diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index a16d889..f02e5e9 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2504,6 +2504,37 @@ PKI; acceptance is the explicit per-DC command on voffice1, and the gate is the literal line `DNS SANs are all in this DC's expected zone`, not the PASS verdict (while unarmed, A12/A13 RECORD rather than fail). +- **F9 IS CLOSED. THE REISSUE WAS EXECUTED 2026-07-30 ON BOTH DCs** (operator-gated per DC, + "Go ahead" each, never batched; dc0 verified clean before dc1 was touched). Capture: + **`docs/audit/octavia-reissue-executed-20260730.txt`**. Ran on voffice1 via Step 1.0-REISSUE + at HEAD `7b6a2e4`. This supersedes the "must be reissued" obligation recorded above. + - **vr1-dc0** -> CN + both DNS SANs `...omega.dc0.vr1.cloud.neumatrix.local`; IP SANs + `10.12.4.57` + `2602:f3e2:f02:11::57`; serial `...250D -> ...250E`. + - **vr1-dc1** -> CN + both DNS SANs `...omega.dc1.vr1.cloud.neumatrix.local`; IP SANs + `10.12.64.57` + `2602:f3e2:f03:11::57`; serial `...674DD -> ...674DE`. (dc1's OLD cert + carried dc0's LABEL as well as the wrong region -- the cross-DC case A11 exists for.) + - Both: FRESH EC P-256 key, public key proven to differ from the outgoing cert's; neither CA + regenerated so the amphora trust domain is untouched; overlay surgery measured + `other_values_identical=4 changed_lines=1 line_count=8`. + - **ACCEPTANCE: `verify` PASS 37/0 on BOTH DCs** (was 29/0 pre-session; +8 from the new + A13/A14/A15/A16/A17), each carrying the LITERAL positive line -- the gate, since a PASS + verdict alone is compatible with a wrong cert while unarmed. **A11 confirms per-DC + independence on both.** `creds-matrix` = the same 5 pre-existing findings, NO new finding + and NO S4 drift (Step 1.0-REISSUE was appended after 1.0-GEN.e precisely to protect the F10 + line anchors). No staging residue; neither run reached exit 5. + - **The hardened re-run guard proved itself on the live tree:** re-running dc0 immediately + after returned exit 4 "already correct", explicitly noting *"and verify returns clean"* -- + the fix for the defect that had it refusing to repair certs `verify` was failing. + - **STILL OWED (operator-gated, NOT done):** backup custody. Each archive holds BOTH CA + private keys, BOTH passphrases and the overlay, and currently sits on the headend at + `~/octavia-pki/backups/`. Step 1.0-REISSUE.4 pulls them to the jumphost creds folder, + sha256-compares, then removes the headend copy. They are also an UNREGISTERED credential + class (no `creds-matrix.tsv` row, no retention rule, so P5 cannot see them) -- logged, not + actioned. And: **never restore one over a workspace that has issued since** (it carries the + CA's issuance state; burned serials today are `...250E` dc0 / `...674DE` dc1). + - The certs remain INERT by design -- `os-public-hostname` is still set nowhere (R5 refused + setting it at Stage 5 as a D-019 repeat). The names are now correct IN ADVANCE of D-106's + Stage-7 work arming them. - **CAPTURE: `docs/audit/docfix205-d117-annotation-20260730.txt`** -- the quoted D-117 Status line, D-119's selector-only discharge, annotation coverage measured 0/4 -> 4/4, the phase-6 expansion old-vs-new run in a real shell, live A12 on the headend, and both hosts' gates diff --git a/docs/audit/octavia-reissue-executed-20260730.txt b/docs/audit/octavia-reissue-executed-20260730.txt new file mode 100644 index 0000000..7befdd5 --- /dev/null +++ b/docs/audit/octavia-reissue-executed-20260730.txt @@ -0,0 +1,126 @@ +OCTAVIA CONTROLLER-CERT REISSUE -- AS EXECUTED, 2026-07-30 +========================================================== +F9 CLOSED on both DCs. Executed on voffice1 (the D-128 Plane-2 headend, where the PKI +lives per the D-109 ruling note (b)), via runbooks/phase-01-bundle-deploy.md Step +1.0-REISSUE, tool scripts/octavia-pki.sh at repo HEAD 7b6a2e4. + +Operator authorisation: "Let's reissue now" (2026-07-30), then an explicit "Go ahead" +per DC -- dc0 and dc1 gated SEPARATELY, never batched, with dc0 verified clean before +dc1 was touched. + +SCOPE ACTUALLY EXECUTED (matches the ruling exactly) +---------------------------------------------------- +Controller LEAF certificate + its private key ONLY, signed by the EXISTING controller CA. +NEITHER CA regenerated -- the amphora trust domain is untouched, so anything already +issued against it stays valid. Exactly ONE overlay value rewritten per DC. + +WHAT WAS WRONG, AND WHY (the defect this closes) +------------------------------------------------ +Both DCs' controller certs carried CN and both DNS SANs = `...omega.dc0.vr0...` -- the +VR0 region, from a literal paste in 1.0-GEN.c. On vr1-dc1 the DC LABEL was wrong too +(dc0 on dc1's certificate). Inert only because os-public-hostname is set in no deploy +artifact (bundle.yaml posture B5, IP-only), and A12/A13 arm themselves the moment that +changes. Correct names are DERIVED, never typed: region = ${SITE%%-*}, dc = ${SITE#*-}, +zone = omega...cloud.neumatrix.local (D-008 shape, D-117 labels). + +RESULT -- vr1-dc0 +----------------- + CN: octavia-controller.omega.dc0.vr1.cloud.neumatrix.local + DNS SANs: octavia-controller.omega.dc0.vr1.cloud.neumatrix.local + octavia.omega.dc0.vr1.cloud.neumatrix.local + IP SANs: 10.12.4.57, 2602:f3e2:f02:11::57 (derived from overlays/vr1-dc0-vips.yaml) + serial: 48919FCC728FFAC08D74758AE95A9D29956F250D -> ...250E + CA serial file advanced in step (consumed, not invented) + key: FRESH EC P-256; public key DIFFERS from the outgoing certificate's (asserted) + overlay: other_values_identical=4 changed_lines=1 line_count=8 + backup: ~/octavia-pki/backups/vr1-dc0-octavia-pki-20260730T070146Z-1520866.tar.gz + sha256: e30cab7f512c97a8a07eba052568859dd5f0e9540a1b9a104f6f23d12e3f3487 + +RESULT -- vr1-dc1 +----------------- + CN: octavia-controller.omega.dc1.vr1.cloud.neumatrix.local + DNS SANs: octavia-controller.omega.dc1.vr1.cloud.neumatrix.local + octavia.omega.dc1.vr1.cloud.neumatrix.local + IP SANs: 10.12.64.57, 2602:f3e2:f03:11::57 (derived from overlays/vr1-dc1-vips.yaml) + serial: 480F90BCD27F24EF96B6991516E818FB170674DD -> ...674DE + CA serial file advanced in step + key: FRESH EC P-256; public key DIFFERS from the outgoing certificate's (asserted) + overlay: other_values_identical=4 changed_lines=1 line_count=8 + backup: ~/octavia-pki/backups/vr1-dc1-octavia-pki-20260730T070905Z-1522295.tar.gz + sha256: 17c831da91e34283e89f9bb7253c69ae69477c5bc1d3807eed136290a1c96fe3 + +ACCEPTANCE -- AND THE GATE IS THE LINE, NOT THE VERDICT +------------------------------------------------------- +A PASS verdict alone would NOT have been acceptance. While unarmed (os-public-hostname +unset), A12/A13 RECORD rather than fail, so a wrong-zone certificate also reports PASS -- +it merely carries a "(recorded, not failed)" line instead of the positive one. The gate +is the literal positive, and it is now present on both DCs: + + vr1-dc0: ok A12 DNS SANs are all in this DC's expected zone 'omega.dc0.vr1.cloud.neumatrix.local' + ok A13 controller cert CN is 'octavia-controller.omega.dc0.vr1.cloud.neumatrix.local' + vr1-dc1: ok A12 DNS SANs are all in this DC's expected zone 'omega.dc1.vr1.cloud.neumatrix.local' + ok A13 controller cert CN is 'octavia-controller.omega.dc1.vr1.cloud.neumatrix.local' + + octavia-pki verify (vr1-dc0): PASS -- 37 assertion(s), 0 failed + octavia-pki verify (vr1-dc1): PASS -- 37 assertion(s), 0 failed + (was 29/0 before this session; +8 from the new A13/A14/A15/A16/A17) + + A11 all 3 compared artifacts differ from the other DC -- per-DC independence HOLDS, + on BOTH DCs. This is the assertion that would have caught a cross-DC mix-up, which + was the most plausible failure here because dc1's OLD cert carried dc0's label. + + A14 cert<->key pairing and bundle integrity: PASS both DCs (the bundle's CERTIFICATE + and PRIVATE KEY blocks are byte-identical to their files). + A15 keyUsage (critical: digitalSignature, keyEncipherment) + EKU (clientAuth, + serverAuth) present ON THE ISSUED CERT: PASS both DCs. + A16 valid, not expiring within 30 days: PASS both DCs. + A17 the overlay's values decode byte-identically to this workspace's bundle and + controller CA: PASS both DCs -- workspace and deploy input agree. + +SECOND GATE (they check different things and have disagreed in BOTH directions) +------------------------------------------------------------------------------- + python3 scripts/creds-matrix.py -> 99 rows, 8 check groups clean, 5 findings. + ALL FIVE ARE PRE-EXISTING AND NAMED, none attributable to this change: + S2 dc0-edge-api 'opnsense-api.txt' (SEC-021) + S5 x3 power-key asymmetries (id_ed25519, maas-virsh_ed25519, id_dcN_power) + S6 maas-region-admin principal conflation + NO S4 findings -- the mint-ref line anchors survived appending Step 1.0-REISSUE + (which is WHY it was appended AFTER 1.0-GEN.e; F10 records that S4 only checks + within-EOF and cannot see an anchor that has drifted onto unrelated text). + +HYGIENE +------- + Staging residue: NONE (no ~/octavia-pki/.reissue-* directories left behind). + Both runs exited 0. Neither exit 5 (PARTIAL) nor any rollback was reached. + +THE RE-RUN GUARD, PROVEN ON THE LIVE TREE +----------------------------------------- +Immediately after dc0 succeeded, re-running it returned: + ok re-run guard: CN and both DNS SANs already match the derived zone + 'omega.dc0.vr1.cloud.neumatrix.local', and verify returns clean + REFUSE (already correct) -- nothing to fix, so nothing is rotated. [exit 4] +Note "AND verify returns clean". The first version of this guard compared NAMES only, +and an adversarial review measured it REFUSING to fix a certificate that `verify` FAILED +(names right, IP SAN stale) -- the remedy refusing the defect the checker had just +reported. It would not have bitten today; it would have bitten at the 2-year renewal +this reissue starts the clock on. + +STANDING OBLIGATIONS THIS LEAVES +-------------------------------- +1. BACKUP CUSTODY -- NOT YET DONE, operator-gated. Each archive holds BOTH CA private + keys, BOTH passphrases and the overlay. Per Step 1.0-REISSUE.4 they should be pulled + to the jumphost per-DC creds folder, sha256-compared, and only then removed from the + headend. Until that happens the headend carries two unregistered credential archives. +2. THE ARCHIVES ARE AN UNREGISTERED CREDENTIAL CLASS. No creds-matrix.tsv row, no + retention rule, so P5 cannot see them. Logged, not actioned (hard rule 1) -- it needs + its own gated change. +3. NEVER RESTORE AN ARCHIVE OVER A WORKSPACE THAT HAS ISSUED SINCE. It carries the CA's + issuance state (controller-ca.cert.srl), so restoring rolls the serial backwards and + the next mint reuses a serial the estate already holds. If a post-promotion rollback + is ever performed, RECORD the burned serial -- the two burned today are ...250E (dc0) + and ...674DE (dc1). +4. octavia-pki.sh is invoked by NO gate; preflight.sh checks only that the overlay + EXISTS. "Gauntlet ALL GREEN" is still not evidence about the live PKI. +5. F9's obligation is now CLOSED, but the certs remain INERT by design: os-public-hostname + is still set nowhere (R5 refused setting it at Stage 5 as a D-019 repeat). A12/A13 arm + themselves when D-106's Stage-7 work sets it; the names are now correct in advance.