OCTAVIA CONTROLLER-CERT REISSUE -- AS EXECUTED, 2026-07-30 ========================================================== F9 CLOSED on both DCs. Executed on voffice1 (the D-128 Plane-2 headend, where the PKI lives per the D-109 ruling note (b)), via runbooks/phase-01-bundle-deploy.md Step 1.0-REISSUE, tool scripts/octavia-pki.sh at repo HEAD 7b6a2e4. Operator authorisation: "Let's reissue now" (2026-07-30), then an explicit "Go ahead" per DC -- dc0 and dc1 gated SEPARATELY, never batched, with dc0 verified clean before dc1 was touched. SCOPE ACTUALLY EXECUTED (matches the ruling exactly) ---------------------------------------------------- Controller LEAF certificate + its private key ONLY, signed by the EXISTING controller CA. NEITHER CA regenerated -- the amphora trust domain is untouched, so anything already issued against it stays valid. Exactly ONE overlay value rewritten per DC. WHAT WAS WRONG, AND WHY (the defect this closes) ------------------------------------------------ Both DCs' controller certs carried CN and both DNS SANs = `...omega.dc0.vr0...` -- the VR0 region, from a literal paste in 1.0-GEN.c. On vr1-dc1 the DC LABEL was wrong too (dc0 on dc1's certificate). Inert only because os-public-hostname is set in no deploy artifact (bundle.yaml posture B5, IP-only), and A12/A13 arm themselves the moment that changes. Correct names are DERIVED, never typed: region = ${SITE%%-*}, dc = ${SITE#*-}, zone = omega...cloud.neumatrix.local (D-008 shape, D-117 labels). RESULT -- vr1-dc0 ----------------- CN: octavia-controller.omega.dc0.vr1.cloud.neumatrix.local DNS SANs: octavia-controller.omega.dc0.vr1.cloud.neumatrix.local octavia.omega.dc0.vr1.cloud.neumatrix.local IP SANs: 10.12.4.57, 2602:f3e2:f02:11::57 (derived from overlays/vr1-dc0-vips.yaml) serial: 48919FCC728FFAC08D74758AE95A9D29956F250D -> ...250E CA serial file advanced in step (consumed, not invented) key: FRESH EC P-256; public key DIFFERS from the outgoing certificate's (asserted) overlay: other_values_identical=4 changed_lines=1 line_count=8 backup: ~/octavia-pki/backups/vr1-dc0-octavia-pki-20260730T070146Z-1520866.tar.gz sha256: e30cab7f512c97a8a07eba052568859dd5f0e9540a1b9a104f6f23d12e3f3487 RESULT -- vr1-dc1 ----------------- CN: octavia-controller.omega.dc1.vr1.cloud.neumatrix.local DNS SANs: octavia-controller.omega.dc1.vr1.cloud.neumatrix.local octavia.omega.dc1.vr1.cloud.neumatrix.local IP SANs: 10.12.64.57, 2602:f3e2:f03:11::57 (derived from overlays/vr1-dc1-vips.yaml) serial: 480F90BCD27F24EF96B6991516E818FB170674DD -> ...674DE CA serial file advanced in step key: FRESH EC P-256; public key DIFFERS from the outgoing certificate's (asserted) overlay: other_values_identical=4 changed_lines=1 line_count=8 backup: ~/octavia-pki/backups/vr1-dc1-octavia-pki-20260730T070905Z-1522295.tar.gz sha256: 17c831da91e34283e89f9bb7253c69ae69477c5bc1d3807eed136290a1c96fe3 ACCEPTANCE -- AND THE GATE IS THE LINE, NOT THE VERDICT ------------------------------------------------------- A PASS verdict alone would NOT have been acceptance. While unarmed (os-public-hostname unset), A12/A13 RECORD rather than fail, so a wrong-zone certificate also reports PASS -- it merely carries a "(recorded, not failed)" line instead of the positive one. The gate is the literal positive, and it is now present on both DCs: vr1-dc0: ok A12 DNS SANs are all in this DC's expected zone 'omega.dc0.vr1.cloud.neumatrix.local' ok A13 controller cert CN is 'octavia-controller.omega.dc0.vr1.cloud.neumatrix.local' vr1-dc1: ok A12 DNS SANs are all in this DC's expected zone 'omega.dc1.vr1.cloud.neumatrix.local' ok A13 controller cert CN is 'octavia-controller.omega.dc1.vr1.cloud.neumatrix.local' octavia-pki verify (vr1-dc0): PASS -- 37 assertion(s), 0 failed octavia-pki verify (vr1-dc1): PASS -- 37 assertion(s), 0 failed (was 29/0 before this session; +8 from the new A13/A14/A15/A16/A17) A11 all 3 compared artifacts differ from the other DC -- per-DC independence HOLDS, on BOTH DCs. This is the assertion that would have caught a cross-DC mix-up, which was the most plausible failure here because dc1's OLD cert carried dc0's label. A14 cert<->key pairing and bundle integrity: PASS both DCs (the bundle's CERTIFICATE and PRIVATE KEY blocks are byte-identical to their files). A15 keyUsage (critical: digitalSignature, keyEncipherment) + EKU (clientAuth, serverAuth) present ON THE ISSUED CERT: PASS both DCs. A16 valid, not expiring within 30 days: PASS both DCs. A17 the overlay's values decode byte-identically to this workspace's bundle and controller CA: PASS both DCs -- workspace and deploy input agree. SECOND GATE (they check different things and have disagreed in BOTH directions) ------------------------------------------------------------------------------- python3 scripts/creds-matrix.py -> 99 rows, 8 check groups clean, 5 findings. ALL FIVE ARE PRE-EXISTING AND NAMED, none attributable to this change: S2 dc0-edge-api 'opnsense-api.txt' (SEC-021) S5 x3 power-key asymmetries (id_ed25519, maas-virsh_ed25519, id_dcN_power) S6 maas-region-admin principal conflation NO S4 findings -- the mint-ref line anchors survived appending Step 1.0-REISSUE (which is WHY it was appended AFTER 1.0-GEN.e; F10 records that S4 only checks within-EOF and cannot see an anchor that has drifted onto unrelated text). HYGIENE ------- Staging residue: NONE (no ~/octavia-pki/.reissue-* directories left behind). Both runs exited 0. Neither exit 5 (PARTIAL) nor any rollback was reached. THE RE-RUN GUARD, PROVEN ON THE LIVE TREE ----------------------------------------- Immediately after dc0 succeeded, re-running it returned: ok re-run guard: CN and both DNS SANs already match the derived zone 'omega.dc0.vr1.cloud.neumatrix.local', and verify returns clean REFUSE (already correct) -- nothing to fix, so nothing is rotated. [exit 4] Note "AND verify returns clean". The first version of this guard compared NAMES only, and an adversarial review measured it REFUSING to fix a certificate that `verify` FAILED (names right, IP SAN stale) -- the remedy refusing the defect the checker had just reported. It would not have bitten today; it would have bitten at the 2-year renewal this reissue starts the clock on. STANDING OBLIGATIONS THIS LEAVES -------------------------------- 1. BACKUP CUSTODY -- NOT YET DONE, operator-gated. Each archive holds BOTH CA private keys, BOTH passphrases and the overlay. Per Step 1.0-REISSUE.4 they should be pulled to the jumphost per-DC creds folder, sha256-compared, and only then removed from the headend. Until that happens the headend carries two unregistered credential archives. 2. THE ARCHIVES ARE AN UNREGISTERED CREDENTIAL CLASS. No creds-matrix.tsv row, no retention rule, so P5 cannot see them. Logged, not actioned (hard rule 1) -- it needs its own gated change. 3. NEVER RESTORE AN ARCHIVE OVER A WORKSPACE THAT HAS ISSUED SINCE. It carries the CA's issuance state (controller-ca.cert.srl), so restoring rolls the serial backwards and the next mint reuses a serial the estate already holds. If a post-promotion rollback is ever performed, RECORD the burned serial -- the two burned today are ...250E (dc0) and ...674DE (dc1). 4. octavia-pki.sh is invoked by NO gate; preflight.sh checks only that the overlay EXISTS. "Gauntlet ALL GREEN" is still not evidence about the live PKI. 5. F9's obligation is now CLOSED, but the certs remain INERT by design: os-public-hostname is still set nowhere (R5 refused setting it at Stage 5 as a D-019 repeat). A12/A13 arm themselves when D-106's Stage-7 work sets it; the names are now correct in advance.