OCTAVIA CONTROLLER-CERT REISSUE -- AS EXECUTED, 2026-07-30
==========================================================
F9 CLOSED on both DCs. Executed on voffice1 (the D-128 Plane-2 headend, where the PKI
lives per the D-109 ruling note (b)), via runbooks/phase-01-bundle-deploy.md Step
1.0-REISSUE, tool scripts/octavia-pki.sh at repo HEAD 7b6a2e4.
Operator authorisation: "Let's reissue now" (2026-07-30), then an explicit "Go ahead"
per DC -- dc0 and dc1 gated SEPARATELY, never batched, with dc0 verified clean before
dc1 was touched.
SCOPE ACTUALLY EXECUTED (matches the ruling exactly)
----------------------------------------------------
Controller LEAF certificate + its private key ONLY, signed by the EXISTING controller CA.
NEITHER CA regenerated -- the amphora trust domain is untouched, so anything already
issued against it stays valid. Exactly ONE overlay value rewritten per DC.
WHAT WAS WRONG, AND WHY (the defect this closes)
------------------------------------------------
Both DCs' controller certs carried CN and both DNS SANs = `...omega.dc0.vr0...` -- the
VR0 region, from a literal paste in 1.0-GEN.c. On vr1-dc1 the DC LABEL was wrong too
(dc0 on dc1's certificate). Inert only because os-public-hostname is set in no deploy
artifact (bundle.yaml posture B5, IP-only), and A12/A13 arm themselves the moment that
changes. Correct names are DERIVED, never typed: region = ${SITE%%-*}, dc = ${SITE#*-},
zone = omega.<dc>.<region>.cloud.neumatrix.local (D-008 shape, D-117 labels).
RESULT -- vr1-dc0
-----------------
CN: octavia-controller.omega.dc0.vr1.cloud.neumatrix.local
DNS SANs: octavia-controller.omega.dc0.vr1.cloud.neumatrix.local
octavia.omega.dc0.vr1.cloud.neumatrix.local
IP SANs: 10.12.4.57, 2602:f3e2:f02:11::57 (derived from overlays/vr1-dc0-vips.yaml)
serial: 48919FCC728FFAC08D74758AE95A9D29956F250D -> ...250E
CA serial file advanced in step (consumed, not invented)
key: FRESH EC P-256; public key DIFFERS from the outgoing certificate's (asserted)
overlay: other_values_identical=4 changed_lines=1 line_count=8
backup: ~/octavia-pki/backups/vr1-dc0-octavia-pki-20260730T070146Z-1520866.tar.gz
sha256: e30cab7f512c97a8a07eba052568859dd5f0e9540a1b9a104f6f23d12e3f3487
RESULT -- vr1-dc1
-----------------
CN: octavia-controller.omega.dc1.vr1.cloud.neumatrix.local
DNS SANs: octavia-controller.omega.dc1.vr1.cloud.neumatrix.local
octavia.omega.dc1.vr1.cloud.neumatrix.local
IP SANs: 10.12.64.57, 2602:f3e2:f03:11::57 (derived from overlays/vr1-dc1-vips.yaml)
serial: 480F90BCD27F24EF96B6991516E818FB170674DD -> ...674DE
CA serial file advanced in step
key: FRESH EC P-256; public key DIFFERS from the outgoing certificate's (asserted)
overlay: other_values_identical=4 changed_lines=1 line_count=8
backup: ~/octavia-pki/backups/vr1-dc1-octavia-pki-20260730T070905Z-1522295.tar.gz
sha256: 17c831da91e34283e89f9bb7253c69ae69477c5bc1d3807eed136290a1c96fe3
ACCEPTANCE -- AND THE GATE IS THE LINE, NOT THE VERDICT
-------------------------------------------------------
A PASS verdict alone would NOT have been acceptance. While unarmed (os-public-hostname
unset), A12/A13 RECORD rather than fail, so a wrong-zone certificate also reports PASS --
it merely carries a "(recorded, not failed)" line instead of the positive one. The gate
is the literal positive, and it is now present on both DCs:
vr1-dc0: ok A12 DNS SANs are all in this DC's expected zone 'omega.dc0.vr1.cloud.neumatrix.local'
ok A13 controller cert CN is 'octavia-controller.omega.dc0.vr1.cloud.neumatrix.local'
vr1-dc1: ok A12 DNS SANs are all in this DC's expected zone 'omega.dc1.vr1.cloud.neumatrix.local'
ok A13 controller cert CN is 'octavia-controller.omega.dc1.vr1.cloud.neumatrix.local'
octavia-pki verify (vr1-dc0): PASS -- 37 assertion(s), 0 failed
octavia-pki verify (vr1-dc1): PASS -- 37 assertion(s), 0 failed
(was 29/0 before this session; +8 from the new A13/A14/A15/A16/A17)
A11 all 3 compared artifacts differ from the other DC -- per-DC independence HOLDS,
on BOTH DCs. This is the assertion that would have caught a cross-DC mix-up, which
was the most plausible failure here because dc1's OLD cert carried dc0's label.
A14 cert<->key pairing and bundle integrity: PASS both DCs (the bundle's CERTIFICATE
and PRIVATE KEY blocks are byte-identical to their files).
A15 keyUsage (critical: digitalSignature, keyEncipherment) + EKU (clientAuth,
serverAuth) present ON THE ISSUED CERT: PASS both DCs.
A16 valid, not expiring within 30 days: PASS both DCs.
A17 the overlay's values decode byte-identically to this workspace's bundle and
controller CA: PASS both DCs -- workspace and deploy input agree.
SECOND GATE (they check different things and have disagreed in BOTH directions)
-------------------------------------------------------------------------------
python3 scripts/creds-matrix.py -> 99 rows, 8 check groups clean, 5 findings.
ALL FIVE ARE PRE-EXISTING AND NAMED, none attributable to this change:
S2 dc0-edge-api 'opnsense-api.txt' (SEC-021)
S5 x3 power-key asymmetries (id_ed25519, maas-virsh_ed25519, id_dcN_power)
S6 maas-region-admin principal conflation
NO S4 findings -- the mint-ref line anchors survived appending Step 1.0-REISSUE
(which is WHY it was appended AFTER 1.0-GEN.e; F10 records that S4 only checks
within-EOF and cannot see an anchor that has drifted onto unrelated text).
HYGIENE
-------
Staging residue: NONE (no ~/octavia-pki/.reissue-* directories left behind).
Both runs exited 0. Neither exit 5 (PARTIAL) nor any rollback was reached.
THE RE-RUN GUARD, PROVEN ON THE LIVE TREE
-----------------------------------------
Immediately after dc0 succeeded, re-running it returned:
ok re-run guard: CN and both DNS SANs already match the derived zone
'omega.dc0.vr1.cloud.neumatrix.local', and verify returns clean
REFUSE (already correct) -- nothing to fix, so nothing is rotated. [exit 4]
Note "AND verify returns clean". The first version of this guard compared NAMES only,
and an adversarial review measured it REFUSING to fix a certificate that `verify` FAILED
(names right, IP SAN stale) -- the remedy refusing the defect the checker had just
reported. It would not have bitten today; it would have bitten at the 2-year renewal
this reissue starts the clock on.
STANDING OBLIGATIONS THIS LEAVES
--------------------------------
1. BACKUP CUSTODY -- NOT YET DONE, operator-gated. Each archive holds BOTH CA private
keys, BOTH passphrases and the overlay. Per Step 1.0-REISSUE.4 they should be pulled
to the jumphost per-DC creds folder, sha256-compared, and only then removed from the
headend. Until that happens the headend carries two unregistered credential archives.
2. THE ARCHIVES ARE AN UNREGISTERED CREDENTIAL CLASS. No creds-matrix.tsv row, no
retention rule, so P5 cannot see them. Logged, not actioned (hard rule 1) -- it needs
its own gated change.
3. NEVER RESTORE AN ARCHIVE OVER A WORKSPACE THAT HAS ISSUED SINCE. It carries the CA's
issuance state (controller-ca.cert.srl), so restoring rolls the serial backwards and
the next mint reuses a serial the estate already holds. If a post-promotion rollback
is ever performed, RECORD the burned serial -- the two burned today are ...250E (dc0)
and ...674DE (dc1).
4. octavia-pki.sh is invoked by NO gate; preflight.sh checks only that the overlay
EXISTS. "Gauntlet ALL GREEN" is still not evidence about the live PKI.
5. F9's obligation is now CLOSED, but the certs remain INERT by design: os-public-hostname
is still set nowhere (R5 refused setting it at Stage 5 as a D-019 repeat). A12/A13 arm
themselves when D-106's Stage-7 work sets it; the names are now correct in advance.