Changelog 2026-07-30 (part 2) -- the Octavia controller-cert REISSUE tool, hardened

Session changelog part 2 (part 1 = changelog-20260730-docfix205-d117-annotation.md). Branch dc-dc-stage5-preconditions. Status claims live ONLY in docs/CURRENT-STATE.md.

Trigger. Operator: "Let's reissue now. Engineer the artifact, variable, based workflow to reduce the chance of errors. Deploy agents to engineer, code, and valid the coding." F9's obligation: the controller certs on BOTH DCs carry dc0.vr0 names -- the wrong region on both, and the wrong DC label on dc1.

No new D-number (GA-R3). The naming is ruled (D-008 shape + D-117 labels); the tool is OPS. Next-free UNCHANGED: D 138 / DOCFIX 206 / BUNDLEFIX 053.

Operator rulings recorded (GA-R5, one exchange each, verbatim)

1. SCOPE -- "Controller cert only (Recommended)". Signed by the EXISTING controller CA; neither CA regenerated; only lb-mgmt-controller-cert changes in the overlay.
2. KEY -- "Fresh P-256 key (Recommended)".
3. ARTIFACT -- "Script subcommand + runbook step (Recommended)".
4. RE-RUN GUARD -- "Refuse unless --force (Recommended)".
5. MINT PATH -- "Full script minter now", taken after I escalated a measured concern (below).
6. Scope of this batch -- "Yes, they all need to be processed and the hardened tool driven process needs to be documented and saved for future stand ups."

The escalation, and its correction -- recorded because I got the framing wrong first

Presenting ruling 3 I told the operator a reissue subcommand "does not touch generate, so the D-137 fork-1 ruling stays untouched." That was wrong twice over.

First, measured: guard-destructive.py is registered "matcher": "Bash" and inspects only the command STRING, keyed on .pem. Empirically -- bash scripts/octavia-pki.sh reissue vr1-dc0 ALLOWED, openssl genpkey ... -out controller.key ALLOWED, only the .pem shape BLOCKED. So the guard has no opinion on ANY script wrapping openssl, and the subcommand's NAME is irrelevant. I escalated on that basis.

Second, and this is the correction: D-137 fork 1 is NOT open. Sub-ruling 1 RULED it 2026-07-25 at option (b) "Blocking in preflight", explicitly recording "NOT adopted: the PreToolUse guard (so an agent is not constrained at write time, only at the deploy gate)". guard-destructive.py is a separate, older mechanism (CLAUDE.md secrets norms / DOCFIX-006), not D-137's enforcement point. A scripted minter therefore decides nothing unruled -- it is consistent with what D-137 actually ruled. docs/audit/queued-findings-20260730.txt Q2 calls fork 1 "still unruled", which is the same stale-premise class as Q1 in part 1 of this session. Corrected there by appended note.

1. scripts/octavia-pki.sh -- the reissue action

reissue <site> [--force] [--dry-run]. Exit codes: 0 ok/dry-run, 1 FAIL (measured defect or a staged assertion failed -- nothing promoted), 2 args, 3 REFUSE (could not evaluate / could not make it safe), 4 REFUSE already-correct, 5 PARTIAL (promotion interrupted -- restore command printed). verify's 0/1/2/3 semantics unchanged.

Shape, in order: precondition sweep (host authority FIRST, before any file is created) -> derive -> re-run guard -> backup -> printf-built config -> structural config assertions -> mint into STAGING -> post-mint assertions in staging -> promote -> single-value overlay surgery -> prove the surgery -> final verify.

Properties that matter, each chosen against a measured failure:

• Zone DERIVED, never typed (${SITE%%-*} / ${SITE#*-}), so vr1-dc0 -> dc0.vr1 and vr1-dc1 -> dc1.vr1. Same code path verify uses -- generator and gate cannot drift.
• printf per line, no heredoc -- the F8 payoff. Values passed as %s ARGUMENTS so a % in a value cannot become a format directive.
• Stage, assert, promote. 1.0-GEN.c overwrites the key BEFORE signing with no set -e, so any mid-way failure yields a bundle whose cert and key do not match. Nothing detected that.
• -CAserial, NOT -CAcreateserial -- with -CAserial alone openssl FAILS on a missing serial file instead of silently starting a new random sequence.
• Overlay surgery via python3 reading both files itself -- never sed/awk with the value in argv, because the bundle holds an UNENCRYPTED private key and /proc/<pid>/cmdline is world-readable.

Revert: git checkout <prev> -- scripts/octavia-pki.sh.

2. Three verify gaps closed, then three MORE found by validation

Closed in the first pass: A13 (controller cert subject CN -- asserted nowhere before; check_subject ran only on the two CAs), A14 (cert<->key pairing + bundle integrity -- nothing read the bundle Octavia actually consumes), and a symmetric positive in A12's inert branch (a correct and an incorrect cert were previously distinguishable only by assertion count).

Then adversarial + mutation agents found three more, all now closed:

• A15 -- keyUsage / EKU on the issued cert. Deleting BOTH extension lines produced a cert with no KU and no EKU that minted, promoted, and left the harness fully green. keyUsage appeared exactly ONCE in the whole file -- the printf that writes it. Losing clientAuth breaks amphora->controller mTLS ONLY: a one-directional failure at LB-create time with every gate reading PASS.
• A16 -- validity. openssl x509 -req defaults to -days 30. checkend, notAfter and enddate appeared ZERO times in the verify half, so a 30-day controller certificate passed every gate in this repo forever. Reachable by dropping one token from the hand-pasted block that produced the live certs.
• A17 -- the overlay must carry the material the workspace holds. A10 graded the overlay's SHAPE and A1-A16 graded the workspace; nothing joined them. Measured: an overlay whose embedded cert decoded to a DIFFERENT bundle reported PASS, 0 failed. The overlay is what reaches the charm.

Also hardened: A8's negative proves its instrument first -- openssl verify -CAfile X exits non-zero both for "does not verify" and "could not load X", so a corrupt issuing CA made the trust-domain-separation check pass VACUOUSLY (measured: it printed the reassuring line and promoted).

3. The re-run guard was refusing to fix broken certificates

The most consequential validation finding. The guard compared NAMES and treated that as "is the certificate right?". Measured: a cert with correct CN/SANs but a stale IP SAN made verify FAIL while reissue exited 4 "nothing to fix" -- the remedy refusing the defect the checker had just reported, same workspace, seconds apart. It does not bite today (the names ARE wrong, so it proceeds) but it bites every run afterwards, including the 2-year renewal this reissue starts the clock on, where an EXPIRED cert with correct names would be refused as already-correct.

Now "already correct" means names match AND verify returns clean, so the two cannot disagree -- and A16/A17 put expiry and overlay desync inside that definition.

4. Two pre-existing overlay defects moved to pre-mint

Both are pure functions of state that exists BEFORE the run, but were detected only by the post-surgery proof -- producing exit 5 (workspace promoted, serial burned) when the honest answer was "your overlay was already wrong, nothing was touched": (a) exactly one lb-mgmt-controller-cert line; (b) lb-mgmt-controller-cacert must already decode to THIS DC's controller CA (an overlay from another DC is the F1 cross-DC mix-up).

5. Smaller hardening, each from a measured failure

• Backup filename gains $$ -- the 1-second stamp plus tar czf overwriting was measured destroying 3 of 12 pre-images, worst exactly in an exit-5-then-retry sequence.
• Leftover .reissue-<site>.* staging dirs are REPORTED (not deleted): SIGKILL between signing and promotion leaves a valid CA-signed cert and an unencrypted key that no register declares.
• Two overstated in-file claims corrected: the "hoist is PURE, proven by diffing" comment was backed by nothing executable; and "the other four values are byte-identical" is a line-diff result, which proves the RESULT unchanged, not that they were never rewritten.

6. tests/octavia-pki/run-tests.sh -- 21/21 -> 48/48

T22-T36 cover reissue. T37-T44 exist because mutation testing proved the new assertions could not fail: deleting A13, A14 or the A12 positive outright left the suite fully green. An assertion with no failing-direction fixture is decoration.

T37/T37b (no KU/EKU; partial EKU loss), T38 (short-dated cert), T39/T39b (overlay desync; empty value), T40 (cert/key mismatch), T41 (A13 armed, correct SANs + wrong CN -- the half-fix), T42 (the inert positive must not be vacuous -- a SAN-less cert must not earn it), T43 (a stale THIRD DNS SAN must be reissued down to two; mutation refuted the assumption that T25 covered the count guard), T44 (names-correct-but-verify-FAILS must reissue, not refuse).

Revert: git checkout <prev> -- tests/octavia-pki/run-tests.sh.

7. runbooks/phase-01-bundle-deploy.md -- Step 1.0-REISSUE

Appended AFTER 1.0-GEN.e deliberately: F10 records that creds-matrix S4 mint-refs are line-anchored and only checked within-EOF, so inserting above would have silently misaligned 20 octavia refs. Verified after the append -- creds-matrix.py reports no S4 findings.

Four sub-steps: read-only decide -> --dry-run -> the mint ONE DC AT A TIME -> verify + backup hygiene. Carries the exit-code table as an action table, and the standing warning that a PASS verdict alone is not acceptance -- while unarmed, A12/A13 RECORD rather than fail, so the gate is the literal line DNS SANs are all in this DC's expected zone.

8. Skill sweep -- saved for future standups (operator-directed)

.claude/skills/openstack-cloud-ops/SKILL.md gains two standing invariants and a routing row:

• EVERY PER-DC SECRET NEEDS A ROTATION TOOL, NOT JUST A GENERATION RECIPE. A generation step guarded by REFUSE-IF-PRESENT leaves no sanctioned way to rotate a leaf, so its only documented escape moves the CAs -- the dangerous path becomes the only path. Carries the four properties that made this tool trustworthy (derive identity; assert the artifact not the config; stage-assert-promote; join workspace to deploy input) and the Roosevelt analog.
• A HAND-PASTED openssl CHAIN IS A DEFECT CLASS, AND THIS REPO HAS THE RECEIPTS -- F8 and F9 from one block; plus "prove each new assertion can FAIL", with the two properties that had no assertion at all (-days defaulting to 30, and a dropped EKU).

Revert: git checkout <prev> -- .claude/skills/openstack-cloud-ops/SKILL.md.

Verification

bash tests/octavia-pki/run-tests.sh   -> 48/48 PASS
bash scripts/repo-lint.sh             -> 0 fail, 1 warn (legacy L1, unchanged)
bash scripts/run-tests-all.sh         -> GAUNTLET: ALL GREEN (89 harnesses)
python3 scripts/creds-matrix.py       -> 5 findings, ALL pre-existing and named
                                         (dc0-edge-api SEC-021, three power-key
                                         asymmetries, maas-region-admin conflation);
                                         NO S4 findings, so mint-ref anchors are intact

NOT done, deliberately

• The mint itself has not been run. This changelog ships the tool; the reissue is a gated operator action per 1.0-REISSUE.
• generate remains unimplemented. reissue REFUSES when there is no existing certificate, precisely so it cannot become a generate backdoor.
• The backup archive is still an unregistered credential class (it holds both CA keys plus passphrases). Logged, not actioned -- it needs a creds-matrix.tsv row and a retention rule, which is its own gated change.
• octavia-pki.sh is still called by NO gate: preflight.sh checks only that the overlay EXISTS. So "gauntlet ALL GREEN" remains not-evidence about the live PKI. Wiring verify into preflight is a gate change with its own blast radius -- proposed, not smuggled in here.

PART 3 (same session) -- custody, register, and F8/F9 fixed AT SOURCE in GEN.c

Operator: "Update the matrix as needed. Transfer the artifacts to the vcloud. keep the pin that during the secrets workflow planning that the creds and certs from this step need to be included. continue on with the rest of the steps."

9. Backup custody executed (Step 1.0-REISSUE.4)

Both pre-reissue archives pulled from the headend to the jumphost SEC-009 creds folders, sha256 compared and identical at both ends before anything was deleted (a truncated scp would otherwise leave a verified-looking backup of nothing). 0600; 15 entries each including the deploy overlay and the CA serial. Headend ~/octavia-pki/backups/ removed entirely; no .reissue-* staging residue.

10. The archives are now REGISTERED (they were an unregistered credential class)

Two octavia-reissue-backup rows (per-DC, jumphost, consolidated, templated filename <site>-octavia-pki-<stamp>.tar.gz), the DERIVED manifests updated to match (D-137 ruling 2 -- manifests are generated, never hand-authored), and a new n-reissue-backup note.

Deliberately a SEPARATE id from n-pki-backup: that one is the generation-time workspace archive, this is per-ROTATION and additionally carries the deploy overlay, which the generation archive does not. Restoring only the workspace would leave the half that reaches the charm unrecoverable.

Measured after: tier 1 = 101 rows, the SAME 5 pre-existing findings; tier 2 = the same 7, with no reissue-attributable finding -- and tier 2 FOUND both archives at their declared location, which is the point of registering them.

Hazard recorded in the note: these archives contain controller-ca.cert.srl, the CA's ISSUANCE STATE. Restoring one over a workspace that has issued since rolls the serial counter BACKWARDS and the next mint reuses a serial the estate already holds. Serials burned 2026-07-30: ...250E (dc0), ...674DE (dc1).

PINNED (operator-directed): the note carries the standing requirement that the creds and certs from this step be included in the secrets-storage workflow planning, with the full scope enumerated -- both CAs (key, passphrase, cert), the controller LEAF key/cert/bundle, the CA serial state, and the deploy overlay, at BOTH DCs. The jumphost creds folder is INTERIM only.

11. Q2 withdrawn -- the THIRD stale premise in one findings file

docs/audit/queued-findings-20260730.txt Q2 opens "D-137 OPEN FORK 1 -- still unruled". It was ruled 2026-07-25 at option (b), "Blocking in preflight", whose CONSEQUENCE block explicitly declines option (c). Appended correction, not a rewrite. The misfire tally is now SEVEN -- the two newest collected today, when the guard blocked a command whose only purpose was to TEST it, and then blocked the heredoc writing the correction that documents its own misfires.

12. F8 AND F9 FIXED AT SOURCE IN 1.0-GEN.c -- the part that matters for future standups

Everything above repairs the two EXISTING DCs. The generation path still carried both defects, so the next DC standup would have recreated them. Both are now fixed where they originate:

• F9 at source. GEN.c baked omega.dc0.vr0 as a literal in THREE places (CN + both DNS SANs). It now DERIVES DC_ZONE from $DC using the same two expansions the tool uses (${DC%%-*} / ${DC#*-}), so the runbook and the gate cannot disagree, and echoes the zone for confirmation before the sign.
• F8 at source. The cat > controller.cnf <<'CNF' heredoc is replaced by one printf per line, values passed as %s ARGUMENTS. This was explicitly deferred on 2026-07-29 as "an untested rewrite ... reasonable when someone can run a real generation" -- that condition is now met: the identical shape was exercised end to end by two real mints (both DCs) plus 48 harness cases.
• Plus structural assertions before signing, because printf removes the paste hazard but not the failure CLASS: all four sections present, subjectAltName wired, CN equal to the derived zone, exactly 2 DNS entries. A typo'd section name now stops the step instead of producing a SAN-less certificate behind a wall of OK output.
• The stale "NOT changed -- outside R7's ruled scope" note is superseded in place, keeping the reasoning: R7 recorded this cert's SAN as "already DERIVED per-DC by design", which was true of the IP SAN and NOT the DNS names -- a claim accurate about one half of a field, read as covering both. That is how F9 survived.

F10 handled deliberately. Editing GEN.c shifts every mint-ref anchored below it. All 13 octavia anchors were re-resolved by MARKER in a SINGLE PASS keyed by row id (never sequential seds -- a line number can be simultaneously an old value for one row and a new value for another), then each verified to point at its correct command. 12 rows rewritten; creds-matrix S4 reports clean. The runbook block was bash -n syntax-checked as extracted.

Revert: git checkout <prev> -- runbooks/phase-01-bundle-deploy.md creds-matrix.tsv creds-matrix-notes.md creds-manifests/vr1-dc0.manifest creds-manifests/vr1-dc1.manifest.

Verification (part 3)

bash tests/octavia-pki/run-tests.sh   -> 48/48 PASS
bash tests/creds-matrix/run-tests.sh  -> 65/65 PASS
python3 scripts/creds-matrix.py       -> 101 rows, 5 findings (all pre-existing), S4 CLEAN
bash scripts/run-tests-all.sh         -> GAUNTLET: ALL GREEN (89)
bash scripts/repo-lint.sh             -> 0 fail

PART 4 (same session) -- P7: the Octavia PKI becomes an actual GATE

Operator: "Yes, I accept the recommendation. Convert to a gate."

13. scripts/preflight.sh gains P7

THE GAP. scripts/octavia-pki.sh was invoked by NOTHING executable. P4's CHECK 0 asserts only that overlays/<dc>-octavia-pki.yaml EXISTS; the octavia-pki harness runs against throwaway fixtures. So "gauntlet ALL GREEN" said nothing whatever about the live PKI, and the only real check ran when a human remembered to type it. That is how F9 survived for weeks.

Three design problems had to be solved, each from a measured precedent:

1. rc 3 must not reach note()'s *) arm. octavia-pki.sh uses exit 3 for REFUSE, but note() treats any rc>=3 as "UNEXPECTED exit code (127=command not found...)". Wiring it naively would report a legitimate wrong-host refusal as an unknown failure. P7 maps rc explicitly: 0 -> evaluate the output, 1 -> FAIL, 3 -> FAIL named as an unevaluated PKI, anything else -> FAIL.
2. Host authority. The PKI lives on the headend (D-109 ruling note (b)). On any other host P7 reports NOT EVALUATED (WARN) -- never a silent pass, and never a hard FAIL either, since running preflight from the jumphost for other purposes is legitimate. The binding is read from the SAME creds-manifests/host-identity that octavia-pki.sh reads, so the two cannot drift. This is P5's own F6 defect (2026-07-30: probing the wrong host's filesystem and reporting it as fact, 34 findings against a true 7) designed out rather than repeated.
3. A PASS VERDICT IS NOT SUFFICIENT, and this is the subtle one. While os-public-hostname is unset (B5 IP-only), A12/A13 RECORD a wrong-zone certificate rather than failing it -- correctly, since nothing resolves those names yet. So verify exits 0 on exactly the defect this gate exists to catch. P7 therefore also requires the literal line DNS SANs are all in this DC's expected zone, which is emitted only when every DNS SAN is in that DC's derived zone. Satisfiable by construction now: 1.0-GEN.c derives the zone, and both live DCs were reissued into theirs.

Also: SKIPS cleanly for vr0-dc0 (the amphora PKI is a VR1 per-DC artifact -- a gate that fires on a target it does not apply to is how gates get bypassed), and FAILS CLOSED if octavia-pki.sh is absent, mirroring P4/P5.

Revert: git checkout <prev> -- scripts/preflight.sh tests/preflight/run-tests.sh.

14. Harness 26/26 -> 33/33, and ten cases went red first

When P7 landed, ten existing cases failed -- correctly: the fixtures had no octavia-pki.sh, so the gate failed closed. The fix was to make mkfix model a healthy headend (host-identity binding, a fake hostname on PATH, an octavia-pki stub), NOT to patch the cases individually -- that would have been the moment someone demoted the gate to a warning to go green.

New cases: T27 (verify exits 0 on a wrong-zone cert and P7 still FAILS -- the F9 shape, and the case a naive exit-code-only implementation fails), T28 (off-headend -> NOT EVALUATED, WARN, never a silent pass), T29 (checker missing -> fails closed), T30 (verify FAIL -> overall FAIL), T31 (REFUSE on the headend -> FAIL, named correctly rather than as "UNEXPECTED"), T32 (absent headend binding -> FAIL, never assumed), T33 (vr0-dc0 skips cleanly and the run still PASSES).

Two harness bugs of my own, both caught by running rather than reading: the stub first emitted DC.s where the real line has an apostrophe (so it correctly did not match), then \x27\x27\x27 produced three apostrophes because \x27 inside a printf FORMAT already yields one. Fixed the stub, not the gate.

Verification (part 4)

bash tests/preflight/run-tests.sh  -> 33/33 ALL PASS
bash scripts/run-tests-all.sh      -> GAUNTLET: ALL GREEN (89)
bash scripts/repo-lint.sh          -> 0 fail

Note on preflight's overall verdict

Preflight still exits FAIL today, and that is P5 working as designed: the credential register is red on real pre-existing defects (SEC-021's declaration gaps, the SEC-020 identity conflation, the per-DC power-key asymmetry). P7 does not change that verdict on the headend -- it passes there. Going green is remediation, gated per row.