# Changelog 2026-07-30 (part 2) -- the Octavia controller-cert REISSUE tool, hardened Session changelog part 2 (part 1 = `changelog-20260730-docfix205-d117-annotation.md`). Branch `dc-dc-stage5-preconditions`. Status claims live ONLY in `docs/CURRENT-STATE.md`. **Trigger.** Operator: "Let's reissue now. Engineer the artifact, variable, based workflow to reduce the chance of errors. Deploy agents to engineer, code, and valid the coding." F9's obligation: the controller certs on BOTH DCs carry `dc0.vr0` names -- the wrong region on both, and the wrong DC label on dc1. **No new D-number (GA-R3).** The naming is ruled (D-008 shape + D-117 labels); the tool is OPS. Next-free UNCHANGED: D 138 / DOCFIX 206 / BUNDLEFIX 053. --- ## Operator rulings recorded (GA-R5, one exchange each, verbatim) 1. SCOPE -- **"Controller cert only (Recommended)"**. Signed by the EXISTING controller CA; neither CA regenerated; only `lb-mgmt-controller-cert` changes in the overlay. 2. KEY -- **"Fresh P-256 key (Recommended)"**. 3. ARTIFACT -- **"Script subcommand + runbook step (Recommended)"**. 4. RE-RUN GUARD -- **"Refuse unless --force (Recommended)"**. 5. MINT PATH -- **"Full script minter now"**, taken after I escalated a measured concern (below). 6. Scope of this batch -- **"Yes, they all need to be processed and the hardened tool driven process needs to be documented and saved for future stand ups."** ### The escalation, and its correction -- recorded because I got the framing wrong first Presenting ruling 3 I told the operator a `reissue` subcommand "does not touch `generate`, so the D-137 fork-1 ruling stays untouched." **That was wrong twice over.** First, measured: `guard-destructive.py` is registered `"matcher": "Bash"` and inspects only the command STRING, keyed on `.pem`. Empirically -- `bash scripts/octavia-pki.sh reissue vr1-dc0` **ALLOWED**, `openssl genpkey ... -out controller.key` **ALLOWED**, only the `.pem` shape BLOCKED. So the guard has no opinion on ANY script wrapping openssl, and the subcommand's NAME is irrelevant. I escalated on that basis. Second, and this is the correction: **D-137 fork 1 is NOT open.** Sub-ruling 1 RULED it 2026-07-25 at option (b) "Blocking in preflight", explicitly recording *"NOT adopted: the PreToolUse guard (so an agent is not constrained at write time, only at the deploy gate)"*. `guard-destructive.py` is a separate, older mechanism (CLAUDE.md secrets norms / DOCFIX-006), not D-137's enforcement point. A scripted minter therefore decides nothing unruled -- it is consistent with what D-137 actually ruled. **`docs/audit/queued-findings-20260730.txt` Q2 calls fork 1 "still unruled", which is the same stale-premise class as Q1 in part 1 of this session.** Corrected there by appended note. --- ## 1. `scripts/octavia-pki.sh` -- the `reissue` action `reissue [--force] [--dry-run]`. Exit codes: 0 ok/dry-run, 1 FAIL (measured defect or a staged assertion failed -- nothing promoted), 2 args, 3 REFUSE (could not evaluate / could not make it safe), 4 REFUSE already-correct, 5 PARTIAL (promotion interrupted -- restore command printed). `verify`'s 0/1/2/3 semantics unchanged. Shape, in order: precondition sweep (host authority FIRST, before any file is created) -> derive -> re-run guard -> backup -> `printf`-built config -> structural config assertions -> mint into STAGING -> post-mint assertions in staging -> promote -> single-value overlay surgery -> prove the surgery -> final verify. Properties that matter, each chosen against a measured failure: - **Zone DERIVED, never typed** (`${SITE%%-*}` / `${SITE#*-}`), so `vr1-dc0` -> `dc0.vr1` and `vr1-dc1` -> `dc1.vr1`. Same code path `verify` uses -- generator and gate cannot drift. - **`printf` per line, no heredoc** -- the F8 payoff. Values passed as `%s` ARGUMENTS so a `%` in a value cannot become a format directive. - **Stage, assert, promote.** 1.0-GEN.c overwrites the key BEFORE signing with no `set -e`, so any mid-way failure yields a bundle whose cert and key do not match. Nothing detected that. - **`-CAserial`, NOT `-CAcreateserial`** -- with `-CAserial` alone openssl FAILS on a missing serial file instead of silently starting a new random sequence. - **Overlay surgery via python3 reading both files itself** -- never `sed`/`awk` with the value in argv, because the bundle holds an UNENCRYPTED private key and `/proc//cmdline` is world-readable. **Revert:** `git checkout -- scripts/octavia-pki.sh`. ## 2. Three `verify` gaps closed, then three MORE found by validation Closed in the first pass: **A13** (controller cert subject CN -- asserted nowhere before; `check_subject` ran only on the two CAs), **A14** (cert<->key pairing + bundle integrity -- nothing read the bundle Octavia actually consumes), and a symmetric positive in **A12**'s inert branch (a correct and an incorrect cert were previously distinguishable only by assertion count). Then adversarial + mutation agents found three more, all now closed: - **A15 -- keyUsage / EKU on the issued cert.** Deleting BOTH extension lines produced a cert with no KU and no EKU that minted, promoted, and left the harness fully green. `keyUsage` appeared exactly ONCE in the whole file -- the printf that writes it. Losing `clientAuth` breaks amphora->controller mTLS ONLY: a one-directional failure at LB-create time with every gate reading PASS. - **A16 -- validity.** `openssl x509 -req` defaults to **`-days 30`**. `checkend`, `notAfter` and `enddate` appeared ZERO times in the verify half, so a 30-day controller certificate passed every gate in this repo forever. Reachable by dropping one token from the hand-pasted block that produced the live certs. - **A17 -- the overlay must carry the material the workspace holds.** A10 graded the overlay's SHAPE and A1-A16 graded the workspace; nothing joined them. Measured: an overlay whose embedded cert decoded to a DIFFERENT bundle reported PASS, 0 failed. The overlay is what reaches the charm. Also hardened: **A8's negative proves its instrument first** -- `openssl verify -CAfile X` exits non-zero both for "does not verify" and "could not load X", so a corrupt issuing CA made the trust-domain-separation check pass VACUOUSLY (measured: it printed the reassuring line and promoted). ## 3. The re-run guard was refusing to fix broken certificates **The most consequential validation finding.** The guard compared NAMES and treated that as "is the certificate right?". Measured: a cert with correct CN/SANs but a stale IP SAN made `verify` FAIL while `reissue` exited 4 "nothing to fix" -- the remedy refusing the defect the checker had just reported, same workspace, seconds apart. It does not bite today (the names ARE wrong, so it proceeds) but it bites every run afterwards, including the 2-year renewal this reissue starts the clock on, where an EXPIRED cert with correct names would be refused as already-correct. Now "already correct" means names match **AND `verify` returns clean**, so the two cannot disagree -- and A16/A17 put expiry and overlay desync inside that definition. ## 4. Two pre-existing overlay defects moved to pre-mint Both are pure functions of state that exists BEFORE the run, but were detected only by the post-surgery proof -- producing exit 5 (workspace promoted, serial burned) when the honest answer was "your overlay was already wrong, nothing was touched": (a) exactly one `lb-mgmt-controller-cert` line; (b) `lb-mgmt-controller-cacert` must already decode to THIS DC's controller CA (an overlay from another DC is the F1 cross-DC mix-up). ## 5. Smaller hardening, each from a measured failure - Backup filename gains `$$` -- the 1-second stamp plus `tar czf` overwriting was measured destroying 3 of 12 pre-images, worst exactly in an exit-5-then-retry sequence. - Leftover `.reissue-.*` staging dirs are REPORTED (not deleted): SIGKILL between signing and promotion leaves a valid CA-signed cert and an unencrypted key that no register declares. - Two overstated in-file claims corrected: the "hoist is PURE, proven by diffing" comment was backed by nothing executable; and "the other four values are byte-identical" is a line-diff result, which proves the RESULT unchanged, not that they were never rewritten. ## 6. `tests/octavia-pki/run-tests.sh` -- 21/21 -> 48/48 T22-T36 cover `reissue`. **T37-T44 exist because mutation testing proved the new assertions could not fail:** deleting A13, A14 or the A12 positive outright left the suite fully green. An assertion with no failing-direction fixture is decoration. T37/T37b (no KU/EKU; partial EKU loss), T38 (short-dated cert), T39/T39b (overlay desync; empty value), T40 (cert/key mismatch), T41 (A13 armed, correct SANs + wrong CN -- the half-fix), T42 (the inert positive must not be vacuous -- a SAN-less cert must not earn it), T43 (a stale THIRD DNS SAN must be reissued down to two; mutation refuted the assumption that T25 covered the count guard), T44 (names-correct-but-verify-FAILS must reissue, not refuse). **Revert:** `git checkout -- tests/octavia-pki/run-tests.sh`. ## 7. `runbooks/phase-01-bundle-deploy.md` -- Step 1.0-REISSUE Appended AFTER 1.0-GEN.e **deliberately**: F10 records that `creds-matrix` S4 mint-refs are line-anchored and only checked within-EOF, so inserting above would have silently misaligned 20 octavia refs. Verified after the append -- `creds-matrix.py` reports **no S4 findings**. Four sub-steps: read-only decide -> `--dry-run` -> the mint ONE DC AT A TIME -> verify + backup hygiene. Carries the exit-code table as an action table, and the standing warning that **a PASS verdict alone is not acceptance** -- while unarmed, A12/A13 RECORD rather than fail, so the gate is the literal line `DNS SANs are all in this DC's expected zone`. ## 8. Skill sweep -- saved for future standups (operator-directed) `.claude/skills/openstack-cloud-ops/SKILL.md` gains two standing invariants and a routing row: - **EVERY PER-DC SECRET NEEDS A ROTATION TOOL, NOT JUST A GENERATION RECIPE.** A generation step guarded by REFUSE-IF-PRESENT leaves no sanctioned way to rotate a leaf, so its only documented escape moves the CAs -- **the dangerous path becomes the only path**. Carries the four properties that made this tool trustworthy (derive identity; assert the artifact not the config; stage-assert-promote; join workspace to deploy input) and the Roosevelt analog. - **A HAND-PASTED `openssl` CHAIN IS A DEFECT CLASS, AND THIS REPO HAS THE RECEIPTS** -- F8 and F9 from one block; plus "prove each new assertion can FAIL", with the two properties that had no assertion at all (`-days` defaulting to 30, and a dropped EKU). **Revert:** `git checkout -- .claude/skills/openstack-cloud-ops/SKILL.md`. --- ## Verification bash tests/octavia-pki/run-tests.sh -> 48/48 PASS bash scripts/repo-lint.sh -> 0 fail, 1 warn (legacy L1, unchanged) bash scripts/run-tests-all.sh -> GAUNTLET: ALL GREEN (89 harnesses) python3 scripts/creds-matrix.py -> 5 findings, ALL pre-existing and named (dc0-edge-api SEC-021, three power-key asymmetries, maas-region-admin conflation); NO S4 findings, so mint-ref anchors are intact ## NOT done, deliberately - **The mint itself has not been run.** This changelog ships the tool; the reissue is a gated operator action per 1.0-REISSUE. - `generate` remains unimplemented. `reissue` REFUSES when there is no existing certificate, precisely so it cannot become a `generate` backdoor. - The backup archive is still an unregistered credential class (it holds both CA keys plus passphrases). Logged, not actioned -- it needs a `creds-matrix.tsv` row and a retention rule, which is its own gated change. - `octavia-pki.sh` is still called by NO gate: `preflight.sh` checks only that the overlay EXISTS. So "gauntlet ALL GREEN" remains not-evidence about the live PKI. Wiring `verify` into preflight is a gate change with its own blast radius -- proposed, not smuggled in here.