Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260730-octavia-reissue-tool.md
@JANeumatrix JANeumatrix 11 hours ago 11 KB octavia-pki: a hardened, tool-driven controller-cert REISSUE path (F9's remedy)
Raw Blame History

Changelog 2026-07-30 (part 2) -- the Octavia controller-cert REISSUE tool, hardened

Session changelog part 2 (part 1 = changelog-20260730-docfix205-d117-annotation.md). Branch dc-dc-stage5-preconditions. Status claims live ONLY in docs/CURRENT-STATE.md.

Trigger. Operator: "Let's reissue now. Engineer the artifact, variable, based workflow to reduce the chance of errors. Deploy agents to engineer, code, and valid the coding." F9's obligation: the controller certs on BOTH DCs carry dc0.vr0 names -- the wrong region on both, and the wrong DC label on dc1.

No new D-number (GA-R3). The naming is ruled (D-008 shape + D-117 labels); the tool is OPS. Next-free UNCHANGED: D 138 / DOCFIX 206 / BUNDLEFIX 053.

Operator rulings recorded (GA-R5, one exchange each, verbatim)

  1. SCOPE -- "Controller cert only (Recommended)". Signed by the EXISTING controller CA; neither CA regenerated; only lb-mgmt-controller-cert changes in the overlay.
  2. KEY -- "Fresh P-256 key (Recommended)".
  3. ARTIFACT -- "Script subcommand + runbook step (Recommended)".
  4. RE-RUN GUARD -- "Refuse unless --force (Recommended)".
  5. MINT PATH -- "Full script minter now", taken after I escalated a measured concern (below).
  6. Scope of this batch -- "Yes, they all need to be processed and the hardened tool driven process needs to be documented and saved for future stand ups."

The escalation, and its correction -- recorded because I got the framing wrong first

Presenting ruling 3 I told the operator a reissue subcommand "does not touch generate, so the D-137 fork-1 ruling stays untouched." That was wrong twice over.

First, measured: guard-destructive.py is registered "matcher": "Bash" and inspects only the command STRING, keyed on .pem. Empirically -- bash scripts/octavia-pki.sh reissue vr1-dc0 ALLOWED, openssl genpkey ... -out controller.key ALLOWED, only the .pem shape BLOCKED. So the guard has no opinion on ANY script wrapping openssl, and the subcommand's NAME is irrelevant. I escalated on that basis.

Second, and this is the correction: D-137 fork 1 is NOT open. Sub-ruling 1 RULED it 2026-07-25 at option (b) "Blocking in preflight", explicitly recording "NOT adopted: the PreToolUse guard (so an agent is not constrained at write time, only at the deploy gate)". guard-destructive.py is a separate, older mechanism (CLAUDE.md secrets norms / DOCFIX-006), not D-137's enforcement point. A scripted minter therefore decides nothing unruled -- it is consistent with what D-137 actually ruled. docs/audit/queued-findings-20260730.txt Q2 calls fork 1 "still unruled", which is the same stale-premise class as Q1 in part 1 of this session. Corrected there by appended note.

1. scripts/octavia-pki.sh -- the reissue action

reissue <site> [--force] [--dry-run]. Exit codes: 0 ok/dry-run, 1 FAIL (measured defect or a staged assertion failed -- nothing promoted), 2 args, 3 REFUSE (could not evaluate / could not make it safe), 4 REFUSE already-correct, 5 PARTIAL (promotion interrupted -- restore command printed). verify's 0/1/2/3 semantics unchanged.

Shape, in order: precondition sweep (host authority FIRST, before any file is created) -> derive -> re-run guard -> backup -> printf-built config -> structural config assertions -> mint into STAGING -> post-mint assertions in staging -> promote -> single-value overlay surgery -> prove the surgery -> final verify.

Properties that matter, each chosen against a measured failure:

  • Zone DERIVED, never typed (${SITE%%-*} / ${SITE#*-}), so vr1-dc0 -> dc0.vr1 and vr1-dc1 -> dc1.vr1. Same code path verify uses -- generator and gate cannot drift.
  • printf per line, no heredoc -- the F8 payoff. Values passed as %s ARGUMENTS so a % in a value cannot become a format directive.
  • Stage, assert, promote. 1.0-GEN.c overwrites the key BEFORE signing with no set -e, so any mid-way failure yields a bundle whose cert and key do not match. Nothing detected that.
  • -CAserial, NOT -CAcreateserial -- with -CAserial alone openssl FAILS on a missing serial file instead of silently starting a new random sequence.
  • Overlay surgery via python3 reading both files itself -- never sed/awk with the value in argv, because the bundle holds an UNENCRYPTED private key and /proc/<pid>/cmdline is world-readable.

Revert: git checkout <prev> -- scripts/octavia-pki.sh.

2. Three verify gaps closed, then three MORE found by validation

Closed in the first pass: A13 (controller cert subject CN -- asserted nowhere before; check_subject ran only on the two CAs), A14 (cert<->key pairing + bundle integrity -- nothing read the bundle Octavia actually consumes), and a symmetric positive in A12's inert branch (a correct and an incorrect cert were previously distinguishable only by assertion count).

Then adversarial + mutation agents found three more, all now closed:

  • A15 -- keyUsage / EKU on the issued cert. Deleting BOTH extension lines produced a cert with no KU and no EKU that minted, promoted, and left the harness fully green. keyUsage appeared exactly ONCE in the whole file -- the printf that writes it. Losing clientAuth breaks amphora->controller mTLS ONLY: a one-directional failure at LB-create time with every gate reading PASS.
  • A16 -- validity. openssl x509 -req defaults to -days 30. checkend, notAfter and enddate appeared ZERO times in the verify half, so a 30-day controller certificate passed every gate in this repo forever. Reachable by dropping one token from the hand-pasted block that produced the live certs.
  • A17 -- the overlay must carry the material the workspace holds. A10 graded the overlay's SHAPE and A1-A16 graded the workspace; nothing joined them. Measured: an overlay whose embedded cert decoded to a DIFFERENT bundle reported PASS, 0 failed. The overlay is what reaches the charm.

Also hardened: A8's negative proves its instrument first -- openssl verify -CAfile X exits non-zero both for "does not verify" and "could not load X", so a corrupt issuing CA made the trust-domain-separation check pass VACUOUSLY (measured: it printed the reassuring line and promoted).

3. The re-run guard was refusing to fix broken certificates

The most consequential validation finding. The guard compared NAMES and treated that as "is the certificate right?". Measured: a cert with correct CN/SANs but a stale IP SAN made verify FAIL while reissue exited 4 "nothing to fix" -- the remedy refusing the defect the checker had just reported, same workspace, seconds apart. It does not bite today (the names ARE wrong, so it proceeds) but it bites every run afterwards, including the 2-year renewal this reissue starts the clock on, where an EXPIRED cert with correct names would be refused as already-correct.

Now "already correct" means names match AND verify returns clean, so the two cannot disagree -- and A16/A17 put expiry and overlay desync inside that definition.

4. Two pre-existing overlay defects moved to pre-mint

Both are pure functions of state that exists BEFORE the run, but were detected only by the post-surgery proof -- producing exit 5 (workspace promoted, serial burned) when the honest answer was "your overlay was already wrong, nothing was touched": (a) exactly one lb-mgmt-controller-cert line; (b) lb-mgmt-controller-cacert must already decode to THIS DC's controller CA (an overlay from another DC is the F1 cross-DC mix-up).

5. Smaller hardening, each from a measured failure

  • Backup filename gains $$ -- the 1-second stamp plus tar czf overwriting was measured destroying 3 of 12 pre-images, worst exactly in an exit-5-then-retry sequence.
  • Leftover .reissue-<site>.* staging dirs are REPORTED (not deleted): SIGKILL between signing and promotion leaves a valid CA-signed cert and an unencrypted key that no register declares.
  • Two overstated in-file claims corrected: the "hoist is PURE, proven by diffing" comment was backed by nothing executable; and "the other four values are byte-identical" is a line-diff result, which proves the RESULT unchanged, not that they were never rewritten.

6. tests/octavia-pki/run-tests.sh -- 21/21 -> 48/48

T22-T36 cover reissue. T37-T44 exist because mutation testing proved the new assertions could not fail: deleting A13, A14 or the A12 positive outright left the suite fully green. An assertion with no failing-direction fixture is decoration.

T37/T37b (no KU/EKU; partial EKU loss), T38 (short-dated cert), T39/T39b (overlay desync; empty value), T40 (cert/key mismatch), T41 (A13 armed, correct SANs + wrong CN -- the half-fix), T42 (the inert positive must not be vacuous -- a SAN-less cert must not earn it), T43 (a stale THIRD DNS SAN must be reissued down to two; mutation refuted the assumption that T25 covered the count guard), T44 (names-correct-but-verify-FAILS must reissue, not refuse).

Revert: git checkout <prev> -- tests/octavia-pki/run-tests.sh.

7. runbooks/phase-01-bundle-deploy.md -- Step 1.0-REISSUE

Appended AFTER 1.0-GEN.e deliberately: F10 records that creds-matrix S4 mint-refs are line-anchored and only checked within-EOF, so inserting above would have silently misaligned 20 octavia refs. Verified after the append -- creds-matrix.py reports no S4 findings.

Four sub-steps: read-only decide -> --dry-run -> the mint ONE DC AT A TIME -> verify + backup hygiene. Carries the exit-code table as an action table, and the standing warning that a PASS verdict alone is not acceptance -- while unarmed, A12/A13 RECORD rather than fail, so the gate is the literal line DNS SANs are all in this DC's expected zone.

8. Skill sweep -- saved for future standups (operator-directed)

.claude/skills/openstack-cloud-ops/SKILL.md gains two standing invariants and a routing row:

  • EVERY PER-DC SECRET NEEDS A ROTATION TOOL, NOT JUST A GENERATION RECIPE. A generation step guarded by REFUSE-IF-PRESENT leaves no sanctioned way to rotate a leaf, so its only documented escape moves the CAs -- the dangerous path becomes the only path. Carries the four properties that made this tool trustworthy (derive identity; assert the artifact not the config; stage-assert-promote; join workspace to deploy input) and the Roosevelt analog.
  • A HAND-PASTED openssl CHAIN IS A DEFECT CLASS, AND THIS REPO HAS THE RECEIPTS -- F8 and F9 from one block; plus "prove each new assertion can FAIL", with the two properties that had no assertion at all (-days defaulting to 30, and a dropped EKU).

Revert: git checkout <prev> -- .claude/skills/openstack-cloud-ops/SKILL.md.

Verification

bash tests/octavia-pki/run-tests.sh   -> 48/48 PASS
bash scripts/repo-lint.sh             -> 0 fail, 1 warn (legacy L1, unchanged)
bash scripts/run-tests-all.sh         -> GAUNTLET: ALL GREEN (89 harnesses)
python3 scripts/creds-matrix.py       -> 5 findings, ALL pre-existing and named
                                         (dc0-edge-api SEC-021, three power-key
                                         asymmetries, maas-region-admin conflation);
                                         NO S4 findings, so mint-ref anchors are intact

NOT done, deliberately

  • The mint itself has not been run. This changelog ships the tool; the reissue is a gated operator action per 1.0-REISSUE.
  • generate remains unimplemented. reissue REFUSES when there is no existing certificate, precisely so it cannot become a generate backdoor.
  • The backup archive is still an unregistered credential class (it holds both CA keys plus passphrases). Logged, not actioned -- it needs a creds-matrix.tsv row and a retention rule, which is its own gated change.
  • octavia-pki.sh is still called by NO gate: preflight.sh checks only that the overlay EXISTS. So "gauntlet ALL GREEN" remains not-evidence about the live PKI. Wiring verify into preflight is a gate change with its own blast radius -- proposed, not smuggled in here.