Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260730-octavia-reissue-tool.md
@JANeumatrix JANeumatrix 6 hours ago 17 KB F8 + F9 fixed AT SOURCE in 1.0-GEN.c -- the next DC standup will not recreate them
Raw Blame History

Changelog 2026-07-30 (part 2) -- the Octavia controller-cert REISSUE tool, hardened

Session changelog part 2 (part 1 = changelog-20260730-docfix205-d117-annotation.md). Branch dc-dc-stage5-preconditions. Status claims live ONLY in docs/CURRENT-STATE.md.

Trigger. Operator: "Let's reissue now. Engineer the artifact, variable, based workflow to reduce the chance of errors. Deploy agents to engineer, code, and valid the coding." F9's obligation: the controller certs on BOTH DCs carry dc0.vr0 names -- the wrong region on both, and the wrong DC label on dc1.

No new D-number (GA-R3). The naming is ruled (D-008 shape + D-117 labels); the tool is OPS. Next-free UNCHANGED: D 138 / DOCFIX 206 / BUNDLEFIX 053.

Operator rulings recorded (GA-R5, one exchange each, verbatim)

  1. SCOPE -- "Controller cert only (Recommended)". Signed by the EXISTING controller CA; neither CA regenerated; only lb-mgmt-controller-cert changes in the overlay.
  2. KEY -- "Fresh P-256 key (Recommended)".
  3. ARTIFACT -- "Script subcommand + runbook step (Recommended)".
  4. RE-RUN GUARD -- "Refuse unless --force (Recommended)".
  5. MINT PATH -- "Full script minter now", taken after I escalated a measured concern (below).
  6. Scope of this batch -- "Yes, they all need to be processed and the hardened tool driven process needs to be documented and saved for future stand ups."

The escalation, and its correction -- recorded because I got the framing wrong first

Presenting ruling 3 I told the operator a reissue subcommand "does not touch generate, so the D-137 fork-1 ruling stays untouched." That was wrong twice over.

First, measured: guard-destructive.py is registered "matcher": "Bash" and inspects only the command STRING, keyed on .pem. Empirically -- bash scripts/octavia-pki.sh reissue vr1-dc0 ALLOWED, openssl genpkey ... -out controller.key ALLOWED, only the .pem shape BLOCKED. So the guard has no opinion on ANY script wrapping openssl, and the subcommand's NAME is irrelevant. I escalated on that basis.

Second, and this is the correction: D-137 fork 1 is NOT open. Sub-ruling 1 RULED it 2026-07-25 at option (b) "Blocking in preflight", explicitly recording "NOT adopted: the PreToolUse guard (so an agent is not constrained at write time, only at the deploy gate)". guard-destructive.py is a separate, older mechanism (CLAUDE.md secrets norms / DOCFIX-006), not D-137's enforcement point. A scripted minter therefore decides nothing unruled -- it is consistent with what D-137 actually ruled. docs/audit/queued-findings-20260730.txt Q2 calls fork 1 "still unruled", which is the same stale-premise class as Q1 in part 1 of this session. Corrected there by appended note.

1. scripts/octavia-pki.sh -- the reissue action

reissue <site> [--force] [--dry-run]. Exit codes: 0 ok/dry-run, 1 FAIL (measured defect or a staged assertion failed -- nothing promoted), 2 args, 3 REFUSE (could not evaluate / could not make it safe), 4 REFUSE already-correct, 5 PARTIAL (promotion interrupted -- restore command printed). verify's 0/1/2/3 semantics unchanged.

Shape, in order: precondition sweep (host authority FIRST, before any file is created) -> derive -> re-run guard -> backup -> printf-built config -> structural config assertions -> mint into STAGING -> post-mint assertions in staging -> promote -> single-value overlay surgery -> prove the surgery -> final verify.

Properties that matter, each chosen against a measured failure:

  • Zone DERIVED, never typed (${SITE%%-*} / ${SITE#*-}), so vr1-dc0 -> dc0.vr1 and vr1-dc1 -> dc1.vr1. Same code path verify uses -- generator and gate cannot drift.
  • printf per line, no heredoc -- the F8 payoff. Values passed as %s ARGUMENTS so a % in a value cannot become a format directive.
  • Stage, assert, promote. 1.0-GEN.c overwrites the key BEFORE signing with no set -e, so any mid-way failure yields a bundle whose cert and key do not match. Nothing detected that.
  • -CAserial, NOT -CAcreateserial -- with -CAserial alone openssl FAILS on a missing serial file instead of silently starting a new random sequence.
  • Overlay surgery via python3 reading both files itself -- never sed/awk with the value in argv, because the bundle holds an UNENCRYPTED private key and /proc/<pid>/cmdline is world-readable.

Revert: git checkout <prev> -- scripts/octavia-pki.sh.

2. Three verify gaps closed, then three MORE found by validation

Closed in the first pass: A13 (controller cert subject CN -- asserted nowhere before; check_subject ran only on the two CAs), A14 (cert<->key pairing + bundle integrity -- nothing read the bundle Octavia actually consumes), and a symmetric positive in A12's inert branch (a correct and an incorrect cert were previously distinguishable only by assertion count).

Then adversarial + mutation agents found three more, all now closed:

  • A15 -- keyUsage / EKU on the issued cert. Deleting BOTH extension lines produced a cert with no KU and no EKU that minted, promoted, and left the harness fully green. keyUsage appeared exactly ONCE in the whole file -- the printf that writes it. Losing clientAuth breaks amphora->controller mTLS ONLY: a one-directional failure at LB-create time with every gate reading PASS.
  • A16 -- validity. openssl x509 -req defaults to -days 30. checkend, notAfter and enddate appeared ZERO times in the verify half, so a 30-day controller certificate passed every gate in this repo forever. Reachable by dropping one token from the hand-pasted block that produced the live certs.
  • A17 -- the overlay must carry the material the workspace holds. A10 graded the overlay's SHAPE and A1-A16 graded the workspace; nothing joined them. Measured: an overlay whose embedded cert decoded to a DIFFERENT bundle reported PASS, 0 failed. The overlay is what reaches the charm.

Also hardened: A8's negative proves its instrument first -- openssl verify -CAfile X exits non-zero both for "does not verify" and "could not load X", so a corrupt issuing CA made the trust-domain-separation check pass VACUOUSLY (measured: it printed the reassuring line and promoted).

3. The re-run guard was refusing to fix broken certificates

The most consequential validation finding. The guard compared NAMES and treated that as "is the certificate right?". Measured: a cert with correct CN/SANs but a stale IP SAN made verify FAIL while reissue exited 4 "nothing to fix" -- the remedy refusing the defect the checker had just reported, same workspace, seconds apart. It does not bite today (the names ARE wrong, so it proceeds) but it bites every run afterwards, including the 2-year renewal this reissue starts the clock on, where an EXPIRED cert with correct names would be refused as already-correct.

Now "already correct" means names match AND verify returns clean, so the two cannot disagree -- and A16/A17 put expiry and overlay desync inside that definition.

4. Two pre-existing overlay defects moved to pre-mint

Both are pure functions of state that exists BEFORE the run, but were detected only by the post-surgery proof -- producing exit 5 (workspace promoted, serial burned) when the honest answer was "your overlay was already wrong, nothing was touched": (a) exactly one lb-mgmt-controller-cert line; (b) lb-mgmt-controller-cacert must already decode to THIS DC's controller CA (an overlay from another DC is the F1 cross-DC mix-up).

5. Smaller hardening, each from a measured failure

  • Backup filename gains $$ -- the 1-second stamp plus tar czf overwriting was measured destroying 3 of 12 pre-images, worst exactly in an exit-5-then-retry sequence.
  • Leftover .reissue-<site>.* staging dirs are REPORTED (not deleted): SIGKILL between signing and promotion leaves a valid CA-signed cert and an unencrypted key that no register declares.
  • Two overstated in-file claims corrected: the "hoist is PURE, proven by diffing" comment was backed by nothing executable; and "the other four values are byte-identical" is a line-diff result, which proves the RESULT unchanged, not that they were never rewritten.

6. tests/octavia-pki/run-tests.sh -- 21/21 -> 48/48

T22-T36 cover reissue. T37-T44 exist because mutation testing proved the new assertions could not fail: deleting A13, A14 or the A12 positive outright left the suite fully green. An assertion with no failing-direction fixture is decoration.

T37/T37b (no KU/EKU; partial EKU loss), T38 (short-dated cert), T39/T39b (overlay desync; empty value), T40 (cert/key mismatch), T41 (A13 armed, correct SANs + wrong CN -- the half-fix), T42 (the inert positive must not be vacuous -- a SAN-less cert must not earn it), T43 (a stale THIRD DNS SAN must be reissued down to two; mutation refuted the assumption that T25 covered the count guard), T44 (names-correct-but-verify-FAILS must reissue, not refuse).

Revert: git checkout <prev> -- tests/octavia-pki/run-tests.sh.

7. runbooks/phase-01-bundle-deploy.md -- Step 1.0-REISSUE

Appended AFTER 1.0-GEN.e deliberately: F10 records that creds-matrix S4 mint-refs are line-anchored and only checked within-EOF, so inserting above would have silently misaligned 20 octavia refs. Verified after the append -- creds-matrix.py reports no S4 findings.

Four sub-steps: read-only decide -> --dry-run -> the mint ONE DC AT A TIME -> verify + backup hygiene. Carries the exit-code table as an action table, and the standing warning that a PASS verdict alone is not acceptance -- while unarmed, A12/A13 RECORD rather than fail, so the gate is the literal line DNS SANs are all in this DC's expected zone.

8. Skill sweep -- saved for future standups (operator-directed)

.claude/skills/openstack-cloud-ops/SKILL.md gains two standing invariants and a routing row:

  • EVERY PER-DC SECRET NEEDS A ROTATION TOOL, NOT JUST A GENERATION RECIPE. A generation step guarded by REFUSE-IF-PRESENT leaves no sanctioned way to rotate a leaf, so its only documented escape moves the CAs -- the dangerous path becomes the only path. Carries the four properties that made this tool trustworthy (derive identity; assert the artifact not the config; stage-assert-promote; join workspace to deploy input) and the Roosevelt analog.
  • A HAND-PASTED openssl CHAIN IS A DEFECT CLASS, AND THIS REPO HAS THE RECEIPTS -- F8 and F9 from one block; plus "prove each new assertion can FAIL", with the two properties that had no assertion at all (-days defaulting to 30, and a dropped EKU).

Revert: git checkout <prev> -- .claude/skills/openstack-cloud-ops/SKILL.md.

Verification

bash tests/octavia-pki/run-tests.sh   -> 48/48 PASS
bash scripts/repo-lint.sh             -> 0 fail, 1 warn (legacy L1, unchanged)
bash scripts/run-tests-all.sh         -> GAUNTLET: ALL GREEN (89 harnesses)
python3 scripts/creds-matrix.py       -> 5 findings, ALL pre-existing and named
                                         (dc0-edge-api SEC-021, three power-key
                                         asymmetries, maas-region-admin conflation);
                                         NO S4 findings, so mint-ref anchors are intact

NOT done, deliberately

  • The mint itself has not been run. This changelog ships the tool; the reissue is a gated operator action per 1.0-REISSUE.
  • generate remains unimplemented. reissue REFUSES when there is no existing certificate, precisely so it cannot become a generate backdoor.
  • The backup archive is still an unregistered credential class (it holds both CA keys plus passphrases). Logged, not actioned -- it needs a creds-matrix.tsv row and a retention rule, which is its own gated change.
  • octavia-pki.sh is still called by NO gate: preflight.sh checks only that the overlay EXISTS. So "gauntlet ALL GREEN" remains not-evidence about the live PKI. Wiring verify into preflight is a gate change with its own blast radius -- proposed, not smuggled in here.

PART 3 (same session) -- custody, register, and F8/F9 fixed AT SOURCE in GEN.c

Operator: "Update the matrix as needed. Transfer the artifacts to the vcloud. keep the pin that during the secrets workflow planning that the creds and certs from this step need to be included. continue on with the rest of the steps."

9. Backup custody executed (Step 1.0-REISSUE.4)

Both pre-reissue archives pulled from the headend to the jumphost SEC-009 creds folders, sha256 compared and identical at both ends before anything was deleted (a truncated scp would otherwise leave a verified-looking backup of nothing). 0600; 15 entries each including the deploy overlay and the CA serial. Headend ~/octavia-pki/backups/ removed entirely; no .reissue-* staging residue.

10. The archives are now REGISTERED (they were an unregistered credential class)

Two octavia-reissue-backup rows (per-DC, jumphost, consolidated, templated filename <site>-octavia-pki-<stamp>.tar.gz), the DERIVED manifests updated to match (D-137 ruling 2 -- manifests are generated, never hand-authored), and a new n-reissue-backup note.

Deliberately a SEPARATE id from n-pki-backup: that one is the generation-time workspace archive, this is per-ROTATION and additionally carries the deploy overlay, which the generation archive does not. Restoring only the workspace would leave the half that reaches the charm unrecoverable.

Measured after: tier 1 = 101 rows, the SAME 5 pre-existing findings; tier 2 = the same 7, with no reissue-attributable finding -- and tier 2 FOUND both archives at their declared location, which is the point of registering them.

Hazard recorded in the note: these archives contain controller-ca.cert.srl, the CA's ISSUANCE STATE. Restoring one over a workspace that has issued since rolls the serial counter BACKWARDS and the next mint reuses a serial the estate already holds. Serials burned 2026-07-30: ...250E (dc0), ...674DE (dc1).

PINNED (operator-directed): the note carries the standing requirement that the creds and certs from this step be included in the secrets-storage workflow planning, with the full scope enumerated -- both CAs (key, passphrase, cert), the controller LEAF key/cert/bundle, the CA serial state, and the deploy overlay, at BOTH DCs. The jumphost creds folder is INTERIM only.

11. Q2 withdrawn -- the THIRD stale premise in one findings file

docs/audit/queued-findings-20260730.txt Q2 opens "D-137 OPEN FORK 1 -- still unruled". It was ruled 2026-07-25 at option (b), "Blocking in preflight", whose CONSEQUENCE block explicitly declines option (c). Appended correction, not a rewrite. The misfire tally is now SEVEN -- the two newest collected today, when the guard blocked a command whose only purpose was to TEST it, and then blocked the heredoc writing the correction that documents its own misfires.

12. F8 AND F9 FIXED AT SOURCE IN 1.0-GEN.c -- the part that matters for future standups

Everything above repairs the two EXISTING DCs. The generation path still carried both defects, so the next DC standup would have recreated them. Both are now fixed where they originate:

  • F9 at source. GEN.c baked omega.dc0.vr0 as a literal in THREE places (CN + both DNS SANs). It now DERIVES DC_ZONE from $DC using the same two expansions the tool uses (${DC%%-*} / ${DC#*-}), so the runbook and the gate cannot disagree, and echoes the zone for confirmation before the sign.
  • F8 at source. The cat > controller.cnf <<'CNF' heredoc is replaced by one printf per line, values passed as %s ARGUMENTS. This was explicitly deferred on 2026-07-29 as "an untested rewrite ... reasonable when someone can run a real generation" -- that condition is now met: the identical shape was exercised end to end by two real mints (both DCs) plus 48 harness cases.
  • Plus structural assertions before signing, because printf removes the paste hazard but not the failure CLASS: all four sections present, subjectAltName wired, CN equal to the derived zone, exactly 2 DNS entries. A typo'd section name now stops the step instead of producing a SAN-less certificate behind a wall of OK output.
  • The stale "NOT changed -- outside R7's ruled scope" note is superseded in place, keeping the reasoning: R7 recorded this cert's SAN as "already DERIVED per-DC by design", which was true of the IP SAN and NOT the DNS names -- a claim accurate about one half of a field, read as covering both. That is how F9 survived.

F10 handled deliberately. Editing GEN.c shifts every mint-ref anchored below it. All 13 octavia anchors were re-resolved by MARKER in a SINGLE PASS keyed by row id (never sequential seds -- a line number can be simultaneously an old value for one row and a new value for another), then each verified to point at its correct command. 12 rows rewritten; creds-matrix S4 reports clean. The runbook block was bash -n syntax-checked as extracted.

Revert: git checkout <prev> -- runbooks/phase-01-bundle-deploy.md creds-matrix.tsv creds-matrix-notes.md creds-manifests/vr1-dc0.manifest creds-manifests/vr1-dc1.manifest.

Verification (part 3)

bash tests/octavia-pki/run-tests.sh   -> 48/48 PASS
bash tests/creds-matrix/run-tests.sh  -> 65/65 PASS
python3 scripts/creds-matrix.py       -> 101 rows, 5 findings (all pre-existing), S4 CLEAN
bash scripts/run-tests-all.sh         -> GAUNTLET: ALL GREEN (89)
bash scripts/repo-lint.sh             -> 0 fail