| 2026-07-30 |
octavia-pki: a hardened, tool-driven controller-cert REISSUE path (F9's remedy)
...
Operator: 'Let's reissue now. Engineer the artifact, variable, based workflow to
reduce the chance of errors.' Five GA-R5 rulings, one exchange each, quoted in
the changelog; ruling 5 was 'Full script minter now'.
THE TOOL IS BUILT; THE MINT IS NOT YET RUN (gated operator action, 1.0-REISSUE).
scripts/octavia-pki.sh gains 'reissue <site> [--force] [--dry-run]': controller
LEAF only, signed by the EXISTING controller CA (neither CA regenerated, amphora
trust domain untouched), fresh P-256 key, zone DERIVED from the site token and
never typed, printf-built config (the F8 payoff), stage-assert-promote, and a
single-value overlay surgery that never puts the bundle in argv.
SIX GATE DEFECTS FOUND BY VALIDATION AGENTS, ALL CLOSED. Mutation testing is
what earned this: deleting the three assertions I had just written left the
harness fully GREEN every time, so they were decoration until each got a
failing-direction fixture.
- A15 keyUsage/EKU on the ISSUED cert. A cert with NEITHER minted, promoted and
passed everything. keyUsage appeared once in the file: the printf writing it.
- A16 validity. openssl x509 -req defaults to -days 30, and checkend/notAfter/
enddate appeared ZERO times in verify -- a 30-day cert passed forever.
- A17 the overlay must decode to what the workspace holds. A10 graded shape,
A1-A16 graded the workspace, nothing joined them; a desynced overlay read
PASS 0-failed and the overlay is what reaches the charm.
- The re-run guard REFUSED to fix broken certs: names-correct + stale IP SAN
gave verify FAIL and reissue exit 4 'nothing to fix'. Now gated on verify.
- A8's negative proves its instrument loads first (openssl verify exits non-zero
for 'could not load the CA' too, so a corrupt CA passed it vacuously).
- Two pre-existing overlay defects moved pre-mint, so they stop producing a
false exit-5 with the workspace already promoted.
Harness 23/23 -> 48/48. Runbook Step 1.0-REISSUE appended AFTER GEN.e
deliberately (F10 line-anchored mint-refs; creds-matrix reports no S4 findings
after the append). Skill gains two standing invariants for future DC standups:
every per-DC secret needs a ROTATION tool not just a generation recipe -- a
REFUSE-IF-PRESENT generation gate makes the dangerous path the only path -- and
a hand-pasted openssl chain is a defect class, with F8 and F9 as the receipts.
Also corrected: D-137 fork 1 is NOT open (sub-ruling 1, 2026-07-25, ruled
'Blocking in preflight' and explicitly declined the PreToolUse guard), so the
queued Q2 premise is stale in the same way Q1 was.
gauntlet ALL GREEN (89); repo-lint 0 fail; creds-matrix 5 findings, all
pre-existing and named; ledger-scan unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
DOCFIX-205: Q1 withdrawn -- D-117 ruled it 2026-07-13; annotate the four it owed
...
The operator asked to rule on queued Q1 (is substrate vr1-dc1 'dc1' by token or
'dc2' by position, for cert/DNS identity). Measured before presenting options.
The question was WITHDRAWN, not ruled: D-117 (ADOPTED 2026-07-13, four days after
D-106) names the supersession in its own Status line -- "Supersedes ... the D-106
dc1/dc2 zone labels" -- and rules the replacement: "The repo's dc1/dc2 labels are
retired in favour of dc0/dc1". vr1-dc0 -> dc0, vr1-dc1 -> dc1.
WHY IT RESURFACED, and this is the reusable half: D-117 ruled that the ADOPTED
decision texts D-101/D-106/D-111/D-115 be ANNOTATED in place. Measured: ZERO of
the four carried any D-117 annotation. It stayed invisible because D-117's OWN
Status line claimed "FULLY EXECUTED BY D-119", while D-119 scopes its discharge
to the SELECTOR half only. A reader checking whether the annotation was owed was
told it was already done. Ruled is not built -- check the artifact.
Landed (operator-approved batch, "All four + code fixes"):
- All four decisions annotated in place, per D-117's own ruled treatment. D-101
gets the hardest one: it carries BOTH namespaces, so it is annotated BY DATE
(dc1 means different datacenters in its two halves -- D-117 TRAP 2).
- D-117's Status line corrected by measurement (GA-R1/C2) to "EXECUTED IN TWO
HALVES", with the standing lesson that a Status line is a CLAIM about
execution, not evidence of it.
- phase-6 EXECUTABLE DEFECT fixed: os-public-hostname was built as
"keystone.omega.${DC}.vr1..." with $DC the D-119 selector, expanding to
omega.vr1-dc0.vr1... -- the region TWICE -- and would have been baked into
Vault-issued SANs at Stage 7. Both labels are now DERIVED from the site token
(${DC%%-*} / ${DC#*-}); verified by running it, not by reading it.
- octavia-pki.sh A12: refusal -> assertion on the full derived zone; a SAN-less
cert now also fails. Harness 21/21 -> 23/23, with T21 REPLACED (not deleted)
and new T21b (the other DC's label in the right region FAILS -- the cross-DC
mix-up region-only checking passed) and T21c (per-DC derivation proven).
- CURRENT-STATE + ledger corrected in the same commit (L10). "BLOCKING" was
wrong as stated: A12 measures INERT and PASSES live on both DCs. F9's reissue
obligation stands.
gauntlet ALL GREEN (89); repo-lint 0 fail; ledger-scan unchanged (3 decisions,
21 SEC, D 138 / BUNDLEFIX 053; DOCFIX 205 -> next-free 206).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
A12: the DNS SAN check ARMS ITSELF, so F9 cannot be missed
...
Operator direction: make the DNS SANs part of the process so it is not a step that gets missed
in future.
The risk in F9 is not the defect, it is that a future session forgets. So A12 arms from the
change that makes the SANs load-bearing -- os-public-hostname appearing as a real option key in
the merged deploy input -- rather than from a note somebody has to read. Unarmed, it states the
names are inert and why; armed, it asserts them.
WHAT IT CAN ASSERT, AND A GAP IN A RULED DECISION THAT IT CANNOT. D-008 fixes the shape
<service>.<cloud>.<dc>.<region>.cloud.neumatrix.local and D-106:2563 instantiates VR1 as
"omega.dc1.vr1" / "dc2.vr1" -- while the substrate's DCs are vr1-dc0 and vr1-dc1. So vr1-dc1 is
"dc1" by token or "dc2" by position: the DC1/DC2 ambiguity item 3.1 retired elsewhere, surviving
inside a ruling, where it decides certificate identity. The REGION is unambiguous and is
asserted. The DC LABEL REFUSES pending a ruling rather than blessing a name that cannot be
validated. Both live certs currently say dc0.vr0 -- the VR0 region, wrong for a VR1 DC under
either reading.
A PRECEDENCE BUG THE HARNESS CAUGHT. Because the armed branch always refuses on the label, a
definite wrong-region SAN was being reported as "could not evaluate" -- REFUSE was checked before
FAIL and masked a confirmed defect. A known defect outranks an unevaluated item, so the verdict
now reports FAIL first and still names the refusal, which is never lost.
Harness 21/21 (was 18). T19 proves the unarmed live posture still PASSES -- without it, arming
A12 would have broken every current verify, the same trap T17 guards for the deleted
intermediates. T20 is F9 armed. T21 proves the right region still refuses on the unruled label.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
tfstate backup path built; E2/E3 closed live; a gate disagreement found and fixed
...
BACKUP PROVEN BY RESTORE, not by existence. Both PKI archives were extracted and compared
byte-for-byte against the live tree: 12/12 sha256 MATCH across both DCs, covering both encrypted
CA keys, both passphrases, both CA serials and both controller bundles. Archives are 600 inside
700 folders and differ between DCs, so per-DC independence survives into the backup.
E2/E3 CLOSED LIVE. Operator ran the chmod and the intermediate removal; octavia-pki.sh verify
now reads PASS 26/0 on BOTH DCs, workspaces 12 -> 10 files, and P5 on the headend fell 18 -> 8.
A GATE DISAGREEMENT, recorded as the pattern rather than the incident. verify read 26/0 while
creds-matrix E2 still reported the CA serial mode 664 on both DCs. The chmod list predated the
serial being declared, and verify's mode assertions covered six private files and three certs --
the serial was in neither. Two gates that can both see a file must not disagree about it. Fixed
in three places: verify now mode-checks the serial (integrity, not secrecy -- rewriting it forces
the next issuance to reuse a serial), the generator's chmod includes it, and T18 pins it. The
harness then caught its own fixture creating the serial at the inherited umask; baseline went red
until the fixture matched the corrected generator. 18/18.
TFSTATE BACKUP as dc-dc-phase2 step 13, 2 rows, notes key n-tfstate-backup. Register 99 rows;
schema, render-drift, mint-ref and notes checks clean.
Three deliberate differences from the PKI backup: tfstate is DYNAMIC so the serial is recorded
both sides and the step must be re-run after every apply (restoring a stale state is actively
dangerous -- tofu then believes everything created since does not exist); the pull block proves
RESTORE by decompressing to check the archive parses as JSON and reports a serial, because a
truncated gzip passes a size check; and mint-stage is stage3, which is REACHED, so P5 correctly
reports these EXPECTED-BUT-ABSENT until the step runs rather than deferring them.
Classification recorded honestly: the outer state is credential-bearing (DOCFIX-175, MAAS API key
in plaintext); the inner states show no credential-shaped key names, consistent with the inner
root using only libvirt over qemu+ssh. Strong evidence, NOT proof -- a name-based absence cannot
exclude material inside a value -- so they are registered and stored as if sensitive. Uncertainty
resolves toward more auditing, and an unaudited backup directory beside audited ones is how the
SEC-022 shadow stores happened.
Two of my own defects caught by the gates: repo-lint L9 rejected a hardcoded clone name in the
new step (the repo has been renamed once, D-110) -- now $REPO; and two mint-refs had drifted when
a later edit added lines above their anchors, which S4 cannot catch because it only checks
line <= EOF. Both re-anchored.
Also corrected: phase-01:605 said the overlay must be in the backup set. It is derivable from the
archived workspace by 1.0-GEN.d, so the line implied a gap that does not exist.
Known, not fixed: both live inner states are 664, group-writable, and they are the authority tofu
trusts. Tightening needs its own gated change plus proof the provider preserves the mode.
creds-matrix 65/65, octavia-pki 18/18, repo-lint 0 fail / 621 files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
E3 resolved: declare the two artifacts that persist, delete the two intermediates at source
...
Operator-approved disposition after review. The four undeclared per-DC generator outputs were
separated by whether anything will read them again.
DECLARED. The controller certificate, because octavia-pki.sh verify reads it for the subject,
chain and SAN assertions -- a gate that depends on an undeclared artifact is incoherent. And
the CA serial file, because it is issuance STATE rather than residue: delete it and the next
-CAcreateserial starts a fresh sequence, so the same CA can issue a duplicate serial.
Reissuance is scheduled, not hypothetical -- the controller cert is 2-year and F9's D-106 work
will want new SANs.
DELETED AT SOURCE. The signing request is spent the moment the cert is issued. The openssl
config is fully derived from $DC plus the VIP overlay, so the repo already determines it, and
verify now asserts the SAN set on the CERT, which is where the F8 failure actually shows.
WHY NOT DECLARE ALL FOUR, which was lower-risk and less work: a register that accumulates rows
for artifacts that should not exist trains the reader to add a row rather than ask whether the
file belongs -- the register-as-rubber-stamp failure, the same family as this project's
false-green problems. Concretely, both intermediates are 664, so declaring them would have
produced four MORE world-readable findings for files nothing reads.
Also fixed at source, beyond the three requested steps and flagged as such: the generator now
chmod 600s all three certificates, so the next DC is correct by construction instead of by a
remembered follow-up.
Register 95 rows / 22 octavia rows; schema, mint-ref and notes checks clean; no new findings
(still 7 on the jumphost, the same pre-existing set). verify A2 extended 9 -> 10 so the register
and the gate agree on what should exist.
Harness 17/17 (was 15). T16 proves a missing serial FAILS. T17 proves a workspace with the
intermediates DELETED still PASSES -- without it, adding the serial to A2 would have silently
failed every clean workspace.
Gauntlet ALL GREEN (89), repo-lint 0 fail / 621 files.
Still owed, operator-executed: the one-off chmod on the six existing certs and removal of the
four existing intermediates. The guard blocks both for the session -- including the chmod, which
is worth recording as fork-1 evidence since it blocks an operation that strictly improves
posture.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-29 |
Anchor the gitignore pattern that silently swallowed the new harness
...
The previous commit added tests/HARNESS-MANIFEST's octavia-pki entry and the script, but git
REFUSED the harness directory and the commit went through without it. Cause: .gitignore
carried `octavia-pki/`, which git matches at ANY depth, so it also matched
`tests/octavia-pki/`.
Consequence had this not been caught: the manifest names a harness that is not tracked, so a
fresh clone runs the gauntlet against a missing directory. repo-lint does not catch it,
because locally the files exist -- only a clone would have shown it.
Anchored to `/octavia-pki/`. Verified three ways rather than assumed: tests/octavia-pki is
now trackable, a root-level workspace copy is STILL ignored so the original protection is
intact, and both per-DC overlays remain ignored.
Same class as F4 earlier today -- a gitignore pattern whose real scope differed from its
intent, in the direction that loses data rather than the direction that leaks it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
