diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 44ba0cf..ccbea3e 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2445,6 +2445,36 @@ within-EOF; this bit three times in two sessions, caught by hand every time, never by a gate). All 24 octavia refs were re-anchored today in a SINGLE-PASS mapping keyed by row id, because 391 was simultaneously an old and a new value and sequential seds would have corrupted it. +- **HEADEND RECONCILE COMPLETE 2026-07-30, and running the gauntlet THERE -- which had never + actually been done -- found a gate that was RED ON THE ONLY HOST THAT DEPLOYS.** Operator asked + whether voffice1 was fully reconciled. It was not. + **RECONCILED:** headend was 1 commit behind, 0 local commits, now at HEAD with a CLEAN working + tree; both tfstate sha256 byte-identical before and after every pull; no stale remote-tracking + refs. One asymmetry found and fixed: `opentofu/vr1-dc1-substrate/.terraform.lock.hcl` was + UNTRACKED and not gitignored while **dc0's equivalent IS tracked** -- a fresh clone could not + reproduce dc1's provider resolution the way it can dc0's. Content divergence was RULED OUT by + measurement first (both pin libvirt 0.9.8) before the file was landed. + **THE GATE DEFECT:** gauntlet read ALL GREEN (89) on the jumphost and **1/89 FAILED on the + headend**, on `opentofu-validate`. Cause: `tofu fmt -recursive` walks EVERY `.tf`/`.tfvars` + under the tree INCLUDING GITIGNORED ones, and the sole flagged file was + `vr1-dc0-substrate/d124-inner.auto.tfvars` -- gitignored operator input that exists only on the + host which runs the applies. **So the gate was red where it mattered and green where it did + not**, the same host-dependent-verdict class as the LC_ALL manifest sort. Version drift was + ruled out first (both hosts run tofu 1.12.4). The fmt check is now scoped to repo content and + SAYS when it skips; harness **16/16** with both directions pinned (T16 a tracked badly-formatted + file still FAILS, T17 a gitignored tfvars is skipped -- and T17 fails loudly if the ignore rule + ever stops covering per-DC tfvars, so it cannot silently stop testing what it claims). + **BOTH HOSTS NOW ALL GREEN (89), repo-lint 0 fail on both.** + **TWO OWNED MISTAKES, both of the same shape -- a silenced error:** my first T16 read as a + failure because `bash "$SCRIPT" | grep -q` inherits the gate's correct exit 1 under + `set -o pipefail` even though grep matched; and a `git pull -q && echo` on the headend SWALLOWED + a real pull failure (git refused to overwrite the untracked lock file I had just committed), so + the headend silently stayed 1 behind and I re-ran the gauntlet against a stale tree and + mis-read the result as "the fix did not work". The missing confirmation line was the only clue. + Both fixed by capturing/checking status explicitly rather than chaining on `&&`. + **NOTED, NOT FIXED:** `repo-lint` scans **623 files on the headend vs 621 on the jumphost** -- + it walks untracked/ignored files too, so its file count is host-dependent. 0 fail on both, so + not urgent, but it is the same unscoped-walk class as the fmt defect just fixed. **F3 -- both `~/octavia-pki/` and `overlays/octavia-pki.yaml` are ABSENT here** (existence checked, no contents read). So this is generation FROM SCRATCH for both DCs: there is nothing to reuse, which retires the reuse-vs-regenerate choice diff --git a/docs/changelog-20260729-stage5-preconditions.md b/docs/changelog-20260729-stage5-preconditions.md index 4bd89a4..93e7124 100644 --- a/docs/changelog-20260729-stage5-preconditions.md +++ b/docs/changelog-20260729-stage5-preconditions.md @@ -165,6 +165,49 @@ circumventing -- and once CORRECTLY, on the PKI generation itself. Worth tightening the first class of match rule; the third is the guard working as designed. +## 13. `scripts/octavia-pki.sh verify` + harness (`a22bcb4`, `7aef401`) + +WHAT: new verify-only gate, 29 assertions, harness `tests/octavia-pki` 21/21, manifest 88 -> 89. +WHY: F8 established that nothing in the repo asserted the SAN set, so a controller certificate +with NO SANs passed every check. Prose cannot be tested. `generate` deliberately exits 2 pending +the D-137 fork-1 ruling -- building it would silently convert the guard from "blocks secret +minting" to "blocks it except via any script". +REVERT: `git revert 7aef401 a22bcb4`, then re-run `run-tests-all.sh` -- this adds a harness, so +the manifest must be reconciled after a revert. + +## 14. E2/E3 closed at source + the serial omission (`8091f15`, `cfae8a7` follow-on) + +WHAT: generator `chmod 600`s all three certs AND the CA serial; `rm -f` the two build +intermediates; 4 register rows; `verify` A2 9 -> 10 and A3 covers the serial. +WHY: certs landed 664 -- group-WRITABLE trust anchors, an INTEGRITY exposure this session first +mis-called harmless. The serial was missed in the first chmod list and caught by P5 while +`verify` still read 26/0: two gates disagreeing about a file both could see. +REVERT: `git revert` the pair; live modes are unaffected (they were changed by hand). + +## 15. Both backup paths (`2801b97`, `9a96843`, `f581599`) + +WHAT: `phase-01` 1.0-GEN.e (PKI) and `dc-dc-phase2` step 13 (inner tfstate), 4 register rows, +notes keys `n-pki-backup` / `n-tfstate-backup`. +WHY: three surfaces asserted a backup set that did not exist. Both restores were PROVEN, not +asserted -- PKI 12/12 sha256 vs live, tfstate byte-identical with correct serials. +REVERT: doc/register only. **Do NOT delete the archives** -- they are the only copies. + +## 16. F9 self-arming + the operator's working commands (`68987ec`) + +WHAT: A12 arms from `os-public-hostname`; the operator's portable `base64` form, `VIP_OVERLAY:?` +guard, heredoc column-1 CAUTION and `${DC_LABEL:?}` all folded in; all 24 mint-refs re-anchored. +WHY: the commands that ACTUALLY produced both DCs' PKI were not in the repo -- the runbook +described a `base64 -w0` nobody ran. Found only because the operator asked before the bookend. +REVERT: `git revert 68987ec` restores a runbook describing unexecuted commands. Not advised. + +## 17. Headend reconcile + a gate red on the deploy host (`893d958`) + +WHAT: dc1's provider lock tracked (dc0's already was); `opentofu-validate`'s fmt check scoped to +repo content; harness 16/16. +WHY: the gauntlet had never been run on voffice1. It failed there and passed on vcloud, because +`tofu fmt -recursive` walks gitignored local tfvars that exist only on the deploy host. +REVERT: `git revert 893d958` re-breaks the headend gauntlet. + ## Agent deviation, disclosed and verified harmless One delivery agent used `git stash push/pop` on one file despite being forbidden any git write. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 81868bd..ce68ab3 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -261,3 +261,20 @@ - **OWNED:** three strict-bash quoting traps, two false-positive greps, one fabricated flag, one red-lint push. All corrected on-surface; traps in `docs/audit/queued-findings-20260729.txt`. - **NEXT:** octavia-pki generation -- the last item hard-failing preflight, and the generator is now per-DC. Gauntlet ALL GREEN (87) vcloud; repo-lint 0 fail. - Body: `docs/changelog-20260728-vip-arity-gate.md`. Status ONLY in CURRENT-STATE.md. + +## SESSION CLOSE 2026-07-30 -- Stage-5 preconditions: 19 Phase-3 items, gate integrity, per-DC Octavia PKI LIVE (bounded, GA-R4) + +- Branch `dc-dc-stage5-preconditions`, **25 commits pushed** (`ab4c842..`). NO stage opened/closed. Scan unchanged: 3 decisions, SEC 21, D 138 / DOCFIX 205 / BUNDLEFIX 053. +- **6 agents** (3 read-only recon, 4 delivery). Phase-3 re-measured **2 FIXED / 19 REMAIN / 5 NEW**, not the 21 the 07-27 doc implied; **13 items + 3.9/3.10 + R5 delivered**, phase-4 runbook 434 -> 929 lines. +- **F1 (HIGH): R7's per-DC Octavia PKI was HALF BUILT** -- subject and VIP gate per-DC, `$DC` in NO path, so generating dc1 would have DESTROYED dc0's CA and left one fixed-name overlay applying dc1's CA to dc0. Fixed end to end + REFUSE-IF-PRESENT. +- **F4:** the per-DC rename would have UN-IGNORED a CA private key in a repo SEC-004 calls PUBLIC. Glob + a `check-ignore` self-assert, proven both ways. **F2:** register was blind to a missing 2nd CA set (was RULED work, R13 Part 1). **F6:** P5 was probing the WRONG HOST'S filesystem and reporting it as fact -- 34 findings vs the true 7. +- **PKI GENERATED (operator-executed; guard blocks the mint by design, not worked around), both DCs verified PASS 29/0**, independence proven by sha256, backups **proven by restore** (PKI 12/12 vs live; tfstate byte-identical, correct serials). **E2 + E3 fully closed.** +- **`scripts/octavia-pki.sh verify` NEW** (harness 21/21): asserts the SAN set nothing asserted before (F8's SAN-less cert), and **A12 ARMS ITSELF** from `os-public-hostname` so F9 cannot be missed. +- **Gate integrity:** decorative HA, machines-overlay no-op, cloud-assert arity, G17 node check, gauntlet locale. P4 DC-aware (DC refs 1 -> 45); 3.7 closed; lib-net dc1 arm. +- **NEW ruling-shaped, now BLOCKING:** D-008 + D-106:2563 do not say whether substrate `vr1-dc1` is `dc1` by TOKEN or `dc2` by POSITION -- item 3.1's ambiguity inside a ruling, deciding cert identity. A12 REFUSES rather than picking. +- **Headend reconcile found a gate RED ON THE ONLY HOST THAT DEPLOYS** (`opentofu-validate` fmt walking gitignored tfvars). Fixed + scoped; **both hosts now ALL GREEN (89)**, repo-lint 0 fail. +- **Three wrong records found, none catchable by any gate:** D-124 told a reader to source a provider MAC from a file holding only boot MACs (would have bridged `br-ex` onto the PXE NIC); the skill cites D-107 for a tailnet path it does not rule; `office1-tailscale` is labelled a subnet router and advertises none. +- **A "backup set" asserted on three surfaces did not exist.** Built for the PKI (1.0-GEN.e) and the inner tfstates (phase-2 step 13), declared, restores proven. Forward requirement recorded: the pinned secrets-storage solution MUST absorb both. +- OWNED: the operator's own working commands were NOT in the repo until the pre-bookend sweep asked; I waved through 664 CA certs as "harmless" (integrity, not secrecy); two silenced errors (`| grep -q` under pipefail, `git pull -q &&`); four consecutive cwd mistakes. All corrected on-surface. +- **NEXT:** F8 residue (heredoc -> printf when a real generation can be run); D-137 fork-1 guard ruling (blocked correctly once, misfired 5x); preflight still has NO verified-green reading; G17 fixture-green only; the two `phase-04-network-*` scripts. +- Sweep capture `docs/audit/queued-findings-20260730.txt` (F10-F13 + 2 ruling questions). Body: `docs/changelog-20260729-stage5-preconditions.md`. Status ONLY in CURRENT-STATE.md.