diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index b2fb863..afe22dc 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1937,6 +1937,49 @@ observed failing is as untrustworthy as one only ever observed passing. `dc-node-v6-carve` then demonstrated it live -- flipping dc0 to PASS while dc1 still read FAIL. This is a COMPLEMENT to an existing invariant, not a new one, so it folds into that paragraph. +- **SUCCESSOR SESSION OPENED 2026-07-29 (operator: "Run through all items left all the way + through the end of stage ... Dispatch agents to assist"). FIVE FINDINGS RAISED, NONE + EXECUTED -- capture `docs/audit/stage5-findings-20260729-successor.md`.** Three read-only + reconciliation agents were dispatched over the Phase-3 runbook batch, the Phase-4 gate + set + a RULED-IS-NOT-BUILT sweep, and the Plane-2 execution environment; their results are + recorded separately when they land. The findings below are the orchestrator's own, each + MEASURED before being put to the operator. + **F1 (HIGH) -- R7's per-DC Octavia PKI is HALF BUILT: the artifact PATHS were never + parameterised.** The D-109 amendment rules "each DC gets its own Octavia CA; no cross-DC + amphora root-of-trust", and the 2026-07-29 execution correctly made the CA SUBJECT and the + VIP gate per-DC. But `$DC` appears in NO path: `phase-01-bundle-deploy.md:293` is + `WORKDIR="$HOME/octavia-pki"` and `:473` writes `overlays/octavia-pki.yaml`, both fixed. + Generating dc1's PKI after dc0's therefore OVERWRITES dc0's issuing-CA key, controller-CA + key and both passphrases, and leaves the single fixed-name overlay carrying dc1's CA -- so a + later dc0 redeploy reads that overlay by its fixed name and applies **dc1's CA to dc0**, + exactly the cross-DC shared root R7 refused. Nothing detects it: `phase-01:86` and + `pre-flight-checks.sh:65` both ask whether the file EXISTS, never whose CA it is (the + assert-on-existence class fixed in `dc-mirror.sh check` on 2026-07-27). **The gap is in + R7's own "Work implied" list, which names the subject, the gate and the generation but not + the paths -- the execution followed it faithfully.** Corroborating contradiction already on + this surface: line 990 describes one unscoped path as holding "per-DC generated CA material". + **F2 (MEDIUM) -- the credential register cannot see a missing second PKI set.** + `creds-matrix.tsv:103-111` declares all nine Octavia credentials `singleton`, and + `creds-matrix.py:368-398` enforces both-DC existence ONLY for rows marked `per-DC`. Behind + the BLOCKING P5 gate. Baseline measured so the delta is attributable: `--tier2` = 82 rows, + 15 groups clean, **7 findings, exit 1**. Must land BEFORE generation, or dc0's set alone + satisfies the register. + **F3 -- both `~/octavia-pki/` and `overlays/octavia-pki.yaml` are ABSENT here** (existence + checked, no contents read). So this is generation FROM SCRATCH for both DCs: there is + nothing to reuse, which retires the reuse-vs-regenerate choice + `dc-dc-phase4:204-209` frames as an operator call -- and means F1/F2 are fixable at ZERO + risk right now. That window closes the moment the first DC's PKI exists. + **F4 (HIGH, conditional) -- renaming the overlay would SILENTLY UN-IGNORE a CA private + key.** `.gitignore:40` is an exact path, not a glob; a per-DC name would not match it, and a + file holding CA key blobs plus a plaintext passphrase would become committable in a repo + SEC-004 records as PUBLIC. Recorded so the F1 fix cannot be taken without widening the + glob in the SAME commit, plus a `git check-ignore` self-assert in the generator. + **F5 (MEDIUM) -- preflight's `DC` selector does not reach P4.** `preflight.sh:99` sets + `DC` without exporting it, so propagation depends on the caller's invocation form; and it + is moot because `pre-flight-checks.sh` never reads `DC` at all (one hit, in a comment). + `DC=vr1-dc1 bash scripts/preflight.sh` runs a DC-aware P2 beside a dc0-frozen P4 under one + combined verdict. `preflight.sh:102` also hardcodes the octavia overlay name and inherits + F1's rename. - **RENDER-PIPELINE STEP 3 IS COMPLETE 2026-07-27 -- ALL FOUR LAYERS, BOTH DCs.** (Heading corrected at session close: it read "MAAS, lib-net AND APEX HALVES COMPLETE", which was true when written but became an UNDERSTATEMENT once the node carve landed above -- a reader diff --git a/docs/audit/stage5-findings-20260729-successor.md b/docs/audit/stage5-findings-20260729-successor.md new file mode 100644 index 0000000..23f8e72 --- /dev/null +++ b/docs/audit/stage5-findings-20260729-successor.md @@ -0,0 +1,200 @@ +# Stage-5 precondition findings -- successor session 2026-07-29 + +Findings raised by the successor session that opened after the 2026-07-29 close +bookend (`1cd45cd`). Each is an OBSERVATION MEASURED BEFORE BEING PUT to the +operator, per the skill's standing rule. Nothing here is adopted; nothing here is +executed. Status authority remains `docs/CURRENT-STATE.md`. + +--- + +## F1 -- R7's per-DC Octavia PKI is HALF BUILT: the artifact PATHS are still single and unscoped + +**Class: RULED-IS-NOT-BUILT.** Severity: HIGH -- silent destruction of a live DC's +CA material, with no gate anywhere that can detect it. + +### The ruling + +D-109 AMENDMENT (2026-07-27, R7), operator utterance *"Per-DC independent Octavia +PKI; fix the generator first (Recommended)"*. Decision text, verbatim: **"each DC +gets its own Octavia CA; no cross-DC amphora root-of-trust."** Reuse was refused +explicitly on posture -- one amphora CA private key across two clouds D-100 +defines as independent would widen the SEC-004 exposure rather than contain it. + +### What was built (verified present) + +- The CA SUBJECT is per-DC: `runbooks/phase-01-bundle-deploy.md:313-316` exports + `DC` / `DC_LABEL` / `VIP_OVERLAY`; `:328` uses `/CN=${DC_LABEL} ...`. +- The VIP gate reads the per-DC merged overlay rather than `bundle.yaml`'s dc0 + band (R7's explicit caveat), and the D-109 ruling note of 2026-07-29 added the + IPv6 IP SAN (`IP.1` v4 + `IP.2` v6, both derived per-DC). + +### What was NOT built (the defect) + +The artifact LOCATIONS are not parameterised by `$DC` at all: + +| Location | path:line | Value | +|---|---|---| +| Workspace root | `phase-01-bundle-deploy.md:293` | `WORKDIR="$HOME/octavia-pki"` | +| Issuing CA dir | `:320` | `"$WORKDIR/issuing-ca"` | +| Controller CA dir | `:342` | `"$WORKDIR/controller-ca"` | +| Controller dir | `:364` | `"$WORKDIR/controller"` | +| Overlay write | `:465,473,483` | `"$REPO/overlays/octavia-pki.yaml"` | + +`$DC` appears in the subject and the VIP gate and NOWHERE in a path. + +### Failure mode, stated concretely + +Generate dc0's PKI, deploy dc0. Later generate dc1's PKI per the same ruled step. +`1.0-GEN.0` runs `mkdir -p` over the SAME three directories and every subsequent +`openssl` write lands on dc0's files: `passphrase.txt`, `issuing-ca.key.enc`, +`issuing-ca.cert.pem`, `controller-ca.*`, `controller.key`, `controller.bundle.pem`. +`1.0-GEN.d` then overwrites `overlays/octavia-pki.yaml`. **dc0's issuing CA private +key, its controller CA private key and both passphrases are destroyed on disk**, and +the single fixed-name overlay now carries dc1's CA. + +Consequences beyond the loss itself: + +1. Any dc0 redeploy or `juju config` refresh reads the fixed overlay name at + `phase-01:34,86,148-153,180,241` and silently applies **dc1's CA** to dc0's + Octavia -- a cross-DC shared amphora root-of-trust, precisely the outcome R7 + refused. +2. Controller-cert rotation for dc0 becomes impossible: the controller CA key it + must be signed by is gone. +3. Nothing detects it. The existence checks are DC-blind -- `phase-01:86` is + `[ -f overlays/octavia-pki.yaml ]` and `scripts/pre-flight-checks.sh:65` sets + `OVL="overlays/octavia-pki.yaml"`. Both ask *does a file exist*, never *whose + CA is it*. This is the assert-on-existence-not-content class the repo fixed in + `dc-mirror.sh check` on 2026-07-27. + +### Why the ruling did not catch it + +R7's own "Work implied" list enumerates: parameterise the CA subject per DC; +DC-parameterise the VIP gate; then generate dc1's own PKI. It does not name the +paths. The 2026-07-29 execution followed that list faithfully and completely -- +the gap is in the list, not in the execution. + +Corroborating contradiction already on a status surface: `docs/CURRENT-STATE.md:990` +describes `overlays/octavia-pki.yaml` as "per-DC generated CA material under the R7 +amendment" -- a single unscoped path described as holding per-DC material. + +### Blast radius of the fix (measured, `grep -rn`) + +`runbooks/phase-01-bundle-deploy.md` (5 path blocks + 6 gate/deploy references), +`scripts/pre-flight-checks.sh:65`, `.gitignore:40-41`, `creds-matrix.tsv:103-111` +(see F2), `bundle.yaml:40,649,654` (comments), `README.md:35` (already stale -- +says `overlays/` holds one overlay; there are four tracked), and +`runbooks/dc-dc-phase4-juju-bundle-per-dc.md`'s deploy command. + +### Disposition + +LOGGED, NOT EXECUTED (hard rule 1). The PRINCIPLE is already ruled by the D-109 +amendment, so conformance is OPS under GA-R3 and needs no new D-number. The PATH +SCHEME is a choice on a secret-handling surface and is put to the operator. + +--- + +## F2 -- the credential register cannot see a missing second Octavia PKI set + +**Class: a register cannot report the absence of something it has no ROW for.** +Severity: MEDIUM. Direct consequence of F1 and fixed with it. + +Measured: `creds-matrix.tsv:103-111` declares all nine Octavia PKI credentials with +cardinality **`singleton`**, site-key `-`, host-role `jumphost`, mint-stage +`vr0-phase01`: + + octavia-issuing-ca-passphrase / -key / -cert + octavia-controller-ca-pass / -key / -cert + octavia-controller-key / -bundle + octavia-pki-overlay + +`scripts/creds-matrix.py:44` defines the vocabulary +`{singleton, per-site, per-DC, per-node, per-tenant}`, and `s5_per_dc_symmetry` +(`:368-398`) enforces both-DC existence **only for rows whose cardinality is +`per-DC`** -- it selects on `r.cardinality != "per-DC"` and skips. So under R7 the +register would match one PKI set and be structurally blind to whether the second +exists. That is the exact class the D-137 register was built to eliminate, and it +sits behind the BLOCKING preflight gate P5 (`scripts/preflight.sh:144`). + +Baseline measured this session, so the delta is attributable: +`python3 scripts/creds-matrix.py --tier2` -> **82 rows, 15 check groups clean, +7 findings, exit 1**. The nine Octavia rows are currently E0-SKIPPED because +`~/octavia-pki/*` does not exist ("a mint stage this deployment has not reached"), +which is correct today and is exactly why the blindness has not yet bitten. + +Correct fix is derived from the generator, not invented: the rows become `per-DC` +with region-qualified site-keys and whatever per-DC paths F1 settles on. **F2 must +land BEFORE the artifacts are generated** -- generating dc0's set against a +`singleton` register produces a register that looks satisfied while dc1's set is +absent. + +--- + +## F3 -- `~/octavia-pki/` and `overlays/octavia-pki.yaml` are ABSENT on this jumphost + +Measured, existence only, no contents read: `test -d ~/octavia-pki` -> ABSENT; +`ls overlays/` -> `dc-dc-ipv6-family-matrix.yaml dc-ha-scaleup.yaml +vr1-dc0-vips.yaml vr1-dc1-machines.yaml vr1-dc1-vips.yaml` (five tracked overlays, +no octavia-pki). So this is generation FROM SCRATCH for both DCs, not a +regeneration and not a reuse decision -- which removes the reuse-vs-regenerate +choice `dc-dc-phase4-juju-bundle-per-dc.md:204-209` frames as an operator call. +Nothing exists to reuse. + +Consequence for sequencing: F1 and F2 can both be fixed with ZERO risk to existing +material, because there is no existing material. The window in which this fix is +free closes the moment the first DC's PKI is generated. + +--- + +## F4 -- renaming the overlay would SILENTLY UN-IGNORE a CA private key + +**Class: coupling trap.** Severity: HIGH, but only if F1 is fixed carelessly. It is +recorded here so the F1 fix cannot be taken without it. + +`.gitignore:40` is an EXACT path, not a glob: + + overlays/octavia-pki.yaml + octavia-pki/ + passphrase.txt + +F1's fix renames the overlay per DC (e.g. `overlays/vr1-dc0-octavia-pki.yaml`). That +new name **does not match `.gitignore:40`**, so the moment the generator writes it, +a file containing base64 CA private-key blobs plus the issuing-CA passphrase IN +PLAINTEXT becomes an untracked-but-committable file -- in a repo `SEC-004` records +as currently **PUBLIC**. `docs/design-decisions.md` (R7 posture paragraph) states +that this gitignore is "the only thing standing between a CA private key and +publication"; `creds-matrix-notes.md:182-185` (`n-overlay-in-clone`) says the same. + +The F1 edit MUST therefore widen line 40 to a glob covering every per-DC name +(`overlays/*octavia-pki.yaml`) **in the same commit that renames anything**, and the +generator should verify its own output is ignored (`git check-ignore -q "$OUT"`) +before writing key material into it -- an assert-on-content gate rather than a +convention anybody must remember. `octavia-pki/` at line 41 is already a directory +glob and covers `~/octavia-pki//` without change. + +--- + +## F5 -- preflight's `DC` selector does not reach P4, and P4 is dc0-frozen + +**Class: gate does not measure what the operator asked it to measure.** +Severity: MEDIUM. Adjacent to, but not covered by, the 2026-07-29 P2 DC-awareness fix. + +Measured: + +- `scripts/preflight.sh:99` sets `DC="${DC:-vr1-dc0}"` and **never exports it**. + Invoked as `DC=vr1-dc1 bash scripts/preflight.sh` the variable arrives in the + environment and survives into children; invoked BARE it is a plain shell variable + and children see nothing. So propagation depends on the caller's invocation form + -- two call shapes, two different behaviours, neither documented. +- It does not matter yet, because `scripts/pre-flight-checks.sh` (P4) never reads + `DC` at all: `grep -n 'DC'` returns exactly one hit, inside a comment at `:80`. + Its own CHECK 1 comment admits the state -- "this check is still dc0-only -- it + never calls `lib_net_select_dc`" -- but that admission is scoped to CHECK 1, while + CHECK 0's `OVL="overlays/octavia-pki.yaml"` (`:65`) is equally unscoped. + +Net effect: `DC=vr1-dc1 bash scripts/preflight.sh` runs a DC-aware P2 and a +dc0-frozen P4 in the same invocation and prints one combined verdict. A dc1 operator +reading `PREFLIGHT` output cannot tell which halves measured dc1. + +Also note `preflight.sh:102` hardcodes `overlays/octavia-pki.yaml` in P2's overlay +assembly loop, so it inherits F1's rename and must change with it. +