diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 4114e16..f37e343 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2348,6 +2348,89 @@ **SCOPE NOTE, because it bounds what P8 can ever mean: tofu owns the SUBSTRATE only.** MAAS carves, the rack services, and the v6 node carve are script-driven by design (Model B / D-123). "Keep tofu current" never covered those and P8 does not check them. + **>>> STAGE-5 DEPLOY PREP 2026-08-01 (new session, post-`/clear`): THE dc0 MAAS PATH WAS + DOWN AND IS RESTORED; PREFLIGHT RE-RUN; P8 HAD A LIVE FALSE-FAIL DEFECT, NOW FIXED. <<<** + Session changelog `docs/changelog-20260801-stage5-deploy-prep.md`. + **THE MAAS PATH FAILURE AND ITS CAUSE, MEASURED.** Every dc0 MAAS call from voffice1 + failed -- `dc-node-v6-verify.sh plan vr1-dc0 --profile vr1-dc0-region`, the exact command + captured PASS/exit 0 at 01:53Z, returned **REFUSE exit 3**. **The `vr1-dc0-region` profile is + registered against `http://127.0.0.1:5241/` -- an SSH port-forward on voffice1, NOT the + region's address -- and that forward died when the dc0 rack rebooted at 05:48:34 during the + containment-VM resize recorded above.** Restored with the documented command + (`changelog-20260730-dc0-region-migration.md:408`) plus `-o ExitOnForwardFailure=yes` so a + failed forward cannot exit 0 silently. **PROVEN, not assumed:** endpoint 200 with real + capability JSON; `plan vr1-dc0` PASS exit 0; `maas-profile-assert vr1-dc0-region hot-kid` OK + exit 0 **with the control `admin`-asserted-as-`hot-kid` still FAILing exit 1**, so the gate + keeps its failing direction. **THIS ALSO ANSWERS AN SEC-010 QUESTION RAISED DURING THE + DIAGNOSIS, AND THE ANSWER IS NO: the 01:53Z working state was NOT a puncture.** An SSH + forward terminates ON the rack and originates the API call there -- exactly the + rack-originated traffic SEC-010 permits. No kernel forwarding; nothing ruled was violated. + **FINDING, LOGGED NOT FIXED, and it is the consequential one: the whole dc0 MAAS toolchain + (preflight P4, `maas-profile-assert`, `maas-role-tags`, `dc-plane-ipam`, + `dc-node-v6-verify`, `dc-node-v6-carve`) depends on a session-lifetime tunnel with no unit, + no restart and no dependency on the rack it traverses.** Any rack reboot silently disarms + all of them, and they fail in a shape that reads as "the region is unreachable" rather than + "your transport is down". Same class as the D-126/D-128 base-leg gap `site-baseleg.sh` + exists to close, except this reach is APPLICATION-layer, so that tool does not cover it and + D-138 ruled out a host-side L3 leg. Whether it gets a durable unit is UNRULED. + **NODE STATE CONFIRMED FROM MAAS, not merely from virsh:** `plan vr1-dc0` enumerates **all + nine role nodes under their ruled names** (`vr1-dc0-{compute-01,compute-02,control-01, + control-02,control-03,storage-01..04}`) with an IDENTICAL six-plane v6 set across all nine. + `virsh list --all` on the rack independently shows those nine `shut off` (consistent with + `Ready`, and it rules out anything still deployed) plus the three infra VMs running. + **MODEL `vr1-dc0` DOES NOT EXIST** -- `juju models` lists only `controller`; it went with + the 07-31 teardown, so Step 3.5 is owed. Controller `vr1-dc0-controller` is LIVE + ("Last connection: just now"). + **PREFLIGHT RE-RUN, `DC=vr1-dc0 MAAS_PROFILE=vr1-dc0-region`, ON voffice1 at HEAD `f79c9e8`. + EXACTLY 12 `[FAIL]` LINES: 11 from P5, 1 from P8. Nothing else is red.** P3 PASS; **P4 PASS** + (six planes by CIDR, 13 aligned VIPs 0 bad, `MAAS reachable (profile=vr1-dc0-region)`, + metal-internal UNTAGGED per D-133, overlay present with 5 `lb-mgmt-*` keys, ASCII clean) -- + **the 19 region-blindness false negatives recorded 2026-07-31 are DISCHARGED by using the + region profile**; **P7 PASS 37 assertions / 0 failed** including the literal zone line. + **P5's 11 findings were compared BY IDENTITY, not by count, because the 07-30 ruling covers + "these six, enumerated, and nothing else": the six (dc0 `opnsense-api.txt` SEC-021; three S5 + power-key asymmetries; S6 `maas-region-admin` conflation; E4 x2) plus the five ruled + 2026-07-31 (dc1 `maas-region-{admin-password,api-key.txt,db-password}` SEC-027; dc1 + `maas-juju-{api-key.txt,user-password}` SEC-028). ELEVEN accepted, eleven present, ZERO + NEW** -- so no fresh GA-R5 exchange is owed and the P5 RED remains ruled-accepted. + **P8 FAILED FOR THE WRONG REASON -- A LIVE DEFECT IN A GATE BUILT THE PREVIOUS DAY, NOW + FIXED.** Its host guard read `[ ! -f opentofu/terraform.tfstate ] && [ ! -d + opentofu/.terraform ]`. voffice1 has NO state file but DOES have `opentofu/.terraform`, + because D-128 has it run the INNER tofu roots and `tofu init` leaves a provider cache -- so + neither warn branch fired, P8 ran a real `tofu plan`, and it died on a gitignored tfvar + (`vr1_dc0_rack_transit_peer_ip`) -> exit 1 -> hard FAIL **on the very host the Stage-5 + runbook designates for preflight**, contradicting the gate's own rule that "not the + substrate host" is a legitimate state. **`terraform.tfstate` is the OWNERSHIP marker; + `.terraform` is a cache present on any host that ever initialised any root** -- the two are + not equivalent evidence and BOTH must be present to evaluate, so the operator is `||`. + **WHY NO TEST CAUGHT IT: T38 removes the WHOLE `opentofu/` directory, so both markers vanish + together and it can never distinguish them -- and the fixture itself sat in the defective + state, creating `.terraform` with no state file and passing only BECAUSE the `&&` was + wrong.** Fixture now creates both. **New case T39** (cache present, state absent = voffice1's + real shape -> WARN) **proven able to fail: reverting `||` to `&&` turns T39 and ONLY T39 red; + script restored from a pristine snapshot and sha256-verified identical.** STATED LIMIT: the + harness's fake `tofu` returns zero-diff, so the mutant yields `rc=0` rather than reproducing + the live `exit 1` -- T39 pins the BRANCH SELECTION, which is the defect, not the downstream + error. The `*)` catch-all deliberately still FAILs: on the substrate host a `tofu plan` error + IS a real failure. `tests/preflight` 38 -> **39/39**; gauntlet **ALL GREEN (96)**; repo-lint + **0 fail, 1 warn**. **P8's ACTUAL SUBJECT IS CLEAN, measured on vcloud (D-128 Plane 1): + `tofu plan -detailed-exitcode` exit 0, 0 pending actions, 0 observed drift, "No changes. + Your infrastructure matches the configuration."** + **BOTH CLONES NOW AT `f79c9e8`** -- voffice1 pulled `844b2e4` -> `f79c9e8` (it was 3 behind, + and those 3 carried P8, so preflight there would have used yesterday's gate set). The 08-01 + sweep's F2 stale-clone hazard is CLOSED. + **ALSO LOGGED NOT FIXED: `dc-rack-net.sh`'s `DNS_UPSTREAM="10.10.0.20"`** still points node + DNS at voffice1's BIND across the fiber while the 07-30 record says node DNS now uses the + DC-local `10.12.8.6` and lists "rack DNS forwarder upstream" as OWED; `dc-rack-net.sh check + dc0` PASSes (10 assertions, both units active+enabled) because it checks units and legs, not + the upstream's correctness. The 05:48 rack reboot otherwise recovered EVERYTHING + reboot-persistent unaided: region API 200, mirror 200, snap proxy listening, juju controller + connected. **The single casualty was the un-unitised tunnel.** + **STILL OWED BEFORE THE DEPLOY:** Step 3.5 (`add-model vr1-dc0` + spaces gate + + `apt-mirror`), then Step 4; and the UNRULED ordering question of whether D-139's apex + CREATE-only push precedes the bundle deploy -- deploying on the current ULA carve means + D-139 steps 2-6 later re-carve node addresses under a live cloud, the expensive path that + entry warns about since addresses bake into `agent.conf` at deploy time. - Project: Omega Cloud, VR1 DC-DC rehearsal -- a two-DC + Office1-headend virtual rehearsal on KVM (vcloud host), rehearsing the future bare-metal Roosevelt deployment (D-100, `docs/design-decisions.md:1946`). diff --git a/docs/changelog-20260801-stage5-deploy-prep.md b/docs/changelog-20260801-stage5-deploy-prep.md new file mode 100644 index 0000000..30ec381 --- /dev/null +++ b/docs/changelog-20260801-stage5-deploy-prep.md @@ -0,0 +1,176 @@ +# Session changelog 2026-08-01 -- Stage 5 deploy prep (dc0) + +Session opened after `/clear` with the GA-R1 bookend: `git pull` (already current), +`repo-lint` 0 fail / 1 warn, `ledger-scan` reconciled against the 08-01 close summary +(3 open decisions, 26 open SEC, next-free D-140 / DOCFIX-207 / BUNDLEFIX-053 -- all +matching). `run-tests-all` deliberately SKIPPED at open (no tooling change yet; the +08-01 close recorded 96 GREEN) and run in full at item 3 once a script changed. + +Status claims live in `docs/CURRENT-STATE.md` ONLY. Nothing here is a status claim. + +--- + +## Item 1 -- Four stale processes cleared from the jumphost (operator-approved) + +**What.** Three orphaned background polling loops (PPID 1, ~34h old, from before the +`/clear`) and one suspended Claude Code session (7 days, state `T`, 417 MiB) were +terminated on vcloud. + +**Why.** Two of the three loops were structurally incapable of ever exiting: +`until ! pgrep -f run-tests-all` matches **its own `bash -c` command line**, so the +guard can never be true. Proven by measurement rather than argued -- `pgrep -f +run-tests-all` returned both loops AND the probe asking the question, while +`pgrep -af 'scripts/run-tests-all\.sh'` returned no real script. The gauntlet they +waited to announce had finished 34 hours earlier, and its payload (`ALL GREEN (92)`) +was itself superseded by the current 96. The third loop awaited a +`maas=(Ready|Failed)` pattern in a file with 0 matches, unmodified since 07-30 and +with no open writers. + +**How.** Identity asserted immediately before each signal (PPID==1 + cmdline +substring), with a hard PROTECT list (live session, its shell, tmux server, the +live daemon) that REFUSES rather than signals. TERM for the loops; the suspended +session needed CONT+TERM and then SIGKILL, because its TERM handler resumed, touched +a terminal it does not own (`pgid` != `tpgid`), took SIGTTIN and re-stopped. It had +no children to orphan. Read-back: all four gone, all four protected alive, zero +zombies, zero `T`-state processes. + +**NOT a leak, checked rather than assumed:** the `claude daemon` at PPID 1 with a +dead `--spawned-by` pid is the LIVE supervisor (5 unix sockets, 23 fds, +`auth: proactive refresh succeeded` 4h earlier). Outliving its spawner is what +`--origin transient` does. Reporting it as stale would have been a false finding. + +**Two defects in the sweep's own instruments, recorded because they are reusable:** +(i) `grep -v grep` **eats `pgrep`** -- the substring matches -- so the first shell +inventory read clean while filtering out exactly the processes being hunted; +(ii) the `pgrep -f run-tests-all` probe matched itself, reproducing the bug live. +Guards: match on a PATH (`pgrep -f 'scripts/run-tests-all\.sh'`) or `pgrep -x`, and +bound waiters with `timeout` so a wrong predicate expires instead of spinning. + +**Revert.** None possible or wanted (processes). Nothing in the repo changed. + +--- + +## Item 2 -- The dc0 MAAS path was down; DIAGNOSED, then restored (operator-gated) + +**What.** Every dc0 MAAS call from voffice1 failed. `maas-profile-assert.sh +vr1-dc0-region hot-kid` REFUSED; `dc-node-v6-verify.sh plan vr1-dc0 --profile +vr1-dc0-region` -- the exact command captured PASS/exit 0 at 01:53Z -- returned +**REFUSE, exit 3**. + +**Root cause, measured.** The `vr1-dc0-region` profile is registered against +`http://127.0.0.1:5241/MAAS/api/2.0/` -- an SSH port-forward on voffice1, not the +region's real address. That forward **died when the dc0 rack rebooted at 05:48:34** +during the 08-01 containment-VM resize. Nothing listens on `127.0.0.1:524x`; no +`ssh -L` process survives. Documented at `changelog-20260730-dc0-region-migration.md:408` +and `CURRENT-STATE.md:452`. + +**This settles an SEC-010 question raised during the diagnosis and answers it NO:** +the working state at 01:53Z was **not** a puncture. An SSH forward terminates on the +rack and originates the API call there, which is precisely the rack-originated traffic +SEC-010 permits. No kernel forwarding was ever involved and nothing ruled was violated. + +**How restored.** The documented command, with `-o ExitOnForwardFailure=yes` added so +a failed forward cannot exit 0 silently (declared deviation, not verbatim): + + ssh voffice1 "ssh -f -N -o BatchMode=yes -o ExitOnForwardFailure=yes \ + -L 127.0.0.1:5241:10.12.8.6:5240 jessea123@172.31.0.2" + +Pre-checked that nothing was already listening, so tunnels could not stack. + +**Proven, not assumed.** Endpoint answers 200 with real capability JSON; +`plan vr1-dc0` PASS exit 0 (was REFUSE exit 3); `maas-profile-assert vr1-dc0-region +hot-kid` OK exit 0 with the control `admin`-asserted-as-`hot-kid` still FAILing exit 1, +so the gate retains its failing direction. + +**FINDING, logged not fixed (hard rule 1).** The entire dc0 MAAS toolchain -- +preflight P4, `maas-profile-assert`, `maas-role-tags`, `dc-plane-ipam`, +`dc-node-v6-verify`, `dc-node-v6-carve` -- depends on a **session-lifetime tunnel with +no unit, no restart and no dependency on the rack it traverses**. Any rack reboot +silently disarms all of them, and they fail in a shape that reads as "the region is +unreachable" rather than "your transport is down". Same class as the D-126/D-128 +base-leg problem `site-baseleg.sh` exists to solve, except this reach is +APPLICATION-layer, so `site-baseleg.sh` does not cover it and D-138 ruled out a +host-side L3 leg. Whether it gets a durable unit is an unruled operator decision. + +**Also logged not fixed:** `dc-rack-net.sh`'s `DNS_UPSTREAM="10.10.0.20"` still points +node DNS at voffice1's BIND across the fiber, while the 07-30 record says node DNS now +uses the DC-local `10.12.8.6` and lists "rack DNS forwarder upstream" as OWED. The gate +PASSes because it checks units and legs, not the upstream's correctness. + +**Revert.** `ssh voffice1 "pkill -f '5241:10.12.8.6'"`. Nothing in the cloud was touched. + +--- + +## Item 3 -- preflight P8: host guard used `&&` where both markers are required (FIX) + +**What.** `scripts/preflight.sh:284`, one operator character: + + - elif [ ! -f opentofu/terraform.tfstate ] && [ ! -d opentofu/.terraform ]; then + + elif [ ! -f opentofu/terraform.tfstate ] || [ ! -d opentofu/.terraform ]; then + +plus a comment block stating why the two markers are not equivalent evidence. + +**Why -- a LIVE false FAIL, not a review finding.** P8 hard-FAILED the dc0 preflight on +voffice1 with `tofu plan exited 1 ... variable "vr1_dc0_rack_transit_peer_ip" is not set`. +voffice1 has NO `terraform.tfstate` but DOES have `opentofu/.terraform`, because D-128 +has it run the INNER tofu roots and `tofu init` leaves a provider cache. Under `&&` +neither warn branch fired, so P8 ran a real `tofu plan` that died on a gitignored tfvar. +**`terraform.tfstate` is the OWNERSHIP marker; `.terraform` is a cache present on any host +that ever initialised any root.** The gate therefore false-FAILED on the exact host the +Stage-5 runbook designates for preflight, contradicting its own rule at `preflight.sh:274` +("'not the substrate host' is a legitimate state, not a missing gate") and its own +rationale that a permanently-red gate gets ignored. + +**Why no test caught it.** T38 removes the WHOLE `opentofu/` directory, so both markers +vanish together and it can never distinguish them. **The fixture itself sat in the +defective state** -- it created `.terraform` and no state file, and worked only BECAUSE +the `&&` was wrong. Fixed: `mkfix` now creates both markers, which is what a real +substrate host looks like. + +**New harness case T39** -- provider cache present, state absent (voffice1's actual +shape) -> must WARN, not FAIL. **Proven able to fail:** reverting `||` to `&&` turns +**T39 and only T39** red (`rc=0 want=2`); script restored from a pristine snapshot and +verified sha256-identical afterwards. + +**Stated limit, because the mutant's failure mode is not the live one:** the harness's +fake `tofu` returns zero-diff, so the mutant yields `rc=0` rather than reproducing the +live `exit 1`. T39 pins the BRANCH SELECTION, which is the defect; it does not reproduce +the downstream tofu error. + +**Deliberately NOT changed (scope):** the `*)` catch-all still FAILs on a non-0/2 tofu +exit. On the substrate host a `tofu plan` error IS a real failure, and demoting it would +weaken the gate. + +**Verification.** `tests/preflight` 38 -> **39/39 PASS**; gauntlet **ALL GREEN (96)**; +repo-lint **0 fail, 1 warn** (standing legacy-ASCII carve-out). + +**Revert.** `git revert `, or restore `&&` at `preflight.sh:284` and drop +T39 plus the `mkfix` state-file line. + +--- + +## Item 4 -- voffice1 clone pulled to HEAD + +**What.** `git pull --ff-only` on voffice1: `844b2e4` -> `f79c9e8`, tree clean, same +branch. Both clones now report identical HEAD hashes. + +**Why.** The 08-01 close recorded voffice1 36 commits behind (F2 of that sweep); it had +since closed to 3, but those 3 carried gate **P8** -- so running preflight there would +have silently used yesterday's gate set. This is the 07-27 stale-clone hazard on the +D-128 Plane-2 host. + +**Revert.** `git -C ~/openstack-caracal-dc-dc reset --hard 844b2e4` on voffice1 (not +wanted; the remote is authoritative). + +--- + +## Not done, and why + +- **Step 3.5** (`add-model vr1-dc0` + spaces gate + `apt-mirror`) -- gated mutation, + not yet approved this session. +- **D-139 step 1, the apex CREATE-only push** -- the recorded NEXT, but its ordering + against the bundle deploy is UNRULED and is being put to the operator as its own + GA-R5 exchange. Deploying on the current ULA carve means D-139 steps 2-6 later + re-carve node addresses under a live cloud, which the entry warns is the expensive + path since addresses bake into `agent.conf` at deploy time. +- **A durable unit for the 5241 tunnel** -- item 2's finding; unruled, logged only. diff --git a/scripts/preflight.sh b/scripts/preflight.sh index 31f5c11..ffb5eb3 100644 --- a/scripts/preflight.sh +++ b/scripts/preflight.sh @@ -281,7 +281,16 @@ echo " [warn] tofu not on PATH -- substrate drift NOT evaluated here (run on the vcloud" echo " substrate host). This is 'could not look', never 'nothing there'." note 2 "P8 substrate drift" warn -elif [ ! -f opentofu/terraform.tfstate ] && [ ! -d opentofu/.terraform ]; then +elif [ ! -f opentofu/terraform.tfstate ] || [ ! -d opentofu/.terraform ]; then + # `||`, NOT `&&` (fixed 2026-08-01 after a live false FAIL on voffice1). The two markers + # are NOT equivalent evidence and must BOTH be present to evaluate: terraform.tfstate is + # the OWNERSHIP marker (only the substrate host holds it, and it is gitignored), while + # .terraform is merely a provider cache that `tofu init` leaves on ANY host that ever + # initialised ANY root -- which by D-128 includes voffice1, since it runs the INNER roots. + # Under the old `&&`, voffice1 satisfied neither warn branch, fell through to `tofu plan`, + # and died on a gitignored tfvar -> exit 1 -> hard FAIL on the very host the Stage-5 + # runbook designates for preflight. That contradicts this gate's own rule above: "not the + # substrate host" is a legitimate state, not a missing gate. echo " [warn] no outer-root state here -- substrate drift NOT evaluated (D-128: the" echo " outer root lives on vcloud). Not a failure; not a pass either." note 2 "P8 substrate drift" warn diff --git a/tests/preflight/run-tests.sh b/tests/preflight/run-tests.sh index 74bc703..8d92993 100644 --- a/tests/preflight/run-tests.sh +++ b/tests/preflight/run-tests.sh @@ -77,7 +77,11 @@ # so the seam supplies BOTH. $7 selects the plan shape: zerodiff | pending | observed | # unparsed | error. Asserting P8 needs a fake `tofu` because the real one would touch # the live substrate, which a harness must never do. + # BOTH markers, because P8 evaluates only when BOTH are present (see preflight.sh:284). + # The state file is the OWNERSHIP marker; .terraform alone is just a provider cache and + # is NOT evidence this host owns the outer root -- T39 pins that distinction. mkdir -p "$d/opentofu/.terraform" + : > "$d/opentofu/terraform.tfstate" cat > "$d/fakebin/tofu" < WARN, never silent" "$D" +# T39: THE SHAPE THAT ACTUALLY BIT, and that T38 could not catch (added 2026-08-01 after a +# live false FAIL). voffice1 has NO terraform.tfstate but DOES have opentofu/.terraform, +# because D-128 has it run the INNER tofu roots and `tofu init` leaves a provider cache +# behind. T38 removes the WHOLE directory, so both markers vanish together and it can never +# distinguish them. Under the old `&&` guard this fell through to a real `tofu plan`, which +# died on a gitignored tfvar and hard-FAILED preflight on the one host the Stage-5 runbook +# designates for it. The state file is the OWNERSHIP marker; the cache is not. +D=$(mkfix p8_cache_no_state 0 0 1 0 ok) +rm -f "$D/opentofu/terraform.tfstate" # cache REMAINS -- that is the whole point +[[ -d "$D/opentofu/.terraform" ]] || { echo "FIXTURE BROKEN: T39 needs .terraform present"; exit 1; } +run 2 'no outer-root state here' "T39 P8 provider cache but NO state -> WARN, not FAIL" "$D" + echo; echo "RESULT: PASS=$PASS FAIL=$FAIL" [[ "$FAIL" -eq 0 ]] && { echo "ALL PASS"; exit 0; } || exit 1