Session changelog 2026-08-01 -- Stage 5 deploy prep (dc0)

Session opened after /clear with the GA-R1 bookend: git pull (already current), repo-lint 0 fail / 1 warn, ledger-scan reconciled against the 08-01 close summary (3 open decisions, 26 open SEC, next-free D-140 / DOCFIX-207 / BUNDLEFIX-053 -- all matching). run-tests-all deliberately SKIPPED at open (no tooling change yet; the 08-01 close recorded 96 GREEN) and run in full at item 3 once a script changed.

Status claims live in docs/CURRENT-STATE.md ONLY. Nothing here is a status claim.

Item 1 -- Four stale processes cleared from the jumphost (operator-approved)

What. Three orphaned background polling loops (PPID 1, ~34h old, from before the /clear) and one suspended Claude Code session (7 days, state T, 417 MiB) were terminated on vcloud.

Why. Two of the three loops were structurally incapable of ever exiting: until ! pgrep -f run-tests-all matches its own bash -c command line, so the guard can never be true. Proven by measurement rather than argued -- pgrep -f run-tests-all returned both loops AND the probe asking the question, while pgrep -af 'scripts/run-tests-all\.sh' returned no real script. The gauntlet they waited to announce had finished 34 hours earlier, and its payload (ALL GREEN (92)) was itself superseded by the current 96. The third loop awaited a maas=(Ready|Failed) pattern in a file with 0 matches, unmodified since 07-30 and with no open writers.

How. Identity asserted immediately before each signal (PPID==1 + cmdline substring), with a hard PROTECT list (live session, its shell, tmux server, the live daemon) that REFUSES rather than signals. TERM for the loops; the suspended session needed CONT+TERM and then SIGKILL, because its TERM handler resumed, touched a terminal it does not own (pgid != tpgid), took SIGTTIN and re-stopped. It had no children to orphan. Read-back: all four gone, all four protected alive, zero zombies, zero T-state processes.

NOT a leak, checked rather than assumed: the claude daemon at PPID 1 with a dead --spawned-by pid is the LIVE supervisor (5 unix sockets, 23 fds, auth: proactive refresh succeeded 4h earlier). Outliving its spawner is what --origin transient does. Reporting it as stale would have been a false finding.

Two defects in the sweep's own instruments, recorded because they are reusable: (i) grep -v grep eats pgrep -- the substring matches -- so the first shell inventory read clean while filtering out exactly the processes being hunted; (ii) the pgrep -f run-tests-all probe matched itself, reproducing the bug live. Guards: match on a PATH (pgrep -f 'scripts/run-tests-all\.sh') or pgrep -x, and bound waiters with timeout so a wrong predicate expires instead of spinning.

Revert. None possible or wanted (processes). Nothing in the repo changed.

Item 2 -- The dc0 MAAS path was down; DIAGNOSED, then restored (operator-gated)

What. Every dc0 MAAS call from voffice1 failed. maas-profile-assert.sh vr1-dc0-region hot-kid REFUSED; dc-node-v6-verify.sh plan vr1-dc0 --profile vr1-dc0-region -- the exact command captured PASS/exit 0 at 01:53Z -- returned REFUSE, exit 3.

Root cause, measured. The vr1-dc0-region profile is registered against http://127.0.0.1:5241/MAAS/api/2.0/ -- an SSH port-forward on voffice1, not the region's real address. That forward died when the dc0 rack rebooted at 05:48:34 during the 08-01 containment-VM resize. Nothing listens on 127.0.0.1:524x; no ssh -L process survives. Documented at changelog-20260730-dc0-region-migration.md:408 and CURRENT-STATE.md:452.

This settles an SEC-010 question raised during the diagnosis and answers it NO: the working state at 01:53Z was not a puncture. An SSH forward terminates on the rack and originates the API call there, which is precisely the rack-originated traffic SEC-010 permits. No kernel forwarding was ever involved and nothing ruled was violated.

How restored. The documented command, with -o ExitOnForwardFailure=yes added so a failed forward cannot exit 0 silently (declared deviation, not verbatim):

ssh voffice1 "ssh -f -N -o BatchMode=yes -o ExitOnForwardFailure=yes \
  -L 127.0.0.1:5241:10.12.8.6:5240 jessea123@172.31.0.2"

Pre-checked that nothing was already listening, so tunnels could not stack.

Proven, not assumed. Endpoint answers 200 with real capability JSON; plan vr1-dc0 PASS exit 0 (was REFUSE exit 3); maas-profile-assert vr1-dc0-region hot-kid OK exit 0 with the control admin-asserted-as-hot-kid still FAILing exit 1, so the gate retains its failing direction.

FINDING, logged not fixed (hard rule 1). The entire dc0 MAAS toolchain -- preflight P4, maas-profile-assert, maas-role-tags, dc-plane-ipam, dc-node-v6-verify, dc-node-v6-carve -- depends on a session-lifetime tunnel with no unit, no restart and no dependency on the rack it traverses. Any rack reboot silently disarms all of them, and they fail in a shape that reads as "the region is unreachable" rather than "your transport is down". Same class as the D-126/D-128 base-leg problem site-baseleg.sh exists to solve, except this reach is APPLICATION-layer, so site-baseleg.sh does not cover it and D-138 ruled out a host-side L3 leg. Whether it gets a durable unit is an unruled operator decision.

Also logged not fixed: dc-rack-net.sh's DNS_UPSTREAM="10.10.0.20" still points node DNS at voffice1's BIND across the fiber, while the 07-30 record says node DNS now uses the DC-local 10.12.8.6 and lists "rack DNS forwarder upstream" as OWED. The gate PASSes because it checks units and legs, not the upstream's correctness.

Revert. ssh voffice1 "pkill -f '5241:10.12.8.6'". Nothing in the cloud was touched.

Item 3 -- preflight P8: host guard used && where both markers are required (FIX)

What. scripts/preflight.sh:284, one operator character:

- elif [ ! -f opentofu/terraform.tfstate ] && [ ! -d opentofu/.terraform ]; then
+ elif [ ! -f opentofu/terraform.tfstate ] || [ ! -d opentofu/.terraform ]; then

plus a comment block stating why the two markers are not equivalent evidence.

Why -- a LIVE false FAIL, not a review finding. P8 hard-FAILED the dc0 preflight on voffice1 with tofu plan exited 1 ... variable "vr1_dc0_rack_transit_peer_ip" is not set. voffice1 has NO terraform.tfstate but DOES have opentofu/.terraform, because D-128 has it run the INNER tofu roots and tofu init leaves a provider cache. Under && neither warn branch fired, so P8 ran a real tofu plan that died on a gitignored tfvar. terraform.tfstate is the OWNERSHIP marker; .terraform is a cache present on any host that ever initialised any root. The gate therefore false-FAILED on the exact host the Stage-5 runbook designates for preflight, contradicting its own rule at preflight.sh:274 ("'not the substrate host' is a legitimate state, not a missing gate") and its own rationale that a permanently-red gate gets ignored.

Why no test caught it. T38 removes the WHOLE opentofu/ directory, so both markers vanish together and it can never distinguish them. The fixture itself sat in the defective state -- it created .terraform and no state file, and worked only BECAUSE the && was wrong. Fixed: mkfix now creates both markers, which is what a real substrate host looks like.

New harness case T39 -- provider cache present, state absent (voffice1's actual shape) -> must WARN, not FAIL. Proven able to fail: reverting || to && turns T39 and only T39 red (rc=0 want=2); script restored from a pristine snapshot and verified sha256-identical afterwards.

Stated limit, because the mutant's failure mode is not the live one: the harness's fake tofu returns zero-diff, so the mutant yields rc=0 rather than reproducing the live exit 1. T39 pins the BRANCH SELECTION, which is the defect; it does not reproduce the downstream tofu error.

Deliberately NOT changed (scope): the *) catch-all still FAILs on a non-0/2 tofu exit. On the substrate host a tofu plan error IS a real failure, and demoting it would weaken the gate.

Verification. tests/preflight 38 -> 39/39 PASS; gauntlet ALL GREEN (96); repo-lint 0 fail, 1 warn (standing legacy-ASCII carve-out).

Revert. git revert <this commit>, or restore && at preflight.sh:284 and drop T39 plus the mkfix state-file line.

Item 4 -- voffice1 clone pulled to HEAD

What. git pull --ff-only on voffice1: 844b2e4 -> f79c9e8, tree clean, same branch. Both clones now report identical HEAD hashes.

Why. The 08-01 close recorded voffice1 36 commits behind (F2 of that sweep); it had since closed to 3, but those 3 carried gate P8 -- so running preflight there would have silently used yesterday's gate set. This is the 07-27 stale-clone hazard on the D-128 Plane-2 host.

Revert. git -C ~/openstack-caracal-dc-dc reset --hard 844b2e4 on voffice1 (not wanted; the remote is authoritative).

Not done, and why

• Step 3.5 (add-model vr1-dc0 + spaces gate + apt-mirror) -- gated mutation, not yet approved this session.
• D-139 step 1, the apex CREATE-only push -- the recorded NEXT, but its ordering against the bundle deploy is UNRULED and is being put to the operator as its own GA-R5 exchange. Deploying on the current ULA carve means D-139 steps 2-6 later re-carve node addresses under a live cloud, which the entry warns is the expensive path since addresses bake into agent.conf at deploy time.
• A durable unit for the 5241 tunnel -- item 2's finding; unruled, logged only.

Item 5 -- D-139 step 1 EXECUTED (apex GUA carve, vr1-dc0)

See docs/audit/d139-step1-apex-push-dc0-20260801.txt and the CURRENT-STATE block. --commit exit 0, 13/13 read-back, apex 139 -> 152 verified by an INDEPENDENT query. Nothing deleted: 9 ULA prefixes and all 26 ULA VIP legs still resolve.

Revert. Delete the 13 created prefixes (they are new and carry no dependents).

Item 6 -- the VIP overlay re-rendered onto GUA, and the two apex readers taught FAMILY

What. overlays/vr1-dc0-vips.yaml now carries GUA on all 39 v6 VIP legs (was 13 GUA + 26 ULA). render/values/vr1-dc0-vips.yaml gains the two GUA prefixes, a refreshed apex_record, and a new v6_family: gua provenance key. scripts/render-dc-overlays.py gains v6_family() + --v6-family; scripts/provider-bundle-check.py gains per-leg, per-application family resolution.

Why it was needed. Step 1 creates GUA ALONGSIDE the ULA rows and step 6 retires the ULA rows, so for the WHOLE transition every v6-only plane has two /64s under one (role, kind) key. Both apex readers keyed without a family dimension, so both could only REFUSE. Measured live minutes after the push: derive --dual-family -> rc 2, and the gate -> "cannot evaluate barbican's dual-family vip". The D-136 pipeline was unrunnable for exactly the migration it exists to serve.

A LOSSY PATH FOUND AND NOT TAKEN. derive does not reproduce the per-app comment: fields, so a full re-derive silently dropped 13 comments and shrank the overlay 4368 -> 3593 bytes. Reverted; the values file was edited surgically instead (3 lines), giving exactly 26 changed overlay lines -- 13 vip lines x 2 -- with every comment intact. derive being lossy against its own committed values file is a real gap, LOGGED NOT FIXED.

TWO BUGS I INTRODUCED, BOTH CAUGHT BY THE HARNESS AND NOT BY REVIEW. (i) I inferred the family ONCE from parts[3], the provider leg -- but provider-public's VIP /64 is GUA in BOTH worlds (:11::), so every dc1 fixture asked for GUA bands on a still-ULA DC: 7 cases red. (ii) The corrected version still resolved bands ONCE for the whole bundle and reused them across all 13 applications, grading every later app against the first app's families. Fixed by resolving per application AND per leg, cached on the family triple.

>>> A PROPERTY WAS TRADED AWAY, AND IT IS NOT A WIN: THE GATE NO LONGER CATCHES A WRONG-FAMILY LEG. <<< Measured: substituting a ULA admin leg into the GUA dc0 overlay now PASSES, because per-leg resolution grades it against the ULA band -- which still exists until step 6. This is NECESSARY (dc1 is legitimately ULA, so no gate can hardcode a family) but it means D-139 family conformance is now checked by NOTHING. That was already recorded as a gap (docs/audit/d139-conformance-dc0-20260801.txt D.2); this change makes it the ONLY thing that would catch a DC deployed on the wrong family. A ruled-table conformance gate is OWED.

Verification. tests/render-dc-overlays 18 -> 23/23 (T17/T17b/T18/T19/T20), three mutations each killing named cases, script restored sha256-identical. tests/provider-bundle-check 55/55. Gauntlet ALL GREEN (96); repo-lint 0 fail / 1 warn. Renderer check reproduces the new overlay byte-for-byte. The real deploy input gates PASS with 13 clustered VIP(s) ... (13 dual-family), and a corrupted GUA leg still FAILS.

vr1-dc1 is UNTOUCHED -- its apex has no GUA, so its overlay stays ULA and now gates correctly through the same per-leg path.

Revert. git revert <commit>; the overlay's prior bytes are recoverable from the values file by restoring the three prefix lines and re-rendering.