# Session changelog 2026-08-01 -- Stage 5 deploy prep (dc0)

Session opened after `/clear` with the GA-R1 bookend: `git pull` (already current),
`repo-lint` 0 fail / 1 warn, `ledger-scan` reconciled against the 08-01 close summary
(3 open decisions, 26 open SEC, next-free D-140 / DOCFIX-207 / BUNDLEFIX-053 -- all
matching). `run-tests-all` deliberately SKIPPED at open (no tooling change yet; the
08-01 close recorded 96 GREEN) and run in full at item 3 once a script changed.

Status claims live in `docs/CURRENT-STATE.md` ONLY. Nothing here is a status claim.

---

## Item 1 -- Four stale processes cleared from the jumphost (operator-approved)

**What.** Three orphaned background polling loops (PPID 1, ~34h old, from before the
`/clear`) and one suspended Claude Code session (7 days, state `T`, 417 MiB) were
terminated on vcloud.

**Why.** Two of the three loops were structurally incapable of ever exiting:
`until ! pgrep -f run-tests-all` matches **its own `bash -c` command line**, so the
guard can never be true. Proven by measurement rather than argued -- `pgrep -f
run-tests-all` returned both loops AND the probe asking the question, while
`pgrep -af 'scripts/run-tests-all\.sh'` returned no real script. The gauntlet they
waited to announce had finished 34 hours earlier, and its payload (`ALL GREEN (92)`)
was itself superseded by the current 96. The third loop awaited a
`maas=(Ready|Failed)` pattern in a file with 0 matches, unmodified since 07-30 and
with no open writers.

**How.** Identity asserted immediately before each signal (PPID==1 + cmdline
substring), with a hard PROTECT list (live session, its shell, tmux server, the
live daemon) that REFUSES rather than signals. TERM for the loops; the suspended
session needed CONT+TERM and then SIGKILL, because its TERM handler resumed, touched
a terminal it does not own (`pgid` != `tpgid`), took SIGTTIN and re-stopped. It had
no children to orphan. Read-back: all four gone, all four protected alive, zero
zombies, zero `T`-state processes.

**NOT a leak, checked rather than assumed:** the `claude daemon` at PPID 1 with a
dead `--spawned-by` pid is the LIVE supervisor (5 unix sockets, 23 fds,
`auth: proactive refresh succeeded` 4h earlier). Outliving its spawner is what
`--origin transient` does. Reporting it as stale would have been a false finding.

**Two defects in the sweep's own instruments, recorded because they are reusable:**
(i) `grep -v grep` **eats `pgrep`** -- the substring matches -- so the first shell
inventory read clean while filtering out exactly the processes being hunted;
(ii) the `pgrep -f run-tests-all` probe matched itself, reproducing the bug live.
Guards: match on a PATH (`pgrep -f 'scripts/run-tests-all\.sh'`) or `pgrep -x`, and
bound waiters with `timeout` so a wrong predicate expires instead of spinning.

**Revert.** None possible or wanted (processes). Nothing in the repo changed.

---

## Item 2 -- The dc0 MAAS path was down; DIAGNOSED, then restored (operator-gated)

**What.** Every dc0 MAAS call from voffice1 failed. `maas-profile-assert.sh
vr1-dc0-region hot-kid` REFUSED; `dc-node-v6-verify.sh plan vr1-dc0 --profile
vr1-dc0-region` -- the exact command captured PASS/exit 0 at 01:53Z -- returned
**REFUSE, exit 3**.

**Root cause, measured.** The `vr1-dc0-region` profile is registered against
`http://127.0.0.1:5241/MAAS/api/2.0/` -- an SSH port-forward on voffice1, not the
region's real address. That forward **died when the dc0 rack rebooted at 05:48:34**
during the 08-01 containment-VM resize. Nothing listens on `127.0.0.1:524x`; no
`ssh -L` process survives. Documented at `changelog-20260730-dc0-region-migration.md:408`
and `CURRENT-STATE.md:452`.

**This settles an SEC-010 question raised during the diagnosis and answers it NO:**
the working state at 01:53Z was **not** a puncture. An SSH forward terminates on the
rack and originates the API call there, which is precisely the rack-originated traffic
SEC-010 permits. No kernel forwarding was ever involved and nothing ruled was violated.

**How restored.** The documented command, with `-o ExitOnForwardFailure=yes` added so
a failed forward cannot exit 0 silently (declared deviation, not verbatim):

    ssh voffice1 "ssh -f -N -o BatchMode=yes -o ExitOnForwardFailure=yes \
      -L 127.0.0.1:5241:10.12.8.6:5240 jessea123@172.31.0.2"

Pre-checked that nothing was already listening, so tunnels could not stack.

**Proven, not assumed.** Endpoint answers 200 with real capability JSON;
`plan vr1-dc0` PASS exit 0 (was REFUSE exit 3); `maas-profile-assert vr1-dc0-region
hot-kid` OK exit 0 with the control `admin`-asserted-as-`hot-kid` still FAILing exit 1,
so the gate retains its failing direction.

**FINDING, logged not fixed (hard rule 1).** The entire dc0 MAAS toolchain --
preflight P4, `maas-profile-assert`, `maas-role-tags`, `dc-plane-ipam`,
`dc-node-v6-verify`, `dc-node-v6-carve` -- depends on a **session-lifetime tunnel with
no unit, no restart and no dependency on the rack it traverses**. Any rack reboot
silently disarms all of them, and they fail in a shape that reads as "the region is
unreachable" rather than "your transport is down". Same class as the D-126/D-128
base-leg problem `site-baseleg.sh` exists to solve, except this reach is
APPLICATION-layer, so `site-baseleg.sh` does not cover it and D-138 ruled out a
host-side L3 leg. Whether it gets a durable unit is an unruled operator decision.

**Also logged not fixed:** `dc-rack-net.sh`'s `DNS_UPSTREAM="10.10.0.20"` still points
node DNS at voffice1's BIND across the fiber, while the 07-30 record says node DNS now
uses the DC-local `10.12.8.6` and lists "rack DNS forwarder upstream" as OWED. The gate
PASSes because it checks units and legs, not the upstream's correctness.

**Revert.** `ssh voffice1 "pkill -f '5241:10.12.8.6'"`. Nothing in the cloud was touched.

---

## Item 3 -- preflight P8: host guard used `&&` where both markers are required (FIX)

**What.** `scripts/preflight.sh:284`, one operator character:

    - elif [ ! -f opentofu/terraform.tfstate ] && [ ! -d opentofu/.terraform ]; then
    + elif [ ! -f opentofu/terraform.tfstate ] || [ ! -d opentofu/.terraform ]; then

plus a comment block stating why the two markers are not equivalent evidence.

**Why -- a LIVE false FAIL, not a review finding.** P8 hard-FAILED the dc0 preflight on
voffice1 with `tofu plan exited 1 ... variable "vr1_dc0_rack_transit_peer_ip" is not set`.
voffice1 has NO `terraform.tfstate` but DOES have `opentofu/.terraform`, because D-128
has it run the INNER tofu roots and `tofu init` leaves a provider cache. Under `&&`
neither warn branch fired, so P8 ran a real `tofu plan` that died on a gitignored tfvar.
**`terraform.tfstate` is the OWNERSHIP marker; `.terraform` is a cache present on any host
that ever initialised any root.** The gate therefore false-FAILED on the exact host the
Stage-5 runbook designates for preflight, contradicting its own rule at `preflight.sh:274`
("'not the substrate host' is a legitimate state, not a missing gate") and its own
rationale that a permanently-red gate gets ignored.

**Why no test caught it.** T38 removes the WHOLE `opentofu/` directory, so both markers
vanish together and it can never distinguish them. **The fixture itself sat in the
defective state** -- it created `.terraform` and no state file, and worked only BECAUSE
the `&&` was wrong. Fixed: `mkfix` now creates both markers, which is what a real
substrate host looks like.

**New harness case T39** -- provider cache present, state absent (voffice1's actual
shape) -> must WARN, not FAIL. **Proven able to fail:** reverting `||` to `&&` turns
**T39 and only T39** red (`rc=0 want=2`); script restored from a pristine snapshot and
verified sha256-identical afterwards.

**Stated limit, because the mutant's failure mode is not the live one:** the harness's
fake `tofu` returns zero-diff, so the mutant yields `rc=0` rather than reproducing the
live `exit 1`. T39 pins the BRANCH SELECTION, which is the defect; it does not reproduce
the downstream tofu error.

**Deliberately NOT changed (scope):** the `*)` catch-all still FAILs on a non-0/2 tofu
exit. On the substrate host a `tofu plan` error IS a real failure, and demoting it would
weaken the gate.

**Verification.** `tests/preflight` 38 -> **39/39 PASS**; gauntlet **ALL GREEN (96)**;
repo-lint **0 fail, 1 warn** (standing legacy-ASCII carve-out).

**Revert.** `git revert <this commit>`, or restore `&&` at `preflight.sh:284` and drop
T39 plus the `mkfix` state-file line.

---

## Item 4 -- voffice1 clone pulled to HEAD

**What.** `git pull --ff-only` on voffice1: `844b2e4` -> `f79c9e8`, tree clean, same
branch. Both clones now report identical HEAD hashes.

**Why.** The 08-01 close recorded voffice1 36 commits behind (F2 of that sweep); it had
since closed to 3, but those 3 carried gate **P8** -- so running preflight there would
have silently used yesterday's gate set. This is the 07-27 stale-clone hazard on the
D-128 Plane-2 host.

**Revert.** `git -C ~/openstack-caracal-dc-dc reset --hard 844b2e4` on voffice1 (not
wanted; the remote is authoritative).

---

## Not done, and why

- **Step 3.5** (`add-model vr1-dc0` + spaces gate + `apt-mirror`) -- gated mutation,
  not yet approved this session.
- **D-139 step 1, the apex CREATE-only push** -- the recorded NEXT, but its ordering
  against the bundle deploy is UNRULED and is being put to the operator as its own
  GA-R5 exchange. Deploying on the current ULA carve means D-139 steps 2-6 later
  re-carve node addresses under a live cloud, which the entry warns is the expensive
  path since addresses bake into `agent.conf` at deploy time.
- **A durable unit for the 5241 tunnel** -- item 2's finding; unruled, logged only.

---

## Item 5 -- D-139 step 1 EXECUTED (apex GUA carve, vr1-dc0)

See `docs/audit/d139-step1-apex-push-dc0-20260801.txt` and the CURRENT-STATE block.
`--commit` exit 0, 13/13 read-back, apex 139 -> 152 verified by an INDEPENDENT query.
Nothing deleted: 9 ULA prefixes and all 26 ULA VIP legs still resolve.

**Revert.** Delete the 13 created prefixes (they are new and carry no dependents).

---

## Item 6 -- the VIP overlay re-rendered onto GUA, and the two apex readers taught FAMILY

**What.** `overlays/vr1-dc0-vips.yaml` now carries GUA on all 39 v6 VIP legs (was 13 GUA +
26 ULA). `render/values/vr1-dc0-vips.yaml` gains the two GUA prefixes, a refreshed
`apex_record`, and a new `v6_family: gua` provenance key. `scripts/render-dc-overlays.py`
gains `v6_family()` + `--v6-family`; `scripts/provider-bundle-check.py` gains per-leg,
per-application family resolution.

**Why it was needed.** Step 1 creates GUA ALONGSIDE the ULA rows and step 6 retires the ULA
rows, so for the WHOLE transition every v6-only plane has two `/64`s under one (role, kind)
key. Both apex readers keyed without a family dimension, so both could only REFUSE. Measured
live minutes after the push: `derive --dual-family` -> rc 2, and the gate -> "cannot evaluate
barbican's dual-family vip". The D-136 pipeline was unrunnable for exactly the migration it
exists to serve.

**A LOSSY PATH FOUND AND NOT TAKEN.** `derive` does not reproduce the per-app `comment:`
fields, so a full re-derive silently dropped 13 comments and shrank the overlay 4368 -> 3593
bytes. Reverted; the values file was edited surgically instead (3 lines), giving exactly 26
changed overlay lines -- 13 vip lines x 2 -- with every comment intact. **`derive` being lossy
against its own committed values file is a real gap, LOGGED NOT FIXED.**

**TWO BUGS I INTRODUCED, BOTH CAUGHT BY THE HARNESS AND NOT BY REVIEW.** (i) I inferred the
family ONCE from `parts[3]`, the provider leg -- but provider-public's VIP `/64` is GUA in
BOTH worlds (`:11::`), so every dc1 fixture asked for GUA bands on a still-ULA DC: 7 cases
red. (ii) The corrected version still resolved bands ONCE for the whole bundle and reused
them across all 13 applications, grading every later app against the first app's families.
Fixed by resolving per application AND per leg, cached on the family triple.

**>>> A PROPERTY WAS TRADED AWAY, AND IT IS NOT A WIN: THE GATE NO LONGER CATCHES A
WRONG-FAMILY LEG. <<<** Measured: substituting a ULA admin leg into the GUA dc0 overlay now
PASSES, because per-leg resolution grades it against the ULA band -- which still exists until
step 6. This is NECESSARY (dc1 is legitimately ULA, so no gate can hardcode a family) but it
means D-139 family conformance is now checked by NOTHING. That was already recorded as a gap
(`docs/audit/d139-conformance-dc0-20260801.txt` D.2); this change makes it the ONLY thing that
would catch a DC deployed on the wrong family. A ruled-table conformance gate is OWED.

**Verification.** `tests/render-dc-overlays` 18 -> **23/23** (T17/T17b/T18/T19/T20), three
mutations each killing named cases, script restored sha256-identical.
`tests/provider-bundle-check` **55/55**. Gauntlet **ALL GREEN (96)**; repo-lint 0 fail / 1
warn. Renderer `check` reproduces the new overlay byte-for-byte. The real deploy input gates
PASS with `13 clustered VIP(s) ... (13 dual-family)`, and a corrupted GUA leg still FAILS.

**vr1-dc1 is UNTOUCHED** -- its apex has no GUA, so its overlay stays ULA and now gates
correctly through the same per-leg path.

**Revert.** `git revert <commit>`; the overlay's prior bytes are recoverable from the values
file by restoring the three prefix lines and re-rendering.

---

## Item 7 -- D-139 steps 2 and 3 EXECUTED for vr1-dc0 (operator-gated)

Capture: `docs/audit/d139-steps2-3-dc0-20260801.txt`. Step 2 created the 5 GUA `/64`s on the
same vlan as their v4 twins (`applied=5`, 2 correct refusals for `lbaas-mgmt`/`oob`, which have
no v4 plane). Step 3 migrated the node statics **ULA -> GUA while v4 stayed in place**:
`applied=45 errors=0`, `READ-BACK 45/45`, and an independent query confirms **GUA 54 / ULA 0
with ZERO multi-global NICs**, so G19's predicate holds.

`dc-plane-ipam check` went 7 fail -> 2 fail, and both remaining are expected absences that
existed before this work (`lbaas-mgmt` is step 5; `oob` has no MAAS plane).

**Revert.** `dc-node-v6-carve.py replace vr1-dc0 --v6-family ula --commit` restores the node
statics; both families remain in MAAS until step 6. Nothing was deleted at any point.

---

## Item 8 -- BOTH gaps closed: a DC EGRESS gate, wired where it will actually run

**The gaps.** (1) No gate anywhere tested DC egress -- `dc-rack-net.sh check` PASSED through
a 19-hour outage because it asserts bridge legs and unit states, and **a leg is not a path**;
preflight had no egress gate at all; the mirror answered 200 from its own nginx the whole
time, and **a 200 from a LOCAL service says nothing about upstream**. (2) This session's own
post-reboot verification cited five true facts, none of which tested egress -- i.e. the check
set was a matter of my judgement, and my judgement was wrong.

**`scripts/dc-egress-check.sh check <site>` (NEW).** Site-keyed, runs ON the rack like
`dc-mirror.sh` / `dc-rack-net.sh`, read-only. **LAYERED, and that is the point:** A1 default
route -> A2 the edge ANSWERS -> A3 traffic leaves (probed by ICMP *and* TCP so an
ICMP-filtering path cannot fake a failure) -> A4 the three upstreams the deploy needs. The
first failure is reported as the cause and the rest SKIP, because a gate that restates one
fault as five trains people to skim it. A4 does not duplicate the mirror/proxy/snap checkers:
those assert the LOCAL service SERVES, this asserts the UPSTREAM it syncs FROM is reachable.
Both matter and they failed independently on 08-01.

**PROVEN LIVE ON TWO DIFFERENT FAILURE MODES, not fixture-green.** dc0: `A1 OK` ->
`A2 FAIL ... neigh: '10.12.4.1 dev virbr5 FAILED'`, A3/A4 skipped -- the outage, named at its
layer. dc1: `A1 OK, A2 OK (edge ANSWERS)` -> A3/A4 fail -- a *different* diagnosis, traffic
not leaving a healthy edge. That discrimination is the whole value.

**preflight gate P9.** WARNs off-rack with the exact command (preflight normally runs on
voffice1, which is never a rack -- "not the rack" is legitimate, same reasoning as P8), FAILs
when run on the rack with egress broken, and WARNs if the checker is missing: **a gate that
vanishes must not read as a green one.**

**WIRED WHERE IT BITES, which is what actually closes gap 2** -- the check set is no longer a
judgement call: `ops-restart-procedure.md` gains **Stage 0, before anything that fetches**,
with a first-FAIL-to-cause table; `dc-dc-phase4-juju-bundle-per-dc.md` gains **Step 3.9**
immediately before the deploy, stating that Step 3.5's `apt-mirror` covers apt ONLY while the
agent stream and snaps have no local copy, and that **P9 passing is not a substitute for
running it on the rack**.

**A MUTATION PASS CAUGHT A DECORATIVE TEST OF MY OWN.** T11 claimed to cover "unrecognised
HTTP code" but its fixture made A3's anchor odd, which A3 catches first -- so A4's `*)` branch
was never exercised and flipping it to `ok()` left the suite fully green. Added T14 with the
A3 anchor healthy and only the UCA upstream odd, so only A4 can fail it; the mutation now
kills T14. The P9 fixture also had to default to HEALTHY: modelling an unhealthy default drags
every case testing something else, which is the moment someone demotes the gate (mkfix's own
comment records exactly that happening when P7 landed).

**Verification.** `tests/dc-egress-check` **14/14** (3 mutations, restored sha256-identical);
`tests/preflight` 39 -> **43/43** (2 further mutations, each killing a named case); gauntlet
**ALL GREEN (97)**, manifest re-recorded 96 -> 97; repo-lint 0 fail / 1 warn.

**Revert.** `git revert <commit>`, then `bash scripts/run-tests-all.sh --record-manifest`.
