queued-findings-20260802-stage5-edge-fold.txt ============================================== Close sweep for the 2026-08-02 session (edge outage + rebuild, D-139 steps 1-3, Step 3.5, the egress gate, and the opening of the runbook fold). Method (ruled 2026-07-31): read back over the whole session, enumerate every finding/decision/measurement/mistake, then GREP each candidate against repo surfaces. A hit = ALREADY ON SURFACE, and where. No hit = FIRST SURFACE and would have been lost on a context clear. Session body: this session wrote no single changelog; its record is spread across docs/CURRENT-STATE.md (status), docs/design-decisions.md (rulings), and the per-topic captures cited below. Status claims live in CURRENT-STATE only. -------------------------------------------------------------------------------------- FIRST SURFACE -- existed ONLY in the transcript. Listed first, by consequence. -------------------------------------------------------------------------------------- F1. >>> THE dc0 RACK'S STAGED DEPLOY INPUT IS NOW STALE, AND IT IS THE VIP OVERLAY. <<< greps: "ed19d989" hits ONLY the 2026-08-01 sweep, which recorded it MATCHING. MEASURED at this close: rack ~/repo-stage/overlays/vr1-dc0-vips.yaml ed19d989e80da8da repo overlays/vr1-dc0-vips.yaml 80d861560a6b3c52 DIVERGED bundle.yaml 4c8a7852 and vr1-dc0-machines.yaml e3be85e4 still MATCH. WHY IT DIVERGED: this session re-rendered the VIP overlay onto GUA (26 of 39 v6 legs moved from the retired ULA /64s to f02:20::/f02:21::). The rack's copy is the PRE-GUA version and nothing propagated the change. WHY IT MATTERS AND WHY IT IS THE TOP ITEM: under D-138 the rack IS the deploy client. `juju deploy` there would consume the STALE overlay and place 26 VIP legs on prefixes D-139 ruling B retires -- undoing, silently, the exact work this session did. The 2026-08-01 sweep recorded these digests MATCHING, so a reader trusting that note would conclude the staged copy is fine. LOGGED, NOT FIXED (hard rule 1 + this skill does not fix). The fix is one copy plus a re-verify, and it belongs to whoever runs Step 4. DURABLE LESSON: ~/repo-stage has NO git, so nothing detects its drift. Any session that edits a deploy artifact owes a re-stage or a recorded digest. F2. THE MIRROR SYNC EXITS 1 WHILE ITS OWN LOG SAYS THE SYNC SUCCEEDED, AND THE CONTENT IS CURRENT. grep "Everything OK": 0 hits. MEASURED after re-triggering the sync at 08:22 (egress restored): journal: "Files to download: 0 B" / "Downloaded 15 kiB in 3s" / "Everything OK. Moving meta files ..." / "Cleanup mirror." / "All done." then: systemd "Main process exited, code=exited, status=1/FAILURE" status: FAIL 2026-08-02T08:28:03Z ubuntu=1 uca=0 (UCA now SUCCEEDS: 0) content: ubuntu jammy Release mtime 2026-08-02 08:22:51 UCA caracal Release mtime 2026-08-02 08:28:03 -- BOTH refreshed today So the mirror IS current and `dc-mirror.sh check dc0` will stay RED, because its success criterion is debmirror's EXIT CODE and debmirror reported "All done" while exiting 1. The 08-02 upstream failure is genuinely fixed (uca=0 proves egress); what remains is an exit-code disagreement. NOT DIAGNOSED FURTHER and NOT FIXED. Two readings remain open and only one is right: debmirror has a non-fatal non-zero path, OR something after "All done" fails silently. Do not assume the first. CONSEQUENCE FOR STAGE 5: a real gate is red for a reason that is probably not staleness. Do not wave it through and do not "fix" it by relaxing the check -- that is the 2026-07-27 false-green defect in reverse. F3. `systemctl show` RETURNS SUCCESS-LOOKING DEFAULTS FOR A UNIT THAT DOES NOT EXIST. grep "success-looking default": 0 hits. MEASURED: `systemctl show dc0-rack-net -p Type -p RemainAfterExit -p Result` on a unit name that does not exist returned `Type=`, `UnitFileState=`, `RemainAfterExit=no`, `Result=success`, `ExecMainStatus=0` and NO journal entries -- which reads exactly like a healthy oneshot that has completed. The real units are `dc0-rack-legs` and `dc0-node-dns` (from `dc-rack-net.sh:16,22`). This caused a false "inactive" reading that was reported before being caught. DURABLE RULE: `systemctl show` is not an existence check. Use `systemctl cat` (which errors on a missing unit) or check UnitFileState is non-empty. Same class as the repo's standing instrument-currency lesson. F4. A HARNESS CASE THAT NEVER RAN, IN A SUITE THAT REPORTED ALL PASS. grep "fok": 0 hits. While extending tests/dc-egress-check, two new cases were written using `fok`/ `fbad` -- helper names from a DIFFERENT harness (tests/dc-node-v6-carve). The suite printed `fbad: command not found` to stderr, kept its old count of 14, and still reported ALL PASS. Both mutations then "survived", which looked like the tool being robust and was actually the tests not existing. DURABLE RULE: after adding harness cases, assert the CASE COUNT moved. A suite that says ALL PASS with the old count has silently dropped what was added. F5. WHEN TWO SCRIPTS PROBE THE SAME ENDPOINT THEY MUST SHARE THE PROBE DEFINITION. grep "share the probe definition": 0 hits. dc-egress-check.sh's A4 snap probe and dc-snap-proxy.sh:215 both hit the snap store; only the latter sent `Snap-Device-Series: 16`, which the store REQUIRES. Measured: 400 without it BOTH through the proxy and direct, 200 with it. The new gate therefore reported a healthy proxy as broken on its first live run, while the older script returned PASS with a real payload. A fixture that mocks curl cannot catch a wrong REAL request. (Raised by the dc0 rebuild agent; the header fix and its harness case are on surface, this GENERAL rule was not.) F6. `pg_dump maasdb` -- THE ONE dc0 CORRUPTION TEST THAT WAS NOT COMPLETED. grep "pg_dump": 0 hits. The MAAS region VM was hard-cut on 2026-08-01 and holds the region database this session wrote ~95 changes to. Evidence AGAINST damage is good but not a proof: ext4 with no EXT4-fs errors, no orphan recovery, MAAS services active, ZERO checksum/corrupt/invalid-page entries in the postgres logs, and every one of those ~95 writes read back correctly. A full-table read was attempted FIVE times and blocked every time by snap confinement (`_daemon_` has HOME=/nonexistent; the cgroup is rejected; the raw binary needs the snap's library path). OWED: run `pg_dump maasdb > /dev/null` from the VM's own login shell, where the snap cgroup is valid. One command, definitive, and it should happen before the deploy rather than be assumed. -------------------------------------------------------------------------------------- ALREADY ON SURFACE -- verified by grep, recorded here for completeness -------------------------------------------------------------------------------------- A1. The edge outage, its root cause (UFS soft-update damage from the in-place resize bounce, NOT a partial update) and the corrected diagnosis -- docs/audit/dc0-edge-egress-outage-20260802.txt + CURRENT-STATE. A2. The dc0 edge rebuild, incl. the wrong base-image path in my agent brief -- docs/audit/dc0-edge-rebuild-20260802.txt + CURRENT-STATE. A3. The dc1 edge assessment: forwards-but-does-not-translate, config INTACT, repair-not-rebuild -- docs/audit/dc1-edge-assessment-20260802.txt + SEC-031. A4. D-139 steps 1-3 executed -- docs/audit/d139-step1-apex-push-dc0-20260801.txt, docs/audit/d139-steps2-3-dc0-20260801.txt + CURRENT-STATE. A5. The D-139 conformance audit and the VIP-overlay blocker -- docs/audit/d139-conformance-dc0-20260801.txt. A6. Both GA-R5 rulings of this date (D-135 amendment; D-140 PINNED) with exact utterances -- docs/design-decisions.md + CURRENT-STATE. A7. The runbook fold register and both Class-A fixes -- docs/runbook-fold-register.md. A8. `Snap-Device-Series: 16` as a store requirement -- scripts/dc-egress-check.sh comment + tests/dc-egress-check T15. A9. SEC-031 (dc1 edge is an open router) and SEC-032 (dc0 edge API credential). A10. The dnsmasq v6 `constructor:` latent risk -- CURRENT-STATE. -------------------------------------------------------------------------------------- THE FIVE STRUCTURAL SWEEPS -------------------------------------------------------------------------------------- S1. GITIGNORED STATE. `.claude/settings.local.json` was NOT modified this session -- allow/ask/deny counts unchanged from the 2026-07-30 verbatim record, which therefore still stands as the recovery copy. No new permission rules were added; two classifier refusals were hit and NOT retried in altered shapes (the sqlite profile enumeration, and nothing else). S2. DANGLING REFERENCES. Every docs/audit path cited by this session's commits resolves; the four new captures are committed. S3. RULING FIDELITY. Six GA-R5 rulings this session, each with its exact utterance quoted, dated, committed and pushed BEFORE dependent work: D-139 ordering; OOB dual-stack; OOB v4 allocation (10.12.40.0/22 / 10.12.88.0/22, superseding 10.12.60.0/22); VPN deferral; D-135 amendment; D-140 PINNED. None paraphrased. S4. AS-EXECUTED LOG. `run-logged.sh` was NOT used -- it needs an interactive shell. All evidence is in the docs/audit captures and the commit messages. THIS WINDOW IS NOT COVERED BY AN AS-EXECUTED LOG and that is declared here rather than left to be discovered. S5. CONTRADICTION DETECTOR. (a) preflight's P6 reminder still quotes "plan: 50 apps / 97 relations" against the 2026-07-31 MEASURED 56 apps / 108 relations. Pre-existing, still unfixed, and measurement wins (GA-R1 C2). DOCFIX owed. (b) `dc-rack-net.sh`'s DNS_UPSTREAM=10.10.0.20 still points node DNS at voffice1's BIND across the fiber while the 07-30 record says node DNS now uses the DC-local 10.12.8.6. Pre-existing; its gate passes because it checks units and legs, not the upstream's correctness. (c) D-135's "there is no fallback and no later convergence" is now QUALIFIED by the 2026-08-02 amendment -- recorded as a qualification in the amendment itself rather than left to contradict silently.