queued-findings-20260802-stage5-edge-fold.txt
==============================================
Close sweep for the 2026-08-02 session (edge outage + rebuild, D-139 steps 1-3,
Step 3.5, the egress gate, and the opening of the runbook fold).
Method (ruled 2026-07-31): read back over the whole session, enumerate every
finding/decision/measurement/mistake, then GREP each candidate against repo
surfaces. A hit = ALREADY ON SURFACE, and where. No hit = FIRST SURFACE and would
have been lost on a context clear.
Session body: this session wrote no single changelog; its record is spread across
docs/CURRENT-STATE.md (status), docs/design-decisions.md (rulings), and the
per-topic captures cited below. Status claims live in CURRENT-STATE only.
--------------------------------------------------------------------------------------
FIRST SURFACE -- existed ONLY in the transcript. Listed first, by consequence.
--------------------------------------------------------------------------------------
F1. >>> THE dc0 RACK'S STAGED DEPLOY INPUT IS NOW STALE, AND IT IS THE VIP OVERLAY. <<<
greps: "ed19d989" hits ONLY the 2026-08-01 sweep, which recorded it MATCHING.
MEASURED at this close:
rack ~/repo-stage/overlays/vr1-dc0-vips.yaml ed19d989e80da8da
repo overlays/vr1-dc0-vips.yaml 80d861560a6b3c52 DIVERGED
bundle.yaml 4c8a7852 and vr1-dc0-machines.yaml e3be85e4 still MATCH.
WHY IT DIVERGED: this session re-rendered the VIP overlay onto GUA (26 of 39 v6
legs moved from the retired ULA /64s to f02:20::/f02:21::). The rack's copy is
the PRE-GUA version and nothing propagated the change.
WHY IT MATTERS AND WHY IT IS THE TOP ITEM: under D-138 the rack IS the deploy
client. `juju deploy` there would consume the STALE overlay and place 26 VIP
legs on prefixes D-139 ruling B retires -- undoing, silently, the exact work
this session did. The 2026-08-01 sweep recorded these digests MATCHING, so a
reader trusting that note would conclude the staged copy is fine.
LOGGED, NOT FIXED (hard rule 1 + this skill does not fix). The fix is one copy
plus a re-verify, and it belongs to whoever runs Step 4.
DURABLE LESSON: ~/repo-stage has NO git, so nothing detects its drift. Any
session that edits a deploy artifact owes a re-stage or a recorded digest.
F2. THE MIRROR SYNC EXITS 1 WHILE ITS OWN LOG SAYS THE SYNC SUCCEEDED, AND THE
CONTENT IS CURRENT. grep "Everything OK": 0 hits.
MEASURED after re-triggering the sync at 08:22 (egress restored):
journal: "Files to download: 0 B" / "Downloaded 15 kiB in 3s" /
"Everything OK. Moving meta files ..." / "Cleanup mirror." / "All done."
then: systemd "Main process exited, code=exited, status=1/FAILURE"
status: FAIL 2026-08-02T08:28:03Z ubuntu=1 uca=0 (UCA now SUCCEEDS: 0)
content: ubuntu jammy Release mtime 2026-08-02 08:22:51
UCA caracal Release mtime 2026-08-02 08:28:03 -- BOTH refreshed today
So the mirror IS current and `dc-mirror.sh check dc0` will stay RED, because its
success criterion is debmirror's EXIT CODE and debmirror reported "All done"
while exiting 1. The 08-02 upstream failure is genuinely fixed (uca=0 proves
egress); what remains is an exit-code disagreement.
NOT DIAGNOSED FURTHER and NOT FIXED. Two readings remain open and only one is
right: debmirror has a non-fatal non-zero path, OR something after "All done"
fails silently. Do not assume the first.
CONSEQUENCE FOR STAGE 5: a real gate is red for a reason that is probably not
staleness. Do not wave it through and do not "fix" it by relaxing the check --
that is the 2026-07-27 false-green defect in reverse.
F3. `systemctl show` RETURNS SUCCESS-LOOKING DEFAULTS FOR A UNIT THAT DOES NOT EXIST.
grep "success-looking default": 0 hits.
MEASURED: `systemctl show dc0-rack-net -p Type -p RemainAfterExit -p Result` on
a unit name that does not exist returned `Type=`, `UnitFileState=`,
`RemainAfterExit=no`, `Result=success`, `ExecMainStatus=0` and NO journal
entries -- which reads exactly like a healthy oneshot that has completed. The
real units are `dc0-rack-legs` and `dc0-node-dns` (from `dc-rack-net.sh:16,22`).
This caused a false "inactive" reading that was reported before being caught.
DURABLE RULE: `systemctl show` is not an existence check. Use `systemctl cat`
(which errors on a missing unit) or check UnitFileState is non-empty. Same class
as the repo's standing instrument-currency lesson.
F4. A HARNESS CASE THAT NEVER RAN, IN A SUITE THAT REPORTED ALL PASS.
grep "fok": 0 hits.
While extending tests/dc-egress-check, two new cases were written using `fok`/
`fbad` -- helper names from a DIFFERENT harness (tests/dc-node-v6-carve). The
suite printed `fbad: command not found` to stderr, kept its old count of 14, and
still reported ALL PASS. Both mutations then "survived", which looked like the
tool being robust and was actually the tests not existing.
DURABLE RULE: after adding harness cases, assert the CASE COUNT moved. A suite
that says ALL PASS with the old count has silently dropped what was added.
F5. WHEN TWO SCRIPTS PROBE THE SAME ENDPOINT THEY MUST SHARE THE PROBE DEFINITION.
grep "share the probe definition": 0 hits.
dc-egress-check.sh's A4 snap probe and dc-snap-proxy.sh:215 both hit the snap
store; only the latter sent `Snap-Device-Series: 16`, which the store REQUIRES.
Measured: 400 without it BOTH through the proxy and direct, 200 with it. The new
gate therefore reported a healthy proxy as broken on its first live run, while
the older script returned PASS with a real payload. A fixture that mocks curl
cannot catch a wrong REAL request. (Raised by the dc0 rebuild agent; the header
fix and its harness case are on surface, this GENERAL rule was not.)
F6. `pg_dump maasdb` -- THE ONE dc0 CORRUPTION TEST THAT WAS NOT COMPLETED.
grep "pg_dump": 0 hits.
The MAAS region VM was hard-cut on 2026-08-01 and holds the region database this
session wrote ~95 changes to. Evidence AGAINST damage is good but not a proof:
ext4 with no EXT4-fs errors, no orphan recovery, MAAS services active, ZERO
checksum/corrupt/invalid-page entries in the postgres logs, and every one of
those ~95 writes read back correctly. A full-table read was attempted FIVE times
and blocked every time by snap confinement (`_daemon_` has HOME=/nonexistent;
the cgroup is rejected; the raw binary needs the snap's library path).
OWED: run `pg_dump maasdb > /dev/null` from the VM's own login shell, where the
snap cgroup is valid. One command, definitive, and it should happen before the
deploy rather than be assumed.
--------------------------------------------------------------------------------------
ALREADY ON SURFACE -- verified by grep, recorded here for completeness
--------------------------------------------------------------------------------------
A1. The edge outage, its root cause (UFS soft-update damage from the in-place
resize bounce, NOT a partial update) and the corrected diagnosis --
docs/audit/dc0-edge-egress-outage-20260802.txt + CURRENT-STATE.
A2. The dc0 edge rebuild, incl. the wrong base-image path in my agent brief --
docs/audit/dc0-edge-rebuild-20260802.txt + CURRENT-STATE.
A3. The dc1 edge assessment: forwards-but-does-not-translate, config INTACT,
repair-not-rebuild -- docs/audit/dc1-edge-assessment-20260802.txt + SEC-031.
A4. D-139 steps 1-3 executed -- docs/audit/d139-step1-apex-push-dc0-20260801.txt,
docs/audit/d139-steps2-3-dc0-20260801.txt + CURRENT-STATE.
A5. The D-139 conformance audit and the VIP-overlay blocker --
docs/audit/d139-conformance-dc0-20260801.txt.
A6. Both GA-R5 rulings of this date (D-135 amendment; D-140 PINNED) with exact
utterances -- docs/design-decisions.md + CURRENT-STATE.
A7. The runbook fold register and both Class-A fixes -- docs/runbook-fold-register.md.
A8. `Snap-Device-Series: 16` as a store requirement -- scripts/dc-egress-check.sh
comment + tests/dc-egress-check T15.
A9. SEC-031 (dc1 edge is an open router) and SEC-032 (dc0 edge API credential).
A10. The dnsmasq v6 `constructor:` latent risk -- CURRENT-STATE.
--------------------------------------------------------------------------------------
THE FIVE STRUCTURAL SWEEPS
--------------------------------------------------------------------------------------
S1. GITIGNORED STATE. `.claude/settings.local.json` was NOT modified this session --
allow/ask/deny counts unchanged from the 2026-07-30 verbatim record, which
therefore still stands as the recovery copy. No new permission rules were added;
two classifier refusals were hit and NOT retried in altered shapes (the sqlite
profile enumeration, and nothing else).
S2. DANGLING REFERENCES. Every docs/audit path cited by this session's commits
resolves; the four new captures are committed.
S3. RULING FIDELITY. Six GA-R5 rulings this session, each with its exact utterance
quoted, dated, committed and pushed BEFORE dependent work: D-139 ordering; OOB
dual-stack; OOB v4 allocation (10.12.40.0/22 / 10.12.88.0/22, superseding
10.12.60.0/22); VPN deferral; D-135 amendment; D-140 PINNED. None paraphrased.
S4. AS-EXECUTED LOG. `run-logged.sh` was NOT used -- it needs an interactive shell.
All evidence is in the docs/audit captures and the commit messages. THIS WINDOW
IS NOT COVERED BY AN AS-EXECUTED LOG and that is declared here rather than left
to be discovered.
S5. CONTRADICTION DETECTOR.
(a) preflight's P6 reminder still quotes "plan: 50 apps / 97 relations" against
the 2026-07-31 MEASURED 56 apps / 108 relations. Pre-existing, still unfixed,
and measurement wins (GA-R1 C2). DOCFIX owed.
(b) `dc-rack-net.sh`'s DNS_UPSTREAM=10.10.0.20 still points node DNS at
voffice1's BIND across the fiber while the 07-30 record says node DNS now
uses the DC-local 10.12.8.6. Pre-existing; its gate passes because it checks
units and legs, not the upstream's correctness.
(c) D-135's "there is no fallback and no later convergence" is now QUALIFIED by
the 2026-08-02 amendment -- recorded as a qualification in the amendment
itself rather than left to contradict silently.
