queued-findings-20260802-stage5-edge-fold.txt
==============================================
Close sweep for the 2026-08-02 session (edge outage + rebuild, D-139 steps 1-3,
Step 3.5, the egress gate, and the opening of the runbook fold).
Method (ruled 2026-07-31): read back over the whole session, enumerate every
finding/decision/measurement/mistake, then GREP each candidate against repo
surfaces. A hit = ALREADY ON SURFACE, and where. No hit = FIRST SURFACE and would
have been lost on a context clear.

Session body: this session wrote no single changelog; its record is spread across
docs/CURRENT-STATE.md (status), docs/design-decisions.md (rulings), and the
per-topic captures cited below. Status claims live in CURRENT-STATE only.

--------------------------------------------------------------------------------------
FIRST SURFACE -- existed ONLY in the transcript. Listed first, by consequence.
--------------------------------------------------------------------------------------

F1. >>> THE dc0 RACK'S STAGED DEPLOY INPUT IS NOW STALE, AND IT IS THE VIP OVERLAY. <<<
    greps: "ed19d989" hits ONLY the 2026-08-01 sweep, which recorded it MATCHING.
    MEASURED at this close:
      rack ~/repo-stage/overlays/vr1-dc0-vips.yaml   ed19d989e80da8da
      repo               overlays/vr1-dc0-vips.yaml   80d861560a6b3c52   DIVERGED
      bundle.yaml 4c8a7852 and vr1-dc0-machines.yaml e3be85e4 still MATCH.
    WHY IT DIVERGED: this session re-rendered the VIP overlay onto GUA (26 of 39 v6
    legs moved from the retired ULA /64s to f02:20::/f02:21::). The rack's copy is
    the PRE-GUA version and nothing propagated the change.
    WHY IT MATTERS AND WHY IT IS THE TOP ITEM: under D-138 the rack IS the deploy
    client. `juju deploy` there would consume the STALE overlay and place 26 VIP
    legs on prefixes D-139 ruling B retires -- undoing, silently, the exact work
    this session did. The 2026-08-01 sweep recorded these digests MATCHING, so a
    reader trusting that note would conclude the staged copy is fine.
    LOGGED, NOT FIXED (hard rule 1 + this skill does not fix). The fix is one copy
    plus a re-verify, and it belongs to whoever runs Step 4.
    DURABLE LESSON: ~/repo-stage has NO git, so nothing detects its drift. Any
    session that edits a deploy artifact owes a re-stage or a recorded digest.

F2. THE MIRROR SYNC EXITS 1 WHILE ITS OWN LOG SAYS THE SYNC SUCCEEDED, AND THE
    CONTENT IS CURRENT.  grep "Everything OK": 0 hits.
    MEASURED after re-triggering the sync at 08:22 (egress restored):
      journal: "Files to download: 0 B" / "Downloaded 15 kiB in 3s" /
               "Everything OK. Moving meta files ..." / "Cleanup mirror." / "All done."
      then:    systemd "Main process exited, code=exited, status=1/FAILURE"
      status:  FAIL 2026-08-02T08:28:03Z ubuntu=1 uca=0     (UCA now SUCCEEDS: 0)
      content: ubuntu jammy Release mtime 2026-08-02 08:22:51
               UCA caracal Release mtime 2026-08-02 08:28:03   -- BOTH refreshed today
    So the mirror IS current and `dc-mirror.sh check dc0` will stay RED, because its
    success criterion is debmirror's EXIT CODE and debmirror reported "All done"
    while exiting 1. The 08-02 upstream failure is genuinely fixed (uca=0 proves
    egress); what remains is an exit-code disagreement.
    NOT DIAGNOSED FURTHER and NOT FIXED. Two readings remain open and only one is
    right: debmirror has a non-fatal non-zero path, OR something after "All done"
    fails silently. Do not assume the first.
    CONSEQUENCE FOR STAGE 5: a real gate is red for a reason that is probably not
    staleness. Do not wave it through and do not "fix" it by relaxing the check --
    that is the 2026-07-27 false-green defect in reverse.

F3. `systemctl show` RETURNS SUCCESS-LOOKING DEFAULTS FOR A UNIT THAT DOES NOT EXIST.
    grep "success-looking default": 0 hits.
    MEASURED: `systemctl show dc0-rack-net -p Type -p RemainAfterExit -p Result` on
    a unit name that does not exist returned `Type=`, `UnitFileState=`,
    `RemainAfterExit=no`, `Result=success`, `ExecMainStatus=0` and NO journal
    entries -- which reads exactly like a healthy oneshot that has completed. The
    real units are `dc0-rack-legs` and `dc0-node-dns` (from `dc-rack-net.sh:16,22`).
    This caused a false "inactive" reading that was reported before being caught.
    DURABLE RULE: `systemctl show` is not an existence check. Use `systemctl cat`
    (which errors on a missing unit) or check UnitFileState is non-empty. Same class
    as the repo's standing instrument-currency lesson.

F4. A HARNESS CASE THAT NEVER RAN, IN A SUITE THAT REPORTED ALL PASS.
    grep "fok": 0 hits.
    While extending tests/dc-egress-check, two new cases were written using `fok`/
    `fbad` -- helper names from a DIFFERENT harness (tests/dc-node-v6-carve). The
    suite printed `fbad: command not found` to stderr, kept its old count of 14, and
    still reported ALL PASS. Both mutations then "survived", which looked like the
    tool being robust and was actually the tests not existing.
    DURABLE RULE: after adding harness cases, assert the CASE COUNT moved. A suite
    that says ALL PASS with the old count has silently dropped what was added.

F5. WHEN TWO SCRIPTS PROBE THE SAME ENDPOINT THEY MUST SHARE THE PROBE DEFINITION.
    grep "share the probe definition": 0 hits.
    dc-egress-check.sh's A4 snap probe and dc-snap-proxy.sh:215 both hit the snap
    store; only the latter sent `Snap-Device-Series: 16`, which the store REQUIRES.
    Measured: 400 without it BOTH through the proxy and direct, 200 with it. The new
    gate therefore reported a healthy proxy as broken on its first live run, while
    the older script returned PASS with a real payload. A fixture that mocks curl
    cannot catch a wrong REAL request. (Raised by the dc0 rebuild agent; the header
    fix and its harness case are on surface, this GENERAL rule was not.)

F6. `pg_dump maasdb` -- THE ONE dc0 CORRUPTION TEST THAT WAS NOT COMPLETED.
    grep "pg_dump": 0 hits.
    The MAAS region VM was hard-cut on 2026-08-01 and holds the region database this
    session wrote ~95 changes to. Evidence AGAINST damage is good but not a proof:
    ext4 with no EXT4-fs errors, no orphan recovery, MAAS services active, ZERO
    checksum/corrupt/invalid-page entries in the postgres logs, and every one of
    those ~95 writes read back correctly. A full-table read was attempted FIVE times
    and blocked every time by snap confinement (`_daemon_` has HOME=/nonexistent;
    the cgroup is rejected; the raw binary needs the snap's library path).
    OWED: run `pg_dump maasdb > /dev/null` from the VM's own login shell, where the
    snap cgroup is valid. One command, definitive, and it should happen before the
    deploy rather than be assumed.

--------------------------------------------------------------------------------------
ALREADY ON SURFACE -- verified by grep, recorded here for completeness
--------------------------------------------------------------------------------------

A1. The edge outage, its root cause (UFS soft-update damage from the in-place
    resize bounce, NOT a partial update) and the corrected diagnosis --
    docs/audit/dc0-edge-egress-outage-20260802.txt + CURRENT-STATE.
A2. The dc0 edge rebuild, incl. the wrong base-image path in my agent brief --
    docs/audit/dc0-edge-rebuild-20260802.txt + CURRENT-STATE.
A3. The dc1 edge assessment: forwards-but-does-not-translate, config INTACT,
    repair-not-rebuild -- docs/audit/dc1-edge-assessment-20260802.txt + SEC-031.
A4. D-139 steps 1-3 executed -- docs/audit/d139-step1-apex-push-dc0-20260801.txt,
    docs/audit/d139-steps2-3-dc0-20260801.txt + CURRENT-STATE.
A5. The D-139 conformance audit and the VIP-overlay blocker --
    docs/audit/d139-conformance-dc0-20260801.txt.
A6. Both GA-R5 rulings of this date (D-135 amendment; D-140 PINNED) with exact
    utterances -- docs/design-decisions.md + CURRENT-STATE.
A7. The runbook fold register and both Class-A fixes -- docs/runbook-fold-register.md.
A8. `Snap-Device-Series: 16` as a store requirement -- scripts/dc-egress-check.sh
    comment + tests/dc-egress-check T15.
A9. SEC-031 (dc1 edge is an open router) and SEC-032 (dc0 edge API credential).
A10. The dnsmasq v6 `constructor:` latent risk -- CURRENT-STATE.

--------------------------------------------------------------------------------------
THE FIVE STRUCTURAL SWEEPS
--------------------------------------------------------------------------------------

S1. GITIGNORED STATE. `.claude/settings.local.json` was NOT modified this session --
    allow/ask/deny counts unchanged from the 2026-07-30 verbatim record, which
    therefore still stands as the recovery copy. No new permission rules were added;
    two classifier refusals were hit and NOT retried in altered shapes (the sqlite
    profile enumeration, and nothing else).
S2. DANGLING REFERENCES. Every docs/audit path cited by this session's commits
    resolves; the four new captures are committed.
S3. RULING FIDELITY. Six GA-R5 rulings this session, each with its exact utterance
    quoted, dated, committed and pushed BEFORE dependent work: D-139 ordering; OOB
    dual-stack; OOB v4 allocation (10.12.40.0/22 / 10.12.88.0/22, superseding
    10.12.60.0/22); VPN deferral; D-135 amendment; D-140 PINNED. None paraphrased.
S4. AS-EXECUTED LOG. `run-logged.sh` was NOT used -- it needs an interactive shell.
    All evidence is in the docs/audit captures and the commit messages. THIS WINDOW
    IS NOT COVERED BY AN AS-EXECUTED LOG and that is declared here rather than left
    to be discovered.
S5. CONTRADICTION DETECTOR.
    (a) preflight's P6 reminder still quotes "plan: 50 apps / 97 relations" against
        the 2026-07-31 MEASURED 56 apps / 108 relations. Pre-existing, still unfixed,
        and measurement wins (GA-R1 C2). DOCFIX owed.
    (b) `dc-rack-net.sh`'s DNS_UPSTREAM=10.10.0.20 still points node DNS at
        voffice1's BIND across the fiber while the 07-30 record says node DNS now
        uses the DC-local 10.12.8.6. Pre-existing; its gate passes because it checks
        units and legs, not the upstream's correctness.
    (c) D-135's "there is no fallback and no later convergence" is now QUALIFIED by
        the 2026-08-02 amendment -- recorded as a qualification in the amendment
        itself rather than left to contradict silently.
