diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 06c35e5..c50c463 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2605,6 +2605,36 @@ Now an error; the assertion was REPLACED with that invariant and a further case added for the refusal path the original was written for. Harness 14 -> **23**, three mutations each killing named cases, script restored sha256-identical. + **>>> STEP 3.5 DONE, AND THEN A HARD BLOCKER: THE dc0 EDGE HAS BEEN DOWN SINCE THE + 2026-08-01 05:48 REBOOT. <<<** Capture `docs/audit/dc0-edge-egress-outage-20260802.txt`. + **STEP 3.5:** `juju add-model vr1-dc0` created with credential `vr1-dc0-cred`; + `juju-spaces-check.sh vr1-dc0` **PASS, 0 fatal, exit 0** (all six ruled spaces, none of the + five stale); `apt-mirror` already `http://10.12.8.4/ubuntu`, key VERIFIED against the live + client and the address read from the DC's own checker. The spaces script needs the repo + (`BASH_SOURCE`-relative `lib-net.sh`, so piping breaks it) -- run from the 07-30 staged copy + in `~/repo-stage/scripts/` on the rack, **all three files sha256-verified against the repo + before being trusted**. + **THE BLOCKER, MEASURED:** `dc-mirror.sh check dc0` FAILs with `last-sync FAILED ... + ubuntu=2 uca=2`; the sync journal reads `500 Can't connect to + ubuntu-cloud.archive.canonical.com`. The rack cannot reach `archive.ubuntu.com`, + `streams.canonical.com`, `api.snapcraft.io` through the snap proxy, `1.1.1.1`, **or its OWN + default gateway `10.12.4.1`** -- and `ip neigh dev virbr5` shows **`10.12.4.1 FAILED` while + `10.12.4.5` (juju controller) and `10.12.4.6` (MAAS region) on the same segment answer + normally.** The edge VM is RUNNING with both NICs attached; qga is not connected (the G13 + retrofit was office1-opnsense ONLY); its qemu log's last write is 2026-08-01 05:48. Last + successful mirror sync was 2026-08-01 00:04, BEFORE that reboot. **Step 4 cannot proceed: + apt would work off the mirror's Aug-1 content, but the deploy also needs the juju agent + stream and the snap proxy's upstream, which is the exact failure class the 2026-07-31 deploy + hit.** + **>>> TWO GAPS THIS EXPOSES, both worth more than the incident. <<<** **(1) NO GATE ANYWHERE + TESTS DC EGRESS** -- `dc-rack-net.sh check dc0` PASSES with egress dead because it asserts + bridge legs and unit states, not reachability, and preflight has no egress gate either; a + 19-hour outage on the deploy's critical path surfaced only because a nightly timer failed. + **(2) THIS SESSION'S OWN POST-REBOOT VERIFICATION LOOKED THOROUGH AND WAS NOT.** It reported + "everything reboot-persistent recovered unaided", citing rack units, region API 200, mirror + 200, snap proxy LISTENING, juju controller connected. Every one was true; none tested egress. + **A listener is not a path, and an HTTP 200 from a LOCAL service says nothing about + upstream.** The check set was incomplete and the summary outran its evidence. **STILL OWED BEFORE THE DEPLOY, in ruled order:** ~~D-139 step 1~~ DONE (apex CREATE-only push, `netbox/d139-gua-carve.py --dc vr1-dc0 --commit`; tool built, independently reviewed, dry-run byte-identical, `--commit` never yet passed), step 2 (MAAS GUA `/64`s alongside the diff --git a/docs/audit/dc0-edge-egress-outage-20260802.txt b/docs/audit/dc0-edge-egress-outage-20260802.txt new file mode 100644 index 0000000..8bfc46d --- /dev/null +++ b/docs/audit/dc0-edge-egress-outage-20260802.txt @@ -0,0 +1,58 @@ +dc0-edge-egress-outage-20260802.txt +==================================== +The vr1-dc0 OPNsense edge has been down since the 2026-08-01 05:48 containment-VM +reboot. Found 2026-08-02 while running Step 3.5, by `dc-mirror.sh check dc0`. +Read-only throughout. Nothing was restarted by this capture. + +HOW IT SURFACED -- a gate doing exactly what it was fixed to do + dc-mirror.sh check dc0 -> FAIL + "MISS last-sync FAILED: FAIL 2026-08-02T00:42:36Z ubuntu=2 uca=2 + -- the mirror content is NOT attested current" + This is the 2026-07-27 fix earning its keep: before it, `check` asserted that + last-sync.status EXISTED and printed it behind an unconditional OK, so a FAIL + status passed. Content-assertion caught a 19-hour outage. + +MEASURED + sync journal "rsync error: error in socket IO (code 10)" + "Download of dists/jammy-updates/caracal/Release failed: + 500 Can't connect to ubuntu-cloud.archive.canonical.com" + dc0-mirror-sync.service: Failed with result 'exit-code' + rack -> archive.ubuntu.com 000 + rack -> streams.canonical.com/juju/tools/ 000 + rack -> api.snapcraft.io VIA the snap proxy 000 + rack -> 1.1.1.1 100% packet loss + rack -> 10.12.4.1 (its OWN default gateway) 100% packet loss + ip neigh dev virbr5: 10.12.4.1 FAILED + 10.12.4.5 lladdr ... STALE (juju controller, fine) + 10.12.4.6 lladdr ... STALE (MAAS region, fine) + virsh: vr1-dc0-opnsense RUNNING, vnet0 on vr1-dc0-provider-public, + vnet1 on vr1-dc0-wan -- both attached + qga: "QEMU guest agent is not connected" (the G13 qga retrofit was + office1-opnsense ONLY; the DC edges never got it) + qemu log: last write 2026-08-01 05:48 -- the reboot + last mirror sync SUCCESS: 2026-08-01 00:04:19, i.e. BEFORE that reboot + +SO: the edge VM is running with its NICs attached and is not answering at L2 on +its LAN leg, while two neighbours on the same segment answer normally. + +WHY IT BLOCKS STAGE 5 STEP 4 + apt would be fine -- the mirror still serves jammy, jammy-updates, + jammy-backports and jammy-security at 200 from its Aug-1 content. + The deploy ALSO needs streams.canonical.com for juju agent binaries and the + snap proxy's upstream for snaps. Both are dead. That is exactly the failure + class the 2026-07-31 deploy hit (four artifact defects, each masked by the one + before it). + +>>> TWO GAPS THIS EXPOSES, both worth more than the incident <<< + 1. NO GATE ANYWHERE TESTS DC EGRESS. `dc-rack-net.sh check dc0` PASSES with + egress dead -- it asserts bridge legs and unit states, not reachability. + preflight has no egress gate either. A 19-hour outage on the deploy's + critical path surfaced only because a nightly timer happened to fail. + 2. A POST-REBOOT VERIFICATION THAT LOOKED THOROUGH AND WAS NOT. This session + reported after the 05:48 reboot that "everything reboot-persistent recovered + unaided", citing the rack units, the region API 200, the mirror answering + 200, the snap proxy LISTENING and the juju controller connected. Every one + of those was true. None of them tested egress, which was the thing broken. + A listener is not a path; an HTTP 200 from a LOCAL service says nothing + about upstream. The check set was incomplete and the summary was more + confident than its evidence.