diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index e8f3612..6394765 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2641,6 +2641,36 @@ there is no copy-on-write base to roll back to. **SEPARATE THREAD, LOGGED NOT CONFLATED: dc1's rack also cannot reach `1.1.1.1` despite a healthy edge on both legs and a default route via it** -- an edge forwarding/NAT question at dc1, off the deploy path, NOT diagnosed here. + **>>> CORRECTION 2026-08-02: THE "PARTIALLY-APPLIED UPDATE" DIAGNOSIS ABOVE IS WRONG. + THE CAUSE IS UFS DAMAGE FROM AN UNCLEAN POWER CUT, AND IT HIT BOTH DCs. <<<** That earlier + reading was an INFERENCE FROM THE SHAPE of the symptom (libraries missing, so something must + have replaced them) and it is **REFUTED BY THE LOG: zero `pkg-static` / `opnsense-update` / + firmware lines in the entire 477 KB serial history.** What actually precedes the first + missing-library error is an **fsck salvage of a badly damaged UFS root** -- `UNREF FILE` + x**2533**, `UNEXPECTED SOFT UPDATE INCONSISTENCY` x**785**, `SALVAGE? yes` x**257**, ending + `FILE SYSTEM MARKED CLEAN` / `FILE SYSTEM WAS MODIFIED`, then immediately `ld-elf.so.1: + Shared object "libcrypto.so.17" not found`. **The libraries were fsck casualties.** + **CAUSAL CHAIN:** the 2026-08-01 memory resize was an **IN-PLACE tofu update, which this + document already records BOUNCES the guest** (confirmed that day on dc1 as the canary, domain + id 9 -> 12, uptime 0). **A bounce of a CONTAINMENT VM is a hard power cut to every INNER VM** + -- nothing shuts them down first. Under **D-127 the DC edges are the ONLY inner VMs with + `autostart=true`** ("comes up with its containment VM"); the nine role nodes were powered off + and the region/juju VMs are `autostart=false` and were started BY HAND afterwards, cleanly. + So the edges were the only inner guests hard-cut and auto-restarted onto damaged filesystems. + **CONFIRMED BY THE dc1 CONTROL -- same event, same class, different severity:** dc0 `2533` + UNREF / `785` inconsistencies / **ld-elf errors present**; dc1 `76` / `181` / **zero ld-elf + errors**. dc0 lost its libraries and cannot reach multi-user; dc1 kept its libraries and boots + to a login prompt but **its damage is real too -- which is the likely explanation for the + "healthy edge that does not forward" logged earlier as a SEPARATE thread. IT IS NOT SEPARATE: + same incident, lesser damage**, and that earlier note is superseded by this one. + **WHY A "RESTART" DID THIS, plainly: it was not a restart of the edge -- it was a POWER CUT.** + The edge was never asked to shut down; its parent VM was replaced under it while it held + unflushed metadata. **THE DURABLE FINDING IS A PROCEDURE GAP:** the resize was executed with a + plan asserted on CONTENT (1 in-place, 0 create/destroy/replace, 0 MAC changes) and a capacity + gate -- all correct, and **none of it covered the INNER guests. "In-place" describes the tofu + RESOURCE, not the BLAST RADIUS**: for a containment VM it means every VM inside it is hard-cut. + **Any future in-place change to a containment VM must SHUT DOWN its inner guests first, or + accept filesystem damage to every running one.** **>>> BOTH GAPS ARE NOW CLOSED, 2026-08-02: `scripts/dc-egress-check.sh` + preflight gate P9, wired into the restart procedure and the phase-4 runbook. <<<** The check is LAYERED -- A1 default route -> A2 **the edge ANSWERS** -> A3 traffic leaves (ICMP *and* TCP, so an diff --git a/docs/audit/dc0-edge-egress-outage-20260802.txt b/docs/audit/dc0-edge-egress-outage-20260802.txt index 14da616..fa9e5ee 100644 --- a/docs/audit/dc0-edge-egress-outage-20260802.txt +++ b/docs/audit/dc0-edge-egress-outage-20260802.txt @@ -101,3 +101,67 @@ even though its edge is healthy on BOTH legs and it holds a default route via it. That is an edge forwarding/NAT question at dc1, not the dc0 outage, and dc1 is not on the deploy path. Logged so it is not lost; NOT diagnosed here. + +================================================================================ +>>> CORRECTION: THE "PARTIALLY-APPLIED UPDATE" DIAGNOSIS ABOVE IS WRONG. <<< + The real cause is UFS damage from an unclean power cut. Measured, not reasoned. +================================================================================ +The section above concluded "a PARTIALLY-APPLIED OPNsense UPDATE". That was an +inference from the SHAPE of the symptom (missing libs => something replaced them) +and it is REFUTED by the log: + + update / pkg-static / opnsense-update / firmware activity in the whole log: 0 lines + +What the log actually shows, immediately BEFORE the first missing-library error: + + UNREF FILE I=1348940 OWNER=root MODE=100644 MTIME=Jul 6 19:48 2026 + RECONNECT? yes ... x2533 + UNEXPECTED SOFT UPDATE INCONSISTENCY ... x785 + SALVAGE? yes ... x257 + ** Phase 5 - Check Cyl groups + FREE BLK COUNT(S) WRONG IN SUPERBLK / BLK(S) MISSING IN BIT MAPS / SUMMARY INFORMATION BAD + ***** FILE SYSTEM MARKED CLEAN ***** + ***** FILE SYSTEM WAS MODIFIED ***** + ld-elf.so.1: Shared object "libcrypto.so.17" not found, required by "php" + +fsck salvaged a badly damaged UFS root and the shared libraries did not survive it. +`libcrypto.so.17` and `libpython3.13.so.1.0` were fsck casualties. + +THE CAUSAL CHAIN, end to end + 1. The 2026-08-01 memory resize (416 -> 480 GiB) was an IN-PLACE tofu update, and + the repo already records that an in-place update BOUNCES the guest -- confirmed + on dc1 as the canary that day (domain id 9 -> 12, uptime 0 min). + 2. A bounce of the CONTAINMENT VM is a hard power cut to every INNER VM. Nothing + shuts the inner guests down first. + 3. Under D-127 the DC edges are the only inner VMs with autostart=true ("comes up + with its containment VM"). The nine role nodes were powered off; the MAAS region + and juju controller VMs are autostart=false and were started BY HAND afterwards, + cleanly. So the edges were the ONLY inner guests hard-cut and auto-restarted. + 4. Both edges restarted onto damaged UFS. fsck salvaged both. + +CONFIRMED BY THE dc1 CONTROL -- same event, same class, different severity: + dc0 edge dc1 edge + UNREF FILE 2533 76 + SOFT UPDATE incons. 785 181 + SALVAGE? yes 257 46 + FILE SYSTEM MODIFIED 1 1 + ld-elf errors present 0 + qemu restart 05:48:31 05:45:30 + + dc0 lost its shared libraries -> cannot reach multi-user -> no LAN service at all. + dc1 kept its libraries -> boots to a login prompt -> BUT its damage is real too, + which is the likely explanation for the "healthy edge that does not forward" + logged earlier as a SEPARATE thread. IT IS NOT SEPARATE: same incident, lesser + damage. That earlier note should be read as superseded by this one. + +WHY A "RESTART" DID THIS, stated plainly: it was not a restart of the edge. It was a +POWER CUT. The edge was never asked to shut down; its parent VM was replaced under it +while it had unflushed metadata. + +THE PROCEDURE GAP, which is the durable finding: + The resize was executed with a plan asserted on CONTENT (1 in-place, 0 create/ + destroy/replace, 0 MAC changes) and a capacity gate -- all correct, and none of it + covered the INNER guests. "In-place" describes the tofu resource, not the blast + radius: for a containment VM it means every VM inside it is hard-cut. Any future + in-place change to a containment VM must SHUT DOWN its inner guests first, or accept + filesystem damage to every one of them that is running.