diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index c50c463..f1736ed 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2626,6 +2626,21 @@ apt would work off the mirror's Aug-1 content, but the deploy also needs the juju agent stream and the snap proxy's upstream, which is the exact failure class the 2026-07-31 deploy hit.** + **ROOT CAUSE, appended after reading the console: THE dc0 EDGE IS AT A SINGLE-USER PROMPT + WITH MISSING SHARED LIBRARIES** -- `libcrypto.so.17` and `libpython3.13.so.1.0` not found, + `10-configd` / `15-templates` / `90-carp` all failing, ending at `Enter full pathname of + shell`. **That is a PARTIALLY-APPLIED OPNsense UPDATE: the reboot did not break the edge, it + EXPOSED it** -- the running system had survived on already-mapped libraries. **The symptom + appears NOWHERE in appendix-A; it is a new class.** **SCOPE MEASURED, NOT ASSUMED -- dc0 + ONLY:** dc1's edge serial log has ZERO matches for those libraries and booted clean to a login + prompt with `WAN (vtnet1) -> v4: 172.30.3.2/24`. **The upstream chain is HEALTHY** -- vcloud + and voffice1 both reach `1.1.1.1` and `archive.ubuntu.com` 200, both uplink nets are ACTIVE + with `forward mode='nat'`, and both racks hold a default route via their own edge. + **RECOVERY MATERIAL EXISTS: `opnsense-26.7-nano.qcow2` sits in the same pool** as the broken + disk -- but that disk has an EMPTY ``, so it is standalone, not an overlay, and + there is no copy-on-write base to roll back to. **SEPARATE THREAD, LOGGED NOT CONFLATED: dc1's + rack also cannot reach `1.1.1.1` despite a healthy edge on both legs and a default route via + it** -- an edge forwarding/NAT question at dc1, off the deploy path, NOT diagnosed here. **>>> TWO GAPS THIS EXPOSES, both worth more than the incident. <<<** **(1) NO GATE ANYWHERE TESTS DC EGRESS** -- `dc-rack-net.sh check dc0` PASSES with egress dead because it asserts bridge legs and unit states, not reachability, and preflight has no egress gate either; a diff --git a/docs/audit/dc0-edge-egress-outage-20260802.txt b/docs/audit/dc0-edge-egress-outage-20260802.txt index 8bfc46d..14da616 100644 --- a/docs/audit/dc0-edge-egress-outage-20260802.txt +++ b/docs/audit/dc0-edge-egress-outage-20260802.txt @@ -56,3 +56,48 @@ A listener is not a path; an HTTP 200 from a LOCAL service says nothing about upstream. The check set was incomplete and the summary was more confident than its evidence. + +================================================================================ +ROOT CAUSE FOUND -- AND IT IS NOT THE REBOOT (appended after the console read) +================================================================================ +The dc0 edge is sitting at a SINGLE-USER PROMPT with missing shared libraries: + + ld-elf.so.1: Shared object "libcrypto.so.17" not found, required by "php" + ld-elf.so.1: Shared object "libpython3.13.so.1.0" not found, required by "python3" + >>> Error in early script '10-configd' + >>> Error in early script '15-templates' + >>> Error in early script '90-carp' + Launching the init system...ld-elf.so.1: ... libcrypto.so.17 not found + Enter full pathname of shell or RETURN for /bin/sh: + +That is a PARTIALLY-APPLIED OPNsense UPDATE. The reboot did not break the edge -- +it EXPOSED it. The running system had survived on already-mapped libraries; the +2026-08-01 05:49 boot is where it fell over. Serial log mtime 05:49:39 confirms. +This symptom appears NOWHERE in appendix-A or the docs -- it is a new class. + +SCOPE, MEASURED RATHER THAN ASSUMED -- IT IS dc0-ONLY + dc1 edge serial log: ZERO matches for libcrypto.so.17 / libpython3.13; it booted + clean to a login prompt with "WAN (vtnet1) -> v4: 172.30.3.2/24". + dc1 rack -> its edge 10.12.64.1 : 0% loss. vcloud -> 172.30.3.2 : 0% loss. + vcloud -> 172.30.2.2 (dc0 edge WAN) : 100% loss -- consistent with no config loaded. + +THE UPSTREAM CHAIN IS HEALTHY -- the break is at the dc0 edge, nowhere else + vcloud -> 1.1.1.1 0% loss, archive.ubuntu.com 200 + voffice1 -> 1.1.1.1 0% loss, archive.ubuntu.com 200, office1 edge 10.10.0.1 0% loss + both uplink nets ACTIVE, forward mode='nat', gateways 172.30.2.1 / 172.30.3.1 + both pingable from vcloud + both racks hold a default route via their own edge + +RECOVERY MATERIAL AVAILABLE + The pristine base EXISTS in the same pool as the broken disk: + vr1-dc0-inner-pool -> opnsense-26.7-nano.qcow2 + vr1-dc0-opnsense-disk.qcow2 (the broken one) + The broken disk has an EMPTY -- it is a standalone qcow2, NOT an + overlay, so there is no copy-on-write base to roll back to. A rebuild would mean + a fresh disk from the nano base plus the D-112(c) console bootstrap and D-113 + API-key mint, then re-addressing (WAN 172.30.2.2/24, LAN 10.12.4.1/22). + +SEPARATE THREAD, NOT CONFLATED WITH THIS ONE: dc1's rack also cannot reach 1.1.1.1 +even though its edge is healthy on BOTH legs and it holds a default route via it. +That is an edge forwarding/NAT question at dc1, not the dc0 outage, and dc1 is not +on the deploy path. Logged so it is not lost; NOT diagnosed here.