dc1-edge-assessment-20260802.txt ================================================================================ READ-ONLY assessment of the vr1-dc1 OPNsense edge after the 2026-08-01 05:45:30 containment-VM power cut. NOTHING was mutated: no virsh start/destroy/define, no config write, no service restart, no tofu. Every API call issued was a GET. Companion to docs/audit/dc0-edge-egress-outage-20260802.txt (the dc0 half). Every line below is tagged MEASURED (observed this session, or quoted from a captured artifact) or INFERRED (reasoning over measurements). Per this repo's discipline a finding is an observation, not a conclusion, until it is measured. ================================================================================ VERDICT ================================================================================ Q1 Does the dc1 edge forward/NAT? It FORWARDS. It does NOT NAT. And it is enforcing NO EFFECTIVE FIREWALL RULESET -- measured: no outbound NAT on the wire, and no WAN block, in a controlled comparison against a healthy sibling edge. (Whether pf is disabled or loaded-but-permissive is NOT separable read-only; `pfctl -s info` in section 8 settles it, and nothing in the recommendation turns on which it is.) Not a D-125 isolation posture -- D-125 is the decision that RESTORES per-DC egress, and dc1 rack egress was MEASURED WORKING on 2026-07-30. This is a regression, not a design. Q2 Is the config INTACT? YES, on positive evidence rather than on "it booted." fsck names every inode it deletes; all 50 named casualties are listed in section 5a and /conf/config.xml IS NOT AMONG THEM -- the whole /conf tree lost exactly one transient event file. The edge came up on its exact ruled as-built addresses (LAN 10.12.64.1/22, WAN 172.30.3.2/24) with routes, Unbound, Dnsmasq, NTP and the gateway monitor all "done." What fsck destroyed is the FreeBSD BASE SYSTEM user database -- /etc/master.passwd and /etc/group -- which is NOT part of config.xml, plus BOTH of their /var/backups/*.bak copies. Everything broken is downstream of that. QUALIFIER: strong but not absolute -- fsck's other 92 casualties are reported by inode, not by name (see 5a(i)). One `xmllint --noout /conf/config.xml` settles it and needs a shell (section 8). Q3 Rebuild, repair, or nothing? >>> REPAIR, not rebuild. <<< dc0 lost its shared libraries and cannot reach multi-user; dc1 lost its passwd/group database and reaches multi-user with its whole configuration loaded. Different severity AND a different repair. See "RECOMMENDATION" for the caveat that decides how the repair has to be delivered. Q4 Other damage in dc1? NONE FOUND. The edge was the ONLY inner guest running at the cut (the other ten dc1 domains are shut off with autostart=disable). The rack host vvr1-dc1 (ext4) is clean: no ext4/IO errors, zero failed units, all four dc1 units active. ================================================================================ 1. THE BASELINE THAT FRAMES EVERYTHING -- this was working two days ago ================================================================================ MEASURED (quoted from docs/CURRENT-STATE.md:115-121, a captured 2026-07-30 measurement, not prose): "ARTIFACT SOURCES AND EGRESS RE-MEASURED 2026-07-30, both DCs. ... dc1 proxy PASS (apt-cacher-ng active on 10.12.68.4:3142, serves archive AND UCA 200). Egress re-probed from BOTH rack hosts with `--noproxy '*'` so a cache hit could not fake it: juju agent stream `streams.canonical.com/juju/tools/` 200, snap store `api.snapcraft.io` answering, `archive.ubuntu.com` jammy Release 200, 1.1.1.1 0% loss, default routes via 10.12.4.1 / 10.12.64.1." This closes three of the four alternatives in the task's hypothesis list before any live probe is taken: - "a deliberate D-125 isolation posture I am misreading" -- REFUTED. D-125 is the ruling that GIVES each DC a working ISP uplink (bridge-in, single NAT); its whole purpose is egress. And egress was measured working. - "no NAT rule was ever configured" -- REFUTED. Outbound NAT was translating on 2026-07-30. - "interfaces mis-assigned" -- REFUTED, and re-refuted live in section 3. The only event recorded between that measurement and now is the 2026-08-01 05:45:30 hard cut. INFERRED: this is a regression caused by that cut. Sections 2-4 measure the mechanism. ================================================================================ 2. THE SERIAL LOG -- a controlled before/after inside ONE file ================================================================================ Serial log path taken from the live domain XML, not assumed: MEASURED `virsh dumpxml vr1-dc1-opnsense` -> INSTRUMENT CURRENCY (stated before any negative is drawn from this file): MEASURED size=79424 mtime=2026-08-01 07:31:36.982889689 +0000 MEASURED the file contains exactly TWO boots, and `append='on'` means it was never rotated: boot 1 ends line 259, banner dated "Wed Jul 22 21:51:39 UTC 2026", LAN (vtnet0) -> v4: 192.168.1.1/24 (FACTORY addressing; the D-112(c) console bootstrap session follows it inline) boot 2 ends line 2670, banner dated "Sat Aug 1 07:31:36 UTC 2026", LAN 10.12.64.1/22, WAN 172.30.3.2/24 (AS-BUILT addressing) MEASURED fsck damage counts in the file: UNREF FILE 76, UNEXPECTED SOFT UPDATE INCONSISTENCY 181, SALVAGE 46, ld-elf errors 0. (Matches the dc0 report's dc1 control column exactly.) NOTE, and it is load-bearing: there is NO boot between 2026-07-22 and 2026-08-01. The as-built config was applied LIVE over the D-113 API on 2026-07-23 and the edge then ran continuously. The 2026-08-01 boot is therefore the FIRST TIME this edge has ever booted on its as-built config. See section 7, finding (c). --- 2a. The pre-cut boot (lines 225-245, VERBATIM) ------------------------- Setting hostname: OPNsense.internal Generating /etc/resolv.conf...done. Generating /etc/hosts...done. Configuring system logging...done. Configuring firewall.......done. Configuring hardware interfaces...done. Configuring loopback interface...done. Configuring LAN interface...done. Configuring WAN interface...done. Generating /etc/resolv.conf...done. Generating /etc/hosts...done. Configuring firewall.......done. Starting web GUI...done. Setting up routes...done. Starting Dnsmasq...done. Starting Unbound DNS...done. Configuring firewall.......done. --- 2b. fsck, then the first casualty (lines 2548-2565, VERBATIM) ---------- SUMMARY INFORMATION BAD SALVAGE? yes BLK(S) MISSING IN BIT MAPS SALVAGE? yes 77557 files, 534225 used, 2165014 free (86 frags, 270616 blocks, 0.0% frag.) ***** FILE SYSTEM MARKED CLEAN ***** ***** FILE SYSTEM WAS MODIFIED ***** pwd_mkdb: line #1 too long pwd_mkdb: /etc/master.passwd: Inappropriate file type or format pwd_mkdb: line #1 too long pwd_mkdb: /etc/master.passwd: Inappropriate file type or format This is the dc1 equivalent of dc0's `ld-elf.so.1: libcrypto.so.17 not found`: the FIRST line after "FILE SYSTEM WAS MODIFIED" names the casualty. On dc0 it was the shared libraries. On dc1 it is /etc/master.passwd. --- 2c. The post-cut boot (lines 2615-2650, VERBATIM) ---------------------- Setting hostname: OPNsense.internal Generating /etc/resolv.conf...done. Generating /etc/hosts...done. Configuring system logging...done. Configuring firewall.....failed. <<<< Configuring hardware interfaces...done. Configuring loopback interface...done. Configuring LAN interface...done. Configuring WAN interface...done. Generating /etc/resolv.conf...done. Generating /etc/hosts...done. Configuring firewall.....failed. <<<< Configuring OpenSSH...failed. <<<< Starting web GUI...pwd_mkdb: line #1 too long pwd_mkdb: /etc/master.passwd: Inappropriate file type or format pw: passwd file update: No error: 0 chown: wheel: illegal group name <<<< x9 ... done. Setting up routes...done. Starting Dnsmasq...done. Starting Unbound DNS...done. Configuring firewall.....failed. <<<< Setting up gateway monitor...done. Syncing OpenVPN settings...done. Starting NTP service...done. Starting Unbound DNS...done. and further down, same boot: Configuring login behaviour...pw: group `wheel' does not exist Starting Cron: error in configd communication, see syslog for details error in configd communication, see syslog for details (openvpn) chown: hostd: illegal user name / illegal group name /usr/local/etc/rc.d/hostwatch: WARNING: failed to setup hostwatch MEASURED, exhaustive grep over the whole file: "Configuring firewall" appears at lines 229, 236, 241 (all "......done.") and 2619, 2626, 2644 (all "....failed."). "Configuring OpenSSH" appears ONCE, at line 2627, and it FAILED. HONEST QUALIFIER on 2a: the three "done." lines are from the FACTORY-config boot, so they prove the firewall subsystem works on THIS IMAGE, not that the as-built config booted cleanly (it had never booted). The evidence that the as-built config's NAT worked is the 2026-07-30 measurement in section 1, which is stronger anyway because it is behavioral. MEASURED qemu started the edge at 2026-08-01 05:45 (mtime of /var/log/libvirt/qemu/vr1-dc1-opnsense.log); the boot banner is stamped 07:31:36. ~1h46m elapsed. INFERRED that gap is the fsck pass over a damaged UFS root (the ~2,000 lines of salvage output between the two). Not independently timed. ================================================================================ 3. LIVE STATE -- what the edge is doing right now ================================================================================ All probes run 2026-08-02 ~06:00-06:20 UTC. Rack identity confirmed first: MEASURED `ssh vr1-dc1-rack 'hostname'` -> vvr1-dc1 ; uptime "up 1 day, 10 min" (consistent with the 2026-08-01 05:45 boot) MEASURED ip -4: enp1s0 172.31.0.6/30, virbr4 10.12.64.2/22, virbr6 10.12.68.2/.3/.4/22, virbr0 192.168.122.1/24 MEASURED ip route: "default via 10.12.64.1 dev virbr4" MEASURED ip neigh dev virbr4: 10.12.64.1 lladdr 52:54:00:95:65:64 STALE -- and 52:54:00:95:65:64 is exactly the net0 MAC in the edge's domain XML. The gateway answering IS the edge VM, not a squatter. Domain XML (MEASURED, `virsh dumpxml vr1-dc1-opnsense`): net0 mac 52:54:00:95:65:64 network vr1-dc1-provider-public bridge virbr4 target vnet0 (LAN leg) net1 mac 52:54:00:87:3b:74 network vr1-dc1-wan bridge br-vr1-dc1-wan target vnet1 (WAN leg, D-125 bridge-in) disk pool vr1-dc1-inner-pool vol vr1-dc1-opnsense-disk.qcow2, EMPTY -- standalone qcow2, no copy-on-write base to roll back to. Identical to dc0. (qemu-img could not be run on it: the file is locked by the running guest. The empty is the evidence.) Interfaces are CORRECTLY assigned. "mis-assigned interfaces" is refuted. Reachability (MEASURED): from the rack (vvr1-dc1): ping 10.12.64.1 (edge LAN) -> 2/2, 0% loss ping 172.30.3.2 (edge WAN) -> 2/2, 0% loss ping 1.1.1.1 -> 0/2, 100% loss from vcloud: ping 172.30.3.2 -> 2/2, 0% loss ip neigh -> 172.30.3.2 dev virbr1 lladdr 52:54:00:87:3b:74 REACHABLE (matches the net1 MAC: the D-125 bridge-in L2 path is intact end to end, inner tap -> br-vr1-dc1-wan -> vcloud virbr1) virsh net-dumpxml vr1-dc1-uplink -> forward mode='nat', ip 172.30.3.1/24, bridge virbr1 -- the simulated ISP is up and is a NAT for 172.30.3.0/24. Service ports on the edge (MEASURED): LAN 10.12.64.1 WAN 172.30.3.2 (from vcloud) tcp/22 (sshd) closed / no answer closed / no answer tcp/80 301 301 tcp/443 200 200 GET / title "Login | OPNsense" sshd is DOWN -- consistent with "Configuring OpenSSH...failed." The web GUI is UP and serving PHP. API (D-113 key, MEASURED; the key was never printed, never copied to another host -- opnsense-api.sh passes the secret to curl via --config on stdin): unauthenticated GET /api/core/menu/search -> 302 in 0.022s unauthenticated GET /api/core/firmware/status -> 302 in 0.017s AUTHENTICATED GET core/menu/search -> NO RESPONSE, killed at 40s AUTHENTICATED GET diagnostics/interface/getInterfaceNames -> killed at 30s AUTHENTICATED GET diagnostics/interface/getRoutes -> killed at 30s The web stack answers unauthenticated requests in ~20ms and hangs forever the moment authentication is attempted -- including on core/menu/search, which does not use configd. INFERRED: the hang is in the authentication path, which is the one thing that must consult a user database that no longer parses. Not proven further; the API is simply not a usable instrument on this box. ================================================================================ 4. THE DECISIVE TEST -- forwards yes, translates no ================================================================================ tcpdump on BOTH of the edge's tap devices simultaneously, on the rack (the taps belong to the rack's libvirt), while the rack pinged 1.1.1.1. Read-only. MEASURED vnet0 (edge LAN tap): 06:04:16.702929 IP 10.12.64.2 > 1.1.1.1: ICMP echo request, id 34238, seq 1 06:04:17.757057 IP 10.12.64.2 > 1.1.1.1: ICMP echo request, id 34238, seq 2 06:04:18.781116 IP 10.12.64.2 > 1.1.1.1: ICMP echo request, id 34238, seq 3 06:04:19.805049 IP 10.12.64.2 > 1.1.1.1: ICMP echo request, id 34238, seq 4 MEASURED vnet1 (edge WAN tap), SAME four packets, ~0.8 ms later: 06:04:16.703747 IP 10.12.64.2 > 1.1.1.1: ICMP echo request, id 34238, seq 1 06:04:17.757724 IP 10.12.64.2 > 1.1.1.1: ICMP echo request, id 34238, seq 2 06:04:18.781871 IP 10.12.64.2 > 1.1.1.1: ICMP echo request, id 34238, seq 3 06:04:19.805778 IP 10.12.64.2 > 1.1.1.1: ICMP echo request, id 34238, seq 4 This settles Q1 completely: * The edge HAS a working default route out its WAN. Packets are routed LAN->WAN. * pf is NOT blocking them -- they are passed, not dropped. * The source address is STILL 10.12.64.2 on the WAN side. NO OUTBOUND NAT IS APPLIED. The packet lands on vcloud's virbr1, whose libvirt NAT masquerades only 172.30.3.0/24, so a 10.12.64.2 source is never translated and no reply can ever come back. That is exactly the observed "0% loss to the gateway, 100% loss to the internet." So the failure is precisely: pf has no outbound-NAT ruleset loaded. It is NOT a routing failure, NOT an interface failure, NOT an L2 failure, NOT an upstream failure. --- 4a. CONTROL: pf is enforcing NO ruleset at all ------------------------- An OPNsense edge with any ruleset loaded blocks unsolicited inbound on WAN -- there is no default WAN pass rule, and the anti-lockout rule is LAN-only. So "can the ISP side reach the GUI?" is a direct test for a loaded ruleset. MEASURED, from vcloud, against the HEALTHY office1 edge (same OPNsense 26.7 image, same D-112/D-113 build path -- the control): 172.30.1.2 found via `ip neigh show dev virbr11` (not guessed) ping 172.30.1.2 -> 0/2, 100% loss (ICMP blocked on WAN) https://172.30.1.2/ -> 000 after 12.006s (no answer) http://172.30.1.2/ -> 000 after 12.006s (no answer) tcp/22 -> closed MEASURED, from vcloud, against the SUBJECT dc1 edge: ping 172.30.3.2 -> 2/2, 0% loss (ICMP answered on WAN) https://172.30.3.2/ -> 200 in 0.014s (GUI served to the ISP) http://172.30.3.2/ -> 301 tcp/22 -> closed (sshd is down, not firewalled) The dc1 edge is answering ICMP and serving its management GUI to the simulated ISP. A healthy sibling on the identical image does neither. Conclusion, stated to exactly the strength of the evidence: the dc1 edge has NO EFFECTIVE pf RULESET -- MEASURED, there is no outbound NAT (section 4) and no WAN block (this control). WHAT THIS DOES NOT DISTINGUISH: pf disabled / ruleset empty, versus pf loaded with a partial or permit-all ruleset. Both produce identical external behaviour and both produce no NAT, and neither can be separated without `pfctl -s info` on the guest (section 8). The recommendation is the same either way, so this is recorded as an open sub-question, not chased. >>> SECURITY CONSEQUENCE, flagged not actioned (read-only session): MEASURED, >>> there is NO INBOUND FILTERING IN FORCE ON THE dc1 EDGE'S WAN -- its >>> management GUI answers 200 on its WAN address from the simulated ISP >>> segment, and it answers ICMP there, neither of which the healthy control >>> does. It is functioning as an open router between 172.30.3.0/24 and the dc1 >>> provider-public plane. The exposure is bounded by the lab's own topology >>> (172.30.3.0/24 is a vcloud-local libvirt NAT, not routable inbound from the >>> real internet), which is why this is a finding and not an incident. It >>> belongs in docs/security-ledger.md as a new SEC row. I did not open one: >>> numbering and ledger rows are the operator's to assign. ================================================================================ 5. IS THE CONFIG INTACT? (Q2) ================================================================================ EVIDENCE THAT config.xml SURVIVED -- and none of it is "it booted": MEASURED The post-cut boot configured both interfaces onto the EXACT ruled as-built values, which exist ONLY in config.xml: LAN (vtnet0) -> v4: 10.12.64.1/22 WAN (vtnet1) -> v4: 172.30.3.2/24 These match changelog-20260723-g12-dc1-edge.md Item 5 and lib-net.sh's provider-public gateway exactly. A damaged or reverted config would have come up on the factory 192.168.1.1/24 (which is what boot 1 in the very same log shows it doing). MEASURED "Configuring LAN interface...done." / "Configuring WAN interface...done." / "Setting up routes...done." -- the config parsed and drove the interface + routing layer. MEASURED Dnsmasq, Unbound (twice), NTP, the gateway monitor and the web GUI all started "done." on the post-cut boot. MEASURED Section 4 proves a default route exists and works -- i.e. the data is present and usable. MEASURED The web GUI serves its login page (200, correct ), so the PHP stack that READS config.xml is functional. dc0's is not. WHAT WAS ACTUALLY DESTROYED -- and it is not config: MEASURED /etc/master.passwd is unparseable: pwd_mkdb reports "line #1 too long" and "Inappropriate file type or format" on every one of four attempts across the boot. So /etc/pwd.db and /etc/spwd.db were never rebuilt. MEASURED /etc/group is damaged: `pw: group 'wheel' does not exist`, and 17 x `chown: wheel: illegal group name` plus 4 x `chown: hostd: illegal user name` / `illegal group name` (exact grep counts). Both are FreeBSD BASE SYSTEM files. Neither is part of config.xml. They are fsck casualties in the same class as dc0's libcrypto.so.17 -- the file was reconnected as garbage or truncated, not removed. --- 5a. THE NAMED-CASUALTY INVENTORY -- the strongest evidence in this file --- fsck names every inode it DELETES. MEASURED, exhaustive over the serial log: REMOVE? yes 50 UNALLOCATED 50 NAME= 50 RECONNECT? yes 92 All 50 named casualties, deduped and complete: /conf/event_config_changed.json <-- the ONLY /conf casualty /root/.history /usr/local/etc/bogons /usr/local/etc/bogonsv6 /usr/local/etc/unbound.opnsense.d/access_lists.conf /usr/local/etc/unbound.opnsense.d/domainoverrides.conf /usr/local/etc/unbound.opnsense.d/dot.conf /usr/local/etc/unbound.opnsense.d/safesearch.conf /usr/local/etc/unbound/unbound-blocklists.conf /usr/local/etc/unbound_dhcpd.conf /usr/local/lib/perl5/5.42/perl/man/mandoc.db /usr/local/lib/perl5/site_perl/man/mandoc.db /usr/local/lib/python3.13/site-packages/__pycache__/unboundmodule.cpython-313.pyc /usr/local/share/man/mandoc.db /usr/share/openssl/man/mandoc.db /var/backups/gpart.vtbd0.bak /var/backups/group.bak <<<< THE BACKUP OF A CORRUPT FILE /var/backups/kern.geom.conftxt.bak /var/backups/master.passwd.bak <<<< THE BACKUP OF A CORRUPT FILE /var/backups/pkg.sql.xz.1 .2 .3 .4 .7 /var/cron/tabs/root /var/db/dhcp6c_duid /var/db/entropy/saved-entropy.8 /var/db/locate.database /var/db/pkg/vuln.xml /var/run/cron.pid dhcp6c.pid dpinger_WAN_GW.pid dpinger_WAN_GW.sock /var/run/gateway_watcher.pid rtsold.pid sshd.pid unbound.pid utx.active /var/unbound/advanced.conf cache.dump.gz host_entries.conf /var/unbound/private_domains.conf resolv.conf.root root.hints root.key /var/unbound/unbound.conf unbound_control.key unbound_control.pem /var/unbound/unbound_server.key unbound_server.pem TWO CONCLUSIONS, both directly measured: (i) /conf/config.xml IS NOT IN THE LIST. The entire /conf tree lost exactly one file, and it is `event_config_changed.json` -- a transient event marker, not configuration. Combined with the fact that the config demonstrably drove the boot onto its exact as-built addresses, this is positive evidence of survival rather than an inference from "it booted." QUALIFIER, because absence of evidence is not evidence of absence: the 92 RECONNECT (UNREF FILE) entries are reported by INODE, not by name, so fsck could in principle have unlinked a file into lost+found without printing its path. Absence from this list is strong but not absolute. The check that settles it is `xmllint --noout /conf/config.xml` -- see section 8. (ii) BOTH FreeBSD-NATIVE BACKUPS OF THE CORRUPT FILES WERE ALSO DESTROYED. /var/backups/master.passwd.bak and /var/backups/group.bak are on the REMOVE list. So the obvious repair -- copy the .bak back over the corrupt original -- IS NOT AVAILABLE. This corrects an earlier draft of this assessment which read the master.passwd.bak line as evidence the backup still existed; it appears in the log as `UNALLOCATED I=77473 OWNER=0 MODE=0 SIZE=0 ... REMOVE? yes`, i.e. as a deletion, not a survival. Repair must regenerate the user database from config.xml instead (section 7). A third, minor observation: nearly every /var/unbound file was destroyed, including unbound's TLS keypairs -- yet "Starting Unbound DNS...done." appears twice on the post-cut boot. INFERRED: OPNsense regenerates the whole /var/unbound tree from config.xml at boot, which is another datapoint that config.xml is intact and that the config->runtime generators work. The one generator that did NOT work is the firewall's. CAUSAL CHAIN MEASURED the damage and the failures are in the same boot, in this order: fsck salvage -> master.passwd/group unparseable -> firewall configure failed x3 -> OpenSSH configure failed -> configd communication errors -> API auth hangs -> no pf ruleset -> no NAT. INFERRED that the passwd/group corruption is the CAUSE of the firewall configure failure. The adjacency is measured; the specific internal dependency (OPNsense's filter_configure resolving users/groups, or configd failing to start for the same reason) is NOT proven, because proving it needs a shell on the edge, which I do not have read-only. ALTERNATIVE NOT FULLY EXCLUDED, stated honestly: because 2026-08-01 was the FIRST boot this edge ever took on its as-built config, a latent boot-time config defect cannot be excluded by this evidence alone. It is unlikely -- the fsck damage to base files is measured and sufficient, and outbound NAT was working live on 2026-07-30 -- but it is not zero, and the repair path in section 7 distinguishes them for free (if repairing the user DB restores NAT, the alternative is dead). WHAT I COULD NOT READ, and why (this is a gap, not a pass): I have NOT read config.xml itself, nor /conf/backup/, nor the live pf ruleset (`pfctl -sr` / `pfctl -sn`). All three need a shell on the edge, and all three read-only routes to one are closed: - SSH: tcp/22 does not answer (MEASURED). sshd failed to configure. - REST API: authenticated GETs hang indefinitely (MEASURED). - Serial console: the guest is sitting at `login:`. Typing at it is an interaction with the guest, which I read as outside "READ-ONLY, make no mutations," so I did not attempt it. INFERRED, and worth the operator's attention before anyone tries: a console login will most likely FAIL anyway, because authentication needs the same /etc/master.passwd that pwd_mkdb cannot parse. If the operator wants config.xml and the pf ruleset read directly, the commands are listed in section 8. I am stopping rather than guessing. ================================================================================ 6. ANY OTHER dc1 DAMAGE FROM THE SAME CUT? (Q4) ================================================================================ INNER GUESTS -- MEASURED, `virsh list --all` + per-domain `dominfo`: vr1-dc1-opnsense running Autostart=enable vr1-dc1-control-01 shut off Autostart=disable vr1-dc1-control-02 shut off Autostart=disable vr1-dc1-control-03 shut off Autostart=disable vr1-dc1-compute-01 shut off Autostart=disable vr1-dc1-compute-02 shut off Autostart=disable vr1-dc1-storage-01 shut off Autostart=disable vr1-dc1-storage-02 shut off Autostart=disable vr1-dc1-storage-03 shut off Autostart=disable vr1-dc1-storage-04 shut off Autostart=disable vr1-dc1-juju-01 shut off Autostart=disable Eleven domains, one running. The edge is the ONLY guest that was hard-cut. This is D-127 behaving exactly as written and it BOUNDS the blast radius: there is no other inner filesystem to inspect. (Consistent with the task's note that dc1 has no per-DC MAAS region VM -- MEASURED: no such domain exists.) THE RACK HOST vvr1-dc1 (ext4) -- MEASURED: df/findmnt /dev/vda1 ext4 2.9T, 1% used, rw,relatime,discard, errors=remount-ro -- NOT remounted read-only journal Aug 01 05:45:41 EXT4-fs (vda1): mounted ... ro, ordered data Aug 01 05:45:41 EXT4-fs (vda1): re-mounted ... r/w systemd-fsck-root skipped (unmet condition) -- normal BOOT: recovering journal; BOOT: clean, 604/58496 files UEFI (vfat): "differences between boot sector and its backup ... mostly harmless" journal -p err for the whole boot: ONE kernel line ("RDSEED32 is broken. Disabling the corresponding CPUID bit." -- a CPU-feature notice, unrelated) plus a repeating libvirtd line, below. systemctl --failed: "0 loaded units listed." dc1 units: apt-cacher-ng.service active running dc1-cache-proxy-net.service active exited dc1-node-dns.service active running dc1-rack-legs.service active exited libvirtd.service active running ext4 came through the cut cleanly. A journalling filesystem with an intact journal is the expected outcome; UFS-with-soft-updates is not, which is the whole reason the two edges are damaged and neither rack is. INSTRUMENT-CURRENCY NOTE, because a negative is being reported here: `dmesg` is NOT a valid instrument for the boot window on this host. It holds 4834 lines whose OLDEST timestamp is [75228.4] (~20.9h uptime) against an uptime of ~24.5h -- the ring buffer HAS WRAPPED and the boot messages are gone from it. Its "zero ext4 errors" covers only the last ~3.6 hours. The boot-window evidence above is from `journalctl -b`, which does cover 2026-08-01 05:45:41 onward. The dmesg ring is dominated by a flood of `apparmor="DENIED" ... profile="snap.maas.pebble" ... comm="ps"` lines -- that is the MAAS RACK controller snap on this host reading /proc/N/environ; pre-existing noise, not damage, and not related to this incident. PRE-EXISTING NOISE, explicitly NOT attributed to the cut -- MEASURED: `libvirtd[2531]: End of file while reading data: Input/output error` recurs in ~10-minute clusters from Aug 01 05:51:57 onward; 1406 occurrences before this session began. INFERRED: a periodic libvirt client disconnecting (power-state polling shape). It predates and is unaffected by my probes. Logged so nobody later mistakes it for new damage. ================================================================================ 7. RECOMMENDATION ================================================================================ >>> REPAIR the dc1 edge. Do NOT rebuild it. <<< WHY REPAIR IS THE RIGHT CALL (all measured): a) The configuration survived. The edge is running its exact as-built identity -- LAN 10.12.64.1/22, WAN 172.30.3.2/24, working default route, working forwarding, DNS/NTP/GUI up. A rebuild would throw all of that away and re-incur the D-112(c) console bootstrap, the D-113 API-key mint, the re-addressing, and a fresh SEC row for the transient key exposure -- to recreate a config that is already correct. b) The break is a bounded, named, base-system defect: two corrupt files (/etc/master.passwd, /etc/group) and the services that depend on them (pf ruleset generation, sshd, configd, API auth). c) dc0 is genuinely different and genuinely needs the rebuild: it lost libcrypto.so.17 and libpython3.13.so.1.0, so PHP and python cannot even load and it never reaches multi-user. dc1's PHP runs and serves pages. Same incident, same damage class, materially different remedy. The task's hypothesis -- "same incident, milder outcome" -- is CONFIRMED, and the "milder" is load-bearing: it changes the remedy. THE CAVEAT THAT DECIDES HOW, and it is the important part of this section: MEASURED, dc1 currently has NO working credential path into the guest: - sshd is down (tcp/22 does not answer) - authenticated API calls hang - INFERRED: console login will fail too, since it needs the same unparseable /etc/master.passwd dc0, paradoxically, is EASIER to get a shell on -- it dropped to a single-user prompt, which is a root shell. dc1 is sitting at a `login:` that probably cannot authenticate anyone. So the repair almost certainly requires BOOTING THE dc1 EDGE INTO SINGLE-USER MODE (interrupt the loader, `boot -s`) to get a shell, then rebuilding the user database. AND THE OBVIOUS RESTORE PATH IS CLOSED -- MEASURED, section 5a(ii): fsck DELETED both /var/backups/master.passwd.bak and /var/backups/group.bak, so there is no native backup to copy back. The repair therefore has to REGENERATE the user/group database rather than restore it. The two candidate sources, in preference order: 1. OPNsense's own generators, driven from the surviving config.xml -- the same `local_user_set()` path the D-112(c) console bootstrap already used on this exact edge (see changelog-20260723-g12-dc1-edge.md Item 2, which records the working PHP shape including the `require_once("util.inc")` fix). This is the repo-proven route and it reuses tooling that exists. 2. The pristine base image in the same pool -- MEASURED, `virsh vol-list vr1-dc1-inner-pool` holds `opnsense-26.7-nano.qcow2` -- as a donor for the stock /etc/master.passwd and /etc/group, which for an appliance are essentially image constants. Then re-run (1) to re-apply config users. THAT IS A REBOOT. It is a mutation. I did not do it and I am not proposing to do it inside this assessment -- it needs its own gated step, and it should be preceded by a disk snapshot/copy of vr1-dc1-opnsense-disk.qcow2 (the domain XML shows an EMPTY <backingStore/>: there is no rollback point today, on either edge). Copying that disk requires the guest to be shut down or the copy to be taken with the guest quiesced -- also a mutation, also gated. FALLBACK IF REPAIR FAILS: the rebuild material is present and identical to dc0's -- MEASURED, `virsh vol-list vr1-dc1-inner-pool` contains the pristine `opnsense-26.7-nano.qcow2` alongside `vr1-dc1-opnsense-disk.qcow2`. So a rebuild remains available at no additional risk; it is simply the more expensive answer and the evidence does not call for it. DURABLE FINDINGS THIS RAISES (logged, not actioned -- hard rule 1): (a) The dc0 report's procedure gap is confirmed by a second, independent datapoint: an "in-place" tofu change to a containment VM is a power cut to every inner guest, and BOTH edges took filesystem damage from it. The remedy the dc0 report proposes -- shut inner guests down first -- would have prevented both. (b) dc-egress-check.sh's A2 assertion cannot see this failure mode. On dc1 the edge ANSWERS at 10.12.64.1 (A2 OK) and forwards, and the DC still has no egress. A2 is doing its job for the dc0 shape and is simply not the instrument for the dc1 shape. A candidate A2b: "the edge translates" -- and it should be written in the POSITIVE form: assert that traffic observed leaving the edge's WAN side carries the WAN SOURCE ADDRESS (the section 4 tcpdump), which is a real positive assertion and can fail loudly. The tempting cheap version -- "assert the edge does NOT serve its GUI on its WAN address", i.e. the section 4a control -- is an INVERTED assertion that reads PASS whenever a curl times out for any reason, which is exactly the "a checker that cannot fail is not a gate" / "could not look is never nothing there" shape this repo has already been bitten by twice. Use the control as corroboration, never as the gate itself. Worth raising as a gate improvement; I have NOT touched the script. (c) NEITHER DC EDGE HAD EVER BEEN REBOOT-TESTED. MEASURED for dc1: the serial log contains exactly two boots, 2026-07-22 (factory) and 2026-08-01 (as-built, first ever). The as-built config was applied live over the API on 2026-07-23 and never proven to survive a restart. Both edges' first real reboot was an unclean one, which is the worst possible way to find out. Roosevelt delta: an edge's config is not "built" until it has been proven across a clean reboot. That reads like D-NNN material rather than a changelog line, but assigning the number is the operator's call. (d) SECURITY: section 4a's open-router finding needs a SEC row. ================================================================================ 8. WHAT I COULD NOT ANSWER READ-ONLY, and the exact commands that would ================================================================================ Stated rather than guessed, per the ground rules. Q: Is config.xml byte-intact, and are /conf/backup/ revisions present? Q: What exactly is in the live pf ruleset -- empty, or absent? Q: Is it really master.passwd/group that breaks filter_configure? All three need a shell on the edge. Every read-only route to one is closed (section 5). The minimal commands, once a shell exists, are: ls -la /conf/config.xml /conf/backup/ | tail -20 xmllint --noout /conf/config.xml && echo XML-WELL-FORMED grep -c '<rule>' /conf/config.xml ; grep -o '<outbound>' /conf/config.xml pfctl -s info ; pfctl -sr | head -40 ; pfctl -sn | head -40 wc -c /etc/master.passwd /etc/group ; head -3 /etc/group ls -la /var/backups/ # EXPECT master.passwd.bak and group.bak to # be ABSENT -- fsck removed both (5a(ii)) ls -la /lost+found | head # the 92 RECONNECTed inodes land here Getting that shell requires `boot -s` at the loader, i.e. a REBOOT of the edge guest. That is a mutation and it is where I stop. ================================================================================ END. Nothing in this session was mutated. The repo tree is left dirty and uncommitted by instruction. ================================================================================