dc0-edge-egress-outage-20260802.txt
====================================
The vr1-dc0 OPNsense edge has been down since the 2026-08-01 05:48 containment-VM
reboot. Found 2026-08-02 while running Step 3.5, by `dc-mirror.sh check dc0`.
Read-only throughout. Nothing was restarted by this capture.

HOW IT SURFACED -- a gate doing exactly what it was fixed to do
  dc-mirror.sh check dc0 -> FAIL
    "MISS last-sync FAILED: FAIL 2026-08-02T00:42:36Z ubuntu=2 uca=2
     -- the mirror content is NOT attested current"
  This is the 2026-07-27 fix earning its keep: before it, `check` asserted that
  last-sync.status EXISTED and printed it behind an unconditional OK, so a FAIL
  status passed. Content-assertion caught a 19-hour outage.

MEASURED
  sync journal   "rsync error: error in socket IO (code 10)"
                 "Download of dists/jammy-updates/caracal/Release failed:
                  500 Can't connect to ubuntu-cloud.archive.canonical.com"
                 dc0-mirror-sync.service: Failed with result 'exit-code'
  rack -> archive.ubuntu.com                     000
  rack -> streams.canonical.com/juju/tools/      000
  rack -> api.snapcraft.io VIA the snap proxy    000
  rack -> 1.1.1.1                                100% packet loss
  rack -> 10.12.4.1 (its OWN default gateway)    100% packet loss
  ip neigh dev virbr5:  10.12.4.1 FAILED
                        10.12.4.5 lladdr ... STALE   (juju controller, fine)
                        10.12.4.6 lladdr ... STALE   (MAAS region, fine)
  virsh:         vr1-dc0-opnsense RUNNING, vnet0 on vr1-dc0-provider-public,
                 vnet1 on vr1-dc0-wan -- both attached
  qga:           "QEMU guest agent is not connected" (the G13 qga retrofit was
                 office1-opnsense ONLY; the DC edges never got it)
  qemu log:      last write 2026-08-01 05:48 -- the reboot
  last mirror sync SUCCESS: 2026-08-01 00:04:19, i.e. BEFORE that reboot

SO: the edge VM is running with its NICs attached and is not answering at L2 on
its LAN leg, while two neighbours on the same segment answer normally.

WHY IT BLOCKS STAGE 5 STEP 4
  apt would be fine -- the mirror still serves jammy, jammy-updates,
  jammy-backports and jammy-security at 200 from its Aug-1 content.
  The deploy ALSO needs streams.canonical.com for juju agent binaries and the
  snap proxy's upstream for snaps. Both are dead. That is exactly the failure
  class the 2026-07-31 deploy hit (four artifact defects, each masked by the one
  before it).

>>> TWO GAPS THIS EXPOSES, both worth more than the incident <<<
  1. NO GATE ANYWHERE TESTS DC EGRESS. `dc-rack-net.sh check dc0` PASSES with
     egress dead -- it asserts bridge legs and unit states, not reachability.
     preflight has no egress gate either. A 19-hour outage on the deploy's
     critical path surfaced only because a nightly timer happened to fail.
  2. A POST-REBOOT VERIFICATION THAT LOOKED THOROUGH AND WAS NOT. This session
     reported after the 05:48 reboot that "everything reboot-persistent recovered
     unaided", citing the rack units, the region API 200, the mirror answering
     200, the snap proxy LISTENING and the juju controller connected. Every one
     of those was true. None of them tested egress, which was the thing broken.
     A listener is not a path; an HTTP 200 from a LOCAL service says nothing
     about upstream. The check set was incomplete and the summary was more
     confident than its evidence.

================================================================================
ROOT CAUSE FOUND -- AND IT IS NOT THE REBOOT (appended after the console read)
================================================================================
The dc0 edge is sitting at a SINGLE-USER PROMPT with missing shared libraries:

  ld-elf.so.1: Shared object "libcrypto.so.17" not found, required by "php"
  ld-elf.so.1: Shared object "libpython3.13.so.1.0" not found, required by "python3"
  >>> Error in early script '10-configd'
  >>> Error in early script '15-templates'
  >>> Error in early script '90-carp'
  Launching the init system...ld-elf.so.1: ... libcrypto.so.17 not found
  Enter full pathname of shell or RETURN for /bin/sh:

That is a PARTIALLY-APPLIED OPNsense UPDATE. The reboot did not break the edge --
it EXPOSED it. The running system had survived on already-mapped libraries; the
2026-08-01 05:49 boot is where it fell over. Serial log mtime 05:49:39 confirms.
This symptom appears NOWHERE in appendix-A or the docs -- it is a new class.

SCOPE, MEASURED RATHER THAN ASSUMED -- IT IS dc0-ONLY
  dc1 edge serial log: ZERO matches for libcrypto.so.17 / libpython3.13; it booted
  clean to a login prompt with "WAN (vtnet1) -> v4: 172.30.3.2/24".
  dc1 rack -> its edge 10.12.64.1 : 0% loss.  vcloud -> 172.30.3.2 : 0% loss.
  vcloud -> 172.30.2.2 (dc0 edge WAN) : 100% loss -- consistent with no config loaded.

THE UPSTREAM CHAIN IS HEALTHY -- the break is at the dc0 edge, nowhere else
  vcloud   -> 1.1.1.1 0% loss, archive.ubuntu.com 200
  voffice1 -> 1.1.1.1 0% loss, archive.ubuntu.com 200, office1 edge 10.10.0.1 0% loss
  both uplink nets ACTIVE, forward mode='nat', gateways 172.30.2.1 / 172.30.3.1
  both pingable from vcloud
  both racks hold a default route via their own edge

RECOVERY MATERIAL AVAILABLE
  The pristine base EXISTS in the same pool as the broken disk:
    vr1-dc0-inner-pool -> opnsense-26.7-nano.qcow2
                          vr1-dc0-opnsense-disk.qcow2  (the broken one)
  The broken disk has an EMPTY <backingStore/> -- it is a standalone qcow2, NOT an
  overlay, so there is no copy-on-write base to roll back to. A rebuild would mean
  a fresh disk from the nano base plus the D-112(c) console bootstrap and D-113
  API-key mint, then re-addressing (WAN 172.30.2.2/24, LAN 10.12.4.1/22).

SEPARATE THREAD, NOT CONFLATED WITH THIS ONE: dc1's rack also cannot reach 1.1.1.1
even though its edge is healthy on BOTH legs and it holds a default route via it.
That is an edge forwarding/NAT question at dc1, not the dc0 outage, and dc1 is not
on the deploy path. Logged so it is not lost; NOT diagnosed here.

================================================================================
>>> CORRECTION: THE "PARTIALLY-APPLIED UPDATE" DIAGNOSIS ABOVE IS WRONG. <<<
    The real cause is UFS damage from an unclean power cut. Measured, not reasoned.
================================================================================
The section above concluded "a PARTIALLY-APPLIED OPNsense UPDATE". That was an
inference from the SHAPE of the symptom (missing libs => something replaced them)
and it is REFUTED by the log:

  update / pkg-static / opnsense-update / firmware activity in the whole log: 0 lines

What the log actually shows, immediately BEFORE the first missing-library error:

  UNREF FILE  I=1348940 OWNER=root MODE=100644 MTIME=Jul 6 19:48 2026
  RECONNECT? yes                                    ... x2533
  UNEXPECTED SOFT UPDATE INCONSISTENCY              ... x785
  SALVAGE? yes                                      ... x257
  ** Phase 5 - Check Cyl groups
  FREE BLK COUNT(S) WRONG IN SUPERBLK / BLK(S) MISSING IN BIT MAPS / SUMMARY INFORMATION BAD
  ***** FILE SYSTEM MARKED CLEAN *****
  ***** FILE SYSTEM WAS MODIFIED *****
  ld-elf.so.1: Shared object "libcrypto.so.17" not found, required by "php"

fsck salvaged a badly damaged UFS root and the shared libraries did not survive it.
`libcrypto.so.17` and `libpython3.13.so.1.0` were fsck casualties.

THE CAUSAL CHAIN, end to end
  1. The 2026-08-01 memory resize (416 -> 480 GiB) was an IN-PLACE tofu update, and
     the repo already records that an in-place update BOUNCES the guest -- confirmed
     on dc1 as the canary that day (domain id 9 -> 12, uptime 0 min).
  2. A bounce of the CONTAINMENT VM is a hard power cut to every INNER VM. Nothing
     shuts the inner guests down first.
  3. Under D-127 the DC edges are the only inner VMs with autostart=true ("comes up
     with its containment VM"). The nine role nodes were powered off; the MAAS region
     and juju controller VMs are autostart=false and were started BY HAND afterwards,
     cleanly. So the edges were the ONLY inner guests hard-cut and auto-restarted.
  4. Both edges restarted onto damaged UFS. fsck salvaged both.

CONFIRMED BY THE dc1 CONTROL -- same event, same class, different severity:
                       dc0 edge      dc1 edge
  UNREF FILE             2533           76
  SOFT UPDATE incons.     785          181
  SALVAGE? yes            257           46
  FILE SYSTEM MODIFIED      1            1
  ld-elf errors        present            0
  qemu restart      05:48:31       05:45:30

  dc0 lost its shared libraries -> cannot reach multi-user -> no LAN service at all.
  dc1 kept its libraries -> boots to a login prompt -> BUT its damage is real too,
  which is the likely explanation for the "healthy edge that does not forward"
  logged earlier as a SEPARATE thread. IT IS NOT SEPARATE: same incident, lesser
  damage. That earlier note should be read as superseded by this one.

WHY A "RESTART" DID THIS, stated plainly: it was not a restart of the edge. It was a
POWER CUT. The edge was never asked to shut down; its parent VM was replaced under it
while it had unflushed metadata.

THE PROCEDURE GAP, which is the durable finding:
  The resize was executed with a plan asserted on CONTENT (1 in-place, 0 create/
  destroy/replace, 0 MAC changes) and a capacity gate -- all correct, and none of it
  covered the INNER guests. "In-place" describes the tofu resource, not the blast
  radius: for a containment VM it means every VM inside it is hard-cut. Any future
  in-place change to a containment VM must SHUT DOWN its inner guests first, or accept
  filesystem damage to every one of them that is running.
