dc0-edge-egress-outage-20260802.txt
====================================
The vr1-dc0 OPNsense edge has been down since the 2026-08-01 05:48 containment-VM
reboot. Found 2026-08-02 while running Step 3.5, by `dc-mirror.sh check dc0`.
Read-only throughout. Nothing was restarted by this capture.

HOW IT SURFACED -- a gate doing exactly what it was fixed to do
  dc-mirror.sh check dc0 -> FAIL
    "MISS last-sync FAILED: FAIL 2026-08-02T00:42:36Z ubuntu=2 uca=2
     -- the mirror content is NOT attested current"
  This is the 2026-07-27 fix earning its keep: before it, `check` asserted that
  last-sync.status EXISTED and printed it behind an unconditional OK, so a FAIL
  status passed. Content-assertion caught a 19-hour outage.

MEASURED
  sync journal   "rsync error: error in socket IO (code 10)"
                 "Download of dists/jammy-updates/caracal/Release failed:
                  500 Can't connect to ubuntu-cloud.archive.canonical.com"
                 dc0-mirror-sync.service: Failed with result 'exit-code'
  rack -> archive.ubuntu.com                     000
  rack -> streams.canonical.com/juju/tools/      000
  rack -> api.snapcraft.io VIA the snap proxy    000
  rack -> 1.1.1.1                                100% packet loss
  rack -> 10.12.4.1 (its OWN default gateway)    100% packet loss
  ip neigh dev virbr5:  10.12.4.1 FAILED
                        10.12.4.5 lladdr ... STALE   (juju controller, fine)
                        10.12.4.6 lladdr ... STALE   (MAAS region, fine)
  virsh:         vr1-dc0-opnsense RUNNING, vnet0 on vr1-dc0-provider-public,
                 vnet1 on vr1-dc0-wan -- both attached
  qga:           "QEMU guest agent is not connected" (the G13 qga retrofit was
                 office1-opnsense ONLY; the DC edges never got it)
  qemu log:      last write 2026-08-01 05:48 -- the reboot
  last mirror sync SUCCESS: 2026-08-01 00:04:19, i.e. BEFORE that reboot

SO: the edge VM is running with its NICs attached and is not answering at L2 on
its LAN leg, while two neighbours on the same segment answer normally.

WHY IT BLOCKS STAGE 5 STEP 4
  apt would be fine -- the mirror still serves jammy, jammy-updates,
  jammy-backports and jammy-security at 200 from its Aug-1 content.
  The deploy ALSO needs streams.canonical.com for juju agent binaries and the
  snap proxy's upstream for snaps. Both are dead. That is exactly the failure
  class the 2026-07-31 deploy hit (four artifact defects, each masked by the one
  before it).

>>> TWO GAPS THIS EXPOSES, both worth more than the incident <<<
  1. NO GATE ANYWHERE TESTS DC EGRESS. `dc-rack-net.sh check dc0` PASSES with
     egress dead -- it asserts bridge legs and unit states, not reachability.
     preflight has no egress gate either. A 19-hour outage on the deploy's
     critical path surfaced only because a nightly timer happened to fail.
  2. A POST-REBOOT VERIFICATION THAT LOOKED THOROUGH AND WAS NOT. This session
     reported after the 05:48 reboot that "everything reboot-persistent recovered
     unaided", citing the rack units, the region API 200, the mirror answering
     200, the snap proxy LISTENING and the juju controller connected. Every one
     of those was true. None of them tested egress, which was the thing broken.
     A listener is not a path; an HTTP 200 from a LOCAL service says nothing
     about upstream. The check set was incomplete and the summary was more
     confident than its evidence.