dc0-edge-rebuild-20260802.txt
=============================
REBUILD of the vr1-dc0 OPNsense edge, destroyed by UFS soft-update damage after
the 2026-08-01 05:48 hard power cut of its containment VM (full diagnosis:
docs/audit/dc0-edge-egress-outage-20260802.txt -- not re-derived here).

Definition of done, set by the operator: `ssh vr1-dc0-rack 'bash -s -- check dc0'
< scripts/dc-egress-check.sh` exits 0.

Written incrementally as each step completed. Every value below was LOOKED UP in a
script constant, a tfvars file, or a live object -- none was inferred. Where a value
in the task brief disagreed with the measured one, the measured one is used and the
divergence is called out.

================================================================================
SECTION 0 -- READ-ONLY BASELINE (before any mutation)
================================================================================

REPO / CLONE IDENTITY
  jumphost (vcloud) HEAD : 95a6f5f  branch dc-dc-stage5-preconditions
  voffice1 clone HEAD    : 22af620  (git status clean)
  The two clones are at DIFFERENT commits. That matters only if it changes the
  tofu root, so it was MEASURED rather than assumed -- sha256 of all seven files
  the inner root consumes, both hosts:

    c81aa41b093af531f81a97254116f98c524b07a44f2e02240c03876bbbce9ab3  vr1-dc0-substrate/main.tf
    f4020e606c1fc4d2a2dbf5430d0d38c518962ca3b4bd1b549de223a7e51250f2  vr1-dc0-substrate/variables.tf
    17de7c023fa1353f8f14eebf29b7bc30dd11f0bcfec1112d8b41c83fd1c3c081  vr1-dc0-substrate/versions.tf
    cd117906d8f4baf526240e20eda53931746aaff2d2df8fa40bc7f75df30e7229  modules/opnsense-edge/main.tf
    7e38c5b88e6f5c83efd25f7487af87857ca37f52b17831cfb0e9e8a12f733712  modules/opnsense-edge/outputs.tf
    b027fe2566e1e4aca89ddcade89051ea01b7b7de7470f9a2c61a2fe0cfb0e7f3  modules/opnsense-edge/variables.tf
    f11b24565b04f42959706ad305416eced71310386ff78e029a882ac38382ec2d  modules/opnsense-edge/versions.tf

  IDENTICAL on both hosts. The clone divergence therefore cannot affect this
  root, and NO `git pull` was run on voffice1 (the operator is committing on this
  branch concurrently; pulling would drag unrelated commits in mid-apply).

INSTRUMENT TRAP CAUGHT AT THE FIRST PROBE -- recorded because it would have
produced a confident wrong negative:
  `ssh vr1-dc0-rack 'virsh list --all'`  ->  EMPTY TABLE, and
  `virsh dominfo vr1-dc0-opnsense`       ->  "error: failed to get domain"
  That reads exactly like "the domain is gone". It is not.
    `virsh uri` -> qemu:///session
  The non-root default URI is the SESSION daemon, which owns nothing. With
  `-c qemu:///system` all 12 domains are listed and the edge is Id 1 running.
  EVERY virsh call in this capture carries `-c qemu:///system` for that reason.

LIVE STATE, dc0 rack (vvr1-dc0), `virsh -c qemu:///system list --all`:
   Id   Name                 State
    1    vr1-dc0-opnsense     running        <- the broken edge (single-user)
    2    vr1-dc0-maas-01      running
    3    vr1-dc0-juju-01      running
    -    9 role nodes         shut off

RACK NETWORK FACTS (measured, `ip -4 -o addr`, `ip route`, `virsh net-info`):
  provider-public plane  = libvirt net vr1-dc0-provider-public = bridge virbr5
  rack provider-public leg = 10.12.4.2/22 on virbr5     (permanent, dc-rack-net)
  rack default route       = default via 10.12.4.1 dev virbr5   <- the edge LAN
  metal-admin  = virbr2, carrying 10.12.8.2/.3/.4 (rack leg, node-DNS, utility)
  WAN plane    = libvirt net vr1-dc0-wan = bridge br-vr1-dc0-wan (D-125 bridge-in)
  So the rack's A1 precondition for the egress gate is ALREADY satisfied; only the
  edge itself is missing.

TOFU ROOT (on voffice1, per D-128 -- inner roots run from the Office1 headend)
  path     : ~/openstack-caracal-dc-dc/opentofu/vr1-dc0-substrate/
  tofu     : OpenTofu v1.12.4, provider dmacvicar/libvirt v0.9.8
  state    : 36 resources, of which the two edge resources are exactly
               module.vr1_dc0_opnsense.libvirt_volume.disk
               module.vr1_dc0_opnsense.libvirt_domain.vm

>>> CORRECTION TO THE TASK BRIEF -- BASE IMAGE PATH. <<<
  The brief cited the base image as
      /var/lib/libvirt/vr1-dc0-inner/opnsense-26.7-nano.qcow2
  That is the module DEFAULT in variables.tf, and it is NOT the effective value.
  `d124-inner.auto.tfvars` on voffice1 OVERRIDES it:
      opnsense_base_path = "/home/jessea123/vr1-dc0-images/opnsense-26.7-nano.qcow2"
  and variables.tf:27 explains why, in a note dated at the FIRST inner apply:
      "Path ON THE EXECUTING HOST (voffice1, D-128 Plane 2) ... the remote
       qemu+ssh provider UPLOADS volume content from ITS OWN filesystem -- a path
       on vvr1-dc0 fails with 'Could not open URL for upload: failed to stat'."
  This is load-bearing: the apply DESTROYS the volume before creating it, so a
  base path that resolves only on the rack would have left the edge with no disk
  at all. VERIFIED PRESENT ON voffice1 BEFORE ANY MUTATION:
      -rw-r--r-- 1 jessea123 jessea123 2192703664 Jul 20 06:13
        /home/jessea123/vr1-dc0-images/opnsense-26.7-nano.qcow2
      sha256 3981821e3a3c7be49f6d96f405266806a619756d8816f8a9d92e7fdf5476627d
  (The rack ALSO holds a copy at the default path -- same filename, staged by the
  bootstrap step -- which is exactly why the wrong one is easy to cite.)

BOOTSTRAP MATERIAL ALREADY ON THE RACK (from the 2026-07-20 original build)
  ~/d112c-console.py         2481 B  D-112(c) console driver v6 (materialize leg)
  ~/d112c-bootstrap.php.b64   992 B  the config-write payload
  ~/scripts/{opnsense-api.sh,opnsense-mint-apikey.php,
             opnsense-set-iface-v4.php,opnsense-set-interface-v4.sh}  (Jul 20)
  ~/opnsense-bootstrap-apikey.sh, ~/opnsense-mint-apikey.php          (Jul 20)
  python3 pexpect 4.9.0 present on the rack.

  The `.b64` payload was DECODED and read. Two things confirmed by reading it
  rather than trusting the record:
   (a) It carries only `require_once("config.inc")` -- NO `util.inc`. That is
       exactly the DOCFIX-199 / dc1-lesson-(iv) defect: `write_config()` calls
       `shell_safe()`, which lives in util.inc, so this payload dies at
       "Call to undefined function shell_safe()" BEFORE writing. The runbook's
       own warning ("Any surviving dc0 .b64 artifact on a rack predates this line
       and is missing the include -- do not replay it verbatim") is CORRECT and
       is now confirmed against the artifact itself.
   (b) The public key it installs is
         ssh-ed25519 AAAAC3...tzoc vr1-dc0-edge_svc (D-112c per-env)
       which is BYTE-IDENTICAL to ~/vr1-dc0-creds/vr1-dc0-edge_ed25519.pub on the
       jumphost (SHA256:2Gknnn3FsMtJov7MFBQlAjT6XJAd5EVz4OaCuY2ZLaU). The existing
       private key therefore still opens the rebuilt edge -- no new keypair needed.

>>> PRE-EXISTING CREDENTIAL FINDING, FOUND WHILE STAGING (logged, not actioned
    beyond what the rebuild requires) -- SEE SECTION 6. <<<
  `ls` (never a read) of vr1-dc0-rack:~/vr1-dc0-creds/ shows, dated 2026-07-20:
      -rw------- 173 B  opnsense-api.txt
      -rw------- 432 B  vr1-dc0-edge_ed25519          (the PRIVATE half)
  SEC-021(a) records that the dc0 edge API credential is ABSENT from its recorded
  jumphost location and asks for "a decision between RE-MINT and locating an
  off-jumphost copy". THE OFF-JUMPHOST COPY IS THIS ONE. It has been on the rack
  for 13 days. The dc1 build shredded its equivalents the same phase and opened
  SEC-015 for the transient; the dc0 build never did, and nothing noticed --
  `creds-manifests/vm-secret-locations:123` DOES declare
  `rack vr1-dc0 ssh vr1-dc0-rack:~/vr1-dc0-creds/*` as an audited location, so
  this was visible to a `--remote` sweep and simply had not been reconciled.
  CONSEQUENCE FOR THIS TASK: the 173 B key authenticates to a config that this
  rebuild DESTROYS, so it is dead material after the apply and must be re-minted
  regardless. It also blocks `opnsense-bootstrap-apikey.sh`, which refuses to
  overwrite an existing output file (by design -- overwriting would strand a live
  key). Handling is in Section 4.

================================================================================
SECTION 1 -- RE-PLAN AND CONTENT ASSERTION (step 1)
================================================================================

The stale plan the brief mentioned (/tmp/edge-rebuild.plan, Aug 2 03:36) was NOT
used. A fresh one was cut. Pre-mutation safety captures first:

  voffice1: cp -p terraform.tfstate ~/vr1-dc0-substrate-tfstate-pre-edge-rebuild-20260802.json
            sha256 45e9aad95e37cdbf2840508173aa7cea56ddc469fdc922a57f0516742ec77b9c
            (identical both files -- the backup is a real copy, verified not assumed)
  rack:     serial log BASELINE recorded so the post-rebuild boot can be asserted on
            NEW content only.  The domain's serial device has append="on", so the file
            still holds the DAMAGED 08-01 boot; a grep of the whole file for a
            clean-boot marker would match the old boot and look exactly like success.
              /var/lib/libvirt/vr1/staging/vr1-dc0-opnsense-serial.log
              size=477629  lines=21912  mtime=2026-08-01 05:49:39

PLAN (run on voffice1, D-128 Plane 2):
  cd ~/openstack-caracal-dc-dc/opentofu/vr1-dc0-substrate/
  tofu plan -replace=module.vr1_dc0_opnsense.libvirt_volume.disk \
            -replace=module.vr1_dc0_opnsense.libvirt_domain.vm \
            -out=/tmp/edge-rebuild-20260802.plan

  Plan: 2 to add, 0 to change, 2 to destroy.

ASSERTION ON CONTENT, machine-read from `tofu show -json`, not from prose.
Prose was deliberately not the assertion surface: the refresh reports 20 resources
under "changed outside of OpenTofu", which is a DRIFT NOTE and not an action, and a
grep over prose blurs the two.

  $ jq '.resource_changes[] | select(.change.actions != ["no-op"]) | .address + " " + (.change.actions|join(","))'
      module.vr1_dc0_opnsense.libvirt_domain.vm     ACTIONS=delete,create
      module.vr1_dc0_opnsense.libvirt_volume.disk   ACTIONS=delete,create

  count of non-no-op changes ................................. 2   (expected 2)
  count of non-no-op changes NOT naming vr1_dc0_opnsense ..... 0   (required 0)

Both halves were asserted, not just the "nothing extra" half. Asserting only that
every action line names the edge would PASS on a plan that replaced the domain and
left the corrupt qcow2 in place -- which would reattach the damaged disk and make
the whole rebuild a no-op on the actual damage. So the positive half (exactly these
TWO addresses, each delete+create) is asserted as well.

BASE IMAGE THE PLAN WILL ACTUALLY USE, read out of the plan rather than trusted:
  $ jq '.resource_changes[] | select(.address|test("libvirt_volume.disk")) | .change.after.create.content.url'
      /home/jessea123/vr1-dc0-images/opnsense-26.7-nano.qcow2
  -- the voffice1 path, confirmed present at 2192703664 B before the apply. This is
  the check that made the brief's cited path worth correcting: the apply destroys
  the volume BEFORE creating its replacement, so a source path that resolved only on
  the rack would have left the edge with no disk at all and no way back.

VERDICT: plan matches the intent exactly. Proceed to apply.

================================================================================
SECTION 2 -- APPLY (step 2)
================================================================================

  $ tofu apply /tmp/edge-rebuild-20260802.plan       (on voffice1)
    module.vr1_dc0_opnsense.libvirt_domain.vm: Destroying... [name=vr1-dc0-opnsense]
    module.vr1_dc0_opnsense.libvirt_domain.vm: Destruction complete after 0s
    module.vr1_dc0_opnsense.libvirt_volume.disk: Destroying... [id=/var/lib/libvirt/vr1-dc0-inner/vr1-dc0-opnsense-disk.qcow2]
    module.vr1_dc0_opnsense.libvirt_volume.disk: Destruction complete after 0s
    module.vr1_dc0_opnsense.libvirt_volume.disk: Creating...
    module.vr1_dc0_opnsense.libvirt_volume.disk: Creation complete after 4s
    module.vr1_dc0_opnsense.libvirt_domain.vm: Creating...
    module.vr1_dc0_opnsense.libvirt_domain.vm: Creation complete after 2s
    Apply complete! Resources: 2 added, 0 changed, 2 destroyed.

A 2.2 GB volume "created in 4s" is the kind of number worth distrusting, so the
ARTIFACT was measured rather than the log believed:
  rack: -rw------- libvirt-qemu kvm 2198142976 Aug 2 06:14
          /var/lib/libvirt/vr1-dc0-inner/vr1-dc0-opnsense-disk.qcow2
  (the old disk was 2559705088 B and dated Aug 1 05:51 -- this is a new file of
  base-image size, not the survivor.)

DOMAIN AS BUILT (virsh -c qemu:///system, on the rack):
  Id 4  vr1-dc0-opnsense  running   Autostart: enable   2 vCPU   2048 MiB
  vnet14  network  vr1-dc0-provider-public  virtio  52:54:00:6d:a9:27   (LAN, idx 0)
  vnet15  bridge   vr1-dc0-wan              virtio  52:54:00:ae:b2:89   (WAN, idx 1)
  <channel type='unix'> target org.qemu.guest_agent.0 state='disconnected'
    -- D-127 autostart=true and D-129 expose_qga_channel=true both came back with
       the module, as designed. The channel reads 'disconnected' because the
       os-qemu-guest-agent PACKAGE was never installed on the DC edges (the G13
       retrofit was office1-opnsense only). PRE-EXISTING, not a rebuild regression.
  Edge MACs are NOT pinned by the module (only the nine node VMs are), so both
  edge MACs are new. Nothing keys on them: MAAS knows the nodes, not the edge.

FIRST BOOT -- ASSERTED ON NEW SERIAL CONTENT ONLY
The serial device has append="on", so the file still contains the damaged 08-01
boot. Grepping the whole file for a healthy-boot marker would match the OLD boot
and look exactly like success. Assertions were run on `tail -n +21913` only:

  ld-elf.so.1 .............. 0     (was: present, the failure)
  UNREF FILE ............... 0     (was: 2533)
  UNEXPECTED SOFT UPDATE ... 0     (was: 785)
  SALVAGE? ................. 0     (was: 257)
  libcrypto / libpython .... 0 / 0
  Error in early script .... 0     (was: 10-configd, 15-templates, 90-carp)
  login: ................... 1
  OPNsense 26.7 ............ 1

  ONE apparent hit needed chasing rather than dismissing: "Enter full pathname"
  counted 1 in the "new" region. That is the single-user prompt -- the exact
  symptom being rebuilt away. Cause, measured by reading the first line of the
  region: the OLD log's final line had NO trailing newline, so `wc -l` counted
  21912 while a 21913th partial line existed; `tail -n +21913` therefore begins
  mid-way through the OLD prompt, and the NEW boot starts on the very next
  characters (`/boot/config: -S115200 -h -D`). An off-by-one in my instrument,
  not a fault in the edge. Recorded because the alternative -- assuming it away --
  is the failure mode this repo keeps logging.

  Boot banner (new content):
      *** OPNsense.internal: OPNsense 26.7 (amd64) ***
       LAN (vtnet0)    -> v4: 192.168.1.1/24
       WAN (vtnet1)    ->
  vtnet0 = LAN = provider-public, vtnet1 = WAN -- the same mapping the 2026-07-20
  dc0 build and the 2026-07-23 dc1 build both measured. WAN is address-less
  because `vr1-dc0-uplink` carries an <ip> but NO <dhcp> block (measured at the
  D-125 gate, 2026-07-20): the WAN static is set explicitly below, by design.

================================================================================
SECTION 3 -- D-112(c) CONSOLE BOOTSTRAP (step 3)
================================================================================

TOOLING STAGED FRESH FROM THE REPO, sha256-verified BOTH ENDS before use. The
rack's own copies date from 2026-07-20 and predate the sshd-readiness wait and
scp retry that were added on 2026-07-23; the Jul-20 copies were not used.
  -> vr1-dc0-rack:~/repo-stage/scripts/
    c1bd904489fe6e819463f1c9db483ea054d15586acd05e08fd85285dbbf1b450  opnsense-bootstrap-apikey.sh
    2a85225dc6329524b86986c0c1ab3ccb1863b0fe0fd522fd7f6c88e1acca0938  opnsense-mint-apikey.php
    5dd78f323df0261b4f9bc32cdb8fbf5e7fcc411e58527d49742716ad8a4bb7ef  opnsense-set-interface-v4.sh
    ed3c4784e20046aab09b6041030176301bddb5442e2ff4cbb5911390d8bfdc5a  opnsense-set-iface-v4.php
    8c2329f87fe72a939cfe48b7fece33685ff8f392949af96b490772453b35c905  opnsense-api.sh
  (both PHP helpers staged BESIDE their drivers -- both scripts resolve them
  from their own directory and fail at run time otherwise.)

DRIVER: ~/d112c-console-dc0-rebuild.py on the rack (sha256
6dbf985697815e030bdd29031a75b7142570e92558336a00d5194bdb9f78fa04, byte-identical
to the local copy). It merges the two legs the 2026-07-20 build ran as separate
artifacts, and it does NOT replay ~/d112c-bootstrap.php.b64 -- that artifact is
missing require_once("util.inc") and would have died inside write_config() at
"Call to undefined function shell_safe()" BEFORE writing anything. Confirmed on
the live edge: `grep -c shell_safe /usr/local/etc/inc/util.inc` -> 7.
Payload shipped as 10 base64 chunks of <=160 chars (lesson i: the serial console
wraps long lines).

TRANSCRIPT (~/d112c-console-dc0-rebuild-20260802.log on the rack), key lines:
    root@OPNsense:~ # grep -c authorizedkeys /conf/config.xml
    1
    root@OPNsense:~ # grep -c shell_safe /usr/local/etc/inc/util.inc
    7
    root@OPNsense:~ # php -l /tmp/b.php
    No syntax errors detected in /tmp/b.php
    root@OPNsense:~ # php /tmp/b.php
    CONFIG-WRITTEN
    AK-IN-CONFIG=152
    USER-MATERIALIZED
    root@OPNsense:~ # configctl openssh restart
    OK
  AK-IN-CONFIG=152 matches the dc1 datapoint exactly (152). USER-MATERIALIZED is
  the local_user_set() call -- lesson v: write_config() alone does NOT create
  /root/.ssh/authorized_keys, which is the measured reason key auth "mysteriously"
  fails after a clean config write.

  MINOR, RECORDED NOT FIXED: two diagnostic lines inherited verbatim from the v6
  driver use `2>&1`, and the edge root shell is tcsh, so both printed
  "Ambiguous output redirect." (lesson iii). They are diagnostics only -- the
  bootstrap itself is unaffected -- but the dc1 session made its equivalents
  csh-safe and the dc0 rack copy still has not been. Queued as a finding.

>>> A TRAP THAT WOULD HAVE READ AS A SCRIPT BUG, CAUGHT BY LOOKING FIRST <<<
The rack's ~/.ssh/known_hosts held the DESTROYED edge's host keys for BOTH
192.168.1.1 (3 entries) and 10.12.4.1 (1 entry), from 2026-07-20. The rebuilt
edge generates NEW host keys, so every script here -- all of which use
`-o BatchMode=yes` -- would have failed with REMOTE HOST IDENTIFICATION HAS
CHANGED and no interactive prompt to resolve it. Measured with `ssh-keygen -F`
BEFORE running anything, then:
    cp -p ~/.ssh/known_hosts ~/.ssh/known_hosts.pre-edge-rebuild-20260802
    ssh-keygen -R 192.168.1.1 ; ssh-keygen -R 10.12.4.1
    ssh-keyscan -T 10 192.168.1.1 >> ~/.ssh/known_hosts     (and 10.12.4.1 later)
This belongs in the runbook: a REBUILT edge invalidates the rack's known_hosts,
and the dc0/dc1 first-build records could not have caught it because on a first
build there is no prior entry.

INTERIM REACH LEG (the factory LAN is 192.168.1.0/24 and the rack has no route):
    sudo ip addr add 192.168.1.2/24 dev virbr5      (non-persistent; removed later)

KEY-ONLY SSH PROVEN -- the D-112(c) exit condition:
    ssh -i ~/vr1-dc0-creds/vr1-dc0-edge_ed25519 root@192.168.1.1
      uname -r            -> 15.1-RELEASE-p1
      ifconfig -l         -> vtnet0 vtnet1 lo0 enc0 pfsync0 pflog0
      ifconfig vtnet0     -> inet 192.168.1.1 netmask 0xffffff00
      ifconfig vtnet1     -> (no inet)
      opnsense-version    -> OPNsense 26.7 (amd64)
  NO new keypair was minted: the pubkey in the surviving .b64 was verified
  byte-identical to ~/vr1-dc0-creds/vr1-dc0-edge_ed25519.pub on the jumphost, so
  the EXISTING private key opens the rebuilt edge.

API KEY MINTED (D-113(a2), the vendor model -- no GUI click, no re-implemented crypto)
    cd ~/repo-stage/scripts
    OPNSENSE_SSH_KEY=$HOME/vr1-dc0-creds/vr1-dc0-edge_ed25519 \
      bash opnsense-bootstrap-apikey.sh 192.168.1.1 \
           $HOME/vr1-dc0-creds/opnsense-api-rebuild-20260802.txt
    -> OK: minted for root -- key=80 chars, secret=80 chars, wrote 173 bytes (0600,
       secret NOT printed)
  Output went to a DATED path, not the canonical name, so the script's
  refuse-to-overwrite guard was never worked around: a stale
  ~/vr1-dc0-creds/opnsense-api.txt already sat on the rack (Section 0) and that
  guard exists precisely to stop a live key being stranded. Nothing was deleted to
  make a script run.
  SMOKE TEST: `opnsense-api.sh GET core/firmware/status` -> exit 0,
    {"product":{"CORE_ABI":"26.7","CORE_ARCH":"amd64",...,"CORE_PRODUCT":"OPNsens...
  Third D-113(a2)-on-26.7 datapoint (after dc0 2026-07-20 and dc1 2026-07-23).

CONSOLIDATED PER D-137, verified by SHAPE only -- no secret was ever read:
    ~/vr1-dc0-creds/opnsense-api.txt on the JUMPHOST, 0600, 173 bytes
    key= lines 1, secret= lines 1; key value 80 chars, secret value 80 chars
    sha256 identical to the rack copy (ad59dca4...956378, compared not assumed)
  This materially CHANGES SEC-021(a), which recorded that dc0 had no consolidated
  edge API credential and therefore "no API management path from vcloud". It now
  has one. See Section 6.

================================================================================
SECTION 4 -- ADDRESSING, BOTH LEGS, READ BACK (step 4)
================================================================================

Ordering follows the script's own warning: WAN FIRST, verified, LAN LAST -- the
LAN move relocates the very interface the session arrives on.

WAN (dry run first, then --commit):
    bash opnsense-set-interface-v4.sh --commit 192.168.1.1 wan 172.30.2.2 24 172.30.2.1
    interface 'wan' is device 'vtnet1' (measured from the edge's own config)
    ipaddr: dhcp -> 172.30.2.2 | subnet: (none) -> 24 | gateway: (none) -> WAN_GW (172.30.2.1)
    OK: saved and read back -- wan = 172.30.2.2/24 (3 interfaces intact)
    OK: gateway WAN_GW = 172.30.2.1 is the default route for wan
    ...
    inet 172.30.2.2 netmask 0xffffff00 broadcast 172.30.2.255
    OK: 172.30.2.2/24 is live on wan (vtnet1).
  The pf reload ALSO produced real automatic outbound NAT at this point --
    nat on vtnet1 inet from (vtnet0:network) to any -> (vtnet1:0) port 1024:65535
  which is the appendix-A "addressed perfectly, forwards nothing" defect NOT
  happening. That check is a WARN-only report in the script, so it was read
  deliberately rather than left to scroll past.

EGRESS PROVEN FROM THE EDGE ITSELF, BEFORE THE LAN WAS TOUCHED. Deliberate
ordering: after the LAN moves, the reach path is gone, and a WAN gateway that did
not take is far harder to diagnose from the other side.
    default            172.30.2.1         UGS          vtnet1
    ping 172.30.2.1  -> 3 packets transmitted, 3 received, 0.0% packet loss
    ping 1.1.1.1     -> 3 packets transmitted, 3 received, 0.0% packet loss
    host archive.ubuntu.com -> 91.189.92.22 / 91.189.92.24
    fetch http://archive.ubuntu.com/ubuntu/dists/jammy/Release -> FETCH-OK

LAN (dry run first, then --commit):
    bash opnsense-set-interface-v4.sh --commit 192.168.1.1 lan 10.12.4.1 22
    interface 'lan' is device 'vtnet0' (measured from the edge's own config)
    ipaddr: 192.168.1.1 -> 10.12.4.1 | subnet: 24 -> 22 | gateway: (none) [UNTOUCHED]
    OK: saved and read back -- lan = 10.12.4.1/22 (3 interfaces intact)
    === APPLY === -> the invocation was killed at the 240s timeout (EXIT=124)
  EXPECTED AND DOCUMENTED: `configctl interface reconfigure lan` drops the session
  it is issued over, and the foreground ssh then hangs on the dead interface --
  the identical behaviour the dc1 build recorded ("TaskStop-ped; config was
  already saved+applied"). The config was saved and applied; what the timeout cost
  was the script's own post-drop legs (filter reload + read-back), which were then
  run explicitly over a FRESH connection to the new address:

    from the rack (permanent leg 10.12.4.2/22, no interim leg needed):
      ping 10.12.4.1  -> 3 packets transmitted, 3 received, 0% packet loss
      ip neigh        -> 10.12.4.1 dev virbr5 lladdr 52:54:00:6d:a9:27 DELAY
        (contrast the outage capture's `10.12.4.1 ... FAILED`)
    on the edge:
      ifconfig vtnet0 -> inet 10.12.4.1 netmask 0xfffffc00 broadcast 10.12.7.255   (/22)
      ifconfig vtnet1 -> inet 172.30.2.2 netmask 0xffffff00
      netstat -rn     -> default 172.30.2.1 UGS vtnet1
      configctl filter reload -> OK
      pfctl -s nat (carp boilerplate removed):
        nat on vtnet1 inet from (vtnet0:network) to any -> (vtnet1:0) port 1024:65535
        nat on vtnet1 inet from (lo0:network) to any -> (vtnet1:0) port 1024:65535
        nat on vtnet1 inet from 127.0.0.0/8 to any -> (vtnet1:0) port 1024:65535
      -- (vtnet0:network) is now 10.12.4.0/22, so the rack and everything on
      provider-public NATs out of vtnet1. Read explicitly; not assumed from the
      earlier WAN-stage output, because the LAN network changed underneath it.

  Interim leg removed:  sudo ip addr del 192.168.1.2/24 dev virbr5
  virbr5 now carries only 10.12.4.2/22 -- the dc0 end-state exactly as before the
  outage, and identical in shape to dc1's.

BOTH LEGS, FINAL, READ FROM THE KERNEL:
    WAN  vtnet1  172.30.2.2/24  gw 172.30.2.1   (egress 0% loss)
    LAN  vtnet0  10.12.4.1/22                   (the ruled provider-public gateway)

================================================================================
SECTION 5 -- THE GATE (step 5, the definition of done)
================================================================================

RUN 1 -- against repo HEAD's dc-egress-check.sh, 2026-08-02 ~06:40. FAIL, 7/8:

    == dc-egress-check dc0 ==
       edge LAN (gateway): 10.12.4.1     rack leg: 10.12.4.2
       snap proxy: http://10.12.8.4:3129
      OK     A1 default route via 10.12.4.1 (this site's edge)
      OK     A2 edge 10.12.4.1 ANSWERS
      OK     A3 off-site ICMP anchor 1.1.1.1 answers
      OK     A3 off-site TCP/HTTP anchor answers 200 (archive.ubuntu.com)
      OK     A4 apt upstream (archive.ubuntu.com) reachable (HTTP 200)
      OK     A4 UCA upstream (ubuntu-cloud.archive.canonical.com) reachable (HTTP 200)
      OK     A4 juju agent stream (streams.canonical.com) reachable (HTTP 200)
      FAIL   A4 snap store VIA this site's proxy answered '400' -- unrecognised, not assumed healthy
    RESULT: pass=7 fail=1     exit 1

The gate is LAYERED and reports the FIRST failure as the cause. A1, A2, A3 and
three of four A4 probes were green, so by the gate's own design the edge was NOT
the cause. The failure was ISOLATED rather than reasoned about -- three probes
from the rack:

  1. THROUGH the proxy, no header (exactly what A4 sends):
       http_code=400 curl_exit=0
       {"error-list":[{"code":"bad-argument","message":"Snap-Device-Series header is required."}]}
  2. THROUGH the proxy, WITH `-H 'Snap-Device-Series: 16'`:
       http_code=200 curl_exit=0
       {"channel-map":[{"channel":{"architecture":"amd64","name":"stable",...
  3. DIRECT, no proxy, no header -- the control that separates store from proxy:
       http_code=400 curl_exit=0    (identical error)

  Probe 3 is the decisive one: the 400 is produced with the proxy taken out of
  the path entirely, so it is the STORE rejecting a malformed request, not the
  proxy refusing a tunnel. A 400 in fact PROVES the tunnel end to end -- squid
  admitted the CONNECT, TLS completed to api.snapcraft.io, and the store replied.

  Corroborated by the repo's own snap gate, which sends the header:
    `dc-snap-proxy.sh check dc0` -> PASS, 16/16, including
      OK  CONNECT to the store API: curl exit 0, HTTP 200, channel-map + sha3-384 present
      OK  CDN tunnel allowed (curl exit 0; the CDN's own status was '403')
      OK  non-allowlisted destination is DENIED (403 at CONNECT) -- the allowlist is in force

  So: a DEFECT IN THE GATE, not in the edge. `dc-snap-proxy.sh:215,469` sends
  `Snap-Device-Series: 16`; `dc-egress-check.sh`'s A4 `probe()` did not. The
  assertion had never passed anywhere -- dc0 previously failed at A2 with A4
  SKIPPED, and dc1 has no ruled snap proxy and was failing A3/A4 for other
  reasons -- so A4's snap branch had never once been exercised against a live
  proxy. It was decoration until this run.

  The finding was LOGGED, NOT FIXED (hard rule 1; and a gate change needs a
  failing-direction harness case + a changelog, which would have collided with
  the operator's concurrent work on this branch).

RUN 2 -- 2026-08-02 ~06:55. Between the two runs the operator INDEPENDENTLY
found and fixed the same defect in the working tree (`scripts/dc-egress-check.sh`
mtime 06:53:27, uncommitted), adding `-H 'Snap-Device-Series: 16'` to `probe()`
with a comment citing the same three measurements. Re-run against that:

    $ ssh vr1-dc0-rack 'bash -s -- check dc0' < scripts/dc-egress-check.sh
    == dc-egress-check dc0 ==
       edge LAN (gateway): 10.12.4.1     rack leg: 10.12.4.2
       snap proxy: http://10.12.8.4:3129
      OK     A1 default route via 10.12.4.1 (this site's edge)
      OK     A2 edge 10.12.4.1 ANSWERS
      OK     A3 off-site ICMP anchor 1.1.1.1 answers
      OK     A3 off-site TCP/HTTP anchor answers 200 (archive.ubuntu.com)
      OK     A4 apt upstream (archive.ubuntu.com) reachable (HTTP 200)
      OK     A4 UCA upstream (ubuntu-cloud.archive.canonical.com) reachable (HTTP 200)
      OK     A4 juju agent stream (streams.canonical.com) reachable (HTTP 200)
      OK     A4 snap store VIA this site's proxy reachable (HTTP 200)

    RESULT: pass=8 fail=0
    PASS: dc-egress-check dc0
    GATE-EXIT=0

>>> DEFINITION OF DONE MET: exit 0, 8/8. <<<
The two runs differ ONLY in the gate's own probe. No edge, rack, proxy or network
state changed between them -- the edge was already fully working at run 1, which
is what the A1/A2/A3 + three-upstream greens in run 1 already showed.

CONVERGENCE RE-PLAN (the repo's standing post-apply evidence -- G16, step B,
step C and dc1 step C all cite one):
    $ tofu plan            (on voffice1)
      No changes. Your infrastructure matches the configuration.
    machine-asserted from `tofu show -json`:
      non-no-op resource_changes: 0
      addresses: (empty)
  State matches the built artifact. The 20 "changed outside of OpenTofu" drift
  lines still appear on every plan of this root; they are notes, not actions, and
  the JSON assertion is what distinguishes them.

================================================================================
SECTION 6 -- CREDENTIALS
================================================================================

MINTED : dc0 edge REST API key, via the vendor model (D-113(a2)), on the rebuilt
         edge. Secret never printed, never read into context; verified by
         length/format only.
CONSOLIDATED (D-137): ~/vr1-dc0-creds/opnsense-api.txt on the JUMPHOST, 0600,
         173 B, 1x `key=` (80 chars) + 1x `secret=` (80 chars), sha256-compared
         to the source copy before the source was destroyed.
CLEANED: the dated rack copy `~/vr1-dc0-creds/opnsense-api-rebuild-20260802.txt`
         was `shred -u`'d after consolidation -- a second copy of a LIVE secret
         is an exposure I introduced, so I removed it (dc1 SEC-015 precedent).

REGISTER STATE, run rather than assumed (`python3 scripts/creds-matrix.py --tier2`
on the jumphost, 121 rows, 20 check groups clean, 11 findings):
  - `E1/E3 existence: every expected artifact present and nothing undeclared,
    across 4 fully-probed role(s)` -- CLEAN. Previously CURRENT-STATE recorded
    `E1 EXPECTED-BUT-ABSENT: dc0-edge-api 'opnsense-api.txt'`. The mint closed it.
  - `[FAIL] S2 vr1-dc0 EXPECTED-BUT-ABSENT: 'opnsense-api.txt' (id dc0-edge-api,
    SEC-021)` -- STILL RED, and now red for a DIFFERENT REASON than when the
    operator accepted it on 2026-07-30. Then, the credential did not exist. Now it
    exists and `creds-manifests/vr1-dc0.manifest` simply does not declare it
    (dc1's manifest:25 does declare its equivalent). The manifest is DERIVED from
    the matrix via `creds-matrix.py --render`, so this is a mechanical
    regeneration -- deliberately NOT run here: it rewrites a generated file across
    all three sites while the operator has uncommitted work on this branch.
  - The total (11) is NOT comparable to CURRENT-STATE's 2026-07-30 figure of 6:
    that run had 101 rows, this one 121, the difference being the SEC-027/SEC-028
    dc1 forward-register rows added since. Quoting a delta across two different
    row sets would be a fabricated number.
  - NO test anywhere asserts the literal string `n-dc0-edge-api-absent`
    (grepped `--include=*.py --include=*.sh`); the only occurrences are
    `creds-matrix.tsv:76` and `creds-matrix-notes.md:123`. So nothing turns red
    from the remediation -- but both are now factually STALE and their wording
    needs REPLACING with the new invariant, never deleting.

MATRIX ROW: `dc0-edge-api` ALREADY EXISTS at `creds-matrix.tsv:76`. No duplicate
was added -- the register was already expecting this credential, which is exactly
why its absence was detectable.

LEDGER: SEC-032 added (next-free confirmed by grep at write time; SEC-031 was
taken by the operator's concurrent dc1-edge row). It records the dc0 analog of
SEC-015 -- edge private key + a now-dead API credential resident on the rack since
2026-07-20, undisclosed -- and notes that this ANSWERS SEC-021(a)'s open question
("locate an off-jumphost copy"). Nothing was actioned on it.

================================================================================
SECTION 7 -- WHAT WAS LEFT BEHIND, NAMED
================================================================================

ON THE RACK (vr1-dc0-rack):
  ~/vr1-dc0-creds/vr1-dc0-edge_ed25519      LEFT. Pre-existing since 2026-07-20.
      Structurally required by the current tooling: opnsense-bootstrap-apikey.sh
      and opnsense-set-interface-v4.sh both SSH the edge DIRECT with no
      ProxyJump, and the edge LAN is reachable only from the rack. Disclosed as
      SEC-032; NOT silently kept.
  ~/vr1-dc0-creds/opnsense-api.txt          LEFT. The 2026-07-20 key, now DEAD
      (the config holding its hash was destroyed by this rebuild). Left rather
      than shredded because it is the physical evidence behind SEC-021(a) and the
      operator has not ruled. Named in SEC-032 item (1).
  ~/repo-stage/scripts/opnsense-*  (5 files) LEFT. Non-secret repo scripts,
      sha256-verified against the repo when staged. Same convention as the
      2026-07-30 juju staging already in that directory. They WILL go stale --
      re-verify hashes before any future use rather than trusting them.
  ~/d112c-console-dc0-rebuild.py            LEFT. Non-secret (embeds only the
      PUBLIC key). Kept as the reproducible artifact for the next rebuild.
  ~/d112c-console-dc0-rebuild-20260802.log  LEFT. Non-secret console transcript,
      the dc1 precedent for retaining one as an audit artifact.
  ~/.ssh/known_hosts.pre-edge-rebuild-20260802 and ~/.ssh/known_hosts.old
      LEFT. Public host keys only.
  ~/d112c-bootstrap.php.b64 (2026-07-20)    LEFT, UNUSED, AND DANGEROUS TO REPLAY
      -- it is missing require_once("util.inc"). Section 3.

ON voffice1:
  ~/vr1-dc0-substrate-tfstate-pre-edge-rebuild-20260802.json   the pre-apply state
      backup. Delete once the rebuild is accepted.
  /tmp/edge-rebuild-20260802.plan, .plan.json, /tmp/edge-rebuild-converge-20260802.plan
  /tmp/edge-rebuild.plan -- the STALE plan named in the task brief, NOT used.

REPO TREE (left dirty for the operator to commit, as instructed):
  docs/audit/dc0-edge-rebuild-20260802.txt   this file (new)
  docs/security-ledger.md                    SEC-032 appended
  (scripts/dc-egress-check.sh and tests/dc-egress-check/run-tests.sh were already
   modified by the operator's concurrent session and were NOT touched here.)

================================================================================
SECTION 8 -- FINDINGS RAISED, NONE ACTIONED (hard rule 1)
================================================================================

F1. A REBUILT EDGE INVALIDATES THE RACK'S known_hosts, AND NO RUNBOOK SAYS SO.
    Both edge management scripts use `-o BatchMode=yes`, so a changed host key
    fails with REMOTE HOST IDENTIFICATION HAS CHANGED and no way to answer the
    prompt. Neither the dc0 nor the dc1 build could have caught this: on a FIRST
    build there is no prior entry. This is a REBUILD-only trap and belongs in the
    D-112(c) chain and in appendix-A by symptom.

F2. `dc-egress-check.sh`'s A4 snap probe omitted `Snap-Device-Series: 16` --
    found here, and independently fixed by the operator mid-session. Recorded
    because the CLASS is what matters: the assertion had never been exercised
    (dc0 failed earlier at A2, dc1 has no ruled proxy), so it was decoration on a
    gate that shipped the same day with a 14/14 harness and a mutation pass. A
    harness that mocks curl cannot catch a wrong REAL request. The durable rule:
    when two scripts probe the same endpoint, they must share the probe
    definition -- `dc-snap-proxy.sh:215` already had the correct one.

F3. THE EDGE STILL SERVES DHCP ON THE LAN, WITH A STALE FACTORY RANGE.
    Measured on the rebuilt edge AFTER the LAN move:
      /usr/local/etc/dnsmasq.conf : interface=vtnet0
                                    dhcp-range=tag:vtnet0,192.168.1.100,192.168.1.199,86400
                                    dhcp-range=tag:vtnet0,::1000,::2000,constructor:vtnet0,slaac,64,86400
      sockstat -4 -l              : nobody dnsmasq ... udp4 *:67
    vtnet0 is now 10.12.4.0/22 -- the segment carrying the juju controller
    (10.12.4.5) and the MAAS region VM (10.12.4.6). Nothing in the addressing
    toolchain regenerates dnsmasq.conf: `configctl interface reconfigure` and
    `configctl filter reload` do not touch it, so the range still describes the
    factory LAN. NOT CHANGED (out of scope; the edge has no ruled DHCP role and
    changing it is a config decision).
    WHAT IS MEASURED vs WHAT IS NOT, stated separately:
      MEASURED: dnsmasq is running, bound to udp4 *:67, with a 192.168.1.x range
                on vtnet0, and dhcpd/Kea blocks in config.xml count ZERO.
      NOT MEASURED: whether it would actually answer a DHCP request on the new
                subnet. dnsmasq matches a range to the interface's own subnet and
                192.168.1.0/24 is no longer on vtnet0, which SUGGESTS it is inert
                -- but that is reasoning, not a measurement, and it is flagged as
                such rather than reported as a fact.
      NOT ESTABLISHED: whether the destroyed edge was in the same state. It very
                likely was (it took the identical factory-boot-then-re-address
                path), but it cannot be checked now and dc1 was out of scope for
                this task, so this is NOT claimed as "restored prior state".
    Worth an operator decision before Stage 5 puts nine nodes on that segment.

F4. THE `.b64` BOOTSTRAP ARTIFACT ON THE dc0 RACK IS STILL THE BROKEN ONE.
    DOCFIX-199 warned about it in prose; this session confirmed it against the
    artifact. It remains on the rack, unmarked, next to a working driver. Prose in
    a runbook does not disarm a file on a host -- either delete it or rename it to
    something that cannot be replayed by mistake.

F5. Two diagnostic lines in the console driver still use `2>&1` and print
    "Ambiguous output redirect." on the edge's tcsh. Inherited from the v6
    driver; dc1 fixed its equivalents and the dc0 rack copy was never updated.
    Cosmetic, but it is the documented tcsh trap firing in a live transcript.

F6. `creds-manifests/vr1-dc0.manifest` does not declare `opnsense-api.txt` even
    though the credential now exists and `creds-matrix.tsv:76` expects it. One
    `creds-matrix.py --render` closes it. See Section 6 and SEC-032 item (2).

================================================================================
SECTION 9 -- WHAT WAS *NOT* RESTORED (scope, stated so it is not assumed)
================================================================================

The rebuilt edge is a FACTORY 26.7 nano plus SSH, an API key, and two static
addresses. It does NOT carry:
  - os-qemu-guest-agent. The D-129 qga CHANNEL is present (module input
    expose_qga_channel=true) and the domain XML shows it `state='disconnected'`,
    but the PACKAGE was never installed on either DC edge -- the G13 retrofit was
    office1-opnsense only. PRE-EXISTING, not a regression: the outage capture
    records `qga: "QEMU guest agent is not connected"` on the OLD edge too.
  - any interface-level IPv6. The dc0 edge's v6 config was blocked by SEC-021(a)
    (no API credential) and had never been done. That blocker is now cleared.
  - firmware 26.7.1. The office1 edge was updated to 26.7.1 in the G13 bundle;
    both DC edges were and remain 26.7 (`CORE_ABI 26.7`, measured).
  - any firewall rule beyond the vendor defaults, which is what the previous edge
    also had -- D-113(a2) config is done over REST and none was applied to dc0
    beyond addressing.
Nothing was lost that the old edge had, on the evidence available; but the two
statements "the rebuild restored what was there" and "the rebuild produced a
working edge" are different claims, and only the second one is proven here.
