dc0-edge-rebuild-20260802.txt
=============================
REBUILD of the vr1-dc0 OPNsense edge, destroyed by UFS soft-update damage after
the 2026-08-01 05:48 hard power cut of its containment VM (full diagnosis:
docs/audit/dc0-edge-egress-outage-20260802.txt -- not re-derived here).
Definition of done, set by the operator: `ssh vr1-dc0-rack 'bash -s -- check dc0'
< scripts/dc-egress-check.sh` exits 0.
Written incrementally as each step completed. Every value below was LOOKED UP in a
script constant, a tfvars file, or a live object -- none was inferred. Where a value
in the task brief disagreed with the measured one, the measured one is used and the
divergence is called out.
================================================================================
SECTION 0 -- READ-ONLY BASELINE (before any mutation)
================================================================================
REPO / CLONE IDENTITY
jumphost (vcloud) HEAD : 95a6f5f branch dc-dc-stage5-preconditions
voffice1 clone HEAD : 22af620 (git status clean)
The two clones are at DIFFERENT commits. That matters only if it changes the
tofu root, so it was MEASURED rather than assumed -- sha256 of all seven files
the inner root consumes, both hosts:
c81aa41b093af531f81a97254116f98c524b07a44f2e02240c03876bbbce9ab3 vr1-dc0-substrate/main.tf
f4020e606c1fc4d2a2dbf5430d0d38c518962ca3b4bd1b549de223a7e51250f2 vr1-dc0-substrate/variables.tf
17de7c023fa1353f8f14eebf29b7bc30dd11f0bcfec1112d8b41c83fd1c3c081 vr1-dc0-substrate/versions.tf
cd117906d8f4baf526240e20eda53931746aaff2d2df8fa40bc7f75df30e7229 modules/opnsense-edge/main.tf
7e38c5b88e6f5c83efd25f7487af87857ca37f52b17831cfb0e9e8a12f733712 modules/opnsense-edge/outputs.tf
b027fe2566e1e4aca89ddcade89051ea01b7b7de7470f9a2c61a2fe0cfb0e7f3 modules/opnsense-edge/variables.tf
f11b24565b04f42959706ad305416eced71310386ff78e029a882ac38382ec2d modules/opnsense-edge/versions.tf
IDENTICAL on both hosts. The clone divergence therefore cannot affect this
root, and NO `git pull` was run on voffice1 (the operator is committing on this
branch concurrently; pulling would drag unrelated commits in mid-apply).
INSTRUMENT TRAP CAUGHT AT THE FIRST PROBE -- recorded because it would have
produced a confident wrong negative:
`ssh vr1-dc0-rack 'virsh list --all'` -> EMPTY TABLE, and
`virsh dominfo vr1-dc0-opnsense` -> "error: failed to get domain"
That reads exactly like "the domain is gone". It is not.
`virsh uri` -> qemu:///session
The non-root default URI is the SESSION daemon, which owns nothing. With
`-c qemu:///system` all 12 domains are listed and the edge is Id 1 running.
EVERY virsh call in this capture carries `-c qemu:///system` for that reason.
LIVE STATE, dc0 rack (vvr1-dc0), `virsh -c qemu:///system list --all`:
Id Name State
1 vr1-dc0-opnsense running <- the broken edge (single-user)
2 vr1-dc0-maas-01 running
3 vr1-dc0-juju-01 running
- 9 role nodes shut off
RACK NETWORK FACTS (measured, `ip -4 -o addr`, `ip route`, `virsh net-info`):
provider-public plane = libvirt net vr1-dc0-provider-public = bridge virbr5
rack provider-public leg = 10.12.4.2/22 on virbr5 (permanent, dc-rack-net)
rack default route = default via 10.12.4.1 dev virbr5 <- the edge LAN
metal-admin = virbr2, carrying 10.12.8.2/.3/.4 (rack leg, node-DNS, utility)
WAN plane = libvirt net vr1-dc0-wan = bridge br-vr1-dc0-wan (D-125 bridge-in)
So the rack's A1 precondition for the egress gate is ALREADY satisfied; only the
edge itself is missing.
TOFU ROOT (on voffice1, per D-128 -- inner roots run from the Office1 headend)
path : ~/openstack-caracal-dc-dc/opentofu/vr1-dc0-substrate/
tofu : OpenTofu v1.12.4, provider dmacvicar/libvirt v0.9.8
state : 36 resources, of which the two edge resources are exactly
module.vr1_dc0_opnsense.libvirt_volume.disk
module.vr1_dc0_opnsense.libvirt_domain.vm
>>> CORRECTION TO THE TASK BRIEF -- BASE IMAGE PATH. <<<
The brief cited the base image as
/var/lib/libvirt/vr1-dc0-inner/opnsense-26.7-nano.qcow2
That is the module DEFAULT in variables.tf, and it is NOT the effective value.
`d124-inner.auto.tfvars` on voffice1 OVERRIDES it:
opnsense_base_path = "/home/jessea123/vr1-dc0-images/opnsense-26.7-nano.qcow2"
and variables.tf:27 explains why, in a note dated at the FIRST inner apply:
"Path ON THE EXECUTING HOST (voffice1, D-128 Plane 2) ... the remote
qemu+ssh provider UPLOADS volume content from ITS OWN filesystem -- a path
on vvr1-dc0 fails with 'Could not open URL for upload: failed to stat'."
This is load-bearing: the apply DESTROYS the volume before creating it, so a
base path that resolves only on the rack would have left the edge with no disk
at all. VERIFIED PRESENT ON voffice1 BEFORE ANY MUTATION:
-rw-r--r-- 1 jessea123 jessea123 2192703664 Jul 20 06:13
/home/jessea123/vr1-dc0-images/opnsense-26.7-nano.qcow2
sha256 3981821e3a3c7be49f6d96f405266806a619756d8816f8a9d92e7fdf5476627d
(The rack ALSO holds a copy at the default path -- same filename, staged by the
bootstrap step -- which is exactly why the wrong one is easy to cite.)
BOOTSTRAP MATERIAL ALREADY ON THE RACK (from the 2026-07-20 original build)
~/d112c-console.py 2481 B D-112(c) console driver v6 (materialize leg)
~/d112c-bootstrap.php.b64 992 B the config-write payload
~/scripts/{opnsense-api.sh,opnsense-mint-apikey.php,
opnsense-set-iface-v4.php,opnsense-set-interface-v4.sh} (Jul 20)
~/opnsense-bootstrap-apikey.sh, ~/opnsense-mint-apikey.php (Jul 20)
python3 pexpect 4.9.0 present on the rack.
The `.b64` payload was DECODED and read. Two things confirmed by reading it
rather than trusting the record:
(a) It carries only `require_once("config.inc")` -- NO `util.inc`. That is
exactly the DOCFIX-199 / dc1-lesson-(iv) defect: `write_config()` calls
`shell_safe()`, which lives in util.inc, so this payload dies at
"Call to undefined function shell_safe()" BEFORE writing. The runbook's
own warning ("Any surviving dc0 .b64 artifact on a rack predates this line
and is missing the include -- do not replay it verbatim") is CORRECT and
is now confirmed against the artifact itself.
(b) The public key it installs is
ssh-ed25519 AAAAC3...tzoc vr1-dc0-edge_svc (D-112c per-env)
which is BYTE-IDENTICAL to ~/vr1-dc0-creds/vr1-dc0-edge_ed25519.pub on the
jumphost (SHA256:2Gknnn3FsMtJov7MFBQlAjT6XJAd5EVz4OaCuY2ZLaU). The existing
private key therefore still opens the rebuilt edge -- no new keypair needed.
>>> PRE-EXISTING CREDENTIAL FINDING, FOUND WHILE STAGING (logged, not actioned
beyond what the rebuild requires) -- SEE SECTION 6. <<<
`ls` (never a read) of vr1-dc0-rack:~/vr1-dc0-creds/ shows, dated 2026-07-20:
-rw------- 173 B opnsense-api.txt
-rw------- 432 B vr1-dc0-edge_ed25519 (the PRIVATE half)
SEC-021(a) records that the dc0 edge API credential is ABSENT from its recorded
jumphost location and asks for "a decision between RE-MINT and locating an
off-jumphost copy". THE OFF-JUMPHOST COPY IS THIS ONE. It has been on the rack
for 13 days. The dc1 build shredded its equivalents the same phase and opened
SEC-015 for the transient; the dc0 build never did, and nothing noticed --
`creds-manifests/vm-secret-locations:123` DOES declare
`rack vr1-dc0 ssh vr1-dc0-rack:~/vr1-dc0-creds/*` as an audited location, so
this was visible to a `--remote` sweep and simply had not been reconciled.
CONSEQUENCE FOR THIS TASK: the 173 B key authenticates to a config that this
rebuild DESTROYS, so it is dead material after the apply and must be re-minted
regardless. It also blocks `opnsense-bootstrap-apikey.sh`, which refuses to
overwrite an existing output file (by design -- overwriting would strand a live
key). Handling is in Section 4.
================================================================================
SECTION 1 -- RE-PLAN AND CONTENT ASSERTION (step 1)
================================================================================
The stale plan the brief mentioned (/tmp/edge-rebuild.plan, Aug 2 03:36) was NOT
used. A fresh one was cut. Pre-mutation safety captures first:
voffice1: cp -p terraform.tfstate ~/vr1-dc0-substrate-tfstate-pre-edge-rebuild-20260802.json
sha256 45e9aad95e37cdbf2840508173aa7cea56ddc469fdc922a57f0516742ec77b9c
(identical both files -- the backup is a real copy, verified not assumed)
rack: serial log BASELINE recorded so the post-rebuild boot can be asserted on
NEW content only. The domain's serial device has append="on", so the file
still holds the DAMAGED 08-01 boot; a grep of the whole file for a
clean-boot marker would match the old boot and look exactly like success.
/var/lib/libvirt/vr1/staging/vr1-dc0-opnsense-serial.log
size=477629 lines=21912 mtime=2026-08-01 05:49:39
PLAN (run on voffice1, D-128 Plane 2):
cd ~/openstack-caracal-dc-dc/opentofu/vr1-dc0-substrate/
tofu plan -replace=module.vr1_dc0_opnsense.libvirt_volume.disk \
-replace=module.vr1_dc0_opnsense.libvirt_domain.vm \
-out=/tmp/edge-rebuild-20260802.plan
Plan: 2 to add, 0 to change, 2 to destroy.
ASSERTION ON CONTENT, machine-read from `tofu show -json`, not from prose.
Prose was deliberately not the assertion surface: the refresh reports 20 resources
under "changed outside of OpenTofu", which is a DRIFT NOTE and not an action, and a
grep over prose blurs the two.
$ jq '.resource_changes[] | select(.change.actions != ["no-op"]) | .address + " " + (.change.actions|join(","))'
module.vr1_dc0_opnsense.libvirt_domain.vm ACTIONS=delete,create
module.vr1_dc0_opnsense.libvirt_volume.disk ACTIONS=delete,create
count of non-no-op changes ................................. 2 (expected 2)
count of non-no-op changes NOT naming vr1_dc0_opnsense ..... 0 (required 0)
Both halves were asserted, not just the "nothing extra" half. Asserting only that
every action line names the edge would PASS on a plan that replaced the domain and
left the corrupt qcow2 in place -- which would reattach the damaged disk and make
the whole rebuild a no-op on the actual damage. So the positive half (exactly these
TWO addresses, each delete+create) is asserted as well.
BASE IMAGE THE PLAN WILL ACTUALLY USE, read out of the plan rather than trusted:
$ jq '.resource_changes[] | select(.address|test("libvirt_volume.disk")) | .change.after.create.content.url'
/home/jessea123/vr1-dc0-images/opnsense-26.7-nano.qcow2
-- the voffice1 path, confirmed present at 2192703664 B before the apply. This is
the check that made the brief's cited path worth correcting: the apply destroys
the volume BEFORE creating its replacement, so a source path that resolved only on
the rack would have left the edge with no disk at all and no way back.
VERDICT: plan matches the intent exactly. Proceed to apply.
================================================================================
SECTION 2 -- APPLY (step 2)
================================================================================
$ tofu apply /tmp/edge-rebuild-20260802.plan (on voffice1)
module.vr1_dc0_opnsense.libvirt_domain.vm: Destroying... [name=vr1-dc0-opnsense]
module.vr1_dc0_opnsense.libvirt_domain.vm: Destruction complete after 0s
module.vr1_dc0_opnsense.libvirt_volume.disk: Destroying... [id=/var/lib/libvirt/vr1-dc0-inner/vr1-dc0-opnsense-disk.qcow2]
module.vr1_dc0_opnsense.libvirt_volume.disk: Destruction complete after 0s
module.vr1_dc0_opnsense.libvirt_volume.disk: Creating...
module.vr1_dc0_opnsense.libvirt_volume.disk: Creation complete after 4s
module.vr1_dc0_opnsense.libvirt_domain.vm: Creating...
module.vr1_dc0_opnsense.libvirt_domain.vm: Creation complete after 2s
Apply complete! Resources: 2 added, 0 changed, 2 destroyed.
A 2.2 GB volume "created in 4s" is the kind of number worth distrusting, so the
ARTIFACT was measured rather than the log believed:
rack: -rw------- libvirt-qemu kvm 2198142976 Aug 2 06:14
/var/lib/libvirt/vr1-dc0-inner/vr1-dc0-opnsense-disk.qcow2
(the old disk was 2559705088 B and dated Aug 1 05:51 -- this is a new file of
base-image size, not the survivor.)
DOMAIN AS BUILT (virsh -c qemu:///system, on the rack):
Id 4 vr1-dc0-opnsense running Autostart: enable 2 vCPU 2048 MiB
vnet14 network vr1-dc0-provider-public virtio 52:54:00:6d:a9:27 (LAN, idx 0)
vnet15 bridge vr1-dc0-wan virtio 52:54:00:ae:b2:89 (WAN, idx 1)
<channel type='unix'> target org.qemu.guest_agent.0 state='disconnected'
-- D-127 autostart=true and D-129 expose_qga_channel=true both came back with
the module, as designed. The channel reads 'disconnected' because the
os-qemu-guest-agent PACKAGE was never installed on the DC edges (the G13
retrofit was office1-opnsense only). PRE-EXISTING, not a rebuild regression.
Edge MACs are NOT pinned by the module (only the nine node VMs are), so both
edge MACs are new. Nothing keys on them: MAAS knows the nodes, not the edge.
FIRST BOOT -- ASSERTED ON NEW SERIAL CONTENT ONLY
The serial device has append="on", so the file still contains the damaged 08-01
boot. Grepping the whole file for a healthy-boot marker would match the OLD boot
and look exactly like success. Assertions were run on `tail -n +21913` only:
ld-elf.so.1 .............. 0 (was: present, the failure)
UNREF FILE ............... 0 (was: 2533)
UNEXPECTED SOFT UPDATE ... 0 (was: 785)
SALVAGE? ................. 0 (was: 257)
libcrypto / libpython .... 0 / 0
Error in early script .... 0 (was: 10-configd, 15-templates, 90-carp)
login: ................... 1
OPNsense 26.7 ............ 1
ONE apparent hit needed chasing rather than dismissing: "Enter full pathname"
counted 1 in the "new" region. That is the single-user prompt -- the exact
symptom being rebuilt away. Cause, measured by reading the first line of the
region: the OLD log's final line had NO trailing newline, so `wc -l` counted
21912 while a 21913th partial line existed; `tail -n +21913` therefore begins
mid-way through the OLD prompt, and the NEW boot starts on the very next
characters (`/boot/config: -S115200 -h -D`). An off-by-one in my instrument,
not a fault in the edge. Recorded because the alternative -- assuming it away --
is the failure mode this repo keeps logging.
Boot banner (new content):
*** OPNsense.internal: OPNsense 26.7 (amd64) ***
LAN (vtnet0) -> v4: 192.168.1.1/24
WAN (vtnet1) ->
vtnet0 = LAN = provider-public, vtnet1 = WAN -- the same mapping the 2026-07-20
dc0 build and the 2026-07-23 dc1 build both measured. WAN is address-less
because `vr1-dc0-uplink` carries an <ip> but NO <dhcp> block (measured at the
D-125 gate, 2026-07-20): the WAN static is set explicitly below, by design.
================================================================================
SECTION 3 -- D-112(c) CONSOLE BOOTSTRAP (step 3)
================================================================================
TOOLING STAGED FRESH FROM THE REPO, sha256-verified BOTH ENDS before use. The
rack's own copies date from 2026-07-20 and predate the sshd-readiness wait and
scp retry that were added on 2026-07-23; the Jul-20 copies were not used.
-> vr1-dc0-rack:~/repo-stage/scripts/
c1bd904489fe6e819463f1c9db483ea054d15586acd05e08fd85285dbbf1b450 opnsense-bootstrap-apikey.sh
2a85225dc6329524b86986c0c1ab3ccb1863b0fe0fd522fd7f6c88e1acca0938 opnsense-mint-apikey.php
5dd78f323df0261b4f9bc32cdb8fbf5e7fcc411e58527d49742716ad8a4bb7ef opnsense-set-interface-v4.sh
ed3c4784e20046aab09b6041030176301bddb5442e2ff4cbb5911390d8bfdc5a opnsense-set-iface-v4.php
8c2329f87fe72a939cfe48b7fece33685ff8f392949af96b490772453b35c905 opnsense-api.sh
(both PHP helpers staged BESIDE their drivers -- both scripts resolve them
from their own directory and fail at run time otherwise.)
DRIVER: ~/d112c-console-dc0-rebuild.py on the rack (sha256
6dbf985697815e030bdd29031a75b7142570e92558336a00d5194bdb9f78fa04, byte-identical
to the local copy). It merges the two legs the 2026-07-20 build ran as separate
artifacts, and it does NOT replay ~/d112c-bootstrap.php.b64 -- that artifact is
missing require_once("util.inc") and would have died inside write_config() at
"Call to undefined function shell_safe()" BEFORE writing anything. Confirmed on
the live edge: `grep -c shell_safe /usr/local/etc/inc/util.inc` -> 7.
Payload shipped as 10 base64 chunks of <=160 chars (lesson i: the serial console
wraps long lines).
TRANSCRIPT (~/d112c-console-dc0-rebuild-20260802.log on the rack), key lines:
root@OPNsense:~ # grep -c authorizedkeys /conf/config.xml
1
root@OPNsense:~ # grep -c shell_safe /usr/local/etc/inc/util.inc
7
root@OPNsense:~ # php -l /tmp/b.php
No syntax errors detected in /tmp/b.php
root@OPNsense:~ # php /tmp/b.php
CONFIG-WRITTEN
AK-IN-CONFIG=152
USER-MATERIALIZED
root@OPNsense:~ # configctl openssh restart
OK
AK-IN-CONFIG=152 matches the dc1 datapoint exactly (152). USER-MATERIALIZED is
the local_user_set() call -- lesson v: write_config() alone does NOT create
/root/.ssh/authorized_keys, which is the measured reason key auth "mysteriously"
fails after a clean config write.
MINOR, RECORDED NOT FIXED: two diagnostic lines inherited verbatim from the v6
driver use `2>&1`, and the edge root shell is tcsh, so both printed
"Ambiguous output redirect." (lesson iii). They are diagnostics only -- the
bootstrap itself is unaffected -- but the dc1 session made its equivalents
csh-safe and the dc0 rack copy still has not been. Queued as a finding.
>>> A TRAP THAT WOULD HAVE READ AS A SCRIPT BUG, CAUGHT BY LOOKING FIRST <<<
The rack's ~/.ssh/known_hosts held the DESTROYED edge's host keys for BOTH
192.168.1.1 (3 entries) and 10.12.4.1 (1 entry), from 2026-07-20. The rebuilt
edge generates NEW host keys, so every script here -- all of which use
`-o BatchMode=yes` -- would have failed with REMOTE HOST IDENTIFICATION HAS
CHANGED and no interactive prompt to resolve it. Measured with `ssh-keygen -F`
BEFORE running anything, then:
cp -p ~/.ssh/known_hosts ~/.ssh/known_hosts.pre-edge-rebuild-20260802
ssh-keygen -R 192.168.1.1 ; ssh-keygen -R 10.12.4.1
ssh-keyscan -T 10 192.168.1.1 >> ~/.ssh/known_hosts (and 10.12.4.1 later)
This belongs in the runbook: a REBUILT edge invalidates the rack's known_hosts,
and the dc0/dc1 first-build records could not have caught it because on a first
build there is no prior entry.
INTERIM REACH LEG (the factory LAN is 192.168.1.0/24 and the rack has no route):
sudo ip addr add 192.168.1.2/24 dev virbr5 (non-persistent; removed later)
KEY-ONLY SSH PROVEN -- the D-112(c) exit condition:
ssh -i ~/vr1-dc0-creds/vr1-dc0-edge_ed25519 root@192.168.1.1
uname -r -> 15.1-RELEASE-p1
ifconfig -l -> vtnet0 vtnet1 lo0 enc0 pfsync0 pflog0
ifconfig vtnet0 -> inet 192.168.1.1 netmask 0xffffff00
ifconfig vtnet1 -> (no inet)
opnsense-version -> OPNsense 26.7 (amd64)
NO new keypair was minted: the pubkey in the surviving .b64 was verified
byte-identical to ~/vr1-dc0-creds/vr1-dc0-edge_ed25519.pub on the jumphost, so
the EXISTING private key opens the rebuilt edge.
API KEY MINTED (D-113(a2), the vendor model -- no GUI click, no re-implemented crypto)
cd ~/repo-stage/scripts
OPNSENSE_SSH_KEY=$HOME/vr1-dc0-creds/vr1-dc0-edge_ed25519 \
bash opnsense-bootstrap-apikey.sh 192.168.1.1 \
$HOME/vr1-dc0-creds/opnsense-api-rebuild-20260802.txt
-> OK: minted for root -- key=80 chars, secret=80 chars, wrote 173 bytes (0600,
secret NOT printed)
Output went to a DATED path, not the canonical name, so the script's
refuse-to-overwrite guard was never worked around: a stale
~/vr1-dc0-creds/opnsense-api.txt already sat on the rack (Section 0) and that
guard exists precisely to stop a live key being stranded. Nothing was deleted to
make a script run.
SMOKE TEST: `opnsense-api.sh GET core/firmware/status` -> exit 0,
{"product":{"CORE_ABI":"26.7","CORE_ARCH":"amd64",...,"CORE_PRODUCT":"OPNsens...
Third D-113(a2)-on-26.7 datapoint (after dc0 2026-07-20 and dc1 2026-07-23).
CONSOLIDATED PER D-137, verified by SHAPE only -- no secret was ever read:
~/vr1-dc0-creds/opnsense-api.txt on the JUMPHOST, 0600, 173 bytes
key= lines 1, secret= lines 1; key value 80 chars, secret value 80 chars
sha256 identical to the rack copy (ad59dca4...956378, compared not assumed)
This materially CHANGES SEC-021(a), which recorded that dc0 had no consolidated
edge API credential and therefore "no API management path from vcloud". It now
has one. See Section 6.
================================================================================
SECTION 4 -- ADDRESSING, BOTH LEGS, READ BACK (step 4)
================================================================================
Ordering follows the script's own warning: WAN FIRST, verified, LAN LAST -- the
LAN move relocates the very interface the session arrives on.
WAN (dry run first, then --commit):
bash opnsense-set-interface-v4.sh --commit 192.168.1.1 wan 172.30.2.2 24 172.30.2.1
interface 'wan' is device 'vtnet1' (measured from the edge's own config)
ipaddr: dhcp -> 172.30.2.2 | subnet: (none) -> 24 | gateway: (none) -> WAN_GW (172.30.2.1)
OK: saved and read back -- wan = 172.30.2.2/24 (3 interfaces intact)
OK: gateway WAN_GW = 172.30.2.1 is the default route for wan
...
inet 172.30.2.2 netmask 0xffffff00 broadcast 172.30.2.255
OK: 172.30.2.2/24 is live on wan (vtnet1).
The pf reload ALSO produced real automatic outbound NAT at this point --
nat on vtnet1 inet from (vtnet0:network) to any -> (vtnet1:0) port 1024:65535
which is the appendix-A "addressed perfectly, forwards nothing" defect NOT
happening. That check is a WARN-only report in the script, so it was read
deliberately rather than left to scroll past.
EGRESS PROVEN FROM THE EDGE ITSELF, BEFORE THE LAN WAS TOUCHED. Deliberate
ordering: after the LAN moves, the reach path is gone, and a WAN gateway that did
not take is far harder to diagnose from the other side.
default 172.30.2.1 UGS vtnet1
ping 172.30.2.1 -> 3 packets transmitted, 3 received, 0.0% packet loss
ping 1.1.1.1 -> 3 packets transmitted, 3 received, 0.0% packet loss
host archive.ubuntu.com -> 91.189.92.22 / 91.189.92.24
fetch http://archive.ubuntu.com/ubuntu/dists/jammy/Release -> FETCH-OK
LAN (dry run first, then --commit):
bash opnsense-set-interface-v4.sh --commit 192.168.1.1 lan 10.12.4.1 22
interface 'lan' is device 'vtnet0' (measured from the edge's own config)
ipaddr: 192.168.1.1 -> 10.12.4.1 | subnet: 24 -> 22 | gateway: (none) [UNTOUCHED]
OK: saved and read back -- lan = 10.12.4.1/22 (3 interfaces intact)
=== APPLY === -> the invocation was killed at the 240s timeout (EXIT=124)
EXPECTED AND DOCUMENTED: `configctl interface reconfigure lan` drops the session
it is issued over, and the foreground ssh then hangs on the dead interface --
the identical behaviour the dc1 build recorded ("TaskStop-ped; config was
already saved+applied"). The config was saved and applied; what the timeout cost
was the script's own post-drop legs (filter reload + read-back), which were then
run explicitly over a FRESH connection to the new address:
from the rack (permanent leg 10.12.4.2/22, no interim leg needed):
ping 10.12.4.1 -> 3 packets transmitted, 3 received, 0% packet loss
ip neigh -> 10.12.4.1 dev virbr5 lladdr 52:54:00:6d:a9:27 DELAY
(contrast the outage capture's `10.12.4.1 ... FAILED`)
on the edge:
ifconfig vtnet0 -> inet 10.12.4.1 netmask 0xfffffc00 broadcast 10.12.7.255 (/22)
ifconfig vtnet1 -> inet 172.30.2.2 netmask 0xffffff00
netstat -rn -> default 172.30.2.1 UGS vtnet1
configctl filter reload -> OK
pfctl -s nat (carp boilerplate removed):
nat on vtnet1 inet from (vtnet0:network) to any -> (vtnet1:0) port 1024:65535
nat on vtnet1 inet from (lo0:network) to any -> (vtnet1:0) port 1024:65535
nat on vtnet1 inet from 127.0.0.0/8 to any -> (vtnet1:0) port 1024:65535
-- (vtnet0:network) is now 10.12.4.0/22, so the rack and everything on
provider-public NATs out of vtnet1. Read explicitly; not assumed from the
earlier WAN-stage output, because the LAN network changed underneath it.
Interim leg removed: sudo ip addr del 192.168.1.2/24 dev virbr5
virbr5 now carries only 10.12.4.2/22 -- the dc0 end-state exactly as before the
outage, and identical in shape to dc1's.
BOTH LEGS, FINAL, READ FROM THE KERNEL:
WAN vtnet1 172.30.2.2/24 gw 172.30.2.1 (egress 0% loss)
LAN vtnet0 10.12.4.1/22 (the ruled provider-public gateway)
================================================================================
SECTION 5 -- THE GATE (step 5, the definition of done)
================================================================================
RUN 1 -- against repo HEAD's dc-egress-check.sh, 2026-08-02 ~06:40. FAIL, 7/8:
== dc-egress-check dc0 ==
edge LAN (gateway): 10.12.4.1 rack leg: 10.12.4.2
snap proxy: http://10.12.8.4:3129
OK A1 default route via 10.12.4.1 (this site's edge)
OK A2 edge 10.12.4.1 ANSWERS
OK A3 off-site ICMP anchor 1.1.1.1 answers
OK A3 off-site TCP/HTTP anchor answers 200 (archive.ubuntu.com)
OK A4 apt upstream (archive.ubuntu.com) reachable (HTTP 200)
OK A4 UCA upstream (ubuntu-cloud.archive.canonical.com) reachable (HTTP 200)
OK A4 juju agent stream (streams.canonical.com) reachable (HTTP 200)
FAIL A4 snap store VIA this site's proxy answered '400' -- unrecognised, not assumed healthy
RESULT: pass=7 fail=1 exit 1
The gate is LAYERED and reports the FIRST failure as the cause. A1, A2, A3 and
three of four A4 probes were green, so by the gate's own design the edge was NOT
the cause. The failure was ISOLATED rather than reasoned about -- three probes
from the rack:
1. THROUGH the proxy, no header (exactly what A4 sends):
http_code=400 curl_exit=0
{"error-list":[{"code":"bad-argument","message":"Snap-Device-Series header is required."}]}
2. THROUGH the proxy, WITH `-H 'Snap-Device-Series: 16'`:
http_code=200 curl_exit=0
{"channel-map":[{"channel":{"architecture":"amd64","name":"stable",...
3. DIRECT, no proxy, no header -- the control that separates store from proxy:
http_code=400 curl_exit=0 (identical error)
Probe 3 is the decisive one: the 400 is produced with the proxy taken out of
the path entirely, so it is the STORE rejecting a malformed request, not the
proxy refusing a tunnel. A 400 in fact PROVES the tunnel end to end -- squid
admitted the CONNECT, TLS completed to api.snapcraft.io, and the store replied.
Corroborated by the repo's own snap gate, which sends the header:
`dc-snap-proxy.sh check dc0` -> PASS, 16/16, including
OK CONNECT to the store API: curl exit 0, HTTP 200, channel-map + sha3-384 present
OK CDN tunnel allowed (curl exit 0; the CDN's own status was '403')
OK non-allowlisted destination is DENIED (403 at CONNECT) -- the allowlist is in force
So: a DEFECT IN THE GATE, not in the edge. `dc-snap-proxy.sh:215,469` sends
`Snap-Device-Series: 16`; `dc-egress-check.sh`'s A4 `probe()` did not. The
assertion had never passed anywhere -- dc0 previously failed at A2 with A4
SKIPPED, and dc1 has no ruled snap proxy and was failing A3/A4 for other
reasons -- so A4's snap branch had never once been exercised against a live
proxy. It was decoration until this run.
The finding was LOGGED, NOT FIXED (hard rule 1; and a gate change needs a
failing-direction harness case + a changelog, which would have collided with
the operator's concurrent work on this branch).
RUN 2 -- 2026-08-02 ~06:55. Between the two runs the operator INDEPENDENTLY
found and fixed the same defect in the working tree (`scripts/dc-egress-check.sh`
mtime 06:53:27, uncommitted), adding `-H 'Snap-Device-Series: 16'` to `probe()`
with a comment citing the same three measurements. Re-run against that:
$ ssh vr1-dc0-rack 'bash -s -- check dc0' < scripts/dc-egress-check.sh
== dc-egress-check dc0 ==
edge LAN (gateway): 10.12.4.1 rack leg: 10.12.4.2
snap proxy: http://10.12.8.4:3129
OK A1 default route via 10.12.4.1 (this site's edge)
OK A2 edge 10.12.4.1 ANSWERS
OK A3 off-site ICMP anchor 1.1.1.1 answers
OK A3 off-site TCP/HTTP anchor answers 200 (archive.ubuntu.com)
OK A4 apt upstream (archive.ubuntu.com) reachable (HTTP 200)
OK A4 UCA upstream (ubuntu-cloud.archive.canonical.com) reachable (HTTP 200)
OK A4 juju agent stream (streams.canonical.com) reachable (HTTP 200)
OK A4 snap store VIA this site's proxy reachable (HTTP 200)
RESULT: pass=8 fail=0
PASS: dc-egress-check dc0
GATE-EXIT=0
>>> DEFINITION OF DONE MET: exit 0, 8/8. <<<
The two runs differ ONLY in the gate's own probe. No edge, rack, proxy or network
state changed between them -- the edge was already fully working at run 1, which
is what the A1/A2/A3 + three-upstream greens in run 1 already showed.
CONVERGENCE RE-PLAN (the repo's standing post-apply evidence -- G16, step B,
step C and dc1 step C all cite one):
$ tofu plan (on voffice1)
No changes. Your infrastructure matches the configuration.
machine-asserted from `tofu show -json`:
non-no-op resource_changes: 0
addresses: (empty)
State matches the built artifact. The 20 "changed outside of OpenTofu" drift
lines still appear on every plan of this root; they are notes, not actions, and
the JSON assertion is what distinguishes them.
================================================================================
SECTION 6 -- CREDENTIALS
================================================================================
MINTED : dc0 edge REST API key, via the vendor model (D-113(a2)), on the rebuilt
edge. Secret never printed, never read into context; verified by
length/format only.
CONSOLIDATED (D-137): ~/vr1-dc0-creds/opnsense-api.txt on the JUMPHOST, 0600,
173 B, 1x `key=` (80 chars) + 1x `secret=` (80 chars), sha256-compared
to the source copy before the source was destroyed.
CLEANED: the dated rack copy `~/vr1-dc0-creds/opnsense-api-rebuild-20260802.txt`
was `shred -u`'d after consolidation -- a second copy of a LIVE secret
is an exposure I introduced, so I removed it (dc1 SEC-015 precedent).
REGISTER STATE, run rather than assumed (`python3 scripts/creds-matrix.py --tier2`
on the jumphost, 121 rows, 20 check groups clean, 11 findings):
- `E1/E3 existence: every expected artifact present and nothing undeclared,
across 4 fully-probed role(s)` -- CLEAN. Previously CURRENT-STATE recorded
`E1 EXPECTED-BUT-ABSENT: dc0-edge-api 'opnsense-api.txt'`. The mint closed it.
- `[FAIL] S2 vr1-dc0 EXPECTED-BUT-ABSENT: 'opnsense-api.txt' (id dc0-edge-api,
SEC-021)` -- STILL RED, and now red for a DIFFERENT REASON than when the
operator accepted it on 2026-07-30. Then, the credential did not exist. Now it
exists and `creds-manifests/vr1-dc0.manifest` simply does not declare it
(dc1's manifest:25 does declare its equivalent). The manifest is DERIVED from
the matrix via `creds-matrix.py --render`, so this is a mechanical
regeneration -- deliberately NOT run here: it rewrites a generated file across
all three sites while the operator has uncommitted work on this branch.
- The total (11) is NOT comparable to CURRENT-STATE's 2026-07-30 figure of 6:
that run had 101 rows, this one 121, the difference being the SEC-027/SEC-028
dc1 forward-register rows added since. Quoting a delta across two different
row sets would be a fabricated number.
- NO test anywhere asserts the literal string `n-dc0-edge-api-absent`
(grepped `--include=*.py --include=*.sh`); the only occurrences are
`creds-matrix.tsv:76` and `creds-matrix-notes.md:123`. So nothing turns red
from the remediation -- but both are now factually STALE and their wording
needs REPLACING with the new invariant, never deleting.
MATRIX ROW: `dc0-edge-api` ALREADY EXISTS at `creds-matrix.tsv:76`. No duplicate
was added -- the register was already expecting this credential, which is exactly
why its absence was detectable.
LEDGER: SEC-032 added (next-free confirmed by grep at write time; SEC-031 was
taken by the operator's concurrent dc1-edge row). It records the dc0 analog of
SEC-015 -- edge private key + a now-dead API credential resident on the rack since
2026-07-20, undisclosed -- and notes that this ANSWERS SEC-021(a)'s open question
("locate an off-jumphost copy"). Nothing was actioned on it.
================================================================================
SECTION 7 -- WHAT WAS LEFT BEHIND, NAMED
================================================================================
ON THE RACK (vr1-dc0-rack):
~/vr1-dc0-creds/vr1-dc0-edge_ed25519 LEFT. Pre-existing since 2026-07-20.
Structurally required by the current tooling: opnsense-bootstrap-apikey.sh
and opnsense-set-interface-v4.sh both SSH the edge DIRECT with no
ProxyJump, and the edge LAN is reachable only from the rack. Disclosed as
SEC-032; NOT silently kept.
~/vr1-dc0-creds/opnsense-api.txt LEFT. The 2026-07-20 key, now DEAD
(the config holding its hash was destroyed by this rebuild). Left rather
than shredded because it is the physical evidence behind SEC-021(a) and the
operator has not ruled. Named in SEC-032 item (1).
~/repo-stage/scripts/opnsense-* (5 files) LEFT. Non-secret repo scripts,
sha256-verified against the repo when staged. Same convention as the
2026-07-30 juju staging already in that directory. They WILL go stale --
re-verify hashes before any future use rather than trusting them.
~/d112c-console-dc0-rebuild.py LEFT. Non-secret (embeds only the
PUBLIC key). Kept as the reproducible artifact for the next rebuild.
~/d112c-console-dc0-rebuild-20260802.log LEFT. Non-secret console transcript,
the dc1 precedent for retaining one as an audit artifact.
~/.ssh/known_hosts.pre-edge-rebuild-20260802 and ~/.ssh/known_hosts.old
LEFT. Public host keys only.
~/d112c-bootstrap.php.b64 (2026-07-20) LEFT, UNUSED, AND DANGEROUS TO REPLAY
-- it is missing require_once("util.inc"). Section 3.
ON voffice1:
~/vr1-dc0-substrate-tfstate-pre-edge-rebuild-20260802.json the pre-apply state
backup. Delete once the rebuild is accepted.
/tmp/edge-rebuild-20260802.plan, .plan.json, /tmp/edge-rebuild-converge-20260802.plan
/tmp/edge-rebuild.plan -- the STALE plan named in the task brief, NOT used.
REPO TREE (left dirty for the operator to commit, as instructed):
docs/audit/dc0-edge-rebuild-20260802.txt this file (new)
docs/security-ledger.md SEC-032 appended
(scripts/dc-egress-check.sh and tests/dc-egress-check/run-tests.sh were already
modified by the operator's concurrent session and were NOT touched here.)
================================================================================
SECTION 8 -- FINDINGS RAISED, NONE ACTIONED (hard rule 1)
================================================================================
F1. A REBUILT EDGE INVALIDATES THE RACK'S known_hosts, AND NO RUNBOOK SAYS SO.
Both edge management scripts use `-o BatchMode=yes`, so a changed host key
fails with REMOTE HOST IDENTIFICATION HAS CHANGED and no way to answer the
prompt. Neither the dc0 nor the dc1 build could have caught this: on a FIRST
build there is no prior entry. This is a REBUILD-only trap and belongs in the
D-112(c) chain and in appendix-A by symptom.
F2. `dc-egress-check.sh`'s A4 snap probe omitted `Snap-Device-Series: 16` --
found here, and independently fixed by the operator mid-session. Recorded
because the CLASS is what matters: the assertion had never been exercised
(dc0 failed earlier at A2, dc1 has no ruled proxy), so it was decoration on a
gate that shipped the same day with a 14/14 harness and a mutation pass. A
harness that mocks curl cannot catch a wrong REAL request. The durable rule:
when two scripts probe the same endpoint, they must share the probe
definition -- `dc-snap-proxy.sh:215` already had the correct one.
F3. THE EDGE STILL SERVES DHCP ON THE LAN, WITH A STALE FACTORY RANGE.
Measured on the rebuilt edge AFTER the LAN move:
/usr/local/etc/dnsmasq.conf : interface=vtnet0
dhcp-range=tag:vtnet0,192.168.1.100,192.168.1.199,86400
dhcp-range=tag:vtnet0,::1000,::2000,constructor:vtnet0,slaac,64,86400
sockstat -4 -l : nobody dnsmasq ... udp4 *:67
vtnet0 is now 10.12.4.0/22 -- the segment carrying the juju controller
(10.12.4.5) and the MAAS region VM (10.12.4.6). Nothing in the addressing
toolchain regenerates dnsmasq.conf: `configctl interface reconfigure` and
`configctl filter reload` do not touch it, so the range still describes the
factory LAN. NOT CHANGED (out of scope; the edge has no ruled DHCP role and
changing it is a config decision).
WHAT IS MEASURED vs WHAT IS NOT, stated separately:
MEASURED: dnsmasq is running, bound to udp4 *:67, with a 192.168.1.x range
on vtnet0, and dhcpd/Kea blocks in config.xml count ZERO.
NOT MEASURED: whether it would actually answer a DHCP request on the new
subnet. dnsmasq matches a range to the interface's own subnet and
192.168.1.0/24 is no longer on vtnet0, which SUGGESTS it is inert
-- but that is reasoning, not a measurement, and it is flagged as
such rather than reported as a fact.
NOT ESTABLISHED: whether the destroyed edge was in the same state. It very
likely was (it took the identical factory-boot-then-re-address
path), but it cannot be checked now and dc1 was out of scope for
this task, so this is NOT claimed as "restored prior state".
Worth an operator decision before Stage 5 puts nine nodes on that segment.
F4. THE `.b64` BOOTSTRAP ARTIFACT ON THE dc0 RACK IS STILL THE BROKEN ONE.
DOCFIX-199 warned about it in prose; this session confirmed it against the
artifact. It remains on the rack, unmarked, next to a working driver. Prose in
a runbook does not disarm a file on a host -- either delete it or rename it to
something that cannot be replayed by mistake.
F5. Two diagnostic lines in the console driver still use `2>&1` and print
"Ambiguous output redirect." on the edge's tcsh. Inherited from the v6
driver; dc1 fixed its equivalents and the dc0 rack copy was never updated.
Cosmetic, but it is the documented tcsh trap firing in a live transcript.
F6. `creds-manifests/vr1-dc0.manifest` does not declare `opnsense-api.txt` even
though the credential now exists and `creds-matrix.tsv:76` expects it. One
`creds-matrix.py --render` closes it. See Section 6 and SEC-032 item (2).
================================================================================
SECTION 9 -- WHAT WAS *NOT* RESTORED (scope, stated so it is not assumed)
================================================================================
The rebuilt edge is a FACTORY 26.7 nano plus SSH, an API key, and two static
addresses. It does NOT carry:
- os-qemu-guest-agent. The D-129 qga CHANNEL is present (module input
expose_qga_channel=true) and the domain XML shows it `state='disconnected'`,
but the PACKAGE was never installed on either DC edge -- the G13 retrofit was
office1-opnsense only. PRE-EXISTING, not a regression: the outage capture
records `qga: "QEMU guest agent is not connected"` on the OLD edge too.
- any interface-level IPv6. The dc0 edge's v6 config was blocked by SEC-021(a)
(no API credential) and had never been done. That blocker is now cleared.
- firmware 26.7.1. The office1 edge was updated to 26.7.1 in the G13 bundle;
both DC edges were and remain 26.7 (`CORE_ABI 26.7`, measured).
- any firewall rule beyond the vendor defaults, which is what the previous edge
also had -- D-113(a2) config is done over REST and none was applied to dc0
beyond addressing.
Nothing was lost that the old edge had, on the evidence available; but the two
statements "the rebuild restored what was there" and "the rebuild produced a
working edge" are different claims, and only the second one is proven here.
