Runbook: DC OPNsense edge REBUILD (filesystem / power-cut damage)

Use when a DC edge forwards but does not NAT, or its GUI answers on the WAN, or dc-egress-check <dc> fails A3/A4 while the edge answers A2 -- i.e. the edge lost its pf ruleset (typically fsck damage to /etc/master.passwd + /etc/group after an unclean power cut; the SEC-031 class). Verdict is REBUILD, not in-place repair: a fresh pristine disk + this bootstrap is faster and more reliable than single-user user-DB surgery, and the edge config is fully recreated by the addressing step below.

Proven twice: dc0 2026-08-02 (docs/audit/dc0-edge-rebuild-20260802.txt, the worked reference) and dc1 2026-08-07 (docs/changelog-20260807-dc1-region-sequence.md Item 4). Both ended dc-egress-check <dc> 8/8. This runbook GENERALISES those; it exists because the dc0 run lived only as an audit capture, so dc1 had to reconstruct it.

Tools (all in-repo -- do NOT reinvent): scripts/opnsense-console-rebuild.py (site-agnostic D-112(c) console driver), scripts/opnsense-bootstrap-apikey.sh, scripts/opnsense-api.sh, scripts/opnsense-set-interface-v4.sh, scripts/dc-egress-check.sh.

Per-DC values (fill in, measured -- never inferred)

All virsh on the rack takes -c qemu:///system (rack default is qemu:///session).

Step 1 -- SAFETY PREP (read-only + backups)

1. Rollback copy of the edge disk (no backingStore = no other rollback point): virsh -c qemu:///system shutdown $DOMAIN (ACPI works even with a broken login); wait for shut off; then on the rack sudo cp -p <edge-disk>.qcow2 <edge-disk>.qcow2.pre-rebuild-<date>.
2. Backup the inner tfstate on voffice1: cp -p terraform.tfstate ~/<dc>-substrate-tfstate-pre-edge-rebuild-<date>.json + sha256-verify identical.
3. Cut a TARGETED plan and machine-assert it is edge-only (this is the gate that protects the 9 node VMs sharing this state):

   cd "$REPO"/opentofu/<dc>-substrate/          # $REPO set per session bootstrap
   tofu plan -replace=module.<dc>_opnsense.libvirt_volume.disk \
             -replace=module.<dc>_opnsense.libvirt_domain.vm -out=/tmp/edge-rebuild.plan
   tofu show -json /tmp/edge-rebuild.plan | jq -r '.resource_changes[] | select(.change.actions!=["no-op"]) | .address+" "+(.change.actions|join(","))'

   REQUIRE: exactly 2 non-no-op changes, BOTH the edge domain+disk, each delete,create; 0 non-no-op changes naming anything else. (The refresh reports ~20 resources "changed outside OpenTofu" -- that is DRIFT, not an action; assert on ACTIONS only.)
4. Confirm the base image the plan will use is PRESENT on voffice1 (the apply destroys the volume before recreating -- a missing source strands the edge with no disk).

Step 2 -- REBUILD (destroy+recreate the edge, edge-only)

tofu apply /tmp/edge-rebuild.plan -> 2 added, 0 changed, 2 destroyed. Fresh disk is base-image size / new mtime (measure the artifact, don't trust the log). Domain boots factory: LAN 192.168.1.1/24, WAN address-less, root/opnsense login.

Step 3 -- CONSOLE BOOTSTRAP (enables ssh + installs the edge key)

Runs ON the rack (has pexpect). Stage the driver + the site edge PUBKEY (public):

scp scripts/opnsense-console-rebuild.py <edge-pubkey>.pub -> rack
ssh $RACK 'python3 ~/opnsense-console-rebuild.py --domain '$DOMAIN' --pubkey ~/<edge>.pub'

Expect CONSOLE-STATE=0 then BOOTSTRAP-SEQUENCE-COMPLETE. Verify the transcript shows CONFIG-WRITTEN, AK-IN-CONFIG=152, USER-MATERIALIZED, OK (sshd restart).

Step 4 -- REACH + KEY-ONLY SSH (D-112(c) exit condition)

A REBUILT edge has NEW host keys -- clear the rack's stale entries or every BatchMode script fails with HOST IDENTIFICATION CHANGED:

ssh $RACK 'cp -p ~/.ssh/known_hosts ~/.ssh/known_hosts.pre-rebuild-<date>;
           ssh-keygen -R 192.168.1.1; ssh-keygen -R $LAN_IP;
           sudo ip addr add 192.168.1.2/24 dev <rack-provider-public-bridge>;   # interim reach to factory LAN
           ssh-keyscan -T8 192.168.1.1 >> ~/.ssh/known_hosts'

Stage the edge PRIVATE key to the rack transiently (0600, SEC-015 pattern -- SHRED in Step 7) OR reach the edge from the jumphost via ProxyCommand. Prove: ssh -i <edge-key> root@192.168.1.1 opnsense-version -> OPNsense 26.7.

Step 5 -- API KEY + ADDRESSING (WAN first, LAN last)

cd ~/repo-stage/scripts   # sha256-verify the 5 opnsense-*.sh/php against the repo first
OPNSENSE_SSH_KEY=<edge-key> bash opnsense-bootstrap-apikey.sh 192.168.1.1 <api-out>.txt
# WAN (also REGENERATES the pf ruleset -> automatic outbound NAT appears here):
OPNSENSE_SSH_KEY=<edge-key> bash opnsense-set-interface-v4.sh --commit 192.168.1.1 wan $WAN_ADDR $WAN_PL $WAN_GW
# PROVE egress FROM THE EDGE before the LAN move (after it, the reach path is gone):
ssh -i <edge-key> root@192.168.1.1 'ping -c3 1.1.1.1; fetch -o /dev/null http://archive.ubuntu.com/ubuntu/dists/jammy/Release'
# LAN LAST -- the apply drops the session it rides on. EXPECTED to time out; config saves+applies.
OPNSENSE_SSH_KEY=<edge-key> bash opnsense-set-interface-v4.sh --commit 192.168.1.1 lan $LAN_ADDR 22

After the LAN move the edge is at $LAN_ADDR, reachable from the rack's PERMANENT provider-public leg (no interim leg needed). If the script version only SAVES lan, apply it: ssh -i <edge-key> root@$LAN_ADDR configctl interface reconfigure lan. Then configctl filter reload and confirm pfctl -s nat shows nat on vtnet1 ... from (vtnet0:network) to any -> (vtnet1:0). tcsh trap: the edge root shell is tcsh -- NO 2>&1 in edge commands ("Ambiguous output redirect").

Step 6 -- VERIFY (the gate)

ssh $RACK 'sudo -n bash -s -- check <dc>' < scripts/dc-egress-check.sh -> pass=8 fail=0. Confirm the region VM egresses: from the .6, ping 1.1.1.1 and curl images.maas.io 200.

Step 7 -- CLEANUP + RECORD

• SHRED the transient edge private key on the rack (shred -u); remove the interim 192.168.1.2/24 reach leg. Consolidate the API key to the jumphost ~/<dc>-creds/ (0600, sha256-verified, secret never printed).
• SEC row: the open-router exposure ENDS with the pf ruleset restored -- close it.
• Changelog + docs/CURRENT-STATE.md (status). A REBUILT edge invalidated the rack known_hosts (Step 4) -- fold any new lesson here.

Gotchas (measured, do not relearn)

• A rebuilt edge = new host keys -> clear rack known_hosts (Step 4).
• The LAN move DROPS the session mid-apply; config is saved+applied first (Step 5).
• Edge root shell is tcsh: no 2>&1, no nested-quoted one-liners.
• Assert tofu scope on ACTIONS, not the drift note (Step 1.3).
• opnsense-set-interface-v4.sh uses the LEGACY interfaces page, not REST (D-113 amdt) -- do not "fix" it to the API.