# Runbook: DC OPNsense edge REBUILD (filesystem / power-cut damage) **Use when** a DC edge forwards but does not NAT, or its GUI answers on the WAN, or `dc-egress-check ` fails A3/A4 while the edge answers A2 -- i.e. the edge lost its pf ruleset (typically fsck damage to `/etc/master.passwd` + `/etc/group` after an unclean power cut; the SEC-031 class). **Verdict is REBUILD, not in-place repair:** a fresh pristine disk + this bootstrap is faster and more reliable than single-user user-DB surgery, and the edge config is fully recreated by the addressing step below. **Proven twice:** dc0 2026-08-02 (`docs/audit/dc0-edge-rebuild-20260802.txt`, the worked reference) and dc1 2026-08-07 (`docs/changelog-20260807-dc1-region-sequence.md` Item 4). Both ended `dc-egress-check ` **8/8**. This runbook GENERALISES those; it exists because the dc0 run lived only as an audit capture, so dc1 had to reconstruct it. **Tools (all in-repo -- do NOT reinvent):** `scripts/opnsense-console-rebuild.py` (site-agnostic D-112(c) console driver), `scripts/opnsense-bootstrap-apikey.sh`, `scripts/opnsense-api.sh`, `scripts/opnsense-set-interface-v4.sh`, `scripts/dc-egress-check.sh`. --- ## Per-DC values (fill in, measured -- never inferred) | var | dc0 | dc1 | source | |---|---|---|---| | `RACK` (transit IP) | 172.31.0.2 | 172.31.0.6 | `scripts/lib-hosts.sh` | | `DOMAIN` | vr1-dc0-opnsense | vr1-dc1-opnsense | inner tofu module | | inner root | `opentofu/vr1-dc0-substrate` | `opentofu/vr1-dc1-substrate` | on voffice1 | | `WAN` addr/pl gw | 172.30.2.2/24 172.30.2.1 | 172.30.3.2/24 172.30.3.1 | G12 / D-125 | | `LAN` (provider-public gw) | 10.12.4.1/22 | 10.12.64.1/22 | lib-net `PLANE_GW` | | rack provider-public leg / bridge | 10.12.4.2 / virbrN | 10.12.64.2 / virbr4 | `ip -4 -o addr` on rack | | edge key (jumphost) | `~/vr1-dc0-creds/vr1-dc0-edge_ed25519` | `~/vr1-dc1-creds/vr1-dc1-edge_ed25519` | SEC-013/-015 | | base image (voffice1) | `~/vr1-dc0-images/opnsense-26.7-nano.qcow2` | `~/vr1-dc1-images/opnsense-26.7-nano.qcow2` | present-before-apply | All `virsh` on the rack takes `-c qemu:///system` (rack default is `qemu:///session`). --- ## Step 1 -- SAFETY PREP (read-only + backups) 1. Rollback copy of the edge disk (no backingStore = no other rollback point): `virsh -c qemu:///system shutdown $DOMAIN` (ACPI works even with a broken login); wait for `shut off`; then on the rack `sudo cp -p .qcow2 .qcow2.pre-rebuild-`. 2. Backup the inner tfstate on voffice1: `cp -p terraform.tfstate ~/-substrate-tfstate-pre-edge-rebuild-.json` + sha256-verify identical. 3. Cut a TARGETED plan and **machine-assert it is edge-only** (this is the gate that protects the 9 node VMs sharing this state): ``` cd "$REPO"/opentofu/-substrate/ # $REPO set per session bootstrap tofu plan -replace=module._opnsense.libvirt_volume.disk \ -replace=module._opnsense.libvirt_domain.vm -out=/tmp/edge-rebuild.plan tofu show -json /tmp/edge-rebuild.plan | jq -r '.resource_changes[] | select(.change.actions!=["no-op"]) | .address+" "+(.change.actions|join(","))' ``` REQUIRE: exactly 2 non-no-op changes, BOTH the edge domain+disk, each `delete,create`; 0 non-no-op changes naming anything else. (The refresh reports ~20 resources "changed outside OpenTofu" -- that is DRIFT, not an action; assert on ACTIONS only.) 4. Confirm the base image the plan will use is PRESENT on voffice1 (the apply destroys the volume before recreating -- a missing source strands the edge with no disk). ## Step 2 -- REBUILD (destroy+recreate the edge, edge-only) `tofu apply /tmp/edge-rebuild.plan` -> `2 added, 0 changed, 2 destroyed`. Fresh disk is base-image size / new mtime (measure the artifact, don't trust the log). Domain boots factory: LAN `192.168.1.1/24`, WAN address-less, root/opnsense login. ## Step 3 -- CONSOLE BOOTSTRAP (enables ssh + installs the edge key) Runs ON the rack (has pexpect). Stage the driver + the site edge PUBKEY (public): ``` scp scripts/opnsense-console-rebuild.py .pub -> rack ssh $RACK 'python3 ~/opnsense-console-rebuild.py --domain '$DOMAIN' --pubkey ~/.pub' ``` Expect `CONSOLE-STATE=0` then `BOOTSTRAP-SEQUENCE-COMPLETE`. Verify the transcript shows `CONFIG-WRITTEN`, `AK-IN-CONFIG=152`, `USER-MATERIALIZED`, `OK` (sshd restart). ## Step 4 -- REACH + KEY-ONLY SSH (D-112(c) exit condition) A REBUILT edge has NEW host keys -- clear the rack's stale entries or every `BatchMode` script fails with HOST IDENTIFICATION CHANGED: ``` ssh $RACK 'cp -p ~/.ssh/known_hosts ~/.ssh/known_hosts.pre-rebuild-; ssh-keygen -R 192.168.1.1; ssh-keygen -R $LAN_IP; sudo ip addr add 192.168.1.2/24 dev ; # interim reach to factory LAN ssh-keyscan -T8 192.168.1.1 >> ~/.ssh/known_hosts' ``` Stage the edge PRIVATE key to the rack transiently (0600, SEC-015 pattern -- SHRED in Step 7) OR reach the edge from the jumphost via ProxyCommand. Prove: `ssh -i root@192.168.1.1 opnsense-version` -> `OPNsense 26.7`. ## Step 5 -- API KEY + ADDRESSING (WAN first, LAN last) ``` cd ~/repo-stage/scripts # sha256-verify the 5 opnsense-*.sh/php against the repo first OPNSENSE_SSH_KEY= bash opnsense-bootstrap-apikey.sh 192.168.1.1 .txt # WAN (also REGENERATES the pf ruleset -> automatic outbound NAT appears here): OPNSENSE_SSH_KEY= bash opnsense-set-interface-v4.sh --commit 192.168.1.1 wan $WAN_ADDR $WAN_PL $WAN_GW # PROVE egress FROM THE EDGE before the LAN move (after it, the reach path is gone): ssh -i root@192.168.1.1 'ping -c3 1.1.1.1; fetch -o /dev/null http://archive.ubuntu.com/ubuntu/dists/jammy/Release' # LAN LAST -- the apply drops the session it rides on. EXPECTED to time out; config saves+applies. OPNSENSE_SSH_KEY= bash opnsense-set-interface-v4.sh --commit 192.168.1.1 lan $LAN_ADDR 22 ``` After the LAN move the edge is at `$LAN_ADDR`, reachable from the rack's PERMANENT provider-public leg (no interim leg needed). If the script version only SAVES lan, apply it: `ssh -i root@$LAN_ADDR configctl interface reconfigure lan`. Then `configctl filter reload` and confirm `pfctl -s nat` shows `nat on vtnet1 ... from (vtnet0:network) to any -> (vtnet1:0)`. **tcsh trap:** the edge root shell is tcsh -- NO `2>&1` in edge commands ("Ambiguous output redirect"). ## Step 6 -- VERIFY (the gate) `ssh $RACK 'sudo -n bash -s -- check ' < scripts/dc-egress-check.sh` -> **pass=8 fail=0**. Confirm the region VM egresses: from the `.6`, `ping 1.1.1.1` and `curl images.maas.io` 200. ## Step 7 -- CLEANUP + RECORD - SHRED the transient edge private key on the rack (`shred -u`); remove the interim `192.168.1.2/24` reach leg. Consolidate the API key to the jumphost `~/-creds/` (0600, sha256-verified, secret never printed). - SEC row: the open-router exposure ENDS with the pf ruleset restored -- close it. - Changelog + `docs/CURRENT-STATE.md` (status). A REBUILT edge invalidated the rack known_hosts (Step 4) -- fold any new lesson here. ## Gotchas (measured, do not relearn) - A rebuilt edge = new host keys -> clear rack known_hosts (Step 4). - The LAN move DROPS the session mid-apply; config is saved+applied first (Step 5). - Edge root shell is tcsh: no `2>&1`, no nested-quoted one-liners. - Assert tofu scope on ACTIONS, not the drift note (Step 1.3). - `opnsense-set-interface-v4.sh` uses the LEGACY interfaces page, not REST (D-113 amdt) -- do not "fix" it to the API.