# Runbook: DC OPNsense edge REBUILD (filesystem / power-cut damage)

**Use when** a DC edge forwards but does not NAT, or its GUI answers on the WAN, or
`dc-egress-check <dc>` fails A3/A4 while the edge answers A2 -- i.e. the edge lost its pf
ruleset (typically fsck damage to `/etc/master.passwd` + `/etc/group` after an unclean
power cut; the SEC-031 class). **Verdict is REBUILD, not in-place repair:** a fresh
pristine disk + this bootstrap is faster and more reliable than single-user user-DB
surgery, and the edge config is fully recreated by the addressing step below.

**Proven twice:** dc0 2026-08-02 (`docs/audit/dc0-edge-rebuild-20260802.txt`, the
worked reference) and dc1 2026-08-07 (`docs/changelog-20260807-dc1-region-sequence.md`
Item 4). Both ended `dc-egress-check <dc>` **8/8**. This runbook GENERALISES those; it
exists because the dc0 run lived only as an audit capture, so dc1 had to reconstruct it.

**Tools (all in-repo -- do NOT reinvent):** `scripts/opnsense-console-rebuild.py`
(site-agnostic D-112(c) console driver), `scripts/opnsense-bootstrap-apikey.sh`,
`scripts/opnsense-api.sh`, `scripts/opnsense-set-interface-v4.sh`,
`scripts/dc-egress-check.sh`.

---

## Per-DC values (fill in, measured -- never inferred)

| var | dc0 | dc1 | source |
|---|---|---|---|
| `RACK` (transit IP) | 172.31.0.2 | 172.31.0.6 | `scripts/lib-hosts.sh` |
| `DOMAIN` | vr1-dc0-opnsense | vr1-dc1-opnsense | inner tofu module |
| inner root | `opentofu/vr1-dc0-substrate` | `opentofu/vr1-dc1-substrate` | on voffice1 |
| `WAN` addr/pl gw | 172.30.2.2/24 172.30.2.1 | 172.30.3.2/24 172.30.3.1 | G12 / D-125 |
| `LAN` (provider-public gw) | 10.12.4.1/22 | 10.12.64.1/22 | lib-net `PLANE_GW` |
| rack provider-public leg / bridge | 10.12.4.2 / virbrN | 10.12.64.2 / virbr4 | `ip -4 -o addr` on rack |
| edge key (jumphost) | `~/vr1-dc0-creds/vr1-dc0-edge_ed25519` | `~/vr1-dc1-creds/vr1-dc1-edge_ed25519` | SEC-013/-015 |
| base image (voffice1) | `~/vr1-dc0-images/opnsense-26.7-nano.qcow2` | `~/vr1-dc1-images/opnsense-26.7-nano.qcow2` | present-before-apply |

All `virsh` on the rack takes `-c qemu:///system` (rack default is `qemu:///session`).

---

## Step 1 -- SAFETY PREP (read-only + backups)

1. Rollback copy of the edge disk (no backingStore = no other rollback point):
   `virsh -c qemu:///system shutdown $DOMAIN` (ACPI works even with a broken login);
   wait for `shut off`; then on the rack `sudo cp -p <edge-disk>.qcow2 <edge-disk>.qcow2.pre-rebuild-<date>`.
2. Backup the inner tfstate on voffice1: `cp -p terraform.tfstate ~/<dc>-substrate-tfstate-pre-edge-rebuild-<date>.json` + sha256-verify identical.
3. Cut a TARGETED plan and **machine-assert it is edge-only** (this is the gate that
   protects the 9 node VMs sharing this state):
   ```
   cd "$REPO"/opentofu/<dc>-substrate/          # $REPO set per session bootstrap
   tofu plan -replace=module.<dc>_opnsense.libvirt_volume.disk \
             -replace=module.<dc>_opnsense.libvirt_domain.vm -out=/tmp/edge-rebuild.plan
   tofu show -json /tmp/edge-rebuild.plan | jq -r '.resource_changes[] | select(.change.actions!=["no-op"]) | .address+" "+(.change.actions|join(","))'
   ```
   REQUIRE: exactly 2 non-no-op changes, BOTH the edge domain+disk, each `delete,create`;
   0 non-no-op changes naming anything else. (The refresh reports ~20 resources "changed
   outside OpenTofu" -- that is DRIFT, not an action; assert on ACTIONS only.)
4. Confirm the base image the plan will use is PRESENT on voffice1 (the apply destroys
   the volume before recreating -- a missing source strands the edge with no disk).

## Step 2 -- REBUILD (destroy+recreate the edge, edge-only)

`tofu apply /tmp/edge-rebuild.plan` -> `2 added, 0 changed, 2 destroyed`. Fresh disk is
base-image size / new mtime (measure the artifact, don't trust the log). Domain boots
factory: LAN `192.168.1.1/24`, WAN address-less, root/opnsense login.

## Step 3 -- CONSOLE BOOTSTRAP (enables ssh + installs the edge key)

Runs ON the rack (has pexpect). Stage the driver + the site edge PUBKEY (public):
```
scp scripts/opnsense-console-rebuild.py <edge-pubkey>.pub -> rack
ssh $RACK 'python3 ~/opnsense-console-rebuild.py --domain '$DOMAIN' --pubkey ~/<edge>.pub'
```
Expect `CONSOLE-STATE=0` then `BOOTSTRAP-SEQUENCE-COMPLETE`. Verify the transcript shows
`CONFIG-WRITTEN`, `AK-IN-CONFIG=152`, `USER-MATERIALIZED`, `OK` (sshd restart).

## Step 4 -- REACH + KEY-ONLY SSH (D-112(c) exit condition)

A REBUILT edge has NEW host keys -- clear the rack's stale entries or every `BatchMode`
script fails with HOST IDENTIFICATION CHANGED:
```
ssh $RACK 'cp -p ~/.ssh/known_hosts ~/.ssh/known_hosts.pre-rebuild-<date>;
           ssh-keygen -R 192.168.1.1; ssh-keygen -R $LAN_IP;
           sudo ip addr add 192.168.1.2/24 dev <rack-provider-public-bridge>;   # interim reach to factory LAN
           ssh-keyscan -T8 192.168.1.1 >> ~/.ssh/known_hosts'
```
Stage the edge PRIVATE key to the rack transiently (0600, SEC-015 pattern -- SHRED in
Step 7) OR reach the edge from the jumphost via ProxyCommand. Prove:
`ssh -i <edge-key> root@192.168.1.1 opnsense-version` -> `OPNsense 26.7`.

## Step 5 -- API KEY + ADDRESSING (WAN first, LAN last)

```
cd ~/repo-stage/scripts   # sha256-verify the 5 opnsense-*.sh/php against the repo first
OPNSENSE_SSH_KEY=<edge-key> bash opnsense-bootstrap-apikey.sh 192.168.1.1 <api-out>.txt
# WAN (also REGENERATES the pf ruleset -> automatic outbound NAT appears here):
OPNSENSE_SSH_KEY=<edge-key> bash opnsense-set-interface-v4.sh --commit 192.168.1.1 wan $WAN_ADDR $WAN_PL $WAN_GW
# PROVE egress FROM THE EDGE before the LAN move (after it, the reach path is gone):
ssh -i <edge-key> root@192.168.1.1 'ping -c3 1.1.1.1; fetch -o /dev/null http://archive.ubuntu.com/ubuntu/dists/jammy/Release'
# LAN LAST -- the apply drops the session it rides on. EXPECTED to time out; config saves+applies.
OPNSENSE_SSH_KEY=<edge-key> bash opnsense-set-interface-v4.sh --commit 192.168.1.1 lan $LAN_ADDR 22
```
After the LAN move the edge is at `$LAN_ADDR`, reachable from the rack's PERMANENT
provider-public leg (no interim leg needed). If the script version only SAVES lan, apply
it: `ssh -i <edge-key> root@$LAN_ADDR configctl interface reconfigure lan`. Then
`configctl filter reload` and confirm `pfctl -s nat` shows
`nat on vtnet1 ... from (vtnet0:network) to any -> (vtnet1:0)`.
**tcsh trap:** the edge root shell is tcsh -- NO `2>&1` in edge commands ("Ambiguous
output redirect").

## Step 6 -- VERIFY (the gate)

`ssh $RACK 'sudo -n bash -s -- check <dc>' < scripts/dc-egress-check.sh` -> **pass=8 fail=0**.
Confirm the region VM egresses: from the `.6`, `ping 1.1.1.1` and `curl images.maas.io` 200.

## Step 7 -- CLEANUP + RECORD

- SHRED the transient edge private key on the rack (`shred -u`); remove the interim
  `192.168.1.2/24` reach leg. Consolidate the API key to the jumphost `~/<dc>-creds/`
  (0600, sha256-verified, secret never printed).
- SEC row: the open-router exposure ENDS with the pf ruleset restored -- close it.
- Changelog + `docs/CURRENT-STATE.md` (status). A REBUILT edge invalidated the rack
  known_hosts (Step 4) -- fold any new lesson here.

## Gotchas (measured, do not relearn)
- A rebuilt edge = new host keys -> clear rack known_hosts (Step 4).
- The LAN move DROPS the session mid-apply; config is saved+applied first (Step 5).
- Edge root shell is tcsh: no `2>&1`, no nested-quoted one-liners.
- Assert tofu scope on ACTIONS, not the drift note (Step 1.3).
- `opnsense-set-interface-v4.sh` uses the LEGACY interfaces page, not REST (D-113 amdt) --
  do not "fix" it to the API.
