# Incident 2026-07-12: Office1 OPNsense edge triple-faults at the BTX loader (UNRESOLVED)

**Status:** OPEN. The Office1 OPNsense edge VM is built and *starts*, but OPNsense
triple-faults ~262 bytes into boot. Blocks the Office1 headend (router+DHCP for
`office1-local`) and therefore the Office1 NetBox VM. Written at a context limit as a
resume artifact -- a fresh session should read this + `docs/session-ledger.md` first.

## Symptom (confirmed, reproducible)
- Domain `office1-opnsense` (libvirt, `qemu:///system`) enters `running (booted)` then
  goes to `paused (unknown)`. `virsh resume` fails with *"cont: Resetting the Virtual
  Machine is required"* -> the guest **triple-faulted**.
- Serial capture (`/var/lib/libvirt/vr1/staging/office1-opnsense-serial.log`, `root:600`,
  read with `sudo cat`) contains exactly, every boot:
  ```
  /boot.config: -S115200 -h -D
  ```
  i.e. FreeBSD `boot2` echoes `/boot.config` (serial 115200, `-h` serial console, `-D`
  dual console) and then triple-faults **handing off to the BTX `/boot/loader`**. The log
  mtime updates on each boot (confirmed fresh, not stale); it is deterministically 262 bytes.

## Environment
- **Double-nested virt:** OPNsense guest -> `vcloud` host (itself a VM: virtio NIC/disk)
  -> outer hypervisor. Host CPU model reported by libvirt: **AMD `Opteron_G3`**. Nested
  KVM on (`kvm_amd/parameters/nested = 1`).
- Image: OPNsense **26.1 nano** amd64 (`scripts/opnsense-prep-image.sh 26.1`), prepped to
  `/var/lib/libvirt/vr1/office1/opnsense-26.1-nano.qcow2` (11 GiB virtual).
- Module: `opentofu/modules/opnsense-edge`, instantiated as `module "office1_opnsense"` in
  `opentofu/main.tf`. LAN=`office1-local` (`vtnet0`), WAN=`office1-wan` (`vtnet1`, NAT
  `172.30.1.0/24`). Config ISO (real-ISP-router config, DOCFIX-185) at
  `/var/lib/libvirt/vr1/staging/office1-opnsense-config.iso`.

## What was tried -- ALL applied and verified in the domain XML, NONE resolved it
1. **Serial console added** (module gap -- nano is serial-only). Got the boot *to* the
   loader stage (from a no-console early fault to the 262-byte `/boot.config` point).
2. **`machine` q35 -> i440fx** (`pc-i440fx-noble`, verified). No change. (Note: machine +
   cpu are create-time; the provider does an in-place "change" that does NOT apply --
   must recreate the domain: `virsh destroy && virsh undefine`, then `tofu apply`.)
3. **Disk: COW overlay -> direct per-VM copy** of the nano (verified 11 GiB / 2.14 GiB
   allocated, no backing). No change.
4. **CPU `host-passthrough`** (verified). No change.
5. **Disable AMD `svm`** (`<feature policy='disable' name='svm'/>` in XML -- the documented
   AMD nested-virt fix, forum: `-cpu host,-svm`). **Still 262 bytes.** NOTE: the provider's
   CPU-feature key is `features` (plural); `feature` validates but is silently dropped.

## Ranked next steps for a fresh session
1. **Full forum CPU flag set**, not just `-svm`: add `+kvm_pv_eoi,+kvm_pv_unhalt` (and try
   without `svm` masking too). Likely via `libvirt_domain.qemu_commandline` since the
   provider's CPU `features` may not fully translate under `host-passthrough`. Confirm the
   masking actually reaches the guest CPUID.
2. **Video device (`-D` dual console).** `/boot.config` requests dual console but the domain
   has **no video/graphics device** -- `boot2`'s VGA init may fault. Add a `graphics` +
   `video` (VGA/std) device and retry. (Cheap, plausible, not yet tried.)
3. **Memory 2 GB -> 4 GB** (below OPNsense's 3 GB min; guides use 4096). Recreate to apply.
4. **UEFI boot (OVMF)** instead of legacy BIOS/BTX -- sidesteps BTX entirely, but the nano
   is a BIOS/MBR image, so this needs care (or a different image build).
5. **Outer-hypervisor CPU:** under double-nesting, vcloud's exposed CPU (`Opteron_G3`) may
   not provide what FreeBSD's BTX needs. Consider testing a minimal FreeBSD/OPNsense boot
   directly on vcloud to isolate whether it's nesting-depth-specific.

## Reproduce / operate
- Recreate + boot: `cd opentofu && source ~/vr1-stage1.env && virsh -c qemu:///system
  destroy office1-opnsense; virsh -c qemu:///system undefine office1-opnsense; tofu apply`.
- Watch: `virsh domstate --reason office1-opnsense`; serial via `sudo cat` the log above.
- Provider CPU/serial schema is attribute-style + nested under `devices`/`cpu`; introspect
  with `tofu providers schema -json` (see how `serials`/`features` were found this session).
- As-executed log: `~/as-executed/2026-07-12-dc-dc-phase1-office1.log`.
- Creds (jumphost-only, 0600, `~/vr1-office1-creds/`): SSH key + OPNsense root pw/hash.
  **Operator: revoke the pasted NetBox token + these at close.**

## Cross-refs
- `docs/changelog-20260712-opnsense-edge-boot-fixes.md` (DOCFIX-187, the module changes).
- `docs/changelog-20260712-office1-opnsense-edge-build.md` (DOCFIX-186, the build + the
  apparmor/config-iso-staging findings).
- `docs/changelog-20260712-opnsense-edge-real-isp-router.md` (DOCFIX-185, config posture).