# Incident 2026-07-12: Office1 OPNsense edge triple-faults at the BTX loader

> ## ROOT CAUSE FOUND 2026-07-12 (DOCFIX-188) -- READ THIS FIRST, THEN STOP
>
> **The guest had 2 MiB of RAM.** Not a CPU, nesting, machine-type, or console problem.
> Everything below this box is the ORIGINAL (wrong) investigation, retained for the record.
> **Do not work the "ranked next steps" -- they all chase the wrong layer.**
>
> `dmacvicar/libvirt` >= 0.9 changed `memory` from MiB (the 0.8-era meaning the modules were
> written against) to **raw libvirt units, defaulting to KiB**. With no `memory_unit`,
> `memory = 2048` rendered `<memory unit='KiB'>2048</memory>` -> QEMU **`-m size=2048k`** =
> **2 MiB**. Measured end-to-end: module input -> tofu state -> domain XML -> `virsh dominfo`
> (`Max memory: 2048 KiB`) -> the live QEMU cmdline.
>
> `boot2` is tiny and fits in 2 MiB, so it echoes `/boot.config` and *then* triple-faults
> handing off to `/boot/loader`, which does not fit. **That is why the fault was
> deterministic at exactly 262 bytes and immune to every CPU/machine/disk/console change
> tried -- none of them touched the cause.**
>
> **Fix:** `memory_unit = "MiB"` on the `libvirt_domain` in all three VM modules
> (`opnsense-edge`, `cloudinit-vm`, `node-vm` -- the same defect was latent in all of them
> and would have broken every future VR1 VM). Guarded against recurrence by
> `scripts/opentofu-validate.sh` S1. See
> `docs/changelog-20260712-libvirt-memory-unit-rootcause.md`.
>
> **Lesson for the next incident:** a bootloader that dies at a fixed byte offset, immune to
> every knob you turn, is a *resource* problem, not a CPU-feature problem. The domain XML and
> the QEMU cmdline are ground truth -- read them before theorising about nested virt.
>
> **Status:** root cause fixed in repo; live boot verification is the remaining gated step.

**Original status (SUPERSEDED):** OPEN. The Office1 OPNsense edge VM is built and *starts*, but OPNsense
triple-faults ~262 bytes into boot. Blocks the Office1 headend (router+DHCP for
`office1-local`) and therefore the Office1 NetBox VM. Written at a context limit as a
resume artifact -- a fresh session should read this + `docs/session-ledger.md` first.

## Symptom (confirmed, reproducible)
- Domain `office1-opnsense` (libvirt, `qemu:///system`) enters `running (booted)` then
  goes to `paused (unknown)`. `virsh resume` fails with *"cont: Resetting the Virtual
  Machine is required"* -> the guest **triple-faulted**.
- Serial capture (`/var/lib/libvirt/vr1/staging/office1-opnsense-serial.log`, `root:600`,
  read with `sudo cat`) contains exactly, every boot:
  ```
  /boot.config: -S115200 -h -D
  ```
  i.e. FreeBSD `boot2` echoes `/boot.config` (serial 115200, `-h` serial console, `-D`
  dual console) and then triple-faults **handing off to the BTX `/boot/loader`**. The log
  mtime updates on each boot (confirmed fresh, not stale); it is deterministically 262 bytes.

## Environment
- **Double-nested virt:** OPNsense guest -> `vcloud` host (itself a VM: virtio NIC/disk)
  -> outer hypervisor. Host CPU model reported by libvirt: **AMD `Opteron_G3`**. Nested
  KVM on (`kvm_amd/parameters/nested = 1`).
- Image: OPNsense **26.1 nano** amd64 (`scripts/opnsense-prep-image.sh 26.1`), prepped to
  `/var/lib/libvirt/vr1/office1/opnsense-26.1-nano.qcow2` (11 GiB virtual).
- Module: `opentofu/modules/opnsense-edge`, instantiated as `module "office1_opnsense"` in
  `opentofu/main.tf`. LAN=`office1-local` (`vtnet0`), WAN=`office1-wan` (`vtnet1`, NAT
  `172.30.1.0/24`). Config ISO (real-ISP-router config, DOCFIX-185) at
  `/var/lib/libvirt/vr1/staging/office1-opnsense-config.iso`.

## What was tried -- ALL applied and verified in the domain XML, NONE resolved it
1. **Serial console added** (module gap -- nano is serial-only). Got the boot *to* the
   loader stage (from a no-console early fault to the 262-byte `/boot.config` point).
2. **`machine` q35 -> i440fx** (`pc-i440fx-noble`, verified). No change. (Note: machine +
   cpu are create-time; the provider does an in-place "change" that does NOT apply --
   must recreate the domain: `virsh destroy && virsh undefine`, then `tofu apply`.)
3. **Disk: COW overlay -> direct per-VM copy** of the nano (verified 11 GiB / 2.14 GiB
   allocated, no backing). No change.
4. **CPU `host-passthrough`** (verified). No change.
5. **Disable AMD `svm`** (`<feature policy='disable' name='svm'/>` in XML -- the documented
   AMD nested-virt fix, forum: `-cpu host,-svm`). **Still 262 bytes.** NOTE: the provider's
   CPU-feature key is `features` (plural); `feature` validates but is silently dropped.

## Ranked next steps for a fresh session
1. **Full forum CPU flag set**, not just `-svm`: add `+kvm_pv_eoi,+kvm_pv_unhalt` (and try
   without `svm` masking too). Likely via `libvirt_domain.qemu_commandline` since the
   provider's CPU `features` may not fully translate under `host-passthrough`. Confirm the
   masking actually reaches the guest CPUID.
2. **Video device (`-D` dual console).** `/boot.config` requests dual console but the domain
   has **no video/graphics device** -- `boot2`'s VGA init may fault. Add a `graphics` +
   `video` (VGA/std) device and retry. (Cheap, plausible, not yet tried.)
3. **Memory 2 GB -> 4 GB** (below OPNsense's 3 GB min; guides use 4096). Recreate to apply.
4. **UEFI boot (OVMF)** instead of legacy BIOS/BTX -- sidesteps BTX entirely, but the nano
   is a BIOS/MBR image, so this needs care (or a different image build).
5. **Outer-hypervisor CPU:** under double-nesting, vcloud's exposed CPU (`Opteron_G3`) may
   not provide what FreeBSD's BTX needs. Consider testing a minimal FreeBSD/OPNsense boot
   directly on vcloud to isolate whether it's nesting-depth-specific.

## Reproduce / operate
- Recreate + boot: `cd opentofu && source ~/vr1-stage1.env && virsh -c qemu:///system
  destroy office1-opnsense; virsh -c qemu:///system undefine office1-opnsense; tofu apply`.
- Watch: `virsh domstate --reason office1-opnsense`; serial via `sudo cat` the log above.
- Provider CPU/serial schema is attribute-style + nested under `devices`/`cpu`; introspect
  with `tofu providers schema -json` (see how `serials`/`features` were found this session).
- As-executed log: `~/as-executed/2026-07-12-dc-dc-phase1-office1.log`.
- Creds (jumphost-only, 0600, `~/vr1-office1-creds/`): SSH key + OPNsense root pw/hash.
  **Operator: revoke the pasted NetBox token + these at close.**

## Cross-refs
- `docs/changelog-20260712-opnsense-edge-boot-fixes.md` (DOCFIX-187, the module changes).
- `docs/changelog-20260712-office1-opnsense-edge-build.md` (DOCFIX-186, the build + the
  apparmor/config-iso-staging findings).
- `docs/changelog-20260712-opnsense-edge-real-isp-router.md` (DOCFIX-185, config posture).