Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / incident-20260712-opnsense-edge-boot-triplefault.md
@JANeumatrix JANeumatrix 1 day ago 6 KB DOCFIX-188: root-cause the OPNsense boot triple-fault -- libvirt memory_unit (guest had 2 MiB)
Raw Blame History

Incident 2026-07-12: Office1 OPNsense edge triple-faults at the BTX loader

ROOT CAUSE FOUND 2026-07-12 (DOCFIX-188) -- READ THIS FIRST, THEN STOP

The guest had 2 MiB of RAM. Not a CPU, nesting, machine-type, or console problem. Everything below this box is the ORIGINAL (wrong) investigation, retained for the record. Do not work the "ranked next steps" -- they all chase the wrong layer.

dmacvicar/libvirt >= 0.9 changed memory from MiB (the 0.8-era meaning the modules were written against) to raw libvirt units, defaulting to KiB. With no memory_unit, memory = 2048 rendered <memory unit='KiB'>2048</memory> -> QEMU -m size=2048k = 2 MiB. Measured end-to-end: module input -> tofu state -> domain XML -> virsh dominfo (Max memory: 2048 KiB) -> the live QEMU cmdline.

boot2 is tiny and fits in 2 MiB, so it echoes /boot.config and then triple-faults handing off to /boot/loader, which does not fit. That is why the fault was deterministic at exactly 262 bytes and immune to every CPU/machine/disk/console change tried -- none of them touched the cause.

Fix: memory_unit = "MiB" on the libvirt_domain in all three VM modules (opnsense-edge, cloudinit-vm, node-vm -- the same defect was latent in all of them and would have broken every future VR1 VM). Guarded against recurrence by scripts/opentofu-validate.sh S1. See docs/changelog-20260712-libvirt-memory-unit-rootcause.md.

Lesson for the next incident: a bootloader that dies at a fixed byte offset, immune to every knob you turn, is a resource problem, not a CPU-feature problem. The domain XML and the QEMU cmdline are ground truth -- read them before theorising about nested virt.

Status: root cause fixed in repo; live boot verification is the remaining gated step.

Original status (SUPERSEDED): OPEN. The Office1 OPNsense edge VM is built and starts, but OPNsense triple-faults ~262 bytes into boot. Blocks the Office1 headend (router+DHCP for office1-local) and therefore the Office1 NetBox VM. Written at a context limit as a resume artifact -- a fresh session should read this + docs/session-ledger.md first.

Symptom (confirmed, reproducible)

  • Domain office1-opnsense (libvirt, qemu:///system) enters running (booted) then goes to paused (unknown). virsh resume fails with "cont: Resetting the Virtual Machine is required" -> the guest triple-faulted.
  • Serial capture (/var/lib/libvirt/vr1/staging/office1-opnsense-serial.log, root:600, read with sudo cat) contains exactly, every boot: 
    /boot.config: -S115200 -h -D
    i.e. FreeBSD boot2 echoes /boot.config (serial 115200, -h serial console, -D dual console) and then triple-faults handing off to the BTX /boot/loader. The log mtime updates on each boot (confirmed fresh, not stale); it is deterministically 262 bytes.

Environment

  • Double-nested virt: OPNsense guest -> vcloud host (itself a VM: virtio NIC/disk) -> outer hypervisor. Host CPU model reported by libvirt: AMD Opteron_G3. Nested KVM on (kvm_amd/parameters/nested = 1).
  • Image: OPNsense 26.1 nano amd64 (scripts/opnsense-prep-image.sh 26.1), prepped to /var/lib/libvirt/vr1/office1/opnsense-26.1-nano.qcow2 (11 GiB virtual).
  • Module: opentofu/modules/opnsense-edge, instantiated as module "office1_opnsense" in opentofu/main.tf. LAN=office1-local (vtnet0), WAN=office1-wan (vtnet1, NAT 172.30.1.0/24). Config ISO (real-ISP-router config, DOCFIX-185) at /var/lib/libvirt/vr1/staging/office1-opnsense-config.iso.

What was tried -- ALL applied and verified in the domain XML, NONE resolved it

  1. Serial console added (module gap -- nano is serial-only). Got the boot to the loader stage (from a no-console early fault to the 262-byte /boot.config point).
  2. machine q35 -> i440fx (pc-i440fx-noble, verified). No change. (Note: machine + cpu are create-time; the provider does an in-place "change" that does NOT apply -- must recreate the domain: virsh destroy && virsh undefine, then tofu apply.)
  3. Disk: COW overlay -> direct per-VM copy of the nano (verified 11 GiB / 2.14 GiB allocated, no backing). No change.
  4. CPU host-passthrough (verified). No change.
  5. Disable AMD svm (<feature policy='disable' name='svm'/> in XML -- the documented AMD nested-virt fix, forum: -cpu host,-svm). Still 262 bytes. NOTE: the provider's CPU-feature key is features (plural); feature validates but is silently dropped.

Ranked next steps for a fresh session

  1. Full forum CPU flag set, not just -svm: add +kvm_pv_eoi,+kvm_pv_unhalt (and try without svm masking too). Likely via libvirt_domain.qemu_commandline since the provider's CPU features may not fully translate under host-passthrough. Confirm the masking actually reaches the guest CPUID.
  2. Video device (-D dual console). /boot.config requests dual console but the domain has no video/graphics device -- boot2's VGA init may fault. Add a graphics + video (VGA/std) device and retry. (Cheap, plausible, not yet tried.)
  3. Memory 2 GB -> 4 GB (below OPNsense's 3 GB min; guides use 4096). Recreate to apply.
  4. UEFI boot (OVMF) instead of legacy BIOS/BTX -- sidesteps BTX entirely, but the nano is a BIOS/MBR image, so this needs care (or a different image build).
  5. Outer-hypervisor CPU: under double-nesting, vcloud's exposed CPU (Opteron_G3) may not provide what FreeBSD's BTX needs. Consider testing a minimal FreeBSD/OPNsense boot directly on vcloud to isolate whether it's nesting-depth-specific.

Reproduce / operate

  • Recreate + boot: cd opentofu && source ~/vr1-stage1.env && virsh -c qemu:///system destroy office1-opnsense; virsh -c qemu:///system undefine office1-opnsense; tofu apply.
  • Watch: virsh domstate --reason office1-opnsense; serial via sudo cat the log above.
  • Provider CPU/serial schema is attribute-style + nested under devices/cpu; introspect with tofu providers schema -json (see how serials/features were found this session).
  • As-executed log: ~/as-executed/2026-07-12-dc-dc-phase1-office1.log.
  • Creds (jumphost-only, 0600, ~/vr1-office1-creds/): SSH key + OPNsense root pw/hash. Operator: revoke the pasted NetBox token + these at close.

Cross-refs

  • docs/changelog-20260712-opnsense-edge-boot-fixes.md (DOCFIX-187, the module changes).
  • docs/changelog-20260712-office1-opnsense-edge-build.md (DOCFIX-186, the build + the apparmor/config-iso-staging findings).
  • docs/changelog-20260712-opnsense-edge-real-isp-router.md (DOCFIX-185, config posture).