Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260712-opnsense-edge-boot-fixes.md

Changelog 2026-07-12 -- DOCFIX-187: opnsense-edge module first-boot fixes (boot still OPEN)

SUPERSEDED IN PART -- read docs/changelog-20260712-libvirt-memory-unit-rootcause.md (DOCFIX-188) and the DOCFIX-189 audit first.

This changelog was written under a WRONG diagnosis, at a context limit. The boot fault was a 2 MiB guest (missing memory_unit), not a CPU/machine/disk/console problem.

The CHANGES below are retained (each is defensible on its own merits), but the CAUSAL CLAIMS attached to items 2, 3 and 4 are FALSE and were corrected in the module source by DOCFIX-189:

  • item 2 (q35 -> i440fx): did NOT fix the fault. q35 was never shown to be a problem.
  • item 3 (COW overlay -> direct copy): did NOT fix the fault. It also silently orphaned disk_size_bytes, so the edge got 11 GiB while main.tf claimed 16 GiB (DOCFIX-189).
  • item 4 (svm disable): did NOT fix the fault.
  • item 1 (serial console) is the one that earned its keep -- but it revealed the fault, it did not change it.

Standing lesson: when a change does not fix the symptom, do not write it up as though it did. Every one of these was recorded as a contributing cause on no evidence.

Context. First real boot of the Office1 OPNsense edge. Several correct-but-necessary module fixes were made while debugging; the boot is NOT yet resolved (a BTX-loader triple-fault remains -- see docs/incident-20260712-opnsense-edge-boot-triplefault.md). These changes are kept because a working config needs them regardless.

opentofu/modules/opnsense-edge/main.tf

  1. Serial console added (devices.serials, file-backed to /var/lib/libvirt/vr1/staging/${vm_name}-serial.log). OPNsense nano is serial-only; the module had no console, so the guest had no output and faulted with nothing to show. This got the boot from a blank early fault to the loader stage (and gives boot-log capture).
  2. machine q35 -> pc (i440fx). FreeBSD's legacy BTX loader is far more commonly booted on i440fx; q35 is a known early-fault source. (Did not fix this incident, but is the safer, documented choice.)
  3. Disk: COW overlay -> direct per-VM copy (create.content.url = base_volume_path, no backing_store). Matches the documented virt-install --import <nano.qcow2> flow; the nano is designed to boot + auto-expand in place, not through a read-only backing. var.disk_size_bytes is now unused (nano is pre-grown by opnsense-prep-image.sh).
  4. CPU host-passthrough with AMD svm disabled (cpu.features = [{name=svm, policy=disable}]). The documented AMD nested-virt fix (-cpu host,-svm). NOTE: the provider key is features (plural) -- feature validates but is silently dropped.

Important operational note discovered

  • machine and cpu are create-time; the dmacvicar provider plans them as an in-place "change" that libvirt does NOT apply to an existing domain. To actually change them: virsh destroy <dom>; virsh undefine <dom>; tofu apply (recreate). tofu apply -replace hit "domain already exists" mid-replace this session.

Status

  • Module tofu validate -> Success. NOT booting: OPNsense triple-faults at the BTX loader regardless of the above (incident report has the full trail + ranked next steps).

Revert

  • git checkout opentofu/modules/opnsense-edge/main.tf to the pre-DOCFIX-187 revision. (Reverts the serial console + i440fx + direct-disk + svm-disable together.)