Changelog 2026-07-12 -- DOCFIX-190: SECOND boot bug -- libvirt domains had ACPI disabled

Context. DOCFIX-188 fixed the OPNsense boot triple-fault (a 2 MiB guest). With the guest finally getting its RAM, the kernel got far enough to reach interrupt initialisation -- and hit a second, independent defect that the first bug had been masking.

Symptom

Domain reports running, burns 100% of one core indefinitely, emits nothing further on the console, and never touches the network. Serial log (measured) ends at:

---<<BOOT>>---
panic: running without device atpic requires a local APIC
cpuid = 0
KDB: stack backtrace:
apic_init() at apic_init+0xfc/frame ...
mi_startup() at mi_startup+0xb5/frame ...
KDB: enter: panic
db>

The 100% CPU spin is the guest parked at the db> kernel debugger prompt. A domain in this state looks healthy to virsh -- running (booted), no error -- which is exactly how it evades casual checks.

Root cause

The QEMU cmdline carried -machine pc-i440fx-noble,...,acpi=off.

None of the three VM modules emitted a features block, so libvirt defaulted ACPI off. FreeBSD discovers the local APIC from ACPI's MADT table, and the OPNsense kernel ships no atpic fallback -- so with ACPI disabled it has no usable interrupt controller and panics in apic_init() before userland.

Fix -- all three VM modules (same foundational class as DOCFIX-188)

features = {
  acpi = true
  apic = {}
}

1. modules/opnsense-edge -- the live blocker; FreeBSD panics outright without it.
2. modules/cloudinit-vm -- MAAS/NetBox/GitBucket. A Linux guest boots without ACPI but degraded: no clean ACPI shutdown/reboot signalling, unreliable CPU/IRQ enumeration.
3. modules/node-vm -- the MAAS-managed node VMs. MAAS drives power off/on via ACPI signalling, so without it a graceful virsh shutdown (and therefore MAAS power control) never works -- nodes could only ever be hard-stopped. This one would have been a genuinely nasty Stage-3 debug.

Every ordinary libvirt guest sets both; there was no reason ours did not.

Guard -- S2

Like memory_unit, tofu validate cannot catch this: the entire features block is optional, so its absence is schema-valid. scripts/opentofu-validate.sh gains S2, which fails any libvirt_domain that does not enable ACPI. The --check-memory-unit flag is renamed --static-only (old name kept as a back-compat alias) since it now runs S1 + S2.

Harness tests/opentofu-validate/ 4 -> 6 PASS. New fixtures/s2-bad/ is the key case: it passes S1 and still fails S2, proving the two guards catch independent classes rather than one masking the other.

Verification (measured, live)

OPNsense now boots and runs. Both boot bugs are closed.

• tests/opentofu-validate/run-tests.sh: 6 PASS / 0 FAIL.
• scripts/repo-lint.sh: 0 fail (1 documented legacy warn).
• tofu fmt/validate: clean.

STILL OPEN -- the Configuration Importer did not apply (next session's item)

OPNsense came up on factory defaults (WAN via DHCP) instead of our static 172.30.1.2 / LAN 10.10.0.1. The config ISO is verified well-formed -- it is ISO9660 labelled OPNSENSE_CFG and contains CONF/CONFIG.XML with <opnsense> and 10.10.0.1 inside (grepped from the raw image) -- and it is attached as a SATA cdrom. So the payload is right and the Importer simply did not consume it.

This is the mechanism modules/opnsense-edge's own header has always flagged UNVERIFIED ("whether the Configuration Importer's ISO9660 support actually behaves as described once booted for real"). The research is well-sourced; it had just never been exercised. Now it has, and it did not work as assumed.

Do NOT guess the fix. Read the Importer's own console output first -- it runs early in boot and reports what it scanned. Untested hypothesis worth checking (NOT a conclusion): ISO9660 8.3 uppercase naming vs the lowercase /conf/config.xml the Importer looks for.

Note the unanswered ping to the WAN address is NOT a fault -- OPNsense blocks inbound on WAN by default. Do not chase it.

Operational finding -- the boot log is unreadable to the agent (real gap)

libvirt creates the serial log root:0600 and recreates it on every domain create, so reading the single most valuable artifact during a boot incident needs an interactive sudo each time. This materially contributed to the original misdiagnosis. A default ACL on the staging dir does NOT fix it: libvirt's 0600 creation mode sets the ACL mask to ---, which nullifies any named-user entry (measured: user:jessea123:r-- #effective:---).

Proposed durable fix (NOT yet implemented/verified): have the runbook/module pre-create the serial log 0644 before tofu apply, so libvirt appends to an already-readable file rather than minting a root:0600 one. Verify on a real recreate before trusting it.

Revert

• Remove the features blocks from the three libvirt_domain resources (restores the panic -- do not).
• Guard: drop s2_check + the --static-only rename in scripts/opentofu-validate.sh, and tests/opentofu-validate/fixtures/s2-bad/ + T5/T6/T7.
• Whole change: git revert <sha>. No live-state dependency.