# Changelog 2026-07-12 -- DOCFIX-190: SECOND boot bug -- libvirt domains had ACPI disabled

**Context.** DOCFIX-188 fixed the OPNsense boot triple-fault (a 2 MiB guest). With the guest
finally getting its RAM, the kernel got far enough to reach interrupt initialisation -- and
hit a **second, independent defect** that the first bug had been masking.

## Symptom

Domain reports `running`, burns **100% of one core indefinitely**, emits nothing further on
the console, and never touches the network. Serial log (measured) ends at:

```
---<<BOOT>>---
panic: running without device atpic requires a local APIC
cpuid = 0
KDB: stack backtrace:
apic_init() at apic_init+0xfc/frame ...
mi_startup() at mi_startup+0xb5/frame ...
KDB: enter: panic
db>
```

The 100% CPU spin is the guest parked at the `db>` kernel debugger prompt. A domain in this
state looks healthy to `virsh` -- `running (booted)`, no error -- which is exactly how it
evades casual checks.

## Root cause

The QEMU cmdline carried **`-machine pc-i440fx-noble,...,acpi=off`**.

None of the three VM modules emitted a `features` block, so libvirt defaulted **ACPI off**.
FreeBSD discovers the local APIC from ACPI's MADT table, and the OPNsense kernel ships no
`atpic` fallback -- so with ACPI disabled it has no usable interrupt controller and panics
in `apic_init()` before userland.

## Fix -- all three VM modules (same foundational class as DOCFIX-188)

```hcl
features = {
  acpi = true
  apic = {}
}
```

1. `modules/opnsense-edge` -- the live blocker; FreeBSD **panics outright** without it.
2. `modules/cloudinit-vm` -- MAAS/NetBox/GitBucket. A Linux guest boots without ACPI but
   degraded: no clean ACPI shutdown/reboot signalling, unreliable CPU/IRQ enumeration.
3. `modules/node-vm` -- the MAAS-managed node VMs. **MAAS drives power off/on via ACPI
   signalling**, so without it a graceful `virsh shutdown` (and therefore MAAS power
   control) never works -- nodes could only ever be hard-stopped. This one would have been
   a genuinely nasty Stage-3 debug.

Every ordinary libvirt guest sets both; there was no reason ours did not.

## Guard -- S2

Like `memory_unit`, `tofu validate` **cannot** catch this: the entire `features` block is
optional, so its absence is schema-valid. `scripts/opentofu-validate.sh` gains **S2**, which
fails any `libvirt_domain` that does not enable ACPI. The `--check-memory-unit` flag is
renamed `--static-only` (old name kept as a back-compat alias) since it now runs S1 + S2.

Harness `tests/opentofu-validate/` 4 -> **6 PASS**. New `fixtures/s2-bad/` is the key case:
it **passes S1 and still fails S2**, proving the two guards catch independent classes rather
than one masking the other.

## Verification (measured, live)

| | Before | After |
|---|---|---|
| QEMU machine | `acpi=off` | **`acpi=on`** |
| Kernel | `panic: ... requires a local APIC`, `db>` | boots to userland |
| CPU | **100.0% of one core** (ddb spin) | idle (~19s total) |
| Network | nothing | **DHCP lease `172.30.1.126`, hostname `OPNsense`** |
| Memory (DOCFIX-188, re-confirmed) | -- | `BIOS 639kB/2096108kB available memory` = full 2 GiB |

**OPNsense now boots and runs.** Both boot bugs are closed.

- `tests/opentofu-validate/run-tests.sh`: **6 PASS / 0 FAIL**.
- `scripts/repo-lint.sh`: 0 fail (1 documented legacy warn).
- `tofu fmt`/`validate`: clean.

## STILL OPEN -- the Configuration Importer did not apply (next session's item)

OPNsense came up on **factory defaults** (WAN via DHCP) instead of our static
`172.30.1.2` / LAN `10.10.0.1`. The config ISO is **verified well-formed** -- it is ISO9660
labelled `OPNSENSE_CFG` and contains `CONF/CONFIG.XML` with `<opnsense>` and `10.10.0.1`
inside (grepped from the raw image) -- and it is attached as a SATA cdrom. So the payload is
right and the Importer simply did not consume it.

This is the mechanism `modules/opnsense-edge`'s own header has always flagged UNVERIFIED
("whether the Configuration Importer's ISO9660 support actually behaves as described once
booted for real"). The research is well-sourced; it had just never been exercised. Now it
has, and it did not work as assumed.

**Do NOT guess the fix.** Read the Importer's own console output first -- it runs early in
boot and reports what it scanned. Untested hypothesis worth checking (NOT a conclusion):
ISO9660 8.3 uppercase naming vs the lowercase `/conf/config.xml` the Importer looks for.

**Note the unanswered ping to the WAN address is NOT a fault** -- OPNsense blocks inbound on
WAN by default. Do not chase it.

## Operational finding -- the boot log is unreadable to the agent (real gap)

libvirt creates the serial log **`root:0600`** and **recreates it on every domain create**, so
reading the single most valuable artifact during a boot incident needs an interactive `sudo`
each time. This materially contributed to the original misdiagnosis. A default ACL on the
staging dir does NOT fix it: libvirt's `0600` creation mode sets the ACL **mask** to `---`,
which nullifies any named-user entry (measured: `user:jessea123:r-- #effective:---`).

Proposed durable fix (NOT yet implemented/verified): have the runbook/module pre-create the
serial log `0644` before `tofu apply`, so libvirt appends to an already-readable file rather
than minting a `root:0600` one. Verify on a real recreate before trusting it.

## Revert

- Remove the `features` blocks from the three `libvirt_domain` resources (restores the panic
  -- do not).
- Guard: drop `s2_check` + the `--static-only` rename in `scripts/opentofu-validate.sh`, and
  `tests/opentofu-validate/fixtures/s2-bad/` + T5/T6/T7.
- Whole change: `git revert <sha>`. No live-state dependency.