diff --git a/docs/changelog-20260712-libvirt-acpi-kernel-panic.md b/docs/changelog-20260712-libvirt-acpi-kernel-panic.md new file mode 100644 index 0000000..2a19a6e --- /dev/null +++ b/docs/changelog-20260712-libvirt-acpi-kernel-panic.md @@ -0,0 +1,120 @@ +# Changelog 2026-07-12 -- DOCFIX-190: SECOND boot bug -- libvirt domains had ACPI disabled + +**Context.** DOCFIX-188 fixed the OPNsense boot triple-fault (a 2 MiB guest). With the guest +finally getting its RAM, the kernel got far enough to reach interrupt initialisation -- and +hit a **second, independent defect** that the first bug had been masking. + +## Symptom + +Domain reports `running`, burns **100% of one core indefinitely**, emits nothing further on +the console, and never touches the network. Serial log (measured) ends at: + +``` +---<>--- +panic: running without device atpic requires a local APIC +cpuid = 0 +KDB: stack backtrace: +apic_init() at apic_init+0xfc/frame ... +mi_startup() at mi_startup+0xb5/frame ... +KDB: enter: panic +db> +``` + +The 100% CPU spin is the guest parked at the `db>` kernel debugger prompt. A domain in this +state looks healthy to `virsh` -- `running (booted)`, no error -- which is exactly how it +evades casual checks. + +## Root cause + +The QEMU cmdline carried **`-machine pc-i440fx-noble,...,acpi=off`**. + +None of the three VM modules emitted a `features` block, so libvirt defaulted **ACPI off**. +FreeBSD discovers the local APIC from ACPI's MADT table, and the OPNsense kernel ships no +`atpic` fallback -- so with ACPI disabled it has no usable interrupt controller and panics +in `apic_init()` before userland. + +## Fix -- all three VM modules (same foundational class as DOCFIX-188) + +```hcl +features = { + acpi = true + apic = {} +} +``` + +1. `modules/opnsense-edge` -- the live blocker; FreeBSD **panics outright** without it. +2. `modules/cloudinit-vm` -- MAAS/NetBox/GitBucket. A Linux guest boots without ACPI but + degraded: no clean ACPI shutdown/reboot signalling, unreliable CPU/IRQ enumeration. +3. `modules/node-vm` -- the MAAS-managed node VMs. **MAAS drives power off/on via ACPI + signalling**, so without it a graceful `virsh shutdown` (and therefore MAAS power + control) never works -- nodes could only ever be hard-stopped. This one would have been + a genuinely nasty Stage-3 debug. + +Every ordinary libvirt guest sets both; there was no reason ours did not. + +## Guard -- S2 + +Like `memory_unit`, `tofu validate` **cannot** catch this: the entire `features` block is +optional, so its absence is schema-valid. `scripts/opentofu-validate.sh` gains **S2**, which +fails any `libvirt_domain` that does not enable ACPI. The `--check-memory-unit` flag is +renamed `--static-only` (old name kept as a back-compat alias) since it now runs S1 + S2. + +Harness `tests/opentofu-validate/` 4 -> **6 PASS**. New `fixtures/s2-bad/` is the key case: +it **passes S1 and still fails S2**, proving the two guards catch independent classes rather +than one masking the other. + +## Verification (measured, live) + +| | Before | After | +|---|---|---| +| QEMU machine | `acpi=off` | **`acpi=on`** | +| Kernel | `panic: ... requires a local APIC`, `db>` | boots to userland | +| CPU | **100.0% of one core** (ddb spin) | idle (~19s total) | +| Network | nothing | **DHCP lease `172.30.1.126`, hostname `OPNsense`** | +| Memory (DOCFIX-188, re-confirmed) | -- | `BIOS 639kB/2096108kB available memory` = full 2 GiB | + +**OPNsense now boots and runs.** Both boot bugs are closed. + +- `tests/opentofu-validate/run-tests.sh`: **6 PASS / 0 FAIL**. +- `scripts/repo-lint.sh`: 0 fail (1 documented legacy warn). +- `tofu fmt`/`validate`: clean. + +## STILL OPEN -- the Configuration Importer did not apply (next session's item) + +OPNsense came up on **factory defaults** (WAN via DHCP) instead of our static +`172.30.1.2` / LAN `10.10.0.1`. The config ISO is **verified well-formed** -- it is ISO9660 +labelled `OPNSENSE_CFG` and contains `CONF/CONFIG.XML` with `` and `10.10.0.1` +inside (grepped from the raw image) -- and it is attached as a SATA cdrom. So the payload is +right and the Importer simply did not consume it. + +This is the mechanism `modules/opnsense-edge`'s own header has always flagged UNVERIFIED +("whether the Configuration Importer's ISO9660 support actually behaves as described once +booted for real"). The research is well-sourced; it had just never been exercised. Now it +has, and it did not work as assumed. + +**Do NOT guess the fix.** Read the Importer's own console output first -- it runs early in +boot and reports what it scanned. Untested hypothesis worth checking (NOT a conclusion): +ISO9660 8.3 uppercase naming vs the lowercase `/conf/config.xml` the Importer looks for. + +**Note the unanswered ping to the WAN address is NOT a fault** -- OPNsense blocks inbound on +WAN by default. Do not chase it. + +## Operational finding -- the boot log is unreadable to the agent (real gap) + +libvirt creates the serial log **`root:0600`** and **recreates it on every domain create**, so +reading the single most valuable artifact during a boot incident needs an interactive `sudo` +each time. This materially contributed to the original misdiagnosis. A default ACL on the +staging dir does NOT fix it: libvirt's `0600` creation mode sets the ACL **mask** to `---`, +which nullifies any named-user entry (measured: `user:jessea123:r-- #effective:---`). + +Proposed durable fix (NOT yet implemented/verified): have the runbook/module pre-create the +serial log `0644` before `tofu apply`, so libvirt appends to an already-readable file rather +than minting a `root:0600` one. Verify on a real recreate before trusting it. + +## Revert + +- Remove the `features` blocks from the three `libvirt_domain` resources (restores the panic + -- do not). +- Guard: drop `s2_check` + the `--static-only` rename in `scripts/opentofu-validate.sh`, and + `tests/opentofu-validate/fixtures/s2-bad/` + T5/T6/T7. +- Whole change: `git revert `. No live-state dependency. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 05b8396..baedb0b 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -1787,7 +1787,42 @@ it did. Four successive non-fixes were each recorded as a cause, and that accumulated fiction is what a fresh session would have inherited. A fault that is deterministic and immune to every knob means you have not yet touched the cause. -- **Numbers:** D next-free 112, DOCFIX next-free 190. +## Jumphost stream -- session 2026-07-12 (cont.): edge BOOTS. Both boot bugs closed (DOCFIX-190) + +- **LIVE APPLY EXECUTED (operator-gated, 3 recreates).** DOCFIX-188's memory fix went live: + `Max memory: 2048 KiB -> 2097152 KiB`, QEMU `-m size=2097152k`, domain `running (booted)` + not `paused`, serial 262 -> 3882 bytes. **Triple-fault CONFIRMED DEAD.** +- **SECOND BUG, found underneath it (DOCFIX-190): ACPI was OFF.** With RAM finally correct + the kernel reached interrupt init and PANICKED: `panic: running without device atpic + requires a local APIC` -> parked at ddb `db>` = the 100%-of-one-core spin, while `virsh` + still cheerfully reported `running`. Cause: NO `features` block in ANY of the 3 VM modules + -> libvirt renders `-machine ...,acpi=off`. FreeBSD gets its local APIC from ACPI's MADT + and OPNsense has no `atpic` fallback. FIXED: `features = { acpi = true, apic = {} }` in + opnsense-edge + cloudinit-vm + node-vm. **node-vm matters most long-term: MAAS drives + power via ACPI, so without it a graceful shutdown/MAAS power-off would NEVER work.** +- **GUARD S2** added to opentofu-validate (same reason as S1: `features` is optional, so + `tofu validate` structurally cannot see its absence). Flag renamed `--static-only` + (`--check-memory-unit` kept as alias). Harness 4 -> **6 PASS**; `fixtures/s2-bad` PASSES S1 + and still FAILS S2, proving the guards catch independent classes. +- **RESULT: OPNSENSE BOOTS AND RUNS.** acpi=on; kernel to userland; CPU idle (~19s, no spin); + **DHCP lease 172.30.1.126, hostname `OPNsense`** on office1-wan. `BIOS 639kB/2096108kB + available memory` re-confirms the 2 GiB. +- **STILL OPEN -- Configuration Importer did NOT apply.** OPNsense came up on FACTORY + DEFAULTS (WAN=DHCP) instead of static 172.30.1.2 / LAN 10.10.0.1. The ISO is VERIFIED + well-formed (ISO9660 `OPNSENSE_CFG`, contains `CONF/CONFIG.XML` with `` + + `10.10.0.1`) and attached as SATA cdrom -- so the payload is right, the Importer just did + not consume it. This is exactly the mechanism the module header has ALWAYS flagged + UNVERIFIED. **Do NOT guess the fix** -- read the Importer's own early-boot console output. + Untested hypothesis only: ISO9660 8.3 UPPERCASE vs the lowercase `/conf/config.xml` it + looks for. **The unanswered WAN ping is NOT a fault** -- OPNsense blocks inbound on WAN by + default; do not chase it. +- **OPS GAP (contributed to the original misdiagnosis):** libvirt creates the serial log + `root:0600` and RECREATES it on every domain create, so the most valuable boot-incident + artifact needs an interactive `sudo` every retry. A default ACL does NOT work: the 0600 + creation mode sets the ACL **mask** to `---`, nullifying the named-user entry (measured: + `user:jessea123:r-- #effective:---`). Proposed (NOT yet verified): pre-create the log 0644 + before `tofu apply` so libvirt appends to an already-readable file. +- **Numbers:** D next-free 112, DOCFIX next-free 191. diff --git a/opentofu/modules/cloudinit-vm/main.tf b/opentofu/modules/cloudinit-vm/main.tf index 83732c7..7e3b52f 100644 --- a/opentofu/modules/cloudinit-vm/main.tf +++ b/opentofu/modules/cloudinit-vm/main.tf @@ -77,6 +77,16 @@ type = "kvm" running = true + # ACPI/APIC: with no `features` block libvirt renders `acpi=off` -- see the + # opnsense-edge module for the full write-up (a FreeBSD guest PANICS outright). + # A Linux guest will usually still boot without ACPI, but degraded: no clean + # ACPI shutdown/reboot signalling, and unreliable CPU/IRQ enumeration. Every + # normal libvirt guest sets both; there is no reason for ours not to. + features = { + acpi = true + apic = {} + } + os = { type = "hvm" type_arch = "x86_64" diff --git a/opentofu/modules/node-vm/main.tf b/opentofu/modules/node-vm/main.tf index 04938a1..986acba 100644 --- a/opentofu/modules/node-vm/main.tf +++ b/opentofu/modules/node-vm/main.tf @@ -44,6 +44,16 @@ type = "kvm" running = true + # ACPI/APIC: with no `features` block libvirt renders `acpi=off` -- see the + # opnsense-edge module for the full write-up. Especially important here: these are + # the MAAS-managed node VMs, and MAAS drives power/shutdown via ACPI signalling. + # Without ACPI a graceful `virsh shutdown` (and therefore MAAS power-off) does not + # work -- the node would only ever hard-stop. + features = { + acpi = true + apic = {} + } + os = { type = "hvm" type_arch = "x86_64" diff --git a/opentofu/modules/opnsense-edge/main.tf b/opentofu/modules/opnsense-edge/main.tf index 6fb67c8..86ee676 100644 --- a/opentofu/modules/opnsense-edge/main.tf +++ b/opentofu/modules/opnsense-edge/main.tf @@ -98,6 +98,24 @@ type = "kvm" running = true + # ACPI IS REQUIRED. With no `features` block, libvirt renders the machine with + # `acpi=off`, and FreeBSD/OPNsense then panics on the FIRST interrupt setup: + # + # panic: running without device atpic requires a local APIC + # apic_init() at ... mi_startup() + # + # FreeBSD discovers the local APIC from ACPI's MADT table, and the OPNsense kernel + # carries no `atpic` fallback -- so no ACPI means no usable interrupt controller. The + # guest lands in the ddb debugger (`db>`), which presents as a domain that is "running" + # while burning 100% of a core and emitting nothing further on the console. + # + # This was the SECOND boot bug (2026-07-12), found only after the memory-unit fix let + # the kernel get far enough to reach interrupt init. See the incident report. + features = { + acpi = true + apic = {} + } + # host-passthrough with the AMD `svm` (nested-virt) feature disabled. NOTE: this was # tried as a triple-fault fix on 2026-07-12 and did NOT resolve it (the real cause was # the memory unit above). Retained because it is a legitimate nested-virt hardening for diff --git a/scripts/opentofu-validate.sh b/scripts/opentofu-validate.sh index 2077cf9..5e6af67 100644 --- a/scripts/opentofu-validate.sh +++ b/scripts/opentofu-validate.sh @@ -1,23 +1,29 @@ #!/usr/bin/env bash -# scripts/opentofu-validate.sh [--check-memory-unit] [tofu-dir] +# scripts/opentofu-validate.sh [--static-only] [tofu-dir] # -# Syntax/schema/semantic gate for opentofu/ (VR1 IaC, D-103). Read-only: runs a -# static semantic guard (S1) + `tofu fmt -check` + `tofu init -backend=false` + -# `tofu validate` against the root module. Does NOT plan or apply -- no provider +# Syntax/schema/semantic gate for opentofu/ (VR1 IaC, D-103). Read-only: runs the +# static semantic guards (S1, S2) + `tofu fmt -check` + `tofu init -backend=false` +# + `tofu validate` against the root module. Does NOT plan or apply -- no provider # connection, no libvirt reach required. This is the harness opentofu/README.md # tells you to run before trusting anything under opentofu/, since it was # authored in a session with no tofu binary available to self-check. # -# --check-memory-unit runs ONLY the S1 static guard and exits (no tofu binary, -# no network needed). That is how the harness exercises S1 cheaply. +# The S* guards catch libvirt-domain defects that `tofu validate` structurally +# CANNOT see (optional attributes whose ABSENCE is the bug). Both were paid for in +# blood by the 2026-07-12 Office1 OPNsense boot incident: +# S1 memory without memory_unit -> 1024x too little RAM (guest triple-faults) +# S2 domain without ACPI enabled -> FreeBSD panics; Linux loses clean shutdown # -# Exit: 0 clean | 1 S1/fmt/init/validate failed | 2 tofu binary not found. +# --static-only runs ONLY the S* guards and exits (no tofu binary, no network +# needed). That is how the harness exercises them cheaply. +# +# Exit: 0 clean | 1 S*/fmt/init/validate failed | 2 tofu binary not found. set -uo pipefail HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" MODE="full" -if [ "${1:-}" = "--check-memory-unit" ]; then - MODE="s1" +if [ "${1:-}" = "--static-only" ] || [ "${1:-}" = "--check-memory-unit" ]; then + MODE="static" # --check-memory-unit kept as a back-compat alias shift fi TOFU_DIR="${1:-$HERE/../opentofu}" @@ -62,6 +68,36 @@ return 0 } +# S2: every libvirt_domain MUST enable ACPI (`features = { acpi = true }`). +# +# Also invisible to `tofu validate` (the whole `features` block is optional). With no +# features block libvirt renders the machine `acpi=off`. A FreeBSD/OPNsense guest then +# PANICS at interrupt init ("running without device atpic requires a local APIC") and +# spins in ddb at 100% CPU while looking "running". A Linux guest boots but loses clean +# ACPI shutdown -- which breaks MAAS power control on the node VMs. Cost: the second half +# of the 2026-07-12 boot incident, found only once the S1 fix let the kernel get far +# enough to reach interrupt init. +s2_check() { + local out + out="$(find "$TOFU_DIR" -name '*.tf' -not -path '*/.terraform/*' -print0 2>/dev/null \ + | xargs -0 -r awk ' + /^resource "libvirt_domain"/ { inblk=1; acpi=0; rline=FNR; next } + inblk && /^[[:space:]]*acpi[[:space:]]*=[[:space:]]*true/ { acpi=1 } + inblk && /^}/ { + if (!acpi) + printf " [FAIL] %s:%d libvirt_domain does not enable ACPI -- libvirt renders acpi=off; FreeBSD guests panic, Linux guests lose clean shutdown\n", FILENAME, rline + inblk=0 + } + ')" + if [ -n "$out" ]; then + echo "$out" + echo ' FIX: add features = { acpi = true, apic = {} } to each domain listed above.' + return 1 + fi + echo " [PASS] every libvirt_domain enables ACPI" + return 0 +} + echo "== S1 libvirt_domain memory_unit guard ==" if s1_check; then s1_rc=0 @@ -69,8 +105,16 @@ s1_rc=1 fi -if [ "$MODE" = "s1" ]; then - exit "$s1_rc" +echo "== S2 libvirt_domain ACPI guard ==" +if s2_check; then + s2_rc=0 +else + s2_rc=1 +fi + +if [ "$MODE" = "static" ]; then + [ "$s1_rc" -eq 0 ] && [ "$s2_rc" -eq 0 ] && exit 0 + exit 1 fi if ! command -v tofu >/dev/null 2>&1; then @@ -78,7 +122,7 @@ exit 2 fi -fail="$s1_rc" +fail=$(( s1_rc | s2_rc )) echo "== tofu fmt -check -recursive ==" if ! tofu fmt -check -recursive -diff "$TOFU_DIR"; then diff --git a/tests/opentofu-validate/fixtures/s1-good/main.tf b/tests/opentofu-validate/fixtures/s1-good/main.tf index b2da60b..fdf5c6e 100644 --- a/tests/opentofu-validate/fixtures/s1-good/main.tf +++ b/tests/opentofu-validate/fixtures/s1-good/main.tf @@ -1,4 +1,4 @@ -# S1 fixture (GOOD): libvirt_domain sets `memory` AND `memory_unit`. +# S1+S2 fixture (GOOD): libvirt_domain sets `memory` AND `memory_unit`, AND enables ACPI. # Also pins the near-miss: `memory_backing` / `memory_tune` are DIFFERENT # attributes and must not be mistaken for `memory` by the guard's scan. resource "libvirt_domain" "good" { @@ -8,6 +8,11 @@ vcpu = 2 type = "kvm" running = true + + features = { + acpi = true + apic = {} + } } resource "libvirt_domain" "no_memory_at_all" { @@ -15,4 +20,8 @@ vcpu = 1 type = "kvm" running = true + + features = { + acpi = true + } } diff --git a/tests/opentofu-validate/fixtures/s2-bad/main.tf b/tests/opentofu-validate/fixtures/s2-bad/main.tf new file mode 100644 index 0000000..69876c6 --- /dev/null +++ b/tests/opentofu-validate/fixtures/s2-bad/main.tf @@ -0,0 +1,15 @@ +# S2 fixture (BAD): memory/memory_unit are correct, but the domain does NOT enable ACPI. +# libvirt then renders the machine `acpi=off`, and a FreeBSD/OPNsense guest PANICS at +# interrupt init ("running without device atpic requires a local APIC") and spins in ddb +# at 100% CPU while still reporting as "running". That was the SECOND half of the +# 2026-07-12 boot incident. scripts/opentofu-validate.sh S2 MUST reject this. +# +# Note this fixture PASSES S1 -- it exists to prove S2 catches its own class independently. +resource "libvirt_domain" "no_acpi" { + name = "s2-bad" + memory = 2048 + memory_unit = "MiB" + vcpu = 2 + type = "kvm" + running = true +} diff --git a/tests/opentofu-validate/run-tests.sh b/tests/opentofu-validate/run-tests.sh index f572b0c..804d19b 100644 --- a/tests/opentofu-validate/run-tests.sh +++ b/tests/opentofu-validate/run-tests.sh @@ -32,19 +32,27 @@ run 2 'tofu binary not found' "T2 tofu binary missing FAILS (rc 2)" "$TMP" fi -# --- S1: libvirt_domain memory_unit guard (2026-07-12 triple-fault regression) --- -# --check-memory-unit runs the static scan only: no tofu binary, no network. +# --- S1/S2: libvirt_domain static guards (both halves of the 2026-07-12 boot incident) --- +# --static-only runs the static scans only: no tofu binary, no network. run 1 'sets memory with no memory_unit' \ "T3 S1 REJECTS a libvirt_domain with memory and no memory_unit (rc 1)" \ - --check-memory-unit "$HERE/fixtures/s1-bad" + --static-only "$HERE/fixtures/s1-bad" run 0 'every libvirt_domain that sets memory also sets memory_unit' \ - "T4 S1 ACCEPTS memory+memory_unit, and a domain with no memory at all (rc 0)" \ - --check-memory-unit "$HERE/fixtures/s1-good" + "T4 S1+S2 ACCEPT memory+memory_unit+acpi, and a domain with no memory at all (rc 0)" \ + --static-only "$HERE/fixtures/s1-good" + +run 1 'does not enable ACPI' \ + "T5 S2 REJECTS a libvirt_domain with no ACPI (rc 1) -- and it PASSES S1, proving S2 is independent" \ + --static-only "$HERE/fixtures/s2-bad" + +run 0 'every libvirt_domain enables ACPI' \ + "T6 S1+S2 PASS against the real opentofu/ tree (all VM modules carry memory_unit + acpi)" \ + --static-only "$HERE/../../opentofu" run 0 'every libvirt_domain that sets memory also sets memory_unit' \ - "T5 S1 PASSES against the real opentofu/ tree (all VM modules carry memory_unit)" \ + "T7 back-compat: the old --check-memory-unit flag still works" \ --check-memory-unit "$HERE/../../opentofu" echo; echo "RESULT: PASS=$PASS FAIL=$FAIL"