diff --git a/docs/changelog-20260712-libvirt-memory-unit-rootcause.md b/docs/changelog-20260712-libvirt-memory-unit-rootcause.md new file mode 100644 index 0000000..db7ac64 --- /dev/null +++ b/docs/changelog-20260712-libvirt-memory-unit-rootcause.md @@ -0,0 +1,104 @@ +# Changelog 2026-07-12 -- DOCFIX-188: ROOT CAUSE of the OPNsense boot triple-fault (libvirt `memory_unit`) + +**The 2026-07-12 boot incident is diagnosed. It was never a CPU, nesting, machine-type, +or console problem.** The Office1 OPNsense edge guest was running with **2 MiB of RAM.** + +## Evidence (measured this session, read-only, every hop) + +| Hop | Measured | +|---|---| +| Module input | `memory_mib = 2048` (`opentofu/main.tf:105`) -- intent: 2 GiB | +| Module wiring | `memory = var.memory_mib` (`modules/opnsense-edge/main.tf:82`), **no `memory_unit`** | +| tofu state | `memory = 2048` | +| Rendered libvirt XML | `2048` | +| `virsh dominfo` | `Max memory: 2048 KiB` | +| **Actual QEMU cmdline** | **`-m size=2048k`** -- i.e. **2 MiB** | + +**Why it presents as a BTX triple-fault:** FreeBSD's `boot2` is tiny and fits in 2 MiB, so +it happily echoes `/boot.config` (the deterministic 262-byte serial capture) -- and then +triple-faults the instant it hands off to `/boot/loader`, which does not fit. This is +exactly why the fault was immune to every CPU/machine/disk/console change tried on +2026-07-12: none of them touched the cause. + +## The defect + +`dmacvicar/libvirt` **>= 0.9** changed `memory` semantics. Per the provider's own schema +(`tofu providers schema -json`): + +- `memory` -- *"interpreted in libvirt memory units (typically **KiB** unless a unit is + specified elsewhere)"* +- `memory_unit` -- *"Sets the unit for the domain's main memory value ... KiB, MiB, or GiB"* + +The old 0.8-era meaning of `memory` was MiB. The modules were authored against that +assumption. With no `memory_unit`, libvirt defaults to KiB, so **every VM this tree creates +gets 1024x too little RAM.** + +## Fix -- all three VM modules (identical one-line defect) + +`memory_unit = "MiB"` added to the `libvirt_domain` resource in each: + +1. `opentofu/modules/opnsense-edge/main.tf` -- the Office1 edge (**the live blocker**). +2. `opentofu/modules/cloudinit-vm/main.tf` -- MAAS / NetBox / GitBucket VMs (Stage 2, the + **next** thing to be built; would have hit this identically). +3. `opentofu/modules/node-vm/main.tf` -- DC node VMs (Stage 3). + +Only `opnsense-edge` was ever instantiated, which is the only reason this surfaced there +first. The `memory_mib = 2048` value is UNCHANGED -- with the unit correct it now means the +2 GiB that was always intended. + +**Also corrected (same file, flagged not smuggled):** the `opnsense-edge` comment asserting +that disabling AMD `svm` was the *"ROOT CAUSE of the first-boot triple-fault"*. That claim +is disproven by this session's evidence (it never resolved the fault). The `svm = disable` +setting itself is RETAINED -- it is legitimate nested-virt hardening for a guest with no +business seeing `svm` -- but the comment now says it was tried and did not fix anything. +Leaving a false root-cause claim in the tree would have misled the next session. + +## Guard -- so this cannot silently recur + +`tofu validate` **cannot** catch this: `memory_unit` is optional, so omitting it is +schema-valid. That is precisely how it shipped. Following the DOCFIX-137 precedent (make +the standing lesson enforceable at the source): + +- `scripts/opentofu-validate.sh` gains **S1**: a static scan failing any `libvirt_domain` + that sets `memory` without `memory_unit`. Runs first, needs no tofu binary and no network. +- New `--check-memory-unit` flag runs S1 alone, so the harness can exercise it without + paying for `tofu init`. +- `tests/opentofu-validate/` gains fixtures + T3/T4/T5. **T3 is the negative test**: the + fixture is the exact defect that shipped, and S1 rejects it. T5 asserts the real tree is + clean. + +## Verification + +- `tests/opentofu-validate/run-tests.sh`: **4 PASS / 0 FAIL** (was 1 PASS). +- `scripts/opentofu-validate.sh`: S1 PASS + fmt + init + validate **Success**. +- `scripts/repo-lint.sh`: **0 fail** (1 documented legacy warn). +- `scripts/run-tests-all.sh`: **ALL GREEN (52 harnesses)**. +- `tofu plan`: `0 to add, 1 to change, 0 to destroy` -- domain only; the 11 GiB direct-copy + disk volume is NOT touched. + +## Live apply -- SEPARATELY GATED, NOT DONE IN THIS CHANGE + +The repo fix above is module source only (no live mutation). Booting the edge is a separate +operator-gated step. **Do not trust the plan's "update in-place":** max-boot-memory is +create-time to libvirt, the same trap DOCFIX-187 recorded for `machine`/`cpu` (provider +plans in-place, libvirt ignores it). The domain must be recreated: + + cd opentofu && source ~/vr1-stage1.env + virsh -c qemu:///system destroy office1-opnsense # currently paused/faulted + virsh -c qemu:///system undefine office1-opnsense # leaves volumes intact + tofu apply + +**Pass criteria (do not declare success on `tofu apply` exiting 0):** +1. `virsh dominfo office1-opnsense` -> `Max memory: 2097152 KiB` (2 GiB), and the QEMU + cmdline shows `-m size=2048` (not `2048k`). +2. The serial log grows **past 262 bytes** into the loader/kernel. +3. The router actually works: WAN 172.30.1.2 reachable, LAN DHCP serving `office1-local`. + +## Revert (per item) + +- Module fix: remove the `memory_unit = "MiB"` line from the three `libvirt_domain` + resources (restores the 2 MiB bug -- do not). +- Comment correction: `git revert` the doc hunk in `modules/opnsense-edge/main.tf`. +- Guard: delete the S1 block + `--check-memory-unit` in `scripts/opentofu-validate.sh`, drop + T3/T4/T5 and `tests/opentofu-validate/fixtures/`. +- Whole change: `git revert `. Nothing here is live-state dependent. diff --git a/docs/incident-20260712-opnsense-edge-boot-triplefault.md b/docs/incident-20260712-opnsense-edge-boot-triplefault.md index 057342f..f98cacb 100644 --- a/docs/incident-20260712-opnsense-edge-boot-triplefault.md +++ b/docs/incident-20260712-opnsense-edge-boot-triplefault.md @@ -1,6 +1,35 @@ -# Incident 2026-07-12: Office1 OPNsense edge triple-faults at the BTX loader (UNRESOLVED) +# Incident 2026-07-12: Office1 OPNsense edge triple-faults at the BTX loader -**Status:** OPEN. The Office1 OPNsense edge VM is built and *starts*, but OPNsense +> ## ROOT CAUSE FOUND 2026-07-12 (DOCFIX-188) -- READ THIS FIRST, THEN STOP +> +> **The guest had 2 MiB of RAM.** Not a CPU, nesting, machine-type, or console problem. +> Everything below this box is the ORIGINAL (wrong) investigation, retained for the record. +> **Do not work the "ranked next steps" -- they all chase the wrong layer.** +> +> `dmacvicar/libvirt` >= 0.9 changed `memory` from MiB (the 0.8-era meaning the modules were +> written against) to **raw libvirt units, defaulting to KiB**. With no `memory_unit`, +> `memory = 2048` rendered `2048` -> QEMU **`-m size=2048k`** = +> **2 MiB**. Measured end-to-end: module input -> tofu state -> domain XML -> `virsh dominfo` +> (`Max memory: 2048 KiB`) -> the live QEMU cmdline. +> +> `boot2` is tiny and fits in 2 MiB, so it echoes `/boot.config` and *then* triple-faults +> handing off to `/boot/loader`, which does not fit. **That is why the fault was +> deterministic at exactly 262 bytes and immune to every CPU/machine/disk/console change +> tried -- none of them touched the cause.** +> +> **Fix:** `memory_unit = "MiB"` on the `libvirt_domain` in all three VM modules +> (`opnsense-edge`, `cloudinit-vm`, `node-vm` -- the same defect was latent in all of them +> and would have broken every future VR1 VM). Guarded against recurrence by +> `scripts/opentofu-validate.sh` S1. See +> `docs/changelog-20260712-libvirt-memory-unit-rootcause.md`. +> +> **Lesson for the next incident:** a bootloader that dies at a fixed byte offset, immune to +> every knob you turn, is a *resource* problem, not a CPU-feature problem. The domain XML and +> the QEMU cmdline are ground truth -- read them before theorising about nested virt. +> +> **Status:** root cause fixed in repo; live boot verification is the remaining gated step. + +**Original status (SUPERSEDED):** OPEN. The Office1 OPNsense edge VM is built and *starts*, but OPNsense triple-faults ~262 bytes into boot. Blocks the Office1 headend (router+DHCP for `office1-local`) and therefore the Office1 NetBox VM. Written at a context limit as a resume artifact -- a fresh session should read this + `docs/session-ledger.md` first. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 8a349ca..cb1c9a0 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -1714,6 +1714,49 @@ - **CONTEXT-LIMIT CHECKPOINT.** Session near max; resume in a fresh session from the incident report + this ledger. **Numbers:** D next-free 112, DOCFIX next-free 188. +## Jumphost stream -- session 2026-07-12 (cont.): BOOT INCIDENT ROOT-CAUSED (DOCFIX-188) + +- **The triple-fault was NEVER a CPU/nesting/machine/console problem. The guest had 2 MiB + of RAM.** `dmacvicar/libvirt` >=0.9 changed `memory` from MiB (0.8-era meaning, which the + modules were authored against) to raw libvirt units defaulting to **KiB**. No + `memory_unit` => `memory = 2048` rendered `2048` => QEMU + **`-m size=2048k`** = 2 MiB. Measured every hop: module input -> tofu state -> domain XML + -> `virsh dominfo` (`Max memory: 2048 KiB`) -> live QEMU cmdline. `boot2` fits in 2 MiB + and echoes `/boot.config`; `/boot/loader` does not fit -> deterministic 262-byte fault, + immune to every knob the prior window turned. +- **FOUNDATIONAL, not local to OPNsense:** the identical one-line defect was latent in ALL + THREE VM modules -- `opnsense-edge` (live blocker), `cloudinit-vm` (MAAS/NetBox/GitBucket, + the literal NEXT step), `node-vm` (Stage 3 DC nodes). Every VR1 VM would have gotten 1024x + too little RAM. Only opnsense-edge had been instantiated, which is why it alone detonated. + Same class as the apparmor finding: it gates every VR1 VM. +- **FIXED (DOCFIX-188):** `memory_unit = "MiB"` on all three domains; `memory_mib = 2048` + value untouched (it now means the 2 GiB always intended). Disproven `svm`-is-root-cause + comment in opnsense-edge CORRECTED (setting retained as legitimate hardening; the false + claim would have misled the next session). Incident report gains a READ-THIS-FIRST box so + nobody re-chases the CPU theories. +- **GUARDED (DOCFIX-188):** `tofu validate` CANNOT catch this (`memory_unit` is optional => + omitting it is schema-valid -- exactly how it shipped). Added `opentofu-validate.sh` **S1** + static guard + `--check-memory-unit` flag (no tofu binary/network needed) + harness T3/T4/T5. + **T3 is a true negative test:** the fixture IS the shipped defect and S1 rejects it. + Harness 1 -> 4 PASS. Gauntlet ALL GREEN (52). repo-lint 0 fail. +- **NEXT (operator-gated live step -- the ONLY live mutation pending):** recreate the edge. + `tofu plan` says `0 add / 1 change / 0 destroy` (disk volume safe) BUT **do not trust + "update in-place"** -- max-boot-memory is create-time to libvirt, the same trap DOCFIX-187 + logged for `machine`/`cpu`. Must be: `virsh destroy office1-opnsense; virsh undefine + office1-opnsense; tofu apply`. **Pass criteria (NOT "apply exited 0"):** (1) `virsh dominfo` + shows `Max memory: 2097152 KiB` and cmdline `-m size=2048`; (2) serial log grows past 262 + bytes; (3) router actually works -- WAN 172.30.1.2 up, LAN DHCP serving `office1-local`. + Then: the Office1 NetBox VM (which `cloudinit-vm` now no longer breaks on). +- **STANDING LESSON:** a bootloader dying at a FIXED byte offset, immune to every knob, is a + RESOURCE problem, not a CPU-feature problem. Read the domain XML + QEMU cmdline (ground + truth) before theorising about nested virt. +- **LOGGED-NOT-ACTIONED (DOCFIX candidate, no number consumed):** `run-tests-all.sh` leaves + the working tree dirty -- it `chmod +x`'s the `tests/*/fakebin/*` stubs, so a gauntlet run + produces **37 spurious `100644 -> 100755` mode-change diffs** (zero content change). Had to + `git checkout` them out of this session's commit. Fix is either commit the stubs 0755 or + have the harness restore the mode; either way a gauntlet run should not dirty the tree. +- **Numbers:** D next-free 112, DOCFIX next-free 189. + diff --git a/opentofu/modules/cloudinit-vm/main.tf b/opentofu/modules/cloudinit-vm/main.tf index d36b768..83732c7 100644 --- a/opentofu/modules/cloudinit-vm/main.tf +++ b/opentofu/modules/cloudinit-vm/main.tf @@ -66,11 +66,16 @@ } resource "libvirt_domain" "vm" { - name = var.vm_name - memory = var.memory_mib - vcpu = var.vcpu - type = "kvm" - running = true + name = var.vm_name + # memory_unit is REQUIRED -- see the opnsense-edge module for the full write-up. On + # dmacvicar/libvirt >=0.9, bare `memory` is interpreted in libvirt's default unit (KiB), + # NOT MiB, so omitting this gives the guest 1024x too little RAM and it triple-faults + # in its bootloader. (2026-07-12 incident.) + memory = var.memory_mib + memory_unit = "MiB" + vcpu = var.vcpu + type = "kvm" + running = true os = { type = "hvm" diff --git a/opentofu/modules/node-vm/main.tf b/opentofu/modules/node-vm/main.tf index cd73642..04938a1 100644 --- a/opentofu/modules/node-vm/main.tf +++ b/opentofu/modules/node-vm/main.tf @@ -33,11 +33,16 @@ } resource "libvirt_domain" "node" { - name = var.vm_name - memory = var.memory_mib - vcpu = var.vcpu - type = "kvm" - running = true + name = var.vm_name + # memory_unit is REQUIRED -- see the opnsense-edge module for the full write-up. On + # dmacvicar/libvirt >=0.9, bare `memory` is interpreted in libvirt's default unit (KiB), + # NOT MiB, so omitting this gives the guest 1024x too little RAM and it triple-faults + # in its bootloader. (2026-07-12 incident.) + memory = var.memory_mib + memory_unit = "MiB" + vcpu = var.vcpu + type = "kvm" + running = true os = { type = "hvm" diff --git a/opentofu/modules/opnsense-edge/main.tf b/opentofu/modules/opnsense-edge/main.tf index c9eeea6..4ee85fc 100644 --- a/opentofu/modules/opnsense-edge/main.tf +++ b/opentofu/modules/opnsense-edge/main.tf @@ -78,18 +78,23 @@ } resource "libvirt_domain" "vm" { - name = var.vm_name - memory = var.memory_mib - vcpu = var.vcpu - type = "kvm" - running = true + name = var.vm_name + # memory_unit is REQUIRED. On dmacvicar/libvirt >=0.9, `memory` is a raw libvirt + # value, NOT MiB (that was the 0.8-era meaning): with no unit libvirt defaults to + # KiB, so `memory = 2048` renders `-m size=2048k` = 2 MiB. That was the ACTUAL root + # cause of the 2026-07-12 boot triple-fault (measured: `virsh dominfo` Max memory + # 2048 KiB) -- boot2 fits in 2 MiB and echoes /boot.config, then dies handing off to + # /boot/loader, which does not. See docs/incident-20260712-opnsense-edge-boot-triplefault.md. + memory = var.memory_mib + memory_unit = "MiB" + vcpu = var.vcpu + type = "kvm" + running = true - # host-passthrough, but DISABLE the AMD `svm` (nested-virt) feature in the - # guest. ROOT CAUSE of the first-boot triple-fault (2026-07-12): on an AMD host - # (this one is Opteron_G3), passing `svm` to a FreeBSD/OPNsense guest makes its - # loader triple-fault -- a well-documented KVM bug. Equivalent to the forum's - # `-cpu host,-svm` fix. (qemu64 default also lacks features, so host-passthrough - # is still right; we just drop svm.) + # host-passthrough with the AMD `svm` (nested-virt) feature disabled. NOTE: this was + # tried as a triple-fault fix on 2026-07-12 and did NOT resolve it (the real cause was + # the memory unit above). Retained because it is a legitimate nested-virt hardening for + # a guest that has no business seeing svm, not because it fixed anything. cpu = { mode = "host-passthrough" features = [ diff --git a/scripts/opentofu-validate.sh b/scripts/opentofu-validate.sh index 63165f7..2077cf9 100644 --- a/scripts/opentofu-validate.sh +++ b/scripts/opentofu-validate.sh @@ -1,16 +1,25 @@ #!/usr/bin/env bash -# scripts/opentofu-validate.sh [tofu-dir] +# scripts/opentofu-validate.sh [--check-memory-unit] [tofu-dir] # -# Thin syntax/schema gate for opentofu/ (VR1 IaC, D-103). Read-only: runs -# `tofu fmt -check` + `tofu init -backend=false` + `tofu validate` against the -# root module. Does NOT plan or apply -- no provider connection, no libvirt -# reach required. This is the harness opentofu/README.md tells you to run -# before trusting anything under opentofu/, since it was authored in a -# session with no tofu binary available to self-check. +# Syntax/schema/semantic gate for opentofu/ (VR1 IaC, D-103). Read-only: runs a +# static semantic guard (S1) + `tofu fmt -check` + `tofu init -backend=false` + +# `tofu validate` against the root module. Does NOT plan or apply -- no provider +# connection, no libvirt reach required. This is the harness opentofu/README.md +# tells you to run before trusting anything under opentofu/, since it was +# authored in a session with no tofu binary available to self-check. # -# Exit: 0 clean | 1 fmt/init/validate failed | 2 tofu binary not found. +# --check-memory-unit runs ONLY the S1 static guard and exits (no tofu binary, +# no network needed). That is how the harness exercises S1 cheaply. +# +# Exit: 0 clean | 1 S1/fmt/init/validate failed | 2 tofu binary not found. set -uo pipefail HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +MODE="full" +if [ "${1:-}" = "--check-memory-unit" ]; then + MODE="s1" + shift +fi TOFU_DIR="${1:-$HERE/../opentofu}" if [ ! -d "$TOFU_DIR" ]; then @@ -18,12 +27,58 @@ exit 1 fi +# S1: every libvirt_domain that sets `memory` MUST also set `memory_unit`. +# +# `tofu validate` CANNOT catch this: memory_unit is an optional attribute, so +# omitting it is schema-valid. But on dmacvicar/libvirt >=0.9 a bare `memory` is +# interpreted in libvirt's DEFAULT unit (KiB), not MiB (the 0.8-era meaning), so +# `memory = 2048` silently yields a 2 MiB guest that triple-faults in its +# bootloader. That cost a full session on 2026-07-12 (Office1 OPNsense edge; see +# docs/incident-20260712-opnsense-edge-boot-triplefault.md). This is the guard +# that stops the next VM module from reintroducing it. +# +# Block-scan is safe because `tofu fmt -check` (below) enforces canonical layout: +# top-level resource braces at column 0, attributes at two-space indent. Run +# order puts S1 first so it also works with no tofu binary present. +s1_check() { + local out + out="$(find "$TOFU_DIR" -name '*.tf' -not -path '*/.terraform/*' -print0 2>/dev/null \ + | xargs -0 -r awk ' + /^resource "libvirt_domain"/ { inblk=1; mem=0; unit=0; rline=FNR; next } + inblk && /^ memory[[:space:]]*=/ { mem=1 } + inblk && /^ memory_unit[[:space:]]*=/ { unit=1 } + inblk && /^}/ { + if (mem && !unit) + printf " [FAIL] %s:%d sets memory with no memory_unit -- guest gets KiB, not MiB (1024x too little RAM)\n", FILENAME, rline + inblk=0 + } + ')" + if [ -n "$out" ]; then + echo "$out" + echo ' FIX: add memory_unit = "MiB" to each domain listed above.' + return 1 + fi + echo " [PASS] every libvirt_domain that sets memory also sets memory_unit" + return 0 +} + +echo "== S1 libvirt_domain memory_unit guard ==" +if s1_check; then + s1_rc=0 +else + s1_rc=1 +fi + +if [ "$MODE" = "s1" ]; then + exit "$s1_rc" +fi + if ! command -v tofu >/dev/null 2>&1; then echo "FAIL: tofu binary not found on PATH -- install OpenTofu, then re-run this script" exit 2 fi -fail=0 +fail="$s1_rc" echo "== tofu fmt -check -recursive ==" if ! tofu fmt -check -recursive -diff "$TOFU_DIR"; then diff --git a/tests/opentofu-validate/fixtures/s1-bad/main.tf b/tests/opentofu-validate/fixtures/s1-bad/main.tf new file mode 100644 index 0000000..067c049 --- /dev/null +++ b/tests/opentofu-validate/fixtures/s1-bad/main.tf @@ -0,0 +1,10 @@ +# S1 fixture (BAD): libvirt_domain sets `memory` with NO `memory_unit`. +# This is the exact defect that shipped the 2026-07-12 Office1 OPNsense +# 2 MiB guest. scripts/opentofu-validate.sh S1 MUST reject this. +resource "libvirt_domain" "bad" { + name = "s1-bad" + memory = 2048 + vcpu = 2 + type = "kvm" + running = true +} diff --git a/tests/opentofu-validate/fixtures/s1-good/main.tf b/tests/opentofu-validate/fixtures/s1-good/main.tf new file mode 100644 index 0000000..b2da60b --- /dev/null +++ b/tests/opentofu-validate/fixtures/s1-good/main.tf @@ -0,0 +1,18 @@ +# S1 fixture (GOOD): libvirt_domain sets `memory` AND `memory_unit`. +# Also pins the near-miss: `memory_backing` / `memory_tune` are DIFFERENT +# attributes and must not be mistaken for `memory` by the guard's scan. +resource "libvirt_domain" "good" { + name = "s1-good" + memory = 2048 + memory_unit = "MiB" + vcpu = 2 + type = "kvm" + running = true +} + +resource "libvirt_domain" "no_memory_at_all" { + name = "s1-no-memory" + vcpu = 1 + type = "kvm" + running = true +} diff --git a/tests/opentofu-validate/run-tests.sh b/tests/opentofu-validate/run-tests.sh index 7c2586b..f572b0c 100644 --- a/tests/opentofu-validate/run-tests.sh +++ b/tests/opentofu-validate/run-tests.sh @@ -32,5 +32,20 @@ run 2 'tofu binary not found' "T2 tofu binary missing FAILS (rc 2)" "$TMP" fi +# --- S1: libvirt_domain memory_unit guard (2026-07-12 triple-fault regression) --- +# --check-memory-unit runs the static scan only: no tofu binary, no network. + +run 1 'sets memory with no memory_unit' \ + "T3 S1 REJECTS a libvirt_domain with memory and no memory_unit (rc 1)" \ + --check-memory-unit "$HERE/fixtures/s1-bad" + +run 0 'every libvirt_domain that sets memory also sets memory_unit' \ + "T4 S1 ACCEPTS memory+memory_unit, and a domain with no memory at all (rc 0)" \ + --check-memory-unit "$HERE/fixtures/s1-good" + +run 0 'every libvirt_domain that sets memory also sets memory_unit' \ + "T5 S1 PASSES against the real opentofu/ tree (all VM modules carry memory_unit)" \ + --check-memory-unit "$HERE/../../opentofu" + echo; echo "RESULT: PASS=$PASS FAIL=$FAIL" [[ "$FAIL" -eq 0 ]] && { echo "ALL PASS"; exit 0; } || exit 1