Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260712-libvirt-memory-unit-rootcause.md

Changelog 2026-07-12 -- DOCFIX-188: ROOT CAUSE of the OPNsense boot triple-fault (libvirt memory_unit)

The 2026-07-12 boot incident is diagnosed. It was never a CPU, nesting, machine-type, or console problem. The Office1 OPNsense edge guest was running with 2 MiB of RAM.

Evidence (measured this session, read-only, every hop)

Hop Measured
Module input memory_mib = 2048 (opentofu/main.tf:105) -- intent: 2 GiB
Module wiring memory = var.memory_mib (modules/opnsense-edge/main.tf:82), no memory_unit
tofu state memory = 2048
Rendered libvirt XML <memory unit='KiB'>2048</memory>
virsh dominfo Max memory: 2048 KiB
Actual QEMU cmdline -m size=2048k -- i.e. 2 MiB

Why it presents as a BTX triple-fault: FreeBSD's boot2 is tiny and fits in 2 MiB, so it happily echoes /boot.config (the deterministic 262-byte serial capture) -- and then triple-faults the instant it hands off to /boot/loader, which does not fit. This is exactly why the fault was immune to every CPU/machine/disk/console change tried on 2026-07-12: none of them touched the cause.

The defect

dmacvicar/libvirt >= 0.9 changed memory semantics. Per the provider's own schema (tofu providers schema -json):

  • memory -- "interpreted in libvirt memory units (typically KiB unless a unit is specified elsewhere)"
  • memory_unit -- "Sets the unit for the domain's main memory value ... KiB, MiB, or GiB"

The old 0.8-era meaning of memory was MiB. The modules were authored against that assumption. With no memory_unit, libvirt defaults to KiB, so every VM this tree creates gets 1024x too little RAM.

Fix -- all three VM modules (identical one-line defect)

memory_unit = "MiB" added to the libvirt_domain resource in each:

  1. opentofu/modules/opnsense-edge/main.tf -- the Office1 edge (the live blocker).
  2. opentofu/modules/cloudinit-vm/main.tf -- MAAS / NetBox / GitBucket VMs (Stage 2, the next thing to be built; would have hit this identically).
  3. opentofu/modules/node-vm/main.tf -- DC node VMs (Stage 3).

Only opnsense-edge was ever instantiated, which is the only reason this surfaced there first. The memory_mib = 2048 value is UNCHANGED -- with the unit correct it now means the 2 GiB that was always intended.

Also corrected (same file, flagged not smuggled): the opnsense-edge comment asserting that disabling AMD svm was the "ROOT CAUSE of the first-boot triple-fault". That claim is disproven by this session's evidence (it never resolved the fault). The svm = disable setting itself is RETAINED -- it is legitimate nested-virt hardening for a guest with no business seeing svm -- but the comment now says it was tried and did not fix anything. Leaving a false root-cause claim in the tree would have misled the next session.

Guard -- so this cannot silently recur

tofu validate cannot catch this: memory_unit is optional, so omitting it is schema-valid. That is precisely how it shipped. Following the DOCFIX-137 precedent (make the standing lesson enforceable at the source):

  • scripts/opentofu-validate.sh gains S1: a static scan failing any libvirt_domain that sets memory without memory_unit. Runs first, needs no tofu binary and no network.
  • New --check-memory-unit flag runs S1 alone, so the harness can exercise it without paying for tofu init.
  • tests/opentofu-validate/ gains fixtures + T3/T4/T5. T3 is the negative test: the fixture is the exact defect that shipped, and S1 rejects it. T5 asserts the real tree is clean.

Verification

  • tests/opentofu-validate/run-tests.sh: 4 PASS / 0 FAIL (was 1 PASS).
  • scripts/opentofu-validate.sh: S1 PASS + fmt + init + validate Success.
  • scripts/repo-lint.sh: 0 fail (1 documented legacy warn).
  • scripts/run-tests-all.sh: ALL GREEN (52 harnesses).
  • tofu plan: 0 to add, 1 to change, 0 to destroy -- domain only; the 11 GiB direct-copy disk volume is NOT touched.

Live apply -- SEPARATELY GATED, NOT DONE IN THIS CHANGE

The repo fix above is module source only (no live mutation). Booting the edge is a separate operator-gated step. Do not trust the plan's "update in-place": max-boot-memory is create-time to libvirt, the same trap DOCFIX-187 recorded for machine/cpu (provider plans in-place, libvirt ignores it). The domain must be recreated:

cd opentofu && source ~/vr1-stage1.env
virsh -c qemu:///system destroy  office1-opnsense   # currently paused/faulted
virsh -c qemu:///system undefine office1-opnsense   # leaves volumes intact
tofu apply

Re-run tofu plan AFTER the undefine -- the 1 to change / in-place plan captured above is the pre-undefine plan and is NOT what will execute. Post-undefine, tofu refreshes, sees the domain gone, and switches to create. Confirm it then reads 1 to add, 0 to destroy (disk + config_seed volumes untouched by the recreate).

Pass criteria (do NOT declare success on tofu apply exiting 0):

  1. Authoritative: virsh dominfo office1-opnsense -> Max memory: 2097152 KiB (2 GiB). Anchor on this number; it is unambiguous. Do NOT anchor on the QEMU -m string -- with unit='MiB' libvirt may normalize and emit -m size=2097152k, which is success, not failure. (Stated as a guess here, not a measurement, until the apply runs.)
  2. The serial log grows past 262 bytes into the loader/kernel.
  3. The router actually works: WAN 172.30.1.2 reachable, LAN DHCP serving office1-local.

Pre-staged fallbacks (so a surprise does not cost another session):

  • If dominfo still shows the wrong size, the provider is not honouring memory_unit: fix is memory = var.memory_mib * 1024 (leaning on the KiB default), NOT more debugging.
  • If it clears 262 bytes and boots further but does not reach a working router, that is a sizing question, not a regression of this fix -- check OPNsense 26.1's documented minimum RAM (the incident report's "3 GB" is UNVERIFIED) and bump to 4096 if warranted. Clearing the triple-fault and reaching a working router are two separate bars.

Revert (per item)

  • Module fix: remove the memory_unit = "MiB" line from the three libvirt_domain resources (restores the 2 MiB bug -- do not).
  • Comment correction: git revert the doc hunk in modules/opnsense-edge/main.tf.
  • Guard: delete the S1 block + --check-memory-unit in scripts/opentofu-validate.sh, drop T3/T4/T5 and tests/opentofu-validate/fixtures/.
  • Whole change: git revert <sha>. Nothing here is live-state dependent.