Platform traps and version pins -- libvirt/KVM, MAAS+LXD, OPNsense

The substrate BELOW the cloud (the hypervisor, the provisioner, the edge routers) has its own version boundaries and its own silent-failure modes. Every item below is either (a) sourced to a vendor/upstream URL, or (b) MEASURED on this project and sourced to the repo's own D-NNN / changelog. Nothing here is generic advice: each entry exists because it cost, or would have cost, a session.

Companion files: references/opentofu-provider-docs.md (provider schema + fetch methodology), references/troubleshooting.md (method). Start at the verbatim-error index at the bottom if you have an error string in hand.

Version pins that actually matter (and why)

Charm channels/revisions are appendix-B's job, not this file's. This file covers only the substrate the charms sit on.

1. dmacvicar/libvirt provider (pinned 0.9.8)

1a. The 0.8 -> 0.9 break invalidates most examples you will find

v0.9.0 (2025-11-08) is a deliberate, compatibility-breaking rewrite on the plugin framework, with the design principle "HCL maps almost 1:1 to libvirt XML"; the legacy provider stayed on the v0.8 branch. v0.9.1 (2025-11-30) then code-generated the schema from libvirt's XML schema, renaming attributes again (unit -> memory_unit, os.arch -> os.type_arch, os.machine -> os.type_machine, os.kernel_args -> os.cmdline, camelCase -> snake_case throughout).

• https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.0
• https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.1

Consequence: blog posts, StackOverflow answers, and model memory almost all describe 0.8. Nested structures are attribute-style objects (os = { ... }, devices = { ... }), not HCL blocks. Confirm against docs/resources/*.md + examples/*.tf at the pinned tag, or tofu providers schema -json.

1b. Bare memory is KiB, not MiB -- 1024x too little RAM (DOCFIX-188)

The 0.9.1 migration guide states the rule explicitly: "Value/unit pairs are explicit -- whenever libvirt exposes a value with a unit attribute the provider now has two attributes (memory + memory_unit, capacity + capacity_unit, etc.). Leaving the unit unset lets libvirt use its default." (https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.1)

And libvirt's default is KiB: "The units for this value are determined by the optional attribute unit, which defaults to 'KiB'" (https://libvirt.org/formatdomain.html#memory-allocation).

So memory = 2048 with no memory_unit is 2 MiB, not 2 GiB. In 0.8, memory meant MiB -- that is exactly how this shipped here.

memory      = var.memory_mib
memory_unit = "MiB"      # REQUIRED. Never omit.

Signatures (all measured, 2026-07-12): QEMU cmdline -m size=2048k; virsh dominfo -> Max memory: 2048 KiB; a FreeBSD guest echoes /boot.config (262 bytes of serial output) and then triple-faults handing off to /boot/loader. tofu validate CANNOT catch it (the attribute is optional) -- scripts/opentofu-validate.sh check S1 does. Evidence: docs/changelog-20260712-libvirt-memory-unit-rootcause.md, docs/incident-20260712-opnsense-edge-boot-triplefault.md.

The provider's own current domain examples now carry memory_unit = "MiB" (docs/resources/domain.md) -- a fresh reader of the docs would get this right; a reader of any 0.8-era example would not.

1c. No features block => acpi=off => FreeBSD panics (DOCFIX-190)

The whole features attribute is optional, so omitting it is schema-valid -- and libvirt then renders the machine with acpi=off (measured on the QEMU cmdline: -machine pc-i440fx-noble,...,acpi=off).

features = {
  acpi = true
  apic = {}          # acpi is a Boolean; apic is a nested object
}

(schema: docs/resources/domain.md, features.acpi (Boolean), features.apic (Attributes).)

Three distinct consequences, only one of which is loud:

• FreeBSD/OPNsense panics (it finds the local APIC via ACPI's MADT and ships no atpic fallback): panic: running without device atpic requires a local APIC -> db> prompt -> domain shows running while burning 100% of one core forever.
• Linux guests boot but degrade: no clean ACPI shutdown/reboot signalling.
• MAAS drives power off/on via ACPI, so a MAAS-managed node VM without ACPI can only ever be hard-stopped. That would have been a nasty Stage-3 debug. Evidence: docs/changelog-20260712-libvirt-acpi-kernel-panic.md. Guard: scripts/opentofu-validate.sh check S2.

1d. No cpu block => generic CPU => NO svm => nested KVM impossible

Found 2026-07-13. With no cpu attribute, libvirt hands the guest a generic emulated CPU model with no virtualization flag. Measured: host is an AMD EPYC 9965, the default-CPU guest was handed an Opteron_G3. Nothing errors -- the guest just has no /dev/kvm and no svm in /proc/cpuinfo.

Ubuntu's own nested-virt guide states the guest-side requirement: set the guest cpu mode to host-model or host-passthrough for the guest to see svm/vmx (https://ubuntu.com/server/docs/how-to/virtualisation/enable-nested-virtualisation/).

cpu = {
  mode = "host-passthrough"
  # per-caller: pass svm through, or disable it as router hardening
  features = var.expose_nested_virt ? [] : [
    { name = "svm", policy = "disable" }
  ]
}

The host side is a SEPARATE requirement and bites at every level. The vcloud host is ITSELF a KVM guest (systemd-detect-virt -> kvm), so nesting must be enabled at each level, not just ours:

• check: cat /sys/module/kvm_amd/parameters/nested -> Y or 1 (Intel: /sys/module/kvm_intel/parameters/nested)
• persist: options kvm-amd nested=1 in a file under /etc/modprobe.d/ (same source as above)

Why this matters here (D-114): LXD virtual machines are qemu/KVM guests, so MAAS composing service VMs inside a site containment VM needs working nested KVM at that depth; and nova-compute on the DC nodes needs it one level deeper still. modules/node-vm still has the missing-cpu-block defect -- logged, NOT fixed, because DC1 is gated behind Office1 (docs/changelog-20260713-d114-voffice1-nested-virt.md).

1e. "will be updated in-place" DOES NOT mean "the guest stays up"

A plan line reading libvirt_domain.vm will be updated in-place bounced a live guest (measured 2026-07-13: uptime 8m36s -> 6s; ~30s with no routing and no DHCP on the Office1 edge). "In-place" is a statement about the Terraform RESOURCE, not about the domain.

Upstream confirms the mechanism: the v0.9.4 release notes describe domain update as "updating a domain (e.g. changing memory) would undefine and re-define it" (the 0.9.4 fix was only to preserve NVRAM/TPM files across that undefine), and v0.9.6 added "configurable update shutdown behavior so domain updates can request guest shutdown and wait with a configurable timeout before forcing a stop."

• https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.4
• https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.6

Rule: treat ANY tofu apply whose plan touches a libvirt_domain as an OUTAGE of that guest. Schedule and gate it as one. Read the plan's resource list before applying; 0 to change on the domains is the only assurance that a running guest is not about to bounce (docs/session-ledger.md, standing lesson 2).

Corollary (DOCFIX-187/DOCFIX-188): some domain attributes are CREATE-time to libvirt (max boot memory, machine, cpu). The provider will happily plan an in-place update that libvirt then ignores. If a changed value does not show up in virsh dominfo, destroy + undefine the domain and re-apply (volumes survive an undefine).

1f. Two more upstream behaviours worth knowing (from the release notes)

• capacity_unit = "GiB" on a volume caused "Provider produced inconsistent result after apply" on every apply before 0.9.4 (libvirt normalizes to bytes). Fixed in 0.9.4 -- if you see that error string, check the provider version first.
• tofu destroy of a dir pool used to DELETE the backing directory (StoragePoolDelete). Since 0.9.4 the directory is preserved by default, with an explicit destroy.delete = true to opt back in. Relevant to modules/dc-storage-pool. (https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.4)
• UNVERIFIED: the 0.9.4 notes name a destroy.shutdown.timeout option while docs/resources/domain.md documents destroy = { graceful, timeout }, and I could not find the 0.9.6 update-shutdown attribute documented at all. Before relying on either, read the real schema: tofu providers schema -json | jq '.provider_schemas[].resource_schemas.libvirt_domain'.

2. AppArmor blocks qemu on any non-default pool path

libvirt's stock abstractions/libvirt-qemu grants qemu the DEFAULT pool path (/var/lib/libvirt/images) only. A pool anywhere else is blocked by AppArmor even with perfect POSIX permissions, and the failure names nothing:

• libvirt/qemu says only: Could not open '<path>': Permission denied
• the actual denial is in the kernel log: apparmor="DENIED" operation="open" profile="libvirt-<domain-uuid>" name="<path>"
• the domain DEFINES fine and fails at START -- so tofu apply can "succeed" into a dead guest.

Upstream (still open, provider issue #920, with the same workaround): https://github.com/dmacvicar/terraform-provider-libvirt/issues/920

Fix, in the vendor-sanctioned override include (survives package upgrade -- do NOT edit the shipped abstraction):

# /etc/apparmor.d/local/abstractions/libvirt-qemu
/var/lib/libvirt/vr1/** rwk,

This is now installed by bash scripts/prereqs/install-apparmor-libvirt.sh (pool parent overridable via VR1_POOL_PARENT), wired into scripts/prereqs/install-all.sh and reported by check-prereqs.sh. It cost a session on 2026-07-12 (DOCFIX-186) because it existed only as hand-applied host state; docs/changelog-20260713-apparmor-libvirt-prereq.md.

3. MAAS + LXD

3a. MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7 (D-114)

Canonical's own announcement: LXD 6.7 "consolidates some API endpoints" that MAAS's pinned pylxd 2.3.5 cannot speak to (fixed in pylxd >= 2.3.9, not yet in a MAAS release). Verbatim guidance: "We recommend using LXD <=6.6 or the 5.21 LTS release until further notice." https://discourse.maas.io/t/maas-incompatibility-with-lxd-6-7/15749

VR0 runs LXD 5.21.4 and is therefore safe by accident of timing; D-114 makes it a decision. LXD 5.21 LTS is supported to June 2029 (https://canonical.com/blog/lxd_5-21-0_lts).

The live hazard is the snap auto-refresh, not the install. LXD's docs: "By default, installed snaps update automatically when new releases are published to the channel they're tracking"; the latest track "typically points to the latest feature release", is "a continuously rolling release track", and is "not recommended for general use"; a bare snap install lxd uses the most recent LTS track, "which is currently 5.21". https://canonical.com/lxd/docs/latest/reference/releases-snap/ => install/refresh explicitly on 5.21/stable and verify the tracked channel (snap list lxd, snap info lxd) before blaming MAAS for an LXD it can no longer talk to.

3b. LXD's own bridge runs a dnsmasq DHCP server -- MAAS says turn it off

"Bridges created by LXD are managed, which means that in addition to creating the bridge interface itself, LXD also sets up a local dnsmasq process to provide DHCP, IPv6 route announcements and DNS services to the network" (ipv4.dhcp defaults to true). https://canonical.com/lxd/docs/latest/reference/network_bridge/

MAAS's own LXD VM-host procedure therefore has you disable exactly that, because MAAS must be the DHCP/PXE authority for the machines it composes:

lxc network set lxdbr0 dns.mode=none
lxc network set lxdbr0 ipv4.dhcp=false
lxc network set lxdbr0 ipv6.dhcp=false

https://canonical.com/maas/docs/how-to-manage-machines

This is the concrete form of D-114's DHCP warning. On office1-local, OPNsense/Kea is authoritative (10.10.0.0/24, pool .100-.199). Two DHCP servers on one L2 is an intermittent, genuinely unpleasant failure. Never bridge lxdbr0 onto the site LAN without disabling its DHCP first.

3c. Two different LXDs in this cloud -- do not conflate them (D-114)

Conflating these caused a real, recorded error:

• Juju-created LXD CONTAINERS on the OpenStack nodes (lxd:N placements in bundle.yaml, 21 of them). MAAS never sees these and never needs to.
• A MAAS-registered LXD KVM host (the VR0 lxd machine) into which MAAS COMPOSES VMs (maas $PROFILE vm-host compose ...) -- VR0's tailscale machine is one. These ARE MAAS machines: enlisted, commissioned, deployed, powered, released.

D-114 extends the second pattern per site; the first is explicitly out of scope. Also note: MAAS only sees VMs in the LXD project the VM host was registered with -- VMs in other projects are invisible to it and unaffected (MAAS LP#1923251, https://bugs.launchpad.net/maas/+bug/1923251).

Composed-VM networking -- MEASURE, do not assume. A MAAS-bundled copy of the MAAS docs states: "When composing a virtual machine with LXD, MAAS uses either the 'maas' LXD profile, or (if that doesn't exist) the 'default' LXD profile. The profile is used to determine which bridge to use." (served by a MAAS instance's own docs: https://maas.cloud.cbh.kth.se/MAAS/docs/cli/how-to-use-lxd.html -- I could NOT locate this sentence on canonical.com/maas.io today; the docs site has been reorganized and the old URLs 404. Treat the profile->bridge mechanism as PLAUSIBLE-BUT-UNCONFIRMED against a primary source.) Since a composed machine that PXEs on the wrong L2 is the failure this predicts, do the cheap measurement instead of trusting either source: lxc profile show maas / lxc profile show default on the LXD host, and confirm the NIC's parent bridge before composing.

Because LXD VMs (not containers) are qemu/KVM guests, this pattern needs nested KVM at whatever depth the LXD host sits -- see 1d.

4. OPNsense 26.1 edge

4a. config.xml is GUI-owned. Do not hand-author it. (D-112, D-113)

The config-xml path is DELETED (template, renderer, ISO builder, and the module's config_seed volume + cdrom disk -- D-113 amendment, 2026-07-13). Configure the edge over the REST API (scripts/opnsense-api.sh); mint an API key with scripts/opnsense-bootstrap-apikey.sh. Three separate defects (no sshd, no console, an inert <dhcpd> block) all traced to hand-writing that file, and a full-config push DROPS ~667 migration-populated elements -- including the box's only two firewall pass rules -- surviving only on an UNDOCUMENTED self-heal at boot. If you think the API cannot express something, that is a finding to raise against D-113, not a reason to write XML.

Two traps that follow:

• A config ISO can NEVER be read on a nano image (D-112, root-caused in upstream opnsense/core:src/sbin/opnsense-importer: the importer probes for a read-only root and bootstrap_and_exit 0s on a pre-installed image before scanning any media). Any instruction to build one is a trap.
• Factory default: LAN 192.168.1.1/24, SSH disabled, all inbound blocked on WAN. An unanswered ping to the WAN address is NOT a fault.

4b. ISC dhcpd is GONE from core in 26.1 -- an <dhcpd> config is inert

26.1 release notes: "ISC-DHCP moves to a plugin. It will be automatically installed during upgrades. It is not installed on new installations because it is not being used, but you can still install and keep using it." Also: "Dnsmasq is now the default for DHCPv4 and DHCPv6 as well as RA out of the box." https://docs.opnsense.org/releases/CE_26.1.html

So on a fresh 26.1 image: an ISC <dhcpd> block configures nothing (that was DOCFIX-193), and the GUI default backend is dnsmasq, not Kea. This edge deliberately runs Kea (10.10.0.0/24, pool .100-.199, first real lease served to voffice1 2026-07-13).

4c. REST API: auth, endpoints, and the reconfigure step everyone forgets

• Auth is HTTP basic with an API key/secret pair, not the root password: "API access is part of the local user authentication system, but uses key/secret pairs to separate account information from machine to machine communication." URL pattern: https://<host>/api/<module>/<controller>/<command>. https://docs.opnsense.org/development/how-tos/api.html
• Kea DHCPv4 (exact paths): /api/kea/dhcpv4/get, /set, /add_subnet, /set_subnet/$uuid, /del_subnet/$uuid, /add_reservation, /set_reservation/$uuid, /get_reservation, /add_option; service control: /api/kea/service/reconfigure|restart|start|stop|status. A set* call only edits config -- it is NOT live until you POST /api/kea/service/reconfigure. A silently-unapplied change looks exactly like a change that did not take. https://docs.opnsense.org/development/api/core/kea.html
• Firewall rules have a self-rollback safety net -- use it. The filter controller exposes savepoint, apply/$rollback_revision, cancel_rollback/$rollback_revision, revert/$revision: "When calling savepoint() a new config revision will be created and the timestamp will be returned for later use. If the cancelRollback(savepoint) is not called within 60 seconds, the firewall will rollback to the previous state." For any remote rule change on a router you reach THROUGH that router, a savepoint means a mistake self-reverts instead of locking you out. https://docs.opnsense.org/development/api/core/firewall.html
• Do NOT re-implement vendor internals to bootstrap. OPNsense stores API secrets as crypt(secret,'$6$'); we deliberately do not mint keys offline -- a format drift would fail SILENTLY (keys that never authenticate). Call the vendor's own model instead (scripts/opnsense-bootstrap-apikey.sh). (D-113 amendment.)

4d. root's shell is tcsh -- $(...) dies QUIETLY over SSH

Measured on the Office1 edge: root's shell is tcsh. tcsh has no $(...) command substitution (it uses backquotes); a $(...) inside a quoted remote command is rejected with:

Illegal variable name.

It already cost a defect: a pre-install snapshot silently no-op'd and the install then ran with no rollback point behind it (docs/changelog-20260713-office1-dhcp-apply.md). Because the step was non-fatal, nothing stopped.

Always feed remote commands to sh -s:

ssh -i "$OPNSENSE_SSH_KEY" root@<edge> 'sh -s' <<'SH'
...POSIX shell here...
SH

(tcsh's substitution syntax: https://man.freebsd.org/cgi/man.cgi?query=tcsh)

Verbatim error -> cause index (grep this first)