# Platform traps and version pins -- libvirt/KVM, MAAS+LXD, OPNsense The substrate BELOW the cloud (the hypervisor, the provisioner, the edge routers) has its own version boundaries and its own silent-failure modes. Every item below is either (a) sourced to a vendor/upstream URL, or (b) MEASURED on this project and sourced to the repo's own D-NNN / changelog. Nothing here is generic advice: each entry exists because it cost, or would have cost, a session. Companion files: `references/opentofu-provider-docs.md` (provider schema + fetch methodology), `references/troubleshooting.md` (method). Start at the verbatim-error index at the bottom if you have an error string in hand. ## Version pins that actually matter (and why) | Thing | Pin | Why the pin exists | |---|---|---| | `dmacvicar/libvirt` provider | `0.9.8` | 0.9.0 was a compatibility-BREAKING rewrite; anything remembered from 0.8-era examples is wrong (see below) | | OpenTofu | `v1.12.3` (as-run) | the binary that actually executed Stage 1 | | LXD | **`5.21` LTS track** | MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7 (D-114; see below) | | MAAS | 3.7.2 (VR0) | the LXD ceiling above is a property of MAAS 3.6/3.7 | | OPNsense | 26.1 "Witty Woodpecker" | ISC-DHCP left core in 26.1 -- Kea/dnsmasq only (see below) | | Vault charm | `1.8/stable` | 1.16 is an incompatible charm, deliberately NOT an upgrade (appendix-B, D-068) | Charm channels/revisions are appendix-B's job, not this file's. This file covers only the substrate the charms sit on. --- ## 1. dmacvicar/libvirt provider (pinned 0.9.8) ### 1a. The 0.8 -> 0.9 break invalidates most examples you will find v0.9.0 (2025-11-08) is a deliberate, compatibility-breaking rewrite on the plugin framework, with the design principle "HCL maps almost 1:1 to libvirt XML"; the legacy provider stayed on the `v0.8` branch. v0.9.1 (2025-11-30) then code-generated the schema from libvirt's XML schema, renaming attributes again (`unit` -> `memory_unit`, `os.arch` -> `os.type_arch`, `os.machine` -> `os.type_machine`, `os.kernel_args` -> `os.cmdline`, camelCase -> snake_case throughout). - https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.0 - https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.1 **Consequence:** blog posts, StackOverflow answers, and model memory almost all describe 0.8. Nested structures are attribute-style objects (`os = { ... }`, `devices = { ... }`), not HCL blocks. Confirm against `docs/resources/*.md` + `examples/*.tf` at the pinned tag, or `tofu providers schema -json`. ### 1b. Bare `memory` is KiB, not MiB -- 1024x too little RAM (DOCFIX-188) The 0.9.1 migration guide states the rule explicitly: "Value/unit pairs are explicit -- whenever libvirt exposes a value with a unit attribute the provider now has two attributes (`memory` + `memory_unit`, `capacity` + `capacity_unit`, etc.). **Leaving the unit unset lets libvirt use its default.**" (https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.1) And libvirt's default is KiB: "The units for this value are determined by the optional attribute `unit`, which defaults to 'KiB'" (https://libvirt.org/formatdomain.html#memory-allocation). So `memory = 2048` with no `memory_unit` is **2 MiB**, not 2 GiB. In 0.8, `memory` meant MiB -- that is exactly how this shipped here. memory = var.memory_mib memory_unit = "MiB" # REQUIRED. Never omit. Signatures (all measured, 2026-07-12): QEMU cmdline `-m size=2048k`; `virsh dominfo` -> `Max memory: 2048 KiB`; a FreeBSD guest echoes `/boot.config` (262 bytes of serial output) and then triple-faults handing off to `/boot/loader`. `tofu validate` CANNOT catch it (the attribute is optional) -- `scripts/opentofu-validate.sh` check **S1** does. Evidence: `docs/changelog-20260712-libvirt-memory-unit-rootcause.md`, `docs/incident-20260712-opnsense-edge-boot-triplefault.md`. The provider's own current domain examples now carry `memory_unit = "MiB"` (`docs/resources/domain.md`) -- a fresh reader of the docs would get this right; a reader of any 0.8-era example would not. ### 1c. No `features` block => `acpi=off` => FreeBSD panics (DOCFIX-190) The whole `features` attribute is optional, so omitting it is schema-valid -- and libvirt then renders the machine with `acpi=off` (measured on the QEMU cmdline: `-machine pc-i440fx-noble,...,acpi=off`). features = { acpi = true apic = {} # acpi is a Boolean; apic is a nested object } (schema: `docs/resources/domain.md`, `features.acpi` (Boolean), `features.apic` (Attributes).) Three distinct consequences, only one of which is loud: - FreeBSD/OPNsense **panics** (it finds the local APIC via ACPI's MADT and ships no `atpic` fallback): `panic: running without device atpic requires a local APIC` -> `db>` prompt -> domain shows `running` while burning 100% of one core forever. - Linux guests boot but degrade: no clean ACPI shutdown/reboot signalling. - **MAAS drives power off/on via ACPI**, so a MAAS-managed node VM without ACPI can only ever be hard-stopped. That would have been a nasty Stage-3 debug. Evidence: `docs/changelog-20260712-libvirt-acpi-kernel-panic.md`. Guard: `scripts/opentofu-validate.sh` check **S2**. ### 1d. No `cpu` block => generic CPU => NO `svm` => nested KVM impossible Found 2026-07-13. With no `cpu` attribute, libvirt hands the guest a generic emulated CPU model with no virtualization flag. Measured: host is an **AMD EPYC 9965**, the default-CPU guest was handed an **`Opteron_G3`**. Nothing errors -- the guest just has no `/dev/kvm` and no `svm` in `/proc/cpuinfo`. Ubuntu's own nested-virt guide states the guest-side requirement: set the guest `cpu mode` to `host-model` or `host-passthrough` for the guest to see `svm`/`vmx` (https://ubuntu.com/server/docs/how-to/virtualisation/enable-nested-virtualisation/). cpu = { mode = "host-passthrough" # per-caller: pass svm through, or disable it as router hardening features = var.expose_nested_virt ? [] : [ { name = "svm", policy = "disable" } ] } **The host side is a SEPARATE requirement and bites at every level.** The vcloud host is ITSELF a KVM guest (`systemd-detect-virt` -> `kvm`), so nesting must be enabled at each level, not just ours: - check: `cat /sys/module/kvm_amd/parameters/nested` -> `Y` or `1` (Intel: `/sys/module/kvm_intel/parameters/nested`) - persist: `options kvm-amd nested=1` in a file under `/etc/modprobe.d/` (same source as above) Why this matters here (D-114): LXD **virtual machines** are qemu/KVM guests, so MAAS composing service VMs inside a site containment VM needs working nested KVM at that depth; and `nova-compute` on the DC nodes needs it one level deeper still. `modules/node-vm` still has the missing-`cpu`-block defect -- logged, NOT fixed, because DC1 is gated behind Office1 (`docs/changelog-20260713-d114-voffice1-nested-virt.md`). ### 1e. "will be updated in-place" DOES NOT mean "the guest stays up" A plan line reading `libvirt_domain.vm will be updated in-place` **bounced a live guest** (measured 2026-07-13: uptime 8m36s -> 6s; ~30s with no routing and no DHCP on the Office1 edge). "In-place" is a statement about the Terraform RESOURCE, not about the domain. Upstream confirms the mechanism: the v0.9.4 release notes describe domain update as "updating a domain (e.g. changing memory) **would undefine and re-define it**" (the 0.9.4 fix was only to preserve NVRAM/TPM files across that undefine), and v0.9.6 added "configurable update shutdown behavior so domain updates can request guest shutdown and wait with a configurable timeout before forcing a stop." - https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.4 - https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.6 **Rule: treat ANY `tofu apply` whose plan touches a `libvirt_domain` as an OUTAGE of that guest. Schedule and gate it as one.** Read the plan's resource list before applying; `0 to change` on the domains is the only assurance that a running guest is not about to bounce (`docs/session-ledger.md`, standing lesson 2). Corollary (DOCFIX-187/DOCFIX-188): some domain attributes are CREATE-time to libvirt (max boot memory, `machine`, `cpu`). The provider will happily plan an in-place update that libvirt then ignores. If a changed value does not show up in `virsh dominfo`, `destroy` + `undefine` the domain and re-apply (volumes survive an `undefine`). ### 1f. Two more upstream behaviours worth knowing (from the release notes) - `capacity_unit = "GiB"` on a volume caused **"Provider produced inconsistent result after apply"** on every apply before 0.9.4 (libvirt normalizes to bytes). Fixed in 0.9.4 -- if you see that error string, check the provider version first. - `tofu destroy` of a **`dir` pool** used to DELETE the backing directory (`StoragePoolDelete`). Since 0.9.4 the directory is preserved by default, with an explicit `destroy.delete = true` to opt back in. Relevant to `modules/dc-storage-pool`. (https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.4) - **UNVERIFIED:** the 0.9.4 notes name a `destroy.shutdown.timeout` option while `docs/resources/domain.md` documents `destroy = { graceful, timeout }`, and I could not find the 0.9.6 update-shutdown attribute documented at all. Before relying on either, read the real schema: `tofu providers schema -json | jq '.provider_schemas[].resource_schemas.libvirt_domain'`. ## 2. AppArmor blocks qemu on any non-default pool path libvirt's stock `abstractions/libvirt-qemu` grants qemu the DEFAULT pool path (`/var/lib/libvirt/images`) only. A pool anywhere else is blocked by AppArmor **even with perfect POSIX permissions**, and the failure names nothing: - libvirt/qemu says only: `Could not open '': Permission denied` - the actual denial is in the kernel log: `apparmor="DENIED" operation="open" profile="libvirt-" name=""` - the domain DEFINES fine and fails at START -- so `tofu apply` can "succeed" into a dead guest. Upstream (still open, provider issue #920, with the same workaround): https://github.com/dmacvicar/terraform-provider-libvirt/issues/920 Fix, in the vendor-sanctioned override include (survives package upgrade -- do NOT edit the shipped abstraction): # /etc/apparmor.d/local/abstractions/libvirt-qemu /var/lib/libvirt/vr1/** rwk, This is now installed by `bash scripts/prereqs/install-apparmor-libvirt.sh` (pool parent overridable via `VR1_POOL_PARENT`), wired into `scripts/prereqs/install-all.sh` and reported by `check-prereqs.sh`. It cost a session on 2026-07-12 (DOCFIX-186) because it existed only as hand-applied host state; `docs/changelog-20260713-apparmor-libvirt-prereq.md`. ## 3. MAAS + LXD ### 3a. MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7 (D-114) Canonical's own announcement: LXD 6.7 "consolidates some API endpoints" that MAAS's pinned **pylxd 2.3.5** cannot speak to (fixed in **pylxd >= 2.3.9**, not yet in a MAAS release). Verbatim guidance: "We recommend using LXD <=6.6 or the 5.21 LTS release until further notice." https://discourse.maas.io/t/maas-incompatibility-with-lxd-6-7/15749 VR0 runs LXD **5.21.4** and is therefore safe by accident of timing; D-114 makes it a decision. **LXD 5.21 LTS is supported to June 2029** (https://canonical.com/blog/lxd_5-21-0_lts). **The live hazard is the snap auto-refresh, not the install.** LXD's docs: "By default, installed snaps update automatically when new releases are published to the channel they're tracking"; the `latest` track "typically points to the latest feature release", is "a continuously rolling release track", and is "not recommended for general use"; a bare `snap install lxd` uses the most recent LTS track, "which is currently 5.21". https://canonical.com/lxd/docs/latest/reference/releases-snap/ => install/refresh explicitly on `5.21/stable` and verify the tracked channel (`snap list lxd`, `snap info lxd`) before blaming MAAS for an LXD it can no longer talk to. ### 3b. LXD's own bridge runs a dnsmasq DHCP server -- MAAS says turn it off "Bridges created by LXD are managed, which means that in addition to creating the bridge interface itself, LXD also sets up a local `dnsmasq` process to provide DHCP, IPv6 route announcements and DNS services to the network" (`ipv4.dhcp` defaults to **true**). https://canonical.com/lxd/docs/latest/reference/network_bridge/ MAAS's own LXD VM-host procedure therefore has you disable exactly that, because MAAS must be the DHCP/PXE authority for the machines it composes: lxc network set lxdbr0 dns.mode=none lxc network set lxdbr0 ipv4.dhcp=false lxc network set lxdbr0 ipv6.dhcp=false https://canonical.com/maas/docs/how-to-manage-machines This is the concrete form of D-114's DHCP warning. On `office1-local`, **OPNsense/Kea is authoritative** (10.10.0.0/24, pool .100-.199). Two DHCP servers on one L2 is an intermittent, genuinely unpleasant failure. Never bridge `lxdbr0` onto the site LAN without disabling its DHCP first. ### 3c. Two different LXDs in this cloud -- do not conflate them (D-114) Conflating these caused a real, recorded error: - **Juju-created LXD CONTAINERS** on the OpenStack nodes (`lxd:N` placements in `bundle.yaml`, 21 of them). MAAS never sees these and never needs to. - **A MAAS-registered LXD KVM host** (the VR0 `lxd` machine) into which MAAS **COMPOSES VMs** (`maas $PROFILE vm-host compose ...`) -- VR0's `tailscale` machine is one. These ARE MAAS machines: enlisted, commissioned, deployed, powered, released. D-114 extends the second pattern per site; the first is explicitly out of scope. Also note: MAAS only sees VMs in the LXD **project** the VM host was registered with -- VMs in other projects are invisible to it and unaffected (MAAS LP#1923251, https://bugs.launchpad.net/maas/+bug/1923251). **Composed-VM networking -- MEASURE, do not assume.** A MAAS-bundled copy of the MAAS docs states: "When composing a virtual machine with LXD, MAAS uses either the 'maas' LXD profile, or (if that doesn't exist) the 'default' LXD profile. The profile is used to determine which bridge to use." (served by a MAAS instance's own docs: https://maas.cloud.cbh.kth.se/MAAS/docs/cli/how-to-use-lxd.html -- **I could NOT locate this sentence on canonical.com/maas.io today; the docs site has been reorganized and the old URLs 404. Treat the profile->bridge mechanism as PLAUSIBLE-BUT-UNCONFIRMED against a primary source.**) Since a composed machine that PXEs on the wrong L2 is the failure this predicts, do the cheap measurement instead of trusting either source: `lxc profile show maas` / `lxc profile show default` on the LXD host, and confirm the NIC's parent bridge before composing. Because LXD **VMs** (not containers) are qemu/KVM guests, this pattern needs nested KVM at whatever depth the LXD host sits -- see 1d. ## 4. OPNsense 26.1 edge ### 4a. `config.xml` is GUI-owned. Do not hand-author it. (D-112, D-113) The config-xml path is **DELETED** (template, renderer, ISO builder, and the module's `config_seed` volume + cdrom disk -- D-113 amendment, 2026-07-13). Configure the edge over the **REST API** (`scripts/opnsense-api.sh`); mint an API key with `scripts/opnsense-bootstrap-apikey.sh`. Three separate defects (no sshd, no console, an inert `` block) all traced to hand-writing that file, and a full-config push DROPS ~667 migration-populated elements -- including the box's only two firewall pass rules -- surviving only on an UNDOCUMENTED self-heal at boot. If you think the API cannot express something, that is a finding to raise against D-113, not a reason to write XML. Two traps that follow: - **A config ISO can NEVER be read on a nano image** (D-112, root-caused in upstream `opnsense/core:src/sbin/opnsense-importer`: the importer probes for a read-only root and `bootstrap_and_exit 0`s on a pre-installed image before scanning any media). Any instruction to build one is a trap. - Factory default: LAN `192.168.1.1/24`, **SSH disabled**, **all inbound blocked on WAN**. An unanswered ping to the WAN address is NOT a fault. ### 4b. ISC dhcpd is GONE from core in 26.1 -- an `` config is inert 26.1 release notes: "ISC-DHCP moves to a plugin. It will be automatically installed during upgrades. **It is not installed on new installations because it is not being used**, but you can still install and keep using it." Also: "Dnsmasq is now the default for DHCPv4 and DHCPv6 as well as RA out of the box." https://docs.opnsense.org/releases/CE_26.1.html So on a fresh 26.1 image: an ISC `` block configures nothing (that was DOCFIX-193), and the GUI default backend is **dnsmasq**, not Kea. This edge deliberately runs **Kea** (10.10.0.0/24, pool .100-.199, first real lease served to `voffice1` 2026-07-13). ### 4c. REST API: auth, endpoints, and the reconfigure step everyone forgets - Auth is **HTTP basic with an API key/secret pair**, not the root password: "API access is part of the local user authentication system, but uses key/secret pairs to separate account information from machine to machine communication." URL pattern: `https:///api///`. https://docs.opnsense.org/development/how-tos/api.html - Kea DHCPv4 (exact paths): `/api/kea/dhcpv4/get`, `/set`, `/add_subnet`, `/set_subnet/$uuid`, `/del_subnet/$uuid`, `/add_reservation`, `/set_reservation/$uuid`, `/get_reservation`, `/add_option`; service control: `/api/kea/service/reconfigure|restart|start|stop|status`. **A `set*` call only edits config -- it is NOT live until you POST `/api/kea/service/reconfigure`.** A silently-unapplied change looks exactly like a change that did not take. https://docs.opnsense.org/development/api/core/kea.html - **Firewall rules have a self-rollback safety net -- use it.** The filter controller exposes `savepoint`, `apply/$rollback_revision`, `cancel_rollback/$rollback_revision`, `revert/$revision`: "When calling `savepoint()` a new config revision will be created and the timestamp will be returned for later use. **If the `cancelRollback(savepoint)` is not called within 60 seconds, the firewall will rollback to the previous state**." For any remote rule change on a router you reach THROUGH that router, a savepoint means a mistake self-reverts instead of locking you out. https://docs.opnsense.org/development/api/core/firewall.html - Do NOT re-implement vendor internals to bootstrap. OPNsense stores API secrets as `crypt(secret,'$6$')`; we deliberately do not mint keys offline -- a format drift would fail SILENTLY (keys that never authenticate). Call the vendor's own model instead (`scripts/opnsense-bootstrap-apikey.sh`). (D-113 amendment.) ### 4d. root's shell is tcsh -- `$(...)` dies QUIETLY over SSH Measured on the Office1 edge: root's shell is **tcsh**. tcsh has no `$(...)` command substitution (it uses backquotes); a `$(...)` inside a quoted remote command is rejected with: Illegal variable name. It already cost a defect: a pre-install snapshot silently no-op'd and the install then ran with no rollback point behind it (`docs/changelog-20260713-office1-dhcp-apply.md`). Because the step was non-fatal, nothing stopped. **Always feed remote commands to `sh -s`:** ssh -i "$OPNSENSE_SSH_KEY" root@ 'sh -s' <<'SH' ...POSIX shell here... SH (tcsh's substitution syntax: https://man.freebsd.org/cgi/man.cgi?query=tcsh) --- ## Verbatim error -> cause index (grep this first) | Exact string you see | Cause | Where | |---|---|---| | guest triple-faults right after echoing `/boot.config`; `virsh dominfo` -> `Max memory: 2048 KiB`; QEMU `-m size=2048k` | `libvirt_domain.memory` with no `memory_unit` -- KiB, not MiB | 1b | | `panic: running without device atpic requires a local APIC` (then `db>`, 100% of one core, domain "running") | no `features` block -> `acpi=off` | 1c | | guest has no `/dev/kvm`; `/proc/cpuinfo` has no `svm`; CPU model reads `Opteron_G3` on an EPYC host | no `cpu` block -> generic emulated CPU | 1d | | `Could not open '': Permission denied` (domain defines, then fails to START; nothing names AppArmor) -- check `dmesg` for `apparmor="DENIED" operation="open" profile="libvirt-"` | non-default libvirt pool path not granted in AppArmor | 2 | | `Provider produced inconsistent result after apply` on a volume with `capacity_unit` | provider < 0.9.4 | 1f | | plan says `will be updated in-place`, guest uptime resets anyway | provider undefines + redefines the domain | 1e | | `Illegal variable name.` from an `ssh root@ '...'` | root's shell is tcsh; `$(...)` unsupported | 4d | | OPNsense boots on FACTORY DEFAULTS with a correct config ISO attached; console shows `>>> Invoking import script 'importer'` and nothing more | the Configuration Importer can never fire on a nano image | 4a | | a Kea/firewall API `set` returns success but the box's behaviour does not change | no `service/reconfigure` (or no firewall `apply`) call | 4c | | MAAS cannot talk to an LXD VM host after an LXD update | LXD auto-refreshed past 6.6; MAAS 3.6/3.7 needs LXD <= 6.6 or 5.21 LTS | 3a | | a MAAS-composed VM gets a lease MAAS did not issue, or PXEs on the wrong L2 | `lxdbr0` runs its own dnsmasq (verified); the composing LXD profile's bridge is the other candidate (unconfirmed -- measure it) | 3b, 3c | --- ## NetBox (upstream, `netbox.baldurkeep.com`) ### `HTTP 403 Forbidden` on every API call -- but the SAME token works from `curl` **Symptom.** `urllib`/`requests`/`pynetbox` get `403 Forbidden` from the upstream NetBox on every endpoint, including ones a valid token plainly reads. `curl` with the identical token and URL returns `200`. It reads exactly like a bad or revoked token; it is not. **Cause.** Something in front of upstream NetBox (WAF / reverse proxy) filters on **User-Agent**. The default Python UA (`Python-urllib/3.12`) is blocked; `curl/8.5.0` is allowed. Measured 2026-07-13: same token, same URL, UA the only variable -- `None -> 403`, `curl/8.5.0 -> 200`. **Fix.** Send an accepted `User-Agent` header. This applies to **`pynetbox`** too (it sends `python-requests/...`), so the repo's NetBox tooling must set it: ```python nb = pynetbox.api(url, token=tok) nb.http_session.headers["User-Agent"] = "curl/8.5.0" ``` **Do not** diagnose this as a token/permission problem and go re-mint credentials -- that is the trap. Confirm with `curl` first; if curl works and Python does not, it is the UA. **Note:** this is an UPSTREAM-only trap so far. The Office1 sandbox NetBox (4.6.4, no proxy in front) accepts the default Python UA. Upstream is 4.5.8 and also uses **v1** (40-char, bare) tokens, while the sandbox is 4.6 and uses **v2** (`nbt_.`). Any tool pointed at BOTH must handle both auth shapes and the UA -- do not assume they are the same NetBox.