# Platform traps and version pins -- libvirt/KVM, MAAS+LXD, OPNsense

The substrate BELOW the cloud (the hypervisor, the provisioner, the edge
routers) has its own version boundaries and its own silent-failure modes.
Every item below is either (a) sourced to a vendor/upstream URL, or (b)
MEASURED on this project and sourced to the repo's own D-NNN / changelog.
Nothing here is generic advice: each entry exists because it cost, or would
have cost, a session.

Companion files: `references/opentofu-provider-docs.md` (provider schema +
fetch methodology), `references/troubleshooting.md` (method). Start at the
verbatim-error index at the bottom if you have an error string in hand.

## Version pins that actually matter (and why)

| Thing | Pin | Why the pin exists |
|---|---|---|
| `dmacvicar/libvirt` provider | `0.9.8` | 0.9.0 was a compatibility-BREAKING rewrite; anything remembered from 0.8-era examples is wrong (see below) |
| OpenTofu | live pin: CURRENT-STATE.md section 7 | Stage-1 as-run value is historical (git history) |
| LXD | **`5.21` LTS track** | MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7 (D-114; see below) |
| MAAS | 3.7.2 (VR0) | the LXD ceiling above is a property of MAAS 3.6/3.7 |
| OPNsense | live pin: CURRENT-STATE.md section 7 (26.7 as of 2026-07-18) | ISC-DHCP left core AT 26.1 -- Kea/dnsmasq only on every later release (see below) |
| Vault charm | `1.8/stable` | 1.16 is an incompatible charm, deliberately NOT an upgrade (appendix-B, D-068) |

Charm channels/revisions are appendix-B's job, not this file's. This file
covers only the substrate the charms sit on.

---

## 1. dmacvicar/libvirt provider (pinned 0.9.8)

### 1a. The 0.8 -> 0.9 break invalidates most examples you will find

v0.9.0 (2025-11-08) is a deliberate, compatibility-breaking rewrite on the
plugin framework, with the design principle "HCL maps almost 1:1 to libvirt
XML"; the legacy provider stayed on the `v0.8` branch. v0.9.1 (2025-11-30)
then code-generated the schema from libvirt's XML schema, renaming attributes
again (`unit` -> `memory_unit`, `os.arch` -> `os.type_arch`, `os.machine` ->
`os.type_machine`, `os.kernel_args` -> `os.cmdline`, camelCase -> snake_case
throughout).
- https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.0
- https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.1

**Consequence:** blog posts, StackOverflow answers, and model memory almost
all describe 0.8. Nested structures are attribute-style objects
(`os = { ... }`, `devices = { ... }`), not HCL blocks. Confirm against
`docs/resources/*.md` + `examples/*.tf` at the pinned tag, or
`tofu providers schema -json`.

### 1b. Bare `memory` is KiB, not MiB -- 1024x too little RAM (DOCFIX-188)

The 0.9.1 migration guide states the rule explicitly: "Value/unit pairs are
explicit -- whenever libvirt exposes a value with a unit attribute the
provider now has two attributes (`memory` + `memory_unit`, `capacity` +
`capacity_unit`, etc.). **Leaving the unit unset lets libvirt use its
default.**"
(https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.1)

And libvirt's default is KiB: "The units for this value are determined by the
optional attribute `unit`, which defaults to 'KiB'"
(https://libvirt.org/formatdomain.html#memory-allocation).

So `memory = 2048` with no `memory_unit` is **2 MiB**, not 2 GiB. In 0.8,
`memory` meant MiB -- that is exactly how this shipped here.

    memory      = var.memory_mib
    memory_unit = "MiB"      # REQUIRED. Never omit.

Signatures (all measured, 2026-07-12): QEMU cmdline `-m size=2048k`;
`virsh dominfo` -> `Max memory: 2048 KiB`; a FreeBSD guest echoes
`/boot.config` (262 bytes of serial output) and then triple-faults handing off
to `/boot/loader`. `tofu validate` CANNOT catch it (the attribute is
optional) -- `scripts/opentofu-validate.sh` check **S1** does.
Evidence: `docs/archive/changelogs/changelog-20260712-libvirt-memory-unit-rootcause.md`,
`docs/archive/incident-20260712-opnsense-edge-boot-triplefault.md`.

The provider's own current domain examples now carry `memory_unit = "MiB"`
(`docs/resources/domain.md`) -- a fresh reader of the docs would get this
right; a reader of any 0.8-era example would not.

### 1c. No `features` block => `acpi=off` => FreeBSD panics (DOCFIX-190)

The whole `features` attribute is optional, so omitting it is schema-valid --
and libvirt then renders the machine with `acpi=off` (measured on the QEMU
cmdline: `-machine pc-i440fx-noble,...,acpi=off`).

    features = {
      acpi = true
      apic = {}          # acpi is a Boolean; apic is a nested object
    }

(schema: `docs/resources/domain.md`, `features.acpi` (Boolean),
`features.apic` (Attributes).)

Three distinct consequences, only one of which is loud:
- FreeBSD/OPNsense **panics** (it finds the local APIC via ACPI's MADT and
  ships no `atpic` fallback): `panic: running without device atpic requires a
  local APIC` -> `db>` prompt -> domain shows `running` while burning 100% of
  one core forever.
- Linux guests boot but degrade: no clean ACPI shutdown/reboot signalling.
- **MAAS drives power off/on via ACPI**, so a MAAS-managed node VM without
  ACPI can only ever be hard-stopped. That would have been a nasty Stage-3
  debug.
Evidence: `docs/archive/changelogs/changelog-20260712-libvirt-acpi-kernel-panic.md`.
Guard: `scripts/opentofu-validate.sh` check **S2**.

### 1d. No `cpu` block => generic CPU => NO `svm` => nested KVM impossible

Found 2026-07-13. With no `cpu` attribute, libvirt hands the guest a generic
emulated CPU model with no virtualization flag. Measured: host is an **AMD
EPYC 9965**, the default-CPU guest was handed an **`Opteron_G3`**. Nothing
errors -- the guest just has no `/dev/kvm` and no `svm` in `/proc/cpuinfo`.

Ubuntu's own nested-virt guide states the guest-side requirement: set the
guest `cpu mode` to `host-model` or `host-passthrough` for the guest to see
`svm`/`vmx`
(https://ubuntu.com/server/docs/how-to/virtualisation/enable-nested-virtualisation/).

    cpu = {
      mode = "host-passthrough"
      # per-caller: pass svm through, or disable it as router hardening
      features = var.expose_nested_virt ? [] : [
        { name = "svm", policy = "disable" }
      ]
    }

**The host side is a SEPARATE requirement and bites at every level.** The
vcloud host is ITSELF a KVM guest (`systemd-detect-virt` -> `kvm`), so nesting
must be enabled at each level, not just ours:
- check: `cat /sys/module/kvm_amd/parameters/nested` -> `Y` or `1`
  (Intel: `/sys/module/kvm_intel/parameters/nested`)
- persist: `options kvm-amd nested=1` in a file under `/etc/modprobe.d/`
  (same source as above)

Why this matters here (D-114): LXD **virtual machines** are qemu/KVM guests,
so MAAS composing service VMs inside a site containment VM needs working
nested KVM at that depth; and `nova-compute` on the DC nodes needs it one
level deeper still. `modules/node-vm` still has the missing-`cpu`-block defect
-- logged, NOT fixed, because DC1 is gated behind Office1
(`docs/archive/changelogs/changelog-20260713-d114-voffice1-nested-virt.md`).

### 1e. "will be updated in-place" DOES NOT mean "the guest stays up"

A plan line reading `libvirt_domain.vm will be updated in-place` **bounced a
live guest** (measured 2026-07-13: uptime 8m36s -> 6s; ~30s with no routing and
no DHCP on the Office1 edge). "In-place" is a statement about the Terraform
RESOURCE, not about the domain.

Upstream confirms the mechanism: the v0.9.4 release notes describe domain
update as "updating a domain (e.g. changing memory) **would undefine and
re-define it**" (the 0.9.4 fix was only to preserve NVRAM/TPM files across
that undefine), and v0.9.6 added "configurable update shutdown behavior so
domain updates can request guest shutdown and wait with a configurable timeout
before forcing a stop."
- https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.4
- https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.6

**Rule: treat ANY `tofu apply` whose plan touches a `libvirt_domain` as an
OUTAGE of that guest. Schedule and gate it as one.** Read the plan's resource
list before applying; `0 to change` on the domains is the only assurance that
a running guest is not about to bounce
(`docs/session-ledger.md`, standing lesson 2).

Corollary (DOCFIX-187/DOCFIX-188): some domain attributes are CREATE-time to
libvirt (max boot memory, `machine`, `cpu`). The provider will happily plan an
in-place update that libvirt then ignores. If a changed value does not show up
in `virsh dominfo`, `destroy` + `undefine` the domain and re-apply (volumes
survive an `undefine`).

Second corollary (measured 2026-07-20/21): the undefine/redefine also
**REGENERATES every UNPINNED NIC MAC**. The plan reads `0 to add, N to change,
0 to destroy`, the apply succeeds -- and every touched domain now has brand-new
MACs. Anything that keys on those MACs is silently stranded: in the 2026-07-21
commissioning incident all 9 MAAS-enlisted nodes became UNKNOWN at PXE (no
"Performing PXE boot" event, silent 30-min timeouts), because a serial-console
retrofit apply had drifted every boot-interface MAC. **Rule: any domain whose
MACs are load-bearing (MAAS-enlisted nodes above all) carries config-pinned
MACs** -- `mac = { address = ... }` per interface; see `modules/node-vm`'s
`interface_macs` + `tests/node-vm`. VR-only trap: Roosevelt metal has fixed
MACs.

### 1f. Two more upstream behaviours worth knowing (from the release notes)

- `capacity_unit = "GiB"` on a volume caused **"Provider produced inconsistent
  result after apply"** on every apply before 0.9.4 (libvirt normalizes to
  bytes). Fixed in 0.9.4 -- if you see that error string, check the provider
  version first.
- `tofu destroy` of a **`dir` pool** used to DELETE the backing directory
  (`StoragePoolDelete`). Since 0.9.4 the directory is preserved by default,
  with an explicit `destroy.delete = true` to opt back in. Relevant to
  `modules/dc-storage-pool`.
  (https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.4)
- **UNVERIFIED:** the 0.9.4 notes name a `destroy.shutdown.timeout` option
  while `docs/resources/domain.md` documents `destroy = { graceful, timeout }`,
  and I could not find the 0.9.6 update-shutdown attribute documented at all.
  Before relying on either, read the real schema:
  `tofu providers schema -json | jq '.provider_schemas[].resource_schemas.libvirt_domain'`.

### 1g. `forward=bridge` networks + `br_netfilter`: a global FORWARD-drop silently kills bridged frames (D-125)

A `libvirt_network` with `forward = { mode = "bridge" }` (dmacvicar schema:
`bridge = { name = "<host-bridge>" }`, no `ips`, no NAT) attaches guest taps to a
**pre-existing** host bridge -- libvirt does NOT create the bridge, so it must
already exist (declare it in the host's netplan, not the tofu) or the network
fails to start. This is the Model B "bridge-in" WAN path (D-125): the inner
OPNsense WAN rides `br-vr1-dc0-wan` out through `vvr1-dc0`'s IP-less uplink NIC to
the single vcloud ISP NAT.

Two silent-failure traps ride along:

- **`br_netfilter` makes bridged frames traverse the L3 `FORWARD` chain.** So a
  GLOBAL forward-drop -- `iptables -P FORWARD DROP`, or an nftables forward hook
  with `policy drop` and no interface match -- silently kills the WAN bridge even
  though it is "just L2". Keep any DC-LOCAL forward-drop (SEC-010) **scoped to the
  transit interface** (`oifname/iifname "<transit>"`), never global. Note
  `net.ipv4.ip_forward=0` is L3-only and does NOT affect pure bridging -- do not
  reach for it expecting to gate a bridge, and do not assume `ip_forward=1` is
  required just because an older NAT-based design needed it.
- **Foreign-MAC egress across a nested bridge is unprovable until apply.** When the
  bridged guest is itself inside another guest (OPNsense inside `vvr1-dc0`), the
  outer NAT's `virbr` sees a foreign source MAC arriving on the containment VM's
  port; an anti-spoof/nwfilter drop there blackholes egress. `tofu validate` is
  clean regardless. TEST IN ISOLATION FIRST: attach a throwaway guest to the
  bridge, confirm it gets an address and pings out, BEFORE the real chain rides
  it. (D-125's fallback for this failure is double-NAT: masquerade at `vvr1-dc0`
  instead of relying on nested L2.)

## 2. AppArmor blocks qemu on any non-default pool path

libvirt's stock `abstractions/libvirt-qemu` grants qemu the DEFAULT pool path
(`/var/lib/libvirt/images`) only. A pool anywhere else is blocked by AppArmor
**even with perfect POSIX permissions**, and the failure names nothing:

- libvirt/qemu says only: `Could not open '<path>': Permission denied`
- the actual denial is in the kernel log:
  `apparmor="DENIED" operation="open" profile="libvirt-<domain-uuid>" name="<path>"`
- the domain DEFINES fine and fails at START -- so `tofu apply` can "succeed"
  into a dead guest.

Upstream (still open, provider issue #920, with the same workaround):
https://github.com/dmacvicar/terraform-provider-libvirt/issues/920

Fix, in the vendor-sanctioned override include (survives package upgrade --
do NOT edit the shipped abstraction):

    # /etc/apparmor.d/local/abstractions/libvirt-qemu
    /var/lib/libvirt/vr1/** rwk,

This is now installed by `bash scripts/prereqs/install-apparmor-libvirt.sh`
(pool parent overridable via `VR1_POOL_PARENT`), wired into
`scripts/prereqs/install-all.sh` and reported by `check-prereqs.sh`. It cost a
session on 2026-07-12 (DOCFIX-186) because it existed only as hand-applied host
state; `docs/archive/changelogs/changelog-20260713-apparmor-libvirt-prereq.md`.

## 3. MAAS + LXD

### 3a. MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7 (D-114)

Canonical's own announcement: LXD 6.7 "consolidates some API endpoints" that
MAAS's pinned **pylxd 2.3.5** cannot speak to (fixed in **pylxd >= 2.3.9**,
not yet in a MAAS release). Verbatim guidance: "We recommend using LXD <=6.6
or the 5.21 LTS release until further notice."
https://discourse.maas.io/t/maas-incompatibility-with-lxd-6-7/15749

VR0 runs LXD **5.21.4** and is therefore safe by accident of timing; D-114
makes it a decision. **LXD 5.21 LTS is supported to June 2029**
(https://canonical.com/blog/lxd_5-21-0_lts).

**The live hazard is the snap auto-refresh, not the install.** LXD's docs:
"By default, installed snaps update automatically when new releases are
published to the channel they're tracking"; the `latest` track "typically
points to the latest feature release", is "a continuously rolling release
track", and is "not recommended for general use"; a bare `snap install lxd`
uses the most recent LTS track, "which is currently 5.21".
https://canonical.com/lxd/docs/latest/reference/releases-snap/
=> install/refresh explicitly on `5.21/stable` and verify the tracked channel
(`snap list lxd`, `snap info lxd`) before blaming MAAS for an LXD it can no
longer talk to.

### 3b. LXD's own bridge runs a dnsmasq DHCP server -- MAAS says turn it off

"Bridges created by LXD are managed, which means that in addition to creating
the bridge interface itself, LXD also sets up a local `dnsmasq` process to
provide DHCP, IPv6 route announcements and DNS services to the network"
(`ipv4.dhcp` defaults to **true**).
https://canonical.com/lxd/docs/latest/reference/network_bridge/

MAAS's own LXD VM-host procedure therefore has you disable exactly that,
because MAAS must be the DHCP/PXE authority for the machines it composes:

    lxc network set lxdbr0 dns.mode=none
    lxc network set lxdbr0 ipv4.dhcp=false
    lxc network set lxdbr0 ipv6.dhcp=false

https://canonical.com/maas/docs/how-to-manage-machines

This is the concrete form of D-114's DHCP warning. On `office1-local`,
**OPNsense/Kea is authoritative** (10.10.0.0/24, pool .100-.199). Two DHCP
servers on one L2 is an intermittent, genuinely unpleasant failure. Never
bridge `lxdbr0` onto the site LAN without disabling its DHCP first.

### 3c. Two different LXDs in this cloud -- do not conflate them (D-114)

Conflating these caused a real, recorded error:
- **Juju-created LXD CONTAINERS** on the OpenStack nodes (`lxd:N` placements
  in `bundle.yaml`, 21 of them). MAAS never sees these and never needs to.
- **A MAAS-registered LXD KVM host** (the VR0 `lxd` machine) into which MAAS
  **COMPOSES VMs** (`maas $PROFILE vm-host compose ...`) -- VR0's `tailscale`
  machine is one. These ARE MAAS machines: enlisted, commissioned, deployed,
  powered, released.

D-114 extends the second pattern per site; the first is explicitly out of
scope. Also note: MAAS only sees VMs in the LXD **project** the VM host was
registered with -- VMs in other projects are invisible to it and unaffected
(MAAS LP#1923251, https://bugs.launchpad.net/maas/+bug/1923251).

**Composed-VM networking -- MEASURE, do not assume.** A MAAS-bundled copy of
the MAAS docs states: "When composing a virtual machine with LXD, MAAS uses
either the 'maas' LXD profile, or (if that doesn't exist) the 'default' LXD
profile. The profile is used to determine which bridge to use."
(served by a MAAS instance's own docs:
https://maas.cloud.cbh.kth.se/MAAS/docs/cli/how-to-use-lxd.html -- **I could
NOT locate this sentence on canonical.com/maas.io today; the docs site has been
reorganized and the old URLs 404. Treat the profile->bridge mechanism as
PLAUSIBLE-BUT-UNCONFIRMED against a primary source.**) Since a composed machine
that PXEs on the wrong L2 is the failure this predicts, do the cheap
measurement instead of trusting either source: `lxc profile show maas` /
`lxc profile show default` on the LXD host, and confirm the NIC's parent bridge
before composing.

Because LXD **VMs** (not containers) are qemu/KVM guests, this pattern needs
nested KVM at whatever depth the LXD host sits -- see 1d.

## 4. OPNsense edge (traps entered at 26.1; they persist on the live 26.7)

### 4a. `config.xml` is GUI-owned. Do not hand-author it. (D-112, D-113)

The config-xml path is **DELETED** (template, renderer, ISO builder, and the
module's `config_seed` volume + cdrom disk -- D-113 amendment, 2026-07-13).
Configure the edge over the **REST API** (`scripts/opnsense-api.sh`); mint an
API key with `scripts/opnsense-bootstrap-apikey.sh`. Three separate defects
(no sshd, no console, an inert `<dhcpd>` block) all traced to hand-writing
that file, and a full-config push DROPS ~667 migration-populated elements --
including the box's only two firewall pass rules -- surviving only on an
UNDOCUMENTED self-heal at boot. If you think the API cannot express something,
that is a finding to raise against D-113, not a reason to write XML.

Two traps that follow:
- **A config ISO can NEVER be read on a nano image** (D-112, root-caused in
  upstream `opnsense/core:src/sbin/opnsense-importer`: the importer probes for
  a read-only root and `bootstrap_and_exit 0`s on a pre-installed image before
  scanning any media). Any instruction to build one is a trap.
- Factory default: LAN `192.168.1.1/24`, **SSH disabled**, **all inbound
  blocked on WAN**. An unanswered ping to the WAN address is NOT a fault.

### 4b. ISC dhcpd is GONE from core in 26.1 -- an `<dhcpd>` config is inert

26.1 release notes: "ISC-DHCP moves to a plugin. It will be automatically
installed during upgrades. **It is not installed on new installations because
it is not being used**, but you can still install and keep using it." Also:
"Dnsmasq is now the default for DHCPv4 and DHCPv6 as well as RA out of the
box."
https://docs.opnsense.org/releases/CE_26.1.html

So on a fresh 26.1-or-later image: an ISC `<dhcpd>` block configures nothing (that was
DOCFIX-193), and the GUI default backend is **dnsmasq**, not Kea. This edge
deliberately runs **Kea** (10.10.0.0/24, pool .100-.199, first real lease
served to `voffice1` 2026-07-13).

### 4c. REST API: auth, endpoints, and the reconfigure step everyone forgets

- Auth is **HTTP basic with an API key/secret pair**, not the root password:
  "API access is part of the local user authentication system, but uses
  key/secret pairs to separate account information from machine to machine
  communication." URL pattern: `https://<host>/api/<module>/<controller>/<command>`.
  https://docs.opnsense.org/development/how-tos/api.html
- Kea DHCPv4 (exact paths):
  `/api/kea/dhcpv4/get`, `/set`, `/add_subnet`, `/set_subnet/$uuid`,
  `/del_subnet/$uuid`, `/add_reservation`, `/set_reservation/$uuid`,
  `/get_reservation`, `/add_option`; service control:
  `/api/kea/service/reconfigure|restart|start|stop|status`.
  **A `set*` call only edits config -- it is NOT live until you POST
  `/api/kea/service/reconfigure`.** A silently-unapplied change looks exactly
  like a change that did not take.
  https://docs.opnsense.org/development/api/core/kea.html
- **Firewall rules have a self-rollback safety net -- use it.** The filter
  controller exposes `savepoint`, `apply/$rollback_revision`,
  `cancel_rollback/$rollback_revision`, `revert/$revision`: "When calling
  `savepoint()` a new config revision will be created and the timestamp will be
  returned for later use. **If the `cancelRollback(savepoint)` is not called
  within 60 seconds, the firewall will rollback to the previous state**."
  For any remote rule change on a router you reach THROUGH that router, a
  savepoint means a mistake self-reverts instead of locking you out.
  https://docs.opnsense.org/development/api/core/firewall.html
- Do NOT re-implement vendor internals to bootstrap. OPNsense stores API
  secrets as `crypt(secret,'$6$')`; we deliberately do not mint keys offline --
  a format drift would fail SILENTLY (keys that never authenticate). Call the
  vendor's own model instead (`scripts/opnsense-bootstrap-apikey.sh`).
  (D-113 amendment.)

### 4d. root's shell is tcsh -- `$(...)` dies QUIETLY over SSH

Measured on the Office1 edge: root's shell is **tcsh**. tcsh has no `$(...)`
command substitution (it uses backquotes); a `$(...)` inside a quoted remote
command is rejected with:

    Illegal variable name.

It already cost a defect: a pre-install snapshot silently no-op'd and the
install then ran with no rollback point behind it
(`docs/archive/changelogs/changelog-20260713-office1-dhcp-apply.md`). Because the step was
non-fatal, nothing stopped.

**Always feed remote commands to `sh -s`:**

    ssh -i "$OPNSENSE_SSH_KEY" root@<edge> 'sh -s' <<'SH'
    ...POSIX shell here...
    SH

(tcsh's substitution syntax: https://man.freebsd.org/cgi/man.cgi?query=tcsh)

---

## 5. systemd timers -- "stopped" is not "dormant" across a reboot

Measured on the dc1 rack, 2026-07-27, while auditing a debmirror that had been paused
by ruling three days earlier.

**5a. A STOPPED-but-ENABLED timer re-arms at boot, and `Persistent=yes` makes it fire
IMMEDIATELY.** The state that looks dormant:

```
systemctl is-enabled <unit>.timer   -> enabled
systemctl is-active  <unit>.timer   -> inactive (dead)
systemctl show <unit>.timer -p NextElapseUSecRealtime  -> (EMPTY)
systemctl show <unit>.timer -p Persistent              -> yes
```

An empty `NextElapseUSecRealtime` is the trap: nothing is scheduled *right now*, so the
unit reads harmless and `systemctl list-timers` shows `NEXT: -`. But `enabled` means
systemd STARTS the timer at the next boot, and `Persistent=yes` means a timer whose
scheduled time has already elapsed triggers **at once** on activation to catch up. So a
job paused days ago does not wait for its next window -- it runs seconds after boot. On
this project that meant a multi-hundred-GB debmirror on a rack whose ruled artifact path
had changed, and the vector is recurring: DC racks live inside containment VMs on the
vcloud host, so a vcloud reboot reboots every rack.

Corollaries worth keeping:
- **`systemctl reset-failed` on the SERVICE does not re-arm the TIMER.** If the timer is
  inactive, clearing the service's failed state changes nothing schedulable. An earlier
  reading of this same incident blamed `reset-failed` and was wrong; the vector is the
  reboot (correction appended to `docs/audit/stage4-mirror-gate-20260727.txt`).
- **`Result=signal` + `ExecMainStatus=15` on a oneshot means someone SIGTERM'd it**, i.e.
  it was killed mid-run and never wrote a terminal state -- which is also why its status
  file can sit at `RUNNING` indefinitely. Do not read a stale `RUNNING` as in-flight;
  cross-check `systemctl is-active`.
- To make a pause DECLARED rather than incidental: `systemctl disable --now <unit>.timer`
  (survives reboot) or `mask` (also survives a later `enable`, e.g. from an idempotent
  installer re-running). Leaving it merely stopped encodes the intent nowhere.

**5b. A `oneshot` with `RemainAfterExit=yes` does not undo its work when stopped.** So
`systemctl stop <net-unit>` proves NOTHING about whether something else still depends on
the addresses/routes it placed. To test a dependency for real, remove the live state
(`ip addr del`, `ip route del`) and re-apply from the unit you believe owns it. Stopping
the unit and seeing everything still work is a false green.

**5c. `systemctl show` IS NOT AN EXISTENCE CHECK -- it fabricates a healthy-looking
oneshot for a unit that does not exist.** Measured 2026-08-02 on the dc0 rack against the
invented name `dc0-rack-net` (the real units are `dc0-rack-legs` and `dc0-node-dns`,
defined at `scripts/dc-rack-net.sh:16,22`):

```
systemctl show dc0-rack-net -p Type -p RemainAfterExit -p Result
  Type=
  UnitFileState=
  RemainAfterExit=no
  Result=success          <-- success. For a unit that has never existed.
  ExecMainStatus=0
```

`show` answers from property DEFAULTS, so a typo'd or renamed unit returns
`Result=success` / `ExecMainStatus=0` and no journal entries -- which reads exactly like
a oneshot that ran cleanly and is now idle. On this project it produced a false
"inactive" reading that was reported before it was caught.

Use an instrument that ERRORS on absence:
- `systemctl cat <unit>` -- exits non-zero with `No files found for <unit>`.
- `systemctl list-unit-files '<unit>'` -- returns 0 entries.
- or, if you must use `show`, assert `UnitFileState` is NON-EMPTY before reading any
  other property. The empty string is the only field that betrays the absence.

Same class as the standing instrument-currency lesson: take unit names from the script
that defines them, never from memory, and prove your instrument can report absence
before you trust it to report health.

---

## 6. MAAS IPAM -- two behaviours that silently invert an expectation

Both MEASURED 2026-07-27 while carving IPv6 into the VR1 DC planes.

### 6a. MAAS ALREADY reserves the whole low IPv6 block on every v6 subnet

Creating an explicit reserved iprange inside a v6 subnet FAILS:

```
{"start_ip": ["Requested reserved range conflicts with an existing range."],
 "end_ip":   ["Requested reserved range conflicts with an existing range."]}
```

...and `maas <profile> ipranges read` shows **zero** v6 ranges, so the "existing range"
is invisible there. It is MAAS's own default, visible only via:

```
maas <profile> subnet reserved-ip-ranges <subnet_id>
  ::            - ::            purpose ['rfc-4291-2.6.1']   <- subnet-router anycast
  ::1           - ::ffff:ffff   purpose ['reserved']         <- the whole first 2^32
```

Allocatable space begins only at `<prefix>:0:1::`. **CONSEQUENCE: a v4-style "reserve the
VIP/utility band" pass is both impossible AND unnecessary on IPv6** -- the addresses are
already protected, far more broadly than any band you would have written. Verify the
property instead of writing a range. This inverted a ruled execution plan (D-134's R4
amendment sequenced "v6 bands as a second pass"); the ruling's INTENT was satisfied by
MAAS's default, and only the mechanism changed.

### 6b. `mode=static` means EXPLICITLY CONFIGURED -- a subnet does not address anything

Creating a subnet gives nodes NOTHING. Every address a node holds under `mode=static` was
written by an explicit `interface link-subnet ... ip_address=<addr>` call. Measured: after
carving 12 v6 plane subnets, all 18 Ready nodes still reported **zero IPv6 links**, while
their v4 links (set by an earlier carve) were all present and octet-mirrored.

So "MAAS assigns the address" is only true if something assigned it. Ruling that node
addressing is "MAAS static assignment" does NOT make addresses appear -- it names the
mechanism, and the carve is still owed (18 nodes x 6 planes = 108 links here). Read
`interface_set[].links[]` before believing a subnet is in use.

## NetBox -- the same object reads differently live vs in a dump

`scope.name` from the LIVE API is the DISPLAY name; `scope.slug` is the key:

```
live:  "scope": {"name": "VR1 DC0", "slug": "vr1-dc0", ...}
dump:  "scope": {"name": "vr1-dc0"}          <- this repo's dumpers NORMALISE name to the slug
```

A tool written against a dump and pointed at the live API matches **zero** objects. Match
`slug` first with `name` as fallback so both shapes work. Related lesson from the same
session: a tool that REFUSES on "found nothing" surfaces this immediately, where one that
treats zero matches as "nothing to do" would have written nothing and reported success.

## Juju -- `deploy --dry-run` on a bundle shows FAR less than a plan reader expects

**MEASURED 2026-08-02, juju 3.6.27, a 56-application bundle plus three overlays**
(`docs/audit/stage5-dryrun-dc0-20260802.txt`). This generalizes past any one gate: if you
are writing an assertion against a bundle dry-run ANYWHERE in the chain, check first that
the property is even printed.

`juju deploy <bundle> --overlay ... --dry-run` prints ONLY:

- `Located charm "<name>" in charm-hub, channel <ch>`
- `- upload charm ... / - deploy application ...`
- `- add new machine N` -- **the BARE id, with no constraints**
- `- add unit <app>/N to <machine|container>` / `- add lxd container ...`
- `- add relation a:x - b:y`

It does **NOT** print application options, VIPs, or machine constraints. Adding `--debug`
adds exactly ONE thing: the per-machine constraint lines (`bundlehandler.go:1035`,
`with constraints "arch=amd64 tags=..."`). **Option values are absent at every
verbosity** -- greps for `bridge-interface-mappings`, `physnet1`, `openstack-origin`,
`cloud-archive`, `prefer-chassis` all returned 0 on the `--debug` capture. Beware two
decoys when grepping: the controller's own API addresses look like VIPs from the DC's
band, and `--debug` echoes the full command line, so the overlay FILENAMES appear as hits.

**Consequences to carry:**

1. **A dry-run cannot verify an overlay's `options` merged.** Whether juju merges an
   `options` map key-by-key or REPLACES it is invisible here -- and no offline merger can
   settle it either, because a hand-written deep-merge merges key-by-key by construction
   and so cannot distinguish the two behaviours. **The only instrument is `juju config
   <app>` on the deployed model.** Run it the moment `deploy` returns, before convergence,
   while a `juju config` correction is still cheap.
2. **If your gate reads tags/constraints, `--debug` is load-bearing, not noise.** A gate
   naming a bare `--dry-run` while asserting on tags cannot fail.
3. This is how three assertions in one repo gate came to be ungradable -- one for months,
   two written and removed within the same hour. Assert what the command PRINTS; measure
   the print, do not assume it from the command's name.

## Verbatim error -> cause index (grep this first)

| Exact string you see | Cause | Where |
|---|---|---|
| guest triple-faults right after echoing `/boot.config`; `virsh dominfo` -> `Max memory: 2048 KiB`; QEMU `-m size=2048k` | `libvirt_domain.memory` with no `memory_unit` -- KiB, not MiB | 1b |
| `panic: running without device atpic requires a local APIC` (then `db>`, 100% of one core, domain "running") | no `features` block -> `acpi=off` | 1c |
| guest has no `/dev/kvm`; `/proc/cpuinfo` has no `svm`; CPU model reads `Opteron_G3` on an EPYC host | no `cpu` block -> generic emulated CPU | 1d |
| `Could not open '<path>': Permission denied` (domain defines, then fails to START; nothing names AppArmor) -- check `dmesg` for `apparmor="DENIED" operation="open" profile="libvirt-<uuid>"` | non-default libvirt pool path not granted in AppArmor | 2 |
| `Provider produced inconsistent result after apply` on a volume with `capacity_unit` | provider < 0.9.4 | 1f |
| plan says `will be updated in-place`, guest uptime resets anyway | provider undefines + redefines the domain | 1e |
| MAAS commissioning times out at 30 min, NO PXE event for a known machine | an in-place apply regenerated unpinned NIC MACs; MAAS records stale | 1e |
| `Illegal variable name.` from an `ssh root@<edge> '...'` | root's shell is tcsh; `$(...)` unsupported | 4d |
| OPNsense boots on FACTORY DEFAULTS with a correct config ISO attached; console shows `>>> Invoking import script 'importer'` and nothing more | the Configuration Importer can never fire on a nano image | 4a |
| a Kea/firewall API `set` returns success but the box's behaviour does not change | no `service/reconfigure` (or no firewall `apply`) call | 4c |
| MAAS cannot talk to an LXD VM host after an LXD update | LXD auto-refreshed past 6.6; MAAS 3.6/3.7 needs LXD <= 6.6 or 5.21 LTS | 3a |
| a MAAS-composed VM gets a lease MAAS did not issue, or PXEs on the wrong L2 | `lxdbr0` runs its own dnsmasq (verified); the composing LXD profile's bridge is the other candidate (unconfirmed -- measure it) | 3b, 3c |
| a timer reads `is-enabled=enabled` but `is-active=inactive` and `NextElapseUSecRealtime` is EMPTY, and `list-timers` shows `NEXT: -` | it is stopped, NOT dormant -- `enabled` restarts it at boot and `Persistent=yes` then fires it immediately | 5a |
| a `last-sync`/status file sits at `RUNNING` for days | the oneshot was SIGTERM'd (`Result=signal`, `ExecMainStatus=15`) and never wrote a terminal state; cross-check `is-active` before believing it | 5a |
| stopping a `<x>-net.service` changes nothing, so "nothing depends on it" | a `oneshot` with `RemainAfterExit=yes` does not undo its work on stop -- delete the live addr/route to test dependency for real | 5b |

---

## NetBox (upstream, `netbox.baldurkeep.com`)

### `HTTP 403 Forbidden` on every API call -- but the SAME token works from `curl`

**Symptom.** `urllib`/`requests`/`pynetbox` get `403 Forbidden` from the upstream NetBox on every
endpoint, including ones a valid token plainly reads. `curl` with the identical token and URL
returns `200`. It reads exactly like a bad or revoked token; it is not.

**Cause.** Something in front of upstream NetBox (WAF / reverse proxy) filters on **User-Agent**.
The default Python UA (`Python-urllib/3.12`) is blocked; `curl/8.5.0` is allowed. Measured
2026-07-13: same token, same URL, UA the only variable -- `None -> 403`, `curl/8.5.0 -> 200`.

**Fix.** Send an accepted `User-Agent` header. This applies to **`pynetbox`** too (it sends
`python-requests/...`), so the repo's NetBox tooling must set it:

```python
nb = pynetbox.api(url, token=tok)
nb.http_session.headers["User-Agent"] = "curl/8.5.0"
```

**Do not** diagnose this as a token/permission problem and go re-mint credentials -- that is the
trap. Confirm with `curl` first; if curl works and Python does not, it is the UA.

**Note:** this is an UPSTREAM-only trap so far. The Office1 sandbox NetBox (4.6.4, no proxy in
front) accepts the default Python UA. Upstream is 4.5.8 and also uses **v1** (40-char, bare) tokens,
while the sandbox is 4.6 and uses **v2** (`nbt_<key>.<plaintext>`). Any tool pointed at BOTH must
handle both auth shapes and the UA -- do not assume they are the same NetBox.
