changelog 2026-07-13 -- Office1 headend LIVE: MAAS 3.7.2 + LXD 5.21, D-114 proven end to end

What

voffice1 is a working VR1 site headend. MAAS 3.7.2 (region+rack, PostgreSQL 16.14) and LXD 5.21.5 are installed on it, that LXD is registered back into MAAS as an LXD VM host, and MAAS has composed, PXE-booted and COMMISSIONED its first service VM. D-114's model is proven.

New: scripts/site-headend-install.sh + tests/site-headend-install/run-tests.sh (18/18 PASS). The script is the install sequence we actually executed, codified -- not written from docs. Its --check mode was run against the live voffice1 and reports every item OK.

The proof (measured, end to end)

vcloud     (L1 -- itself a KVM guest, measured)
  |- voffice1        (L2)  MAAS 3.7.2 region+rack + LXD 5.21.5, 10.10.0.20 (Kea reservation)
       |- office1-netbox  (L3)  LXD VIRTUAL-MACHINE, composed by MAAS, PXE-booted by MAAS

• office1-netbox: composed via maas admin vm-host compose, leased 10.10.1.100 from MAAS's own DHCP on lxdbr0, PXE-booted, ran commissioning, and reached status Ready with MAAS correctly detecting 2 cores / 4096 MiB. It then powered off -- correct for a commissioned machine.
• This is the DEFINITIVE L3 nested-KVM proof D-114 asked for. Previously we had only shown nested KVM was AVAILABLE (/dev/kvm + svm). Now a guest has actually booted, PXE'd, commissioned and reported its hardware three levels deep. D-114's Office1 gate is CLEARED.
• It reproduces VR0's lxd + tailscale pattern exactly: MAAS's unit of visibility is the machine; Juju's LXD containers (the OpenStack API services) remain invisible to MAAS, as in VR0.

The DHCP split -- structurally safe, not safe-by-convention

Two DHCP servers, two physically separate L2s -- lxdbr0 is never bridged onto the site LAN, so they cannot see each other. The script asserts this after configuring DHCP and FAILS if MAAS DHCP is on for any subnet other than the one it was told about.

10.10.1.0/24 is a NEW allocation that NetBox has not blessed. It was chosen deliberately (DC1 is 10.12.x, DC2 10.13.0.0/19, the tenant pool is 10.20.0.0/16 per D-016 -- the obvious-looking 10.20.x would have collided with tenant space). It must be registered in NetBox once NetBox exists. This is the known chicken-and-egg: NetBox is one of the VMs this headend composes.

FOUR TRAPS, all hit for real, all now encoded in the script and asserted by the harness

1. lxd init --auto FAILS on a host that also runs MAAS. It creates lxdbr0 with LXD's own DHCP+DNS on, which starts a dnsmasq; dnsmasq binds the wildcard 0.0.0.0:53, and MAAS's bind9 (named) already holds :53. Verbatim: Failed starting network: The DNS and DHCP service exited prematurely: exit status 2 ("dnsmasq: failed to create listening socket for 10.10.1.1: Address already in use") The message names the BRIDGE address, which sends you hunting the wrong thing -- the real conflict is the wildcard bind. ipv4.dhcp=false + dns.mode=none are NOT sufficient: LXD still spawns dnsmasq for a managed bridge and it still tries to bind :53. The fix that works is raw.dnsmasq="port=0". Verified after the fact: dnsmasq runs but holds zero sockets on :53 and zero on :67.
2. lxc reads its config from STDIN. Pipe this script to bash -s over ssh and the first lxc call without a redirect eats the rest of the script as YAML (Error: yaml: unmarshal errors: line 1: cannot unmarshal !!str 'echo "...' into api.StoragePoolPut) and everything after it silently vanishes. Every lxc call now ends in </dev/null. Same class as the OPNsense tcsh trap.
3. LXD snap track changes are ONE-WAY. LXD must be installed DIRECTLY onto 5.21/stable and never allowed to land on latest first, because MAAS 3.6/3.7 is incompatible with LXD >= 6.7 (pinned pylxd 2.3.5 vs LXD 6.7's consolidated endpoints). Bonus discovered in research: core.trust_password still exists in 5.21 and is GONE in 6.x, so the pin also preserves the scriptable registration path. https://discourse.maas.io/t/maas-incompatibility-with-lxd-6-7/15749
4. MAAS DHCP on the compose network ONLY. --compose-cidr is REQUIRED and has no default, precisely so it can never be picked by accident, and the script hard-fails if MAAS DHCP ends up enabled on any other subnet.

Also encoded: systemd-timesyncd must be disabled (MAAS manages time via chrony); a dynamic iprange must exist before dhcp_on=True leases anything; dir storage driver chosen over zfs deliberately (no zfs loop file inside an already-nested guest).

Discipline notes

• Every MAAS/LXD flag used here was read from the tool's OWN --help on the box before use (maas init --help, maas createadmin --help, maas vm-host compose --help), rather than trusted from documentation. --password on createadmin and storage=label:size on compose were confirmed this way.
• All secrets (DB password, MAAS admin password, MAAS API key, LXD trust password) are generated ON the target, stored 0600 under /root/maas-secrets/, and were never printed into the session.
• The harness asserts the four traps are still present in the script -- a "simplification" that drops one goes RED in the gauntlet rather than in production.

Verification

• tests/site-headend-install/run-tests.sh: 18/18 PASS. Includes a self-check that the harness actually catches a regression (removing the 5.21 pin makes it fail).
• scripts/site-headend-install.sh --check against the LIVE voffice1: every item OK.
• repo-lint: 0 fail. Full gauntlet: ALL GREEN.

Revert

ssh voffice1 'sudo snap remove maas postgresql lxd'   # or destroy voffice1 entirely:
tofu -chdir=opentofu destroy -target=module.voffice1
git revert <this commit>

Reverting the commit removes the script + harness. The OPNsense edge is untouched. Note the Kea reservation for voffice1 lives on the edge, not in this commit -- remove it via the REST API (kea/dhcpv4/del_reservation) if voffice1 is destroyed for good.