# changelog 2026-07-13 -- Office1 headend LIVE: MAAS 3.7.2 + LXD 5.21, D-114 proven end to end ## What `voffice1` is a working VR1 site headend. **MAAS 3.7.2** (region+rack, PostgreSQL 16.14) and **LXD 5.21.5** are installed on it, that LXD is **registered back into MAAS as an LXD VM host**, and MAAS has **composed, PXE-booted and COMMISSIONED** its first service VM. D-114's model is proven. New: `scripts/site-headend-install.sh` + `tests/site-headend-install/run-tests.sh` (18/18 PASS). The script is the install sequence we actually executed, codified -- not written from docs. Its `--check` mode was run against the live `voffice1` and reports every item OK. ## The proof (measured, end to end) ``` vcloud (L1 -- itself a KVM guest, measured) |- voffice1 (L2) MAAS 3.7.2 region+rack + LXD 5.21.5, 10.10.0.20 (Kea reservation) |- office1-netbox (L3) LXD VIRTUAL-MACHINE, composed by MAAS, PXE-booted by MAAS ``` - `office1-netbox`: composed via `maas admin vm-host compose`, leased **10.10.1.100** from MAAS's own DHCP on `lxdbr0`, PXE-booted, ran commissioning, and reached **status `Ready`** with MAAS correctly detecting 2 cores / 4096 MiB. It then powered off -- correct for a commissioned machine. - **This is the DEFINITIVE L3 nested-KVM proof** D-114 asked for. Previously we had only shown nested KVM was AVAILABLE (`/dev/kvm` + `svm`). Now a guest has actually booted, PXE'd, commissioned and reported its hardware three levels deep. D-114's Office1 gate is CLEARED. - It reproduces VR0's `lxd` + `tailscale` pattern exactly: MAAS's unit of visibility is the machine; Juju's LXD containers (the OpenStack API services) remain invisible to MAAS, as in VR0. ## The DHCP split -- structurally safe, not safe-by-convention | network | authority | MAAS `dhcp_on` | |---|---|---| | `10.10.0.0/24` (office1-local, the site LAN) | **Kea**, on the OPNsense edge | **False** | | `10.10.1.0/24` (`lxdbr0`, the compose network) | **MAAS** | **True** | Two DHCP servers, two **physically separate L2s** -- `lxdbr0` is never bridged onto the site LAN, so they cannot see each other. The script asserts this after configuring DHCP and FAILS if MAAS DHCP is on for any subnet other than the one it was told about. **`10.10.1.0/24` is a NEW allocation that NetBox has not blessed.** It was chosen deliberately (DC1 is `10.12.x`, DC2 `10.13.0.0/19`, the tenant pool is `10.20.0.0/16` per D-016 -- the obvious-looking `10.20.x` would have collided with tenant space). It must be registered in NetBox once NetBox exists. This is the known chicken-and-egg: NetBox is one of the VMs this headend composes. ## FOUR TRAPS, all hit for real, all now encoded in the script and asserted by the harness 1. **`lxd init --auto` FAILS on a host that also runs MAAS.** It creates `lxdbr0` with LXD's own DHCP+DNS on, which starts a dnsmasq; dnsmasq binds the **wildcard** `0.0.0.0:53`, and MAAS's bind9 (`named`) already holds `:53`. Verbatim: `Failed starting network: The DNS and DHCP service exited prematurely: exit status 2 ("dnsmasq: failed to create listening socket for 10.10.1.1: Address already in use")` The message names the BRIDGE address, which sends you hunting the wrong thing -- the real conflict is the wildcard bind. **`ipv4.dhcp=false` + `dns.mode=none` are NOT sufficient**: LXD still spawns dnsmasq for a managed bridge and it still tries to bind `:53`. The fix that works is `raw.dnsmasq="port=0"`. Verified after the fact: dnsmasq runs but holds **zero** sockets on `:53` and **zero** on `:67`. 2. **`lxc` reads its config from STDIN.** Pipe this script to `bash -s` over ssh and the first `lxc` call without a redirect **eats the rest of the script as YAML** (`Error: yaml: unmarshal errors: line 1: cannot unmarshal !!str 'echo "...' into api.StoragePoolPut`) and everything after it silently vanishes. Every `lxc` call now ends in `= 6.7 (pinned pylxd 2.3.5 vs LXD 6.7's consolidated endpoints). Bonus discovered in research: `core.trust_password` still exists in 5.21 and is GONE in 6.x, so the pin also preserves the scriptable registration path. https://discourse.maas.io/t/maas-incompatibility-with-lxd-6-7/15749 4. **MAAS DHCP on the compose network ONLY.** `--compose-cidr` is REQUIRED and has no default, precisely so it can never be picked by accident, and the script hard-fails if MAAS DHCP ends up enabled on any other subnet. Also encoded: `systemd-timesyncd` must be disabled (MAAS manages time via chrony); a **dynamic iprange must exist before `dhcp_on=True` leases anything**; `dir` storage driver chosen over zfs deliberately (no zfs loop file inside an already-nested guest). ## Discipline notes - Every MAAS/LXD flag used here was read from the tool's OWN `--help` **on the box** before use (`maas init --help`, `maas createadmin --help`, `maas vm-host compose --help`), rather than trusted from documentation. `--password` on `createadmin` and `storage=label:size` on `compose` were confirmed this way. - All secrets (DB password, MAAS admin password, MAAS API key, LXD trust password) are generated ON the target, stored `0600` under `/root/maas-secrets/`, and were never printed into the session. - The harness asserts the four traps are still present in the script -- a "simplification" that drops one goes RED in the gauntlet rather than in production. ## Verification - `tests/site-headend-install/run-tests.sh`: **18/18 PASS**. Includes a self-check that the harness actually catches a regression (removing the 5.21 pin makes it fail). - `scripts/site-headend-install.sh --check` against the LIVE `voffice1`: every item OK. - `repo-lint`: 0 fail. Full gauntlet: ALL GREEN. ## Revert ``` ssh voffice1 'sudo snap remove maas postgresql lxd' # or destroy voffice1 entirely: tofu -chdir=opentofu destroy -target=module.voffice1 git revert ``` Reverting the commit removes the script + harness. The OPNsense edge is untouched. Note the Kea reservation for voffice1 lives on the edge, not in this commit -- remove it via the REST API (`kea/dhcpv4/del_reservation`) if voffice1 is destroyed for good.