office1-netbox DEPLOYED: NetBox 4.6.4 live on the Office1 headend, three levels deep ...

MAAS composed the machine, PXE-booted it, commissioned it, and DEPLOYED Ubuntu 24.04.4 onto it --
an LXD VM inside voffice1, inside vcloud, which is itself a KVM guest. NetBox 4.6.4 (Django 6.0.6)
runs on it and its API returns HTTP 200. The D-114 chain is complete in its final form.

FOUR FINDINGS. TWO ARE NOW GUARDED IN CODE.

1. MAAS-DISCOVERED SUBNETS HAVE NO GATEWAY (FIXED). MAAS discovers the compose subnet from lxdbr0's
   interface, and that discovery carries no gateway_ip. The first deploy produced a machine that
   LOOKED healthy -- it had an address and DNS even resolved (MAAS's bind9 is link-local) -- but had
   NO default route and NO egress. It only surfaces when the first apt install hangs.
   scripts/site-headend-install.sh now sets gateway_ip on the compose subnet; the harness asserts it
   (19/19). office1-netbox was released and REDEPLOYED to prove the fix rather than hand-patching a
   route onto the running box.

2. NETBOX 4.6 v2 TOKENS: THE WIRE FORMAT IS `nbt_<key>.<plaintext>`, NOT THE API's `token` FIELD.
   NetBox 4.6 hashes tokens: `key` is a 12-char PREFIX, plus `plaintext` and an hmac_digest.
   Authentication infers the version FROM A PREFIX (TOKEN_PREFIX = 'nbt_') and splits on '.'. So the
   57-char assembled form is what authenticates. POST /api/users/tokens/provision/ returns `key` and
   `token` as SEPARATE fields, and the `token` field alone is NOT usable -- present it raw and NetBox
   parses it as a legacy v1 token and returns 403 "Invalid v1 token", an error that names v1 while
   you are holding a v2 token. Anything scripted against this NetBox (including
   netbox/dc-dc-prefixes-import.py and any pynetbox client) must ASSEMBLE the token.
   Also: netbox-docker's SUPERUSER_API_TOKEN env is NOT honored on this version, and hand-building a
   Token row trips the DB's enforce_version_dependent_fields constraint. Call the interface; do not
   re-implement it -- the same lesson this repo already learned minting OPNsense API keys, which I
   proceeded to re-learn the hard way.

3. A WRONG HYPOTHESIS, RECORDED BECAUSE THE MISTAKE IS REUSABLE. When the token kept 403-ing I
   concluded that rotating SECRET_KEY after NetBox had initialised must have broken the token HMAC
   pepper, and I WIPED THE DATABASE (docker compose down -v) to re-init. That was WRONG -- the 403
   was purely the wire format above. The wipe was harmless because the DB was empty, but the
   reasoning was not: on a populated NetBox it would have been destructive. "Invalid v1 token" is a
   FORMAT error, not a crypto or key-rotation error. Read the auth code before reaching for a
   destructive reset.

4. netbox-docker ships a PUBLIC default SECRET_KEY (it is in the public repo) -- session-forgery
   material on any reachable instance. Rotated to a generated value. (SKIP_SUPERUSER=true is the
   shipped default, so no default admin/admin exists.)

All secrets (SECRET_KEY, admin password, API token) generated ON the VM, stored 0600 under
/root/netbox-secrets/, never printed into the session.

REACHABILITY CAVEAT, not solved: NetBox is on 10.10.1.201, behind voffice1's NAT. Reachable from
voffice1, NOT from office1-local or a workstation. Making it reachable needs a static route for
10.10.1.0/24 via 10.10.0.20 on the OPNsense edge -- a live edge change, deliberately NOT done
unilaterally.

NEXT: import the D-115 carve into this NetBox.

repo-lint 0 fail. tests/site-headend-install 19/19 PASS.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Office1 headend LIVE: MAAS 3.7.2 + LXD 5.21 on voffice1. D-114 PROVEN END TO END. ...

MAAS composed, PXE-booted and COMMISSIONED its first VM inside voffice1. The model works.

  vcloud            (L1 -- itself a KVM guest, measured)
    +- voffice1     (L2)  MAAS 3.7.2 region+rack + PostgreSQL 16.14 + LXD 5.21.5
         +- office1-netbox (L3)  LXD VIRTUAL-MACHINE, MAAS-composed, MAAS-PXE-booted,
                                 leased 10.10.1.100 from MAAS, status READY, 2c/4096MiB detected

THE L3 PROOF IS NOW DEFINITIVE. Earlier today we could only say nested KVM was AVAILABLE
(/dev/kvm + svm present). Now a guest has actually BOOTED, PXE'd, COMMISSIONED and reported its
hardware three levels deep. D-114's Office1 gate is CLEARED. This reproduces VR0's `lxd` +
`tailscale` pattern exactly -- and the Juju-created LXD CONTAINERS remain invisible to MAAS, as in VR0.

THE DHCP SPLIT IS STRUCTURAL, NOT CONVENTIONAL:
  10.10.0.0/24 (site LAN)   -> Kea on the OPNsense edge   MAAS dhcp_on=False
  10.10.1.0/24 (lxdbr0)     -> MAAS                        MAAS dhcp_on=True
Separate L2s; lxdbr0 is never bridged onto the site LAN, so they cannot see each other. The
installer asserts after configuring DHCP that MAAS owns NO other subnet, and hard-fails otherwise.

NEW: scripts/site-headend-install.sh + tests/site-headend-install/ (18/18 PASS). It is the sequence
we ACTUALLY EXECUTED, codified -- not written from docs -- and reusable verbatim for DC1/DC2. Its
--check mode was run against the live voffice1 and reports every item OK.

FOUR TRAPS, ALL HIT FOR REAL TODAY, ALL NOW ENCODED AND HARNESS-ASSERTED:
1. `lxd init --auto` FAILS on a MAAS host. It brings up lxdbr0 with LXD's own DHCP+DNS, which starts
   a dnsmasq; dnsmasq binds the WILDCARD 0.0.0.0:53 and MAAS's bind9 already holds :53. The error
   names the BRIDGE address ("failed to create listening socket for 10.10.1.1"), which sends you
   hunting the wrong thing. `ipv4.dhcp=false` + `dns.mode=none` are NOT enough -- LXD still spawns
   dnsmasq. The fix is raw.dnsmasq="port=0". Verified after: dnsmasq holds ZERO :53 and ZERO :67.
2. `lxc` READS CONFIG FROM STDIN. Piped to `bash -s` over ssh, the first unredirected `lxc` call
   EATS THE REST OF THE SCRIPT as YAML and everything after it silently vanishes. Every lxc call now
   ends in </dev/null. Same class as the OPNsense tcsh trap.
3. LXD snap track changes are ONE-WAY -- install DIRECTLY onto 5.21/stable, never via `latest`.
   MAAS 3.6/3.7 is incompatible with LXD >= 6.7. (Bonus: core.trust_password exists in 5.21, gone in
   6.x -- the pin also preserves the scriptable registration path.)
4. MAAS DHCP on the compose network ONLY. --compose-cidr is REQUIRED with no default so it can never
   be picked by accident.

DISCIPLINE: every MAAS/LXD flag was read from the tool's own --help ON THE BOX before use (maas init,
createadmin, vm-host compose), not trusted from docs. All secrets (DB, admin, API key, LXD trust) are
generated on the target, stored 0600 under /root/maas-secrets/, and never printed into the session.

10.10.1.0/24 IS AN UNBLESSED ALLOCATION -- chosen deliberately (DC1=10.12.x, DC2=10.13.0.0/19, tenant
pool=10.20.0.0/16 per D-016; the obvious 10.20.x would have COLLIDED with tenant space), but NetBox
has not assigned it. Register once NetBox exists -- it is one of the VMs this headend composes.

Also: the ops skill is now tuned to this exact stack (new references/platform-traps.md with a
verbatim-error -> cause index; the OPNsense section that still routed sessions into the DELETED
config-ISO path is replaced). Upstream corroboration found for two of our own findings: the libvirt
provider's release notes document that an "in-place" domain update undefines and re-defines the
domain (our measured guest-bounce), and there is an upstream issue for the AppArmor denial.

Gauntlet ALL GREEN (53 harnesses). repo-lint 0 fail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>