Stage 3 (vr1-dc0) prep batch: D-121..D-124 rulings + substrate/rack wiring, HA overlay, NetBox importers ...

Prepares the VR1 Stage-3 DC substrate so the live steps reduce to review-plan-and-apply.
Authoring + offline validation only -- NO cloud mutation; every tofu apply, NetBox --commit,
and rack install stays operator-gated. Four decisions ruled this batch, all recorded:

- D-121 (ADOPTED): make the decorative single-unit control plane REAL. 14 services scale 1->3
  (D-009's mechanical scale-up pulled into VR1); node layout = Option C (3 control + 2 compute +
  3 storage/DC, sizing set) after a whole-host resource validation (256 vCPU/1024 GiB/10 TiB);
  vault sub-ruling = v-a (MySQL-backed, charm-verify gated). Placement-balance rule for Stage 5.
- D-122 (ADOPTED): VR1 site shape -- nested-per-site containment, dark fiber East-West only,
  dedicated per-site L3 ISP uplink, 2-NIC fabric-routed DC edge (LAN=provider-public, WAN=uplink),
  per-site MAAS. Edge NIC count baremetal-matched (nodes 6, edge 2).
- D-123 (ADOPTED Model A): DC nodes vcloud-level; site-down = destroy the vr1-dc0-* group; MAAS
  region on Office1 + a rack controller per DC (vvr1-dc0).
- D-124 (ADOPTED Scheme A): the Office1-region<->DC-rack management overlay -- a transit link on
  the office1<->dc0 mesh leg; rack sizing 4/8192/80.

Code (all validated, NOT applied):
- OpenTofu: new modules/site-wan (NAT /24 uplink, provider-schema-confirmed); main.tf Stage-3
  section -- vr1_dc0_wan, the 2-NIC edge, 8 Option C node VMs (for_each), the vvr1-dc0 rack
  (two legs, static IPs from new vr1_dc0_rack_* tfvars -- NetBox-assigned, not invented), netem
  held, mesh-link network_name output; stale vr1_dc1_planes comment fixed. Step-9 maas-vm-host
  deliberately deferred (DOCFIX-179: its maas provider would force every plan to demand creds).
  tofu validate Success; opentofu-validate 11/11.
- NetBox: dc-edge-wan-import.py (58/58) + dc-rack-mgmt-import.py (96/96) -- register the DC-edge
  /24s + the rack transit/IP, dry-by-default, whole-plan preflight, operator-supplied literals.
- overlays/dc-ha-scaleup.yaml: the D-121 14-service HA scale-up (Stage-5), placement tokens +
  rabbitmq min-cluster-size + hacluster cluster_count + re-declared vault-hacluster.
- scripts/site-headend-install.sh: rack-only mode (enrolls a DC rack to Office1's region).
- runbooks/dc-dc-phase2-tofu-dc-substrate.md: fix-forward (D-119 naming, Steps 4-5 REST-API,
  D-121/122 folded in, stale gate corrected).

docs/session-ledger.md + dc-dc-deployment-workflow.md reconciled; three changelogs added.
Gauntlet ALL GREEN; repo-lint 0-fail (1 documented legacy WARN).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH

D-119: region-qualify the VR1 DC namespace -- C3 / gap #19 CLOSED (Stage 3 unblocked) ...

THE BUG: the token `dc0` meant TWO DIFFERENT CLOUDS depending on the file --
VR0's LIVE testcloud in scripts/lib-net.sh, VR1's FIRST DC in the NetBox
importer. One string, two clouds, one of them in production.

THE RULING (operator): the repo adopts the apex's names verbatim.
  vr0-dc0  VR0's DC0        the LIVE testcloud (a DIFFERENT region)
  vr1-dc0  VR1's FIRST  DC  GUA 2602:f3e2:f02::/48
  vr1-dc1  VR1's SECOND DC  GUA 2602:f3e2:f03::/48
Bare dc0/dc1/dc2 are now REJECTED LOUDLY everywhere.

D-119 COMPLETES D-117, it does not reverse it: it executes D-117's own amendment
across the three surfaces D-117 never touched. ZERO NetBox writes -- the apex was
already correct and self-consistent (MEASURED: f02::/48 -> vr1-dc0, f03::/48 ->
vr1-dc1); the REPO was the only surface out of step. Renaming the apex to match
the repo was considered and REJECTED (production IPAM write; VR1 would become the
only 1-indexed region; and vr1-dc1 would mean VR1's FIRST DC while vr0-dc1 means
VR0's SECOND).

THE PRIZE: the importer's DC->site map is now an IDENTITY -- there is no offset
table left to get wrong, and an assert enforces it. The original defect was
precisely a WRONG LOOKUP TABLE. The bug class is deleted, not defended.

TWO REAL BUGS the naming fix ALONE would NOT have closed (found by review sweep):

- BREAK-1: descriptions were built with f"VR1 {dc.upper()} ..." -- under D-119 that
  renders "VR1 VR1-DC0 provider-public" on all 36 prefixes. Same class as the
  original bug (deriving a label by munging a token), hidden in the description
  field where slug-focused review missed it. Now looked up from SITES[name].

- BREAK-2 (the important one): DC_GUA_PREFIX was NEVER cross-checked against --dc.
  `--dc vr1-dc0 DC_GUA_PREFIX=<f03>` was ACCEPTED: it writes the SECOND DC's GUA,
  carved with the FIRST DC's ULA nibble, scoped to the FIRST DC's site -- a
  silently mis-bound datacenter assembled from two disagreeing sources. Identity-
  mapping the slug leaves the ADDRESSING free-floating. Now guarded by EXPECTED_GUA.

ALSO: vdc1/vdc2 -> vvr1-dc0/vvr1-dc1 (D-114 amendment). D-114's own DR primitive is
`virsh destroy vdc1`, which read as "destroy VR1 DC1" while MEANING "destroy VR1
DC0". A mislabelled destroy command in a DR drill is not cosmetic. Neither VM is
built, so it is free. voffice1/Office1 KEEP their number (operator ruling).
Env var: DC1_/DC2_V4_SUPERNET -> VR1_DC1_V4_SUPERNET (both old names rejected).
rbd-mirror's --site-name aligned: `--dc vr1-dc0 --site-name dc1` was a re-created
two-namespace collision on one command line.

TESTS: dc-selector 21->30, prefixes-import 40->82 (identity invariant; EXPECTED_GUA
in BOTH mismatch directions; mismatched run writes NOTHING; matching pair still
succeeds; no munged description). GAUNTLET ALL GREEN (57). repo-lint 0 fail.
The old harnesses PINNED THE WRONG MAPPING -- they would have gone green while
enforcing the bug.

THE tofu apply IS NOT DONE -- IT IS GATED. Plan: 11 to add, 0 to change, 11 to
destroy (libvirt object names are ForceNew; moved{} makes the plan reviewable but
cannot suppress a replace). MEASURED SAFE: all 11 are EMPTY -- no guests, no
volumes, Stage 3 hasn't run. Office1's live objects are not in the plan.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

office1-netbox DEPLOYED: NetBox 4.6.4 live on the Office1 headend, three levels deep ...

MAAS composed the machine, PXE-booted it, commissioned it, and DEPLOYED Ubuntu 24.04.4 onto it --
an LXD VM inside voffice1, inside vcloud, which is itself a KVM guest. NetBox 4.6.4 (Django 6.0.6)
runs on it and its API returns HTTP 200. The D-114 chain is complete in its final form.

FOUR FINDINGS. TWO ARE NOW GUARDED IN CODE.

1. MAAS-DISCOVERED SUBNETS HAVE NO GATEWAY (FIXED). MAAS discovers the compose subnet from lxdbr0's
   interface, and that discovery carries no gateway_ip. The first deploy produced a machine that
   LOOKED healthy -- it had an address and DNS even resolved (MAAS's bind9 is link-local) -- but had
   NO default route and NO egress. It only surfaces when the first apt install hangs.
   scripts/site-headend-install.sh now sets gateway_ip on the compose subnet; the harness asserts it
   (19/19). office1-netbox was released and REDEPLOYED to prove the fix rather than hand-patching a
   route onto the running box.

2. NETBOX 4.6 v2 TOKENS: THE WIRE FORMAT IS `nbt_<key>.<plaintext>`, NOT THE API's `token` FIELD.
   NetBox 4.6 hashes tokens: `key` is a 12-char PREFIX, plus `plaintext` and an hmac_digest.
   Authentication infers the version FROM A PREFIX (TOKEN_PREFIX = 'nbt_') and splits on '.'. So the
   57-char assembled form is what authenticates. POST /api/users/tokens/provision/ returns `key` and
   `token` as SEPARATE fields, and the `token` field alone is NOT usable -- present it raw and NetBox
   parses it as a legacy v1 token and returns 403 "Invalid v1 token", an error that names v1 while
   you are holding a v2 token. Anything scripted against this NetBox (including
   netbox/dc-dc-prefixes-import.py and any pynetbox client) must ASSEMBLE the token.
   Also: netbox-docker's SUPERUSER_API_TOKEN env is NOT honored on this version, and hand-building a
   Token row trips the DB's enforce_version_dependent_fields constraint. Call the interface; do not
   re-implement it -- the same lesson this repo already learned minting OPNsense API keys, which I
   proceeded to re-learn the hard way.

3. A WRONG HYPOTHESIS, RECORDED BECAUSE THE MISTAKE IS REUSABLE. When the token kept 403-ing I
   concluded that rotating SECRET_KEY after NetBox had initialised must have broken the token HMAC
   pepper, and I WIPED THE DATABASE (docker compose down -v) to re-init. That was WRONG -- the 403
   was purely the wire format above. The wipe was harmless because the DB was empty, but the
   reasoning was not: on a populated NetBox it would have been destructive. "Invalid v1 token" is a
   FORMAT error, not a crypto or key-rotation error. Read the auth code before reaching for a
   destructive reset.

4. netbox-docker ships a PUBLIC default SECRET_KEY (it is in the public repo) -- session-forgery
   material on any reachable instance. Rotated to a generated value. (SKIP_SUPERUSER=true is the
   shipped default, so no default admin/admin exists.)

All secrets (SECRET_KEY, admin password, API token) generated ON the VM, stored 0600 under
/root/netbox-secrets/, never printed into the session.

REACHABILITY CAVEAT, not solved: NetBox is on 10.10.1.201, behind voffice1's NAT. Reachable from
voffice1, NOT from office1-local or a workstation. Making it reachable needs a static route for
10.10.1.0/24 via 10.10.0.20 on the OPNsense edge -- a live edge change, deliberately NOT done
unilaterally.

NEXT: import the D-115 carve into this NetBox.

repo-lint 0 fail. tests/site-headend-install 19/19 PASS.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Office1 headend LIVE: MAAS 3.7.2 + LXD 5.21 on voffice1. D-114 PROVEN END TO END. ...

MAAS composed, PXE-booted and COMMISSIONED its first VM inside voffice1. The model works.

  vcloud            (L1 -- itself a KVM guest, measured)
    +- voffice1     (L2)  MAAS 3.7.2 region+rack + PostgreSQL 16.14 + LXD 5.21.5
         +- office1-netbox (L3)  LXD VIRTUAL-MACHINE, MAAS-composed, MAAS-PXE-booted,
                                 leased 10.10.1.100 from MAAS, status READY, 2c/4096MiB detected

THE L3 PROOF IS NOW DEFINITIVE. Earlier today we could only say nested KVM was AVAILABLE
(/dev/kvm + svm present). Now a guest has actually BOOTED, PXE'd, COMMISSIONED and reported its
hardware three levels deep. D-114's Office1 gate is CLEARED. This reproduces VR0's `lxd` +
`tailscale` pattern exactly -- and the Juju-created LXD CONTAINERS remain invisible to MAAS, as in VR0.

THE DHCP SPLIT IS STRUCTURAL, NOT CONVENTIONAL:
  10.10.0.0/24 (site LAN)   -> Kea on the OPNsense edge   MAAS dhcp_on=False
  10.10.1.0/24 (lxdbr0)     -> MAAS                        MAAS dhcp_on=True
Separate L2s; lxdbr0 is never bridged onto the site LAN, so they cannot see each other. The
installer asserts after configuring DHCP that MAAS owns NO other subnet, and hard-fails otherwise.

NEW: scripts/site-headend-install.sh + tests/site-headend-install/ (18/18 PASS). It is the sequence
we ACTUALLY EXECUTED, codified -- not written from docs -- and reusable verbatim for DC1/DC2. Its
--check mode was run against the live voffice1 and reports every item OK.

FOUR TRAPS, ALL HIT FOR REAL TODAY, ALL NOW ENCODED AND HARNESS-ASSERTED:
1. `lxd init --auto` FAILS on a MAAS host. It brings up lxdbr0 with LXD's own DHCP+DNS, which starts
   a dnsmasq; dnsmasq binds the WILDCARD 0.0.0.0:53 and MAAS's bind9 already holds :53. The error
   names the BRIDGE address ("failed to create listening socket for 10.10.1.1"), which sends you
   hunting the wrong thing. `ipv4.dhcp=false` + `dns.mode=none` are NOT enough -- LXD still spawns
   dnsmasq. The fix is raw.dnsmasq="port=0". Verified after: dnsmasq holds ZERO :53 and ZERO :67.
2. `lxc` READS CONFIG FROM STDIN. Piped to `bash -s` over ssh, the first unredirected `lxc` call
   EATS THE REST OF THE SCRIPT as YAML and everything after it silently vanishes. Every lxc call now
   ends in </dev/null. Same class as the OPNsense tcsh trap.
3. LXD snap track changes are ONE-WAY -- install DIRECTLY onto 5.21/stable, never via `latest`.
   MAAS 3.6/3.7 is incompatible with LXD >= 6.7. (Bonus: core.trust_password exists in 5.21, gone in
   6.x -- the pin also preserves the scriptable registration path.)
4. MAAS DHCP on the compose network ONLY. --compose-cidr is REQUIRED with no default so it can never
   be picked by accident.

DISCIPLINE: every MAAS/LXD flag was read from the tool's own --help ON THE BOX before use (maas init,
createadmin, vm-host compose), not trusted from docs. All secrets (DB, admin, API key, LXD trust) are
generated on the target, stored 0600 under /root/maas-secrets/, and never printed into the session.

10.10.1.0/24 IS AN UNBLESSED ALLOCATION -- chosen deliberately (DC1=10.12.x, DC2=10.13.0.0/19, tenant
pool=10.20.0.0/16 per D-016; the obvious 10.20.x would have COLLIDED with tenant space), but NetBox
has not assigned it. Register once NetBox exists -- it is one of the VMs this headend composes.

Also: the ops skill is now tuned to this exact stack (new references/platform-traps.md with a
verbatim-error -> cause index; the OPNsense section that still routed sessions into the DELETED
config-ISO path is replaced). Upstream corroboration found for two of our own findings: the libvirt
provider's release notes document that an "in-place" domain update undefines and re-defines the
domain (our measured guest-bounce), and there is an upstream issue for the AppArmor denial.

Gauntlet ALL GREEN (53 harnesses). repo-lint 0 fail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>