D-125: Model B per-DC ISP egress (bridge-in, single-NAT) -- resolves OBS-3 design gap ...

OBS-3: Model B nesting put vr1-dc0-wan (a libvirt NAT) inside vvr1-dc0, whose
only leg is the SEC-010 FORWARD-dropped transit -- so the per-DC simulated ISP
had no egress. Operator ruled bridge-in (single-NAT) over double-NAT.

- modules/wan-bridge (NEW): libvirt forward=bridge onto a pre-existing host
  bridge (bridge={name}, schema measured from the provider, not guessed). No
  subnet/NAT. Sibling to site-wan so every NAT user stays untouched. Covered by
  opentofu-validate S3 (standalone PASS).
- OUTER root: vcloud ISP NAT vr1_dc0_uplink (the single NAT) + a 2nd IP-less
  uplink NIC on vvr1-dc0 + the br-vr1-dc0-wan bridge declared in netplan (transit
  stays NIC1 so SEC-010/--transit-if keep keying on it). New HELD var
  vr1_dc0_uplink_cidr (no default; office1-netbox apex, hard rule 2).
- INNER root: vr1_dc0_wan site-wan(NAT) -> wan-bridge onto br-vr1-dc0-wan;
  OPNsense WAN re-addresses into the vcloud ISP /24 (edge config, D-113, HELD).
- Bootstrap (site-headend-install.sh): --uplink-if/--wan-bridge; --check verifies
  the bridge exists + uplink ENSLAVED (fail-open guard, same class as the transit
  check); SEC-010 rationale corrected (bridge-in has no inner NAT) + br_netfilter
  CONSTRAINT (keep the FORWARD-drop interface-scoped, NEVER global). Harness 49->54.

Records: D-125 (new); D-122 amendment (forward-pointer -- WAN realization
superseded, intent stands); SEC-010 (br_netfilter + hardened --check);
model-a-fallback B->A delta item 8 + revert; platform-traps 1g (bridge +
br_netfilter + nested-foreign-MAC trap).

NOT closed: the L2-NAT-of-foreign-MAC path is a DEPLOY-TIME gate (bridge-in
equivalent of the depth-4 boot gate); fallback = double-NAT. HELD: vr1_dc0_uplink_cidr
+ OPNsense WAN re-address. Both roots validate; gauntlet ALL GREEN (63); repo-lint
0 fail. Present-only; NOT pushed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH

Stage-3 review sweep C+D: Model B two-root reshape + SEC-010 hardening ...

Phase C (Model B reshape, D-123 operator ruling):
- outer root (opentofu/main.tf): vvr1-dc0 resized to whole-DC containment VM
  (108 vCPU / 425984 MiB / 3221225472000 bytes; expose_nested_virt=true;
  single Office1 transit leg); node/plane/wan/opnsense modules removed from
  vcloud level (MOVED markers), storage-04 added per R-3.
- inner root (opentofu/vr1-dc0-substrate/): NEW second root, provider
  qemu+ssh to vvr1-dc0; reuses ../modules/* verbatim to build 9 nodes +
  6 planes + wan + opnsense inside the containment VM. dmacvicar/libvirt
  0.9.8 pinned (.terraform.lock.hcl committed). Both roots tofu-validate.
- variables.tf: vvr1_dc0_vcpu/memory_mib/disk_bytes with derivation comment.

Phase D (SEC-010, R3-F02): site-headend-install.sh --host-nodes node-host
bootstrap installs qemu-kvm+libvirt, persists+verifies nested=1, writes a
transit FORWARD-drop (nftables inet sec010) enforcing DC-LOCAL under Model B.
Gate hardened (advisor catch): --check now verifies the keyed transit
interface EXISTS, else nftables oifname loads clean but matches nothing
(fail-open). Harness 49/49.

Records: D-103/107/114/121/122/123/124 amendments (append-only);
model-a-fallback-plan synced; ledgers + Phase-C changelog.

KNOWN GAP (OBS-3, operator-confirmed 2026-07-16): under Model B the inner
vr1-dc0-wan NAT is nested inside vvr1-dc0 (single transit NIC), severing the
simulated-ISP egress that Model A had at vcloud level. DC0/DC1 ARE meant to
have ISP connections -- egress path resolution is follow-up design work, not
resolved by this commit.

Present-only; not pushed. Gauntlet ALL GREEN (63); repo-lint 0 fail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH

Stage 3 (vr1-dc0) prep batch: D-121..D-124 rulings + substrate/rack wiring, HA overlay, NetBox importers ...

Prepares the VR1 Stage-3 DC substrate so the live steps reduce to review-plan-and-apply.
Authoring + offline validation only -- NO cloud mutation; every tofu apply, NetBox --commit,
and rack install stays operator-gated. Four decisions ruled this batch, all recorded:

- D-121 (ADOPTED): make the decorative single-unit control plane REAL. 14 services scale 1->3
  (D-009's mechanical scale-up pulled into VR1); node layout = Option C (3 control + 2 compute +
  3 storage/DC, sizing set) after a whole-host resource validation (256 vCPU/1024 GiB/10 TiB);
  vault sub-ruling = v-a (MySQL-backed, charm-verify gated). Placement-balance rule for Stage 5.
- D-122 (ADOPTED): VR1 site shape -- nested-per-site containment, dark fiber East-West only,
  dedicated per-site L3 ISP uplink, 2-NIC fabric-routed DC edge (LAN=provider-public, WAN=uplink),
  per-site MAAS. Edge NIC count baremetal-matched (nodes 6, edge 2).
- D-123 (ADOPTED Model A): DC nodes vcloud-level; site-down = destroy the vr1-dc0-* group; MAAS
  region on Office1 + a rack controller per DC (vvr1-dc0).
- D-124 (ADOPTED Scheme A): the Office1-region<->DC-rack management overlay -- a transit link on
  the office1<->dc0 mesh leg; rack sizing 4/8192/80.

Code (all validated, NOT applied):
- OpenTofu: new modules/site-wan (NAT /24 uplink, provider-schema-confirmed); main.tf Stage-3
  section -- vr1_dc0_wan, the 2-NIC edge, 8 Option C node VMs (for_each), the vvr1-dc0 rack
  (two legs, static IPs from new vr1_dc0_rack_* tfvars -- NetBox-assigned, not invented), netem
  held, mesh-link network_name output; stale vr1_dc1_planes comment fixed. Step-9 maas-vm-host
  deliberately deferred (DOCFIX-179: its maas provider would force every plan to demand creds).
  tofu validate Success; opentofu-validate 11/11.
- NetBox: dc-edge-wan-import.py (58/58) + dc-rack-mgmt-import.py (96/96) -- register the DC-edge
  /24s + the rack transit/IP, dry-by-default, whole-plan preflight, operator-supplied literals.
- overlays/dc-ha-scaleup.yaml: the D-121 14-service HA scale-up (Stage-5), placement tokens +
  rabbitmq min-cluster-size + hacluster cluster_count + re-declared vault-hacluster.
- scripts/site-headend-install.sh: rack-only mode (enrolls a DC rack to Office1's region).
- runbooks/dc-dc-phase2-tofu-dc-substrate.md: fix-forward (D-119 naming, Steps 4-5 REST-API,
  D-121/122 folded in, stale gate corrected).

docs/session-ledger.md + dc-dc-deployment-workflow.md reconciled; three changelogs added.
Gauntlet ALL GREEN; repo-lint 0-fail (1 documented legacy WARN).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH

office1-netbox DEPLOYED: NetBox 4.6.4 live on the Office1 headend, three levels deep ...

MAAS composed the machine, PXE-booted it, commissioned it, and DEPLOYED Ubuntu 24.04.4 onto it --
an LXD VM inside voffice1, inside vcloud, which is itself a KVM guest. NetBox 4.6.4 (Django 6.0.6)
runs on it and its API returns HTTP 200. The D-114 chain is complete in its final form.

FOUR FINDINGS. TWO ARE NOW GUARDED IN CODE.

1. MAAS-DISCOVERED SUBNETS HAVE NO GATEWAY (FIXED). MAAS discovers the compose subnet from lxdbr0's
   interface, and that discovery carries no gateway_ip. The first deploy produced a machine that
   LOOKED healthy -- it had an address and DNS even resolved (MAAS's bind9 is link-local) -- but had
   NO default route and NO egress. It only surfaces when the first apt install hangs.
   scripts/site-headend-install.sh now sets gateway_ip on the compose subnet; the harness asserts it
   (19/19). office1-netbox was released and REDEPLOYED to prove the fix rather than hand-patching a
   route onto the running box.

2. NETBOX 4.6 v2 TOKENS: THE WIRE FORMAT IS `nbt_<key>.<plaintext>`, NOT THE API's `token` FIELD.
   NetBox 4.6 hashes tokens: `key` is a 12-char PREFIX, plus `plaintext` and an hmac_digest.
   Authentication infers the version FROM A PREFIX (TOKEN_PREFIX = 'nbt_') and splits on '.'. So the
   57-char assembled form is what authenticates. POST /api/users/tokens/provision/ returns `key` and
   `token` as SEPARATE fields, and the `token` field alone is NOT usable -- present it raw and NetBox
   parses it as a legacy v1 token and returns 403 "Invalid v1 token", an error that names v1 while
   you are holding a v2 token. Anything scripted against this NetBox (including
   netbox/dc-dc-prefixes-import.py and any pynetbox client) must ASSEMBLE the token.
   Also: netbox-docker's SUPERUSER_API_TOKEN env is NOT honored on this version, and hand-building a
   Token row trips the DB's enforce_version_dependent_fields constraint. Call the interface; do not
   re-implement it -- the same lesson this repo already learned minting OPNsense API keys, which I
   proceeded to re-learn the hard way.

3. A WRONG HYPOTHESIS, RECORDED BECAUSE THE MISTAKE IS REUSABLE. When the token kept 403-ing I
   concluded that rotating SECRET_KEY after NetBox had initialised must have broken the token HMAC
   pepper, and I WIPED THE DATABASE (docker compose down -v) to re-init. That was WRONG -- the 403
   was purely the wire format above. The wipe was harmless because the DB was empty, but the
   reasoning was not: on a populated NetBox it would have been destructive. "Invalid v1 token" is a
   FORMAT error, not a crypto or key-rotation error. Read the auth code before reaching for a
   destructive reset.

4. netbox-docker ships a PUBLIC default SECRET_KEY (it is in the public repo) -- session-forgery
   material on any reachable instance. Rotated to a generated value. (SKIP_SUPERUSER=true is the
   shipped default, so no default admin/admin exists.)

All secrets (SECRET_KEY, admin password, API token) generated ON the VM, stored 0600 under
/root/netbox-secrets/, never printed into the session.

REACHABILITY CAVEAT, not solved: NetBox is on 10.10.1.201, behind voffice1's NAT. Reachable from
voffice1, NOT from office1-local or a workstation. Making it reachable needs a static route for
10.10.1.0/24 via 10.10.0.20 on the OPNsense edge -- a live edge change, deliberately NOT done
unilaterally.

NEXT: import the D-115 carve into this NetBox.

repo-lint 0 fail. tests/site-headend-install 19/19 PASS.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Office1 headend LIVE: MAAS 3.7.2 + LXD 5.21 on voffice1. D-114 PROVEN END TO END. ...

MAAS composed, PXE-booted and COMMISSIONED its first VM inside voffice1. The model works.

  vcloud            (L1 -- itself a KVM guest, measured)
    +- voffice1     (L2)  MAAS 3.7.2 region+rack + PostgreSQL 16.14 + LXD 5.21.5
         +- office1-netbox (L3)  LXD VIRTUAL-MACHINE, MAAS-composed, MAAS-PXE-booted,
                                 leased 10.10.1.100 from MAAS, status READY, 2c/4096MiB detected

THE L3 PROOF IS NOW DEFINITIVE. Earlier today we could only say nested KVM was AVAILABLE
(/dev/kvm + svm present). Now a guest has actually BOOTED, PXE'd, COMMISSIONED and reported its
hardware three levels deep. D-114's Office1 gate is CLEARED. This reproduces VR0's `lxd` +
`tailscale` pattern exactly -- and the Juju-created LXD CONTAINERS remain invisible to MAAS, as in VR0.

THE DHCP SPLIT IS STRUCTURAL, NOT CONVENTIONAL:
  10.10.0.0/24 (site LAN)   -> Kea on the OPNsense edge   MAAS dhcp_on=False
  10.10.1.0/24 (lxdbr0)     -> MAAS                        MAAS dhcp_on=True
Separate L2s; lxdbr0 is never bridged onto the site LAN, so they cannot see each other. The
installer asserts after configuring DHCP that MAAS owns NO other subnet, and hard-fails otherwise.

NEW: scripts/site-headend-install.sh + tests/site-headend-install/ (18/18 PASS). It is the sequence
we ACTUALLY EXECUTED, codified -- not written from docs -- and reusable verbatim for DC1/DC2. Its
--check mode was run against the live voffice1 and reports every item OK.

FOUR TRAPS, ALL HIT FOR REAL TODAY, ALL NOW ENCODED AND HARNESS-ASSERTED:
1. `lxd init --auto` FAILS on a MAAS host. It brings up lxdbr0 with LXD's own DHCP+DNS, which starts
   a dnsmasq; dnsmasq binds the WILDCARD 0.0.0.0:53 and MAAS's bind9 already holds :53. The error
   names the BRIDGE address ("failed to create listening socket for 10.10.1.1"), which sends you
   hunting the wrong thing. `ipv4.dhcp=false` + `dns.mode=none` are NOT enough -- LXD still spawns
   dnsmasq. The fix is raw.dnsmasq="port=0". Verified after: dnsmasq holds ZERO :53 and ZERO :67.
2. `lxc` READS CONFIG FROM STDIN. Piped to `bash -s` over ssh, the first unredirected `lxc` call
   EATS THE REST OF THE SCRIPT as YAML and everything after it silently vanishes. Every lxc call now
   ends in </dev/null. Same class as the OPNsense tcsh trap.
3. LXD snap track changes are ONE-WAY -- install DIRECTLY onto 5.21/stable, never via `latest`.
   MAAS 3.6/3.7 is incompatible with LXD >= 6.7. (Bonus: core.trust_password exists in 5.21, gone in
   6.x -- the pin also preserves the scriptable registration path.)
4. MAAS DHCP on the compose network ONLY. --compose-cidr is REQUIRED with no default so it can never
   be picked by accident.

DISCIPLINE: every MAAS/LXD flag was read from the tool's own --help ON THE BOX before use (maas init,
createadmin, vm-host compose), not trusted from docs. All secrets (DB, admin, API key, LXD trust) are
generated on the target, stored 0600 under /root/maas-secrets/, and never printed into the session.

10.10.1.0/24 IS AN UNBLESSED ALLOCATION -- chosen deliberately (DC1=10.12.x, DC2=10.13.0.0/19, tenant
pool=10.20.0.0/16 per D-016; the obvious 10.20.x would have COLLIDED with tenant space), but NetBox
has not assigned it. Register once NetBox exists -- it is one of the VMs this headend composes.

Also: the ops skill is now tuned to this exact stack (new references/platform-traps.md with a
verbatim-error -> cause index; the OPNsense section that still routed sessions into the DELETED
config-ISO path is replaced). Upstream corroboration found for two of our own findings: the libvirt
provider's release notes document that an "in-place" domain update undefines and re-defines the
domain (our measured guest-bounce), and there is an upstream issue for the AppArmor denial.

Gauntlet ALL GREEN (53 harnesses). repo-lint 0 fail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>