# DC-DC (VR1) NetBox Buildout -- Planning & Decision Scope **Status:** planning/scope reference for the Code stream. NOT a decision record -- it consolidates ADOPTED decisions and flags the sub-decisions still needing operator ratification (candidate D-111+; assign via `ledger-scan` at adoption). **Authority order:** repo `design-decisions.md` (ADOPTED) > this doc > live NetBox (a reconciliation target, not a source) > design-exploration chats. NetBox is the IPAM apex (D-010, D-103) -- everything below it consumes it; it is never read back as a planning input. **Scope:** what must exist in `netbox.baldurkeep.com` before Phase 1 completes so the VR1 two-DC buildout can consume it. Reconciled against a live read of NetBox on 2026-07-10 (v4.5.8). --- ## 1. Authoritative address/family model (D-101, D-100, D-108) Six planes per DC, family follows reachability. Verbatim from D-101 / `dc-dc-buildout-design.md`: | Plane | v4 (DC1 = DC0 template) | v6 family | Notes | |---|---|---|---| | provider-public | `10.12.4.0/22` | **GUA** (dual) | public API VIPs + FIP/ext_net; GUA via owned OPNsense edge (D-015 gate N/A in VR1) | | metal-admin | `10.12.8.0/22` | **ULA** (dual) | admin-API ULA leg (2026-07-09 amendment); PXE stays v4-first | | metal-internal | `10.12.12.0/22` | **ULA** (dual) | API endpoints dual-stack; MySQL/RabbitMQ stay v4 | | data-tenant | `10.12.16.0/22` | **ULA-only** | geneve overlay; tenant GUA delegated separately | | storage | `10.12.32.0/22` | **ULA-only** | Ceph public; `ms_bind_ipv6` | | replication | `10.12.36.0/22` | **ULA-only** | Ceph cluster; the cross-DC leg rides this ULA plane (D-108) | Also ULA-only: **Octavia lb-mgmt** and **P2P links**. IPv4-retained: **MAAS PXE** (v4-first) and **datastore east-west** (MySQL/RabbitMQ). Tenant GUA: **native routing, no NAT66**, SLAAC default, **`/60` per tenant** (~4080/pool; `/56` on request) -- delegated from a per-DC tenant GUA pool, distinct from the infra planes. **Per-DC v4 (D-101):** DC1 **inherits DC0's six-plane CIDRs unchanged** (the six above). DC2 gets a **distinct, non-overlapping** parallel supernet, same six-plane structure, non-colliding with DC1 + operator infra + client VPN (D-074). **Cross-DC replication carrier = IPv6-only ULA** `/64` (D-101/D-108; supersedes the DR-seed `10.12.36.0/22` v4 line). **Three literals are NetBox-assigned, NOT hardcoded in any decision (D-101 open item / tooling-gap #3):** the org ULA `/48`, the per-DC GUA carve out of the `/36`, and DC2's v4 supernet. The import tool takes them as required env vars with no defaults. --- ## 2. Live NetBox state (2026-07-10) -- reconciliation `netbox.baldurkeep.com` v4.5.8. Regions: US West -> Eugene (prod); **Virtual Regions -> VR0, VR1**. Sites incl. **VR1 DC0 (`vr1-dc0`), VR1 DC1 (`vr1-dc1`)**. 90 prefixes (77 v6 / 13 v4), 56 VLANs, 2 VRFs (Roosevelt, Willamette), 1 RIR (ARIN). What is already right, and what is missing/divergent -- **Code must close these before the import runs**: | # | Finding | State | Action | |---|---|---|---| | G1 | **Six-plane IPAM roles** | Only `storage` exists; `provider-public`, `metal-admin`, `metal-internal`, `data-tenant`, `replication` **MISSING** | Create the 5 roles out-of-band (import script does NOT create roles) | | G2 | **Aggregates** | **0 aggregates** despite ARIN RIR present | Create aggregates (section 4b) -- IPAM apex should declare its allocations | | G3 | **Org ULA `/48`** | Not present (only `fc00::/8`, `fd00::/8` containers) | Assign + create (section 4c); the D-101 literal | | G4 | **VR1 VLAN groups** | VR1 DC0/DC1 have **no VLAN groups** (VR0 DC0 + prod sites do) | Create per-DC VLAN groups + VLANs | | G5 | **DC naming** | ~~NetBox = `vr1-dc0`/`vr1-dc1`; tool = DC1/DC2~~ | **CLOSED by D-117 (ADOPTED 2026-07-13).** The REPO moved to the apex's 0-indexed names: `--dc dc0` -> `vr1-dc0`, `--dc dc1` -> `vr1-dc1`. The recommendation BELOW (rename the apex to `vr1-dc1`/`vr1-dc2`) was **REJECTED** -- the apex is not bent to fit the repo. The office KEEPS its number (`vr1-off1`). | | G6 | **VR0 DC0 / Willamette prefixes** | Fully built on the **OLD** model: single roles (`provider`/`metal`/`data`/`storage`/`repl`/`lbaas-mgmt`), **all GUA**, no ULA | This is the v1/DC0-era model, **not** the VR1 template. Leave as-is or retire (hygiene call, D-101 site-collision note) | --- ## 3. Structure to build on (RECONCILED template) The deployed carve and D-101 agree on a byte-aligned layout; use it for VR1. **Template `2602:f3e2:0RSS:CCNN`** where nibble-1 = `0` (region-0 = the held `/36`; varies once expanded to the `/32`), nibble-2 = the region/metro selector (Eugene=`1`, VR0=`e`, **VR1=`f`**), `SS` = site byte (`/48`), `CC` = cloud byte (`/56`), `NN` = net byte (`/64`). Net-byte role mnemonic already in use: `10` provider, `11` provider-VIP, `20` metal, `30` data, `40` storage, `50` replication, `80` lbaas, `e0` VPN, `f0` OOB. VR1 already occupies `2602:f3e2:f00::/40` with DC0=`f02::/48`, DC1=`f03::/48` as sites. **The per-DC GUA carve (D-101 literal) should be these existing `/48` site blocks** (`DC_GUA_PREFIX` = `2602:f3e2:f02::/48` for DC1, `2602:f3e2:f03::/48` for DC2), aligned to G5's naming decision -- not the README's placeholder `2602:f3e2:1000::/40`. > **RECONCILIATION NOTE (design-chat vs repo -- for the operator, so nothing surprises you):** > The parallel IPAM-design chat produced a carve (v3 / dual-region bundle) that diverged from what > is deployed and adopted. Those artifacts are **superseded for the DC-DC buildout**; the deployed > pattern + D-101 govern. Specifically: > - **Region label:** chat used `RMSS` (region=`/36`, a "metro" at `/40`); deployed + D-101 use > region-0=`/36` with the `/40` as the region/metro selector (`0RSS`). Addresses match for > production; **only VR0/VR1 placement differed** -- the chat bundle put them at nibble-1 > (`2602:f3e2:e000::/36`, `f000::/36`); NetBox has them at nibble-2 (`0e00`/`0f00::/40`). > **Deployed nibble-2 placement wins.** > - **Tenant size:** chat concluded `/64`-per-tenant (assuming tenants share the infra cloud > `/56`); D-101 says **`/60`**, because tenant GUA is a **separate delegation pool**, not the > infra `/56`. **D-101 `/60` wins.** > - **Roles:** chat used the old 5-6 role set; VR1 uses the **six-plane** set. **Six-plane wins.** --- ## 4. Buildout steps for Code (ordered; all idempotent, dry-run first) **4a. IPAM roles (G1).** Create the 5 missing six-plane roles (`provider-public`, `metal-admin`, `metal-internal`, `data-tenant`, `replication`; `storage` exists). Slugs must match `scripts/lib-net.sh` `SPACES6` exactly -- the import tool references them by slug. Created by `netbox/roles-aggregates-import.py` (this delivery). **4b. Aggregates (G2).** Under RIR ARIN: **`2602:f3e2::/36`** (the *held* allocation = region-0 of the planned `/32`; update to `/32` post-expand -- declaring a `/32` you don't hold is not honest for an IPAM apex), `23.157.124.0/24`. Under a new RFC-4193 RIR: the org ULA `/48` (4c). Under a new RFC-1918 RIR: `10.0.0.0/8`, `172.16.0.0/12`. All created idempotently by `netbox/roles-aggregates-import.py` (this delivery). **4c. Org ULA `/48` (G3, D-101 literal).** Assign one RFC-4193 pseudo-random `/48`. **Recommended value: `fd50:840e:74e2::/48`** -- CSPRNG-generated (Global ID `0x50840e74e2`), collision-checked against the Tailscale `fd7a:115c:a1e0::/48` already in NetBox (distinct). This is a *recommendation for the operator to ratify as the literal*, not an invented fact; if adopted, it becomes `ORG_ULA_48` for the import tool. **4d. Per-DC prefixes via `netbox/dc-dc-prefixes-import.py` (D-101).** Run per DC with the three literals. Recommended literals (pending G5 naming + section 5 ratification): ``` # DC1 (v4 inherited; only v6 top-level blocks required) ORG_ULA_48=fd50:840e:74e2::/48 DC_GUA_PREFIX=2602:f3e2:f02::/48 --dc dc0 # D-117: VR1 DC0 # DC2 (distinct v4 supernet ALSO required -- >= /19; tool re-packs 6 sequential /22s, see 5.3) ORG_ULA_48=fd50:840e:74e2::/48 DC_GUA_PREFIX=2602:f3e2:f03::/48 DC1_V4_SUPERNET=10.12.64.0/19 --dc dc1 # D-117: VR1 DC1 ``` Run `--verify-only` first. The tool creates the six-plane v4 + the D-101 family-matrix v6 legs. **4e. VLAN groups + VLANs (G4).** Per-DC VLAN groups scoped to `vr1-dc1`/`vr1-dc2`, VLANs per the plane->VID scheme (mirror `docs/maas-as-built-reference.md`; note metal-internal VID 103 per the buildout doc). VIDs reuse per DC (separate fabrics). **4f. Tenant GUA delegation pool.** Per D-101, a per-DC tenant GUA pool (from the DC's GUA carve or a dedicated block) from which `/60`s are delegated (SLAAC, native-routed). Model the pool; per- tenant `/60`s are Neutron-managed (D-016/D-074), not enumerated in NetBox. --- ## 5. Sub-decisions requiring operator ratification (candidate D-111+) None of these block *reading* NetBox; they block *trusting the exact values written*. Each is a grep-before-assign D-NNN when ruled. 1. **Per-plane subcarve (the big one).** `dc-dc-prefixes-import.py`'s ULA/GUA subdivision is the script's OWN proposal, explicitly un-ratified: ULA = a `/56`-per-DC out of the org `/48` (dc1 `::0/56`, dc2 `::1:00/56`), then a `/64` per ULA-bearing plane in fixed order; GUA = provider-public gets the whole `DC_GUA_PREFIX` as one `/64`. **Two conflicts to resolve:** (a) this does NOT use the deployed `NN` role mnemonic (provider at `NN=10`, etc.) -- decide whether to align the tool to the deployed mnemonic or accept the tool's index scheme; (b) the ULA `/56`-per-DC index is independent of the GUA geography -- confirm that's acceptable (it matches the "internal `/48` reused across the domain" intent). **Recommend:** align the tool's per-plane `/64` offsets to the `NN` mnemonic so GUA and ULA read consistently, then ratify. **RATIFIED as D-111 (2026-07-11) + implemented in `carve_v6` (DOCFIX-182):** NN offsets, `/60`+`/64` per plane, metal `/60` shared by admin `:20` / internal `:21`, ULA `/56` indexed to the GUA nibble so the 4th hextet reads `DC.NN` in both families. 2. ~~**DC naming (G5).**~~ **SUPERSEDED -- do NOT follow this.** This recommended renaming the APEX to `vr1-dc1`/`vr1-dc2`. **D-117 (ADOPTED 2026-07-13) ruled the opposite:** the repo moves to the apex's names (`dc0`/`dc1`), because NetBox is the IPAM apex and is not bent to fit the repo. The import tool now REJECTS `--dc dc2` and the `DC2_V4_SUPERNET` env var by name. 3. **DC2 v4 supernet value + size.** D-101 leaves it NetBox-assigned. The import tool (`DC2_MIN_SUPERNET_PREFIXLEN = 19`) carves **six SEQUENTIAL `/22`s** from the supernet base and FAILS LOUD below `/19` (a `/19` = 8 `/22`s; six used, two spare). So a **`/19` is correct** (`10.12.64.0/19` per D-115; was `10.13.0.0/19`); DC2's planes land at sequential offsets (`.0/.4/.8/.12/.16/.20`), NOT mirroring DC1's `.4/.8/.12/.16/.32/.36` -- fine per D-101 (same six-plane *structure*, not same offsets). Just pick a non-colliding `/19` (per D-074). 4. **Tenant size confirmation.** D-101 says `/60`. Confirm the design-chat `/64` is dropped (it is, per section 3) so Code builds the `/60` delegation model. 5. **VR0 DC0 old-model retirement.** Keep for history vs retire once VR1 supersedes the rehearsal (D-101 site-collision note leaves this to the operator). --- ## 6. Cross-references - **Decisions:** D-100 (fabric/topology), **D-101** (address plan + family matrix; folds D-102 tenant/MTU), D-103 (IaC seam; NetBox apex), D-104 (controllers), D-106 (Designate A/AAAA), D-107 (per-DC mirror), D-108 (replication carrier = ULA), D-109 (Vault v6-SAN). D-010 (NetBox-upstream), D-052/D-053 (six-plane), D-016/D-074 (tenant CIDR). - **Tooling:** `netbox/dc-dc-prefixes-import.py` (+ `README.md`), `netbox/ipv6-mark-reserved.py`, `scripts/lib-net.sh` (`SPACES6`/`PLANE_CIDRS`), `docs/dc-dc-buildout-design.md` (Phase 1 gate), `docs/maas-as-built-reference.md` (VLANs/fabrics), `docs/dc-dc-replication-DR-seed.md`. - **Phase:** this is the **Phase 1** NetBox-population gate ("NetBox authoritative and populated: planes, per-DC v4, ULA/GUA") in `dc-dc-buildout-design.md`. ## 7. Open literals summary (hand to whoever assigns them) | Literal | Status | Recommendation | |---|---|---| | `ORG_ULA_48` | unassigned | `fd50:840e:74e2::/48` (ratify) | | `ORG_ULA_48` | **ASSIGNED (D-118)** | **`fd50:840e:74e2::/48`** -- RFC 4193 valid, no collision with Tailscale`s `fd7a:115c:a1e0::/48`. Unblocks the DC import. | | `DC_GUA_PREFIX` **dc0** | **MEASURED** (D-117) | `2602:f3e2:f02::/48` -- the apex's `vr1-dc0` block. Not renamed; the repo adopted the apex's name. | | `DC_GUA_PREFIX` **dc1** | **MEASURED** (D-117) | `2602:f3e2:f03::/48` -- the apex's `vr1-dc1` block. | | `DC2_V4_SUPERNET` | **ASSIGNED (D-115)** | **`10.12.64.0/19`** -- INSIDE the Cloud `/16`, free (DC1 holds `10.12.{4,8,12,16,32,36}`), DC1<->DC2 stay routable. The old `10.13.0.0/19` recommendation was OUTSIDE every allocated block (it would have squatted); D-115 supersedes it. Tool needs >= `/19` and re-packs 6 sequential `/22`s -- see 5.3. | ## 8. The NetBox sandbox loop (C4) ### 8.1 Why a second NetBox exists **DOCFIX-195 (operator ruling 2026-07-15) -- READ THIS FIRST; it INVERTS the roles the rest of this section was originally written around.** During VR1 the VR1 IPAM APEX is `office1-netbox` (`10.10.1.10:8000`, NetBox 4.6.4 on the Office1 VM, `docs/changelog-20260713-office1-netbox-deployed.md`): it takes ALL VR1 writes (the section-4 buildout, the D-120 additions, any further refinement) and every consuming system (OpenTofu, MAAS, overlays, the Juju bundle) derives its NetBox values from it, not from `netbox.baldurkeep.com`. It is a *working draft* -- edited as the buildout proceeds -- but it is the authoritative NetBox. `netbox.baldurkeep.com` is held as a READ-ONLY v1 REFERENCE DRAFT for the duration -- it gets NO write during VR1. Merging the validated `office1-netbox` dataset back into `netbox.baldurkeep.com` is a DEFERRED, end-of-deployment MAYBE (8.6), not a Stage 2 gate. This DEFERS the apex write; it does NOT revoke D-010 -- `netbox.baldurkeep.com` remains the eventual IPAM source of truth. The loop that builds and validates `office1-netbox` from the draft is unchanged; only the FINAL step (feeding the apex) moved OUT of Stage 2: upstream draft (READ-ONLY, v1 reference) --[prod-draft-dump.py]--> netbox/draft/vr1-draft.json vr1-draft.json --[sandbox-seed.py]------> office1-netbox (the RECORD) ...apply the planned delta in office1-netbox (the section-4 importers + the D-120 additions)... ...prove office1-netbox == draft + EXACTLY the planned delta (fidelity check)... validated dataset --[DEFERRED, end-of-deployment, operator-gated]-> apex The importers NEVER `--commit` the apex as a side effect (8.4). Writing `office1-netbox` is normal and unguarded (it is in `SANDBOX_HOSTS`); writing `netbox.baldurkeep.com` stays behind `--yes-write-upstream` and is out of scope until the deferred merge-back. ### 8.2 Why a FIDELITY CHECK, and not just counts Counts and idempotency CANNOT prove fidelity. On 2026-07-13 the seeder silently dropped the scope on 17 of 90 prefixes -- it mapped only `dcim.site`, and most of the VR1 draft is `dcim.region`-scoped -- and every count still matched and re-running stayed idempotent. It was caught by luck on a spot-check printing `site None`. `netbox/sandbox-fidelity-check.py` is the check that catches it on purpose: it diffs FIELD BY FIELD on every shared object, so a dropped or mangled field is a hard failure. ### 8.3 The 2026-07-14 hardening -- do NOT trust a pre-hardening "faithful" verdict Until 2026-07-14 the fidelity check could return a FALSE GREEN. It only proved the sandbox delta was a SUBSET of the plan (`unexpected = extra - expected`); it NEVER proved the plan was CARRIED OUT. With ZERO of the 36 DC prefixes created, `extra` and `unexpected` were both empty, and it printed "Nothing lost, nothing stray" and exited 0. `EXPECTED_NEW` was an upper bound masquerading as an assertion. Fixed 2026-07-14 (`docs/changelog-20260714-netbox-write-path-hardening.md`): `EXPECTED_NEW` is now BOTH bounds (`EXPECTED-BUT-ABSENT` is a hard failure), and delta prefixes are checked for scope and role. A NEW harness `tests/sandbox-fidelity-check/run-tests.sh` (10/10; T3 IS the false-green scenario) ships with it -- the checker had shipped with NO harness, which is why the false green survived. CONSEQUENCE: any "sandbox is faithful" claim dated before 2026-07-14 was made by a checker that could not see an unapplied or under-applied plan. **Re-run the current checker before trusting fidelity** (this is why the ledger flags a sandbox re-verify owed before C2). Two blind spots remain OPEN (logged, not fixed): SAME-DUMPER (both sides use the same field list, so a field the dumper omits is invisible on both), and DUPLICATE-CIDR collapse (prefix-keyed dicts collapse duplicates; latent -- none upstream today). "Faithful" means faithful within the dumped field list, not absolutely. ### 8.4 Write-guard architecture -- FIVE tools, and how each is kept off the apex "The sim never writes upstream" is enforced differently per tool. As of 2026-07-14 the three write-capable importers ALL gate an upstream `--commit`, but by two different mechanisms: | Tool | Default | Upstream write-guard | |---|---|---| | `prod-draft-dump.py` | read-only; no `--commit` | N/A -- performs no writes | | `sandbox-seed.py` | DRY; `--commit` writes | STRUCTURAL: holds no upstream URL/token and dies if pointed at the draft's own source host -- it CANNOT reach upstream | | `d115-office-carve.py` | DRY; `--commit` writes | `SANDBOX_HOSTS` allowlist `{localhost,127.0.0.1,10.10.1.10}`; a non-sandbox `--commit` REFUSES without `--yes-write-upstream` | | `roles-aggregates-import.py` | PREVIEW; `--commit` writes | **`SANDBOX_HOSTS` + `--yes-write-upstream` (added 2026-07-14)** -- plus a whole-plan PREFLIGHT so it cannot half-write | | `dc-dc-prefixes-import.py` | DRY; `--commit` writes | **`SANDBOX_HOSTS` + `--yes-write-upstream` (added 2026-07-14)** -- plus a whole-plan role PREFLIGHT so it cannot half-write | Before 2026-07-14 the two pynetbox importers had NO target guard at all; the only thing stopping an accidental upstream `--commit` was the WAF 403 (8.5) -- and fixing the WAF (below) removed that accidental safety, which is exactly why the explicit `--yes-write-upstream` gate was added in the same change (`docs/changelog-20260714-netbox-importers-waf-useragent.md`). The whole-plan preflights are the OTHER layer: both importers had previously died mid-loop and left the production IPAM half-populated (`docs/changelog-20260714-netbox-write-path-hardening.md`). ### 8.5 The upstream WAF trap (FIXED 2026-07-14) Upstream NetBox sits behind a User-Agent-filtering WAF: `urllib`/`requests`/`pynetbox` get a 403 that looks EXACTLY like an auth failure, while the SAME token works from `curl` (measured 2026-07-13, re-measured 2026-07-14: `curl` -> 200, `urllib`/pynetbox -> 403; `references/platform-traps.md`). Do NOT diagnose it as a token problem and re-mint credentials -- confirm with `curl` first. The three stdlib tools always sent an accepted UA (`curl/8.5.0`). The two pynetbox importers did NOT until 2026-07-14 -- meaning they could NEVER have written to the apex, only to the WAF-less sandbox. `get_nb()` in both now sets `nb.http_session.headers["User-Agent"] = "curl/8.5.0"` (`docs/changelog-20260714-netbox-importers-waf-useragent.md`). NOTE: `pynetbox` is NOT installed on `vcloud` -- the two pynetbox importers must run on `office1-netbox` (pynetbox 7.0.0), which reaches upstream through the edge. ### 8.6 C2 -- office1-netbox holds the validated dataset (apex write-back DEFERRED) **DOCFIX-195 (operator ruling 2026-07-15) REDEFINED C2.** C2 is NO LONGER "feed the carve upstream" -- there is NO `netbox.baldurkeep.com` write in Stage 2 at all. C2 is met when `office1-netbox` (the VR1 IPAM apex) holds the COMPLETE validated dataset AND it is VERIFIED. Two sub-tasks remain: (i) Re-run the HARDENED fidelity check against `office1-netbox` for the shared carve/roles/DCs -- the "faithful" verdict predates the 2026-07-14 hardening (8.3), so it must be re-established. (ii) Load the D-120 additions into `office1-netbox` -- `10.10.1.0/24`'s child ranges (`ip-ranges`: static .2-.49 / dynamic .100-.200 / node .201-.254) and the two service IP assignments (`ip-addresses`: office1-netbox 10.10.1.10, office1-tailscale 10.10.1.11) -- and verify them. **TOOLING GAP (logged DOCFIX-195, NOT fixed here).** The D-120 objects are `ip-ranges`/`ip-addresses` -- object classes the sandbox loop does NOT cover: `prod-draft-dump.py`'s ENDPOINTS and `sandbox-fidelity-check.py`'s KEY/EXPECTED_NEW both stop at rirs/roles/regions/sites/aggregates/ prefixes. So the hardened fidelity check CANNOT see the D-120 load. Verifying sub-task (ii) requires EITHER extending both tools (add the two endpoints + KEY entries + EXPECTED_NEW bounds + delta field checks, each with its harness) OR a recorded manual read-back against `office1-netbox`. That choice is an open C2-execution sub-question, to be ruled when C2 runs. **The DEFERRED merge-back (end-of-deployment, MAYBE).** If/when the operator elects to seed `netbox.baldurkeep.com` from the validated `office1-netbox` dataset, it is a production IPAM write and NOT part of the sim: (a) run the two pynetbox importers from `office1-netbox` (pynetbox + apex reach); (b) each needs `--yes-write-upstream` on top of `--commit` for the non-sandbox target; (c) the NetBox write-path bugs (session-ledger "NetBox WRITE-PATH BUGS") gate THAT write, not Stage 2 close; (d) present and individually approve each mutation -- never batched. As of 2026-07-15 the apex is untouched and still holds only the v1 draft (measured read-only).