Newer
Older
openstack-caracal-dc-dc / docs / dc-dc-netbox-buildout-scope.md

DC-DC (VR1) NetBox Buildout -- Planning & Decision Scope

Status: planning/scope reference for the Code stream. NOT a decision record -- it consolidates ADOPTED decisions and flags the sub-decisions still needing operator ratification (candidate D-111+; assign via ledger-scan at adoption). Authority order: repo design-decisions.md (ADOPTED) > this doc > live NetBox (a reconciliation target, not a source) > design-exploration chats. NetBox is the IPAM apex (D-010, D-103) -- everything below it consumes it; it is never read back as a planning input. Scope: what must exist in netbox.baldurkeep.com before Phase 1 completes so the VR1 two-DC buildout can consume it. Reconciled against a live read of NetBox on 2026-07-10 (v4.5.8).


1. Authoritative address/family model (D-101, D-100, D-108)

Six planes per DC, family follows reachability. Verbatim from D-101 / dc-dc-buildout-design.md:

Plane v4 (DC1 = DC0 template) v6 family Notes
provider-public 10.12.4.0/22 GUA (dual) public API VIPs + FIP/ext_net; GUA via owned OPNsense edge (D-015 gate N/A in VR1)
metal-admin 10.12.8.0/22 ULA (dual) admin-API ULA leg (2026-07-09 amendment); PXE stays v4-first
metal-internal 10.12.12.0/22 ULA (dual) API endpoints dual-stack; MySQL/RabbitMQ stay v4
data-tenant 10.12.16.0/22 ULA-only geneve overlay; tenant GUA delegated separately
storage 10.12.32.0/22 ULA-only Ceph public; ms_bind_ipv6
replication 10.12.36.0/22 ULA-only Ceph cluster; the cross-DC leg rides this ULA plane (D-108)

Also ULA-only: Octavia lb-mgmt and P2P links. IPv4-retained: MAAS PXE (v4-first) and datastore east-west (MySQL/RabbitMQ). Tenant GUA: native routing, no NAT66, SLAAC default, /60 per tenant (~4080/pool; /56 on request) -- delegated from a per-DC tenant GUA pool, distinct from the infra planes.

Per-DC v4 (D-101): DC1 inherits DC0's six-plane CIDRs unchanged (the six above). DC2 gets a distinct, non-overlapping parallel supernet, same six-plane structure, non-colliding with DC1 + operator infra + client VPN (D-074). Cross-DC replication carrier = IPv6-only ULA /64 (D-101/D-108; supersedes the DR-seed 10.12.36.0/22 v4 line).

**Three literals are NetBox-assigned, NOT hardcoded in any decision (D-101 open item / tooling-gap

#3):** the org ULA /48, the per-DC GUA carve out of the /36, and DC2's v4 supernet. The import tool takes them as required env vars with no defaults.


2. Live NetBox state (2026-07-10) -- reconciliation

netbox.baldurkeep.com v4.5.8. Regions: US West -> Eugene (prod); Virtual Regions -> VR0, VR1. Sites incl. VR1 DC0 (vr1-dc0), VR1 DC1 (vr1-dc1). 90 prefixes (77 v6 / 13 v4), 56 VLANs, 2 VRFs (Roosevelt, Willamette), 1 RIR (ARIN).

What is already right, and what is missing/divergent -- Code must close these before the import runs:

# Finding State Action
G1 Six-plane IPAM roles Only storage exists; provider-public, metal-admin, metal-internal, data-tenant, replication MISSING Create the 5 roles out-of-band (import script does NOT create roles)
G2 Aggregates 0 aggregates despite ARIN RIR present Create aggregates (section 4b) -- IPAM apex should declare its allocations
G3 Org ULA /48 Not present (only fc00::/8, fd00::/8 containers) Assign + create (section 4c); the D-101 literal
G4 VR1 VLAN groups VR1 DC0/DC1 have no VLAN groups (VR0 DC0 + prod sites do) Create per-DC VLAN groups + VLANs
G5 DC naming NetBox = vr1-dc0/vr1-dc1; decisions + import tool = DC1/DC2 (--dc dc1 targets vr1-dc1, DC2 -> vr1-dc2 which does not exist) Operator IPAM-hygiene call: rename to vr1-dc1/vr1-dc2, or re-point the tool. Blocks the import until resolved
G6 VR0 DC0 / Willamette prefixes Fully built on the OLD model: single roles (provider/metal/data/storage/repl/lbaas-mgmt), all GUA, no ULA This is the v1/DC0-era model, not the VR1 template. Leave as-is or retire (hygiene call, D-101 site-collision note)

3. Structure to build on (RECONCILED template)

The deployed carve and D-101 agree on a byte-aligned layout; use it for VR1. Template 2602:f3e2:0RSS:CCNN where nibble-1 = 0 (region-0 = the held /36; varies once expanded to the /32), nibble-2 = the region/metro selector (Eugene=1, VR0=e, VR1=f), SS = site byte (/48), CC = cloud byte (/56), NN = net byte (/64). Net-byte role mnemonic already in use: 10 provider, 11 provider-VIP, 20 metal, 30 data, 40 storage, 50 replication, 80 lbaas, e0 VPN, f0 OOB.

VR1 already occupies 2602:f3e2:f00::/40 with DC0=f02::/48, DC1=f03::/48 as sites. The per-DC GUA carve (D-101 literal) should be these existing /48 site blocks (DC_GUA_PREFIX = 2602:f3e2:f02::/48 for DC1, 2602:f3e2:f03::/48 for DC2), aligned to G5's naming decision -- not the README's placeholder 2602:f3e2:1000::/40.

RECONCILIATION NOTE (design-chat vs repo -- for the operator, so nothing surprises you): The parallel IPAM-design chat produced a carve (v3 / dual-region bundle) that diverged from what is deployed and adopted. Those artifacts are superseded for the DC-DC buildout; the deployed pattern + D-101 govern. Specifically:

  • Region label: chat used RMSS (region=/36, a "metro" at /40); deployed + D-101 use region-0=/36 with the /40 as the region/metro selector (0RSS). Addresses match for production; only VR0/VR1 placement differed -- the chat bundle put them at nibble-1 (2602:f3e2:e000::/36, f000::/36); NetBox has them at nibble-2 (0e00/0f00::/40). Deployed nibble-2 placement wins.
  • Tenant size: chat concluded /64-per-tenant (assuming tenants share the infra cloud /56); D-101 says /60, because tenant GUA is a separate delegation pool, not the infra /56. D-101 /60 wins.
  • Roles: chat used the old 5-6 role set; VR1 uses the six-plane set. Six-plane wins.

4. Buildout steps for Code (ordered; all idempotent, dry-run first)

4a. IPAM roles (G1). Create the 5 missing six-plane roles (provider-public, metal-admin, metal-internal, data-tenant, replication; storage exists). Slugs must match scripts/lib-net.sh SPACES6 exactly -- the import tool references them by slug. Created by netbox/roles-aggregates-import.py (this delivery).

4b. Aggregates (G2). Under RIR ARIN: 2602:f3e2::/36 (the held allocation = region-0 of the planned /32; update to /32 post-expand -- declaring a /32 you don't hold is not honest for an IPAM apex), 23.157.124.0/24. Under a new RFC-4193 RIR: the org ULA /48 (4c). Under a new RFC-1918 RIR: 10.0.0.0/8, 172.16.0.0/12. All created idempotently by netbox/roles-aggregates-import.py (this delivery).

4c. Org ULA /48 (G3, D-101 literal). Assign one RFC-4193 pseudo-random /48. Recommended value: fd50:840e:74e2::/48 -- CSPRNG-generated (Global ID 0x50840e74e2), collision-checked against the Tailscale fd7a:115c:a1e0::/48 already in NetBox (distinct). This is a recommendation for the operator to ratify as the literal, not an invented fact; if adopted, it becomes ORG_ULA_48 for the import tool.

4d. Per-DC prefixes via netbox/dc-dc-prefixes-import.py (D-101). Run per DC with the three literals. Recommended literals (pending G5 naming + section 5 ratification):

# DC1 (v4 inherited; only v6 top-level blocks required)
ORG_ULA_48=fd50:840e:74e2::/48  DC_GUA_PREFIX=2602:f3e2:f02::/48  --dc dc1
# DC2 (distinct v4 supernet ALSO required -- >= /19; tool re-packs 6 sequential /22s, see 5.3)
ORG_ULA_48=fd50:840e:74e2::/48  DC_GUA_PREFIX=2602:f3e2:f03::/48  DC2_V4_SUPERNET=10.13.0.0/19 --dc dc2

Run --verify-only first. The tool creates the six-plane v4 + the D-101 family-matrix v6 legs.

4e. VLAN groups + VLANs (G4). Per-DC VLAN groups scoped to vr1-dc1/vr1-dc2, VLANs per the plane->VID scheme (mirror docs/maas-as-built-reference.md; note metal-internal VID 103 per the buildout doc). VIDs reuse per DC (separate fabrics).

4f. Tenant GUA delegation pool. Per D-101, a per-DC tenant GUA pool (from the DC's GUA carve or a dedicated block) from which /60s are delegated (SLAAC, native-routed). Model the pool; per- tenant /60s are Neutron-managed (D-016/D-074), not enumerated in NetBox.


5. Sub-decisions requiring operator ratification (candidate D-111+)

None of these block reading NetBox; they block trusting the exact values written. Each is a grep-before-assign D-NNN when ruled.

  1. Per-plane subcarve (the big one). dc-dc-prefixes-import.py's ULA/GUA subdivision is the script's OWN proposal, explicitly un-ratified: ULA = a /56-per-DC out of the org /48 (dc1 ::0/56, dc2 ::1:00/56), then a /64 per ULA-bearing plane in fixed order; GUA = provider-public gets the whole DC_GUA_PREFIX as one /64. Two conflicts to resolve: (a) this does NOT use the deployed NN role mnemonic (provider at NN=10, etc.) -- decide whether to align the tool to the deployed mnemonic or accept the tool's index scheme; (b) the ULA /56-per-DC index is independent of the GUA geography -- confirm that's acceptable (it matches the "internal /48 reused across the domain" intent). Recommend: align the tool's per-plane /64 offsets to the NN mnemonic so GUA and ULA read consistently, then ratify.

  2. DC naming (G5). vr1-dc0/vr1-dc1 (NetBox) vs DC1/DC2 (decisions/tool). Recommend: rename sites to vr1-dc1/vr1-dc2 to match the adopted DC1/DC2 language and the tool's --dc dc2 -> vr1-dc2 expectation; retire or repurpose the now-orphaned vr1-dc0.

  3. DC2 v4 supernet value + size. D-101 leaves it NetBox-assigned. The import tool (DC2_MIN_SUPERNET_PREFIXLEN = 19) carves six SEQUENTIAL /22s from the supernet base and FAILS LOUD below /19 (a /19 = 8 /22s; six used, two spare). So a /19 is correct (e.g. 10.13.0.0/19); DC2's planes land at sequential offsets (.0/.4/.8/.12/.16/.20), NOT mirroring DC1's .4/.8/.12/.16/.32/.36 -- fine per D-101 (same six-plane structure, not same offsets). Just pick a non-colliding /19 (per D-074).

  4. Tenant size confirmation. D-101 says /60. Confirm the design-chat /64 is dropped (it is, per section 3) so Code builds the /60 delegation model.

  5. VR0 DC0 old-model retirement. Keep for history vs retire once VR1 supersedes the rehearsal (D-101 site-collision note leaves this to the operator).


6. Cross-references

  • Decisions: D-100 (fabric/topology), D-101 (address plan + family matrix; folds D-102 tenant/MTU), D-103 (IaC seam; NetBox apex), D-104 (controllers), D-106 (Designate A/AAAA), D-107 (per-DC mirror), D-108 (replication carrier = ULA), D-109 (Vault v6-SAN). D-010 (NetBox-upstream), D-052/D-053 (six-plane), D-016/D-074 (tenant CIDR).
  • Tooling: netbox/dc-dc-prefixes-import.py (+ README.md), netbox/ipv6-mark-reserved.py, scripts/lib-net.sh (SPACES6/PLANE_CIDRS), docs/dc-dc-buildout-design.md (Phase 1 gate), docs/maas-as-built-reference.md (VLANs/fabrics), docs/dc-dc-replication-DR-seed.md.
  • Phase: this is the Phase 1 NetBox-population gate ("NetBox authoritative and populated: planes, per-DC v4, ULA/GUA") in dc-dc-buildout-design.md.

7. Open literals summary (hand to whoever assigns them)

Literal Status Recommendation
ORG_ULA_48 unassigned fd50:840e:74e2::/48 (ratify)
DC_GUA_PREFIX dc1 ambiguous 2602:f3e2:f02::/48 (existing VR1 DC0 site block, renamed dc1)
DC_GUA_PREFIX dc2 ambiguous 2602:f3e2:f03::/48 (existing VR1 DC1 site block, renamed dc2)
DC2_V4_SUPERNET unassigned 10.13.0.0/19 (distinct; tool needs >= /19, re-packs 6 sequential /22s -- see 5.3)