Newer
Older
openstack-caracal-dc-dc / docs / audit / decision-recon-20260725.md

Full decision reconciliation -- VR1 DC-DC (IP / VLAN / node / decision-status)

Date: 2026-07-25. Branch: dc-dc-stage4-phase3-maas-deploy. Convened by: operator ("Deploy a committee for a full decision recon. Are there any other reconciliations that should be queued afterwards?").

Status: ADVISORY / audit record. Reconciles RULED decisions against the authoritative surfaces; adopts nothing. docs/CURRENT-STATE.md remains the sole status authority (GA-R1). Companion to docs/audit/committee-20260724-track2-bundle-render.md and docs/audit/stage5-expansion-review-20260724.md.

Method: four independent READ-ONLY reviewers, each a recon dimension (IPAM prefixes/bands; VLAN/space/fabric; node/tag/MAC/power; decision-status + backlog), reconciling from COMMITTED surfaces: the office1-netbox apex snapshot netbox/draft/vr1-office1-current-20260725.json, scripts/lib-net.sh + lib-hosts.sh, bundle.yaml + overlays/, opentofu/, the MAAS carve captures (docs/audit/stage4-carve-verify-20260723.txt, stage4-discovery-20260723.txt), and docs/design-decisions.md + docs/CURRENT-STATE.md. Live-truth items are flagged NEEDS-LIVE-VERIFY.

Bottom line (the operator's question: is office1-netbox updated with all IP + VLAN decisions?)

Layer In the apex? Verdict
VLANs zero CORRECT BY DESIGN -- VR1 is untagged-per-fabric (D-133); VLANs are a MAAS-layer construct, never NetBox
Plane /22s, transit, uplink, v6 GUA/ULA, aggregates all present, drift-free COMPLETE -- all 12 planes + both transit/uplink + both /48s, matching lib-net value-for-value
D-134 per-DC bands (utility/VIP/node/dynamic) absent GAP -- exist only as decision prose; not in NetBox OR lib-net
The 33 VIP addresses (.50-.60 x 3 legs) absent GAP -- overlay-only literals; a generator has nothing to pull

Dimension verdicts

1. IPAM (prefixes / bands)

Prefix-layer COMPLETE + ZERO DRIFT across netbox <-> lib-net <-> artifacts:

  • All 12 plane /22s (6 dc0: 10.12.4/8/12/16/32/36; 6 dc1: 10.12.64/68/72/76/80/84), correctly scope_site-scoped + role-tagged.
  • Transit /30s (dc0 172.31.0.0/30, dc1 172.31.0.4/30); uplinks (172.30.2/24, 172.30.3/24); GUA /48s (2602:f3e2:f02 dc0, :f03 dc1) + ARIN 2602:f3e2::/36 + full v6 GUA/ULA subcarves; org ULA fd50:840e:74e2::/48. D-120 Office1 bands present.
  • No drift between lib-net.sh and the apex dump -- every value matches (the dc1 arm and opentofu/variables.tf twin honor "change one, change both"). GAP = sub-prefix only:
  • D-134 per-DC bands (utility .4-.49, VIP .50-.99, node .100-.200 [control .100-.119 / compute .120-.149 / storage .150-.200], dynamic .201-.254) -- MISSING in NetBox AND in lib-net (lib-net carries only the dc0 VIP octet band .50-.60, UNSET for dc1); exist only as D-134 prose. [blocks-coupling]
  • The 33 VIP addresses (11 services x provider/admin/internal, .50-.60) -- MISSING in NetBox; overlay-only literals in overlays/vr1-dc1-vips.yaml (internally consistent with bundle octet-for-octet, but not an apex projection). [blocks-coupling]
  • Per-DC infra statics (gw .1, DNS-fwd .3 D-131, mirror .4 D-135) + the dc1 10.12.64.0/19 container prefix -- MISSING. [hygiene]

2. VLAN / space / fabric

  • office1-netbox holds ZERO VLAN objects = CORRECT BY DESIGN. VR1 runs untagged-per-fabric on every plane, both DCs (D-133 supersedes VR0's VID-103/104 for VR1 nodes; carve capture confirms flat static: links, no .103/.104 sub-interfaces). A VLAN object would document tagging that does not exist on the wire -- exactly what design-decisions.md:217 prohibits.
  • The 6 D-052/D-053 spaces MATCH character-for-character across bundle / lib-net / carve. The "12 MAAS VLANs" = 6 planes x 2 DCs on default untagged (VID 0) fabric VLANs (MAAS constructs). VLAN 5005 (dc0) = MAAS db-id of the untagged boot fabric; dc1 twin = fabric-142/vid 0. 10 named plane fabrics + 2 boot = 12.
  • Residual VR0 VID-103/104 constants in lib-net.sh are caveated (D-133) + UNSET for the vr1-dc1 arm -- functionally inert (LOW hygiene).
  • Coupling input: VLAN/space/fabric data, if ever needed, comes from MAAS, not NetBox. The bundle binds by space NAME -> MAAS subnet by CIDR (never by VLAN/fabric/subnet id, which drift). NetBox = IPAM (prefixes) only.

3. Node / tag / MAC / power

  • Config-of-record DRIFT-FREE: D-121 Option C shapes match (substrate <-> lib-hosts <-> discovery, 9/9 both DCs); pinned MACs match across substrate main.tf <-> lib-hosts HOST_BOOT_MAC (:01) <-> bundle ovn-chassis (:02 compute provider) <-> discovery; per-DC power isolation intact (SEC-012 dc0 172.31.0.2 / SEC-016 dc1 172.31.0.6, distinct IdentityFile; keys off-repo).
  • Two pending gated deltas (NOT drift): (a) the Fork-3 per-role tags the rendered bundle now hard-depends on are NOT yet on the MAAS nodes -- HIGH, deploy-blocks placement if skipped; (b) the D-104 10th controller VM is un-built in both substrates (expected; needs its own juju-controller-vr1-dc1 tag, no role tag).
  • NEEDS-LIVE-VERIFY: the discovery capture has NO tag column, so even the DC-scoping tag's live application on the nodes is unproven by committed evidence ("tags applied 9+9" is prose). Verify live MAAS tag state before deploy. Also: per-node vCPU/mem live values; the SEC key IdentityFile mapping.

4. Decision-status integrity

Session-touched decisions (D-104 amendment, BUNDLEFIX-052, the render, Fork 2/3 as OPS, the coupling routed to Chat) are mutually consistent across CURRENT-STATE + changelog + design-decisions; ledger-scan PROPOSED/OPEN {D-068, D-131, D-132} matches CURRENT-STATE section 8. Stale surfaces found (to fix -- deferred per operator):

  • HIGH -- CURRENT-STATE section 4 "RULED-BUT-NOT-BUILT" is comprehensively stale (authored 2026-07-18, never updated): 4 of 5 bullets contradicted by section 1 + on-disk -- claims "Nothing DC0 is built" (DC0 built/commissioned, G10 CLOSED), dc1 substrate "UNBUILT" (roots exist, G12 CLOSED), D-100 netem "HELD as comment main.tf:309-319" (instantiated main.tf:353, applied Step E), D-129 qga "not written" (shipped, G13 CLOSED). The status authority contradicts itself; only GA-R1/C2 + "read section 1 first" masks it.
  • MED -- G14 SEC count stale: says 12 open; ledger-scan now 15 (adds SEC-017/-018/-019).
  • MED -- runbooks/dc-dc-phase4-juju-bundle-per-dc.md:192 still says bundle "ships NO designate" (contradicts ADOPTED D-106 + the actual bundle:830).
  • LOW -- changelog "Next" block internally stale (items 6-7 done); opentofu/main.tf:18-19 comment says netem "not instantiated yet" (it is, :353).

Ranked gaps -- what must land in office1-netbox for a NetBox->deploy generator

  1. D-134 per-DC bands as ip-ranges on every DC plane, both DCs (utility/VIP/node [sub-role]/dynamic). Prose only today. [blocks-coupling]
  2. The 33 VIP addresses (.50-.60) as scoped ip-addresses per DC so the overlay is a projection of NetBox, not a hand-maintained literal. [blocks-coupling]
  3. Per-DC infra statics (gw .1, DNS-fwd .3, mirror .4). [hygiene]
  4. dc1 10.12.64.0/19 container prefix. [hygiene]

Reconciliation backlog (the "what else to queue" answer)

Prio Recon Surfaces When Blocks
P0 VIP-overlay collision + vault-VIP completeness vr1-dc1-vips vs ipv6-family-matrix (both own vip:); rendered vault (no vip) vs D-036 before dc1 deploy dc1 deploy
P0 rendered bundle/overlays <-> apex bundle + overlays (VIP band, tags, ovn MACs) vs the apex dump before deploy (--dc vr1-dc1 + --dry-run) deploy correctness
P1 extend netbox/sandbox-fidelity-check.py (STALE) the tool: covers only D-115/117/120, diffs the frozen 90-prefix draft not the 139-prefix apex; blind to D-124/D-134 before the coupling design trustworthy apex recons
P1 lib-net/lib-hosts literals <-> apex 28 IP + 26 IP/MAC literals vs current apex at coupling design; on any apex mutation coupling adoption
P1 MAAS live <-> NetBox D-134 bands + the 12 VLANs / 6 spaces (MAAS-only) vs apex end-of-deploy + coupling apex completeness / merge-back
P2 post-deploy cloud <-> decisions (BOM) cloud-assert --capture vs D-121 HA / D-036 vault / D-108 / Ceph size/CRUSH / cluster_count after dc1 deploys Stage-5 close
P2 IPv6 / dual-stack ipv6-family-matrix v6 tokens vs apex v6 once v6 rendered (depends on P0 phasing) dual-stack
P2 office1-netbox -> netbox.baldurkeep.com merge-back 4.6.4 apex -> 4.5.8 frozen v1 reference end-of-deployment (DOCFIX-195) IPAM-of-record loop
P2 SEC-ledger <-> reality security-ledger 15 open rows vs rotations/scope G14 reconcile now; bulk at v1 close security close-out

Sequencing insight: extend the fidelity tool FIRST -- it is the prerequisite that makes every apex recon (bundle<->apex, libs<->apex, MAAS<->NetBox) trustworthy, and the coupling's whole value proposition is eliminating the hand-render/carried-literal drift those recons chase.

Implications for the NetBox->deploy coupling (the Chat D-number)

The recon VALIDATES the coupling's scope precisely: the gap is NOT the plane/prefix modeling (already faithful, drift-free) -- it is the sub-prefix band + VIP-address layer living in prose/overlays. So the coupling should: (1) model the D-134 bands + VIP addresses IN the apex; (2) generate the overlays from them; (3) reconcile the two VIP overlays into one per-app string; (4) ride an EXTENDED fidelity check. VLAN/space/ fabric stays MAAS-sourced (NetBox = IPAM only).