Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / audit / container-elim-pass / pass0-w2-network-map.md
@JANeumatrix JANeumatrix 8 hours ago 14 KB container-elim planning pass RAN (phases 0-4 + advisor): FINAL-PLAN + operator-report + D-144 framing (READ-ONLY)
Raw Blame History

Pass 0 / W0.2 -- Network & wiring map of the container layer

Agent: W0.2 (sonnet worker), container-elim pass, Phase 0. Read-only. All values below are cited to path:line or to a ruled D-NNN/SEC-NNN entry; nothing is inferred. Dated 2026-08-09.

1. As-is topology -- text diagram (dc0 arm; dc1 is the structural mirror)

vcloud (outer libvirt, qemu:///system) -- opentofu/main.tf
|
|-- office1_network (module, MTU=underlay_mtu=9000)  -- main.tf:76-80
|     `-- voffice1 (LXD/MAAS-region/NetBox host)      -- main.tf:175 (NIC1 enp1s0, DHCP/Kea)
|-- office1-wan (site-wan NAT, MTU 1500, not jumbo)
|-- office1_opnsense edge                              -- main.tf:98-114
|
|-- D-100 dark-fiber mesh TRIANGLE (mesh-link module, MTU=underlay_mtu=9000) -- main.tf:125-141
|     |-- mesh_vr1_dc0_vr1_dc1  (dc0<->dc1)   bridge virbr5 (MEASURED, `virsh net-info`)
|     |     netem: delay 3ms jitter 1ms loss 0.01% (PLACEHOLDER, D-100 gap #11 unruled)
|     |                                        -- main.tf:352-357 netem_vr1_dc0_vr1_dc1
|     |-- mesh_vr1_dc0_office1  (dc0<->office1, region<->rack TRANSIT)
|     |     region end: voffice1 NIC2 static 172.31.0.1/30      -- main.tf:191-192
|     |     rack   end: vvr1-dc0 NIC1 (enp1s0) static 172.31.0.2/30 -- main.tf:498-505, D-124
|     `-- mesh_vr1_dc1_office1  (dc1<->office1 transit)
|           region end: voffice1 NIC3 static 172.31.0.5/30       -- main.tf:193
|           rack   end: vvr1-dc1 NIC1 static 172.31.0.6/30       -- main.tf:602-609, D-124 amdt
|     (transit supernet 172.31.0.0/24, D-124 AMENDMENT 2026-07-16, design-decisions.md:5031)
|
|-- vr1_dc0_uplink (site-wan NAT, 172.30.2.0/24, MTU 1500)  -- main.tf:379-385, D-125/D-115
|-- vr1_dc1_uplink (site-wan NAT, 172.30.3.0/24, MTU 1500)  -- main.tf:388-395
|
`-- vvr1-dc0 CONTAINMENT VM (416 GiB/108 vCPU, expose_nested_virt=true) -- main.tf:410-522, D-123/D-124
      NIC1 enp1s0 = transit (region-facing; SEC-010 --transit-if KEYS HERE)
      NIC2 enp2s0 = uplink, IP-LESS, enslaved into netplan bridge br-vr1-dc0-wan (declared
                    IN vvr1-dc0's own cloud-init netplan, D-125 bridge-in)     -- main.tf:507-522
      |
      `== INNER root: opentofu/vr1-dc0-substrate/ (qemu+ssh to vvr1-dc0 over the transit,
          R-5, run from Office1 per D-128) ==
           |-- inner_storage pool                                  -- vr1-dc0-substrate/main.tf:26-30
           |-- vr1_dc0_planes (dc-planes module, isolated L2, no forward/DHCP,
           |     MTU = var.underlay_mtu = 9000)                    -- vr1-dc0-substrate/main.tf:32-40
           |     SIX PLANES (D-052/D-100 template, D-139-amended families):
           |       provider-public   10.12.4.0/22   dual-stack v4 + GUA 2602:f3e2:f02:10::/64
           |       metal-admin       10.12.8.0/22    dual-stack v4 + GUA 2602:f3e2:f02:20::/64
           |       metal-internal    10.12.12.0/22   IPv6-ONLY GUA 2602:f3e2:f02:21::/64 (v4 REMOVED, D-139 Ruling A)
           |       data-tenant       10.12.16.0/22   IPv6-ONLY GUA 2602:f3e2:f02:30::/64 (geneve underlay)
           |       storage           10.12.32.0/22   IPv6-ONLY GUA 2602:f3e2:f02:40::/64 (Ceph public)
           |       replication       10.12.36.0/22   IPv6-ONLY GUA 2602:f3e2:f02:50::/64 (Ceph cluster, cross-DC leg)
           |       lb-mgmt (NEW)     n/a             IPv6-ONLY GUA 2602:f3e2:f02:80::/64 (RESERVED, no charm consumer -- G18)
           |     lib-net.sh mirrors dc0's v4 CIDRs: PLANE_CIDRS (lib-net.sh:22), PLANE_NAME (:23-30)
           |-- vr1_dc0_wan (wan-bridge module: forward={mode=bridge} onto br-vr1-dc0-wan;
           |     NO mtu block -- provider REJECTS <mtu> on bridge-mode; inherits host
           |     bridge MTU=1500 from the outer netplan)            -- vr1-dc0-substrate/main.tf:51-56
           |-- vr1_dc0_opnsense DC edge (LAN=provider-public, WAN=vr1_dc0_wan bridge)
           |                                                       -- vr1-dc0-substrate/main.tf:60-73
           `-- vr1_dc0_node x 12 (9 role + juju-01 + maas-01 + tailscale-01)
                 NIC_PLANE_ORDER (lib-hosts.sh:69): metal-admin, provider-public,
                   metal-internal, data-tenant, storage, replication
                 role nodes (9): all 6 planes + OVS br-ex parented on enp2s0
                   (provider-public leg; BREX_PARENT_NIC, lib-hosts.sh:72) -- enp2s0
                   itself carries NO L3, br-ex carries the static (Pattern A, D-100/D-060)
                 juju-01 / maas-01 / tailscale-01 (utility VMs): 2 planes only
                   (metal-admin + provider-public, raw NIC WITH gateway, NO br-ex)
                   -- lib-hosts.sh:74-93

dc1 arm is structurally identical: vvr1-dc1 (main.tf:524-618), its own inner root opentofu/vr1-dc1-substrate/, planes 10.12.64/68/72/76/80/84.0/22 (lib-net.sh:171-179, D-124 AMENDMENT 2026-07-21), transit 172.31.0.4/30, uplink 172.30.3.0/24.

2. SEC-010 -- transit FORWARD-drop (forwarding choke point)

  • Scope: interface-scoped FORWARD-drop on the TRANSIT leg only (enp1s0 on vvr1-dc0), never a global ip_forward=0 and never globalized on the bridge -- br_netfilter makes bridged WAN frames traverse the L3 FORWARD chain, so a global drop would silently kill br-vr1-dc0-wan (design-decisions.md D-125, "br_netfilter CONSTRAINT").
  • Artifact: scripts/site-headend-install.sh --host-nodes writes /etc/nftables-sec010.nft + boot-persistent sec010-fw.service; --host-nodes --check is the mechanical gate (fails if the rule is absent OR the keyed transit interface does not exist -- an nftables oifname on an absent iface loads clean but matches nothing, fail-open class). CLOSED 2026-07-20, both ends (security-ledger.md SEC-010 row).
  • Why it sits where it does (nesting-specific rationale, verbatim from the ledger): "after the D-123 Model B reshape ... vvr1-dc0 now bridges ALL 6 inner planes + the transit, and belongs on the inner libvirt host, not just a 2-leg rack." SEC-010 is therefore the control that keeps vvr1-dc0's six bridged inner planes from being reachable FROM the region side of the transit -- it is doing the isolation job a separate host would otherwise do "for free."
  • The same pin follows onto voffice1's own transit leg (its enp2s0), independently verified (SEC-010 row, security-ledger.md).

3. MTU / jumbo / geneve-over-v6 budget

  • underlay_mtu = 9000 (jumbo) is the ONE knob applied to every plane network AND every mesh-link network: opentofu/dc-dc-phase0.auto.tfvars:7 -- "Step 3 ruling: jumbo internal fabric (tenant MTU stays 1500)". Consumed by vr1-dc0-substrate/main.tf:38 (mtu = var.underlay_mtu) and main.tf:127/133/139 (mesh links) and main.tf:78 (office1_network).
  • The WAN path is deliberately NOT jumbo: modules/wan-bridge (bridge-mode, no mtu block -- the provider REJECTS it in bridge mode, measured 2026-07-20) inherits the host bridge's 1500 from vvr1-dc0's outer netplan; modules/site-wan (office1-wan, the two uplink NATs) defaults to 1500 -- "the ISP-uplink domain; NOT the jumbo planes/mesh" (vr1-dc0-substrate/main.tf:55).
  • D-101 tenant-MTU sub-policy (design-decisions.md ~line 2280): geneve-over-v6 overhead is roughly 56 bytes (IPv6 40 + UDP 8 + Geneve base 8) before nested-virt/options. With the jumbo (9000) underlay, tenant MTU stays 1500 and amphora/geneve fit; if the underlay were pinned at 1500 instead, tenant MTU would drop to ~1444 (v6-geneve) and must be set consistently across ovn geneve, tenant-network MTU, and amphora. The jumbo underlay is what makes tenant MTU 1500 possible at all -- this is the plane-level fact the container-elim must preserve.
  • D-139 (2026-07-31): data-tenant (the geneve underlay plane) moved from IPv4 to IPv6-only GUA (2602:f3e2:f02:30::/64 dc0) -- geneve now rides GUA, not the historically planned ULA (Ruling B, "full GUA on every plane").
  • The 2026-08-08/09 geneve-over-v6 root cause (memory + scripts/geneve-encap-assert.sh header) was NOT an MTU/byte-budget defect. Two distinct causes, both at the OVN/OVS layer, neither is the vvr1-dcN containment hop itself:
    1. Encap-family SPLIT between the containerized control plane (LXD, v4) and metal compute chassis (v6) -- cross-family tunnels never form. Gate: geneve-encap-assert.sh C1.
    2. ovn-chassis 24.03 emitted a BRACKETED v6 ovn-encap-ip ([2602:...]) that OVS's geneve implementation rejects ("bad geneve 'remote_ip'"), leaving v6 tunnels at ofport -1. Gate: geneve-encap-assert.sh C2. Fixed live: unbracket + carve v6 on the LXD/containerized chassis + overlay_ip_version=6; VM->VM 8/8 0% loss confirmed (memory: dc0-checkpoint-then-reip-redeploy.md). NAMING TRAP reminder (per this pass's own framing): the "containerized control plane" here is the LXD API-charm containers on nodes, NOT the vvr1-dcN containment VM this pass eliminates -- do not credit or blame container-elim for this fix.
  • Net conclusion: the plane-level MTU/geneve byte budget (9000 underlay / 1500 tenant / ~56-byte v6-geneve overhead) is unaffected by container-elim -- the vvr1-dcN hop for plane traffic was already a same-MTU (9000) isolated libvirt bridge with no extra encapsulation, so removing it removes a bridging HOP, not a budget constraint.

4. What "simplify the wiring" precisely touches if vvr1-dcN is eliminated

COLLAPSES entirely (exists only because of the nesting):

  • vvr1-dc0's own two outer NICs (transit enp1s0, uplink enp2s0) and the containment VM itself -- their sole purpose is to give the inner qemu+ssh provider (R-5) something to dial and to give the inner WAN bridge an egress path.
  • br-vr1-dc0-wan (the netplan bridge INSIDE vvr1-dc0) and the modules/wan-bridge network realizing D-125's bridge-in -- this exists purely to fix OBS-3 (nested WAN NAT losing egress under Model B). Flat topology reverts to Model A's direct site-wan NAT; the repo already names this exact revert: "Fallback: docs/archive/model-a-fallback-plan.md section 3 (revert removes the uplink NIC/network/bridge + wan-bridge and restores the OPNsense WAN addr)" (D-125 entry).
  • The two-stage OpenTofu apply ordering (outer boots+sizes vvr1-dc0 -> bootstrap gate installs inner libvirtd -> inner root applies) and the qemu+ssh inner-provider URI itself (vvr1_dc0_transit_ip dial) -- one fewer apply stage, one fewer credential/URI to carry (SEC-026 already flags the MAAS-admin-key residency problem this ordering created).

RE-HOMES onto vcloud libvirt directly (same module, same shape, different provider target):

  • The SIX plane bridges (modules/dc-planes) -- same CIDRs, same MTU var (9000), same isolated/no-forward/no-DHCP posture (NetBox/OpenStack still own L3). D-139's family matrix is an OpenStack/NetBox-layer concern, not a libvirt-nesting one -- unaffected.
  • The 12 node-vm module calls per DC (9 role + juju-01/maas-01/tailscale-01) -- same NIC_PLANE_ORDER, same per-role carve shape (6 planes + br-ex for role nodes; 2 planes, no br-ex, for utility VMs), same MAC-pin-after-first-apply discipline.
  • The DC OPNsense edge -- LAN stays provider-public; WAN reverts to a direct site-wan NAT (Model A shape) instead of routing through a bridge inside a containment VM.

REQUIRES A TARGET-TOPOLOGY DECISION (flagged for W0.4, not resolved here):

  • SEC-010's isolation boundary. Today SEC-010 protects ONE DC's forwarding surface because vvr1-dc0 and vvr1-dc1 are SEPARATE nested libvirt hosts -- each bridges only its own six planes. If both DCs' planes flatten onto vcloud's SAME libvirtd, dc0's and dc1's plane bridges become co-resident on ONE host for the first time, a cross-DC adjacency that did not exist before and that SEC-010 (interface-scoped to a transit leg that no longer carries this function) does not cover. This is the single largest wiring risk of elimination and needs an explicit new host-level control (or a proposed topology that avoids co-residency) before Phase 1.
  • The transit leg's (D-124) surviving purpose. Its qemu+ssh-dial role disappears with the inner root, but MAAS PXE/commissioning reachability from Office1 to a DC-local region (D-132) and jujud's MAAS-API dial may still need SOME region<->rack leg depending on whether the target topology keeps "DC-local" as a routing/administrative concept at all. W0.3/W0.4's call.
  • The D-100 mesh triangle's fate. The three mesh-link segments represent inter-SITE links (dark-fiber stand-ins), a concept orthogonal to intra-site containment -- they likely survive container-elim UNLESS the target topology also collapses "DC" as a distinct site into one shared vcloud fabric with no site-to-site fiction needed. Also a W0.4 call.

5. Files / lines this map is built from (for W0.4 and later phases)

  • opentofu/main.tf: 76-148 (office1 net/edge, mesh triangle), 175-233 (voffice1 3-NIC transit host), 341-397 (netem, uplink NATs), 399-522 (vvr1-dc0 containment VM), 524-618 (vvr1-dc1 mirror).
  • opentofu/vr1-dc0-substrate/main.tf: 1-73 (inner provider, planes, wan-bridge, edge), 75-265 (node fleet + utility VMs, NIC order, carved planes per role).
  • opentofu/modules/dc-planes/main.tf, opentofu/modules/wan-bridge/main.tf, opentofu/modules/mesh-link/main.tf -- the three bridge-shape module bodies.
  • scripts/lib-net.sh: 22-30 (PLANE_CIDRS/PLANE_NAME dc0), 171-179 (dc1).
  • scripts/lib-hosts.sh: 53-100 (NIC_PLANE_ORDER, BREX_PARENT_NIC, per-role carve shape).
  • scripts/geneve-encap-assert.sh: 1-30 (the C1/C2 gate header, root-cause citation).
  • docs/design-decisions.md: D-100 (~2223), D-101 (~2246-2330), D-124 AMENDMENTs (~5022-5083), D-125 (~5084-5161), D-139 (~7253-7370).
  • docs/security-ledger.md: SEC-010 row (line 21), SEC-026 row (line 79).
  • docs/CURRENT-STATE.md: 7606-7710 (Sec 2.3-iii containment relocation note, Sec 3 authored-not- applied inventory) -- NOTE this section is dated 2026-07-18/20 and predates the later amphora/geneve/D-139 work; used here only for the nesting mechanism, not current apply state.

6. Top risks for the administrator / W0.4

  1. SEC-010 co-residency gap (Section 4) -- the single biggest wiring risk; no existing artifact addresses cross-DC plane-bridge adjacency on one flattened host.
  2. D-125's whole bridge-in mechanism is Model-B-only debt -- eliminating the container layer should DELETE it (revert to Model A direct NAT), not carry it forward unused.
  3. MTU/geneve budget is NOT the source of container-elim benefit -- do not let Phase 4's change-set imply an MTU fix; the real payoff is removing OBS-3-class nesting fragility and one apply-ordering stage, not a byte-budget change.
  4. Transit leg's surviving purpose is undetermined pending the target topology (W0.4) -- this doc intentionally stops short of ruling it out.