# Pass 0 / W0.2 -- Network & wiring map of the container layer **Agent:** W0.2 (sonnet worker), container-elim pass, Phase 0. Read-only. All values below are cited to path:line or to a ruled D-NNN/SEC-NNN entry; nothing is inferred. Dated 2026-08-09. --- ## 1. As-is topology -- text diagram (dc0 arm; dc1 is the structural mirror) ``` vcloud (outer libvirt, qemu:///system) -- opentofu/main.tf | |-- office1_network (module, MTU=underlay_mtu=9000) -- main.tf:76-80 | `-- voffice1 (LXD/MAAS-region/NetBox host) -- main.tf:175 (NIC1 enp1s0, DHCP/Kea) |-- office1-wan (site-wan NAT, MTU 1500, not jumbo) |-- office1_opnsense edge -- main.tf:98-114 | |-- D-100 dark-fiber mesh TRIANGLE (mesh-link module, MTU=underlay_mtu=9000) -- main.tf:125-141 | |-- mesh_vr1_dc0_vr1_dc1 (dc0<->dc1) bridge virbr5 (MEASURED, `virsh net-info`) | | netem: delay 3ms jitter 1ms loss 0.01% (PLACEHOLDER, D-100 gap #11 unruled) | | -- main.tf:352-357 netem_vr1_dc0_vr1_dc1 | |-- mesh_vr1_dc0_office1 (dc0<->office1, region<->rack TRANSIT) | | region end: voffice1 NIC2 static 172.31.0.1/30 -- main.tf:191-192 | | rack end: vvr1-dc0 NIC1 (enp1s0) static 172.31.0.2/30 -- main.tf:498-505, D-124 | `-- mesh_vr1_dc1_office1 (dc1<->office1 transit) | region end: voffice1 NIC3 static 172.31.0.5/30 -- main.tf:193 | rack end: vvr1-dc1 NIC1 static 172.31.0.6/30 -- main.tf:602-609, D-124 amdt | (transit supernet 172.31.0.0/24, D-124 AMENDMENT 2026-07-16, design-decisions.md:5031) | |-- vr1_dc0_uplink (site-wan NAT, 172.30.2.0/24, MTU 1500) -- main.tf:379-385, D-125/D-115 |-- vr1_dc1_uplink (site-wan NAT, 172.30.3.0/24, MTU 1500) -- main.tf:388-395 | `-- vvr1-dc0 CONTAINMENT VM (416 GiB/108 vCPU, expose_nested_virt=true) -- main.tf:410-522, D-123/D-124 NIC1 enp1s0 = transit (region-facing; SEC-010 --transit-if KEYS HERE) NIC2 enp2s0 = uplink, IP-LESS, enslaved into netplan bridge br-vr1-dc0-wan (declared IN vvr1-dc0's own cloud-init netplan, D-125 bridge-in) -- main.tf:507-522 | `== INNER root: opentofu/vr1-dc0-substrate/ (qemu+ssh to vvr1-dc0 over the transit, R-5, run from Office1 per D-128) == |-- inner_storage pool -- vr1-dc0-substrate/main.tf:26-30 |-- vr1_dc0_planes (dc-planes module, isolated L2, no forward/DHCP, | MTU = var.underlay_mtu = 9000) -- vr1-dc0-substrate/main.tf:32-40 | SIX PLANES (D-052/D-100 template, D-139-amended families): | provider-public 10.12.4.0/22 dual-stack v4 + GUA 2602:f3e2:f02:10::/64 | metal-admin 10.12.8.0/22 dual-stack v4 + GUA 2602:f3e2:f02:20::/64 | metal-internal 10.12.12.0/22 IPv6-ONLY GUA 2602:f3e2:f02:21::/64 (v4 REMOVED, D-139 Ruling A) | data-tenant 10.12.16.0/22 IPv6-ONLY GUA 2602:f3e2:f02:30::/64 (geneve underlay) | storage 10.12.32.0/22 IPv6-ONLY GUA 2602:f3e2:f02:40::/64 (Ceph public) | replication 10.12.36.0/22 IPv6-ONLY GUA 2602:f3e2:f02:50::/64 (Ceph cluster, cross-DC leg) | lb-mgmt (NEW) n/a IPv6-ONLY GUA 2602:f3e2:f02:80::/64 (RESERVED, no charm consumer -- G18) | lib-net.sh mirrors dc0's v4 CIDRs: PLANE_CIDRS (lib-net.sh:22), PLANE_NAME (:23-30) |-- vr1_dc0_wan (wan-bridge module: forward={mode=bridge} onto br-vr1-dc0-wan; | NO mtu block -- provider REJECTS on bridge-mode; inherits host | bridge MTU=1500 from the outer netplan) -- vr1-dc0-substrate/main.tf:51-56 |-- vr1_dc0_opnsense DC edge (LAN=provider-public, WAN=vr1_dc0_wan bridge) | -- vr1-dc0-substrate/main.tf:60-73 `-- vr1_dc0_node x 12 (9 role + juju-01 + maas-01 + tailscale-01) NIC_PLANE_ORDER (lib-hosts.sh:69): metal-admin, provider-public, metal-internal, data-tenant, storage, replication role nodes (9): all 6 planes + OVS br-ex parented on enp2s0 (provider-public leg; BREX_PARENT_NIC, lib-hosts.sh:72) -- enp2s0 itself carries NO L3, br-ex carries the static (Pattern A, D-100/D-060) juju-01 / maas-01 / tailscale-01 (utility VMs): 2 planes only (metal-admin + provider-public, raw NIC WITH gateway, NO br-ex) -- lib-hosts.sh:74-93 ``` **dc1 arm** is structurally identical: `vvr1-dc1` (main.tf:524-618), its own inner root `opentofu/vr1-dc1-substrate/`, planes 10.12.64/68/72/76/80/84.0/22 (lib-net.sh:171-179, D-124 AMENDMENT 2026-07-21), transit 172.31.0.4/30, uplink 172.30.3.0/24. --- ## 2. SEC-010 -- transit FORWARD-drop (forwarding choke point) - **Scope:** interface-scoped FORWARD-drop on the TRANSIT leg only (`enp1s0` on vvr1-dc0), never a global `ip_forward=0` and never globalized on the bridge -- `br_netfilter` makes bridged WAN frames traverse the L3 FORWARD chain, so a global drop would silently kill `br-vr1-dc0-wan` (design-decisions.md D-125, "br_netfilter CONSTRAINT"). - **Artifact:** `scripts/site-headend-install.sh --host-nodes` writes `/etc/nftables-sec010.nft` + boot-persistent `sec010-fw.service`; `--host-nodes --check` is the mechanical gate (fails if the rule is absent OR the keyed transit interface does not exist -- an nftables oifname on an absent iface loads clean but matches nothing, fail-open class). CLOSED 2026-07-20, both ends (security-ledger.md SEC-010 row). - **Why it sits where it does (nesting-specific rationale, verbatim from the ledger):** *"after the D-123 Model B reshape ... vvr1-dc0 now bridges ALL 6 inner planes + the transit, and belongs on the inner libvirt host, not just a 2-leg rack."* SEC-010 is therefore the control that keeps vvr1-dc0's six bridged inner planes from being reachable FROM the region side of the transit -- it is doing the isolation job a separate host would otherwise do "for free." - **The same pin follows onto voffice1's own transit leg** (its `enp2s0`), independently verified (SEC-010 row, security-ledger.md). --- ## 3. MTU / jumbo / geneve-over-v6 budget - `underlay_mtu = 9000` (jumbo) is the ONE knob applied to every plane network AND every mesh-link network: `opentofu/dc-dc-phase0.auto.tfvars:7` -- *"Step 3 ruling: jumbo internal fabric (tenant MTU stays 1500)"*. Consumed by `vr1-dc0-substrate/main.tf:38` (`mtu = var.underlay_mtu`) and `main.tf:127/133/139` (mesh links) and `main.tf:78` (office1_network). - **The WAN path is deliberately NOT jumbo:** `modules/wan-bridge` (bridge-mode, no `mtu` block -- the provider REJECTS it in bridge mode, measured 2026-07-20) inherits the host bridge's 1500 from vvr1-dc0's outer netplan; `modules/site-wan` (office1-wan, the two uplink NATs) defaults to 1500 -- "the ISP-uplink domain; NOT the jumbo planes/mesh" (`vr1-dc0-substrate/main.tf:55`). - **D-101 tenant-MTU sub-policy** (design-decisions.md ~line 2280): geneve-over-v6 overhead is roughly 56 bytes (IPv6 40 + UDP 8 + Geneve base 8) before nested-virt/options. With the jumbo (9000) underlay, tenant MTU stays 1500 and amphora/geneve fit; if the underlay were pinned at 1500 instead, tenant MTU would drop to ~1444 (v6-geneve) and must be set consistently across ovn geneve, tenant-network MTU, and amphora. **The jumbo underlay is what makes tenant MTU 1500 possible at all** -- this is the plane-level fact the container-elim must preserve. - **D-139 (2026-07-31):** data-tenant (the geneve underlay plane) moved from IPv4 to IPv6-only GUA (`2602:f3e2:f02:30::/64` dc0) -- geneve now rides GUA, not the historically planned ULA (Ruling B, "full GUA on every plane"). - **The 2026-08-08/09 geneve-over-v6 root cause (memory + `scripts/geneve-encap-assert.sh` header) was NOT an MTU/byte-budget defect.** Two distinct causes, both at the OVN/OVS layer, neither is the vvr1-dcN containment hop itself: 1. Encap-family SPLIT between the containerized control plane (LXD, v4) and metal compute chassis (v6) -- cross-family tunnels never form. Gate: `geneve-encap-assert.sh` C1. 2. ovn-chassis 24.03 emitted a BRACKETED v6 `ovn-encap-ip` (`[2602:...]`) that OVS's geneve implementation rejects ("bad geneve 'remote_ip'"), leaving v6 tunnels at ofport -1. Gate: `geneve-encap-assert.sh` C2. Fixed live: unbracket + carve v6 on the LXD/containerized chassis + `overlay_ip_version=6`; VM->VM 8/8 0% loss confirmed (memory: dc0-checkpoint-then-reip-redeploy.md). **NAMING TRAP reminder (per this pass's own framing): the "containerized control plane" here is the LXD API-charm containers on nodes, NOT the `vvr1-dcN` containment VM this pass eliminates -- do not credit or blame container-elim for this fix.** - **Net conclusion:** the plane-level MTU/geneve byte budget (9000 underlay / 1500 tenant / ~56-byte v6-geneve overhead) is **unaffected by container-elim** -- the vvr1-dcN hop for plane traffic was already a same-MTU (9000) isolated libvirt bridge with no extra encapsulation, so removing it removes a bridging HOP, not a budget constraint. --- ## 4. What "simplify the wiring" precisely touches if vvr1-dcN is eliminated **COLLAPSES entirely (exists only because of the nesting):** - vvr1-dc0's own two outer NICs (transit `enp1s0`, uplink `enp2s0`) and the containment VM itself -- their sole purpose is to give the inner `qemu+ssh` provider (R-5) something to dial and to give the inner WAN bridge an egress path. - `br-vr1-dc0-wan` (the netplan bridge INSIDE vvr1-dc0) and the `modules/wan-bridge` network realizing D-125's bridge-in -- this exists purely to fix OBS-3 (nested WAN NAT losing egress under Model B). Flat topology reverts to Model A's direct `site-wan` NAT; the repo already names this exact revert: "Fallback: `docs/archive/model-a-fallback-plan.md` section 3 (revert removes the uplink NIC/network/bridge + `wan-bridge` and restores the OPNsense WAN addr)" (D-125 entry). - The two-stage OpenTofu apply ordering (outer boots+sizes vvr1-dc0 -> bootstrap gate installs inner libvirtd -> inner root applies) and the `qemu+ssh` inner-provider URI itself (`vvr1_dc0_transit_ip` dial) -- one fewer apply stage, one fewer credential/URI to carry (SEC-026 already flags the MAAS-admin-key residency problem this ordering created). **RE-HOMES onto vcloud libvirt directly (same module, same shape, different provider target):** - The SIX plane bridges (`modules/dc-planes`) -- same CIDRs, same MTU var (9000), same isolated/no-forward/no-DHCP posture (NetBox/OpenStack still own L3). D-139's family matrix is an OpenStack/NetBox-layer concern, not a libvirt-nesting one -- unaffected. - The 12 node-vm module calls per DC (9 role + juju-01/maas-01/tailscale-01) -- same NIC_PLANE_ORDER, same per-role carve shape (6 planes + br-ex for role nodes; 2 planes, no br-ex, for utility VMs), same MAC-pin-after-first-apply discipline. - The DC OPNsense edge -- LAN stays provider-public; WAN reverts to a direct `site-wan` NAT (Model A shape) instead of routing through a bridge inside a containment VM. **REQUIRES A TARGET-TOPOLOGY DECISION (flagged for W0.4, not resolved here):** - **SEC-010's isolation boundary.** Today SEC-010 protects ONE DC's forwarding surface because vvr1-dc0 and vvr1-dc1 are SEPARATE nested libvirt hosts -- each bridges only its own six planes. If both DCs' planes flatten onto vcloud's SAME libvirtd, dc0's and dc1's plane bridges become co-resident on ONE host for the first time, a cross-DC adjacency that did not exist before and that SEC-010 (interface-scoped to a transit leg that no longer carries this function) does not cover. **This is the single largest wiring risk of elimination** and needs an explicit new host-level control (or a proposed topology that avoids co-residency) before Phase 1. - **The transit leg's (D-124) surviving purpose.** Its `qemu+ssh`-dial role disappears with the inner root, but MAAS PXE/commissioning reachability from Office1 to a DC-local region (D-132) and jujud's MAAS-API dial may still need SOME region<->rack leg depending on whether the target topology keeps "DC-local" as a routing/administrative concept at all. W0.3/W0.4's call. - **The D-100 mesh triangle's fate.** The three mesh-link segments represent inter-SITE links (dark-fiber stand-ins), a concept orthogonal to intra-site containment -- they likely survive container-elim UNLESS the target topology also collapses "DC" as a distinct site into one shared vcloud fabric with no site-to-site fiction needed. Also a W0.4 call. --- ## 5. Files / lines this map is built from (for W0.4 and later phases) - `opentofu/main.tf`: 76-148 (office1 net/edge, mesh triangle), 175-233 (voffice1 3-NIC transit host), 341-397 (netem, uplink NATs), 399-522 (vvr1-dc0 containment VM), 524-618 (vvr1-dc1 mirror). - `opentofu/vr1-dc0-substrate/main.tf`: 1-73 (inner provider, planes, wan-bridge, edge), 75-265 (node fleet + utility VMs, NIC order, carved planes per role). - `opentofu/modules/dc-planes/main.tf`, `opentofu/modules/wan-bridge/main.tf`, `opentofu/modules/mesh-link/main.tf` -- the three bridge-shape module bodies. - `scripts/lib-net.sh`: 22-30 (PLANE_CIDRS/PLANE_NAME dc0), 171-179 (dc1). - `scripts/lib-hosts.sh`: 53-100 (NIC_PLANE_ORDER, BREX_PARENT_NIC, per-role carve shape). - `scripts/geneve-encap-assert.sh`: 1-30 (the C1/C2 gate header, root-cause citation). - `docs/design-decisions.md`: D-100 (~2223), D-101 (~2246-2330), D-124 AMENDMENTs (~5022-5083), D-125 (~5084-5161), D-139 (~7253-7370). - `docs/security-ledger.md`: SEC-010 row (line 21), SEC-026 row (line 79). - `docs/CURRENT-STATE.md`: 7606-7710 (Sec 2.3-iii containment relocation note, Sec 3 authored-not- applied inventory) -- NOTE this section is dated 2026-07-18/20 and predates the later amphora/geneve/D-139 work; used here only for the nesting mechanism, not current apply state. --- ## 6. Top risks for the administrator / W0.4 1. **SEC-010 co-residency gap** (Section 4) -- the single biggest wiring risk; no existing artifact addresses cross-DC plane-bridge adjacency on one flattened host. 2. **D-125's whole bridge-in mechanism is Model-B-only debt** -- eliminating the container layer should DELETE it (revert to Model A direct NAT), not carry it forward unused. 3. **MTU/geneve budget is NOT the source of container-elim benefit** -- do not let Phase 4's change-set imply an MTU fix; the real payoff is removing OBS-3-class nesting fragility and one apply-ordering stage, not a byte-budget change. 4. **Transit leg's surviving purpose is undetermined** pending the target topology (W0.4) -- this doc intentionally stops short of ruling it out.