# Pass 0 / W0.2 -- Network & wiring map of the container layer

**Agent:** W0.2 (sonnet worker), container-elim pass, Phase 0. Read-only. All values below are
cited to path:line or to a ruled D-NNN/SEC-NNN entry; nothing is inferred. Dated 2026-08-09.

---

## 1. As-is topology -- text diagram (dc0 arm; dc1 is the structural mirror)

```
vcloud (outer libvirt, qemu:///system) -- opentofu/main.tf
|
|-- office1_network (module, MTU=underlay_mtu=9000)  -- main.tf:76-80
|     `-- voffice1 (LXD/MAAS-region/NetBox host)      -- main.tf:175 (NIC1 enp1s0, DHCP/Kea)
|-- office1-wan (site-wan NAT, MTU 1500, not jumbo)
|-- office1_opnsense edge                              -- main.tf:98-114
|
|-- D-100 dark-fiber mesh TRIANGLE (mesh-link module, MTU=underlay_mtu=9000) -- main.tf:125-141
|     |-- mesh_vr1_dc0_vr1_dc1  (dc0<->dc1)   bridge virbr5 (MEASURED, `virsh net-info`)
|     |     netem: delay 3ms jitter 1ms loss 0.01% (PLACEHOLDER, D-100 gap #11 unruled)
|     |                                        -- main.tf:352-357 netem_vr1_dc0_vr1_dc1
|     |-- mesh_vr1_dc0_office1  (dc0<->office1, region<->rack TRANSIT)
|     |     region end: voffice1 NIC2 static 172.31.0.1/30      -- main.tf:191-192
|     |     rack   end: vvr1-dc0 NIC1 (enp1s0) static 172.31.0.2/30 -- main.tf:498-505, D-124
|     `-- mesh_vr1_dc1_office1  (dc1<->office1 transit)
|           region end: voffice1 NIC3 static 172.31.0.5/30       -- main.tf:193
|           rack   end: vvr1-dc1 NIC1 static 172.31.0.6/30       -- main.tf:602-609, D-124 amdt
|     (transit supernet 172.31.0.0/24, D-124 AMENDMENT 2026-07-16, design-decisions.md:5031)
|
|-- vr1_dc0_uplink (site-wan NAT, 172.30.2.0/24, MTU 1500)  -- main.tf:379-385, D-125/D-115
|-- vr1_dc1_uplink (site-wan NAT, 172.30.3.0/24, MTU 1500)  -- main.tf:388-395
|
`-- vvr1-dc0 CONTAINMENT VM (416 GiB/108 vCPU, expose_nested_virt=true) -- main.tf:410-522, D-123/D-124
      NIC1 enp1s0 = transit (region-facing; SEC-010 --transit-if KEYS HERE)
      NIC2 enp2s0 = uplink, IP-LESS, enslaved into netplan bridge br-vr1-dc0-wan (declared
                    IN vvr1-dc0's own cloud-init netplan, D-125 bridge-in)     -- main.tf:507-522
      |
      `== INNER root: opentofu/vr1-dc0-substrate/ (qemu+ssh to vvr1-dc0 over the transit,
          R-5, run from Office1 per D-128) ==
           |-- inner_storage pool                                  -- vr1-dc0-substrate/main.tf:26-30
           |-- vr1_dc0_planes (dc-planes module, isolated L2, no forward/DHCP,
           |     MTU = var.underlay_mtu = 9000)                    -- vr1-dc0-substrate/main.tf:32-40
           |     SIX PLANES (D-052/D-100 template, D-139-amended families):
           |       provider-public   10.12.4.0/22   dual-stack v4 + GUA 2602:f3e2:f02:10::/64
           |       metal-admin       10.12.8.0/22    dual-stack v4 + GUA 2602:f3e2:f02:20::/64
           |       metal-internal    10.12.12.0/22   IPv6-ONLY GUA 2602:f3e2:f02:21::/64 (v4 REMOVED, D-139 Ruling A)
           |       data-tenant       10.12.16.0/22   IPv6-ONLY GUA 2602:f3e2:f02:30::/64 (geneve underlay)
           |       storage           10.12.32.0/22   IPv6-ONLY GUA 2602:f3e2:f02:40::/64 (Ceph public)
           |       replication       10.12.36.0/22   IPv6-ONLY GUA 2602:f3e2:f02:50::/64 (Ceph cluster, cross-DC leg)
           |       lb-mgmt (NEW)     n/a             IPv6-ONLY GUA 2602:f3e2:f02:80::/64 (RESERVED, no charm consumer -- G18)
           |     lib-net.sh mirrors dc0's v4 CIDRs: PLANE_CIDRS (lib-net.sh:22), PLANE_NAME (:23-30)
           |-- vr1_dc0_wan (wan-bridge module: forward={mode=bridge} onto br-vr1-dc0-wan;
           |     NO mtu block -- provider REJECTS <mtu> on bridge-mode; inherits host
           |     bridge MTU=1500 from the outer netplan)            -- vr1-dc0-substrate/main.tf:51-56
           |-- vr1_dc0_opnsense DC edge (LAN=provider-public, WAN=vr1_dc0_wan bridge)
           |                                                       -- vr1-dc0-substrate/main.tf:60-73
           `-- vr1_dc0_node x 12 (9 role + juju-01 + maas-01 + tailscale-01)
                 NIC_PLANE_ORDER (lib-hosts.sh:69): metal-admin, provider-public,
                   metal-internal, data-tenant, storage, replication
                 role nodes (9): all 6 planes + OVS br-ex parented on enp2s0
                   (provider-public leg; BREX_PARENT_NIC, lib-hosts.sh:72) -- enp2s0
                   itself carries NO L3, br-ex carries the static (Pattern A, D-100/D-060)
                 juju-01 / maas-01 / tailscale-01 (utility VMs): 2 planes only
                   (metal-admin + provider-public, raw NIC WITH gateway, NO br-ex)
                   -- lib-hosts.sh:74-93
```

**dc1 arm** is structurally identical: `vvr1-dc1` (main.tf:524-618), its own inner root
`opentofu/vr1-dc1-substrate/`, planes 10.12.64/68/72/76/80/84.0/22 (lib-net.sh:171-179,
D-124 AMENDMENT 2026-07-21), transit 172.31.0.4/30, uplink 172.30.3.0/24.

---

## 2. SEC-010 -- transit FORWARD-drop (forwarding choke point)

- **Scope:** interface-scoped FORWARD-drop on the TRANSIT leg only (`enp1s0` on vvr1-dc0),
  never a global `ip_forward=0` and never globalized on the bridge -- `br_netfilter` makes
  bridged WAN frames traverse the L3 FORWARD chain, so a global drop would silently kill
  `br-vr1-dc0-wan` (design-decisions.md D-125, "br_netfilter CONSTRAINT").
- **Artifact:** `scripts/site-headend-install.sh --host-nodes` writes
  `/etc/nftables-sec010.nft` + boot-persistent `sec010-fw.service`; `--host-nodes --check`
  is the mechanical gate (fails if the rule is absent OR the keyed transit interface does
  not exist -- an nftables oifname on an absent iface loads clean but matches nothing,
  fail-open class). CLOSED 2026-07-20, both ends (security-ledger.md SEC-010 row).
- **Why it sits where it does (nesting-specific rationale, verbatim from the ledger):**
  *"after the D-123 Model B reshape ... vvr1-dc0 now bridges ALL 6 inner planes + the
  transit, and belongs on the inner libvirt host, not just a 2-leg rack."* SEC-010 is
  therefore the control that keeps vvr1-dc0's six bridged inner planes from being reachable
  FROM the region side of the transit -- it is doing the isolation job a separate host would
  otherwise do "for free."
- **The same pin follows onto voffice1's own transit leg** (its `enp2s0`), independently
  verified (SEC-010 row, security-ledger.md).

---

## 3. MTU / jumbo / geneve-over-v6 budget

- `underlay_mtu = 9000` (jumbo) is the ONE knob applied to every plane network AND every
  mesh-link network: `opentofu/dc-dc-phase0.auto.tfvars:7` -- *"Step 3 ruling: jumbo internal
  fabric (tenant MTU stays 1500)"*. Consumed by `vr1-dc0-substrate/main.tf:38` (`mtu =
  var.underlay_mtu`) and `main.tf:127/133/139` (mesh links) and `main.tf:78`
  (office1_network).
- **The WAN path is deliberately NOT jumbo:** `modules/wan-bridge` (bridge-mode, no `mtu`
  block -- the provider REJECTS it in bridge mode, measured 2026-07-20) inherits the host
  bridge's 1500 from vvr1-dc0's outer netplan; `modules/site-wan` (office1-wan, the two
  uplink NATs) defaults to 1500 -- "the ISP-uplink domain; NOT the jumbo planes/mesh"
  (`vr1-dc0-substrate/main.tf:55`).
- **D-101 tenant-MTU sub-policy** (design-decisions.md ~line 2280): geneve-over-v6 overhead
  is roughly 56 bytes (IPv6 40 + UDP 8 + Geneve base 8) before nested-virt/options. With the
  jumbo (9000) underlay, tenant MTU stays 1500 and amphora/geneve fit; if the underlay were
  pinned at 1500 instead, tenant MTU would drop to ~1444 (v6-geneve) and must be set
  consistently across ovn geneve, tenant-network MTU, and amphora. **The jumbo underlay is
  what makes tenant MTU 1500 possible at all** -- this is the plane-level fact the
  container-elim must preserve.
- **D-139 (2026-07-31):** data-tenant (the geneve underlay plane) moved from IPv4 to
  IPv6-only GUA (`2602:f3e2:f02:30::/64` dc0) -- geneve now rides GUA, not the historically
  planned ULA (Ruling B, "full GUA on every plane").
- **The 2026-08-08/09 geneve-over-v6 root cause (memory + `scripts/geneve-encap-assert.sh`
  header) was NOT an MTU/byte-budget defect.** Two distinct causes, both at the OVN/OVS
  layer, neither is the vvr1-dcN containment hop itself:
  1. Encap-family SPLIT between the containerized control plane (LXD, v4) and metal compute
     chassis (v6) -- cross-family tunnels never form. Gate: `geneve-encap-assert.sh` C1.
  2. ovn-chassis 24.03 emitted a BRACKETED v6 `ovn-encap-ip` (`[2602:...]`) that OVS's
     geneve implementation rejects ("bad geneve 'remote_ip'"), leaving v6 tunnels at
     ofport -1. Gate: `geneve-encap-assert.sh` C2.
  Fixed live: unbracket + carve v6 on the LXD/containerized chassis + `overlay_ip_version=6`;
  VM->VM 8/8 0% loss confirmed (memory: dc0-checkpoint-then-reip-redeploy.md). **NAMING TRAP
  reminder (per this pass's own framing): the "containerized control plane" here is the LXD
  API-charm containers on nodes, NOT the `vvr1-dcN` containment VM this pass eliminates --
  do not credit or blame container-elim for this fix.**
- **Net conclusion:** the plane-level MTU/geneve byte budget (9000 underlay / 1500 tenant /
  ~56-byte v6-geneve overhead) is **unaffected by container-elim** -- the vvr1-dcN hop for
  plane traffic was already a same-MTU (9000) isolated libvirt bridge with no extra
  encapsulation, so removing it removes a bridging HOP, not a budget constraint.

---

## 4. What "simplify the wiring" precisely touches if vvr1-dcN is eliminated

**COLLAPSES entirely (exists only because of the nesting):**
- vvr1-dc0's own two outer NICs (transit `enp1s0`, uplink `enp2s0`) and the containment VM
  itself -- their sole purpose is to give the inner `qemu+ssh` provider (R-5) something to
  dial and to give the inner WAN bridge an egress path.
- `br-vr1-dc0-wan` (the netplan bridge INSIDE vvr1-dc0) and the `modules/wan-bridge`
  network realizing D-125's bridge-in -- this exists purely to fix OBS-3 (nested WAN NAT
  losing egress under Model B). Flat topology reverts to Model A's direct `site-wan` NAT;
  the repo already names this exact revert: "Fallback: `docs/archive/model-a-fallback-plan.md`
  section 3 (revert removes the uplink NIC/network/bridge + `wan-bridge` and restores the
  OPNsense WAN addr)" (D-125 entry).
- The two-stage OpenTofu apply ordering (outer boots+sizes vvr1-dc0 -> bootstrap gate
  installs inner libvirtd -> inner root applies) and the `qemu+ssh` inner-provider URI
  itself (`vvr1_dc0_transit_ip` dial) -- one fewer apply stage, one fewer credential/URI to
  carry (SEC-026 already flags the MAAS-admin-key residency problem this ordering created).

**RE-HOMES onto vcloud libvirt directly (same module, same shape, different provider target):**
- The SIX plane bridges (`modules/dc-planes`) -- same CIDRs, same MTU var (9000), same
  isolated/no-forward/no-DHCP posture (NetBox/OpenStack still own L3). D-139's family matrix
  is an OpenStack/NetBox-layer concern, not a libvirt-nesting one -- unaffected.
- The 12 node-vm module calls per DC (9 role + juju-01/maas-01/tailscale-01) -- same
  NIC_PLANE_ORDER, same per-role carve shape (6 planes + br-ex for role nodes; 2 planes, no
  br-ex, for utility VMs), same MAC-pin-after-first-apply discipline.
- The DC OPNsense edge -- LAN stays provider-public; WAN reverts to a direct `site-wan` NAT
  (Model A shape) instead of routing through a bridge inside a containment VM.

**REQUIRES A TARGET-TOPOLOGY DECISION (flagged for W0.4, not resolved here):**
- **SEC-010's isolation boundary.** Today SEC-010 protects ONE DC's forwarding surface
  because vvr1-dc0 and vvr1-dc1 are SEPARATE nested libvirt hosts -- each bridges only its
  own six planes. If both DCs' planes flatten onto vcloud's SAME libvirtd, dc0's and dc1's
  plane bridges become co-resident on ONE host for the first time, a cross-DC adjacency
  that did not exist before and that SEC-010 (interface-scoped to a transit leg that no
  longer carries this function) does not cover. **This is the single largest wiring risk of
  elimination** and needs an explicit new host-level control (or a proposed topology that
  avoids co-residency) before Phase 1.
- **The transit leg's (D-124) surviving purpose.** Its `qemu+ssh`-dial role disappears with
  the inner root, but MAAS PXE/commissioning reachability from Office1 to a DC-local
  region (D-132) and jujud's MAAS-API dial may still need SOME region<->rack leg depending
  on whether the target topology keeps "DC-local" as a routing/administrative concept at
  all. W0.3/W0.4's call.
- **The D-100 mesh triangle's fate.** The three mesh-link segments represent inter-SITE
  links (dark-fiber stand-ins), a concept orthogonal to intra-site containment -- they likely
  survive container-elim UNLESS the target topology also collapses "DC" as a distinct site
  into one shared vcloud fabric with no site-to-site fiction needed. Also a W0.4 call.

---

## 5. Files / lines this map is built from (for W0.4 and later phases)

- `opentofu/main.tf`: 76-148 (office1 net/edge, mesh triangle), 175-233 (voffice1 3-NIC
  transit host), 341-397 (netem, uplink NATs), 399-522 (vvr1-dc0 containment VM), 524-618
  (vvr1-dc1 mirror).
- `opentofu/vr1-dc0-substrate/main.tf`: 1-73 (inner provider, planes, wan-bridge, edge),
  75-265 (node fleet + utility VMs, NIC order, carved planes per role).
- `opentofu/modules/dc-planes/main.tf`, `opentofu/modules/wan-bridge/main.tf`,
  `opentofu/modules/mesh-link/main.tf` -- the three bridge-shape module bodies.
- `scripts/lib-net.sh`: 22-30 (PLANE_CIDRS/PLANE_NAME dc0), 171-179 (dc1).
- `scripts/lib-hosts.sh`: 53-100 (NIC_PLANE_ORDER, BREX_PARENT_NIC, per-role carve shape).
- `scripts/geneve-encap-assert.sh`: 1-30 (the C1/C2 gate header, root-cause citation).
- `docs/design-decisions.md`: D-100 (~2223), D-101 (~2246-2330), D-124 AMENDMENTs (~5022-5083),
  D-125 (~5084-5161), D-139 (~7253-7370).
- `docs/security-ledger.md`: SEC-010 row (line 21), SEC-026 row (line 79).
- `docs/CURRENT-STATE.md`: 7606-7710 (Sec 2.3-iii containment relocation note, Sec 3 authored-not-
  applied inventory) -- NOTE this section is dated 2026-07-18/20 and predates the later
  amphora/geneve/D-139 work; used here only for the nesting mechanism, not current apply state.

---

## 6. Top risks for the administrator / W0.4

1. **SEC-010 co-residency gap** (Section 4) -- the single biggest wiring risk; no existing
   artifact addresses cross-DC plane-bridge adjacency on one flattened host.
2. **D-125's whole bridge-in mechanism is Model-B-only debt** -- eliminating the container
   layer should DELETE it (revert to Model A direct NAT), not carry it forward unused.
3. **MTU/geneve budget is NOT the source of container-elim benefit** -- do not let Phase 4's
   change-set imply an MTU fix; the real payoff is removing OBS-3-class nesting fragility
   and one apply-ordering stage, not a byte-budget change.
4. **Transit leg's surviving purpose is undetermined** pending the target topology (W0.4) --
   this doc intentionally stops short of ruling it out.
