Pass 0 / W0.1 -- Substrate/tofu map of the container layer

READ-ONLY planning artifact. Container-layer-elimination pass, Phase 0, Worker 1. Repo HEAD at write time: branch dc-dc-stage5-preconditions. All refs are path:line against files as read this session; re-check line numbers if the file has moved since.

1. As-is substrate diagram (text)

vcloud (outer libvirt, qemu:///system)                         <- OUTER ROOT: opentofu/main.tf
|
|-- module.office1_storage / office1_network / office1_opnsense   (Office1 -- NOT container-layer)
|-- module.voffice1                                                (Office1 headend VM -- NOT container-layer)
|-- module.ubuntu_noble_base                                        (shared base image, multi-consumer)
|-- module.mesh_vr1_dc0_vr1_dc1 / _office1 / vr1_dc1_office1        (D-100 mesh triangle -- outer, persists)
|-- module.netem_vr1_dc0_vr1_dc1                                    (tc netem on a mesh bridge -- outer, persists)
|-- module.vr1_dc0_storage / vr1_dc1_storage   (outer per-DC pool -- holds ONLY the containment VM's own disk)
|-- module.vr1_dc0_uplink / vr1_dc1_uplink     (D-125 vcloud-level ISP NAT /24 -- the ONE real NAT egress)
|
|-- module.vvr1_dc0  <==== CONTAINMENT VM (D-122/D-123 Model B) ====
|     cloudinit-vm: 108 vCPU / 480 GiB / ~3000 GiB disk (opentofu/variables.tf:137-156)
|     NIC1 -> mesh_vr1_dc0_office1 (transit, region-facing, SEC-010 keys here)
|     NIC2 -> vr1_dc0_uplink (IP-less bridge port -> br-vr1-dc0-wan)
|     expose_nested_virt = true (LOAD-BEARING: svm passthrough for inner KVM)
|     |
|     |==== BOOTSTRAP GATE (scripts/site-headend-install.sh, node-host mode) ====
|     |     installs: nested libvirtd, inner storage-pool dir + AppArmor grant,
|     |     kvm nested=1, stages OPNsense base image, verifies bridge + SEC-010
|     |     FORWARD-drop scoped to the transit NIC. NOT an OpenTofu artifact --
|     |     a bash script gate between the two tofu roots.
|     |
|     +-- INNER ROOT: opentofu/vr1-dc0-substrate/  (provider = qemu+ssh -> vvr1-dc0, R-5)
|           |-- module.inner_storage          (dc-storage-pool, INSIDE vvr1-dc0)
|           |-- module.vr1_dc0_planes         (dc-planes x6 -- the RELOCATED planes, D-123 Model B)
|           |-- module.vr1_dc0_wan            (wan-bridge -> br-vr1-dc0-wan, D-125 bridge-in)
|           |-- module.vr1_dc0_opnsense       (opnsense-edge, LAN=provider-public, WAN=vr1_dc0_wan)
|           +-- module.vr1_dc0_node[9]        (node-vm x9: 3 control+2 compute+4 storage,
|                                               PLUS juju-01, maas-01, tailscale-01 =
|                                               12 for_each entries total, D-104/D-132/D-129(iii))
|
+-- module.vvr1_dc1  <==== CONTAINMENT VM (mirror of vvr1_dc0) ====
      cloudinit-vm: 108 vCPU / 480 GiB / ~3000 GiB disk (opentofu/variables.tf:175-194)
      NIC1 -> mesh_vr1_dc1_office1, NIC2 -> vr1_dc1_uplink
      |
      |==== BOOTSTRAP GATE (same script, dc1 target) ====
      |
      +-- INNER ROOT: opentofu/vr1-dc1-substrate/  (provider = qemu+ssh -> vvr1-dc1)
            mirrors dc0's inner root exactly: inner_storage, vr1_dc1_planes (x6),
            vr1_dc1_wan, vr1_dc1_opnsense, vr1_dc1_node[N] (schematic MAC scheme
            52:54:01:d1:NN:PP, pre-pinned at authoring -- opentofu/vr1-dc1-substrate/main.tf:84-119)

Target (post-elimination, NOT yet confirmed -- Phase-0 W0.4's job, referenced here only for contrast): vcloud (libvirt) -> node VMs directly, planes/wan/edge modules re-homed onto the outer root or vcloud itself, containment VM + bootstrap gate + inner root deleted.

2. Every file/module involved, with path:line

2.1 Outer root (opentofu/)

• opentofu/main.tf -- the whole outer root. Load-bearing container-layer blocks:

  • module "vvr1_dc0" at main.tf:410-519 -- the containment VM itself (cloudinit-vm).
  • module "vvr1_dc1" at main.tf:537-623 -- mirror for dc1.
  • module "vr1_dc0_uplink" at main.tf:379-384, module "vr1_dc1_uplink" at main.tf:391-396 -- the D-125 vcloud-level ISP NAT that the containment VM's NIC2 bridges into (site-wan module; this is the SOLE real internet egress for each DC).
  • module "vr1_dc0_storage" (main.tf:35-39) / vr1_dc1_storage (main.tf:49-53) -- outer per-DC pool. Post-Model-B this pool holds ONLY the containment VM's own boot disk/seed, NOT node disks (those moved to inner_storage in the inner root).
  • module "mesh_vr1_dc0_vr1_dc1" / _office1 / mesh_vr1_dc1_office1 (main.tf:125-141) -- the D-100 mesh triangle. dc0/dc1<->office1 legs carry the D-124 transit the containment VM's NIC1 rides; NOT container-layer-specific themselves (mesh survives elimination, only what rides it changes -- see W0.2 network map).
  • module "netem_vr1_dc0_vr1_dc1" (main.tf:353-358) -- tc netem on mesh bridge virbr5; outer, independent of the containment layer.
  • Comment block main.tf:22-33 -- explicitly documents the D-123 Model B move: the six vr1-dc0 planes moved to the inner root because "a libvirt provider cannot be configured from a resource created in the same apply."
  • Comment block main.tf:300-331 -- names the apply order and the bootstrap-gate script explicitly (quoted in section 3 below).
  • moved blocks main.tf:275-298 -- state-address rewrites from the pre-D-119/pre- Model-B naming; historical, not currently container-layer-live, but relevant if Phase 1 needs precedent for a future moved{} migration when the layer collapses.

• opentofu/variables.tf -- sizing/addressing vars for the containment VMs:

  • vvr1_dc0_vcpu/_memory_mib/_disk_bytes (variables.tf:137-156, defaults 108 / 491520 MiB / 3,221,225,472,000 bytes) and the dc1 mirror (variables.tf:175-194) -- derived via scripts/dc-dc-whole-host-budget.py (comment cites the derivation, not invented).
  • vr1_dc0_rack_metal_admin_ip / _transit_ip / _transit_prefix / _transit_peer_ip (variables.tf:226-244) and the dc1 mirror (variables.tf:201-219) -- NO defaults, HELD values, populated from opentofu/d124-rack.auto.tfvars (gitignored; confirmed content read this session -- dc0 10.12.8.2 / 172.31.0.2/30 peer 172.31.0.1; dc1 10.12.68.2 / 172.31.0.6/30 peer 172.31.0.5).
  • vr1_dc0_ssh_pubkey_path / vr1_dc1_ssh_pubkey_path (variables.tf:105-114,158-168) -- D-126 per-env dedicated SSH key each containment VM's cloud-init authorizes; the inner root's qemu+ssh provider authenticates with the matching private half.
  • vr1_dc0_planes / vr1_dc1_planes (variables.tf:39-86) -- the six-plane CIDR maps. Declared in the OUTER root's variables.tf but consumed by the INNER root's dc-planes module call, not the outer root itself (outer no longer instantiates dc-planes for either DC) -- the outer var is the values-of-record copy the inner tfvars mirrors (comment variables.tf:70-73).

• opentofu/d124-rack.auto.tfvars -- gitignored; the four rack-addressing values per DC (metal-admin IP, transit IP/prefix/peer) that key the containment VM's netplan AND the inner root's qemu+ssh connection variable (vvr1_dc0_transit_ip in the inner root is a SEPARATE var populated from the SAME measured IP, not auto-derived from this file -- confirmed by reading vr1-dc0-substrate/variables.tf:4-7, which is a plain string var with no default).

2.2 Inner roots

• opentofu/vr1-dc0-substrate/main.tf (266 lines, read in full):
  • provider "libvirt" block (main.tf:12-22) -- the qemu+ssh dial into vvr1-dc0, keyfile+sshauth REQUIRED (measured trap, no default identity works).
  • module "inner_storage" (main.tf:26-30) -- dc-storage-pool INSIDE vvr1-dc0.
  • module "vr1_dc0_planes" (main.tf:34-40) -- the six RELOCATED planes (dc-planes module reused verbatim from ../modules/).
  • module "vr1_dc0_wan" (main.tf:51-56) -- wan-bridge onto br-vr1-dc0-wan.
  • module "vr1_dc0_opnsense" (main.tf:60-73) -- the DC edge, opnsense-edge module.
  • locals.vr1_dc0_node_nics (main.tf:77-85) + locals.vr1_dc0_nodes map (main.tf:96-247) -- 12 entries: 9 D-121 Option-C role nodes (3 control/2 compute/4 storage) + vr1-dc0-juju-01 (D-104 amendment) + vr1-dc0-maas-01 (D-132(iii)) + vr1-dc0-tailscale-01 (D-129(iii)) -- all MAC-pinned from live measurement.
  • module "vr1_dc0_node" (main.tf:250-265) -- for_each over the 12-entry map, instantiating ../modules/node-vm once per node.
• opentofu/vr1-dc0-substrate/variables.tf (48 lines) -- vvr1_dc0_transit_ip / vvr1_dc0_ssh_user / vvr1_dc0_ssh_keyfile (connection vars, MEASURED post-outer- apply, no defaults on the IP/keyfile), inner_pool_path (default /var/lib/libvirt/vr1-dc0-inner), opnsense_base_path (default staged path, must be on the EXECUTING host's filesystem -- measured trap, see variables.tf:26-33), domain_suffix, underlay_mtu, vr1_dc0_planes (mirrors outer var).
• opentofu/vr1-dc0-substrate/versions.tf -- separate terraform{} block, same provider pin (dmacvicar/libvirt 0.9.8) as the outer root's versions.tf.
• opentofu/vr1-dc1-substrate/{main,variables,versions}.tf -- structural MIRROR of the dc0 inner root (confirmed by reading both main.tf files side by side): same module set (inner_storage, vr1_dc1_planes, vr1_dc1_wan, vr1_dc1_opnsense, vr1_dc1_node[for_each]), only the addressing/MAC values differ (dc1 uses a DETERMINISTIC pre-assigned MAC scheme 52:54:01:d1:NN:PP, main.tf:84-98, vs dc0's measured-and-pinned 52:54:00:* values -- a genuine divergence in HOW the MACs were assigned, not a container-layer-elimination-relevant difference).

2.3 Modules touched by the container layer

Reused VERBATIM by both the outer root (for the containment VM itself) and the inner roots (for what runs inside it) -- same module source, different provider/root:

None of modules/{cloudinit-vm,dc-storage-pool,dc-planes,wan-bridge,opnsense-edge, node-vm,site-wan,mesh-link,netem-link,base-image} contain any hardcoded reference to "vvr1-dc0"/"containment" in their own HCL -- they are generic and reused verbatim. The container-layer-SPECIFIC facts live entirely in the two ROOTS (which provider/state each module instance is created under) and in the containment VM's own module call (vvr1_dc0/vvr1_dc1 in the outer root). This is exactly what main.tf:326-330's comment states: "Their HCL is UNCHANGED -- the same node-vm / site-wan / opnsense-edge / dc-planes modules are reused verbatim; only the provider they run against moved from vcloud to vvr1-dc0."

3. Outer<->inner apply ordering (as documented, main.tf:326-330)

Verbatim from the outer root's own comment:

"Apply order: OUTER (this root -- boots + sizes vvr1-dc0) -> BOOTSTRAP GATE (site-headend-install.sh: install nested libvirtd + inner pool + kvm nested=1 + stage the opnsense base image) -> INNER root (planes/wan/edge/nodes)."

Concretely, three distinct steps with two different tools:

1. OUTER apply (opentofu/ root, qemu:///system on vcloud, run per D-128 Plane 1 from vcloud itself, confirmed docs/CURRENT-STATE.md:159,178) -- creates vvr1_dc0/vvr1_dc1 (the containment VM), the D-125 uplink NAT, the mesh legs, the outer per-DC storage pool.
2. BOOTSTRAP GATE (scripts/site-headend-install.sh, "node-host mode" / --role rack) -- a bash script, NOT OpenTofu, run against the now-booted containment VM over SSH. Installs nested libvirtd, creates the inner pool directory + AppArmor grant (site-headend-install.sh:264-272), sets kvm nested=1, stages the OPNsense base image onto the EXECUTING host's filesystem (voffice1, per the inner root's opnsense_base_path var comment), and verifies the transit/uplink bridges + the SEC-010 FORWARD-drop scoped to the transit interface.
3. INNER apply (opentofu/vr1-dc0-substrate/ or vr1-dc1-substrate/, qemu+ssh dialed from the EXECUTING host -- per D-128 the inner tofu roots run from voffice1, confirmed docs/CURRENT-STATE.md:3678 "D-128 has it run the INNER tofu roots and tofu init leaves a provider cache") -- creates the inner storage pool, the six relocated planes, the WAN bridge, the DC edge, and the 12 node VMs per DC.

This ordering is STRUCTURALLY FORCED, not a preference: main.tf:28-29's comment states the reason plainly -- "A libvirt provider cannot be configured from a resource created in the same apply, so the inner substrate is a separate root/state applied AFTER vvr1-dc0 is up (the bootstrap gate)." Eliminating the containment VM removes this structural forcing function entirely -- a flat topology would let ONE root (or a set of modules under one root/state) create the planes/edge/nodes directly on vcloud's own qemu:///system provider, with no bootstrap-gate script and no second tofu init/apply cycle. This is very likely the single largest simplification the elimination buys, structurally speaking (Phase 1/2 should confirm the operational-workflow-step count this removes).

4. Disk/pool/cloud-init/seed-volume dependency chain

Two INDEPENDENT chains today, one per root:

Outer chain (containment VM's own boot disk): module.ubuntu_noble_base (base-image, downloaded once, main.tf:165-173) --backing_store--> module.vvr1_dc0/vvr1_dc1 (cloudinit-vm) which creates, per instance:

• libvirt_volume.disk (COW backing off the shared base image, modules/cloudinit-vm/main.tf:24-47)
• libvirt_cloudinit_disk.seed + libvirt_volume.seed (the NoCloud ISO, modules/cloudinit-vm/main.tf:49-76) -- lifecycle { ignore_changes = [create] } guards against the D-130 staging-path-remint-on-reboot trap.
• both volumes live in module.vr1_dc0_storage/vr1_dc1_storage (the OUTER per-DC pool, dc-storage-pool, main.tf:35-53).

Inner chain (everything the containment VM hosts):

• module.inner_storage (dc-storage-pool, vr1-dc0-substrate/main.tf:26-30) is a FRESH pool created at var.inner_pool_path (default /var/lib/libvirt/vr1-dc0-inner) -- a directory the BOOTSTRAP GATE (not OpenTofu) provisions on the containment VM's own filesystem before this root can apply.
• module.vr1_dc0_node[*] (node-vm) creates a BLANK libvirt_volume.disk per node (PXE-boot pattern, no backing image -- modules/node-vm/main.tf:44-62) plus an opt-in OSD volume for storage-role nodes, all in inner_storage.
• module.vr1_dc0_opnsense (opnsense-edge) creates its disk as a DIRECT COPY (not COW) of var.opnsense_base_path (modules/opnsense-edge/main.tf:55-85) -- a file that must exist on the EXECUTING host's filesystem (the qemu+ssh provider uploads from its own client-side filesystem, NOT from a path on vvr1-dc0 -- measured trap, vr1-dc0-substrate/variables.tf:26-33), also placed in inner_storage.

Key coupling for elimination planning: the inner chain's inner_storage pool is NOT the same pool as the outer vr1_dc0_storage/vr1_dc1_storage pool -- they are two separate dc-storage-pool module instances under two separate roots/providers, with the inner one's target directory living INSIDE the containment VM's own disk image (itself a volume in the outer pool). A flat topology collapses this to ONE pool per DC on vcloud's own filesystem holding node/edge/pool volumes directly -- no nested-disk-inside-a-disk indirection, and no bootstrap-gate-provisioned directory dependency.

5. Bulleted list -- exactly what is containment-layer-specific

Would be REMOVED entirely if the containment VM is eliminated:

• module "vvr1_dc0" / module "vvr1_dc1" (opentofu/main.tf:410-519, 537-623) -- the containment VM domain + its disk + seed volumes.
• The ENTIRE inner roots: opentofu/vr1-dc0-substrate/{main,variables,versions}.tf and opentofu/vr1-dc1-substrate/{main,variables,versions}.tf as separate roots/states (their MODULE CALLS would likely be re-homed into the outer root or a new flat root, not deleted -- see below).
  • module.inner_storage in each inner root -- re-homed (folds into, or replaces, the outer per-DC storage pool).
  • The BOOTSTRAP GATE step of scripts/site-headend-install.sh "node-host mode" -- nested-libvirtd / inner-pool / kvm-nested=1 install -- has NOTHING to install once there is no nested hypervisor.
• The two qemu+ssh provider connections (vr1-dc0-substrate/main.tf:11-22, vr1-dc1-substrate/main.tf:11-19) and their connection variables (vvr1_dc0_transit_ip/_ssh_user/_ssh_keyfile and dc1 mirror) -- a flat topology needs no remote-libvirt dial at all; the outer root's qemu:///system covers everything.
• module "vr1_dc0_wan" / vr1_dc1_wan (wan-bridge module calls) -- these exist ONLY to bridge the inner OPNsense edge onto the containment VM's own br-vr1-dc0-wan netplan bridge; with no containment VM there is no such bridge to attach to. The DC edge's WAN would need to reconnect DIRECTLY to vr1_dc0_uplink/vr1_dc1_uplink (the outer D-125 NAT network) instead -- likely reverting toward the pre-Model-B ("Model A") shape docs/model-a-fallback-plan.md already describes as the fallback.
• The containment VM's own two-NIC netplan (main.tf:495-518, 599-621): the br-vr1-dc0-wan/br-vr1-dc1-wan bridge declarations, the transit-leg static addressing consumed only by the inner qemu+ssh dial, and the D-124 rack-addressing vars that exist ONLY to address the containment VM (vr1_dc0_rack_transit_ip/_prefix/_peer_ip, vr1_dc0_rack_metal_admin_ip, and dc1 mirrors -- variables.tf:196-244).
• vvr1_dc0_vcpu/_memory_mib/_disk_bytes and dc1 mirror (variables.tf:137-156, 175-194) -- sizing vars that exist only because the containment VM must hold the ENTIRE node fleet (104 node vCPU + 4-96 GiB overhead derivation). A flat topology sizes each node VM directly against vcloud's own budget instead (no "containment overhead" line item at all -- scripts/dc-dc-whole-host-budget.py's --containment-overhead-* flags would need re-deriving or dropping).
• vr1_dc0_ssh_pubkey_path/vr1_dc1_ssh_pubkey_path (D-126 per-env keys) and the ~/vr1-dc0-creds/, ~/vr1-dc1-creds/ key material they authorize -- these exist specifically to authenticate the inner qemu+ssh provider to the containment VM; a flat topology has no SSH-to-a-nested-hypervisor step to key.

Would be RE-HOMED (kept, but moved to the outer root / vcloud's own provider), not deleted:

• module "vr1_dc0_planes" / vr1_dc1_planes (dc-planes x6 each) -- the six planes themselves are NOT container-layer artifacts; only WHERE they are created (inside vs. outside the containment VM) is. This is precisely the D-123 Model B move already documented at main.tf:22-33 and CURRENT-STATE.md:7676-7682 (2.3-iii) -- eliminating the container layer is largely UNDOING that specific move: planes go back to being created directly by the outer root (or a flat per-DC root) on vcloud's own libvirt, the way they stood before D-123 Model B (Model A).
• module "vr1_dc0_opnsense" / vr1_dc1_opnsense (opnsense-edge) -- the DC edge VM itself; only its WAN attachment (currently vr1_dc0_wan/wan-bridge) needs to change to attach directly to the outer vr1_dc0_uplink/vr1_dc1_uplink NAT network instead.
• module "vr1_dc0_node" / vr1_dc1_node (for_each node-vm, 12 entries each) -- the node VMs themselves; only their provider (containment-VM qemu+ssh -> vcloud qemu:///system) and their storage-pool parent (inner_storage -> a per-DC pool on vcloud) change.
• module.inner_storage -- effectively MERGES with (or is replaced by) the existing outer vr1_dc0_storage/vr1_dc1_storage pool, since there is no longer an "inner" vs "outer" distinction once everything is on vcloud.

Would be UNCHANGED / NOT container-layer-specific (persist regardless):

• module "voffice1", office1_opnsense, office1_network, office1_storage, ubuntu_noble_base -- Office1 headend, untouched.
• module "mesh_vr1_dc0_vr1_dc1", mesh_vr1_dc0_office1, mesh_vr1_dc1_office1 (mesh-link) and netem_vr1_dc0_vr1_dc1 -- the D-100 mesh triangle. NOTE for W0.2: what RIDES the office1<->dc0/dc1 mesh legs changes (today: the transit to the containment VM's NIC1; post-elimination: whatever replaces the region<->rack path, if anything is still needed once there is no separate rack/region split) -- flagged as a W0.2 item, not resolved here.
• module "vr1_dc0_uplink" / vr1_dc1_uplink (site-wan, the D-125 vcloud-level ISP NAT) -- persists as the DC's egress; only what attaches to it changes (today: the containment VM's bridge port; post-elimination: directly the DC edge's WAN NIC, likely reverting to a site-wan-module-style NAT attachment matching Office1's own pattern).
• modules/maas-vm-host -- already NOT instantiated for either DC (refuted by measurement, unrelated to this pass).

6. Top risks / unknowns for eliminating this layer (structural, from this dimension only)

1. The D-125 bridge-in mechanism (wan-bridge module) has no proven Model-A equivalent in this codebase's CURRENT config -- Model A's original NAT-based WAN shape was RETIRED when Model B replaced it (main.tf:44-47's comment: "the old pre-Model-B module vr1_dc1_planes skeleton... is retired -- do not re-add"). Reverting needs either resurrecting that retired shape or re-deriving it; docs/model-a-fallback- plan.md exists and should be W0.4's/Phase-1's first read, not re-invented.
2. Node VM count triples per DC's libvirt host under a flat topology: today ONE containment-VM domain exists per DC at the outer-root level; post-elimination the outer root (or its replacement) would directly own 12 node-vm domains + 1 edge domain
   • 6 planes PER DC = ~19 domains x 2 DCs = ~38 domains, all on vcloud's single outer libvirt/state. This is a state-management and blast-radius change beyond "flatten the topology" -- Phase 4's module design should weigh one-state-per-DC vs. one-state- total for this reason.
3. MAC-pinning discipline must survive the move. Every node/edge MAC is currently pinned from LIVE MEASUREMENT taken against the containment VM's virsh (virsh domiflist on vvr1-dc0/vvr1-dc1). Re-homing these for_each maps to a new provider will very likely register as a -/+ (force-replace, since name is ForceNew per the existing moved{} block precedent at main.tf:256-271) or at minimum demands re-measuring and re-pinning 12x2=24 nodes' MACs plus 2 edges' MACs after the move -- MAAS re-enlistment risk is real and large, not cosmetic (the exact "MAC regeneration trap" incident class this repo already suffered once, per multiple comments cross-referenced above).
4. Whether the outer root's qemu:///system (vcloud) provider is even the RIGHT single target is a W0.4/operator question, not resolved here -- this doc only maps what exists; it takes no position on the target topology.
5. The bootstrap-gate script (site-headend-install.sh) has a SECOND role (region+ rack MAAS enrollment, SEC-010 nftables) beyond nested-libvirt bring-up -- eliminating the containment layer removes the nested-libvirt PART of its job but the script likely still needs SOME of its other duties (rack MAAS role, SEC-010) reassigned somewhere; W0.3 (consumer inventory) owns confirming exactly which parts survive.
6. Not verified this session (out of W0.1's read-only substrate scope, flagged for W0.2/W0.3): whether underlay_mtu (jumbo, 9000) assumptions inside the currently- nested plane networks change once they run on vcloud's own bridges directly (host NIC MTU vs. a nested-guest's virtio MTU) -- geneve-over-v6 budget is explicitly a W0.2 deliverable per phase-prompts.md; this doc does not check it.