# Pass 0 / W0.4 -- Target-topology options (container-layer elimination) **Worker:** W0.4 (target-topology options). **Feeds:** the Phase-0 OPERATOR GATE (`SCOPE-AND-EXECUTION-PLAN.md` Section 4). **Scope discipline:** READ-ONLY; every claim below cites `path:line` or a named durable record; anything not resolvable from the repo is marked **OWED** rather than invented (hard rule 2). --- ## 0. Framing recap (verified against the repo, not assumed) - **Current nesting (Model B, D-123):** `vcloud (outer libvirt, qemu:///system) -> vvr1-dcN (containment VM, inner libvirtd) -> node VMs`. The outer root creates `vvr1-dc0`/`vvr1-dc1` (`opentofu/main.tf:410-519,537-623`); the INNER root (`opentofu/vr1-dc0-substrate/main.tf:1-266`) creates the 6 planes + WAN bridge + edge + **12** node VMs (9 D-121 role nodes + the D-104 juju-controller `vr1-dc0-juju-01` + the D-132 MAAS-region `vr1-dc0-maas-01` + the D-129(iii) Tailscale router `vr1-dc0-tailscale-01`, `opentofu/vr1-dc0-substrate/main.tf:96-247`) via a `qemu+ssh` provider dialled FROM Office1 (`opentofu/vr1-dc0-substrate/main.tf:12-22`). - **The eliminate-and-pull-up-one target, stated in the operator's own words** (quoted in `SCOPE-AND-EXECUTION-PLAN.md:34-36`): collapse to `vcloud (libvirt) -> node VMs` directly. - **This is not a green-field question.** Before Model B was ruled, the exact flat shape existed, was committed, and validated: `docs/archive/model-a-fallback-plan.md` ("Model A", git tag `model-a-fallback` at `114d392`, R-3-compliant, `tofu validate` 11/11 modules). Both options below are read against that precedent rather than invented from scratch. - **Two rulings POST-DATE Model A and change what a flat topology must additionally provide** -- Model A's own spec did not have to solve them: - **D-138** (`docs/design-decisions.md:7064-7126`, RULED 2026-07-30): the cloud-facing Juju client + `openstack` CLI (phase-03..phase-06) MUST run from a host with L3 reach to the DC's node planes (`voffice1` cannot reach them; SEC-010/D-052/D-125 forbid routing there). CURRENT STATE: that host is `vvr1-dc0` itself -- "the dc0 rack" (`docs/CURRENT-STATE.md:1046,3014-3016`; `lib-hosts.sh:212-213` shows `vvr1-dc0` holds a live metal-admin address, `qemu+ssh://jessea123@10.12.8.2/system`, the D-134 rack-utility `.2` -- distinct from `.5` juju-01, `.6` maas-01, `.7` tailscale-01). - **D-132 amendment** (`docs/design-decisions.md:7135-7204`, RULED 2026-07-30): each DC gets its OWN MAAS region (now a node VM, `vr1-dc0-maas-01`), not a migrated/shared Office1 region. - **SEC-026/SEC-028** (`docs/security-ledger.md:79,81`): "the rack" is now a **credential-bearing host** -- it holds the DC-scoped MAAS admin key (SEC-018) and the per-DC Juju service key (SEC-028), OPEN rotation obligations, and an explicit per-DC isolation requirement ("each DC's client host receives ONLY THAT DC's credential"). - **Consequence for this pass:** eliminating the container layer must still answer "where does the D-138 client + its SEC-026/SEC-028 credential live", because that requirement did not exist when Model A was archived. This is the crux item (d) below. --- ## 1. OPTION 1 (RECOMMENDED) -- "Model A, D-132/D-138-updated": flat nodes + a small per-DC client VM ``` vcloud (host, L0, qemu:///system) |-- vr1-dc0-client small VM (4 vCPU / 8 GiB / 80 GiB, D-138 role only -- see (d)) | legs: metal-admin + office1<->dc0 transit; NO nested libvirt, | NOT a hypervisor for anything -- a flat sibling VM like any other |-- vr1-dc0-control-01..03 node VMs (16/65536/150) \ |-- vr1-dc0-compute-01..02 node VMs (12/49152/100) \ vcloud-level libvirt siblings, |-- vr1-dc0-storage-01..04 node VMs (8/24576/550) > attached DIRECTLY to the 6 |-- vr1-dc0-juju-01 node VM (4/8192/100) / vcloud-level plane networks |-- vr1-dc0-maas-01 node VM (4/8192/150) / |-- vr1-dc0-tailscale-01 node VM (2/2048/25) / |-- vr1-dc0-opnsense edge (2/2048, 2-NIC: provider-public LAN + vr1-dc0-wan WAN) |-- vr1-dc0-{provider-public,metal-admin,metal-internal,data-tenant,storage,replication} | 6 isolated-L2 libvirt networks (dc-planes), AT VCLOUD LEVEL |-- vr1-dc0-wan NAT /24 simulated ISP uplink, AT VCLOUD LEVEL (site-wan, direct) `-- mesh-vr1-dc0-office1 transit leg (office1 <-> dc0), unchanged nesting depth = 2 (vcloud -> node VM -> nova KVM guest) <- VR0-PROVEN, same as Model A site-down = scripted group-destroy of the vr1-dc0-* domain set (Model A's mechanism) ``` ### (a) Where node VMs live Directly on vcloud's outer libvirt provider (`qemu:///system`), as siblings -- identical to `docs/archive/model-a-fallback-plan.md:24-39`'s Model A shape, but with the 3 utility node classes (juju-controller, MAAS region, Tailscale router) added as additional flat siblings using the SAME `modules/node-vm` bodies already authored in `opentofu/vr1-dc0-substrate/main.tf:250-265` (only the provider target changes, per that file's own header note at lines 1-10: "All module bodies are UNCHANGED from the outer root -- only the provider they run against moved"). ### (b) Where the six planes land Back at vcloud level as `modules/dc-planes` outputs, reversing the D-123 move documented at `opentofu/main.tf:22-33` ("the 6 vr1-dc0 planes MOVED to the INNER root ... under Model B"). `lib-net.sh`'s `PLANE_CIDRS`/`PLANE_NAME` (unchanged, `scripts/lib-net.sh:21-30`) and the D-101/ D-134 CIDR values stay IDENTICAL -- only the libvirt network objects' host moves, not their IPAM identity. metal-admin remains the PXE/boot plane, `NIC_PLANE_ORDER` unchanged (`lib-hosts.sh:69`). ### (c) Where transit/uplink/mesh/br-ex land - Mesh legs (office1<->dc0, office1<->dc1, dc0<->dc1): UNCHANGED, already vcloud-level (`opentofu/main.tf:125-141`). - Uplink/WAN: the D-125 bridge-in plumbing (`module "vr1_dc0_uplink"` NAT + `wan-bridge` module + the IP-less 2nd NIC + `br-vr1-dc0-wan` netplan bridge, `opentofu/main.tf:360-397,505-518`) is REMOVED. The DC edge's WAN attaches to a vcloud-level NAT directly, exactly like Office1's edge does today (`opentofu/main.tf:99-117`) -- this is Model A's item 8 (`docs/archive/model-a-fallback-plan.md:67-75`): "same `172.30.2.0/24` ... no uplink NIC, no bridge, no `wan-bridge` module ... NO re-address needed". - br-ex: unchanged in kind -- still an OVS bridge parented on each ROLE node's provider-public NIC (`lib-hosts.sh:70-72`), a per-node fact independent of where the node's hypervisor sits. ### (d) Where the juju/openstack execution host goes (THE CRUX) A **dedicated, small, per-DC client VM** on vcloud libvirt, occupying the ROLE Model A's original rack headend already had by construction: two legs (metal-admin + office1 transit, `docs/archive/model-a-fallback-plan.md:25-26`), 4 vCPU / 8 GiB / 80 GiB. Model A's own headend was authored BEFORE D-138/D-132 existed and was scoped as "MAAS rack headend" only -- this option REPURPOSES that same artifact shape to also satisfy D-138 (cloud-facing client execution) and SEC-026/SEC-028 (the DC-scoped credential residency), because it already has the one property those rulings require: an L3 leg on metal-admin. It is explicitly **not a hypervisor for anything** -- no `expose_nested_virt`, no inner libvirtd, no nodes attached to it -- so it does not reintroduce a container layer; it is a flat utility sibling, same class as `vr1-dc0-juju-01`. Per-DC credential isolation (SEC-026 "(1) ISOLATION IS THE LOAD-BEARING CONTROL") is preserved exactly as it is today, since this VM plays the identical role the current rack plays. maas-vm-host registration (Step 9, deferred DOCFIX-179) targets **vcloud's own virsh**, not this VM's (Model A table row 4, `docs/archive/model-a-fallback-plan.md:48`) -- MAAS discovers the vcloud-level node domains directly. ### (e) Capacity/FIT impact on vcloud Directionally FREES host RAM, does not need it. Today's containment VM is sized to hold BOTH the node fleet AND its own containment overhead (`opentofu/variables.tf:143-150`: RAM raised 416->480 GiB per DC, "384 GiB node fleet + 96 GiB overhead"; the 96 GiB is inner-libvirtd/OS/page- cache overhead that exists ONLY because of the nesting). Flat placement removes that per-DC 96 GiB overhead layer and replaces it with the small client VM's 8 GiB -- a swing of roughly (96-8) x 2 DCs ~ 176 GiB of host RAM, all else equal. vCPU impact is smaller: Model B's overhead is 4 vCPU/DC (`opentofu/variables.tf:137-141`, unchanged since the RAM-only raise), and Model A's headend is also 4 vCPU, so vCPU is roughly a wash. **This is a qualitative, repo-grounded direction, not a verified number** -- see Section 4 OWED items: the sizing calculator (`scripts/dc-dc-whole-host-budget.py`) does not yet have flags for the 3 utility node classes (juju/maas/tailscale) added after its authoring, so an exact FIT verdict needs either extending it or hand-totaling before the operator relies on a number. ### (f) What breaks - The inner OpenTofu root (`opentofu/vr1-dc0-substrate/`, its own state file, its `qemu+ssh` provider dialled from Office1) is retired; its module bodies fold back into the outer root. - The "OUTER -> BOOTSTRAP GATE -> INNER" apply-ordering contract (`opentofu/main.tf:301-331`) goes away -- no bootstrap gate, no cross-root apply sequencing. - `site-headend-install.sh`'s "node-host mode" (nested libvirtd, `kvm nested=1`, inner pool, the SEC-010 nftables writer scoped to the inner host, D-125 `--uplink-if`/`--wan-bridge` verification) is retired; only whatever the small client VM still needs (metal-admin address assignment, the D-138 client role) remains, in a much smaller mode. - **SEC-010's CLOSED implementation must be REBUILT, not moved.** Today's mechanism (`docs/security-ledger.md:21`) is `nftables-sec010.nft` + `sec010-fw.service` written onto `vvr1-dc0`'s MEASURED interface names (`enp1s0` transit / `br-vr1-dc0-wan`/`enp2s0` uplink, captured live 2026-07-20). The new client VM is a DIFFERENT artifact with its own NIC-naming trap (`opentofu/main.tf:473-481` documents this exact trap for `vvr1-dc0`'s q35 shape) -- its transit-scoped FORWARD-drop has to be re-authored and re-measured, not copy-pasted. - `VIRSH_POWER_ADDRESS_FROM_OFFICE1`/`_FROM_DCREGION` (`lib-hosts.sh:212-213,246-250`) currently point at the rack's transit/metal-admin address for `qemu+ssh` NODE POWER CONTROL through the inner libvirt. With nodes flat on vcloud, power control becomes vcloud's own local virsh (`qemu:///system` or a local-equivalent address) -- these constants and every consumer that reads them (`maas-node-power.sh`, the teardown runbook's virsh probes) need retargeting. - **The D-123 Model B ruling's headline benefit is lost:** "site-down = a single `virsh destroy vvr1-dc0`" (`opentofu/main.tf:405`) reverts to Model A's scripted group-destroy of the `vr1-dc0-*` domain set (`docs/archive/model-a-fallback-plan.md:36,64`). This is a REAL regression against what D-123 was ruled for, not a free simplification -- flagged here so Phase 4's [ARCH] decision framing carries it honestly, per `SCOPE-AND-EXECUTION-PLAN.md:197-200`. - MAC-pin discipline for the 12 node VMs (`opentofu/vr1-dc0-substrate/main.tf:87-245`) needs a fresh capture pass once nodes are re-created flat -- moot for the 10.13 redeploy itself since the whole fleet is being torn down and rebuilt regardless. ### (g) What simplifies - Nesting depth 4 -> 2 (VR0-proven), eliminating the entire "unproven depth-4 nested-virt" risk class that Model A's own fallback-trigger list names (`docs/archive/model-a-fallback-plan.md:97, 100`: "nova-compute guests fail to boot or are unusably slow at 3x-nested KVM"; "expose_nested_ virt=true ... destabilises the headend"). - No inner-root/outer-root apply-ordering gate, no separate inner state file, no cross-host `qemu+ssh`-from-Office1 provider dial for the substrate apply. - No D-125 bridge-in plumbing at all (`wan-bridge` module, the IP-less uplink NIC, the `br-vr1-dc0-wan` bridge, the "unprovable pre-apply" deploy-time gate at `opentofu/main.tf:375-378`) -- the DC edge's WAN becomes an ordinary direct NAT, one fewer bespoke module and one fewer "unprovable until you try it" gate. - Fewer NIC-naming traps to carry (today TWO artifacts -- `vvr1-dc0` and `voffice1` -- each independently measured `enp1s0`/`enp2s0` on first boot; flat placement needs this measured once, for the single small client VM, not for a containment VM AND its inner substrate). - `lib-net.sh` plane CIDRs/names are untouched (still 6 planes/DC, same D-101/D-134 values) -- only the low-level libvirt/carve/power scripts (`lib-hosts.sh`'s power-address constants, `maas-node-power.sh`, `dc-node-carve`) need retargeting, not the IPAM layer. ### (h) Roosevelt-delta Strongly POSITIVE. Bare metal has no hypervisor containment layer at all -- Roosevelt's node "VMs" are literal servers on the DC's physical planes, so flat placement is the closer bare-metal analog and nesting was always a simulation-only artifact (Model A's fallback doc never claimed Model B carried Roosevelt fidelity; its one stated advantage was the single-object site-down convenience, which is a simulation-only DR primitive, not a transferable property). The retained small client VM in (d) DOES transfer: D-138 names its own Roosevelt analog explicitly -- "each DC has a management entry point inside it; the NOC reaches that entry point ... the per-DC management bastion holds only its own DC's cloud credential" (`docs/design-decisions.md:7121- 7124`). So this option's client VM is not throwaway scaffolding; it is that bastion, rehearsed early, minimizing delta-to-Roosevelt exactly as `SCOPE-AND-EXECUTION-PLAN.md:203-204` asks. --- ## 2. OPTION 2 (NOT RECOMMENDED, given for contrast) -- fully flat: client runs on vcloud itself ``` vcloud (host, L0) <-- juju/openstack CLI + the D-138/SEC-026/SEC-028 credential live HERE, | on the jumphost OS itself, via a new host-side NIC/veth onto the | (now vcloud-level) metal-admin plane bridge |-- vr1-dc0-control-01..03 / compute-01..02 / storage-01..04 / juju-01 / maas-01 / tailscale-01 | (same flat node-VM siblings as Option 1 -- (a)/(b)/(c) IDENTICAL to Option 1) `-- ... (planes/uplink/mesh identical to Option 1; NO small client VM) ``` Everything in (a)/(b)/(c)/(f)/(g) is IDENTICAL to Option 1 -- the only difference is (d): no per-DC client VM at all; the D-138 execution host collapses one layer further, onto vcloud's own OS. ### (d) Execution host -- the crux, and why this is worse The juju/openstack CLI, and the SEC-026/SEC-028 per-DC credentials, would live directly on the shared jumphost that ALSO runs Plane-1 substrate tooling for BOTH DCs and Office1 (`docs/design-decisions.md:5354-5358`, D-128 Plane 1). This inverts SEC-026's own stated control: "(1) ISOLATION IS THE LOAD-BEARING CONTROL: each DC's client host receives ONLY THAT DC's credential" (`docs/security-ledger.md:79`) -- on vcloud, BOTH DCs' credentials would sit on the one host that already has the widest blast radius in the whole project (it is the CLAUDE.md "live operations clone", governed by the PreToolUse guard and every permission `ask` rule). A credential-scoping boundary that today survives a "rebuild the rack VM" event would no longer exist to survive anything short of rebuilding vcloud itself. ### (e) Capacity/FIT Same node-fleet math as Option 1 (nodes are identical either way); this option additionally saves the ~8 GiB the small client VM would have cost. Negligible vs. Option 1's ~176 GiB swing. ### (h) Roosevelt-delta -- negative No bare-metal analog exists for "run the DC client from the shared jumphost" -- Roosevelt's own management model is per-DC bastions (D-138's own analog, quoted above), which is what Option 1's small client VM already rehearses. Option 2 would need to be UNDONE and re-built as something like Option 1's shape at the pre-Roosevelt bare-metal test, making it throwaway work rather than transferable. --- ## 3. Tradeoff table | Dimension | Option 1 (small per-DC client VM) | Option 2 (client on vcloud) | |---|---|---| | Nesting depth | 2 (same as Option 1's node placement) | 2 | | D-138 execution host | dedicated, isolated, per-DC | shared jumphost, cross-DC | | SEC-026/SEC-028 isolation | preserved (matches today's model) | WEAKENED (both DCs' admin-scoped MAAS keys land on one shared host) | | Host RAM freed vs. today | ~176 GiB (2 DCs, qualitative -- see OWED) | ~184 GiB (marginally more) | | Roosevelt-delta | POSITIVE -- the client VM IS the D-138-named per-DC bastion | NEGATIVE -- no bare-metal analog, throwaway | | Git/spec precedent | Model A archive (`docs/archive/model-a-fallback-plan.md`), already validated 11/11 | none -- net-new design | | D-123 site-down benefit | lost (reverts to Model A's group-destroy) | lost, identically | | New engineering needed | SEC-010 re-author for the new small VM; power-address retarget; MAC re-capture | same, PLUS a new host-side bridge/veth into a MAAS plane on vcloud itself (unprecedented surface) | --- ## 4. Recommendation **Option 1** (flat nodes + a small, single-purpose, per-DC client VM). Rationale, in order of weight: 1. It is the lower-risk delta -- Model A is a previously-validated, git-tagged artifact, not a new design; only the D-138/D-132 layer on top of it is new work, and that work is small (repurpose an existing headend shape, not invent a host class). 2. It preserves the credential-isolation property SEC-026/SEC-028 were opened to protect, where Option 2 actively erodes it. 3. It has a NAMED Roosevelt analog in D-138's own text, so it is not scaffolding to be re-done at the bare-metal test -- it satisfies `SCOPE-AND-EXECUTION-PLAN.md`'s minimize-delta-to- Roosevelt framing directly. 4. The RAM difference between the two options (~176 vs ~184 GiB freed) is marginal next to the isolation and Roosevelt-fidelity costs of Option 2. **Framing note for the operator gate:** "eliminate the container layer" is fully satisfied by Option 1 under the reading that matters -- no VM in this topology is a HYPERVISOR for another VM (no nested libvirt, no `expose_nested_virt`, no inner OpenTofu root). The retained small VM is a flat utility sibling, the same class of thing as the Juju-controller or MAAS-region node VMs already in the fleet, not a resurrection of the containment pattern. If the operator's intent is literally zero additional VMs of any kind, that is Option 2, with the tradeoffs above accepted knowingly. --- ## 5. OWED live measurements (not resolvable from the repo; do not infer) 1. **Current vcloud host capacity.** `scripts/dc-dc-whole-host-budget.py:66-67` carries a committed default it labels "MEASURED host budget" (256 vCPU / 1024 GiB / 10240 GiB), but its CURRENCY for the 10.13 redeploy (same hardware? any change since it was set?) needs a fresh read-only measurement on vcloud before any FIT verdict is finalized. 2. **An exact FIT number for Option 1 vs. today's Model B sizing.** The calculator's `--control`/ `--compute`/`--storage` flags do not yet cover the 3 utility node classes added after it was authored (`vr1-dc0-juju-01`, `vr1-dc0-maas-01`, `vr1-dc0-tailscale-01` -- `opentofu/vr1-dc0-substrate/main.tf:133-245`). Extending the script (or hand-totaling) and re-running `--model A` vs. `--model B` with the CURRENT 12-VM/DC roster is owed before citing a precise freed-capacity number in the Phase-4 change-set. 3. **MTU/jumbo budget for vcloud-level plane bridges** once the 6 planes move up a layer -- this is W0.2's (network/wiring map) dimension, not measured here; flag as a dependency for the Phase-0 administrator synthesis.