# Pass 4 / W4.2 -- THE LAYERED MODULE-WORKFLOW DESIGN (container-layer elimination)

**Worker:** W4.2 (Phase 4, container-layer-elimination pass, `SCOPE-AND-EXECUTION-PLAN.md`
Section 4 -- "the pass's headline deliverable"). **Date:** 2026-08-09. **Scope:** READ-ONLY
planning. Produces the concrete design of the layered module system the redeploy becomes;
performs no mutation.

**Baseline consumed in full, this session:** `SCOPE-AND-EXECUTION-PLAN.md`,
`pass0-admin-report.md` (Option 1 CONFIRMED, Section 7a), `pass1-admin-report.md` +
`pass1-w4-module-planning.md` (the L0-L5 layer model, adopted here as the backbone),
`pass2-admin-report.md` + `pass2-w1-tofu-modules.md` + `pass2-w4-module-decomposition.md`
(the module change-set, the `dc-site` IaC module, the IaC<->procedure boundary, the 13 owed
artifacts), `pass3-admin-report.md` (the unified tests change-set, the gate-home map, the
per-module harness contract). Every module/artifact below is grounded in one of these
documents or the live repo; anything not yet built is marked **[NEW]** and cites which pass
scoped it. No inferred values -- unresolved design points (root naming, the (a)/(iii)
mechanisms, rack-controller placement) are carried as OPEN, matching their source reports.

---

## 0. What this document is (and is not)

This is the CONCRETE module design that `pass1-w4-module-planning.md`'s L0-L5 model
scaffolds and `pass2`/`pass3`'s change-sets populate. It answers three questions:

1. **Per layer, what are the modules, and what is each one's contract?** (Section 2)
2. **How does a full per-DC redeploy compose these modules into one ordered
   invocation, parameterized by site token?** (Section 3)
3. **What design principles make this repeatable and Roosevelt-transferable, and
   which layers/modules actually carry over to the pre-Roosevelt bare-metal test
   unchanged?** (Sections 4-5)

It does NOT re-rule anything already settled (Option 1, handling (a), MAAS-region
placement) or re-open anything explicitly OPEN in the source reports (root naming, the
(a)/(iii) mechanisms, rack-controller-remainder placement, root topology's final
ratification). Where a module's shape depends on an OPEN item, this document states the
dependency and defers -- it does not invent a value (hard rule 2).

---

## 1. The layer model, restated as the design's backbone

Adopted verbatim from `pass1-w4-module-planning.md` Section 2, consolidated at
`pass1-admin-report.md` Section 5:

```
L0  Host & inter-site substrate      (IaC)         -- Stage 1
L1  Site/edge nodes                  (IaC)         -- Stage 2 (Office1) / Stage 3 (DC, flattened in)
L2  DC substrate: planes + node VMs  (IaC)         -- Stage 3
L3  Enlist/commission                (procedure)   -- Stage 4
L4  Juju/OpenStack deploy            (procedure)   -- Stage 5 (+6 DR, +7 Designate/COS/Magnum, additive)
L5  Verify/gate                      (cross-cutting procedure, re-invoked at every boundary)
```

**Layer-boundary rule** (the design's one hard invariant, `pass1-w4-module-planning.md`
Section 2, restated by `pass2-w4-module-decomposition.md` Section 4 as "identity, not
orchestration"): each layer's input is the layer directly below's OUTPUT only. OpenTofu
(L0-L2) owns up to *"a booted libvirt domain exists, with its network identity (MAC per
NIC, and any statically-assigned IP) correctly wired to the right plane bridges."* The
procedure layer (L3+) begins at the first live dial into that object and **never reads
OpenTofu state** -- it re-derives everything from live, independently observable identity
(MAC, hostname, a fresh API/SSH probe), per `lib-hosts.sh:6-11`'s own stated design rule.

**The container layer was the one place this rule was violated**, and it is the reason
this pass exists: the inner root (`opentofu/vr1-dc0-substrate/main.tf`) dialed OUT to a
`qemu+ssh` provider INTO the outer root's own `vvr1_dc0` output -- IaC reaching into IaC
across a live-dial boundary that only a procedure module should cross
(`opentofu/main.tf:28-29`; `pass2-w4-module-decomposition.md` Section 4). Option 1 removes
this violation structurally: with one flat per-DC root, L2 is IaC end to end, and the
FIRST live dial into anything L2 produced is L3's MAAS commissioning -- exactly where the
boundary rule says it should be. Every module table below is built so this stays true.

---

## 2. Per-layer module design

Legend: **kind** = IaC module / procedure module / library / gate. **[EXISTS]** = re-homed
or unchanged artifact, cited to its current path. **[NEW]** = owed artifact, cited to the
pass that scoped it (`pass1` #N / `pass2` #N referring to the numbered owed-artifact lists
in `pass1-admin-report.md` Section 6 / `pass2-admin-report.md` Section 5).

### L0 -- Host & inter-site substrate (IaC, Stage 1)

**Contract:** given vcloud's `qemu:///system` connection and the site tokens
(`office1`/`vr1-dc0`/`vr1-dc1`), produce the cross-site fabric every other layer plugs
into: the mesh triangle, per-site storage pools, the Office1 L2 network, and the
inter-DC netem link. No DC-specific node/plane content lives here.

| Module | Kind | Inputs | Outputs | Current/planned artifact | Harness |
|---|---|---|---|---|---|
| Mesh triangle | IaC | 3 site pairs, MTU 9000 | 3 mesh-link bridges (dc0<->dc1, dc0<->office1, dc1<->office1) | `modules/mesh-link`, outer `main.tf:125-141` **[EXISTS, unchanged]** -- confirmed non-consumer of the container layer (`pass2-w1-tofu-modules.md` row `mesh-link`) | `tests/opentofu-validate/` (module-standalone validate) |
| Netem link | IaC | dc0<->dc1 mesh bridge (`virbr5`) | tc-shaped DR-drill link | `modules/netem-link`, `main.tf:353-358` **[EXISTS, unchanged]** -- targets the bridge directly, no dependency on either DC's internal shape | `tests/opentofu-validate/` |
| Per-site storage pools | IaC | host path per site | `office1_storage` pool (DC pools re-home to L2, Section below) | `modules/dc-storage-pool`, `office1_storage` call **[EXISTS, unchanged]** | `tests/opentofu-validate/` |
| Office1 L2 network | IaC | -- | `office1-network` | `modules/office1-network`, `main.tf:76-80` **[EXISTS, unchanged, D-114 territory, out of scope]** | `tests/opentofu-validate/` |
| Shared base image | IaC | source image | `ubuntu_noble_base`, consumed by every `cloudinit-vm` instance | `modules/base-image`, `main.tf:165-173` **[EXISTS, unchanged, gains client-VM consumers]** | `tests/opentofu-validate/` |
| Cross-DC host isolation control **[NEW]** | Gate (procedure, L5 kind, L0-scoped invocation) | live bridge enumeration on vcloud, both DCs' plane tags | PASS/REFUSE; nftables rules asserting no inter-plane/inter-DC forwarding | `pass1` #2 / `pass2` Sec 2.1 -- concern (i); spec: Section 6 below | **[NEW]** `tests/<name>/run-tests.sh`, offline fixture model (`pass3` C2) |

**Composes onto:** nothing below it -- L0 is the floor. **Stage owner:** Stage 1
(`docs/dc-dc-deployment-workflow.md`, gate content unchanged per `pass1-admin-report.md`
Section 2.2). The (a) control's recommended home is also Stage 1 -- host-scoped, not
per-DC-apply-scoped (`pass1-admin-report.md` Section 3).

### L1 -- Site/edge nodes (IaC, Stage 2 Office1 / Stage 3 DC-flattened)

**Contract:** given an L0 network + pool, produce a booted, network-identified
non-hypervisor VM: Office1's headend, each DC's edge, and (Option 1's new instance)
each DC's client VM. All three are the SAME module type -- `cloudinit-vm` -- differing
only by inputs.

| Module | Kind | Inputs | Outputs | Current/planned artifact | Harness |
|---|---|---|---|---|---|
| Office1 headend (`voffice1`) | IaC | L0 office1-network + pool | booted VM, MAAS-composed LXD host | `modules/cloudinit-vm`, `main.tf:175` **[EXISTS, unchanged -- D-114, out of scope]** | `tests/opentofu-validate/`, `tests/node-vm/` (adjacent) |
| DC edge (OPNsense) | IaC | L0 pool; L2 planes (WAN leg); `uplink_network_name` string (an L0 `site-wan` output) | booted edge VM, WAN attached directly to the per-DC NAT (no `wan-bridge`) | `modules/opnsense-edge`, re-homed into `dc-site` (Section below) **[EXISTS body, re-homed]** -- input `wan_network_name` rewires from `module.vr1_dc0_wan.network_name` (dead) to `module.vr1_dc0_uplink.network_name` (`pass2-w1-tofu-modules.md` row `opnsense-edge`) | `tests/opentofu-validate/` |
| **Client VM** (`vr1-dcN-client`) **[NEW instance, existing module]** | IaC | L0 pool + mesh-triangle transit leg; NetBox-assigned `metal_admin_ip`/`transit_ip`; `expose_nested_virt=false`; ~4 vCPU/8192 MiB/80 GiB | booted, non-hypervisor VM carrying the D-138 client role + SEC-028/029 credential residencies + (open) rack/D-131/mirror placement | `modules/cloudinit-vm`, NEW call inside `dc-site` (`pass0-admin-report.md` Option 1; `pass1-w4-module-planning.md` Section 3; `pass2-w1-tofu-modules.md` row `cloudinit-vm`) -- **duty roster SHRUNK to client role + credentials + transit leg** once rack-controller retirement is adopted (`pass2-admin-report.md` Section 4.2, 4.5) | `tests/opentofu-validate/` T14/T15 (autostart -- **new case owed, D-127 value not yet ruled**) |
| Site-WAN NAT (L0/L1 boundary) | IaC | -- | per-DC egress NAT, now the edge's DIRECT upstream | `modules/site-wan`, `main.tf:379-396` **[EXISTS, unchanged, gains a direct consumer]** | `tests/opentofu-validate/` |
| SEC-010 transit-leg successor **[NEW]** | Procedure, installed on L1 objects | the client VM's + voffice1's re-measured transit interface names | nftables FORWARD-drop on both transit endpoints | `pass1` #3 / `pass2` Sec 2.2 -- concern (ii); ONE role-agnostic installer subcommand, extracted from `site-headend-install.sh`'s `node_host_setup()` (`:273-320`), replacing today's hand-mirrored voffice1 install | `pass3` A3 -- extends `tests/site-headend-install/` (migrates every proven SEC-010 assertion) |

**Composes onto:** L0's network + pool outputs only. **Stage owner:** Stage 2 (Office1,
untouched) / Stage 3 (DC edge + client VM, now co-located with L2's per-DC apply --
Section 2.4 of `pass2-w1-tofu-modules.md`: teardown symmetry + SEC-026 state isolation +
D-138 fidelity all argue for grouping the client VM's APPLY with its DC's flat root even
though its module TYPE is L1).

### L2 -- DC substrate: planes + node VMs (IaC, Stage 3)

**Contract:** given the L0 pool + L1 client VM's transit identity, and a site token +
D-121/R-3 node roster + D-134 octet map, produce the DC's six plane networks and its 12
node-VM libvirt domains (9 role nodes + 3 utility nodes), each with a MAC-pinned network
identity MAAS can discover. **This is the layer the container layer's boundary
violation lived in and Option 1 collapses it to a single root/state.**

| Module | Kind | Inputs | Outputs | Current/planned artifact | Harness |
|---|---|---|---|---|---|
| `dc-planes` | IaC | `planes` map (name->CIDR), `mtu`, `domain_suffix` | 6 isolated plane bridges, MTU 9000 (D-101/D-139 own the CIDR values) | `modules/dc-planes` **[EXISTS body, re-homed]** from inner root (qemu+ssh) to the per-DC-flat root (`pass2-w1-tofu-modules.md` row `dc-planes`) | `tests/opentofu-validate/` |
| `dc-storage-pool` (per-DC) | IaC | host path | one per-DC pool (collapses the outer "containment-VM-disk" pool + the inner "node-disk" pool into ONE) | `modules/dc-storage-pool` **[EXISTS body, COLLAPSES 2-per-DC -> 1-per-DC]** (`pass2-w1-tofu-modules.md` row `dc-storage-pool`) | `tests/opentofu-validate/` |
| `node-vm` (x12/DC) | IaC | `nodes` map (vcpu/mem/disk/osd_gib?/macs), attaches to `dc-planes` outputs | 12 MAC-pinned libvirt domains: 9 D-121 role nodes + `vr1-dcN-juju-01` (.5) + `vr1-dcN-maas-01` (.6) + `vr1-dcN-tailscale-01` (.7) | `modules/node-vm` **[EXISTS body, re-homed]**, `for_each` map moves verbatim from the inner root (`pass2-w1-tofu-modules.md` row `node-vm`) | `tests/node-vm/` T1-T15 (T8-T11 re-point the hardcoded `INNER=` path; T1-T7/T12-T15 STAY) |
| **`modules/dc-site`** **[NEW, highest-leverage new artifact]** | IaC (composing module) | `site_token`, `domain_suffix`, `underlay_mtu`, `pool_path`, `planes`, `opnsense_base_path`, `uplink_network_name` (string), `nodes` (12-entry roster), `client_vm` (object: sizing, macs, `transit_network_name`, `metal_admin_ip`/`transit_ip`) | plane `network_names`, node `domain_names`/`ids`, client-VM `domain_id`, edge `domain_id` | Replaces the ~230-266-line copy-pasted per-DC inner-root bodies with ONE module both per-DC-flat roots call once; composes `dc-storage-pool` -> `dc-planes` -> `opnsense-edge` -> `node-vm` `for_each` -> `cloudinit-vm` (client), in that order (`pass2-w1-tofu-modules.md` Section 3.1) | **[NEW]** `tests/dc-site/run-tests.sh`, static-fixture `.tf`-tree model (`pass3` C1: 5-plane tree FAILS count; 11/13-node roster FAILS; 2 client-VM calls FAILS exactly-one; `mtu=1500` FAILS; hardcoded DC literal FAILS site-token check) |
| Per-DC-flat root (`vr1-dcN-flat/`, name OPEN) | IaC (root, not a module) | one `provider "libvirt" { }` block pointed at vcloud's own `qemu:///system` (no keyfile/sshauth/qemu+ssh trap class); one `module "site" { source = "../modules/dc-site" }` call; one tfvars file | the DC's entire flat substrate, one apply cycle, one state file | **[NEW root]**, direct successor to today's inner root (`pass2-w1-tofu-modules.md` Section 3.2) -- **root topology (B) shared-outer + per-DC-flat RECOMMENDED, Phase-4 ratifies; naming OPEN (avoid unqualified `-substrate` if reserved for Roosevelt)** | `tests/opentofu-validate/` (extend P8-equivalent to a DECLARED root list, not a glob -- `pass3-admin-report.md` Section 3 "H6") |
| `wan-bridge` | IaC | -- | -- | `modules/wan-bridge` **[COLLAPSES/DELETED]** -- D-125 bridge-in dead; edge WAN -> direct NAT (`pass2-w1-tofu-modules.md` row `wan-bridge`) | retired with its parent |

**Composes onto:** L0's pool (via `pool_path`) and L1's client-VM transit identity (via
`client_vm.transit_network_name`, a cross-root STRING reference -- the same pattern
`office1_opnsense`'s `wan_network_name` literal already uses, `main.tf:112`). **Does NOT
reach into L3** -- `dc-site`'s outputs are network_names/domain_ids/MACs only; it never
calls MAAS or juju. **Stage owner:** Stage 3, the HEAVIEST rewrite
(`pass1-admin-report.md` Section 2.2): Step B (bootstrap gate) eliminated wholesale, Step
C (inner apply) merges into Step A -- one apply, one root-scope, one host, one state.

### L3 -- Enlist/commission (procedure, Stage 4)

**Contract:** given L2's MAC-pinned node VMs and a reachable MAAS region (L1's
`vr1-dcN-maas-01`, D-132 addendum, unchanged by flattening), produce READY, carved,
tagged MAAS machines and the per-DC artifact-mirror/DNS-forwarder services those
machines need during commissioning. **The handoff artifact from L2 is a MAC address,
observed independently on both sides -- no tofu state crosses this boundary**
(`pass2-w4-module-decomposition.md` Section 4).

| Module | Kind | Inputs | Outputs | Current/planned artifact | Harness |
|---|---|---|---|---|---|
| `dc-node-carve.sh` | Procedure | `<site>`, live MAAS API | v4 NIC/br-ex carve | **[EXISTS, no code change]** -- grep-verified zero containment hits, pure MAAS-API (`pass2-admin-report.md` Sec 3.3) | `tests/dc-node-carve/` |
| `dc-node-v6-carve.py` | Procedure | `<site>`, live MAAS API, MAAS tag `openstack-<site>` | v6 static assignment | **[EXISTS, no code change]** | `tests/dc-node-v6-carve/` |
| `maas-node-power.sh` | Procedure | power address as `$1` (topology-agnostic) | `power_type=virsh` set, MAC-matched | **[EXISTS, no code change to the script body]** -- every invocation-site/runbook LITERAL updates, **BLOCKED on the power-key mitigation (#11) mechanism** (Section 6) | `pass3` A4/A5 -- **BLOCKED on #11**, edited together, same session |
| `maas-role-tags.sh` | Procedure | `<site>` | per-role MAAS tags the bundle constrains on | **[EXISTS, no code change]** | `tests/maas-role-tags/` |
| `maas-region-power-key.sh` | Procedure | region host | installs/verifies the per-DC MAAS->libvirt power key (SEC-012 dc0 / SEC-016 dc1) | **[EXISTS body unchanged]** -- the KEY it installs IS the power-key blast-radius object; URI/key shape re-derives per the #11 mitigation | `pass3` A5 -- **BLOCKED on #11** |
| `dc-region-topology.sh` | Procedure (+ gate mode) | `<site>` | region fabric/space/subnet/tag topology | **[EXISTS, no change]** | `tests/dc-region-topology/` |
| `dc-plane-ipam.sh` | Procedure (+ gate mode) | `<site>` | plane IPAM state, D-134's executable gate | **[EXISTS, no change]** | `tests/dc-plane-ipam/` |
| `dc-mirror.sh` / `dc-cache-proxy.sh` / `dc-snap-proxy.sh` | Procedure | target host, `<site>` | per-DC apt/UCA mirror or cache, snap proxy | **[EXISTS, host RE-TARGET only]** -- neither Option-1 VM fits dc0's mirror as authored (maas-01: 150 GiB earmarked, no spare; client VM: ~80 GiB) -- **sizing decision rides the FIT-calculator extension (#7)** | rides `tests/dc-mirror/` etc., extended sizing case |
| `site-headend-install.sh --role rack` remainder | Procedure | target host | MAAS `--role rack` enrollment (contingent) | **[EXISTS, CONTINGENT RETIRE]** -- `--host-nodes` (~134 lines, the bootstrap-gate duty) DELETES wholesale; `--role rack` retires IF rack-controller retirement is ratified (both maas-01 VMs already run `region+rack`; measured, `pass2-admin-report.md` Sec 1.2/Sec 4.2) | `pass3` B2 (retire), B3 (contingent) |
| `dc-rack-net.sh` (D-131 forwarder half) | Procedure | rack host | node-DNS forwarder | **[EXISTS, LEGS half retires structurally (no flat VM is a libvirt host with own bridges); forwarder half CONTINGENT on D-131 retire-with-evidence]** | `pass3` B4 (retire), B5 (contingent) |
| Cross-DC isolation control's `--check` re-verify | Gate | live bridge state, post-apply | PASS/REFUSE | Re-invocation of the L0-homed (a) control at each per-DC apply's close (`pass1-admin-report.md` Section 3) | shares C2's harness |
| MAAS machine-record release/delete + rack-controller decommission **[NEW, teardown-adjacent]** | Procedure | MAAS API, region-side residue | zero-record confirmation | `pass1` #5, AMENDED by `pass2` Sec 4.2(i) to include `maas rack-controller delete` + `primary_rack`/DHCP-reference cleanup | **[NEW]** `pass3` C7, fakebin `maas` model |
| D-131 retirement-evidence step **[NEW, one-time]** | Procedure (evidence capture, not a standing gate) | fresh region's own BIND | dig-proven resolver identity | `pass2` #13 | **[NEW]** `pass3` C5, offline dig-capture fixtures; **dc1's current config is a standing-RED case until retirement is live** |
| Power-key blast-radius mitigation **[NEW]** | Procedure (credential-scope control) | vcloud's libvirtd connection scoping mechanism (candidate: restricted SSH key / per-DC virsh wrapper / polkit ACL) | a power key that can control ONLY its own DC's domain set | `pass2` Sec 2.3, concern (iii) -- **BLOCKS `maas-node-power.sh`/`maas-region-power-key.sh` literal updates and the `lib-hosts.sh` power-address re-derivation** | **[NEW]** `pass3` C4, offline rendered-ACL/`authorized_keys`/polkit parsing model; itself the precondition for A4/A5/A9 |

**Composes onto:** L2's MAC-pinned domains + L1's reachable MAAS region only; never
dials L0 or reads tofu state. **Stage owner:** Stage 4 (`dc-dc-phase3-maas-enlist-
deploy.md`, LOW DELTA -- only the two SSH-jump-target literals change, per
`pass1-admin-report.md` Section 2.1).

### L4 -- Juju/OpenStack deploy (procedure, Stage 5 + additive Stage 6/7)

**Contract:** given L3's READY machines, a site token, `bundle.yaml` + per-DC overlays,
and a Vault root, stand up an independently-running OpenStack cloud per DC. D-140 PINS
this as a procedure layer for THIS redeploy -- settled, not folded into IaC
(`pass1-admin-report.md` Section 5).

| Module | Kind | Inputs | Outputs | Current/planned artifact | Harness |
|---|---|---|---|---|---|
| Juju bootstrap + `bundle.yaml` deploy | Procedure | L3's READY machines, `$DC` | running controller + bundle | `runbooks/dc-dc-phase4-juju-bundle-per-dc.md` **[EXISTS, MODERATE change]** -- RUN-LOCATION table row 1 (juju/openstack CLI) re-points `rack` -> `vr1-dcN-client`; row 3 ("never vcloud") doctrine intact | -- (runbook-level, gated by `preflight.sh` below) |
| `render-dc-overlays.py` | Procedure | `<site>` | deterministic per-DC bundle-overlay files | **[EXISTS, no change -- derive/render split already site-parameterized]** | `tests/render-dc-overlays/` |
| VR0 template chain (`phase-00`..`phase-07-*.sh`) | Procedure | `<site>` | admin creds, network stand-up, Octavia amphora pipeline, Magnum/CAPI stack, Ceph rbd-mirror/radosgw-multisite | **[EXISTS, no change]** -- run twice, once per DC, per `pass1-w4-module-planning.md` Section 1.2 "the VR0 template it runs twice" | Each ships its own `tests/phase-0N-*/` dir (`pass2-w4-module-decomposition.md` Section 2) |
| Preflight gate | Gate | live cloud state | PASS/FAIL, P1-P10 | `scripts/preflight.sh` **[EXISTS, EXTENDED -- new P10 for concern (ii); P4 gains an #11-dependent content case]** | `tests/preflight/` |
| Credential minting (SEC-026/028/029) | Procedure (data, not a module body) | the client VM's identity | freshly-minted per-DC credentials | Register rows re-point (`vm-secret-locations`); no material migrates -- freshly minted on the new host (`pass1-admin-report.md` Section 1 check 7) | `pass3` C6 (R7 revocation checklist, rides teardown) |

**Composes onto:** L3's READY-machine handoff only. **Stage owner:** Stage 5 (primary),
Stage 6 (DR drill, additive per `vr0-to-vr1-is-additive`), Stage 7 (Designate/COS/Magnum,
additive) -- Stages 6-7 have NO container-layer dependency, verified by W1.2
(`pass1-admin-report.md` Section 2.2).

### L5 -- Verify/gate (cross-cutting procedure, invoked at every layer boundary)

**Contract:** at every layer's declared-done state, produce a failable PASS/FAIL (never
existence-only, GA-R6) plus, at milestones, a committed BOM. L5 is not owned by one
stage; it re-runs at each stage's close.

| Gate | Scope | Home / re-invocation points | Current/planned artifact | Harness |
|---|---|---|---|---|
| `opentofu-validate.sh` | L0-L2, every module standalone + every root | Delivery-time + Stage 1-3 close | **[EXISTS, extended]** -- root enumeration moves to a DECLARED list (not a glob, `pass3` "H6"); T13 (containment autostart) RETIRES with a negative-assertion case | `tests/opentofu-validate/` |
| `preflight.sh` | L3->L4 boundary, P1-P10 | Before every L4 deploy step | **[EXISTS, extended]** -- new P10 (concern ii), P4 content case (concern iii, `#11`-blocked) | `tests/preflight/` |
| `cloud-assert.sh` (+ `--capture`) | L4, behavioral | Post-deploy/restart/pre-change/post-incident (periodic) | **[EXISTS, extended]** -- new A11a (concern i re-verify) and A11b (concern iii standing re-verify) | `tests/cloud-assert/` |
| `geneve-encap-assert.sh` | L2/L4, OVN geneve family/tunnel health | Post-build on the vcloud-level planes (live assert OWED -- `pass2` #10) | **[EXISTS, unchanged body, NEW invocation point only]** -- MTU/geneve budget analytically unaffected by flattening (`pass0-admin-report.md` Section 1.4) | `tests/geneve-encap-assert/` |
| `dc-egress-check.sh` | L3, DC-egress probe | Invocation-host literal re-points to the ruled B.5 host | **[EXISTS, host literal only]** | `tests/dc-egress-check/` |
| `dc-node-v6-verify.sh` | L2/L3, v6 statics + forwarding | unchanged | **[EXISTS, no change]** | `tests/dc-node-v6-verify/` |
| Budget calculators (`dc-dc-mtu-geneve-budget.sh`, `dc-dc-ceph-disk-budget.sh`, `dc-dc-whole-host-budget.py`) | L0/L2 | Delivery-time, before any FIT verdict enters the change-set | **[EXISTS, the whole-host one EXTENDED]** -- `pass2` #7: add the 3 utility-node classes + artifact-service disk-sizing branch; the Model-A/B comparison cases must not remain the SOLE coverage against a retired topology (`pass3-admin-report.md` "H4") | `tests/dc-dc-whole-host-budget/` |
| `maas-profile-assert.sh` | L3 | Region-profile resolution proof | **[EXISTS, fixture change]** -- drop the two `vvr1-dcN` rows from the simulated roster (rack-retirement-contingent) | `tests/maas-profile-assert/` |
| Cross-DC host isolation control **[NEW, concern i]** | L0-scoped, cross-DC | Stage 1 install; re-verify at each per-DC apply's close; re-verify at Stage-5 live verify (A11a) | `pass1` #2 -- Section 6 below | **[NEW]** `pass3` C2 |
| SEC-010 transit-drop successor **[NEW, concern ii]** | L1-scoped, both transit endpoints | preflight P10 | `pass1` #3 (amended) -- Section 2 L1 table | extends `tests/site-headend-install/` |
| Power-key mitigation gate **[NEW, concern iii]** | L3/L4-scoped, credential blast radius | preflight P4 dependency; cloud-assert A11b standing | `pass2` #11 | **[NEW]** `pass3` C4 |

**Composes onto:** any layer's declared-done state; L5 gates never mutate by default
(gate = "its sole job is verify," `pass2-w4-module-decomposition.md` Section 2 legend).

---

## 3. THE COMPOSITION MECHANISM -- one ordered invocation, parameterized by site token

A full per-DC redeploy is the following ordered sequence. `$SITE` is the site token
(`vr1-dc0` / `vr1-dc1`, D-119); steps marked **(once)** run before the per-DC loop and are
NOT re-run per DC; steps marked **(per $SITE)** run once for each DC, in either order
(root shape (B) makes them structurally independent -- Section 3.1 below). Axis tags
`[D-143]`/`[CE]`/`[both]` carried from `pass1-admin-report.md` Section 4 so the two
riding changes (the 10.13 re-IP and the container-elim) stay distinguishable in this
same sequence.

```
STAGE 1  (once, L0 + L5)
  1.1 [unchanged]  tofu apply: outer/shared root -- mesh triangle, per-site pools,
                    office1-network, netem link, base-image                         (L0)
  1.2 [CE, NEW]     install + --check the cross-DC host isolation control            (L5)
                     -- MUST precede 1.4/2.x for EITHER DC (fork-robust invariant,
                        `pass1-admin-report.md` Section 3)

STAGE 2  (once, L1)
  2.1 [unchanged]  Office1 headend (voffice1) standup -- D-114, untouched            (L1)

  --- per-$SITE loop begins; (B) root shape makes each iteration state-isolated ---

STAGE 3  (per $SITE, L1 + L2)
  3.1 [both]        tofu apply: vr1-$SITE-flat root -> module "site" = dc-site(
                       site_token=$SITE, planes=..., nodes=..., client_vm=...)
                     composes, IN ORDER: dc-storage-pool -> dc-planes -> opnsense-edge
                     -> node-vm[12] -> cloudinit-vm(client)                     (L1+L2)
  3.2 [CE]           re-verify the (a) control's --check now that this DC's planes
                     exist (post-apply close)                                        (L5)
  3.3 [NEW]          MAC re-measurement pass (owed #8) -- confirm every domain's MAC
                     before Stage 4 trusts one                                       (L2/L3 handoff)

STAGE 3.5 (per $SITE, L1-hosted procedure installs)
  3.5.1 [CE]         SEC-010 successor installer, BOTH ends (client VM + voffice1)   (L1)
  3.5.2 [CE, cont.]  site-headend-install.sh --role rack  (IF rack-retirement is
                       NOT ratified; else SKIPPED)                                   (L1/L3)
  3.5.3 [both]       maas-region-power-key.sh -- key shape per the #11 mitigation    (L3)

STAGE 4  (per $SITE, L3)
  4.1 [D-143]        dc-region-topology.sh --commit                                  (L3)
  4.2 [D-143]        dc-plane-ipam.sh --commit                                       (L3)
  4.3 [unchanged*]   dc-node-carve.sh / dc-node-v6-carve.py -- per-machine
                     power_type=virsh; *only the power-ADDRESS value re-derives      (L3)
  4.4 [both]         maas-node-power.sh -- value BLOCKED on #11                      (L3)
  4.5 [unchanged]    maas-role-tags.sh                                               (L3)
  4.6 [CE, cont.]    dc-rack-net.sh (D-131 forwarder) -- IF retire-with-evidence
                       is REJECTED for this DC; else the D-131 evidence step (4.7)   (L3)
  4.7 [CE, cont.]    D-131 retirement-evidence step -- IF retirement adopted         (L3)
  4.8 [CE]           dc-mirror.sh / dc-cache-proxy.sh / dc-snap-proxy.sh, sized
                     per the FIT-calculator extension (#7)                           (L3)

STAGE 5  (per $SITE, L4 + L5)
  5.1 [gate]         preflight.sh (P1-P10)                                           (L5)
  5.2 [unchanged]    juju bootstrap + bundle.yaml + overlays, FROM vr1-$SITE-client   (L4)
  5.3 [both]         SEC-026/028/029 credentials freshly minted on the client VM      (L4)
  5.4 [gate]         cloud-assert.sh (incl. A11a/A11b) + geneve-encap-assert.sh       (L5)

  --- per-$SITE loop ends ---

STAGE 6  (once, cross-DC, L4, additive)
  6.1 [unchanged]    DR/failover drill: netem-link mechanism + Ceph rbd-mirror /
                     radosgw-multisite (D-108)                                       (L4)

STAGE 7  (once per $SITE, L4, additive)
  7.1 [unchanged]    Designate / COS / Magnum (D-106/D-105, `vr0-to-vr1-is-additive`) (L4)

CLOSE-OUT (per $SITE, L5 + record-keeping)
  C.1 [CE]           NetBox DCIM: decommission vvr1-$SITE, register client VM+roster (record)
  C.2 [D-143/CE]     the container-elim [ARCH] ruling itself is NOT a gate -- it is a
                     precondition the operator must have already ruled before Stage 3
                     runs for the FIRST time (Section 7)
```

### 3.1 Why the loop is safe to run per-DC in either order (root shape (B))

Under the recommended root shape -- shared-outer + per-DC-flat roots
(`pass2-w1-tofu-modules.md` Section 2, RECOMMENDED, Phase-4 ratifies) -- DC1's Stage-3
apply cannot create or touch any DC0 resource: it is not in DC0's state file, full stop.
This makes the composition mechanism's ordering invariant reduce to a single, simple
contract: **"the (a) control's `--check` must pass before the first Stage-3 apply of
EITHER per-DC-flat root"** (Step 1.2), rather than depending on `-target` discipline
being followed correctly on every apply (the risk a merged single root would carry).
This is why Stage 1's step 1.2 is drawn OUTSIDE the per-`$SITE` loop and BEFORE it in the
sequence above -- it is a precondition for the loop, not a loop step.

### 3.2 Teardown primitive's place in the same composition

The teardown/site-down lever is the composition mechanism's INVERSE, not a separate
design: under (B), site-down for `$SITE` is `cd opentofu/vr1-$SITE-flat/ && tofu destroy`
(gated) or a scripted `virsh destroy` loop over that root's own state-listed domains
(emergency) -- touching only that DC's root, never the shared-outer root or the other
DC's root (`pass2-w1-tofu-modules.md` Section 2.3). This is owed artifact `pass1` #1 (+
#6 the emergency lever, riding the same fixture library per `pass3-admin-report.md`
Section 2 rides list) and sits at the **L2/L5 boundary**: it is a procedure module that
WRAPS an IaC destroy, verified by an L5-style completeness check (a dc1 domain leaking
into a dc0-targeted set FAILS; an empty resolved set REFUSES rather than reporting
"nothing to do" success -- `pass3-admin-report.md` C3).

---

## 4. Design principles that make this repeatable + Roosevelt-transferable

Adopted from `pass1-w4-module-planning.md` Section 4, sharpened by `pass2`/`pass3`'s
findings. Each principle is grounded in an EXISTING repo pattern, not invented for this
pass (`pass1-w4-module-planning.md` Section 1's framing: "the container-elim does not
need to invent module mechanics, only re-home").

1. **Site-token parameterization, never hardcoded identity.** The `$DC`/`$SITE`
   selector (D-119, DOCFIX-151, `lib_net_select_dc`/`lib_hosts_select_dc`) already does
   this for every procedure module; the IaC layer already does it structurally (no
   module BODY names a DC). Every new artifact this pass introduces takes the same
   token as an input rather than being written DC0-specific and copy-pasted for DC1:
   `dc-site`'s `site_token` input (Section 2, L2), the (a) control's per-bridge-tag
   enumeration (Section 2, L0/L5), the power-key mitigation's per-DC domain-set scoping
   (Section 2, L3). This is the exact anti-pattern
   `docs/dc-dc-deployment-workflow.md`'s gap-register item 1 was created to close.

2. **Every module ships its tested harness, no exception.** `tests/<name>/run-tests.sh`
   for procedure modules, `opentofu-validate.sh`'s standalone-module coverage for IaC
   (CLAUDE.md "Delivery"; `pass2-w4-module-decomposition.md` Section 1.3). Of the 34
   procedure modules surveyed at Phase 2, 32 already comply; the two pre-existing gaps
   (`maas-fabric-prune.sh` / `maas_fabric_classify.py`) are NOT silently waved through --
   they are routed as a named Phase-4 decision (build vs. accept-as-named-exception,
   `pass3-admin-report.md` Section 7 item 8), because this pass's own harness-discipline
   principle would otherwise be inconsistent about them. Every one of the 7 net-new
   artifacts in Section 2/6 ships its `tests/<name>/run-tests.sh` FROM THE SAME COMMIT
   that ships the artifact, per the per-module harness contract adopted at
   `pass3-admin-report.md` Section 6: prove-it-can-fail, assert the artifact not the
   intent, offline/fixture-driven by default with a separately-named LIVE re-run,
   `$SITE`-parameterized fixtures, standard exit contract, delivery discipline.

3. **Idempotence at every layer.** L0-L2 get this from `tofu apply`'s own semantics;
   L3/L4 procedure modules stay re-run-safe via the existing `check`/`apply --commit`
   split (`dc-region-topology.sh`, `dc-plane-ipam.sh`, `phase-00-maas-standup.sh`) and
   the MAAS "READY not deployed" handoff that `preflight.sh`'s drift-detection gate
   already enforces at the L3/L4 boundary. The teardown primitive (Section 3.2) and
   every new L5 gate inherit the same "safe to re-run, safe to re-verify" contract --
   the (a) control's `--check` and cloud-assert's A11a/A11b are explicitly RE-INVOKED
   at multiple points in Section 3's sequence, not one-shot.

4. **The strict IaC<->procedure boundary is the load-bearing rule, not a preference.**
   Restated from Section 1: the boundary is identity (a MAC, an IP, a hostname), never
   orchestration (a state file read, a cross-host provider dial). This is the rule the
   container layer violated and the ONE thing Option 1 fixes structurally. Every new
   module in Section 2/6 is placed on the correct side of this line by construction: the
   (a) control and the power-key mitigation are procedure/L5, NOT tofu resources, even
   though both are "about" IaC-produced objects (`pass1-admin-report.md` Section 3;
   `pass2-w1-tofu-modules.md` Section 3.4) -- because both are LIVE verifications of a
   running kernel/credential state, not declarations of desired infrastructure shape.

5. **No layer reaches past the one directly below it.** L1 does not dial L3; L3 does not
   dial L0's infrastructure directly; a Stage-4 script never reads `opentofu/*.tfstate`
   (`pass2-w4-module-decomposition.md` Section 4's boundary-summary table). This is what
   makes the per-`$SITE` loop in Section 3 safe to reorder, parallelize, or re-run a
   single stage in isolation without re-deriving the whole sequence's state by hand.

6. **Findings/design stay logged at their true layer, not folded upward.** The
   rack-controller-remainder/D-131/mirror placement question is explicitly an L3
   placement decision (Section 2, L3 table) that this design does NOT resolve by
   picking a value -- it states the dependency (Stage 3.5/4.6-4.8's conditional steps)
   and defers to the ruling. Mirrors `pass1-w4-module-planning.md` Section 4 item 5's
   own discipline.

7. **D-140 is a distinct, future axis -- noted, not folded in.** D-140 (PINNED, not
   ruled) would eventually make L4 IaC-managed too, consuming `dc-site`'s outputs from a
   SEPARATE later root (e.g. `opentofu/vr1-dcN-juju/`) without reshaping L0-L3
   (`pass2-w1-tofu-modules.md` Section 4). The root-naming convention this design
   recommends (`<site_token>-<purpose>`, Section 2 L2) stays generic enough to admit a
   future `-juju` root per DC without a rename.

---

## 5. THE ROOSEVELT-TRANSFER STORY -- per layer, not per artifact

Per `pass1-w4-module-planning.md` Section 4 item 7's stated lens ("Roosevelt-transfer
judged per layer, not per artifact"): the pre-Roosevelt bare-metal test replaces the
VIRTUALIZATION substrate (vcloud's libvirt) with physical hosts. The question for each
layer is whether its module bodies assume libvirt/qemu, or whether they already operate
purely on live-observed identity (MAC, IP, hostname) that is substrate-agnostic.

| Layer | Transfers to bare metal? | Why / why not |
|---|---|---|
| **L0** | **NO -- does not transfer** | The mesh triangle (dark-fiber `mesh-link` legs) and the netem-link DR-drill mechanism are virtualization-only shims with no bare-metal analog (`pass1-w4-module-planning.md` Section 4 item 7, citing `docs/dc-dc-deployment-workflow.md:11-14`'s own shim register). A physical multi-site test either has real inter-site links (no `mesh-link` needed) or none at all -- this layer's IaC bodies are simply not reused. |
| **L1** | **YES -- the client-VM PATTERN is the direct pre-Roosevelt deliverable** | The client VM ("the cloud-facing client lives IN the DC," D-138, `docs/design-decisions.md:7118-7124`) is explicitly the D-138 Roosevelt bastion analog, "rehearsed early" (`pass0-admin-report.md` Option-1 "For" bullet). On bare metal this becomes a physical or minimally-virtualized bastion host in the same role -- same D-138 principle, same L1 contract (booted, network-identified, non-hypervisor), different provisioning mechanism underneath. voffice1's D-114 pattern is separate and out of this pass's scope either way. |
| **L2** | **NO, as an IaC mechanism -- but the CONTRACT transfers** | `node-vm`/`dc-planes`'s libvirt-domain bodies have no bare-metal analog (physical hosts are not libvirt domains) -- this is the same "node-VM shim" `pass1-w4-module-planning.md` Section 4 item 7 names as non-transferring. What DOES transfer is the L2 CONTRACT itself: "produce a booted object with correct MAC-per-NIC network identity wired to the right planes" is exactly what a physical host's out-of-band provisioning (PXE + BMC) must also satisfy before L3 can start -- the CONTRACT is substrate-agnostic even though `dc-site`'s OpenTofu implementation is not. |
| **L3** | **YES -- unchanged, verbatim** | This is the clean transfer point: `dc-node-carve.sh`, `maas-node-power.sh`, `maas-role-tags.sh`, `dc-region-topology.sh`, `dc-plane-ipam.sh` already work by re-deriving everything from LIVE identity (`lib-hosts.sh:6-11`'s own design rule) and take the power/commissioning TARGET as an input, never a baked value. The one substitution point is `maas-node-power.sh`'s power backend (`power_type=virsh` -> a physical BMC/IPMI power type) -- a parameter change to the SAME script, not a rewrite, exactly the class of change D-143's address-substitution axis already demonstrates this repo tolerates cleanly. |
| **L4** | **YES -- wholly unchanged; this IS "the good test of the module deployment project"** | `bundle.yaml`, the per-DC overlays (`render-dc-overlays.py`), the VR0 template chain (`phase-00`..`phase-07-*.sh`), and `preflight.sh`/`cloud-assert.sh` have zero dependency on the compute substrate below L3's MAAS handoff -- they consume READY machines by API, not by libvirt identity. This is precisely the operator's own framing of the bare-metal test: *"a good test of the module deployment project we are developing during the teardown and redeploy"* (`SCOPE-AND-EXECUTION-PLAN.md` Section 1) -- L4 (and L3) are the module deployment project's actual payload; L0/L2's virtualization substrate is scaffolding around it. |
| **L5** | **MOSTLY YES, with one exception** | `preflight.sh`, `cloud-assert.sh`, `geneve-encap-assert.sh` are topology-agnostic (verified clean of container-layer assumptions, `pass1-admin-report.md` Section 2.3 / `pass3-admin-report.md` Section 1 check 4). The cross-DC host isolation control (concern i) and the power-key blast-radius mitigation (concern iii) are BOTH specific to vcloud's single-libvirtd co-residency problem -- on physical hosts there is no shared hypervisor kernel for two DCs' domains to leak across, so neither control's PROBLEM exists in the same shape on bare metal (a bare-metal equivalent, if any, would be a physical-network isolation control -- explicitly out of this design's scope, a Roosevelt-time question). |

**Bottom line, matching the operator's own framing:** the layers that DO NOT transfer
(L0's mesh/netem shim, L2's libvirt-domain mechanics) are exactly the layers that exist
ONLY because vcloud is virtualized hardware standing in for real DCs -- not because they
encode anything about how OpenStack gets deployed. The layers that DO transfer (L1's
client-VM pattern, L3's MAAS-enlist procedures, L4's Juju/bundle procedures) are the
actual "module deployment project" the operator named -- rehearsing them on the 10.13
flat topology now is the direct dry run for Roosevelt.

---

## 6. Where the three isolation controls + the client VM + the teardown primitive sit

Consolidated view (each already placed in Section 2/3 above; gathered here per the
task's explicit requirement):

| Object | Layer | Kind | Why there |
|---|---|---|---|
| **Client VM** (`vr1-dcN-client`) | **L1** (IaC instantiation, same `cloudinit-vm` module type as voffice1/edges) | IaC module instance | It is a booted, network-identified, non-hypervisor VM -- exactly L1's contract. Its APPLY groups with L2's per-DC-flat root (teardown symmetry + SEC-026 state isolation + D-138 fidelity, `pass2-w1-tofu-modules.md` Section 2.4) even though its module TYPE is L1 -- layer classification and apply grouping are orthogonal, both hold (`pass1-admin-report.md` Section 1 check 7). |
| **Cross-DC host isolation control** (concern i, the "(a)" control) | **L0-scoped, L5 kind** (a procedure/verify artifact ABOUT an L0-level fact -- vcloud's own kernel-level forwarding) | Gate (procedure), NOT a tofu module | Confirmed explicitly: "SEC-010's actual pattern... NOT an OpenTofu module" (`pass1-admin-report.md` Section 3). Installed and `--check`-verified BEFORE the first Stage-3 (L2) apply of either per-DC root (Section 3.1's fork-robust invariant) -- it gates the layer above it without being IN that layer. |
| **SEC-010 transit-leg successor** (concern ii) | **L1-scoped** (installed on two L1 objects: the client VM + voffice1) | Procedure, one role-agnostic installer for both ends | It protects the transit LEG -- an L1-object property (both endpoints are `cloudinit-vm` instances) -- not a DC-substrate (L2) or commissioning (L3) fact. Homed at preflight P10, DC-scoped (`pass3-admin-report.md` Section 3). |
| **Power-key blast-radius mitigation** (concern iii) | **L3-scoped** (the credential itself is installed and used at L3 -- `maas-region-power-key.sh` / `maas-node-power.sh`), with its **gate split across L3/L4** (preflight P4 dependency + cloud-assert A11b standing re-verify, L5) | Procedure (credential-scope control) + L5 gate | It is fundamentally about WHO can dial WHICH domains via libvirt -- an L3 commissioning-credential fact, not an L2 substrate fact or an L0 network fact (`pass2-admin-report.md` Section 2.3's own distinction: "two network controls and one credential-scope control"). |
| **Teardown primitive** | **L2/L5 boundary** | Procedure that WRAPS an IaC destroy, verified by an L5-style completeness check | It operates on L2's state (a `tofu destroy` scoped to one DC's flat root, or a `virsh destroy` loop over that root's state-listed domains) but its CORRECTNESS property (no cross-DC leakage, no incomplete teardown) is verified the same way an L5 gate verifies anything else -- hence "L2/L5 boundary," matching `pass1-w4-module-planning.md` Section 3's own row for this artifact. |

---

## 7. Open design questions (not resolved by this worker -- carried to the operator / delivery)

Every item below is already OPEN in an earlier pass report; restated here only because
it bears directly on this module design's final shape. None is invented or pre-picked.

1. **Root naming** (`vr1-dcN-flat` vs. reserving `-substrate`) -- blocks A1/A2 test
   path re-points; Phase-4's call (`pass2-admin-report.md` Section 6 item 2).
2. **Root topology ratification** -- (B) is RECOMMENDED (Section 3.1), not yet ruled;
   the whole composition sequence's per-DC independence claim depends on it.
3. **Rack-controller-remainder + D-131 forwarder + artifact-service (`.4`) placement**
   -- gates Stage 3.5/4.6-4.8's conditional steps in Section 3; "THE highest-leverage
   open item" per `pass1-admin-report.md` Section 7.
4. **The (a) control's concrete nftables rule set** -- kind/home/ordering are settled
   (Section 2/6), the RULE SET itself is not.
5. **The power-key mitigation's mechanism** (restricted key / per-DC virsh wrapper /
   polkit ACL) -- blocks `maas-node-power.sh`/`maas-region-power-key.sh` literals and
   the `lib-hosts.sh` power-address re-derivation (Section 2, L3; the H1
   "plausible-URI-swap" hazard, `pass3-admin-report.md` Section 5).
6. **D-127's client-VM autostart value** -- needed for the new `opentofu-validate` T14/
   T15 case (Section 2, L1 table).
7. **The container-elim [ARCH] ruling itself** (D-123 amendment vs. new D-number, +
   ride-alongs: D-125 retirement, D-138 concrete-host, D-128 amendment, D-122 site-down
   re-earn, D-124 sizing-void) -- this is a PRECONDITION for Stage 3 ever running for
   the first time (Section 3, C.2), framed by Phase 4's W4.4 dimension, ruled by the
   operator (GA-R5), not by this document.
8. **`maas-fabric-prune.sh` harness gap** -- pre-existing, named Phase-4 decision
   (Section 4 item 2).

---

## 8. Verification note

Author = "the worker" (no model name asserted, operator instruction). This document
performs no live measurement of its own -- every module/artifact claim traces to a
direct read of `pass0`-`pass3`'s administrator reports and their cited worker docs
(`pass1-w4-module-planning.md`, `pass2-w1-tofu-modules.md`,
`pass2-w4-module-decomposition.md`), plus `docs/tool-index.md`'s stated discipline
(consulted, not newly operated against). Every **[NEW]** artifact cites the pass/section
that scoped it; every **[EXISTS]** artifact cites its current path. No inferred values --
every OPEN item in Section 7 is carried as OPEN, not resolved here. READ-ONLY; no
mutation performed.