Pass 4 / W4.2 -- THE LAYERED MODULE-WORKFLOW DESIGN (container-layer elimination)

Worker: W4.2 (Phase 4, container-layer-elimination pass, SCOPE-AND-EXECUTION-PLAN.md Section 4 -- "the pass's headline deliverable"). Date: 2026-08-09. Scope: READ-ONLY planning. Produces the concrete design of the layered module system the redeploy becomes; performs no mutation.

Baseline consumed in full, this session: SCOPE-AND-EXECUTION-PLAN.md, pass0-admin-report.md (Option 1 CONFIRMED, Section 7a), pass1-admin-report.md + pass1-w4-module-planning.md (the L0-L5 layer model, adopted here as the backbone), pass2-admin-report.md + pass2-w1-tofu-modules.md + pass2-w4-module-decomposition.md (the module change-set, the dc-site IaC module, the IaC<->procedure boundary, the 13 owed artifacts), pass3-admin-report.md (the unified tests change-set, the gate-home map, the per-module harness contract). Every module/artifact below is grounded in one of these documents or the live repo; anything not yet built is marked [NEW] and cites which pass scoped it. No inferred values -- unresolved design points (root naming, the (a)/(iii) mechanisms, rack-controller placement) are carried as OPEN, matching their source reports.

0. What this document is (and is not)

This is the CONCRETE module design that pass1-w4-module-planning.md's L0-L5 model scaffolds and pass2/pass3's change-sets populate. It answers three questions:

1. Per layer, what are the modules, and what is each one's contract? (Section 2)
2. How does a full per-DC redeploy compose these modules into one ordered invocation, parameterized by site token? (Section 3)
3. What design principles make this repeatable and Roosevelt-transferable, and which layers/modules actually carry over to the pre-Roosevelt bare-metal test unchanged? (Sections 4-5)

It does NOT re-rule anything already settled (Option 1, handling (a), MAAS-region placement) or re-open anything explicitly OPEN in the source reports (root naming, the (a)/(iii) mechanisms, rack-controller-remainder placement, root topology's final ratification). Where a module's shape depends on an OPEN item, this document states the dependency and defers -- it does not invent a value (hard rule 2).

1. The layer model, restated as the design's backbone

Adopted verbatim from pass1-w4-module-planning.md Section 2, consolidated at pass1-admin-report.md Section 5:

L0  Host & inter-site substrate      (IaC)         -- Stage 1
L1  Site/edge nodes                  (IaC)         -- Stage 2 (Office1) / Stage 3 (DC, flattened in)
L2  DC substrate: planes + node VMs  (IaC)         -- Stage 3
L3  Enlist/commission                (procedure)   -- Stage 4
L4  Juju/OpenStack deploy            (procedure)   -- Stage 5 (+6 DR, +7 Designate/COS/Magnum, additive)
L5  Verify/gate                      (cross-cutting procedure, re-invoked at every boundary)

Layer-boundary rule (the design's one hard invariant, pass1-w4-module-planning.md Section 2, restated by pass2-w4-module-decomposition.md Section 4 as "identity, not orchestration"): each layer's input is the layer directly below's OUTPUT only. OpenTofu (L0-L2) owns up to "a booted libvirt domain exists, with its network identity (MAC per NIC, and any statically-assigned IP) correctly wired to the right plane bridges." The procedure layer (L3+) begins at the first live dial into that object and never reads OpenTofu state -- it re-derives everything from live, independently observable identity (MAC, hostname, a fresh API/SSH probe), per lib-hosts.sh:6-11's own stated design rule.

The container layer was the one place this rule was violated, and it is the reason this pass exists: the inner root (opentofu/vr1-dc0-substrate/main.tf) dialed OUT to a qemu+ssh provider INTO the outer root's own vvr1_dc0 output -- IaC reaching into IaC across a live-dial boundary that only a procedure module should cross (opentofu/main.tf:28-29; pass2-w4-module-decomposition.md Section 4). Option 1 removes this violation structurally: with one flat per-DC root, L2 is IaC end to end, and the FIRST live dial into anything L2 produced is L3's MAAS commissioning -- exactly where the boundary rule says it should be. Every module table below is built so this stays true.

2. Per-layer module design

Legend: kind = IaC module / procedure module / library / gate. [EXISTS] = re-homed or unchanged artifact, cited to its current path. [NEW] = owed artifact, cited to the pass that scoped it (pass1 #N / pass2 #N referring to the numbered owed-artifact lists in pass1-admin-report.md Section 6 / pass2-admin-report.md Section 5).

L0 -- Host & inter-site substrate (IaC, Stage 1)

Contract: given vcloud's qemu:///system connection and the site tokens (office1/vr1-dc0/vr1-dc1), produce the cross-site fabric every other layer plugs into: the mesh triangle, per-site storage pools, the Office1 L2 network, and the inter-DC netem link. No DC-specific node/plane content lives here.

Composes onto: nothing below it -- L0 is the floor. Stage owner: Stage 1 (docs/dc-dc-deployment-workflow.md, gate content unchanged per pass1-admin-report.md Section 2.2). The (a) control's recommended home is also Stage 1 -- host-scoped, not per-DC-apply-scoped (pass1-admin-report.md Section 3).

L1 -- Site/edge nodes (IaC, Stage 2 Office1 / Stage 3 DC-flattened)

Contract: given an L0 network + pool, produce a booted, network-identified non-hypervisor VM: Office1's headend, each DC's edge, and (Option 1's new instance) each DC's client VM. All three are the SAME module type -- cloudinit-vm -- differing only by inputs.

Composes onto: L0's network + pool outputs only. Stage owner: Stage 2 (Office1, untouched) / Stage 3 (DC edge + client VM, now co-located with L2's per-DC apply -- Section 2.4 of pass2-w1-tofu-modules.md: teardown symmetry + SEC-026 state isolation + D-138 fidelity all argue for grouping the client VM's APPLY with its DC's flat root even though its module TYPE is L1).

L2 -- DC substrate: planes + node VMs (IaC, Stage 3)

Contract: given the L0 pool + L1 client VM's transit identity, and a site token + D-121/R-3 node roster + D-134 octet map, produce the DC's six plane networks and its 12 node-VM libvirt domains (9 role nodes + 3 utility nodes), each with a MAC-pinned network identity MAAS can discover. This is the layer the container layer's boundary violation lived in and Option 1 collapses it to a single root/state.

Composes onto: L0's pool (via pool_path) and L1's client-VM transit identity (via client_vm.transit_network_name, a cross-root STRING reference -- the same pattern office1_opnsense's wan_network_name literal already uses, main.tf:112). Does NOT reach into L3 -- dc-site's outputs are network_names/domain_ids/MACs only; it never calls MAAS or juju. Stage owner: Stage 3, the HEAVIEST rewrite (pass1-admin-report.md Section 2.2): Step B (bootstrap gate) eliminated wholesale, Step C (inner apply) merges into Step A -- one apply, one root-scope, one host, one state.

L3 -- Enlist/commission (procedure, Stage 4)

Contract: given L2's MAC-pinned node VMs and a reachable MAAS region (L1's vr1-dcN-maas-01, D-132 addendum, unchanged by flattening), produce READY, carved, tagged MAAS machines and the per-DC artifact-mirror/DNS-forwarder services those machines need during commissioning. The handoff artifact from L2 is a MAC address, observed independently on both sides -- no tofu state crosses this boundary (pass2-w4-module-decomposition.md Section 4).

Composes onto: L2's MAC-pinned domains + L1's reachable MAAS region only; never dials L0 or reads tofu state. Stage owner: Stage 4 (dc-dc-phase3-maas-enlist- deploy.md, LOW DELTA -- only the two SSH-jump-target literals change, per pass1-admin-report.md Section 2.1).

L4 -- Juju/OpenStack deploy (procedure, Stage 5 + additive Stage 6/7)

Contract: given L3's READY machines, a site token, bundle.yaml + per-DC overlays, and a Vault root, stand up an independently-running OpenStack cloud per DC. D-140 PINS this as a procedure layer for THIS redeploy -- settled, not folded into IaC (pass1-admin-report.md Section 5).

Composes onto: L3's READY-machine handoff only. Stage owner: Stage 5 (primary), Stage 6 (DR drill, additive per vr0-to-vr1-is-additive), Stage 7 (Designate/COS/Magnum, additive) -- Stages 6-7 have NO container-layer dependency, verified by W1.2 (pass1-admin-report.md Section 2.2).

L5 -- Verify/gate (cross-cutting procedure, invoked at every layer boundary)

Contract: at every layer's declared-done state, produce a failable PASS/FAIL (never existence-only, GA-R6) plus, at milestones, a committed BOM. L5 is not owned by one stage; it re-runs at each stage's close.

Composes onto: any layer's declared-done state; L5 gates never mutate by default (gate = "its sole job is verify," pass2-w4-module-decomposition.md Section 2 legend).

3. THE COMPOSITION MECHANISM -- one ordered invocation, parameterized by site token

A full per-DC redeploy is the following ordered sequence. $SITE is the site token (vr1-dc0 / vr1-dc1, D-119); steps marked (once) run before the per-DC loop and are NOT re-run per DC; steps marked (per $SITE) run once for each DC, in either order (root shape (B) makes them structurally independent -- Section 3.1 below). Axis tags [D-143]/[CE]/[both] carried from pass1-admin-report.md Section 4 so the two riding changes (the 10.13 re-IP and the container-elim) stay distinguishable in this same sequence.

STAGE 1  (once, L0 + L5)
  1.1 [unchanged]  tofu apply: outer/shared root -- mesh triangle, per-site pools,
                    office1-network, netem link, base-image                         (L0)
  1.2 [CE, NEW]     install + --check the cross-DC host isolation control            (L5)
                     -- MUST precede 1.4/2.x for EITHER DC (fork-robust invariant,
                        `pass1-admin-report.md` Section 3)

STAGE 2  (once, L1)
  2.1 [unchanged]  Office1 headend (voffice1) standup -- D-114, untouched            (L1)

  --- per-$SITE loop begins; (B) root shape makes each iteration state-isolated ---

STAGE 3  (per $SITE, L1 + L2)
  3.1 [both]        tofu apply: vr1-$SITE-flat root -> module "site" = dc-site(
                       site_token=$SITE, planes=..., nodes=..., client_vm=...)
                     composes, IN ORDER: dc-storage-pool -> dc-planes -> opnsense-edge
                     -> node-vm[12] -> cloudinit-vm(client)                     (L1+L2)
  3.2 [CE]           re-verify the (a) control's --check now that this DC's planes
                     exist (post-apply close)                                        (L5)
  3.3 [NEW]          MAC re-measurement pass (owed #8) -- confirm every domain's MAC
                     before Stage 4 trusts one                                       (L2/L3 handoff)

STAGE 3.5 (per $SITE, L1-hosted procedure installs)
  3.5.1 [CE]         SEC-010 successor installer, BOTH ends (client VM + voffice1)   (L1)
  3.5.2 [CE, cont.]  site-headend-install.sh --role rack  (IF rack-retirement is
                       NOT ratified; else SKIPPED)                                   (L1/L3)
  3.5.3 [both]       maas-region-power-key.sh -- key shape per the #11 mitigation    (L3)

STAGE 4  (per $SITE, L3)
  4.1 [D-143]        dc-region-topology.sh --commit                                  (L3)
  4.2 [D-143]        dc-plane-ipam.sh --commit                                       (L3)
  4.3 [unchanged*]   dc-node-carve.sh / dc-node-v6-carve.py -- per-machine
                     power_type=virsh; *only the power-ADDRESS value re-derives      (L3)
  4.4 [both]         maas-node-power.sh -- value BLOCKED on #11                      (L3)
  4.5 [unchanged]    maas-role-tags.sh                                               (L3)
  4.6 [CE, cont.]    dc-rack-net.sh (D-131 forwarder) -- IF retire-with-evidence
                       is REJECTED for this DC; else the D-131 evidence step (4.7)   (L3)
  4.7 [CE, cont.]    D-131 retirement-evidence step -- IF retirement adopted         (L3)
  4.8 [CE]           dc-mirror.sh / dc-cache-proxy.sh / dc-snap-proxy.sh, sized
                     per the FIT-calculator extension (#7)                           (L3)

STAGE 5  (per $SITE, L4 + L5)
  5.1 [gate]         preflight.sh (P1-P10)                                           (L5)
  5.2 [unchanged]    juju bootstrap + bundle.yaml + overlays, FROM vr1-$SITE-client   (L4)
  5.3 [both]         SEC-026/028/029 credentials freshly minted on the client VM      (L4)
  5.4 [gate]         cloud-assert.sh (incl. A11a/A11b) + geneve-encap-assert.sh       (L5)

  --- per-$SITE loop ends ---

STAGE 6  (once, cross-DC, L4, additive)
  6.1 [unchanged]    DR/failover drill: netem-link mechanism + Ceph rbd-mirror /
                     radosgw-multisite (D-108)                                       (L4)

STAGE 7  (once per $SITE, L4, additive)
  7.1 [unchanged]    Designate / COS / Magnum (D-106/D-105, `vr0-to-vr1-is-additive`) (L4)

CLOSE-OUT (per $SITE, L5 + record-keeping)
  C.1 [CE]           NetBox DCIM: decommission vvr1-$SITE, register client VM+roster (record)
  C.2 [D-143/CE]     the container-elim [ARCH] ruling itself is NOT a gate -- it is a
                     precondition the operator must have already ruled before Stage 3
                     runs for the FIRST time (Section 7)

3.1 Why the loop is safe to run per-DC in either order (root shape (B))

Under the recommended root shape -- shared-outer + per-DC-flat roots (pass2-w1-tofu-modules.md Section 2, RECOMMENDED, Phase-4 ratifies) -- DC1's Stage-3 apply cannot create or touch any DC0 resource: it is not in DC0's state file, full stop. This makes the composition mechanism's ordering invariant reduce to a single, simple contract: "the (a) control's --check must pass before the first Stage-3 apply of EITHER per-DC-flat root" (Step 1.2), rather than depending on -target discipline being followed correctly on every apply (the risk a merged single root would carry). This is why Stage 1's step 1.2 is drawn OUTSIDE the per-$SITE loop and BEFORE it in the sequence above -- it is a precondition for the loop, not a loop step.

3.2 Teardown primitive's place in the same composition

The teardown/site-down lever is the composition mechanism's INVERSE, not a separate design: under (B), site-down for $SITE is cd opentofu/vr1-$SITE-flat/ && tofu destroy (gated) or a scripted virsh destroy loop over that root's own state-listed domains (emergency) -- touching only that DC's root, never the shared-outer root or the other DC's root (pass2-w1-tofu-modules.md Section 2.3). This is owed artifact pass1 #1 (+

#6 the emergency lever, riding the same fixture library per pass3-admin-report.md Section 2 rides list) and sits at the L2/L5 boundary: it is a procedure module that WRAPS an IaC destroy, verified by an L5-style completeness check (a dc1 domain leaking into a dc0-targeted set FAILS; an empty resolved set REFUSES rather than reporting "nothing to do" success -- pass3-admin-report.md C3).

4. Design principles that make this repeatable + Roosevelt-transferable

Adopted from pass1-w4-module-planning.md Section 4, sharpened by pass2/pass3's findings. Each principle is grounded in an EXISTING repo pattern, not invented for this pass (pass1-w4-module-planning.md Section 1's framing: "the container-elim does not need to invent module mechanics, only re-home").

1. Site-token parameterization, never hardcoded identity. The $DC/$SITE selector (D-119, DOCFIX-151, lib_net_select_dc/lib_hosts_select_dc) already does this for every procedure module; the IaC layer already does it structurally (no module BODY names a DC). Every new artifact this pass introduces takes the same token as an input rather than being written DC0-specific and copy-pasted for DC1: dc-site's site_token input (Section 2, L2), the (a) control's per-bridge-tag enumeration (Section 2, L0/L5), the power-key mitigation's per-DC domain-set scoping (Section 2, L3). This is the exact anti-pattern docs/dc-dc-deployment-workflow.md's gap-register item 1 was created to close.

2. Every module ships its tested harness, no exception. tests/<name>/run-tests.sh for procedure modules, opentofu-validate.sh's standalone-module coverage for IaC (CLAUDE.md "Delivery"; pass2-w4-module-decomposition.md Section 1.3). Of the 34 procedure modules surveyed at Phase 2, 32 already comply; the two pre-existing gaps (maas-fabric-prune.sh / maas_fabric_classify.py) are NOT silently waved through -- they are routed as a named Phase-4 decision (build vs. accept-as-named-exception, pass3-admin-report.md Section 7 item 8), because this pass's own harness-discipline principle would otherwise be inconsistent about them. Every one of the 7 net-new artifacts in Section 2/6 ships its tests/<name>/run-tests.sh FROM THE SAME COMMIT that ships the artifact, per the per-module harness contract adopted at pass3-admin-report.md Section 6: prove-it-can-fail, assert the artifact not the intent, offline/fixture-driven by default with a separately-named LIVE re-run, $SITE-parameterized fixtures, standard exit contract, delivery discipline.

3. Idempotence at every layer. L0-L2 get this from tofu apply's own semantics; L3/L4 procedure modules stay re-run-safe via the existing check/apply --commit split (dc-region-topology.sh, dc-plane-ipam.sh, phase-00-maas-standup.sh) and the MAAS "READY not deployed" handoff that preflight.sh's drift-detection gate already enforces at the L3/L4 boundary. The teardown primitive (Section 3.2) and every new L5 gate inherit the same "safe to re-run, safe to re-verify" contract -- the (a) control's --check and cloud-assert's A11a/A11b are explicitly RE-INVOKED at multiple points in Section 3's sequence, not one-shot.

4. The strict IaC<->procedure boundary is the load-bearing rule, not a preference. Restated from Section 1: the boundary is identity (a MAC, an IP, a hostname), never orchestration (a state file read, a cross-host provider dial). This is the rule the container layer violated and the ONE thing Option 1 fixes structurally. Every new module in Section 2/6 is placed on the correct side of this line by construction: the (a) control and the power-key mitigation are procedure/L5, NOT tofu resources, even though both are "about" IaC-produced objects (pass1-admin-report.md Section 3; pass2-w1-tofu-modules.md Section 3.4) -- because both are LIVE verifications of a running kernel/credential state, not declarations of desired infrastructure shape.

5. No layer reaches past the one directly below it. L1 does not dial L3; L3 does not dial L0's infrastructure directly; a Stage-4 script never reads opentofu/*.tfstate (pass2-w4-module-decomposition.md Section 4's boundary-summary table). This is what makes the per-$SITE loop in Section 3 safe to reorder, parallelize, or re-run a single stage in isolation without re-deriving the whole sequence's state by hand.

6. Findings/design stay logged at their true layer, not folded upward. The rack-controller-remainder/D-131/mirror placement question is explicitly an L3 placement decision (Section 2, L3 table) that this design does NOT resolve by picking a value -- it states the dependency (Stage 3.5/4.6-4.8's conditional steps) and defers to the ruling. Mirrors pass1-w4-module-planning.md Section 4 item 5's own discipline.

7. D-140 is a distinct, future axis -- noted, not folded in. D-140 (PINNED, not ruled) would eventually make L4 IaC-managed too, consuming dc-site's outputs from a SEPARATE later root (e.g. opentofu/vr1-dcN-juju/) without reshaping L0-L3 (pass2-w1-tofu-modules.md Section 4). The root-naming convention this design recommends (<site_token>-<purpose>, Section 2 L2) stays generic enough to admit a future -juju root per DC without a rename.

5. THE ROOSEVELT-TRANSFER STORY -- per layer, not per artifact

Per pass1-w4-module-planning.md Section 4 item 7's stated lens ("Roosevelt-transfer judged per layer, not per artifact"): the pre-Roosevelt bare-metal test replaces the VIRTUALIZATION substrate (vcloud's libvirt) with physical hosts. The question for each layer is whether its module bodies assume libvirt/qemu, or whether they already operate purely on live-observed identity (MAC, IP, hostname) that is substrate-agnostic.

Bottom line, matching the operator's own framing: the layers that DO NOT transfer (L0's mesh/netem shim, L2's libvirt-domain mechanics) are exactly the layers that exist ONLY because vcloud is virtualized hardware standing in for real DCs -- not because they encode anything about how OpenStack gets deployed. The layers that DO transfer (L1's client-VM pattern, L3's MAAS-enlist procedures, L4's Juju/bundle procedures) are the actual "module deployment project" the operator named -- rehearsing them on the 10.13 flat topology now is the direct dry run for Roosevelt.

6. Where the three isolation controls + the client VM + the teardown primitive sit

Consolidated view (each already placed in Section 2/3 above; gathered here per the task's explicit requirement):

7. Open design questions (not resolved by this worker -- carried to the operator / delivery)

Every item below is already OPEN in an earlier pass report; restated here only because it bears directly on this module design's final shape. None is invented or pre-picked.

1. Root naming (vr1-dcN-flat vs. reserving -substrate) -- blocks A1/A2 test path re-points; Phase-4's call (pass2-admin-report.md Section 6 item 2).
2. Root topology ratification -- (B) is RECOMMENDED (Section 3.1), not yet ruled; the whole composition sequence's per-DC independence claim depends on it.
3. Rack-controller-remainder + D-131 forwarder + artifact-service (.4) placement -- gates Stage 3.5/4.6-4.8's conditional steps in Section 3; "THE highest-leverage open item" per pass1-admin-report.md Section 7.
4. The (a) control's concrete nftables rule set -- kind/home/ordering are settled (Section 2/6), the RULE SET itself is not.
5. The power-key mitigation's mechanism (restricted key / per-DC virsh wrapper / polkit ACL) -- blocks maas-node-power.sh/maas-region-power-key.sh literals and the lib-hosts.sh power-address re-derivation (Section 2, L3; the H1 "plausible-URI-swap" hazard, pass3-admin-report.md Section 5).
6. D-127's client-VM autostart value -- needed for the new opentofu-validate T14/ T15 case (Section 2, L1 table).
7. The container-elim [ARCH] ruling itself (D-123 amendment vs. new D-number, + ride-alongs: D-125 retirement, D-138 concrete-host, D-128 amendment, D-122 site-down re-earn, D-124 sizing-void) -- this is a PRECONDITION for Stage 3 ever running for the first time (Section 3, C.2), framed by Phase 4's W4.4 dimension, ruled by the operator (GA-R5), not by this document.
8. maas-fabric-prune.sh harness gap -- pre-existing, named Phase-4 decision (Section 4 item 2).

8. Verification note

Author = "the worker" (no model name asserted, operator instruction). This document performs no live measurement of its own -- every module/artifact claim traces to a direct read of pass0-pass3's administrator reports and their cited worker docs (pass1-w4-module-planning.md, pass2-w1-tofu-modules.md, pass2-w4-module-decomposition.md), plus docs/tool-index.md's stated discipline (consulted, not newly operated against). Every [NEW] artifact cites the pass/section that scoped it; every [EXISTS] artifact cites its current path. No inferred values -- every OPEN item in Section 7 is carried as OPEN, not resolved here. READ-ONLY; no mutation performed.