Newer
Older
openstack-caracal-dc-dc / docs / audit / container-elim-pass / pass1-w4-module-planning.md

Pass 1 -- W1.4: how planning artifacts map onto the layered module model

Worker: W1.4 (Phase 1, container-layer-elimination pass). Date: 2026-08-09. Scope: READ-ONLY. This is the conceptual backbone for Phase 4's module-workflow design (SCOPE-AND-EXECUTION-PLAN.md W4.2). Inputs read in full: SCOPE-AND-EXECUTION-PLAN.md, pass0-admin-report.md (Option 1 CONFIRMED at the Phase-0 gate, Section 7a), plus live repo survey below. No inferred values -- every claim below cites path:line or a directory listing taken this session.


1. What module structure ALREADY exists (survey, not proposal)

1.1 IaC modules -- opentofu/modules/ (12 modules, opentofu/modules/*)

Listing (find opentofu/modules -maxdepth 2 -type d, this session): base-image, cloudinit-vm, dc-planes, dc-storage-pool, maas-vm-host, mesh-link, netem-link, node-vm, office1-network, opnsense-edge, site-wan, wan-bridge.

Each module is already: (a) independently tofu validate-able (scripts/opentofu-validate.sh, opentofu/README.md:5-6,40-41 -- "validates EVERY module standalone"), (b) called from one or both of the two CURRENT roots -- the outer root opentofu/main.tf (vcloud provider) and the inner root opentofu/vr1-dc0-substrate/main.tf (qemu+ssh provider, D-123 Model B) -- and (c) already parameterized (no hardcoded DC identity inside a module body; the calling root supplies dc/CIDR/sizing values). This is a working IaC-module layer TODAY; the container-elim does not need to invent module mechanics, only re-home module CALLS from two roots into one and delete the modules that existed only to bridge the two-root split.

Outer-root calls (grep '^module "' opentofu/main.tf): vr1_dc0_storage, vr1_dc1_storage, office1_storage, office1_network, office1_opnsense, three mesh_* legs, ubuntu_noble_base, voffice1, netem_vr1_dc0_vr1_dc1, vr1_dc0_uplink, vr1_dc1_uplink, vvr1_dc0, vvr1_dc1 (opentofu/main.tf:35-537).

Inner-root calls (grep '^module "' opentofu/vr1-dc0-substrate/main.tf): inner_storage, vr1_dc0_planes, vr1_dc0_wan, vr1_dc0_opnsense, vr1_dc0_node (x12, D-121/R-3) (opentofu/vr1-dc0-substrate/main.tf:26-250).

The vvr1_dc0/vvr1_dc1 outer-root calls ARE the containment layer (pass0-admin-report.md Section 1.1); everything in the inner-root list is a module CALL that Option 1 re-homes, not a module BODY that changes shape (pass0-admin-report.md row 2: "module CALLS ... re-home into the outer/flat root -- bodies unchanged, provider changes").

1.2 Procedure/runbook modularity -- runbooks/dc-dc-phaseN-*.md + docs/dc-dc-deployment-workflow.md

docs/dc-dc-deployment-workflow.md already defines an 8-stage sequence (Stage 0 decision ratification through Stage 7 Designate/COS/Magnum), each stage with a fixed schema -- Goal / Build / Gate / Owns (D-numbers) / Reuse-vs-new / Authoring status -- and each stage's authoring status POINTS AT one runbooks/dc-dc-phaseN-*.md file:

Stage Runbook IaC layer it drives
1 dc-dc-phase0-vcloud-prep.md outer root: mesh/planes/pools/office1-network
2 dc-dc-phase1-office1-standup.md cloudinit-vm+base-image (voffice1), MAAS-composed LXD
3 dc-dc-phase2-tofu-dc-substrate.md outer vvr1_dcN + bootstrap gate + inner root (THE container layer)
4 dc-dc-phase3-maas-enlist-deploy.md no tofu; MAAS commission/deploy of already-created node VMs
5 dc-dc-phase4-juju-bundle-per-dc.md no tofu; Juju controller + bundle.yaml
6 dc-dc-phase5-dr-failover-drill.md netem-link mechanism + Ceph replication
7 dc-dc-phase6-designate-cos-magnum.md no tofu; Designate/COS/Magnum additive apps (D-106/D-105)

Each runbook is already parameterized by $DC (lib_net_select_dc "$DC" / lib_hosts_select_dc "$DC", DOCFIX-151, docs/dc-dc-deployment-workflow.md:485-498) and each stage's close-out is gated by repo-lint + a harness sweep + a changelog + a branch-merge (docs/dc-dc-deployment-workflow.md:424-436, "Cross-cutting discipline"). This IS a procedure- module system already -- the container-elim pass does not need to invent the concept, only (a) rewrite Stage 3's content for the flat topology and (b) formalize the pattern the operator is asking for so it is named and reusable, not just an emergent convention.

Underneath the stage runbooks, two shared libraries act as the procedure layer's own "IaC modules": scripts/lib-net.sh and scripts/lib-hosts.sh -- every stage-4/5 script sources them for CIDR/host facts, keyed by the $DC selector (vr1-dc0/vr1-dc1, D-119). This is the existing parameterization mechanism the layer model below reuses rather than replaces.

1.3 Test-harness convention (governs BOTH module kinds)

  • IaC: one shared gate, scripts/opentofu-validate.sh (tests/opentofu-validate/), validates every module standalone plus both roots.
  • Procedure: per-script harnesses, tests/<script-name>/run-tests.sh (tool-index: "65 scripts with their own tests/<name>/ harness"; docs/tool-index.md:27).

No module of either kind ships without its harness (CLAUDE.md "Delivery" rule); this is already the repo norm the layer model's design principles (Section 4) restate for the new modules Option 1 introduces, not a new rule.


2. Proposed layer model

Six layers, named to match the EXISTING Stage numbers where a layer is Stage-owned, so the layer model is a REFRAME of what's already built, not a parallel taxonomy. "IaC or procedure" marks whether the layer's artifact is an OpenTofu module/root or a runbook+script procedure. Verify layers (L5) are cross-cutting and re-invoked at every layer boundary, not a one-time final step.

Layer Name Kind Inputs Outputs Current artifacts Stage owner
L0 Host & inter-site substrate IaC module vcloud libvirt connection; site tokens (Office1/dc0/dc1); MTU plan (D-101) mesh triangle (dark-fiber legs), per-site storage pools, office1-network modules/mesh-link, modules/dc-storage-pool, modules/office1-network, outer main.tf:35-163 Stage 1
L1 Site/edge nodes IaC module L0 outputs (a network + a pool); per-VM cloud-init or edge-image spec Office1 headend VM (voffice1); per-site OPNsense edge VMs; (Option 1 NEW) the per-DC client VM modules/cloudinit-vm, modules/base-image, modules/opnsense-edge, modules/site-wan Stage 2 (Office1); Stage 3 (DC edges, currently split by the container layer -- Option 1 flattens into this same layer)
L2 DC substrate: planes + node VMs IaC module L0 pool + L1 edge; $DC token; D-121/R-3 node counts/sizing; D-134 octet map six per-DC plane networks; 9+3 node-VM libvirt domains (role nodes + utility nodes) per DC modules/dc-planes, modules/node-vm; TODAY split outer(vvr1_dcN)/inner(vr1_dc0_planes,vr1_dc0_nodex12) -- Option 1 COLLAPSES this to ONE root, one state Stage 3
L3 Enlist/commission procedure Procedure module L2 node VMs (MAC-pinned, discoverable); MAAS region reachable (L1's vr1-dcN-maas-01, D-132 addendum) READY, carved, tagged MAAS machines; per-DC artifact-mirror answering runbooks/dc-dc-phase3-maas-enlist-deploy.md, scripts/dc-rack-net.sh, scripts/maas-node-power.sh, scripts/site-headend-install.sh (rack-role remainder), modules/maas-vm-host Stage 4
L4 Juju/OpenStack deploy procedure Procedure module L3's READY machines; $DC selector; bundle.yaml + per-DC overlays; Vault root a running independent OpenStack cloud per DC (controller + bundle + Vault) runbooks/dc-dc-phase4-juju-bundle-per-dc.md, runbooks/phase-01..08-*.md (the VR0 template it runs twice), scripts/preflight.sh (entry gate) Stage 5 (+ Stage 6 DR, Stage 7 Designate/COS/Magnum as ADDITIVE sub-procedures riding the same L4 mechanism, per vr0-to-vr1-is-additive -- not separate layers)
L5 Verify/gate Procedure module (cross-cutting) any layer's declared-done state PASS/FAIL + a committed BOM at milestones scripts/opentofu-validate.sh (L0-L2), scripts/cloud-assert.sh/--capture (L4), scripts/preflight.sh (L3->L4 boundary), scripts/geneve-encap-assert.sh (L2/L4 network correctness), scripts/repo-lint.sh (every layer's artifact hygiene) Invoked at every stage close, not owned by one stage

Layer-boundary rule (why this ordering, not invented): each layer's INPUT is the layer below's OUTPUT only -- L1 does not reach into L3, L3 does not dial L0 directly. The one place this rule is currently VIOLATED is exactly the container layer: the inner root (part of L2) dials OUT to a cross-host qemu+ssh provider (opentofu/main.tf:28-29, "a libvirt provider cannot be configured from a resource created in the same apply") because L2's outer half (vvr1_dcN) had to exist as infrastructure BEFORE L2's inner half (planes/nodes) could be declared. Option 1 removes this violation structurally (Section 3) -- it is the single clearest justification, in layer-model terms, for why flattening also SIMPLIFIES the module system, not just the topology.


3. How Option 1's changes map onto the layer model

Option-1 change (from pass0-admin-report.md) Layer What happens to the module/procedure
Flat single tofu root/state (no outer/inner split) L2 The two-root split collapses INTO L2: vvr1_dc0/vvr1_dc1 module calls (outer main.tf:410-623) are DELETED; the inner root's module calls (vr1_dc0_planes, vr1_dc0_node x12, vr1_dc0_opnsense) become direct calls in the single flat root, same module bodies, qemu+ssh provider replaced by the flat root's own qemu:///system (pass0-admin-report.md row 2). The BOOTSTRAP GATE script (site-headend-install.sh --host-nodes) that sat BETWEEN the two roots is eliminated as a step -- L2 becomes a single tofu apply, no cross-host provider dial.
modules/wan-bridge deletion; DC edge WAN attaches directly to the outer uplink NAT L1/L2 boundary modules/wan-bridge (existed only to fix the OBS-3 nesting-egress problem, D-125) is DELETED; the DC OPNsense edge (L1) attaches its WAN leg directly to vr1_dcN_uplink (already an L0/L1 artifact) -- no re-address, same /24 (pass0-admin-report.md row 3).
Per-DC client VM standup (D-138 role, Model-A shape) L1 (new module instance) A NEW small cloudinit-vm instantiation per DC (~4/8192/80, non-hypervisor, expose_nested_virt=false) -- the SAME L1 module type Office1's voffice1 and the DC edges already use, not a new module. Carries the D-138 client role + SEC-028/SEC-029 credential residencies (a procedure/config concern layered on top at L3/L4, not an IaC concern).
(a) cross-DC host-level isolation control L5 (new artifact) Confirmed at the Phase-0 gate (pass0-admin-report.md Section 7a) as a Phase-1 DESIGN ITEM: a NEW vcloud-level nftables isolation artifact + mechanical --check gate + its own SEC-NNN row -- structurally the SAME pattern as scripts/geneve-encap-assert.sh/cloud-assert.sh (an L5 verify module), one layer up from SEC-010's interface-scoped drop. This is a genuinely NEW L5 module, not a re-home of an existing one.
Rack-controller remainder (MAAS --role rack) + D-131 forwarder + artifact-service (.4) placement L3 (open placement, not yet a module change) Currently site-headend-install.sh --host-nodes bootstraps the rack role INSIDE vvr1-dcN; that call site becomes dead code for VR1 once the containment VM is gone. Where the rack role, dc-rack-net.sh's forwarder, and the mirror land (the new L1 client VM vs. vr1-dcN-maas-01 vs. retire-with-evidence) is UNRESOLVED -- pass0-admin-report.md Section 6 item 3, carried into Phase 1 explicitly as an open item, not resolved by this worker.
MAAS region stays on vr1-dcN-maas-01 L2 (no change) Already a flat L2 sibling node VM under D-132's addendum; flattening does not touch it.
Credential residencies (SEC-028/029) migrate to the client VM L3/L4 (procedure, not IaC) The register rows (vm-secret-locations, SEC-026 isolation control) re-point to the new L1 client VM; this is a data/procedure change at the L3-L4 boundary, not a module-body change.
Teardown primitive (virsh destroy vvr1-dcN = site-down) L5/cross-layer D-123's one-command site-down is LOST by flattening (no single containment domain to destroy); Option-1's design must re-earn it as a MODULE-SCOPED group-destroy (a tofu-state-scoped destroy -target set, or a scripted domain-group teardown) -- an L2-layer verify/teardown primitive, owed to Phase 4's module-workflow design (pass0-admin-report.md Section 6 item 7).

D-143 stays a separate axis, threaded through every layer, not a layer of its own. The 10.12->10.13 re-IP touches L0's CIDR inputs, L2's node addressing, and L4's overlay files identically whether or not the container layer exists -- SCOPE-AND-EXECUTION-PLAN.md Section 7 requires the two changes stay distinguishable in the change-set, and the layer model above keeps that distinguishability structural: D-143 is a PARAMETER change at every layer's input; the container-elim is a LAYER-COUNT/BOUNDARY change at L1/L2/L5 specifically.


4. Design principles for the module system (repeatable + Roosevelt-transferable)

  1. Parameterize by site token, not by hardcoded identity. The $DC selector convention (D-119, DOCFIX-151, lib_net_select_dc/lib_hosts_select_dc) already does this for the procedure layer; the IaC layer already does it structurally (no module body names a DC). Option 1's new artifacts (the client VM, the L5 isolation control) MUST take the same token rather than being written DC0-specific and copy-pasted for DC1 -- the exact anti-pattern dc-dc-deployment-workflow.md gap-register item 1 was created to close.
  2. Every module ships its tested harness, no exception (CLAUDE.md "Delivery"; Section 1.3 above). A new L1 client-VM instantiation reuses opentofu-validate.sh's existing coverage (it's the same cloudinit-vm module, no new module body); the NEW L5 isolation control needs its own tests/<name>/run-tests.sh from day one, matching geneve-encap-assert.sh's and SEC-010's own precedent -- a mechanical --check, not a prose claim.
  3. Idempotence at every layer. L0-L2 already get this from tofu apply; L3/L4 procedure modules must stay re-run-safe (the existing MAAS "READY not deployed" handoff and the preflight.sh drift-detection gate already enforce this at the L3/L4 boundary -- D-140's own "OWED BEFORE THE REVIEW" note on P8 drift-detection value is the same principle one layer up).
  4. Clear layer boundaries -- no layer reaches past the one directly below it. Section 2's layer-boundary rule; the container layer is the one place this was violated (L2's inner half dialing a cross-host provider), and removing it is what makes the OTHER four principles easier to hold, not a side effect.
  5. Findings/design stay LOGGED at their true layer, not folded upward. The rack-controller/ D-131/mirror placement question (Section 3) is explicitly NOT resolved here -- it is an L3 procedure-placement decision that this worker's L1/L2 mapping cannot answer without inventing a value, so it stays an open item for the phase administrator and Phase-2's tools workers.
  6. D-140 is a DISTINCT axis from this layer model -- do not conflate. D-140 (docs/design-decisions.md:7781-7822) is PINNED, not ruled, and governs whether the L4 Juju layer ITSELF becomes OpenTofu-managed -- sequenced strictly AFTER "dc0 deploys and is HARDENED ... that deployment method is TESTED." It does not currently govern how L0-L2's IaC modules are structured (that structure already exists and predates D-140). The layer model above is written so that IF D-140 is later adopted, L4 becomes an IaC-module layer too WITHOUT changing L0-L3's shape -- but that is a future trigger, not something this pass proposes now.
  7. Roosevelt-transfer lens applied per layer, not per artifact. L0's node-VM/netem-link shim (no bare-metal analog, dc-dc-deployment-workflow.md:11-14 Section-9 shim register) does NOT transfer; L1's client-VM pattern, L3's MAAS enlist procedure, and L4's Juju/bundle procedure ARE the D-138 Roosevelt bastion analog and the direct pre-Roosevelt bare-metal deliverable (pass0-admin-report.md Option-1 "For" bullet: "rehearsed early ... transfers to the pre-Roosevelt bare-metal test"). The layer model's job for Phase 4 is to keep these two classes visibly separate so the module-workflow design does not present shim-layer work as reusable.

5. Open questions for Phase 4 (not resolved by this worker)

  1. L3 rack-role/D-131/mirror placement (Section 3, pass0-admin-report.md Section 6 item 3) -- unresolved; blocks writing the concrete L3 module change-set until ruled or proposed.
  2. The L5 cross-DC isolation control's exact mechanism -- confirmed IN SCOPE (handling (a)) at the Phase-0 gate, but its concrete design (nftables rule set, --check gate shape, SEC-NNN number) is Phase-1/2 work, not yet started.
  3. The site-down teardown primitive's replacement shape (module-scoped group-destroy) -- named as owed (Section 3, L5/cross-layer row) but not designed here; it is the biggest single piece of NEW tooling this layer model implies beyond re-homing existing module calls.
  4. Biggest open question about the layering itself: whether L3 (MAAS enlist) and L4 (Juju deploy) should be reframed as OpenTofu-orchestrated procedure invocations once D-140 is eventually adopted (Section 4 item 6) -- this pass deliberately keeps them procedure modules now, consistent with D-140's PINNED status, but Phase 4's module-workflow design should name this as a FUTURE layer-model revision trigger rather than silently assuming procedure-only is permanent.