Newer
Older
openstack-caracal-dc-dc / docs / audit / container-elim-pass / pass2-w4-module-decomposition.md

Pass 2 -- W2.4: module decomposition (which tools become procedure-modules; the IaC/procedure layering)

Worker: W2.4 (Phase 2, container-layer-elimination pass). Date: 2026-08-09. Scope: READ-ONLY. Builds on the L0-L5 layer model adopted at Phase 1 (pass1-w4-module-planning.md, consolidated in pass1-admin-report.md Section 5) and the Option-1-confirmed target (pass0-admin-report.md Section 7a). Inputs read in full: SCOPE-AND-EXECUTION-PLAN.md, pass0-admin-report.md, pass1-admin-report.md, pass1-w4-module-planning.md, docs/tool-index.md, plus a live inventory of scripts/, tests/*/, and script headers taken this session (cited by path throughout). D-140 (L4-as-IaC) is PINNED, not ruled -- treated as a future trigger, not folded into this decomposition (per CLAUDE.md instruction and pass1-w4-module-planning.md Section 4 item 6).


1. Method

Every script under scripts/ matching the task's deploy-path filter (preflight, cloud-assert, phase-NN-*, lib-*, dc-*, maas-*, site-*, carve-*, geneve-encap-assert, opentofu-validate) was read (header + role), placed on the L0-L5 model, classed as IaC / procedure / library / gate, and checked against tests/ for a harness (ls tests/*/ cross-referenced by basename; a script with no matching tests/<name>/ dir is flagged NO). 51 scripts matched the filter; opentofu/modules/* (12 IaC modules) are W2.1's domain and are cited here only at the handoff boundary (Section 4), not re-decomposed.


2. Script -> module decomposition table

Legend: kind = IaC (OpenTofu-owned) / Procedure (bash/python driving live MAAS/juju/OS state) / Library (sourced, no independent action) / Gate (procedure, but its sole job is verify, never mutate by default). Layer per Section 2 of pass1-w4-module-planning.md. Harness = tests/<name>/run-tests.sh exists (checked live this session).

Tool Current role Kind Layer Harness?
lib-hosts.sh host/power-address/NIC-map constants, sourced Library L2/L3 (consumed) Selector fn covered by tests/dc-selector/ (DOCFIX-151); the constants table itself is exercised only transitively via every consumer's harness -- no dedicated full-library harness
lib-net.sh CIDR/plane/space constants, sourced Library L0/L2 (consumed) Same as above (tests/dc-selector/)
lib-identity.sh identity constants, sourced Library L2/L3 (consumed) Not independently verified this session -- flag for W2.2 (its dimension)
lib-validate.sh shared exit-contract + emit() lib for scripts/checks/*.sh Library L5 (verify-layer support) tests/lib-validate/ YES
opentofu-validate.sh validates every IaC module standalone + both roots, S1-S3 guards Gate L0-L2 tests/opentofu-validate/ YES
preflight.sh THE single pre-deploy gate; sequences P1-P4 Gate L3->L4 boundary tests/preflight/ YES
cloud-assert.sh behavioral cloud verifier, --capture BOM Gate L4 tests/cloud-assert/ YES
geneve-encap-assert.sh OVN geneve family/tunnel-health gate Gate L2/L4 (network correctness) tests/geneve-encap-assert/ YES
dc-egress-check.sh layered DC-egress probe (read-only) Gate L3 (rack-host) tests/dc-egress-check/ YES
dc-node-v6-verify.sh gate G19: v6 statics + plane forwarding Gate L2/L3 tests/dc-node-v6-verify/ YES
dc-dc-mtu-geneve-budget.sh MTU/geneve budget calculator Gate (calculator) L0 tests/dc-dc-mtu-geneve-budget/ YES
dc-dc-ceph-disk-budget.sh Ceph disk-budget calculator Gate (calculator) L2 tests/dc-dc-ceph-disk-budget/ YES
dc-dc-whole-host-budget.py whole-host RAM/vCPU/disk FIT calculator Gate (calculator) L0/L2 tests/dc-dc-whole-host-budget/ YES
maas-profile-assert.sh proves which region a profile resolves to Gate L3 tests/maas-profile-assert/ YES
site-headend-install.sh installs MAAS region+rack (or rack-only) + LXD on the host it runs on Procedure L1 (install) / L3 (rack-role output) tests/site-headend-install/ YES
dc-rack-net.sh rack bridge legs + D-131 node-DNS forwarder, runs ON the rack host Procedure L3 tests/dc-rack-net/ YES
dc-node-carve.sh v4 NIC/br-ex carve for a DC's nodes Procedure L3 tests/dc-node-carve/ YES
dc-node-v6-carve.py v6 static assignment mirroring the v4 carve Procedure L3 tests/dc-node-v6-carve/ YES
carve-host-interfaces.sh Pattern-A single-host interface carve (VR0) Procedure L3 tests/carve-host-interfaces/ YES
maas-node-power.sh sets power_type=virsh per enlisted machine, MAC-matched Procedure L2->L3 handoff (Section 4) tests/maas-node-power/ YES
maas-role-tags.sh creates + applies per-role MAAS tags the bundle constrains on Procedure L3 tests/maas-role-tags/ YES
maas-region-power-key.sh installs/verifies the per-DC MAAS->libvirt power key Procedure L3 tests/maas-region-power-key/ YES
maas-fabric-prune.sh deletes orphaned auto-fabrics (recurring maintenance) Procedure L3 NO -- no tests/ dir found this session (gap, logged not executed)
maas_fabric_classify.py pure classifier backing the above (no mutation) Library (pure fn) L3 (support) NO -- same gap
dc-region-topology.sh builds/verifies a per-DC MAAS region's fabric/space/subnet/tag topology Procedure L3 tests/dc-region-topology/ YES
dc-plane-ipam.sh site-keyed plane IPAM state incl. v6 carve, D-134's executable gate Procedure (+ gate mode) L2/L3 tests/dc-plane-ipam/ YES
dc-mirror.sh per-DC apt+UCA artifact mirror, runs ON the rack host Procedure L3 tests/dc-mirror/ YES
dc-cache-proxy.sh per-DC apt caching proxy (interim/DC1-strategy artifact path) Procedure L3 tests/dc-cache-proxy/ YES
dc-snap-proxy.sh per-DC snap forward proxy Procedure L3 tests/dc-snap-proxy/ YES
dc-node-etchosts.sh cloudinit-userdata generator for node-local /etc/hosts fix Procedure (generator, feeds L1 cloud-init) L1/L3 boundary tests/dc-node-etchosts/ YES
site-baseleg.sh durable base L3 leg vcloud -> site-local libvirt net Procedure L0/L1 tests/site-baseleg/ YES
site-forward.sh rootless systemd port-forward jumphost -> site VM Procedure L1 (access) tests/site-forward/ YES
site-ssh-config.sh ssh_config Host-alias generator for site VMs Procedure L1 (access) tests/site-ssh-config/ YES
site-tailscale.sh per-DC Tailscale subnet-router install/check Procedure L1 tests/site-tailscale/ YES
render-dc-overlays.py deterministic per-DC bundle-overlay renderer (derive/render split) Procedure L4 tests/render-dc-overlays/ YES
phase-00-maas-standup.sh MAAS fabric/VLAN/subnet/space stand-up (VR0 plane scheme) Procedure L3 (VR0 template L4 reuses per-DC per pass1-w4 Section 1.2) tests/phase-00-maas-standup/ YES
phase-00-teardown-destroy.sh / -release.sh juju-model teardown, VR0-scoped (D-061) Procedure L4/L5 (destroy) tests/phase-00-teardown-d061/ YES
phase-02-vault-preflight.sh Vault preflight for the VR0 template Procedure L4 tests/phase-02/ YES
phase-03-admin-openrc.sh, phase-03-core-verify.sh admin creds + core-service verify Procedure L4 tests/phase-03-adminrc/, tests/phase-03/ YES
phase-04-network-create.sh, -verify.sh, -internal-cert-san-verify.sh network stand-up + verify + cert SAN check Procedure L4 tests/phase-04-create/, tests/phase-04/, tests/phase-04-internal-cert-san/ YES
phase-05-amphora-pipeline.sh, -octavia-verify.sh Octavia amphora image pipeline + verify Procedure L4 tests/phase-05-amphora/, tests/phase-05/ YES
phase-06-bootstrap.sh, -capi-stack.sh, -k8s-bootstrap.sh, -kubeconfig-gate.sh, -mgmt-vm.sh, -net-setup.sh Magnum/CAPI tenant-K8s stand-up chain Procedure L4 (Stage-7 additive, per vr0-to-vr1-is-additive) Each has its own tests/phase-06-*/ dir -- YES
phase-07-conductor-graft.sh Magnum conductor graft step Procedure L4 tests/phase-07-conductor-graft/ YES
dc-dc-rbd-mirror.sh, dc-dc-radosgw-multisite.sh, dc-dc-dr-drill.sh Ceph rbd-mirror / radosgw multisite bootstrap + failover-failback sequences (D-108) Procedure L4 (Stage 6, additive) tests/dc-dc-rbd-mirror/, tests/dc-dc-radosgw-multisite/, tests/dc-dc-dr-drill/ YES

Shape of the decomposition (this filtered set of 51 scripts): 4 library units (2 with a dedicated selector-mechanism harness, 2 flagged for W2.2), 13 gates (verify-only, cross-cutting L5 or layer-scoped calculators), 34 procedure modules (L1-L4), of which 32/34 ship a harness today and 2 do not (maas-fabric-prune.sh / maas_fabric_classify.py -- a pre-existing gap, unrelated to container-elim, logged here because this pass's harness-discipline principle (pass1-w4-module-planning.md Section 4 item 2) would otherwise silently wave it through).


3. What a "procedure module" IS in this repo's terms -- the contract

Grounded entirely in patterns that already exist (pass1-w4-module-planning.md Section 1.2-1.3; docs/tool-index.md; CLAUDE.md "Delivery"), not invented for this pass. A procedure module is:

  1. A named, single-purpose script under scripts/, one file = one job (the repo's existing granularity -- dc-rack-net.sh does rack-net only, dc-mirror.sh does the mirror only; no script is asked to do two jobs at once).
  2. $SITE/$DC-parameterized, never DC-hardcoded -- the DOCFIX-151 convention (lib_net_select_dc "$DC" / lib_hosts_select_dc "$DC"), or a positional <site> argument read by the same underlying selector (every dc-*.sh / maas-*.sh script in Section 2 takes <site> this way). This is the mechanism that lets one module body run DC0 today and DC1 tomorrow without a copy-paste fork -- named explicitly as the anti-pattern to avoid in pass1-w4-module-planning.md Section 4 item 1.
  3. Independently testable with its own harness -- tests/<script-name>/run-tests.sh (docs/tool-index.md:27, "65 scripts with their own tests/<name>/ harness"; CLAUDE.md "Delivery": "every script change ships with its tests/<name>/run-tests.sh harness green"). Section 2's table shows this held for 32/34 procedure modules in the filtered set.
  4. Declared inputs, declared outputs, a stated exit contract. The pattern is explicit in the library layer already (lib-validate.sh's 0/1/2/3/4 PASS/FAIL/HOLD/PASS_PENDING_MANUAL/SKIPPED vocabulary) and echoed in every gate script (geneve-encap-assert.sh: "Exit: 0 all pass | 1 any FAIL | 2 usage/precondition"). A procedure module's inputs are its CLI args + whatever it reads live (never a baked table -- hard rule 2); its outputs are the mutation it performs (or, in check mode, the PASS/FAIL verdict) plus what it leaves behind for the layer above to consume (e.g. dc-node-carve.sh leaves a carved br-ex; dc-node-v6-carve.py needs that br-ex to already exist -- an explicit inter-module input/output chain, not implicit state).
  5. Idempotent, safe to re-run. Universal across the table's procedure modules: check/apply split with apply DRY by default and --commit required to mutate (dc-region-topology.sh, dc-plane-ipam.sh, phase-00-maas-standup.sh); install actions that no-op when already correct (dc-rack-net.sh, dc-mirror.sh, site-tailscale.sh explicitly say "idempotent; safe to re-run").
  6. Composes onto the layer strictly below it, never reaches past it. The layer-boundary rule from pass1-w4-module-planning.md Section 2: an L3 module's live inputs come from L2's output (MAC-pinned node VMs) or L1's output (a reachable MAAS region), never by dialing L0 or L2's infrastructure directly (Section 4 makes this concrete for the container-elim case).
  7. Ships with a changelog entry + a revert, and is repo-lint clean (CLAUDE.md "Delivery" -- applies identically to procedure and IaC modules; not a procedure-only rule, restated here so the contract is complete).

Precedent this contract is built on, not invented against: the $DC-parameterized Stage runbooks (DOCFIX-151, docs/dc-dc-deployment-workflow.md:485-498) already run this exact contract at the stage granularity (one runbook file, $DC-selected, gated by repo-lint + a harness sweep + a changelog + a branch-merge per stage close-out, docs/dc-dc-deployment-workflow.md:424-436). A procedure module is the SAME contract one level down -- the runbook is the composition of several procedure-module invocations in a stated order; the module is the individually-testable unit the runbook calls.


4. The IaC <-> procedure BOUNDARY, stated precisely

The boundary is identity, not orchestration. OpenTofu owns everything up to "a booted libvirt domain exists, with its network identity (MAC per NIC, and any statically-assigned IP) correctly wired to the right plane bridges." The procedure layer begins at the first live dial into that object -- an SSH session, a MAAS API call, a libvirt power query -- and by design never reads OpenTofu state. It re-derives everything it needs from LIVE, independently observable identity: MAC address, hostname, or a fresh API/SSH probe. This is not a convenience; it is a stated design rule with its own incident history:

lib-hosts.sh:6-11 -- "WHY hostname-keyed (NOT system_id-keyed): MAAS system_ids are minted fresh on every (re-)enrollment... The stable identities are the hostname and the libvirt domain name, so every map here keys on hostname and the live system_id is resolved AT RUNTIME."

dc-node-v6-carve.py -- "EVERYTHING IS DERIVED FROM LIVE STATE -- there is no plane table here (hard rule 2): site membership <- the MAAS tag openstack-."

maas-node-power.sh:8-9 -- "Domains are matched to MAAS machines BY MAC ADDRESS -- never by name, because MAAS assigns its own random hostnames at enlistment."

Concretely, at each L1/L2 -> L3 handoff point in Section 2's table:

  • Node VMs (L2, modules/node-vm) -> MAAS commissioning (L3): the handoff artifact is the MAC address baked into the libvirt domain XML (IaC output) that MAAS discovers at commissioning and maas-node-power.sh later matches power config against (procedure input). No file, no state read, no data pipe crosses the boundary -- only a MAC address that both sides independently observe.
  • Client VM / edges / voffice1 (L1, modules/cloudinit-vm) -> install procedures (L1/L3): the handoff artifact is a reachable IP + a cloud-init-seeded initial state (IaC output); site-headend-install.sh / dc-rack-net.sh / site-tailscale.sh then dial in over SSH and configure from there, keyed by hostname/IP resolved live (lib-hosts.sh), never by reading opentofu/*.tfstate.
  • DC edge WAN (L1) -> egress verification (L5): dc-egress-check.sh probes the live path end to end; it does not consult tofu at all.

Why this rule matters here specifically -- it is the diagnostic for the container-elim's core defect. pass1-w4-module-planning.md Section 2 already names the ONE place this boundary rule is currently violated: the inner root (opentofu/vr1-dc0-substrate/main.tf) dials OUT to a qemu+ssh provider INTO the outer root's own vvr1_dc0 output, because "a libvirt provider cannot be configured from a resource created in the same apply" (opentofu/main.tf:28-29). That is IaC reaching into IaC across a live-dial boundary that should only ever be crossed by a procedure module -- L2's inner half is doing L2-to-L2 what only L2-to-L3 is supposed to do (dial a live object by observed identity, not by direct provider coupling). Option 1 removes the violation structurally: with one flat root, L2 is IaC end to end (module bodies only, no cross-host provider dial), and the FIRST live dial into anything L2 produced is L3's MAAS commissioning -- exactly where the boundary rule says it should be. This is the single clearest argument, in module-decomposition terms, for why flattening also simplifies the module system and not merely the topology (echoing pass1-w4-module-planning.md Section 2's own framing).

Boundary summary table:

From (IaC, L0-L2) To (Procedure, L3+) Crossing artifact Never crosses
modules/node-vm (L2) MAAS commissioning (L3) MAC address (observed both sides) tofu state, module output vars
modules/cloudinit-vm (L1) site-headend-install.sh / access scripts (L1/L3) IP + cloud-init seed tofu state
modules/dc-planes (L2) dc-plane-ipam.sh / dc-node-v6-carve.py (L3) plane CIDR/VLAN (both re-derive from live MAAS, per D-134/D-139) tofu state
Any IaC module Any L5 gate the live object itself, probed independently tofu state (opentofu-validate.sh is the ONE gate that DOES read tofu directly -- because it IS the IaC-layer's own gate, not a procedure-layer consumer)

5. Option-1's new tools, mapped onto the decomposition

New tool (Option 1) Module kind Layer Placement / what's actually new
Cross-DC host-isolation control (the "(a)" control, pass0-admin-report.md Section 7a / pass1-admin-report.md Section 3) Gate (procedure) -- confirmed in pass1-admin-report.md Section 3: "SEC-010's actual pattern -- a script-installed nftables control + --check + harness ... NOT an OpenTofu module" L5, new artifact A genuinely NEW script + tests/<name>/run-tests.sh + SEC-NNN row, on the geneve-encap-assert.sh/SEC-010 pattern one layer up. Recommended stage home: Stage 1 (host-scoped, not per-DC-apply-scoped), re-verified at each per-DC apply's close and at Stage-5 live verify. This is the one wholly new procedure-module BODY this pass's decomposition requires -- everything else re-targets existing bodies.
Client-VM standup (vr1-dcN-client, D-138 role) Split: IaC instantiation + procedure configuration L1 (IaC) for the VM itself; L1/L3 (procedure) for what runs on it The VM is a NEW instance of the EXISTING modules/cloudinit-vm module type (same body Office1's voffice1 and the DC edges already use, per pass1-w4-module-planning.md Section 3) -- no new IaC module, so it rides opentofu-validate.sh's existing coverage. What lands ON it once booted is EXISTING procedure modules RE-TARGETED, not new bodies: site-headend-install.sh --role rack (rack-controller portion, sans the now-dead --host-nodes bootstrap-gate duty), and -- pending the OPEN Section-6-item-3 placement ruling from Phase 0/1 -- dc-rack-net.sh (D-131 forwarder) and the artifact-mirror scripts (dc-mirror.sh/dc-cache-proxy.sh/dc-snap-proxy.sh). This is a call-site change (which host these scripts SSH into), not a module-body change -- consistent with the contract's "declared inputs" (the target host is an input, not baked in).
Teardown-primitive (module/root-scoped group-destroy, re-earning D-122's one-command site-down) Procedure (new) L2/L5 boundary -- a procedure module that WRAPS an IaC destroy (invokes tofu destroy -target=... against the flat root's DC-scoped module set, or drives a scripted virsh destroy loop over the roster lib-hosts.sh derives) The second wholly new procedure-module BODY. Does not exist today (pass1-admin-report.md Section 6 item 1 / Section 6 item 6 names it as owed, distinct from the emergency virsh destroy loop). Its exact shape (root-scoped -target set vs. scripted domain-group destroy) depends on the OPEN root-topology fork (pass1-admin-report.md Section 7 item 1, W2.1's domain) -- this worker does not resolve that fork, only places the resulting artifact's layer. Must carry the same contract as Section 3: $DC-parameterized, its own harness, idempotent-on-already-torn-down state.
Re-homed rack-controller / D-131 forwarder / artifact-service (.4) No new module -- re-target of existing L3 procedure modules L3, placement OPEN Confirmed by both pass0-admin-report.md Section 6 item 3 and pass1-w4-module-planning.md Section 3: this is a call-site/placement decision (client VM vs. vr1-dcN-maas-01 vs. retire-with-evidence), not a module-body change. dc-rack-net.sh, site-headend-install.sh --role rack, and the mirror/proxy scripts already take <site> as a parameter and run over SSH to whatever host is named -- the decomposition table in Section 2 shows every one of them already meets the procedure-module contract (Section 3) independent of which host wins. This worker's contribution is confirming there is no tooling gap here -- the gap is a ruling, not a build.

Net new procedure-module bodies this pass's decomposition surfaces: exactly TWO -- the (a) cross-DC isolation gate and the teardown-primitive. Everything else Option 1 needs is either (i) a new instance of an existing IaC module type (the client VM), or (ii) an existing procedure module re-targeted at a new host via its existing <site>/host-argument parameterization (no body change). This matches pass1-w4-module-planning.md Section 1's framing: "the container-elim does not need to invent module mechanics, only re-home."


6. Open items (not resolved by this worker; feed the Phase-2 administrator / Phase 4)

  1. lib-identity.sh's harness status -- not independently verified this session; flagged for W2.2 (its assigned dimension, lib-hosts/lib-net containment-keyed values).
  2. maas-fabric-prune.sh / maas_fabric_classify.py have no harness -- pre-existing gap, unrelated to container-elim causally, but the harness-discipline principle this pass restates (Section 3 item 3) would be inconsistent if silently excluded; logged for the Phase-2 administrator to route (build-a-harness item vs. accept-as-a-named-exception).
  3. The teardown-primitive's exact shape depends on W2.1's root-topology design (the OPEN fork from pass1-admin-report.md Section 7 item 1) -- this worker places its LAYER, not its final form. Feed forward to whichever worker/administrator resolves the fork.
  4. The rack-controller-remainder placement ruling (Section 5 row 4) is the actual blocker for writing the FINAL L3 change-set (which scripts get new call-site arguments); this worker confirms readiness (no tooling gap) but cannot pick a value (hard rule 2).
  5. opentofu/modules/*'s own decomposition (W2.1's domain) was consulted only at the boundary (Section 4) -- this doc does not re-derive the IaC-module inventory or recommend root-splitting; that is W2.1's report to write.

7. Verification note

Every script's role/kind/layer classification in Section 2 is grounded in a direct header read this session (paths cited); harness presence was checked live via ls tests/*/ cross-referenced against each script's basename, not assumed from the tool-index's aggregate counts. The two "NO harness" flags and the lib-identity.sh open item are the only claims in this doc not independently confirmed against a second source. No mutation performed; no live cloud state queried; findings are LOGGED only, per the pass's read-only-planning charter.