DC-DC Phase 2 -- OpenTofu builds each DC substrate (Stage 3)

From Office1, script each DC's "physical" layer into existence: the node-VM libvirt domains MAAS will discover and deploy, the DC's plane networks, and its OPNsense edge -- then register the DC's libvirt host to that DC's own MAAS rack controller as a virsh VM-host. This runbook builds vr1-dc0 (VR1's FIRST DC -- the build target this pass). vr1-dc1 (VR1's SECOND DC) is explicitly gated (see the callout below) and is NOT part of this session's run until that gate clears -- do not duplicate these steps for vr1-dc1 with invented values in the meantime.

!!! NAMING -- this runbook was written pre-D-119 and has since been reconciled to the region-qualified namespace. Its original prose spoke the bare dcN namespace (dc1 = the build target, dc2 = the second DC). D-119 REJECTS bare dcN for VR1 datacenters -- the token dc0 already means VR0's live DC in scripts/lib-net.sh, so an unqualified dcN is a latent off-by-one. The mapping applied throughout this file is: runbook "DC1" -> vr1-dc0 (already live in opentofu/main.tf as the vr1_dc0_* modules), runbook "DC2" -> vr1-dc1 (still gated). All object names, module labels, plan files, and network/pool names below are region-qualified to match main.tf. HAZARD: following ANY residual bare-dcN literal in a copy of this runbook (or in muscle memory) literally reintroduces the exact dc0/dc1 cross-cloud collision D-119 was written to delete -- region-qualify it or stop. (D-119.)

Governing docs: docs/dc-dc-buildout-design.md Section 4 "Phase 2" (the goal/build/gate this runbook satisfies), Section 9 (shim register -- the node-VM creation module has NO Roosevelt analog, see the callout below), and Section 6 (WAN simulation / netem, same-metro lean). docs/dc-dc-deployment-workflow.md Stage 3 row and the Tooling gap register (items 1, 4, 11 govern this stage directly). opentofu/README.md and opentofu/templates/README.md for exact module/token status -- read both before touching any .tf file or the OPNsense template.

Decisions this runbook owns: D-103 (the OpenTofu/MAAS/Juju lifecycle seam -- OpenTofu creates, MAAS discovers/commissions, never composes), D-100 (the dark-fiber mesh + per-site OPNsense edge, and the netem WAN-simulation mechanism riding on it).

!!! Depends on Stage 2 (Office1 headend standup). This stage's MAAS registration step needs a REACHABLE MAAS region controller (Stage 2's gate) and Stage 1's OpenTofu-reaches-vcloud-libvirt gate already closed. If Stage 2 has not completed when this session runs, STOP at Step 9 (MAAS registration) -- everything through Step 8 (planes/pool/edge/node-VM creation at the libvirt layer) does not itself require MAAS and may proceed, but do not fabricate a maas_api_url/maas_api_key pair to push past that dependency. Confirm Stage 2's actual state in docs/dc-dc-deployment-workflow.md before starting.

!!! vr1-dc1 is explicitly gated -- by SEQUENCING, not by a missing literal. The earlier rationale ("NetBox has not assigned the supernet") is STALE: D-115 ALREADY assigned it -- vr1-dc1 gets 10.12.64.0/19 (moved inside the Cloud /16, colliding with nothing vr1-dc0 occupies), and that carve is imported to office1-netbox. So the blocker is no longer a missing address; vr1-dc1 stays out of scope THIS pass purely because we build one DC at a time and vr1-dc0 is the target. opentofu/main.tf's vr1_dc1_planes block is still commented out (its vr1_dc1_storage and the three mesh legs ARE wired, being address-independent). Do NOT uncomment the planes block, do NOT add vr1-dc1 build steps, and do NOT run this runbook's steps against vr1-dc1 this pass. Re-run the whole runbook for vr1-dc1 when it is sequenced. This is a hard gate, re-stated in the GATE section. NB: main.tf's commented vr1_dc1_planes block still carries the OLD "wait for NetBox to assign D-101's supernet" comment -- that comment is now stale per D-115; flagged as a divergence to fix under the main.tf owner, out of scope for this doc-only pass.

!!! Node-VM creation has NO Roosevelt analog (Section 9 shim register). modules/node-vm (the blank-disk, PXE-boot libvirt domain this stage calls) is a virtual-only stand-in for what, in Roosevelt, is physical racking plus BMC enlistment -- MAAS enlists hardware that already exists there. Do not treat this module, or the act of Terraform-creating "node hardware," as reusable production IaC. Everything from MAAS enlistment onward (commission/deploy, Juju bundle) IS Roosevelt-transferable; this one step is not.

!!! Node sizing is now RULED (D-121, Option C, 2026-07-15). modules/node-vm's memory_mib/vcpu/disk_size_bytes still have no module defaults by design, but the VALUES to pass are no longer open: D-121 ruled the per-DC layout as 3 control + 2 compute + 3 storage (8 node VMs) at the sizes wired into Step 6. This was the exact decision that used to block Step 6; it is now UNBLOCKED. Still open (one sub-ruling D-121 notes): the vault-HA backend fork -- but that is a Stage-5 bundle concern, NOT a Step-6 blocker (node layout is fully ruled). Do not invent any OTHER node spec; use D-121's numbers exactly.

!!! netem parameters are still an unruled D-100 sub-item (gap #11). This stage's own gate text says "netem parameters applied and measured," but the exact latency/jitter/loss/rate values do not exist as a ruled decision -- only the qualitative buildout-design Section 6 lean exists: "same-metro dark fiber... low single-digit ms, jitter-capable, jumbo- capable." Step 11 below applies ONLY that qualitative lean as an explicitly-labeled PLACEHOLDER profile (recorded as such, with a citation to Section 6), pending a firmer numeric ruling. Do not write a specific number (e.g. "delay 3ms") into netem_args and represent it as measured or ruled -- if you pick a number to instantiate the mechanism for real, log it as exactly what it is: a placeholder pending D-100's still-open sub-item, not a final value.

!!! OPNsense prep tools. scripts/opnsense-prep-image.sh requires bunzip2 + qemu-img on PATH. The ISO9660 builder requirement is GONE (2026-07-13): the config-ISO builder (opnsense-build-config-iso.sh, formerly under scripts/) is DELETED (D-112 -- the Importer can never fire on a nano image, so the ISO was never read; D-113(a2) -- config is done over the REST API). genisoimage/xorriso are no longer needed and are no longer installed by scripts/prereqs/install-image-tools.sh. Step 2 checks the remaining tools.

Sequence (vr1-dc0; vr1-dc1 remains gated -- see callout above)

Honest achievable scope for a first run is Steps 1-8 (libvirt-layer objects + the edge). Step 9 onward is gated on vr1-dc0's own MAAS rack controller existing -- a Stage-4 gap (per-DC rack-controller source is not yet stood up); do not treat Steps 9-12 as runnable until it is.

1.  Pre-flight: confirm Stage 1 + Stage 2 gates closed       (read-only)
2.  Confirm OPNsense prep-tool prereqs on the vcloud host    (read-only)
3.  Confirm/record decisions this stage depends on (node
    sizing = D-121 RESOLVED; edge/WAN/LAN = D-122 RESOLVED;
    MAAS zone/pool; netem parameters) -- STOP on any open     (read-only)
4.  Prepare vr1-dc0's OPNsense base image (prep only; edge
    config is done over the REST API -- D-113(a2))            [MUTATION: host-local files, gated]
5.  Wire modules/opnsense-edge for vr1-dc0 into main.tf       [repo change, gated]
    (prereq: build the vr1-dc0-wan uplink, office1-wan shape)
6.  Wire the 8 modules/node-vm calls for vr1-dc0's node VMs
    into main.tf (D-121 Option C sizes)                       [repo change, gated]
7.  tofu init / validate / plan -- review before apply         (read-only)
8.  tofu apply -- vr1-dc0 OPNsense edge + node-VM domains      [MUTATION: creates libvirt objects, gated]
9.  Wire + apply modules/maas-vm-host -- register vr1-dc0's
    libvirt host with vr1-dc0's MAAS rack controller           [MUTATION: MAAS + repo change, gated]
10. Verify MAAS discovers the node VMs                          (read-only)
11. Wire + apply modules/netem-link on vr1-dc0's mesh legs,
    using the Section-6 placeholder lean, explicitly flagged     [MUTATION: gated]
12. Verify edge routing + simulated ISP uplink + netem applied   (read-only)
    -> EXIT GATE -> Stage 4 (per-DC, once vr1-dc1 clears its own gate)

D-123 MODEL B RESHAPE (2026-07-16) -- the single-root sequence above is REGROUPED; read this

The operator ruled D-123 = Model B: the vr1-dc0 node fleet nests INSIDE the containment VM vvr1-dc0 (single-object virsh destroy vvr1-dc0 = site-down). A libvirt provider cannot target a VM created in the same apply, so the substrate is now TWO ROOTS with a bootstrap gate between. The Steps above still describe the WORK; they regroup under the new roots + a new apply order:

• A. OUTER apply (opentofu/, vcloud) -- creates + sizes vvr1-dc0 (~416 GiB, expose_nested_virt=true, single transit leg) + the transit. Steps 1-3 (preflight/decisions) apply here. [MUTATION: gated]
• B. BOOTSTRAP GATE on vvr1-dc0 (after it boots): scripts/site-headend-install.sh --role rack --host-nodes --region-url ... --enroll-secret-file ... -- enroll the rack to the Office1 region AND make vvr1-dc0 a nested libvirt host (qemu/libvirt, kvm nested=1, inner pool, AppArmor, the SEC-010 transit FORWARD-drop). Prep the OPNsense base ON vvr1-dc0 (Step 4 -- run opnsense-prep-image.sh there). GATE: site-headend-install.sh --role rack --host-nodes --check must pass (incl. SEC-010).
• C. INNER apply (opentofu/vr1-dc0-substrate/, provider = qemu+ssh to vvr1-dc0 over the transit, R-5) -- the 6 planes + vr1-dc0-wan + edge + inner pool + 9 node VMs. Steps 5-6 (wire edge/nodes) are ALREADY authored in this inner root; Step 7 (init/validate/plan) + Step 8 (apply) run HERE, against vvr1-dc0's libvirt, NOT vcloud. [MUTATION: gated]
• D. maas-vm-host (Step 9) -- register vvr1-dc0's OWN (inner, local qemu:///system) virsh to the Office1 REGION; MAAS discovers the inner node VMs (Step 10). Still DEFERRED (DOCFIX-179).
• E. netem (Step 11) -- UNCHANGED: runs on the vcloud-level mesh bridges (virbr7), outer.

Node count is R-3 = 3 control + 2 compute + 4 storage = 9/DC (Step 6's "8" is superseded). The Model A single-root layout is preserved as a revert (git tag model-a-fallback + docs/model-a-fallback-plan.md). Sizing derived by scripts/dc-dc-whole-host-budget.py (870/1024 GiB, FIT). SEC-010 DC-LOCAL is enforced by the bootstrap's transit FORWARD-drop (NOT a global ip_forward=0), kept interface-scoped -- never globalize it (D-125 br_netfilter constraint: bridged WAN frames traverse L3 FORWARD). See docs/changelog-20260716-review-sweep-phaseC.md.

D-125 (bridge-in per-DC ISP egress -- closes OBS-3). Under Model B the inner vr1-dc0-wan was a NAT with no egress (its host vvr1-dc0 is transit-only). Fix, folded into the roots above:

• A (OUTER) ALSO creates the vcloud ISP NAT vr1-dc0-uplink (HELD var.vr1_dc0_uplink_cidr -- assign the /24 in office1-netbox before tofu plan) + a 2nd IP-less uplink NIC + the br-vr1-dc0-wan netplan bridge on vvr1-dc0.
• B (BOOTSTRAP) --check ALSO verifies br-vr1-dc0-wan exists with uplink enslaved (fail-open guard).
• C (INNER) vr1-dc0-wan is now a BRIDGE (modules/wan-bridge) onto br-vr1-dc0-wan; the OPNsense WAN takes a static address in the vcloud ISP /24 (edge config, D-113 -- HELD).
• DEPLOY-TIME GATE (Step 12): before trusting the OPNsense chain, isolation-test egress -- a throwaway guest on br-vr1-dc0-wan must get a vcloud-ISP address + ping out. FAIL => revert to the D-125 double-NAT fallback (not a redesign). See D-125 + docs/changelog-20260716-review-sweep-phaseC2-D125.md.

Step 1 -- Pre-flight: confirm Stage 1 + Stage 2 gates closed (READ-ONLY)

CHECK -- Stage 1 (vcloud host prep) closed?

virsh net-list --all
virsh pool-list --all

Expect: vr1-dc0's six plane networks, the three mesh-link networks (vr1-dc0-vr1-dc1, vr1-dc0-office1, vr1-dc1-office1 -- the region- qualified names now in main.tf), and the vr1-dc0/office1 storage pools, all active -- Stage 1's Step 11 verification. If any are missing, Stage 1 is not actually done regardless of what docs/dc-dc-deployment-workflow.md says -- stop and reconcile there first.

CHECK -- Stage 2 (Office1 headend) closed? Confirm against docs/dc-dc-deployment-workflow.md's Stage 2 row and, if a Stage 2 runbook exists by the time you run this, its own exit gate. At minimum, confirm:

# from Office1 (or wherever this session's tofu apply will run from)
tofu -chdir=opentofu version
curl -sI "$MAAS_API_URL" 2>/dev/null | head -1   # MAAS_API_URL: this session's real value, not invented

Expect a real MAAS region controller reachable at a real URL. If Stage 2 is not done, everything through Step 8 below (libvirt-layer objects only) may still proceed since it needs no MAAS reach; Step 9 onward (MAAS registration) is a hard stop until Stage 2 closes -- do not substitute a placeholder MAAS endpoint to push past it.

GATE: Stage 1 objects present and active; Stage 2's MAAS reachability either confirmed, or explicitly recorded as NOT YET (in which case this session's scope for today is Steps 1-8 only).

Step 2 -- Confirm OPNsense prep-tool prerequisites on the vcloud host (READ-ONLY)

CHECK

for bin in bunzip2 qemu-img genisoimage xorriso curl wget; do
  command -v "$bin" >/dev/null 2>&1 && echo "present: $bin" || echo "absent:  $bin"
done

Required: bunzip2 + qemu-img (for scripts/opnsense-prep-image.sh), and (The genisoimage/xorriso requirement is GONE -- the config-ISO path is deleted.) plus curl or wget for the image fetch. If any required tool is absent, install it on the vcloud host (an OS-specific package install, out of scope to prescribe here) before Step 4 -- do not skip the tool and hand-build the ISO/qcow2 by some improvised method not covered by this repo's tested scripts.

GATE: all required tools present (or installed and re-checked).

Step 3 -- Confirm/record still-open decisions this stage depends on (READ-ONLY)

Before wiring anything into main.tf, walk this list. Two of the items that used to block this stage are now RULED (D-121, D-122) and are recorded as RESOLVED below; the genuinely-open ones remain STOPs. Record each plainly rather than inventing a value to keep moving:

• Node sizing (memory_mib/vcpu/disk_size_bytes and count for modules/node-vm) -- RESOLVED (D-121, Option C, 2026-07-15). Per DC: 3 control @ 16 vCPU / 65536 MiB / 150 GiB; 2 compute @ 12 vCPU / 49152 MiB / 100 GiB; 3 storage @ 8 vCPU / 24576 MiB / 550 GiB -- 8 node VMs total. Step 6 wires exactly these. Open sub-ruling D-121 still carries (the vault-HA backend fork) does NOT block Step 6 -- it is a Stage-5 bundle concern.
• Edge sizing / WAN uplink / LAN role -- RESOLVED (D-122, 2026-07-15). The DC edge follows the proven Office1 pattern: memory_mib = 2048, vcpu = 2, nano base image (NO disk_size_bytes, like office1_opnsense). The edge is 2-NIC (baremetal-matched): wan_network_name is a dedicated per-site ISP uplink vr1-dc0-wan (172.30.2.0/24, D-115) built on the office1-wan pattern -- NOT a mesh leg (dark fiber is East-West only). lan_network_name is provider-public -- the six planes are routed by the fabric (OVN/OpenStack on isolated-L2 segments), so the edge is only the external gateway on provider-public (D-100), not an inter-plane router. See Step 5. PREREQUISITES: build vr1-dc0-wan (as a tofu module) and register 172.30.2.0/24 in office1-netbox (not yet loaded).
• MAAS zone/pool for vr1-dc0's VM-host registration (modules/maas-vm-host's zone/pool variables) -- both are optional (MAAS computes them if left null, confirmed safe per opentofu/README.md's audit pass), so this is NOT necessarily a blocker; record whichever real zone/pool name the operator wants, or confirm the null no-op is intentional. Affects Step 9.
• vr1-dc0's real power_address (the virsh URI MAAS uses to reach the vcloud host) -- must be measured this session (or carried from Stage 1's libvirt_uri tfvar, if it is the SAME endpoint -- confirm, do not assume identical per modules/maas-vm-host's own variable note). Blocks Step 9.
• vr1-dc0's MAAS rack controller -- a Stage-4 gap: the per-DC rack-controller source is not yet stood up. This is the reason honest Step-9 scope is BLOCKED and this pass's real reach is Steps 1-8 (see the GATE section). Blocks Steps 9-12.
• Edge post-boot interface mapping (formerly the {{WAN_IF}}/{{LAN_IF}} config.xml tokens) -- NO LONGER a config-render blocker. There is no config.xml render (D-112/D-113(a2)); the real vtnetN mapping is measured AFTER boot over SSH (ifconfig), then applied over the REST API. See the REPLACEMENT chain in Step 4. Nothing here blocks wiring Step 5.
• netem parameters -- still an unruled D-100 sub-item (gap #11); Step 11 uses the Section-6 qualitative placeholder only, explicitly labeled as such. Not a hard blocker for Step 11 (the mechanism can be exercised with a labeled placeholder), but IS a blocker for calling this stage's netem gate "final" -- see the GATE section.

GATE: every item above is either resolved with a real, measured/ruled value, or explicitly recorded as NOT YET with the corresponding downstream step named as blocked. Do not proceed past a step whose input is still "NOT YET" by inventing one.

Step 4 -- Prepare vr1-dc0's OPNsense base image [MUTATION: host-local files, gated]

THE EDGE-CONFIG PATH (settled 2026-07-13 -- this IS the path)

This step is now prep the base image only. Edge configuration is NOT baked into an image or seeded from an ISO -- it is done over the REST API after boot, the path proven end to end on the live Office1 edge (2026-07-12/13). The two deleted approaches, and why:

1. The config ISO was INERT -- it could never be read. Per D-112 (root-caused from upstream source): opnsense-importer -b probes for a read-only root. On INSTALLER media the probe fails and the importer scans attached media. On our PRE-INSTALLED NANO image the root is writable AND a factory /conf/config.xml already exists, so BOTH conditions hold and it bootstrap_and_exit 0s without enumerating a single device. The Configuration Importer can NEVER fire on a nano image, by design -- so the ISO builder is DELETED and the module no longer takes a config_iso_path.
2. Full-config.xml rendering is SUPERSEDED by D-113(a2) (ruled 2026-07-13): edge config is done over the REST API (scripts/opnsense-api.sh), proven end to end -- read AND write -- against the live Office1 edge on 2026-07-13. Hand-authoring the appliance's GUI-owned XML caused DOCFIX-191 (management lockout), DOCFIX-192 (dead console) and DOCFIX-193 (no DHCP, and an ISC <dhcpd> block that would have been inert against a Kea backend). None of those bugs is expressible through the API. The renderer is DELETED.

There is no WAN_IF/LAN_IF chicken-and-egg. It only ever existed because we tried to seed a full config before first boot. The D-112(c) bootstrap (boot on factory defaults -> reach the console -> enable SSH + install the key -> configure over the network) measures the real vtnetN mapping after boot, where it is knowable -- nothing to guess before Step 8.

What vr1-dc0's edge does (the proven Office1 path): prep the image (this step) -> Step 5 wires the domain -> Step 8 boots it -> D-112(c) console bootstrap -> configure over SSH + the REST API (the REPLACEMENT chain below). See docs/changelog-20260712-office1-opnsense-edge-build.md, docs/changelog-20260713-opnsense-api-proven.md, and docs/changelog-20260713-opnsense-api-write-proven.md.

This step produces one plain file on the vcloud host filesystem -- the prepped base image -- no libvirt or MAAS object is created yet. Run from the vcloud host (or wherever this script executes with reach to write there).

MUTATION -- prep the base image (needs Step 2's tools)

bash scripts/opnsense-prep-image.sh   # see the script's own header for its exact args/output path

Expect: a decompressed, qcow2-converted, resized OPNsense nano image at a real output path -- record that path, it feeds Step 5's base_volume_path. That single path is this step's ONLY output. There is no config.xml render and no config-ISO build -- both are deleted (D-112 / D-113(a2), see the box above).

The REPLACEMENT chain (proven on Office1, 2026-07-12/13)

No config.xml anywhere. After Step 5 wires the domain and Step 8 boots it on FACTORY DEFAULTS:

1. Console bootstrap (D-112(c)) -- reach the serial console, enable SSH, install the service public key. This is the only step that needs the console; everything after it is scripted.
2. Mint the API key -- no GUI click:

   export OPNSENSE_SSH_KEY=...   # the service private key
   bash scripts/opnsense-bootstrap-apikey.sh <edge-lan-ip> <creds-out-file>

3. Configure over REST -- interfaces, DHCP, firewall:

   export OPNSENSE_API_HOST=<edge-lan-ip> OPNSENSE_API_CREDS=<creds-out-file>
   bash scripts/opnsense-api.sh GET  core/firmware/status        # auth smoke test
   bash scripts/opnsense-api.sh POST kea/dhcpv4/set_subnet/<uuid> '<json>'
   bash scripts/opnsense-api.sh POST kea/service/reconfigure '{}'

The real vtnetN interface mapping is measured AFTER boot (ifconfig over SSH), where it is knowable -- there is no interface value to guess before the domain exists.

GATE: the prepped base image path recorded as a real host-local file path (it feeds Step 5's base_volume_path). Nothing else is produced or pending here -- no config-ISO path exists to record.

Step 5 -- Wire modules/opnsense-edge for vr1-dc0 into main.tf [repo change, gated]

Proceed once Step 4 produced a real base_volume_path. The edge SHAPE is ruled (D-122): it mirrors the already-applied office1_opnsense block in main.tf -- memory_mib = 2048, vcpu = 2, nano image, no disk_size_bytes (the disk is a direct copy of the prepped nano; the module does not accept a size input -- DOCFIX-189), and no config_iso_path (the module dropped it -- D-112). There is nothing to invent for the edge specs themselves; the two values that ARE still per-site (the base image path, and the two network names) are called out below.

PREREQUISITE -- build the vr1-dc0-wan uplink first. D-122 gives the DC its OWN dedicated L3 "ISP" uplink, NOT a mesh leg (dark fiber is East-West replication only). Build a per-site NAT'd /24 uplink network vr1-dc0-wan on the office1-wan pattern (the same mechanical shape office1-wan was created in -- a documented D-103 debt to formalize into a module later; today it is a literal network name the edge's wan_network_name points at). This is a prerequisite to wiring the block below with a real wan_network_name.

LAN side -- RESOLVED (D-122, operator ruling 2026-07-15): provider-public. The edge is 2-NIC (WAN + LAN), matching the baremetal target: the six planes are routed by the fabric (the dc-planes segments are isolated L2; OVN/OpenStack own their L3, not the edge), so the edge is NOT an inter-plane router -- only the external boundary. Its LAN attaches to provider-public (10.12.4.0/22), where the edge is the upstream/external gateway the provider network exits through (FIP/SNAT egress + GUA injection, D-100 br-ex/provider model). metal-admin and the other planes reach external via fabric routing, not a direct edge leg. So lan_network_name = the vr1_dc0_planes provider-public output (confirm the exact output name in modules/dc-planes). No new site-internal network module is needed (this SUPERSEDES the earlier "build an office1-network analog" flag).

MUTATION -- edit opentofu/main.tf, adding a block shaped like office1_opnsense (main.tf, the applied reference):

module "vr1_dc0_opnsense" {
  source           = "./modules/opnsense-edge"
  vm_name          = "vr1-dc0-opnsense"
  memory_mib       = 2048                              # D-122: Office1 pattern
  vcpu             = 2                                  # D-122: Office1 pattern
  pool_name        = module.vr1_dc0_storage.pool_name
  # No disk_size_bytes: nano direct-copy, size from opnsense-prep-image.sh's GROW (DOCFIX-189).
  base_volume_path = "<Step 4's real prepped-image path>"
  lan_network_name = <the vr1_dc0_planes provider-public output -- D-122: edge is the external gateway on provider-public; confirm the output name in modules/dc-planes>
  wan_network_name = "vr1-dc0-wan"                     # D-122: dedicated per-site ISP uplink, office1-wan pattern (build first, promote to a tofu module)
}

Before commit: the base image path (Step 4) must be real, vr1-dc0-wan must be built (as a tofu module -- D-122) and 172.30.2.0/24 registered in office1-netbox (D-115, not yet loaded), and lan_network_name bound to the real provider-public plane output. memory_mib/vcpu/the nano-no-disk shape and the 2-NIC WAN/LAN model are ruled (D-122), not invented.

GATE: opentofu/main.tf diff shows exactly one new module "vr1_dc0_opnsense" block, no disk_size_bytes and no config_iso_path line, every argument a real value (with vr1-dc0-wan and the site-internal LAN network genuinely built), nothing else changed.

Step 6 -- Wire modules/node-vm calls for vr1-dc0's node VMs into main.tf [repo change, gated]

Node sizing is RULED (D-121, Option C). Wire 8 module "vr1_dc0_node_*" blocks in three role-sizes:

disk_size_bytes is BYTES: express each as GiB x 1024^3 in HCL -- 150 * 1024 * 1024 * 1024, 100 * 1024 * 1024 * 1024, 550 * 1024 * 1024 * 1024 (the module's disk_size_bytes expects bytes; the GROW/qcow2 layer is thin-provisioned so this is a max, not a preallocation).

Why 3 control (context): this layout exists to carry the Stage-5 HA scale-up (D-121). The bundle's 14 currently-single-unit services scale 1 -> 3: the 11 hacluster+VIP API apps (keystone, glance, neutron-api, nova-cloud-controller, placement, cinder, openstack-dashboard, octavia, barbican, magnum, designate) plus ceph-radosgw, rabbitmq-server, and vault. Those 3-unit control-plane services LIVE on the 3 control nodes -- alongside the services that are ALREADY at 3 and do not change (mysql-innodb-cluster, ovn-central, ceph-mon). Meanwhile ceph-osd = 3 units on the 3 storage nodes and nova-compute = 2 units on the 2 compute nodes. The node-COUNT/sizing here is fully ruled; the still-open D-121 sub-item (the vault-HA backend fork) is a Stage-5 bundle concern and does NOT gate this step.

MUTATION -- edit opentofu/main.tf, one block per node VM (control shown; compute/storage identical shape with their row's vCPU/memory/disk/name):

module "vr1_dc0_control_01" {
  source          = "./modules/node-vm"
  vm_name         = "vr1-dc0-control-01"
  memory_mib      = 65536                              # D-121: control
  vcpu            = 16                                  # D-121: control
  pool_name       = module.vr1_dc0_storage.pool_name
  disk_size_bytes = 150 * 1024 * 1024 * 1024           # D-121: control 150 GiB
  network_names   = [ <ordered list of module.vr1_dc0_planes outputs -- confirm which plane goes first for PXE priority, per D-103> ]
}

Repeat for vr1_dc0_control_02/03, vr1_dc0_compute_01/02 (12 vCPU / 49152 MiB / 100 * 1024 * 1024 * 1024), and vr1_dc0_storage_01/02/03 (8 vCPU / 24576 MiB / 550 * 1024 * 1024 * 1024). network_names is STILL a flagged STOP in every block: D-121 ruled the SIZES, not the plane attachment/PXE order -- that ordered plane list is an unresolved value (D-103 PXE priority) and must not be invented to make the block parse.

GATE: exactly 8 module "vr1_dc0_node_*"-shaped blocks (3 control / 2 compute / 3 storage) with D-121's sizes verbatim; each network_names either a genuinely-resolved plane list or explicitly recorded as the remaining STOP, never a guessed order.

Step 7 -- tofu init / validate / plan (READ-ONLY against providers)

cd opentofu
tofu fmt -check -recursive -diff .
tofu init -backend=false -input=false
tofu validate
tofu plan -out=phase2-vr1-dc0.tfplan

Review the plan line by line: expect creates for module.vr1_dc0_opnsense (the edge VM's boot volume and domain -- NO config-seed volume/cdrom anymore; that path is deleted, D-112) and each module.vr1_dc0_* node (a blank boot volume + domain per node: 3 control, 2 compute, 3 storage). Confirm nothing else is planned -- in particular, confirm no vr1-dc1 resource appears (its vr1_dc1_planes block should still be commented out per the gate above).

GATE: plan matches this expectation exactly; no unexpected creates, updates, or destroys. Do not apply a plan you have not read.

Step 8 -- tofu apply -- vr1-dc0 OPNsense edge + node-VM domains [MUTATION: gated]

cd opentofu
tofu apply phase2-vr1-dc0.tfplan

Confirm this is the exact reviewed plan file from Step 7 (not re-planned) before running. This is the first live mutation THIS stage performs.

VERIFY

virsh list --all
virsh domblklist vr1-dc0-opnsense
virsh domblklist vr1-dc0-control-01   # per real node name

Expect the new domains present (likely shut off until first boot/PXE), the OPNsense domain showing a SINGLE boot disk (a direct copy of the prepped nano -- no config-ISO cdrom; that path is deleted, D-112), each node VM showing a single blank boot disk.

GATE: all new domains defined as planned; disk attachments match Step 5/6's wiring.

State file reminder: re-check chmod 600 opentofu/terraform.tfstate / git status after this apply too, same as every stage since Stage 1 Step 10 first created the file (opentofu/README.md "State file handling", DOCFIX-175) -- Step 9 next is the first step in this stage that needs a REAL (non-placeholder) TF_VAR_maas_api_key to actually succeed, so this is the last checkpoint before a genuine credential is likely to enter state if it hasn't already.

Step 9 -- Wire + apply modules/maas-vm-host -- register vr1-dc0's libvirt host with vr1-dc0's MAAS rack controller [MUTATION: gated]

Hard dependency -- vr1-dc0's MAAS controller. This step requires vr1-dc0's own MAAS to exist and be reachable. D-123 (PROPOSED, recommend Model A) resolves what that is: a vvr1-dc0 site headend VM built the proven Office1 way -- an OpenTofu cloudinit-vm block shaped like module "voffice1" + a run of scripts/site-headend-install.sh (which already names vvr1-dc0/vvr1-dc1 as targets) to install MAAS region+rack + LXD. Under Model A the node VMs stay vcloud-level and this MAAS DISCOVERS them via the maas-vm-host registration below (power_address = vcloud's virsh). Until Model A is ruled and vvr1-dc0 is stood up, this is the honest scope boundary: Steps 1-8 are this pass's achievable scope; Steps 9-12 STOP here -- vr1-dc0's Phase 2 is then INCOMPLETE, not silently skippable. (Also inherits Stage 2's reachable-MAAS-region dependency.)

MUTATION -- edit opentofu/main.tf

module "vr1_dc0_maas_vm_host" {
  source         = "./modules/maas-vm-host"
  vm_host_name   = "vr1-dc0-vcloud"
  power_address  = "<Step 3's real, measured virsh URI -- confirm same as var.libvirt_uri or a distinct value>"
  zone           = <real zone name, or leave unset for MAAS's computed default -- per Step 3>
  pool           = <real pool name, or leave unset -- per Step 3>
}

cd opentofu
tofu plan -out=phase2-vr1-dc0-maashost.tfplan   # review: exactly one new maas_vm_host resource
tofu apply phase2-vr1-dc0-maashost.tfplan

GATE: maas_vm_host resource created; no other resource touched by this plan.

Step 10 -- Verify MAAS discovers the node VMs (READ-ONLY)

CHECK

maas admin vm-hosts read | jq -r '.[] | select(.name=="vr1-dc0-vcloud") | {id, name, resources}'
maas admin machines read | jq -r '.[] | select(.hostname|test("^vr1-dc0-")) | "\(.hostname)\t\(.status_name)"'

Expect vr1-dc0's node VMs listed as MAAS machines (status likely New -- enlistment discovered them; commissioning is Stage 4's job, not this stage's). If the node VMs do not appear, do not proceed to declare this stage's MAAS-discovery gate met -- troubleshoot the VM-host registration (power_address correctness, virsh reachability from the MAAS rack controller) before moving on.

GATE: vr1-dc0's OpenTofu-created node VMs visible in MAAS, associated with the vr1-dc0-vcloud VM host.

Step 11 -- Wire + apply modules/netem-link on vr1-dc0's mesh legs, placeholder profile [MUTATION: gated]

Per the netem callout above: this step applies ONLY the buildout design's qualitative Section 6 lean ("same-metro dark fiber... low single-digit ms, jitter-capable, jumbo-capable"), explicitly recorded as a PLACEHOLDER, not a ruled numeric value. If the operator has since ruled a specific numeric profile, use that instead and cite the ruling; do not use this placeholder language once a real ruling exists.

MUTATION -- edit opentofu/main.tf, one block per relevant mesh leg. The only leg with substrate on both ends this pass is vr1-dc0<->office1 (vr1-dc0<->vr1-dc1 and vr1-dc1<->office1 have no vr1-dc1-side substrate while vr1-dc1 remains gated):

module "netem_vr1_dc0_office1" {
  source                  = "./modules/netem-link"
  link_name               = "vr1-dc0-office1"
  vcloud_host_ssh_target  = "<real SSH target for the vcloud host -- measured, not invented>"
  bridge_name             = "<real OS bridge name for this mesh-link network -- confirm via `virsh net-info vr1-dc0-office1` or `tofu show`, not assumed>"
  netem_args              = "<PLACEHOLDER pending D-100 sub-item ruling -- record explicitly as provisional, e.g. a low single-digit-ms delay with modest jitter and negligible loss, cited to buildout-design Section 6, NOT presented as a measured or ruled value>"
}

(A vr1-dc0<->vr1-dc1 block -- module "netem_vr1_dc0_vr1_dc1", link_name "vr1-dc0-vr1-dc1" -- follows the same shape once vr1-dc1 itself is ungated; until then, applying netem on that leg has no vr1-dc1-side substrate to matter for, so it is naturally deferred alongside the rest of vr1-dc1's work.)

cd opentofu
tofu plan -out=phase2-vr1-dc0-netem.tfplan
tofu apply phase2-vr1-dc0-netem.tfplan

GATE: the terraform_data/local-exec resource applies without error; record the exact netem_args string used and its PLACEHOLDER status in the as-executed log, alongside a note that D-100's exact parameters remain open (gap #11).

Step 12 -- Verify edge routing + simulated ISP uplink + netem applied (READ-ONLY)

CHECK -- from the vcloud host

sudo tc qdisc show dev <bridge_name>   # per Step 11's real bridge name

Expect a netem qdisc present with the applied (placeholder) parameters.

CHECK -- OPNsense edge reachability (only meaningful once the domain has been booted and configured over the REST API per Step 4's REPLACEMENT chain -- the vtnetN mapping and WAN/LAN addressing are measured/applied AFTER boot)

virsh domstate vr1-dc0-opnsense

If the domain has not yet been through the console + REST-API bootstrap (the D-112(c) chain), this verification is necessarily partial this session -- record that honestly rather than declaring edge routing verified when it is not.

GATE (this step): netem qdisc present and matching Step 11's recorded placeholder profile; OPNsense domain running IF it has been through the post-boot REST-API bootstrap, otherwise recorded as blocked pending that bootstrap (console -> SSH -> API, per Step 4).

GATE -- Stage 3 exit condition (honest state, not aspirational)

Buildout-design Section 4 / deployment-workflow Stage 3 states the gate as: "MAAS rack controller per DC up; MAAS discovers the OpenTofu-created node VMs; edge routing + simulated ISP uplink up; netem parameters applied and measured." As of what this runbook can actually close this session:

• MAAS rack controller per DC up: achievable ONLY once vr1-dc0's own MAAS rack controller is stood up -- a Stage-4 gap that is NOT yet done (D-122 confirms each site runs its own controller; the SOURCE for it is the open item). Steps 1/9 depend on it. This is why honest scope for a first run is Steps 1-8. Record actual state, do not assume it.
• MAAS discovers the OpenTofu-created node VMs: achievable through Step 10, CONDITIONAL on the per-DC rack controller existing (Step 9's dependency) and Stage 2's MAAS region being reachable. Node sizing itself is no longer a condition -- it is RULED (D-121, Step 6).
• Edge routing + simulated ISP uplink up: PARTIALLY achievable. The OPNsense domain and its network attachments can be created (Step 8), and the edge shape/uplink are RULED (D-122: vr1-dc0-wan dedicated ISP uplink, Office1-pattern edge), but full edge routing verification depends on the post-boot REST-API bootstrap (Step 4's REPLACEMENT chain -- console -> SSH -> API, where the real vtnetN mapping and WAN/LAN addressing are measured and applied). Do not mark this sub-gate fully closed until that bootstrap has actually run against the booted domain.
• netem parameters applied and measured: applied, YES (Step 11), using an explicitly-labeled PLACEHOLDER same-metro-lean profile per buildout-design Section 6 -- MEASURED in the sense of "the qdisc is present and its effect can be observed" (e.g. via ping/iperf once the domains are reachable), but NOT measured against a ruled, final numeric target, because that target does not exist yet (D-100 gap #11). Do not represent this sub-gate as "done" in the same sense as a stage with no open decisions -- it is done AS FAR AS A PLACEHOLDER ALLOWS, with the remaining gap named.

This stage's exit gate is therefore CONDITIONALLY MET at best on any run before (a) vr1-dc0's per-DC MAAS rack controller is stood up (Stage-4 gap) and Stage 2's MAAS region is reachable, (b) vr1-dc0's OPNsense edge has been through the post-boot REST-API bootstrap (vtnetN + WAN/LAN applied), and (c) D-100's exact netem parameters are ruled. Node sizing (D-121) and the edge shape/uplink (D-122) are NO LONGER open. Update docs/dc-dc-deployment-workflow.md's Stage 3 row to reflect the REAL state after running this (e.g. "PARTIAL -- libvirt objects created, MAAS registration blocked on the per-DC rack controller" or similar), never to a blanket DONE unless every sub-condition above is genuinely true.

vr1-dc1: none of the above applies to vr1-dc1 this pass. Its v4 supernet IS assigned (D-115, 10.12.64.0/19) -- the blocker is sequencing, not a missing literal -- but its vr1_dc1_planes block stays commented out and it stays out of scope. Re-run this entire runbook for vr1-dc1 when it is sequenced; do not attempt a partial vr1-dc1 pass now.

-> Proceed to Stage 4 (MAAS enlist/commission/deploy) for vr1-dc0 once this gate's applicable sub-conditions are genuinely met; vr1-dc1 remains blocked at Stage 3 by sequencing until it is taken up as its own pass.

Delivery checklist (this repo's standard discipline)

• bash scripts/repo-lint.sh clean (0 fail) before committing any repo

  changes made while executing this runbook (`main.tf` edits, any new
  tfvars, minus secrets).

• bash scripts/opentofu-validate.sh green (the harness for opentofu/,

  per its own README) -- re-run after every `main.tf` change in Steps
  5/6/9/11, not just once at the end.

• Changelog entry for this runbook's first real execution (next DOCFIX

  number via `bash scripts/ledger-scan.sh`), noting the ACTUAL measured/
  decided values used (node sizing, MAAS zone/pool, power_address, the
  netem placeholder profile and its provisional status) -- redact
  nothing that isn't a secret.

• docs/session-ledger.md updated with the outcome, including which

  sub-steps were blocked and why (per-DC MAAS rack-controller Stage-4 gap,
  Stage 2 MAAS-region dependency, the edge post-boot bootstrap, D-100
  netem sub-item).

• docs/dc-dc-deployment-workflow.md Stage 3 row updated to the REAL,

  honest state (PARTIAL/DONE/NOT STARTED per the GATE section above),
  never rounded up to DONE with open sub-conditions.

• opentofu/README.md updated to reflect vr1-dc0's OPNsense edge /

  node-VM / MAAS-vm-host / netem-link modules moving from "not
  instantiated" to instantiated (or partially so) once this runbook
  actually applies them, including any real bug found while doing so
  (matching this repo's practice of logging schema surprises as they're
  found).

• vr1-dc1 gate re-confirmed as still closed BY SEQUENCING (its supernet

  IS assigned -- D-115 `10.12.64.0/19`) if this session did not take it
  up -- do not let this runbook's `vr1-dc0` completion be misread as
  covering `vr1-dc1`.