# DC-DC Phase 2 -- OpenTofu builds each DC substrate (Stage 3) From Office1, script each DC's "physical" layer into existence: the node-VM libvirt domains MAAS will discover and deploy, the DC's plane networks, and its OPNsense edge -- then register the DC's libvirt host to that DC's own MAAS rack controller as a virsh VM-host. This runbook builds **`vr1-dc0`** (VR1's FIRST DC -- the build target this pass). **`vr1-dc1`** (VR1's SECOND DC) is explicitly gated (see the callout below) and is NOT part of this session's run until that gate clears -- do not duplicate these steps for `vr1-dc1` with invented values in the meantime. !!! **NAMING -- this runbook was written pre-D-119 and has since been reconciled to the region-qualified namespace.** Its original prose spoke the bare `dcN` namespace (`dc1` = the build target, `dc2` = the second DC). **D-119 REJECTS bare `dcN` for VR1 datacenters** -- the token `dc0` already means VR0's live DC in `scripts/lib-net.sh`, so an unqualified `dcN` is a latent off-by-one. The mapping applied throughout this file is: runbook "DC1" -> **`vr1-dc0`** (already live in `opentofu/main.tf` as the `vr1_dc0_*` modules), runbook "DC2" -> **`vr1-dc1`** (still gated). All object names, module labels, plan files, and network/pool names below are region-qualified to match `main.tf`. **HAZARD:** following ANY residual bare-`dcN` literal in a copy of this runbook (or in muscle memory) literally reintroduces the exact `dc0`/`dc1` cross-cloud collision D-119 was written to delete -- region-qualify it or stop. (D-119.) **Governing docs:** `docs/dc-dc-buildout-design.md` Section 4 "Phase 2" (the goal/build/gate this runbook satisfies), Section 9 (shim register -- the node-VM creation module has **NO Roosevelt analog**, see the callout below), and Section 6 (WAN simulation / netem, same-metro lean). `docs/dc-dc-deployment-workflow.md` Stage 3 row and the Tooling gap register (items 1, 4, 11 govern this stage directly). `opentofu/README.md` and `opentofu/templates/README.md` for exact module/token status -- read both before touching any `.tf` file or the OPNsense template. Decisions this runbook owns: **D-103** (the OpenTofu/MAAS/Juju lifecycle seam -- OpenTofu creates, MAAS discovers/commissions, never composes), **D-100** (the dark-fiber mesh + per-site OPNsense edge, and the netem WAN-simulation mechanism riding on it). --- !!! **Depends on Stage 2 (Office1 headend standup).** This stage's MAAS registration step needs a REACHABLE MAAS region controller (Stage 2's gate) and Stage 1's OpenTofu-reaches-vcloud-libvirt gate already closed. If Stage 2 has not completed when this session runs, STOP at Step 9 (MAAS registration) -- everything through Step 8 (planes/pool/edge/node-VM creation at the libvirt layer) does not itself require MAAS and may proceed, but do not fabricate a `maas_api_url`/`maas_api_key` pair to push past that dependency. Confirm Stage 2's actual state in `docs/dc-dc-deployment-workflow.md` before starting. !!! **`vr1-dc1` is explicitly gated -- by SEQUENCING, not by a missing literal.** The earlier rationale ("NetBox has not assigned the supernet") is STALE: **D-115 ALREADY assigned it** -- `vr1-dc1` gets `10.12.64.0/19` (moved inside the Cloud `/16`, colliding with nothing `vr1-dc0` occupies), and that carve is imported to `office1-netbox`. So the blocker is no longer a missing address; `vr1-dc1` stays out of scope THIS pass purely because we build one DC at a time and `vr1-dc0` is the target. `opentofu/main.tf`'s `vr1_dc1_planes` block is still commented out (its `vr1_dc1_storage` and the three mesh legs ARE wired, being address-independent). Do NOT uncomment the planes block, do NOT add `vr1-dc1` build steps, and do NOT run this runbook's steps against `vr1-dc1` this pass. Re-run the whole runbook for `vr1-dc1` when it is sequenced. This is a hard gate, re-stated in the GATE section. NB: `main.tf`'s commented `vr1_dc1_planes` block still carries the OLD "wait for NetBox to assign D-101's supernet" comment -- that comment is now stale per D-115; flagged as a divergence to fix under the `main.tf` owner, out of scope for this doc-only pass. !!! **Node-VM creation has NO Roosevelt analog (Section 9 shim register).** `modules/node-vm` (the blank-disk, PXE-boot libvirt domain this stage calls) is a virtual-only stand-in for what, in Roosevelt, is physical racking plus BMC enlistment -- MAAS enlists hardware that already exists there. Do not treat this module, or the act of Terraform-creating "node hardware," as reusable production IaC. Everything from MAAS enlistment onward (commission/deploy, Juju bundle) IS Roosevelt-transferable; this one step is not. !!! **Node sizing is now RULED (D-121, Option C, 2026-07-15).** `modules/node-vm`'s `memory_mib`/`vcpu`/`disk_size_bytes` still have no module defaults by design, but the VALUES to pass are no longer open: D-121 ruled the per-DC layout as **3 control + 2 compute + 3 storage** (8 node VMs) at the sizes wired into Step 6. This was the exact decision that used to block Step 6; it is now UNBLOCKED. Still open (one sub-ruling D-121 notes): the vault-HA backend fork -- but that is a Stage-5 bundle concern, NOT a Step-6 blocker (node layout is fully ruled). Do not invent any OTHER node spec; use D-121's numbers exactly. !!! **netem parameters are still an unruled D-100 sub-item (gap #11).** This stage's own gate text says "netem parameters applied and measured," but the exact latency/jitter/loss/rate values do not exist as a ruled decision -- only the qualitative buildout-design Section 6 lean exists: "same-metro dark fiber... low single-digit ms, jitter-capable, jumbo- capable." Step 11 below applies ONLY that qualitative lean as an explicitly-labeled PLACEHOLDER profile (recorded as such, with a citation to Section 6), pending a firmer numeric ruling. Do not write a specific number (e.g. "delay 3ms") into `netem_args` and represent it as measured or ruled -- if you pick a number to instantiate the mechanism for real, log it as exactly what it is: a placeholder pending D-100's still-open sub-item, not a final value. !!! **OPNsense prep tools.** `scripts/opnsense-prep-image.sh` requires `bunzip2` + `qemu-img` on PATH. **The ISO9660 builder requirement is GONE (2026-07-13):** the config-ISO builder (`opnsense-build-config-iso.sh`, formerly under scripts/) is DELETED (D-112 -- the Importer can never fire on a nano image, so the ISO was never read; D-113(a2) -- config is done over the REST API). `genisoimage`/`xorriso` are no longer needed and are no longer installed by `scripts/prereqs/install-image-tools.sh`. Step 2 checks the remaining tools. --- ## Sequence (`vr1-dc0`; `vr1-dc1` remains gated -- see callout above) Honest achievable scope for a first run is **Steps 1-8** (libvirt-layer objects + the edge). Step 9 onward is gated on `vr1-dc0`'s own MAAS rack controller existing -- a Stage-4 gap (per-DC rack-controller source is not yet stood up); do not treat Steps 9-12 as runnable until it is. ``` 1. Pre-flight: confirm Stage 1 + Stage 2 gates closed (read-only) 2. Confirm OPNsense prep-tool prereqs on the vcloud host (read-only) 3. Confirm/record decisions this stage depends on (node sizing = D-121 RESOLVED; edge/WAN/LAN = D-122 RESOLVED; MAAS zone/pool; netem parameters) -- STOP on any open (read-only) 4. Prepare vr1-dc0's OPNsense base image (prep only; edge config is done over the REST API -- D-113(a2)) [MUTATION: host-local files, gated] 5. Wire modules/opnsense-edge for vr1-dc0 into main.tf [repo change, gated] (prereq: build the vr1-dc0-wan uplink, office1-wan shape) 6. Wire the 8 modules/node-vm calls for vr1-dc0's node VMs into main.tf (D-121 Option C sizes) [repo change, gated] 7. tofu init / validate / plan -- review before apply (read-only) 8. tofu apply -- vr1-dc0 OPNsense edge + node-VM domains [MUTATION: creates libvirt objects, gated] 9. Wire + apply modules/maas-vm-host -- register vr1-dc0's libvirt host with vr1-dc0's MAAS rack controller [MUTATION: MAAS + repo change, gated] 10. Verify MAAS discovers the node VMs (read-only) 11. Wire + apply modules/netem-link on vr1-dc0's mesh legs, using the Section-6 placeholder lean, explicitly flagged [MUTATION: gated] 12. Verify edge routing + simulated ISP uplink + netem applied (read-only) -> EXIT GATE -> Stage 4 (per-DC, once vr1-dc1 clears its own gate) ``` ### D-123 MODEL B RESHAPE (2026-07-16) -- the single-root sequence above is REGROUPED; read this The operator ruled D-123 = **Model B**: the vr1-dc0 node fleet nests INSIDE the containment VM `vvr1-dc0` (single-object `virsh destroy vvr1-dc0` = site-down). A libvirt provider cannot target a VM created in the same apply, so the substrate is now TWO ROOTS with a bootstrap gate between. The Steps above still describe the WORK; they regroup under the new roots + a new apply order: - **A. OUTER apply** (`opentofu/`, vcloud) -- creates + sizes `vvr1-dc0` (~416 GiB, `expose_nested_virt=true`, single transit leg) + the transit. Steps 1-3 (preflight/decisions) apply here. `[MUTATION: gated]` - **B. BOOTSTRAP GATE** on vvr1-dc0 (after it boots): `scripts/site-headend-install.sh --role rack --host-nodes --region-url ... --enroll-secret-file ...` -- enroll the rack to the Office1 region AND make vvr1-dc0 a nested libvirt host (qemu/libvirt, `kvm nested=1`, inner pool, AppArmor, the SEC-010 transit FORWARD-drop). Prep the OPNsense base ON vvr1-dc0 (Step 4 -- run `opnsense-prep-image.sh` there). GATE: `site-headend-install.sh --role rack --host-nodes --check` must pass (incl. SEC-010). - **C. INNER apply** (`opentofu/vr1-dc0-substrate/`, provider = qemu+ssh to vvr1-dc0 over the transit, R-5) -- the 6 planes + `vr1-dc0-wan` + edge + inner pool + 9 node VMs. Steps 5-6 (wire edge/nodes) are ALREADY authored in this inner root; Step 7 (init/validate/plan) + Step 8 (apply) run HERE, against vvr1-dc0's libvirt, NOT vcloud. `[MUTATION: gated]` - **D. maas-vm-host** (Step 9) -- register vvr1-dc0's OWN (inner, local `qemu:///system`) virsh to the Office1 REGION; MAAS discovers the inner node VMs (Step 10). Still DEFERRED (DOCFIX-179). - **E. netem** (Step 11) -- UNCHANGED: runs on the vcloud-level mesh bridges (virbr7), outer. Node count is R-3 = 3 control + 2 compute + **4 storage = 9/DC** (Step 6's "8" is superseded). The Model A single-root layout is preserved as a revert (git tag `model-a-fallback` + `docs/model-a-fallback-plan.md`). Sizing derived by `scripts/dc-dc-whole-host-budget.py` (870/1024 GiB, FIT). SEC-010 DC-LOCAL is enforced by the bootstrap's transit FORWARD-drop (NOT a global `ip_forward=0` -- the inner WAN NAT forces `ip_forward=1`). See `docs/changelog-20260716-review-sweep-phaseC.md`. --- ## Step 1 -- Pre-flight: confirm Stage 1 + Stage 2 gates closed (READ-ONLY) **CHECK -- Stage 1 (vcloud host prep) closed?** ```bash virsh net-list --all virsh pool-list --all ``` Expect: `vr1-dc0`'s six plane networks, the three mesh-link networks (`vr1-dc0-vr1-dc1`, `vr1-dc0-office1`, `vr1-dc1-office1` -- the region- qualified names now in `main.tf`), and the `vr1-dc0`/`office1` storage pools, all `active` -- Stage 1's Step 11 verification. If any are missing, Stage 1 is not actually done regardless of what `docs/dc-dc-deployment-workflow.md` says -- stop and reconcile there first. **CHECK -- Stage 2 (Office1 headend) closed?** Confirm against `docs/dc-dc-deployment-workflow.md`'s Stage 2 row and, if a Stage 2 runbook exists by the time you run this, its own exit gate. At minimum, confirm: ```bash # from Office1 (or wherever this session's tofu apply will run from) tofu -chdir=opentofu version curl -sI "$MAAS_API_URL" 2>/dev/null | head -1 # MAAS_API_URL: this session's real value, not invented ``` Expect a real MAAS region controller reachable at a real URL. If Stage 2 is not done, everything through Step 8 below (libvirt-layer objects only) may still proceed since it needs no MAAS reach; Step 9 onward (MAAS registration) is a hard stop until Stage 2 closes -- do not substitute a placeholder MAAS endpoint to push past it. **GATE:** Stage 1 objects present and active; Stage 2's MAAS reachability either confirmed, or explicitly recorded as NOT YET (in which case this session's scope for today is Steps 1-8 only). --- ## Step 2 -- Confirm OPNsense prep-tool prerequisites on the vcloud host (READ-ONLY) **CHECK** ```bash for bin in bunzip2 qemu-img genisoimage xorriso curl wget; do command -v "$bin" >/dev/null 2>&1 && echo "present: $bin" || echo "absent: $bin" done ``` Required: `bunzip2` + `qemu-img` (for `scripts/opnsense-prep-image.sh`), and (The `genisoimage`/`xorriso` requirement is GONE -- the config-ISO path is deleted.) plus `curl` or `wget` for the image fetch. If any required tool is absent, install it on the vcloud host (an OS-specific package install, out of scope to prescribe here) before Step 4 -- do not skip the tool and hand-build the ISO/qcow2 by some improvised method not covered by this repo's tested scripts. **GATE:** all required tools present (or installed and re-checked). --- ## Step 3 -- Confirm/record still-open decisions this stage depends on (READ-ONLY) Before wiring anything into `main.tf`, walk this list. Two of the items that used to block this stage are now RULED (D-121, D-122) and are recorded as RESOLVED below; the genuinely-open ones remain STOPs. Record each plainly rather than inventing a value to keep moving: - **Node sizing** (`memory_mib`/`vcpu`/`disk_size_bytes` and count for `modules/node-vm`) -- **RESOLVED (D-121, Option C, 2026-07-15).** Per DC: 3 control @ 16 vCPU / 65536 MiB / 150 GiB; 2 compute @ 12 vCPU / 49152 MiB / 100 GiB; 3 storage @ 8 vCPU / 24576 MiB / 550 GiB -- 8 node VMs total. Step 6 wires exactly these. Open sub-ruling D-121 still carries (the vault-HA backend fork) does NOT block Step 6 -- it is a Stage-5 bundle concern. - **Edge sizing / WAN uplink / LAN role** -- **RESOLVED (D-122, 2026-07-15).** The DC edge follows the proven Office1 pattern: `memory_mib = 2048`, `vcpu = 2`, nano base image (NO `disk_size_bytes`, like `office1_opnsense`). The edge is **2-NIC** (baremetal-matched): `wan_network_name` is a **dedicated per-site ISP uplink `vr1-dc0-wan`** (`172.30.2.0/24`, D-115) built on the `office1-wan` pattern -- NOT a mesh leg (dark fiber is East-West only). `lan_network_name` is **provider-public** -- the six planes are routed by the fabric (OVN/OpenStack on isolated-L2 segments), so the edge is only the external gateway on provider-public (D-100), not an inter-plane router. See Step 5. PREREQUISITES: build `vr1-dc0-wan` (as a tofu module) and register `172.30.2.0/24` in office1-netbox (not yet loaded). - **MAAS zone/pool for `vr1-dc0`'s VM-host registration** (`modules/maas-vm-host`'s `zone`/`pool` variables) -- both are optional (MAAS computes them if left `null`, confirmed safe per `opentofu/README.md`'s audit pass), so this is NOT necessarily a blocker; record whichever real zone/pool name the operator wants, or confirm the `null` no-op is intentional. Affects Step 9. - **`vr1-dc0`'s real `power_address`** (the virsh URI MAAS uses to reach the vcloud host) -- must be measured this session (or carried from Stage 1's `libvirt_uri` tfvar, if it is the SAME endpoint -- confirm, do not assume identical per `modules/maas-vm-host`'s own variable note). Blocks Step 9. - **`vr1-dc0`'s MAAS rack controller** -- a Stage-4 gap: the per-DC rack-controller source is not yet stood up. This is the reason honest Step-9 scope is BLOCKED and this pass's real reach is Steps 1-8 (see the GATE section). Blocks Steps 9-12. - **Edge post-boot interface mapping** (formerly the `{{WAN_IF}}`/`{{LAN_IF}}` `config.xml` tokens) -- NO LONGER a config-render blocker. There is no `config.xml` render (D-112/D-113(a2)); the real `vtnetN` mapping is measured AFTER boot over SSH (`ifconfig`), then applied over the REST API. See the REPLACEMENT chain in Step 4. Nothing here blocks wiring Step 5. - **netem parameters** -- still an unruled D-100 sub-item (gap #11); Step 11 uses the Section-6 qualitative placeholder only, explicitly labeled as such. Not a hard blocker for Step 11 (the mechanism can be exercised with a labeled placeholder), but IS a blocker for calling this stage's netem gate "final" -- see the GATE section. **GATE:** every item above is either resolved with a real, measured/ruled value, or explicitly recorded as NOT YET with the corresponding downstream step named as blocked. Do not proceed past a step whose input is still "NOT YET" by inventing one. --- ## Step 4 -- Prepare `vr1-dc0`'s OPNsense base image [MUTATION: host-local files, gated] > ## THE EDGE-CONFIG PATH (settled 2026-07-13 -- this IS the path) > > This step is now **prep the base image only.** Edge configuration is NOT baked into an image > or seeded from an ISO -- it is done over the **REST API after boot**, the path proven end to > end on the live Office1 edge (2026-07-12/13). The two deleted approaches, and why: > > 1. **The config ISO was INERT -- it could never be read.** Per **D-112** (root-caused from > upstream source): `opnsense-importer -b` probes for a read-only root. On INSTALLER media the > probe fails and the importer scans attached media. On our PRE-INSTALLED NANO image the root > is writable AND a factory `/conf/config.xml` already exists, so BOTH conditions hold and it > `bootstrap_and_exit 0`s **without enumerating a single device**. The Configuration Importer > can NEVER fire on a nano image, by design -- so the ISO builder is DELETED and the module no > longer takes a `config_iso_path`. > 2. **Full-`config.xml` rendering is SUPERSEDED** by **D-113(a2)** (ruled 2026-07-13): edge > config is done over the **REST API** (`scripts/opnsense-api.sh`), proven end to end -- read > AND write -- against the live Office1 edge on 2026-07-13. Hand-authoring the appliance's > GUI-owned XML caused DOCFIX-191 (management lockout), DOCFIX-192 (dead console) and > DOCFIX-193 (no DHCP, and an ISC `` block that would have been inert against a Kea > backend). None of those bugs is expressible through the API. The renderer is DELETED. > > **There is no `WAN_IF`/`LAN_IF` chicken-and-egg.** It only ever existed because we tried to seed > a full config before first boot. The D-112(c) bootstrap (boot on factory defaults -> reach the > console -> enable SSH + install the key -> configure over the network) measures the real > `vtnetN` mapping *after* boot, where it is knowable -- nothing to guess before Step 8. > > **What `vr1-dc0`'s edge does (the proven Office1 path):** prep the image (this step) -> Step 5 > wires the domain -> Step 8 boots it -> D-112(c) console bootstrap -> configure over SSH + the > REST API (the REPLACEMENT chain below). See > `docs/changelog-20260712-office1-opnsense-edge-build.md`, > `docs/changelog-20260713-opnsense-api-proven.md`, and > `docs/changelog-20260713-opnsense-api-write-proven.md`. This step produces one plain file on the vcloud host filesystem -- the prepped base image -- no libvirt or MAAS object is created yet. Run from the vcloud host (or wherever this script executes with reach to write there). **MUTATION -- prep the base image (needs Step 2's tools)** ```bash bash scripts/opnsense-prep-image.sh # see the script's own header for its exact args/output path ``` **Expect:** a decompressed, qcow2-converted, resized OPNsense nano image at a real output path -- record that path, it feeds Step 5's `base_volume_path`. That single path is this step's ONLY output. There is no config.xml render and no config-ISO build -- both are deleted (D-112 / D-113(a2), see the box above). --- ### The REPLACEMENT chain (proven on Office1, 2026-07-12/13) No `config.xml` anywhere. After Step 5 wires the domain and Step 8 boots it on FACTORY DEFAULTS: 1. **Console bootstrap (D-112(c))** -- reach the serial console, enable SSH, install the service public key. This is the only step that needs the console; everything after it is scripted. 2. **Mint the API key** -- no GUI click: ```bash export OPNSENSE_SSH_KEY=... # the service private key bash scripts/opnsense-bootstrap-apikey.sh ``` 3. **Configure over REST** -- interfaces, DHCP, firewall: ```bash export OPNSENSE_API_HOST= OPNSENSE_API_CREDS= bash scripts/opnsense-api.sh GET core/firmware/status # auth smoke test bash scripts/opnsense-api.sh POST kea/dhcpv4/set_subnet/ '' bash scripts/opnsense-api.sh POST kea/service/reconfigure '{}' ``` **The real `vtnetN` interface mapping is measured AFTER boot** (`ifconfig` over SSH), where it is knowable -- there is no interface value to guess before the domain exists. **GATE:** the prepped base image path recorded as a real host-local file path (it feeds Step 5's `base_volume_path`). Nothing else is produced or pending here -- no config-ISO path exists to record. --- ## Step 5 -- Wire `modules/opnsense-edge` for `vr1-dc0` into `main.tf` [repo change, gated] Proceed once Step 4 produced a real `base_volume_path`. The edge SHAPE is ruled (D-122): it mirrors the already-applied `office1_opnsense` block in `main.tf` -- `memory_mib = 2048`, `vcpu = 2`, nano image, **no `disk_size_bytes`** (the disk is a direct copy of the prepped nano; the module does not accept a size input -- DOCFIX-189), and **no `config_iso_path`** (the module dropped it -- D-112). There is nothing to invent for the edge specs themselves; the two values that ARE still per-site (the base image path, and the two network names) are called out below. **PREREQUISITE -- build the `vr1-dc0-wan` uplink first.** D-122 gives the DC its OWN dedicated L3 "ISP" uplink, NOT a mesh leg (dark fiber is East-West replication only). Build a per-site NAT'd `/24` uplink network `vr1-dc0-wan` on the **`office1-wan` pattern** (the same mechanical shape `office1-wan` was created in -- a documented D-103 debt to formalize into a module later; today it is a literal network name the edge's `wan_network_name` points at). This is a prerequisite to wiring the block below with a real `wan_network_name`. **LAN side -- RESOLVED (D-122, operator ruling 2026-07-15): `provider-public`.** The edge is **2-NIC** (WAN + LAN), matching the baremetal target: the six planes are **routed by the fabric** (the `dc-planes` segments are isolated L2; OVN/OpenStack own their L3, not the edge), so the edge is NOT an inter-plane router -- only the external boundary. Its LAN attaches to **provider-public** (`10.12.4.0/22`), where the edge is the upstream/external gateway the provider network exits through (FIP/SNAT egress + GUA injection, D-100 br-ex/provider model). metal-admin and the other planes reach external via fabric routing, not a direct edge leg. So `lan_network_name` = the `vr1_dc0_planes` provider-public output (confirm the exact output name in `modules/dc-planes`). No new site-internal network module is needed (this SUPERSEDES the earlier "build an office1-network analog" flag). **MUTATION -- edit `opentofu/main.tf`**, adding a block shaped like `office1_opnsense` (main.tf, the applied reference): ```hcl module "vr1_dc0_opnsense" { source = "./modules/opnsense-edge" vm_name = "vr1-dc0-opnsense" memory_mib = 2048 # D-122: Office1 pattern vcpu = 2 # D-122: Office1 pattern pool_name = module.vr1_dc0_storage.pool_name # No disk_size_bytes: nano direct-copy, size from opnsense-prep-image.sh's GROW (DOCFIX-189). base_volume_path = "" lan_network_name = wan_network_name = "vr1-dc0-wan" # D-122: dedicated per-site ISP uplink, office1-wan pattern (build first, promote to a tofu module) } ``` Before commit: the base image path (Step 4) must be real, `vr1-dc0-wan` must be built (as a tofu module -- D-122) and `172.30.2.0/24` registered in office1-netbox (D-115, not yet loaded), and `lan_network_name` bound to the real provider-public plane output. `memory_mib`/`vcpu`/the nano-no-disk shape and the 2-NIC WAN/LAN model are ruled (D-122), not invented. **GATE:** `opentofu/main.tf` diff shows exactly one new `module "vr1_dc0_opnsense"` block, no `disk_size_bytes` and no `config_iso_path` line, every argument a real value (with `vr1-dc0-wan` and the site-internal LAN network genuinely built), nothing else changed. --- ## Step 6 -- Wire `modules/node-vm` calls for `vr1-dc0`'s node VMs into `main.tf` [repo change, gated] **Node sizing is RULED (D-121, Option C).** Wire **8** `module "vr1_dc0_node_*"` blocks in three role-sizes: | Role | Count | vCPU | `memory_mib` | disk | Names | |---------|-------|------|--------------|-------------|------------------------------------| | control | 3 | 16 | 65536 | 150 GiB | `vr1-dc0-control-01..03` | | compute | 2 | 12 | 49152 | 100 GiB | `vr1-dc0-compute-01..02` | | storage | 3 | 8 | 24576 | 550 GiB | `vr1-dc0-storage-01..03` | `disk_size_bytes` is BYTES: express each as GiB x 1024^3 in HCL -- `150 * 1024 * 1024 * 1024`, `100 * 1024 * 1024 * 1024`, `550 * 1024 * 1024 * 1024` (the module's `disk_size_bytes` expects bytes; the GROW/qcow2 layer is thin-provisioned so this is a max, not a preallocation). **Why 3 control (context):** this layout exists to carry the Stage-5 HA scale-up (D-121). The bundle's **14 currently-single-unit services scale 1 -> 3**: the 11 hacluster+VIP API apps (keystone, glance, neutron-api, nova-cloud-controller, placement, cinder, openstack-dashboard, octavia, barbican, magnum, designate) plus `ceph-radosgw`, `rabbitmq-server`, and `vault`. Those 3-unit control-plane services LIVE on the 3 control nodes -- alongside the services that are ALREADY at 3 and do not change (`mysql-innodb-cluster`, `ovn-central`, `ceph-mon`). Meanwhile `ceph-osd` = 3 units on the 3 storage nodes and `nova-compute` = 2 units on the 2 compute nodes. The node-COUNT/sizing here is fully ruled; the still-open D-121 sub-item (the vault-HA backend fork) is a Stage-5 bundle concern and does NOT gate this step. **MUTATION -- edit `opentofu/main.tf`**, one block per node VM (control shown; compute/storage identical shape with their row's vCPU/memory/disk/name): ```hcl module "vr1_dc0_control_01" { source = "./modules/node-vm" vm_name = "vr1-dc0-control-01" memory_mib = 65536 # D-121: control vcpu = 16 # D-121: control pool_name = module.vr1_dc0_storage.pool_name disk_size_bytes = 150 * 1024 * 1024 * 1024 # D-121: control 150 GiB network_names = [ ] } ``` Repeat for `vr1_dc0_control_02`/`03`, `vr1_dc0_compute_01`/`02` (12 vCPU / 49152 MiB / `100 * 1024 * 1024 * 1024`), and `vr1_dc0_storage_01`/`02`/`03` (8 vCPU / 24576 MiB / `550 * 1024 * 1024 * 1024`). **`network_names` is STILL a flagged STOP** in every block: D-121 ruled the SIZES, not the plane attachment/PXE order -- that ordered plane list is an unresolved value (D-103 PXE priority) and must not be invented to make the block parse. **GATE:** exactly 8 `module "vr1_dc0_node_*"`-shaped blocks (3 control / 2 compute / 3 storage) with D-121's sizes verbatim; each `network_names` either a genuinely-resolved plane list or explicitly recorded as the remaining STOP, never a guessed order. --- ## Step 7 -- `tofu init` / `validate` / `plan` (READ-ONLY against providers) ```bash cd opentofu tofu fmt -check -recursive -diff . tofu init -backend=false -input=false tofu validate tofu plan -out=phase2-vr1-dc0.tfplan ``` Review the plan line by line: expect creates for `module.vr1_dc0_opnsense` (the edge VM's boot volume and domain -- NO config-seed volume/cdrom anymore; that path is deleted, D-112) and each `module.vr1_dc0_*` node (a blank boot volume + domain per node: 3 control, 2 compute, 3 storage). Confirm nothing else is planned -- in particular, confirm no `vr1-dc1` resource appears (its `vr1_dc1_planes` block should still be commented out per the gate above). **GATE:** plan matches this expectation exactly; no unexpected creates, updates, or destroys. Do not apply a plan you have not read. --- ## Step 8 -- `tofu apply` -- `vr1-dc0` OPNsense edge + node-VM domains [MUTATION: gated] ```bash cd opentofu tofu apply phase2-vr1-dc0.tfplan ``` Confirm this is the exact reviewed plan file from Step 7 (not re-planned) before running. This is the first live mutation THIS stage performs. **VERIFY** ```bash virsh list --all virsh domblklist vr1-dc0-opnsense virsh domblklist vr1-dc0-control-01 # per real node name ``` Expect the new domains present (likely `shut off` until first boot/PXE), the OPNsense domain showing a SINGLE boot disk (a direct copy of the prepped nano -- no config-ISO cdrom; that path is deleted, D-112), each node VM showing a single blank boot disk. **GATE:** all new domains defined as planned; disk attachments match Step 5/6's wiring. **State file reminder:** re-check `chmod 600 opentofu/terraform.tfstate` / `git status` after this apply too, same as every stage since Stage 1 Step 10 first created the file (`opentofu/README.md` "State file handling", DOCFIX-175) -- Step 9 next is the first step in this stage that needs a REAL (non-placeholder) `TF_VAR_maas_api_key` to actually succeed, so this is the last checkpoint before a genuine credential is likely to enter state if it hasn't already. --- ## Step 9 -- Wire + apply `modules/maas-vm-host` -- register `vr1-dc0`'s libvirt host with `vr1-dc0`'s MAAS rack controller [MUTATION: gated] **Hard dependency -- `vr1-dc0`'s MAAS controller.** This step requires `vr1-dc0`'s own MAAS to exist and be reachable. **D-123 (PROPOSED, recommend Model A) resolves what that is:** a `vvr1-dc0` **site headend VM** built the proven Office1 way -- an OpenTofu `cloudinit-vm` block shaped like `module "voffice1"` + a run of `scripts/site-headend-install.sh` (which already names `vvr1-dc0/vvr1-dc1` as targets) to install MAAS region+rack + LXD. Under Model A the node VMs stay vcloud-level and this MAAS DISCOVERS them via the `maas-vm-host` registration below (power_address = vcloud's virsh). Until Model A is ruled and `vvr1-dc0` is stood up, this is the honest scope boundary: **Steps 1-8 are this pass's achievable scope; Steps 9-12 STOP here** -- `vr1-dc0`'s Phase 2 is then INCOMPLETE, not silently skippable. (Also inherits Stage 2's reachable-MAAS-region dependency.) **MUTATION -- edit `opentofu/main.tf`** ```hcl module "vr1_dc0_maas_vm_host" { source = "./modules/maas-vm-host" vm_host_name = "vr1-dc0-vcloud" power_address = "" zone = pool = } ``` ```bash cd opentofu tofu plan -out=phase2-vr1-dc0-maashost.tfplan # review: exactly one new maas_vm_host resource tofu apply phase2-vr1-dc0-maashost.tfplan ``` **GATE:** `maas_vm_host` resource created; no other resource touched by this plan. --- ## Step 10 -- Verify MAAS discovers the node VMs (READ-ONLY) **CHECK** ```bash maas admin vm-hosts read | jq -r '.[] | select(.name=="vr1-dc0-vcloud") | {id, name, resources}' maas admin machines read | jq -r '.[] | select(.hostname|test("^vr1-dc0-")) | "\(.hostname)\t\(.status_name)"' ``` Expect `vr1-dc0`'s node VMs listed as MAAS machines (status likely `New` -- enlistment discovered them; commissioning is Stage 4's job, not this stage's). If the node VMs do not appear, do not proceed to declare this stage's MAAS-discovery gate met -- troubleshoot the VM-host registration (power_address correctness, virsh reachability from the MAAS rack controller) before moving on. **GATE:** `vr1-dc0`'s OpenTofu-created node VMs visible in MAAS, associated with the `vr1-dc0-vcloud` VM host. --- ## Step 11 -- Wire + apply `modules/netem-link` on `vr1-dc0`'s mesh legs, placeholder profile [MUTATION: gated] Per the netem callout above: this step applies ONLY the buildout design's qualitative Section 6 lean ("same-metro dark fiber... low single-digit ms, jitter-capable, jumbo-capable"), explicitly recorded as a PLACEHOLDER, not a ruled numeric value. If the operator has since ruled a specific numeric profile, use that instead and cite the ruling; do not use this placeholder language once a real ruling exists. **MUTATION -- edit `opentofu/main.tf`**, one block per relevant mesh leg. The only leg with substrate on both ends this pass is `vr1-dc0`<->`office1` (`vr1-dc0`<->`vr1-dc1` and `vr1-dc1`<->`office1` have no `vr1-dc1`-side substrate while `vr1-dc1` remains gated): ```hcl module "netem_vr1_dc0_office1" { source = "./modules/netem-link" link_name = "vr1-dc0-office1" vcloud_host_ssh_target = "" bridge_name = "" netem_args = "" } ``` (A `vr1-dc0`<->`vr1-dc1` block -- `module "netem_vr1_dc0_vr1_dc1"`, `link_name "vr1-dc0-vr1-dc1"` -- follows the same shape once `vr1-dc1` itself is ungated; until then, applying netem on that leg has no `vr1-dc1`-side substrate to matter for, so it is naturally deferred alongside the rest of `vr1-dc1`'s work.) ```bash cd opentofu tofu plan -out=phase2-vr1-dc0-netem.tfplan tofu apply phase2-vr1-dc0-netem.tfplan ``` **GATE:** the `terraform_data`/`local-exec` resource applies without error; record the exact `netem_args` string used and its PLACEHOLDER status in the as-executed log, alongside a note that D-100's exact parameters remain open (gap #11). --- ## Step 12 -- Verify edge routing + simulated ISP uplink + netem applied (READ-ONLY) **CHECK -- from the vcloud host** ```bash sudo tc qdisc show dev # per Step 11's real bridge name ``` Expect a `netem` qdisc present with the applied (placeholder) parameters. **CHECK -- OPNsense edge reachability** (only meaningful once the domain has been booted and configured over the REST API per Step 4's REPLACEMENT chain -- the `vtnetN` mapping and WAN/LAN addressing are measured/applied AFTER boot) ```bash virsh domstate vr1-dc0-opnsense ``` If the domain has not yet been through the console + REST-API bootstrap (the D-112(c) chain), this verification is necessarily partial this session -- record that honestly rather than declaring edge routing verified when it is not. **GATE (this step):** netem qdisc present and matching Step 11's recorded placeholder profile; OPNsense domain running IF it has been through the post-boot REST-API bootstrap, otherwise recorded as blocked pending that bootstrap (console -> SSH -> API, per Step 4). --- ## GATE -- Stage 3 exit condition (honest state, not aspirational) Buildout-design Section 4 / deployment-workflow Stage 3 states the gate as: "MAAS rack controller per DC up; MAAS discovers the OpenTofu-created node VMs; edge routing + simulated ISP uplink up; netem parameters applied and measured." As of what this runbook can actually close this session: - **MAAS rack controller per DC up:** achievable ONLY once `vr1-dc0`'s own MAAS rack controller is stood up -- a Stage-4 gap that is NOT yet done (D-122 confirms each site runs its own controller; the SOURCE for it is the open item). Steps 1/9 depend on it. This is why honest scope for a first run is Steps 1-8. Record actual state, do not assume it. - **MAAS discovers the OpenTofu-created node VMs:** achievable through Step 10, CONDITIONAL on the per-DC rack controller existing (Step 9's dependency) and Stage 2's MAAS region being reachable. Node sizing itself is no longer a condition -- it is RULED (D-121, Step 6). - **Edge routing + simulated ISP uplink up:** PARTIALLY achievable. The OPNsense domain and its network attachments can be created (Step 8), and the edge shape/uplink are RULED (D-122: `vr1-dc0-wan` dedicated ISP uplink, Office1-pattern edge), but full edge routing verification depends on the post-boot REST-API bootstrap (Step 4's REPLACEMENT chain -- console -> SSH -> API, where the real `vtnetN` mapping and WAN/LAN addressing are measured and applied). Do not mark this sub-gate fully closed until that bootstrap has actually run against the booted domain. - **netem parameters applied and measured:** applied, YES (Step 11), using an explicitly-labeled PLACEHOLDER same-metro-lean profile per buildout-design Section 6 -- MEASURED in the sense of "the qdisc is present and its effect can be observed" (e.g. via `ping`/`iperf` once the domains are reachable), but NOT measured against a ruled, final numeric target, because that target does not exist yet (D-100 gap #11). Do not represent this sub-gate as "done" in the same sense as a stage with no open decisions -- it is done AS FAR AS A PLACEHOLDER ALLOWS, with the remaining gap named. **This stage's exit gate is therefore CONDITIONALLY MET at best** on any run before (a) `vr1-dc0`'s per-DC MAAS rack controller is stood up (Stage-4 gap) and Stage 2's MAAS region is reachable, (b) `vr1-dc0`'s OPNsense edge has been through the post-boot REST-API bootstrap (`vtnetN` + WAN/LAN applied), and (c) D-100's exact netem parameters are ruled. Node sizing (D-121) and the edge shape/uplink (D-122) are NO LONGER open. Update `docs/dc-dc-deployment-workflow.md`'s Stage 3 row to reflect the REAL state after running this (e.g. "PARTIAL -- libvirt objects created, MAAS registration blocked on the per-DC rack controller" or similar), never to a blanket DONE unless every sub-condition above is genuinely true. **`vr1-dc1`:** none of the above applies to `vr1-dc1` this pass. Its v4 supernet IS assigned (D-115, `10.12.64.0/19`) -- the blocker is sequencing, not a missing literal -- but its `vr1_dc1_planes` block stays commented out and it stays out of scope. Re-run this entire runbook for `vr1-dc1` when it is sequenced; do not attempt a partial `vr1-dc1` pass now. -> Proceed to Stage 4 (MAAS enlist/commission/deploy) for `vr1-dc0` once this gate's applicable sub-conditions are genuinely met; `vr1-dc1` remains blocked at Stage 3 by sequencing until it is taken up as its own pass. --- ## Delivery checklist (this repo's standard discipline) - [ ] `bash scripts/repo-lint.sh` clean (0 fail) before committing any repo changes made while executing this runbook (`main.tf` edits, any new tfvars, minus secrets). - [ ] `bash scripts/opentofu-validate.sh` green (the harness for `opentofu/`, per its own README) -- re-run after every `main.tf` change in Steps 5/6/9/11, not just once at the end. - [ ] Changelog entry for this runbook's first real execution (next DOCFIX number via `bash scripts/ledger-scan.sh`), noting the ACTUAL measured/ decided values used (node sizing, MAAS zone/pool, power_address, the netem placeholder profile and its provisional status) -- redact nothing that isn't a secret. - [ ] `docs/session-ledger.md` updated with the outcome, including which sub-steps were blocked and why (per-DC MAAS rack-controller Stage-4 gap, Stage 2 MAAS-region dependency, the edge post-boot bootstrap, D-100 netem sub-item). - [ ] `docs/dc-dc-deployment-workflow.md` Stage 3 row updated to the REAL, honest state (PARTIAL/DONE/NOT STARTED per the GATE section above), never rounded up to DONE with open sub-conditions. - [ ] `opentofu/README.md` updated to reflect `vr1-dc0`'s OPNsense edge / node-VM / MAAS-vm-host / netem-link modules moving from "not instantiated" to instantiated (or partially so) once this runbook actually applies them, including any real bug found while doing so (matching this repo's practice of logging schema surprises as they're found). - [ ] `vr1-dc1` gate re-confirmed as still closed BY SEQUENCING (its supernet IS assigned -- D-115 `10.12.64.0/19`) if this session did not take it up -- do not let this runbook's `vr1-dc0` completion be misread as covering `vr1-dc1`.