# DC-DC Phase 0 -- vcloud host preparation (Stage 1) Turn the bare vcloud host into a substrate Office1 can deploy VR1 (DC1, DC2, Office1) from. This is the FIRST DC-DC runbook executed against real infrastructure -- everything before this session was repo-only prep (docs, scripts, OpenTofu modules, all UNVALIDATED against a real `tofu` binary or libvirt connection). Run from the vcloud host itself, or from a machine with `libvirt_uri` SSH reach to it -- confirm which before starting (Step 1). **Governing docs:** `docs/dc-dc-buildout-design.md` Section 3 (the gates this runbook satisfies) and Section 4 Phase 0 (goal/build/gate summary); `docs/dc-dc-deployment-workflow.md` Stage 1 (tracker row -- update its `**State:**` line when this runbook completes a real run); `opentofu/README.md` (module scope/status -- **SCAFFOLD, UNVALIDATED** as of this writing: no `tofu` binary has run against these modules yet. This runbook's Step 4 is the FIRST time that changes). Decisions this runbook owns: **D-100** (fabric: the dark-fiber mesh + per-site ISP-edge topology), **D-101** (MTU sub-policy, folded in from D-102 -- the geneve-over-v6 budget math in Step 3), and buildout-design Section 3's three Phase-0 gates (nested KVM, Ceph size decision, MTU). !!! Every value below marked MEASURE is filled in AT RUNTIME from this session's real output -- never carried over from the buildout design's ~256 vCPU / 1 TiB / 10 TiB planning estimate, which is a PLANNING NUMBER, not a measurement. If the real host doesn't match that estimate, the Ceph size=3-vs-size=2 arithmetic in Step 3 changes accordingly -- that is the gate working as designed, not a problem to paper over. --- ## Known gap, flagged before you start (not fixed in this runbook) `opentofu/main.tf` declares `provider "maas" { api_url = var.maas_api_url; api_key = var.maas_api_key }` UNCONDITIONALLY at the root, even though this Phase-0 stage only touches `libvirt`-backed resources (planes, storage pools, mesh links) -- no MAAS resource is instantiated yet (Stage 3's job). Depending on how strictly OpenTofu validates a declared-but-unused provider block, `tofu init`/`plan` in Step 4 MAY demand `maas_api_url`/`maas_api_key` be set even though nothing in this stage's plan uses MAAS. If you hit that: - Cheapest workaround: pass placeholder-shaped-but-clearly-fake values (e.g. `maas_api_url=http://not-yet-provisioned.invalid:5240/MAAS`) ONLY if OpenTofu merely wants the variable non-null and does not attempt to actually reach it during `plan` for a provider with zero resources. Verify this is true (a `plan`, not `apply`, should never open a real connection) before relying on it. - If OpenTofu actually attempts to validate MAAS reachability at `plan` time even with zero MAAS resources, that is a real structural finding -- LOG it (a DOCFIX candidate: split the MAAS provider block into a Stage-3-only root module) rather than working around it by fabricating real-looking credentials or skipping `tofu validate`. --- ## Sequence ``` 1. Identify the target host + confirm reach (read-only) 2. Measure: CPU/RAM/disk, nested-KVM support (read-only) 3. Measure: L2 MTU; compute Ceph size + MTU decisions (read-only + a recorded decision) 4. Enable nested KVM if not already on [MUTATION: kernel module param, gated] 5. Prepare libvirt storage pool paths [MUTATION: mkdir, gated] 6. Install/confirm OpenTofu + provider registry reach (read-only once installed) 7. Write opentofu/dc-dc-phase0.auto.tfvars from measured values (repo change, gated) 8. tofu init / validate / fmt (read-only against providers) 9. tofu plan -- review before apply (read-only) 10. tofu apply -- DC1 planes + Office1 pool + mesh links [MUTATION: creates libvirt objects, gated] 11. Post-apply verify against the Phase-0 gate (read-only) -> EXIT GATE -> Stage 2 (Office1 headend standup) ``` --- ## Step 1 -- Identify the target host + confirm reach (READ-ONLY) **CHECK -- from wherever you are running this session** ```bash hostname whoami uname -a ``` Record the actual hostname/user -- this is the value that goes into `libvirt_uri` in Step 7 (e.g. `qemu:///system` if running directly on the vcloud host as a user in the `libvirt` group, or `qemu+ssh://@/system` if running remotely). Do NOT assume the shape from any other repo's `VIRSH_POWER_ADDRESS` convention (`scripts/lib-hosts.sh`'s DC0 value is a DIFFERENT host/connection -- see that file's own DC-selector comments, DOCFIX-151) -- this is a distinct endpoint, confirmed here, not carried over. **CHECK -- libvirt reachable at all** ```bash virsh -c qemu:///system list --all # if running ON the vcloud host # or, if remote: # virsh -c qemu+ssh://@/system list --all ``` Expect: a (possibly empty) domain list with no connection error. If this fails, stop -- nothing past this point works without it. --- ## Step 2 -- Measure CPU/RAM/disk + nested-KVM support (READ-ONLY) **CHECK -- host resources** ```bash nproc --all free -h lsblk -o NAME,SIZE,TYPE,MOUNTPOINT df -h ``` Record the REAL numbers. The buildout design's "~256 vCPU / 1 TiB RAM / 10 TiB disk" is a planning estimate (`docs/dc-dc-buildout-design.md` Section 3) -- if actual capacity differs, Step 3's disk-budget arithmetic uses the MEASURED number, not the plan's. **CHECK -- CPU virtualization extensions present** ```bash egrep -c '(vmx|svm)' /proc/cpuinfo ``` Expect: a non-zero count. `vmx` = Intel VT-x, `svm` = AMD-V. Record which one -- it determines which kernel module's `nested` parameter Step 4 touches. **CHECK -- nested virtualization already enabled?** ```bash # Intel: cat /sys/module/kvm_intel/parameters/nested 2>/dev/null # AMD: cat /sys/module/kvm_amd/parameters/nested 2>/dev/null ``` Expect `Y` or `1` if already on. If the file doesn't exist for your CPU vendor, the `kvm_intel`/`kvm_amd` module may not be loaded yet -- `lsmod | grep kvm` to check, and consult your distro's docs for loading it before proceeding (out of scope for this runbook to prescribe -- this is a one-time host-OS-level fact to confirm, not a VR1-specific step). **GATE (Section 3, first bullet):** nested KVM is either already enabled (skip Step 4) or CPU-capable and not yet enabled (Step 4 turns it on). If the CPU shows zero `vmx`/`svm` support, STOP -- this host cannot run nested KVM at all, and the entire VR1 virtual-regional design (DC node VMs running their own nested libvirt/OpenStack) is not viable on it. That is a blocking finding for the operator, not something to route around. --- ## Step 3 -- Measure L2 MTU; record the Ceph size + MTU decisions (READ-ONLY + a recorded decision) **CHECK -- host L2 MTU on the interface(s) VR1 will use** ```bash ip -o link show | awk '{print $2, $0}' | grep -i mtu ``` Identify which interface(s) carry the vcloud host's real uplink (the one OpenTofu's virtual networks will ultimately ride over, or bridge to, for the simulated ISP edges) and record its MTU. Per D-101 (folded in from D-102): "Prefer jumbo (9000) end-to-end... if pinned at 1500, set the reduced tenant MTU consistently." **Do not assume jumbo** -- use the measured value. **Decision to record (D-101 MTU sub-policy, buildout-design Section 3) -- use the tested calculator (DOCFIX-162, tooling gap #7, 19/19 tests) instead of hand arithmetic:** ```bash bash scripts/dc-dc-mtu-geneve-budget.sh --underlay-mtu ``` This reproduces D-101's own worked example exactly (a measured 1500 -> tenant MTU 1444) and recommends leaving tenant MTU at 1500 for a measured jumbo (>=9000) underlay -- read the script's own `--help` for the full arithmetic it quotes verbatim from D-101. Set `underlay_mtu` in Step 7's tfvars to the MEASURED value from Step 3's own `ip -o link show` check above (never the script's output value itself -- the script tells you the DOWNSTREAM tenant MTU consequence, not what to put in `underlay_mtu`, which is always the raw measured underlay number). The propagation to ovn geneve/tenant-network MTU/amphora settings happens in later stages (Stage 5/7), not this one -- this step only records the decision. **Decision to record (Ceph size=3 vs size=2, Section 3 + the 2026-07-09 Stage-0 ruling) -- use the tested calculator (DOCFIX-162, 16/16 tests) instead of hand arithmetic:** ```bash bash scripts/dc-dc-ceph-disk-budget.sh --total-disk \ --dc1-nodes --dc1-per-node-osd \ --dc2-nodes --dc2-per-node-osd \ --backup-overhead-fraction ``` Target is size=3/min_size=2 BY DEFAULT (ADOPTED, not still open). The exact per-node OSD footprint and the backup/mirror/image overhead FRACTION are NOT knowable/measurable until DC1's actual Ceph cluster exists and reports real usage (Stage 5) -- for THIS stage, pass your best PLANNED/measured estimate for `--total-disk` (Step 2's `df`/`lsblk` output) and a plausibility-level `--backup-overhead-fraction` (the script has no default for this fraction on purpose -- it is not specified as a hard number anywhere in this repo), to get a PLAUSIBILITY verdict, not a final one. If the script's verdict is a shortfall, record that now as a flag for Stage 5's real budget pass rather than silently hoping it works out -- size=2 fallback is CONFIRMED ADOPTED as the only path if the real numbers don't fit (D-101/Section 3), but it must be an explicit, logged decision at the point real usage is measured (Stage 5), not guessed here. The script itself never applies size=2 silently -- it only names the fallback path and leaves the decision to the operator, per its own `--help` text. **GATE (Section 3, second + third bullets):** MTU measured and recorded; disk budget plausibility-checked and Ceph size decision path recorded (size=3 default, or flagged for a size=2 fallback review at Stage 5). --- ## Step 4 -- Enable nested KVM if not already on [MUTATION: gated] Only run this if Step 2's check showed nested virtualization is NOT already enabled. Skip straight to Step 5 if it was already `Y`/`1`. **MUTATION -- requires operator approval, one CPU vendor path or the other** ```bash # Intel host: echo "options kvm_intel nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf sudo rmmod kvm_intel && sudo modprobe kvm_intel # AMD host: echo "options kvm_amd nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf sudo rmmod kvm_amd && sudo modprobe kvm_amd ``` `rmmod`/`modprobe` fails if any VM is currently running on this host (module busy) -- if this vcloud host has other live workloads, this step may require a maintenance window / reboot instead. Confirm no unrelated VMs are running first (`virsh list --all` from Step 1). **VERIFY** ```bash cat /sys/module/kvm_intel/parameters/nested # or kvm_amd ``` Expect `Y`/`1`. --- ## Step 5 -- Prepare libvirt storage pool paths [MUTATION: gated] DC1 and Office1 each need a real filesystem path for their libvirt storage pool (`dc1_pool_path`/`office1_pool_path` in `opentofu/variables.tf` -- DC2's is deliberately not requested yet, per that file's own comment: no CIDRs assigned, gap #3's DATA half still open even after DOCFIX-152's mechanism closed). **CHECK -- pick real paths with adequate free space (from Step 2's `df -h`)** ```bash df -h /var/lib/libvirt/images 2>/dev/null || df -h / ``` Choose paths under whichever filesystem has the room computed in Step 3's disk-budget check. Do not reuse VR0/DC0's existing pool path if this vcloud host is the SAME physical host that ran the single-DC testcloud (check `virsh pool-list --all` for any existing pool already pointed at a candidate path) -- a path collision between VR0's retiring pool and VR1's new one is exactly the kind of silent-overlap risk this repo's discipline flags rather than assumes away. **MUTATION** ```bash sudo mkdir -p /path/you/chose/dc1 sudo mkdir -p /path/you/chose/office1 ``` (DC2's path is not created yet -- no CIDRs, no plan to instantiate it in `main.tf` yet either; see `opentofu/main.tf`'s commented DC2 block.) Record the two real paths -- they go into Step 7's tfvars as `dc1_pool_path`/`office1_pool_path`. --- ## Step 6 -- Install/confirm OpenTofu + provider registry reach **CHECK -- is `tofu` already present?** ```bash tofu version ``` If absent, install per OpenTofu's own official install instructions for this host's OS (not prescribed here -- an OS-specific package/binary install is a one-time host-prep action the operator runs directly, matching this repo's practice of not hardcoding install mechanics that vary by distro). **CHECK -- registry network access** (needed for `tofu init` in Step 8 to fetch `dmacvicar/libvirt` 0.9.8 and `canonical/maas` 2.7.2) ```bash curl -sI https://registry.opentofu.org/ | head -1 ``` Expect an HTTP response (200/301/etc.), not a connection failure. If this host is airgapped from the public internet by design (D-107's node-level airgap applies to OpenStack NODES, not necessarily the OpenTofu control point at Office1 -- confirm which category the vcloud host falls into before assuming either way), a local provider mirror may be needed -- out of scope for this runbook; flag and resolve before Step 8 if so. --- ## Step 7 -- Write `opentofu/dc-dc-phase0.auto.tfvars` from measured values [repo change, gated] Create a new tfvars file (NOT committed with real secrets -- `maas_api_key` stays out of any file that gets committed; pass it via `TF_VAR_maas_api_key` environment variable instead, never written to disk in the repo clone). ```hcl # opentofu/dc-dc-phase0.auto.tfvars # MEASURED this session (Stage 1, dc-dc-phase0-vcloud-prep.md) -- do not # hand-edit without re-measuring; every value here traces to a specific # Step above. libvirt_uri = "qemu:///system" # or qemu+ssh://... -- Step 1 underlay_mtu = 1500 # or 9000 -- Step 3 dc1_pool_path = "/path/you/chose/dc1" # Step 5 office1_pool_path = "/path/you/chose/office1" # Step 5 # maas_api_url / maas_api_key: see this runbook's "Known gap" section above. # If MAAS is not yet stood up (Stage 2 hasn't run), these may need a # placeholder per that section's guidance -- confirm the plan-time behavior # before committing to one path. maas_api_url = "TBD -- see Known Gap section" ``` Do NOT commit `maas_api_key` in any form. Use: ```bash export TF_VAR_maas_api_key="" ``` `domain_suffix` and `dc1_planes` already have ratified defaults in `opentofu/variables.tf` (D-106 naming; D-101 inherited DC1 CIDRs) -- no override needed unless you have a specific reason to deviate (log it as a decision if so). --- ## Step 8 -- `tofu init` / `validate` / `fmt` (read-only against providers) ```bash cd opentofu tofu fmt -check -recursive -diff . tofu init -backend=false -input=false tofu validate ``` Or equivalently, from the repo root: `bash scripts/opentofu-validate.sh`. This is the FIRST real run of this script against a real `tofu` binary -- `opentofu/README.md` has carried a **SCAFFOLD, UNVALIDATED** banner since authoring; a clean run here is the first evidence closing that banner (do not remove the banner from the README until this has actually run clean -- update it as part of this stage's completion, not preemptively). If `validate` surfaces a schema mismatch against the flagged UNVERIFIED notes in `opentofu/README.md` (the `node-vm` boot-order attribute shape is the most likely -- though `node-vm` isn't even instantiated in `main.tf` yet for this stage, so it wouldn't surface here; more relevant to THIS stage's plan are `dc-planes`, `mesh-link`, `dc-storage-pool`, all previously corrected for the attribute-vs-block syntax bug, DOCFIX-144) -- fix the specific module, re-run, and log a changelog entry the same way prior syntax fixes were logged this session. --- ## Step 9 -- `tofu plan` -- review before apply (READ-ONLY) ```bash cd opentofu tofu plan -out=phase0.tfplan ``` Review the plan output line by line against what Step 7's tfvars specify: expect creates for `module.dc1_planes` (six `libvirt_network` resources), `module.dc1_storage` + `module.office1_storage` (two `libvirt_pool` resources), and `module.mesh_dc1_dc2` / `module.mesh_dc1_office1` / `module.mesh_dc2_office1` (three more `libvirt_network` resources for the D-100 dark-fiber triangle legs). Confirm nothing else is planned (no DC2 plane resources -- that module block is commented out in `main.tf` and should stay that way until gap #3's DATA half closes). **GATE:** the plan matches this expectation exactly. If it doesn't (extra resources, missing resources, or an unexpected diff), STOP and reconcile before Step 10 -- do not apply a plan you have not read. --- ## Step 10 -- `tofu apply` [MUTATION: creates real libvirt objects, gated] ```bash cd opentofu tofu apply phase0.tfplan ``` This is the FIRST live mutation this repo's DC-DC work has performed against real infrastructure. Individually confirm this is the reviewed plan from Step 9 (same `.tfplan` file, not re-planned) before running. --- ## Step 11 -- Post-apply verify against the Phase-0 gate (READ-ONLY) ```bash virsh net-list --all virsh pool-list --all ``` Expect: the six DC1 plane networks, the three mesh-link networks, and the `dc1`/`office1` storage pools, all `active`. Cross-check each plane network's CIDR against `opentofu/variables.tf`'s `dc1_planes` default (which mirrors `scripts/lib-net.sh`'s `PLANE_CIDRS` -- DOCFIX-151's `lib_net_select_dc dc1` no-op is the same six values, confirming both sources still agree). ```bash virsh net-dumpxml | grep -i mtu ``` Confirm the MTU matches Step 3's recorded decision on every network. **GATE (buildout-design Section 4, Phase 0):** nested KVM verified (Step 2/4); disk budget computed and Ceph size decision recorded (Step 3); measured MTU recorded (Step 3); virtual networks present and isolated as designed (this step). All four true -> Stage 1 of `docs/dc-dc-deployment-workflow.md` moves from NOT STARTED to DONE; update that doc's Stage 1 row and Section 4's tracker table accordingly, and update `opentofu/README.md`'s SCAFFOLD/UNVALIDATED banner to reflect the first real validated run. -> Proceed to Stage 2 (Office1 headend standup): MAAS region controller, OpenTofu already reaching this host, NetBox, GitBucket, Tailscale. --- ## Delivery checklist (this repo's standard discipline) - [ ] `bash scripts/repo-lint.sh` clean (0 fail) before committing any repo changes made while executing this runbook (e.g. the tfvars file, minus secrets; any module fix from Step 8). - [ ] `bash scripts/opentofu-validate.sh` green (this IS the harness for `opentofu/`, per its own README). - [ ] Changelog entry for this runbook's first real execution (next DOCFIX number via `bash scripts/ledger-scan.sh`), noting the ACTUAL measured values (host specs, MTU, pool paths -- redact nothing that isn't a secret; these are exactly the kind of as-built facts this repo commits). - [ ] `docs/session-ledger.md` updated with the outcome. - [ ] `docs/dc-dc-deployment-workflow.md` Stage 1 row and tracker table updated from NOT STARTED to DONE (or to whatever partial state is honest, if the run surfaces a blocker).