DC-DC Phase 0 -- vcloud host preparation (Stage 1)

Turn the bare vcloud host into a substrate Office1 can deploy VR1 (DC1, DC2, Office1) from. This is the FIRST DC-DC runbook executed against real infrastructure -- everything before this session was repo-only prep (docs, scripts, OpenTofu modules, all UNVALIDATED against a real tofu binary or libvirt connection). Run from the vcloud host itself, or from a machine with libvirt_uri SSH reach to it -- confirm which before starting (Step 1).

Governing docs: docs/dc-dc-buildout-design.md Section 3 (the gates this runbook satisfies) and Section 4 Phase 0 (goal/build/gate summary); docs/dc-dc-deployment-workflow.md Stage 1 (tracker row -- update its **State:** line when this runbook completes a real run); opentofu/README.md (module scope/status -- SCAFFOLD, UNVALIDATED as of this writing: no tofu binary has run against these modules yet. This runbook's Step 4 is the FIRST time that changes).

Decisions this runbook owns: D-100 (fabric: the dark-fiber mesh + per-site ISP-edge topology), D-101 (MTU sub-policy, folded in from D-102 -- the geneve-over-v6 budget math in Step 3), and buildout-design Section 3's three Phase-0 gates (nested KVM, Ceph size decision, MTU).

!!! Every value below marked MEASURE is filled in AT RUNTIME from this session's real output -- never carried over from the buildout design's ~256 vCPU / 1 TiB / 10 TiB planning estimate, which is a PLANNING NUMBER, not a measurement. If the real host doesn't match that estimate, the Ceph size=3-vs-size=2 arithmetic in Step 3 changes accordingly -- that is the gate working as designed, not a problem to paper over.

Known gap, flagged before you start (not fixed in this runbook)

opentofu/main.tf declares provider "maas" { api_url = var.maas_api_url; api_key = var.maas_api_key } UNCONDITIONALLY at the root, even though this Phase-0 stage only touches libvirt-backed resources (planes, storage pools, mesh links) -- no MAAS resource is instantiated yet (Stage 3's job). Depending on how strictly OpenTofu validates a declared-but-unused provider block, tofu init/plan in Step 4 MAY demand maas_api_url/maas_api_key be set even though nothing in this stage's plan uses MAAS. If you hit that:

• Cheapest workaround: pass placeholder-shaped-but-clearly-fake values (e.g. maas_api_url=http://not-yet-provisioned.invalid:5240/MAAS) ONLY if OpenTofu merely wants the variable non-null and does not attempt to actually reach it during plan for a provider with zero resources. Verify this is true (a plan, not apply, should never open a real connection) before relying on it.
• If OpenTofu actually attempts to validate MAAS reachability at plan time even with zero MAAS resources, that is a real structural finding -- LOG it (a DOCFIX candidate: split the MAAS provider block into a Stage-3-only root module) rather than working around it by fabricating real-looking credentials or skipping tofu validate.

Sequence

1.  Identify the target host + confirm reach       (read-only)
2.  Measure: CPU/RAM/disk, nested-KVM support        (read-only)
3.  Measure: L2 MTU; compute Ceph size + MTU decisions (read-only + a recorded decision)
4.  Enable nested KVM if not already on               [MUTATION: kernel module param, gated]
5.  Prepare libvirt storage pool paths                [MUTATION: mkdir, gated]
6.  Install/confirm OpenTofu + provider registry reach (read-only once installed)
7.  Write opentofu/dc-dc-phase0.auto.tfvars from measured values (repo change, gated)
8.  tofu init / validate / fmt                        (read-only against providers)
9.  tofu plan -- review before apply                  (read-only)
10. tofu apply -- DC1 planes + Office1 pool + mesh links [MUTATION: creates libvirt objects, gated]
11. Post-apply verify against the Phase-0 gate          (read-only)
    -> EXIT GATE -> Stage 2 (Office1 headend standup)

Step 1 -- Identify the target host + confirm reach (READ-ONLY)

CHECK -- from wherever you are running this session

hostname
whoami
uname -a

Record the actual hostname/user -- this is the value that goes into libvirt_uri in Step 7 (e.g. qemu:///system if running directly on the vcloud host as a user in the libvirt group, or qemu+ssh://<user>@<host>/system if running remotely). Do NOT assume the shape from any other repo's VIRSH_POWER_ADDRESS convention (scripts/lib-hosts.sh's DC0 value is a DIFFERENT host/connection -- see that file's own DC-selector comments, DOCFIX-151) -- this is a distinct endpoint, confirmed here, not carried over.

CHECK -- libvirt reachable at all

virsh -c qemu:///system list --all   # if running ON the vcloud host
# or, if remote:
# virsh -c qemu+ssh://<user>@<host>/system list --all

Expect: a (possibly empty) domain list with no connection error. If this fails, stop -- nothing past this point works without it.

Step 2 -- Measure CPU/RAM/disk + nested-KVM support (READ-ONLY)

CHECK -- host resources

nproc --all
free -h
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT
df -h

Record the REAL numbers. The buildout design's "~256 vCPU / 1 TiB RAM / 10 TiB disk" is a planning estimate (docs/dc-dc-buildout-design.md Section 3) -- if actual capacity differs, Step 3's disk-budget arithmetic uses the MEASURED number, not the plan's.

CHECK -- CPU virtualization extensions present

egrep -c '(vmx|svm)' /proc/cpuinfo

Expect: a non-zero count. vmx = Intel VT-x, svm = AMD-V. Record which one -- it determines which kernel module's nested parameter Step 4 touches.

CHECK -- nested virtualization already enabled?

# Intel:
cat /sys/module/kvm_intel/parameters/nested 2>/dev/null
# AMD:
cat /sys/module/kvm_amd/parameters/nested 2>/dev/null

Expect Y or 1 if already on. If the file doesn't exist for your CPU vendor, the kvm_intel/kvm_amd module may not be loaded yet -- lsmod | grep kvm to check, and consult your distro's docs for loading it before proceeding (out of scope for this runbook to prescribe -- this is a one-time host-OS-level fact to confirm, not a VR1-specific step).

GATE (Section 3, first bullet): nested KVM is either already enabled (skip Step 4) or CPU-capable and not yet enabled (Step 4 turns it on). If the CPU shows zero vmx/svm support, STOP -- this host cannot run nested KVM at all, and the entire VR1 virtual-regional design (DC node VMs running their own nested libvirt/OpenStack) is not viable on it. That is a blocking finding for the operator, not something to route around.

Step 3 -- Measure L2 MTU; record the Ceph size + MTU decisions (READ-ONLY + a recorded decision)

CHECK -- host L2 MTU on the interface(s) VR1 will use

ip -o link show | awk '{print $2, $0}' | grep -i mtu

Identify which interface(s) carry the vcloud host's real uplink (the one OpenTofu's virtual networks will ultimately ride over, or bridge to, for the simulated ISP edges) and record its MTU. Per D-101 (folded in from D-102): "Prefer jumbo (9000) end-to-end... if pinned at 1500, set the reduced tenant MTU consistently." Do not assume jumbo -- use the measured value.

Decision to record (D-101 MTU sub-policy, buildout-design Section 3) -- use the tested calculator (DOCFIX-162, tooling gap #7, 19/19 tests) instead of hand arithmetic:

bash scripts/dc-dc-mtu-geneve-budget.sh --underlay-mtu <MEASURED_MTU>

This reproduces D-101's own worked example exactly (a measured 1500 -> tenant MTU 1444) and recommends leaving tenant MTU at 1500 for a measured jumbo (>=9000) underlay -- read the script's own --help for the full arithmetic it quotes verbatim from D-101. Set underlay_mtu in Step 7's tfvars to the MEASURED value from Step 3's own ip -o link show check above (never the script's output value itself -- the script tells you the DOWNSTREAM tenant MTU consequence, not what to put in underlay_mtu, which is always the raw measured underlay number). The propagation to ovn geneve/tenant-network MTU/amphora settings happens in later stages (Stage 5/7), not this one -- this step only records the decision.

Decision to record (Ceph size=3 vs size=2, Section 3 + the 2026-07-09 Stage-0 ruling) -- use the tested calculator (DOCFIX-162, 16/16 tests) instead of hand arithmetic:

bash scripts/dc-dc-ceph-disk-budget.sh --total-disk <MEASURED_TOTAL_DISK> \
  --dc1-nodes <N> --dc1-per-node-osd <SIZE> \
  --dc2-nodes <N> --dc2-per-node-osd <SIZE> \
  --backup-overhead-fraction <FRACTION>

Target is size=3/min_size=2 BY DEFAULT (ADOPTED, not still open). The exact per-node OSD footprint and the backup/mirror/image overhead FRACTION are NOT knowable/measurable until DC1's actual Ceph cluster exists and reports real usage (Stage 5) -- for THIS stage, pass your best PLANNED/measured estimate for --total-disk (Step 2's df/lsblk output) and a plausibility-level --backup-overhead-fraction (the script has no default for this fraction on purpose -- it is not specified as a hard number anywhere in this repo), to get a PLAUSIBILITY verdict, not a final one. If the script's verdict is a shortfall, record that now as a flag for Stage 5's real budget pass rather than silently hoping it works out -- size=2 fallback is CONFIRMED ADOPTED as the only path if the real numbers don't fit (D-101/Section 3), but it must be an explicit, logged decision at the point real usage is measured (Stage 5), not guessed here. The script itself never applies size=2 silently -- it only names the fallback path and leaves the decision to the operator, per its own --help text.

GATE (Section 3, second + third bullets): MTU measured and recorded; disk budget plausibility-checked and Ceph size decision path recorded (size=3 default, or flagged for a size=2 fallback review at Stage 5).

Step 4 -- Enable nested KVM if not already on [MUTATION: gated]

Only run this if Step 2's check showed nested virtualization is NOT already enabled. Skip straight to Step 5 if it was already Y/1.

MUTATION -- requires operator approval, one CPU vendor path or the other

# Intel host:
echo "options kvm_intel nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf
sudo rmmod kvm_intel && sudo modprobe kvm_intel
# AMD host:
echo "options kvm_amd nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf
sudo rmmod kvm_amd && sudo modprobe kvm_amd

rmmod/modprobe fails if any VM is currently running on this host (module busy) -- if this vcloud host has other live workloads, this step may require a maintenance window / reboot instead. Confirm no unrelated VMs are running first (virsh list --all from Step 1).

VERIFY

cat /sys/module/kvm_intel/parameters/nested   # or kvm_amd

Expect Y/1.

Step 5 -- Prepare libvirt storage pool paths [MUTATION: gated]

DC1 and Office1 each need a real filesystem path for their libvirt storage pool (dc1_pool_path/office1_pool_path in opentofu/variables.tf -- DC2's is deliberately not requested yet, per that file's own comment: no CIDRs assigned, gap #3's DATA half still open even after DOCFIX-152's mechanism closed).

CHECK -- pick real paths with adequate free space (from Step 2's df -h)

df -h /var/lib/libvirt/images 2>/dev/null || df -h /

Choose paths under whichever filesystem has the room computed in Step 3's disk-budget check. Do not reuse VR0/DC0's existing pool path if this vcloud host is the SAME physical host that ran the single-DC testcloud (check virsh pool-list --all for any existing pool already pointed at a candidate path) -- a path collision between VR0's retiring pool and VR1's new one is exactly the kind of silent-overlap risk this repo's discipline flags rather than assumes away.

MUTATION

sudo mkdir -p /path/you/chose/dc1
sudo mkdir -p /path/you/chose/office1

(DC2's path is not created yet -- no CIDRs, no plan to instantiate it in main.tf yet either; see opentofu/main.tf's commented DC2 block.)

Record the two real paths -- they go into Step 7's tfvars as dc1_pool_path/office1_pool_path.

Step 6 -- Install/confirm OpenTofu + provider registry reach

CHECK -- is tofu already present?

tofu version

If absent, install per OpenTofu's own official install instructions for this host's OS (not prescribed here -- an OS-specific package/binary install is a one-time host-prep action the operator runs directly, matching this repo's practice of not hardcoding install mechanics that vary by distro).

CHECK -- registry network access (needed for tofu init in Step 8 to fetch dmacvicar/libvirt 0.9.8 and canonical/maas 2.7.2)

curl -sI https://registry.opentofu.org/ | head -1

Expect an HTTP response (200/301/etc.), not a connection failure. If this host is airgapped from the public internet by design (D-107's node-level airgap applies to OpenStack NODES, not necessarily the OpenTofu control point at Office1 -- confirm which category the vcloud host falls into before assuming either way), a local provider mirror may be needed -- out of scope for this runbook; flag and resolve before Step 8 if so.

Step 7 -- Write opentofu/dc-dc-phase0.auto.tfvars from measured values [repo change, gated]

Create a new tfvars file (NOT committed with real secrets -- maas_api_key stays out of any file that gets committed; pass it via TF_VAR_maas_api_key environment variable instead, never written to disk in the repo clone).

# opentofu/dc-dc-phase0.auto.tfvars
# MEASURED this session (Stage 1, dc-dc-phase0-vcloud-prep.md) -- do not
# hand-edit without re-measuring; every value here traces to a specific
# Step above.

libvirt_uri       = "qemu:///system"          # or qemu+ssh://... -- Step 1
underlay_mtu      = 1500                       # or 9000 -- Step 3
dc1_pool_path     = "/path/you/chose/dc1"      # Step 5
office1_pool_path = "/path/you/chose/office1"  # Step 5

# maas_api_url / maas_api_key: see this runbook's "Known gap" section above.
# If MAAS is not yet stood up (Stage 2 hasn't run), these may need a
# placeholder per that section's guidance -- confirm the plan-time behavior
# before committing to one path.
maas_api_url = "TBD -- see Known Gap section"

Do NOT commit maas_api_key in any form. Use:

export TF_VAR_maas_api_key="<real key, once Stage 2 stands up MAAS>"

domain_suffix and dc1_planes already have ratified defaults in opentofu/variables.tf (D-106 naming; D-101 inherited DC1 CIDRs) -- no override needed unless you have a specific reason to deviate (log it as a decision if so).

Step 8 -- tofu init / validate / fmt (read-only against providers)

cd opentofu
tofu fmt -check -recursive -diff .
tofu init -backend=false -input=false
tofu validate

Or equivalently, from the repo root: bash scripts/opentofu-validate.sh. This is the FIRST real run of this script against a real tofu binary -- opentofu/README.md has carried a SCAFFOLD, UNVALIDATED banner since authoring; a clean run here is the first evidence closing that banner (do not remove the banner from the README until this has actually run clean -- update it as part of this stage's completion, not preemptively).

If validate surfaces a schema mismatch against the flagged UNVERIFIED notes in opentofu/README.md (the node-vm boot-order attribute shape is the most likely -- though node-vm isn't even instantiated in main.tf yet for this stage, so it wouldn't surface here; more relevant to THIS stage's plan are dc-planes, mesh-link, dc-storage-pool, all previously corrected for the attribute-vs-block syntax bug, DOCFIX-144) -- fix the specific module, re-run, and log a changelog entry the same way prior syntax fixes were logged this session.

Step 9 -- tofu plan -- review before apply (READ-ONLY)

cd opentofu
tofu plan -out=phase0.tfplan

Review the plan output line by line against what Step 7's tfvars specify: expect creates for module.dc1_planes (six libvirt_network resources), module.dc1_storage + module.office1_storage (two libvirt_pool resources), and module.mesh_dc1_dc2 / module.mesh_dc1_office1 / module.mesh_dc2_office1 (three more libvirt_network resources for the D-100 dark-fiber triangle legs). Confirm nothing else is planned (no DC2 plane resources -- that module block is commented out in main.tf and should stay that way until gap #3's DATA half closes).

GATE: the plan matches this expectation exactly. If it doesn't (extra resources, missing resources, or an unexpected diff), STOP and reconcile before Step 10 -- do not apply a plan you have not read.

Step 10 -- tofu apply [MUTATION: creates real libvirt objects, gated]

cd opentofu
tofu apply phase0.tfplan

This is the FIRST live mutation this repo's DC-DC work has performed against real infrastructure. Individually confirm this is the reviewed plan from Step 9 (same .tfplan file, not re-planned) before running.

Step 11 -- Post-apply verify against the Phase-0 gate (READ-ONLY)

virsh net-list --all
virsh pool-list --all

Expect: the six DC1 plane networks, the three mesh-link networks, and the dc1/office1 storage pools, all active. Cross-check each plane network's CIDR against opentofu/variables.tf's dc1_planes default (which mirrors scripts/lib-net.sh's PLANE_CIDRS -- DOCFIX-151's lib_net_select_dc dc1 no-op is the same six values, confirming both sources still agree).

virsh net-dumpxml <plane-network-name> | grep -i mtu

Confirm the MTU matches Step 3's recorded decision on every network.

GATE (buildout-design Section 4, Phase 0): nested KVM verified (Step 2/4); disk budget computed and Ceph size decision recorded (Step 3); measured MTU recorded (Step 3); virtual networks present and isolated as designed (this step). All four true -> Stage 1 of docs/dc-dc-deployment-workflow.md moves from NOT STARTED to DONE; update that doc's Stage 1 row and Section 4's tracker table accordingly, and update opentofu/README.md's SCAFFOLD/UNVALIDATED banner to reflect the first real validated run.

-> Proceed to Stage 2 (Office1 headend standup): MAAS region controller, OpenTofu already reaching this host, NetBox, GitBucket, Tailscale.

Delivery checklist (this repo's standard discipline)

• bash scripts/repo-lint.sh clean (0 fail) before committing any repo

  changes made while executing this runbook (e.g. the tfvars file, minus
  secrets; any module fix from Step 8).

• bash scripts/opentofu-validate.sh green (this IS the harness for

  `opentofu/`, per its own README).

• Changelog entry for this runbook's first real execution (next DOCFIX

  number via `bash scripts/ledger-scan.sh`), noting the ACTUAL measured
  values (host specs, MTU, pool paths -- redact nothing that isn't a
  secret; these are exactly the kind of as-built facts this repo commits).

• docs/session-ledger.md updated with the outcome.
• docs/dc-dc-deployment-workflow.md Stage 1 row and tracker table

  updated from NOT STARTED to DONE (or to whatever partial state is
  honest, if the run surfaces a blocker).