DC-DC Phase 0 -- vcloud host preparation (Stage 1)

FIRST EXECUTED 2026-07-10 (vcloud host, OpenTofu v1.12.3) -- run end-to-end against real infrastructure. As-executed corrections are folded in inline, tagged (DOCFIX-180); the OpenTofu scaffold fixes that run required are DOCFIX-179; the workstation prereq installers are DOCFIX-178. See the matching docs/changelog-20260710-*.md.

Turn the bare vcloud host into a substrate Office1 can deploy VR1 (DC1, DC2, Office1) from. This is the FIRST DC-DC runbook executed against real infrastructure -- everything before this session was repo-only prep (docs, scripts, OpenTofu modules, all UNVALIDATED against a real tofu binary or libvirt connection). Run from the vcloud host itself, or from a machine with libvirt_uri SSH reach to it -- confirm which before starting (Step 1).

Governing docs: docs/dc-dc-buildout-design.md Section 3 (the gates this runbook satisfies) and Section 4 Phase 0 (goal/build/gate summary); docs/dc-dc-deployment-workflow.md Stage 1 (tracker row -- update its **State:** line when this runbook completes a real run); opentofu/README.md (module scope/status -- SCAFFOLD, UNVALIDATED as of this writing: no tofu binary has run against these modules yet. This runbook's Step 4 is the FIRST time that changes).

Decisions this runbook owns: D-100 (fabric: the dark-fiber mesh + per-site ISP-edge topology), D-101 (MTU sub-policy, folded in from D-102 -- the geneve-over-v6 budget math in Step 3), and buildout-design Section 3's three Phase-0 gates (nested KVM, Ceph size decision, MTU).

!!! Every value below marked MEASURE is filled in AT RUNTIME from this session's real output -- never carried over from the buildout design's ~256 vCPU / 1 TiB / 10 TiB planning estimate, which is a PLANNING NUMBER, not a measurement. If the real host doesn't match that estimate, the Ceph size=3-vs-size=2 arithmetic in Step 3 changes accordingly -- that is the gate working as designed, not a problem to paper over.

Prerequisites -- runtimes on the vcloud host / Office1 control point (DOCFIX-178)

Do NOT assume these are already installed. This runbook drives a real libvirt host and a real tofu binary; confirm each BEFORE Step 1. Idempotent installer scripts live in scripts/prereqs/ (Debian/Ubuntu; each supports --check and --dry-run) -- see that directory's README.md. One-pass setup on a fresh workstation/workspace:

bash scripts/prereqs/check-prereqs.sh        # read-only: what's missing?
bash scripts/prereqs/install-all.sh          # install the missing prereqs (uses sudo)
# re-login (or: newgrp libvirt) if the libvirt group was just added, then:
bash scripts/prereqs/check-prereqs.sh        # confirm all required present

If the host is airgapped by policy, the Step 8 tofu init registry fetch needs a local provider mirror instead -- out of scope here; resolve before Step 8.

Known gap, flagged before you start (not fixed in this runbook)

RESOLVED on first execution (2026-07-10, DOCFIX-179/180). The root provider "maas" block DID force tofu plan to demand BOTH maas_api_url AND the SENSITIVE maas_api_key for a Stage-1 plan that creates zero MAAS resources (it fails at variable validation, before any provider is configured -- so no real MAAS connection is ever attempted). Rather than the placeholder workaround (which bakes a fake sensitive key into phase0.tfplan/terraform.tfstate -- the DOCFIX-175 surface), the operator ruled the STRUCTURAL fix: the provider "maas" block and the maas_api_url/maas_api_key variables are now REMOVED from the root module until Stage 3 instantiates maas-vm-host (DOCFIX-179). maas stays in the root versions.tf required_providers (version pinned, lock file complete). Stage 1 therefore needs no MAAS input at all. Stage 3's runbook MUST re-add the provider block + both variables when it wires maas-vm-host (noted in main.tf/variables.tf in-file).

Sequence

1.  Identify the target host + confirm reach       (read-only)
2.  Measure: CPU/RAM/disk, nested-KVM support        (read-only)
3.  Measure: L2 MTU; compute Ceph size + MTU decisions (read-only + a recorded decision)
4.  Enable nested KVM if not already on               [MUTATION: kernel module param, gated]
5.  Prepare libvirt storage pool paths                [MUTATION: mkdir, gated]
6.  Install/confirm OpenTofu + provider registry reach (read-only once installed)
7.  Write opentofu/dc-dc-phase0.auto.tfvars from measured values (repo change, gated)
8.  tofu init / validate / fmt                        (read-only against providers)
9.  tofu plan -- review before apply                  (read-only)
10. tofu apply -- DC1 planes + Office1 pool + mesh links [MUTATION: creates libvirt objects, gated]
11. Post-apply verify against the Phase-0 gate          (read-only)
    -> EXIT GATE -> Stage 2 (Office1 headend standup)

Step 1 -- Identify the target host + confirm reach (READ-ONLY)

CHECK -- from wherever you are running this session

hostname
whoami
uname -a

Record the actual hostname/user -- this is the value that goes into libvirt_uri in Step 7 (e.g. qemu:///system if running directly on the vcloud host as a user in the libvirt group, or qemu+ssh://<user>@<host>/system if running remotely). Do NOT assume the shape from any other repo's VIRSH_POWER_ADDRESS convention (scripts/lib-hosts.sh's DC0 value is a DIFFERENT host/connection -- see that file's own DC-selector comments, DOCFIX-151) -- this is a distinct endpoint, confirmed here, not carried over.

CHECK -- libvirt reachable at all

virsh -c qemu:///system list --all   # if running ON the vcloud host
# or, if remote:
# virsh -c qemu+ssh://<user>@<host>/system list --all

Expect: a (possibly empty) domain list with no connection error. If this fails, stop -- nothing past this point works without it.

Step 2 -- Measure CPU/RAM/disk + nested-KVM support (READ-ONLY)

CHECK -- host resources

nproc --all
free -h
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT
df -h

Record the REAL numbers. The buildout design's "~256 vCPU / 1 TiB RAM / 10 TiB disk" is a planning estimate (docs/dc-dc-buildout-design.md Section 3) -- if actual capacity differs, Step 3's disk-budget arithmetic uses the MEASURED number, not the plan's.

CHECK -- CPU virtualization extensions present

egrep -c '(vmx|svm)' /proc/cpuinfo

Expect: a non-zero count. vmx = Intel VT-x, svm = AMD-V. Record which one -- it determines which kernel module's nested parameter Step 4 touches.

CHECK -- nested virtualization already enabled?

# Intel:
cat /sys/module/kvm_intel/parameters/nested 2>/dev/null
# AMD:
cat /sys/module/kvm_amd/parameters/nested 2>/dev/null

Expect Y or 1 if already on. If the file doesn't exist for your CPU vendor, the kvm_intel/kvm_amd module may not be loaded yet -- lsmod | grep kvm to check, and consult your distro's docs for loading it before proceeding (out of scope for this runbook to prescribe -- this is a one-time host-OS-level fact to confirm, not a VR1-specific step).

GATE (Section 3, first bullet): nested KVM is either already enabled (skip Step 4) or CPU-capable and not yet enabled (Step 4 turns it on). If the CPU shows zero vmx/svm support, STOP -- this host cannot run nested KVM at all, and the entire VR1 virtual-regional design (DC node VMs running their own nested libvirt/OpenStack) is not viable on it. That is a blocking finding for the operator, not something to route around.

Step 3 -- Measure L2 MTU; record the Ceph size + MTU decisions (READ-ONLY + a recorded decision)

CHECK -- host L2 MTU on the interface(s) VR1 will use

ip -o link show | awk '{print $2, $0}' | grep -i mtu

Identify which interface(s) carry the vcloud host's real uplink and record its MTU. Per D-101 (folded in from D-102): "Prefer jumbo (9000) end-to-end... if pinned at 1500, set the reduced tenant MTU consistently."

As-executed clarification (2026-07-10, DOCFIX-180) -- there are TWO distinct MTU domains; do not conflate them: the measured host uplink MTU (e.g. enp1s0) is the ISP/WAN-edge MTU and governs the future per-site ISP-uplink networks (Stage 2/3), which are naturally ~1500 like real internet. But underlay_mtu here drives the six planes + mesh legs + office1-local, which are host-internal isolated virtio bridges -- they never traverse the physical uplink and ARE jumbo-capable regardless of it. Per D-101's explicit "prefer jumbo end-to-end" intent (and because real dark fiber is jumbo-capable), set underlay_mtu to the JUMBO value (9000) for the internal fabric when the host supports it -- keeping tenant MTU at 1500 with full geneve headroom and avoiding the nested-virt+geneve MTU-stacking failure the design itself flags. First run: measured uplink 1500, ruled underlay_mtu=9000. Run the calculator below for BOTH the measured uplink and the chosen jumbo value to see each verdict.

Decision to record (D-101 MTU sub-policy, buildout-design Section 3) -- use the tested calculator (DOCFIX-162, tooling gap #7, 19/19 tests) instead of hand arithmetic:

bash scripts/dc-dc-mtu-geneve-budget.sh --underlay-mtu <MEASURED_MTU>

This reproduces D-101's own worked example exactly (a measured 1500 -> tenant MTU 1444) and recommends leaving tenant MTU at 1500 for a measured jumbo (>=9000) underlay -- read the script's own --help for the full arithmetic it quotes verbatim from D-101. Set underlay_mtu in Step 7's tfvars to the MEASURED value from Step 3's own ip -o link show check above (never the script's output value itself -- the script tells you the DOWNSTREAM tenant MTU consequence, not what to put in underlay_mtu, which is always the raw measured underlay number). The propagation to ovn geneve/tenant-network MTU/amphora settings happens in later stages (Stage 5/7), not this one -- this step only records the decision.

Decision to record (Ceph size=3 vs size=2, Section 3 + the 2026-07-09 Stage-0 ruling) -- use the tested calculator (DOCFIX-162, 16/16 tests) instead of hand arithmetic:

bash scripts/dc-dc-ceph-disk-budget.sh --total-disk <MEASURED_TOTAL_DISK> \
  --dc1-nodes <N> --dc1-per-node-osd <SIZE> \
  --dc2-nodes <N> --dc2-per-node-osd <SIZE> \
  --backup-overhead-fraction <FRACTION>

Target is size=3/min_size=2 BY DEFAULT (ADOPTED, not still open). The exact per-node OSD footprint and the backup/mirror/image overhead FRACTION are NOT knowable/measurable until DC1's actual Ceph cluster exists and reports real usage (Stage 5) -- for THIS stage, pass your best PLANNED/measured estimate for --total-disk (Step 2's df/lsblk output) and a plausibility-level --backup-overhead-fraction (the script has no default for this fraction on purpose -- it is not specified as a hard number anywhere in this repo), to get a PLAUSIBILITY verdict, not a final one. If the script's verdict is a shortfall, record that now as a flag for Stage 5's real budget pass rather than silently hoping it works out -- size=2 fallback is CONFIRMED ADOPTED as the only path if the real numbers don't fit (D-101/Section 3), but it must be an explicit, logged decision at the point real usage is measured (Stage 5), not guessed here. The script itself never applies size=2 silently -- it only names the fallback path and leaves the decision to the operator, per its own --help text.

GATE (Section 3, second + third bullets): MTU measured and recorded; disk budget plausibility-checked and Ceph size decision path recorded (size=3 default, or flagged for a size=2 fallback review at Stage 5).

Step 4 -- Enable nested KVM if not already on [MUTATION: gated]

Only run this if Step 2's check showed nested virtualization is NOT already enabled. Skip straight to Step 5 if it was already Y/1.

MUTATION -- requires operator approval, one CPU vendor path or the other

# Intel host:
echo "options kvm_intel nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf
sudo rmmod kvm_intel && sudo modprobe kvm_intel
# AMD host:
echo "options kvm_amd nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf
sudo rmmod kvm_amd && sudo modprobe kvm_amd

rmmod/modprobe fails if any VM is currently running on this host (module busy) -- if this vcloud host has other live workloads, this step may require a maintenance window / reboot instead. Confirm no unrelated VMs are running first (virsh list --all from Step 1).

VERIFY

cat /sys/module/kvm_intel/parameters/nested   # or kvm_amd

Expect Y/1.

Step 5 -- Prepare libvirt storage pool paths [MUTATION: gated]

DC1 and Office1 each need a real filesystem path for their libvirt storage pool (dc1_pool_path/office1_pool_path in opentofu/variables.tf -- DC2's is deliberately not requested yet, per that file's own comment: no CIDRs assigned, gap #3's DATA half still open even after DOCFIX-152's mechanism closed).

CHECK -- pick real paths with adequate free space (from Step 2's df -h)

df -h /var/lib/libvirt/images 2>/dev/null || df -h /

Choose paths under whichever filesystem has the room computed in Step 3's disk-budget check. Do not reuse VR0/DC0's existing pool path if this vcloud host is the SAME physical host that ran the single-DC testcloud.

As-executed (2026-07-10, DOCFIX-180) -- the vcloud host was NOT pristine. It carried a prior hand-built VR0 topology (9 vr0-* isolated networks + a wan NAT network) plus the stock default pool at /var/lib/libvirt/images. Inventory FIRST:

virsh -c qemu:///system net-list --all
virsh -c qemu:///system pool-list --all

Tear down leftover prior-generation networks (gated, per runbooks/dc-dc-teardown-rollback.md) for a clean substrate + clean Step-11 verify -- BUT KEEP a pre-existing wan NAT-to-uplink network: it is a ready-made model for the per-site ISP-uplink network (gap #17) Stage 2/3 needs. New pool paths must NOT be /var/lib/libvirt/images itself (that is default's path).

MUTATION -- /var/lib/libvirt is root-owned. Either run the mkdir with sudo, OR (recommended, sudo-free thereafter) have the operator create a self-owned VR1 parent ONCE, then create the pool subdirs as your own user:

# one-time, operator (real terminal for the sudo password):
sudo install -d -o "$USER" -g libvirt -m 2775 /var/lib/libvirt/vr1
# then, no sudo, and DC2's pool too (see below):
mkdir -p /var/lib/libvirt/vr1/dc1 /var/lib/libvirt/vr1/office1 /var/lib/libvirt/vr1/dc2

DC2's pool dir + the dc2_storage module ARE wired now (operator ruling: a storage pool has no address dependency); only DC2's PLANES stay deferred pending NetBox's supernet. Record the real paths -- they become Step 7's dc1_pool_path/office1_pool_path/dc2_pool_path. First run used /var/lib/libvirt/vr1/{dc1,office1,dc2}.

Step 6 -- Install/confirm OpenTofu + provider registry reach

CHECK -- is tofu already present?

tofu version

If absent (or older than 1.6.0), install it with the repo's own idempotent installer -- this is now a documented prerequisite (see the Prerequisites section at the top of this runbook), not an out-of-band step:

bash scripts/prereqs/install-opentofu.sh        # deb method -> /usr/bin/tofu; needs sudo

Ideally the whole prereq set is already satisfied before Step 1 (bash scripts/prereqs/check-prereqs.sh); this Step is the in-sequence confirmation that tofu specifically is ready. For a non-Debian/non-Ubuntu control point, install-opentofu.sh fails loud with a pointer to OpenTofu's own portable install options rather than guessing a package manager.

CHECK -- registry network access (needed for tofu init in Step 8 to fetch dmacvicar/libvirt 0.9.8 and canonical/maas 2.7.2)

curl -sI https://registry.opentofu.org/ | head -1

Expect an HTTP response (200/301/etc.), not a connection failure. If this host is airgapped from the public internet by design (D-107's node-level airgap applies to OpenStack NODES, not necessarily the OpenTofu control point at Office1 -- confirm which category the vcloud host falls into before assuming either way), a local provider mirror may be needed -- out of scope for this runbook; flag and resolve before Step 8 if so.

Step 7 -- Write opentofu/dc-dc-phase0.auto.tfvars from measured values [repo change, gated]

Create a new tfvars file (NOT committed with real secrets -- maas_api_key stays out of any file that gets committed; pass it via TF_VAR_maas_api_key environment variable instead, never written to disk in the repo clone).

# opentofu/dc-dc-phase0.auto.tfvars
# MEASURED this session (Stage 1, dc-dc-phase0-vcloud-prep.md) -- do not
# hand-edit without re-measuring; every value here traces to a specific
# Step above.

libvirt_uri       = "qemu:///system"          # or qemu+ssh://... -- Step 1
underlay_mtu      = 1500                       # or 9000 -- Step 3
dc1_pool_path     = "/path/you/chose/dc1"      # Step 5
office1_pool_path = "/path/you/chose/office1"  # Step 5

# maas_api_url / maas_api_key: see this runbook's "Known gap" section above.
# If MAAS is not yet stood up (Stage 2 hasn't run), these may need a
# placeholder per that section's guidance -- confirm the plan-time behavior
# before committing to one path.
maas_api_url = "TBD -- see Known Gap section"

Do NOT commit maas_api_key in any form. Use:

export TF_VAR_maas_api_key="<real key, once Stage 2 stands up MAAS>"

domain_suffix and dc1_planes already have ratified defaults in opentofu/variables.tf (D-106 naming; D-101 inherited DC1 CIDRs) -- no override needed unless you have a specific reason to deviate (log it as a decision if so).

Step 8 -- tofu init / validate / fmt (read-only against providers)

cd opentofu
tofu fmt -check -recursive -diff .
tofu init -backend=false -input=false
tofu validate

Or equivalently, from the repo root: bash scripts/opentofu-validate.sh.

As-executed (2026-07-10, DOCFIX-179): the first real tofu init FAILED until two fixes landed, now both in the repo (a fresh clone inits clean):

1. Each child module using a provider resource needs its OWN required_providers (modules/*/versions.tf). Child modules do NOT inherit provider SOURCE mapping, so OpenTofu inferred the libvirt_ prefix as a nonexistent hashicorp/libvirt and init failed. Fixed for all libvirt + maas modules.
2. tofu fmt -recursive on the never-fmt'd tree (11 files). The opentofu/README.md banner is now updated from SCAFFOLD/UNVALIDATED to STAGE-1-VALIDATED (the Stage 2/3 modules remain unexercised).

If validate surfaces a schema mismatch against the flagged UNVERIFIED notes in opentofu/README.md (the node-vm boot-order attribute shape is the most likely -- though node-vm isn't even instantiated in main.tf yet for this stage, so it wouldn't surface here; more relevant to THIS stage's plan are dc-planes, mesh-link, dc-storage-pool, all previously corrected for the attribute-vs-block syntax bug, DOCFIX-144) -- fix the specific module, re-run, and log a changelog entry the same way prior syntax fixes were logged this session.

Step 9 -- tofu plan -- review before apply (READ-ONLY)

cd opentofu
tofu plan -input=false -out=phase0.tfplan

(-input=false so a missing required var ERRORS instead of hanging on an interactive prompt in a non-interactive/agent shell -- DOCFIX-180.)

Review the plan line by line. As-executed (2026-07-10, DOCFIX-180): the plan is 13 resources, not the 11 an earlier draft of this step implied:

• module.dc1_planes -- six libvirt_network (the six planes)
• module.dc1_storage + module.office1_storage + module.dc2_storage -- THREE libvirt_pool (DC2's storage pool is wired; only its planes are deferred)
• module.office1_network -- one libvirt_network (office1-local; DOCFIX-163 added this, and an earlier draft's expected-list omitted it)
• module.mesh_dc1_dc2 / mesh_dc1_office1 / mesh_dc2_office1 -- three libvirt_network (the D-100 dark-fiber legs)

Confirm NO DC2 plane resources (that block stays commented in main.tf until NetBox assigns the supernet). Expect Plan: 13 to add, 0 to change, 0 to destroy.

GATE: the plan matches this expectation exactly. If it doesn't (extra resources, missing resources, or an unexpected diff), STOP and reconcile before Step 10 -- do not apply a plan you have not read.

Step 10 -- tofu apply [MUTATION: creates real libvirt objects, gated]

cd opentofu
tofu apply phase0.tfplan

This is the FIRST live mutation this repo's DC-DC work has performed against real infrastructure. Individually confirm this is the reviewed plan from Step 9 (same .tfplan file, not re-planned) before running.

SECURE THE STATE FILE (do this immediately, before Step 11): this apply creates opentofu/terraform.tfstate for the first time. Regardless of whether Step 7's maas_api_key is still a placeholder at this point (per the Known-gap section above) or already a real value, treat the file as sensitive from here on -- it will carry whatever TF_VAR_maas_api_key value is exported the NEXT time tofu plan/apply runs (Stage 2 or 3, once MAAS is real), and sensitive = true only redacts CLI/plan OUTPUT, never state storage (verified against HashiCorp's own docs; see opentofu/README.md "State file handling", DOCFIX-175). Starting the file-permission/backup discipline now costs nothing and avoids a gap between "state exists" and "state is protected":

chmod 700 opentofu
chmod 600 opentofu/terraform.tfstate

Confirm git status shows opentofu/terraform.tfstate as untracked/ignored, not staged (.gitignore excludes it as of DOCFIX-175) -- if it shows as a trackable change, STOP and fix .gitignore before proceeding, do not commit it. Back the file up out-of-band per the operator's existing ~/vault-init/-class secret-handling process; repeat this same permission/backup check after every subsequent stage's tofu apply (Stage 2/3/etc.) -- this is a standing discipline from here forward, not a one-time Stage 1 step.

Step 11 -- Post-apply verify against the Phase-0 gate (READ-ONLY)

virsh net-list --all
virsh pool-list --all

Expect: the six DC1 plane networks, the three mesh-link networks, and the dc1/office1 storage pools, all active. Cross-check each plane network's CIDR against opentofu/variables.tf's dc1_planes default (which mirrors scripts/lib-net.sh's PLANE_CIDRS -- DOCFIX-151's lib_net_select_dc dc1 no-op is the same six values, confirming both sources still agree).

virsh net-dumpxml <plane-network-name> | grep -i mtu

Confirm the MTU matches Step 3's recorded decision on every network.

GATE (buildout-design Section 4, Phase 0): nested KVM verified (Step 2/4); disk budget computed and Ceph size decision recorded (Step 3); measured MTU recorded (Step 3); virtual networks present and isolated as designed (this step). All four true -> Stage 1 of docs/dc-dc-deployment-workflow.md moves from NOT STARTED to DONE; update that doc's Stage 1 row and Section 4's tracker table accordingly, and update opentofu/README.md's SCAFFOLD/UNVALIDATED banner to reflect the first real validated run.

-> Proceed to Stage 2 (Office1 headend standup): MAAS region controller, OpenTofu already reaching this host, NetBox, GitBucket, Tailscale.

Delivery checklist (this repo's standard discipline)

• bash scripts/repo-lint.sh clean (0 fail) before committing any repo

  changes made while executing this runbook (e.g. the tfvars file, minus
  secrets; any module fix from Step 8).

• bash scripts/opentofu-validate.sh green (this IS the harness for

  `opentofu/`, per its own README).

• Changelog entry for this runbook's first real execution (next DOCFIX

  number via `bash scripts/ledger-scan.sh`), noting the ACTUAL measured
  values (host specs, MTU, pool paths -- redact nothing that isn't a
  secret; these are exactly the kind of as-built facts this repo commits).

• docs/session-ledger.md updated with the outcome.
• docs/dc-dc-deployment-workflow.md Stage 1 row and tracker table

  updated from NOT STARTED to DONE (or to whatever partial state is
  honest, if the run surfaces a blocker).