# DC-DC Phase 0 -- vcloud host preparation (Stage 1)

Turn the bare vcloud host into a substrate Office1 can deploy VR1 (DC1, DC2,
Office1) from. This is the FIRST DC-DC runbook executed against real
infrastructure -- everything before this session was repo-only prep (docs,
scripts, OpenTofu modules, all UNVALIDATED against a real `tofu` binary or
libvirt connection). Run from the vcloud host itself, or from a machine with
`libvirt_uri` SSH reach to it -- confirm which before starting (Step 1).

**Governing docs:** `docs/dc-dc-buildout-design.md` Section 3 (the gates this
runbook satisfies) and Section 4 Phase 0 (goal/build/gate summary);
`docs/dc-dc-deployment-workflow.md` Stage 1 (tracker row -- update its
`**State:**` line when this runbook completes a real run); `opentofu/README.md`
(module scope/status -- **SCAFFOLD, UNVALIDATED** as of this writing: no
`tofu` binary has run against these modules yet. This runbook's Step 4 is the
FIRST time that changes).

Decisions this runbook owns: **D-100** (fabric: the dark-fiber mesh + per-site
ISP-edge topology), **D-101** (MTU sub-policy, folded in from D-102 -- the
geneve-over-v6 budget math in Step 3), and buildout-design Section 3's three
Phase-0 gates (nested KVM, Ceph size decision, MTU).

!!! Every value below marked MEASURE is filled in AT RUNTIME from this
    session's real output -- never carried over from the buildout design's
    ~256 vCPU / 1 TiB / 10 TiB planning estimate, which is a PLANNING NUMBER,
    not a measurement. If the real host doesn't match that estimate, the
    Ceph size=3-vs-size=2 arithmetic in Step 3 changes accordingly -- that is
    the gate working as designed, not a problem to paper over.

---

## Known gap, flagged before you start (not fixed in this runbook)

`opentofu/main.tf` declares `provider "maas" { api_url = var.maas_api_url;
api_key = var.maas_api_key }` UNCONDITIONALLY at the root, even though this
Phase-0 stage only touches `libvirt`-backed resources (planes, storage pools,
mesh links) -- no MAAS resource is instantiated yet (Stage 3's job). Depending
on how strictly OpenTofu validates a declared-but-unused provider block, `tofu
init`/`plan` in Step 4 MAY demand `maas_api_url`/`maas_api_key` be set even
though nothing in this stage's plan uses MAAS. If you hit that:
- Cheapest workaround: pass placeholder-shaped-but-clearly-fake values (e.g.
  `maas_api_url=http://not-yet-provisioned.invalid:5240/MAAS`) ONLY if OpenTofu
  merely wants the variable non-null and does not attempt to actually reach
  it during `plan` for a provider with zero resources. Verify this is true
  (a `plan`, not `apply`, should never open a real connection) before relying
  on it.
- If OpenTofu actually attempts to validate MAAS reachability at `plan` time
  even with zero MAAS resources, that is a real structural finding -- LOG it
  (a DOCFIX candidate: split the MAAS provider block into a Stage-3-only root
  module) rather than working around it by fabricating real-looking
  credentials or skipping `tofu validate`.

---

## Sequence

```
1.  Identify the target host + confirm reach       (read-only)
2.  Measure: CPU/RAM/disk, nested-KVM support        (read-only)
3.  Measure: L2 MTU; compute Ceph size + MTU decisions (read-only + a recorded decision)
4.  Enable nested KVM if not already on               [MUTATION: kernel module param, gated]
5.  Prepare libvirt storage pool paths                [MUTATION: mkdir, gated]
6.  Install/confirm OpenTofu + provider registry reach (read-only once installed)
7.  Write opentofu/dc-dc-phase0.auto.tfvars from measured values (repo change, gated)
8.  tofu init / validate / fmt                        (read-only against providers)
9.  tofu plan -- review before apply                  (read-only)
10. tofu apply -- DC1 planes + Office1 pool + mesh links [MUTATION: creates libvirt objects, gated]
11. Post-apply verify against the Phase-0 gate          (read-only)
    -> EXIT GATE -> Stage 2 (Office1 headend standup)
```

---

## Step 1 -- Identify the target host + confirm reach (READ-ONLY)

**CHECK -- from wherever you are running this session**
```bash
hostname
whoami
uname -a
```
Record the actual hostname/user -- this is the value that goes into
`libvirt_uri` in Step 7 (e.g. `qemu:///system` if running directly on the
vcloud host as a user in the `libvirt` group, or `qemu+ssh://<user>@<host>/system`
if running remotely). Do NOT assume the shape from any other repo's
`VIRSH_POWER_ADDRESS` convention (`scripts/lib-hosts.sh`'s DC0 value is a
DIFFERENT host/connection -- see that file's own DC-selector comments,
DOCFIX-151) -- this is a distinct endpoint, confirmed here, not carried over.

**CHECK -- libvirt reachable at all**
```bash
virsh -c qemu:///system list --all   # if running ON the vcloud host
# or, if remote:
# virsh -c qemu+ssh://<user>@<host>/system list --all
```
Expect: a (possibly empty) domain list with no connection error. If this
fails, stop -- nothing past this point works without it.

---

## Step 2 -- Measure CPU/RAM/disk + nested-KVM support (READ-ONLY)

**CHECK -- host resources**
```bash
nproc --all
free -h
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT
df -h
```
Record the REAL numbers. The buildout design's "~256 vCPU / 1 TiB RAM / 10 TiB
disk" is a planning estimate (`docs/dc-dc-buildout-design.md` Section 3) --
if actual capacity differs, Step 3's disk-budget arithmetic uses the MEASURED
number, not the plan's.

**CHECK -- CPU virtualization extensions present**
```bash
egrep -c '(vmx|svm)' /proc/cpuinfo
```
Expect: a non-zero count. `vmx` = Intel VT-x, `svm` = AMD-V. Record which one
-- it determines which kernel module's `nested` parameter Step 4 touches.

**CHECK -- nested virtualization already enabled?**
```bash
# Intel:
cat /sys/module/kvm_intel/parameters/nested 2>/dev/null
# AMD:
cat /sys/module/kvm_amd/parameters/nested 2>/dev/null
```
Expect `Y` or `1` if already on. If the file doesn't exist for your CPU
vendor, the `kvm_intel`/`kvm_amd` module may not be loaded yet -- `lsmod |
grep kvm` to check, and consult your distro's docs for loading it before
proceeding (out of scope for this runbook to prescribe -- this is a one-time
host-OS-level fact to confirm, not a VR1-specific step).

**GATE (Section 3, first bullet):** nested KVM is either already enabled
(skip Step 4) or CPU-capable and not yet enabled (Step 4 turns it on). If the
CPU shows zero `vmx`/`svm` support, STOP -- this host cannot run nested KVM at
all, and the entire VR1 virtual-regional design (DC node VMs running their
own nested libvirt/OpenStack) is not viable on it. That is a blocking finding
for the operator, not something to route around.

---

## Step 3 -- Measure L2 MTU; record the Ceph size + MTU decisions (READ-ONLY + a recorded decision)

**CHECK -- host L2 MTU on the interface(s) VR1 will use**
```bash
ip -o link show | awk '{print $2, $0}' | grep -i mtu
```
Identify which interface(s) carry the vcloud host's real uplink (the one
OpenTofu's virtual networks will ultimately ride over, or bridge to, for the
simulated ISP edges) and record its MTU. Per D-101 (folded in from D-102):
"Prefer jumbo (9000) end-to-end... if pinned at 1500, set the reduced tenant
MTU consistently." **Do not assume jumbo** -- use the measured value.

**Decision to record (D-101 MTU sub-policy, buildout-design Section 3):**
- If measured MTU >= 9000: set `underlay_mtu = 9000` in Step 7's tfvars.
  Tenant MTU stays 1500; geneve/amphora headroom is generous.
- If measured MTU == 1500 (the common case for a plain host NIC): set
  `underlay_mtu = 1500`, and record that tenant-network MTU will need to
  drop to ~1444 (v6-geneve overhead: 40 IPv6 + 8 UDP + 8 Geneve base = 56
  bytes, before nested-virt/options) -- this propagates to ovn geneve,
  tenant-network MTU, and amphora settings in later stages (Stage 5/7), not
  this one. Just RECORD the decision here; the propagation happens where
  those components are actually configured.

**Decision to record (Ceph size=3 vs size=2, Section 3 + the 2026-07-09
Stage-0 ruling):** target is size=3/min_size=2 BY DEFAULT (ADOPTED, not
still open). This is conditional on the measured disk budget from Step 2:
```
required_disk_TiB ~= (DC1_Ceph_size3 + DC2_Ceph_size3 + per-DC_radosgw_backup_pools
                       + Glance_rbd-mirror_targets + images + per-DC_artifact_mirrors)
```
The exact per-node OSD footprint isn't knowable until DC1's actual Ceph
cluster exists and reports real usage (Stage 5) -- for THIS stage, the gate
is narrower: confirm the MEASURED total disk (Step 2's `df`/`lsblk` output)
is large enough that size=3 across both DCs plus backup/mirror/image
overhead is PLAUSIBLE, not exact. If the measured disk is clearly short (a
back-of-envelope check, not final math), record that now as a flag for
Stage 5's real budget pass rather than silently hoping it works out --
size=2 fallback is CONFIRMED ADOPTED as the only path if the real numbers
don't fit (D-101/Section 3), but it must be an explicit, logged decision at
the point real usage is measured (Stage 5), not guessed here.

**GATE (Section 3, second + third bullets):** MTU measured and recorded;
disk budget plausibility-checked and Ceph size decision path recorded
(size=3 default, or flagged for a size=2 fallback review at Stage 5).

---

## Step 4 -- Enable nested KVM if not already on [MUTATION: gated]

Only run this if Step 2's check showed nested virtualization is NOT already
enabled. Skip straight to Step 5 if it was already `Y`/`1`.

**MUTATION -- requires operator approval, one CPU vendor path or the other**
```bash
# Intel host:
echo "options kvm_intel nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf
sudo rmmod kvm_intel && sudo modprobe kvm_intel
# AMD host:
echo "options kvm_amd nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf
sudo rmmod kvm_amd && sudo modprobe kvm_amd
```
`rmmod`/`modprobe` fails if any VM is currently running on this host (module
busy) -- if this vcloud host has other live workloads, this step may require
a maintenance window / reboot instead. Confirm no unrelated VMs are running
first (`virsh list --all` from Step 1).

**VERIFY**
```bash
cat /sys/module/kvm_intel/parameters/nested   # or kvm_amd
```
Expect `Y`/`1`.

---

## Step 5 -- Prepare libvirt storage pool paths [MUTATION: gated]

DC1 and Office1 each need a real filesystem path for their libvirt storage
pool (`dc1_pool_path`/`office1_pool_path` in `opentofu/variables.tf` --
DC2's is deliberately not requested yet, per that file's own comment: no
CIDRs assigned, gap #3's DATA half still open even after DOCFIX-152's
mechanism closed).

**CHECK -- pick real paths with adequate free space (from Step 2's `df -h`)**
```bash
df -h /var/lib/libvirt/images 2>/dev/null || df -h /
```
Choose paths under whichever filesystem has the room computed in Step 3's
disk-budget check. Do not reuse VR0/DC0's existing pool path if this vcloud
host is the SAME physical host that ran the single-DC testcloud (check
`virsh pool-list --all` for any existing pool already pointed at a
candidate path) -- a path collision between VR0's retiring pool and VR1's
new one is exactly the kind of silent-overlap risk this repo's discipline
flags rather than assumes away.

**MUTATION**
```bash
sudo mkdir -p /path/you/chose/dc1
sudo mkdir -p /path/you/chose/office1
```
(DC2's path is not created yet -- no CIDRs, no plan to instantiate it in
`main.tf` yet either; see `opentofu/main.tf`'s commented DC2 block.)

Record the two real paths -- they go into Step 7's tfvars as
`dc1_pool_path`/`office1_pool_path`.

---

## Step 6 -- Install/confirm OpenTofu + provider registry reach

**CHECK -- is `tofu` already present?**
```bash
tofu version
```
If absent, install per OpenTofu's own official install instructions for
this host's OS (not prescribed here -- an OS-specific package/binary install
is a one-time host-prep action the operator runs directly, matching this
repo's practice of not hardcoding install mechanics that vary by distro).

**CHECK -- registry network access** (needed for `tofu init` in Step 8 to
fetch `dmacvicar/libvirt` 0.9.8 and `canonical/maas` 2.7.2)
```bash
curl -sI https://registry.opentofu.org/ | head -1
```
Expect an HTTP response (200/301/etc.), not a connection failure. If this
host is airgapped from the public internet by design (D-107's node-level
airgap applies to OpenStack NODES, not necessarily the OpenTofu control
point at Office1 -- confirm which category the vcloud host falls into
before assuming either way), a local provider mirror may be needed --
out of scope for this runbook; flag and resolve before Step 8 if so.

---

## Step 7 -- Write `opentofu/dc-dc-phase0.auto.tfvars` from measured values [repo change, gated]

Create a new tfvars file (NOT committed with real secrets -- `maas_api_key`
stays out of any file that gets committed; pass it via `TF_VAR_maas_api_key`
environment variable instead, never written to disk in the repo clone).

```hcl
# opentofu/dc-dc-phase0.auto.tfvars
# MEASURED this session (Stage 1, dc-dc-phase0-vcloud-prep.md) -- do not
# hand-edit without re-measuring; every value here traces to a specific
# Step above.

libvirt_uri       = "qemu:///system"          # or qemu+ssh://... -- Step 1
underlay_mtu      = 1500                       # or 9000 -- Step 3
dc1_pool_path     = "/path/you/chose/dc1"      # Step 5
office1_pool_path = "/path/you/chose/office1"  # Step 5

# maas_api_url / maas_api_key: see this runbook's "Known gap" section above.
# If MAAS is not yet stood up (Stage 2 hasn't run), these may need a
# placeholder per that section's guidance -- confirm the plan-time behavior
# before committing to one path.
maas_api_url = "TBD -- see Known Gap section"
```

Do NOT commit `maas_api_key` in any form. Use:
```bash
export TF_VAR_maas_api_key="<real key, once Stage 2 stands up MAAS>"
```

`domain_suffix` and `dc1_planes` already have ratified defaults in
`opentofu/variables.tf` (D-106 naming; D-101 inherited DC1 CIDRs) -- no
override needed unless you have a specific reason to deviate (log it as a
decision if so).

---

## Step 8 -- `tofu init` / `validate` / `fmt` (read-only against providers)

```bash
cd opentofu
tofu fmt -check -recursive -diff .
tofu init -backend=false -input=false
tofu validate
```
Or equivalently, from the repo root: `bash scripts/opentofu-validate.sh`.
This is the FIRST real run of this script against a real `tofu` binary --
`opentofu/README.md` has carried a **SCAFFOLD, UNVALIDATED** banner since
authoring; a clean run here is the first evidence closing that banner (do
not remove the banner from the README until this has actually run clean --
update it as part of this stage's completion, not preemptively).

If `validate` surfaces a schema mismatch against the flagged UNVERIFIED
notes in `opentofu/README.md` (the `node-vm` boot-order attribute shape is
the most likely -- though `node-vm` isn't even instantiated in `main.tf` yet
for this stage, so it wouldn't surface here; more relevant to THIS stage's
plan are `dc-planes`, `mesh-link`, `dc-storage-pool`, all previously
corrected for the attribute-vs-block syntax bug, DOCFIX-144) -- fix the
specific module, re-run, and log a changelog entry the same way prior syntax
fixes were logged this session.

---

## Step 9 -- `tofu plan` -- review before apply (READ-ONLY)

```bash
cd opentofu
tofu plan -out=phase0.tfplan
```
Review the plan output line by line against what Step 7's tfvars specify:
expect creates for `module.dc1_planes` (six `libvirt_network` resources),
`module.dc1_storage` + `module.office1_storage` (two `libvirt_pool`
resources), and `module.mesh_dc1_dc2` / `module.mesh_dc1_office1` /
`module.mesh_dc2_office1` (three more `libvirt_network` resources for the
D-100 dark-fiber triangle legs). Confirm nothing else is planned (no DC2
plane resources -- that module block is commented out in `main.tf` and
should stay that way until gap #3's DATA half closes).

**GATE:** the plan matches this expectation exactly. If it doesn't (extra
resources, missing resources, or an unexpected diff), STOP and reconcile
before Step 10 -- do not apply a plan you have not read.

---

## Step 10 -- `tofu apply` [MUTATION: creates real libvirt objects, gated]

```bash
cd opentofu
tofu apply phase0.tfplan
```
This is the FIRST live mutation this repo's DC-DC work has performed against
real infrastructure. Individually confirm this is the reviewed plan from
Step 9 (same `.tfplan` file, not re-planned) before running.

---

## Step 11 -- Post-apply verify against the Phase-0 gate (READ-ONLY)

```bash
virsh net-list --all
virsh pool-list --all
```
Expect: the six DC1 plane networks, the three mesh-link networks, and the
`dc1`/`office1` storage pools, all `active`. Cross-check each plane network's
CIDR against `opentofu/variables.tf`'s `dc1_planes` default (which mirrors
`scripts/lib-net.sh`'s `PLANE_CIDRS` -- DOCFIX-151's `lib_net_select_dc dc1`
no-op is the same six values, confirming both sources still agree).

```bash
virsh net-dumpxml <plane-network-name> | grep -i mtu
```
Confirm the MTU matches Step 3's recorded decision on every network.

**GATE (buildout-design Section 4, Phase 0):** nested KVM verified (Step 2/4);
disk budget computed and Ceph size decision recorded (Step 3); measured MTU
recorded (Step 3); virtual networks present and isolated as designed
(this step). All four true -> Stage 1 of `docs/dc-dc-deployment-workflow.md`
moves from NOT STARTED to DONE; update that doc's Stage 1 row and Section
4's tracker table accordingly, and update `opentofu/README.md`'s
SCAFFOLD/UNVALIDATED banner to reflect the first real validated run.

-> Proceed to Stage 2 (Office1 headend standup): MAAS region controller,
OpenTofu already reaching this host, NetBox, GitBucket, Tailscale.

---

## Delivery checklist (this repo's standard discipline)

- [ ] `bash scripts/repo-lint.sh` clean (0 fail) before committing any repo
      changes made while executing this runbook (e.g. the tfvars file, minus
      secrets; any module fix from Step 8).
- [ ] `bash scripts/opentofu-validate.sh` green (this IS the harness for
      `opentofu/`, per its own README).
- [ ] Changelog entry for this runbook's first real execution (next DOCFIX
      number via `bash scripts/ledger-scan.sh`), noting the ACTUAL measured
      values (host specs, MTU, pool paths -- redact nothing that isn't a
      secret; these are exactly the kind of as-built facts this repo commits).
- [ ] `docs/session-ledger.md` updated with the outcome.
- [ ] `docs/dc-dc-deployment-workflow.md` Stage 1 row and tracker table
      updated from NOT STARTED to DONE (or to whatever partial state is
      honest, if the run surfaces a blocker).
