diff --git a/docs/changelog-20260709-dc-dc-phase0-runbook.md b/docs/changelog-20260709-dc-dc-phase0-runbook.md new file mode 100644 index 0000000..ced3740 --- /dev/null +++ b/docs/changelog-20260709-dc-dc-phase0-runbook.md @@ -0,0 +1,75 @@ +# Changelog 2026-07-09 -- Stage 1 runbook: Phase 0 vcloud host prep (DOCFIX-153) + +No live infrastructure touched -- new runbook document only. Operator +clarified this session runs from the workstation (not the vopenstack-jesse +jumphost or the vcloud host); tonight's scope is prep so tomorrow morning's +switch to the vcloud host can execute against real infrastructure +immediately. This is the FIRST runbook that will be, since Stage 0 +(ratification) is a decision-record exercise, not an infrastructure step. + +## Item + +### 1. DOCFIX-153 -- `runbooks/dc-dc-phase0-vcloud-prep.md` +FILES: `runbooks/dc-dc-phase0-vcloud-prep.md` (new), `docs/dc-dc-deployment- +workflow.md` (Stage 1 row + tracker table updated). + +WHAT: command-level runbook for buildout-design Section 4's Phase 0 (turn +the bare vcloud host into a substrate Office1 can deploy from), matching +this repo's established runbook conventions (see `runbooks/phase-00- +teardown-maas-reset.md` as the style template): a sequence table, read-only +CHECK blocks before every mutation, each MUTATION individually gated, an +explicit gate section at the end, and a delivery checklist. + +Eleven steps: identify the target host + confirm libvirt reach; measure +CPU/RAM/disk and CPU virtualization-extension support; measure host L2 MTU +and record the D-101 MTU decision (jumbo vs. 1500-with-reduced-tenant-MTU) +plus the Ceph size=3-vs-size=2 disk-budget plausibility check (Section 3's +three Phase-0 gates); enable nested KVM if not already on; prepare libvirt +storage pool paths for DC1 and Office1 (explicitly NOT DC2 -- no CIDRs +assigned yet, gap #3's data half); install/confirm OpenTofu and registry +reach; write a `dc-dc-phase0.auto.tfvars` file from the session's measured +values (with `maas_api_key` explicitly kept OUT of any committed file, via +`TF_VAR_maas_api_key` instead); `tofu init`/`validate`/`fmt`; `tofu plan` +reviewed line-by-line against an explicit expected-resource list before +`tofu apply`; post-apply verification against the Phase-0 gate criteria. + +**Genuine open question surfaced and flagged, not silently resolved:** +`opentofu/main.tf` declares the MAAS provider block unconditionally, even +though this Phase-0 stage's plan only touches libvirt resources (planes, +pools, mesh links) -- no MAAS resource is instantiated until Stage 3. It is +not yet known whether `tofu init`/`plan` will demand real +`maas_api_url`/`maas_api_key` values before Stage 2 has even stood up MAAS. +The runbook documents both possible outcomes and what to do for each +(placeholder values IF plan-time validation only checks non-null, vs. a +logged structural finding -- splitting the MAAS provider into a Stage-3-only +root module -- if OpenTofu actually attempts real connectivity at plan +time with zero MAAS resources). This is exactly the kind of thing this +runbook is supposed to catch by being read carefully before tomorrow's +execution, rather than being discovered live. + +**Also flagged, not silently assumed:** if the vcloud host tomorrow is the +SAME physical host that ran the VR0/DC0 single-DC testcloud (buildout-design's +own Ground Truth section implies this -- the "~256 vCPU/1 TiB/1 TiB disk +dedicated to VR1" host is a repurposing, not a new machine), Step 5 tells the +operator to check for an existing libvirt storage pool at any candidate path +before creating DC1's pool there, to avoid a silent path collision with +VR0's own pool. + +VERIFIED: `bash scripts/repo-lint.sh` 0 fail, 1 documented legacy warn +(unchanged -- prose-only addition, no script logic touched). This runbook +itself is NOT YET EXECUTED -- that is tomorrow morning's work, per the +operator's own plan; nothing in this delivery claims otherwise. + +REVERT: `rm runbooks/dc-dc-phase0-vcloud-prep.md && git checkout HEAD~ -- +docs/dc-dc-deployment-workflow.md` (safe -- no infrastructure has been +touched by this delivery, purely a document). + +## Next actionable step + +This runbook is the hand-off artifact for tomorrow morning's switch to the +vcloud host. Stage 2 (Office1 headend standup) is the next runbook to draft +tonight, continuing while the operator is away -- its content depends on +Stage 1 actually having run (real libvirt reach, pool paths, MTU decision) +only loosely; it can be drafted in parallel since MAAS-region/NetBox/ +GitBucket/Tailscale standup on Office1 is a mostly independent set of +steps once OpenTofu itself is confirmed working. diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md index f3b5932..255fbe6 100644 --- a/docs/dc-dc-deployment-workflow.md +++ b/docs/dc-dc-deployment-workflow.md @@ -41,9 +41,9 @@ | **Gate** | Nested KVM verified; disk budget computed + Ceph size decision recorded; measured MTU recorded; virtual networks present and isolated as designed. | | **Owns** | D-100 (fabric), D-101 (MTU -- folds in former D-102), buildout-design Section 3 (host/disk/MTU gates). | | **Reuse vs new** | NEW. No VR0-DC0 analog -- that testcloud IS the single vcloud host, already prepared; this is preparing the host to carry TWO independent DC substrates plus Office1. | -| **Authoring status** | Runbook not started. `opentofu/` SCAFFOLD started 2026-07-09: `modules/dc-planes` (six per-DC planes) + `modules/mesh-link` (dark-fiber legs) + `modules/dc-storage-pool` wired for DC1 + Office1 -- see `opentofu/README.md` for what's built vs deliberately deferred (node-VM/domain resources, DC2 CIDRs, netem). UNVALIDATED: no `tofu` binary available to run `scripts/opentofu-validate.sh` yet. | +| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase0-vcloud-prep.md`** -- command-level steps (host identify -> measure CPU/RAM/disk/nested-KVM/MTU -> enable nested KVM -> prepare pool paths -> install/confirm OpenTofu -> tfvars -> init/validate/plan/apply -> post-apply gate verify), each MUTATION individually gated per this repo's discipline. Flags a real open structural question up front (the `provider "maas"` block in `main.tf` is unconditional even though this stage only touches libvirt resources -- may force `maas_api_url`/`maas_api_key` to be set at `plan` time with nothing yet using them; noted as a possible DOCFIX candidate, not silently worked around). NOT YET EXECUTED against real infrastructure -- this is the first runbook that will be, tomorrow morning, per the operator's own stated plan. `opentofu/` SCAFFOLD started 2026-07-09: `modules/dc-planes` (six per-DC planes) + `modules/mesh-link` (dark-fiber legs) + `modules/dc-storage-pool` wired for DC1 + Office1 -- see `opentofu/README.md` for what's built vs deliberately deferred (node-VM/domain resources, DC2 CIDRs, netem). UNVALIDATED: no `tofu` binary available to run `scripts/opentofu-validate.sh` yet -- this runbook's Step 8 is the first real run. | -**State:** NOT STARTED. +**State:** RUNBOOK WRITTEN, NOT YET EXECUTED. --- @@ -325,7 +325,7 @@ | Design (`dc-dc-buildout-design.md`) | DONE, PROPOSED | | Decisions (D-100..D-110) | **ADOPTED 2026-07-09** (D-102 merged into D-101; D-109 standalone) -- Stage 0 CLEARED | | DR mechanism seed | DONE (`dc-dc-replication-DR-seed.md`), superseded-carrier corrected into D-108 | -| Stage 0-7 runbooks | NOT YET WRITTEN -- see Tooling gap register above for the itemized breakdown | +| Stage 0-7 runbooks | Stage 0 DONE (ratification). **Stage 1 WRITTEN 2026-07-09** (`runbooks/dc-dc-phase0-vcloud-prep.md`), not yet executed. Stages 2-7 NOT YET WRITTEN -- see Tooling gap register above for the itemized breakdown | | `opentofu/` (networks/pools/node-VM PXE/cloud-init-VM patterns) | SCAFFOLDED 2026-07-09, UNVALIDATED (no `tofu` binary available this session) -- see gap #2 and `opentofu/README.md` | | OPNsense image+config mechanism (`modules/opnsense-edge`) | BUILT 2026-07-09, UNVALIDATED (no `tofu`/ISO-tool/real boot to confirm), see `opentofu/README.md` | | OPNsense config.xml design (`templates/opnsense-config.xml.tmpl`) | BUILT 2026-07-09, renderer fully tested (8/8) -- several tokens still pending Stage 0 ratification or a real boot, see `opentofu/templates/README.md` | diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 2d0b6ff..6cd9966 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -21,8 +21,8 @@ ## Machine-derived (re-seed from `scripts/ledger-scan.sh`; do not hand-edit) -_As of 2026-07-09 scan (post-DOCFIX-152, NetBox DC-DC pipeline mechanism -closed); re-run the scan to refresh:_ +_As of 2026-07-09 scan (post-DOCFIX-153, Stage 1 runbook written); re-run +the scan to refresh:_ - **PROPOSED / OPEN decisions:** D-068 (Vault substrate hardening, Roosevelt -- 1.16 ruled out per amendment; off-EOL path OPEN), D-071 (routine update cadence + @@ -37,8 +37,8 @@ - **OPEN security rows:** SEC-001 (rotate at v1 close, re-ruled 2026-07-06), SEC-003 (custodians + second-person unseal -- DEFERRED; keeps d011-06 MANUAL), SEC-004 (flip repo to private at v1 close -- deferral reaffirmed). -- **Next-free numbers:** D = 111, DOCFIX = 153, BUNDLEFIX = 012. - (Seeded from the 2026-07-09 scan, post DOCFIX-152.) +- **Next-free numbers:** D = 111, DOCFIX = 154, BUNDLEFIX = 012. + (Seeded from the 2026-07-09 scan, post DOCFIX-153.) --- @@ -978,6 +978,22 @@ reachable this session). This closes the TOOLING half of gap #3 only -- the literals themselves (ULA /48, GUA carve, DC2 supernet) still need real assignment before this script can be run for real. +- **DONE -- Stage 1 runbook WRITTEN 2026-07-09 (changelog + `docs/changelog-20260709-dc-dc-phase0-runbook.md`, DOCFIX-153):** + `runbooks/dc-dc-phase0-vcloud-prep.md` -- command-level Phase 0 (vcloud + host prep) steps, matching the phase-00 runbook's style (sequence table, + read-only CHECK before every mutation, individually gated mutations, an + explicit gate section, delivery checklist). Surfaces a genuine open + structural question rather than silently resolving it: `opentofu/main.tf`'s + unconditional `provider "maas"` block may force MAAS credentials to be set + at `tofu plan` time even though this stage's plan touches only libvirt + resources -- documented both possible outcomes and what to do for each. + Also flags the VR0/DC0-pool-path-collision risk if the vcloud host is the + same physical machine being repurposed (buildout design's own framing). + NOT YET EXECUTED -- this is the runbook the operator will run tomorrow + morning after switching to the vcloud host; nothing in this delivery + claims a real run happened. repo-lint 0 fail/1 documented warn (unchanged, + prose-only). diff --git a/runbooks/dc-dc-phase0-vcloud-prep.md b/runbooks/dc-dc-phase0-vcloud-prep.md new file mode 100644 index 0000000..bb1f463 --- /dev/null +++ b/runbooks/dc-dc-phase0-vcloud-prep.md @@ -0,0 +1,415 @@ +# DC-DC Phase 0 -- vcloud host preparation (Stage 1) + +Turn the bare vcloud host into a substrate Office1 can deploy VR1 (DC1, DC2, +Office1) from. This is the FIRST DC-DC runbook executed against real +infrastructure -- everything before this session was repo-only prep (docs, +scripts, OpenTofu modules, all UNVALIDATED against a real `tofu` binary or +libvirt connection). Run from the vcloud host itself, or from a machine with +`libvirt_uri` SSH reach to it -- confirm which before starting (Step 1). + +**Governing docs:** `docs/dc-dc-buildout-design.md` Section 3 (the gates this +runbook satisfies) and Section 4 Phase 0 (goal/build/gate summary); +`docs/dc-dc-deployment-workflow.md` Stage 1 (tracker row -- update its +`**State:**` line when this runbook completes a real run); `opentofu/README.md` +(module scope/status -- **SCAFFOLD, UNVALIDATED** as of this writing: no +`tofu` binary has run against these modules yet. This runbook's Step 4 is the +FIRST time that changes). + +Decisions this runbook owns: **D-100** (fabric: the dark-fiber mesh + per-site +ISP-edge topology), **D-101** (MTU sub-policy, folded in from D-102 -- the +geneve-over-v6 budget math in Step 3), and buildout-design Section 3's three +Phase-0 gates (nested KVM, Ceph size decision, MTU). + +!!! Every value below marked MEASURE is filled in AT RUNTIME from this + session's real output -- never carried over from the buildout design's + ~256 vCPU / 1 TiB / 10 TiB planning estimate, which is a PLANNING NUMBER, + not a measurement. If the real host doesn't match that estimate, the + Ceph size=3-vs-size=2 arithmetic in Step 3 changes accordingly -- that is + the gate working as designed, not a problem to paper over. + +--- + +## Known gap, flagged before you start (not fixed in this runbook) + +`opentofu/main.tf` declares `provider "maas" { api_url = var.maas_api_url; +api_key = var.maas_api_key }` UNCONDITIONALLY at the root, even though this +Phase-0 stage only touches `libvirt`-backed resources (planes, storage pools, +mesh links) -- no MAAS resource is instantiated yet (Stage 3's job). Depending +on how strictly OpenTofu validates a declared-but-unused provider block, `tofu +init`/`plan` in Step 4 MAY demand `maas_api_url`/`maas_api_key` be set even +though nothing in this stage's plan uses MAAS. If you hit that: +- Cheapest workaround: pass placeholder-shaped-but-clearly-fake values (e.g. + `maas_api_url=http://not-yet-provisioned.invalid:5240/MAAS`) ONLY if OpenTofu + merely wants the variable non-null and does not attempt to actually reach + it during `plan` for a provider with zero resources. Verify this is true + (a `plan`, not `apply`, should never open a real connection) before relying + on it. +- If OpenTofu actually attempts to validate MAAS reachability at `plan` time + even with zero MAAS resources, that is a real structural finding -- LOG it + (a DOCFIX candidate: split the MAAS provider block into a Stage-3-only root + module) rather than working around it by fabricating real-looking + credentials or skipping `tofu validate`. + +--- + +## Sequence + +``` +1. Identify the target host + confirm reach (read-only) +2. Measure: CPU/RAM/disk, nested-KVM support (read-only) +3. Measure: L2 MTU; compute Ceph size + MTU decisions (read-only + a recorded decision) +4. Enable nested KVM if not already on [MUTATION: kernel module param, gated] +5. Prepare libvirt storage pool paths [MUTATION: mkdir, gated] +6. Install/confirm OpenTofu + provider registry reach (read-only once installed) +7. Write opentofu/dc-dc-phase0.auto.tfvars from measured values (repo change, gated) +8. tofu init / validate / fmt (read-only against providers) +9. tofu plan -- review before apply (read-only) +10. tofu apply -- DC1 planes + Office1 pool + mesh links [MUTATION: creates libvirt objects, gated] +11. Post-apply verify against the Phase-0 gate (read-only) + -> EXIT GATE -> Stage 2 (Office1 headend standup) +``` + +--- + +## Step 1 -- Identify the target host + confirm reach (READ-ONLY) + +**CHECK -- from wherever you are running this session** +```bash +hostname +whoami +uname -a +``` +Record the actual hostname/user -- this is the value that goes into +`libvirt_uri` in Step 7 (e.g. `qemu:///system` if running directly on the +vcloud host as a user in the `libvirt` group, or `qemu+ssh://@/system` +if running remotely). Do NOT assume the shape from any other repo's +`VIRSH_POWER_ADDRESS` convention (`scripts/lib-hosts.sh`'s DC0 value is a +DIFFERENT host/connection -- see that file's own DC-selector comments, +DOCFIX-151) -- this is a distinct endpoint, confirmed here, not carried over. + +**CHECK -- libvirt reachable at all** +```bash +virsh -c qemu:///system list --all # if running ON the vcloud host +# or, if remote: +# virsh -c qemu+ssh://@/system list --all +``` +Expect: a (possibly empty) domain list with no connection error. If this +fails, stop -- nothing past this point works without it. + +--- + +## Step 2 -- Measure CPU/RAM/disk + nested-KVM support (READ-ONLY) + +**CHECK -- host resources** +```bash +nproc --all +free -h +lsblk -o NAME,SIZE,TYPE,MOUNTPOINT +df -h +``` +Record the REAL numbers. The buildout design's "~256 vCPU / 1 TiB RAM / 10 TiB +disk" is a planning estimate (`docs/dc-dc-buildout-design.md` Section 3) -- +if actual capacity differs, Step 3's disk-budget arithmetic uses the MEASURED +number, not the plan's. + +**CHECK -- CPU virtualization extensions present** +```bash +egrep -c '(vmx|svm)' /proc/cpuinfo +``` +Expect: a non-zero count. `vmx` = Intel VT-x, `svm` = AMD-V. Record which one +-- it determines which kernel module's `nested` parameter Step 4 touches. + +**CHECK -- nested virtualization already enabled?** +```bash +# Intel: +cat /sys/module/kvm_intel/parameters/nested 2>/dev/null +# AMD: +cat /sys/module/kvm_amd/parameters/nested 2>/dev/null +``` +Expect `Y` or `1` if already on. If the file doesn't exist for your CPU +vendor, the `kvm_intel`/`kvm_amd` module may not be loaded yet -- `lsmod | +grep kvm` to check, and consult your distro's docs for loading it before +proceeding (out of scope for this runbook to prescribe -- this is a one-time +host-OS-level fact to confirm, not a VR1-specific step). + +**GATE (Section 3, first bullet):** nested KVM is either already enabled +(skip Step 4) or CPU-capable and not yet enabled (Step 4 turns it on). If the +CPU shows zero `vmx`/`svm` support, STOP -- this host cannot run nested KVM at +all, and the entire VR1 virtual-regional design (DC node VMs running their +own nested libvirt/OpenStack) is not viable on it. That is a blocking finding +for the operator, not something to route around. + +--- + +## Step 3 -- Measure L2 MTU; record the Ceph size + MTU decisions (READ-ONLY + a recorded decision) + +**CHECK -- host L2 MTU on the interface(s) VR1 will use** +```bash +ip -o link show | awk '{print $2, $0}' | grep -i mtu +``` +Identify which interface(s) carry the vcloud host's real uplink (the one +OpenTofu's virtual networks will ultimately ride over, or bridge to, for the +simulated ISP edges) and record its MTU. Per D-101 (folded in from D-102): +"Prefer jumbo (9000) end-to-end... if pinned at 1500, set the reduced tenant +MTU consistently." **Do not assume jumbo** -- use the measured value. + +**Decision to record (D-101 MTU sub-policy, buildout-design Section 3):** +- If measured MTU >= 9000: set `underlay_mtu = 9000` in Step 7's tfvars. + Tenant MTU stays 1500; geneve/amphora headroom is generous. +- If measured MTU == 1500 (the common case for a plain host NIC): set + `underlay_mtu = 1500`, and record that tenant-network MTU will need to + drop to ~1444 (v6-geneve overhead: 40 IPv6 + 8 UDP + 8 Geneve base = 56 + bytes, before nested-virt/options) -- this propagates to ovn geneve, + tenant-network MTU, and amphora settings in later stages (Stage 5/7), not + this one. Just RECORD the decision here; the propagation happens where + those components are actually configured. + +**Decision to record (Ceph size=3 vs size=2, Section 3 + the 2026-07-09 +Stage-0 ruling):** target is size=3/min_size=2 BY DEFAULT (ADOPTED, not +still open). This is conditional on the measured disk budget from Step 2: +``` +required_disk_TiB ~= (DC1_Ceph_size3 + DC2_Ceph_size3 + per-DC_radosgw_backup_pools + + Glance_rbd-mirror_targets + images + per-DC_artifact_mirrors) +``` +The exact per-node OSD footprint isn't knowable until DC1's actual Ceph +cluster exists and reports real usage (Stage 5) -- for THIS stage, the gate +is narrower: confirm the MEASURED total disk (Step 2's `df`/`lsblk` output) +is large enough that size=3 across both DCs plus backup/mirror/image +overhead is PLAUSIBLE, not exact. If the measured disk is clearly short (a +back-of-envelope check, not final math), record that now as a flag for +Stage 5's real budget pass rather than silently hoping it works out -- +size=2 fallback is CONFIRMED ADOPTED as the only path if the real numbers +don't fit (D-101/Section 3), but it must be an explicit, logged decision at +the point real usage is measured (Stage 5), not guessed here. + +**GATE (Section 3, second + third bullets):** MTU measured and recorded; +disk budget plausibility-checked and Ceph size decision path recorded +(size=3 default, or flagged for a size=2 fallback review at Stage 5). + +--- + +## Step 4 -- Enable nested KVM if not already on [MUTATION: gated] + +Only run this if Step 2's check showed nested virtualization is NOT already +enabled. Skip straight to Step 5 if it was already `Y`/`1`. + +**MUTATION -- requires operator approval, one CPU vendor path or the other** +```bash +# Intel host: +echo "options kvm_intel nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf +sudo rmmod kvm_intel && sudo modprobe kvm_intel +# AMD host: +echo "options kvm_amd nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf +sudo rmmod kvm_amd && sudo modprobe kvm_amd +``` +`rmmod`/`modprobe` fails if any VM is currently running on this host (module +busy) -- if this vcloud host has other live workloads, this step may require +a maintenance window / reboot instead. Confirm no unrelated VMs are running +first (`virsh list --all` from Step 1). + +**VERIFY** +```bash +cat /sys/module/kvm_intel/parameters/nested # or kvm_amd +``` +Expect `Y`/`1`. + +--- + +## Step 5 -- Prepare libvirt storage pool paths [MUTATION: gated] + +DC1 and Office1 each need a real filesystem path for their libvirt storage +pool (`dc1_pool_path`/`office1_pool_path` in `opentofu/variables.tf` -- +DC2's is deliberately not requested yet, per that file's own comment: no +CIDRs assigned, gap #3's DATA half still open even after DOCFIX-152's +mechanism closed). + +**CHECK -- pick real paths with adequate free space (from Step 2's `df -h`)** +```bash +df -h /var/lib/libvirt/images 2>/dev/null || df -h / +``` +Choose paths under whichever filesystem has the room computed in Step 3's +disk-budget check. Do not reuse VR0/DC0's existing pool path if this vcloud +host is the SAME physical host that ran the single-DC testcloud (check +`virsh pool-list --all` for any existing pool already pointed at a +candidate path) -- a path collision between VR0's retiring pool and VR1's +new one is exactly the kind of silent-overlap risk this repo's discipline +flags rather than assumes away. + +**MUTATION** +```bash +sudo mkdir -p /path/you/chose/dc1 +sudo mkdir -p /path/you/chose/office1 +``` +(DC2's path is not created yet -- no CIDRs, no plan to instantiate it in +`main.tf` yet either; see `opentofu/main.tf`'s commented DC2 block.) + +Record the two real paths -- they go into Step 7's tfvars as +`dc1_pool_path`/`office1_pool_path`. + +--- + +## Step 6 -- Install/confirm OpenTofu + provider registry reach + +**CHECK -- is `tofu` already present?** +```bash +tofu version +``` +If absent, install per OpenTofu's own official install instructions for +this host's OS (not prescribed here -- an OS-specific package/binary install +is a one-time host-prep action the operator runs directly, matching this +repo's practice of not hardcoding install mechanics that vary by distro). + +**CHECK -- registry network access** (needed for `tofu init` in Step 8 to +fetch `dmacvicar/libvirt` 0.9.8 and `canonical/maas` 2.7.2) +```bash +curl -sI https://registry.opentofu.org/ | head -1 +``` +Expect an HTTP response (200/301/etc.), not a connection failure. If this +host is airgapped from the public internet by design (D-107's node-level +airgap applies to OpenStack NODES, not necessarily the OpenTofu control +point at Office1 -- confirm which category the vcloud host falls into +before assuming either way), a local provider mirror may be needed -- +out of scope for this runbook; flag and resolve before Step 8 if so. + +--- + +## Step 7 -- Write `opentofu/dc-dc-phase0.auto.tfvars` from measured values [repo change, gated] + +Create a new tfvars file (NOT committed with real secrets -- `maas_api_key` +stays out of any file that gets committed; pass it via `TF_VAR_maas_api_key` +environment variable instead, never written to disk in the repo clone). + +```hcl +# opentofu/dc-dc-phase0.auto.tfvars +# MEASURED this session (Stage 1, dc-dc-phase0-vcloud-prep.md) -- do not +# hand-edit without re-measuring; every value here traces to a specific +# Step above. + +libvirt_uri = "qemu:///system" # or qemu+ssh://... -- Step 1 +underlay_mtu = 1500 # or 9000 -- Step 3 +dc1_pool_path = "/path/you/chose/dc1" # Step 5 +office1_pool_path = "/path/you/chose/office1" # Step 5 + +# maas_api_url / maas_api_key: see this runbook's "Known gap" section above. +# If MAAS is not yet stood up (Stage 2 hasn't run), these may need a +# placeholder per that section's guidance -- confirm the plan-time behavior +# before committing to one path. +maas_api_url = "TBD -- see Known Gap section" +``` + +Do NOT commit `maas_api_key` in any form. Use: +```bash +export TF_VAR_maas_api_key="" +``` + +`domain_suffix` and `dc1_planes` already have ratified defaults in +`opentofu/variables.tf` (D-106 naming; D-101 inherited DC1 CIDRs) -- no +override needed unless you have a specific reason to deviate (log it as a +decision if so). + +--- + +## Step 8 -- `tofu init` / `validate` / `fmt` (read-only against providers) + +```bash +cd opentofu +tofu fmt -check -recursive -diff . +tofu init -backend=false -input=false +tofu validate +``` +Or equivalently, from the repo root: `bash scripts/opentofu-validate.sh`. +This is the FIRST real run of this script against a real `tofu` binary -- +`opentofu/README.md` has carried a **SCAFFOLD, UNVALIDATED** banner since +authoring; a clean run here is the first evidence closing that banner (do +not remove the banner from the README until this has actually run clean -- +update it as part of this stage's completion, not preemptively). + +If `validate` surfaces a schema mismatch against the flagged UNVERIFIED +notes in `opentofu/README.md` (the `node-vm` boot-order attribute shape is +the most likely -- though `node-vm` isn't even instantiated in `main.tf` yet +for this stage, so it wouldn't surface here; more relevant to THIS stage's +plan are `dc-planes`, `mesh-link`, `dc-storage-pool`, all previously +corrected for the attribute-vs-block syntax bug, DOCFIX-144) -- fix the +specific module, re-run, and log a changelog entry the same way prior syntax +fixes were logged this session. + +--- + +## Step 9 -- `tofu plan` -- review before apply (READ-ONLY) + +```bash +cd opentofu +tofu plan -out=phase0.tfplan +``` +Review the plan output line by line against what Step 7's tfvars specify: +expect creates for `module.dc1_planes` (six `libvirt_network` resources), +`module.dc1_storage` + `module.office1_storage` (two `libvirt_pool` +resources), and `module.mesh_dc1_dc2` / `module.mesh_dc1_office1` / +`module.mesh_dc2_office1` (three more `libvirt_network` resources for the +D-100 dark-fiber triangle legs). Confirm nothing else is planned (no DC2 +plane resources -- that module block is commented out in `main.tf` and +should stay that way until gap #3's DATA half closes). + +**GATE:** the plan matches this expectation exactly. If it doesn't (extra +resources, missing resources, or an unexpected diff), STOP and reconcile +before Step 10 -- do not apply a plan you have not read. + +--- + +## Step 10 -- `tofu apply` [MUTATION: creates real libvirt objects, gated] + +```bash +cd opentofu +tofu apply phase0.tfplan +``` +This is the FIRST live mutation this repo's DC-DC work has performed against +real infrastructure. Individually confirm this is the reviewed plan from +Step 9 (same `.tfplan` file, not re-planned) before running. + +--- + +## Step 11 -- Post-apply verify against the Phase-0 gate (READ-ONLY) + +```bash +virsh net-list --all +virsh pool-list --all +``` +Expect: the six DC1 plane networks, the three mesh-link networks, and the +`dc1`/`office1` storage pools, all `active`. Cross-check each plane network's +CIDR against `opentofu/variables.tf`'s `dc1_planes` default (which mirrors +`scripts/lib-net.sh`'s `PLANE_CIDRS` -- DOCFIX-151's `lib_net_select_dc dc1` +no-op is the same six values, confirming both sources still agree). + +```bash +virsh net-dumpxml | grep -i mtu +``` +Confirm the MTU matches Step 3's recorded decision on every network. + +**GATE (buildout-design Section 4, Phase 0):** nested KVM verified (Step 2/4); +disk budget computed and Ceph size decision recorded (Step 3); measured MTU +recorded (Step 3); virtual networks present and isolated as designed +(this step). All four true -> Stage 1 of `docs/dc-dc-deployment-workflow.md` +moves from NOT STARTED to DONE; update that doc's Stage 1 row and Section +4's tracker table accordingly, and update `opentofu/README.md`'s +SCAFFOLD/UNVALIDATED banner to reflect the first real validated run. + +-> Proceed to Stage 2 (Office1 headend standup): MAAS region controller, +OpenTofu already reaching this host, NetBox, GitBucket, Tailscale. + +--- + +## Delivery checklist (this repo's standard discipline) + +- [ ] `bash scripts/repo-lint.sh` clean (0 fail) before committing any repo + changes made while executing this runbook (e.g. the tfvars file, minus + secrets; any module fix from Step 8). +- [ ] `bash scripts/opentofu-validate.sh` green (this IS the harness for + `opentofu/`, per its own README). +- [ ] Changelog entry for this runbook's first real execution (next DOCFIX + number via `bash scripts/ledger-scan.sh`), noting the ACTUAL measured + values (host specs, MTU, pool paths -- redact nothing that isn't a + secret; these are exactly the kind of as-built facts this repo commits). +- [ ] `docs/session-ledger.md` updated with the outcome. +- [ ] `docs/dc-dc-deployment-workflow.md` Stage 1 row and tracker table + updated from NOT STARTED to DONE (or to whatever partial state is + honest, if the run surfaces a blocker).