DC-DC (VR1) deployment workflow overview

Purpose. A single followable checklist across the whole VR1 (two-DC virtual regional) buildout, spanning many sessions. docs/dc-dc-buildout-design.md is the WHY/WHAT (goals, gates, D-refs per phase); this doc is the HOW-TO-FOLLOW-IT sequence, cross-referenced to that plan, to D-100..D-110, and to what already exists in this repo vs what still needs to be authored. Update the "state" column as stages complete -- this doc is a tracker, not a one-time snapshot.

Governing constraint (unchanged throughout): minimize delta to Roosevelt. Reuse VR0 DC0 material wherever it transfers; the only deliberate no-Roosevelt-analog steps are the Section 9 shim register in the buildout design (OpenTofu node-VM creation, tc netem WAN simulation, single-unit controllers/rbd-mirror).

Stage 0 -- Decision ratification (BLOCKING)

Nothing in Stage 1 onward starts until this clears. D-100..D-110 are ALL PROPOSED, not ADOPTED (see docs/design-decisions.md).

1. Operator reviews docs/dc-dc-buildout-design.md in full.
2. Rule on each of D-100..D-110 individually (ADOPTED / AMENDED / REJECTED) -- not a blanket approval; each carries distinct, separable scope.
3. Rule on the 6 open-redline items (buildout design Section 10):
   • Ceph size=3 vs a documented size=2 fallback (gated on Phase-0 disk math)
   • tc netem link profile (same-metro lean vs a specific Roosevelt target)
   • Whether D-102 folds into D-101, and whether D-109 stands alone
   • metal-admin admin-API ULA leg (lean: no)
   • COS per-DC-only vs an Office1 roll-up (lean: per-DC only)
   • Mirror sync topology (independent upstream pull per DC vs seed-from-peer)
4. Record each ruling as an amendment in docs/design-decisions.md (same convention as every prior D-NNN ruling in this repo).

Exit gate: every D-100..D-110 is ADOPTED or explicitly deferred with a documented fallback. bash scripts/ledger-scan.sh should show none of them under "PROPOSED / OPEN decisions" (or only the explicitly-deferred ones).

State: NOT STARTED.

Stage 1 -- Phase 0: vcloud host preparation

State: NOT STARTED.

Stage 2 -- Phase 1: Office1 headend standup

State: NOT STARTED.

Stage 3 -- Phase 2: OpenTofu builds each DC substrate (from Office1)

State: NOT STARTED.

Stage 4 -- Phase 3: MAAS enlist / commission / deploy (per DC)

State: NOT STARTED.

Stage 5 -- Phase 4: Juju controller + OpenStack bundle (per DC) -- HIGH REUSE

State: NOT STARTED. This is where the existing repo tooling (preflight.sh, cloud-assert.sh, scripts/lib-net.sh/lib-hosts.sh, run-logged.sh) does the most direct work -- confirmed repo-agnostic as of the 2026-07-09 sweep, so they apply to either DC's clone/session without modification.

Stage 6 -- Phase 5: DR wiring and failover drill

State: NOT STARTED.

Stage 7 -- Phase 6: Designate, COS, Magnum/CAPI (per DC)

State: NOT STARTED.

Cross-cutting discipline (applies at every stage, every session)

• Session bootstrap, every session: git pull -> bash scripts/repo-lint.sh (0 fail / 1 documented legacy WARN expected) -> bash scripts/ledger-scan.sh to reconcile open decisions/next-free numbers against docs/session-ledger.md.
• Any session that will mutate the cloud: bash scripts/run-logged.sh <label> first (as-executed log).
• Before any add-model-like mutation: bash scripts/preflight.sh PASS.
• After any deploy/restart/change, and as an incident-triage first step: bash scripts/cloud-assert.sh (add --capture at real deploy-completion milestones for the committed BOM).
• Before touching ANY built surface: grep docs/design-decisions.md for the governing D-NNN. PROPOSED/OPEN means the operator has not ruled -- present options, do not implement.
• Every deliverable (script, runbook, doc): repo-lint clean, the touched script's harness green (write one if missing), a changelog entry with a per-item revert, and a docs/session-ledger.md update.
• Before assigning a new D-/DOCFIX-/BUNDLEFIX-NNN: bash scripts/ledger-scan.sh for next-free (it is the numbering authority, not repo-lint's informational count).
• Hard rule reminder: no inferred values -- every IP/ID/name/scope is measured this session or carried from confirmed as-built; verify before mutate, always; destructive/secret-handling steps are individually gated, never batched.

Tooling gap register (audited 2026-07-09; update as items close)

Confirmed by grepping the actual repo (scripts/, tests/, netbox/, runbooks/), not inferred from the design doc alone. Ordered roughly by "you'll hit this first," not by size -- #1 is cheap and unblocks a reuse claim made elsewhere in this doc;

#2 is the single largest build.

1. $DC selector convention (BLOCKS Stage 5's reuse claim). scripts/lib-net.sh and lib-hosts.sh are hardcoded to ONE deployment's literals (one set of six plane CIDRs, one KEYSTONE_VIP_DEFAULT, a fixed 4-host list) -- confirmed by reading both files directly, no DC parameter exists anywhere in them. Running the "reused" phase-01..08 runbooks against DC2 today would silently reuse DC1's values. Needs the same treatment as the 2026-07-09 $REPO convention: a $DC (or --dc) selector, with both libraries becoming DC-keyed lookups instead of flat constants. Do this BEFORE writing the Stage 5 runbook, or the "high reuse" claim above is not actually true operationally.
2. OpenTofu -- network/pool/node-VM/cloud-init layer scaffolded 2026-07-09 (was: zero files). opentofu/ now has: modules/dc-planes (six per-DC planes), modules/mesh-link (the D-100 dark-fiber triangle legs), modules/dc-storage-pool, modules/node-vm (blank disk, PXE-boot MAAS-managed node VM, the D-103 pattern for Stage 3), and modules/base-image + modules/cloudinit-vm (the OTHER VM pattern -- pre-built image + cloud-init, for Office1's own service VMs). DC1 + Office1 pool/plane wiring is instantiated in root main.tf; node-vm and cloudinit-vm are NOT (node sizing and Office1 VM image/config choices are both pending decisions -- neither invented here). Provider schema was confirmed against the actual current docs AND real example .tf files (not memory) -- it differs materially from older common examples: nested structures are attribute-style objects (key = { ... }), not classic HCL blocks. A real bug from the first pass was found and fixed: dc-planes/mesh-link/dc-storage-pool originally used block syntax for libvirt_network's domain/mtu and libvirt_pool's target; corrected once real provider examples surfaced the actual convention -- discovered precisely BECAUSE building the domain module forced fetching real example code instead of doc-summaries alone. Still missing, deliberately deferred (see opentofu/README.md): OPNsense's own bootability via cloudinit-vm is NOT confirmed (FreeBSD, not the same cloud-init/NoCloud guarantee as Linux images); the per-device boot-order attribute's exact internal shape (flagged UNVERIFIED in node-vm/main.tf, an inference by pattern not a confirmed field); and tc netem application. UNVALIDATED -- no tofu binary was available to run scripts/opentofu-validate.sh this session; run it on a connected machine before trusting any of this.
3. NetBox multi-DC/dual-stack import pipeline. netbox/ has exactly two scripts (ipv4-prefixes-import.py, ipv6-mark-reserved.py) -- single-DC v4 import and v6-reservation-marking only. D-101 requires DC2's supernet, the org ULA /48, and per-DC GUA carves to be NetBox-authoritative via "the extended import pipeline" -- that pipeline is not written. Blocks Stage 0's exit condition (literals must be real, not placeholders) and Stage 2.
4. OPNsense edge deployment/config, and netem WAN-simulation -- (b)/(c) were RESEARCHED 2026-07-09 (opentofu/README.md's dedicated section, sourced against OPNsense's own docs/forum/GitHub), (d) is still a real gap. Each site (Office1, DC1, DC2) gets its own independent OPNsense edge with a simulated ISP uplink (D-100); OpenTofu owns creating the edge VMs (D-103). Blocks Stage 1 (Office1 edge) and Stage 3 (per-DC edges) gates, which require netem parameters "applied and measured."
   • (a) Network substrate -- DONE. opentofu/modules/mesh-link creates the inter-site L2 segments (DC1<->DC2, DC1<->Office1, DC2<->Office1) an OPNsense VM's interfaces would eventually attach to.
   • (b)+(c) The OPNsense VM/image AND its config -- RESEARCHED, confirmed as ONE pattern, not built yet. Cloud-init is confirmed unreliable on OPNsense (FreeBSD) -- OPNsense's own forum consensus, not just a hedge. The real native mechanism is the Configuration Importer (official docs): scans an attached volume for /conf/config.xml in a 2-3 second boot-time window; ISO9660 support was added to it specifically for VM/cloud automation (attach a config ISO as a secondary CD-ROM) -- mechanically IDENTICAL to modules/cloudinit-vm's shape (a secondary volume attached as cdrom), just a different payload (a plain ISO9660 image with /conf/config.xml, not a NoCloud seed). Use the nano image (pre-installed, serial-ready), not the installer images. Still needed to actually build it: a preprocessing script for the nano image's decompress+convert (OpenTofu's volume-download mechanism won't do that itself), a way to build the config.xml ISO, and the actual config.xml content (real design work). No image source formally chosen yet either. Full sourcing in opentofu/README.md.
   • (d) tc netem WAN-simulation -- a wholly separate mechanism, not an OPNsense setting. No Terraform/OpenTofu-native equivalent exists at all (confirmed against the libvirt_network schema, gap #2); it is a Linux tc qdisc setting applied to the host-side bridge interface after the network exists, needing either a provisioner or a small companion script -- neither written. The exact parameters (latency/jitter/loss/ rate) are themselves still an unruled D-100 open sub-item (gap #11), so this needs a decision before it can be scripted, not just scripting.
5. Ceph replication tooling (radosgw multisite + rbd-mirror). No script in the repo touches either (checked; osd-blank-check.sh is unrelated). D-108's entire mechanism -- zonegroup setup, rbd-mirror daemon peering, the failover/failback drill commands -- is net-new. Blocks Stage 6.
6. Designate reactivation is a bundle change, not just a runbook. runbooks/phase-01-bundle-deploy.md states the bundle explicitly ships "NO designate (D-019)" -- reactivating it per D-106 means editing bundle.yaml itself (a built-surface change, gated behind D-106 leaving PROPOSED), in addition to writing the reactivation runbook. Blocks Stage 7.
7. MTU/geneve budget + Ceph disk-budget calculators. D-102 and buildout-design Section 3 both describe measurement-then-computation gates (host L2 MTU -> geneve-over-v6 overhead; measured per-node OSD footprint -> size=3 vs size=2 fallback) that are manual arithmetic in prose today, not a script. Blocks the Stage 1 Phase-0 gates.
8. The "developed IPv6 plan" D-101 cites does not exist in this repo. D-101's context line references "the developed family-follows-reachability IPv6 plan (in-project, not yet committed)" as prior art -- grepped the whole repo, it is not present as a standalone doc anywhere. If the exact Phase A/B/C detail behind D-101/D-102 is needed by a future session, it currently is NOT retrievable from the repo alone. Not a blocker for Stage 0 ratification of what IS written, but a real content gap if finer detail is needed later.
9. Runbook debt. Confirmed zero DC-DC-specific runbooks exist. Stage 1, 2, 3, 6 need net-new runbooks; Stage 5 and 7 need an adaptation pass over phase-01..08 / phase-06/07/08 (Stage 5's pass is gated on #1 above).
10. Harness debt. This repo's own delivery rule -- no script change ships without its tests/<name>/run-tests.sh harness -- applies to everything in #2-7 above. None exist yet because none of those scripts exist yet; expect roughly 5-6 new test suites alongside whatever gets built.
11. Decision sub-items still "leaning," not ruled, that block the runbooks depending on them even after Stage 0 closes the big items: exact netem parameters (D-100), exact NetBox literals for DC2/ULA/GUA (D-101), whether D-102 folds into D-101, COS per-DC-vs-Office1-rollup (D-105), mirror sync topology (D-107).

Where things stand today (2026-07-09) -- update this as stages close