DC-DC (VR1) deployment workflow overview

Purpose. A single followable checklist across the whole VR1 (two-DC virtual regional) buildout, spanning many sessions. docs/dc-dc-buildout-design.md is the WHY/WHAT (goals, gates, D-refs per phase); this doc is the HOW-TO-FOLLOW-IT sequence, cross-referenced to that plan, to D-100..D-110, and to what already exists in this repo vs what still needs to be authored. Update the "state" column as stages complete -- this doc is a tracker, not a one-time snapshot.

Governing constraint (unchanged throughout): minimize delta to Roosevelt. Reuse VR0 DC0 material wherever it transfers; the only deliberate no-Roosevelt-analog steps are the Section 9 shim register in the buildout design (OpenTofu node-VM creation, tc netem WAN simulation, single-unit controllers/rbd-mirror).

Stage 0 -- Decision ratification (CLEARED 2026-07-09)

RATIFIED. D-100..D-110 are ALL ADOPTED (docs/design-decisions.md; D-102 MERGED into D-101 per the operator's own ruling, not deleted -- retained for numbering/history; D-109 confirmed standalone). All 6 redline items ruled (docs/dc-dc-buildout-design.md Section 10 is now a ruling record, not an open list): Ceph size=3-by-default (size=2 only as an explicit Phase-0 fallback); netem same-metro lean; D-102/D-101 merge + D-109 standalone; metal-admin GAINS a ULA leg (a real reversal of the stated lean, not a rubber-stamp); COS per-DC-only; mirror sync independent per-DC. Verified via bash scripts/ledger-scan.sh: none of D-100..D-110 appear under "PROPOSED / OPEN decisions." Full ruling record: docs/changelog-20260709-stage0-ratification.md.

State: DONE.

Stage 1 -- Phase 0: vcloud host preparation

State: RUNBOOK WRITTEN, NOT YET EXECUTED.

Stage 2 -- Phase 1: Office1 headend standup

State: NOT STARTED.

Stage 3 -- Phase 2: OpenTofu builds each DC substrate (from Office1)

State: NOT STARTED.

Stage 4 -- Phase 3: MAAS enlist / commission / deploy (per DC)

State: NOT STARTED.

Stage 5 -- Phase 4: Juju controller + OpenStack bundle (per DC) -- HIGH REUSE

State: NOT STARTED. This is where the existing repo tooling (preflight.sh, cloud-assert.sh, scripts/lib-net.sh/lib-hosts.sh, run-logged.sh) does the most direct work -- confirmed repo-agnostic as of the 2026-07-09 sweep, so they apply to either DC's clone/session without modification.

Stage 6 -- Phase 5: DR wiring and failover drill

State: NOT STARTED.

Stage 7 -- Phase 6: Designate, COS, Magnum/CAPI (per DC)

State: NOT STARTED.

Cross-cutting discipline (applies at every stage, every session)

• Session bootstrap, every session: git pull -> bash scripts/repo-lint.sh (0 fail / 1 documented legacy WARN expected) -> bash scripts/ledger-scan.sh to reconcile open decisions/next-free numbers against docs/session-ledger.md.
• Any session that will mutate the cloud: bash scripts/run-logged.sh <label> first (as-executed log).
• Before any add-model-like mutation: bash scripts/preflight.sh PASS.
• After any deploy/restart/change, and as an incident-triage first step: bash scripts/cloud-assert.sh (add --capture at real deploy-completion milestones for the committed BOM).
• Before touching ANY built surface: grep docs/design-decisions.md for the governing D-NNN. PROPOSED/OPEN means the operator has not ruled -- present options, do not implement.
• Every deliverable (script, runbook, doc): repo-lint clean, the touched script's harness green (write one if missing), a changelog entry with a per-item revert, and a docs/session-ledger.md update.
• Before assigning a new D-/DOCFIX-/BUNDLEFIX-NNN: bash scripts/ledger-scan.sh for next-free (it is the numbering authority, not repo-lint's informational count).
• Hard rule reminder: no inferred values -- every IP/ID/name/scope is measured this session or carried from confirmed as-built; verify before mutate, always; destructive/secret-handling steps are individually gated, never batched.

Tooling gap register (audited 2026-07-09; update as items close)

Confirmed by grepping the actual repo (scripts/, tests/, netbox/, runbooks/), not inferred from the design doc alone. Ordered roughly by "you'll hit this first," not by size -- #1 is cheap and unblocks a reuse claim made elsewhere in this doc;

#2 is the single largest build.

1. $DC selector convention -- CLOSED 2026-07-09 (DOCFIX-151). scripts/lib-net.sh gained lib_net_select_dc() (dc0/dc1 no-op, since D-101 rules DC1 inherits DC0's v4 layout unchanged; dc2 fails loud -- NetBox literals not yet assigned, gap #3) and scripts/lib-hosts.sh gained lib_hosts_select_dc() (dc0 no-op; dc1 AND dc2 BOTH fail loud -- no real per-DC host enrollment exists yet for either, an intentional asymmetry vs. the net selector, documented in both files' comments and docs/changelog-20260709-dc-selector-convention.md). Backward compatible by construction -- every existing caller of either library is unaffected. New harness tests/dc-selector/run-tests.sh: 21/21 PASS. The mechanism exists; Stage 5's runbook still needs to actually CALL lib_net_select_dc "$DC" / lib_hosts_select_dc "$DC" when it's authored -- that call-site wiring is part of writing that runbook, not this delivery.
2. OpenTofu -- network/pool/node-VM/cloud-init/MAAS/netem layer scaffolded 2026-07-09 (was: zero files). opentofu/ now has: modules/dc-planes (six per-DC planes), modules/mesh-link (the D-100 dark-fiber triangle legs), modules/dc-storage-pool, modules/node-vm (blank disk, PXE-boot MAAS-managed node VM, the D-103 pattern for Stage 3), modules/base-image
   • modules/cloudinit-vm (the OTHER VM pattern -- pre-built image + cloud-init, for Office1's own service VMs), modules/opnsense-edge (see gap #4), modules/maas-vm-host (registers the vcloud host with MAAS as a virsh VM host -- via the official canonical/maas provider, confirmed against its real schema; deliberately NOT maas_vm_host_machine, which composes/creates new VMs itself and would fight node-vm for VM creation -- D-103 explicitly rules out MAAS composing), and modules/netem-link (terraform_data + SSH-wrapped local-exec provisioners applying tc qdisc ... netem to a mesh-link bridge -- OpenTofu runs from Office1, not the vcloud host, so a bare local command can't reach the bridge). DC1 + Office1 pool/plane wiring is instantiated in root main.tf; everything else is NOT (node/VM sizing, MAAS zone/pool, netem parameters are all pending decisions -- neither invented here). Provider schema was confirmed against the actual current docs AND real example .tf files (not memory) -- it differs materially from older common examples: nested structures are attribute-style objects (key = { ... }), not classic HCL blocks. A real bug from the first pass was found and fixed: dc-planes/mesh-link/dc-storage-pool originally used block syntax for libvirt_network's domain/mtu and libvirt_pool's target; corrected once real provider examples surfaced the actual convention -- discovered precisely BECAUSE building the domain module forced fetching real example code instead of doc-summaries alone. Still missing, deliberately deferred (see opentofu/README.md): OPNsense's own bootability via cloudinit-vm is NOT confirmed (FreeBSD, not the same cloud-init/NoCloud guarantee as Linux images); the per-device boot-order attribute's exact internal shape (flagged UNVERIFIED in node-vm/main.tf, an inference by pattern not a confirmed field); and whether devices.interfaces list order reliably maps to LAN/WAN NIC enumeration (flagged in opnsense-edge/main.tf). UNVALIDATED -- no tofu binary was available to run scripts/opentofu-validate.sh this session; run it on a connected machine before trusting any of this.
3. NetBox multi-DC/dual-stack import pipeline -- MECHANISM CLOSED 2026-07-09 (DOCFIX-152), literals still pending. netbox/dc-dc-prefixes- import.py extends the v1 pattern to VR1's two-DC dual-stack model: per-DC (--dc dc1/dc2) creates the six-plane IPv4 (DC1 inherited, hardcoded per D-101; DC2 via required DC2_V4_SUPERNET, no default) plus the D-101 family-matrix IPv6 legs (ULA/GUA), carved from two REQUIRED env vars (ORG_ULA_48, DC_GUA_PREFIX) with no defaults -- fails loud rather than inventing either. 37/37 tests (tests/dc-dc-prefixes- import/, fake-NetBox end-to-end). UNVALIDATED against a real NetBox (no instance reachable this session). The literals themselves (org ULA /48, per-DC GUA carve, DC2 supernet) still don't exist -- that operator/NetBox assignment step is unchanged by this delivery; this closes the TOOLING gap, not the DATA gap. Still blocks Stage 2/3 until those literals are assigned and this script is actually run against a live NetBox.
4. OPNsense edge deployment/config, and netem WAN-simulation -- (a)-(c) DONE 2026-07-09, (d) is the only real gap left. Each site (Office1, DC1, DC2) gets its own independent OPNsense edge with a simulated ISP uplink (D-100); OpenTofu owns creating the edge VMs (D-103). Blocks Stage 1 (Office1 edge) and Stage 3 (per-DC edges) gates, which require netem parameters "applied and measured."
   • (a) Network substrate -- DONE. opentofu/modules/mesh-link creates the inter-site L2 segments (DC1<->DC2, DC1<->Office1, DC2<->Office1) an OPNsense VM's interfaces would eventually attach to.
   • (b)+(c) The OPNsense VM/image AND its config -- BUILT (opentofu/modules/opnsense-edge), UNVALIDATED. Cloud-init is confirmed unreliable on OPNsense (FreeBSD) -- OPNsense's own forum consensus, not just a hedge. The real native mechanism is the Configuration Importer (official docs): scans an attached volume for /conf/config.xml in a 2-3 second boot-time window; ISO9660 support was added to it specifically for VM/cloud automation (attach a config ISO as a secondary CD-ROM) -- mechanically IDENTICAL to modules/cloudinit-vm's shape (a secondary volume attached as cdrom), just a different payload. Built as opentofu/modules/opnsense-edge + two prep scripts (scripts/opnsense-prep-image.sh for the nano image's decompress+ convert, scripts/opnsense-build-config-iso.sh for the config.xml ISO). The real config.xml DESIGN is now also done: opentofu/templates/opnsense-config.xml.tmpl (a {{TOKEN}}-parameterized template built from OPNsense's own real shipped sample, covering interfaces/gateway/staticroutes/NAT/DNS/NTP and D-107-shaped firewall rules) + scripts/opnsense-render-config.sh (the renderer -- fully tested end-to-end, 8/8, unlike the other opnsense-* scripts, since it needs no external tool). UNVALIDATED regardless: no tofu binary, no ISO-building tool, and no real OPNsense boot were available to confirm the whole chain end-to-end. Several template tokens still can't be filled with real per-site values -- pending Stage 0 ratification (D-100/D-101/D-107) or only measurable on a real boot (which vtnetN device a network becomes). Full sourcing in opentofu/README.md and opentofu/templates/README.md's token legend.
   • (d) tc netem WAN-simulation -- mechanism BUILT (modules/netem-link), only the real parameters remain. No Terraform/OpenTofu-native equivalent exists (confirmed against the libvirt_network schema, gap #2); built instead as a terraform_data + SSH-wrapped local-exec provisioner (OpenTofu runs from Office1, not the vcloud host where the bridge lives -- a bare local command can't reach it). UNVALIDATED, not instantiated. The exact parameters (latency/jitter/loss/rate) are themselves still an unruled D-100 open sub-item (gap #11), so this needs a decision before it can be applied for real, not just scripting.
5. Ceph replication tooling (radosgw multisite + rbd-mirror). No script in the repo touches either (checked; osd-blank-check.sh is unrelated). D-108's entire mechanism -- zonegroup setup, rbd-mirror daemon peering, the failover/failback drill commands -- is net-new. Blocks Stage 6.
6. Designate reactivation is a bundle change, not just a runbook. runbooks/phase-01-bundle-deploy.md states the bundle explicitly ships "NO designate (D-019)" -- reactivating it per D-106 means editing bundle.yaml itself (a built-surface change, gated behind D-106 leaving PROPOSED), in addition to writing the reactivation runbook. Blocks Stage 7.
7. MTU/geneve budget + Ceph disk-budget calculators. D-102 and buildout-design Section 3 both describe measurement-then-computation gates (host L2 MTU -> geneve-over-v6 overhead; measured per-node OSD footprint -> size=3 vs size=2 fallback) that are manual arithmetic in prose today, not a script. Blocks the Stage 1 Phase-0 gates.
8. The "developed IPv6 plan" D-101 cites does not exist in this repo. D-101's context line references "the developed family-follows-reachability IPv6 plan (in-project, not yet committed)" as prior art -- grepped the whole repo, it is not present as a standalone doc anywhere. If the exact Phase A/B/C detail behind D-101/D-102 is needed by a future session, it currently is NOT retrievable from the repo alone. Not a blocker for Stage 0 ratification of what IS written, but a real content gap if finer detail is needed later.
9. Runbook debt. Confirmed zero DC-DC-specific runbooks exist. Stage 1, 2, 3, 6 need net-new runbooks; Stage 5 and 7 need an adaptation pass over phase-01..08 / phase-06/07/08 (Stage 5's pass is gated on #1 above).
10. Harness debt. This repo's own delivery rule -- no script change ships without its tests/<name>/run-tests.sh harness -- applies to everything in #2-7 above. None exist yet because none of those scripts exist yet; expect roughly 5-6 new test suites alongside whatever gets built.
11. Decision sub-items still "leaning," not ruled, that block the runbooks depending on them even after Stage 0 closes the big items: exact netem parameters (D-100), exact NetBox literals for DC2/ULA/GUA (D-101), whether D-102 folds into D-101, COS per-DC-vs-Office1-rollup (D-105), mirror sync topology (D-107).

Where things stand today (2026-07-09) -- update this as stages close