DC-DC (VR1) deployment workflow overview

Purpose. A single followable checklist across the whole VR1 (two-DC virtual regional) buildout, spanning many sessions. docs/dc-dc-buildout-design.md is the WHY/WHAT (goals, gates, D-refs per phase); this doc is the HOW-TO-FOLLOW-IT sequence, cross-referenced to that plan, to D-100..D-110, and to what already exists in this repo vs what still needs to be authored. Update the "state" column as stages complete -- this doc is a tracker, not a one-time snapshot.

Governing constraint (unchanged throughout): minimize delta to Roosevelt. Reuse VR0 DC0 material wherever it transfers; the only deliberate no-Roosevelt-analog steps are the Section 9 shim register in the buildout design (OpenTofu node-VM creation, tc netem WAN simulation, single-unit controllers/rbd-mirror).

Stage 0 -- Decision ratification (CLEARED 2026-07-09)

RATIFIED. D-100..D-110 are ALL ADOPTED (docs/design-decisions.md; D-102 MERGED into D-101 per the operator's own ruling, not deleted -- retained for numbering/history; D-109 confirmed standalone). All 6 redline items ruled (docs/dc-dc-buildout-design.md Section 10 is now a ruling record, not an open list): Ceph size=3-by-default (size=2 only as an explicit Phase-0 fallback); netem same-metro lean; D-102/D-101 merge + D-109 standalone; metal-admin GAINS a ULA leg (a real reversal of the stated lean, not a rubber-stamp); COS per-DC-only; mirror sync independent per-DC. Verified via bash scripts/ledger-scan.sh: none of D-100..D-110 appear under "PROPOSED / OPEN decisions." Full ruling record: docs/changelog-20260709-stage0-ratification.md.

State: DONE.

Stage 1 -- Phase 0: vcloud host preparation

State: RUNBOOK WRITTEN, NOT YET EXECUTED.

Stage 2 -- Phase 1: Office1 headend standup

State: RUNBOOK WRITTEN, NOT YET EXECUTED. Honest exit state per the runbook's own GATE section: MAAS/GitBucket/Tailscale/OpenTofu-reach bullets closeable in one run; NetBox's "authoritative and populated" bullet only PARTIAL (mechanism done via DOCFIX-152, literals still unassigned).

Stage 3 -- Phase 2: OpenTofu builds each DC substrate (from Office1)

State: RUNBOOK WRITTEN, NOT YET EXECUTED, hard-dependent on Stage 2 for its MAAS-registration half.

Stage 4 -- Phase 3: MAAS enlist / commission / deploy (per DC)

State: RUNBOOK WRITTEN, NOT YET EXECUTED.

Stage 5 -- Phase 4: Juju controller + OpenStack bundle (per DC) -- HIGH REUSE

State: RUNBOOK WRITTEN, NOT YET EXECUTED. This is where the existing repo tooling (preflight.sh, cloud-assert.sh, scripts/lib-net.sh/lib-hosts.sh, run-logged.sh) does the most direct work -- confirmed repo-agnostic as of the 2026-07-09 sweep, so they apply to either DC's clone/session without modification.

Stage 6 -- Phase 5: DR wiring and failover drill

State: RUNBOOK WRITTEN, NOT YET EXECUTED.

Stage 7 -- Phase 6: Designate, COS, Magnum/CAPI (per DC)

State: RUNBOOK WRITTEN, NOT YET EXECUTED.

Cross-cutting discipline (applies at every stage, every session)

• Session bootstrap, every session: git pull -> bash scripts/repo-lint.sh (0 fail / 1 documented legacy WARN expected) -> bash scripts/ledger-scan.sh to reconcile open decisions/next-free numbers against docs/session-ledger.md.
• Any session that will mutate the cloud: bash scripts/run-logged.sh <label> first (as-executed log).
• Before any add-model-like mutation: bash scripts/preflight.sh PASS.
• After any deploy/restart/change, and as an incident-triage first step: bash scripts/cloud-assert.sh (add --capture at real deploy-completion milestones for the committed BOM).
• Before touching ANY built surface: grep docs/design-decisions.md for the governing D-NNN. PROPOSED/OPEN means the operator has not ruled -- present options, do not implement.
• Every deliverable (script, runbook, doc): repo-lint clean, the touched script's harness green (write one if missing), a changelog entry with a per-item revert, and a docs/session-ledger.md update.
• Before assigning a new D-/DOCFIX-/BUNDLEFIX-NNN: bash scripts/ledger-scan.sh for next-free (it is the numbering authority, not repo-lint's informational count).
• Hard rule reminder: no inferred values -- every IP/ID/name/scope is measured this session or carried from confirmed as-built; verify before mutate, always; destructive/secret-handling steps are individually gated, never batched.

Tooling gap register (audited 2026-07-09; update as items close)

Confirmed by grepping the actual repo (scripts/, tests/, netbox/, runbooks/), not inferred from the design doc alone. Ordered roughly by "you'll hit this first," not by size -- #1 is cheap and unblocks a reuse claim made elsewhere in this doc;

#2 is the single largest build.

1. $DC selector convention -- CLOSED 2026-07-09 (DOCFIX-151). scripts/lib-net.sh gained lib_net_select_dc() (dc0/dc1 no-op, since D-101 rules DC1 inherits DC0's v4 layout unchanged; dc2 fails loud -- NetBox literals not yet assigned, gap #3) and scripts/lib-hosts.sh gained lib_hosts_select_dc() (dc0 no-op; dc1 AND dc2 BOTH fail loud -- no real per-DC host enrollment exists yet for either, an intentional asymmetry vs. the net selector, documented in both files' comments and docs/changelog-20260709-dc-selector-convention.md). Backward compatible by construction -- every existing caller of either library is unaffected. New harness tests/dc-selector/run-tests.sh: 21/21 PASS. The mechanism exists; Stage 5's runbook still needs to actually CALL lib_net_select_dc "$DC" / lib_hosts_select_dc "$DC" when it's authored -- that call-site wiring is part of writing that runbook, not this delivery.
2. OpenTofu -- network/pool/node-VM/cloud-init/MAAS/netem layer scaffolded 2026-07-09 (was: zero files). opentofu/ now has: modules/dc-planes (six per-DC planes), modules/mesh-link (the D-100 dark-fiber triangle legs), modules/dc-storage-pool, modules/node-vm (blank disk, PXE-boot MAAS-managed node VM, the D-103 pattern for Stage 3), modules/base-image
   • modules/cloudinit-vm (the OTHER VM pattern -- pre-built image + cloud-init, for Office1's own service VMs), modules/opnsense-edge (see gap #4), modules/maas-vm-host (registers the vcloud host with MAAS as a virsh VM host -- via the official canonical/maas provider, confirmed against its real schema; deliberately NOT maas_vm_host_machine, which composes/creates new VMs itself and would fight node-vm for VM creation -- D-103 explicitly rules out MAAS composing), and modules/netem-link (terraform_data + SSH-wrapped local-exec provisioners applying tc qdisc ... netem to a mesh-link bridge -- OpenTofu runs from Office1, not the vcloud host, so a bare local command can't reach the bridge). DC1 + Office1 pool/plane wiring is instantiated in root main.tf; everything else is NOT (node/VM sizing, MAAS zone/pool, netem parameters are all pending decisions -- neither invented here). Provider schema was confirmed against the actual current docs AND real example .tf files (not memory) -- it differs materially from older common examples: nested structures are attribute-style objects (key = { ... }), not classic HCL blocks. A real bug from the first pass was found and fixed: dc-planes/mesh-link/dc-storage-pool originally used block syntax for libvirt_network's domain/mtu and libvirt_pool's target; corrected once real provider examples surfaced the actual convention -- discovered precisely BECAUSE building the domain module forced fetching real example code instead of doc-summaries alone. Still missing, deliberately deferred (see opentofu/README.md): OPNsense's own bootability via cloudinit-vm is NOT confirmed (FreeBSD, not the same cloud-init/NoCloud guarantee as Linux images); the per-device boot-order attribute's exact internal shape (flagged UNVERIFIED in node-vm/main.tf, an inference by pattern not a confirmed field); and whether devices.interfaces list order reliably maps to LAN/WAN NIC enumeration (flagged in opnsense-edge/main.tf). UNVALIDATED -- no tofu binary was available to run scripts/opentofu-validate.sh this session; run it on a connected machine before trusting any of this.
3. NetBox multi-DC/dual-stack import pipeline -- MECHANISM CLOSED 2026-07-09 (DOCFIX-152), literals still pending. netbox/dc-dc-prefixes- import.py extends the v1 pattern to VR1's two-DC dual-stack model: per-DC (--dc dc1/dc2) creates the six-plane IPv4 (DC1 inherited, hardcoded per D-101; DC2 via required DC2_V4_SUPERNET, no default) plus the D-101 family-matrix IPv6 legs (ULA/GUA), carved from two REQUIRED env vars (ORG_ULA_48, DC_GUA_PREFIX) with no defaults -- fails loud rather than inventing either. 40/40 tests (tests/dc-dc-prefixes- import/, fake-NetBox end-to-end). UNVALIDATED against a real NetBox (no instance reachable this session). The literals themselves (org ULA /48, per-DC GUA carve, DC2 supernet) still don't exist -- that operator/NetBox assignment step is unchanged by this delivery; this closes the TOOLING gap, not the DATA gap. Still blocks Stage 2/3 until those literals are assigned and this script is actually run against a live NetBox.
4. OPNsense edge deployment/config, and netem WAN-simulation -- (a)-(c) DONE 2026-07-09, (d) is the only real gap left. Each site (Office1, DC1, DC2) gets its own independent OPNsense edge with a simulated ISP uplink (D-100); OpenTofu owns creating the edge VMs (D-103). Blocks Stage 1 (Office1 edge) and Stage 3 (per-DC edges) gates, which require netem parameters "applied and measured."
   • (a) Network substrate -- DONE. opentofu/modules/mesh-link creates the inter-site L2 segments (DC1<->DC2, DC1<->Office1, DC2<->Office1) an OPNsense VM's interfaces would eventually attach to.
   • (b)+(c) The OPNsense VM/image AND its config -- BUILT (opentofu/modules/opnsense-edge), UNVALIDATED. Cloud-init is confirmed unreliable on OPNsense (FreeBSD) -- OPNsense's own forum consensus, not just a hedge. The real native mechanism is the Configuration Importer (official docs): scans an attached volume for /conf/config.xml in a 2-3 second boot-time window; ISO9660 support was added to it specifically for VM/cloud automation (attach a config ISO as a secondary CD-ROM) -- mechanically IDENTICAL to modules/cloudinit-vm's shape (a secondary volume attached as cdrom), just a different payload. Built as opentofu/modules/opnsense-edge + two prep scripts (scripts/opnsense-prep-image.sh for the nano image's decompress+ convert, scripts/opnsense-build-config-iso.sh for the config.xml ISO). The real config.xml DESIGN is now also done: opentofu/templates/opnsense-config.xml.tmpl (a {{TOKEN}}-parameterized template built from OPNsense's own real shipped sample, covering interfaces/gateway/staticroutes/NAT/DNS/NTP and D-107-shaped firewall rules) + scripts/opnsense-render-config.sh (the renderer -- fully tested end-to-end, 8/8, unlike the other opnsense-* scripts, since it needs no external tool). UNVALIDATED regardless: no tofu binary, no ISO-building tool, and no real OPNsense boot were available to confirm the whole chain end-to-end. Several template tokens still can't be filled with real per-site values -- pending Stage 0 ratification (D-100/D-101/D-107) or only measurable on a real boot (which vtnetN device a network becomes). Full sourcing in opentofu/README.md and opentofu/templates/README.md's token legend.
   • (d) tc netem WAN-simulation -- mechanism BUILT (modules/netem-link), only the real parameters remain. No Terraform/OpenTofu-native equivalent exists (confirmed against the libvirt_network schema, gap #2); built instead as a terraform_data + SSH-wrapped local-exec provisioner (OpenTofu runs from Office1, not the vcloud host where the bridge lives -- a bare local command can't reach it). UNVALIDATED, not instantiated. The exact parameters (latency/jitter/loss/rate) are themselves still an unruled D-100 open sub-item (gap #11), so this needs a decision before it can be applied for real, not just scripting.
5. Ceph replication tooling (radosgw multisite + rbd-mirror). No script in the repo touches either (checked; osd-blank-check.sh is unrelated). D-108's entire mechanism -- zonegroup setup, rbd-mirror daemon peering, the failover/failback drill commands -- is net-new. Blocks Stage 6.
6. Designate reactivation is a bundle change, not just a runbook. runbooks/phase-01-bundle-deploy.md states the bundle explicitly ships "NO designate (D-019)" -- reactivating it per D-106 means editing bundle.yaml itself (a built-surface change, gated behind D-106 leaving PROPOSED), in addition to writing the reactivation runbook. Blocks Stage 7.
7. MTU/geneve budget + Ceph disk-budget calculators. D-102 and buildout-design Section 3 both describe measurement-then-computation gates (host L2 MTU -> geneve-over-v6 overhead; measured per-node OSD footprint -> size=3 vs size=2 fallback) that are manual arithmetic in prose today, not a script. Blocks the Stage 1 Phase-0 gates.
8. The "developed IPv6 plan" D-101 cites does not exist in this repo. D-101's context line references "the developed family-follows-reachability IPv6 plan (in-project, not yet committed)" as prior art -- grepped the whole repo, it is not present as a standalone doc anywhere. If the exact Phase A/B/C detail behind D-101/D-102 is needed by a future session, it currently is NOT retrievable from the repo alone. Not a blocker for Stage 0 ratification of what IS written, but a real content gap if finer detail is needed later.
9. Runbook debt -- CLOSED 2026-07-09. All seven Stage 1-7 runbooks are now written (runbooks/dc-dc-phase0-vcloud-prep.md through dc-dc-phase6- designate-cos-magnum.md), Stage 1 authored directly, Stages 2/3/4/6/7 delegated to parallel subagents and Stage 5 authored directly, all reviewed and repo-lint clean. NONE have been executed against real infrastructure yet -- this was a prep-only session (the operator was away from the workstation, not the vcloud host). Each runbook is explicit about what it can and cannot close given today's real state (NetBox literals, node sizing, and several other decisions remain genuinely open -- see each runbook's own GATE section for the honest per-stage breakdown).
10. Harness debt. This repo's own delivery rule -- no script change ships without its tests/<name>/run-tests.sh harness -- applies to everything in #2-7 above. None exist yet because none of those scripts exist yet; expect roughly 5-6 new test suites alongside whatever gets built.
11. Decision sub-items still genuinely open (the big items are RULED per Stage 0; these specific literals/params are not): exact netem parameters (D-100), exact NetBox literals for DC2/ULA/GUA (D-101).
12. No Office1-local virtual network is modeled anywhere in opentofu/ (found 2026-07-09 while drafting the Stage 2 runbook). main.tf wires office1_pool_path (storage only) and the three inter-site mesh links (management-traffic-only per D-100's own sub-item ruling) -- nothing the three Office1 service VMs (MAAS-region, NetBox, GitBucket), or the vcloud host's own management interface, could actually attach to as a shared local network. Blocks designing real cloudinit-vm network_config content for any of the three VMs. Needs either a new modules/dc-planes- shaped Office1-local network, or a deliberate, documented reuse of an existing host bridge -- not decided yet.
13. The D-101 IPv6 family-matrix overlay does not exist (found 2026-07-09 while drafting the Stage 5 runbook). Which planes need ULA-only vs. dual-stack vs. GUA addressing is fully specified by D-101 (Section 2 of the buildout design), but the actual Juju charm config option names needed to configure that per application (ceph-mon, ovn-central, octavia, the HA-cluster/VIP charms, etc.) have not been confirmed against any charm's real current schema. Blocks Stage 5's "Ceph-over-v6 and geneve-over-v6 verified" gate criterion. D-109's IPv6-SAN cert- issuance charm support is a related, separately-unverified assumption.
14. Three real bundle.yaml gaps for D-108's DR mechanism (found 2026-07-09 while drafting the Stage 6 runbook, by grepping the actual bundle): no cinder-backup charm deployed; no ceph-rbd-mirror charm deployed; ceph-mon's existing rbd-mirror endpoint binding is on storage, not replication (inherited from the VR0/DC0 seed, predates D-108's ULA-carrier requirement). All three are Stage 6's own first gated mutation (its Step 2), not fixed by this delivery -- no live Ceph cluster exists this session to test the fix against.
15. Seven real MAAS/multi-rack gaps found 2026-07-09 while drafting the Stage 4 runbook (full detail in that runbook's own "Known gaps" section): no OpenTofu module stands up a per-DC MAAS rack controller VM itself (only registration, modules/maas-vm-host, exists); reenroll- hosts.sh/carve-host-interfaces.sh/phase-00-maas-standup.sh are not $DC-parameterized (DOCFIX-151 only touched the two library files); per-DC node count/sizing is an unmade Phase-0 decision; the per-DC artifact mirror (D-107) has no identified owning stage/module; the MAAS status a VM-host-discovered (vs. PXE-enlisted) machine lands in is unconfirmed; HOST_TAG/boot-fabric-name literals are DC0-specific.

Where things stand today (2026-07-09) -- update this as stages close