DC-DC virtual regional two-DC buildout design (VR1)

Status: DESIGN / PLAN. Originating design for the virtual regional two-DC test environment (VR1: DC1, DC2, Office1 headend). PROPOSED decisions live in docs/design-decisions.md as D-100..D-110. No live mutation is authorized from this document; it is the plan the phased runbooks execute against.

Governing constraint: minimize delta to Roosevelt (the future bare-metal multi-DC production site). Every virtual choice maps to a defensible real one; the one deliberate exception is tagged in the shim register (Section 9).

Ground truth / copy point: seeded from openstack-caracal-ipv4 at commit 8813efc into openstack-caracal-dc-dc (verified). Six-plane values are scripts/lib-net.sh (CIDR-keyed); fabrics / VLANs are docs/maas-as-built-reference.md; NetBox (vcloud-local) is the IPAM apex for all address literals.

1. Settled inputs (locked with the operator)

• Topology: full dark-fiber triangle (DC1<->DC2, DC1<->Office1, DC2<->Office1) plus a per-site OPNsense simulated ISP edge. Mesh, not star. (D-100)
• Independent clouds: DC1 and DC2 are separate Charmed OpenStack clouds, separate Keystones. No control-plane federation. Only the replication plane crosses fiber. (D-100)
• IaC seam: OpenTofu creates the libvirt "physical" layer (domains, virtual networks, OPNsense, Office1 service VMs) from Office1; MAAS enlists / commissions / deploys the node VMs; Juju deploys the bundle. NetBox is IPAM apex. (D-103)
• Controllers: one single-unit Juju controller per DC, in-DC, with backups; HA deferred to Roosevelt. Office1 hosts no OpenStack controller. (D-104)
• CAPI mgmt per DC, in-cloud single-homed; COS and Magnum / CAPI in both DCs. (D-105)
• DNS: Designate reactivated in-bundle per DC (backend designate-bind), per-DC zones, A + AAAA. Supersedes D-019, reactivates D-008. (D-106)
• Airgap at the node level; per-DC artifact mirror; NTP from each DC's edge (edge syncs internet); Office1 out of the core-service path. (D-107)
• Replication: Tier A, bidirectional -- cinder-backup via radosgw multisite; Glance via rbd-mirror; Nova ephemeral not replicated; one-way then two-way; carrier is the IPv6-only ULA replication plane. (D-108, corrects the DR-seed v4 carrier)
• Vault: per-DC independent roots, pinned 1.8/stable; v4 + IPv6-SAN certs. (D-109)
• IPv6: family follows reachability -- v6 wherever possible, v4 only where forced. (D-101)
• Numbering: DC-DC series starts D-100; 076-099 reserved source-band. (D-110)

2. Address families (per DC) -- summary of D-101

DC2 mirrors this structure on a distinct, non-overlapping supernet. All literals (ULA /48, per-DC GUA carve from the /36, DC2 v4 supernet) are NetBox-assigned -- not hardcoded. PXE and datastore east-west are the two deliberate v4-retained points.

3. Host and resource plan (Phase-0 gates)

Single vcloud host: ~256 vCPU / 1 TiB RAM / 10 TiB disk, dedicated to VR1. CPU and RAM are abundant; disk is the binding constraint (two Ceph clusters + per-DC backup/mirror + images). Phase-0 gates:

• Nested KVM: confirm / enable (nested module param; Intel vs AMD differs). Verify guest virtualization is exposed to the DC node VMs.
• Ceph replication factor: RULED 2026-07-09 (operator): target size=3 / min_size=2 by default (production-honest, lowest delta), CONFIRMED. This is CONDITIONAL on the disk budget. The Phase-0 disk math must account for: DC1 Ceph (size=3) + DC2 Ceph (size=3) + per-DC radosgw backup pools (themselves replicated within each cluster, and duplicated cross-DC by multisite) + Glance rbd-mirror targets + images + per-DC mirrors. If size=3 across both clusters plus the backup/mirror footprint exceeds 10 TiB, fall back to size=2 as an EXPLICIT, documented test-only deviation (its own decision note) -- CONFIRMED as the ONLY path to size=2: never a silent choice, Roosevelt sizing must not inherit a hidden 2x. Thin-provisioning is the lever; the reserve split (Ceph vs backup/mirror/images) is set in Phase 0 from the measured DC0 per-node OSD footprint.
• MTU: measure the host L2 MTU. Compute the geneve-over-v6 budget (D-101, folded in from D-102 per the 2026-07-09 operator ruling). Prefer jumbo (9000) end-to-end; if pinned at 1500, set the reduced tenant MTU consistently. Do not assume jumbo. MTU stacking (nested virt + geneve + VLAN) is the classic failure mode.

4. Phase spine

Each phase has a single goal, a verify-before-mutate gate, and owning D-entries. Phases run in order; a phase is not entered until the prior gate passes. Per operating discipline, the per-phase runbooks (later deliverables) carry the command-level steps with harnesses; this document is the plan they execute.

Phase 0 -- vcloud host preparation

Goal: turn the bare vcloud host into a substrate the Office1 headend can deploy from. Build: nested KVM enabled; libvirt storage pools + thin-provisioning; the dark-fiber and per-site ISP-edge virtual networks; the six per-DC plane segments (as virtual networks); the MTU plan set on the underlay. Gate: nested KVM verified; disk budget computed and the Ceph size decision recorded; measured MTU recorded; virtual networks present and isolated as designed. Owns: D-100 (fabric), D-101 (MTU -- folds in former D-102 per the 2026-07-09 operator ruling), Section 3 gates.

Phase 1 -- Office1 headend standup

Goal: stand up the operator's deployment headend first, before any DC. Build: MAAS region controller; OpenTofu; NetBox (vcloud-local, importer extended for multi-DC + dual-stack); GitBucket (vcloud-local mirror); Tailscale front door (Office1 only). Gate: MAAS region reachable; NetBox authoritative and populated (planes, per-DC v4, ULA/GUA carve); GitBucket serving; OpenTofu can reach the vcloud host libvirt; Tailscale access confirmed to Office1 only. Owns: D-103 (Office1 service VMs are OpenTofu-created), D-107 (headend is not a core-service provider).

Phase 2 -- OpenTofu builds each DC substrate (from Office1)

Goal: from Office1, script each DC's "physical" layer into existence. Build: per DC, OpenTofu creates the node-VM libvirt domains (SHIM -- Section 9), the DC's plane networks, and the DC's OPNsense edge; registers the DC libvirt host to that DC's MAAS rack controller as a virsh VM-host. Gate: MAAS rack controller per DC up; MAAS discovers the OpenTofu-created node VMs; edge routing + simulated ISP uplink up; netem parameters applied and measured. Owns: D-103 (seam), D-100 (edges).

Phase 3 -- MAAS enlist / commission / deploy (per DC)

Goal: bring the discovered node VMs to a deployed OS, ready for Juju. Build: per DC, MAAS commissions and deploys the node VMs; provider NIC left RAW (br-ex discipline, D-100); the VLAN-103 metal-internal bridge stack pre-built by MAAS. Gate: nodes deployed; the six planes present per node with correct fabrics/VLANs; provider NIC raw; PXE (v4) working; per-DC mirror reachable from nodes; NTP from edge working. Owns: D-103, D-100 (br-ex), D-107 (mirror + NTP).

Phase 4 -- Juju controller + OpenStack bundle (per DC)

Goal: a running independent OpenStack cloud per DC. Build: per DC, bootstrap the single-unit Juju controller in-DC; deploy the Caracal bundle; Vault init with per-DC independent root, issuing v4 + IPv6-SAN certs; the family matrix applied (ULA on data/storage/replication, dual-stack metal-internal API, GUA provider). Gate: preflight.sh PASS before add-model; post-deploy cloud-assert.sh --capture per DC; controller backup taken and stored; Ceph-over-v6 and geneve-over-v6 verified. Owns: D-104 (controllers), D-109 (Vault), D-101 (families).

Phase 5 -- DR wiring and failover drill

Goal: prove Tier-A recoverability, one-way first, then two-way. Build: replication-plane peering (ULA); radosgw multisite zonegroup (two DCs as zones); rbd-mirror for the Glance pool; cinder-backup to the replicated object store; consistency groups where needed. Run the failover / failback drill (Section 8). Stage one-way, then enable two-way after a clean drill. Gate: one-way drill clean (RTO/RPO measured); then two-way; controller restore drill (D-104) exercised; RTO recorded as a measured output, not an SLA. Owns: D-108 (replication), D-104 (control-plane backup/restore in the drill).

Phase 6 -- Designate, COS, Magnum / CAPI (per DC)

Goal: complete the service set in both DCs. Build: per DC, Designate (designate-bind) with per-DC zones + A/AAAA per the D-008 bootstrap order; COS observability; Magnum / CAPI with in-cloud single-homed capi-mgmt (domain-setup re-run per D-046). Gate: Designate resolving A + AAAA; FQDN-SAN certs valid (D-019/D-021 root cause closed); COS scraping; a CAPI workload cluster comes up per DC. Owns: D-106 (Designate), D-105 (CAPI/COS).

5. Cross-cutting: OpenTofu / MAAS / NetBox / Juju boundary

• NetBox: IPAM apex. All address values originate here; MAAS and overlays consume them.
• OpenTofu: create/destroy of libvirt domains, virtual networks, OPNsense, Office1 VMs; runs from Office1. Owns the "physical" layer only.
• MAAS: enlist (discover) / commission / deploy / power / release of node VMs. Does not compose. One region (Office1) + one rack per DC (virsh VM-host).
• Juju: deploys and manages the OpenStack bundle onto MAAS machines; one controller per DC.

6. WAN simulation

tc netem on the inter-DC and site-uplink virtual segments. Default lean: same-metro dark fiber (low single-digit ms, jumbo-capable). Parameters (latency / jitter / loss / rate) are a Phase-0 knob and are recorded per drill so RTO/RPO actuals are attributable to a known link profile. Re-tune to the confirmed Roosevelt inter-DC profile when known.

7. IPv6 rehearsal scope

VR1 rehearses the full developed IPv6 plan because the simulated OPNsense edge is ours (the D-015 upstream-router gate does not apply): internal IPv6-only ULA (data / storage / replication / Octavia lb-mgmt / P2P), internal dual-stack ULA on metal-internal API endpoints, and external GUA (provider public VIPs, native v6 ext_net, tenant delegation). Ceph-over-v6 (ms_bind_ipv6), geneve-over-v6 MTU, and native v6 tenant routing are the high-value Roosevelt proofs. Phase-A internal-three goes straight to v6-only (lean; dual-then-flip is the conservative fallback if Phase-0 verification is not clean).

8. Failover / failback drill skeleton (from the DR seed, carrier corrected)

Failover (DC1 lost, recover at DC2):

1. Confirm DC1 truly down (avoid split-brain from a transient partition).
2. Glance: promote the mirrored Glance pool at DC2 (force-promote if DC1 unreachable); re-register images into DC2 Glance.
3. Cinder: restore required volumes into DC2 Cinder from the replicated backups.
4. Rebuild instances from image + restored volume; re-create Neutron ports / FIPs / SGs.
5. Record RTO / RPO actuals.

Failback (DC1 recovered) -- split-brain-safe for two-way:

1. Do NOT let recovered DC1 resume as primary automatically.
2. Demote DC1's Glance pool; resync from DC2 (now primary).
3. Reconcile Cinder: back up any DC2-side changes; restore into DC1.
4. In a controlled window, optionally flip primary back (demote DC2, promote DC1).
5. Verify both directions healthy; record the drill.

9. Shim register -- the no-Roosevelt-analog steps

The only steps with NO Roosevelt analog (tag them clearly; do not treat as reusable production IaC):

• OpenTofu "create the node VMs" module (D-103). In Roosevelt, node "hardware" already exists; that step is physical racking + BMC enlistment. Everything from MAAS enlistment onward IS Roosevelt-transferable.
• tc netem WAN simulation (D-100). In Roosevelt the inter-DC link is real fiber; netem is replaced by the physical link. The route + bandwidth budget it stands in for IS real.
• Single-unit controllers (D-104) and single-unit rbd-mirror daemon (D-108) are VR1 test economies; Roosevelt scales both to HA.

10. Operator redline -- RULED 2026-07-09 (Stage 0 ratification complete)

All six items below were ruled by the operator on 2026-07-09; see the referenced D-NNN entries in docs/design-decisions.md for the authoritative recorded text. D-100 through D-110 are now ADOPTED (D-102 MERGED into D-101; D-109 confirmed standalone) -- Stage 0 of docs/dc-dc-deployment-workflow.md is CLEARED.

• Ceph size=3 vs a documented size=2 fallback: RULED -- size=3 by default, size=2 only as an explicit, logged Phase-0 fallback if the disk math doesn't fit (Section 3).
• netem link profile: RULED -- same-metro lean now (D-100), re-tune when a specific Roosevelt inter-DC target is known.
• Whether D-102 folds into D-101, and whether D-109 stands alone: RULED -- D-102 folds into D-101; D-109 stays standalone (D-101, D-109).
• metal-admin admin-API ULA leg: RULED -- ADDED (reverses the stated lean; D-101).
• COS per-DC-only vs an Office1 roll-up: RULED -- per-DC only, confirming the stated lean (D-105).
• Mirror sync topology: RULED -- independent upstream pull per DC, confirming the stated lean (D-107).

11. Landing this deliverable

1. Append the body of docs/dc-dc-D100-D110.append.md (from its first ## D-100 line) to the end of docs/design-decisions.md, then remove the fragment file.
2. Run bash scripts/ledger-scan.sh; confirm D next-free = 111 and the machine-derived ledger block regenerates cleanly. Do not hand-edit the machine-derived block.
3. Add a one-line entry in the main-chat fence of docs/session-ledger.md recording this deliverable (design doc + D-100..D-110 PROPOSED).
4. bash scripts/repo-lint.sh for ASCII/LF/format (ignore any next-free noise above 099 -- known band bug; ledger-scan is the next-free authority).
5. Commit. These are PROPOSED: present for redline, do not implement until the operator ratifies each to ADOPTED.