# DC-DC virtual regional two-DC buildout design (VR1)

**Status:** DESIGN / PLAN. Originating design for the virtual regional two-DC test
environment (VR1: DC1, DC2, Office1 headend). PROPOSED decisions live in
`docs/design-decisions.md` as D-100..D-110. No live mutation is authorized from this
document; it is the plan the phased runbooks execute against.

**Governing constraint:** minimize delta to Roosevelt (the future bare-metal multi-DC
production site). Every virtual choice maps to a defensible real one; the one deliberate
exception is tagged in the shim register (Section 9).

**Ground truth / copy point:** seeded from `openstack-caracal-ipv4` at commit 8813efc into
`openstack-caracal-dc-dc` (verified). Six-plane values are `scripts/lib-net.sh`
(CIDR-keyed); fabrics / VLANs are `docs/maas-as-built-reference.md`; NetBox (vcloud-local)
is the IPAM apex for all address literals.

---

## 1. Settled inputs (locked with the operator)

- Topology: full dark-fiber triangle (DC1<->DC2, DC1<->Office1, DC2<->Office1) plus a
  per-site OPNsense simulated ISP edge. Mesh, not star. (D-100)
- Independent clouds: DC1 and DC2 are separate Charmed OpenStack clouds, separate
  Keystones. No control-plane federation. Only the replication plane crosses fiber. (D-100)
- IaC seam: OpenTofu creates the libvirt "physical" layer (domains, virtual networks,
  OPNsense, Office1 service VMs) from Office1; MAAS enlists / commissions / deploys the
  node VMs; Juju deploys the bundle. NetBox is IPAM apex. (D-103)
- Controllers: one single-unit Juju controller per DC, in-DC, with backups; HA deferred to
  Roosevelt. Office1 hosts no OpenStack controller. (D-104)
- CAPI mgmt per DC, in-cloud single-homed; COS and Magnum / CAPI in both DCs. (D-105)
- DNS: Designate reactivated in-bundle per DC (backend designate-bind), per-DC zones,
  A + AAAA. Supersedes D-019, reactivates D-008. (D-106)
- Airgap at the node level; per-DC artifact mirror; NTP from each DC's edge (edge syncs
  internet); Office1 out of the core-service path. (D-107)
- Replication: Tier A, bidirectional -- cinder-backup via radosgw multisite; Glance via
  rbd-mirror; Nova ephemeral not replicated; one-way then two-way; carrier is the
  IPv6-only ULA replication plane. (D-108, corrects the DR-seed v4 carrier)
- Vault: per-DC independent roots, pinned 1.8/stable; v4 + IPv6-SAN certs. (D-109)
- IPv6: family follows reachability -- v6 wherever possible, v4 only where forced. (D-101)
- Numbering: DC-DC series starts D-100; 076-099 reserved source-band. (D-110)

## 2. Address families (per DC) -- summary of D-101

| Plane | v1/DC0 v4 (DC1 inherits) | VR1 family | Note |
|---|---|---|---|
| provider-public | 10.12.4.0/22 | v4 + GUA | public API VIPs + FIP/ext_net; GUA via owned edge |
| metal-admin | 10.12.8.0/22 | v4 only | operator / MAAS / PXE / admin API; DC-local |
| metal-internal | 10.12.12.0/22 (VID 103) | v4 + ULA | API endpoints dual-stack; MySQL/RabbitMQ stay v4 |
| data-tenant | 10.12.16.0/22 | ULA only | geneve overlay; tenant GUA delegated separately |
| storage | 10.12.32.0/22 | ULA only | Ceph public; ms_bind_ipv6 |
| replication | 10.12.36.0/22 | ULA only | Ceph cluster; cross-DC leg rides this (ULA) |

DC2 mirrors this structure on a distinct, non-overlapping supernet. All literals
(ULA /48, per-DC GUA carve from the /36, DC2 v4 supernet) are NetBox-assigned -- not
hardcoded. PXE and datastore east-west are the two deliberate v4-retained points.

## 3. Host and resource plan (Phase-0 gates)

Single vcloud host: ~256 vCPU / 1 TiB RAM / 10 TiB disk, dedicated to VR1. CPU and RAM
are abundant; disk is the binding constraint (two Ceph clusters + per-DC backup/mirror +
images). Phase-0 gates:

- Nested KVM: confirm / enable (`nested` module param; Intel vs AMD differs). Verify guest
  virtualization is exposed to the DC node VMs.
- Ceph replication factor: target size=3 / min_size=2 (production-honest, lowest delta).
  This is CONDITIONAL on the disk budget. The Phase-0 disk math must account for: DC1 Ceph
  (size=3) + DC2 Ceph (size=3) + per-DC radosgw backup pools (themselves replicated within
  each cluster, and duplicated cross-DC by multisite) + Glance rbd-mirror targets + images
  + per-DC mirrors. If size=3 across both clusters plus the backup/mirror footprint exceeds
  10 TiB, fall back to size=2 as an EXPLICIT, documented test-only deviation (its own
  decision note), never a silent choice -- Roosevelt sizing must not inherit a hidden 2x.
  Thin-provisioning is the lever; the reserve split (Ceph vs backup/mirror/images) is set in
  Phase 0 from the measured DC0 per-node OSD footprint.
- MTU: measure the host L2 MTU. Compute the geneve-over-v6 budget (Section D-102). Prefer
  jumbo (9000) end-to-end; if pinned at 1500, set the reduced tenant MTU consistently. Do
  not assume jumbo. MTU stacking (nested virt + geneve + VLAN) is the classic failure mode.

## 4. Phase spine

Each phase has a single goal, a verify-before-mutate gate, and owning D-entries. Phases run
in order; a phase is not entered until the prior gate passes. Per operating discipline, the
per-phase runbooks (later deliverables) carry the command-level steps with harnesses; this
document is the plan they execute.

### Phase 0 -- vcloud host preparation
Goal: turn the bare vcloud host into a substrate the Office1 headend can deploy from.
Build: nested KVM enabled; libvirt storage pools + thin-provisioning; the dark-fiber and
per-site ISP-edge virtual networks; the six per-DC plane segments (as virtual networks);
the MTU plan set on the underlay.
Gate: nested KVM verified; disk budget computed and the Ceph size decision recorded;
measured MTU recorded; virtual networks present and isolated as designed.
Owns: D-100 (fabric), D-101/D-102 (MTU), Section 3 gates.

### Phase 1 -- Office1 headend standup
Goal: stand up the operator's deployment headend first, before any DC.
Build: MAAS region controller; OpenTofu; NetBox (vcloud-local, importer extended for
multi-DC + dual-stack); GitBucket (vcloud-local mirror); Tailscale front door (Office1 only).
Gate: MAAS region reachable; NetBox authoritative and populated (planes, per-DC v4, ULA/GUA
carve); GitBucket serving; OpenTofu can reach the vcloud host libvirt; Tailscale access
confirmed to Office1 only.
Owns: D-103 (Office1 service VMs are OpenTofu-created), D-107 (headend is not a core-service
provider).

### Phase 2 -- OpenTofu builds each DC substrate (from Office1)
Goal: from Office1, script each DC's "physical" layer into existence.
Build: per DC, OpenTofu creates the node-VM libvirt domains (SHIM -- Section 9), the DC's
plane networks, and the DC's OPNsense edge; registers the DC libvirt host to that DC's MAAS
rack controller as a virsh VM-host.
Gate: MAAS rack controller per DC up; MAAS discovers the OpenTofu-created node VMs; edge
routing + simulated ISP uplink up; netem parameters applied and measured.
Owns: D-103 (seam), D-100 (edges).

### Phase 3 -- MAAS enlist / commission / deploy (per DC)
Goal: bring the discovered node VMs to a deployed OS, ready for Juju.
Build: per DC, MAAS commissions and deploys the node VMs; provider NIC left RAW (br-ex
discipline, D-100); the VLAN-103 metal-internal bridge stack pre-built by MAAS.
Gate: nodes deployed; the six planes present per node with correct fabrics/VLANs; provider
NIC raw; PXE (v4) working; per-DC mirror reachable from nodes; NTP from edge working.
Owns: D-103, D-100 (br-ex), D-107 (mirror + NTP).

### Phase 4 -- Juju controller + OpenStack bundle (per DC)
Goal: a running independent OpenStack cloud per DC.
Build: per DC, bootstrap the single-unit Juju controller in-DC; deploy the Caracal bundle;
Vault init with per-DC independent root, issuing v4 + IPv6-SAN certs; the family matrix
applied (ULA on data/storage/replication, dual-stack metal-internal API, GUA provider).
Gate: `preflight.sh` PASS before add-model; post-deploy `cloud-assert.sh --capture` per DC;
controller backup taken and stored; Ceph-over-v6 and geneve-over-v6 verified.
Owns: D-104 (controllers), D-109 (Vault), D-101 (families).

### Phase 5 -- DR wiring and failover drill
Goal: prove Tier-A recoverability, one-way first, then two-way.
Build: replication-plane peering (ULA); radosgw multisite zonegroup (two DCs as zones);
rbd-mirror for the Glance pool; cinder-backup to the replicated object store; consistency
groups where needed. Run the failover / failback drill (Section 8). Stage one-way, then
enable two-way after a clean drill.
Gate: one-way drill clean (RTO/RPO measured); then two-way; controller restore drill
(D-104) exercised; RTO recorded as a measured output, not an SLA.
Owns: D-108 (replication), D-104 (control-plane backup/restore in the drill).

### Phase 6 -- Designate, COS, Magnum / CAPI (per DC)
Goal: complete the service set in both DCs.
Build: per DC, Designate (designate-bind) with per-DC zones + A/AAAA per the D-008
bootstrap order; COS observability; Magnum / CAPI with in-cloud single-homed capi-mgmt
(`domain-setup` re-run per D-046).
Gate: Designate resolving A + AAAA; FQDN-SAN certs valid (D-019/D-021 root cause closed);
COS scraping; a CAPI workload cluster comes up per DC.
Owns: D-106 (Designate), D-105 (CAPI/COS).

## 5. Cross-cutting: OpenTofu / MAAS / NetBox / Juju boundary

- NetBox: IPAM apex. All address values originate here; MAAS and overlays consume them.
- OpenTofu: create/destroy of libvirt domains, virtual networks, OPNsense, Office1 VMs;
  runs from Office1. Owns the "physical" layer only.
- MAAS: enlist (discover) / commission / deploy / power / release of node VMs. Does not
  compose. One region (Office1) + one rack per DC (virsh VM-host).
- Juju: deploys and manages the OpenStack bundle onto MAAS machines; one controller per DC.

## 6. WAN simulation

`tc netem` on the inter-DC and site-uplink virtual segments. Default lean: same-metro dark
fiber (low single-digit ms, jumbo-capable). Parameters (latency / jitter / loss / rate) are
a Phase-0 knob and are recorded per drill so RTO/RPO actuals are attributable to a known
link profile. Re-tune to the confirmed Roosevelt inter-DC profile when known.

## 7. IPv6 rehearsal scope

VR1 rehearses the full developed IPv6 plan because the simulated OPNsense edge is ours (the
D-015 upstream-router gate does not apply): internal IPv6-only ULA (data / storage /
replication / Octavia lb-mgmt / P2P), internal dual-stack ULA on metal-internal API
endpoints, and external GUA (provider public VIPs, native v6 ext_net, tenant delegation).
Ceph-over-v6 (`ms_bind_ipv6`), geneve-over-v6 MTU, and native v6 tenant routing are the
high-value Roosevelt proofs. Phase-A internal-three goes straight to v6-only (lean;
dual-then-flip is the conservative fallback if Phase-0 verification is not clean).

## 8. Failover / failback drill skeleton (from the DR seed, carrier corrected)

Failover (DC1 lost, recover at DC2):
1. Confirm DC1 truly down (avoid split-brain from a transient partition).
2. Glance: promote the mirrored Glance pool at DC2 (force-promote if DC1 unreachable);
   re-register images into DC2 Glance.
3. Cinder: restore required volumes into DC2 Cinder from the replicated backups.
4. Rebuild instances from image + restored volume; re-create Neutron ports / FIPs / SGs.
5. Record RTO / RPO actuals.

Failback (DC1 recovered) -- split-brain-safe for two-way:
1. Do NOT let recovered DC1 resume as primary automatically.
2. Demote DC1's Glance pool; resync from DC2 (now primary).
3. Reconcile Cinder: back up any DC2-side changes; restore into DC1.
4. In a controlled window, optionally flip primary back (demote DC2, promote DC1).
5. Verify both directions healthy; record the drill.

## 9. Shim register -- the no-Roosevelt-analog steps

The only steps with NO Roosevelt analog (tag them clearly; do not treat as reusable
production IaC):

- OpenTofu "create the node VMs" module (D-103). In Roosevelt, node "hardware" already
  exists; that step is physical racking + BMC enlistment. Everything from MAAS enlistment
  onward IS Roosevelt-transferable.
- `tc netem` WAN simulation (D-100). In Roosevelt the inter-DC link is real fiber; netem
  is replaced by the physical link. The route + bandwidth budget it stands in for IS real.
- Single-unit controllers (D-104) and single-unit rbd-mirror daemon (D-108) are VR1 test
  economies; Roosevelt scales both to HA.

## 10. Open items for operator redline

- Ceph size=3 vs a documented size=2 fallback (gated on the Phase-0 disk math).
- netem link profile (same-metro lean vs a specific Roosevelt inter-DC target).
- Whether D-102 folds into D-101, and whether D-109 stands alone.
- metal-admin admin-API ULA leg (lean: no).
- COS per-DC-only vs an Office1 roll-up (lean: per-DC only).
- Mirror sync topology (independent upstream pull per DC vs seed-from-peer).

## 11. Landing this deliverable

1. Append the body of `docs/dc-dc-D100-D110.append.md` (from its first `## D-100` line) to
   the end of `docs/design-decisions.md`, then remove the fragment file.
2. Run `bash scripts/ledger-scan.sh`; confirm D next-free = 111 and the machine-derived
   ledger block regenerates cleanly. Do not hand-edit the machine-derived block.
3. Add a one-line entry in the main-chat fence of `docs/session-ledger.md` recording this
   deliverable (design doc + D-100..D-110 PROPOSED).
4. `bash scripts/repo-lint.sh` for ASCII/LF/format (ignore any next-free noise above 099 --
   known band bug; ledger-scan is the next-free authority).
5. Commit. These are PROPOSED: present for redline, do not implement until the operator
   ratifies each to ADOPTED.