diff --git a/docs/dc-dc-replication-DR-seed.md b/docs/dc-dc-replication-DR-seed.md new file mode 100644 index 0000000..948ec6e --- /dev/null +++ b/docs/dc-dc-replication-DR-seed.md @@ -0,0 +1,75 @@ +# SEED: DC-DC cross-DC replication / DR design (Tier A) + +**Status:** SEED / DRAFT for the DC-DC planning chat. Authored by the main-chat stream +during DC-DC planning. This is a settled *decision* to formalize, not a committed record. +The planning chat files the D-entry: **assign next-free via `ledger-scan` at commit -- +note D-076 is already claimed by the PINNED dashboard workstream (draft in a worktree), so +the DR decision is the next free after that; confirm and coordinate before consuming.** + +**Governing constraint:** minimize delta to Roosevelt. + +--- + +## Decision (settled with the operator) + +Two **independent** clouds (separate Keystones), so RBD-mirror is a **data** primitive, not +service failover. DR posture = **Tier A: data recoverability**, **bidirectional** (each DC +protects the other), operator-runbook failover. + +Mechanism, per data class: +- **Cinder volumes -> `cinder-backup`** to a **cross-DC-replicated object store**. This is + the honest cross-independent-cloud path: a restore reconstitutes the volume *with its + Cinder metadata* in the peer cloud, which raw RBD-mirror cannot. Low delta -- `cinder` + already exposes the `backup-backend` endpoint (bound metal-internal) and `ceph-radosgw` + is present as the S3 target. +- **Glance images -> snapshot-based RBD-mirror** of the Glance pool, two-way, with a + re-register step in the failover runbook. +- **Nova ephemeral -> NOT replicated.** Ephemeral root disks are cattle; recovering them + would also need the un-mirrored Nova instance records. Instances are rebuilt from + image + volume on failover. + +Parameters: +- **RPO** = replication/backup interval; start ~15 min, tunable. +- **RTO** = a **measured output** of the failover drill, not a pre-committed SLA. +- **Staging:** prove **one-way** (single mechanism, single direction) + a clean + failover/failback drill FIRST, then enable **two-way**. +- **Carrier:** the **replication plane (`10.12.36.0/22`)** with an explicit **inter-DC + route + bandwidth budget** -- distinct from intra-DC OSD cluster traffic that already + uses this plane. +- **Not replicated:** Neutron state (ports, FIPs, security groups) -- recovered workloads + land on the peer's networks with new addresses; the runbook re-creates this. + +## Options considered + +- **Tier A (CHOSEN):** mirror/backup + operator rehydration runbook. RPO minutes, RTO + hours/measured. Lowest complexity, most robust, delta-minimal. +- **Tier B (folded in for volumes):** `cinder-backup`/restore -- adopted as the Cinder path + because it crosses the independent-cloud boundary cleanly. +- **Tier C (deferred):** orchestrated workload failover via control-plane metadata replay. + Low RTO, high build cost, brittle (no first-class cross-independent-cloud OpenStack DR). + Revisit only if a hard minutes-RTO requirement emerges. + +## Failover / failback runbook -- SKELETON (planning chat to flesh out against the live env) + +Failover (DC1 lost, recover at DC2): +1. Confirm DC1 truly down (avoid split-brain from a transient partition). +2. Glance: promote the mirrored Glance pool at DC2 (force-promote if DC1 unreachable); + re-register images into DC2 Glance. +3. Cinder: restore required volumes into DC2 Cinder from the replicated backups. +4. Rebuild instances from image + restored volume; re-create Neutron ports/FIPs/SGs. +5. Record RTO/RPO actuals. + +Failback (DC1 recovered) -- split-brain-safe ordering for two-way: +1. Do NOT let recovered DC1 resume as primary automatically. +2. Demote DC1's Glance pool; resync from DC2 (now primary). +3. Reconcile Cinder: back up any DC2-side changes; restore into DC1. +4. In a controlled window, optionally flip primary back (demote DC2, promote DC1). +5. Verify both directions healthy; record the drill. + +## Open sub-items for the planning chat +- radosgw **multisite** vs. a mirrored backup bucket as the replicated backup target. +- Exact mirrored pools; **consistency groups** for multi-volume apps. +- Inter-DC replication route + bandwidth budget on the simulated WAN (MTU-aware). +- Interaction with the **D-071** controller-HA decision (backup/restore of per-DC control + planes). +- `ceph-rbd-mirror` charm topology + daemon HA; peer bootstrap on the replication plane.