# SEED: DC-DC cross-DC replication / DR design (Tier A)

**Status:** SEED / DRAFT for the DC-DC planning chat. Authored by the main-chat stream
during DC-DC planning. This is a settled *decision* to formalize, not a committed record.
The planning chat files the D-entry: **assign next-free via `ledger-scan` at commit --
note D-076 is already claimed by the PINNED dashboard workstream (draft in a worktree), so
the DR decision is the next free after that; confirm and coordinate before consuming.**

**Governing constraint:** minimize delta to Roosevelt.

---

## Decision (settled with the operator)

Two **independent** clouds (separate Keystones), so RBD-mirror is a **data** primitive, not
service failover. DR posture = **Tier A: data recoverability**, **bidirectional** (each DC
protects the other), operator-runbook failover.

Mechanism, per data class:
- **Cinder volumes -> `cinder-backup`** to a **cross-DC-replicated object store**. This is
  the honest cross-independent-cloud path: a restore reconstitutes the volume *with its
  Cinder metadata* in the peer cloud, which raw RBD-mirror cannot. Low delta -- `cinder`
  already exposes the `backup-backend` endpoint (bound metal-internal) and `ceph-radosgw`
  is present as the S3 target.
- **Glance images -> snapshot-based RBD-mirror** of the Glance pool, two-way, with a
  re-register step in the failover runbook.
- **Nova ephemeral -> NOT replicated.** Ephemeral root disks are cattle; recovering them
  would also need the un-mirrored Nova instance records. Instances are rebuilt from
  image + volume on failover.

Parameters:
- **RPO** = replication/backup interval; start ~15 min, tunable.
- **RTO** = a **measured output** of the failover drill, not a pre-committed SLA.
- **Staging:** prove **one-way** (single mechanism, single direction) + a clean
  failover/failback drill FIRST, then enable **two-way**.
- **Carrier:** the **replication plane (`10.12.36.0/22`)** with an explicit **inter-DC
  route + bandwidth budget** -- distinct from intra-DC OSD cluster traffic that already
  uses this plane.
- **Not replicated:** Neutron state (ports, FIPs, security groups) -- recovered workloads
  land on the peer's networks with new addresses; the runbook re-creates this.

## Options considered

- **Tier A (CHOSEN):** mirror/backup + operator rehydration runbook. RPO minutes, RTO
  hours/measured. Lowest complexity, most robust, delta-minimal.
- **Tier B (folded in for volumes):** `cinder-backup`/restore -- adopted as the Cinder path
  because it crosses the independent-cloud boundary cleanly.
- **Tier C (deferred):** orchestrated workload failover via control-plane metadata replay.
  Low RTO, high build cost, brittle (no first-class cross-independent-cloud OpenStack DR).
  Revisit only if a hard minutes-RTO requirement emerges.

## Failover / failback runbook -- SKELETON (planning chat to flesh out against the live env)

Failover (DC1 lost, recover at DC2):
1. Confirm DC1 truly down (avoid split-brain from a transient partition).
2. Glance: promote the mirrored Glance pool at DC2 (force-promote if DC1 unreachable);
   re-register images into DC2 Glance.
3. Cinder: restore required volumes into DC2 Cinder from the replicated backups.
4. Rebuild instances from image + restored volume; re-create Neutron ports/FIPs/SGs.
5. Record RTO/RPO actuals.

Failback (DC1 recovered) -- split-brain-safe ordering for two-way:
1. Do NOT let recovered DC1 resume as primary automatically.
2. Demote DC1's Glance pool; resync from DC2 (now primary).
3. Reconcile Cinder: back up any DC2-side changes; restore into DC1.
4. In a controlled window, optionally flip primary back (demote DC2, promote DC1).
5. Verify both directions healthy; record the drill.

## Open sub-items for the planning chat
- radosgw **multisite** vs. a mirrored backup bucket as the replicated backup target.
- Exact mirrored pools; **consistency groups** for multi-volume apps.
- Inter-DC replication route + bandwidth budget on the simulated WAN (MTU-aware).
- Interaction with the **D-071** controller-HA decision (backup/restore of per-DC control
  planes).
- `ceph-rbd-mirror` charm topology + daemon HA; peer bootstrap on the replication plane.