SEED: DC-DC cross-DC replication / DR design (Tier A)

Status: SEED / DRAFT for the DC-DC planning chat. Authored by the main-chat stream during DC-DC planning. This is a settled decision to formalize, not a committed record. The planning chat files the D-entry: assign next-free via ledger-scan at commit -- note D-076 is already claimed by the PINNED dashboard workstream (draft in a worktree), so the DR decision is the next free after that; confirm and coordinate before consuming.

Governing constraint: minimize delta to Roosevelt.

Decision (settled with the operator)

Two independent clouds (separate Keystones), so RBD-mirror is a data primitive, not service failover. DR posture = Tier A: data recoverability, bidirectional (each DC protects the other), operator-runbook failover.

Mechanism, per data class:

Cinder volumes -> cinder-backup to a cross-DC-replicated object store. This is the honest cross-independent-cloud path: a restore reconstitutes the volume with its Cinder metadata in the peer cloud, which raw RBD-mirror cannot. Low delta -- cinder already exposes the backup-backend endpoint (bound metal-internal) and ceph-radosgw is present as the S3 target.
Glance images -> snapshot-based RBD-mirror of the Glance pool, two-way, with a re-register step in the failover runbook.
Nova ephemeral -> NOT replicated. Ephemeral root disks are cattle; recovering them would also need the un-mirrored Nova instance records. Instances are rebuilt from image + volume on failover.

Parameters:

RPO = replication/backup interval; start ~15 min, tunable.
RTO = a measured output of the failover drill, not a pre-committed SLA.
Staging: prove one-way (single mechanism, single direction) + a clean failover/failback drill FIRST, then enable two-way.
Carrier: the replication plane (10.12.36.0/22) with an explicit inter-DC route + bandwidth budget -- distinct from intra-DC OSD cluster traffic that already uses this plane.
Not replicated: Neutron state (ports, FIPs, security groups) -- recovered workloads land on the peer's networks with new addresses; the runbook re-creates this.

Options considered

Tier A (CHOSEN): mirror/backup + operator rehydration runbook. RPO minutes, RTO hours/measured. Lowest complexity, most robust, delta-minimal.
Tier B (folded in for volumes): cinder-backup/restore -- adopted as the Cinder path because it crosses the independent-cloud boundary cleanly.
Tier C (deferred): orchestrated workload failover via control-plane metadata replay. Low RTO, high build cost, brittle (no first-class cross-independent-cloud OpenStack DR). Revisit only if a hard minutes-RTO requirement emerges.

Failover / failback runbook -- SKELETON (planning chat to flesh out against the live env)

Failover (DC1 lost, recover at DC2):

Confirm DC1 truly down (avoid split-brain from a transient partition).
Glance: promote the mirrored Glance pool at DC2 (force-promote if DC1 unreachable); re-register images into DC2 Glance.
Cinder: restore required volumes into DC2 Cinder from the replicated backups.
Rebuild instances from image + restored volume; re-create Neutron ports/FIPs/SGs.
Record RTO/RPO actuals.

Failback (DC1 recovered) -- split-brain-safe ordering for two-way:

Do NOT let recovered DC1 resume as primary automatically.
Demote DC1's Glance pool; resync from DC2 (now primary).
Reconcile Cinder: back up any DC2-side changes; restore into DC1.
In a controlled window, optionally flip primary back (demote DC2, promote DC1).
Verify both directions healthy; record the drill.

Open sub-items for the planning chat

radosgw multisite vs. a mirrored backup bucket as the replicated backup target.
Exact mirrored pools; consistency groups for multi-volume apps.
Inter-DC replication route + bandwidth budget on the simulated WAN (MTU-aware).
Interaction with the D-071 controller-HA decision (backup/restore of per-DC control planes).
ceph-rbd-mirror charm topology + daemon HA; peer bootstrap on the replication plane.