# Changelog 2026-07-09 -- Stage 6 runbook: DR wiring and failover drill (DOCFIX-158)

No live infrastructure touched -- new runbook document only. Authored by a
delegated subagent (general-purpose, briefed with this repo's discipline,
the DR seed doc, D-108/D-104, Section 8's failover skeleton, and the
tooling gap register), reviewed before this commit -- repo-lint was clean
on the first pass.

## Item

### 1. DOCFIX-158 -- `runbooks/dc-dc-phase5-dr-failover-drill.md`
FILES: `runbooks/dc-dc-phase5-dr-failover-drill.md` (new), `docs/dc-dc-
deployment-workflow.md` (Stage 6 row updated, tooling gap register item 14
added).

WHAT: command-level implementation of `docs/dc-dc-buildout-design.md`
Section 8's failover/failback skeleton, staged one-way-then-two-way per
D-108's explicit requirement, plus the D-104 per-DC controller backup/
restore drill (exercised here, not deferred).

**Three real bundle gaps found and flagged before any mutation is
proposed** (grepped `bundle.yaml` directly, not assumed): no `cinder-
backup` charm is deployed in either DC's bundle (the DR seed's own text
assumes cinder already exposes a backup-backend endpoint, which is true,
but the separate charm that runs the backup driver isn't there); no
`ceph-rbd-mirror` charm is deployed either; and `ceph-mon`'s existing
`rbd-mirror` endpoint binding is on `storage`, not `replication`
(inherited from the VR0/DC0 seed at commit 8813efc, predates D-108's ULA-
carrier requirement). All three become the runbook's own Step 2 (a gated,
reviewed bundle-relation-binding change with its own changelog/revert
requirement) rather than being silently routed around or left unmentioned.

**Every realm/zonegroup/zone/pool/endpoint name is an explicit
`<UPPER_SNAKE_TBD>` placeholder** -- the runbook is explicit that the
underlying Ceph commands (`radosgw-admin realm/zonegroup/zone create`,
`rbd mirror pool enable/peer bootstrap/promote/demote`) are standard,
well-documented upstream Ceph administration, not invented for this repo,
while the specific names/endpoints/addresses genuinely don't exist yet and
must never be guessed.

**A negative-control drill precedes the real one:** Step 10.0 runs a
replication-plane-ONLY partition first (DC1 stays otherwise reachable) and
explicitly states the split-brain-avoidance check in Step 10.1 SHOULD
REFUSE to treat that as "DC1 down" -- if it promotes DC2 anyway under a
mere partition, that's flagged as a finding against the drill design, not
a pass. Only after that negative control does Step 10 run the real
hard-down drill.

RTO/RPO are treated strictly as MEASURED DRILL OUTPUTS throughout (Steps
6, 7, 9, 10.5, 11.5), never asserted as a pre-committed SLA, matching
D-108's own explicit language.

VERIFIED: `bash scripts/repo-lint.sh` 0 fail, 1 documented legacy warn (no
fixes needed). NOT YET EXECUTED -- no live Ceph cluster this session.

REVERT: `rm runbooks/dc-dc-phase5-dr-failover-drill.md && git checkout
HEAD~ -- docs/dc-dc-deployment-workflow.md` (safe).

## Next actionable step

Once this drill has been run manually end-to-end at least once, the
radosgw multisite/rbd-mirror command sequences it documents are strong
candidates for extraction into real `scripts/` with a test harness,
closing tooling gap register item #5 for real -- explicitly logged as
future work in the runbook's own delivery checklist, not built here (no
live Ceph cluster exists this session to test a script against).