Changelog 2026-07-09 -- Stage 6 runbook: DR wiring and failover drill (DOCFIX-158)

No live infrastructure touched -- new runbook document only. Authored by a delegated subagent (general-purpose, briefed with this repo's discipline, the DR seed doc, D-108/D-104, Section 8's failover skeleton, and the tooling gap register), reviewed before this commit -- repo-lint was clean on the first pass.

Item

1. DOCFIX-158 -- runbooks/dc-dc-phase5-dr-failover-drill.md

FILES: runbooks/dc-dc-phase5-dr-failover-drill.md (new), docs/dc-dc- deployment-workflow.md (Stage 6 row updated, tooling gap register item 14 added).

WHAT: command-level implementation of docs/dc-dc-buildout-design.md Section 8's failover/failback skeleton, staged one-way-then-two-way per D-108's explicit requirement, plus the D-104 per-DC controller backup/ restore drill (exercised here, not deferred).

Three real bundle gaps found and flagged before any mutation is proposed (grepped bundle.yaml directly, not assumed): no cinder- backup charm is deployed in either DC's bundle (the DR seed's own text assumes cinder already exposes a backup-backend endpoint, which is true, but the separate charm that runs the backup driver isn't there); no ceph-rbd-mirror charm is deployed either; and ceph-mon's existing rbd-mirror endpoint binding is on storage, not replication (inherited from the VR0/DC0 seed at commit 8813efc, predates D-108's ULA- carrier requirement). All three become the runbook's own Step 2 (a gated, reviewed bundle-relation-binding change with its own changelog/revert requirement) rather than being silently routed around or left unmentioned.

Every realm/zonegroup/zone/pool/endpoint name is an explicit <UPPER_SNAKE_TBD> placeholder -- the runbook is explicit that the underlying Ceph commands (radosgw-admin realm/zonegroup/zone create, rbd mirror pool enable/peer bootstrap/promote/demote) are standard, well-documented upstream Ceph administration, not invented for this repo, while the specific names/endpoints/addresses genuinely don't exist yet and must never be guessed.

A negative-control drill precedes the real one: Step 10.0 runs a replication-plane-ONLY partition first (DC1 stays otherwise reachable) and explicitly states the split-brain-avoidance check in Step 10.1 SHOULD REFUSE to treat that as "DC1 down" -- if it promotes DC2 anyway under a mere partition, that's flagged as a finding against the drill design, not a pass. Only after that negative control does Step 10 run the real hard-down drill.

RTO/RPO are treated strictly as MEASURED DRILL OUTPUTS throughout (Steps 6, 7, 9, 10.5, 11.5), never asserted as a pre-committed SLA, matching D-108's own explicit language.

VERIFIED: bash scripts/repo-lint.sh 0 fail, 1 documented legacy warn (no fixes needed). NOT YET EXECUTED -- no live Ceph cluster this session.

REVERT: rm runbooks/dc-dc-phase5-dr-failover-drill.md && git checkout HEAD~ -- docs/dc-dc-deployment-workflow.md (safe).

Next actionable step

Once this drill has been run manually end-to-end at least once, the radosgw multisite/rbd-mirror command sequences it documents are strong candidates for extraction into real scripts/ with a test harness, closing tooling gap register item #5 for real -- explicitly logged as future work in the runbook's own delivery checklist, not built here (no live Ceph cluster exists this session to test a script against).