# DC-DC Phase 5 -- DR wiring and failover drill (Stage 6) Wire the Tier-A cross-DC replication mechanism (`docs/dc-dc-replication-DR-seed.md`, engineered by **D-108**) between DC1 and DC2, stage it one-way, prove it clean, enable two-way, then run the actual failover and failback drills plus the per-DC Juju controller backup/restore drill (**D-104**). This is the first DC-DC runbook that deliberately takes a live DC down (simulated, not physical) as part of the drill itself -- treat Steps 9-11 with the same seriousness as `phase-00-teardown-maas-reset.md`'s own destructive steps. Entry condition: **Stage 5's gate** (`docs/dc-dc-deployment-workflow.md`) has passed for BOTH DC1 and DC2 independently -- `preflight.sh` PASS before add-model, post-deploy `cloud-assert.sh --capture` per DC, a controller backup already taken and stored, Ceph-over-v6 and geneve-over-v6 verified. Do not start this runbook against a DC that hasn't cleared that gate; whichever sibling runbook carries Stage 5 is the authority for confirming it, not this document. Decisions this runbook executes against: **D-108** (cross-DC replication mechanism: cinder-backup via `ceph-radosgw` MULTISITE zonegroup; Glance via `ceph-rbd-mirror`; Nova ephemeral not replicated; carrier is the IPv6-only ULA replication plane per **D-101**, correcting the DR-seed's original v4 carrier language; one-way staged before two-way; RPO ~15 min tunable; RTO is a measured drill output, not an SLA), and **D-104** (per-DC single-unit Juju controllers, `create-backup`/`download-backup`, restore drilled HERE, not just storage failover). Governing design: `docs/dc-dc-buildout-design.md` Section 4's Phase 5 entry and Section 8's failover/failback skeleton (reproduced faithfully below, at command-level, not redesigned). Tracker row: `docs/dc-dc-deployment-workflow.md` Stage 6 and its Tooling gap register item #5 (Ceph replication tooling -- confirmed NO script in this repo touches radosgw multisite or rbd-mirror; genuinely net-new) and item #1 ($DC selector -- DOCFIX-151 -- this runbook operates against TWO DCs, so any `scripts/lib-net.sh` session calls `lib_net_select_dc "$DC"` explicitly, once, immediately after sourcing). --- !!! DESTRUCTIVE. Steps 9-11 deliberately take DC1 "down" (a controlled simulated outage -- either a replication-plane partition or a hard shutdown of DC1's node VMs, see Step 9) to run a real failover drill, then bring it back in a split-brain-safe failback. There is no scripted rollback for a botched promote/demote sequence on live Ceph pools -- a wrong-order `rbd mirror pool promote --force` or radosgw zone demote on BOTH sides at once is exactly the split-brain this drill exists to avoid. Each promote/demote/restore action is DISCRETE and individually gated -- do not batch them, and do not run this against Roosevelt or any cloud you cannot afford to actually lose data on. !!! GENUINELY NEW TOOLING. No script in this repo touches radosgw multisite or rbd-mirror (Tooling gap register item #5) -- the commands below are STANDARD, well-documented upstream Ceph administration commands (`radosgw-admin realm/zonegroup/zone create`, `rbd mirror pool enable/peer bootstrap/promote/demote`, etc.), not invented for this repo. What IS genuinely unknown and MUST NOT be inferred: the exact realm/zonegroup/zone names, pool names, radosgw endpoint URLs/ports, and the real replication-plane ULA addresses for DC1 and DC2 -- none of that exists yet. Every placeholder below is written as `` and MUST be replaced with a value measured against the real DC1/DC2 Ceph clusters at execution time (Stage 5's job, confirmed again here before this runbook mutates anything). If a placeholder is still a placeholder when you reach a MUTATION step, STOP -- that is an inferred value entering a command, the hardest rule this repo has. --- ## Known gaps / prerequisites -- confirmed missing, not fixed by this runbook Grepped `bundle.yaml` before writing this (2026-07-09/10 sweep, same repo state as the Tooling gap register audit): - **No `cinder-backup` charm is deployed in either DC's bundle.** The DR seed's own text (`docs/dc-dc-replication-DR-seed.md` line 24) assumes cinder already exposes the `backup-backend` endpoint -- true for `cinder` itself, but the separate `cinder-backup` subordinate/charm that actually runs the backup driver is NOT in `bundle.yaml` today. Adding it is a bundle change (a built-surface edit, gated the same as any other bundle change) -- Step 1 below confirms this live and, if still absent, this runbook's first real MUTATION is adding it, NOT skipping past it. - **No `ceph-rbd-mirror` charm is deployed either.** Same situation: D-108's Glance mechanism needs the charm; it is not in `bundle.yaml`. - **`ceph-mon`'s existing `rbd-mirror` endpoint is bound to `storage`, not `replication`** (`bundle.yaml` line 515: `rbd-mirror: storage`). D-108 requires the cross-DC rbd-mirror daemon traffic to ride the IPv6-only ULA **replication** plane (`10.12.36.0/22` family), not the intra-DC OSD storage plane. This binding predates D-108 (inherited from the VR0/DC0 seed at commit 8813efc) and needs correcting as part of standing up rbd-mirror here -- flag it, correct it as its own gated bundle-relation-binding change, do not silently leave the cross-DC daemon traffic on the wrong plane. **UPDATE 2026-07-10 (DOCFIX-167):** `bundle.yaml` ITSELF now carries `cinder-backup`, `ceph-rbd-mirror`, and the corrected `rbd-mirror: replication` binding (see `docs/changelog-20260709-designate-cinderbackup-rbdmirror.md`) -- so any DC deployed via Stage 5's runbook AFTER this date gets all three from the initial `juju deploy`, and Step 1's CHECK below should find them already correct (skip straight to Step 3). Step 2 remains here, unmodified, for the case this runbook is run against a DC deployed from an OLDER bundle.yaml snapshot (pre-2026-07-10) that predates this fix -- do not assume Step 2 is dead code just because a fresh deploy no longer needs it. Still not live-tested: no Ceph cluster exists this session (2026-07-09/10 is PREP-ONLY) to confirm either path actually works end-to-end. --- ## Sequence ``` 1. Confirm Stage 5 gate + bundle prerequisites (read-only; may surface bundle gaps above) 2. Add/fix cinder-backup + ceph-rbd-mirror wiring [MUTATION: bundle change, gated, per DC] 3. Confirm replication-plane (ULA) peering (read-only; latency/bandwidth recorded) 4. Stage ONE-WAY: radosgw realm/zonegroup/zone [MUTATION: Ceph multisite, gated] 5. Stage ONE-WAY: rbd-mirror peer bootstrap (rx-only) [MUTATION: Ceph rbd-mirror, gated] 6. One-way validation drill (write DC1 -> observe DC2) (read-only observation of RPO) 7. One-way failover/failback micro-drill; RECORD RTO/RPO [DESTRUCTIVE: small-scale promote] -> GATE 1: one-way clean, RTO/RPO measured 8. Enable TWO-WAY (bidirectional zone sync; rbd-mirror rx-tx both sides) [MUTATION, gated] 9. Controller backup + restore drill (D-104), per DC [MUTATION: backup + restore test] 10. FULL failover drill: DC1 "down" -> promote/restore/rebuild at DC2 [DESTRUCTIVE] 11. FULL failback drill: split-brain-safe, controlled window [DESTRUCTIVE] 12. Record final RTO/RPO actuals; update tracker docs -> EXIT GATE -> Stage 7 (Designate, COS, Magnum/CAPI) ``` --- ## Step 1 -- Confirm Stage 5 gate + bundle prerequisites (READ-ONLY) **CHECK -- both DCs independently, from wherever this session runs** ```bash source scripts/lib-net.sh lib_net_select_dc "dc1" # explicit, once -- gap #1 / DOCFIX-151 call-site wiring # repeat the pair below for dc2 in its own shell/session context; do not assume dc2's # values from dc1 -- lib_net_select_dc fails loud for dc2 until its literals are real bash scripts/cloud-assert.sh ``` Expect PASS on both DCs. If either is not clean, STOP -- this runbook's entry condition is unmet; go fix Stage 5 first, do not proceed with a shaky base under a DR drill. **CHECK -- bundle prerequisites (per DC)** ```bash juju status --format=json | jq -r '.applications | keys[]' | grep -E '^cinder-backup$|^ceph-rbd-mirror$' juju status --format=json | jq -r '.applications["ceph-mon"].endpoint-bindings // {}' ``` Expect `cinder-backup` and `ceph-rbd-mirror` both present, and the `rbd-mirror` binding already on the replication space. If either charm is absent, or the binding is still on `storage`, this confirms the "Known gaps" section above live -- proceed to Step 2. If both are already correct (e.g. a prior partial run of this runbook already fixed them), skip to Step 3. **GATE:** Stage 5 clean on both DCs; bundle prerequisites either confirmed present+correct, or confirmed absent (routed to Step 2). --- ## Step 2 -- Add/fix cinder-backup + ceph-rbd-mirror wiring [MUTATION: bundle change, gated, per DC] > CAUTION: edits `bundle.yaml` (a built-surface change) and applies it via `juju deploy`/ > `juju add-relation`/`juju bind` against a LIVE cloud. This is the first mutation this > runbook performs. Present the diff, confirm the charm channel matches the rest of the > bundle's Caracal/squid pins, and get explicit go before applying -- do not batch this with > Step 1's read-only checks. Only the specific fixes identified in Step 1 -- do not use this step to make any other bundle change (no adjacent improvements mid-step, per this repo's hard rule 1). **MUTATION -- per DC, only for whichever gap Step 1 found live** ```bash # (a) if cinder-backup is absent: juju deploy cinder-backup --channel 2024.1/stable --to lxd: juju add-relation cinder-backup:backup-backend cinder:backup-backend juju add-relation cinder-backup:ceph cinder-ceph:ceph # or the real endpoint name -- # confirm against `juju info cinder-backup` # rather than assuming this shape juju add-relation cinder-backup:amqp rabbitmq-server:amqp # (b) if ceph-rbd-mirror is absent: juju deploy ceph-rbd-mirror --channel squid/stable --to lxd: juju add-relation ceph-rbd-mirror:ceph-rbd-mirror ceph-mon:rbd-mirror # (c) if ceph-mon's rbd-mirror binding is on storage, not replication: juju bind ceph-mon rbd-mirror=replication ``` `` is a real MAAS/LXD placement decision on this DC's machines -- resolve it live (`juju status` machine list) the same way every other phase runbook in this repo resolves placement, never a copied number from DC0's bundle. **VERIFY** ```bash juju status --format=json | jq -r '.applications | keys[]' | grep -E '^cinder-backup$|^ceph-rbd-mirror$' juju status --format=json | jq -r '.applications["ceph-mon"].endpoint-bindings["rbd-mirror"]' bash scripts/cloud-assert.sh ``` **GATE:** both charms active/idle, `rbd-mirror` binding shows `replication`, `cloud-assert` still PASS after the bundle change. Log this as a changelog entry (this IS a built-surface change) with a revert (`juju remove-application`/`juju bind ... =storage` back) before moving on. --- ## Step 3 -- Confirm replication-plane (ULA) peering (READ-ONLY) **CHECK -- per DC, the replication plane's real ULA addresses** ```bash ip -6 -o addr show scope global | grep -i fd # or whatever prefix NetBox assigned -- confirm # against NetBox, do not assume fd00::/8 shape ``` Record the real ULA address each DC's ceph-mon/ceph-rbd-mirror/ceph-radosgw units hold on the replication plane (`10.12.36.0/22` family, per D-101 -- v6 ULA side). These are the addresses every command from Step 4 onward uses; nothing here is invented. **CHECK -- inter-DC route + latency/bandwidth budget on the simulated WAN** ```bash # from a DC1 replication-plane host to its DC2 peer's REAL measured ULA address: ping6 -c 20 ``` Record latency/jitter/loss actuals. Per `docs/dc-dc-buildout-design.md` Section 6, this is the `tc netem`-simulated inter-DC leg (`opentofu/modules/netem-link`, mesh_dc1_dc2) -- confirm its parameters are ACTUALLY APPLIED (not just planned) before trusting any RTT number recorded here; a netem link with no rule applied yet will look artificially fast. **GATE:** both DCs' real replication-plane ULA addresses recorded; inter-DC reachability and link-profile actuals recorded against the netem parameters in effect. If the link is down or unmeasured, STOP -- Step 4 onward needs it. --- ## Step 4 -- Stage ONE-WAY: radosgw realm/zonegroup/zone (multisite) [MUTATION: gated] Per D-108's own explicit staging requirement, prove ONE mechanism in ONE direction before anything bidirectional. This step wires radosgw multisite for the cinder-backup target, DC1 as master zone only. > CAUTION: `radosgw-admin ... --commit` mutates the realm/period on a LIVE radosgw. Run each > command, verify its output, THEN proceed to the next -- do not chain the whole sequence > unattended. **MUTATION -- on a DC1 ceph-radosgw unit** ```bash radosgw-admin realm create --rgw-realm= --default radosgw-admin zonegroup create --rgw-zonegroup= \ --endpoints=http:// --master --default radosgw-admin zone create --rgw-zonegroup= --rgw-zone= \ --endpoints=http:// --master --default radosgw-admin period update --commit ``` Restart the DC1 radosgw unit (`juju run ceph-radosgw/0 restart` or the charm's own action -- confirm the current action name against `juju actions ceph-radosgw` rather than assuming) so it picks up realm membership. **VERIFY** ```bash radosgw-admin realm list radosgw-admin zonegroup get --rgw-zonegroup= ``` **GATE:** DC1 shows as the sole zone in the new zonegroup, master, default. DC2 is NOT yet joined -- that is deliberate (one-way staging); joining DC2 happens only after Step 6/7's one-way drill is clean (see "then enable two-way" -- Step 8, not here). For the ONE-WAY drill itself, DC2's radosgw still needs to be a member zone of the SAME zonegroup (multisite sync is inherently zone-to-zone; "one-way" here means DC2 is configured `--read-only`/non-master, i.e. it can only receive, never author, matching D-108's "single mechanism, single direction" staging intent), not a fully separate radosgw: ```bash # on the DC2 ceph-radosgw unit, joining the SAME realm/zonegroup as a read-only zone: radosgw-admin zone create --rgw-zonegroup= --rgw-zone= \ --endpoints=http:// --read-only \ --access-key= --secret= radosgw-admin period update --commit ``` The access-key/secret pair is the multisite system user radosgw itself generates at `realm create` time on DC1 -- capture it there (`radosgw-admin user list` / `radosgw-admin user info --uid=<...>`), never invent it. Restart DC2's radosgw unit. **VERIFY** ```bash radosgw-admin sync status --rgw-zone= ``` **GATE (Step 4 complete):** DC2 zone shows in the zonegroup, `--read-only`, sync status reports it caught up (not "behind" indefinitely). This is the multisite half of one-way staging; Step 5 does the same for rbd-mirror. --- ## Step 5 -- Stage ONE-WAY: rbd-mirror peer bootstrap (rx-only) [MUTATION: gated] **CHECK -- confirm the Glance pool name** (do not assume `glance` -- confirm against the live ceph-mon: `ceph osd pool ls` and cross-check which pool glance-charm's `rbd-pool` config points at) ```bash ceph osd pool ls juju config glance rbd-pool ``` **MUTATION -- on DC1's ceph-rbd-mirror / ceph-mon unit** ```bash rbd mirror pool enable image rbd mirror pool peer bootstrap create --site-name \ > /tmp/rbd-mirror-bootstrap-token ``` Transfer the token to DC2 out of band (it is secret-adjacent -- treat it like any other credential material this repo's guard hook flags: never printed into this session's context, copied via a channel the operator controls directly). **MUTATION -- on DC2's ceph-rbd-mirror / ceph-mon unit** ```bash rbd mirror pool enable image rbd mirror pool peer bootstrap import --site-name --direction rx-only \ /tmp/rbd-mirror-bootstrap-token ``` `--direction rx-only` is the one-way staging: DC2 can only RECEIVE mirrored snapshots, never originate. Two-way (`rx-tx`) is Step 8, only after the one-way drill (Step 6/7) is clean. **VERIFY** ```bash rbd mirror pool info --all rbd mirror pool status --verbose ``` **GATE (Step 5 complete):** DC2 shows as an `rx-only` peer of DC1 for the Glance pool; pool status is `health OK`, no images stuck in an error/split-brain state. --- ## Step 6 -- One-way validation drill: write DC1, observe DC2 (READ-ONLY OBSERVATION) **Cinder-backup path** ```bash # on DC1 (a real tenant/volume this session has, not a fabricated id): openstack volume backup create --name dr-drill-oneway- ``` Then, from DC2's radosgw: ```bash radosgw-admin bucket list --rgw-zone= radosgw-admin sync status --rgw-zone= ``` Record the wall-clock time from the `backup create` call to the object appearing synced at DC2 -- this is one real RPO data point (target ~15 min per D-108, tunable, MEASURED here, not assumed). **Glance rbd-mirror path** ```bash # on DC1: take/verify a snapshot on a real Glance-backed image; a normal glance image import # already creates the pool object rbd-mirror snapshots against. rbd mirror image status / ``` Then on DC2: ```bash rbd mirror image status / ``` Record the time for `state` to reach `up+replaying` with a matching snapshot timestamp on DC2. This is the Glance-side RPO data point. **GATE:** at least one clean cinder-backup sync and one clean rbd-mirror sync observed at DC2, both within a recorded, measured time -- not assumed to already work because the mirror "looks enabled." --- ## Step 7 -- One-way failover/failback micro-drill; RECORD RTO/RPO [DESTRUCTIVE: small-scale] Before the FULL drill (Step 10-11), run a small-scale version against the ONE-WAY setup, using only the test volume/image from Step 6 -- this proves the mechanics (promote/restore/re-register) work at all before staking a real DC-down scenario on them. > CAUTION: `rbd mirror image promote` and `openstack volume backup restore` are real actions > against the test artifacts from Step 6. Confirm you are promoting/restoring the DRILL > objects, not a real tenant's data, before running. **MUTATION** ```bash # force-promote is used here deliberately (DC1 stays technically reachable in this micro-drill # -- this exercises the promote MECHANISM, the split-brain-avoidance CHECK is exercised for # real only in Step 10, where DC1 is genuinely unreachable): rbd mirror image promote --force / openstack --os-cloud dc2 volume backup restore dr-drill-oneway- --volume dr-drill-oneway-restored- ``` **VERIFY / RECORD** ```bash openstack --os-cloud dc2 volume show dr-drill-oneway-restored- rbd mirror image status / ``` Record: time from the "confirm DC1 down" decision point (here, decision-to-drill) to a usable restored volume/promoted image at DC2 -- this is the RTO measured output. Do NOT assert a target number; the buildout design and D-108 are explicit that RTO is the drill's OUTPUT, not a pre-committed SLA. **Demote back** (this micro-drill does not leave DC2 promoted afterward): ```bash rbd mirror image demote / ``` **GATE 1 (Stage 6 first exit sub-condition):** one-way drill clean (Step 6 RPO observed, Step 7 RTO measured and recorded), radosgw multisite and rbd-mirror both proven end-to-end at small scale. Only after this gate passes does Step 8 enable two-way. --- ## Step 8 -- Enable TWO-WAY [MUTATION: gated] **MUTATION -- radosgw: make DC2's zone a full read-write member** ```bash radosgw-admin zone modify --rgw-zone= --read-only=false radosgw-admin period update --commit ``` **MUTATION -- rbd-mirror: re-import the peer with bidirectional direction** ```bash # on DC1, generate a fresh bootstrap token (or reuse Step 5's peer relationship and just # flip direction -- confirm against `rbd mirror pool peer` current subcommand behavior, # it may require remove+re-add rather than an in-place direction change): rbd mirror pool peer bootstrap create --site-name \ > /tmp/rbd-mirror-bootstrap-token-twoway # on DC2: rbd mirror pool peer bootstrap import --site-name --direction rx-tx \ /tmp/rbd-mirror-bootstrap-token-twoway ``` **VERIFY** ```bash radosgw-admin sync status --rgw-zone= radosgw-admin sync status --rgw-zone= rbd mirror pool status --verbose ``` **GATE:** both zones show each other caught up in `sync status`; rbd-mirror pool status shows `rx-tx` in both directions, health OK. Re-run Step 6's write-and-observe check in the OPPOSITE direction (write at DC2, observe at DC1) to prove two-way is real, not just configured. --- ## Step 9 -- Controller backup + restore drill (D-104), per DC [MUTATION: backup + restore test] D-104 requires this drill exercised HERE, not deferred -- this is control-plane recovery, a distinct surface from the storage failover above. **MUTATION -- per DC, take a fresh controller backup** ```bash bash scripts/cloud-snapshot.sh --with-controller-backup ``` This runs `juju create-backup -m admin/controller` and downloads the archive to `~/openstack-baseline/` (jumphost-local; secret-adjacent; never committed -- confirmed by this script's own header, DOCFIX-088). Record the archive path/size/sha256 from its manifest. **CHECK -- confirm the real restore command/flags before using them** ```bash juju restore-backup --help ``` Do not assume flags from memory or from a different juju version's documentation -- confirm against this controller's actual installed juju client (3.6 per D-104) before running restore for real. This repo's own discipline (never an inferred value) applies to CLI flags too, not just IPs/IDs. **MUTATION -- restore drill** (decide live, per the `--help` output and this DC's actual topology, whether the drill targets a fresh scratch controller machine or an in-place restore path; do not assume either shape without confirming what the installed juju version supports) ```bash juju restore-backup --file ``` **VERIFY** ```bash juju status -m admin/controller juju status -m openstack ``` **GATE:** the controller resumes management of this DC's `openstack` model post-restore; `cloud-assert.sh` still PASS. Record wall-clock time from backup-taken to restore-verified-working as this drill's own RTO data point (control-plane, distinct from the storage RTO recorded in Step 7/12). Repeat for the OTHER DC's controller independently -- D-104's whole point is per-DC independence; a DC2 controller drill proves nothing about DC1's and vice versa. --- ## Step 10 -- FULL failover drill: DC1 "down" [DESTRUCTIVE] > CAUTION: this step deliberately makes DC1 unreachable/non-authoritative, then promotes DC2 > and rebuilds real workload state there. Confirm every real tenant/workload this touches is > either drill-only test material or something you are fully prepared to have live at DC2 > only until failback. Individually gated -- present each promote/restore/rebuild action, > confirm, then proceed. Do not batch. **Step 10.0 -- choose and record how "down" is simulated, then confirm it is truly down, not a transient partition** (Section 8 skeleton item 1). Two shapes, pick deliberately and record which: - **(a) Replication-plane-only partition** (tests the split-brain-avoidance CHECK itself): sever only the DC1<->DC2 leg on the replication plane, e.g. via the `netem-link` mechanism (`opentofu/modules/netem-link` / a direct `tc qdisc ... netem loss 100%` on the mesh_dc1_dc2 bridge -- confirm the real interface/bridge name against this session's own OpenTofu state, do not assume it). DC1 itself stays fully up and reachable from Office1/its own tenants. **Expected correct behavior: the split-brain check in Step 10.1 below should REFUSE to treat this as "DC1 down"** (it's still reachable via an out-of-band path) -- if this drill promotes DC2 anyway under a mere partition, that is a FINDING against the drill design, not a pass. - **(b) Hard down** (the real failover scenario): shut down DC1's node VMs entirely (`virsh shutdown ` for each, or the DC1 OpenTofu-managed equivalent) -- confirm real domain names against this session's `virsh list --all`, never assumed from DC0's naming. Run (a) FIRST as a negative-control drill (prove the check correctly refuses), THEN (b) as the real drill this stage's gate is measuring. **Step 10.1 -- confirm DC1 truly down, avoiding split-brain** (Section 8 skeleton item 1) ```bash # from Office1 or any vantage point INDEPENDENT of the DC1<->DC2 replication leg: ping openstack --os-cloud dc1 token issue # or any cheap DC1 API reachability probe ``` Require BOTH the replication-plane path AND at least one independent out-of-band path to agree DC1 is unreachable before proceeding -- a DC2-only view (replication link down) is exactly the ambiguous signal Section 8's skeleton warns about. Record the evidence. **Step 10.2 -- Glance: promote at DC2** (Section 8 skeleton item 2) ```bash rbd mirror pool promote --force ``` `--force` because DC1 is (by 10.1's confirmation) genuinely unreachable to demote first -- this is the documented force-promote path, not an improvised shortcut. Re-register images into DC2 Glance: for each image that needs to exist at DC2, `openstack --os-cloud dc2 image create` pointing at the now-promoted RBD object (or the equivalent re-register mechanism Glance's rbd driver exposes) -- confirm the exact invocation against the deployed Glance charm version's driver behavior at execution time; this per-image re-registration is exactly the kind of mechanism the Delivery checklist below flags as a good `scripts/` extraction candidate once this has been run manually once. **Step 10.3 -- Cinder: restore from replicated backups** (Section 8 skeleton item 3) ```bash openstack --os-cloud dc2 volume backup restore --volume ``` Repeat per volume that needs recovery. Record each restore's start/end time. **Step 10.4 -- rebuild instances; re-create Neutron state** (Section 8 skeleton item 4) ```bash openstack --os-cloud dc2 server create --image \ --volume --flavor --network openstack --os-cloud dc2 floating ip create openstack --os-cloud dc2 security group create ``` Neutron state (ports/FIPs/security groups) is NOT replicated by design (D-108) -- this is expected manual re-creation against DC2's own address space, not a bug to route around. **Step 10.5 -- RECORD RTO/RPO actuals** (Section 8 skeleton item 5) Tabulate: time-of-last-known-good-sync (RPO, from Step 6/Step 8's data) vs. time-DC1-confirmed-down (10.1) vs. time-workloads-usable-at-DC2 (10.4 complete). This tuple is the drill's real output -- write it into this runbook's execution log / changelog, not as an assumed number. **GATE (Stage 6 second exit sub-condition):** DC2 is running the drill workload(s) successfully; RTO/RPO recorded as measured, not assumed. --- ## Step 11 -- FULL failback drill: split-brain-safe, controlled window [DESTRUCTIVE] Only after DC1 is confirmed genuinely recovered (reverse of Step 10.1's check -- DC1 reachable again via an independent path, not just "the partition healed"). > CAUTION: the ordering below is the split-brain-safety mechanism itself. Demote the CURRENT > primary (DC2) before promoting the recovering side (DC1) -- promoting both, or promoting DC1 > before demoting DC2, creates two masters writing independently. Do not reorder these steps > even if it seems faster. **Step 11.1 -- do NOT let DC1 auto-resume as primary** (Section 8 skeleton item 1) No command here by design -- this is a negative instruction. Confirm no automation (charm hook, cron, systemd timer) on DC1 is configured to auto-promote on recovery; if one exists, that is itself a finding to log, not silently accept. **Step 11.2 -- demote DC1's Glance pool; resync from DC2 (now primary)** (item 2) ```bash rbd mirror pool demote # on DC1, now that it's back rbd mirror pool status --verbose # confirm DC1 catching up from DC2 ``` **Step 11.3 -- reconcile Cinder: back up DC2-side changes made during the outage, restore into DC1** (item 3) ```bash openstack --os-cloud dc2 volume backup create --name failback-reconcile- # once synced to DC1's radosgw zone (multisite sync, now bidirectional per Step 8): openstack --os-cloud dc1 volume backup restore failback-reconcile- --volume ``` **Step 11.4 -- in a controlled window, optionally flip primary back** (item 4) ```bash rbd mirror pool demote # DC2 (current primary) demotes FIRST rbd mirror pool promote # DC1 promotes SECOND -- never simultaneous ``` This is "optional" per Section 8's own skeleton -- if the operator's plan is to leave DC2 as primary for a while, stop after 11.3 and record that as the deliberate state, not an incomplete drill. **Step 11.5 -- verify both directions healthy; record the drill** (item 5) ```bash rbd mirror pool status --verbose radosgw-admin sync status --rgw-zone= radosgw-admin sync status --rgw-zone= ``` **GATE:** both pools/zones report healthy bidirectional sync; no image or bucket left in a half-migrated state. Record the failback drill's own timing the same way Step 10.5 did. --- ## Step 12 -- Record final RTO/RPO actuals; update tracker docs Consolidate every timing recorded in Steps 6, 7, 9, 10.5, and 11.5 into this runbook's execution log entry (changelog + `docs/session-ledger.md`, per this repo's standard delivery discipline). Update `docs/dc-dc-deployment-workflow.md` Stage 6's `**State:**` line from NOT STARTED to whatever is honestly true after this run (fully DONE only if every GATE above passed; partial otherwise, with the specific blocking item named). --- ## GATE (Stage 6 exit condition, restated from `docs/dc-dc-buildout-design.md` Section 4 and `docs/dc-dc-deployment-workflow.md` Stage 6) - One-way drill clean: Step 6 RPO observed for both cinder-backup and rbd-mirror; Step 7 RTO measured and recorded (GATE 1). - Two-way enabled and proven bidirectionally (Step 8). - Controller backup/restore drill exercised per DC (Step 9, D-104) -- not deferred to a later stage. - Full failover drill (Step 10) and full failback drill (Step 11) both run, split-brain-safe ordering observed throughout. - RTO recorded as a MEASURED output of the drills above (Steps 7, 9, 10.5, 11.5) -- never asserted as an achieved SLA. RPO recorded against the ~15 min tunable target, also measured, not assumed. - All bundle prerequisite gaps from the "Known gaps" section resolved and logged (Step 2). All true -> Stage 6 of `docs/dc-dc-deployment-workflow.md` moves to DONE; proceed to Stage 7 (Designate, COS, Magnum/CAPI, per DC). --- ## Delivery checklist (this repo's standard discipline) - [ ] `bash scripts/repo-lint.sh` clean (0 fail) before committing any repo changes made while executing this runbook (changelog entry, tracker doc updates). - [ ] Every bundle change from Step 2 has its own changelog entry with a stated revert (`juju remove-application`, `juju bind ... =storage` back), per this repo's delivery rule for built-surface changes. - [ ] Changelog entry for this runbook's first real execution (next DOCFIX/D number via `bash scripts/ledger-scan.sh`, do not guess next-free), recording the ACTUAL measured RTO/RPO values from Steps 6/7/9/10/11 -- these are exactly the as-built facts this repo commits, not secrets. - [ ] `docs/session-ledger.md` updated with the outcome. - [ ] `docs/dc-dc-deployment-workflow.md` Stage 6 row and tracker table updated (Step 12). - [x] **Extraction DONE 2026-07-09 (DOCFIX-165), not future work anymore.** The radosgw multisite / rbd-mirror command sequences above (Steps 4, 5, 8, 10.2, 11.2-11.4) are now real, callable, parameterized scripts, closing Tooling gap register item #5's MECHANISM half: `scripts/dc-dc-radosgw-multisite.sh` (`master-init`/`join-readonly`/ `enable-two-way` -- Steps 4 and 8's radosgw half), `scripts/dc-dc-rbd-mirror.sh` (`bootstrap-primary`/`bootstrap-secondary --direction rx-only|rx-tx` -- Steps 5 and 8's rbd-mirror half), and `scripts/dc-dc-dr-drill.sh` (`failover` -- Step 10.2's force-promote + a flagged, never-auto-executed Glance re-registration reminder; `failback` -- Steps 11.2-11.4, with the demote-CURRENT-PRIMARY-before-promote-RECOVERING ordering HARD-CODED as fixed call order, not left to documentation). Every name/endpoint/unit is a REQUIRED argument (no invented defaults); default mode is `--dry-run`, `--apply` executes one command at a time via `juju ssh`/`juju run`, aborting on first failure. Matching harnesses `tests/dc-dc-radosgw-multisite/`, `tests/dc-dc-rbd-mirror/`, `tests/dc-dc-dr-drill/` (19/19, 19/19, 23/23 PASS) validate ONLY these scripts' own argument parsing, guard clauses, and dry-run output (including the failback step ORDER) -- they do NOT and CANNOT validate real radosgw/rbd-mirror behavior; no live Ceph cluster exists this session to run any of this against (still true tonight, unchanged from when this item was first logged). When this runbook is actually executed, Steps 4/5/8/10.2/11.2-11.4 above call these scripts (starting in `--dry-run` to review the exact plan first) instead of typing the command sequences by hand. See `docs/changelog-20260709-ceph-replication- tooling.md`. - [ ] The per-image Glance re-registration mechanism (Step 10.2) and the Cinder backup/restore-with-metadata reconciliation (Step 10.3/11.3) remain the least-scripted, most manual parts of the drill even after the above extraction -- `dc-dc-dr-drill.sh failover` deliberately prints the re-registration step as a flagged MANUAL reminder rather than executing it (the runbook is explicit the exact invocation depends on unconfirmed Glance driver behavior), and `failback`'s Step 11.3 Cinder reconciliation is not scripted at all (tenant/volume-specific). Both remain the highest-value future automation targets if gap #5 is picked up again after a real drill run surfaces the concrete Glance driver invocation and a general Cinder reconciliation shape. ## Next Stage 7 -- Designate, COS, Magnum/CAPI (per DC).