# DC-DC Phase 5 -- DR wiring and failover drill (Stage 6)

Wire the Tier-A cross-DC replication mechanism (`docs/dc-dc-replication-DR-seed.md`,
engineered by **D-108**) between DC1 and DC2, stage it one-way, prove it clean, enable
two-way, then run the actual failover and failback drills plus the per-DC Juju controller
backup/restore drill (**D-104**). This is the first DC-DC runbook that deliberately takes a
live DC down (simulated, not physical) as part of the drill itself -- treat Steps 9-11 with
the same seriousness as `phase-00-teardown-maas-reset.md`'s own destructive steps.

Entry condition: **Stage 5's gate** (`docs/dc-dc-deployment-workflow.md`) has passed for
BOTH DC1 and DC2 independently -- `preflight.sh` PASS before add-model, post-deploy
`cloud-assert.sh --capture` per DC, a controller backup already taken and stored, Ceph-over-v6
and geneve-over-v6 verified. Do not start this runbook against a DC that hasn't cleared that
gate; whichever sibling runbook carries Stage 5 is the authority for confirming it, not this
document.

Decisions this runbook executes against: **D-108** (cross-DC replication mechanism:
cinder-backup via `ceph-radosgw` MULTISITE zonegroup; Glance via `ceph-rbd-mirror`; Nova
ephemeral not replicated; carrier is the IPv6-only ULA replication plane per **D-101**,
correcting the DR-seed's original v4 carrier language; one-way staged before two-way; RPO
~15 min tunable; RTO is a measured drill output, not an SLA), and **D-104** (per-DC
single-unit Juju controllers, `create-backup`/`download-backup`, restore drilled HERE, not
just storage failover). Governing design: `docs/dc-dc-buildout-design.md` Section 4's Phase 5
entry and Section 8's failover/failback skeleton (reproduced faithfully below, at
command-level, not redesigned). Tracker row: `docs/dc-dc-deployment-workflow.md` Stage 6 and
its Tooling gap register item #5 (Ceph replication tooling -- confirmed NO script in this
repo touches radosgw multisite or rbd-mirror; genuinely net-new) and item #1 ($DC selector --
DOCFIX-151 -- this runbook operates against TWO DCs, so any `scripts/lib-net.sh` session
calls `lib_net_select_dc "$DC"` explicitly, once, immediately after sourcing).

---

!!! DESTRUCTIVE. Steps 9-11 deliberately take DC1 "down" (a controlled simulated outage --
    either a replication-plane partition or a hard shutdown of DC1's node VMs, see Step 9) to
    run a real failover drill, then bring it back in a split-brain-safe failback. There is no
    scripted rollback for a botched promote/demote sequence on live Ceph pools -- a wrong-order
    `rbd mirror pool promote --force` or radosgw zone demote on BOTH sides at once is exactly
    the split-brain this drill exists to avoid. Each promote/demote/restore action is DISCRETE
    and individually gated -- do not batch them, and do not run this against Roosevelt or any
    cloud you cannot afford to actually lose data on.

!!! GENUINELY NEW TOOLING. No script in this repo touches radosgw multisite or rbd-mirror
    (Tooling gap register item #5) -- the commands below are STANDARD, well-documented upstream
    Ceph administration commands (`radosgw-admin realm/zonegroup/zone create`, `rbd mirror pool
    enable/peer bootstrap/promote/demote`, etc.), not invented for this repo. What IS genuinely
    unknown and MUST NOT be inferred: the exact realm/zonegroup/zone names, pool names, radosgw
    endpoint URLs/ports, and the real replication-plane ULA addresses for DC1 and DC2 -- none of
    that exists yet. Every placeholder below is written as `<UPPER_SNAKE_TBD>` and MUST be
    replaced with a value measured against the real DC1/DC2 Ceph clusters at execution time
    (Stage 5's job, confirmed again here before this runbook mutates anything). If a placeholder
    is still a placeholder when you reach a MUTATION step, STOP -- that is an inferred value
    entering a command, the hardest rule this repo has.

---

## Known gaps / prerequisites -- confirmed missing, not fixed by this runbook

Grepped `bundle.yaml` before writing this (2026-07-09/10 sweep, same repo state as the
Tooling gap register audit):

- **No `cinder-backup` charm is deployed in either DC's bundle.** The DR seed's own text
  (`docs/dc-dc-replication-DR-seed.md` line 24) assumes cinder already exposes the
  `backup-backend` endpoint -- true for `cinder` itself, but the separate `cinder-backup`
  subordinate/charm that actually runs the backup driver is NOT in `bundle.yaml` today. Adding
  it is a bundle change (a built-surface edit, gated the same as any other bundle change) --
  Step 1 below confirms this live and, if still absent, this runbook's first real MUTATION is
  adding it, NOT skipping past it.
- **No `ceph-rbd-mirror` charm is deployed either.** Same situation: D-108's Glance mechanism
  needs the charm; it is not in `bundle.yaml`.
- **`ceph-mon`'s existing `rbd-mirror` endpoint is bound to `storage`, not `replication`**
  (`bundle.yaml` line 515: `rbd-mirror: storage`). D-108 requires the cross-DC rbd-mirror
  daemon traffic to ride the IPv6-only ULA **replication** plane (`10.12.36.0/22` family), not
  the intra-DC OSD storage plane. This binding predates D-108 (inherited from the VR0/DC0
  seed at commit 8813efc) and needs correcting as part of standing up rbd-mirror here -- flag
  it, correct it as its own gated bundle-relation-binding change, do not silently leave the
  cross-DC daemon traffic on the wrong plane.

**UPDATE 2026-07-10 (DOCFIX-167):** `bundle.yaml` ITSELF now carries `cinder-backup`,
`ceph-rbd-mirror`, and the corrected `rbd-mirror: replication` binding (see
`docs/changelog-20260709-designate-cinderbackup-rbdmirror.md`) -- so any DC deployed via
Stage 5's runbook AFTER this date gets all three from the initial `juju deploy`, and Step 1's
CHECK below should find them already correct (skip straight to Step 3). Step 2 remains here,
unmodified, for the case this runbook is run against a DC deployed from an OLDER bundle.yaml
snapshot (pre-2026-07-10) that predates this fix -- do not assume Step 2 is dead code just
because a fresh deploy no longer needs it. Still not live-tested: no Ceph cluster exists this
session (2026-07-09/10 is PREP-ONLY) to confirm either path actually works end-to-end.

---

## Sequence

```
 1.  Confirm Stage 5 gate + bundle prerequisites   (read-only; may surface bundle gaps above)
 2.  Add/fix cinder-backup + ceph-rbd-mirror wiring [MUTATION: bundle change, gated, per DC]
 3.  Confirm replication-plane (ULA) peering        (read-only; latency/bandwidth recorded)
 4.  Stage ONE-WAY: radosgw realm/zonegroup/zone     [MUTATION: Ceph multisite, gated]
 5.  Stage ONE-WAY: rbd-mirror peer bootstrap (rx-only) [MUTATION: Ceph rbd-mirror, gated]
 6.  One-way validation drill (write DC1 -> observe DC2)  (read-only observation of RPO)
 7.  One-way failover/failback micro-drill; RECORD RTO/RPO  [DESTRUCTIVE: small-scale promote]
     -> GATE 1: one-way clean, RTO/RPO measured
 8.  Enable TWO-WAY (bidirectional zone sync; rbd-mirror rx-tx both sides) [MUTATION, gated]
 9.  Controller backup + restore drill (D-104), per DC       [MUTATION: backup + restore test]
10.  FULL failover drill: DC1 "down" -> promote/restore/rebuild at DC2  [DESTRUCTIVE]
11.  FULL failback drill: split-brain-safe, controlled window            [DESTRUCTIVE]
12.  Record final RTO/RPO actuals; update tracker docs
     -> EXIT GATE -> Stage 7 (Designate, COS, Magnum/CAPI)
```

---

## Step 1 -- Confirm Stage 5 gate + bundle prerequisites (READ-ONLY)

**CHECK -- both DCs independently, from wherever this session runs**
```bash
source scripts/lib-net.sh
lib_net_select_dc "vr1-dc0"    # explicit, once -- gap #1 / DOCFIX-151 call-site wiring
# repeat the pair below for dc2 in its own shell/session context; do not assume dc2's
# values from dc1 -- lib_net_select_dc fails loud for dc2 until its literals are real
bash scripts/cloud-assert.sh
```
Expect PASS on both DCs. If either is not clean, STOP -- this runbook's entry condition is
unmet; go fix Stage 5 first, do not proceed with a shaky base under a DR drill.

**CHECK -- bundle prerequisites (per DC)**
```bash
juju status --format=json | jq -r '.applications | keys[]' | grep -E '^cinder-backup$|^ceph-rbd-mirror$'
juju status --format=json | jq -r '.applications["ceph-mon"].endpoint-bindings // {}'
```
Expect `cinder-backup` and `ceph-rbd-mirror` both present, and the `rbd-mirror` binding
already on the replication space. If either charm is absent, or the binding is still on
`storage`, this confirms the "Known gaps" section above live -- proceed to Step 2. If both are
already correct (e.g. a prior partial run of this runbook already fixed them), skip to Step 3.

**GATE:** Stage 5 clean on both DCs; bundle prerequisites either confirmed present+correct, or
confirmed absent (routed to Step 2).

---

## Step 2 -- Add/fix cinder-backup + ceph-rbd-mirror wiring [MUTATION: bundle change, gated, per DC]

> CAUTION: edits `bundle.yaml` (a built-surface change) and applies it via `juju deploy`/
> `juju add-relation`/`juju bind` against a LIVE cloud. This is the first mutation this
> runbook performs. Present the diff, confirm the charm channel matches the rest of the
> bundle's Caracal/squid pins, and get explicit go before applying -- do not batch this with
> Step 1's read-only checks.

Only the specific fixes identified in Step 1 -- do not use this step to make any other bundle
change (no adjacent improvements mid-step, per this repo's hard rule 1).

**MUTATION -- per DC, only for whichever gap Step 1 found live**
```bash
# (a) if cinder-backup is absent:
juju deploy cinder-backup --channel 2024.1/stable --to lxd:<UNIT_NUMBER_TBD>
juju add-relation cinder-backup:backup-backend cinder:backup-backend
juju add-relation cinder-backup:ceph cinder-ceph:ceph   # or the real endpoint name --
                                                          # confirm against `juju info cinder-backup`
                                                          # rather than assuming this shape
juju add-relation cinder-backup:amqp rabbitmq-server:amqp

# (b) if ceph-rbd-mirror is absent:
juju deploy ceph-rbd-mirror --channel squid/stable --to lxd:<UNIT_NUMBER_TBD>
juju add-relation ceph-rbd-mirror:ceph-rbd-mirror ceph-mon:rbd-mirror

# (c) if ceph-mon's rbd-mirror binding is on storage, not replication:
juju bind ceph-mon rbd-mirror=replication
```
`<UNIT_NUMBER_TBD>` is a real MAAS/LXD placement decision on this DC's machines -- resolve it
live (`juju status` machine list) the same way every other phase runbook in this repo resolves
placement, never a copied number from DC0's bundle.

**VERIFY**
```bash
juju status --format=json | jq -r '.applications | keys[]' | grep -E '^cinder-backup$|^ceph-rbd-mirror$'
juju status --format=json | jq -r '.applications["ceph-mon"].endpoint-bindings["rbd-mirror"]'
bash scripts/cloud-assert.sh
```
**GATE:** both charms active/idle, `rbd-mirror` binding shows `replication`, `cloud-assert`
still PASS after the bundle change. Log this as a changelog entry (this IS a built-surface
change) with a revert (`juju remove-application`/`juju bind ... =storage` back) before moving
on.

---

## Step 3 -- Confirm replication-plane (ULA) peering (READ-ONLY)

**CHECK -- per DC, the replication plane's real ULA addresses**
```bash
ip -6 -o addr show scope global | grep -i fd    # or whatever prefix NetBox assigned -- confirm
                                                  # against NetBox, do not assume fd00::/8 shape
```
Record the real ULA address each DC's ceph-mon/ceph-rbd-mirror/ceph-radosgw units hold on the
replication plane (`10.12.36.0/22` family, per D-101 -- v6 ULA side). These are the addresses
every command from Step 4 onward uses; nothing here is invented.

**CHECK -- inter-DC route + latency/bandwidth budget on the simulated WAN**
```bash
# from a DC1 replication-plane host to its DC2 peer's REAL measured ULA address:
ping6 -c 20 <DC2_REPLICATION_ULA_TBD>
```
Record latency/jitter/loss actuals. Per `docs/dc-dc-buildout-design.md` Section 6, this is
the `tc netem`-simulated inter-DC leg (`opentofu/modules/netem-link`, mesh_dc1_dc2) -- confirm
its parameters are ACTUALLY APPLIED (not just planned) before trusting any RTT number recorded
here; a netem link with no rule applied yet will look artificially fast.

**GATE:** both DCs' real replication-plane ULA addresses recorded; inter-DC reachability and
link-profile actuals recorded against the netem parameters in effect. If the link is down or
unmeasured, STOP -- Step 4 onward needs it.

---

## Step 4 -- Stage ONE-WAY: radosgw realm/zonegroup/zone (multisite) [MUTATION: gated]

Per D-108's own explicit staging requirement, prove ONE mechanism in ONE direction before
anything bidirectional. This step wires radosgw multisite for the cinder-backup target,
DC1 as master zone only.

**PREFER `scripts/dc-dc-radosgw-multisite.sh master-init ...`** (DOCFIX-165) over typing the
raw commands below by hand -- it runs the SAME sequence with a `--dry-run` preview by
default and `--apply` to execute one command at a time via `juju ssh`, aborting on first
failure (run `bash scripts/dc-dc-radosgw-multisite.sh --help` for its exact flags). The raw
commands below are kept as the reference for what the script actually does, and as a
fallback if the script is unavailable.

> CAUTION: `radosgw-admin ... --commit` mutates the realm/period on a LIVE radosgw. Run each
> command, verify its output, THEN proceed to the next -- do not chain the whole sequence
> unattended (the script does this for you in `--apply` mode).

**MUTATION -- on a DC1 ceph-radosgw unit**
```bash
radosgw-admin realm create --rgw-realm=<REALM_NAME_TBD> --default
radosgw-admin zonegroup create --rgw-zonegroup=<ZONEGROUP_NAME_TBD> \
  --endpoints=http://<DC1_RGW_ENDPOINT_TBD> --master --default
radosgw-admin zone create --rgw-zonegroup=<ZONEGROUP_NAME_TBD> --rgw-zone=<DC1_ZONE_NAME_TBD> \
  --endpoints=http://<DC1_RGW_ENDPOINT_TBD> --master --default
radosgw-admin period update --commit
```
Restart the DC1 radosgw unit (`juju run ceph-radosgw/0 restart` or the charm's own action --
confirm the current action name against `juju actions ceph-radosgw` rather than assuming) so
it picks up realm membership.

**VERIFY**
```bash
radosgw-admin realm list
radosgw-admin zonegroup get --rgw-zonegroup=<ZONEGROUP_NAME_TBD>
```
**GATE:** DC1 shows as the sole zone in the new zonegroup, master, default. DC2 is NOT yet
joined -- that is deliberate (one-way staging); joining DC2 happens only after Step 6/7's
one-way drill is clean (see "then enable two-way" -- Step 8, not here).

For the ONE-WAY drill itself, DC2's radosgw still needs to be a member zone of the SAME
zonegroup (multisite sync is inherently zone-to-zone; "one-way" here means DC2 is configured
`--read-only`/non-master, i.e. it can only receive, never author, matching D-108's "single
mechanism, single direction" staging intent), not a fully separate radosgw:
```bash
# on the DC2 ceph-radosgw unit, joining the SAME realm/zonegroup as a read-only zone:
radosgw-admin zone create --rgw-zonegroup=<ZONEGROUP_NAME_TBD> --rgw-zone=<DC2_ZONE_NAME_TBD> \
  --endpoints=http://<DC2_RGW_ENDPOINT_TBD> --read-only \
  --access-key=<DC1_SYSTEM_USER_ACCESS_KEY_TBD> --secret=<DC1_SYSTEM_USER_SECRET_TBD>
radosgw-admin period update --commit
```
The access-key/secret pair is the multisite system user radosgw itself generates at
`realm create` time on DC1 -- capture it there (`radosgw-admin user list` /
`radosgw-admin user info --uid=<...>`), never invent it. Restart DC2's radosgw unit.

**VERIFY**
```bash
radosgw-admin sync status --rgw-zone=<DC2_ZONE_NAME_TBD>
```
**GATE (Step 4 complete):** DC2 zone shows in the zonegroup, `--read-only`, sync status
reports it caught up (not "behind" indefinitely). This is the multisite half of one-way
staging; Step 5 does the same for rbd-mirror.

---

## Step 5 -- Stage ONE-WAY: rbd-mirror peer bootstrap (rx-only) [MUTATION: gated]

**PREFER `scripts/dc-dc-rbd-mirror.sh bootstrap-primary ...` / `bootstrap-secondary --direction
rx-only ...`** (DOCFIX-165) over typing the raw commands below by hand -- dry-run by default,
`--apply` to execute, and the bootstrap token file is written with an unpredictable path and
`chmod 600`'d immediately (`bash scripts/dc-dc-rbd-mirror.sh --help` for exact flags). The raw
commands below are the reference for what the script does, and the fallback if unavailable.

**CHECK -- confirm the Glance pool name** (do not assume `glance` -- confirm against the live
ceph-mon: `ceph osd pool ls` and cross-check which pool glance-charm's `rbd-pool` config
points at)
```bash
ceph osd pool ls
juju config glance rbd-pool
```

**MUTATION -- on DC1's ceph-rbd-mirror / ceph-mon unit**
```bash
rbd mirror pool enable <GLANCE_POOL_NAME_TBD> image
rbd mirror pool peer bootstrap create --site-name <DC1_SITE_NAME_TBD> <GLANCE_POOL_NAME_TBD> \
  > /tmp/rbd-mirror-bootstrap-token
```
Transfer the token to DC2 out of band (it is secret-adjacent -- treat it like any other
credential material this repo's guard hook flags: never printed into this session's context,
copied via a channel the operator controls directly).

**MUTATION -- on DC2's ceph-rbd-mirror / ceph-mon unit**
```bash
rbd mirror pool enable <GLANCE_POOL_NAME_TBD> image
rbd mirror pool peer bootstrap import --site-name <DC2_SITE_NAME_TBD> --direction rx-only \
  <GLANCE_POOL_NAME_TBD> /tmp/rbd-mirror-bootstrap-token
```
`--direction rx-only` is the one-way staging: DC2 can only RECEIVE mirrored snapshots, never
originate. Two-way (`rx-tx`) is Step 8, only after the one-way drill (Step 6/7) is clean.

**VERIFY**
```bash
rbd mirror pool info <GLANCE_POOL_NAME_TBD> --all
rbd mirror pool status <GLANCE_POOL_NAME_TBD> --verbose
```
**GATE (Step 5 complete):** DC2 shows as an `rx-only` peer of DC1 for the Glance pool; pool
status is `health OK`, no images stuck in an error/split-brain state.

---

## Step 6 -- One-way validation drill: write DC1, observe DC2 (READ-ONLY OBSERVATION)

**Cinder-backup path**
```bash
# on DC1 (a real tenant/volume this session has, not a fabricated id):
openstack volume backup create --name dr-drill-oneway-<TS> <REAL_VOLUME_ID>
```
Then, from DC2's radosgw:
```bash
radosgw-admin bucket list --rgw-zone=<DC2_ZONE_NAME_TBD>
radosgw-admin sync status --rgw-zone=<DC2_ZONE_NAME_TBD>
```
Record the wall-clock time from the `backup create` call to the object appearing synced at
DC2 -- this is one real RPO data point (target ~15 min per D-108, tunable, MEASURED here, not
assumed).

**Glance rbd-mirror path**
```bash
# on DC1: take/verify a snapshot on a real Glance-backed image; a normal glance image import
# already creates the pool object rbd-mirror snapshots against.
rbd mirror image status <GLANCE_POOL_NAME_TBD>/<REAL_IMAGE_RBD_ID>
```
Then on DC2:
```bash
rbd mirror image status <GLANCE_POOL_NAME_TBD>/<REAL_IMAGE_RBD_ID>
```
Record the time for `state` to reach `up+replaying` with a matching snapshot timestamp on
DC2. This is the Glance-side RPO data point.

**GATE:** at least one clean cinder-backup sync and one clean rbd-mirror sync observed at
DC2, both within a recorded, measured time -- not assumed to already work because the mirror
"looks enabled."

---

## Step 7 -- One-way failover/failback micro-drill; RECORD RTO/RPO [DESTRUCTIVE: small-scale]

Before the FULL drill (Step 10-11), run a small-scale version against the ONE-WAY setup, using
only the test volume/image from Step 6 -- this proves the mechanics (promote/restore/re-register)
work at all before staking a real DC-down scenario on them.

> CAUTION: `rbd mirror image promote` and `openstack volume backup restore` are real actions
> against the test artifacts from Step 6. Confirm you are promoting/restoring the DRILL
> objects, not a real tenant's data, before running.

**MUTATION**
```bash
# force-promote is used here deliberately (DC1 stays technically reachable in this micro-drill
# -- this exercises the promote MECHANISM, the split-brain-avoidance CHECK is exercised for
# real only in Step 10, where DC1 is genuinely unreachable):
rbd mirror image promote --force <GLANCE_POOL_NAME_TBD>/<REAL_IMAGE_RBD_ID>
openstack --os-cloud dc2 volume backup restore dr-drill-oneway-<TS> --volume dr-drill-oneway-restored-<TS>
```
**VERIFY / RECORD**
```bash
openstack --os-cloud dc2 volume show dr-drill-oneway-restored-<TS>
rbd mirror image status <GLANCE_POOL_NAME_TBD>/<REAL_IMAGE_RBD_ID>
```
Record: time from the "confirm DC1 down" decision point (here, decision-to-drill) to a
usable restored volume/promoted image at DC2 -- this is the RTO measured output. Do NOT
assert a target number; the buildout design and D-108 are explicit that RTO is the drill's
OUTPUT, not a pre-committed SLA.

**Demote back** (this micro-drill does not leave DC2 promoted afterward):
```bash
rbd mirror image demote <GLANCE_POOL_NAME_TBD>/<REAL_IMAGE_RBD_ID>
```

**GATE 1 (Stage 6 first exit sub-condition):** one-way drill clean (Step 6 RPO observed, Step
7 RTO measured and recorded), radosgw multisite and rbd-mirror both proven end-to-end at small
scale. Only after this gate passes does Step 8 enable two-way.

---

## Step 8 -- Enable TWO-WAY [MUTATION: gated]

**PREFER `scripts/dc-dc-radosgw-multisite.sh enable-two-way ...` and `scripts/dc-dc-rbd-mirror.sh
bootstrap-secondary --direction rx-tx ...`** (DOCFIX-165) over the raw commands below -- same
dry-run/`--apply` safety as Steps 4/5. The raw commands are the reference/fallback.

**MUTATION -- radosgw: make DC2's zone a full read-write member**
```bash
radosgw-admin zone modify --rgw-zone=<DC2_ZONE_NAME_TBD> --read-only=false
radosgw-admin period update --commit
```

**MUTATION -- rbd-mirror: re-import the peer with bidirectional direction**
```bash
# on DC1, generate a fresh bootstrap token (or reuse Step 5's peer relationship and just
# flip direction -- confirm against `rbd mirror pool peer` current subcommand behavior,
# it may require remove+re-add rather than an in-place direction change):
rbd mirror pool peer bootstrap create --site-name <DC1_SITE_NAME_TBD> <GLANCE_POOL_NAME_TBD> \
  > /tmp/rbd-mirror-bootstrap-token-twoway
# on DC2:
rbd mirror pool peer bootstrap import --site-name <DC2_SITE_NAME_TBD> --direction rx-tx \
  <GLANCE_POOL_NAME_TBD> /tmp/rbd-mirror-bootstrap-token-twoway
```

**VERIFY**
```bash
radosgw-admin sync status --rgw-zone=<DC1_ZONE_NAME_TBD>
radosgw-admin sync status --rgw-zone=<DC2_ZONE_NAME_TBD>
rbd mirror pool status <GLANCE_POOL_NAME_TBD> --verbose
```
**GATE:** both zones show each other caught up in `sync status`; rbd-mirror pool status shows
`rx-tx` in both directions, health OK. Re-run Step 6's write-and-observe check in the OPPOSITE
direction (write at DC2, observe at DC1) to prove two-way is real, not just configured.

---

## Step 9 -- Controller backup + restore drill (D-104), per DC [MUTATION: backup + restore test]

D-104 requires this drill exercised HERE, not deferred -- this is control-plane recovery, a
distinct surface from the storage failover above.

**MUTATION -- per DC, take a fresh controller backup**
```bash
bash scripts/cloud-snapshot.sh --with-controller-backup
```
This runs `juju create-backup -m admin/controller` and downloads the archive to
`~/openstack-baseline/` (jumphost-local; secret-adjacent; never committed -- confirmed by
this script's own header, DOCFIX-088). Record the archive path/size/sha256 from its manifest.

**CHECK -- confirm the real restore command/flags before using them**
```bash
juju restore-backup --help
```
Do not assume flags from memory or from a different juju version's documentation -- confirm
against this controller's actual installed juju client (3.6 per D-104) before running restore
for real. This repo's own discipline (never an inferred value) applies to CLI flags too, not
just IPs/IDs.

**MUTATION -- restore drill** (decide live, per the `--help` output and this DC's actual
topology, whether the drill targets a fresh scratch controller machine or an in-place restore
path; do not assume either shape without confirming what the installed juju version supports)
```bash
juju restore-backup --file <BACKUP_ARCHIVE_PATH_FROM_ABOVE>
```
**VERIFY**
```bash
juju status -m admin/controller
juju status -m openstack
```
**GATE:** the controller resumes management of this DC's `openstack` model post-restore;
`cloud-assert.sh` still PASS. Record wall-clock time from backup-taken to
restore-verified-working as this drill's own RTO data point (control-plane, distinct from the
storage RTO recorded in Step 7/12).

Repeat for the OTHER DC's controller independently -- D-104's whole point is per-DC
independence; a DC2 controller drill proves nothing about DC1's and vice versa.

---

## Step 10 -- FULL failover drill: DC1 "down" [DESTRUCTIVE]

> CAUTION: this step deliberately makes DC1 unreachable/non-authoritative, then promotes DC2
> and rebuilds real workload state there. Confirm every real tenant/workload this touches is
> either drill-only test material or something you are fully prepared to have live at DC2
> only until failback. Individually gated -- present each promote/restore/rebuild action,
> confirm, then proceed. Do not batch.

**Step 10.0 -- choose and record how "down" is simulated, then confirm it is truly down, not
a transient partition** (Section 8 skeleton item 1). Two shapes, pick deliberately and record
which:
- **(a) Replication-plane-only partition** (tests the split-brain-avoidance CHECK itself):
  sever only the DC1<->DC2 leg on the replication plane, e.g. via the `netem-link` mechanism
  (`opentofu/modules/netem-link` / a direct `tc qdisc ... netem loss 100%` on the mesh_dc1_dc2
  bridge -- confirm the real interface/bridge name against this session's own OpenTofu state,
  do not assume it). DC1 itself stays fully up and reachable from Office1/its own tenants.
  **Expected correct behavior: the split-brain check in Step 10.1 below should REFUSE to treat
  this as "DC1 down"** (it's still reachable via an out-of-band path) -- if this drill
  promotes DC2 anyway under a mere partition, that is a FINDING against the drill design, not
  a pass.
- **(b) Hard down** (the real failover scenario): shut down DC1's node VMs entirely
  (`virsh shutdown <DC1_DOMAIN_TBD>` for each, or the DC1 OpenTofu-managed equivalent) --
  confirm real domain names against this session's `virsh list --all`, never assumed from
  DC0's naming.

Run (a) FIRST as a negative-control drill (prove the check correctly refuses), THEN (b) as
the real drill this stage's gate is measuring.

**Step 10.1 -- confirm DC1 truly down, avoiding split-brain** (Section 8 skeleton item 1)
```bash
# from Office1 or any vantage point INDEPENDENT of the DC1<->DC2 replication leg:
ping <DC1_METAL_ADMIN_OR_PROVIDER_ADDRESS_TBD>
openstack --os-cloud dc1 token issue   # or any cheap DC1 API reachability probe
```
Require BOTH the replication-plane path AND at least one independent out-of-band path to
agree DC1 is unreachable before proceeding -- a DC2-only view (replication link down) is
exactly the ambiguous signal Section 8's skeleton warns about. Record the evidence.

**Step 10.2 -- Glance: promote at DC2** (Section 8 skeleton item 2)

**PREFER `scripts/dc-dc-dr-drill.sh failover ...`** (DOCFIX-165) over the raw command below --
same dry-run/`--apply` safety, plus it always prints the per-image Glance re-registration
reminder below as a flagged MANUAL step, never auto-executed.
```bash
rbd mirror pool promote --force <GLANCE_POOL_NAME_TBD>
```
`--force` because DC1 is (by 10.1's confirmation) genuinely unreachable to demote first --
this is the documented force-promote path, not an improvised shortcut.

Re-register images into DC2 Glance: for each image that needs to exist at DC2,
`openstack --os-cloud dc2 image create` pointing at the now-promoted RBD object (or the
equivalent re-register mechanism Glance's rbd driver exposes) -- confirm the exact invocation
against the deployed Glance charm version's driver behavior at execution time; this per-image
re-registration is exactly the kind of mechanism the Delivery checklist below flags as a good
`scripts/` extraction candidate once this has been run manually once.

**Step 10.3 -- Cinder: restore from replicated backups** (Section 8 skeleton item 3)
```bash
openstack --os-cloud dc2 volume backup restore <REAL_BACKUP_ID> --volume <NEW_VOLUME_NAME>
```
Repeat per volume that needs recovery. Record each restore's start/end time.

**Step 10.4 -- rebuild instances; re-create Neutron state** (Section 8 skeleton item 4)
```bash
openstack --os-cloud dc2 server create --image <RE-REGISTERED_IMAGE_ID> \
  --volume <RESTORED_VOLUME_ID> --flavor <REAL_FLAVOR_TBD> --network <DC2_NETWORK_TBD> <NAME>
openstack --os-cloud dc2 floating ip create <DC2_EXTERNAL_NET_TBD>
openstack --os-cloud dc2 security group create <REAL_SG_NAME_TBD>
```
Neutron state (ports/FIPs/security groups) is NOT replicated by design (D-108) -- this is
expected manual re-creation against DC2's own address space, not a bug to route around.

**Step 10.5 -- RECORD RTO/RPO actuals** (Section 8 skeleton item 5)
Tabulate: time-of-last-known-good-sync (RPO, from Step 6/Step 8's data) vs. time-DC1-confirmed-down
(10.1) vs. time-workloads-usable-at-DC2 (10.4 complete). This tuple is the drill's real output
-- write it into this runbook's execution log / changelog, not as an assumed number.

**GATE (Stage 6 second exit sub-condition):** DC2 is running the drill workload(s)
successfully; RTO/RPO recorded as measured, not assumed.

---

## Step 11 -- FULL failback drill: split-brain-safe, controlled window [DESTRUCTIVE]

Only after DC1 is confirmed genuinely recovered (reverse of Step 10.1's check -- DC1
reachable again via an independent path, not just "the partition healed").

> CAUTION: the ordering below is the split-brain-safety mechanism itself. Demote the CURRENT
> primary (DC2) before promoting the recovering side (DC1) -- promoting both, or promoting DC1
> before demoting DC2, creates two masters writing independently. Do not reorder these steps
> even if it seems faster.

**Step 11.1 -- do NOT let DC1 auto-resume as primary** (Section 8 skeleton item 1)
No command here by design -- this is a negative instruction. Confirm no automation (charm
hook, cron, systemd timer) on DC1 is configured to auto-promote on recovery; if one exists,
that is itself a finding to log, not silently accept.

**Step 11.2 -- demote DC1's Glance pool; resync from DC2 (now primary)** (item 2)

**PREFER `scripts/dc-dc-dr-drill.sh failback ...`** (DOCFIX-165) for Steps 11.2-11.4 together
over the raw commands below -- the demote-CURRENT-PRIMARY-before-promote-RECOVERING ordering
this whole step's CAUTION describes is HARD-CODED as fixed call order inside the script itself
(not left to documentation), and it defaults to dry-run with `--apply` to execute.
```bash
rbd mirror pool demote <GLANCE_POOL_NAME_TBD>   # on DC1, now that it's back
rbd mirror pool status <GLANCE_POOL_NAME_TBD> --verbose   # confirm DC1 catching up from DC2
```

**Step 11.3 -- reconcile Cinder: back up DC2-side changes made during the outage, restore into DC1** (item 3)
```bash
openstack --os-cloud dc2 volume backup create --name failback-reconcile-<TS> <DRIFTED_VOLUME_ID>
# once synced to DC1's radosgw zone (multisite sync, now bidirectional per Step 8):
openstack --os-cloud dc1 volume backup restore failback-reconcile-<TS> --volume <DC1_TARGET_VOLUME>
```

**Step 11.4 -- in a controlled window, optionally flip primary back** (item 4)
```bash
rbd mirror pool demote <GLANCE_POOL_NAME_TBD>    # DC2 (current primary) demotes FIRST
rbd mirror pool promote <GLANCE_POOL_NAME_TBD>   # DC1 promotes SECOND -- never simultaneous
```
This is "optional" per Section 8's own skeleton -- if the operator's plan is to leave DC2 as
primary for a while, stop after 11.3 and record that as the deliberate state, not an
incomplete drill.

**Step 11.5 -- verify both directions healthy; record the drill** (item 5)
```bash
rbd mirror pool status <GLANCE_POOL_NAME_TBD> --verbose
radosgw-admin sync status --rgw-zone=<DC1_ZONE_NAME_TBD>
radosgw-admin sync status --rgw-zone=<DC2_ZONE_NAME_TBD>
```
**GATE:** both pools/zones report healthy bidirectional sync; no image or bucket left in a
half-migrated state. Record the failback drill's own timing the same way Step 10.5 did.

---

## Step 12 -- Record final RTO/RPO actuals; update tracker docs

Consolidate every timing recorded in Steps 6, 7, 9, 10.5, and 11.5 into this runbook's
execution log entry (changelog + `docs/session-ledger.md`, per this repo's standard delivery
discipline). Record Stage 6's honest outcome in `docs/CURRENT-STATE.md` (GA-R1: the only status
authority; fully done only if every GATE above passed; partial otherwise, with the
specific blocking item named).

---

## GATE (Stage 6 exit condition, restated from `docs/dc-dc-buildout-design.md` Section 4 and
`docs/dc-dc-deployment-workflow.md` Stage 6)

- One-way drill clean: Step 6 RPO observed for both cinder-backup and rbd-mirror; Step 7 RTO
  measured and recorded (GATE 1).
- Two-way enabled and proven bidirectionally (Step 8).
- Controller backup/restore drill exercised per DC (Step 9, D-104) -- not deferred to a later
  stage.
- Full failover drill (Step 10) and full failback drill (Step 11) both run, split-brain-safe
  ordering observed throughout.
- RTO recorded as a MEASURED output of the drills above (Steps 7, 9, 10.5, 11.5) -- never
  asserted as an achieved SLA. RPO recorded against the ~15 min tunable target, also measured,
  not assumed.
- All bundle prerequisite gaps from the "Known gaps" section resolved and logged (Step 2).

All true -> Stage 6 is done: record it in `docs/CURRENT-STATE.md` (GA-R1); proceed to Stage 7
(Designate, COS, Magnum/CAPI, per DC).

---

## Delivery checklist (this repo's standard discipline)

- [ ] `bash scripts/repo-lint.sh` clean (0 fail) before committing any repo changes made while
      executing this runbook (changelog entry, tracker doc updates).
- [ ] Every bundle change from Step 2 has its own changelog entry with a stated revert
      (`juju remove-application`, `juju bind ... =storage` back), per this repo's delivery
      rule for built-surface changes.
- [ ] Changelog entry for this runbook's first real execution (next DOCFIX/D number via
      `bash scripts/ledger-scan.sh`, do not guess next-free), recording the ACTUAL measured
      RTO/RPO values from Steps 6/7/9/10/11 -- these are exactly the as-built facts this repo
      commits, not secrets.
- [ ] `docs/session-ledger.md` updated with the outcome.
- [ ] `docs/dc-dc-deployment-workflow.md` Stage 6 row and tracker table updated (Step 12).
- [x] **Extraction done 2026-07-09 (DOCFIX-165), not future work anymore -- and Steps 4, 5, 8,
      10.2, 11.2-11.4 above now each carry a "PREFER `scripts/dc-dc-...`" pointer at their own
      top (added 2026-07-10 during a full-project sweep, after the first version of this
      checklist item claimed this was already done when the step bodies themselves still only
      showed raw commands -- fixed so the claim and the actual text agree).** The radosgw
      multisite / rbd-mirror command sequences above (Steps 4, 5, 8, 10.2, 11.2-11.4) are now
      real, callable, parameterized scripts, closing Tooling gap register item #5's MECHANISM
      half: `scripts/dc-dc-radosgw-multisite.sh` (`master-init`/`join-readonly`/
      `enable-two-way` -- Steps 4 and 8's radosgw half), `scripts/dc-dc-rbd-mirror.sh`
      (`bootstrap-primary`/`bootstrap-secondary --direction rx-only|rx-tx` -- Steps 5 and 8's
      rbd-mirror half), and `scripts/dc-dc-dr-drill.sh` (`failover` -- Step 10.2's force-promote
      + a flagged, never-auto-executed Glance re-registration reminder; `failback` -- Steps
      11.2-11.4, with the demote-CURRENT-PRIMARY-before-promote-RECOVERING ordering HARD-CODED
      as fixed call order, not left to documentation). Every name/endpoint/unit is a REQUIRED
      argument (no invented defaults); default mode is `--dry-run`, `--apply` executes one
      command at a time via `juju ssh`/`juju run`, aborting on first failure. Matching harnesses
      `tests/dc-dc-radosgw-multisite/`, `tests/dc-dc-rbd-mirror/`, `tests/dc-dc-dr-drill/`
      (19/19, 19/19, 23/23 PASS) validate ONLY these scripts' own argument parsing, guard
      clauses, and dry-run output (including the failback step ORDER) -- they do NOT and
      CANNOT validate real radosgw/rbd-mirror behavior; no live Ceph cluster exists this session
      to run any of this against (still true tonight, unchanged from when this item was first
      logged). When this runbook is actually executed, Steps 4/5/8/10.2/11.2-11.4 above call
      these scripts (starting in `--dry-run` to review the exact plan first) instead of typing
      the command sequences by hand. See `docs/changelog-20260709-ceph-replication-
      tooling.md`.
- [ ] The per-image Glance re-registration mechanism (Step 10.2) and the Cinder
      backup/restore-with-metadata reconciliation (Step 10.3/11.3) remain the least-scripted,
      most manual parts of the drill even after the above extraction -- `dc-dc-dr-drill.sh
      failover` deliberately prints the re-registration step as a flagged MANUAL reminder
      rather than executing it (the runbook is explicit the exact invocation depends on
      unconfirmed Glance driver behavior), and `failback`'s Step 11.3 Cinder reconciliation is
      not scripted at all (tenant/volume-specific). Both remain the highest-value future
      automation targets if gap #5 is picked up again after a real drill run surfaces the
      concrete Glance driver invocation and a general Cinder reconciliation shape.

## Next
Stage 7 -- Designate, COS, Magnum/CAPI (per DC).
