# Changelog 2026-07-09 -- Ceph replication tooling: radosgw multisite + rbd-mirror scripts (DOCFIX-165)

No live infrastructure touched -- new scripts + test harnesses only. Closes the
MECHANISM half of tooling gap register item #5 in `docs/dc-dc-deployment-
workflow.md` ("Ceph replication tooling (radosgw multisite + rbd-mirror). No
script in the repo touches either ... D-108's entire mechanism ... is
net-new. Blocks Stage 6."). The DATA half (no live Ceph cluster exists this
session to run any of this against) stays open -- this is a prep-only
session, same as every other tooling delivery tonight.

Primary source: `runbooks/dc-dc-phase5-dr-failover-drill.md` Steps 4, 5, 8,
10.2, and 11.2-11.4 already carried the exact `radosgw-admin`/`rbd mirror`
command sequences (verbatim upstream Ceph administration, not invented) --
this delivery extracts them into parameterized, callable scripts instead of
hand-typed runbook prose, per that runbook's own "Follow-up, NOT built now"
delivery-checklist item. Style/posture template: `scripts/opnsense-prep-
image.sh` / `scripts/opnsense-build-config-iso.sh` (real commands, no live
system to test against tonight, same posture).

## Items

### 1. DOCFIX-165 -- `scripts/dc-dc-radosgw-multisite.sh`
FILES: `scripts/dc-dc-radosgw-multisite.sh`, new
`tests/dc-dc-radosgw-multisite/run-tests.sh`.

WHAT: subcommands `master-init` (Step 4 DC1 master realm/zonegroup/zone
create), `join-readonly` (Step 4 DC2 read-only zone join), `enable-two-way`
(Step 8's radosgw half -- zone modify --read-only=false). Every name (realm/
zonegroup/zone), endpoint URL, juju model/unit, and the charm's restart-
action name is a REQUIRED argument -- no invented defaults, matching the
runbook's own `<UPPER_SNAKE_TBD>` placeholder discipline. `--restart-action`
is the sole optional flag (unknown action name is not guessed; omit it and
the script reminds the operator to confirm+restart manually instead).
Default mode is a dry-run PLAN print (secrets redacted); `--apply` executes
one command at a time via `juju ssh`, aborting on the first failure (per the
runbook's own "verify each command before proceeding -- do not chain
unattended" caution).

### 2. DOCFIX-165 -- `scripts/dc-dc-rbd-mirror.sh`
FILES: `scripts/dc-dc-rbd-mirror.sh`, new `tests/dc-dc-rbd-mirror/run-
tests.sh`.

WHAT: subcommands `bootstrap-primary` (Step 5 DC1: pool enable + peer
bootstrap create, token written to a file ON THE REMOTE UNIT, never
captured/printed into this session -- secret-adjacent, per the runbook's own
out-of-band-transfer instruction) and `bootstrap-secondary` (Step 5 DC2 /
Step 8 two-way: pool enable + peer bootstrap import, `--direction` REQUIRED
with no default -- `rx-only` for one-way staging, `rx-tx` for two-way, a
conscious choice every time, never assumed). `--pool` is required and its
guard message explicitly echoes the runbook's "do not assume glance --
confirm against `ceph osd pool ls`" caution.

### 3. DOCFIX-165 -- `scripts/dc-dc-dr-drill.sh`
FILES: `scripts/dc-dc-dr-drill.sh`, new `tests/dc-dc-dr-drill/run-tests.sh`.

WHAT: subcommand `failover` (Step 10.2: `rbd mirror pool promote --force`,
gated on an explicit `--confirmed-down` flag acknowledging the runbook's
Step 10.1 independent-path split-brain check was already done elsewhere --
this script does not perform that check itself; the per-image Glance
re-registration step is ALWAYS printed as a flagged MANUAL reminder and is
NEVER executed by this script even under `--apply`, because the runbook is
explicit the exact invocation depends on unconfirmed Glance driver
behavior). Subcommand `failback` (Steps 11.2-11.4): demote the recovering
side (11.2), a Step 11.3 reminder (Cinder backup/restore reconciliation --
tenant/volume-specific, intentionally NOT scripted here), then -- THE
SAFETY-CRITICAL PART -- demote the CURRENT PRIMARY before promoting the
RECOVERING side (11.4a then 11.4b), hard-coded as fixed call order in the
script (not a flag an operator could reorder, not just documentation): in
apply mode, 11.4b is only ever invoked if 11.4a's demote succeeded first.
`--skip-11-4` matches the runbook's own "optional, may stay on the current
primary for a while" clause. `--apply` additionally requires a typed
`FAILBACK` confirmation read from `/dev/tty` (matching `scripts/phase-00-
teardown-destroy.sh`'s typed-approval gate pattern; `--no-prompt` bypasses it
for tested automation only).

### Shared `$DC` gate design (all three scripts)
Each script sources `scripts/lib-net.sh` and calls `lib_net_select_dc`
(DOCFIX-151 convention) for every DC token it touches. The gate is
INFORMATIONAL in dry-run mode (a plan preview is not a mutation -- the whole
point of `--dry-run` is to stay reviewable even for a DC whose network
literals aren't assigned yet, which is exactly today's DC2 state) and HARD
BLOCKING immediately before any `--apply` execution (refuses to mutate an
unassigned DC, exit 3). Each script also prints, purely as a reminder, the
replication-plane CIDR key from `lib-net.sh`'s `PLANE_CIDRS`/`PLANE_NAME`
(the "10.12.36.0/22" -> "replication" mapping, per D-101/D-108's correction
of the DR-seed's original v4 carrier language) -- this value is NEVER
substituted into a command; the real ULA peer/endpoint address is always a
separately measured, required argument.

## What is STANDARD vs REPO-SPECIFIC (stated in each script's own header)

STANDARD, trust these: every `radosgw-admin`/`rbd mirror pool` subcommand
and flag used is copied verbatim from the runbook (itself copied from
upstream Ceph multisite/rbd-mirror docs) -- realm/zonegroup/zone create,
zone modify --read-only=false, period update --commit, pool enable image,
peer bootstrap create/import --direction, pool promote --force, pool
demote/promote. Nothing here is invented Ceph administration.

REPO-SPECIFIC, needs review before trusting: every name (realm/zonegroup/
zone/pool/site), every endpoint URL, every juju model/unit, and the charm
restart-action name -- none of these exist yet (Stage 6's own "Known gaps"
section: no `cinder-backup` charm, no `ceph-rbd-mirror` charm, `rbd-mirror`
binding still on `storage` not `replication`). All required arguments, zero
invented defaults.

## VERIFIED

- New harnesses, offline only (no live juju/Ceph reachable or attempted):
  - `tests/dc-dc-radosgw-multisite/run-tests.sh`: 19/19 PASS.
  - `tests/dc-dc-rbd-mirror/run-tests.sh`: 19/19 PASS.
  - `tests/dc-dc-dr-drill/run-tests.sh`: 23/23 PASS -- including the
    single most important assertion in this delivery: the failback PLAN's
    printed line order is 11.2 < 11.4a(demote CURRENT PRIMARY) <
    11.4b(promote RECOVERING), verified by grepping actual line numbers in
    the dry-run output, plus a check that 11.4a is a `demote` and 11.4b is a
    `promote` (not swapped).
  - Each harness explicitly tests ONLY this script's own logic (argument
    parsing, guard clauses, dry-run plan text/order, the `$DC` gate's
    dry-run-informational/apply-blocking split, the `--apply`/typed-
    confirmation gates) -- none validate real radosgw/rbd-mirror daemon
    behavior, stated in each harness's own header, matching
    `tests/opnsense-prep-image/run-tests.sh`'s disclosure style.
  - `bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy WARN
    (unchanged).
  - `bash scripts/run-tests-all.sh`: full gauntlet re-run after this
    delivery; no new failures introduced by these three scripts/harnesses.

REVERT: `rm -f scripts/dc-dc-radosgw-multisite.sh scripts/dc-dc-rbd-mirror.sh
scripts/dc-dc-dr-drill.sh && rm -rf tests/dc-dc-radosgw-multisite tests/dc-dc-
rbd-mirror tests/dc-dc-dr-drill` (safe -- nothing else in the repo calls any
of these three scripts yet).

## Next actionable step

Tooling gap register item #5's MECHANISM is CLOSED; its DATA half (a real
DC1/DC2 Ceph cluster, real realm/zonegroup/zone/pool names, real endpoints,
the three `bundle.yaml` gaps from Stage 6's own "Known gaps" section) is
unchanged and still blocks Stage 6 for real. The Stage 6 runbook
(`runbooks/dc-dc-phase5-dr-failover-drill.md`) Steps 4/5/8/10.2/11.2-11.4 and
its delivery checklist now point at these scripts instead of describing them
as future work -- the runbook's own command sequences remain the reference
if a script's behavior is ever in question. `docs/dc-dc-deployment-
workflow.md` tooling gap register item #5 updated to CLOSED (mechanism),
mirroring the phrasing pattern used for items #1/#3/#9.
