Changelog 2026-07-09 -- Ceph replication tooling: radosgw multisite + rbd-mirror scripts (DOCFIX-165)

No live infrastructure touched -- new scripts + test harnesses only. Closes the MECHANISM half of tooling gap register item #5 in docs/dc-dc-deployment- workflow.md ("Ceph replication tooling (radosgw multisite + rbd-mirror). No script in the repo touches either ... D-108's entire mechanism ... is net-new. Blocks Stage 6."). The DATA half (no live Ceph cluster exists this session to run any of this against) stays open -- this is a prep-only session, same as every other tooling delivery tonight.

Primary source: runbooks/dc-dc-phase5-dr-failover-drill.md Steps 4, 5, 8, 10.2, and 11.2-11.4 already carried the exact radosgw-admin/rbd mirror command sequences (verbatim upstream Ceph administration, not invented) -- this delivery extracts them into parameterized, callable scripts instead of hand-typed runbook prose, per that runbook's own "Follow-up, NOT built now" delivery-checklist item. Style/posture template: scripts/opnsense-prep- image.sh / scripts/opnsense-build-config-iso.sh (real commands, no live system to test against tonight, same posture).

Items

1. DOCFIX-165 -- scripts/dc-dc-radosgw-multisite.sh

FILES: scripts/dc-dc-radosgw-multisite.sh, new tests/dc-dc-radosgw-multisite/run-tests.sh.

WHAT: subcommands master-init (Step 4 DC1 master realm/zonegroup/zone create), join-readonly (Step 4 DC2 read-only zone join), enable-two-way (Step 8's radosgw half -- zone modify --read-only=false). Every name (realm/ zonegroup/zone), endpoint URL, juju model/unit, and the charm's restart- action name is a REQUIRED argument -- no invented defaults, matching the runbook's own <UPPER_SNAKE_TBD> placeholder discipline. --restart-action is the sole optional flag (unknown action name is not guessed; omit it and the script reminds the operator to confirm+restart manually instead). Default mode is a dry-run PLAN print (secrets redacted); --apply executes one command at a time via juju ssh, aborting on the first failure (per the runbook's own "verify each command before proceeding -- do not chain unattended" caution).

2. DOCFIX-165 -- scripts/dc-dc-rbd-mirror.sh

FILES: scripts/dc-dc-rbd-mirror.sh, new tests/dc-dc-rbd-mirror/run- tests.sh.

WHAT: subcommands bootstrap-primary (Step 5 DC1: pool enable + peer bootstrap create, token written to a file ON THE REMOTE UNIT, never captured/printed into this session -- secret-adjacent, per the runbook's own out-of-band-transfer instruction) and bootstrap-secondary (Step 5 DC2 / Step 8 two-way: pool enable + peer bootstrap import, --direction REQUIRED with no default -- rx-only for one-way staging, rx-tx for two-way, a conscious choice every time, never assumed). --pool is required and its guard message explicitly echoes the runbook's "do not assume glance -- confirm against ceph osd pool ls" caution.

3. DOCFIX-165 -- scripts/dc-dc-dr-drill.sh

FILES: scripts/dc-dc-dr-drill.sh, new tests/dc-dc-dr-drill/run-tests.sh.

WHAT: subcommand failover (Step 10.2: rbd mirror pool promote --force, gated on an explicit --confirmed-down flag acknowledging the runbook's Step 10.1 independent-path split-brain check was already done elsewhere -- this script does not perform that check itself; the per-image Glance re-registration step is ALWAYS printed as a flagged MANUAL reminder and is NEVER executed by this script even under --apply, because the runbook is explicit the exact invocation depends on unconfirmed Glance driver behavior). Subcommand failback (Steps 11.2-11.4): demote the recovering side (11.2), a Step 11.3 reminder (Cinder backup/restore reconciliation -- tenant/volume-specific, intentionally NOT scripted here), then -- THE SAFETY-CRITICAL PART -- demote the CURRENT PRIMARY before promoting the RECOVERING side (11.4a then 11.4b), hard-coded as fixed call order in the script (not a flag an operator could reorder, not just documentation): in apply mode, 11.4b is only ever invoked if 11.4a's demote succeeded first. --skip-11-4 matches the runbook's own "optional, may stay on the current primary for a while" clause. --apply additionally requires a typed FAILBACK confirmation read from /dev/tty (matching scripts/phase-00- teardown-destroy.sh's typed-approval gate pattern; --no-prompt bypasses it for tested automation only).

Shared $DC gate design (all three scripts)

Each script sources scripts/lib-net.sh and calls lib_net_select_dc (DOCFIX-151 convention) for every DC token it touches. The gate is INFORMATIONAL in dry-run mode (a plan preview is not a mutation -- the whole point of --dry-run is to stay reviewable even for a DC whose network literals aren't assigned yet, which is exactly today's DC2 state) and HARD BLOCKING immediately before any --apply execution (refuses to mutate an unassigned DC, exit 3). Each script also prints, purely as a reminder, the replication-plane CIDR key from lib-net.sh's PLANE_CIDRS/PLANE_NAME (the "10.12.36.0/22" -> "replication" mapping, per D-101/D-108's correction of the DR-seed's original v4 carrier language) -- this value is NEVER substituted into a command; the real ULA peer/endpoint address is always a separately measured, required argument.

What is STANDARD vs REPO-SPECIFIC (stated in each script's own header)

STANDARD, trust these: every radosgw-admin/rbd mirror pool subcommand and flag used is copied verbatim from the runbook (itself copied from upstream Ceph multisite/rbd-mirror docs) -- realm/zonegroup/zone create, zone modify --read-only=false, period update --commit, pool enable image, peer bootstrap create/import --direction, pool promote --force, pool demote/promote. Nothing here is invented Ceph administration.

REPO-SPECIFIC, needs review before trusting: every name (realm/zonegroup/ zone/pool/site), every endpoint URL, every juju model/unit, and the charm restart-action name -- none of these exist yet (Stage 6's own "Known gaps" section: no cinder-backup charm, no ceph-rbd-mirror charm, rbd-mirror binding still on storage not replication). All required arguments, zero invented defaults.

VERIFIED

• New harnesses, offline only (no live juju/Ceph reachable or attempted):
  • tests/dc-dc-radosgw-multisite/run-tests.sh: 19/19 PASS.
  • tests/dc-dc-rbd-mirror/run-tests.sh: 19/19 PASS.
  • tests/dc-dc-dr-drill/run-tests.sh: 23/23 PASS -- including the single most important assertion in this delivery: the failback PLAN's printed line order is 11.2 < 11.4a(demote CURRENT PRIMARY) < 11.4b(promote RECOVERING), verified by grepping actual line numbers in the dry-run output, plus a check that 11.4a is a demote and 11.4b is a promote (not swapped).
  • Each harness explicitly tests ONLY this script's own logic (argument parsing, guard clauses, dry-run plan text/order, the $DC gate's dry-run-informational/apply-blocking split, the --apply/typed- confirmation gates) -- none validate real radosgw/rbd-mirror daemon behavior, stated in each harness's own header, matching tests/opnsense-prep-image/run-tests.sh's disclosure style.
  • bash scripts/repo-lint.sh: 0 fail, 1 documented legacy WARN (unchanged).
  • bash scripts/run-tests-all.sh: full gauntlet re-run after this delivery; no new failures introduced by these three scripts/harnesses.

REVERT: rm -f scripts/dc-dc-radosgw-multisite.sh scripts/dc-dc-rbd-mirror.sh scripts/dc-dc-dr-drill.sh && rm -rf tests/dc-dc-radosgw-multisite tests/dc-dc- rbd-mirror tests/dc-dc-dr-drill (safe -- nothing else in the repo calls any of these three scripts yet).

Next actionable step

Tooling gap register item #5's MECHANISM is CLOSED; its DATA half (a real DC1/DC2 Ceph cluster, real realm/zonegroup/zone/pool names, real endpoints, the three bundle.yaml gaps from Stage 6's own "Known gaps" section) is unchanged and still blocks Stage 6 for real. The Stage 6 runbook (runbooks/dc-dc-phase5-dr-failover-drill.md) Steps 4/5/8/10.2/11.2-11.4 and its delivery checklist now point at these scripts instead of describing them as future work -- the runbook's own command sequences remain the reference if a script's behavior is ever in question. docs/dc-dc-deployment- workflow.md tooling gap register item #5 updated to CLOSED (mechanism), mirroring the phrasing pattern used for items #1/#3/#9.