# Changelog 2026-07-09 -- Ceph replication tooling: radosgw multisite + rbd-mirror scripts (DOCFIX-165) No live infrastructure touched -- new scripts + test harnesses only. Closes the MECHANISM half of tooling gap register item #5 in `docs/dc-dc-deployment- workflow.md` ("Ceph replication tooling (radosgw multisite + rbd-mirror). No script in the repo touches either ... D-108's entire mechanism ... is net-new. Blocks Stage 6."). The DATA half (no live Ceph cluster exists this session to run any of this against) stays open -- this is a prep-only session, same as every other tooling delivery tonight. Primary source: `runbooks/dc-dc-phase5-dr-failover-drill.md` Steps 4, 5, 8, 10.2, and 11.2-11.4 already carried the exact `radosgw-admin`/`rbd mirror` command sequences (verbatim upstream Ceph administration, not invented) -- this delivery extracts them into parameterized, callable scripts instead of hand-typed runbook prose, per that runbook's own "Follow-up, NOT built now" delivery-checklist item. Style/posture template: `scripts/opnsense-prep- image.sh` / `scripts/opnsense-build-config-iso.sh` (real commands, no live system to test against tonight, same posture). ## Items ### 1. DOCFIX-165 -- `scripts/dc-dc-radosgw-multisite.sh` FILES: `scripts/dc-dc-radosgw-multisite.sh`, new `tests/dc-dc-radosgw-multisite/run-tests.sh`. WHAT: subcommands `master-init` (Step 4 DC1 master realm/zonegroup/zone create), `join-readonly` (Step 4 DC2 read-only zone join), `enable-two-way` (Step 8's radosgw half -- zone modify --read-only=false). Every name (realm/ zonegroup/zone), endpoint URL, juju model/unit, and the charm's restart- action name is a REQUIRED argument -- no invented defaults, matching the runbook's own `` placeholder discipline. `--restart-action` is the sole optional flag (unknown action name is not guessed; omit it and the script reminds the operator to confirm+restart manually instead). Default mode is a dry-run PLAN print (secrets redacted); `--apply` executes one command at a time via `juju ssh`, aborting on the first failure (per the runbook's own "verify each command before proceeding -- do not chain unattended" caution). ### 2. DOCFIX-165 -- `scripts/dc-dc-rbd-mirror.sh` FILES: `scripts/dc-dc-rbd-mirror.sh`, new `tests/dc-dc-rbd-mirror/run- tests.sh`. WHAT: subcommands `bootstrap-primary` (Step 5 DC1: pool enable + peer bootstrap create, token written to a file ON THE REMOTE UNIT, never captured/printed into this session -- secret-adjacent, per the runbook's own out-of-band-transfer instruction) and `bootstrap-secondary` (Step 5 DC2 / Step 8 two-way: pool enable + peer bootstrap import, `--direction` REQUIRED with no default -- `rx-only` for one-way staging, `rx-tx` for two-way, a conscious choice every time, never assumed). `--pool` is required and its guard message explicitly echoes the runbook's "do not assume glance -- confirm against `ceph osd pool ls`" caution. ### 3. DOCFIX-165 -- `scripts/dc-dc-dr-drill.sh` FILES: `scripts/dc-dc-dr-drill.sh`, new `tests/dc-dc-dr-drill/run-tests.sh`. WHAT: subcommand `failover` (Step 10.2: `rbd mirror pool promote --force`, gated on an explicit `--confirmed-down` flag acknowledging the runbook's Step 10.1 independent-path split-brain check was already done elsewhere -- this script does not perform that check itself; the per-image Glance re-registration step is ALWAYS printed as a flagged MANUAL reminder and is NEVER executed by this script even under `--apply`, because the runbook is explicit the exact invocation depends on unconfirmed Glance driver behavior). Subcommand `failback` (Steps 11.2-11.4): demote the recovering side (11.2), a Step 11.3 reminder (Cinder backup/restore reconciliation -- tenant/volume-specific, intentionally NOT scripted here), then -- THE SAFETY-CRITICAL PART -- demote the CURRENT PRIMARY before promoting the RECOVERING side (11.4a then 11.4b), hard-coded as fixed call order in the script (not a flag an operator could reorder, not just documentation): in apply mode, 11.4b is only ever invoked if 11.4a's demote succeeded first. `--skip-11-4` matches the runbook's own "optional, may stay on the current primary for a while" clause. `--apply` additionally requires a typed `FAILBACK` confirmation read from `/dev/tty` (matching `scripts/phase-00- teardown-destroy.sh`'s typed-approval gate pattern; `--no-prompt` bypasses it for tested automation only). ### Shared `$DC` gate design (all three scripts) Each script sources `scripts/lib-net.sh` and calls `lib_net_select_dc` (DOCFIX-151 convention) for every DC token it touches. The gate is INFORMATIONAL in dry-run mode (a plan preview is not a mutation -- the whole point of `--dry-run` is to stay reviewable even for a DC whose network literals aren't assigned yet, which is exactly today's DC2 state) and HARD BLOCKING immediately before any `--apply` execution (refuses to mutate an unassigned DC, exit 3). Each script also prints, purely as a reminder, the replication-plane CIDR key from `lib-net.sh`'s `PLANE_CIDRS`/`PLANE_NAME` (the "10.12.36.0/22" -> "replication" mapping, per D-101/D-108's correction of the DR-seed's original v4 carrier language) -- this value is NEVER substituted into a command; the real ULA peer/endpoint address is always a separately measured, required argument. ## What is STANDARD vs REPO-SPECIFIC (stated in each script's own header) STANDARD, trust these: every `radosgw-admin`/`rbd mirror pool` subcommand and flag used is copied verbatim from the runbook (itself copied from upstream Ceph multisite/rbd-mirror docs) -- realm/zonegroup/zone create, zone modify --read-only=false, period update --commit, pool enable image, peer bootstrap create/import --direction, pool promote --force, pool demote/promote. Nothing here is invented Ceph administration. REPO-SPECIFIC, needs review before trusting: every name (realm/zonegroup/ zone/pool/site), every endpoint URL, every juju model/unit, and the charm restart-action name -- none of these exist yet (Stage 6's own "Known gaps" section: no `cinder-backup` charm, no `ceph-rbd-mirror` charm, `rbd-mirror` binding still on `storage` not `replication`). All required arguments, zero invented defaults. ## VERIFIED - New harnesses, offline only (no live juju/Ceph reachable or attempted): - `tests/dc-dc-radosgw-multisite/run-tests.sh`: 19/19 PASS. - `tests/dc-dc-rbd-mirror/run-tests.sh`: 19/19 PASS. - `tests/dc-dc-dr-drill/run-tests.sh`: 23/23 PASS -- including the single most important assertion in this delivery: the failback PLAN's printed line order is 11.2 < 11.4a(demote CURRENT PRIMARY) < 11.4b(promote RECOVERING), verified by grepping actual line numbers in the dry-run output, plus a check that 11.4a is a `demote` and 11.4b is a `promote` (not swapped). - Each harness explicitly tests ONLY this script's own logic (argument parsing, guard clauses, dry-run plan text/order, the `$DC` gate's dry-run-informational/apply-blocking split, the `--apply`/typed- confirmation gates) -- none validate real radosgw/rbd-mirror daemon behavior, stated in each harness's own header, matching `tests/opnsense-prep-image/run-tests.sh`'s disclosure style. - `bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy WARN (unchanged). - `bash scripts/run-tests-all.sh`: full gauntlet re-run after this delivery; no new failures introduced by these three scripts/harnesses. REVERT: `rm -f scripts/dc-dc-radosgw-multisite.sh scripts/dc-dc-rbd-mirror.sh scripts/dc-dc-dr-drill.sh && rm -rf tests/dc-dc-radosgw-multisite tests/dc-dc- rbd-mirror tests/dc-dc-dr-drill` (safe -- nothing else in the repo calls any of these three scripts yet). ## Next actionable step Tooling gap register item #5's MECHANISM is CLOSED; its DATA half (a real DC1/DC2 Ceph cluster, real realm/zonegroup/zone/pool names, real endpoints, the three `bundle.yaml` gaps from Stage 6's own "Known gaps" section) is unchanged and still blocks Stage 6 for real. The Stage 6 runbook (`runbooks/dc-dc-phase5-dr-failover-drill.md`) Steps 4/5/8/10.2/11.2-11.4 and its delivery checklist now point at these scripts instead of describing them as future work -- the runbook's own command sequences remain the reference if a script's behavior is ever in question. `docs/dc-dc-deployment- workflow.md` tooling gap register item #5 updated to CLOSED (mechanism), mirroring the phrasing pattern used for items #1/#3/#9.