diff --git a/docs/changelog-20260709-ceph-replication-tooling.md b/docs/changelog-20260709-ceph-replication-tooling.md new file mode 100644 index 0000000..4725ac1 --- /dev/null +++ b/docs/changelog-20260709-ceph-replication-tooling.md @@ -0,0 +1,144 @@ +# Changelog 2026-07-09 -- Ceph replication tooling: radosgw multisite + rbd-mirror scripts (DOCFIX-165) + +No live infrastructure touched -- new scripts + test harnesses only. Closes the +MECHANISM half of tooling gap register item #5 in `docs/dc-dc-deployment- +workflow.md` ("Ceph replication tooling (radosgw multisite + rbd-mirror). No +script in the repo touches either ... D-108's entire mechanism ... is +net-new. Blocks Stage 6."). The DATA half (no live Ceph cluster exists this +session to run any of this against) stays open -- this is a prep-only +session, same as every other tooling delivery tonight. + +Primary source: `runbooks/dc-dc-phase5-dr-failover-drill.md` Steps 4, 5, 8, +10.2, and 11.2-11.4 already carried the exact `radosgw-admin`/`rbd mirror` +command sequences (verbatim upstream Ceph administration, not invented) -- +this delivery extracts them into parameterized, callable scripts instead of +hand-typed runbook prose, per that runbook's own "Follow-up, NOT built now" +delivery-checklist item. Style/posture template: `scripts/opnsense-prep- +image.sh` / `scripts/opnsense-build-config-iso.sh` (real commands, no live +system to test against tonight, same posture). + +## Items + +### 1. DOCFIX-165 -- `scripts/dc-dc-radosgw-multisite.sh` +FILES: `scripts/dc-dc-radosgw-multisite.sh`, new +`tests/dc-dc-radosgw-multisite/run-tests.sh`. + +WHAT: subcommands `master-init` (Step 4 DC1 master realm/zonegroup/zone +create), `join-readonly` (Step 4 DC2 read-only zone join), `enable-two-way` +(Step 8's radosgw half -- zone modify --read-only=false). Every name (realm/ +zonegroup/zone), endpoint URL, juju model/unit, and the charm's restart- +action name is a REQUIRED argument -- no invented defaults, matching the +runbook's own `` placeholder discipline. `--restart-action` +is the sole optional flag (unknown action name is not guessed; omit it and +the script reminds the operator to confirm+restart manually instead). +Default mode is a dry-run PLAN print (secrets redacted); `--apply` executes +one command at a time via `juju ssh`, aborting on the first failure (per the +runbook's own "verify each command before proceeding -- do not chain +unattended" caution). + +### 2. DOCFIX-165 -- `scripts/dc-dc-rbd-mirror.sh` +FILES: `scripts/dc-dc-rbd-mirror.sh`, new `tests/dc-dc-rbd-mirror/run- +tests.sh`. + +WHAT: subcommands `bootstrap-primary` (Step 5 DC1: pool enable + peer +bootstrap create, token written to a file ON THE REMOTE UNIT, never +captured/printed into this session -- secret-adjacent, per the runbook's own +out-of-band-transfer instruction) and `bootstrap-secondary` (Step 5 DC2 / +Step 8 two-way: pool enable + peer bootstrap import, `--direction` REQUIRED +with no default -- `rx-only` for one-way staging, `rx-tx` for two-way, a +conscious choice every time, never assumed). `--pool` is required and its +guard message explicitly echoes the runbook's "do not assume glance -- +confirm against `ceph osd pool ls`" caution. + +### 3. DOCFIX-165 -- `scripts/dc-dc-dr-drill.sh` +FILES: `scripts/dc-dc-dr-drill.sh`, new `tests/dc-dc-dr-drill/run-tests.sh`. + +WHAT: subcommand `failover` (Step 10.2: `rbd mirror pool promote --force`, +gated on an explicit `--confirmed-down` flag acknowledging the runbook's +Step 10.1 independent-path split-brain check was already done elsewhere -- +this script does not perform that check itself; the per-image Glance +re-registration step is ALWAYS printed as a flagged MANUAL reminder and is +NEVER executed by this script even under `--apply`, because the runbook is +explicit the exact invocation depends on unconfirmed Glance driver +behavior). Subcommand `failback` (Steps 11.2-11.4): demote the recovering +side (11.2), a Step 11.3 reminder (Cinder backup/restore reconciliation -- +tenant/volume-specific, intentionally NOT scripted here), then -- THE +SAFETY-CRITICAL PART -- demote the CURRENT PRIMARY before promoting the +RECOVERING side (11.4a then 11.4b), hard-coded as fixed call order in the +script (not a flag an operator could reorder, not just documentation): in +apply mode, 11.4b is only ever invoked if 11.4a's demote succeeded first. +`--skip-11-4` matches the runbook's own "optional, may stay on the current +primary for a while" clause. `--apply` additionally requires a typed +`FAILBACK` confirmation read from `/dev/tty` (matching `scripts/phase-00- +teardown-destroy.sh`'s typed-approval gate pattern; `--no-prompt` bypasses it +for tested automation only). + +### Shared `$DC` gate design (all three scripts) +Each script sources `scripts/lib-net.sh` and calls `lib_net_select_dc` +(DOCFIX-151 convention) for every DC token it touches. The gate is +INFORMATIONAL in dry-run mode (a plan preview is not a mutation -- the whole +point of `--dry-run` is to stay reviewable even for a DC whose network +literals aren't assigned yet, which is exactly today's DC2 state) and HARD +BLOCKING immediately before any `--apply` execution (refuses to mutate an +unassigned DC, exit 3). Each script also prints, purely as a reminder, the +replication-plane CIDR key from `lib-net.sh`'s `PLANE_CIDRS`/`PLANE_NAME` +(the "10.12.36.0/22" -> "replication" mapping, per D-101/D-108's correction +of the DR-seed's original v4 carrier language) -- this value is NEVER +substituted into a command; the real ULA peer/endpoint address is always a +separately measured, required argument. + +## What is STANDARD vs REPO-SPECIFIC (stated in each script's own header) + +STANDARD, trust these: every `radosgw-admin`/`rbd mirror pool` subcommand +and flag used is copied verbatim from the runbook (itself copied from +upstream Ceph multisite/rbd-mirror docs) -- realm/zonegroup/zone create, +zone modify --read-only=false, period update --commit, pool enable image, +peer bootstrap create/import --direction, pool promote --force, pool +demote/promote. Nothing here is invented Ceph administration. + +REPO-SPECIFIC, needs review before trusting: every name (realm/zonegroup/ +zone/pool/site), every endpoint URL, every juju model/unit, and the charm +restart-action name -- none of these exist yet (Stage 6's own "Known gaps" +section: no `cinder-backup` charm, no `ceph-rbd-mirror` charm, `rbd-mirror` +binding still on `storage` not `replication`). All required arguments, zero +invented defaults. + +## VERIFIED + +- New harnesses, offline only (no live juju/Ceph reachable or attempted): + - `tests/dc-dc-radosgw-multisite/run-tests.sh`: 19/19 PASS. + - `tests/dc-dc-rbd-mirror/run-tests.sh`: 19/19 PASS. + - `tests/dc-dc-dr-drill/run-tests.sh`: 23/23 PASS -- including the + single most important assertion in this delivery: the failback PLAN's + printed line order is 11.2 < 11.4a(demote CURRENT PRIMARY) < + 11.4b(promote RECOVERING), verified by grepping actual line numbers in + the dry-run output, plus a check that 11.4a is a `demote` and 11.4b is a + `promote` (not swapped). + - Each harness explicitly tests ONLY this script's own logic (argument + parsing, guard clauses, dry-run plan text/order, the `$DC` gate's + dry-run-informational/apply-blocking split, the `--apply`/typed- + confirmation gates) -- none validate real radosgw/rbd-mirror daemon + behavior, stated in each harness's own header, matching + `tests/opnsense-prep-image/run-tests.sh`'s disclosure style. + - `bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy WARN + (unchanged). + - `bash scripts/run-tests-all.sh`: full gauntlet re-run after this + delivery; no new failures introduced by these three scripts/harnesses. + +REVERT: `rm -f scripts/dc-dc-radosgw-multisite.sh scripts/dc-dc-rbd-mirror.sh +scripts/dc-dc-dr-drill.sh && rm -rf tests/dc-dc-radosgw-multisite tests/dc-dc- +rbd-mirror tests/dc-dc-dr-drill` (safe -- nothing else in the repo calls any +of these three scripts yet). + +## Next actionable step + +Tooling gap register item #5's MECHANISM is CLOSED; its DATA half (a real +DC1/DC2 Ceph cluster, real realm/zonegroup/zone/pool names, real endpoints, +the three `bundle.yaml` gaps from Stage 6's own "Known gaps" section) is +unchanged and still blocks Stage 6 for real. The Stage 6 runbook +(`runbooks/dc-dc-phase5-dr-failover-drill.md`) Steps 4/5/8/10.2/11.2-11.4 and +its delivery checklist now point at these scripts instead of describing them +as future work -- the runbook's own command sequences remain the reference +if a script's behavior is ever in question. `docs/dc-dc-deployment- +workflow.md` tooling gap register item #5 updated to CLOSED (mechanism), +mirroring the phrasing pattern used for items #1/#3/#9. diff --git a/scripts/dc-dc-dr-drill.sh b/scripts/dc-dc-dr-drill.sh new file mode 100644 index 0000000..e3dc4fd --- /dev/null +++ b/scripts/dc-dc-dr-drill.sh @@ -0,0 +1,291 @@ +#!/usr/bin/env bash +# scripts/dc-dc-dr-drill.sh -- failover promote and failback demote/promote +# sequences for D-108's Glance rbd-mirror mechanism (tooling gap register +# item #5). Extracts runbooks/dc-dc-phase5-dr-failover-drill.md Step 10.2 +# (FULL failover: force-promote at the surviving DC + a flagged, NOT +# auto-executed re-registration reminder) and Steps 11.2-11.4 (FULL failback: +# demote the recovering side, then -- ONLY after that -- demote the current +# primary before promoting the recovering side back). +# +# STANDARD, well-documented upstream Ceph administration (trust these): every +# `rbd mirror pool promote --force` / `rbd mirror pool demote` / +# `rbd mirror pool promote` call below is copied verbatim from the runbook, +# itself copied from upstream rbd-mirror docs -- nothing here is invented. +# +# GENUINELY REPO-SPECIFIC (needs review before trusting; none of it exists +# yet): the Glance POOL name, which DC is "down"/"recovering" vs "primary" at +# drill time (a live decision, not a fixed default), and the juju model/unit +# for each side. All REQUIRED arguments -- no invented defaults. +# +# THE SAFETY-CRITICAL PROPERTY OF THIS SCRIPT: failback's ordering -- +# "demote the CURRENT PRIMARY before promoting the RECOVERING side" -- is the +# single most safety-critical property in the whole drill (runbook Step 11 +# caution: promoting both, or promoting before demoting, creates two masters +# writing independently -- split-brain). This is HARD-CODED as call order in +# main() below (step_11_2, then step_11_4a demote-primary, then step_11_4b +# promote-recovering), not left to documentation or a flag an operator could +# reorder; step_11_4b is only ever invoked if step_11_4a's demote succeeded +# (apply mode) or was printed first (dry-run mode -- verifiable by grepping +# line order in the PLAN output). Step 11.3 (Cinder backup/restore +# reconciliation) is intentionally NOT implemented here -- it is a +# tenant/volume-specific manual reconciliation, out of scope for an +# rbd-mirror/radosgw mechanism script; a reminder is printed instead. +# +# Step 10.2's Glance re-registration (per-image `openstack image create` +# against the promoted RBD objects) is NEVER executed by this script, even +# under --apply -- the runbook is explicit that the exact invocation depends +# on the deployed Glance charm driver's behavior at execution time, which is +# unconfirmed. This script only prints a CLEARLY FLAGGED manual reminder +# listing the image identifiers supplied via --reregister-image. +# +# NO LIVE CEPH CLUSTER EXISTS THIS SESSION (2026-07-10, prep-only) -- never +# run for real, cannot be tonight. Default mode is a DRY-RUN plan print; +# --apply requires an additional typed confirmation (matching this repo's +# phase-00-teardown-destroy.sh --apply pattern) and executes one command at +# a time, aborting on the first failure. Its matching harness +# (tests/dc-dc-dr-drill/run-tests.sh) validates ONLY this script's own logic +# (argument parsing, guard clauses, dry-run output, the demote-before-promote +# line ORDER, and that --apply is blocked without confirmation) -- it does +# NOT and CANNOT validate real rbd-mirror promote/demote behavior. +# +# $DC gate: sources scripts/lib-net.sh and calls `lib_net_select_dc` once for +# EACH DC token this invocation touches (DOCFIX-151 convention) -- this +# deliberately makes any use of dc2 FAIL LOUD today (D-101/gap #3: DC2 has no +# assigned network literals, so no real DC2 Ceph cluster exists to target). +# +# Usage: +# dc-dc-dr-drill.sh failover --dc --model M --unit U --pool NAME \ +# --confirmed-down [--reregister-image ID ...] [--apply] +# dc-dc-dr-drill.sh failback --pool NAME \ +# --recovering-dc --recovering-model M --recovering-unit U \ +# --primary-dc --primary-model M --primary-unit U \ +# [--skip-11-4] [--apply] [--no-prompt] +# dc-dc-dr-drill.sh --help +# +# Exit: 0 ok | 1 bad/missing args | 2 juju not on PATH (apply only) | +# 3 $DC gate refused | 4 apply confirmation refused | +# 5 apply-mode command failed. ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +usage() { + cat <<'EOF' +usage: + dc-dc-dr-drill.sh failover --dc dc1|dc2 --model M --unit U --pool NAME \ + --confirmed-down [--reregister-image ID ...] [--apply] + (--dc/--model/--unit identify the DC being PROMOTED -- the surviving + side. --confirmed-down is a required acknowledgement that runbook Step + 10.1's independent-path-plus-replication-plane split-brain check has + already been done; this script does not perform that check itself.) + + dc-dc-dr-drill.sh failback --pool NAME \ + --recovering-dc dc1|dc2 --recovering-model M --recovering-unit U \ + --primary-dc dc1|dc2 --primary-model M --primary-unit U \ + [--skip-11-4] [--apply] [--no-prompt] + (recovering = the DC coming back up; primary = the DC currently + serving as primary since the failover. Order is hard-coded: demote + recovering (11.2) -> [11.3 reminder] -> demote primary (11.4a) -> + promote recovering (11.4b), each gated on the previous apply-mode step + succeeding. --skip-11-4 stops after 11.2/11.3, matching the runbook's + own "optional, may stay on DC2 for a while" clause.) + + dc-dc-dr-drill.sh --help + +Default mode is dry-run. --apply additionally requires a typed confirmation +read from /dev/tty (bypassed only by --no-prompt, for tested automation). +EOF +} + +SUBCMD="${1:-}" +case "$SUBCMD" in + -h|--help) usage; exit 0 ;; + failover|failback) shift ;; + "") echo "FAIL: a subcommand is required" >&2; usage >&2; exit 1 ;; + *) echo "FAIL: unknown subcommand '$SUBCMD'" >&2; usage >&2; exit 1 ;; +esac + +# shellcheck source=scripts/lib-net.sh +. "$HERE/lib-net.sh" + +hdr() { echo; echo "=== $* ==="; } +note() { echo " $*"; } + +# dc_gate_check -- computes gate status, does NOT exit. Prints +# nothing; sets DC_GATE_RC/DC_GATE_MSG for the caller to report and, later, +# to hard-block --apply on. A dry-run plan preview is not a mutation, so the +# gate is informational there; it is enforced (fail loud, exit 3) only right +# before any real execution below. +dc_gate_check() { + DC_GATE_MSG="$(lib_net_select_dc "$1" 2>&1 1>/dev/null)"; DC_GATE_RC=$? +} +dc_gate_line() { # dc_gate_line -- formats the last dc_gate_check result + if [ "$DC_GATE_RC" -eq 0 ]; then echo "\$DC gate: OK ($1)"; else echo "\$DC gate: FAILED for '$1' -- $DC_GATE_MSG"; fi +} +dc_gate_enforce() { # dc_gate_enforce -- HARD block before --apply execution + dc_gate_check "$1" + [ "$DC_GATE_RC" -eq 0 ] || { echo "FAIL: \$DC gate refused for '$1' -- STOP, do not mutate an unassigned DC ($DC_GATE_MSG)" >&2; exit 3; } +} + +# ===================================================================== +# failover: Step 10.2 -- force-promote at the surviving DC + a flagged, +# NEVER auto-executed re-registration reminder. +# ===================================================================== +run_failover() { + local DC="" MODEL="${MODEL:-openstack}" UNIT="" POOL="" APPLY=0 CONFIRMED_DOWN=0 + local -a REREG=() + while [ $# -gt 0 ]; do + case "$1" in + -h|--help) usage; exit 0 ;; + --dc) DC="${2:-}"; shift 2 ;; + --model) MODEL="${2:-}"; shift 2 ;; + --unit) UNIT="${2:-}"; shift 2 ;; + --pool) POOL="${2:-}"; shift 2 ;; + --confirmed-down) CONFIRMED_DOWN=1; shift ;; + --reregister-image) REREG+=("${2:-}"); shift 2 ;; + --apply) APPLY=1; shift ;; + *) echo "FAIL: unknown argument '$1'" >&2; usage >&2; exit 1 ;; + esac + done + + [ -n "$DC" ] || { echo "FAIL: --dc is required (dc1|dc2 -- the DC being promoted)" >&2; exit 1; } + case "$DC" in dc1|dc2) ;; *) echo "FAIL: --dc must be dc1 or dc2 (got '$DC')" >&2; exit 1 ;; esac + [ -n "$UNIT" ] || { echo "FAIL: --unit is required" >&2; exit 1; } + [ -n "$POOL" ] || { echo "FAIL: --pool is required (Glance pool name -- confirm, never assume 'glance')" >&2; exit 1; } + + dc_gate_check "$DC" + + hdr "dc-dc-dr-drill failover dc=$DC mode=$([ "$APPLY" -eq 1 ] && echo apply || echo dry-run)" + note "juju model=$MODEL unit=$UNIT pool=$POOL" + note "$(dc_gate_line "$DC")" + + hdr "PLAN" + note "STEP 1 (Step 10.2 force-promote): juju ssh -m $MODEL $UNIT -- \"rbd mirror pool promote --force $POOL\"" + if [ "${#REREG[@]}" -gt 0 ]; then + note "STEP 2 (Step 10.2 re-registration -- MANUAL, NEVER EXECUTED BY THIS SCRIPT):" + for img in "${REREG[@]}"; do + note " CONFIRM against the deployed Glance charm driver, then run by hand: openstack --os-cloud image create --name ... (re-register $img)" + done + else + note "STEP 2 (Step 10.2 re-registration): no --reregister-image given -- remember every image that needs re-registering into this DC's Glance is a separate, MANUAL, driver-specific step this script does not perform" + fi + + if [ "$CONFIRMED_DOWN" -ne 1 ]; then + echo + echo "FAIL: --confirmed-down is required before this script will --apply -- Step 10.1's independent-path split-brain check must be done FIRST (out of scope for this script); dry-run plan above is printed regardless" >&2 + [ "$APPLY" -eq 1 ] && exit 1 + fi + + if [ "$APPLY" -ne 1 ]; then + echo + echo "re-run with --confirmed-down --apply to execute." + echo "OK (dry-run)" + exit 0 + fi + + dc_gate_enforce "$DC" + command -v juju >/dev/null 2>&1 || { echo "FAIL: juju required on PATH for --apply" >&2; exit 2; } + + hdr "EXECUTE (apply mode)" + echo " DO STEP 1: juju ssh -m $MODEL $UNIT -- \"rbd mirror pool promote --force $POOL\"" + juju ssh -m "$MODEL" "$UNIT" -- "rbd mirror pool promote --force $POOL" &2; exit 5; } + echo " STEP 2 (re-registration) NOT executed -- MANUAL, per the header comment; do it now per the plan above" + echo "OK (apply) -- verify with: rbd mirror image status (per runbook Step 10.2/10.5 VERIFY)" +} + +# ===================================================================== +# failback: Steps 11.2-11.4 -- demote recovering, [11.3 reminder], then +# demote CURRENT PRIMARY before promote RECOVERING (hard-coded order). +# ===================================================================== +run_failback() { + local POOL="" + local REC_DC="" REC_MODEL="${MODEL:-openstack}" REC_UNIT="" + local PRI_DC="" PRI_MODEL="${MODEL:-openstack}" PRI_UNIT="" + local APPLY=0 SKIP114=0 PROMPT=1 + + while [ $# -gt 0 ]; do + case "$1" in + -h|--help) usage; exit 0 ;; + --pool) POOL="${2:-}"; shift 2 ;; + --recovering-dc) REC_DC="${2:-}"; shift 2 ;; + --recovering-model) REC_MODEL="${2:-}"; shift 2 ;; + --recovering-unit) REC_UNIT="${2:-}"; shift 2 ;; + --primary-dc) PRI_DC="${2:-}"; shift 2 ;; + --primary-model) PRI_MODEL="${2:-}"; shift 2 ;; + --primary-unit) PRI_UNIT="${2:-}"; shift 2 ;; + --skip-11-4) SKIP114=1; shift ;; + --apply) APPLY=1; shift ;; + --no-prompt) PROMPT=0; shift ;; + *) echo "FAIL: unknown argument '$1'" >&2; usage >&2; exit 1 ;; + esac + done + + [ -n "$POOL" ] || { echo "FAIL: --pool is required" >&2; exit 1; } + [ -n "$REC_DC" ] || { echo "FAIL: --recovering-dc is required (dc1|dc2)" >&2; exit 1; } + [ -n "$PRI_DC" ] || { echo "FAIL: --primary-dc is required (dc1|dc2)" >&2; exit 1; } + case "$REC_DC" in dc1|dc2) ;; *) echo "FAIL: --recovering-dc must be dc1 or dc2 (got '$REC_DC')" >&2; exit 1 ;; esac + case "$PRI_DC" in dc1|dc2) ;; *) echo "FAIL: --primary-dc must be dc1 or dc2 (got '$PRI_DC')" >&2; exit 1 ;; esac + [ "$REC_DC" != "$PRI_DC" ] || { echo "FAIL: --recovering-dc and --primary-dc must differ (got '$REC_DC' twice)" >&2; exit 1; } + [ -n "$REC_UNIT" ] || { echo "FAIL: --recovering-unit is required" >&2; exit 1; } + [ -n "$PRI_UNIT" ] || { echo "FAIL: --primary-unit is required" >&2; exit 1; } + + hdr "dc-dc-dr-drill failback recovering=$REC_DC primary=$PRI_DC mode=$([ "$APPLY" -eq 1 ] && echo apply || echo dry-run)" + note "recovering: model=$REC_MODEL unit=$REC_UNIT" + note "primary: model=$PRI_MODEL unit=$PRI_UNIT" + note "pool=$POOL" + dc_gate_check "$REC_DC"; note "$(dc_gate_line "$REC_DC")" + dc_gate_check "$PRI_DC"; note "$(dc_gate_line "$PRI_DC")" + + hdr "PLAN (fixed order -- do not reorder; this is the split-brain-safety mechanism itself)" + note "STEP 11.2 (demote recovering, do NOT let it auto-resume as primary): juju ssh -m $REC_MODEL $REC_UNIT -- \"rbd mirror pool demote $POOL\"" + note "STEP 11.3 (Cinder reconciliation -- MANUAL, NOT implemented here): back up DC2-side drift, restore into the recovering DC once multisite sync catches up -- see runbook Step 11.3" + if [ "$SKIP114" -eq 1 ]; then + note "STEP 11.4 SKIPPED (--skip-11-4): stopping after 11.2/11.3, per the runbook's own 'optional, may stay on the current primary for a while' clause" + else + note "STEP 11.4a (demote CURRENT PRIMARY -- FIRST): juju ssh -m $PRI_MODEL $PRI_UNIT -- \"rbd mirror pool demote $POOL\"" + note "STEP 11.4b (promote RECOVERING -- SECOND, only after 11.4a succeeds): juju ssh -m $REC_MODEL $REC_UNIT -- \"rbd mirror pool promote $POOL\"" + fi + + if [ "$APPLY" -ne 1 ]; then + echo + echo "re-run with --apply to execute (typed confirmation required)." + echo "OK (dry-run)" + exit 0 + fi + + dc_gate_enforce "$REC_DC" + dc_gate_enforce "$PRI_DC" + command -v juju >/dev/null 2>&1 || { echo "FAIL: juju required on PATH for --apply" >&2; exit 2; } + + if [ "$PROMPT" -eq 1 ]; then + [ -r /dev/tty ] || { echo "FAIL: --apply needs a terminal (or --no-prompt for tested automation)" >&2; exit 4; } + printf 'Type FAILBACK to confirm demote-then-promote sequence on pool %s: ' "$POOL" > /dev/tty + read -r ans < /dev/tty + [ "$ans" = "FAILBACK" ] || { echo "FAIL: confirmation refused (got '$ans') -- nothing changed" >&2; exit 4; } + fi + + hdr "EXECUTE (apply mode -- hard-coded order, abort on first failure)" + echo " DO STEP 11.2: juju ssh -m $REC_MODEL $REC_UNIT -- \"rbd mirror pool demote $POOL\"" + juju ssh -m "$REC_MODEL" "$REC_UNIT" -- "rbd mirror pool demote $POOL" &2; exit 5; } + echo " STEP 11.3 NOT executed -- MANUAL, per the header comment; do the Cinder reconciliation now" + + if [ "$SKIP114" -eq 1 ]; then + echo "OK (apply, stopped after 11.2/11.3 per --skip-11-4)" + exit 0 + fi + + echo " DO STEP 11.4a (demote CURRENT PRIMARY, first): juju ssh -m $PRI_MODEL $PRI_UNIT -- \"rbd mirror pool demote $POOL\"" + juju ssh -m "$PRI_MODEL" "$PRI_UNIT" -- "rbd mirror pool demote $POOL" &2; exit 5; } + echo " DO STEP 11.4b (promote RECOVERING, second): juju ssh -m $REC_MODEL $REC_UNIT -- \"rbd mirror pool promote $POOL\"" + juju ssh -m "$REC_MODEL" "$REC_UNIT" -- "rbd mirror pool promote $POOL" &2; exit 5; } + + echo "OK (apply) -- verify with: rbd mirror pool status --verbose + radosgw-admin sync status both zones (per runbook Step 11.5 VERIFY)" +} + +case "$SUBCMD" in + failover) run_failover "$@" ;; + failback) run_failback "$@" ;; +esac diff --git a/scripts/dc-dc-radosgw-multisite.sh b/scripts/dc-dc-radosgw-multisite.sh new file mode 100644 index 0000000..815e36a --- /dev/null +++ b/scripts/dc-dc-radosgw-multisite.sh @@ -0,0 +1,225 @@ +#!/usr/bin/env bash +# scripts/dc-dc-radosgw-multisite.sh -- radosgw MULTISITE setup for D-108's +# cinder-backup cross-DC replication mechanism (tooling gap register item #5). +# Extracts runbooks/dc-dc-phase5-dr-failover-drill.md Step 4 (stage ONE-WAY: +# DC1 master realm/zonegroup/zone, DC2 read-only join) and Step 8's radosgw +# half (flip DC2 to read-write -- TWO-WAY) into a callable, parameterized +# script instead of a hand-typed command sequence. +# +# STANDARD, well-documented upstream Ceph administration (trust these): every +# `radosgw-admin` subcommand/flag below (realm create, zonegroup create +# --master, zone create --master / --read-only, zone modify --read-only=false, +# period update --commit) is copied verbatim from the runbook, which is itself +# copied from upstream radosgw multisite docs -- nothing here is invented. +# +# GENUINELY REPO-SPECIFIC (needs review before trusting; none of it exists +# yet): realm/zonegroup/zone NAMES, radosgw endpoint URLs, the juju model/unit +# this targets, and the charm's current restart-action name (runbook Step 4: +# "confirm the current action name against `juju actions ceph-radosgw` rather +# than assuming"). Every one of these is a REQUIRED argument below -- no +# invented defaults, matching this runbook's own `` +# placeholder discipline. --restart-action is the one OPTIONAL flag: if the +# operator does not yet know the confirmed action name, omit it and this +# script reminds you to restart manually rather than guessing a name. +# +# NO LIVE RADOSGW CLUSTER EXISTS THIS SESSION (2026-07-10, prep-only) -- this +# script has never been run for real and cannot be, tonight. Default mode is +# a DRY-RUN plan print (the exact `juju ssh ...` / `juju run ...` command +# sequence it would issue, secrets redacted); --apply executes it, ONE +# command at a time, ABORTING immediately on the first non-zero exit -- per +# the runbook's own caution ("run each command, verify its output, THEN +# proceed -- do not chain the whole sequence unattended"). Its matching +# harness (tests/dc-dc-radosgw-multisite/run-tests.sh) validates ONLY this +# script's own logic (argument parsing, guard clauses, dry-run output) -- it +# does NOT and CANNOT validate real radosgw behavior. +# +# $DC gate: sources scripts/lib-net.sh and calls `lib_net_select_dc "$DC"` +# once (DOCFIX-151 convention), immediately after argument parsing. This +# deliberately makes `--dc dc2` FAIL LOUD today -- D-101/gap #3: DC2's +# network literals are not assigned yet, so there is no real DC2 Ceph +# cluster to target regardless of what endpoint URL an operator might type +# in. Also prints the replication-plane CIDR key (PLANE_NAME in lib-net.sh, +# "10.12.36.0/22" per D-101) as a REMINDER that the endpoint above must ride +# that plane -- it is NEVER substituted into any command; the real +# replication-plane ULA address is always a separately measured value, not +# inferred from this constant. +# +# Usage: +# dc-dc-radosgw-multisite.sh master-init --dc dc1 --model \ +# --unit --realm --zonegroup --zone \ +# --endpoint [--restart-action ] [--apply] +# dc-dc-radosgw-multisite.sh join-readonly --dc dc2 --model \ +# --unit --zonegroup --zone \ +# --endpoint --access-key --secret \ +# [--restart-action ] [--apply] +# dc-dc-radosgw-multisite.sh enable-two-way --dc dc2 --model \ +# --unit --zone [--apply] +# dc-dc-radosgw-multisite.sh --help +# +# Exit: 0 ok | 1 bad/missing args | 2 juju not on PATH (apply only) | +# 3 $DC gate refused | 5 apply-mode command failed. ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +usage() { + cat <<'EOF' +usage: + dc-dc-radosgw-multisite.sh master-init --dc dc1|dc2 --model M --unit U \ + --realm NAME --zonegroup NAME --zone NAME --endpoint URL \ + [--restart-action ACTION] [--apply] + dc-dc-radosgw-multisite.sh join-readonly --dc dc1|dc2 --model M --unit U \ + --zonegroup NAME --zone NAME --endpoint URL \ + --access-key KEY --secret SECRET [--restart-action ACTION] [--apply] + dc-dc-radosgw-multisite.sh enable-two-way --dc dc1|dc2 --model M --unit U \ + --zone NAME [--apply] + dc-dc-radosgw-multisite.sh --help + +Default mode is dry-run (prints the plan, executes nothing). --apply executes, +one command at a time, aborting on the first failure. No values are defaulted +or inferred -- realm/zonegroup/zone/endpoint/unit/model/keys are all required. +EOF +} + +SUBCMD="${1:-}" +case "$SUBCMD" in + -h|--help) usage; exit 0 ;; + master-init|join-readonly|enable-two-way) shift ;; + "") echo "FAIL: a subcommand is required" >&2; usage >&2; exit 1 ;; + *) echo "FAIL: unknown subcommand '$SUBCMD'" >&2; usage >&2; exit 1 ;; +esac + +DC=""; MODEL="${MODEL:-openstack}"; UNIT=""; APPLY=0 +REALM=""; ZONEGROUP=""; ZONE=""; ENDPOINT=""; RESTART_ACTION="" +ACCESS_KEY=""; SECRET="" + +while [ $# -gt 0 ]; do + case "$1" in + -h|--help) usage; exit 0 ;; + --dc) DC="${2:-}"; shift 2 ;; + --model) MODEL="${2:-}"; shift 2 ;; + --unit) UNIT="${2:-}"; shift 2 ;; + --realm) REALM="${2:-}"; shift 2 ;; + --zonegroup) ZONEGROUP="${2:-}"; shift 2 ;; + --zone) ZONE="${2:-}"; shift 2 ;; + --endpoint) ENDPOINT="${2:-}"; shift 2 ;; + --restart-action) RESTART_ACTION="${2:-}"; shift 2 ;; + --access-key) ACCESS_KEY="${2:-}"; shift 2 ;; + --secret) SECRET="${2:-}"; shift 2 ;; + --apply) APPLY=1; shift ;; + *) echo "FAIL: unknown argument '$1'" >&2; usage >&2; exit 1 ;; + esac +done + +# --- guard clauses: required args, no invented defaults --- +# --zonegroup/--endpoint are only meaningful for the two zone-CREATE +# subcommands (Step 4) -- enable-two-way (Step 8) only MODIFIES an existing +# zone's read-only flag, so it does not need them; forcing them here would +# make an operator supply irrelevant values just to satisfy the parser. +[ -n "$DC" ] || { echo "FAIL: --dc is required (dc1|dc2)" >&2; exit 1; } +case "$DC" in dc1|dc2) ;; *) echo "FAIL: --dc must be dc1 or dc2 (got '$DC')" >&2; exit 1 ;; esac +[ -n "$UNIT" ] || { echo "FAIL: --unit is required (e.g. ceph-radosgw/0 -- measured this session, never assumed)" >&2; exit 1; } +[ -n "$ZONE" ] || { echo "FAIL: --zone is required" >&2; exit 1; } + +case "$SUBCMD" in + master-init) + [ -n "$REALM" ] || { echo "FAIL: --realm is required for master-init" >&2; exit 1; } + [ -n "$ZONEGROUP" ] || { echo "FAIL: --zonegroup is required for master-init" >&2; exit 1; } + [ -n "$ENDPOINT" ] || { echo "FAIL: --endpoint is required for master-init" >&2; exit 1; } + ;; + join-readonly) + [ -n "$ZONEGROUP" ] || { echo "FAIL: --zonegroup is required for join-readonly" >&2; exit 1; } + [ -n "$ENDPOINT" ] || { echo "FAIL: --endpoint is required for join-readonly" >&2; exit 1; } + [ -n "$ACCESS_KEY" ] || { echo "FAIL: --access-key is required for join-readonly (capture live from DC1's multisite system user -- never invent)" >&2; exit 1; } + [ -n "$SECRET" ] || { echo "FAIL: --secret is required for join-readonly" >&2; exit 1; } + ;; + enable-two-way) : ;; +esac + +# --- $DC gate (DOCFIX-151 convention) --- +# Informational in dry-run (a plan preview is not a mutation -- the whole +# point of --dry-run is to be usable/reviewable even for a DC whose literals +# aren't assigned yet); HARD BLOCKING before any --apply execution below. +# shellcheck source=scripts/lib-net.sh +. "$HERE/lib-net.sh" +DC_GATE_MSG="$(lib_net_select_dc "$DC" 2>&1 1>/dev/null)"; DC_GATE_RC=$? +if [ "$DC_GATE_RC" -eq 0 ]; then + DC_GATE_LINE="\$DC gate: OK ($DC)" +else + DC_GATE_LINE="\$DC gate: FAILED for '$DC' -- $DC_GATE_MSG" +fi + +REPL_CIDR="" +for c in "${PLANE_CIDRS[@]}"; do + [ "${PLANE_NAME[$c]:-}" = "replication" ] && REPL_CIDR="$c" +done + +hdr() { echo; echo "=== $* ==="; } +note() { echo " $*"; } + +hdr "dc-dc-radosgw-multisite subcmd=$SUBCMD dc=$DC mode=$([ "$APPLY" -eq 1 ] && echo apply || echo dry-run)" +note "juju model=$MODEL unit=$UNIT" +note "$DC_GATE_LINE" +note "reminder: the endpoint above must ride the REPLICATION plane (${REPL_CIDR:-unassigned}, per D-101/lib-net.sh) -- not substituted into any command, confirm the real ULA address separately" + +# --- build the remote radosgw-admin command sequence (verbatim upstream ops) --- +SSH_CMDS=() +case "$SUBCMD" in + master-init) + SSH_CMDS+=("radosgw-admin realm create --rgw-realm=$REALM --default") + SSH_CMDS+=("radosgw-admin zonegroup create --rgw-zonegroup=$ZONEGROUP --endpoints=$ENDPOINT --master --default") + SSH_CMDS+=("radosgw-admin zone create --rgw-zonegroup=$ZONEGROUP --rgw-zone=$ZONE --endpoints=$ENDPOINT --master --default") + SSH_CMDS+=("radosgw-admin period update --commit") + ;; + join-readonly) + SSH_CMDS+=("radosgw-admin zone create --rgw-zonegroup=$ZONEGROUP --rgw-zone=$ZONE --endpoints=$ENDPOINT --read-only --access-key=$ACCESS_KEY --secret=$SECRET") + SSH_CMDS+=("radosgw-admin period update --commit") + ;; + enable-two-way) + SSH_CMDS+=("radosgw-admin zone modify --rgw-zone=$ZONE --read-only=false") + SSH_CMDS+=("radosgw-admin period update --commit") + ;; +esac + +redact() { # redact -- hides --secret's value in printed output only + local s="$1" + if [ -n "$SECRET" ]; then s="${s//$SECRET/}"; fi + printf '%s' "$s" +} + +hdr "PLAN" +n=0 +for cmd in "${SSH_CMDS[@]}"; do + n=$((n+1)) + note "STEP $n: juju ssh -m $MODEL $UNIT -- \"$(redact "$cmd")\"" +done +if [ -n "$RESTART_ACTION" ]; then + n=$((n+1)) + note "STEP $n: juju run $UNIT $RESTART_ACTION -m $MODEL (restart to pick up realm/zone membership)" +else + note "NOTE: no --restart-action given -- restart the unit manually after this sequence (confirm the action name via 'juju actions ceph-radosgw', per the runbook; not guessed here)" +fi + +if [ "$APPLY" -ne 1 ]; then + echo + echo "re-run with --apply to execute (one command at a time, aborts on first failure)." + echo "OK (dry-run)" + exit 0 +fi + +[ "$DC_GATE_RC" -eq 0 ] || { echo "FAIL: \$DC gate refused for '$DC' -- STOP, do not mutate an unassigned DC ($DC_GATE_MSG)" >&2; exit 3; } +command -v juju >/dev/null 2>&1 || { echo "FAIL: juju required on PATH for --apply" >&2; exit 2; } + +hdr "EXECUTE (apply mode -- one command at a time, abort on first failure)" +n=0 +for cmd in "${SSH_CMDS[@]}"; do + n=$((n+1)) + echo " DO STEP $n: juju ssh -m $MODEL $UNIT -- \"$(redact "$cmd")\"" + juju ssh -m "$MODEL" "$UNIT" -- "$cmd" &2; exit 5; } +done +if [ -n "$RESTART_ACTION" ]; then + n=$((n+1)) + echo " DO STEP $n: juju run $UNIT $RESTART_ACTION -m $MODEL" + juju run "$UNIT" "$RESTART_ACTION" -m "$MODEL" &2; exit 5; } +fi + +echo "OK (apply) -- verify with: radosgw-admin realm list / zonegroup get / sync status (per runbook Step 4/8 VERIFY)" diff --git a/scripts/dc-dc-rbd-mirror.sh b/scripts/dc-dc-rbd-mirror.sh new file mode 100644 index 0000000..2983cb9 --- /dev/null +++ b/scripts/dc-dc-rbd-mirror.sh @@ -0,0 +1,188 @@ +#!/usr/bin/env bash +# scripts/dc-dc-rbd-mirror.sh -- rbd-mirror pool peer bootstrap for D-108's +# Glance replication mechanism (tooling gap register item #5). Extracts +# runbooks/dc-dc-phase5-dr-failover-drill.md Step 5 (stage ONE-WAY: rx-only +# peer bootstrap) and Step 8's rbd-mirror half (flip to rx-tx -- TWO-WAY) +# into a callable, parameterized script. +# +# STANDARD, well-documented upstream Ceph administration (trust these): every +# `rbd mirror pool` subcommand/flag below (enable image, peer bootstrap +# create/import, --direction rx-only|rx-tx) is copied verbatim from the +# runbook, which is itself copied from upstream rbd-mirror docs -- nothing +# here is invented. +# +# GENUINELY REPO-SPECIFIC (needs review before trusting; none of it exists +# yet): the Glance POOL name (runbook Step 5 explicitly warns "do not assume +# glance -- confirm against the live ceph-mon / `juju config glance +# rbd-pool`"), the site names, and the juju model/unit this targets. All +# REQUIRED arguments below -- no invented defaults. +# +# SECRET-ADJACENT TOKEN HANDLING: the bootstrap token is credential material +# (whoever holds it can join the mirror relationship). This script NEVER +# reads the token's contents into its own context -- bootstrap-primary +# redirects the remote command's stdout straight to a file ON THE REMOTE +# UNIT (via the ssh'd shell's own `>`), and bootstrap-secondary expects the +# token file to already exist on ITS target unit (transferred out of band by +# the operator, per the runbook's own "transfer the token to DC2 out of band +# ... never printed into this session's context" instruction) -- this script +# does not perform that transfer itself. +# +# NO LIVE RBD-MIRROR DAEMON EXISTS THIS SESSION (2026-07-10, prep-only) -- +# never run for real, cannot be tonight. Default mode is a DRY-RUN plan +# print; --apply executes, one command at a time, aborting on first failure. +# Its matching harness (tests/dc-dc-rbd-mirror/run-tests.sh) validates ONLY +# this script's own logic (argument parsing, guard clauses, dry-run output, +# the required --direction choice) -- it does NOT and CANNOT validate real +# rbd-mirror daemon behavior. +# +# $DC gate: sources scripts/lib-net.sh and calls `lib_net_select_dc "$DC"` +# once (DOCFIX-151 convention), immediately after argument parsing -- +# deliberately makes `--dc dc2` FAIL LOUD today (D-101/gap #3: DC2 has no +# assigned network literals, so no real DC2 Ceph cluster exists to target). +# Also prints the replication-plane CIDR key (PLANE_NAME in lib-net.sh) as a +# REMINDER that rbd-mirror daemon peering must ride that plane, per D-108's +# explicit correction of the DR-seed's original v4 carrier language -- never +# substituted into a command, the real ULA peer address is always separately +# measured. +# +# Usage: +# dc-dc-rbd-mirror.sh bootstrap-primary --dc dc1 --model M --unit U \ +# --pool NAME --site-name NAME [--token-out /remote/path] [--apply] +# dc-dc-rbd-mirror.sh bootstrap-secondary --dc dc2 --model M --unit U \ +# --pool NAME --site-name NAME --direction rx-only|rx-tx \ +# --token-in /remote/path [--apply] +# dc-dc-rbd-mirror.sh --help +# +# Exit: 0 ok | 1 bad/missing args | 2 juju not on PATH (apply only) | +# 3 $DC gate refused | 5 apply-mode command failed. ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +usage() { + cat <<'EOF' +usage: + dc-dc-rbd-mirror.sh bootstrap-primary --dc dc1|dc2 --model M --unit U \ + --pool NAME --site-name NAME [--token-out /remote/path] [--apply] + dc-dc-rbd-mirror.sh bootstrap-secondary --dc dc1|dc2 --model M --unit U \ + --pool NAME --site-name NAME --direction rx-only|rx-tx \ + --token-in /remote/path [--apply] + dc-dc-rbd-mirror.sh --help + +Default mode is dry-run (prints the plan, executes nothing). --apply executes, +one command at a time, aborting on the first failure. --direction has no +default -- rx-only (Step 5, one-way staging) and rx-tx (Step 8, two-way) are +a deliberate, conscious choice each time, never assumed. +EOF +} + +SUBCMD="${1:-}" +case "$SUBCMD" in + -h|--help) usage; exit 0 ;; + bootstrap-primary|bootstrap-secondary) shift ;; + "") echo "FAIL: a subcommand is required" >&2; usage >&2; exit 1 ;; + *) echo "FAIL: unknown subcommand '$SUBCMD'" >&2; usage >&2; exit 1 ;; +esac + +DC=""; MODEL="${MODEL:-openstack}"; UNIT=""; APPLY=0 +POOL=""; SITE_NAME=""; DIRECTION="" +TOKEN_OUT="/tmp/rbd-mirror-bootstrap-token" +TOKEN_IN="" + +while [ $# -gt 0 ]; do + case "$1" in + -h|--help) usage; exit 0 ;; + --dc) DC="${2:-}"; shift 2 ;; + --model) MODEL="${2:-}"; shift 2 ;; + --unit) UNIT="${2:-}"; shift 2 ;; + --pool) POOL="${2:-}"; shift 2 ;; + --site-name) SITE_NAME="${2:-}"; shift 2 ;; + --direction) DIRECTION="${2:-}"; shift 2 ;; + --token-out) TOKEN_OUT="${2:-}"; shift 2 ;; + --token-in) TOKEN_IN="${2:-}"; shift 2 ;; + --apply) APPLY=1; shift ;; + *) echo "FAIL: unknown argument '$1'" >&2; usage >&2; exit 1 ;; + esac +done + +# --- guard clauses: required args, no invented defaults --- +[ -n "$DC" ] || { echo "FAIL: --dc is required (dc1|dc2)" >&2; exit 1; } +case "$DC" in dc1|dc2) ;; *) echo "FAIL: --dc must be dc1 or dc2 (got '$DC')" >&2; exit 1 ;; esac +[ -n "$UNIT" ] || { echo "FAIL: --unit is required (e.g. ceph-rbd-mirror/0 or ceph-mon/0 -- measured, never assumed)" >&2; exit 1; } +[ -n "$POOL" ] || { echo "FAIL: --pool is required (confirm against 'ceph osd pool ls' + 'juju config glance rbd-pool' -- do not assume 'glance', per the runbook)" >&2; exit 1; } +[ -n "$SITE_NAME" ] || { echo "FAIL: --site-name is required" >&2; exit 1; } + +case "$SUBCMD" in + bootstrap-secondary) + [ -n "$DIRECTION" ] || { echo "FAIL: --direction is required for bootstrap-secondary (rx-only|rx-tx -- a conscious choice, never defaulted)" >&2; exit 1; } + case "$DIRECTION" in rx-only|rx-tx) ;; *) echo "FAIL: --direction must be rx-only or rx-tx (got '$DIRECTION')" >&2; exit 1 ;; esac + [ -n "$TOKEN_IN" ] || { echo "FAIL: --token-in is required for bootstrap-secondary (the path, ON THIS UNIT, where the out-of-band-transferred bootstrap token file lives -- this script does not transfer it)" >&2; exit 1; } + ;; +esac + +# --- $DC gate (DOCFIX-151 convention) --- +# Informational in dry-run (a plan preview is not a mutation); HARD BLOCKING +# before any --apply execution below. +# shellcheck source=scripts/lib-net.sh +. "$HERE/lib-net.sh" +DC_GATE_MSG="$(lib_net_select_dc "$DC" 2>&1 1>/dev/null)"; DC_GATE_RC=$? +if [ "$DC_GATE_RC" -eq 0 ]; then + DC_GATE_LINE="\$DC gate: OK ($DC)" +else + DC_GATE_LINE="\$DC gate: FAILED for '$DC' -- $DC_GATE_MSG" +fi + +REPL_CIDR="" +for c in "${PLANE_CIDRS[@]}"; do + [ "${PLANE_NAME[$c]:-}" = "replication" ] && REPL_CIDR="$c" +done + +hdr() { echo; echo "=== $* ==="; } +note() { echo " $*"; } + +hdr "dc-dc-rbd-mirror subcmd=$SUBCMD dc=$DC mode=$([ "$APPLY" -eq 1 ] && echo apply || echo dry-run)" +note "juju model=$MODEL unit=$UNIT pool=$POOL site-name=$SITE_NAME" +note "$DC_GATE_LINE" +note "reminder: rbd-mirror daemon peering must ride the REPLICATION plane (${REPL_CIDR:-unassigned}, per D-101/D-108/lib-net.sh) -- not substituted into any command, confirm the real ULA peer address separately" + +# --- build the remote rbd command sequence (verbatim upstream ops) --- +SSH_CMDS=() +case "$SUBCMD" in + bootstrap-primary) + SSH_CMDS+=("rbd mirror pool enable $POOL image") + SSH_CMDS+=("rbd mirror pool peer bootstrap create --site-name $SITE_NAME $POOL > $TOKEN_OUT") + ;; + bootstrap-secondary) + SSH_CMDS+=("rbd mirror pool enable $POOL image") + SSH_CMDS+=("rbd mirror pool peer bootstrap import --site-name $SITE_NAME --direction $DIRECTION $POOL $TOKEN_IN") + ;; +esac + +hdr "PLAN" +n=0 +for cmd in "${SSH_CMDS[@]}"; do + n=$((n+1)) + note "STEP $n: juju ssh -m $MODEL $UNIT -- \"$cmd\"" +done +if [ "$SUBCMD" = "bootstrap-primary" ]; then + note "NOTE: the token now lives at $TOKEN_OUT ON THE REMOTE UNIT -- transfer it to the DC2 unit OUT OF BAND (never printed/piped through this session), then run bootstrap-secondary there" +fi + +if [ "$APPLY" -ne 1 ]; then + echo + echo "re-run with --apply to execute (one command at a time, aborts on first failure)." + echo "OK (dry-run)" + exit 0 +fi + +[ "$DC_GATE_RC" -eq 0 ] || { echo "FAIL: \$DC gate refused for '$DC' -- STOP, do not mutate an unassigned DC ($DC_GATE_MSG)" >&2; exit 3; } +command -v juju >/dev/null 2>&1 || { echo "FAIL: juju required on PATH for --apply" >&2; exit 2; } + +hdr "EXECUTE (apply mode -- one command at a time, abort on first failure)" +n=0 +for cmd in "${SSH_CMDS[@]}"; do + n=$((n+1)) + echo " DO STEP $n: juju ssh -m $MODEL $UNIT -- \"$cmd\"" + juju ssh -m "$MODEL" "$UNIT" -- "$cmd" &2; exit 5; } +done + +echo "OK (apply) -- verify with: rbd mirror pool info/status --verbose (per runbook Step 5/8 VERIFY)" diff --git a/tests/dc-dc-dr-drill/run-tests.sh b/tests/dc-dc-dr-drill/run-tests.sh new file mode 100644 index 0000000..fb42b11 --- /dev/null +++ b/tests/dc-dc-dr-drill/run-tests.sh @@ -0,0 +1,93 @@ +#!/usr/bin/env bash +# tests/dc-dc-dr-drill/run-tests.sh -- offline harness for +# scripts/dc-dc-dr-drill.sh. Only tests THIS SCRIPT's own logic: argument +# parsing / guard clauses, the --confirmed-down gate on failover --apply, +# the failback --apply typed-confirmation gate, and -- the single most +# important assertion in this suite -- that the failback PLAN's printed +# step order is demote-recovering, then demote-CURRENT-PRIMARY-before- +# promote-recovering, verified by grepping the actual LINE ORDER of the +# dry-run output (not just that all four lines are present). Does NOT and +# CANNOT validate real rbd-mirror promote/demote behavior -- no live +# cluster exists this session (2026-07-10, prep-only). +# Exit: 0 all pass | 1 any case failed. ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SCRIPT="$HERE/../../scripts/dc-dc-dr-drill.sh" +PASS=0; FAIL=0 + +run() { # run