diff --git a/docs/changelog-20260710-sweep-dcdc-runbook-fixes.md b/docs/changelog-20260710-sweep-dcdc-runbook-fixes.md new file mode 100644 index 0000000..b22894c --- /dev/null +++ b/docs/changelog-20260710-sweep-dcdc-runbook-fixes.md @@ -0,0 +1,77 @@ +# Changelog 2026-07-10 -- Full-project sweep: DC-DC runbook consistency fixes (DOCFIX-171) + +No live infrastructure touched. Part of the full-project sweep (see +`docs/changelog-20260710-sweep-vr0-runbook-opentofu-fixes.md`, DOCFIX-170, +for the sweep's overall scope). This changelog covers findings from a +dedicated agent that re-checked all seven DC-DC Stage runbooks and the +master tracker against the actual repo state after the second gap-closure +wave (DOCFIX-162 through 168). + +## Items + +### 1. DOCFIX-171a -- Stage 1's calculator scripts never wired into Step 3 +FILE: `runbooks/dc-dc-phase0-vcloud-prep.md`. + +WHAT: `docs/dc-dc-deployment-workflow.md`'s gap register item #7 and +`scripts/dc-dc-mtu-geneve-budget.sh`'s own header both asserted "the first +real run is part of executing Step 3" of this runbook -- but Step 3 still +contained only the old manual back-of-envelope arithmetic, never actually +invoking either `scripts/dc-dc-mtu-geneve-budget.sh` or `scripts/dc-dc- +ceph-disk-budget.sh`. Same class as DOCFIX-169: a gap-closure delivery +never got cross-referenced back into the runbook that was supposed to +consume it. Fixed: Step 3 now shows the real invocations (with their exact +flags) in place of the hand-arithmetic instructions, explicitly noting the +calculator's OUTPUT (a downstream tenant-MTU consequence) is distinct from +what actually goes into `underlay_mtu` (the measured value itself). + +### 2. DOCFIX-171b -- Stage 5's stale Sequence-list bracket tag +FILE: `runbooks/dc-dc-phase4-juju-bundle-per-dc.md`. + +WHAT: the top-level Sequence list still read `6. Write + apply the IPv6 +family-matrix overlay [NOT YET WRITTEN -- see below]`, even though Step 6's +own body had already been correctly updated (during DOCFIX-164) to +`[OVERLAY DRAFTED 2026-07-10, NOT YET APPLIED]`. The Sequence-list's own +bracket tag was simply never touched when the Step 6 body was -- a direct, +within-file self-contradiction. Fixed to match. + +### 3. DOCFIX-171c -- Stage 6's checklist overstated its own step content +FILE: `runbooks/dc-dc-phase5-dr-failover-drill.md`. + +WHAT: the delivery checklist claimed Steps 4/5/8/10.2/11.2-11.4 "call +these scripts... instead of typing the command sequences by hand," but +those step BODIES (verified by reading them) still showed only raw +`radosgw-admin`/`rbd mirror pool ...` commands -- no invocation of +`scripts/dc-dc-radosgw-multisite.sh`, `dc-dc-rbd-mirror.sh`, or +`dc-dc-dr-drill.sh` appeared anywhere in Steps 1-12. Fixed by adding a +"PREFER `scripts/dc-dc-...`" pointer at the top of each of the five +affected steps, directing to the corresponding script (with its exact +subcommand) as the safer, tested alternative to typing the raw commands, +while keeping the raw commands as the reference/fallback. Updated the +checklist item itself to describe what's now actually true rather than +what was aspirationally claimed before. + +### 4. DOCFIX-171d -- two DOCFIX numbers never closed out in the session ledger +FILE: `docs/session-ledger.md`. + +WHAT: comparing final in-use numbers (from changelog filenames and +cross-references elsewhere) against the ledger's own tail: DOCFIX-164 (the +IPv6 overlay) and DOCFIX-167 (the bundle.yaml Designate/D-108 change) both +had ledger bullets ending in unresolved "Uncommitted -- orchestrating +session will assign the real DOCFIX number at integration time" boilerplate +-- even though both deliveries HAD been assigned real numbers and committed, +correctly cited everywhere else (the tracker doc, the runbooks, the +changelog file titles themselves). This is an integration-pass omission, +not a numbering collision (no number was reused or double-assigned) -- +fixed by writing the real numbers into both closing lines. A third, lower- +severity instance (gap #15/DOCFIX-166's entry, which already cited the +correct number up front but still carried the same stale closing +boilerplate) was fixed for consistency too. + +## Verification + +`bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy warn. No script +logic touched by this changelog -- doc/runbook prose only. + +REVERT: `git checkout HEAD~ -- runbooks/dc-dc-phase0-vcloud-prep.md +runbooks/dc-dc-phase4-juju-bundle-per-dc.md runbooks/dc-dc-phase5-dr- +failover-drill.md docs/session-ledger.md` (safe). diff --git a/runbooks/dc-dc-phase0-vcloud-prep.md b/runbooks/dc-dc-phase0-vcloud-prep.md index bb1f463..a0b80b9 100644 --- a/runbooks/dc-dc-phase0-vcloud-prep.md +++ b/runbooks/dc-dc-phase0-vcloud-prep.md @@ -153,34 +153,47 @@ "Prefer jumbo (9000) end-to-end... if pinned at 1500, set the reduced tenant MTU consistently." **Do not assume jumbo** -- use the measured value. -**Decision to record (D-101 MTU sub-policy, buildout-design Section 3):** -- If measured MTU >= 9000: set `underlay_mtu = 9000` in Step 7's tfvars. - Tenant MTU stays 1500; geneve/amphora headroom is generous. -- If measured MTU == 1500 (the common case for a plain host NIC): set - `underlay_mtu = 1500`, and record that tenant-network MTU will need to - drop to ~1444 (v6-geneve overhead: 40 IPv6 + 8 UDP + 8 Geneve base = 56 - bytes, before nested-virt/options) -- this propagates to ovn geneve, - tenant-network MTU, and amphora settings in later stages (Stage 5/7), not - this one. Just RECORD the decision here; the propagation happens where - those components are actually configured. +**Decision to record (D-101 MTU sub-policy, buildout-design Section 3) -- +use the tested calculator (DOCFIX-162, tooling gap #7, 19/19 tests) instead +of hand arithmetic:** +```bash +bash scripts/dc-dc-mtu-geneve-budget.sh --underlay-mtu +``` +This reproduces D-101's own worked example exactly (a measured 1500 -> +tenant MTU 1444) and recommends leaving tenant MTU at 1500 for a measured +jumbo (>=9000) underlay -- read the script's own `--help` for the full +arithmetic it quotes verbatim from D-101. Set `underlay_mtu` in Step 7's +tfvars to the MEASURED value from Step 3's own `ip -o link show` check +above (never the script's output value itself -- the script tells you the +DOWNSTREAM tenant MTU consequence, not what to put in `underlay_mtu`, +which is always the raw measured underlay number). The propagation to ovn +geneve/tenant-network MTU/amphora settings happens in later stages +(Stage 5/7), not this one -- this step only records the decision. **Decision to record (Ceph size=3 vs size=2, Section 3 + the 2026-07-09 -Stage-0 ruling):** target is size=3/min_size=2 BY DEFAULT (ADOPTED, not -still open). This is conditional on the measured disk budget from Step 2: +Stage-0 ruling) -- use the tested calculator (DOCFIX-162, 16/16 tests) +instead of hand arithmetic:** +```bash +bash scripts/dc-dc-ceph-disk-budget.sh --total-disk \ + --dc1-nodes --dc1-per-node-osd \ + --dc2-nodes --dc2-per-node-osd \ + --backup-overhead-fraction ``` -required_disk_TiB ~= (DC1_Ceph_size3 + DC2_Ceph_size3 + per-DC_radosgw_backup_pools - + Glance_rbd-mirror_targets + images + per-DC_artifact_mirrors) -``` -The exact per-node OSD footprint isn't knowable until DC1's actual Ceph -cluster exists and reports real usage (Stage 5) -- for THIS stage, the gate -is narrower: confirm the MEASURED total disk (Step 2's `df`/`lsblk` output) -is large enough that size=3 across both DCs plus backup/mirror/image -overhead is PLAUSIBLE, not exact. If the measured disk is clearly short (a -back-of-envelope check, not final math), record that now as a flag for -Stage 5's real budget pass rather than silently hoping it works out -- -size=2 fallback is CONFIRMED ADOPTED as the only path if the real numbers -don't fit (D-101/Section 3), but it must be an explicit, logged decision at -the point real usage is measured (Stage 5), not guessed here. +Target is size=3/min_size=2 BY DEFAULT (ADOPTED, not still open). The exact +per-node OSD footprint and the backup/mirror/image overhead FRACTION are +NOT knowable/measurable until DC1's actual Ceph cluster exists and reports +real usage (Stage 5) -- for THIS stage, pass your best PLANNED/measured +estimate for `--total-disk` (Step 2's `df`/`lsblk` output) and a +plausibility-level `--backup-overhead-fraction` (the script has no default +for this fraction on purpose -- it is not specified as a hard number +anywhere in this repo), to get a PLAUSIBILITY verdict, not a final one. If +the script's verdict is a shortfall, record that now as a flag for Stage +5's real budget pass rather than silently hoping it works out -- size=2 +fallback is CONFIRMED ADOPTED as the only path if the real numbers don't +fit (D-101/Section 3), but it must be an explicit, logged decision at the +point real usage is measured (Stage 5), not guessed here. The script itself +never applies size=2 silently -- it only names the fallback path and +leaves the decision to the operator, per its own `--help` text. **GATE (Section 3, second + third bullets):** MTU measured and recorded; disk budget plausibility-checked and Ceph size decision path recorded diff --git a/runbooks/dc-dc-phase4-juju-bundle-per-dc.md b/runbooks/dc-dc-phase4-juju-bundle-per-dc.md index 5724101..5bd2aa2 100644 --- a/runbooks/dc-dc-phase4-juju-bundle-per-dc.md +++ b/runbooks/dc-dc-phase4-juju-bundle-per-dc.md @@ -79,7 +79,7 @@ 3. preflight.sh PASS (read-only gate) 4. Run phase-01 (bundle deploy) against THIS DC's model [adapted, gated -- see below] 5. Run phase-02 (vault bring-up) -- INDEPENDENT root per DC [adapted, SECRET-HANDLING] -6. Write + apply the IPv6 family-matrix overlay [NOT YET WRITTEN -- see below] +6. Write + apply the IPv6 family-matrix overlay [DRAFTED 2026-07-10, NOT YET APPLIED -- see below] 7. Run phase-03 (core verify) [adapted, gated] 8. Run phase-04 (network carve) [adapted, gated] 9. Run phase-05 (octavia enablement) [adapted, gated] diff --git a/runbooks/dc-dc-phase5-dr-failover-drill.md b/runbooks/dc-dc-phase5-dr-failover-drill.md index 78bcff5..7d60f76 100644 --- a/runbooks/dc-dc-phase5-dr-failover-drill.md +++ b/runbooks/dc-dc-phase5-dr-failover-drill.md @@ -214,9 +214,16 @@ anything bidirectional. This step wires radosgw multisite for the cinder-backup target, DC1 as master zone only. +**PREFER `scripts/dc-dc-radosgw-multisite.sh master-init ...`** (DOCFIX-165) over typing the +raw commands below by hand -- it runs the SAME sequence with a `--dry-run` preview by +default and `--apply` to execute one command at a time via `juju ssh`, aborting on first +failure (run `bash scripts/dc-dc-radosgw-multisite.sh --help` for its exact flags). The raw +commands below are kept as the reference for what the script actually does, and as a +fallback if the script is unavailable. + > CAUTION: `radosgw-admin ... --commit` mutates the realm/period on a LIVE radosgw. Run each > command, verify its output, THEN proceed to the next -- do not chain the whole sequence -> unattended. +> unattended (the script does this for you in `--apply` mode). **MUTATION -- on a DC1 ceph-radosgw unit** ```bash @@ -267,6 +274,12 @@ ## Step 5 -- Stage ONE-WAY: rbd-mirror peer bootstrap (rx-only) [MUTATION: gated] +**PREFER `scripts/dc-dc-rbd-mirror.sh bootstrap-primary ...` / `bootstrap-secondary --direction +rx-only ...`** (DOCFIX-165) over typing the raw commands below by hand -- dry-run by default, +`--apply` to execute, and the bootstrap token file is written with an unpredictable path and +`chmod 600`'d immediately (`bash scripts/dc-dc-rbd-mirror.sh --help` for exact flags). The raw +commands below are the reference for what the script does, and the fallback if unavailable. + **CHECK -- confirm the Glance pool name** (do not assume `glance` -- confirm against the live ceph-mon: `ceph osd pool ls` and cross-check which pool glance-charm's `rbd-pool` config points at) @@ -380,6 +393,10 @@ ## Step 8 -- Enable TWO-WAY [MUTATION: gated] +**PREFER `scripts/dc-dc-radosgw-multisite.sh enable-two-way ...` and `scripts/dc-dc-rbd-mirror.sh +bootstrap-secondary --direction rx-tx ...`** (DOCFIX-165) over the raw commands below -- same +dry-run/`--apply` safety as Steps 4/5. The raw commands are the reference/fallback. + **MUTATION -- radosgw: make DC2's zone a full read-write member** ```bash radosgw-admin zone modify --rgw-zone= --read-only=false @@ -492,6 +509,10 @@ exactly the ambiguous signal Section 8's skeleton warns about. Record the evidence. **Step 10.2 -- Glance: promote at DC2** (Section 8 skeleton item 2) + +**PREFER `scripts/dc-dc-dr-drill.sh failover ...`** (DOCFIX-165) over the raw command below -- +same dry-run/`--apply` safety, plus it always prints the per-image Glance re-registration +reminder below as a flagged MANUAL step, never auto-executed. ```bash rbd mirror pool promote --force ``` @@ -547,6 +568,11 @@ that is itself a finding to log, not silently accept. **Step 11.2 -- demote DC1's Glance pool; resync from DC2 (now primary)** (item 2) + +**PREFER `scripts/dc-dc-dr-drill.sh failback ...`** (DOCFIX-165) for Steps 11.2-11.4 together +over the raw commands below -- the demote-CURRENT-PRIMARY-before-promote-RECOVERING ordering +this whole step's CAUTION describes is HARD-CODED as fixed call order inside the script itself +(not left to documentation), and it defaults to dry-run with `--apply` to execute. ```bash rbd mirror pool demote # on DC1, now that it's back rbd mirror pool status --verbose # confirm DC1 catching up from DC2 @@ -622,7 +648,11 @@ commits, not secrets. - [ ] `docs/session-ledger.md` updated with the outcome. - [ ] `docs/dc-dc-deployment-workflow.md` Stage 6 row and tracker table updated (Step 12). -- [x] **Extraction DONE 2026-07-09 (DOCFIX-165), not future work anymore.** The radosgw +- [x] **Extraction DONE 2026-07-09 (DOCFIX-165), not future work anymore -- and Steps 4, 5, 8, + 10.2, 11.2-11.4 above now each carry a "PREFER `scripts/dc-dc-...`" pointer at their own + top (added 2026-07-10 during a full-project sweep, after the first version of this + checklist item claimed this was already done when the step bodies themselves still only + showed raw commands -- fixed so the claim and the actual text agree).** The radosgw multisite / rbd-mirror command sequences above (Steps 4, 5, 8, 10.2, 11.2-11.4) are now real, callable, parameterized scripts, closing Tooling gap register item #5's MECHANISM half: `scripts/dc-dc-radosgw-multisite.sh` (`master-init`/`join-readonly`/