diff --git a/docs/changelog-20260709-dc-dc-phase3-runbook.md b/docs/changelog-20260709-dc-dc-phase3-runbook.md new file mode 100644 index 0000000..2ff2f3a --- /dev/null +++ b/docs/changelog-20260709-dc-dc-phase3-runbook.md @@ -0,0 +1,73 @@ +# Changelog 2026-07-09 -- Stage 4 runbook: MAAS enlist/commission/deploy per DC (DOCFIX-157) + +No live infrastructure touched -- new runbook document only. Authored by a +delegated subagent (general-purpose, briefed with this repo's discipline, +D-103/D-100/D-107, the tooling gap register, `phase-00-teardown-maas- +reset.md` and `scripts/lib-net.sh`/`lib-hosts.sh` as primary sources), +reviewed before this commit -- repo-lint was clean on the first pass. + +## Item + +### 1. DOCFIX-157 -- `runbooks/dc-dc-phase3-maas-enlist-deploy.md` +FILES: `runbooks/dc-dc-phase3-maas-enlist-deploy.md` (new), `docs/dc-dc- +deployment-workflow.md` (Stage 4 row updated, tooling gap register items +11/12/13/14/15 added or corrected). + +WHAT: adapts `phase-00-teardown-maas-reset.md`'s validated single-rack +MAAS commission/deploy/carve sequence to VR1's multi-rack, per-DC shape. +Run once per `$DC` (dc1/dc2), explicitly not mixed in one shell session. + +**Mandates the `$DC` selector convention as its first real step** -- +`lib_net_select_dc "$DC"` (a documented no-op for dc1, fail-loud for dc2) +followed HONESTLY by `lib_hosts_select_dc "$DC"`, which the runbook states +is EXPECTED TO FAIL for both DCs right now: no per-DC host data exists in +`scripts/lib-hosts.sh` yet, and populating it from THIS stage's own real +enrollment is exactly the follow-up delivery that lets the selector start +passing. This is documented as a deliberate chicken-and-egg, not a bug to +route around -- every step in the runbook resolves host identity LIVE from +MAAS instead. + +**Seven real gaps found and named in a dedicated section, each logged as a +finding rather than silently worked around:** +1. No OpenTofu module stands up a per-DC MAAS rack controller VM itself -- + only registration (`modules/maas-vm-host`) exists. +2. `reenroll-hosts.sh`, `carve-host-interfaces.sh`, and `phase-00-maas- + standup.sh` do not accept a `$DC` argument or call the new selector + functions -- only `lib-net.sh`/`lib-hosts.sh` themselves gained them + (DOCFIX-151). The runbook describes adapting each script's MECHANICS by + hand rather than inventing a `--dc` flag those scripts don't have. +3. Per-DC node count/sizing is an unmade Phase-0 decision -- the runbook + discovers however many nodes actually exist live (Step 2), never + assuming DC0's four hosts carry over. +4. DC2 is additionally blocked at the `lib_net_select_dc` layer (no NetBox + literals yet). +5. The per-DC artifact mirror (D-107) has no identified owning stage/ + module anywhere in this repo as surveyed -- flagged before Step 7's + reachability check assumes one exists. +6. The MAAS status a VM-host-DISCOVERED (vs. PXE-enlisted) pre-existing + domain lands in has never been observed in this repo -- Step 2 measures + it live rather than assuming `New` or an auto-`Ready` per phase-00's + different code path. +7. `lib-hosts.sh`'s `HOST_TAG`/boot-fabric-name literals (`openstack`, + `2_metal`) are DC0-specific -- confirmed live per DC, not assumed reused. + +**A real sequencing subtlety flagged, not glossed over:** phase-00's own +precedent carves interfaces BEFORE deploy (Ready-only); if Stage 3's +discovery+commission flow for a DC leaves nodes at `Ready` before this +stage's carve step, the runbook instructs reordering Steps 4/5 to match +phase-00's actual order and recording which order was used, rather than +blindly following this document's own numbering against contradicting live +machine states. + +VERIFIED: `bash scripts/repo-lint.sh` 0 fail, 1 documented legacy warn (no +fixes needed). NOT YET EXECUTED. + +REVERT: `rm runbooks/dc-dc-phase3-maas-enlist-deploy.md && git checkout +HEAD~ -- docs/dc-dc-deployment-workflow.md` (safe). + +## Next actionable step + +This stage's own delivery checklist requires populating `scripts/lib- +hosts.sh` with real per-DC host data once it actually runs -- that follow- +up unblocks `lib_hosts_select_dc` for the DC it ran against and is a +prerequisite named explicitly in Stage 5's own runbook. diff --git a/docs/changelog-20260709-dc-dc-phase5-runbook.md b/docs/changelog-20260709-dc-dc-phase5-runbook.md new file mode 100644 index 0000000..801792b --- /dev/null +++ b/docs/changelog-20260709-dc-dc-phase5-runbook.md @@ -0,0 +1,66 @@ +# Changelog 2026-07-09 -- Stage 6 runbook: DR wiring and failover drill (DOCFIX-158) + +No live infrastructure touched -- new runbook document only. Authored by a +delegated subagent (general-purpose, briefed with this repo's discipline, +the DR seed doc, D-108/D-104, Section 8's failover skeleton, and the +tooling gap register), reviewed before this commit -- repo-lint was clean +on the first pass. + +## Item + +### 1. DOCFIX-158 -- `runbooks/dc-dc-phase5-dr-failover-drill.md` +FILES: `runbooks/dc-dc-phase5-dr-failover-drill.md` (new), `docs/dc-dc- +deployment-workflow.md` (Stage 6 row updated, tooling gap register item 14 +added). + +WHAT: command-level implementation of `docs/dc-dc-buildout-design.md` +Section 8's failover/failback skeleton, staged one-way-then-two-way per +D-108's explicit requirement, plus the D-104 per-DC controller backup/ +restore drill (exercised here, not deferred). + +**Three real bundle gaps found and flagged before any mutation is +proposed** (grepped `bundle.yaml` directly, not assumed): no `cinder- +backup` charm is deployed in either DC's bundle (the DR seed's own text +assumes cinder already exposes a backup-backend endpoint, which is true, +but the separate charm that runs the backup driver isn't there); no +`ceph-rbd-mirror` charm is deployed either; and `ceph-mon`'s existing +`rbd-mirror` endpoint binding is on `storage`, not `replication` +(inherited from the VR0/DC0 seed at commit 8813efc, predates D-108's ULA- +carrier requirement). All three become the runbook's own Step 2 (a gated, +reviewed bundle-relation-binding change with its own changelog/revert +requirement) rather than being silently routed around or left unmentioned. + +**Every realm/zonegroup/zone/pool/endpoint name is an explicit +`` placeholder** -- the runbook is explicit that the +underlying Ceph commands (`radosgw-admin realm/zonegroup/zone create`, +`rbd mirror pool enable/peer bootstrap/promote/demote`) are standard, +well-documented upstream Ceph administration, not invented for this repo, +while the specific names/endpoints/addresses genuinely don't exist yet and +must never be guessed. + +**A negative-control drill precedes the real one:** Step 10.0 runs a +replication-plane-ONLY partition first (DC1 stays otherwise reachable) and +explicitly states the split-brain-avoidance check in Step 10.1 SHOULD +REFUSE to treat that as "DC1 down" -- if it promotes DC2 anyway under a +mere partition, that's flagged as a finding against the drill design, not +a pass. Only after that negative control does Step 10 run the real +hard-down drill. + +RTO/RPO are treated strictly as MEASURED DRILL OUTPUTS throughout (Steps +6, 7, 9, 10.5, 11.5), never asserted as a pre-committed SLA, matching +D-108's own explicit language. + +VERIFIED: `bash scripts/repo-lint.sh` 0 fail, 1 documented legacy warn (no +fixes needed). NOT YET EXECUTED -- no live Ceph cluster this session. + +REVERT: `rm runbooks/dc-dc-phase5-dr-failover-drill.md && git checkout +HEAD~ -- docs/dc-dc-deployment-workflow.md` (safe). + +## Next actionable step + +Once this drill has been run manually end-to-end at least once, the +radosgw multisite/rbd-mirror command sequences it documents are strong +candidates for extraction into real `scripts/` with a test harness, +closing tooling gap register item #5 for real -- explicitly logged as +future work in the runbook's own delivery checklist, not built here (no +live Ceph cluster exists this session to test a script against). diff --git a/docs/changelog-20260709-dc-dc-phase6-runbook.md b/docs/changelog-20260709-dc-dc-phase6-runbook.md new file mode 100644 index 0000000..604d18d --- /dev/null +++ b/docs/changelog-20260709-dc-dc-phase6-runbook.md @@ -0,0 +1,69 @@ +# Changelog 2026-07-09 -- Stage 7 runbook: Designate, COS, Magnum/CAPI (DOCFIX-159) + +No live infrastructure touched -- new runbook document only. Authored by a +delegated subagent (general-purpose, briefed with this repo's discipline, +D-106/D-008/D-019/D-021/D-105/D-046, `bundle.yaml`, `phase-01-bundle- +deploy.md`, and `phase-06/07/08` as primary sources), reviewed before this +commit -- repo-lint was clean on the first pass. + +## Item + +### 1. DOCFIX-159 -- `runbooks/dc-dc-phase6-designate-cos-magnum.md` +FILES: `runbooks/dc-dc-phase6-designate-cos-magnum.md` (new), `docs/dc-dc- +deployment-workflow.md` (Stage 7 row updated). + +WHAT: per-DC Designate reactivation (D-106, superseding D-019/reactivating +D-008), COS observability, and Magnum/CAPI, reusing `phase-06/07/08` by +filename rather than duplicating their content. + +**Quotes, rather than paraphrases, the exact current repo state being +reversed:** `bundle.yaml`'s own D-019 comment line and `phase-01-bundle- +deploy.md`'s "NO designate (D-019)" gate text are both reproduced verbatim +before describing the reversal, so the diff is traceable to real text, not +a vague "add designate" gloss. + +**Correctly identifies the reversal is bigger than one application:** +D-106's own reactivated bootstrap order includes `os-public-hostname` per +API charm as its second step -- directly reversing `bundle.yaml`'s +separate B5 posture ("IP-ONLY -- os-public-hostname dropped on all API +charms... No control-plane DNS dependency"). The runbook treats this as +two coupled changes under one D-106 umbrella (add designate; re-add +os-public-hostname fleet-wide) rather than quietly bundling it into a +one-line description, and proposes -- without silently adopting -- +pushing the new per-DC FQDN literals into new `overlays/dc1-hostnames.yaml` +/`dc2-hostnames.yaml` files (mirroring the existing `octavia-pki.yaml` +overlay pattern) so `bundle.yaml` itself stays DC-agnostic. + +**D-046's domain-setup trap is quoted verbatim twice** (a top-of-file +`!!!` callout and again inline at the Magnum step) -- "magnum reports +active / 'Unit is ready' regardless of whether it has run" -- with an +explicit instruction that this re-run/re-assertion applies even when this +whole Stage 7 pass is itself a redo after an earlier Stage 4-6 rebuild. + +**COS deployment mechanism explicitly NOT invented:** D-105 rules only the +SCOPE (per-DC, in-scope, no Office1 roll-up); no decision in this repo +specifies whether COS means the `cos-lite` k8s bundle or machine-charm- +native observability. The runbook presents both real options with their +tradeoffs and defers the choice to the operator at execution time, logged +as a real architectural decision rather than picked for them. + +**DC-parameterization notes are traced, not assumed:** confirmed by +reading which phase-06/07/08 scripts actually source `scripts/lib-net.sh` +(`phase-06-k8s-bootstrap.sh`, `phase-06-kubeconfig-gate.sh`, both via +`KEYSTONE_VIP_DEFAULT`) and require `lib_net_select_dc "$DC"` first, versus +which literals those runbooks already resolve dynamically per-run and need +no further changes. + +VERIFIED: `bash scripts/repo-lint.sh` 0 fail, 1 documented legacy warn (no +fixes needed). NOT YET EXECUTED. + +REVERT: `rm runbooks/dc-dc-phase6-designate-cos-magnum.md && git checkout +HEAD~ -- docs/dc-dc-deployment-workflow.md` (safe). + +## Next actionable step + +This is the last of the seven Stage 1-7 runbooks (tooling gap register +item #9 now CLOSED for authoring). Remaining work is the adversarial +review pass across everything built this session and the final +consolidation of the workflow doc, gap register, companion Artifact, +changelogs, and session ledger. diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md index 27c207f..6d9cca1 100644 --- a/docs/dc-dc-deployment-workflow.md +++ b/docs/dc-dc-deployment-workflow.md @@ -86,9 +86,9 @@ | **Gate** | Nodes deployed; six planes present per node with correct fabrics/VLANs; provider NIC raw; PXE (v4) working; per-DC mirror reachable; NTP from edge working. | | **Owns** | D-103, D-100 (br-ex), D-107 (mirror + NTP). | | **Reuse vs new** | PARTIAL reuse. `runbooks/phase-00-teardown-maas-reset.md` carries the MAAS commission/deploy + br-ex discipline this repo already validated for a single rack; the per-DC multi-rack registration is new, the deploy mechanics are precedent. | -| **Authoring status** | No dc-dc runbook yet; adapt phase-00's MAAS section per-DC. | +| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase3-maas-enlist-deploy.md`.** Adapts `phase-00-teardown-maas-reset.md`'s validated single-rack commission/deploy/carve mechanics per DC, run once per `$DC` (dc1/dc2). Requires `lib_net_select_dc "$DC"` first (DOCFIX-151); explicitly documents that `lib_hosts_select_dc "$DC"` is EXPECTED to fail for both DCs until this stage's own real host enrollment populates `lib-hosts.sh` -- a deliberate chicken-and-egg, not a bug. Surfaces 7 real gaps in a named section (no OpenTofu module stands up a per-DC MAAS rack controller VM itself; `reenroll-hosts.sh`/`carve-host-interfaces.sh`/`phase-00-maas-standup.sh` are not `$DC`-parameterized; node count/sizing undecided; DC2 additionally blocked at the selector layer; no identified owner for the per-DC artifact mirror; MAAS status for VM-host-discovered domains unconfirmed; `HOST_TAG`/fabric-name literals are DC0-specific). NOT YET EXECUTED. | -**State:** NOT STARTED. +**State:** RUNBOOK WRITTEN, NOT YET EXECUTED. --- @@ -119,9 +119,9 @@ | **Gate** | One-way drill clean (RTO/RPO measured); then two-way; controller restore drill (D-104) exercised; RTO recorded as a measured output, not an SLA. | | **Owns** | D-108 (replication), D-104 (control-plane backup/restore in the drill). | | **Reuse vs new** | Ground truth: `docs/dc-dc-replication-DR-seed.md` (the original Tier-A/bidirectional/cinder-backup+rbd-mirror decision) -- D-108 corrected its carrier from the seed's v4 replication plane to IPv6-only ULA. The failover/failback drill skeleton is already drafted (buildout design Section 8): confirm DC1 down -> promote Glance -> restore Cinder -> rebuild instances -> record RTO/RPO; failback is split-brain-safe (no auto-resume, controlled window to flip primary back). | -| **Authoring status** | Skeleton exists; needs a runbook with command-level steps + harness. | +| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase5-dr-failover-drill.md`.** Command-level implementation of Section 8's failover/failback skeleton, staged one-way-then-two-way per D-108's own requirement, plus the D-104 controller backup/restore drill. Found and flagged three real bundle gaps while writing it (grepped `bundle.yaml`): no `cinder-backup` charm deployed, no `ceph-rbd-mirror` charm deployed, and `ceph-mon`'s existing `rbd-mirror` endpoint binding is on `storage` instead of `replication` (inherited from the VR0/DC0 seed, predates D-108) -- all three become this runbook's own first gated mutation (Step 2), not silently routed around. Every realm/zone/pool/endpoint name is an explicit `` placeholder. Includes a negative-control sub-drill (sever only the replication link first, confirm the split-brain check correctly refuses to promote) before the real hard-down drill. NOT YET EXECUTED -- no live Ceph cluster this session (harness for gap #5's scripts is future work, logged, not built). | -**State:** NOT STARTED. +**State:** RUNBOOK WRITTEN, NOT YET EXECUTED. --- @@ -134,9 +134,9 @@ | **Gate** | Designate resolving A + AAAA; FQDN-SAN certs valid (D-019/D-021 root cause closed); COS scraping; a CAPI workload cluster comes up per DC. | | **Owns** | D-106 (Designate), D-105 (CAPI/COS). | | **Reuse vs new** | HIGH REUSE for CAPI/Magnum: `runbooks/phase-06-incloud-mgmt-cluster.md` + `phase-07-conductor-graft.md` + `phase-08-workload-cluster-acceptance.md` are direct precedent, per-DC. Designate is a REVERSAL, not new build: VR0 DC0 deferred it (D-019, "DNS scope reduction"); D-106 reactivates it (supersedes D-019, reactivates the original D-008 bootstrap order) -- read D-008 and D-019 together before writing this stage's runbook, the static-`/etc/hosts` workaround they document goes away. | -| **Authoring status** | No dc-dc runbook yet; adapt phase-06/07/08 per-DC, write Designate reactivation fresh. | +| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase6-designate-cos-magnum.md`.** Quotes `bundle.yaml`'s and `phase-01-bundle-deploy.md`'s actual current "NO designate (D-019)" text verbatim rather than paraphrasing, then the D-106 reversal against it. Flags that the reversal is bigger than "add designate" -- D-106's bootstrap order also reactivates `os-public-hostname` per API charm, reversing the bundle's separate B5 IP-only posture; proposes (not silently adopts) pushing per-DC hostname literals into new `overlays/dc1-hostnames.yaml`/`dc2-hostnames.yaml` files to keep `bundle.yaml` itself DC-agnostic, mirroring the existing octavia-pki overlay pattern. D-046's domain-setup trap quoted verbatim in a top callout and again inline. COS deployment mechanism explicitly left as an operator decision point (D-105 rules only the scope) rather than invented. Reuses `phase-06/07/08` by filename with concrete DC-parameterization notes (confirmed via tracing which scripts source `lib-net.sh`). NOT YET EXECUTED. | -**State:** NOT STARTED. +**State:** RUNBOOK WRITTEN, NOT YET EXECUTED. --- @@ -303,18 +303,60 @@ behind D-101/D-102 is needed by a future session, it currently is NOT retrievable from the repo alone. Not a blocker for Stage 0 ratification of what IS written, but a real content gap if finer detail is needed later. -9. **Runbook debt.** Confirmed zero DC-DC-specific runbooks exist. Stage 1, 2, 3, - 6 need net-new runbooks; Stage 5 and 7 need an adaptation pass over - `phase-01..08` / `phase-06/07/08` (Stage 5's pass is gated on #1 above). +9. **Runbook debt -- CLOSED 2026-07-09.** All seven Stage 1-7 runbooks are now + written (`runbooks/dc-dc-phase0-vcloud-prep.md` through `dc-dc-phase6- + designate-cos-magnum.md`), Stage 1 authored directly, Stages 2/3/4/6/7 + delegated to parallel subagents and Stage 5 authored directly, all + reviewed and repo-lint clean. NONE have been executed against real + infrastructure yet -- this was a prep-only session (the operator was away + from the workstation, not the vcloud host). Each runbook is explicit about + what it can and cannot close given today's real state (NetBox literals, + node sizing, and several other decisions remain genuinely open -- see + each runbook's own GATE section for the honest per-stage breakdown). 10. **Harness debt.** This repo's own delivery rule -- no script change ships without its `tests//run-tests.sh` harness -- applies to everything in #2-7 above. None exist yet because none of those scripts exist yet; expect roughly 5-6 new test suites alongside whatever gets built. -11. **Decision sub-items still "leaning," not ruled**, that block the runbooks - depending on them even after Stage 0 closes the big items: exact netem - parameters (D-100), exact NetBox literals for DC2/ULA/GUA (D-101), whether - D-102 folds into D-101, COS per-DC-vs-Office1-rollup (D-105), mirror sync - topology (D-107). +11. **Decision sub-items still genuinely open** (the big items are RULED per + Stage 0; these specific literals/params are not): exact netem parameters + (D-100), exact NetBox literals for DC2/ULA/GUA (D-101). +12. **No Office1-local virtual network is modeled anywhere in `opentofu/`** + (found 2026-07-09 while drafting the Stage 2 runbook). `main.tf` wires + `office1_pool_path` (storage only) and the three inter-site mesh links + (management-traffic-only per D-100's own sub-item ruling) -- nothing the + three Office1 service VMs (MAAS-region, NetBox, GitBucket), or the vcloud + host's own management interface, could actually attach to as a shared + local network. Blocks designing real `cloudinit-vm` `network_config` + content for any of the three VMs. Needs either a new `modules/dc-planes`- + shaped Office1-local network, or a deliberate, documented reuse of an + existing host bridge -- not decided yet. +13. **The D-101 IPv6 family-matrix overlay does not exist** (found 2026-07-09 + while drafting the Stage 5 runbook). Which planes need ULA-only vs. + dual-stack vs. GUA addressing is fully specified by D-101 (Section 2 of + the buildout design), but the actual Juju charm config option names + needed to configure that per application (`ceph-mon`, `ovn-central`, + `octavia`, the HA-cluster/VIP charms, etc.) have not been confirmed + against any charm's real current schema. Blocks Stage 5's "Ceph-over-v6 + and geneve-over-v6 verified" gate criterion. D-109's IPv6-SAN cert- + issuance charm support is a related, separately-unverified assumption. +14. **Three real `bundle.yaml` gaps for D-108's DR mechanism** (found + 2026-07-09 while drafting the Stage 6 runbook, by grepping the actual + bundle): no `cinder-backup` charm deployed; no `ceph-rbd-mirror` charm + deployed; `ceph-mon`'s existing `rbd-mirror` endpoint binding is on + `storage`, not `replication` (inherited from the VR0/DC0 seed, predates + D-108's ULA-carrier requirement). All three are Stage 6's own first + gated mutation (its Step 2), not fixed by this delivery -- no live Ceph + cluster exists this session to test the fix against. +15. **Seven real MAAS/multi-rack gaps found 2026-07-09 while drafting the + Stage 4 runbook** (full detail in that runbook's own "Known gaps" + section): no OpenTofu module stands up a per-DC MAAS rack controller VM + itself (only registration, `modules/maas-vm-host`, exists); `reenroll- + hosts.sh`/`carve-host-interfaces.sh`/`phase-00-maas-standup.sh` are not + `$DC`-parameterized (DOCFIX-151 only touched the two library files); + per-DC node count/sizing is an unmade Phase-0 decision; the per-DC + artifact mirror (D-107) has no identified owning stage/module; the MAAS + status a VM-host-discovered (vs. PXE-enlisted) machine lands in is + unconfirmed; `HOST_TAG`/boot-fabric-name literals are DC0-specific. --- @@ -325,7 +367,7 @@ | Design (`dc-dc-buildout-design.md`) | DONE, PROPOSED | | Decisions (D-100..D-110) | **ADOPTED 2026-07-09** (D-102 merged into D-101; D-109 standalone) -- Stage 0 CLEARED | | DR mechanism seed | DONE (`dc-dc-replication-DR-seed.md`), superseded-carrier corrected into D-108 | -| Stage 0-7 runbooks | Stage 0 DONE (ratification). **Stage 1 WRITTEN 2026-07-09** (`runbooks/dc-dc-phase0-vcloud-prep.md`), not yet executed. Stages 2-7 NOT YET WRITTEN -- see Tooling gap register above for the itemized breakdown | +| Stage 0-7 runbooks | Stage 0 DONE (ratification). **Stages 1-7 ALL WRITTEN 2026-07-09** (gap #9 CLOSED) -- `runbooks/dc-dc-phase0-vcloud-prep.md` through `dc-dc-phase6-designate-cos-magnum.md`. NONE executed against real infrastructure (prep-only session). See gaps #12-15 above for real design gaps the authoring pass surfaced. | | `opentofu/` (networks/pools/node-VM PXE/cloud-init-VM patterns) | SCAFFOLDED 2026-07-09, UNVALIDATED (no `tofu` binary available this session) -- see gap #2 and `opentofu/README.md` | | OPNsense image+config mechanism (`modules/opnsense-edge`) | BUILT 2026-07-09, UNVALIDATED (no `tofu`/ISO-tool/real boot to confirm), see `opentofu/README.md` | | OPNsense config.xml design (`templates/opnsense-config.xml.tmpl`) | BUILT 2026-07-09, renderer fully tested (8/8) -- several tokens still pending Stage 0 ratification or a real boot, see `opentofu/templates/README.md` | diff --git a/docs/session-ledger.md b/docs/session-ledger.md index cc5afe7..fbd1ea2 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -21,8 +21,8 @@ ## Machine-derived (re-seed from `scripts/ledger-scan.sh`; do not hand-edit) -_As of 2026-07-09 scan (post-DOCFIX-156, Stages 2/3/5 runbooks written); -re-run the scan to refresh:_ +_As of 2026-07-09 scan (post-DOCFIX-159, ALL seven Stage 1-7 runbooks +written); re-run the scan to refresh:_ - **PROPOSED / OPEN decisions:** D-068 (Vault substrate hardening, Roosevelt -- 1.16 ruled out per amendment; off-EOL path OPEN), D-071 (routine update cadence + @@ -37,8 +37,8 @@ - **OPEN security rows:** SEC-001 (rotate at v1 close, re-ruled 2026-07-06), SEC-003 (custodians + second-person unseal -- DEFERRED; keeps d011-06 MANUAL), SEC-004 (flip repo to private at v1 close -- deferral reaffirmed). -- **Next-free numbers:** D = 111, DOCFIX = 157, BUNDLEFIX = 012. - (Seeded from the 2026-07-09 scan, post DOCFIX-156.) +- **Next-free numbers:** D = 111, DOCFIX = 160, BUNDLEFIX = 012. + (Seeded from the 2026-07-09 scan, post DOCFIX-159.) --- @@ -1025,9 +1025,41 @@ unconfirmed) and D-109's IPv6-SAN cert-issuance charm support (unverified assumption). All three: repo-lint 0 fail/1 documented warn; NOT YET EXECUTED (prep-only - session, no live infrastructure reachable). Stages 4, 6, 7 were also - delegated to parallel subagents and completed around the same time -- - see their own changelog entries once reviewed and committed. + session, no live infrastructure reachable). +- **DONE -- Stages 4, 6, 7 runbooks WRITTEN 2026-07-09 -- ALL SEVEN Stage + 1-7 runbooks now complete, tooling gap register item #9 CLOSED:** + - **Stage 4** (changelog `docs/changelog-20260709-dc-dc-phase3-runbook.md`, + DOCFIX-157): `runbooks/dc-dc-phase3-maas-enlist-deploy.md`. Mandates + `lib_net_select_dc "$DC"` then HONESTLY documents `lib_hosts_select_dc + "$DC"` as expected to fail for both DCs until this stage's own + enrollment populates real host data -- a deliberate chicken-and-egg. + Names 7 real gaps in its own section (no OpenTofu module stands up a + per-DC MAAS rack controller VM; three scripts not `$DC`-parameterized; + node sizing undecided; DC2 blocked at the selector layer; no owner for + the per-DC mirror; VM-host-discovery MAAS status unconfirmed; DC0- + specific tag/fabric literals). + - **Stage 6** (changelog `docs/changelog-20260709-dc-dc-phase5-runbook.md`, + DOCFIX-158): `runbooks/dc-dc-phase5-dr-failover-drill.md`. Found three + real `bundle.yaml` gaps by grepping it directly (no `cinder-backup` + charm, no `ceph-rbd-mirror` charm, `ceph-mon`'s `rbd-mirror` binding on + the wrong plane) and makes fixing them the runbook's own first gated + mutation. Includes a negative-control drill (partition-only, expected + to correctly refuse promotion) before the real hard-down drill. Every + Ceph object name is an explicit placeholder. + - **Stage 7** (changelog `docs/changelog-20260709-dc-dc-phase6-runbook.md`, + DOCFIX-159): `runbooks/dc-dc-phase6-designate-cos-magnum.md`. Quotes + the bundle's actual current "NO designate" text verbatim before + describing the reversal; correctly identifies D-106's reactivation + also reverses the bundle's separate B5 os-public-hostname-dropped + posture (two coupled changes, not one); proposes per-DC hostname + overlays to keep `bundle.yaml` DC-agnostic; quotes D-046's domain-setup + trap verbatim twice; leaves COS's deployment mechanism as an explicit + operator decision point rather than inventing one. + All three: repo-lint 0 fail/1 documented warn; NOT YET EXECUTED. Four new + tooling-gap-register items (#12-15) added for the cross-cutting findings + this whole runbook-authoring pass surfaced (Office1-local network + missing, IPv6 family-matrix overlay missing, D-108 bundle gaps, Stage-4 + MAAS/multi-rack gaps) -- see `docs/dc-dc-deployment-workflow.md`. diff --git a/runbooks/dc-dc-phase3-maas-enlist-deploy.md b/runbooks/dc-dc-phase3-maas-enlist-deploy.md new file mode 100644 index 0000000..16efa97 --- /dev/null +++ b/runbooks/dc-dc-phase3-maas-enlist-deploy.md @@ -0,0 +1,506 @@ +# DC-DC Phase 3 -- MAAS enlist / commission / deploy (per DC) (Stage 4) + +Bring the per-DC node VMs that Stage 3 (`docs/dc-dc-deployment-workflow.md` -- +"Stage 3 -- Phase 2: OpenTofu builds each DC substrate") caused MAAS to +DISCOVER through to a deployed OS, ready for Juju (Stage 5 / Phase 4). Run +this runbook ONCE PER DC: set `DC=dc1` or `DC=dc2` for the whole session and +do not mix the two DCs' commands in one shell. This is the FIRST DC-DC +runbook whose subject is MAAS itself in the multi-rack VR1 shape -- everything +before it (Stage 0-3) either ratified decisions or built the substrate MAAS +now sits on top of. + +**Governing docs:** `docs/dc-dc-buildout-design.md` Section 4 "### Phase 3" +(goal/build/gate, quoted in the GATE section below) and Section 5 +(OpenTofu/MAAS/Juju boundary: "MAAS: enlist (discover) / commission / deploy +/ power / release of node VMs. Does not compose. One region (Office1) + one +rack per DC (virsh VM-host)."); `docs/dc-dc-deployment-workflow.md` Stage 4 +row (State: NOT STARTED as of this writing; "Reuse vs new: PARTIAL reuse... +the per-DC multi-rack registration is new, the deploy mechanics are +precedent"); `docs/design-decisions.md` **D-103** (lifecycle seam: MAAS owns +commission/deploy/power/release, does NOT compose), **D-100** (br-ex / +provider-NIC raw discipline, Pattern A), **D-107** (per-DC airgap mirror + +edge NTP). `runbooks/phase-00-teardown-maas-reset.md` Steps 5-7 are the +validated single-rack precedent this runbook adapts -- read it first if you +have not; this doc assumes you have and does not re-derive what it already +established. + +Decisions this runbook owns: **D-103**, **D-100** (br-ex), **D-107** (mirror ++ NTP). + +--- + +## Precondition -- Stage 3's gate must already be green (READ-ONLY) + +Stage 3's own runbook does not exist yet as of this writing (workflow doc: +"Runbook not started"; state NOT STARTED) -- if it has been written and run +by the time you execute this, confirm its EXIT GATE against +`docs/dc-dc-buildout-design.md` Section 4 Phase 2 gate text: "MAAS rack +controller per DC up; MAAS discovers the OpenTofu-created node VMs; edge +routing + simulated ISP uplink up; netem parameters applied and measured." +If that gate is not green for `$DC`, STOP here -- do not attempt to +stand up a rack controller, register a VM host, or create node VMs from +inside THIS runbook. That is Stage 3's job (D-103: OpenTofu creates the node +VMs and registers the libvirt host as a MAAS VM host); this runbook only +picks up from "MAAS already sees them." + +**CHECK (read-only) -- confirm the rack controller MAAS thinks it has for this DC** +```bash +maas "${MAAS_PROFILE:-admin}" rack-controllers read | jq -r '.[] | "\(.hostname)\t\(.system_id)"' +``` +Identify, by eye, which entry (if any) is `$DC`'s rack controller -- there is +no established naming convention yet to grep for (Stage 3 has not run; do +not assume a name like `dc1-rack`). If nothing plausible appears, Stage 3's +gate is not met -- stop and go finish Stage 3. + +--- + +## Known gaps flagged before you start (not fixed by this runbook) + +These are real, current repo gaps surfaced while adapting phase-00's +single-rack sequence to the multi-rack VR1 shape. Each is a FINDING to log +(changelog / D-NNN / DOCFIX candidate per this repo's discipline) at the +point it is actually hit, not something to route around silently here. + +1. **No OpenTofu module stands up a per-DC MAAS rack controller VM itself.** + `opentofu/README.md`'s built-module list has `maas-vm-host` (registers an + existing libvirt/virsh connection as a VM host WITH a MAAS rack), but + nothing that creates the rack-controller service VM the buildout design's + Section 5 says each DC needs ("one region (Office1) + one rack per DC"). + If Stage 3's precondition check above finds no rack controller for `$DC`, + that absence is this gap manifesting, not a Stage 4 problem to solve here. +2. **`scripts/reenroll-hosts.sh`, `scripts/carve-host-interfaces.sh`, and + `scripts/phase-00-maas-standup.sh` do not accept a `$DC` argument and do + not call `lib_net_select_dc`/`lib_hosts_select_dc`.** Only `lib-net.sh` + and `lib-hosts.sh` themselves gained the selector functions (DOCFIX-151, + 2026-07-09). This runbook therefore describes ADAPTING each script's + MECHANICS by hand per DC (the CLI calls it makes, in the order it makes + them) rather than literally invoking e.g. `bash scripts/reenroll-hosts.sh + dc1` -- that invocation does not exist. Parameterizing these three + scripts for `$DC` directly is a real follow-up (log it), not done here. +3. **Node count and sizing per DC is an unmade Phase-0 decision.** + `opentofu/README.md`'s `node-vm` module note: "node count/memory/vcpu/disk + sizing is a Phase-0 host/disk-budget decision... that hasn't been made -- + call the module once real values exist." Do NOT assume DC0's four hosts + (`openstack0-3`) carry over to `$DC` -- discover however many nodes Stage + 3 actually created, live, in Step 2 below. +4. **DC2 is additionally blocked at the network-selector layer.** + `lib_net_select_dc dc2` FAILS LOUD today (D-101's NetBox-literals open + item, tooling gap register #3 -- no CIDRs assigned yet for DC2). This + runbook cannot proceed past Step 1 for `DC=dc2` until that closes. `dc1` + is unblocked at this layer (D-101: DC1 inherits DC0's v4 layout + unchanged, so `lib_net_select_dc dc1` is a documented no-op). +5. **The per-DC artifact mirror (D-107) has no identified owning + stage/module in this repo as surveyed for this runbook.** Stage 2's build + list (Office1 headend) mentions GitBucket (the REPO mirror) and NetBox -- + neither is the node-package/apt/snap/charmhub/image mirror D-107 + describes. Before Step 7's mirror-reachability check, CONFIRM the mirror + actually exists and where; if it does not, that is a gap to log against + Stage 2 or a new stage, not something to invent a reachability check for. +6. **The MAAS status a VM-host-discovered (not PXE-enlisted) pre-existing + domain lands in is not confirmed anywhere in this repo.** Phase-00's + `reenroll-hosts.sh` header states PXE-enlisted hosts "auto-commission on + create" -- but Stage 3's node VMs are discovered via `maas_vm_host` pod + scan of already-existing libvirt domains, a different MAAS code path. + Step 2 below measures the real status live rather than assuming either + "New" or "auto-Ready." +7. **`lib-hosts.sh`'s `HOST_TAG`/boot-fabric-name conventions (`openstack`, + `2_metal`) are DC0-specific literals.** Whether `$DC`'s bundle placement + tag and PXE fabric carry the same names is not established -- confirm + live per DC rather than assuming reuse of DC0's literal strings. + +--- + +## Sequence +``` +0. Precondition confirm Stage 3 gate (above) (read-only) +1. DC selector lib_net_select_dc "$DC"; note lib_hosts_select_dc fails (read-only) +2. Discover find $DC's node VMs live in MAAS (read-only) +3. Commission per discovered node [MUTATION: gated] +4. Deploy per discovered node [MUTATION: gated] +5. Pattern A carve (adapted) per node, RAW provider NIC + VLAN-103 stack [MUTATION: gated] +6. PXE / boot-fabric verify confirm v4 PXE + fabric landing (read-only) +7. Mirror + NTP verify per-DC mirror reachable; edge NTP (read-only) +8. Topology check (adapted) phase-00-maas-standup.sh mechanics, per DC (read-only) + -> EXIT GATE -> Stage 5 (Juju controller + bundle, per DC) +``` + +--- + +## Step 1 -- DC selector (READ-ONLY) + +Every session/script that sources `scripts/lib-net.sh` for this stage calls +the selector ONCE, immediately after sourcing, before using any flat plane +variable: + +```bash +SCRIPT_DIR="$(pwd)/scripts" +. "$SCRIPT_DIR/lib-net.sh" +. "$SCRIPT_DIR/lib-hosts.sh" +lib_net_select_dc "$DC" # dc1: documented no-op (D-101 inherits DC0 v4). + # dc2: FAILS LOUD today -- gap #3, stop here for dc2. +``` + +Then, honestly, also call the host selector and expect it to fail for +BOTH DCs right now: + +```bash +lib_hosts_select_dc "$DC" # EXPECTED TO FAIL for dc1 AND dc2 until THIS + # stage's own completion populates real per-DC + # host data into lib-hosts.sh (Step 5 below). +``` + +This is a real, expected chicken-and-egg, not a bug to work around: the +selector exists specifically to PREVENT a `$DC` script from silently reusing +`HOSTS`/`HOST_OCTET`/`HOST_BOOT_MAC`/`VIRSH_POWER_ADDRESS` from DC0's real, +already-enrolled hosts for a different DC's different VMs. Until this +stage's enrollment produces `$DC`'s own real hostnames, MACs, and +system_ids, there is nothing correct for `lib_hosts_select_dc "$DC"` to +select -- so it fails, on purpose, for both `dc1` and `dc2`, and will keep +failing until Step 5's data goes into `lib-hosts.sh` for real. Do not +comment out the call, do not pre-populate placeholder host data to make it +pass -- populating it with REAL data once Step 5 has measured it is this +stage's own delivery item (see the checklist). + +Because `lib_hosts_select_dc` cannot yet be relied on, every step below +resolves `$DC`'s host identity LIVE from MAAS (by whatever the discovery in +Step 2 actually finds), never from `lib-hosts.sh`'s DC0 maps. + +--- + +## Step 2 -- Discover `$DC`'s node VMs live in MAAS (READ-ONLY) + +There is no fixed hostname list for `$DC` (gap #3 above) and no established +zone/pool/tag naming convention yet (Stage 3 has not run to record one). +Confirm, from what Stage 3 actually did for `$DC` (its as-executed log / +session notes), which of these MAAS groupings it used, then discover +against THAT -- do not guess a name like `"dc1"`: + +**CHECK -- by zone (if Stage 3 recorded a zone name for `$DC`)** +```bash +maas "${MAAS_PROFILE:-admin}" zones read | jq -r '.[] | "\(.name)"' +# then, once you have confirmed the real zone name for $DC (call it Z below): +maas "${MAAS_PROFILE:-admin}" machines read \ + | jq -r --arg z "Z" '.[] | select(.zone.name==$z) | "\(.hostname)\t\(.system_id)\t\(.status_name)"' +``` + +**CHECK -- by resource pool (if Stage 3 used a pool instead)** +```bash +maas "${MAAS_PROFILE:-admin}" resource-pools read | jq -r '.[] | "\(.name)"' +maas "${MAAS_PROFILE:-admin}" machines read \ + | jq -r --arg p "P" '.[] | select(.pool.name==$p) | "\(.hostname)\t\(.system_id)\t\(.status_name)"' +``` + +**Cross-check -- by fabric/VLAN** (independent confirmation: D-100 makes +each DC's metal-admin/PXE plane its own isolated segment, so `$DC`'s +machines' boot interfaces should all land on the SAME fabric, and that +fabric should be DIFFERENT from DC0's `2_metal` and from the other DC's): +```bash +maas "${MAAS_PROFILE:-admin}" interfaces read \ + | jq -r '.[] | select(.type=="physical") | "\(.mac_address)\t\(.vlan.fabric)"' +``` +Run this per candidate system_id from the zone/pool query above; every node +claimed for `$DC` should share one fabric name, and that name should not be +`2_metal` (DC0's) nor match the other DC's. If the grouping mechanism +disagrees with the fabric cross-check (e.g. a machine tagged into `$DC`'s +pool but sitting on a different DC's fabric), STOP -- that is a real +enrollment inconsistency, not something to paper over by picking whichever +signal you prefer. + +**GATE (informal, before mutating):** you have a concrete, confirmed list of +system_ids that both (a) belong to `$DC` by whatever grouping Stage 3 used +and (b) share one boot fabric distinct from DC0's and the peer DC's. Record +that list -- Steps 3-5 iterate over it. Record the count too; it is +whatever Stage 3 actually created (gap #3), not an assumed 4. + +--- + +## Step 3 -- Commission each discovered node [MUTATION: gated, per node] + +Phase-00 precedent (`reenroll-hosts.sh`): freshly PXE-enlisted hosts +auto-commission. Stage 3's nodes arrived via `maas_vm_host` pod discovery of +pre-existing libvirt domains, a different code path whose landing status +this repo has not previously observed (gap #6) -- Step 2's `status_name` +column tells you what you actually got. If a node already reads `Ready`, +skip it (idempotent). If it reads `New`, commission it explicitly: + +**CHECK (read-only) -- current status, one more time, immediately before mutating** +```bash +maas "${MAAS_PROFILE:-admin}" machine read | jq -r '.status_name' +``` + +> CAUTION: commissioning boots the machine and runs MAAS's commissioning +> scripts against it -- gated, one node at a time, re-read status after each. + +**RUN -- per node that reads `New`** +```bash +maas "${MAAS_PROFILE:-admin}" machine commission +``` + +**VERIFY -- poll to `Ready`** +```bash +maas "${MAAS_PROFILE:-admin}" machine read | jq -r '.status_name' +``` +Expect `Ready` within a reasonable commissioning window (phase-00 uses a +~20 min poll budget as a reference, not a hard rule). If a node lands +anywhere other than `Ready`/`Failed commissioning` after a full poll cycle, +treat the unexpected status itself as gap #6 manifesting -- log the actual +status observed rather than assuming it will resolve. + +--- + +## Step 4 -- Deploy each node [MUTATION: gated, per node] + +Do not pass a hardcoded `distro_series`/`osystem` -- no script in this repo +sets one (checked: `reenroll-hosts.sh`, `phase-00-maas-standup.sh`, and the +rest of `scripts/` are all silent on OS release, meaning DC0's own hosts +deploy off MAAS's region-level default). Confirm `$DC`'s rack/region default +live before deploying, rather than assuming it matches DC0's: + +**CHECK -- MAAS's configured default OS release** +```bash +maas "${MAAS_PROFILE:-admin}" maas get-config name=default_distro_series +``` +If `$DC` needs a different release than the default (a real design +decision, not assumed here), that override is itself a value to record and +justify, not silently pass. + +> CAUTION: deploy installs the OS and reboots the machine -- gated, one node +> at a time. + +**RUN -- per Ready node** +```bash +maas "${MAAS_PROFILE:-admin}" machine deploy +``` + +**VERIFY** +```bash +maas "${MAAS_PROFILE:-admin}" machine read | jq -r '.status_name' +``` +Expect `Deployed`. + +**Apply the deploy-placement tag** (phase-00 precedent: `reenroll-hosts.sh`'s +"Apply MAAS tag ... (deploy placement prereq)" step -- without a tag the +bundle cannot bind a unit to the machine via `constraint tags=...`). +`lib-hosts.sh`'s `HOST_TAG="openstack"` is DC0's literal (gap #7) -- confirm +whether `$DC`'s Stage-5 bundle will constrain on the same tag name or a +DC-specific one before tagging; do not assume reuse without checking +whatever Stage 5's authoring records. +```bash +maas "${MAAS_PROFILE:-admin}" tags read | jq -r '.[].name' +# create/confirm the confirmed tag name, then: +maas "${MAAS_PROFILE:-admin}" tag update-nodes "" add= +``` + +--- + +## Step 5 -- Pattern A interface carve, adapted per node [MUTATION: gated, per node] + +`scripts/carve-host-interfaces.sh` is the validated precedent (D-060 Pattern +A revert, D-100 br-ex/raw-NIC discipline) -- read its header tree again: + +``` +enp1s0 --> br-ex (OVS bridge) + STATIC provider-public (Pattern A; MAAS builds + the OVS bridge; ovn-chassis + consumes it by MAC, does not + build its own -- D-100) +enp7s0 --> br-metal (bridge) + STATIC metal-admin + br-metal.103 (VLAN, VID 103) + --> br-internal (bridge) + STATIC metal-internal +enp8/9/10s0 raw + STATIC data-tenant / storage / replication +enp11s0 idle (ex-lbaas; no VR1 analog needed unless re-added deliberately) +``` + +The VLAN-103 metal-internal stack is a REAL, already-established repo +constant, safe to cite verbatim (`scripts/lib-net.sh`): `METAL_INTERNAL_VID="103"`, +`METAL_INTERNAL_IFACE="br-internal"`, `METAL_INTERNAL_CIDR="10.12.12.0/22"` (the +last one is DC0/DC1's value only under `lib_net_select_dc dc1`'s no-op -- +confirm the same call resolves the right CIDR for whichever `$DC` you are +running against; it fails loud for `dc2` per gap #4). + +**The script itself is not yet DC-parameterized (gap #2).** It resolves a +per-host static octet from `lib-hosts.sh`'s `HOST_OCTET[]`, keyed +by hostname -- for a `$DC` node whose hostname is not yet in that map (true +for every DC1/DC2 node right now), do NOT run +`bash scripts/carve-host-interfaces.sh --apply` as-is: the +octet lookup would return empty and the script would either fail cleanly +(preferred) or, worse, carve an interface with no static address. Before +carving each node: + +1. Assign it a real octet within whatever per-DC octet band Stage 1/3 + planning actually recorded (not invented here -- if no band was recorded, + that is itself a finding to log before proceeding). +2. Resolve its system_id and boot MAC live (`maas machine read`, + `maas interfaces read` -- same pattern as `host_sysid()` and + `HOST_BOOT_MAC` in `lib-hosts.sh`, just not yet captured there for `$DC`). +3. Record hostname -> octet -> boot MAC -> system_id as you measure each, + per node, THIS SESSION -- this record is exactly what Step 5's completion + feeds back into `lib-hosts.sh` as a real `$DC` block (delivery checklist + item below), not a placeholder. +4. Only then either (a) temporarily add that mapping to a local, uncommitted + copy of `lib-hosts.sh` so `carve-host-interfaces.sh --apply` + resolves correctly, or (b) run the equivalent `maas` CLI calls the script + itself issues (`link-subnet`, bridge/VLAN create) by hand with the same + measured values. Either way, re-read the resulting interface tree per + node before moving to the next (per the script's own "apply ONE host at a + time" discipline). + +> CAUTION: mutates MAAS interface definitions on each node -- requires the +> node in `Ready`... but Step 4 already deployed it. **Sequencing note:** +> phase-00's precedent carves BEFORE deploy (Ready-only; deploy applies the +> carved netplan on boot). If Stage 3's discovery+commission flow leaves +> `$DC`'s nodes at `Ready` before you reach this step, carve THEN deploy +> (matching phase-00's order) rather than the Step-3/4/5 ordering written +> above -- reorder Steps 4 and 5 for your actual live sequence and note which +> order you used in the as-executed log; do not silently follow this doc's +> numbering if the real machine states say otherwise. + +**GATE (per node):** `br-ex` (OVS) STATIC on provider-public; `br-metal` +STATIC on metal-admin; `br-internal` (VID 103) STATIC on metal-internal; +data/storage/replication planes raw + STATIC. Provider NIC (`enp1s0`) +carries NO other configuration -- confirm it stayed raw/untagged per D-100 +(MAAS must not bridge the provider NIC itself outside the OVS build ovn- +chassis expects). + +--- + +## Step 6 -- PXE / boot-fabric verify (READ-ONLY) + +**CHECK -- boot NIC landed on `$DC`'s own metal/PXE fabric** +```bash +maas "${MAAS_PROFILE:-admin}" interfaces read \ + | jq -r --arg m "" '.[]|select(.mac_address==$m)|.vlan.fabric' +``` +Expect the SAME fabric name every `$DC` node cross-checked against in Step +2 -- not `2_metal` (DC0's), not the peer DC's fabric. + +**CHECK -- PXE (v4) actually working** +Confirm via the commission/deploy transcripts already captured in Steps 3-4 +(a machine that successfully commissioned and deployed necessarily PXE'd +successfully) rather than re-deriving a separate synthetic PXE test -- +per D-101, PXE is v4-first regardless of the metal-admin dual-stack +amendment, so no v6 PXE path exists to additionally check. + +--- + +## Step 7 -- Per-DC mirror + edge NTP verify (READ-ONLY) + +Per gap #5: CONFIRM the per-DC artifact mirror this D-107 gate item +requires actually exists and where, before running a reachability check +against it. If it does not exist yet, STOP this sub-check, log the gap +(it blocks the "airgap nodes pull only from an in-DC mirror" posture +D-107 requires), and do not fabricate a mirror endpoint to check against. + +**CHECK (once the real mirror address is known) -- reachable from a deployed node** +```bash +# from the deployed node, or via its MAAS-reported address: +curl -sI http:/// | head -1 +``` + +**CHECK -- NTP from `$DC`'s own OPNsense edge (D-107: nodes get chrony from +their DC edge, which syncs upstream; Office1 is NOT in this path)** +```bash +chronyc sources +# or, if chrony is not yet configured on a freshly-deployed node: +timedatectl show -p NTP -p NTPSynchronized +``` +Confirm the source is `$DC`'s own edge address (measured, not assumed) -- +not Office1, not a public pool directly (D-107: the edge is the only +component with controlled internet egress; nodes are airgapped at the node +boundary). + +--- + +## Step 8 -- Topology consistency check, adapted per DC (READ-ONLY) + +`scripts/phase-00-maas-standup.sh` is the validated precedent for "does +MAAS's fabric/VLAN/subnet/space topology match the target plane scheme" -- +but it is hardcoded to DC0/D-052/D-053's literal space names and is not yet +`$DC`-aware (gap #2). Adapt its CHECKS (not its literals) per DC: + +```bash +maas "${MAAS_PROFILE:-admin}" spaces read \ + | jq -r '.[] | "\(.name)\t\([.subnets[]?.cidr] | join(", "))"' | sort +``` +Cross-check the six spaces/CIDRs against whatever `lib_net_select_dc "$DC"` +resolved in Step 1 (a no-op equal to DC0's values for `dc1`; blocked for +`dc2`) -- confirm no DRIFT (a subnet present but bound to the wrong plane or +VID) before treating `$DC` as ready for Stage 5's Juju controller bootstrap. +Do NOT run `phase-00-maas-standup.sh` itself unmodified against `$DC` -- +its target-state literals are DC0's; running it as-is against `$DC` would +either no-op uselessly (dc1, since the CIDRs happen to match) or produce +false DRIFT findings (dc2, once its CIDRs exist and differ from DC0's). + +**Explicitly NOT this stage's job:** `scripts/provider-bundle-check.py` (the +bundle-invariant gate phase-00 runs alongside the topology check) validates +`bundle.yaml`'s DC0-specific chassis MACs, VIP triples, and unit counts. +Stage 5's own authoring status already flags that `bundle.yaml` needs a +per-DC parameterization pass before reuse -- do not run +`provider-bundle-check.py` against `$DC` from this stage; that check belongs +at the START of Stage 5, once that parameterization exists. + +--- + +## GATE (Stage 4 exit condition, per DC -- restated honestly from the buildout design) + +Per `docs/dc-dc-buildout-design.md` Section 4 Phase 3: +- Nodes deployed (Steps 3-4: all discovered `$DC` nodes read `Deployed`). +- The six planes present per node with correct fabrics/VLANs (Step 5's + per-node GATE, Step 8's topology cross-check). +- Provider NIC raw (Step 5 GATE: `enp1s0`/equivalent carries no config + beyond what MAAS's own br-ex build applies). +- PXE (v4) working (Step 6). +- Per-DC mirror reachable from nodes (Step 7 -- CONDITIONAL on gap #5 being + resolved; if the mirror does not exist yet, this GATE bullet is honestly + NOT MET and Stage 4 is not complete for `$DC`, full stop, regardless of + how clean the MAAS-side steps came out). +- NTP from the DC's own OPNsense edge working (Step 7). + +All six true, for every node Step 2 discovered for `$DC` -> Stage 4 complete +for that DC. Run the whole runbook again with the other `$DC` value for the +second DC; the two DCs' completions are independent (D-100: no shared +control plane), so one can be done before the other. + +--- + +## Next + +Stage 5 (`docs/dc-dc-deployment-workflow.md` -- Phase 4: Juju controller + +OpenStack bundle, per DC): bootstrap the per-DC Juju controller onto these +now-deployed machines, parameterize `bundle.yaml` for `$DC` (D-101 family +matrix, D-109 Vault root), and run `provider-bundle-check.py` for the first +time against that parameterized bundle before `preflight.sh`. + +--- + +## Delivery checklist (this repo's standard discipline) + +- [ ] `bash scripts/repo-lint.sh` clean (0 fail) before committing anything + touched while executing this runbook. +- [ ] **Populate `scripts/lib-hosts.sh`'s per-DC host data from this stage's + real enrollment, as its own follow-up delivery** -- the real hostnames, + boot MACs, octets, and a note on how to resolve each node's system_id + live, captured in Steps 2-5 above, go into a new `$DC`-scoped block in + `lib-hosts.sh` (mirroring the existing DC0 arrays' shape, not + overwriting them) so that `lib_hosts_select_dc "$DC"` can go from + FAIL to a real, populated selection. Do not commit this data before it + has actually been measured against a live enrollment. +- [ ] Log gap #1 (no OpenTofu module creates a per-DC MAAS rack controller + VM) against Stage 3 if it was hit as a real blocker, not just noted. +- [ ] Log gap #2 (reenroll-hosts.sh / carve-host-interfaces.sh / + phase-00-maas-standup.sh are not `$DC`-parameterized) as a DOCFIX + candidate once the manual adaptation in Steps 3/5/8 has been run at + least once for real, so the next DC's run (or a redeploy of this one) + has a real script to call instead of hand-adapted CLI. +- [ ] Log gap #5 (no identified owning stage/module for the per-DC artifact + mirror) against Stage 2 (or a new stage) if the mirror was found not + to exist when Step 7 was reached -- this GATE bullet cannot be met + until that closes, so it blocks Stage 4 completion honestly rather + than being paperable-over. +- [ ] Changelog entry for this runbook's first real execution per DC (next + DOCFIX number via `bash scripts/ledger-scan.sh`), noting actual + measured values (node count, hostnames, octets, fabric names, tag + name used) -- these are as-built facts this repo commits, not secrets. +- [ ] `docs/session-ledger.md` updated with the outcome, per DC. +- [ ] `docs/dc-dc-deployment-workflow.md` Stage 4 row and tracker table + updated from NOT STARTED to the honest actual state (may be PARTIAL + -- e.g. DC1 done, DC2 still blocked on gap #4's NetBox literals). diff --git a/runbooks/dc-dc-phase5-dr-failover-drill.md b/runbooks/dc-dc-phase5-dr-failover-drill.md new file mode 100644 index 0000000..79ba623 --- /dev/null +++ b/runbooks/dc-dc-phase5-dr-failover-drill.md @@ -0,0 +1,633 @@ +# DC-DC Phase 5 -- DR wiring and failover drill (Stage 6) + +Wire the Tier-A cross-DC replication mechanism (`docs/dc-dc-replication-DR-seed.md`, +engineered by **D-108**) between DC1 and DC2, stage it one-way, prove it clean, enable +two-way, then run the actual failover and failback drills plus the per-DC Juju controller +backup/restore drill (**D-104**). This is the first DC-DC runbook that deliberately takes a +live DC down (simulated, not physical) as part of the drill itself -- treat Steps 9-11 with +the same seriousness as `phase-00-teardown-maas-reset.md`'s own destructive steps. + +Entry condition: **Stage 5's gate** (`docs/dc-dc-deployment-workflow.md`) has passed for +BOTH DC1 and DC2 independently -- `preflight.sh` PASS before add-model, post-deploy +`cloud-assert.sh --capture` per DC, a controller backup already taken and stored, Ceph-over-v6 +and geneve-over-v6 verified. Do not start this runbook against a DC that hasn't cleared that +gate; whichever sibling runbook carries Stage 5 is the authority for confirming it, not this +document. + +Decisions this runbook executes against: **D-108** (cross-DC replication mechanism: +cinder-backup via `ceph-radosgw` MULTISITE zonegroup; Glance via `ceph-rbd-mirror`; Nova +ephemeral not replicated; carrier is the IPv6-only ULA replication plane per **D-101**, +correcting the DR-seed's original v4 carrier language; one-way staged before two-way; RPO +~15 min tunable; RTO is a measured drill output, not an SLA), and **D-104** (per-DC +single-unit Juju controllers, `create-backup`/`download-backup`, restore drilled HERE, not +just storage failover). Governing design: `docs/dc-dc-buildout-design.md` Section 4's Phase 5 +entry and Section 8's failover/failback skeleton (reproduced faithfully below, at +command-level, not redesigned). Tracker row: `docs/dc-dc-deployment-workflow.md` Stage 6 and +its Tooling gap register item #5 (Ceph replication tooling -- confirmed NO script in this +repo touches radosgw multisite or rbd-mirror; genuinely net-new) and item #1 ($DC selector -- +DOCFIX-151 -- this runbook operates against TWO DCs, so any `scripts/lib-net.sh` session +calls `lib_net_select_dc "$DC"` explicitly, once, immediately after sourcing). + +--- + +!!! DESTRUCTIVE. Steps 9-11 deliberately take DC1 "down" (a controlled simulated outage -- + either a replication-plane partition or a hard shutdown of DC1's node VMs, see Step 9) to + run a real failover drill, then bring it back in a split-brain-safe failback. There is no + scripted rollback for a botched promote/demote sequence on live Ceph pools -- a wrong-order + `rbd mirror pool promote --force` or radosgw zone demote on BOTH sides at once is exactly + the split-brain this drill exists to avoid. Each promote/demote/restore action is DISCRETE + and individually gated -- do not batch them, and do not run this against Roosevelt or any + cloud you cannot afford to actually lose data on. + +!!! GENUINELY NEW TOOLING. No script in this repo touches radosgw multisite or rbd-mirror + (Tooling gap register item #5) -- the commands below are STANDARD, well-documented upstream + Ceph administration commands (`radosgw-admin realm/zonegroup/zone create`, `rbd mirror pool + enable/peer bootstrap/promote/demote`, etc.), not invented for this repo. What IS genuinely + unknown and MUST NOT be inferred: the exact realm/zonegroup/zone names, pool names, radosgw + endpoint URLs/ports, and the real replication-plane ULA addresses for DC1 and DC2 -- none of + that exists yet. Every placeholder below is written as `` and MUST be + replaced with a value measured against the real DC1/DC2 Ceph clusters at execution time + (Stage 5's job, confirmed again here before this runbook mutates anything). If a placeholder + is still a placeholder when you reach a MUTATION step, STOP -- that is an inferred value + entering a command, the hardest rule this repo has. + +--- + +## Known gaps / prerequisites -- confirmed missing, not fixed by this runbook + +Grepped `bundle.yaml` before writing this (2026-07-09/10 sweep, same repo state as the +Tooling gap register audit): + +- **No `cinder-backup` charm is deployed in either DC's bundle.** The DR seed's own text + (`docs/dc-dc-replication-DR-seed.md` line 24) assumes cinder already exposes the + `backup-backend` endpoint -- true for `cinder` itself, but the separate `cinder-backup` + subordinate/charm that actually runs the backup driver is NOT in `bundle.yaml` today. Adding + it is a bundle change (a built-surface edit, gated the same as any other bundle change) -- + Step 1 below confirms this live and, if still absent, this runbook's first real MUTATION is + adding it, NOT skipping past it. +- **No `ceph-rbd-mirror` charm is deployed either.** Same situation: D-108's Glance mechanism + needs the charm; it is not in `bundle.yaml`. +- **`ceph-mon`'s existing `rbd-mirror` endpoint is bound to `storage`, not `replication`** + (`bundle.yaml` line 515: `rbd-mirror: storage`). D-108 requires the cross-DC rbd-mirror + daemon traffic to ride the IPv6-only ULA **replication** plane (`10.12.36.0/22` family), not + the intra-DC OSD storage plane. This binding predates D-108 (inherited from the VR0/DC0 + seed at commit 8813efc) and needs correcting as part of standing up rbd-mirror here -- flag + it, correct it as its own gated bundle-relation-binding change, do not silently leave the + cross-DC daemon traffic on the wrong plane. + +None of the three above are fixed by writing this document -- there is no live Ceph cluster +this session to test a bundle change against (2026-07-09/10 is PREP-ONLY). They are Step 1's +first checks and Step 2's first gated mutations when this runbook is actually run. + +--- + +## Sequence + +``` + 1. Confirm Stage 5 gate + bundle prerequisites (read-only; may surface bundle gaps above) + 2. Add/fix cinder-backup + ceph-rbd-mirror wiring [MUTATION: bundle change, gated, per DC] + 3. Confirm replication-plane (ULA) peering (read-only; latency/bandwidth recorded) + 4. Stage ONE-WAY: radosgw realm/zonegroup/zone [MUTATION: Ceph multisite, gated] + 5. Stage ONE-WAY: rbd-mirror peer bootstrap (rx-only) [MUTATION: Ceph rbd-mirror, gated] + 6. One-way validation drill (write DC1 -> observe DC2) (read-only observation of RPO) + 7. One-way failover/failback micro-drill; RECORD RTO/RPO [DESTRUCTIVE: small-scale promote] + -> GATE 1: one-way clean, RTO/RPO measured + 8. Enable TWO-WAY (bidirectional zone sync; rbd-mirror rx-tx both sides) [MUTATION, gated] + 9. Controller backup + restore drill (D-104), per DC [MUTATION: backup + restore test] +10. FULL failover drill: DC1 "down" -> promote/restore/rebuild at DC2 [DESTRUCTIVE] +11. FULL failback drill: split-brain-safe, controlled window [DESTRUCTIVE] +12. Record final RTO/RPO actuals; update tracker docs + -> EXIT GATE -> Stage 7 (Designate, COS, Magnum/CAPI) +``` + +--- + +## Step 1 -- Confirm Stage 5 gate + bundle prerequisites (READ-ONLY) + +**CHECK -- both DCs independently, from wherever this session runs** +```bash +source scripts/lib-net.sh +lib_net_select_dc "dc1" # explicit, once -- gap #1 / DOCFIX-151 call-site wiring +# repeat the pair below for dc2 in its own shell/session context; do not assume dc2's +# values from dc1 -- lib_net_select_dc fails loud for dc2 until its literals are real +bash scripts/cloud-assert.sh +``` +Expect PASS on both DCs. If either is not clean, STOP -- this runbook's entry condition is +unmet; go fix Stage 5 first, do not proceed with a shaky base under a DR drill. + +**CHECK -- bundle prerequisites (per DC)** +```bash +juju status --format=json | jq -r '.applications | keys[]' | grep -E '^cinder-backup$|^ceph-rbd-mirror$' +juju status --format=json | jq -r '.applications["ceph-mon"].endpoint-bindings // {}' +``` +Expect `cinder-backup` and `ceph-rbd-mirror` both present, and the `rbd-mirror` binding +already on the replication space. If either charm is absent, or the binding is still on +`storage`, this confirms the "Known gaps" section above live -- proceed to Step 2. If both are +already correct (e.g. a prior partial run of this runbook already fixed them), skip to Step 3. + +**GATE:** Stage 5 clean on both DCs; bundle prerequisites either confirmed present+correct, or +confirmed absent (routed to Step 2). + +--- + +## Step 2 -- Add/fix cinder-backup + ceph-rbd-mirror wiring [MUTATION: bundle change, gated, per DC] + +> CAUTION: edits `bundle.yaml` (a built-surface change) and applies it via `juju deploy`/ +> `juju add-relation`/`juju bind` against a LIVE cloud. This is the first mutation this +> runbook performs. Present the diff, confirm the charm channel matches the rest of the +> bundle's Caracal/squid pins, and get explicit go before applying -- do not batch this with +> Step 1's read-only checks. + +Only the specific fixes identified in Step 1 -- do not use this step to make any other bundle +change (no adjacent improvements mid-step, per this repo's hard rule 1). + +**MUTATION -- per DC, only for whichever gap Step 1 found live** +```bash +# (a) if cinder-backup is absent: +juju deploy cinder-backup --channel 2024.1/stable --to lxd: +juju add-relation cinder-backup:backup-backend cinder:backup-backend +juju add-relation cinder-backup:ceph cinder-ceph:ceph # or the real endpoint name -- + # confirm against `juju info cinder-backup` + # rather than assuming this shape +juju add-relation cinder-backup:amqp rabbitmq-server:amqp + +# (b) if ceph-rbd-mirror is absent: +juju deploy ceph-rbd-mirror --channel squid/stable --to lxd: +juju add-relation ceph-rbd-mirror:ceph-rbd-mirror ceph-mon:rbd-mirror + +# (c) if ceph-mon's rbd-mirror binding is on storage, not replication: +juju bind ceph-mon rbd-mirror=replication +``` +`` is a real MAAS/LXD placement decision on this DC's machines -- resolve it +live (`juju status` machine list) the same way every other phase runbook in this repo resolves +placement, never a copied number from DC0's bundle. + +**VERIFY** +```bash +juju status --format=json | jq -r '.applications | keys[]' | grep -E '^cinder-backup$|^ceph-rbd-mirror$' +juju status --format=json | jq -r '.applications["ceph-mon"].endpoint-bindings["rbd-mirror"]' +bash scripts/cloud-assert.sh +``` +**GATE:** both charms active/idle, `rbd-mirror` binding shows `replication`, `cloud-assert` +still PASS after the bundle change. Log this as a changelog entry (this IS a built-surface +change) with a revert (`juju remove-application`/`juju bind ... =storage` back) before moving +on. + +--- + +## Step 3 -- Confirm replication-plane (ULA) peering (READ-ONLY) + +**CHECK -- per DC, the replication plane's real ULA addresses** +```bash +ip -6 -o addr show scope global | grep -i fd # or whatever prefix NetBox assigned -- confirm + # against NetBox, do not assume fd00::/8 shape +``` +Record the real ULA address each DC's ceph-mon/ceph-rbd-mirror/ceph-radosgw units hold on the +replication plane (`10.12.36.0/22` family, per D-101 -- v6 ULA side). These are the addresses +every command from Step 4 onward uses; nothing here is invented. + +**CHECK -- inter-DC route + latency/bandwidth budget on the simulated WAN** +```bash +# from a DC1 replication-plane host to its DC2 peer's REAL measured ULA address: +ping6 -c 20 +``` +Record latency/jitter/loss actuals. Per `docs/dc-dc-buildout-design.md` Section 6, this is +the `tc netem`-simulated inter-DC leg (`opentofu/modules/netem-link`, mesh_dc1_dc2) -- confirm +its parameters are ACTUALLY APPLIED (not just planned) before trusting any RTT number recorded +here; a netem link with no rule applied yet will look artificially fast. + +**GATE:** both DCs' real replication-plane ULA addresses recorded; inter-DC reachability and +link-profile actuals recorded against the netem parameters in effect. If the link is down or +unmeasured, STOP -- Step 4 onward needs it. + +--- + +## Step 4 -- Stage ONE-WAY: radosgw realm/zonegroup/zone (multisite) [MUTATION: gated] + +Per D-108's own explicit staging requirement, prove ONE mechanism in ONE direction before +anything bidirectional. This step wires radosgw multisite for the cinder-backup target, +DC1 as master zone only. + +> CAUTION: `radosgw-admin ... --commit` mutates the realm/period on a LIVE radosgw. Run each +> command, verify its output, THEN proceed to the next -- do not chain the whole sequence +> unattended. + +**MUTATION -- on a DC1 ceph-radosgw unit** +```bash +radosgw-admin realm create --rgw-realm= --default +radosgw-admin zonegroup create --rgw-zonegroup= \ + --endpoints=http:// --master --default +radosgw-admin zone create --rgw-zonegroup= --rgw-zone= \ + --endpoints=http:// --master --default +radosgw-admin period update --commit +``` +Restart the DC1 radosgw unit (`juju run ceph-radosgw/0 restart` or the charm's own action -- +confirm the current action name against `juju actions ceph-radosgw` rather than assuming) so +it picks up realm membership. + +**VERIFY** +```bash +radosgw-admin realm list +radosgw-admin zonegroup get --rgw-zonegroup= +``` +**GATE:** DC1 shows as the sole zone in the new zonegroup, master, default. DC2 is NOT yet +joined -- that is deliberate (one-way staging); joining DC2 happens only after Step 6/7's +one-way drill is clean (see "then enable two-way" -- Step 8, not here). + +For the ONE-WAY drill itself, DC2's radosgw still needs to be a member zone of the SAME +zonegroup (multisite sync is inherently zone-to-zone; "one-way" here means DC2 is configured +`--read-only`/non-master, i.e. it can only receive, never author, matching D-108's "single +mechanism, single direction" staging intent), not a fully separate radosgw: +```bash +# on the DC2 ceph-radosgw unit, joining the SAME realm/zonegroup as a read-only zone: +radosgw-admin zone create --rgw-zonegroup= --rgw-zone= \ + --endpoints=http:// --read-only \ + --access-key= --secret= +radosgw-admin period update --commit +``` +The access-key/secret pair is the multisite system user radosgw itself generates at +`realm create` time on DC1 -- capture it there (`radosgw-admin user list` / +`radosgw-admin user info --uid=<...>`), never invent it. Restart DC2's radosgw unit. + +**VERIFY** +```bash +radosgw-admin sync status --rgw-zone= +``` +**GATE (Step 4 complete):** DC2 zone shows in the zonegroup, `--read-only`, sync status +reports it caught up (not "behind" indefinitely). This is the multisite half of one-way +staging; Step 5 does the same for rbd-mirror. + +--- + +## Step 5 -- Stage ONE-WAY: rbd-mirror peer bootstrap (rx-only) [MUTATION: gated] + +**CHECK -- confirm the Glance pool name** (do not assume `glance` -- confirm against the live +ceph-mon: `ceph osd pool ls` and cross-check which pool glance-charm's `rbd-pool` config +points at) +```bash +ceph osd pool ls +juju config glance rbd-pool +``` + +**MUTATION -- on DC1's ceph-rbd-mirror / ceph-mon unit** +```bash +rbd mirror pool enable image +rbd mirror pool peer bootstrap create --site-name \ + > /tmp/rbd-mirror-bootstrap-token +``` +Transfer the token to DC2 out of band (it is secret-adjacent -- treat it like any other +credential material this repo's guard hook flags: never printed into this session's context, +copied via a channel the operator controls directly). + +**MUTATION -- on DC2's ceph-rbd-mirror / ceph-mon unit** +```bash +rbd mirror pool enable image +rbd mirror pool peer bootstrap import --site-name --direction rx-only \ + /tmp/rbd-mirror-bootstrap-token +``` +`--direction rx-only` is the one-way staging: DC2 can only RECEIVE mirrored snapshots, never +originate. Two-way (`rx-tx`) is Step 8, only after the one-way drill (Step 6/7) is clean. + +**VERIFY** +```bash +rbd mirror pool info --all +rbd mirror pool status --verbose +``` +**GATE (Step 5 complete):** DC2 shows as an `rx-only` peer of DC1 for the Glance pool; pool +status is `health OK`, no images stuck in an error/split-brain state. + +--- + +## Step 6 -- One-way validation drill: write DC1, observe DC2 (READ-ONLY OBSERVATION) + +**Cinder-backup path** +```bash +# on DC1 (a real tenant/volume this session has, not a fabricated id): +openstack volume backup create --name dr-drill-oneway- +``` +Then, from DC2's radosgw: +```bash +radosgw-admin bucket list --rgw-zone= +radosgw-admin sync status --rgw-zone= +``` +Record the wall-clock time from the `backup create` call to the object appearing synced at +DC2 -- this is one real RPO data point (target ~15 min per D-108, tunable, MEASURED here, not +assumed). + +**Glance rbd-mirror path** +```bash +# on DC1: take/verify a snapshot on a real Glance-backed image; a normal glance image import +# already creates the pool object rbd-mirror snapshots against. +rbd mirror image status / +``` +Then on DC2: +```bash +rbd mirror image status / +``` +Record the time for `state` to reach `up+replaying` with a matching snapshot timestamp on +DC2. This is the Glance-side RPO data point. + +**GATE:** at least one clean cinder-backup sync and one clean rbd-mirror sync observed at +DC2, both within a recorded, measured time -- not assumed to already work because the mirror +"looks enabled." + +--- + +## Step 7 -- One-way failover/failback micro-drill; RECORD RTO/RPO [DESTRUCTIVE: small-scale] + +Before the FULL drill (Step 10-11), run a small-scale version against the ONE-WAY setup, using +only the test volume/image from Step 6 -- this proves the mechanics (promote/restore/re-register) +work at all before staking a real DC-down scenario on them. + +> CAUTION: `rbd mirror image promote` and `openstack volume backup restore` are real actions +> against the test artifacts from Step 6. Confirm you are promoting/restoring the DRILL +> objects, not a real tenant's data, before running. + +**MUTATION** +```bash +# force-promote is used here deliberately (DC1 stays technically reachable in this micro-drill +# -- this exercises the promote MECHANISM, the split-brain-avoidance CHECK is exercised for +# real only in Step 10, where DC1 is genuinely unreachable): +rbd mirror image promote --force / +openstack --os-cloud dc2 volume backup restore dr-drill-oneway- --volume dr-drill-oneway-restored- +``` +**VERIFY / RECORD** +```bash +openstack --os-cloud dc2 volume show dr-drill-oneway-restored- +rbd mirror image status / +``` +Record: time from the "confirm DC1 down" decision point (here, decision-to-drill) to a +usable restored volume/promoted image at DC2 -- this is the RTO measured output. Do NOT +assert a target number; the buildout design and D-108 are explicit that RTO is the drill's +OUTPUT, not a pre-committed SLA. + +**Demote back** (this micro-drill does not leave DC2 promoted afterward): +```bash +rbd mirror image demote / +``` + +**GATE 1 (Stage 6 first exit sub-condition):** one-way drill clean (Step 6 RPO observed, Step +7 RTO measured and recorded), radosgw multisite and rbd-mirror both proven end-to-end at small +scale. Only after this gate passes does Step 8 enable two-way. + +--- + +## Step 8 -- Enable TWO-WAY [MUTATION: gated] + +**MUTATION -- radosgw: make DC2's zone a full read-write member** +```bash +radosgw-admin zone modify --rgw-zone= --read-only=false +radosgw-admin period update --commit +``` + +**MUTATION -- rbd-mirror: re-import the peer with bidirectional direction** +```bash +# on DC1, generate a fresh bootstrap token (or reuse Step 5's peer relationship and just +# flip direction -- confirm against `rbd mirror pool peer` current subcommand behavior, +# it may require remove+re-add rather than an in-place direction change): +rbd mirror pool peer bootstrap create --site-name \ + > /tmp/rbd-mirror-bootstrap-token-twoway +# on DC2: +rbd mirror pool peer bootstrap import --site-name --direction rx-tx \ + /tmp/rbd-mirror-bootstrap-token-twoway +``` + +**VERIFY** +```bash +radosgw-admin sync status --rgw-zone= +radosgw-admin sync status --rgw-zone= +rbd mirror pool status --verbose +``` +**GATE:** both zones show each other caught up in `sync status`; rbd-mirror pool status shows +`rx-tx` in both directions, health OK. Re-run Step 6's write-and-observe check in the OPPOSITE +direction (write at DC2, observe at DC1) to prove two-way is real, not just configured. + +--- + +## Step 9 -- Controller backup + restore drill (D-104), per DC [MUTATION: backup + restore test] + +D-104 requires this drill exercised HERE, not deferred -- this is control-plane recovery, a +distinct surface from the storage failover above. + +**MUTATION -- per DC, take a fresh controller backup** +```bash +bash scripts/cloud-snapshot.sh --with-controller-backup +``` +This runs `juju create-backup -m admin/controller` and downloads the archive to +`~/openstack-baseline/` (jumphost-local; secret-adjacent; never committed -- confirmed by +this script's own header, DOCFIX-088). Record the archive path/size/sha256 from its manifest. + +**CHECK -- confirm the real restore command/flags before using them** +```bash +juju restore-backup --help +``` +Do not assume flags from memory or from a different juju version's documentation -- confirm +against this controller's actual installed juju client (3.6 per D-104) before running restore +for real. This repo's own discipline (never an inferred value) applies to CLI flags too, not +just IPs/IDs. + +**MUTATION -- restore drill** (decide live, per the `--help` output and this DC's actual +topology, whether the drill targets a fresh scratch controller machine or an in-place restore +path; do not assume either shape without confirming what the installed juju version supports) +```bash +juju restore-backup --file +``` +**VERIFY** +```bash +juju status -m admin/controller +juju status -m openstack +``` +**GATE:** the controller resumes management of this DC's `openstack` model post-restore; +`cloud-assert.sh` still PASS. Record wall-clock time from backup-taken to +restore-verified-working as this drill's own RTO data point (control-plane, distinct from the +storage RTO recorded in Step 7/12). + +Repeat for the OTHER DC's controller independently -- D-104's whole point is per-DC +independence; a DC2 controller drill proves nothing about DC1's and vice versa. + +--- + +## Step 10 -- FULL failover drill: DC1 "down" [DESTRUCTIVE] + +> CAUTION: this step deliberately makes DC1 unreachable/non-authoritative, then promotes DC2 +> and rebuilds real workload state there. Confirm every real tenant/workload this touches is +> either drill-only test material or something you are fully prepared to have live at DC2 +> only until failback. Individually gated -- present each promote/restore/rebuild action, +> confirm, then proceed. Do not batch. + +**Step 10.0 -- choose and record how "down" is simulated, then confirm it is truly down, not +a transient partition** (Section 8 skeleton item 1). Two shapes, pick deliberately and record +which: +- **(a) Replication-plane-only partition** (tests the split-brain-avoidance CHECK itself): + sever only the DC1<->DC2 leg on the replication plane, e.g. via the `netem-link` mechanism + (`opentofu/modules/netem-link` / a direct `tc qdisc ... netem loss 100%` on the mesh_dc1_dc2 + bridge -- confirm the real interface/bridge name against this session's own OpenTofu state, + do not assume it). DC1 itself stays fully up and reachable from Office1/its own tenants. + **Expected correct behavior: the split-brain check in Step 10.1 below should REFUSE to treat + this as "DC1 down"** (it's still reachable via an out-of-band path) -- if this drill + promotes DC2 anyway under a mere partition, that is a FINDING against the drill design, not + a pass. +- **(b) Hard down** (the real failover scenario): shut down DC1's node VMs entirely + (`virsh shutdown ` for each, or the DC1 OpenTofu-managed equivalent) -- + confirm real domain names against this session's `virsh list --all`, never assumed from + DC0's naming. + +Run (a) FIRST as a negative-control drill (prove the check correctly refuses), THEN (b) as +the real drill this stage's gate is measuring. + +**Step 10.1 -- confirm DC1 truly down, avoiding split-brain** (Section 8 skeleton item 1) +```bash +# from Office1 or any vantage point INDEPENDENT of the DC1<->DC2 replication leg: +ping +openstack --os-cloud dc1 token issue # or any cheap DC1 API reachability probe +``` +Require BOTH the replication-plane path AND at least one independent out-of-band path to +agree DC1 is unreachable before proceeding -- a DC2-only view (replication link down) is +exactly the ambiguous signal Section 8's skeleton warns about. Record the evidence. + +**Step 10.2 -- Glance: promote at DC2** (Section 8 skeleton item 2) +```bash +rbd mirror pool promote --force +``` +`--force` because DC1 is (by 10.1's confirmation) genuinely unreachable to demote first -- +this is the documented force-promote path, not an improvised shortcut. + +Re-register images into DC2 Glance: for each image that needs to exist at DC2, +`openstack --os-cloud dc2 image create` pointing at the now-promoted RBD object (or the +equivalent re-register mechanism Glance's rbd driver exposes) -- confirm the exact invocation +against the deployed Glance charm version's driver behavior at execution time; this per-image +re-registration is exactly the kind of mechanism the Delivery checklist below flags as a good +`scripts/` extraction candidate once this has been run manually once. + +**Step 10.3 -- Cinder: restore from replicated backups** (Section 8 skeleton item 3) +```bash +openstack --os-cloud dc2 volume backup restore --volume +``` +Repeat per volume that needs recovery. Record each restore's start/end time. + +**Step 10.4 -- rebuild instances; re-create Neutron state** (Section 8 skeleton item 4) +```bash +openstack --os-cloud dc2 server create --image \ + --volume --flavor --network +openstack --os-cloud dc2 floating ip create +openstack --os-cloud dc2 security group create +``` +Neutron state (ports/FIPs/security groups) is NOT replicated by design (D-108) -- this is +expected manual re-creation against DC2's own address space, not a bug to route around. + +**Step 10.5 -- RECORD RTO/RPO actuals** (Section 8 skeleton item 5) +Tabulate: time-of-last-known-good-sync (RPO, from Step 6/Step 8's data) vs. time-DC1-confirmed-down +(10.1) vs. time-workloads-usable-at-DC2 (10.4 complete). This tuple is the drill's real output +-- write it into this runbook's execution log / changelog, not as an assumed number. + +**GATE (Stage 6 second exit sub-condition):** DC2 is running the drill workload(s) +successfully; RTO/RPO recorded as measured, not assumed. + +--- + +## Step 11 -- FULL failback drill: split-brain-safe, controlled window [DESTRUCTIVE] + +Only after DC1 is confirmed genuinely recovered (reverse of Step 10.1's check -- DC1 +reachable again via an independent path, not just "the partition healed"). + +> CAUTION: the ordering below is the split-brain-safety mechanism itself. Demote the CURRENT +> primary (DC2) before promoting the recovering side (DC1) -- promoting both, or promoting DC1 +> before demoting DC2, creates two masters writing independently. Do not reorder these steps +> even if it seems faster. + +**Step 11.1 -- do NOT let DC1 auto-resume as primary** (Section 8 skeleton item 1) +No command here by design -- this is a negative instruction. Confirm no automation (charm +hook, cron, systemd timer) on DC1 is configured to auto-promote on recovery; if one exists, +that is itself a finding to log, not silently accept. + +**Step 11.2 -- demote DC1's Glance pool; resync from DC2 (now primary)** (item 2) +```bash +rbd mirror pool demote # on DC1, now that it's back +rbd mirror pool status --verbose # confirm DC1 catching up from DC2 +``` + +**Step 11.3 -- reconcile Cinder: back up DC2-side changes made during the outage, restore into DC1** (item 3) +```bash +openstack --os-cloud dc2 volume backup create --name failback-reconcile- +# once synced to DC1's radosgw zone (multisite sync, now bidirectional per Step 8): +openstack --os-cloud dc1 volume backup restore failback-reconcile- --volume +``` + +**Step 11.4 -- in a controlled window, optionally flip primary back** (item 4) +```bash +rbd mirror pool demote # DC2 (current primary) demotes FIRST +rbd mirror pool promote # DC1 promotes SECOND -- never simultaneous +``` +This is "optional" per Section 8's own skeleton -- if the operator's plan is to leave DC2 as +primary for a while, stop after 11.3 and record that as the deliberate state, not an +incomplete drill. + +**Step 11.5 -- verify both directions healthy; record the drill** (item 5) +```bash +rbd mirror pool status --verbose +radosgw-admin sync status --rgw-zone= +radosgw-admin sync status --rgw-zone= +``` +**GATE:** both pools/zones report healthy bidirectional sync; no image or bucket left in a +half-migrated state. Record the failback drill's own timing the same way Step 10.5 did. + +--- + +## Step 12 -- Record final RTO/RPO actuals; update tracker docs + +Consolidate every timing recorded in Steps 6, 7, 9, 10.5, and 11.5 into this runbook's +execution log entry (changelog + `docs/session-ledger.md`, per this repo's standard delivery +discipline). Update `docs/dc-dc-deployment-workflow.md` Stage 6's `**State:**` line from NOT +STARTED to whatever is honestly true after this run (fully DONE only if every GATE above +passed; partial otherwise, with the specific blocking item named). + +--- + +## GATE (Stage 6 exit condition, restated from `docs/dc-dc-buildout-design.md` Section 4 and +`docs/dc-dc-deployment-workflow.md` Stage 6) + +- One-way drill clean: Step 6 RPO observed for both cinder-backup and rbd-mirror; Step 7 RTO + measured and recorded (GATE 1). +- Two-way enabled and proven bidirectionally (Step 8). +- Controller backup/restore drill exercised per DC (Step 9, D-104) -- not deferred to a later + stage. +- Full failover drill (Step 10) and full failback drill (Step 11) both run, split-brain-safe + ordering observed throughout. +- RTO recorded as a MEASURED output of the drills above (Steps 7, 9, 10.5, 11.5) -- never + asserted as an achieved SLA. RPO recorded against the ~15 min tunable target, also measured, + not assumed. +- All bundle prerequisite gaps from the "Known gaps" section resolved and logged (Step 2). + +All true -> Stage 6 of `docs/dc-dc-deployment-workflow.md` moves to DONE; proceed to Stage 7 +(Designate, COS, Magnum/CAPI, per DC). + +--- + +## Delivery checklist (this repo's standard discipline) + +- [ ] `bash scripts/repo-lint.sh` clean (0 fail) before committing any repo changes made while + executing this runbook (changelog entry, tracker doc updates). +- [ ] Every bundle change from Step 2 has its own changelog entry with a stated revert + (`juju remove-application`, `juju bind ... =storage` back), per this repo's delivery + rule for built-surface changes. +- [ ] Changelog entry for this runbook's first real execution (next DOCFIX/D number via + `bash scripts/ledger-scan.sh`, do not guess next-free), recording the ACTUAL measured + RTO/RPO values from Steps 6/7/9/10/11 -- these are exactly the as-built facts this repo + commits, not secrets. +- [ ] `docs/session-ledger.md` updated with the outcome. +- [ ] `docs/dc-dc-deployment-workflow.md` Stage 6 row and tracker table updated (Step 12). +- [ ] **Follow-up, NOT built now:** once this drill has been run manually at least once + end-to-end, the radosgw multisite / rbd-mirror command sequences above (Steps 4, 5, 8, + 10.2, 10.3, 11.2-11.4) are strong candidates for extraction into real `scripts/` with a + `tests//run-tests.sh` harness (closing Tooling gap register item #5 for real, not + just documenting the gap). Out of scope for THIS deliverable -- there is no live Ceph + cluster this session to test a script against, and this task was scoped as + runbook-authoring only. Log this as a follow-up item in `docs/session-ledger.md`, not as + a TODO buried in this file. +- [ ] The per-image Glance re-registration mechanism (Step 10.2) and the Cinder + backup/restore-with-metadata reconciliation (Step 10.3/11.3) are the least-scripted, + most manual parts of the drill -- flag both as the highest-value future automation + targets if gap #5 is picked up. + +## Next +Stage 7 -- Designate, COS, Magnum/CAPI (per DC). diff --git a/runbooks/dc-dc-phase6-designate-cos-magnum.md b/runbooks/dc-dc-phase6-designate-cos-magnum.md new file mode 100644 index 0000000..f06f311 --- /dev/null +++ b/runbooks/dc-dc-phase6-designate-cos-magnum.md @@ -0,0 +1,700 @@ +# DC-DC Phase 6 -- Designate, COS, Magnum/CAPI (per DC) (Stage 7) + +Complete the per-DC service set: reactivate Designate (cloud-internal DNS), stand +up COS observability, and bring up the Magnum/CAPI management + workload cluster +path -- once per DC, against that DC's OWN Juju model and Keystone. This is the +LAST per-DC build stage before the buildout is functionally complete in both +DCs; DR wiring (Stage 6 / Phase 5) is a peer stage this runbook does not +re-verify beyond confirming its gate passed. + +This runbook is written PREP-ONLY: no DC1/DC2 cloud exists or is reachable this +session. Every command below is the procedure to run LATER, once each DC has a +live Charmed OpenStack deployment and its own Juju controller (Stage 4 / Phase 4 +in `docs/dc-dc-buildout-design.md`). Every literal that depends on that future +live state is marked `MEASURE` and must be resolved from real output at +execution time -- nothing here is carried over from the VR0 DC0 as-built values +quoted for reference. + +**Entry condition:** Stage 6's gate (DR wiring / failover drill) has passed for +the DC you are about to work on. This runbook does not re-verify Stage 6; it +only requires that gate closed before starting. + +Decisions: **D-106** (Designate reactivation -- supersedes D-019, reactivates +D-008), **D-008** (original DNS architecture / bootstrap order, now reactivated +verbatim), **D-019** (the v1 Designate deferral D-106 supersedes; its root +cause is what this stage closes), **D-021** (the amphora-pipeline constraint +whose root cause is the D-019 no-DNS posture -- resolved here), **D-105** (CAPI +mgmt-cluster placement + COS/Magnum scope, per DC, carries D-035 and D-046), +**D-046** (Magnum trustee domain-setup re-run trap), **D-101** (IPv6 family +matrix -- gates whether AAAA records are populated), **D-110** (numbering; no +new D-NNN is assigned by this runbook). + +Troubleshooting: `runbooks/appendix-A-troubleshooting.md` for anything charm/ +driver-specific inherited from phase-06/07/08 below; this runbook does not +duplicate that index. + +--- + +!!! D-046 TRAP -- READ BEFORE TOUCHING MAGNUM, EVERY TIME, PER DC: + "the magnum charm action `domain-setup` is MANUAL, not automatic; magnum + reports active / 'Unit is ready' regardless of whether it has run" (D-046). + Per D-105: "The Magnum `domain-setup` action is re-run after every + teardown / redeploy per DC (D-046 trap: magnum reports 'Unit is ready' + regardless)." Magnum's unit status is NOT evidence the trustee domain + exists. Re-run and re-assert Step 7.0 of `phase-07-conductor-graft.md` + (reproduced as Step 6.3 below) after ANY teardown/redeploy of either DC's + magnum unit -- do not skip it because the unit looks healthy. + +!!! BUNDLE.YAML IS A BUILT SURFACE -- this runbook's Step 1 changes it. + Per this repo's standing discipline ("grep design-decisions for the + governing D-NNN before ANY change to a built surface"): D-106 is the + governing decision and its Status line reads ADOPTED (2026-07-09 + operator ruling), so the edit below IS authorized in principle. That + is NOT a license to wave the edit through unreviewed -- it is still a + verify-before-mutate, individually-gated repo change like any other + built-surface edit (repo-lint clean, `preflight.sh` PASS before any + live apply, changelog entry with a revert). Do not fold it into a + "just add designate" one-liner; read Step 1 in full before editing. + +!!! COS MECHANISM IS UNDETERMINED -- do not invent it. D-105 rules the SCOPE + (COS in-scope, per-DC, no Office1 roll-up) but no D-NNN or design doc in + this repo specifies the COS deployment MECHANISM (bundle vs. separate + Juju model, exact channel, cross-model-relation topology). Step 5 below + presents the standard options for the operator to pick from at execution + time -- it does not pre-select one. + +--- + +## Sequence (run ONCE PER DC -- DC1 first, then DC2, independently; do not +## assume DC2 repeats DC1's literals -- each DC has its own Keystone) + +``` +0. Entry gate + DC context (read-only: confirm Stage 6 gate + which + DC's controller/model you are targeting) +1. bundle.yaml edit (Designate) [REPO CHANGE: reviewed once, applies to + both DCs -- gated behind D-106] +2. Static /etc/hosts bootstrap (D-008 step 1, per DC) +3. os-public-hostname per API charm [MUTATION: juju config, per DC] +4. Vault issues FQDN-SAN certs [MUTATION: cert cascade, per DC] +5. Deploy designate + designate-bind [MUTATION: bundle-diff apply into the + into the LIVE per-DC model live model, per DC] +6. Designate zone + A/AAAA records [MUTATION: per DC, from measured values] +7. Neutron default_dns_domain/ [MUTATION: per DC] + dns_servers -> Designate +8. Tenant subnets --dns-nameserver [MUTATION: per DC, incl. capi-mgmt-subnet] +9. Verify D-008/D-106 bootstrap (read-only: resolution + cert validity) +10. COS observability [DECISION POINT + MUTATION, per DC] +11. CAPI mgmt cluster (reuse phase-06-incloud-mgmt-cluster.md, + per DC, DC-parameterized) +12. Magnum conductor graft (reuse phase-07-conductor-graft.md, per + DC -- D-046 trap, Step 6.3 below) +13. Workload cluster acceptance (reuse phase-08-workload-cluster- + acceptance.md, per DC) + -> GATE (this doc) -> Stage 7 exit; repeat 0-13 for the other DC if not + already done +``` + +--- + +## Command-label convention +Every command block below is bracketed by bold labels, so a command line is never mistaken +for surrounding prose (these render in GitBucket and read clearly in a raw editor): +- **RUN -- LOC** -- the block CHANGES state; run it at LOC (e.g. `jumphost`, `vault/0`, `jumphost -> magnum/0`). +- **CHECK (read-only) -- LOC** -- a read-only verification; safe to re-run. +- **GATE:** -- a hard stop; do NOT proceed past the block unless the stated condition holds. +- **Expect:** -- what a passing result looks like. +- `> CAUTION:` -- marks a destructive, secret-handling, or irreversible step. + +--- + +## Step 0 -- Entry gate + DC context (READ-ONLY) + +MEASURE, never assume: which DC you are working, which Juju controller/model +fronts it, and which admin credentials scope to it. Stage 4 (`docs/dc-dc- +buildout-design.md` Phase 4) is where each DC's controller and Keystone came +into existence; this runbook does not know that stage's naming choices in +advance (they are not yet made this session) -- confirm them from that stage's +own as-built record before substituting `$DC`, `$DC_CONTROLLER`, `$DC_MODEL` +below. + +**CHECK (read-only) -- jumphost / Office1 operator seat** +```bash +# Set once per DC-pass through this runbook -- MEASURE from Stage 4's as-built +# record, never guess a naming scheme: +# DC=dc1|dc2 +# DC_CONTROLLER= +# DC_MODEL= +juju controllers # confirm DC_CONTROLLER is listed and reachable +juju switch "${DC_CONTROLLER}:${DC_MODEL}" +juju status -m "${DC_MODEL}" --format=short | grep -vE 'active|idle' || echo "all active/idle" +``` +**GATE:** the DC's controller is reachable, its model shows all charms +`active/idle` (Stage 4/5/6 already delivered this), and you have confirmed +which physical DC you are about to mutate. Do not proceed on an ambiguous +target. + +**CHECK (read-only)** +```bash +# Confirm Stage 6 (DR wiring / failover drill) closed for this DC -- this +# runbook's stated entry condition. Reference the Stage 6 runbook's own EXIT +# GATE for the exact check; it is not reproduced here (sibling-stage content). +``` +**GATE:** Stage 6's own EXIT GATE has passed for this DC. + +--- + +## Step 1 -- bundle.yaml edit: reactivate Designate (REPO CHANGE, gated behind D-106) + +### 1.1 -- What is currently in the repo (quoted, not paraphrased) + +`bundle.yaml`'s own description block states, at the line documenting decision +D-019: +``` + D-019 (supersedes D-008) Designate deferred to v2 +``` +`runbooks/phase-01-bundle-deploy.md` states the dry-run/deploy expectation +explicitly: +``` +relations include +`octavia:certificates - vault:certificates`, `vault:shared-db - vault-mysql-router`, +`mysql-innodb-cluster:certificates - vault:certificates`; NO `vault:ha`, NO designate +(D-019). +``` +And `bundle.yaml` itself carries the deferred application/subordinate as +comments, not live entries: +``` + # designate-hacluster stays deferred (D-019). + ... + # v2-deferred (D-019): designate-hacluster: { charm: hacluster, channel: 2.4/stable } +``` +There is no `designate:` or `designate-bind:` application block anywhere in +`bundle.yaml` today. This is the concrete state D-106 reverses. + +### 1.2 -- The governing decision, quoted + +D-106 (Status: ADOPTED 2026-07-09) states: +> Designate is deployed in-bundle in each DC, backend `designate-bind` (bind9, +> the Charmed-OpenStack-native backend, lowest delta). + +and the reactivated bootstrap order, quoted verbatim (this is D-008's original +text, reactivated by D-106 without change): +> static /etc/hosts bootstrap -> `os-public-hostname` per API charm -> Vault +> issues FQDN-SAN certs -> Designate zone plus A/AAAA -> Neutron +> `default_dns_domain` and `dns_servers` -> tenant subnets with +> `--dns-nameserver`. + +D-106 also carries a naming instantiation, not a new invention -- this is the +one place a "zone name" appears in this runbook, and it is cited from the +decision text itself, not invented: +> Naming per D-008 instantiated with the VR1 labels: +> `.omega.dc1.vr1.cloud.neumatrix.local` and `...dc2.vr1...`. +> Per-DC INDEPENDENT zones (independent Keystones), served with A and AAAA +> records (AAAA for the v6-enabled planes per D-101). + +### 1.3 -- The reversal is bigger than "add one application" -- flag this honestly + +D-106's bootstrap order includes `os-public-hostname` per API charm as its +SECOND step. `bundle.yaml`'s current header records the opposite posture, +under its own B5 label (not a separate D-NNN -- a 2026-06-01 bundle-cleanup +item): +``` +# Endpoints: IP-ONLY -- os-public-hostname dropped on all API charms; the dual VIPs ARE the +# catalog endpoints (public 10.12.4.N / internal+admin 10.12.8.N). Vault issues +# per-VIP IP-SAN certs. No control-plane DNS dependency. (B5) +``` +D-106's own text explicitly names `os-public-hostname` per API charm as part +of the order it reactivates, so re-adding it is NOT an inferred scope +addition -- it is in the ratified decision text itself. But it IS a second, +larger surface than "add designate": every API charm's `os-public-hostname` +option goes from unset back to a real FQDN, and Vault's cert issuance moves +from per-VIP IP-SAN back to FQDN-SAN. Treat this as two coupled changes under +one D-106 umbrella, not one: +1. Add `designate` + `designate-bind` (+ `designate-hacluster`) applications + and their relations. +2. Re-add `os-public-hostname: ` to every clustered API charm's + `options:` block (the same charm list D-020 already dual-VIPs: keystone, + glance, neutron-api, nova-cloud-controller, placement, cinder, octavia, + barbican, magnum, openstack-dashboard, ceph-radosgw). + +If, on review, the operator wants item 2 scoped or sequenced differently from +item 1 (e.g. reactivate DNS-serving without yet flipping every charm's +hostname), that is a real design fork -- log it as a proposal against D-106 +rather than deciding it inside this runbook. + +### 1.4 -- Keep bundle.yaml DC-agnostic; push per-DC literals to an overlay + +`bundle.yaml` is ONE shared file used to deploy both DC1 and DC2 (D-101: DC1 +inherits DC0's v4 layout unchanged; DC2 gets its own NetBox-assigned +supernet, still pending -- gap #3 in `docs/dc-dc-deployment-workflow.md`'s +tooling gap register). The new `os-public-hostname` values are NOT DC-agnostic +-- they embed `dc1` or `dc2` per D-106's naming convention. Do not hardcode a +DC-specific FQDN into the shared `bundle.yaml`. Follow the existing overlay +pattern already established for secrets (`overlays/octavia-pki.yaml`, +Step 1.0 of `phase-01-bundle-deploy.md`) and add one per-DC hostnames overlay: +`overlays/dc1-hostnames.yaml`, `overlays/dc2-hostnames.yaml`, each setting +only the `os-public-hostname` options for that DC, applied at deploy time +with an additional `--overlay` alongside whichever overlays that DC's deploy +already uses. This is a proposed pattern, not yet reviewed -- present it at +the same review gate as the rest of Step 1, do not silently adopt it. + +### 1.5 -- The application/relation shape (template -- CONFIRM before writing it live) + +The block below is a STARTING TEMPLATE only. The exact channel and the exact +relation/binding names are NOT verified against this repo's pinned Charmhub +revisions this session -- confirm both via `scripts/preflight.sh`'s Charmhub +channel assert (the same mechanism that already gates every other charm in +this bundle) before this goes into a real commit: + +```yaml + designate: + charm: designate + channel: 2024.1/stable # CONFIRM against Charmhub -- matches this + # bundle's D-002 channel matrix pattern for + # every other Caracal API charm; not yet + # independently verified for designate. + num_units: 1 + options: + # os-public-hostname etc. as needed once the DC-specific overlay (1.4) + # is drafted -- do not hardcode a zone/fqdn literal here. + bindings: + '': metal-admin + identity-service: metal-internal + shared-db: metal-internal + dns-backend: metal-internal + public: provider-public + # CONFIRM these binding/relation endpoint names against the designate + # charm's actual metadata.yaml -- written here from the standard + # Charmed OpenStack pattern (identity-service / shared-db / dns-backend), + # not from a live read of this repo's pinned revision. + + designate-bind: + charm: designate-bind + channel: 2024.1/stable # CONFIRM, as above + num_units: 1 + bindings: + '': metal-admin + dns-backend: metal-internal + + designate-mysql-router: + charm: mysql-router + channel: 8.0/stable # matches every other *-mysql-router in this bundle + bindings: + '': metal-admin + db-router: metal-internal + shared-db: metal-internal + + # un-comment and activate (currently deferred at bundle.yaml's HA-subordinates block): + designate-hacluster: + charm: hacluster + channel: 2.4/stable + options: {cluster_count: 1} + bindings: {'': metal-admin, ha: metal-internal, hanode: metal-internal, + pacemaker-remote: metal-internal, peer-availability: metal-internal} +``` +Relations to add (mirroring the existing pattern for every other clustered API +charm in this bundle -- `keystone:identity-service`, the charm's own +`:shared-db` to its mysql-router, `:certificates` to vault, `:ha` to its +hacluster subordinate, plus `designate:dns-backend - designate-bind:dns-backend`). +Do not copy these blindly -- diff against an actual `juju deploy --dry-run` on +a real DC before treating this as final. + +**CHECK (read-only) -- jumphost, once bundle.yaml + overlays are edited** +```bash +bash scripts/repo-lint.sh # expect 0 fail +bash scripts/preflight.sh # expect PREFLIGHT: PASS (incl. Charmhub channel assert) +``` +**GATE:** repo-lint clean; preflight PASS. Only after this passes does Step 5 +(the live per-DC apply) become eligible. Deliver this edit with a changelog +entry (what changed, why -- D-106, how to revert) per this repo's standing +delivery rule, same as any other script/runbook/bundle change. + +--- + +## Step 2 -- Static /etc/hosts bootstrap (D-008 step 1, per DC) + +MEASURE this DC's own hostnames and IPs from its live `juju status` / +`maas admin machines read` output -- do not carry over DC0's `/etc/hosts` +entries or VR0's IP ranges. Per D-101, DC1's v4 layout equals DC0's +(deliberate, ratified reuse -- not an assumption), so DC1's measured values +may look identical to DC0's on paper; measure them live anyway. DC2's values +are, per gap #3, not yet assigned in NetBox as of this writing -- if DC2's +literals still are not populated by the time you reach this step, STOP and +resolve that gap first (do not invent a DC2 supernet here). + +**CHECK (read-only) -- this DC's jumphost-equivalent seat** +```bash +juju status -m "${DC_MODEL}" --format=yaml | grep -E 'public-address|hostname' +``` +**RUN -- every host + LXD container in this DC's model (mirrors phase-00's +Pattern A hosts, D-008 step 1's "jumphost + all openstack hosts + all LXD +containers" scope, but for THIS DC)** +```bash +# Populate /etc/hosts on each unit with THIS DC's own measured +# pairs, one line per API charm, using D-106's naming: +# .omega.${DC}.vr1.cloud.neumatrix.local +# Do not template a value you have not just measured. +``` +**GATE:** every host/LXD container in this DC resolves every other API +charm's FQDN via `/etc/hosts` (the D-008 bootstrap floor, before Designate +itself exists). + +--- + +## Step 3 -- os-public-hostname per API charm (D-008 step 2, per DC) + +**RUN -- jumphost, per charm (repeat for every clustered API charm listed in Step 1.3)** +```bash +juju config -m "${DC_MODEL}" keystone os-public-hostname="keystone.omega.${DC}.vr1.cloud.neumatrix.local" +# ... repeat per charm: glance, neutron-api, nova-cloud-controller, placement, +# cinder, octavia, barbican, magnum, openstack-dashboard, ceph-radosgw +``` +**CHECK (read-only) -- jumphost** +```bash +juju status -m "${DC_MODEL}" --format=short | grep -vE 'active|idle' || echo "all active/idle" +``` +**GATE:** all charms return to `active/idle` after the config change settles +(a hostname-only config change is non-disruptive on this charm family, but +confirm rather than assume for this DC). + +--- + +## Step 4 -- Vault issues FQDN-SAN certs (D-008 step 3, per DC) + +This DC's Vault is an INDEPENDENT root (D-109) issuing its own certs -- do not +reuse a cert or CA from the other DC or from VR0 DC0. Trigger the certificates +relation to re-issue now that `os-public-hostname` is set; the charm-native +cert cascade (Vault's `certificates` interface) picks up the new SAN +automatically once the relation re-fires -- no manual CSR step is expected on +this charm family (confirm this is still true for the designate charm's own +certificate once Step 5 relates it). + +**CHECK (read-only) -- jumphost, per charm** +```bash +juju ssh -m "${DC_MODEL}" keystone/leader \ + "sudo openssl x509 -in /etc/apache2/ssl/keystone_1/cert -noout -text | grep -A1 'Subject Alternative Name'" Designate (D-008 step 5, per DC) + +**RUN -- jumphost** +```bash +DESIGNATE_VIP=$( ... ) # MEASURE: this DC's live designate-bind / designate API VIP +juju config -m "${DC_MODEL}" neutron-api \ + dns-domain="omega.${DC}.vr1.cloud.neumatrix.local." \ + dns-servers="${DESIGNATE_VIP}" +``` +**GATE:** `neutron-api` returns to `active/idle`; `openstack network show +` (or the relevant config surface) reflects the new dns_domain. + +--- + +## Step 8 -- Tenant subnets --dns-nameserver (D-008 step 6, per DC) + +Every existing tenant subnet in this DC created BEFORE Designate existed (most +notably `capi-mgmt-subnet` from phase-06 Step 6.1, which set +`--dns-nameserver 1.1.1.1 --dns-nameserver 1.0.0.1` under the D-019 "no +cloud DNS, tenants use public resolvers" posture) needs updating, and every +subnet created AFTER this point should use the Designate VIP from the start. + +**RUN -- jumphost, per existing tenant subnet** +```bash +DESIGNATE_VIP=$( ... ) # MEASURE: same value as Step 7 +openstack subnet set --dns-nameserver "$DESIGNATE_VIP" capi-mgmt-subnet +# clear the old public-resolver entries only if the subnet update replaces +# rather than appends -- confirm openstack-client's actual semantics for +# --dns-nameserver on `subnet set` live, do not assume append vs. replace +``` +**GATE:** `openstack subnet show capi-mgmt-subnet` shows the Designate VIP as +the (or a) DNS nameserver. + +--- + +## Step 9 -- Verify the D-008/D-106 bootstrap end to end (read-only) + +**CHECK (read-only) -- jumphost -> a tenant VM in this DC (e.g. capi-mgmt-v2)** +```bash +source ~/capi-mgmt-net.env # this DC's own env file, per phase-06 Step 6.2 -- confirm it is THIS DC's, not the other DC's stale file +ssh ... ubuntu@"$MGMT_FIP" bash -s <<'REOF' +getent hosts keystone.omega.DC.vr1.cloud.neumatrix.local # substitute the real DC label +dig +short AAAA keystone.omega.DC.vr1.cloud.neumatrix.local # if AAAA populated +curl -sk https://keystone.omega.DC.vr1.cloud.neumatrix.local:5000/v3 -o /dev/null -w '%{http_code}\n' +REOF +``` +**GATE:** the FQDN resolves (A, and AAAA where populated); the HTTPS call +against the FQDN succeeds with a cert the client trusts (Vault's CA is this +DC's own root, per D-109 -- the tenant VM/OS trust store must carry it, same +as any other charm's client already does). This is the concrete proof the +D-019/D-021 root cause (metal-only charms pulling a public FQDN endpoint that +does not resolve/route, behind a cert with no FQDN SAN) is closed for this DC. + +--- + +## Step 10 -- COS observability (DECISION POINT + MUTATION, per DC) + +No D-NNN or design doc in this repo specifies the COS deployment mechanism -- +only its scope (D-105: per-DC, in scope, no Office1 roll-up). Do not pick one +silently. Present the standard options to the operator at execution time: + +- **Option A -- `cos-lite` bundle in a dedicated per-DC `cos` Juju model** + (Canonical's standard COS Lite: Prometheus, Grafana, Loki, Alertmanager, + Traefik ingress), related to this DC's `openstack` model applications via + Juju cross-model relations (`juju offer` / `juju consume` + + `cos-configuration-k8s` or the `grafana-agent` subordinate pattern for + metrics/log forwarding). Needs a Kubernetes substrate for the `cos` + model (microk8s or equivalent) -- confirm this DC has one before picking + this option. +- **Option B -- machine-charm-native observability** (`prometheus2` / + `grafana` / `telegraf` charms in the SAME model as the OpenStack bundle, + no k8s substrate needed) -- lower delta if no k8s host is available in + this DC for a `cos` model, but is not "COS" in the strict Canonical-bundle + sense; confirm this is an acceptable substitution for the D-105 "COS" + scope before treating it as satisfying the gate. + +**CHECK (read-only) -- confirm before choosing** +```bash +# Does this DC have a k8s substrate available for a cos-lite model? +# (microk8s, or another already-enrolled k8s -- MEASURE, do not assume) +``` +**GATE (decision, log it):** record which option was chosen and why in the +session ledger / a changelog entry -- this is a real architectural choice, +not a detail to bury in a command block. + +Once chosen, the deployment steps are option-specific and are NOT detailed +further here (this is genuinely new tooling territory for this repo -- author +the concrete steps as their own change when the option is picked, with its own +harness per this repo's delivery rule, rather than improvising them live). + +**GATE (Stage 7's COS criterion):** COS scraping -- at minimum, this DC's +OpenStack API charms' metrics/logs and the CAPI mgmt cluster + workload +cluster (Steps 11-13) are visible in whichever COS surface was chosen. + +--- + +## Step 11 -- CAPI management cluster (reuse `phase-06-incloud-mgmt-cluster.md`, per DC) + +Run `runbooks/phase-06-incloud-mgmt-cluster.md` Steps 6.0-BOOT through 6.6f +IN FULL, once per DC, against this DC's own project/Keystone/model. Do not +duplicate its content here -- point at it by filename. DC-parameterization +notes (found by tracing what that runbook's canonical scripts actually +resolve dynamically vs. hardcode): + +- `scripts/phase-06-k8s-bootstrap.sh` sources `scripts/lib-net.sh` and reads + `KEYSTONE_VIP_DEFAULT` from it (DOCFIX-138 centralized this). Before running + it for a DC other than DC0, call `lib_net_select_dc "$DC"` (DOCFIX-151) + FIRST: for `dc1` this is currently a documented no-op (D-101: DC1 inherits + DC0's v4 layout unchanged -- a ratified reuse, not a stale default), for + `dc2` it currently FAILS LOUDLY (`lib-net.sh`'s dc2 literals are not yet + assigned -- tooling gap register #3). If dc2's literals are still unassigned + by the time you reach this stage, STOP; do not bypass the fail-loud guard. +- `scripts/phase-06-kubeconfig-gate.sh` -- same `lib-net.sh` dependency; same + `lib_net_select_dc "$DC"` call required first. +- Every other literal in phase-06 (`ENV(project)`, `ENV(ext-net)`, + `ENV(image)`, `ENV(mgmt-fip)`, project/image/network UUIDs) is ALREADY + written as "resolve by name" / "resolve dynamically" in that runbook -- it + re-discovers these fresh on every run, so it needs no further DC- + parameterization beyond pointing it at this DC's own `admin-openrc` and + Keystone. Confirm you have sourced THIS DC's own admin-openrc (naming + convention for per-DC openrc files was not established this session -- + confirm it from Stage 4's as-built record, do not assume a filename). +- `ENV(mgmt-cidr) 10.20.0.0/24` is a non-IPAM tenant overlay CIDR -- safe to + reuse identically in both DCs (they are independent clouds; no collision + risk). Do not route this through NetBox. +- `scripts/lib-hosts.sh`'s `lib_hosts_select_dc()` is NOT used by phase-06 (it + governs MAAS host enrollment, a Stage 3 concern, not the tenant-VM layer + phase-06 operates at) -- no action needed here, noted so it is not + mistakenly assumed required. + +**GATE (this runbook, restating phase-06's EXIT GATE):** per DC, GATE 1 +(VIP-OK) and GATE 2 (agnhost pod-egress `Completed`) both pass; the CAPI +provider stack is up with no crash-looping CAPO. + +--- + +## Step 12 -- Magnum conductor graft (reuse `phase-07-conductor-graft.md`, per DC) + +Run `runbooks/phase-07-conductor-graft.md` in full, once per DC, against this +DC's own `magnum/0` and this DC's phase-06 kubeconfig/FIP. Point at it by +filename; do not duplicate its content. + +**!!! D-046 GATE -- REPRODUCED HERE BECAUSE IT IS THE MOST-MISSED STEP IN +THIS DECISION-DERIVED CHECKLIST, restated verbatim from D-105/D-046:** +> The magnum charm action `domain-setup` is MANUAL, not automatic; magnum +> reports active / "Unit is ready" regardless of whether it has run. + +Concretely, this means: for THIS DC, run phase-07's Step 7.0 (`juju run +magnum/leader domain-setup`, then assert `openstack domain show magnum` and +`openstack user show magnum_domain_admin --domain magnum` both succeed, then +gate `openstack coe service list` with no 403) EVERY time this DC's magnum +unit is torn down and redeployed -- including if this whole Stage 7 pass is +itself a redo after an earlier Stage 4-6 rebuild of this DC. Do not treat a +green `juju status` for magnum as evidence this step already happened. + +DC-parameterization: phase-07's `ENV(conductor-unit)`, `ENV(mgmt-fip)`, +`ENV(magnum-ns)` are all already written as "resolve by name" / per-rebuild -- +re-resolve them fresh against THIS DC's model; do not carry over the other +DC's or DC0's recorded values (the as-built reference blocks in that runbook +are VR0/DC0 audit trail, not this DC's expected values). + +**GATE (this runbook, restating phase-07's EXIT GATE):** Step 7.0's domain- +setup assertion passes; conductor reaches the mgmt apiserver; magnum-capi-helm +1.4.0 (or whatever the then-current contract-coherent pin is -- confirm +against D-042/D-034 rather than assuming 1.4.0 is still current) installed; +`health_status` gate discharged per that runbook's FRESH DEPLOY routing (i.e. +deferred to Step 13 if no cluster exists yet in this DC). + +--- + +## Step 13 -- Workload-cluster acceptance (reuse `phase-08-workload-cluster-acceptance.md`, per DC) + +Run `runbooks/phase-08-workload-cluster-acceptance.md` in full, once per DC. +Point at it by filename; do not duplicate its content. One correction to +apply when running it under D-106 (rather than the D-019-amended acceptance +bar that runbook currently documents): D-011 item 8 ("Designate zones + tenant +hostname resolution") is NO LONGER deferred for VR1 -- Step 9 above is what +discharges it. Do not mark it DEFERRED per the old D-019 wording when running +this under D-106; record it PASS/FAIL against Step 9's actual result instead. + +DC-parameterization: same shape as Steps 11-12 -- every `ENV(...)` in that +runbook already resolves by name; point it at this DC's own project/model, +re-seed this DC's own workload image via stage-and-verify (D-048) rather than +assuming the image already exists in a fresh DC. + +**GATE (this runbook, restating phase-08's EXIT GATE, D-106-corrected):** +`capi-test-1`-equivalent cluster reaches `CREATE_COMPLETE`; 3 (or as-templated) +nodes Ready; CNI/CCM/CSI/CoreDNS Running; D-011 items 1-6 PASS; item 7 +(snapshot) per this DC's own baseline policy; item 8 PASS (not deferred) via +Step 9's resolution proof. + +--- + +## GATE -- Stage 7 exit condition (per DC; both DC1 and DC2 must pass independently) + +Restating `docs/dc-dc-buildout-design.md` Phase 6 / `docs/dc-dc-deployment- +workflow.md` Stage 7's gate: + +- Designate resolving A + AAAA for this DC's own zone (Step 6/9; AAAA only + where D-101's v6 planes are confirmed up). +- FQDN-SAN certs valid (Step 4/9) -- the D-019/D-021 root cause (metal-only + charms pulling an unresolvable/unroutable public FQDN behind an IP-only + cert) is CLOSED for this DC. +- COS scraping this DC's OpenStack + CAPI mgmt + workload cluster (Step 10). +- A CAPI workload cluster comes up in this DC (Step 13, `CREATE_COMPLETE`). + +Do not mark Stage 7 complete for the DC-DC buildout as a whole until BOTH DCs +independently clear all four bullets -- one DC passing does not imply the +other DC's independent Keystone/Vault/Designate zone/CAPI stack also passes. + +--- + +## Delivery checklist + +- [ ] `bundle.yaml` + new `overlays/dc1-hostnames.yaml` / `overlays/dc2- + hostnames.yaml` edited per Step 1; `scripts/repo-lint.sh` 0 fail; + `scripts/preflight.sh` PASS (incl. Charmhub channel assert on the new + designate/designate-bind charms). +- [ ] Changelog entry for the bundle.yaml + overlay change: what changed + (designate reactivation + os-public-hostname re-add), why (D-106, + superseding D-019), how to revert (re-comment the designate blocks, + drop the hostnames overlays, re-run preflight to confirm the revert is + clean). +- [ ] `docs/session-ledger.md` updated with this runbook's authoring and, + once executed, each DC's Stage 7 pass/fail state. +- [ ] `docs/dc-dc-deployment-workflow.md` Stage 7 row's `**State:**` line + updated from NOT STARTED once execution begins/completes. +- [ ] Per DC: Steps 0-9 (Designate reactivation) executed and Step 9's gate + passed BEFORE Step 10 (COS) is attempted -- do not build COS scraping + against a DNS layer that has not yet proven itself. +- [ ] Per DC: Step 10's COS mechanism choice recorded (which option, why). +- [ ] Per DC: phase-06/07/08 run to their own EXIT GATES with THIS DC's own + resolved values (never a copy-pasted literal from DC0 or the other DC). +- [ ] D-046 domain-setup re-run + assertion done for THIS pass, not assumed + from a prior one. +- [ ] This runbook's own GATE section cleared for BOTH DCs before Stage 7 is + marked complete in the deployment workflow tracker. + +## Next + +Once both DCs clear this runbook's GATE, the DC-DC buildout's per-DC service +set is complete. Remaining work is whatever the deployment-workflow doc's +"Where things stand today" table still lists as outstanding at that point +(consult it fresh -- it is a living tracker, not reproduced here).