diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index bde2998..49d065d 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1994,6 +1994,39 @@ input, which already encodes the bands, the count and the 2026-07-28 dual-family arity coupling -- one source of truth instead of a second copy that drifts. The 2026-06-03 as-built line retains the old command verbatim: it is HISTORY, not instruction. +- **THE PHASE-3 REMEDIATION BATCH IS RE-MEASURED AND PART-EXECUTED 2026-07-29.** The 21-item + batch was written 2026-07-27 and LOGGED-NOT-EXECUTED; re-verified against HEAD by a + read-only agent it is **2 FIXED / 19 REMAIN / 0 SUPERSEDED**, plus **5 NEW**. Zero + SUPERSEDED is deliberate and load-bearing: ruling 3 CREATED 3.3's hazard rather than + retiring it, and the no-DC-ordering ruling `81d8e11` moved 3.8 the wrong way, promoting it + to live. Two measurements corrected the readiness doc rather than inheriting it: 3.6's "21 + non-selector consumers" is really **23 real consumers, 8 calling the selector, 15 not, and + only 5 carrying DC-dependent values** (the doc's 21 counted 29 grep-hits minus 8 and + included four files that never source `lib-net`); and the D-133 guard is already SATISFIED + (`lib-net.sh:175` unsets VID/IFACE -- do not "fix" it). + **13 items DELIVERED into `runbooks/dc-dc-phase4-juju-bundle-per-dc.md` (434 -> 929 lines):** + 3.1 (the DC1/DC2 namespace retired -- it meant `vr1-dc0` in one place and `vr1-dc1` in + another), 3.2, 3.4 (controller-tag constraint DERIVED from `maas-role-tags.sh`, preceded by + a check refusing on 0 OR >1 matching machines), 3.11's phase-4 side, 3.13, 3.14 (now asserts + the observable via `ovs-vsctl external_ids`, refusing on no-encap), 3.15, 3.16 (all four + VERIFY-LIVE gates), 3.17 (dry-run now physically precedes the deploy it gates), 3.18, 3.19, + 3.20, 3.21. repo-lint 0 fail, zero non-ASCII, and the per-DC octavia overlay name threaded + through to match the F1 rename. + **THREE ITEMS HONESTLY NOT FIXED rather than papered over:** the dc0 `apt-mirror` + model-config key rests on a single repo comment with no client here to verify it (a mistyped + model-config key is accepted SILENTLY and leaves the model pointing nowhere), the + `juju create-backup` flag shape is corrected only where established and marked + unverified-at-authoring with the gate moved onto the resulting FILE so it holds regardless + of spelling, and whether `--unit /leader` resolves for a SUBORDINATE could not be + established -- so the `ovn-chassis` step lists units and probes a NAMED one, following this + repo's own precedent. **NEW-6/7/8 logged not fixed:** two overlays document the now-wrong + octavia-pki and phantom hostnames names in their usage comments, `vr1-dc1-machines.yaml` + shows `dc-ha-scaleup.yaml` in the SAME deploy command as the VIP overlay (which R6 rules + must not happen) under a third model-variable spelling, and phase-01's plan gate still + describes VR0's 4-machine hyperconverged layout rather than D-121's 3/2/4 split -- fixing + its counts without that sentence would leave it wrong in a way that READS as fixed. + **OWED: a human read of the expanded runbook.** It more than doubled in one pass; its + individual claims were re-grepped and it is lint-clean, but length is not correctness. **F5 (MEDIUM) -- preflight's `DC` selector does not reach P4.** `preflight.sh:99` sets `DC` without exporting it, so propagation depends on the caller's invocation form; and it is moot because `pre-flight-checks.sh` never reads `DC` at all (one hit, in a comment). diff --git a/runbooks/dc-dc-phase4-juju-bundle-per-dc.md b/runbooks/dc-dc-phase4-juju-bundle-per-dc.md index 817df84..c392461 100644 --- a/runbooks/dc-dc-phase4-juju-bundle-per-dc.md +++ b/runbooks/dc-dc-phase4-juju-bundle-per-dc.md @@ -1,7 +1,22 @@ # DC-DC Phase 4 -- Juju controller + OpenStack bundle, per DC (Stage 5) -Stand up a running, independent OpenStack cloud in DC1, then repeat the same -loop for DC2. This is the ground-truth/copy-point stage: `docs/dc-dc-buildout- +> **NAMESPACE (D-119, ADOPTED 2026-07-14) -- this runbook speaks ONLY +> region-qualified site slugs: `vr1-dc0` and `vr1-dc1`.** The bare `DC1`/`DC2` +> and `dc1`/`dc2` tokens this file used to carry are RETIRED. They were +> ambiguous in the worst possible way: read positionally, "DC1" meant VR1's +> FIRST DC (`vr1-dc0`) in this file's selector callout and VR1's SECOND DC +> (`vr1-dc1`) in its deploy step -- the same token naming two different +> datacenters in one document. `scripts/lib-net.sh:179-182` REJECTS a bare +> `dc0`/`dc1`/`dc2` outright, so any command that still carried one could not +> run. Everywhere below, `$DC` is exactly one of `vr1-dc0` | `vr1-dc1`. +> +> **There is NO ruled DC ordering for this deployment (GA-R5, 2026-07-27).** +> Both DCs deploy from the Office1 headend by the SAME procedure. Run this +> runbook once per DC, in whichever order the operator chooses; nothing here +> depends on one DC preceding the other. + +Stand up a running, independent OpenStack cloud in one DC, then repeat the same +loop for the other. This is the ground-truth/copy-point stage: `docs/dc-dc-buildout- design.md` line 12 states this repo was seeded from `openstack-caracal-ipv4` at commit 8813efc specifically so `bundle.yaml` and `runbooks/phase-01..08- *.md` could be reused here almost unchanged. **This runbook does not @@ -26,11 +41,11 @@ ruling / DOCFIX-200 -- THIS runbook is what provisions them, and expecting Deployed nodes here is exactly the breakage that ruling prevented); six planes present with correct fabrics/VLANs; provider NIC raw; PXE (v4) working; that -DC's own artifact source reachable on its own address (dc0 = the D-135 item-1 -mirror, dc1 = the D-135-AMENDED apt caching proxy consumed via +DC's own artifact source reachable on its own address (`vr1-dc0` = the D-135 +item-1 mirror, `vr1-dc1` = the D-135-AMENDED apt caching proxy consumed via `juju apt-http-proxy` -- the two DCs differ BY RULING, so do not assume a -mirror at dc1); `scripts/lib-hosts.sh`'s per-DC host data populated (that -stage's own follow-up delivery) so `lib_hosts_select_dc "$dc"` no longer +mirror at `vr1-dc1`); `scripts/lib-hosts.sh`'s per-DC host data populated (that +stage's own follow-up delivery) so `lib_hosts_select_dc "$DC"` no longer fails loud for it. **THIS STAGE OWES GATE G17 (inherited from Stage 4, split out 2026-07-27 by @@ -40,9 +55,13 @@ its nodes never boot an OS. **First boot in this stage is the first moment either is observable, so capture both then** -- per DC, from a booted node: -- dc0: `curl -sI http://10.12.8.4/` returns 200. -- dc1: apt fetches through the proxy at `10.12.68.4:3142` (NOT a mirror -- D-135 - amendment). +- `vr1-dc0`: the mirror at `10.12.8.4` answers -- and assert on the BODY, not on + `curl` exiting 0: `curl -sI` returns 0 on a 404 or a 500, and the dc0 URL is an + autoindex root that answers 200 with nothing behind it. Fetch a real index file + and check its content. +- `vr1-dc1`: apt fetches through the proxy at `10.12.68.4:3142` (NOT a mirror -- + D-135 amendment); confirm from the apt output that the fetch went THROUGH the + proxy, not around it. - both: `chronyc sources` shows the MAAS-served time source, NOT the DC edge (D-129(iv) removed the edge from that path). @@ -56,69 +75,75 @@ octets, `KEYSTONE_VIP_DEFAULT`, etc.) MUST be preceded, in THIS stage's session, by: ```bash - source scripts/lib-net.sh; lib_net_select_dc "$DC" # $DC = dc1 or dc2 + source scripts/lib-net.sh; lib_net_select_dc "$DC" # $DC = vr1-dc0 | vr1-dc1 source scripts/lib-hosts.sh; lib_hosts_select_dc "$DC" ``` - Do not skip this because it "worked before" on DC0/DC1 -- for DC1 both - calls are a documented no-op (D-101: DC1 inherits DC0 unchanged), so - skipping them happens to produce the same values by accident; for DC2 it - is NOT a no-op and skipping it silently reuses DC1's values, exactly the - bug DOCFIX-151 exists to prevent. Always call both, for both DCs, + Do not skip this because it "worked before" on the other DC. For + `vr1-dc0` the net selector is a documented no-op (D-101: VR1's first DC + inherits VR0 DC0's v4 layout unchanged -- `scripts/lib-net.sh:120-135`), + so skipping it happens to produce the right values by accident. For + `vr1-dc1` it is NOT a no-op -- that arm DIVERGES (`scripts/lib-net.sh:136-178`: + the `10.12.64/68/72/76/80/84` planes, its own gateways, its own FIP pool) + -- and skipping it silently reuses `vr1-dc0`'s addresses, exactly the bug + DOCFIX-151 exists to prevent. Always call both, for both DCs, unconditionally. ---- - -## What is genuinely IDENTICAL to VR0/DC0 for DC1, and what is not - -**SUPERSEDED 2026-07-29 by ruling 3 (2026-07-25) -- the paragraph below was -written when `bundle.yaml` carried DC0's VIPs inline. It no longer does: the -bundle is VIP-FREE (topology only) and EVERY VIP arrives via a per-DC overlay, -dc0 included. So the correct statement is now: DC1 needs no VIP *band* change --- it keeps the same `.50-.60` host octets -- but it absolutely does need its -OWN overlay, `overlays/vr1-dc1-vips.yaml`, because the prefixes differ -(`10.12.64/68/72` vs dc0's `10.12.4/8/12`). Deploying the bare bundle with no -overlay gives 11 charms of decorative HA and NO gate catches it at deploy time -except `provider-bundle-check`. Both overlays are GENERATED by -`scripts/render-dc-overlays.py`; edit `render/values/-vips.yaml` and -re-render rather than hand-editing an overlay.** - -The original text, kept because its D-101 reasoning about the plane layout is -still correct: DC1 inherits DC0's six-plane v4 layout UNCHANGED, and the host -octets are identical across DCs. `phase-01..08`'s command blocks -that reference `lib-net.sh`/`lib-hosts.sh` values will resolve to the exact -same numbers for DC1 as they did for the original DC0 rehearsal -- this is -expected, not a bug (see the `$DC` selector callout above). - -**DC2 needs a distinct, non-overlapping v4 supernet + all-new VIPs -- NOT -YET ASSIGNED.** Per D-101 and tooling gap register item #3: DC2's supernet, -the org ULA /48, and the per-DC GUA carve are NetBox-authoritative and -still genuinely unassigned (the import PIPELINE closed 2026-07-09, DOCFIX- -152, but the literals themselves have not been generated/assigned). **DC2's -bundle deploy is BLOCKED on this** -- you cannot pick DC2's VIP band, the -octavia SAN IP (per phase-01's own `1.0-GEN.c` step, which derives the SAN -IP FROM the bundle's real VIP value, not a literal), or any other DC2 -address until NetBox has real values. Do not invent a "DC2 test range" to -get moving faster -- that is exactly the inference this repo's hard rule -forbids. If DC2's literals still don't exist when you reach this stage, -STOP here for DC2 and only proceed with DC1. + A bare `dc0`/`dc1`/`dc2` is REFUSED by the selector, by design + (`scripts/lib-net.sh:179-182`). If you see that FAIL, you have typed the + retired namespace -- re-type the region-qualified slug, do not "fix" the + library. --- -## Sequence (repeat entire sequence once per DC, DC1 first) +## What is genuinely IDENTICAL across the two VR1 DCs, and what is not + +**Ruling 3 (2026-07-25): `bundle.yaml` is VIP-FREE (topology only) and EVERY +VIP arrives via a per-DC overlay, `vr1-dc0` INCLUDED.** Neither DC needs a VIP +*band* change -- both keep the same `.50-.99` host octets -- but each +absolutely needs its OWN overlay, because the prefixes differ: `vr1-dc0` is +`10.12.4/8/12`, `vr1-dc1` is `10.12.64/68/72` (`scripts/lib-net.sh:54-56` and +`:146-154`). Deploying the bare bundle with no overlay gives a control plane of +decorative HA, and NO gate catches it at deploy time except +`provider-bundle-check`. Both overlays are GENERATED by +`scripts/render-dc-overlays.py`; edit `render/values/${DC}-vips.yaml` and +re-render rather than hand-editing an overlay. + +Still correct, and the reason the two DCs read alike: D-101 rules that VR1's +FIRST DC (`vr1-dc0`) inherits VR0 DC0's six-plane v4 layout UNCHANGED, and the +host octets are identical across DCs. `phase-01..08`'s command blocks that +reference `lib-net.sh`/`lib-hosts.sh` values resolve to the same numbers for +`vr1-dc0` as they did for the original VR0 DC0 rehearsal -- expected, not a bug +(see the `$DC` selector callout above). `vr1-dc1` is where they diverge, which +is why the selector call is unconditional. + +**A THIRD DC would need a distinct, non-overlapping v4 supernet + all-new VIPs +-- NOT YET ASSIGNED, and OUT OF SCOPE for this stage.** Per D-101 and tooling +gap register item #3, any further DC's supernet, the org ULA /48 and its GUA +carve are NetBox-authoritative; the import PIPELINE closed 2026-07-09 +(DOCFIX-152) but no literals beyond the two ruled DCs have been assigned. Do +not invent a "test range" to get moving faster -- that is exactly the inference +hard rule 2 forbids. **This stage deploys `vr1-dc0` and `vr1-dc1` only; both +have real, ruled, apex-verified literals today.** + +--- + +## Sequence (repeat entire sequence once per DC; no ruled DC ordering) ``` 1. Confirm Stage 4 gate + call the $DC selectors (read-only) 2. Bootstrap a DEDICATED single-unit Juju controller, IN-DC [MUTATION, gated] -3. preflight.sh PASS (read-only gate) -4. Run phase-01 (bundle deploy) against THIS DC's model [adapted, gated -- see below] +3. preflight.sh PASS for THIS DC (read-only gate) +3.5 add-model, spaces gate, per-DC artifact source [MUTATION, gated] +4. Deploy bundle.yaml + THIS DC's overlays [MUTATION, gated -- see below] +4b. Mid-deploy watch + failure decision tree (read-only, runs alongside 4) 5. Run phase-02 (vault bring-up) -- INDEPENDENT root per DC [adapted, SECRET-HANDLING] 6. Write + apply the IPv6 family-matrix overlay [DRAFTED 2026-07-10, NOT YET APPLIED -- see below] 7. Run phase-03 (core verify) [adapted, gated] 8. Run phase-04 (network carve) [adapted, gated] 9. Run phase-05 (octavia enablement) [adapted, gated] 10. cloud-assert.sh --capture (read-only + committed BOM) -11. Controller backup: juju create-backup / download-backup [MUTATION, gated] -12. Ceph-over-v6 + geneve-over-v6 verification (read-only gate) +11. Controller backup [MUTATION, gated] +12. VERIFY-LIVE gates + Ceph-over-v6 / geneve-over-v6 (read-only gate) -> EXIT GATE (this DC) -> repeat 1-12 for the other DC -> Stage 6 ``` @@ -126,66 +151,141 @@ ## Step 1 -- Confirm Stage 4 gate + call the $DC selectors (READ-ONLY) -**CHECK -- jumphost / wherever this session runs juju against this DC** +> **RUN LOCATION -- `voffice1`, the D-128 Plane-2 host. NOT the vcloud +> jumphost.** Every `juju` and `maas` command in this runbook runs there. +> Measured 2026-07-29 on vcloud: `command -v juju` -> ABSENT, `command -v maas` +> -> ABSENT. The GA-R5 ruling of 2026-07-27 says it directly: both DCs deploy +> from the Office1 headend by the same procedure. Claude's own host only picks +> the transport (`ssh voffice1 '...'`); it is not where the work runs. + +**CHECK (read-only) -- voffice1** ```bash -export DC=vr1-dc0 # or dc2 -- explicit, per this repo's $DC convention (never omitted) +export DC=vr1-dc0 # or vr1-dc1 -- explicit, region-qualified (D-119), never omitted +export DC_MODEL="$DC" # ONE juju model per DC, named for the site slug. + # Used by every -m below; do NOT leave it unset -- + # an unset ${DC_MODEL} expands to -m "" and juju + # operates on whatever model happens to be current. +export MAAS_PROFILE="${MAAS_PROFILE:-admin}" # same default the scripts use + # (scripts/maas-role-tags.sh:37, pre-flight-checks.sh:53); + # never `maas list` -- DOCFIX-016. source scripts/lib-net.sh; lib_net_select_dc "$DC" source scripts/lib-hosts.sh; lib_hosts_select_dc "$DC" +echo "DC=$DC DC_MODEL=$DC_MODEL" echo "PLANE_CIDRS: ${PLANE_CIDRS[*]}" echo "HOSTS: ${HOSTS[*]}" ``` -**GATE:** both selector calls exit 0 (for DC2, this REQUIRES Stage 4 having -already populated real host data into `scripts/lib-hosts.sh` for dc2 -- if -`lib_hosts_select_dc vr1-dc1` still fails loud here, Stage 4 for DC2 is not -actually done yet, stop and go back). +**GATE:** both selector calls exit 0, and `DC`/`DC_MODEL` echo non-empty. If +`lib_hosts_select_dc "$DC"` fails loud, Stage 4 has not populated real host +data for this DC -- stop and go back, do not hand-supply the values. --- ## Step 2 -- Bootstrap a dedicated, in-DC, single-unit Juju controller [MUTATION, gated] -Per D-104: ONE Juju controller PER DC, deployed IN that DC (on a MAAS- -deployed machine in that DC, from Stage 4), single-unit (not HA) for VR1, -managing ONLY its own cloud. This is explicitly NOT the same controller -model as VR0/DC0's original bootstrap -- a fresh `juju bootstrap` targeting -THIS DC's MAAS cloud/region, not a reused controller. +Per D-104 (as AMENDED): ONE Juju controller PER DC, deployed IN that DC on +that DC's OWN dedicated controller VM, single-unit (not HA) for VR1, managing +ONLY its own cloud. This is explicitly NOT the same controller model as +VR0/DC0's original bootstrap -- a fresh `juju bootstrap` targeting THIS DC's +MAAS cloud/region, not a reused controller. -**Step 2.0 -- Per-DC MAAS service credential [DEPLOYMENT-TIME TASK; MUST precede bootstrap]** +**The controller target is a dedicated 10th VM, and it is READY, not Deployed.** +Under the READY-handoff ruling (DOCFIX-200, 2026-07-23) Stage 4 hands off nodes +in `Ready` with MAAS-deploy SKIPPED -- Juju provisions them here. Any surface +saying the bootstrap lands "on a MAAS-deployed machine" is stating the premise +that ruling removed. The controller VM is NOT one of the nine bundle machines +(`bundle.yaml` machines block: "The Juju CONTROLLER is a dedicated 10th VM +(D-104 amendment), bootstrapped by its own tag -- NOT a bundle machine"). -Juju authenticates to the MAAS region with an API key -- this is NOT created by -any prior stage, so it is a standing deployment-time task per DC. Per the per-DC +**Step 2.0 -- Per-DC MAAS service credential [CHECK FIRST; both may already exist]** + +Juju authenticates to the MAAS region with an API key. Per the per-DC credential-isolation discipline (SEC-012/-016; Roosevelt analog: per-DC cloud -credentials), issue a DEDICATED MAAS admin service user + its OWN key per DC -- -never reuse one key across DCs. On the MAAS region host (VR1: `voffice1`): +credentials), each DC gets a DEDICATED MAAS admin service user + its OWN key -- +never reuse one key across DCs. + +> **CHECK BEFORE YOU MINT. Both per-DC credentials already existed as of +> 2026-07-27** (`vr1-dc0-cred` and `vr1-dc1-cred`, SEC-018 / SEC-019; +> `juju show-cloud vr1-maas --client` reported `credential-count: 2`). +> Re-running the mint blocks below as written creates a DUPLICATE MAAS admin +> user and a SECOND key for a DC that already has one -- credential sprawl +> against SEC-018/-019 and D-137, on a register that is already red. Minting a +> credential is also a SEC-row obligation, not a free action. + +**CHECK (read-only) -- voffice1** +```bash +juju credentials --client # is "$DC-cred" already listed? +maas "$MAAS_PROFILE" users read | grep -o "juju-$DC" || echo "no juju-$DC user" +ls -d ~/"$DC"-creds 2>/dev/null || echo "no ~/$DC-creds folder" # SEC-009 site folder +``` +(`maas users read` is the enumeration used to account for every MAAS +account on this deployment -- `docs/CURRENT-STATE.md:2784`. Do NOT open anything +inside the creds folder; its existence is all this gate needs.) +**GATE -- three outcomes, and only one of them proceeds:** +- This DC's credential is LISTED and its MAAS user exists -> **SKIP the rest of + Step 2.0 entirely** and go to the bootstrap CHECK below. Do not re-mint. +- Neither exists -> run the mint blocks that follow, and open the per-DC SEC row. +- One exists without the other (a user with no registered credential, or a + credential naming a user MAAS does not have) -> **STOP.** That is a half-made + credential, not a missing one; reconcile it against `docs/security-ledger.md` + before touching either side. Do not "make it consistent" by minting the + missing half. + +The mint blocks below run ONLY on the second outcome. On the MAAS region host +(VR1: `voffice1`): ```bash # service account -- Juju auths via the API key, not this password; use a random one -sudo maas createadmin --username=juju- --email=juju-@ +sudo maas createadmin --username="juju-$DC" --email="juju-$DC@" ``` Generate the key so it lands ONLY in the JUMPHOST per-site creds folder -(`~/vr1--creds/maas-api-key.txt` on vcloud, SEC-009 convention -- NOT a new -location on the region), never echoed to a terminal (a printed MAAS key is -burned, SEC-006). Run the generate FROM the jumphost so the key transits ssh -stdout straight into the vcloud file (no persistent copy on the region): +(`~/$DC-creds/maas-api-key.txt` on vcloud -- the SEC-009 `~/-creds/` +convention, and the site key IS the D-119 slug; NOT a new location on the +region), never echoed to a terminal (a printed MAAS key is burned, SEC-006). +Run the generate FROM the jumphost so the key transits ssh stdout straight into +the vcloud file (no persistent copy on the region): ```bash -ssh 'sudo maas apikey --username=juju- --generate' \ - > ~/vr1--creds/maas-api-key.txt -chmod 600 ~/vr1--creds/maas-api-key.txt -awk -F: '{print "parts:", NF}' ~/vr1--creds/maas-api-key.txt # expect 3; do NOT cat +ssh "sudo maas apikey --username=juju-$DC --generate" \ + > ~/"$DC"-creds/maas-api-key.txt +chmod 600 ~/"$DC"-creds/maas-api-key.txt +awk -F: '{print "parts:", NF}' ~/"$DC"-creds/maas-api-key.txt # expect 3; do NOT cat ``` Register the MAAS cloud + this credential in Juju consuming the key BY PATH/stdin (never argv). Each key is a rotation obligation -- open a per-DC SEC row when issued. -**CHECK (read-only) -- confirm the target MAAS cloud is registered to Juju** +**CHECK (read-only) -- voffice1 -- the cloud is registered AND the controller tag exists** ```bash juju clouds --client | grep -i maas +# The bootstrap constraint below is only as good as the tag behind it. Fork 3's +# per-role tagger owns the D-104 controller tag (scripts/maas-role-tags.sh:125, +# "juju-controller-$SITE") -- confirm THIS DC's tag exists and is on exactly one +# machine before constraining on it: +bash scripts/maas-role-tags.sh check "$DC" +maas "$MAAS_PROFILE" machines read \ + | jq -r --arg t "juju-controller-$DC" '.[] | select(.tag_names[]? == $t) | "\(.hostname) \(.status_name)"' ``` +**GATE:** the tag `juju-controller-$DC` exists AND the query returns EXACTLY ONE +machine, in `Ready`. Zero machines -> the tag was never applied; `juju bootstrap` +would find no candidate and fail at allocation. More than one -> STOP; two +machines carrying a single-controller tag means the bootstrap target is +ambiguous and you cannot predict which it takes. A status other than `Ready` +(including a cached `error` power state) is NOT a pass -- refresh it with +`maas "$MAAS_PROFILE" machine query-power-state ` and re-read, rather +than assuming the node is broken (a stale cached `error` on nodes that are +simply off is a measured trap, 2026-07-27). -**MUTATION** +**MUTATION -- voffice1** ```bash -juju bootstrap +juju bootstrap \ + --constraints "tags=juju-controller-$DC" ``` -The exact MAAS cloud name and the controller name are session-specific -identifiers you choose/confirm at execution time (e.g. reflecting `dc1`/ -`dc2` in the controller name for operator clarity) -- not invented here. +**The `--constraints` is not optional.** Without it Juju asks MAAS for any +available machine and can land the controller on one of the nine role nodes -- +the exact outcome the D-104 amendment rejected, and one you only discover after +the bundle deploy finds a node missing. The tag name is DERIVED from `$DC`, so +it stays correct for both DCs by construction; it is not a literal to retype. + +The MAAS cloud name and the controller name are session-specific identifiers you +choose/confirm at execution time (reflecting the site slug in the controller name +for operator clarity) -- not invented here. Per D-104's rationale: this keeps DR-independence honest (a controller loss affects only its own DC) and is a deliberate VR1 test-only economy (HA @@ -194,82 +294,314 @@ --- -## Step 3 -- `preflight.sh` PASS (read-only gate) +## Step 3 -- `preflight.sh` PASS for THIS DC (read-only gate) ```bash -bash scripts/preflight.sh +DC="$DC" bash scripts/preflight.sh ``` -Confirmed repo-agnostic as of the 2026-07-09 sweep (`docs/dc-dc-deployment- -workflow.md` Stage 5 row) -- runs unmodified against either DC's clone/ -session. Expect `PREFLIGHT: PASS`. +**`DC=` is load-bearing -- do not run this bare.** P2 defaults to `vr1-dc0` +(`scripts/preflight.sh:99`) and assembles the overlay set from `$DC` +(`:100-105`), so a bare run on the other DC validates dc0's numbers against +dc0's bands and never parses this DC's artifact at all -- a green gate that +examined the wrong input. + +**Expect** the banner to name THIS DC before anything else: +`validating the MERGED deploy input: bundle.yaml --overlay ... --dc `. +If that line names a different DC than the one you exported, STOP -- `$DC` did +not reach the script. + +**GATE:** `PREFLIGHT: PASS`. Two failures are known and expected on the vcloud +jumphost rather than on `voffice1` (the deliberately-absent gitignored octavia +PKI overlay, and MAAS unreachable) -- which is another reason this runs on +`voffice1`. A red preflight is a stop, not a note: read every FAIL and decide +per row, and record the decision. Do not run the sub-gates piecemeal to get a +greener-looking answer. --- -## Step 4 -- Run phase-01 (bundle deploy) against this DC's model [adapted, gated] +## Step 3.5 -- Model, spaces gate, and this DC's artifact source [MUTATION, gated] -Follow `runbooks/phase-01-bundle-deploy.md` verbatim, with these DC-DC- -specific notes, not a rewrite: +Three things that must exist BEFORE the deploy and that no earlier step creates. -- **DC1: deploy `bundle.yaml` with ALL of its per-DC overlays, in this order.** - Corrected 2026-07-29 -- this step previously named ONLY the VIP overlay, and a - deploy run exactly as written would have merged a machines block still saying - `tags=openstack-vr1-dc0`, which is precisely what `overlays/vr1-dc1-machines.yaml` - exists to prevent (one Office1 MAAS region sees BOTH DCs' nodes, so dc1's model - could grab a dc0 node). Measured at the time: `grep -rn vr1-dc1-machines - runbooks/ scripts/` returned ZERO -- the overlay existed in no executable path. +**MUTATION -- voffice1 -- create THIS DC's model (guarded)** +```bash +( { + # Same capture-then-test idiom as scripts/juju-spaces-check.sh:40-42 (DOCFIX-115): + # capture first, then test. The live-pipe form races SIGPIPE under pipefail and + # reports a juju AUTH failure as "model not present" -- which here would mean + # add-model runs against a controller you are not actually talking to. + MJSON="$(juju models --format json)" || { echo "ABORT: 'juju models' failed (auth/macaroon? re-login, or append /dev/null | sed 's#.*/##')" || true + if grep -qx "$DC_MODEL" <<<"$MNAMES"; then + echo "ABORT: model '$DC_MODEL' already exists -- do not deploy into a used model" + else + juju add-model "$DC_MODEL" + fi +} ) +``` +One model per DC, named for the site slug. Deploying two DCs into a model named +`openstack` on one client is how the second deploy lands on top of the first. - ```bash - juju deploy ./bundle.yaml \ - --overlay overlays/vr1-dc1-vips.yaml \ - --overlay overlays/vr1-dc1-machines.yaml \ - --overlay overlays/octavia-pki.yaml \ - -m "${DC_MODEL}" - ``` +**CHECK (read-only) -- voffice1 -- per-model spaces** +```bash +bash scripts/juju-spaces-check.sh "$DC_MODEL" +``` +Spaces are PER-MODEL in Juju, so this can only run after `add-model` and must +run before `juju deploy` -- a missing or stale space name mis-binds charms at +deploy time and is expensive to unpick afterwards. +**GATE:** exit 0. Exit 1 = missing or stale spaces, STOP. Exit 2 = the model is +not present yet, which means the block above did not do what you think it did. - Make NO hand edit to `bundle.yaml` or to either generated overlay (ruling 3, - 2026-07-25; both `*-vips.yaml` are rendered -- edit `render/values/-vips.yaml` - and re-render, and `tests/render-drift` fails the gauntlet if you do not). The base - bundle carries no VIPs at all now. +**MUTATION -- voffice1 -- point this DC's model at ITS OWN artifact source** - **`overlays/dc-ha-scaleup.yaml` is DELIBERATELY NOT in that command.** Per R6 the - VIPs land BEFORE the HA overlay; applying it is a separate, later step once the base - deploy has settled. Validate whichever set you are about to deploy with - `DC=vr1-dc1 bash scripts/preflight.sh` -- P2 now merges the same overlays this - command does. +> **ORDERING TRAP -- this is why the step is here and not later.** If the model +> has no artifact source set, every unit apts from the INTERNET during deploy. +> The D-135/D-107 egress narrowing then lands on a half-built model, and unlike +> bootstrap the recovery is not "re-open egress and retry" -- it is a +> partially-installed cloud. Set it BEFORE the deploy, not during. - **VERIFY BEFORE THE REAL DEPLOY:** run the identical command with `--dry-run` and - confirm the machines block merged (no offline juju merger exists to prove it, per - `overlays/vr1-dc1-machines.yaml`'s own VERIFY-LIVE note). Confirm every machine - reads `tags=openstack-vr1-dc1,`; if it still reads `-dc0`, STOP -- the - fallback is inlining the dc1 tags into a rendered dc1 bundle. The Octavia PKI overlay (Step 1.0) - can reuse the existing `overlays/octavia-pki.yaml` if its CAs are still - valid (10y validity, per that runbook), OR regenerate fresh per-DC PKI via - Step 1.0-GEN if independent per-DC Octavia trust domains are desired -- - this is a call the operator should make explicitly (log it as a decision - note either way), not silently default to reuse. -- **DC2:** BLOCKED until its VIP band exists (see above). Once NetBox - assigns DC2's real supernet, the artifact is a NEW per-DC overlay rendered - from the apex -- `render/values/-vips.yaml` then - `scripts/render-dc-overlays.py render` -- and NOT an edit to `bundle.yaml`, - which no longer carries VIP lines to edit (ruling 3, 2026-07-25). The - substance of the original warning stands: it must be that DC's own measured - band, never a copy-paste of DC1's values. -- Both DCs: `bundle.yaml` explicitly ships "NO designate" per phase-01's own - gate text (D-019) -- this is UNCHANGED here; Designate reactivation is - Stage 7's job (D-106), do not reactivate it mid-Stage-5. -- Run phase-01's own EXIT GATE checks exactly as written, against THIS DC's - model. +**The two DCs DIFFER BY RULING (D-135 + its 2026-07-24/27 amendments) -- there +is no common command here.** `vr1-dc0` runs a FULL MIRROR; `vr1-dc1` runs an APT +CACHING PROXY. This is the experiment, not an interim state. Read the address +from that DC's OWN checker rather than typing it: + +```bash +# vr1-dc1 -- apt caching proxy (apt-cacher-ng). Confirm it is up and note its LISTEN: +bash scripts/dc-cache-proxy.sh check vr1-dc1 +juju model-config -m "$DC_MODEL" apt-http-proxy="http://:3142" +juju model-config -m "$DC_MODEL" apt-http-proxy # READ BACK -- assert on content +``` +Reference values, for recognising a wrong answer rather than for typing: +`scripts/dc-cache-proxy.sh:81,87` give `vr1-dc1` LISTEN `10.12.68.4` port `3142`; +`scripts/dc-mirror.sh:79` gives `vr1-dc0` LISTEN `10.12.8.4`. If the checker +reports anything else, the checker wins and you stop to reconcile. + +> **`vr1-dc0` (mirror) -- VERIFY THE KEY NAME BEFORE YOU RUN IT.** The mirror is +> consumed via a DIFFERENT model-config key than the proxy. The only in-repo +> statement of that key is a comment (`scripts/dc-mirror.sh:234`, +> `apt-mirror=http:///ubuntu`), and no juju client exists on the vcloud +> jumphost to check it against, so it is NOT written here as an executable line. +> At execution, confirm the exact key with `juju model-config --help` (or +> `juju model-defaults`) on `voffice1` FIRST, then set it and read it back the +> same way as above. Do not guess a key name -- a mistyped model-config key is +> accepted silently and leaves the model pointing nowhere. + +**GATE:** the read-back prints the address you intended, and it is THIS DC's own +artifact service. An empty read-back is a FAIL, not a pass -- checking `vr1-dc1` +as a mirror (or `vr1-dc0` as a proxy) fails BY DESIGN and is not a reason to +skip the step. + +--- + +## Step 4 -- Deploy `bundle.yaml` + THIS DC's overlays [MUTATION, gated] + +This step carries its OWN DC-parameterised deploy command. It no longer says +"follow phase-01 verbatim": phase-01's deploy blocks were written around VR0's +single-DC testcloud, and following them unchanged against a VR1 DC is a known +defect class, not a shortcut. Use `runbooks/phase-01-bundle-deploy.md` for the +things that genuinely transfer -- Step 1.0's Octavia PKI overlay sanity, the +convergence-watch pattern, and the PRE-vault-init end-state description -- with +`$DC` and `$DC_MODEL` exported per Step 1 above. **Dependency: phase-01's own +per-DC parameterisation (its `$DC`/`$DC_MODEL` handling and its per-DC VIP +guard) is delivered separately in this same batch; if you reach a phase-01 block +that still hardcodes `10.12.4.`, VR0 system_ids, `openstack0..3` disk paths, a +4-machine plan, or `juju add-model openstack`, that block has NOT been +parameterised yet -- STOP and reconcile rather than adapting it live.** + +### 4.1 -- Assemble THIS DC's overlay set + +Deploy input = `bundle.yaml` + this DC's overlays. The base bundle is VIP-FREE +and is NOT a deployable input on its own. + +| Overlay | `vr1-dc0` | `vr1-dc1` | +|---|---|---| +| `overlays/${DC}-vips.yaml` | REQUIRED (`vr1-dc0-vips.yaml`) | REQUIRED (`vr1-dc1-vips.yaml`) | +| machines retag | **none -- and that is correct**: the base bundle's machines block already carries `tags=openstack-vr1-dc0,` | REQUIRED: `overlays/vr1-dc1-machines.yaml` | +| `overlays/${DC}-octavia-pki.yaml` | REQUIRED (gitignored secret) | REQUIRED (gitignored secret) | + +**Do not interpolate a machines overlay path for `vr1-dc0`** -- there is no +`overlays/vr1-dc0-machines.yaml`, and juju errors out on a missing `--overlay` +path. The asymmetry is deliberate; the gate that catches BOTH arms is the merged +tag assertion in 4.2, not the presence of a file. Never "guard" the argument +with a silent skip: an overlay that quietly disappears is exactly how a +`vr1-dc1` deploy would merge `vr1-dc0`'s tags. + +Make NO hand edit to `bundle.yaml` or to either generated VIP overlay (ruling 3, +2026-07-25; both `*-vips.yaml` are RENDERED -- edit `render/values/${DC}-vips.yaml` +and re-render, and `tests/render-drift` fails the gauntlet if you do not). + +**`overlays/dc-ha-scaleup.yaml` is DELIBERATELY NOT in this command.** Per R6 the +VIPs land BEFORE the HA overlay; applying it is a separate, later step once the +base deploy has settled. That is also why the base deploy legitimately shows +`cluster_count: 1` -- see Step 12's gate, which states when `1` is correct and +when it is the failure. + +**Octavia PKI is PER-DC.** Under the R7 amendment (D-109, RULED 2026-07-27) the +generator is no longer dc0-frozen and each DC gets its own CA material, so this +is a per-DC artifact (`overlays/${DC}-octavia-pki.yaml`), not one shared file. +Reuse this DC's existing overlay if its CAs are still valid (10y, per +phase-01 Step 1.0), OR regenerate via phase-01 Step 1.0-GEN with `DC` exported. +Either way it is an explicit operator call, logged as a decision note -- not a +silent default to reuse. Cross-DC reuse was refused on posture: the overlay +carries CA private keys plus a plaintext passphrase in a repo SEC-004 records as +PUBLIC. + +### 4.2 -- DRY-RUN FIRST, and read the machines block [read-only gate] + +> **Run this BEFORE 4.3. It is the only thing standing between a typo in the +> overlay set and nine wrongly-allocated MAAS machines.** There is no offline +> juju merger that can prove the machines-block override merges (see +> `overlays/vr1-dc1-machines.yaml`'s own VERIFY-LIVE note), so the dry-run IS +> the proof. + +**CHECK (read-only) -- voffice1 -- run the block for YOUR DC.** These are +EXACTLY the 4.3 commands plus `--dry-run`; if the two ever differ, this gate is +validating something you are not deploying. + +`vr1-dc0` (no machines overlay -- the base bundle already carries its tags): +```bash +juju deploy ./bundle.yaml \ + --overlay overlays/vr1-dc0-vips.yaml \ + --overlay overlays/vr1-dc0-octavia-pki.yaml \ + -m "${DC_MODEL}" --dry-run +``` +`vr1-dc1` (machines overlay REQUIRED, order: vips -> machines -> pki): +```bash +juju deploy ./bundle.yaml \ + --overlay overlays/vr1-dc1-vips.yaml \ + --overlay overlays/vr1-dc1-machines.yaml \ + --overlay overlays/vr1-dc1-octavia-pki.yaml \ + -m "${DC_MODEL}" --dry-run +``` +**GATE -- assert on the CONTENT of the plan, not on the command exiting 0:** +1. **Every machine reads `tags=openstack-$DC,`.** If any machine still + names the OTHER DC, STOP -- one Office1 MAAS region sees BOTH DCs' nodes, so + a wrong tag here means this model can allocate the other DC's hardware. The + fallback is inlining this DC's tags into a rendered per-DC bundle, not + deploying anyway. +2. **Nine machines, `"0"` through `"8"`**, matching `bundle.yaml`'s machines + block (D-121 Option C: 3 control / 2 compute / 4 storage). A four-machine + plan means you are looking at VR0's topology. +3. **Every VIP in the plan is on THIS DC's prefixes** -- `10.12.4/8/12` for + `vr1-dc0`, `10.12.64/68/72` for `vr1-dc1` (`scripts/lib-net.sh:54-56`, + `:146-154`). A VIP from the other DC means the wrong overlay merged. +4. If the plan reports an ERROR on any `--overlay` path, do NOT drop the overlay + to make it run. A missing overlay is a missing input, not an optional one. + +Capture this output. It is the plan-count evidence the exit gate cites -- plan +counts are quoted only from captured command output, never from this file's +prose (GA-R1 rule 2). + +### 4.3 -- The real deploy [MUTATION, gated] + +**The SAME block you dry-ran in 4.2, minus `--dry-run`. Nothing else changes.** + +`vr1-dc0`: +```bash +juju deploy ./bundle.yaml \ + --overlay overlays/vr1-dc0-vips.yaml \ + --overlay overlays/vr1-dc0-octavia-pki.yaml \ + -m "${DC_MODEL}" +``` +`vr1-dc1`: +```bash +juju deploy ./bundle.yaml \ + --overlay overlays/vr1-dc1-vips.yaml \ + --overlay overlays/vr1-dc1-machines.yaml \ + --overlay overlays/vr1-dc1-octavia-pki.yaml \ + -m "${DC_MODEL}" +``` + +Open the Step 4b watch windows before this returns. + +- `bundle.yaml` ships "NO designate" for THIS stage per phase-01's own gate text + (D-019, as superseded by D-106): Designate reactivation is Stage 7's job, do + not reactivate it mid-Stage-5. +- Settle to phase-01's documented PRE-vault-init end state, read with `$DC_MODEL` + substituted for its hardcoded model name: ZERO units in `error`; + mysql-innodb-cluster ACTIVE; `vault/0` BLOCKED "Vault needs to be initialized" + (the Step 5 trigger, not a fault); the TLS consumers waiting on vault certs; + octavia BLOCKED "Awaiting configure-resources". + +--- + +## Step 4b -- Mid-deploy watch + failure decision tree (read-only, runs alongside Step 4) + +Open BOTH windows before Step 4.3 returns. A deploy you are not watching is one +you diagnose from logs afterwards. + +**CHECK (read-only) -- voffice1, window 1 (detail)** +```bash +juju status -m "$DC_MODEL" --watch 5s +``` +**CHECK (read-only) -- voffice1, window 2 (signal)** +```bash +bash scripts/deploy-watch.sh "$DC_MODEL" 15 +``` +Compact machine/unit state COUNTS plus the NAMED principal units in +error/blocked. Health-at-a-glance: the error/blocked section stays EMPTY until +the expected late blocks (vault needs-init, octavia awaiting-configure). +Neither window descends into subordinates and neither replaces a phase gate. + +**Expected end-state shape** (not a count to quote -- confirm against Step 4.2's +captured plan, GA-R1 rule 2): nine machines reaching `started`; zero units in +`error`; the only `blocked` units are `vault/0` (needs init) and `octavia` +(awaiting configure-resources). Anything else named in the error/blocked list is +this tree's input. + +### Decision tree -- something went wrong mid-deploy + +Work top to bottom. **Default is FIX-FORWARD**; juju converges, and tearing down +a partially-good model costs more than repairing it. + +1. **Units are still `waiting`/`maintenance` and counts are MOVING.** Not a + failure. Snap installs and cert exchange are slow on first boot. Keep + watching; do not intervene because it is quiet. +2. **A machine never leaves `pending` / allocation fails.** This is a MAAS-side + answer, not a juju one. Check this DC's node pool: the right tag on the right + count of `Ready` machines (Step 2's tag check, widened to the role tags). A + cached `power_state=error` on nodes that are merely off reads as a hard + blocker and is not one -- refresh with + `maas "$MAAS_PROFILE" machine query-power-state ` and re-read + before diagnosing a fault. **Fix-forward:** correct the tags/power state and + let juju retry. Do NOT destroy the model. +3. **Units are in `error` with a hook failure.** Read the actual hook error + (`juju debug-log -m "$DC_MODEL" --replay --include `). If it is a + transient fetch failure, first check whether this model's artifact source + (Step 3.5) is set and reachable -- a model apting from the internet behind a + narrowed edge fails exactly this way. **Fix-forward:** repair the cause, then + `juju resolved --no-retry ` or `juju resolved ` as appropriate. +4. **The wrong DC's hardware got allocated, or a VIP from the other DC appears + in the model.** STOP. This is the one case where fix-forward is wrong: the + model has bound to another DC's addresses/machines, and repairing it in place + leaves you unsure what is bound to what. Tear the MODEL down + (`juju destroy-model "$DC_MODEL"`, operator-gated) and return to Step 4.2 -- + the dry-run gate exists precisely to make this case unreachable. +5. **You are considering destroying anything BELOW the juju layer** (the + containment VM, the DC's libvirt resources, MAAS records). Read + `runbooks/dc-dc-teardown-rollback.md`'s rollback decision tree FIRST, and + read its Model-B banner before acting on any of its resource-level steps. + **Standing caution as of 2026-07-29:** that runbook's Step 2 MAAS-side check + and Step 3 scoped-destroy targets have known defects against the current + two-root shape -- treat both as reference, verify every module name against + the live `opentofu/` tree before running anything, and never let a whole-DC + destroy be the first thing you reach for while 18 nodes are enrolled in MAAS. + +At every branch: record what you observed and what you chose. A mid-deploy +decision that is not written down is one the next session has to re-derive. --- ## Step 5 -- Run phase-02 (vault bring-up) -- INDEPENDENT root per DC [adapted, SECRET-HANDLING] -Follow `runbooks/phase-02-vault-bringup.md` verbatim per DC. Per D-109: each +Follow `runbooks/phase-02-vault-bringup.md` per DC, with `$DC`/`$DC_MODEL` +exported per Step 1 and substituted for any hardcoded model name in its command +blocks. Per D-109: each DC's Vault is its OWN independent root CA -- no regional root-of-trust this phase. This means: - The `vault operator init` one-shot step (Step 2.1) runs ONCE PER DC, - producing DISTINCT unseal-key shares and a DISTINCT root token for DC1 vs. - DC2. Store them separately, clearly labeled by DC (this repo's existing + producing DISTINCT unseal-key shares and a DISTINCT root token for `vr1-dc0` + vs. `vr1-dc1`. Store them separately, clearly labeled by DC (this repo's existing secrets discipline -- `~/vault-init/`-style off-host capture -- applies identically per DC, just not shared between them). - D-109 also requires each DC's Vault to issue certs with BOTH IPv4 and @@ -313,17 +645,53 @@ (confirmed no such charm-config option exists -- geneve family follows the bound interface automatically once the plane itself is ULA-only). -**Apply via** (stacked overlays, same pattern as the existing octavia-pki -overlay): +**This is an INCREMENTAL ADD against the LIVE `$DC_MODEL` Step 4 already +deployed -- not a fresh deploy.** It re-passes the SAME overlay set Step 4 used +and adds the family matrix on top, so juju's diff introduces only the +address-family options; the base topology, the VIPs and the machines block must +come out UNCHANGED. That is precisely what the GATE below checks. Re-passing the +full set is deliberate: dropping the VIP or machines overlay here would present +juju with an input that contradicts the running model. + +Corrected 2026-07-29: this command previously named `overlays/octavia-pki.yaml` +and `overlays/${DC}-hostnames.yaml` and passed NO VIP overlay at all -- so since +ruling 3 its merged input carried ZERO VIPs, and `${DC}-hostnames.yaml` has +NEVER BEEN AUTHORED (it is unbuilt D-106 work, proposed in +`runbooks/dc-dc-phase6-designate-cos-magnum.md`), on which juju errors out. + +> **THIS STEP HAS NOT BEEN APPLIED TO ANY LIVE MODEL** (overlay drafted +> 2026-07-10, still unvalidated -- see open item 3 below). Every block here is +> `--dry-run`; the real apply is a separate, gated decision once the diff has +> been reviewed. **Step 12's v6 gates depend on this step having actually been +> applied** -- if you have only dry-run it, 12.1 and 12.2 are BLOCKED, and +> "blocked on Step 6" is the honest recorded answer, not a soft pass. + +`vr1-dc0`: ```bash juju deploy ./bundle.yaml \ - --overlay overlays/octavia-pki.yaml \ - --overlay "overlays/${DC}-hostnames.yaml" \ + --overlay overlays/vr1-dc0-vips.yaml \ + --overlay overlays/vr1-dc0-octavia-pki.yaml \ --overlay overlays/dc-dc-ipv6-family-matrix.yaml \ -m "${DC_MODEL}" --dry-run ``` +`vr1-dc1`: +```bash +juju deploy ./bundle.yaml \ + --overlay overlays/vr1-dc1-vips.yaml \ + --overlay overlays/vr1-dc1-machines.yaml \ + --overlay overlays/vr1-dc1-octavia-pki.yaml \ + --overlay overlays/dc-dc-ipv6-family-matrix.yaml \ + -m "${DC_MODEL}" --dry-run +``` +Same overlay-set rules as Step 4.1: `vr1-dc0` takes no machines overlay because +the base bundle already carries its tags, and no overlay is ever dropped to make +the command run. + Review the diff before applying for real, same discipline as every other -overlay in this repo. +overlay in this repo. **GATE:** the diff changes ONLY the address-family options +this overlay sets; every VIP still reads on THIS DC's prefixes, and the machines +block still reads `tags=openstack-$DC,`. If either moved, the overlay set +is wrong -- stop, do not apply. **Still genuinely open, not resolved by the overlay:** 1. The 8 charms whose `prefer-ipv6` presence is INFERRED-BY-PATTERN (not @@ -351,13 +719,24 @@ ## Steps 7-9 -- phase-03 / phase-04 / phase-05, adapted per DC Follow `runbooks/phase-03-core-verify.md`, `runbooks/phase-04-network- -carve.md`, `runbooks/phase-05-octavia-enablement.md` verbatim per DC, with -the same `$DC` selector discipline as Step 1 applied to every command block -in them that reads `lib-net.sh`/`lib-hosts.sh` values. No other adaptation -is currently known to be needed in these three -- if executing them surfaces -a DC0-specific literal baked into a check block that these runbooks' -authors didn't anticipate needing DC-parameterization, LOG it (a DOCFIX -candidate) rather than silently patching around it live. +carve.md`, `runbooks/phase-05-octavia-enablement.md` per DC, with `$DC` and +`$DC_MODEL` exported per Step 1, the selector discipline applied to every +command block in them that reads `lib-net.sh`/`lib-hosts.sh` values, and +`$DC_MODEL` substituted for any hardcoded model name. + +**Known, measured caveat -- these three phases call scripts that do NOT select +a DC.** Measured 2026-07-29: of the scripts that source `scripts/lib-net.sh`, +these Stage-5 ones read a DC-dependent value while never calling +`lib_net_select_dc`, so they resolve `vr1-dc0`'s literals whichever DC you are +on: `scripts/phase-03-admin-openrc.sh` (`KEYSTONE_VIP_DEFAULT`), +`scripts/phase-04-network-create.sh` and `scripts/phase-04-network-verify.sh` +(both also carry a hardcoded `PROVIDER_CIDR="10.12.4.0/22"`, which a selector +would not reach), and `scripts/vault-kv-health.sh` (`METAL_INTERNAL_CIDR`). +On `vr1-dc0` that is harmless and correct. **On `vr1-dc1` it is wrong and +SILENT** -- no error, just dc0's addresses. Until those scripts are fixed, +either export the value explicitly (each honours an env override) or verify +every address in their output against this DC's planes before believing a PASS. +LOG anything further you find (a DOCFIX candidate); do not patch around it live. --- @@ -368,50 +747,163 @@ ``` Per DC, at real deploy-completion. Confirmed repo-agnostic (2026-07-09 sweep). Commit the resulting `asbuilt/` BOM per this repo's existing -deploy-loop convention, labeled clearly by DC (do not let DC1's and DC2's -BOMs collide/overwrite -- confirm the script's own output path is already -DC-distinguishing, or add a DC-qualified path if not, before running for -the second DC). +deploy-loop convention, labeled clearly by DC (do not let `vr1-dc0`'s and +`vr1-dc1`'s BOMs collide/overwrite -- confirm the script's own output path is +already DC-distinguishing, or add a DC-qualified path if not, BEFORE running +for the second DC; a silently overwritten BOM loses the first DC's evidence). --- ## Step 11 -- Controller backup [MUTATION, gated] +> **THE FLAG SHAPE BELOW IS UNVERIFIED AT AUTHORING -- CONFIRM IT FIRST.** +> No juju client exists on the vcloud jumphost (measured 2026-07-29: +> `command -v juju` -> ABSENT), so this shape could not be checked against a +> real client when it was written, and a repo-wide grep corroborates the +> subcommands only from other prose. What IS established: `-m` takes a MODEL, +> not a controller name -- so the previous form here (`-m `) +> was wrong on both lines; and `create-backup` downloads by default, which made +> the old second line redundant rather than necessary. **At execution, run +> `juju create-backup --help` on `voffice1` and use what it says.** Then gate on +> the resulting FILE, which is true regardless of flag spelling. + ```bash -juju create-backup -m -juju download-backup -m +# Confirm the flags FIRST (see the caveat above): +juju create-backup --help + +# Then take the backup against the CONTROLLER MODEL of this DC's controller: +juju switch +juju create-backup -m controller ``` -Per D-104: scheduled controller backups, confirmed backup posture (juju 3.6 -`create-backup`/`download-backup`). Store the downloaded backup off-host, -labeled by DC, same secrets-adjacent handling discipline as Vault's init -capture. The RESTORE half of this drill is exercised in Stage 6 (DR wiring -and failover drill) -- this step only takes and stores the backup, it does -not restore it. +**GATE -- assert on the artifact, not on the command exiting 0:** a backup file +exists in the working directory, is NON-ZERO in size, and its timestamp is from +this run. Record its name and size. If `create-backup` printed a path but no +file is there, that is a FAIL -- do not record "backup taken". + +Then store it OFF-HOST, labeled by DC, with the same secrets-adjacent handling +discipline as Vault's init capture, and confirm the off-host copy's size matches +before deleting anything locally. Per D-104: scheduled controller backups, one +per DC. The RESTORE half of this drill is exercised in Stage 6 (DR wiring and +failover drill) -- this step only takes and stores the backup, it does not +restore it. --- -## Step 12 -- Ceph-over-v6 + geneve-over-v6 verification (READ-ONLY gate) +## Step 12 -- VERIFY-LIVE gates + Ceph-over-v6 / geneve-over-v6 (READ-ONLY gate) + +`juju run` on juju 3.x is the ACTION runner, so the old `juju run 'ceph -s'` +form asked ceph-mon to run an action NAMED `ceph -s` and could never execute. +`juju exec` is the command runner; the ` ceph-mon/leader 'ceph -s' 2>&1 | grep -i "v6\|v2" -# OVN geneve encapsulation over the data-tenant ULA plane: -juju config -m ovn-central | grep -i encap +juju exec -m "$DC_MODEL" --unit ceph-mon/leader -- 'ceph -s' /leader` resolves for a +subordinate is NOT verified from this repo. The precedent that IS established +here uses a NAMED unit, not `/leader` +(`runbooks/phase-05-octavia-enablement.md:93` -> `--unit octavia/0`). So: + +```bash +# 1. list this DC's chassis units and pick one that is on a COMPUTE host: +juju status -m "$DC_MODEL" ovn-chassis +# 2. probe THAT unit by name (substitute for ovn-chassis/0): +juju exec -m "$DC_MODEL" --unit ovn-chassis/0 -- \ + 'sudo ovs-vsctl get Open_vSwitch . external_ids' + `data-tenant`, after `lib_net_select_dc "$DC"`) -> PASS. +- An encapsulation address is reported and it is IPv4, or outside that prefix + -> FAIL. The chassis is tunnelling on the wrong plane or the wrong family. +- No encapsulation entry appears at all, or the command errors -> **REFUSE, do + not pass.** "Could not look" is never "nothing there". Record the raw output, + confirm the unit name from step 1 is one juju actually lists, and check Step 6 + was APPLIED rather than only dry-run. If it still shows nothing, this gate is + BLOCKED and is recorded as BLOCKED -- never as a pass. + +### 12.3 -- The four VERIFY-LIVE gates + +Each asserts a RULED value that no offline gate can reach. Sources: +`docs/CURRENT-STATE.md:247-252` (the VERIFY-LIVE list) and +`docs/design-decisions.md:824` (the keystone `PO:` mechanism). + +**(a) hacluster `cluster_count` -- and WHEN each value is correct.** +```bash +juju config -m "$DC_MODEL" -hacluster cluster_count +``` +**GATE, and read the precondition before judging it:** after the Step 4 base +deploy, `cluster_count: 1` is EXPECTED AND CORRECT -- the base bundle pins it +deliberately and `overlays/dc-ha-scaleup.yaml` is excluded from Step 4 per R6 +(`overlays/dc-ha-scaleup.yaml:51-58`). The `3` assertion applies ONLY after the +HA scale-up overlay has been applied; at THAT point a `cluster_count` still +reading `1` behind a 3-unit principal is the failure -- pacemaker will bootstrap +a VIP off a single node and every offline checker stays green. State which side +of the overlay you are on when you record this result. + +**(b) Ceph pool redundancy.** +```bash +juju exec -m "$DC_MODEL" --unit ceph-mon/leader -- 'ceph osd pool ls detail' preflight.sh` PASS before add-model (Step 3); the Step 4.2 +dry-run captured and its machines-block tags confirmed; post-deploy +`cloud-assert.sh --capture` per DC (Step 10); controller backup taken + stored, +gated on the FILE (Step 11); the four VERIFY-LIVE gates plus Ceph-over-v6 and +geneve-over-v6 recorded (Step 12, the v6 halves contingent on Step 6). --> Repeat this entire runbook for the other DC (DC1 first is recommended, -since only DC1 is unblocked by real address literals today) -> proceed to -Stage 6 (DR wiring and failover drill) once BOTH DCs pass this gate. +-> Repeat this entire runbook for the other DC. **There is no ruled DC ordering +for this deployment (GA-R5, 2026-07-27): both DCs deploy from the Office1 +headend by the same procedure, in whichever order the operator chooses.** -> +proceed to Stage 6 (DR wiring and failover drill) once BOTH DCs pass this gate. --- @@ -419,15 +911,18 @@ - [ ] `bash scripts/repo-lint.sh` clean (0 fail) before committing any repo changes made while executing this runbook (the IPv6 overlay file, any - a NEW per-DC VIP overlay rendered for DC2 -- NOT a `bundle.yaml` edit, - which carries no VIPs since ruling 3 -- any DC-qualified `asbuilt/` path fix). + re-rendered per-DC VIP overlay -- NOT a `bundle.yaml` edit, which carries + no VIPs since ruling 3 -- any DC-qualified `asbuilt/` path fix). +- [ ] The Step 4.2 dry-run output CAPTURED to a dated file and cited in the + commit that closes this stage for that DC (plan counts are quoted only + from captured output, GA-R1 rule 2). - [ ] Changelog entry per real repo change this runbook produces (the overlay, any bundle edit, any script fix) -- next DOCFIX number via `bash scripts/ledger-scan.sh`, same as every other delivery this session. - [ ] `docs/session-ledger.md` updated with the outcome, per DC. - [ ] `docs/dc-dc-deployment-workflow.md` Stage 5 row updated once BOTH DCs - have actually passed Step 12's gate (not just DC1). + have actually passed Step 12's gate (not just the first one done). - [ ] Log the Step 6 charm-config research (v6 SAN support, v6 addressing option names) as its own follow-up item if not resolved inline during this stage's real execution -- do not leave it silently unresolved in