# DC-DC Phase 4 -- Juju controller + OpenStack bundle, per DC (Stage 5) > **NAMESPACE (D-119, ADOPTED 2026-07-14) -- this runbook speaks ONLY > region-qualified site slugs: `vr1-dc0` and `vr1-dc1`.** The bare `DC1`/`DC2` > and `dc1`/`dc2` tokens this file used to carry are RETIRED. They were > ambiguous in the worst possible way: read positionally, "DC1" meant VR1's > FIRST DC (`vr1-dc0`) in this file's selector callout and VR1's SECOND DC > (`vr1-dc1`) in its deploy step -- the same token naming two different > datacenters in one document. `scripts/lib-net.sh:179-182` REJECTS a bare > `dc0`/`dc1`/`dc2` outright, so any command that still carried one could not > run. Everywhere below, `$DC` is exactly one of `vr1-dc0` | `vr1-dc1`. > > **There is NO ruled DC ordering for this deployment (GA-R5, 2026-07-27).** > Both DCs deploy from the Office1 headend by the SAME procedure. Run this > runbook once per DC, in whichever order the operator chooses; nothing here > depends on one DC preceding the other. Stand up a running, independent OpenStack cloud in one DC, then repeat the same loop for the other. This is the ground-truth/copy-point stage: `docs/dc-dc-buildout- design.md` line 12 states this repo was seeded from `openstack-caracal-ipv4` at commit 8813efc specifically so `bundle.yaml` and `runbooks/phase-01..08- *.md` could be reused here almost unchanged. **This runbook does not duplicate those eight runbooks' command-level content.** It describes what changes to run them TWICE, independently, in the VR1 two-DC dual-stack context, and calls out the real adaptation work plainly rather than asserting "just re-run it" glosses over anything. **Governing docs:** `docs/dc-dc-buildout-design.md` Section 4 "Phase 4" entry and Section 2 (address-family table); `docs/design-decisions.md` D-104 (controller topology), D-109 (per-DC independent Vault roots + IPv6-SAN certs), D-101 (address plan / family matrix); `docs/dc-dc-deployment- workflow.md` Stage 5 row (**"This is where the existing repo tooling (preflight.sh, cloud-assert.sh, lib-net.sh/lib-hosts.sh, run-logged.sh) does the most direct work"**) and tooling gap register item #1 (the `$DC` selector convention, CLOSED 2026-07-09 DOCFIX-151 -- this is the stage that actually calls it). **Prerequisites (must be true entering this stage, per DC):** Stage 4 (`runbooks/dc-dc-phase3-maas-enlist-deploy.md`) done for that DC -- nodes **READY** (NOT Deployed: MAAS-deploy is SKIPPED per the 2026-07-23 READY-handoff ruling / DOCFIX-200 -- THIS runbook is what provisions them, and expecting Deployed nodes here is exactly the breakage that ruling prevented); six planes present with correct fabrics/VLANs; provider NIC raw; PXE (v4) working; that DC's own artifact source reachable on its own address (`vr1-dc0` = the D-135 item-1 mirror, `vr1-dc1` = the D-135-AMENDED apt caching proxy consumed via `juju apt-http-proxy` -- the two DCs differ BY RULING, so do not assume a mirror at `vr1-dc1`); `scripts/lib-hosts.sh`'s per-DC host data populated (that stage's own follow-up delivery) so `lib_hosts_select_dc "$DC"` no longer fails loud for it. **THIS STAGE OWES GATE G17 (inherited from Stage 4, split out 2026-07-27 by operator ruling -- see the G17 row in `docs/CURRENT-STATE.md`, the authority).** Stage 4 could not verify that a NODE reaches its DC's artifact source, or that a node's time source is the MAAS hierarchy, because under the READY-handoff ruling its nodes never boot an OS. **First boot in this stage is the first moment either is observable, so capture both then** -- per DC, from a booted node: - `vr1-dc0`: the mirror at `10.12.8.4` answers -- and assert on the BODY, not on `curl` exiting 0: `curl -sI` returns 0 on a 404 or a 500, and the dc0 URL is an autoindex root that answers 200 with nothing behind it. Fetch a real index file and check its content. - `vr1-dc1`: apt fetches through the proxy at `10.12.68.4:3142` (NOT a mirror -- D-135 amendment); confirm from the apt output that the fetch went THROUGH the proxy, not around it. - both: `chronyc sources` shows the MAAS-served time source, NOT the DC edge (D-129(iv) removed the edge from that path). Capture the output to a dated file and cite it in the commit that closes G17. If first boot happens without these captures being taken, the observation window has passed and G17 needs a deliberate rescue-boot to answer -- so take them the first time. !!! Every command block in the referenced `phase-01..08` runbooks that reads `scripts/lib-net.sh` or `scripts/lib-hosts.sh` values (VIP bands, host octets, `KEYSTONE_VIP_DEFAULT`, etc.) MUST be preceded, in THIS stage's session, by: ```bash source scripts/lib-net.sh; lib_net_select_dc "$DC" # $DC = vr1-dc0 | vr1-dc1 source scripts/lib-hosts.sh; lib_hosts_select_dc "$DC" ``` Do not skip this because it "worked before" on the other DC. For `vr1-dc0` the net selector is a documented no-op (D-101: VR1's first DC inherits VR0 DC0's v4 layout unchanged -- `scripts/lib-net.sh:120-135`), so skipping it happens to produce the right values by accident. For `vr1-dc1` it is NOT a no-op -- that arm DIVERGES (`scripts/lib-net.sh:136-178`: the `10.12.64/68/72/76/80/84` planes, its own gateways, its own FIP pool) -- and skipping it silently reuses `vr1-dc0`'s addresses, exactly the bug DOCFIX-151 exists to prevent. Always call both, for both DCs, unconditionally. A bare `dc0`/`dc1`/`dc2` is REFUSED by the selector, by design (`scripts/lib-net.sh:179-182`). If you see that FAIL, you have typed the retired namespace -- re-type the region-qualified slug, do not "fix" the library. --- ## What is genuinely IDENTICAL across the two VR1 DCs, and what is not **Ruling 3 (2026-07-25): `bundle.yaml` is VIP-FREE (topology only) and EVERY VIP arrives via a per-DC overlay, `vr1-dc0` INCLUDED.** Neither DC needs a VIP *band* change -- both keep the same `.50-.99` host octets -- but each absolutely needs its OWN overlay, because the prefixes differ: `vr1-dc0` is `10.12.4/8/12`, `vr1-dc1` is `10.12.64/68/72` (`scripts/lib-net.sh:54-56` and `:146-154`). Deploying the bare bundle with no overlay gives a control plane of decorative HA, and NO gate catches it at deploy time except `provider-bundle-check`. Both overlays are GENERATED by `scripts/render-dc-overlays.py`; edit `render/values/${DC}-vips.yaml` and re-render rather than hand-editing an overlay. Still correct, and the reason the two DCs read alike: D-101 rules that VR1's FIRST DC (`vr1-dc0`) inherits VR0 DC0's six-plane v4 layout UNCHANGED, and the host octets are identical across DCs. `phase-01..08`'s command blocks that reference `lib-net.sh`/`lib-hosts.sh` values resolve to the same numbers for `vr1-dc0` as they did for the original VR0 DC0 rehearsal -- expected, not a bug (see the `$DC` selector callout above). `vr1-dc1` is where they diverge, which is why the selector call is unconditional. **A THIRD DC would need a distinct, non-overlapping v4 supernet + all-new VIPs -- NOT YET ASSIGNED, and OUT OF SCOPE for this stage.** Per D-101 and tooling gap register item #3, any further DC's supernet, the org ULA /48 and its GUA carve are NetBox-authoritative; the import PIPELINE closed 2026-07-09 (DOCFIX-152) but no literals beyond the two ruled DCs have been assigned. Do not invent a "test range" to get moving faster -- that is exactly the inference hard rule 2 forbids. **This stage deploys `vr1-dc0` and `vr1-dc1` only; both have real, ruled, apex-verified literals today.** --- ## Sequence (repeat entire sequence once per DC; no ruled DC ordering) ``` 1. Confirm Stage 4 gate + call the $DC selectors (read-only) 2. Bootstrap a DEDICATED single-unit Juju controller, IN-DC [MUTATION, gated] 3. preflight.sh PASS for THIS DC (read-only gate) 3.5 add-model, spaces gate, per-DC artifact source [MUTATION, gated] 4. Deploy bundle.yaml + THIS DC's overlays [MUTATION, gated -- see below] 4b. Mid-deploy watch + failure decision tree (read-only, runs alongside 4) 5. Run phase-02 (vault bring-up) -- INDEPENDENT root per DC [adapted, SECRET-HANDLING] 6. Write + apply the IPv6 family-matrix overlay [DRAFTED 2026-07-10, NOT YET APPLIED -- see below] 7. Run phase-03 (core verify) [adapted, gated] 8. Run phase-04 (network carve) [adapted, gated] 9. Run phase-05 (octavia enablement) [adapted, gated] 10. cloud-assert.sh --capture (read-only + committed BOM) 11. Controller backup [MUTATION, gated] 12. VERIFY-LIVE gates + Ceph-over-v6 / geneve-over-v6 (read-only gate) -> EXIT GATE (this DC) -> repeat 1-12 for the other DC -> Stage 6 ``` --- ## Step 1 -- Confirm Stage 4 gate + call the $DC selectors (READ-ONLY) > **RUN LOCATION -- `voffice1`, the D-128 Plane-2 host. NOT the vcloud > jumphost.** Every `juju` and `maas` command in this runbook runs there. > Measured 2026-07-29 on vcloud: `command -v juju` -> ABSENT, `command -v maas` > -> ABSENT. The GA-R5 ruling of 2026-07-27 says it directly: both DCs deploy > from the Office1 headend by the same procedure. Claude's own host only picks > the transport (`ssh voffice1 '...'`); it is not where the work runs. **CHECK (read-only) -- voffice1** ```bash export DC=vr1-dc0 # or vr1-dc1 -- explicit, region-qualified (D-119), never omitted export DC_MODEL="$DC" # ONE juju model per DC, named for the site slug. # Used by every -m below; do NOT leave it unset -- # an unset ${DC_MODEL} expands to -m "" and juju # operates on whatever model happens to be current. export MAAS_PROFILE="${MAAS_PROFILE:-admin}" # same default the scripts use # (scripts/maas-role-tags.sh:37, pre-flight-checks.sh:53); # never `maas list` -- DOCFIX-016. source scripts/lib-net.sh; lib_net_select_dc "$DC" source scripts/lib-hosts.sh; lib_hosts_select_dc "$DC" echo "DC=$DC DC_MODEL=$DC_MODEL" echo "PLANE_CIDRS: ${PLANE_CIDRS[*]}" echo "HOSTS: ${HOSTS[*]}" ``` **GATE:** both selector calls exit 0, and `DC`/`DC_MODEL` echo non-empty. If `lib_hosts_select_dc "$DC"` fails loud, Stage 4 has not populated real host data for this DC -- stop and go back, do not hand-supply the values. --- ## Step 2 -- Bootstrap a dedicated, in-DC, single-unit Juju controller [MUTATION, gated] Per D-104 (as AMENDED): ONE Juju controller PER DC, deployed IN that DC on that DC's OWN dedicated controller VM, single-unit (not HA) for VR1, managing ONLY its own cloud. This is explicitly NOT the same controller model as VR0/DC0's original bootstrap -- a fresh `juju bootstrap` targeting THIS DC's MAAS cloud/region, not a reused controller. **The controller target is a dedicated 10th VM, and it is READY, not Deployed.** Under the READY-handoff ruling (DOCFIX-200, 2026-07-23) Stage 4 hands off nodes in `Ready` with MAAS-deploy SKIPPED -- Juju provisions them here. Any surface saying the bootstrap lands "on a MAAS-deployed machine" is stating the premise that ruling removed. The controller VM is NOT one of the nine bundle machines (`bundle.yaml` machines block: "The Juju CONTROLLER is a dedicated 10th VM (D-104 amendment), bootstrapped by its own tag -- NOT a bundle machine"). **Step 2.0 -- Per-DC MAAS service credential [CHECK FIRST; both may already exist]** Juju authenticates to the MAAS region with an API key. Per the per-DC credential-isolation discipline (SEC-012/-016; Roosevelt analog: per-DC cloud credentials), each DC gets a DEDICATED MAAS admin service user + its OWN key -- never reuse one key across DCs. > **CHECK BEFORE YOU MINT. Both per-DC credentials already existed as of > 2026-07-27** (`vr1-dc0-cred` and `vr1-dc1-cred`, SEC-018 / SEC-019; > `juju show-cloud vr1-maas --client` reported `credential-count: 2`). > Re-running the mint blocks below as written creates a DUPLICATE MAAS admin > user and a SECOND key for a DC that already has one -- credential sprawl > against SEC-018/-019 and D-137, on a register that is already red. Minting a > credential is also a SEC-row obligation, not a free action. **CHECK (read-only) -- voffice1** ```bash juju credentials --client # is "$DC-cred" already listed? maas "$MAAS_PROFILE" users read | grep -o "juju-$DC" || echo "no juju-$DC user" ls -d ~/"$DC"-creds 2>/dev/null || echo "no ~/$DC-creds folder" # SEC-009 site folder ``` (`maas users read` is the enumeration used to account for every MAAS account on this deployment -- `docs/CURRENT-STATE.md:2784`. Do NOT open anything inside the creds folder; its existence is all this gate needs.) **GATE -- three outcomes, and only one of them proceeds:** - This DC's credential is LISTED and its MAAS user exists -> **SKIP the rest of Step 2.0 entirely** and go to the bootstrap CHECK below. Do not re-mint. - Neither exists -> run the mint blocks that follow, and open the per-DC SEC row. - One exists without the other (a user with no registered credential, or a credential naming a user MAAS does not have) -> **STOP.** That is a half-made credential, not a missing one; reconcile it against `docs/security-ledger.md` before touching either side. Do not "make it consistent" by minting the missing half. The mint blocks below run ONLY on the second outcome. On the MAAS region host (VR1: `voffice1`): ```bash # service account -- Juju auths via the API key, not this password; use a random one sudo maas createadmin --username="juju-$DC" --email="juju-$DC@" ``` Generate the key so it lands ONLY in the JUMPHOST per-site creds folder (`~/$DC-creds/maas-api-key.txt` on vcloud -- the SEC-009 `~/-creds/` convention, and the site key IS the D-119 slug; NOT a new location on the region), never echoed to a terminal (a printed MAAS key is burned, SEC-006). Run the generate FROM the jumphost so the key transits ssh stdout straight into the vcloud file (no persistent copy on the region): ```bash ssh "sudo maas apikey --username=juju-$DC --generate" \ > ~/"$DC"-creds/maas-api-key.txt chmod 600 ~/"$DC"-creds/maas-api-key.txt awk -F: '{print "parts:", NF}' ~/"$DC"-creds/maas-api-key.txt # expect 3; do NOT cat ``` Register the MAAS cloud + this credential in Juju consuming the key BY PATH/stdin (never argv). Each key is a rotation obligation -- open a per-DC SEC row when issued. **CHECK (read-only) -- voffice1 -- the cloud is registered AND the controller tag exists** ```bash juju clouds --client | grep -i maas # The bootstrap constraint below is only as good as the tag behind it. Fork 3's # per-role tagger owns the D-104 controller tag (scripts/maas-role-tags.sh:125, # "juju-controller-$SITE") -- confirm THIS DC's tag exists and is on exactly one # machine before constraining on it: bash scripts/maas-role-tags.sh check "$DC" maas "$MAAS_PROFILE" machines read \ | jq -r --arg t "juju-controller-$DC" '.[] | select(.tag_names[]? == $t) | "\(.hostname) \(.status_name)"' ``` **GATE:** the tag `juju-controller-$DC` exists AND the query returns EXACTLY ONE machine, in `Ready`. Zero machines -> the tag was never applied; `juju bootstrap` would find no candidate and fail at allocation. More than one -> STOP; two machines carrying a single-controller tag means the bootstrap target is ambiguous and you cannot predict which it takes. A status other than `Ready` (including a cached `error` power state) is NOT a pass -- refresh it with `maas "$MAAS_PROFILE" machine query-power-state ` and re-read, rather than assuming the node is broken (a stale cached `error` on nodes that are simply off is a measured trap, 2026-07-27). **MUTATION -- voffice1** ```bash juju bootstrap \ --constraints "tags=juju-controller-$DC" ``` **The `--constraints` is not optional.** Without it Juju asks MAAS for any available machine and can land the controller on one of the nine role nodes -- the exact outcome the D-104 amendment rejected, and one you only discover after the bundle deploy finds a node missing. The tag name is DERIVED from `$DC`, so it stays correct for both DCs by construction; it is not a literal to retype. The MAAS cloud name and the controller name are session-specific identifiers you choose/confirm at execution time (reflecting the site slug in the controller name for operator clarity) -- not invented here. Per D-104's rationale: this keeps DR-independence honest (a controller loss affects only its own DC) and is a deliberate VR1 test-only economy (HA deferred to Roosevelt) -- do not "upgrade" this to HA mid-stream without a new D-NNN, that would silently reverse an ADOPTED decision. --- ## Step 3 -- `preflight.sh` PASS for THIS DC (read-only gate) ```bash DC="$DC" bash scripts/preflight.sh ``` **`DC=` is load-bearing -- do not run this bare.** P2 defaults to `vr1-dc0` (`scripts/preflight.sh:99`) and assembles the overlay set from `$DC` (`:100-105`), so a bare run on the other DC validates dc0's numbers against dc0's bands and never parses this DC's artifact at all -- a green gate that examined the wrong input. **Expect** the banner to name THIS DC before anything else: `validating the MERGED deploy input: bundle.yaml --overlay ... --dc `. If that line names a different DC than the one you exported, STOP -- `$DC` did not reach the script. **GATE:** `PREFLIGHT: PASS`. Two failures are known and expected on the vcloud jumphost rather than on `voffice1` (the deliberately-absent gitignored octavia PKI overlay, and MAAS unreachable) -- which is another reason this runs on `voffice1`. A red preflight is a stop, not a note: read every FAIL and decide per row, and record the decision. Do not run the sub-gates piecemeal to get a greener-looking answer. --- ## Step 3.5 -- Model, spaces gate, and this DC's artifact source [MUTATION, gated] Three things that must exist BEFORE the deploy and that no earlier step creates. **MUTATION -- voffice1 -- create THIS DC's model (guarded)** ```bash ( { # Same capture-then-test idiom as scripts/juju-spaces-check.sh:40-42 (DOCFIX-115): # capture first, then test. The live-pipe form races SIGPIPE under pipefail and # reports a juju AUTH failure as "model not present" -- which here would mean # add-model runs against a controller you are not actually talking to. MJSON="$(juju models --format json)" || { echo "ABORT: 'juju models' failed (auth/macaroon? re-login, or append /dev/null | sed 's#.*/##')" || true if grep -qx "$DC_MODEL" <<<"$MNAMES"; then echo "ABORT: model '$DC_MODEL' already exists -- do not deploy into a used model" else juju add-model "$DC_MODEL" fi } ) ``` One model per DC, named for the site slug. Deploying two DCs into a model named `openstack` on one client is how the second deploy lands on top of the first. **CHECK (read-only) -- voffice1 -- per-model spaces** ```bash bash scripts/juju-spaces-check.sh "$DC_MODEL" ``` Spaces are PER-MODEL in Juju, so this can only run after `add-model` and must run before `juju deploy` -- a missing or stale space name mis-binds charms at deploy time and is expensive to unpick afterwards. **GATE:** exit 0. Exit 1 = missing or stale spaces, STOP. Exit 2 = the model is not present yet, which means the block above did not do what you think it did. **MUTATION -- voffice1 -- point this DC's model at ITS OWN artifact source** > **ORDERING TRAP -- this is why the step is here and not later.** If the model > has no artifact source set, every unit apts from the INTERNET during deploy. > The D-135/D-107 egress narrowing then lands on a half-built model, and unlike > bootstrap the recovery is not "re-open egress and retry" -- it is a > partially-installed cloud. Set it BEFORE the deploy, not during. **The two DCs DIFFER BY RULING (D-135 + its 2026-07-24/27 amendments) -- there is no common command here.** `vr1-dc0` runs a FULL MIRROR; `vr1-dc1` runs an APT CACHING PROXY. This is the experiment, not an interim state. Read the address from that DC's OWN checker rather than typing it: ```bash # vr1-dc1 -- apt caching proxy (apt-cacher-ng). Confirm it is up and note its LISTEN: bash scripts/dc-cache-proxy.sh check vr1-dc1 juju model-config -m "$DC_MODEL" apt-http-proxy="http://:3142" juju model-config -m "$DC_MODEL" apt-http-proxy # READ BACK -- assert on content ``` Reference values, for recognising a wrong answer rather than for typing: `scripts/dc-cache-proxy.sh:81,87` give `vr1-dc1` LISTEN `10.12.68.4` port `3142`; `scripts/dc-mirror.sh:79` gives `vr1-dc0` LISTEN `10.12.8.4`. If the checker reports anything else, the checker wins and you stop to reconcile. > **`vr1-dc0` (mirror) -- VERIFY THE KEY NAME BEFORE YOU RUN IT.** The mirror is > consumed via a DIFFERENT model-config key than the proxy. The only in-repo > statement of that key is a comment (`scripts/dc-mirror.sh:234`, > `apt-mirror=http:///ubuntu`), and no juju client exists on the vcloud > jumphost to check it against, so it is NOT written here as an executable line. > At execution, confirm the exact key with `juju model-config --help` (or > `juju model-defaults`) on `voffice1` FIRST, then set it and read it back the > same way as above. Do not guess a key name -- a mistyped model-config key is > accepted silently and leaves the model pointing nowhere. **GATE:** the read-back prints the address you intended, and it is THIS DC's own artifact service. An empty read-back is a FAIL, not a pass -- checking `vr1-dc1` as a mirror (or `vr1-dc0` as a proxy) fails BY DESIGN and is not a reason to skip the step. --- ## Step 4 -- Deploy `bundle.yaml` + THIS DC's overlays [MUTATION, gated] This step carries its OWN DC-parameterised deploy command. It no longer says "follow phase-01 verbatim": phase-01's deploy blocks were written around VR0's single-DC testcloud, and following them unchanged against a VR1 DC is a known defect class, not a shortcut. Use `runbooks/phase-01-bundle-deploy.md` for the things that genuinely transfer -- Step 1.0's Octavia PKI overlay sanity, the convergence-watch pattern, and the PRE-vault-init end-state description -- with `$DC` and `$DC_MODEL` exported per Step 1 above. **Dependency: phase-01's own per-DC parameterisation (its `$DC`/`$DC_MODEL` handling and its per-DC VIP guard) is delivered separately in this same batch; if you reach a phase-01 block that still hardcodes `10.12.4.`, VR0 system_ids, `openstack0..3` disk paths, a 4-machine plan, or `juju add-model openstack`, that block has NOT been parameterised yet -- STOP and reconcile rather than adapting it live.** ### 4.1 -- Assemble THIS DC's overlay set Deploy input = `bundle.yaml` + this DC's overlays. The base bundle is VIP-FREE and is NOT a deployable input on its own. | Overlay | `vr1-dc0` | `vr1-dc1` | |---|---|---| | `overlays/${DC}-vips.yaml` | REQUIRED (`vr1-dc0-vips.yaml`) | REQUIRED (`vr1-dc1-vips.yaml`) | | machines retag | **none -- and that is correct**: the base bundle's machines block already carries `tags=openstack-vr1-dc0,` | REQUIRED: `overlays/vr1-dc1-machines.yaml` | | `overlays/${DC}-octavia-pki.yaml` | REQUIRED (gitignored secret) | REQUIRED (gitignored secret) | **Do not interpolate a machines overlay path for `vr1-dc0`** -- there is no `overlays/vr1-dc0-machines.yaml`, and juju errors out on a missing `--overlay` path. The asymmetry is deliberate; the gate that catches BOTH arms is the merged tag assertion in 4.2, not the presence of a file. Never "guard" the argument with a silent skip: an overlay that quietly disappears is exactly how a `vr1-dc1` deploy would merge `vr1-dc0`'s tags. Make NO hand edit to `bundle.yaml` or to either generated VIP overlay (ruling 3, 2026-07-25; both `*-vips.yaml` are RENDERED -- edit `render/values/${DC}-vips.yaml` and re-render, and `tests/render-drift` fails the gauntlet if you do not). **`overlays/dc-ha-scaleup.yaml` is DELIBERATELY NOT in this command.** Per R6 the VIPs land BEFORE the HA overlay; applying it is a separate, later step once the base deploy has settled. That is also why the base deploy legitimately shows `cluster_count: 1` -- see Step 12's gate, which states when `1` is correct and when it is the failure. **Octavia PKI is PER-DC.** Under the R7 amendment (D-109, RULED 2026-07-27) the generator is no longer dc0-frozen and each DC gets its own CA material, so this is a per-DC artifact (`overlays/${DC}-octavia-pki.yaml`), not one shared file. Reuse this DC's existing overlay if its CAs are still valid (10y, per phase-01 Step 1.0), OR regenerate via phase-01 Step 1.0-GEN with `DC` exported. Either way it is an explicit operator call, logged as a decision note -- not a silent default to reuse. Cross-DC reuse was refused on posture: the overlay carries CA private keys plus a plaintext passphrase in a repo SEC-004 records as PUBLIC. ### 4.2 -- DRY-RUN FIRST, and read the machines block [read-only gate] > **Run this BEFORE 4.3. It is the only thing standing between a typo in the > overlay set and nine wrongly-allocated MAAS machines.** There is no offline > juju merger that can prove the machines-block override merges (see > `overlays/vr1-dc1-machines.yaml`'s own VERIFY-LIVE note), so the dry-run IS > the proof. **CHECK (read-only) -- voffice1 -- run the block for YOUR DC.** These are EXACTLY the 4.3 commands plus `--dry-run`; if the two ever differ, this gate is validating something you are not deploying. `vr1-dc0` (no machines overlay -- the base bundle already carries its tags): ```bash juju deploy ./bundle.yaml \ --overlay overlays/vr1-dc0-vips.yaml \ --overlay overlays/vr1-dc0-octavia-pki.yaml \ -m "${DC_MODEL}" --dry-run ``` `vr1-dc1` (machines overlay REQUIRED, order: vips -> machines -> pki): ```bash juju deploy ./bundle.yaml \ --overlay overlays/vr1-dc1-vips.yaml \ --overlay overlays/vr1-dc1-machines.yaml \ --overlay overlays/vr1-dc1-octavia-pki.yaml \ -m "${DC_MODEL}" --dry-run ``` **GATE -- assert on the CONTENT of the plan, not on the command exiting 0:** 1. **Every machine reads `tags=openstack-$DC,`.** If any machine still names the OTHER DC, STOP -- one Office1 MAAS region sees BOTH DCs' nodes, so a wrong tag here means this model can allocate the other DC's hardware. The fallback is inlining this DC's tags into a rendered per-DC bundle, not deploying anyway. 2. **Nine machines, `"0"` through `"8"`**, matching `bundle.yaml`'s machines block (D-121 Option C: 3 control / 2 compute / 4 storage). A four-machine plan means you are looking at VR0's topology. 3. **Every VIP in the plan is on THIS DC's prefixes** -- `10.12.4/8/12` for `vr1-dc0`, `10.12.64/68/72` for `vr1-dc1` (`scripts/lib-net.sh:54-56`, `:146-154`). A VIP from the other DC means the wrong overlay merged. 4. If the plan reports an ERROR on any `--overlay` path, do NOT drop the overlay to make it run. A missing overlay is a missing input, not an optional one. Capture this output. It is the plan-count evidence the exit gate cites -- plan counts are quoted only from captured command output, never from this file's prose (GA-R1 rule 2). ### 4.3 -- The real deploy [MUTATION, gated] **The SAME block you dry-ran in 4.2, minus `--dry-run`. Nothing else changes.** `vr1-dc0`: ```bash juju deploy ./bundle.yaml \ --overlay overlays/vr1-dc0-vips.yaml \ --overlay overlays/vr1-dc0-octavia-pki.yaml \ -m "${DC_MODEL}" ``` `vr1-dc1`: ```bash juju deploy ./bundle.yaml \ --overlay overlays/vr1-dc1-vips.yaml \ --overlay overlays/vr1-dc1-machines.yaml \ --overlay overlays/vr1-dc1-octavia-pki.yaml \ -m "${DC_MODEL}" ``` Open the Step 4b watch windows before this returns. - `bundle.yaml` ships "NO designate" for THIS stage per phase-01's own gate text (D-019, as superseded by D-106): Designate reactivation is Stage 7's job, do not reactivate it mid-Stage-5. - Settle to phase-01's documented PRE-vault-init end state, read with `$DC_MODEL` substituted for its hardcoded model name: ZERO units in `error`; mysql-innodb-cluster ACTIVE; `vault/0` BLOCKED "Vault needs to be initialized" (the Step 5 trigger, not a fault); the TLS consumers waiting on vault certs; octavia BLOCKED "Awaiting configure-resources". --- ## Step 4b -- Mid-deploy watch + failure decision tree (read-only, runs alongside Step 4) Open BOTH windows before Step 4.3 returns. A deploy you are not watching is one you diagnose from logs afterwards. **CHECK (read-only) -- voffice1, window 1 (detail)** ```bash juju status -m "$DC_MODEL" --watch 5s ``` **CHECK (read-only) -- voffice1, window 2 (signal)** ```bash bash scripts/deploy-watch.sh "$DC_MODEL" 15 ``` Compact machine/unit state COUNTS plus the NAMED principal units in error/blocked. Health-at-a-glance: the error/blocked section stays EMPTY until the expected late blocks (vault needs-init, octavia awaiting-configure). Neither window descends into subordinates and neither replaces a phase gate. **Expected end-state shape** (not a count to quote -- confirm against Step 4.2's captured plan, GA-R1 rule 2): nine machines reaching `started`; zero units in `error`; the only `blocked` units are `vault/0` (needs init) and `octavia` (awaiting configure-resources). Anything else named in the error/blocked list is this tree's input. ### Decision tree -- something went wrong mid-deploy Work top to bottom. **Default is FIX-FORWARD**; juju converges, and tearing down a partially-good model costs more than repairing it. 1. **Units are still `waiting`/`maintenance` and counts are MOVING.** Not a failure. Snap installs and cert exchange are slow on first boot. Keep watching; do not intervene because it is quiet. 2. **A machine never leaves `pending` / allocation fails.** This is a MAAS-side answer, not a juju one. Check this DC's node pool: the right tag on the right count of `Ready` machines (Step 2's tag check, widened to the role tags). A cached `power_state=error` on nodes that are merely off reads as a hard blocker and is not one -- refresh with `maas "$MAAS_PROFILE" machine query-power-state ` and re-read before diagnosing a fault. **Fix-forward:** correct the tags/power state and let juju retry. Do NOT destroy the model. 3. **Units are in `error` with a hook failure.** Read the actual hook error (`juju debug-log -m "$DC_MODEL" --replay --include `). If it is a transient fetch failure, first check whether this model's artifact source (Step 3.5) is set and reachable -- a model apting from the internet behind a narrowed edge fails exactly this way. **Fix-forward:** repair the cause, then `juju resolved --no-retry ` or `juju resolved ` as appropriate. 4. **The wrong DC's hardware got allocated, or a VIP from the other DC appears in the model.** STOP. This is the one case where fix-forward is wrong: the model has bound to another DC's addresses/machines, and repairing it in place leaves you unsure what is bound to what. Tear the MODEL down (`juju destroy-model "$DC_MODEL"`, operator-gated) and return to Step 4.2 -- the dry-run gate exists precisely to make this case unreachable. 5. **You are considering destroying anything BELOW the juju layer** (the containment VM, the DC's libvirt resources, MAAS records). Read `runbooks/dc-dc-teardown-rollback.md`'s rollback decision tree FIRST, and read its Model-B banner before acting on any of its resource-level steps. **Standing caution as of 2026-07-29:** that runbook's Step 2 MAAS-side check and Step 3 scoped-destroy targets have known defects against the current two-root shape -- treat both as reference, verify every module name against the live `opentofu/` tree before running anything, and never let a whole-DC destroy be the first thing you reach for while 18 nodes are enrolled in MAAS. At every branch: record what you observed and what you chose. A mid-deploy decision that is not written down is one the next session has to re-derive. --- ## Step 5 -- Run phase-02 (vault bring-up) -- INDEPENDENT root per DC [adapted, SECRET-HANDLING] Follow `runbooks/phase-02-vault-bringup.md` per DC, with `$DC`/`$DC_MODEL` exported per Step 1 and substituted for any hardcoded model name in its command blocks. Per D-109: each DC's Vault is its OWN independent root CA -- no regional root-of-trust this phase. This means: - The `vault operator init` one-shot step (Step 2.1) runs ONCE PER DC, producing DISTINCT unseal-key shares and a DISTINCT root token for `vr1-dc0` vs. `vr1-dc1`. Store them separately, clearly labeled by DC (this repo's existing secrets discipline -- `~/vault-init/`-style off-host capture -- applies identically per DC, just not shared between them). - D-109 also requires each DC's Vault to issue certs with BOTH IPv4 and IPv6 (ULA) SANs for the dual-stacked metal-internal API endpoints. **This is a real charm-behavior question not yet confirmed this session:** whether the `vault`/`vault-pki`-relation charms used in `bundle.yaml` actually support requesting/issuing IPv6 SANs today (vs. only IPv4) is UNVERIFIED -- check the actual charm config/relation-data schema (e.g. `openstack-loadbalancer`/certificates interface docs, or the vault charm's own config options) before assuming this "just works" once IPv6 addresses exist on the plane. If it does not support v6 SANs natively, that is a real finding to log (a D-NNN amendment or a new decision), not something to route around silently. --- ## Step 6 -- Write + apply the IPv6 family-matrix overlay [OVERLAY DRAFTED 2026-07-10, NOT YET APPLIED] D-101's family matrix (Section 2 of the buildout design) requires, per DC: - `data-tenant`, `storage`, `replication`: IPv6-ONLY ULA (no v4 config at all on these planes for the relevant charms -- e.g. `ceph-mon`'s `public-network`/`cluster-network`, OVN's geneve encapsulation address). - `metal-internal`, `metal-admin`: DUAL-STACK (v4 retained + a ULA leg added -- the metal-admin ULA leg is the 2026-07-09 operator AMENDMENT to D-101, a reversal of that decision's own original lean, so do not assume older documentation/training-data patterns for "metal-admin stays v4-only" -- this repo's ADOPTED text says otherwise). - `provider-public`: DUAL-STACK v4 + GUA (native, no NAT66 -- D-101's tenant/MTU sub-policy folded in from D-102). **`overlays/dc-dc-ipv6-family-matrix.yaml` now exists**, drafted 2026-07-10 against REAL charm source/config (not guessed option names) -- see `docs/archive/dc-dc-ipv6-charm-research.md` for full sourcing and per-charm confidence level. Summary: `prefer-ipv6` (confirmed real option, additive dual-stack HAProxy binding, confirmed via `charm-nova-cloud-controller`'s actual "Dual Stack VIPs" commit) + appended v6/GUA entries on the existing `vip:` option, for the nine dual-stack API charms directly confirmed or sharing the same charms.openstack framework; `ceph-mon`'s own separate `prefer-ipv6` + `ceph-public-network`/`ceph-cluster-network` (a clean switch-to-v6-only, correct for the ULA-only planes); NO entry for OVN (confirmed no such charm-config option exists -- geneve family follows the bound interface automatically once the plane itself is ULA-only). **This is an INCREMENTAL ADD against the LIVE `$DC_MODEL` Step 4 already deployed -- not a fresh deploy.** It re-passes the SAME overlay set Step 4 used and adds the family matrix on top, so juju's diff introduces only the address-family options; the base topology, the VIPs and the machines block must come out UNCHANGED. That is precisely what the GATE below checks. Re-passing the full set is deliberate: dropping the VIP or machines overlay here would present juju with an input that contradicts the running model. Corrected 2026-07-29: this command previously named `overlays/octavia-pki.yaml` and `overlays/${DC}-hostnames.yaml` and passed NO VIP overlay at all -- so since ruling 3 its merged input carried ZERO VIPs, and `${DC}-hostnames.yaml` has NEVER BEEN AUTHORED (it is unbuilt D-106 work, proposed in `runbooks/dc-dc-phase6-designate-cos-magnum.md`), on which juju errors out. > **THIS STEP HAS NOT BEEN APPLIED TO ANY LIVE MODEL** (overlay drafted > 2026-07-10, still unvalidated -- see open item 3 below). Every block here is > `--dry-run`; the real apply is a separate, gated decision once the diff has > been reviewed. **Step 12's v6 gates depend on this step having actually been > applied** -- if you have only dry-run it, 12.1 and 12.2 are BLOCKED, and > "blocked on Step 6" is the honest recorded answer, not a soft pass. `vr1-dc0`: ```bash juju deploy ./bundle.yaml \ --overlay overlays/vr1-dc0-vips.yaml \ --overlay overlays/vr1-dc0-octavia-pki.yaml \ --overlay overlays/dc-dc-ipv6-family-matrix.yaml \ -m "${DC_MODEL}" --dry-run ``` `vr1-dc1`: ```bash juju deploy ./bundle.yaml \ --overlay overlays/vr1-dc1-vips.yaml \ --overlay overlays/vr1-dc1-machines.yaml \ --overlay overlays/vr1-dc1-octavia-pki.yaml \ --overlay overlays/dc-dc-ipv6-family-matrix.yaml \ -m "${DC_MODEL}" --dry-run ``` Same overlay-set rules as Step 4.1: `vr1-dc0` takes no machines overlay because the base bundle already carries its tags, and no overlay is ever dropped to make the command run. Review the diff before applying for real, same discipline as every other overlay in this repo. **GATE:** the diff changes ONLY the address-family options this overlay sets; every VIP still reads on THIS DC's prefixes, and the machines block still reads `tags=openstack-$DC,`. If either moved, the overlay set is wrong -- stop, do not apply. **Still genuinely open, not resolved by the overlay:** 1. The 8 charms whose `prefer-ipv6` presence is INFERRED-BY-PATTERN (not individually fetched from their own `config.yaml`) need live `juju config ` confirmation once first deployed. 2. **RESOLVED 2026-07-27 by R8 -- this item is CLOSED and the text below is superseded.** Octavia's `lb-mgmt-net` is **IPv6 ULA, charm-created and charm-owned** (operator: "I want to take a Octavia creates and owns its own IPv6 network"). The two Launchpad numbers this item rested on were READ IN FULL and BOTH FAILED: #1913409 is **kolla-ansible**, a different installer, **Fix Released 2021**; #1911788 is **Incomplete**, a duplicate, and **is not an IPv6 defect** -- its cause is an OVN shortname/FQDN port-binding mismatch fixed in OVS 2.15, and these nodes install 2.17. Upstream research also found the charm creates a v6 subnet and **has no IPv4 code path at all**, so v4-only was never the safe option -- it was the departure. Do NOT re-raise this as an open decision. What IS owed here is the MTU obligation (LP #2018998), now asserted by `scripts/phase-05-octavia-verify.sh`, and gate **G18** (apex back-fill of the charm-created prefix), which blocks project close. 3. UNVALIDATED against any live model -- this overlay has not been applied or tested, same posture as every other artifact from tonight's prep-only session. --- ## Steps 7-9 -- phase-03 / phase-04 / phase-05, adapted per DC Follow `runbooks/phase-03-core-verify.md`, `runbooks/phase-04-network- carve.md`, `runbooks/phase-05-octavia-enablement.md` per DC, with `$DC` and `$DC_MODEL` exported per Step 1, the selector discipline applied to every command block in them that reads `lib-net.sh`/`lib-hosts.sh` values, and `$DC_MODEL` substituted for any hardcoded model name. **Known, measured caveat -- these three phases call scripts that do NOT select a DC.** Measured 2026-07-29: of the scripts that source `scripts/lib-net.sh`, these Stage-5 ones read a DC-dependent value while never calling `lib_net_select_dc`, so they resolve `vr1-dc0`'s literals whichever DC you are on: `scripts/phase-03-admin-openrc.sh` (`KEYSTONE_VIP_DEFAULT`), `scripts/phase-04-network-create.sh` and `scripts/phase-04-network-verify.sh` (both also carry a hardcoded `PROVIDER_CIDR="10.12.4.0/22"`, which a selector would not reach), and `scripts/vault-kv-health.sh` (`METAL_INTERNAL_CIDR`). On `vr1-dc0` that is harmless and correct. **On `vr1-dc1` it is wrong and SILENT** -- no error, just dc0's addresses. Until those scripts are fixed, either export the value explicitly (each honours an env override) or verify every address in their output against this DC's planes before believing a PASS. LOG anything further you find (a DOCFIX candidate); do not patch around it live. --- ## Step 10 -- `cloud-assert.sh --capture` (read-only + committed BOM) ```bash bash scripts/cloud-assert.sh --capture ``` Per DC, at real deploy-completion. Confirmed repo-agnostic (2026-07-09 sweep). Commit the resulting `asbuilt/` BOM per this repo's existing deploy-loop convention, labeled clearly by DC (do not let `vr1-dc0`'s and `vr1-dc1`'s BOMs collide/overwrite -- confirm the script's own output path is already DC-distinguishing, or add a DC-qualified path if not, BEFORE running for the second DC; a silently overwritten BOM loses the first DC's evidence). --- ## Step 11 -- Controller backup [MUTATION, gated] > **THE FLAG SHAPE BELOW IS UNVERIFIED AT AUTHORING -- CONFIRM IT FIRST.** > No juju client exists on the vcloud jumphost (measured 2026-07-29: > `command -v juju` -> ABSENT), so this shape could not be checked against a > real client when it was written, and a repo-wide grep corroborates the > subcommands only from other prose. What IS established: `-m` takes a MODEL, > not a controller name -- so the previous form here (`-m `) > was wrong on both lines; and `create-backup` downloads by default, which made > the old second line redundant rather than necessary. **At execution, run > `juju create-backup --help` on `voffice1` and use what it says.** Then gate on > the resulting FILE, which is true regardless of flag spelling. ```bash # Confirm the flags FIRST (see the caveat above): juju create-backup --help # Then take the backup against the CONTROLLER MODEL of this DC's controller: juju switch juju create-backup -m controller ``` **GATE -- assert on the artifact, not on the command exiting 0:** a backup file exists in the working directory, is NON-ZERO in size, and its timestamp is from this run. Record its name and size. If `create-backup` printed a path but no file is there, that is a FAIL -- do not record "backup taken". Then store it OFF-HOST, labeled by DC, with the same secrets-adjacent handling discipline as Vault's init capture, and confirm the off-host copy's size matches before deleting anything locally. Per D-104: scheduled controller backups, one per DC. The RESTORE half of this drill is exercised in Stage 6 (DR wiring and failover drill) -- this step only takes and stores the backup, it does not restore it. --- ## Step 12 -- VERIFY-LIVE gates + Ceph-over-v6 / geneve-over-v6 (READ-ONLY gate) `juju run` on juju 3.x is the ACTION runner, so the old `juju run 'ceph -s'` form asked ceph-mon to run an action NAMED `ceph -s` and could never execute. `juju exec` is the command runner; the `/leader` resolves for a subordinate is NOT verified from this repo. The precedent that IS established here uses a NAMED unit, not `/leader` (`runbooks/phase-05-octavia-enablement.md:93` -> `--unit octavia/0`). So: ```bash # 1. list this DC's chassis units and pick one that is on a COMPUTE host: juju status -m "$DC_MODEL" ovn-chassis # 2. probe THAT unit by name (substitute for ovn-chassis/0): juju exec -m "$DC_MODEL" --unit ovn-chassis/0 -- \ 'sudo ovs-vsctl get Open_vSwitch . external_ids' `data-tenant`, after `lib_net_select_dc "$DC"`) -> PASS. - An encapsulation address is reported and it is IPv4, or outside that prefix -> FAIL. The chassis is tunnelling on the wrong plane or the wrong family. - No encapsulation entry appears at all, or the command errors -> **REFUSE, do not pass.** "Could not look" is never "nothing there". Record the raw output, confirm the unit name from step 1 is one juju actually lists, and check Step 6 was APPLIED rather than only dry-run. If it still shows nothing, this gate is BLOCKED and is recorded as BLOCKED -- never as a pass. ### 12.3 -- The four VERIFY-LIVE gates Each asserts a RULED value that no offline gate can reach. Sources: `docs/CURRENT-STATE.md:247-252` (the VERIFY-LIVE list) and `docs/design-decisions.md:824` (the keystone `PO:` mechanism). **(a) hacluster `cluster_count` -- and WHEN each value is correct.** ```bash juju config -m "$DC_MODEL" -hacluster cluster_count ``` **GATE, and read the precondition before judging it:** after the Step 4 base deploy, `cluster_count: 1` is EXPECTED AND CORRECT -- the base bundle pins it deliberately and `overlays/dc-ha-scaleup.yaml` is excluded from Step 4 per R6 (`overlays/dc-ha-scaleup.yaml:51-58`). The `3` assertion applies ONLY after the HA scale-up overlay has been applied; at THAT point a `cluster_count` still reading `1` behind a 3-unit principal is the failure -- pacemaker will bootstrap a VIP off a single node and every offline checker stays green. State which side of the overlay you are on when you record this result. **(b) Ceph pool redundancy.** ```bash juju exec -m "$DC_MODEL" --unit ceph-mon/leader -- 'ceph osd pool ls detail' preflight.sh` PASS before add-model (Step 3); the Step 4.2 dry-run captured and its machines-block tags confirmed; post-deploy `cloud-assert.sh --capture` per DC (Step 10); controller backup taken + stored, gated on the FILE (Step 11); the four VERIFY-LIVE gates plus Ceph-over-v6 and geneve-over-v6 recorded (Step 12, the v6 halves contingent on Step 6). -> Repeat this entire runbook for the other DC. **There is no ruled DC ordering for this deployment (GA-R5, 2026-07-27): both DCs deploy from the Office1 headend by the same procedure, in whichever order the operator chooses.** -> proceed to Stage 6 (DR wiring and failover drill) once BOTH DCs pass this gate. --- ## Delivery checklist - [ ] `bash scripts/repo-lint.sh` clean (0 fail) before committing any repo changes made while executing this runbook (the IPv6 overlay file, any re-rendered per-DC VIP overlay -- NOT a `bundle.yaml` edit, which carries no VIPs since ruling 3 -- any DC-qualified `asbuilt/` path fix). - [ ] The Step 4.2 dry-run output CAPTURED to a dated file and cited in the commit that closes this stage for that DC (plan counts are quoted only from captured output, GA-R1 rule 2). - [ ] Changelog entry per real repo change this runbook produces (the overlay, any bundle edit, any script fix) -- next DOCFIX number via `bash scripts/ledger-scan.sh`, same as every other delivery this session. - [ ] `docs/session-ledger.md` updated with the outcome, per DC. - [ ] `docs/dc-dc-deployment-workflow.md` Stage 5 row updated once BOTH DCs have actually passed Step 12's gate (not just the first one done). - [ ] Log the Step 6 charm-config research (v6 SAN support, v6 addressing option names) as its own follow-up item if not resolved inline during this stage's real execution -- do not leave it silently unresolved in the workflow doc's tooling gap register.