DC-DC Phase 4 -- Juju controller + OpenStack bundle, per DC (Stage 5)

NAMESPACE (D-119, ADOPTED 2026-07-14) -- this runbook speaks ONLY region-qualified site slugs: vr1-dc0 and vr1-dc1. The bare DC1/DC2 and dc1/dc2 tokens this file used to carry are RETIRED. They were ambiguous in the worst possible way: read positionally, "DC1" meant VR1's FIRST DC (vr1-dc0) in this file's selector callout and VR1's SECOND DC (vr1-dc1) in its deploy step -- the same token naming two different datacenters in one document. scripts/lib-net.sh:179-182 REJECTS a bare dc0/dc1/dc2 outright, so any command that still carried one could not run. Everywhere below, $DC is exactly one of vr1-dc0 | vr1-dc1.

There is NO ruled DC ordering for this deployment (GA-R5, 2026-07-27). Both DCs deploy from the Office1 headend by the SAME procedure. Run this runbook once per DC, in whichever order the operator chooses; nothing here depends on one DC preceding the other.

Stand up a running, independent OpenStack cloud in one DC, then repeat the same loop for the other. This is the ground-truth/copy-point stage: docs/dc-dc-buildout- design.md line 12 states this repo was seeded from openstack-caracal-ipv4 at commit 8813efc specifically so bundle.yaml and runbooks/phase-01..08- *.md could be reused here almost unchanged. This runbook does not duplicate those eight runbooks' command-level content. It describes what changes to run them TWICE, independently, in the VR1 two-DC dual-stack context, and calls out the real adaptation work plainly rather than asserting "just re-run it" glosses over anything.

Governing docs: docs/dc-dc-buildout-design.md Section 4 "Phase 4" entry and Section 2 (address-family table); docs/design-decisions.md D-104 (controller topology), D-109 (per-DC independent Vault roots + IPv6-SAN certs), D-101 (address plan / family matrix); docs/dc-dc-deployment- workflow.md Stage 5 row ("This is where the existing repo tooling (preflight.sh, cloud-assert.sh, lib-net.sh/lib-hosts.sh, run-logged.sh) does the most direct work") and tooling gap register item #1 (the $DC selector convention, CLOSED 2026-07-09 DOCFIX-151 -- this is the stage that actually calls it).

Prerequisites (must be true entering this stage, per DC): Stage 4 (runbooks/dc-dc-phase3-maas-enlist-deploy.md) done for that DC -- nodes READY (NOT Deployed: MAAS-deploy is SKIPPED per the 2026-07-23 READY-handoff ruling / DOCFIX-200 -- THIS runbook is what provisions them, and expecting Deployed nodes here is exactly the breakage that ruling prevented); six planes present with correct fabrics/VLANs; provider NIC raw; PXE (v4) working; that DC's own artifact source reachable on its own address (vr1-dc0 = the D-135 item-1 mirror, vr1-dc1 = the D-135-AMENDED apt caching proxy consumed via juju apt-http-proxy -- the two DCs differ BY RULING, so do not assume a mirror at vr1-dc1); scripts/lib-hosts.sh's per-DC host data populated (that stage's own follow-up delivery) so lib_hosts_select_dc "$DC" no longer fails loud for it.

THIS STAGE OWES GATE G17 (inherited from Stage 4, split out 2026-07-27 by operator ruling -- see the G17 row in docs/CURRENT-STATE.md, the authority). Stage 4 could not verify that a NODE reaches its DC's artifact source, or that a node's time source is the MAAS hierarchy, because under the READY-handoff ruling its nodes never boot an OS. First boot in this stage is the first moment either is observable, so capture both then -- per DC, from a booted node:

vr1-dc0: the mirror at 10.12.8.4 answers -- and assert on the BODY, not on curl exiting 0: curl -sI returns 0 on a 404 or a 500, and the dc0 URL is an autoindex root that answers 200 with nothing behind it. Fetch a real index file and check its content.
vr1-dc1: apt fetches through the proxy at 10.12.68.4:3142 (NOT a mirror -- D-135 amendment); confirm from the apt output that the fetch went THROUGH the proxy, not around it.
both: chronyc sources shows the MAAS-served time source, NOT the DC edge (D-129(iv) removed the edge from that path).

Capture the output to a dated file and cite it in the commit that closes G17. If first boot happens without these captures being taken, the observation window has passed and G17 needs a deliberate rescue-boot to answer -- so take them the first time.

!!! Every command block in the referenced phase-01..08 runbooks that reads scripts/lib-net.sh or scripts/lib-hosts.sh values (VIP bands, host octets, KEYSTONE_VIP_DEFAULT, etc.) MUST be preceded, in THIS stage's session, by:

```bash
source scripts/lib-net.sh;   lib_net_select_dc   "$DC"   # $DC = vr1-dc0 | vr1-dc1
source scripts/lib-hosts.sh; lib_hosts_select_dc "$DC"
```
Do not skip this because it "worked before" on the other DC. For
`vr1-dc0` the net selector is a documented no-op (D-101: VR1's first DC
inherits VR0 DC0's v4 layout unchanged -- `scripts/lib-net.sh:120-135`),
so skipping it happens to produce the right values by accident. For
`vr1-dc1` it is NOT a no-op -- that arm DIVERGES (`scripts/lib-net.sh:136-178`:
the `10.12.64/68/72/76/80/84` planes, its own gateways, its own FIP pool)
-- and skipping it silently reuses `vr1-dc0`'s addresses, exactly the bug
DOCFIX-151 exists to prevent. Always call both, for both DCs,
unconditionally.

A bare `dc0`/`dc1`/`dc2` is REFUSED by the selector, by design
(`scripts/lib-net.sh:179-182`). If you see that FAIL, you have typed the
retired namespace -- re-type the region-qualified slug, do not "fix" the
library.

What is genuinely IDENTICAL across the two VR1 DCs, and what is not

Ruling 3 (2026-07-25): bundle.yaml is VIP-FREE (topology only) and EVERY VIP arrives via a per-DC overlay, vr1-dc0 INCLUDED. Neither DC needs a VIP band change -- both keep the same .50-.99 host octets -- but each absolutely needs its OWN overlay, because the prefixes differ: vr1-dc0 is 10.12.4/8/12, vr1-dc1 is 10.12.64/68/72 (scripts/lib-net.sh:54-56 and :146-154). Deploying the bare bundle with no overlay gives a control plane of decorative HA, and NO gate catches it at deploy time except provider-bundle-check. Both overlays are GENERATED by scripts/render-dc-overlays.py; edit render/values/${DC}-vips.yaml and re-render rather than hand-editing an overlay.

Still correct, and the reason the two DCs read alike: D-101 rules that VR1's FIRST DC (vr1-dc0) inherits VR0 DC0's six-plane v4 layout UNCHANGED, and the host octets are identical across DCs. phase-01..08's command blocks that reference lib-net.sh/lib-hosts.sh values resolve to the same numbers for vr1-dc0 as they did for the original VR0 DC0 rehearsal -- expected, not a bug (see the $DC selector callout above). vr1-dc1 is where they diverge, which is why the selector call is unconditional.

A THIRD DC would need a distinct, non-overlapping v4 supernet + all-new VIPs -- NOT YET ASSIGNED, and OUT OF SCOPE for this stage. Per D-101 and tooling gap register item #3, any further DC's supernet, the org ULA /48 and its GUA carve are NetBox-authoritative; the import PIPELINE closed 2026-07-09 (DOCFIX-152) but no literals beyond the two ruled DCs have been assigned. Do not invent a "test range" to get moving faster -- that is exactly the inference hard rule 2 forbids. This stage deploys vr1-dc0 and vr1-dc1 only; both have real, ruled, apex-verified literals today.

Sequence (repeat entire sequence once per DC; no ruled DC ordering)

1.  Confirm Stage 4 gate + call the $DC selectors        (read-only)
2.  Bootstrap a DEDICATED single-unit Juju controller, IN-DC  [MUTATION, gated]
3.  preflight.sh PASS for THIS DC                         (read-only gate)
3.5 add-model, spaces gate, per-DC artifact source        [MUTATION, gated]
4.  Deploy bundle.yaml + THIS DC's overlays               [MUTATION, gated -- see below]
4b. Mid-deploy watch + failure decision tree              (read-only, runs alongside 4)
5.  Run phase-02 (vault bring-up) -- INDEPENDENT root per DC [adapted, SECRET-HANDLING]
6.  Write + apply the IPv6 family-matrix overlay          [DRAFTED 2026-07-10, NOT YET APPLIED -- see below]
7.  Run phase-03 (core verify)                            [adapted, gated]
8.  Run phase-04 (network carve)                          [adapted, gated]
9.  Run phase-05 (octavia enablement)                     [adapted, gated]
10. cloud-assert.sh --capture                             (read-only + committed BOM)
11. Controller backup                                     [MUTATION, gated]
12. VERIFY-LIVE gates + Ceph-over-v6 / geneve-over-v6     (read-only gate)
    -> EXIT GATE (this DC) -> repeat 1-12 for the other DC -> Stage 6

Step 1 -- Confirm Stage 4 gate + call the $DC selectors (READ-ONLY)

RUN LOCATION -- voffice1, the D-128 Plane-2 host. NOT the vcloud jumphost. Every juju and maas command in this runbook runs there. Measured 2026-07-29 on vcloud: command -v juju -> ABSENT, command -v maas -> ABSENT. The GA-R5 ruling of 2026-07-27 says it directly: both DCs deploy from the Office1 headend by the same procedure. Claude's own host only picks the transport (ssh voffice1 '...'); it is not where the work runs.

CHECK (read-only) -- voffice1

export DC=vr1-dc0            # or vr1-dc1 -- explicit, region-qualified (D-119), never omitted
export DC_MODEL="$DC"        # ONE juju model per DC, named for the site slug.
                             # Used by every -m below; do NOT leave it unset --
                             # an unset ${DC_MODEL} expands to -m "" and juju
                             # operates on whatever model happens to be current.
export MAAS_PROFILE="${MAAS_PROFILE:-admin}"   # same default the scripts use
                             # (scripts/maas-role-tags.sh:37, pre-flight-checks.sh:53);
                             # never `maas list` -- DOCFIX-016.
source scripts/lib-net.sh;   lib_net_select_dc   "$DC"
source scripts/lib-hosts.sh; lib_hosts_select_dc "$DC"
echo "DC=$DC  DC_MODEL=$DC_MODEL"
echo "PLANE_CIDRS: ${PLANE_CIDRS[*]}"
echo "HOSTS: ${HOSTS[*]}"

GATE: both selector calls exit 0, and DC/DC_MODEL echo non-empty. If lib_hosts_select_dc "$DC" fails loud, Stage 4 has not populated real host data for this DC -- stop and go back, do not hand-supply the values.

Step 2 -- Bootstrap a dedicated, in-DC, single-unit Juju controller [MUTATION, gated]

Per D-104 (as AMENDED): ONE Juju controller PER DC, deployed IN that DC on that DC's OWN dedicated controller VM, single-unit (not HA) for VR1, managing ONLY its own cloud. This is explicitly NOT the same controller model as VR0/DC0's original bootstrap -- a fresh juju bootstrap targeting THIS DC's MAAS cloud/region, not a reused controller.

The controller target is a dedicated 10th VM, and it is READY, not Deployed. Under the READY-handoff ruling (DOCFIX-200, 2026-07-23) Stage 4 hands off nodes in Ready with MAAS-deploy SKIPPED -- Juju provisions them here. Any surface saying the bootstrap lands "on a MAAS-deployed machine" is stating the premise that ruling removed. The controller VM is NOT one of the nine bundle machines (bundle.yaml machines block: "The Juju CONTROLLER is a dedicated 10th VM (D-104 amendment), bootstrapped by its own tag -- NOT a bundle machine").

Step 2.0 -- Per-DC MAAS service credential [CHECK FIRST; both may already exist]

Juju authenticates to the MAAS region with an API key. Per the per-DC credential-isolation discipline (SEC-012/-016; Roosevelt analog: per-DC cloud credentials), each DC gets a DEDICATED MAAS admin service user + its OWN key -- never reuse one key across DCs.

CHECK BEFORE YOU MINT. Both per-DC credentials already existed as of 2026-07-27 (vr1-dc0-cred and vr1-dc1-cred, SEC-018 / SEC-019; juju show-cloud vr1-maas --client reported credential-count: 2). Re-running the mint blocks below as written creates a DUPLICATE MAAS admin user and a SECOND key for a DC that already has one -- credential sprawl against SEC-018/-019 and D-137, on a register that is already red. Minting a credential is also a SEC-row obligation, not a free action.

CHECK (read-only) -- voffice1

juju credentials --client                       # is "$DC-cred" already listed?
maas "$MAAS_PROFILE" users read | grep -o "juju-$DC" || echo "no juju-$DC user"
ls -d ~/"$DC"-creds 2>/dev/null || echo "no ~/$DC-creds folder"   # SEC-009 site folder

(maas <profile> users read is the enumeration used to account for every MAAS account on this deployment -- docs/CURRENT-STATE.md:2784. Do NOT open anything inside the creds folder; its existence is all this gate needs.) GATE -- three outcomes, and only one of them proceeds:

This DC's credential is LISTED and its MAAS user exists -> SKIP the rest of Step 2.0 entirely and go to the bootstrap CHECK below. Do not re-mint.
Neither exists -> run the mint blocks that follow, and open the per-DC SEC row.
One exists without the other (a user with no registered credential, or a credential naming a user MAAS does not have) -> STOP. That is a half-made credential, not a missing one; reconcile it against docs/security-ledger.md before touching either side. Do not "make it consistent" by minting the missing half.

The mint blocks below run ONLY on the second outcome. On the MAAS region host (VR1: voffice1):

# service account -- Juju auths via the API key, not this password; use a random one
sudo maas createadmin --username="juju-$DC" --email="juju-$DC@<realm>"

Generate the key so it lands ONLY in the JUMPHOST per-site creds folder (~/$DC-creds/maas-api-key.txt on vcloud -- the SEC-009 ~/<site>-creds/ convention, and the site key IS the D-119 slug; NOT a new location on the region), never echoed to a terminal (a printed MAAS key is burned, SEC-006). Run the generate FROM the jumphost so the key transits ssh stdout straight into the vcloud file (no persistent copy on the region):

ssh <region> "sudo maas apikey --username=juju-$DC --generate" \
  > ~/"$DC"-creds/maas-api-key.txt
chmod 600 ~/"$DC"-creds/maas-api-key.txt
awk -F: '{print "parts:", NF}' ~/"$DC"-creds/maas-api-key.txt   # expect 3; do NOT cat

Register the MAAS cloud + this credential in Juju consuming the key BY PATH/stdin (never argv). Each key is a rotation obligation -- open a per-DC SEC row when issued.

CHECK (read-only) -- voffice1 -- the cloud is registered AND the controller tag exists

juju clouds --client | grep -i maas
# The bootstrap constraint below is only as good as the tag behind it. Fork 3's
# per-role tagger owns the D-104 controller tag (scripts/maas-role-tags.sh:125,
# "juju-controller-$SITE") -- confirm THIS DC's tag exists and is on exactly one
# machine before constraining on it:
bash scripts/maas-role-tags.sh check "$DC"
maas "$MAAS_PROFILE" machines read \
  | jq -r --arg t "juju-controller-$DC" '.[] | select(.tag_names[]? == $t) | "\(.hostname) \(.status_name)"'

GATE: the tag juju-controller-$DC exists AND the query returns EXACTLY ONE machine, in Ready. Zero machines -> the tag was never applied; juju bootstrap would find no candidate and fail at allocation. More than one -> STOP; two machines carrying a single-controller tag means the bootstrap target is ambiguous and you cannot predict which it takes. A status other than Ready (including a cached error power state) is NOT a pass -- refresh it with maas "$MAAS_PROFILE" machine query-power-state <system_id> and re-read, rather than assuming the node is broken (a stale cached error on nodes that are simply off is a measured trap, 2026-07-27).

MUTATION -- voffice1

juju bootstrap <this-dc-maas-cloud-name> <controller-name-per-DC> \
  --constraints "tags=juju-controller-$DC"

The --constraints is not optional. Without it Juju asks MAAS for any available machine and can land the controller on one of the nine role nodes -- the exact outcome the D-104 amendment rejected, and one you only discover after the bundle deploy finds a node missing. The tag name is DERIVED from $DC, so it stays correct for both DCs by construction; it is not a literal to retype.

The MAAS cloud name and the controller name are session-specific identifiers you choose/confirm at execution time (reflecting the site slug in the controller name for operator clarity) -- not invented here.

Per D-104's rationale: this keeps DR-independence honest (a controller loss affects only its own DC) and is a deliberate VR1 test-only economy (HA deferred to Roosevelt) -- do not "upgrade" this to HA mid-stream without a new D-NNN, that would silently reverse an ADOPTED decision.

Step 3 -- `preflight.sh` PASS for THIS DC (read-only gate)

DC="$DC" bash scripts/preflight.sh

DC= is load-bearing -- do not run this bare. P2 defaults to vr1-dc0 (scripts/preflight.sh:99) and assembles the overlay set from $DC (:100-105), so a bare run on the other DC validates dc0's numbers against dc0's bands and never parses this DC's artifact at all -- a green gate that examined the wrong input.

Expect the banner to name THIS DC before anything else: validating the MERGED <your DC> deploy input: bundle.yaml --overlay ... --dc <your DC>. If that line names a different DC than the one you exported, STOP -- $DC did not reach the script.

GATE: PREFLIGHT: PASS. Two failures are known and expected on the vcloud jumphost rather than on voffice1 (the deliberately-absent gitignored octavia PKI overlay, and MAAS unreachable) -- which is another reason this runs on voffice1. A red preflight is a stop, not a note: read every FAIL and decide per row, and record the decision. Do not run the sub-gates piecemeal to get a greener-looking answer.

Step 3.5 -- Model, spaces gate, and this DC's artifact source [MUTATION, gated]

Three things that must exist BEFORE the deploy and that no earlier step creates.

MUTATION -- voffice1 -- create THIS DC's model (guarded)

( {
  # Same capture-then-test idiom as scripts/juju-spaces-check.sh:40-42 (DOCFIX-115):
  # capture first, then test. The live-pipe form races SIGPIPE under pipefail and
  # reports a juju AUTH failure as "model not present" -- which here would mean
  # add-model runs against a controller you are not actually talking to.
  MJSON="$(juju models --format json)" || { echo "ABORT: 'juju models' failed (auth/macaroon? re-login, or append </dev/tty)"; exit 1; }
  MNAMES="$(jq -r '.models[]?.name' <<<"$MJSON" 2>/dev/null | sed 's#.*/##')" || true
  if grep -qx "$DC_MODEL" <<<"$MNAMES"; then
    echo "ABORT: model '$DC_MODEL' already exists -- do not deploy into a used model"
  else
    juju add-model "$DC_MODEL"
  fi
} )

One model per DC, named for the site slug. Deploying two DCs into a model named openstack on one client is how the second deploy lands on top of the first.

CHECK (read-only) -- voffice1 -- per-model spaces

bash scripts/juju-spaces-check.sh "$DC_MODEL"

Spaces are PER-MODEL in Juju, so this can only run after add-model and must run before juju deploy -- a missing or stale space name mis-binds charms at deploy time and is expensive to unpick afterwards. GATE: exit 0. Exit 1 = missing or stale spaces, STOP. Exit 2 = the model is not present yet, which means the block above did not do what you think it did.

MUTATION -- voffice1 -- point this DC's model at ITS OWN artifact source

ORDERING TRAP -- this is why the step is here and not later. If the model has no artifact source set, every unit apts from the INTERNET during deploy. The D-135/D-107 egress narrowing then lands on a half-built model, and unlike bootstrap the recovery is not "re-open egress and retry" -- it is a partially-installed cloud. Set it BEFORE the deploy, not during.

The two DCs DIFFER BY RULING (D-135 + its 2026-07-24/27 amendments) -- there is no common command here. vr1-dc0 runs a FULL MIRROR; vr1-dc1 runs an APT CACHING PROXY. This is the experiment, not an interim state. Read the address from that DC's OWN checker rather than typing it:

# vr1-dc1 -- apt caching proxy (apt-cacher-ng). Confirm it is up and note its LISTEN:
bash scripts/dc-cache-proxy.sh check vr1-dc1
juju model-config -m "$DC_MODEL" apt-http-proxy="http://<LISTEN from the check>:3142"
juju model-config -m "$DC_MODEL" apt-http-proxy          # READ BACK -- assert on content

Reference values, for recognising a wrong answer rather than for typing: scripts/dc-cache-proxy.sh:81,87 give vr1-dc1 LISTEN 10.12.68.4 port 3142; scripts/dc-mirror.sh:79 gives vr1-dc0 LISTEN 10.12.8.4. If the checker reports anything else, the checker wins and you stop to reconcile.

vr1-dc0 (mirror) -- VERIFY THE KEY NAME BEFORE YOU RUN IT. The mirror is consumed via a DIFFERENT model-config key than the proxy. The only in-repo statement of that key is a comment (scripts/dc-mirror.sh:234, apt-mirror=http://<LISTEN>/ubuntu), and no juju client exists on the vcloud jumphost to check it against, so it is NOT written here as an executable line. At execution, confirm the exact key with juju model-config --help (or juju model-defaults) on voffice1 FIRST, then set it and read it back the same way as above. Do not guess a key name -- a mistyped model-config key is accepted silently and leaves the model pointing nowhere.

GATE: the read-back prints the address you intended, and it is THIS DC's own artifact service. An empty read-back is a FAIL, not a pass -- checking vr1-dc1 as a mirror (or vr1-dc0 as a proxy) fails BY DESIGN and is not a reason to skip the step.

Step 4 -- Deploy `bundle.yaml` + THIS DC's overlays [MUTATION, gated]

This step carries its OWN DC-parameterised deploy command. It no longer says "follow phase-01 verbatim": phase-01's deploy blocks were written around VR0's single-DC testcloud, and following them unchanged against a VR1 DC is a known defect class, not a shortcut. Use runbooks/phase-01-bundle-deploy.md for the things that genuinely transfer -- Step 1.0's Octavia PKI overlay sanity, the convergence-watch pattern, and the PRE-vault-init end-state description -- with $DC and $DC_MODEL exported per Step 1 above. Dependency: phase-01's own per-DC parameterisation (its $DC/$DC_MODEL handling and its per-DC VIP guard) is delivered separately in this same batch; if you reach a phase-01 block that still hardcodes 10.12.4., VR0 system_ids, openstack0..3 disk paths, a 4-machine plan, or juju add-model openstack, that block has NOT been parameterised yet -- STOP and reconcile rather than adapting it live.

4.1 -- Assemble THIS DC's overlay set

Deploy input = bundle.yaml + this DC's overlays. The base bundle is VIP-FREE and is NOT a deployable input on its own.

Overlay	`vr1-dc0`	`vr1-dc1`
`overlays/${DC}-vips.yaml`	REQUIRED (`vr1-dc0-vips.yaml`)	REQUIRED (`vr1-dc1-vips.yaml`)
machines retag	none -- and that is correct: the base bundle's machines block already carries `tags=openstack-vr1-dc0,<role>`	REQUIRED: `overlays/vr1-dc1-machines.yaml`
`overlays/${DC}-octavia-pki.yaml`	REQUIRED (gitignored secret)	REQUIRED (gitignored secret)

Do not interpolate a machines overlay path for vr1-dc0 -- there is no overlays/vr1-dc0-machines.yaml, and juju errors out on a missing --overlay path. The asymmetry is deliberate; the gate that catches BOTH arms is the merged tag assertion in 4.2, not the presence of a file. Never "guard" the argument with a silent skip: an overlay that quietly disappears is exactly how a vr1-dc1 deploy would merge vr1-dc0's tags.

Make NO hand edit to bundle.yaml or to either generated VIP overlay (ruling 3, 2026-07-25; both *-vips.yaml are RENDERED -- edit render/values/${DC}-vips.yaml and re-render, and tests/render-drift fails the gauntlet if you do not).

overlays/dc-ha-scaleup.yaml is DELIBERATELY NOT in this command. Per R6 the VIPs land BEFORE the HA overlay; applying it is a separate, later step once the base deploy has settled. That is also why the base deploy legitimately shows cluster_count: 1 -- see Step 12's gate, which states when 1 is correct and when it is the failure.

Octavia PKI is PER-DC. Under the R7 amendment (D-109, RULED 2026-07-27) the generator is no longer dc0-frozen and each DC gets its own CA material, so this is a per-DC artifact (overlays/${DC}-octavia-pki.yaml), not one shared file. Reuse this DC's existing overlay if its CAs are still valid (10y, per phase-01 Step 1.0), OR regenerate via phase-01 Step 1.0-GEN with DC exported. Either way it is an explicit operator call, logged as a decision note -- not a silent default to reuse. Cross-DC reuse was refused on posture: the overlay carries CA private keys plus a plaintext passphrase in a repo SEC-004 records as PUBLIC.

4.2 -- DRY-RUN FIRST, and read the machines block [read-only gate]

Run this BEFORE 4.3. It is the only thing standing between a typo in the overlay set and nine wrongly-allocated MAAS machines. There is no offline juju merger that can prove the machines-block override merges (see overlays/vr1-dc1-machines.yaml's own VERIFY-LIVE note), so the dry-run IS the proof.

CHECK (read-only) -- voffice1 -- run the block for YOUR DC. These are EXACTLY the 4.3 commands plus --dry-run; if the two ever differ, this gate is validating something you are not deploying.

vr1-dc0 (no machines overlay -- the base bundle already carries its tags):

juju deploy ./bundle.yaml \
  --overlay overlays/vr1-dc0-vips.yaml \
  --overlay overlays/vr1-dc0-octavia-pki.yaml \
  -m "${DC_MODEL}" --dry-run

vr1-dc1 (machines overlay REQUIRED, order: vips -> machines -> pki):

juju deploy ./bundle.yaml \
  --overlay overlays/vr1-dc1-vips.yaml \
  --overlay overlays/vr1-dc1-machines.yaml \
  --overlay overlays/vr1-dc1-octavia-pki.yaml \
  -m "${DC_MODEL}" --dry-run

GATE -- assert on the CONTENT of the plan, not on the command exiting 0:

Every machine reads tags=openstack-$DC,<role>. If any machine still names the OTHER DC, STOP -- one Office1 MAAS region sees BOTH DCs' nodes, so a wrong tag here means this model can allocate the other DC's hardware. The fallback is inlining this DC's tags into a rendered per-DC bundle, not deploying anyway.
Nine machines, "0" through "8", matching bundle.yaml's machines block (D-121 Option C: 3 control / 2 compute / 4 storage). A four-machine plan means you are looking at VR0's topology.
Every VIP in the plan is on THIS DC's prefixes -- 10.12.4/8/12 for vr1-dc0, 10.12.64/68/72 for vr1-dc1 (scripts/lib-net.sh:54-56, :146-154). A VIP from the other DC means the wrong overlay merged.
If the plan reports an ERROR on any --overlay path, do NOT drop the overlay to make it run. A missing overlay is a missing input, not an optional one.

Capture this output. It is the plan-count evidence the exit gate cites -- plan counts are quoted only from captured command output, never from this file's prose (GA-R1 rule 2).

4.3 -- The real deploy [MUTATION, gated]

The SAME block you dry-ran in 4.2, minus --dry-run. Nothing else changes.

vr1-dc0:

juju deploy ./bundle.yaml \
  --overlay overlays/vr1-dc0-vips.yaml \
  --overlay overlays/vr1-dc0-octavia-pki.yaml \
  -m "${DC_MODEL}"

vr1-dc1:

juju deploy ./bundle.yaml \
  --overlay overlays/vr1-dc1-vips.yaml \
  --overlay overlays/vr1-dc1-machines.yaml \
  --overlay overlays/vr1-dc1-octavia-pki.yaml \
  -m "${DC_MODEL}"

Open the Step 4b watch windows before this returns.

bundle.yaml ships "NO designate" for THIS stage per phase-01's own gate text (D-019, as superseded by D-106): Designate reactivation is Stage 7's job, do not reactivate it mid-Stage-5.
Settle to phase-01's documented PRE-vault-init end state, read with $DC_MODEL substituted for its hardcoded model name: ZERO units in error; mysql-innodb-cluster ACTIVE; vault/0 BLOCKED "Vault needs to be initialized" (the Step 5 trigger, not a fault); the TLS consumers waiting on vault certs; octavia BLOCKED "Awaiting configure-resources".

Step 4b -- Mid-deploy watch + failure decision tree (read-only, runs alongside Step 4)

Open BOTH windows before Step 4.3 returns. A deploy you are not watching is one you diagnose from logs afterwards.

CHECK (read-only) -- voffice1, window 1 (detail)

juju status -m "$DC_MODEL" --watch 5s

CHECK (read-only) -- voffice1, window 2 (signal)

bash scripts/deploy-watch.sh "$DC_MODEL" 15

Compact machine/unit state COUNTS plus the NAMED principal units in error/blocked. Health-at-a-glance: the error/blocked section stays EMPTY until the expected late blocks (vault needs-init, octavia awaiting-configure). Neither window descends into subordinates and neither replaces a phase gate.

Expected end-state shape (not a count to quote -- confirm against Step 4.2's captured plan, GA-R1 rule 2): nine machines reaching started; zero units in error; the only blocked units are vault/0 (needs init) and octavia (awaiting configure-resources). Anything else named in the error/blocked list is this tree's input.

Decision tree -- something went wrong mid-deploy

Work top to bottom. Default is FIX-FORWARD; juju converges, and tearing down a partially-good model costs more than repairing it.

Units are still waiting/maintenance and counts are MOVING. Not a failure. Snap installs and cert exchange are slow on first boot. Keep watching; do not intervene because it is quiet.
A machine never leaves pending / allocation fails. This is a MAAS-side answer, not a juju one. Check this DC's node pool: the right tag on the right count of Ready machines (Step 2's tag check, widened to the role tags). A cached power_state=error on nodes that are merely off reads as a hard blocker and is not one -- refresh with maas "$MAAS_PROFILE" machine query-power-state <system_id> and re-read before diagnosing a fault. Fix-forward: correct the tags/power state and let juju retry. Do NOT destroy the model.
Units are in error with a hook failure. Read the actual hook error (juju debug-log -m "$DC_MODEL" --replay --include <unit>). If it is a transient fetch failure, first check whether this model's artifact source (Step 3.5) is set and reachable -- a model apting from the internet behind a narrowed edge fails exactly this way. Fix-forward: repair the cause, then juju resolved --no-retry <unit> or juju resolved <unit> as appropriate.
The wrong DC's hardware got allocated, or a VIP from the other DC appears in the model. STOP. This is the one case where fix-forward is wrong: the model has bound to another DC's addresses/machines, and repairing it in place leaves you unsure what is bound to what. Tear the MODEL down (juju destroy-model "$DC_MODEL", operator-gated) and return to Step 4.2 -- the dry-run gate exists precisely to make this case unreachable.
You are considering destroying anything BELOW the juju layer (the containment VM, the DC's libvirt resources, MAAS records). Read runbooks/dc-dc-teardown-rollback.md's rollback decision tree FIRST, and read its Model-B banner before acting on any of its resource-level steps. Standing caution as of 2026-07-29: that runbook's Step 2 MAAS-side check and Step 3 scoped-destroy targets have known defects against the current two-root shape -- treat both as reference, verify every module name against the live opentofu/ tree before running anything, and never let a whole-DC destroy be the first thing you reach for while 18 nodes are enrolled in MAAS.

At every branch: record what you observed and what you chose. A mid-deploy decision that is not written down is one the next session has to re-derive.

Step 5 -- Run phase-02 (vault bring-up) -- INDEPENDENT root per DC [adapted, SECRET-HANDLING]

Follow runbooks/phase-02-vault-bringup.md per DC, with $DC/$DC_MODEL exported per Step 1 and substituted for any hardcoded model name in its command blocks. Per D-109: each DC's Vault is its OWN independent root CA -- no regional root-of-trust this phase. This means:

The vault operator init one-shot step (Step 2.1) runs ONCE PER DC, producing DISTINCT unseal-key shares and a DISTINCT root token for vr1-dc0 vs. vr1-dc1. Store them separately, clearly labeled by DC (this repo's existing secrets discipline -- ~/vault-init/-style off-host capture -- applies identically per DC, just not shared between them).
D-109 also requires each DC's Vault to issue certs with BOTH IPv4 and IPv6 (ULA) SANs for the dual-stacked metal-internal API endpoints. This is a real charm-behavior question not yet confirmed this session: whether the vault/vault-pki-relation charms used in bundle.yaml actually support requesting/issuing IPv6 SANs today (vs. only IPv4) is UNVERIFIED -- check the actual charm config/relation-data schema (e.g. openstack-loadbalancer/certificates interface docs, or the vault charm's own config options) before assuming this "just works" once IPv6 addresses exist on the plane. If it does not support v6 SANs natively, that is a real finding to log (a D-NNN amendment or a new decision), not something to route around silently.

Step 6 -- Write + apply the IPv6 family-matrix overlay [OVERLAY DRAFTED 2026-07-10, NOT YET APPLIED]

D-101's family matrix (Section 2 of the buildout design) requires, per DC:

data-tenant, storage, replication: IPv6-ONLY ULA (no v4 config at all on these planes for the relevant charms -- e.g. ceph-mon's public-network/cluster-network, OVN's geneve encapsulation address).
metal-internal, metal-admin: DUAL-STACK (v4 retained + a ULA leg added -- the metal-admin ULA leg is the 2026-07-09 operator AMENDMENT to D-101, a reversal of that decision's own original lean, so do not assume older documentation/training-data patterns for "metal-admin stays v4-only" -- this repo's ADOPTED text says otherwise).
provider-public: DUAL-STACK v4 + GUA (native, no NAT66 -- D-101's tenant/MTU sub-policy folded in from D-102).

overlays/dc-dc-ipv6-family-matrix.yaml now exists, drafted 2026-07-10 against REAL charm source/config (not guessed option names) -- see docs/archive/dc-dc-ipv6-charm-research.md for full sourcing and per-charm confidence level. Summary: prefer-ipv6 (confirmed real option, additive dual-stack HAProxy binding, confirmed via charm-nova-cloud-controller's actual "Dual Stack VIPs" commit) + appended v6/GUA entries on the existing vip: option, for the nine dual-stack API charms directly confirmed or sharing the same charms.openstack framework; ceph-mon's own separate prefer-ipv6 + ceph-public-network/ceph-cluster-network (a clean switch-to-v6-only, correct for the ULA-only planes); NO entry for OVN (confirmed no such charm-config option exists -- geneve family follows the bound interface automatically once the plane itself is ULA-only).

This is an INCREMENTAL ADD against the LIVE $DC_MODEL Step 4 already deployed -- not a fresh deploy. It re-passes the SAME overlay set Step 4 used and adds the family matrix on top, so juju's diff introduces only the address-family options; the base topology, the VIPs and the machines block must come out UNCHANGED. That is precisely what the GATE below checks. Re-passing the full set is deliberate: dropping the VIP or machines overlay here would present juju with an input that contradicts the running model.

Corrected 2026-07-29: this command previously named overlays/octavia-pki.yaml and overlays/${DC}-hostnames.yaml and passed NO VIP overlay at all -- so since ruling 3 its merged input carried ZERO VIPs, and ${DC}-hostnames.yaml has NEVER BEEN AUTHORED (it is unbuilt D-106 work, proposed in runbooks/dc-dc-phase6-designate-cos-magnum.md), on which juju errors out.

THIS STEP HAS NOT BEEN APPLIED TO ANY LIVE MODEL (overlay drafted 2026-07-10, still unvalidated -- see open item 3 below). Every block here is --dry-run; the real apply is a separate, gated decision once the diff has been reviewed. Step 12's v6 gates depend on this step having actually been applied -- if you have only dry-run it, 12.1 and 12.2 are BLOCKED, and "blocked on Step 6" is the honest recorded answer, not a soft pass.

vr1-dc0:

juju deploy ./bundle.yaml \
  --overlay overlays/vr1-dc0-vips.yaml \
  --overlay overlays/vr1-dc0-octavia-pki.yaml \
  --overlay overlays/dc-dc-ipv6-family-matrix.yaml \
  -m "${DC_MODEL}" --dry-run

vr1-dc1:

juju deploy ./bundle.yaml \
  --overlay overlays/vr1-dc1-vips.yaml \
  --overlay overlays/vr1-dc1-machines.yaml \
  --overlay overlays/vr1-dc1-octavia-pki.yaml \
  --overlay overlays/dc-dc-ipv6-family-matrix.yaml \
  -m "${DC_MODEL}" --dry-run

Same overlay-set rules as Step 4.1: vr1-dc0 takes no machines overlay because the base bundle already carries its tags, and no overlay is ever dropped to make the command run.

Review the diff before applying for real, same discipline as every other overlay in this repo. GATE: the diff changes ONLY the address-family options this overlay sets; every VIP still reads on THIS DC's prefixes, and the machines block still reads tags=openstack-$DC,<role>. If either moved, the overlay set is wrong -- stop, do not apply.

Still genuinely open, not resolved by the overlay:

The 8 charms whose prefer-ipv6 presence is INFERRED-BY-PATTERN (not individually fetched from their own config.yaml) need live juju config <app> confirmation once first deployed.
RESOLVED 2026-07-27 by R8 -- this item is CLOSED and the text below is superseded. Octavia's lb-mgmt-net is IPv6 ULA, charm-created and charm-owned (operator: "I want to take a Octavia creates and owns its own IPv6 network"). The two Launchpad numbers this item rested on were READ IN FULL and BOTH FAILED: #1913409 is kolla-ansible, a different installer, Fix Released 2021; #1911788 is Incomplete, a duplicate, and is not an IPv6 defect -- its cause is an OVN shortname/FQDN port-binding mismatch fixed in OVS 2.15, and these nodes install 2.17. Upstream research also found the charm creates a v6 subnet and has no IPv4 code path at all, so v4-only was never the safe option -- it was the departure. Do NOT re-raise this as an open decision. What IS owed here is the MTU obligation (LP #2018998), now asserted by scripts/phase-05-octavia-verify.sh, and gate G18 (apex back-fill of the charm-created prefix), which blocks project close.
UNVALIDATED against any live model -- this overlay has not been applied or tested, same posture as every other artifact from tonight's prep-only session.

Steps 7-9 -- phase-03 / phase-04 / phase-05, adapted per DC

Follow runbooks/phase-03-core-verify.md, runbooks/phase-04-network- carve.md, runbooks/phase-05-octavia-enablement.md per DC, with $DC and $DC_MODEL exported per Step 1, the selector discipline applied to every command block in them that reads lib-net.sh/lib-hosts.sh values, and $DC_MODEL substituted for any hardcoded model name.

Known, measured caveat -- these three phases call scripts that do NOT select a DC. Measured 2026-07-29: of the scripts that source scripts/lib-net.sh, these Stage-5 ones read a DC-dependent value while never calling lib_net_select_dc, so they resolve vr1-dc0's literals whichever DC you are on: scripts/phase-03-admin-openrc.sh (KEYSTONE_VIP_DEFAULT), scripts/phase-04-network-create.sh and scripts/phase-04-network-verify.sh (both also carry a hardcoded PROVIDER_CIDR="10.12.4.0/22", which a selector would not reach), and scripts/vault-kv-health.sh (METAL_INTERNAL_CIDR). On vr1-dc0 that is harmless and correct. On vr1-dc1 it is wrong and SILENT -- no error, just dc0's addresses. Until those scripts are fixed, either export the value explicitly (each honours an env override) or verify every address in their output against this DC's planes before believing a PASS. LOG anything further you find (a DOCFIX candidate); do not patch around it live.

Step 10 -- `cloud-assert.sh --capture` (read-only + committed BOM)

bash scripts/cloud-assert.sh --capture

Per DC, at real deploy-completion. Confirmed repo-agnostic (2026-07-09 sweep). Commit the resulting asbuilt/ BOM per this repo's existing deploy-loop convention, labeled clearly by DC (do not let vr1-dc0's and vr1-dc1's BOMs collide/overwrite -- confirm the script's own output path is already DC-distinguishing, or add a DC-qualified path if not, BEFORE running for the second DC; a silently overwritten BOM loses the first DC's evidence).

Step 11 -- Controller backup [MUTATION, gated]

THE FLAG SHAPE BELOW IS UNVERIFIED AT AUTHORING -- CONFIRM IT FIRST. No juju client exists on the vcloud jumphost (measured 2026-07-29: command -v juju -> ABSENT), so this shape could not be checked against a real client when it was written, and a repo-wide grep corroborates the subcommands only from other prose. What IS established: -m takes a MODEL, not a controller name -- so the previous form here (-m <controller-name>) was wrong on both lines; and create-backup downloads by default, which made the old second line redundant rather than necessary. At execution, run juju create-backup --help on voffice1 and use what it says. Then gate on the resulting FILE, which is true regardless of flag spelling.

# Confirm the flags FIRST (see the caveat above):
juju create-backup --help

# Then take the backup against the CONTROLLER MODEL of this DC's controller:
juju switch <this-dc-controller-name>
juju create-backup -m controller

GATE -- assert on the artifact, not on the command exiting 0: a backup file exists in the working directory, is NON-ZERO in size, and its timestamp is from this run. Record its name and size. If create-backup printed a path but no file is there, that is a FAIL -- do not record "backup taken".

Then store it OFF-HOST, labeled by DC, with the same secrets-adjacent handling discipline as Vault's init capture, and confirm the off-host copy's size matches before deleting anything locally. Per D-104: scheduled controller backups, one per DC. The RESTORE half of this drill is exercised in Stage 6 (DR wiring and failover drill) -- this step only takes and stores the backup, it does not restore it.

Step 12 -- VERIFY-LIVE gates + Ceph-over-v6 / geneve-over-v6 (READ-ONLY gate)

juju run on juju 3.x is the ACTION runner, so the old juju run <unit> 'ceph -s' form asked ceph-mon to run an action NAMED ceph -s and could never execute. juju exec is the command runner; the </dev/null is this repo's convention for it (scripts/phase-05-octavia-verify.sh:96, runbooks/phase-05-octavia-enablement.md:93).

12.1 -- Ceph over v6

# Ceph binds v6 on the storage/replication ULA planes (ms_bind_ipv6, D-101):
juju exec -m "$DC_MODEL" --unit ceph-mon/leader -- 'ceph -s' </dev/null

GATE: the mon addresses reported are on THIS DC's storage/replication ULA planes, in v2:/v1: form. A v4 mon address on those planes is a FAIL, not a "close enough". If the command errors, that is a FAIL too -- an unreadable cluster is not a passing one.

12.2 -- Geneve encapsulation family (assert the OBSERVABLE, not a config key)

The old form here grepped juju config ovn-central for encap. Step 6 of this same runbook states there is NO such charm-config option -- geneve family follows the bound interface automatically once the plane is ULA-only -- so that grep could only ever return empty and read as "nothing wrong". Assert what is actually observable on a chassis instead:

Name the unit first, then probe it. ovn-chassis is a SUBORDINATE (to nova-compute -- bundle.yaml:450,453), and whether <app>/leader resolves for a subordinate is NOT verified from this repo. The precedent that IS established here uses a NAMED unit, not /leader (runbooks/phase-05-octavia-enablement.md:93 -> --unit octavia/0). So:

# 1. list this DC's chassis units and pick one that is on a COMPUTE host:
juju status -m "$DC_MODEL" ovn-chassis
# 2. probe THAT unit by name (substitute for ovn-chassis/0):
juju exec -m "$DC_MODEL" --unit ovn-chassis/0 -- \
  'sudo ovs-vsctl get Open_vSwitch . external_ids' </dev/null

Repeat for each chassis unit -- one chassis tunnelling on the wrong family is still a defect, and checking one unit does not clear the others.

GATE -- three outcomes, and only one passes:

An encapsulation address is reported and it is an IPv6 address inside THIS DC's data-tenant ULA prefix (scripts/lib-net.sh PLANE_NAME -> data-tenant, after lib_net_select_dc "$DC") -> PASS.
An encapsulation address is reported and it is IPv4, or outside that prefix -> FAIL. The chassis is tunnelling on the wrong plane or the wrong family.
No encapsulation entry appears at all, or the command errors -> REFUSE, do not pass. "Could not look" is never "nothing there". Record the raw output, confirm the unit name from step 1 is one juju actually lists, and check Step 6 was APPLIED rather than only dry-run. If it still shows nothing, this gate is BLOCKED and is recorded as BLOCKED -- never as a pass.

12.3 -- The four VERIFY-LIVE gates

Each asserts a RULED value that no offline gate can reach. Sources: docs/CURRENT-STATE.md:247-252 (the VERIFY-LIVE list) and docs/design-decisions.md:824 (the keystone PO: mechanism).

(a) hacluster cluster_count -- and WHEN each value is correct.

juju config -m "$DC_MODEL" <app>-hacluster cluster_count

GATE, and read the precondition before judging it: after the Step 4 base deploy, cluster_count: 1 is EXPECTED AND CORRECT -- the base bundle pins it deliberately and overlays/dc-ha-scaleup.yaml is excluded from Step 4 per R6 (overlays/dc-ha-scaleup.yaml:51-58). The 3 assertion applies ONLY after the HA scale-up overlay has been applied; at THAT point a cluster_count still reading 1 behind a 3-unit principal is the failure -- pacemaker will bootstrap a VIP off a single node and every offline checker stays green. State which side of the overlay you are on when you record this result.

(b) Ceph pool redundancy.

juju exec -m "$DC_MODEL" --unit ceph-mon/leader -- 'ceph osd pool ls detail' </dev/null
juju exec -m "$DC_MODEL" --unit ceph-mon/leader -- 'ceph osd crush rule dump' </dev/null

GATE: size=3 min_size=2 on the OpenStack pools, and failure-domain host (the RULED target -- docs/CURRENT-STATE.md:248, and size=3/min_size=2 is ADOPTED per runbooks/dc-dc-phase0-vcloud-prep.md:223). A size=2 fallback is permitted ONLY as an explicit, logged decision taken when real capacity is measured -- never as something you find already set and wave through. Any other failure-domain REFUSES: record it and stop rather than assuming a synonym.

(c) keystone policy override -- RULED 2026-07-25.

juju status -m "$DC_MODEL" keystone

GATE: PO: appears on EVERY keystone unit. PO (broken): on ANY unit is a FAIL, and it is not cosmetic: a broken override is ATOMIC -- the whole policy is discarded, silently reverting every tenant domain-manager to a plain user (protects D-051/D-064). One good unit does not cover for a broken one; check the whole column, and re-check after any scale-up.

(d) machines-block overlay merge. Already gated at Step 4.2 -- the dry-run tag assertion. Do not author a second copy here; cite Step 4.2's captured output in the exit-gate record.

The v6 expectations in 12.1/12.2 depend on Step 6's overlay actually having been applied. If Step 6 is still open, this gate CANNOT pass yet -- record "blocked on Step 6" as the honest answer rather than forcing a pass.

GATE (Stage 5 exit condition, docs/dc-dc-deployment-workflow.md): DC=<this DC> preflight.sh PASS before add-model (Step 3); the Step 4.2 dry-run captured and its machines-block tags confirmed; post-deploy cloud-assert.sh --capture per DC (Step 10); controller backup taken + stored, gated on the FILE (Step 11); the four VERIFY-LIVE gates plus Ceph-over-v6 and geneve-over-v6 recorded (Step 12, the v6 halves contingent on Step 6).

-> Repeat this entire runbook for the other DC. There is no ruled DC ordering for this deployment (GA-R5, 2026-07-27): both DCs deploy from the Office1 headend by the same procedure, in whichever order the operator chooses. -> proceed to Stage 6 (DR wiring and failover drill) once BOTH DCs pass this gate.

Delivery checklist

bash scripts/repo-lint.sh clean (0 fail) before committing any repo

changes made while executing this runbook (the IPv6 overlay file, any
re-rendered per-DC VIP overlay -- NOT a `bundle.yaml` edit, which carries
no VIPs since ruling 3 -- any DC-qualified `asbuilt/` path fix).

The Step 4.2 dry-run output CAPTURED to a dated file and cited in the

commit that closes this stage for that DC (plan counts are quoted only
from captured output, GA-R1 rule 2).

Changelog entry per real repo change this runbook produces (the

overlay, any bundle edit, any script fix) -- next DOCFIX number via
`bash scripts/ledger-scan.sh`, same as every other delivery this
session.

docs/session-ledger.md updated with the outcome, per DC.

docs/dc-dc-deployment-workflow.md Stage 5 row updated once BOTH DCs

have actually passed Step 12's gate (not just the first one done).

Log the Step 6 charm-config research (v6 SAN support, v6 addressing

option names) as its own follow-up item if not resolved inline during
this stage's real execution -- do not leave it silently unresolved in
the workflow doc's tooling gap register.