# DC-DC Phase 4 -- Juju controller + OpenStack bundle, per DC (Stage 5)

> **NAMESPACE (D-119, ADOPTED 2026-07-14) -- this runbook speaks ONLY
> region-qualified site slugs: `vr1-dc0` and `vr1-dc1`.** The bare `DC1`/`DC2`
> and `dc1`/`dc2` tokens this file used to carry are RETIRED. They were
> ambiguous in the worst possible way: read positionally, "DC1" meant VR1's
> FIRST DC (`vr1-dc0`) in this file's selector callout and VR1's SECOND DC
> (`vr1-dc1`) in its deploy step -- the same token naming two different
> datacenters in one document. `scripts/lib-net.sh:179-182` REJECTS a bare
> `dc0`/`dc1`/`dc2` outright, so any command that still carried one could not
> run. Everywhere below, `$DC` is exactly one of `vr1-dc0` | `vr1-dc1`.
>
> **There is NO ruled DC ordering for this deployment (GA-R5, 2026-07-27).**
> Run this runbook once per DC, in whichever order the operator chooses; nothing
> here depends on one DC preceding the other.
>
> **CORRECTED 2026-08-02 (fold register F1).** This paragraph used to continue
> "Both DCs deploy from the Office1 headend by the SAME procedure." **The
> same-procedure half stands; the Office1-headend half is WRONG and was reversed
> by D-138 (2026-07-30).** The 2026-07-27 ruling was about DC *ordering*; the
> run-location claim was attached to it and is not part of it. See the RUN
> LOCATION block below for the split that actually applies.

Stand up a running, independent OpenStack cloud in one DC, then repeat the same
loop for the other. This is the ground-truth/copy-point stage: `docs/dc-dc-buildout-
design.md` line 12 states this repo was seeded from `openstack-caracal-ipv4`
at commit 8813efc specifically so `bundle.yaml` and `runbooks/phase-01..08-
*.md` could be reused here almost unchanged. **This runbook does not
duplicate those eight runbooks' command-level content.** It describes what
changes to run them TWICE, independently, in the VR1 two-DC dual-stack
context, and calls out the real adaptation work plainly rather than
asserting "just re-run it" glosses over anything.

**Governing docs:** `docs/dc-dc-buildout-design.md` Section 4 "Phase 4" entry
and Section 2 (address-family table); `docs/design-decisions.md` D-104
(controller topology), D-109 (per-DC independent Vault roots + IPv6-SAN
certs), D-101 (address plan / family matrix); `docs/dc-dc-deployment-
workflow.md` Stage 5 row (**"This is where the existing repo tooling
(preflight.sh, cloud-assert.sh, lib-net.sh/lib-hosts.sh, run-logged.sh) does
the most direct work"**) and tooling gap register item #1 (the `$DC`
selector convention, CLOSED 2026-07-09 DOCFIX-151 -- this is the stage that
actually calls it).

**Prerequisites (must be true entering this stage, per DC):** Stage 4
(`runbooks/dc-dc-phase3-maas-enlist-deploy.md`) done for that DC -- nodes
**READY** (NOT Deployed: MAAS-deploy is SKIPPED per the 2026-07-23 READY-handoff
ruling / DOCFIX-200 -- THIS runbook is what provisions them, and expecting
Deployed nodes here is exactly the breakage that ruling prevented); six planes
present with correct fabrics/VLANs; provider NIC raw; PXE (v4) working; that
DC's own artifact source reachable on its own address (`vr1-dc0` = the D-135
item-1 mirror, `vr1-dc1` = the D-135-AMENDED apt caching proxy consumed via
`juju apt-http-proxy` -- the two DCs differ BY RULING, so do not assume a
mirror at `vr1-dc1`); `scripts/lib-hosts.sh`'s per-DC host data populated (that
stage's own follow-up delivery) so `lib_hosts_select_dc "$DC"` no longer
fails loud for it.

**THIS STAGE OWES GATE G17 (inherited from Stage 4, split out 2026-07-27 by
operator ruling -- see the G17 row in `docs/CURRENT-STATE.md`, the authority).**
Stage 4 could not verify that a NODE reaches its DC's artifact source, or that a
node's time source is the MAAS hierarchy, because under the READY-handoff ruling
its nodes never boot an OS. **First boot in this stage is the first moment either
is observable, so capture both then** -- per DC, from a booted node:

- `vr1-dc0`: the mirror at `10.12.8.4` answers -- and assert on the BODY, not on
  `curl` exiting 0: `curl -sI` returns 0 on a 404 or a 500, and the dc0 URL is an
  autoindex root that answers 200 with nothing behind it. Fetch a real index file
  and check its content.
- `vr1-dc1`: apt fetches through the proxy at `10.12.68.4:3142` (NOT a mirror --
  D-135 amendment); confirm from the apt output that the fetch went THROUGH the
  proxy, not around it.
- both: `chronyc sources` shows the MAAS-served time source, NOT the DC edge
  (D-129(iv) removed the edge from that path).

Capture the output to a dated file and cite it in the commit that closes G17. If
first boot happens without these captures being taken, the observation window has
passed and G17 needs a deliberate rescue-boot to answer -- so take them the first
time.

!!! Every command block in the referenced `phase-01..08` runbooks that reads
    `scripts/lib-net.sh` or `scripts/lib-hosts.sh` values (VIP bands, host
    octets, `KEYSTONE_VIP_DEFAULT`, etc.) MUST be preceded, in THIS stage's
    session, by:
    ```bash
    source scripts/lib-net.sh;   lib_net_select_dc   "$DC"   # $DC = vr1-dc0 | vr1-dc1
    source scripts/lib-hosts.sh; lib_hosts_select_dc "$DC"
    ```
    Do not skip this because it "worked before" on the other DC. For
    `vr1-dc0` the net selector is a documented no-op (D-101: VR1's first DC
    inherits VR0 DC0's v4 layout unchanged -- `scripts/lib-net.sh:120-135`),
    so skipping it happens to produce the right values by accident. For
    `vr1-dc1` it is NOT a no-op -- that arm DIVERGES (`scripts/lib-net.sh:136-178`:
    the `10.12.64/68/72/76/80/84` planes, its own gateways, its own FIP pool)
    -- and skipping it silently reuses `vr1-dc0`'s addresses, exactly the bug
    DOCFIX-151 exists to prevent. Always call both, for both DCs,
    unconditionally.

    A bare `dc0`/`dc1`/`dc2` is REFUSED by the selector, by design
    (`scripts/lib-net.sh:179-182`). If you see that FAIL, you have typed the
    retired namespace -- re-type the region-qualified slug, do not "fix" the
    library.

---

## What is genuinely IDENTICAL across the two VR1 DCs, and what is not

**Ruling 3 (2026-07-25): `bundle.yaml` is VIP-FREE (topology only) and EVERY
VIP arrives via a per-DC overlay, `vr1-dc0` INCLUDED.** Neither DC needs a VIP
*band* change -- both keep the same `.50-.99` host octets -- but each
absolutely needs its OWN overlay, because the prefixes differ: `vr1-dc0` is
`10.12.4/8/12`, `vr1-dc1` is `10.12.64/68/72` (`scripts/lib-net.sh:54-56` and
`:146-154`). Deploying the bare bundle with no overlay gives a control plane of
decorative HA, and NO gate catches it at deploy time except
`provider-bundle-check`. Both overlays are GENERATED by
`scripts/render-dc-overlays.py`; edit `render/values/${DC}-vips.yaml` and
re-render rather than hand-editing an overlay.

Still correct, and the reason the two DCs read alike: D-101 rules that VR1's
FIRST DC (`vr1-dc0`) inherits VR0 DC0's six-plane v4 layout UNCHANGED, and the
host octets are identical across DCs. `phase-01..08`'s command blocks that
reference `lib-net.sh`/`lib-hosts.sh` values resolve to the same numbers for
`vr1-dc0` as they did for the original VR0 DC0 rehearsal -- expected, not a bug
(see the `$DC` selector callout above). `vr1-dc1` is where they diverge, which
is why the selector call is unconditional.

**A THIRD DC would need a distinct, non-overlapping v4 supernet + all-new VIPs
-- NOT YET ASSIGNED, and OUT OF SCOPE for this stage.** Per D-101 and tooling
gap register item #3, any further DC's supernet, the org ULA /48 and its GUA
carve are NetBox-authoritative; the import PIPELINE closed 2026-07-09
(DOCFIX-152) but no literals beyond the two ruled DCs have been assigned. Do
not invent a "test range" to get moving faster -- that is exactly the inference
hard rule 2 forbids. **This stage deploys `vr1-dc0` and `vr1-dc1` only; both
have real, ruled, apex-verified literals today.**

---

## Sequence (repeat entire sequence once per DC; no ruled DC ordering)

```
1.  Confirm Stage 4 gate + call the $DC selectors        (read-only)
2.  Bootstrap a DEDICATED single-unit Juju controller, IN-DC  [MUTATION, gated]
3.  preflight.sh PASS for THIS DC                         (read-only gate)
3.5 add-model, spaces gate, per-DC artifact source        [MUTATION, gated]
4.  Deploy bundle.yaml + THIS DC's overlays               [MUTATION, gated -- see below]
4b. Mid-deploy watch + failure decision tree              (read-only, runs alongside 4)
5.  Run phase-02 (vault bring-up) -- INDEPENDENT root per DC [adapted, SECRET-HANDLING]
6.  Write + apply the IPv6 family-matrix overlay          [DRAFTED 2026-07-10, NOT YET APPLIED -- see below]
7.  Run phase-03 (core verify)                            [adapted, gated]
8.  Run phase-04 (network carve)                          [adapted, gated]
9.  Run phase-05 (octavia enablement)                     [adapted, gated]
10. cloud-assert.sh --capture                             (read-only + committed BOM)
11. Controller backup                                     [MUTATION, gated]
12. VERIFY-LIVE gates + Ceph-over-v6 / geneve-over-v6     (read-only gate)
    -> EXIT GATE (this DC) -> repeat 1-12 for the other DC -> Stage 6
```

---

## Step 1 -- Confirm Stage 4 gate + call the $DC selectors (READ-ONLY)

> **RUN LOCATION -- THE TOOLS SPLIT. This is the single most load-bearing fact
> in this runbook, and it was WRONG here until 2026-08-02 (fold register F1).**
>
> | Tool | Runs on | Why |
> |---|---|---|
> | `juju` client, `openstack` CLI (phase-03..06) | **THIS DC's RACK** | **D-138**: they dial the CLOUD at L3, and there is NO L3 path from `voffice1` to any DC node plane -- SEC-010 drops forwarding across the transit leg and the plane nets are isolated libvirt nets |
> | `maas` CLI, NetBox, the inner `tofu` roots | **`voffice1`** | application-proxied or `qemu+ssh`-mediated; D-128 Plane 2 is UNCHANGED for these |
> | anything at all | **never the vcloud jumphost** | measured 2026-07-29: `command -v juju` -> ABSENT, `command -v maas` -> ABSENT. It picks the transport only |
>
> **THE SUPERSEDED TEXT, kept so the error is legible:** this block used to read
> "*Every `juju` and `maas` command in this runbook runs there [voffice1]*",
> citing the 2026-07-27 GA-R5 ruling. That ruling was about DC **ordering**, not
> run location, and **D-138 (2026-07-30) moved the cloud-facing half into the
> DC** after three bootstrap attempts failed on exactly this. **MEASURED
> 2026-08-02, which is what makes this unambiguous: `juju controllers` on
> `voffice1` returns `ERROR No controllers registered`, while the same command
> on the dc0 rack returns the live `vr1-dc0-controller`.** A session following
> the old text would not merely stall -- it would run `juju` against nothing and
> read the failure as a network fault.
>
> **PRACTICAL CONSEQUENCE, recorded because it bit on 2026-08-02:** the rack has
> NO repo clone, and scripts that source `lib-net.sh` relative to their own path
> (`juju-spaces-check.sh`) therefore cannot simply be piped through `bash -s`.
> The 2026-07-30 session staged the needed files to `~/repo-stage/scripts/` on
> the rack. **Verify those staged copies by sha256 against the repo before
> trusting them** -- a stale staged script is a confidently wrong answer.

**CHECK (read-only) -- voffice1**
```bash
export DC=vr1-dc0            # or vr1-dc1 -- explicit, region-qualified (D-119), never omitted
export DC_MODEL="$DC"        # ONE juju model per DC, named for the site slug.
                             # Used by every -m below; do NOT leave it unset --
                             # an unset ${DC_MODEL} expands to -m "" and juju
                             # operates on whatever model happens to be current.
export MAAS_PROFILE="${MAAS_PROFILE:-admin}"   # same default the scripts use
                             # (scripts/maas-role-tags.sh:37, pre-flight-checks.sh:53);
                             # never `maas list` -- DOCFIX-016.
source scripts/lib-net.sh;   lib_net_select_dc   "$DC"
source scripts/lib-hosts.sh; lib_hosts_select_dc "$DC"
echo "DC=$DC  DC_MODEL=$DC_MODEL"
echo "PLANE_CIDRS: ${PLANE_CIDRS[*]}"
echo "HOSTS: ${HOSTS[*]}"
```
**GATE:** both selector calls exit 0, and `DC`/`DC_MODEL` echo non-empty. If
`lib_hosts_select_dc "$DC"` fails loud, Stage 4 has not populated real host
data for this DC -- stop and go back, do not hand-supply the values.

---

## Step 2 -- Bootstrap a dedicated, in-DC, single-unit Juju controller [MUTATION, gated]

Per D-104 (as AMENDED): ONE Juju controller PER DC, deployed IN that DC on
that DC's OWN dedicated controller VM, single-unit (not HA) for VR1, managing
ONLY its own cloud. This is explicitly NOT the same controller model as
VR0/DC0's original bootstrap -- a fresh `juju bootstrap` targeting THIS DC's
MAAS cloud/region, not a reused controller.

**The controller target is a dedicated 10th VM, and it is READY, not Deployed.**
Under the READY-handoff ruling (DOCFIX-200, 2026-07-23) Stage 4 hands off nodes
in `Ready` with MAAS-deploy SKIPPED -- Juju provisions them here. Any surface
saying the bootstrap lands "on a MAAS-deployed machine" is stating the premise
that ruling removed. The controller VM is NOT one of the nine bundle machines
(`bundle.yaml` machines block: "The Juju CONTROLLER is a dedicated 10th VM
(D-104 amendment), bootstrapped by its own tag -- NOT a bundle machine").

**Step 2.0 -- Per-DC MAAS service credential [CHECK FIRST; both may already exist]**

Juju authenticates to the MAAS region with an API key. Per the per-DC
credential-isolation discipline (SEC-012/-016; Roosevelt analog: per-DC cloud
credentials), each DC gets a DEDICATED MAAS admin service user + its OWN key --
never reuse one key across DCs.

> **CHECK BEFORE YOU MINT. Both per-DC credentials already existed as of
> 2026-07-27** (`vr1-dc0-cred` and `vr1-dc1-cred`, SEC-018 / SEC-019;
> `juju show-cloud vr1-maas --client` reported `credential-count: 2`).
> Re-running the mint blocks below as written creates a DUPLICATE MAAS admin
> user and a SECOND key for a DC that already has one -- credential sprawl
> against SEC-018/-019 and D-137, on a register that is already red. Minting a
> credential is also a SEC-row obligation, not a free action.

**CHECK (read-only) -- voffice1**
```bash
juju credentials --client                       # is "$DC-cred" already listed?
maas "$MAAS_PROFILE" users read | grep -o "juju-$DC" || echo "no juju-$DC user"
ls -d ~/"$DC"-creds 2>/dev/null || echo "no ~/$DC-creds folder"   # SEC-009 site folder
```
(`maas <profile> users read` is the enumeration used to account for every MAAS
account on this deployment -- `docs/CURRENT-STATE.md:2784`. Do NOT open anything
inside the creds folder; its existence is all this gate needs.)
> **DOCFIX-206, 2026-07-30 -- THE CHECK ABOVE IS NOT REGION-SCOPED, AND UNDER
> D-132 q1 THAT MAKES IT WRONG.** Once each DC has its OWN MAAS region, all
> three commands can report "already exists" while the user exists in the
> WRONG region. MEASURED on the dc0 migration: `juju credentials --client`
> listed `vr1-dc0-cred`, `~/vr1-dc0-creds` was present, and `juju-vr1-dc0`
> existed -- **in OFFICE1**. The new region held only `MAAS`, `admin`,
> `maas-init-node`. Following the SKIP branch would have gone straight to a
> bootstrap that fails on authentication, which reads like a network fault.
> **The regions hold SEPARATE DATABASES, so a user in one is not a user in the
> other, and a key minted in one will NEVER authenticate against the other.**
> Every command in the CHECK block must therefore name the region explicitly:
> run `maas <profile> users read` with the PER-DC profile (and prove which
> region that profile resolves to with `scripts/maas-profile-assert.sh
> <profile> <rack>` FIRST -- a machine count is not proof), and read the
> cloud's endpoint with `juju show-cloud vr1-maas --client` rather than
> trusting the credential's NAME. A credential's name says nothing about which
> region it authenticates against.

**GATE -- three outcomes, and only one of them proceeds:**
- This DC's credential is LISTED **and its MAAS user exists IN THIS DC'S OWN
  REGION** (proven by profile assert, not by name) -> **SKIP the rest of
  Step 2.0 entirely** and go to the bootstrap CHECK below. Do not re-mint.
- The credential is listed but the user lives in ANOTHER region -> the
  credential is STALE, not present: remove it (`juju remove-credential
  vr1-maas <name> --client`) and mint into the correct region. Leaving both
  would give the cloud `credential-count: 2` and juju may pick either.
- Neither exists -> run the mint blocks that follow, and open the per-DC SEC row.
- One exists without the other (a user with no registered credential, or a
  credential naming a user MAAS does not have) -> **STOP.** That is a half-made
  credential, not a missing one; reconcile it against `docs/security-ledger.md`
  before touching either side. Do not "make it consistent" by minting the
  missing half.

The mint blocks below run ONLY on the second outcome. On the MAAS region host
-- **which under D-132 q1 is that DC's OWN region VM (`vr1-dc<N>-maas`, the
D-134 `.6`), NOT `voffice1`.** `voffice1` is the OFFICE1 region and minting
there produces a user the DC's own region has never heard of:
```bash
# service account -- Juju auths via the API key, not this password; use a random one
sudo maas createadmin --username="juju-$DC" --email="juju-$DC@<realm>"
```
Generate the key so it lands ONLY in the JUMPHOST per-site creds folder
(`~/$DC-creds/maas-api-key.txt` on vcloud -- the SEC-009 `~/<site>-creds/`
convention, and the site key IS the D-119 slug; NOT a new location on the
region), never echoed to a terminal (a printed MAAS key is burned, SEC-006).
Run the generate FROM the jumphost so the key transits ssh stdout straight into
the vcloud file (no persistent copy on the region):
```bash
ssh <region> "sudo maas apikey --username=juju-$DC --generate" \
  > ~/"$DC"-creds/maas-api-key.txt
chmod 600 ~/"$DC"-creds/maas-api-key.txt
awk -F: '{print "parts:", NF}' ~/"$DC"-creds/maas-api-key.txt   # expect 3; do NOT cat
```
Register the MAAS cloud + this credential in Juju consuming the key BY PATH/stdin
(never argv). Each key is a rotation obligation -- open a per-DC SEC row when issued.

> **RE-POINT THE CLOUD ENDPOINT TOO, and read it back.** A new credential against
> a cloud still pointing at the old region fixes nothing: `jujud` dials the region
> API for its whole life, and an Office1 endpoint is what killed bootstrap attempt
> 3 on 2026-07-30. `juju update-cloud vr1-maas --client -f <file>` with
> `endpoint: http://<dc-region>:5240/MAAS`, then **read it back** with
> `juju show-cloud vr1-maas --client` -- a command that returns is not evidence.
>
> **TWO SNAP-CONFINEMENT TRAPS, both measured 2026-07-30, both costing an attempt.**
> `juju` is a SNAP, so the YAML file you pass to `update-cloud` / `add-credential`:
> (a) must NOT live in `/tmp` -- a snap has a PRIVATE `/tmp` namespace, so a file
> the shell writes there is INVISIBLE to juju, which reports `no such file or
> directory` for a path that demonstrably exists; and (b) must NOT be under a
> DOT-directory in `$HOME` -- the snap `home` interface grants access to
> NON-HIDDEN files only, so `$HOME/.stage/cloud.yaml` fails `permission denied`
> on a directory the caller owns. Stage under a plain `$HOME/<name>` directory.
> Two different error messages, one cause; neither names confinement.
>
> **Build the YAML with `printf`, one line per statement, values passed as `%s`
> ARGUMENTS** -- not a heredoc. A heredoc's correctness depends on invisible
> leading whitespace on a line nobody reads, and this repo has two separate
> defects (F8, F9) from exactly that.
>
> **PROVE THE KEY AUTHENTICATES BEFORE BOOTSTRAPPING.** Bootstrap is one-shot and
> has no `--dry-run`. A scoped `maas login <tmp-profile> <url> "$(cat <keyfile>)"`
> followed by `users read`, `machines read`, and a `rack-controllers read` that
> confirms the DC-LOCAL rack -- then `maas logout <tmp-profile>` -- proves
> authentication, superuser scope, and REGION IDENTITY in four read-only calls.
> A credential that EXISTS is not a credential that WORKS.

**CHECK (read-only) -- voffice1 -- the cloud is registered AND the controller tag exists**
```bash
juju clouds --client | grep -i maas
# The bootstrap constraint below is only as good as the tag behind it. Fork 3's
# per-role tagger owns the D-104 controller tag (scripts/maas-role-tags.sh:125,
# "juju-controller-$SITE") -- confirm THIS DC's tag exists and is on exactly one
# machine before constraining on it:
bash scripts/maas-role-tags.sh check "$DC"
maas "$MAAS_PROFILE" machines read \
  | jq -r --arg t "juju-controller-$DC" '.[] | select(.tag_names[]? == $t) | "\(.hostname) \(.status_name)"'
```
**GATE:** the tag `juju-controller-$DC` exists AND the query returns EXACTLY ONE
machine, in `Ready`. Zero machines -> the tag was never applied; `juju bootstrap`
would find no candidate and fail at allocation. More than one -> STOP; two
machines carrying a single-controller tag means the bootstrap target is
ambiguous and you cannot predict which it takes. A status other than `Ready`
(including a cached `error` power state) is NOT a pass -- refresh it with
`maas "$MAAS_PROFILE" machine query-power-state <system_id>` and re-read, rather
than assuming the node is broken (a stale cached `error` on nodes that are
simply off is a measured trap, 2026-07-27).

**MUTATION -- voffice1**
```bash
juju bootstrap <this-dc-maas-cloud-name> <controller-name-per-DC> \
  --constraints "tags=juju-controller-$DC"
```
**The `--constraints` is not optional.** Without it Juju asks MAAS for any
available machine and can land the controller on one of the nine role nodes --
the exact outcome the D-104 amendment rejected, and one you only discover after
the bundle deploy finds a node missing. The tag name is DERIVED from `$DC`, so
it stays correct for both DCs by construction; it is not a literal to retype.

The MAAS cloud name and the controller name are session-specific identifiers you
choose/confirm at execution time (reflecting the site slug in the controller name
for operator clarity) -- not invented here.

Per D-104's rationale: this keeps DR-independence honest (a controller loss
affects only its own DC) and is a deliberate VR1 test-only economy (HA
deferred to Roosevelt) -- do not "upgrade" this to HA mid-stream without a
new D-NNN, that would silently reverse an ADOPTED decision.

---

## Step 3 -- `preflight.sh` PASS for THIS DC (read-only gate)

```bash
DC="$DC" bash scripts/preflight.sh
```
**`DC=` is load-bearing -- do not run this bare.** P2 defaults to `vr1-dc0`
(`scripts/preflight.sh:99`) and assembles the overlay set from `$DC`
(`:100-105`), so a bare run on the other DC validates dc0's numbers against
dc0's bands and never parses this DC's artifact at all -- a green gate that
examined the wrong input.

**Expect** the banner to name THIS DC before anything else:
`validating the MERGED <your DC> deploy input: bundle.yaml --overlay ... --dc <your DC>`.
If that line names a different DC than the one you exported, STOP -- `$DC` did
not reach the script.

**GATE:** `PREFLIGHT: PASS`. Two failures are known and expected on the vcloud
jumphost rather than on `voffice1` (the deliberately-absent gitignored octavia
PKI overlay, and MAAS unreachable) -- which is another reason this runs on
`voffice1`. A red preflight is a stop, not a note: read every FAIL and decide
per row, and record the decision. Do not run the sub-gates piecemeal to get a
greener-looking answer.

---

## Step 3.5 -- Model, spaces gate, and this DC's artifact source [MUTATION, gated]

Three things that must exist BEFORE the deploy and that no earlier step creates.

**MUTATION -- voffice1 -- create THIS DC's model (guarded)**
```bash
( {
  # Same capture-then-test idiom as scripts/juju-spaces-check.sh:40-42 (DOCFIX-115):
  # capture first, then test. The live-pipe form races SIGPIPE under pipefail and
  # reports a juju AUTH failure as "model not present" -- which here would mean
  # add-model runs against a controller you are not actually talking to.
  MJSON="$(juju models --format json)" || { echo "ABORT: 'juju models' failed (auth/macaroon? re-login, or append </dev/tty)"; exit 1; }
  MNAMES="$(jq -r '.models[]?.name' <<<"$MJSON" 2>/dev/null | sed 's#.*/##')" || true
  if grep -qx "$DC_MODEL" <<<"$MNAMES"; then
    echo "ABORT: model '$DC_MODEL' already exists -- do not deploy into a used model"
  else
    juju add-model "$DC_MODEL"
  fi
} )
```
One model per DC, named for the site slug. Deploying two DCs into a model named
`openstack` on one client is how the second deploy lands on top of the first.

**CHECK (read-only) -- voffice1 -- per-model spaces**
```bash
bash scripts/juju-spaces-check.sh "$DC_MODEL"
```
Spaces are PER-MODEL in Juju, so this can only run after `add-model` and must
run before `juju deploy` -- a missing or stale space name mis-binds charms at
deploy time and is expensive to unpick afterwards.
**GATE:** exit 0. Exit 1 = missing or stale spaces, STOP. Exit 2 = the model is
not present yet, which means the block above did not do what you think it did.

**MUTATION -- voffice1 -- point this DC's model at ITS OWN artifact source**

> **ORDERING TRAP -- this is why the step is here and not later.** If the model
> has no artifact source set, every unit apts from the INTERNET during deploy.
> The D-135/D-107 egress narrowing then lands on a half-built model, and unlike
> bootstrap the recovery is not "re-open egress and retry" -- it is a
> partially-installed cloud. Set it BEFORE the deploy, not during.

**The two DCs DIFFER BY RULING (D-135 + its 2026-07-24/27 amendments) -- there
is no common command here.** `vr1-dc0` runs a FULL MIRROR; `vr1-dc1` runs an APT
CACHING PROXY. This is the experiment, not an interim state. Read the address
from that DC's OWN checker rather than typing it:

```bash
# vr1-dc1 -- apt caching proxy (apt-cacher-ng). Confirm it is up and note its LISTEN:
bash scripts/dc-cache-proxy.sh check vr1-dc1
juju model-config -m "$DC_MODEL" apt-http-proxy="http://<LISTEN from the check>:3142"
juju model-config -m "$DC_MODEL" apt-http-proxy          # READ BACK -- assert on content
```
Reference values, for recognising a wrong answer rather than for typing:
`scripts/dc-cache-proxy.sh:81,87` give `vr1-dc1` LISTEN `10.12.68.4` port `3142`;
`scripts/dc-mirror.sh:79` gives `vr1-dc0` LISTEN `10.12.8.4`. If the checker
reports anything else, the checker wins and you stop to reconcile.

> **`vr1-dc0` (mirror) -- VERIFY THE KEY NAME BEFORE YOU RUN IT.** The mirror is
> consumed via a DIFFERENT model-config key than the proxy. The only in-repo
> statement of that key is a comment (`scripts/dc-mirror.sh:234`,
> `apt-mirror=http://<LISTEN>/ubuntu`), and no juju client exists on the vcloud
> jumphost to check it against, so it is NOT written here as an executable line.
> At execution, confirm the exact key with `juju model-config --help` (or
> `juju model-defaults`) on `voffice1` FIRST, then set it and read it back the
> same way as above. Do not guess a key name -- a mistyped model-config key is
> accepted silently and leaves the model pointing nowhere.

**GATE:** the read-back prints the address you intended, and it is THIS DC's own
artifact service. An empty read-back is a FAIL, not a pass -- checking `vr1-dc1`
as a mirror (or `vr1-dc0` as a proxy) fails BY DESIGN and is not a reason to
skip the step.

---

## Step 3.9 -- DC EGRESS gate, immediately before the deploy [read-only, REQUIRED]

> **ADDED 2026-08-02 after a 19-hour dc0 egress outage no gate could see.** Step
> 3.5 already points the model at THIS DC's artifact source, but that only covers
> **apt**. The deploy also needs two sources with **no local copy at all** --
> `streams.canonical.com` for juju agent binaries and `api.snapcraft.io` for
> snaps (the D-135 items 2-3 gap). On 2026-08-01 the mirror served 200 all day
> while the DC had no path off-site, so an apt-only check would have passed and
> the deploy would have failed the same way the 2026-07-31 attempt did.

```bash
ssh <this DC's rack> 'bash -s -- check <dc0|dc1>' < scripts/dc-egress-check.sh
```

**GATE:** exit 0. Capture the output and cite it in the commit that records the
deploy. Exit 2 is REFUSE ("could not look" -- usually run off the rack), never a
pass. The check is layered: read the FIRST FAIL as the cause. `A2` failing means
the edge itself is down and nothing downstream can work.

`preflight.sh` gate **P9** covers the same ground but WARNs when run off the rack
(which is where preflight normally runs), so **P9 passing is not a substitute for
running this on the rack.**

---

## Step 4 -- Deploy `bundle.yaml` + THIS DC's overlays [MUTATION, gated]

This step carries its OWN DC-parameterised deploy command. It no longer says
"follow phase-01 verbatim": phase-01's deploy blocks were written around VR0's
single-DC testcloud, and following them unchanged against a VR1 DC is a known
defect class, not a shortcut. Use `runbooks/phase-01-bundle-deploy.md` for the
things that genuinely transfer -- Step 1.0's Octavia PKI overlay sanity, the
convergence-watch pattern, and the PRE-vault-init end-state description -- with
`$DC` and `$DC_MODEL` exported per Step 1 above. **Dependency: phase-01's own
per-DC parameterisation (its `$DC`/`$DC_MODEL` handling and its per-DC VIP
guard) is delivered separately in this same batch; if you reach a phase-01 block
that still hardcodes `10.12.4.`, VR0 system_ids, `openstack0..3` disk paths, a
4-machine plan, or `juju add-model openstack`, that block has NOT been
parameterised yet -- STOP and reconcile rather than adapting it live.**

### 4.1 -- Assemble THIS DC's overlay set

Deploy input = `bundle.yaml` + this DC's overlays. The base bundle is VIP-FREE
and is NOT a deployable input on its own.

| Overlay | `vr1-dc0` | `vr1-dc1` |
|---|---|---|
| `overlays/${DC}-vips.yaml` | REQUIRED (`vr1-dc0-vips.yaml`) | REQUIRED (`vr1-dc1-vips.yaml`) |
| `overlays/${DC}-machines.yaml` | **REQUIRED** (`vr1-dc0-machines.yaml`) | **REQUIRED** (`vr1-dc1-machines.yaml`) |
| `overlays/${DC}-octavia-pki.yaml` | REQUIRED (gitignored secret) | REQUIRED (gitignored secret) |

**BOTH DCs take a machines overlay. DOCFIX-208, 2026-08-02.** This row and the
`vr1-dc0` command blocks below previously read "none -- and that is correct" and
"there is no `overlays/vr1-dc0-machines.yaml`". That file HAS EXISTED since
2026-07-29 and the dc0 deploy REQUIRES it. What made the old text wrong is that
it reasoned from the FILE NAME: the overlay's job is no longer a machines retag,
and dc0 still needs none -- `bundle.yaml`'s machines block already carries
`tags=openstack-vr1-dc0,<role>`, and `overlays/vr1-dc0-machines.yaml` has NO
`machines:` block at all (measured: its only top-level key is `applications:`,
so passing it cannot move a tag or a machine id). It is the per-DC
HAND-MAINTAINED overlay, and its `applications:` half is load-bearing.

**MEASURED 2026-08-02 -- the dc0 overlay's delta to the merged input is SIXTEEN
applications, not one:**

- `ovn-chassis.bridge-interface-mappings` (ADDS) -- the two dc0 compute
  provider MACs. `bundle.yaml:494` states this key "IS NOT IN THIS FILE, and
  must never be re-added here", so the overlay is its ONLY source. Deploy dc0
  without it and ovn-chassis builds no `br-ex` mapping: tenant networks come up
  with no external path, surfacing later as broken egress rather than as a
  deploy error. This is the exact defect the 2026-07-29 move out of the base
  bundle was made to fix.
- **15 apps repointed at the in-DC mirror** (12 `openstack-origin`, 3 ceph
  `source`; 6 OVERRIDE a base value, 9 ADD one) per the 2026-07-31 GA-R5 ruling
  "Point origin/source at the mirrored UCA, per-DC overlay". Without it those
  charms fetch from `ubuntu-cloud.archive.canonical.com`, MEASURED UNREACHABLE
  from a node (000) under the D-107 airgap.

Supersedes the narrower 2026-07-31 finding in `docs/CURRENT-STATE.md` that the
overlay's "only delta" is `bridge-interface-mappings` -- true when measured, and
overtaken hours later by the UCA build in the same session (GA-R1 C2:
measurement wins, and this is the correcting capture).

The `[ -f ]` guard in `scripts/preflight.sh:144-145` and
`runbooks/phase-01-bundle-deploy.md:199-201` assembles this same three-overlay
set, so P2 was validating the right input and this runbook was the wrong
surface. Never "guard" a `--overlay` argument with a silent skip in a DEPLOY
command: an overlay that quietly disappears is exactly how a `vr1-dc1` deploy
would merge `vr1-dc0`'s tags, and how a dc0 deploy lost its provider mapping.

Make NO hand edit to `bundle.yaml` or to either generated VIP overlay (ruling 3,
2026-07-25; both `*-vips.yaml` are RENDERED -- edit `render/values/${DC}-vips.yaml`
and re-render, and `tests/render-drift` fails the gauntlet if you do not).

**There is no separate HA overlay in this command, by design.** The HA chain now
lives IN the base bundle (BUNDLEFIX-053, 2026-08-05); `overlays/dc-ha-scaleup.yaml`
was RETIRED 2026-08-06 (its content folded into `bundle.yaml`; D-121 R6 supersession
note). So this single deploy of base + the per-DC VIP overlay lands 3-unit HA
directly -- `num_units: 3` and `cluster_count: 3` are the EXPECTED post-deploy values
(see Step 12's gate). There is no "apply the overlay later" step to sequence.

**Octavia PKI is PER-DC.** Under the R7 amendment (D-109, RULED 2026-07-27) the
generator is no longer dc0-frozen and each DC gets its own CA material, so this
is a per-DC artifact (`overlays/${DC}-octavia-pki.yaml`), not one shared file.
Reuse this DC's existing overlay if its CAs are still valid (10y, per
phase-01 Step 1.0), OR regenerate via phase-01 Step 1.0-GEN with `DC` exported.
Either way it is an explicit operator call, logged as a decision note -- not a
silent default to reuse. Cross-DC reuse was refused on posture: the overlay
carries CA private keys plus a plaintext passphrase in a repo SEC-004 records as
PUBLIC.

### 4.2 -- DRY-RUN FIRST, and read the machines block [read-only gate]

> **Run this BEFORE 4.3. It is the only thing standing between a typo in the
> overlay set and nine wrongly-allocated MAAS machines.** There is no offline
> juju merger that can prove either half of `overlays/${DC}-machines.yaml`
> merges -- dc1's machines-block retag, or BOTH DCs' `applications.options`
> key-by-key merge (see each overlay's own VERIFY-LIVE note) -- so the dry-run
> IS the proof. Gate items 5 and 6 below are that proof for the options half.

**CHECK (read-only) -- voffice1 -- run the block for YOUR DC.** These are
EXACTLY the 4.3 commands plus `--dry-run`; if the two ever differ, this gate is
validating something you are not deploying.

**`--debug` IS NOT OPTIONAL HERE, AND IT IS THE ONLY REASON ITEM 1 CAN FAIL.**
MEASURED 2026-08-02 (`docs/audit/stage5-dryrun-dc0-20260802.txt`): a plain
`juju deploy --dry-run` on a bundle prints ONLY located charms, upload/deploy
application lines, BARE machine ids, unit/container placement, and relations. It
prints no constraints at all. `--debug` adds the per-machine constraint lines
(`bundlehandler.go:1035`) that item 1 reads, and nothing else. Run the block for
YOUR DC and keep `--debug` in it.

`vr1-dc0` (order: vips -> machines -> pki):
```bash
juju deploy ./bundle.yaml \
  --overlay overlays/vr1-dc0-vips.yaml \
  --overlay overlays/vr1-dc0-machines.yaml \
  --overlay overlays/vr1-dc0-octavia-pki.yaml \
  -m "${DC_MODEL}" --dry-run --debug
```
`vr1-dc1` (machines overlay REQUIRED, order: vips -> machines -> pki):
```bash
juju deploy ./bundle.yaml \
  --overlay overlays/vr1-dc1-vips.yaml \
  --overlay overlays/vr1-dc1-machines.yaml \
  --overlay overlays/vr1-dc1-octavia-pki.yaml \
  -m "${DC_MODEL}" --dry-run --debug
```
**GATE -- assert on the CONTENT of the plan, not on the command exiting 0:**
1. **Every machine reads `tags=openstack-$DC,<role>`.** If any machine still
   names the OTHER DC, STOP -- one Office1 MAAS region sees BOTH DCs' nodes, so
   a wrong tag here means this model can allocate the other DC's hardware. The
   fallback is inlining this DC's tags into a rendered per-DC bundle, not
   deploying anyway.
2. **Nine machines, `"0"` through `"8"`**, matching `bundle.yaml`'s machines
   block (D-121 Option C: 3 control / 2 compute / 4 storage). A four-machine
   plan means you are looking at VR0's topology.
3. If the plan reports an ERROR on any `--overlay` path, do NOT drop the overlay
   to make it run. A missing overlay is a missing input, not an optional one.

> **>>> WHAT THIS DRY-RUN CANNOT SEE. DO NOT ADD AN ASSERTION HERE FOR ANY OF IT.
> <<<** MEASURED 2026-08-02, both verbosities
> (`docs/audit/stage5-dryrun-dc0-20260802.txt`): the plan carries **NO
> application options and NO VIPs**, at any verbosity. Greps over the `--debug`
> capture return ZERO for `bridge-interface-mappings`, `physnet1`,
> `openstack-origin`, `cloud-archive` and `prefer-chassis`; the only `10.12.4.`
> hits are the controller's own API addresses and the only `vip` hit is the
> echoed command line. This gate previously carried a fourth item asserting
> "every VIP in the plan is on THIS DC's prefixes" -- **it could never have
> failed**, because no VIP is ever in the plan. Two more option-reading items
> were added and removed within the same session, 2026-08-02, for the same
> reason: the runbook is not exempt from the standing rule that a checker which
> cannot fail is not a gate. The properties are real and still checked -- below.

**WHERE THE VIP AND OPTION PROPERTIES ARE ACTUALLY GRADED:**

- **VIPs, and the merged input generally -- OFFLINE, BEFORE this step.**
  `scripts/preflight.sh` P2 runs `provider-bundle-check.py` over `bundle.yaml`
  plus this DC's SAME three overlays and asserts the VIP bands per DC
  (`scripts/lib-net.sh:54-56`, `:146-154`). That gate reads the merged INPUT,
  which is exactly the artifact the dry-run does not expose. A VIP from the
  other DC fails there, not here.
- **`ovn-chassis`'s merged options map, and the 15 mirror origins -- LIVE, in
  Step 4c, immediately after the deploy returns.** No offline check can prove
  them: `provider-bundle-check.py`'s `_deep_merge` merges key-by-key BY
  CONSTRUCTION, so it cannot distinguish juju merging an `options` map
  key-by-key from juju REPLACING it. Only the deployed application's own config
  can, and the dry-run is not it.

Capture this output. It is the plan-count evidence the exit gate cites -- plan
counts are quoted only from captured command output, never from this file's
prose (GA-R1 rule 2).

### 4.3 -- The real deploy [MUTATION, gated]

**The SAME block you dry-ran in 4.2, minus `--dry-run`. Nothing else changes.**

`vr1-dc0`:
```bash
juju deploy ./bundle.yaml \
  --overlay overlays/vr1-dc0-vips.yaml \
  --overlay overlays/vr1-dc0-machines.yaml \
  --overlay overlays/vr1-dc0-octavia-pki.yaml \
  -m "${DC_MODEL}"
```
`vr1-dc1`:
```bash
juju deploy ./bundle.yaml \
  --overlay overlays/vr1-dc1-vips.yaml \
  --overlay overlays/vr1-dc1-machines.yaml \
  --overlay overlays/vr1-dc1-octavia-pki.yaml \
  -m "${DC_MODEL}"
```

Open the Step 4b watch windows before this returns.

- **DESIGNATE DEPLOYS HERE, AND THAT IS CORRECT. CORRECTED 2026-08-02 (DOCFIX-208).**
  This bullet used to read `bundle.yaml` ships "NO designate" for this stage. That
  was VR0's posture and it has been wrong since 2026-07-10. **D-019 (v1 ships no
  cloud DNS) is SUPERSEDED by D-106, which REACTIVATES Designate for VR1** --
  Designate is one of this generation's ADDED services, not a deferred one, and
  the VR1 buildout layers new services onto each previous successful deployment
  by design. DOCFIX-167 put `designate`, `designate-bind`, `designate-mysql-router`
  and `designate-hacluster` INTO the `bundle.yaml` this stage deploys;
  `runbooks/phase-01-bundle-deploy.md:13-16` and `:174` already record that and
  say explicitly it "corrects this GATE's old 'NO designate (D-019)' text".
  Confirmed live: the dc0 dry-run deploys all four and places `designate/0` on
  `0/lxd/5` (`docs/audit/stage5-dryrun-dc0-20260802.txt`). **EXPECT THEM IN THE
  PLAN; their absence would be the defect.**
- **What IS Stage 7's job is the DNS ACTIVATION, not the charm deploy** -- per-DC
  zones with A/AAAA per the D-008 bootstrap order, the FQDN-SAN certs that close
  the D-019/D-021 root cause, and the coupled B5 `os-public-hostname` reversal
  (`docs/dc-dc-deployment-workflow.md` Stage 7; `bundle.yaml:898-902` says the
  same: "the zone name/A/AAAA population itself is NOT part of this bundle").
  Do not perform that activation mid-Stage-5; do not read it as a reason to keep
  the charms out.
- Settle to phase-01's documented PRE-vault-init end state, read with `$DC_MODEL`
  substituted for its hardcoded model name: ZERO units in `error`;
  mysql-innodb-cluster ACTIVE; `vault/0` BLOCKED "Vault needs to be initialized"
  (the Step 5 trigger, not a fault); the TLS consumers waiting on vault certs;
  octavia BLOCKED "Awaiting configure-resources".

### 4.4 -- Assert the MERGED CONFIG live, before the model settles [read-only gate]

**Run this as soon as `juju deploy` returns -- do not wait for convergence.**
Added DOCFIX-208. Both properties below decide whether this deploy can ever
work, and **NOTHING before this point can read either of them**: the dry-run
carries no options (4.2's note, measured), and no offline merger can prove
juju's merge behaviour. If either fails, the fix is a `juju config` correction
on a young model, not a diagnosis after hours of failed hook retries.

**CHECK (read-only) -- from the DC's juju client**
```bash
juju config ovn-chassis -m "${DC_MODEL}" | grep -A2 -E \
  'ovn-bridge-mappings|prefer-chassis-as-gw|bridge-interface-mappings'
for app in barbican cinder designate glance keystone magnum neutron-api \
           nova-cloud-controller nova-compute octavia openstack-dashboard placement; do
  printf '%-24s %s\n' "$app" "$(juju config "$app" -m "${DC_MODEL}" --format=json \
    | python3 -c 'import json,sys; s=json.load(sys.stdin).get("settings",{}); print(s.get("openstack-origin",{}).get("value","<<UNSET-OR-APP-ABSENT>>"))' \
    2>&1 | tail -1)"
done
for app in ceph-mon ceph-osd ceph-radosgw; do
  printf '%-24s %s\n' "$app" "$(juju config "$app" -m "${DC_MODEL}" --format=json \
    | python3 -c 'import json,sys; s=json.load(sys.stdin).get("settings",{}); print(s.get("source",{}).get("value","<<UNSET-OR-APP-ABSENT>>"))' \
    2>&1 | tail -1)"
done
```

> **Why `.get(...)` and a sentinel rather than a direct index.** A missing key
> would raise a `KeyError`, and a traceback printed into a results column reads
> like output -- this repo has a logged instance of an assertion being satisfied
> by a traceback. `<<UNSET-OR-APP-ABSENT>>` is a VISIBLE failure. Treat it as a
> failure, not as a blank: it means either the charm never got the option or the
> application is not in this model.
>
> **THIS BLOCK IS AUTHORED, NOT YET EXERCISED (2026-08-02).** It is written
> against a measured instrument -- `juju config` is what holds the merged
> configuration -- but it has never run, because the model was empty when the
> gate was written. Its FIRST execution is at this deploy. Expect to correct its
> shape, not its intent; report a correction rather than working around it.
> `designate` belongs in the loop: D-106 reactivates it for VR1 and the charm
> deploys at this stage (see Step 4.3's note).

**GATE:**
1. **`ovn-chassis` carries all THREE options.** `bridge-interface-mappings`
   (from `overlays/${DC}-machines.yaml`) **alongside** `ovn-bridge-mappings:
   physnet1:br-ex` and `prefer-chassis-as-gw: true` (B2, gateway-chassis
   election). All three = juju merged the `options` map key-by-key, as the whole
   `*-vips.yaml` architecture assumes. **Only `bridge-interface-mappings` = the
   map was REPLACED**: the physnet mapping and the gateway election are gone
   from the deployed config, and provider egress dies by a different route than
   the missing-overlay defect DOCFIX-208 fixes. STOP and `juju config` them back
   before anything relates. Exactly TWO MACs in the value; a third means a wrong
   node.
2. **Every origin/source resolves to an address THIS DC's nodes can reach.**
   `vr1-dc0`: all 15 read `deb http://10.12.8.4/cloud-archive
   jammy-updates/caracal main`. Any app still reading `cloud:jammy-caracal` or a
   bare `ubuntu-cloud.archive.canonical.com` URL is MEASURED UNREACHABLE from a
   node (000) under the D-107 airgap and will fail at package install, not at
   deploy. `vr1-dc1` correctly has NO such block -- its apt-cacher-ng forwards
   whatever URL it is handed. That asymmetry is D-135's experiment RESULT, not a
   defect to "fix" for symmetry.
3. **`ovn-central` is EXPECTED to read `source: zed`, not caracal** -- its charm
   default, logged and deliberately left alone in
   `overlays/vr1-dc0-machines.yaml`'s header because repointing it would change
   its RELEASE, not just its URL. Do not "correct" it here.

Capture this output; it is the first evidence in the chain that reads the
deployed configuration rather than an input to it.

---

## Step 4b -- Mid-deploy watch + failure decision tree (read-only, runs alongside Step 4)

Open BOTH windows before Step 4.3 returns. A deploy you are not watching is one
you diagnose from logs afterwards.

**CHECK (read-only) -- voffice1, window 1 (detail)**
```bash
juju status -m "$DC_MODEL" --watch 5s
```
**CHECK (read-only) -- voffice1, window 2 (signal)**
```bash
bash scripts/deploy-watch.sh "$DC_MODEL" 15
```
Compact machine/unit state COUNTS plus the NAMED principal units in
error/blocked. Health-at-a-glance: the error/blocked section stays EMPTY until
the expected late blocks (vault needs-init, octavia awaiting-configure).
Neither window descends into subordinates and neither replaces a phase gate.

**Expected end-state shape** (not a count to quote -- confirm against Step 4.2's
captured plan, GA-R1 rule 2): nine machines reaching `started`; zero units in
`error`; the only `blocked` units are `vault/0` (needs init) and `octavia`
(awaiting configure-resources). Anything else named in the error/blocked list is
this tree's input.

### Decision tree -- something went wrong mid-deploy

Work top to bottom. **Default is FIX-FORWARD**; juju converges, and tearing down
a partially-good model costs more than repairing it.

1. **Units are still `waiting`/`maintenance` and counts are MOVING.** Not a
   failure. Snap installs and cert exchange are slow on first boot. Keep
   watching; do not intervene because it is quiet.
2. **A machine never leaves `pending` / allocation fails.** This is a MAAS-side
   answer, not a juju one. Check this DC's node pool: the right tag on the right
   count of `Ready` machines (Step 2's tag check, widened to the role tags). A
   cached `power_state=error` on nodes that are merely off reads as a hard
   blocker and is not one -- refresh with
   `maas "$MAAS_PROFILE" machine query-power-state <system_id>` and re-read
   before diagnosing a fault. **Fix-forward:** correct the tags/power state and
   let juju retry. Do NOT destroy the model.
3. **Units are in `error` with a hook failure.** Read the actual hook error
   (`juju debug-log -m "$DC_MODEL" --replay --include <unit>`). If it is a
   transient fetch failure, first check whether this model's artifact source
   (Step 3.5) is set and reachable -- a model apting from the internet behind a
   narrowed edge fails exactly this way. **Fix-forward:** repair the cause, then
   `juju resolved --no-retry <unit>` or `juju resolved <unit>` as appropriate.
4. **The wrong DC's hardware got allocated, or a VIP from the other DC appears
   in the model.** STOP. This is the one case where fix-forward is wrong: the
   model has bound to another DC's addresses/machines, and repairing it in place
   leaves you unsure what is bound to what. Tear the MODEL down
   (`juju destroy-model "$DC_MODEL"`, operator-gated) and return to Step 4.2 --
   the dry-run gate exists precisely to make this case unreachable.
5. **You are considering destroying anything BELOW the juju layer** (the
   containment VM, the DC's libvirt resources, MAAS records). Read
   `runbooks/dc-dc-teardown-rollback.md`'s rollback decision tree FIRST, and
   read its Model-B banner before acting on any of its resource-level steps.
   **Standing caution as of 2026-07-29:** that runbook's Step 2 MAAS-side check
   and Step 3 scoped-destroy targets have known defects against the current
   two-root shape -- treat both as reference, verify every module name against
   the live `opentofu/` tree before running anything, and never let a whole-DC
   destroy be the first thing you reach for while 18 nodes are enrolled in MAAS.

At every branch: record what you observed and what you chose. A mid-deploy
decision that is not written down is one the next session has to re-derive.

---

## Step 5 -- Run phase-02 (vault bring-up) -- INDEPENDENT root per DC [adapted, SECRET-HANDLING]

Follow `runbooks/phase-02-vault-bringup.md` per DC, with `$DC`/`$DC_MODEL`
exported per Step 1 and substituted for any hardcoded model name in its command
blocks. Per D-109: each
DC's Vault is its OWN independent root CA -- no regional root-of-trust this
phase. This means:
- The `vault operator init` one-shot step (Step 2.1) runs ONCE PER DC,
  producing DISTINCT unseal-key shares and a DISTINCT root token for `vr1-dc0`
  vs. `vr1-dc1`. Store them separately, clearly labeled by DC (this repo's existing
  secrets discipline -- `~/vault-init/`-style off-host capture -- applies
  identically per DC, just not shared between them).
- D-109 also requires each DC's Vault to issue certs with BOTH IPv4 and
  IPv6 (ULA) SANs for the dual-stacked metal-internal API endpoints. **This
  is a real charm-behavior question not yet confirmed this session:**
  whether the `vault`/`vault-pki`-relation charms used in `bundle.yaml`
  actually support requesting/issuing IPv6 SANs today (vs. only IPv4) is
  UNVERIFIED -- check the actual charm config/relation-data schema (e.g.
  `openstack-loadbalancer`/certificates interface docs, or the vault charm's
  own config options) before assuming this "just works" once IPv6 addresses
  exist on the plane. If it does not support v6 SANs natively, that is a
  real finding to log (a D-NNN amendment or a new decision), not something
  to route around silently.

---

## Step 6 -- Write + apply the IPv6 family-matrix overlay [OVERLAY DRAFTED 2026-07-10, NOT YET APPLIED]

D-101's family matrix (Section 2 of the buildout design) requires, per DC:
- `data-tenant`, `storage`, `replication`: IPv6-ONLY ULA (no v4 config at
  all on these planes for the relevant charms -- e.g. `ceph-mon`'s
  `public-network`/`cluster-network`, OVN's geneve encapsulation address).
- `metal-internal`, `metal-admin`: DUAL-STACK (v4 retained + a ULA leg
  added -- the metal-admin ULA leg is the 2026-07-09 operator AMENDMENT to
  D-101, a reversal of that decision's own original lean, so do not assume
  older documentation/training-data patterns for "metal-admin stays v4-only"
  -- this repo's ADOPTED text says otherwise).
- `provider-public`: DUAL-STACK v4 + GUA (native, no NAT66 -- D-101's
  tenant/MTU sub-policy folded in from D-102).

**`overlays/dc-dc-ipv6-family-matrix.yaml` now exists**, drafted 2026-07-10
against REAL charm source/config (not guessed option names) -- see
`docs/archive/dc-dc-ipv6-charm-research.md` for full sourcing and per-charm
confidence level. Summary: `prefer-ipv6` (confirmed real option, additive
dual-stack HAProxy binding, confirmed via `charm-nova-cloud-controller`'s
actual "Dual Stack VIPs" commit) + appended v6/GUA entries on the existing
`vip:` option, for the nine dual-stack API charms directly confirmed or
sharing the same charms.openstack framework; `ceph-mon`'s own separate
`prefer-ipv6` + `ceph-public-network`/`ceph-cluster-network` (a clean
switch-to-v6-only, correct for the ULA-only planes); NO entry for OVN
(confirmed no such charm-config option exists -- geneve family follows the
bound interface automatically once the plane itself is v6-only).
**CAVEAT (measured 2026-08-09, do NOT read "follows the interface" as "just
works"):** a v6-only data-tenant is NECESSARY but NOT SUFFICIENT for
geneve-over-v6. Two extra conditions must hold, both verified at Step 12.2:
(i) the CONTAINERIZED chassis (octavia LXD) must actually take a v6 data-tenant
address -- they auto-picked v4 on vr1-dc0 (D-134), so carve/assign v6, do not
rely on auto-pick; and (ii) `ovn-chassis` 24.03 emits the v6 `ovn-encap-ip`
BRACKETED (`"[2602:...]"`), which OVS geneve rejects (`bad remote_ip`, tunnel
`ofport -1`) -- it must reach OVS UNbracketed (fixed charm rev or post-deploy
override). geneve-over-v6 IS confirmed working on this stack (OVS 3.3 / OVN
24.03 / kernel 5.15) once both hold. Full evidence + tested sequence:
`docs/audit/geneve-over-v6-rootcause-20260808.md`.

**This is an INCREMENTAL ADD against the LIVE `$DC_MODEL` Step 4 already
deployed -- not a fresh deploy.** It re-passes the SAME overlay set Step 4 used
and adds the family matrix on top, so juju's diff introduces only the
address-family options; the base topology, the VIPs and the machines block must
come out UNCHANGED. That is precisely what the GATE below checks. Re-passing the
full set is deliberate: dropping the VIP or machines overlay here would present
juju with an input that contradicts the running model.

Corrected 2026-07-29: this command previously named `overlays/octavia-pki.yaml`
and `overlays/${DC}-hostnames.yaml` and passed NO VIP overlay at all -- so since
ruling 3 its merged input carried ZERO VIPs, and `${DC}-hostnames.yaml` has
NEVER BEEN AUTHORED (it is unbuilt D-106 work, proposed in
`runbooks/dc-dc-phase6-designate-cos-magnum.md`), on which juju errors out.

> **THIS STEP HAS NOT BEEN APPLIED TO ANY LIVE MODEL** (overlay drafted
> 2026-07-10, still unvalidated -- see open item 3 below). Every block here is
> `--dry-run`; the real apply is a separate, gated decision once the diff has
> been reviewed. **Step 12's v6 gates depend on this step having actually been
> applied** -- if you have only dry-run it, 12.1 and 12.2 are BLOCKED, and
> "blocked on Step 6" is the honest recorded answer, not a soft pass.

`vr1-dc0`:
```bash
juju deploy ./bundle.yaml \
  --overlay overlays/vr1-dc0-vips.yaml \
  --overlay overlays/vr1-dc0-machines.yaml \
  --overlay overlays/vr1-dc0-octavia-pki.yaml \
  --overlay overlays/dc-dc-ipv6-family-matrix.yaml \
  -m "${DC_MODEL}" --dry-run
```
`vr1-dc1`:
```bash
juju deploy ./bundle.yaml \
  --overlay overlays/vr1-dc1-vips.yaml \
  --overlay overlays/vr1-dc1-machines.yaml \
  --overlay overlays/vr1-dc1-octavia-pki.yaml \
  --overlay overlays/dc-dc-ipv6-family-matrix.yaml \
  -m "${DC_MODEL}" --dry-run
```
Same overlay-set rules as Step 4.1: BOTH DCs pass their `${DC}-machines.yaml`,
and no overlay is ever dropped to make the command run. (DOCFIX-208: this
sentence previously exempted `vr1-dc0`, contradicting its own paragraph four
lines above -- "dropping the VIP or machines overlay here would present juju
with an input that contradicts the running model".)

Review the diff before applying for real, same discipline as every other
overlay in this repo. **GATE:** the diff changes ONLY the address-family options
this overlay sets; every VIP still reads on THIS DC's prefixes, and the machines
block still reads `tags=openstack-$DC,<role>`. If either moved, the overlay set
is wrong -- stop, do not apply.

**Still genuinely open, not resolved by the overlay:**
1. The 8 charms whose `prefer-ipv6` presence is INFERRED-BY-PATTERN (not
   individually fetched from their own `config.yaml`) need live `juju
   config <app>` confirmation once first deployed.
2. **RESOLVED 2026-07-27 by R8 -- this item is CLOSED and the text below is
   superseded.** Octavia's `lb-mgmt-net` is **IPv6 ULA, charm-created and
   charm-owned** (operator: "I want to take a Octavia creates and owns its own
   IPv6 network"). The two Launchpad numbers this item rested on were READ IN
   FULL and BOTH FAILED: #1913409 is **kolla-ansible**, a different installer,
   **Fix Released 2021**; #1911788 is **Incomplete**, a duplicate, and **is not
   an IPv6 defect** -- its cause is an OVN shortname/FQDN port-binding mismatch
   fixed in OVS 2.15, and these nodes install 2.17. Upstream research also found
   the charm creates a v6 subnet and **has no IPv4 code path at all**, so v4-only
   was never the safe option -- it was the departure. Do NOT re-raise this as an
   open decision. What IS owed here is the MTU obligation (LP #2018998), now
   asserted by `scripts/phase-05-octavia-verify.sh`, and gate **G18** (apex
   back-fill of the charm-created prefix), which blocks project close.
3. UNVALIDATED against any live model -- this overlay has not been applied
   or tested, same posture as every other artifact from tonight's
   prep-only session.

---

## Steps 7-9 -- phase-03 / phase-04 / phase-05, adapted per DC

Follow `runbooks/phase-03-core-verify.md`, `runbooks/phase-04-network-
carve.md`, `runbooks/phase-05-octavia-enablement.md` per DC, with `$DC` and
`$DC_MODEL` exported per Step 1, the selector discipline applied to every
command block in them that reads `lib-net.sh`/`lib-hosts.sh` values, and
`$DC_MODEL` substituted for any hardcoded model name.

**Known, measured caveat -- these three phases call scripts that do NOT select
a DC.** Measured 2026-07-29: of the scripts that source `scripts/lib-net.sh`,
these Stage-5 ones read a DC-dependent value while never calling
`lib_net_select_dc`, so they resolve `vr1-dc0`'s literals whichever DC you are
on: `scripts/phase-03-admin-openrc.sh` (`KEYSTONE_VIP_DEFAULT`),
`scripts/phase-04-network-create.sh` and `scripts/phase-04-network-verify.sh`
(both also carry a hardcoded `PROVIDER_CIDR="10.12.4.0/22"`, which a selector
would not reach), and `scripts/vault-kv-health.sh` (`METAL_INTERNAL_CIDR`).
On `vr1-dc0` that is harmless and correct. **On `vr1-dc1` it is wrong and
SILENT** -- no error, just dc0's addresses. Until those scripts are fixed,
either export the value explicitly (each honours an env override) or verify
every address in their output against this DC's planes before believing a PASS.
LOG anything further you find (a DOCFIX candidate); do not patch around it live.

---

## Step 10 -- `cloud-assert.sh --capture` (read-only + committed BOM)

```bash
bash scripts/cloud-assert.sh --capture
```
Per DC, at real deploy-completion. Confirmed repo-agnostic (2026-07-09
sweep). Commit the resulting `asbuilt/` BOM per this repo's existing
deploy-loop convention, labeled clearly by DC (do not let `vr1-dc0`'s and
`vr1-dc1`'s BOMs collide/overwrite -- confirm the script's own output path is
already DC-distinguishing, or add a DC-qualified path if not, BEFORE running
for the second DC; a silently overwritten BOM loses the first DC's evidence).

---

## Step 11 -- Controller backup [MUTATION, gated]

> **THE FLAG SHAPE BELOW IS UNVERIFIED AT AUTHORING -- CONFIRM IT FIRST.**
> No juju client exists on the vcloud jumphost (measured 2026-07-29:
> `command -v juju` -> ABSENT), so this shape could not be checked against a
> real client when it was written, and a repo-wide grep corroborates the
> subcommands only from other prose. What IS established: `-m` takes a MODEL,
> not a controller name -- so the previous form here (`-m <controller-name>`)
> was wrong on both lines; and `create-backup` downloads by default, which made
> the old second line redundant rather than necessary. **At execution, run
> `juju create-backup --help` on `voffice1` and use what it says.** Then gate on
> the resulting FILE, which is true regardless of flag spelling.

```bash
# Confirm the flags FIRST (see the caveat above):
juju create-backup --help

# Then take the backup against the CONTROLLER MODEL of this DC's controller:
juju switch <this-dc-controller-name>
juju create-backup -m controller
```
**GATE -- assert on the artifact, not on the command exiting 0:** a backup file
exists in the working directory, is NON-ZERO in size, and its timestamp is from
this run. Record its name and size. If `create-backup` printed a path but no
file is there, that is a FAIL -- do not record "backup taken".

Then store it OFF-HOST, labeled by DC, with the same secrets-adjacent handling
discipline as Vault's init capture, and confirm the off-host copy's size matches
before deleting anything locally. Per D-104: scheduled controller backups, one
per DC. The RESTORE half of this drill is exercised in Stage 6 (DR wiring and
failover drill) -- this step only takes and stores the backup, it does not
restore it.

---

## Step 12 -- VERIFY-LIVE gates + Ceph-over-v6 / geneve-over-v6 (READ-ONLY gate)

`juju run` on juju 3.x is the ACTION runner, so the old `juju run <unit> 'ceph -s'`
form asked ceph-mon to run an action NAMED `ceph -s` and could never execute.
`juju exec` is the command runner; the `</dev/null` is this repo's convention for
it (`scripts/phase-05-octavia-verify.sh:96`,
`runbooks/phase-05-octavia-enablement.md:93`).

### 12.1 -- Ceph over v6

```bash
# Ceph binds v6 on the storage/replication ULA planes (ms_bind_ipv6, D-101):
juju exec -m "$DC_MODEL" --unit ceph-mon/leader -- 'ceph -s' </dev/null
```
**GATE:** the mon addresses reported are on THIS DC's storage/replication ULA
planes, in `v2:`/`v1:` form. A v4 mon address on those planes is a FAIL, not a
"close enough". If the command errors, that is a FAIL too -- an unreadable
cluster is not a passing one.

### 12.2 -- Geneve encapsulation family (assert the OBSERVABLE, not a config key)

The old form here grepped `juju config ovn-central` for `encap`. Step 6 of this
same runbook states there is NO such charm-config option -- geneve family follows
the bound interface automatically once the plane is ULA-only -- so that grep could
only ever return empty and read as "nothing wrong". Assert what is actually
observable on a chassis instead:

**FAMILY CONSISTENCY IS NECESSARY BUT NOT SUFFICIENT -- both were learned the
hard way on vr1-dc0 (`docs/audit/geneve-over-v6-rootcause-20260808.md`).** A
family-only check PASSES on a v6 encap whose tunnel never instantiated. Two
distinct failures were measured, so the gate asserts BOTH:
- **C1 family split** (2026-08-08): the containerized control chassis (octavia
  LXD) took IPv4 data-tenant addresses while metal compute took IPv6 -> no
  cross-family tunnel forms -> 100% cross-node loss.
- **C2 bracketed encap** (2026-08-09): `ovn-chassis` 24.03 emits the v6
  `ovn-encap-ip` BRACKETED (`"[2602:...]"`), which OVS geneve REJECTS
  (`bad geneve 'remote_ip'`), leaving every v6 tunnel at `ofport -1`. C1 alone
  reads PASS while no traffic passes. Live-CONFIRMED fix: the encap delivered
  UNbracketed -> tunnel `ofport 14` -> real VM->VM 0% loss.

**PRIMARY GATE (executable):**
```bash
# from the DC rack (D-138). --expect-family v6 pins the ruled family (D-139 full-GUA);
# the check asserts C1 (single family + == v6) AND C2 (every geneve tunnel ofport >= 0).
bash scripts/geneve-encap-assert.sh --live "$DC_MODEL" --expect-family v6
```
PASS only if it prints `geneve-encap-assert: PASS` (exit 0). A `C1 ... SPLIT`
FAIL = the family split; a `C2 ... ofport -1` FAIL = the bracket bug (apply the
unbracket fix -- fixed charm revision, or per-chassis
`sudo ovs-vsctl set open_vswitch . external_ids:ovn-encap-ip=<unbracketed v6>` --
then re-run). Note the encap prefix is **GUA** `2602:f3e2:f0X:30::/64` (D-139
retired the ULA); a check citing "data-tenant ULA" is stale.

**MANUAL back-up probe** (if the script cannot run): `ovn-chassis` is a
SUBORDINATE (to nova-compute -- `bundle.yaml:450,453`); name a unit on a COMPUTE
host, do not assume `/leader`. Check the Encap.ip family AND the tunnel ofport:
```bash
juju status -m "$DC_MODEL" ovn-chassis
juju exec -m "$DC_MODEL" --unit ovn-central/leader -- \
  'sudo ovn-sbctl --columns=chassis_name,ip list encap' </dev/null    # C1: all one family, v6
juju exec -m "$DC_MODEL" --unit ovn-chassis/0 -- \
  'sudo ovs-vsctl --columns=name,ofport,error find interface type=geneve' </dev/null  # C2: ofport>=0, error []
```
Repeat C2 per chassis unit. No encap entry / a command error -> **REFUSE, do not
pass** ("could not look" is never "nothing there"); an `ofport -1`/error is a
FAIL, not a skip; confirm Step 6 was APPLIED, else record BLOCKED (never a pass).

### 12.3 -- The four VERIFY-LIVE gates

Each asserts a RULED value that no offline gate can reach. Sources:
`docs/CURRENT-STATE.md:247-252` (the VERIFY-LIVE list) and
`docs/design-decisions.md:824` (the keystone `PO:` mechanism).

**(a) hacluster `cluster_count` -- and WHEN each value is correct.**
```bash
juju config -m "$DC_MODEL" <app>-hacluster cluster_count
```
**GATE (RECONCILED 2026-08-06 to BUNDLEFIX-053; the old two-phase model is RETIRED):**
the base bundle now carries `num_units: 3` + `cluster_count: 3` for the HA apps
DIRECTLY. The former "base deploys at `cluster_count: 1`, then apply
`dc-ha-scaleup.yaml` to reach `3`" sequence is gone -- that overlay was retired
2026-08-06 (its content folded into `bundle.yaml`; see the D-121 R6 supersession
note). So after the Step 4 deploy of base + the per-DC VIP overlay,
`cluster_count: 3` is EXPECTED AND CORRECT for each of the 12 hacluster subs. A
`cluster_count` reading `1` behind a 3-unit principal is the FAILURE -- pacemaker
bootstraps a VIP off a single node (decorative HA) and every offline checker stays
green, which is exactly why `provider-bundle-check.py`'s arity gate (cases
T32-T34) asserts `cluster_count == num_units` on the merged deploy input and
catches the mismatch pre-deploy.

**(b) Ceph pool redundancy.**
```bash
juju exec -m "$DC_MODEL" --unit ceph-mon/leader -- 'ceph osd pool ls detail' </dev/null
juju exec -m "$DC_MODEL" --unit ceph-mon/leader -- 'ceph osd crush rule dump' </dev/null
```
**GATE:** `size=3 min_size=2` on the OpenStack pools, and failure-domain `host`
(the RULED target -- `docs/CURRENT-STATE.md:248`, and size=3/min_size=2 is
ADOPTED per `runbooks/dc-dc-phase0-vcloud-prep.md:223`). A `size=2` fallback is
permitted ONLY as an explicit, logged decision taken when real capacity is
measured -- never as something you find already set and wave through. Any other
failure-domain REFUSES: record it and stop rather than assuming a synonym.

**(c) keystone policy override -- RULED 2026-07-25.**
```bash
juju status -m "$DC_MODEL" keystone
```
**GATE:** `PO:` appears on EVERY keystone unit. `PO (broken):` on ANY unit is a
FAIL, and it is not cosmetic: a broken override is ATOMIC -- the whole policy is
discarded, silently reverting every tenant domain-manager to a plain user
(protects D-051/D-064). One good unit does not cover for a broken one; check the
whole column, and re-check after any scale-up.

**(d) machines-block overlay merge.** Already gated at Step 4.2 -- the dry-run
tag assertion. Do not author a second copy here; cite Step 4.2's captured output
in the exit-gate record.

The v6 expectations in 12.1/12.2 depend on Step 6's overlay actually having been
applied. If Step 6 is still open, this gate CANNOT pass yet -- record "blocked on
Step 6" as the honest answer rather than forcing a pass.

**GATE (Stage 5 exit condition, `docs/dc-dc-deployment-workflow.md`):**
`DC=<this DC> preflight.sh` PASS before add-model (Step 3); the Step 4.2
dry-run captured and its machines-block tags confirmed; post-deploy
`cloud-assert.sh --capture` per DC (Step 10); controller backup taken + stored,
gated on the FILE (Step 11); the four VERIFY-LIVE gates plus Ceph-over-v6 and
geneve-over-v6 recorded (Step 12, the v6 halves contingent on Step 6).

-> Repeat this entire runbook for the other DC. **There is no ruled DC ordering
for this deployment (GA-R5, 2026-07-27): both DCs deploy from the Office1
headend by the same procedure, in whichever order the operator chooses.** ->
proceed to Stage 6 (DR wiring and failover drill) once BOTH DCs pass this gate.

---

## Delivery checklist

- [ ] `bash scripts/repo-lint.sh` clean (0 fail) before committing any repo
      changes made while executing this runbook (the IPv6 overlay file, any
      re-rendered per-DC VIP overlay -- NOT a `bundle.yaml` edit, which carries
      no VIPs since ruling 3 -- any DC-qualified `asbuilt/` path fix).
- [ ] The Step 4.2 dry-run output CAPTURED to a dated file and cited in the
      commit that closes this stage for that DC (plan counts are quoted only
      from captured output, GA-R1 rule 2).
- [ ] Changelog entry per real repo change this runbook produces (the
      overlay, any bundle edit, any script fix) -- next DOCFIX number via
      `bash scripts/ledger-scan.sh`, same as every other delivery this
      session.
- [ ] `docs/session-ledger.md` updated with the outcome, per DC.
- [ ] `docs/dc-dc-deployment-workflow.md` Stage 5 row updated once BOTH DCs
      have actually passed Step 12's gate (not just the first one done).
- [ ] Log the Step 6 charm-config research (v6 SAN support, v6 addressing
      option names) as its own follow-up item if not resolved inline during
      this stage's real execution -- do not leave it silently unresolved in
      the workflow doc's tooling gap register.
