# DC-DC Phase 4 -- Juju controller + OpenStack bundle, per DC (Stage 5)

Stand up a running, independent OpenStack cloud in DC1, then repeat the same
loop for DC2. This is the ground-truth/copy-point stage: `docs/dc-dc-buildout-
design.md` line 12 states this repo was seeded from `openstack-caracal-ipv4`
at commit 8813efc specifically so `bundle.yaml` and `runbooks/phase-01..08-
*.md` could be reused here almost unchanged. **This runbook does not
duplicate those eight runbooks' command-level content.** It describes what
changes to run them TWICE, independently, in the VR1 two-DC dual-stack
context, and calls out the real adaptation work plainly rather than
asserting "just re-run it" glosses over anything.

**Governing docs:** `docs/dc-dc-buildout-design.md` Section 4 "Phase 4" entry
and Section 2 (address-family table); `docs/design-decisions.md` D-104
(controller topology), D-109 (per-DC independent Vault roots + IPv6-SAN
certs), D-101 (address plan / family matrix); `docs/dc-dc-deployment-
workflow.md` Stage 5 row (**"This is where the existing repo tooling
(preflight.sh, cloud-assert.sh, lib-net.sh/lib-hosts.sh, run-logged.sh) does
the most direct work"**) and tooling gap register item #1 (the `$DC`
selector convention, CLOSED 2026-07-09 DOCFIX-151 -- this is the stage that
actually calls it).

**Prerequisites (must be true entering this stage, per DC):** Stage 4
(`runbooks/dc-dc-phase3-maas-enlist-deploy.md`) done for that DC -- nodes
**READY** (NOT Deployed: MAAS-deploy is SKIPPED per the 2026-07-23 READY-handoff
ruling / DOCFIX-200 -- THIS runbook is what provisions them, and expecting
Deployed nodes here is exactly the breakage that ruling prevented); six planes
present with correct fabrics/VLANs; provider NIC raw; PXE (v4) working; that
DC's own artifact source reachable on its own address (dc0 = the D-135 item-1
mirror, dc1 = the D-135-AMENDED apt caching proxy consumed via
`juju apt-http-proxy` -- the two DCs differ BY RULING, so do not assume a
mirror at dc1); `scripts/lib-hosts.sh`'s per-DC host data populated (that
stage's own follow-up delivery) so `lib_hosts_select_dc "$dc"` no longer
fails loud for it.

**THIS STAGE OWES GATE G17 (inherited from Stage 4, split out 2026-07-27 by
operator ruling -- see the G17 row in `docs/CURRENT-STATE.md`, the authority).**
Stage 4 could not verify that a NODE reaches its DC's artifact source, or that a
node's time source is the MAAS hierarchy, because under the READY-handoff ruling
its nodes never boot an OS. **First boot in this stage is the first moment either
is observable, so capture both then** -- per DC, from a booted node:

- dc0: `curl -sI http://10.12.8.4/` returns 200.
- dc1: apt fetches through the proxy at `10.12.68.4:3142` (NOT a mirror -- D-135
  amendment).
- both: `chronyc sources` shows the MAAS-served time source, NOT the DC edge
  (D-129(iv) removed the edge from that path).

Capture the output to a dated file and cite it in the commit that closes G17. If
first boot happens without these captures being taken, the observation window has
passed and G17 needs a deliberate rescue-boot to answer -- so take them the first
time.

!!! Every command block in the referenced `phase-01..08` runbooks that reads
    `scripts/lib-net.sh` or `scripts/lib-hosts.sh` values (VIP bands, host
    octets, `KEYSTONE_VIP_DEFAULT`, etc.) MUST be preceded, in THIS stage's
    session, by:
    ```bash
    source scripts/lib-net.sh;   lib_net_select_dc   "$DC"   # $DC = dc1 or dc2
    source scripts/lib-hosts.sh; lib_hosts_select_dc "$DC"
    ```
    Do not skip this because it "worked before" on DC0/DC1 -- for DC1 both
    calls are a documented no-op (D-101: DC1 inherits DC0 unchanged), so
    skipping them happens to produce the same values by accident; for DC2 it
    is NOT a no-op and skipping it silently reuses DC1's values, exactly the
    bug DOCFIX-151 exists to prevent. Always call both, for both DCs,
    unconditionally.

---

## What is genuinely IDENTICAL to VR0/DC0 for DC1, and what is not

**DC1 needs NO VIP/CIDR changes at all.** D-101: DC1 inherits DC0's six-plane
v4 layout UNCHANGED. `bundle.yaml`'s existing VIP values (`.50-.60` band,
per phase-01's own text) are ALREADY correct for DC1 -- deploy the SAME
`bundle.yaml` this repo already carries, in DC1's own separate Juju
controller/model, with no v4-address edits. `phase-01..08`'s command blocks
that reference `lib-net.sh`/`lib-hosts.sh` values will resolve to the exact
same numbers for DC1 as they did for the original DC0 rehearsal -- this is
expected, not a bug (see the `$DC` selector callout above).

**DC2 needs a distinct, non-overlapping v4 supernet + all-new VIPs -- NOT
YET ASSIGNED.** Per D-101 and tooling gap register item #3: DC2's supernet,
the org ULA /48, and the per-DC GUA carve are NetBox-authoritative and
still genuinely unassigned (the import PIPELINE closed 2026-07-09, DOCFIX-
152, but the literals themselves have not been generated/assigned). **DC2's
bundle deploy is BLOCKED on this** -- you cannot pick DC2's VIP band, the
octavia SAN IP (per phase-01's own `1.0-GEN.c` step, which derives the SAN
IP FROM the bundle's real VIP value, not a literal), or any other DC2
address until NetBox has real values. Do not invent a "DC2 test range" to
get moving faster -- that is exactly the inference this repo's hard rule
forbids. If DC2's literals still don't exist when you reach this stage,
STOP here for DC2 and only proceed with DC1.

---

## Sequence (repeat entire sequence once per DC, DC1 first)

```
1.  Confirm Stage 4 gate + call the $DC selectors        (read-only)
2.  Bootstrap a DEDICATED single-unit Juju controller, IN-DC  [MUTATION, gated]
3.  preflight.sh PASS                                     (read-only gate)
4.  Run phase-01 (bundle deploy) against THIS DC's model  [adapted, gated -- see below]
5.  Run phase-02 (vault bring-up) -- INDEPENDENT root per DC [adapted, SECRET-HANDLING]
6.  Write + apply the IPv6 family-matrix overlay          [DRAFTED 2026-07-10, NOT YET APPLIED -- see below]
7.  Run phase-03 (core verify)                            [adapted, gated]
8.  Run phase-04 (network carve)                          [adapted, gated]
9.  Run phase-05 (octavia enablement)                     [adapted, gated]
10. cloud-assert.sh --capture                             (read-only + committed BOM)
11. Controller backup: juju create-backup / download-backup [MUTATION, gated]
12. Ceph-over-v6 + geneve-over-v6 verification              (read-only gate)
    -> EXIT GATE (this DC) -> repeat 1-12 for the other DC -> Stage 6
```

---

## Step 1 -- Confirm Stage 4 gate + call the $DC selectors (READ-ONLY)

**CHECK -- jumphost / wherever this session runs juju against this DC**
```bash
export DC=vr1-dc0   # or dc2 -- explicit, per this repo's $DC convention (never omitted)
source scripts/lib-net.sh;   lib_net_select_dc   "$DC"
source scripts/lib-hosts.sh; lib_hosts_select_dc "$DC"
echo "PLANE_CIDRS: ${PLANE_CIDRS[*]}"
echo "HOSTS: ${HOSTS[*]}"
```
**GATE:** both selector calls exit 0 (for DC2, this REQUIRES Stage 4 having
already populated real host data into `scripts/lib-hosts.sh` for dc2 -- if
`lib_hosts_select_dc vr1-dc1` still fails loud here, Stage 4 for DC2 is not
actually done yet, stop and go back).

---

## Step 2 -- Bootstrap a dedicated, in-DC, single-unit Juju controller [MUTATION, gated]

Per D-104: ONE Juju controller PER DC, deployed IN that DC (on a MAAS-
deployed machine in that DC, from Stage 4), single-unit (not HA) for VR1,
managing ONLY its own cloud. This is explicitly NOT the same controller
model as VR0/DC0's original bootstrap -- a fresh `juju bootstrap` targeting
THIS DC's MAAS cloud/region, not a reused controller.

**Step 2.0 -- Per-DC MAAS service credential [DEPLOYMENT-TIME TASK; MUST precede bootstrap]**

Juju authenticates to the MAAS region with an API key -- this is NOT created by
any prior stage, so it is a standing deployment-time task per DC. Per the per-DC
credential-isolation discipline (SEC-012/-016; Roosevelt analog: per-DC cloud
credentials), issue a DEDICATED MAAS admin service user + its OWN key per DC --
never reuse one key across DCs. On the MAAS region host (VR1: `voffice1`):
```bash
# service account -- Juju auths via the API key, not this password; use a random one
sudo maas createadmin --username=juju-<dc> --email=juju-<dc>@<realm>
```
Generate the key so it lands ONLY in the JUMPHOST per-site creds folder
(`~/vr1-<dc>-creds/maas-api-key.txt` on vcloud, SEC-009 convention -- NOT a new
location on the region), never echoed to a terminal (a printed MAAS key is
burned, SEC-006). Run the generate FROM the jumphost so the key transits ssh
stdout straight into the vcloud file (no persistent copy on the region):
```bash
ssh <region> 'sudo maas apikey --username=juju-<dc> --generate' \
  > ~/vr1-<dc>-creds/maas-api-key.txt
chmod 600 ~/vr1-<dc>-creds/maas-api-key.txt
awk -F: '{print "parts:", NF}' ~/vr1-<dc>-creds/maas-api-key.txt   # expect 3; do NOT cat
```
Register the MAAS cloud + this credential in Juju consuming the key BY PATH/stdin
(never argv). Each key is a rotation obligation -- open a per-DC SEC row when issued.

**CHECK (read-only) -- confirm the target MAAS cloud is registered to Juju**
```bash
juju clouds --client | grep -i maas
```

**MUTATION**
```bash
juju bootstrap <this-dc-maas-cloud-name> <controller-name-per-DC>
```
The exact MAAS cloud name and the controller name are session-specific
identifiers you choose/confirm at execution time (e.g. reflecting `dc1`/
`dc2` in the controller name for operator clarity) -- not invented here.

Per D-104's rationale: this keeps DR-independence honest (a controller loss
affects only its own DC) and is a deliberate VR1 test-only economy (HA
deferred to Roosevelt) -- do not "upgrade" this to HA mid-stream without a
new D-NNN, that would silently reverse an ADOPTED decision.

---

## Step 3 -- `preflight.sh` PASS (read-only gate)

```bash
bash scripts/preflight.sh
```
Confirmed repo-agnostic as of the 2026-07-09 sweep (`docs/dc-dc-deployment-
workflow.md` Stage 5 row) -- runs unmodified against either DC's clone/
session. Expect `PREFLIGHT: PASS`.

---

## Step 4 -- Run phase-01 (bundle deploy) against this DC's model [adapted, gated]

Follow `runbooks/phase-01-bundle-deploy.md` verbatim, with these DC-DC-
specific notes, not a rewrite:

- **DC1:** deploy the EXISTING `bundle.yaml` with NO VIP edits (see the
  "identical to DC0" section above). The Octavia PKI overlay (Step 1.0)
  can reuse the existing `overlays/octavia-pki.yaml` if its CAs are still
  valid (10y validity, per that runbook), OR regenerate fresh per-DC PKI via
  Step 1.0-GEN if independent per-DC Octavia trust domains are desired --
  this is a call the operator should make explicitly (log it as a decision
  note either way), not silently default to reuse.
- **DC2:** BLOCKED until its VIP band exists (see above). Once NetBox
  assigns DC2's real supernet and it's imported via `netbox/dc-dc-prefixes-
  import.py --dc vr1-dc1` (DOCFIX-152), `bundle.yaml`'s VIP lines need editing
  to DC2's real `.50-.60`-equivalent band within DC2's own provider-public
  plane -- a genuine, reviewed bundle edit, not a copy-paste of DC1's values.
- Both DCs: `bundle.yaml` explicitly ships "NO designate" per phase-01's own
  gate text (D-019) -- this is UNCHANGED here; Designate reactivation is
  Stage 7's job (D-106), do not reactivate it mid-Stage-5.
- Run phase-01's own EXIT GATE checks exactly as written, against THIS DC's
  model.

---

## Step 5 -- Run phase-02 (vault bring-up) -- INDEPENDENT root per DC [adapted, SECRET-HANDLING]

Follow `runbooks/phase-02-vault-bringup.md` verbatim per DC. Per D-109: each
DC's Vault is its OWN independent root CA -- no regional root-of-trust this
phase. This means:
- The `vault operator init` one-shot step (Step 2.1) runs ONCE PER DC,
  producing DISTINCT unseal-key shares and a DISTINCT root token for DC1 vs.
  DC2. Store them separately, clearly labeled by DC (this repo's existing
  secrets discipline -- `~/vault-init/`-style off-host capture -- applies
  identically per DC, just not shared between them).
- D-109 also requires each DC's Vault to issue certs with BOTH IPv4 and
  IPv6 (ULA) SANs for the dual-stacked metal-internal API endpoints. **This
  is a real charm-behavior question not yet confirmed this session:**
  whether the `vault`/`vault-pki`-relation charms used in `bundle.yaml`
  actually support requesting/issuing IPv6 SANs today (vs. only IPv4) is
  UNVERIFIED -- check the actual charm config/relation-data schema (e.g.
  `openstack-loadbalancer`/certificates interface docs, or the vault charm's
  own config options) before assuming this "just works" once IPv6 addresses
  exist on the plane. If it does not support v6 SANs natively, that is a
  real finding to log (a D-NNN amendment or a new decision), not something
  to route around silently.

---

## Step 6 -- Write + apply the IPv6 family-matrix overlay [OVERLAY DRAFTED 2026-07-10, NOT YET APPLIED]

D-101's family matrix (Section 2 of the buildout design) requires, per DC:
- `data-tenant`, `storage`, `replication`: IPv6-ONLY ULA (no v4 config at
  all on these planes for the relevant charms -- e.g. `ceph-mon`'s
  `public-network`/`cluster-network`, OVN's geneve encapsulation address).
- `metal-internal`, `metal-admin`: DUAL-STACK (v4 retained + a ULA leg
  added -- the metal-admin ULA leg is the 2026-07-09 operator AMENDMENT to
  D-101, a reversal of that decision's own original lean, so do not assume
  older documentation/training-data patterns for "metal-admin stays v4-only"
  -- this repo's ADOPTED text says otherwise).
- `provider-public`: DUAL-STACK v4 + GUA (native, no NAT66 -- D-101's
  tenant/MTU sub-policy folded in from D-102).

**`overlays/dc-dc-ipv6-family-matrix.yaml` now exists**, drafted 2026-07-10
against REAL charm source/config (not guessed option names) -- see
`docs/archive/dc-dc-ipv6-charm-research.md` for full sourcing and per-charm
confidence level. Summary: `prefer-ipv6` (confirmed real option, additive
dual-stack HAProxy binding, confirmed via `charm-nova-cloud-controller`'s
actual "Dual Stack VIPs" commit) + appended v6/GUA entries on the existing
`vip:` option, for the nine dual-stack API charms directly confirmed or
sharing the same charms.openstack framework; `ceph-mon`'s own separate
`prefer-ipv6` + `ceph-public-network`/`ceph-cluster-network` (a clean
switch-to-v6-only, correct for the ULA-only planes); NO entry for OVN
(confirmed no such charm-config option exists -- geneve family follows the
bound interface automatically once the plane itself is ULA-only).

**Apply via** (stacked overlays, same pattern as the existing octavia-pki
overlay):
```bash
juju deploy ./bundle.yaml \
  --overlay overlays/octavia-pki.yaml \
  --overlay "overlays/${DC}-hostnames.yaml" \
  --overlay overlays/dc-dc-ipv6-family-matrix.yaml \
  -m "${DC_MODEL}" --dry-run
```
Review the diff before applying for real, same discipline as every other
overlay in this repo.

**Still genuinely open, not resolved by the overlay:**
1. The 8 charms whose `prefer-ipv6` presence is INFERRED-BY-PATTERN (not
   individually fetched from their own `config.yaml`) need live `juju
   config <app>` confirmation once first deployed.
2. **Octavia's `lb-mgmt-net` IPv6 support is a real, open risk, not
   resolved.** The overlay deliberately excludes an Octavia entry --
   `docs/archive/dc-dc-ipv6-charm-research.md` section 6 documents two real,
   still-referenced upstream Launchpad bugs (#1911788, #1913409) about
   IPv6 lb-mgmt-net failures. Decide explicitly before this stage's
   Ceph-over-v6/geneve-over-v6 gate is declared closed: accept the risk
   and test it for real once DC1 exists, or keep Octavia's lb-mgmt-net
   v4-only as a deliberate, logged D-101 exception.
3. UNVALIDATED against any live model -- this overlay has not been applied
   or tested, same posture as every other artifact from tonight's
   prep-only session.

---

## Steps 7-9 -- phase-03 / phase-04 / phase-05, adapted per DC

Follow `runbooks/phase-03-core-verify.md`, `runbooks/phase-04-network-
carve.md`, `runbooks/phase-05-octavia-enablement.md` verbatim per DC, with
the same `$DC` selector discipline as Step 1 applied to every command block
in them that reads `lib-net.sh`/`lib-hosts.sh` values. No other adaptation
is currently known to be needed in these three -- if executing them surfaces
a DC0-specific literal baked into a check block that these runbooks'
authors didn't anticipate needing DC-parameterization, LOG it (a DOCFIX
candidate) rather than silently patching around it live.

---

## Step 10 -- `cloud-assert.sh --capture` (read-only + committed BOM)

```bash
bash scripts/cloud-assert.sh --capture
```
Per DC, at real deploy-completion. Confirmed repo-agnostic (2026-07-09
sweep). Commit the resulting `asbuilt/` BOM per this repo's existing
deploy-loop convention, labeled clearly by DC (do not let DC1's and DC2's
BOMs collide/overwrite -- confirm the script's own output path is already
DC-distinguishing, or add a DC-qualified path if not, before running for
the second DC).

---

## Step 11 -- Controller backup [MUTATION, gated]

```bash
juju create-backup -m <this-dc-controller-name>
juju download-backup <backup-id> -m <this-dc-controller-name>
```
Per D-104: scheduled controller backups, confirmed backup posture (juju 3.6
`create-backup`/`download-backup`). Store the downloaded backup off-host,
labeled by DC, same secrets-adjacent handling discipline as Vault's init
capture. The RESTORE half of this drill is exercised in Stage 6 (DR wiring
and failover drill) -- this step only takes and stores the backup, it does
not restore it.

---

## Step 12 -- Ceph-over-v6 + geneve-over-v6 verification (READ-ONLY gate)

```bash
# Ceph binds v6 on the storage/replication ULA planes (ms_bind_ipv6, D-101):
juju run -m <this-dc-model> ceph-mon/leader 'ceph -s' 2>&1 | grep -i "v6\|v2"
# OVN geneve encapsulation over the data-tenant ULA plane:
juju config -m <this-dc-model> ovn-central | grep -i encap
```
The exact expected values here depend on Step 6's overlay actually having
been written and applied -- if Step 6 is still open (charm options
unconfirmed), this gate CANNOT pass yet; do not force a "looks close enough"
pass. Record whichever real state you observe, including "blocked on Step
6" if that's the honest answer.

**GATE (Stage 5 exit condition, `docs/dc-dc-deployment-workflow.md`):**
`preflight.sh` PASS before add-model (Step 3); post-deploy `cloud-assert.sh
--capture` per DC (Step 10); controller backup taken + stored (Step 11);
Ceph-over-v6 and geneve-over-v6 verified (Step 12, contingent on Step 6).

-> Repeat this entire runbook for the other DC (DC1 first is recommended,
since only DC1 is unblocked by real address literals today) -> proceed to
Stage 6 (DR wiring and failover drill) once BOTH DCs pass this gate.

---

## Delivery checklist

- [ ] `bash scripts/repo-lint.sh` clean (0 fail) before committing any repo
      changes made while executing this runbook (the IPv6 overlay file, any
      `bundle.yaml` VIP edit for DC2, any DC-qualified `asbuilt/` path fix).
- [ ] Changelog entry per real repo change this runbook produces (the
      overlay, any bundle edit, any script fix) -- next DOCFIX number via
      `bash scripts/ledger-scan.sh`, same as every other delivery this
      session.
- [ ] `docs/session-ledger.md` updated with the outcome, per DC.
- [ ] `docs/dc-dc-deployment-workflow.md` Stage 5 row updated once BOTH DCs
      have actually passed Step 12's gate (not just DC1).
- [ ] Log the Step 6 charm-config research (v6 SAN support, v6 addressing
      option names) as its own follow-up item if not resolved inline during
      this stage's real execution -- do not leave it silently unresolved in
      the workflow doc's tooling gap register.
