Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / runbooks / dc-dc-phase4-juju-bundle-per-dc.md
@JANeumatrix JANeumatrix 1 hour ago 16 KB DOCFIX-171: full-project sweep -- DC-DC runbook consistency fixes
Raw Blame History

DC-DC Phase 4 -- Juju controller + OpenStack bundle, per DC (Stage 5)

Stand up a running, independent OpenStack cloud in DC1, then repeat the same loop for DC2. This is the ground-truth/copy-point stage: docs/dc-dc-buildout- design.md line 12 states this repo was seeded from openstack-caracal-ipv4 at commit 8813efc specifically so bundle.yaml and runbooks/phase-01..08- *.md could be reused here almost unchanged. This runbook does not duplicate those eight runbooks' command-level content. It describes what changes to run them TWICE, independently, in the VR1 two-DC dual-stack context, and calls out the real adaptation work plainly rather than asserting "just re-run it" glosses over anything.

Governing docs: docs/dc-dc-buildout-design.md Section 4 "Phase 4" entry and Section 2 (address-family table); docs/design-decisions.md D-104 (controller topology), D-109 (per-DC independent Vault roots + IPv6-SAN certs), D-101 (address plan / family matrix); docs/dc-dc-deployment- workflow.md Stage 5 row ("This is where the existing repo tooling (preflight.sh, cloud-assert.sh, lib-net.sh/lib-hosts.sh, run-logged.sh) does the most direct work") and tooling gap register item #1 (the $DC selector convention, CLOSED 2026-07-09 DOCFIX-151 -- this is the stage that actually calls it).

Prerequisites (must be true entering this stage, per DC): Stage 4 (runbooks/dc-dc-phase3-maas-enlist-deploy.md) done for that DC -- nodes Deployed; six planes present with correct fabrics/VLANs; provider NIC raw; PXE (v4) working; that DC's own artifact mirror reachable; NTP from that DC's edge working; scripts/lib-hosts.sh's per-DC host data populated (that stage's own follow-up delivery) so lib_hosts_select_dc "$dc" no longer fails loud for it.

!!! Every command block in the referenced phase-01..08 runbooks that reads scripts/lib-net.sh or scripts/lib-hosts.sh values (VIP bands, host octets, KEYSTONE_VIP_DEFAULT, etc.) MUST be preceded, in THIS stage's session, by:

```bash
source scripts/lib-net.sh;   lib_net_select_dc   "$DC"   # $DC = dc1 or dc2
source scripts/lib-hosts.sh; lib_hosts_select_dc "$DC"
```
Do not skip this because it "worked before" on DC0/DC1 -- for DC1 both
calls are a documented no-op (D-101: DC1 inherits DC0 unchanged), so
skipping them happens to produce the same values by accident; for DC2 it
is NOT a no-op and skipping it silently reuses DC1's values, exactly the
bug DOCFIX-151 exists to prevent. Always call both, for both DCs,
unconditionally.

What is genuinely IDENTICAL to VR0/DC0 for DC1, and what is not

DC1 needs NO VIP/CIDR changes at all. D-101: DC1 inherits DC0's six-plane v4 layout UNCHANGED. bundle.yaml's existing VIP values (.50-.60 band, per phase-01's own text) are ALREADY correct for DC1 -- deploy the SAME bundle.yaml this repo already carries, in DC1's own separate Juju controller/model, with no v4-address edits. phase-01..08's command blocks that reference lib-net.sh/lib-hosts.sh values will resolve to the exact same numbers for DC1 as they did for the original DC0 rehearsal -- this is expected, not a bug (see the $DC selector callout above).

DC2 needs a distinct, non-overlapping v4 supernet + all-new VIPs -- NOT YET ASSIGNED. Per D-101 and tooling gap register item #3: DC2's supernet, the org ULA /48, and the per-DC GUA carve are NetBox-authoritative and still genuinely unassigned (the import PIPELINE closed 2026-07-09, DOCFIX- 152, but the literals themselves have not been generated/assigned). DC2's bundle deploy is BLOCKED on this -- you cannot pick DC2's VIP band, the octavia SAN IP (per phase-01's own 1.0-GEN.c step, which derives the SAN IP FROM the bundle's real VIP value, not a literal), or any other DC2 address until NetBox has real values. Do not invent a "DC2 test range" to get moving faster -- that is exactly the inference this repo's hard rule forbids. If DC2's literals still don't exist when you reach this stage, STOP here for DC2 and only proceed with DC1.

Sequence (repeat entire sequence once per DC, DC1 first)

1.  Confirm Stage 4 gate + call the $DC selectors        (read-only)
2.  Bootstrap a DEDICATED single-unit Juju controller, IN-DC  [MUTATION, gated]
3.  preflight.sh PASS                                     (read-only gate)
4.  Run phase-01 (bundle deploy) against THIS DC's model  [adapted, gated -- see below]
5.  Run phase-02 (vault bring-up) -- INDEPENDENT root per DC [adapted, SECRET-HANDLING]
6.  Write + apply the IPv6 family-matrix overlay          [DRAFTED 2026-07-10, NOT YET APPLIED -- see below]
7.  Run phase-03 (core verify)                            [adapted, gated]
8.  Run phase-04 (network carve)                          [adapted, gated]
9.  Run phase-05 (octavia enablement)                     [adapted, gated]
10. cloud-assert.sh --capture                             (read-only + committed BOM)
11. Controller backup: juju create-backup / download-backup [MUTATION, gated]
12. Ceph-over-v6 + geneve-over-v6 verification              (read-only gate)
    -> EXIT GATE (this DC) -> repeat 1-12 for the other DC -> Stage 6

Step 1 -- Confirm Stage 4 gate + call the $DC selectors (READ-ONLY)

CHECK -- jumphost / wherever this session runs juju against this DC

export DC=dc1   # or dc2 -- explicit, per this repo's $DC convention (never omitted)
source scripts/lib-net.sh;   lib_net_select_dc   "$DC"
source scripts/lib-hosts.sh; lib_hosts_select_dc "$DC"
echo "PLANE_CIDRS: ${PLANE_CIDRS[*]}"
echo "HOSTS: ${HOSTS[*]}"

GATE: both selector calls exit 0 (for DC2, this REQUIRES Stage 4 having already populated real host data into scripts/lib-hosts.sh for dc2 -- if lib_hosts_select_dc dc2 still fails loud here, Stage 4 for DC2 is not actually done yet, stop and go back).

Step 2 -- Bootstrap a dedicated, in-DC, single-unit Juju controller [MUTATION, gated]

Per D-104: ONE Juju controller PER DC, deployed IN that DC (on a MAAS- deployed machine in that DC, from Stage 4), single-unit (not HA) for VR1, managing ONLY its own cloud. This is explicitly NOT the same controller model as VR0/DC0's original bootstrap -- a fresh juju bootstrap targeting THIS DC's MAAS cloud/region, not a reused controller.

CHECK (read-only) -- confirm the target MAAS cloud is registered to Juju

juju clouds --client | grep -i maas

MUTATION

juju bootstrap <this-dc-maas-cloud-name> <controller-name-per-DC>

The exact MAAS cloud name and the controller name are session-specific identifiers you choose/confirm at execution time (e.g. reflecting dc1/ dc2 in the controller name for operator clarity) -- not invented here.

Per D-104's rationale: this keeps DR-independence honest (a controller loss affects only its own DC) and is a deliberate VR1 test-only economy (HA deferred to Roosevelt) -- do not "upgrade" this to HA mid-stream without a new D-NNN, that would silently reverse an ADOPTED decision.

Step 3 -- preflight.sh PASS (read-only gate)

bash scripts/preflight.sh

Confirmed repo-agnostic as of the 2026-07-09 sweep (docs/dc-dc-deployment- workflow.md Stage 5 row) -- runs unmodified against either DC's clone/ session. Expect PREFLIGHT: PASS.

Step 4 -- Run phase-01 (bundle deploy) against this DC's model [adapted, gated]

Follow runbooks/phase-01-bundle-deploy.md verbatim, with these DC-DC- specific notes, not a rewrite:

  • DC1: deploy the EXISTING bundle.yaml with NO VIP edits (see the "identical to DC0" section above). The Octavia PKI overlay (Step 1.0) can reuse the existing overlays/octavia-pki.yaml if its CAs are still valid (10y validity, per that runbook), OR regenerate fresh per-DC PKI via Step 1.0-GEN if independent per-DC Octavia trust domains are desired -- this is a call the operator should make explicitly (log it as a decision note either way), not silently default to reuse.
  • DC2: BLOCKED until its VIP band exists (see above). Once NetBox assigns DC2's real supernet and it's imported via netbox/dc-dc-prefixes- import.py --dc dc2 (DOCFIX-152), bundle.yaml's VIP lines need editing to DC2's real .50-.60-equivalent band within DC2's own provider-public plane -- a genuine, reviewed bundle edit, not a copy-paste of DC1's values.
  • Both DCs: bundle.yaml explicitly ships "NO designate" per phase-01's own gate text (D-019) -- this is UNCHANGED here; Designate reactivation is Stage 7's job (D-106), do not reactivate it mid-Stage-5.
  • Run phase-01's own EXIT GATE checks exactly as written, against THIS DC's model.

Step 5 -- Run phase-02 (vault bring-up) -- INDEPENDENT root per DC [adapted, SECRET-HANDLING]

Follow runbooks/phase-02-vault-bringup.md verbatim per DC. Per D-109: each DC's Vault is its OWN independent root CA -- no regional root-of-trust this phase. This means:

  • The vault operator init one-shot step (Step 2.1) runs ONCE PER DC, producing DISTINCT unseal-key shares and a DISTINCT root token for DC1 vs. DC2. Store them separately, clearly labeled by DC (this repo's existing secrets discipline -- ~/vault-init/-style off-host capture -- applies identically per DC, just not shared between them).
  • D-109 also requires each DC's Vault to issue certs with BOTH IPv4 and IPv6 (ULA) SANs for the dual-stacked metal-internal API endpoints. This is a real charm-behavior question not yet confirmed this session: whether the vault/vault-pki-relation charms used in bundle.yaml actually support requesting/issuing IPv6 SANs today (vs. only IPv4) is UNVERIFIED -- check the actual charm config/relation-data schema (e.g. openstack-loadbalancer/certificates interface docs, or the vault charm's own config options) before assuming this "just works" once IPv6 addresses exist on the plane. If it does not support v6 SANs natively, that is a real finding to log (a D-NNN amendment or a new decision), not something to route around silently.

Step 6 -- Write + apply the IPv6 family-matrix overlay [OVERLAY DRAFTED 2026-07-10, NOT YET APPLIED]

D-101's family matrix (Section 2 of the buildout design) requires, per DC:

  • data-tenant, storage, replication: IPv6-ONLY ULA (no v4 config at all on these planes for the relevant charms -- e.g. ceph-mon's public-network/cluster-network, OVN's geneve encapsulation address).
  • metal-internal, metal-admin: DUAL-STACK (v4 retained + a ULA leg added -- the metal-admin ULA leg is the 2026-07-09 operator AMENDMENT to D-101, a reversal of that decision's own original lean, so do not assume older documentation/training-data patterns for "metal-admin stays v4-only" -- this repo's ADOPTED text says otherwise).
  • provider-public: DUAL-STACK v4 + GUA (native, no NAT66 -- D-101's tenant/MTU sub-policy folded in from D-102).

overlays/dc-dc-ipv6-family-matrix.yaml now exists, drafted 2026-07-10 against REAL charm source/config (not guessed option names) -- see docs/dc-dc-ipv6-charm-research.md for full sourcing and per-charm confidence level. Summary: prefer-ipv6 (confirmed real option, additive dual-stack HAProxy binding, confirmed via charm-nova-cloud-controller's actual "Dual Stack VIPs" commit) + appended v6/GUA entries on the existing vip: option, for the nine dual-stack API charms directly confirmed or sharing the same charms.openstack framework; ceph-mon's own separate prefer-ipv6 + ceph-public-network/ceph-cluster-network (a clean switch-to-v6-only, correct for the ULA-only planes); NO entry for OVN (confirmed no such charm-config option exists -- geneve family follows the bound interface automatically once the plane itself is ULA-only).

Apply via (stacked overlays, same pattern as the existing octavia-pki overlay):

juju deploy ./bundle.yaml \
  --overlay overlays/octavia-pki.yaml \
  --overlay "overlays/${DC}-hostnames.yaml" \
  --overlay overlays/dc-dc-ipv6-family-matrix.yaml \
  -m "${DC_MODEL}" --dry-run

Review the diff before applying for real, same discipline as every other overlay in this repo.

Still genuinely open, not resolved by the overlay:

  1. The 8 charms whose prefer-ipv6 presence is INFERRED-BY-PATTERN (not individually fetched from their own config.yaml) need live juju config <app> confirmation once first deployed.
  2. Octavia's lb-mgmt-net IPv6 support is a real, open risk, not resolved. The overlay deliberately excludes an Octavia entry -- docs/dc-dc-ipv6-charm-research.md section 6 documents two real, still-referenced upstream Launchpad bugs (#1911788, #1913409) about IPv6 lb-mgmt-net failures. Decide explicitly before this stage's Ceph-over-v6/geneve-over-v6 gate is declared closed: accept the risk and test it for real once DC1 exists, or keep Octavia's lb-mgmt-net v4-only as a deliberate, logged D-101 exception.
  3. UNVALIDATED against any live model -- this overlay has not been applied or tested, same posture as every other artifact from tonight's prep-only session.

Steps 7-9 -- phase-03 / phase-04 / phase-05, adapted per DC

Follow runbooks/phase-03-core-verify.md, runbooks/phase-04-network- carve.md, runbooks/phase-05-octavia-enablement.md verbatim per DC, with the same $DC selector discipline as Step 1 applied to every command block in them that reads lib-net.sh/lib-hosts.sh values. No other adaptation is currently known to be needed in these three -- if executing them surfaces a DC0-specific literal baked into a check block that these runbooks' authors didn't anticipate needing DC-parameterization, LOG it (a DOCFIX candidate) rather than silently patching around it live.

Step 10 -- cloud-assert.sh --capture (read-only + committed BOM)

bash scripts/cloud-assert.sh --capture

Per DC, at real deploy-completion. Confirmed repo-agnostic (2026-07-09 sweep). Commit the resulting asbuilt/ BOM per this repo's existing deploy-loop convention, labeled clearly by DC (do not let DC1's and DC2's BOMs collide/overwrite -- confirm the script's own output path is already DC-distinguishing, or add a DC-qualified path if not, before running for the second DC).

Step 11 -- Controller backup [MUTATION, gated]

juju create-backup -m <this-dc-controller-name>
juju download-backup <backup-id> -m <this-dc-controller-name>

Per D-104: scheduled controller backups, confirmed backup posture (juju 3.6 create-backup/download-backup). Store the downloaded backup off-host, labeled by DC, same secrets-adjacent handling discipline as Vault's init capture. The RESTORE half of this drill is exercised in Stage 6 (DR wiring and failover drill) -- this step only takes and stores the backup, it does not restore it.

Step 12 -- Ceph-over-v6 + geneve-over-v6 verification (READ-ONLY gate)

# Ceph binds v6 on the storage/replication ULA planes (ms_bind_ipv6, D-101):
juju run -m <this-dc-model> ceph-mon/leader 'ceph -s' 2>&1 | grep -i "v6\|v2"
# OVN geneve encapsulation over the data-tenant ULA plane:
juju config -m <this-dc-model> ovn-central | grep -i encap

The exact expected values here depend on Step 6's overlay actually having been written and applied -- if Step 6 is still open (charm options unconfirmed), this gate CANNOT pass yet; do not force a "looks close enough" pass. Record whichever real state you observe, including "blocked on Step 6" if that's the honest answer.

GATE (Stage 5 exit condition, docs/dc-dc-deployment-workflow.md): preflight.sh PASS before add-model (Step 3); post-deploy cloud-assert.sh --capture per DC (Step 10); controller backup taken + stored (Step 11); Ceph-over-v6 and geneve-over-v6 verified (Step 12, contingent on Step 6).

-> Repeat this entire runbook for the other DC (DC1 first is recommended, since only DC1 is unblocked by real address literals today) -> proceed to Stage 6 (DR wiring and failover drill) once BOTH DCs pass this gate.

Delivery checklist

  • bash scripts/repo-lint.sh clean (0 fail) before committing any repo 
    changes made while executing this runbook (the IPv6 overlay file, any
`bundle.yaml` VIP edit for DC2, any DC-qualified `asbuilt/` path fix).
  • Changelog entry per real repo change this runbook produces (the 
    overlay, any bundle edit, any script fix) -- next DOCFIX number via
`bash scripts/ledger-scan.sh`, same as every other delivery this
session.
  • docs/session-ledger.md updated with the outcome, per DC.
  • docs/dc-dc-deployment-workflow.md Stage 5 row updated once BOTH DCs 
    have actually passed Step 12's gate (not just DC1).
  • Log the Step 6 charm-config research (v6 SAN support, v6 addressing 
    option names) as its own follow-up item if not resolved inline during
this stage's real execution -- do not leave it silently unresolved in
the workflow doc's tooling gap register.