Phase 01 -- Bundle Deploy

Deploy the hardened bundle + the octavia-pki overlay onto the freshly-prepped MAAS machines, and verify it settles to the expected PRE-vault-init state (zero errors, vault awaiting init, the TLS consumers awaiting vault certs). Vault init is phase-02.

Decisions: B5 (IP-only), D-019 (no designate -- SUPERSEDED 2026-07-10 by D-106, see the callout below), D-020 (dual provider+metal VIPs), R14 (VIPs front-loaded to .50-.60), Section-G NIC bindings. Troubleshooting: appendix-A -- R14 (VIP relocation), R15 (.10 phantom resolver), L1 (no set -e on count-gate blocks), L3 (metal-side dual-VIP eyeball check), DOCFIX-016 (maas list leak).

!!! bundle.yaml CHANGED 2026-07-10 (DOCFIX-167, VR1 DC-DC buildout, D-106/ D-108 both ADOPTED): designate/designate-bind/designate-mysql-router/ designate-hacluster and cinder-backup/ceph-rbd-mirror are now IN the same bundle.yaml this phase deploys -- the counts and "NO designate" GATE text below are updated to match. Genuinely unresolved, flagged rather than decided: this is the SAME shared bundle file used for both a plain VR0/DC0 single-DC rehearsal redeploy and for standing up VR1's DC1 (per the buildout design's own framing, VR0's testcloud IS becoming DC1, not staying a separate thing) -- so ANY phase-01 run from this point forward gets these new charms, whether the intent is a VR0-only continuation or a DC-DC DC1 build. If a future session needs a plain VR0-only redeploy WITHOUT Designate/cinder-backup/ceph-rbd-mirror, that needs an explicit decision (a bundle fragment/override removing them for that one run) -- do not assume either interpretation silently.

Prerequisites (must be true entering phase-01)

phase-00 done: 4 machines Ready/power=off; MAAS carve applied (front-loaded VIP /26 reserved, FIP pool reserved, stale iprange gone); enp8s0 data NIC linked on ALL four hosts; OSD /dev/vdb wiped blank.
overlays/${DC}-octavia-pki.yaml present (Step 1.0). PER-DC since F1 (2026-07-29) -- export DC (region-qualified, e.g. vr1-dc0) before running any step in this phase.
Hardened bundle.yaml in the working dir (channels pinned; VIPs .50-.60; reserved-host-memory 8192; image-conversion; use-policyd-override).

GATE (DOCFIX-073) -- run THE pre-deploy gate before add-model:

bash scripts/preflight.sh

Expect PREFLIGHT: PASS (repo lint + bundle invariants + Charmhub channel assert + live MAAS pre-flight, aggregated). WARN = review then decide; FAIL = do not deploy. This is the single entry point that replaced the six separate gate artifacts an operator previously had to remember.

Constants and env-literals

Host identity / system_ids: resolve LIVE via scripts/lib-hosts.sh (DOCFIX-040: system_ids are re-minted on every reenrollment -- a documented id is stale by definition; never paste one into a command).
Planes / CIDRs / VIP bands: scripts/lib-net.sh is the single authority (D-052/D-053). Resolve MAAS subnets BY CIDR (PATTERN-1), never by numeric subnet id -- ids drift across MAAS lifecycles. (DOCFIX-068 removed a stale pre-D-052 literal map that sat here.)
expected plan (as of bundle.yaml post-DOCFIX-167, 2026-07-10): 56 apps, 108 relations, 4 machines (bundle 8/9/10/11 -> juju 0/1/2/3), 24 LXD. (Was 50 apps/97 relations before the Designate + cinder-backup + ceph-rbd-mirror reactivation -- see the callout above.)

Run-location legend

# RUN: jumphost -- juju + maas admin (MAAS profile is admin; never maas list -- DOCFIX-016).

Command-label convention

Every command block below is bracketed by bold labels, so a command line is never mistaken for surrounding prose (these render in GitBucket and read clearly in a raw editor):

RUN -- LOC -- the block CHANGES state; run it at LOC (e.g. jumphost, vault/0, jumphost -> magnum/0).
CHECK (read-only) -- LOC -- a read-only verification; safe to re-run.
GATE: -- a hard stop; do NOT proceed past the block unless the stated condition holds.
Expect: -- what a passing result looks like.
> CAUTION: -- marks a destructive, secret-handling, or irreversible step.

Step 1.0 -- Octavia PKI overlay (secret-handling prereq) DISCRETE

overlays/${DC}-octavia-pki.yaml carries the 5 lb-mgmt-* PKI keys (controller CA/cert, issuing CA key+passphrase+cert). It is the ONLY overlay in the deploy command and is secret-safe + ASCII. PRIMARY path: reuse the existing validated overlay (the CAs are 10y, so it survives rebuilds). REGENERATION path (fresh CAs): run the discrete secret procedure inlined as "Step 1.0-GEN" at the end of this phase. Either way, confirm the overlay parses and contains exactly the 5 keys (sanity block below) before deploying.

CHECK (read-only) -- jumphost

# RUN: jumphost -- sanity only (does NOT print key material)
DC="${DC:?export DC=vr1-dc0|vr1-dc1 -- the PKI overlay is per-DC (F1)}"
OVL="overlays/${DC}-octavia-pki.yaml"
# Assert on CONTENT, not existence: an absent file, a wrong key count and a non-ASCII
# byte are three DIFFERENT answers and none of them may read as OK.
if [ ! -f "$OVL" ]; then
  echo "MISSING $OVL -- generate it with Step 1.0-GEN for DC=$DC"
else
  K=$(grep -cE 'lb-mgmt-' "$OVL" || true)
  [ "$K" -eq 5 ] && echo "OK 5 lb-mgmt-* keys" || echo "FAIL $K lb-mgmt-* keys (want 5)"
  LC_ALL=C grep -qP '[^\x00-\x7F]' "$OVL" && echo "FAIL non-ASCII bytes" || echo "OK ASCII clean"
fi

Step 1.1 -- Pre-deploy verify (read-only; 4 checks)

One consolidated read-only block. NO set -e (a guarded count of 0 is a valid answer, not a failure -- appendix-A: L1); count greps are || true.

CHECK (read-only) -- jumphost

( {
  echo "=== CHECK 1: bundle VIPs (quote-tolerant, octet-anchored) ==="
  grep -nE '^[[:space:]]+vip:' bundle.yaml overlays/vr1-dc0-vips.yaml
  TOT=$(grep -hcE '^[[:space:]]+vip:[[:space:]]*"?10\.12\.4\.' bundle.yaml overlays/vr1-dc0-vips.yaml 2>/dev/null | awk '{s+=$1} END{print s+0}')
  HI=$(grep -hcE '^[[:space:]]+vip:[[:space:]]*"?10\.12\.4\.(5[0-9]|6[0-2])("|$|[[:space:]])' bundle.yaml overlays/vr1-dc0-vips.yaml 2>/dev/null | awk '{s+=$1} END{print s+0}')
  LO=$(grep -hcE '^[[:space:]]+vip:[[:space:]]*"?10\.12\.4\.(1[0-9]|20)("|$|[[:space:]])' bundle.yaml overlays/vr1-dc0-vips.yaml 2>/dev/null | awk '{s+=$1} END{print s+0}')
  echo "  provider VIPs total=$TOT  in .50-.60=$HI  in .10-.20(stale)=$LO   (want 13/13/0)"
  # RULING 3 (2026-07-25): the counts span bundle.yaml AND its per-DC VIP overlay,
  # because the base is now VIP-FREE. `grep -c` over two files prints one count PER
  # FILE, so these sum with awk -- a bare $( ) would capture a two-line string and
  # every numeric test below would silently fail.
  # metal side is the second token of each dual vip; eyeball that all 11 are .8.50-.60,
  # clear of metal infra .8.10(maas)/.8.20(lxd)/.8.21(capi)/.8.30(juju) -- appendix-A: L3.

  echo "=== CHECK 2: enp8s0 data NIC linked on ALL FOUR hosts (10.12.12.0/22) ==="
  for SID in 4na83t qdbqd6 h8frng tmsafc; do
    echo -n "  $SID: "
    maas admin interfaces read "$SID" | jq -r '.[] | select(.name=="enp8s0")
      | [.links[]? | select(.subnet.cidr=="10.12.12.0/22") | .ip_address] | join(",")'
  done   # expect 10.12.12.40 / .41 / .42 / .43 (select by .subnet.cidr -> robust to id drift)

  echo "=== CHECK 3: subnet DNS resolvers ==="
  for ID in 1 2 6 7 8 9; do maas admin subnet read "$ID" | jq -c '{id,cidr,dns_servers}'; done
  # expect subnet 1 (provider) -> [10.12.4.1]; 2/6/7/8/9 -> [10.12.8.1]

  echo "=== CHECK 4a: nodes Ready / power off ==="
  maas admin machines read | jq -r '.[] | select(.system_id|IN("4na83t","qdbqd6","h8frng","tmsafc"))
    | "\(.hostname) \(.status_name) power=\(.power_state)"'
} )

CHECK (read-only) -- jumphost

# CHECK 4b: OSD /dev/vdb blank (DOCFIX-027 -- LOCAL libvirt-host loop, NOT ssh: the four
# hosts are Released/powered-off entering phase-01, and /var/lib/libvirt/images is a
# hypervisor (jumphost) path that does not exist on the hosts. RUN: jumphost (libvirt host; sudo).
for h in openstack0 openstack1 openstack2 openstack3; do
  echo "== $h =="
  sudo qemu-img info "/var/lib/libvirt/images/${h}-1.qcow2" | grep -E 'virtual size|disk size'
done   # expect virtual 512 GiB, disk ~200 KiB (sparse/blank)

GATE: VIPs 11/11/0; enp8s0 linked on all 4; subnet DNS as above; 4 nodes Ready; OSD blank.

Step 1.2 -- Dry-run (guarded)

Refuse to add a model if openstack already exists; require the overlay.

RUN -- jumphost

( {
  juju models 2>&1 | tee /tmp/jmodels.txt
  if grep -qE '(^|[[:space:]]|/)openstack([[:space:]*]|$)' /tmp/jmodels.txt; then
    echo "ABORT: an 'openstack' model already exists (teardown is phase-00)"; 
  elif [ ! -f "overlays/${DC:?export DC first}-octavia-pki.yaml" ]; then
    echo "ABORT: overlays/${DC}-octavia-pki.yaml missing (Step 1.0-GEN for DC=$DC)";
  else
    juju add-model openstack
    juju deploy ./bundle.yaml --overlay "overlays/${DC}-vips.yaml" \
      --overlay "overlays/${DC}-octavia-pki.yaml" -m openstack --dry-run
  fi
} )

GATE: (from the plan): 56 apps, 108 relations, 4 machines (8/9/10/11 -> 0/1/2/3), 24 LXD; ceph-osd/0-3 one per node; nova-compute/0-2 on machines 1/2/3 ONLY (machine 0 = OSD+LXD host, no compute); channels match the matrix; relations include octavia:certificates - vault:certificates, vault:shared-db - vault-mysql-router, mysql-innodb-cluster:certificates - vault:certificates; NO vault:ha (still absent, unchanged); designate/designate-bind/designate-mysql-router/designate- hacluster and cinder-backup/ceph-rbd-mirror NOW PRESENT (D-106/D-108, 2026-07-10 -- corrects this GATE's old "NO designate (D-019)" text, since D-019 is superseded). Only the two benign R11 warnings (L34 name, L55 variables).

Step 1.3 -- Deploy (VIP-guarded)

Re-run the VIP guard inline (the dry-run never echoes vip values), then deploy only if the guard exits 0.

THE GUARD IS NOW THE MAINTAINED CHECKER, NOT A HAND-ROLLED GREP (2026-07-29). This block used to compute a 13/13/0 triple with three grep -hcE counts anchored on the literal 10\.12\.4\.. Three defects, all measured: it hard-ABORTED on vr1-dc1, whose provider band is 10.12.64.x; it counted only IPv4, so after R2's dual-stack ADD the ruled v6 legs were checked by NOTHING (chain-audit finding 22); and its prose said "11/11/0" while its code demanded 13, a drift that appeared the moment R11 added vault .61 and designate .62. scripts/provider-bundle-check.py already encodes every one of those invariants -- per-DC bands, the VIP count, and since 2026-07-28 the dual-family ARITY coupling -- so the runbook defers to it instead of keeping a second copy that drifts. One source of truth; a new ruling updates the checker and this step inherits it.

RUN -- jumphost

( {
  DC="${DC:?export DC=vr1-dc0|vr1-dc1 first}"
  set -- ./bundle.yaml
  for OVL in "overlays/${DC}-vips.yaml" "overlays/${DC}-machines.yaml" \
             "overlays/${DC}-octavia-pki.yaml"; do
    [ -f "$OVL" ] && set -- "$@" --overlay "$OVL"
  done
  # The octavia-pki overlay is REQUIRED for deploy (it carries the 5 lb-mgmt-* keys) --
  # absence is an ABORT, not a silently-skipped overlay.
  [ -f "overlays/${DC}-octavia-pki.yaml" ] || {
    echo "ABORT: overlays/${DC}-octavia-pki.yaml missing -- run Step 1.0-GEN for DC=$DC"; exit 1; }

  echo "validating the MERGED $DC deploy input: $*"
  if python3 scripts/provider-bundle-check.py "$@" --dc "$DC"; then
    juju deploy "$@" -m "${DC_MODEL:?export DC_MODEL first}"
  else
    echo "ABORT: VIP/bundle invariants failed for $DC -- fix before deploying"; exit 1
  fi
} )

CONVERGENCE WATCH (ENHANCEMENT-1): keep two windows open for the whole deploy arc.

Window 1 (detail): juju status -m openstack --watch 5s (always explicit -m; on a 50-app model the table exceeds one screen, so this is a slice -- optionally filter, e.g. ... 'mysql*' vault).
Window 2 (signal): scripts/deploy-watch.sh openstack 15 -- compact machine/unit state counts
- named error/blocked units. Health-at-a-glance: the error/blocked section stays EMPTY until the expected late blocks (vault needs-init, octavia awaiting-configure). Neither window descends into subordinates; neither replaces the phase gates.

Step 1.4 -- DNS gate during deploy (as machines come up)

Run when machine 0 reaches started, then per LXD unit as they appear (flag BEFORE the target; logic inside the remote quotes; no outer 2>/dev/null):

CHECK (read-only) -- jumphost

juju ssh -m openstack 0 -- 'resolvectl status | grep -i "DNS Server"; getent hosts api.snapcraft.io && echo OK || echo FAIL'
# repeat for ceph-mon/0, mysql-innodb-cluster/0 as they appear

GATE: each returns OK (api.snapcraft.io resolves -> the snap install storm proceeds clean). FINDING (non-blocking, R15): the unreachable region resolver 10.12.8.10 (MAAS region/rack controller, advertised on the metal VLAN independent of the subnet field) may still appear in a node's resolver list -- resolution succeeds because systemd-resolved deprioritizes .10 and falls through to .1. Latent fragility if .1 ever drops; understand/eliminate for Roosevelt. (appendix-A: R15.)

EXIT GATE (phase-01 complete)

Deploy settled to the PRE-vault-init end state:
- ZERO units in error.
- mysql-innodb-cluster x3 ACTIVE ("Cluster is ONLINE").
- vault/0 BLOCKED "Vault needs to be initialized" (the phase-02 trigger, not a fault).
- Waiting on vault certs (expected pre-init): ovn-central x3, ovn-chassis x3 (incl nova-compute subordinates), ovn-chassis-octavia, neutron-api-plugin-ovn, barbican-vault.
- octavia BLOCKED "Awaiting configure-resources" (D-021); gss unknown (pre-run).
- NOT YET OBSERVED LIVE (2026-07-10, D-106/D-108 reactivation) -- add to this checklist once a real deploy confirms it: designate and designate-bind have a certificates: metal-internal relation to vault (per bundle.yaml), so BY THE SAME PATTERN as the apps above they are LIKELY also waiting on vault certs pre-init -- inferred from the relation graph, not confirmed against a real deploy. cinder-backup (subordinate) and ceph-rbd-mirror have no certificates relation, so no equivalent expectation for them. If a real deploy shows anything different for designate/designate-bind, correct this list rather than trusting this inference.
- magnum/0 BLOCKED "Ports which should be open, but are not: 9501" -- pre-vault posture: magnum-api is loopback-bound ([api] host not yet templated) and haproxy backends target unit IPs. EXPECTED phase-01 end-state; self-resolves at the phase-02 cert rollout (apache2 takes *:9501). Confirmed self-resolving 2026-06-12 (FINDING-2); verify in the phase-02 post-init sweep.
- keystone/0 "PO (broken): Unit is ready" -- expected while use-policyd-override=true with no policy zip attached (FINDING-1); keystone runs the DEFAULT policy. No mutation this arc.
Section-G NIC payoff confirmed (no subset/binding errors): ceph-mon -> storage 10.12.16.x; octavia -> data 10.12.12.1; nova-compute -> data 10.12.12.4x; vault -> metal 10.12.8.x.
Proceed to phase-02 (vault init).

As-built reference (2026-06-03 second redeploy -- audit trail)

juju deploy ./bundle.yaml --overlay overlays/octavia-pki.yaml -m openstack on maas/default (cred maas-api).
Plan: 50 apps / 97 relations / 4 machines / 24 LXD; placement as above.
Pre-deploy verify: VIPs 11/11/0; enp8s0 -> 10.12.12.40-43 (all 4); subnet DNS as above; nodes Ready; OSD blank.
Settled: zero errors; mysql /0 R/W (10.12.8.173), /1 (.179) /2 (.185) R/O; vault blocked needs-init.

Balance / stability observations (Roosevelt rebalancing inputs -- post-deploy item 6)

Quorum triads (mysql-innodb-cluster, ovn-central, ceph-mon) all on machines 0/1/2: correct anti-affinity; machine 3 loss breaks no quorum; any single loss of 0/1/2 leaves a 2-of-3 majority.
Machine 0 = no-compute control host, largest container count: prefigures the Roosevelt role split.
FLAG: machine 3 concentrates six singletons (vault, glance, nova-cloud-controller, octavia, placement, barbican) + compute + OSD. Acceptable on testcloud; Roosevelt answer is role split + HA, informed by measured footprints at the 32 GiB envelope (DEVIATION-2 caveat).
rabbitmq-server single unit: messaging SPOF, as designed for v1.

PATTERN-1 (standing convention) -- dynamic lookup vs. pinned identifiers

READ/VERIFY ops discover values at runtime (never hardcode what resolves: hostname->system_id via maas admin machines read | jq; subnet id by CIDR). DESTRUCTIVE/IRREVERSIBLE ops discover dynamically, ASSERT against a pinned EXPECTED set, ABORT on mismatch, then operate on the pinned values (a filter bug or an unexpected new machine must not become collateral damage). Retrofit candidates (apply as fixture-tested gated blocks, NOT bulk edits): phase-00 release/host loops -> discover-assert-pin; subnet ids -> resolve by CIDR; octet maps -> derive from hostname index. Canonical statement in runbooks/README.md.

phase-02 -- vault bring-up.

Step 1.0-GEN -- Octavia management-PKI generation (regeneration path) DISCRETE / SECRET

CAUTION: SECRET step -- generates Octavia CA private keys + passphrases. Do NOT echo or log key material (only cert dates/subjects + verify-OK are printed). The overlay it writes is gitignored.

Run ONLY if this DC has no PKI yet -- 1.0-GEN.0 REFUSES if overlays/${DC}-octavia-pki.yaml or this DC's workspace already exists, because regenerating invalidates every amphora already issued against that CA. Each DC is generated separately. Produces the two-tier EC PKI for Charmed Octavia's amphora trust domain and writes the overlay. Decisions (Workstream 3a, 2026-05-22): fresh generation; EC P-384 CAs (SHA-384, 10y); EC P-256 controller cert (2y); overlay-file distribution (gitignored); artifacts under $HOME/octavia-pki/; passphrases = 32 random bytes base64 (44 chars). SECRET step -- do NOT echo key material; the only printed values are cert dates/subjects and verify OK.

The five octavia charm options the overlay sets:

lb-mgmt-issuing-cacert = base64(issuing CA cert)
lb-mgmt-issuing-ca-private-key = base64(issuing CA ENCRYPTED key)
lb-mgmt-issuing-ca-key-passphrase = the issuing CA passphrase (PLAIN string, NOT base64)
lb-mgmt-controller-cacert = base64(controller CA cert)
lb-mgmt-controller-cert = base64(controller cert + key, concatenated)