# STAGE-5 READINESS -- ordered precondition checklist (2026-07-27)

**Read this first.** It is the answer to "are we ready to start Stage 5 error-free?"
The full committee record is `docs/audit/stage5-committee-raw-20260727.md`; the
questions needing your decision are `docs/audit/queued-rulings-20260727.md`; this
session's own measurements are `docs/audit/stage5-live-measurement-20260727.txt`.

**This file carries NO status tokens.** Stage/gate status lives only in
`docs/CURRENT-STATE.md` (GA-R1). Rows below are readiness assessments, not stage
states.

---

## The short answer

**No -- Stage 5 would not run error-free today, and it would fail early.**

That is not a bad outcome for this audit; it is the point of it. The substrate
underneath is in excellent shape. What is not ready is the layer between the
substrate and the deploy: the execution host, the deploy artifacts, several
literals, and the gates meant to catch exactly these problems.

**What is genuinely solid (measured, not read):**
- All three OpenTofu roots plan **ZERO DIFF** -- outer on vcloud and both inner
  roots on voffice1, in one session for the first time. The substrate matches
  its configuration exactly.
- 18 nodes `Ready`, shapes EXACT to D-121 Option C, all 18 pinned MACs matching
  `lib-hosts.sh`, all 18 `power_type=virsh` with correct power addresses.
- The D-134 statics are perfect: one octet per node across all six planes, 18/18.
- 17 fabrics, exactly as the carve-residue cleanup recorded. Seven interfaces per
  node, **zero orphans**.
- Both artifact paths genuinely serve (dc0 mirror 200 + sync attested current;
  dc1 proxy 200 for archive and UCA). Both D-131 forwarders answer.
- Gauntlet ALL GREEN (81); repo-lint 0 fail; preflight red for exactly the known
  reasons with nothing new joined.
- **The dc1 edge's egress is MEASURED OPEN right now** (probed directly from the
  rack, bypassing the apt proxy so a cache hit could not fake it):
  `streams.canonical.com/juju/tools/` -> 200 -- the juju agent stream bootstrap
  actually needs -- plus `api.snapcraft.io` reachable, `archive.ubuntu.com` -> 200
  direct, and `ping 1.1.1.1` 0% loss, with the default route via `10.12.64.1`.
  This was previously carried as a record, not a measurement. It means the
  bootstrap window is open today (see 5.2).

**What blocks it** -- in the order you would hit it:

1. **Stage 5 executes on voffice1, and voffice1 is 105 commits behind `main`,**
   sitting on a branch retired four days ago. `overlays/vr1-dc1-vips.yaml` and
   `overlays/vr1-dc1-machines.yaml` are ABSENT there and `bundle.yaml` is still
   the VR0 4-node hyperconverged layout. You would deploy the wrong topology.
2. **The `openstack` client is installed on neither host,** and ten Stage-5/6/7
   scripts invoke it.
3. **The Juju controller has nowhere to land.** The dedicated 10th VM of the D-104
   amendment is UNAUTHORED -- no OpenTofu resource anywhere, not merely un-applied
   -- and the `juju-controller-vr1-dc1` tag does not exist in MAAS.
4. **`ceph-osd` has no device.** `bundle.yaml` says `/dev/vdb`; all 18 nodes have
   exactly one disk, `vda`.
5. **Per-role MAAS tags are consumed by the machines block and authored nowhere;**
   an unmatched tag yields no candidate machine, so allocation fails hard.
6. **Dual-stack is RULED and there is no IPv6 anywhere in the DC substrate.**
7. **Stage 5's Step 4 says "follow phase-01 verbatim", and phase-01 actively
   REFUSES to deploy dc1.** Its VIP guard greps `bundle.yaml` for eleven
   `10.12.4.x` VIPs; dc1's live in an overlay on `10.12.64/68/72`, so the guard
   evaluates 0/0/0 and takes the `ABORT: VIP guard failed` branch. It also
   carries hardcoded VR0 system_ids, a jumphost-local libvirt loop over disks
   that do not exist, and a 4-machine plan gate against a 9-machine bundle.
   **After the ruled VIP extraction it will abort for dc0 too.**
8. **Several gates that would have caught these cannot fail** -- including
   `repo-lint` returning `PASS (0 fail, 0 warn)` over ZERO files on a
   one-character typo, which is the gate every stage close cites.

**And three gate commands in the Stage-5 runbook cannot execute at all**, measured
against the juju actually installed (3.6.27): `juju run` is used for a shell
command when it is the *action* runner (`juju exec` is the shell one);
`juju download-backup` is given a backup-id when it takes a controller-side path;
and the geneve gate greps `ovn-central` for a config key that the same runbook
states does not exist. Separately, **four of the five VERIFY-LIVE gates the record
says Stage 5 owes have no step in the runbook at all** -- including the keystone
policyd-override check, which is a RULED gate.

**Fifteen decisions are yours**, queued in `queued-rulings-20260727.md`: **11 in
Part A that block Stage 5 (R1-R11)** and **4 in Part B that do not (R12-R15)**.
Answer them ONE AT A TIME -- GA-R5 makes a batch answer invalid, so "yes to all"
rules nothing. Everything else on this list is mechanical.

**Answer R1, R2 and R3 first.** They change what gets deployed, and several later
questions have different right answers depending on them -- R2 in particular feeds
R8, R9 and R11.

**Two convergent confirmations worth noting**, because they raise confidence in
the whole set: the Ceph OSD blocker was found INDEPENDENTLY by two lenses using
different methods (lens 2 via `virsh domblklist` on both racks; lens 5 via MAAS
`blockdevice_set` on dc1's nine), and the stale-clone blocker was found
independently by this session and by lens 5.

---

## Legend

- **DONE** -- verified this session, with evidence.
- **MECHANICAL** -- no choice required; a session can execute it under normal gating.
- **NEEDS-RULING** -- requires your decision first; see the R-number.
- **UNMEASURED** -- could not be looked at. Never read as "fine".

Order matters: within a phase, items are independent; phases are sequential.

---

## Phase 0 -- Execution environment (before anything else)

| # | Precondition | Status | Evidence | If skipped |
|---|---|---|---|---|
| 0.1 | The Office1 headend clone is at `main` | **MECHANICAL** | `ssh voffice1 ... rev-list --left-right --count origin/main...HEAD` -> `105 0`, HEAD `61c416e` on retired branch `dc-dc-g12-dc1-substrate`; both dc1 overlays ABSENT; `bundle.yaml` sha differs and its machines block is `"8"-"11" tags=openstack` | Deploy fails on missing overlay paths, or silently deploys VR0's 4-node hyperconverged topology into dc1 |
| 0.2 | The `openstack` client exists on the D-128 Plane-2 host | **MECHANICAL** | `command -v openstack` ABSENT on vcloud AND on voffice1; voffice1 snaps are core24/juju/lxd/maas/postgresql/snapd only; ten `phase-0*` scripts invoke it | Every step from Stage 5 Step 7 onward has no client to run |
| 0.3 | Stale remote-tracking refs pruned on voffice1 | **MECHANICAL** | that clone still lists `origin/dc-dc-g12-dc1-substrate` and `origin/dc-dc-stage3-phase2-dc-substrate`; ground truth `git ls-remote --heads origin` shows only `main` + this audit branch | A naive `git pull` there operates on a branch that no longer exists upstream |
| 0.4 | Juju client version recorded | **DONE** | measured 3.6.27 rev 35621 `3/stable` on voffice1; supersedes the recorded 3.6.25. In-channel patch refresh, which D-071 permits -- policy-compliant, now corrected in CURRENT-STATE section 7 | -- |

---

## Phase 1 -- Decisions that change what gets deployed

Answer these before touching artifacts; several later items have different right
answers depending on them.

| # | Precondition | Status | Ruling | Why it is first |
|---|---|---|---|---|
| 1.1 | Ceph OSD device resolved | **NEEDS-RULING** | **R1** | `bundle.yaml:560 osd-devices: /dev/vdb`; all 18 nodes have only `vda` (measured twice). Option (a) re-opens both substrate roots, which currently plan zero diff |
| 1.2 | Address family resolved | **NEEDS-RULING** | **R2** | D-101 RULED dual-stack for both DCs *this deployment*; measured ZERO IPv6 on any of the 12 plane fabrics and on all 18 nodes. Addresses become as-built Keystone endpoints and cert SANs at deploy |
| 1.3 | MTU branch chosen | **NEEDS-RULING** | **R3** | Measured state is in NEITHER of D-101's two permitted shapes (bridges 9000, MAAS VLANs 1500, transit 1500, no MTU option anywhere). Failure appears later as tenant/geneve blackholing |
| 1.4 | Designate's stage resolved | **NEEDS-RULING** | **R5** | It ships in the bundle, so Stage 5 deploys it, while Stage 7's gate requires it to be newly added |
| 1.5 | HA scale-up overlay decided | **NEEDS-RULING** | **R6** | Stage 6's radosgw tooling is single-unit-shaped; scaling afterwards is a live-cloud change |
| 1.6 | Octavia CA scope decided | **NEEDS-RULING** | **R7** | Shared CA across two independent clouds vs per-DC trust domains; D-109 already rules per-DC Vault roots |
| 1.7 | Octavia `lb-mgmt-net` family decided | **NEEDS-RULING** | **R8** | Stage 6's entry condition depends on a gate Stage 5 can close without producing |
| 1.8 | vault + designate VIPs decided | **NEEDS-RULING** | **R11** | Both scaled 3 with `cluster_count: 3` and NO vip; the checker skips any app without one |

---

## Phase 2 -- Substrate and MAAS preconditions (each a gated live mutation)

| # | Precondition | Status | Evidence | If skipped |
|---|---|---|---|---|
| 2.1 | The 10th Juju-controller VM authored, applied, tagged | **MECHANICAL** (shape ruled by the D-104 amendment) | UNAUTHORED: no controller module in either substrate root, `grep -rn "juju-controller-vr1-dc1" opentofu/ runbooks/ scripts/ bundle.yaml overlays/` -> zero hits; live MAAS has 5 tags and none is a controller tag; `virsh list --all` shows exactly 9 nodes + 1 edge per DC | `juju bootstrap` has no target machine, or lands on one of the nine role nodes -- the outcome the D-104 amendment explicitly rejected |
| 2.2 | Per-role MAAS tags created | **MECHANICAL** (Fork 3 already OPS-ruled) | Consumed by `bundle.yaml:81-97` and `overlays/vr1-dc1-machines.yaml:23-39` (`tags=openstack-vr1-dc1,<role>`); the only tag-create in the tree makes ONE tag (`reenroll-hosts.sh:177-180`); none exist live | Juju passes `tags=` to MAAS as an allocation constraint; an unmatched tag yields no candidate machine and the deploy fails hard at allocation |
| 2.3 | Reserved VIP/utility bands exist in MAAS | **NEEDS-RULING** -> then gated mutation | **R4**. Measured: THREE ipranges cloud-wide, all `type=dynamic`; ZERO reserved; `.50-.99` allocatable on 12/12 DC plane subnets; `phase-00-maas-standup.sh:136` refuses non-VR0 | Nothing stops MAAS handing a VIP-band address to a container mid-deploy; `phase-04-network-verify.sh:100` hard-fails on the missing FIP pool |
| 2.4 | OSD volumes present (only if R1 = option a) | **NEEDS-RULING** first | `modules/node-vm/main.tf:104-125` declares a single `vda` | `ceph-osd` finds no device |
| 2.5 | IPv6 carved into MAAS (only if R2 = dual-stack) | **NEEDS-RULING** first | D-101's own remaining open item is the un-assigned org ULA /48 + per-DC GUA carve | Nothing to bind the ruled ULA legs against |

---

## Phase 3 -- Deploy-artifact and runbook corrections (all DOCFIX, no choice required)

The Stage-5 runbook is the single most defective surface found. **Twenty-one
corrections are listed below**; following the runbook verbatim today fails or
misdeploys in at least nine independent ways (3.1, 3.2, 3.4, 3.5, 3.11, 3.12,
3.13, 3.14, 3.15), and three of its named gate commands cannot execute at all
against the juju that is actually installed.

| # | Correction | Finding | If skipped |
|---|---|---|---|
| 3.1 | Retire the `DC1`/`DC2` namespace throughout the phase-4 runbook | L1-6 | Two readings, both wrong and both destructive: read literally, Stage 5 STOPs on the very DC it is meant to build ("DC2 ... BLOCKED ... STOP"); read by position, `vr1-dc1` is treated as "DC1" and gets dc0's `10.12.4/8/12` VIPs |
| 3.2 | Fix Step 6's `juju deploy` overlay set | L1-2, L6-10 | It names two overlays that do not exist (`octavia-pki.yaml`, `${DC}-hostnames.yaml`) and omits all three that constitute the ruled dc1 deploy input |
| 3.3 | Remove Step 4's "edit `bundle.yaml`'s VIP lines" instruction | L1-3 | Directly reversed by ruling 3 of 2026-07-25 (bundle becomes VIP-free; per-DC values arrive via overlay) |
| 3.4 | Step 2: add the controller-tag constraint; drop the "MAAS-deployed machine" target | L1-4 | Bootstraps onto a role node; the READY-handoff ruling (DOCFIX-200) already removed the Deployed premise |
| 3.5 | Steps 1/2: state that juju runs on voffice1, not the jumphost | L1-5 | The first mutation has no binary to run on the host the runbook names |
| 3.6 | Resolve `lib-net.sh`'s dc1 arm + sweep the 21 non-selector consumers | L1-1, L7-3, L7-10 | Scripts either die under `set -u` or silently write dc0's literals into dc1. **Guard: `METAL_INTERNAL_VID`/`IFACE` must STAY unset -- D-133 abolished that stack for VR1** |
| 3.7 | `pre-flight-checks.sh`: remove the VID-103 / `br-internal` assertions | L3-6 | Two checks contradict a RULED decision and can never pass on either VR1 DC; D-133's own affected-surfaces list omits this file |
| 3.8 | `bundle.yaml`'s `ovn-chassis` MACs are dc1's, in the file documented as dc0's source of truth | L3-8 | Harmless for a dc1-first Stage 5; for any dc0 deploy, no local MAC matches, `ovn-chassis` builds no br-ex mapping, provider egress is dead on dc0 compute, and no gate fails |
| 3.9 | Teardown/rollback: Step 2's false clear and Step 3's phantom modules | L6-2, L6-3 | **The most dangerous item in the audit.** Step 2 decides "nothing to clean up in MAAS" by reading a `vm_host` that deliberately does not exist (per-machine virsh power, D-103/D-123), then routes you to destroy the containment VM out from under 18 live MAAS records. Step 3's scoped destroy names five modules that exist in neither root. This runbook is what you reach for *during* a failed Stage 5 |
| 3.10 | Gap register: stale-OPEN item 17, duplicate item 19, unrecorded item 20 | L6-11, L6-12, L6-13 | The skill routes you to read this register FIRST; item 17 produces a false STOP, and item 20's per-DC reach verdict was never measured or recorded for either DC |
| 3.11 | **Stop delegating "follow phase-01 verbatim"** -- phase-4 must carry its own DC-parameterised deploy step, or phase-01 must be parameterised | L5-2 | phase-01 REFUSES dc1: VIP guard aborts (0/0/0), hardcoded VR0 system_ids, a jumphost-local libvirt loop over `/var/lib/libvirt/images/${h}-1.qcow2` for nodes that live inside `vvr1-dc1`, and a 4-machine plan gate against 9 machines. **This is the single largest runbook defect found** |
| 3.12 | The octavia-pki generator is dc0-frozen and cannot produce a dc1 artifact | L5-3 | Its VIP gate hard-fails anything not matching `^10\.12\.4\.`; dc1's octavia VIP is `10.12.64.57` and lives in an overlay, not in `bundle.yaml`. CN/SAN are `...dc0.vr0...` literals. phase-4 offers reuse-vs-regenerate as a choice **without noting regenerate is currently impossible** -- so R7 must be answered knowing that |
| 3.13 | `juju run` -> `juju exec` in Step 12's Ceph gate | L5-5 | Measured on 3.6.27: `juju run` is the ACTION runner. As written the gate asks ceph-mon to run an action named `ceph -s`. A named exit-gate command that cannot execute |
| 3.14 | Step 12's geneve gate asserts a nonexistent charm option | L5-6 | It greps `ovn-central` for `encap` while Step 6 of the SAME runbook states no such option exists. DOCFIX-204 class: the gate must assert the observable (encap family on a live chassis), not a config key |
| 3.15 | Step 11's backup commands are wrong-shaped | L5-7 | Measured on 3.6.27: `download-backup` takes a controller-side PATH not a backup-id, `-m` takes a MODEL not a controller name, and `create-backup` already downloads by default -- so the second command is redundant. The exit gate requires "controller backup taken + stored" |
| 3.16 | Place the four missing VERIFY-LIVE gates | L5-8 | `grep -niE 'cluster_count\|min_size\|failure-domain\|policyd\|PO:'` on the runbook returns **ZERO lines**. The keystone policyd-override gate is RULED (2026-07-25), so its absence contradicts a ruled decision |
| 3.17 | Move the machines-block `--dry-run` AHEAD of the deploy | L5-9 | The only `--dry-run` sits in Step 6; the real deploy is Step 4. The gate that proves the dc1 retag merged runs after machines are already allocated. It also carries no pass/fail criterion |
| 3.18 | Add the `apt-http-proxy` model-config step | L5-10 | **Ordering trap.** `apt-http-proxy` appears only as prose at `:31`, no command. Unset, every unit apts from the internet during deploy; the D-135/D-107 egress narrowing then kills a half-built model -- and unlike bootstrap the recovery is not "re-open egress and retry" but a partially-installed cloud |
| 3.19 | Add `juju add-model` and define `${DC_MODEL}`; invoke `juju-spaces-check.sh` | L5-11 | `${DC_MODEL}` is used once and defined nowhere, so Step 6's command expands to `-m ""`. The only add-model is phase-01's hardcoded `juju add-model openstack` |
| 3.20 | Make Step 2.0 CHECK-first for the MAAS credential | L5-12 | Measured: `vr1-dc0-cred` and `vr1-dc1-cred` already exist (`credential-count: 2`). Re-running the step as written mints a duplicate admin + a second key -- credential sprawl against SEC-018/-019 and D-137 |
| 3.21 | Add a mid-deploy watch and failure decision tree | L5-18 | phase-4 references no `deploy-watch.sh`, no expected settle-state, no stop/fix-forward/rollback branch. The apparatus exists only in phase-01, whose expected end-state is the 4-machine VR0 model |

---

## Phase 4 -- Gate integrity (fix before relying on any gate to authorise Stage 5)

Each was demonstrated with a constructed, measured false green.

| # | Gate defect | Finding | Consequence |
|---|---|---|---|
| 4.1 | `repo-lint` PASSES over ZERO files on a one-character typo of its flag or root | L4-1 (**re-verified by this session**) | The gate whose "0-fail" every GA-R6 stage-close commit cites cannot distinguish a clean repo from an unexamined one |
| 4.2 | `provider-bundle-check` passes decorative HA | L4-3 (**re-verified**: `grep -rn cluster_count scripts/ tests/` -> nothing, against 20 occurrences in the overlay) | 13 hacluster subs at `cluster_count: 1` behind 3-unit principals produce a byte-identical PASS |
| 4.3 | `provider-bundle-check` treats an empty or mis-keyed overlay as a silent no-op | L4-4 | A truncated or typo'd Stage-5 overlay is indistinguishable from a correct one |
| 4.4 | `preflight` ignores every sub-gate exit code that is not 1 or 2 | L4-5, L4-6 | A gate that crashes (127) leaves `PREFLIGHT: PASS -- clear to add-model / deploy`; a checker's deliberate hard-error 2 is downgraded to WARN in P1, P2 and P3 |
| 4.5 | P3 verified ZERO of 33 charm-channel pins today | L4-2 | `juju` is not on the host's PATH and a missing BINARY is folded into the same WARN as an unreachable charmhub. **No committed Caracal channel pin is currently verified by any executable check, on the eve of the deploy stage** |
| 4.6 | The gauntlet's harness count is compared to nothing | L4-8 | A renamed or deleted harness is neither run nor failed; it still prints ALL GREEN. 81 exists only as prose |
| 4.7 | `cloud-assert.sh` asserts a unit count exactly once | L4-10 | The Stage-5 completion verifier catches a sick unit and cannot catch a missing one; "Cluster ID uniform across units" passes over a SINGLE unit |
| 4.8 | G17's named check cannot fail | L4-7 | `curl -sI` exits 0 on 404/500 and the dc0 URL is an autoindex root answering 200 with nothing behind it; the dc1 half names no command at all |

---

## Phase 5 -- The deploy itself

| # | Precondition | Status | Note |
|---|---|---|---|
| 5.1 | Stage-5 entry basis against a red preflight | **NEEDS-RULING** | **R10**. Preflight exits 1 for exactly the known reasons; "PASS" is unreachable as written |
| 5.2 | Edge egress OPEN at bootstrap | **MEASURED OPEN 2026-07-27** | Probed directly from the dc1 rack, bypassing the apt proxy so a cache hit could not fake it: `streams.canonical.com/juju/tools/` -> **200** (the agent stream bootstrap needs), `api.snapcraft.io` reachable, `archive.ubuntu.com` -> 200, `ping 1.1.1.1` 0% loss, default route via `10.12.64.1`. Previously carried as a record only. **STANDING CAVEAT UNCHANGED: `juju bootstrap` fetches the agent stream and the juju/juju-db SNAPS pre-apt, which the dc1 apt proxy does NOT cover and D-135 items 2-3 do not exist -- so do not apply the egress narrowing until after bootstrap.** Re-probe immediately before bootstrap; this is a measurement with a shelf life |
| 5.3 | `juju bootstrap` with the controller-tag constraint | blocked on 2.1 | No target machine exists yet |
| 5.4 | `juju deploy` with the three ruled overlays and `--dc vr1-dc1` | blocked on Phase 1 + 3 | The runbook's current command is 3.2 |
| 5.5 | Pre-allocation power-state refresh | **MECHANICAL -- add as a one-line check** | L5-15: all nine dc1 nodes read a CACHED `power_state=error` until `maas admin machine query-power-state` was run explicitly, which returned `{"state":"off"}` on 9/9 and cleared it. A stale `error` immediately before deploy reads as a hard blocker and will stall Juju provisioning. **The remedy is a query-power-state refresh, not a rebuild** -- worth a runbook line so nobody diagnoses a phantom fault mid-deploy |

### Already DONE on the Juju side (measured, so they are not re-litigated)

- The MAAS cloud is registered: `juju show-cloud vr1-maas --client` -> `type: maas`,
  `endpoint: http://10.10.0.20:5240/MAAS`, `credential-count: 2`, region `default`.
- Both per-DC credentials exist: `juju credentials --client` -> `vr1-dc0-cred`,
  `vr1-dc1-cred` (SEC-018 / SEC-019).
- MAAS spaces are complete and correct for both DCs -- all six
  (`provider-public`, `metal-admin`, `metal-internal`, `data-tenant`, `storage`,
  `replication`), each carrying both DCs' /22s, matching `lib-net.sh:31` exactly
  with none of the `STALE_SPACES` present.
- `juju controllers` -> `ERROR No controllers registered.` -- expected; nothing to
  clean up before bootstrap.

---

## UNMEASURED -- could not look, and not to be read as "fine"

- The canonical piped rack-side checker invocations were run by lens 2 and PASSED
  on all three (`dc-mirror` dc0, `dc-cache-proxy` dc1, `dc-rack-net` both). This
  session's own first attempt ran them LOCALLY and produced a comprehensive false
  FAIL -- an owned error, recorded in the measurement capture.
- Juju's actual overlay-merge behaviour for the `machines:` block. No Juju client
  exists on the jumphost, so `provider-bundle-check`'s merge is a MIRROR of juju's
  documented behaviour, not juju itself. The overlay itself demands a
  `--dry-run` confirmation before deploy.
- `juju restore-backup` (Stage 6 Step 9, D-104 restore drill). Repo-wide grep
  returns only the two runbook lines; no script or decision corroborates the
  subcommand. Verify the moment a Juju client is in hand -- do NOT assume it is
  absent.
- Whether `curl` is on the racks' PATH, which both artifact checkers' only
  behavioural probe depends on.
- Charm option existence for `prefer-ipv6`, `min-cluster-size`, and `hacluster`'s
  `cluster_count` semantics. Not fetched from Charmhub this session; the overlays
  carry their own VERIFY-AT-DEPLOY caveats and nothing here asserts beyond them.
- Node-side anything. All 18 nodes are powered off in `Ready` under the
  READY-handoff ruling -- that is G17's territory by construction.