Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / audit / stage5-readiness-20260727.md
@JANeumatrix JANeumatrix 3 hours ago 24 KB Session-close sweep: three stale surfaces the audit itself created, plus transcript-only facts
Raw Blame History

STAGE-5 READINESS -- ordered precondition checklist (2026-07-27)

!! SUPERSEDED IN PART -- READ THIS BANNER FIRST (added 2026-07-27, same day) !!

Every NEEDS-RULING marker in this document is now HISTORICAL. This file was written while R1-R15 were open questions. They are now CLOSED: R1-R13 and R15 RULED individually (GA-R5, one exchange each), R2a and R14 WITHDRAWN as raised in error, and R16 DEFERRED BY DIRECTION and gated as G18. The 13 NEEDS-RULING rows below record what was owed AT THE TIME, not what is owed now.

Authorities, in order: docs/CURRENT-STATE.md for status and the gate table; docs/design-decisions.md for each ruling's D-number Status block; docs/audit/queued-rulings-20260727.md for the questions as put and the exact utterances.

What is still true and still worth reading here: the ORDERED PRECONDITION SEQUENCE (Phases 0-5), the evidence column, and the "what breaks if skipped" column. The rulings decided WHAT to do; this document is still the best statement of IN WHAT ORDER.

Also corrected by later measurement: Phase 3's intro and several row counts were written before the R6-R15 measurement pass. Where this document and docs/audit/r9-r15-final-measurements-20260727.txt disagree, the measurement capture wins -- notably R9's blast radius (TWO Stage-5 scripts, not twenty) and R14's premise (withdrawn).

Read this first. It is the answer to "are we ready to start Stage 5 error-free?" The full committee record is docs/audit/stage5-committee-raw-20260727.md; the questions needing your decision are docs/audit/queued-rulings-20260727.md; this session's own measurements are docs/audit/stage5-live-measurement-20260727.txt.

This file carries NO status tokens. Stage/gate status lives only in docs/CURRENT-STATE.md (GA-R1). Rows below are readiness assessments, not stage states.

The short answer

No -- Stage 5 would not run error-free today, and it would fail early.

That is not a bad outcome for this audit; it is the point of it. The substrate underneath is in excellent shape. What is not ready is the layer between the substrate and the deploy: the execution host, the deploy artifacts, several literals, and the gates meant to catch exactly these problems.

What is genuinely solid (measured, not read):

  • All three OpenTofu roots plan ZERO DIFF -- outer on vcloud and both inner roots on voffice1, in one session for the first time. The substrate matches its configuration exactly.
  • 18 nodes Ready, shapes EXACT to D-121 Option C, all 18 pinned MACs matching lib-hosts.sh, all 18 power_type=virsh with correct power addresses.
  • The D-134 statics are perfect: one octet per node across all six planes, 18/18.
  • 17 fabrics, exactly as the carve-residue cleanup recorded. Seven interfaces per node, zero orphans.
  • Both artifact paths genuinely serve (dc0 mirror 200 + sync attested current; dc1 proxy 200 for archive and UCA). Both D-131 forwarders answer.
  • Gauntlet ALL GREEN (81); repo-lint 0 fail; preflight red for exactly the known reasons with nothing new joined.
  • The dc1 edge's egress is MEASURED OPEN right now (probed directly from the rack, bypassing the apt proxy so a cache hit could not fake it): streams.canonical.com/juju/tools/ -> 200 -- the juju agent stream bootstrap actually needs -- plus api.snapcraft.io reachable, archive.ubuntu.com -> 200 direct, and ping 1.1.1.1 0% loss, with the default route via 10.12.64.1. This was previously carried as a record, not a measurement. It means the bootstrap window is open today (see 5.2).

What blocks it -- in the order you would hit it:

  1. Stage 5 executes on voffice1, and voffice1 is 105 commits behind main, sitting on a branch retired four days ago. overlays/vr1-dc1-vips.yaml and overlays/vr1-dc1-machines.yaml are ABSENT there and bundle.yaml is still the VR0 4-node hyperconverged layout. You would deploy the wrong topology.
  2. The openstack client is installed on neither host, and ten Stage-5/6/7 scripts invoke it.
  3. The Juju controller has nowhere to land. The dedicated 10th VM of the D-104 amendment is UNAUTHORED -- no OpenTofu resource anywhere, not merely un-applied -- and the juju-controller-vr1-dc1 tag does not exist in MAAS.
  4. ceph-osd has no device. bundle.yaml says /dev/vdb; all 18 nodes have exactly one disk, vda.
  5. Per-role MAAS tags are consumed by the machines block and authored nowhere; an unmatched tag yields no candidate machine, so allocation fails hard.
  6. Dual-stack is RULED and there is no IPv6 anywhere in the DC substrate.
  7. Stage 5's Step 4 says "follow phase-01 verbatim", and phase-01 actively REFUSES to deploy dc1. Its VIP guard greps bundle.yaml for eleven 10.12.4.x VIPs; dc1's live in an overlay on 10.12.64/68/72, so the guard evaluates 0/0/0 and takes the ABORT: VIP guard failed branch. It also carries hardcoded VR0 system_ids, a jumphost-local libvirt loop over disks that do not exist, and a 4-machine plan gate against a 9-machine bundle. After the ruled VIP extraction it will abort for dc0 too.
  8. Several gates that would have caught these cannot fail -- including repo-lint returning PASS (0 fail, 0 warn) over ZERO files on a one-character typo, which is the gate every stage close cites.

And three gate commands in the Stage-5 runbook cannot execute at all, measured against the juju actually installed (3.6.27): juju run is used for a shell command when it is the action runner (juju exec is the shell one); juju download-backup is given a backup-id when it takes a controller-side path; and the geneve gate greps ovn-central for a config key that the same runbook states does not exist. Separately, four of the five VERIFY-LIVE gates the record says Stage 5 owes have no step in the runbook at all -- including the keystone policyd-override check, which is a RULED gate.

Fifteen decisions are yours, queued in queued-rulings-20260727.md: 11 in Part A that block Stage 5 (R1-R11) and 4 in Part B that do not (R12-R15). Answer them ONE AT A TIME -- GA-R5 makes a batch answer invalid, so "yes to all" rules nothing. Everything else on this list is mechanical.

Answer R1, R2 and R3 first. They change what gets deployed, and several later questions have different right answers depending on them -- R2 in particular feeds R8, R9 and R11.

Two convergent confirmations worth noting, because they raise confidence in the whole set: the Ceph OSD blocker was found INDEPENDENTLY by two lenses using different methods (lens 2 via virsh domblklist on both racks; lens 5 via MAAS blockdevice_set on dc1's nine), and the stale-clone blocker was found independently by this session and by lens 5.

Legend

  • DONE -- verified this session, with evidence.
  • MECHANICAL -- no choice required; a session can execute it under normal gating.
  • NEEDS-RULING -- requires your decision first; see the R-number.
  • UNMEASURED -- could not be looked at. Never read as "fine".

Order matters: within a phase, items are independent; phases are sequential.

Phase 0 -- Execution environment (before anything else)

# Precondition Status Evidence If skipped
0.1 The Office1 headend clone is at main MECHANICAL ssh voffice1 ... rev-list --left-right --count origin/main...HEAD -> 105 0, HEAD 61c416e on retired branch dc-dc-g12-dc1-substrate; both dc1 overlays ABSENT; bundle.yaml sha differs and its machines block is "8"-"11" tags=openstack Deploy fails on missing overlay paths, or silently deploys VR0's 4-node hyperconverged topology into dc1
0.2 The openstack client exists on the D-128 Plane-2 host MECHANICAL command -v openstack ABSENT on vcloud AND on voffice1; voffice1 snaps are core24/juju/lxd/maas/postgresql/snapd only; ten phase-0* scripts invoke it Every step from Stage 5 Step 7 onward has no client to run
0.3 Stale remote-tracking refs pruned on voffice1 MECHANICAL that clone still lists origin/dc-dc-g12-dc1-substrate and origin/dc-dc-stage3-phase2-dc-substrate; ground truth git ls-remote --heads origin shows only main + this audit branch A naive git pull there operates on a branch that no longer exists upstream
0.4 Juju client version recorded DONE measured 3.6.27 rev 35621 3/stable on voffice1; supersedes the recorded 3.6.25. In-channel patch refresh, which D-071 permits -- policy-compliant, now corrected in CURRENT-STATE section 7 --

Phase 1 -- Decisions that change what gets deployed

Answer these before touching artifacts; several later items have different right answers depending on them.

# Precondition Status Ruling Why it is first
1.1 Ceph OSD device resolved NEEDS-RULING R1 bundle.yaml:560 osd-devices: /dev/vdb; all 18 nodes have only vda (measured twice). Option (a) re-opens both substrate roots, which currently plan zero diff
1.2 Address family resolved NEEDS-RULING R2 D-101 RULED dual-stack for both DCs this deployment; measured ZERO IPv6 on any of the 12 plane fabrics and on all 18 nodes. Addresses become as-built Keystone endpoints and cert SANs at deploy
1.3 MTU branch chosen NEEDS-RULING R3 Measured state is in NEITHER of D-101's two permitted shapes (bridges 9000, MAAS VLANs 1500, transit 1500, no MTU option anywhere). Failure appears later as tenant/geneve blackholing
1.4 Designate's stage resolved NEEDS-RULING R5 It ships in the bundle, so Stage 5 deploys it, while Stage 7's gate requires it to be newly added
1.5 HA scale-up overlay decided NEEDS-RULING R6 Stage 6's radosgw tooling is single-unit-shaped; scaling afterwards is a live-cloud change
1.6 Octavia CA scope decided NEEDS-RULING R7 Shared CA across two independent clouds vs per-DC trust domains; D-109 already rules per-DC Vault roots
1.7 Octavia lb-mgmt-net family decided NEEDS-RULING R8 Stage 6's entry condition depends on a gate Stage 5 can close without producing
1.8 vault + designate VIPs decided NEEDS-RULING R11 Both scaled 3 with cluster_count: 3 and NO vip; the checker skips any app without one

Phase 2 -- Substrate and MAAS preconditions (each a gated live mutation)

# Precondition Status Evidence If skipped
2.1 The 10th Juju-controller VM authored, applied, tagged MECHANICAL (shape ruled by the D-104 amendment) UNAUTHORED: no controller module in either substrate root, grep -rn "juju-controller-vr1-dc1" opentofu/ runbooks/ scripts/ bundle.yaml overlays/ -> zero hits; live MAAS has 5 tags and none is a controller tag; virsh list --all shows exactly 9 nodes + 1 edge per DC juju bootstrap has no target machine, or lands on one of the nine role nodes -- the outcome the D-104 amendment explicitly rejected
2.2 Per-role MAAS tags created MECHANICAL (Fork 3 already OPS-ruled) Consumed by bundle.yaml:81-97 and overlays/vr1-dc1-machines.yaml:23-39 (tags=openstack-vr1-dc1,<role>); the only tag-create in the tree makes ONE tag (reenroll-hosts.sh:177-180); none exist live Juju passes tags= to MAAS as an allocation constraint; an unmatched tag yields no candidate machine and the deploy fails hard at allocation
2.3 Reserved VIP/utility bands exist in MAAS NEEDS-RULING -> then gated mutation R4. Measured: THREE ipranges cloud-wide, all type=dynamic; ZERO reserved; .50-.99 allocatable on 12/12 DC plane subnets; phase-00-maas-standup.sh:136 refuses non-VR0 Nothing stops MAAS handing a VIP-band address to a container mid-deploy; phase-04-network-verify.sh:100 hard-fails on the missing FIP pool
2.4 OSD volumes present (only if R1 = option a) NEEDS-RULING first modules/node-vm/main.tf:104-125 declares a single vda ceph-osd finds no device
2.5 IPv6 carved into MAAS (only if R2 = dual-stack) NEEDS-RULING first D-101's own remaining open item is the un-assigned org ULA /48 + per-DC GUA carve Nothing to bind the ruled ULA legs against

Phase 3 -- Deploy-artifact and runbook corrections (all DOCFIX, no choice required)

The Stage-5 runbook is the single most defective surface found. Twenty-one corrections are listed below; following the runbook verbatim today fails or misdeploys in at least nine independent ways (3.1, 3.2, 3.4, 3.5, 3.11, 3.12, 3.13, 3.14, 3.15), and three of its named gate commands cannot execute at all against the juju that is actually installed.

# Correction Finding If skipped
3.1 Retire the DC1/DC2 namespace throughout the phase-4 runbook L1-6 Two readings, both wrong and both destructive: read literally, Stage 5 STOPs on the very DC it is meant to build ("DC2 ... BLOCKED ... STOP"); read by position, vr1-dc1 is treated as "DC1" and gets dc0's 10.12.4/8/12 VIPs
3.2 Fix Step 6's juju deploy overlay set L1-2, L6-10 It names two overlays that do not exist (octavia-pki.yaml, ${DC}-hostnames.yaml) and omits all three that constitute the ruled dc1 deploy input
3.3 Remove Step 4's "edit bundle.yaml's VIP lines" instruction L1-3 Directly reversed by ruling 3 of 2026-07-25 (bundle becomes VIP-free; per-DC values arrive via overlay)
3.4 Step 2: add the controller-tag constraint; drop the "MAAS-deployed machine" target L1-4 Bootstraps onto a role node; the READY-handoff ruling (DOCFIX-200) already removed the Deployed premise
3.5 Steps 1/2: state that juju runs on voffice1, not the jumphost L1-5 The first mutation has no binary to run on the host the runbook names
3.6 Resolve lib-net.sh's dc1 arm + sweep the 21 non-selector consumers L1-1, L7-3, L7-10 Scripts either die under set -u or silently write dc0's literals into dc1. Guard: METAL_INTERNAL_VID/IFACE must STAY unset -- D-133 abolished that stack for VR1
3.7 pre-flight-checks.sh: remove the VID-103 / br-internal assertions L3-6 Two checks contradict a RULED decision and can never pass on either VR1 DC; D-133's own affected-surfaces list omits this file
3.8 bundle.yaml's ovn-chassis MACs are dc1's, in the file documented as dc0's source of truth L3-8 Harmless for a dc1-first Stage 5; for any dc0 deploy, no local MAC matches, ovn-chassis builds no br-ex mapping, provider egress is dead on dc0 compute, and no gate fails
3.9 Teardown/rollback: Step 2's false clear and Step 3's phantom modules L6-2, L6-3 The most dangerous item in the audit. Step 2 decides "nothing to clean up in MAAS" by reading a vm_host that deliberately does not exist (per-machine virsh power, D-103/D-123), then routes you to destroy the containment VM out from under 18 live MAAS records. Step 3's scoped destroy names five modules that exist in neither root. This runbook is what you reach for during a failed Stage 5
3.10 Gap register: stale-OPEN item 17, duplicate item 19, unrecorded item 20 L6-11, L6-12, L6-13 The skill routes you to read this register FIRST; item 17 produces a false STOP, and item 20's per-DC reach verdict was never measured or recorded for either DC
3.11 Stop delegating "follow phase-01 verbatim" -- phase-4 must carry its own DC-parameterised deploy step, or phase-01 must be parameterised L5-2 phase-01 REFUSES dc1: VIP guard aborts (0/0/0), hardcoded VR0 system_ids, a jumphost-local libvirt loop over /var/lib/libvirt/images/${h}-1.qcow2 for nodes that live inside vvr1-dc1, and a 4-machine plan gate against 9 machines. This is the single largest runbook defect found
3.12 The octavia-pki generator is dc0-frozen and cannot produce a dc1 artifact L5-3 Its VIP gate hard-fails anything not matching ^10\.12\.4\.; dc1's octavia VIP is 10.12.64.57 and lives in an overlay, not in bundle.yaml. CN/SAN are ...dc0.vr0... literals. phase-4 offers reuse-vs-regenerate as a choice without noting regenerate is currently impossible -- so R7 must be answered knowing that
3.13 juju run -> juju exec in Step 12's Ceph gate L5-5 Measured on 3.6.27: juju run is the ACTION runner. As written the gate asks ceph-mon to run an action named ceph -s. A named exit-gate command that cannot execute
3.14 Step 12's geneve gate asserts a nonexistent charm option L5-6 It greps ovn-central for encap while Step 6 of the SAME runbook states no such option exists. DOCFIX-204 class: the gate must assert the observable (encap family on a live chassis), not a config key
3.15 Step 11's backup commands are wrong-shaped L5-7 Measured on 3.6.27: download-backup takes a controller-side PATH not a backup-id, -m takes a MODEL not a controller name, and create-backup already downloads by default -- so the second command is redundant. The exit gate requires "controller backup taken + stored"
3.16 Place the four missing VERIFY-LIVE gates L5-8 `grep -niE 'cluster_count\ min_size\ failure-domain\ policyd\ PO:'` on the runbook returns ZERO lines. The keystone policyd-override gate is RULED (2026-07-25), so its absence contradicts a ruled decision
3.17 Move the machines-block --dry-run AHEAD of the deploy L5-9 The only --dry-run sits in Step 6; the real deploy is Step 4. The gate that proves the dc1 retag merged runs after machines are already allocated. It also carries no pass/fail criterion
3.18 Add the apt-http-proxy model-config step L5-10 Ordering trap. apt-http-proxy appears only as prose at :31, no command. Unset, every unit apts from the internet during deploy; the D-135/D-107 egress narrowing then kills a half-built model -- and unlike bootstrap the recovery is not "re-open egress and retry" but a partially-installed cloud
3.19 Add juju add-model and define ${DC_MODEL}; invoke juju-spaces-check.sh L5-11 ${DC_MODEL} is used once and defined nowhere, so Step 6's command expands to -m "". The only add-model is phase-01's hardcoded juju add-model openstack
3.20 Make Step 2.0 CHECK-first for the MAAS credential L5-12 Measured: vr1-dc0-cred and vr1-dc1-cred already exist (credential-count: 2). Re-running the step as written mints a duplicate admin + a second key -- credential sprawl against SEC-018/-019 and D-137
3.21 Add a mid-deploy watch and failure decision tree L5-18 phase-4 references no deploy-watch.sh, no expected settle-state, no stop/fix-forward/rollback branch. The apparatus exists only in phase-01, whose expected end-state is the 4-machine VR0 model

Phase 4 -- Gate integrity (fix before relying on any gate to authorise Stage 5)

Each was demonstrated with a constructed, measured false green.

# Gate defect Finding Consequence
4.1 repo-lint PASSES over ZERO files on a one-character typo of its flag or root L4-1 (re-verified by this session) The gate whose "0-fail" every GA-R6 stage-close commit cites cannot distinguish a clean repo from an unexamined one
4.2 provider-bundle-check passes decorative HA L4-3 (re-verified: grep -rn cluster_count scripts/ tests/ -> nothing, against 20 occurrences in the overlay) 13 hacluster subs at cluster_count: 1 behind 3-unit principals produce a byte-identical PASS
4.3 provider-bundle-check treats an empty or mis-keyed overlay as a silent no-op L4-4 A truncated or typo'd Stage-5 overlay is indistinguishable from a correct one
4.4 preflight ignores every sub-gate exit code that is not 1 or 2 L4-5, L4-6 A gate that crashes (127) leaves PREFLIGHT: PASS -- clear to add-model / deploy; a checker's deliberate hard-error 2 is downgraded to WARN in P1, P2 and P3
4.5 P3 verified ZERO of 33 charm-channel pins today L4-2 juju is not on the host's PATH and a missing BINARY is folded into the same WARN as an unreachable charmhub. No committed Caracal channel pin is currently verified by any executable check, on the eve of the deploy stage
4.6 The gauntlet's harness count is compared to nothing L4-8 A renamed or deleted harness is neither run nor failed; it still prints ALL GREEN. 81 exists only as prose
4.7 cloud-assert.sh asserts a unit count exactly once L4-10 The Stage-5 completion verifier catches a sick unit and cannot catch a missing one; "Cluster ID uniform across units" passes over a SINGLE unit
4.8 G17's named check cannot fail L4-7 curl -sI exits 0 on 404/500 and the dc0 URL is an autoindex root answering 200 with nothing behind it; the dc1 half names no command at all

Phase 5 -- The deploy itself

# Precondition Status Note
5.1 Stage-5 entry basis against a red preflight NEEDS-RULING R10. Preflight exits 1 for exactly the known reasons; "PASS" is unreachable as written
5.2 Edge egress OPEN at bootstrap MEASURED OPEN 2026-07-27 Probed directly from the dc1 rack, bypassing the apt proxy so a cache hit could not fake it: streams.canonical.com/juju/tools/ -> 200 (the agent stream bootstrap needs), api.snapcraft.io reachable, archive.ubuntu.com -> 200, ping 1.1.1.1 0% loss, default route via 10.12.64.1. Previously carried as a record only. STANDING CAVEAT UNCHANGED: juju bootstrap fetches the agent stream and the juju/juju-db SNAPS pre-apt, which the dc1 apt proxy does NOT cover and D-135 items 2-3 do not exist -- so do not apply the egress narrowing until after bootstrap. Re-probe immediately before bootstrap; this is a measurement with a shelf life
5.3 juju bootstrap with the controller-tag constraint blocked on 2.1 No target machine exists yet
5.4 juju deploy with the three ruled overlays and --dc vr1-dc1 blocked on Phase 1 + 3 The runbook's current command is 3.2
5.5 Pre-allocation power-state refresh MECHANICAL -- add as a one-line check L5-15: all nine dc1 nodes read a CACHED power_state=error until maas admin machine query-power-state was run explicitly, which returned {"state":"off"} on 9/9 and cleared it. A stale error immediately before deploy reads as a hard blocker and will stall Juju provisioning. The remedy is a query-power-state refresh, not a rebuild -- worth a runbook line so nobody diagnoses a phantom fault mid-deploy

Already DONE on the Juju side (measured, so they are not re-litigated)

  • The MAAS cloud is registered: juju show-cloud vr1-maas --client -> type: maas, endpoint: http://10.10.0.20:5240/MAAS, credential-count: 2, region default.
  • Both per-DC credentials exist: juju credentials --client -> vr1-dc0-cred, vr1-dc1-cred (SEC-018 / SEC-019).
  • MAAS spaces are complete and correct for both DCs -- all six (provider-public, metal-admin, metal-internal, data-tenant, storage, replication), each carrying both DCs' /22s, matching lib-net.sh:31 exactly with none of the STALE_SPACES present.
  • juju controllers -> ERROR No controllers registered. -- expected; nothing to clean up before bootstrap.

UNMEASURED -- could not look, and not to be read as "fine"

  • The canonical piped rack-side checker invocations were run by lens 2 and PASSED on all three (dc-mirror dc0, dc-cache-proxy dc1, dc-rack-net both). This session's own first attempt ran them LOCALLY and produced a comprehensive false FAIL -- an owned error, recorded in the measurement capture.
  • Juju's actual overlay-merge behaviour for the machines: block. No Juju client exists on the jumphost, so provider-bundle-check's merge is a MIRROR of juju's documented behaviour, not juju itself. The overlay itself demands a --dry-run confirmation before deploy.
  • juju restore-backup (Stage 6 Step 9, D-104 restore drill). Repo-wide grep returns only the two runbook lines; no script or decision corroborates the subcommand. Verify the moment a Juju client is in hand -- do NOT assume it is absent.
  • Whether curl is on the racks' PATH, which both artifact checkers' only behavioural probe depends on.
  • Charm option existence for prefer-ipv6, min-cluster-size, and hacluster's cluster_count semantics. Not fetched from Charmhub this session; the overlays carry their own VERIFY-AT-DEPLOY caveats and nothing here asserts beyond them.
  • Node-side anything. All 18 nodes are powered off in Ready under the READY-handoff ruling -- that is G17's territory by construction.