# Pass 3 / W3.2 -- preflight / cloud-assert / stage-gate change table (flat topology) **Worker:** W3.2 (Phase 3 -- Tests review), container-layer-elimination pass. **Date:** 2026-08-09. **READ-ONLY.** No mutation; findings LOGGED, per SCOPE Section 7/5.5. **Inputs consumed in full:** `SCOPE-AND-EXECUTION-PLAN.md`, `pass0-admin-report.md`, `pass1-admin-report.md`, `pass2-admin-report.md`. Baseline facts carried without re-deriving: **Option 1 CONFIRMED** (flat node VMs on vcloud libvirt + one small non-hypervisor `vr1-dcN-client` VM per DC, `.8`); cross-DC handling **(a) CONFIRMED** (new vcloud-level host isolation control); MAAS region stays on `vr1-dcN-maas-01`; **THREE isolation concerns**, each real, each open, none substituting for another (pass2 Section 2): (i) cross-DC plane-bridge network adjacency -- the (a) control; (ii) the SEC-010 transit-leg FORWARD-drop successor (client VM + voffice1); (iii) the MAAS power-key blast radius (NEW, pass2 Section 2.3) -- owed artifact #11. Rack controller: **RETIRE the standalone registration** (pass2 Section 4.2(i)), D-131 **retire-with-evidence** (pass2 4.2(ii)). All riding D-143 (10.13 re-IP). Repo discipline applied throughout: assert on CONTENT not existence; an unrecognised result REFUSES, never passes silently; a checker must be provably able to FAIL (GA-R6 "checker that cannot fail is not a gate," `SKILL.md:321-333`); every gate cites `docs/tool-index.md` before an operation is named in a runbook. --- ## 1. `scripts/preflight.sh` -- per-gate change table Current gates read directly (`scripts/preflight.sh:1-386`): P1 repo-lint, P2 bundle invariants (`provider-bundle-check.py`, DC-aware since F5), P3 channel assert, P4 live pre-flight (MAAS/overlay/nodes), P5 credential matrix (D-137, hermetic tier1+tier2-local), P7 octavia PKI (HEADEND-ONLY, `creds-manifests/host-identity` `headend` binding), P8 substrate drift (outer tofu root, vcloud-only), P9 DC egress (rack-only), P6 stage-2 reminders (not run here). | Gate | Current container-layer assumption | Option-1 change | New/changed assertion (failable?) | |---|---|---|---| | **P1 repo-lint** | None (static hygiene) | None | Unchanged. L11 (`*.original.md` residue) etc. stay as-is | | **P2 provider-bundle-check.py** | DC-aware (`$DC` selects overlay set); no host-topology assumption in the bundle content itself | None to the bundle logic. Overlay literals (VIPs, machine placement) carry D-143's 10.13 addresses -- **address axis only**, not container-elim | Unchanged mechanism. `overlays/${DC}-machines.yaml` values change under D-143, tracked there not here | | **P3 channel_assert.py** | None | None | Unchanged | | **P4 pre-flight-checks.sh** | **YES -- the flagged assumption.** "live pre-flight (MAAS/overlay/nodes)" reads MAAS machine state, overlay/VIP data, and node readiness -- all of it currently reasoned about through the two-root/two-host Model-B shape implicitly (the runbooks it backs assume a `vvr1-dcN` rack host exists to enroll/carve against, per pass0 rows 4-7). Needs direct read of `pre-flight-checks.sh`'s body (not yet done this pass) to enumerate literal host/topology assumptions -- **flagged to W3.1's harness sweep**, since W3.2's charter is the gate SHAPE, not this script's full body | Node discovery/carve mechanism is **unchanged** (per-machine `power_type=virsh`, pass1 check 7) -- only the power-ADDRESS value re-derives, and that value is **BLOCKED on the concern-(iii) mitigation** (pass2 Section 2.3). P4 must not assert a specific power URI until #11 is designed | **Content assertion needed, new:** once #11 lands, P4 (or a new P-gate) should assert the LIVE power-address value in use matches the mitigation's issued form (e.g. `command=`-restricted key / wrapper), not the raw `qemu+ssh://` shape -- a regression-detector for the exact defect concern (iii) names. Failable: a stale containment-shaped URI in `lib-hosts.sh` or a live MAAS power-parameters read fails it | | **P5 creds-matrix.py --tier2** | **YES.** D-137's register rows are keyed to host CLASSES including `rack` (SEC-028's "first `rack` rows," pass0 row 8) -- Option 1 retires the rack-controller registration (pass2 4.2(i)) and re-points those rows to `vr1-dcN-client`. Matrix content is data (rows), not code, so **no script change** -- but the row SET must be updated or P5 asserts against retired host classes and either false-FAILs (row references a host that no longer exists) or false-PASSes (a client-VM credential with no row at all, SEC-022's exact failure class) | Register rows: `rack` class rows -> `client` class rows (SEC-028/-029 residencies); **NEW rows owed for concern (ii) and (iii)'s minted artifacts** (the transit-drop installer's key material if any, the power-key mitigation's restricted key) | Already failable by construction (D-137 ruling 1: hard-fails on expected-but-absent/undeclared/asymmetric). **Change is DATA not CODE**: the matrix's row source must be updated when #11/#3/#5's artifacts are built, or P5 silently stops covering the new credential-bearing host (a coverage gap, not a false pass -- but coverage gaps are how SEC-022 happened, pass2 check 6, and D-137's founding incident). Not this pass's job to edit the matrix; flagged as a delivery dependency on owed artifacts #3/#5/#11 | | **P7 octavia-pki.sh verify** | Host-bound to the **headend** via `creds-manifests/host-identity`'s `headend` row (D-109 note (b)) -- NOT container-layer-keyed at all; the PKI lives where D-138/D-128 put the deploy execution host, already re-pointed to the DC client VM by D-138 (pass0 check 3). **No change from container-elim**: P7 already reads "whichever host `host-identity` names," and that binding tracks D-138 independent of this pass | None -- confirms it is ALREADY flat-topology-correct by construction, a positive finding | Unchanged. Worth noting in the change-set as a NON-finding so it isn't re-litigated: P7's host-binding indirection is the pattern P4/P5 should imitate once concern (iii) picks a mechanism | | **P8 substrate drift** | **YES, explicitly two-root shaped by naming.** Comment says "outer tofu root" and only evaluates `opentofu/terraform.tfstate` + `opentofu/.terraform` on vcloud; guards for "not the substrate host" (voffice1, which under Model B holds the INNER root's state) as a legitimate non-evaluating state (`:284-296`). Under Option 1 there is **no inner root and no voffice1-side state** for the DC substrate -- the guard's premise (the inner root lives elsewhere) is now false, and the shared-outer + per-DC-flat root fork (pass2 recommend (B), Phase-4-ratified) means potentially THREE state files (shared outer + `vr1-dc0-flat` + `vr1-dc1-flat`), not one | Must evaluate **every root that exists post-flatten**: the shared-outer root PLUS each per-DC-flat root (if (B) is ratified), each independently, each printing which root it evaluated. The voffice1 "not the substrate host, WARN not FAIL" branch becomes dead code for the DC-substrate half (voffice1 keeps only Plane-2/MAAS-NetBox duties per the D-128 amendment, pass1 check 6) | **Change needed, failable already in shape:** extend P8 to loop over a root LIST (`opentofu/` + `opentofu/vr1-dc0-flat/` + `opentofu/vr1-dc1-flat/`, names TBD at Phase-4 ratification) instead of the single hardcoded `opentofu/` path at `:284,298`. Each root keeps the existing pending-action-FAILs / refresh-only-WARNs / no-state-here-WARNs logic (content-based, already correct) -- only the root ENUMERATION is container-elim-shaped and needs the update. Root NAMING is Phase-4's call (pass2 Section 6.2); this gate cannot be finished until that's ratified | | **P9 dc-egress-check.sh** | Runs **ON THE RACK** ("every value it uses describes the rack," `:335-336`); the rack IS `vvr1-dcN` today | The egress-testing HOST changes: under Option 1 there is no rack-as-libvirt-host to run this from. The natural new host is **`vr1-dcN-client`** (the DC-side VM with a transit+metal-admin leg, structurally the only candidate per pass2 Section 2.2's SEC-010-successor reasoning) OR a node itself. Needs a Phase-4 pick, same open item as B.5 placement (pass1 Section 7 item 2) | Mechanism (egress reachability from inside the DC) is unaffected; only the invocation-host literal in the warn/fail messaging (`:353-354`, `ssh <$DC rack>`) needs updating to name the new host once ruled. Already failable/content-based; no new assertion needed, a literal-currency edit only | | **P6 stage-2 reminders** | None (printed only) | None | Unchanged | ### 1.1 New P-gates this pass identifies as owed (not yet slotted with a number -- Phase-4/delivery mints) Three isolation controls' `--check` gates do **not** slot into the existing P1-P9 set cleanly, because preflight's charter is **pre-deploy, per-DC** (`DC=` selects one site, `:36-66`) while concern (i) and concern (iii) are **cross-DC, host-scoped, single vcloud kernel** facts that exist independent of which DC is being gated. Recommendation: - **Concern (i) [the (a) control]:** belongs as a **new stage gate at Stage 1** (pass1 Section 3's own recommendation, re-affirmed here), NOT a preflight P-gate -- preflight runs `DC=`-scoped and re-running a cross-DC assertion once per DC either duplicates work or silently only checks the last-invoked DC. A `--check` subcommand of the new artifact (SEC-010's pattern) is the right shape; preflight's P4/P9-style REFUSE-on-wrong-host guard applies (it must run on vcloud, the only host that can see both DCs' bridges). - **Concern (ii) [SEC-010 successor]:** **DOES fit as a preflight P-gate**, DC-scoped, because the transit-leg drop is inherently per-DC (one client VM + voffice1, one pair per DC) -- model it exactly on P7's shape (host-bound `--check`, REFUSE off-host, content assertion on the nftables table + interface existing, matching SEC-010's own `node_host_check()` pattern at `scripts/site-headend-install.sh:206-231`). Candidate slot: **P10** (next free preflight letter; P6 is reserved as the non-executing reminder block). - **Concern (iii) [power-key blast radius]:** belongs as **P4's dependency**, not a freestanding P-gate on its own merits (it gates whether P4's power-address literal is trustworthy) -- but it is ALSO a standing, non-preflight assertion (the mitigation must hold at all times libvirtd is up, not just at deploy time), so it needs BOTH a preflight-time check (does the live power-address match the mitigation's issued shape) and a `cloud-assert.sh`-time check (Section 2 below) that the mitigation is still enforced. None of these can be given a fully concrete `--check` body in this pass -- their underlying artifacts (#2, #3, #11) are UNDESIGNED (pass1/pass2 explicitly leave mechanism open); this table gives each a **gate home + assertion SPEC**, not an implementation (Section 3). --- ## 2. `scripts/cloud-assert.sh` -- container-layer / two-host assumptions Direct read (`scripts/cloud-assert.sh:1-294`): sections A0-A10 are ALL post-deploy juju-model/OpenStack-service behavioral checks (vault seal state, mysql cluster, OVN uniformity/chassis, compute plane, octavia LBs, keystone/magnum, conductor graft, vault-kv AppRole, HA arity). **Zero container-layer or two-host assumptions found** -- every section either runs `juju ssh` into a unit or calls the OpenStack API; none reads `vvr1-dcN`, dials qemu+ssh, or otherwise depends on the containment shape. This is consistent with pass1/pass2's finding that D-140 keeps L4 (juju/openstack deploy) a PROCEDURE layer sitting entirely above the substrate -- cloud-assert is an L4/L5 artifact and the substrate reshape underneath it is, by design, invisible to it. **What a flat-topology cloud-assert must ADD (net-new, not a modification of A0-A10):** A flat topology's actual NEW risk is that isolation the containment layer provided "for free" (pass0 Section 5: separate kernels per DC) must now be asserted explicitly. None of A0-A10 tests network/credential isolation between DCs -- they test the OpenStack control plane's OWN health, which is DC-scoped by the model they run against (`-m "$MODEL"`, one DC's juju model per invocation). **Recommend a new section, `A11: cross-DC isolation still enforced`**, appended to cloud-assert as the runtime-verification half of concern (i) and (iii) (concern (ii)'s transit-drop is a preflight/deploy-time concern, not an ongoing service-health one, though it could be echoed here too): - **A11a (concern i, network):** re-run the (a) control's `--check` from vcloud itself (cloud-assert already assumes jumphost/headend execution context for some sections, same class of host-binding as P7) -- content assertion: the nftables ruleset that blocks inter-plane/inter-DC forwarding is LOADED and its rule COUNT/hash matches the artifact's own expected state (not just "a table named X exists" -- SEC-010's own fail-open lesson: keying to an absent interface loads clean but matches nothing, `site-headend-install.sh:224`). This is the **B.7 re-run** pass1 already specifies ("the (a) control's `--check` re-run now that both DCs' planes are actually co-resident," pass1 Section 4 B.7) -- cloud-assert is the natural PERIODIC home for that re-run (post-deploy, post-restart, pre-change baseline, post-incident -- exactly cloud-assert's stated invocation points, `:6-9`), not a one-time deploy-gate. - **A11b (concern iii, credential-scope):** assert the power-key mitigation is still in effect -- e.g. if the mechanism is a `command=`-restricted key, grep the live `authorized_keys` forced-command on vcloud for the expected wrapper/allowlist rather than a bare key; if it is a polkit ACL, assert the ACL file's content matches the per-DC scoping. **Content-based, failable**: an unrestricted key or a missing ACL fails it. This directly operationalizes pass2's own warning that "a read of vcloud's LIVE polkit/libvirt config is delivery work, not asserted here" (pass2 Section 2.3) -- A11b IS that assertion, turned into a standing gate rather than a one-off read. **Framing note:** cloud-assert's own doc-comment (`:4-9`) explains it exists because "juju status is BLIND to" certain classes of defect learned from incident history (D-045/046/051/042). Concerns (i) and (iii) are the SAME shape of blind spot one layer down the stack -- `juju status` and the OpenStack API are equally blind to a host-level libvirt/nftables isolation failure. A11 is the structurally consistent place to add them, not a bolt-on. --- ## 3. The three isolation controls + rack-retirement evidence -- gate homes + failable assertion specs | # | Control | Gate home (recommendation) | Assertion spec (content-based, failable) | |---|---|---|---| | **(i)** | Cross-DC vcloud host-level isolation control (the "(a)" control) | **NEW Stage-1 gate** (pass1 Section 3's recommendation, confirmed here) -- `scripts/-check.sh` (name TBD, Phase-4/delivery), a `--check` subcommand run on vcloud; ALSO re-run at (per pass1 Section 3) each per-DC substrate apply's close AND at cloud-assert's `A11a` (Section 2 above) for ongoing verification | `nft list table inet ` must show a FORWARD-drop scoping EVERY pair of DC plane bridges (not just one -- the fail-open class SEC-010 already taught this repo: keying to one interface and silently matching nothing is not a control). Concretely: enumerate the live bridge set for both DCs' six planes (from `lib-hosts.sh`'s `NIC_PLANE_ORDER`/`BREX_PARENT_NIC` conventions, which persist per pass0 row 14), assert the ruleset denies forwarding between any bridge tagged `dc0` and any tagged `dc1`, and REFUSE (not pass) if fewer than the full plane count resolves to a live interface -- the SEC-010 fail-open lesson generalized. Ordering invariant (pass1 Section 3): must be installed + `--check`-verified **before the first flat substrate apply of EITHER root** | | **(ii)** | SEC-010 transit-leg FORWARD-drop successor | **NEW preflight gate, P10** (Section 1.1), DC-scoped, host-bound to the client VM's own `--check` (mirroring `node_host_check()`, `site-headend-install.sh:205-231`) PLUS the voffice1-side install verified the same way. **One extracted role-agnostic installer/checker for BOTH ends** (pass2 Section 2.2's resolved spec) | `nft list table inet sec010` (or its successor table name) present on the client VM, keyed to the CLIENT VM'S OWN transit interface (re-measured, not assumed -- pass2 carries forward the "NIC-naming trap" lesson: dc0's live interface was `enp1s0` not the script default `mgmt`); mirrored check on voffice1's DC-facing transit leg. REFUSE if the keyed interface does not exist (exact SEC-010 fail-open precedent) | | **(iii)** | MAAS power-key blast-radius mitigation | **Split gate, two homes:** (a) preflight P4 dependency -- before P4 asserts a power-address literal, verify it matches the mitigation's issued shape (Section 1's P4 row); (b) `cloud-assert.sh` **A11b** (Section 2) for standing/periodic re-verification, since the exposure exists any time vcloud's libvirtd is up, not only at deploy time | **What it must assert (pass2 Section 2.3's "consequence" made concrete):** that connecting with EITHER DC's region-VM power key to vcloud's `qemu:///system` endpoint CANNOT enumerate/control domains outside that DC's own set. Candidate concrete check (mechanism-dependent, Phase-4 picks the mechanism per pass2 Section 6 item 5): if a `command=`-restricted key, assert the `authorized_keys` forced-command wraps every DC-scoped invocation and REJECTS a raw `virsh -c qemu:///system list --all` from that key; if a polkit ACL, assert the ACL rule's domain-name-prefix match covers exactly that DC's `vr1-dcN-*` roster and denies the other DC's + voffice1's + the vcloud substrate's own domains. **A pass verdict must be tied to a NEGATIVE test** (the key CANNOT reach the wrong domains), not merely "the key exists" -- an existence-only check is exactly the class of non-gate GA-R6 rules out (`SKILL.md:321-333`) and exactly what SEC-012/-016's own ledger rows already flag as unaddressed ("blast radius... broader than the power verbs MAAS actually needs," `security-ledger.md` SEC-012) | | **rack-retirement evidence** | D-131 retire-with-evidence (pass2 4.2(ii)) + MAAS rack-controller decommission (pass2 owed artifact #5, amended) | **NEW step in the promoted MAAS machine-record release/delete Part-A step** (pass1 owed artifact #5) -- not a standing gate, a ONE-TIME evidence capture at build time, per pass2's framing ("the retirement EVIDENCE step... is owed artifact #13") | The dc0 migration's own proof shape is the template (`docs/changelog-20260730-dc0-region-migration.md` Item 9, cited pass2 check 4): `dig` against the fresh region's own BIND from a node, asserting `dns_servers` resolves via the DC-LOCAL region (not a remote forwarder), ANSWER section non-empty, `flags: qr rd ra`. Run once per fresh 10.13 region at build time as evidence the D-131 forwarder is genuinely unneeded, captured into the build's changelog -- NOT wired as a recurring gate (the asymmetry pass2 found, dc0-load-bearing-false / dc1-load-bearing-true, was itself only caught by exactly this kind of direct dig-test, check 4) | --- ## 4. Top risks / gaps this dimension surfaces 1. **P4 and P5 are BLOCKED on undesigned artifacts.** Neither `pre-flight-checks.sh`'s power-address assertions nor the creds-matrix row set can be finalized until concern-(iii)'s mitigation mechanism is chosen (pass2 Section 6 item 5) -- this gate work has a hard external dependency, already flagged upstream (pass2 Section 2.3 "BLOCKED on this mitigation's design"). 2. **P8 (substrate drift) is currently single-root-hardcoded** (`opentofu/` literal at `:284,298`) and will silently under-evaluate if the root-topology fork resolves to (B) shared-outer + per-DC-flat (pass2's recommendation) without this loop-extension landing first -- a real regression risk if the redeploy ships before this gate is updated. 3. **`pre-flight-checks.sh`'s own body was not read this pass** (W3.2's charter names it as the P4 delegate but the full literal-by-literal container-layer sweep of ITS internals belongs to W3.1's harness-assumption charter, per the phase-prompt split) -- flagged so Phase-3's administrator does not read this table as a complete P4 audit. 4. **No `--check` body exists yet for any of the three isolation controls** -- this table specifies WHAT each must assert (content, failable, REFUSE-on-ambiguous), not HOW; per SCOPE Section 7 the pass plans, delivery builds, each with its own `tests//run-tests.sh` harness (repo discipline, not yet started -- W3.3's charter). 5. **A11's placement in cloud-assert is a recommendation, not a ratified decision** -- Phase 4 should confirm cloud-assert (periodic/behavioral) vs. a dedicated new `isolation-assert.sh` (single-purpose, callable independent of the full A0-A10 sweep) is the right home; the case for folding in is cloud-assert's own stated charter (catching what status/API checks are blind to) and its existing periodic invocation points, not a structural necessity. --- ## 5. Verification note Direct reads this session: `scripts/preflight.sh` (full, 386 lines), `scripts/ cloud-assert.sh` (full, 294 lines), `scripts/site-headend-install.sh` (SEC-010 `node_host_check()`/writer, lines ~205-320), `scripts/dc-egress-check.sh` (host-binding + REFUSE shape, lines ~22-60,335-354), `docs/security-ledger.md` (SEC-010/-012/-016 rows in full, plus a sweep for the highest-numbered row = SEC-033, next-free SEC-034), `docs/design-decisions.md` (highest-numbered decision = D-143, next-free D-144), `docs/CURRENT-STATE.md:7786-7814` (G1-G18 in full), `.claude/skills/openstack-cloud-ops/ SKILL.md:310-369` (GA-R6 gate discipline, deploy-loop pointers). No inferred values used; every host/interface/mechanism cited as OWED or UNKNOWN where the source pass docs left it open (concern-(iii) mechanism, root naming, P10's exact number, A11's exact name) is marked as such rather than guessed. READ-ONLY; nothing executed; findings LOGGED only.