# Pass 3 / W3.2 -- preflight / cloud-assert / stage-gate change table (flat topology)

**Worker:** W3.2 (Phase 3 -- Tests review), container-layer-elimination pass.
**Date:** 2026-08-09. **READ-ONLY.** No mutation; findings LOGGED, per SCOPE Section 7/5.5.
**Inputs consumed in full:** `SCOPE-AND-EXECUTION-PLAN.md`, `pass0-admin-report.md`,
`pass1-admin-report.md`, `pass2-admin-report.md`. Baseline facts carried without
re-deriving: **Option 1 CONFIRMED** (flat node VMs on vcloud libvirt + one small
non-hypervisor `vr1-dcN-client` VM per DC, `.8`); cross-DC handling **(a) CONFIRMED**
(new vcloud-level host isolation control); MAAS region stays on `vr1-dcN-maas-01`;
**THREE isolation concerns**, each real, each open, none substituting for another
(pass2 Section 2): (i) cross-DC plane-bridge network adjacency -- the (a) control;
(ii) the SEC-010 transit-leg FORWARD-drop successor (client VM + voffice1); (iii) the
MAAS power-key blast radius (NEW, pass2 Section 2.3) -- owed artifact #11. Rack
controller: **RETIRE the standalone registration** (pass2 Section 4.2(i)), D-131
**retire-with-evidence** (pass2 4.2(ii)). All riding D-143 (10.13 re-IP).

Repo discipline applied throughout: assert on CONTENT not existence; an unrecognised
result REFUSES, never passes silently; a checker must be provably able to FAIL
(GA-R6 "checker that cannot fail is not a gate," `SKILL.md:321-333`); every gate cites
`docs/tool-index.md` before an operation is named in a runbook.

---

## 1. `scripts/preflight.sh` -- per-gate change table

Current gates read directly (`scripts/preflight.sh:1-386`): P1 repo-lint, P2 bundle
invariants (`provider-bundle-check.py`, DC-aware since F5), P3 channel assert, P4 live
pre-flight (MAAS/overlay/nodes), P5 credential matrix (D-137, hermetic tier1+tier2-local),
P7 octavia PKI (HEADEND-ONLY, `creds-manifests/host-identity` `headend` binding), P8
substrate drift (outer tofu root, vcloud-only), P9 DC egress (rack-only), P6 stage-2
reminders (not run here).

| Gate | Current container-layer assumption | Option-1 change | New/changed assertion (failable?) |
|---|---|---|---|
| **P1 repo-lint** | None (static hygiene) | None | Unchanged. L11 (`*.original.md` residue) etc. stay as-is |
| **P2 provider-bundle-check.py** | DC-aware (`$DC` selects overlay set); no host-topology assumption in the bundle content itself | None to the bundle logic. Overlay literals (VIPs, machine placement) carry D-143's 10.13 addresses -- **address axis only**, not container-elim | Unchanged mechanism. `overlays/${DC}-machines.yaml` values change under D-143, tracked there not here |
| **P3 channel_assert.py** | None | None | Unchanged |
| **P4 pre-flight-checks.sh** | **YES -- the flagged assumption.** "live pre-flight (MAAS/overlay/nodes)" reads MAAS machine state, overlay/VIP data, and node readiness -- all of it currently reasoned about through the two-root/two-host Model-B shape implicitly (the runbooks it backs assume a `vvr1-dcN` rack host exists to enroll/carve against, per pass0 rows 4-7). Needs direct read of `pre-flight-checks.sh`'s body (not yet done this pass) to enumerate literal host/topology assumptions -- **flagged to W3.1's harness sweep**, since W3.2's charter is the gate SHAPE, not this script's full body | Node discovery/carve mechanism is **unchanged** (per-machine `power_type=virsh`, pass1 check 7) -- only the power-ADDRESS value re-derives, and that value is **BLOCKED on the concern-(iii) mitigation** (pass2 Section 2.3). P4 must not assert a specific power URI until #11 is designed | **Content assertion needed, new:** once #11 lands, P4 (or a new P-gate) should assert the LIVE power-address value in use matches the mitigation's issued form (e.g. `command=`-restricted key / wrapper), not the raw `qemu+ssh://<containment-vm>` shape -- a regression-detector for the exact defect concern (iii) names. Failable: a stale containment-shaped URI in `lib-hosts.sh` or a live MAAS power-parameters read fails it |
| **P5 creds-matrix.py --tier2** | **YES.** D-137's register rows are keyed to host CLASSES including `rack` (SEC-028's "first `rack` rows," pass0 row 8) -- Option 1 retires the rack-controller registration (pass2 4.2(i)) and re-points those rows to `vr1-dcN-client`. Matrix content is data (rows), not code, so **no script change** -- but the row SET must be updated or P5 asserts against retired host classes and either false-FAILs (row references a host that no longer exists) or false-PASSes (a client-VM credential with no row at all, SEC-022's exact failure class) | Register rows: `rack` class rows -> `client` class rows (SEC-028/-029 residencies); **NEW rows owed for concern (ii) and (iii)'s minted artifacts** (the transit-drop installer's key material if any, the power-key mitigation's restricted key) | Already failable by construction (D-137 ruling 1: hard-fails on expected-but-absent/undeclared/asymmetric). **Change is DATA not CODE**: the matrix's row source must be updated when #11/#3/#5's artifacts are built, or P5 silently stops covering the new credential-bearing host (a coverage gap, not a false pass -- but coverage gaps are how SEC-022 happened, pass2 check 6, and D-137's founding incident). Not this pass's job to edit the matrix; flagged as a delivery dependency on owed artifacts #3/#5/#11 |
| **P7 octavia-pki.sh verify** | Host-bound to the **headend** via `creds-manifests/host-identity`'s `headend` row (D-109 note (b)) -- NOT container-layer-keyed at all; the PKI lives where D-138/D-128 put the deploy execution host, already re-pointed to the DC client VM by D-138 (pass0 check 3). **No change from container-elim**: P7 already reads "whichever host `host-identity` names," and that binding tracks D-138 independent of this pass | None -- confirms it is ALREADY flat-topology-correct by construction, a positive finding | Unchanged. Worth noting in the change-set as a NON-finding so it isn't re-litigated: P7's host-binding indirection is the pattern P4/P5 should imitate once concern (iii) picks a mechanism |
| **P8 substrate drift** | **YES, explicitly two-root shaped by naming.** Comment says "outer tofu root" and only evaluates `opentofu/terraform.tfstate` + `opentofu/.terraform` on vcloud; guards for "not the substrate host" (voffice1, which under Model B holds the INNER root's state) as a legitimate non-evaluating state (`:284-296`). Under Option 1 there is **no inner root and no voffice1-side state** for the DC substrate -- the guard's premise (the inner root lives elsewhere) is now false, and the shared-outer + per-DC-flat root fork (pass2 recommend (B), Phase-4-ratified) means potentially THREE state files (shared outer + `vr1-dc0-flat` + `vr1-dc1-flat`), not one | Must evaluate **every root that exists post-flatten**: the shared-outer root PLUS each per-DC-flat root (if (B) is ratified), each independently, each printing which root it evaluated. The voffice1 "not the substrate host, WARN not FAIL" branch becomes dead code for the DC-substrate half (voffice1 keeps only Plane-2/MAAS-NetBox duties per the D-128 amendment, pass1 check 6) | **Change needed, failable already in shape:** extend P8 to loop over a root LIST (`opentofu/` + `opentofu/vr1-dc0-flat/` + `opentofu/vr1-dc1-flat/`, names TBD at Phase-4 ratification) instead of the single hardcoded `opentofu/` path at `:284,298`. Each root keeps the existing pending-action-FAILs / refresh-only-WARNs / no-state-here-WARNs logic (content-based, already correct) -- only the root ENUMERATION is container-elim-shaped and needs the update. Root NAMING is Phase-4's call (pass2 Section 6.2); this gate cannot be finished until that's ratified |
| **P9 dc-egress-check.sh** | Runs **ON THE RACK** ("every value it uses describes the rack," `:335-336`); the rack IS `vvr1-dcN` today | The egress-testing HOST changes: under Option 1 there is no rack-as-libvirt-host to run this from. The natural new host is **`vr1-dcN-client`** (the DC-side VM with a transit+metal-admin leg, structurally the only candidate per pass2 Section 2.2's SEC-010-successor reasoning) OR a node itself. Needs a Phase-4 pick, same open item as B.5 placement (pass1 Section 7 item 2) | Mechanism (egress reachability from inside the DC) is unaffected; only the invocation-host literal in the warn/fail messaging (`:353-354`, `ssh <$DC rack>`) needs updating to name the new host once ruled. Already failable/content-based; no new assertion needed, a literal-currency edit only |
| **P6 stage-2 reminders** | None (printed only) | None | Unchanged |

### 1.1 New P-gates this pass identifies as owed (not yet slotted with a number -- Phase-4/delivery mints)

Three isolation controls' `--check` gates do **not** slot into the existing P1-P9 set
cleanly, because preflight's charter is **pre-deploy, per-DC** (`DC=` selects one site,
`:36-66`) while concern (i) and concern (iii) are **cross-DC, host-scoped, single vcloud
kernel** facts that exist independent of which DC is being gated. Recommendation:

- **Concern (i) [the (a) control]:** belongs as a **new stage gate at Stage 1** (pass1
  Section 3's own recommendation, re-affirmed here), NOT a preflight P-gate --
  preflight runs `DC=`-scoped and re-running a cross-DC assertion once per DC either
  duplicates work or silently only checks the last-invoked DC. A `--check` subcommand
  of the new artifact (SEC-010's pattern) is the right shape; preflight's P4/P9-style
  REFUSE-on-wrong-host guard applies (it must run on vcloud, the only host that can see
  both DCs' bridges).
- **Concern (ii) [SEC-010 successor]:** **DOES fit as a preflight P-gate**, DC-scoped,
  because the transit-leg drop is inherently per-DC (one client VM + voffice1, one pair
  per DC) -- model it exactly on P7's shape (host-bound `--check`, REFUSE off-host,
  content assertion on the nftables table + interface existing, matching SEC-010's own
  `node_host_check()` pattern at `scripts/site-headend-install.sh:206-231`). Candidate
  slot: **P10** (next free preflight letter; P6 is reserved as the non-executing
  reminder block).
- **Concern (iii) [power-key blast radius]:** belongs as **P4's dependency**, not a
  freestanding P-gate on its own merits (it gates whether P4's power-address literal is
  trustworthy) -- but it is ALSO a standing, non-preflight assertion (the mitigation
  must hold at all times libvirtd is up, not just at deploy time), so it needs BOTH a
  preflight-time check (does the live power-address match the mitigation's issued
  shape) and a `cloud-assert.sh`-time check (Section 2 below) that the mitigation is
  still enforced.

None of these can be given a fully concrete `--check` body in this pass -- their
underlying artifacts (#2, #3, #11) are UNDESIGNED (pass1/pass2 explicitly leave
mechanism open); this table gives each a **gate home + assertion SPEC**, not an
implementation (Section 3).

---

## 2. `scripts/cloud-assert.sh` -- container-layer / two-host assumptions

Direct read (`scripts/cloud-assert.sh:1-294`): sections A0-A10 are ALL post-deploy
juju-model/OpenStack-service behavioral checks (vault seal state, mysql cluster,
OVN uniformity/chassis, compute plane, octavia LBs, keystone/magnum, conductor graft,
vault-kv AppRole, HA arity). **Zero container-layer or two-host assumptions found** --
every section either runs `juju ssh` into a unit or calls the OpenStack API; none reads
`vvr1-dcN`, dials qemu+ssh, or otherwise depends on the containment shape. This is
consistent with pass1/pass2's finding that D-140 keeps L4 (juju/openstack deploy) a
PROCEDURE layer sitting entirely above the substrate -- cloud-assert is an L4/L5
artifact and the substrate reshape underneath it is, by design, invisible to it.

**What a flat-topology cloud-assert must ADD (net-new, not a modification of A0-A10):**

A flat topology's actual NEW risk is that isolation the containment layer provided
"for free" (pass0 Section 5: separate kernels per DC) must now be asserted explicitly.
None of A0-A10 tests network/credential isolation between DCs -- they test the OpenStack
control plane's OWN health, which is DC-scoped by the model they run against (`-m
"$MODEL"`, one DC's juju model per invocation). **Recommend a new section, `A11: cross-DC
isolation still enforced`**, appended to cloud-assert as the runtime-verification
half of concern (i) and (iii) (concern (ii)'s transit-drop is a preflight/deploy-time
concern, not an ongoing service-health one, though it could be echoed here too):

- **A11a (concern i, network):** re-run the (a) control's `--check` from vcloud itself
  (cloud-assert already assumes jumphost/headend execution context for some sections,
  same class of host-binding as P7) -- content assertion: the nftables ruleset that
  blocks inter-plane/inter-DC forwarding is LOADED and its rule COUNT/hash matches the
  artifact's own expected state (not just "a table named X exists" -- SEC-010's own
  fail-open lesson: keying to an absent interface loads clean but matches nothing,
  `site-headend-install.sh:224`). This is the **B.7 re-run** pass1 already specifies
  ("the (a) control's `--check` re-run now that both DCs' planes are actually
  co-resident," pass1 Section 4 B.7) -- cloud-assert is the natural PERIODIC home for
  that re-run (post-deploy, post-restart, pre-change baseline, post-incident -- exactly
  cloud-assert's stated invocation points, `:6-9`), not a one-time deploy-gate.
- **A11b (concern iii, credential-scope):** assert the power-key mitigation is still in
  effect -- e.g. if the mechanism is a `command=`-restricted key, grep the live
  `authorized_keys` forced-command on vcloud for the expected wrapper/allowlist rather
  than a bare key; if it is a polkit ACL, assert the ACL file's content matches the
  per-DC scoping. **Content-based, failable**: an unrestricted key or a missing ACL
  fails it. This directly operationalizes pass2's own warning that "a read of vcloud's
  LIVE polkit/libvirt config is delivery work, not asserted here" (pass2 Section 2.3) --
  A11b IS that assertion, turned into a standing gate rather than a one-off read.

**Framing note:** cloud-assert's own doc-comment (`:4-9`) explains it exists because
"juju status is BLIND to" certain classes of defect learned from incident history
(D-045/046/051/042). Concerns (i) and (iii) are the SAME shape of blind spot one layer
down the stack -- `juju status` and the OpenStack API are equally blind to a host-level
libvirt/nftables isolation failure. A11 is the structurally consistent place to add
them, not a bolt-on.

---

## 3. The three isolation controls + rack-retirement evidence -- gate homes + failable assertion specs

| # | Control | Gate home (recommendation) | Assertion spec (content-based, failable) |
|---|---|---|---|
| **(i)** | Cross-DC vcloud host-level isolation control (the "(a)" control) | **NEW Stage-1 gate** (pass1 Section 3's recommendation, confirmed here) -- `scripts/<name>-check.sh` (name TBD, Phase-4/delivery), a `--check` subcommand run on vcloud; ALSO re-run at (per pass1 Section 3) each per-DC substrate apply's close AND at cloud-assert's `A11a` (Section 2 above) for ongoing verification | `nft list table inet <newtable>` must show a FORWARD-drop scoping EVERY pair of DC plane bridges (not just one -- the fail-open class SEC-010 already taught this repo: keying to one interface and silently matching nothing is not a control). Concretely: enumerate the live bridge set for both DCs' six planes (from `lib-hosts.sh`'s `NIC_PLANE_ORDER`/`BREX_PARENT_NIC` conventions, which persist per pass0 row 14), assert the ruleset denies forwarding between any bridge tagged `dc0` and any tagged `dc1`, and REFUSE (not pass) if fewer than the full plane count resolves to a live interface -- the SEC-010 fail-open lesson generalized. Ordering invariant (pass1 Section 3): must be installed + `--check`-verified **before the first flat substrate apply of EITHER root** |
| **(ii)** | SEC-010 transit-leg FORWARD-drop successor | **NEW preflight gate, P10** (Section 1.1), DC-scoped, host-bound to the client VM's own `--check` (mirroring `node_host_check()`, `site-headend-install.sh:205-231`) PLUS the voffice1-side install verified the same way. **One extracted role-agnostic installer/checker for BOTH ends** (pass2 Section 2.2's resolved spec) | `nft list table inet sec010` (or its successor table name) present on the client VM, keyed to the CLIENT VM'S OWN transit interface (re-measured, not assumed -- pass2 carries forward the "NIC-naming trap" lesson: dc0's live interface was `enp1s0` not the script default `mgmt`); mirrored check on voffice1's DC-facing transit leg. REFUSE if the keyed interface does not exist (exact SEC-010 fail-open precedent) |
| **(iii)** | MAAS power-key blast-radius mitigation | **Split gate, two homes:** (a) preflight P4 dependency -- before P4 asserts a power-address literal, verify it matches the mitigation's issued shape (Section 1's P4 row); (b) `cloud-assert.sh` **A11b** (Section 2) for standing/periodic re-verification, since the exposure exists any time vcloud's libvirtd is up, not only at deploy time | **What it must assert (pass2 Section 2.3's "consequence" made concrete):** that connecting with EITHER DC's region-VM power key to vcloud's `qemu:///system` endpoint CANNOT enumerate/control domains outside that DC's own set. Candidate concrete check (mechanism-dependent, Phase-4 picks the mechanism per pass2 Section 6 item 5): if a `command=`-restricted key, assert the `authorized_keys` forced-command wraps every DC-scoped invocation and REJECTS a raw `virsh -c qemu:///system list --all` from that key; if a polkit ACL, assert the ACL rule's domain-name-prefix match covers exactly that DC's `vr1-dcN-*` roster and denies the other DC's + voffice1's + the vcloud substrate's own domains. **A pass verdict must be tied to a NEGATIVE test** (the key CANNOT reach the wrong domains), not merely "the key exists" -- an existence-only check is exactly the class of non-gate GA-R6 rules out (`SKILL.md:321-333`) and exactly what SEC-012/-016's own ledger rows already flag as unaddressed ("blast radius... broader than the power verbs MAAS actually needs," `security-ledger.md` SEC-012) |
| **rack-retirement evidence** | D-131 retire-with-evidence (pass2 4.2(ii)) + MAAS rack-controller decommission (pass2 owed artifact #5, amended) | **NEW step in the promoted MAAS machine-record release/delete Part-A step** (pass1 owed artifact #5) -- not a standing gate, a ONE-TIME evidence capture at build time, per pass2's framing ("the retirement EVIDENCE step... is owed artifact #13") | The dc0 migration's own proof shape is the template (`docs/changelog-20260730-dc0-region-migration.md` Item 9, cited pass2 check 4): `dig` against the fresh region's own BIND from a node, asserting `dns_servers` resolves via the DC-LOCAL region (not a remote forwarder), ANSWER section non-empty, `flags: qr rd ra`. Run once per fresh 10.13 region at build time as evidence the D-131 forwarder is genuinely unneeded, captured into the build's changelog -- NOT wired as a recurring gate (the asymmetry pass2 found, dc0-load-bearing-false / dc1-load-bearing-true, was itself only caught by exactly this kind of direct dig-test, check 4) |

---

## 4. Top risks / gaps this dimension surfaces

1. **P4 and P5 are BLOCKED on undesigned artifacts.** Neither `pre-flight-checks.sh`'s
   power-address assertions nor the creds-matrix row set can be finalized until
   concern-(iii)'s mitigation mechanism is chosen (pass2 Section 6 item 5) -- this
   gate work has a hard external dependency, already flagged upstream (pass2 Section
   2.3 "BLOCKED on this mitigation's design").
2. **P8 (substrate drift) is currently single-root-hardcoded** (`opentofu/` literal at
   `:284,298`) and will silently under-evaluate if the root-topology fork resolves to
   (B) shared-outer + per-DC-flat (pass2's recommendation) without this loop-extension
   landing first -- a real regression risk if the redeploy ships before this gate is
   updated.
3. **`pre-flight-checks.sh`'s own body was not read this pass** (W3.2's charter names
   it as the P4 delegate but the full literal-by-literal container-layer sweep of ITS
   internals belongs to W3.1's harness-assumption charter, per the phase-prompt split)
   -- flagged so Phase-3's administrator does not read this table as a complete P4
   audit.
4. **No `--check` body exists yet for any of the three isolation controls** -- this
   table specifies WHAT each must assert (content, failable, REFUSE-on-ambiguous), not
   HOW; per SCOPE Section 7 the pass plans, delivery builds, each with its own
   `tests/<name>/run-tests.sh` harness (repo discipline, not yet started -- W3.3's
   charter).
5. **A11's placement in cloud-assert is a recommendation, not a ratified decision** --
   Phase 4 should confirm cloud-assert (periodic/behavioral) vs. a dedicated new
   `isolation-assert.sh` (single-purpose, callable independent of the full A0-A10
   sweep) is the right home; the case for folding in is cloud-assert's own stated
   charter (catching what status/API checks are blind to) and its existing periodic
   invocation points, not a structural necessity.

---

## 5. Verification note

Direct reads this session: `scripts/preflight.sh` (full, 386 lines), `scripts/
cloud-assert.sh` (full, 294 lines), `scripts/site-headend-install.sh` (SEC-010
`node_host_check()`/writer, lines ~205-320), `scripts/dc-egress-check.sh` (host-binding
+ REFUSE shape, lines ~22-60,335-354), `docs/security-ledger.md` (SEC-010/-012/-016 rows
in full, plus a sweep for the highest-numbered row = SEC-033, next-free SEC-034),
`docs/design-decisions.md` (highest-numbered decision = D-143, next-free D-144),
`docs/CURRENT-STATE.md:7786-7814` (G1-G18 in full), `.claude/skills/openstack-cloud-ops/
SKILL.md:310-369` (GA-R6 gate discipline, deploy-loop pointers). No inferred values used;
every host/interface/mechanism cited as OWED or UNKNOWN where the source pass docs left
it open (concern-(iii) mechanism, root naming, P10's exact number, A11's exact name) is
marked as such rather than guessed. READ-ONLY; nothing executed; findings LOGGED only.
