Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / audit / container-elim-pass / pass3-w2-gates.md
@JANeumatrix JANeumatrix 5 hours ago 21 KB container-elim planning pass RAN (phases 0-4 + advisor): FINAL-PLAN + operator-report + D-144 framing (READ-ONLY)
Raw Blame History

Pass 3 / W3.2 -- preflight / cloud-assert / stage-gate change table (flat topology)

Worker: W3.2 (Phase 3 -- Tests review), container-layer-elimination pass. Date: 2026-08-09. READ-ONLY. No mutation; findings LOGGED, per SCOPE Section 7/5.5. Inputs consumed in full: SCOPE-AND-EXECUTION-PLAN.md, pass0-admin-report.md, pass1-admin-report.md, pass2-admin-report.md. Baseline facts carried without re-deriving: Option 1 CONFIRMED (flat node VMs on vcloud libvirt + one small non-hypervisor vr1-dcN-client VM per DC, .8); cross-DC handling (a) CONFIRMED (new vcloud-level host isolation control); MAAS region stays on vr1-dcN-maas-01; THREE isolation concerns, each real, each open, none substituting for another (pass2 Section 2): (i) cross-DC plane-bridge network adjacency -- the (a) control; (ii) the SEC-010 transit-leg FORWARD-drop successor (client VM + voffice1); (iii) the MAAS power-key blast radius (NEW, pass2 Section 2.3) -- owed artifact #11. Rack controller: RETIRE the standalone registration (pass2 Section 4.2(i)), D-131 retire-with-evidence (pass2 4.2(ii)). All riding D-143 (10.13 re-IP).

Repo discipline applied throughout: assert on CONTENT not existence; an unrecognised result REFUSES, never passes silently; a checker must be provably able to FAIL (GA-R6 "checker that cannot fail is not a gate," SKILL.md:321-333); every gate cites docs/tool-index.md before an operation is named in a runbook.

1. scripts/preflight.sh -- per-gate change table

Current gates read directly (scripts/preflight.sh:1-386): P1 repo-lint, P2 bundle invariants (provider-bundle-check.py, DC-aware since F5), P3 channel assert, P4 live pre-flight (MAAS/overlay/nodes), P5 credential matrix (D-137, hermetic tier1+tier2-local), P7 octavia PKI (HEADEND-ONLY, creds-manifests/host-identity headend binding), P8 substrate drift (outer tofu root, vcloud-only), P9 DC egress (rack-only), P6 stage-2 reminders (not run here).

Gate Current container-layer assumption Option-1 change New/changed assertion (failable?)
P1 repo-lint None (static hygiene) None Unchanged. L11 (*.original.md residue) etc. stay as-is
P2 provider-bundle-check.py DC-aware ($DC selects overlay set); no host-topology assumption in the bundle content itself None to the bundle logic. Overlay literals (VIPs, machine placement) carry D-143's 10.13 addresses -- address axis only, not container-elim Unchanged mechanism. overlays/${DC}-machines.yaml values change under D-143, tracked there not here
P3 channel_assert.py None None Unchanged
P4 pre-flight-checks.sh YES -- the flagged assumption. "live pre-flight (MAAS/overlay/nodes)" reads MAAS machine state, overlay/VIP data, and node readiness -- all of it currently reasoned about through the two-root/two-host Model-B shape implicitly (the runbooks it backs assume a vvr1-dcN rack host exists to enroll/carve against, per pass0 rows 4-7). Needs direct read of pre-flight-checks.sh's body (not yet done this pass) to enumerate literal host/topology assumptions -- flagged to W3.1's harness sweep, since W3.2's charter is the gate SHAPE, not this script's full body Node discovery/carve mechanism is unchanged (per-machine power_type=virsh, pass1 check 7) -- only the power-ADDRESS value re-derives, and that value is BLOCKED on the concern-(iii) mitigation (pass2 Section 2.3). P4 must not assert a specific power URI until #11 is designed Content assertion needed, new: once #11 lands, P4 (or a new P-gate) should assert the LIVE power-address value in use matches the mitigation's issued form (e.g. command=-restricted key / wrapper), not the raw qemu+ssh://<containment-vm> shape -- a regression-detector for the exact defect concern (iii) names. Failable: a stale containment-shaped URI in lib-hosts.sh or a live MAAS power-parameters read fails it
P5 creds-matrix.py --tier2 YES. D-137's register rows are keyed to host CLASSES including rack (SEC-028's "first rack rows," pass0 row 8) -- Option 1 retires the rack-controller registration (pass2 4.2(i)) and re-points those rows to vr1-dcN-client. Matrix content is data (rows), not code, so no script change -- but the row SET must be updated or P5 asserts against retired host classes and either false-FAILs (row references a host that no longer exists) or false-PASSes (a client-VM credential with no row at all, SEC-022's exact failure class) Register rows: rack class rows -> client class rows (SEC-028/-029 residencies); NEW rows owed for concern (ii) and (iii)'s minted artifacts (the transit-drop installer's key material if any, the power-key mitigation's restricted key) Already failable by construction (D-137 ruling 1: hard-fails on expected-but-absent/undeclared/asymmetric). Change is DATA not CODE: the matrix's row source must be updated when #11/#3/#5's artifacts are built, or P5 silently stops covering the new credential-bearing host (a coverage gap, not a false pass -- but coverage gaps are how SEC-022 happened, pass2 check 6, and D-137's founding incident). Not this pass's job to edit the matrix; flagged as a delivery dependency on owed artifacts #3/#5/#11
P7 octavia-pki.sh verify Host-bound to the headend via creds-manifests/host-identity's headend row (D-109 note (b)) -- NOT container-layer-keyed at all; the PKI lives where D-138/D-128 put the deploy execution host, already re-pointed to the DC client VM by D-138 (pass0 check 3). No change from container-elim: P7 already reads "whichever host host-identity names," and that binding tracks D-138 independent of this pass None -- confirms it is ALREADY flat-topology-correct by construction, a positive finding Unchanged. Worth noting in the change-set as a NON-finding so it isn't re-litigated: P7's host-binding indirection is the pattern P4/P5 should imitate once concern (iii) picks a mechanism
P8 substrate drift YES, explicitly two-root shaped by naming. Comment says "outer tofu root" and only evaluates opentofu/terraform.tfstate + opentofu/.terraform on vcloud; guards for "not the substrate host" (voffice1, which under Model B holds the INNER root's state) as a legitimate non-evaluating state (:284-296). Under Option 1 there is no inner root and no voffice1-side state for the DC substrate -- the guard's premise (the inner root lives elsewhere) is now false, and the shared-outer + per-DC-flat root fork (pass2 recommend (B), Phase-4-ratified) means potentially THREE state files (shared outer + vr1-dc0-flat + vr1-dc1-flat), not one Must evaluate every root that exists post-flatten: the shared-outer root PLUS each per-DC-flat root (if (B) is ratified), each independently, each printing which root it evaluated. The voffice1 "not the substrate host, WARN not FAIL" branch becomes dead code for the DC-substrate half (voffice1 keeps only Plane-2/MAAS-NetBox duties per the D-128 amendment, pass1 check 6) Change needed, failable already in shape: extend P8 to loop over a root LIST (opentofu/ + opentofu/vr1-dc0-flat/ + opentofu/vr1-dc1-flat/, names TBD at Phase-4 ratification) instead of the single hardcoded opentofu/ path at :284,298. Each root keeps the existing pending-action-FAILs / refresh-only-WARNs / no-state-here-WARNs logic (content-based, already correct) -- only the root ENUMERATION is container-elim-shaped and needs the update. Root NAMING is Phase-4's call (pass2 Section 6.2); this gate cannot be finished until that's ratified
P9 dc-egress-check.sh Runs ON THE RACK ("every value it uses describes the rack," :335-336); the rack IS vvr1-dcN today The egress-testing HOST changes: under Option 1 there is no rack-as-libvirt-host to run this from. The natural new host is vr1-dcN-client (the DC-side VM with a transit+metal-admin leg, structurally the only candidate per pass2 Section 2.2's SEC-010-successor reasoning) OR a node itself. Needs a Phase-4 pick, same open item as B.5 placement (pass1 Section 7 item 2) Mechanism (egress reachability from inside the DC) is unaffected; only the invocation-host literal in the warn/fail messaging (:353-354, ssh <$DC rack>) needs updating to name the new host once ruled. Already failable/content-based; no new assertion needed, a literal-currency edit only
P6 stage-2 reminders None (printed only) None Unchanged

1.1 New P-gates this pass identifies as owed (not yet slotted with a number -- Phase-4/delivery mints)

Three isolation controls' --check gates do not slot into the existing P1-P9 set cleanly, because preflight's charter is pre-deploy, per-DC (DC= selects one site, :36-66) while concern (i) and concern (iii) are cross-DC, host-scoped, single vcloud kernel facts that exist independent of which DC is being gated. Recommendation:

  • Concern (i) [the (a) control]: belongs as a new stage gate at Stage 1 (pass1 Section 3's own recommendation, re-affirmed here), NOT a preflight P-gate -- preflight runs DC=-scoped and re-running a cross-DC assertion once per DC either duplicates work or silently only checks the last-invoked DC. A --check subcommand of the new artifact (SEC-010's pattern) is the right shape; preflight's P4/P9-style REFUSE-on-wrong-host guard applies (it must run on vcloud, the only host that can see both DCs' bridges).
  • Concern (ii) [SEC-010 successor]: DOES fit as a preflight P-gate, DC-scoped, because the transit-leg drop is inherently per-DC (one client VM + voffice1, one pair per DC) -- model it exactly on P7's shape (host-bound --check, REFUSE off-host, content assertion on the nftables table + interface existing, matching SEC-010's own node_host_check() pattern at scripts/site-headend-install.sh:206-231). Candidate slot: P10 (next free preflight letter; P6 is reserved as the non-executing reminder block).
  • Concern (iii) [power-key blast radius]: belongs as P4's dependency, not a freestanding P-gate on its own merits (it gates whether P4's power-address literal is trustworthy) -- but it is ALSO a standing, non-preflight assertion (the mitigation must hold at all times libvirtd is up, not just at deploy time), so it needs BOTH a preflight-time check (does the live power-address match the mitigation's issued shape) and a cloud-assert.sh-time check (Section 2 below) that the mitigation is still enforced.

None of these can be given a fully concrete --check body in this pass -- their underlying artifacts (#2, #3, #11) are UNDESIGNED (pass1/pass2 explicitly leave mechanism open); this table gives each a gate home + assertion SPEC, not an implementation (Section 3).

2. scripts/cloud-assert.sh -- container-layer / two-host assumptions

Direct read (scripts/cloud-assert.sh:1-294): sections A0-A10 are ALL post-deploy juju-model/OpenStack-service behavioral checks (vault seal state, mysql cluster, OVN uniformity/chassis, compute plane, octavia LBs, keystone/magnum, conductor graft, vault-kv AppRole, HA arity). Zero container-layer or two-host assumptions found -- every section either runs juju ssh into a unit or calls the OpenStack API; none reads vvr1-dcN, dials qemu+ssh, or otherwise depends on the containment shape. This is consistent with pass1/pass2's finding that D-140 keeps L4 (juju/openstack deploy) a PROCEDURE layer sitting entirely above the substrate -- cloud-assert is an L4/L5 artifact and the substrate reshape underneath it is, by design, invisible to it.

What a flat-topology cloud-assert must ADD (net-new, not a modification of A0-A10):

A flat topology's actual NEW risk is that isolation the containment layer provided "for free" (pass0 Section 5: separate kernels per DC) must now be asserted explicitly. None of A0-A10 tests network/credential isolation between DCs -- they test the OpenStack control plane's OWN health, which is DC-scoped by the model they run against (-m "$MODEL", one DC's juju model per invocation). Recommend a new section, A11: cross-DC isolation still enforced, appended to cloud-assert as the runtime-verification half of concern (i) and (iii) (concern (ii)'s transit-drop is a preflight/deploy-time concern, not an ongoing service-health one, though it could be echoed here too):

  • A11a (concern i, network): re-run the (a) control's --check from vcloud itself (cloud-assert already assumes jumphost/headend execution context for some sections, same class of host-binding as P7) -- content assertion: the nftables ruleset that blocks inter-plane/inter-DC forwarding is LOADED and its rule COUNT/hash matches the artifact's own expected state (not just "a table named X exists" -- SEC-010's own fail-open lesson: keying to an absent interface loads clean but matches nothing, site-headend-install.sh:224). This is the B.7 re-run pass1 already specifies ("the (a) control's --check re-run now that both DCs' planes are actually co-resident," pass1 Section 4 B.7) -- cloud-assert is the natural PERIODIC home for that re-run (post-deploy, post-restart, pre-change baseline, post-incident -- exactly cloud-assert's stated invocation points, :6-9), not a one-time deploy-gate.
  • A11b (concern iii, credential-scope): assert the power-key mitigation is still in effect -- e.g. if the mechanism is a command=-restricted key, grep the live authorized_keys forced-command on vcloud for the expected wrapper/allowlist rather than a bare key; if it is a polkit ACL, assert the ACL file's content matches the per-DC scoping. Content-based, failable: an unrestricted key or a missing ACL fails it. This directly operationalizes pass2's own warning that "a read of vcloud's LIVE polkit/libvirt config is delivery work, not asserted here" (pass2 Section 2.3) -- A11b IS that assertion, turned into a standing gate rather than a one-off read.

Framing note: cloud-assert's own doc-comment (:4-9) explains it exists because "juju status is BLIND to" certain classes of defect learned from incident history (D-045/046/051/042). Concerns (i) and (iii) are the SAME shape of blind spot one layer down the stack -- juju status and the OpenStack API are equally blind to a host-level libvirt/nftables isolation failure. A11 is the structurally consistent place to add them, not a bolt-on.

3. The three isolation controls + rack-retirement evidence -- gate homes + failable assertion specs

# Control Gate home (recommendation) Assertion spec (content-based, failable)
(i) Cross-DC vcloud host-level isolation control (the "(a)" control) NEW Stage-1 gate (pass1 Section 3's recommendation, confirmed here) -- scripts/<name>-check.sh (name TBD, Phase-4/delivery), a --check subcommand run on vcloud; ALSO re-run at (per pass1 Section 3) each per-DC substrate apply's close AND at cloud-assert's A11a (Section 2 above) for ongoing verification nft list table inet <newtable> must show a FORWARD-drop scoping EVERY pair of DC plane bridges (not just one -- the fail-open class SEC-010 already taught this repo: keying to one interface and silently matching nothing is not a control). Concretely: enumerate the live bridge set for both DCs' six planes (from lib-hosts.sh's NIC_PLANE_ORDER/BREX_PARENT_NIC conventions, which persist per pass0 row 14), assert the ruleset denies forwarding between any bridge tagged dc0 and any tagged dc1, and REFUSE (not pass) if fewer than the full plane count resolves to a live interface -- the SEC-010 fail-open lesson generalized. Ordering invariant (pass1 Section 3): must be installed + --check-verified before the first flat substrate apply of EITHER root
(ii) SEC-010 transit-leg FORWARD-drop successor NEW preflight gate, P10 (Section 1.1), DC-scoped, host-bound to the client VM's own --check (mirroring node_host_check(), site-headend-install.sh:205-231) PLUS the voffice1-side install verified the same way. One extracted role-agnostic installer/checker for BOTH ends (pass2 Section 2.2's resolved spec) nft list table inet sec010 (or its successor table name) present on the client VM, keyed to the CLIENT VM'S OWN transit interface (re-measured, not assumed -- pass2 carries forward the "NIC-naming trap" lesson: dc0's live interface was enp1s0 not the script default mgmt); mirrored check on voffice1's DC-facing transit leg. REFUSE if the keyed interface does not exist (exact SEC-010 fail-open precedent)
(iii) MAAS power-key blast-radius mitigation Split gate, two homes: (a) preflight P4 dependency -- before P4 asserts a power-address literal, verify it matches the mitigation's issued shape (Section 1's P4 row); (b) cloud-assert.sh A11b (Section 2) for standing/periodic re-verification, since the exposure exists any time vcloud's libvirtd is up, not only at deploy time What it must assert (pass2 Section 2.3's "consequence" made concrete): that connecting with EITHER DC's region-VM power key to vcloud's qemu:///system endpoint CANNOT enumerate/control domains outside that DC's own set. Candidate concrete check (mechanism-dependent, Phase-4 picks the mechanism per pass2 Section 6 item 5): if a command=-restricted key, assert the authorized_keys forced-command wraps every DC-scoped invocation and REJECTS a raw virsh -c qemu:///system list --all from that key; if a polkit ACL, assert the ACL rule's domain-name-prefix match covers exactly that DC's vr1-dcN-* roster and denies the other DC's + voffice1's + the vcloud substrate's own domains. A pass verdict must be tied to a NEGATIVE test (the key CANNOT reach the wrong domains), not merely "the key exists" -- an existence-only check is exactly the class of non-gate GA-R6 rules out (SKILL.md:321-333) and exactly what SEC-012/-016's own ledger rows already flag as unaddressed ("blast radius... broader than the power verbs MAAS actually needs," security-ledger.md SEC-012)
rack-retirement evidence D-131 retire-with-evidence (pass2 4.2(ii)) + MAAS rack-controller decommission (pass2 owed artifact #5, amended) NEW step in the promoted MAAS machine-record release/delete Part-A step (pass1 owed artifact #5) -- not a standing gate, a ONE-TIME evidence capture at build time, per pass2's framing ("the retirement EVIDENCE step... is owed artifact #13") The dc0 migration's own proof shape is the template (docs/changelog-20260730-dc0-region-migration.md Item 9, cited pass2 check 4): dig against the fresh region's own BIND from a node, asserting dns_servers resolves via the DC-LOCAL region (not a remote forwarder), ANSWER section non-empty, flags: qr rd ra. Run once per fresh 10.13 region at build time as evidence the D-131 forwarder is genuinely unneeded, captured into the build's changelog -- NOT wired as a recurring gate (the asymmetry pass2 found, dc0-load-bearing-false / dc1-load-bearing-true, was itself only caught by exactly this kind of direct dig-test, check 4)

4. Top risks / gaps this dimension surfaces

  1. P4 and P5 are BLOCKED on undesigned artifacts. Neither pre-flight-checks.sh's power-address assertions nor the creds-matrix row set can be finalized until concern-(iii)'s mitigation mechanism is chosen (pass2 Section 6 item 5) -- this gate work has a hard external dependency, already flagged upstream (pass2 Section 2.3 "BLOCKED on this mitigation's design").
  2. P8 (substrate drift) is currently single-root-hardcoded (opentofu/ literal at :284,298) and will silently under-evaluate if the root-topology fork resolves to (B) shared-outer + per-DC-flat (pass2's recommendation) without this loop-extension landing first -- a real regression risk if the redeploy ships before this gate is updated.
  3. pre-flight-checks.sh's own body was not read this pass (W3.2's charter names it as the P4 delegate but the full literal-by-literal container-layer sweep of ITS internals belongs to W3.1's harness-assumption charter, per the phase-prompt split) -- flagged so Phase-3's administrator does not read this table as a complete P4 audit.
  4. No --check body exists yet for any of the three isolation controls -- this table specifies WHAT each must assert (content, failable, REFUSE-on-ambiguous), not HOW; per SCOPE Section 7 the pass plans, delivery builds, each with its own tests/<name>/run-tests.sh harness (repo discipline, not yet started -- W3.3's charter).
  5. A11's placement in cloud-assert is a recommendation, not a ratified decision -- Phase 4 should confirm cloud-assert (periodic/behavioral) vs. a dedicated new isolation-assert.sh (single-purpose, callable independent of the full A0-A10 sweep) is the right home; the case for folding in is cloud-assert's own stated charter (catching what status/API checks are blind to) and its existing periodic invocation points, not a structural necessity.

5. Verification note

Direct reads this session: scripts/preflight.sh (full, 386 lines), scripts/ cloud-assert.sh (full, 294 lines), scripts/site-headend-install.sh (SEC-010 node_host_check()/writer, lines ~205-320), scripts/dc-egress-check.sh (host-binding

  • REFUSE shape, lines ~22-60,335-354), docs/security-ledger.md (SEC-010/-012/-016 rows in full, plus a sweep for the highest-numbered row = SEC-033, next-free SEC-034), docs/design-decisions.md (highest-numbered decision = D-143, next-free D-144), docs/CURRENT-STATE.md:7786-7814 (G1-G18 in full), .claude/skills/openstack-cloud-ops/ SKILL.md:310-369 (GA-R6 gate discipline, deploy-loop pointers). No inferred values used; every host/interface/mechanism cited as OWED or UNKNOWN where the source pass docs left it open (concern-(iii) mechanism, root naming, P10's exact number, A11's exact name) is marked as such rather than guessed. READ-ONLY; nothing executed; findings LOGGED only.