diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 6d31ab2..26dd378 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2043,6 +2043,42 @@ workspaces hold identically-named basenames and a shared scope would collapse them into one namespace where a missing dc1 file is indistinguishable from a present dc0 one, reintroducing precisely the blindness F2 had just removed. +- **P4 IS DC-AWARE, ITEM 3.7 CLOSED, AND R9's `lib-net` RESIDUE POPULATED, 2026-07-29.** + `DC` references in `pre-flight-checks.sh` went from **1 (a comment) to 45**. Both selectors + are called once after sourcing, CHECK 1 reads `overlays/${DC}-vips.yaml`, and CHECK 2/4 + iterate the DC's own `HOSTS` resolving system_ids by **pinned boot MAC** on VR1 -- which was + the cause of 8 of P4's 11 fatals (it had been probing VR0's `openstack0..3`). Role-node + scoping REUSES `maas-role-tags.sh`'s existing D-134 utility-band rule rather than defining a + second copy; the D-104 controller at `.5` is named and excluded with its reason, its MAAS + record still asserted, and deliberately NOT asserted `Ready` since it is legitimately + `Deployed` after bootstrap. An octet in neither band REFUSES. + **The two VR0-inherited expectations were RETIRED, NOT DELETED:** VID 103 (superseded by + D-133) and the metal-admin gateway pin (superseded by D-134 -- measured, `10.12.8.1` / + `10.12.68.1` are held by nothing, so the PIN was the defect). A spurious gateway still warns + and a missing provider-public gateway still fails, so the checks kept their teeth. + **Item 3.7's guards are not skips:** each guarded branch asserts D-133's own invariant + instead -- VID must be `0` (untagged) on VR1, and the metal-internal link must be MAAS + `type == physical`, which asserts the flat carve itself and is falsifiable (a bridge or VLAN + there fails). **An UNREADABLE VLAN now REFUSES** rather than satisfying the untagged branch: + `.vlan.vid // 0` had rendered "missing" and "untagged" identically, a false-green. + **F5 fixed:** `DC` is validated against the D-119 token set and REFUSES before any gate runs, + is exported, and appears in both the header and the verdict line -- while keeping the + `PREFLIGHT: ` token contiguous, which ten existing cases grep. + **R9 residue:** the dc1 arm is populated (`KEYSTONE_VIP_DEFAULT=10.12.64.50`), and the D-133 + guard HELD -- `METAL_INTERNAL_VID`/`IFACE` stay unset, now pinned for both variables on both + VR1 DCs, verified independently here. Two dc-selector assertions were REPLACED with the new + invariant and the replacement explained in-file, never deleted to go green. Honest framing + retained: the arm is **hand-authored with a drift check, and R9's generator half is still + owed**. Harnesses `pre-flight-checks` **31/31** (new), `preflight` **26/26** (was 16), + `dc-selector` **69 checks** (was 45). + **CORRECTED FROM PREDICTION TO MEASUREMENT, and it changes the remaining work:** the four + other DC-dependent consumers all carry `set -u`, so adding the selector ALONE to + `phase-04-network-{create,verify}.sh` makes them **ABORT** with `PLANE_GW[...]: unbound + variable` on dc1 (reproduced) -- the hardcoded `PROVIDER_CIDR="10.12.4.0/22"` must be derived + in the SAME edit. `vault-kv-health.sh` and `phase-03-admin-openrc.sh` need only the selector. + **UNRULED AND NOT PICKED:** whether dc1's public catalog endpoint becomes the v6 GUA leg + under R2/`prefer-ipv6` is ruling-shaped; `KEYSTONE_VIP_DEFAULT` mirrors dc0's v4 provider-leg + definition pending that. **F3 -- both `~/octavia-pki/` and `overlays/octavia-pki.yaml` are ABSENT here** (existence checked, no contents read). So this is generation FROM SCRATCH for both DCs: there is nothing to reuse, which retires the reuse-vs-regenerate choice diff --git a/scripts/lib-net.sh b/scripts/lib-net.sh index a8d3545..72d3972 100644 --- a/scripts/lib-net.sh +++ b/scripts/lib-net.sh @@ -33,7 +33,10 @@ # Names that MUST be gone after the D-052 / D-053 cutover (deploy fails or mis-binds if any reappear). STALE_SPACES=( provider metal data fabric-data lbaas provider-vip ) -# Gateways: only provider-public and metal-admin route; the other four are gw=none. +# Gateways. VR0 AS-BUILT: provider-public and metal-admin both route; the other four +# are gw=none. 10.12.8.1 is VR0's REAL as-built value (phase-00-maas-standup.sh:119) and +# stays here for the vr0-dc0 arm. It does NOT hold for either VR1 DC -- see the per-arm +# overrides below (D-134; measured 2026-07-27). declare -A PLANE_GW=( ["10.12.4.0/22"]="10.12.4.1" ["10.12.8.0/22"]="10.12.8.1" ) # The four non-API, non-PXE planes whose host NICs MAAS must have provisioned. @@ -118,20 +121,42 @@ # real, measured values. Explicit no-op, not a guess. ;; vr1-dc0) - : # VR1's FIRST DC. Same values as vr0-dc0, but for a DIFFERENT REASON: - # D-101 rules that VR1's first DC INHERITS VR0 DC0's v4 layout UNCHANGED. - # This is a SEPARATE case arm ON PURPOSE. The day D-101 stops holding, - # this arm diverges and vr0-dc0's must not. Folding the two arms together - # would encode the coincidence and lose the reason -- and the reason is - # the thing that changes (VR1 DC0 gains its own v6 per the family matrix). - # - # D-133 CAVEAT (2026-07-23): the inherited METAL_INTERNAL_VID=103 / - # METAL_INTERNAL_IFACE=br-internal are VR0 CARVE ARTIFACTS ONLY. VR1 DC - # nodes carve every plane FLAT on its dedicated NIC (D-133 ADOPTED; - # metal-internal = untagged on enp3s0). Do NOT consume these two values - # for any VR1 carve/netplan surface -- they survive here solely because - # the D-101 inheritance is value-for-value and VR0 tooling still reads - # them under the vr0-dc0 arm. + # VR1's FIRST DC. Same PLANE values as vr0-dc0, but for a DIFFERENT REASON: + # D-101 rules that VR1's first DC INHERITS VR0 DC0's v4 layout UNCHANGED. + # This is a SEPARATE case arm ON PURPOSE. The day D-101 stops holding, + # this arm diverges and vr0-dc0's must not. Folding the two arms together + # would encode the coincidence and lose the reason -- and the reason is + # the thing that changes (VR1 DC0 gains its own v6 per the family matrix). + # + # THAT DAY ARRIVED for two groups (executed 2026-07-29). The arm is no longer + # a pure no-op: D-101's inheritance is value-for-value only where the value is + # still TRUE for VR1. + # + # (1) METAL_INTERNAL_VID=103 / METAL_INTERNAL_IFACE=br-internal are VR0 CARVE + # ARTIFACTS ONLY. D-133 ADOPTED: VR1 DC nodes carve every plane FLAT on its + # dedicated NIC, metal-internal untagged on enp3s0, "no br-metal.103 / + # br-internal stack exists in VR1. VID 103 is a VR0-only fact." D-133's own + # "Affected surfaces" list names THIS arm as owing an explicit unset, WITH its + # harness, at the carve delivery -- that is this. Leaving them set made every + # DC-aware consumer inherit an expectation that can never pass on a VR1 DC + # (pre-flight-checks CHECK 3's VID assertion was exactly that failure). + # UNSET, not re-pointed: they must fail loud under `set -u`, never quietly + # carry a substitute for a fact that does not exist. Mirrors the vr1-dc1 arm. + unset METAL_INTERNAL_VID METAL_INTERNAL_IFACE + # (2) metal-admin has NO ROUTER in either VR1 DC. MEASURED 2026-07-27 -- cited by + # PHRASE, not by line offset: grep "metal-admin has no router" in + # docs/audit/stage5-committee-raw-20260727.md, and "held by NOTHING" in + # docs/CURRENT-STATE.md, which reconciles it. + # `maas admin subnets read` returns gateway + # `none` for 10.12.8.0/22, and `ping 10.12.8.1` from the dc0 rack is 100% + # loss with an INCOMPLETE ARP entry -- the address is held by NOTHING. + # Control: 10.12.4.1 answers 0% loss and carries the rack default route. + # MAAS is RIGHT and the inherited pin was the defect. Consistent with the + # D-134 carve, which rules `.1` gateways for the PROVIDER subnets only; + # VR1's edges carry a provider-public LAN leg and nothing on metal-admin. + # A whole-array assignment RESETS an associative array in bash, so this + # drops the 10.12.8.0/22 key rather than adding to it. + PLANE_GW=( ["10.12.4.0/22"]="10.12.4.1" ) ;; vr1-dc1) # VR1's SECOND DC -- literals RATIFIED by the D-124 AMENDMENT (2026-07-21) @@ -152,9 +177,17 @@ ["10.12.80.0/22"]="storage" ["10.12.84.0/22"]="replication" ) - # .1 site gateways on the two routed planes (D-120 convention; provider-public - # gw 10.12.64.1 is the ruled edge LAN gateway, D-124 amendment). - PLANE_GW=( ["10.12.64.0/22"]="10.12.64.1" ["10.12.68.0/22"]="10.12.68.1" ) + # provider-public gw 10.12.64.1 is the RULED edge LAN gateway (D-124 amendment). + # metal-admin's 10.12.68.1 was REMOVED 2026-07-29: it came from the D-120 ".1 site + # gateway" convention, not from a ruling or a measurement, and MEASUREMENT refutes + # it -- `maas admin subnets read` returns gateway `none` for 10.12.68.0/22 and + # `ping 10.12.68.1` from the dc1 rack is 100% loss with a FAILED ARP entry + # (grep "metal-admin has no router" in docs/audit/stage5-committee-raw-20260727.md; + # docs/CURRENT-STATE.md records the dc0 twin -- grep "held by NOTHING" -- and + # predicts this one "will fail identically"). D-134's + # carve rules `.1` gateways for the PROVIDER subnets only. Same change as the + # vr1-dc0 arm above, for the same reason. + PLANE_GW=( ["10.12.64.0/22"]="10.12.64.1" ) DATA_PLANE_CIDRS=( "10.12.72.0/22" "10.12.76.0/22" "10.12.80.0/22" "10.12.84.0/22" ) METAL_INTERNAL_CIDR="10.12.72.0/22" # FIP pool RULED 2026-07-27 (GA-R5, operator: "Rule dc1 FIP pool @@ -165,16 +198,48 @@ # scripts/dc-plane-ipam.sh reserve and asserted by phase-04-network-verify.sh:100. FIP_POOL_START="10.12.65.0" FIP_POOL_END="10.12.67.254" - # The REMAINING OpenStack-layer values (VIP bands, keystone VIP) and the VR0 metal - # fabric facts (VLAN id, bridge iface) are still NOT ruled/measured for vr1-dc1. - # UNSET so any use fails loud under `set -u` instead of silently borrowing another - # DC's addresses. Per R9 (D-119 amendment) the VIP group becomes GENERATED from - # overlays/vr1-dc1-vips.yaml -- so it must NOT be hand-set here; and per D-133 the - # METAL_INTERNAL_VID/IFACE pair must STAY unset, since that VLAN stack is retired - # for VR1 and a generator repopulating it would silently reintroduce it. + # --- R9 group: the OpenStack-layer VIP literals (item 3.6 residue) ---------- + # POPULATED 2026-07-29. These were unset "until ruled"; R9 (the D-119 AMENDMENT, + # RULED 2026-07-27) ruled them, and overlays/vr1-dc1-vips.yaml is the ratified, + # deploy-input copy the ruling names as the single AUTHORED source. Leaving them + # unset was costing more than it bought: `lib_net_select_dc vr1-dc1` left + # KEYSTONE_VIP_DEFAULT unbound, so every selector-CALLER that touched it died + # under `set -u` mid-run (measured 2026-07-29), while the non-callers silently + # used dc0's 10.12.4.50 -- the silent half being the dangerous half, exactly as + # R9's "MEASURED BASIS" paragraph states. + # + # HONEST STATUS -- do not read these as "generated": R9's ruled shape is DERIVE + # + drift-check, and only the DRIFT half exists. The values below are HAND-COPIED + # from overlays/vr1-dc1-vips.yaml (read 2026-07-29) and pinned against it by + # tests/dc-selector/run-tests.sh, which re-parses the overlay and fails on + # divergence in either direction. R9's GENERATOR half is still OWED; when it + # lands it replaces this block and the harness becomes its drift gate. + # VIP_PREFIX_* the three v4 legs of every `vip:` line, in ruled column + # order provider-public / metal-admin / metal-internal. + # VIP_COUNT_EXPECT 13 = the `vip:` line count in that overlay (R11's ruled + # 11 -> 13, vault .61 + designate .62). + # KEYSTONE_VIP_DEFAULT keystone's PROVIDER-PUBLIC v4 leg (.50). Same + # definition as dc0's own KEYSTONE_VIP_DEFAULT=10.12.4.50: + # bundle.yaml:11-13 (B5) makes the public catalog endpoint the + # provider VIP. NOT settled here: R2 made these dual-family and + # the overlay carries `prefer-ipv6: true`, so whether the public + # catalog endpoint ultimately becomes the v6 GUA leg is a + # RULING-shaped question, raised and left open on purpose. + # VIP_OCTET_MIN/MAX are NOT unset by this arm and stay at the R11 band 50-99. + VIP_PREFIX_PROVIDER="10.12.64" + VIP_PREFIX_ADMIN="10.12.68" + VIP_PREFIX_INTERNAL="10.12.72" + VIP_COUNT_EXPECT=13 + KEYSTONE_VIP_DEFAULT="10.12.64.50" + # --- NOT in the R9 group, and must STAY unset ------------------------------ + # The VR0 metal fabric facts (VLAN id, bridge iface) are not "unruled" -- D-133 + # ABOLISHED them for VR1 (flat per-NIC carve; metal-internal untagged on enp3s0). + # They genuinely do not exist, so they must fail loud under `set -u` rather than + # carry a substitute. R9's own GUARD paragraph says the same in terms: a + # derivation that populated all nine would silently reintroduce a retired stack. + # Consumers assert D-133's invariant instead (untagged VLAN + a `physical` + # interface type) -- see scripts/pre-flight-checks.sh CHECK 3 / CHECK 2. unset METAL_INTERNAL_VID METAL_INTERNAL_IFACE - unset VIP_PREFIX_PROVIDER VIP_PREFIX_ADMIN VIP_PREFIX_INTERNAL VIP_COUNT_EXPECT - unset KEYSTONE_VIP_DEFAULT ;; dc0|dc1|dc2) echo "FAIL: bare '$dc' is RETIRED (D-119). It was AMBIGUOUS ACROSS REGIONS: 'dc0' meant VR0's live DC0 here, but VR1's FIRST DC in the NetBox importer. Use the region-qualified selector: vr0-dc0 | vr1-dc0 | vr1-dc1" >&2 diff --git a/scripts/pre-flight-checks.sh b/scripts/pre-flight-checks.sh index 1fd8095..b9a548d 100644 --- a/scripts/pre-flight-checks.sh +++ b/scripts/pre-flight-checks.sh @@ -5,14 +5,26 @@ # (or set REPO=) BEFORE 'juju add-model' / 'juju deploy'. Surfaces issues that # would cause the deploy to fail or mis-bind during settle. # -# Covers: +# DC-AWARE since 2026-07-29 (F5 / readiness item 3.7). Set DC to pick the datacenter; +# it defaults to vr1-dc0, and scripts/preflight.sh EXPORTS its own validated selection +# so P4 always measures the same DC the rest of preflight did. +# DC=vr1-dc1 bash scripts/pre-flight-checks.sh +# Before this, every check below was frozen on VR0: it probed hosts openstack0..3 that +# exist in no VR1 DC, asserted VR0's VLAN-103 metal fabric that D-133 abolished for VR1, +# and read a single unscoped octavia-pki overlay. On a VR1 DC it produced fatals that +# described the CHECKER, not the cloud. +# +# Covers (every item resolved for the SELECTED DC): # - repo HEAD / cleanliness (informational) -# - octavia-pki overlay sanity (5 keys + ASCII; no key material printed) -# - VIP validator (provider/admin/internal columns, aligned, .50-.99; a v4 triple -# or an R2 dual-family sextet). Deep v6 checking is provider-bundle-check's job. -# - MAAS six-plane layout resolved BY CIDR (id/vid/gw/dns; metal-internal VID 103) -# - per-host data/storage NIC links resolved BY CIDR (metal-internal on br-internal) -# - the four KVM nodes Ready / power state +# - per-DC octavia-pki overlay sanity (5 keys + ASCII; no key material printed) +# - VIP validator over the per-DC overlay (provider/admin/internal columns, aligned, +# .50-.99; a v4 triple or an R2 dual-family sextet). Deep v6 checking is +# provider-bundle-check's job. +# - MAAS six-plane layout resolved BY CIDR (id/vid/gw/dns), incl. the metal-internal +# tagging invariant: VR0's VID 103, or D-133's untagged carve on a VR1 DC +# - per-node data/storage NIC links resolved BY CIDR, incl. D-133's flat-carve +# invariant (VR0: br-internal; VR1: a PHYSICAL interface, no bridge/VLAN stack) +# - the DC's OpenStack role nodes Ready / power state # # NOT covered here (by design): # - juju per-model SPACE names -> scripts/juju-spaces-check.sh (runs AFTER add-model) @@ -54,6 +66,79 @@ cd "$REPO" || { fail "cannot cd to REPO=$REPO"; finish; } # --------------------------------------------------------------------------- +# $DC selector (DOCFIX-166 wiring pattern, as used by carve-host-interfaces.sh:48-67). +# Called ONCE, immediately after sourcing both libs and before ANY flat variable is +# read -- that is the libs' own documented contract, and both refuse a second, +# different selection in the same shell. +# +# The default matches scripts/preflight.sh's so the two agree when P4 is run +# standalone. An unknown or RETIRED token is REFUSED by the selectors themselves; we +# turn that into a fatal + summary rather than letting `set -e` kill the run with no +# verdict, because "could not evaluate" must never look like "nothing found". +DC="${DC:-vr1-dc0}" +hdr "DC selection" +if ! lib_net_select_dc "$DC"; then + fail "lib-net refused DC='$DC' -- nothing below can be measured against an unknown DC" + finish +fi +if ! lib_hosts_select_dc "$DC"; then + fail "lib-hosts refused DC='$DC' -- nothing below can be measured against an unknown DC" + finish +fi +pass "gating DC=$DC (planes ${PLANE_CIDRS[0]} .. ${PLANE_CIDRS[5]}; ${#HOSTS[@]} node(s))" + +# Which nodes does the DEPLOY actually bind to? Not all of them. +# D-134 AMENDMENT (2026-07-29) rules octets .4-.49 the per-DC UTILITY band -- "per-DC +# INFRASTRUCTURE the OpenStack nodes consume, NOT OpenStack role nodes" -- which is +# where the D-104 dedicated Juju controller sits at .5. It carries the +# `juju-controller-` tag and NONE of the role tags precisely so no OpenStack +# constraint can grab it (D-104 amendment), so bundle placement, the six-plane node +# carve and "Ready before deploy" are all statements about the ROLE nodes only. +# scripts/maas-role-tags.sh:70-98 classifies by exactly this band; reusing its rule +# keeps one definition of "role node" in the repo instead of two. +# VR0 is EXEMPT: openstack0..3 sit at .40-.43 under the D-052 index, which predates +# the D-134 bands entirely -- classifying them by band would silently drop the whole +# VR0 fleet into "utility" and leave this gate asserting nothing. +# +# Classified ONCE, here, into NODE_CLASS -- not re-derived per check. CHECK 2 and +# CHECK 4 both need the answer, and a per-check helper emitted the refusal TWICE for +# an unclassifiable node and then printed a "utility-band" note contradicting the +# refusal that had just fired. +D134_UTILITY_LO=4 +D134_UTILITY_HI=49 +declare -A NODE_CLASS=() +for _h in "${HOSTS[@]}"; do + _o="${HOST_OCTET[$_h]}" + case "$DC" in + vr0-dc0) NODE_CLASS[$_h]="role" ;; + *) + if [ "$_o" -ge "$D134_UTILITY_LO" ] && [ "$_o" -le "$D134_UTILITY_HI" ]; then NODE_CLASS[$_h]="utility" + elif [ "$_o" -ge 100 ]; then NODE_CLASS[$_h]="role" + else + # Neither band. REFUSE rather than guess which side it falls on, and say so + # once. The node is then asserted on by nothing, which is why this is fatal. + NODE_CLASS[$_h]="unclassified" + fail "$_h octet .$_o is in NO ruled D-134 band (utility .$D134_UTILITY_LO-.$D134_UTILITY_HI, role .100+) -- cannot classify, refusing to skip or to assert" + fi ;; + esac +done + +# system_id resolution differs BY REGION and getting it wrong is what produced eight +# of P4's eleven fatals. lib-hosts.sh:54-57 / :64-67: VR0 hostnames are stable +# (hostname == libvirt domain == power_id) so host_sysid() is correct there; VR1 MAAS +# hostnames are RANDOM and RE-MINTED on every re-enlistment (measured: the 2026-07-21 +# pod-delete recovery renamed all 9), so the only stable key is the config-pinned boot +# MAC. `|| true` is deliberate: under `set -e` + inherit_errexit a failing pipeline +# inside $( ) aborts the script instead of reaching the refusal branch below. +sysid_for() { + local h="$1" + case "$DC" in + vr0-dc0) host_sysid "$h" || true ;; + *) host_sysid_by_bootmac "${HOST_BOOT_MAC[$h]}" || true ;; + esac +} + +# --------------------------------------------------------------------------- hdr "Repo (informational)" note "REPO=$REPO" note "HEAD: $(git --no-pager log --oneline -1 2>/dev/null || echo '(not a git tree)')" @@ -61,8 +146,13 @@ if [ "$DIRTY" = "0" ]; then note "working tree clean"; else warn "working tree has $DIRTY modified file(s)"; fi # --------------------------------------------------------------------------- -hdr "CHECK 0: octavia-pki overlay (no key material printed)" -OVL="overlays/octavia-pki.yaml" +hdr "CHECK 0: per-DC octavia-pki overlay (no key material printed)" +# PER-DC since F1 (2026-07-29). The old fixed `overlays/octavia-pki.yaml` was a single +# unscoped path holding what the D-109 R7 amendment rules must be per-DC CA material +# ("each DC gets its own Octavia CA; no cross-DC amphora root-of-trust"): generating +# dc1's PKI overwrote dc0's, and every existing check asked only *does a file exist*, +# never *whose CA is it*. Naming the file per DC makes the existence check a DC check. +OVL="overlays/${DC}-octavia-pki.yaml" if [ -f "$OVL" ]; then KEYS="$(grep -cE 'lb-mgmt-' "$OVL" || true)" if [ "$KEYS" -eq 5 ]; then pass "overlay present with 5 lb-mgmt-* keys"; else fail "overlay has $KEYS lb-mgmt-* keys (want 5)"; fi @@ -81,19 +171,31 @@ # ZERO VIPs and hard-fail, so this reads the MERGED input. The ruled fix is # exactly this, and explicitly NOT bumping the expected count to paper over it. # - # KNOWN AND DELIBERATELY NOT FIXED HERE (logged, hard rule 1): this check is - # still dc0-only -- it never calls lib_net_select_dc, and lib-net.sh UNSETS - # VIP_PREFIX_* for dc1 by design, so it cannot evaluate dc1 at all. It is also - # triple-only (n!=3 below), so it will reject the ruled dual-family sextet that - # provider-bundle-check already accepts. Both are pre-existing VR0-frozen - # defects, widened in scope by this ruling but not created by it. + # DC-AWARE since 2026-07-29. The previous note here said this check was "still + # dc0-only -- it never calls lib_net_select_dc, and lib-net.sh UNSETS VIP_PREFIX_* + # for dc1 by design, so it cannot evaluate dc1 at all". Both halves are now closed: + # the selector runs at the top of this file, and lib-net's dc1 arm carries the R9 + # group. The overlay read is the SELECTED DC's, so a dc1 run validates dc1's numbers + # against dc1's bands instead of silently re-validating dc0. # An ARRAY, not a space-joined string: under `set -euo pipefail` an unsplit # "a b" reaches grep as ONE filename, grep exits 2, pipefail propagates it and # the whole gate dies silently mid-check. Measured here, not theorised. - VIP_SRC=(bundle.yaml) - [ -f overlays/vr1-dc0-vips.yaml ] && VIP_SRC+=(overlays/vr1-dc0-vips.yaml) - VIPLINES="$(grep -hcE '^[[:space:]]+vip:' "${VIP_SRC[@]}" 2>/dev/null | awk '{s+=$1} END{print s+0}')" - if [ "$VIPLINES" -eq "$VIP_COUNT_EXPECT" ]; then pass "vip: line count = $VIPLINES"; else warn "vip: line count = $VIPLINES (want $VIP_COUNT_EXPECT)"; fi + VIP_OVL="overlays/${DC}-vips.yaml" + if [ ! -f "$VIP_OVL" ]; then + # REFUSE, do not fall through. Under ruling 3 bundle.yaml is VIP-FREE, so without + # the per-DC overlay there is nothing to validate -- and the old code path would + # have read zero VIPs and reported a count mismatch, which describes the wrong + # problem. `vr0-dc0` is a legal selector with no overlay in this tree and lands + # here by design. + fail "no VIP overlay for DC=$DC ($VIP_OVL). bundle.yaml is VIP-FREE (ruling 3, 2026-07-25), so this DC has NO VIPs to validate -- render it or select a DC that has one" + else + VIP_SRC=(bundle.yaml "$VIP_OVL") + # `|| true` on the grep: it exits 1 when a file has no matches, pipefail propagates + # that, and under inherit_errexit the command substitution ABORTS the whole script + # instead of reaching any branch. bundle.yaml is VIP-free, so the zero-match case is + # the NORMAL one here, not an edge case. + VIPLINES="$( { grep -hcE '^[[:space:]]+vip:' "${VIP_SRC[@]}" 2>/dev/null || true; } | awk '{s+=$1} END{print s+0}')" + if [ "$VIPLINES" -eq "$VIP_COUNT_EXPECT" ]; then pass "vip: line count = $VIPLINES (from $VIP_OVL)"; else warn "vip: line count = $VIPLINES (want $VIP_COUNT_EXPECT, from $VIP_OVL)"; fi VIPOUT="$(awk -v pp="$VIP_PREFIX_PROVIDER" -v pa="$VIP_PREFIX_ADMIN" -v pi="$VIP_PREFIX_INTERNAL" \ -v lo="$VIP_OCTET_MIN" -v hi="$VIP_OCTET_MAX" ' BEGIN{ok=0;bad=0} @@ -123,10 +225,11 @@ VOK="$(echo "$VIPOUT" | awk '/^RESULT /{print $2}')" VBAD="$(echo "$VIPOUT" | awk '/^RESULT /{print $3}')" if [ "${VOK:-0}" -eq "$VIP_COUNT_EXPECT" ] && [ "${VBAD:-1}" -eq 0 ]; then - pass "aligned triple VIPs OK=$VOK bad=$VBAD" + pass "aligned VIPs OK=$VOK bad=$VBAD (DC=$DC bands $VIP_PREFIX_PROVIDER/$VIP_PREFIX_ADMIN/$VIP_PREFIX_INTERNAL)" else fail "VIP validation OK=${VOK:-0} bad=${VBAD:-?} (want OK=$VIP_COUNT_EXPECT bad=0)" fi + fi fi # --------------------------------------------------------------------------- @@ -150,38 +253,107 @@ fi done if [ "$present" -eq 6 ]; then pass "all six planes present (by CIDR)"; else fail "only $present/6 planes present"; fi -mivid="$(printf '%s' "$SUBJSON" | jq -r --arg c "$METAL_INTERNAL_CIDR" '.[]|select(.cidr==$c)|.vlan.vid // 0')" -if [ "$mivid" = "$METAL_INTERNAL_VID" ]; then pass "metal-internal is VID $METAL_INTERNAL_VID"; else fail "metal-internal VID=$mivid (want $METAL_INTERNAL_VID)"; fi + +# --- metal-internal TAGGING (readiness item 3.7) ------------------------------------ +# RETIRED EXPECTATION: this asserted `vid == METAL_INTERNAL_VID` unconditionally, i.e. +# VLAN 103. D-133 (ADOPTED 2026-07-23) ABOLISHED that stack for VR1 -- "VR1 DCs carve +# every plane FLAT on its dedicated node NIC. Metal-internal is an untagged subnet on +# the enp3s0 plane network ... VID 103 is a VR0-only fact." The assertion could +# therefore NEVER pass on either VR1 DC; the live `vid 0` it kept failing on is the +# CORRECT carve. lib-net's VR1 arms now UNSET the pair, so the guard below is a real +# branch, not a paper one -- and under `set -u` an unguarded read would abort the run. +# +# The guard does NOT weaken the gate: it swaps VR0's expectation for D-133's, which is +# the stronger of the two on a VR1 DC. Both branches can fail, and an unreadable VLAN +# refuses instead of defaulting to success -- `.vlan.vid // 0` used to render a MISSING +# vlan object and a genuine untagged VLAN identically as "0", so a subnet whose VLAN +# could not be read would have silently satisfied the untagged branch. +miraw="$(printf '%s' "$SUBJSON" | jq -r --arg c "$METAL_INTERNAL_CIDR" '.[]|select(.cidr==$c)|.vlan.vid')" +if [ -z "$miraw" ] || [ "$miraw" = "null" ]; then + fail "metal-internal ($METAL_INTERNAL_CIDR) carries NO readable VLAN in MAAS -- cannot evaluate its tagging, refusing to treat that as untagged" +elif [ -n "${METAL_INTERNAL_VID:-}" ]; then + # VR0 (vr0-dc0): the tagged VLAN-103 stack is this DC's real, validated as-built. + if [ "$miraw" = "$METAL_INTERNAL_VID" ]; then pass "metal-internal is VID $METAL_INTERNAL_VID (VR0 Pattern A)" + else fail "metal-internal VID=$miraw (want $METAL_INTERNAL_VID)"; fi +else + # VR1 (D-133): untagged. MAAS represents the untagged/default VLAN as vid 0. + if [ "$miraw" = "0" ]; then pass "metal-internal is UNTAGGED (vid 0) -- D-133 flat carve" + else fail "metal-internal VID=$miraw but D-133 rules it UNTAGGED for VR1 -- a VLAN tag here means the abolished VLAN stack came back"; fi +fi note "stale-NAME check is juju-side (run scripts/juju-spaces-check.sh after add-model)" # --------------------------------------------------------------------------- -hdr "CHECK 2: data/storage NIC links BY CIDR (per host; expect octet .40-.43)" +hdr "CHECK 2: data/storage NIC links BY CIDR (DC=$DC role nodes; octet per D-134 band)" +ROLE_SEEN=0 for h in "${HOSTS[@]}"; do - sid="$(host_sysid "$h" || true)" + if [ "${NODE_CLASS[$h]}" = "unclassified" ]; then continue; fi # already refused above + if [ "${NODE_CLASS[$h]}" = "utility" ]; then + # Not an OpenStack role node (D-134 utility band). It is still a pinned node of + # this DC, so its ENROLLMENT is asserted -- but the six-plane carve and the + # bundle's binding are statements about role nodes only. + usid="$(sysid_for "$h")" + if [ -z "$usid" ]; then fail "$h (utility band .${HOST_OCTET[$h]}) has NO MAAS record for its pinned boot MAC" + else note "$h ($usid, octet .${HOST_OCTET[$h]}) is D-134 utility-band infrastructure, not an OpenStack role node -- data-plane carve not asserted"; fi + continue + fi + ROLE_SEEN=$((ROLE_SEEN+1)) + sid="$(sysid_for "$h")" if [ -z "$sid" ]; then fail "$h not enrolled in MAAS"; continue; fi echo " == $h ($sid, octet .${HOST_OCTET[$h]}) ==" IFJSON="$(maas "$MAAS_PROFILE" interfaces read "$sid" 2>/dev/null)" || { fail "cannot read interfaces for $h"; continue; } for c in "${DATA_PLANE_CIDRS[@]}"; do - line="$(printf '%s' "$IFJSON" | jq -r --arg c "$c" '.[] as $if | $if.links[]? | select(.subnet.cidr==$c) | "\($if.name) \(.ip_address // "(no-ip)")"' | head -1)" + # Tab-separated on purpose: IFS is $'\n\t' at the top of this file, so a + # space-separated payload does NOT word-split and a third field would be lost. + line="$(printf '%s' "$IFJSON" | jq -r --arg c "$c" '.[] as $if | $if.links[]? | select(.subnet.cidr==$c) | "\($if.name)\t\(.ip_address // "(no-ip)")\t\($if.type // "unknown")"' | head -1)" if [ -z "$line" ]; then fail "$h missing link on ${PLANE_NAME[$c]} ($c)"; continue; fi - ifname="${line%% *}"; ip="${line##* }" - printf " %-14s -> %-16s %s\n" "$ifname" "$c" "$ip" - if [ "$c" = "$METAL_INTERNAL_CIDR" ] && [ "$ifname" != "$METAL_INTERNAL_IFACE" ]; then - fail "$h metal-internal on '$ifname' (want $METAL_INTERNAL_IFACE)" + IFS=$'\t' read -r ifname ip iftype <<<"$line" + printf " %-14s -> %-16s %-16s type=%s\n" "$ifname" "$c" "$ip" "$iftype" + if [ "$c" = "$METAL_INTERNAL_CIDR" ]; then + # RETIRED EXPECTATION (readiness item 3.7): this asserted the link name equals + # METAL_INTERNAL_IFACE, i.e. `br-internal`. D-133 ABOLISHED that bridge for VR1 + # ("no br-metal.103 / br-internal stack exists in VR1"), and lib-net's VR1 arms + # UNSET the constant, so the assertion is guarded to the DC that still has it. + # The VR1 branch is NOT a skip -- it asserts D-133's own invariant, that the + # plane lands FLAT on a dedicated PHYSICAL NIC. That is falsifiable in the exact + # way that matters: rebuild a bridge or a VLAN stack over metal-internal and + # MAAS reports type `bridge` / `vlan` and this fails. It is also stronger than + # pinning the NIC NAME would be, and needs no new literal in a gate. + # Measured basis: docs/audit/stage4-carve-verify-20260723.txt shows + # `enp3s0 physical -metal-internal static:...` on all 18 role nodes, both DCs. + if [ -n "${METAL_INTERNAL_IFACE:-}" ]; then + [ "$ifname" = "$METAL_INTERNAL_IFACE" ] || fail "$h metal-internal on '$ifname' (want $METAL_INTERNAL_IFACE)" + else + [ "$iftype" = "physical" ] || fail "$h metal-internal on '$ifname' is type '$iftype' (D-133 rules a FLAT carve on a dedicated PHYSICAL NIC for VR1 -- a bridge/vlan here is the abolished VLAN-103 stack returning)" + fi fi o="$(fourth_octet "$ip")" if [ "$o" != "${HOST_OCTET[$h]}" ]; then warn "$h ${PLANE_NAME[$c]} ip=$ip (want last octet .${HOST_OCTET[$h]})"; fi done done +# Zero-floor: a gate that asserted nothing must not read as a gate that found nothing. +if [ "$ROLE_SEEN" -eq 0 ]; then fail "DC=$DC yielded ZERO OpenStack role nodes -- nothing was carve-checked; a check over an empty set is not a check"; fi # --------------------------------------------------------------------------- -hdr "CHECK 4: the four KVM hosts -- status / power" +hdr "CHECK 4: DC=$DC OpenStack role nodes -- status / power" MJSON="$(maas "$MAAS_PROFILE" machines read 2>/dev/null)" || { fail "cannot read machines"; finish; } +# Resolved BY THE SAME KEY as CHECK 2, not by hostname. Selecting on `.hostname` is +# what produced four of P4's eleven fatals on a VR1 DC: lib-hosts' VR1 HOSTS entries +# are TOFU NODE NAMES (config-of-record), while the MAAS hostname is random and +# re-minted on every re-enlistment (lib-hosts.sh:83-88). for h in "${HOSTS[@]}"; do - row="$(printf '%s' "$MJSON" | jq -r --arg n "$h" '.[]|select(.hostname==$n)|"\(.hostname) \(.status_name) power=\(.power_state)"')" - if [ -z "$row" ]; then fail "host $h not found in MAAS"; continue; fi - echo " $row" - st="$(printf '%s' "$MJSON" | jq -r --arg n "$h" '.[]|select(.hostname==$n)|.status_name')" + # The D-104 per-DC Juju controller is DELIBERATELY not asserted Ready here: it is a + # `juju bootstrap` precondition, not a deploy one, and once bootstrapped it is + # legitimately `Deployed`. Its presence is asserted in CHECK 2 above. An + # unclassified node was already refused in the DC-selection block. + [ "${NODE_CLASS[$h]}" = "role" ] || continue + sid="$(sysid_for "$h")" + if [ -z "$sid" ]; then fail "host $h not found in MAAS"; continue; fi + row="$(printf '%s' "$MJSON" | jq -r --arg s "$sid" '.[]|select(.system_id==$s)|"\(.hostname) \(.status_name) power=\(.power_state)"')" + if [ -z "$row" ]; then fail "host $h ($sid) resolved but absent from the machines list"; continue; fi + echo " $h -> $row" + st="$(printf '%s' "$MJSON" | jq -r --arg s "$sid" '.[]|select(.system_id==$s)|.status_name')" + # DOCFIX-200: Stage 4 hands off READY nodes, not Deployed ones -- Juju provisions at + # Stage 5, so Ready is the correct pre-deploy state for a role node. if [ "$st" = "Ready" ]; then pass "$h Ready"; else fail "$h status=$st (want Ready)"; fi done diff --git a/scripts/preflight.sh b/scripts/preflight.sh index 64dc07c..b5eb983 100644 --- a/scripts/preflight.sh +++ b/scripts/preflight.sh @@ -18,12 +18,50 @@ # P6 reminders stage-2 gates: juju-spaces-check.sh (AFTER add-model), # osd-blank-check.sh (sudo), 'juju deploy --dry-run' (phase-01 1.2) # +# DC-AWARE END TO END (F5, 2026-07-29). `DC` selects the datacenter this run gates; +# it defaults to vr1-dc0, is validated against the D-119 region-qualified token set, +# is EXPORTED so every sub-gate measures the same DC, and is printed in both the +# opening header and the verdict line. P2 and P4 are the DC-sensitive gates. +# DC=vr1-dc1 bash scripts/preflight.sh +# # Exit: 0 all pass | 1 any FAIL (do NOT deploy) | 2 warnings only (review, then decide). # ASCII + LF. set -uo pipefail REPO="${1:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}" cd "$REPO" || { echo "FAIL: cannot cd to repo root $REPO"; exit 1; } WORST=0 + +# ---- F5: the DC selection is ONE decision, made ONCE, EXPORTED, and PRINTED ----- +# MEASURED DEFECT (2026-07-29, docs/audit/stage5-findings-20260729-successor.md F5): +# this used to be set at the top of P2 as a plain shell variable and never exported, so +# whether the sub-gates saw it depended on the CALLER'S INVOCATION FORM -- +# `DC=vr1-dc1 bash scripts/preflight.sh` put it in the environment (children saw it), +# a bare invocation did not (children saw nothing and silently used their own default). +# Two call shapes, two behaviours, neither documented. `export` collapses them to one. +# +# REFUSE an unrecognised DC rather than defaulting to success: an unknown token used to +# flow straight into `--dc $DC` and the overlay paths, where it produced "overlay not +# found" noise instead of "you named a DC that does not exist". The token set is the +# D-119 region-qualified one enforced by lib-net.sh/lib-hosts.sh; bare `dcN` is RETIRED +# there and is rejected here with the same reason so the message does not depend on +# which gate happens to see it first. +DC="${DC:-vr1-dc0}" +case "$DC" in + vr0-dc0|vr1-dc0|vr1-dc1) ;; + dc0|dc1|dc2) + echo "FAIL: DC='$DC' -- bare 'dcN' is RETIRED (D-119): it meant VR0's live DC0 in" + echo " lib-net.sh but VR1's FIRST DC in the NetBox importer. Use a" + echo " region-qualified selector: vr0-dc0 | vr1-dc0 | vr1-dc1" + exit 1 ;; + *) + echo "FAIL: DC='$DC' is not a recognised datacenter selector (expected vr0-dc0," + echo " vr1-dc0 or vr1-dc1). REFUSING rather than gating an unknown target --" + echo " a gate that cannot say WHAT it measured is not a gate." + exit 1 ;; +esac +export DC +echo "PREFLIGHT TARGET: DC=$DC (override: DC=vr1-dc1 bash scripts/preflight.sh)" +echo " Every gate below is run against THIS DC; the verdict line repeats it." # ---- R15(3): preflight must not FAIL OPEN (ruled 2026-07-27, OPS) -------------- # MEASURED DEFECT: note() tested ONLY rc 1 and 2, so a sub-gate exiting 127 (command # not found), 126 (not executable), 130 (interrupted) or any rc>=3 left WORST @@ -96,9 +134,12 @@ # deploy is not validating the deploy. Each overlay is added only if present, so a tree # missing the gitignored octavia-pki secret still runs (CHECK 0 in P4 is what fails on # that, deliberately, and it is a different gate). -DC="${DC:-vr1-dc0}" +# $DC is selected, validated and EXPORTED at the top of this file (F5). +# The octavia-pki overlay is PER-DC since F1 (2026-07-29) -- one shared +# overlays/octavia-pki.yaml would have carried whichever DC's CA was generated last, +# which is the cross-DC amphora root-of-trust the D-109 R7 amendment refused. P2_ARGS="bundle.yaml" -for _ovl in "overlays/${DC}-vips.yaml" "overlays/${DC}-machines.yaml" overlays/octavia-pki.yaml; do +for _ovl in "overlays/${DC}-vips.yaml" "overlays/${DC}-machines.yaml" "overlays/${DC}-octavia-pki.yaml"; do [ -f "$_ovl" ] && P2_ARGS="$P2_ARGS --overlay $_ovl" done P2_ARGS="$P2_ARGS --dc $DC" @@ -151,9 +192,12 @@ echo " - with sudo: bash scripts/osd-blank-check.sh" echo " - phase-01 Step 1.2: juju deploy --dry-run (plan: 50 apps / 97 relations)" +# The `PREFLIGHT: ` token stays CONTIGUOUS and leading -- the harness and the +# runbooks grep for it verbatim. $DC is appended so a verdict can never be read against +# the wrong datacenter (F5: two DCs, one combined verdict line, no way to tell). case "$WORST" in - 0) echo; echo "PREFLIGHT: PASS -- clear to add-model / deploy" ;; - 2) echo; echo "PREFLIGHT: WARN -- review warnings, then decide" ;; - *) echo; echo "PREFLIGHT: FAIL -- do NOT deploy" ;; + 0) echo; echo "PREFLIGHT: PASS (DC=$DC) -- clear to add-model / deploy" ;; + 2) echo; echo "PREFLIGHT: WARN (DC=$DC) -- review warnings, then decide" ;; + *) echo; echo "PREFLIGHT: FAIL (DC=$DC) -- do NOT deploy" ;; esac exit "$WORST" diff --git a/tests/HARNESS-MANIFEST b/tests/HARNESS-MANIFEST index 4034cf5..a8a8ba5 100644 --- a/tests/HARNESS-MANIFEST +++ b/tests/HARNESS-MANIFEST @@ -63,6 +63,7 @@ phase-06-net-setup phase-07-conductor-graft preflight +pre-flight-checks prereqs provider-bundle-check record-audit diff --git a/tests/dc-selector/run-tests.sh b/tests/dc-selector/run-tests.sh index d431620..27315ad 100644 --- a/tests/dc-selector/run-tests.sh +++ b/tests/dc-selector/run-tests.sh @@ -59,12 +59,48 @@ chk "net vr1-dc1 provider-public gw is 10.12.64.1 (ruled edge LAN gw)" "$DC1_GW" "10.12.64.1" DC1_MI="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "$METAL_INTERNAL_CIDR")" chk "net vr1-dc1 metal-internal CIDR is 10.12.72.0/22" "$DC1_MI" "10.12.72.0/22" -# not-yet-ruled OpenStack-layer values are UNSET (fail loud under set -u, never -# silently another DC's addresses). -DC1_KV="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${KEYSTONE_VIP_DEFAULT+still-set}")" -chk "net vr1-dc1 unsets KEYSTONE_VIP_DEFAULT" "$DC1_KV" "" -DC1_VP="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${VIP_PREFIX_PROVIDER+still-set}")" -chk "net vr1-dc1 unsets VIP_PREFIX_PROVIDER" "$DC1_VP" "" +# --- R9 group: REPLACED 2026-07-29, not deleted ------------------------------------- +# These two cases asserted that vr1-dc1 UNSETS KEYSTONE_VIP_DEFAULT and VIP_PREFIX_*. +# That state was correct only while the values were unruled. R9 (the D-119 AMENDMENT, +# RULED 2026-07-27) ruled them and named overlays/vr1-dc1-vips.yaml the single AUTHORED +# source, and the unset was actively costing: `lib_net_select_dc vr1-dc1` left +# KEYSTONE_VIP_DEFAULT unbound, so every selector-CALLER touching it died under `set -u` +# while the non-callers silently used dc0's address. +# The assertions are REPOINTED at the ruled values -- a strictly stronger check than the +# absence they replace -- and paired with the DRIFT cases below, which re-derive the same +# values from the overlay so the arm and the ratified artifact cannot diverge silently. +DC1_KV="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${KEYSTONE_VIP_DEFAULT:-}")" +chk "net vr1-dc1 sets KEYSTONE_VIP_DEFAULT (R9)" "$DC1_KV" "10.12.64.50" +DC1_VP="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${VIP_PREFIX_PROVIDER:-}")" +chk "net vr1-dc1 sets VIP_PREFIX_PROVIDER (R9)" "$DC1_VP" "10.12.64" +DC1_VA="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${VIP_PREFIX_ADMIN:-}")" +chk "net vr1-dc1 sets VIP_PREFIX_ADMIN (R9)" "$DC1_VA" "10.12.68" +DC1_VI="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${VIP_PREFIX_INTERNAL:-}")" +chk "net vr1-dc1 sets VIP_PREFIX_INTERNAL (R9)" "$DC1_VI" "10.12.72" +DC1_VC="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${VIP_COUNT_EXPECT:-}")" +chk "net vr1-dc1 sets VIP_COUNT_EXPECT 13 (R11)" "$DC1_VC" "13" + +# --- R9's DRIFT half: the arm is checked against the artifact it was copied from ------ +# R9's ruled shape is DERIVE + drift-check. Only the drift half exists today (the arm is +# hand-authored, see lib-net.sh's own note), and this is it: re-read the RATIFIED +# overlay and require the arm to agree, in both directions. Re-rendering the overlay +# without updating the arm, or editing the arm without the overlay, turns this red. +OVL="$SD/../../overlays/vr1-dc1-vips.yaml" +if [ ! -r "$OVL" ]; then + no "R9 drift: overlays/vr1-dc1-vips.yaml is missing -- the arm's authored source is gone" +else + # First v4 leg of each vip line = provider column; 2nd = admin; 3rd = internal. + OVL_PP="$(awk -F'"' '/^[[:space:]]+vip:/{split($2,a," "); split(a[1],p,"."); print p[1]"."p[2]"."p[3]; exit}' "$OVL")" + OVL_PA="$(awk -F'"' '/^[[:space:]]+vip:/{split($2,a," "); split(a[2],p,"."); print p[1]"."p[2]"."p[3]; exit}' "$OVL")" + OVL_PI="$(awk -F'"' '/^[[:space:]]+vip:/{split($2,a," "); split(a[3],p,"."); print p[1]"."p[2]"."p[3]; exit}' "$OVL")" + OVL_CNT="$(grep -cE '^[[:space:]]+vip:' "$OVL")" + OVL_KS="$(awk -F'"' '/^ keystone:/{k=1} k&&/^[[:space:]]+vip:/{split($2,a," "); print a[1]; exit}' "$OVL")" + chk "R9 drift: arm provider prefix == overlay" "$DC1_VP" "$OVL_PP" + chk "R9 drift: arm admin prefix == overlay" "$DC1_VA" "$OVL_PA" + chk "R9 drift: arm internal prefix == overlay" "$DC1_VI" "$OVL_PI" + chk "R9 drift: arm VIP_COUNT_EXPECT == overlay vip: line count" "$DC1_VC" "$OVL_CNT" + chk "R9 drift: arm KEYSTONE_VIP_DEFAULT == overlay keystone provider leg" "$DC1_KV" "$OVL_KS" +fi # REPLACED 2026-07-27, not deleted. This asserted that vr1-dc1 UNSETS FIP_POOL_START. # The pool was RULED that day (GA-R5: "Rule dc1 FIP pool 10.12.65.0-10.12.67.254"; # recorded on the D-134 R4 amendment), so the old assertion encoded a state the ruling @@ -75,13 +111,46 @@ chk "net vr1-dc1 sets the RULED FIP_POOL_START" "$DC1_FP" "10.12.65.0" DC1_FPE="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${FIP_POOL_END:-}")" chk "net vr1-dc1 sets the RULED FIP_POOL_END" "$DC1_FPE" "10.12.67.254" -# GUARD: the ruling narrowed the unset list by exactly one pair. The VIP group must -# STAY unset (R9 rules it GENERATED from the overlay) and so must the D-133-retired -# metal-internal VLAN stack. -DC1_VP="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${VIP_PREFIX_PROVIDER+still-set}")" -chk "net vr1-dc1 STILL unsets VIP_PREFIX_PROVIDER (R9: generated, not hand-set)" "$DC1_VP" "" +# GUARD, NON-NEGOTIABLE. The unset list shrank twice (the FIP pool in 2026-07-27's +# ruling, the R9 VIP group in 2026-07-29's delivery) and both times the same pair had to +# survive: METAL_INTERNAL_VID / METAL_INTERNAL_IFACE are not "unruled", they are +# ABOLISHED for VR1 by D-133 (flat per-NIC carve; metal-internal untagged on enp3s0). +# Anything that repopulates them silently reintroduces a retired VLAN stack, which is +# exactly what R9's own GUARD paragraph forbids. Both DCs, both variables -- the +# vr1-dc0 arm inherited them from VR0 until 2026-07-29 and now unsets them too. DC1_MV="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${METAL_INTERNAL_VID+still-set}")" chk "net vr1-dc1 STILL unsets METAL_INTERNAL_VID (D-133 retired that stack)" "$DC1_MV" "" +DC1_MIF="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${METAL_INTERNAL_IFACE+still-set}")" +chk "net vr1-dc1 STILL unsets METAL_INTERNAL_IFACE (D-133 retired br-internal)" "$DC1_MIF" "" +DC0_MV="$(lib_net_select_dc vr1-dc0 >/dev/null 2>&1; echo "${METAL_INTERNAL_VID+still-set}")" +chk "net vr1-dc0 unsets METAL_INTERNAL_VID (D-133 'affected surfaces', executed 2026-07-29)" "$DC0_MV" "" +DC0_MIF="$(lib_net_select_dc vr1-dc0 >/dev/null 2>&1; echo "${METAL_INTERNAL_IFACE+still-set}")" +chk "net vr1-dc0 unsets METAL_INTERNAL_IFACE (D-133)" "$DC0_MIF" "" +# VR0's own as-built is UNTOUCHED: D-133 supersedes Pattern A "FOR VR1 DC NODES ONLY". +VR0_MV="$(lib_net_select_dc vr0-dc0 >/dev/null 2>&1; echo "${METAL_INTERNAL_VID:-}")" +chk "net vr0-dc0 KEEPS METAL_INTERNAL_VID 103 (VR0 as-built untouched)" "$VR0_MV" "103" +VR0_MIF="$(lib_net_select_dc vr0-dc0 >/dev/null 2>&1; echo "${METAL_INTERNAL_IFACE:-}")" +chk "net vr0-dc0 KEEPS METAL_INTERNAL_IFACE br-internal" "$VR0_MIF" "br-internal" + +# --- metal-admin carries NO gateway in either VR1 DC (retired VR0 inheritance) ------- +# MEASURED 2026-07-27 (docs/audit/stage5-committee-raw-20260727.md:309, reconciled at +# docs/CURRENT-STATE.md:938-950): MAAS returns gateway `none` for 10.12.8.0/22 and +# 10.12.68.0/22, and both .1 addresses are held by NOTHING (100% loss, ARP never +# resolves) while both provider-public gateways answer and carry the rack default route. +# D-134's carve rules `.1` gateways for the PROVIDER subnets only. The pin was the +# defect, not MAAS -- it made scripts/pre-flight-checks.sh hard-fail on a correct cloud. +VR0_GWN="$(lib_net_select_dc vr0-dc0 >/dev/null 2>&1; echo "${#PLANE_GW[@]}")" +chk "net vr0-dc0 keeps BOTH gateways (VR0 real as-built)" "$VR0_GWN" "2" +DC0_GWN="$(lib_net_select_dc vr1-dc0 >/dev/null 2>&1; echo "${#PLANE_GW[@]}")" +chk "net vr1-dc0 has exactly ONE routed plane" "$DC0_GWN" "1" +DC0_GWK="$(lib_net_select_dc vr1-dc0 >/dev/null 2>&1; echo "${!PLANE_GW[*]}")" +chk "net vr1-dc0 routed plane is provider-public" "$DC0_GWK" "10.12.4.0/22" +DC0_GWA="$(lib_net_select_dc vr1-dc0 >/dev/null 2>&1; echo "${PLANE_GW[10.12.8.0/22]:-unset}")" +chk "net vr1-dc0 metal-admin has NO gateway (measured: 10.12.8.1 answers nothing)" "$DC0_GWA" "unset" +DC1_GWN="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${#PLANE_GW[@]}")" +chk "net vr1-dc1 has exactly ONE routed plane" "$DC1_GWN" "1" +DC1_GWA="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${PLANE_GW[10.12.68.0/22]:-unset}")" +chk "net vr1-dc1 metal-admin has NO gateway (measured: 10.12.68.1 answers nothing)" "$DC1_GWA" "unset" # ONE SELECTION PER SHELL (guard added with the diverging dc1 arm): changing # selection after dc1 is REFUSED; re-selecting the SAME DC stays a no-op. diff --git a/tests/pre-flight-checks/run-tests.sh b/tests/pre-flight-checks/run-tests.sh new file mode 100644 index 0000000..e315700 --- /dev/null +++ b/tests/pre-flight-checks/run-tests.sh @@ -0,0 +1,301 @@ +#!/usr/bin/env bash +# tests/pre-flight-checks/run-tests.sh -- offline harness for scripts/pre-flight-checks.sh +# (preflight's P4 gate). +# +# WHY IT EXISTS. Until 2026-07-29 this script had NO harness at all, and the only place +# it had ever been observed running was a live jumphost where it FAILED -- eleven fatals, +# of which zero described the cloud. That is the repo's own inverse rule biting: "a +# checker that cannot fail is not a gate" has a twin, and a gate only ever observed +# FAILING needs a constructed fixture proving it can go green. So T1/T2/T3 build a +# CORRECT DC and require exit 0, and everything after them flips exactly one field of +# that same fixture and requires the gate to go red on the assertion that field feeds. +# +# HERMETIC: a fake `maas` on PATH serves JSON from files in the fixture; nothing reaches +# a real MAAS, a real cloud or the real repo. Fixtures are BUILT FROM lib-net.sh + +# lib-hosts.sh for the selected DC, so a green case attests that the gate agrees with the +# libraries rather than with a second hand-typed copy of the same numbers. +# +# Exit: 0 all pass | 1 any case failed. ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO="$(cd "$HERE/../.." && pwd)" +TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT +PASS=0; FAIL=0 +ok() { echo " PASS $1"; PASS=$((PASS+1)); } +bad() { echo " FAIL $1"; FAIL=$((FAIL+1)); } + +command -v jq >/dev/null 2>&1 || { echo " FAIL jq is required by the script under test"; echo "RESULT: PASS=0 FAIL=1"; exit 1; } + +# --------------------------------------------------------------------------- +# mkfix -> echoes the fixture repo path +# Builds a repo whose MAAS state is exactly what the selected DC's libraries describe. +mkfix() { + local name="$1" dc="$2" d="$TMP/$1" + mkdir -p "$d/scripts" "$d/overlays" "$d/fakebin" "$d/maasdata" + cp "$REPO/scripts/pre-flight-checks.sh" "$REPO/scripts/lib-net.sh" \ + "$REPO/scripts/lib-hosts.sh" "$d/scripts/" + printf 'applications:\n keystone:\n charm: keystone\n' > "$d/bundle.yaml" + + # The octavia-pki overlay is a gitignored SECRET in the real tree, so the fixture + # mints its own with the 5 lb-mgmt-* keys the gate counts. No key material: the + # values are placeholders and the gate never prints them anyway. + { + printf 'applications:\n octavia:\n options:\n' + for k in lb-mgmt-issuing-cacert lb-mgmt-issuing-ca-private-key \ + lb-mgmt-issuing-ca-key-passphrase lb-mgmt-controller-cacert \ + lb-mgmt-controller-cert; do + printf ' %s: PLACEHOLDER\n' "$k" + done + } > "$d/overlays/${dc}-octavia-pki.yaml" + + # Dump the DC's own view of planes and hosts. A SUBSHELL per dump: both libs refuse a + # second, different selection in the same shell. + ( . "$REPO/scripts/lib-net.sh"; lib_net_select_dc "$dc" >/dev/null + for c in "${PLANE_CIDRS[@]}"; do printf 'P\t%s\t%s\t%s\n' "$c" "${PLANE_NAME[$c]}" "${PLANE_GW[$c]:-}"; done + for c in "${DATA_PLANE_CIDRS[@]}"; do printf 'D\t%s\n' "$c"; done + printf 'M\t%s\n' "$METAL_INTERNAL_CIDR" + printf 'I\t%s\n' "${METAL_INTERNAL_IFACE:-}" + printf 'T\t%s\n' "${METAL_INTERNAL_VID:-0}" + printf 'V\t%s\t%s\t%s\t%s\t%s\n' "$VIP_PREFIX_PROVIDER" "$VIP_PREFIX_ADMIN" \ + "$VIP_PREFIX_INTERNAL" "$VIP_OCTET_MIN" "$VIP_COUNT_EXPECT" + ) > "$d/maasdata/planes.tsv" + ( . "$REPO/scripts/lib-hosts.sh"; lib_hosts_select_dc "$dc" >/dev/null + for h in "${HOSTS[@]}"; do printf 'H\t%s\t%s\t%s\n' "$h" "${HOST_OCTET[$h]}" "${HOST_BOOT_MAC[$h]}"; done + ) > "$d/maasdata/hosts.tsv" + + D="$d" python3 - <<'PY' +import json, os +d = os.environ["D"] +planes, data, hosts = [], [], [] +mi = mi_iface = "" +mi_vid = 0 +vip = None +for line in open(d + "/maasdata/planes.tsv"): + f = line.rstrip("\n").split("\t") + if f[0] == "P": planes.append((f[1], f[2], f[3])) + elif f[0] == "D": data.append(f[1]) + elif f[0] == "M": mi = f[1] + elif f[0] == "I": mi_iface = f[1] + elif f[0] == "T": mi_vid = int(f[1] or 0) + elif f[0] == "V": vip = f[1:] +for line in open(d + "/maasdata/hosts.tsv"): + f = line.rstrip("\n").split("\t") + if f[0] == "H": hosts.append((f[1], int(f[2]), f[3])) + +subnets = [] +for i, (cidr, name, gw) in enumerate(planes): + # VR0's metal-internal is the tagged VLAN-103 stack; every VR1 plane is untagged + # (D-133 flat carve), and MAAS renders the untagged/default VLAN as vid 0. + subnets.append({"id": 100 + i, "cidr": cidr, "name": name, + "vlan": {"vid": mi_vid if cidr == mi else 0}, + "gateway_ip": gw or None, "dns_servers": []}) +json.dump(subnets, open(d + "/maasdata/subnets.json", "w"), indent=1) + +# The D-134 utility band (.4-.49) is per-DC infrastructure, not an OpenStack role node. +# VR0's .40-.43 index predates those bands, so vr0-dc0's fleet is all role nodes. +vr0 = any(h[0].startswith("openstack") for h in hosts) +machines = [] +for n, (host, octet, mac) in enumerate(hosts): + role = vr0 or octet >= 100 + sid = "sysid%02d" % n + # A DELIBERATELY random MAAS hostname for VR1: lib-hosts' VR1 HOSTS entries are tofu + # node names, and MAAS re-mints its own name on every re-enlistment. A gate that + # resolves by hostname cannot pass this fixture. + maas_hostname = host if vr0 else "maas-random-%02d" % n + machines.append({"system_id": sid, "hostname": maas_hostname, + # role nodes are handed off Ready (DOCFIX-200); the D-104 Juju + # controller is legitimately Deployed once bootstrapped. + "status_name": "Ready" if role else "Deployed", + "power_state": "off", + "boot_interface": {"mac_address": mac}}) + ifaces = [] + if not vr0: + ifaces.append({"name": "enp1s0", "type": "physical", + "links": [{"subnet": {"cidr": planes[1][0]}, + "ip_address": planes[1][0].split("/")[0].rsplit(".", 1)[0] + ".%d" % octet}]}) + ifaces.append({"name": "enp2s0", "type": "physical", "links": []}) + for j, cidr in enumerate(data): + base = cidr.split("/")[0].rsplit(".", 1)[0] + if cidr == mi and mi_iface: + nm, ty = mi_iface, "bridge" # VR0 Pattern A: the br-internal stack + else: + nm, ty = "enp%ds0" % (3 + j), "physical" # D-133 flat carve + ifaces.append({"name": nm, "type": ty, + "links": [{"subnet": {"cidr": cidr}, "ip_address": "%s.%d" % (base, octet)}]}) + json.dump(ifaces, open(d + "/maasdata/if-%s.json" % sid, "w"), indent=1) +json.dump(machines, open(d + "/maasdata/machines.json", "w"), indent=1) + +# A VIP overlay for THIS DC, built from the DC's own bands so a green run attests the +# gate and lib-net agree. Column order provider / metal-admin / metal-internal. +pp, pa, pi, lo, cnt = vip[0], vip[1], vip[2], int(vip[3]), int(vip[4]) +with open(d + "/overlays/%s-vips.yaml" % os.environ["DCNAME"], "w") as fh: + fh.write("applications:\n") + for k in range(cnt): + o = lo + k + fh.write(" app%02d:\n options:\n vip: \"%s.%d %s.%d %s.%d\"\n" % (k, pp, o, pa, o, pi, o)) +PY + echo "$d" +} + +# The python block needs the DC name for the overlay filename. +mkfix_dc() { DCNAME="$2" mkfix "$@"; } + +# fakebin maas: serves the fixture JSON. Nothing else is answerable, and an +# unrecognised call exits non-zero rather than printing an empty success. +mkmaas() { + local d="$1" + cat > "$d/fakebin/maas" <<'FB' +#!/usr/bin/env bash +DD="$(dirname "$(dirname "$0")")/maasdata" +case "$2 $3" in + "subnets read") cat "$DD/subnets.json" ;; + "machines read") cat "$DD/machines.json" ;; + "interfaces read") cat "$DD/if-$4.json" 2>/dev/null || echo "[]" ;; + *) echo "fake maas: unhandled '$*'" >&2; exit 2 ;; +esac +FB + chmod +x "$d/fakebin/maas" +} + +# run