diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 26dd378..c2ab809 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2079,6 +2079,60 @@ **UNRULED AND NOT PICKED:** whether dc1's public catalog endpoint becomes the v6 GUA leg under R2/`prefer-ipv6` is ruling-shaped; `KEYSTONE_VIP_DEFAULT` mirrors dc0's v4 provider-leg definition pending that. +- **THE FOUR PHASE-4 GATE-INTEGRITY DEFECTS ARE FIXED, EACH DEMONSTRATED RED THEN GREEN, + 2026-07-29.** The repo's rule is that a checker which cannot fail is not a gate, and its + inverse -- a gate only ever seen failing is equally untrustworthy -- so every item below + carries BOTH a constructed fixture proving it goes red on the defect and a run proving it + goes green on the real tree. + **4.2 DECORATIVE HA is now gated.** `cluster_count` was asserted NOWHERE against 20 + occurrences in `dc-ha-scaleup.yaml`, so rewriting all 20 from `3` to `1` produced a + BYTE-IDENTICAL PASS. Re-verified independently here rather than from reported output: the + same rewrite now yields **13 `DECORATIVE HA` failures**, and the real overlay still PASSes. + A missing `cluster_count` and the scale-down direction both fail too. + **4.3 the machines overlay can no longer be a silent no-op** -- missing, gutted (`machines: + {}`) and mis-keyed (`"01"` vs `"0"`) each fail with distinct messages, the mis-keyed one + under EITHER `--dc`. **A BYPASS WAS CAUGHT ON REVIEW BEFORE DELIVERY:** the check first sat + inside the `role_sep` branch, so an overlay stripping every `tags=` made the whole placement + block SELF-SKIP behind an `[ok]` -- rc=0 verified, then hoisted out and locked by a case. + **4.7 `cloud-assert.sh` gained a real arity check (A10) covering 14/14 scaled apps** + (13 by `cluster_count`, rabbitmq by `min-cluster-size`), discovered from status JSON alone + and measured against a real captured `juju-status.json`. It refuses on an unreadable value + rather than passing, single-unit `ovn-central` is no longer "uniform", and a named + `*-hacluster` app with no charm-name match refuses the skip. + **4.8 G17 has a real node-side check:** `dc-cache-proxy.sh node ` content-asserts + proxied fetches of archive AND UCA `Release` (Codename derived from the URL's `/dists/` + path), carries the `chronyc` time-source assertion R12 folded in, and adds **exit 3 = + REFUSE**. Red on HTML-body-with-200, wrong dist, missing checksum section, 404, unreachable + proxy, edge time source, and silent/unrecognised chronyc. **STATED HONESTLY: its green is a + FIXTURE green** -- no node has run it, because the nodes are powered off and G17's window is + Stage-5 first boot. G17 stays OPEN. + **THE GAUNTLET NOW GIVES THE SAME VERDICT ON BOTH HOSTS.** The voffice1 "drift" was pure + LOCALE COLLATION -- `LC_ALL=C` now applies to the manifest sort and to BOTH sides of the + compare, deliberately WITHOUT re-recording `HARNESS-MANIFEST`, which would merely have moved + the failure to vcloud (the gauntlet's own hint suggests exactly that trap). The + `site-headend-install` snap test now takes a pre-state reading instead of asserting + snap-absence, demonstrated both ways on a simulated headend: new 59/59 PASS, old assertion + 1/59 FAIL on the same host. + **AN AGENT CONCLUSION CORRECTED RATHER THAN ACTED ON.** The delivering agent logged that + `preflight` P2 should also assemble `dc-ha-scaleup.yaml`, on the reasoning that "the deploy + applies the 3-unit overlay". **Measured, that is wrong and acting on it would have + reintroduced the chain-audit defect in reverse:** `dc-ha-scaleup.yaml` is DELIBERATELY + excluded from the Step 4 deploy per R6 (which orders the VIPs first), and `grep -rn` finds + NO step anywhere that applies it -- the scale-up is a later, still-unwritten operation. Adding + it to P2 would make the gate validate an input the deploy does not use, which is precisely + the "a gate validating a different input than the deploy is not validating the deploy" finding + the 2026-07-29 chain audit fixed. The `cluster_count` gate is built and correct; it acquires + deploy-gate reach when a step exists to apply the overlay. LOGGED, NOT EXECUTED. + **DISCLOSED DEVIATION, verified harmless:** the agent used `git stash push/pop` on one file + to demonstrate the old-vs-new snap assertion on the same host, which it had been forbidden. + Checked here because other agents were writing the tree concurrently: `git stash list` empty, + the prior commit intact at HEAD, the D-133 guard still holding, and the modified set exactly + its declared files. No loss -- and it was disclosed rather than hidden. + **COULD NOT BE DONE, recorded rather than papered over:** `juju config` could not be verified + against a live juju (absent on this host; A10 REFUSES on an unreadable value rather than + passing), `chronyc sources` column format could not be measured (chrony absent -- so the + check matches address literals only, parses no columns and invents no flag), `node dc1` could + not be run on a real node, and the gauntlet could not be run on voffice1. **F3 -- both `~/octavia-pki/` and `overlays/octavia-pki.yaml` are ABSENT here** (existence checked, no contents read). So this is generation FROM SCRATCH for both DCs: there is nothing to reuse, which retires the reuse-vs-regenerate choice diff --git a/scripts/cloud-assert.sh b/scripts/cloud-assert.sh index c17377b..0ef2fda 100644 --- a/scripts/cloud-assert.sh +++ b/scripts/cloud-assert.sh @@ -25,6 +25,11 @@ # A9 vault-kv consumer AppRole auth (D-068 item 3, ruled 2026-07-21): # delegates to vault-kv-health.sh (one real 60s-TTL login per consumer); # SKIPS (ok) when the model has no vault application -- pre-Stage-5 VR1 +# A10 HA arity: every hacluster principal's LIVE unit count equals the quorum its +# own subordinate declares (cluster_count), and rabbitmq-server's equals +# min-cluster-size. Added 2026-07-29 -- before it, this script arity-checked +# exactly ONE application (mysql, A2), so a control plane that came up with two +# of three units of everything else read PASS. # # --capture: additionally writes a deploy BOM (bill of materials) to # asbuilt// : exported bundle, juju status yaml, driver pip @@ -106,8 +111,16 @@ CID=$(awk '/^Cluster ID/{print $3; exit}' <<<"$O") [ -n "$CID" ] && IDS="$IDS $CID" || fail "$u: no Cluster ID for $db" done <<<"$UNITS" + # A uniformity assertion over ONE contributing unit is vacuous -- it passed a + # single-unit ovn-central and reported "uniform across units" (found 2026-07-29). + # bundle.yaml pins ovn-central num_units=3, so fewer than 3 IDs means units are + # MISSING or unreadable, and "could not look" is never "nothing there". + NC=$(tr ' ' '\n' <<<"$IDS" | sed '/^$/d' | wc -l) U=$(tr ' ' '\n' <<<"$IDS" | sed '/^$/d' | sort -u | wc -l) - [ "$U" -eq 1 ] && ok "$db Cluster ID uniform across units" || fail "$db Cluster IDs differ:$IDS" + if [ "$NC" -lt 3 ]; then + fail "$db uniformity NOT assertable: only $NC unit(s) returned a Cluster ID (bundle.yaml pins ovn-central num_units=3) -- one unit is trivially 'uniform'" + elif [ "$U" -eq 1 ]; then ok "$db Cluster ID uniform across $NC units" + else fail "$db Cluster IDs differ:$IDS"; fi done fi @@ -198,6 +211,68 @@ esac fi +echo "================ A10: HA arity -- declared quorum vs live units ================" +# THE LIFECYCLE GAP THIS CLOSES (measured 2026-07-29): provider-bundle-check asserts +# cluster_count in the BUNDLE, but nothing compared it to a RUNNING model, and this +# script asserted a unit count exactly once (mysql, A2). So HA was asserted by NOTHING +# after deploy: a model up with 2 of 3 units of every API charm read PASS. +# DISCOVERY is from status JSON alone -- MEASURED shape, asbuilt/20260706-224851/ +# juju-status.json: an hacluster subordinate application carries charm-name "hacluster" +# and subordinate-to [""]. No app list is hardcoded, so the 14 apps +# overlays/dc-ha-scaleup.yaml scales are covered without naming any of them. +# The DECLARED quorum comes from the charm's OWN config via `juju config -m +# ` -- the form already in scripts/phase-05-octavia-verify.sh:54-57 and +# runbooks/dc-dc-phase4-juju-bundle-per-dc.md:855 (`juju config -m "$DC_MODEL" +# -hacluster cluster_count`). An unreadable or non-numeric value REFUSES. +HAP=$(jq -r '.applications // {} | to_entries[] + | select(.value["charm-name"] == "hacluster") + | .key as $sub | (.value["subordinate-to"] // [])[] | "\($sub) \(.)"' <<<"$ST" || true) +if [ -z "$HAP" ]; then + # A skip must be able to tell "none present" from "discovery missed them" -- otherwise + # this section becomes the very silent pass it was added to remove. Cross-check by + # NAME: this bundle's hacluster subordinates are all -hacluster, and no + # other application carries that suffix (mysql-innodb-cluster / ovn-central / ceph-mon + # cluster natively and have none). + NAMED=$(jq -r '.applications // {} | keys[] | select(endswith("-hacluster"))' <<<"$ST" || true) + if [ -n "$NAMED" ]; then + fail "charm-name discovery found NO hacluster subordinates, yet the model carries $(tr '\n' ' ' <<<"$NAMED")-- REFUSING to skip A10; the status-JSON shape changed and this section would pass over every principal" + else + ok "no hacluster subordinates in model '$MODEL' -- A10 hacluster arity skipped (VR1 HA arrives Stage 5)" + fi +else + while read -r sub prin; do + [ -n "$sub" ] && [ -n "$prin" ] || continue + LIVE=$(jq -r --arg a "$prin" '.applications[$a].units // {} | length' <<<"$ST" || true) + CC=$(J config -m "$MODEL" "$sub" cluster_count | head -1 | tr -d ' \r' || true) + case "$CC" in + ''|*[!0-9]*) + fail "cannot read $sub cluster_count (got '$CC') -- REFUSING to report $prin's HA arity as verified" ;; + *) + if [ "${LIVE:-x}" = "$CC" ]; then ok "$prin: $LIVE live unit(s) == $sub cluster_count=$CC" + else fail "$prin has ${LIVE:-?} live unit(s) but $sub declares cluster_count=$CC -- HA arity broken (a missing unit, or a quorum left over from a scale change)"; fi ;; + esac + done <<<"$HAP" +fi +# rabbitmq-server clusters NATIVELY (quorum, no hacluster), so the loop above cannot +# see it -- yet it is one of the 14 apps dc-ha-scaleup scales. Its declared quorum is +# min-cluster-size, MANDATORY at multi-unit per the D-009 AMENDMENT (2026-07-02): +# without it the charm accepts client relations BEFORE the cluster forms. +if jq -e '.applications["rabbitmq-server"]' >/dev/null 2>&1 <<<"$ST"; then + RN=$(jq -r '.applications["rabbitmq-server"].units // {} | length' <<<"$ST" || true) + MCS=$(J config -m "$MODEL" rabbitmq-server min-cluster-size | head -1 | tr -d ' \r' || true) + case "$MCS" in + ''|*[!0-9]*) + if [ "${RN:-0}" -gt 1 ]; then + fail "rabbitmq-server has $RN units but min-cluster-size is unset/unreadable ('$MCS') -- the charm accepts client relations BEFORE the cluster forms (D-009 AMENDMENT 2026-07-02)" + else + ok "rabbitmq-server min-cluster-size unset at ${RN:-0} unit(s) -- the single-unit VR0 shape (D-009); rabbitmq arity deliberately NOT asserted here" + fi ;; + *) + if [ "${RN:-x}" = "$MCS" ]; then ok "rabbitmq-server: $RN live unit(s) == min-cluster-size=$MCS" + else fail "rabbitmq-server has ${RN:-?} live unit(s) but min-cluster-size=$MCS -- the native quorum will not form (D-009 AMENDMENT)"; fi ;; + esac +fi + if [ "$CAPTURE" -eq 1 ]; then echo "================ CAPTURE: as-built BOM ================" DIR="asbuilt/$(date -u +%Y%m%d-%H%M%S)"; mkdir -p "$DIR" diff --git a/scripts/dc-cache-proxy.sh b/scripts/dc-cache-proxy.sh index 040ee37..1563ffe 100644 --- a/scripts/dc-cache-proxy.sh +++ b/scripts/dc-cache-proxy.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# scripts/dc-cache-proxy.sh -- per-DC apt CACHING PROXY +# scripts/dc-cache-proxy.sh -- per-DC apt CACHING PROXY # (apt-cacher-ng). D-135 amendment 2026-07-24 (operator-ruled): the INTERIM DC1 # apt path so the DC1 build starts WITHOUT waiting on the full ~950 GiB # debmirror. DC0 stays the full-mirror path (D-135 item 1); DC1's proxy is @@ -15,6 +15,26 @@ # 'sudo bash -s' -- check dc1 < scripts/dc-cache-proxy.sh # install: same with 'install dc1' (idempotent; safe to re-run). # +# ...EXCEPT `node`, WHICH RUNS ON A DC NODE (gate G17). Added 2026-07-29. +# G17's dc1 half named `dc-cache-proxy.sh:210-217` as its node-side check, but that +# code had NO subcommand entry point and lives inside `check`, whose other assertions +# are RACK-side facts a node cannot satisfy (the acng drop-in, the systemd net unit, +# the .4 alias on the libvirt bridge, the rack default route). Pointing a node at +# `check` therefore fails for reasons that say nothing about node reachability. +# `node` asserts ONLY what a node can, and touches ONLY curl + chronyc + the network +# -- no root, no systemctl, no virsh, no generated-file diffs: +# node: bash -s -- node dc1 < scripts/dc-cache-proxy.sh (ON the node) +# G17's three assertions, in order: +# (1) ARTIFACT REACHABILITY ON CONTENT, with an exit-code predicate -- a proxied +# fetch of archive AND UCA `Release` through :3142, asserting curl +# exit 0 AND HTTP 200 AND that the BODY is that dist's Release file. A 200 is +# not evidence: apt-cacher-ng serves its own HTML report pages with one. +# (2) NODE TIME SOURCE -- `chronyc sources` names the MAAS-served rack and NOT the +# DC edge (D-129(iv): the edge has NO NTP role; the MAAS hierarchy is the time +# authority). +# (3) An UNRECOGNISED or UNREACHABLE result REFUSES (exit 3) rather than defaulting +# to success -- "could not look" is never "nothing there". +# # PREREQS on the rack: # - apt-cacher-ng must be apt-installed first (fail-loud with the exact line). # @@ -50,13 +70,14 @@ # juju model-config apt-http-proxy=http://:3142 # (or per-node /etc/apt/apt.conf.d: Acquire::http::Proxy "http://:3142";) # -# EXIT: 0 ok | 1 check failed | 2 bad args/unknown site | 4 install failed. +# EXIT: 0 ok | 1 check failed | 2 bad args/unknown site | 3 node check REFUSED +# (could not evaluate -- never a pass) | 4 install failed. # ASCII + LF only. set -uo pipefail MODE="${1:-}"; SITE="${2:-}" -case "$MODE" in check|install) ;; *) - echo "usage: dc-cache-proxy.sh " >&2; exit 2 ;; esac +case "$MODE" in check|install|node) ;; *) + echo "usage: dc-cache-proxy.sh " >&2; exit 2 ;; esac # --------------------------------------------------------------------------- # Site table. Same MEASURED/RULED utility .4 + edge gateway per DC as @@ -69,9 +90,22 @@ # MEASURED/RULED 2026-07-23 (D-134 amendment + D-135). NOTE: dc0's RULED # path is the FULL MIRROR (D-135 item 1); this row exists for tooling # parity only -- dc0 is not deployed via the proxy. + # CONSEQUENCE, so a failing run is not mistaken for a regression (the same + # warning dc-mirror.sh's dc1 row now carries): `node dc0` will REFUSE on the + # artifact half BY DESIGN -- there is no proxy listening at 10.12.8.4:3142 on + # dc0. dc0's G17 artifact assertion is a package-path fetch from the MIRROR, + # which is dc-mirror.sh's territory, not this script's. Do NOT "fix" it by + # installing the proxy on dc0: that is a deliberate strategy change (D-135 + # AMENDMENT 2026-07-27), never a repair. UTIL_NET="vr1-dc0-metal-admin" LISTEN_CIDR="10.12.8.4/22" EDGE_GW="10.12.4.1" + # RACK_IP -- the rack's metal-admin leg, i.e. the MAAS-served time source a node + # must sync from under D-129(iv). MEASURED: scripts/dc-rack-net.sh:62 LEGS, + # docs/audit/dc-rack-net-install-20260721.txt (the metal-admin leg row), and + # the rendered dhcpd option itself -- "option ntp-servers 10.12.8.2" in + # docs/audit/d131-sub3-dhcpd-option6-20260721.txt. + RACK_IP="10.12.8.2" ;; dc1) # MEASURED/RULED 2026-07-24 (D-135 amendment -- interim DC1 apt path): @@ -80,11 +114,27 @@ UTIL_NET="vr1-dc1-metal-admin" LISTEN_CIDR="10.12.68.4/22" EDGE_GW="10.12.64.1" + # RACK_IP -- MEASURED as an ADDRESS (scripts/dc-rack-net.sh:78 LEGS; + # docs/audit/dc1-rack-net-install-20260723.txt, the metal-admin leg row; + # D-124 amendment / docs/design-decisions.md:4587,4597). HONEST LIMIT (hard rule 2): + # unlike dc0, dc1's RENDERED dhcpd `option ntp-servers` has NOT been captured. The + # expectation here is the D-129(iv) INVARIANT (MAAS hierarchy is the time authority), + # not a measurement of dc1's live DHCP render -- so if dc1 serves something else at + # first boot this check goes red, which is the intended behaviour, not a false red. + RACK_IP="10.12.68.2" ;; *) echo "FAIL: unknown site '$SITE' -- add a MEASURED row block first" >&2; exit 2 ;; esac LISTEN="${LISTEN_CIDR%%/*}" PROXY_PORT="3142" +# The two artifact paths the deploy actually needs (Ubuntu archive + UCA), shared by +# the rack-side smoke test and the node-side G17 check so the two cannot drift apart. +# An ARRAY, never a space-joined string -- word-splitting a URL list is a measured +# strict-bash trap in this repo. +ARTIFACT_URLS=( + "http://archive.ubuntu.com/ubuntu/dists/jammy/Release" + "http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/jammy-updates/caracal/Release" +) ACNG_DROPIN="/etc/apt-cacher-ng/acng.conf.d/${SITE}-cache-proxy.conf" # Net layer owned HERE (see the header note). Named for the PROXY, never "mirror". NET_HELPER="/usr/local/sbin/${SITE}-cache-proxy-net-apply" @@ -208,8 +258,7 @@ # Behavioral smoke test: fetch through the proxy (proxy mode) -- archive AND # UCA, the two the deploy needs; proves cache-miss passthrough + upstream reach. if command -v curl >/dev/null 2>&1; then - for u in "http://archive.ubuntu.com/ubuntu/dists/jammy/Release" \ - "http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/jammy-updates/caracal/Release"; do + for u in "${ARTIFACT_URLS[@]}"; do HTTP="$(curl -s -o /dev/null -w '%{http_code}' -x "http://${LISTEN}:${PROXY_PORT}" -m 20 "$u" 2>/dev/null)" [ "$HTTP" = "200" ] && say "OK proxy serves $u (200)" \ || miss "proxy did NOT serve $u (got '$HTTP')" @@ -263,7 +312,85 @@ do_check } +# --------------------------------------------------------------------------- +# node -- GATE G17, run FROM A DC NODE at Stage-5 first boot. Read-only, no root, +# and deliberately touches NOTHING rack-side: only curl, chronyc, and the network. +# Three outcomes, never two: PASS (0), FAIL (1), REFUSE (3). +# --------------------------------------------------------------------------- +NFAILED=0; NREFUSED=0 +nok(){ echo " OK $1"; } +nno(){ echo " FAIL $1"; NFAILED=1; } +nrefuse(){ echo " REFUSE $1"; NREFUSED=1; } + +do_node() { + echo "dc-cache-proxy node check ($SITE) -- from this node, against ${LISTEN}:${PROXY_PORT}" + + # -- G17 (1): artifact reachability asserted on CONTENT, with an exit-code predicate. + # A bare HTTP 200 proves nothing here: apt-cacher-ng answers its own report and error + # pages with 200, and the superseded G17 wording (`curl -sI`) exited 0 on 404/403/500. + # So all four must hold: curl exit 0, HTTP 200, the body is not HTML, and the body is + # THAT DIST's Release file. The Codename expectation is derived from the URL's own + # /dists/ path rather than hardcoded -- MEASURED 2026-07-29 against both upstreams: + # archive .../dists/jammy/Release -> "Codename: jammy" + # UCA .../dists/jammy-updates/caracal/Release -> "Codename: jammy-updates/caracal" + if ! command -v curl >/dev/null 2>&1; then + nrefuse "curl is not present on this node -- cannot evaluate artifact reachability" + else + for u in "${ARTIFACT_URLS[@]}"; do + dist="${u#*/dists/}"; dist="${dist%/Release}" + body="$(mktemp)"; cerr="$(mktemp)" + code="$(curl -sS -o "$body" -w '%{http_code}' \ + -x "http://${LISTEN}:${PROXY_PORT}" -m 30 "$u" 2>"$cerr")"; crc=$? + if [ "$crc" -ne 0 ]; then + nrefuse "could not reach $u THROUGH http://${LISTEN}:${PROXY_PORT} (curl exit $crc: $(head -1 "$cerr")) -- unreachable is not 'nothing there'" + elif [ "$code" != "200" ]; then + nno "proxy returned HTTP $code for $u (want 200)" + elif grep -qi '/dev/null 2>&1; then + nrefuse "chronyc is not present on this node -- cannot evaluate the node time source" + else + CS="$(chronyc sources 2>&1)"; crc=$? + if [ "$crc" -ne 0 ] || [ -z "$CS" ]; then + nrefuse "chronyc sources returned nothing (exit $crc) -- the time source is UNKNOWN, not absent" + elif grep -qw "$EDGE_GW" <<<"$CS"; then + nno "node time source includes the DC edge ${EDGE_GW} -- D-129(iv) gives the edge NO NTP role; the MAAS hierarchy is the time authority" + elif grep -qw "$RACK_IP" <<<"$CS"; then + nok "node time source is the MAAS-served rack ${RACK_IP} (D-129(iv))" + else + nrefuse "chronyc sources names NEITHER the MAAS rack ${RACK_IP} nor the edge ${EDGE_GW} -- unrecognised time source; refusing to default to success. Output:" + printf '%s\n' "$CS" | sed 's/^/ /' + fi + fi + + if [ "$NFAILED" -ne 0 ]; then + echo "dc-cache-proxy node ($SITE): FAIL"; exit 1 + fi + if [ "$NREFUSED" -ne 0 ]; then + echo "dc-cache-proxy node ($SITE): REFUSE -- could not evaluate; this is NOT a pass"; exit 3 + fi + echo "dc-cache-proxy node ($SITE): PASS"; exit 0 +} + case "$MODE" in check) do_check ;; install) do_install ;; + node) do_node ;; esac diff --git a/scripts/provider-bundle-check.py b/scripts/provider-bundle-check.py index f84128d..2830d28 100644 --- a/scripts/provider-bundle-check.py +++ b/scripts/provider-bundle-check.py @@ -26,13 +26,31 @@ byte compare, so zip mtimes cannot false-fail it) 8. R11 (D-020 AMENDMENT, 2026-07-27): a principal wired to an hacluster subordinate MUST carry a vip. Without one it binds a UNIT address and pacemaker has nothing - to fail over to -- decorative HA. `cluster_count` is asserted NOWHERE in this - repo, so a 3->1 rewrite of every value produces a byte-identical PASS; this is - the ruled gate hardening that makes the shape checkable at all. + to fail over to -- decorative HA. 9. prefer-ipv6 and dual-family VIP arity are COUPLED. prefer-ipv6 makes HAProxy bind :::port in ADDITION to *:port, so the two must travel together. Measured (L3-9): the overlay merge order that keeps prefer-ipv6 while silently dropping the v6 VIP legs is the one that exits 0 -- the dangerous order is the GREEN one. + 10. HA ARITY (2026-07-29 gate hardening, the second half of R11's shape): an hacluster + subordinate's `cluster_count` MUST equal its principal's `num_units`. Until this + landed, `cluster_count` was asserted NOWHERE in scripts/ or tests/ -- measured + against 20 occurrences in overlays/dc-ha-scaleup.yaml -- so rewriting every one of + those values 3->1 produced a BYTE-IDENTICAL PASS while pacemaker was free to + bootstrap the VIP off a single node: exactly the decorative HA D-121 exists to + retire. A MISSING cluster_count also FAILS (an unrecognised state refuses; the + charm default is not a substitute for the declared quorum). + 11. MACHINES-BLOCK DC IDENTITY (same hardening). ONE Office1 MAAS region sees BOTH + DCs' nodes (D-104/D-123), so the machines block's `tags=` is the only thing + keeping dc1's model off dc0's hardware. A machines overlay that PARSES but + contributes nothing -- gutted, or keyed "01" where the base says "0" -- was a + byte-identical PASS and would have allocated dc1 against `tags=openstack-vr1-dc0`. + Three assertions: every machine carries a DC tag; the whole block carries exactly + ONE distinct DC tag (a mis-keyed overlay yields base dc0 + overlay dc1 = mixed, + and fails under EITHER --dc); and that tag matches --dc. Plus: a machine that no + application places `to:` is an overlay entry that contributes nothing. + RESIDUAL LIMIT, stated plainly: a gutted overlay is only caught because --dc + DECLARES the intended DC. Validating a dc1 input while calling it dc0 is outside + what any bundle-static check can see. It is the single deploy-gate: it REPLACED the retired scripts/d057-bundle-check.py (D-060) and now also scripts/review-bundle.py (DOCFIX-070). FAIL -> exit 1. ASCII-only output. @@ -51,6 +69,14 @@ "vr1-dc0": ("10.12.4.0/22", "10.12.8.0/22", "10.12.12.0/22"), "vr1-dc1": ("10.12.64.0/22", "10.12.68.0/22", "10.12.72.0/22"), } +# The MAAS DC-scoping tag each --dc expects in the machines block's `tags=`. +# Written out per DC rather than glued from the flag ("openstack-" + dc): these are +# the literal tag strings carried by scripts/lib-hosts.sh (HOST_TAG) and +# overlays/vr1-dc1-machines.yaml, and a greppable table is what the next session reads. +DC_MACHINE_TAGS = { + "vr1-dc0": "openstack-vr1-dc0", + "vr1-dc1": "openstack-vr1-dc1", +} # D-020 AMENDMENT / R11 (RULED 2026-07-27): the band widens 50-60 -> 50-99 to admit # vault .61 and designate .62. EXPECT_PUBLIC_VIP deliberately STAYS 11 -- measured, # NEITHER vault nor designate carries a `public` binding, so they do not join that @@ -182,15 +208,31 @@ return s.rsplit(".", 1)[-1] if ip.version == 4 else s.rsplit(":", 1)[-1] +def _tags_of(spec): + """The `tags=` set from a machine spec's constraints string ({} when absent).""" + m = re.search(r"tags=(\S+)", str((spec or {}).get("constraints", ""))) + return set(m.group(1).split(",")) if m else set() + + def _role_of(spec): """Extract control/compute/storage from a machine spec's `constraints: tags=...`.""" - m = re.search(r"tags=(\S+)", str((spec or {}).get("constraints", ""))) - tags = set(m.group(1).split(",")) if m else set() + tags = _tags_of(spec) for r in ROLES: if r in tags: return r return None + +def _dc_tag_of(spec): + """The DC-scoping tag (openstack-vr1-dcN) from a machine spec, or None. + + Matched against the DC_MACHINE_TAGS table rather than a pattern, so a typo'd + tag reads as ABSENT (and fails) instead of being accepted as some new DC. + """ + tags = _tags_of(spec) + hits = sorted(t for t in tags if t in set(DC_MACHINE_TAGS.values())) + return hits[0] if len(hits) == 1 else None + def main(): ap = argparse.ArgumentParser(description="Pattern A / D-052-D-053 + placement bundle invariants") ap.add_argument("path", nargs="?", default="bundle.yaml", help="base bundle (default bundle.yaml)") @@ -348,6 +390,7 @@ # nothing to fail over to. Nothing in scripts/ or tests/ asserts # cluster_count, so this shape passed every gate until now. ha_principals = set() + ha_pairs = set() # (principal, its hacluster subordinate app) for r in rels: if not isinstance(r, list) or len(r) != 2: continue @@ -358,6 +401,7 @@ app, peer = side.split(":")[0], sides[1 - i].split(":")[0] if app in apps and (apps.get(peer) or {}).get("charm") == "hacluster": ha_principals.add(app) + ha_pairs.add((app, peer)) no_vip = sorted(a for a in ha_principals if not ((apps.get(a) or {}).get("options") or {}).get("vip")) if no_vip: @@ -367,6 +411,37 @@ elif ha_principals: oks.append("%d hacluster principal(s) all carry a VIP (R11)" % len(ha_principals)) + # -- 10. HA ARITY: cluster_count == the principal's num_units. + # The hacluster charm configures its resources only once cluster_count peers + # have joined; at 3 units with cluster_count still 1 pacemaker can bootstrap + # the VIP off ONE node, which is decorative HA wearing a 3-unit costume. + # Equality catches BOTH directions -- a stale 1 under a scaled-up overlay, + # and a 3 that outlives a scale-down. + cc_pf0 = len(fails) + for principal, sub in sorted(ha_pairs): + opts = ((apps.get(sub) or {}).get("options") or {}) + if "cluster_count" not in opts: + fails.append("%s (hacluster for %s) declares NO cluster_count -- the quorum " + "size must be explicit; a charm default is not a declaration" + % (sub, principal)); continue + cc = opts.get("cluster_count") + # `num_units` ABSENT means juju deploys exactly ONE unit. Written out rather + # than defaulted silently, because "1" here is a juju behaviour, not a guess. + nu = (apps.get(principal) or {}).get("num_units", 1) + if not isinstance(cc, int) or isinstance(cc, bool): + fails.append("%s cluster_count=%r is not an integer -- refusing to compare " + "it with %s num_units" % (sub, cc, principal)); continue + if not isinstance(nu, int) or isinstance(nu, bool): + fails.append("%s num_units=%r is not an integer -- refusing to compare it " + "with %s cluster_count" % (principal, nu, sub)); continue + if cc != nu: + fails.append("%s cluster_count=%d but principal %s num_units=%d -- DECORATIVE " + "HA: pacemaker forms (or waits for) the wrong quorum and can bind " + "the VIP off a single node (D-121 / R11)" % (sub, cc, principal, nu)) + if ha_pairs and len(fails) == cc_pf0: + oks.append("%d hacluster subordinate(s) declare cluster_count == principal " + "num_units" % len(ha_pairs)) + # -- 7. DOCFIX-071: keystone policy ships in-bundle, zip content matches source -- import os, zipfile ks = apps.get("keystone") or {} @@ -398,6 +473,42 @@ except Exception as e: fails.append("cannot read policyd zip: %s" % e) + # -- 11. MACHINES-BLOCK DC IDENTITY. The machines `tags=` is the ONLY thing keeping + # one DC's model off the other DC's hardware (one MAAS region, both DCs -- + # D-104/D-123). A machines overlay that parses but contributes nothing + # (gutted, or keyed "01" where the base says "0") used to be a byte-identical + # PASS. + # DELIBERATELY OUTSIDE the role_sep gate below, and keyed on `machines` alone. + # Keying it on role tags was ITSELF a bypass: an overlay that rewrites every + # machine's constraints to a bare `arch=amd64` strips the role tags AND the DC + # tag together, role_sep goes False, and the whole placement block -- these + # checks included -- self-skipped behind an [ok]. A bundle with NO machines + # block still skips (zero iterations), which is the VR0 shape. + if machines: + want_tag = DC_MACHINE_TAGS[args.dc] + dc_tags = {} + for mid, spec in machines.items(): + t = _dc_tag_of(spec) + if t is None: + fails.append("machine %s carries no single DC tag in its constraints " + "(expected one of %s) -- an untagged machine can be " + "allocated from EITHER DC" % (mid, "/".join(sorted(DC_MACHINE_TAGS.values())))) + else: + dc_tags[str(mid)] = t + seen = sorted(set(dc_tags.values())) + if len(seen) > 1: + fails.append("machines block MIXES DC tags %s -- the signature of a mis-keyed " + "machines overlay whose entries land beside the base's instead of " + "over them (both sets present, only one referenced)" % ", ".join(seen)) + elif seen and seen[0] != want_tag: + fails.append("machines block is tagged %s but --dc %s was requested -- deploying " + "this input would allocate %s against the OTHER DC's nodes (the " + "machines overlay is absent, gutted, or not merged)" + % (seen[0], args.dc, args.dc)) + elif seen: + oks.append("machines block: all %d machine(s) tagged %s, matching --dc %s" + % (len(dc_tags), want_tag, args.dc)) + # -- 8. Placement / anti-affinity (role separation). Activates ONLY when the # machines block carries control/compute/storage role tags; a base or # un-rendered bundle has none, so the checks self-skip (base stays green). @@ -408,12 +519,14 @@ n_ctl = sum(1 for r in roles.values() if r == "control") n_cmp = sum(1 for r in roles.values() if r == "compute") n_stg = sum(1 for r in roles.values() if r == "storage") + placed = set() for n, s in apps.items(): s = s or {} tos = [str(t) for t in (s.get("to") or [])] if not tos: continue ids = [t.split(":")[-1] for t in tos] + placed.update(ids) for t, mid in zip(tos, ids): # a. no dangling placement if mid not in roles: fails.append("%s to: references undefined machine %r" % (n, t)) @@ -427,6 +540,11 @@ if isinstance(nu, int) and nu >= 2 and len(set(ids)) < nu: fails.append("%s num_units=%d but to: has %d distinct machine(s) -- anti-affinity defeat: %s" % (n, nu, len(set(ids)), ids)) + orphans = sorted(m for m in map(str, machines) if m not in placed) + if orphans: # e. no inert machine entries + fails.append("machine(s) %s are DEFINED but no application places `to:` them " + "-- an overlay whose machine keys do not match the base's adds " + "inert entries and overrides NOTHING" % ", ".join(orphans)) for n, want, lbl in (("ceph-osd", n_stg, "storage"), ("nova-compute", n_cmp, "compute")): nu = (apps.get(n) or {}).get("num_units") # c. bare-metal role counts if isinstance(nu, int) and nu != want: diff --git a/scripts/run-tests-all.sh b/scripts/run-tests-all.sh index d30c68d..ddd0d70 100644 --- a/scripts/run-tests-all.sh +++ b/scripts/run-tests-all.sh @@ -31,10 +31,20 @@ # repo-lint 0 fail) -- the ruling explicitly warns a manifest must not be seeded from # whatever happens to be on disk. # Re-record deliberately, and say why in the commit: bash scripts/run-tests-all.sh --record-manifest +# +# LOCALE (fixed 2026-07-29). Every sort here runs under LC_ALL=C, and the comparison +# RE-SORTS the checked-in manifest the same way. MEASURED: the harness set sorts +# DIFFERENTLY under en_US.UTF-8 (vcloud) and C (voffice1) -- glibc's en_US collation +# ignores the hyphen, so `cloud-snapshot` and `cloudinit-vm` swap places. The manifest +# was recorded on vcloud, so the gauntlet read ALL GREEN there and "harness set DRIFTED" +# on voffice1 while the two trees were byte-identical. The gauntlet's own hint +# (--record-manifest) would only have MOVED the failure to vcloud. Normalising BOTH +# sides at compare time fixes it in either direction without re-recording a file the +# R15 ruling says must be re-recorded deliberately. MANIFEST="$REPO/tests/HARNESS-MANIFEST" if [ "${1:-}" = "--record-manifest" ]; then ls -1 "$REPO"/tests/*/run-tests.sh 2>/dev/null | xargs -n1 dirname | xargs -n1 basename \ - | sort > "$MANIFEST" + | LC_ALL=C sort > "$MANIFEST" echo "recorded $(wc -l < "$MANIFEST") harness name(s) to tests/HARNESS-MANIFEST" exit 0 fi @@ -64,8 +74,12 @@ exit 1 fi ACTUAL="$(ls -1 "$REPO"/tests/*/run-tests.sh 2>/dev/null | xargs -n1 dirname \ - | xargs -n1 basename | sort)" - if ! DIFF="$(diff <(cat "$MANIFEST") <(printf '%s\n' "$ACTUAL") 2>&1)"; then + | xargs -n1 basename | LC_ALL=C sort)" + # Both sides re-sorted under LC_ALL=C: the manifest is compared as a SET of names, + # never as a byte-for-byte file, so the recording host's locale cannot decide the + # verdict. The drift this gate exists to catch (a renamed/deleted/unpinned harness) + # is unaffected -- only the ORDER is normalised. + if ! DIFF="$(diff <(LC_ALL=C sort "$MANIFEST") <(printf '%s\n' "$ACTUAL") 2>&1)"; then echo "GAUNTLET: FAIL -- harness set DRIFTED from tests/HARNESS-MANIFEST." echo " '<' = pinned but MISSING (renamed or deleted -- it was neither run nor failed)" echo " '>' = present but UNPINNED (added without recording)" diff --git a/tests/cloud-assert/run-tests.sh b/tests/cloud-assert/run-tests.sh index 4fa4ecc..16a3fcb 100644 --- a/tests/cloud-assert/run-tests.sh +++ b/tests/cloud-assert/run-tests.sh @@ -15,6 +15,15 @@ cat > "$BIN/juju" <<'FB' #!/usr/bin/env bash # fake juju: replay fixtures from $FIXDIR keyed by argv patterns +# `juju config -m ` is matched POSITIONALLY (argv 4 and 5), not by +# a glob on the joined string: the A10 arity check needs per-app/per-key answers, and +# a missing fixture must reproduce real juju's "key not found" error + rc1 so the +# REFUSAL branch is exercised rather than being handed an empty success. +if [ "${1:-}" = "config" ]; then + f="$FIXDIR/config/${4:-}.${5:-}" + [ -f "$f" ] && { cat "$f"; exit 0; } + echo "ERROR key \"${5:-}\" not found" >&2; exit 1 +fi A="$*" case "$A" in *"status -m"*"keystone --format=json"*) cat "$FIXDIR/ks-status.json" ;; @@ -156,6 +165,77 @@ printf '#!/usr/bin/env bash\necho "FAIL: barbican-vault login_http=403"\nexit 1\n' > "$TMP/vkh-fail.sh" run 1 'vault-kv health FAILED' "T15 vault-kv health fail -> FAIL (D-068 item 3)" "$F" CLOUD_ASSERT_VKH="$TMP/vkh-fail.sh" +# ---- T16-T21 (2026-07-29): A10 HA arity + the non-vacuous A3 uniformity ---- +# Before these, HA was asserted by NOTHING after deploy: this script arity-checked one +# application (mysql, A2) and A3 reported "uniform across units" over a SINGLE unit. +addha() { # addha + python3 - "$1/status.json" "$2" <<'PY' +import json,sys +p,n=sys.argv[1],int(sys.argv[2]); d=json.load(open(p)) +d["applications"]["keystone"]={"charm-name":"keystone","units":{ + "keystone/%d"%i:{"workload-status":{"current":"active"}} for i in range(n)}} +# MEASURED shape from asbuilt/20260706-224851/juju-status.json: the hacluster +# subordinate app has charm-name "hacluster", subordinate-to [principal], units {}. +d["applications"]["keystone-hacluster"]={"charm-name":"hacluster", + "subordinate-to":["keystone"]} +json.dump(d,open(p,"w")) +PY + mkdir -p "$1/config" + [ -n "${3:-}" ] && printf '%s\n' "$3" > "$1/config/keystone-hacluster.cluster_count" + return 0 +} +# T16 GREEN: 3 live units, cluster_count 3 -> the check can pass, not only fail +F="$TMP/ha3"; mkgood "$F"; addha "$F" 3 3 + run 0 'keystone: 3 live unit\(s\) == keystone-hacluster cluster_count=3' \ + "T16 3 units vs cluster_count=3 -> ok" "$F" +# T17 RED: the missing unit A2's mysql-only arity check could never see +F="$TMP/ha2"; mkgood "$F"; addha "$F" 2 3 + run 1 'keystone has 2 live unit\(s\) but keystone-hacluster declares cluster_count=3' \ + "T17 2 live units vs cluster_count=3 FAILS" "$F" +# T18 RED the other way: a quorum left over from a scale-down +F="$TMP/hadown"; mkgood "$F"; addha "$F" 1 3 + run 1 'HA arity broken' "T18 scale-down leaving cluster_count=3 FAILS" "$F" +# T19 REFUSAL: cluster_count unreadable -> never "verified by default" +F="$TMP/hanocc"; mkgood "$F"; addha "$F" 3 "" + run 1 'REFUSING to report keystone' \ + "T19 unreadable cluster_count REFUSES (not a pass)" "$F" +# T20 rabbitmq clusters natively, so A10's hacluster loop cannot see it: multi-unit +# without min-cluster-size is the D-009 AMENDMENT race +F="$TMP/rabbit"; mkgood "$F"; python3 - "$F/status.json" <<'PY' +import json,sys; p=sys.argv[1]; d=json.load(open(p)) +d["applications"]["rabbitmq-server"]={"charm-name":"rabbitmq-server","units":{ + "rabbitmq-server/%d"%i:{"workload-status":{"current":"active"}} for i in range(3)}} +json.dump(d,open(p,"w")) +PY + run 1 'min-cluster-size is unset/unreadable' \ + "T20 3-unit rabbitmq with no min-cluster-size FAILS (D-009 amendment)" "$F" +mkdir -p "$TMP/rabbit-ok"; mkgood "$TMP/rabbit-ok" +cp "$F/status.json" "$TMP/rabbit-ok/status.json"; mkdir -p "$TMP/rabbit-ok/config" +printf '3\n' > "$TMP/rabbit-ok/config/rabbitmq-server.min-cluster-size" + run 0 'rabbitmq-server: 3 live unit\(s\) == min-cluster-size=3' \ + "T20b 3-unit rabbitmq with min-cluster-size=3 -> ok" "$TMP/rabbit-ok" +# T21b the skip must distinguish "none present" from "discovery missed them": an app +# NAMED *-hacluster while charm-name discovery returns nothing means the status +# shape changed, and skipping would pass over every principal. +F="$TMP/hashape"; mkgood "$F"; python3 - "$F/status.json" <<'PY' +import json,sys; p=sys.argv[1]; d=json.load(open(p)) +d["applications"]["keystone-hacluster"]={"charm-name":"SOMETHING-ELSE","subordinate-to":["keystone"]} +json.dump(d,open(p,"w")) +PY + run 1 'REFUSING to skip A10' \ + "T21b named -hacluster app but no charm-name match REFUSES the skip" "$F" + +# T21 A3 must not certify uniformity over one unit -- it used to print +# "Cluster ID uniform across units" for a single ovn-central unit. +F="$TMP/ovn1"; mkgood "$F"; python3 - "$F/status.json" <<'PY' +import json,sys; p=sys.argv[1]; d=json.load(open(p)) +u=d["applications"]["ovn-central"]["units"] +for k in list(u)[1:]: del u[k] +json.dump(d,open(p,"w")) +PY + run 1 'uniformity NOT assertable: only 1 unit' \ + "T21 single-unit ovn-central is NOT 'uniform' (vacuous pass removed)" "$F" + # T9: no admin scope -> A5-A7 HELD, exit 2 (never a silent pass) F="$TMP/held"; mkgood "$F" out="$(cd "$TMP" && env -u OS_AUTH_URL FIXDIR="$F" PATH="$BIN:$PATH" bash "$REPO/scripts/cloud-assert.sh" 2>&1)"; rc=$? diff --git a/tests/dc-cache-proxy/run-tests.sh b/tests/dc-cache-proxy/run-tests.sh index 19f07ee..9c20855 100644 --- a/tests/dc-cache-proxy/run-tests.sh +++ b/tests/dc-cache-proxy/run-tests.sh @@ -120,5 +120,87 @@ bash "$SCRIPT" check zz >/dev/null 2>&1; [ "$?" = "2" ] \ && ok "T15 unknown site -> exit 2" || no "T15 unknown site -> exit 2" +# =========================================================================== +# T16-T25 (2026-07-29): the NODE-SIDE subcommand -- gate G17. +# G17's dc1 half named `dc-cache-proxy.sh:210-217` as its node-side check, but that +# had no entry point and sat inside `check`, which asserts three RACK-side facts a +# node cannot satisfy. These cases drive `node dc1` behaviorally against a fake +# curl/chronyc on PATH, because a grep cannot tell a gate that fails from one that +# cannot. NOTE what this does NOT prove: no DC node has run it -- the nodes are +# powered off in Ready and G17's window is Stage-5 first boot -- so the GREEN below +# is a fixture green, not a node green. +# =========================================================================== +TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT +FB="$TMP/bin"; mkdir -p "$FB" +cat > "$FB/curl" <<'FAKE' +#!/usr/bin/env bash +# fake curl: honours -o and returns %{http_code} on stdout, like the real one. +out=""; url="" +while [ $# -gt 0 ]; do + case "$1" in + -o) out="$2"; shift 2 ;; + -w|-x|-m) shift 2 ;; + -*) shift ;; + *) url="$1"; shift ;; + esac +done +dist="${url#*/dists/}"; dist="${dist%/Release}" +# Body shape MEASURED 2026-07-29 against both real upstreams (see the script header). +release() { printf 'Origin: Ubuntu\nSuite: x\nCodename: %s\nComponents: main\nMD5Sum:\n abc123 42 main/binary-amd64/Packages\n' "$1" > "$out"; } +case "$(cat "$FAKEDIR/curl.mode")" in + ok) release "$dist"; printf '200' ;; + wrongdist) release "bookworm"; printf '200' ;; + nosum) printf 'Codename: %s\nComponents: main\n' "$dist" > "$out"; printf '200' ;; + html) printf 'apt-cacher-ng report\n' > "$out"; printf '200' ;; + http404) printf 'not found\n' > "$out"; printf '404' ;; + unreach) echo "curl: (7) Failed to connect to 10.12.68.4 port 3142" >&2; exit 7 ;; +esac +FAKE +cat > "$FB/chronyc" <<'FAKE' +#!/usr/bin/env bash +case "$(cat "$FAKEDIR/chrony.mode")" in + rack) printf 'MS Name/IP address Stratum Poll Reach LastRx Last sample\n^* 10.12.68.2 3 6 377 21 +12us\n' ;; + edge) printf 'MS Name/IP address Stratum Poll Reach LastRx Last sample\n^* 10.12.64.1 3 6 377 21 +12us\n' ;; + other) printf 'MS Name/IP address Stratum Poll Reach LastRx Last sample\n^* 91.189.94.4 2 6 377 21 +9us\n' ;; + empty) : ;; + err) echo "506 Cannot talk to daemon" >&2; exit 1 ;; +esac +FAKE +chmod +x "$FB/curl" "$FB/chronyc" + +noderun() { # noderun