diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 5b481dc..1e326cc 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1661,6 +1661,44 @@ figures are stale (4 machines vs 9, 50 apps vs 56, 11/11/0 vs the 13/13/0 demanded 20 lines later); and the apex dump the chain reads predates the 2026-07-27 population with nothing re-dumping or comparing it to live. +- **THE DEPLOY-PATH HALF OF THE CHAIN AUDIT ACTIONED 2026-07-29** (operator: "approved + continue"). Changelog `docs/changelog-20260728-vip-arity-gate.md` items 26-30. + **`preflight` P2 is now DC-AWARE and validates the deploy's ACTUAL overlay set.** It + hardcoded `--dc vr1-dc0` with no dc1 branch while `dc-dc-phase4` invokes it BARE as the + **dc1** gate and tells the operator to expect PASS -- so a dc1 operator got a green gate + that validated dc0's numbers against dc0's bands and never parsed a dc1 artifact. It also + passed ONE overlay where the dc0 deploy passes two. `DC=vr1-dc1 bash scripts/preflight.sh` + now merges vips + machines for that site. + **dc1's deploy command now names ALL its overlays.** phase-4 Step 4 named only the vips + overlay, so a dc1 deploy run exactly as written merged a machines block still saying + `tags=openstack-vr1-dc0` -- `grep -rn vr1-dc1-machines runbooks/ scripts/` returned ZERO, + i.e. that overlay existed in NO executable path. The step now carries the full command, + states that `dc-ha-scaleup.yaml` is deliberately excluded (R6 orders VIPs first), and adds + the `--dry-run` machines-merge confirmation that the overlay's own VERIFY-LIVE note + required and no runbook step performed. + **phase-6 deployed with NO VIP overlay and named a file that never existed.** Both its + `juju deploy` blocks -- including the real apply against a LIVE model -- passed + `overlays/${DC}-hostnames.yaml`, which HAS NEVER BEEN AUTHORED, and no VIP overlay at all; + post-ruling-3 that is a merged input with zero VIPs including the designate those commands + exist to add. Both fixed; the unbuilt hostnames overlay dropped with a note. + **`scripts/maas-role-tags.sh` SHIPPED (Fork 3, harness 9/9) -- NOT APPLIED.** Every machine + constrains on TWO tags and only the SITE tag is ever created; the three ROLE tags exist + NOWHERE, so juju's allocation constraint matches no machine and the deploy aborts at + allocation -- loud, which is the good failure mode, but it blocks BOTH DCs. Role is DERIVED + from two signals that must AGREE (the ruled D-134 octet band and the host's own name + token); disagreement REFUSES rather than picking one. Nodes match by PINNED BOOT MAC, never + hostname. Dry by default, every write read back, and an ABSENT `maas` CLI refuses saying so + rather than "MAAS unreachable". **`apply --commit` is a live MAAS mutation and is + operator-gated.** + **OWNED -- the `IFS` word-splitting trap, THIRD appearance:** `ROLES="control compute + storage"` under `IFS=$'\n\t'` does not word-split, so the script reported one absurd tag + named `'control compute storage'`. Caught by its own harness pre-ship. Same class as + commit 1's CHECK 1 death and phase-05's aborting captures -- **this repo's strict-bash + gates should use arrays, never space-separated string lists.** Also fixed: the + role-derivation refusal swallowed its own per-node diagnostic. + Gauntlet **ALL GREEN (87) on vcloud** (was 86); repo-lint 0 fail / 614 files; `preflight` + verdict unchanged in kind (2 fatal -- the absent octavia-pki secret, MAAS unreachable from + vcloud). - **THREE PLATFORM BEHAVIOURS GRADUATED to `references/platform-traps.md`** at session close, having been recorded only in this status document (which is consolidated over time, so a durable trap does not belong here alone): MAAS auto-reserves `::1`-`::ffff:ffff` on EVERY diff --git a/docs/changelog-20260728-vip-arity-gate.md b/docs/changelog-20260728-vip-arity-gate.md index 8272de8..f6adea6 100644 --- a/docs/changelog-20260728-vip-arity-gate.md +++ b/docs/changelog-20260728-vip-arity-gate.md @@ -710,3 +710,88 @@ 13/13/0 the guard 20 lines later demands) train an operator to wave the gate through. 25. The apex dump the chain reads (`vr1-office1-current-20260725.json`) predates the 2026-07-27 apex population and nothing re-dumps or compares it to live. + +--- + +# 2026-07-29 -- the deploy-path half of the chain audit + +Operator: "approved continue". Findings 16-20 from the chain audit, actioned. + +## 26. `preflight` P2 is DC-AWARE and validates the deploy's ACTUAL overlay set + +**What.** `DC=` (default `vr1-dc0`) selects the site; P2 assembles the same overlays the +deploy uses, in the same order, each added only if present. + +**Why.** P2 hardcoded `overlays/vr1-dc0-vips.yaml --dc vr1-dc0` with NO dc1 branch, while +`dc-dc-phase4-juju-bundle-per-dc.md:197-204` invokes this script BARE as the **dc1** gate +and tells the operator to expect PASS. A dc1 operator got a green gate that had validated +dc0's numbers against dc0's bands and never parsed a dc1 artifact. It also passed one +overlay where the dc0 deploy passes two -- a gate validating a different input than the +deploy is not validating the deploy. + +**Verified:** `DC=vr1-dc1 bash scripts/preflight.sh` now reports +`validating the MERGED vr1-dc1 deploy input: bundle.yaml --overlay overlays/vr1-dc1-vips.yaml +--overlay overlays/vr1-dc1-machines.yaml --dc vr1-dc1`. + +## 27. dc1's deploy command now names ALL its overlays + +`phase-4` Step 4 named ONLY the vips overlay, so a dc1 deploy run exactly as written would +have merged a machines block still saying `tags=openstack-vr1-dc0` -- precisely what +`overlays/vr1-dc1-machines.yaml` exists to prevent, since one Office1 MAAS region sees BOTH +DCs' nodes. Measured: `grep -rn vr1-dc1-machines runbooks/ scripts/` returned **ZERO** -- +the overlay existed in no executable path at all. + +Step 4 now carries the full command, states that `dc-ha-scaleup.yaml` is DELIBERATELY +excluded (R6: VIPs land BEFORE the HA overlay), points at `DC=vr1-dc1 preflight`, and adds +the `--dry-run` machines-merge confirmation that `vr1-dc1-machines.yaml`'s own VERIFY-LIVE +note required and that no runbook step performed. + +## 28. `phase-6` deployed with NO VIP overlay, and named a file that never existed + +Its two `juju deploy` blocks (dry-run and the real apply against a LIVE model) passed +`octavia-pki.yaml` and `overlays/${DC}-hostnames.yaml` -- and no VIP overlay. Since ruling 3 +that means a merged input with **zero VIPs, including the designate those commands exist to +add**; invariant 8 would reject it, but nothing runs the checker there. And +`${DC}-hostnames.yaml` **has never been authored** -- juju errors out on a missing overlay. +Both blocks now pass the per-DC vips + machines overlays, and the never-authored hostnames +overlay is dropped with a note that it is unbuilt D-106 work. + +## 29. `scripts/maas-role-tags.sh` -- Fork 3's per-role MAAS tags, with a harness + +**The blocker.** Every machine in `bundle.yaml` constrains on TWO tags +(`tags=openstack-vr1-dc0,control`). Measured: only the SITE tag is ever created +(`reenroll-hosts.sh`); the three ROLE tags are created **NOWHERE**. Juju passes `tags=` to +MAAS as an allocation constraint, so an unmatched tag yields no candidate machine. It fails +LOUDLY at allocation -- the good failure mode -- but it blocks BOTH DCs. + +**Role is DERIVED from two independent signals that must AGREE** (hard rule 2): the ruled +D-134 octet band (control .100-.119 / compute .120-.149 / storage .150-.200) and the role +token in the host's own name. Disagreement REFUSES for that node rather than picking one; +an octet in no ruled band also refuses. Nodes match MAAS by **pinned boot MAC**, never by +hostname -- MAAS renames machines at enlistment and the current names are random. + +Dry by default; every write READ BACK; an ABSENT `maas` CLI refuses with a message that +says so rather than "MAAS unreachable" (this deployment has misdiagnosed that three times). + +**Harness 9/9**, including the two directions the audit cares about: clean when everything +is tagged, red in the exact state measured today, and a REFUSAL when the two role signals +are made to disagree. + +**NOT APPLIED.** `apply --commit` is a live MAAS mutation and is operator-gated. + +## 30. OWNED: the `IFS` word-splitting trap, third appearance + +`ROLES="control compute storage"` with `IFS=$'\n\t'` does not word-split, so +`for r in $ROLES` saw ONE item and the script reported a single absurd tag named +`'control compute storage'`. Caught by its own harness before shipping. Same class as +commit 1's `pre-flight-checks` CHECK 1 death and phase-05's aborting captures. **Arrays are +immune; this repo's strict-bash gates should not use space-separated string lists.** + +Also fixed on the way: the role-derivation refusal SWALLOWED its own diagnostic -- the +python printed which node and why, the command substitution captured it, and the failure +path discarded it, leaving the operator "disagreed with itself" and nothing else. Now the +per-node reason is printed before the refusal. + +**Evidence.** Gauntlet **ALL GREEN (87) on vcloud** (was 86); repo-lint 0 fail / 614 files. +`preflight` verdict unchanged in kind (2 fatal: the absent octavia-pki secret and MAAS +unreachable from vcloud). diff --git a/runbooks/dc-dc-phase4-juju-bundle-per-dc.md b/runbooks/dc-dc-phase4-juju-bundle-per-dc.md index b715182..817df84 100644 --- a/runbooks/dc-dc-phase4-juju-bundle-per-dc.md +++ b/runbooks/dc-dc-phase4-juju-bundle-per-dc.md @@ -210,10 +210,38 @@ Follow `runbooks/phase-01-bundle-deploy.md` verbatim, with these DC-DC- specific notes, not a rewrite: -- **DC1:** deploy `bundle.yaml` WITH its per-DC VIP overlay -- - `--overlay overlays/vr1-dc1-vips.yaml` -- and make no hand edit to either - file (ruling 3, 2026-07-25; the overlay is generated, see the superseded - note above). The base bundle carries no VIPs at all now. The Octavia PKI overlay (Step 1.0) +- **DC1: deploy `bundle.yaml` with ALL of its per-DC overlays, in this order.** + Corrected 2026-07-29 -- this step previously named ONLY the VIP overlay, and a + deploy run exactly as written would have merged a machines block still saying + `tags=openstack-vr1-dc0`, which is precisely what `overlays/vr1-dc1-machines.yaml` + exists to prevent (one Office1 MAAS region sees BOTH DCs' nodes, so dc1's model + could grab a dc0 node). Measured at the time: `grep -rn vr1-dc1-machines + runbooks/ scripts/` returned ZERO -- the overlay existed in no executable path. + + ```bash + juju deploy ./bundle.yaml \ + --overlay overlays/vr1-dc1-vips.yaml \ + --overlay overlays/vr1-dc1-machines.yaml \ + --overlay overlays/octavia-pki.yaml \ + -m "${DC_MODEL}" + ``` + + Make NO hand edit to `bundle.yaml` or to either generated overlay (ruling 3, + 2026-07-25; both `*-vips.yaml` are rendered -- edit `render/values/-vips.yaml` + and re-render, and `tests/render-drift` fails the gauntlet if you do not). The base + bundle carries no VIPs at all now. + + **`overlays/dc-ha-scaleup.yaml` is DELIBERATELY NOT in that command.** Per R6 the + VIPs land BEFORE the HA overlay; applying it is a separate, later step once the base + deploy has settled. Validate whichever set you are about to deploy with + `DC=vr1-dc1 bash scripts/preflight.sh` -- P2 now merges the same overlays this + command does. + + **VERIFY BEFORE THE REAL DEPLOY:** run the identical command with `--dry-run` and + confirm the machines block merged (no offline juju merger exists to prove it, per + `overlays/vr1-dc1-machines.yaml`'s own VERIFY-LIVE note). Confirm every machine + reads `tags=openstack-vr1-dc1,`; if it still reads `-dc0`, STOP -- the + fallback is inlining the dc1 tags into a rendered dc1 bundle. The Octavia PKI overlay (Step 1.0) can reuse the existing `overlays/octavia-pki.yaml` if its CAs are still valid (10y validity, per that runbook), OR regenerate fresh per-DC PKI via Step 1.0-GEN if independent per-DC Octavia trust domains are desired -- diff --git a/runbooks/dc-dc-phase6-designate-cos-magnum.md b/runbooks/dc-dc-phase6-designate-cos-magnum.md index cd1a364..df6f5fd 100644 --- a/runbooks/dc-dc-phase6-designate-cos-magnum.md +++ b/runbooks/dc-dc-phase6-designate-cos-magnum.md @@ -358,9 +358,18 @@ **CHECK (read-only) -- jumphost -- dry-run against the live model first** ```bash +# CORRECTED 2026-07-29 (chain audit findings 18/19). This command previously passed NO +# VIP overlay -- and since ruling 3 (2026-07-25) `bundle.yaml` is VIP-FREE, so the merged +# input carried ZERO VIPs including the designate this step exists to add; +# provider-bundle-check invariant 8 would reject it, but nothing ran the checker here. +# It also named `overlays/${DC}-hostnames.yaml`, which HAS NEVER BEEN AUTHORED -- juju +# errors out on a missing overlay file. That overlay is D-106 hostname work and is not +# built; it is omitted rather than referenced, and `os-public-hostname` stays unset +# (bundle.yaml records IP-ONLY endpoints as the current posture, B5). juju deploy ./bundle.yaml \ + --overlay "overlays/${DC}-vips.yaml" \ + --overlay "overlays/${DC}-machines.yaml" \ --overlay overlays/octavia-pki.yaml \ - --overlay "overlays/${DC}-hostnames.yaml" \ -m "${DC_MODEL}" --dry-run ``` **GATE:** the diff shows ONLY the new designate/designate-bind/designate- @@ -372,8 +381,9 @@ **RUN -- jumphost** ```bash juju deploy ./bundle.yaml \ + --overlay "overlays/${DC}-vips.yaml" \ + --overlay "overlays/${DC}-machines.yaml" \ --overlay overlays/octavia-pki.yaml \ - --overlay "overlays/${DC}-hostnames.yaml" \ -m "${DC_MODEL}" ``` **GATE:** `designate`, `designate-bind`, `designate-mysql-router`, diff --git a/scripts/maas-role-tags.sh b/scripts/maas-role-tags.sh new file mode 100755 index 0000000..85726cb --- /dev/null +++ b/scripts/maas-role-tags.sh @@ -0,0 +1,186 @@ +#!/usr/bin/env bash +# scripts/maas-role-tags.sh [--commit] +# +# Fork 3 (D-121 Option C placement): create the per-role MAAS tags the bundle's machines +# block CONSTRAINS ON, and apply them to this site's nodes. +# +# WHY THIS EXISTS. Every machine in `bundle.yaml` constrains on TWO tags, e.g. +# `tags=openstack-vr1-dc0,control`. Measured 2026-07-29 (D-136 chain audit, finding 20): +# `grep -rn "tags create" scripts/ runbooks/` finds ONLY the site tag +# (`reenroll-hosts.sh`), and the three ROLE tags -- control / compute / storage -- are +# created NOWHERE. Juju passes `tags=` to MAAS as an allocation constraint, so an +# unmatched tag yields no candidate machine and the deploy aborts at allocation. That is +# the GOOD failure mode (loud, not silent), but it blocks BOTH DCs. +# +# HOW THE ROLE IS DERIVED -- two independent signals that must AGREE (hard rule 2: no +# inferred values): +# 1. the D-134-AMENDED octet band, RULED: control .100-.119 / compute .120-.149 / +# storage .150-.200, read from lib-hosts' HOST_OCTET +# 2. the role token in the host's own name (vr1-dcN--NN), read from HOST_OCTET's +# key set +# If they disagree the script REFUSES for that node rather than picking one. A node whose +# octet falls in no ruled band also refuses -- an unrecognised state is never a default. +# +# Nodes are matched to MAAS records by PINNED BOOT MAC, never by hostname: MAAS renames +# machines at enlistment (2026-07-21 incident) and the current names are random. +# +# DRY BY DEFAULT. `apply` plans and prints; nothing is written without --commit, and every +# write is READ BACK. Precedent: opnsense-plugins.sh apply ALWAYS silently dry-ran, which +# voided every prior "applied" claim from it. +# +# Exit: 0 ok/clean | 1 findings or write error | 2 could-not-evaluate. ASCII + LF. +set -uo pipefail +IFS=$'\n\t' + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO="$(cd "$SCRIPT_DIR/.." && pwd)" +MAAS_PROFILE="${MAAS_PROFILE:-admin}" +ROLES=(control compute storage) + +usage() { echo "usage: $(basename "$0") [--commit]" >&2; exit 2; } +refuse() { echo "REFUSE: $*" >&2; exit 2; } + +ACTION="${1:-}"; SITE="${2:-}"; COMMIT=0 +[ "${3:-}" = "--commit" ] && COMMIT=1 +case "$ACTION" in check|apply) ;; *) usage ;; esac +case "$SITE" in vr1-dc0|vr1-dc1) ;; *) usage ;; esac + +# An ABSENT client is not an unreachable service. This deployment has misdiagnosed that +# three times (preflight's "MAAS unreachable" was a missing binary), so they are split. +command -v maas >/dev/null 2>&1 \ + || refuse "the 'maas' CLI is ABSENT on $(hostname) -- this is not 'MAAS unreachable'. Run this where maas lives (the D-128 Plane-2 headend)." + +MACHINES="$(maas "$MAAS_PROFILE" machines read 2>/dev/null)" \ + || refuse "'maas $MAAS_PROFILE machines read' failed -- MAAS unreachable or the CLI is not logged in (profile=$MAAS_PROFILE)" +TMPD="$(mktemp -d)"; trap 'rm -rf "$TMPD"' EXIT +printf '%s' "$MACHINES" > "$TMPD/machines.json" +export MJSON="$TMPD/machines.json" + +# ---- expected roles, derived from lib-hosts + the D-134 bands ------------------------ +EXPECT="$(bash -c ' + source "'"$REPO"'/scripts/lib-hosts.sh" >/dev/null 2>&1 || exit 3 + lib_hosts_select_dc "'"$SITE"'" >/dev/null 2>&1 || exit 3 + for h in "${HOSTS[@]}"; do + printf "%s\t%s\t%s\n" "$h" "${HOST_OCTET[$h]}" "${HOST_BOOT_MAC[$h]}" + done' 2>/dev/null)" || refuse "lib-hosts did not yield hosts for $SITE" +[ -n "$EXPECT" ] || refuse "lib-hosts produced no hosts for $SITE" + +PLAN="$(printf '%s\n' "$EXPECT" | python3 -c ' +import sys +BANDS = (("control", 100, 119), ("compute", 120, 149), ("storage", 150, 200)) +rc = 0 +for line in sys.stdin.read().splitlines(): + if not line.strip(): + continue + host, octet, mac = line.split("\t") + o = int(octet) + by_band = next((r for r, lo, hi in BANDS if lo <= o <= hi), None) + by_name = next((r for r, _, _ in BANDS if "-%s-" % r in host), None) + if by_band is None: + print("REFUSE\t%s\t%s\toctet .%d is in NO ruled D-134 band" % (host, mac, o)); rc = 1 + elif by_name is None: + print("REFUSE\t%s\t%s\tname carries no role token" % (host, mac)); rc = 1 + elif by_band != by_name: + print("REFUSE\t%s\t%s\tband says %s but name says %s" % (host, mac, by_band, by_name)); rc = 1 + else: + print("OK\t%s\t%s\t%s" % (host, mac.lower(), by_band)) +sys.exit(rc) +')" || { printf '%s\n' "${PLAN:-}" | grep '^REFUSE' | while IFS=$'\t' read -r _ h m why; do + echo " [REFUSE] $h ($m): $why" >&2; done + refuse "role derivation disagreed with itself for the node(s) above -- refusing rather than picking one"; } + +echo "== maas-role-tags $ACTION $SITE $([ "$COMMIT" -eq 1 ] && echo '--commit' || echo '(DRY RUN)') ==" +echo " role derived from the D-134 band AND the host name; both must agree" +echo + +# ---- which role tags exist in MAAS? -------------------------------------------------- +TAGS="$(maas "$MAAS_PROFILE" tags read 2>/dev/null | python3 -c ' +import json,sys +try: print(" ".join(t.get("name","") for t in json.load(sys.stdin))) +except Exception: print("")')" +missing_tags="" +for r in "${ROLES[@]}"; do + case " $TAGS " in *" $r "*) echo " [ok] tag '$r' exists" ;; + *) echo " [MISS] tag '$r' does NOT exist -- every machine constraining on it would fail allocation" + missing_tags="${missing_tags}${r}\n" ;; esac +done +echo + +# ---- per-node current vs expected ---------------------------------------------------- +REPORT="$(printf '%s\n' "$PLAN" | python3 -c ' +import json, sys, os +plan = [l.split("\t") for l in sys.stdin.read().splitlines() if l.startswith("OK\t")] +machines = json.load(open(os.environ["MJSON"])) +by_mac = {} +for m in machines: + bi = m.get("boot_interface") or {} + mac = (bi.get("mac_address") or "").lower() + if mac: + by_mac[mac] = m +for _, host, mac, role in plan: + m = by_mac.get(mac) + if not m: + print("ABSENT\t%s\t%s\t%s\t-" % (host, mac, role)); continue + have = set(m.get("tag_names") or []) + print("%s\t%s\t%s\t%s\t%s" % ("HAS" if role in have else "NEED", host, mac, role, m["system_id"])) +' 2>/dev/null)" || refuse "could not correlate lib-hosts nodes with MAAS machines" + +printf '%s\n' "$REPORT" | while IFS=$'\t' read -r st host mac role sid; do + case "$st" in + HAS) printf " [ok] %-22s %s -> tag '%s' present\n" "$host" "$sid" "$role" ;; + NEED) printf " [NEED] %-22s %s -> tag '%s' MISSING\n" "$host" "$sid" "$role" ;; + ABSENT) printf " [MISS] %-22s pinned MAC %s not found in MAAS\n" "$host" "$mac" ;; + esac +done + +n_need="$(printf '%s\n' "$REPORT" | grep -c '^NEED' || true)" +n_absent="$(printf '%s\n' "$REPORT" | grep -c '^ABSENT' || true)" +n_tagmiss="$(printf '%b' "${missing_tags:-}" | grep -c . || true)" +echo +echo " summary: $n_tagmiss role tag(s) missing, $n_need node(s) needing a tag, $n_absent node(s) not in MAAS" + +if [ "$ACTION" = "check" ]; then + if [ "$n_tagmiss" -eq 0 ] && [ "$n_need" -eq 0 ] && [ "$n_absent" -eq 0 ]; then + echo "PASS: every ruled role tag exists and every node carries its role"; exit 0 + fi + echo "FAIL: role tagging incomplete -- 'apply $SITE --commit' after review"; exit 1 +fi + +# ---- apply --------------------------------------------------------------------------- +[ "$n_absent" -eq 0 ] || refuse "$n_absent node(s) have no MAAS record for their pinned MAC -- resolve before tagging" +if [ "$COMMIT" -ne 1 ]; then + echo + echo "DRY RUN -- would create $n_tagmiss tag(s) and tag $n_need node(s). Re-run with --commit." + exit 0 +fi + +rc=0 +while IFS= read -r r; do + [ -n "$r" ] || continue + if maas "$MAAS_PROFILE" tags create name="$r" comment="D-121 Option C role tag (Fork 3); consumed by bundle.yaml machines constraints" >/dev/null 2>&1; then + # read back -- a create that is not verified is a claim, not a fact + if maas "$MAAS_PROFILE" tags read 2>/dev/null | grep -q "\"name\": \"$r\""; then + echo " [applied] tag '$r' created (read back)" + else + echo " [ERROR] tag '$r' create returned ok but is ABSENT on read-back"; rc=1 + fi + else + echo " [ERROR] tag '$r' create failed"; rc=1 + fi +done < <(printf '%b' "${missing_tags:-}") + +printf '%s\n' "$REPORT" | grep '^NEED' | while IFS=$'\t' read -r _ host mac role sid; do + if maas "$MAAS_PROFILE" tag update-nodes "$role" add="$sid" >/dev/null 2>&1; then + if maas "$MAAS_PROFILE" machine read "$sid" 2>/dev/null | grep -q "\"$role\""; then + echo " [applied] $host ($sid) tagged '$role' (read back)" + else + echo " [ERROR] $host ($sid) tag '$role' not present on read-back" + fi + else + echo " [ERROR] $host ($sid) tag '$role' failed" + fi +done + +echo +[ "$rc" -eq 0 ] && echo "apply complete -- re-run 'check $SITE' to confirm convergence" +exit "$rc" diff --git a/scripts/preflight.sh b/scripts/preflight.sh index b011811..64dc07c 100644 --- a/scripts/preflight.sh +++ b/scripts/preflight.sh @@ -83,8 +83,26 @@ # relation but no vip" failures for VIPs that simply moved. P2 therefore validates # the MERGED dc0 deploy input, which is what actually gets deployed. The overlay is # passed only when present so this stays runnable on a tree without it. +# +# DC-AWARE since 2026-07-29 (chain audit finding 16). P2 used to hardcode +# `overlays/vr1-dc0-vips.yaml --dc vr1-dc0` with NO dc1 branch, while +# `runbooks/dc-dc-phase4-juju-bundle-per-dc.md` invokes this script bare as the *dc1* +# gate and tells the operator to expect PASS. A dc1 operator therefore got a green gate +# that had validated dc0's numbers against dc0's bands and never parsed a dc1 artifact. +# Set DC= to gate the other site; the default keeps every existing caller unchanged. +# +# It now assembles the SAME overlay set the deploy uses, in the SAME order, rather than a +# partial one -- the audit's point was that a gate validating a different input than the +# deploy is not validating the deploy. Each overlay is added only if present, so a tree +# missing the gitignored octavia-pki secret still runs (CHECK 0 in P4 is what fails on +# that, deliberately, and it is a different gate). +DC="${DC:-vr1-dc0}" P2_ARGS="bundle.yaml" -[ -f overlays/vr1-dc0-vips.yaml ] && P2_ARGS="$P2_ARGS --overlay overlays/vr1-dc0-vips.yaml --dc vr1-dc0" +for _ovl in "overlays/${DC}-vips.yaml" "overlays/${DC}-machines.yaml" overlays/octavia-pki.yaml; do + [ -f "$_ovl" ] && P2_ARGS="$P2_ARGS --overlay $_ovl" +done +P2_ARGS="$P2_ARGS --dc $DC" +echo " validating the MERGED $DC deploy input: $P2_ARGS" python3 scripts/provider-bundle-check.py $P2_ARGS; note $? "P2 bundle invariants" fail echo "================ P3: channel assert (charmhub) ================" diff --git a/tests/HARNESS-MANIFEST b/tests/HARNESS-MANIFEST index f19d227..4034cf5 100644 --- a/tests/HARNESS-MANIFEST +++ b/tests/HARNESS-MANIFEST @@ -33,6 +33,7 @@ ledger-scan lib-validate maas-node-power +maas-role-tags netem-link netem-sudoers node-vm diff --git a/tests/maas-role-tags/run-tests.sh b/tests/maas-role-tags/run-tests.sh new file mode 100755 index 0000000..baad3e5 --- /dev/null +++ b/tests/maas-role-tags/run-tests.sh @@ -0,0 +1,121 @@ +#!/usr/bin/env bash +# tests/maas-role-tags/run-tests.sh +# +# Offline regression harness for scripts/maas-role-tags.sh. `maas` is STUBBED on PATH and +# driven by scenario files; no live MAAS, no writes anywhere outside $TMP. +# +# The script's job is to create the per-role tags the bundle constrains on. Its gate must +# therefore be provable in BOTH directions -- clean when everything is tagged, red when a +# tag or a node is missing -- and it must REFUSE rather than guess when its two role +# signals disagree. All three are exercised here. +# +# Exit: 0 all pass | 1 any case failed. ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO="$(cd "$HERE/../.." && pwd)" +S="$REPO/scripts/maas-role-tags.sh" +TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT +PASS=0; FAIL=0 +ok() { echo " PASS $1"; PASS=$((PASS+1)); } +bad() { echo " FAIL $1"; shift; [ $# -gt 0 ] && printf ' %s\n' "$@"; FAIL=$((FAIL+1)); } + +mkdir -p "$TMP/bin" +cat > "$TMP/bin/maas" <<'EOF' +#!/usr/bin/env bash +# $1=profile $2=command ... +[ "${SC_UNREACHABLE:-0}" = "1" ] && exit 1 +case "$2 $3" in + "machines read") cat "${SC_MACHINES:?}" ;; + "tags read") cat "${SC_TAGS:?}" ;; + "tags create") echo '{"name":"stub"}' ;; + "tag update-nodes") echo '{}' ;; + "machine read") cat "${SC_MACHINES:?}" ;; +esac +exit 0 +EOF +chmod +x "$TMP/bin/maas" + +# dc0's nine pinned boot MACs, read from lib-hosts so the fixture cannot drift from it. +MACS="$(bash -c 'source "'"$REPO"'/scripts/lib-hosts.sh" >/dev/null 2>&1 + lib_hosts_select_dc vr1-dc0 >/dev/null 2>&1 + for h in "${HOSTS[@]}"; do printf "%s %s\n" "$h" "${HOST_BOOT_MAC[$h]}"; done')" +[ -n "$MACS" ] || { echo " FAIL fixture: lib-hosts yielded no dc0 hosts"; exit 1; } + +mk_machines() { # mk_machines + python3 - "$1" "$2" < "$TMP/t-all.json" +mk_tags openstack-vr1-dc0 > "$TMP/t-none.json" + +run() { # run