diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index f1736ed..e8f3612 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2641,6 +2641,22 @@ there is no copy-on-write base to roll back to. **SEPARATE THREAD, LOGGED NOT CONFLATED: dc1's rack also cannot reach `1.1.1.1` despite a healthy edge on both legs and a default route via it** -- an edge forwarding/NAT question at dc1, off the deploy path, NOT diagnosed here. + **>>> BOTH GAPS ARE NOW CLOSED, 2026-08-02: `scripts/dc-egress-check.sh` + preflight gate + P9, wired into the restart procedure and the phase-4 runbook. <<<** The check is LAYERED -- + A1 default route -> A2 **the edge ANSWERS** -> A3 traffic leaves (ICMP *and* TCP, so an + ICMP-filtering path cannot fake a failure) -> A4 the three upstreams the deploy needs -- and + it reports the FIRST failure as the cause while the rest SKIP, because a gate that restates + one fault as five trains people to skim it. **PROVEN LIVE ON TWO DIFFERENT FAILURE MODES:** + dc0 `A2 FAIL ... neigh '10.12.4.1 dev virbr5 FAILED'` with A3/A4 skipped; dc1 `A2 OK (edge + ANSWERS)` with A3/A4 failing -- a different diagnosis, traffic not leaving a healthy edge. + **P9** WARNs off-rack with the exact command, FAILs on the rack when egress is broken, and + WARNs if the checker is absent (a gate that vanishes must not read as green). **GAP 2 IS + CLOSED BY THE WIRING, not by the script:** `ops-restart-procedure.md` gains **Stage 0, before + anything that fetches**, and `dc-dc-phase4-juju-bundle-per-dc.md` gains **Step 3.9** before + the deploy -- so the post-reboot check set is no longer a matter of judgement. A mutation + pass caught a DECORATIVE test of my own (T11 never reached A4's branch; T14 added so only A4 + can fail it). `tests/dc-egress-check` **14/14**, `tests/preflight` **43/43**, gauntlet **ALL + GREEN (97)**, manifest 96 -> 97. **>>> TWO GAPS THIS EXPOSES, both worth more than the incident. <<<** **(1) NO GATE ANYWHERE TESTS DC EGRESS** -- `dc-rack-net.sh check dc0` PASSES with egress dead because it asserts bridge legs and unit states, not reachability, and preflight has no egress gate either; a diff --git a/docs/changelog-20260801-stage5-deploy-prep.md b/docs/changelog-20260801-stage5-deploy-prep.md index 5eb1d97..074a686 100644 --- a/docs/changelog-20260801-stage5-deploy-prep.md +++ b/docs/changelog-20260801-stage5-deploy-prep.md @@ -250,3 +250,54 @@ **Revert.** `dc-node-v6-carve.py replace vr1-dc0 --v6-family ula --commit` restores the node statics; both families remain in MAAS until step 6. Nothing was deleted at any point. + +--- + +## Item 8 -- BOTH gaps closed: a DC EGRESS gate, wired where it will actually run + +**The gaps.** (1) No gate anywhere tested DC egress -- `dc-rack-net.sh check` PASSED through +a 19-hour outage because it asserts bridge legs and unit states, and **a leg is not a path**; +preflight had no egress gate at all; the mirror answered 200 from its own nginx the whole +time, and **a 200 from a LOCAL service says nothing about upstream**. (2) This session's own +post-reboot verification cited five true facts, none of which tested egress -- i.e. the check +set was a matter of my judgement, and my judgement was wrong. + +**`scripts/dc-egress-check.sh check ` (NEW).** Site-keyed, runs ON the rack like +`dc-mirror.sh` / `dc-rack-net.sh`, read-only. **LAYERED, and that is the point:** A1 default +route -> A2 the edge ANSWERS -> A3 traffic leaves (probed by ICMP *and* TCP so an +ICMP-filtering path cannot fake a failure) -> A4 the three upstreams the deploy needs. The +first failure is reported as the cause and the rest SKIP, because a gate that restates one +fault as five trains people to skim it. A4 does not duplicate the mirror/proxy/snap checkers: +those assert the LOCAL service SERVES, this asserts the UPSTREAM it syncs FROM is reachable. +Both matter and they failed independently on 08-01. + +**PROVEN LIVE ON TWO DIFFERENT FAILURE MODES, not fixture-green.** dc0: `A1 OK` -> +`A2 FAIL ... neigh: '10.12.4.1 dev virbr5 FAILED'`, A3/A4 skipped -- the outage, named at its +layer. dc1: `A1 OK, A2 OK (edge ANSWERS)` -> A3/A4 fail -- a *different* diagnosis, traffic +not leaving a healthy edge. That discrimination is the whole value. + +**preflight gate P9.** WARNs off-rack with the exact command (preflight normally runs on +voffice1, which is never a rack -- "not the rack" is legitimate, same reasoning as P8), FAILs +when run on the rack with egress broken, and WARNs if the checker is missing: **a gate that +vanishes must not read as a green one.** + +**WIRED WHERE IT BITES, which is what actually closes gap 2** -- the check set is no longer a +judgement call: `ops-restart-procedure.md` gains **Stage 0, before anything that fetches**, +with a first-FAIL-to-cause table; `dc-dc-phase4-juju-bundle-per-dc.md` gains **Step 3.9** +immediately before the deploy, stating that Step 3.5's `apt-mirror` covers apt ONLY while the +agent stream and snaps have no local copy, and that **P9 passing is not a substitute for +running it on the rack**. + +**A MUTATION PASS CAUGHT A DECORATIVE TEST OF MY OWN.** T11 claimed to cover "unrecognised +HTTP code" but its fixture made A3's anchor odd, which A3 catches first -- so A4's `*)` branch +was never exercised and flipping it to `ok()` left the suite fully green. Added T14 with the +A3 anchor healthy and only the UCA upstream odd, so only A4 can fail it; the mutation now +kills T14. The P9 fixture also had to default to HEALTHY: modelling an unhealthy default drags +every case testing something else, which is the moment someone demotes the gate (mkfix's own +comment records exactly that happening when P7 landed). + +**Verification.** `tests/dc-egress-check` **14/14** (3 mutations, restored sha256-identical); +`tests/preflight` 39 -> **43/43** (2 further mutations, each killing a named case); gauntlet +**ALL GREEN (97)**, manifest re-recorded 96 -> 97; repo-lint 0 fail / 1 warn. + +**Revert.** `git revert `, then `bash scripts/run-tests-all.sh --record-manifest`. diff --git a/runbooks/dc-dc-phase4-juju-bundle-per-dc.md b/runbooks/dc-dc-phase4-juju-bundle-per-dc.md index 745edbd..261972e 100644 --- a/runbooks/dc-dc-phase4-juju-bundle-per-dc.md +++ b/runbooks/dc-dc-phase4-juju-bundle-per-dc.md @@ -447,6 +447,31 @@ --- +## Step 3.9 -- DC EGRESS gate, immediately before the deploy [read-only, REQUIRED] + +> **ADDED 2026-08-02 after a 19-hour dc0 egress outage no gate could see.** Step +> 3.5 already points the model at THIS DC's artifact source, but that only covers +> **apt**. The deploy also needs two sources with **no local copy at all** -- +> `streams.canonical.com` for juju agent binaries and `api.snapcraft.io` for +> snaps (the D-135 items 2-3 gap). On 2026-08-01 the mirror served 200 all day +> while the DC had no path off-site, so an apt-only check would have passed and +> the deploy would have failed the same way the 2026-07-31 attempt did. + +```bash +ssh 'bash -s -- check ' < scripts/dc-egress-check.sh +``` + +**GATE:** exit 0. Capture the output and cite it in the commit that records the +deploy. Exit 2 is REFUSE ("could not look" -- usually run off the rack), never a +pass. The check is layered: read the FIRST FAIL as the cause. `A2` failing means +the edge itself is down and nothing downstream can work. + +`preflight.sh` gate **P9** covers the same ground but WARNs when run off the rack +(which is where preflight normally runs), so **P9 passing is not a substitute for +running this on the rack.** + +--- + ## Step 4 -- Deploy `bundle.yaml` + THIS DC's overlays [MUTATION, gated] This step carries its OWN DC-parameterised deploy command. It no longer says diff --git a/runbooks/ops-restart-procedure.md b/runbooks/ops-restart-procedure.md index 025fd46..baa2801 100644 --- a/runbooks/ops-restart-procedure.md +++ b/runbooks/ops-restart-procedure.md @@ -60,6 +60,41 @@ ## Power-on / recovery +### Stage 0 -- DC EGRESS, before anything that fetches [VR1 DC-DC only] + +> **RUN THIS FIRST AND DO NOT SKIP IT.** Added 2026-08-02 after a **19-hour dc0 +> egress outage that every existing check missed.** A containment-VM reboot left +> the DC's OPNsense edge broken (a partially-applied update), so the DC had no +> path off-site -- and `dc-rack-net.sh check dc0` PASSED throughout (it asserts +> bridge legs and unit states; **a leg is not a path**), the mirror answered +> HTTP 200 from its own nginx (**a 200 from a LOCAL service says nothing about +> upstream**), and the session that verified that reboot cited five true facts, +> none of which tested egress. It surfaced only when a nightly debmirror timer +> failed, three layers away from the cause. + +```bash +# ON THE RACK -- every value the check uses describes the rack, so running it +# anywhere else measures a different machine's path. +ssh 'bash -s -- check ' < scripts/dc-egress-check.sh +``` + +**GATE:** exit 0. The check is LAYERED and reports the FIRST failure as the +cause rather than the last, so read the first FAIL and fix that: + +| First FAIL | What it means | +|---|---| +| `A1` | no default route, or it points somewhere other than this site's edge | +| `A2 edge ... does NOT answer` | **the edge itself is down** -- this is the 2026-08-01 outage. Nothing below it can work; do not chase the downstream symptoms | +| `A3` | the edge answers but traffic is not leaving it -- forwarding/NAT on the edge | +| `A4` | egress works; one upstream artifact source is down. Named individually so a single dead source is not read as a total outage | + +Exit 2 is REFUSE ("could not look"), never a pass -- most often it means the +check was run somewhere other than the rack. + +**Why here and not later:** everything from Stage 1 onward fetches. Discovering +a dead edge after the agents start is how a half-built model happens, and unlike +bootstrap the recovery is not "re-open egress and retry". + ### Stage 1 -- foundation **GATE:** every juju machine agent reports `started` (typically 2-5 min). ```bash diff --git a/scripts/dc-egress-check.sh b/scripts/dc-egress-check.sh new file mode 100755 index 0000000..e36cdd1 --- /dev/null +++ b/scripts/dc-egress-check.sh @@ -0,0 +1,191 @@ +#!/usr/bin/env bash +# scripts/dc-egress-check.sh +# +# Site-keyed DC EGRESS gate. RUNS ON THE RACK HOST, like dc-mirror.sh / +# dc-cache-proxy.sh / dc-rack-net.sh / dc-snap-proxy.sh. Read-only: it probes, +# it never configures. +# +# WHY THIS EXISTS -- a measured 19-hour outage that no gate could see. +# On 2026-08-01 05:48 the dc0 containment VM rebooted and the dc0 OPNsense edge +# came up broken (a partially-applied update; missing libcrypto.so.17 / +# libpython3.13.so.1.0, dropping to a single-user prompt). The DC therefore had +# NO path off-site. Nothing detected it for 19 hours: +# * `dc-rack-net.sh check dc0` PASSED -- it asserts bridge legs and unit +# states, which were all fine. A leg is not a path. +# * `preflight.sh` had no egress gate at all. +# * the mirror answered HTTP 200 from its own nginx -- a 200 from a LOCAL +# service says nothing about upstream. +# * the session that verified the reboot cited five true facts (rack units, +# region API 200, mirror 200, snap proxy LISTENING, juju controller +# connected) and none of them tested egress. +# It surfaced only when a nightly debmirror timer failed, and it surfaced as +# "mirror sync FAILED" -- three layers away from the cause. This gate exists so +# the cause is named at the layer it happens. +# +# LAYERING IS THE POINT. Each assertion is a strictly deeper dependency of the +# next, and the FIRST failure is reported as the cause rather than the last: +# A1 default route -> exists and points at THIS site's ruled edge LAN address +# A2 edge answers -> the gateway itself replies (this is what broke on 08-01) +# A3 traffic leaves -> an off-site anchor answers, probed BOTH by ICMP and by +# TCP, so an ICMP-filtering path cannot fake a failure +# A4 upstreams -> the three artifact sources the deploy actually needs +# A4 does NOT duplicate dc-mirror.sh / dc-cache-proxy.sh / dc-snap-proxy.sh: +# those assert the LOCAL service serves content; this asserts the UPSTREAM those +# services sync FROM is reachable. Both matter and they fail independently -- on +# 08-01 the mirror served fine and its upstream was gone. +# +# ASSERT ON CONTENT, NEVER ON EXISTENCE, and REFUSE rather than default to a +# pass: an unrecognised state, a missing tool, or the wrong host all exit 2. +# "Could not look" is never "nothing there". +# +# Exit: 0 all pass | 1 an assertion FAILED | 2 REFUSE (could not evaluate). +# ASCII + LF. +set -uo pipefail + +ACTION="${1:-}"; SITE="${2:-}" +case "$ACTION" in + check) ;; + *) echo "usage: dc-egress-check.sh check " >&2; exit 2 ;; +esac + +PASS=0; FAILN=0 +ok() { PASS=$((PASS+1)); printf ' OK %s\n' "$1"; } +bad() { FAILN=$((FAILN+1)); printf ' FAIL %s\n' "$1"; } +refuse() { printf 'REFUSE: %s\n' "$1" >&2 + printf ' (could not evaluate -- this is NOT a pass)\n' >&2; exit 2; } + +# --------------------------------------------------------------------------- +# Site table. ADD A SITE ONLY WITH MEASURED VALUES (hard rule 2). Every row +# cites where its value was measured; an uncited row is not admissible. +# --------------------------------------------------------------------------- +case "$SITE" in + dc0|vr1-dc0) + # MEASURED 2026-08-02 on vvr1-dc0: `ip route` -> "default via 10.12.4.1 dev + # virbr5"; `ip -4 -o addr show virbr5` -> 10.12.4.2/22. Matches D-134's + # provider-public band and dc-rack-net.sh's dc0 rows. + EDGE_LAN="10.12.4.1"; RACK_LEG="10.12.4.2" + # MEASURED 2026-08-01 (dc0 snap proxy built + proven, squid on the D-134 + # utility .4): LISTEN 10.12.8.4 port 3129. + SNAP_PROXY="http://10.12.8.4:3129" + ;; + dc1|vr1-dc1) + # MEASURED 2026-08-02 on vvr1-dc1: `ip route` -> "default via 10.12.64.1 dev + # virbr4". Rack provider-public leg 10.12.64.2/22 recorded at the 2026-07-23 + # dc1 edge addressing (changelog-20260723-g12-dc1-edge.md). + EDGE_LAN="10.12.64.1"; RACK_LEG="10.12.64.2" + # dc1 has NO snap proxy ruled (D-135 gives it the apt CACHING PROXY at + # 10.12.68.4:3142; the snap path is the D-135 items 2-3 gap). Probe direct. + SNAP_PROXY="" + ;; + *) refuse "unknown site '$SITE' -- add a MEASURED row block first" ;; +esac + +command -v ip >/dev/null 2>&1 || refuse "no 'ip' on this host -- cannot evaluate egress" +command -v curl >/dev/null 2>&1 || refuse "no 'curl' on this host -- cannot evaluate egress" + +# WRONG-HOST GUARD. Every value above describes the RACK. Run anywhere else and +# the probes measure a different machine's path -- the exact class recorded for +# dc-mirror.sh, which reports every item MISS when run from voffice1. +ip -4 -o addr show 2>/dev/null | grep -qw "${RACK_LEG}/22" \ + || refuse "this host does not carry ${RACK_LEG}/22, so it is not the ${SITE} rack -- \ +run it there (ssh 'bash -s -- check ${SITE}' < scripts/dc-egress-check.sh)" + +echo "== dc-egress-check $SITE ==" +echo " edge LAN (gateway): $EDGE_LAN rack leg: $RACK_LEG" +[ -n "$SNAP_PROXY" ] && echo " snap proxy: $SNAP_PROXY" || echo " snap proxy: none ruled for this site" +echo + +# ---- A1 default route ------------------------------------------------------ +DEF="$(ip route 2>/dev/null | awk '/^default/{print $3; exit}')" +if [ -z "$DEF" ]; then + bad "A1 no default route on this rack -- nothing can leave the DC" +elif [ "$DEF" != "$EDGE_LAN" ]; then + bad "A1 default route points at '$DEF', not this site's edge '$EDGE_LAN'" +else + ok "A1 default route via $EDGE_LAN (this site's edge)" +fi + +# ---- A2 the edge itself answers -------------------------------------------- +# THE 2026-08-01 FAILURE. The edge VM was RUNNING with both NICs attached and +# did not answer at L2: `ip neigh` read "10.12.4.1 FAILED" while two neighbours +# on the same segment answered. Asserting the gateway REPLIES -- not that a +# route to it exists, not that the VM is running -- is what makes this catch it. +if ping -c2 -W3 "$EDGE_LAN" >/dev/null 2>&1; then + ok "A2 edge $EDGE_LAN ANSWERS" + EDGE_UP=1 +else + NEIGH="$(ip neigh show "$EDGE_LAN" 2>/dev/null | head -1)" + bad "A2 edge $EDGE_LAN does NOT answer -- neigh: '${NEIGH:-none}'. The DC has no \ +path off-site; everything below fails as a CONSEQUENCE, not as separate faults" + EDGE_UP=0 +fi + +# ---- A3 traffic actually leaves -------------------------------------------- +# Probed TWO ways on purpose: an ICMP-only probe would report a false outage on +# a path that filters ICMP, and a TCP-only probe would miss a DNS-layer break. +if [ "$EDGE_UP" -eq 1 ]; then + ping -c2 -W3 1.1.1.1 >/dev/null 2>&1 \ + && ok "A3 off-site ICMP anchor 1.1.1.1 answers" \ + || bad "A3 off-site ICMP anchor 1.1.1.1 does NOT answer" + C="$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 --noproxy '*' \ + http://archive.ubuntu.com/ubuntu/dists/jammy/Release 2>/dev/null)" + case "$C" in + 200) ok "A3 off-site TCP/HTTP anchor answers 200 (archive.ubuntu.com)" ;; + 000|"") bad "A3 off-site TCP/HTTP anchor unreachable (curl code '${C:-none}')" ;; + *) bad "A3 off-site TCP/HTTP anchor answered '$C', not 200 -- unrecognised, \ +not assumed healthy" ;; + esac +else + bad "A3 SKIPPED -- the edge does not answer, so an off-site probe would only \ +restate A2. Fix the edge first" +fi + +# ---- A4 the upstreams the deploy needs ------------------------------------- +# These are the sources the LOCAL artifact services sync FROM. dc-mirror.sh / +# dc-cache-proxy.sh / dc-snap-proxy.sh assert the local service SERVES; this +# asserts its upstream is REACHABLE. On 2026-08-01 the mirror served 200 all day +# with its upstream gone, which is why both halves are needed. +probe() { # $1 label, $2 url, $3 optional proxy + local lbl="$1" url="$2" px="${3:-}" code + if [ -n "$px" ]; then + code="$(curl -s -o /dev/null -w '%{http_code}' --max-time 20 -x "$px" "$url" 2>/dev/null)" + else + code="$(curl -s -o /dev/null -w '%{http_code}' --max-time 20 --noproxy '*' "$url" 2>/dev/null)" + fi + case "$code" in + 200|30[0-9]) ok "A4 $lbl reachable (HTTP $code)" ;; + 000|"") bad "A4 $lbl UNREACHABLE (curl code '${code:-none}')" ;; + *) bad "A4 $lbl answered '$code' -- unrecognised, not assumed healthy" ;; + esac +} +if [ "$EDGE_UP" -eq 1 ]; then + # apt upstream -- what debmirror pulls. THIS is what failed on 2026-08-02: + # "500 Can't connect to ubuntu-cloud.archive.canonical.com". + probe "apt upstream (archive.ubuntu.com)" \ + "http://archive.ubuntu.com/ubuntu/dists/jammy/Release" + probe "UCA upstream (ubuntu-cloud.archive.canonical.com)" \ + "http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/jammy-updates/caracal/Release" + # juju agent binaries -- the bootstrap/deploy dependency with NO local copy. + probe "juju agent stream (streams.canonical.com)" \ + "https://streams.canonical.com/juju/tools/" + # snaps -- through this site's ruled proxy where one exists, else direct. + if [ -n "$SNAP_PROXY" ]; then + probe "snap store VIA this site's proxy" \ + "https://api.snapcraft.io/v2/snaps/info/core22" "$SNAP_PROXY" + else + probe "snap store (direct -- no proxy ruled for this site)" \ + "https://api.snapcraft.io/v2/snaps/info/core22" + fi +else + bad "A4 SKIPPED -- the edge does not answer. Reporting four more failures here \ +would restate one cause as five faults" +fi + +echo +echo "RESULT: pass=$PASS fail=$FAILN" +if [ "$FAILN" -eq 0 ]; then + echo "PASS: dc-egress-check $SITE" + exit 0 +fi +echo "FAIL: dc-egress-check $SITE -- $FAILN assertion(s) failed" +exit 1 diff --git a/scripts/preflight.sh b/scripts/preflight.sh index ffb5eb3..01d3db4 100644 --- a/scripts/preflight.sh +++ b/scripts/preflight.sh @@ -324,6 +324,45 @@ esac fi +echo "================ P9: DC egress (the path off-site) ================" +# ADDED 2026-08-02 after a 19-hour dc0 egress outage that NO gate could see. The dc0 +# OPNsense edge came up broken after a reboot, so the DC had no path off-site -- +# and `dc-rack-net.sh check dc0` PASSED throughout (it asserts bridge legs and unit +# states; a leg is not a path), the mirror answered 200 from its own nginx, and +# preflight had no egress gate at all. It surfaced only when a nightly debmirror +# timer failed, three layers from the cause. +# +# The check RUNS ON THE RACK -- every value it uses describes the rack, and run +# anywhere else it measures a different machine's path (the recorded dc-mirror.sh +# trap). preflight normally runs on voffice1, which is NOT a rack, so this WARNS +# with the exact command rather than failing: "not the rack" is a legitimate state, +# the same reasoning as P8's substrate guard. It FAILS only when preflight is itself +# run on the rack and egress is genuinely broken. +EGRESS_SH="$REPO/scripts/dc-egress-check.sh" +EGRESS_SITE="${DC#vr1-}" +if [ ! -x "$EGRESS_SH" ]; then + echo " [warn] scripts/dc-egress-check.sh missing or not executable -- DC egress NOT evaluated" + note 2 "P9 DC egress" warn +elif bash "$EGRESS_SH" check "$EGRESS_SITE" >/tmp/p9.$$ 2>&1; then + echo " [ok] DC egress verified from this host for $DC" + note 0 "P9 DC egress" warn +else + P9_RC=$? + if [ "$P9_RC" -eq 2 ]; then + echo " [warn] not the $DC rack, so DC egress was NOT evaluated here. This is" + echo " 'could not look', never 'nothing there'. Run it where it belongs:" + echo " ssh <$DC rack> 'bash -s -- check $EGRESS_SITE' < scripts/dc-egress-check.sh" + echo " REQUIRED before Step 4 and after any DC reboot." + note 2 "P9 DC egress" warn + else + echo " [FAIL] DC egress is BROKEN for $DC -- the deploy needs the juju agent stream" + echo " and the snap upstream, neither of which is local:" + sed 's/^/ /' /tmp/p9.$$ | grep -E 'FAIL|REFUSE' | head -6 + note 1 "P9 DC egress" fail + fi +fi +rm -f /tmp/p9.$$ + echo "================ P6: stage-2 reminders (NOT run here) ================" echo " - after 'juju add-model': bash scripts/juju-spaces-check.sh" echo " - with sudo: bash scripts/osd-blank-check.sh" diff --git a/tests/HARNESS-MANIFEST b/tests/HARNESS-MANIFEST index 1f5cd20..bdb15e2 100644 --- a/tests/HARNESS-MANIFEST +++ b/tests/HARNESS-MANIFEST @@ -23,6 +23,7 @@ dc-dc-rbd-mirror dc-dc-whole-host-budget dc-edge-wan-import +dc-egress-check dc-mirror dc-node-carve dc-node-v6-carve diff --git a/tests/dc-egress-check/run-tests.sh b/tests/dc-egress-check/run-tests.sh new file mode 100755 index 0000000..074268f --- /dev/null +++ b/tests/dc-egress-check/run-tests.sh @@ -0,0 +1,140 @@ +#!/usr/bin/env bash +# tests/dc-egress-check/run-tests.sh -- offline harness for scripts/dc-egress-check.sh. +# Stubs `ip`, `ping` and `curl` in a fakebin so NOTHING touches a live rack or the +# internet. Exit: 0 all pass | 1 any case failed. ASCII + LF. +# +# WHAT THIS HARNESS IS FOR. The gate exists because a 19-hour dc0 egress outage went +# undetected: dc-rack-net.sh check PASSED throughout (it asserts legs and units, not a +# path) and the mirror answered 200 from its own nginx the whole time. So the cases below +# are not decoration -- each one is a shape that ACTUALLY OCCURRED or that would have made +# the outage invisible again. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +S="$(cd "$HERE/../../scripts" && pwd)/dc-egress-check.sh" +TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT +PASS=0; FAIL=0 +ok() { PASS=$((PASS+1)); printf ' PASS %s\n' "$1"; } +bad() { FAIL=$((FAIL+1)); printf ' FAIL %s\n' "$1"; [ $# -gt 1 ] && printf ' %s\n' "$2"; } + +# mkfix +mkfix() { + local d="$1" onrack="$2" gw="$3" edge="$4" anchor="$5" up="$6" + mkdir -p "$d/fakebin" + # ip: addr show decides the wrong-host guard; route decides A1 + { echo '#!/usr/bin/env bash' + echo "ONRACK=$onrack; GW=$gw" + echo 'if [ "$1" = "-4" ]; then [ "$ONRACK" = yes ] && echo "2: virbr5 inet 10.12.4.2/22 brd 10.12.7.255 scope global virbr5"; exit 0; fi' + echo 'if [ "$1" = "route" ]; then' + echo ' case "$GW" in edge) echo "default via 10.12.4.1 dev virbr5" ;;' + echo ' other) echo "default via 10.10.0.1 dev enp1s0" ;;' + echo ' none) : ;; esac; exit 0; fi' + echo 'if [ "$1" = "neigh" ]; then echo "10.12.4.1 FAILED"; exit 0; fi' + echo 'exit 0' + } > "$d/fakebin/ip"; chmod +x "$d/fakebin/ip" + # ping: the edge and the off-site anchor answer independently + { echo '#!/usr/bin/env bash' + echo "EDGE=$edge; ANCHOR=$anchor" + echo 'for a; do case "$a" in' + echo ' 10.12.4.1) [ "$EDGE" = up ] && exit 0 || exit 1 ;;' + echo ' 1.1.1.1) [ "$ANCHOR" = up ] && exit 0 || exit 1 ;;' + echo 'esac; done; exit 1' + } > "$d/fakebin/ping"; chmod +x "$d/fakebin/ping" + # curl: one code per upstream, so each A4 probe can fail alone + { echo '#!/usr/bin/env bash' + echo "UP=$up; ANCHOR=$anchor" + echo 'URL=""; for a; do case "$a" in http*) URL="$a" ;; esac; done' + echo 'case "$URL" in' + echo ' *archive.ubuntu.com*) if [ "$ANCHOR" = down ]; then echo -n 000; elif [ "$UP" = aptdown ]; then echo -n 000; elif [ "$UP" = weird ]; then echo -n 403; else echo -n 200; fi ;;' + echo ' *ubuntu-cloud*) if [ "$UP" = aptdown ]; then echo -n 000; elif [ "$UP" = weirdup ]; then echo -n 403; else echo -n 200; fi ;;' + echo ' *streams.canonical*) [ "$UP" = jujudown ] && echo -n 000 || echo -n 200 ;;' + echo ' *api.snapcraft.io*) [ "$UP" = snapdown ] && echo -n 000 || echo -n 200 ;;' + echo ' *) echo -n 200 ;; esac; exit 0' + } > "$d/fakebin/curl"; chmod +x "$d/fakebin/curl" +} +run() { local d="$1"; shift + OUT="$(PATH="$d/fakebin:$PATH" bash "$S" "$@" 2>&1)"; RC=$?; } + +# T01 the healthy path. +mkfix "$TMP/t1" yes edge up up ok; run "$TMP/t1" check dc0 +[ "$RC" -eq 0 ] && ok "T01 fully healthy egress PASSES (rc=0)" || bad "T01 healthy run failed (rc=$RC)" "$OUT" + +# T02 THE 2026-08-01 OUTAGE, exactly: the edge does not answer. This is the case the +# whole gate exists for -- dc-rack-net.sh check passed in this state. +mkfix "$TMP/t2" yes edge down down ok; run "$TMP/t2" check dc0 +{ [ "$RC" -eq 1 ] && printf '%s' "$OUT" | grep -q "A2 edge 10.12.4.1 does NOT answer"; } \ + && ok "T02 edge-not-answering FAILS and names the edge (the 08-01 outage)" \ + || bad "T02 did not fail/name the edge (rc=$RC)" "$OUT" + +# T03 ...and it reports ONE cause, not five. A gate that restates a single fault as a +# cascade trains people to skim it. +printf '%s' "$OUT" | grep -q "A3 SKIPPED" && printf '%s' "$OUT" | grep -q "A4 SKIPPED" \ + && ok "T03 downstream assertions SKIP rather than restate one cause as five faults" \ + || bad "T03 downstream assertions did not skip" "$OUT" + +# T04 wrong host REFUSES (2). Every value in the site table describes the RACK; run +# elsewhere and the probes measure a different machine -- the recorded dc-mirror.sh trap. +mkfix "$TMP/t4" no edge up up ok; run "$TMP/t4" check dc0 +[ "$RC" -eq 2 ] && ok "T04 wrong host REFUSES (rc=2), never reports a clean tree" \ + || bad "T04 wrong host did not refuse (rc=$RC)" "$OUT" + +# T05 unknown site REFUSES rather than guessing a gateway. +run "$TMP/t1" check dc9 +[ "$RC" -eq 2 ] && ok "T05 unknown site REFUSES (rc=2)" || bad "T05 unknown site not refused (rc=$RC)" + +# T06 a default route pointing somewhere OTHER than this site's edge is a FAIL, not a +# pass -- routing out through the wrong path is how a DC silently loses its isolation. +mkfix "$TMP/t6" yes other up up ok; run "$TMP/t6" check dc0 +{ [ "$RC" -eq 1 ] && printf '%s' "$OUT" | grep -q "not this site's edge"; } \ + && ok "T06 default route via the WRONG gateway FAILS" || bad "T06 wrong gateway not caught" "$OUT" + +# T07 no default route at all is a FAIL. +mkfix "$TMP/t7" yes none up up ok; run "$TMP/t7" check dc0 +{ [ "$RC" -eq 1 ] && printf '%s' "$OUT" | grep -q "no default route"; } \ + && ok "T07 absent default route FAILS" || bad "T07 absent default route not caught" "$OUT" + +# T08 THE 2026-08-02 SYMPTOM: edge and anchor fine, apt upstream gone. The mirror served +# 200 from its own nginx throughout, so only an UPSTREAM probe sees this. +mkfix "$TMP/t8" yes edge up up aptdown; run "$TMP/t8" check dc0 +{ [ "$RC" -eq 1 ] && printf '%s' "$OUT" | grep -q "UCA upstream"; } \ + && ok "T08 apt/UCA upstream down FAILS even with a healthy edge" \ + || bad "T08 apt upstream failure not caught" "$OUT" + +# T09 juju agent stream down FAILS on its own -- the deploy dependency with no local copy. +mkfix "$TMP/t9" yes edge up up jujudown; run "$TMP/t9" check dc0 +{ [ "$RC" -eq 1 ] && printf '%s' "$OUT" | grep -q "juju agent stream"; } \ + && ok "T09 juju agent stream down FAILS alone" || bad "T09 agent stream failure not caught" "$OUT" + +# T10 snap store down FAILS on its own. +mkfix "$TMP/t10" yes edge up up snapdown; run "$TMP/t10" check dc0 +{ [ "$RC" -eq 1 ] && printf '%s' "$OUT" | grep -q "snap store"; } \ + && ok "T10 snap store down FAILS alone" || bad "T10 snap failure not caught" "$OUT" + +# T11 an UNRECOGNISED http code is a FAIL, never "not 000 so probably fine". This is the +# standing rule that an unrecognised state refuses rather than defaulting to success. +mkfix "$TMP/t11" yes edge up up weird; run "$TMP/t11" check dc0 +{ [ "$RC" -eq 1 ] && printf '%s' "$OUT" | grep -q "unrecognised, not assumed healthy"; } \ + && ok "T11 an unrecognised HTTP code FAILS rather than passing" || bad "T11 weird code passed" "$OUT" + +# T12 dc1 is a DIFFERENT site with its own measured gateway, and it has NO ruled snap +# proxy -- checking it with dc0's values would be the cross-DC error this repo has hit. +mkfix "$TMP/t12" yes edge up up ok; run "$TMP/t12" check dc1 +{ [ "$RC" -eq 2 ] && printf '%s' "$OUT" | grep -q "not the dc1 rack"; } \ + && ok "T12 dc0 host checked as dc1 REFUSES (cross-DC guard)" || bad "T12 cross-DC not guarded (rc=$RC)" "$OUT" + +# T14 A4's OWN unrecognised-code branch. ADDED after a mutation pass proved T11 was +# DECORATION: T11's fixture makes archive.ubuntu.com odd, which A3 catches first, so +# A4's `*)` branch was never exercised and flipping it to ok() left the suite fully +# green. This fixture keeps the A3 anchor healthy (200) and makes only the UCA +# upstream odd, so the failure can only come from A4. +mkfix "$TMP/t14" yes edge up up weirdup; run "$TMP/t14" check dc0 +{ [ "$RC" -eq 1 ] && printf '%s' "$OUT" | grep -q "A4 UCA upstream.*answered '403'"; } \ + && ok "T14 A4 unrecognised HTTP code FAILS (A3 healthy, so only A4 can fail it)" \ + || bad "T14 A4 unrecognised code not caught" "$OUT" + +# T13 usage without an action REFUSES. +OUT="$(bash "$S" 2>&1)"; RC=$? +[ "$RC" -eq 2 ] && ok "T13 no action REFUSES (rc=2)" || bad "T13 no action not refused (rc=$RC)" + +echo +echo "RESULT: PASS=$PASS FAIL=$FAIL" +[ "$FAIL" -eq 0 ] && { echo "ALL PASS"; exit 0; } || exit 1 diff --git a/tests/preflight/run-tests.sh b/tests/preflight/run-tests.sh index 8d92993..0f89d56 100644 --- a/tests/preflight/run-tests.sh +++ b/tests/preflight/run-tests.sh @@ -93,6 +93,22 @@ esac FB chmod +x "$d/fakebin/tofu" + # P9 seam (2026-08-02). The real check RUNS ON THE RACK and preflight normally does + # not; the stub lets all three branches be exercised offline. $EGRESS_FIXTURE: + # offrack (rc2 -> WARN) | broken (rc1 -> FAIL) | ok (rc0, the DEFAULT) | absent. + # Defaults to ok for the same reason P7 does: a fixture that models an unhealthy state + # by default drags every case testing something else, and that is the moment someone + # "fixes" it by demoting the gate. + mkdir -p "$d/scripts" + cat > "$d/scripts/dc-egress-check.sh" <<'EG' +#!/usr/bin/env bash +case "${EGRESS_FIXTURE:-ok}" in + ok) echo " OK A2 edge answers"; exit 0 ;; + broken) echo " FAIL A2 edge 10.12.4.1 does NOT answer"; exit 1 ;; + *) echo "REFUSE: this host does not carry 10.12.4.2/22" >&2; exit 2 ;; +esac +EG + chmod +x "$d/scripts/dc-egress-check.sh" echo "$d" } run() { # run