Close both egress gaps: dc-egress-check.sh + preflight P9, wired into restart + phase-4
Gap 1: nothing tested DC egress. dc-rack-net.sh check PASSED through a 19-hour outage
because it asserts legs and units -- a leg is not a path -- and the mirror answered 200
from its own nginx, which says nothing about upstream.
Gap 2: the post-reboot check set was my judgement, and it omitted egress.

New scripts/dc-egress-check.sh: site-keyed, runs on the rack, LAYERED (route -> edge
answers -> traffic leaves, ICMP and TCP -> the three upstreams), reporting the FIRST
failure as the cause while the rest SKIP. A4 asserts the UPSTREAM the local artifact
services sync FROM, which is independent of whether they serve.

Proven live on two different failure modes: dc0 fails at A2 naming the dead edge; dc1
passes A2 and fails A3/A4 -- traffic not leaving a healthy edge.

preflight P9 WARNs off-rack with the exact command, FAILs on the rack when broken, WARNs
if the checker is absent. Gap 2 is closed by the WIRING: restart procedure Stage 0
(before anything fetches) and phase-4 Step 3.9 (before the deploy), both stating P9 is
not a substitute for running it on the rack.

Mutation pass caught a decorative test of mine: T11 never reached A4's unrecognised-code
branch, so flipping it to ok() left the suite green. T14 added to exercise A4 alone.

tests/dc-egress-check 14/14; tests/preflight 39 -> 43/43; 5 mutations across both, each
killing a named case, scripts restored sha256-identical. Gauntlet ALL GREEN (97),
manifest 96 -> 97; repo-lint 0 fail / 1 warn.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
1 parent 453cecb commit aa64a18517998df0a3b4b9e65e70c60743e1d960
@JANeumatrix JANeumatrix authored 2 days ago
Showing 9 changed files
View
docs/CURRENT-STATE.md
View
docs/changelog-20260801-stage5-deploy-prep.md
View
runbooks/dc-dc-phase4-juju-bundle-per-dc.md
View
runbooks/ops-restart-procedure.md
View
scripts/dc-egress-check.sh 0 → 100755
View
scripts/preflight.sh
View
tests/HARNESS-MANIFEST
View
tests/dc-egress-check/run-tests.sh 0 → 100755
View
tests/preflight/run-tests.sh