diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 3a59084..ae11b19 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2581,6 +2581,31 @@ Worth carrying forward: R7 recorded this cert's SAN as "already DERIVED per-DC by design", which was true of the IP SAN and NOT the DNS names -- **a claim accurate about one half of a field and read as covering both. That is how F9 survived.** + - **THE PKI IS NOW AN ACTUAL GATE: `preflight.sh` P7, added 2026-07-30** (operator: "Yes, I + accept the recommendation. Convert to a gate."). This closes the "invoked by NO gate" + finding recorded above -- `octavia-pki.sh` was executed by nothing, P4's CHECK 0 asserted + only that the overlay FILE EXISTS, and the harness runs on throwaway fixtures, so "gauntlet + ALL GREEN" said nothing about the live PKI. **That is how F9 survived for weeks.** + - **HEADEND-ONLY, and NOT EVALUATED (WARN) anywhere else** -- never a silent pass, never a + hard FAIL for being on the jumphost. The binding is read from the SAME + `creds-manifests/host-identity` `octavia-pki.sh` reads, so the two cannot drift. This + designs out P5's own F6 defect rather than repeating it. + - **rc 3 is mapped EXPLICITLY.** `note()` treats any rc>=3 as "UNEXPECTED exit code", so a + documented REFUSE would have been reported as an unknown failure. On the headend a REFUSE + is a FAIL, named as an unevaluated amphora trust domain. + - **A PASS VERDICT IS NOT SUFFICIENT, and P7 encodes that.** While `os-public-hostname` is + unset, A12/A13 RECORD a wrong-zone cert rather than failing it, so `verify` exits 0 on + exactly the defect the gate exists to catch. P7 additionally requires the literal line + `DNS SANs are all in this DC's expected zone`. Satisfiable by construction now. + - Skips cleanly for `vr0-dc0` (VR1 per-DC artifact); FAILS CLOSED if the checker is absent. + - **Preflight harness 26/26 -> 33/33.** Ten existing cases went RED when P7 landed -- + correctly, since the fixtures had no `octavia-pki.sh`. Fixed by making `mkfix` model a + healthy headend, NOT by patching cases: that would have been the moment the gate got + demoted to a warning to go green. T27 is the one that matters -- a wrong-zone cert on + which `verify` exits 0 must still FAIL P7. + - Preflight's overall verdict is still FAIL, which is **P5 working as designed** on the + pre-existing red register (SEC-021, the SEC-020 conflation, the power-key asymmetry). + P7 itself PASSES on the headend. - The certs remain INERT by design -- `os-public-hostname` is still set nowhere (R5 refused setting it at Stage 5 as a D-019 repeat). The names are now correct IN ADVANCE of D-106's Stage-7 work arming them. diff --git a/docs/changelog-20260730-octavia-reissue-tool.md b/docs/changelog-20260730-octavia-reissue-tool.md index 529d436..9e48c0f 100644 --- a/docs/changelog-20260730-octavia-reissue-tool.md +++ b/docs/changelog-20260730-octavia-reissue-tool.md @@ -283,3 +283,75 @@ python3 scripts/creds-matrix.py -> 101 rows, 5 findings (all pre-existing), S4 CLEAN bash scripts/run-tests-all.sh -> GAUNTLET: ALL GREEN (89) bash scripts/repo-lint.sh -> 0 fail + +--- + +# PART 4 (same session) -- P7: the Octavia PKI becomes an actual GATE + +Operator: "Yes, I accept the recommendation. Convert to a gate." + +## 13. `scripts/preflight.sh` gains P7 + +**THE GAP.** `scripts/octavia-pki.sh` was invoked by NOTHING executable. P4's CHECK 0 asserts +only that `overlays/-octavia-pki.yaml` EXISTS; the octavia-pki harness runs against +throwaway fixtures. So "gauntlet ALL GREEN" said nothing whatever about the live PKI, and the +only real check ran when a human remembered to type it. **That is how F9 survived for weeks.** + +Three design problems had to be solved, each from a measured precedent: + +1. **rc 3 must not reach `note()`'s `*)` arm.** `octavia-pki.sh` uses exit 3 for REFUSE, but + `note()` treats any rc>=3 as "UNEXPECTED exit code (127=command not found...)". Wiring it + naively would report a legitimate wrong-host refusal as an unknown failure. P7 maps rc + explicitly: 0 -> evaluate the output, 1 -> FAIL, 3 -> FAIL **named as an unevaluated PKI**, + anything else -> FAIL. +2. **Host authority.** The PKI lives on the headend (D-109 ruling note (b)). On any other host + P7 reports **NOT EVALUATED (WARN)** -- never a silent pass, and never a hard FAIL either, + since running preflight from the jumphost for other purposes is legitimate. The binding is + read from the SAME `creds-manifests/host-identity` that `octavia-pki.sh` reads, so the two + cannot drift. This is P5's own F6 defect (2026-07-30: probing the wrong host's filesystem + and reporting it as fact, 34 findings against a true 7) designed out rather than repeated. +3. **A PASS VERDICT IS NOT SUFFICIENT, and this is the subtle one.** While `os-public-hostname` + is unset (B5 IP-only), A12/A13 **RECORD** a wrong-zone certificate rather than failing it -- + correctly, since nothing resolves those names yet. So `verify` **exits 0 on exactly the + defect this gate exists to catch.** P7 therefore also requires the literal line + `DNS SANs are all in this DC's expected zone`, which is emitted only when every DNS SAN is + in that DC's derived zone. Satisfiable by construction now: 1.0-GEN.c derives the zone, and + both live DCs were reissued into theirs. + +Also: SKIPS cleanly for `vr0-dc0` (the amphora PKI is a VR1 per-DC artifact -- a gate that +fires on a target it does not apply to is how gates get bypassed), and FAILS CLOSED if +`octavia-pki.sh` is absent, mirroring P4/P5. + +**Revert:** `git checkout -- scripts/preflight.sh tests/preflight/run-tests.sh`. + +## 14. Harness 26/26 -> 33/33, and ten cases went red first + +When P7 landed, **ten existing cases failed** -- correctly: the fixtures had no +`octavia-pki.sh`, so the gate failed closed. The fix was to make `mkfix` model a healthy +headend (host-identity binding, a fake `hostname` on PATH, an octavia-pki stub), NOT to patch +the cases individually -- that would have been the moment someone demoted the gate to a warning +to go green. + +New cases: **T27** (verify exits 0 on a wrong-zone cert and P7 still FAILS -- the F9 shape, and +the case a naive exit-code-only implementation fails), T28 (off-headend -> NOT EVALUATED, WARN, +never a silent pass), T29 (checker missing -> fails closed), T30 (verify FAIL -> overall FAIL), +T31 (REFUSE on the headend -> FAIL, named correctly rather than as "UNEXPECTED"), T32 (absent +headend binding -> FAIL, never assumed), T33 (vr0-dc0 skips cleanly and the run still PASSES). + +Two harness bugs of my own, both caught by running rather than reading: the stub first emitted +`DC.s` where the real line has an apostrophe (so it correctly did not match), then +`\x27\x27\x27` produced three apostrophes because `\x27` inside a printf FORMAT already yields +one. Fixed the stub, not the gate. + +## Verification (part 4) + + bash tests/preflight/run-tests.sh -> 33/33 ALL PASS + bash scripts/run-tests-all.sh -> GAUNTLET: ALL GREEN (89) + bash scripts/repo-lint.sh -> 0 fail + +## Note on preflight's overall verdict + +Preflight still exits FAIL today, and that is P5 working as designed: the credential register is +red on real pre-existing defects (SEC-021's declaration gaps, the SEC-020 identity conflation, +the per-DC power-key asymmetry). P7 does not change that verdict on the headend -- it passes +there. Going green is remediation, gated per row. diff --git a/scripts/preflight.sh b/scripts/preflight.sh index b5eb983..76fc95c 100644 --- a/scripts/preflight.sh +++ b/scripts/preflight.sh @@ -15,6 +15,8 @@ # P4 live pre-flight scripts/pre-flight-checks.sh (needs MAAS; overlay/VIP/planes/nodes) # P5 credential matrix scripts/creds-matrix.py --tier2 (D-137 ruling 1; # tier 1 STATIC + tier 2 LOCAL. Hermetic: no ssh, no sudo.) +# P7 octavia PKI scripts/octavia-pki.sh verify (amphora trust domain; +# HEADEND-ONLY -- NOT EVALUATED elsewhere, never a silent pass) # P6 reminders stage-2 gates: juju-spaces-check.sh (AFTER add-model), # osd-blank-check.sh (sudo), 'juju deploy --dry-run' (phase-01 1.2) # @@ -187,6 +189,83 @@ echo " [FAIL] scripts/creds-matrix.py missing"; note 1 "P5 credential matrix" fail fi +echo "================ P7: Octavia amphora PKI ================" +# WHY THIS GATE EXISTS (added 2026-07-30). `scripts/octavia-pki.sh` was invoked by NOTHING +# executable: P4's CHECK 0 asserts only that overlays/-octavia-pki.yaml EXISTS, and the +# harness runs against throwaway fixtures. So "gauntlet ALL GREEN" said nothing whatever about +# the live PKI, and the only real check ran when a human remembered to type it. That is how F9 +# survived for weeks -- both DCs' controller certs carried another region's names, and no gate +# read them. Octavia cannot establish amphora mTLS without this material being correct, so it +# belongs in the deploy gate. +# +# HOST-AWARE, AND THAT IS THE WHOLE DESIGN PROBLEM. The PKI lives on the HEADEND (D-109 ruling +# note (b)); `octavia-pki.sh verify` REFUSES with exit 3 anywhere else rather than describing +# the wrong machine's filesystem. Two consequences: +# (a) rc 3 must NOT reach note()'s `*)` arm, which would report a legitimate wrong-host +# refusal as "UNEXPECTED exit code" and hard-FAIL preflight on the jumphost; +# (b) on a non-headend host this gate must say NOT EVALUATED -- never silently pass. P5's own +# F6 defect (2026-07-30) was probing the WRONG HOST'S filesystem and reporting it as +# fact, producing 34 findings against a true 7. The binding is read from the SAME +# register file octavia-pki.sh reads, so the two cannot drift. +# +# THE PASS VERDICT ALONE IS NOT SUFFICIENT, WHICH IS WHY THE POSITIVE LINE IS ALSO REQUIRED. +# While `os-public-hostname` is unset (bundle.yaml posture B5, IP-only), A12/A13 RECORD a +# wrong-zone certificate rather than failing it -- deliberately, since nothing resolves those +# names yet. So `verify` exits 0 on exactly the defect this gate is meant to catch. Asserting +# the literal positive line closes that: it is emitted ONLY when every DNS SAN is in this DC's +# derived zone. Satisfiable by construction -- 1.0-GEN.c now derives the zone, and both live +# DCs were reissued into theirs on 2026-07-30. +if [ "$DC" = "vr0-dc0" ]; then + echo " [skip] P7 not applicable to $DC -- the Octavia amphora PKI is a VR1 per-DC artifact" +elif [ ! -f scripts/octavia-pki.sh ]; then + # FAIL CLOSED, mirroring P4/P5: a gate that VANISHES must never read as a pass. + echo " [FAIL] scripts/octavia-pki.sh missing -- the Octavia PKI gate cannot run" + note 1 "P7 octavia PKI" fail +else + P7_HOST="$(hostname)" + P7_EXPECT="$(awk '!/^[[:space:]]*#/ && NF==2 && $1=="headend" {print $2; exit}' \ + creds-manifests/host-identity 2>/dev/null)" + if [ -z "$P7_EXPECT" ]; then + echo " [FAIL] no 'headend' binding in creds-manifests/host-identity -- cannot tell which" + echo " host should hold the Octavia PKI, and will not assume this one does" + note 1 "P7 octavia PKI" fail + elif [ "$P7_HOST" != "$P7_EXPECT" ]; then + echo " [warn] P7 NOT EVALUATED on '$P7_HOST' -- the Octavia PKI lives on '$P7_EXPECT'" + echo " (D-109 ruling note (b)), and measuring this host's filesystem would" + echo " describe the wrong machine. This is NOT a pass. Run preflight ON the" + echo " headend for a real pre-deploy gate: D-128 puts the Stage-5+ deploy there." + note 2 "P7 octavia PKI" warn + else + P7_OUT="$(bash scripts/octavia-pki.sh verify "$DC" 2>&1)"; P7_RC=$? + printf '%s\n' "$P7_OUT" + case "$P7_RC" in + 0) + # The verdict passed. Now the acceptance criterion the verdict cannot express. + if printf '%s' "$P7_OUT" | grep -q "DNS SANs are all in this DC's expected zone"; then + echo " [ok] P7 octavia PKI verified for $DC, and its DNS SANs are in this DC's own zone" + else + echo " [FAIL] P7 octavia-pki verify PASSED for $DC but did NOT report that the DNS SANs" + echo " are in this DC's expected zone. While os-public-hostname is unset those" + echo " names are INERT, so A12/A13 record rather than fail -- which means a PASS" + echo " verdict here is compatible with a certificate carrying another DC's or" + echo " another region's names (F9, measured on BOTH DCs 2026-07-30). Reissue it:" + echo " runbooks/phase-01-bundle-deploy.md Step 1.0-REISSUE" + note 1 "P7 octavia PKI" fail + fi ;; + 1) note 1 "P7 octavia PKI" fail ;; + 3) + echo " [FAIL] P7 octavia-pki verify REFUSED (exit 3) ON THE HEADEND ITSELF -- it could" + echo " not evaluate the PKI (missing openssl/python3, unreadable VIP overlay, or" + echo " an absent host binding). 'Could not look' is never 'nothing there', and a" + echo " deploy must not proceed over an unevaluated amphora trust domain." + note 1 "P7 octavia PKI" fail ;; + *) + echo " [FAIL] P7 octavia-pki verify exited $P7_RC -- unexpected for this checker" + note 1 "P7 octavia PKI" fail ;; + esac + fi +fi + echo "================ P6: stage-2 reminders (NOT run here) ================" echo " - after 'juju add-model': bash scripts/juju-spaces-check.sh" echo " - with sudo: bash scripts/osd-blank-check.sh" diff --git a/tests/preflight/run-tests.sh b/tests/preflight/run-tests.sh index 90a1fc7..c422475 100644 --- a/tests/preflight/run-tests.sh +++ b/tests/preflight/run-tests.sh @@ -11,7 +11,41 @@ PASS=0; FAIL=0 mkfix() { # mkfix [creds_rc|"omit"] - local d="$TMP/$1"; mkdir -p "$d/scripts" "$d/fakebin" + # [octavia: ok|wrongzone|fail|refuse|omit|offhost|nobinding] + # P7 (2026-07-30) is HEADEND-ONLY and FAILS CLOSED, so every fixture must model a healthy + # headend or the gate correctly stops the deploy in cases that are testing something else. + # Adding it to mkfix rather than to individual cases is deliberate: when P7 landed, ten + # existing cases went red for the RIGHT reason (no octavia-pki.sh => fail closed), and + # patching them one by one would have been the moment someone "fixed" it by demoting the + # gate to a warning. + local d="$TMP/$1"; mkdir -p "$d/scripts" "$d/fakebin" "$d/creds-manifests" + local oct="${6:-ok}" + # The headend binding P7 reads. `nobinding` exercises the refusal-to-assume path. + if [ "$oct" != "nobinding" ]; then + printf '# role host\nheadend fixturehost\n' > "$d/creds-manifests/host-identity" + else + printf '# role host\n' > "$d/creds-manifests/host-identity" + fi + # A FAKE hostname on PATH is what makes the host-authority branch testable at all. + # `offhost` makes this fixture claim to be a different machine, which is the live jumphost + # case: the PKI is elsewhere, so P7 must say NOT EVALUATED and must NOT pass silently. + if [ "$oct" = "offhost" ]; then + printf '#!/usr/bin/env bash\necho someotherhost\n' > "$d/fakebin/hostname" + else + printf '#!/usr/bin/env bash\necho fixturehost\n' > "$d/fakebin/hostname" + fi + chmod +x "$d/fakebin/hostname" + # The octavia-pki stub. `ok` emits the LITERAL positive line P7 requires; `wrongzone` + # exits 0 WITHOUT it, which is the real F9 shape -- verify passes because the names are + # inert, while the certificate carries another DC's or another region's zone. + case "$oct" in + omit) : ;; + wrongzone) printf '#!/usr/bin/env bash\necho " ok A12 (recorded, not failed) the DNS SANs are not in this DC\x27s expected zone"\necho "octavia-pki verify: PASS -- 37 assertion(s), 0 failed"\nexit 0\n' > "$d/scripts/octavia-pki.sh" ;; + fail) printf '#!/usr/bin/env bash\necho " FAIL A9 SAN missing this DC.s provider VIP"\nexit 1\n' > "$d/scripts/octavia-pki.sh" ;; + refuse) printf '#!/usr/bin/env bash\necho " REFUSE openssl is not on PATH"\nexit 3\n' > "$d/scripts/octavia-pki.sh" ;; + *) printf '#!/usr/bin/env bash\necho " ok A12 DNS SANs are all in this DC\x27s expected zone \x27omega.dc0.vr1.cloud.neumatrix.local\x27"\necho "octavia-pki verify: PASS -- 37 assertion(s), 0 failed"\nexit 0\n' > "$d/scripts/octavia-pki.sh" ;; + esac + [ "$oct" = "omit" ] || chmod +x "$d/scripts/octavia-pki.sh" cp "$REPO/scripts/preflight.sh" "$REPO/scripts/channel_assert.py" "$d/scripts/" printf 'applications:\n keystone: {charm: keystone, channel: 2024.1/stable}\n' > "$d/bundle.yaml" printf '#!/usr/bin/env bash\necho " [stub] lint"\nexit %s\n' "$2" > "$d/scripts/repo-lint.sh" @@ -154,5 +188,56 @@ echo " PASS T26 the old unscoped octavia-pki overlay name is gone"; PASS=$((PASS+1)) fi +# =================================================================================== +# T27..T33 -- P7, the Octavia amphora PKI gate (added 2026-07-30) +# +# WHY P7 EXISTS: octavia-pki.sh was invoked by NOTHING executable. P4's CHECK 0 asserted +# only that the overlay FILE EXISTS, and the octavia-pki harness runs against throwaway +# fixtures -- so "gauntlet ALL GREEN" said nothing about the live PKI, and the only real +# check ran when a human remembered to type it. That is how F9 survived: both DCs carried +# another region's names for weeks with every gate green. + +# T27 -- THE CASE THIS GATE WAS BUILT FOR, and the one a naive implementation fails. +# `verify` exits 0 on a WRONG-ZONE certificate while os-public-hostname is unset, because +# A12/A13 RECORD rather than fail when the names are inert. So a P7 that only checked the +# exit code would have passed the exact defect it exists to catch. It must additionally +# require the literal positive line. +D=$(mkfix p7_wrongzone 0 0 1 0 wrongzone) +run 1 'did NOT report that the DNS SANs' "T27 verify exits 0 on a WRONG-ZONE cert -> P7 still FAILS (F9)" "$D" + +# T28 -- HOST AUTHORITY. The PKI lives on the headend (D-109 (b)); anywhere else the gate +# must say NOT EVALUATED. It must NOT silently pass (that is P5's own F6 defect: probing +# the wrong host's filesystem and reporting it as fact) and must NOT hard-FAIL either, +# since running preflight from the jumphost for other purposes is legitimate. +D=$(mkfix p7_offhost 0 0 1 0 offhost) +run 2 'NOT EVALUATED on .someotherhost.' "T28 off-headend -> NOT EVALUATED (WARN), never a silent pass" "$D" + +# T29 -- FAIL CLOSED, same rule T5/T9 encode for P4/P5. A gate that VANISHES must never +# read as a pass; deleting the checker must not be a way to go green. +D=$(mkfix p7_gone 0 0 1 0 omit) +run 1 'octavia-pki.sh missing' "T29 P7 checker missing -> FAILS CLOSED" "$D" + +# T30 -- a real verify failure blocks the deploy. +D=$(mkfix p7_fail 0 0 1 0 fail) +run 1 'PREFLIGHT: FAIL' "T30 octavia-pki verify FAIL -> overall FAIL" "$D" + +# T31 -- REFUSE (exit 3) ON THE HEADEND is a FAIL, and must be reported as an unevaluated +# PKI rather than as note()'s generic "UNEXPECTED exit code". rc 3 is octavia-pki's +# DOCUMENTED refusal code; letting it reach note()'s `*)` arm would describe a known, +# meaningful state as an unknown one. +D=$(mkfix p7_refuse 0 0 1 0 refuse) +run 1 'REFUSED .exit 3. ON THE HEADEND' "T31 REFUSE on the headend -> FAIL, named correctly" "$D" + +# T32 -- no headend binding: refuse to ASSUME this host holds the PKI. Same reasoning as +# octavia-pki.sh's own binding check -- both read the SAME register file so they cannot drift. +D=$(mkfix p7_nobinding 0 0 1 0 nobinding) +run 1 "no .headend. binding" "T32 absent headend binding -> FAIL, never assumed" "$D" + +# T33 -- vr0-dc0 has no Octavia amphora PKI (it is a VR1 per-DC artifact), so P7 SKIPS +# rather than inventing a failure. A gate that fires on a target it does not apply to is +# how gates get bypassed. +D=$(mkfix p7_vr0 0 0 1 0 ok) +run 0 'not applicable to vr0-dc0' "T33 vr0-dc0 -> P7 skips cleanly, run still PASSES" "$D" vr0-dc0 + echo; echo "RESULT: PASS=$PASS FAIL=$FAIL" [[ "$FAIL" -eq 0 ]] && { echo "ALL PASS"; exit 0; } || exit 1