diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index fad0b3c..5b481dc 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1618,6 +1618,49 @@ ruling makes the SAN set consistent with the VIP it has always been derived from -- whether that is the right derivation is a separate question to settle by inspection at the Octavia step of Stage 5. +- **RENDER-PIPELINE STEP 6 DONE 2026-07-29 -- the D-136 CHAIN AUDIT, and it found real holes.** + Three read-only lenses over `data source -> rendering tool -> deployment mechanism`, the ruled + final step of D-136 option (D). Lens 3 was adversarial, tasked with making the chain produce a + WRONG result that still comes out GREEN. **It succeeded three times, all executed.** + **ALL THREE LENSES CONVERGED ON ONE ROOT CAUSE: dc1's overlay was byte-gated and dc0's was + gated by NOTHING** -- `render/values/` was referenced by ZERO scripts and ZERO harnesses, and + `preflight` P2 feeds the deploy gate dc0's overlay. Because **R9's render-drift check was + RULED (D-119 amendment) and never BUILT.** Executed wrong-but-green: hand-transposing two + apps' octets in the dc0 overlay (consistent, in-band, correct prefixes) PASSED; and changing + ONE WORD, `family: dual` -> `v4`, silently reverted dc0 to IPv4-only against R2's ruling and + PASSED, because dropping `prefer-ipv6` and the v6 legs together satisfies invariant 9's + coupling check and `vip_dual` is printed but never asserted. + **R9's GATE IS NOW BUILT: `tests/render-drift/`** -- iterates `render/values/*.yaml`, reads + each file's own `output:` field (inert until now), renders and byte-compares. A new per-DC + values file is gated the moment it is committed. Non-zero floor plus a PROOF-OF-TEETH case. + **THREE REFUSALS ADDED where the chain silently took the last writer:** a duplicate + `(role, kind)` in the apex (measured -- appending ONE spare prefix re-homed every dc0 VIP into + a different `/64` and went green, because the gate checks overlay-vs-apex AGREEMENT and both + read the same wrong value; the winner depended on JSON array order); and duplicate / + out-of-band entries in `APP_OCTET` (a duplicate app name emitted two YAML blocks, + `safe_load` kept the last, one app's ruled octet vanished with nothing red). + **TWO HARNESS CASES THAT COULD NOT FAIL, FIXED:** T3b's byte-compare was CIRCULAR (header + derived `--from-overlay $LIVE` then compared `--against $LIVE`; ~a third of the bytes + self-copied, proven by rewriting a header line to a false claim and still matching), and + T29's own fixture was INERT. **OWNED: my first T29 re-point asserted a `--show-legs` flag + THAT DOES NOT EXIST** -- an invented interface, caught by the harness and replaced with real + behaviour; recorded rather than quietly fixed. + Gauntlet **ALL GREEN (86) on vcloud** (was 85); repo-lint 0 fail / 612 files. + **LOGGED NOT ACTIONED (the deploy-path half, which is larger):** `preflight` P2 validates a + DIFFERENT and partial overlay set vs any real deploy and hardcodes `--dc vr1-dc0` while + phase-4 uses it as the **dc1** gate; **dc1's ruled 3-overlay input exists in NO executable + path** (phase-4 Step 4 names only the vips overlay, so a documented dc1 deploy merges a + machines block still tagged `openstack-vr1-dc0`); phase-6's deploys pass NO VIP overlay at + all; `overlays/${DC}-hostnames.yaml` is named in three commands and was never authored; + **role sub-tags `control`/`compute`/`storage` are created NOWHERE** though every machine + constrains on them (fails hard at allocation -- the good failure mode -- and is the + per-role-tags Stage-5 blocker, now evidenced); `octavia-pki.yaml` lands LAST and writes the + same `applications.octavia.options` map, and the checker deep-merges so it returns PASS under + EITHER juju merge semantics and cannot discriminate (needs a `--dry-run`); the in-runbook VIP + guard reads only the first v4 leg; no harness pins its own case count; phase-01's GATE + figures are stale (4 machines vs 9, 50 apps vs 56, 11/11/0 vs the 13/13/0 demanded 20 lines + later); and the apex dump the chain reads predates the 2026-07-27 population with nothing + re-dumping or comparing it to live. - **THREE PLATFORM BEHAVIOURS GRADUATED to `references/platform-traps.md`** at session close, having been recorded only in this status document (which is consolidated over time, so a durable trap does not belong here alone): MAAS auto-reserves `::1`-`::ffff:ffff` on EVERY diff --git a/docs/changelog-20260728-vip-arity-gate.md b/docs/changelog-20260728-vip-arity-gate.md index c9ca63b..8272de8 100644 --- a/docs/changelog-20260728-vip-arity-gate.md +++ b/docs/changelog-20260728-vip-arity-gate.md @@ -614,3 +614,99 @@ reverse direction was not established from source. This ruling makes the SAN set consistent with the VIP it has always been derived from -- whether that is the right derivation is a separate question, to settle by inspection at the Octavia step. + +--- + +# 2026-07-29 -- D-136 STEP 6: the chain audit, and what it found + +Three read-only lenses over `data source -> rendering tool -> deployment mechanism` +(D-136 option (D)'s ruled final step). Lens 3 was adversarial and was told to make the +chain produce a wrong result that still comes out GREEN. **It succeeded three times.** + +## 21. All three lenses converged: dc1 was gated, dc0 was not + +`tests/render-dc-overlays` byte-compared the **dc1** overlay only. dc0's -- the artifact +`preflight.sh` P2 actually feeds the deploy gate -- was reproducible by nobody, and +`render/values/` was referenced by **ZERO scripts and ZERO harnesses**. Executed attacks +that passed every gate: + +- **Hand-transposing two apps' octets in the dc0 overlay** (glance .53 <-> magnum .54, + all six legs). Internally consistent, in-band, right /22 and /64, unique -> `PASS`. +- **One word: `family: dual` -> `family: v4`** in dc0's values file. That drops + `prefer-ipv6` and the v6 legs TOGETHER, so invariant 9's coupling check is satisfied, + and `vip_dual` is printed but never asserted -> dc0 silently reverts to IPv4-only + against R2's ruling, `PASS`. + +**Root cause: R9's render-drift check was RULED (D-119 amendment, 2026-07-27) and never +built.** `lib-net.sh:171-177` still carried the future-tense "becomes GENERATED". + +## 22. `tests/render-drift/` -- R9's gate, built + +Iterates `render/values/*.yaml`, reads each file's own `output:` field (inert until now), +renders, and byte-compares against the committed overlay. **A new per-DC values file is +gated the moment it is committed -- no harness edit.** Carries a non-zero floor (a drift +gate over zero files is not a gate) and a PROOF-OF-TEETH case that mutates a copy and +requires the check to go red. 4/4; both DCs byte-match today. + +## 23. Three refusals added where the chain took the last writer + +- **Duplicate `(role, kind)` in the apex.** Both readers were last-writer-wins. Measured: + appending ONE spare prefix (`2602:f3e2:f02:99::/64`, routine IPAM work) silently re-homed + **every dc0 VIP into a different /64** and the whole chain went green -- because the gate + only checks overlay-vs-apex AGREEMENT, and both sides read the same wrong value. Which + prefix won depended on JSON array order, which is not a contract. Both + `render-dc-overlays.py` and `provider-bundle-check.py` now REFUSE (rc=2) and name both + candidates. +- **`APP_OCTET` duplicates / out-of-band octets.** A duplicate app name emitted two YAML + blocks for one app; `yaml.safe_load` kept the last and the gate passed, so one app's + ruled octet vanished with nothing red. Now refused at source, where the cause is named. + +## 24. Two harness cases that could not fail, fixed + +- **T3b's byte-compare was CIRCULAR.** It derived the header `--from-overlay "$LIVE"` and + then compared `--against "$LIVE"` -- ~24 of 76 lines self-copied, so the header could not + fail. Proven by rewriting a header line to a false claim and still getting + "byte-for-byte match". Now reads the COMMITTED values file, the same artifact + `render-drift` gates. +- **T29's own fixture was INERT** -- its assertion held from the dc1 overlay alone, so it + could not detect a fault in the thing it constructed. Re-pointed to strip dc1's prefixes + from the apex and require a REFUSAL, which only the dc1 path can show. + +**OWNED:** my first re-point of T29 asserted a `--show-legs` flag **that does not exist** -- +I invented an interface rather than reading one. Caught immediately by the harness and +replaced with a real behaviour. Fabricating a flag is the cardinal sin in this repo's +posture; recording it rather than quietly fixing it. + +**Evidence.** `provider-bundle-check` 33/33, `render-dc-overlays` 18/18, `render-drift` +4/4 (new); gauntlet **ALL GREEN (86) on vcloud** (was 85); repo-lint 0 fail / 612 files. + +## 25. Chain findings LOGGED, NOT ACTIONED + +16. **`preflight.sh` P2 validates a DIFFERENT overlay set than any deploy uses** -- it + passes one overlay where dc0's deploy passes two, and it hardcodes `--dc vr1-dc0` with + no dc1 branch, while `dc-dc-phase4:197-204` uses it as the **dc1** gate. +17. **dc1's ruled 3-overlay input exists in no executable path.** `grep dc-ha-scaleup + runbooks/ scripts/` and `grep vr1-dc1-machines runbooks/ scripts/` both return ZERO. + phase-4 Step 4 names only the vips overlay -> a dc1 deploy as documented merges a + machines block still saying `tags=openstack-vr1-dc0`, which is exactly what that + overlay was written to prevent. +18. **phase-6's deploy commands pass no VIP overlay at all** -- since ruling 3 that means a + merged input with zero VIPs, including the designate those commands exist to add. +19. **`overlays/${DC}-hostnames.yaml` is named in three deploy commands and was never + authored.** +20. **Role sub-tags `control`/`compute`/`storage` are created NOWHERE** while every machine + constrains on them -- confirmed by measurement, and it fails hard at allocation (the + good failure mode). This is the per-role-tags Stage-5 blocker, now evidenced. +21. **`octavia-pki.yaml` lands LAST and writes the same `applications.octavia.options` map + as the vips overlay.** Whether octavia keeps `vip` + `prefer-ipv6` depends on juju's + merge semantics, which cannot be settled offline -- and `provider-bundle-check` + deep-merges, so it returns PASS under EITHER behaviour and structurally cannot + discriminate. Needs a `--dry-run` with both overlays. +22. **The in-runbook VIP guard reads only the first v4 leg**, so dropping all three v6 legs + plus `prefer-ipv6` still scores 13/13/0. +23. **No harness pins its own case count** -- deleting the three seeded-fault cases that + prove the byte-compare has teeth still yields `ALL PASS`. +24. Stale GATE figures in `phase-01` (4 machines vs 9, 50 apps vs 56, VIPs 11/11/0 vs the + 13/13/0 the guard 20 lines later demands) train an operator to wave the gate through. +25. The apex dump the chain reads (`vr1-office1-current-20260725.json`) predates the + 2026-07-27 apex population and nothing re-dumps or compares it to live. diff --git a/scripts/provider-bundle-check.py b/scripts/provider-bundle-check.py index 1f124d7..f84128d 100644 --- a/scripts/provider-bundle-check.py +++ b/scripts/provider-bundle-check.py @@ -143,6 +143,16 @@ role = p.get("role") or "" role = role.get("slug") if isinstance(role, dict) else role kind = "vip" if "VIP" in str(p.get("description") or "") else "plane" + # REFUSE a duplicate rather than take the last. This reader and the renderer's + # share the record, so last-writer-wins here would make the gate AGREE with a + # wrongly-rendered overlay: measured in the 2026-07-29 chain audit, adding ONE + # spare prefix to the apex re-homed every dc0 VIP into a different /64 and the + # whole chain went GREEN. Which one won depended on JSON array order, which is + # not a contract. + if (role, kind) in found and found[(role, kind)] != pre: + return None, ("apex record %s has TWO %s/%s /64s for %s (%s and %s) -- refusing " + "to pick one by array order" + % (rec, role, kind, dc, found[(role, kind)], pre)) found[(role, kind)] = pre want = (("provider", ("provider-public", "vip")), ("admin", ("metal-admin", "plane")), diff --git a/scripts/render-dc-overlays.py b/scripts/render-dc-overlays.py index f186c73..29222ed 100755 --- a/scripts/render-dc-overlays.py +++ b/scripts/render-dc-overlays.py @@ -44,6 +44,10 @@ REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) APEX_ENV, APEX_GLOB = "APEX_RECORD", "netbox/draft/vr1-office1-current-*.json" +# The ruled VIP octet band (D-020 amendment / R11). Kept in step with +# provider-bundle-check.py OCTET_LO/HI and dc-plane-apex-import.py BAND_VIP. +OCTET_LO, OCTET_HI = 50, 99 + # The ruled app -> VIP octet map. Single source is netbox/dc-plane-apex-import.py:49-56 # (the tool that wrote these into the apex); duplicated here would be a second # hand-maintained copy, so it is READ from that file rather than restated. @@ -73,8 +77,26 @@ for node in tree.body: if isinstance(node, ast.Assign) and any( getattr(t, "id", "") == "APP_OCTET" for t in node.targets): - return [(k.value, v.value) for k, v in - ((e.elts[0], e.elts[1]) for e in node.value.elts)] + pairs = [(k.value, v.value) for k, v in + ((e.elts[0], e.elts[1]) for e in node.value.elts)] + # REFUSE rather than emit. Measured (2026-07-29 chain audit): a duplicate app + # name emitted TWO blocks for one app, `yaml.safe_load` silently kept the last, + # and the gate passed -- one app's ruled octet vanished with nothing red. A + # duplicate octet or an out-of-band one would be caught downstream by the gate, + # but catching it HERE names the actual cause instead of a symptom. + names = [n for n, _ in pairs] + octs = [o for _, o in pairs] + dupn = sorted({n for n in names if names.count(n) > 1}) + dupo = sorted({o for o in octs if octs.count(o) > 1}) + bad = sorted({o for o in octs if not (OCTET_LO <= int(o) <= OCTET_HI)}) + if dupn: + die("APP_OCTET in %s has duplicate app name(s): %s" % (OCTET_SRC, ", ".join(dupn))) + if dupo: + die("APP_OCTET in %s has duplicate octet(s): %s" % (OCTET_SRC, ", ".join(map(str, dupo)))) + if bad: + die("APP_OCTET in %s has octet(s) outside the ruled %d-%d band: %s" + % (OCTET_SRC, OCTET_LO, OCTET_HI, ", ".join(map(str, bad)))) + return pairs die("APP_OCTET not found in %s" % OCTET_SRC) @@ -122,7 +144,17 @@ continue role = p.get("role") or "" role = role.get("slug") if isinstance(role, dict) else role - out[(role, "vip" if "VIP" in str(p.get("description") or "") else "plane")] = pre + key = (role, "vip" if "VIP" in str(p.get("description") or "") else "plane") + # REFUSE on a duplicate key. Measured (2026-07-29 chain audit): this loop was + # last-writer-wins, so adding ONE spare prefix to the apex -- routine IPAM work -- + # silently re-homed every dc0 VIP into a different /64 and the whole chain went + # GREEN, because the gate only checks overlay-vs-apex AGREEMENT and both read the + # same wrong value. Which prefix won depended on JSON array order, which is not a + # contract. + if key in out and out[key] != pre: + die("apex record %s has TWO %s/%s /64s for %s (%s and %s) -- refusing to pick " + "one by array order" % (rec, key[0], key[1], dc, out[key], pre)) + out[key] = pre if not out: die("apex record %s carries no /64 prefixes scoped to %s" % (rec, dc)) return out, rec diff --git a/tests/HARNESS-MANIFEST b/tests/HARNESS-MANIFEST index ab59405..f19d227 100644 --- a/tests/HARNESS-MANIFEST +++ b/tests/HARNESS-MANIFEST @@ -68,6 +68,7 @@ reenroll-hosts render-baseline render-dc-overlays +render-drift repo-lint roles-aggregates-import sandbox-fidelity-check diff --git a/tests/provider-bundle-check/run-tests.sh b/tests/provider-bundle-check/run-tests.sh index 3b49b8a..c691624 100644 --- a/tests/provider-bundle-check/run-tests.sh +++ b/tests/provider-bundle-check/run-tests.sh @@ -281,10 +281,21 @@ prefer-ipv6: true vip: "10.12.64.50 10.12.68.50 10.12.72.50 2602:f3e2:f03:11::50 fd50:840e:74e2:320::50 fd50:840e:74e2:321::50" YAML -runargs 0 '13 clustered VIP\(s\).*13 dual-family' \ - "T29 dual-family resolves the dc1 apex bands" \ - "$TMP/good.yaml" --overlay "$REPO/overlays/vr1-dc1-vips.yaml" \ - --overlay "$TMP/dc1-dual.yaml" --dc vr1-dc1 +# T29 asserted a string the dc1 overlay alone already produces, so its own fixture was +# INERT and the case could not fail on it (2026-07-29 chain audit; same defect class as +# T20/T21 the day before). Re-pointed at what ONLY this path proves: a dc1 v6 leg is +# validated against dc1's apex /64s, which a dc0 run cannot reach. +python3 - "$REPO" "$TMP/apex-nodc1.json" <<'PY' +import json, glob, sys +rec = sorted(glob.glob(sys.argv[1] + "/netbox/draft/vr1-office1-current-*.json"))[-1] +d = json.load(open(rec)) +d["ipam/prefixes"] = [p for p in d.get("ipam/prefixes", []) + if (p.get("scope_site") or "") != "vr1-dc1"] +json.dump(d, open(sys.argv[2], "w")) +PY +APEX_RECORD="$TMP/apex-nodc1.json" runargs 2 'carries no v6 prefix for .* in vr1-dc1' \ + "T29 dc1 v6 legs really come from dc1's OWN apex entries (strip them -> REFUSE)" \ + "$TMP/good.yaml" --overlay "$REPO/overlays/vr1-dc1-vips.yaml" --dc vr1-dc1 # T30 the v6 analogue of T15: dc0 v6 legs under --dc vr1-dc1 must FAIL, so a # cross-DC copy-paste of a rendered overlay cannot pass. diff --git a/tests/render-dc-overlays/run-tests.sh b/tests/render-dc-overlays/run-tests.sh index c560841..9b710ea 100755 --- a/tests/render-dc-overlays/run-tests.sh +++ b/tests/render-dc-overlays/run-tests.sh @@ -32,11 +32,15 @@ # ---- derive ------------------------------------------------------------------ V4="$TMP/dc1-v4.yaml" # v4 target = the FROZEN fixture (immutable) -V="$TMP/dc1-dual.yaml" # live target = the dual-family overlay the tree deploys +# The dual target is the COMMITTED values file, not a fresh derive. `derive --from-overlay` +# lifts the header verbatim FROM the artifact being compared against, so deriving here and +# comparing against $LIVE made ~a third of the bytes self-referential and the header +# assertion unfalsifiable -- measured in the 2026-07-29 chain audit. Reading the committed +# values file makes T3b a real comparison and keeps it consistent with tests/render-drift. +V="$TMP/dc1-dual.yaml" if python3 "$R" derive --dc vr1-dc1 --from-overlay "$FIX" \ --output overlays/vr1-dc1-vips.yaml -o "$V4" 2>"$TMP/e1" \ - && python3 "$R" derive --dc vr1-dc1 --dual-family --include-unbuilt \ - --from-overlay "$LIVE" --output overlays/vr1-dc1-vips.yaml -o "$V" 2>>"$TMP/e1"; then + && cp "$REPO/render/values/vr1-dc1-vips.yaml" "$V"; then ok "T1 derive vr1-dc1 succeeds for both the v4 and dual-family shapes" else bad "T1 derive failed: $(head -1 "$TMP/e1")" diff --git a/tests/render-drift/run-tests.sh b/tests/render-drift/run-tests.sh new file mode 100755 index 0000000..b6575b1 --- /dev/null +++ b/tests/render-drift/run-tests.sh @@ -0,0 +1,74 @@ +#!/usr/bin/env bash +# tests/render-drift/run-tests.sh +# +# R9's RULED render-drift gate (D-119 AMENDMENT, 2026-07-27): "a render-drift check that +# fails the gauntlet on divergence", mirroring the D-137 creds-manifests derive-plus-drift +# pattern already proven in this repo. +# +# WHY IT EXISTS: until 2026-07-29 the ONLY byte-compare in the tree targeted the dc1 +# overlay. dc0's -- the artifact `scripts/preflight.sh` P2 actually feeds to the deploy +# gate -- was reproducible by nobody, and `render/values/` was referenced by ZERO scripts +# and ZERO harnesses. A chain audit executed two wrong-but-green attacks straight through +# that hole: hand-transposing two apps' octets in the dc0 overlay, and flipping ONE WORD +# (`family: dual` -> `v4`) to silently de-dual-stack dc0 against R2's ruling. Both passed +# every gate. This closes that class for every values file, present and future. +# +# It iterates render/values/*.yaml, so a NEW per-DC values file is gated the moment it is +# committed -- no harness edit required. +# +# Exit: 0 all pass | 1 any drift. ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO="$(cd "$HERE/../.." && pwd)" +R="$REPO/scripts/render-dc-overlays.py" +VALDIR="$REPO/render/values" +TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT +PASS=0; FAIL=0 +ok() { echo " PASS $1"; PASS=$((PASS+1)); } +bad() { echo " FAIL $1"; FAIL=$((FAIL+1)); } + +echo "== render-drift: every committed values file still renders its committed overlay ==" + +[ -d "$VALDIR" ] || { echo " FAIL render/values/ missing"; echo "RESULT: PASS=0 FAIL=1"; exit 1; } +n=0 +for v in "$VALDIR"/*.yaml; do + [ -e "$v" ] || continue + n=$((n+1)) + rel="${v#$REPO/}" + # The values file NAMES its own output; that field was inert until now. + out="$(python3 -c "import yaml,sys;print(yaml.safe_load(open(sys.argv[1])).get('output',''))" "$v" 2>/dev/null)" + if [ -z "$out" ]; then bad "$rel declares no output: path"; continue; fi + tgt="$REPO/$out" + if [ ! -r "$tgt" ]; then bad "$rel -> $out (target missing)"; continue; fi + if python3 "$R" check "$v" --against "$tgt" >"$TMP/d" 2>&1; then + ok "$rel renders $out byte-for-byte" + else + bad "$rel DRIFTED from $out -- re-render, do not hand-edit the overlay" + sed 's/^/ /' "$TMP/d" | head -8 + fi +done +# A zero-floor: an empty values dir must not read as a clean gate. +if [ "$n" -eq 0 ]; then bad "no values files found -- a drift gate over zero files is not a gate"; else ok "$n values file(s) checked (non-zero floor)"; fi + +# PROVEN ABLE TO FAIL: tamper with a COPY and require the same check to go red. A drift +# gate only ever observed passing is exactly what this file exists to replace. +probe="$(ls -1 "$VALDIR"/*.yaml 2>/dev/null | head -1)" +if [ -n "$probe" ]; then + cp "$probe" "$TMP/t.yaml" + python3 - "$TMP/t.yaml" <<'PY' +import sys, yaml +v = yaml.safe_load(open(sys.argv[1])) +v["apps"][0]["octet"] = int(v["apps"][0]["octet"]) + 40 # in-band, unique, plausible +yaml.safe_dump(v, open(sys.argv[1], "w"), sort_keys=False, width=4096) +PY + out="$(python3 -c "import yaml,sys;print(yaml.safe_load(open(sys.argv[1]))['output'])" "$probe")" + if python3 "$R" check "$TMP/t.yaml" --against "$REPO/$out" >/dev/null 2>&1; then + bad "PROOF-OF-TEETH: a mutated values file still matched -- this gate cannot fail" + else + ok "PROOF-OF-TEETH: a mutated values file is detected" + fi +fi + +echo +echo "RESULT: PASS=$PASS FAIL=$FAIL" +[ "$FAIL" -eq 0 ] && { echo "ALL PASS"; exit 0; } || exit 1