diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 1149a3f..3d0d275 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -855,8 +855,21 @@ `*.pem`, `*_ed25519`, `*_rsa`, `*openrc*`) -- the old six patterns could not have seen the SEC-020 secret or the predicted Stage-5 `~/admin-openrc`. Harnesses **creds-matrix 33/33** and **creds-audit 13/13** (was 7); gauntlet ALL GREEN (80), - repo-lint 0-fail. The LIVE three-site sweep (the plan's acceptance test for SEC-021's - on-disk half, SEC-022 and SEC-023) is the next gated step and has NOT run. + repo-lint 0-fail. + **LIVE TIER-2 SWEEP RUN 2026-07-26, read-only, no sudo** (capture + `docs/audit/d137-tier2-sweep-20260726.txt`; jumphost local + voffice1 + office1-netbox, + `stat` over ssh, metadata only). **SEC-021's ON-DISK half is now REPRODUCED as a named + failure** -- `vr1-dc0-maas-power_ed25519{,.pub}` are genuinely ABSENT from the dc0 + jumphost creds folder, not merely undeclared. The sweep ALSO corrected two of its own + false-greens, both found by running it: (a) an unprivileged `[ -d ]` on a root-owned + directory is indistinguishable from absent, so `/root/maas-secrets` and + `/root/netbox-secrets` first reported "does not exist" -- they are now correctly + reported **UNREADABLE** ("could not look" is never "nothing there"), which is a FAIL, + not a skip; (b) a role with any unprobed location no longer lets its other locations' + listings manufacture false EXPECTED-BUT-ABSENT findings -- 14 headend/netbox rows are + explicitly NOT JUDGED instead. **STILL OUTSTANDING: the two root-owned directories need + a privileged read**, so SEC-022's shadow-store verification and the SEC-020 region + secrets remain unconfirmed; that run is a remote-sudo shape and is operator-gated. ## 9. Additional defect found while authoring (FIXED in sweep Batch 0.3, 2026-07-19 -- wrap-aware exclusion, GA-F15; history below) diff --git a/docs/audit/d137-tier2-sweep-20260726.txt b/docs/audit/d137-tier2-sweep-20260726.txt new file mode 100644 index 0000000..8bafed6 --- /dev/null +++ b/docs/audit/d137-tier2-sweep-20260726.txt @@ -0,0 +1,33 @@ +# D-137 tier-2 EXISTENCE sweep -- live, read-only, 2026-07-26 +# +# Command (metadata only: stat over ssh, NO sudo, no file content read): +# python3 scripts/creds-matrix.py --tier2 --remote \ +# --pending-stage vr0-phase01 --pending-stage vr0-phase02 \ +# --pending-stage vr0-phase03 --pending-stage tenant-onboard +# +# Pending stages supplied by the CALLER from CURRENT-STATE (GA-R1: the script +# carries no status claim). Hosts probed: jumphost (local), voffice1, office1-netbox. + +=== creds-matrix: tier 1 (STATIC) === +=== creds-matrix: tier 2 (EXISTENCE) === + [ok] S1 schema: 72 rows, all enums valid, site-keys region-qualified, no duplicate (id,site,host-role,filename) + [ok] S3 render: 2 source field(s) SKIPPED -- rendering them needs the declared path from creds-manifests/vm-secret-locations (ruling 3); the list now EXISTS but the source-field derivation is not wired + [ok] S3 render drift: rendered row fields (mode, source) match checked-in; header prose and non-jumphost rows are OUT OF SCOPE of this compare + [ok] S4 mint-ref: every script:/runbook: reference resolves to a real location + [ok] S4 provenance debt: 27 row(s) are mint-ref=operator-terminal -- NOT reproducible from the repo (research FINDING 1). Admitted by design; converting them is remediation, not a checker fix. + [ok] S7 notes: 32 note key(s) referenced, all resolve, none orphaned + [ok] E0 jumphost location '~/vault-init/*' does not exist -- SKIPPED (typically a mint stage this deployment has not reached) + [ok] E0 jumphost location '~/octavia-pki/*' does not exist -- SKIPPED (typically a mint stage this deployment has not reached) + [ok] E1 17 expected artifact(s) deferred as not-yet-minted (--pending-stage) + [ok] E1 14 expected artifact(s) NOT JUDGED -- their role (headend, netbox) has at least one location that could not be probed, so absence cannot be asserted over it + [FAIL] S2 vr1-dc0 EXPECTED-BUT-ABSENT: 'opnsense-api.txt' (id dc0-edge-api, SEC-021) is expected by the matrix but NOT declared in the manifest -- the credential is either missing or undeclared + [FAIL] S2 vr1-dc0 EXPECTED-BUT-ABSENT: 'vr1-dc0-maas-power_ed25519' (id dc0-maas-power-key, SEC-021) is expected by the matrix but NOT declared in the manifest -- the credential is either missing or undeclared + [FAIL] S2 vr1-dc0 EXPECTED-BUT-ABSENT: 'vr1-dc0-maas-power_ed25519.pub' (id dc0-maas-power-key, SEC-021) is expected by the matrix but NOT declared in the manifest -- the credential is either missing or undeclared + [FAIL] S5 ASYMMETRY: vr1-dc0 declares id=dcN-maas-power-key file=maas-virsh_ed25519 on headend (custody=off-manifest-known) with no counterpart in vr1-dc1 -- a per-DC credential must exist at BOTH DCs in the same shape + [FAIL] S6 IDENTITY CONFLATION: id 'maas-region-admin' serves 2 principal types (human via gui; service via api, cli-profile) -- ruling 5 requires one identity to serve one principal type + [FAIL] E0 UNREADABLE: headend location '/root/maas-secrets/*' EXISTS but could not be listed -- no conclusion may be drawn about its contents; re-run with sufficient privilege + [FAIL] E0 UNREADABLE: netbox location '/root/netbox-secrets/*' EXISTS but could not be listed -- no conclusion may be drawn about its contents; re-run with sufficient privilege + [FAIL] E1 EXPECTED-BUT-ABSENT: dc0-maas-power-key 'vr1-dc0-maas-power_ed25519' expected at jumphost, not found (mint-stage stage3, SEC-021) + [FAIL] E1 EXPECTED-BUT-ABSENT: dc0-maas-power-key 'vr1-dc0-maas-power_ed25519.pub' expected at jumphost, not found (mint-stage stage3, SEC-021) + +FAIL: creds-matrix tier 1 -- 72 row(s), 10 check group(s) clean, 9 finding(s) diff --git a/docs/changelog-20260726-d137-tier1.md b/docs/changelog-20260726-d137-tier1.md index 98446f0..68f02b4 100644 --- a/docs/changelog-20260726-d137-tier1.md +++ b/docs/changelog-20260726-d137-tier1.md @@ -216,3 +216,38 @@ remediation is the departure and needs an exchange. - SEC-009's demotion to a pointer (ruling 4) -- not done; it belongs with tier 2. - No manifest was rewritten; no cloud state was touched. + +--- + +## Item 9 -- LIVE tier-2 sweep run (read-only) + two probe false-greens it exposed + +Capture: `docs/audit/d137-tier2-sweep-20260726.txt`. Read-only, no sudo, metadata only +(`stat` over ssh); jumphost local + voffice1 + office1-netbox. + +**ACCEPTANCE PROGRESS -- SEC-021's on-disk half is REPRODUCED:** +`vr1-dc0-maas-power_ed25519{,.pub}` are genuinely ABSENT from the dc0 jumphost creds +folder. Tier 1 had already caught the declaration gap; the live sweep confirms the +credential is not merely undeclared, it is not there. + +**Two false-greens in my own probe, both found only by running it against real hosts:** + +1. **"Absent" vs "could not look."** `/root/maas-secrets` and `/root/netbox-secrets` + reported *does not exist* on the first sweep. They exist -- SEC-020 is precisely about + `admin.pass` living in the first one. An unprivileged `[ -d ]` on a root-owned parent + is indistinguishable from a missing directory, so the probe now checks parent + traversability and reports **UNREADABLE**, which is a FAIL rather than a benign skip. + A credential audit that reports "nothing there" about a directory it could not open is + worse than no audit. +2. **Role-level aggregation manufactured false absences.** Skipping ONE location for a + role still let that role's other locations "cover" it, so every credential living in + the unprobed location reported EXPECTED-BUT-ABSENT. Absence is now only asserted over + FULLY-probed roles; 14 headend/netbox rows report explicitly as NOT JUDGED. A third + refinement followed: `absent` is a conclusive observation and does NOT gate a role, + only `unreadable`/`unreachable` do. + +Harness grew to 35 (T33 unreadable-location, T34 absent-location). + +**STILL OUTSTANDING -- needs an operator-gated privileged read.** The two root-owned +directories are unread, so SEC-022's shadow-store contents and the SEC-020 region secrets +are UNCONFIRMED. SEC-023's sprawl half is also unverified live. The remaining command is +a remote-sudo shape and was NOT run. diff --git a/scripts/creds-matrix.py b/scripts/creds-matrix.py index 0c37f34..39dfab7 100644 --- a/scripts/creds-matrix.py +++ b/scripts/creds-matrix.py @@ -420,32 +420,79 @@ return out +# Probes return (state, {basename: mode}) where state is one of: +# "ok" the location was read +# "absent" the directory does not exist (a stage not yet reached, usually) +# "unreadable" it EXISTS but could not be listed -- typically root-owned +# "unreachable" the host could not be contacted +# +# `unreadable` exists as its own state because of a false green found during the build: +# a shell glob over an unreadable directory does not expand, the loop finds no files, and +# the probe would have returned an EMPTY listing -- reporting "nothing undeclared here" +# about a directory it was never able to open. The region secrets dir (SEC-020) is exactly +# such a directory. "Could not look" must never be recorded as "nothing there". +def _dir_of(pattern): + return os.path.dirname(pattern) if pattern.endswith("/*") else None + + def probe_local(pattern): - """{basename: mode} for files matching a local pattern. METADATA ONLY.""" + """METADATA ONLY -- stat, never file content.""" + d = _dir_of(pattern) + if d: + dx = os.path.expanduser(d) + # A glob (~/tenant-*/*) may legitimately match nothing; only a literal + # directory path can be judged absent or unreadable. + if not glob.has_magic(dx): + if not os.path.isdir(dx): + # CANNOT conclude "absent" if the PARENT is not traversable: to an + # unprivileged reader a root-owned directory is indistinguishable from a + # missing one. Reporting it absent would be a benign-looking skip over a + # location that exists and holds secrets -- measured on the region + # secrets dir (SEC-020), which read "does not exist" on the first sweep. + parent = os.path.dirname(dx.rstrip("/")) or "/" + if not os.access(parent, os.R_OK | os.X_OK): + return ("unreadable", {}) + return ("absent", {}) + if not os.access(dx, os.R_OK | os.X_OK): + return ("unreadable", {}) found = {} for f in glob.glob(os.path.expanduser(pattern)): if os.path.isfile(f): found[os.path.basename(f)] = oct(os.stat(f).st_mode & 0o777)[2:] - return found + return ("ok", found) def probe_remote(target, pattern): - """{basename: mode} over ssh. METADATA ONLY -- `stat`, never file content. - Returns None if the host is unreachable, so the caller can skip EXPLICITLY.""" - cmd = 'for f in %s; do [ -f "$f" ] && stat -c "%%n %%a" "$f"; done' % pattern + """METADATA ONLY over ssh -- stat, never file content.""" + d = _dir_of(pattern) + probe = "" + if d: + # See probe_local: "not a directory" is only ABSENT when the parent is + # traversable; otherwise the truth is that we could not look. + probe = ('p=$(dirname %s); ' + 'if [ ! -d %s ]; then ' + ' if [ -r "$p" ] && [ -x "$p" ]; then echo "__STATE__ absent"; ' + ' else echo "__STATE__ unreadable"; fi; ' + 'elif [ ! -r %s ] || [ ! -x %s ]; then echo "__STATE__ unreadable"; fi; ' + % (d, d, d, d)) + cmd = probe + 'for f in %s; do [ -f "$f" ] && stat -c "%%n %%a" "$f"; done' % pattern try: p = subprocess.run(["ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=10", target, cmd], capture_output=True, text=True, timeout=60) except (OSError, subprocess.SubprocessError): - return None - if p.returncode != 0 and not p.stdout.strip(): - return None + return ("unreachable", {}) + if p.returncode == 255: # ssh transport failure + return ("unreachable", {}) found = {} for line in p.stdout.splitlines(): + if line.startswith("__STATE__ "): + return (line.split(None, 1)[1].strip(), {}) bits = line.rsplit(None, 1) if len(bits) == 2: found[os.path.basename(bits[0])] = bits[1] - return found + if p.returncode != 0 and not found: + return ("unreachable", {}) + return ("ok", found) def tier2_existence(rows, locations, use_remote, pending_stages, oks, fails): @@ -456,20 +503,44 @@ by_role.setdefault(role, []).append((target, pat)) observed = {} # role -> {basename: mode} - skipped_remote = unreachable = 0 + # A role is COMPLETE only if EVERY declared location for it was probed. Absence can + # only be asserted over a fully-probed role: with one location skipped, the other + # locations' listings would "cover" the role and manufacture false EXPECTED-BUT-ABSENT + # findings for credentials that live in the location nobody looked at. Measured on the + # first live sweep, where skipping the region secrets dir made every credential in it + # report as missing from the headend. + incomplete = set() + for role, _t, _p in locations: + observed.setdefault(role, {}) + skipped_remote = 0 for role, target, pat in locations: if target != "local": if not use_remote: skipped_remote += 1 + incomplete.add(role) continue - seen = probe_remote(target, pat) - if seen is None: - unreachable += 1 - oks.append("E0 %s (%s) UNREACHABLE -- location '%s' SKIPPED explicitly; " - "an unreachable host is never a pass" % (role, target, pat)) - continue + state, seen = probe_remote(target, pat) else: - seen = probe_local(pat) + state, seen = probe_local(pat) + # "absent" is a CONCLUSIVE observation -- we looked and the location is not there, + # so it hides nothing. Only "could not look" states gate the role. + if state in ("unreadable", "unreachable"): + incomplete.add(role) + if state == "absent": + oks.append("E0 %s location '%s' does not exist -- SKIPPED (typically a mint " + "stage this deployment has not reached)" % (role, pat)) + continue + if state == "unreadable": + # NOT a pass and NOT an empty listing: the location exists and could not be + # opened, so nothing may be concluded about what is or is not inside it. + fails.append("E0 UNREADABLE: %s location '%s' EXISTS but could not be listed " + "-- no conclusion may be drawn about its contents; re-run with " + "sufficient privilege" % (role, pat)) + continue + if state == "unreachable": + oks.append("E0 %s (%s) UNREACHABLE -- location '%s' SKIPPED explicitly; " + "an unreachable host is never a pass" % (role, target, pat)) + continue observed.setdefault(role, {}).update(seen) if skipped_remote: @@ -480,6 +551,7 @@ # bound 1: every expected artifact present at its expected role absent = 0 deferred = 0 + gated = 0 for r in rows: if not r.has_artifact() or r.host_role not in by_role: continue @@ -492,6 +564,9 @@ fails.append("E2 MODE: %s '%s' at %s is %s, want %s" % (r.id, r.filename, r.host_role, got, want)) continue + if r.host_role in incomplete: + gated += 1 + continue if r.mint_stage in pending_stages: # NOT-YET-EXPECTED, declared by the CALLER. The script carries no status # claim of its own (GA-R1: status lives in CURRENT-STATE, nowhere else). @@ -520,9 +595,14 @@ if deferred: oks.append("E1 %d expected artifact(s) deferred as not-yet-minted (--pending-stage)" % deferred) + if gated: + oks.append("E1 %d expected artifact(s) NOT JUDGED -- their role (%s) has at least " + "one location that could not be probed, so absence cannot be asserted " + "over it" % (gated, ", ".join(sorted(incomplete)))) if not (absent or undeclared): oks.append("E1/E3 existence: every expected artifact present and nothing " - "undeclared, across %d probed role(s)" % len(observed)) + "undeclared, across %d fully-probed role(s)" + % len([r for r in observed if r not in incomplete])) def main(): diff --git a/tests/creds-matrix/run-tests.sh b/tests/creds-matrix/run-tests.sh index 39ae751..144e4f2 100644 --- a/tests/creds-matrix/run-tests.sh +++ b/tests/creds-matrix/run-tests.sh @@ -310,6 +310,27 @@ && ok "T32 an unreachable host -> explicit UNREACHABLE skip, never counted as clean" \ || { no "T32 unreachable must skip loudly (rc=$RC)"; sed 's/^/ /' "$D/out"; } +# T33 -- FALSE-GREEN REGRESSION: a declared location that EXISTS but cannot be listed. +# A shell glob over an unreadable directory does not expand, so a naive probe finds no +# files and reports "nothing undeclared here" about a directory it never opened. The +# region secrets dir (SEC-020) is exactly such a directory. +D=$(mkenv); mkloc "$D"; mkdir -p "$D/locked"; printf 'x\n' > "$D/locked/hidden.key" +chmod 000 "$D/locked" +printf 'jumphost local %s/locked/*\n' "$D" >> "$D/locations" +run "$D" --tier2 +chmod 755 "$D/locked" +[ "$RC" = 1 ] && grep -q 'E0 UNREADABLE' "$D/out" \ + && ok "T33 an unreadable declared location -> UNREADABLE FAIL, never an empty listing" \ + || { no "T33 unreadable must not read as empty (rc=$RC)"; sed 's/^/ /' "$D/out"; } + +# T34 -- a declared location that simply does not exist yet is a SKIP, not a failure. +D=$(mkenv); mkloc "$D" +printf 'jumphost local %s/not-created-yet/*\n' "$D" >> "$D/locations" +run "$D" --tier2 +[ "$RC" = 0 ] && grep -q 'does not exist -- SKIPPED' "$D/out" \ + && ok "T34 a not-yet-created declared location -> explicit SKIP, not a failure" \ + || { no "T34 absent location handling (rc=$RC)"; sed 's/^/ /' "$D/out"; } + echo if [ "$F" = 0 ]; then echo "creds-matrix: $P/$P PASS"; exit 0; fi echo "creds-matrix: FAILURES: $F (passed $P)"; exit 1