| 2026-07-26 |
D-137 tier 3 behavioural half: V2 declared-state + creds-probe.sh
...
Boundary and cadence operator-ruled: "Full -- including OpenStack service
credentials", "Stage close + DC standup close". I had recommended
passive-only and flagged the costs; the operator ruled otherwise.
V2 (passive, zero authentication) fires TODAY on a real defect: SEC-006 has
recorded the NetBox upstream token as BURNED since 2026-07-12 while the
matrix carried it as a healthy consolidated row. Nothing reconciled the two
surfaces. Keyword matching on ledger prose is fragile, so it is conservative
and reports the matched text as its basis.
creds-probe.sh is SEPARATE by design: a probe must USE a credential, and
creds-matrix.py is metadata-only by an enforced invariant (T22/T51). Folding
probes in would have relaxed that guard, and a guard relaxed once stays
relaxed.
Safety: never retry a failed auth (retry loops lock accounts; the MAAS admin
identity backs 19 call sites), never argv, never echo, stage-aware skips, and
per-tenant credentials REFUSED BY CONSTRUCTION -- a declaration cannot opt
them back in (D-069, ruling 3). Not a deploy gate.
FALSE GREEN in my own first run: ssh inside a while-read loop ate the loop's
stdin, so 1 of 13 probes ran and the run exited 0. A partial run reporting
clean is the exact failure class this tool exists to catch. Fixed with ssh -n
and locked by a total-accounting check that makes a partial run VOID.
13 probes declared: 3 active, 4 stage-pending, 6 declared-only with reasons
recorded. OpenStack credentials declared now, skipping until Stage 5.
tests/creds-probe 9/9; gauntlet 80 -> 81; phase-3 DoD wired.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-137 tier 2: local half blocking in P5; --pending-stage -> coupled projection
...
Two operator decisions, both taking the recommendation.
GATE SHAPE: local-only blocking, remote operator-run. preflight P5 now runs
creds-matrix.py --tier2 -- tier 1 STATIC plus tier 2 LOCAL, hermetic (no ssh,
no sudo). It already catches SEC-021's three dc0 on-disk absences without
leaving the jumphost. The remote half stays an operator-run check at
DC-standup close and stage close.
Wiring --remote --privileged into the deploy gate would let a host outage, a
NOPASSWD change, or a MAAS snap refresh turn the gate red for reasons that
are not about credentials, and a gate that fails environmentally is one
people learn to bypass. The repo already made this call once
(run-tests-all.sh:7-8, "different cadence"), and preflight already shows the
failure mode: P4 reads "MAAS unreachable" because it runs on vcloud while
MAAS lives on the headend.
STAGE INPUT: --pending-stage replaced by creds-manifests/stages-reached, a
machine-readable projection of CURRENT-STATE, coupled by new repo-lint L12
(the L10 pattern). A hand-typed gate parameter can be widened silently, and
GA-R1 puts status in CURRENT-STATE alone. A drifted projection would make the
gate defer REAL absences -- a false green.
Exhaustive both ways: an unclassified mint-stage is a hard FAIL, a
classified-but-unused one is a stale-declaration FAIL. That caught `stage4`
immediately -- Stage 4 is open and executing but mints no credential.
Self-inflicted break caught by the existing harness: I first renamed L10's
message to "L10/L12", breaking five cases that grep the established L10 id.
Reverted -- new rules get their own message id, they do not rebrand old ones.
Harnesses: creds-matrix 52 -> 56, repo-lint 45 -> 47, preflight 10.
Gauntlet ALL GREEN (80), repo-lint 0-fail, baseline unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-137 tier 3 (VALIDITY) BUILT: cross-copy sha256 provenance
...
Closes the gap the plan names: creds-audit PARSES declared provenance and
never verifies it, so SEC-020's stale-trap warning has been unchecked prose
and its sha256 equality was a ONE-TIME manual act on 2026-07-25.
sha256sum runs as an EXTERNAL command on both sides so only a 64-hex digest
enters the checker -- hashing bytes in Python would meet the letter of "never
transfer content" and not the spirit.
RESULT: 4 groups compared, all byte-identical, INCLUDING the SEC-020 pair
(maas-region-admin, region source-of-record vs jumphost copy). Now
machine-verified instead of remembered. The [ok] line names the groups.
TWO FALSE POSITIVES IN MY OWN DRAFT, caught by verifying before reporting.
Keying digests by (role, site) let an identity's DISTINCT artifacts at one
scope overwrite each other, so it compared unrelated files and reported drift
on maas-region-admin and dc1-svc-key. Manual sha256sum showed both IDENTICAL.
Reporting that would have raised a false credential-rotation alarm.
Root cause was a schema gap: nothing expressed which rows are byte-copies.
custody gains `verbatim-copy` [OPS]. V1 now compares only rows sharing
(id, filename) across scopes, or a source-of-record row against declared
verbatim-copy rows -- SEC-020's exact shape. Everything else is skipped with
a reason: a pair we cannot justify comparing is not a pair we may report
drift on. verbatim-copy is manifest-bearing; it asserts byte-identity, it
does not move the file.
Harness 44 -> 52: T47/T48 regression-lock both false positives; T49 keeps
SEC-020's case from falling into T48's skip; T51 asserts the digest path uses
sha256sum and that no content-transfer verb exists anywhere in the checker.
NOT built: tier 3's behavioural probes. Liveness, expiry, revocation state
and grant/scope drift remain unverified.
Gauntlet ALL GREEN (80), repo-lint 0-fail, finding-class baseline unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-137 phase 3: re-measure the register; SEC-024 opened
...
~21 inferred filenames corrected against their own mint-refs. The session had
owned six; those were only the rows the first sweep physically touched.
Octavia's 8 real basenames plus its THREE SUBDIRECTORIES (the bare
~/octavia-pki/* pattern matched ZERO of the 8 artifacts, two of them CA
private keys); vault's init.txt; the tenant rows' <client>- instance prefix,
with a placeholder matcher so they can match a real basename at all;
kube-config -> config; .maas.cli -> .maascli.db (measured: ~/.maas.cli does
not exist).
sec-ref mis-attribution corrected: both juju-maas-user rows cited SEC-020;
the on-subject rows are SEC-018/-019. This is the SEC-015 class recurring --
one instance was fixed by hand earlier without checking whether it was
systemic. It was.
mint-ref corrected: admin.pass cited a CONSUMER (:452), not the mint (:450);
the RGW key cites a line that takes it as INPUT, so it is operator-terminal.
Rows added: the libvirt SSH power password (standing rotation obligation from
a 2026-06-26 plaintext exposure, in neither the matrix nor the exclusions),
vault-ca-root, and the tfstate siblings. ~/as-executed/ deliberately NOT
declared, reason recorded: it holds session logs, so declaring it made every
log a false UNDECLARED.
Four further defects in my own phase-2 changes, found by RUNNING the sweep:
shlex.quote broke ~ expansion; a non-existent parent read as unreadable
rather than absent; site-key `-` needed shared-location semantics; the
tfstate siblings needed rows.
SEC-024 OPENED -- new exposure found by the fixed checker on its first clean
run: opentofu/terraform.tfstate.backup is mode 0664 and carries the MAAS API
key in plaintext (DOCFIX-175); the live state file is 0600. Invisible to
every prior control because the world-readable check was custody-gated and
the siblings were undeclared. LOGGED, NOT ACTIONED (hard rule 1).
Findings 35 -> 13, all true. Matrix 81 rows. Harness 44/44, creds-audit
15/15, gauntlet ALL GREEN (80), repo-lint 0-fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-137 phase 2: fix the eight confirmed false-greens, regression-lock each
...
Tier 2 is now scoped by (host-role, SITE-KEY). Keying on role alone merged
all three jumphost creds folders into one basename namespace, so a credential
present at ANY site satisfied the row for EVERY site -- how SEC-021's dc0
opnsense-api.txt absence passed a green sweep while tier 1 flagged it. Four
committee lenses converged on this. vm-secret-locations gains a site column.
Also fixed, each reproduced by the committee and each now regression-locked:
- probe_remote called a successfully-read location UNREACHABLE when its glob
matched nothing, gating the role and turning every absence FAIL there into
an [ok]; a __DONE__ liveness marker now separates "ran, found nothing" from
"could not reach".
- literal-file locations skipped absent/unreadable detection entirely (the
T33 false green surviving in the other path form).
- an EMPTY locations list bypassed ruling 3's refusal.
- a zero-row matrix reported clean over the empty set.
- a mint-ref pointing at a directory crashed with exit 1 -- indistinguishable
from findings -- and skipped S5/S6/S7, silently dropping the conflation and
asymmetry findings.
- mode enforcement was custody-gated, leaving 43 of 77 rows unchecked
including the SEC-020 and SEC-022 stores; a world-readable check now
applies at any custody.
- S5's dc1->dc0 direction was untested; deleting it left the harness green.
- rows with no declared location for their scope are disclosed, not skipped.
- declared paths/targets are charset-validated and shell-quoted: an
unvalidated `;` reached a remote shell, under sudo when --privileged.
- creds-audit sprawl globs now cover dotfiles; the prior widening could not
see .admin.pass, the dotfile sibling of its own motivating secret.
T24 replaced with a checked-in finding-CLASS baseline. The old form grepped
for the literal "IDENTITY CONFLATION", so REMEDIATING that defect would have
turned the gauntlet red for whoever did the right thing -- making "edit the
test" the cheapest response exactly when the guard mattered.
Harness 35 -> 43, creds-audit 13 -> 15, gauntlet ALL GREEN (80), lint 0-fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-137 tier 2: wire --privileged escalation (run BLOCKED by the auto-mode classifier)
...
--privileged retries a remote probe with `sudo -n` ONLY where the
unprivileged probe returned `unreadable`, keeping the privileged surface as
small as ruling 3 keeps the search surface. Metadata only; the T22
metadata-only guard still passes; verified inert (zero escalations) when
nothing is unreadable.
The operator approved the privileged read, but the auto-mode classifier
denied the remote-sudo shape -- the same wall the 2026-07-23 close recorded,
whose noted fix is manual permission mode. NOT worked around: reading a
root-owned directory is exactly the privilege being denied, so any
alternative would fail identically or misrepresent the result.
SEC-022 shadow-store contents and the SEC-020 region secrets stay
UNCONFIRMED; the sweep reports them as E0 UNREADABLE FAILs, which is the
correct outcome for an audit that could not look.
Also de-staled the docstring tier table (tier 2 is built).
Harness 35/35, gauntlet ALL GREEN (80), repo-lint 0-fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-137 tier 2: live read-only sweep + two probe false-greens it exposed
...
SEC-021's ON-DISK half REPRODUCED: vr1-dc0-maas-power_ed25519{,.pub} are
genuinely absent from the dc0 jumphost creds folder, not merely undeclared.
Capture: docs/audit/d137-tier2-sweep-20260726.txt (read-only, no sudo,
stat-over-ssh metadata only; jumphost + voffice1 + office1-netbox).
Running it against real hosts exposed two false-greens in the probe:
1. "absent" vs "could not look". /root/maas-secrets and /root/netbox-secrets
reported "does not exist" -- they exist; SEC-020 is precisely about
admin.pass living in the first. An unprivileged [ -d ] on a root-owned
parent is indistinguishable from a missing directory. The probe now tests
parent traversability and reports UNREADABLE as a FAIL, not a skip.
2. Role-level aggregation manufactured false absences: skipping one location
let a role's other locations cover it, so credentials in the unprobed
location reported EXPECTED-BUT-ABSENT. Absence is now asserted only over
fully-probed roles (14 rows report NOT JUDGED). Refinement: "absent" is a
conclusive observation and does not gate a role; only unreadable and
unreachable do.
Harness 33 -> 35 (T33 unreadable location, T34 absent location).
Gauntlet ALL GREEN (80), repo-lint 0-fail.
OUTSTANDING: the two root-owned dirs need a privileged read, so SEC-022's
shadow-store contents and the SEC-020 region secrets stay UNCONFIRMED. That
is a remote-sudo shape and was not run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-137 tier 2 (EXISTENCE): declared-location bound, existence checks, widened globs
...
Tooling only -- the LIVE three-site sweep has NOT run, so SEC-021's on-disk
half, SEC-022 and SEC-023 are not yet reproduced as live findings.
creds-manifests/vm-secret-locations (ruling 3): the absolute bound on
discovery. Covers the jumphost creds folders, the SEC-022 headend shadow
stores, the SEC-020 region secrets dir, the three dirs outside the SEC-009
convention, plus two the ruling's note did not name -- the in-clone PKI
overlay and the DOCFIX-175 plaintext tfstate. Tenant dirs are LOCAL-only:
no tenant VM/project/host is reachable from this file, so D-069 holds by
construction rather than by good behaviour. `<repo>` is substituted at
runtime (D-110: the repo name is never hardcoded).
creds-matrix.py --tier2 [--remote]: E1 expected-but-absent, E2 mode, E3
undeclared-at-a-declared-location. Probes are `stat` over ssh -- metadata
only, never content. An unreachable host SKIPS with an explicit [ok],
because "could not look" must never read as "nothing there".
--pending-stage is supplied by the CALLER so a not-yet-reached mint stage
defers instead of failing; encoding which stages have run would put a
status claim in code, and GA-R1 puts status in CURRENT-STATE only.
creds-audit.sh sprawl globs widened for the SEC-023 blind spots
(admin.pass, *.apikey, *.key, *.pem, *_ed25519, *_rsa, *openrc*). The old
six patterns could not have matched the SEC-020 secret that sat undetected
12 days, any bare SSH key or PEM, or the predicted Stage-5 ~/admin-openrc.
Harnesses: creds-matrix 33/33 (8 new tier-2 cases), creds-audit 13/13
(was 7; five blind-spot fixtures plus T13 asserting the widening did NOT
become trigger-happy). Gauntlet ALL GREEN (80), repo-lint 0-fail.
FLAGGED DIVERGENCE from the plan's Files list: --remote is implemented in
creds-matrix.py, not creds-audit.sh. Tier 2 IS the preflight Pn and the
matrix plus locations list both live there; splitting it would mean two
implementations of one bounded walk. Ruling 3 names a bound, not a script.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-137 tier 1 (STATIC): credential matrix, checker, harness -- RED by design
...
Schema amended 10 -> 12 columns (operator-ruled 2026-07-26, "go with the
12-column amendment"): adds `custody` + `notes-ref` and re-keys a row to one
(credential, location) pair. A read-first round-trip check of the three
manifests proved the 10-column form could not carry what D-137 ruling 2
forbids dropping -- and ruling 3 puts four deliberate credential copies inside
--remote's declared locations, where creds-audit.sh:63-67 would report each as
UNDECLARED. Implements ruling 2 ("prose must become MATRIX FIELDS"); OPS under
GA-R3, the five sub-rulings untouched.
Ships: creds-matrix.tsv (72 rows), creds-matrix-notes.md (32 keys),
scripts/creds-matrix.py (S1 schema / S2 both-bounds manifest coverage / S3
render drift / S4 mint-ref resolution / S5 per-DC symmetry / S6 ruling-5
principal invariant / S7 notes integrity), tests/creds-matrix 24/24.
Gauntlet ALL GREEN (80, was 79); repo-lint 0-fail.
RED BY DESIGN, 5 findings -- do not go green by deleting rows: the ruling-5
identity conflation on maas-region-admin (the live SEC-020 defect), 3x
EXPECTED-BUT-ABSENT for SEC-021's declaration half, and the S5 asymmetry on
dc0's divergently-named headend power key. 27 operator-terminal rows are the
FINDING 1 reproducibility debt, counted not faulted.
Two self-inflicted false-greens found and closed during the build: S2
enumerated only matrix-mentioned sites (an empty matrix read CLEAN -- the
sandbox-fidelity-check.py:131-143 defect one level up; now the union of both
sides, regression-locked by T9), and the uniqueness key false-failed two
identities sharing one file (T6).
NOT done, stated so the boundary is unambiguous: tiers 2-3, vm-secret-locations,
the SEC-009 demotion, and the ruling-1 preflight Pn -- whose blocking-vs-deferred
sequencing is an OPEN operator question. No manifest rewritten; no cloud state
touched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
