# Session changelog 2026-07-26 -- D-137 credential matrix, tier 1 (STATIC) Session-scoped scratch (GA-R2/D1). NOT a status or decision authority -- status lives only in `docs/CURRENT-STATE.md`, rulings only in `docs/design-decisions.md`. Consolidated at stage close. Branch `dc-dc-stage4-phase3-maas-deploy`. No cloud state touched: every item below is offline repo work. Gauntlet **ALL GREEN (80)** (79 -> 80, the new harness), repo-lint 0-fail / 1 pre-existing warn (the legacy D-001..018 ASCII carve-out). --- ## Item 1 -- D-137 matrix schema amended 10 -> 12 columns (operator-ruled) **What.** `docs/D-137-implementation-plan.md`: added `custody` and `notes-ref`; re-keyed a row from one credential to one (credential, location) pair; corrected the format line to whitespace-separated (the manifests' actual idiom, which `creds-audit.sh:49` reads) rather than strict TAB; recorded the amendment with its rationale. **Why.** A read-first round-trip check of the three checked-in manifests against the 10-column schema found it could not carry three classes of governance prose that D-137 ruling 2 forbids dropping: - **off-manifest carve-outs** -- four deliberate, reasoned credential copies (both DCs' Juju client-store keys, the dc1 MAAS-snap power key, and the three ruled not-consolidated region files). Ruling 3 puts their locations INSIDE `--remote`'s declared set, so `creds-audit.sh:63-67` will visit every one and report it UNDECLARED. Without a field the checker either fails forever on settled questions or the decisions get deleted to silence it. - **multi-copy credentials** -- one credential, several locations, DIFFERENT filenames (`admin.pass` vs `maas-admin-password`), one designated authoritative copy. Tier 3's digest comparison cannot know what to compare without it. - **verification-relevant detail** -- format specs, mint dates, the not-reproducible debt. Ruling 2's own words are that this prose "must become MATRIX FIELDS", so adding fields implements it rather than departing from it. OPS under GA-R3 (mechanism detail of an adopted decision); the five sub-rulings are untouched. **Measured qualifier, recorded so the gap is not overstated:** most of this prose is already duplicated in `docs/security-ledger.md` (SEC-018 carries the Juju-store copy; SEC-020 carries source-of-record, the stale-trap warning, and all three carve-out reasons). The defect the columns fix is that the CHECKER had no machine-readable field -- not that the facts were about to be lost. **Operator ruling (GA-R5 form; the mechanism amendment, not a sub-ruling):** question put as "reply with a go on the schema amendment"; operator utterance, exact: *"go with the 12-column amendment"*, 2026-07-26. **Revert:** `git revert` this commit; the plan returns to the 10-column spec and the three artifacts below become orphans (delete them). ## Item 2 -- `creds-matrix.tsv` + `creds-matrix-notes.md` (the register) **What.** 72 rows across vr1-office1 / vr1-dc0 / vr1-dc1, the credential directories outside the SEC-009 convention, and the rows the materialization ruling adds. 32 note keys. **Why the row count is not 55.** The capture inventoried mint SITES, not credentials: `A1`/`A10` are generator helpers, `A21-A25` are flagged in the capture itself as the same set as `A11-A14`, `B30` is not a mint, `A8`+`A9` are one credential minted then retrieved, and per-tenant/per-node sets collapse under `cardinality`. Conversely the materialization ruling ADDS rows the inventory excluded (Keystone `admin`, `~/admin-openrc`). Derived from identity, never transcribed 1:1. **Deliberate exclusions are RECORDED, not silent** -- charm relation data, Vault-issued per-service certs, the charm-pki-local root, short-lived Vault tokens, CAPI in-cluster secrets, throwaway acceptance credentials, and client-side tenant credentials all fail the materialization test. The reasoning is in the notes file so a later reader can challenge it. **Revert:** delete both files. ## Item 3 -- `scripts/creds-matrix.py` tier 1 (STATIC) **What.** Seven check groups in the `provider-bundle-check.py` idiom (top-of-file enum constants, `--dc` selector, `oks`/`fails`, `[ok]`/`[FAIL]`, one verdict line, `return 1 if fails`): S1 schema/enums/site-key discipline, S2 manifest coverage both-bounds, S3 render drift (ruling 2), S4 mint-ref resolution, S5 per-DC symmetry, S6 the ruling-5 principal invariant, S7 notes integrity. `--render` prints derived manifests; it does NOT write. **Two design decisions worth challenging at review:** - The manifest **mode column is DERIVED** (`.pub` -> 644, else 600) rather than becoming a 13th column. That rule reproduces all three checked-in manifests exactly. - **`--render` deliberately does not write.** Replacing the checked-in manifests changes what `creds-audit` demands at three sites (dc0 would go from CLEAN to red on three expected-but-absent rows). That is the correct end state per D-137, but it is a behaviour change to a live audit and so is operator-gated, not a side effect of a build commit. `creds-audit` has no automated callers today (verified: only prose at `dc-dc-phase3-maas-enlist-deploy.md:498`), so nothing is destabilised by waiting. **Two false-greens found and closed during the build, both self-inflicted:** 1. **S2 enumerated only the sites the MATRIX mentions**, so an empty or site-truncated matrix examined no manifest, found nothing undeclared, and reported "both bounds clean". That is precisely the defect `netbox/sandbox-fidelity-check.py:131-143` records in-repo ("an upper bound masquerading as an assertion. It must be BOTH bounds") reproduced one level up. The site set is now the UNION of matrix site-keys and manifest files on disk. Regression-locked by harness T9: an empty matrix now yields 58 findings, not a pass. 2. The uniqueness key was (site, host-role, filename), which false-failed two DISTINCT identities sharing one file -- the Vault init artifact holds both the unseal shares and the root token. Key now includes `id`. Regression-locked by T6. **Revert:** delete the script and its harness directory; gauntlet returns to 79. ## Item 4 -- `tests/creds-matrix/run-tests.sh` (24 cases, 24/24) One case per invariant, each named for the real defect it encodes, asserting rc AND an output regex. Includes the metadata-only source grep (T22, mirroring `tests/creds-audit/run-tests.sh:71-73`) and **T24, which asserts the real tree stays RED** -- a guard against someone making the register green by deleting the rows that fail. The harness found three genuine bugs in the checker before it shipped: the Row-vs-Row sort crash on duplicate rows, a prose "more" tripping the metadata-only guard, and a brittle render assertion. All three fixed. **Revert:** delete `tests/creds-matrix/`. ## Item 5 -- corrected a wrong `sec-ref` before it could be cited as authority `dc1-edge-api` carried `SEC-015`. Verified against `docs/security-ledger.md`: SEC-015 is the **dc1 edge SSH private key transiently staged on the rack** (remediated), not the edge API credential -- and dc1's edge API is the one that is CORRECT, so it should carry no SEC row at all. Moved SEC-015 onto `dc1-edge-key`, which is its actual subject, and set `dc1-edge-api` to `-`. Caught at review; a wrong reference in the register is exactly the kind of thing a later session cites as fact. Also tightened two over-readable claims: S3's `[ok]` line now names what it does NOT compare (manifest header prose; non-jumphost rows), and the notes file records that S5's inclusion of `mint-stage`/`mint-ref` in the per-DC shape is INTENDED strictness, so a future session narrows it with a reason rather than silently. --- ## EXPECTED RED -- the 5 findings are the deliverable, not a defect in it `python3 scripts/creds-matrix.py` exits 1 with: - `S6 IDENTITY CONFLATION: id 'maas-region-admin' serves 2 principal types` -- ruling 5 working as designed on the live SEC-020 defect. Clears when the 19 `maas admin` call sites migrate to a service identity. - `S2 ... EXPECTED-BUT-ABSENT` x3 (dc0 `opnsense-api.txt`, `vr1-dc0-maas-power_ed25519`, `.pub`) -- SEC-021's declaration half. - `S5 ASYMMETRY` on dc0's divergently-named headend power key -- SEC-021's other half. 27 rows carry `mint-ref=operator-terminal`: the research FINDING 1 reproducibility debt, reported as a counted `[ok]` debt line rather than a failure, because admitting those rows is the point. Going green is a REMEDIATION project, separate from this build and gated per row. --- # TIER 2 (EXISTENCE) -- tooling built, live sweep NOT run ## Item 6 -- `creds-manifests/vm-secret-locations` (ruling 3) The absolute bound on discovery. Includes every class the ruling's faithful-implementation note names, plus two the note did not: the in-clone PKI overlay and the DOCFIX-175 plaintext `terraform.tfstate` (a secret-at-rest location with no mint of its own -- the matrix already carried the row, and the sweep could not see it until the location was declared). **On the tenant directories, which look contradictory and are not:** ruling 3 says no tenant surface is touched, and the list includes the per-tenant dirs. Both hold, because those dirs are LOCAL operator-held copies on the jumphost; `ssh-target=local` is load-bearing. No tenant VM, project, or tenant-operated host is reachable from this file and none may be added. D-069 is preserved by construction, not by good behaviour. `` is substituted at runtime from the checker's own location -- the repo has been renamed once (D-110) and its name is never hardcoded into a data file. **Revert:** delete the file; tier 2 then refuses to run (by design, T30). ## Item 7 -- tier-2 EXISTENCE checks in `creds-matrix.py` E1 expected-but-absent, E2 mode, E3 undeclared-at-a-declared-location. `--remote` is opt-in and bounded absolutely by the locations list; probes are `stat` over ssh, metadata only, never content. **Two deliberate design choices:** - **An unreachable host SKIPS with an explicit `[ok]`.** Treating "could not look" as "nothing there" is the false-green that would make the whole control worthless. T32. - **`--pending-stage` is supplied by the CALLER, not the script.** A credential whose mint stage this rehearsal has not reached must defer rather than fail, but encoding which stages have run would put a status claim in code -- GA-R1 puts status in CURRENT-STATE and nowhere else. T27. **Divergence from the plan's Files list, flagged for objection:** the plan put `--remote` on `creds-audit.sh`. It is implemented in `creds-matrix.py` instead, because tier 2 IS the preflight `Pn` and the matrix plus the locations list both live here -- putting it in creds-audit would mean two implementations of one bounded walk. `creds-audit.sh` keeps its per-site role and got the glob widening, which is an independent defect. Ruling 3 does not name a script, only the bound. ## Item 8 -- `creds-audit.sh` sprawl globs widened (SEC-023) Added `*.pass`, `*.apikey`, `*.key`, `*.pem`, `*_ed25519`, `*_rsa`, `*openrc*`. The original six patterns had measured blind spots: they would not have matched `admin.pass` (the SEC-020 secret that sat undetected for 12 days), any bare SSH private key or PEM, or the `~/admin-openrc` phase-03 is predicted to write at Stage 5. Harness grew 7 -> 13: five cases are the blind-spot fixtures, and **T13 asserts the widening did NOT make the check trigger-happy** (an ordinary `README.md`/`notes.txt` in the home root is still CLEAN). **Revert:** restore the six-pattern glob line and drop harness T8-T13. --- ## NOT done in this session (stated so the boundary is unambiguous) - **The LIVE tier-2 sweep has NOT run.** The tooling exists and is harnessed, but no remote host has been probed, so SEC-021's on-disk half, SEC-022's shadow stores and SEC-023's sprawl are NOT yet reproduced as live findings. The plan's acceptance test is therefore still only partly met. Next gated step. - Tier 3 (VALIDITY) -- not built. `--all` names it unbuilt with an explicit `[ok]`. - `--render`'s source-field derivation from the locations list -- not wired. Two fields are explicitly skipped as pending, not silently matched. - The preflight `Pn` of ruling 1 -- **not wired**, and this is an OPEN QUESTION for the operator, not an omission: wiring it blocking (the default, since it is what was ruled) hard-fails `preflight.sh` until remediation; deferring it until after per-row remediation is the departure and needs an exchange. - SEC-009's demotion to a pointer (ruling 4) -- not done; it belongs with tier 2. - No manifest was rewritten; no cloud state was touched. --- ## Item 9 -- LIVE tier-2 sweep run (read-only) + two probe false-greens it exposed Capture: `docs/audit/d137-tier2-sweep-20260726.txt`. Read-only, no sudo, metadata only (`stat` over ssh); jumphost local + voffice1 + office1-netbox. **ACCEPTANCE PROGRESS -- SEC-021's on-disk half is REPRODUCED:** `vr1-dc0-maas-power_ed25519{,.pub}` are genuinely ABSENT from the dc0 jumphost creds folder. Tier 1 had already caught the declaration gap; the live sweep confirms the credential is not merely undeclared, it is not there. **Two false-greens in my own probe, both found only by running it against real hosts:** 1. **"Absent" vs "could not look."** `/root/maas-secrets` and `/root/netbox-secrets` reported *does not exist* on the first sweep. They exist -- SEC-020 is precisely about `admin.pass` living in the first one. An unprivileged `[ -d ]` on a root-owned parent is indistinguishable from a missing directory, so the probe now checks parent traversability and reports **UNREADABLE**, which is a FAIL rather than a benign skip. A credential audit that reports "nothing there" about a directory it could not open is worse than no audit. 2. **Role-level aggregation manufactured false absences.** Skipping ONE location for a role still let that role's other locations "cover" it, so every credential living in the unprobed location reported EXPECTED-BUT-ABSENT. Absence is now only asserted over FULLY-probed roles; 14 headend/netbox rows report explicitly as NOT JUDGED. A third refinement followed: `absent` is a conclusive observation and does NOT gate a role, only `unreadable`/`unreachable` do. Harness grew to 35 (T33 unreadable-location, T34 absent-location). **STILL OUTSTANDING -- needs an operator-gated privileged read.** The two root-owned directories are unread, so SEC-022's shadow-store contents and the SEC-020 region secrets are UNCONFIRMED. SEC-023's sprawl half is also unverified live. The remaining command is a remote-sudo shape and was NOT run.