# Session body 2026-07-26 -- D-137 credential matrix: build, committee audit, remediation Full session record (GA-R4 rule 2). The bounded summary is in `docs/session-ledger.md`; status is in `docs/CURRENT-STATE.md` ONLY (GA-R1). Session changelog with per-item reverts: `docs/changelog-20260726-d137-tier1.md` (items 1-22 -- the filename says "tier1" and the content outgrew it; noted rather than renamed, since the name is cited in commits). Branch `dc-dc-stage4-phase3-maas-deploy`, 17 commits `af1b682..1696d08`, all pushed. Opened at `8e72ff1`. Stage 4 remains OPEN; no stage was closed by this session. ## What was built D-137 was ADOPTED but unimplemented at session open. All three tiers now exist. - **Tier 1 STATIC** -- `creds-matrix.tsv` (81 rows), `creds-matrix-notes.md`, `scripts/creds-matrix.py` (S1 schema / S2 manifest coverage both-bounds / S3 render drift / S4 mint-ref resolution / S5 per-DC symmetry / S6 ruling-5 principal invariant / S7 notes integrity), harness `tests/creds-matrix`. - **Tier 2 EXISTENCE** -- `creds-manifests/vm-secret-locations` (ruling 3's absolute bound), E0-E4 checks, `--remote`/`--privileged` with escalation retried only where needed. - **Tier 3 VALIDITY** -- V1 cross-copy sha256 provenance (external `sha256sum` both sides so only digests enter the process), V2 declared-state reconciliation, and `scripts/creds-probe.sh` for the behavioural half. - **Enforcement** -- preflight **P5** runs tier 1 + tier 2 LOCAL, blocking, fail-closed. `creds-manifests/stages-reached` replaces `--pending-stage`, coupled to CURRENT-STATE by new **repo-lint L12**. - **Schema** -- 10 -> 12 columns (`custody`, `notes-ref`), rows re-keyed to (credential, location); `custody` later gained `verbatim-copy` for tier 3. Harnesses: creds-matrix 56, creds-probe 9, creds-audit 15, preflight 10, repo-lint 47. Gauntlet 79 -> **81 ALL GREEN**. repo-lint 0-fail throughout. ## Operator rulings (GA-R5 form; utterances quoted in design-decisions.md D-137) 1. "go with the 12-column amendment" -- schema amendment. 2. "wire tier 1 as the blocking preflight Pn" -- ruling 1 enforcement, tier 1. 3. Tier 2 gate shape: "Local-only blocking; remote operator-run"; stage input: "Declared file + lint coupling". 4. Tier 3 boundary: "Full -- including OpenStack service credentials"; cadence: "Stage close + DC standup close". (I recommended passive-only and flagged the costs; ruled otherwise.) 5. SEC-024: "chmod 600 both, now"; snapshots "Keep, decide at stage close". Ruling 4 (SEC-009 demotion) needed no exchange -- already ruled, trigger passed. ## The committee, and what it found Six independent read-only lenses (correctness, coverage, claim-verification, ruling-fidelity, record-integrity, Roosevelt-transfer). **Four converged independently on the same root cause.** Every defect was reproduced, not theorised. **Eight confirmed false greens in the delivered build**, all since fixed and individually regression-locked. The most consequential: tier 2 keyed existence on host-role ALONE and never consulted `site-key`, so a credential present at one site satisfied the row for every site -- which had masked SEC-021's dc0 `opnsense-api.txt` on-disk absence while tier 1 flagged it. Also: `probe_remote` reporting UNREACHABLE for a location it had read; literal paths skipping absent/unreadable detection; an empty locations file bypassing ruling 3; no non-empty floor; a mint-ref crash exiting 1 indistinguishably from findings; mode enforcement custody-gated (43 of 77 rows unchecked); and S5's second direction untested. **~21 inferred filenames, not the six I had owned.** Only the rows the first sweep physically touched had been re-measured. Octavia's 8 were invented AND its declared location matched none of them (the artifacts live in three subdirectories); vault used a directory name as a filename; the tenant rows lacked the `-` prefix; `.maas.cli` was never measured at all and does not exist (the MAAS snap CLI profile is `.maascli.db`). **`sec-ref` mis-attribution was systemic.** One instance was fixed by hand earlier in the session without asking whether the class recurred. It did. ## Defects found in my own work by running it Recorded because the pattern is the lesson: several defects were invisible to review and appeared only on contact with real hosts. - The first privileged sweep reported `/root/maas-secrets` as "does not exist". It exists -- an unprivileged `[ -d ]` on a root-owned parent is indistinguishable from absent. - Role-level gating manufactured false absences when one location was unprobed. - `shlex.quote` on the probe directory broke `~` expansion, so every home-relative remote location read as unreadable. - **Tier 3's first draft reported provenance DRIFT on `maas-region-admin` and `dc1-svc-key`.** Manual `sha256sum` showed both IDENTICAL. Reporting it would have raised a false credential-rotation alarm on exactly the SEC-020 pair. Root cause was a schema gap: nothing expressed which rows are byte-copies. - **The first `creds-probe.sh --run` was a false green:** `ssh` ate the `while read` loop's stdin, 1 of 13 probes ran, exit 0. Now locked by a total-accounting check. ## Records corrected CURRENT-STATE had asserted remote filenames and modes citing a checker VERDICT file that contained none of them (GA-R1 rule 2); it said the `virsh` gap was unfixed after it was fixed; it carried contradictory counts. The "ZERO undeclared files / fully accounted for" claim was WITHDRAWN as resting on the site-blind check, and "SEC-021's on-disk half REPRODUCED" was narrowed to the power-key artifacts -- then genuinely earned after the fix, when `E1 dc0-edge-api 'opnsense-api.txt'` appeared for the first time. The 12-column amendment was recorded on D-137, having existed only in a non-citable changelog. ## Security ledger - **SEC-024 OPENED then REMEDIATED (mode).** `terraform.tfstate.backup` at 0664 carrying the MAAS API key in plaintext, found by the fixed checker on its first clean run. Severity was CORRECTED before remediation: `opentofu/` is 0700 and `~` is 0750, so it was never effectively readable -- defence-in-depth, not live exposure. Retention of the two `pre-*` snapshots deferred to stage close. - **V2 surfaced a standing defect:** SEC-006 records the NetBox upstream token as BURNED since 2026-07-12 while the matrix carried it as a healthy consolidated row. Nothing had reconciled those surfaces. - Open SEC rows 19 -> 20. ## Housekeeping Permission rules: `Bash(ssh * sudo *)` never matched quoted remote commands, so remote sudo fell through to the classifier; quoted variants added for `sudo` and `virsh`, all `ask`. Session ledger rotated oldest-first 413 -> 280 (GA-R4 F1), discharging a debt the 07-25 close recorded as OWED. ## Advice given, not built Secrets-storage recommendation delivered on request (no HSM/KMS available): resolve the Vault-vs-OpenBao licensing fork first; build `creds-mint.sh` as an atomic mint->store->register; enable MAAS 3.7's Vault integration (measured `disabled`) rehearsed in VR1 because of the MAAS/Vault circular dependency on bare metal; Vault SSH CA to retire the static keypairs; Tang/Clevis + transit auto-unseal for the no-HSM bootstrap. Recommended DROPPING the sops/age stopgap as churn. None of this is built or ruled. ## State at close Stage 4 OPEN. **The live mission did not move**: the DC1 Stage-5 chain is exactly where it was at session open -- verify MAAS role tags live, the 10th controller VM, the dc0 VIP extract, the vault-VIP repair, BLOCKER-1/2 and the DC-aware gate fixes, then `juju bootstrap` with edge egress OPEN. This session hardened the credential control that will protect Stage 5; it did not advance Stage 5. Preflight exits 1, with P5 among the reasons -- but it was ALREADY exiting 1 before P5 existed (P4: `overlays/octavia-pki.yaml` absent, MAAS unreachable from the jumphost). No deploy path that was open is closed by this session's work.