# Session body 2026-07-26 -- D-137 credential matrix: build, committee audit, remediation

Full session record (GA-R4 rule 2). The bounded summary is in `docs/session-ledger.md`;
status is in `docs/CURRENT-STATE.md` ONLY (GA-R1). Session changelog with per-item reverts:
`docs/changelog-20260726-d137-tier1.md` (items 1-22 -- the filename says "tier1" and the
content outgrew it; noted rather than renamed, since the name is cited in commits).

Branch `dc-dc-stage4-phase3-maas-deploy`, 17 commits `af1b682..1696d08`, all pushed.
Opened at `8e72ff1`. Stage 4 remains OPEN; no stage was closed by this session.

## What was built

D-137 was ADOPTED but unimplemented at session open. All three tiers now exist.

- **Tier 1 STATIC** -- `creds-matrix.tsv` (81 rows), `creds-matrix-notes.md`,
  `scripts/creds-matrix.py` (S1 schema / S2 manifest coverage both-bounds / S3 render drift
  / S4 mint-ref resolution / S5 per-DC symmetry / S6 ruling-5 principal invariant / S7 notes
  integrity), harness `tests/creds-matrix`.
- **Tier 2 EXISTENCE** -- `creds-manifests/vm-secret-locations` (ruling 3's absolute bound),
  E0-E4 checks, `--remote`/`--privileged` with escalation retried only where needed.
- **Tier 3 VALIDITY** -- V1 cross-copy sha256 provenance (external `sha256sum` both sides so
  only digests enter the process), V2 declared-state reconciliation, and
  `scripts/creds-probe.sh` for the behavioural half.
- **Enforcement** -- preflight **P5** runs tier 1 + tier 2 LOCAL, blocking, fail-closed.
  `creds-manifests/stages-reached` replaces `--pending-stage`, coupled to CURRENT-STATE by
  new **repo-lint L12**.
- **Schema** -- 10 -> 12 columns (`custody`, `notes-ref`), rows re-keyed to
  (credential, location); `custody` later gained `verbatim-copy` for tier 3.

Harnesses: creds-matrix 56, creds-probe 9, creds-audit 15, preflight 10, repo-lint 47.
Gauntlet 79 -> **81 ALL GREEN**. repo-lint 0-fail throughout.

## Operator rulings (GA-R5 form; utterances quoted in design-decisions.md D-137)

1. "go with the 12-column amendment" -- schema amendment.
2. "wire tier 1 as the blocking preflight Pn" -- ruling 1 enforcement, tier 1.
3. Tier 2 gate shape: "Local-only blocking; remote operator-run"; stage input: "Declared
   file + lint coupling".
4. Tier 3 boundary: "Full -- including OpenStack service credentials"; cadence: "Stage close
   + DC standup close". (I recommended passive-only and flagged the costs; ruled otherwise.)
5. SEC-024: "chmod 600 both, now"; snapshots "Keep, decide at stage close".
Ruling 4 (SEC-009 demotion) needed no exchange -- already ruled, trigger passed.

## The committee, and what it found

Six independent read-only lenses (correctness, coverage, claim-verification,
ruling-fidelity, record-integrity, Roosevelt-transfer). **Four converged independently on
the same root cause.** Every defect was reproduced, not theorised.

**Eight confirmed false greens in the delivered build**, all since fixed and individually
regression-locked. The most consequential: tier 2 keyed existence on host-role ALONE and
never consulted `site-key`, so a credential present at one site satisfied the row for every
site -- which had masked SEC-021's dc0 `opnsense-api.txt` on-disk absence while tier 1
flagged it. Also: `probe_remote` reporting UNREACHABLE for a location it had read; literal
paths skipping absent/unreadable detection; an empty locations file bypassing ruling 3;
no non-empty floor; a mint-ref crash exiting 1 indistinguishably from findings; mode
enforcement custody-gated (43 of 77 rows unchecked); and S5's second direction untested.

**~21 inferred filenames, not the six I had owned.** Only the rows the first sweep
physically touched had been re-measured. Octavia's 8 were invented AND its declared location
matched none of them (the artifacts live in three subdirectories); vault used a directory
name as a filename; the tenant rows lacked the `<client>-` prefix; `.maas.cli` was never
measured at all and does not exist (the MAAS snap CLI profile is `.maascli.db`).

**`sec-ref` mis-attribution was systemic.** One instance was fixed by hand earlier in the
session without asking whether the class recurred. It did.

## Defects found in my own work by running it

Recorded because the pattern is the lesson: several defects were invisible to review and
appeared only on contact with real hosts.

- The first privileged sweep reported `/root/maas-secrets` as "does not exist". It exists --
  an unprivileged `[ -d ]` on a root-owned parent is indistinguishable from absent.
- Role-level gating manufactured false absences when one location was unprobed.
- `shlex.quote` on the probe directory broke `~` expansion, so every home-relative remote
  location read as unreadable.
- **Tier 3's first draft reported provenance DRIFT on `maas-region-admin` and
  `dc1-svc-key`.** Manual `sha256sum` showed both IDENTICAL. Reporting it would have raised
  a false credential-rotation alarm on exactly the SEC-020 pair. Root cause was a schema
  gap: nothing expressed which rows are byte-copies.
- **The first `creds-probe.sh --run` was a false green:** `ssh` ate the `while read` loop's
  stdin, 1 of 13 probes ran, exit 0. Now locked by a total-accounting check.

## Records corrected

CURRENT-STATE had asserted remote filenames and modes citing a checker VERDICT file that
contained none of them (GA-R1 rule 2); it said the `virsh` gap was unfixed after it was
fixed; it carried contradictory counts. The "ZERO undeclared files / fully accounted for"
claim was WITHDRAWN as resting on the site-blind check, and "SEC-021's on-disk half
REPRODUCED" was narrowed to the power-key artifacts -- then genuinely earned after the fix,
when `E1 dc0-edge-api 'opnsense-api.txt'` appeared for the first time. The 12-column
amendment was recorded on D-137, having existed only in a non-citable changelog.

## Security ledger

- **SEC-024 OPENED then REMEDIATED (mode).** `terraform.tfstate.backup` at 0664 carrying the
  MAAS API key in plaintext, found by the fixed checker on its first clean run. Severity was
  CORRECTED before remediation: `opentofu/` is 0700 and `~` is 0750, so it was never
  effectively readable -- defence-in-depth, not live exposure. Retention of the two `pre-*`
  snapshots deferred to stage close.
- **V2 surfaced a standing defect:** SEC-006 records the NetBox upstream token as BURNED
  since 2026-07-12 while the matrix carried it as a healthy consolidated row. Nothing had
  reconciled those surfaces.
- Open SEC rows 19 -> 20.

## Housekeeping

Permission rules: `Bash(ssh * sudo *)` never matched quoted remote commands, so remote sudo
fell through to the classifier; quoted variants added for `sudo` and `virsh`, all `ask`.
Session ledger rotated oldest-first 413 -> 280 (GA-R4 F1), discharging a debt the 07-25
close recorded as OWED.

## Advice given, not built

Secrets-storage recommendation delivered on request (no HSM/KMS available): resolve the
Vault-vs-OpenBao licensing fork first; build `creds-mint.sh` as an atomic
mint->store->register; enable MAAS 3.7's Vault integration (measured `disabled`) rehearsed
in VR1 because of the MAAS/Vault circular dependency on bare metal; Vault SSH CA to retire
the static keypairs; Tang/Clevis + transit auto-unseal for the no-HSM bootstrap. Recommended
DROPPING the sops/age stopgap as churn. None of this is built or ruled.

## State at close

Stage 4 OPEN. **The live mission did not move**: the DC1 Stage-5 chain is exactly where it
was at session open -- verify MAAS role tags live, the 10th controller VM, the dc0 VIP
extract, the vault-VIP repair, BLOCKER-1/2 and the DC-aware gate fixes, then `juju
bootstrap` with edge egress OPEN. This session hardened the credential control that will
protect Stage 5; it did not advance Stage 5.

Preflight exits 1, with P5 among the reasons -- but it was ALREADY exiting 1 before P5
existed (P4: `overlays/octavia-pki.yaml` absent, MAAS unreachable from the jumphost).
No deploy path that was open is closed by this session's work.
