# D-137 implementation plan -- credential matrix build spec

**Status of this document.** Build spec for D-137, ADOPTED 2026-07-26 with all five
sub-rulings recorded. `docs/design-decisions.md` D-137 is the ruling authority; this file
is the executable detail. Written 2026-07-26 so the build survives a fresh session --
the design originated in a plan file outside the repo, which the GA-R4 bookend path would
not surface. Precedent: `docs/D-068-vault-migration-plan-draft.md`.

**NOTHING IN HERE IS BUILT YET.** No matrix file, no checker, no `--render`, no preflight
`Pn`, no SEC-009 demotion.

## Why (the short version)

`creds-audit` is DECLARATION-based: the manifest IS the declaration, so an undeclared
secret is structurally invisible. It reported CLEAN on 2026-07-15 while four region-VM
secrets minted 2026-07-13 sat undeclared; `admin.pass` surfaced 12 days later only
because someone looked (SEC-020). The operator's insight closed the other half: **a
discovery sweep can never detect a credential that was never minted at all** -- absence is
invisible to discovery. Hence a forward EXPECTED-state register (the matrix), reconciled
against measured state.

## The five ruled decisions (authority: D-137 Status block)

1. **Enforcement = blocking in preflight.** The check runs as a new `Pn` in
   `scripts/preflight.sh` and HARD-FAILS (exit 1) on expected-but-absent, undeclared, or
   per-DC-asymmetric credentials. No PreToolUse guard. Advisory was rejected as the
   posture that demonstrably failed.
2. **Manifests are DERIVED from the matrix.** Matrix is single source; `--render`
   regenerates `creds-manifests/*.manifest`; the gauntlet FAILS if checked-in differs from
   rendered. ACCEPTED COST: manifests become generated, so their governance prose (SEC
   linkage, mint dates, off-manifest carve-outs, un-consolidated findings) must become
   MATRIX FIELDS or be regenerated into the rendered header -- it must not be dropped. That
   prose is load-bearing; `vr1-dc1.manifest:18-20` is what recorded the dc0 backfill debt.
3. **`--remote` is bounded to declared locations.** New `creds-manifests/vm-secret-locations`
   (`<host-role> <path>`); never walks outside it. No tenant surface, D-069 preserved.
   The list MUST include (faithful-implementation note in the ruling): per-site creds
   folders on the jumphost AND the headend (SEC-022 shadow stores), the region's
   `/root/maas-secrets/` (SEC-020), and the three dirs outside the SEC-009 convention
   (the Vault init dir, the Octavia PKI dir, the per-tenant dirs -- capture FINDING 2),
   plus `overlays/octavia-pki.yaml` in the clone. Adding a row is DoD for any new mint site.
4. **D-137 is the policy authority.** SEC-009's convention block demotes to a POINTER;
   the ledger returns to being purely an exposure/rotation register. The demotion MOVES
   the rule text -- SEC-009's founding history (the NetBox-token miss) stays as history.
5. **Identity folded in.** Invariant: **one identity serves one principal type.** The
   `principal` column enforces it -- a credential serving both `human` and `service` is a
   FAIL. Absorbs the SEC-020 conflation question; no separate D-number.

## Matrix schema (logical keys ONLY -- SEC-004, repo is currently PUBLIC)

`creds-matrix.tsv`, positional records in the manifests' idiom (do NOT invent a format
language; `scripts/site-ssh-config.sh:44-50` uses the same shape):

    id | cardinality | site-key | host-role | filename | access-type | principal | custody | mint-stage | mint-ref | sec-ref | notes-ref

Whitespace-separated positional records -- the same shape `creds-audit.sh:49` already
reads, which is what "the manifests' idiom" means; the file is column-aligned for human
review and NO FIELD MAY CONTAIN WHITESPACE. `#` comments and blank lines ignored; `-`
means not-applicable. The doc renders the columns pipe-delimited for readability only.

- `cardinality`: `singleton` | `per-site` | `per-DC` | `per-node` | `per-tenant`.
  REQUIRED for Roosevelt transfer -- without it the matrix cannot express "9 nodes".
- `site-key`: region-qualified ONLY (`vr1-dc0`), bare `dc0` REJECTED -- import the
  discipline at `scripts/lib-net.sh:160-162` / `lib-hosts.sh:153-155`.
- `host-role`: role token (`jumphost`, `headend`, `rack`, `edge`) -- NOT `voffice1` or an
  IP. This still expresses "the copy is on the wrong role" without publishing topology.
- `access-type`: `gui` | `api` | `ssh` | `cli-profile` | `console` | `none`.
- `principal`: `human` | `service`. Ruling 5's invariant keys off this.
- `mint-ref`: MUST admit THREE provenance kinds -- `script:line`, `runbook:step`, and
  `operator-terminal`. Capture FINDING 1: `ssh-keygen` returns ZERO hits repo-wide, so 12
  declared secrets have NO mint command; those rows are the DEBT the matrix exists to
  surface and must be admitted, not treated as parse errors.
- NO grants, hostnames, IPs, or values (ruling: logical keys only).

### AMENDMENT 2026-07-26 -- 10 columns -> 12; rows keyed by (credential, location)

Operator-approved 2026-07-26 ("go with the 12-column amendment") after a read-first
round-trip check of the three manifests against the 10-column schema found it could not
carry what ruling 2 forbids dropping. This is ruling 2 being IMPLEMENTED, not departed
from -- its own words are that the prose "must become MATRIX FIELDS". OPS under GA-R3
(mechanism detail of an adopted decision, no Roosevelt-delta of its own); D-137's five
sub-rulings are untouched and remain the authority.

**A row is one (credential, location) pair, not one credential.** A credential with
copies in several places gets one row per place, sharing an `id`. This was already
`host-role`'s stated intent ("expresses that the copy is on the wrong role") and it is
what makes the SEC-020 stale-trap machine-checkable: tier 3 cannot compare digests across
hosts without knowing which locations hold the same `id` and which one is authoritative.

**`custody`** (NEW) -- `consolidated` | `source-of-record` | `off-manifest-known` |
`not-consolidated-ruled`. A four-token logical enum, NOT the "custody detail" the original
bullet excludes: no paths, no hostnames, no holders. It exists because ruling 3 puts
`/root/maas-secrets/` and the headend shadow stores INSIDE `--remote`'s declared
locations, so `creds-audit.sh:63-67`'s undeclared-file check will visit four credential
copies that are deliberate, reasoned decisions and report every one as UNDECLARED:
  - the Juju client credential store copies of both DCs' `maas-api-key.txt`
    (`vr1-dc0.manifest:27-28`, `vr1-dc1.manifest:34-35`; also SEC-018);
  - the MAAS snap copy of the dc1 power key (`vr1-dc1.manifest:27-28`; also SEC-016);
  - `voffice1:/root/maas-secrets/{admin.apikey,db.pass,lxd-trust.pass}`, ruled
    not-consolidated with per-file reasons (`vr1-office1.manifest:24-27`; SEC-020(i)).
Without the column the checker either fails forever on settled questions or the decisions
get deleted to silence it. `source-of-record` additionally carries SEC-020's ruling that
the REGION copy of the `admin` password is authoritative and the jumphost copy is the
working copy -- the fact the stale-trap warning depends on.

**`notes-ref`** (NEW) -- an `id`-keyed pointer into `creds-matrix-notes.md`, or `-`.
Free text lives THERE, never in the TSV, so the TSV stays logical-keys-only and greppable.
`--render` emits the referenced note as the manifest comment above its row, which is how
ruling 2's "regenerated into the rendered header" obligation is met for per-row prose.
Load-bearing content this preserves: mint dates; the not-reproducible debt on the 12
`operator-terminal` rows; format specifications used for verify-by-format (e.g. the
`admin` password's 28 chars, alphanumeric, NO trailing newline); and the reason text on
every `not-consolidated-ruled` / `off-manifest-known` row.

**Knowledge-loss qualifier (measured, do not overstate the gap).** Most of this prose is
ALREADY duplicated in `docs/security-ledger.md` -- SEC-018 carries the Juju-store copy,
SEC-020 carries source-of-record, the stale-trap warning, and all three carve-out reasons.
The real defect the two columns fix is that the CHECKER had no machine-readable field for
facts that are otherwise well recorded, not that the facts were about to be lost.

## Checker: `scripts/creds-matrix.py`

Model on `scripts/provider-bundle-check.py` -- the repo's canonical declarative gate:
expectations as top-of-file constants keyed by a region-qualified slug, `--dc` selector
with a safe default, `oks`/`fails` accumulators, `[ok]`/`[FAIL]` lines, ONE verdict line,
`return 1 if fails`, and checks that SELF-SKIP with an explicit ok when a precondition is
absent (`:281`) rather than silently passing -- needed for sites not yet deployed.

Import the BOTH-BOUNDS discipline from `netbox/sandbox-fidelity-check.py:131-143`, which
records the identical false-green in-repo: *"EXPECTED_NEW was an upper bound masquerading
as an assertion. It must be BOTH bounds."* A credential matrix has the same failure mode
one level up: enumerate no sites -> nothing undeclared -> false CLEAN. Reuse its four-way
drift vocabulary -- MISSING / UNEXPECTED-EXTRA / EXPECTED-BUT-ABSENT / FIELD DRIFT.
Provenance drift is a FIELD mismatch, not a presence mismatch.

### Three tiers, increasing cost

1. **STATIC** (offline, gauntlet): matrix parses; every row has a manifest row; rendered
   manifests == checked-in (ruling 2); every `script:line`/`runbook:step` `mint-ref`
   resolves to a real location; per-DC rows are SYMMETRIC; principal invariant (ruling 5).
2. **EXISTENCE** (local + remote, preflight `Pn`): every expected credential present at
   the expected host-role with expected mode; nothing undeclared at any declared location
   INCLUDING remote ones (ruling 3).
3. **VALIDITY** (stage close, `--probe`): provenance digests match across hosts; per-row
   behavioral probes where defined. Motivated by: `creds-audit` PARSES provenance and
   NEVER verifies it (`fsrc` used only in the MISSING message), so the SEC-020 stale-trap
   warning is unchecked today.

## Hard constraints inherited

- **Metadata only, enforced by source-grep.** `tests/creds-audit/run-tests.sh:90-92` fails
  on any `cat|head|tail|less|more|od|xxd|hexdump` in the audit source. Remote provenance
  verification MUST compare `sha256sum` digests over ssh -- never transfer content. A new
  checker should carry the same guard test.
- **Gauntlet registration is pure glob** over `tests/*/run-tests.sh`
  (`scripts/run-tests-all.sh:22`); only `tail -1` is shown, so the LAST line must
  self-describe. Failure-detail excerpting matches `^\s*(FAIL|\[XX\]|MISS|LEAK|COUNT)` --
  it does NOT match `[FAIL]`, which is why creds-audit failures currently surface with no
  detail. Lead detail lines with bare `FAIL`.
- **preflight** aggregation is `note $?` worst-exit (1 dominates 2 dominates 0); add one
  banner + one call honouring the 0/1/2 contract.
- **repo-lint L1**: new `design-decisions.md` text must be pure ASCII. **L10**: any commit
  touching `docs/audit/`, a `**Status:` line, or a gate row must touch CURRENT-STATE in the
  SAME commit.
- Harness style: mutation-of-a-real-artifact fixtures + `rc` AND output-regex assertions,
  one case per invariant, each named for the real defect it encodes
  (`tests/provider-bundle-check/run-tests.sh`).

## Files

NEW: `creds-matrix.tsv`, `creds-matrix-notes.md` (the 2026-07-26 amendment's `notes-ref`
target), `creds-manifests/vm-secret-locations`, `scripts/creds-matrix.py`,
`tests/creds-matrix/run-tests.sh`.
MODIFY: `scripts/preflight.sh` (new `Pn`), `scripts/creds-audit.sh` (`--all`, `--remote`,
VERIFY provenance, widen sprawl globs -- currently miss `admin.pass`, `*.apikey`, `*.key`,
`*.pem`, `*_ed25519`, `admin-openrc`), `creds-manifests/*.manifest` (become generated),
`docs/security-ledger.md` (SEC-009 demotion per ruling 4), `docs/CURRENT-STATE.md` (C1).

## Verification / acceptance

- `bash tests/creds-matrix/run-tests.sh` -- every invariant FAILS when violated.
- **ACCEPTANCE TEST:** `scripts/creds-matrix.py --all` must reproduce SEC-021/-022/-023 and
  the openrc case as NAMED failures on today's tree. A checker that goes green on the
  current tree is wrong.
  - **Tier boundary (clarified 2026-07-26; corrected after the matrix was seeded).**
    SEC-022 and SEC-023 are pure EXISTENCE failures needing tier 2's remote reads across
    three sites. **SEC-021 SPLITS across the tiers:** its DECLARATION asymmetry -- dc1
    declares an edge API credential and a jumphost-consolidated power key, dc0 declares
    neither -- is caught at TIER 1 by the per-DC symmetry and manifest-coverage checks; only
    its on-disk absence needs tier 2. So tier 1 + its harness partially satisfies the
    acceptance test and must be reported that way -- neither as fully satisfying it nor as
    unable to touch it.
  - **The `admin-openrc` case resolves to BOTH readings, which are complementary.**
    `~/admin-openrc` is PREDICTED, not present: `phase-03-admin-openrc.sh` runs at Stage 5,
    not yet reached. So it is (a) a matrix row whose `mint-stage` is in the future, which
    the checker reports as not-yet-expected via the explicit-skip-with-`[ok]` pattern
    (`provider-bundle-check.py:281`) rather than a silent pass, AND (b) a harness fixture
    proving the widened sprawl glob CATCHES it when it does appear. Build both.
- `bash scripts/run-tests-all.sh` ALL GREEN; `bash scripts/repo-lint.sh` 0-fail;
  `bash scripts/preflight.sh` exercises the new `Pn`.

## EXPECTED RED BY DESIGN -- do not "fix" by weakening the check

The first full run SHOULD fail, and these are the defects, not bugs in the checker:
- `admin` serves a `human` GUI row AND a `service` api row -> ruling-5 invariant FAIL.
  Remediation is the `operator`/`admin` split already begun 2026-07-25, i.e. migrate the
  19 `maas admin` call sites to a service identity.
- 27 rows carry `mint-ref=operator-terminal` (the plan's original figure of 12 counted
  only FINDING 1's declared secrets; the delivered matrix admits every undocumented mint) -> reproducibility debt to convert.
- SEC-021 (dc0 `opnsense-api.txt` ABSENT; power-key name/host divergence), SEC-022
  (headend shadow stores), SEC-023 (sprawl blind spots + predicted `~/admin-openrc`).

Going green is a REMEDIATION project separate from the build. Do not scope-creep it into
the build; land the checker red, record the failures, then gate remediation per row.

## Fresh-session start

1. Normal bootstrap: `git pull`, `bash scripts/repo-lint.sh`, read
   `docs/CURRENT-STATE.md` in full, then `docs/session-ledger.md` +
   `bash scripts/ledger-scan.sh`.
2. Read `docs/design-decisions.md` D-137 (the ruling authority -- all five sub-rulings and
   their verbatim operator selections), then this file.
3. Read the research capture `docs/audit/creds-creation-points-20260725.md` -- it is the
   matrix SEED (55 MINT sites classified by host, destination, stage, human/service).
   **55 mint SITES is not 55 rows** (clarified 2026-07-26): A1/A10 are generator helpers,
   not credentials; A21-A25 are flagged in the capture itself as the SAME set as A11-A14
   ("do not double-count"); B30 is explicitly not a mint; A8+A9 are one credential minted
   then retrieved; and `per-tenant`/`per-node`/`per-cluster` sets collapse to ONE row under
   `cardinality`. Conversely the materialization ruling ADDS rows the inventory excluded as
   MATERIALIZE (the Keystone `admin` password, `~/admin-openrc`), and the 12 no-mint
   declared secrets of FINDING 1 are rows with `mint-ref=operator-terminal`. Derive the row
   set from credential IDENTITY; never transcribe the table 1:1.
4. Build tier 1 first (offline, no live access needed), harness it, then tier 2, then tier 3.