D-137 implementation plan -- credential matrix build spec

Status of this document. Build spec for D-137, ADOPTED 2026-07-26 with all five sub-rulings recorded. docs/design-decisions.md D-137 is the ruling authority; this file is the executable detail. Written 2026-07-26 so the build survives a fresh session -- the design originated in a plan file outside the repo, which the GA-R4 bookend path would not surface. Precedent: docs/D-068-vault-migration-plan-draft.md.

NOTHING IN HERE IS BUILT YET. No matrix file, no checker, no --render, no preflight Pn, no SEC-009 demotion.

Why (the short version)

creds-audit is DECLARATION-based: the manifest IS the declaration, so an undeclared secret is structurally invisible. It reported CLEAN on 2026-07-15 while four region-VM secrets minted 2026-07-13 sat undeclared; admin.pass surfaced 12 days later only because someone looked (SEC-020). The operator's insight closed the other half: a discovery sweep can never detect a credential that was never minted at all -- absence is invisible to discovery. Hence a forward EXPECTED-state register (the matrix), reconciled against measured state.

The five ruled decisions (authority: D-137 Status block)

1. Enforcement = blocking in preflight. The check runs as a new Pn in scripts/preflight.sh and HARD-FAILS (exit 1) on expected-but-absent, undeclared, or per-DC-asymmetric credentials. No PreToolUse guard. Advisory was rejected as the posture that demonstrably failed.
2. Manifests are DERIVED from the matrix. Matrix is single source; --render regenerates creds-manifests/*.manifest; the gauntlet FAILS if checked-in differs from rendered. ACCEPTED COST: manifests become generated, so their governance prose (SEC linkage, mint dates, off-manifest carve-outs, un-consolidated findings) must become MATRIX FIELDS or be regenerated into the rendered header -- it must not be dropped. That prose is load-bearing; vr1-dc1.manifest:18-20 is what recorded the dc0 backfill debt.
3. --remote is bounded to declared locations. New creds-manifests/vm-secret-locations (<host-role> <path>); never walks outside it. No tenant surface, D-069 preserved. The list MUST include (faithful-implementation note in the ruling): per-site creds folders on the jumphost AND the headend (SEC-022 shadow stores), the region's /root/maas-secrets/ (SEC-020), and the three dirs outside the SEC-009 convention (the Vault init dir, the Octavia PKI dir, the per-tenant dirs -- capture FINDING 2), plus overlays/octavia-pki.yaml in the clone. Adding a row is DoD for any new mint site.
4. D-137 is the policy authority. SEC-009's convention block demotes to a POINTER; the ledger returns to being purely an exposure/rotation register. The demotion MOVES the rule text -- SEC-009's founding history (the NetBox-token miss) stays as history.
5. Identity folded in. Invariant: one identity serves one principal type. The principal column enforces it -- a credential serving both human and service is a FAIL. Absorbs the SEC-020 conflation question; no separate D-number.

Matrix schema (logical keys ONLY -- SEC-004, repo is currently PUBLIC)

creds-matrix.tsv, positional records in the manifests' idiom (do NOT invent a format language; scripts/site-ssh-config.sh:44-50 uses the same shape):

id | cardinality | site-key | host-role | filename | access-type | principal | custody | mint-stage | mint-ref | sec-ref | notes-ref

Whitespace-separated positional records -- the same shape creds-audit.sh:49 already reads, which is what "the manifests' idiom" means; the file is column-aligned for human review and NO FIELD MAY CONTAIN WHITESPACE. # comments and blank lines ignored; - means not-applicable. The doc renders the columns pipe-delimited for readability only.

• cardinality: singleton | per-site | per-DC | per-node | per-tenant. REQUIRED for Roosevelt transfer -- without it the matrix cannot express "9 nodes".
• site-key: region-qualified ONLY (vr1-dc0), bare dc0 REJECTED -- import the discipline at scripts/lib-net.sh:160-162 / lib-hosts.sh:153-155.
• host-role: role token (jumphost, headend, rack, edge) -- NOT voffice1 or an IP. This still expresses "the copy is on the wrong role" without publishing topology.
• access-type: gui | api | ssh | cli-profile | console | none.
• principal: human | service. Ruling 5's invariant keys off this.
• mint-ref: MUST admit THREE provenance kinds -- script:line, runbook:step, and operator-terminal. Capture FINDING 1: ssh-keygen returns ZERO hits repo-wide, so 12 declared secrets have NO mint command; those rows are the DEBT the matrix exists to surface and must be admitted, not treated as parse errors.
• NO grants, hostnames, IPs, or values (ruling: logical keys only).

AMENDMENT 2026-07-26 -- 10 columns -> 12; rows keyed by (credential, location)

Operator-approved 2026-07-26 ("go with the 12-column amendment") after a read-first round-trip check of the three manifests against the 10-column schema found it could not carry what ruling 2 forbids dropping. This is ruling 2 being IMPLEMENTED, not departed from -- its own words are that the prose "must become MATRIX FIELDS". OPS under GA-R3 (mechanism detail of an adopted decision, no Roosevelt-delta of its own); D-137's five sub-rulings are untouched and remain the authority.

A row is one (credential, location) pair, not one credential. A credential with copies in several places gets one row per place, sharing an id. This was already host-role's stated intent ("expresses that the copy is on the wrong role") and it is what makes the SEC-020 stale-trap machine-checkable: tier 3 cannot compare digests across hosts without knowing which locations hold the same id and which one is authoritative.

custody (NEW) -- consolidated | source-of-record | off-manifest-known | not-consolidated-ruled. A four-token logical enum, NOT the "custody detail" the original bullet excludes: no paths, no hostnames, no holders. It exists because ruling 3 puts /root/maas-secrets/ and the headend shadow stores INSIDE --remote's declared locations, so creds-audit.sh:63-67's undeclared-file check will visit four credential copies that are deliberate, reasoned decisions and report every one as UNDECLARED:

• the Juju client credential store copies of both DCs' maas-api-key.txt (vr1-dc0.manifest:27-28, vr1-dc1.manifest:34-35; also SEC-018);
• the MAAS snap copy of the dc1 power key (vr1-dc1.manifest:27-28; also SEC-016);
• voffice1:/root/maas-secrets/{admin.apikey,db.pass,lxd-trust.pass}, ruled not-consolidated with per-file reasons (vr1-office1.manifest:24-27; SEC-020(i)). Without the column the checker either fails forever on settled questions or the decisions get deleted to silence it. source-of-record additionally carries SEC-020's ruling that the REGION copy of the admin password is authoritative and the jumphost copy is the working copy -- the fact the stale-trap warning depends on.

notes-ref (NEW) -- an id-keyed pointer into creds-matrix-notes.md, or -. Free text lives THERE, never in the TSV, so the TSV stays logical-keys-only and greppable. --render emits the referenced note as the manifest comment above its row, which is how ruling 2's "regenerated into the rendered header" obligation is met for per-row prose. Load-bearing content this preserves: mint dates; the not-reproducible debt on the 12 operator-terminal rows; format specifications used for verify-by-format (e.g. the admin password's 28 chars, alphanumeric, NO trailing newline); and the reason text on every not-consolidated-ruled / off-manifest-known row.

Knowledge-loss qualifier (measured, do not overstate the gap). Most of this prose is ALREADY duplicated in docs/security-ledger.md -- SEC-018 carries the Juju-store copy, SEC-020 carries source-of-record, the stale-trap warning, and all three carve-out reasons. The real defect the two columns fix is that the CHECKER had no machine-readable field for facts that are otherwise well recorded, not that the facts were about to be lost.

Checker: scripts/creds-matrix.py

Model on scripts/provider-bundle-check.py -- the repo's canonical declarative gate: expectations as top-of-file constants keyed by a region-qualified slug, --dc selector with a safe default, oks/fails accumulators, [ok]/[FAIL] lines, ONE verdict line, return 1 if fails, and checks that SELF-SKIP with an explicit ok when a precondition is absent (:281) rather than silently passing -- needed for sites not yet deployed.

Import the BOTH-BOUNDS discipline from netbox/sandbox-fidelity-check.py:131-143, which records the identical false-green in-repo: "EXPECTED_NEW was an upper bound masquerading as an assertion. It must be BOTH bounds." A credential matrix has the same failure mode one level up: enumerate no sites -> nothing undeclared -> false CLEAN. Reuse its four-way drift vocabulary -- MISSING / UNEXPECTED-EXTRA / EXPECTED-BUT-ABSENT / FIELD DRIFT. Provenance drift is a FIELD mismatch, not a presence mismatch.

Three tiers, increasing cost

1. STATIC (offline, gauntlet): matrix parses; every row has a manifest row; rendered manifests == checked-in (ruling 2); every script:line/runbook:step mint-ref resolves to a real location; per-DC rows are SYMMETRIC; principal invariant (ruling 5).
2. EXISTENCE (local + remote, preflight Pn): every expected credential present at the expected host-role with expected mode; nothing undeclared at any declared location INCLUDING remote ones (ruling 3).
3. VALIDITY (stage close, --probe): provenance digests match across hosts; per-row behavioral probes where defined. Motivated by: creds-audit PARSES provenance and NEVER verifies it (fsrc used only in the MISSING message), so the SEC-020 stale-trap warning is unchecked today.

Hard constraints inherited

• Metadata only, enforced by source-grep. tests/creds-audit/run-tests.sh:71-73 fails on any cat|head|tail|less|more|od|xxd|hexdump in the audit source. Remote provenance verification MUST compare sha256sum digests over ssh -- never transfer content. A new checker should carry the same guard test.
• Gauntlet registration is pure glob over tests/*/run-tests.sh (scripts/run-tests-all.sh:22); only tail -1 is shown, so the LAST line must self-describe. Failure-detail excerpting matches ^\s*(FAIL|\[XX\]|MISS|LEAK|COUNT) -- it does NOT match [FAIL], which is why creds-audit failures currently surface with no detail. Lead detail lines with bare FAIL.
• preflight aggregation is note $? worst-exit (1 dominates 2 dominates 0); add one banner + one call honouring the 0/1/2 contract.
• repo-lint L1: new design-decisions.md text must be pure ASCII. L10: any commit touching docs/audit/, a **Status: line, or a gate row must touch CURRENT-STATE in the SAME commit.
• Harness style: mutation-of-a-real-artifact fixtures + rc AND output-regex assertions, one case per invariant, each named for the real defect it encodes (tests/provider-bundle-check/run-tests.sh).

Files

NEW: creds-matrix.tsv, creds-matrix-notes.md (the 2026-07-26 amendment's notes-ref target), creds-manifests/vm-secret-locations, scripts/creds-matrix.py, tests/creds-matrix/run-tests.sh. MODIFY: scripts/preflight.sh (new Pn), scripts/creds-audit.sh (--all, --remote, VERIFY provenance, widen sprawl globs -- currently miss admin.pass, *.apikey, *.key, *.pem, *_ed25519, admin-openrc), creds-manifests/*.manifest (become generated), docs/security-ledger.md (SEC-009 demotion per ruling 4), docs/CURRENT-STATE.md (C1).

Verification / acceptance

• bash tests/creds-matrix/run-tests.sh -- every invariant FAILS when violated.
• ACCEPTANCE TEST: scripts/creds-matrix.py --all must reproduce SEC-021/-022/-023 and the openrc case as NAMED failures on today's tree. A checker that goes green on the current tree is wrong.
  • Tier boundary (clarified 2026-07-26; corrected after the matrix was seeded). SEC-022 and SEC-023 are pure EXISTENCE failures needing tier 2's remote reads across three sites. SEC-021 SPLITS across the tiers: its DECLARATION asymmetry -- dc1 declares an edge API credential and a jumphost-consolidated power key, dc0 declares neither -- is caught at TIER 1 by the per-DC symmetry and manifest-coverage checks; only its on-disk absence needs tier 2. So tier 1 + its harness partially satisfies the acceptance test and must be reported that way -- neither as fully satisfying it nor as unable to touch it.
  • The admin-openrc case resolves to BOTH readings, which are complementary. ~/admin-openrc is PREDICTED, not present: phase-03-admin-openrc.sh runs at Stage 5, not yet reached. So it is (a) a matrix row whose mint-stage is in the future, which the checker reports as not-yet-expected via the explicit-skip-with-[ok] pattern (provider-bundle-check.py:281) rather than a silent pass, AND (b) a harness fixture proving the widened sprawl glob CATCHES it when it does appear. Build both.
• bash scripts/run-tests-all.sh ALL GREEN; bash scripts/repo-lint.sh 0-fail; bash scripts/preflight.sh exercises the new Pn.

EXPECTED RED BY DESIGN -- do not "fix" by weakening the check

The first full run SHOULD fail, and these are the defects, not bugs in the checker:

• admin serves a human GUI row AND a service api row -> ruling-5 invariant FAIL. Remediation is the operator/admin split already begun 2026-07-25, i.e. migrate the 19 maas admin call sites to a service identity.
• 12 rows carry mint-ref=operator-terminal -> reproducibility debt to convert.
• SEC-021 (dc0 opnsense-api.txt ABSENT; power-key name/host divergence), SEC-022 (headend shadow stores), SEC-023 (sprawl blind spots + predicted ~/admin-openrc).

Going green is a REMEDIATION project separate from the build. Do not scope-creep it into the build; land the checker red, record the failures, then gate remediation per row.

Fresh-session start

1. Normal bootstrap: git pull, bash scripts/repo-lint.sh, read docs/CURRENT-STATE.md in full, then docs/session-ledger.md + bash scripts/ledger-scan.sh.
2. Read docs/design-decisions.md D-137 (the ruling authority -- all five sub-rulings and their verbatim operator selections), then this file.
3. Read the research capture docs/audit/creds-creation-points-20260725.md -- it is the matrix SEED (55 MINT sites classified by host, destination, stage, human/service). 55 mint SITES is not 55 rows (clarified 2026-07-26): A1/A10 are generator helpers, not credentials; A21-A25 are flagged in the capture itself as the SAME set as A11-A14 ("do not double-count"); B30 is explicitly not a mint; A8+A9 are one credential minted then retrieved; and per-tenant/per-node/per-cluster sets collapse to ONE row under cardinality. Conversely the materialization ruling ADDS rows the inventory excluded as MATERIALIZE (the Keystone admin password, ~/admin-openrc), and the 12 no-mint declared secrets of FINDING 1 are rows with mint-ref=operator-terminal. Derive the row set from credential IDENTITY; never transcribe the table 1:1.
4. Build tier 1 first (offline, no live access needed), harness it, then tier 2, then tier 3.