Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260726-d137-tier1.md

Session changelog 2026-07-26 -- D-137 credential matrix, tier 1 (STATIC)

Session-scoped scratch (GA-R2/D1). NOT a status or decision authority -- status lives only in docs/CURRENT-STATE.md, rulings only in docs/design-decisions.md. Consolidated at stage close.

Branch dc-dc-stage4-phase3-maas-deploy. No cloud state touched: every item below is offline repo work. Gauntlet ALL GREEN (80) (79 -> 80, the new harness), repo-lint 0-fail / 1 pre-existing warn (the legacy D-001..018 ASCII carve-out).


Item 1 -- D-137 matrix schema amended 10 -> 12 columns (operator-ruled)

What. docs/D-137-implementation-plan.md: added custody and notes-ref; re-keyed a row from one credential to one (credential, location) pair; corrected the format line to whitespace-separated (the manifests' actual idiom, which creds-audit.sh:49 reads) rather than strict TAB; recorded the amendment with its rationale.

Why. A read-first round-trip check of the three checked-in manifests against the 10-column schema found it could not carry three classes of governance prose that D-137 ruling 2 forbids dropping:

  • off-manifest carve-outs -- four deliberate, reasoned credential copies (both DCs' Juju client-store keys, the dc1 MAAS-snap power key, and the three ruled not-consolidated region files). Ruling 3 puts their locations INSIDE --remote's declared set, so creds-audit.sh:63-67 will visit every one and report it UNDECLARED. Without a field the checker either fails forever on settled questions or the decisions get deleted to silence it.
  • multi-copy credentials -- one credential, several locations, DIFFERENT filenames (admin.pass vs maas-admin-password), one designated authoritative copy. Tier 3's digest comparison cannot know what to compare without it.
  • verification-relevant detail -- format specs, mint dates, the not-reproducible debt.

Ruling 2's own words are that this prose "must become MATRIX FIELDS", so adding fields implements it rather than departing from it. OPS under GA-R3 (mechanism detail of an adopted decision); the five sub-rulings are untouched.

Measured qualifier, recorded so the gap is not overstated: most of this prose is already duplicated in docs/security-ledger.md (SEC-018 carries the Juju-store copy; SEC-020 carries source-of-record, the stale-trap warning, and all three carve-out reasons). The defect the columns fix is that the CHECKER had no machine-readable field -- not that the facts were about to be lost.

Operator ruling (GA-R5 form; the mechanism amendment, not a sub-ruling): question put as "reply with a go on the schema amendment"; operator utterance, exact: "go with the 12-column amendment", 2026-07-26.

Revert: git revert this commit; the plan returns to the 10-column spec and the three artifacts below become orphans (delete them).

Item 2 -- creds-matrix.tsv + creds-matrix-notes.md (the register)

What. 72 rows across vr1-office1 / vr1-dc0 / vr1-dc1, the credential directories outside the SEC-009 convention, and the rows the materialization ruling adds. 32 note keys.

Why the row count is not 55. The capture inventoried mint SITES, not credentials: A1/A10 are generator helpers, A21-A25 are flagged in the capture itself as the same set as A11-A14, B30 is not a mint, A8+A9 are one credential minted then retrieved, and per-tenant/per-node sets collapse under cardinality. Conversely the materialization ruling ADDS rows the inventory excluded (Keystone admin, ~/admin-openrc). Derived from identity, never transcribed 1:1.

Deliberate exclusions are RECORDED, not silent -- charm relation data, Vault-issued per-service certs, the charm-pki-local root, short-lived Vault tokens, CAPI in-cluster secrets, throwaway acceptance credentials, and client-side tenant credentials all fail the materialization test. The reasoning is in the notes file so a later reader can challenge it.

Revert: delete both files.

Item 3 -- scripts/creds-matrix.py tier 1 (STATIC)

What. Seven check groups in the provider-bundle-check.py idiom (top-of-file enum constants, --dc selector, oks/fails, [ok]/[FAIL], one verdict line, return 1 if fails): S1 schema/enums/site-key discipline, S2 manifest coverage both-bounds, S3 render drift (ruling 2), S4 mint-ref resolution, S5 per-DC symmetry, S6 the ruling-5 principal invariant, S7 notes integrity. --render prints derived manifests; it does NOT write.

Two design decisions worth challenging at review:

  • The manifest mode column is DERIVED (.pub -> 644, else 600) rather than becoming a 13th column. That rule reproduces all three checked-in manifests exactly.
  • --render deliberately does not write. Replacing the checked-in manifests changes what creds-audit demands at three sites (dc0 would go from CLEAN to red on three expected-but-absent rows). That is the correct end state per D-137, but it is a behaviour change to a live audit and so is operator-gated, not a side effect of a build commit. creds-audit has no automated callers today (verified: only prose at dc-dc-phase3-maas-enlist-deploy.md:498), so nothing is destabilised by waiting.

Two false-greens found and closed during the build, both self-inflicted:

  1. S2 enumerated only the sites the MATRIX mentions, so an empty or site-truncated matrix examined no manifest, found nothing undeclared, and reported "both bounds clean". That is precisely the defect netbox/sandbox-fidelity-check.py:131-143 records in-repo ("an upper bound masquerading as an assertion. It must be BOTH bounds") reproduced one level up. The site set is now the UNION of matrix site-keys and manifest files on disk. Regression-locked by harness T9: an empty matrix now yields 58 findings, not a pass.
  2. The uniqueness key was (site, host-role, filename), which false-failed two DISTINCT identities sharing one file -- the Vault init artifact holds both the unseal shares and the root token. Key now includes id. Regression-locked by T6.

Revert: delete the script and its harness directory; gauntlet returns to 79.

Item 4 -- tests/creds-matrix/run-tests.sh (24 cases, 24/24)

One case per invariant, each named for the real defect it encodes, asserting rc AND an output regex. Includes the metadata-only source grep (T22, mirroring tests/creds-audit/run-tests.sh:71-73) and T24, which asserts the real tree stays RED -- a guard against someone making the register green by deleting the rows that fail.

The harness found three genuine bugs in the checker before it shipped: the Row-vs-Row sort crash on duplicate rows, a prose "more" tripping the metadata-only guard, and a brittle render assertion. All three fixed.

Revert: delete tests/creds-matrix/.

Item 5 -- corrected a wrong sec-ref before it could be cited as authority

dc1-edge-api carried SEC-015. Verified against docs/security-ledger.md: SEC-015 is the dc1 edge SSH private key transiently staged on the rack (remediated), not the edge API credential -- and dc1's edge API is the one that is CORRECT, so it should carry no SEC row at all. Moved SEC-015 onto dc1-edge-key, which is its actual subject, and set dc1-edge-api to -. Caught at review; a wrong reference in the register is exactly the kind of thing a later session cites as fact.

Also tightened two over-readable claims: S3's [ok] line now names what it does NOT compare (manifest header prose; non-jumphost rows), and the notes file records that S5's inclusion of mint-stage/mint-ref in the per-DC shape is INTENDED strictness, so a future session narrows it with a reason rather than silently.


EXPECTED RED -- the 5 findings are the deliverable, not a defect in it

python3 scripts/creds-matrix.py exits 1 with:

  • S6 IDENTITY CONFLATION: id 'maas-region-admin' serves 2 principal types -- ruling 5 working as designed on the live SEC-020 defect. Clears when the 19 maas admin call sites migrate to a service identity.
  • S2 ... EXPECTED-BUT-ABSENT x3 (dc0 opnsense-api.txt, vr1-dc0-maas-power_ed25519, .pub) -- SEC-021's declaration half.
  • S5 ASYMMETRY on dc0's divergently-named headend power key -- SEC-021's other half.

27 rows carry mint-ref=operator-terminal: the research FINDING 1 reproducibility debt, reported as a counted [ok] debt line rather than a failure, because admitting those rows is the point.

Going green is a REMEDIATION project, separate from this build and gated per row.

NOT done in this session (stated so the boundary is unambiguous)

  • Tier 2 (EXISTENCE) and tier 3 (VALIDITY) -- not built. --all names them as unbuilt with an explicit [ok]; they never pass silently.
  • creds-manifests/vm-secret-locations (ruling 3) -- not written. Two --render source fields depend on it and are explicitly skipped as pending, not silently matched.
  • The preflight Pn of ruling 1 -- not wired, and this is an OPEN QUESTION for the operator, not an omission: wiring it blocking (the default, since it is what was ruled) hard-fails preflight.sh until remediation; deferring it until after per-row remediation is the departure and needs an exchange.
  • SEC-009's demotion to a pointer (ruling 4) -- not done; it belongs with tier 2.
  • No manifest was rewritten; no cloud state was touched.