# creds-matrix-notes.md -- `notes-ref` targets for creds-matrix.tsv (D-137)

The `notes-ref` column of `creds-matrix.tsv` points here. Free text lives HERE and never
in the TSV, so the TSV stays logical-keys-only (SEC-004) and greppable. Added by the
12-column amendment 2026-07-26 (`docs/D-137-implementation-plan.md`).

**Parse contract.** One `## <key>` heading per note; the body is every line until the next
heading. `scripts/creds-matrix.py --render` emits a note as the manifest comment above the
rows that reference it -- this is how D-137 ruling 2's "prose must not be dropped"
obligation is met for per-row governance text.

**No secret values, no paths, no hostnames** -- the same discipline as the TSV. Where a
note must refer to a location it names the role token and the register row that holds the
detail (`docs/security-ledger.md` remains the custody/exposure register of record).

---

## n-maas-region-admin
The MAAS region `admin` superuser. Password 28 characters, alphanumeric, NO trailing
newline -- that format spec is the verify-by-format basis; never verify by printing.
The REGION copy is source-of-record and the jumphost copy is a byte-identical working
copy (sha256-verified 2026-07-25). SEC-020 ruling: `admin`'s password was deliberately
NOT rotated, so **any future rotation MUST update both copies in one operation or the
region copy becomes a stale trap**.
This identity is also the automation identity, which is why it carries both a `human`
and a `service` row -- see `n-maas-cli-profile`. That is the ruling-5 violation D-137
exists to make machine-detectable, not a data-entry error.

## n-maas-admin-apikey
Deliberately NOT consolidated (SEC-020(i)): trivially regenerable, and a second copy is
pure added surface. `not-consolidated-ruled` records that this is a settled decision, so
`--remote` must not report it as an undeclared find.

## n-maas-cli-profile
The installer logs the CLI in as profile `admin`; 19 `maas admin ...` call sites across
4 scripts hardcode that profile (SEC-020(b)). ROOT CAUSE of the SEC-020 miss: automation
only ever needed the API key, so nobody ever needed the password, so nothing forced it
into the folder. Remediation is the `operator`/`admin` split begun 2026-07-25 -- migrate
those call sites to a service identity, which is what clears the ruling-5 FAIL.

## n-maas-region-operator
Minted 2026-07-25 for HUMAN GUI login so a person no longer authenticates as the service
identity. 32-char base64, generated on the jumphost, set via **stdin -- never argv**
(`createadmin --password` would expose it in argv, the shape the installer uses).
The first human-only MAAS identity.

## n-region-infra-carveout
Region infrastructure, ruled OUT of the consolidation task's scope (SEC-020(i)). Present
at a declared location by decision, not by oversight.

## n-rack-enroll
Side effect of `maas init region+rack`; consumed by each DC rack at enrollment via a
0600 copy. Mint and consume are different stages -- minted at region standup, consumed at
each DC standup.

## n-no-mint-command
**Research FINDING 1 -- reproducibility debt.** `ssh-keygen` returns ZERO hits repo-wide;
there is likewise no `mkpasswd`, `htpasswd`, `pwgen`, `uuidgen`, nor any Terraform
`random_password`/`tls_private_key`. This credential's only provenance record is a
manifest comment, so **it is not reproducible**: if the jumphost is rebuilt, no repo
artifact states how to recreate it. That is a Roosevelt-transfer defect, not merely
hygiene. `mint-ref=operator-terminal` rows are the debt to convert, and must be admitted
by the parser rather than treated as errors.

## n-edge-api
OPNsense REST API key+secret, minted ON the edge via the vendor model (D-113(a2)) into a
0600 temp file that is then wiped, and retrieved to the jumphost creds folder by the
bootstrap driver.

## n-third-party
Operator-supplied from a third-party control server. Correctly has NO repo mint command --
this is the one FINDING 1 row that is not a defect.

## n-netbox-upstream
Minted on the upstream NetBox apex. SEC-006 records this credential as BURNED; the
rotation obligation stands at v1 close.

## n-netbox-sandbox
**The founding SEC-009 miss.** Minted on the NetBox VM and never pulled into the creds
folder; it went unnoticed until the fidelity re-check needed it. That case is why
`creds-audit` exists at all, and why the source-of-record vs consolidated distinction is
modelled: a VM-minted secret has no forcing function to reach the folder.

## n-netbox-vm-secrets
Held on the NetBox VM, outside the jumphost creds folder. `netbox-secret-key` was rotated
off netbox-docker's PUBLIC default. The admin password is a human GUI login and is a
separate identity from the service SECRET_KEY -- do not merge these ids.

`netbox-admin` CONSOLIDATED 2026-07-27 (SEC-025): the VM copy is now `source-of-record` and
`~/vr1-office1-creds/netbox-admin-password` is a `verbatim-copy`, so tier-3 V1 asserts the two
stay byte-identical. It had never left the VM it was minted on -- the THIRD instance of the
miss class that SEC-009 (`api.token`) founded and SEC-020 (`admin.pass`) repeated: a
VM-minted HUMAN login with no forcing function to reach the operator's folder. It was
invisible to every control because `off-manifest-known` asserts nothing about the jumphost,
so no check could report it missing. Found only by ENUMERATING the accounts that actually
exist (`maas admin users read` for MAAS, this row's account by inspection) rather than
auditing the ones the register already knew about -- the register cannot report an absence
it has no row for. `netbox-secret-key` is deliberately NOT consolidated: it is a service
secret consumed only by the container, with no human who needs to read it.

## n-tofu-apikey
Office1 MAAS API key consumed by OpenTofu as `TF_VAR_maas_api_key`.

## n-tfstate-plaintext
DOCFIX-175: the same API key reaches `terraform.tfstate` in PLAINTEXT. A secret-at-rest
location with no mint of its own -- it exists only as a second copy, which is exactly what
the (credential, location) row shape is for. Any rotation of `n-tofu-apikey` must account
for this copy.

## n-office1-env
Environment file carrying the edge root-password hash and an SSH key path. One of the
three files whose loose, 0664 placement in the home directory WAS SEC-009.

## n-readme
NOT a credential -- the human index of the creds folder, declared in the manifest so the
folder's undeclared-file check does not fire on it. `principal` is `-` deliberately; the
ruling-5 invariant must skip rows with no principal rather than fault them.

## n-edge-key
Edge service SSH keypair for the D-112(c) console bootstrap. The runbook describes the
bootstrap step but does NOT contain the key generation itself -- see `n-no-mint-command`.

## n-dc0-edge-api-absent
**SEC-021, tier-1 detectable.** dc1 declares an edge API credential and dc0 does not, so
this expected row has no manifest row to satisfy it. This is an EXPECTED-BUT-ABSENT
failure, not a matrix error: do not delete the row to make the checker green.

## n-dc0-power-key-divergence
**SEC-021, per-DC asymmetry.** dc1's power key is consolidated on the jumphost under a
DC-qualified name; dc0's equivalent exists under a DIFFERENT name and on a DIFFERENT
host-role (the headend shadow store, SEC-022). Both the name and the location diverge from
the per-DC pattern, so the expected jumphost rows fail EXPECTED-BUT-ABSENT while the
divergent copy shows up at a declared remote location.

## n-snap-key-fragility
The private half is installed into the region MAAS snap so MAAS's own power operations can
dial the rack. **The snap path is per-revision and may not survive a snap refresh -- it
must be re-asserted after any refresh** (a standing DC-standup item). Per-DC isolation is
mandatory here: NEVER a cross-DC power key (SEC-012 dc0, SEC-016 dc1).

## n-dc1-power-key
Dedicated dc1 MAAS->libvirt power key, minted under the SEC-016 ruling "Mint a dedicated
dc1 power key" (per-DC isolation). Roosevelt analog: per-DC IPMI/BMC credentials.

## n-dc-maas-apikey
Per-DC MAAS admin API key backing the Juju cloud credential. Generated FROM the jumphost
so it never persisted on the region -- which is why its provenance is local rather than a
VM path. Verified by FORMAT ONLY (parts=3, 70 chars); never printed, echoed, or placed in
argv. GRANT is admin-scoped MAAS API. The owning MAAS superuser's password is random and
unstored by ruling -- see `n-juju-user-nopassword`.

## n-juju-cred-store
A runtime copy lives in the Juju client credential store on the HEADEND, a consequence of
running the Juju client there per D-128. The jumphost file stays canonical. Recorded
off-manifest in the manifests today (SEC-018/-019); this row is what makes it visible to
the checker instead.

## n-juju-user-nopassword
`createadmin` is invoked with NO `--password`, so the value is stored NOWHERE. Kept random
and unstored by operator decision recorded in SEC-018/SEC-019 (2026-07-24): the API key
is what is load-bearing, and the
`operator` identity removes any need to log in as this user. `filename` is `-` because
there is no host artifact -- existence checks must skip such rows rather than fail them.

## n-headend-shadow-store
**SEC-022 -- a scope gap in the control itself, not a one-off miss.** An entire parallel
`*-creds/` store exists on the headend holding real key material. `creds-audit.sh:29-31`
resolves the folder against `$HOME` on whatever host it runs on, and the script contains
no ssh/scp at all, so these stores are structurally invisible to every check that exists
today. The D-128 operating model puts Plane-2 execution on the headend, so headend-resident
credentials are EXPECTED to keep appearing -- this is why ruling 3's declared-locations
list must include the headend, and why adding a location row is DoD for any new mint site.

## n-outside-convention
**Research FINDING 2.** A credential directory outside the SEC-009 `*-creds/` convention
and outside `creds-audit` entirely -- the audit checks the per-site folders plus a loose
FILE glob in the home directory, and never inspects these DIRECTORIES. Named in CLAUDE.md
as a secret location, so it is KNOWN; it is simply outside the control. Materially extends
SEC-023. The Vault directory holds the cloud's entire root of trust and its mint is an
irreversible one-shot.

## n-overlay-in-clone
Contains base64 CA cert/key blobs PLUS the issuing-CA passphrase in PLAINTEXT, written
**inside the repo clone** and gitignored. Given SEC-004 (this repo is currently PUBLIC),
that gitignore is the only thing standing between a CA private key and publication.

**PER-DC SINCE 2026-07-29 (F2, under the D-109 amendment / R7 and R13 Part 1).** The nine
Octavia PKI ids were `cardinality=singleton` with `site-key=-` and `mint-stage=vr0-phase01`.
Two consequences, both measured:

- `creds-matrix.py`'s `s5_per_dc_symmetry` enforces both-DC existence ONLY for rows whose
  cardinality is `per-DC`, so a `singleton` declaration made the register **structurally
  blind to a missing second PKI set** -- the exact "a register cannot report the absence of
  something it has no ROW for" class this file exists to prevent, sitting behind the BLOCKING
  P5 gate.
- R13 Part 1 (RULED 2026-07-27) separately requires these rows to be staged to what Stage 5
  ACTUALLY mints, because staging them to a `pending` VR0 phase meant flipping `stage5` to
  `reached` would never make them expected -- a false green over the largest mint event of
  the deployment. R13's own text names R7 as making this MORE urgent: per-DC independence
  means TWO CA mints where the register expected none.

So the nine ids became **18 rows** (9 x 2 DCs), `per-DC`, region-qualified site-keys,
`mint-stage=stage5`, and the overlay row's filename is now per-DC
(`vr1-dc0-octavia-pki.yaml` / `vr1-dc1-octavia-pki.yaml`) to match the generator rename.
Mint-refs were RE-DERIVED from the edited `phase-01` -- `s4_mint_ref` only checks that a
line number is within EOF, so the old refs would have silently pointed at wrong lines.

PROVEN, not assumed: with both DCs declared the run is 91 rows / 7 findings (unchanged from
the 82/7 baseline, so this introduced none); deleting ONE dc1 row makes S5 report
`ASYMMETRY ... no counterpart in vr1-dc1` at 8 findings. The gate was demonstrated capable
of failing before being trusted.

**`stage5` DELIBERATELY STAYS `pending` in `creds-manifests/stages-reached`** until the PKI
is actually generated -- the rows are now correctly ATTRIBUTED, not yet expected. **Trap to
check before flipping it:** `stage5` is one coarse token that also covers eight already-
existing rows, so flipping it makes those expected simultaneously. Whether that is right, or
whether the mint needs a finer-grained token, is a question to settle at flip time.

**REMAINDER OF R13 PART 1, NOT DONE HERE:** the vault-init and `admin-openrc` rows are still
mis-staged the same way. vault-init is additionally per-DC under D-109's ORIGINAL text
(per-DC independent Vault roots), so it carries the same cardinality defect -- but its blast
radius reaches `~/vault-init/`, which CLAUDE.md designates operator-only one-shot territory,
so it was deliberately NOT bundled into an R7-conformance edit.

## n-tfstate-backup
**Recovery copy of a per-DC INNER substrate tfstate, added 2026-07-30 (operator-directed:
"extend the same pattern to the tfstates").** One gzip per DC in that DC's jumphost creds
folder, produced by `dc-dc-phase2-tofu-dc-substrate.md` step 13.

**WHY.** The inner `terraform.tfstate` is the substrate's STATE-OF-RECORD for 20 node VMs across
two DCs, lives ONLY on the Office1 headend, and is gitignored AND untracked by design.
CURRENT-STATE said it should be added to "the site backup set"; measured 2026-07-30, **no such
set existed**. Lose the state and `tofu` no longer knows those resources exist -- the next plan
proposes to CREATE what is already running.

**REGISTERED DESPITE PROBABLY NOT BEING A CREDENTIAL, and the reasoning matters.** The OUTER
state IS credential-bearing (DOCFIX-175: the MAAS API key in PLAINTEXT, id
`office1-tofu-maas-apikey`). The INNER states are different -- a name-based inspection found NO
`*api_key*`, `*password*`, `*secret*`, `*private_key*`, `*token*`, `user_data`, `cloudinit` or
`authorized*` keys in either, consistent with the inner root talking only to libvirt over
`qemu+ssh`, whose key is not in state. **That is strong evidence, not proof:** a name-based
absence cannot exclude material embedded inside a value. So it is registered and stored as if
sensitive. Two reasons, stated so a later reader does not "correct" it: uncertainty about
content should resolve toward MORE auditing, not less; and leaving an unaudited backup directory
beside audited ones is precisely how the SEC-022 headend shadow stores came about. **If the
classification is ever settled properly, this row is the thing to revisit.**

**DIFFERENT FROM `n-pki-backup` IN ONE WAY THAT BITES.** The Octavia CAs are static (10-year), so
one backup holds indefinitely. **tfstate is DYNAMIC** -- it changes on every apply, so the backup
is only meaningful with its `serial`, and restoring a STALE state is ACTIVELY DANGEROUS because
`tofu` will believe anything created since that serial does not exist. The procedure records the
serial on both sides and the step must be **re-run after every apply**.

**Restore is PROVEN, not assumed:** the pull block compares sha256 against the source AND
decompresses the archive to check it parses as JSON and reports a serial, because a truncated
gzip passes a size check and fails only on decompression.

**KNOWN, NOT FIXED:** both live inner states are mode **664** -- group-writable, and they are the
authority `tofu` trusts, so a group member who rewrites one can make `tofu` destroy or orphan
real resources. Tightening needs its own gated change plus verification that the provider
preserves the mode across the rewrite it performs on every apply.

**Same forward requirement as `n-pki-backup`:** the pinned secrets-storage solution must carry a
certificate/credential backup process, and this step is one of the steps that must fold into it.

## n-pki-backup
**Recovery copy of a per-DC Octavia PKI workspace, added 2026-07-30 (operator-directed).**
One gzipped archive per DC in that DC's jumphost creds folder, produced by
`phase-01-bundle-deploy.md` step 1.0-GEN.e.

**WHY IT EXISTS.** Three surfaces asserted this material "MUST be in the per-DC backup set" /
"backed up securely" and **no backup set existed** -- `scripts/cloud-snapshot.sh` is a
juju-layer capture and mentions octavia, tfstate and terraform ZERO times. So both DCs' 10-year
amphora trust roots sat in exactly one place, on one VM, with no copy. Losing the ISSUING CA key
means Octavia can never sign another amphora certificate (no new load balancers, no amphora
replacement); losing the CONTROLLER CA key means the controller certificate can never be
reissued, which F9 establishes it must be at Stage 7.

**WHAT IT COSTS.** A SECOND at-rest copy of both encrypted CA private keys BESIDE their
plaintext passphrases, so the archive's own contents defeat encryption-at-rest -- treat it
exactly as the live keys. This is the trade D-109 option (b) was refused for, and the
distinction is deliberate: that ruling governed where the AUTHORITATIVE artifact lives and
which host the deploy reads. A recovery copy is not a second source of truth.

**Integrity is asserted, not assumed:** the archive's sha256 is compared against the source
before the staging copy is removed from the headend, because a truncated `scp` would otherwise
leave a verified-looking backup of nothing.

**STANDING FORWARD REQUIREMENT -- carry this into the pinned secrets-storage design.** When the
secrets-storage solution is settled, it MUST include a documented PROCESS AND PROCEDURE for
CERTIFICATE and CREDENTIAL backup, and **this step is one of the steps that must be folded into
it.** The jumphost creds folder is the SEC-009 convention location and is the INTERIM home only.
Recorded here rather than in a runbook comment so the requirement is found by anyone reading the
register, and so it is not rediscovered the next time a 10-year CA turns out to exist in one
place. Related: R13 Part 2's reproducibility debt, and the 12 declared secrets with no mint
command anywhere in the repo.

## n-tenant-dir
Per-tenant credential directory outside the `*-creds/` convention (research FINDING 2).
Hard tenant isolation applies: D-069 keeps custody off-repo, and D-137 ruling 3 bounds
`--remote` to declared locations precisely so no tenant surface is ever walked. The
runbook `tenant-onboarding-v2-DRAFT.md` documents the SAME credential set as the onboard
script -- two creation locations, ONE set; do not double-count them into extra rows.

## n-charm-minted
Charm-minted at bundle apply with no repo mint site, then MATERIALIZED to a file by the
operator. In matrix scope per the operator-ruled materialization test (in-matrix if a
credential is ever written to a file, presented by a human, or consumed out-of-band,
regardless of who generated it) -- a generation-based test would have missed it entirely.

## n-predicted-openrc
**PREDICTED, not present (SEC-023).** The script that writes it runs at a stage this
deployment has not reached, so the checker must report this row as not-yet-expected via an
explicit skip with an `[ok]`, never as a silent pass. Both readings of the acceptance test
apply: this row exercises the not-yet-expected path, and the harness separately proves the
widened sprawl glob CATCHES the file once it appears.

## n-cloud-only
Lives in the cloud or on a unit rather than in a host file, but is consumed out-of-band
(captured by an operator command at a documented step), so the materialization test puts
it in scope. `filename` may be `-`; existence checks skip such rows.

## n-ephemeral-unit
Written to a 0600 temp file ON the unit during a stage step. In scope because it is written
to a file, but it is short-lived by design -- absence outside its step is normal, so this
row must never drive an existence failure.

## n-libvirt-power-pw
The libvirt SSH password for MAAS virsh power control. Read INTERACTIVELY (never a CLI
arg, never echoed, never logged, never in the repo) and injected into MAAS `power_pass`.
`reenroll-hosts.sh:22-25` records that it was exposed in plaintext on 2026-06-26 --
`maas machine power-parameters` echoes `power_pass` -- with a standing instruction to
ROTATE it. It is in scope by the materialization test (presented by a human, consumed
out-of-band) and had NO row until the 2026-07-26 committee found it: a credential with a
live rotation obligation that the credential-lifecycle authority could not see.

---

## Checker behaviour worth knowing before you "fix" it

**E2 reports FILE MODE, not reachability -- read an E2 finding before calling it an
exposure.** SEC-024 was opened on a 0664 state file and first written up as
"group- and world-readable"; measuring the path showed `opentofu/` is 0700 and `~` is 0750,
so no other account could actually reach it. It was a defence-in-depth defect, not a live
exposure, and the row had to be corrected the same day. Reporting mode alone is the RIGHT
default for a credential gate -- directory modes change, a `cp -p` or a tarball escapes
them, and a gate that reasons about traversal would go quiet the moment someone tightened a
parent directory for an unrelated reason. But it means the human writing up an E2 finding
owes a reachability check before using the word "exposed".

**S5 compares `mint-stage` and `mint-ref` as part of the per-DC shape, deliberately.**
Today both DCs agree on both, so it does not fire on them. A future DC that legitimately
mints the same credential at a DIFFERENT stage -- one standing up after Stage 5, say --
would trip S5 on a non-defect. That strictness is INTENDED: a per-DC credential appearing
at a different stage or from a different mint site is exactly the drift the register
exists to surface, and it should be argued about rather than absorbed. If a future DC
makes it genuinely wrong, narrow it with a recorded reason -- do not silently drop the two
fields from the comparison.

**S3 is a partial gate on ruling 2, not a full one.** It compares parsed row fields for
rows present on both sides; it does not compare manifest header prose, and `--render`
emits only the consolidated jumphost rows. Byte-equality becomes enforceable once the
manifests are actually replaced by rendered output, which is operator-gated because it
changes what `creds-audit` demands at three sites.

## n-colocated-nonsecret
NOT a credential -- a config or known-hosts artifact that happens to sit in a declared
credential location. Declared so the undeclared-file check stays meaningful instead of
crying wolf on every config file next to a key. `principal` is `-`, so ruling 5 skips it.

---

## Deliberate EXCLUSIONS (recorded so the decision is not silent)

These credentials appear in the research capture but are NOT matrix rows, because they
fail the materialization test -- never written to a file, never presented to a human,
never consumed out-of-band:

- Per-service DB / RabbitMQ / Ceph keyring credentials -- charm relation data (the capture
  confirms 56 apps, zero hardcoded secret options in the bundle).
- Vault-issued per-service TLS certificates -- they cascade from the charm-pki-local root;
  the verify script checks them, nothing mints them.
- The charm-pki-local root CA itself -- created inside Vault and never materialized.
- The short-lived (10-minute TTL) child Vault token and the charm authorization token --
  printed once, deliberately so the root token never reaches juju logs.
- CAPI/CAPO per-cluster CAs and bootstrap tokens, and per-cluster trustee users/trusts --
  in-cluster Kubernetes Secrets and cloud-side objects, minted at every cluster create.
- Throwaway acceptance-run credentials -- created and deleted within the same run.
- Tenant end-user passwords and client-minted CI application credentials -- these live on
  CLIENT infrastructure, outside this cloud's custody entirely.

If any of these later becomes materialized to a file, it acquires a row at that moment --
adding one is DoD for the change that materializes it.