Newer
Older
openstack-caracal-dc-dc / docs / audit / creds-creation-points-20260725.md

Credential creation-point inventory -- 2026-07-25

Read-only research capture for the D-137 credential-matrix design. No credential file was opened; every row below is a file:line reference to CODE or RUNBOOK PROSE that mints, materializes, or uploads credential material.

Classification used: MINT (new secret material comes into existence), MATERIALIZE (an existing secret is fetched/copied/written somewhere new), UPLOAD (public key registered -- no secret created). Only MINT rows are creation points; the other two classes were checked and excluded deliberately.

Operator-ruled scope for the matrix (this session): the materialization test -- in-matrix if a credential is ever written to a file, presented by a human, or consumed out-of-band, regardless of who generated it.

Counts

Class Count Where
Scripted MINT (repo code generates the secret) 25 rows scripts/
Runbook/prose MINT (operator runs by hand, no script owns it) 30 rows runbooks/, clientdocs/
Declared secrets with NO creation point anywhere in the repo 12 see below
Charm/platform-minted at bundle apply summarised bundle.yaml deploy
MATERIALIZE / UPLOAD (excluded, verified non-creating) 20+ see source agent record

FINDING 1 -- twelve declared secrets have NO mint command anywhere

This is the most consequential result and it qualifies the "creation points are already known" premise: for twelve standing credentials, they are not.

grep -rnI "ssh-keygen" . returns ZERO hits repo-wide (searched everything except .git/.terraform/tfstate/tfplan, including tests/, docs/archive/, clientdocs/, .claude/). There is likewise no mkpasswd, htpasswd, password_hash, pwgen, uuidgen, nor any Terraform random_password / tls_private_key. Yet the manifests declare:

Credential Declared at Only provenance record
office1_svc_ed25519{,.pub} vr1-office1.manifest:30-31 vr1-office1-as-built.md:126 (SEC-007)
vr1-dc0_svc_ed25519{,.pub} vr1-dc0.manifest:16-17 a manifest COMMENT (:10-12)
vr1-dc1_svc_ed25519{,.pub} vr1-dc1.manifest:16-17 a manifest COMMENT (:10-12)
vr1-dc0-edge_ed25519{,.pub} vr1-dc0.manifest:22-23 manifest comment (backfilled 2026-07-25)
vr1-dc1-edge_ed25519{,.pub} vr1-dc1.manifest:21-22 manifest comment (:18-20)
vr1-dc1-maas-power_ed25519{,.pub} vr1-dc1.manifest:29-30 manifest comment (:26-28, SEC-016)
opnsense-root-password vr1-office1.manifest:34 vr1-office1-as-built.md:128, SEC-007 -- HUMAN GUI login
opnsense-root-hash vr1-office1.manifest:33 same (bcrypt of the above, for config.xml)
tailscale-authkey.txt vr1-office1.manifest:36 SEC-008 -- operator-supplied, third-party control server (correctly has no repo mint)
vr1-netbox.env vr1-office1.manifest:37 minted on the upstream apex; SEC-006 (BURNED)
vr1-office1.env / vr1-stage1.env .manifest:39-40 vr1-office1-as-built.md:132-133

Six SSH keypairs and the OPNsense root password/hash are pure operator-terminal mints whose only record is a manifest comment. Consequence: they are not reproducible. If the jumphost is rebuilt, no repo artifact states how to recreate them -- which is a Roosevelt-transfer defect, not merely a hygiene one. The tailscale key is genuinely third-party and correctly has no repo mint.

FINDING 2 -- credential directories OUTSIDE the *-creds/ convention

SEC-009 says all site secrets live in ~/<site>-creds/. The inventory shows at least three other jumphost directories holding credential material, none of which creds-audit covers (it checks ~/<site>-creds/ folders plus a loose-FILE sprawl glob in $HOME -- it never inspects these DIRECTORIES):

  • ~/vault-init/init.txt -- Vault 5 unseal shares + root token, the cloud's entire root of trust (runbooks/phase-02-vault-bringup.md:71, irreversible one-shot).
  • ~/octavia-pki/ -- 8 artifacts: 2 CA passphrases, 2 CA private keys, 2 CA certs, the controller key, the signed bundle (runbooks/phase-01-bundle-deploy.md:299-410).
  • ~/tenant-<client>/ -- per-tenant domain-admin / cluster / svc passwords, an application credential, and a Nova private key (scripts/tenant-onboard.sh:70-71, 94-95, 141, 167).

All three are named in CLAUDE.md as secret locations, so they are KNOWN -- they are simply outside the control. This materially extends SEC-023.

Related: runbooks/phase-01-bundle-deploy.md:391-410 writes overlays/octavia-pki.yaml containing base64 CA key blobs plus the issuing-CA passphrase in plaintext, inside the repo clone (gitignored). Given SEC-004 (repo currently PUBLIC), that gitignore is the only thing standing between a CA private key and publication.

FINDING 3 -- highest-concentration mint sites (matrix seeding order)

  1. scripts/site-headend-install.sh -- 7 mints (:418 primitive, :430 DB pass, :445 rack-enrollment secret, :450/:452 admin password, :453 API key, :485 LXD trust). Every Office1/DC-region MAAS + LXD secret originates here.
  2. scripts/tenant-onboard.sh -- 7 mints, the entire tenant identity set.
  3. runbooks/phase-01-bundle-deploy.md:299-410 -- 8 Octavia PKI artifacts.
  4. runbooks/tenant-onboarding-v2-DRAFT.md:162-320 -- 6 mints DUPLICATING #2 (two creation locations, one credential set -- do not double-count).
  5. runbooks/phase-02-vault-bringup.md:71-144 -- 4 mints incl. the root of trust.
  6. runbooks/dc-dc-phase4-juju-bundle-per-dc.md:119-151 -- 4 mints PER DC.

FINDING 4 -- corroborations of already-recorded defects

  • dc-dc-phase4-juju-bundle-per-dc.md:128 mints juju-<dc> via maas createadmin with no --password, so MAAS prompts and the value is stored nowhere -- independently confirming the SEC-020 finding about those two superusers.
  • site-headend-install.sh:452 uses --password in argv, the exposure SEC-020 flagged and which this session's operator mint avoided via stdin.
  • scripts/phase-03-admin-openrc.sh:46-52 is classified MATERIALIZE (charm-generated, operator-materialized) -- exactly the case the ruled materialization test captures and a generation-based test would have missed (SEC-023).

What this means for the matrix

The matrix must carry a mint-ref column that can hold three provenance kinds, not one: script:line, runbook:step, and operator-terminal (undocumented). Finding 1 means a meaningful fraction of today's estate is the third kind, so a checker that requires every row to name executable code would fail 12 rows on day one. Those rows are the backlog the matrix exists to surface -- they should be admitted as operator-terminal and tracked as a debt to convert, not treated as parse errors.

Source: three read-only Explore agents, 2026-07-25. The full per-row tables ARE NOW IN THIS FILE (see APPENDIX below), transcribed 2026-07-26 before the originating session was cleared -- they are the creds-matrix.tsv seed and must not depend on a transcript.


APPENDIX -- full per-row inventory (the matrix seed)

Transcribed into the repo 2026-07-26 because the body above pointed at a session transcript that was about to be cleared. These rows ARE the creds-matrix.tsv seed; they are what docs/D-137-implementation-plan.md step 3 tells the build to read. Classification: MINT = new secret material; MATERIALIZE/UPLOAD excluded (section E).

A. Scripted MINT points (repo code generates the secret)

# file:line Credential For Written to Stage/step Principal
A1 site-headend-install.sh:416-419 newpass() primitive: `openssl rand -base64 30 \ tr -dc A-Za-z0-9 \ head -c 28`, 0600 voffice1 (helper) Stage 2 --
A2 site-headend-install.sh:430 (:151 SECRETS=/root/maas-secrets) PostgreSQL maas role password region DB voffice1:/root/maas-secrets/db.pass REMOTE Stage 2 step 3 service
A3 site-headend-install.sh:450,452 MAAS superuser admin password (--password in ARGV) region voffice1:/root/maas-secrets/admin.pass REMOTE; copy consolidated 2026-07-25 Stage 2 step 4 BOTH (conflated)
A4 site-headend-install.sh:453 MAAS API key for admin region voffice1:/root/maas-secrets/admin.apikey REMOTE, deliberately not consolidated Stage 2 step 4 service
A5 site-headend-install.sh:455 MAAS CLI profile admin (~/.maas.cli) region region VM home Stage 2 step 4 service
A6 site-headend-install.sh:445-447 (doc :25-29) MAAS rack-enrollment secret (side effect of maas init region+rack) region -> DC racks voffice1:/var/snap/maas/common/maas/secret; 0600 copy to rack via --enroll-secret-file Stage 2 mint / Stage 3 consume service
A7 site-headend-install.sh:485-486 LXD trust password voffice1 LXD voffice1:/root/maas-secrets/lxd-trust.pass REMOTE Stage 2 step 6 service
A8 opnsense-mint-apikey.php:43 (write :52-60) OPNsense REST API key+secret (vendor model) any edge <edge>:/tmp/opnsense-apikey.$$.txt 0600, then wiped D-113(a2) service
A9 opnsense-bootstrap-apikey.sh:89-90,114-120 (cleanup :93) driver for A8; retrieves key to jumphost edge -> jumphost ~/vr1-<site>-creds/opnsense-api.txt 0600 Stage 3 replacement chain step 2 service
A10 tenant-onboard.sh:39 newpw() secrets.token_urlsafe(24) primitive Keystone (helper) tenant onboard --
A11 tenant-onboard.sh:64-65 (write :70-71) <client>-domain-admin password tenant domain ~/tenant-<client>/<client>-domain-admin-cred.txt 0600 onboard stage1 human
A12 tenant-onboard.sh:90-91 (write :94-95) <client>-cluster + <client>-svc passwords tenant project ~/tenant-<client>/*-cred.txt 0600 onboard stage2 cluster=human-ish, svc=service
A13 tenant-onboard.sh:141 Keystone application credential <client>-svc-cred Keystone ~/tenant-<client>/<client>-svc-appcred.txt 0600 onboard stage3 (re-run guard :125-139) service
A14 tenant-onboard.sh:167 Nova keypair PRIVATE key <client>-key Nova ~/tenant-<client>/<client>-key.pem 0600 onboard stage3 (guard :154-166) service
A15 dc-dc-rbd-mirror.sh:88,156 Ceph rbd-mirror pool peer bootstrap token primary-DC Ceph /tmp/rbd-mirror-bootstrap-token.<rand> 0600 ON THE UNIT Stage 6 step 11.x service
A16 dc-dc-radosgw-multisite.sh:172,178 RGW multisite system user access-key+secret (radosgw-generated at realm create) radosgw DC1 stays in the RGW realm on the unit Stage 6 step 4 service
A17 phase-06-capi-stack.sh:91 (def :78) cluster-admin kubeconfig (sudo k8s config) capi-mgmt VM <mgmt VM>:~/.kube/config 0600 REMOTE phase-06 step 6.6a service
A18 phase-06-capi-stack.sh:147-153 CAPI/CAPO stack: per-cluster CA + bootstrap tokens at every cluster create capi-mgmt in-cluster k8s Secrets phase-06 step 6.6d service
A19 clientdocs/scripts/acceptance-run.sh:349 throwaway Nova keypair ci-accept-key tenant client host; deleted :129 acceptance D2 service (ephemeral)
A20 clientdocs/scripts/acceptance-run.sh:464 Barbican secret ci-accept-secret Barbican Barbican; deleted :470 acceptance G1 service
A21-A25 tenant-onboarding-v2-DRAFT.md:162,225-226,290,295,308-312,319-320 SAME credential set as A11-A14 -- the DRAFT runbook duplicates tenant-onboard.sh. Two creation LOCATIONS, one set. Do not double-count. Keystone/Nova $HOME/<client>-* onboard-v2 mixed

B. Runbook / prose MINT points (operator by hand; no script owns it)

# file:line Credential Written to Stage/step Principal
B1 phase-02-vault-bringup.md:71 Vault operator init: 5 unseal shares + root token. IRREVERSIBLE one-shot (:14-15,63-64) the Vault init dir on the jumphost -- OUTSIDE any *-creds/ phase-02 step 2.1 service (root of trust)
B2 phase-02-vault-bringup.md:127-130 short-lived child Vault token (-ttl=10m), so root never hits juju logs printed once phase-02 step 2.3 service
B3 phase-02-vault-bringup.md:142 charm-vault credential (authorize-charm token=) inside the charm phase-02 step 2.3, DOCFIX-011 service
B4 phase-02-vault-bringup.md:144 charm-pki-local root CA (private key + cert) -- the whole cloud's TLS root inside Vault phase-02 step 2.3, DOCFIX-014 service
B5-B12 phase-01-bundle-deploy.md:299-386 8 Octavia PKI artifacts: issuing-CA passphrase (:299-301), issuing-CA key EC P-384 AES-256 (:302-304), issuing-CA cert (:305-307), controller-CA passphrase (:321-323), controller-CA key (:324-326), controller-CA cert (:327-329), controller key EC P-256 UNENCRYPTED (:343-344), signed controller.bundle.pem (:375-386) the Octavia PKI dir on the jumphost -- OUTSIDE any *-creds/ phase-01 step 1.0-GEN.a/b/c service
B13 phase-01-bundle-deploy.md:391-410 overlays/octavia-pki.yaml -- base64 CA cert/key blobs + the issuing-CA passphrase in PLAINTEXT INSIDE the repo clone (gitignored) phase-01 step 1.0-GEN.d service
B14 dc-dc-phase4-juju-bundle-per-dc.md:128 MAAS superuser juju-<dc> -- createadmin with NO --password, so stored nowhere nowhere Stage 5 step 2.0 BOTH (no custody)
B15 dc-dc-phase4-juju-bundle-per-dc.md:136-139 per-DC MAAS API key (apikey --generate, run FROM the jumphost) ~/vr1-<dc>-creds/maas-api-key.txt 0600 (SEC-018/019) Stage 5 step 2.0 service
B16 dc-dc-phase4-juju-bundle-per-dc.md:141-142 Juju cloud credential wrapping B15 Juju client store on voffice1 -- explicitly OFF-MANIFEST Stage 5 step 2.0 service
B17 dc-dc-phase4-juju-bundle-per-dc.md:151 juju bootstrap: controller admin credential + controller CA/agent secrets, per DC (D-104) controller + ~/.local/share/juju/ Stage 5 step 2 service
B18 dc-dc-phase1-office1-standup.md:464-471 Office1 MAAS API key for OpenTofu (TF_VAR_maas_api_key); DOCFIX-175 warns it lands in terraform.tfstate PLAINTEXT ~/vr1-office1-creds/vr1-stage1.env 0600 Stage 2 step 5 service
B19 dc-dc-phase1-office1-standup.md:643-647 NetBox API token office1-netbox:/root/netbox-secrets/api.token REMOTE; copy in vr1-netbox-sandbox.env (the founding SEC-009 miss) Stage 2 step 9 service
B20 implied vr1-office1-as-built.md:130, prose :631-637 NetBox SECRET_KEY (rotated off netbox-docker's PUBLIC default) + NetBox admin password office1-netbox:/root/netbox-secrets/ REMOTE Stage 2 step 9 SECRET_KEY=service, admin=human
B21 dc-dc-phase2-tofu-dc-substrate.md:371-395 (D-112(c)) edge service SSH keypair + root access via serial-console payload ~/vr1-dc<N>-creds/vr1-dc<N>-edge_ed25519 Stage 3 replacement chain step 1 service
B22 dc-dc-phase2-tofu-dc-substrate.md:770-778 (SEC-012/-016) per-DC MAAS->libvirt power keypair, never cross-DC private half in voffice1:/var/snap/maas/current/root/.ssh/ (per-snap-revision, re-assert after refresh); dc1 copy on jumphost, dc0 has none (SEC-021) Stage 3 DoD item 5 service
B23 dc-dc-phase5-dr-failover-drill.md:230,261-263 RGW multisite system key ("never invent it"; capture via radosgw-admin user info) remote RGW realm Stage 6 step 4 service
B24 dc-dc-phase5-dr-failover-drill.md:294,411 rbd-mirror peer bootstrap token (manual form of A15) remote unit Stage 6 step 11.5 service
B25 phase-07-conductor-graft.md:89 / phase-07-conductor-graft.sh:90 Keystone magnum domain + magnum_domain_admin user + password (D-046, re-run after every redeploy) Magnum charm config phase-07 step 7.0 service
B26 appendix-D-magnum-trust-model.md:18-25,77; tenant-acceptance.sh:54; phase-08-*.md:35,179 per-cluster trustee user + Keystone trust + CAPO child app credential, minted at EVERY cluster create (D-039/D-046) cloud only, no host file phase-08 / every create service
B27 changelog-20260725-maas-admin-recovery.md:24-29 MAAS superuser operator -- 32-char base64, stdin never argv ~/vr1-office1-creds/maas-operator-password 0600 ad-hoc 2026-07-25 (SEC-020c) human (first human-only MAAS identity)
B28 clientdocs/self-service-guide.md:51-52; tenant-skill/references/day2-operations.md:147-148 tenant end-user Keystone passwords (--password-prompt), client-run client host tenant day-2 human
B29 clientdocs/ci-automation.md:63-70; ci-integration-guide.md:102-109 client-minted app credential pipeline-nightly client CI secret store tenant CI service
B30 dc-dc-phase3-maas-enlist-deploy.md:491-497 not a mint -- the standing instruction enumerating what a DC standup mints and requiring immediate consolidation + a manifest row ~/vr1-<dc>-creds/ Stage 4 close-out --

D. Charm/platform-minted at bundle apply (no repo mint site)

  • Keystone admin password -- charm-minted; MATERIALIZED by phase-03-admin-openrc.sh:52 via juju run keystone/leader get-admin-password + extract_admin_password.py. In matrix scope per the materialization ruling.
  • Per-service DB / RabbitMQ / Ceph keyring credentials -- charm relation data, never materialized. OUT of scope (56 apps, zero hardcoded secret options in bundle.yaml).
  • Vault-issued per-service TLS certs -- cascade from B4; phase-04-internal-cert-san-verify.sh verifies, does not mint.

E. MATERIALIZE / UPLOAD -- checked and EXCLUDED (not creation points)

phase-03-admin-openrc.sh:46-52,66-80 (writes ~/admin-openrc, SEC-023); extract_admin_password.py; phase-06-kubeconfig-gate.sh:36,59-77; phase-07-conductor-graft.sh:121-129 (base64 to the conductor unit, sha256-verified); phase-06-net-setup.sh:23,48 and phase-06-incloud-mgmt-cluster.md:206 (UPLOAD of an existing pubkey); opentofu/main.tf:218-219,444-447,575-578 (cloud-init ssh_authorized_keys via file()); vr1-dc{0,1}-substrate/main.tf:21,18 (qemu+ssh?keyfile= -- path only, D-126 boundary); opnsense-api.sh:32,75 (consume); reenroll-hosts.sh:122-123 (read -rsp a libvirt password into MAAS power params -- rotation warning :22-25); vault-kv-inner-probe.sh:12-13,22 (consume); site-ssh-config.sh / site-forward.sh / site-baseleg.sh (explicit no-key-material boundaries); netbox/*.py (consume NETBOX_TOKEN from env only); creds-audit.sh (metadata only); .claude/hooks/guard-destructive.py:28-42; snapshot_scrub.py:10.

G. Coverage boundary

Searched beyond scripts/+runbooks/: bundle.yaml, overlays/, clientdocs/ (incl. scripts/ and tenant-skill/), policies/, skills/, .claude/, creds-manifests/, _handoff-not-committed/. tests/ EXCLUDED as fixtures (tests/tenant-onboard/run-tests.sh:21,86 mints a fake CA/PEM; test_logic.py uses NETBOX_TOKEN="fake-token") -- no real credential originates there. NEVER opened, and excluded from content-printing greps: opentofu/terraform.tfstate*, the 8 *.tfplan, logs/, asbuilt/, docs/audit/, netbox/draft/*.json. opentofu/README.md + variables.tf:6-10 confirm maas_api_key reaches tfstate in plaintext (DOCFIX-175) -- a secret-at-rest location with no mint of its own.