Newer
Older
openstack-caracal-dc-dc / docs / audit / creds-consolidation-audit-20260727.txt
=== creds consolidation audit -- 2026-07-27 (read-only; metadata only, no credential value read) ===
repo HEAD at audit: 4d62f66709366b17f21d36d53c976145e03a9470  branch: dc-dc-stage4-phase3-maas-deploy
question (operator, 2026-07-27): confirm a consolidated set of login creds on vcloud for all
accounts currently created; identify creds missing, needing creation, or not moved into the
vcloud folders.  Direction after the audit: 'clear the whole consolidation batch first'.
NOTE: sections below are the POST-REMEDIATION state. The pre-remediation run returned 13
findings; see docs/changelog-20260727-creds-consolidation.md for the per-item delta.

--- 1. ACCOUNTS THAT ACTUALLY EXIST on the MAAS region (ssh voffice1 'maas admin users read') ---
    Enumerated from the live surface, NOT from the register -- the register can only answer
    for accounts it has rows for, which is the blind spot this section exists to close.
  count=6
    MAAS               superuser=True  email=maas@localhost
    admin              superuser=True  email=admin@office1.local
    juju-vr1-dc0       superuser=True  email=juju-vr1-dc0@vr1.local
    juju-vr1-dc1       superuser=True  email=juju-vr1-dc1@vr1.local
    maas-init-node     superuser=False email=node-init-user@localhost
    operator           superuser=True  email=jesse.austin@neumatrix.com

--- 2. creds-audit.sh (SEC-009 declaration-based audit), all three sites ---
=== creds-audit: vr1-office1 (/home/jessea123/vr1-office1-creds) ===
  [ok]   folder mode 0700
  [ok]   all 14 manifest entries present, correct mode, non-empty

creds-audit vr1-office1: CLEAN
=== creds-audit: vr1-dc0 (/home/jessea123/vr1-dc0-creds) ===
  [ok]   folder mode 0700
  [ok]   all 7 manifest entries present, correct mode, non-empty

creds-audit vr1-dc0: CLEAN
=== creds-audit: vr1-dc1 (/home/jessea123/vr1-dc1-creds) ===
  [ok]   folder mode 0700
  [ok]   all 8 manifest entries present, correct mode, non-empty

creds-audit vr1-dc1: CLEAN

--- 3. jumphost (vcloud) creds folder contents, filenames only ---
$HOME/vr1-office1-creds/:
  maas-admin-password
  maas-operator-password
  netbox-admin-password
  office1_svc_ed25519
  office1_svc_ed25519.pub
  opnsense-api.txt
  opnsense-root-hash
  opnsense-root-password
  README
  tailscale-authkey.txt
  vr1-netbox.env
  vr1-netbox-sandbox.env
  vr1-office1.env
  vr1-stage1.env
$HOME/vr1-dc0-creds/:
  maas-api-key.txt
  vr1-dc0-edge_ed25519
  vr1-dc0-edge_ed25519.pub
  vr1-dc0-maas-power_ed25519
  vr1-dc0-maas-power_ed25519.pub
  vr1-dc0_svc_ed25519
  vr1-dc0_svc_ed25519.pub
$HOME/vr1-dc1-creds/:
  maas-api-key.txt
  opnsense-api.txt
  vr1-dc1-edge_ed25519
  vr1-dc1-edge_ed25519.pub
  vr1-dc1-maas-power_ed25519
  vr1-dc1-maas-power_ed25519.pub
  vr1-dc1_svc_ed25519
  vr1-dc1_svc_ed25519.pub

--- 4. headend (voffice1) shadow stores, filenames only (SEC-022) ---
voffice1:~/vr1-dc0-creds/:
  maas-virsh_ed25519
  vr1-dc0_svc_ed25519
  vr1-dc0_svc_ed25519.pub
voffice1:~/vr1-dc1-creds/:
  vr1-dc1_svc_ed25519
  vr1-dc1_svc_ed25519.pub

--- 5. creds-matrix.py --tier2 --tier3 --remote --privileged (D-137 register, ALL tiers) ---
=== creds-matrix: tier 1 (STATIC) ===
=== creds-matrix: tier 2 (EXISTENCE) ===
=== creds-matrix: tier 3 (VALIDITY) ===
  [ok]   S1 schema: 82 rows, all enums valid, site-keys region-qualified, no duplicate (id,site,host-role,filename)
  [ok]   S3 render: 3 source field(s) SKIPPED -- rendering them needs the declared path from creds-manifests/vm-secret-locations (ruling 3); the list now EXISTS but the source-field derivation is not wired
  [ok]   S3 render drift: rendered row fields (mode, source) match checked-in; header prose and non-jumphost rows are OUT OF SCOPE of this compare
  [ok]   S4 mint-ref: every script:/runbook: reference resolves to a real location
  [ok]   S4 provenance debt: 30 row(s) are mint-ref=operator-terminal -- NOT reproducible from the repo (research FINDING 1). Admitted by design; converting them is remediation, not a checker fix.
  [ok]   S7 notes: 34 note key(s) referenced, all resolve, none orphaned
  [ok]   E0 jumphost location '~/vault-init/*' does not exist -- SKIPPED (typically a mint stage this deployment has not reached)
  [ok]   E0 jumphost location '~/octavia-pki/*' does not exist -- SKIPPED (typically a mint stage this deployment has not reached)
  [ok]   E0 jumphost location '~/octavia-pki/issuing-ca/*' does not exist -- SKIPPED (typically a mint stage this deployment has not reached)
  [ok]   E0 jumphost location '~/octavia-pki/controller-ca/*' does not exist -- SKIPPED (typically a mint stage this deployment has not reached)
  [ok]   E0 jumphost location '~/octavia-pki/controller/*' does not exist -- SKIPPED (typically a mint stage this deployment has not reached)
  [ok]   E0 headend location '/root/maas-secrets/*' read via ESCALATION (sudo -n, metadata only) after an unprivileged probe could not open it
  [ok]   E0 headend location '/var/snap/maas/current/root/.ssh/*' read via ESCALATION (sudo -n, metadata only) after an unprivileged probe could not open it
  [ok]   E0 netbox location '/root/netbox-secrets/*' read via ESCALATION (sudo -n, metadata only) after an unprivileged probe could not open it
  [ok]   E1 18 expected artifact(s) deferred as not-yet-minted per creds-manifests/stages-reached
  [ok]   V2 ACKNOWLEDGED DEFERRAL: id 'netbox-upstream-token' (vr1-netbox.env) is custody=consolidated and SEC-006 declares it BURNED, but remediation is DEFERRED BY OPERATOR RULING -- the credential is still live and still exposed until that ruling is discharged. Ruling text: 'DEFERRED by operator ruling'
  [ok]   V2 declared state: no credential is registered healthy while its SEC row declares it burned/revoked/compromised
  [ok]   V1 provenance: all copies byte-identical for 6 group(s): dc0-svc-key; dc0-svc-key; dc1-svc-key; dc1-svc-key; maas-region-admin (declared verbatim copies); netbox-admin (declared verbatim copies)
  [FAIL] S2 vr1-dc0 EXPECTED-BUT-ABSENT: 'opnsense-api.txt' (id dc0-edge-api, SEC-021) is expected by the matrix but NOT declared in the manifest -- the credential is either missing or undeclared
  [FAIL] S5 ASYMMETRY: vr1-dc0 declares id=dcN-maas-power-key file=id_ed25519 on headend (custody=off-manifest-known) with no counterpart in vr1-dc1 -- a per-DC credential must exist at BOTH DCs in the same shape
  [FAIL] S5 ASYMMETRY: vr1-dc0 declares id=dcN-maas-power-key file=maas-virsh_ed25519 on headend (custody=off-manifest-known) with no counterpart in vr1-dc1 -- a per-DC credential must exist at BOTH DCs in the same shape
  [FAIL] S5 ASYMMETRY: vr1-dc1 declares id=dcN-maas-power-key file=id_dcN_power on headend (custody=off-manifest-known) with no counterpart in vr1-dc0 -- a per-DC credential must exist at BOTH DCs in the same shape
  [FAIL] S6 IDENTITY CONFLATION: id 'maas-region-admin' serves 2 principal types (human via gui; service via api, cli-profile) -- ruling 5 requires one identity to serve one principal type
  [FAIL] E1 EXPECTED-BUT-ABSENT: dc0-edge-api 'opnsense-api.txt' expected at jumphost/vr1-dc0, not found (mint-stage stage3, SEC-021)
  [FAIL] E4 UNCHECKABLE: 2 row(s) have no declared location for their (role, site) and can never be verified -- add a location row or correct the matrix: capi-mgmt-kubeconfig 'config' (cloud/-); rbd-mirror-peer-token 'rbd-mirror-bootstrap-token' (unit/-)

FAIL: creds-matrix tier 1 -- 82 row(s), 18 check group(s) clean, 7 finding(s)

--- 6. cardinality distribution + every per-site row (blind-spot evidence, item 3(a)) ---
     28 per-DC
      6 per-site
      5 per-tenant
     43 singleton
per-site rows (all vr1-office1; S5 compares only cardinality=per-DC, creds-matrix.py:380):
  office1-svc-key            vr1-office1  office1_svc_ed25519                ssh
  office1-svc-key            vr1-office1  office1_svc_ed25519.pub            none
  office1-edge-api           vr1-office1  opnsense-api.txt                   api
  office1-edge-root          vr1-office1  opnsense-root-password             gui
  office1-edge-root          vr1-office1  opnsense-root-hash                 none
  creds-folder-readme        vr1-office1  README                             none

--- 7. human-principal rows (the login-credential set) ---
  maas-region-admin        vr1-office1  headend   admin.pass                         gui         source-of-record         SEC-020
  maas-region-admin        vr1-office1  jumphost  maas-admin-password                gui         verbatim-copy            SEC-020
  maas-region-operator     vr1-office1  jumphost  maas-operator-password             gui         consolidated             SEC-020
  office1-edge-root        vr1-office1  jumphost  opnsense-root-password             gui         consolidated             SEC-007
  office1-edge-root        vr1-office1  jumphost  opnsense-root-hash                 none        consolidated             SEC-007
  netbox-admin             vr1-office1  netbox    admin.pass                         gui         source-of-record         SEC-025
  netbox-admin             vr1-office1  jumphost  netbox-admin-password              gui         verbatim-copy            SEC-025
  tenant-domain-admin      -            jumphost  <client>-domain-admin-cred.txt     gui         off-manifest-known       SEC-023
  tenant-cluster-user      -            jumphost  <client>-cluster-cred.txt          api         off-manifest-known       SEC-023
  libvirt-power-password   -            -         -                                  console     off-manifest-known       -

--- 8. host-role declaration gap (blind-spot evidence, item 3(b)) ---
  checker enum: 45:HOST_ROLE   = {"jumphost", "headend", "rack", "edge", "netbox", "cloud", "unit", "client", "-"}
  roles actually declared in creds-manifests/vm-secret-locations:
    headend
    jumphost
    netbox

--- 9. key-identity measurements behind the dc0 power-key decision (fingerprints only) ---
  dc0 power key, vcloud consolidated copy : 256 SHA256:Dt/YXTXSF4nXGj9cz8f0owL+qreVpMYBVm/igW10FXY maas-virsh-pod (D-123 amendment; MAAS snap -> local libvirt) (ED25519)
  dc0 SERVICE key (must DIFFER, SEC-012)  : 256 SHA256:DBkeStCyf2qGi3SYuvN6hQ+EsepmhpkN9oulLz09uAs vr1-dc0_svc (D-126 per-env key) (ED25519)
  dc1 power key, vcloud (the model shape) : 256 SHA256:BUAdNZEWbbn8Rg+QVSLaY8M6NSvc2nyTaMsQUHQEJhQ vr1-dc1-maas-power (SEC-016 dedicated MAAS->libvirt, per-DC) (ED25519)