Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260727-creds-consolidation.md
@JANeumatrix JANeumatrix 6 hours ago 11 KB creds consolidation batch: SEC-021(b) dc0 power key, SEC-025 NetBox GUI, V2 ruled deferral
Raw Blame History

changelog 2026-07-27 -- credential consolidation batch (SEC-021(b), SEC-025, V2 deferral)

Session changelog (ONE per session, GA-R2/D1). Status claims live ONLY in docs/CURRENT-STATE.md; this file records WHAT changed, WHY, and HOW TO REVERT.

Branch dc-dc-stage4-phase3-maas-deploy. Stage 4 stays OPEN; no gate closed.

Operator question that opened the work (2026-07-27): "Check and confirm that we have a consolidated set of login creds on vcloud for all accounts currently created. Are there any creds that should be in place that are missing, need to be created, or have not been moved into their vcloud folders?" Operator direction after the audit: "clear the whole consolidation batch first" -- i.e. the consolidation items, NOT the dc0 edge-API re-mint, which was presented as a separate live-mutation item and is deliberately NOT done here.

1. Audit capture (read-only)

docs/audit/creds-consolidation-audit-20260727.txt -- the full three-site sweep, metadata only, no credential value read anywhere. Sections: live MAAS account enumeration, creds-audit x3, both jumphost and headend folder listings, the full creds-matrix.py --tier2 --tier3 --remote --privileged run, the cardinality/per-site blind-spot evidence, the human-principal row set, and the host-role declaration gap.

The finding that drove everything else: creds-audit read CLEAN on all three sites while the matrix returned 13 findings, and the two most consequential gaps were in NEITHER surface -- they were found by enumerating the accounts that ACTUALLY EXIST. The register cannot report the absence of a credential it has no row for. creds-audit CLEAN remains not-evidence-of-completeness.

Revert: delete the capture file (it is a record, nothing depends on it).

2. dc1 svc public key backfilled to the headend store

voffice1:~/vr1-dc1-creds/vr1-dc1_svc_ed25519.pub placed (0644), sha256-verified identical to the vcloud copy. Clears one E1 EXPECTED-BUT-ABSENT (SEC-022). Public-key material only. dc0's shadow store already held both halves; dc1's held only the private -- the mirror image of the dc0 gap in item 3.

Revert: ssh voffice1 'rm ~/vr1-dc1-creds/vr1-dc1_svc_ed25519.pub'.

3. dc0 MAAS->libvirt power key consolidated to vcloud (SEC-021(b))

~/vr1-dc0-creds/vr1-dc0-maas-power_ed25519 (0600) + .pub (0644), plus two rows in creds-manifests/vr1-dc0.manifest. This is SEC-021(b) executed as written: "dc0's key should gain a jumphost copy + manifest row". Clears S2 x2 and E1 x2.

MEASURED before acting, because the naming looked like a defect and was not:

  • The headend's maas-virsh_ed25519 and the region MAAS snap's id_ed25519 are the SAME key (fingerprint SHA256:Dt/YXTXSF4nXGj9cz8f0owL+qreVpMYBVm/igW10FXY).
  • It IS the dedicated SEC-012 key, not a reuse: the dc0 SERVICE key is a different key (SHA256:DBkeStCyf2qGi3SYuvN6hQ+EsepmhpkN9oulLz09uAs), which is exactly what SEC-012's ruling ("Dedicated MAAS->libvirt key" over reusing the dc0 service key) requires.
  • dc0 having NO per-host block in the snap's ssh config (so it uses the default identity) is the RULED design, not drift -- SEC-016 states it: "Host 172.31.0.6 -> that IdentityFile, so 172.31.0.2/dc0 keeps SEC-012".

So NO re-mint was performed and no live power path was touched. The private half was copied from the voffice1 shadow store, sha256-verified byte-identical at both ends. The public half had never existed anywhere and was DERIVED with ssh-keygen -y, its fingerprint verified equal to the snap's installed key.

Two things deliberately NOT done, both recorded rather than fixed:

  • The key's INTERNAL comment still reads maas-virsh-pod (D-123 amendment) -- it was minted for the qemu+ssh pod attempt that was refuted the same day, and the per-machine power_type=virsh ruling correctly retained it. Rewriting a comment alters key material; the file name is now authoritative and the manifest comment carries the explanation.
  • Manifest source reads local to stay consistent with --render (which derives it and would otherwise raise an S3 drift FAIL); the true provenance is in the manifest comment, which S3 explicitly treats as out of scope. Writing voffice1:<path> there would have traded one finding for another, and writing a custody change would have created two NEW S5 asymmetries, since S5 compares custody.

Revert: rm ~/vr1-dc0-creds/vr1-dc0-maas-power_ed25519{,.pub} and drop the two rows + comment block from creds-manifests/vr1-dc0.manifest. The headend and snap copies were never modified, so power control is unaffected either way.

4. NetBox web-GUI admin password consolidated (SEC-025 OPENED)

~/vr1-office1-creds/netbox-admin-password (0600), byte-identical to office1-netbox:/root/netbox-secrets/admin.pass (sha256-verified both ends, 28 bytes, value never printed or echoed). Register changes: the netbox-admin netbox row custody off-manifest-known -> source-of-record; a NEW jumphost row with custody verbatim-copy (so tier-3 V1 now asserts the two copies stay byte-identical); a manifest row; an extended n-netbox-vm-secrets note.

This was a HUMAN GUI login that had never left the VM it was minted on -- the third instance of one miss class (SEC-009's api.token, SEC-020's admin.pass, now this). It produced no finding before this session because off-manifest-known asserts nothing about the jumphost.

SEC-025 is opened for the exposure the remediation CREATES (a plaintext human password at rest on the jumphost), not for the gap it closed -- the same shape as SEC-007 and SEC-020, with the same rotation obligation and the same both-copies-or-stale-trap hazard.

Revert: rm ~/vr1-office1-creds/netbox-admin-password; revert the matrix row pair, the manifest row, the note paragraph, and the SEC-025 ledger row. The VM copy is untouched.

5. V2 learns the ruled-deferral state, and gains its first harness cases

scripts/creds-matrix.py: load_sec_states now also reports whether a declared-dead SEC row carries an operator-ruled deferral; v2_declared_state emits V2 ACKNOWLEDGED DEFERRAL instead of a FAIL when it does.

WHY: SEC-006 has read "DEFERRED by operator ruling 2026-07-13: revoke at completion of this deployment" since long before this checker existed. V2 was reporting that standing operator ruling as a defect on every run -- which is both wrong and corrosive, because a finding that can never be actioned trains readers to skip findings. Reissuing the token would have contravened the ruling, so the register was the thing that needed to change, not the credential.

The acknowledgement is NOT a pass. It names the id, the SEC row, the state word and the ruling text, and states that the credential "is still live and still exposed until that ruling is discharged".

DEFERRAL_RE demands BOTH the emphasis-cased state word AND explicit attribution to an operator ruling (the GA-R5 bar), and [^|] stops it crossing a ledger table cell. A looser match would let prose that merely DISCUSSES deferring buy silence -- the false-green class this register exists to prevent. T58 and T59 are the regression locks for exactly that.

Also added --ledger (default unchanged). V2 read the SEC register via --repo, which the harness must point at the REAL repo for S4 mint-ref resolution -- so there was no fixture path to the ledger and V2 had shipped with ZERO harness cases. The override is what makes it testable; it also de-couples the existing tier-3 cases from the live ledger.

New cases T56-T59 (harness 56 -> 60): the founding invariant still fails; a ruled deferral is acknowledged and still says EXPOSED; unruled "deferred" prose still fails; a claim spanning two ledger cells does not count as ruled.

NOTE: tier 3 is NOT part of preflight P5 (which runs tier 1 + tier-2-local), so this does not change the blocking deploy gate's behaviour.

Revert: git revert this commit's changes to scripts/creds-matrix.py and tests/creds-matrix/run-tests.sh; V2 returns to failing on SEC-006 every run.

Findings LOGGED, not actioned (hard rule 1)

  1. dc0 edge API credential (opnsense-api.txt) is absent from every DECLARED location -- not the jumphost folder, not the headend shadow store. It exists only inside the dc0 edge's own config. Recovery is a re-mint via scripts/opnsense-bootstrap-apikey.sh (the dc1-proven path; dc0's edge SSH key IS present on vcloud), which is a live mutation on a running edge and was explicitly OUT of this batch. This is the sole remaining SEC-021(a) item and accounts for 2 of the 7 residual findings.
  2. No registered root/console credential at EITHER DC edge. MEASURED: no matrix row, no manifest entry, no SEC row. office1-edge-root is cardinality=per-site and office1-only. NOT MEASURED, and deliberately not probed: what those root passwords actually are. The phase-2 runbook says DC edges boot on FACTORY DEFAULTS and the D-112(c) chain installs sshd + the service key without setting a root password -- but that is a repo reading, not a measurement (hard rule 2). Root SSH is key-only (proven), so SSH is not the vector; the web GUI is LAN-reachable at both edges and the serial console exists. Proposed as the next-free SEC row once the operator gates a probe.
  3. Two structural blind spots that let (2) hide. (a) S5 compares only cardinality=per-DC (creds-matrix.py:380); all SIX per-site rows are office1-only, so a per-site credential missing at two of three sites is invisible to every check. This is the "cardinality largely inert" item CURRENT-STATE already carries -- now with a concrete consequence. (b) creds-manifests/vm-secret-locations declares only jumphost, headend, netbox, while the checker also accepts rack, edge, cloud, unit, client -- all with zero declared locations, therefore unaudited. SEC-015 was a rack-resident edge-key copy, so the class is not hypothetical.
  4. The 3 residual S5 power-key asymmetries are RULED, not defects. SEC-016 designs dc0 to keep the snap's default identity while dc1 gets a named per-host key, so the filenames can never match and DC_TOKEN_RE cannot normalise them. The register needs a way to record a ruled exception; suppressing them silently would be wrong and renaming live files to satisfy a checker would be worse. Needs an operator decision on mechanism.
  5. dc0/dc1 creds folders carry no README while office1 does (creds-folder-readme, also per-site). Cosmetic; noted for symmetry.
  6. The headend shadow stores (SEC-022) are now redundant for dc0's power key -- vcloud holds the copy SEC-021(b) asked for. Deleting the shadow copy is a plausible follow-up but is destructive and out of scope.

Verification

  • tests/creds-matrix/run-tests.sh -- 60/60 PASS (was 56).
  • scripts/creds-audit.sh all three sites -- CLEAN (office1 14 entries, dc0 7, dc1 8).
  • creds-matrix.py --tier2 --tier3 --remote --privileged -- 82 rows, findings 13 -> 7. Cleared: S2 x2, E1 x3 (2 power-key + 1 dc1 .pub), V2 x1 (now an acknowledged deferral). Residual 7: dc0-edge-api (2, needs the excluded re-mint), S5 x3 (ruled, item 4 above), S6 conflation x1 (the SEC-020 defect, separate), E4 uncheckable x2 (Stage-5/6 rows). The register is NOT green and is not expected to be.
  • scripts/repo-lint.sh + full gauntlet: see the commit message for the recorded results.