Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260727-creds-consolidation.md
@JANeumatrix JANeumatrix 5 hours ago 17 KB Stage 4 close-out: fix dc-mirror.sh check false-greening the mirror gate
Raw Blame History

changelog 2026-07-27 -- credential consolidation batch (SEC-021(b), SEC-025, V2 deferral)

Session changelog (ONE per session, GA-R2/D1). Status claims live ONLY in docs/CURRENT-STATE.md; this file records WHAT changed, WHY, and HOW TO REVERT.

Branch dc-dc-stage4-phase3-maas-deploy. Stage 4 stays OPEN; no gate closed.

Operator question that opened the work (2026-07-27): "Check and confirm that we have a consolidated set of login creds on vcloud for all accounts currently created. Are there any creds that should be in place that are missing, need to be created, or have not been moved into their vcloud folders?" Operator direction after the audit: "clear the whole consolidation batch first" -- i.e. the consolidation items, NOT the dc0 edge-API re-mint, which was presented as a separate live-mutation item and is deliberately NOT done here.

1. Audit capture (read-only)

docs/audit/creds-consolidation-audit-20260727.txt -- the full three-site sweep, metadata only, no credential value read anywhere. Sections: live MAAS account enumeration, creds-audit x3, both jumphost and headend folder listings, the full creds-matrix.py --tier2 --tier3 --remote --privileged run, the cardinality/per-site blind-spot evidence, the human-principal row set, and the host-role declaration gap.

The finding that drove everything else: creds-audit read CLEAN on all three sites while the matrix returned 13 findings, and the two most consequential gaps were in NEITHER surface -- they were found by enumerating the accounts that ACTUALLY EXIST. The register cannot report the absence of a credential it has no row for. creds-audit CLEAN remains not-evidence-of-completeness.

Revert: delete the capture file (it is a record, nothing depends on it).

2. dc1 svc public key backfilled to the headend store

voffice1:~/vr1-dc1-creds/vr1-dc1_svc_ed25519.pub placed (0644), sha256-verified identical to the vcloud copy. Clears one E1 EXPECTED-BUT-ABSENT (SEC-022). Public-key material only. dc0's shadow store already held both halves; dc1's held only the private -- the mirror image of the dc0 gap in item 3.

Revert: ssh voffice1 'rm ~/vr1-dc1-creds/vr1-dc1_svc_ed25519.pub'.

3. dc0 MAAS->libvirt power key consolidated to vcloud (SEC-021(b))

~/vr1-dc0-creds/vr1-dc0-maas-power_ed25519 (0600) + .pub (0644), plus two rows in creds-manifests/vr1-dc0.manifest. This is SEC-021(b) executed as written: "dc0's key should gain a jumphost copy + manifest row". Clears S2 x2 and E1 x2.

MEASURED before acting, because the naming looked like a defect and was not:

  • The headend's maas-virsh_ed25519 and the region MAAS snap's id_ed25519 are the SAME key (fingerprint SHA256:Dt/YXTXSF4nXGj9cz8f0owL+qreVpMYBVm/igW10FXY).
  • It IS the dedicated SEC-012 key, not a reuse: the dc0 SERVICE key is a different key (SHA256:DBkeStCyf2qGi3SYuvN6hQ+EsepmhpkN9oulLz09uAs), which is exactly what SEC-012's ruling ("Dedicated MAAS->libvirt key" over reusing the dc0 service key) requires.
  • dc0 having NO per-host block in the snap's ssh config (so it uses the default identity) is the RULED design, not drift -- SEC-016 states it: "Host 172.31.0.6 -> that IdentityFile, so 172.31.0.2/dc0 keeps SEC-012".

So NO re-mint was performed and no live power path was touched. The private half was copied from the voffice1 shadow store, sha256-verified byte-identical at both ends. The public half had never existed anywhere and was DERIVED with ssh-keygen -y, its fingerprint verified equal to the snap's installed key.

Two things deliberately NOT done, both recorded rather than fixed:

  • The key's INTERNAL comment still reads maas-virsh-pod (D-123 amendment) -- it was minted for the qemu+ssh pod attempt that was refuted the same day, and the per-machine power_type=virsh ruling correctly retained it. Rewriting a comment alters key material; the file name is now authoritative and the manifest comment carries the explanation.
  • Manifest source reads local to stay consistent with --render (which derives it and would otherwise raise an S3 drift FAIL); the true provenance is in the manifest comment, which S3 explicitly treats as out of scope. Writing voffice1:<path> there would have traded one finding for another, and writing a custody change would have created two NEW S5 asymmetries, since S5 compares custody.

Revert: rm ~/vr1-dc0-creds/vr1-dc0-maas-power_ed25519{,.pub} and drop the two rows + comment block from creds-manifests/vr1-dc0.manifest. The headend and snap copies were never modified, so power control is unaffected either way.

4. NetBox web-GUI admin password consolidated (SEC-025 OPENED)

~/vr1-office1-creds/netbox-admin-password (0600), byte-identical to office1-netbox:/root/netbox-secrets/admin.pass (sha256-verified both ends, 28 bytes, value never printed or echoed). Register changes: the netbox-admin netbox row custody off-manifest-known -> source-of-record; a NEW jumphost row with custody verbatim-copy (so tier-3 V1 now asserts the two copies stay byte-identical); a manifest row; an extended n-netbox-vm-secrets note.

This was a HUMAN GUI login that had never left the VM it was minted on -- the third instance of one miss class (SEC-009's api.token, SEC-020's admin.pass, now this). It produced no finding before this session because off-manifest-known asserts nothing about the jumphost.

SEC-025 is opened for the exposure the remediation CREATES (a plaintext human password at rest on the jumphost), not for the gap it closed -- the same shape as SEC-007 and SEC-020, with the same rotation obligation and the same both-copies-or-stale-trap hazard.

Revert: rm ~/vr1-office1-creds/netbox-admin-password; revert the matrix row pair, the manifest row, the note paragraph, and the SEC-025 ledger row. The VM copy is untouched.

5. V2 learns the ruled-deferral state, and gains its first harness cases

scripts/creds-matrix.py: load_sec_states now also reports whether a declared-dead SEC row carries an operator-ruled deferral; v2_declared_state emits V2 ACKNOWLEDGED DEFERRAL instead of a FAIL when it does.

WHY: SEC-006 has read "DEFERRED by operator ruling 2026-07-13: revoke at completion of this deployment" since long before this checker existed. V2 was reporting that standing operator ruling as a defect on every run -- which is both wrong and corrosive, because a finding that can never be actioned trains readers to skip findings. Reissuing the token would have contravened the ruling, so the register was the thing that needed to change, not the credential.

The acknowledgement is NOT a pass. It names the id, the SEC row, the state word and the ruling text, and states that the credential "is still live and still exposed until that ruling is discharged".

DEFERRAL_RE demands BOTH the emphasis-cased state word AND explicit attribution to an operator ruling (the GA-R5 bar), and [^|] stops it crossing a ledger table cell. A looser match would let prose that merely DISCUSSES deferring buy silence -- the false-green class this register exists to prevent. T58 and T59 are the regression locks for exactly that.

Also added --ledger (default unchanged). V2 read the SEC register via --repo, which the harness must point at the REAL repo for S4 mint-ref resolution -- so there was no fixture path to the ledger and V2 had shipped with ZERO harness cases. The override is what makes it testable; it also de-couples the existing tier-3 cases from the live ledger.

New cases T56-T59 (harness 56 -> 60): the founding invariant still fails; a ruled deferral is acknowledged and still says EXPOSED; unruled "deferred" prose still fails; a claim spanning two ledger cells does not count as ruled.

NOTE: tier 3 is NOT part of preflight P5 (which runs tier 1 + tier-2-local), so this does not change the blocking deploy gate's behaviour.

Revert: git revert this commit's changes to scripts/creds-matrix.py and tests/creds-matrix/run-tests.sh; V2 returns to failing on SEC-006 every run.

6. Permission allowlist widened for read-only commands (.claude/settings.json)

Transcript scan of the 44 most-recent session JSONLs across both project dirs, counting Bash leading command+subcommand pairs (and splitting each candidate into LOCAL vs nested-inside-ssh, since a command that only ever runs inside ssh host '...' gains nothing from its own rule -- the outer ssh is what the permission layer matches).

ADDED to permissions.allow only -- 13 entries, 31 -> 44. ask, deny, env and hooks untouched; nothing removed or reordered; JSON re-validated and de-duplicated:

bash scripts/creds-audit.sh *          (33)   bash scripts/dc-dc-ceph-disk-budget.sh * (13)
bash scripts/site-baseleg.sh check*     (5)   tofu validate* + tofu -chdir=* validate*  (25)
tofu state list* + -chdir variant      (16)   tofu state show*                           (3)
tofu providers schema*                 (13)   tofu version*                             (11)
ip route*                              (22)   systemctl is-active *                      (6)

DELIBERATELY NOT ADDED, and this is the substantive half of the item:

  • ssh (693 calls), scp (47), virsh (234) -- the three biggest prompt sources, left alone. They are this cloud's MUTATION TRANSPORT. CLAUDE.md hard rule 3 makes the ask rules the enforcement mechanism for per-mutation approval and says not to work around them, and the Bash(ssh *) disposition is an OPEN operator item queued at the 2026-07-23 close. Allowlisting any ssh shape would bypass the gate AND pre-empt an unresolved operator decision. This item does NOT discharge that queued prune -- it is still open.
  • python3 scripts/creds-matrix.py (76 local runs -- the largest genuine miss) -- skipped because Bash(*creds-matrix.py *--privileged*) is an ask rule added 2026-07-26 precisely after the auto-mode classifier incident. Every prefix pattern wide enough to cover the observed ... 2>&1 | tail forms would ALSO match --privileged, and allow-vs-ask precedence was not verified this session. Adding a rule that might silently downgrade an operator-installed gate is exactly the class of change hard rule 3 forbids. Needs the precedence question answered first, then a narrow set of EXACT forms.
  • tofu plan (54) -- read-only in effect, but CURRENT-STATE section 10 says in terms "Do NOT re-run tofu plan casually against live state". The prompt is deliberate friction.
  • python3 -c/heredocs (324) and shells: arbitrary code execution. curl (114): a mutation vector via -X/-d. tofu fmt (26): rewrites files. tofu init (10): mutates .terraform/. nft list (10): normally needs sudo, which correctly stays ask.

TWO OBSERVATIONS, no action taken (flagged, not fixed): several existing allow entries are redundant with Claude Code's built-in auto-allow (Bash(git status*), Bash(git log*), Bash(grep *), Bash(ls *), Bash(jq *), Bash(cat scripts/*)) -- harmless, and removing entries was out of scope; and Bash(git pull*) sits in allow although it is a fetch-and-merge that mutates the working tree.

Revert: drop the 13 appended lines from permissions.allow in .claude/settings.json. No other field was touched, so the revert is a contiguous deletion.

7. Stage 4 close-out opened: dc-mirror.sh check was FALSE-GREENING the mirror gate

Operator direction: close Stage 4 before resuming the DC1 Stage-5 chain. The Stage 4 DoD (runbooks/dc-dc-phase3-maas-enlist-deploy.md:472-485) has six bullets; four are already met and captured. Verifying the fifth -- "per-DC mirror reachable" -- found that the check the gate would close on could not fail.

scripts/dc-mirror.sh:271-273 tested [ -f last-sync.status ] and printed the file's contents behind an unconditional OK. So the status word was displayed but never asserted on, and the check exited PASS regardless. MEASURED on both racks, 2026-07-27:

  • dc0 read FAIL 2026-07-27T00:54:54Z ubuntu=255 uca=0 -> printed OK, check PASS.
  • dc1 read RUNNING 2026-07-23T21:49:35Z -- a marker FOUR DAYS stale, left by the debmirror the D-135 amendment killed with a signal, so it never wrote a terminal state -> printed OK, check PASS.

FIXED: the status word is now case-analysed. OK* passes; FAIL* misses; an ABSENT file misses (it was a passing note, but nothing has attested the content); an unrecognised word REFUSES rather than defaulting to success. RUNNING* is deliberately neither -- in-flight and died-mid-run are indistinguishable from the file alone, so it reports an explicit UNKNOWN and cross-checks the unit: a RUNNING marker with no active unit is named as a corpse, not a success. Harness tests/dc-mirror/run-tests.sh 24/24 (was 19); T20-T24 lock each status word, T24 specifically locks out the return of the exact defective shape.

PROOF THE FIX WORKS -- re-run against both live racks, and the gate is now honestly NOT MET: dc-mirror check (dc0): FAIL on the failed sync, dc-mirror check (dc1): FAIL on the stale marker. Capture docs/audit/stage4-mirror-gate-20260727.txt.

SUBSTANCE BEHIND THE TWO FAILURES (measured, not inferred):

  • dc0's mirror CONTENT is complete and intact -- 949G ubuntu + 342M cloud-archive, all three dists (jammy, jammy-updates, jammy-security) plus pool and project. What failed is the overnight INCREMENTAL: Download of dists/jammy/Release failed: 500 read timeout -> "Failed to download some Release, Release.gpg or InRelease files!" -> debmirror aborted the ubuntu leg with 255. The UCA leg succeeded (uca=0). A transient upstream fetch failure, not mirror damage. The daily timer re-arms for 2026-07-28 00:14:48.
  • dc1's debmirror is dormant only ACCIDENTALLY. dc1-mirror-sync.timer is enabled with an EMPTY NextElapseUSecRealtime, because the service sits in failed/Result=signal from the 07-24 pause. The D-135 amendment RULED it paused and "kept dormant as the fallback", but nothing enforces that -- the dormancy is a side effect of a failed unit, not a declared state, so a systemctl reset-failed (or possibly a reboot) re-arms a 330G -> ~949G pull on a rack whose ruled artifact path is now the proxy. dc1's PROXY meanwhile checks PASS genuinely: apt-cacher-ng active on .4:3142, serving both archive and UCA Release with 200.

Revert: git revert this commit's scripts/dc-mirror.sh + tests/dc-mirror/run-tests.sh changes -- the check returns to passing unconditionally on last-sync. No live state was altered by this item; both racks were read only.

Findings LOGGED, not actioned (hard rule 1)

  1. dc0 edge API credential (opnsense-api.txt) is absent from every DECLARED location -- not the jumphost folder, not the headend shadow store. It exists only inside the dc0 edge's own config. Recovery is a re-mint via scripts/opnsense-bootstrap-apikey.sh (the dc1-proven path; dc0's edge SSH key IS present on vcloud), which is a live mutation on a running edge and was explicitly OUT of this batch. This is the sole remaining SEC-021(a) item and accounts for 2 of the 7 residual findings.
  2. No registered root/console credential at EITHER DC edge. MEASURED: no matrix row, no manifest entry, no SEC row. office1-edge-root is cardinality=per-site and office1-only. NOT MEASURED, and deliberately not probed: what those root passwords actually are. The phase-2 runbook says DC edges boot on FACTORY DEFAULTS and the D-112(c) chain installs sshd + the service key without setting a root password -- but that is a repo reading, not a measurement (hard rule 2). Root SSH is key-only (proven), so SSH is not the vector; the web GUI is LAN-reachable at both edges and the serial console exists. Proposed as the next-free SEC row once the operator gates a probe.
  3. Two structural blind spots that let (2) hide. (a) S5 compares only cardinality=per-DC (creds-matrix.py:380); all SIX per-site rows are office1-only, so a per-site credential missing at two of three sites is invisible to every check. This is the "cardinality largely inert" item CURRENT-STATE already carries -- now with a concrete consequence. (b) creds-manifests/vm-secret-locations declares only jumphost, headend, netbox, while the checker also accepts rack, edge, cloud, unit, client -- all with zero declared locations, therefore unaudited. SEC-015 was a rack-resident edge-key copy, so the class is not hypothetical.
  4. The 3 residual S5 power-key asymmetries are RULED, not defects. SEC-016 designs dc0 to keep the snap's default identity while dc1 gets a named per-host key, so the filenames can never match and DC_TOKEN_RE cannot normalise them. The register needs a way to record a ruled exception; suppressing them silently would be wrong and renaming live files to satisfy a checker would be worse. Needs an operator decision on mechanism.
  5. dc0/dc1 creds folders carry no README while office1 does (creds-folder-readme, also per-site). Cosmetic; noted for symmetry.
  6. The headend shadow stores (SEC-022) are now redundant for dc0's power key -- vcloud holds the copy SEC-021(b) asked for. Deleting the shadow copy is a plausible follow-up but is destructive and out of scope.

Verification

  • tests/creds-matrix/run-tests.sh -- 60/60 PASS (was 56).
  • scripts/creds-audit.sh all three sites -- CLEAN (office1 14 entries, dc0 7, dc1 8).
  • creds-matrix.py --tier2 --tier3 --remote --privileged -- 82 rows, findings 13 -> 7. Cleared: S2 x2, E1 x3 (2 power-key + 1 dc1 .pub), V2 x1 (now an acknowledged deferral). Residual 7: dc0-edge-api (2, needs the excluded re-mint), S5 x3 (ruled, item 4 above), S6 conflation x1 (the SEC-020 defect, separate), E4 uncheckable x2 (Stage-5/6 rows). The register is NOT green and is not expected to be.
  • scripts/repo-lint.sh + full gauntlet: see the commit message for the recorded results.
  • .claude/settings.json (item 6): json.load parses, 44 allow entries, zero duplicates, ask 52 / deny 15 / env / hooks byte-unchanged.