Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / archive / session-20260726-d137-build.md
@JANeumatrix JANeumatrix 12 hours ago 7 KB SESSION CLOSE 2026-07-26 -- D-137 build + committee audit + remediation (GA-R4)
Raw Blame History

Session body 2026-07-26 -- D-137 credential matrix: build, committee audit, remediation

Full session record (GA-R4 rule 2). The bounded summary is in docs/session-ledger.md; status is in docs/CURRENT-STATE.md ONLY (GA-R1). Session changelog with per-item reverts: docs/changelog-20260726-d137-tier1.md (items 1-22 -- the filename says "tier1" and the content outgrew it; noted rather than renamed, since the name is cited in commits).

Branch dc-dc-stage4-phase3-maas-deploy, 17 commits af1b682..1696d08, all pushed. Opened at 8e72ff1. Stage 4 remains OPEN; no stage was closed by this session.

What was built

D-137 was ADOPTED but unimplemented at session open. All three tiers now exist.

  • Tier 1 STATIC -- creds-matrix.tsv (81 rows), creds-matrix-notes.md, scripts/creds-matrix.py (S1 schema / S2 manifest coverage both-bounds / S3 render drift / S4 mint-ref resolution / S5 per-DC symmetry / S6 ruling-5 principal invariant / S7 notes integrity), harness tests/creds-matrix.
  • Tier 2 EXISTENCE -- creds-manifests/vm-secret-locations (ruling 3's absolute bound), E0-E4 checks, --remote/--privileged with escalation retried only where needed.
  • Tier 3 VALIDITY -- V1 cross-copy sha256 provenance (external sha256sum both sides so only digests enter the process), V2 declared-state reconciliation, and scripts/creds-probe.sh for the behavioural half.
  • Enforcement -- preflight P5 runs tier 1 + tier 2 LOCAL, blocking, fail-closed. creds-manifests/stages-reached replaces --pending-stage, coupled to CURRENT-STATE by new repo-lint L12.
  • Schema -- 10 -> 12 columns (custody, notes-ref), rows re-keyed to (credential, location); custody later gained verbatim-copy for tier 3.

Harnesses: creds-matrix 56, creds-probe 9, creds-audit 15, preflight 10, repo-lint 47. Gauntlet 79 -> 81 ALL GREEN. repo-lint 0-fail throughout.

Operator rulings (GA-R5 form; utterances quoted in design-decisions.md D-137)

  1. "go with the 12-column amendment" -- schema amendment.
  2. "wire tier 1 as the blocking preflight Pn" -- ruling 1 enforcement, tier 1.
  3. Tier 2 gate shape: "Local-only blocking; remote operator-run"; stage input: "Declared file + lint coupling".
  4. Tier 3 boundary: "Full -- including OpenStack service credentials"; cadence: "Stage close
    • DC standup close". (I recommended passive-only and flagged the costs; ruled otherwise.)
  5. SEC-024: "chmod 600 both, now"; snapshots "Keep, decide at stage close". Ruling 4 (SEC-009 demotion) needed no exchange -- already ruled, trigger passed.

The committee, and what it found

Six independent read-only lenses (correctness, coverage, claim-verification, ruling-fidelity, record-integrity, Roosevelt-transfer). Four converged independently on the same root cause. Every defect was reproduced, not theorised.

Eight confirmed false greens in the delivered build, all since fixed and individually regression-locked. The most consequential: tier 2 keyed existence on host-role ALONE and never consulted site-key, so a credential present at one site satisfied the row for every site -- which had masked SEC-021's dc0 opnsense-api.txt on-disk absence while tier 1 flagged it. Also: probe_remote reporting UNREACHABLE for a location it had read; literal paths skipping absent/unreadable detection; an empty locations file bypassing ruling 3; no non-empty floor; a mint-ref crash exiting 1 indistinguishably from findings; mode enforcement custody-gated (43 of 77 rows unchecked); and S5's second direction untested.

~21 inferred filenames, not the six I had owned. Only the rows the first sweep physically touched had been re-measured. Octavia's 8 were invented AND its declared location matched none of them (the artifacts live in three subdirectories); vault used a directory name as a filename; the tenant rows lacked the <client>- prefix; .maas.cli was never measured at all and does not exist (the MAAS snap CLI profile is .maascli.db).

sec-ref mis-attribution was systemic. One instance was fixed by hand earlier in the session without asking whether the class recurred. It did.

Defects found in my own work by running it

Recorded because the pattern is the lesson: several defects were invisible to review and appeared only on contact with real hosts.

  • The first privileged sweep reported /root/maas-secrets as "does not exist". It exists -- an unprivileged [ -d ] on a root-owned parent is indistinguishable from absent.
  • Role-level gating manufactured false absences when one location was unprobed.
  • shlex.quote on the probe directory broke ~ expansion, so every home-relative remote location read as unreadable.
  • Tier 3's first draft reported provenance DRIFT on maas-region-admin and dc1-svc-key. Manual sha256sum showed both IDENTICAL. Reporting it would have raised a false credential-rotation alarm on exactly the SEC-020 pair. Root cause was a schema gap: nothing expressed which rows are byte-copies.
  • The first creds-probe.sh --run was a false green: ssh ate the while read loop's stdin, 1 of 13 probes ran, exit 0. Now locked by a total-accounting check.

Records corrected

CURRENT-STATE had asserted remote filenames and modes citing a checker VERDICT file that contained none of them (GA-R1 rule 2); it said the virsh gap was unfixed after it was fixed; it carried contradictory counts. The "ZERO undeclared files / fully accounted for" claim was WITHDRAWN as resting on the site-blind check, and "SEC-021's on-disk half REPRODUCED" was narrowed to the power-key artifacts -- then genuinely earned after the fix, when E1 dc0-edge-api 'opnsense-api.txt' appeared for the first time. The 12-column amendment was recorded on D-137, having existed only in a non-citable changelog.

Security ledger

  • SEC-024 OPENED then REMEDIATED (mode). terraform.tfstate.backup at 0664 carrying the MAAS API key in plaintext, found by the fixed checker on its first clean run. Severity was CORRECTED before remediation: opentofu/ is 0700 and ~ is 0750, so it was never effectively readable -- defence-in-depth, not live exposure. Retention of the two pre-* snapshots deferred to stage close.
  • V2 surfaced a standing defect: SEC-006 records the NetBox upstream token as BURNED since 2026-07-12 while the matrix carried it as a healthy consolidated row. Nothing had reconciled those surfaces.
  • Open SEC rows 19 -> 20.

Housekeeping

Permission rules: Bash(ssh * sudo *) never matched quoted remote commands, so remote sudo fell through to the classifier; quoted variants added for sudo and virsh, all ask. Session ledger rotated oldest-first 413 -> 280 (GA-R4 F1), discharging a debt the 07-25 close recorded as OWED.

Advice given, not built

Secrets-storage recommendation delivered on request (no HSM/KMS available): resolve the Vault-vs-OpenBao licensing fork first; build creds-mint.sh as an atomic mint->store->register; enable MAAS 3.7's Vault integration (measured disabled) rehearsed in VR1 because of the MAAS/Vault circular dependency on bare metal; Vault SSH CA to retire the static keypairs; Tang/Clevis + transit auto-unseal for the no-HSM bootstrap. Recommended DROPPING the sops/age stopgap as churn. None of this is built or ruled.

State at close

Stage 4 OPEN. The live mission did not move: the DC1 Stage-5 chain is exactly where it was at session open -- verify MAAS role tags live, the 10th controller VM, the dc0 VIP extract, the vault-VIP repair, BLOCKER-1/2 and the DC-aware gate fixes, then juju bootstrap with edge egress OPEN. This session hardened the credential control that will protect Stage 5; it did not advance Stage 5.

Preflight exits 1, with P5 among the reasons -- but it was ALREADY exiting 1 before P5 existed (P4: overlays/octavia-pki.yaml absent, MAAS unreachable from the jumphost). No deploy path that was open is closed by this session's work.