Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260725-maas-admin-recovery.md
@JANeumatrix JANeumatrix 14 hours ago 17 KB D-137: repo-carried build spec for the fresh-session handoff
Raw Blame History

Session changelog 2026-07-25 -- MAAS admin-account recovery + creds consolidation

Branch dc-dc-stage4-phase3-maas-deploy. Operator task: "complete the recovery of the MAAS admin accounts ... complete password resets, and save the creds in their proper folders so I can retrieve them." Status lives ONLY in docs/CURRENT-STATE.md (GA-R1); this changelog is the review surface -- each item states what / why / revert.

Measured evidence for every claim below: docs/audit/maas-admin-recovery-20260725.txt (42 lines, no secret values).

The task premise was FALSIFIED by measurement before any mutation. The session-ledger's 2026-07-25 "IMMEDIATE NEXT" asserted MAAS web-GUI login "does NOT exist ... never minted; no prior task did it". It did exist: the admin superuser's password was minted 2026-07-13 by scripts/site-headend-install.sh:452 and is MEASURED working (login POST 204 with the stored value, 400 with a wrong-password control; has_usable_password=True). It was merely root-only on voffice1 at /root/maas-secrets/admin.pass and never consolidated to ~/vr1-office1-creds/ -- the SEC-009 miss class, not a missing account. Only the ledger's juju-* clause was correct. That finding was reported and the scope RE-RULED by the operator before execution: "New account + consolidate only" -- no existing password rotated.

Item 1 -- NEW MAAS superuser operator (live region mutation, gated)

What: minted MAAS superuser operator / jesse.austin@neumatrix.com on the Office1 region for HUMAN GUI login. Password 32-char base64, generated ON vcloud straight into ~/vr1-office1-creds/maas-operator-password (0600), then fed to maas createadmin over stdin -- never in argv, never printed, never in context. Verified: account present + superuser=True; stored password authenticates (204) against a wrong-password control (400). Why: operator-ruled scope. Gives a human identity distinct from the service identity -- until now the only usable GUI login was admin, which is simultaneously the automation identity (item 6). Note maas changepassword has NO --password flag and createadmin --password would place the secret in argv (the shape site-headend-install.sh:452 uses); the stdin path satisfies the SEC-018 "never printed/echoed/in argv" discipline and is the transferable technique. Revert: ssh voffice1 'maas admin user delete operator' then rm ~/vr1-office1-creds/maas-operator-password and drop its manifest row (item 3).

Item 2 -- admin password CONSOLIDATED to the jumphost (no rotation)

What: copied voffice1:/root/maas-secrets/admin.pass -> ~/vr1-office1-creds/maas-admin-password (0600, 28 bytes). Byte-identical, sha256-verified both ends. Value moved over ssh into the file; never printed. Verified the CONSOLIDATED copy authenticates as admin (204). Why: closes the SEC-009 miss for this secret -- it is now retrievable from the operator's own folder instead of being root-only on a VM. Deliberately NOT rotated (operator ruling), so the VM copy REMAINS source-of-record and the vcloud file is a working copy; a future admin rotation must update BOTH in one operation or the VM copy becomes a stale trap. Recorded in SEC-020. Revert: rm ~/vr1-office1-creds/maas-admin-password + drop its manifest row. No loss -- the source-of-record copy on voffice1 was never modified.

Item 3 -- creds-manifests/vr1-office1.manifest: two new declared secrets

What: declared maas-admin-password (provenance voffice1:/root/maas-secrets/admin.pass -- the <vm>:<path> class the manifest format exists for) and maas-operator-password (local), with a header comment recording the mint mechanism, the byte format, and the still-un-consolidated residue (item 7). Why: the manifest is the forcing function -- an undeclared secret is exactly how the NetBox token went missing. Declaring makes creds-audit enforce presence+mode. Revert: git checkout HEAD -- creds-manifests/vr1-office1.manifest.

Item 4 -- dc0/dc1 manifest BACKFILL -> creds-audit CLEAN on all three sites

What: declared the four files that were failing creds-audit as UNDECLARED: dc0 maas-api-key.txt + vr1-dc0-edge_ed25519{,.pub}, dc1 maas-api-key.txt. Provenance local for all four (SEC-018/019 state the API keys were generated FROM the jumphost so they never persisted on the region; the edge keypair likewise). Why: creds-audit was RED on two of three sites at session open (dc0 3 fails, dc1 1 fail) -- the same convention the operator's task is about, so leaving the gate red would have been incoherent. The dc0 edge-keypair rows were a KNOWN queued finding, named in creds-manifests/vr1-dc1.manifest:18-20 ("dc0 precedent, whose manifest rows are a queued backfill finding") -- this closes it. Result: creds-audit CLEAN x3 (13/5/8 entries), harness 7/7. Revert: git checkout HEAD -- creds-manifests/vr1-dc0.manifest creds-manifests/vr1-dc1.manifest (returns both sites to FAIL).

Item 5 -- SEC-020 opened (register of record)

What: new row in docs/security-ledger.md covering (a) the un-consolidated VM-minted admin password + the falsified "no GUI login" claim, (b) the human/service identity conflation that CAUSED the miss, (c) the new operator standing credential. Status: OPEN -- rotation obligation on TWO region superusers, plus three logged-not- actioned residues. Open SEC rows 15 -> 16. Why: ledger mandate -- every credential exposure/obligation gets a row, never only a comment. A new standing credential is a new rotation obligation. Revert: delete the SEC-020 row.

Item 6 -- session-ledger: falsified claim corrected + machine block re-seeded

What: (a) struck the 2026-07-25 "IMMEDIATE NEXT" block and annotated it FALSIFIED + RESOLVED with what was actually true and what was done. (b) Re-seeded the machine-derived block from a fresh ledger-scan.sh: it was stale on EVERY line -- open SEC 12 -> 16, next-free D 133 -> 137 / DOCFIX 200 -> 204 / BUNDLEFIX 052 -> 053, and D-136 was missing from the open-decision list while D-071/D-129 were still listed despite being ADOPTED/RESOLVED. Why: (a) GA-R1 -- a falsified claim left standing misdirects the next session, and this one would have sent it to create an account that already existed. (b) my SEC-020 changes the count that block carries, so re-seeding is required by this change, not added scope; the block's own rule is "regenerate, never hand-edit". Revert: git checkout HEAD -- docs/session-ledger.md.

Item 7 -- Findings LOGGED, not executed (hard rule 1)

What: recorded in SEC-020, no action taken:

  • (i) voffice1:/root/maas-secrets/{admin.apikey,db.pass,lxd-trust.pass} remain un-consolidated. admin.apikey deliberately NOT copied -- trivially regenerable (maas apikey --username=admin) so a second copy is pure added surface; db.pass/lxd-trust.pass are region infrastructure, outside this task's scope.
  • (ii) the DURABLE fix for the miss class is a site-headend-install.sh amendment to consolidate at mint time (the script currently leaves all four secrets on the VM). Queued, unruled -- a script change needs its harness and a ruling.
  • (iii) the human/service conflation is proposable as a D-number, [ARCH] -- still UNASSIGNED after item 9 took D-137 for the credential pipeline; the two are kept separate deliberately (see D-137's "Related, deliberately NOT folded in"): admin is both the GUI login and the automation identity (its API key drives the maas admin CLI profile across 19 call sites in 4 scripts -- site-headend-install.sh 10, phase-04-network-verify.sh 5, phase-04-network-create.sh 3, phase-00-teardown-release.sh 1). Roosevelt-delta: per-identity MAAS separation / audit attribution on a multi-tenant cloud. NOT assigned -- GA-R3 doubt-resolves-DOWN, operator's call. Why: findings are logged, never executed mid-task. Revert: n/a (records only).

Item 8 -- evidence capture

What: docs/audit/maas-admin-recovery-20260725.txt -- generated, not hand-written: user list, Django password-state booleans, both login probes with wrong-password controls, the sha256 fidelity compare, file modes/sizes, creds-audit x3 + harness, and an automation-unaffected check. Machine-scanned for password bytes (0 hits) before commit. Why: GA-R1 rule 2 -- claims are quoted from captured command output, never prose. Revert: git rm docs/audit/maas-admin-recovery-20260725.txt.

Not done / still open

  • admin, juju-vr1-dc0, juju-vr1-dc1 passwords UNCHANGED by ruling. The two juju-* passwords remain random + unstored; their API keys are load-bearing for the blocked Stage-5 bootstrap and operator removes any need to log in as them.
  • MAAS and maas-init-node are MAAS-internal accounts -- deliberately untouched (maas-init-node is also is_active=False).
  • Items 7(i)-(iii) above.
  • Pre-existing, NOT touched by this session: CURRENT-STATE G14 SEC count (reads 12, actual 16) and section 4 ("RULED-BUT-NOT-BUILT", flagged comprehensively stale 2026-07-25) -- both already operator-deferred to Stage 4 close.

Item 9 -- D-137 PROPOSED: credential mint-and-consolidate pipeline [ARCH]

What: new docs/design-decisions.md entry D-137, PROPOSED (not ruled, not implemented). Answers the operator's follow-up ask -- a better minting/saving method plus "a durable rule to make sure when accounts are created there is a consolidation that happens every time". Records the measured diagnosis, a three-part proposal (the invariant; creds-mint.sh + creds-audit --all/--remote + gate wiring + site-headend-install.sh amendment; the Roosevelt delta), and THREE open forks. Section 8 of CURRENT-STATE gains item 9; the ledger's machine block goes 4 -> 5 open decisions. Why: the diagnosis is that the existing SEC-009 control cannot detect the failure class it was built for -- the audit is DECLARATION-based, so an undeclared secret is structurally invisible. creds-audit vr1-office1 reported CLEAN on 2026-07-15 while four secrets minted on the region VM on 2026-07-13 sat undeclared; admin.pass surfaced only 2026-07-25, twelve days later, and only because a task went looking (SEC-020). Compounding it: no --all (creds-audit.sh:25 takes one mandatory site), and the only enforcement is PROSE at runbooks/dc-dc-phase3-maas-enlist-deploy.md:498 -- absent from preflight/cloud-assert/repo-lint/gauntlet -- which demonstrably did not fire at either DC standup (dc0 3 + dc1 1 undeclared at this session's open). Admitted as a D under GA-R3 on the A1 test: Roosevelt mints per-DC IPMI/BMC and Vault material on real hardware, so a Roosevelt build session should grep this before creating any credential; today the rule lives only in docs/security-ledger.md, a register of exposures, which no gate can cite as authority. Deliberately NOT done: no tooling built, no enforcement wired, no SEC-009 demotion -- PROPOSED means present options, never implement (hard rule 1 + GA-R5). The SEC-020 identity-conflation question is kept SEPARATE to avoid batching two rulings. Revert: delete the D-137 block from docs/design-decisions.md, CURRENT-STATE section 8 item 9, and the ledger machine-block D-137 clause (restore count to 4).

Item 10 -- SEC-021/-022/-023 opened: live credential defects found during D-137 research

What: three new docs/security-ledger.md rows recording defects MEASURED read-only while researching the credential matrix. None actioned (hard rule 1). Open SEC rows 16 -> 19; CURRENT-STATE G14's evidence cell reconciled to 19 in this commit (its title text keeps the 07-23 figure and is marked superseded); ledger machine block re-seeded.

  • SEC-021 -- dc0 custody defects. (a) changelog-20260719-dc0-deploy-stepB.md:263 records the dc0 edge REST API credential written to ~/vr1-dc0-creds/opnsense-api.txt; a find over vcloud $HOME returns that filename for office1 and dc1 ONLY -- no dc0 copy -- and creds-audit vr1-dc0 reports CLEAN because it was never manifested. Severity is a custody/record defect + API-access gap, NOT a lockout: the dc0 edge SSH key is present and D-112(c) makes key-only SSH the primary management path. (b) dc0's power key is maas-virsh_ed25519 on voffice1 vs dc1's declared vr1-dc1-maas-power_ed25519{,.pub} on vcloud -- same purpose, different name, host and custody, dc0 with no .pub.
  • SEC-022 -- unaudited shadow stores. voffice1 holds ~/vr1-dc0-creds/ AND ~/vr1-dc1-creds/ with real key material. creds-audit.sh:29-31 resolves the folder under $HOME on whatever host it runs on (vcloud) and contains no ssh at all, so these are structurally invisible. A SCOPE gap in the control, not a one-off -- D-128 puts Plane-2 execution on voffice1, so headend-resident credentials will keep appearing.
  • SEC-023 -- sprawl blind spots + a PREDICTED Stage-5 exposure. The globs at creds-audit.sh:72-73 miss admin.pass, *.apikey, *.key, *.pem, *_ed25519 and even maas-api-key.txt. phase-03-admin-openrc.sh:32,72 will write OS_PASSWORD into $HOME/admin-openrc at Stage 5 -- home root, outside any creds folder, matching no glob. Recorded BEFORE it exists so the mint is not missed. Why: the ledger mandate is a row at discovery, never only a note. These are live defects independent of any D-137 ruling, and the approved plan lists recording them as the one action not gated behind the five open forks. They also constitute the empirical case for the matrix: SEC-021/-023 are absence-and-drift (invisible to discovery) while SEC-022 is an unknown-unknown (invisible to declaration) -- the two halves are complementary, which could not be argued from the discovery-only D-137. NOT done: no remediation, no tooling, no glob widening -- all coupled to unruled forks. Interim obligations are stated per row (notably: whoever runs Stage 5 phase-03 must consolidate ~/admin-openrc BY HAND at mint time). Revert: delete the three rows, restore the G14 evidence cell, re-seed the ledger block to 16.

Item 11 -- D-137 ADOPTED: all five sub-rulings ruled (GA-R5), implementation unblocked

What: the operator's follow-up ask ("better best practices method ... and a durable rule so credentials aren't misplaced or lost") was designed, committee-reviewed, and ruled. D-137 moves PROPOSED -> ADOPTED 2026-07-26. Five sub-rulings, each its own operator exchange, each committed+pushed BEFORE the next was asked (GA-R5), verbatim selections quoted in the Status block: (1) enforcement = "Blocking in preflight"; (2) derivation = "Derive manifests from matrix"; (3) remote scope = "Declared locations only"; (4) policy home = "D-137 is the authority"; (5) identity = "Fold in as a D-137 invariant". Commits a3a2d46, df8fa21, aa67634, db85531, and this one. Why the design changed: the operator identified the flaw in D-137-as-first-proposed -- a discovery sweep can never detect a credential that was NEVER MINTED, because absence is invisible to discovery. The known creation points support a forward EXPECTED-state register, which is the only instrument that catches a missed mint. Research (3 read-only agents) then produced four measured findings that validated it and one that qualified the premise: ssh-keygen returns ZERO hits repo-wide, so 12 declared secrets have no mint command anywhere -- recorded in docs/audit/creds-creation-points-20260725.md, and the reason mint-ref must admit an operator-terminal provenance kind. Committee (5 lenses) produced three amendments now folded into the ruled design: cardinality + host-role as first-class columns; manifests DERIVED not dual-maintained (the GA-F06 mitigation); and a VALIDITY tier distinct from existence -- prompted by finding that creds-audit parses provenance and NEVER verifies it, so the SEC-020 stale trap is unchecked today. NOT done: nothing built. No matrix file, no checker, no --render, no preflight Pn, no SEC-009 demotion. Implementation is unblocked but unstarted. Note the first run of the identity invariant is expected RED by design (admin serves both a human and a service row -- that IS the defect). Revert: flip the D-137 Status line back to PROPOSED and drop the five sub-ruling blocks; restore CURRENT-STATE item 9 and the ledger open-decision count to 5.

Item 12 -- D-137 build spec made repo-durable (handoff to a fresh session)

What: new docs/D-137-implementation-plan.md -- the executable build detail for the ADOPTED D-137: the five ruled decisions and their consequences, the matrix schema (logical keys only per the SEC-004 ruling), the checker model and precedents with file:line, the three tiers, the inherited hard constraints, the file list, the acceptance test, and an explicit "EXPECTED RED BY DESIGN" section. CURRENT-STATE item 9 and the ledger addendum now point at it. Also updated the creds-folder-convention auto-memory pointer: policy authority is now D-137 (ruling 4), with a verify-first caveat that creds-audit CLEAN is not evidence of completeness and the matrix is not built. Why: the design and committee review originated in a plan file under ~/.claude/plans/, which is OUTSIDE the repo -- so a fresh session bootstrapping via the GA-R4 path (CURRENT-STATE -> session-ledger -> ledger-scan) would never surface it, and the repo-is-authoritative rule would leave the build spec undiscoverable. Precedent for a D-NNN plan doc in-repo: docs/D-068-vault-migration-plan-draft.md. Operator directed the build to a fresh session, which makes this durability gap load-bearing rather than tidy. Note: the spec deliberately tells the next session NOT to make the first run green -- the acceptance test is that the checker REPRODUCES SEC-021/-022/-023 and the openrc case as named failures. A checker that passes on today's tree is wrong. Revert: git rm docs/D-137-implementation-plan.md and drop the two pointers.