Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260725-maas-admin-recovery.md
@JANeumatrix JANeumatrix 4 hours ago 11 KB D-137 PROPOSED: credential mint-and-consolidate pipeline -- make consolidation structural [ARCH]
Raw Blame History

Session changelog 2026-07-25 -- MAAS admin-account recovery + creds consolidation

Branch dc-dc-stage4-phase3-maas-deploy. Operator task: "complete the recovery of the MAAS admin accounts ... complete password resets, and save the creds in their proper folders so I can retrieve them." Status lives ONLY in docs/CURRENT-STATE.md (GA-R1); this changelog is the review surface -- each item states what / why / revert.

Measured evidence for every claim below: docs/audit/maas-admin-recovery-20260725.txt (42 lines, no secret values).

The task premise was FALSIFIED by measurement before any mutation. The session-ledger's 2026-07-25 "IMMEDIATE NEXT" asserted MAAS web-GUI login "does NOT exist ... never minted; no prior task did it". It did exist: the admin superuser's password was minted 2026-07-13 by scripts/site-headend-install.sh:452 and is MEASURED working (login POST 204 with the stored value, 400 with a wrong-password control; has_usable_password=True). It was merely root-only on voffice1 at /root/maas-secrets/admin.pass and never consolidated to ~/vr1-office1-creds/ -- the SEC-009 miss class, not a missing account. Only the ledger's juju-* clause was correct. That finding was reported and the scope RE-RULED by the operator before execution: "New account + consolidate only" -- no existing password rotated.

Item 1 -- NEW MAAS superuser operator (live region mutation, gated)

What: minted MAAS superuser operator / jesse.austin@neumatrix.com on the Office1 region for HUMAN GUI login. Password 32-char base64, generated ON vcloud straight into ~/vr1-office1-creds/maas-operator-password (0600), then fed to maas createadmin over stdin -- never in argv, never printed, never in context. Verified: account present + superuser=True; stored password authenticates (204) against a wrong-password control (400). Why: operator-ruled scope. Gives a human identity distinct from the service identity -- until now the only usable GUI login was admin, which is simultaneously the automation identity (item 6). Note maas changepassword has NO --password flag and createadmin --password would place the secret in argv (the shape site-headend-install.sh:452 uses); the stdin path satisfies the SEC-018 "never printed/echoed/in argv" discipline and is the transferable technique. Revert: ssh voffice1 'maas admin user delete operator' then rm ~/vr1-office1-creds/maas-operator-password and drop its manifest row (item 3).

Item 2 -- admin password CONSOLIDATED to the jumphost (no rotation)

What: copied voffice1:/root/maas-secrets/admin.pass -> ~/vr1-office1-creds/maas-admin-password (0600, 28 bytes). Byte-identical, sha256-verified both ends. Value moved over ssh into the file; never printed. Verified the CONSOLIDATED copy authenticates as admin (204). Why: closes the SEC-009 miss for this secret -- it is now retrievable from the operator's own folder instead of being root-only on a VM. Deliberately NOT rotated (operator ruling), so the VM copy REMAINS source-of-record and the vcloud file is a working copy; a future admin rotation must update BOTH in one operation or the VM copy becomes a stale trap. Recorded in SEC-020. Revert: rm ~/vr1-office1-creds/maas-admin-password + drop its manifest row. No loss -- the source-of-record copy on voffice1 was never modified.

Item 3 -- creds-manifests/vr1-office1.manifest: two new declared secrets

What: declared maas-admin-password (provenance voffice1:/root/maas-secrets/admin.pass -- the <vm>:<path> class the manifest format exists for) and maas-operator-password (local), with a header comment recording the mint mechanism, the byte format, and the still-un-consolidated residue (item 7). Why: the manifest is the forcing function -- an undeclared secret is exactly how the NetBox token went missing. Declaring makes creds-audit enforce presence+mode. Revert: git checkout HEAD -- creds-manifests/vr1-office1.manifest.

Item 4 -- dc0/dc1 manifest BACKFILL -> creds-audit CLEAN on all three sites

What: declared the four files that were failing creds-audit as UNDECLARED: dc0 maas-api-key.txt + vr1-dc0-edge_ed25519{,.pub}, dc1 maas-api-key.txt. Provenance local for all four (SEC-018/019 state the API keys were generated FROM the jumphost so they never persisted on the region; the edge keypair likewise). Why: creds-audit was RED on two of three sites at session open (dc0 3 fails, dc1 1 fail) -- the same convention the operator's task is about, so leaving the gate red would have been incoherent. The dc0 edge-keypair rows were a KNOWN queued finding, named in creds-manifests/vr1-dc1.manifest:18-20 ("dc0 precedent, whose manifest rows are a queued backfill finding") -- this closes it. Result: creds-audit CLEAN x3 (13/5/8 entries), harness 7/7. Revert: git checkout HEAD -- creds-manifests/vr1-dc0.manifest creds-manifests/vr1-dc1.manifest (returns both sites to FAIL).

Item 5 -- SEC-020 opened (register of record)

What: new row in docs/security-ledger.md covering (a) the un-consolidated VM-minted admin password + the falsified "no GUI login" claim, (b) the human/service identity conflation that CAUSED the miss, (c) the new operator standing credential. Status: OPEN -- rotation obligation on TWO region superusers, plus three logged-not- actioned residues. Open SEC rows 15 -> 16. Why: ledger mandate -- every credential exposure/obligation gets a row, never only a comment. A new standing credential is a new rotation obligation. Revert: delete the SEC-020 row.

Item 6 -- session-ledger: falsified claim corrected + machine block re-seeded

What: (a) struck the 2026-07-25 "IMMEDIATE NEXT" block and annotated it FALSIFIED + RESOLVED with what was actually true and what was done. (b) Re-seeded the machine-derived block from a fresh ledger-scan.sh: it was stale on EVERY line -- open SEC 12 -> 16, next-free D 133 -> 137 / DOCFIX 200 -> 204 / BUNDLEFIX 052 -> 053, and D-136 was missing from the open-decision list while D-071/D-129 were still listed despite being ADOPTED/RESOLVED. Why: (a) GA-R1 -- a falsified claim left standing misdirects the next session, and this one would have sent it to create an account that already existed. (b) my SEC-020 changes the count that block carries, so re-seeding is required by this change, not added scope; the block's own rule is "regenerate, never hand-edit". Revert: git checkout HEAD -- docs/session-ledger.md.

Item 7 -- Findings LOGGED, not executed (hard rule 1)

What: recorded in SEC-020, no action taken:

  • (i) voffice1:/root/maas-secrets/{admin.apikey,db.pass,lxd-trust.pass} remain un-consolidated. admin.apikey deliberately NOT copied -- trivially regenerable (maas apikey --username=admin) so a second copy is pure added surface; db.pass/lxd-trust.pass are region infrastructure, outside this task's scope.
  • (ii) the DURABLE fix for the miss class is a site-headend-install.sh amendment to consolidate at mint time (the script currently leaves all four secrets on the VM). Queued, unruled -- a script change needs its harness and a ruling.
  • (iii) the human/service conflation is proposable as a D-number, [ARCH] -- still UNASSIGNED after item 9 took D-137 for the credential pipeline; the two are kept separate deliberately (see D-137's "Related, deliberately NOT folded in"): admin is both the GUI login and the automation identity (its API key drives the maas admin CLI profile across 19 call sites in 4 scripts -- site-headend-install.sh 10, phase-04-network-verify.sh 5, phase-04-network-create.sh 3, phase-00-teardown-release.sh 1). Roosevelt-delta: per-identity MAAS separation / audit attribution on a multi-tenant cloud. NOT assigned -- GA-R3 doubt-resolves-DOWN, operator's call. Why: findings are logged, never executed mid-task. Revert: n/a (records only).

Item 8 -- evidence capture

What: docs/audit/maas-admin-recovery-20260725.txt -- generated, not hand-written: user list, Django password-state booleans, both login probes with wrong-password controls, the sha256 fidelity compare, file modes/sizes, creds-audit x3 + harness, and an automation-unaffected check. Machine-scanned for password bytes (0 hits) before commit. Why: GA-R1 rule 2 -- claims are quoted from captured command output, never prose. Revert: git rm docs/audit/maas-admin-recovery-20260725.txt.

Not done / still open

  • admin, juju-vr1-dc0, juju-vr1-dc1 passwords UNCHANGED by ruling. The two juju-* passwords remain random + unstored; their API keys are load-bearing for the blocked Stage-5 bootstrap and operator removes any need to log in as them.
  • MAAS and maas-init-node are MAAS-internal accounts -- deliberately untouched (maas-init-node is also is_active=False).
  • Items 7(i)-(iii) above.
  • Pre-existing, NOT touched by this session: CURRENT-STATE G14 SEC count (reads 12, actual 16) and section 4 ("RULED-BUT-NOT-BUILT", flagged comprehensively stale 2026-07-25) -- both already operator-deferred to Stage 4 close.

Item 9 -- D-137 PROPOSED: credential mint-and-consolidate pipeline [ARCH]

What: new docs/design-decisions.md entry D-137, PROPOSED (not ruled, not implemented). Answers the operator's follow-up ask -- a better minting/saving method plus "a durable rule to make sure when accounts are created there is a consolidation that happens every time". Records the measured diagnosis, a three-part proposal (the invariant; creds-mint.sh + creds-audit --all/--remote + gate wiring + site-headend-install.sh amendment; the Roosevelt delta), and THREE open forks. Section 8 of CURRENT-STATE gains item 9; the ledger's machine block goes 4 -> 5 open decisions. Why: the diagnosis is that the existing SEC-009 control cannot detect the failure class it was built for -- the audit is DECLARATION-based, so an undeclared secret is structurally invisible. creds-audit vr1-office1 reported CLEAN on 2026-07-15 while four secrets minted on the region VM on 2026-07-13 sat undeclared; admin.pass surfaced only 2026-07-25, twelve days later, and only because a task went looking (SEC-020). Compounding it: no --all (creds-audit.sh:25 takes one mandatory site), and the only enforcement is PROSE at runbooks/dc-dc-phase3-maas-enlist-deploy.md:498 -- absent from preflight/cloud-assert/repo-lint/gauntlet -- which demonstrably did not fire at either DC standup (dc0 3 + dc1 1 undeclared at this session's open). Admitted as a D under GA-R3 on the A1 test: Roosevelt mints per-DC IPMI/BMC and Vault material on real hardware, so a Roosevelt build session should grep this before creating any credential; today the rule lives only in docs/security-ledger.md, a register of exposures, which no gate can cite as authority. Deliberately NOT done: no tooling built, no enforcement wired, no SEC-009 demotion -- PROPOSED means present options, never implement (hard rule 1 + GA-R5). The SEC-020 identity-conflation question is kept SEPARATE to avoid batching two rulings. Revert: delete the D-137 block from docs/design-decisions.md, CURRENT-STATE section 8 item 9, and the ledger machine-block D-137 clause (restore count to 4).