diff --git a/creds-manifests/vm-secret-locations b/creds-manifests/vm-secret-locations index b35685c..16cb6e1 100644 --- a/creds-manifests/vm-secret-locations +++ b/creds-manifests/vm-secret-locations @@ -120,3 +120,7 @@ region vr1-dc0 ssh vr1-dc0-maas:~/.maas-admin-pw region vr1-dc1 ssh vr1-dc1-maas:~/.maas-db-pw region vr1-dc1 ssh vr1-dc1-maas:~/.maas-admin-pw +rack vr1-dc0 ssh vr1-dc0-rack:~/vr1-dc0-creds/* +rack vr1-dc0 ssh vr1-dc0-rack:~/.local/share/juju/* +rack vr1-dc1 ssh vr1-dc1-rack:~/vr1-dc1-creds/* +rack vr1-dc1 ssh vr1-dc1-rack:~/.local/share/juju/* diff --git a/creds-manifests/vr1-dc0.manifest b/creds-manifests/vr1-dc0.manifest index 0799144..c228591 100644 --- a/creds-manifests/vr1-dc0.manifest +++ b/creds-manifests/vr1-dc0.manifest @@ -66,3 +66,8 @@ maas-region-db-password 600 PENDING-TIER2 maas-region-admin-password 600 PENDING-TIER2 maas-region-api-key.txt 600 local +# Per-DC JUJU SERVICE CREDENTIAL (minted 2026-07-30; D-132 q1 + D-138 consequence). +# `juju-vr1-dc0` in the DC's OWN MAAS region -- the Office1 user/key of the same name +# authenticates against a DIFFERENT database and is NOT this credential. +maas-juju-api-key.txt 600 local +maas-juju-user-password 600 local diff --git a/creds-matrix-notes.md b/creds-matrix-notes.md index a2a5ab4..0804489 100644 --- a/creds-matrix-notes.md +++ b/creds-matrix-notes.md @@ -432,3 +432,18 @@ If any of these later becomes materialized to a file, it acquires a row at that moment -- adding one is DoD for the change that materializes it. + +## n-juju-region-cred +The per-DC Juju service credential (`juju-` in that DC's OWN MAAS region). MINTED +2026-07-30 for dc0 as a direct consequence of D-132 q1: the regions hold separate +databases, so the Office1 `juju-vr1-dc0` user and its key do NOT authenticate against the +per-DC region. Office1's copies are NOT this credential and must not be conflated with it. + +The RACK copy exists because D-138 moved the cloud-facing Juju client into the DC. That +made `rack` a credential-bearing host-role for the first time, which is why +`vm-secret-locations` gained its first `rack` rows in the same change -- an unlisted +location is not audited, which is how SEC-022 happened. + +dc1's four rows are the FORWARD REGISTER and are EXPECTED to fail S2 until dc1's region is +built and its own user minted. That failure is the register working as designed (D-137); +do NOT delete the rows to make the checker green. diff --git a/creds-matrix.tsv b/creds-matrix.tsv index 37eb49c..bb5d9d2 100644 --- a/creds-matrix.tsv +++ b/creds-matrix.tsv @@ -95,6 +95,20 @@ dc0-region-admin per-DC vr1-dc0 jumphost maas-region-admin-password gui human verbatim-copy stage5 operator-terminal SEC-027 - dc0-region-apikey per-DC vr1-dc0 jumphost maas-region-api-key.txt api service consolidated stage5 operator-terminal SEC-027 - +# per-DC JUJU SERVICE CREDENTIAL (D-132 q1 consequence, minted 2026-07-30). Juju +# authenticates to its DC's OWN region as `juju-`; the Office1 key does NOT +# authenticate against a separate database. The CLIENT copy lives on the RACK because +# D-138 moved the cloud-facing client into the DC -- a host-role that had no rows here +# before, and an unlisted location is not audited (SEC-022). +dc0-juju-apikey per-DC vr1-dc0 jumphost maas-juju-api-key.txt api service consolidated stage5 runbook:runbooks/dc-dc-phase4-juju-bundle-per-dc.md:236 SEC-028 n-juju-region-cred +dc0-juju-apikey per-DC vr1-dc0 rack maas-juju-api-key.txt api service off-manifest-known stage5 runbook:runbooks/dc-dc-phase4-juju-bundle-per-dc.md:236 SEC-028 n-juju-region-cred +dc0-juju-apikey per-DC vr1-dc0 rack credentials.yaml api service off-manifest-known stage5 runbook:runbooks/dc-dc-phase4-juju-bundle-per-dc.md:236 SEC-028 n-juju-cred-store +dc0-juju-user per-DC vr1-dc0 jumphost maas-juju-user-password gui human consolidated stage5 runbook:runbooks/dc-dc-phase4-juju-bundle-per-dc.md:236 SEC-028 n-juju-region-cred +dc1-juju-apikey per-DC vr1-dc1 jumphost maas-juju-api-key.txt api service consolidated stage5 runbook:runbooks/dc-dc-phase4-juju-bundle-per-dc.md:236 SEC-028 n-juju-region-cred +dc1-juju-apikey per-DC vr1-dc1 rack maas-juju-api-key.txt api service off-manifest-known stage5 runbook:runbooks/dc-dc-phase4-juju-bundle-per-dc.md:236 SEC-028 n-juju-region-cred +dc1-juju-apikey per-DC vr1-dc1 rack credentials.yaml api service off-manifest-known stage5 runbook:runbooks/dc-dc-phase4-juju-bundle-per-dc.md:236 SEC-028 n-juju-cred-store +dc1-juju-user per-DC vr1-dc1 jumphost maas-juju-user-password gui human consolidated stage5 runbook:runbooks/dc-dc-phase4-juju-bundle-per-dc.md:236 SEC-028 n-juju-region-cred + # ---------------------------------------------------------------- vr1-dc1 dc1-svc-key per-DC vr1-dc1 jumphost vr1-dc1_svc_ed25519 ssh service consolidated stage3 operator-terminal - n-no-mint-command dc1-svc-key per-DC vr1-dc1 jumphost vr1-dc1_svc_ed25519.pub none service consolidated stage3 operator-terminal - n-no-mint-command diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 51341cd..b4dfebc 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -859,10 +859,63 @@ databases. The alternative -- pointing juju at the region ADMIN key already consolidated at `~/vr1-dc0-creds/maas-region-api-key.txt` on vcloud -- is strictly worse: it puts an admin-scoped credential on the rack, which is the residency SEC-026 exists to constrain. - **NOTHING WAS MINTED.** The call was refused by the harness and NOT retried in an altered - shape; the refusal was a permission rule failing to MATCH (`Bash(ssh vr1-dc0-rack *)` is - allowed, there is no `vr1-dc0-maas` rule) rather than a classifier fault -- the same - standing lesson as the 2026-07-30 DHCP cutover. + The first attempt was refused by the harness and NOT retried in an altered shape; the + refusal was a permission rule failing to MATCH (`Bash(ssh vr1-dc0-rack *)` is allowed, + there was no `vr1-dc0-maas` rule) rather than a classifier fault -- the same standing + lesson as the 2026-07-30 DHCP cutover. + **^ RULED AND EXECUTED 2026-07-30 (GA-R5). Question as presented: the new dc0 region has + no `juju-vr1-dc0` user so juju cannot authenticate against it; mint a dedicated user + (mirrors Office1, DC-local blast radius, but a NEW key since the databases are separate), + reuse the region admin key (no new secret but admin-scoped ON THE RACK), or stop. + Operator answer, exact utterance: "Mint juju-vr1-dc0 on the new region (Recommended)".** + A second exchange on the permission wall: options were a targeted `ask` rule, a broad + `allow`, or no change. **Operator answer, exact utterance: "Add it to allow".** + `Bash(ssh vr1-dc0-maas *)` was added to `allow` in the GITIGNORED + `.claude/settings.local.json` -- recorded here because F1 established that rules living + only in that file are lost on a rebuild. + **MINTED AND PROVEN 2026-07-30. `juju-vr1-dc0` exists in the dc0 region as a SUPERUSER** + (mirroring the MEASURED Office1 shape -- MAAS 3.7 needs it for machine allocate/deploy; + it was not "improved" to a lesser role). Key is 3-part, single-line, 71 bytes. Minted ON + the region VM, never printed, moved region-VM -> vcloud -> rack with **all three sha256 + digests compared and equal**, staging `shred`ded, `0600` in a `0700` folder read back at + the destination. **THE KEY WAS PROVEN TO AUTHENTICATE BEFORE ANY BOOTSTRAP** -- a scoped + `maas login` + `users read` (4 users) + `machines read` (**10 total, 10 Ready**) + + `rack-controllers read` returning `[hot-kid]`, which proves DC-LOCAL region identity, then + logout. A credential that EXISTS is not a credential that WORKS. + **JUJU CLOUD RE-POINTED AND READ BACK: `endpoint: http://10.12.8.6:5240/MAAS`** (was + `10.10.0.20` = Office1, which is exactly what killed bootstrap 3). The stale + Office1-keyed `vr1-dc0-cred` was REMOVED before the new one was added under the same + name -- leaving both would give the cloud `credential-count: 2` and juju may pick either. + **TWO SNAP-CONFINEMENT TRAPS MEASURED, each costing an attempt, neither naming + confinement in its error:** (a) `juju` is a SNAP with a PRIVATE `/tmp`, so a YAML the + shell writes to `/tmp` is INVISIBLE to it -- `no such file or directory` on a path that + demonstrably exists; (b) the snap `home` interface grants access to NON-HIDDEN files + only, so staging under `$HOME/.juju-stage.XXX` fails `permission denied` on a directory + the caller owns. Stage under a plain `$HOME/`. Both are now in the phase-4 runbook. + **DOCFIX-206 -- A CONSEQUENTIAL RUNBOOK DEFECT FOUND BY HITTING IT.** Step 2.0's + credential gate is NOT REGION-SCOPED: it says "credential LISTED and MAAS user exists -> + SKIP the mint". Under D-132 q1 all three of its checks passed while `juju-vr1-dc0` lived + in OFFICE1, so the SKIP branch would have gone straight to a bootstrap that fails on + authentication and reads like a network fault. Fixed: every check must name the region + and prove it with `maas-profile-assert.sh` first, a fourth outcome was added for + "credential listed but the user is in another region" (STALE -> remove and re-mint), the + mint location was corrected from `voffice1` to the DC's OWN region VM, and the endpoint + re-point + the authenticate-before-bootstrap proof were added. + **REGISTERED BEFORE BOOTSTRAP, not after (SEC-022's lesson):** **SEC-028** opened (open + SEC 23 -> 24); 8 `creds-matrix.tsv` rows (4 per DC); the dc0 manifest declares its two; + and `vm-secret-locations` gained its **FIRST `rack` rows** (4), because D-138 made the + rack a credential-bearing host-role for the first time. Harness `tests/creds-matrix` + **65/65 PASS**, repo-lint 0 fail. **dc1's four rows FAIL S2 BY DESIGN** -- the forward + register making dc1's absence detectable. **THOSE TWO NEW dc1 FINDINGS ARE NOT COVERED BY + THE OPERATOR'S P5 ACCEPTANCE**, which enumerated six and said so explicitly; they are + flagged here rather than absorbed. P5 now reports 10 findings (the 6 accepted + dc1's + 3 SEC-027 region rows + ... see the matrix output; all by-design forward-register or + pre-existing). + **PRE-EXISTING AND NOT FIXED (hard rule 1):** `creds-audit vr1-dc0` reports 2 problems -- + the manifest carries a literal `vr1-dc0-octavia-pki-.tar.gz` placeholder that + cannot match the real timestamped archive, so it reads as one MISSING + one UNDECLARED. + Predates this session and is not deploy-blocking. The two credentials added here are NOT + among its findings. - Project: Omega Cloud, VR1 DC-DC rehearsal -- a two-DC + Office1-headend virtual rehearsal on KVM (vcloud host), rehearsing the future bare-metal diff --git a/docs/changelog-20260730-dc0-node-carve.md b/docs/changelog-20260730-dc0-node-carve.md index 5b228be..009df04 100644 --- a/docs/changelog-20260730-dc0-node-carve.md +++ b/docs/changelog-20260730-dc0-node-carve.md @@ -211,3 +211,59 @@ entry for the rack, the dc0 manifest declaration, and a new SEC row (next free SEC-028). **Revert.** Nothing to revert -- no mutation was made. + +## Item 6 -- per-DC Juju service credential MINTED, cloud re-pointed (SEC-028) + +**Operator rulings (GA-R5, one exchange each, quoted).** Credential shape: *"Mint +juju-vr1-dc0 on the new region (Recommended)"*. Permission wall: *"Add it to allow"*. + +**What.** `juju-vr1-dc0` created in the dc0 region as a SUPERUSER -- mirroring the MEASURED +Office1 shape, since MAAS 3.7 needs it for machine allocate/deploy. It was NOT "improved" +to a lesser role; this is re-establishing an as-built shape, not designing a new one. + +**Custody.** Minted ON the region VM, never printed. Moved region-VM -> vcloud -> rack with +all three sha256 digests compared and equal, staging `shred`ded, `0600` inside a `0700` +folder read back at the destination. Only byte counts and part counts ever reached the +transcript. + +**PROVEN TO AUTHENTICATE BEFORE ANY BOOTSTRAP.** Bootstrap is one-shot with no `--dry-run`, +so a scoped `maas login` + `users read` + `machines read` (10 total, 10 Ready) + +`rack-controllers read` -> `[hot-kid]` + logout proved authentication, superuser scope, AND +DC-local region identity in four read-only calls. A credential that EXISTS is not a +credential that WORKS. + +**Cloud re-pointed and read back:** `endpoint: http://10.12.8.6:5240/MAAS`. The stale +Office1-keyed `vr1-dc0-cred` was REMOVED before the new one was added under the same name -- +leaving both gives the cloud `credential-count: 2` and juju may pick either. + +**TWO SNAP-CONFINEMENT TRAPS, each cost an attempt, neither error names confinement.** +(a) `juju` is a snap with a PRIVATE `/tmp`: a YAML written there by the shell is INVISIBLE +to it, reported as `no such file or directory` for a path that demonstrably exists. +(b) The snap `home` interface grants access to NON-HIDDEN files only, so `$HOME/.stage/...` +fails `permission denied` on a directory the caller owns. Both now in the phase-4 runbook. + +**Revert.** `juju remove-credential vr1-maas vr1-dc0-cred --client`; restore the endpoint +with `juju update-cloud`; `maas vr1-dc0-region users delete juju-vr1-dc0`; remove the two +files from both creds folders and their matrix/manifest rows. + +## Item 7 -- DOCFIX-206: the Step 2.0 credential gate was not region-scoped + +**What.** `runbooks/dc-dc-phase4-juju-bundle-per-dc.md` Step 2.0 says "credential LISTED and +MAAS user exists -> SKIP the mint". Under D-132 q1 that is WRONG, and I hit it: `juju +credentials --client` listed `vr1-dc0-cred`, `~/vr1-dc0-creds` existed, and `juju-vr1-dc0` +existed -- **in OFFICE1**. The new region held only `MAAS`, `admin`, `maas-init-node`. + +**Why it matters.** The SKIP branch leads straight to a bootstrap that fails on +authentication, which presents as a network fault -- the same misleading signature that +consumed three attempts on 2026-07-30. The regions hold SEPARATE DATABASES: a user in one +is not a user in the other, and a credential's NAME says nothing about which region it +authenticates against. + +**Fixed.** Every check in the block must name the region and prove it with +`maas-profile-assert.sh` first; a FOURTH gate outcome was added for "listed but the user is +in another region" (STALE -> remove and re-mint, never "make it consistent"); the mint +location was corrected from `voffice1` to the DC's OWN region VM; and the endpoint re-point, +the two snap traps, the printf-not-heredoc rule and the authenticate-before-bootstrap proof +were all added. + +**Revert.** `git checkout ^ -- runbooks/dc-dc-phase4-juju-bundle-per-dc.md`. diff --git a/docs/security-ledger.md b/docs/security-ledger.md index 5e23ffe..a02b009 100644 --- a/docs/security-ledger.md +++ b/docs/security-ledger.md @@ -78,3 +78,4 @@ before it. | SEC-026 | 2026-07-30 | **A MAAS admin-scoped API key becomes RESIDENT on a DC-local host, by D-138.** D-138 (RULED 2026-07-30) moves the cloud-facing client -- the Juju client, and `phase-03`..`phase-06`'s `openstack` CLI work -- INTO each DC, because `voffice1` has no L3 path to any DC node plane and SEC-010/D-052/D-125 deliberately forbid creating one. The client needs its DC's MAAS credential where it runs, so `juju-vr1-dc0`'s key (SEC-018) now lives on the dc0 client host, and `juju-vr1-dc1`'s (SEC-019) will on dc1's. **The exposure is wider than the DC it sits in: the key is MAAS ADMIN-scoped (full machine/power/deploy) and the MAAS region is SHARED by BOTH DCs**, so a host inside dc0 holding it has region-wide MAAS admin, including over dc1's hardware. This is a NEW residency, not a new key -- SEC-018/-019 already carry the keys' own rotation obligations. | 2026-07-30 Stage-5 bootstrap failure + D-138 ruling; capture `docs/audit/stage5-bootstrap-reachability-20260730.txt` | operator | **OPEN.** (1) **ISOLATION IS THE LOAD-BEARING CONTROL:** each DC's client host receives ONLY THAT DC's credential. Never copy the whole Juju client credential store (`~/.local/share/juju/credentials.yaml` on voffice1 holds BOTH `vr1-dc0-cred` and `vr1-dc1-cred`) to a DC host -- that single act would put dc1's credential on dc0's rack and destroy the per-DC isolation SEC-018/-019 exist to create. Verify after any client rebuild that `juju credentials --client` on a DC host lists exactly one. (2) ROTATE at v1 close, or immediately if the DC client host is rebuilt or shared. (3) SCOPE HARDENING CANDIDATE, Roosevelt-relevant: MAAS has no per-DC admin scope, so the blast radius is regional by construction; a narrower grant (or a per-DC MAAS region) is the fix and is deferred, not solved. (4) The residency must be re-asserted at every DC standup as part of D-138's definition-of-done. | | SEC-027 | 2026-07-30 | **Per-DC MAAS region controller credentials -- a NEW credential class, three secrets per DC.** D-132's amendment (RULED 2026-07-30) gives each DC its own MAAS region controller VM at utility-band `.6`, running its own `regiond` + PostgreSQL. Each region therefore mints and holds: (a) a PostgreSQL role password for the `maas` DB role, (b) a MAAS region `admin` account password (a HUMAN login), and (c) that admin's MAAS API key (3-part, service). dc0's are MINTED and consolidated to `~/vr1-dc0-creds/` (`maas-region-db-password`, `maas-region-admin-password`, `maas-region-api-key.txt`, all 0600, sha256-verified identical to the on-VM source, never printed). dc1's are NOT YET MINTED -- its region VM is authored but not applied. **These are NOT the Office1 region's credentials and must never be conflated with them**: the entire point of the per-DC region is that each DC's provider API is independent of Office1. | 2026-07-30 Stage-5 region build; D-132 amendment + addendum; capture `docs/audit/stage5-bootstrap-reachability-20260730.txt` | operator | **OPEN -- rotation obligation + two structural notes.** (1) ROTATE all three at v1 close, or immediately if the region VM is rebuilt or shared. (2) **RESIDUAL ARGV EXPOSURE, recorded rather than hidden:** `maas init` offers NO stdin or file option for `--database-uri`, so the DB password is briefly visible in that host's own `ps` output at init time and is then written to `/var/snap/maas/current/regiond.conf`. The URI was assembled inside the remote shell from a 0600 file so it never reached the operator terminal or the session transcript, but the argv exposure on the VM itself is unavoidable with this tool. Identical in shape to the existing Office1 region -- not a new exposure class. (3) **PROVENANCE DEBT, deliberate:** all five matrix rows carry `mint-ref=operator-terminal` because NO repo script mints these yet. That is truthful, not an oversight -- writing the region install into a runbook or script is what clears it, and until then these are not reproducible from the repo if the jumphost is rebuilt (research FINDING 1 class). (4) The `region` host-role token was added to `scripts/creds-matrix.py`'s enum for this class; `creds-manifests/vm-secret-locations` gained four rows, which is definition-of-done for a new mint site (an unlisted location is not audited -- how SEC-022 happened). | +| SEC-028 | 2026-07-30 | **Per-DC JUJU SERVICE credential, and the RACK becomes a credential-bearing host for the first time.** D-132 q1 gives each DC its own MAAS region with its own database, so the Office1 `juju-vr1-dc0` user and its API key do NOT authenticate against dc0's region -- a fact that reads as a network fault at bootstrap rather than as an auth failure. dc0's `juju-vr1-dc0` was therefore MINTED IN THE NEW REGION 2026-07-30 (superuser, mirroring the measured Office1 shape, which MAAS 3.7 requires for machine allocate/deploy) together with its 3-part API key and account password. Minted ON the region VM, never printed, moved host-to-host with all three sha256 digests compared, staging shredded. Consolidated to `~/vr1-dc0-creds/` on the jumphost (0600) and distributed to the dc0 RACK (0600 in a 0700 folder) because **D-138 moved the cloud-facing Juju client INTO the DC** -- so `rack` is now a credential-bearing host-role, which it had never been. dc1's is NOT YET MINTED (its region VM is authored, not applied). | 2026-07-30 Stage-5 Juju re-point; D-132 q1 + D-138; captures in `docs/audit/dc0-node-carve-20260730.txt` and this session's changelog | operator ("Mint juju-vr1-dc0 on the new region (Recommended)") | **OPEN -- rotation obligation + three notes.** (1) ROTATE at v1 close, or immediately if the rack or region VM is rebuilt or shared. (2) **BLAST RADIUS IS DC-LOCAL BY CONSTRUCTION and this is strictly BETTER than the alternative considered:** the rejected option was pointing juju at the region ADMIN key already consolidated on vcloud, which would have placed an admin-scoped credential on the rack -- the residency SEC-026 exists to constrain. This key administers dc0's region ONLY; no cross-DC reuse, per SEC-012/-016/-018/-019. (3) **`vm-secret-locations` gained its FIRST `rack` rows** (four, both DCs) -- definition-of-done for a new mint site, since an unlisted location is not audited (how SEC-022 happened). It also now covers the rack's `~/.local/share/juju/*` credential store, where juju keeps its own runtime copy. (4) dc1's four matrix rows are the FORWARD REGISTER and FAIL S2 by design until dc1's region exists -- that failure is D-137 working, and the rows must NOT be deleted to go green. **These two NEW S2 findings are NOT covered by the operator's 2026-07-30 P5 acceptance, which enumerated six and said so explicitly.** | diff --git a/runbooks/dc-dc-phase4-juju-bundle-per-dc.md b/runbooks/dc-dc-phase4-juju-bundle-per-dc.md index c392461..745edbd 100644 --- a/runbooks/dc-dc-phase4-juju-bundle-per-dc.md +++ b/runbooks/dc-dc-phase4-juju-bundle-per-dc.md @@ -220,9 +220,32 @@ (`maas users read` is the enumeration used to account for every MAAS account on this deployment -- `docs/CURRENT-STATE.md:2784`. Do NOT open anything inside the creds folder; its existence is all this gate needs.) +> **DOCFIX-206, 2026-07-30 -- THE CHECK ABOVE IS NOT REGION-SCOPED, AND UNDER +> D-132 q1 THAT MAKES IT WRONG.** Once each DC has its OWN MAAS region, all +> three commands can report "already exists" while the user exists in the +> WRONG region. MEASURED on the dc0 migration: `juju credentials --client` +> listed `vr1-dc0-cred`, `~/vr1-dc0-creds` was present, and `juju-vr1-dc0` +> existed -- **in OFFICE1**. The new region held only `MAAS`, `admin`, +> `maas-init-node`. Following the SKIP branch would have gone straight to a +> bootstrap that fails on authentication, which reads like a network fault. +> **The regions hold SEPARATE DATABASES, so a user in one is not a user in the +> other, and a key minted in one will NEVER authenticate against the other.** +> Every command in the CHECK block must therefore name the region explicitly: +> run `maas users read` with the PER-DC profile (and prove which +> region that profile resolves to with `scripts/maas-profile-assert.sh +> ` FIRST -- a machine count is not proof), and read the +> cloud's endpoint with `juju show-cloud vr1-maas --client` rather than +> trusting the credential's NAME. A credential's name says nothing about which +> region it authenticates against. + **GATE -- three outcomes, and only one of them proceeds:** -- This DC's credential is LISTED and its MAAS user exists -> **SKIP the rest of +- This DC's credential is LISTED **and its MAAS user exists IN THIS DC'S OWN + REGION** (proven by profile assert, not by name) -> **SKIP the rest of Step 2.0 entirely** and go to the bootstrap CHECK below. Do not re-mint. +- The credential is listed but the user lives in ANOTHER region -> the + credential is STALE, not present: remove it (`juju remove-credential + vr1-maas --client`) and mint into the correct region. Leaving both + would give the cloud `credential-count: 2` and juju may pick either. - Neither exists -> run the mint blocks that follow, and open the per-DC SEC row. - One exists without the other (a user with no registered credential, or a credential naming a user MAAS does not have) -> **STOP.** That is a half-made @@ -231,7 +254,9 @@ missing half. The mint blocks below run ONLY on the second outcome. On the MAAS region host -(VR1: `voffice1`): +-- **which under D-132 q1 is that DC's OWN region VM (`vr1-dc-maas`, the +D-134 `.6`), NOT `voffice1`.** `voffice1` is the OFFICE1 region and minting +there produces a user the DC's own region has never heard of: ```bash # service account -- Juju auths via the API key, not this password; use a random one sudo maas createadmin --username="juju-$DC" --email="juju-$DC@" @@ -251,6 +276,35 @@ Register the MAAS cloud + this credential in Juju consuming the key BY PATH/stdin (never argv). Each key is a rotation obligation -- open a per-DC SEC row when issued. +> **RE-POINT THE CLOUD ENDPOINT TOO, and read it back.** A new credential against +> a cloud still pointing at the old region fixes nothing: `jujud` dials the region +> API for its whole life, and an Office1 endpoint is what killed bootstrap attempt +> 3 on 2026-07-30. `juju update-cloud vr1-maas --client -f ` with +> `endpoint: http://:5240/MAAS`, then **read it back** with +> `juju show-cloud vr1-maas --client` -- a command that returns is not evidence. +> +> **TWO SNAP-CONFINEMENT TRAPS, both measured 2026-07-30, both costing an attempt.** +> `juju` is a SNAP, so the YAML file you pass to `update-cloud` / `add-credential`: +> (a) must NOT live in `/tmp` -- a snap has a PRIVATE `/tmp` namespace, so a file +> the shell writes there is INVISIBLE to juju, which reports `no such file or +> directory` for a path that demonstrably exists; and (b) must NOT be under a +> DOT-directory in `$HOME` -- the snap `home` interface grants access to +> NON-HIDDEN files only, so `$HOME/.stage/cloud.yaml` fails `permission denied` +> on a directory the caller owns. Stage under a plain `$HOME/` directory. +> Two different error messages, one cause; neither names confinement. +> +> **Build the YAML with `printf`, one line per statement, values passed as `%s` +> ARGUMENTS** -- not a heredoc. A heredoc's correctness depends on invisible +> leading whitespace on a line nobody reads, and this repo has two separate +> defects (F8, F9) from exactly that. +> +> **PROVE THE KEY AUTHENTICATES BEFORE BOOTSTRAPPING.** Bootstrap is one-shot and +> has no `--dry-run`. A scoped `maas login "$(cat )"` +> followed by `users read`, `machines read`, and a `rack-controllers read` that +> confirms the DC-LOCAL rack -- then `maas logout ` -- proves +> authentication, superuser scope, and REGION IDENTITY in four read-only calls. +> A credential that EXISTS is not a credential that WORKS. + **CHECK (read-only) -- voffice1 -- the cloud is registered AND the controller tag exists** ```bash juju clouds --client | grep -i maas