STAGE 5 -- "juju-client blocker" DIAGNOSIS (read-only) Captured 2026-08-01 by agent session; repo at d555b90, branch dc-dc-stage5-preconditions. Governing: D-138 (RULED 2026-07-30) + D-132 amendment. Ledger: SEC-026, SEC-028. NOTHING IN THIS CAPTURE WAS MUTATED. All fix commands below are UNEXECUTED PROPOSALS. ===================================================================== VERDICT ===================================================================== Both reported blockers are NOT blockers. They are D-138 working as ruled. (a) "no juju client on the jumphost" -> CORRECT under D-138. vcloud is not where the cloud-facing client belongs. The client EXISTS, registered and LIVE, on the dc0 rack (vvr1-dc0), which is the D-138 client host. (b) "ssh vr1-dc0-juju -> Permission denied (publickey)" -> CORRECT and expected. The offered key was never installed on that machine and is not needed; `juju ssh -m controller 0` from the rack is the working access path and was used throughout this diagnosis. `juju add-model` is reachable from the rack TODAY. The real remaining gaps are downstream of add-model and are listed in section 5. ===================================================================== 1. ROOT CAUSE OF THE SSH REFUSAL -- MEASURED, NO INFERENCE ===================================================================== Alias (vcloud ~/.ssh/config.d/vr1-sites): Host vr1-dc0-juju HostName 10.12.8.5 User ubuntu ProxyJump vr1-dc0-rack IdentityFile ~/vr1-office1-creds/office1_svc_ed25519 `ssh -v -o BatchMode=yes vr1-dc0-juju` -- the hop chain is CORRECT end to end: hop 1 vcloud -> voffice1 OK hop 2 -> 172.31.0.2 (rack), key vr1-dc0_svc_ed25519 SHA256:DBkeStCyf2qGi3SY... "Server accepts key" / "Authenticated to 172.31.0.2 (via proxy)" hop 3 channel_connect_stdio_fwd: 10.12.8.5:22 -- reached, TCP fine "Host '10.12.8.5' is known and matches the ED25519 host key" "Found key in /home/jessea123/.ssh/known_hosts:28" <- host key CURRENT Offering office1_svc_ed25519 SHA256:iUYex2kmvtdGTPiMKRZ2Nfywqa8pWacVV1GEEx6sscI "Authentications that can continue: publickey" ubuntu@10.12.8.5: Permission denied (publickey). So: hostname, port, user, hop chain, per-hop identity selection and host key are ALL correct and measured working. The single failing element is the final key. WHAT THE TARGET ACTUALLY ACCEPTS (read via `juju ssh -m controller 0`): /home/ubuntu/.ssh/authorized_keys -- exactly 2 lines: 4096 SHA256:VkipepSQPOmCyx1RS5HvM/F17U7c8HZAGO0LWHEgW/w Juju:juju-client-key (RSA) 4096 SHA256:pBDEJsaS7fH/JLdVWm3A9aOlBN0jiYIv4J9AjVPk7YI Juju:juju-system-key (RSA) mtime 2026-07-31 01:00:03 SHA256:Vkipep... is byte-identical to the rack's ~/.local/share/juju/ssh/juju_id_rsa.pub (juju-client-key). The office1 key SHA256:iUYex2... is ABSENT. Refusal is therefore CORRECT. IDENTITY TIE -- the box read is the box that refused (hard rule 2, not "almost certainly"). On `juju ssh -m controller 0`: enp1s0 10.12.8.5/22 enp2s0 10.12.4.5/22 hostname: subtle-grouse /etc/ssh/ssh_host_ed25519_key.pub -> 256 SHA256:Gr7kw5GTa+taeYiknHcLywc6Mjnwr82ly7gm2XetqXQ That fingerprint is EXACTLY the host key `ssh -v` matched at known_hosts:28 for 10.12.8.5. Same host, proven. `juju machines -m controller` reports machine 0 at 10.12.4.5 / inst subtle-grouse -- the two addresses are two NICs on one VM. WHY THE KEY IS ABSENT -- deploy-time evidence, and one candidate RULED OUT: /var/lib/cloud/instance/user-data.txt mtime 2026-07-31 00:58:39 machine boot (`uptime -s`) 2026-07-31 00:58:15 ssh keys found in user-data.txt 0 /var/lib/cloud/instance/cloud-config.txt line 131 "ssh_authorized_keys:" line 133 "Juju:juju-client-key" grep for vr1-office1 / jessea123 in cloud-config.txt -> NO MATCH vendor-cloud-config.txt / vendor-data.txt / vendor-data2.txt -> 0 ssh keys `juju ssh-keys -m controller --full` -> "No keys to display." `juju model-config -m controller authorized-keys` -> ERROR "authorized-keys" is not a key of the currently targeted model (3.6 moved it to `juju ssh-keys`). ~/.local/share/juju/bootstrap-config.yaml:48 authorized-keys: | -> one ssh-rsa entry only (the juju client key). RULED OUT: "the machine was redeployed BEFORE the key import." The MAAS key import into the new dc0 region was 2026-07-30; this machine deployed 2026-07-31 00:58 -- AFTER the import -- and STILL received only Juju keys. Timing is not the explanation. PROXIMATE CAUSE (measured): the deploy-time cloud-config supplied ONLY Juju's own keys. Nothing ever put the office1 key on this VM. DISTAL CAUSE (NOT MEASURED -- declared, not asserted): either the MAAS user juju uses in the dc0 region (juju-vr1-dc0, minted fresh per SEC-028) has no imported ssh key, or juju's bootstrap cloud-config supersedes the user's key set. I could NOT distinguish these -- see section 6. ===================================================================== 2. WHERE THE JUJU CLIENT SHOULD LIVE, AND WHAT EXISTS -- ALL MEASURED ===================================================================== D-138 RULED (1): tools that dial the CLOUD at L3 -- the Juju client and the phase-03..06 `openstack` CLI -- run from INSIDE the DC. voffice1 stays Plane-2 only for application-proxied / qemu+ssh work. => the dc0 client host is the dc0 RACK (vvr1-dc0), consistent with SEC-028, which distributed the juju service credential to the rack precisely because "D-138 moved the cloud-facing Juju client INTO the DC -- so `rack` is now a credential-bearing host-role." COMPONENT vcloud (jumphost) vr1-dc0-rack (D-138 host) --------------------- ----------------------- ----------------------------- juju binary ABSENT (`which juju` PRESENT /snap/bin/juju rc=1; snap list shows 3.6.27-genericlinux-amd64 only core24 + snapd) snap juju 3.6.27 rev 35621 3/stable ~/.local/share/juju ABSENT (No such file) PRESENT, 8 entries controller reg. n/a PRESENT: vr1-dc0-controller* credential n/a PRESENT: exactly ONE repo scripts FULL repo (this clone) PARTIAL hand-stage, no .git openstack CLI not checked (not the MISSING <-- D-138 gap D-138 host) maas CLI ABSENT, no ~/.maascli.db (not checked) vcloud negative instrumented: `which juju` (rc=1), `snap list` (2 rows, neither juju), `ls ~/.local/share/juju/` (No such file or directory). Three independent instruments agree -- this is not a mis-scoped probe. LIVE PROOF the rack client works (run 2026-08-01): $ juju whoami Controller: vr1-dc0-controller / Model: vr1-dc0 / User: admin $ juju controllers vr1-dc0-controller* vr1-dc0 admin superuser vr1-maas 1 model 1 node 3.6.27 $ juju models controller vr1-maas maas available 1 machine 4 cores 1 unit admin Last connection: just now <-- the controller IS UP and reachable $ juju machines -m controller 0 started 10.12.4.5 subtle-grouse ubuntu@22.04 default Deployed $ juju show-controller controller-uuid 5556adb3-9cce-4b4e-8e64-f9c0a14c08f7 api-endpoints ['10.12.4.5:17070','10.12.8.5:17070','[fd50:840e:74e2:220::5]:17070'] ~/.local/share/juju/clouds.yaml vr1-maas / type maas / oauth1 / endpoint http://10.12.8.6:5240/MAAS (the DC-LOCAL region -- D-132 amendment satisfied; no cross-fiber MAAS dep) NOTHING IS MISSING for `add-model`: binary, registration and credential are all present and demonstrably functional against the live controller. ===================================================================== 3. SEC-026 CONTROL (1) -- DISCHARGED BY MEASUREMENT ===================================================================== SEC-026 requires: "Verify after any client rebuild that `juju credentials --client` on a DC host lists exactly one." $ juju credentials --client (on vr1-dc0-rack) Client Credentials: Cloud Credentials vr1-maas vr1-dc0-cred EXACTLY ONE. No vr1-dc1-cred present. The per-DC isolation SEC-018/-019 exist to create is INTACT on the dc0 rack, measured 2026-08-01. BOTH SIDES OF THE CONTROL MEASURED. SEC-026 warns that voffice1's client store "holds BOTH vr1-dc0-cred and vr1-dc1-cred" and that copying it to a DC host "would destroy the per-DC isolation." Confirmed on voffice1 2026-08-01: $ juju credentials --client (on voffice1) Client Credentials: Cloud Credentials vr1-maas vr1-dc0-cred, vr1-dc1-cred Two on voffice1, ONE on the rack -> the forbidden whole-store copy did NOT happen. The control is not merely unviolated, it is measured on both sides. voffice1 juju state, VERIFIED (previously carried, now measured 2026-08-01): which juju -> /snap/bin/juju juju version -> 3.6.27-genericlinux-amd64 juju controllers-> ERROR No controllers registered. ~/.local/share/juju/ -> full store present (accounts, bootstrap-config, clouds, controllers, credentials, models, public-clouds, ssh) So voffice1 has a BINARY and a CREDENTIAL STORE but no controller registration -- it cannot perform the runbook's voffice1-labelled juju steps. This is the measured basis for gap G1. CREDENTIAL RESIDENCY OF THE PROPOSED FIX: NONE NEW. The fix in section 4 moves no key, copies no credential, and creates no new residency. Every credential location it touches is ALREADY REGISTERED in creds-manifests/vm-secret-locations: line 123 rack vr1-dc0 ssh vr1-dc0-rack:~/vr1-dc0-creds/* line 124 rack vr1-dc0 ssh vr1-dc0-rack:~/.local/share/juju/* line 127 rack vr1-dc0 ssh vr1-dc0-rack:~/repo-stage/overlays/vr1-dc0-octavia-pki.yaml NO new security-ledger row is owed for the section-4 fix. ===================================================================== 4. PROPOSED MINIMAL FIX -- UNEXECUTED, FOR OPERATOR APPROVAL ===================================================================== FIX-0 (NO-OP, state it explicitly): do NOT install a juju client on vcloud and do NOT add the office1 key to the controller VM. Both would work AGAINST D-138 and against the measured facts. The reported blocker requires no key movement. FIX-1 -- HYGIENE ONLY; NOT REQUIRED FOR add-model (see the guard measurement below before approving). Clears the dangling current-model pointer left by the 2026-07-31 teardown. MEASURED SYMPTOM (on vr1-dc0-rack): $ juju status ERROR model vr1-dc0-controller:admin/vr1-dc0 not found ~/.local/share/juju/models.yaml has `current-model: admin/vr1-dc0` while its `models:` map contains ONLY admin/controller -- the destroyed model's pointer outlived the model. IMPORTANT -- this does NOT block Step 3.5. Measured, in the runbook's own guard shape: $ juju models --format json | jq -r '.models[]?.name' admin/controller so the guard's `grep -qx "$DC_MODEL"` correctly finds no match and `juju add-model vr1-dc0` proceeds. FIX-1 is hygiene for bare `juju status`, not a precondition. Proposed command, run ON vr1-dc0-rack: juju switch controller Minimal because it uses juju's own state machine rather than hand-editing models.yaml, and `add-model` will re-point current-model anyway. REVERT: juju switch vr1-dc0 (once that model exists) FIX-2 -- SEPARATE, OPTIONAL, DO NOT BATCH WITH FIX-1. Operator ssh to the controller VM. Only if the operator wants `ssh vr1-dc0-juju` to work as a convenience; it is NOT required for Stage 5, because `juju ssh -m controller 0` already works and was the instrument for this entire diagnosis. PREREQUISITE, MEASURED AS ABSENT -- this is a SEPARATE step, not part of the command. The office1 PUBLIC key is NOT on the rack. Measured 2026-08-01: vr1-dc0-rack $ ls -la ~/.ssh/ authorized_keys, known_hosts, known_hosts.old -- NO key files vr1-dc0-rack $ find ~ -maxdepth 3 -name '*.pub' /home/jessea123/vr1-dc0-creds/maas-virsh_ed25519.pub (only this one) vcloud $ ls -la ~/vr1-office1-creds/ office1_svc_ed25519.pub 97 bytes, mode 0644 <- source, exists here So FIX-2 requires FIRST staging that .pub to the rack (public key, not secret), and only then: juju add-ssh-key -m controller "" The `-m controller` is REQUIRED and is not optional decoration: `add-ssh-key` targets the CURRENT model, which is the dangling admin/vr1-dc0, so the bare form fails with the same "model not found" documented under FIX-1. NOT RECOMMENDED as part of this fix. It (a) adds an operator key to a juju-managed authorized_keys set, (b) requires the office1 PUBLIC key to be placed on the rack, and (c) widens shell access to the controller beyond the juju-mediated path. If declined, the correct standing answer is: reach the controller VM with `juju ssh -m controller 0` from the rack, and treat the vr1-dc0-juju alias as non-functional-by-design. If FIX-2 IS taken, the vr1-dc0-juju alias comment in scripts/site-ssh-config.sh needs updating -- the fragment currently asserts DC nodes carry the office1 key, which is FALSE for juju-provisioned machines (measured above). That is a DOCFIX, not part of the mutation. ===================================================================== 5. REAL REMAINING GAPS -- DOWNSTREAM OF add-model. LOGGED, NOT FIXED. ===================================================================== G1 -- HOST SPLIT THE RUNBOOK DOES NOT DESCRIBE. runbooks/dc-dc-phase4-juju-bundle-per-dc.md:379 labels the add-model block **MUTATION -- voffice1 -- create THIS DC's model (guarded)** and :398 labels the spaces gate **CHECK (read-only) -- voffice1 --**. Both are STALE w.r.t. D-138, which is dated 2026-07-30 and moved the juju client into the DC. voffice1 has a juju BINARY (3.6.27) but `juju controllers` there returns ERROR No controllers registered -- so the runbook as written targets a host that cannot perform the step. DOCFIX owed. NOT EDITED here (a) because hard rule 1 forbids adjacent changes mid-step and (b) because the DOCFIX must be sized against every other voffice1-pinned juju reference, not just these two lines. Grep next-free DOCFIX-NNN before assigning a number. G2 -- scripts/juju-spaces-check.sh NEEDS repo + juju ON THE SAME HOST. The runbook runs it immediately after add-model. It is present on the rack at ~/repo-stage/scripts/juju-spaces-check.sh -- and the staged copy is IDENTICAL to the repo copy (sha256 adb9dbd38a27e663083f37b64e17ca98d870180f010b8d499a2 9f3fbfe7d1d5d on BOTH vcloud and vr1-dc0-rack). jq and python3 3.12.3 are present on the rack. So this step CAN run today. But ~/repo-stage is a 9-file HAND-STAGED copy with NO .git (verified: `find / -xdev -maxdepth 5 -name .git -type d` on the rack returned NOTHING), so it has no provenance and will drift silently. Drift measured 2026-08-01: 8 of 9 files byte-identical to the repo; the 9th (vr1-dc0-octavia-pki.yaml) exists ONLY on the rack. G3 -- overlays/vr1-dc0-octavia-pki.yaml IS ABSENT FROM THE vcloud REPO TREE. Two independent instruments agree: `ls -la overlays/` lists 6 files and not this one; `sha256sum overlays/vr1-dc0-octavia-pki.yaml` -> No such file. It IS present on the rack, mode -rw------- (0600), 4682 bytes, and it is gitignored by design (.gitignore:46 `overlays/*octavia-pki.yaml`). vm-secret-locations registers it at BOTH locations (line 90 headend `/overlays/vr1-dc0-octavia-pki.yaml`, line 127 rack), so the register is correct and the rack copy's 0600 handling is correct. CONSEQUENCE -- CITED FROM CODE, NOT EXECUTED. preflight.sh runs on the repo host (vcloud). Its P4 delegate hard-FAILS on the missing file: scripts/pre-flight-checks.sh:155-162 (CHECK 0) OVL="overlays/${DC}-octavia-pki.yaml" if [ -f "$OVL" ]; then ... else fail "MISSING $OVL (gitignored secret; place it or regenerate via runbook Step 1.0-GEN before deploy)" `fail` -> preflight exit 1 -> "do NOT deploy". So the mandated pre-add-model gate CANNOT PASS on vcloud as the tree stands. This is a CODE READ; I did not run preflight.sh (out of mandate scope), so the FAIL is cited, not observed. CORRECTION TO AN EARLIER DRAFT OF THIS CAPTURE: P7 does NOT depend on this overlay and must not be cited here. scripts/octavia-pki.sh reads overlays/${SITE}-vips.yaml (line 169) and WRITES the octavia-pki overlay; it is the generator, not a consumer. Only P4 CHECK 0 gates on the file's presence. Note preflight.sh:144-146 SILENTLY SKIPS a missing overlay at P2 (`[ -f "$_ovl" ] && P2_ARGS=...`), so P2 would validate a bundle merged WITHOUT the octavia overlay and still report on it -- the failure surfaces at P4/P7, not P2. NOT INVESTIGATED FURTHER -- out of mandate scope. Flagged so it is not discovered at deploy time. G4 -- `openstack` CLI is MISSING on the dc0 rack. Measured with THREE instruments, because `which` in a non-interactive ssh shell misses ~/.local/bin: `which openstack` -> MISSING; `ls ~/.local/bin` -> No such file or directory (the dir does not exist at all); `snap list | grep -i openstack` -> no openstack snap. All three agree. D-138 names the phase-03..06 `openstack` CLI work explicitly as DC-resident. This is not a Stage-5 add-model blocker but IS a D-138 definition-of-done gap that will block phase-03. Also missing on the rack: yq, charmcraft. Present: python3 3.12.3, jq, git. ===================================================================== 6. WHAT I COULD NOT MEASURE, AND HOW I KNOW I COULD NOT ===================================================================== UNMEASURED: whether MAAS user `juju-vr1-dc0` in the dc0 region has an imported ssh key. This is the DISTAL half of section 1's root cause. It is DECLARED, not guessed. Attempts and why each failed: - `sudo -u postgres psql -d maasdb` on vr1-dc0-maas -> "sudo: unknown user postgres" (snap postgresql runs as _daemon_). - `/snap/bin/postgresql.psql -U postgres -h 127.0.0.1` -> asks for a password; NOT pursued (would mean handling credential material). - `sudo -u _daemon_ /snap/bin/postgresql.psql` -> snap refuses: "is not a snap cgroup for tag snap.postgresql.psql". - maas CLI profile: ABSENT on vcloud (no ~/.maascli.db) and EFFECTIVELY absent on voffice1 -- ~/.maascli.db there is ZERO BYTES (Jul 29) and sqlite reports "no such table: profiles". vr1-dc0-maas has no ~/.maascli.db either. Operator command that would settle it (UNEXECUTED, needs a logged-in profile pointed at http://10.12.8.6:5240/MAAS -- verify the endpoint BEFORE trusting the answer; a profile aimed at 10.10.0.20 is the OFFICE1 region and would give a confidently wrong answer): maas users read maas sshkeys read This gap does NOT change the fix. Section 4 is correct either way, because the proximate cause (only Juju keys were ever supplied to this machine) is measured directly on the machine. REFUSED BY THE PERMISSION HARNESS -- verbatim, and NOT retried in an altered shape (the query was `ls ~/.maascli.db; maas list; ls -la ~/vr1-dc0-creds/` on vr1-dc0-rack): PreToolUse:Bash hook error: [python3 "$CLAUDE_PROJECT_DIR"/.claude/hooks/ guard-destructive.py]: BLOCKED by .claude/hooks/guard-destructive.py: prints the MAAS API key (DOCFIX-016); use 'maas admin ...' directly Consequence: the rack's maas-profile state and the contents of ~/vr1-dc0-creds/ were NOT enumerated this session. SEC-028's claim that the credential was distributed to the rack is therefore CARRIED FROM THE LEDGER, not re-measured here. INSTRUMENT CORRECTION MADE MID-SESSION (recorded because the first result was a false negative of exactly the shape this repo has been bitten by): an initial `find ~ -maxdepth 2 -name preflight.sh` on the rack could NOT have found a repo clone -- repo scripts sit at depth 3. Re-run as `find / -xdev` over unique filenames, it FOUND ~/repo-stage. The corrected result is what section 5 uses. Likewise `grep -rn ... creds-manifests/vm-secret-locations/` (trailing slash on a FILE) returned nothing and was re-run without the slash, which found all six rack rows. Neither false negative reached a conclusion. ===================================================================== 7. COMMANDS RUN (all read-only; none mutated state) ===================================================================== vcloud: which juju; snap list; ls ~/.local/share/juju; ls ~/ | grep creds; ssh -v -o BatchMode=yes vr1-dc0-juju; sha256sum <9 repo files>; ls -la overlays/; git check-ignore -v; grep over docs/ and creds-manifests/vm-secret-locations; sed over runbook + preflight.sh; python3 sqlite read of ~/.maascli.db (endpoint URL only, no keys) vr1-dc0-rack: hostname; which juju; juju version; snap list; ls ~/.local/share/juju; juju whoami / controllers / models / credentials --client / show-controller / machines -m controller / ssh-keys -m controller / model-config -m controller authorized-keys / models --format json; cat clouds.yaml models.yaml; grep accounts.yaml (password excluded); ssh-keygen -lf juju_id_rsa.pub; find / -xdev for repo + .git; sha256sum <9 staged files>; ls -la repo-stage; which openstack et al subtle-grouse (via `juju ssh -m controller 0`): authorized_keys fingerprints + comments + mtime; ip -4 -o addr; ssh-keygen -lf host key; hostname; stat/grep over /var/lib/cloud/instance/* vr1-dc0-maas: hostname; snap list; ps; ss -lntp; three failed psql attempts ls -la ~/vr1-office1-creds/ (names/modes only); grep+sed over scripts/pre-flight-checks.sh and scripts/octavia-pki.sh; bash scripts/repo-lint.sh; git status --short; git check-ignore -v vr1-dc0-rack (cont.): ls -la ~/.ssh/; find ~ -maxdepth 3 -name '*.pub'; ls ~/.local/bin; snap list | grep openstack voffice1: ls ~/.maascli.db; python3 sqlite read (no profiles table); which juju; juju version; juju controllers; juju credentials --client; ls ~/.local/share/juju/ ===================================================================== 8. REPO-LINT STATE AT CAPTURE TIME -- NOT CAUSED BY, NOT FIXABLE BY, THIS WORK ===================================================================== $ bash scripts/repo-lint.sh [WARN] L1 docs/design-decisions.md: 239 non-ASCII byte(s) (legacy carve-out) [FAIL] L10 pending change set modifies state-bearing surface(s) [docs/audit/: docs/audit/d139-carve-dryrun-20260801.txt, docs/audit/stage5-juju-client-blocker-20260801.txt] without touching docs/CURRENT-STATE.md -- GA-R1 rule 8 (C1) FAIL: repo lint (1 fail, 1 warn, 645 files scanned) EXIT=1 L10 was ALREADY FAILING before this capture existed: the other file it names, docs/audit/d139-carve-dryrun-20260801.txt, is a concurrent agent's and has mtime 00:18 against this file's 00:25. This capture is now ALSO named in the finding but did not create it. NOT RESOLVED HERE, deliberately: clearing L10 requires touching docs/CURRENT-STATE.md in the SAME commit, and this session was scoped out of both that file and any git add/commit/push. Resolution must be coordinated by whoever owns docs/CURRENT-STATE.md for this change set.