STAGE 5 -- "juju-client blocker" DIAGNOSIS (read-only)
Captured 2026-08-01 by agent session; repo at d555b90, branch dc-dc-stage5-preconditions.
Governing: D-138 (RULED 2026-07-30) + D-132 amendment. Ledger: SEC-026, SEC-028.
NOTHING IN THIS CAPTURE WAS MUTATED. All fix commands below are UNEXECUTED PROPOSALS.
=====================================================================
VERDICT
=====================================================================
Both reported blockers are NOT blockers. They are D-138 working as ruled.
(a) "no juju client on the jumphost" -> CORRECT under D-138. vcloud is not
where the cloud-facing client belongs. The client EXISTS, registered and
LIVE, on the dc0 rack (vvr1-dc0), which is the D-138 client host.
(b) "ssh vr1-dc0-juju -> Permission denied (publickey)" -> CORRECT and expected.
The offered key was never installed on that machine and is not needed;
`juju ssh -m controller 0` from the rack is the working access path and
was used throughout this diagnosis.
`juju add-model` is reachable from the rack TODAY. The real remaining gaps are
downstream of add-model and are listed in section 5.
=====================================================================
1. ROOT CAUSE OF THE SSH REFUSAL -- MEASURED, NO INFERENCE
=====================================================================
Alias (vcloud ~/.ssh/config.d/vr1-sites):
Host vr1-dc0-juju HostName 10.12.8.5 User ubuntu
ProxyJump vr1-dc0-rack IdentityFile ~/vr1-office1-creds/office1_svc_ed25519
`ssh -v -o BatchMode=yes vr1-dc0-juju` -- the hop chain is CORRECT end to end:
hop 1 vcloud -> voffice1 OK
hop 2 -> 172.31.0.2 (rack), key vr1-dc0_svc_ed25519 SHA256:DBkeStCyf2qGi3SY...
"Server accepts key" / "Authenticated to 172.31.0.2 (via proxy)"
hop 3 channel_connect_stdio_fwd: 10.12.8.5:22 -- reached, TCP fine
"Host '10.12.8.5' is known and matches the ED25519 host key"
"Found key in /home/jessea123/.ssh/known_hosts:28" <- host key CURRENT
Offering office1_svc_ed25519 SHA256:iUYex2kmvtdGTPiMKRZ2Nfywqa8pWacVV1GEEx6sscI
"Authentications that can continue: publickey"
ubuntu@10.12.8.5: Permission denied (publickey).
So: hostname, port, user, hop chain, per-hop identity selection and host key are
ALL correct and measured working. The single failing element is the final key.
WHAT THE TARGET ACTUALLY ACCEPTS (read via `juju ssh -m controller 0`):
/home/ubuntu/.ssh/authorized_keys -- exactly 2 lines:
4096 SHA256:VkipepSQPOmCyx1RS5HvM/F17U7c8HZAGO0LWHEgW/w Juju:juju-client-key (RSA)
4096 SHA256:pBDEJsaS7fH/JLdVWm3A9aOlBN0jiYIv4J9AjVPk7YI Juju:juju-system-key (RSA)
mtime 2026-07-31 01:00:03
SHA256:Vkipep... is byte-identical to the rack's
~/.local/share/juju/ssh/juju_id_rsa.pub (juju-client-key).
The office1 key SHA256:iUYex2... is ABSENT. Refusal is therefore CORRECT.
IDENTITY TIE -- the box read is the box that refused (hard rule 2, not "almost
certainly"). On `juju ssh -m controller 0`:
enp1s0 10.12.8.5/22
enp2s0 10.12.4.5/22
hostname: subtle-grouse
/etc/ssh/ssh_host_ed25519_key.pub -> 256 SHA256:Gr7kw5GTa+taeYiknHcLywc6Mjnwr82ly7gm2XetqXQ
That fingerprint is EXACTLY the host key `ssh -v` matched at known_hosts:28 for
10.12.8.5. Same host, proven. `juju machines -m controller` reports machine 0
at 10.12.4.5 / inst subtle-grouse -- the two addresses are two NICs on one VM.
WHY THE KEY IS ABSENT -- deploy-time evidence, and one candidate RULED OUT:
/var/lib/cloud/instance/user-data.txt mtime 2026-07-31 00:58:39
machine boot (`uptime -s`) 2026-07-31 00:58:15
ssh keys found in user-data.txt 0
/var/lib/cloud/instance/cloud-config.txt line 131 "ssh_authorized_keys:"
line 133 "Juju:juju-client-key"
grep for vr1-office1 / jessea123 in cloud-config.txt -> NO MATCH
vendor-cloud-config.txt / vendor-data.txt / vendor-data2.txt -> 0 ssh keys
`juju ssh-keys -m controller --full` -> "No keys to display."
`juju model-config -m controller authorized-keys` -> ERROR "authorized-keys" is
not a key of the currently targeted model (3.6 moved it to `juju ssh-keys`).
~/.local/share/juju/bootstrap-config.yaml:48 authorized-keys: | -> one ssh-rsa
entry only (the juju client key).
RULED OUT: "the machine was redeployed BEFORE the key import." The MAAS key
import into the new dc0 region was 2026-07-30; this machine deployed
2026-07-31 00:58 -- AFTER the import -- and STILL received only Juju keys.
Timing is not the explanation.
PROXIMATE CAUSE (measured): the deploy-time cloud-config supplied ONLY Juju's
own keys. Nothing ever put the office1 key on this VM.
DISTAL CAUSE (NOT MEASURED -- declared, not asserted): either the MAAS user
juju uses in the dc0 region (juju-vr1-dc0, minted fresh per SEC-028) has no
imported ssh key, or juju's bootstrap cloud-config supersedes the user's key
set. I could NOT distinguish these -- see section 6.
=====================================================================
2. WHERE THE JUJU CLIENT SHOULD LIVE, AND WHAT EXISTS -- ALL MEASURED
=====================================================================
D-138 RULED (1): tools that dial the CLOUD at L3 -- the Juju client and the
phase-03..06 `openstack` CLI -- run from INSIDE the DC. voffice1 stays Plane-2
only for application-proxied / qemu+ssh work. => the dc0 client host is the dc0
RACK (vvr1-dc0), consistent with SEC-028, which distributed the juju service
credential to the rack precisely because "D-138 moved the cloud-facing Juju
client INTO the DC -- so `rack` is now a credential-bearing host-role."
COMPONENT vcloud (jumphost) vr1-dc0-rack (D-138 host)
--------------------- ----------------------- -----------------------------
juju binary ABSENT (`which juju` PRESENT /snap/bin/juju
rc=1; snap list shows 3.6.27-genericlinux-amd64
only core24 + snapd) snap juju 3.6.27 rev 35621 3/stable
~/.local/share/juju ABSENT (No such file) PRESENT, 8 entries
controller reg. n/a PRESENT: vr1-dc0-controller*
credential n/a PRESENT: exactly ONE
repo scripts FULL repo (this clone) PARTIAL hand-stage, no .git
openstack CLI not checked (not the MISSING <-- D-138 gap
D-138 host)
maas CLI ABSENT, no ~/.maascli.db (not checked)
vcloud negative instrumented: `which juju` (rc=1), `snap list` (2 rows,
neither juju), `ls ~/.local/share/juju/` (No such file or directory).
Three independent instruments agree -- this is not a mis-scoped probe.
LIVE PROOF the rack client works (run 2026-08-01):
$ juju whoami
Controller: vr1-dc0-controller / Model: vr1-dc0 / User: admin
$ juju controllers
vr1-dc0-controller* vr1-dc0 admin superuser vr1-maas 1 model 1 node 3.6.27
$ juju models
controller vr1-maas maas available 1 machine 4 cores 1 unit admin
Last connection: just now <-- the controller IS UP and reachable
$ juju machines -m controller
0 started 10.12.4.5 subtle-grouse ubuntu@22.04 default Deployed
$ juju show-controller
controller-uuid 5556adb3-9cce-4b4e-8e64-f9c0a14c08f7
api-endpoints ['10.12.4.5:17070','10.12.8.5:17070','[fd50:840e:74e2:220::5]:17070']
~/.local/share/juju/clouds.yaml
vr1-maas / type maas / oauth1 / endpoint http://10.12.8.6:5240/MAAS
(the DC-LOCAL region -- D-132 amendment satisfied; no cross-fiber MAAS dep)
NOTHING IS MISSING for `add-model`: binary, registration and credential are
all present and demonstrably functional against the live controller.
=====================================================================
3. SEC-026 CONTROL (1) -- DISCHARGED BY MEASUREMENT
=====================================================================
SEC-026 requires: "Verify after any client rebuild that `juju credentials
--client` on a DC host lists exactly one."
$ juju credentials --client (on vr1-dc0-rack)
Client Credentials:
Cloud Credentials
vr1-maas vr1-dc0-cred
EXACTLY ONE. No vr1-dc1-cred present. The per-DC isolation SEC-018/-019 exist
to create is INTACT on the dc0 rack, measured 2026-08-01.
BOTH SIDES OF THE CONTROL MEASURED. SEC-026 warns that voffice1's client store
"holds BOTH vr1-dc0-cred and vr1-dc1-cred" and that copying it to a DC host
"would destroy the per-DC isolation." Confirmed on voffice1 2026-08-01:
$ juju credentials --client (on voffice1)
Client Credentials:
Cloud Credentials
vr1-maas vr1-dc0-cred, vr1-dc1-cred
Two on voffice1, ONE on the rack -> the forbidden whole-store copy did NOT
happen. The control is not merely unviolated, it is measured on both sides.
voffice1 juju state, VERIFIED (previously carried, now measured 2026-08-01):
which juju -> /snap/bin/juju
juju version -> 3.6.27-genericlinux-amd64
juju controllers-> ERROR No controllers registered.
~/.local/share/juju/ -> full store present (accounts, bootstrap-config,
clouds, controllers, credentials, models,
public-clouds, ssh)
So voffice1 has a BINARY and a CREDENTIAL STORE but no controller
registration -- it cannot perform the runbook's voffice1-labelled juju steps.
This is the measured basis for gap G1.
CREDENTIAL RESIDENCY OF THE PROPOSED FIX: NONE NEW. The fix in section 4 moves
no key, copies no credential, and creates no new residency. Every credential
location it touches is ALREADY REGISTERED in creds-manifests/vm-secret-locations:
line 123 rack vr1-dc0 ssh vr1-dc0-rack:~/vr1-dc0-creds/*
line 124 rack vr1-dc0 ssh vr1-dc0-rack:~/.local/share/juju/*
line 127 rack vr1-dc0 ssh vr1-dc0-rack:~/repo-stage/overlays/vr1-dc0-octavia-pki.yaml
NO new security-ledger row is owed for the section-4 fix.
=====================================================================
4. PROPOSED MINIMAL FIX -- UNEXECUTED, FOR OPERATOR APPROVAL
=====================================================================
FIX-0 (NO-OP, state it explicitly): do NOT install a juju client on vcloud and
do NOT add the office1 key to the controller VM. Both would work AGAINST D-138
and against the measured facts. The reported blocker requires no key movement.
FIX-1 -- HYGIENE ONLY; NOT REQUIRED FOR add-model (see the guard measurement
below before approving). Clears the dangling current-model pointer left by the
2026-07-31 teardown.
MEASURED SYMPTOM (on vr1-dc0-rack):
$ juju status
ERROR model vr1-dc0-controller:admin/vr1-dc0 not found
~/.local/share/juju/models.yaml has `current-model: admin/vr1-dc0` while its
`models:` map contains ONLY admin/controller -- the destroyed model's
pointer outlived the model.
IMPORTANT -- this does NOT block Step 3.5. Measured, in the runbook's own
guard shape:
$ juju models --format json | jq -r '.models[]?.name'
admin/controller
so the guard's `grep -qx "$DC_MODEL"` correctly finds no match and
`juju add-model vr1-dc0` proceeds. FIX-1 is hygiene for bare `juju status`,
not a precondition. Proposed command, run ON vr1-dc0-rack:
juju switch controller
Minimal because it uses juju's own state machine rather than hand-editing
models.yaml, and `add-model` will re-point current-model anyway.
REVERT: juju switch vr1-dc0 (once that model exists)
FIX-2 -- SEPARATE, OPTIONAL, DO NOT BATCH WITH FIX-1. Operator ssh to the
controller VM. Only if the operator wants `ssh vr1-dc0-juju` to work as a
convenience; it is NOT required for Stage 5, because `juju ssh -m controller 0`
already works and was the instrument for this entire diagnosis.
PREREQUISITE, MEASURED AS ABSENT -- this is a SEPARATE step, not part of the
command. The office1 PUBLIC key is NOT on the rack. Measured 2026-08-01:
vr1-dc0-rack $ ls -la ~/.ssh/
authorized_keys, known_hosts, known_hosts.old -- NO key files
vr1-dc0-rack $ find ~ -maxdepth 3 -name '*.pub'
/home/jessea123/vr1-dc0-creds/maas-virsh_ed25519.pub (only this one)
vcloud $ ls -la ~/vr1-office1-creds/
office1_svc_ed25519.pub 97 bytes, mode 0644 <- source, exists here
So FIX-2 requires FIRST staging that .pub to the rack (public key, not secret),
and only then:
juju add-ssh-key -m controller "<contents of office1_svc_ed25519.pub>"
The `-m controller` is REQUIRED and is not optional decoration: `add-ssh-key`
targets the CURRENT model, which is the dangling admin/vr1-dc0, so the bare
form fails with the same "model not found" documented under FIX-1.
NOT RECOMMENDED as part of this fix. It (a) adds an operator key to a
juju-managed authorized_keys set, (b) requires the office1 PUBLIC key to be
placed on the rack, and (c) widens shell access to the controller beyond the
juju-mediated path. If declined, the correct standing answer is: reach the
controller VM with `juju ssh -m controller 0` from the rack, and treat the
vr1-dc0-juju alias as non-functional-by-design.
If FIX-2 IS taken, the vr1-dc0-juju alias comment in
scripts/site-ssh-config.sh needs updating -- the fragment currently asserts
DC nodes carry the office1 key, which is FALSE for juju-provisioned machines
(measured above). That is a DOCFIX, not part of the mutation.
=====================================================================
5. REAL REMAINING GAPS -- DOWNSTREAM OF add-model. LOGGED, NOT FIXED.
=====================================================================
G1 -- HOST SPLIT THE RUNBOOK DOES NOT DESCRIBE.
runbooks/dc-dc-phase4-juju-bundle-per-dc.md:379 labels the add-model block
**MUTATION -- voffice1 -- create THIS DC's model (guarded)**
and :398 labels the spaces gate **CHECK (read-only) -- voffice1 --**.
Both are STALE w.r.t. D-138, which is dated 2026-07-30 and moved the juju
client into the DC. voffice1 has a juju BINARY (3.6.27) but `juju controllers`
there returns ERROR No controllers registered -- so the runbook as written
targets a host that cannot perform the step. DOCFIX owed. NOT EDITED here
(a) because hard rule 1 forbids adjacent changes mid-step and (b) because the
DOCFIX must be sized against every other voffice1-pinned juju reference, not
just these two lines. Grep next-free DOCFIX-NNN before assigning a number.
G2 -- scripts/juju-spaces-check.sh NEEDS repo + juju ON THE SAME HOST.
The runbook runs it immediately after add-model. It is present on the rack at
~/repo-stage/scripts/juju-spaces-check.sh -- and the staged copy is IDENTICAL
to the repo copy (sha256 adb9dbd38a27e663083f37b64e17ca98d870180f010b8d499a2
9f3fbfe7d1d5d on BOTH vcloud and vr1-dc0-rack). jq and python3 3.12.3 are
present on the rack. So this step CAN run today. But ~/repo-stage is a
9-file HAND-STAGED copy with NO .git (verified: `find / -xdev -maxdepth 5
-name .git -type d` on the rack returned NOTHING), so it has no provenance
and will drift silently. Drift measured 2026-08-01: 8 of 9 files byte-identical
to the repo; the 9th (vr1-dc0-octavia-pki.yaml) exists ONLY on the rack.
G3 -- overlays/vr1-dc0-octavia-pki.yaml IS ABSENT FROM THE vcloud REPO TREE.
Two independent instruments agree: `ls -la overlays/` lists 6 files and not
this one; `sha256sum overlays/vr1-dc0-octavia-pki.yaml` -> No such file.
It IS present on the rack, mode -rw------- (0600), 4682 bytes, and it is
gitignored by design (.gitignore:46 `overlays/*octavia-pki.yaml`).
vm-secret-locations registers it at BOTH locations (line 90 headend
`<repo>/overlays/vr1-dc0-octavia-pki.yaml`, line 127 rack), so the register is
correct and the rack copy's 0600 handling is correct.
CONSEQUENCE -- CITED FROM CODE, NOT EXECUTED. preflight.sh runs on the repo
host (vcloud). Its P4 delegate hard-FAILS on the missing file:
scripts/pre-flight-checks.sh:155-162 (CHECK 0)
OVL="overlays/${DC}-octavia-pki.yaml"
if [ -f "$OVL" ]; then ... else
fail "MISSING $OVL (gitignored secret; place it or regenerate via
runbook Step 1.0-GEN before deploy)"
`fail` -> preflight exit 1 -> "do NOT deploy". So the mandated pre-add-model
gate CANNOT PASS on vcloud as the tree stands. This is a CODE READ; I did not
run preflight.sh (out of mandate scope), so the FAIL is cited, not observed.
CORRECTION TO AN EARLIER DRAFT OF THIS CAPTURE: P7 does NOT depend on this
overlay and must not be cited here. scripts/octavia-pki.sh reads
overlays/${SITE}-vips.yaml (line 169) and WRITES the octavia-pki overlay; it
is the generator, not a consumer. Only P4 CHECK 0 gates on the file's presence.
Note preflight.sh:144-146 SILENTLY SKIPS a missing overlay at P2
(`[ -f "$_ovl" ] && P2_ARGS=...`), so P2 would validate a bundle merged
WITHOUT the octavia overlay and still report on it -- the failure surfaces at
P4/P7, not P2. NOT INVESTIGATED FURTHER -- out of mandate scope. Flagged so it
is not discovered at deploy time.
G4 -- `openstack` CLI is MISSING on the dc0 rack.
Measured with THREE instruments, because `which` in a non-interactive ssh
shell misses ~/.local/bin: `which openstack` -> MISSING; `ls ~/.local/bin`
-> No such file or directory (the dir does not exist at all); `snap list |
grep -i openstack` -> no openstack snap. All three agree. D-138 names the
phase-03..06 `openstack` CLI work explicitly as DC-resident. This is not a
Stage-5 add-model blocker but IS a D-138 definition-of-done gap that will
block phase-03. Also missing on the rack: yq, charmcraft. Present: python3
3.12.3, jq, git.
=====================================================================
6. WHAT I COULD NOT MEASURE, AND HOW I KNOW I COULD NOT
=====================================================================
UNMEASURED: whether MAAS user `juju-vr1-dc0` in the dc0 region has an imported
ssh key. This is the DISTAL half of section 1's root cause. It is DECLARED, not
guessed. Attempts and why each failed:
- `sudo -u postgres psql -d maasdb` on vr1-dc0-maas -> "sudo: unknown user
postgres" (snap postgresql runs as _daemon_).
- `/snap/bin/postgresql.psql -U postgres -h 127.0.0.1` -> asks for a password;
NOT pursued (would mean handling credential material).
- `sudo -u _daemon_ /snap/bin/postgresql.psql` -> snap refuses: "is not a snap
cgroup for tag snap.postgresql.psql".
- maas CLI profile: ABSENT on vcloud (no ~/.maascli.db) and EFFECTIVELY absent
on voffice1 -- ~/.maascli.db there is ZERO BYTES (Jul 29) and sqlite reports
"no such table: profiles". vr1-dc0-maas has no ~/.maascli.db either.
Operator command that would settle it (UNEXECUTED, needs a logged-in profile
pointed at http://10.12.8.6:5240/MAAS -- verify the endpoint BEFORE trusting
the answer; a profile aimed at 10.10.0.20 is the OFFICE1 region and would give
a confidently wrong answer):
maas <profile> users read
maas <profile> sshkeys read
This gap does NOT change the fix. Section 4 is correct either way, because the
proximate cause (only Juju keys were ever supplied to this machine) is measured
directly on the machine.
REFUSED BY THE PERMISSION HARNESS -- verbatim, and NOT retried in an altered
shape (the query was `ls ~/.maascli.db; maas list; ls -la ~/vr1-dc0-creds/` on
vr1-dc0-rack):
PreToolUse:Bash hook error: [python3 "$CLAUDE_PROJECT_DIR"/.claude/hooks/
guard-destructive.py]: BLOCKED by .claude/hooks/guard-destructive.py:
prints the MAAS API key (DOCFIX-016); use 'maas admin ...' directly
Consequence: the rack's maas-profile state and the contents of
~/vr1-dc0-creds/ were NOT enumerated this session. SEC-028's claim that the
credential was distributed to the rack is therefore CARRIED FROM THE LEDGER,
not re-measured here.
INSTRUMENT CORRECTION MADE MID-SESSION (recorded because the first result was a
false negative of exactly the shape this repo has been bitten by): an initial
`find ~ -maxdepth 2 -name preflight.sh` on the rack could NOT have found a repo
clone -- repo scripts sit at depth 3. Re-run as `find / -xdev` over unique
filenames, it FOUND ~/repo-stage. The corrected result is what section 5 uses.
Likewise `grep -rn ... creds-manifests/vm-secret-locations/` (trailing slash on
a FILE) returned nothing and was re-run without the slash, which found all six
rack rows. Neither false negative reached a conclusion.
=====================================================================
7. COMMANDS RUN (all read-only; none mutated state)
=====================================================================
vcloud: which juju; snap list; ls ~/.local/share/juju; ls ~/ | grep creds;
ssh -v -o BatchMode=yes vr1-dc0-juju; sha256sum <9 repo files>;
ls -la overlays/; git check-ignore -v; grep over docs/ and
creds-manifests/vm-secret-locations; sed over runbook + preflight.sh;
python3 sqlite read of ~/.maascli.db (endpoint URL only, no keys)
vr1-dc0-rack: hostname; which juju; juju version; snap list; ls ~/.local/share/juju;
juju whoami / controllers / models / credentials --client /
show-controller / machines -m controller / ssh-keys -m controller /
model-config -m controller authorized-keys / models --format json;
cat clouds.yaml models.yaml; grep accounts.yaml (password excluded);
ssh-keygen -lf juju_id_rsa.pub; find / -xdev for repo + .git;
sha256sum <9 staged files>; ls -la repo-stage; which openstack et al
subtle-grouse (via `juju ssh -m controller 0`): authorized_keys fingerprints +
comments + mtime; ip -4 -o addr; ssh-keygen -lf host key; hostname;
stat/grep over /var/lib/cloud/instance/*
vr1-dc0-maas: hostname; snap list; ps; ss -lntp; three failed psql attempts
ls -la ~/vr1-office1-creds/ (names/modes only); grep+sed over
scripts/pre-flight-checks.sh and scripts/octavia-pki.sh;
bash scripts/repo-lint.sh; git status --short; git check-ignore -v
vr1-dc0-rack (cont.): ls -la ~/.ssh/; find ~ -maxdepth 3 -name '*.pub';
ls ~/.local/bin; snap list | grep openstack
voffice1: ls ~/.maascli.db; python3 sqlite read (no profiles table);
which juju; juju version; juju controllers; juju credentials --client;
ls ~/.local/share/juju/
=====================================================================
8. REPO-LINT STATE AT CAPTURE TIME -- NOT CAUSED BY, NOT FIXABLE BY, THIS WORK
=====================================================================
$ bash scripts/repo-lint.sh
[WARN] L1 docs/design-decisions.md: 239 non-ASCII byte(s) (legacy carve-out)
[FAIL] L10 pending change set modifies state-bearing surface(s)
[docs/audit/: docs/audit/d139-carve-dryrun-20260801.txt,
docs/audit/stage5-juju-client-blocker-20260801.txt] without touching
docs/CURRENT-STATE.md -- GA-R1 rule 8 (C1)
FAIL: repo lint (1 fail, 1 warn, 645 files scanned) EXIT=1
L10 was ALREADY FAILING before this capture existed: the other file it names,
docs/audit/d139-carve-dryrun-20260801.txt, is a concurrent agent's and has
mtime 00:18 against this file's 00:25. This capture is now ALSO named in the
finding but did not create it.
NOT RESOLVED HERE, deliberately: clearing L10 requires touching
docs/CURRENT-STATE.md in the SAME commit, and this session was scoped out of
both that file and any git add/commit/push. Resolution must be coordinated by
whoever owns docs/CURRENT-STATE.md for this change set.
