diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index ef725b1..a2441bb 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1890,6 +1890,63 @@ equally at precedence 40); it buys conformance and Roosevelt fidelity, and nothing else. A later session reasoning "GUA was chosen so v6 would beat v4" is reasoning from the struck argument and is wrong. **Stage-2 execution of D-139 is therefore UNBLOCKED.** + **>>> THE "JUJU CLIENT BLOCKER" IS NOT A BLOCKER -- IT IS D-138 WORKING CORRECTLY. + STAGE 5 CAN REACH `add-model` TODAY, FROM THE dc0 RACK. <<<** Capture + `docs/audit/stage5-juju-client-blocker-20260801.txt` (391 lines). **VERIFIED INDEPENDENTLY + from the main session, not accepted on the agent's word:** `/snap/bin/juju` + **3.6.27-genericlinux-amd64** on the rack; controller **`vr1-dc0-controller*`**, admin / + superuser, cloud `vr1-maas`; the `controller` model reads **"Last connection: just now"**; + client credentials list **exactly one** entry, `vr1-maas -> vr1-dc0-cred`. **CORRECTION TO + THIS DOCUMENT'S OWN EARLIER ENTRY (2026-07-31, finding 9d), which called this a blocker for + the next phase: it is not, and no key movement is needed.** + **ROOT CAUSE OF THE `ssh vr1-dc0-juju` REFUSAL, MEASURED ON THE MACHINE ITSELF.** The alias + is entirely correct -- hop chain, host, port, per-hop identity and host key all measured + working; `ssh -v` reaches `10.12.8.5:22`, matches the known_hosts entry, offers + `office1_svc_ed25519` and is refused. **The target simply does not carry that key:** read + via `juju ssh -m controller 0`, `subtle-grouse`'s `/home/ubuntu/.ssh/authorized_keys` holds + **exactly two lines, both Juju's** (`Juju:juju-client-key`, byte-identical to the rack's own + `~/.local/share/juju/ssh/juju_id_rsa.pub`, and `Juju:juju-system-key`). Identity tied rather + than inferred: machine 0 holds BOTH `10.12.8.5/22` and `10.12.4.5/22`, hostname + `subtle-grouse`, and the host key `ssh -v` matched. **The "redeployed after the key import" + hypothesis is REFUTED** -- the machine booted 2026-07-31 00:58, AFTER the 2026-07-30 import, + and still received only Juju keys; deploy-time `cloud-config.txt` carries + `ssh_authorized_keys:` with `Juju:juju-client-key` ONLY, and `juju ssh-keys -m controller` + returns "No keys to display". `juju ssh -m controller 0` already works, so operator ssh is a + convenience, not a requirement. + **SEC-026 CONTROL (1) IS DISCHARGED BY MEASUREMENT, BOTH SIDES:** the rack lists ONE + credential while voffice1 lists `vr1-dc0-cred, vr1-dc1-cred` -- the forbidden whole-store + copy did NOT happen. **No new credential residency and NO new security-ledger row is owed**; + every location the proposed fix touches is already registered in + `creds-manifests/vm-secret-locations`. + **ONLY HYGIENE IS OWED, NOT A FIX:** `juju controllers` shows a dangling `Model: vr1-dc0` + pointing at the model destroyed on 2026-07-31, so a bare `juju status` errors; + `juju switch controller` clears it. **It is NOT an `add-model` precondition** -- measured, + the runbook's own guard shape (`juju models --format json | jq -r '.models[]?.name'` -> + `admin/controller`) works regardless. + **FOUR REAL GAPS FOUND DOWNSTREAM, LOGGED NOT FIXED.** **(G4, and it is the one that bites: + the `openstack` CLI is MISSING ON THE RACK** -- three instruments agree. That is a D-138 + definition-of-done gap which blocks `phase-03`+, and it is the SAME item as F1 from the + 2026-07-30 sweep, now re-confirmed on the D-138 host rather than on voffice1.) **(G3)** + `overlays/vr1-dc0-octavia-pki.yaml` is ABSENT from the vcloud repo tree (it is gitignored + PKI material, present 0600 on the rack) while `scripts/pre-flight-checks.sh:161` hard-fails + on it -- so **preflight CANNOT PASS on vcloud by construction**; code-cited, not executed. + **(G2)** `~/repo-stage` on the rack is a **hand-staged 9-file copy with NO `.git`** -- 8 of 9 + byte-identical to the repo by sha256, so it works today, but the D-138 client host's deploy + input has NO PROVENANCE. This is a THIRD copy beside the two clones the 2026-07-31 entry + was careful to verify at `c58bf95`, and the 07-27 stale-clone hazard is why that mattered. + **(G1)** `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:379,398` still label the add-model and + spaces steps **"voffice1"**, stale against D-138 -- and voffice1 measurably has the binary + but `juju controllers` -> "No controllers registered", so following the runbook as written + fails. DOCFIX owed, not numbered. + **DECLARED UNMEASURED, and it is the DISTAL half of the root cause:** whether MAAS user + `juju-vr1-dc0` has an imported ssh key. Four attempts failed and the reason is recorded -- + there is **no usable maas CLI profile anywhere** (vcloud has none; voffice1's `~/.maascli.db` + is ZERO BYTES with no `profiles` table, the inert residue already noted on 2026-07-30). Does + not change the fix: the proximate cause was measured directly on the machine. **The + PreToolUse guard also refused a command that would print the MAAS API key (DOCFIX-016), and + it was NOT retried in an altered shape** -- so `~/vr1-dc0-creds/` contents and the rack's + maas-profile state were not enumerated, and SEC-028's distribution claim is carried from the + ledger rather than re-measured. - Project: Omega Cloud, VR1 DC-DC rehearsal -- a two-DC + Office1-headend virtual rehearsal on KVM (vcloud host), rehearsing the future bare-metal Roosevelt deployment (D-100, `docs/design-decisions.md:1946`). diff --git a/docs/audit/stage5-juju-client-blocker-20260801.txt b/docs/audit/stage5-juju-client-blocker-20260801.txt new file mode 100644 index 0000000..31ad504 --- /dev/null +++ b/docs/audit/stage5-juju-client-blocker-20260801.txt @@ -0,0 +1,391 @@ +STAGE 5 -- "juju-client blocker" DIAGNOSIS (read-only) +Captured 2026-08-01 by agent session; repo at d555b90, branch dc-dc-stage5-preconditions. +Governing: D-138 (RULED 2026-07-30) + D-132 amendment. Ledger: SEC-026, SEC-028. +NOTHING IN THIS CAPTURE WAS MUTATED. All fix commands below are UNEXECUTED PROPOSALS. + +===================================================================== +VERDICT +===================================================================== +Both reported blockers are NOT blockers. They are D-138 working as ruled. + + (a) "no juju client on the jumphost" -> CORRECT under D-138. vcloud is not + where the cloud-facing client belongs. The client EXISTS, registered and + LIVE, on the dc0 rack (vvr1-dc0), which is the D-138 client host. + (b) "ssh vr1-dc0-juju -> Permission denied (publickey)" -> CORRECT and expected. + The offered key was never installed on that machine and is not needed; + `juju ssh -m controller 0` from the rack is the working access path and + was used throughout this diagnosis. + +`juju add-model` is reachable from the rack TODAY. The real remaining gaps are +downstream of add-model and are listed in section 5. + +===================================================================== +1. ROOT CAUSE OF THE SSH REFUSAL -- MEASURED, NO INFERENCE +===================================================================== +Alias (vcloud ~/.ssh/config.d/vr1-sites): + Host vr1-dc0-juju HostName 10.12.8.5 User ubuntu + ProxyJump vr1-dc0-rack IdentityFile ~/vr1-office1-creds/office1_svc_ed25519 + +`ssh -v -o BatchMode=yes vr1-dc0-juju` -- the hop chain is CORRECT end to end: + hop 1 vcloud -> voffice1 OK + hop 2 -> 172.31.0.2 (rack), key vr1-dc0_svc_ed25519 SHA256:DBkeStCyf2qGi3SY... + "Server accepts key" / "Authenticated to 172.31.0.2 (via proxy)" + hop 3 channel_connect_stdio_fwd: 10.12.8.5:22 -- reached, TCP fine + "Host '10.12.8.5' is known and matches the ED25519 host key" + "Found key in /home/jessea123/.ssh/known_hosts:28" <- host key CURRENT + Offering office1_svc_ed25519 SHA256:iUYex2kmvtdGTPiMKRZ2Nfywqa8pWacVV1GEEx6sscI + "Authentications that can continue: publickey" + ubuntu@10.12.8.5: Permission denied (publickey). + +So: hostname, port, user, hop chain, per-hop identity selection and host key are +ALL correct and measured working. The single failing element is the final key. + +WHAT THE TARGET ACTUALLY ACCEPTS (read via `juju ssh -m controller 0`): + /home/ubuntu/.ssh/authorized_keys -- exactly 2 lines: + 4096 SHA256:VkipepSQPOmCyx1RS5HvM/F17U7c8HZAGO0LWHEgW/w Juju:juju-client-key (RSA) + 4096 SHA256:pBDEJsaS7fH/JLdVWm3A9aOlBN0jiYIv4J9AjVPk7YI Juju:juju-system-key (RSA) + mtime 2026-07-31 01:00:03 + SHA256:Vkipep... is byte-identical to the rack's + ~/.local/share/juju/ssh/juju_id_rsa.pub (juju-client-key). + The office1 key SHA256:iUYex2... is ABSENT. Refusal is therefore CORRECT. + +IDENTITY TIE -- the box read is the box that refused (hard rule 2, not "almost +certainly"). On `juju ssh -m controller 0`: + enp1s0 10.12.8.5/22 + enp2s0 10.12.4.5/22 + hostname: subtle-grouse + /etc/ssh/ssh_host_ed25519_key.pub -> 256 SHA256:Gr7kw5GTa+taeYiknHcLywc6Mjnwr82ly7gm2XetqXQ + That fingerprint is EXACTLY the host key `ssh -v` matched at known_hosts:28 for + 10.12.8.5. Same host, proven. `juju machines -m controller` reports machine 0 + at 10.12.4.5 / inst subtle-grouse -- the two addresses are two NICs on one VM. + +WHY THE KEY IS ABSENT -- deploy-time evidence, and one candidate RULED OUT: + /var/lib/cloud/instance/user-data.txt mtime 2026-07-31 00:58:39 + machine boot (`uptime -s`) 2026-07-31 00:58:15 + ssh keys found in user-data.txt 0 + /var/lib/cloud/instance/cloud-config.txt line 131 "ssh_authorized_keys:" + line 133 "Juju:juju-client-key" + grep for vr1-office1 / jessea123 in cloud-config.txt -> NO MATCH + vendor-cloud-config.txt / vendor-data.txt / vendor-data2.txt -> 0 ssh keys + `juju ssh-keys -m controller --full` -> "No keys to display." + `juju model-config -m controller authorized-keys` -> ERROR "authorized-keys" is + not a key of the currently targeted model (3.6 moved it to `juju ssh-keys`). + ~/.local/share/juju/bootstrap-config.yaml:48 authorized-keys: | -> one ssh-rsa + entry only (the juju client key). + + RULED OUT: "the machine was redeployed BEFORE the key import." The MAAS key + import into the new dc0 region was 2026-07-30; this machine deployed + 2026-07-31 00:58 -- AFTER the import -- and STILL received only Juju keys. + Timing is not the explanation. + + PROXIMATE CAUSE (measured): the deploy-time cloud-config supplied ONLY Juju's + own keys. Nothing ever put the office1 key on this VM. + + DISTAL CAUSE (NOT MEASURED -- declared, not asserted): either the MAAS user + juju uses in the dc0 region (juju-vr1-dc0, minted fresh per SEC-028) has no + imported ssh key, or juju's bootstrap cloud-config supersedes the user's key + set. I could NOT distinguish these -- see section 6. + +===================================================================== +2. WHERE THE JUJU CLIENT SHOULD LIVE, AND WHAT EXISTS -- ALL MEASURED +===================================================================== +D-138 RULED (1): tools that dial the CLOUD at L3 -- the Juju client and the +phase-03..06 `openstack` CLI -- run from INSIDE the DC. voffice1 stays Plane-2 +only for application-proxied / qemu+ssh work. => the dc0 client host is the dc0 +RACK (vvr1-dc0), consistent with SEC-028, which distributed the juju service +credential to the rack precisely because "D-138 moved the cloud-facing Juju +client INTO the DC -- so `rack` is now a credential-bearing host-role." + + COMPONENT vcloud (jumphost) vr1-dc0-rack (D-138 host) + --------------------- ----------------------- ----------------------------- + juju binary ABSENT (`which juju` PRESENT /snap/bin/juju + rc=1; snap list shows 3.6.27-genericlinux-amd64 + only core24 + snapd) snap juju 3.6.27 rev 35621 3/stable + ~/.local/share/juju ABSENT (No such file) PRESENT, 8 entries + controller reg. n/a PRESENT: vr1-dc0-controller* + credential n/a PRESENT: exactly ONE + repo scripts FULL repo (this clone) PARTIAL hand-stage, no .git + openstack CLI not checked (not the MISSING <-- D-138 gap + D-138 host) + maas CLI ABSENT, no ~/.maascli.db (not checked) + + vcloud negative instrumented: `which juju` (rc=1), `snap list` (2 rows, + neither juju), `ls ~/.local/share/juju/` (No such file or directory). + Three independent instruments agree -- this is not a mis-scoped probe. + +LIVE PROOF the rack client works (run 2026-08-01): + $ juju whoami + Controller: vr1-dc0-controller / Model: vr1-dc0 / User: admin + $ juju controllers + vr1-dc0-controller* vr1-dc0 admin superuser vr1-maas 1 model 1 node 3.6.27 + $ juju models + controller vr1-maas maas available 1 machine 4 cores 1 unit admin + Last connection: just now <-- the controller IS UP and reachable + $ juju machines -m controller + 0 started 10.12.4.5 subtle-grouse ubuntu@22.04 default Deployed + $ juju show-controller + controller-uuid 5556adb3-9cce-4b4e-8e64-f9c0a14c08f7 + api-endpoints ['10.12.4.5:17070','10.12.8.5:17070','[fd50:840e:74e2:220::5]:17070'] + ~/.local/share/juju/clouds.yaml + vr1-maas / type maas / oauth1 / endpoint http://10.12.8.6:5240/MAAS + (the DC-LOCAL region -- D-132 amendment satisfied; no cross-fiber MAAS dep) + + NOTHING IS MISSING for `add-model`: binary, registration and credential are + all present and demonstrably functional against the live controller. + +===================================================================== +3. SEC-026 CONTROL (1) -- DISCHARGED BY MEASUREMENT +===================================================================== +SEC-026 requires: "Verify after any client rebuild that `juju credentials +--client` on a DC host lists exactly one." + + $ juju credentials --client (on vr1-dc0-rack) + Client Credentials: + Cloud Credentials + vr1-maas vr1-dc0-cred + + EXACTLY ONE. No vr1-dc1-cred present. The per-DC isolation SEC-018/-019 exist + to create is INTACT on the dc0 rack, measured 2026-08-01. + + BOTH SIDES OF THE CONTROL MEASURED. SEC-026 warns that voffice1's client store + "holds BOTH vr1-dc0-cred and vr1-dc1-cred" and that copying it to a DC host + "would destroy the per-DC isolation." Confirmed on voffice1 2026-08-01: + $ juju credentials --client (on voffice1) + Client Credentials: + Cloud Credentials + vr1-maas vr1-dc0-cred, vr1-dc1-cred + Two on voffice1, ONE on the rack -> the forbidden whole-store copy did NOT + happen. The control is not merely unviolated, it is measured on both sides. + + voffice1 juju state, VERIFIED (previously carried, now measured 2026-08-01): + which juju -> /snap/bin/juju + juju version -> 3.6.27-genericlinux-amd64 + juju controllers-> ERROR No controllers registered. + ~/.local/share/juju/ -> full store present (accounts, bootstrap-config, + clouds, controllers, credentials, models, + public-clouds, ssh) + So voffice1 has a BINARY and a CREDENTIAL STORE but no controller + registration -- it cannot perform the runbook's voffice1-labelled juju steps. + This is the measured basis for gap G1. + +CREDENTIAL RESIDENCY OF THE PROPOSED FIX: NONE NEW. The fix in section 4 moves +no key, copies no credential, and creates no new residency. Every credential +location it touches is ALREADY REGISTERED in creds-manifests/vm-secret-locations: + line 123 rack vr1-dc0 ssh vr1-dc0-rack:~/vr1-dc0-creds/* + line 124 rack vr1-dc0 ssh vr1-dc0-rack:~/.local/share/juju/* + line 127 rack vr1-dc0 ssh vr1-dc0-rack:~/repo-stage/overlays/vr1-dc0-octavia-pki.yaml +NO new security-ledger row is owed for the section-4 fix. + +===================================================================== +4. PROPOSED MINIMAL FIX -- UNEXECUTED, FOR OPERATOR APPROVAL +===================================================================== +FIX-0 (NO-OP, state it explicitly): do NOT install a juju client on vcloud and +do NOT add the office1 key to the controller VM. Both would work AGAINST D-138 +and against the measured facts. The reported blocker requires no key movement. + +FIX-1 -- HYGIENE ONLY; NOT REQUIRED FOR add-model (see the guard measurement +below before approving). Clears the dangling current-model pointer left by the +2026-07-31 teardown. + MEASURED SYMPTOM (on vr1-dc0-rack): + $ juju status + ERROR model vr1-dc0-controller:admin/vr1-dc0 not found + ~/.local/share/juju/models.yaml has `current-model: admin/vr1-dc0` while its + `models:` map contains ONLY admin/controller -- the destroyed model's + pointer outlived the model. + IMPORTANT -- this does NOT block Step 3.5. Measured, in the runbook's own + guard shape: + $ juju models --format json | jq -r '.models[]?.name' + admin/controller + so the guard's `grep -qx "$DC_MODEL"` correctly finds no match and + `juju add-model vr1-dc0` proceeds. FIX-1 is hygiene for bare `juju status`, + not a precondition. Proposed command, run ON vr1-dc0-rack: + juju switch controller + Minimal because it uses juju's own state machine rather than hand-editing + models.yaml, and `add-model` will re-point current-model anyway. + REVERT: juju switch vr1-dc0 (once that model exists) + +FIX-2 -- SEPARATE, OPTIONAL, DO NOT BATCH WITH FIX-1. Operator ssh to the +controller VM. Only if the operator wants `ssh vr1-dc0-juju` to work as a +convenience; it is NOT required for Stage 5, because `juju ssh -m controller 0` +already works and was the instrument for this entire diagnosis. + PREREQUISITE, MEASURED AS ABSENT -- this is a SEPARATE step, not part of the + command. The office1 PUBLIC key is NOT on the rack. Measured 2026-08-01: + vr1-dc0-rack $ ls -la ~/.ssh/ + authorized_keys, known_hosts, known_hosts.old -- NO key files + vr1-dc0-rack $ find ~ -maxdepth 3 -name '*.pub' + /home/jessea123/vr1-dc0-creds/maas-virsh_ed25519.pub (only this one) + vcloud $ ls -la ~/vr1-office1-creds/ + office1_svc_ed25519.pub 97 bytes, mode 0644 <- source, exists here + So FIX-2 requires FIRST staging that .pub to the rack (public key, not secret), + and only then: + juju add-ssh-key -m controller "" + The `-m controller` is REQUIRED and is not optional decoration: `add-ssh-key` + targets the CURRENT model, which is the dangling admin/vr1-dc0, so the bare + form fails with the same "model not found" documented under FIX-1. + NOT RECOMMENDED as part of this fix. It (a) adds an operator key to a + juju-managed authorized_keys set, (b) requires the office1 PUBLIC key to be + placed on the rack, and (c) widens shell access to the controller beyond the + juju-mediated path. If declined, the correct standing answer is: reach the + controller VM with `juju ssh -m controller 0` from the rack, and treat the + vr1-dc0-juju alias as non-functional-by-design. + If FIX-2 IS taken, the vr1-dc0-juju alias comment in + scripts/site-ssh-config.sh needs updating -- the fragment currently asserts + DC nodes carry the office1 key, which is FALSE for juju-provisioned machines + (measured above). That is a DOCFIX, not part of the mutation. + +===================================================================== +5. REAL REMAINING GAPS -- DOWNSTREAM OF add-model. LOGGED, NOT FIXED. +===================================================================== +G1 -- HOST SPLIT THE RUNBOOK DOES NOT DESCRIBE. + runbooks/dc-dc-phase4-juju-bundle-per-dc.md:379 labels the add-model block + **MUTATION -- voffice1 -- create THIS DC's model (guarded)** + and :398 labels the spaces gate **CHECK (read-only) -- voffice1 --**. + Both are STALE w.r.t. D-138, which is dated 2026-07-30 and moved the juju + client into the DC. voffice1 has a juju BINARY (3.6.27) but `juju controllers` + there returns ERROR No controllers registered -- so the runbook as written + targets a host that cannot perform the step. DOCFIX owed. NOT EDITED here + (a) because hard rule 1 forbids adjacent changes mid-step and (b) because the + DOCFIX must be sized against every other voffice1-pinned juju reference, not + just these two lines. Grep next-free DOCFIX-NNN before assigning a number. + +G2 -- scripts/juju-spaces-check.sh NEEDS repo + juju ON THE SAME HOST. + The runbook runs it immediately after add-model. It is present on the rack at + ~/repo-stage/scripts/juju-spaces-check.sh -- and the staged copy is IDENTICAL + to the repo copy (sha256 adb9dbd38a27e663083f37b64e17ca98d870180f010b8d499a2 + 9f3fbfe7d1d5d on BOTH vcloud and vr1-dc0-rack). jq and python3 3.12.3 are + present on the rack. So this step CAN run today. But ~/repo-stage is a + 9-file HAND-STAGED copy with NO .git (verified: `find / -xdev -maxdepth 5 + -name .git -type d` on the rack returned NOTHING), so it has no provenance + and will drift silently. Drift measured 2026-08-01: 8 of 9 files byte-identical + to the repo; the 9th (vr1-dc0-octavia-pki.yaml) exists ONLY on the rack. + +G3 -- overlays/vr1-dc0-octavia-pki.yaml IS ABSENT FROM THE vcloud REPO TREE. + Two independent instruments agree: `ls -la overlays/` lists 6 files and not + this one; `sha256sum overlays/vr1-dc0-octavia-pki.yaml` -> No such file. + It IS present on the rack, mode -rw------- (0600), 4682 bytes, and it is + gitignored by design (.gitignore:46 `overlays/*octavia-pki.yaml`). + vm-secret-locations registers it at BOTH locations (line 90 headend + `/overlays/vr1-dc0-octavia-pki.yaml`, line 127 rack), so the register is + correct and the rack copy's 0600 handling is correct. + CONSEQUENCE -- CITED FROM CODE, NOT EXECUTED. preflight.sh runs on the repo + host (vcloud). Its P4 delegate hard-FAILS on the missing file: + scripts/pre-flight-checks.sh:155-162 (CHECK 0) + OVL="overlays/${DC}-octavia-pki.yaml" + if [ -f "$OVL" ]; then ... else + fail "MISSING $OVL (gitignored secret; place it or regenerate via + runbook Step 1.0-GEN before deploy)" + `fail` -> preflight exit 1 -> "do NOT deploy". So the mandated pre-add-model + gate CANNOT PASS on vcloud as the tree stands. This is a CODE READ; I did not + run preflight.sh (out of mandate scope), so the FAIL is cited, not observed. + CORRECTION TO AN EARLIER DRAFT OF THIS CAPTURE: P7 does NOT depend on this + overlay and must not be cited here. scripts/octavia-pki.sh reads + overlays/${SITE}-vips.yaml (line 169) and WRITES the octavia-pki overlay; it + is the generator, not a consumer. Only P4 CHECK 0 gates on the file's presence. + Note preflight.sh:144-146 SILENTLY SKIPS a missing overlay at P2 + (`[ -f "$_ovl" ] && P2_ARGS=...`), so P2 would validate a bundle merged + WITHOUT the octavia overlay and still report on it -- the failure surfaces at + P4/P7, not P2. NOT INVESTIGATED FURTHER -- out of mandate scope. Flagged so it + is not discovered at deploy time. + +G4 -- `openstack` CLI is MISSING on the dc0 rack. + Measured with THREE instruments, because `which` in a non-interactive ssh + shell misses ~/.local/bin: `which openstack` -> MISSING; `ls ~/.local/bin` + -> No such file or directory (the dir does not exist at all); `snap list | + grep -i openstack` -> no openstack snap. All three agree. D-138 names the + phase-03..06 `openstack` CLI work explicitly as DC-resident. This is not a + Stage-5 add-model blocker but IS a D-138 definition-of-done gap that will + block phase-03. Also missing on the rack: yq, charmcraft. Present: python3 + 3.12.3, jq, git. + +===================================================================== +6. WHAT I COULD NOT MEASURE, AND HOW I KNOW I COULD NOT +===================================================================== +UNMEASURED: whether MAAS user `juju-vr1-dc0` in the dc0 region has an imported +ssh key. This is the DISTAL half of section 1's root cause. It is DECLARED, not +guessed. Attempts and why each failed: + - `sudo -u postgres psql -d maasdb` on vr1-dc0-maas -> "sudo: unknown user + postgres" (snap postgresql runs as _daemon_). + - `/snap/bin/postgresql.psql -U postgres -h 127.0.0.1` -> asks for a password; + NOT pursued (would mean handling credential material). + - `sudo -u _daemon_ /snap/bin/postgresql.psql` -> snap refuses: "is not a snap + cgroup for tag snap.postgresql.psql". + - maas CLI profile: ABSENT on vcloud (no ~/.maascli.db) and EFFECTIVELY absent + on voffice1 -- ~/.maascli.db there is ZERO BYTES (Jul 29) and sqlite reports + "no such table: profiles". vr1-dc0-maas has no ~/.maascli.db either. + Operator command that would settle it (UNEXECUTED, needs a logged-in profile + pointed at http://10.12.8.6:5240/MAAS -- verify the endpoint BEFORE trusting + the answer; a profile aimed at 10.10.0.20 is the OFFICE1 region and would give + a confidently wrong answer): + maas users read + maas sshkeys read + This gap does NOT change the fix. Section 4 is correct either way, because the + proximate cause (only Juju keys were ever supplied to this machine) is measured + directly on the machine. + +REFUSED BY THE PERMISSION HARNESS -- verbatim, and NOT retried in an altered +shape (the query was `ls ~/.maascli.db; maas list; ls -la ~/vr1-dc0-creds/` on +vr1-dc0-rack): + PreToolUse:Bash hook error: [python3 "$CLAUDE_PROJECT_DIR"/.claude/hooks/ + guard-destructive.py]: BLOCKED by .claude/hooks/guard-destructive.py: + prints the MAAS API key (DOCFIX-016); use 'maas admin ...' directly + Consequence: the rack's maas-profile state and the contents of + ~/vr1-dc0-creds/ were NOT enumerated this session. SEC-028's claim that the + credential was distributed to the rack is therefore CARRIED FROM THE LEDGER, + not re-measured here. + +INSTRUMENT CORRECTION MADE MID-SESSION (recorded because the first result was a +false negative of exactly the shape this repo has been bitten by): an initial +`find ~ -maxdepth 2 -name preflight.sh` on the rack could NOT have found a repo +clone -- repo scripts sit at depth 3. Re-run as `find / -xdev` over unique +filenames, it FOUND ~/repo-stage. The corrected result is what section 5 uses. +Likewise `grep -rn ... creds-manifests/vm-secret-locations/` (trailing slash on +a FILE) returned nothing and was re-run without the slash, which found all six +rack rows. Neither false negative reached a conclusion. + +===================================================================== +7. COMMANDS RUN (all read-only; none mutated state) +===================================================================== +vcloud: which juju; snap list; ls ~/.local/share/juju; ls ~/ | grep creds; + ssh -v -o BatchMode=yes vr1-dc0-juju; sha256sum <9 repo files>; + ls -la overlays/; git check-ignore -v; grep over docs/ and + creds-manifests/vm-secret-locations; sed over runbook + preflight.sh; + python3 sqlite read of ~/.maascli.db (endpoint URL only, no keys) +vr1-dc0-rack: hostname; which juju; juju version; snap list; ls ~/.local/share/juju; + juju whoami / controllers / models / credentials --client / + show-controller / machines -m controller / ssh-keys -m controller / + model-config -m controller authorized-keys / models --format json; + cat clouds.yaml models.yaml; grep accounts.yaml (password excluded); + ssh-keygen -lf juju_id_rsa.pub; find / -xdev for repo + .git; + sha256sum <9 staged files>; ls -la repo-stage; which openstack et al +subtle-grouse (via `juju ssh -m controller 0`): authorized_keys fingerprints + + comments + mtime; ip -4 -o addr; ssh-keygen -lf host key; hostname; + stat/grep over /var/lib/cloud/instance/* +vr1-dc0-maas: hostname; snap list; ps; ss -lntp; three failed psql attempts + ls -la ~/vr1-office1-creds/ (names/modes only); grep+sed over + scripts/pre-flight-checks.sh and scripts/octavia-pki.sh; + bash scripts/repo-lint.sh; git status --short; git check-ignore -v +vr1-dc0-rack (cont.): ls -la ~/.ssh/; find ~ -maxdepth 3 -name '*.pub'; + ls ~/.local/bin; snap list | grep openstack +voffice1: ls ~/.maascli.db; python3 sqlite read (no profiles table); + which juju; juju version; juju controllers; juju credentials --client; + ls ~/.local/share/juju/ + +===================================================================== +8. REPO-LINT STATE AT CAPTURE TIME -- NOT CAUSED BY, NOT FIXABLE BY, THIS WORK +===================================================================== + $ bash scripts/repo-lint.sh + [WARN] L1 docs/design-decisions.md: 239 non-ASCII byte(s) (legacy carve-out) + [FAIL] L10 pending change set modifies state-bearing surface(s) + [docs/audit/: docs/audit/d139-carve-dryrun-20260801.txt, + docs/audit/stage5-juju-client-blocker-20260801.txt] without touching + docs/CURRENT-STATE.md -- GA-R1 rule 8 (C1) + FAIL: repo lint (1 fail, 1 warn, 645 files scanned) EXIT=1 + + L10 was ALREADY FAILING before this capture existed: the other file it names, + docs/audit/d139-carve-dryrun-20260801.txt, is a concurrent agent's and has + mtime 00:18 against this file's 00:25. This capture is now ALSO named in the + finding but did not create it. + NOT RESOLVED HERE, deliberately: clearing L10 requires touching + docs/CURRENT-STATE.md in the SAME commit, and this session was scoped out of + both that file and any git add/commit/push. Resolution must be coordinated by + whoever owns docs/CURRENT-STATE.md for this change set. diff --git a/docs/changelog-20260731-snap-proxy-apply-ipv6.md b/docs/changelog-20260731-snap-proxy-apply-ipv6.md index 7cdb81f..2e6a437 100644 --- a/docs/changelog-20260731-snap-proxy-apply-ipv6.md +++ b/docs/changelog-20260731-snap-proxy-apply-ipv6.md @@ -479,3 +479,74 @@ the original deciding-reason paragraph and the flag. Reverting does not un-refute the measurement, and the ruling note quotes an operator utterance -- so a revert would DESTROY a GA-R5 record and must not be done casually. + +--- + +## Item 12 -- THE "JUJU CLIENT BLOCKER" IS NOT ONE. Correction to item 9's finding 9d. + +**What.** `docs/audit/stage5-juju-client-blocker-20260801.txt` (NEW, 391 lines) + +`docs/CURRENT-STATE.md`. No script, no cloud mutation, no key moved. + +**Item 9 recorded finding 9d as "BLOCKS the next phase". That was WRONG and is corrected +here.** The juju client exists, is registered and is live -- on the dc0 RACK, which is +exactly where D-138 places it. **Stage 5 can reach `add-model` today.** Verified from the +main session rather than accepted from the agent: juju `3.6.27-genericlinux-amd64` at +`/snap/bin/juju`; controller `vr1-dc0-controller*` admin/superuser on cloud `vr1-maas`; the +`controller` model reading **"Last connection: just now"**; and client credentials listing +**exactly one** entry. + +**Root cause of the ssh refusal, measured ON THE MACHINE.** The alias is correct in every +part. `subtle-grouse`'s `authorized_keys` holds exactly two lines, both Juju's -- so the +`office1_svc_ed25519` the alias offers is genuinely absent, and the refusal is right. **The +obvious hypothesis was REFUTED rather than assumed:** the machine booted AFTER the 2026-07-30 +region key import and still received only Juju keys, because deploy-time `cloud-config.txt` +carries `Juju:juju-client-key` alone and `juju ssh-keys -m controller` reports none. `juju ssh +-m controller 0` already works, so operator ssh is convenience, not requirement. + +**A NEGATIVE FINDING WAS TIED TO IDENTITY BEFORE IT WAS TRUSTED** -- machine 0's two addresses, +hostname and host key were all matched to what `ssh -v` actually reached. This repo's standing +instrument rule is that a confident negative needs proof the instrument was pointed at the +right thing; here it was, and it is recorded. + +**SEC-026 control (1) discharged by measurement on BOTH sides:** the rack lists ONE credential +while voffice1 lists two, so the forbidden whole-store copy did not happen. **No new +credential residency and NO new ledger row is owed** -- every location the fix touches is +already in `creds-manifests/vm-secret-locations`. + +**Only hygiene is owed:** a dangling `current-model: admin/vr1-dc0` (the model destroyed +2026-07-31) makes bare `juju status` error; `juju switch controller` clears it. Measured NOT +to be an `add-model` precondition -- the runbook's own guard shape works regardless. **NOT +EXECUTED** (hard rule 3; it is a state change on the live client, presented not taken). + +**FOUR GAPS LOGGED NOT FIXED**, in consequence order: +- **G4 -- the `openstack` CLI is MISSING ON THE RACK** (three instruments agree). A D-138 + definition-of-done gap that blocks `phase-03`+. **Same item as F1 from the 2026-07-30 + sweep**, now re-confirmed on the D-138 host rather than on voffice1 -- so it has survived two + sweeps and should stop being re-discovered. +- **G3 -- preflight CANNOT PASS on vcloud by construction.** + `overlays/vr1-dc0-octavia-pki.yaml` is gitignored PKI material (0600 on the rack, absent from + the vcloud tree) and `scripts/pre-flight-checks.sh:161` hard-`fail`s on it. Code-cited, not + executed. This is a gate that cannot pass on one host and can on another -- the + host-dependence class this repo already records for `dc-mirror.sh` and P5/P7. +- **G2 -- the D-138 client host's deploy input has NO PROVENANCE.** `~/repo-stage` on the rack + is a hand-staged 9-file copy with **no `.git`**; 8 of 9 are byte-identical to the repo by + sha256, so it works today. It is a THIRD copy beside the two clones the 2026-07-31 entry + verified at `c58bf95` before trusting any gate output, and the 2026-07-27 stale-clone hazard + is precisely why that verification existed. +- **G1 -- `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:379,398` still say "voffice1"** for the + add-model and spaces steps, stale against D-138. Measured: voffice1 HAS the binary but + `juju controllers` returns "No controllers registered", so the runbook as written FAILS. + DOCFIX owed, deliberately not numbered mid-flight. + +**DECLARED UNMEASURED, with the reason:** whether MAAS user `juju-vr1-dc0` has an imported ssh +key -- the DISTAL half of the root cause. There is no usable maas CLI profile anywhere +(vcloud has none; voffice1's `~/.maascli.db` is ZERO BYTES with no `profiles` table, the inert +residue already recorded 2026-07-30). It does not change the fix, because the proximate cause +was measured on the machine directly. **The PreToolUse guard refused a command that would +print the MAAS API key (DOCFIX-016) and it was NOT retried in an altered shape** -- so +`~/vr1-dc0-creds/` and the rack's profile state are unenumerated, and SEC-028's distribution +claim is carried from the ledger rather than re-measured. Both the guard firing and the +non-retry are the correct outcomes and are recorded as such. + +**Revert.** `git rm docs/audit/stage5-juju-client-blocker-20260801.txt` and drop the +CURRENT-STATE paragraph. Nothing live was changed, so there is nothing else to undo.