diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 10b6366..0acfbdb 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -297,6 +297,14 @@ domain (grant member or delete); Role-dropdown question on the dashboard-override test (why load-balancer_admin was submitted) -- evidence folded into the D-076 draft (worktree, uncommitted). +- **TestCluster NoValidHost + decommission audit (addendum 40):** devteam + 1+3 cluster stalled on RAM capacity (NOT quota); operator deleted it, + devteam re-created 1+1 (TestCluster2). QUEUED RULINGS from the audit: + (a) beta-cluster teardown (~17GB reclaim) -- cross-stream, main-chat + owns the resize-acceptance role; (b) lbtest LB leftover reclaim via + kubectl (beta kubeconfig, main-chat asset); (c) foil1 offboard + (RECOMMENDED -- dormant shell); (d) quota-vs-capacity expectation + (trim testcloud quotas or document in clientdocs) -- unruled. ## REMAINING OPEN WORK (single list for session resume; 2026-07-07, updated post-devteam-window) diff --git a/docs/v1-redeploy-changelog.md b/docs/v1-redeploy-changelog.md index aa35752..a34e2d3 100644 --- a/docs/v1-redeploy-changelog.md +++ b/docs/v1-redeploy-changelog.md @@ -2914,4 +2914,43 @@ - Numbers consumed: DOCFIX-121, DOCFIX-122. REVERT: git checkout HEAD~ -- clientdocs runbooks/appendix-A-troubleshooting.md docs/session-ledger.md docs/v1-redeploy-changelog.md; then bash scripts/repo-lint.sh --record-clientdocs-sweep (re-pin the receipt); re-render the two package files from the reverted templates (or restore from the operator's previous copy); no cloud-side change to revert. +### 2026-07-08 (addendum 40, jumphost stream) -- devteam cluster NoValidHost diagnosed (capacity, not quota); decommission audit + recommendations (no numbers consumed) + +Read-only diagnosis + audit; the only mutation this window was the operator's +own delete of the wedged cluster. Nothing agent-executed. + +- **TestCluster CREATE stall root-caused:** devteam 1+3 gp.mid cluster + (created 00:18, CAPI driver); control plane + 1 worker ACTIVE, workers + 2-3 ERROR with nova fault "No valid host was found" (verbatim, by UUID -- + admin name-lookup is project-scoped, same trap as template show). NOT + quota (4x gp.mid = 8 cores/32768MB vs 10/20/51200 limits; quota rejects + at the API, never lands ERROR). Physical RAM was the limiter: free + ~3.4/8.5/7.5 GB across the 3 hypervisors; gp.mid=8192MB, so after 2 + landed no host could fit another. Operator deleted TestCluster; + devteam re-created as TestCluster2 1+1 (the fitting shape; matches + stage6/H2). Also noted: cluster was built from a GUI template + "TestTemplate", not the walkthrough's "test1". +- **FINDING (logged, unruled): tenant quota over-promises testcloud + capacity** -- devteam hit NoValidHost at under 2/3 of its RAM quota + (51200MB vs ~19GB schedulable, fragmented). Fine at Roosevelt scale; + misleading here. Options for a later ruling: trim testcloud tenant + quotas to schedulable reality, or set the expectation in clientdocs. +- **Decommission audit (operator-requested, read-only):** cloud-wide + running set = 3x beta-cluster nodes, capi-mgmt-v2, 3 amphorae; zero + stray/SHUTOFF servers, zero volumes. Domains: 5 platform + devteam + (live) + beta + foil1; acme fully gone. Recommendations queued: + (1) lbtest LB leftover (beta): reclaim ~1GB + FIP 10.12.6.97 -- clean + path is kubectl delete service from beta's kubeconfig (k8s-managed; + direct Octavia delete would dangle/recreate), main-chat asset; + (2) beta-cluster teardown = the big reclaim (~17GB, doubles free RAM) + -- BLOCKED on the cross-stream ruling that its resize-acceptance role + is done (ledger marks node_count=2 deliberate); + (3) foil1: dormant shell (no instances/LBs/FIPs/volumes) -- RECOMMEND + offboard via tenant-offboard.sh (retires a live domain, closes its + waived Horizon rows; re-onboarding is proven cheap if a foil is needed + again); + (4) capi-mgmt-v2 and platform domains: KEEP (CAPI control plane). +- Numbers consumed: none. +REVERT: documentation-only entry (this file + docs/session-ledger.md); git checkout HEAD~ -- docs/v1-redeploy-changelog.md docs/session-ledger.md. + Next-free after this push (per scan, keep this line unwrapped): D-076, DOCFIX-121, BUNDLEFIX-012.