|
Stage 5 dc0: remediate ceph-mon/2 + ceph-radosgw/0 "allocating" (apt CLOSE-WAIT); durable capture
Live-cloud, operator-gated (2 exchanges). Two units stuck `allocating` were root-caused NOT to a down proxy (both dc0 proxies PASS; security.ubuntu.com 200/0.67s) but to cloud-init `apt-get update` parked in CLOSE-WAIT to 10.12.8.4:3142 for ~12.5h (apt has no client read timeout) since the 08-04 redeploy. Rebuilt each from the rack (D-138): remove-unit --no-prompt -> remove-machine --force --no-prompt (REQUIRED: remove-unit does NOT cascade to a never-provisioned dead-agent container) -> add-unit --to <bundle placement>. Both fresh containers' cloud-init finished (~211s); mons bootstrapped 3/3, 4 OSDs active, storage cascade cleared. A 2nd apt-cacher-ng mode (radosgw-hacluster 404 on a rotated point-release -> exit 100) SELF-HEALED via juju hook retry. Measured after-state 17:34:08Z: census 62 active; remaining non-active all deferred-by-design + gss + barbican-vault settling. Durable capture (operator-directed "a then b"): - docs/audit/stage5-dc0-ceph-remediation-20260805.txt: transcribed capture (NOT script(1)); the before-state (CLOSE-WAIT sockets, ~45000s etimes, both containers) is unrecoverable and lives only here. - docs/CURRENT-STATE.md: dated status block; measured 62 active (NOT the unmeasured 40/47, C2); records the --force fact and the phase-03 Step-3.1 gate defect (expects 1, VR1 has 4 deferred + gss) as durable finding. - runbooks/appendix-A-troubleshooting.md: NEW entry for the symptom pair (both modes + gated remove/--force/re-add remediation). Drafted; operator-reviewed. - docs/changelog-20260805-stage5-dc0-ceph-remediation.md: session changelog (separate same-day file from the skill-DOCFIX changelog, disclosed in header); F2/F3/F4 owed, unnumbered. repo-lint 0 fail / 1 legacy warn; ledger-scan DOCFIX next-free still 210 (no token leaked); all touched files ASCII-clean. Status lives in docs/CURRENT-STATE.md ONLY. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728 |
|---|
|
|
| docs/CURRENT-STATE.md |
|---|
| docs/audit/stage5-dc0-ceph-remediation-20260805.txt 0 → 100644 |
|---|
| docs/changelog-20260805-stage5-dc0-ceph-remediation.md 0 → 100644 |
|---|
| runbooks/appendix-A-troubleshooting.md |
|---|