Stage 5 dc0: remediate ceph-mon/2 + ceph-radosgw/0 "allocating" (apt CLOSE-WAIT); durable capture
...
Live-cloud, operator-gated (2 exchanges). Two units stuck `allocating` were root-caused
NOT to a down proxy (both dc0 proxies PASS; security.ubuntu.com 200/0.67s) but to
cloud-init `apt-get update` parked in CLOSE-WAIT to 10.12.8.4:3142 for ~12.5h (apt has no
client read timeout) since the 08-04 redeploy. Rebuilt each from the rack (D-138):
remove-unit --no-prompt -> remove-machine --force --no-prompt (REQUIRED: remove-unit does
NOT cascade to a never-provisioned dead-agent container) -> add-unit --to <bundle
placement>. Both fresh containers' cloud-init finished (~211s); mons bootstrapped 3/3, 4
OSDs active, storage cascade cleared. A 2nd apt-cacher-ng mode (radosgw-hacluster 404 on a
rotated point-release -> exit 100) SELF-HEALED via juju hook retry. Measured after-state
17:34:08Z: census 62 active; remaining non-active all deferred-by-design + gss +
barbican-vault settling.
Durable capture (operator-directed "a then b"):
- docs/audit/stage5-dc0-ceph-remediation-20260805.txt: transcribed capture (NOT script(1));
the before-state (CLOSE-WAIT sockets, ~45000s etimes, both containers) is unrecoverable
and lives only here.
- docs/CURRENT-STATE.md: dated status block; measured 62 active (NOT the unmeasured 40/47,
C2); records the --force fact and the phase-03 Step-3.1 gate defect (expects 1, VR1 has 4
deferred + gss) as durable finding.
- runbooks/appendix-A-troubleshooting.md: NEW entry for the symptom pair (both modes +
gated remove/--force/re-add remediation). Drafted; operator-reviewed.
- docs/changelog-20260805-stage5-dc0-ceph-remediation.md: session changelog (separate
same-day file from the skill-DOCFIX changelog, disclosed in header); F2/F3/F4 owed,
unnumbered.
repo-lint 0 fail / 1 legacy warn; ledger-scan DOCFIX next-free still 210 (no token leaked);
all touched files ASCII-clean. Status lives in docs/CURRENT-STATE.md ONLY.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728