Stage 5 dc0: remediate ceph-mon/2 + ceph-radosgw/0 "allocating" (apt CLOSE-WAIT); durable capture
Live-cloud, operator-gated (2 exchanges). Two units stuck `allocating` were root-caused
NOT to a down proxy (both dc0 proxies PASS; security.ubuntu.com 200/0.67s) but to
cloud-init `apt-get update` parked in CLOSE-WAIT to 10.12.8.4:3142 for ~12.5h (apt has no
client read timeout) since the 08-04 redeploy. Rebuilt each from the rack (D-138):
remove-unit --no-prompt -> remove-machine --force --no-prompt (REQUIRED: remove-unit does
NOT cascade to a never-provisioned dead-agent container) -> add-unit --to <bundle
placement>. Both fresh containers' cloud-init finished (~211s); mons bootstrapped 3/3, 4
OSDs active, storage cascade cleared. A 2nd apt-cacher-ng mode (radosgw-hacluster 404 on a
rotated point-release -> exit 100) SELF-HEALED via juju hook retry. Measured after-state
17:34:08Z: census 62 active; remaining non-active all deferred-by-design + gss +
barbican-vault settling.

Durable capture (operator-directed "a then b"):
- docs/audit/stage5-dc0-ceph-remediation-20260805.txt: transcribed capture (NOT script(1));
  the before-state (CLOSE-WAIT sockets, ~45000s etimes, both containers) is unrecoverable
  and lives only here.
- docs/CURRENT-STATE.md: dated status block; measured 62 active (NOT the unmeasured 40/47,
  C2); records the --force fact and the phase-03 Step-3.1 gate defect (expects 1, VR1 has 4
  deferred + gss) as durable finding.
- runbooks/appendix-A-troubleshooting.md: NEW entry for the symptom pair (both modes +
  gated remove/--force/re-add remediation). Drafted; operator-reviewed.
- docs/changelog-20260805-stage5-dc0-ceph-remediation.md: session changelog (separate
  same-day file from the skill-DOCFIX changelog, disclosed in header); F2/F3/F4 owed,
  unnumbered.

repo-lint 0 fail / 1 legacy warn; ledger-scan DOCFIX next-free still 210 (no token leaked);
all touched files ASCII-clean. Status lives in docs/CURRENT-STATE.md ONLY.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728
1 parent 33f9158 commit 71c5b9778dc6b6b4be4b82b749a4366740c16005
@JANeumatrix JANeumatrix authored 2 hours ago
Showing 4 changed files
View
docs/CURRENT-STATE.md
View
docs/audit/stage5-dc0-ceph-remediation-20260805.txt 0 → 100644
View
docs/changelog-20260805-stage5-dc0-ceph-remediation.md 0 → 100644
View
runbooks/appendix-A-troubleshooting.md