# Changelog 2026-08-05 (Stage-5 dc0 ceph apt-wedge remediation) -- the "allocating" pair fixed

Session changelog. Branch `dc-dc-stage5-preconditions`, HEAD `33f9158` at the start of
this work. `ledger-scan` at open: 4 open decisions, 29 open SEC, next-free D-143 /
DOCFIX-210 / BUNDLEFIX-053. Status claims live in `docs/CURRENT-STATE.md` ONLY; nothing
here is a status claim.

DISCLOSURE (GA-R2/D1 one-changelog-per-session): this is a SECOND changelog written the
same day and, strictly, the same post-/clear session as
`docs/changelog-20260805-skill-close-convention-docfix.md` (which is committed + pushed).
It is kept SEPARATE rather than appended because it is a distinct work-stream (a live-cloud
deploy triage vs a doc fix) and retitling a delivered, pushed changelog to cover unrelated
live-cloud mutations would rewrite a delivered record. Precedent for a same-day second
changelog: `changelog-20260802-{deploy-input,queued-items}.md`. Flagged for the close
bookend so it cites BOTH bodies.

Scope: the item the 08-05 vault-init close named NEXT -- triage the ceph-mon/2 +
ceph-radosgw/0 units stuck `allocating` -- worked to completion (operator-gated live
mutations), plus the durable capture the operator directed ("a then b").

---

## Item 1 -- ceph-mon/2 + ceph-radosgw/0 "allocating" REMEDIATED; dc0 storage layer converged

**WHAT WAS WRONG (measured, not the carried label).** The 08-05 close called this an
"apt-cacher-ng hang class" blocker. Read-only triage this session REFUTED the "proxy is
down" reading: `dc-cache-proxy check dc0` and `dc-snap-proxy check dc0` both PASS, and the
proxy serves the upstream the hang stalled on (`security.ubuntu.com` InRelease ->
200/0.67s). Actual root cause, measured on the two containers: cloud-init `modules:final`
hung in `apt-get update` for ~12.5h (etimes 45241/45297s), the apt `http` method parked in
**CLOSE-WAIT** to the proxy `10.12.8.4:3142`. apt has no client read timeout, so a
connection the proxy half-closed during the 08-04 redeploy's concurrent load was never
recovered -> cloud-init never finished -> no jujud -> unit `allocating`. Everything
downstream (ceph-osd, cinder-ceph/backup, glance, nova-compute) was a cascade of
no-mon-quorum. Full evidence (before-state is UNRECOVERABLE, verbatim from transcript):
`docs/audit/stage5-dc0-ceph-remediation-20260805.txt`.

**THE MUTATIONS (operator-gated, run from vr1-dc0-rack per D-138).** Two approval
exchanges: "Yes, try it" (ceph-mon), "Approved" (ceph-radosgw). Per app:
`remove-unit <u> --no-prompt` -> `remove-machine <c> --force --no-prompt` ->
`add-unit <app> -n 1 --to <placement>`. Placement from `bundle.yaml` (ceph-mon
`to:[lxd:0,lxd:1,lxd:2]` -> `lxd:2`; ceph-radosgw `to:[lxd:0]`), never inferred.
- **`--force` was REQUIRED, not optional:** `remove-unit` did NOT cascade to the
  never-provisioned container (`life` stayed alive); a graceful `remove-machine` marked it
  dying but the reaper blocked waiting to stop an agent that never came up.
- `remove-unit`/`remove-machine` need `--no-prompt` in a non-tty context (the first
  `remove-unit` aborted on the interactive "Continue [y/N]?").

**RESULT.** Both fresh containers' cloud-init FINISHED (~211s, no apt hang); jujud active;
mons bootstrapped quorum 3/3; 4 OSDs active; the storage cascade cleared (cinder-ceph,
cinder-backup, glance, nova-compute -> active). A SECOND, distinct apt-cacher-ng mode
surfaced on the radosgw rebuild and SELF-HEALED: `ceph-radosgw-hacluster/0` install hook
failed 4x on apt exit 100 (404 on `linux-libc-dev_5.15.0-187.197_amd64.deb` via the proxy
= stale cached index vs a rotated point-release), then juju's auto-retry succeeded once the
index refreshed. Measured after-state (17:34:08Z): **census 62 active** (settle progression
measured 59 -> 61 -> 62; no pre-remediation total was captured, so the close-block "40/47"
figures are NOT a citable transition -- C2). Remaining non-active are all deferred-by-design
(octavia D-021/phase-05, designate Stage-7 DNS, ceph-rbd-mirror dc1-peer/Stage-6, gss
one-shot) except `barbican-vault/0` (settling; verify owed -- item (b)).

**Revert.** n/a -- a live-cloud fix of a broken state, not a git change. To "undo" would be
to remove the rebuilt units, which re-breaks the storage layer. Do not.

## Item 2 -- durable capture of the live mutations + the findings (this write-up)

Per operator direction ("a then b"): (i) NEW
`docs/audit/stage5-dc0-ceph-remediation-20260805.txt` -- the transcribed capture, plainly
labelled NOT a script(1) capture, carrying the unrecoverable before-state verbatim; (ii)
`docs/CURRENT-STATE.md` section 1 -- a dated status block recording the convergence, the
measured root cause, the `--force` fact, and the named-gate defect below (GA-R1: status
lives only there; the block cites the capture and quotes only measured counts); (iii)
appendix-A entry drafted (Item 3). No numbered identifiers were written above the
high-water mark (`ledger-scan` next-free DOCFIX still 210 after this write).

**Revert.** `git revert` this commit -> drops the capture + CURRENT-STATE block + appendix-A
entry. Not recommended: the before-state evidence exists nowhere else and would be lost.

## Item 3 -- appendix-A entry drafted for the apt-cacher-ng provisioning-hang symptom pair

**WHAT WAS MISSING.** No appendix-A entry existed for this class, despite 3 recurrences
(08-03/08-04 mysql, 08-05 ceph x2) -- so each was re-diagnosed from scratch (CLAUDE.md hard
rule 4: a gap is a finding). NEW entry drafted covering BOTH measured modes ((a) CLOSE-WAIT
hang -> `allocating`; (b) stale-index 404 -> hook exit 100, self-heals) with the
verify-proxy-first -> remove-unit -> remove-machine --force -> add-unit remediation. Drafted
for operator review before commit.

**Revert.** `git revert` (or drop the entry). Not recommended -- it removes the only codified
guidance for a thrice-recurring provisioning hazard.

## Findings OWED (logged, not executed this session -- hard rule 1)

- **F2 -- named-gate defect:** `phase-03-core-verify.md` Step 3.1 asserts expected
  non-active/idle = 1 (octavia); the VR1 roster measured 4 deferred + gss. DOCFIX owed
  (unnumbered until implemented). Same runbook also owes `-m openstack` -> `-m vr1-dc0`
  and run-from-dc0-rack (D-138). Recorded in CURRENT-STATE as the durable finding.
- **F3 -- checker blind spot:** `dc-cache-proxy.sh check` does not test
  `security.ubuntu.com`. DOCFIX owed (unnumbered).
- **F4 -- design/hardening:** the apt CLOSE-WAIT / stale-index class has recurred 3x;
  remove/re-add is a workaround. Preventive `Acquire::http::Timeout`/`Retries` injected at
  provision is a D-NNN candidate for an operator ruling.