STAGE 5 dc0 -- ceph apt-wedge remediation (ceph-mon/2 + ceph-radosgw/0)
========================================================================
Date: 2026-08-05. Model: vr1-dc0. Operator-gated live-cloud mutation session.
PROVENANCE / HOW TO READ THIS FILE
----------------------------------
This is TRANSCRIBED FROM THE CLAUDE-CODE SESSION TRANSCRIPT, not a script(1)
`-aqe` capture -- no as-executed wrapper ran (the harness classifier refuses the
wrapped form; see memory as-executed-log-vs-classifier). Every command below was
run from the vcloud jumphost via `ssh vr1-dc0-rack '<juju ...>'` (D-138: the juju
client lives on the dc0 rack). The command lines and their output blocks are
VERBATIM as observed this session. GA-R1 rule 2 basis: quoted command + its output.
>>> THE BEFORE-STATE IS UNRECOVERABLE. <<< The CLOSE-WAIT sockets, the ~12.5h
apt etimes, and the two hung containers (2/lxd/0, 0/lxd/2) were DESTROYED by the
remediation. The only evidence they ever existed is this transcript. That is why
this section is captured verbatim: without it the root cause would be re-derived
from scratch (this is the 3rd recurrence of the apt-cacher-ng class).
== BEFORE (measured this session, pre-remediation) ==============================
(1) juju status -- ceph layer BLOCKED, two units allocating on PENDING containers:
ceph-mon blocked 2/3 Insufficient peer units to bootstrap cluster (require 3)
ceph-osd waiting 4 Incomplete relation: monitor
ceph-radosgw waiting 0/1 waiting for machine
ceph-rbd-mirror blocked 1 'ceph-local' incomplete, 'ceph-remote' missing [deferred: dc1 peer]
cinder-backup waiting 1 Incomplete relations: ceph
glance waiting 1 Incomplete relations: storage-backend
nova-compute waiting 2 Incomplete relations: storage-backend
ceph-mon/2 waiting allocating 2/lxd/0 waiting for machine
ceph-radosgw/0 waiting allocating 0/lxd/2 waiting for machine
Machine 2/lxd/0 pending juju-d4f45a-2-lxd-0 Container started
Machine 0/lxd/2 pending juju-d4f45a-0-lxd-2 Container started
(2) The PROXY IS HEALTHY -- refutes the carried "apt-cacher-ng is down" conclusion:
dc-cache-proxy check (dc0): PASS (apt-cacher-ng active, :3142, archive+UCA 200)
dc-snap-proxy check (dc0): PASS (squid :3129, store API 200)
(3) Container level -- both RUNNING with addresses, but cloud-init NEVER finished,
no jujud agent installed:
juju-d4f45a-0-lxd-2 RUNNING 10.12.9.4/10.12.4.147/10.12.32.103/10.12.12.118
juju-d4f45a-2-lxd-0 RUNNING 10.12.8.193/10.12.36.1/10.12.32.1
cloud-init status --long -> status: running (both)
systemctl is-active "jujud-machine-*" -> (empty; agent never installed)
(4) THE SMOKING GUN -- cloud-init modules:final hung in apt-get update for ~12.5h:
(ps -eo pid,etimes,stat,args ; etimes = elapsed SECONDS)
0/lxd/2: 1201 45241 S /usr/bin/apt-get ... --quiet update
1212 45241 S /usr/lib/apt/methods/http
1213 45241 S /usr/lib/apt/methods/http
2/lxd/0: 1329 45297 S /usr/bin/apt-get ... --quiet update
1340 45297 S /usr/lib/apt/methods/http
cloud-init-output.log tail: a run of "Get:.. archive.ubuntu.com .." lines then
STOPS mid-fetch. Started Tue 04 Aug 2026 20:24-20:25 = the 08-04 redeploy window.
(5) ROOT CAUSE -- apt http methods parked in CLOSE-WAIT to the proxy, no timeout:
(ss -tnp)
0/lxd/2: CLOSE-WAIT 10.12.9.4:40210 10.12.8.4:3142 users:(("http",pid=1212))
CLOSE-WAIT 10.12.9.4:40242 10.12.8.4:3142 users:(("http",pid=1213))
2/lxd/0: CLOSE-WAIT 10.12.8.193:40236 10.12.8.4:3142 users:(("http",pid=1340))
apt proxy config (correct): Acquire::http::Proxy "http://10.12.8.4:3142";
Proxy serves the upstream the hang stalled on:
curl -x http://10.12.8.4:3142 http://security.ubuntu.com/.../jammy-security/InRelease
-> HTTP 200 (0.668s)
CONCLUSION: proxy closed its side ~12.5h ago (concurrent-load blip during the
08-04 redeploy); apt never closed its side and has NO read timeout -> parked
forever. NOT a live proxy fault, config fault, or routing fault. Stale half-closed
connections apt never recovered from. Same class remediated 08-03/08-04 (mysql).
== REMEDIATION (operator-gated mutations, run from vr1-dc0-rack) =================
Bundle placement (source of truth, from bundle.yaml -- NO inferred values):
ceph-mon to: [lxd:0, lxd:1, lxd:2]
ceph-radosgw to: [lxd:0]
--- ceph-mon/2 (operator-approved "Yes, try it") ---
juju remove-unit ceph-mon/2 -m vr1-dc0
-> ERROR unit removal: aborted (interactive "Continue [y/N]?" with no tty)
juju remove-unit ceph-mon/2 -m vr1-dc0 --no-prompt -> will remove unit ceph-mon/2 (rc=0)
>>> KEY OPERATIONAL FACT: remove-unit did NOT cascade to the never-provisioned
container. `life` stayed alive; 2/lxd/0 lingered RUNNING/pending. A graceful
`remove-machine 2/lxd/0` marked it dying but the reaper BLOCKED waiting to
stop an agent that never came up. Cleanup required --force. <<<
juju remove-machine 2/lxd/0 -m vr1-dc0 --force --no-prompt (operator-approved)
-> 2/lxd/0 GONE from juju + CONFIRMED-GONE-ON-HOST
juju add-unit ceph-mon -m vr1-dc0 -n 1 --to lxd:2 -> ceph-mon/3 on fresh 2/lxd/5
RESULT: fresh container cloud-init FINISHED in ~211s (no apt hang; jujud active).
ceph-mon 3/3 -> "Monitor bootstrapped" -> all 3 mons active "Unit is ready and
clustered"; 4 ceph-osd -> active "Unit is ready (1 OSD)". Storage cascade cleared:
cinder-ceph, cinder-backup, glance, nova-compute -> active.
--- ceph-radosgw/0 (operator-approved "Approved") ---
juju remove-unit ceph-radosgw/0 -m vr1-dc0 --no-prompt
juju remove-machine 0/lxd/2 -m vr1-dc0 --force --no-prompt -> GONE (juju + host)
juju add-unit ceph-radosgw -m vr1-dc0 -n 1 --to lxd:0 -> ceph-radosgw/1 on 0/lxd/20
RESULT: container provisioned cleanly. SECOND apt-cacher-ng failure MODE observed
here (distinct from CLOSE-WAIT): ceph-radosgw-hacluster/0 install hook failed 4x,
apt exit status 100:
E: Failed to fetch http://archive.ubuntu.com/.../linux-libc-dev_5.15.0-187.197_amd64.deb
404 Not Found [IP: 10.12.8.4 3142]
= STALE apt-cacher-ng cached index pointing at a point-release .deb upstream had
rotated out of the pool. SELF-HEALED: juju auto-retried; once the cached index
refreshed, `apt-get update` in-container returned rc=0 (all Hits) and the retry
installed pacemaker/corosync. hacluster -> active "Unit is ready and clustered";
rgw + apache2 services active (running); ceph-radosgw/1 -> active (stale
"Services not running" status cleared on the next update-status).
== AFTER (freshly measured 2026-08-05 17:34:08Z) ===============================
CENSUS: {active: 62, waiting: 1, blocked: 3, unknown: 1}
Non-active, all accounted for:
barbican-vault/0 waiting 'secrets-storage' incomplete [settling -- verify]
ceph-rbd-mirror/0 blocked 'ceph-remote' missing [DEFERRED: dc1 peer, Stage 6 / dc-dc-phase5]
designate/0 blocked nameservers must be set [DEFERRED: DNS activation, Stage 7 / dc-dc-phase6]
glance-simplestreams-sync/0 unknown [EXPECTED: gss one-shot resting state]
octavia/0 blocked Awaiting configure-resources [DEFERRED: D-021, Stage 5 / phase-05]
Census progression actually measured this session: 59 (mid-settle, both re-adds
done) -> 61 -> 62. NOTE: no pre-remediation TOTAL was measured; do not cite a
"40->" or "47->" transition (those are prose figures from the 08-05 close block,
not a census). The measured transition is qualitative (ceph layer blocked -> the
storage layer fully converged) plus 59->62 on the settle.
== FINDINGS (owed; logged here + in the session changelog, NOT executed) ========
F1. appendix-A GAP -> entry drafted this session. TWO apt-cacher-ng failure modes:
(a) apt http method CLOSE-WAIT to :3142, no client timeout -> cloud-init hangs
forever -> unit stuck `allocating`, container `pending`/"Container started".
(b) stale cached index -> 404 on a rotated point-release .deb -> charm apt-get
install fails (exit 100); self-heals on index refresh + juju hook retry.
Remediation both: confirm proxy healthy -> remove-unit --no-prompt ->
remove-machine <c> --force --no-prompt (dead-agent container does NOT cascade)
-> add-unit <app> --to <bundle placement>.
F2. phase-03-core-verify.md Step 3.1 gate is STALE for VR1. It asserts expected
non-active/idle = 1 (octavia). MEASURED here: 4 deferred-by-design (octavia,
designate, ceph-rbd-mirror) + gss unknown. A DOCFIX is owed (unnumbered until
implemented). Also owed: -m openstack -> -m vr1-dc0, and run-from-dc0-rack
(D-138), throughout that runbook.
F3. dc-cache-proxy.sh `check` does not test security.ubuntu.com (the exact upstream
the CLOSE-WAIT hang stalled on; it serves fine, 200/0.67s, but the checker is
blind to it). A DOCFIX is owed (unnumbered).
F4. RECURRENCE (3rd: 08-03 mysql, 08-04 mysql, 08-05 ceph x2). remove/re-add is a
WORKAROUND. Preventive fix is DESIGN material for an operator ruling: inject
Acquire::http::Timeout / Acquire::Retries into node apt config at provision so
apt cannot hang (mode a) or hard-fail without retry (mode b). D-NNN candidate.
