diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 1f941e1..2a19df1 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2277,6 +2277,47 @@ is scan-visible (`ledger-scan`: 4 PROPOSED/OPEN decisions, next-free D-143). voffice1 synced to HEAD. SESSION CLOSE 2026-08-05 (GA-R4 bookend) -- vault init DONE + ovn-central RESOLVED; D-142 saved. + **>>> 2026-08-05 (LATER SESSION) -- THE ceph "allocating" PAIR IS REMEDIATED; dc0 STORAGE + LAYER CONVERGED. Stage 5 remains OPEN. <<<** Full capture (transcribed from the session + transcript, NOT a script(1) capture -- the before-state is unrecoverable and lives only there): + `docs/audit/stage5-dc0-ceph-remediation-20260805.txt`. **The 08-05 close's "apt-wedge" label was + an OBSERVATION carried as a CONCLUSION and is REFUTED by measurement:** both dc0 proxies read + PASS (`dc-cache-proxy`/`dc-snap-proxy check dc0`), and the proxy serves the exact upstream the + hang stalled on (`security.ubuntu.com` InRelease -> 200/0.67s). **Measured root cause:** + `ceph-mon/2` (`2/lxd/0`) and `ceph-radosgw/0` (`0/lxd/2`) were stuck `allocating` because each + container's cloud-init `modules:final` hung in `apt-get update` for ~12.5h (etimes 45241/45297s), + its apt `http` method parked in **CLOSE-WAIT** to the proxy `10.12.8.4:3142` -- apt has no + client read timeout, so a connection the proxy half-closed during the 08-04 redeploy's + concurrent load was never recovered. Everything else (ceph-osd, cinder-ceph, cinder-backup, + glance, nova-compute) was a DOWNSTREAM cascade of no-mon-quorum; `ceph-rbd-mirror/0` blocked is + DEFERRED-by-design (dc1 peer, Stage 6). **Remediation (operator-gated, 2 exchanges -- "Yes, try + it" for ceph-mon, "Approved" for radosgw), run from the dc0 rack (D-138):** per app, + `remove-unit --no-prompt` -> `remove-machine --force --no-prompt` -> `add-unit -n 1 + --to ` (ceph-mon `lxd:2`, ceph-radosgw `lxd:0`; placement taken from + `bundle.yaml`, not inferred). **LOAD-BEARING FACT for the next session: `remove-unit` did NOT + cascade to the never-provisioned container (`life` stayed alive; a graceful `remove-machine` + marked it dying but the reaper BLOCKED waiting to stop an agent that never came up), so the + `--force` on the dead-agent container was REQUIRED, not optional.** Both fresh containers' + cloud-init FINISHED (~211s, no apt hang); mons bootstrapped quorum 3/3, 4 OSDs active, storage + cascade cleared. **A SECOND, DISTINCT apt-cacher-ng failure mode surfaced on the radosgw rebuild + and SELF-HEALED:** `ceph-radosgw-hacluster/0` install hook failed 4x on apt exit 100 -- a 404 on + `linux-libc-dev_5.15.0-187.197_amd64.deb` via the proxy = a STALE cached index pointing at a + point-release upstream had rotated out; juju's auto-retry succeeded once the index refreshed. + **MEASURED AFTER-STATE (2026-08-05 17:34:08Z, `juju status -m vr1-dc0`): census 62 active** (the + settle progression actually measured was 59 -> 61 -> 62; NO pre-remediation total was captured, + so the earlier close-block "40/47" figures are NOT a citable transition -- C2, quote captured + output only). Remaining non-active are all accounted for: `octavia/0` (D-021, phase-05), + `designate/0` (DNS activation, Stage 7), `ceph-rbd-mirror/0` (dc1 peer, Stage 6) and + `glance-simplestreams-sync/0` (gss one-shot) are DEFERRED-by-design; `barbican-vault/0` + ('secrets-storage' incomplete) is settling and OWED a verify (next: item (b)). + **NAMED-GATE DEFECT found by measurement -- `phase-03-core-verify.md` Step 3.1 asserts expected + non-active/idle = 1 (octavia only); the VR1 roster yields 4 deferred-by-design + gss.** That gate + is STALE for VR1 and a DOCFIX is owed (also owed on that runbook: `-m openstack` -> `-m vr1-dc0` + and run-from-dc0-rack per D-138). Two more owed DOCFIX/design items: `dc-cache-proxy.sh check` + does not test `security.ubuntu.com`; and the apt CLOSE-WAIT/stale-index class has now recurred 3x + (08-03/08-04 mysql, 08-05 ceph) -- a preventive apt `Timeout`/`Retries` inject at provision is a + D-NNN candidate for operator ruling. appendix-A entry drafted this session for the symptom pair. + Body: `docs/changelog-20260805-stage5-dc0-ceph-remediation.md`. **>>> SESSION CLOSE 2026-08-03 (GA-R4 bookend). Stage 5 remains OPEN; this is a session bookend, NOT a stage close. <<<** DEPLOY IS MID-STAGE-5: bundle deployed + mostly converged (9 machines started, mysql ONLINE, vault init+unseal+root-CA, ~25 units active, 0 error), diff --git a/docs/audit/stage5-dc0-ceph-remediation-20260805.txt b/docs/audit/stage5-dc0-ceph-remediation-20260805.txt new file mode 100644 index 0000000..2687aa2 --- /dev/null +++ b/docs/audit/stage5-dc0-ceph-remediation-20260805.txt @@ -0,0 +1,158 @@ +STAGE 5 dc0 -- ceph apt-wedge remediation (ceph-mon/2 + ceph-radosgw/0) +======================================================================== +Date: 2026-08-05. Model: vr1-dc0. Operator-gated live-cloud mutation session. + +PROVENANCE / HOW TO READ THIS FILE +---------------------------------- +This is TRANSCRIBED FROM THE CLAUDE-CODE SESSION TRANSCRIPT, not a script(1) +`-aqe` capture -- no as-executed wrapper ran (the harness classifier refuses the +wrapped form; see memory as-executed-log-vs-classifier). Every command below was +run from the vcloud jumphost via `ssh vr1-dc0-rack ''` (D-138: the juju +client lives on the dc0 rack). The command lines and their output blocks are +VERBATIM as observed this session. GA-R1 rule 2 basis: quoted command + its output. + +>>> THE BEFORE-STATE IS UNRECOVERABLE. <<< The CLOSE-WAIT sockets, the ~12.5h +apt etimes, and the two hung containers (2/lxd/0, 0/lxd/2) were DESTROYED by the +remediation. The only evidence they ever existed is this transcript. That is why +this section is captured verbatim: without it the root cause would be re-derived +from scratch (this is the 3rd recurrence of the apt-cacher-ng class). + + +== BEFORE (measured this session, pre-remediation) ============================== + +(1) juju status -- ceph layer BLOCKED, two units allocating on PENDING containers: + + ceph-mon blocked 2/3 Insufficient peer units to bootstrap cluster (require 3) + ceph-osd waiting 4 Incomplete relation: monitor + ceph-radosgw waiting 0/1 waiting for machine + ceph-rbd-mirror blocked 1 'ceph-local' incomplete, 'ceph-remote' missing [deferred: dc1 peer] + cinder-backup waiting 1 Incomplete relations: ceph + glance waiting 1 Incomplete relations: storage-backend + nova-compute waiting 2 Incomplete relations: storage-backend + + ceph-mon/2 waiting allocating 2/lxd/0 waiting for machine + ceph-radosgw/0 waiting allocating 0/lxd/2 waiting for machine + Machine 2/lxd/0 pending juju-d4f45a-2-lxd-0 Container started + Machine 0/lxd/2 pending juju-d4f45a-0-lxd-2 Container started + +(2) The PROXY IS HEALTHY -- refutes the carried "apt-cacher-ng is down" conclusion: + dc-cache-proxy check (dc0): PASS (apt-cacher-ng active, :3142, archive+UCA 200) + dc-snap-proxy check (dc0): PASS (squid :3129, store API 200) + +(3) Container level -- both RUNNING with addresses, but cloud-init NEVER finished, + no jujud agent installed: + juju-d4f45a-0-lxd-2 RUNNING 10.12.9.4/10.12.4.147/10.12.32.103/10.12.12.118 + juju-d4f45a-2-lxd-0 RUNNING 10.12.8.193/10.12.36.1/10.12.32.1 + cloud-init status --long -> status: running (both) + systemctl is-active "jujud-machine-*" -> (empty; agent never installed) + +(4) THE SMOKING GUN -- cloud-init modules:final hung in apt-get update for ~12.5h: + (ps -eo pid,etimes,stat,args ; etimes = elapsed SECONDS) + + 0/lxd/2: 1201 45241 S /usr/bin/apt-get ... --quiet update + 1212 45241 S /usr/lib/apt/methods/http + 1213 45241 S /usr/lib/apt/methods/http + 2/lxd/0: 1329 45297 S /usr/bin/apt-get ... --quiet update + 1340 45297 S /usr/lib/apt/methods/http + cloud-init-output.log tail: a run of "Get:.. archive.ubuntu.com .." lines then + STOPS mid-fetch. Started Tue 04 Aug 2026 20:24-20:25 = the 08-04 redeploy window. + +(5) ROOT CAUSE -- apt http methods parked in CLOSE-WAIT to the proxy, no timeout: + (ss -tnp) + 0/lxd/2: CLOSE-WAIT 10.12.9.4:40210 10.12.8.4:3142 users:(("http",pid=1212)) + CLOSE-WAIT 10.12.9.4:40242 10.12.8.4:3142 users:(("http",pid=1213)) + 2/lxd/0: CLOSE-WAIT 10.12.8.193:40236 10.12.8.4:3142 users:(("http",pid=1340)) + apt proxy config (correct): Acquire::http::Proxy "http://10.12.8.4:3142"; + Proxy serves the upstream the hang stalled on: + curl -x http://10.12.8.4:3142 http://security.ubuntu.com/.../jammy-security/InRelease + -> HTTP 200 (0.668s) + CONCLUSION: proxy closed its side ~12.5h ago (concurrent-load blip during the + 08-04 redeploy); apt never closed its side and has NO read timeout -> parked + forever. NOT a live proxy fault, config fault, or routing fault. Stale half-closed + connections apt never recovered from. Same class remediated 08-03/08-04 (mysql). + + +== REMEDIATION (operator-gated mutations, run from vr1-dc0-rack) ================= + +Bundle placement (source of truth, from bundle.yaml -- NO inferred values): + ceph-mon to: [lxd:0, lxd:1, lxd:2] + ceph-radosgw to: [lxd:0] + +--- ceph-mon/2 (operator-approved "Yes, try it") --- + juju remove-unit ceph-mon/2 -m vr1-dc0 + -> ERROR unit removal: aborted (interactive "Continue [y/N]?" with no tty) + juju remove-unit ceph-mon/2 -m vr1-dc0 --no-prompt -> will remove unit ceph-mon/2 (rc=0) + + >>> KEY OPERATIONAL FACT: remove-unit did NOT cascade to the never-provisioned + container. `life` stayed alive; 2/lxd/0 lingered RUNNING/pending. A graceful + `remove-machine 2/lxd/0` marked it dying but the reaper BLOCKED waiting to + stop an agent that never came up. Cleanup required --force. <<< + juju remove-machine 2/lxd/0 -m vr1-dc0 --force --no-prompt (operator-approved) + -> 2/lxd/0 GONE from juju + CONFIRMED-GONE-ON-HOST + juju add-unit ceph-mon -m vr1-dc0 -n 1 --to lxd:2 -> ceph-mon/3 on fresh 2/lxd/5 + + RESULT: fresh container cloud-init FINISHED in ~211s (no apt hang; jujud active). + ceph-mon 3/3 -> "Monitor bootstrapped" -> all 3 mons active "Unit is ready and + clustered"; 4 ceph-osd -> active "Unit is ready (1 OSD)". Storage cascade cleared: + cinder-ceph, cinder-backup, glance, nova-compute -> active. + +--- ceph-radosgw/0 (operator-approved "Approved") --- + juju remove-unit ceph-radosgw/0 -m vr1-dc0 --no-prompt + juju remove-machine 0/lxd/2 -m vr1-dc0 --force --no-prompt -> GONE (juju + host) + juju add-unit ceph-radosgw -m vr1-dc0 -n 1 --to lxd:0 -> ceph-radosgw/1 on 0/lxd/20 + + RESULT: container provisioned cleanly. SECOND apt-cacher-ng failure MODE observed + here (distinct from CLOSE-WAIT): ceph-radosgw-hacluster/0 install hook failed 4x, + apt exit status 100: + E: Failed to fetch http://archive.ubuntu.com/.../linux-libc-dev_5.15.0-187.197_amd64.deb + 404 Not Found [IP: 10.12.8.4 3142] + = STALE apt-cacher-ng cached index pointing at a point-release .deb upstream had + rotated out of the pool. SELF-HEALED: juju auto-retried; once the cached index + refreshed, `apt-get update` in-container returned rc=0 (all Hits) and the retry + installed pacemaker/corosync. hacluster -> active "Unit is ready and clustered"; + rgw + apache2 services active (running); ceph-radosgw/1 -> active (stale + "Services not running" status cleared on the next update-status). + + +== AFTER (freshly measured 2026-08-05 17:34:08Z) =============================== + + CENSUS: {active: 62, waiting: 1, blocked: 3, unknown: 1} + Non-active, all accounted for: + barbican-vault/0 waiting 'secrets-storage' incomplete [settling -- verify] + ceph-rbd-mirror/0 blocked 'ceph-remote' missing [DEFERRED: dc1 peer, Stage 6 / dc-dc-phase5] + designate/0 blocked nameservers must be set [DEFERRED: DNS activation, Stage 7 / dc-dc-phase6] + glance-simplestreams-sync/0 unknown [EXPECTED: gss one-shot resting state] + octavia/0 blocked Awaiting configure-resources [DEFERRED: D-021, Stage 5 / phase-05] + + Census progression actually measured this session: 59 (mid-settle, both re-adds + done) -> 61 -> 62. NOTE: no pre-remediation TOTAL was measured; do not cite a + "40->" or "47->" transition (those are prose figures from the 08-05 close block, + not a census). The measured transition is qualitative (ceph layer blocked -> the + storage layer fully converged) plus 59->62 on the settle. + + +== FINDINGS (owed; logged here + in the session changelog, NOT executed) ======== + +F1. appendix-A GAP -> entry drafted this session. TWO apt-cacher-ng failure modes: + (a) apt http method CLOSE-WAIT to :3142, no client timeout -> cloud-init hangs + forever -> unit stuck `allocating`, container `pending`/"Container started". + (b) stale cached index -> 404 on a rotated point-release .deb -> charm apt-get + install fails (exit 100); self-heals on index refresh + juju hook retry. + Remediation both: confirm proxy healthy -> remove-unit --no-prompt -> + remove-machine --force --no-prompt (dead-agent container does NOT cascade) + -> add-unit --to . + +F2. phase-03-core-verify.md Step 3.1 gate is STALE for VR1. It asserts expected + non-active/idle = 1 (octavia). MEASURED here: 4 deferred-by-design (octavia, + designate, ceph-rbd-mirror) + gss unknown. A DOCFIX is owed (unnumbered until + implemented). Also owed: -m openstack -> -m vr1-dc0, and run-from-dc0-rack + (D-138), throughout that runbook. + +F3. dc-cache-proxy.sh `check` does not test security.ubuntu.com (the exact upstream + the CLOSE-WAIT hang stalled on; it serves fine, 200/0.67s, but the checker is + blind to it). A DOCFIX is owed (unnumbered). + +F4. RECURRENCE (3rd: 08-03 mysql, 08-04 mysql, 08-05 ceph x2). remove/re-add is a + WORKAROUND. Preventive fix is DESIGN material for an operator ruling: inject + Acquire::http::Timeout / Acquire::Retries into node apt config at provision so + apt cannot hang (mode a) or hard-fail without retry (mode b). D-NNN candidate. diff --git a/docs/changelog-20260805-stage5-dc0-ceph-remediation.md b/docs/changelog-20260805-stage5-dc0-ceph-remediation.md new file mode 100644 index 0000000..a787fe0 --- /dev/null +++ b/docs/changelog-20260805-stage5-dc0-ceph-remediation.md @@ -0,0 +1,100 @@ +# Changelog 2026-08-05 (Stage-5 dc0 ceph apt-wedge remediation) -- the "allocating" pair fixed + +Session changelog. Branch `dc-dc-stage5-preconditions`, HEAD `33f9158` at the start of +this work. `ledger-scan` at open: 4 open decisions, 29 open SEC, next-free D-143 / +DOCFIX-210 / BUNDLEFIX-053. Status claims live in `docs/CURRENT-STATE.md` ONLY; nothing +here is a status claim. + +DISCLOSURE (GA-R2/D1 one-changelog-per-session): this is a SECOND changelog written the +same day and, strictly, the same post-/clear session as +`docs/changelog-20260805-skill-close-convention-docfix.md` (which is committed + pushed). +It is kept SEPARATE rather than appended because it is a distinct work-stream (a live-cloud +deploy triage vs a doc fix) and retitling a delivered, pushed changelog to cover unrelated +live-cloud mutations would rewrite a delivered record. Precedent for a same-day second +changelog: `changelog-20260802-{deploy-input,queued-items}.md`. Flagged for the close +bookend so it cites BOTH bodies. + +Scope: the item the 08-05 vault-init close named NEXT -- triage the ceph-mon/2 + +ceph-radosgw/0 units stuck `allocating` -- worked to completion (operator-gated live +mutations), plus the durable capture the operator directed ("a then b"). + +--- + +## Item 1 -- ceph-mon/2 + ceph-radosgw/0 "allocating" REMEDIATED; dc0 storage layer converged + +**WHAT WAS WRONG (measured, not the carried label).** The 08-05 close called this an +"apt-cacher-ng hang class" blocker. Read-only triage this session REFUTED the "proxy is +down" reading: `dc-cache-proxy check dc0` and `dc-snap-proxy check dc0` both PASS, and the +proxy serves the upstream the hang stalled on (`security.ubuntu.com` InRelease -> +200/0.67s). Actual root cause, measured on the two containers: cloud-init `modules:final` +hung in `apt-get update` for ~12.5h (etimes 45241/45297s), the apt `http` method parked in +**CLOSE-WAIT** to the proxy `10.12.8.4:3142`. apt has no client read timeout, so a +connection the proxy half-closed during the 08-04 redeploy's concurrent load was never +recovered -> cloud-init never finished -> no jujud -> unit `allocating`. Everything +downstream (ceph-osd, cinder-ceph/backup, glance, nova-compute) was a cascade of +no-mon-quorum. Full evidence (before-state is UNRECOVERABLE, verbatim from transcript): +`docs/audit/stage5-dc0-ceph-remediation-20260805.txt`. + +**THE MUTATIONS (operator-gated, run from vr1-dc0-rack per D-138).** Two approval +exchanges: "Yes, try it" (ceph-mon), "Approved" (ceph-radosgw). Per app: +`remove-unit --no-prompt` -> `remove-machine --force --no-prompt` -> +`add-unit -n 1 --to `. Placement from `bundle.yaml` (ceph-mon +`to:[lxd:0,lxd:1,lxd:2]` -> `lxd:2`; ceph-radosgw `to:[lxd:0]`), never inferred. +- **`--force` was REQUIRED, not optional:** `remove-unit` did NOT cascade to the + never-provisioned container (`life` stayed alive); a graceful `remove-machine` marked it + dying but the reaper blocked waiting to stop an agent that never came up. +- `remove-unit`/`remove-machine` need `--no-prompt` in a non-tty context (the first + `remove-unit` aborted on the interactive "Continue [y/N]?"). + +**RESULT.** Both fresh containers' cloud-init FINISHED (~211s, no apt hang); jujud active; +mons bootstrapped quorum 3/3; 4 OSDs active; the storage cascade cleared (cinder-ceph, +cinder-backup, glance, nova-compute -> active). A SECOND, distinct apt-cacher-ng mode +surfaced on the radosgw rebuild and SELF-HEALED: `ceph-radosgw-hacluster/0` install hook +failed 4x on apt exit 100 (404 on `linux-libc-dev_5.15.0-187.197_amd64.deb` via the proxy += stale cached index vs a rotated point-release), then juju's auto-retry succeeded once the +index refreshed. Measured after-state (17:34:08Z): **census 62 active** (settle progression +measured 59 -> 61 -> 62; no pre-remediation total was captured, so the close-block "40/47" +figures are NOT a citable transition -- C2). Remaining non-active are all deferred-by-design +(octavia D-021/phase-05, designate Stage-7 DNS, ceph-rbd-mirror dc1-peer/Stage-6, gss +one-shot) except `barbican-vault/0` (settling; verify owed -- item (b)). + +**Revert.** n/a -- a live-cloud fix of a broken state, not a git change. To "undo" would be +to remove the rebuilt units, which re-breaks the storage layer. Do not. + +## Item 2 -- durable capture of the live mutations + the findings (this write-up) + +Per operator direction ("a then b"): (i) NEW +`docs/audit/stage5-dc0-ceph-remediation-20260805.txt` -- the transcribed capture, plainly +labelled NOT a script(1) capture, carrying the unrecoverable before-state verbatim; (ii) +`docs/CURRENT-STATE.md` section 1 -- a dated status block recording the convergence, the +measured root cause, the `--force` fact, and the named-gate defect below (GA-R1: status +lives only there; the block cites the capture and quotes only measured counts); (iii) +appendix-A entry drafted (Item 3). No numbered identifiers were written above the +high-water mark (`ledger-scan` next-free DOCFIX still 210 after this write). + +**Revert.** `git revert` this commit -> drops the capture + CURRENT-STATE block + appendix-A +entry. Not recommended: the before-state evidence exists nowhere else and would be lost. + +## Item 3 -- appendix-A entry drafted for the apt-cacher-ng provisioning-hang symptom pair + +**WHAT WAS MISSING.** No appendix-A entry existed for this class, despite 3 recurrences +(08-03/08-04 mysql, 08-05 ceph x2) -- so each was re-diagnosed from scratch (CLAUDE.md hard +rule 4: a gap is a finding). NEW entry drafted covering BOTH measured modes ((a) CLOSE-WAIT +hang -> `allocating`; (b) stale-index 404 -> hook exit 100, self-heals) with the +verify-proxy-first -> remove-unit -> remove-machine --force -> add-unit remediation. Drafted +for operator review before commit. + +**Revert.** `git revert` (or drop the entry). Not recommended -- it removes the only codified +guidance for a thrice-recurring provisioning hazard. + +## Findings OWED (logged, not executed this session -- hard rule 1) + +- **F2 -- named-gate defect:** `phase-03-core-verify.md` Step 3.1 asserts expected + non-active/idle = 1 (octavia); the VR1 roster measured 4 deferred + gss. DOCFIX owed + (unnumbered until implemented). Same runbook also owes `-m openstack` -> `-m vr1-dc0` + and run-from-dc0-rack (D-138). Recorded in CURRENT-STATE as the durable finding. +- **F3 -- checker blind spot:** `dc-cache-proxy.sh check` does not test + `security.ubuntu.com`. DOCFIX owed (unnumbered). +- **F4 -- design/hardening:** the apt CLOSE-WAIT / stale-index class has recurred 3x; + remove/re-add is a workaround. Preventive `Acquire::http::Timeout`/`Retries` injected at + provision is a D-NNN candidate for an operator ruling. diff --git a/runbooks/appendix-A-troubleshooting.md b/runbooks/appendix-A-troubleshooting.md index c18e800..70066bc 100644 --- a/runbooks/appendix-A-troubleshooting.md +++ b/runbooks/appendix-A-troubleshooting.md @@ -857,3 +857,71 @@ The NAT read-back is REPORTED, not gated: at an edge's very first addressing there may legitimately be no gateway yet and therefore no NAT to generate. The hard gate remains address-on-the-kernel. + +## Symptom: a unit sits `allocating` forever on a `pending` LXD container; cloud-init never finishes (per-DC apt caching proxy) [2026-08-05] + +-------------------------------------------------------------------------------- +SYMPTOM: One or more juju units stay `allocating` ("waiting for machine") for + hours. `juju machines` shows the container `pending` with message + "Container started"; on the LXD host the container is RUNNING with + addresses, but `cloud-init status` reads `running` (never `done`) and + no `jujud-machine-*` unit exists. Sibling containers on the SAME host + came up fine. (Seen 3x: 08-03/08-04 mysql-innodb-cluster, 08-05 + ceph-mon + ceph-radosgw.) A related, SEPARATE mode: a charm's own + `install` hook fails with apt `exit status 100` and a `404 Not Found` + on a point-release .deb "[IP: 3142]". +-------------------------------------------------------------------------------- +CAUSE: The DC apt CACHING PROXY (apt-cacher-ng, D-135; dc0/dc1 both on the + proxy after the D-135 amendment (b)). TWO distinct failure modes, both + with the proxy otherwise HEALTHY: + (a) HANG: the container's cloud-init `apt-get update` connection is + half-closed by the proxy under concurrent load (many nodes hit it + at once during a deploy/redeploy). apt's `http` method has NO + client read timeout, so the socket parks in CLOSE-WAIT and apt + waits forever -> cloud-init `modules:final` never returns -> jujud + never installs -> unit `allocating`. The hung apt shows a huge + `etimes` (e.g. ~45000s) and the socket state is the tell. + (b) STALE INDEX: apt-cacher-ng serves a cached Packages/Release index + that points at a point-release .deb upstream has already rotated + out of the pool -> `404` on that .deb -> `apt-get install` exit + 100. This one SELF-HEALS: juju retries the hook, and once the + cached index refreshes, the install succeeds. proxy health checks + stay green throughout. +-------------------------------------------------------------------------------- +CHECK: The proxy being UP does NOT rule this out -- check the CONTAINER, not + the proxy. From the LXD host (juju ssh ): + + sudo lxc exec -- cloud-init status # running, not done + sudo lxc exec -- ps -eo pid,etimes,args | grep 'apt-get.*update' + sudo lxc exec -- ss -tnp | grep :3142 # CLOSE-WAIT = mode (a) + + For mode (b), read the failing hook: + juju debug-log --include --replay | grep -iE '404|E:|exit' + + Confirm the proxy is actually healthy (so a rebuild will succeed) and + DO test security.ubuntu.com, which the stock checker does not: + ssh -rack "curl -sS -x http://:3142 -o /dev/null -w '%{http_code}\n' \ + http://security.ubuntu.com/ubuntu/dists/jammy-security/InRelease" # want 200 +-------------------------------------------------------------------------------- +FIX: Mode (b) usually needs NO action -- let juju's hook retry self-heal + once the index refreshes; verify with a fresh `apt-get update` in the + container (rc=0, all Hits). + + Mode (a) needs the unit rebuilt (the hung apt will never recover). Run + from the DC rack (D-138), one app at a time, GATED: + + juju remove-unit / -m --no-prompt + # remove-unit does NOT cascade to a never-provisioned container -- + # its `life` stays alive and a graceful remove-machine BLOCKS on the + # dead agent. Force-remove the single dead container (host untouched): + juju remove-machine /lxd/ -m --force --no-prompt + juju add-unit -m -n 1 --to lxd: # placement from bundle.yaml + + A fresh container re-runs cloud-init against the now-uncontended proxy + and completes (~200s). Never batch the destructive steps; take the + bundle `to:` placement from `bundle.yaml`, never inferred. + +NOTE: Recurring (3x). Preventive fix is unbuilt/unruled as of 2026-08-05: + inject `Acquire::http::Timeout` / `Acquire::Retries` into node apt + config at provision so mode (a) cannot hang and mode (b) retries + cleanly. D-NNN candidate (see CURRENT-STATE 2026-08-05 ceph block).