Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260805-stage5-dc0-ceph-remediation.md
@JANeumatrix JANeumatrix 4 hours ago 6 KB Stage 5 dc0: remediate ceph-mon/2 + ceph-radosgw/0 "allocating" (apt CLOSE-WAIT); durable capture
Raw Blame History

Changelog 2026-08-05 (Stage-5 dc0 ceph apt-wedge remediation) -- the "allocating" pair fixed

Session changelog. Branch dc-dc-stage5-preconditions, HEAD 33f9158 at the start of this work. ledger-scan at open: 4 open decisions, 29 open SEC, next-free D-143 / DOCFIX-210 / BUNDLEFIX-053. Status claims live in docs/CURRENT-STATE.md ONLY; nothing here is a status claim.

DISCLOSURE (GA-R2/D1 one-changelog-per-session): this is a SECOND changelog written the same day and, strictly, the same post-/clear session as docs/changelog-20260805-skill-close-convention-docfix.md (which is committed + pushed). It is kept SEPARATE rather than appended because it is a distinct work-stream (a live-cloud deploy triage vs a doc fix) and retitling a delivered, pushed changelog to cover unrelated live-cloud mutations would rewrite a delivered record. Precedent for a same-day second changelog: changelog-20260802-{deploy-input,queued-items}.md. Flagged for the close bookend so it cites BOTH bodies.

Scope: the item the 08-05 vault-init close named NEXT -- triage the ceph-mon/2 + ceph-radosgw/0 units stuck allocating -- worked to completion (operator-gated live mutations), plus the durable capture the operator directed ("a then b").

Item 1 -- ceph-mon/2 + ceph-radosgw/0 "allocating" REMEDIATED; dc0 storage layer converged

WHAT WAS WRONG (measured, not the carried label). The 08-05 close called this an "apt-cacher-ng hang class" blocker. Read-only triage this session REFUTED the "proxy is down" reading: dc-cache-proxy check dc0 and dc-snap-proxy check dc0 both PASS, and the proxy serves the upstream the hang stalled on (security.ubuntu.com InRelease -> 200/0.67s). Actual root cause, measured on the two containers: cloud-init modules:final hung in apt-get update for ~12.5h (etimes 45241/45297s), the apt http method parked in CLOSE-WAIT to the proxy 10.12.8.4:3142. apt has no client read timeout, so a connection the proxy half-closed during the 08-04 redeploy's concurrent load was never recovered -> cloud-init never finished -> no jujud -> unit allocating. Everything downstream (ceph-osd, cinder-ceph/backup, glance, nova-compute) was a cascade of no-mon-quorum. Full evidence (before-state is UNRECOVERABLE, verbatim from transcript): docs/audit/stage5-dc0-ceph-remediation-20260805.txt.

THE MUTATIONS (operator-gated, run from vr1-dc0-rack per D-138). Two approval exchanges: "Yes, try it" (ceph-mon), "Approved" (ceph-radosgw). Per app: remove-unit <u> --no-prompt -> remove-machine <c> --force --no-prompt -> add-unit <app> -n 1 --to <placement>. Placement from bundle.yaml (ceph-mon to:[lxd:0,lxd:1,lxd:2] -> lxd:2; ceph-radosgw to:[lxd:0]), never inferred.

  • --force was REQUIRED, not optional: remove-unit did NOT cascade to the never-provisioned container (life stayed alive); a graceful remove-machine marked it dying but the reaper blocked waiting to stop an agent that never came up.
  • remove-unit/remove-machine need --no-prompt in a non-tty context (the first remove-unit aborted on the interactive "Continue [y/N]?").

RESULT. Both fresh containers' cloud-init FINISHED (~211s, no apt hang); jujud active; mons bootstrapped quorum 3/3; 4 OSDs active; the storage cascade cleared (cinder-ceph, cinder-backup, glance, nova-compute -> active). A SECOND, distinct apt-cacher-ng mode surfaced on the radosgw rebuild and SELF-HEALED: ceph-radosgw-hacluster/0 install hook failed 4x on apt exit 100 (404 on linux-libc-dev_5.15.0-187.197_amd64.deb via the proxy = stale cached index vs a rotated point-release), then juju's auto-retry succeeded once the index refreshed. Measured after-state (17:34:08Z): census 62 active (settle progression measured 59 -> 61 -> 62; no pre-remediation total was captured, so the close-block "40/47" figures are NOT a citable transition -- C2). Remaining non-active are all deferred-by-design (octavia D-021/phase-05, designate Stage-7 DNS, ceph-rbd-mirror dc1-peer/Stage-6, gss one-shot) except barbican-vault/0 (settling; verify owed -- item (b)).

Revert. n/a -- a live-cloud fix of a broken state, not a git change. To "undo" would be to remove the rebuilt units, which re-breaks the storage layer. Do not.

Item 2 -- durable capture of the live mutations + the findings (this write-up)

Per operator direction ("a then b"): (i) NEW docs/audit/stage5-dc0-ceph-remediation-20260805.txt -- the transcribed capture, plainly labelled NOT a script(1) capture, carrying the unrecoverable before-state verbatim; (ii) docs/CURRENT-STATE.md section 1 -- a dated status block recording the convergence, the measured root cause, the --force fact, and the named-gate defect below (GA-R1: status lives only there; the block cites the capture and quotes only measured counts); (iii) appendix-A entry drafted (Item 3). No numbered identifiers were written above the high-water mark (ledger-scan next-free DOCFIX still 210 after this write).

Revert. git revert this commit -> drops the capture + CURRENT-STATE block + appendix-A entry. Not recommended: the before-state evidence exists nowhere else and would be lost.

Item 3 -- appendix-A entry drafted for the apt-cacher-ng provisioning-hang symptom pair

WHAT WAS MISSING. No appendix-A entry existed for this class, despite 3 recurrences (08-03/08-04 mysql, 08-05 ceph x2) -- so each was re-diagnosed from scratch (CLAUDE.md hard rule 4: a gap is a finding). NEW entry drafted covering BOTH measured modes ((a) CLOSE-WAIT hang -> allocating; (b) stale-index 404 -> hook exit 100, self-heals) with the verify-proxy-first -> remove-unit -> remove-machine --force -> add-unit remediation. Drafted for operator review before commit.

Revert. git revert (or drop the entry). Not recommended -- it removes the only codified guidance for a thrice-recurring provisioning hazard.

Findings OWED (logged, not executed this session -- hard rule 1)

  • F2 -- named-gate defect: phase-03-core-verify.md Step 3.1 asserts expected non-active/idle = 1 (octavia); the VR1 roster measured 4 deferred + gss. DOCFIX owed (unnumbered until implemented). Same runbook also owes -m openstack -> -m vr1-dc0 and run-from-dc0-rack (D-138). Recorded in CURRENT-STATE as the durable finding.
  • F3 -- checker blind spot: dc-cache-proxy.sh check does not test security.ubuntu.com. DOCFIX owed (unnumbered).
  • F4 -- design/hardening: the apt CLOSE-WAIT / stale-index class has recurred 3x; remove/re-add is a workaround. Preventive Acquire::http::Timeout/Retries injected at provision is a D-NNN candidate for an operator ruling.