|
sweep F2 ROOT-CAUSED: one bad archive.ubuntu.com backend; "not transient" WITHDRAWN
Operator ruling, exact utterance: "Root-cause the curl/debmirror anomaly first" -- taken over accepting the red gate, over recording error shape/count, and over --ignore-small-errors. ROOT CAUSE IS UPSTREAM AND BACKEND-SPECIFIC, not a defect in this deployment. archive.ubuntu.com is round-robin across NINE A records and ONE, 91.189.92.23, HANGS on dists/jammy-backports/main/dep11/Components-amd64.yml.gz while serving its directory siblings normally. The control is what makes it a finding rather than "a backend is down": that same backend returns the .xz sibling 200 / 5776 B / 0.543s, and the other eight return the .gz in 0.14-0.51s. The resolver ROTATES -- six consecutive lookups, six different orderings -- so every fetch re-rolls the backend. MEASURED RATE: 12 sequential fetches -> 11 x 200, 1 timeout (~8%), consistent with exactly one bad record of nine. THEREFORE THE PRIOR COMMIT'S "NOT TRANSIENT" IS WITHDRAWN (GA-R1 C2 -- measurement wins over a document, including one written an hour earlier). The failure is probabilistic; two consecutive debmirror failures were not the proof of persistence they appeared to be. A hypothesis was tested and REFUTED, and is kept because it is the one a reader forms first: debmirror sets keep_alive => 1 and curl does not reuse connections, so a stale persistent connection looked like the answer. The test inverted it -- keep_alive=1 (debmirror's own setting) returned 200 in 1s while keep_alive=0 timed out, same library, seconds apart. ALSO CORRECTED: the "500" was never a server 500. LWP synthesises 500 for CLIENT-side failures, so it is LWP reporting its own 300s timeout (debmirror:629). I had read it as an archive-side error, which points the investigation the wrong way. Two structural facts read from the vendor script, both bearing on options that were on the table: there is NO dep11 exclusion option (dep11_from_release :1201 and get_dep11_files :1384 are unconditional), so the mirror's scope cannot be narrowed without patching debmirror; and --ignore-small-errors is NOT a bounded tolerance -- :2891 DISCARDS the error count, so it would read clean on a wholly failed mirror. It stays refuted. RESIDUAL, declared not explained: debmirror failed 2 of 2, which a naive 1-in-9 model predicts at ~1.2%. The per-IP sweep tested each backend ONCE, so it establishes that 91.189.92.23 was bad, NOT that it is the only bad one. SECOND, INDEPENDENT FINDING: the local Release expects 6349 B while every backend now serves 6361 B, and the object was republished DURING the investigation. A successful download would now fail HASH verification against the stale local Release -- a different failure mode from the timeout, not fixed by fixing the timeout. scripts/dc-mirror.sh UNTOUCHED. repo-lint 0 fail; ledger-scan unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf |
|---|
|
|
| docs/CURRENT-STATE.md |
|---|
| docs/audit/mirror-500-timeout-rootcause-20260802.txt 0 → 100644 |
|---|
| docs/changelog-20260802-queued-items.md |
|---|