diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 4afd600..faa1399 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -3012,6 +3012,33 @@ ACTUALLY assigned is 207 (this session's preflight plan-count fix). Caught both times by re-running `ledger-scan` instead of trusting the edit -- the rule is unenforceable by reading, because the defect looks exactly like correct bookkeeping. + **>>> SWEEP F2 FOLLOW-UP: THE MIRROR FAILURE IS PERSISTENT, AND IT DOES NOT AFFECT THE + DEPLOY. <<<** **OPERATOR RULING 2026-08-02, operational (GA-R3: no architectural + consequence, so no D-number).** Question put: what should `dc-mirror.sh`'s gate ATTEST, + given debmirror's exit code cannot distinguish "nothing mirrored" from "mirrored minus N + transient files" from "debmirror died"? **Exact utterance: "Re-trigger the sync first, + decide after".** Run as a MEASUREMENT, not a remedy; **the attestation question remains + OPEN.** **RESULT: NOT TRANSIENT.** The re-run (10:26:05->10:31:20Z) failed identically -- + `FAIL ubuntu=1 uca=0`, `ExecMainStatus=1` -- same file, same error, same 300s timeout, two + hours on: `dists/jammy-backports/main/dep11/Components-amd64.yml.gz`. **This CORRECTS the + earlier "flaky, not stuck" reading AS APPLIED TO THIS FAILURE** -- the unit has a history + of transient failures AND now one persistent one; the two were being conflated. Waiting + will not clear it. **THE GAP IS NOW EXACT, replacing "at least one file missing":** + debmirror's own `Files to download: 6349 B` equals that single file's size, so the entire + outstanding delta of the 952 GiB tree IS this one file. The `.xz` twin is PRESENT and + size-correct (5748 B, matching `Release`); only the `.gz` is absent. **AND IT DOES NOT + BREAK apt -- MEASURED with an ISOLATED apt state dir on the rack** (nothing system-wide + touched): `apt-get update` against `http://10.12.8.4/ubuntu jammy-backports main` returns + **rc=0**, fetching `Components [5748 B]` -- **the .xz, i.e. apt asks for the format that IS + present and never asks for the missing .gz** -- and `apt-cache policy` resolves the suite + normally. **So the mirror is FUNCTIONALLY COMPLETE for the deploy and the gate is RED over + a file no consumer requests.** ONE ANOMALY LEFT UNEXPLAINED rather than hand-waved: curl + fetches that same `.gz` from the same host in **0.44s, HTTP 200, 6349 B** with a real body + read, while debmirror times out at 300s on it twice. A proxy is ruled OUT (unit env is + `HOME=/root` only; no proxy in `/etc/environment`, `apt.conf.d`, `profile.d`, or root's + login shell). Not chased -- it does not gate the deploy, and guessing at an LWP-vs-curl + difference would be the reasoning-instead-of-measuring failure this project keeps logging. + `scripts/dc-mirror.sh` remains UNTOUCHED; no tolerance relaxed, no assertion weakened. - Project: Omega Cloud, VR1 DC-DC rehearsal -- a two-DC + Office1-headend virtual rehearsal on KVM (vcloud host), rehearsing the future bare-metal Roosevelt deployment (D-100, `docs/design-decisions.md:1946`). diff --git a/docs/changelog-20260802-queued-items.md b/docs/changelog-20260802-queued-items.md index 2f75dcd..4e7edb0 100644 --- a/docs/changelog-20260802-queued-items.md +++ b/docs/changelog-20260802-queued-items.md @@ -323,3 +323,71 @@ - Changed: `docs/audit/mirror-exitcode-diagnosis-dc0-20260802.txt` (new capture). `scripts/dc-mirror.sh` deliberately UNTOUCHED. - **Revert:** n/a -- diagnosis only. + +## Item 8 -- sweep F2, the re-trigger: the failure is PERSISTENT, and it does not matter + +**OPERATOR RULING (operational, not a D-number per GA-R3 -- no architectural consequence).** +Question put: what should `dc-mirror.sh`'s gate attest, given debmirror's exit code cannot +distinguish "nothing mirrored" from "mirrored minus N transient files" from "debmirror +died"? Options offered: record shape+count without relaxing / leave as is / re-trigger +first and decide after / add a bounded tolerance (listed as NOT recommended). **Operator +answer, exact utterance: "Re-trigger the sync first, decide after".** The re-trigger was +therefore run as a MEASUREMENT, not as a remedy, and the attestation question stays OPEN. + +**RESULT: THE FAILURE IS NOT TRANSIENT.** The re-run (10:26:05 -> 10:31:20Z) failed +identically -- `FAIL ... ubuntu=1 uca=0`, `Result=exit-code`, `ExecMainStatus=1` -- on the +SAME file with the SAME error and the SAME 300s timeout, two hours after the previous +attempt: + +``` +08:27:56 Download of dists/jammy-backports/main/dep11/Components-amd64.yml.gz failed: 500 read timeout (300s) +10:31:16 Download of dists/jammy-backports/main/dep11/Components-amd64.yml.gz failed: 500 read timeout +``` + +So the earlier "flaky, not stuck" reading -- which the fail-then-succeed pairs supported -- +does NOT extend to THIS failure. Two distinct things were being conflated: the unit has a +history of transient failures, AND it now has one persistent one. The ruling's "decide +after" is answered: waiting will not clear it. + +**THE SCOPE OF THE GAP IS NOW EXACT, replacing the agent's "at least one file missing".** +debmirror's own `Files to download: 6349 B` equals the size of that single file, so the +entire outstanding delta of the 952 GiB tree IS this one file. On disk: + +``` +present: Components-amd64.yml.xz 5748 B (Release lists 5748 -- size-correct) +ABSENT: Components-amd64.yml.gz 6349 B (Release lists 6349) +``` + +**AND IT DOES NOT BREAK apt -- MEASURED, not reasoned.** The prior agent explicitly +declared this unmeasured and declined to reason it away, which was correct. Tested from the +rack with a fully ISOLATED apt state dir (scratch `Dir::State::Lists`, `Dir::Cache`, +`Dir::Etc::SourceList`; nothing system-wide touched, no package installed): + +``` +deb http://10.12.8.4/ubuntu jammy-backports main + Get:1 InRelease [127 kB] + Get:2 main amd64 Packages [70.2 kB] + Get:3 main amd64 Components [5748 B] <-- the .xz, the format that IS present + Get:4 main amd64 c-n-f Metadata [412 B] + APT_UPDATE_RC=0 + apt-cache policy -> release v=22.04,o=Ubuntu,a=jammy-backports,c=main,b=amd64 +``` + +**apt asks for the `.xz` and never asks for the `.gz`.** So the mirror is FUNCTIONALLY +complete for the deploy and the gate is red over a file no consumer requests. jammy-backports +was chosen as the test suite deliberately -- it is the one carrying the missing file. + +**ONE ANOMALY, UNEXPLAINED AND LEFT THAT WAY rather than hand-waved.** The file fetches +FINE by curl from the same host, right now: `http=200 size=6349 time=0.436s`, with a real +body read (HEAD would prove nothing). Its `.xz` sibling and a jammy control also return 200. +So curl succeeds in 0.44s where debmirror times out after 300s, twice. Ruled OUT: any proxy +(the unit's environment is `HOME=/root` only; no `http_proxy` in `/etc/environment`, +`apt.conf.d`, `profile.d`, or root's login shell). NOT chased further -- it does not gate +the deploy, and guessing at an LWP-vs-curl difference would be exactly the reasoning-instead- +of-measuring this project keeps logging. + +- Changed: nothing in the repo. `scripts/dc-mirror.sh` still UNTOUCHED; no tolerance was + relaxed and no assertion weakened. +- State changed on the rack: one sync run, which wrote `last-sync.status` and fetched + ~1.3 MB of indices. The apt test wrote only to a scratch dir, removed on exit. +- **Revert:** n/a.