mirror-exitcode-diagnosis-dc0-20260802.txt ========================================== Diagnosis of SWEEP FINDING F2 (docs/audit/queued-findings-20260802-stage5-edge-fold.txt lines 37-55): "THE MIRROR SYNC EXITS 1 WHILE ITS OWN LOG SAYS THE SYNC SUCCEEDED". NOTE ON SCOPE: this is the SWEEP F-register (queued-findings-20260802-stage5-edge-fold.txt). It is NOT docs/runbook-fold-register.md, which carries its own separate F-numbers. Nothing in the runbook-fold register was read or touched for this work. Read-only diagnostic session, 2026-08-02. No state was changed on the rack or the jumphost OTHER THAN nginx access-log entries generated by the repo check's own HTTP probe (section 2.5) and by the upstream HEAD in 2.8: no sync re-trigger, no `systemctl reset-failed`, no write to the status file, no repo edit, no git operation. Every command run is reproduced verbatim below. ====================================================================================== VERDICT ====================================================================================== R1 IS TRUE. R2 IS REFUTED. debmirror itself has a non-fatal path that prints "All done." and then exits 1. Nothing in the sync helper runs after debmirror's final "All done." except the second debmirror invocation and the status-file write; the helper's exit status is a faithful propagation of debmirror's own exit code, not an artifact of the helper. The status line `FAIL ... ubuntu=1 uca=0` is literally debmirror-#1's exit code captured in `RC1=$?` on the line immediately following the invocation. There is no intervening command that could have overwritten it. Cause of the 1: ONE upstream download failed -- dists/jammy-backports/main/dep11/Components-amd64.yml.gz -- 500 read timeout (300s) from archive.ubuntu.com. debmirror counted it in $num_errors, completed the mirror move and cleanup anyway, printed "All done.", THEN printed the error block and exited 1. ====================================================================================== 1. REPO-SIDE READING (done first, per method) -- already discriminates R1 from R2 ====================================================================================== Source: /home/jessea123/openstack-caracal-dc-dc/scripts/dc-mirror.sh Values taken FROM THE SCRIPT (not guessed): MIRROR_ROOT /var/lib/dc-mirror (line 110) SYNC_HELPER /usr/local/sbin/dc0-mirror-sync (line 114, ${SITE}=dc0) SYNC_UNIT /etc/systemd/system/dc0-mirror-sync.service (line 115) SYNC_TIMER /etc/systemd/system/dc0-mirror-sync.timer (line 116) STATUS FILE /var/lib/dc-mirror/last-sync.status (helper heredoc, line 180) Ubuntu tree /var/lib/dc-mirror/ubuntu UCA tree /var/lib/dc-mirror/cloud-archive The sync helper is generated by gen_sync_helper() (dc-mirror.sh lines 168-201). Its executable body, stripped of comments, is exactly: set -uo pipefail STATUS="/var/lib/dc-mirror/last-sync.status" echo "RUNNING $(date -u +%FT%TZ)" > "$STATUS" mkdir -p /var/lib/dc-mirror/ubuntu /var/lib/dc-mirror/cloud-archive debmirror /var/lib/dc-mirror/ubuntu \ --host=archive.ubuntu.com --root=ubuntu --method=http \ --dist=jammy,jammy-updates,jammy-security,jammy-backports \ --section=main,restricted,universe,multiverse \ --arch=amd64 --nosource --diff=none \ --keyring=/usr/share/keyrings/ubuntu-archive-keyring.gpg --progress RC1=$? debmirror /var/lib/dc-mirror/cloud-archive \ --host=ubuntu-cloud.archive.canonical.com --root=ubuntu --method=http \ --dist=jammy-updates/caracal --section=main \ --arch=amd64 --nosource --diff=none \ --keyring=/usr/share/keyrings/ubuntu-cloud-keyring.gpg --progress RC2=$? if [ "$RC1" -eq 0 ] && [ "$RC2" -eq 0 ]; then echo "OK $(date -u +%FT%TZ) ubuntu=0 uca=0" > "$STATUS"; exit 0 fi echo "FAIL $(date -u +%FT%TZ) ubuntu=$RC1 uca=$RC2" > "$STATUS"; exit 1 WHAT THIS ESTABLISHES BEFORE ANY LIVE MEASUREMENT: a) There are TWO debmirror invocations (ubuntu, then cloud-archive) and their statuses ARE combined -- by the `if [ "$RC1" -eq 0 ] && [ "$RC2" -eq 0 ]` test. A non-zero from EITHER produces the same helper exit 1. b) `RC1=$?` is the very next command after the first debmirror (the backslashes are line continuations of one command, so $? is unambiguously debmirror's). Same for RC2. NOTHING runs between a debmirror and the capture of its status. c) The ONLY things that run after the second debmirror are: the `if` test, one `echo` to the status file, and `exit`. None of them can fail in a way that manufactures `ubuntu=1`. c2) `set -uo pipefail` is set WITHOUT `-e`, and that absence is LOAD-BEARING for the diagnosis. With `-e`, debmirror's non-zero would have terminated the helper immediately -- before the `echo "FAIL ..."` line ever ran -- and the status file would still read `RUNNING`, not `FAIL`. The mere EXISTENCE of a terminal `FAIL ... ubuntu=1 uca=0` line therefore proves the helper ran all the way to its last statement and propagated a status deliberately. This kills R2 on the status file alone, independently of the journal: a helper that had itself failed somewhere after "All done." could not have written that line. d) Therefore the value `1` in `ubuntu=1` can only have come from debmirror #1's own exit code. If debmirror #1 also printed "All done.", R1 is proven and R2 is structurally impossible. The remaining question the repo alone could not answer: WHICH debmirror printed the "All done." quoted in the sweep, and did debmirror #1 print one too. That is what the live journal settles -- CONFIRMING the repo reading rather than discovering it. ====================================================================================== 2. LIVE MEASUREMENT ====================================================================================== Host reached as `vr1-dc0-rack` (ProxyJump voffice1 from ~/.ssh/config.d/vr1-sites, as configured; no jump was hand-constructed). Rack hostname reports as vvr1-dc0. CONVENTION FOR THIS SECTION: commands are reproduced as issued, except that the trailing `2>&1` used to capture stderr into the transcript is omitted, and long one-line commands are wrapped. Outputs are byte-verbatim; the ONLY edits to an output block are elisions, and every elision is marked inline as `... [ ... ] ...`. -------------------------------------------------------------------------------------- 2.1 The INSTALLED sync helper vs the repo's generated content -- IDENTICAL -------------------------------------------------------------------------------------- COMMAND: ssh vr1-dc0-rack 'hostname; echo ---; sudo cat /usr/local/sbin/dc0-mirror-sync' OUTPUT (verbatim): vvr1-dc0 --- #!/usr/bin/env bash # dc0-mirror-sync -- generated by scripts/dc-mirror.sh (D-135 item 1); # do not hand-edit. Jammy amd64 binary mirror + UCA caracal, GPG-verified. # jammy-backports ADDED by the D-135 AMENDMENT 2026-07-31 (operator: "Sync the # backports into the mirror"). It is NOT optional: the MAAS jammy image ships # jammy-backports in its sources.list, and juju's apt-mirror model-config rewrites # EVERY suite to this mirror -- so a mirror without it 404s and every charm's # install hook fails on 'does not have a Release file'. That cost 22 units on the # 2026-07-31 dc0 deploy. The mirror's suite list is a CONTRACT with the node image. set -uo pipefail STATUS="/var/lib/dc-mirror/last-sync.status" echo "RUNNING $(date -u +%FT%TZ)" > "$STATUS" mkdir -p /var/lib/dc-mirror/ubuntu /var/lib/dc-mirror/cloud-archive debmirror /var/lib/dc-mirror/ubuntu \ --host=archive.ubuntu.com --root=ubuntu --method=http \ --dist=jammy,jammy-updates,jammy-security,jammy-backports \ --section=main,restricted,universe,multiverse \ --arch=amd64 --nosource --diff=none \ --keyring=/usr/share/keyrings/ubuntu-archive-keyring.gpg --progress RC1=$? debmirror /var/lib/dc-mirror/cloud-archive \ --host=ubuntu-cloud.archive.canonical.com --root=ubuntu --method=http \ --dist=jammy-updates/caracal --section=main \ --arch=amd64 --nosource --diff=none \ --keyring=/usr/share/keyrings/ubuntu-cloud-keyring.gpg --progress RC2=$? if [ "$RC1" -eq 0 ] && [ "$RC2" -eq 0 ]; then echo "OK $(date -u +%FT%TZ) ubuntu=0 uca=0" > "$STATUS"; exit 0 fi echo "FAIL $(date -u +%FT%TZ) ubuntu=$RC1 uca=$RC2" > "$STATUS"; exit 1 FINDING: the installed helper matches the repo's gen_sync_helper() output exactly. This is CORROBORATED independently by the repo's own check (section 2.5), whose `check_file` does a real `diff` of the installed file against the generated content and printed "OK /usr/local/sbin/dc0-mirror-sync matches". THERE IS NO DRIFT. -------------------------------------------------------------------------------------- 2.2 The unit, the status file, and the recorded exit status -------------------------------------------------------------------------------------- COMMAND: ssh vr1-dc0-rack 'echo "=== unit ==="; sudo systemctl cat dc0-mirror-sync.service; echo "=== status file ==="; sudo cat /var/lib/dc-mirror/last-sync.status; echo "=== systemctl show ==="; sudo systemctl show dc0-mirror-sync.service -p Result -p ExecMainStatus -p ExecMainExitTimestamp -p UnitFileState -p ActiveState' OUTPUT (verbatim): === unit === # /etc/systemd/system/dc0-mirror-sync.service [Unit] Description=dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA) After=dc0-mirror-net.service network-online.target Requires=dc0-mirror-net.service [Service] Type=oneshot # debmirror dies with "HOME not defined in environment!" under systemd's # empty environment (it needs a GPG home) -- measured on the first dc0 run. Environment=HOME=/root ExecStart=/usr/local/sbin/dc0-mirror-sync Nice=10 IOSchedulingClass=idle === status file === FAIL 2026-08-02T08:28:03Z ubuntu=1 uca=0 === systemctl show === Result=exit-code ExecMainExitTimestamp=Sun 2026-08-02 08:28:03 UTC ExecMainStatus=1 UnitFileState=static ActiveState=failed NOTE: `systemctl cat` was used deliberately (not `systemctl show`) as the existence check, per SWEEP FINDING F3's durable rule in the same sweep file. The unit has ONE ExecStart and no ExecStartPost / ExecStopPost. So systemd's status=1/FAILURE is the helper's own `exit 1` and nothing else. Confirms 1(c). -------------------------------------------------------------------------------------- 2.3 THE DECISIVE EVIDENCE -- full journal of the 2026-08-02 08:22 run -------------------------------------------------------------------------------------- COMMAND: ssh vr1-dc0-rack 'sudo journalctl -u dc0-mirror-sync.service --since "2026-08-02 08:00" --until "2026-08-02 09:00" --no-pager -o short-precise' The run spans two PIDs -- 276574 (debmirror #1, ubuntu) and 277623 (debmirror #2, UCA). The decisive segment, verbatim and in order: Aug 02 08:22:47.841316 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)... Aug 02 08:22:47.959441 vvr1-dc0 dc0-mirror-sync[276574]: Mirroring to /var/lib/dc-mirror/ubuntu from http://archive.ubuntu.com/ubuntu/ Aug 02 08:22:47.959441 vvr1-dc0 dc0-mirror-sync[276574]: Dists: jammy,jammy-updates,jammy-security,jammy-backports ... [meta-file fetches, all "ok"] ... Aug 02 08:22:56.221218 vvr1-dc0 dc0-mirror-sync[276574]: Get DEP-11 metadata files ... Aug 02 08:27:56.506798 vvr1-dc0 dc0-mirror-sync[276574]: [ 44%] Getting: dists/jammy-backports/main/dep11/Components-amd64.yml.gz... #** GET http://archive.ubuntu.com/ubuntu/dists/jammy-backports/main/dep11/Components-amd64.yml.gz ==> 500 read timeout (300s) Aug 02 08:27:56.507531 vvr1-dc0 dc0-mirror-sync[276574]: Download of dists/jammy-backports/main/dep11/Components-amd64.yml.gz failed: 500 read timeout ... [remaining fetches, all "ok"] ... Aug 02 08:27:59.992486 vvr1-dc0 dc0-mirror-sync[276574]: Get command-not-found metadata files ... Aug 02 08:27:59.992486 vvr1-dc0 dc0-mirror-sync[276574]: Files to download: 6349 B Aug 02 08:28:00.109956 vvr1-dc0 dc0-mirror-sync[276574]: Downloaded 2867 kiB in 313s at 9.16 kiB/s. Aug 02 08:28:00.109956 vvr1-dc0 dc0-mirror-sync[276574]: Everything OK. Moving meta files ... Aug 02 08:28:00.127979 vvr1-dc0 dc0-mirror-sync[276574]: Cleanup mirror. Aug 02 08:28:00.830168 vvr1-dc0 dc0-mirror-sync[276574]: All done. Aug 02 08:28:00.840132 vvr1-dc0 dc0-mirror-sync[276574]: Errors: Aug 02 08:28:00.840132 vvr1-dc0 dc0-mirror-sync[276574]: Download of dists/jammy-backports/main/dep11/Components-amd64.yml.gz failed: 500 read timeout Aug 02 08:28:00.840132 vvr1-dc0 dc0-mirror-sync[276574]: Failed to download files (1 errors)! Aug 02 08:28:00.963508 vvr1-dc0 dc0-mirror-sync[277623]: Mirroring to /var/lib/dc-mirror/cloud-archive from http://ubuntu-cloud.archive.canonical.com/ubuntu/ ... [UCA run, all "ok"] ... Aug 02 08:28:03.799415 vvr1-dc0 dc0-mirror-sync[277623]: Files to download: 0 B Aug 02 08:28:03.799737 vvr1-dc0 dc0-mirror-sync[277623]: Downloaded 15 kiB in 3s at 5.04 kiB/s. Aug 02 08:28:03.799737 vvr1-dc0 dc0-mirror-sync[277623]: Everything OK. Moving meta files ... Aug 02 08:28:03.813266 vvr1-dc0 dc0-mirror-sync[277623]: Cleanup mirror. Aug 02 08:28:03.833394 vvr1-dc0 dc0-mirror-sync[277623]: All done. Aug 02 08:28:03.840126 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Main process exited, code=exited, status=1/FAILURE Aug 02 08:28:03.840288 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Failed with result 'exit-code'. Aug 02 08:28:03.840605 vvr1-dc0 systemd[1]: Failed to start dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA). Aug 02 08:28:03.840948 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Consumed 5.294s CPU time. READING OF THE ABOVE -- this is the whole diagnosis: * "All done." appears TWICE, once per debmirror. The sweep quoted only one of them. * debmirror #1 (ubuntu, PID 276574) printed "Everything OK. Moving meta files ...", "Cleanup mirror.", and "All done." at 08:28:00.830 -- and then, TEN MILLISECONDS LATER, in the SAME process, printed its error block: Errors: Download of dists/jammy-backports/main/dep11/Components-amd64.yml.gz failed: 500 read timeout Failed to download files (1 errors)! That is R1, observed directly: "All done." followed by a non-zero exit from debmirror itself. The sweep's line "grep 'Everything OK': 0 hits" was true of the repo, but the message it points at is NOT an attestation of success -- debmirror prints it about the meta-file move, before it reports its error tally. * Between the LAST "All done." (08:28:03.833394, the UCA run) and systemd's failure line (08:28:03.840126) there is NOTHING AT ALL -- 6.7 milliseconds and zero log lines. There is no post-"All done." step to fail. R2 has no candidate. * The two statuses ARE combined exactly as the repo predicted: ubuntu=1 (debmirror #1's real exit) AND uca=0 (debmirror #2 clean) -> the `&&` test fails -> the helper writes FAIL and exits 1 -> systemd records status=1/FAILURE. -------------------------------------------------------------------------------------- 2.4 PRIMARY SOURCE -- debmirror's own exit logic -------------------------------------------------------------------------------------- COMMAND: ssh vr1-dc0-rack 'dpkg-query -W -f="${Package} ${Version}\n" debmirror' OUTPUT: debmirror 1:2.39ubuntu2 COMMAND: ssh vr1-dc0-rack 'sed -n "1605,1625p" /usr/bin/debmirror' OUTPUT (verbatim): system("find . -depth -type d ! -name . ! -name .. -print0 2>/dev/null | xargs -0 rmdir 2>/dev/null") if (! $do_dry_run); } if ($res != 0) { die("Failed to move some meta files."); } # Save the state cache. save_state_cache() if $state_cache_days && !$do_dry_run; say("All done."); $lock->release; print "Errors:\n ".join(" ",@errlog) if (@errlog); if ($num_errors != 0) { print "Failed to download files ($num_errors errors)!\n"; exit 1 if (!$ignore_small_errors); } exit; This is R1 in the vendor's source. Line 1615 prints "All done."; lines 1618-1620 then exit 1 when $num_errors != 0, unless --ignore-small-errors was passed (the helper does not pass it). The "All done." message is emitted BEFORE the error tally is consulted, so it carries no information about success. debmirror's exit-1-with-All-done path is by design, not an anomaly. -------------------------------------------------------------------------------------- 2.5 The repo's own check, run read-only against the rack -------------------------------------------------------------------------------------- COMMAND: ssh vr1-dc0-rack 'sudo bash -s' -- check dc0 < scripts/dc-mirror.sh OUTPUT (verbatim; exit code 1): OK /usr/local/sbin/dc0-mirror-net-apply matches OK /etc/systemd/system/dc0-mirror-net.service matches OK /etc/systemd/resolved.conf.d/dc0-mirror.conf matches OK /usr/local/sbin/dc0-mirror-sync matches OK /etc/systemd/system/dc0-mirror-sync.service matches OK /etc/systemd/system/dc0-mirror-sync.timer matches OK /etc/nginx/sites-available/dc0-mirror matches OK /etc/nginx/sites-enabled/dc0-mirror enabled OK dc0-mirror-net.service enabled OK dc0-mirror-sync.timer enabled OK 10.12.8.4/22 on virbr2 (vr1-dc0-metal-admin) OK default route via edge 10.12.4.1 OK nginx active OK mirror answers http://10.12.8.4/ (200) MISS last-sync FAILED: FAIL 2026-08-02T08:28:03Z ubuntu=1 uca=0 -- the mirror content is NOT attested current; investigate before any gate cites this check dc-mirror check (dc0): FAIL Every structural assertion passes. The SINGLE failing assertion is the status-file one. The apparatus is intact; only the last sync's exit code is red. -------------------------------------------------------------------------------------- 2.6 What the one failed download actually cost the mirror content -------------------------------------------------------------------------------------- COMMAND: ssh vr1-dc0-rack 'echo "=== other dists dep11 ==="; for d in jammy jammy-updates jammy-security jammy-backports; do echo "-- $d/main/dep11:"; sudo ls /var/lib/dc-mirror/ubuntu/dists/$d/main/dep11/ | tr "\n" " "; echo; done; echo "=== dep11 .gz anywhere in mirror ==="; sudo find /var/lib/dc-mirror/ubuntu/dists -name "Components-amd64.yml.gz" | head' (the `tr` is why each dist's listing is one space-separated line) OUTPUT (verbatim): === other dists dep11 === -- jammy/main/dep11: Components-amd64.yml.gz Components-amd64.yml.xz icons-128x128.tar.gz icons-128x128@2.tar.gz icons-48x48.tar.gz icons-48x48@2.tar.gz icons-64x64.tar.gz icons-64x64@2.tar.gz -- jammy-updates/main/dep11: Components-amd64.yml.gz Components-amd64.yml.xz icons-128x128.tar.gz icons-128x128@2.tar.gz icons-48x48.tar.gz icons-48x48@2.tar.gz icons-64x64.tar.gz icons-64x64@2.tar.gz -- jammy-security/main/dep11: Components-amd64.yml.gz Components-amd64.yml.xz icons-128x128.tar.gz icons-128x128@2.tar.gz icons-48x48.tar.gz icons-48x48@2.tar.gz icons-64x64.tar.gz icons-64x64@2.tar.gz -- jammy-backports/main/dep11: Components-amd64.yml.xz icons-128x128.tar.gz icons-128x128@2.tar.gz icons-48x48.tar.gz icons-48x48@2.tar.gz icons-64x64.tar.gz icons-64x64@2.tar.gz === dep11 .gz anywhere in mirror === /var/lib/dc-mirror/ubuntu/dists/jammy-backports/multiverse/dep11/Components-amd64.yml.gz /var/lib/dc-mirror/ubuntu/dists/jammy-backports/restricted/dep11/Components-amd64.yml.gz /var/lib/dc-mirror/ubuntu/dists/jammy-backports/universe/dep11/Components-amd64.yml.gz /var/lib/dc-mirror/ubuntu/dists/jammy-updates/multiverse/dep11/Components-amd64.yml.gz /var/lib/dc-mirror/ubuntu/dists/jammy-updates/main/dep11/Components-amd64.yml.gz /var/lib/dc-mirror/ubuntu/dists/jammy-updates/restricted/dep11/Components-amd64.yml.gz /var/lib/dc-mirror/ubuntu/dists/jammy-updates/universe/dep11/Components-amd64.yml.gz /var/lib/dc-mirror/ubuntu/dists/jammy-security/multiverse/dep11/Components-amd64.yml.gz /var/lib/dc-mirror/ubuntu/dists/jammy-security/main/dep11/Components-amd64.yml.gz /var/lib/dc-mirror/ubuntu/dists/jammy-security/restricted/dep11/Components-amd64.yml.gz (`| head` truncated this list at 10; jammy-backports/main is the one absent from it) COMMAND: ssh vr1-dc0-rack 'sudo ls -la /var/lib/dc-mirror/ubuntu/dists/jammy-backports/main/dep11/' OUTPUT (verbatim): total 76 drwxr-xr-x 2 root root 4096 Aug 2 08:28 . drwxr-xr-x 6 root root 4096 Aug 2 08:28 .. -rw-r--r-- 2 root root 5748 Aug 2 08:27 Components-amd64.yml.xz -rw-r--r-- 2 root root 21211 Jul 31 18:54 icons-128x128.tar.gz -rw-r--r-- 2 root root 29 Jul 31 18:54 icons-128x128@2.tar.gz -rw-r--r-- 2 root root 9535 Jul 31 18:54 icons-48x48.tar.gz -rw-r--r-- 2 root root 29 Jul 31 18:54 icons-48x48@2.tar.gz -rw-r--r-- 2 root root 11254 Jul 31 18:54 icons-64x64.tar.gz -rw-r--r-- 2 root root 29 Jul 31 18:54 icons-64x64@2.tar.gz COMMAND: ssh vr1-dc0-rack 'sudo ls -la --time-style=full-iso /var/lib/dc-mirror/ubuntu/dists/jammy/Release /var/lib/dc-mirror/ubuntu/dists/jammy-backports/Release /var/lib/dc-mirror/cloud-archive/dists/jammy-updates/caracal/Release' OUTPUT (verbatim): -rw-r--r-- 2 root root 6896 2026-08-02 08:28:03.426253832 +0000 /var/lib/dc-mirror/cloud-archive/dists/jammy-updates/caracal/Release -rw-r--r-- 2 root root 126303 2026-08-02 08:22:52.452165933 +0000 /var/lib/dc-mirror/ubuntu/dists/jammy-backports/Release -rw-r--r-- 2 root root 269219 2026-08-02 08:22:51.622157671 +0000 /var/lib/dc-mirror/ubuntu/dists/jammy/Release FINDING, STATED ONLY TO WHAT WAS MEASURED: the file named in THIS run's error tally, dists/jammy-backports/main/dep11/Components-amd64.yml.gz is ABSENT from the tree. Its .xz sibling IS present and was fetched 08:27 the same run. All three other suites' main/dep11 have BOTH compressions. NOT MEASURED -- do not upgrade the above into a completeness claim: I did NOT reconcile the mirror tree against the Release indices, so I cannot say this is the ONLY missing file tree-wide. Two reasons to be careful. (1) The `find` in the command above was truncated by `| head` at 10 results, by my own construction. (2) The Jul 29 run tallied 22 errors (2.7); whether every one of those files was later healed by the subsequent clean runs is an INFERENCE from those runs exiting 0, not a measurement of the tree. The accurate statement is: the mirror is short AT LEAST this one file that the Release file indexes, and that shortfall is the reason for THIS run's exit 1. NOT MEASURED, stated as a gap rather than reasoned: I did NOT run `apt-get update` against http://10.12.8.4/ubuntu from a node, so I cannot say from measurement whether a node's apt will error, warn, or silently prefer the present .xz for that DEP-11 target. Do not treat the missing .gz as proven harmless. -------------------------------------------------------------------------------------- 2.7 Is this exit 1 structural or transient? -- the run history -------------------------------------------------------------------------------------- COMMAND: ssh vr1-dc0-rack 'sudo journalctl -u dc0-mirror-sync.service --no-pager -o short-precise | grep -E "Starting dc0-mirror-sync|Main process exited|Deactivated successfully|Failed to download files|Errors:|Succeeded" | tail -40' OUTPUT (verbatim): Jul 25 06:20:23.086437 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)... Jul 25 06:23:59.740895 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Deactivated successfully. Jul 26 00:17:25.629177 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)... Jul 26 00:17:41.479366 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Deactivated successfully. Jul 27 00:49:42.738255 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)... Jul 27 00:54:50.486133 vvr1-dc0 dc0-mirror-sync[1560492]: Errors: Jul 27 00:54:54.918359 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Main process exited, code=exited, status=1/FAILURE Jul 27 08:43:26.286248 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)... Jul 27 08:43:46.701209 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Deactivated successfully. Jul 28 00:19:22.810166 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)... Jul 28 00:31:50.641036 vvr1-dc0 dc0-mirror-sync[1781467]: Errors: Jul 28 00:31:50.641036 vvr1-dc0 dc0-mirror-sync[1781467]: Failed to download files (3 errors)! Jul 28 00:31:54.219244 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Main process exited, code=exited, status=1/FAILURE Jul 29 00:59:22.814254 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)... Jul 29 01:04:07.202354 vvr1-dc0 dc0-mirror-sync[2010432]: Errors: Jul 29 01:04:07.202975 vvr1-dc0 dc0-mirror-sync[2010432]: Failed to download files (22 errors)! Jul 29 01:04:11.345883 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Main process exited, code=exited, status=1/FAILURE Jul 30 00:19:55.651325 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)... Jul 30 00:20:20.760385 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Deactivated successfully. Jul 31 00:53:04.486154 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)... Jul 31 00:54:09.868870 vvr1-dc0 dc0-mirror-sync[2530452]: Errors: Jul 31 00:54:09.869040 vvr1-dc0 dc0-mirror-sync[2530452]: Failed to download files (1 errors)! Jul 31 00:54:12.765561 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Main process exited, code=exited, status=1/FAILURE Jul 31 02:35:08.020275 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)... Jul 31 02:35:23.157357 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Deactivated successfully. Jul 31 18:29:26.194259 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)... Jul 31 18:29:34.722265 vvr1-dc0 dc0-mirror-sync[2732384]: Errors: Jul 31 18:29:37.767629 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Main process exited, code=exited, status=1/FAILURE Jul 31 18:54:17.662449 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)... Jul 31 18:56:20.793348 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Deactivated successfully. Aug 01 00:04:03.875310 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)... Aug 01 00:04:19.793913 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Deactivated successfully. Aug 02 00:41:12.490244 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)... Aug 02 00:42:11.917232 vvr1-dc0 dc0-mirror-sync[193811]: Errors: Aug 02 00:42:36.622309 vvr1-dc0 dc0-mirror-sync[193911]: Errors: Aug 02 00:42:36.633575 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Main process exited, code=exited, status=1/FAILURE Aug 02 08:22:47.841316 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)... Aug 02 08:28:00.840132 vvr1-dc0 dc0-mirror-sync[276574]: Errors: Aug 02 08:28:00.840132 vvr1-dc0 dc0-mirror-sync[276574]: Failed to download files (1 errors)! Aug 02 08:28:03.840126 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Main process exited, code=exited, status=1/FAILURE FINDING: this is INTERMITTENT, not structural. The window shown (`tail -40`, which may have truncated runs older than Jul 25) contains 14 "Starting" lines: 7 ended "Deactivated successfully" (exit 0): Jul 25 06:20, Jul 26 00:17, Jul 27 08:43, Jul 30 00:19, Jul 31 02:35, Jul 31 18:54, Aug 01 00:04 7 ended status=1/FAILURE: Jul 27 00:49, Jul 28 00:19, Jul 29 00:59, Jul 31 00:53, Jul 31 18:29, Aug 02 00:41, Aug 02 08:22 Clean runs DO happen with this exact helper, four of them in the last four days. The 7 failures further split into TWO distinct shapes, which matters: (i) "Errors:" FOLLOWED BY "Failed to download files (N errors)!" -- N = 3, 22, 1, 1 on Jul 28, Jul 29, Jul 31 00:53 and Aug 02 08:22. This is the R1 path: debmirror completed, printed "All done.", tallied N failed downloads, exit 1. The varying N is the signature of upstream flakiness, not a fixed defect. (ii) "Errors:" with NO tally line -- Jul 27 00:49, Jul 31 18:29, Aug 02 00:41. debmirror did not reach the tally, i.e. it died rather than completing. This is consistent with the dc-mirror.sh comment (lines 291-299) recording "FAIL 2026-07-27T00:54:54Z ubuntu=255" for the Jul 27 00:49 run -- 255, not 1. NOT INVESTIGATED here; noted so shape (ii) is not later mistaken for shape (i). The 2026-08-02 08:22 run under diagnosis is squarely shape (i). NOTE for the record: the Aug 02 00:41 run shows "Errors:" from BOTH PIDs (193811 and 193911) -- i.e. that run had the UCA leg failing too, consistent with the pre-egress- restore state the sweep describes. The 08:22 run has "Errors:" from the ubuntu PID only, which is what uca=0 records. -------------------------------------------------------------------------------------- 2.8 The upstream URL now answers -------------------------------------------------------------------------------------- COMMAND: ssh vr1-dc0-rack 'curl -sS -o /dev/null -w "HTTP %{http_code} size=%{size_download} time=%{time_total}s\n" -m 60 -I http://archive.ubuntu.com/ubuntu/dists/jammy-backports/main/dep11/Components-amd64.yml.gz' OUTPUT (verbatim): HTTP 200 size=0 time=0.434358s (HEAD request; size_download=0 is expected for -I.) The URL that timed out at 08:27 answers 200 in 0.43s now, from the rack, over the restored egress path. WHAT THIS DOES AND DOES NOT SHOW -- the distinction matters, because the original failure was a READ TIMEOUT AT 300s, i.e. the transfer BODY stalled mid-stream, not a connect or lookup failure. `curl -I` retrieves headers and never exercises the body read, so this result shows REACHABILITY ONLY: the host resolves, routes, and does not 404 that path. It is NOT evidence that a GET of the file would complete, and it is NOT by itself proof that the fault was transient. (The run history in 2.7 -- 7 clean runs interleaved with the failures, varying error counts -- is the actual evidence for intermittency; this probe only rules out a hard 404/unreachable.) Throughput on that path remains UNMEASURED; see section 4. ====================================================================================== 3. THINGS MEASURED THAT DIFFER FROM WHAT THE SWEEP FILE SAYS ====================================================================================== None of these are errors of fact about the status line or the content mtimes -- those check out. They are readings the sweep drew that the full journal does not support: (a) The sweep presents a SINGLE narrative -- "Files to download: 0 B" / "Downloaded 15 kiB in 3s" / "Everything OK. Moving meta files ..." / "Cleanup mirror." / "All done." -- as the log of the sync that then failed. Those five lines are ALL from the SECOND debmirror (UCA, PID 277623, 08:28:03), the leg that SUCCEEDED. The failing leg (ubuntu, PID 276574) reported "Files to download: 6349 B" / "Downloaded 2867 kiB in 313s" and, after its own "All done.", an explicit error block. So the premise "its own log says the sync succeeded" comes from reading the tail of the passing leg as if it were the whole run. (b) The sweep says the two readings were "not diagnosed further" and warns against assuming R1. R1 is nonetheless the correct one, and the evidence for it was already IN the journal the sweep quoted -- three lines below the last line it quoted. The error block was simply not read. (c) The sweep says `dc-mirror.sh check dc0` "will stay RED". Measured history (2.7) shows 7 of 14 runs exited 0, four of them in the last four days. The check is red for THIS run's status file; the daily timer's next clean run would flip it green on its own. "Stays red" overstates it -- the gate is flaky, not stuck. (d) The sweep says "the mirror IS current". Nearly, but at least one indexed file is genuinely absent from the tree (2.6). "Current except AT LEAST the DEP-11 Components-amd64.yml.gz in jammy-backports/main" is the accurate statement -- tree-wide completeness was not reconciled against the Release indices, so I cannot narrow "at least" to "exactly". ====================================================================================== 4. WHAT THIS DOES *NOT* SETTLE (declared, not reasoned over) ====================================================================================== * Whether the missing jammy-backports/main/dep11/Components-amd64.yml.gz breaks or merely annoys a node's `apt-get update` against this mirror. NOT MEASURED. * Whether the 500 read timeouts are archive.ubuntu.com being slow, the DC edge path being slow, or both. The 313s for 2867 kiB (9.16 kiB/s) on the failing run is strikingly slow and the timeout was exactly the 300s limit, but I did not run a throughput measurement, so the cause is UNMEASURED. Note the standing repo preference (dc-mirror.sh lines 46-55) to move the MAIN archive to mirror.uoregon.edu -- that preference is recorded as NOT yet applied, and this finding is adjacent to it, but nothing here rules on it. * Whether any OTHER site or script combines debmirror exit codes the same way. Not surveyed; out of scope for F2. ====================================================================================== 5. NO REMEDY IS PROPOSED HERE, BY INSTRUCTION ====================================================================================== The remedy needs an operator ruling. Recording only the shape of the choice, without picking: the check is currently strict on debmirror's exit code, and debmirror's exit code conflates "nothing mirrored" with "mirrored, minus N transient files". Any change that makes the gate tolerant of a non-zero debmirror is a change to what the gate ATTESTS, and relaxing an attestation is the 2026-07-27 false-green defect class this repo already owns. Nothing in dc-mirror.sh was edited, and no patch is offered. END.