mirror-exitcode-diagnosis-dc0-20260802.txt
==========================================
Diagnosis of SWEEP FINDING F2 (docs/audit/queued-findings-20260802-stage5-edge-fold.txt
lines 37-55): "THE MIRROR SYNC EXITS 1 WHILE ITS OWN LOG SAYS THE SYNC SUCCEEDED".

NOTE ON SCOPE: this is the SWEEP F-register (queued-findings-20260802-stage5-edge-fold.txt).
It is NOT docs/runbook-fold-register.md, which carries its own separate F-numbers.
Nothing in the runbook-fold register was read or touched for this work.

Read-only diagnostic session, 2026-08-02. No state was changed on the rack or the
jumphost OTHER THAN nginx access-log entries generated by the repo check's own HTTP
probe (section 2.5) and by the upstream HEAD in 2.8: no sync re-trigger, no
`systemctl reset-failed`, no write to the status file, no repo edit, no git
operation. Every command run is reproduced verbatim below.

======================================================================================
VERDICT
======================================================================================

  R1 IS TRUE. R2 IS REFUTED.

  debmirror itself has a non-fatal path that prints "All done." and then exits 1.
  Nothing in the sync helper runs after debmirror's final "All done." except the
  second debmirror invocation and the status-file write; the helper's exit status is
  a faithful propagation of debmirror's own exit code, not an artifact of the helper.

  The status line `FAIL ... ubuntu=1 uca=0` is literally debmirror-#1's exit code
  captured in `RC1=$?` on the line immediately following the invocation. There is no
  intervening command that could have overwritten it.

  Cause of the 1: ONE upstream download failed --
    dists/jammy-backports/main/dep11/Components-amd64.yml.gz -- 500 read timeout (300s)
  from archive.ubuntu.com. debmirror counted it in $num_errors, completed the mirror
  move and cleanup anyway, printed "All done.", THEN printed the error block and
  exited 1.

======================================================================================
1. REPO-SIDE READING (done first, per method) -- already discriminates R1 from R2
======================================================================================

Source: /home/jessea123/openstack-caracal-dc-dc/scripts/dc-mirror.sh

Values taken FROM THE SCRIPT (not guessed):
  MIRROR_ROOT   /var/lib/dc-mirror                        (line 110)
  SYNC_HELPER   /usr/local/sbin/dc0-mirror-sync           (line 114, ${SITE}=dc0)
  SYNC_UNIT     /etc/systemd/system/dc0-mirror-sync.service   (line 115)
  SYNC_TIMER    /etc/systemd/system/dc0-mirror-sync.timer     (line 116)
  STATUS FILE   /var/lib/dc-mirror/last-sync.status       (helper heredoc, line 180)
  Ubuntu tree   /var/lib/dc-mirror/ubuntu
  UCA tree      /var/lib/dc-mirror/cloud-archive

The sync helper is generated by gen_sync_helper() (dc-mirror.sh lines 168-201). Its
executable body, stripped of comments, is exactly:

    set -uo pipefail
    STATUS="/var/lib/dc-mirror/last-sync.status"
    echo "RUNNING $(date -u +%FT%TZ)" > "$STATUS"
    mkdir -p /var/lib/dc-mirror/ubuntu /var/lib/dc-mirror/cloud-archive
    debmirror /var/lib/dc-mirror/ubuntu \
      --host=archive.ubuntu.com --root=ubuntu --method=http \
      --dist=jammy,jammy-updates,jammy-security,jammy-backports \
      --section=main,restricted,universe,multiverse \
      --arch=amd64 --nosource --diff=none \
      --keyring=/usr/share/keyrings/ubuntu-archive-keyring.gpg --progress
    RC1=$?
    debmirror /var/lib/dc-mirror/cloud-archive \
      --host=ubuntu-cloud.archive.canonical.com --root=ubuntu --method=http \
      --dist=jammy-updates/caracal --section=main \
      --arch=amd64 --nosource --diff=none \
      --keyring=/usr/share/keyrings/ubuntu-cloud-keyring.gpg --progress
    RC2=$?
    if [ "$RC1" -eq 0 ] && [ "$RC2" -eq 0 ]; then
      echo "OK $(date -u +%FT%TZ) ubuntu=0 uca=0" > "$STATUS"; exit 0
    fi
    echo "FAIL $(date -u +%FT%TZ) ubuntu=$RC1 uca=$RC2" > "$STATUS"; exit 1

WHAT THIS ESTABLISHES BEFORE ANY LIVE MEASUREMENT:

  a) There are TWO debmirror invocations (ubuntu, then cloud-archive) and their
     statuses ARE combined -- by the `if [ "$RC1" -eq 0 ] && [ "$RC2" -eq 0 ]` test.
     A non-zero from EITHER produces the same helper exit 1.

  b) `RC1=$?` is the very next command after the first debmirror (the backslashes are
     line continuations of one command, so $? is unambiguously debmirror's). Same for
     RC2. NOTHING runs between a debmirror and the capture of its status.

  c) The ONLY things that run after the second debmirror are: the `if` test, one
     `echo` to the status file, and `exit`. None of them can fail in a way that
     manufactures `ubuntu=1`.

  c2) `set -uo pipefail` is set WITHOUT `-e`, and that absence is LOAD-BEARING for
     the diagnosis. With `-e`, debmirror's non-zero would have terminated the helper
     immediately -- before the `echo "FAIL ..."` line ever ran -- and the status file
     would still read `RUNNING`, not `FAIL`. The mere EXISTENCE of a terminal
     `FAIL ... ubuntu=1 uca=0` line therefore proves the helper ran all the way to
     its last statement and propagated a status deliberately. This kills R2 on the
     status file alone, independently of the journal: a helper that had itself failed
     somewhere after "All done." could not have written that line.

  d) Therefore the value `1` in `ubuntu=1` can only have come from debmirror #1's own
     exit code. If debmirror #1 also printed "All done.", R1 is proven and R2 is
     structurally impossible.

  The remaining question the repo alone could not answer: WHICH debmirror printed the
  "All done." quoted in the sweep, and did debmirror #1 print one too. That is what
  the live journal settles -- CONFIRMING the repo reading rather than discovering it.

======================================================================================
2. LIVE MEASUREMENT
======================================================================================

Host reached as `vr1-dc0-rack` (ProxyJump voffice1 from ~/.ssh/config.d/vr1-sites,
as configured; no jump was hand-constructed). Rack hostname reports as vvr1-dc0.

CONVENTION FOR THIS SECTION: commands are reproduced as issued, except that the
trailing `2>&1` used to capture stderr into the transcript is omitted, and long
one-line commands are wrapped. Outputs are byte-verbatim; the ONLY edits to an
output block are elisions, and every elision is marked inline as `... [ ... ] ...`.

--------------------------------------------------------------------------------------
2.1 The INSTALLED sync helper vs the repo's generated content -- IDENTICAL
--------------------------------------------------------------------------------------

COMMAND:
  ssh vr1-dc0-rack 'hostname; echo ---; sudo cat /usr/local/sbin/dc0-mirror-sync'

OUTPUT (verbatim):
vvr1-dc0
---
#!/usr/bin/env bash
# dc0-mirror-sync -- generated by scripts/dc-mirror.sh (D-135 item 1);
# do not hand-edit. Jammy amd64 binary mirror + UCA caracal, GPG-verified.
# jammy-backports ADDED by the D-135 AMENDMENT 2026-07-31 (operator: "Sync the
# backports into the mirror"). It is NOT optional: the MAAS jammy image ships
# jammy-backports in its sources.list, and juju's apt-mirror model-config rewrites
# EVERY suite to this mirror -- so a mirror without it 404s and every charm's
# install hook fails on 'does not have a Release file'. That cost 22 units on the
# 2026-07-31 dc0 deploy. The mirror's suite list is a CONTRACT with the node image.
set -uo pipefail
STATUS="/var/lib/dc-mirror/last-sync.status"
echo "RUNNING $(date -u +%FT%TZ)" > "$STATUS"
mkdir -p /var/lib/dc-mirror/ubuntu /var/lib/dc-mirror/cloud-archive
debmirror /var/lib/dc-mirror/ubuntu \
  --host=archive.ubuntu.com --root=ubuntu --method=http \
  --dist=jammy,jammy-updates,jammy-security,jammy-backports \
  --section=main,restricted,universe,multiverse \
  --arch=amd64 --nosource --diff=none \
  --keyring=/usr/share/keyrings/ubuntu-archive-keyring.gpg --progress
RC1=$?
debmirror /var/lib/dc-mirror/cloud-archive \
  --host=ubuntu-cloud.archive.canonical.com --root=ubuntu --method=http \
  --dist=jammy-updates/caracal --section=main \
  --arch=amd64 --nosource --diff=none \
  --keyring=/usr/share/keyrings/ubuntu-cloud-keyring.gpg --progress
RC2=$?
if [ "$RC1" -eq 0 ] && [ "$RC2" -eq 0 ]; then
  echo "OK $(date -u +%FT%TZ) ubuntu=0 uca=0" > "$STATUS"; exit 0
fi
echo "FAIL $(date -u +%FT%TZ) ubuntu=$RC1 uca=$RC2" > "$STATUS"; exit 1

FINDING: the installed helper matches the repo's gen_sync_helper() output exactly.
This is CORROBORATED independently by the repo's own check (section 2.5), whose
`check_file` does a real `diff` of the installed file against the generated content
and printed "OK   /usr/local/sbin/dc0-mirror-sync matches". THERE IS NO DRIFT.

--------------------------------------------------------------------------------------
2.2 The unit, the status file, and the recorded exit status
--------------------------------------------------------------------------------------

COMMAND:
  ssh vr1-dc0-rack 'echo "=== unit ==="; sudo systemctl cat dc0-mirror-sync.service;
    echo "=== status file ==="; sudo cat /var/lib/dc-mirror/last-sync.status;
    echo "=== systemctl show ==="; sudo systemctl show dc0-mirror-sync.service
      -p Result -p ExecMainStatus -p ExecMainExitTimestamp -p UnitFileState -p ActiveState'

OUTPUT (verbatim):
=== unit ===
# /etc/systemd/system/dc0-mirror-sync.service
[Unit]
Description=dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)
After=dc0-mirror-net.service network-online.target
Requires=dc0-mirror-net.service

[Service]
Type=oneshot
# debmirror dies with "HOME not defined in environment!" under systemd's
# empty environment (it needs a GPG home) -- measured on the first dc0 run.
Environment=HOME=/root
ExecStart=/usr/local/sbin/dc0-mirror-sync
Nice=10
IOSchedulingClass=idle
=== status file ===
FAIL 2026-08-02T08:28:03Z ubuntu=1 uca=0
=== systemctl show ===
Result=exit-code
ExecMainExitTimestamp=Sun 2026-08-02 08:28:03 UTC
ExecMainStatus=1
UnitFileState=static
ActiveState=failed

NOTE: `systemctl cat` was used deliberately (not `systemctl show`) as the existence
check, per SWEEP FINDING F3's durable rule in the same sweep file.

The unit has ONE ExecStart and no ExecStartPost / ExecStopPost. So systemd's
status=1/FAILURE is the helper's own `exit 1` and nothing else. Confirms 1(c).

--------------------------------------------------------------------------------------
2.3 THE DECISIVE EVIDENCE -- full journal of the 2026-08-02 08:22 run
--------------------------------------------------------------------------------------

COMMAND:
  ssh vr1-dc0-rack 'sudo journalctl -u dc0-mirror-sync.service
    --since "2026-08-02 08:00" --until "2026-08-02 09:00" --no-pager -o short-precise'

The run spans two PIDs -- 276574 (debmirror #1, ubuntu) and 277623 (debmirror #2,
UCA). The decisive segment, verbatim and in order:

Aug 02 08:22:47.841316 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)...
Aug 02 08:22:47.959441 vvr1-dc0 dc0-mirror-sync[276574]: Mirroring to /var/lib/dc-mirror/ubuntu from http://archive.ubuntu.com/ubuntu/
Aug 02 08:22:47.959441 vvr1-dc0 dc0-mirror-sync[276574]: Dists: jammy,jammy-updates,jammy-security,jammy-backports
        ... [meta-file fetches, all "ok"] ...
Aug 02 08:22:56.221218 vvr1-dc0 dc0-mirror-sync[276574]: Get DEP-11 metadata files ...
Aug 02 08:27:56.506798 vvr1-dc0 dc0-mirror-sync[276574]: [ 44%] Getting: dists/jammy-backports/main/dep11/Components-amd64.yml.gz...          #** GET http://archive.ubuntu.com/ubuntu/dists/jammy-backports/main/dep11/Components-amd64.yml.gz ==> 500 read timeout (300s)
Aug 02 08:27:56.507531 vvr1-dc0 dc0-mirror-sync[276574]: Download of dists/jammy-backports/main/dep11/Components-amd64.yml.gz failed: 500 read timeout
        ... [remaining fetches, all "ok"] ...
Aug 02 08:27:59.992486 vvr1-dc0 dc0-mirror-sync[276574]: Get command-not-found metadata files ...
Aug 02 08:27:59.992486 vvr1-dc0 dc0-mirror-sync[276574]: Files to download: 6349 B
Aug 02 08:28:00.109956 vvr1-dc0 dc0-mirror-sync[276574]: Downloaded 2867 kiB in 313s at 9.16 kiB/s.
Aug 02 08:28:00.109956 vvr1-dc0 dc0-mirror-sync[276574]: Everything OK. Moving meta files ...
Aug 02 08:28:00.127979 vvr1-dc0 dc0-mirror-sync[276574]: Cleanup mirror.
Aug 02 08:28:00.830168 vvr1-dc0 dc0-mirror-sync[276574]: All done.
Aug 02 08:28:00.840132 vvr1-dc0 dc0-mirror-sync[276574]: Errors:
Aug 02 08:28:00.840132 vvr1-dc0 dc0-mirror-sync[276574]:  Download of dists/jammy-backports/main/dep11/Components-amd64.yml.gz failed: 500 read timeout
Aug 02 08:28:00.840132 vvr1-dc0 dc0-mirror-sync[276574]: Failed to download files (1 errors)!
Aug 02 08:28:00.963508 vvr1-dc0 dc0-mirror-sync[277623]: Mirroring to /var/lib/dc-mirror/cloud-archive from http://ubuntu-cloud.archive.canonical.com/ubuntu/
        ... [UCA run, all "ok"] ...
Aug 02 08:28:03.799415 vvr1-dc0 dc0-mirror-sync[277623]: Files to download: 0 B
Aug 02 08:28:03.799737 vvr1-dc0 dc0-mirror-sync[277623]: Downloaded 15 kiB in 3s at 5.04 kiB/s.
Aug 02 08:28:03.799737 vvr1-dc0 dc0-mirror-sync[277623]: Everything OK. Moving meta files ...
Aug 02 08:28:03.813266 vvr1-dc0 dc0-mirror-sync[277623]: Cleanup mirror.
Aug 02 08:28:03.833394 vvr1-dc0 dc0-mirror-sync[277623]: All done.
Aug 02 08:28:03.840126 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Main process exited, code=exited, status=1/FAILURE
Aug 02 08:28:03.840288 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Failed with result 'exit-code'.
Aug 02 08:28:03.840605 vvr1-dc0 systemd[1]: Failed to start dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA).
Aug 02 08:28:03.840948 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Consumed 5.294s CPU time.

READING OF THE ABOVE -- this is the whole diagnosis:

  * "All done." appears TWICE, once per debmirror. The sweep quoted only one of them.

  * debmirror #1 (ubuntu, PID 276574) printed "Everything OK. Moving meta files ...",
    "Cleanup mirror.", and "All done." at 08:28:00.830 -- and then, TEN MILLISECONDS
    LATER, in the SAME process, printed its error block:
        Errors:
         Download of dists/jammy-backports/main/dep11/Components-amd64.yml.gz failed: 500 read timeout
        Failed to download files (1 errors)!
    That is R1, observed directly: "All done." followed by a non-zero exit from
    debmirror itself. The sweep's line "grep 'Everything OK': 0 hits" was true of the
    repo, but the message it points at is NOT an attestation of success -- debmirror
    prints it about the meta-file move, before it reports its error tally.

  * Between the LAST "All done." (08:28:03.833394, the UCA run) and systemd's
    failure line (08:28:03.840126) there is NOTHING AT ALL -- 6.7 milliseconds and
    zero log lines. There is no post-"All done." step to fail. R2 has no candidate.

  * The two statuses ARE combined exactly as the repo predicted: ubuntu=1 (debmirror
    #1's real exit) AND uca=0 (debmirror #2 clean) -> the `&&` test fails -> the
    helper writes FAIL and exits 1 -> systemd records status=1/FAILURE.

--------------------------------------------------------------------------------------
2.4 PRIMARY SOURCE -- debmirror's own exit logic
--------------------------------------------------------------------------------------

COMMAND:
  ssh vr1-dc0-rack 'dpkg-query -W -f="${Package} ${Version}\n" debmirror'
OUTPUT:
debmirror 1:2.39ubuntu2

COMMAND:
  ssh vr1-dc0-rack 'sed -n "1605,1625p" /usr/bin/debmirror'
OUTPUT (verbatim):
  system("find . -depth -type d ! -name . ! -name .. -print0 2>/dev/null | xargs -0 rmdir 2>/dev/null") if (! $do_dry_run);
}

if ($res != 0) {
  die("Failed to move some meta files.");
}

# Save the state cache.
save_state_cache() if $state_cache_days && !$do_dry_run;

say("All done.");
$lock->release;
print "Errors:\n ".join(" ",@errlog) if (@errlog);
if ($num_errors != 0) {
  print "Failed to download files ($num_errors errors)!\n";
  exit 1 if (!$ignore_small_errors);
}

exit;

This is R1 in the vendor's source. Line 1615 prints "All done."; lines 1618-1620 then
exit 1 when $num_errors != 0, unless --ignore-small-errors was passed (the helper does
not pass it). The "All done." message is emitted BEFORE the error tally is consulted,
so it carries no information about success. debmirror's exit-1-with-All-done path is
by design, not an anomaly.

--------------------------------------------------------------------------------------
2.5 The repo's own check, run read-only against the rack
--------------------------------------------------------------------------------------

COMMAND:
  ssh vr1-dc0-rack 'sudo bash -s' -- check dc0 < scripts/dc-mirror.sh

OUTPUT (verbatim; exit code 1):
  OK   /usr/local/sbin/dc0-mirror-net-apply matches
  OK   /etc/systemd/system/dc0-mirror-net.service matches
  OK   /etc/systemd/resolved.conf.d/dc0-mirror.conf matches
  OK   /usr/local/sbin/dc0-mirror-sync matches
  OK   /etc/systemd/system/dc0-mirror-sync.service matches
  OK   /etc/systemd/system/dc0-mirror-sync.timer matches
  OK   /etc/nginx/sites-available/dc0-mirror matches
  OK   /etc/nginx/sites-enabled/dc0-mirror enabled
  OK   dc0-mirror-net.service enabled
  OK   dc0-mirror-sync.timer enabled
  OK   10.12.8.4/22 on virbr2 (vr1-dc0-metal-admin)
  OK   default route via edge 10.12.4.1
  OK   nginx active
  OK   mirror answers http://10.12.8.4/ (200)
  MISS last-sync FAILED: FAIL 2026-08-02T08:28:03Z ubuntu=1 uca=0 -- the mirror content is NOT attested current; investigate before any gate cites this check
dc-mirror check (dc0): FAIL

Every structural assertion passes. The SINGLE failing assertion is the status-file
one. The apparatus is intact; only the last sync's exit code is red.

--------------------------------------------------------------------------------------
2.6 What the one failed download actually cost the mirror content
--------------------------------------------------------------------------------------

COMMAND:
  ssh vr1-dc0-rack 'echo "=== other dists dep11 ==="; for d in jammy jammy-updates
     jammy-security jammy-backports; do echo "-- $d/main/dep11:";
     sudo ls /var/lib/dc-mirror/ubuntu/dists/$d/main/dep11/ | tr "\n" " "; echo; done;
     echo "=== dep11 .gz anywhere in mirror ==="; sudo find
     /var/lib/dc-mirror/ubuntu/dists -name "Components-amd64.yml.gz" | head'
  (the `tr` is why each dist's listing is one space-separated line)

OUTPUT (verbatim):
=== other dists dep11 ===
-- jammy/main/dep11:
Components-amd64.yml.gz Components-amd64.yml.xz icons-128x128.tar.gz icons-128x128@2.tar.gz icons-48x48.tar.gz icons-48x48@2.tar.gz icons-64x64.tar.gz icons-64x64@2.tar.gz
-- jammy-updates/main/dep11:
Components-amd64.yml.gz Components-amd64.yml.xz icons-128x128.tar.gz icons-128x128@2.tar.gz icons-48x48.tar.gz icons-48x48@2.tar.gz icons-64x64.tar.gz icons-64x64@2.tar.gz
-- jammy-security/main/dep11:
Components-amd64.yml.gz Components-amd64.yml.xz icons-128x128.tar.gz icons-128x128@2.tar.gz icons-48x48.tar.gz icons-48x48@2.tar.gz icons-64x64.tar.gz icons-64x64@2.tar.gz
-- jammy-backports/main/dep11:
Components-amd64.yml.xz icons-128x128.tar.gz icons-128x128@2.tar.gz icons-48x48.tar.gz icons-48x48@2.tar.gz icons-64x64.tar.gz icons-64x64@2.tar.gz
=== dep11 .gz anywhere in mirror ===
/var/lib/dc-mirror/ubuntu/dists/jammy-backports/multiverse/dep11/Components-amd64.yml.gz
/var/lib/dc-mirror/ubuntu/dists/jammy-backports/restricted/dep11/Components-amd64.yml.gz
/var/lib/dc-mirror/ubuntu/dists/jammy-backports/universe/dep11/Components-amd64.yml.gz
/var/lib/dc-mirror/ubuntu/dists/jammy-updates/multiverse/dep11/Components-amd64.yml.gz
/var/lib/dc-mirror/ubuntu/dists/jammy-updates/main/dep11/Components-amd64.yml.gz
/var/lib/dc-mirror/ubuntu/dists/jammy-updates/restricted/dep11/Components-amd64.yml.gz
/var/lib/dc-mirror/ubuntu/dists/jammy-updates/universe/dep11/Components-amd64.yml.gz
/var/lib/dc-mirror/ubuntu/dists/jammy-security/multiverse/dep11/Components-amd64.yml.gz
/var/lib/dc-mirror/ubuntu/dists/jammy-security/main/dep11/Components-amd64.yml.gz
/var/lib/dc-mirror/ubuntu/dists/jammy-security/restricted/dep11/Components-amd64.yml.gz
(`| head` truncated this list at 10; jammy-backports/main is the one absent from it)

COMMAND:
  ssh vr1-dc0-rack 'sudo ls -la /var/lib/dc-mirror/ubuntu/dists/jammy-backports/main/dep11/'
OUTPUT (verbatim):
total 76
drwxr-xr-x 2 root root  4096 Aug  2 08:28 .
drwxr-xr-x 6 root root  4096 Aug  2 08:28 ..
-rw-r--r-- 2 root root  5748 Aug  2 08:27 Components-amd64.yml.xz
-rw-r--r-- 2 root root 21211 Jul 31 18:54 icons-128x128.tar.gz
-rw-r--r-- 2 root root    29 Jul 31 18:54 icons-128x128@2.tar.gz
-rw-r--r-- 2 root root  9535 Jul 31 18:54 icons-48x48.tar.gz
-rw-r--r-- 2 root root    29 Jul 31 18:54 icons-48x48@2.tar.gz
-rw-r--r-- 2 root root 11254 Jul 31 18:54 icons-64x64.tar.gz
-rw-r--r-- 2 root root    29 Jul 31 18:54 icons-64x64@2.tar.gz

COMMAND:
  ssh vr1-dc0-rack 'sudo ls -la --time-style=full-iso
     /var/lib/dc-mirror/ubuntu/dists/jammy/Release
     /var/lib/dc-mirror/ubuntu/dists/jammy-backports/Release
     /var/lib/dc-mirror/cloud-archive/dists/jammy-updates/caracal/Release'
OUTPUT (verbatim):
-rw-r--r-- 2 root root   6896 2026-08-02 08:28:03.426253832 +0000 /var/lib/dc-mirror/cloud-archive/dists/jammy-updates/caracal/Release
-rw-r--r-- 2 root root 126303 2026-08-02 08:22:52.452165933 +0000 /var/lib/dc-mirror/ubuntu/dists/jammy-backports/Release
-rw-r--r-- 2 root root 269219 2026-08-02 08:22:51.622157671 +0000 /var/lib/dc-mirror/ubuntu/dists/jammy/Release

FINDING, STATED ONLY TO WHAT WAS MEASURED: the file named in THIS run's error tally,
  dists/jammy-backports/main/dep11/Components-amd64.yml.gz
is ABSENT from the tree. Its .xz sibling IS present and was fetched 08:27 the same
run. All three other suites' main/dep11 have BOTH compressions.

NOT MEASURED -- do not upgrade the above into a completeness claim: I did NOT
reconcile the mirror tree against the Release indices, so I cannot say this is the
ONLY missing file tree-wide. Two reasons to be careful. (1) The `find` in the command
above was truncated by `| head` at 10 results, by my own construction. (2) The Jul 29
run tallied 22 errors (2.7); whether every one of those files was later healed by the
subsequent clean runs is an INFERENCE from those runs exiting 0, not a measurement of
the tree. The accurate statement is: the mirror is short AT LEAST this one file that
the Release file indexes, and that shortfall is the reason for THIS run's exit 1.

NOT MEASURED, stated as a gap rather than reasoned: I did NOT run `apt-get update`
against http://10.12.8.4/ubuntu from a node, so I cannot say from measurement whether
a node's apt will error, warn, or silently prefer the present .xz for that DEP-11
target. Do not treat the missing .gz as proven harmless.

--------------------------------------------------------------------------------------
2.7 Is this exit 1 structural or transient? -- the run history
--------------------------------------------------------------------------------------

COMMAND:
  ssh vr1-dc0-rack 'sudo journalctl -u dc0-mirror-sync.service --no-pager -o short-precise
     | grep -E "Starting dc0-mirror-sync|Main process exited|Deactivated successfully|Failed to download files|Errors:|Succeeded" | tail -40'

OUTPUT (verbatim):
Jul 25 06:20:23.086437 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)...
Jul 25 06:23:59.740895 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Deactivated successfully.
Jul 26 00:17:25.629177 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)...
Jul 26 00:17:41.479366 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Deactivated successfully.
Jul 27 00:49:42.738255 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)...
Jul 27 00:54:50.486133 vvr1-dc0 dc0-mirror-sync[1560492]: Errors:
Jul 27 00:54:54.918359 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Main process exited, code=exited, status=1/FAILURE
Jul 27 08:43:26.286248 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)...
Jul 27 08:43:46.701209 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Deactivated successfully.
Jul 28 00:19:22.810166 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)...
Jul 28 00:31:50.641036 vvr1-dc0 dc0-mirror-sync[1781467]: Errors:
Jul 28 00:31:50.641036 vvr1-dc0 dc0-mirror-sync[1781467]: Failed to download files (3 errors)!
Jul 28 00:31:54.219244 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Main process exited, code=exited, status=1/FAILURE
Jul 29 00:59:22.814254 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)...
Jul 29 01:04:07.202354 vvr1-dc0 dc0-mirror-sync[2010432]: Errors:
Jul 29 01:04:07.202975 vvr1-dc0 dc0-mirror-sync[2010432]: Failed to download files (22 errors)!
Jul 29 01:04:11.345883 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Main process exited, code=exited, status=1/FAILURE
Jul 30 00:19:55.651325 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)...
Jul 30 00:20:20.760385 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Deactivated successfully.
Jul 31 00:53:04.486154 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)...
Jul 31 00:54:09.868870 vvr1-dc0 dc0-mirror-sync[2530452]: Errors:
Jul 31 00:54:09.869040 vvr1-dc0 dc0-mirror-sync[2530452]: Failed to download files (1 errors)!
Jul 31 00:54:12.765561 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Main process exited, code=exited, status=1/FAILURE
Jul 31 02:35:08.020275 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)...
Jul 31 02:35:23.157357 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Deactivated successfully.
Jul 31 18:29:26.194259 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)...
Jul 31 18:29:34.722265 vvr1-dc0 dc0-mirror-sync[2732384]: Errors:
Jul 31 18:29:37.767629 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Main process exited, code=exited, status=1/FAILURE
Jul 31 18:54:17.662449 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)...
Jul 31 18:56:20.793348 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Deactivated successfully.
Aug 01 00:04:03.875310 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)...
Aug 01 00:04:19.793913 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Deactivated successfully.
Aug 02 00:41:12.490244 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)...
Aug 02 00:42:11.917232 vvr1-dc0 dc0-mirror-sync[193811]: Errors:
Aug 02 00:42:36.622309 vvr1-dc0 dc0-mirror-sync[193911]: Errors:
Aug 02 00:42:36.633575 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Main process exited, code=exited, status=1/FAILURE
Aug 02 08:22:47.841316 vvr1-dc0 systemd[1]: Starting dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA)...
Aug 02 08:28:00.840132 vvr1-dc0 dc0-mirror-sync[276574]: Errors:
Aug 02 08:28:00.840132 vvr1-dc0 dc0-mirror-sync[276574]: Failed to download files (1 errors)!
Aug 02 08:28:03.840126 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Main process exited, code=exited, status=1/FAILURE

FINDING: this is INTERMITTENT, not structural. The window shown (`tail -40`, which
may have truncated runs older than Jul 25) contains 14 "Starting" lines:
  7 ended "Deactivated successfully" (exit 0):
      Jul 25 06:20, Jul 26 00:17, Jul 27 08:43, Jul 30 00:19, Jul 31 02:35,
      Jul 31 18:54, Aug 01 00:04
  7 ended status=1/FAILURE:
      Jul 27 00:49, Jul 28 00:19, Jul 29 00:59, Jul 31 00:53, Jul 31 18:29,
      Aug 02 00:41, Aug 02 08:22
Clean runs DO happen with this exact helper, four of them in the last four days.

The 7 failures further split into TWO distinct shapes, which matters:
  (i) "Errors:" FOLLOWED BY "Failed to download files (N errors)!" -- N = 3, 22, 1, 1
      on Jul 28, Jul 29, Jul 31 00:53 and Aug 02 08:22. This is the R1 path: debmirror
      completed, printed "All done.", tallied N failed downloads, exit 1. The varying
      N is the signature of upstream flakiness, not a fixed defect.
  (ii) "Errors:" with NO tally line -- Jul 27 00:49, Jul 31 18:29, Aug 02 00:41.
      debmirror did not reach the tally, i.e. it died rather than completing. This is
      consistent with the dc-mirror.sh comment (lines 291-299) recording
      "FAIL 2026-07-27T00:54:54Z ubuntu=255" for the Jul 27 00:49 run -- 255, not 1.
      NOT INVESTIGATED here; noted so shape (ii) is not later mistaken for shape (i).
      The 2026-08-02 08:22 run under diagnosis is squarely shape (i).

NOTE for the record: the Aug 02 00:41 run shows "Errors:" from BOTH PIDs (193811 and
193911) -- i.e. that run had the UCA leg failing too, consistent with the pre-egress-
restore state the sweep describes. The 08:22 run has "Errors:" from the ubuntu PID
only, which is what uca=0 records.

--------------------------------------------------------------------------------------
2.8 The upstream URL now answers
--------------------------------------------------------------------------------------

COMMAND:
  ssh vr1-dc0-rack 'curl -sS -o /dev/null -w "HTTP %{http_code}  size=%{size_download}  time=%{time_total}s\n" -m 60 -I http://archive.ubuntu.com/ubuntu/dists/jammy-backports/main/dep11/Components-amd64.yml.gz'

OUTPUT (verbatim):
HTTP 200  size=0  time=0.434358s

(HEAD request; size_download=0 is expected for -I.) The URL that timed out at 08:27
answers 200 in 0.43s now, from the rack, over the restored egress path.

WHAT THIS DOES AND DOES NOT SHOW -- the distinction matters, because the original
failure was a READ TIMEOUT AT 300s, i.e. the transfer BODY stalled mid-stream, not a
connect or lookup failure. `curl -I` retrieves headers and never exercises the body
read, so this result shows REACHABILITY ONLY: the host resolves, routes, and does not
404 that path. It is NOT evidence that a GET of the file would complete, and it is NOT
by itself proof that the fault was transient. (The run history in 2.7 -- 7 clean runs
interleaved with the failures, varying error counts -- is the actual evidence for
intermittency; this probe only rules out a hard 404/unreachable.) Throughput on that
path remains UNMEASURED; see section 4.

======================================================================================
3. THINGS MEASURED THAT DIFFER FROM WHAT THE SWEEP FILE SAYS
======================================================================================

None of these are errors of fact about the status line or the content mtimes -- those
check out. They are readings the sweep drew that the full journal does not support:

 (a) The sweep presents a SINGLE narrative -- "Files to download: 0 B" / "Downloaded
     15 kiB in 3s" / "Everything OK. Moving meta files ..." / "Cleanup mirror." /
     "All done." -- as the log of the sync that then failed. Those five lines are ALL
     from the SECOND debmirror (UCA, PID 277623, 08:28:03), the leg that SUCCEEDED.
     The failing leg (ubuntu, PID 276574) reported "Files to download: 6349 B" /
     "Downloaded 2867 kiB in 313s" and, after its own "All done.", an explicit error
     block. So the premise "its own log says the sync succeeded" comes from reading
     the tail of the passing leg as if it were the whole run.

 (b) The sweep says the two readings were "not diagnosed further" and warns against
     assuming R1. R1 is nonetheless the correct one, and the evidence for it was
     already IN the journal the sweep quoted -- three lines below the last line it
     quoted. The error block was simply not read.

 (c) The sweep says `dc-mirror.sh check dc0` "will stay RED". Measured history (2.7)
     shows 7 of 14 runs exited 0, four of them in the last four days. The check is
     red for THIS run's status file; the daily timer's next clean run would flip it
     green on its own. "Stays red" overstates it -- the gate is flaky, not stuck.

 (d) The sweep says "the mirror IS current". Nearly, but at least one indexed file is
     genuinely absent from the tree (2.6). "Current except AT LEAST the DEP-11
     Components-amd64.yml.gz in jammy-backports/main" is the accurate statement --
     tree-wide completeness was not reconciled against the Release indices, so I
     cannot narrow "at least" to "exactly".

======================================================================================
4. WHAT THIS DOES *NOT* SETTLE (declared, not reasoned over)
======================================================================================

 * Whether the missing jammy-backports/main/dep11/Components-amd64.yml.gz breaks or
   merely annoys a node's `apt-get update` against this mirror. NOT MEASURED.
 * Whether the 500 read timeouts are archive.ubuntu.com being slow, the DC edge path
   being slow, or both. The 313s for 2867 kiB (9.16 kiB/s) on the failing run is
   strikingly slow and the timeout was exactly the 300s limit, but I did not run a
   throughput measurement, so the cause is UNMEASURED. Note the standing repo
   preference (dc-mirror.sh lines 46-55) to move the MAIN archive to
   mirror.uoregon.edu -- that preference is recorded as NOT yet applied, and this
   finding is adjacent to it, but nothing here rules on it.
 * Whether any OTHER site or script combines debmirror exit codes the same way.
   Not surveyed; out of scope for F2.

======================================================================================
5. NO REMEDY IS PROPOSED HERE, BY INSTRUCTION
======================================================================================

The remedy needs an operator ruling. Recording only the shape of the choice, without
picking: the check is currently strict on debmirror's exit code, and debmirror's exit
code conflates "nothing mirrored" with "mirrored, minus N transient files". Any change
that makes the gate tolerant of a non-zero debmirror is a change to what the gate
ATTESTS, and relaxing an attestation is the 2026-07-27 false-green defect class this
repo already owns. Nothing in dc-mirror.sh was edited, and no patch is offered.

END.
