| 2026-08-02 |

sweep F2 ROOT-CAUSED: one bad archive.ubuntu.com backend; "not transient" WITHDRAWN
...
Operator ruling, exact utterance: "Root-cause the curl/debmirror anomaly first" -- taken over
accepting the red gate, over recording error shape/count, and over --ignore-small-errors.
ROOT CAUSE IS UPSTREAM AND BACKEND-SPECIFIC, not a defect in this deployment.
archive.ubuntu.com is round-robin across NINE A records and ONE, 91.189.92.23, HANGS on
dists/jammy-backports/main/dep11/Components-amd64.yml.gz while serving its directory
siblings normally. The control is what makes it a finding rather than "a backend is down":
that same backend returns the .xz sibling 200 / 5776 B / 0.543s, and the other eight return
the .gz in 0.14-0.51s. The resolver ROTATES -- six consecutive lookups, six different
orderings -- so every fetch re-rolls the backend.
MEASURED RATE: 12 sequential fetches -> 11 x 200, 1 timeout (~8%), consistent with exactly
one bad record of nine.
THEREFORE THE PRIOR COMMIT'S "NOT TRANSIENT" IS WITHDRAWN (GA-R1 C2 -- measurement wins over
a document, including one written an hour earlier). The failure is probabilistic; two
consecutive debmirror failures were not the proof of persistence they appeared to be.
A hypothesis was tested and REFUTED, and is kept because it is the one a reader forms first:
debmirror sets keep_alive => 1 and curl does not reuse connections, so a stale persistent
connection looked like the answer. The test inverted it -- keep_alive=1 (debmirror's own
setting) returned 200 in 1s while keep_alive=0 timed out, same library, seconds apart.
ALSO CORRECTED: the "500" was never a server 500. LWP synthesises 500 for CLIENT-side
failures, so it is LWP reporting its own 300s timeout (debmirror:629). I had read it as an
archive-side error, which points the investigation the wrong way.
Two structural facts read from the vendor script, both bearing on options that were on the
table: there is NO dep11 exclusion option (dep11_from_release :1201 and get_dep11_files
:1384 are unconditional), so the mirror's scope cannot be narrowed without patching
debmirror; and --ignore-small-errors is NOT a bounded tolerance -- :2891 DISCARDS the error
count, so it would read clean on a wholly failed mirror. It stays refuted.
RESIDUAL, declared not explained: debmirror failed 2 of 2, which a naive 1-in-9 model
predicts at ~1.2%. The per-IP sweep tested each backend ONCE, so it establishes that
91.189.92.23 was bad, NOT that it is the only bad one.
SECOND, INDEPENDENT FINDING: the local Release expects 6349 B while every backend now serves
6361 B, and the object was republished DURING the investigation. A successful download would
now fail HASH verification against the stale local Release -- a different failure mode from
the timeout, not fixed by fixing the timeout.
scripts/dc-mirror.sh UNTOUCHED. repo-lint 0 fail; ledger-scan unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

sweep F2 re-trigger: failure is PERSISTENT, and measured not to affect the deploy
...
Operator ruling (operational, GA-R3 -- no architectural consequence, no D-number). Question
put: what should dc-mirror.sh's gate ATTEST, given debmirror's exit code cannot distinguish
"nothing mirrored" from "mirrored minus N transient files" from "debmirror died"? Exact
utterance: "Re-trigger the sync first, decide after". Run as a MEASUREMENT, not a remedy.
The attestation question stays OPEN.
NOT TRANSIENT. The re-run (10:26:05->10:31:20Z) failed identically -- same file, same error,
same 300s timeout, two hours on: jammy-backports/main/dep11/Components-amd64.yml.gz. This
CORRECTS the earlier "flaky, not stuck" reading as applied to this failure: the unit has a
history of transient failures AND now one persistent one, and the two were being conflated.
Waiting will not clear it.
The gap is now EXACT, replacing the agent's "at least one file missing": debmirror's own
"Files to download: 6349 B" equals that single file's size, so the entire outstanding delta
of the 952 GiB tree is this one file. Its .xz twin is present and size-correct per Release.
AND IT DOES NOT BREAK apt -- measured, not reasoned, which the prior agent had explicitly
declined to do. With a fully isolated apt state dir on the rack (nothing system-wide
touched, no package installed), apt-get update against the mirror's jammy-backports main
returns rc=0 and fetches Components [5748 B] -- the .xz. apt asks for the format that is
present and never asks for the missing .gz; apt-cache policy resolves the suite normally.
So the mirror is functionally complete for the deploy and the gate is red over a file no
consumer requests.
One anomaly left UNEXPLAINED rather than hand-waved: curl fetches that same .gz from the
same host in 0.44s (HTTP 200, 6349 B, real body read) while debmirror times out at 300s on
it twice. A proxy is ruled out -- unit env is HOME=/root only, and there is no proxy in
/etc/environment, apt.conf.d, profile.d or root's login shell. Not chased: it does not gate
the deploy, and guessing at an LWP-vs-curl difference would be the reasoning-instead-of-
measuring failure this repo keeps logging.
scripts/dc-mirror.sh UNTOUCHED. No tolerance relaxed, no assertion weakened.
repo-lint 0 fail; ledger-scan DOCFIX next-free 208 (unchanged, no new decoy token).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Queued-findings backlog: sweep F1 + F6 CLOSED, F2 diagnosed, F3/F4/F5 graduated, DOCFIX-207
...
Works the 2026-08-02 close sweep (docs/audit/queued-findings-20260802-stage5-edge-fold.txt)
with three read-only agents in parallel. Cites the SWEEP register (F1-F6); the runbook fold
register has its own F1-F12 and is untouched.
sweep F1 CLOSED -- the dc0 rack's staged deploy input matches the repo again. One gated scp
of overlays/vr1-dc0-vips.yaml, ed19d989e80da8da -> 80d861560a6b3c52. A single-file copy was
provably sufficient because the WHOLE staging dir was enumerated first: 14 files, exactly 1
diverged, 0 missing from the Step-4 deploy closure (policies/overrides.zip present at the
repo digest), 0 orphans. All 14 re-verified against CURRENT HEAD after the copy. The 0600
octavia PKI overlay is untouched (same digest/mode/mtime) and was hashed, never read.
Repo-side correctness MEASURED not inherited: 0 ULA legs, 39 GUA, render-drift 4/4 naming
the file with a proof-of-teeth case.
sweep F6 CLOSED, PASS -- the dc0 MAAS region DB is proven uncorrupted. pg_dump read every
page of every table in maasdb: 23,878,796 bytes / 37,199 lines / exit 0 / empty stderr,
completion marker asserted separately. F6's own diagnosis was wrong: snap confinement does
not reproduce as ubuntu. The discriminators are ROLE (maas, not ubuntu) and TRANSPORT (over
TCP the role is password-challenged, over the unix socket it needs no credential). Both
identity values now measured -- maasdb had been prose. Dump streamed, nothing persisted.
sweep F2 DIAGNOSED, not fixed -- reading R1 is true, R2 refuted. debmirror prints "All done."
then exits non-zero; confirmed at vendor source and re-verified independently here
(debmirror 1:2.39ubuntu2, 1615 say("All done."), 1620 exit 1 if !$ignore_small_errors).
Cause: one 500 read timeout on a jammy-backports dep11 index. TWO corrections to the sweep:
its "the log says it succeeded" quotes all came from the PASSING UCA leg; and "stays RED"
overstates it -- measured 16 runs, 7 finished, 9 failed, with four fail-then-succeed pairs
hours apart. No remedy applied: debmirror's exit code conflates "nothing mirrored" with
"mirrored minus N transient files", so any tolerance change alters what the gate attests.
DOCFIX-207 -- preflight P6 quoted "50 apps / 97 relations"; measured is 56 / 108. The same
figures were corrected in phase-01's own gate on 2026-07-10 and this copy was missed.
sweep F3/F4/F5 graduated from the audit capture to durable homes: systemctl show fabricating
Result=success for a non-existent unit -> platform-traps 5c; assert the harness case count
moved, and two scripts probing one endpoint must share the probe definition -> script-authoring.
Logged NOT fixed (hard rule 1): preflight P2 validates a merged input including
vr1-dc0-machines.yaml while phase4:527-528 says the file does not exist and Step 4 does not
pass it -- a gate grading a different artifact than the deploy consumes, on the Step-4 path.
And security-ledger SEC-029(3) calls ~/repo-stage a nine-file copy; it is fourteen.
session-ledger machine-derived block re-seeded (was the 2026-07-27 seed: 21 SEC / D-138;
now 28 SEC / D-141 / DOCFIX-208, each added SEC row verified against the register).
Gates: repo-lint 0 fail / 1 warn (legacy carve-out); run-tests-all ALL GREEN (97 harnesses,
count unchanged, so nothing moved silently); tests/preflight 43/43; tests/render-drift 4/4.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|