# Changelog -- 2026-08-02 (session 2): clearing the queued-findings backlog Session scope: work the queued items left by the 2026-08-02 close bookend (`docs/audit/queued-findings-20260802-stage5-edge-fold.txt`), in consequence order. Stage 5 remains OPEN; nothing here opens or closes a stage. **Numbering note, because two registers collide.** This session's items cite the SWEEP register (F1-F6, the queued-findings file above). The runbook fold register (`docs/runbook-fold-register.md`) has its OWN F1-F12 with different meanings -- sweep F5 is "share the probe definition", fold F5 is "build the OOB plane". Every reference below is prefixed `sweep F` or `fold F`. Do not drop the prefix. --- ## Item 1 -- DOCFIX-207: preflight P6 quoted a plan count stale since 2026-07-10 **Sweep S5(a).** `scripts/preflight.sh:369` printed the phase-01 Step 1.2 reminder as `plan: 50 apps / 97 relations`. The real figure is **56 apps / 108 relations**, MEASURED twice and recorded: `docs/CURRENT-STATE.md:955` (`juju deploy --dry-run` resolved all 56 charms, planned 108 relations, exit 0) and again 2026-07-31 against the fixed deploy input. Measurement wins over a document (GA-R1 C2). The staleness is older than it looks: `docs/archive/changelogs/changelog-20260710-phase01-bundle-staleness-fix.md:18` corrected the SAME figures in `phase-01`'s own Step 1.2 GATE on 2026-07-10 and missed this copy in preflight. So for three weeks the executable gate and the reminder pointing at it disagreed by 6 apps and 11 relations. Scope, stated so it is not overclaimed: this is a REMINDER string inside the `P6: stage-2 reminders (NOT run here)` block, not an assertion. Nothing gated on it and no harness asserts it (`grep '50 apps' tests/` -> one unrelated hit, a line-number probe for the P6 header). The defect is that a session comparing a live dry-run against it would read a correct plan as wrong. - Changed: `scripts/preflight.sh` -- the figure, plus a comment carrying the measurement provenance and the explicit note that the real gate is the dry-run itself. - Verified: `bash tests/preflight/run-tests.sh` -> **43/43 PASS**; `repo-lint` 0 fail. - **Revert:** restore `50 apps / 97 relations` on that echo line and drop the comment block above it. ## Item 2 -- sweep F3 graduated: `systemctl show` is not an existence check The sweep recorded that `systemctl show` on a NON-EXISTENT unit returns `Result=success` / `ExecMainStatus=0` / `RemainAfterExit=no` and no journal entries -- indistinguishable from a healthy oneshot that has completed. Measured 2026-08-02 against the invented name `dc0-rack-net`; the real units are `dc0-rack-legs` and `dc0-node-dns` (`scripts/dc-rack-net.sh:16,22`). It produced a false "inactive" reading that was reported before being caught. It lived only in the sweep capture, which is an audit artifact, not a place a future session looks before running a systemd check. Graduated to its durable home as **platform-traps section 5c**, with the verbatim output, the reason (`show` answers from property DEFAULTS), and the three instruments that DO error on absence (`systemctl cat`, `list-unit-files`, or asserting `UnitFileState` non-empty -- the only field that betrays the absence). - Changed: `.claude/skills/openstack-cloud-ops/references/platform-traps.md` (new 5c). - **Revert:** delete the 5c block. ## Item 3 -- sweep F4 + F5 graduated: two harness-discipline rules Both lived only in the sweep capture. Landed in `.claude/skills/openstack-cloud-ops/references/script-authoring.md`, in the Testing section immediately before the repo-record traps. **Sweep F4 -- assert the CASE COUNT moved.** Two new cases in `tests/dc-egress-check` were written with `fok`/`fbad`, helper names from `tests/dc-node-v6-carve`. The suite printed `fbad: command not found` to stderr, kept its count at 14, and reported ALL PASS. The mutations those cases targeted then "survived", reading as robustness when it was really the tests not existing. The compounding fact is the one worth carrying: the symptom of a non-existent test and of a passing test are the same symptom, so the delivery check has to be arithmetic -- count before, count after, delta equals cases written. Ties into the standing rule that a checker which cannot fail is not a gate: prove a new assertion can FAIL *and* prove it RAN. **Sweep F5 -- two scripts probing one endpoint must share the probe definition.** `dc-egress-check.sh`'s A4 and `dc-snap-proxy.sh:215` both hit `api.snapcraft.io`; only the latter sent `Snap-Device-Series: 16`, which the store requires. Measured 400 without it both through the proxy and direct, 200 with it. The newer gate called a healthy proxy broken on its first live run while the older script returned PASS with a real payload -- two gates disagreeing about one piece of infrastructure. The generalisable half, which was NOT on surface: **a fixture that mocks `curl` cannot catch a wrong REAL request**, so the harness stays green in both worlds. A duplicated request line is a duplicated contract, and only one copy gets fixed when the endpoint tightens. - Changed: `.claude/skills/openstack-cloud-ops/references/script-authoring.md` (two paragraphs). - **Revert:** delete the two paragraphs. ## Item 4 -- sweep F1 repo-side provenance MEASURED (no mutation yet) Before propagating `overlays/vr1-dc0-vips.yaml` to the dc0 rack's `~/repo-stage` (the D-138 deploy client), the repo copy's correctness was measured rather than inherited from the close bookend: - **It conforms to D-139 ruling B.** `grep -o 'fd50:[0-9a-f:]*'` -> **zero** ULA legs. 39 GUA legs, `2602:f3e2:f02:11` x13 (provider-public), `f02:20` x13, `f02:21` x13. That is exactly the 26-leg migration the 2026-08-01 conformance audit named as its blocker C (`docs/audit/d139-conformance-dc0-20260801.txt:44-56`), so blocker C is CLOSED in the repo copy. - **It has renderer provenance.** `bash tests/render-drift/run-tests.sh` -> 4/4 PASS, naming the file: `render/values/vr1-dc0-vips.yaml renders overlays/vr1-dc0-vips.yaml byte-for-byte`, and the suite carries a PROOF-OF-TEETH case (a mutated values file is detected). The values file declares `output: overlays/vr1-dc0-vips.yaml` at line 2. - Repo digest confirmed `80d861560a6b3c52`, matching what the close recorded. - Changed: nothing. Read-only measurement. - **Revert:** n/a. ## Item 5 -- sweep F1 CLOSED: the dc0 rack's deploy input is re-staged A read-only agent enumerated the WHOLE staging directory first, rather than diffing the one file the close bookend named. Capture: `docs/audit/repo-stage-drift-dc0-20260802.txt` (427 lines). Classification of the 14 regular files + 4 dirs under `~/repo-stage` (`.git` re-confirmed absent): - **(A) diverged: 1** -- `overlays/vr1-dc0-vips.yaml` only. Corroboration the bookend did not have: **voffice1's clone also holds `80d86156...`**, so the rack was the ONLY host still carrying the pre-GUA overlay. - **(B) missing from the Step-4 deploy closure: 0.** This was the open question, and it matters because a prior deploy died on a missing `policies/overrides.zip`. That file IS staged, at the repo digest `02fe1fd7...e720a011`, and resolves because `policies/` sits beside `bundle.yaml`. All three scannable merge inputs were checked, not just the bundle: `bundle.yaml` has exactly one local reference (`:227-228`); neither the VIP nor the machines overlay has a `resources:` block or any local path. - **(C) rack-only orphans: 0.** All 14 staged files resolve to a repo path. - Two later-step overlays are unstaged (`dc-dc-ipv6-family-matrix.yaml`, `dc-ha-scaleup.yaml`); not Step-4 inputs, recorded as a fork, not ruled. **THE MUTATION (gated, single file, executed 2026-08-02).** `scp -p overlays/vr1-dc0-vips.yaml vr1-dc0-rack:/home/jessea123/repo-stage/overlays/vr1-dc0-vips.yaml` A single-file copy was the minimal correct action BECAUSE the enumeration proved exactly one file diverged. A wholesale directory refresh was rejected for the same reason the 2026-07-31 re-stage rejected it: it risks clobbering or re-permissioning the `0600` `vr1-dc0-octavia-pki.yaml` (SEC-029). Before / after, measured by this session on both ends: ``` BEFORE AFTER REPO overlays/vr1-dc0-vips.yaml ed19d989e80da8da 80d861560a6b3c52 80d861560a6b3c52 bundle.yaml 4c8a78527a3100f6 (untouched) 4c8a78527a3100f6 overlays/vr1-dc0-machines.yaml e3be85e41b419486 (untouched) e3be85e41b419486 overlays/vr1-dc0-octavia-pki.yaml 5fc117f188332042 5fc117f188332042 (gitignored) ``` **POST-COPY, ALL FOURTEEN STAGED FILES WERE RE-VERIFIED AGAINST CURRENT HEAD**, not just the four in the table -- because the staged `scripts/` copies were last checked at an EARLIER commit and the 2026-08-02 session shipped script changes after that check. All 13 tracked files match the repo digest-for-digest (`bundle.yaml`, both non-PKI overlays, `policies/domain-manager-policy.yaml`, `policies/overrides.zip`, and the eight staged `scripts/`), and the 14th matches voffice1. **The staging directory is now fully in sync, confirmed against HEAD rather than inherited from the prior verification.** Checked and NOT a gap: `dc-egress-check.sh` is absent from `~/repo-stage` by design -- it is streamed over stdin (`ssh 'bash -s -- check ' < scripts/dc-egress-check.sh`, per its own refuse message at `:90-91`), so it is never staged and cannot go stale there. The PKI overlay is verified UNCHANGED -- same digest, same `0600`, same mtime (Jul 31 02:38) -- and was never read, only hashed. It measures IDENTICAL to voffice1's copy, so the SEC-029 superseded-cert hazard is not realized today; nothing enforces the dual update on the next reissue, which stays an open SEC-029 obligation. - **Revert:** copy the pre-GUA overlay back. It is NOT retained anywhere as a file -- reverting means re-rendering from `render/values/` at the parent of the GUA commit. There is no reason to: the staged copy now matches repo HEAD, which is the invariant. **TWO FINDINGS THE AGENT SURFACED THAT WERE NOT ASKED FOR, both logged not fixed:** 1. **The staged inventory is 14 files, not the nine that `docs/security-ledger.md` SEC-029(3) records.** The +5 are `opnsense-*` files left by the 2026-08-02 edge rebuild (digests match `dc0-edge-rebuild-20260802.txt:259-264`). A future re-stage planned off the "nine-file copy" figure would omit five files. The ledger's prose is stale, not wrong-in-kind. 2. **A GATE/DEPLOY ASYMMETRY, DOCFIX-shaped and worth attention before Step 4.** `overlays/vr1-dc0-machines.yaml` exists and `scripts/preflight.sh:144-145` folds it into the merged input that P2 validates -- but `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:527-528` asserts the file does NOT exist and Step 4 does not pass it. **So P2 validates a different artifact than the deploy consumes.** That is the class of defect this project has repeatedly found (a gate grading the workspace while the overlay is what reaches the charm). NOT fixed here -- hard rule 1; it likely bundles with the outstanding D-138 run-location DOCFIX. ## Item 6 -- sweep F6 CLOSED: the dc0 MAAS region DB is proven uncorrupted **PASS.** `pg_dump` read every page of every table in `maasdb`: **23,878,796 bytes / 37,199 lines / exit 0 / empty stderr**, with `-- PostgreSQL database dump complete` asserted in a separate pass. A torn or bad-checksum page anywhere would have aborted the dump, so the 2026-08-01 hard power cut left NO page-level corruption. Evidence appended to `docs/audit/maasdb-pgdump-integrity-dc0-20260802.txt` (the agent's BLOCKED report is left verbatim above it; sections 7-11 are the resolution). **F6's stated blocker was the wrong diagnosis, and this is the durable part.** The finding attributed five prior failures to snap confinement and prescribed a login shell. Neither was operative: - Confinement did NOT reproduce as `ubuntu` -- the `postgresql.dump` wrapper ran, opened the socket, and the server answered. (It DOES reproduce as `_daemon_`: `cannot create snap home dir: mkdir /nonexistent` + `is not a snap cgroup`. So the original reading was true for the user it was measured on and got generalised too far.) - The real discriminators were **ROLE** and **TRANSPORT**. Role `ubuntu` does not exist; the role is `maas`. And over TCP (`-h localhost`) role `maas` is password-challenged (`fe_sendauth: no password supplied`), which would have meant handling the regiond.conf secret -- while over the **UNIX SOCKET (`-h /tmp`) it authenticates with no credential at all.** Dump over the socket; never over TCP. **Both identity values are now MEASURED, and one of them was prose all along.** `maasdb` came from F6's text and had never been verified; `sudo grep -E "^database_name|^database_user|..."` on `regiond.conf` confirms `maasdb` / `maas` / `localhost`. Only non-secret field names were grepped -- `database_pass` was never read, and the dump path chosen above means it never needed to be. The dump was STREAMED through `awk` and nothing was persisted: the MAAS DB carries API keys and password hashes, so writing it to disk would have created a new exposure needing its own SEC row. **Scope, stated so it is not overclaimed:** this proves the pages are readable and the rows decode. It does NOT prove MAAS's application-level invariants hold. Nothing observed suggests otherwise and no such claim is made. **The agent's false-green finding, confirmed and worth keeping:** `SSH_EXIT` was 0 while `pg_dump` exited 1 having produced 0 bytes -- so F6's own proposed form, `ssh host 'pg_dump maasdb > /dev/null' && echo PASS`, would have printed PASS over a dump that never happened. Same family as sweep F3 and sweep F4. Asserting on BYTES plus the completion marker is what makes this a gate rather than a ritual. - Changed: `docs/audit/maasdb-pgdump-integrity-dc0-20260802.txt` (new capture). - **Revert:** n/a -- a read-only measurement. Nothing on the VM was altered. ## Item 7 -- sweep F2 DIAGNOSED: reading R1 is true, R2 is refuted Sweep F2 left two readings open and forbade assuming the first. A read-only agent settled it. Capture: `docs/audit/mirror-exitcode-diagnosis-dc0-20260802.txt`. **VERDICT: R1 -- debmirror itself prints `All done.` and then exits non-zero. Nothing runs after `All done.` that could fail.** R2 (something failing after the message) has no candidate step. **Decisive evidence -- the journal shows `All done.` TWICE, once per debmirror leg, and the failing one is followed 10 ms later by its own error block in the SAME PID:** ``` 08:28:00.830168 dc0-mirror-sync[276574]: All done. 08:28:00.840132 dc0-mirror-sync[276574]: Errors: 08:28:00.840132 dc0-mirror-sync[276574]: Download of dists/jammy-backports/main/dep11/Components-amd64.yml.gz failed: 500 read timeout 08:28:00.840132 dc0-mirror-sync[276574]: Failed to download files (1 errors)! ``` **Confirmed at VENDOR SOURCE, and re-verified independently by this session** (not taken on the agent's word, since it is the load-bearing claim) -- `debmirror 1:2.39ubuntu2` on the dc0 rack: ``` 1615:say("All done."); 1620: exit 1 if (!$ignore_small_errors); ``` The helper does not pass `--ignore-small-errors`, so any non-zero error tally exits 1 AFTER the success-looking message. Corroborating: between the LAST `All done.` (the UCA leg, 08:28:03.833394) and systemd's failure line (08:28:03.840126) there are 6.7 ms and ZERO log lines. And the installed helper is byte-identical to the repo's generated content -- no drift. **Proximate cause of the 1:** exactly one upstream download, `jammy-backports/main/dep11/Components-amd64.yml.gz`, hit a 500 read timeout at 300s. **THE SWEEP'S OWN PREMISE WAS WRONG, AND THE ERROR IS INSTRUCTIVE.** F2 states "its own log says the sync succeeded". Every line it quoted (`Files to download: 0 B`, `Downloaded 15 kiB in 3s`, `Everything OK...`, `All done.`) is from the SECOND debmirror, PID 277623 -- the UCA leg, which SUCCEEDED. The failing ubuntu leg reported `Downloaded 2867 kiB in 313s` and an error block. **The tail of the passing leg was read as the tail of the whole run, and the evidence for R1 sat three lines below the last line quoted.** Same family as this project's standing instrument lessons: a truncated read that looks exactly like a complete one. **A SECOND CORRECTION: "`dc-mirror.sh check dc0` stays RED" OVERSTATES IT -- the gate is FLAKY, not stuck.** The agent reported "7 of 14"; that did not reconcile against the journal, so it was re-counted. **MEASURED, full journal window 2026-07-23..08-02: 16 runs, 7 `Finished`, 9 `Failed with result 'exit-code'`.** The success count was right, the denominator was not. Its "four successes in the last four days" IS correct. **My own first re-count was ALSO wrong, and it is logged rather than quietly dropped:** a `grep -cE "Starting .*mirror sync"` returned 0 because the unit's description reads `dc0 mirror upstream sync`, so "mirror sync" never matched. It looked like "no runs recorded". Checking the instrument before believing a surprising negative is the standing rule and it paid again. **The stronger evidence is the SHAPE, not the ratio.** On FOUR separate days a failed run was followed within hours by a clean one: ``` 2026-07-25T06:20:23 Failed -> 2026-07-25T06:23:59 Finished 2026-07-27T00:54:54 Failed -> 2026-07-27T08:43:46 Finished 2026-07-31T00:54:12 Failed -> 2026-07-31T02:35:23 Finished 2026-07-31T18:29:37 Failed -> 2026-07-31T18:56:20 Finished ``` That is transience DEMONSTRATED, not inferred from an error string. Two caveats kept: the earliest failure (2026-07-23) was the HOME-under-systemd defect fixed the same hour, so the lifetime ratio understates current health; and the two most recent runs, both 2026-08-02, BOTH failed -- they bracket the edge outage, and the last success was 2026-08-01T00:04:19. **Failure shapes are TWO, flagged so they are not later conflated:** 4 runs carry a `Failed to download files (N errors)!` tally (the R1 path, N = 1/3/22/1); 3 carry `Errors:` with no tally, meaning debmirror DIED rather than completing -- the `ubuntu=255` shape `dc-mirror.sh`'s own comment records for 2026-07-27. **NO REMEDY APPLIED, and the decision is the operator's (hard rule 1).** The agent recorded only the SHAPE of the choice, which is the part that matters: debmirror's exit code conflates "nothing mirrored" with "mirrored minus N transient files", so ANY tolerance change alters what the gate ATTESTS. Relaxing it is the 2026-07-27 false-green defect inverted. Options are put to the operator separately. **Declared gaps, not silently filled:** whether the missing `.gz` actually breaks a node's `apt-get update` (the `.xz` sibling is present; apt was NOT run from a node); whether tree-wide completeness holds (the tree was not reconciled against the Release indices, and the agent's `find` was truncated -- so "at least one file missing", never "exactly one"); and whether the 500 timeouts are archive.ubuntu.com or the edge path (a `curl -I` 200 shows reachability only -- HEAD never exercises the body read that timed out, so it is NOT evidence of transience). - Changed: `docs/audit/mirror-exitcode-diagnosis-dc0-20260802.txt` (new capture). `scripts/dc-mirror.sh` deliberately UNTOUCHED. - **Revert:** n/a -- diagnosis only. ## Item 8 -- sweep F2, the re-trigger: the failure is PERSISTENT, and it does not matter **OPERATOR RULING (operational, not a D-number per GA-R3 -- no architectural consequence).** Question put: what should `dc-mirror.sh`'s gate attest, given debmirror's exit code cannot distinguish "nothing mirrored" from "mirrored minus N transient files" from "debmirror died"? Options offered: record shape+count without relaxing / leave as is / re-trigger first and decide after / add a bounded tolerance (listed as NOT recommended). **Operator answer, exact utterance: "Re-trigger the sync first, decide after".** The re-trigger was therefore run as a MEASUREMENT, not as a remedy, and the attestation question stays OPEN. **RESULT: THE FAILURE IS NOT TRANSIENT.** The re-run (10:26:05 -> 10:31:20Z) failed identically -- `FAIL ... ubuntu=1 uca=0`, `Result=exit-code`, `ExecMainStatus=1` -- on the SAME file with the SAME error and the SAME 300s timeout, two hours after the previous attempt: ``` 08:27:56 Download of dists/jammy-backports/main/dep11/Components-amd64.yml.gz failed: 500 read timeout (300s) 10:31:16 Download of dists/jammy-backports/main/dep11/Components-amd64.yml.gz failed: 500 read timeout ``` So the earlier "flaky, not stuck" reading -- which the fail-then-succeed pairs supported -- does NOT extend to THIS failure. Two distinct things were being conflated: the unit has a history of transient failures, AND it now has one persistent one. The ruling's "decide after" is answered: waiting will not clear it. **THE SCOPE OF THE GAP IS NOW EXACT, replacing the agent's "at least one file missing".** debmirror's own `Files to download: 6349 B` equals the size of that single file, so the entire outstanding delta of the 952 GiB tree IS this one file. On disk: ``` present: Components-amd64.yml.xz 5748 B (Release lists 5748 -- size-correct) ABSENT: Components-amd64.yml.gz 6349 B (Release lists 6349) ``` **AND IT DOES NOT BREAK apt -- MEASURED, not reasoned.** The prior agent explicitly declared this unmeasured and declined to reason it away, which was correct. Tested from the rack with a fully ISOLATED apt state dir (scratch `Dir::State::Lists`, `Dir::Cache`, `Dir::Etc::SourceList`; nothing system-wide touched, no package installed): ``` deb http://10.12.8.4/ubuntu jammy-backports main Get:1 InRelease [127 kB] Get:2 main amd64 Packages [70.2 kB] Get:3 main amd64 Components [5748 B] <-- the .xz, the format that IS present Get:4 main amd64 c-n-f Metadata [412 B] APT_UPDATE_RC=0 apt-cache policy -> release v=22.04,o=Ubuntu,a=jammy-backports,c=main,b=amd64 ``` **apt asks for the `.xz` and never asks for the `.gz`.** So the mirror is FUNCTIONALLY complete for the deploy and the gate is red over a file no consumer requests. jammy-backports was chosen as the test suite deliberately -- it is the one carrying the missing file. **ONE ANOMALY, UNEXPLAINED AND LEFT THAT WAY rather than hand-waved.** The file fetches FINE by curl from the same host, right now: `http=200 size=6349 time=0.436s`, with a real body read (HEAD would prove nothing). Its `.xz` sibling and a jammy control also return 200. So curl succeeds in 0.44s where debmirror times out after 300s, twice. Ruled OUT: any proxy (the unit's environment is `HOME=/root` only; no `http_proxy` in `/etc/environment`, `apt.conf.d`, `profile.d`, or root's login shell). NOT chased further -- it does not gate the deploy, and guessing at an LWP-vs-curl difference would be exactly the reasoning-instead- of-measuring this project keeps logging. - Changed: nothing in the repo. `scripts/dc-mirror.sh` still UNTOUCHED; no tolerance was relaxed and no assertion weakened. - State changed on the rack: one sync run, which wrote `last-sync.status` and fetched ~1.3 MB of indices. The apt test wrote only to a scratch dir, removed on exit. - **Revert:** n/a. ## Item 9 -- sweep F2 ROOT-CAUSED, and item 8's "not transient" is WITHDRAWN **OPERATOR RULING, exact utterance: "Root-cause the curl/debmirror anomaly first"** -- taken over accepting the red gate, over recording error shape/count, and over `--ignore-small-errors`. Capture: `docs/audit/mirror-500-timeout-rootcause-20260802.txt`. **ROOT CAUSE: UPSTREAM AND BACKEND-SPECIFIC. `archive.ubuntu.com` is round-robin across NINE A records and ONE of them, `91.189.92.23`, HANGS on this specific object while serving its directory siblings normally.** ``` THE FAILING OBJECT (.yml.gz) CONTROL, SAME DIR (.yml.xz) 185.125.190.81 200 6361 0.305s 91.189.92.23 200 5776 0.543s <-- healthy ...(seven more, all 200) (all nine 200) 91.189.92.23 - - >25s TIMEOUT ``` The control is what makes this a finding rather than "a backend is down": the same backend serves the sibling in half a second. **The resolver ROTATES** -- six consecutive lookups returned six different orderings -- so each fetch re-rolls which backend it gets. **MEASURED RATE: 12 sequential fetches -> 11 x 200, 1 timeout (~8%)**, consistent with exactly one bad record of nine. **SO ITEM 8's "NOT TRANSIENT" IS WITHDRAWN (GA-R1 C2 -- measurement wins over a document, including one written an hour earlier).** The failure is PROBABILISTIC, not deterministic; two consecutive debmirror failures were not the proof of persistence they looked like. **A HYPOTHESIS TESTED AND REFUTED, kept because it is the one a reader forms first.** debmirror sets `keep_alive => 1` and curl does not reuse connections, so a stale persistent connection looked like the answer. The test INVERTED it -- same library, same callback style, seconds apart: ``` keep_alive=1 (debmirror's setting) 200 OK bytes=6361 1s keep_alive=0 (my "control") 500 read timeout bytes=0 30s ``` **ALSO CORRECTED: the `500` was never a server 500.** LWP synthesises status 500 for CLIENT-side failures, so it is LWP reporting its own 300s timeout (`/usr/bin/debmirror:629`, `our $timeout=300;`). I had read it as an archive-side error, which points the investigation the wrong way. **TWO STRUCTURAL FACTS, read from the vendor script rather than assumed** -- both bear on options that were on the table: there is **NO dep11 exclusion option** (`dep11_from_release` `:1201` and `get_dep11_files` `:1384` are unconditional), so the mirror's scope cannot be narrowed to drop AppStream metadata without patching debmirror; and **`--ignore-small-errors` is NOT a bounded tolerance -- `:2891` `$num_errors = $t if ($ignore_small_errors);` DISCARDS the count**, so it would read clean on a wholly failed mirror. **RESIDUAL, DECLARED NOT EXPLAINED:** debmirror failed 2 of 2, which a naive 1-in-9 model predicts at ~1.2%. Untested candidates: LWP's keep-alive pool may pin a whole run to one resolved address so one bad draw poisons the run; or more backends may be intermittently bad. **The per-IP sweep tested each backend ONCE, so it establishes that `91.189.92.23` was bad, NOT that it is the only bad one.** **SECOND, INDEPENDENT FINDING -- A DIFFERENT FAILURE MODE IS NOW LATENT.** The local `Release` expects **6349 B**; every backend now serves **6361 B**. The object was republished DURING this investigation (an early curl measured 6349, every later one 6361). **A successful download would now fail HASH verification against the stale local Release** -- a different error from the timeout, and not fixed by fixing the timeout. Whether debmirror re-fetches `Release` early enough each run to avoid this was NOT measured. - Changed: `docs/audit/mirror-500-timeout-rootcause-20260802.txt` (new capture). `scripts/dc-mirror.sh` STILL untouched. - **Revert:** n/a -- read-only investigation. ## Item 10 -- D-139 step 6 tool BUILT, adversarially reviewed, FOUR defects fixed `netbox/d139-step6-vip-rehome.py` + `tests/d139-step6-vip-rehome/run-tests.sh` (20 cases). Implements the two 2026-08-02 rulings: "Full step 6 first, then deploy" and "Deprecate both, delete nothing". CREATE 26 GUA VIP addresses -> read-back verify -> deprecate 26 ULA addresses + 9 ULA prefixes. **No delete path anywhere**, asserted against the artifact. **Dry run (live apex): `CREATE 26 | ALREADY 0 | DEPRECATE-ADDR 26 | DEPRECATE-PFX 9`.** **The 26 CREATE targets diff EXACTLY against the 26 GUA VIP legs in `overlays/vr1-dc0-vips.yaml`** -- every address written to the apex is one the deploy configures, and vice versa. Nothing has been written; `--commit` has not been run. **AN ADVERSARIAL REVIEW RETURNED "FIX FIRST" AND IT WAS RIGHT ON ALL FOUR COUNTS.** Capture: `docs/audit/d139-step6-tool-review-20260802.txt`. The mapping logic was correct; what was missing was PRECONDITIONS and COVERAGE. - **DEF-1 (CRITICAL) -- `--dc vr1-dc1` would ORPHAN-CREATE.** `gua_net_for_role()` computed targets arithmetically and never asked the apex whether they EXIST. MEASURED: dc1's GUA carve is incomplete -- four rows under `2602:f3e2:f03::/48`, all provider-public, no `:20::/64`, no `:21::/64` -- so dc1 planned 26 creates into non-existent prefixes then deprecated dc1's only authoritative rows, at rc=0 with no warning. **dc0 hid it because all sixteen of its targets happen to exist.** Reachable via the other valid value of a required flag. FIXED: the target must exist in the apex or the tool REFUSES. Verified live -- dc1 now refuses, dc0 unchanged at 26/26/9. - **DEF-2 (CRITICAL) -- the apex-IDENTITY guard was gone.** `d139-gua-carve.py:159-163` refuses any host that is not the working apex (DOCFIX-195), but that guard lives in its `main()`, and importing a module never runs its `main()`. **Subclassing `C.NB` inherited the TRANSPORT and left the SAFETY POSTURE behind** -- pointing this at `netbox.baldurkeep.com` (the v1 reference) would have connected fine and written to the wrong NetBox. FIXED: identity checked before any network call. - **DEF-3 (HIGH) -- silent under-count.** Removing one ULA `/64` row yielded `CREATE=13 / DEPA=13 / DEPP=8` at exit 0 -- 13 live VIPs neither created nor deprecated, reported as success. Same class as DEFECT 2 of the 2026-08-01 CORRECTION NOTE. FIXED: any ULA address claimed by no prefix row REFUSES. **My first fix was itself wrong and running it caught that** -- it scanned the whole retired `/48` and flagged dc1's 26 VIPs while planning dc0. The retired `/48` is SHARED (dc0 `:22x`, dc1 `:32x`), so the check now excludes addresses another DC's row claims, and deliberately does NOT count a `/60` parent as coverage -- the reviewer's scenario was exactly a missing `/64` whose `/60` still existed. - **DEF-4 (HIGH) -- `main()` had ZERO coverage.** The reviewer hoisted the deprecate loops above the create phase and the suite reported ALL PASS. The CREATE-before-DEPRECATE ordering both rulings call load-bearing was untested. FIXED: T16-T18 drive `main()` through a fake client that RECORDS CALL ORDER. **Proven by re-running the reviewer's exact mutation on a copy: T18 goes RED.** **TWO OF MY ASSERTIONS COULD NOT FAIL, and the review killed both.** **T13** asserted the ABSENCE of a string, so a traceback satisfied it -- it passed against a tool file that did not parse. Now requires a positive, well-formed, DIFFERENT target, and **T15** separately asserts the tool parses, closing the escape hatch. **T14** grepped ONE file, so adding a delete to the IMPORTED `d139-gua-carve.py` left it green; it now covers both. **Harness 14 -> 20 cases, count delta checked** (the rule landed in item 3 today, applied to my own delivery). Gauntlet manifest recorded deliberately, 97 -> 98. **Corrected in the ruling record (GA-R1 C2):** the amendment said the GUA records would be `status=active`. MEASURED: the live ULA VIP records are `reserved`, and `dc-plane-apex-import.py:186,200` creates addresses `reserved`. Both `docs/design-decisions.md` and `docs/CURRENT-STATE.md` corrected. **Also corrected -- my own overclaim:** the docstring said "every action is reversible by flipping a status back". True of the 26+9 DEPRECATIONS; NOT true of the 26 CREATES, which would need a delete this repo does not have. Reversal is not wanted (the creates are the records the deploy matches) but the claim was wrong and is now stated precisely. **OPEN SCOPE QUESTION, MEASURED, NOT A TOOL DEFECT -- THE MAAS HALF OF STEP 6.** D-139's list says retire the ULA rows "in the apex **and in MAAS**". This tool is apex-only, so step 6 is NOT complete when it finishes. MEASURED on the dc0 region: MAAS holds BOTH families -- five ULA `/64`s (`:220`, `:221`, `:230`, `:240`, `:250`) beside six GUA. Four are EMPTY; **`fd50:840e:74e2:220::/64` (metal-admin) still holds 2 allocated entries**, so this cannot be a blind cleanup. MAAS has no `deprecated` status for a subnet -- the choice is delete or leave -- which makes it a separate operator decision, not an extension of the "deprecate" ruling. - Changed: `netbox/d139-step6-vip-rehome.py` (new), `tests/d139-step6-vip-rehome/run-tests.sh` (new, 20 cases), `tests/HARNESS-MANIFEST` (+1, recorded deliberately), `docs/audit/d139-step6-tool-review-20260802.txt` (new), plus the two doc corrections. - **Revert:** delete the tool, its harness dir and the manifest line; re-run `bash scripts/run-tests-all.sh --record-manifest`. Nothing was written to the apex. ## Item 11 -- D-139 STEP 6 EXECUTED: apex applied, and 4 of 5 MAAS ULA subnets removed **OPERATOR APPROVAL, exact utterance: "Queue up the MAAS half to run after you complete netbox. I approve the MAAS deletes to clean up the data and config. Go ahead with both now".** **THE APEX HALF IS APPLIED, exit 0.** `d139-step6-vip-rehome.py --dc vr1-dc0 --commit`: `CREATE 26 | ALREADY 0 | DEPRECATE-ADDR 26 | DEPRECATE-PFX 9`, both read-backs OK. INDEPENDENTLY VERIFIED after the fact, not on the tool's own word: a re-run reports `CREATE 0 | ALREADY 26 | DEPRECATE-ADDR 0 | DEPRECATE-PFX 0` -- fully idempotent and converged. **RECORDED SO A LATER READER IS NOT MISLED:** `d139-gua-carve.py` STILL reports `dependent objects ... 26 ip-address(es)`. That is CORRECT and expected -- the 26 ULA records still EXIST, deprecated rather than deleted, per the ruling. The carve tool counts existence, not status. It is not evidence that step 6 did not run. **THE MAAS HALF: FOUR SUBNETS DELETED, ONE DELIBERATELY HELD.** **VERIFY BEFORE MUTATE FOUND A REAL HAZARD, and it is why the fifth was not deleted.** The approval was given before anyone knew WHAT the `fd50:840e:74e2:220::/64` allocations were. Measured: ``` fd50:840e:74e2:220::5 alloc_type=1 (STICKY) user=juju-vr1-dc0 <- the Juju controller fd50:840e:74e2:220::6 alloc_type=1 (STICKY) user=MAAS <- the MAAS region VM ``` **And they have NO GUA counterpart.** The GUA metal-admin subnet holds exactly nine entries -- `::100,::101,::102` control, `::120,::121` compute, `::150-::153` storage -- the nine tagged ROLE NODES and nothing else. This matches the record that `dc-node-v6-carve.py` walks only the nine tagged role nodes and that the controller's own v6 was "restored separately" -- restored on ULA, never migrated. **So deleting that subnet would remove the only recorded v6 address of the deploy client AND of the MAAS region the delete is issued to.** That is not cleanup, and it is outside what the approval could have meant. HELD, pending its own decision. **THE FOUR EMPTY ONES WERE SAFE, and that was established twice over.** Zero allocations AND **zero machine-interface links** for all five ULA subnets -- the stronger check, since "no allocated IP" does not by itself mean "nothing references it". Zero links also CONFIRMS D-139 step 3 succeeded: the nine role nodes are fully on GUA. Deleted INDIVIDUALLY, never looped -- hard rule 3 forbids batching destructive steps, and this repo has a logged incident from exactly that (2026-07-30, nine deletes in a loop; the guard stopped it). Each got a fresh cidr + allocation re-check immediately before its own delete: ``` subnet 9 fd50:840e:74e2:221::/64 metal-internal allocations=0 deleted rc=0 subnet 8 fd50:840e:74e2:230::/64 data-tenant allocations=0 deleted rc=0 subnet 12 fd50:840e:74e2:240::/64 storage allocations=0 deleted rc=0 subnet 11 fd50:840e:74e2:250::/64 replication allocations=0 deleted rc=0 ``` **POST-STATE, MEASURED:** six GUA `/64`s remain (`f02:10/20/21/30/40/50`), one ULA remains (`fd50:840e:74e2:220::/64`, id=2, held). Machines unchanged and healthy: **9 Ready + 1 Deployed** (the controller), total 10. **STILL OPEN, needs a decision:** the held subnet. Options are to migrate the controller's and region's v6 to GUA first and then delete, or to delete and accept losing those two records, or to leave the subnet in place indefinitely. NOT decided here. **NO REPO TOOL EXISTS for the MAAS half** -- these were hand-issued `maas subnet delete` calls. If the dc1 rebuild is to be reproducible, this belongs in a script with the same emptiness + link preconditions asserted, and that is registered as owed rather than built. - Changed: live apex (26 created, 35 deprecated) and the dc0 MAAS region (4 subnets deleted). Repo: this changelog + CURRENT-STATE. - **Revert:** apex -- flip the 35 statuses back (the 26 creates are additive and wanted). MAAS -- the four subnets can be recreated (`dc-region-topology.sh` carries the shape), but their identities are gone; they held nothing, so nothing else is lost.