# Changelog -- 2026-08-02 (session 2): clearing the queued-findings backlog

Session scope: work the queued items left by the 2026-08-02 close bookend
(`docs/audit/queued-findings-20260802-stage5-edge-fold.txt`), in consequence order.
Stage 5 remains OPEN; nothing here opens or closes a stage.

**Numbering note, because two registers collide.** This session's items cite the SWEEP
register (F1-F6, the queued-findings file above). The runbook fold register
(`docs/runbook-fold-register.md`) has its OWN F1-F12 with different meanings -- sweep F5
is "share the probe definition", fold F5 is "build the OOB plane". Every reference below
is prefixed `sweep F<n>` or `fold F<n>`. Do not drop the prefix.

---

## Item 1 -- DOCFIX-207: preflight P6 quoted a plan count stale since 2026-07-10

**Sweep S5(a).** `scripts/preflight.sh:369` printed the phase-01 Step 1.2 reminder as
`plan: 50 apps / 97 relations`. The real figure is **56 apps / 108 relations**, MEASURED
twice and recorded: `docs/CURRENT-STATE.md:955` (`juju deploy --dry-run` resolved all 56
charms, planned 108 relations, exit 0) and again 2026-07-31 against the fixed deploy
input. Measurement wins over a document (GA-R1 C2).

The staleness is older than it looks: `docs/archive/changelogs/changelog-20260710-phase01-bundle-staleness-fix.md:18`
corrected the SAME figures in `phase-01`'s own Step 1.2 GATE on 2026-07-10 and missed
this copy in preflight. So for three weeks the executable gate and the reminder pointing
at it disagreed by 6 apps and 11 relations.

Scope, stated so it is not overclaimed: this is a REMINDER string inside the
`P6: stage-2 reminders (NOT run here)` block, not an assertion. Nothing gated on it and
no harness asserts it (`grep '50 apps' tests/` -> one unrelated hit, a line-number probe
for the P6 header). The defect is that a session comparing a live dry-run against it
would read a correct plan as wrong.

- Changed: `scripts/preflight.sh` -- the figure, plus a comment carrying the measurement
  provenance and the explicit note that the real gate is the dry-run itself.
- Verified: `bash tests/preflight/run-tests.sh` -> **43/43 PASS**; `repo-lint` 0 fail.
- **Revert:** restore `50 apps / 97 relations` on that echo line and drop the comment
  block above it.

## Item 2 -- sweep F3 graduated: `systemctl show` is not an existence check

The sweep recorded that `systemctl show` on a NON-EXISTENT unit returns
`Result=success` / `ExecMainStatus=0` / `RemainAfterExit=no` and no journal entries --
indistinguishable from a healthy oneshot that has completed. Measured 2026-08-02 against
the invented name `dc0-rack-net`; the real units are `dc0-rack-legs` and `dc0-node-dns`
(`scripts/dc-rack-net.sh:16,22`). It produced a false "inactive" reading that was
reported before being caught.

It lived only in the sweep capture, which is an audit artifact, not a place a future
session looks before running a systemd check. Graduated to its durable home as
**platform-traps section 5c**, with the verbatim output, the reason (`show` answers from
property DEFAULTS), and the three instruments that DO error on absence (`systemctl cat`,
`list-unit-files`, or asserting `UnitFileState` non-empty -- the only field that betrays
the absence).

- Changed: `.claude/skills/openstack-cloud-ops/references/platform-traps.md` (new 5c).
- **Revert:** delete the 5c block.

## Item 3 -- sweep F4 + F5 graduated: two harness-discipline rules

Both lived only in the sweep capture. Landed in
`.claude/skills/openstack-cloud-ops/references/script-authoring.md`, in the Testing
section immediately before the repo-record traps.

**Sweep F4 -- assert the CASE COUNT moved.** Two new cases in `tests/dc-egress-check`
were written with `fok`/`fbad`, helper names from `tests/dc-node-v6-carve`. The suite
printed `fbad: command not found` to stderr, kept its count at 14, and reported ALL PASS.
The mutations those cases targeted then "survived", reading as robustness when it was
really the tests not existing. The compounding fact is the one worth carrying: the
symptom of a non-existent test and of a passing test are the same symptom, so the
delivery check has to be arithmetic -- count before, count after, delta equals cases
written. Ties into the standing rule that a checker which cannot fail is not a gate:
prove a new assertion can FAIL *and* prove it RAN.

**Sweep F5 -- two scripts probing one endpoint must share the probe definition.**
`dc-egress-check.sh`'s A4 and `dc-snap-proxy.sh:215` both hit `api.snapcraft.io`; only
the latter sent `Snap-Device-Series: 16`, which the store requires. Measured 400 without
it both through the proxy and direct, 200 with it. The newer gate called a healthy proxy
broken on its first live run while the older script returned PASS with a real payload --
two gates disagreeing about one piece of infrastructure. The generalisable half, which
was NOT on surface: **a fixture that mocks `curl` cannot catch a wrong REAL request**, so
the harness stays green in both worlds. A duplicated request line is a duplicated
contract, and only one copy gets fixed when the endpoint tightens.

- Changed: `.claude/skills/openstack-cloud-ops/references/script-authoring.md`
  (two paragraphs).
- **Revert:** delete the two paragraphs.

## Item 4 -- sweep F1 repo-side provenance MEASURED (no mutation yet)

Before propagating `overlays/vr1-dc0-vips.yaml` to the dc0 rack's `~/repo-stage` (the
D-138 deploy client), the repo copy's correctness was measured rather than inherited from
the close bookend:

- **It conforms to D-139 ruling B.** `grep -o 'fd50:[0-9a-f:]*'` -> **zero** ULA legs.
  39 GUA legs, `2602:f3e2:f02:11` x13 (provider-public), `f02:20` x13, `f02:21` x13.
  That is exactly the 26-leg migration the 2026-08-01 conformance audit named as its
  blocker C (`docs/audit/d139-conformance-dc0-20260801.txt:44-56`), so blocker C is
  CLOSED in the repo copy.
- **It has renderer provenance.** `bash tests/render-drift/run-tests.sh` -> 4/4 PASS,
  naming the file: `render/values/vr1-dc0-vips.yaml renders overlays/vr1-dc0-vips.yaml
  byte-for-byte`, and the suite carries a PROOF-OF-TEETH case (a mutated values file is
  detected). The values file declares `output: overlays/vr1-dc0-vips.yaml` at line 2.
- Repo digest confirmed `80d861560a6b3c52`, matching what the close recorded.

- Changed: nothing. Read-only measurement.
- **Revert:** n/a.

## Item 5 -- sweep F1 CLOSED: the dc0 rack's deploy input is re-staged

A read-only agent enumerated the WHOLE staging directory first, rather than diffing the
one file the close bookend named. Capture: `docs/audit/repo-stage-drift-dc0-20260802.txt`
(427 lines). Classification of the 14 regular files + 4 dirs under `~/repo-stage`
(`.git` re-confirmed absent):

- **(A) diverged: 1** -- `overlays/vr1-dc0-vips.yaml` only. Corroboration the bookend did
  not have: **voffice1's clone also holds `80d86156...`**, so the rack was the ONLY host
  still carrying the pre-GUA overlay.
- **(B) missing from the Step-4 deploy closure: 0.** This was the open question, and it
  matters because a prior deploy died on a missing `policies/overrides.zip`. That file IS
  staged, at the repo digest `02fe1fd7...e720a011`, and resolves because `policies/` sits
  beside `bundle.yaml`. All three scannable merge inputs were checked, not just the
  bundle: `bundle.yaml` has exactly one local reference (`:227-228`); neither the VIP nor
  the machines overlay has a `resources:` block or any local path.
- **(C) rack-only orphans: 0.** All 14 staged files resolve to a repo path.
- Two later-step overlays are unstaged (`dc-dc-ipv6-family-matrix.yaml`,
  `dc-ha-scaleup.yaml`); not Step-4 inputs, recorded as a fork, not ruled.

**THE MUTATION (gated, single file, executed 2026-08-02).**
`scp -p overlays/vr1-dc0-vips.yaml vr1-dc0-rack:/home/jessea123/repo-stage/overlays/vr1-dc0-vips.yaml`

A single-file copy was the minimal correct action BECAUSE the enumeration proved exactly
one file diverged. A wholesale directory refresh was rejected for the same reason the
2026-07-31 re-stage rejected it: it risks clobbering or re-permissioning the `0600`
`vr1-dc0-octavia-pki.yaml` (SEC-029).

Before / after, measured by this session on both ends:

```
                                    BEFORE            AFTER             REPO
overlays/vr1-dc0-vips.yaml          ed19d989e80da8da  80d861560a6b3c52  80d861560a6b3c52
bundle.yaml                         4c8a78527a3100f6  (untouched)       4c8a78527a3100f6
overlays/vr1-dc0-machines.yaml      e3be85e41b419486  (untouched)       e3be85e41b419486
overlays/vr1-dc0-octavia-pki.yaml   5fc117f188332042  5fc117f188332042  (gitignored)
```

**POST-COPY, ALL FOURTEEN STAGED FILES WERE RE-VERIFIED AGAINST CURRENT HEAD**, not just
the four in the table -- because the staged `scripts/` copies were last checked at an
EARLIER commit and the 2026-08-02 session shipped script changes after that check. All 13
tracked files match the repo digest-for-digest (`bundle.yaml`, both non-PKI overlays,
`policies/domain-manager-policy.yaml`, `policies/overrides.zip`, and the eight staged
`scripts/`), and the 14th matches voffice1. **The staging directory is now fully in sync,
confirmed against HEAD rather than inherited from the prior verification.**

Checked and NOT a gap: `dc-egress-check.sh` is absent from `~/repo-stage` by design -- it
is streamed over stdin (`ssh <rack> 'bash -s -- check <site>' < scripts/dc-egress-check.sh`,
per its own refuse message at `:90-91`), so it is never staged and cannot go stale there.

The PKI overlay is verified UNCHANGED -- same digest, same `0600`, same mtime
(Jul 31 02:38) -- and was never read, only hashed. It measures IDENTICAL to voffice1's
copy, so the SEC-029 superseded-cert hazard is not realized today; nothing enforces the
dual update on the next reissue, which stays an open SEC-029 obligation.

- **Revert:** copy the pre-GUA overlay back. It is NOT retained anywhere as a file --
  reverting means re-rendering from `render/values/` at the parent of the GUA commit.
  There is no reason to: the staged copy now matches repo HEAD, which is the invariant.

**TWO FINDINGS THE AGENT SURFACED THAT WERE NOT ASKED FOR, both logged not fixed:**

1. **The staged inventory is 14 files, not the nine that `docs/security-ledger.md`
   SEC-029(3) records.** The +5 are `opnsense-*` files left by the 2026-08-02 edge
   rebuild (digests match `dc0-edge-rebuild-20260802.txt:259-264`). A future re-stage
   planned off the "nine-file copy" figure would omit five files. The ledger's prose is
   stale, not wrong-in-kind.
2. **A GATE/DEPLOY ASYMMETRY, DOCFIX-shaped and worth attention before Step 4.**
   `overlays/vr1-dc0-machines.yaml` exists and `scripts/preflight.sh:144-145` folds it
   into the merged input that P2 validates -- but `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:527-528`
   asserts the file does NOT exist and Step 4 does not pass it. **So P2 validates a
   different artifact than the deploy consumes.** That is the class of defect this
   project has repeatedly found (a gate grading the workspace while the overlay is what
   reaches the charm). NOT fixed here -- hard rule 1; it likely bundles with the
   outstanding D-138 run-location DOCFIX.

## Item 6 -- sweep F6 CLOSED: the dc0 MAAS region DB is proven uncorrupted

**PASS.** `pg_dump` read every page of every table in `maasdb`: **23,878,796 bytes /
37,199 lines / exit 0 / empty stderr**, with `-- PostgreSQL database dump complete`
asserted in a separate pass. A torn or bad-checksum page anywhere would have aborted the
dump, so the 2026-08-01 hard power cut left NO page-level corruption. Evidence appended
to `docs/audit/maasdb-pgdump-integrity-dc0-20260802.txt` (the agent's BLOCKED report is
left verbatim above it; sections 7-11 are the resolution).

**F6's stated blocker was the wrong diagnosis, and this is the durable part.** The
finding attributed five prior failures to snap confinement and prescribed a login shell.
Neither was operative:

- Confinement did NOT reproduce as `ubuntu` -- the `postgresql.dump` wrapper ran, opened
  the socket, and the server answered. (It DOES reproduce as `_daemon_`: `cannot create
  snap home dir: mkdir /nonexistent` + `is not a snap cgroup`. So the original reading
  was true for the user it was measured on and got generalised too far.)
- The real discriminators were **ROLE** and **TRANSPORT**. Role `ubuntu` does not exist;
  the role is `maas`. And over TCP (`-h localhost`) role `maas` is password-challenged
  (`fe_sendauth: no password supplied`), which would have meant handling the regiond.conf
  secret -- while over the **UNIX SOCKET (`-h /tmp`) it authenticates with no credential
  at all.** Dump over the socket; never over TCP.

**Both identity values are now MEASURED, and one of them was prose all along.** `maasdb`
came from F6's text and had never been verified; `sudo grep -E "^database_name|^database_user|..."`
on `regiond.conf` confirms `maasdb` / `maas` / `localhost`. Only non-secret field names
were grepped -- `database_pass` was never read, and the dump path chosen above means it
never needed to be.

The dump was STREAMED through `awk` and nothing was persisted: the MAAS DB carries API
keys and password hashes, so writing it to disk would have created a new exposure needing
its own SEC row.

**Scope, stated so it is not overclaimed:** this proves the pages are readable and the
rows decode. It does NOT prove MAAS's application-level invariants hold. Nothing observed
suggests otherwise and no such claim is made.

**The agent's false-green finding, confirmed and worth keeping:** `SSH_EXIT` was 0 while
`pg_dump` exited 1 having produced 0 bytes -- so F6's own proposed form,
`ssh host 'pg_dump maasdb > /dev/null' && echo PASS`, would have printed PASS over a dump
that never happened. Same family as sweep F3 and sweep F4. Asserting on BYTES plus the
completion marker is what makes this a gate rather than a ritual.

- Changed: `docs/audit/maasdb-pgdump-integrity-dc0-20260802.txt` (new capture).
- **Revert:** n/a -- a read-only measurement. Nothing on the VM was altered.

## Item 7 -- sweep F2 DIAGNOSED: reading R1 is true, R2 is refuted

Sweep F2 left two readings open and forbade assuming the first. A read-only agent settled
it. Capture: `docs/audit/mirror-exitcode-diagnosis-dc0-20260802.txt`.

**VERDICT: R1 -- debmirror itself prints `All done.` and then exits non-zero. Nothing runs
after `All done.` that could fail.** R2 (something failing after the message) has no
candidate step.

**Decisive evidence -- the journal shows `All done.` TWICE, once per debmirror leg, and
the failing one is followed 10 ms later by its own error block in the SAME PID:**

```
08:28:00.830168 dc0-mirror-sync[276574]: All done.
08:28:00.840132 dc0-mirror-sync[276574]: Errors:
08:28:00.840132 dc0-mirror-sync[276574]:  Download of dists/jammy-backports/main/dep11/Components-amd64.yml.gz failed: 500 read timeout
08:28:00.840132 dc0-mirror-sync[276574]: Failed to download files (1 errors)!
```

**Confirmed at VENDOR SOURCE, and re-verified independently by this session** (not taken
on the agent's word, since it is the load-bearing claim) -- `debmirror 1:2.39ubuntu2` on
the dc0 rack:

```
1615:say("All done.");
1620:  exit 1 if (!$ignore_small_errors);
```

The helper does not pass `--ignore-small-errors`, so any non-zero error tally exits 1
AFTER the success-looking message. Corroborating: between the LAST `All done.` (the UCA
leg, 08:28:03.833394) and systemd's failure line (08:28:03.840126) there are 6.7 ms and
ZERO log lines. And the installed helper is byte-identical to the repo's generated
content -- no drift.

**Proximate cause of the 1:** exactly one upstream download,
`jammy-backports/main/dep11/Components-amd64.yml.gz`, hit a 500 read timeout at 300s.

**THE SWEEP'S OWN PREMISE WAS WRONG, AND THE ERROR IS INSTRUCTIVE.** F2 states "its own
log says the sync succeeded". Every line it quoted (`Files to download: 0 B`,
`Downloaded 15 kiB in 3s`, `Everything OK...`, `All done.`) is from the SECOND debmirror,
PID 277623 -- the UCA leg, which SUCCEEDED. The failing ubuntu leg reported
`Downloaded 2867 kiB in 313s` and an error block. **The tail of the passing leg was read
as the tail of the whole run, and the evidence for R1 sat three lines below the last line
quoted.** Same family as this project's standing instrument lessons: a truncated read
that looks exactly like a complete one.

**A SECOND CORRECTION: "`dc-mirror.sh check dc0` stays RED" OVERSTATES IT -- the gate is
FLAKY, not stuck.** The agent reported "7 of 14"; that did not reconcile against the
journal, so it was re-counted. **MEASURED, full journal window 2026-07-23..08-02: 16 runs,
7 `Finished`, 9 `Failed with result 'exit-code'`.** The success count was right, the
denominator was not. Its "four successes in the last four days" IS correct.

**My own first re-count was ALSO wrong, and it is logged rather than quietly dropped:** a
`grep -cE "Starting .*mirror sync"` returned 0 because the unit's description reads
`dc0 mirror upstream sync`, so "mirror sync" never matched. It looked like "no runs
recorded". Checking the instrument before believing a surprising negative is the standing
rule and it paid again.

**The stronger evidence is the SHAPE, not the ratio.** On FOUR separate days a failed run
was followed within hours by a clean one:

```
2026-07-25T06:20:23 Failed  ->  2026-07-25T06:23:59 Finished
2026-07-27T00:54:54 Failed  ->  2026-07-27T08:43:46 Finished
2026-07-31T00:54:12 Failed  ->  2026-07-31T02:35:23 Finished
2026-07-31T18:29:37 Failed  ->  2026-07-31T18:56:20 Finished
```

That is transience DEMONSTRATED, not inferred from an error string. Two caveats kept: the
earliest failure (2026-07-23) was the HOME-under-systemd defect fixed the same hour, so
the lifetime ratio understates current health; and the two most recent runs, both
2026-08-02, BOTH failed -- they bracket the edge outage, and the last success was
2026-08-01T00:04:19.

**Failure shapes are TWO, flagged so they are not later conflated:** 4 runs carry a
`Failed to download files (N errors)!` tally (the R1 path, N = 1/3/22/1); 3 carry
`Errors:` with no tally, meaning debmirror DIED rather than completing -- the `ubuntu=255`
shape `dc-mirror.sh`'s own comment records for 2026-07-27.

**NO REMEDY APPLIED, and the decision is the operator's (hard rule 1).** The agent
recorded only the SHAPE of the choice, which is the part that matters: debmirror's exit
code conflates "nothing mirrored" with "mirrored minus N transient files", so ANY
tolerance change alters what the gate ATTESTS. Relaxing it is the 2026-07-27 false-green
defect inverted. Options are put to the operator separately.

**Declared gaps, not silently filled:** whether the missing `.gz` actually breaks a node's
`apt-get update` (the `.xz` sibling is present; apt was NOT run from a node); whether
tree-wide completeness holds (the tree was not reconciled against the Release indices, and
the agent's `find` was truncated -- so "at least one file missing", never "exactly one");
and whether the 500 timeouts are archive.ubuntu.com or the edge path (a `curl -I` 200
shows reachability only -- HEAD never exercises the body read that timed out, so it is
NOT evidence of transience).

- Changed: `docs/audit/mirror-exitcode-diagnosis-dc0-20260802.txt` (new capture).
  `scripts/dc-mirror.sh` deliberately UNTOUCHED.
- **Revert:** n/a -- diagnosis only.

## Item 8 -- sweep F2, the re-trigger: the failure is PERSISTENT, and it does not matter

**OPERATOR RULING (operational, not a D-number per GA-R3 -- no architectural consequence).**
Question put: what should `dc-mirror.sh`'s gate attest, given debmirror's exit code cannot
distinguish "nothing mirrored" from "mirrored minus N transient files" from "debmirror
died"? Options offered: record shape+count without relaxing / leave as is / re-trigger
first and decide after / add a bounded tolerance (listed as NOT recommended). **Operator
answer, exact utterance: "Re-trigger the sync first, decide after".** The re-trigger was
therefore run as a MEASUREMENT, not as a remedy, and the attestation question stays OPEN.

**RESULT: THE FAILURE IS NOT TRANSIENT.** The re-run (10:26:05 -> 10:31:20Z) failed
identically -- `FAIL ... ubuntu=1 uca=0`, `Result=exit-code`, `ExecMainStatus=1` -- on the
SAME file with the SAME error and the SAME 300s timeout, two hours after the previous
attempt:

```
08:27:56  Download of dists/jammy-backports/main/dep11/Components-amd64.yml.gz failed: 500 read timeout (300s)
10:31:16  Download of dists/jammy-backports/main/dep11/Components-amd64.yml.gz failed: 500 read timeout
```

So the earlier "flaky, not stuck" reading -- which the fail-then-succeed pairs supported --
does NOT extend to THIS failure. Two distinct things were being conflated: the unit has a
history of transient failures, AND it now has one persistent one. The ruling's "decide
after" is answered: waiting will not clear it.

**THE SCOPE OF THE GAP IS NOW EXACT, replacing the agent's "at least one file missing".**
debmirror's own `Files to download: 6349 B` equals the size of that single file, so the
entire outstanding delta of the 952 GiB tree IS this one file. On disk:

```
present:  Components-amd64.yml.xz   5748 B   (Release lists 5748 -- size-correct)
ABSENT:   Components-amd64.yml.gz   6349 B   (Release lists 6349)
```

**AND IT DOES NOT BREAK apt -- MEASURED, not reasoned.** The prior agent explicitly
declared this unmeasured and declined to reason it away, which was correct. Tested from the
rack with a fully ISOLATED apt state dir (scratch `Dir::State::Lists`, `Dir::Cache`,
`Dir::Etc::SourceList`; nothing system-wide touched, no package installed):

```
deb http://10.12.8.4/ubuntu jammy-backports main
  Get:1 InRelease [127 kB]
  Get:2 main amd64 Packages [70.2 kB]
  Get:3 main amd64 Components [5748 B]      <-- the .xz, the format that IS present
  Get:4 main amd64 c-n-f Metadata [412 B]
  APT_UPDATE_RC=0
  apt-cache policy -> release v=22.04,o=Ubuntu,a=jammy-backports,c=main,b=amd64
```

**apt asks for the `.xz` and never asks for the `.gz`.** So the mirror is FUNCTIONALLY
complete for the deploy and the gate is red over a file no consumer requests. jammy-backports
was chosen as the test suite deliberately -- it is the one carrying the missing file.

**ONE ANOMALY, UNEXPLAINED AND LEFT THAT WAY rather than hand-waved.** The file fetches
FINE by curl from the same host, right now: `http=200 size=6349 time=0.436s`, with a real
body read (HEAD would prove nothing). Its `.xz` sibling and a jammy control also return 200.
So curl succeeds in 0.44s where debmirror times out after 300s, twice. Ruled OUT: any proxy
(the unit's environment is `HOME=/root` only; no `http_proxy` in `/etc/environment`,
`apt.conf.d`, `profile.d`, or root's login shell). NOT chased further -- it does not gate
the deploy, and guessing at an LWP-vs-curl difference would be exactly the reasoning-instead-
of-measuring this project keeps logging.

- Changed: nothing in the repo. `scripts/dc-mirror.sh` still UNTOUCHED; no tolerance was
  relaxed and no assertion weakened.
- State changed on the rack: one sync run, which wrote `last-sync.status` and fetched
  ~1.3 MB of indices. The apt test wrote only to a scratch dir, removed on exit.
- **Revert:** n/a.

## Item 9 -- sweep F2 ROOT-CAUSED, and item 8's "not transient" is WITHDRAWN

**OPERATOR RULING, exact utterance: "Root-cause the curl/debmirror anomaly first"** -- taken
over accepting the red gate, over recording error shape/count, and over
`--ignore-small-errors`. Capture: `docs/audit/mirror-500-timeout-rootcause-20260802.txt`.

**ROOT CAUSE: UPSTREAM AND BACKEND-SPECIFIC. `archive.ubuntu.com` is round-robin across
NINE A records and ONE of them, `91.189.92.23`, HANGS on this specific object while serving
its directory siblings normally.**

```
THE FAILING OBJECT (.yml.gz)          CONTROL, SAME DIR (.yml.xz)
185.125.190.81  200  6361  0.305s     91.189.92.23  200  5776  0.543s   <-- healthy
...(seven more, all 200)                (all nine 200)
91.189.92.23     -     -   >25s TIMEOUT
```

The control is what makes this a finding rather than "a backend is down": the same backend
serves the sibling in half a second. **The resolver ROTATES** -- six consecutive lookups
returned six different orderings -- so each fetch re-rolls which backend it gets.

**MEASURED RATE: 12 sequential fetches -> 11 x 200, 1 timeout (~8%)**, consistent with
exactly one bad record of nine.

**SO ITEM 8's "NOT TRANSIENT" IS WITHDRAWN (GA-R1 C2 -- measurement wins over a document,
including one written an hour earlier).** The failure is PROBABILISTIC, not deterministic;
two consecutive debmirror failures were not the proof of persistence they looked like.

**A HYPOTHESIS TESTED AND REFUTED, kept because it is the one a reader forms first.**
debmirror sets `keep_alive => 1` and curl does not reuse connections, so a stale persistent
connection looked like the answer. The test INVERTED it -- same library, same callback
style, seconds apart:

```
keep_alive=1 (debmirror's setting)   200 OK            bytes=6361   1s
keep_alive=0 (my "control")          500 read timeout  bytes=0     30s
```

**ALSO CORRECTED: the `500` was never a server 500.** LWP synthesises status 500 for
CLIENT-side failures, so it is LWP reporting its own 300s timeout
(`/usr/bin/debmirror:629`, `our $timeout=300;`). I had read it as an archive-side error,
which points the investigation the wrong way.

**TWO STRUCTURAL FACTS, read from the vendor script rather than assumed** -- both bear on
options that were on the table: there is **NO dep11 exclusion option** (`dep11_from_release`
`:1201` and `get_dep11_files` `:1384` are unconditional), so the mirror's scope cannot be
narrowed to drop AppStream metadata without patching debmirror; and **`--ignore-small-errors`
is NOT a bounded tolerance -- `:2891` `$num_errors = $t if ($ignore_small_errors);` DISCARDS
the count**, so it would read clean on a wholly failed mirror.

**RESIDUAL, DECLARED NOT EXPLAINED:** debmirror failed 2 of 2, which a naive 1-in-9 model
predicts at ~1.2%. Untested candidates: LWP's keep-alive pool may pin a whole run to one
resolved address so one bad draw poisons the run; or more backends may be intermittently bad.
**The per-IP sweep tested each backend ONCE, so it establishes that `91.189.92.23` was bad,
NOT that it is the only bad one.**

**SECOND, INDEPENDENT FINDING -- A DIFFERENT FAILURE MODE IS NOW LATENT.** The local
`Release` expects **6349 B**; every backend now serves **6361 B**. The object was
republished DURING this investigation (an early curl measured 6349, every later one 6361).
**A successful download would now fail HASH verification against the stale local Release** --
a different error from the timeout, and not fixed by fixing the timeout. Whether debmirror
re-fetches `Release` early enough each run to avoid this was NOT measured.

- Changed: `docs/audit/mirror-500-timeout-rootcause-20260802.txt` (new capture).
  `scripts/dc-mirror.sh` STILL untouched.
- **Revert:** n/a -- read-only investigation.

## Item 10 -- D-139 step 6 tool BUILT, adversarially reviewed, FOUR defects fixed

`netbox/d139-step6-vip-rehome.py` + `tests/d139-step6-vip-rehome/run-tests.sh` (20 cases).
Implements the two 2026-08-02 rulings: "Full step 6 first, then deploy" and "Deprecate
both, delete nothing". CREATE 26 GUA VIP addresses -> read-back verify -> deprecate 26 ULA
addresses + 9 ULA prefixes. **No delete path anywhere**, asserted against the artifact.

**Dry run (live apex): `CREATE 26 | ALREADY 0 | DEPRECATE-ADDR 26 | DEPRECATE-PFX 9`.**
**The 26 CREATE targets diff EXACTLY against the 26 GUA VIP legs in
`overlays/vr1-dc0-vips.yaml`** -- every address written to the apex is one the deploy
configures, and vice versa. Nothing has been written; `--commit` has not been run.

**AN ADVERSARIAL REVIEW RETURNED "FIX FIRST" AND IT WAS RIGHT ON ALL FOUR COUNTS.**
Capture: `docs/audit/d139-step6-tool-review-20260802.txt`. The mapping logic was correct;
what was missing was PRECONDITIONS and COVERAGE.

- **DEF-1 (CRITICAL) -- `--dc vr1-dc1` would ORPHAN-CREATE.** `gua_net_for_role()`
  computed targets arithmetically and never asked the apex whether they EXIST. MEASURED:
  dc1's GUA carve is incomplete -- four rows under `2602:f3e2:f03::/48`, all
  provider-public, no `:20::/64`, no `:21::/64` -- so dc1 planned 26 creates into
  non-existent prefixes then deprecated dc1's only authoritative rows, at rc=0 with no
  warning. **dc0 hid it because all sixteen of its targets happen to exist.** Reachable
  via the other valid value of a required flag. FIXED: the target must exist in the apex
  or the tool REFUSES. Verified live -- dc1 now refuses, dc0 unchanged at 26/26/9.
- **DEF-2 (CRITICAL) -- the apex-IDENTITY guard was gone.** `d139-gua-carve.py:159-163`
  refuses any host that is not the working apex (DOCFIX-195), but that guard lives in its
  `main()`, and importing a module never runs its `main()`. **Subclassing `C.NB` inherited
  the TRANSPORT and left the SAFETY POSTURE behind** -- pointing this at
  `netbox.baldurkeep.com` (the v1 reference) would have connected fine and written to the
  wrong NetBox. FIXED: identity checked before any network call.
- **DEF-3 (HIGH) -- silent under-count.** Removing one ULA `/64` row yielded
  `CREATE=13 / DEPA=13 / DEPP=8` at exit 0 -- 13 live VIPs neither created nor deprecated,
  reported as success. Same class as DEFECT 2 of the 2026-08-01 CORRECTION NOTE. FIXED:
  any ULA address claimed by no prefix row REFUSES. **My first fix was itself wrong and
  running it caught that** -- it scanned the whole retired `/48` and flagged dc1's 26 VIPs
  while planning dc0. The retired `/48` is SHARED (dc0 `:22x`, dc1 `:32x`), so the check
  now excludes addresses another DC's row claims, and deliberately does NOT count a `/60`
  parent as coverage -- the reviewer's scenario was exactly a missing `/64` whose `/60`
  still existed.
- **DEF-4 (HIGH) -- `main()` had ZERO coverage.** The reviewer hoisted the deprecate loops
  above the create phase and the suite reported ALL PASS. The CREATE-before-DEPRECATE
  ordering both rulings call load-bearing was untested. FIXED: T16-T18 drive `main()`
  through a fake client that RECORDS CALL ORDER. **Proven by re-running the reviewer's
  exact mutation on a copy: T18 goes RED.**

**TWO OF MY ASSERTIONS COULD NOT FAIL, and the review killed both.** **T13** asserted the
ABSENCE of a string, so a traceback satisfied it -- it passed against a tool file that did
not parse. Now requires a positive, well-formed, DIFFERENT target, and **T15** separately
asserts the tool parses, closing the escape hatch. **T14** grepped ONE file, so adding a
delete to the IMPORTED `d139-gua-carve.py` left it green; it now covers both.

**Harness 14 -> 20 cases, count delta checked** (the rule landed in item 3 today, applied
to my own delivery). Gauntlet manifest recorded deliberately, 97 -> 98.

**Corrected in the ruling record (GA-R1 C2):** the amendment said the GUA records would be
`status=active`. MEASURED: the live ULA VIP records are `reserved`, and
`dc-plane-apex-import.py:186,200` creates addresses `reserved`. Both
`docs/design-decisions.md` and `docs/CURRENT-STATE.md` corrected.

**Also corrected -- my own overclaim:** the docstring said "every action is reversible by
flipping a status back". True of the 26+9 DEPRECATIONS; NOT true of the 26 CREATES, which
would need a delete this repo does not have. Reversal is not wanted (the creates are the
records the deploy matches) but the claim was wrong and is now stated precisely.

**OPEN SCOPE QUESTION, MEASURED, NOT A TOOL DEFECT -- THE MAAS HALF OF STEP 6.** D-139's
list says retire the ULA rows "in the apex **and in MAAS**". This tool is apex-only, so
step 6 is NOT complete when it finishes. MEASURED on the dc0 region: MAAS holds BOTH
families -- five ULA `/64`s (`:220`, `:221`, `:230`, `:240`, `:250`) beside six GUA. Four
are EMPTY; **`fd50:840e:74e2:220::/64` (metal-admin) still holds 2 allocated entries**, so
this cannot be a blind cleanup. MAAS has no `deprecated` status for a subnet -- the choice
is delete or leave -- which makes it a separate operator decision, not an extension of the
"deprecate" ruling.

- Changed: `netbox/d139-step6-vip-rehome.py` (new), `tests/d139-step6-vip-rehome/run-tests.sh`
  (new, 20 cases), `tests/HARNESS-MANIFEST` (+1, recorded deliberately),
  `docs/audit/d139-step6-tool-review-20260802.txt` (new), plus the two doc corrections.
- **Revert:** delete the tool, its harness dir and the manifest line; re-run
  `bash scripts/run-tests-all.sh --record-manifest`. Nothing was written to the apex.

## Item 11 -- D-139 STEP 6 EXECUTED: apex applied, and 4 of 5 MAAS ULA subnets removed

**OPERATOR APPROVAL, exact utterance: "Queue up the MAAS half to run after you complete
netbox. I approve the MAAS deletes to clean up the data and config. Go ahead with both
now".**

**THE APEX HALF IS APPLIED, exit 0.** `d139-step6-vip-rehome.py --dc vr1-dc0 --commit`:
`CREATE 26 | ALREADY 0 | DEPRECATE-ADDR 26 | DEPRECATE-PFX 9`, both read-backs OK.
INDEPENDENTLY VERIFIED after the fact, not on the tool's own word: a re-run reports
`CREATE 0 | ALREADY 26 | DEPRECATE-ADDR 0 | DEPRECATE-PFX 0` -- fully idempotent and
converged.

**RECORDED SO A LATER READER IS NOT MISLED:** `d139-gua-carve.py` STILL reports
`dependent objects ... 26 ip-address(es)`. That is CORRECT and expected -- the 26 ULA
records still EXIST, deprecated rather than deleted, per the ruling. The carve tool counts
existence, not status. It is not evidence that step 6 did not run.

**THE MAAS HALF: FOUR SUBNETS DELETED, ONE DELIBERATELY HELD.**

**VERIFY BEFORE MUTATE FOUND A REAL HAZARD, and it is why the fifth was not deleted.** The
approval was given before anyone knew WHAT the `fd50:840e:74e2:220::/64` allocations were.
Measured:

```
fd50:840e:74e2:220::5   alloc_type=1 (STICKY)  user=juju-vr1-dc0   <- the Juju controller
fd50:840e:74e2:220::6   alloc_type=1 (STICKY)  user=MAAS           <- the MAAS region VM
```

**And they have NO GUA counterpart.** The GUA metal-admin subnet holds exactly nine
entries -- `::100,::101,::102` control, `::120,::121` compute, `::150-::153` storage -- the
nine tagged ROLE NODES and nothing else. This matches the record that `dc-node-v6-carve.py`
walks only the nine tagged role nodes and that the controller's own v6 was "restored
separately" -- restored on ULA, never migrated. **So deleting that subnet would remove the
only recorded v6 address of the deploy client AND of the MAAS region the delete is issued
to.** That is not cleanup, and it is outside what the approval could have meant. HELD,
pending its own decision.

**THE FOUR EMPTY ONES WERE SAFE, and that was established twice over.** Zero allocations
AND **zero machine-interface links** for all five ULA subnets -- the stronger check, since
"no allocated IP" does not by itself mean "nothing references it". Zero links also
CONFIRMS D-139 step 3 succeeded: the nine role nodes are fully on GUA.

Deleted INDIVIDUALLY, never looped -- hard rule 3 forbids batching destructive steps, and
this repo has a logged incident from exactly that (2026-07-30, nine deletes in a loop; the
guard stopped it). Each got a fresh cidr + allocation re-check immediately before its own
delete:

```
subnet  9  fd50:840e:74e2:221::/64  metal-internal  allocations=0  deleted rc=0
subnet  8  fd50:840e:74e2:230::/64  data-tenant     allocations=0  deleted rc=0
subnet 12  fd50:840e:74e2:240::/64  storage         allocations=0  deleted rc=0
subnet 11  fd50:840e:74e2:250::/64  replication     allocations=0  deleted rc=0
```

**POST-STATE, MEASURED:** six GUA `/64`s remain (`f02:10/20/21/30/40/50`), one ULA remains
(`fd50:840e:74e2:220::/64`, id=2, held). Machines unchanged and healthy: **9 Ready + 1
Deployed** (the controller), total 10.

**STILL OPEN, needs a decision:** the held subnet. Options are to migrate the controller's
and region's v6 to GUA first and then delete, or to delete and accept losing those two
records, or to leave the subnet in place indefinitely. NOT decided here.

**NO REPO TOOL EXISTS for the MAAS half** -- these were hand-issued `maas subnet delete`
calls. If the dc1 rebuild is to be reproducible, this belongs in a script with the same
emptiness + link preconditions asserted, and that is registered as owed rather than built.

- Changed: live apex (26 created, 35 deprecated) and the dc0 MAAS region (4 subnets
  deleted). Repo: this changelog + CURRENT-STATE.
- **Revert:** apex -- flip the 35 statuses back (the 26 creates are additive and wanted).
  MAAS -- the four subnets can be recreated (`dc-region-topology.sh` carries the shape),
  but their identities are gone; they held nothing, so nothing else is lost.

## Item 12 -- the MAAS half gets a tool, and it immediately caught a defect in MY OWN check

**OPERATOR DIRECTION: "Proceed with 1 and 2"** -- 1 = the held ULA subnet, 2 = build the
repo tool for the MAAS half. Item 2 is delivered here. **Item 1 is NOT executed** and is
explained below.

**`scripts/dc-plane-ipam.sh` gains `retire-v6-ula <site> [--commit]`.** It belongs there
rather than in a new script: `dc-region-topology.sh`'s own header assigns v6 plane subnets
to `dc-plane-ipam.sh`, and a one-off script would have split that ownership. Harness
**25 -> 32 cases** (R1-R7), delta checked.

The mode: enumerate this site's subnets inside the retired ULA `/48`; for each, REFUSE if
it holds an allocation OR a machine-interface link, NAMING the holder; otherwise delete --
one at a time, never looped -- and READ BACK that the row is gone, because a delete
returning success is not evidence the row went away (R6 proves that assertion has teeth).

**>>> THE TOOL'S FIRST LIVE RUN CAUGHT A DEFECT IN THE BY-HAND CHECK I USED AN HOUR
EARLIER, AND IT IS THE MORE IMPORTANT FINDING HERE. <<<** Its dry run reported:

```
interface links: subtle-grouse:enp1s0
```

My by-hand link check (item 11) reported **ZERO links for all five subnets**. The cause is
a quoting bug in my own one-liner: the subnet id was interpolated as `sid=\"'$id'\"`, so
the comparison ran `str(id) == "'2'"` and **could never match**. It was a checker that
could not fail -- written by me, in the same session in which I landed exactly that rule
into `script-authoring.md`. The tool found a real link the moment it ran.

**CONSEQUENCE, STATED PLAINLY: item 11's claim that the four deletes were "proven safe
twice" was HALF FALSE.** The allocation count was sound; the link half was inert, so link
status was in fact UNKNOWN at the time of those four deletions.

**THE OUTCOME WAS NEVERTHELESS SAFE, and that is measured, not assumed.** After the
deletes, all nine role nodes hold **v4=6, v6=6** -- six planes in each family -- so every
v6 link they carry is GUA and the four deleted ULA subnets genuinely held none.
`enp2s0` reads unlinked on all nine, which is EXPECTED (provider-public carries no L3 link;
the static lands on `br-ex`, D-133/D-100), not damage. **Right outcome, unsound
verification** -- the two are recorded separately on purpose.

`subtle-grouse` (the juju controller, `Deployed`) reads **v4=2 v6=1**: two planes per the
role-dependent carve, and its single v6 link is the ULA `::5`. That INDEPENDENTLY confirms
holding subnet 2 was correct.

**ITEM 1 IS NOT EXECUTED, and the tool is why.** Run against the live region,
`retire-v6-ula vr1-dc0` REFUSES on `fd50:840e:74e2:220::/64` and names all three holders:
`::5` (`juju-vr1-dc0`), `::6` (`MAAS`), and the interface link `subtle-grouse:enp1s0`.
Deleting it still requires re-homing the juju controller's and the MAAS region's v6 onto
GUA first -- which re-addresses the DEPLOY CLIENT immediately before the deploy, and this
project has already lost three bootstrap attempts to under-carved controller networking.
That is a mutation with its own risk profile and is NOT covered by "I approve the MAAS
deletes to clean up the data and config", which was given about four empty subnets.
**It is also NOT deploy-blocking:** an extra ULA subnet in MAAS harms nothing.

- Changed: `scripts/dc-plane-ipam.sh` (new mode), `tests/dc-plane-ipam/run-tests.sh`
  (R1-R7).
- **Revert:** drop the `retire-v6-ula` block, its usage lines and the `ACTION` case entry;
  drop R1-R7 and the two retire helpers.
