Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260802-queued-items.md
@JANeumatrix JANeumatrix 1 day ago 39 KB retire-v6-ula harness + the defect it found in my own by-hand check
Raw Blame History

Changelog -- 2026-08-02 (session 2): clearing the queued-findings backlog

Session scope: work the queued items left by the 2026-08-02 close bookend (docs/audit/queued-findings-20260802-stage5-edge-fold.txt), in consequence order. Stage 5 remains OPEN; nothing here opens or closes a stage.

Numbering note, because two registers collide. This session's items cite the SWEEP register (F1-F6, the queued-findings file above). The runbook fold register (docs/runbook-fold-register.md) has its OWN F1-F12 with different meanings -- sweep F5 is "share the probe definition", fold F5 is "build the OOB plane". Every reference below is prefixed sweep F<n> or fold F<n>. Do not drop the prefix.

Item 1 -- DOCFIX-207: preflight P6 quoted a plan count stale since 2026-07-10

Sweep S5(a). scripts/preflight.sh:369 printed the phase-01 Step 1.2 reminder as plan: 50 apps / 97 relations. The real figure is 56 apps / 108 relations, MEASURED twice and recorded: docs/CURRENT-STATE.md:955 (juju deploy --dry-run resolved all 56 charms, planned 108 relations, exit 0) and again 2026-07-31 against the fixed deploy input. Measurement wins over a document (GA-R1 C2).

The staleness is older than it looks: docs/archive/changelogs/changelog-20260710-phase01-bundle-staleness-fix.md:18 corrected the SAME figures in phase-01's own Step 1.2 GATE on 2026-07-10 and missed this copy in preflight. So for three weeks the executable gate and the reminder pointing at it disagreed by 6 apps and 11 relations.

Scope, stated so it is not overclaimed: this is a REMINDER string inside the P6: stage-2 reminders (NOT run here) block, not an assertion. Nothing gated on it and no harness asserts it (grep '50 apps' tests/ -> one unrelated hit, a line-number probe for the P6 header). The defect is that a session comparing a live dry-run against it would read a correct plan as wrong.

  • Changed: scripts/preflight.sh -- the figure, plus a comment carrying the measurement provenance and the explicit note that the real gate is the dry-run itself.
  • Verified: bash tests/preflight/run-tests.sh -> 43/43 PASS; repo-lint 0 fail.
  • Revert: restore 50 apps / 97 relations on that echo line and drop the comment block above it.

Item 2 -- sweep F3 graduated: systemctl show is not an existence check

The sweep recorded that systemctl show on a NON-EXISTENT unit returns Result=success / ExecMainStatus=0 / RemainAfterExit=no and no journal entries -- indistinguishable from a healthy oneshot that has completed. Measured 2026-08-02 against the invented name dc0-rack-net; the real units are dc0-rack-legs and dc0-node-dns (scripts/dc-rack-net.sh:16,22). It produced a false "inactive" reading that was reported before being caught.

It lived only in the sweep capture, which is an audit artifact, not a place a future session looks before running a systemd check. Graduated to its durable home as platform-traps section 5c, with the verbatim output, the reason (show answers from property DEFAULTS), and the three instruments that DO error on absence (systemctl cat, list-unit-files, or asserting UnitFileState non-empty -- the only field that betrays the absence).

  • Changed: .claude/skills/openstack-cloud-ops/references/platform-traps.md (new 5c).
  • Revert: delete the 5c block.

Item 3 -- sweep F4 + F5 graduated: two harness-discipline rules

Both lived only in the sweep capture. Landed in .claude/skills/openstack-cloud-ops/references/script-authoring.md, in the Testing section immediately before the repo-record traps.

Sweep F4 -- assert the CASE COUNT moved. Two new cases in tests/dc-egress-check were written with fok/fbad, helper names from tests/dc-node-v6-carve. The suite printed fbad: command not found to stderr, kept its count at 14, and reported ALL PASS. The mutations those cases targeted then "survived", reading as robustness when it was really the tests not existing. The compounding fact is the one worth carrying: the symptom of a non-existent test and of a passing test are the same symptom, so the delivery check has to be arithmetic -- count before, count after, delta equals cases written. Ties into the standing rule that a checker which cannot fail is not a gate: prove a new assertion can FAIL and prove it RAN.

Sweep F5 -- two scripts probing one endpoint must share the probe definition. dc-egress-check.sh's A4 and dc-snap-proxy.sh:215 both hit api.snapcraft.io; only the latter sent Snap-Device-Series: 16, which the store requires. Measured 400 without it both through the proxy and direct, 200 with it. The newer gate called a healthy proxy broken on its first live run while the older script returned PASS with a real payload -- two gates disagreeing about one piece of infrastructure. The generalisable half, which was NOT on surface: a fixture that mocks curl cannot catch a wrong REAL request, so the harness stays green in both worlds. A duplicated request line is a duplicated contract, and only one copy gets fixed when the endpoint tightens.

  • Changed: .claude/skills/openstack-cloud-ops/references/script-authoring.md (two paragraphs).
  • Revert: delete the two paragraphs.

Item 4 -- sweep F1 repo-side provenance MEASURED (no mutation yet)

Before propagating overlays/vr1-dc0-vips.yaml to the dc0 rack's ~/repo-stage (the D-138 deploy client), the repo copy's correctness was measured rather than inherited from the close bookend:

  • It conforms to D-139 ruling B. grep -o 'fd50:[0-9a-f:]*' -> zero ULA legs. 39 GUA legs, 2602:f3e2:f02:11 x13 (provider-public), f02:20 x13, f02:21 x13. That is exactly the 26-leg migration the 2026-08-01 conformance audit named as its blocker C (docs/audit/d139-conformance-dc0-20260801.txt:44-56), so blocker C is CLOSED in the repo copy.
  • It has renderer provenance. bash tests/render-drift/run-tests.sh -> 4/4 PASS, naming the file: render/values/vr1-dc0-vips.yaml renders overlays/vr1-dc0-vips.yaml byte-for-byte, and the suite carries a PROOF-OF-TEETH case (a mutated values file is detected). The values file declares output: overlays/vr1-dc0-vips.yaml at line 2.

  • Repo digest confirmed 80d861560a6b3c52, matching what the close recorded.

  • Changed: nothing. Read-only measurement.

  • Revert: n/a.

Item 5 -- sweep F1 CLOSED: the dc0 rack's deploy input is re-staged

A read-only agent enumerated the WHOLE staging directory first, rather than diffing the one file the close bookend named. Capture: docs/audit/repo-stage-drift-dc0-20260802.txt (427 lines). Classification of the 14 regular files + 4 dirs under ~/repo-stage (.git re-confirmed absent):

  • (A) diverged: 1 -- overlays/vr1-dc0-vips.yaml only. Corroboration the bookend did not have: voffice1's clone also holds 80d86156..., so the rack was the ONLY host still carrying the pre-GUA overlay.
  • (B) missing from the Step-4 deploy closure: 0. This was the open question, and it matters because a prior deploy died on a missing policies/overrides.zip. That file IS staged, at the repo digest 02fe1fd7...e720a011, and resolves because policies/ sits beside bundle.yaml. All three scannable merge inputs were checked, not just the bundle: bundle.yaml has exactly one local reference (:227-228); neither the VIP nor the machines overlay has a resources: block or any local path.
  • (C) rack-only orphans: 0. All 14 staged files resolve to a repo path.
  • Two later-step overlays are unstaged (dc-dc-ipv6-family-matrix.yaml, dc-ha-scaleup.yaml); not Step-4 inputs, recorded as a fork, not ruled.

THE MUTATION (gated, single file, executed 2026-08-02). scp -p overlays/vr1-dc0-vips.yaml vr1-dc0-rack:/home/jessea123/repo-stage/overlays/vr1-dc0-vips.yaml

A single-file copy was the minimal correct action BECAUSE the enumeration proved exactly one file diverged. A wholesale directory refresh was rejected for the same reason the 2026-07-31 re-stage rejected it: it risks clobbering or re-permissioning the 0600 vr1-dc0-octavia-pki.yaml (SEC-029).

Before / after, measured by this session on both ends:

                                    BEFORE            AFTER             REPO
overlays/vr1-dc0-vips.yaml          ed19d989e80da8da  80d861560a6b3c52  80d861560a6b3c52
bundle.yaml                         4c8a78527a3100f6  (untouched)       4c8a78527a3100f6
overlays/vr1-dc0-machines.yaml      e3be85e41b419486  (untouched)       e3be85e41b419486
overlays/vr1-dc0-octavia-pki.yaml   5fc117f188332042  5fc117f188332042  (gitignored)

POST-COPY, ALL FOURTEEN STAGED FILES WERE RE-VERIFIED AGAINST CURRENT HEAD, not just the four in the table -- because the staged scripts/ copies were last checked at an EARLIER commit and the 2026-08-02 session shipped script changes after that check. All 13 tracked files match the repo digest-for-digest (bundle.yaml, both non-PKI overlays, policies/domain-manager-policy.yaml, policies/overrides.zip, and the eight staged scripts/), and the 14th matches voffice1. The staging directory is now fully in sync, confirmed against HEAD rather than inherited from the prior verification.

Checked and NOT a gap: dc-egress-check.sh is absent from ~/repo-stage by design -- it is streamed over stdin (ssh <rack> 'bash -s -- check <site>' < scripts/dc-egress-check.sh, per its own refuse message at :90-91), so it is never staged and cannot go stale there.

The PKI overlay is verified UNCHANGED -- same digest, same 0600, same mtime (Jul 31 02:38) -- and was never read, only hashed. It measures IDENTICAL to voffice1's copy, so the SEC-029 superseded-cert hazard is not realized today; nothing enforces the dual update on the next reissue, which stays an open SEC-029 obligation.

  • Revert: copy the pre-GUA overlay back. It is NOT retained anywhere as a file -- reverting means re-rendering from render/values/ at the parent of the GUA commit. There is no reason to: the staged copy now matches repo HEAD, which is the invariant.

TWO FINDINGS THE AGENT SURFACED THAT WERE NOT ASKED FOR, both logged not fixed:

  1. The staged inventory is 14 files, not the nine that docs/security-ledger.md SEC-029(3) records. The +5 are opnsense-* files left by the 2026-08-02 edge rebuild (digests match dc0-edge-rebuild-20260802.txt:259-264). A future re-stage planned off the "nine-file copy" figure would omit five files. The ledger's prose is stale, not wrong-in-kind.
  2. A GATE/DEPLOY ASYMMETRY, DOCFIX-shaped and worth attention before Step 4. overlays/vr1-dc0-machines.yaml exists and scripts/preflight.sh:144-145 folds it into the merged input that P2 validates -- but runbooks/dc-dc-phase4-juju-bundle-per-dc.md:527-528 asserts the file does NOT exist and Step 4 does not pass it. So P2 validates a different artifact than the deploy consumes. That is the class of defect this project has repeatedly found (a gate grading the workspace while the overlay is what reaches the charm). NOT fixed here -- hard rule 1; it likely bundles with the outstanding D-138 run-location DOCFIX.

Item 6 -- sweep F6 CLOSED: the dc0 MAAS region DB is proven uncorrupted

PASS. pg_dump read every page of every table in maasdb: 23,878,796 bytes / 37,199 lines / exit 0 / empty stderr, with -- PostgreSQL database dump complete asserted in a separate pass. A torn or bad-checksum page anywhere would have aborted the dump, so the 2026-08-01 hard power cut left NO page-level corruption. Evidence appended to docs/audit/maasdb-pgdump-integrity-dc0-20260802.txt (the agent's BLOCKED report is left verbatim above it; sections 7-11 are the resolution).

F6's stated blocker was the wrong diagnosis, and this is the durable part. The finding attributed five prior failures to snap confinement and prescribed a login shell. Neither was operative:

  • Confinement did NOT reproduce as ubuntu -- the postgresql.dump wrapper ran, opened the socket, and the server answered. (It DOES reproduce as _daemon_: cannot create snap home dir: mkdir /nonexistent + is not a snap cgroup. So the original reading was true for the user it was measured on and got generalised too far.)
  • The real discriminators were ROLE and TRANSPORT. Role ubuntu does not exist; the role is maas. And over TCP (-h localhost) role maas is password-challenged (fe_sendauth: no password supplied), which would have meant handling the regiond.conf secret -- while over the UNIX SOCKET (-h /tmp) it authenticates with no credential at all. Dump over the socket; never over TCP.

Both identity values are now MEASURED, and one of them was prose all along. maasdb came from F6's text and had never been verified; sudo grep -E "^database_name|^database_user|..." on regiond.conf confirms maasdb / maas / localhost. Only non-secret field names were grepped -- database_pass was never read, and the dump path chosen above means it never needed to be.

The dump was STREAMED through awk and nothing was persisted: the MAAS DB carries API keys and password hashes, so writing it to disk would have created a new exposure needing its own SEC row.

Scope, stated so it is not overclaimed: this proves the pages are readable and the rows decode. It does NOT prove MAAS's application-level invariants hold. Nothing observed suggests otherwise and no such claim is made.

The agent's false-green finding, confirmed and worth keeping: SSH_EXIT was 0 while pg_dump exited 1 having produced 0 bytes -- so F6's own proposed form, ssh host 'pg_dump maasdb > /dev/null' && echo PASS, would have printed PASS over a dump that never happened. Same family as sweep F3 and sweep F4. Asserting on BYTES plus the completion marker is what makes this a gate rather than a ritual.

  • Changed: docs/audit/maasdb-pgdump-integrity-dc0-20260802.txt (new capture).
  • Revert: n/a -- a read-only measurement. Nothing on the VM was altered.

Item 7 -- sweep F2 DIAGNOSED: reading R1 is true, R2 is refuted

Sweep F2 left two readings open and forbade assuming the first. A read-only agent settled it. Capture: docs/audit/mirror-exitcode-diagnosis-dc0-20260802.txt.

VERDICT: R1 -- debmirror itself prints All done. and then exits non-zero. Nothing runs after All done. that could fail. R2 (something failing after the message) has no candidate step.

Decisive evidence -- the journal shows All done. TWICE, once per debmirror leg, and the failing one is followed 10 ms later by its own error block in the SAME PID:

08:28:00.830168 dc0-mirror-sync[276574]: All done.
08:28:00.840132 dc0-mirror-sync[276574]: Errors:
08:28:00.840132 dc0-mirror-sync[276574]:  Download of dists/jammy-backports/main/dep11/Components-amd64.yml.gz failed: 500 read timeout
08:28:00.840132 dc0-mirror-sync[276574]: Failed to download files (1 errors)!

Confirmed at VENDOR SOURCE, and re-verified independently by this session (not taken on the agent's word, since it is the load-bearing claim) -- debmirror 1:2.39ubuntu2 on the dc0 rack:

1615:say("All done.");
1620:  exit 1 if (!$ignore_small_errors);

The helper does not pass --ignore-small-errors, so any non-zero error tally exits 1 AFTER the success-looking message. Corroborating: between the LAST All done. (the UCA leg, 08:28:03.833394) and systemd's failure line (08:28:03.840126) there are 6.7 ms and ZERO log lines. And the installed helper is byte-identical to the repo's generated content -- no drift.

Proximate cause of the 1: exactly one upstream download, jammy-backports/main/dep11/Components-amd64.yml.gz, hit a 500 read timeout at 300s.

THE SWEEP'S OWN PREMISE WAS WRONG, AND THE ERROR IS INSTRUCTIVE. F2 states "its own log says the sync succeeded". Every line it quoted (Files to download: 0 B, Downloaded 15 kiB in 3s, Everything OK..., All done.) is from the SECOND debmirror, PID 277623 -- the UCA leg, which SUCCEEDED. The failing ubuntu leg reported Downloaded 2867 kiB in 313s and an error block. The tail of the passing leg was read as the tail of the whole run, and the evidence for R1 sat three lines below the last line quoted. Same family as this project's standing instrument lessons: a truncated read that looks exactly like a complete one.

A SECOND CORRECTION: "dc-mirror.sh check dc0 stays RED" OVERSTATES IT -- the gate is FLAKY, not stuck. The agent reported "7 of 14"; that did not reconcile against the journal, so it was re-counted. MEASURED, full journal window 2026-07-23..08-02: 16 runs, 7 Finished, 9 Failed with result 'exit-code'. The success count was right, the denominator was not. Its "four successes in the last four days" IS correct.

My own first re-count was ALSO wrong, and it is logged rather than quietly dropped: a grep -cE "Starting .*mirror sync" returned 0 because the unit's description reads dc0 mirror upstream sync, so "mirror sync" never matched. It looked like "no runs recorded". Checking the instrument before believing a surprising negative is the standing rule and it paid again.

The stronger evidence is the SHAPE, not the ratio. On FOUR separate days a failed run was followed within hours by a clean one:

2026-07-25T06:20:23 Failed  ->  2026-07-25T06:23:59 Finished
2026-07-27T00:54:54 Failed  ->  2026-07-27T08:43:46 Finished
2026-07-31T00:54:12 Failed  ->  2026-07-31T02:35:23 Finished
2026-07-31T18:29:37 Failed  ->  2026-07-31T18:56:20 Finished

That is transience DEMONSTRATED, not inferred from an error string. Two caveats kept: the earliest failure (2026-07-23) was the HOME-under-systemd defect fixed the same hour, so the lifetime ratio understates current health; and the two most recent runs, both 2026-08-02, BOTH failed -- they bracket the edge outage, and the last success was 2026-08-01T00:04:19.

Failure shapes are TWO, flagged so they are not later conflated: 4 runs carry a Failed to download files (N errors)! tally (the R1 path, N = 1/3/22/1); 3 carry Errors: with no tally, meaning debmirror DIED rather than completing -- the ubuntu=255 shape dc-mirror.sh's own comment records for 2026-07-27.

NO REMEDY APPLIED, and the decision is the operator's (hard rule 1). The agent recorded only the SHAPE of the choice, which is the part that matters: debmirror's exit code conflates "nothing mirrored" with "mirrored minus N transient files", so ANY tolerance change alters what the gate ATTESTS. Relaxing it is the 2026-07-27 false-green defect inverted. Options are put to the operator separately.

Declared gaps, not silently filled: whether the missing .gz actually breaks a node's apt-get update (the .xz sibling is present; apt was NOT run from a node); whether tree-wide completeness holds (the tree was not reconciled against the Release indices, and the agent's find was truncated -- so "at least one file missing", never "exactly one"); and whether the 500 timeouts are archive.ubuntu.com or the edge path (a curl -I 200 shows reachability only -- HEAD never exercises the body read that timed out, so it is NOT evidence of transience).

  • Changed: docs/audit/mirror-exitcode-diagnosis-dc0-20260802.txt (new capture). scripts/dc-mirror.sh deliberately UNTOUCHED.
  • Revert: n/a -- diagnosis only.

Item 8 -- sweep F2, the re-trigger: the failure is PERSISTENT, and it does not matter

OPERATOR RULING (operational, not a D-number per GA-R3 -- no architectural consequence). Question put: what should dc-mirror.sh's gate attest, given debmirror's exit code cannot distinguish "nothing mirrored" from "mirrored minus N transient files" from "debmirror died"? Options offered: record shape+count without relaxing / leave as is / re-trigger first and decide after / add a bounded tolerance (listed as NOT recommended). Operator answer, exact utterance: "Re-trigger the sync first, decide after". The re-trigger was therefore run as a MEASUREMENT, not as a remedy, and the attestation question stays OPEN.

RESULT: THE FAILURE IS NOT TRANSIENT. The re-run (10:26:05 -> 10:31:20Z) failed identically -- FAIL ... ubuntu=1 uca=0, Result=exit-code, ExecMainStatus=1 -- on the SAME file with the SAME error and the SAME 300s timeout, two hours after the previous attempt:

08:27:56  Download of dists/jammy-backports/main/dep11/Components-amd64.yml.gz failed: 500 read timeout (300s)
10:31:16  Download of dists/jammy-backports/main/dep11/Components-amd64.yml.gz failed: 500 read timeout

So the earlier "flaky, not stuck" reading -- which the fail-then-succeed pairs supported -- does NOT extend to THIS failure. Two distinct things were being conflated: the unit has a history of transient failures, AND it now has one persistent one. The ruling's "decide after" is answered: waiting will not clear it.

THE SCOPE OF THE GAP IS NOW EXACT, replacing the agent's "at least one file missing". debmirror's own Files to download: 6349 B equals the size of that single file, so the entire outstanding delta of the 952 GiB tree IS this one file. On disk:

present:  Components-amd64.yml.xz   5748 B   (Release lists 5748 -- size-correct)
ABSENT:   Components-amd64.yml.gz   6349 B   (Release lists 6349)

AND IT DOES NOT BREAK apt -- MEASURED, not reasoned. The prior agent explicitly declared this unmeasured and declined to reason it away, which was correct. Tested from the rack with a fully ISOLATED apt state dir (scratch Dir::State::Lists, Dir::Cache, Dir::Etc::SourceList; nothing system-wide touched, no package installed):

deb http://10.12.8.4/ubuntu jammy-backports main
  Get:1 InRelease [127 kB]
  Get:2 main amd64 Packages [70.2 kB]
  Get:3 main amd64 Components [5748 B]      <-- the .xz, the format that IS present
  Get:4 main amd64 c-n-f Metadata [412 B]
  APT_UPDATE_RC=0
  apt-cache policy -> release v=22.04,o=Ubuntu,a=jammy-backports,c=main,b=amd64

apt asks for the .xz and never asks for the .gz. So the mirror is FUNCTIONALLY complete for the deploy and the gate is red over a file no consumer requests. jammy-backports was chosen as the test suite deliberately -- it is the one carrying the missing file.

ONE ANOMALY, UNEXPLAINED AND LEFT THAT WAY rather than hand-waved. The file fetches FINE by curl from the same host, right now: http=200 size=6349 time=0.436s, with a real body read (HEAD would prove nothing). Its .xz sibling and a jammy control also return 200. So curl succeeds in 0.44s where debmirror times out after 300s, twice. Ruled OUT: any proxy (the unit's environment is HOME=/root only; no http_proxy in /etc/environment, apt.conf.d, profile.d, or root's login shell). NOT chased further -- it does not gate the deploy, and guessing at an LWP-vs-curl difference would be exactly the reasoning-instead- of-measuring this project keeps logging.

  • Changed: nothing in the repo. scripts/dc-mirror.sh still UNTOUCHED; no tolerance was relaxed and no assertion weakened.
  • State changed on the rack: one sync run, which wrote last-sync.status and fetched ~1.3 MB of indices. The apt test wrote only to a scratch dir, removed on exit.
  • Revert: n/a.

Item 9 -- sweep F2 ROOT-CAUSED, and item 8's "not transient" is WITHDRAWN

OPERATOR RULING, exact utterance: "Root-cause the curl/debmirror anomaly first" -- taken over accepting the red gate, over recording error shape/count, and over --ignore-small-errors. Capture: docs/audit/mirror-500-timeout-rootcause-20260802.txt.

ROOT CAUSE: UPSTREAM AND BACKEND-SPECIFIC. archive.ubuntu.com is round-robin across NINE A records and ONE of them, 91.189.92.23, HANGS on this specific object while serving its directory siblings normally.

THE FAILING OBJECT (.yml.gz)          CONTROL, SAME DIR (.yml.xz)
185.125.190.81  200  6361  0.305s     91.189.92.23  200  5776  0.543s   <-- healthy
...(seven more, all 200)                (all nine 200)
91.189.92.23     -     -   >25s TIMEOUT

The control is what makes this a finding rather than "a backend is down": the same backend serves the sibling in half a second. The resolver ROTATES -- six consecutive lookups returned six different orderings -- so each fetch re-rolls which backend it gets.

MEASURED RATE: 12 sequential fetches -> 11 x 200, 1 timeout (~8%), consistent with exactly one bad record of nine.

SO ITEM 8's "NOT TRANSIENT" IS WITHDRAWN (GA-R1 C2 -- measurement wins over a document, including one written an hour earlier). The failure is PROBABILISTIC, not deterministic; two consecutive debmirror failures were not the proof of persistence they looked like.

A HYPOTHESIS TESTED AND REFUTED, kept because it is the one a reader forms first. debmirror sets keep_alive => 1 and curl does not reuse connections, so a stale persistent connection looked like the answer. The test INVERTED it -- same library, same callback style, seconds apart:

keep_alive=1 (debmirror's setting)   200 OK            bytes=6361   1s
keep_alive=0 (my "control")          500 read timeout  bytes=0     30s

ALSO CORRECTED: the 500 was never a server 500. LWP synthesises status 500 for CLIENT-side failures, so it is LWP reporting its own 300s timeout (/usr/bin/debmirror:629, our $timeout=300;). I had read it as an archive-side error, which points the investigation the wrong way.

TWO STRUCTURAL FACTS, read from the vendor script rather than assumed -- both bear on options that were on the table: there is NO dep11 exclusion option (dep11_from_release :1201 and get_dep11_files :1384 are unconditional), so the mirror's scope cannot be narrowed to drop AppStream metadata without patching debmirror; and --ignore-small-errors is NOT a bounded tolerance -- :2891 $num_errors = $t if ($ignore_small_errors); DISCARDS the count, so it would read clean on a wholly failed mirror.

RESIDUAL, DECLARED NOT EXPLAINED: debmirror failed 2 of 2, which a naive 1-in-9 model predicts at ~1.2%. Untested candidates: LWP's keep-alive pool may pin a whole run to one resolved address so one bad draw poisons the run; or more backends may be intermittently bad. The per-IP sweep tested each backend ONCE, so it establishes that 91.189.92.23 was bad, NOT that it is the only bad one.

SECOND, INDEPENDENT FINDING -- A DIFFERENT FAILURE MODE IS NOW LATENT. The local Release expects 6349 B; every backend now serves 6361 B. The object was republished DURING this investigation (an early curl measured 6349, every later one 6361). A successful download would now fail HASH verification against the stale local Release -- a different error from the timeout, and not fixed by fixing the timeout. Whether debmirror re-fetches Release early enough each run to avoid this was NOT measured.

  • Changed: docs/audit/mirror-500-timeout-rootcause-20260802.txt (new capture). scripts/dc-mirror.sh STILL untouched.
  • Revert: n/a -- read-only investigation.

Item 10 -- D-139 step 6 tool BUILT, adversarially reviewed, FOUR defects fixed

netbox/d139-step6-vip-rehome.py + tests/d139-step6-vip-rehome/run-tests.sh (20 cases). Implements the two 2026-08-02 rulings: "Full step 6 first, then deploy" and "Deprecate both, delete nothing". CREATE 26 GUA VIP addresses -> read-back verify -> deprecate 26 ULA addresses + 9 ULA prefixes. No delete path anywhere, asserted against the artifact.

Dry run (live apex): CREATE 26 | ALREADY 0 | DEPRECATE-ADDR 26 | DEPRECATE-PFX 9. The 26 CREATE targets diff EXACTLY against the 26 GUA VIP legs in overlays/vr1-dc0-vips.yaml -- every address written to the apex is one the deploy configures, and vice versa. Nothing has been written; --commit has not been run.

AN ADVERSARIAL REVIEW RETURNED "FIX FIRST" AND IT WAS RIGHT ON ALL FOUR COUNTS. Capture: docs/audit/d139-step6-tool-review-20260802.txt. The mapping logic was correct; what was missing was PRECONDITIONS and COVERAGE.

  • DEF-1 (CRITICAL) -- --dc vr1-dc1 would ORPHAN-CREATE. gua_net_for_role() computed targets arithmetically and never asked the apex whether they EXIST. MEASURED: dc1's GUA carve is incomplete -- four rows under 2602:f3e2:f03::/48, all provider-public, no :20::/64, no :21::/64 -- so dc1 planned 26 creates into non-existent prefixes then deprecated dc1's only authoritative rows, at rc=0 with no warning. dc0 hid it because all sixteen of its targets happen to exist. Reachable via the other valid value of a required flag. FIXED: the target must exist in the apex or the tool REFUSES. Verified live -- dc1 now refuses, dc0 unchanged at 26/26/9.
  • DEF-2 (CRITICAL) -- the apex-IDENTITY guard was gone. d139-gua-carve.py:159-163 refuses any host that is not the working apex (DOCFIX-195), but that guard lives in its main(), and importing a module never runs its main(). Subclassing C.NB inherited the TRANSPORT and left the SAFETY POSTURE behind -- pointing this at netbox.baldurkeep.com (the v1 reference) would have connected fine and written to the wrong NetBox. FIXED: identity checked before any network call.
  • DEF-3 (HIGH) -- silent under-count. Removing one ULA /64 row yielded CREATE=13 / DEPA=13 / DEPP=8 at exit 0 -- 13 live VIPs neither created nor deprecated, reported as success. Same class as DEFECT 2 of the 2026-08-01 CORRECTION NOTE. FIXED: any ULA address claimed by no prefix row REFUSES. My first fix was itself wrong and running it caught that -- it scanned the whole retired /48 and flagged dc1's 26 VIPs while planning dc0. The retired /48 is SHARED (dc0 :22x, dc1 :32x), so the check now excludes addresses another DC's row claims, and deliberately does NOT count a /60 parent as coverage -- the reviewer's scenario was exactly a missing /64 whose /60 still existed.
  • DEF-4 (HIGH) -- main() had ZERO coverage. The reviewer hoisted the deprecate loops above the create phase and the suite reported ALL PASS. The CREATE-before-DEPRECATE ordering both rulings call load-bearing was untested. FIXED: T16-T18 drive main() through a fake client that RECORDS CALL ORDER. Proven by re-running the reviewer's exact mutation on a copy: T18 goes RED.

TWO OF MY ASSERTIONS COULD NOT FAIL, and the review killed both. T13 asserted the ABSENCE of a string, so a traceback satisfied it -- it passed against a tool file that did not parse. Now requires a positive, well-formed, DIFFERENT target, and T15 separately asserts the tool parses, closing the escape hatch. T14 grepped ONE file, so adding a delete to the IMPORTED d139-gua-carve.py left it green; it now covers both.

Harness 14 -> 20 cases, count delta checked (the rule landed in item 3 today, applied to my own delivery). Gauntlet manifest recorded deliberately, 97 -> 98.

Corrected in the ruling record (GA-R1 C2): the amendment said the GUA records would be status=active. MEASURED: the live ULA VIP records are reserved, and dc-plane-apex-import.py:186,200 creates addresses reserved. Both docs/design-decisions.md and docs/CURRENT-STATE.md corrected.

Also corrected -- my own overclaim: the docstring said "every action is reversible by flipping a status back". True of the 26+9 DEPRECATIONS; NOT true of the 26 CREATES, which would need a delete this repo does not have. Reversal is not wanted (the creates are the records the deploy matches) but the claim was wrong and is now stated precisely.

OPEN SCOPE QUESTION, MEASURED, NOT A TOOL DEFECT -- THE MAAS HALF OF STEP 6. D-139's list says retire the ULA rows "in the apex and in MAAS". This tool is apex-only, so step 6 is NOT complete when it finishes. MEASURED on the dc0 region: MAAS holds BOTH families -- five ULA /64s (:220, :221, :230, :240, :250) beside six GUA. Four are EMPTY; fd50:840e:74e2:220::/64 (metal-admin) still holds 2 allocated entries, so this cannot be a blind cleanup. MAAS has no deprecated status for a subnet -- the choice is delete or leave -- which makes it a separate operator decision, not an extension of the "deprecate" ruling.

  • Changed: netbox/d139-step6-vip-rehome.py (new), tests/d139-step6-vip-rehome/run-tests.sh (new, 20 cases), tests/HARNESS-MANIFEST (+1, recorded deliberately), docs/audit/d139-step6-tool-review-20260802.txt (new), plus the two doc corrections.
  • Revert: delete the tool, its harness dir and the manifest line; re-run bash scripts/run-tests-all.sh --record-manifest. Nothing was written to the apex.

Item 11 -- D-139 STEP 6 EXECUTED: apex applied, and 4 of 5 MAAS ULA subnets removed

OPERATOR APPROVAL, exact utterance: "Queue up the MAAS half to run after you complete netbox. I approve the MAAS deletes to clean up the data and config. Go ahead with both now".

THE APEX HALF IS APPLIED, exit 0. d139-step6-vip-rehome.py --dc vr1-dc0 --commit: CREATE 26 | ALREADY 0 | DEPRECATE-ADDR 26 | DEPRECATE-PFX 9, both read-backs OK. INDEPENDENTLY VERIFIED after the fact, not on the tool's own word: a re-run reports CREATE 0 | ALREADY 26 | DEPRECATE-ADDR 0 | DEPRECATE-PFX 0 -- fully idempotent and converged.

RECORDED SO A LATER READER IS NOT MISLED: d139-gua-carve.py STILL reports dependent objects ... 26 ip-address(es). That is CORRECT and expected -- the 26 ULA records still EXIST, deprecated rather than deleted, per the ruling. The carve tool counts existence, not status. It is not evidence that step 6 did not run.

THE MAAS HALF: FOUR SUBNETS DELETED, ONE DELIBERATELY HELD.

VERIFY BEFORE MUTATE FOUND A REAL HAZARD, and it is why the fifth was not deleted. The approval was given before anyone knew WHAT the fd50:840e:74e2:220::/64 allocations were. Measured:

fd50:840e:74e2:220::5   alloc_type=1 (STICKY)  user=juju-vr1-dc0   <- the Juju controller
fd50:840e:74e2:220::6   alloc_type=1 (STICKY)  user=MAAS           <- the MAAS region VM

And they have NO GUA counterpart. The GUA metal-admin subnet holds exactly nine entries -- ::100,::101,::102 control, ::120,::121 compute, ::150-::153 storage -- the nine tagged ROLE NODES and nothing else. This matches the record that dc-node-v6-carve.py walks only the nine tagged role nodes and that the controller's own v6 was "restored separately" -- restored on ULA, never migrated. So deleting that subnet would remove the only recorded v6 address of the deploy client AND of the MAAS region the delete is issued to. That is not cleanup, and it is outside what the approval could have meant. HELD, pending its own decision.

THE FOUR EMPTY ONES WERE SAFE, and that was established twice over. Zero allocations AND zero machine-interface links for all five ULA subnets -- the stronger check, since "no allocated IP" does not by itself mean "nothing references it". Zero links also CONFIRMS D-139 step 3 succeeded: the nine role nodes are fully on GUA.

Deleted INDIVIDUALLY, never looped -- hard rule 3 forbids batching destructive steps, and this repo has a logged incident from exactly that (2026-07-30, nine deletes in a loop; the guard stopped it). Each got a fresh cidr + allocation re-check immediately before its own delete:

subnet  9  fd50:840e:74e2:221::/64  metal-internal  allocations=0  deleted rc=0
subnet  8  fd50:840e:74e2:230::/64  data-tenant     allocations=0  deleted rc=0
subnet 12  fd50:840e:74e2:240::/64  storage         allocations=0  deleted rc=0
subnet 11  fd50:840e:74e2:250::/64  replication     allocations=0  deleted rc=0

POST-STATE, MEASURED: six GUA /64s remain (f02:10/20/21/30/40/50), one ULA remains (fd50:840e:74e2:220::/64, id=2, held). Machines unchanged and healthy: 9 Ready + 1 Deployed (the controller), total 10.

STILL OPEN, needs a decision: the held subnet. Options are to migrate the controller's and region's v6 to GUA first and then delete, or to delete and accept losing those two records, or to leave the subnet in place indefinitely. NOT decided here.

NO REPO TOOL EXISTS for the MAAS half -- these were hand-issued maas subnet delete calls. If the dc1 rebuild is to be reproducible, this belongs in a script with the same emptiness + link preconditions asserted, and that is registered as owed rather than built.

  • Changed: live apex (26 created, 35 deprecated) and the dc0 MAAS region (4 subnets deleted). Repo: this changelog + CURRENT-STATE.
  • Revert: apex -- flip the 35 statuses back (the 26 creates are additive and wanted). MAAS -- the four subnets can be recreated (dc-region-topology.sh carries the shape), but their identities are gone; they held nothing, so nothing else is lost.

Item 12 -- the MAAS half gets a tool, and it immediately caught a defect in MY OWN check

OPERATOR DIRECTION: "Proceed with 1 and 2" -- 1 = the held ULA subnet, 2 = build the repo tool for the MAAS half. Item 2 is delivered here. Item 1 is NOT executed and is explained below.

scripts/dc-plane-ipam.sh gains retire-v6-ula <site> [--commit]. It belongs there rather than in a new script: dc-region-topology.sh's own header assigns v6 plane subnets to dc-plane-ipam.sh, and a one-off script would have split that ownership. Harness 25 -> 32 cases (R1-R7), delta checked.

The mode: enumerate this site's subnets inside the retired ULA /48; for each, REFUSE if it holds an allocation OR a machine-interface link, NAMING the holder; otherwise delete -- one at a time, never looped -- and READ BACK that the row is gone, because a delete returning success is not evidence the row went away (R6 proves that assertion has teeth).

>>> THE TOOL'S FIRST LIVE RUN CAUGHT A DEFECT IN THE BY-HAND CHECK I USED AN HOUR EARLIER, AND IT IS THE MORE IMPORTANT FINDING HERE. <<< Its dry run reported:

interface links: subtle-grouse:enp1s0

My by-hand link check (item 11) reported ZERO links for all five subnets. The cause is a quoting bug in my own one-liner: the subnet id was interpolated as sid=\"'$id'\", so the comparison ran str(id) == "'2'" and could never match. It was a checker that could not fail -- written by me, in the same session in which I landed exactly that rule into script-authoring.md. The tool found a real link the moment it ran.

CONSEQUENCE, STATED PLAINLY: item 11's claim that the four deletes were "proven safe twice" was HALF FALSE. The allocation count was sound; the link half was inert, so link status was in fact UNKNOWN at the time of those four deletions.

THE OUTCOME WAS NEVERTHELESS SAFE, and that is measured, not assumed. After the deletes, all nine role nodes hold v4=6, v6=6 -- six planes in each family -- so every v6 link they carry is GUA and the four deleted ULA subnets genuinely held none. enp2s0 reads unlinked on all nine, which is EXPECTED (provider-public carries no L3 link; the static lands on br-ex, D-133/D-100), not damage. Right outcome, unsound verification -- the two are recorded separately on purpose.

subtle-grouse (the juju controller, Deployed) reads v4=2 v6=1: two planes per the role-dependent carve, and its single v6 link is the ULA ::5. That INDEPENDENTLY confirms holding subnet 2 was correct.

ITEM 1 IS NOT EXECUTED, and the tool is why. Run against the live region, retire-v6-ula vr1-dc0 REFUSES on fd50:840e:74e2:220::/64 and names all three holders: ::5 (juju-vr1-dc0), ::6 (MAAS), and the interface link subtle-grouse:enp1s0. Deleting it still requires re-homing the juju controller's and the MAAS region's v6 onto GUA first -- which re-addresses the DEPLOY CLIENT immediately before the deploy, and this project has already lost three bootstrap attempts to under-carved controller networking. That is a mutation with its own risk profile and is NOT covered by "I approve the MAAS deletes to clean up the data and config", which was given about four empty subnets. It is also NOT deploy-blocking: an extra ULA subnet in MAAS harms nothing.

  • Changed: scripts/dc-plane-ipam.sh (new mode), tests/dc-plane-ipam/run-tests.sh (R1-R7).
  • Revert: drop the retire-v6-ula block, its usage lines and the ACTION case entry; drop R1-R7 and the two retire helpers.