# Changelog -- 2026-08-02 (session 2): clearing the queued-findings backlog

Session scope: work the queued items left by the 2026-08-02 close bookend
(`docs/audit/queued-findings-20260802-stage5-edge-fold.txt`), in consequence order.
Stage 5 remains OPEN; nothing here opens or closes a stage.

**Numbering note, because two registers collide.** This session's items cite the SWEEP
register (F1-F6, the queued-findings file above). The runbook fold register
(`docs/runbook-fold-register.md`) has its OWN F1-F12 with different meanings -- sweep F5
is "share the probe definition", fold F5 is "build the OOB plane". Every reference below
is prefixed `sweep F<n>` or `fold F<n>`. Do not drop the prefix.

---

## Item 1 -- DOCFIX-207: preflight P6 quoted a plan count stale since 2026-07-10

**Sweep S5(a).** `scripts/preflight.sh:369` printed the phase-01 Step 1.2 reminder as
`plan: 50 apps / 97 relations`. The real figure is **56 apps / 108 relations**, MEASURED
twice and recorded: `docs/CURRENT-STATE.md:955` (`juju deploy --dry-run` resolved all 56
charms, planned 108 relations, exit 0) and again 2026-07-31 against the fixed deploy
input. Measurement wins over a document (GA-R1 C2).

The staleness is older than it looks: `docs/archive/changelogs/changelog-20260710-phase01-bundle-staleness-fix.md:18`
corrected the SAME figures in `phase-01`'s own Step 1.2 GATE on 2026-07-10 and missed
this copy in preflight. So for three weeks the executable gate and the reminder pointing
at it disagreed by 6 apps and 11 relations.

Scope, stated so it is not overclaimed: this is a REMINDER string inside the
`P6: stage-2 reminders (NOT run here)` block, not an assertion. Nothing gated on it and
no harness asserts it (`grep '50 apps' tests/` -> one unrelated hit, a line-number probe
for the P6 header). The defect is that a session comparing a live dry-run against it
would read a correct plan as wrong.

- Changed: `scripts/preflight.sh` -- the figure, plus a comment carrying the measurement
  provenance and the explicit note that the real gate is the dry-run itself.
- Verified: `bash tests/preflight/run-tests.sh` -> **43/43 PASS**; `repo-lint` 0 fail.
- **Revert:** restore `50 apps / 97 relations` on that echo line and drop the comment
  block above it.

## Item 2 -- sweep F3 graduated: `systemctl show` is not an existence check

The sweep recorded that `systemctl show` on a NON-EXISTENT unit returns
`Result=success` / `ExecMainStatus=0` / `RemainAfterExit=no` and no journal entries --
indistinguishable from a healthy oneshot that has completed. Measured 2026-08-02 against
the invented name `dc0-rack-net`; the real units are `dc0-rack-legs` and `dc0-node-dns`
(`scripts/dc-rack-net.sh:16,22`). It produced a false "inactive" reading that was
reported before being caught.

It lived only in the sweep capture, which is an audit artifact, not a place a future
session looks before running a systemd check. Graduated to its durable home as
**platform-traps section 5c**, with the verbatim output, the reason (`show` answers from
property DEFAULTS), and the three instruments that DO error on absence (`systemctl cat`,
`list-unit-files`, or asserting `UnitFileState` non-empty -- the only field that betrays
the absence).

- Changed: `.claude/skills/openstack-cloud-ops/references/platform-traps.md` (new 5c).
- **Revert:** delete the 5c block.

## Item 3 -- sweep F4 + F5 graduated: two harness-discipline rules

Both lived only in the sweep capture. Landed in
`.claude/skills/openstack-cloud-ops/references/script-authoring.md`, in the Testing
section immediately before the repo-record traps.

**Sweep F4 -- assert the CASE COUNT moved.** Two new cases in `tests/dc-egress-check`
were written with `fok`/`fbad`, helper names from `tests/dc-node-v6-carve`. The suite
printed `fbad: command not found` to stderr, kept its count at 14, and reported ALL PASS.
The mutations those cases targeted then "survived", reading as robustness when it was
really the tests not existing. The compounding fact is the one worth carrying: the
symptom of a non-existent test and of a passing test are the same symptom, so the
delivery check has to be arithmetic -- count before, count after, delta equals cases
written. Ties into the standing rule that a checker which cannot fail is not a gate:
prove a new assertion can FAIL *and* prove it RAN.

**Sweep F5 -- two scripts probing one endpoint must share the probe definition.**
`dc-egress-check.sh`'s A4 and `dc-snap-proxy.sh:215` both hit `api.snapcraft.io`; only
the latter sent `Snap-Device-Series: 16`, which the store requires. Measured 400 without
it both through the proxy and direct, 200 with it. The newer gate called a healthy proxy
broken on its first live run while the older script returned PASS with a real payload --
two gates disagreeing about one piece of infrastructure. The generalisable half, which
was NOT on surface: **a fixture that mocks `curl` cannot catch a wrong REAL request**, so
the harness stays green in both worlds. A duplicated request line is a duplicated
contract, and only one copy gets fixed when the endpoint tightens.

- Changed: `.claude/skills/openstack-cloud-ops/references/script-authoring.md`
  (two paragraphs).
- **Revert:** delete the two paragraphs.

## Item 4 -- sweep F1 repo-side provenance MEASURED (no mutation yet)

Before propagating `overlays/vr1-dc0-vips.yaml` to the dc0 rack's `~/repo-stage` (the
D-138 deploy client), the repo copy's correctness was measured rather than inherited from
the close bookend:

- **It conforms to D-139 ruling B.** `grep -o 'fd50:[0-9a-f:]*'` -> **zero** ULA legs.
  39 GUA legs, `2602:f3e2:f02:11` x13 (provider-public), `f02:20` x13, `f02:21` x13.
  That is exactly the 26-leg migration the 2026-08-01 conformance audit named as its
  blocker C (`docs/audit/d139-conformance-dc0-20260801.txt:44-56`), so blocker C is
  CLOSED in the repo copy.
- **It has renderer provenance.** `bash tests/render-drift/run-tests.sh` -> 4/4 PASS,
  naming the file: `render/values/vr1-dc0-vips.yaml renders overlays/vr1-dc0-vips.yaml
  byte-for-byte`, and the suite carries a PROOF-OF-TEETH case (a mutated values file is
  detected). The values file declares `output: overlays/vr1-dc0-vips.yaml` at line 2.
- Repo digest confirmed `80d861560a6b3c52`, matching what the close recorded.

- Changed: nothing. Read-only measurement.
- **Revert:** n/a.

## Item 5 -- sweep F1 CLOSED: the dc0 rack's deploy input is re-staged

A read-only agent enumerated the WHOLE staging directory first, rather than diffing the
one file the close bookend named. Capture: `docs/audit/repo-stage-drift-dc0-20260802.txt`
(427 lines). Classification of the 14 regular files + 4 dirs under `~/repo-stage`
(`.git` re-confirmed absent):

- **(A) diverged: 1** -- `overlays/vr1-dc0-vips.yaml` only. Corroboration the bookend did
  not have: **voffice1's clone also holds `80d86156...`**, so the rack was the ONLY host
  still carrying the pre-GUA overlay.
- **(B) missing from the Step-4 deploy closure: 0.** This was the open question, and it
  matters because a prior deploy died on a missing `policies/overrides.zip`. That file IS
  staged, at the repo digest `02fe1fd7...e720a011`, and resolves because `policies/` sits
  beside `bundle.yaml`. All three scannable merge inputs were checked, not just the
  bundle: `bundle.yaml` has exactly one local reference (`:227-228`); neither the VIP nor
  the machines overlay has a `resources:` block or any local path.
- **(C) rack-only orphans: 0.** All 14 staged files resolve to a repo path.
- Two later-step overlays are unstaged (`dc-dc-ipv6-family-matrix.yaml`,
  `dc-ha-scaleup.yaml`); not Step-4 inputs, recorded as a fork, not ruled.

**THE MUTATION (gated, single file, executed 2026-08-02).**
`scp -p overlays/vr1-dc0-vips.yaml vr1-dc0-rack:/home/jessea123/repo-stage/overlays/vr1-dc0-vips.yaml`

A single-file copy was the minimal correct action BECAUSE the enumeration proved exactly
one file diverged. A wholesale directory refresh was rejected for the same reason the
2026-07-31 re-stage rejected it: it risks clobbering or re-permissioning the `0600`
`vr1-dc0-octavia-pki.yaml` (SEC-029).

Before / after, measured by this session on both ends:

```
                                    BEFORE            AFTER             REPO
overlays/vr1-dc0-vips.yaml          ed19d989e80da8da  80d861560a6b3c52  80d861560a6b3c52
bundle.yaml                         4c8a78527a3100f6  (untouched)       4c8a78527a3100f6
overlays/vr1-dc0-machines.yaml      e3be85e41b419486  (untouched)       e3be85e41b419486
overlays/vr1-dc0-octavia-pki.yaml   5fc117f188332042  5fc117f188332042  (gitignored)
```

**POST-COPY, ALL FOURTEEN STAGED FILES WERE RE-VERIFIED AGAINST CURRENT HEAD**, not just
the four in the table -- because the staged `scripts/` copies were last checked at an
EARLIER commit and the 2026-08-02 session shipped script changes after that check. All 13
tracked files match the repo digest-for-digest (`bundle.yaml`, both non-PKI overlays,
`policies/domain-manager-policy.yaml`, `policies/overrides.zip`, and the eight staged
`scripts/`), and the 14th matches voffice1. **The staging directory is now fully in sync,
confirmed against HEAD rather than inherited from the prior verification.**

Checked and NOT a gap: `dc-egress-check.sh` is absent from `~/repo-stage` by design -- it
is streamed over stdin (`ssh <rack> 'bash -s -- check <site>' < scripts/dc-egress-check.sh`,
per its own refuse message at `:90-91`), so it is never staged and cannot go stale there.

The PKI overlay is verified UNCHANGED -- same digest, same `0600`, same mtime
(Jul 31 02:38) -- and was never read, only hashed. It measures IDENTICAL to voffice1's
copy, so the SEC-029 superseded-cert hazard is not realized today; nothing enforces the
dual update on the next reissue, which stays an open SEC-029 obligation.

- **Revert:** copy the pre-GUA overlay back. It is NOT retained anywhere as a file --
  reverting means re-rendering from `render/values/` at the parent of the GUA commit.
  There is no reason to: the staged copy now matches repo HEAD, which is the invariant.

**TWO FINDINGS THE AGENT SURFACED THAT WERE NOT ASKED FOR, both logged not fixed:**

1. **The staged inventory is 14 files, not the nine that `docs/security-ledger.md`
   SEC-029(3) records.** The +5 are `opnsense-*` files left by the 2026-08-02 edge
   rebuild (digests match `dc0-edge-rebuild-20260802.txt:259-264`). A future re-stage
   planned off the "nine-file copy" figure would omit five files. The ledger's prose is
   stale, not wrong-in-kind.
2. **A GATE/DEPLOY ASYMMETRY, DOCFIX-shaped and worth attention before Step 4.**
   `overlays/vr1-dc0-machines.yaml` exists and `scripts/preflight.sh:144-145` folds it
   into the merged input that P2 validates -- but `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:527-528`
   asserts the file does NOT exist and Step 4 does not pass it. **So P2 validates a
   different artifact than the deploy consumes.** That is the class of defect this
   project has repeatedly found (a gate grading the workspace while the overlay is what
   reaches the charm). NOT fixed here -- hard rule 1; it likely bundles with the
   outstanding D-138 run-location DOCFIX.

## Item 6 -- sweep F6 CLOSED: the dc0 MAAS region DB is proven uncorrupted

**PASS.** `pg_dump` read every page of every table in `maasdb`: **23,878,796 bytes /
37,199 lines / exit 0 / empty stderr**, with `-- PostgreSQL database dump complete`
asserted in a separate pass. A torn or bad-checksum page anywhere would have aborted the
dump, so the 2026-08-01 hard power cut left NO page-level corruption. Evidence appended
to `docs/audit/maasdb-pgdump-integrity-dc0-20260802.txt` (the agent's BLOCKED report is
left verbatim above it; sections 7-11 are the resolution).

**F6's stated blocker was the wrong diagnosis, and this is the durable part.** The
finding attributed five prior failures to snap confinement and prescribed a login shell.
Neither was operative:

- Confinement did NOT reproduce as `ubuntu` -- the `postgresql.dump` wrapper ran, opened
  the socket, and the server answered. (It DOES reproduce as `_daemon_`: `cannot create
  snap home dir: mkdir /nonexistent` + `is not a snap cgroup`. So the original reading
  was true for the user it was measured on and got generalised too far.)
- The real discriminators were **ROLE** and **TRANSPORT**. Role `ubuntu` does not exist;
  the role is `maas`. And over TCP (`-h localhost`) role `maas` is password-challenged
  (`fe_sendauth: no password supplied`), which would have meant handling the regiond.conf
  secret -- while over the **UNIX SOCKET (`-h /tmp`) it authenticates with no credential
  at all.** Dump over the socket; never over TCP.

**Both identity values are now MEASURED, and one of them was prose all along.** `maasdb`
came from F6's text and had never been verified; `sudo grep -E "^database_name|^database_user|..."`
on `regiond.conf` confirms `maasdb` / `maas` / `localhost`. Only non-secret field names
were grepped -- `database_pass` was never read, and the dump path chosen above means it
never needed to be.

The dump was STREAMED through `awk` and nothing was persisted: the MAAS DB carries API
keys and password hashes, so writing it to disk would have created a new exposure needing
its own SEC row.

**Scope, stated so it is not overclaimed:** this proves the pages are readable and the
rows decode. It does NOT prove MAAS's application-level invariants hold. Nothing observed
suggests otherwise and no such claim is made.

**The agent's false-green finding, confirmed and worth keeping:** `SSH_EXIT` was 0 while
`pg_dump` exited 1 having produced 0 bytes -- so F6's own proposed form,
`ssh host 'pg_dump maasdb > /dev/null' && echo PASS`, would have printed PASS over a dump
that never happened. Same family as sweep F3 and sweep F4. Asserting on BYTES plus the
completion marker is what makes this a gate rather than a ritual.

- Changed: `docs/audit/maasdb-pgdump-integrity-dc0-20260802.txt` (new capture).
- **Revert:** n/a -- a read-only measurement. Nothing on the VM was altered.

## Item 7 -- sweep F2 DIAGNOSED: reading R1 is true, R2 is refuted

Sweep F2 left two readings open and forbade assuming the first. A read-only agent settled
it. Capture: `docs/audit/mirror-exitcode-diagnosis-dc0-20260802.txt`.

**VERDICT: R1 -- debmirror itself prints `All done.` and then exits non-zero. Nothing runs
after `All done.` that could fail.** R2 (something failing after the message) has no
candidate step.

**Decisive evidence -- the journal shows `All done.` TWICE, once per debmirror leg, and
the failing one is followed 10 ms later by its own error block in the SAME PID:**

```
08:28:00.830168 dc0-mirror-sync[276574]: All done.
08:28:00.840132 dc0-mirror-sync[276574]: Errors:
08:28:00.840132 dc0-mirror-sync[276574]:  Download of dists/jammy-backports/main/dep11/Components-amd64.yml.gz failed: 500 read timeout
08:28:00.840132 dc0-mirror-sync[276574]: Failed to download files (1 errors)!
```

**Confirmed at VENDOR SOURCE, and re-verified independently by this session** (not taken
on the agent's word, since it is the load-bearing claim) -- `debmirror 1:2.39ubuntu2` on
the dc0 rack:

```
1615:say("All done.");
1620:  exit 1 if (!$ignore_small_errors);
```

The helper does not pass `--ignore-small-errors`, so any non-zero error tally exits 1
AFTER the success-looking message. Corroborating: between the LAST `All done.` (the UCA
leg, 08:28:03.833394) and systemd's failure line (08:28:03.840126) there are 6.7 ms and
ZERO log lines. And the installed helper is byte-identical to the repo's generated
content -- no drift.

**Proximate cause of the 1:** exactly one upstream download,
`jammy-backports/main/dep11/Components-amd64.yml.gz`, hit a 500 read timeout at 300s.

**THE SWEEP'S OWN PREMISE WAS WRONG, AND THE ERROR IS INSTRUCTIVE.** F2 states "its own
log says the sync succeeded". Every line it quoted (`Files to download: 0 B`,
`Downloaded 15 kiB in 3s`, `Everything OK...`, `All done.`) is from the SECOND debmirror,
PID 277623 -- the UCA leg, which SUCCEEDED. The failing ubuntu leg reported
`Downloaded 2867 kiB in 313s` and an error block. **The tail of the passing leg was read
as the tail of the whole run, and the evidence for R1 sat three lines below the last line
quoted.** Same family as this project's standing instrument lessons: a truncated read
that looks exactly like a complete one.

**A SECOND CORRECTION: "`dc-mirror.sh check dc0` stays RED" OVERSTATES IT -- the gate is
FLAKY, not stuck.** The agent reported "7 of 14"; that did not reconcile against the
journal, so it was re-counted. **MEASURED, full journal window 2026-07-23..08-02: 16 runs,
7 `Finished`, 9 `Failed with result 'exit-code'`.** The success count was right, the
denominator was not. Its "four successes in the last four days" IS correct.

**My own first re-count was ALSO wrong, and it is logged rather than quietly dropped:** a
`grep -cE "Starting .*mirror sync"` returned 0 because the unit's description reads
`dc0 mirror upstream sync`, so "mirror sync" never matched. It looked like "no runs
recorded". Checking the instrument before believing a surprising negative is the standing
rule and it paid again.

**The stronger evidence is the SHAPE, not the ratio.** On FOUR separate days a failed run
was followed within hours by a clean one:

```
2026-07-25T06:20:23 Failed  ->  2026-07-25T06:23:59 Finished
2026-07-27T00:54:54 Failed  ->  2026-07-27T08:43:46 Finished
2026-07-31T00:54:12 Failed  ->  2026-07-31T02:35:23 Finished
2026-07-31T18:29:37 Failed  ->  2026-07-31T18:56:20 Finished
```

That is transience DEMONSTRATED, not inferred from an error string. Two caveats kept: the
earliest failure (2026-07-23) was the HOME-under-systemd defect fixed the same hour, so
the lifetime ratio understates current health; and the two most recent runs, both
2026-08-02, BOTH failed -- they bracket the edge outage, and the last success was
2026-08-01T00:04:19.

**Failure shapes are TWO, flagged so they are not later conflated:** 4 runs carry a
`Failed to download files (N errors)!` tally (the R1 path, N = 1/3/22/1); 3 carry
`Errors:` with no tally, meaning debmirror DIED rather than completing -- the `ubuntu=255`
shape `dc-mirror.sh`'s own comment records for 2026-07-27.

**NO REMEDY APPLIED, and the decision is the operator's (hard rule 1).** The agent
recorded only the SHAPE of the choice, which is the part that matters: debmirror's exit
code conflates "nothing mirrored" with "mirrored minus N transient files", so ANY
tolerance change alters what the gate ATTESTS. Relaxing it is the 2026-07-27 false-green
defect inverted. Options are put to the operator separately.

**Declared gaps, not silently filled:** whether the missing `.gz` actually breaks a node's
`apt-get update` (the `.xz` sibling is present; apt was NOT run from a node); whether
tree-wide completeness holds (the tree was not reconciled against the Release indices, and
the agent's `find` was truncated -- so "at least one file missing", never "exactly one");
and whether the 500 timeouts are archive.ubuntu.com or the edge path (a `curl -I` 200
shows reachability only -- HEAD never exercises the body read that timed out, so it is
NOT evidence of transience).

- Changed: `docs/audit/mirror-exitcode-diagnosis-dc0-20260802.txt` (new capture).
  `scripts/dc-mirror.sh` deliberately UNTOUCHED.
- **Revert:** n/a -- diagnosis only.
