Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260802-queued-items.md
@JANeumatrix JANeumatrix 1 day ago 22 KB sweep F2 re-trigger: failure is PERSISTENT, and measured not to affect the deploy
Raw Blame History

Changelog -- 2026-08-02 (session 2): clearing the queued-findings backlog

Session scope: work the queued items left by the 2026-08-02 close bookend (docs/audit/queued-findings-20260802-stage5-edge-fold.txt), in consequence order. Stage 5 remains OPEN; nothing here opens or closes a stage.

Numbering note, because two registers collide. This session's items cite the SWEEP register (F1-F6, the queued-findings file above). The runbook fold register (docs/runbook-fold-register.md) has its OWN F1-F12 with different meanings -- sweep F5 is "share the probe definition", fold F5 is "build the OOB plane". Every reference below is prefixed sweep F<n> or fold F<n>. Do not drop the prefix.

Item 1 -- DOCFIX-207: preflight P6 quoted a plan count stale since 2026-07-10

Sweep S5(a). scripts/preflight.sh:369 printed the phase-01 Step 1.2 reminder as plan: 50 apps / 97 relations. The real figure is 56 apps / 108 relations, MEASURED twice and recorded: docs/CURRENT-STATE.md:955 (juju deploy --dry-run resolved all 56 charms, planned 108 relations, exit 0) and again 2026-07-31 against the fixed deploy input. Measurement wins over a document (GA-R1 C2).

The staleness is older than it looks: docs/archive/changelogs/changelog-20260710-phase01-bundle-staleness-fix.md:18 corrected the SAME figures in phase-01's own Step 1.2 GATE on 2026-07-10 and missed this copy in preflight. So for three weeks the executable gate and the reminder pointing at it disagreed by 6 apps and 11 relations.

Scope, stated so it is not overclaimed: this is a REMINDER string inside the P6: stage-2 reminders (NOT run here) block, not an assertion. Nothing gated on it and no harness asserts it (grep '50 apps' tests/ -> one unrelated hit, a line-number probe for the P6 header). The defect is that a session comparing a live dry-run against it would read a correct plan as wrong.

  • Changed: scripts/preflight.sh -- the figure, plus a comment carrying the measurement provenance and the explicit note that the real gate is the dry-run itself.
  • Verified: bash tests/preflight/run-tests.sh -> 43/43 PASS; repo-lint 0 fail.
  • Revert: restore 50 apps / 97 relations on that echo line and drop the comment block above it.

Item 2 -- sweep F3 graduated: systemctl show is not an existence check

The sweep recorded that systemctl show on a NON-EXISTENT unit returns Result=success / ExecMainStatus=0 / RemainAfterExit=no and no journal entries -- indistinguishable from a healthy oneshot that has completed. Measured 2026-08-02 against the invented name dc0-rack-net; the real units are dc0-rack-legs and dc0-node-dns (scripts/dc-rack-net.sh:16,22). It produced a false "inactive" reading that was reported before being caught.

It lived only in the sweep capture, which is an audit artifact, not a place a future session looks before running a systemd check. Graduated to its durable home as platform-traps section 5c, with the verbatim output, the reason (show answers from property DEFAULTS), and the three instruments that DO error on absence (systemctl cat, list-unit-files, or asserting UnitFileState non-empty -- the only field that betrays the absence).

  • Changed: .claude/skills/openstack-cloud-ops/references/platform-traps.md (new 5c).
  • Revert: delete the 5c block.

Item 3 -- sweep F4 + F5 graduated: two harness-discipline rules

Both lived only in the sweep capture. Landed in .claude/skills/openstack-cloud-ops/references/script-authoring.md, in the Testing section immediately before the repo-record traps.

Sweep F4 -- assert the CASE COUNT moved. Two new cases in tests/dc-egress-check were written with fok/fbad, helper names from tests/dc-node-v6-carve. The suite printed fbad: command not found to stderr, kept its count at 14, and reported ALL PASS. The mutations those cases targeted then "survived", reading as robustness when it was really the tests not existing. The compounding fact is the one worth carrying: the symptom of a non-existent test and of a passing test are the same symptom, so the delivery check has to be arithmetic -- count before, count after, delta equals cases written. Ties into the standing rule that a checker which cannot fail is not a gate: prove a new assertion can FAIL and prove it RAN.

Sweep F5 -- two scripts probing one endpoint must share the probe definition. dc-egress-check.sh's A4 and dc-snap-proxy.sh:215 both hit api.snapcraft.io; only the latter sent Snap-Device-Series: 16, which the store requires. Measured 400 without it both through the proxy and direct, 200 with it. The newer gate called a healthy proxy broken on its first live run while the older script returned PASS with a real payload -- two gates disagreeing about one piece of infrastructure. The generalisable half, which was NOT on surface: a fixture that mocks curl cannot catch a wrong REAL request, so the harness stays green in both worlds. A duplicated request line is a duplicated contract, and only one copy gets fixed when the endpoint tightens.

  • Changed: .claude/skills/openstack-cloud-ops/references/script-authoring.md (two paragraphs).
  • Revert: delete the two paragraphs.

Item 4 -- sweep F1 repo-side provenance MEASURED (no mutation yet)

Before propagating overlays/vr1-dc0-vips.yaml to the dc0 rack's ~/repo-stage (the D-138 deploy client), the repo copy's correctness was measured rather than inherited from the close bookend:

  • It conforms to D-139 ruling B. grep -o 'fd50:[0-9a-f:]*' -> zero ULA legs. 39 GUA legs, 2602:f3e2:f02:11 x13 (provider-public), f02:20 x13, f02:21 x13. That is exactly the 26-leg migration the 2026-08-01 conformance audit named as its blocker C (docs/audit/d139-conformance-dc0-20260801.txt:44-56), so blocker C is CLOSED in the repo copy.
  • It has renderer provenance. bash tests/render-drift/run-tests.sh -> 4/4 PASS, naming the file: render/values/vr1-dc0-vips.yaml renders overlays/vr1-dc0-vips.yaml byte-for-byte, and the suite carries a PROOF-OF-TEETH case (a mutated values file is detected). The values file declares output: overlays/vr1-dc0-vips.yaml at line 2.

  • Repo digest confirmed 80d861560a6b3c52, matching what the close recorded.

  • Changed: nothing. Read-only measurement.

  • Revert: n/a.

Item 5 -- sweep F1 CLOSED: the dc0 rack's deploy input is re-staged

A read-only agent enumerated the WHOLE staging directory first, rather than diffing the one file the close bookend named. Capture: docs/audit/repo-stage-drift-dc0-20260802.txt (427 lines). Classification of the 14 regular files + 4 dirs under ~/repo-stage (.git re-confirmed absent):

  • (A) diverged: 1 -- overlays/vr1-dc0-vips.yaml only. Corroboration the bookend did not have: voffice1's clone also holds 80d86156..., so the rack was the ONLY host still carrying the pre-GUA overlay.
  • (B) missing from the Step-4 deploy closure: 0. This was the open question, and it matters because a prior deploy died on a missing policies/overrides.zip. That file IS staged, at the repo digest 02fe1fd7...e720a011, and resolves because policies/ sits beside bundle.yaml. All three scannable merge inputs were checked, not just the bundle: bundle.yaml has exactly one local reference (:227-228); neither the VIP nor the machines overlay has a resources: block or any local path.
  • (C) rack-only orphans: 0. All 14 staged files resolve to a repo path.
  • Two later-step overlays are unstaged (dc-dc-ipv6-family-matrix.yaml, dc-ha-scaleup.yaml); not Step-4 inputs, recorded as a fork, not ruled.

THE MUTATION (gated, single file, executed 2026-08-02). scp -p overlays/vr1-dc0-vips.yaml vr1-dc0-rack:/home/jessea123/repo-stage/overlays/vr1-dc0-vips.yaml

A single-file copy was the minimal correct action BECAUSE the enumeration proved exactly one file diverged. A wholesale directory refresh was rejected for the same reason the 2026-07-31 re-stage rejected it: it risks clobbering or re-permissioning the 0600 vr1-dc0-octavia-pki.yaml (SEC-029).

Before / after, measured by this session on both ends:

                                    BEFORE            AFTER             REPO
overlays/vr1-dc0-vips.yaml          ed19d989e80da8da  80d861560a6b3c52  80d861560a6b3c52
bundle.yaml                         4c8a78527a3100f6  (untouched)       4c8a78527a3100f6
overlays/vr1-dc0-machines.yaml      e3be85e41b419486  (untouched)       e3be85e41b419486
overlays/vr1-dc0-octavia-pki.yaml   5fc117f188332042  5fc117f188332042  (gitignored)

POST-COPY, ALL FOURTEEN STAGED FILES WERE RE-VERIFIED AGAINST CURRENT HEAD, not just the four in the table -- because the staged scripts/ copies were last checked at an EARLIER commit and the 2026-08-02 session shipped script changes after that check. All 13 tracked files match the repo digest-for-digest (bundle.yaml, both non-PKI overlays, policies/domain-manager-policy.yaml, policies/overrides.zip, and the eight staged scripts/), and the 14th matches voffice1. The staging directory is now fully in sync, confirmed against HEAD rather than inherited from the prior verification.

Checked and NOT a gap: dc-egress-check.sh is absent from ~/repo-stage by design -- it is streamed over stdin (ssh <rack> 'bash -s -- check <site>' < scripts/dc-egress-check.sh, per its own refuse message at :90-91), so it is never staged and cannot go stale there.

The PKI overlay is verified UNCHANGED -- same digest, same 0600, same mtime (Jul 31 02:38) -- and was never read, only hashed. It measures IDENTICAL to voffice1's copy, so the SEC-029 superseded-cert hazard is not realized today; nothing enforces the dual update on the next reissue, which stays an open SEC-029 obligation.

  • Revert: copy the pre-GUA overlay back. It is NOT retained anywhere as a file -- reverting means re-rendering from render/values/ at the parent of the GUA commit. There is no reason to: the staged copy now matches repo HEAD, which is the invariant.

TWO FINDINGS THE AGENT SURFACED THAT WERE NOT ASKED FOR, both logged not fixed:

  1. The staged inventory is 14 files, not the nine that docs/security-ledger.md SEC-029(3) records. The +5 are opnsense-* files left by the 2026-08-02 edge rebuild (digests match dc0-edge-rebuild-20260802.txt:259-264). A future re-stage planned off the "nine-file copy" figure would omit five files. The ledger's prose is stale, not wrong-in-kind.
  2. A GATE/DEPLOY ASYMMETRY, DOCFIX-shaped and worth attention before Step 4. overlays/vr1-dc0-machines.yaml exists and scripts/preflight.sh:144-145 folds it into the merged input that P2 validates -- but runbooks/dc-dc-phase4-juju-bundle-per-dc.md:527-528 asserts the file does NOT exist and Step 4 does not pass it. So P2 validates a different artifact than the deploy consumes. That is the class of defect this project has repeatedly found (a gate grading the workspace while the overlay is what reaches the charm). NOT fixed here -- hard rule 1; it likely bundles with the outstanding D-138 run-location DOCFIX.

Item 6 -- sweep F6 CLOSED: the dc0 MAAS region DB is proven uncorrupted

PASS. pg_dump read every page of every table in maasdb: 23,878,796 bytes / 37,199 lines / exit 0 / empty stderr, with -- PostgreSQL database dump complete asserted in a separate pass. A torn or bad-checksum page anywhere would have aborted the dump, so the 2026-08-01 hard power cut left NO page-level corruption. Evidence appended to docs/audit/maasdb-pgdump-integrity-dc0-20260802.txt (the agent's BLOCKED report is left verbatim above it; sections 7-11 are the resolution).

F6's stated blocker was the wrong diagnosis, and this is the durable part. The finding attributed five prior failures to snap confinement and prescribed a login shell. Neither was operative:

  • Confinement did NOT reproduce as ubuntu -- the postgresql.dump wrapper ran, opened the socket, and the server answered. (It DOES reproduce as _daemon_: cannot create snap home dir: mkdir /nonexistent + is not a snap cgroup. So the original reading was true for the user it was measured on and got generalised too far.)
  • The real discriminators were ROLE and TRANSPORT. Role ubuntu does not exist; the role is maas. And over TCP (-h localhost) role maas is password-challenged (fe_sendauth: no password supplied), which would have meant handling the regiond.conf secret -- while over the UNIX SOCKET (-h /tmp) it authenticates with no credential at all. Dump over the socket; never over TCP.

Both identity values are now MEASURED, and one of them was prose all along. maasdb came from F6's text and had never been verified; sudo grep -E "^database_name|^database_user|..." on regiond.conf confirms maasdb / maas / localhost. Only non-secret field names were grepped -- database_pass was never read, and the dump path chosen above means it never needed to be.

The dump was STREAMED through awk and nothing was persisted: the MAAS DB carries API keys and password hashes, so writing it to disk would have created a new exposure needing its own SEC row.

Scope, stated so it is not overclaimed: this proves the pages are readable and the rows decode. It does NOT prove MAAS's application-level invariants hold. Nothing observed suggests otherwise and no such claim is made.

The agent's false-green finding, confirmed and worth keeping: SSH_EXIT was 0 while pg_dump exited 1 having produced 0 bytes -- so F6's own proposed form, ssh host 'pg_dump maasdb > /dev/null' && echo PASS, would have printed PASS over a dump that never happened. Same family as sweep F3 and sweep F4. Asserting on BYTES plus the completion marker is what makes this a gate rather than a ritual.

  • Changed: docs/audit/maasdb-pgdump-integrity-dc0-20260802.txt (new capture).
  • Revert: n/a -- a read-only measurement. Nothing on the VM was altered.

Item 7 -- sweep F2 DIAGNOSED: reading R1 is true, R2 is refuted

Sweep F2 left two readings open and forbade assuming the first. A read-only agent settled it. Capture: docs/audit/mirror-exitcode-diagnosis-dc0-20260802.txt.

VERDICT: R1 -- debmirror itself prints All done. and then exits non-zero. Nothing runs after All done. that could fail. R2 (something failing after the message) has no candidate step.

Decisive evidence -- the journal shows All done. TWICE, once per debmirror leg, and the failing one is followed 10 ms later by its own error block in the SAME PID:

08:28:00.830168 dc0-mirror-sync[276574]: All done.
08:28:00.840132 dc0-mirror-sync[276574]: Errors:
08:28:00.840132 dc0-mirror-sync[276574]:  Download of dists/jammy-backports/main/dep11/Components-amd64.yml.gz failed: 500 read timeout
08:28:00.840132 dc0-mirror-sync[276574]: Failed to download files (1 errors)!

Confirmed at VENDOR SOURCE, and re-verified independently by this session (not taken on the agent's word, since it is the load-bearing claim) -- debmirror 1:2.39ubuntu2 on the dc0 rack:

1615:say("All done.");
1620:  exit 1 if (!$ignore_small_errors);

The helper does not pass --ignore-small-errors, so any non-zero error tally exits 1 AFTER the success-looking message. Corroborating: between the LAST All done. (the UCA leg, 08:28:03.833394) and systemd's failure line (08:28:03.840126) there are 6.7 ms and ZERO log lines. And the installed helper is byte-identical to the repo's generated content -- no drift.

Proximate cause of the 1: exactly one upstream download, jammy-backports/main/dep11/Components-amd64.yml.gz, hit a 500 read timeout at 300s.

THE SWEEP'S OWN PREMISE WAS WRONG, AND THE ERROR IS INSTRUCTIVE. F2 states "its own log says the sync succeeded". Every line it quoted (Files to download: 0 B, Downloaded 15 kiB in 3s, Everything OK..., All done.) is from the SECOND debmirror, PID 277623 -- the UCA leg, which SUCCEEDED. The failing ubuntu leg reported Downloaded 2867 kiB in 313s and an error block. The tail of the passing leg was read as the tail of the whole run, and the evidence for R1 sat three lines below the last line quoted. Same family as this project's standing instrument lessons: a truncated read that looks exactly like a complete one.

A SECOND CORRECTION: "dc-mirror.sh check dc0 stays RED" OVERSTATES IT -- the gate is FLAKY, not stuck. The agent reported "7 of 14"; that did not reconcile against the journal, so it was re-counted. MEASURED, full journal window 2026-07-23..08-02: 16 runs, 7 Finished, 9 Failed with result 'exit-code'. The success count was right, the denominator was not. Its "four successes in the last four days" IS correct.

My own first re-count was ALSO wrong, and it is logged rather than quietly dropped: a grep -cE "Starting .*mirror sync" returned 0 because the unit's description reads dc0 mirror upstream sync, so "mirror sync" never matched. It looked like "no runs recorded". Checking the instrument before believing a surprising negative is the standing rule and it paid again.

The stronger evidence is the SHAPE, not the ratio. On FOUR separate days a failed run was followed within hours by a clean one:

2026-07-25T06:20:23 Failed  ->  2026-07-25T06:23:59 Finished
2026-07-27T00:54:54 Failed  ->  2026-07-27T08:43:46 Finished
2026-07-31T00:54:12 Failed  ->  2026-07-31T02:35:23 Finished
2026-07-31T18:29:37 Failed  ->  2026-07-31T18:56:20 Finished

That is transience DEMONSTRATED, not inferred from an error string. Two caveats kept: the earliest failure (2026-07-23) was the HOME-under-systemd defect fixed the same hour, so the lifetime ratio understates current health; and the two most recent runs, both 2026-08-02, BOTH failed -- they bracket the edge outage, and the last success was 2026-08-01T00:04:19.

Failure shapes are TWO, flagged so they are not later conflated: 4 runs carry a Failed to download files (N errors)! tally (the R1 path, N = 1/3/22/1); 3 carry Errors: with no tally, meaning debmirror DIED rather than completing -- the ubuntu=255 shape dc-mirror.sh's own comment records for 2026-07-27.

NO REMEDY APPLIED, and the decision is the operator's (hard rule 1). The agent recorded only the SHAPE of the choice, which is the part that matters: debmirror's exit code conflates "nothing mirrored" with "mirrored minus N transient files", so ANY tolerance change alters what the gate ATTESTS. Relaxing it is the 2026-07-27 false-green defect inverted. Options are put to the operator separately.

Declared gaps, not silently filled: whether the missing .gz actually breaks a node's apt-get update (the .xz sibling is present; apt was NOT run from a node); whether tree-wide completeness holds (the tree was not reconciled against the Release indices, and the agent's find was truncated -- so "at least one file missing", never "exactly one"); and whether the 500 timeouts are archive.ubuntu.com or the edge path (a curl -I 200 shows reachability only -- HEAD never exercises the body read that timed out, so it is NOT evidence of transience).

  • Changed: docs/audit/mirror-exitcode-diagnosis-dc0-20260802.txt (new capture). scripts/dc-mirror.sh deliberately UNTOUCHED.
  • Revert: n/a -- diagnosis only.

Item 8 -- sweep F2, the re-trigger: the failure is PERSISTENT, and it does not matter

OPERATOR RULING (operational, not a D-number per GA-R3 -- no architectural consequence). Question put: what should dc-mirror.sh's gate attest, given debmirror's exit code cannot distinguish "nothing mirrored" from "mirrored minus N transient files" from "debmirror died"? Options offered: record shape+count without relaxing / leave as is / re-trigger first and decide after / add a bounded tolerance (listed as NOT recommended). Operator answer, exact utterance: "Re-trigger the sync first, decide after". The re-trigger was therefore run as a MEASUREMENT, not as a remedy, and the attestation question stays OPEN.

RESULT: THE FAILURE IS NOT TRANSIENT. The re-run (10:26:05 -> 10:31:20Z) failed identically -- FAIL ... ubuntu=1 uca=0, Result=exit-code, ExecMainStatus=1 -- on the SAME file with the SAME error and the SAME 300s timeout, two hours after the previous attempt:

08:27:56  Download of dists/jammy-backports/main/dep11/Components-amd64.yml.gz failed: 500 read timeout (300s)
10:31:16  Download of dists/jammy-backports/main/dep11/Components-amd64.yml.gz failed: 500 read timeout

So the earlier "flaky, not stuck" reading -- which the fail-then-succeed pairs supported -- does NOT extend to THIS failure. Two distinct things were being conflated: the unit has a history of transient failures, AND it now has one persistent one. The ruling's "decide after" is answered: waiting will not clear it.

THE SCOPE OF THE GAP IS NOW EXACT, replacing the agent's "at least one file missing". debmirror's own Files to download: 6349 B equals the size of that single file, so the entire outstanding delta of the 952 GiB tree IS this one file. On disk:

present:  Components-amd64.yml.xz   5748 B   (Release lists 5748 -- size-correct)
ABSENT:   Components-amd64.yml.gz   6349 B   (Release lists 6349)

AND IT DOES NOT BREAK apt -- MEASURED, not reasoned. The prior agent explicitly declared this unmeasured and declined to reason it away, which was correct. Tested from the rack with a fully ISOLATED apt state dir (scratch Dir::State::Lists, Dir::Cache, Dir::Etc::SourceList; nothing system-wide touched, no package installed):

deb http://10.12.8.4/ubuntu jammy-backports main
  Get:1 InRelease [127 kB]
  Get:2 main amd64 Packages [70.2 kB]
  Get:3 main amd64 Components [5748 B]      <-- the .xz, the format that IS present
  Get:4 main amd64 c-n-f Metadata [412 B]
  APT_UPDATE_RC=0
  apt-cache policy -> release v=22.04,o=Ubuntu,a=jammy-backports,c=main,b=amd64

apt asks for the .xz and never asks for the .gz. So the mirror is FUNCTIONALLY complete for the deploy and the gate is red over a file no consumer requests. jammy-backports was chosen as the test suite deliberately -- it is the one carrying the missing file.

ONE ANOMALY, UNEXPLAINED AND LEFT THAT WAY rather than hand-waved. The file fetches FINE by curl from the same host, right now: http=200 size=6349 time=0.436s, with a real body read (HEAD would prove nothing). Its .xz sibling and a jammy control also return 200. So curl succeeds in 0.44s where debmirror times out after 300s, twice. Ruled OUT: any proxy (the unit's environment is HOME=/root only; no http_proxy in /etc/environment, apt.conf.d, profile.d, or root's login shell). NOT chased further -- it does not gate the deploy, and guessing at an LWP-vs-curl difference would be exactly the reasoning-instead- of-measuring this project keeps logging.

  • Changed: nothing in the repo. scripts/dc-mirror.sh still UNTOUCHED; no tolerance was relaxed and no assertion weakened.
  • State changed on the rack: one sync run, which wrote last-sync.status and fetched ~1.3 MB of indices. The apt test wrote only to a scratch dir, removed on exit.
  • Revert: n/a.