| 2026-07-29 |

Four gate-integrity defects fixed, each demonstrated RED then GREEN
...
The rule is that a checker which cannot fail is not a gate -- and its inverse, that a gate only
ever seen failing is equally untrustworthy. So each item carries both a fixture proving it goes
red on the defect and a run proving it goes green on the real tree.
4.2 DECORATIVE HA. cluster_count was asserted nowhere against 20 occurrences in
dc-ha-scaleup.yaml, so rewriting all 20 from 3 to 1 produced a byte-identical PASS. Verified
independently here rather than from reported output: the same rewrite now yields 13 DECORATIVE
HA failures, and the real overlay still passes. Missing cluster_count and the scale-down
direction fail too.
4.3 The machines overlay can no longer be a silent no-op -- missing, gutted and mis-keyed each
fail with distinct messages, the mis-keyed one under either --dc. A bypass was caught on review
before delivery: the check first sat inside the role_sep branch, so an overlay stripping every
tags= made the whole placement block self-skip behind an [ok]. Hoisted out and locked by a case.
4.7 cloud-assert gained an arity check covering 14/14 scaled apps, discovered from status JSON
and measured against a real captured juju-status.json. It refuses on an unreadable value rather
than passing, and single-unit ovn-central is no longer "uniform".
4.8 G17 has a real node-side check: dc-cache-proxy.sh node <site> content-asserts proxied
archive and UCA Release fetches, carries R12's chronyc time-source assertion, and adds exit
3 = REFUSE. Stated honestly: its green is a FIXTURE green -- no node has run it, the nodes are
powered off, and G17's window is Stage-5 first boot. G17 stays OPEN.
THE GAUNTLET NOW AGREES ON BOTH HOSTS. The voffice1 "drift" was pure locale collation; LC_ALL=C
now applies to the sort and to both sides of the compare, deliberately without re-recording
HARNESS-MANIFEST, which would have moved the failure to vcloud. The snap test takes a pre-state
reading instead of asserting snap-absence. Verified as final arbiter: ALL GREEN (88) in the
default locale AND under LC_ALL=C, identical.
AN AGENT CONCLUSION CORRECTED RATHER THAN ACTED ON: it logged that preflight P2 should also
assemble dc-ha-scaleup.yaml. Measured, that is wrong -- the overlay is deliberately excluded
from the Step 4 deploy per R6, and no step anywhere applies it yet. Adding it to P2 would make
the gate validate an input the deploy does not use, exactly the chain-audit defect in reverse.
Logged, not executed.
Disclosed deviation, verified harmless: the agent used git stash despite that being forbidden.
Checked because other agents were writing concurrently -- stash list empty, prior commit intact,
D-133 guard still holding, modified set exactly its declared files.
repo-lint 0 fail / 618 files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-27 |

close-out durability sweep: 4 transcript-only items landed; dc1-is-not-a-mirror warning
...
Operator-requested sweep for content that existed only in the session transcript, before
clearing. Precedent: docs/audit/queued-findings-20260726.txt. Capture:
docs/audit/queued-findings-20260727.txt.
THE CONSEQUENTIAL ONE. scripts/dc-mirror.sh's dc1 site row carried NO warning that dc1 is no
longer a mirror site, so `dc-mirror.sh install dc1` would have SILENTLY REBUILT everything the
same-day teardown removed -- units, nginx vhost, sync helper, and an ENABLED daily debmirror
timer that would start a fresh ~950G pull on a rack whose ruled artifact path is the proxy. That
is precisely the command someone would reach for after treating a failing `check dc1` as a
regression. The row now states dc1 is proxy-only per the D-135 amendment, that `check dc1` FAILS
BY DESIGN, and that `install dc1` is a deliberate strategy change and never a repair. The row is
RETAINED so its measured values survive a legitimate rebuild.
QUEUED, not built (hard rule 1): a RUNTIME guard in do_install for a site whose ruled path is
not the mirror. A comment is strictly weaker than a guard, and prose-only prevention has already
failed twice on this project (DoD item 8 missed dc1; D-137's wiring sat as prose and missed BOTH
DC standups). Recording that the interim is an interim.
A WRONG CLAIM CORRECTED. docs/audit/stage4-mirror-gate-20260727.txt said `systemctl reset-failed`
would re-arm the paused dc1 debmirror. It cannot: reset-failed on the SERVICE does not start an
INACTIVE TIMER. The real vector was a REBOOT -- `enabled` starts the timer at boot and
`Persistent=yes` fires it IMMEDIATELY because its window had passed. Correction APPENDED rather
than edited in place: the file is dated evidence, so a wrong analytical claim is superseded
openly, not quietly rewritten.
THE GENERALISABLE LESSON, previously unrecorded, is now platform-traps section 5:
5a "stopped is not dormant across a reboot" -- enabled + inactive + EMPTY NextElapse looks
harmless and is not; plus reset-failed does not re-arm, and Result=signal/status=15 on a
oneshot means it was SIGTERM'd and never wrote a terminal state (so a stale RUNNING is a
corpse, not progress).
5b a `oneshot` with RemainAfterExit=yes does NOT undo its work on stop, so stopping it proves
NOTHING about dependents -- which is exactly why the dc1 teardown test deleted the live
address and route instead of stopping the mirror net unit.
Three rows added to the verbatim-error index. Note dc0-mirror-sync.timer is still enabled with
Persistent=yes, so 5a describes dc0 today -- benign while its syncs succeed, same shape.
HARD RULE 2. dc-cache-proxy.sh stated that the two net units coexist safely on one host as
FACT. It is REASONED (idempotent replace verbs, identical measured values), never MEASURED --
the proxy has only run on dc1, where the mirror unit was then removed, so nobody has run both.
Marked accordingly, with an instruction to verify rather than trust the sentence.
VERIFICATION: dangling-reference sweep over docs/, runbooks/, scripts/ and the skill -- every
path this session introduced or cited RESOLVES; all pre-existing dangles are legitimate
(deleted-as-history, not-yet-built, or the deliberately-absent octavia PKI overlay preflight P4
fails on). ledger-scan reconciled against the machine-derived block. Gauntlet ALL GREEN (81),
repo-lint 0-fail, dc-mirror 24/24, dc-cache-proxy 20/20.
Revert: text + comments only; the capture correction is an append.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

dc1 full mirror REMOVED (D-135 amendment); dc-cache-proxy.sh now owns its net layer
...
Operator clarified intent, exact utterance: "DC0 is the test of a full mirror. DC1 is the test
of the mirror proxy. DC1 needed the partially created full mirror and associated
script/files/settings removed." Recorded as a D-135 AMENDMENT 2026-07-27 -- the per-DC split
is the DELIBERATE EXPERIMENT, not an interim state. That supersedes both the "PAUSED ... kept
dormant as the fallback" record and the "DC1's proxy is REPLACED by a full mirror afterward"
claim, which sat in THREE places (D-135, CURRENT-STATE:109, dc-cache-proxy.sh:6); all corrected.
dc0 FIRST (gated): sync re-run -> OK 2026-07-27T08:43:46Z ubuntu=0 uca=0 in 20s, confirming
the transient-500 diagnosis. The fixed check now reports PASS -- trustworthy precisely because
the same check FAILED on the same rack twenty minutes earlier. Stage 4 DoD bullet 5 is now MET
for both DCs (dc0 mirror PASS, dc1 proxy PASS), node-side half at G17.
WHY THE ORDER MATTERED. Asked to weigh rip-down-and-rebuild against surgical removal, the
rebuild path turned out to be BLOCKED, not merely slower: dc-cache-proxy.sh did not own its
utility net layer -- it REUSED dc-mirror.sh's <site>-mirror-net.service and refused to install
without it. So on dc1, whose ruled path is the PROXY, three files named dc1-mirror-* (net unit,
apply helper, resolved DNS drop-in) were load-bearing FOR THE PROXY. Consequences: the recorded
removal procedure would have silently killed the proxy, the bullet-5 PASS and the Stage-5 dc1
apt path; and the proxy could not be REBUILT afterward, since nothing but `dc-mirror.sh
install` created the net layer -- rebuilding a PROXY meant reinstalling the whole MIRROR
apparatus, enabled daily debmirror timer included. Operator direction: net layer first, then
rip it all down.
1. NET LAYER MOVED: <site>-cache-proxy-net.service + -net-apply + its own resolved drop-in,
reusing dc-mirror's proven mechanism (bridge from the libvirt net NAME at runtime, hard
rule 3). Proxy-named so a proxy-only DC has NO mirror-named dependency. Idempotent
replace verbs, so both net units coexist safely where a site runs both -- dc0 unaffected.
check asserts the ARTIFACTS, not just the live address/route: asserting only the live
effect would PASS while the layer was really the mirror's, which is the exact failure.
Harness 20/20 (was 15). T12 REPLACED not deleted -- it asserted the superseded design, so
it failed by design; now inverted, plus T12b-T12f. T12b is scoped to non-comment lines on
purpose: the header's account of the removed trap cites dc-mirror.sh install, and deleting
that history to satisfy a grep would lose why the trap existed.
2. INDEPENDENCE PROVEN BEFORE DELETING: a oneshot with RemainAfterExit=yes does not undo its
work on stop, so stopping the mirror unit would have proven nothing. Removed the live state
outright (addr count -> 0, default route count -> 0), applied the proxy-owned unit ALONE,
both restored; check dc1 PASS with dc1-mirror-net.service disabled.
3. WHOLESALE REMOVAL: all mirror units, timer, sync helper, nginx vhost, resolved drop-in, the
debmirror package, and 330G of partial mirror (33,037 files, rm -rf 47.6s; disk 339G ->
9.0G). Zero mirror-named residue, every proxy artifact present, proxy PASS throughout and
never down. Capture docs/audit/dc1-mirror-teardown-20260727.txt.
nginx left installed deliberately: a mirror prereq, but with the vhost gone it is a generic
server on its stock default vhost; removing the metapackage while nginx-core runs is a
half-measure and a full purge is scope creep on a rack host.
Gauntlet ALL GREEN (81), repo-lint 0-fail, dc-cache-proxy 20/20, dc-mirror 24/24.
Revert: `dc-mirror.sh install dc1` (needs debmirror reinstalled) then `sync dc1` to re-pull
~950G; git revert dc-cache-proxy.sh + harness to restore the mirror-owned layer. The 330G
partial is NOT recoverable -- deliberately, per the amendment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-24 |

D-135 amendment: DC1 interim apt caching proxy (dc-cache-proxy.sh)
...
Operator-ruled 2026-07-24 (GA-R5, utterance quoted in D-135): DC1 gets an
interim apt caching proxy (apt-cacher-ng) to unblock its build without waiting
on the full ~950 GiB debmirror; DC0 stays the full-mirror deployment under test;
DC1 proxy is replaced by a full mirror afterward. Both artifact-delivery models
thus get exercised this rehearsal (Roosevelt input, D-132/D-133).
- scripts/dc-cache-proxy.sh: site-keyed check/install; apt-cacher-ng proxy mode
at the D-134 utility .4:3142; reuses the <site>-mirror-net layer (fails loud if
absent); caches archive/security/UCA, consumed via juju apt-http-proxy.
- tests/dc-cache-proxy/run-tests.sh: 15 static cases, 15/15 PASS.
- D-135 AMENDED (design-decisions.md) + CURRENT-STATE Stage-4 DC0/DC1 split.
- DC1 debmirror PAUSED at ~330 GiB (kept dormant as fallback); measured that the
vcloud uplink was NOT the shared bottleneck (pausing dc1 did not speed dc0).
Feasibility recon'd read-only first (apt-cacher-ng installable, upstreams 200 via
edge, port free). Live install on the DC1 rack is the separately-gated mutation.
repo-lint 0 fail; gauntlet ALL GREEN (78).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUjEb7onHpdqFHUnio6iVw
|