SESSION SWEEP 2026-07-31 -- Stage 5 dc0 deploy: prefer-ipv6 research through teardown ===================================================================================== Method (ruled 2026-07-31): the session was read back in full, every candidate enumerated, then each GREPPED against the repo. Items marked FIRST SURFACE had no hit and would have been lost on a clear. FIRST SURFACE items lead the file. A phrase-exact grep produced false negatives on items whose CONTENT is recorded under different wording; each candidate was re-grepped on distinctive TOKENS before being classified. That correction is itself the instrument-error class this repo keeps hitting. -------------------------------------------------------------------------------------- FIRST SURFACE -- would have been lost -------------------------------------------------------------------------------------- F1. MAAS's OWN SQUID ALREADY CONNECT-PROXIES api.snapcraft.io, AND IT WOULD UNBLOCK THE DEPLOY TODAY WITH NOTHING BUILT. Measured from the rack, the controller VM AND one of the failing containers. Machines already use 10.12.8.6:8000 for apt via 90curtin-aptproxy. So `snap-https-proxy=http://10.12.8.6:8000` is a live, working option. **RULED AGAINST 2026-07-31 (GA-R5).** Operator utterance, verbatim: "No, the downsides are real and the upsides for stability are more important then to just breeze past to keep the deployment going. We need to complete the proxy work and use the one that will not be fragile." [sic] The downsides that decided it, all measured or structural: MAAS's squid applies NO destination restriction, so pointing nodes at it hands the node planes general HTTPS egress and makes D-107's "nodes reach NO internet directly" true in letter and hollow in practice; its config is MAAS-generated under a per-revision /var/snap/maas/current/ path, so ACLs cannot be added and a snap refresh can change it with no alarm (the SEC-012 power-key fragility class); and making it load-bearing for snaps is the D-135 hidden-coupling shape exactly, while D-132's per-DC region work is already touching MAAS. RECORDED BECAUSE IT WILL BE REDISCOVERED: a future session WILL notice the MAAS proxy works and reach for it. This is the record that it was found, measured, and deliberately declined. F2. THE FAILING APPS ARE LXD CONTAINERS AND THEY SOURCE FROM METAL-ADMIN, NOT THE ADDRESS `juju status` DISPLAYS. `juju status` prints mysql-innodb-cluster/0 at 10.12.12.116. Measured INSIDE the container: eth0 10.12.8.122/22, eth1 10.12.12.116/22, `ip route get 10.12.8.4` -> src 10.12.8.122, NO default route. An ACL built on the displayed address would have denied every client the proxy exists for. Generalisable: for a juju LXD container, the displayed address is not necessarily the source address for a given destination. F3. PORTS 3128 AND 8000 ARE ALREADY HELD, WILDCARD-BOUND, BY MAAS's OWN SQUID ON BOTH RACKS. noble's squid.conf pins `http_port 3128`, /etc/default/squid is EMPTY, and conf.d loads before `deny all` but cannot unset an http_port -- so the PACKAGED unit can only FATAL on these hosts. Any future rack-side proxy must use a dedicated instance on another port. This is why the build uses 3129. F4. FOUR REAL BUGS IN scripts/dc-snap-proxy.sh, FOUND BY INDEPENDENT REVIEW, LOGGED NOT FIXED (hard rule 1 -- the file was frozen for review): BUG-1 dead `acl snap_probe src 127.0.0.1/32`: squid binds ${LISTEN}:3129 only, so nothing can arrive on loopback. The probes are actually admitted by `snap_clients` because LISTEN is inside CLIENT_CIDR, and the comment misdescribes its own mechanism. Hazard: anyone narrowing CLIENT_CIDR to exclude .4 breaks `check`, and the rule they would reach for would not help. BUG-2 `| grep -q` under `set -o pipefail` -- INHERITED, present identically in dc-mirror.sh:278,281 and dc-cache-proxy.sh:234,252,255. A THREE-SCRIPT SWEEP, not a local fix. Direction is fail-closed (false MISS, never false OK). BUG-3 the deny probe's `403` shape was measured from a PORT deny, never a DSTDOMAIN deny. If squid answers differently the gate REFUSES PERMANENTLY -- and under GA-R6 a gate that cannot PASS blocks a stage close as hard as one that cannot fail. Requires a ONE-TIME apply-time confirmation before `check` is cited. BUG-4 `dpkg -s squid` exits 0 for `Status: deinstall ok config-files`, so a purged package reports as present. The existence-vs-content class again. `[ -x /usr/sbin/squid ]` is shorter and strictly stronger. F5. TEARDOWN INSTRUMENT ERROR: `juju models` SUMMARY COLUMNS read 0 machines / 0 units while `juju status -m vr1-dc0` simultaneously read 26 machines / 37 applications. The summary zeroes during `destroying` and is NOT a progress signal. I reported it as progress. Standing lesson: during a teardown, `juju status -m ` is the instrument; `juju models` counts are not. F6. WHY THE TEARDOWN STALLED, AND WHY IT WAS TERMINAL: all 26 machine/container agents were `stopped`, so NO hook could execute at all -- the destroy worker asks and nothing answers. It is a fixed point, not backoff, and would have retried forever (it reached attempt 30, flat ~19 minutes, app set byte-identical across a 12-minute diff). Units already in `error` cannot run their teardown hooks. `--force` cleared it and stranded NOTHING -- all nine read back Ready/owner=None, no manual MAAS release was needed. F7. A MAAS HOSTNAME RENAME ON A `Deployed` MACHINE IS RECORD-ONLY. Measured: MAAS accepted it (fqdn updated), while `juju status` and the running OS both kept the old name. MAAS applies hostname at DEPLOY time via cloud-init and has no agent reconciling a running host; juju captured the name at provisioning. NOTE THE ASYMMETRY, which is the dangerous part: MAAS REFUSES interface changes on a Deployed machine (loud) but ACCEPTS a hostname change (silent, record-only). STILL UNVERIFIED: that a redeploy applies the new hostname. Post-deploy, check all three -- MAAS record, `juju status`, and `hostname` on the node -- agree. F8. STORAGE-NODE IDENTITY WOULD HAVE BEEN MISLABELLED BY MACHINE ORDER. Resolved by pinned boot MAC: `amused-corgi` is storage-01, not storage-03. Ordering by juju machine number would have mislabelled three of the four storage nodes. F9. `ovn-central`'s charm default is `source: zed`, NOT caracal -- it points at a UCA pocket the dc0 mirror does not carry. Repointing changes its RELEASE, not its URL, so it is a separate question. LOGGED, untouched. F10. GITIGNORED STATE: no new `.claude/settings.local.json` rules were added this session. The 2026-07-30 sweep's F1 record of the existing rules stands unchanged. F11. AS-EXECUTED LOG: NOT USED for this window. `run-logged.sh` opens an interactive `script(1)` subshell, unusable from a non-interactive session. Every action is in the commits with read-backs, and captures went to docs/audit/*. **This window is therefore NOT covered by an as-executed log, and must not be read as if it were.** -------------------------------------------------------------------------------------- ALREADY ON SURFACE (verified by grep; recorded for completeness) -------------------------------------------------------------------------------------- A1. The prefer-ipv6 research and both rulings -- docs/audit/stage5-prefer-ipv6-charm- research-20260731.txt, D-101 RULING NOTES (a) and (b), CURRENT-STATE. A2. The D-135 jammy-backports amendment and its ~1.0 GiB sizing -- D-135 AMENDMENT 2026-07-31, CURRENT-STATE, changelog items 16-18. A3. The UCA-in-DC ruling and its 15-app derived scope -- joint D-135/D-107 RULING NOTE, overlays/vr1-dc0-machines.yaml header, CURRENT-STATE. A4. Bundle re-runnability (explicit base on 56 apps) + invariant 12 -- bundle.yaml header, provider-bundle-check.py, tests, CURRENT-STATE. A5. The three deploy attempts and their causes -- docs/audit/stage5-dc0-deploy- attempt{1,2,3}-20260731.txt. A6. The snap proxy design, measurements and mutation results -- docs/audit/stage5-snap- proxy-measurements-20260731.txt (549 lines), changelog item 19. A7. Per-DC Tailscale as a standing standup requirement -- workflow gap register item 21. A8. ceph-rbd-mirror pinned with options + recommendation -- gap register item 22. A9. model-defaults durability + the owed dc-model-defaults.sh -- CURRENT-STATE. A10. The commit-contamination incident (e57ad09 swept agent files via `git add -A`) -- changelog item 19 and commit 364ff12's message. -------------------------------------------------------------------------------------- CONTRADICTION DETECTOR -- measurement vs standing document -------------------------------------------------------------------------------------- C1. preflight P6's reminder text still quotes "50 apps / 97 relations" against a MEASURED 56 apps / 108 relations. Recorded on 2026-07-31 in CURRENT-STATE; the stale string in scripts/preflight.sh is STILL THERE. DOCFIX owed. C2. G17's own text names `chronyc sources`, and chrony is NOT INSTALLED on the MAAS jammy image. Recorded 2026-07-30; still unfixed. DOCFIX owed. C3. The phase-4 runbook's dc0 deploy block (:553-557) omits overlays/vr1-dc0-machines .yaml, which is NOT a no-op -- it carries ovn-chassis bridge-interface-mappings. Recorded; DOCFIX owed. -------------------------------------------------------------------------------------- DELIBERATELY NOT DONE -------------------------------------------------------------------------------------- N1. The snap proxy is NOT installed and NOTHING is proven end to end. No snap has been fetched through it. The harness green is FIXTURE green. N2. The dc-snap-proxy simplification plan is NOT applied. Applying it REQUIRES re-running the mutation pass -- S8 invalidates M32-M39, M42-M47, M49-M52 as evidence. N3. Tailscale: all four sub-decisions deferred to end-of-step by operator direction. N4. BUG-2 not fixed -- a three-script sweep, out of scope under hard rule 1. N5. dc-model-defaults.sh not built; only the apt-mirror value is set.