SESSION SWEEP 2026-07-31 -- Stage 5 dc0 deploy: prefer-ipv6 research through teardown
=====================================================================================
Method (ruled 2026-07-31): the session was read back in full, every candidate
enumerated, then each GREPPED against the repo. Items marked FIRST SURFACE had no hit
and would have been lost on a clear. FIRST SURFACE items lead the file.
A phrase-exact grep produced false negatives on items whose CONTENT is recorded under
different wording; each candidate was re-grepped on distinctive TOKENS before being
classified. That correction is itself the instrument-error class this repo keeps hitting.
--------------------------------------------------------------------------------------
FIRST SURFACE -- would have been lost
--------------------------------------------------------------------------------------
F1. MAAS's OWN SQUID ALREADY CONNECT-PROXIES api.snapcraft.io, AND IT WOULD UNBLOCK
THE DEPLOY TODAY WITH NOTHING BUILT.
Measured from the rack, the controller VM AND one of the failing containers.
Machines already use 10.12.8.6:8000 for apt via 90curtin-aptproxy. So
`snap-https-proxy=http://10.12.8.6:8000` is a live, working option.
**RULED AGAINST 2026-07-31 (GA-R5).** Operator utterance, verbatim: "No, the
downsides are real and the upsides for stability are more important then to just
breeze past to keep the deployment going. We need to complete the proxy work and
use the one that will not be fragile." [sic]
The downsides that decided it, all measured or structural: MAAS's squid applies NO
destination restriction, so pointing nodes at it hands the node planes general
HTTPS egress and makes D-107's "nodes reach NO internet directly" true in letter
and hollow in practice; its config is MAAS-generated under a per-revision
/var/snap/maas/current/ path, so ACLs cannot be added and a snap refresh can change
it with no alarm (the SEC-012 power-key fragility class); and making it load-bearing
for snaps is the D-135 hidden-coupling shape exactly, while D-132's per-DC region
work is already touching MAAS.
RECORDED BECAUSE IT WILL BE REDISCOVERED: a future session WILL notice the MAAS
proxy works and reach for it. This is the record that it was found, measured, and
deliberately declined.
F2. THE FAILING APPS ARE LXD CONTAINERS AND THEY SOURCE FROM METAL-ADMIN, NOT THE
ADDRESS `juju status` DISPLAYS.
`juju status` prints mysql-innodb-cluster/0 at 10.12.12.116. Measured INSIDE the
container: eth0 10.12.8.122/22, eth1 10.12.12.116/22, `ip route get 10.12.8.4` ->
src 10.12.8.122, NO default route. An ACL built on the displayed address would have
denied every client the proxy exists for. Generalisable: for a juju LXD container,
the displayed address is not necessarily the source address for a given destination.
F3. PORTS 3128 AND 8000 ARE ALREADY HELD, WILDCARD-BOUND, BY MAAS's OWN SQUID ON BOTH
RACKS. noble's squid.conf pins `http_port 3128`, /etc/default/squid is EMPTY, and
conf.d loads before `deny all` but cannot unset an http_port -- so the PACKAGED unit
can only FATAL on these hosts. Any future rack-side proxy must use a dedicated
instance on another port. This is why the build uses 3129.
F4. FOUR REAL BUGS IN scripts/dc-snap-proxy.sh, FOUND BY INDEPENDENT REVIEW, LOGGED NOT
FIXED (hard rule 1 -- the file was frozen for review):
BUG-1 dead `acl snap_probe src 127.0.0.1/32`: squid binds ${LISTEN}:3129 only, so
nothing can arrive on loopback. The probes are actually admitted by
`snap_clients` because LISTEN is inside CLIENT_CIDR, and the comment
misdescribes its own mechanism. Hazard: anyone narrowing CLIENT_CIDR to
exclude .4 breaks `check`, and the rule they would reach for would not help.
BUG-2 `| grep -q` under `set -o pipefail` -- INHERITED, present identically in
dc-mirror.sh:278,281 and dc-cache-proxy.sh:234,252,255. A THREE-SCRIPT SWEEP,
not a local fix. Direction is fail-closed (false MISS, never false OK).
BUG-3 the deny probe's `403` shape was measured from a PORT deny, never a DSTDOMAIN
deny. If squid answers differently the gate REFUSES PERMANENTLY -- and under
GA-R6 a gate that cannot PASS blocks a stage close as hard as one that cannot
fail. Requires a ONE-TIME apply-time confirmation before `check` is cited.
BUG-4 `dpkg -s squid` exits 0 for `Status: deinstall ok config-files`, so a purged
package reports as present. The existence-vs-content class again.
`[ -x /usr/sbin/squid ]` is shorter and strictly stronger.
F5. TEARDOWN INSTRUMENT ERROR: `juju models` SUMMARY COLUMNS read 0 machines / 0 units
while `juju status -m vr1-dc0` simultaneously read 26 machines / 37 applications.
The summary zeroes during `destroying` and is NOT a progress signal. I reported it
as progress. Standing lesson: during a teardown, `juju status -m <model>` is the
instrument; `juju models` counts are not.
F6. WHY THE TEARDOWN STALLED, AND WHY IT WAS TERMINAL: all 26 machine/container agents
were `stopped`, so NO hook could execute at all -- the destroy worker asks and
nothing answers. It is a fixed point, not backoff, and would have retried forever
(it reached attempt 30, flat ~19 minutes, app set byte-identical across a 12-minute
diff). Units already in `error` cannot run their teardown hooks. `--force` cleared
it and stranded NOTHING -- all nine read back Ready/owner=None, no manual MAAS
release was needed.
F7. A MAAS HOSTNAME RENAME ON A `Deployed` MACHINE IS RECORD-ONLY. Measured: MAAS
accepted it (fqdn updated), while `juju status` and the running OS both kept the old
name. MAAS applies hostname at DEPLOY time via cloud-init and has no agent
reconciling a running host; juju captured the name at provisioning. NOTE THE
ASYMMETRY, which is the dangerous part: MAAS REFUSES interface changes on a Deployed
machine (loud) but ACCEPTS a hostname change (silent, record-only).
STILL UNVERIFIED: that a redeploy applies the new hostname. Post-deploy, check all
three -- MAAS record, `juju status`, and `hostname` on the node -- agree.
F8. STORAGE-NODE IDENTITY WOULD HAVE BEEN MISLABELLED BY MACHINE ORDER. Resolved by
pinned boot MAC: `amused-corgi` is storage-01, not storage-03. Ordering by juju
machine number would have mislabelled three of the four storage nodes.
F9. `ovn-central`'s charm default is `source: zed`, NOT caracal -- it points at a UCA
pocket the dc0 mirror does not carry. Repointing changes its RELEASE, not its URL,
so it is a separate question. LOGGED, untouched.
F10. GITIGNORED STATE: no new `.claude/settings.local.json` rules were added this
session. The 2026-07-30 sweep's F1 record of the existing rules stands unchanged.
F11. AS-EXECUTED LOG: NOT USED for this window. `run-logged.sh` opens an interactive
`script(1)` subshell, unusable from a non-interactive session. Every action is in
the commits with read-backs, and captures went to docs/audit/*. **This window is
therefore NOT covered by an as-executed log, and must not be read as if it were.**
--------------------------------------------------------------------------------------
ALREADY ON SURFACE (verified by grep; recorded for completeness)
--------------------------------------------------------------------------------------
A1. The prefer-ipv6 research and both rulings -- docs/audit/stage5-prefer-ipv6-charm-
research-20260731.txt, D-101 RULING NOTES (a) and (b), CURRENT-STATE.
A2. The D-135 jammy-backports amendment and its ~1.0 GiB sizing -- D-135 AMENDMENT
2026-07-31, CURRENT-STATE, changelog items 16-18.
A3. The UCA-in-DC ruling and its 15-app derived scope -- joint D-135/D-107 RULING NOTE,
overlays/vr1-dc0-machines.yaml header, CURRENT-STATE.
A4. Bundle re-runnability (explicit base on 56 apps) + invariant 12 -- bundle.yaml
header, provider-bundle-check.py, tests, CURRENT-STATE.
A5. The three deploy attempts and their causes -- docs/audit/stage5-dc0-deploy-
attempt{1,2,3}-20260731.txt.
A6. The snap proxy design, measurements and mutation results -- docs/audit/stage5-snap-
proxy-measurements-20260731.txt (549 lines), changelog item 19.
A7. Per-DC Tailscale as a standing standup requirement -- workflow gap register item 21.
A8. ceph-rbd-mirror pinned with options + recommendation -- gap register item 22.
A9. model-defaults durability + the owed dc-model-defaults.sh -- CURRENT-STATE.
A10. The commit-contamination incident (e57ad09 swept agent files via `git add -A`) --
changelog item 19 and commit 364ff12's message.
--------------------------------------------------------------------------------------
CONTRADICTION DETECTOR -- measurement vs standing document
--------------------------------------------------------------------------------------
C1. preflight P6's reminder text still quotes "50 apps / 97 relations" against a
MEASURED 56 apps / 108 relations. Recorded on 2026-07-31 in CURRENT-STATE; the
stale string in scripts/preflight.sh is STILL THERE. DOCFIX owed.
C2. G17's own text names `chronyc sources`, and chrony is NOT INSTALLED on the MAAS
jammy image. Recorded 2026-07-30; still unfixed. DOCFIX owed.
C3. The phase-4 runbook's dc0 deploy block (:553-557) omits overlays/vr1-dc0-machines
.yaml, which is NOT a no-op -- it carries ovn-chassis bridge-interface-mappings.
Recorded; DOCFIX owed.
--------------------------------------------------------------------------------------
DELIBERATELY NOT DONE
--------------------------------------------------------------------------------------
N1. The snap proxy is NOT installed and NOTHING is proven end to end. No snap has been
fetched through it. The harness green is FIXTURE green.
N2. The dc-snap-proxy simplification plan is NOT applied. Applying it REQUIRES re-running
the mutation pass -- S8 invalidates M32-M39, M42-M47, M49-M52 as evidence.
N3. Tailscale: all four sub-decisions deferred to end-of-step by operator direction.
N4. BUG-2 not fixed -- a three-script sweep, out of scope under hard rule 1.
N5. dc-model-defaults.sh not built; only the apt-mirror value is set.
