stage5-snap-proxy-measurements-20260731.txt =========================================== Read-only measurement capture backing `scripts/dc-snap-proxy.sh` (the REPO SIDE of D-135/D-107 RULING 2, 2026-07-31: "HTTP(S) forward proxy in the DC utility band + juju snap-https-proxy (Recommended)"). SCOPE OF THIS CAPTURE. Every command below is READ-ONLY. NO cloud mutation was made in the session that produced it: no `juju config`, no `juju model-config`, no package installed, no service started or stopped, no file written on any rack or node. The proxy is BUILT IN THE REPO and NOT APPLIED. Two side effects are declared rather than hidden: the `curl` probes traversed MAAS's existing squid and may have added entries to ITS cache, and `apt-get download squid` wrote a .deb into `/tmp` ON THE JUMPHOST (not on any rack) so the shipped squid config could be read instead of guessed. Host/date: vcloud jumphost, 2026-07-31. Branch dc-dc-stage5-preconditions. 1. THE THREE FAILING APPLICATIONS ARE LXD CONTAINERS, AND THEIR SOURCE ADDRESS TOWARD THE RACK IS ON METAL-ADMIN, NOT METAL-INTERNAL ------------------------------------------------------------------------------ This is the measurement the whole design turns on: it decides the squid client ACL and whether the D-134 utility `.4` is reachable by the clients at all. `juju status -m vr1-dc0` shows `mysql-innodb-cluster/0` at `10.12.12.116`, which is NOT a D-134 node octet (.100-.102 / .120-.121 / .150-.153) -- so the failing units are containers, and the address `juju status` prints is only ONE of their legs. Measured from inside the container: $ juju ssh -m vr1-dc0 mysql-innodb-cluster/0 'ip -4 -o addr show; ip route show; ip route get 10.12.8.4' 60: eth0 inet 10.12.8.122/22 brd 10.12.11.255 scope global eth0 62: eth1 inet 10.12.12.116/22 brd 10.12.15.255 scope global eth1 10.12.8.0/22 dev eth0 proto kernel scope link src 10.12.8.122 10.12.12.0/22 dev eth1 proto kernel scope link src 10.12.12.116 10.12.8.4 dev eth0 src 10.12.8.122 uid 1000 cache => the container has BOTH a metal-admin leg (.8.122) and a metal-internal leg (.12.116); the kernel selects eth0 / src 10.12.8.122 for the utility `.4`; and there is NO default route at all. $ ... 'curl -sS -o /dev/null -w "%{http_code}" -m 12 -H "Snap-Device-Series: 16" https://api.snapcraft.io/v2/snaps/info/core' curl: (7) Couldn't connect to server http=000 <-- DIRECT: FAILS $ ... same URL, but -x http://10.12.8.2:8000 http=200 <-- VIA A PROXY: WORKS => CONFIRMS the D-135 items 2-3 root cause from the client side, and confirms the mechanism the ruling chose: an in-DC CONNECT proxy on the rack works from the exact unit that fails. => CONSEQUENCE FOR THE BUILD: the squid client ACL is the METAL-ADMIN /22 (`10.12.8.0/22` dc0, `10.12.68.0/22` dc1), not metal-internal. A metal-internal ACL would have denied every client the proxy exists for. Corroborating node-side capture (`juju ssh -m vr1-dc0 0`, a bare role node): br-ex 10.12.4.100/22, br-enp1s0 10.12.8.100/22, br-enp3s0 10.12.12.100/22, br-enp5s0 10.12.32.100/22, br-enp6s0 10.12.36.100/22, br-enp4s0 10.12.16.100/22 default via 10.12.4.1 dev br-ex proto static DIRECT api.snapcraft.io -> http=200 (node has a br-ex default route) via 10.12.8.2:8000 -> http=200 http://10.12.8.4/ (mirror) -> http=200 sudo snap get system proxy -> {"proxy":{"http":"","https":"","store":""}} HONEST READING, because this one is easy to misread: the bare node DOES reach the internet directly today via br-ex. The CONTAINERS do not, and they are what fails. Nothing here should be read as "the nodes are already airgapped". Plane CIDR authority (repo, not inferred by mirroring dc0): `scripts/lib-net.sh` vr1-dc0 PLANE_CIDRS = 10.12.4/8/12/16/32/36.0/22, metal-admin = 10.12.8.0/22 vr1-dc1 PLANE_CIDRS = 10.12.64/68/72/76/80/84.0/22, metal-admin = 10.12.68.0/22 2. THE UTILITY `.4` ALREADY EXISTS ON BOTH RACKS; NO NEW OCTET IS NEEDED ------------------------------------------------------------------------------ $ ssh vr1-dc0-rack 'ip -4 -o addr show; ip route show default' virbr2 inet 10.12.8.2/22 | 10.12.8.3/22 secondary | 10.12.8.4/22 secondary virbr5 inet 10.12.4.2/22 ; enp1s0 inet 172.31.0.2/30 default via 10.12.4.1 dev virbr5 $ ssh vr1-dc1-rack 'ip -4 -o addr show; ip route show default' virbr6 inet 10.12.68.2/22 | 10.12.68.3/22 secondary | 10.12.68.4/22 secondary virbr4 inet 10.12.64.2/22 ; enp1s0 inet 172.31.0.6/30 default via 10.12.64.1 dev virbr4 => the D-134 utility `.4` (artifact service) is already aliased on both racks' metal-admin bridge, held by `dc0-mirror-net.service` on dc0 and by `dc1-cache-proxy-net.service` on dc1. Placing the snap proxy ON THE RACK at `.4:` therefore needs NO new ruled octet -- which is what the ruling note asked for. Both racks are Ubuntu 24.04.4 LTS. 3. PORT 3128 IS ALREADY TAKEN -- BY A SQUID, ON BOTH RACKS ------------------------------------------------------------------------------ $ ssh vr1-dc0-rack 'ss -ltn' (dc0, abridged to the relevant rows) LISTEN 10.12.4.2:53 10.12.8.3:53 0.0.0.0:22 0.0.0.0:80 LISTEN 0.0.0.0:5248 0.0.0.0:5247 *:5249 172.31.0.2:5280 LISTEN *:8000 *:3128 $ ssh vr1-dc1-rack 'ss -ltn' (dc1) LISTEN 0.0.0.0:3142 (apt-cacher-ng) 10.12.68.3:53 0.0.0.0:22 0.0.0.0:7911 LISTEN *:8000 *:3128 $ ssh vr1-dc0-rack 'sudo ss -ltnp | grep -E ":(3128|8000) "' LISTEN *:8000 users:(("squid",pid=1871633,fd=13)) LISTEN *:3128 users:(("squid",pid=1871633,fd=12)) $ ssh vr1-dc0-rack 'sudo tr "\0" " " < /proc/1871633/cmdline' squid -N -d 5 -f /var/snap/maas/41649/proxy/maas-proxy.conf => 3128 AND 8000 are held, on BOTH racks, by MAAS's own squid, WILDCARD-bound. A wildcard bind means a later `10.12.8.4:3128` bind would fail. 3142 is dc1's apt-cacher-ng, 80 is dc0's mirror nginx. => CONSEQUENCE: the snap proxy takes **3129**, bound to the LISTEN address specifically, and must NOT be the packaged `squid.service` (see 5). => NOTE: MAAS's squid itself runs with `-f `, i.e. the exact separate-instance shape this build uses. That is a precedent, not a guess. 4. MAAS'S SQUID ALREADY CONNECT-PROXIES THE SNAP STORE. WHY IT WAS NOT USED. ------------------------------------------------------------------------------ $ ssh vr1-dc0-rack 'sudo grep -vE "^\s*#|^\s*$" /var/snap/maas/current/proxy/maas-proxy.conf' ... acl localnet src 10.12.8.0/22 | 10.12.12.0/22 | 10.12.68.0/22 | 10.12.72.0/22 ... acl SSL_ports port 443 acl CONNECT method CONNECT http_access deny CONNECT !SSL_ports http_access allow localnet http_access deny all http_port 3128 transparent http_port 8000 MEASURED to work, three ways: from the rack itself -x http://10.12.8.2:8000 -> api.snapcraft.io http=200 from the juju ctrl VM -x http://10.12.8.2:8000 -> http=200 -x http://10.12.8.6:8000 -> http=200 from mysql-innodb-cluster/0 (container) -x http://10.12.8.2:8000 -> http=200 And juju/MAAS ALREADY point machines at a MAAS squid for apt: $ juju ssh -m controller 0 'grep -rn -i proxy /etc/apt/apt.conf.d/' /etc/apt/apt.conf.d/90curtin-aptproxy:1:Acquire::http::Proxy "http://10.12.8.6:8000/"; SO A WORKING FORWARD PROXY ARGUABLY ALREADY EXISTS. It was NOT adopted, and the reason is recorded here rather than left implicit: (a) `maas-proxy.conf` is MACHINE-GENERATED by MAAS and regenerated whenever MAAS's subnet/proxy settings change -- nothing in this repo owns it, and no diff-against-generator check could ever assert its content. (b) It lives under the per-revision path `/var/snap/maas/current/` (measured real path `/var/snap/maas/41649/...`), which the operating skill already records as NOT surviving a snap refresh for the power keys. (c) It has NO destination restriction: `http_access allow localnet` permits CONNECT to ANY :443 host from every plane MAAS knows. Adopting it as the snap path would silently give the node planes general HTTPS egress. (d) It is the hidden-coupling anti-pattern this repo already has a scar from (dc-cache-proxy.sh reusing dc-mirror.sh's net layer, D-135 amendment 2026-07-27): a service whose prerequisites are owned by another service's machinery cannot be torn down or rebuilt independently. THIS IS AN OPERATOR-VISIBLE ALTERNATIVE, NOT A DISMISSAL. If the operator prefers it at apply time, the equivalent live change is just `juju model-config snap-https-proxy=http://10.12.8.6:8000` with no rack build at all -- cheaper, and worse on (a)-(d). 5. THE SHIPPED UBUNTU SQUID: MEASURED FROM THE PACKAGE, NOT ASSUMED ------------------------------------------------------------------------------ $ ssh vr1-dc0-rack 'apt-cache policy squid tinyproxy' (identical on dc1) squid Candidate: 6.14-0ubuntu0.24.04.4 noble-updates/**main** tinyproxy Candidate: 1.11.1-3ubuntu0.1 noble-updates/**universe** On the JUMPHOST (also noble 24.04.4, same candidate version), the .deb was downloaded and unpacked read-only so the shipped config could be READ: $ apt-get download squid && dpkg-deb -x squid_6.14-0ubuntu0.24.04.4_amd64.deb x $ grep -vE '^\s*#|^\s*$' x/etc/squid/squid.conf acl localnet src 0.0.0.1-0.255.255.255 / 10.0.0.0/8 / 100.64.0.0/10 / 169.254.0.0/16 / 172.16.0.0/12 / 192.168.0.0/16 / fc00::/7 / fe80::/10 acl SSL_ports port 443 acl Safe_ports port 80 21 443 70 210 1025-65535 280 488 591 777 http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost manager http_access deny manager http_access allow localhost http_access deny to_localhost http_access deny to_linklocal include /etc/squid/conf.d/*.conf http_access deny all http_port 3128 coredump_dir /var/spool/squid refresh_pattern ... $ cat x/usr/lib/systemd/system/squid.service [Service] Type=notify PIDFile=/run/squid.pid Group=proxy RuntimeDirectory=squid RuntimeDirectoryMode=0775 ExecStartPre=/usr/sbin/squid --foreground -z ExecStart=/usr/sbin/squid --foreground -sYC ExecReload=/bin/kill -HUP $MAINPID KillMode=mixed NotifyAccess=all $ cat x/etc/default/squid (EMPTY -- there is no SQUID_ARGS / SQUID_CONFIG_FILE indirection to hook) THREE CONSEQUENCES, each load-bearing: (i) `include /etc/squid/conf.d/*.conf` sits BEFORE `http_access deny all`, so a conf.d drop-in CAN add an allow rule -- but it CANNOT unset `http_port 3128`, and 3128 is held by MAAS's squid (section 3). The packaged unit would therefore FATAL on bind. A conf.d drop-in is structurally unusable here. (ii) The default already carries `http_access deny CONNECT !SSL_ports` and a terminal `http_access deny all` -- the posture we want is squid's own default, which is the main reason to prefer it over tinyproxy (whose `ConnectPort` default is documented as "all ports allowed"). (iii) `/etc/default/squid` being empty means there is no supported way to point the packaged unit at another config file. A DEDICATED instance (own config + own unit + own pid file) is the only clean shape, and `install` must `systemctl disable --now squid` so the packaged unit does not crash-loop on 3128 and poison triage. $ zcat x/usr/share/man/man8/squid.8.gz | grep -A2 '"\\-n name"' "-n name Specify Windows Service name ..." => `-n` is documented as Windows-only, so the instance is separated by `-f` plus an explicit `pid_filename`, NOT by `-n`. 6. SHARED-MEMORY COEXISTENCE WITH MAAS'S SQUID: MEASURED, NOT REASONED ------------------------------------------------------------------------------ Two squids on one host can collide on `/dev/shm` segment names. Measured: $ ssh vr1-dc0-rack 'ls -la /dev/shm/' -rw------- snap_daemon snap.maas.squid-cf__metadata.shm -rw------- snap_daemon snap.maas.squid-cf__queues.shm -rw------- snap_daemon snap.maas.squid-cf__readers.shm $ ssh vr1-dc0-rack 'sudo ls -la /run/ | grep -i squid' -> (no /run/squid*) => snapd namespaces MAAS's segments behind a `snap.maas.` prefix; a host squid creates bare `squid-cf__*`. NO COLLISION. And MAAS's squid runs `-N` with no pid file, so `/run/-snap-proxy.pid` collides with nothing either. => This is the one coexistence question dc-cache-proxy.sh had to leave as "REASONED, NOT MEASURED". Here it is measured. 7. SNAP-STORE ENDPOINT SHAPES, MEASURED, SO THE CHECK PREDICATES ARE REAL ------------------------------------------------------------------------------ $ ssh vr1-dc0-rack 'curl -sS -m 25 -H "Snap-Device-Series: 16" https://api.snapcraft.io/v2/snaps/info/core | head -c 400' {"channel-map":[{"channel":{"architecture":"amd64","name":"stable",...}, "download":{"deltas":[],"sha3-384":"13c728f16430310ba467b5768bf3484607a14c45 b3a91e47023928a83ea62f4ea04eb64c922b3d33f1d5a42fad686376","size":110305280, "url":"https://api.snapcraft.io/api/v1/snaps/downlo... => the ALLOW-direction content assertion is `"channel-map"` AND `sha3-384`. Neither appears in a squid error page. A bare 200 is NOT evidence. $ ssh vr1-dc0-rack 'curl -sS -o /dev/null -w "%{http_code}" -m 25 https://canonical-lgw01.cdn.snapcraftcontent.com/' 403 (curl exit 0 -- the CDN itself answers 403 to a bare GET of /) $ ssh vr1-dc0-rack 'curl -sS -o /dev/null -w "code=[%{http_code}]" -m 15 -x http://10.12.8.2:8000 https://api.snapcraft.io:25/' curl: (56) CONNECT tunnel failed, response 403 code=[000] => A PROXY DENIAL and a SERVER 403 are DISTINGUISHABLE, measured: proxy denial : curl exit 56, stderr "CONNECT tunnel failed, response 403", %{http_code} = 000 server 403 : curl exit 0, %{http_code} = 403 That difference is what lets `check` assert the DENY DIRECTION (the destination allowlist is actually in force) separately from the ALLOW direction, from the SAME source address. 8. JUJU'S SNAP PROXY KEYS EXIST IN THIS EXACT CONTROLLER ------------------------------------------------------------------------------ $ ssh vr1-dc0-rack 'juju version; juju model-config -m controller --format=yaml | grep -iE "snap|proxy"' 3.6.27-genericlinux-amd64 apt-http-proxy: apt-https-proxy: apt-ftp-proxy: apt-no-proxy: http-proxy: https-proxy: ftp-proxy: no-proxy: proxy-ssh: juju-http-proxy: juju-https-proxy: juju-ftp-proxy: juju-no-proxy: lxd-snap-channel: snap-http-proxy: snap-https-proxy: snap-store-assertions: snap-store-proxy: snap-store-proxy-url: (identical set from `juju model-defaults`) => the five key names in the ruling note are CONFIRMED against the live controller, not quoted from documentation. NOTE ON THE MODEL: at capture time `juju models` showed `vr1-dc0` in state `destroying` (operator-driven, separate from this work). Probes above ran against machines that still existed. Nothing here mutated the model. 9. VENDOR RESEARCH -- WHAT SNAPD ACTUALLY HONOURS ------------------------------------------------------------------------------ Sources opened and quoted (not summarised from memory): snapd config mechanism https://snapcraft.io/docs/reference/administration/system-options/ "snap set system proxy.http=... / proxy.https=..." CAVEAT FOUND AND NOT SMOOTHED OVER: that page carries an "Available only on Ubuntu Core" note adjacent to the proxy section. THREE primary sources contradict it, and code is the stronger witness: https://raw.githubusercontent.com/canonical/snapd/master/overlord/configstate/configcore/proxy.go supportedConfigurations["core.proxy.http"|"...https"|"...ftp"| "...no-proxy"|"...store"] = true -- registered with NO core-only flag envFilePath = "/etc/environment" -- a CLASSIC-system file handleProxyConfiguration() writes http_proxy/https_proxy/ftp_proxy/no_proxy => `snap set system proxy.*` writes /etc/environment. It does NOT write a systemd drop-in. https://raw.githubusercontent.com/canonical/snapd/master/data/systemd/snapd.service.in "EnvironmentFile=-/var/lib/snapd/environment/snapd.conf" (+ the built-in one) https://raw.githubusercontent.com/canonical/snapd/master/httputil/transport.go "Proxy: http.ProxyFromEnvironment" => snapd's HTTP transport reads the PROCESS environment; a snapd.service.d drop-in with Environment=https_proxy= works by the same path. Canonical's own k8s docs use exactly that drop-in shape: https://documentation.ubuntu.com/canonical-kubernetes/latest/snap/howto/networking/proxy/ https://forum.snapcraft.io/t/preferred-method-for-configuring-proxy-settings-for-snapd/11743 chipaca (Canonical), 2019-06-11: "A proxy set via snap set overrides that obtained from the environment"; the two approaches "are ... equivalent". WHICH HOSTS, AND WHETHER CONNECT IS REQUIRED https://snapcraft.io/docs/reference/administration/network-requirements/ https://forum.snapcraft.io/t/network-requirements/5147 (CDN list 2021-10-25) api.snapcraft.io:443 dashboard.snapcraft.io:443 login.ubuntu.com:443 *.snapcraftcontent.com:443 storage.snapcraftcontent.com:443 canonical-{lgw01,lcy01,lcy02,bos01}.cdn.snapcraftcontent.com:443 cloudfront.cdn.snapcraftcontent.com:443 *.cdn.snapcraft.io:443 [deprecated] => EVERY documented endpoint is :443. The proxy only ever sees `CONNECT host:443`, so a CONNECT-capable forward proxy is REQUIRED and `snap-http-proxy` is effectively inert for snapd's own traffic. Set both anyway; `snap-https-proxy` is the one that matters. https://raw.githubusercontent.com/canonical/snapd/master/store/store.go the SAME client (opts.Proxy = s.cfg.Proxy) serves API calls AND downloads => THE CDN FETCH IS PROXIED TOO. A destination allowlist that names only api.snapcraft.io would establish the session and then fail the download. WHAT JUJU DOES WITH THE KEYS https://canonical.com/juju/docs/juju-cli/3.6/reference/juju-cli/list-of-juju-cli-commands/model-config/ snap-http-proxy "The HTTP proxy value for installing snaps" snap-https-proxy "The HTTPS proxy value for installing snaps" snap-store-proxy "The snap store proxy for installing snaps" snap-store-proxy-url / snap-store-assertions http-proxy/https-proxy -> instance HTTP_PROXY; juju-* -> charm hook env => three DIFFERENT surfaces. Setting the wrong one will not fix snaps. https://raw.githubusercontent.com/juju/juju/3.6/internal/worker/proxyupdater/proxyupdater.go handleSnapProxyValues(): maybeAddSettings("proxy.http", ...), ("proxy.https", ...) then RunFunc(noStdIn, "snap", append([]string{"set","system"}, snapSettings...)) skipped only on CentOS => juju literally shells out `snap set system proxy.http=... proxy.https=...`, i.e. mechanism = section 9 paragraph 1. It is a WATCHER (WatchForProxyConfigAndAPIHostPortChanges), so setting the key AFTER the machines exist is expected to propagate. snapd proxy.store is a 32-character STORE ID needing a signed assertion: configcore/proxy.go: "cannot set proxy.store to %q without a matching store assertion"; client shape documented at https://ubuntu.com/enterprise-store/docs/page/how-to/devices/ => `snap-store-proxy` / `-url` / `-assertions` are the ENTERPRISE STORE product, NOT a forward proxy. LEAVE THEM EMPTY. SQUID VS TINYPROXY, AND THE CACHING QUESTION https://packages.ubuntu.com/jammy/squid -- main, security-supported https://packages.ubuntu.com/jammy/net/tinyproxy -- universe https://ubuntu.com/security/cve?package=tinyproxy jammy: 2 Fixed, 2 Not affected, SIX in "Needs evaluation" https://tinyproxy.github.io/ + tinyproxy.conf(5) ConnectPort: "If no ConnectPort line is found, then all ports are allowed." => tinyproxy's CONNECT default is OPEN; squid's is deny-except-443. https://www.squid-cache.org/Doc/config/http_port/ "Squid normally listens to port 3128" https://www.squid-cache.org/Doc/config/cache/ `cache deny all` documented https://wiki.squid-cache.org/Features/HTTPS "the proxy establishes a TCP connection to the specified server, responds with an HTTP 200 (Connection Established) ... without understanding or interpreting the tunneled traffic"; "Squid does not have access to those encrypted messages." https://wiki.squid-cache.org/Features/SslBump "Decrypting HTTPS tunnels constitutes a man-in-the-middle attack ..." => THERE IS NO CACHING BENEFIT FOR SNAPS. Every endpoint is :443, so a CONNECT proxy sees opaque tunnels and can cache nothing without TLS interception, which is explicitly not wanted. Choose on security support and simplicity, NOT on caching. (This is the opposite of the apt case, where dc1's apt-cacher-ng genuinely caches plain-HTTP apt.) https://github.com/chobits/ngx_http_proxy_connect_module a third-party module exists precisely because stock nginx has no CONNECT. => nginx is NOT a candidate. WHY NOT THE ENTERPRISE STORE (ex Snap Store Proxy) https://ubuntu.com/enterprise-store/docs/page/how-to/install/ prerequisites: an LTS server, a DOMAIN NAME, a PostgreSQL instance with btree_gist + intarray, and a store REGISTRATION step. https://ubuntu.com/enterprise-store/docs/how-to/airgap/ offline mode additionally needs registration on an internet-connected machine, an offline-snap-store.tar.gz carried in, and an export/import cycle. https://ubuntu.com/blog/canonical-announces-the-enterprise-store (2026-07-21) "included as part of Ubuntu Pro" => for ~12 clients behind a rack that already has egress, this is a domain name, a database, a Canonical account dependency and a per-node `snap ack` -- against `apt install squid` plus two model-config keys. 10. WHAT COULD NOT BE VERIFIED (stated, not filled in) ------------------------------------------------------------------------------ a. WHETHER A RUNNING snapd RE-READS THE PROXY WITHOUT A RESTART. The chain is juju -> `snap set system proxy.*` -> snapd rewrites /etc/environment -> snapd.service reads it via EnvironmentFile= AT START. Evidence it may not re-read live: https://bugs.launchpad.net/juju/+bug/1737332 (Fix Released 2018-10-14) "snapd *MUST* be restarted though after proxy.conf is created as naturally there is no sane way to 'reload' environment variables"; and https://bugs.launchpad.net/snapd/+bug/1791587 "snapd ignores proxy settings set via core snap" (Fix Released, snapd 2.36.1, 2018-11-22) whose root cause was exactly this. It could NOT be determined whether that fix made snapd apply the proxy in-memory. OPERATIONAL CONSEQUENCE, carried into the script header and the runbook wording: after setting the key, verify per node with BOTH `snap get system proxy` AND a real egress probe; if the first passes and the second fails, `systemctl restart snapd` and re-probe. b. WHICH snapd VERSION INTRODUCED `snap set system proxy.*`. Two search summaries disagree (2.28 vs 2.33) and neither was from a page that was opened. MOOT here -- both racks measure snapd 2.76.1 and juju 3.6 emits `snap set system` regardless. c. THE "Available only on Ubuntu Core" NOTE on the system-options page (9a). Code, the /etc/environment write target and juju's unconditional use all say it does not apply to classic; no doc source repo was found to confirm the note is an authoring defect. The settling test is one command on one node: `sudo snap set system proxy.https=http://:3129 && sudo snap get system proxy`. d. NOTHING WAS PROVEN END-TO-END. No proxy was installed, so no snap has been installed through one. The build's own `check` and `node` subcommands are the gates that will prove it at apply time; the harness green is a FIXTURE green. e. dc1 HAS NOT BEEN PROBED FROM A CONTAINER. dc1 has no deployed model yet, so the metal-admin-source finding (section 1) is measured on dc0 only. The dc1 site row takes its CIDRs from `lib-net.sh`, and `node dc1` is the assertion that will prove it there. f. The squid config generated by this build has never been parsed by a squid binary. `install` runs `squid -k parse -f ` before enabling anything, so a syntax error fails loud at apply time rather than silently. g. THE DENY-SHAPE MEASUREMENT IS FROM A *PORT* DENY, NOT A *DSTDOMAIN* DENY. Section 7's `curl: (56) CONNECT tunnel failed, response 403` was produced by MAAS's squid refusing `api.snapcraft.io:25` under `http_access deny CONNECT !SSL_ports` -- a PORT rule. This build's deny probe trips a DSTDOMAIN rule (`http_access deny !snap_store`). Squid answers both with 403 at CONNECT, so the two should be byte-identical to curl, but that path was NOT measured. THE FAILURE MODE IS SAFE: check's deny branch treats anything that is neither "403 at CONNECT" nor "curl exit 0" as REFUSE, so an unexpected shape can only withhold a pass, never grant one. Confirm at apply time, once: run the deny probe by hand against the built proxy and read the exact curl line. h. TWO OF check's PROBES ARE LIVENESS-COUPLED TO CANONICAL. The store-API probe and the CDN probe reach real third-party hosts, so an upstream outage or a DNS failure makes a healthy proxy REFUSE. That is deliberate -- a probe that cannot distinguish "the CDN is down" from "the proxy is misconfigured" must not guess -- but it means a REFUSE is not by itself evidence about the proxy, and the script header says so where a triaging operator will read it. The static half of the same property (that the CDN wildcard is PRESENT in STORE_DOMAINS) is asserted by the config diff and by harness case T16, which have no external dependency at all. 11. MUTATION PASS ON tests/dc-snap-proxy/run-tests.sh (2026-07-31) ------------------------------------------------------------------------------ "A checker that cannot fail is not a gate", applied to the harness itself. The 2026-07-30 precedent is explicit: three freshly-written assertions were deleted one at a time and the suite stayed green every time -- they were decoration. METHOD. A one-shot driver (/tmp, NOT a repo artifact) applied ONE exact string replacement to scripts/dc-snap-proxy.sh at a time, ran the suite, restored the file and re-verified its sha256 against the original. Every mutation neuters a PREDICATE, never only a message: `miss` -> `say`, `refuse` -> `say`, `nno` -> `nok`, `elif ` -> `elif false`, a value swapped for a plausible wrong one, or a forbidden construct introduced. RESULT: 53 mutations, 53 KILLED, 0 survived. Baseline and final state both `dc-snap-proxy: 52 passed, 0 failed`; the script restored byte-identical after every mutation (sha256 verified, final sha_ok=True). M01 dc0 LISTEN 10.12.8.4 -> .5 -> T3 red (11 red total) M02 dc0 edge gw -> .254 -> T4 red M03 dc0 client ACL -> metal-internal 10.12.12.0/22 -> T5 red <-- the exact wrong turn the container measurement prevented M04 dc1 LISTEN -> .5 -> T6 red M05 dc1 edge gw -> .254 -> T7 red M06 dc1 client ACL -> metal-internal 10.12.72.0/22 -> T8 red M07 port 3129 -> MAAS's 3128 -> T9 red M08 hardcode virbr2 instead of util_bridge -> T10 red M09 net unit renamed to *-cache-proxy-net.service -> T11 red M10 operative "dc-mirror.sh install" referral added -> T11b red M11 ip addr replace -> ip addr add -> T12 red M12 MEASURED/RULED tags dropped -> T13 red M13 install root guard removed -> T14 red M14 snap-https-proxy key name mangled -> T15 red M15 allowlist narrowed to api.snapcraft.io only -> T16 red M16 http_port LISTEN:PORT -> wildcard PORT -> T17 red M17 install stops disabling the packaged squid -> T18 red M18 install stops parsing the config before enable -> T19 red (first attempt was a MALFORMED mutation -- `true # ...` commented out the `then` and broke `bash -n`, so T2 and 28 others went red while T19 did not. Recorded rather than quietly re-rolled: a mutation that breaks syntax tests nothing. Redone as `if false; then` + `:`, and T19 went red alone.) M19 install apt-installs directly -> T20 red M20 bad mode stops exiting 2 -> T21 red M21 unknown site stops exiting 2 -> T22 red M22 check_file asserts EXISTENCE not content -> T24 red <-- the dc-mirror.sh false-green class, reproduced and caught M23 missing artifact reported as OK -> T25 red M24 unit-enabled miss softened -> T26 red M25 packaged squid ACTIVE not detected -> T27 red M26 packaged squid ENABLED not detected -> T27b red M27 wildcard bind accepted as OK -> T28 red M28 absent listener accepted -> T29 red M29 absent LISTEN alias accepted -> T30 red M30 unresolvable bridge defaults to success -> T31 red M31 missing default route accepted -> T32 red M32 check drops the channel-map content assertion -> T33 red M33 check drops the sha3-384 assertion -> T34 red M34 check ignores a non-200 store status -> T35 red M35 unreachable proxy defaults to success -> T36 red M36 refused CDN tunnel accepted -> T37 red M37 unrecognised CDN result defaults to success -> T38 red M38 OPEN proxy accepted (deny direction gone) -> T39 red <-- the assertion that proves the allowlist is in force, not merely written down M39 unrecognised deny result defaults to success -> T40 red M40 missing squid package accepted -> T41 red M41 no curl defaults to success -> T42 red M42 node: ACL denial reported as OK -> T44 red M43 node: unreachable proxy defaults to success -> T45 red M44 node: drops the channel-map assertion -> T46 red M45 node: OPEN proxy accepted -> T47 red M46 node: absent https_proxy accepted -> T48 red M47 node: wrong https_proxy accepted -> T49 red M48 node makes a rack-side call (systemctl) -> T50 red M49 check: REFUSE treated as a pass -> T31/36/38/40/42 red M50 node: REFUSE treated as a pass -> T45 red M51 check: FAIL treated as a pass -> 15 cases red M52 node: FAIL treated as a pass -> 5 cases red M53 ENV_FILE path moved -> T43 red (the node green baseline is itself killable, so it is not a tautology) THREE OF THE HARNESS'S OWN CASES WERE FOUND TOO WEAK WHILE BUILDING THIS, and were STRENGTHENED rather than left passing -- each would have stayed green after the thing it names was removed: T16 grepped the WHOLE FILE for `.snapcraftcontent.com`, which the header discusses in prose, so narrowing the actual STORE_DOMAINS value left it green. Now scoped to the `STORE_DOMAINS=` line. T18 grepped the whole file for `systemctl disable --now squid`, which check's own failure MESSAGE contains, so install could stop doing it. Now scoped to do_install's body. T20 matched `apt-get install` inside the guidance STRING, so it read green whether or not the script apt-installed. Now blanks double-quoted strings first and matches an INVOCATION position. Same class, found the same way: T10 and T50 had to blank comments / quoted strings for the opposite reason -- they were RED on legitimate prose (measured `virbr5` evidence in a site-row comment; an `ip route get` hint inside a failure message). Rewording the prose to satisfy a regex would have been gaming the test, so the tests were fixed to assert on CODE. WHAT THE MUTATION PASS DOES NOT PROVE: that the assertions are the RIGHT ones, or that the proxy works. It proves only that none of them is decoration.