stage5-snap-proxy-measurements-20260731.txt
===========================================
Read-only measurement capture backing `scripts/dc-snap-proxy.sh` (the REPO SIDE
of D-135/D-107 RULING 2, 2026-07-31: "HTTP(S) forward proxy in the DC utility
band + juju snap-https-proxy (Recommended)").
SCOPE OF THIS CAPTURE. Every command below is READ-ONLY. NO cloud mutation was
made in the session that produced it: no `juju config`, no `juju model-config`,
no package installed, no service started or stopped, no file written on any rack
or node. The proxy is BUILT IN THE REPO and NOT APPLIED. Two side effects are
declared rather than hidden: the `curl` probes traversed MAAS's existing squid
and may have added entries to ITS cache, and `apt-get download squid` wrote a
.deb into `/tmp` ON THE JUMPHOST (not on any rack) so the shipped squid config
could be read instead of guessed.
Host/date: vcloud jumphost, 2026-07-31. Branch dc-dc-stage5-preconditions.
1. THE THREE FAILING APPLICATIONS ARE LXD CONTAINERS, AND THEIR SOURCE ADDRESS
TOWARD THE RACK IS ON METAL-ADMIN, NOT METAL-INTERNAL
------------------------------------------------------------------------------
This is the measurement the whole design turns on: it decides the squid client
ACL and whether the D-134 utility `.4` is reachable by the clients at all.
`juju status -m vr1-dc0` shows `mysql-innodb-cluster/0` at `10.12.12.116`, which
is NOT a D-134 node octet (.100-.102 / .120-.121 / .150-.153) -- so the failing
units are containers, and the address `juju status` prints is only ONE of their
legs. Measured from inside the container:
$ juju ssh -m vr1-dc0 mysql-innodb-cluster/0 'ip -4 -o addr show; ip route show;
ip route get 10.12.8.4'
60: eth0 inet 10.12.8.122/22 brd 10.12.11.255 scope global eth0
62: eth1 inet 10.12.12.116/22 brd 10.12.15.255 scope global eth1
10.12.8.0/22 dev eth0 proto kernel scope link src 10.12.8.122
10.12.12.0/22 dev eth1 proto kernel scope link src 10.12.12.116
10.12.8.4 dev eth0 src 10.12.8.122 uid 1000
cache
=> the container has BOTH a metal-admin leg (.8.122) and a metal-internal leg
(.12.116); the kernel selects eth0 / src 10.12.8.122 for the utility `.4`;
and there is NO default route at all.
$ ... 'curl -sS -o /dev/null -w "%{http_code}" -m 12
-H "Snap-Device-Series: 16" https://api.snapcraft.io/v2/snaps/info/core'
curl: (7) Couldn't connect to server http=000 <-- DIRECT: FAILS
$ ... same URL, but -x http://10.12.8.2:8000
http=200 <-- VIA A PROXY: WORKS
=> CONFIRMS the D-135 items 2-3 root cause from the client side, and confirms
the mechanism the ruling chose: an in-DC CONNECT proxy on the rack works
from the exact unit that fails.
=> CONSEQUENCE FOR THE BUILD: the squid client ACL is the METAL-ADMIN /22
(`10.12.8.0/22` dc0, `10.12.68.0/22` dc1), not metal-internal. A
metal-internal ACL would have denied every client the proxy exists for.
Corroborating node-side capture (`juju ssh -m vr1-dc0 0`, a bare role node):
br-ex 10.12.4.100/22, br-enp1s0 10.12.8.100/22, br-enp3s0 10.12.12.100/22,
br-enp5s0 10.12.32.100/22, br-enp6s0 10.12.36.100/22, br-enp4s0 10.12.16.100/22
default via 10.12.4.1 dev br-ex proto static
DIRECT api.snapcraft.io -> http=200 (node has a br-ex default route)
via 10.12.8.2:8000 -> http=200
http://10.12.8.4/ (mirror) -> http=200
sudo snap get system proxy -> {"proxy":{"http":"","https":"","store":""}}
HONEST READING, because this one is easy to misread: the bare node DOES reach
the internet directly today via br-ex. The CONTAINERS do not, and they are what
fails. Nothing here should be read as "the nodes are already airgapped".
Plane CIDR authority (repo, not inferred by mirroring dc0): `scripts/lib-net.sh`
vr1-dc0 PLANE_CIDRS = 10.12.4/8/12/16/32/36.0/22, metal-admin = 10.12.8.0/22
vr1-dc1 PLANE_CIDRS = 10.12.64/68/72/76/80/84.0/22, metal-admin = 10.12.68.0/22
2. THE UTILITY `.4` ALREADY EXISTS ON BOTH RACKS; NO NEW OCTET IS NEEDED
------------------------------------------------------------------------------
$ ssh vr1-dc0-rack 'ip -4 -o addr show; ip route show default'
virbr2 inet 10.12.8.2/22 | 10.12.8.3/22 secondary | 10.12.8.4/22 secondary
virbr5 inet 10.12.4.2/22 ; enp1s0 inet 172.31.0.2/30
default via 10.12.4.1 dev virbr5
$ ssh vr1-dc1-rack 'ip -4 -o addr show; ip route show default'
virbr6 inet 10.12.68.2/22 | 10.12.68.3/22 secondary | 10.12.68.4/22 secondary
virbr4 inet 10.12.64.2/22 ; enp1s0 inet 172.31.0.6/30
default via 10.12.64.1 dev virbr4
=> the D-134 utility `.4` (artifact service) is already aliased on both racks'
metal-admin bridge, held by `dc0-mirror-net.service` on dc0 and by
`dc1-cache-proxy-net.service` on dc1. Placing the snap proxy ON THE RACK at
`.4:<port>` therefore needs NO new ruled octet -- which is what the ruling
note asked for. Both racks are Ubuntu 24.04.4 LTS.
3. PORT 3128 IS ALREADY TAKEN -- BY A SQUID, ON BOTH RACKS
------------------------------------------------------------------------------
$ ssh vr1-dc0-rack 'ss -ltn' (dc0, abridged to the relevant rows)
LISTEN 10.12.4.2:53 10.12.8.3:53 0.0.0.0:22 0.0.0.0:80
LISTEN 0.0.0.0:5248 0.0.0.0:5247 *:5249 172.31.0.2:5280
LISTEN *:8000 *:3128
$ ssh vr1-dc1-rack 'ss -ltn' (dc1)
LISTEN 0.0.0.0:3142 (apt-cacher-ng) 10.12.68.3:53 0.0.0.0:22 0.0.0.0:7911
LISTEN *:8000 *:3128
$ ssh vr1-dc0-rack 'sudo ss -ltnp | grep -E ":(3128|8000) "'
LISTEN *:8000 users:(("squid",pid=1871633,fd=13))
LISTEN *:3128 users:(("squid",pid=1871633,fd=12))
$ ssh vr1-dc0-rack 'sudo tr "\0" " " < /proc/1871633/cmdline'
squid -N -d 5 -f /var/snap/maas/41649/proxy/maas-proxy.conf
=> 3128 AND 8000 are held, on BOTH racks, by MAAS's own squid, WILDCARD-bound.
A wildcard bind means a later `10.12.8.4:3128` bind would fail. 3142 is
dc1's apt-cacher-ng, 80 is dc0's mirror nginx.
=> CONSEQUENCE: the snap proxy takes **3129**, bound to the LISTEN address
specifically, and must NOT be the packaged `squid.service` (see 5).
=> NOTE: MAAS's squid itself runs with `-f <conf>`, i.e. the exact
separate-instance shape this build uses. That is a precedent, not a guess.
4. MAAS'S SQUID ALREADY CONNECT-PROXIES THE SNAP STORE. WHY IT WAS NOT USED.
------------------------------------------------------------------------------
$ ssh vr1-dc0-rack 'sudo grep -vE "^\s*#|^\s*$" /var/snap/maas/current/proxy/maas-proxy.conf'
... acl localnet src 10.12.8.0/22 | 10.12.12.0/22 | 10.12.68.0/22 | 10.12.72.0/22 ...
acl SSL_ports port 443
acl CONNECT method CONNECT
http_access deny CONNECT !SSL_ports
http_access allow localnet
http_access deny all
http_port 3128 transparent
http_port 8000
MEASURED to work, three ways:
from the rack itself -x http://10.12.8.2:8000 -> api.snapcraft.io http=200
from the juju ctrl VM -x http://10.12.8.2:8000 -> http=200
-x http://10.12.8.6:8000 -> http=200
from mysql-innodb-cluster/0 (container) -x http://10.12.8.2:8000 -> http=200
And juju/MAAS ALREADY point machines at a MAAS squid for apt:
$ juju ssh -m controller 0 'grep -rn -i proxy /etc/apt/apt.conf.d/'
/etc/apt/apt.conf.d/90curtin-aptproxy:1:Acquire::http::Proxy "http://10.12.8.6:8000/";
SO A WORKING FORWARD PROXY ARGUABLY ALREADY EXISTS. It was NOT adopted, and
the reason is recorded here rather than left implicit:
(a) `maas-proxy.conf` is MACHINE-GENERATED by MAAS and regenerated whenever
MAAS's subnet/proxy settings change -- nothing in this repo owns it, and
no diff-against-generator check could ever assert its content.
(b) It lives under the per-revision path `/var/snap/maas/current/` (measured
real path `/var/snap/maas/41649/...`), which the operating skill already
records as NOT surviving a snap refresh for the power keys.
(c) It has NO destination restriction: `http_access allow localnet` permits
CONNECT to ANY :443 host from every plane MAAS knows. Adopting it as the
snap path would silently give the node planes general HTTPS egress.
(d) It is the hidden-coupling anti-pattern this repo already has a scar from
(dc-cache-proxy.sh reusing dc-mirror.sh's net layer, D-135 amendment
2026-07-27): a service whose prerequisites are owned by another service's
machinery cannot be torn down or rebuilt independently.
THIS IS AN OPERATOR-VISIBLE ALTERNATIVE, NOT A DISMISSAL. If the operator
prefers it at apply time, the equivalent live change is just
`juju model-config snap-https-proxy=http://10.12.8.6:8000` with no rack build
at all -- cheaper, and worse on (a)-(d).
5. THE SHIPPED UBUNTU SQUID: MEASURED FROM THE PACKAGE, NOT ASSUMED
------------------------------------------------------------------------------
$ ssh vr1-dc0-rack 'apt-cache policy squid tinyproxy' (identical on dc1)
squid Candidate: 6.14-0ubuntu0.24.04.4 noble-updates/**main**
tinyproxy Candidate: 1.11.1-3ubuntu0.1 noble-updates/**universe**
On the JUMPHOST (also noble 24.04.4, same candidate version), the .deb was
downloaded and unpacked read-only so the shipped config could be READ:
$ apt-get download squid && dpkg-deb -x squid_6.14-0ubuntu0.24.04.4_amd64.deb x
$ grep -vE '^\s*#|^\s*$' x/etc/squid/squid.conf
acl localnet src 0.0.0.1-0.255.255.255 / 10.0.0.0/8 / 100.64.0.0/10 /
169.254.0.0/16 / 172.16.0.0/12 / 192.168.0.0/16 / fc00::/7 / fe80::/10
acl SSL_ports port 443
acl Safe_ports port 80 21 443 70 210 1025-65535 280 488 591 777
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localhost
http_access deny to_localhost
http_access deny to_linklocal
include /etc/squid/conf.d/*.conf
http_access deny all
http_port 3128
coredump_dir /var/spool/squid
refresh_pattern ...
$ cat x/usr/lib/systemd/system/squid.service
[Service] Type=notify PIDFile=/run/squid.pid Group=proxy
RuntimeDirectory=squid RuntimeDirectoryMode=0775
ExecStartPre=/usr/sbin/squid --foreground -z
ExecStart=/usr/sbin/squid --foreground -sYC
ExecReload=/bin/kill -HUP $MAINPID KillMode=mixed NotifyAccess=all
$ cat x/etc/default/squid
(EMPTY -- there is no SQUID_ARGS / SQUID_CONFIG_FILE indirection to hook)
THREE CONSEQUENCES, each load-bearing:
(i) `include /etc/squid/conf.d/*.conf` sits BEFORE `http_access deny all`,
so a conf.d drop-in CAN add an allow rule -- but it CANNOT unset
`http_port 3128`, and 3128 is held by MAAS's squid (section 3). The
packaged unit would therefore FATAL on bind. A conf.d drop-in is
structurally unusable here.
(ii) The default already carries `http_access deny CONNECT !SSL_ports` and a
terminal `http_access deny all` -- the posture we want is squid's own
default, which is the main reason to prefer it over tinyproxy (whose
`ConnectPort` default is documented as "all ports allowed").
(iii) `/etc/default/squid` being empty means there is no supported way to point
the packaged unit at another config file. A DEDICATED instance (own
config + own unit + own pid file) is the only clean shape, and
`install` must `systemctl disable --now squid` so the packaged unit does
not crash-loop on 3128 and poison triage.
$ zcat x/usr/share/man/man8/squid.8.gz | grep -A2 '"\\-n name"'
"-n name Specify Windows Service name ..."
=> `-n` is documented as Windows-only, so the instance is separated by `-f`
plus an explicit `pid_filename`, NOT by `-n`.
6. SHARED-MEMORY COEXISTENCE WITH MAAS'S SQUID: MEASURED, NOT REASONED
------------------------------------------------------------------------------
Two squids on one host can collide on `/dev/shm` segment names. Measured:
$ ssh vr1-dc0-rack 'ls -la /dev/shm/'
-rw------- snap_daemon snap.maas.squid-cf__metadata.shm
-rw------- snap_daemon snap.maas.squid-cf__queues.shm
-rw------- snap_daemon snap.maas.squid-cf__readers.shm
$ ssh vr1-dc0-rack 'sudo ls -la /run/ | grep -i squid' -> (no /run/squid*)
=> snapd namespaces MAAS's segments behind a `snap.maas.` prefix; a host squid
creates bare `squid-cf__*`. NO COLLISION. And MAAS's squid runs `-N` with no
pid file, so `/run/<site>-snap-proxy.pid` collides with nothing either.
=> This is the one coexistence question dc-cache-proxy.sh had to leave as
"REASONED, NOT MEASURED". Here it is measured.
7. SNAP-STORE ENDPOINT SHAPES, MEASURED, SO THE CHECK PREDICATES ARE REAL
------------------------------------------------------------------------------
$ ssh vr1-dc0-rack 'curl -sS -m 25 -H "Snap-Device-Series: 16"
https://api.snapcraft.io/v2/snaps/info/core | head -c 400'
{"channel-map":[{"channel":{"architecture":"amd64","name":"stable",...},
"download":{"deltas":[],"sha3-384":"13c728f16430310ba467b5768bf3484607a14c45
b3a91e47023928a83ea62f4ea04eb64c922b3d33f1d5a42fad686376","size":110305280,
"url":"https://api.snapcraft.io/api/v1/snaps/downlo...
=> the ALLOW-direction content assertion is `"channel-map"` AND `sha3-384`.
Neither appears in a squid error page. A bare 200 is NOT evidence.
$ ssh vr1-dc0-rack 'curl -sS -o /dev/null -w "%{http_code}" -m 25
https://canonical-lgw01.cdn.snapcraftcontent.com/'
403 (curl exit 0 -- the CDN itself answers 403 to a bare GET of /)
$ ssh vr1-dc0-rack 'curl -sS -o /dev/null -w "code=[%{http_code}]" -m 15
-x http://10.12.8.2:8000 https://api.snapcraft.io:25/'
curl: (56) CONNECT tunnel failed, response 403
code=[000]
=> A PROXY DENIAL and a SERVER 403 are DISTINGUISHABLE, measured:
proxy denial : curl exit 56, stderr "CONNECT tunnel failed, response 403",
%{http_code} = 000
server 403 : curl exit 0, %{http_code} = 403
That difference is what lets `check` assert the DENY DIRECTION (the
destination allowlist is actually in force) separately from the ALLOW
direction, from the SAME source address.
8. JUJU'S SNAP PROXY KEYS EXIST IN THIS EXACT CONTROLLER
------------------------------------------------------------------------------
$ ssh vr1-dc0-rack 'juju version; juju model-config -m controller --format=yaml
| grep -iE "snap|proxy"'
3.6.27-genericlinux-amd64
apt-http-proxy: apt-https-proxy: apt-ftp-proxy: apt-no-proxy:
http-proxy: https-proxy: ftp-proxy: no-proxy: proxy-ssh:
juju-http-proxy: juju-https-proxy: juju-ftp-proxy: juju-no-proxy:
lxd-snap-channel:
snap-http-proxy: snap-https-proxy:
snap-store-assertions: snap-store-proxy: snap-store-proxy-url:
(identical set from `juju model-defaults`)
=> the five key names in the ruling note are CONFIRMED against the live
controller, not quoted from documentation.
NOTE ON THE MODEL: at capture time `juju models` showed `vr1-dc0` in state
`destroying` (operator-driven, separate from this work). Probes above ran
against machines that still existed. Nothing here mutated the model.
9. VENDOR RESEARCH -- WHAT SNAPD ACTUALLY HONOURS
------------------------------------------------------------------------------
Sources opened and quoted (not summarised from memory):
snapd config mechanism
https://snapcraft.io/docs/reference/administration/system-options/
"snap set system proxy.http=... / proxy.https=..."
CAVEAT FOUND AND NOT SMOOTHED OVER: that page carries an
"Available only on Ubuntu Core" note adjacent to the proxy section. THREE
primary sources contradict it, and code is the stronger witness:
https://raw.githubusercontent.com/canonical/snapd/master/overlord/configstate/configcore/proxy.go
supportedConfigurations["core.proxy.http"|"...https"|"...ftp"|
"...no-proxy"|"...store"] = true -- registered with NO core-only flag
envFilePath = "/etc/environment" -- a CLASSIC-system file
handleProxyConfiguration() writes http_proxy/https_proxy/ftp_proxy/no_proxy
=> `snap set system proxy.*` writes /etc/environment. It does NOT write a
systemd drop-in.
https://raw.githubusercontent.com/canonical/snapd/master/data/systemd/snapd.service.in
"EnvironmentFile=-/var/lib/snapd/environment/snapd.conf" (+ the built-in one)
https://raw.githubusercontent.com/canonical/snapd/master/httputil/transport.go
"Proxy: http.ProxyFromEnvironment"
=> snapd's HTTP transport reads the PROCESS environment; a
snapd.service.d drop-in with Environment=https_proxy= works by the same
path. Canonical's own k8s docs use exactly that drop-in shape:
https://documentation.ubuntu.com/canonical-kubernetes/latest/snap/howto/networking/proxy/
https://forum.snapcraft.io/t/preferred-method-for-configuring-proxy-settings-for-snapd/11743
chipaca (Canonical), 2019-06-11: "A proxy set via snap set overrides that
obtained from the environment"; the two approaches "are ... equivalent".
WHICH HOSTS, AND WHETHER CONNECT IS REQUIRED
https://snapcraft.io/docs/reference/administration/network-requirements/
https://forum.snapcraft.io/t/network-requirements/5147 (CDN list 2021-10-25)
api.snapcraft.io:443 dashboard.snapcraft.io:443 login.ubuntu.com:443
*.snapcraftcontent.com:443 storage.snapcraftcontent.com:443
canonical-{lgw01,lcy01,lcy02,bos01}.cdn.snapcraftcontent.com:443
cloudfront.cdn.snapcraftcontent.com:443
*.cdn.snapcraft.io:443 [deprecated]
=> EVERY documented endpoint is :443. The proxy only ever sees
`CONNECT host:443`, so a CONNECT-capable forward proxy is REQUIRED and
`snap-http-proxy` is effectively inert for snapd's own traffic. Set both
anyway; `snap-https-proxy` is the one that matters.
https://raw.githubusercontent.com/canonical/snapd/master/store/store.go
the SAME client (opts.Proxy = s.cfg.Proxy) serves API calls AND downloads
=> THE CDN FETCH IS PROXIED TOO. A destination allowlist that names only
api.snapcraft.io would establish the session and then fail the download.
WHAT JUJU DOES WITH THE KEYS
https://canonical.com/juju/docs/juju-cli/3.6/reference/juju-cli/list-of-juju-cli-commands/model-config/
snap-http-proxy "The HTTP proxy value for installing snaps"
snap-https-proxy "The HTTPS proxy value for installing snaps"
snap-store-proxy "The snap store proxy for installing snaps"
snap-store-proxy-url / snap-store-assertions
http-proxy/https-proxy -> instance HTTP_PROXY; juju-* -> charm hook env
=> three DIFFERENT surfaces. Setting the wrong one will not fix snaps.
https://raw.githubusercontent.com/juju/juju/3.6/internal/worker/proxyupdater/proxyupdater.go
handleSnapProxyValues(): maybeAddSettings("proxy.http", ...), ("proxy.https", ...)
then RunFunc(noStdIn, "snap", append([]string{"set","system"}, snapSettings...))
skipped only on CentOS
=> juju literally shells out `snap set system proxy.http=... proxy.https=...`,
i.e. mechanism = section 9 paragraph 1. It is a WATCHER
(WatchForProxyConfigAndAPIHostPortChanges), so setting the key AFTER the
machines exist is expected to propagate.
snapd proxy.store is a 32-character STORE ID needing a signed assertion:
configcore/proxy.go: "cannot set proxy.store to %q without a matching
store assertion"; client shape documented at
https://ubuntu.com/enterprise-store/docs/page/how-to/devices/
=> `snap-store-proxy` / `-url` / `-assertions` are the ENTERPRISE STORE
product, NOT a forward proxy. LEAVE THEM EMPTY.
SQUID VS TINYPROXY, AND THE CACHING QUESTION
https://packages.ubuntu.com/jammy/squid -- main, security-supported
https://packages.ubuntu.com/jammy/net/tinyproxy -- universe
https://ubuntu.com/security/cve?package=tinyproxy
jammy: 2 Fixed, 2 Not affected, SIX in "Needs evaluation"
https://tinyproxy.github.io/ + tinyproxy.conf(5)
ConnectPort: "If no ConnectPort line is found, then all ports are allowed."
=> tinyproxy's CONNECT default is OPEN; squid's is deny-except-443.
https://www.squid-cache.org/Doc/config/http_port/ "Squid normally listens to port 3128"
https://www.squid-cache.org/Doc/config/cache/ `cache deny all` documented
https://wiki.squid-cache.org/Features/HTTPS
"the proxy establishes a TCP connection to the specified server, responds
with an HTTP 200 (Connection Established) ... without understanding or
interpreting the tunneled traffic"; "Squid does not have access to those
encrypted messages."
https://wiki.squid-cache.org/Features/SslBump
"Decrypting HTTPS tunnels constitutes a man-in-the-middle attack ..."
=> THERE IS NO CACHING BENEFIT FOR SNAPS. Every endpoint is :443, so a
CONNECT proxy sees opaque tunnels and can cache nothing without TLS
interception, which is explicitly not wanted. Choose on security
support and simplicity, NOT on caching. (This is the opposite of the
apt case, where dc1's apt-cacher-ng genuinely caches plain-HTTP apt.)
https://github.com/chobits/ngx_http_proxy_connect_module
a third-party module exists precisely because stock nginx has no CONNECT.
=> nginx is NOT a candidate.
WHY NOT THE ENTERPRISE STORE (ex Snap Store Proxy)
https://ubuntu.com/enterprise-store/docs/page/how-to/install/
prerequisites: an LTS server, a DOMAIN NAME, a PostgreSQL instance with
btree_gist + intarray, and a store REGISTRATION step.
https://ubuntu.com/enterprise-store/docs/how-to/airgap/
offline mode additionally needs registration on an internet-connected
machine, an offline-snap-store.tar.gz carried in, and an export/import cycle.
https://ubuntu.com/blog/canonical-announces-the-enterprise-store (2026-07-21)
"included as part of Ubuntu Pro"
=> for ~12 clients behind a rack that already has egress, this is a domain
name, a database, a Canonical account dependency and a per-node `snap
ack` -- against `apt install squid` plus two model-config keys.
10. WHAT COULD NOT BE VERIFIED (stated, not filled in)
------------------------------------------------------------------------------
a. WHETHER A RUNNING snapd RE-READS THE PROXY WITHOUT A RESTART. The chain is
juju -> `snap set system proxy.*` -> snapd rewrites /etc/environment ->
snapd.service reads it via EnvironmentFile= AT START. Evidence it may not
re-read live: https://bugs.launchpad.net/juju/+bug/1737332 (Fix Released
2018-10-14) "snapd *MUST* be restarted though after proxy.conf is created as
naturally there is no sane way to 'reload' environment variables"; and
https://bugs.launchpad.net/snapd/+bug/1791587 "snapd ignores proxy settings
set via core snap" (Fix Released, snapd 2.36.1, 2018-11-22) whose root cause
was exactly this. It could NOT be determined whether that fix made snapd
apply the proxy in-memory. OPERATIONAL CONSEQUENCE, carried into the script
header and the runbook wording: after setting the key, verify per node with
BOTH `snap get system proxy` AND a real egress probe; if the first passes and
the second fails, `systemctl restart snapd` and re-probe.
b. WHICH snapd VERSION INTRODUCED `snap set system proxy.*`. Two search
summaries disagree (2.28 vs 2.33) and neither was from a page that was
opened. MOOT here -- both racks measure snapd 2.76.1 and juju 3.6 emits
`snap set system` regardless.
c. THE "Available only on Ubuntu Core" NOTE on the system-options page (9a).
Code, the /etc/environment write target and juju's unconditional use all say
it does not apply to classic; no doc source repo was found to confirm the
note is an authoring defect. The settling test is one command on one node:
`sudo snap set system proxy.https=http://<listen>:3129 && sudo snap get system proxy`.
d. NOTHING WAS PROVEN END-TO-END. No proxy was installed, so no snap has been
installed through one. The build's own `check` and `node` subcommands are the
gates that will prove it at apply time; the harness green is a FIXTURE green.
e. dc1 HAS NOT BEEN PROBED FROM A CONTAINER. dc1 has no deployed model yet, so
the metal-admin-source finding (section 1) is measured on dc0 only. The dc1
site row takes its CIDRs from `lib-net.sh`, and `node dc1` is the assertion
that will prove it there.
f. The squid config generated by this build has never been parsed by a squid
binary. `install` runs `squid -k parse -f <conf>` before enabling anything,
so a syntax error fails loud at apply time rather than silently.
g. THE DENY-SHAPE MEASUREMENT IS FROM A *PORT* DENY, NOT A *DSTDOMAIN* DENY.
Section 7's `curl: (56) CONNECT tunnel failed, response 403` was produced by
MAAS's squid refusing `api.snapcraft.io:25` under `http_access deny CONNECT
!SSL_ports` -- a PORT rule. This build's deny probe trips a DSTDOMAIN rule
(`http_access deny !snap_store`). Squid answers both with 403 at CONNECT, so
the two should be byte-identical to curl, but that path was NOT measured.
THE FAILURE MODE IS SAFE: check's deny branch treats anything that is neither
"403 at CONNECT" nor "curl exit 0" as REFUSE, so an unexpected shape can only
withhold a pass, never grant one. Confirm at apply time, once: run the deny
probe by hand against the built proxy and read the exact curl line.
h. TWO OF check's PROBES ARE LIVENESS-COUPLED TO CANONICAL. The store-API probe
and the CDN probe reach real third-party hosts, so an upstream outage or a
DNS failure makes a healthy proxy REFUSE. That is deliberate -- a probe that
cannot distinguish "the CDN is down" from "the proxy is misconfigured" must
not guess -- but it means a REFUSE is not by itself evidence about the proxy,
and the script header says so where a triaging operator will read it. The
static half of the same property (that the CDN wildcard is PRESENT in
STORE_DOMAINS) is asserted by the config diff and by harness case T16, which
have no external dependency at all.
11. MUTATION PASS ON tests/dc-snap-proxy/run-tests.sh (2026-07-31)
------------------------------------------------------------------------------
"A checker that cannot fail is not a gate", applied to the harness itself. The
2026-07-30 precedent is explicit: three freshly-written assertions were deleted
one at a time and the suite stayed green every time -- they were decoration.
METHOD. A one-shot driver (/tmp, NOT a repo artifact) applied ONE exact string
replacement to scripts/dc-snap-proxy.sh at a time, ran the suite, restored the
file and re-verified its sha256 against the original. Every mutation neuters a
PREDICATE, never only a message: `miss` -> `say`, `refuse` -> `say`,
`nno` -> `nok`, `elif <test>` -> `elif false`, a value swapped for a plausible
wrong one, or a forbidden construct introduced.
RESULT: 53 mutations, 53 KILLED, 0 survived. Baseline and final state both
`dc-snap-proxy: 52 passed, 0 failed`; the script restored byte-identical after
every mutation (sha256 verified, final sha_ok=True).
M01 dc0 LISTEN 10.12.8.4 -> .5 -> T3 red (11 red total)
M02 dc0 edge gw -> .254 -> T4 red
M03 dc0 client ACL -> metal-internal 10.12.12.0/22 -> T5 red <-- the
exact wrong turn the container measurement prevented
M04 dc1 LISTEN -> .5 -> T6 red
M05 dc1 edge gw -> .254 -> T7 red
M06 dc1 client ACL -> metal-internal 10.12.72.0/22 -> T8 red
M07 port 3129 -> MAAS's 3128 -> T9 red
M08 hardcode virbr2 instead of util_bridge -> T10 red
M09 net unit renamed to *-cache-proxy-net.service -> T11 red
M10 operative "dc-mirror.sh install" referral added -> T11b red
M11 ip addr replace -> ip addr add -> T12 red
M12 MEASURED/RULED tags dropped -> T13 red
M13 install root guard removed -> T14 red
M14 snap-https-proxy key name mangled -> T15 red
M15 allowlist narrowed to api.snapcraft.io only -> T16 red
M16 http_port LISTEN:PORT -> wildcard PORT -> T17 red
M17 install stops disabling the packaged squid -> T18 red
M18 install stops parsing the config before enable -> T19 red
(first attempt was a MALFORMED mutation -- `true # ...` commented out the
`then` and broke `bash -n`, so T2 and 28 others went red while T19 did
not. Recorded rather than quietly re-rolled: a mutation that breaks syntax
tests nothing. Redone as `if false; then` + `:`, and T19 went red alone.)
M19 install apt-installs directly -> T20 red
M20 bad mode stops exiting 2 -> T21 red
M21 unknown site stops exiting 2 -> T22 red
M22 check_file asserts EXISTENCE not content -> T24 red <-- the
dc-mirror.sh false-green class, reproduced and caught
M23 missing artifact reported as OK -> T25 red
M24 unit-enabled miss softened -> T26 red
M25 packaged squid ACTIVE not detected -> T27 red
M26 packaged squid ENABLED not detected -> T27b red
M27 wildcard bind accepted as OK -> T28 red
M28 absent listener accepted -> T29 red
M29 absent LISTEN alias accepted -> T30 red
M30 unresolvable bridge defaults to success -> T31 red
M31 missing default route accepted -> T32 red
M32 check drops the channel-map content assertion -> T33 red
M33 check drops the sha3-384 assertion -> T34 red
M34 check ignores a non-200 store status -> T35 red
M35 unreachable proxy defaults to success -> T36 red
M36 refused CDN tunnel accepted -> T37 red
M37 unrecognised CDN result defaults to success -> T38 red
M38 OPEN proxy accepted (deny direction gone) -> T39 red <-- the
assertion that proves the allowlist is in force, not merely written down
M39 unrecognised deny result defaults to success -> T40 red
M40 missing squid package accepted -> T41 red
M41 no curl defaults to success -> T42 red
M42 node: ACL denial reported as OK -> T44 red
M43 node: unreachable proxy defaults to success -> T45 red
M44 node: drops the channel-map assertion -> T46 red
M45 node: OPEN proxy accepted -> T47 red
M46 node: absent https_proxy accepted -> T48 red
M47 node: wrong https_proxy accepted -> T49 red
M48 node makes a rack-side call (systemctl) -> T50 red
M49 check: REFUSE treated as a pass -> T31/36/38/40/42 red
M50 node: REFUSE treated as a pass -> T45 red
M51 check: FAIL treated as a pass -> 15 cases red
M52 node: FAIL treated as a pass -> 5 cases red
M53 ENV_FILE path moved -> T43 red (the node
green baseline is itself killable, so it is not a tautology)
THREE OF THE HARNESS'S OWN CASES WERE FOUND TOO WEAK WHILE BUILDING THIS, and
were STRENGTHENED rather than left passing -- each would have stayed green after
the thing it names was removed:
T16 grepped the WHOLE FILE for `.snapcraftcontent.com`, which the header
discusses in prose, so narrowing the actual STORE_DOMAINS value left it
green. Now scoped to the `STORE_DOMAINS=` line.
T18 grepped the whole file for `systemctl disable --now squid`, which check's
own failure MESSAGE contains, so install could stop doing it. Now scoped
to do_install's body.
T20 matched `apt-get install` inside the guidance STRING, so it read green
whether or not the script apt-installed. Now blanks double-quoted strings
first and matches an INVOCATION position.
Same class, found the same way: T10 and T50 had to blank comments / quoted
strings for the opposite reason -- they were RED on legitimate prose (measured
`virbr5` evidence in a site-row comment; an `ip route get` hint inside a failure
message). Rewording the prose to satisfy a regex would have been gaming the
test, so the tests were fixed to assert on CODE.
WHAT THE MUTATION PASS DOES NOT PROVE: that the assertions are the RIGHT ones,
or that the proxy works. It proves only that none of them is decoration.
