diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 4fd56c8..82340eb 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1488,6 +1488,39 @@ path (ruling 2), which is the D-135 items 2-3 gap.** Progression across the session, all measured: 22 error -> 17 (backports) -> 4 (prefer-ipv6) -> 7 units in the single snap class (the count rose because more units reached the snap stage, not because more broke). + **RULING 2 BUILT IN THE REPO 2026-07-31 -- AND NOT APPLIED. NOTHING IS CLOSED BY THIS.** + `scripts/dc-snap-proxy.sh` (+ `tests/dc-snap-proxy/`, 52 cases) implements the ruled forward + proxy as a DEDICATED squid instance ON THE RACK at the D-134 utility `.4:3129`, site-keyed + dc0/dc1, CONNECT-only and restricted to the Canonical-documented snap-store/CDN hosts. + **NO cloud mutation was made**: no package installed, no service started, no `juju + model-config` set. **The D-135 items 2-3 gap is therefore still OPEN** -- the build is + repo-side only and the operator applies it separately. + **Four measurements that changed the design, all captured in + `docs/audit/stage5-snap-proxy-measurements-20260731.txt`:** (i) the three failing + applications are LXD CONTAINERS and, measured from inside `mysql-innodb-cluster/0`, they + SOURCE FROM METAL-ADMIN (`eth0 10.12.8.122`, `ip route get 10.12.8.4 -> src 10.12.8.122`) + while having NO default route -- an ACL built on the `10.12.12.116` that `juju status` + displays would have denied every client the proxy exists for; (ii) **port 3128 AND 8000 are + already held, WILDCARD-bound, by MAAS's own squid on BOTH racks**, so the proxy takes 3129 + and the PACKAGED `squid.service` (whose squid.conf pins 3128 and whose `/etc/default/squid` + is empty) can only crash-loop -- `install` disables it; (iii) the utility `.4` is ALREADY + aliased on both racks, so **no new D-134 octet is needed**, which is what the ruling note + required; (iv) squid-vs-MAAS-squid shm coexistence is MEASURED not reasoned (`snap.maas. + squid-cf__*` versus bare `squid-cf__*`). **LOGGED, NOT ADOPTED: MAAS's own squid ALREADY + CONNECT-proxies api.snapcraft.io** and was measured working from the rack, the controller VM + AND the failing container -- so `juju model-config snap-https-proxy=http://10.12.8.6:8000` + would likely work today with no rack build at all. It was not taken because that config is + MAAS-generated under a per-revision `/var/snap/maas/current/` path, has NO destination + restriction, and is the hidden-coupling anti-pattern D-135 already has a scar from -- but it + is a real operator option at apply time, not a dismissal. + **TWO THINGS THIS BUILD DOES NOT PROVE, said plainly:** no proxy has ever been installed, so + no snap has been installed through one (the harness green is a FIXTURE green); and whether a + RUNNING snapd re-reads the proxy without a restart is UNVERIFIED (snapd writes + `/etc/environment` and reads it via `EnvironmentFile=` at service start; LP#1737332, + LP#1791587) -- so per client verify BOTH `snap get system proxy` AND a real egress probe. + **A BUILD-TIME CHOICE THE OPERATOR HAS NOT RULED ON:** the destination allowlist. The ruling + names the mechanism, not the scope; the alternative is allowing CONNECT to any `:443`, which + hands the node planes general HTTPS egress. **NODE NAMING -- CANARY TESTED 2026-07-31, and the answer is that a MAAS rename alone is COSMETIC-ONLY.** Operator asked for the `juju status` `Inst id` values (`mint-roughy`, `pure-condor`, `able-puma` ...) to become role-based. **CORRECTION TO AN EARLIER CLAIM IN @@ -1527,6 +1560,30 @@ rather than remembered. Roosevelt shape: fold these into D-136's per-DC render pipeline so model-config comes from the same per-DC values files that already produce the overlays. **DC1 LESSON: set model-defaults at controller bootstrap, BEFORE its first `add-model`.** + **>>> MODEL TEARDOWN 2026-07-31: STALLED, THEN FORCED. `Model destroyed.` <<<** The plain + `destroy-model` STALLED and would not self-resolve -- `attempt 30 to destroy model failed + (will retry): model not empty, found 26 machines, 37 applications`, flat for ~19 minutes + with the app set BYTE-IDENTICAL across a 12-minute name-level diff. **MECHANISM MEASURED, + and it makes the stall terminal rather than slow: ALL 26 machine/container agents were + `stopped`, so NO hook could execute at all** -- the destroy worker asks and nothing answers. + Residue split 35 error / 2 maintenance: 11 haclusters at `hook failed: "stop"`, four at + `hook failed: "install"`, five at `identity-service-relation-departed`. Units already in + `error` from the snap failures could never run their teardown hooks, so it was never going + to drain. **MAAS WAS NOT THE BOTTLENECK** -- the six nodes juju did release went to + `Ready / owner None` in minutes (`enable_disk_erasing_on_release=false`); juju simply never + issued a release for the other three. **`juju destroy-model --force --no-wait` cleared it + (18 -> 5 -> 2 machines, then `Model destroyed.`), and NO NODES WERE STRANDED** -- read back, + all NINE role nodes are `Ready / owner=None` and no `maas machine release` was needed or + run. `subtle-grouse` correctly remains `Deployed` under `juju-vr1-dc0`: it is the D-104 + controller VM in the `controller` model, not part of the destroyed one. + **INSTRUMENT ERROR, OWNED: I reported "already down to 0 machines / 0 units" from the + `juju models` SUMMARY COLUMNS while `juju status -m vr1-dc0` simultaneously read 26 machines + / 37 applications.** The summary zeroes during `destroying` and is not a progress signal. + The real tell -- three control nodes stuck `Deployed` -- was visible and I explained it away + as "more containers to work through" instead of checking. **Standing lesson: during a + teardown, `juju status -m ` is the instrument; `juju models` counts are not.** + **ENVIRONMENT IS NOW READY FOR `add-model` + `deploy`** except for the snap proxy, which is + the one remaining build. **memcached SCALED to 3 in `overlays/dc-ha-scaleup.yaml`** (operator-directed 2026-07-31); its exclusion comment is RE-POINTED rather than left stale. **`ceph-rbd-mirror` is PINNED, NOT SETTLED** -- gap register **item 22** carries four options and the recommendation diff --git a/docs/audit/stage5-snap-proxy-measurements-20260731.txt b/docs/audit/stage5-snap-proxy-measurements-20260731.txt new file mode 100644 index 0000000..3272802 --- /dev/null +++ b/docs/audit/stage5-snap-proxy-measurements-20260731.txt @@ -0,0 +1,530 @@ +stage5-snap-proxy-measurements-20260731.txt +=========================================== +Read-only measurement capture backing `scripts/dc-snap-proxy.sh` (the REPO SIDE +of D-135/D-107 RULING 2, 2026-07-31: "HTTP(S) forward proxy in the DC utility +band + juju snap-https-proxy (Recommended)"). + +SCOPE OF THIS CAPTURE. Every command below is READ-ONLY. NO cloud mutation was +made in the session that produced it: no `juju config`, no `juju model-config`, +no package installed, no service started or stopped, no file written on any rack +or node. The proxy is BUILT IN THE REPO and NOT APPLIED. Two side effects are +declared rather than hidden: the `curl` probes traversed MAAS's existing squid +and may have added entries to ITS cache, and `apt-get download squid` wrote a +.deb into `/tmp` ON THE JUMPHOST (not on any rack) so the shipped squid config +could be read instead of guessed. + +Host/date: vcloud jumphost, 2026-07-31. Branch dc-dc-stage5-preconditions. + + +1. THE THREE FAILING APPLICATIONS ARE LXD CONTAINERS, AND THEIR SOURCE ADDRESS + TOWARD THE RACK IS ON METAL-ADMIN, NOT METAL-INTERNAL +------------------------------------------------------------------------------ +This is the measurement the whole design turns on: it decides the squid client +ACL and whether the D-134 utility `.4` is reachable by the clients at all. +`juju status -m vr1-dc0` shows `mysql-innodb-cluster/0` at `10.12.12.116`, which +is NOT a D-134 node octet (.100-.102 / .120-.121 / .150-.153) -- so the failing +units are containers, and the address `juju status` prints is only ONE of their +legs. Measured from inside the container: + + $ juju ssh -m vr1-dc0 mysql-innodb-cluster/0 'ip -4 -o addr show; ip route show; + ip route get 10.12.8.4' + 60: eth0 inet 10.12.8.122/22 brd 10.12.11.255 scope global eth0 + 62: eth1 inet 10.12.12.116/22 brd 10.12.15.255 scope global eth1 + 10.12.8.0/22 dev eth0 proto kernel scope link src 10.12.8.122 + 10.12.12.0/22 dev eth1 proto kernel scope link src 10.12.12.116 + 10.12.8.4 dev eth0 src 10.12.8.122 uid 1000 + cache + + => the container has BOTH a metal-admin leg (.8.122) and a metal-internal leg + (.12.116); the kernel selects eth0 / src 10.12.8.122 for the utility `.4`; + and there is NO default route at all. + + $ ... 'curl -sS -o /dev/null -w "%{http_code}" -m 12 + -H "Snap-Device-Series: 16" https://api.snapcraft.io/v2/snaps/info/core' + curl: (7) Couldn't connect to server http=000 <-- DIRECT: FAILS + + $ ... same URL, but -x http://10.12.8.2:8000 + http=200 <-- VIA A PROXY: WORKS + + => CONFIRMS the D-135 items 2-3 root cause from the client side, and confirms + the mechanism the ruling chose: an in-DC CONNECT proxy on the rack works + from the exact unit that fails. + => CONSEQUENCE FOR THE BUILD: the squid client ACL is the METAL-ADMIN /22 + (`10.12.8.0/22` dc0, `10.12.68.0/22` dc1), not metal-internal. A + metal-internal ACL would have denied every client the proxy exists for. + +Corroborating node-side capture (`juju ssh -m vr1-dc0 0`, a bare role node): + br-ex 10.12.4.100/22, br-enp1s0 10.12.8.100/22, br-enp3s0 10.12.12.100/22, + br-enp5s0 10.12.32.100/22, br-enp6s0 10.12.36.100/22, br-enp4s0 10.12.16.100/22 + default via 10.12.4.1 dev br-ex proto static + DIRECT api.snapcraft.io -> http=200 (node has a br-ex default route) + via 10.12.8.2:8000 -> http=200 + http://10.12.8.4/ (mirror) -> http=200 + sudo snap get system proxy -> {"proxy":{"http":"","https":"","store":""}} + + HONEST READING, because this one is easy to misread: the bare node DOES reach + the internet directly today via br-ex. The CONTAINERS do not, and they are what + fails. Nothing here should be read as "the nodes are already airgapped". + +Plane CIDR authority (repo, not inferred by mirroring dc0): `scripts/lib-net.sh` + vr1-dc0 PLANE_CIDRS = 10.12.4/8/12/16/32/36.0/22, metal-admin = 10.12.8.0/22 + vr1-dc1 PLANE_CIDRS = 10.12.64/68/72/76/80/84.0/22, metal-admin = 10.12.68.0/22 + + +2. THE UTILITY `.4` ALREADY EXISTS ON BOTH RACKS; NO NEW OCTET IS NEEDED +------------------------------------------------------------------------------ + $ ssh vr1-dc0-rack 'ip -4 -o addr show; ip route show default' + virbr2 inet 10.12.8.2/22 | 10.12.8.3/22 secondary | 10.12.8.4/22 secondary + virbr5 inet 10.12.4.2/22 ; enp1s0 inet 172.31.0.2/30 + default via 10.12.4.1 dev virbr5 + + $ ssh vr1-dc1-rack 'ip -4 -o addr show; ip route show default' + virbr6 inet 10.12.68.2/22 | 10.12.68.3/22 secondary | 10.12.68.4/22 secondary + virbr4 inet 10.12.64.2/22 ; enp1s0 inet 172.31.0.6/30 + default via 10.12.64.1 dev virbr4 + + => the D-134 utility `.4` (artifact service) is already aliased on both racks' + metal-admin bridge, held by `dc0-mirror-net.service` on dc0 and by + `dc1-cache-proxy-net.service` on dc1. Placing the snap proxy ON THE RACK at + `.4:` therefore needs NO new ruled octet -- which is what the ruling + note asked for. Both racks are Ubuntu 24.04.4 LTS. + + +3. PORT 3128 IS ALREADY TAKEN -- BY A SQUID, ON BOTH RACKS +------------------------------------------------------------------------------ + $ ssh vr1-dc0-rack 'ss -ltn' (dc0, abridged to the relevant rows) + LISTEN 10.12.4.2:53 10.12.8.3:53 0.0.0.0:22 0.0.0.0:80 + LISTEN 0.0.0.0:5248 0.0.0.0:5247 *:5249 172.31.0.2:5280 + LISTEN *:8000 *:3128 + + $ ssh vr1-dc1-rack 'ss -ltn' (dc1) + LISTEN 0.0.0.0:3142 (apt-cacher-ng) 10.12.68.3:53 0.0.0.0:22 0.0.0.0:7911 + LISTEN *:8000 *:3128 + + $ ssh vr1-dc0-rack 'sudo ss -ltnp | grep -E ":(3128|8000) "' + LISTEN *:8000 users:(("squid",pid=1871633,fd=13)) + LISTEN *:3128 users:(("squid",pid=1871633,fd=12)) + + $ ssh vr1-dc0-rack 'sudo tr "\0" " " < /proc/1871633/cmdline' + squid -N -d 5 -f /var/snap/maas/41649/proxy/maas-proxy.conf + + => 3128 AND 8000 are held, on BOTH racks, by MAAS's own squid, WILDCARD-bound. + A wildcard bind means a later `10.12.8.4:3128` bind would fail. 3142 is + dc1's apt-cacher-ng, 80 is dc0's mirror nginx. + => CONSEQUENCE: the snap proxy takes **3129**, bound to the LISTEN address + specifically, and must NOT be the packaged `squid.service` (see 5). + => NOTE: MAAS's squid itself runs with `-f `, i.e. the exact + separate-instance shape this build uses. That is a precedent, not a guess. + + +4. MAAS'S SQUID ALREADY CONNECT-PROXIES THE SNAP STORE. WHY IT WAS NOT USED. +------------------------------------------------------------------------------ + $ ssh vr1-dc0-rack 'sudo grep -vE "^\s*#|^\s*$" /var/snap/maas/current/proxy/maas-proxy.conf' + ... acl localnet src 10.12.8.0/22 | 10.12.12.0/22 | 10.12.68.0/22 | 10.12.72.0/22 ... + acl SSL_ports port 443 + acl CONNECT method CONNECT + http_access deny CONNECT !SSL_ports + http_access allow localnet + http_access deny all + http_port 3128 transparent + http_port 8000 + + MEASURED to work, three ways: + from the rack itself -x http://10.12.8.2:8000 -> api.snapcraft.io http=200 + from the juju ctrl VM -x http://10.12.8.2:8000 -> http=200 + -x http://10.12.8.6:8000 -> http=200 + from mysql-innodb-cluster/0 (container) -x http://10.12.8.2:8000 -> http=200 + + And juju/MAAS ALREADY point machines at a MAAS squid for apt: + $ juju ssh -m controller 0 'grep -rn -i proxy /etc/apt/apt.conf.d/' + /etc/apt/apt.conf.d/90curtin-aptproxy:1:Acquire::http::Proxy "http://10.12.8.6:8000/"; + + SO A WORKING FORWARD PROXY ARGUABLY ALREADY EXISTS. It was NOT adopted, and + the reason is recorded here rather than left implicit: + (a) `maas-proxy.conf` is MACHINE-GENERATED by MAAS and regenerated whenever + MAAS's subnet/proxy settings change -- nothing in this repo owns it, and + no diff-against-generator check could ever assert its content. + (b) It lives under the per-revision path `/var/snap/maas/current/` (measured + real path `/var/snap/maas/41649/...`), which the operating skill already + records as NOT surviving a snap refresh for the power keys. + (c) It has NO destination restriction: `http_access allow localnet` permits + CONNECT to ANY :443 host from every plane MAAS knows. Adopting it as the + snap path would silently give the node planes general HTTPS egress. + (d) It is the hidden-coupling anti-pattern this repo already has a scar from + (dc-cache-proxy.sh reusing dc-mirror.sh's net layer, D-135 amendment + 2026-07-27): a service whose prerequisites are owned by another service's + machinery cannot be torn down or rebuilt independently. + THIS IS AN OPERATOR-VISIBLE ALTERNATIVE, NOT A DISMISSAL. If the operator + prefers it at apply time, the equivalent live change is just + `juju model-config snap-https-proxy=http://10.12.8.6:8000` with no rack build + at all -- cheaper, and worse on (a)-(d). + + +5. THE SHIPPED UBUNTU SQUID: MEASURED FROM THE PACKAGE, NOT ASSUMED +------------------------------------------------------------------------------ + $ ssh vr1-dc0-rack 'apt-cache policy squid tinyproxy' (identical on dc1) + squid Candidate: 6.14-0ubuntu0.24.04.4 noble-updates/**main** + tinyproxy Candidate: 1.11.1-3ubuntu0.1 noble-updates/**universe** + + On the JUMPHOST (also noble 24.04.4, same candidate version), the .deb was + downloaded and unpacked read-only so the shipped config could be READ: + $ apt-get download squid && dpkg-deb -x squid_6.14-0ubuntu0.24.04.4_amd64.deb x + $ grep -vE '^\s*#|^\s*$' x/etc/squid/squid.conf + acl localnet src 0.0.0.1-0.255.255.255 / 10.0.0.0/8 / 100.64.0.0/10 / + 169.254.0.0/16 / 172.16.0.0/12 / 192.168.0.0/16 / fc00::/7 / fe80::/10 + acl SSL_ports port 443 + acl Safe_ports port 80 21 443 70 210 1025-65535 280 488 591 777 + http_access deny !Safe_ports + http_access deny CONNECT !SSL_ports + http_access allow localhost manager + http_access deny manager + http_access allow localhost + http_access deny to_localhost + http_access deny to_linklocal + include /etc/squid/conf.d/*.conf + http_access deny all + http_port 3128 + coredump_dir /var/spool/squid + refresh_pattern ... + + $ cat x/usr/lib/systemd/system/squid.service + [Service] Type=notify PIDFile=/run/squid.pid Group=proxy + RuntimeDirectory=squid RuntimeDirectoryMode=0775 + ExecStartPre=/usr/sbin/squid --foreground -z + ExecStart=/usr/sbin/squid --foreground -sYC + ExecReload=/bin/kill -HUP $MAINPID KillMode=mixed NotifyAccess=all + + $ cat x/etc/default/squid + (EMPTY -- there is no SQUID_ARGS / SQUID_CONFIG_FILE indirection to hook) + + THREE CONSEQUENCES, each load-bearing: + (i) `include /etc/squid/conf.d/*.conf` sits BEFORE `http_access deny all`, + so a conf.d drop-in CAN add an allow rule -- but it CANNOT unset + `http_port 3128`, and 3128 is held by MAAS's squid (section 3). The + packaged unit would therefore FATAL on bind. A conf.d drop-in is + structurally unusable here. + (ii) The default already carries `http_access deny CONNECT !SSL_ports` and a + terminal `http_access deny all` -- the posture we want is squid's own + default, which is the main reason to prefer it over tinyproxy (whose + `ConnectPort` default is documented as "all ports allowed"). + (iii) `/etc/default/squid` being empty means there is no supported way to point + the packaged unit at another config file. A DEDICATED instance (own + config + own unit + own pid file) is the only clean shape, and + `install` must `systemctl disable --now squid` so the packaged unit does + not crash-loop on 3128 and poison triage. + + $ zcat x/usr/share/man/man8/squid.8.gz | grep -A2 '"\\-n name"' + "-n name Specify Windows Service name ..." + => `-n` is documented as Windows-only, so the instance is separated by `-f` + plus an explicit `pid_filename`, NOT by `-n`. + + +6. SHARED-MEMORY COEXISTENCE WITH MAAS'S SQUID: MEASURED, NOT REASONED +------------------------------------------------------------------------------ +Two squids on one host can collide on `/dev/shm` segment names. Measured: + + $ ssh vr1-dc0-rack 'ls -la /dev/shm/' + -rw------- snap_daemon snap.maas.squid-cf__metadata.shm + -rw------- snap_daemon snap.maas.squid-cf__queues.shm + -rw------- snap_daemon snap.maas.squid-cf__readers.shm + $ ssh vr1-dc0-rack 'sudo ls -la /run/ | grep -i squid' -> (no /run/squid*) + + => snapd namespaces MAAS's segments behind a `snap.maas.` prefix; a host squid + creates bare `squid-cf__*`. NO COLLISION. And MAAS's squid runs `-N` with no + pid file, so `/run/-snap-proxy.pid` collides with nothing either. + => This is the one coexistence question dc-cache-proxy.sh had to leave as + "REASONED, NOT MEASURED". Here it is measured. + + +7. SNAP-STORE ENDPOINT SHAPES, MEASURED, SO THE CHECK PREDICATES ARE REAL +------------------------------------------------------------------------------ + $ ssh vr1-dc0-rack 'curl -sS -m 25 -H "Snap-Device-Series: 16" + https://api.snapcraft.io/v2/snaps/info/core | head -c 400' + {"channel-map":[{"channel":{"architecture":"amd64","name":"stable",...}, + "download":{"deltas":[],"sha3-384":"13c728f16430310ba467b5768bf3484607a14c45 + b3a91e47023928a83ea62f4ea04eb64c922b3d33f1d5a42fad686376","size":110305280, + "url":"https://api.snapcraft.io/api/v1/snaps/downlo... + => the ALLOW-direction content assertion is `"channel-map"` AND `sha3-384`. + Neither appears in a squid error page. A bare 200 is NOT evidence. + + $ ssh vr1-dc0-rack 'curl -sS -o /dev/null -w "%{http_code}" -m 25 + https://canonical-lgw01.cdn.snapcraftcontent.com/' + 403 (curl exit 0 -- the CDN itself answers 403 to a bare GET of /) + + $ ssh vr1-dc0-rack 'curl -sS -o /dev/null -w "code=[%{http_code}]" -m 15 + -x http://10.12.8.2:8000 https://api.snapcraft.io:25/' + curl: (56) CONNECT tunnel failed, response 403 + code=[000] + => A PROXY DENIAL and a SERVER 403 are DISTINGUISHABLE, measured: + proxy denial : curl exit 56, stderr "CONNECT tunnel failed, response 403", + %{http_code} = 000 + server 403 : curl exit 0, %{http_code} = 403 + That difference is what lets `check` assert the DENY DIRECTION (the + destination allowlist is actually in force) separately from the ALLOW + direction, from the SAME source address. + + +8. JUJU'S SNAP PROXY KEYS EXIST IN THIS EXACT CONTROLLER +------------------------------------------------------------------------------ + $ ssh vr1-dc0-rack 'juju version; juju model-config -m controller --format=yaml + | grep -iE "snap|proxy"' + 3.6.27-genericlinux-amd64 + apt-http-proxy: apt-https-proxy: apt-ftp-proxy: apt-no-proxy: + http-proxy: https-proxy: ftp-proxy: no-proxy: proxy-ssh: + juju-http-proxy: juju-https-proxy: juju-ftp-proxy: juju-no-proxy: + lxd-snap-channel: + snap-http-proxy: snap-https-proxy: + snap-store-assertions: snap-store-proxy: snap-store-proxy-url: + (identical set from `juju model-defaults`) + => the five key names in the ruling note are CONFIRMED against the live + controller, not quoted from documentation. + + NOTE ON THE MODEL: at capture time `juju models` showed `vr1-dc0` in state + `destroying` (operator-driven, separate from this work). Probes above ran + against machines that still existed. Nothing here mutated the model. + + +9. VENDOR RESEARCH -- WHAT SNAPD ACTUALLY HONOURS +------------------------------------------------------------------------------ +Sources opened and quoted (not summarised from memory): + + snapd config mechanism + https://snapcraft.io/docs/reference/administration/system-options/ + "snap set system proxy.http=... / proxy.https=..." + CAVEAT FOUND AND NOT SMOOTHED OVER: that page carries an + "Available only on Ubuntu Core" note adjacent to the proxy section. THREE + primary sources contradict it, and code is the stronger witness: + https://raw.githubusercontent.com/canonical/snapd/master/overlord/configstate/configcore/proxy.go + supportedConfigurations["core.proxy.http"|"...https"|"...ftp"| + "...no-proxy"|"...store"] = true -- registered with NO core-only flag + envFilePath = "/etc/environment" -- a CLASSIC-system file + handleProxyConfiguration() writes http_proxy/https_proxy/ftp_proxy/no_proxy + => `snap set system proxy.*` writes /etc/environment. It does NOT write a + systemd drop-in. + https://raw.githubusercontent.com/canonical/snapd/master/data/systemd/snapd.service.in + "EnvironmentFile=-/var/lib/snapd/environment/snapd.conf" (+ the built-in one) + https://raw.githubusercontent.com/canonical/snapd/master/httputil/transport.go + "Proxy: http.ProxyFromEnvironment" + => snapd's HTTP transport reads the PROCESS environment; a + snapd.service.d drop-in with Environment=https_proxy= works by the same + path. Canonical's own k8s docs use exactly that drop-in shape: + https://documentation.ubuntu.com/canonical-kubernetes/latest/snap/howto/networking/proxy/ + https://forum.snapcraft.io/t/preferred-method-for-configuring-proxy-settings-for-snapd/11743 + chipaca (Canonical), 2019-06-11: "A proxy set via snap set overrides that + obtained from the environment"; the two approaches "are ... equivalent". + + WHICH HOSTS, AND WHETHER CONNECT IS REQUIRED + https://snapcraft.io/docs/reference/administration/network-requirements/ + https://forum.snapcraft.io/t/network-requirements/5147 (CDN list 2021-10-25) + api.snapcraft.io:443 dashboard.snapcraft.io:443 login.ubuntu.com:443 + *.snapcraftcontent.com:443 storage.snapcraftcontent.com:443 + canonical-{lgw01,lcy01,lcy02,bos01}.cdn.snapcraftcontent.com:443 + cloudfront.cdn.snapcraftcontent.com:443 + *.cdn.snapcraft.io:443 [deprecated] + => EVERY documented endpoint is :443. The proxy only ever sees + `CONNECT host:443`, so a CONNECT-capable forward proxy is REQUIRED and + `snap-http-proxy` is effectively inert for snapd's own traffic. Set both + anyway; `snap-https-proxy` is the one that matters. + https://raw.githubusercontent.com/canonical/snapd/master/store/store.go + the SAME client (opts.Proxy = s.cfg.Proxy) serves API calls AND downloads + => THE CDN FETCH IS PROXIED TOO. A destination allowlist that names only + api.snapcraft.io would establish the session and then fail the download. + + WHAT JUJU DOES WITH THE KEYS + https://canonical.com/juju/docs/juju-cli/3.6/reference/juju-cli/list-of-juju-cli-commands/model-config/ + snap-http-proxy "The HTTP proxy value for installing snaps" + snap-https-proxy "The HTTPS proxy value for installing snaps" + snap-store-proxy "The snap store proxy for installing snaps" + snap-store-proxy-url / snap-store-assertions + http-proxy/https-proxy -> instance HTTP_PROXY; juju-* -> charm hook env + => three DIFFERENT surfaces. Setting the wrong one will not fix snaps. + https://raw.githubusercontent.com/juju/juju/3.6/internal/worker/proxyupdater/proxyupdater.go + handleSnapProxyValues(): maybeAddSettings("proxy.http", ...), ("proxy.https", ...) + then RunFunc(noStdIn, "snap", append([]string{"set","system"}, snapSettings...)) + skipped only on CentOS + => juju literally shells out `snap set system proxy.http=... proxy.https=...`, + i.e. mechanism = section 9 paragraph 1. It is a WATCHER + (WatchForProxyConfigAndAPIHostPortChanges), so setting the key AFTER the + machines exist is expected to propagate. + snapd proxy.store is a 32-character STORE ID needing a signed assertion: + configcore/proxy.go: "cannot set proxy.store to %q without a matching + store assertion"; client shape documented at + https://ubuntu.com/enterprise-store/docs/page/how-to/devices/ + => `snap-store-proxy` / `-url` / `-assertions` are the ENTERPRISE STORE + product, NOT a forward proxy. LEAVE THEM EMPTY. + + SQUID VS TINYPROXY, AND THE CACHING QUESTION + https://packages.ubuntu.com/jammy/squid -- main, security-supported + https://packages.ubuntu.com/jammy/net/tinyproxy -- universe + https://ubuntu.com/security/cve?package=tinyproxy + jammy: 2 Fixed, 2 Not affected, SIX in "Needs evaluation" + https://tinyproxy.github.io/ + tinyproxy.conf(5) + ConnectPort: "If no ConnectPort line is found, then all ports are allowed." + => tinyproxy's CONNECT default is OPEN; squid's is deny-except-443. + https://www.squid-cache.org/Doc/config/http_port/ "Squid normally listens to port 3128" + https://www.squid-cache.org/Doc/config/cache/ `cache deny all` documented + https://wiki.squid-cache.org/Features/HTTPS + "the proxy establishes a TCP connection to the specified server, responds + with an HTTP 200 (Connection Established) ... without understanding or + interpreting the tunneled traffic"; "Squid does not have access to those + encrypted messages." + https://wiki.squid-cache.org/Features/SslBump + "Decrypting HTTPS tunnels constitutes a man-in-the-middle attack ..." + => THERE IS NO CACHING BENEFIT FOR SNAPS. Every endpoint is :443, so a + CONNECT proxy sees opaque tunnels and can cache nothing without TLS + interception, which is explicitly not wanted. Choose on security + support and simplicity, NOT on caching. (This is the opposite of the + apt case, where dc1's apt-cacher-ng genuinely caches plain-HTTP apt.) + https://github.com/chobits/ngx_http_proxy_connect_module + a third-party module exists precisely because stock nginx has no CONNECT. + => nginx is NOT a candidate. + + WHY NOT THE ENTERPRISE STORE (ex Snap Store Proxy) + https://ubuntu.com/enterprise-store/docs/page/how-to/install/ + prerequisites: an LTS server, a DOMAIN NAME, a PostgreSQL instance with + btree_gist + intarray, and a store REGISTRATION step. + https://ubuntu.com/enterprise-store/docs/how-to/airgap/ + offline mode additionally needs registration on an internet-connected + machine, an offline-snap-store.tar.gz carried in, and an export/import cycle. + https://ubuntu.com/blog/canonical-announces-the-enterprise-store (2026-07-21) + "included as part of Ubuntu Pro" + => for ~12 clients behind a rack that already has egress, this is a domain + name, a database, a Canonical account dependency and a per-node `snap + ack` -- against `apt install squid` plus two model-config keys. + + +10. WHAT COULD NOT BE VERIFIED (stated, not filled in) +------------------------------------------------------------------------------ + a. WHETHER A RUNNING snapd RE-READS THE PROXY WITHOUT A RESTART. The chain is + juju -> `snap set system proxy.*` -> snapd rewrites /etc/environment -> + snapd.service reads it via EnvironmentFile= AT START. Evidence it may not + re-read live: https://bugs.launchpad.net/juju/+bug/1737332 (Fix Released + 2018-10-14) "snapd *MUST* be restarted though after proxy.conf is created as + naturally there is no sane way to 'reload' environment variables"; and + https://bugs.launchpad.net/snapd/+bug/1791587 "snapd ignores proxy settings + set via core snap" (Fix Released, snapd 2.36.1, 2018-11-22) whose root cause + was exactly this. It could NOT be determined whether that fix made snapd + apply the proxy in-memory. OPERATIONAL CONSEQUENCE, carried into the script + header and the runbook wording: after setting the key, verify per node with + BOTH `snap get system proxy` AND a real egress probe; if the first passes and + the second fails, `systemctl restart snapd` and re-probe. + b. WHICH snapd VERSION INTRODUCED `snap set system proxy.*`. Two search + summaries disagree (2.28 vs 2.33) and neither was from a page that was + opened. MOOT here -- both racks measure snapd 2.76.1 and juju 3.6 emits + `snap set system` regardless. + c. THE "Available only on Ubuntu Core" NOTE on the system-options page (9a). + Code, the /etc/environment write target and juju's unconditional use all say + it does not apply to classic; no doc source repo was found to confirm the + note is an authoring defect. The settling test is one command on one node: + `sudo snap set system proxy.https=http://:3129 && sudo snap get system proxy`. + d. NOTHING WAS PROVEN END-TO-END. No proxy was installed, so no snap has been + installed through one. The build's own `check` and `node` subcommands are the + gates that will prove it at apply time; the harness green is a FIXTURE green. + e. dc1 HAS NOT BEEN PROBED FROM A CONTAINER. dc1 has no deployed model yet, so + the metal-admin-source finding (section 1) is measured on dc0 only. The dc1 + site row takes its CIDRs from `lib-net.sh`, and `node dc1` is the assertion + that will prove it there. + f. The squid config generated by this build has never been parsed by a squid + binary. `install` runs `squid -k parse -f ` before enabling anything, + so a syntax error fails loud at apply time rather than silently. + + +11. MUTATION PASS ON tests/dc-snap-proxy/run-tests.sh (2026-07-31) +------------------------------------------------------------------------------ +"A checker that cannot fail is not a gate", applied to the harness itself. The +2026-07-30 precedent is explicit: three freshly-written assertions were deleted +one at a time and the suite stayed green every time -- they were decoration. + +METHOD. A one-shot driver (/tmp, NOT a repo artifact) applied ONE exact string +replacement to scripts/dc-snap-proxy.sh at a time, ran the suite, restored the +file and re-verified its sha256 against the original. Every mutation neuters a +PREDICATE, never only a message: `miss` -> `say`, `refuse` -> `say`, +`nno` -> `nok`, `elif ` -> `elif false`, a value swapped for a plausible +wrong one, or a forbidden construct introduced. + +RESULT: 53 mutations, 53 KILLED, 0 survived. Baseline and final state both +`dc-snap-proxy: 52 passed, 0 failed`; the script restored byte-identical after +every mutation (sha256 verified, final sha_ok=True). + + M01 dc0 LISTEN 10.12.8.4 -> .5 -> T3 red (11 red total) + M02 dc0 edge gw -> .254 -> T4 red + M03 dc0 client ACL -> metal-internal 10.12.12.0/22 -> T5 red <-- the + exact wrong turn the container measurement prevented + M04 dc1 LISTEN -> .5 -> T6 red + M05 dc1 edge gw -> .254 -> T7 red + M06 dc1 client ACL -> metal-internal 10.12.72.0/22 -> T8 red + M07 port 3129 -> MAAS's 3128 -> T9 red + M08 hardcode virbr2 instead of util_bridge -> T10 red + M09 net unit renamed to *-cache-proxy-net.service -> T11 red + M10 operative "dc-mirror.sh install" referral added -> T11b red + M11 ip addr replace -> ip addr add -> T12 red + M12 MEASURED/RULED tags dropped -> T13 red + M13 install root guard removed -> T14 red + M14 snap-https-proxy key name mangled -> T15 red + M15 allowlist narrowed to api.snapcraft.io only -> T16 red + M16 http_port LISTEN:PORT -> wildcard PORT -> T17 red + M17 install stops disabling the packaged squid -> T18 red + M18 install stops parsing the config before enable -> T19 red + (first attempt was a MALFORMED mutation -- `true # ...` commented out the + `then` and broke `bash -n`, so T2 and 28 others went red while T19 did + not. Recorded rather than quietly re-rolled: a mutation that breaks syntax + tests nothing. Redone as `if false; then` + `:`, and T19 went red alone.) + M19 install apt-installs directly -> T20 red + M20 bad mode stops exiting 2 -> T21 red + M21 unknown site stops exiting 2 -> T22 red + M22 check_file asserts EXISTENCE not content -> T24 red <-- the + dc-mirror.sh false-green class, reproduced and caught + M23 missing artifact reported as OK -> T25 red + M24 unit-enabled miss softened -> T26 red + M25 packaged squid ACTIVE not detected -> T27 red + M26 packaged squid ENABLED not detected -> T27b red + M27 wildcard bind accepted as OK -> T28 red + M28 absent listener accepted -> T29 red + M29 absent LISTEN alias accepted -> T30 red + M30 unresolvable bridge defaults to success -> T31 red + M31 missing default route accepted -> T32 red + M32 check drops the channel-map content assertion -> T33 red + M33 check drops the sha3-384 assertion -> T34 red + M34 check ignores a non-200 store status -> T35 red + M35 unreachable proxy defaults to success -> T36 red + M36 refused CDN tunnel accepted -> T37 red + M37 unrecognised CDN result defaults to success -> T38 red + M38 OPEN proxy accepted (deny direction gone) -> T39 red <-- the + assertion that proves the allowlist is in force, not merely written down + M39 unrecognised deny result defaults to success -> T40 red + M40 missing squid package accepted -> T41 red + M41 no curl defaults to success -> T42 red + M42 node: ACL denial reported as OK -> T44 red + M43 node: unreachable proxy defaults to success -> T45 red + M44 node: drops the channel-map assertion -> T46 red + M45 node: OPEN proxy accepted -> T47 red + M46 node: absent https_proxy accepted -> T48 red + M47 node: wrong https_proxy accepted -> T49 red + M48 node makes a rack-side call (systemctl) -> T50 red + M49 check: REFUSE treated as a pass -> T31/36/38/40/42 red + M50 node: REFUSE treated as a pass -> T45 red + M51 check: FAIL treated as a pass -> 15 cases red + M52 node: FAIL treated as a pass -> 5 cases red + M53 ENV_FILE path moved -> T43 red (the node + green baseline is itself killable, so it is not a tautology) + +THREE OF THE HARNESS'S OWN CASES WERE FOUND TOO WEAK WHILE BUILDING THIS, and +were STRENGTHENED rather than left passing -- each would have stayed green after +the thing it names was removed: + T16 grepped the WHOLE FILE for `.snapcraftcontent.com`, which the header + discusses in prose, so narrowing the actual STORE_DOMAINS value left it + green. Now scoped to the `STORE_DOMAINS=` line. + T18 grepped the whole file for `systemctl disable --now squid`, which check's + own failure MESSAGE contains, so install could stop doing it. Now scoped + to do_install's body. + T20 matched `apt-get install` inside the guidance STRING, so it read green + whether or not the script apt-installed. Now blanks double-quoted strings + first and matches an INVOCATION position. +Same class, found the same way: T10 and T50 had to blank comments / quoted +strings for the opposite reason -- they were RED on legitimate prose (measured +`virbr5` evidence in a site-row comment; an `ip route get` hint inside a failure +message). Rewording the prose to satisfy a regex would have been gaming the +test, so the tests were fixed to assert on CODE. + +WHAT THE MUTATION PASS DOES NOT PROVE: that the assertions are the RIGHT ones, +or that the proxy works. It proves only that none of them is decoration.