stage5-snap-proxy-measurements-20260731.txt
===========================================
Read-only measurement capture backing `scripts/dc-snap-proxy.sh` (the REPO SIDE
of D-135/D-107 RULING 2, 2026-07-31: "HTTP(S) forward proxy in the DC utility
band + juju snap-https-proxy (Recommended)").

SCOPE OF THIS CAPTURE. Every command below is READ-ONLY. NO cloud mutation was
made in the session that produced it: no `juju config`, no `juju model-config`,
no package installed, no service started or stopped, no file written on any rack
or node. The proxy is BUILT IN THE REPO and NOT APPLIED. Two side effects are
declared rather than hidden: the `curl` probes traversed MAAS's existing squid
and may have added entries to ITS cache, and `apt-get download squid` wrote a
.deb into `/tmp` ON THE JUMPHOST (not on any rack) so the shipped squid config
could be read instead of guessed.

Host/date: vcloud jumphost, 2026-07-31. Branch dc-dc-stage5-preconditions.


1. THE THREE FAILING APPLICATIONS ARE LXD CONTAINERS, AND THEIR SOURCE ADDRESS
   TOWARD THE RACK IS ON METAL-ADMIN, NOT METAL-INTERNAL
------------------------------------------------------------------------------
This is the measurement the whole design turns on: it decides the squid client
ACL and whether the D-134 utility `.4` is reachable by the clients at all.
`juju status -m vr1-dc0` shows `mysql-innodb-cluster/0` at `10.12.12.116`, which
is NOT a D-134 node octet (.100-.102 / .120-.121 / .150-.153) -- so the failing
units are containers, and the address `juju status` prints is only ONE of their
legs. Measured from inside the container:

  $ juju ssh -m vr1-dc0 mysql-innodb-cluster/0 'ip -4 -o addr show; ip route show;
      ip route get 10.12.8.4'
  60: eth0    inet 10.12.8.122/22  brd 10.12.11.255 scope global eth0
  62: eth1    inet 10.12.12.116/22 brd 10.12.15.255 scope global eth1
  10.12.8.0/22  dev eth0 proto kernel scope link src 10.12.8.122
  10.12.12.0/22 dev eth1 proto kernel scope link src 10.12.12.116
  10.12.8.4 dev eth0 src 10.12.8.122 uid 1000
      cache

  => the container has BOTH a metal-admin leg (.8.122) and a metal-internal leg
     (.12.116); the kernel selects eth0 / src 10.12.8.122 for the utility `.4`;
     and there is NO default route at all.

  $ ... 'curl -sS -o /dev/null -w "%{http_code}" -m 12
      -H "Snap-Device-Series: 16" https://api.snapcraft.io/v2/snaps/info/core'
  curl: (7) Couldn't connect to server        http=000        <-- DIRECT: FAILS

  $ ... same URL, but -x http://10.12.8.2:8000
  http=200                                                    <-- VIA A PROXY: WORKS

  => CONFIRMS the D-135 items 2-3 root cause from the client side, and confirms
     the mechanism the ruling chose: an in-DC CONNECT proxy on the rack works
     from the exact unit that fails.
  => CONSEQUENCE FOR THE BUILD: the squid client ACL is the METAL-ADMIN /22
     (`10.12.8.0/22` dc0, `10.12.68.0/22` dc1), not metal-internal. A
     metal-internal ACL would have denied every client the proxy exists for.

Corroborating node-side capture (`juju ssh -m vr1-dc0 0`, a bare role node):
  br-ex 10.12.4.100/22, br-enp1s0 10.12.8.100/22, br-enp3s0 10.12.12.100/22,
  br-enp5s0 10.12.32.100/22, br-enp6s0 10.12.36.100/22, br-enp4s0 10.12.16.100/22
  default via 10.12.4.1 dev br-ex proto static
  DIRECT api.snapcraft.io -> http=200      (node has a br-ex default route)
  via 10.12.8.2:8000      -> http=200
  http://10.12.8.4/ (mirror) -> http=200
  sudo snap get system proxy -> {"proxy":{"http":"","https":"","store":""}}

  HONEST READING, because this one is easy to misread: the bare node DOES reach
  the internet directly today via br-ex. The CONTAINERS do not, and they are what
  fails. Nothing here should be read as "the nodes are already airgapped".

Plane CIDR authority (repo, not inferred by mirroring dc0): `scripts/lib-net.sh`
  vr1-dc0 PLANE_CIDRS = 10.12.4/8/12/16/32/36.0/22, metal-admin = 10.12.8.0/22
  vr1-dc1 PLANE_CIDRS = 10.12.64/68/72/76/80/84.0/22, metal-admin = 10.12.68.0/22


2. THE UTILITY `.4` ALREADY EXISTS ON BOTH RACKS; NO NEW OCTET IS NEEDED
------------------------------------------------------------------------------
  $ ssh vr1-dc0-rack 'ip -4 -o addr show; ip route show default'
  virbr2 inet 10.12.8.2/22 | 10.12.8.3/22 secondary | 10.12.8.4/22 secondary
  virbr5 inet 10.12.4.2/22 ; enp1s0 inet 172.31.0.2/30
  default via 10.12.4.1 dev virbr5

  $ ssh vr1-dc1-rack 'ip -4 -o addr show; ip route show default'
  virbr6 inet 10.12.68.2/22 | 10.12.68.3/22 secondary | 10.12.68.4/22 secondary
  virbr4 inet 10.12.64.2/22 ; enp1s0 inet 172.31.0.6/30
  default via 10.12.64.1 dev virbr4

  => the D-134 utility `.4` (artifact service) is already aliased on both racks'
     metal-admin bridge, held by `dc0-mirror-net.service` on dc0 and by
     `dc1-cache-proxy-net.service` on dc1. Placing the snap proxy ON THE RACK at
     `.4:<port>` therefore needs NO new ruled octet -- which is what the ruling
     note asked for. Both racks are Ubuntu 24.04.4 LTS.


3. PORT 3128 IS ALREADY TAKEN -- BY A SQUID, ON BOTH RACKS
------------------------------------------------------------------------------
  $ ssh vr1-dc0-rack 'ss -ltn'          (dc0, abridged to the relevant rows)
  LISTEN 10.12.4.2:53   10.12.8.3:53   0.0.0.0:22   0.0.0.0:80
  LISTEN 0.0.0.0:5248   0.0.0.0:5247   *:5249   172.31.0.2:5280
  LISTEN *:8000         *:3128

  $ ssh vr1-dc1-rack 'ss -ltn'          (dc1)
  LISTEN 0.0.0.0:3142 (apt-cacher-ng)   10.12.68.3:53   0.0.0.0:22   0.0.0.0:7911
  LISTEN *:8000        *:3128

  $ ssh vr1-dc0-rack 'sudo ss -ltnp | grep -E ":(3128|8000) "'
  LISTEN *:8000  users:(("squid",pid=1871633,fd=13))
  LISTEN *:3128  users:(("squid",pid=1871633,fd=12))

  $ ssh vr1-dc0-rack 'sudo tr "\0" " " < /proc/1871633/cmdline'
  squid -N -d 5 -f /var/snap/maas/41649/proxy/maas-proxy.conf

  => 3128 AND 8000 are held, on BOTH racks, by MAAS's own squid, WILDCARD-bound.
     A wildcard bind means a later `10.12.8.4:3128` bind would fail. 3142 is
     dc1's apt-cacher-ng, 80 is dc0's mirror nginx.
  => CONSEQUENCE: the snap proxy takes **3129**, bound to the LISTEN address
     specifically, and must NOT be the packaged `squid.service` (see 5).
  => NOTE: MAAS's squid itself runs with `-f <conf>`, i.e. the exact
     separate-instance shape this build uses. That is a precedent, not a guess.


4. MAAS'S SQUID ALREADY CONNECT-PROXIES THE SNAP STORE. WHY IT WAS NOT USED.
------------------------------------------------------------------------------
  $ ssh vr1-dc0-rack 'sudo grep -vE "^\s*#|^\s*$" /var/snap/maas/current/proxy/maas-proxy.conf'
  ... acl localnet src 10.12.8.0/22 | 10.12.12.0/22 | 10.12.68.0/22 | 10.12.72.0/22 ...
  acl SSL_ports port 443
  acl CONNECT method CONNECT
  http_access deny CONNECT !SSL_ports
  http_access allow localnet
  http_access deny all
  http_port 3128 transparent
  http_port 8000

  MEASURED to work, three ways:
    from the rack itself   -x http://10.12.8.2:8000 -> api.snapcraft.io http=200
    from the juju ctrl VM  -x http://10.12.8.2:8000 -> http=200
                           -x http://10.12.8.6:8000 -> http=200
    from mysql-innodb-cluster/0 (container) -x http://10.12.8.2:8000 -> http=200

  And juju/MAAS ALREADY point machines at a MAAS squid for apt:
  $ juju ssh -m controller 0 'grep -rn -i proxy /etc/apt/apt.conf.d/'
  /etc/apt/apt.conf.d/90curtin-aptproxy:1:Acquire::http::Proxy "http://10.12.8.6:8000/";

  SO A WORKING FORWARD PROXY ARGUABLY ALREADY EXISTS. It was NOT adopted, and
  the reason is recorded here rather than left implicit:
    (a) `maas-proxy.conf` is MACHINE-GENERATED by MAAS and regenerated whenever
        MAAS's subnet/proxy settings change -- nothing in this repo owns it, and
        no diff-against-generator check could ever assert its content.
    (b) It lives under the per-revision path `/var/snap/maas/current/` (measured
        real path `/var/snap/maas/41649/...`), which the operating skill already
        records as NOT surviving a snap refresh for the power keys.
    (c) It has NO destination restriction: `http_access allow localnet` permits
        CONNECT to ANY :443 host from every plane MAAS knows. Adopting it as the
        snap path would silently give the node planes general HTTPS egress.
    (d) It is the hidden-coupling anti-pattern this repo already has a scar from
        (dc-cache-proxy.sh reusing dc-mirror.sh's net layer, D-135 amendment
        2026-07-27): a service whose prerequisites are owned by another service's
        machinery cannot be torn down or rebuilt independently.
  THIS IS AN OPERATOR-VISIBLE ALTERNATIVE, NOT A DISMISSAL. If the operator
  prefers it at apply time, the equivalent live change is just
  `juju model-config snap-https-proxy=http://10.12.8.6:8000` with no rack build
  at all -- cheaper, and worse on (a)-(d).


5. THE SHIPPED UBUNTU SQUID: MEASURED FROM THE PACKAGE, NOT ASSUMED
------------------------------------------------------------------------------
  $ ssh vr1-dc0-rack 'apt-cache policy squid tinyproxy'     (identical on dc1)
  squid      Candidate: 6.14-0ubuntu0.24.04.4   noble-updates/**main**
  tinyproxy  Candidate: 1.11.1-3ubuntu0.1       noble-updates/**universe**

  On the JUMPHOST (also noble 24.04.4, same candidate version), the .deb was
  downloaded and unpacked read-only so the shipped config could be READ:
  $ apt-get download squid && dpkg-deb -x squid_6.14-0ubuntu0.24.04.4_amd64.deb x
  $ grep -vE '^\s*#|^\s*$' x/etc/squid/squid.conf
  acl localnet src 0.0.0.1-0.255.255.255 / 10.0.0.0/8 / 100.64.0.0/10 /
      169.254.0.0/16 / 172.16.0.0/12 / 192.168.0.0/16 / fc00::/7 / fe80::/10
  acl SSL_ports port 443
  acl Safe_ports port 80 21 443 70 210 1025-65535 280 488 591 777
  http_access deny !Safe_ports
  http_access deny CONNECT !SSL_ports
  http_access allow localhost manager
  http_access deny manager
  http_access allow localhost
  http_access deny to_localhost
  http_access deny to_linklocal
  include /etc/squid/conf.d/*.conf
  http_access deny all
  http_port 3128
  coredump_dir /var/spool/squid
  refresh_pattern ...

  $ cat x/usr/lib/systemd/system/squid.service
  [Service] Type=notify   PIDFile=/run/squid.pid   Group=proxy
  RuntimeDirectory=squid  RuntimeDirectoryMode=0775
  ExecStartPre=/usr/sbin/squid --foreground -z
  ExecStart=/usr/sbin/squid --foreground -sYC
  ExecReload=/bin/kill -HUP $MAINPID   KillMode=mixed   NotifyAccess=all

  $ cat x/etc/default/squid
  (EMPTY -- there is no SQUID_ARGS / SQUID_CONFIG_FILE indirection to hook)

  THREE CONSEQUENCES, each load-bearing:
  (i)   `include /etc/squid/conf.d/*.conf` sits BEFORE `http_access deny all`,
        so a conf.d drop-in CAN add an allow rule -- but it CANNOT unset
        `http_port 3128`, and 3128 is held by MAAS's squid (section 3). The
        packaged unit would therefore FATAL on bind. A conf.d drop-in is
        structurally unusable here.
  (ii)  The default already carries `http_access deny CONNECT !SSL_ports` and a
        terminal `http_access deny all` -- the posture we want is squid's own
        default, which is the main reason to prefer it over tinyproxy (whose
        `ConnectPort` default is documented as "all ports allowed").
  (iii) `/etc/default/squid` being empty means there is no supported way to point
        the packaged unit at another config file. A DEDICATED instance (own
        config + own unit + own pid file) is the only clean shape, and
        `install` must `systemctl disable --now squid` so the packaged unit does
        not crash-loop on 3128 and poison triage.

  $ zcat x/usr/share/man/man8/squid.8.gz | grep -A2 '"\\-n name"'
  "-n name   Specify Windows Service name ..."
  => `-n` is documented as Windows-only, so the instance is separated by `-f`
     plus an explicit `pid_filename`, NOT by `-n`.


6. SHARED-MEMORY COEXISTENCE WITH MAAS'S SQUID: MEASURED, NOT REASONED
------------------------------------------------------------------------------
Two squids on one host can collide on `/dev/shm` segment names. Measured:

  $ ssh vr1-dc0-rack 'ls -la /dev/shm/'
  -rw------- snap_daemon snap.maas.squid-cf__metadata.shm
  -rw------- snap_daemon snap.maas.squid-cf__queues.shm
  -rw------- snap_daemon snap.maas.squid-cf__readers.shm
  $ ssh vr1-dc0-rack 'sudo ls -la /run/ | grep -i squid'   ->  (no /run/squid*)

  => snapd namespaces MAAS's segments behind a `snap.maas.` prefix; a host squid
     creates bare `squid-cf__*`. NO COLLISION. And MAAS's squid runs `-N` with no
     pid file, so `/run/<site>-snap-proxy.pid` collides with nothing either.
  => This is the one coexistence question dc-cache-proxy.sh had to leave as
     "REASONED, NOT MEASURED". Here it is measured.


7. SNAP-STORE ENDPOINT SHAPES, MEASURED, SO THE CHECK PREDICATES ARE REAL
------------------------------------------------------------------------------
  $ ssh vr1-dc0-rack 'curl -sS -m 25 -H "Snap-Device-Series: 16"
      https://api.snapcraft.io/v2/snaps/info/core | head -c 400'
  {"channel-map":[{"channel":{"architecture":"amd64","name":"stable",...},
   "download":{"deltas":[],"sha3-384":"13c728f16430310ba467b5768bf3484607a14c45
   b3a91e47023928a83ea62f4ea04eb64c922b3d33f1d5a42fad686376","size":110305280,
   "url":"https://api.snapcraft.io/api/v1/snaps/downlo...
  => the ALLOW-direction content assertion is `"channel-map"` AND `sha3-384`.
     Neither appears in a squid error page. A bare 200 is NOT evidence.

  $ ssh vr1-dc0-rack 'curl -sS -o /dev/null -w "%{http_code}" -m 25
      https://canonical-lgw01.cdn.snapcraftcontent.com/'
  403        (curl exit 0 -- the CDN itself answers 403 to a bare GET of /)

  $ ssh vr1-dc0-rack 'curl -sS -o /dev/null -w "code=[%{http_code}]" -m 15
      -x http://10.12.8.2:8000 https://api.snapcraft.io:25/'
  curl: (56) CONNECT tunnel failed, response 403
  code=[000]
  => A PROXY DENIAL and a SERVER 403 are DISTINGUISHABLE, measured:
       proxy denial : curl exit 56, stderr "CONNECT tunnel failed, response 403",
                      %{http_code} = 000
       server 403   : curl exit 0,  %{http_code} = 403
     That difference is what lets `check` assert the DENY DIRECTION (the
     destination allowlist is actually in force) separately from the ALLOW
     direction, from the SAME source address.


8. JUJU'S SNAP PROXY KEYS EXIST IN THIS EXACT CONTROLLER
------------------------------------------------------------------------------
  $ ssh vr1-dc0-rack 'juju version; juju model-config -m controller --format=yaml
      | grep -iE "snap|proxy"'
  3.6.27-genericlinux-amd64
  apt-http-proxy: apt-https-proxy: apt-ftp-proxy: apt-no-proxy:
  http-proxy: https-proxy: ftp-proxy: no-proxy: proxy-ssh:
  juju-http-proxy: juju-https-proxy: juju-ftp-proxy: juju-no-proxy:
  lxd-snap-channel:
  snap-http-proxy: snap-https-proxy:
  snap-store-assertions: snap-store-proxy: snap-store-proxy-url:
  (identical set from `juju model-defaults`)
  => the five key names in the ruling note are CONFIRMED against the live
     controller, not quoted from documentation.

  NOTE ON THE MODEL: at capture time `juju models` showed `vr1-dc0` in state
  `destroying` (operator-driven, separate from this work). Probes above ran
  against machines that still existed. Nothing here mutated the model.


9. VENDOR RESEARCH -- WHAT SNAPD ACTUALLY HONOURS
------------------------------------------------------------------------------
Sources opened and quoted (not summarised from memory):

  snapd config mechanism
    https://snapcraft.io/docs/reference/administration/system-options/
      "snap set system proxy.http=... / proxy.https=..."
      CAVEAT FOUND AND NOT SMOOTHED OVER: that page carries an
      "Available only on Ubuntu Core" note adjacent to the proxy section. THREE
      primary sources contradict it, and code is the stronger witness:
    https://raw.githubusercontent.com/canonical/snapd/master/overlord/configstate/configcore/proxy.go
      supportedConfigurations["core.proxy.http"|"...https"|"...ftp"|
        "...no-proxy"|"...store"] = true    -- registered with NO core-only flag
      envFilePath = "/etc/environment"      -- a CLASSIC-system file
      handleProxyConfiguration() writes http_proxy/https_proxy/ftp_proxy/no_proxy
      => `snap set system proxy.*` writes /etc/environment. It does NOT write a
         systemd drop-in.
    https://raw.githubusercontent.com/canonical/snapd/master/data/systemd/snapd.service.in
      "EnvironmentFile=-/var/lib/snapd/environment/snapd.conf" (+ the built-in one)
    https://raw.githubusercontent.com/canonical/snapd/master/httputil/transport.go
      "Proxy: http.ProxyFromEnvironment"
      => snapd's HTTP transport reads the PROCESS environment; a
         snapd.service.d drop-in with Environment=https_proxy= works by the same
         path. Canonical's own k8s docs use exactly that drop-in shape:
         https://documentation.ubuntu.com/canonical-kubernetes/latest/snap/howto/networking/proxy/
    https://forum.snapcraft.io/t/preferred-method-for-configuring-proxy-settings-for-snapd/11743
      chipaca (Canonical), 2019-06-11: "A proxy set via snap set overrides that
      obtained from the environment"; the two approaches "are ... equivalent".

  WHICH HOSTS, AND WHETHER CONNECT IS REQUIRED
    https://snapcraft.io/docs/reference/administration/network-requirements/
    https://forum.snapcraft.io/t/network-requirements/5147   (CDN list 2021-10-25)
      api.snapcraft.io:443  dashboard.snapcraft.io:443  login.ubuntu.com:443
      *.snapcraftcontent.com:443  storage.snapcraftcontent.com:443
      canonical-{lgw01,lcy01,lcy02,bos01}.cdn.snapcraftcontent.com:443
      cloudfront.cdn.snapcraftcontent.com:443
      *.cdn.snapcraft.io:443  [deprecated]
      => EVERY documented endpoint is :443. The proxy only ever sees
         `CONNECT host:443`, so a CONNECT-capable forward proxy is REQUIRED and
         `snap-http-proxy` is effectively inert for snapd's own traffic. Set both
         anyway; `snap-https-proxy` is the one that matters.
    https://raw.githubusercontent.com/canonical/snapd/master/store/store.go
      the SAME client (opts.Proxy = s.cfg.Proxy) serves API calls AND downloads
      => THE CDN FETCH IS PROXIED TOO. A destination allowlist that names only
         api.snapcraft.io would establish the session and then fail the download.

  WHAT JUJU DOES WITH THE KEYS
    https://canonical.com/juju/docs/juju-cli/3.6/reference/juju-cli/list-of-juju-cli-commands/model-config/
      snap-http-proxy  "The HTTP proxy value for installing snaps"
      snap-https-proxy "The HTTPS proxy value for installing snaps"
      snap-store-proxy "The snap store proxy for installing snaps"
      snap-store-proxy-url / snap-store-assertions
      http-proxy/https-proxy -> instance HTTP_PROXY;  juju-* -> charm hook env
      => three DIFFERENT surfaces. Setting the wrong one will not fix snaps.
    https://raw.githubusercontent.com/juju/juju/3.6/internal/worker/proxyupdater/proxyupdater.go
      handleSnapProxyValues(): maybeAddSettings("proxy.http", ...), ("proxy.https", ...)
      then  RunFunc(noStdIn, "snap", append([]string{"set","system"}, snapSettings...))
      skipped only on CentOS
      => juju literally shells out `snap set system proxy.http=... proxy.https=...`,
         i.e. mechanism = section 9 paragraph 1. It is a WATCHER
         (WatchForProxyConfigAndAPIHostPortChanges), so setting the key AFTER the
         machines exist is expected to propagate.
    snapd proxy.store is a 32-character STORE ID needing a signed assertion:
      configcore/proxy.go: "cannot set proxy.store to %q without a matching
      store assertion";  client shape documented at
      https://ubuntu.com/enterprise-store/docs/page/how-to/devices/
      => `snap-store-proxy` / `-url` / `-assertions` are the ENTERPRISE STORE
         product, NOT a forward proxy. LEAVE THEM EMPTY.

  SQUID VS TINYPROXY, AND THE CACHING QUESTION
    https://packages.ubuntu.com/jammy/squid          -- main, security-supported
    https://packages.ubuntu.com/jammy/net/tinyproxy  -- universe
    https://ubuntu.com/security/cve?package=tinyproxy
      jammy: 2 Fixed, 2 Not affected, SIX in "Needs evaluation"
    https://tinyproxy.github.io/  + tinyproxy.conf(5)
      ConnectPort: "If no ConnectPort line is found, then all ports are allowed."
      => tinyproxy's CONNECT default is OPEN; squid's is deny-except-443.
    https://www.squid-cache.org/Doc/config/http_port/  "Squid normally listens to port 3128"
    https://www.squid-cache.org/Doc/config/cache/      `cache deny all` documented
    https://wiki.squid-cache.org/Features/HTTPS
      "the proxy establishes a TCP connection to the specified server, responds
       with an HTTP 200 (Connection Established) ... without understanding or
       interpreting the tunneled traffic"; "Squid does not have access to those
       encrypted messages."
    https://wiki.squid-cache.org/Features/SslBump
      "Decrypting HTTPS tunnels constitutes a man-in-the-middle attack ..."
      => THERE IS NO CACHING BENEFIT FOR SNAPS. Every endpoint is :443, so a
         CONNECT proxy sees opaque tunnels and can cache nothing without TLS
         interception, which is explicitly not wanted. Choose on security
         support and simplicity, NOT on caching. (This is the opposite of the
         apt case, where dc1's apt-cacher-ng genuinely caches plain-HTTP apt.)
    https://github.com/chobits/ngx_http_proxy_connect_module
      a third-party module exists precisely because stock nginx has no CONNECT.
      => nginx is NOT a candidate.

  WHY NOT THE ENTERPRISE STORE (ex Snap Store Proxy)
    https://ubuntu.com/enterprise-store/docs/page/how-to/install/
      prerequisites: an LTS server, a DOMAIN NAME, a PostgreSQL instance with
      btree_gist + intarray, and a store REGISTRATION step.
    https://ubuntu.com/enterprise-store/docs/how-to/airgap/
      offline mode additionally needs registration on an internet-connected
      machine, an offline-snap-store.tar.gz carried in, and an export/import cycle.
    https://ubuntu.com/blog/canonical-announces-the-enterprise-store (2026-07-21)
      "included as part of Ubuntu Pro"
      => for ~12 clients behind a rack that already has egress, this is a domain
         name, a database, a Canonical account dependency and a per-node `snap
         ack` -- against `apt install squid` plus two model-config keys.


10. WHAT COULD NOT BE VERIFIED (stated, not filled in)
------------------------------------------------------------------------------
 a. WHETHER A RUNNING snapd RE-READS THE PROXY WITHOUT A RESTART. The chain is
    juju -> `snap set system proxy.*` -> snapd rewrites /etc/environment ->
    snapd.service reads it via EnvironmentFile= AT START. Evidence it may not
    re-read live: https://bugs.launchpad.net/juju/+bug/1737332 (Fix Released
    2018-10-14) "snapd *MUST* be restarted though after proxy.conf is created as
    naturally there is no sane way to 'reload' environment variables"; and
    https://bugs.launchpad.net/snapd/+bug/1791587 "snapd ignores proxy settings
    set via core snap" (Fix Released, snapd 2.36.1, 2018-11-22) whose root cause
    was exactly this. It could NOT be determined whether that fix made snapd
    apply the proxy in-memory. OPERATIONAL CONSEQUENCE, carried into the script
    header and the runbook wording: after setting the key, verify per node with
    BOTH `snap get system proxy` AND a real egress probe; if the first passes and
    the second fails, `systemctl restart snapd` and re-probe.
 b. WHICH snapd VERSION INTRODUCED `snap set system proxy.*`. Two search
    summaries disagree (2.28 vs 2.33) and neither was from a page that was
    opened. MOOT here -- both racks measure snapd 2.76.1 and juju 3.6 emits
    `snap set system` regardless.
 c. THE "Available only on Ubuntu Core" NOTE on the system-options page (9a).
    Code, the /etc/environment write target and juju's unconditional use all say
    it does not apply to classic; no doc source repo was found to confirm the
    note is an authoring defect. The settling test is one command on one node:
    `sudo snap set system proxy.https=http://<listen>:3129 && sudo snap get system proxy`.
 d. NOTHING WAS PROVEN END-TO-END. No proxy was installed, so no snap has been
    installed through one. The build's own `check` and `node` subcommands are the
    gates that will prove it at apply time; the harness green is a FIXTURE green.
 e. dc1 HAS NOT BEEN PROBED FROM A CONTAINER. dc1 has no deployed model yet, so
    the metal-admin-source finding (section 1) is measured on dc0 only. The dc1
    site row takes its CIDRs from `lib-net.sh`, and `node dc1` is the assertion
    that will prove it there.
 f. The squid config generated by this build has never been parsed by a squid
    binary. `install` runs `squid -k parse -f <conf>` before enabling anything,
    so a syntax error fails loud at apply time rather than silently.
 g. THE DENY-SHAPE MEASUREMENT IS FROM A *PORT* DENY, NOT A *DSTDOMAIN* DENY.
    Section 7's `curl: (56) CONNECT tunnel failed, response 403` was produced by
    MAAS's squid refusing `api.snapcraft.io:25` under `http_access deny CONNECT
    !SSL_ports` -- a PORT rule. This build's deny probe trips a DSTDOMAIN rule
    (`http_access deny !snap_store`). Squid answers both with 403 at CONNECT, so
    the two should be byte-identical to curl, but that path was NOT measured.
    THE FAILURE MODE IS SAFE: check's deny branch treats anything that is neither
    "403 at CONNECT" nor "curl exit 0" as REFUSE, so an unexpected shape can only
    withhold a pass, never grant one. Confirm at apply time, once: run the deny
    probe by hand against the built proxy and read the exact curl line.
 h. TWO OF check's PROBES ARE LIVENESS-COUPLED TO CANONICAL. The store-API probe
    and the CDN probe reach real third-party hosts, so an upstream outage or a
    DNS failure makes a healthy proxy REFUSE. That is deliberate -- a probe that
    cannot distinguish "the CDN is down" from "the proxy is misconfigured" must
    not guess -- but it means a REFUSE is not by itself evidence about the proxy,
    and the script header says so where a triaging operator will read it. The
    static half of the same property (that the CDN wildcard is PRESENT in
    STORE_DOMAINS) is asserted by the config diff and by harness case T16, which
    have no external dependency at all.


11. MUTATION PASS ON tests/dc-snap-proxy/run-tests.sh (2026-07-31)
------------------------------------------------------------------------------
"A checker that cannot fail is not a gate", applied to the harness itself. The
2026-07-30 precedent is explicit: three freshly-written assertions were deleted
one at a time and the suite stayed green every time -- they were decoration.

METHOD. A one-shot driver (/tmp, NOT a repo artifact) applied ONE exact string
replacement to scripts/dc-snap-proxy.sh at a time, ran the suite, restored the
file and re-verified its sha256 against the original. Every mutation neuters a
PREDICATE, never only a message: `miss` -> `say`, `refuse` -> `say`,
`nno` -> `nok`, `elif <test>` -> `elif false`, a value swapped for a plausible
wrong one, or a forbidden construct introduced.

RESULT: 53 mutations, 53 KILLED, 0 survived. Baseline and final state both
`dc-snap-proxy: 52 passed, 0 failed`; the script restored byte-identical after
every mutation (sha256 verified, final sha_ok=True).

  M01 dc0 LISTEN 10.12.8.4 -> .5                      -> T3   red (11 red total)
  M02 dc0 edge gw -> .254                             -> T4   red
  M03 dc0 client ACL -> metal-internal 10.12.12.0/22  -> T5   red   <-- the
      exact wrong turn the container measurement prevented
  M04 dc1 LISTEN -> .5                                -> T6   red
  M05 dc1 edge gw -> .254                             -> T7   red
  M06 dc1 client ACL -> metal-internal 10.12.72.0/22  -> T8   red
  M07 port 3129 -> MAAS's 3128                        -> T9   red
  M08 hardcode virbr2 instead of util_bridge          -> T10  red
  M09 net unit renamed to *-cache-proxy-net.service   -> T11  red
  M10 operative "dc-mirror.sh install" referral added -> T11b red
  M11 ip addr replace -> ip addr add                  -> T12  red
  M12 MEASURED/RULED tags dropped                     -> T13  red
  M13 install root guard removed                      -> T14  red
  M14 snap-https-proxy key name mangled               -> T15  red
  M15 allowlist narrowed to api.snapcraft.io only     -> T16  red
  M16 http_port LISTEN:PORT -> wildcard PORT          -> T17  red
  M17 install stops disabling the packaged squid      -> T18  red
  M18 install stops parsing the config before enable  -> T19  red
      (first attempt was a MALFORMED mutation -- `true # ...` commented out the
      `then` and broke `bash -n`, so T2 and 28 others went red while T19 did
      not. Recorded rather than quietly re-rolled: a mutation that breaks syntax
      tests nothing. Redone as `if false; then` + `:`, and T19 went red alone.)
  M19 install apt-installs directly                   -> T20  red
  M20 bad mode stops exiting 2                        -> T21  red
  M21 unknown site stops exiting 2                    -> T22  red
  M22 check_file asserts EXISTENCE not content        -> T24  red   <-- the
      dc-mirror.sh false-green class, reproduced and caught
  M23 missing artifact reported as OK                 -> T25  red
  M24 unit-enabled miss softened                      -> T26  red
  M25 packaged squid ACTIVE not detected              -> T27  red
  M26 packaged squid ENABLED not detected             -> T27b red
  M27 wildcard bind accepted as OK                    -> T28  red
  M28 absent listener accepted                        -> T29  red
  M29 absent LISTEN alias accepted                    -> T30  red
  M30 unresolvable bridge defaults to success         -> T31  red
  M31 missing default route accepted                  -> T32  red
  M32 check drops the channel-map content assertion   -> T33  red
  M33 check drops the sha3-384 assertion              -> T34  red
  M34 check ignores a non-200 store status            -> T35  red
  M35 unreachable proxy defaults to success           -> T36  red
  M36 refused CDN tunnel accepted                     -> T37  red
  M37 unrecognised CDN result defaults to success     -> T38  red
  M38 OPEN proxy accepted (deny direction gone)       -> T39  red   <-- the
      assertion that proves the allowlist is in force, not merely written down
  M39 unrecognised deny result defaults to success    -> T40  red
  M40 missing squid package accepted                  -> T41  red
  M41 no curl defaults to success                     -> T42  red
  M42 node: ACL denial reported as OK                 -> T44  red
  M43 node: unreachable proxy defaults to success     -> T45  red
  M44 node: drops the channel-map assertion           -> T46  red
  M45 node: OPEN proxy accepted                       -> T47  red
  M46 node: absent https_proxy accepted               -> T48  red
  M47 node: wrong https_proxy accepted                -> T49  red
  M48 node makes a rack-side call (systemctl)         -> T50  red
  M49 check: REFUSE treated as a pass                 -> T31/36/38/40/42 red
  M50 node: REFUSE treated as a pass                  -> T45  red
  M51 check: FAIL treated as a pass                   -> 15 cases red
  M52 node: FAIL treated as a pass                    -> 5 cases red
  M53 ENV_FILE path moved                             -> T43  red (the node
      green baseline is itself killable, so it is not a tautology)

THREE OF THE HARNESS'S OWN CASES WERE FOUND TOO WEAK WHILE BUILDING THIS, and
were STRENGTHENED rather than left passing -- each would have stayed green after
the thing it names was removed:
  T16 grepped the WHOLE FILE for `.snapcraftcontent.com`, which the header
      discusses in prose, so narrowing the actual STORE_DOMAINS value left it
      green. Now scoped to the `STORE_DOMAINS=` line.
  T18 grepped the whole file for `systemctl disable --now squid`, which check's
      own failure MESSAGE contains, so install could stop doing it. Now scoped
      to do_install's body.
  T20 matched `apt-get install` inside the guidance STRING, so it read green
      whether or not the script apt-installed. Now blanks double-quoted strings
      first and matches an INVOCATION position.
Same class, found the same way: T10 and T50 had to blank comments / quoted
strings for the opposite reason -- they were RED on legitimate prose (measured
`virbr5` evidence in a site-row comment; an `ip route get` hint inside a failure
message). Rewording the prose to satisfy a regex would have been gaming the
test, so the tests were fixed to assert on CODE.

WHAT THE MUTATION PASS DOES NOT PROVE: that the assertions are the RIGHT ones,
or that the proxy works. It proves only that none of them is decoration.
