diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 82340eb..517b05c 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1513,11 +1513,18 @@ MAAS-generated under a per-revision `/var/snap/maas/current/` path, has NO destination restriction, and is the hidden-coupling anti-pattern D-135 already has a scar from -- but it is a real operator option at apply time, not a dismissal. - **TWO THINGS THIS BUILD DOES NOT PROVE, said plainly:** no proxy has ever been installed, so - no snap has been installed through one (the harness green is a FIXTURE green); and whether a + **THREE THINGS THIS BUILD DOES NOT PROVE, said plainly:** no proxy has ever been installed, + so no snap has been installed through one (the harness green is a FIXTURE green); whether a RUNNING snapd re-reads the proxy without a restart is UNVERIFIED (snapd writes `/etc/environment` and reads it via `EnvironmentFile=` at service start; LP#1737332, - LP#1791587) -- so per client verify BOTH `snap get system proxy` AND a real egress probe. + LP#1791587) -- so per client verify BOTH `snap get system proxy` AND a real egress probe; + and the exact curl shape of a **dstdomain** denial was not measured (the captured + `CONNECT tunnel failed, response 403` came from a PORT denial), which is safe because an + unrecognised shape REFUSES and can never grant a pass. + **HOW TO READ A `REFUSE` FROM THIS GATE:** two of `check`'s probes reach Canonical hosts, so + an upstream outage or DNS failure makes a HEALTHY proxy REFUSE. That is by design -- a probe + that cannot tell "the CDN is down" from "the proxy is misconfigured" must not guess -- and + REFUSE is never a pass. **A BUILD-TIME CHOICE THE OPERATOR HAS NOT RULED ON:** the destination allowlist. The ruling names the mechanism, not the scope; the alternative is allowing CONNECT to any `:443`, which hands the node planes general HTTPS egress. diff --git a/docs/audit/stage5-snap-proxy-measurements-20260731.txt b/docs/audit/stage5-snap-proxy-measurements-20260731.txt index 3272802..5e715ab 100644 --- a/docs/audit/stage5-snap-proxy-measurements-20260731.txt +++ b/docs/audit/stage5-snap-proxy-measurements-20260731.txt @@ -427,6 +427,25 @@ f. The squid config generated by this build has never been parsed by a squid binary. `install` runs `squid -k parse -f ` before enabling anything, so a syntax error fails loud at apply time rather than silently. + g. THE DENY-SHAPE MEASUREMENT IS FROM A *PORT* DENY, NOT A *DSTDOMAIN* DENY. + Section 7's `curl: (56) CONNECT tunnel failed, response 403` was produced by + MAAS's squid refusing `api.snapcraft.io:25` under `http_access deny CONNECT + !SSL_ports` -- a PORT rule. This build's deny probe trips a DSTDOMAIN rule + (`http_access deny !snap_store`). Squid answers both with 403 at CONNECT, so + the two should be byte-identical to curl, but that path was NOT measured. + THE FAILURE MODE IS SAFE: check's deny branch treats anything that is neither + "403 at CONNECT" nor "curl exit 0" as REFUSE, so an unexpected shape can only + withhold a pass, never grant one. Confirm at apply time, once: run the deny + probe by hand against the built proxy and read the exact curl line. + h. TWO OF check's PROBES ARE LIVENESS-COUPLED TO CANONICAL. The store-API probe + and the CDN probe reach real third-party hosts, so an upstream outage or a + DNS failure makes a healthy proxy REFUSE. That is deliberate -- a probe that + cannot distinguish "the CDN is down" from "the proxy is misconfigured" must + not guess -- but it means a REFUSE is not by itself evidence about the proxy, + and the script header says so where a triaging operator will read it. The + static half of the same property (that the CDN wildcard is PRESENT in + STORE_DOMAINS) is asserted by the config diff and by harness case T16, which + have no external dependency at all. 11. MUTATION PASS ON tests/dc-snap-proxy/run-tests.sh (2026-07-31) diff --git a/docs/changelog-20260731-prefer-ipv6-research.md b/docs/changelog-20260731-prefer-ipv6-research.md index 477f423..0231612 100644 --- a/docs/changelog-20260731-prefer-ipv6-research.md +++ b/docs/changelog-20260731-prefer-ipv6-research.md @@ -577,3 +577,122 @@ (23/1); restored byte-identical, 24/24. Gauntlet ALL GREEN (93); repo-lint 0 fail. **Revert.** `git revert` -- generator and harness move together. + +## Item 19 -- BUILD D-135/D-107 RULING 2, REPO SIDE ONLY: the in-DC snap forward proxy + +**What.** `scripts/dc-snap-proxy.sh` (NEW, `check|install|node`, site-keyed dc0/dc1) + +`tests/dc-snap-proxy/run-tests.sh` (NEW, 52 cases) + `tests/HARNESS-MANIFEST` (one name +added) + `docs/audit/stage5-snap-proxy-measurements-20260731.txt` (NEW) + +`docs/CURRENT-STATE.md` (the RULING-2 status paragraph). It implements the ruled +"HTTP(S) forward proxy in the DC utility band + juju snap-https-proxy" as a **dedicated +squid instance ON THE RACK at the D-134 utility `.4:3129`**, CONNECT-only and restricted +to the Canonical-documented snap-store/CDN hosts. + +**NOTHING WAS APPLIED.** No package installed, no service started, no `juju config` or +`juju model-config` set, no file written on any rack or node. **The D-135 items 2-3 gap +is still OPEN** and the operator applies this separately. Two declared side effects of +the read-only probing: the `curl` probes traversed MAAS's existing squid and may have +added entries to ITS cache, and `apt-get download squid` wrote a .deb into `/tmp` ON THE +JUMPHOST so the shipped squid config could be READ rather than assumed. + +**Why these design choices, each from a MEASUREMENT rather than a preference.** +1. **The client ACL is METAL-ADMIN, not metal-internal.** The three failing applications + are LXD CONTAINERS. `juju status` prints `mysql-innodb-cluster/0` at `10.12.12.116`, + but measured INSIDE it: `eth0 10.12.8.122/22`, `eth1 10.12.12.116/22`, + `ip route get 10.12.8.4 -> src 10.12.8.122`, and NO default route. An ACL built on the + displayed address would have denied every client the proxy exists for. +2. **Port 3129, and a DEDICATED instance.** MAAS's own squid holds `*:3128` AND `*:8000` + WILDCARD-bound on BOTH racks. Ubuntu noble's `squid.conf` pins `http_port 3128` and + `/etc/default/squid` is EMPTY, so the packaged unit cannot be repointed and can only + FATAL on bind -- `install` disables it and `check` fails if it is active or enabled. + (`conf.d` was checked too: its include sits before `http_access deny all`, but a + drop-in cannot UNSET an `http_port`.) +3. **No new D-134 octet.** The utility `.4` is already aliased on both racks, so the + ruling note's "if it takes its own VM it needs a ruled octet first" never triggers. +4. **This script OWNS its net layer** (`-snap-proxy-net.service` + helper + + resolved drop-in), never reusing the mirror's or the cache proxy's -- the 2026-07-27 + D-135 lesson, with `ip addr replace`/`ip route replace` so the units coexist. +5. **squid, not tinyproxy.** Every documented snap endpoint is `:443`, so a CONNECT proxy + is required and **no caching is possible at all** (squid "does not have access to those + encrypted messages"; the only escape is SslBump, a deliberate MITM). So the choice is + support posture: squid is noble **main** and defaults to `deny CONNECT !SSL_ports` + + `deny all`; tinyproxy is universe with six CVEs in "Needs evaluation" and defaults to + "all ports allowed" for CONNECT. nginx cannot do CONNECT at all. + +**LOGGED, NOT ADOPTED, and not buried: MAAS's squid already does this job.** Measured +working to `api.snapcraft.io` from the rack, from the controller VM AND from the failing +container, and machines already consume `http://10.12.8.6:8000` for apt via +`90curtin-aptproxy`. So `juju model-config snap-https-proxy=http://10.12.8.6:8000` would +plausibly work today with no rack build. It was not taken because that config is +MAAS-generated under a per-revision `/var/snap/maas/current/` path, carries NO destination +restriction, and is exactly the hidden-coupling anti-pattern D-135 already has a scar +from. It remains a real operator option at apply time. + +**A BUILD-TIME CHOICE THE OPERATOR HAS NOT RULED ON -- flagged, not smuggled.** The ruling +names the mechanism, not the scope. This build restricts CONNECT to the documented store + +CDN hosts; the alternative is allowing any `:443`, which hands the node planes general +HTTPS egress. Widening is one edit to `STORE_DOMAINS`, and the script header says so. + +**`check` is built as a GA-R6 gate, not a status printer.** Three outcomes (PASS 0 / FAIL 1 +/ REFUSE 3). It diffs all five owned artifacts against their generators (CONTENT, not +existence -- the `dc-mirror.sh` false-green class); asserts the bind is on the LISTEN +address and not wildcard; and makes three behavioral probes from the same source so each +isolates one rule: the store API asserted on `"channel-map"` + `sha3-384` (a 200 is not +evidence), the CDN tunnel asserted as NOT refused (snapd downloads through the same proxy, +so an API-only allowlist would authenticate and then fail), and a **deny-direction probe +that proves the allowlist is in force** -- if it is served, the verdict is "OPEN forward +proxy on the node plane". Every "could not look" REFUSES. + +**Mutation-proven: 53 mutations run; 52 KILLED on the first pass, ONE recorded MALFORMED +and redone, final 53/53 killed, 0 survived** (`docs/audit/...` section 11 lists every one). +The malformed one is stated in that order deliberately: the driver printed `[SURVIVED] M18`, +and the cause was my mutation, not a weak assertion -- `true # squid -k parse -f` commented +out the following `then`, broke `bash -n`, and turned 29 cases red without turning T19 red. +A mutation that breaks syntax tests nothing. Redone as `if false; then` + `:`, T19 went red +alone. Each mutation neuters a PREDICATE, not a message (`miss`->`say`, `refuse`->`say`, +`nno`->`nok`, `elif `->`elif false`, or a value swapped for a plausible wrong one); +the script was restored and sha256-verified byte-identical after each; baseline and final +both `52 passed, 0 failed`. +**THREE OF MY OWN CASES WERE FOUND TOO WEAK while doing this and were STRENGTHENED, not +left green:** T16 grepped the whole file for a CDN host the header discusses in prose, so +narrowing the real allowlist left it green; T18 grepped the whole file for a command that +`check`'s own failure message contains, so `install` could stop doing it; T20 matched +`apt-get install` inside the guidance STRING. All three now assert on the scoped value, +the function body, and an invocation position respectively. + +**HARNESS-MANIFEST re-recorded deliberately** (`bash scripts/run-tests-all.sh +--record-manifest`), because R15(2) requires the reason be stated: exactly one harness was +ADDED (`dc-snap-proxy`), 93 -> 94, and without re-recording the gauntlet fails +"harness set DRIFTED" rather than running it. + +**What this does NOT prove, said plainly.** No proxy has ever been installed, so no snap +has been installed through one -- the harness green is a FIXTURE green. Whether a RUNNING +snapd re-reads the proxy without a restart is UNVERIFIED (snapd writes `/etc/environment` +and reads it via `EnvironmentFile=` at service start; LP#1737332, LP#1791587), so per +client verify BOTH `snap get system proxy` AND a real egress probe. dc1 has no deployed +model, so the container-source finding is measured on dc0 only; `node dc1` is what will +prove it there. And the generated squid config has never been parsed by a squid binary -- +`install` runs `squid -k parse` before enabling anything, so that fails loud at apply time. + +**>>> THIS ITEM'S CHANGE SET IS SPLIT ACROSS HEAD AND THE WORKING TREE, AND NOT BY ME. <<<** +A CONCURRENT session committed **`e57ad09` ("Teardown: stalled on stopped agents, forced +clear, no nodes stranded")** at 21:34:58 with `git add -A` semantics, and it SWEPT UP two of +this item's files -- `docs/audit/stage5-snap-proxy-measurements-20260731.txt` and the +`docs/CURRENT-STATE.md` paragraph -- into a commit whose message is entirely about the model +teardown and does not mention the snap proxy. **I ran no `git commit` and authored no commit.** +Consequences for review: the diff of the working tree does NOT show the whole item, and the +still-untracked `scripts/dc-snap-proxy.sh` + `tests/dc-snap-proxy/` will be swept into that +session's NEXT commit unless they are dealt with first. + +**Revert, stated per half because a single command no longer covers it.** +- ALREADY IN HEAD (e57ad09), so `git checkout --` is a NO-OP on them and would silently leave + the status claim standing: `docs/audit/stage5-snap-proxy-measurements-20260731.txt` (whole + file, added by that commit) and the `docs/CURRENT-STATE.md` RULING-2 paragraph (an addition + inside a file that commit also changed for unrelated teardown reasons). Reverting means a + FOLLOW-UP commit that `git rm`s the capture and deletes the paragraph -- NOT `git revert + e57ad09`, which would also undo that session's teardown record. +- STILL WORKING-TREE ONLY: `git checkout -- docs/changelog-20260731-prefer-ipv6-research.md + tests/HARNESS-MANIFEST` (the manifest returns to 93 names on its own that way); + `rm -f scripts/dc-snap-proxy.sh`; `rm -rf tests/dc-snap-proxy`. +Nothing outside the repo changes either way -- **there is no live state to roll back**, which +is the one thing this split does not put at risk. diff --git a/scripts/dc-snap-proxy.sh b/scripts/dc-snap-proxy.sh new file mode 100644 index 0000000..07900e1 --- /dev/null +++ b/scripts/dc-snap-proxy.sh @@ -0,0 +1,636 @@ +#!/usr/bin/env bash +# scripts/dc-snap-proxy.sh -- per-DC snap FORWARD +# PROXY (squid, CONNECT-only). Builds D-135/D-107 RULING 2, RULED 2026-07-31 +# (GA-R5), operator utterance verbatim: +# "HTTP(S) forward proxy in the DC utility band + juju snap-https-proxy (Recommended)" +# +# WHY IT EXISTS. mysql-innodb-cluster (mysql-shell), ovn-central +# (prometheus-ovn-exporter) and vault (core) all fail their install hook because +# they cannot reach api.snapcraft.io -- the D-135 items 2-3 gap. It hits dc1 +# IDENTICALLY: apt-cacher-ng proxies apt, never snaps. D-107's core statement +# stays TRUE and UNAMENDED -- nodes reach an IN-DC proxy, not the store. +# +# RUNS ON THE DC RACK HOST, invoked over ssh exactly like dc-mirror.sh / +# dc-cache-proxy.sh (no repo clone needed there): +# check: ssh -i -J voffice1 @ \ +# 'sudo bash -s' -- check dc0 < scripts/dc-snap-proxy.sh +# install: same with 'install dc0' (idempotent; safe to re-run). +# +# ...EXCEPT `node`, WHICH RUNS ON A DC NODE OR A CHARM CONTAINER. Same split +# dc-cache-proxy.sh had to make for gate G17: `check` asserts rack-side facts a +# client cannot satisfy, so pointing a client at it fails for reasons that say +# nothing about client reachability. +# node: bash -s -- node dc0 < scripts/dc-snap-proxy.sh (ON the client) +# `node` touches ONLY curl, grep and /etc/environment -- no root, no systemctl, +# no virsh, no generated-file diffs. +# +# --------------------------------------------------------------------------- +# THE MEASUREMENT THE WHOLE DESIGN TURNS ON (docs/audit/stage5-snap-proxy- +# measurements-20260731.txt section 1). The three failing applications are LXD +# CONTAINERS, not nodes: `juju status` prints mysql-innodb-cluster/0 at +# 10.12.12.116, which is no D-134 node octet. Measured INSIDE the container: +# eth0 10.12.8.122/22 (metal-admin) eth1 10.12.12.116/22 (metal-internal) +# ip route get 10.12.8.4 -> dev eth0 src 10.12.8.122 +# NO default route; DIRECT api.snapcraft.io -> curl (7); via a proxy -> 200 +# So the client ACL is the METAL-ADMIN /22, NOT metal-internal. Binding the ACL +# to the address `juju status` displays would have denied every client this +# proxy exists to serve. +# +# WHY A DEDICATED SQUID INSTANCE AND NOT THE PACKAGED squid.service. MEASURED on +# BOTH racks: MAAS's own squid holds *:3128 AND *:8000, WILDCARD-bound +# (`squid -N -d 5 -f /var/snap/maas//proxy/maas-proxy.conf`). Ubuntu noble's +# squid.conf pins `http_port 3128` and /etc/default/squid is EMPTY, so there is +# no supported way to move the packaged unit off 3128 -- it would FATAL on bind. +# `include /etc/squid/conf.d/*.conf` does sit before `http_access deny all`, but +# a drop-in cannot UNSET an http_port. Hence: our own config, our own unit, our +# own pid file, and `install` DISABLES the packaged unit so it cannot crash-loop +# and poison triage. Port 3129 -- 3128/8000 are MAAS's, 3142 is dc1's +# apt-cacher-ng, 80 is dc0's mirror nginx. Bound to LISTEN specifically, never +# wildcard. Coexistence with MAAS's squid is MEASURED, not reasoned: snapd +# namespaces its shm as `snap.maas.squid-cf__*` while a host squid creates bare +# `squid-cf__*`, and MAAS's squid runs `-N` with no pid file. +# +# THIS SCRIPT OWNS ITS OWN NETWORK PREREQUISITES -- the hard-won D-135 lesson +# (dc-cache-proxy.sh, 2026-07-27). It never reuses -mirror-net.service or +# -cache-proxy-net.service: on a DC running only one of those, tearing that +# one down must not be able to kill this proxy, and this proxy must be rebuildable +# without reinstalling the other's apparatus. Artifacts are named for THIS +# service. The three net units set IDENTICAL MEASURED values with `ip addr +# replace` / `ip route replace`, so they are no-ops for each other. +# +# WHY SQUID (research capture section 9; "research is cheap, guesswork is +# expensive"). Every documented snap endpoint is :443 +# (snapcraft.io/docs/reference/administration/network-requirements/), so a +# CONNECT-capable forward proxy is REQUIRED and there is NO caching benefit -- +# squid "does not have access to those encrypted messages" +# (wiki.squid-cache.org/Features/HTTPS) and the only way to change that is +# SslBump, i.e. a deliberate MITM, which is not wanted. So the choice is on +# support and default posture: squid is in noble MAIN (6.14-0ubuntu0.24.04.4, +# security-supported) and ships `http_access deny CONNECT !SSL_ports` plus a +# terminal `deny all` by DEFAULT; tinyproxy is universe with six CVEs in "Needs +# evaluation" and documents "If no ConnectPort line is found, then all ports are +# allowed". nginx cannot do CONNECT without a third-party module. +# +# >>> BUILD-TIME CHOICE, NOT A RULING -- READ BEFORE APPLYING. <<< +# The ruling covers "forward proxy + juju snap-https-proxy" and leaves the +# service choice to build time. It does NOT rule on DESTINATION ALLOWLISTING. +# This build restricts CONNECT to the Canonical-documented snap-store and CDN +# hosts (STORE_DOMAINS below), because "allow CONNECT to any :443" would hand +# the node planes general HTTPS egress and hollow out D-107 in spirit while +# leaving its letter intact. The cost is that an UNDOCUMENTED store/CDN host +# would be denied. Widening is ONE edit: add the host to STORE_DOMAINS, or +# replace the `http_access deny !snap_store` line with nothing to allow any +# :443. If the operator prefers the open form, say so and it changes here. +# +# CONSUMPTION (Stage 5), verified against THIS controller (juju 3.6.27): +# juju model-config -m snap-http-proxy=http://:3129 +# juju model-config -m snap-https-proxy=http://:3129 +# # LEAVE snap-store-proxy / snap-store-proxy-url / snap-store-assertions EMPTY +# # -- those are the Canonical Enterprise Store (ex Snap Store Proxy) product: +# # snapd rejects proxy.store without a matching signed store assertion. +# Both keys take an http:// URL (a plain-HTTP proxy speaking CONNECT), never +# https://. juju's proxyupdater shells out `snap set system proxy.http=... +# proxy.https=...` and is a WATCHER, so setting the key after machines exist is +# expected to propagate. +# UNVERIFIED, CARRIED DELIBERATELY: whether a RUNNING snapd re-reads the value +# without a restart. snapd writes /etc/environment and reads it via +# EnvironmentFile= AT SERVICE START (snapd configcore/proxy.go; LP#1737332, +# LP#1791587). So per client verify BOTH `snap get system proxy` AND a real +# egress probe; if the first passes and the second fails, `systemctl restart +# snapd` and re-probe. `node` below asserts the /etc/environment half without +# needing root. +# +# READ THIS BEFORE TRIAGING A `REFUSE`. Two of check's probes reach THIRD-PARTY +# hosts, so a Canonical-side outage, a DNS failure or a dead upstream makes a +# perfectly healthy proxy REFUSE. THAT IS BY DESIGN, not a defect and not "the +# proxy is broken": the probe cannot tell "the CDN is down" from "the proxy is +# misconfigured", and guessing either way is the failure this repo keeps meeting. +# REFUSE means EVALUATE AGAIN, WITH MORE INFORMATION -- it is never a pass. +# The CDN probe in particular is a LIVENESS-COUPLED assertion: it exists because +# a config diff proves the wildcard `.snapcraftcontent.com` is WRITTEN, and only +# a real tunnel proves squid's dot-prefix matching actually ADMITS a subdomain. +# (Measured aside worth knowing: `core`'s own download URL came back on +# api.snapcraft.io, not a CDN host -- so for THIS snap the CDN entry may be +# unused. It stays because the allowlist must cover every snap, not the one +# probed, and snapd downloads through the same proxy as the API.) +# +# EXIT: 0 ok | 1 check/node failed | 2 bad args/unknown site +# | 3 REFUSED (could not evaluate -- never a pass) | 4 install failed. +# ASCII + LF only. +set -uo pipefail + +MODE="${1:-}"; SITE="${2:-}" +case "$MODE" in check|install|node) ;; *) + echo "usage: dc-snap-proxy.sh " >&2; exit 2 ;; esac + +# --------------------------------------------------------------------------- +# Site table. ADD A SITE ONLY WITH MEASURED/RULED VALUES (hard rule 2); the +# harness rejects rows without a MEASURED tag. CLIENT_CIDR is the plane the +# charm containers SOURCE FROM toward the utility .4 -- measured, never the +# address juju status happens to print. +# --------------------------------------------------------------------------- +case "$SITE" in + dc0) + # MEASURED/RULED 2026-07-31 (this build's capture, + # docs/audit/stage5-snap-proxy-measurements-20260731.txt): + # LISTEN 10.12.8.4/22 -- the D-134 utility .4 (artifact service), ALREADY + # aliased on virbr2 by dc0-mirror-net.service; no new octet is needed, + # which is what the ruling note asked for. + # EDGE_GW 10.12.4.1 -- dc0 edge LAN gw; the rack's measured default route + # (`default via 10.12.4.1 dev virbr5`) and the proxy's upstream egress. + # CLIENT_CIDR 10.12.8.0/22 -- metal-admin. MEASURED from inside + # mysql-innodb-cluster/0: `ip route get 10.12.8.4` -> src 10.12.8.122. + # Cross-check: lib-net.sh vr1-dc0 PLANE_CIDRS[1] = 10.12.8.0/22. + UTIL_NET="vr1-dc0-metal-admin" + LISTEN_CIDR="10.12.8.4/22" + EDGE_GW="10.12.4.1" + CLIENT_CIDR="10.12.8.0/22" + ;; + dc1) + # MEASURED/RULED 2026-07-31: + # LISTEN 10.12.68.4/22 -- utility .4, MEASURED already aliased on virbr6 + # by dc1-cache-proxy-net.service (`ip -4 -o addr` on vr1-dc1-rack). + # EDGE_GW 10.12.64.1 -- MEASURED as the dc1 rack's default route + # (`default via 10.12.64.1 dev virbr4`). + # CLIENT_CIDR 10.12.68.0/22 -- dc1 metal-admin, from lib-net.sh's vr1-dc1 + # arm (PLANE_CIDRS[1]), NOT mirrored from dc0 by symmetry. + # HONEST LIMIT (hard rule 2): dc1 has NO deployed model, so the + # container-sources-from-metal-admin finding is measured on dc0 ONLY. If dc1 + # ever carves containers differently, `node dc1` is what will say so -- and a + # red there is the intended behaviour, not a false red. + UTIL_NET="vr1-dc1-metal-admin" + LISTEN_CIDR="10.12.68.4/22" + EDGE_GW="10.12.64.1" + CLIENT_CIDR="10.12.68.0/22" + ;; + *) echo "FAIL: unknown site '$SITE' -- add a MEASURED row block first" >&2; exit 2 ;; +esac +LISTEN="${LISTEN_CIDR%%/*}" + +# 3129: MEASURED FREE on both racks while 3128 and 8000 are MAAS's squid, 3142 is +# dc1's apt-cacher-ng and 80 is dc0's mirror nginx. +PROXY_PORT="3129" +# TEST SEAM. EMPTY in every production invocation, so the paths below are +# byte-identical to the absolute ones. The harness points it at a mktemp tree so +# that check's five file assertions and node's /etc/environment assertion can be +# driven in BOTH directions -- an assertion nobody can turn red is decoration. +# In-repo precedent for the idiom: MAAS_SNAP_ROOT (maas-region-power-key.sh), +# CREDS_ROOT (creds-audit.sh), CLOUD_SNAPSHOT_ROOT (cloud-snapshot.sh). +SYSROOT="${DC_SNAP_PROXY_SYSROOT:-}" +SQUID_CONF="${SYSROOT}/etc/squid/${SITE}-snap-proxy.conf" +PROXY_UNIT="${SYSROOT}/etc/systemd/system/${SITE}-snap-proxy.service" +NET_HELPER="${SYSROOT}/usr/local/sbin/${SITE}-snap-proxy-net-apply" +NET_UNIT="${SYSROOT}/etc/systemd/system/${SITE}-snap-proxy-net.service" +RESOLVED_DROPIN="${SYSROOT}/etc/systemd/resolved.conf.d/${SITE}-snap-proxy.conf" +ENV_FILE="${SYSROOT}/etc/environment" +RUNTIME_DIR_NAME="${SITE}-snap-proxy" +PID_FILE="/run/${RUNTIME_DIR_NAME}/squid.pid" +LOG_DIR="/var/log/squid" + +# The Canonical-documented snap endpoint set, all :443 +# (snapcraft.io/docs/reference/administration/network-requirements/ and its +# source of record forum.snapcraft.io/t/network-requirements/5147, CDN list dated +# 2021-10-25). A leading dot is squid's "this domain and all subdomains", so +# `.snapcraftcontent.com` covers storage./canonical-lgw01.cdn./cloudfront.cdn. +# `.cdn.snapcraft.io` is marked DEPRECATED upstream and kept only so an older +# client is not denied. THE DOWNLOAD IS PROXIED TOO -- snapd uses ONE http client +# for the API and the CDN fetch (snapd store/store.go), so an allowlist naming +# only api.snapcraft.io would authenticate and then fail the download. +STORE_DOMAINS="api.snapcraft.io dashboard.snapcraft.io login.ubuntu.com .snapcraftcontent.com .cdn.snapcraft.io" + +# Behavioral probe targets, shared by the rack-side `check` and the client-side +# `node` so the two cannot drift apart. Shapes MEASURED 2026-07-31 (capture s7). +STORE_PROBE_URL="https://api.snapcraft.io/v2/snaps/info/core" +CDN_PROBE_URL="https://canonical-lgw01.cdn.snapcraftcontent.com/" +# A destination deliberately NOT in STORE_DOMAINS. RFC 2606 / IANA reserved, so it +# can never become a real dependency. Its only job is to prove the allowlist is in +# force -- a proxy that answers this one is an OPEN proxy. +DENY_PROBE_URL="https://example.com/" + +# --------------------------------------------------------------------------- +# Generated content -- the single source of truth for check AND install. Every +# generator is diffed by `check`, so a hand-edit on the rack is a FAILURE, not a +# silent drift (assert the ARTIFACT, never the intent). +# --------------------------------------------------------------------------- +gen_squid_conf() { + cat < snap-http-proxy=http://${LISTEN}:${PROXY_PORT} +# juju model-config -m snap-https-proxy=http://${LISTEN}:${PROXY_PORT} +# Leave snap-store-proxy / -url / -assertions EMPTY (different product). +visible_hostname ${SITE}-snap-proxy +pid_filename ${PID_FILE} +http_port ${LISTEN}:${PROXY_PORT} + +# Clients: the metal-admin plane. MEASURED as the source the charm CONTAINERS +# select toward this address, not the address juju status displays. +acl snap_clients src ${CLIENT_CIDR} +# The rack's own loopback, so this script's behavioral smoke test can run. It +# proves upstream reach, CONNECT capability and that the destination rule is in +# force; CLIENT AUTHORIZATION is what \`node\` proves, from a real client. +acl snap_probe src 127.0.0.1/32 +acl SSL_ports port 443 +acl Safe_ports port 443 +acl CONNECT method CONNECT +acl snap_store dstdomain ${STORE_DOMAINS} + +http_access deny !Safe_ports +http_access deny CONNECT !SSL_ports +http_access deny !snap_store +http_access allow snap_clients +http_access allow snap_probe +http_access deny all + +# No caching is POSSIBLE here (opaque CONNECT tunnels) -- stated explicitly so +# nobody later "fixes" the absence of a cache_dir by adding SslBump. +cache deny all + +access_log ${LOG_DIR}/${SITE}-snap-proxy.access.log +cache_log ${LOG_DIR}/${SITE}-snap-proxy.cache.log +coredump_dir /var/spool/squid +EOF +} + +gen_proxy_unit() { + cat </dev/null \ + | sed -n "s/.*bridge name='\([^']*\)'.*/\1/p" +} + +# --------------------------------------------------------------------------- +# check -- THREE outcomes, never two: PASS (0), FAIL (1), REFUSE (3). +# GA-R6: this is what a stage would close on, so it must be able to fail, it +# asserts CONTENT and not existence, and a state it cannot evaluate REFUSES +# rather than defaulting to success. "Could not look" is never "nothing there". +# It mutates no config or service state (the probes may add lines to squid's +# access log -- benign, and the proof it actually serves). +# --------------------------------------------------------------------------- +FAILED=0; REFUSED=0 +say(){ echo " OK $1"; } +miss(){ echo " MISS $1"; FAILED=1; } +refuse(){ echo " REFUSE $1"; REFUSED=1; } + +check_file() { # path gen_fn + if [ ! -f "$1" ]; then miss "$1 absent"; return; fi + if diff -q <("$2") "$1" >/dev/null 2>&1; then say "$1 matches generated" + else miss "$1 DIFFERS from generated content"; fi +} + +do_check() { + echo "dc-snap-proxy check ($SITE) -- ${LISTEN}:${PROXY_PORT}, clients ${CLIENT_CIDR}" + + # -- package. An absence here is an OBSERVED absence, so it FAILS; it does not + # refuse. Fail-loud with the exact apt line, like dc-mirror/dc-cache-proxy. + if dpkg -s squid >/dev/null 2>&1; then say "squid installed" + else miss "squid not installed -- run: apt-get update && apt-get install -y squid"; fi + + # -- the five artifacts this script owns, asserted by CONTENT. + check_file "$SQUID_CONF" gen_squid_conf + check_file "$PROXY_UNIT" gen_proxy_unit + check_file "$NET_HELPER" gen_net_helper + check_file "$NET_UNIT" gen_net_unit + check_file "$RESOLVED_DROPIN" gen_resolved_dropin + + # -- units. ENABLED matters as much as active: without it the LISTEN alias and + # the proxy both vanish at the next rack reboot, and this rack has rebooted. + if ! command -v systemctl >/dev/null 2>&1; then + refuse "systemctl is not present -- cannot evaluate unit state" + else + for u in "${SITE}-snap-proxy.service" "${SITE}-snap-proxy-net.service"; do + if [ "$(systemctl is-enabled "$u" 2>/dev/null)" = "enabled" ]; then + say "$u enabled" + else miss "$u not enabled (it would not survive a reboot)"; fi + done + if systemctl is-active --quiet "${SITE}-snap-proxy.service" 2>/dev/null; then + say "${SITE}-snap-proxy.service active" + else miss "${SITE}-snap-proxy.service not active"; fi + # The PACKAGED unit must be neither active nor enabled: its squid.conf pins + # http_port 3128, which MAAS's squid holds, so it can only crash-loop -- and + # a crash-looping squid alongside ours poisons every later triage. + if systemctl is-active --quiet squid 2>/dev/null; then + miss "the PACKAGED squid.service is ACTIVE -- it pins http_port 3128 (MAAS's squid holds it); run: systemctl disable --now squid" + elif [ "$(systemctl is-enabled squid 2>/dev/null)" = "enabled" ]; then + miss "the PACKAGED squid.service is ENABLED -- it will fight for 3128 at the next boot; run: systemctl disable --now squid" + else say "packaged squid.service neither active nor enabled"; fi + fi + + # -- the listener must be on the LISTEN address SPECIFICALLY, not wildcard: a + # wildcard bind would collide with MAAS's squid and would also expose the proxy + # on the transit and provider legs. + if ! command -v ss >/dev/null 2>&1; then + refuse "ss is not present -- cannot evaluate the listener" + elif ss -ltn 2>/dev/null | grep -q "${LISTEN}:${PROXY_PORT} "; then + say "listening on ${LISTEN}:${PROXY_PORT}" + elif ss -ltn 2>/dev/null | grep -qE "(\*|0\.0\.0\.0):${PROXY_PORT} "; then + miss "port ${PROXY_PORT} is bound WILDCARD, not on ${LISTEN} -- refusing to accept a broader bind than generated" + else + miss "nothing listening on ${LISTEN}:${PROXY_PORT}" + fi + + # -- the net layer's live effect (the artifacts were diffed above). + if ! command -v virsh >/dev/null 2>&1; then + refuse "virsh is not present -- cannot resolve ${UTIL_NET}'s bridge, so the LISTEN alias is UNKNOWN" + else + br="$(util_bridge)" + if [ -z "$br" ]; then + refuse "libvirt network '${UTIL_NET}' did not resolve to a bridge -- the alias is UNKNOWN, not absent" + elif ip -4 -o addr show dev "$br" 2>/dev/null | grep -q " ${LISTEN}/"; then + say "${LISTEN_CIDR} on $br ($UTIL_NET)" + else + miss "${LISTEN_CIDR} absent on the metal-admin bridge (run: dc-snap-proxy.sh install $SITE)" + fi + fi + if ip route show default 2>/dev/null | grep -q "via ${EDGE_GW}"; then + say "default route via edge ${EDGE_GW}" + else miss "default route via ${EDGE_GW} absent -- the proxy has no upstream egress"; fi + + # -- BEHAVIORAL. Three probes from the SAME source, so the difference between + # them isolates exactly one rule each. A 200 alone is never evidence. + if ! command -v curl >/dev/null 2>&1; then + refuse "curl is not present -- cannot evaluate whether the proxy actually serves" + else + # (1) ALLOW direction, asserted on CONTENT. MEASURED body shape 2026-07-31: + # the store API answers with a "channel-map" array carrying a sha3-384 + # download digest. No squid error page contains either. + body="$(mktemp)"; cerr="$(mktemp)" + code="$(curl -sS -o "$body" -w '%{http_code}' -x "http://${LISTEN}:${PROXY_PORT}" \ + -m 30 -H 'Snap-Device-Series: 16' "$STORE_PROBE_URL" 2>"$cerr")"; crc=$? + if [ "$crc" -ne 0 ]; then + refuse "could not reach $STORE_PROBE_URL THROUGH http://${LISTEN}:${PROXY_PORT} (curl exit $crc: $(head -1 "$cerr")) -- unreachable is not 'nothing there'" + elif [ "$code" != "200" ]; then + miss "store API returned HTTP $code through the proxy (want 200)" + elif ! grep -q '"channel-map"' "$body"; then + miss "body served for the store API has no \"channel-map\" -- what came back is not the snap store's answer, whatever its status line said" + elif ! grep -q 'sha3-384' "$body"; then + miss "store API body carries no sha3-384 download digest -- not a usable snap info response" + else + say "CONNECT to the store API: curl exit 0, HTTP 200, channel-map + sha3-384 present" + fi + rm -f "$body" "$cerr" + + # (2) The CDN tunnel must be ALLOWED. The CDN answers 403 to a bare GET of /, + # so the assertion is NOT on the status code -- it is that squid did not + # refuse the tunnel. MEASURED distinction: a proxy denial is curl exit 56 + # with "CONNECT tunnel failed, response 403"; a server 403 is curl exit 0. + # Without this the allowlist could name only api.snapcraft.io and every + # snap would authenticate and then fail its DOWNLOAD. + cout="$(curl -sS -o /dev/null -w '%{http_code}' -x "http://${LISTEN}:${PROXY_PORT}" \ + -m 30 "$CDN_PROBE_URL" 2>&1)"; crc=$? + if grep -q 'CONNECT tunnel failed' <<<"$cout"; then + miss "the proxy REFUSED the CDN tunnel to $CDN_PROBE_URL -- the allowlist is too narrow and snap DOWNLOADS would fail after a successful API call" + elif [ "$crc" -eq 0 ]; then + say "CDN tunnel allowed (curl exit 0; the CDN's own status was '$cout')" + else + refuse "CDN probe gave curl exit $crc with '$cout' -- neither an allowed tunnel nor a proxy denial; refusing to interpret it" + fi + + # (3) DENY direction. This is the assertion that proves the destination + # allowlist is actually in force rather than merely written down. A proxy + # that answers this one is an OPEN proxy on the node plane. + dout="$(curl -sS -o /dev/null -w '%{http_code}' -x "http://${LISTEN}:${PROXY_PORT}" \ + -m 20 "$DENY_PROBE_URL" 2>&1)"; drc=$? + if grep -q 'CONNECT tunnel failed, response 403' <<<"$dout"; then + say "non-allowlisted destination is DENIED (403 at CONNECT) -- the allowlist is in force" + elif [ "$drc" -eq 0 ]; then + miss "the proxy SERVED a non-allowlisted destination ($DENY_PROBE_URL, result '$dout') -- this is an OPEN forward proxy on the node plane, not a snap path" + else + refuse "deny probe gave curl exit $drc with '$dout' -- neither a 403 denial nor a success; the allowlist state is UNKNOWN" + fi + fi + + if [ "$FAILED" -ne 0 ]; then echo "dc-snap-proxy check ($SITE): FAIL"; exit 1; fi + if [ "$REFUSED" -ne 0 ]; then + echo "dc-snap-proxy check ($SITE): REFUSE -- could not evaluate; this is NOT a pass"; exit 3 + fi + echo "dc-snap-proxy check ($SITE): PASS"; exit 0 +} + +# --------------------------------------------------------------------------- +# install -- idempotent; requires root. Net layer FIRST (nothing can bind the +# LISTEN address until it exists), then fail loud on the missing package, then +# the config, then a config PARSE before anything is enabled. +# --------------------------------------------------------------------------- +do_install() { + [ "$(id -u)" = "0" ] || { echo "FAIL: install requires root" >&2; exit 4; } + umask 022 + # Prereq: the libvirt utility net must EXIST (we resolve its bridge). That is a + # rack-standup artifact (dc-rack-net.sh), never a mirror or cache-proxy one, so + # this service depends on no other artifact strategy's apparatus. + br="$(util_bridge)" + if [ -z "$br" ]; then + echo "FAIL: libvirt network '${UTIL_NET}' not found -- cannot resolve its bridge" >&2 + echo " the rack net layer comes first: dc-rack-net.sh install $SITE" >&2 + exit 4 + fi + install -m 0755 /dev/null "$NET_HELPER" || exit 4 + gen_net_helper > "$NET_HELPER" || exit 4 + chmod 0755 "$NET_HELPER" || exit 4 + gen_net_unit > "$NET_UNIT" || exit 4 + mkdir -p "$(dirname "$RESOLVED_DROPIN")" || exit 4 + gen_resolved_dropin > "$RESOLVED_DROPIN" || exit 4 + systemctl daemon-reload || exit 4 + systemctl enable "${SITE}-snap-proxy-net.service" >/dev/null 2>&1 || exit 4 + systemctl restart "${SITE}-snap-proxy-net.service" \ + || { echo "FAIL: ${SITE}-snap-proxy-net.service did not apply" >&2; exit 4; } + systemctl restart systemd-resolved >/dev/null 2>&1 || true + + # Prereq 2: the package. NEVER apt-installed from here -- the repo pattern is + # fail-loud with the exact line, so package installs stay operator-visible. + if ! dpkg -s squid >/dev/null 2>&1; then + echo "FAIL: squid not installed" >&2 + echo " net layer is up -- run: apt-get update && apt-get install -y squid" >&2 + echo " then re-run: dc-snap-proxy.sh install $SITE" >&2 + exit 4 + fi + # The packaged unit is stood DOWN before ours goes up: its squid.conf pins + # http_port 3128, which MAAS's squid holds wildcard-bound, so it can only fail. + systemctl disable --now squid >/dev/null 2>&1 || true + + mkdir -p "$(dirname "$SQUID_CONF")" "$LOG_DIR" || exit 4 + gen_squid_conf > "$SQUID_CONF" || exit 4 + # PARSE BEFORE ENABLE. A config that squid rejects must fail here, loudly, and + # not as a mysterious unit that will not start. + if ! squid -k parse -f "$SQUID_CONF" >/dev/null 2>&1; then + echo "FAIL: squid rejected ${SQUID_CONF}:" >&2 + squid -k parse -f "$SQUID_CONF" >&2 + exit 4 + fi + gen_proxy_unit > "$PROXY_UNIT" || exit 4 + systemctl daemon-reload || exit 4 + systemctl enable "${SITE}-snap-proxy.service" >/dev/null 2>&1 || exit 4 + systemctl restart "${SITE}-snap-proxy.service" \ + || { echo "FAIL: ${SITE}-snap-proxy.service did not start" >&2; exit 4; } + echo "dc-snap-proxy install ($SITE): done -- running check:" + do_check +} + +# --------------------------------------------------------------------------- +# node -- run FROM a DC node OR a charm LXD container. Read-only, no root, and +# deliberately touches NOTHING rack-side: only curl, grep and /etc/environment. +# Three outcomes, never two: PASS (0), FAIL (1), REFUSE (3). +# --------------------------------------------------------------------------- +NFAILED=0; NREFUSED=0 +nok(){ echo " OK $1"; } +nno(){ echo " FAIL $1"; NFAILED=1; } +nrefuse(){ echo " REFUSE $1"; NREFUSED=1; } + +do_node() { + echo "dc-snap-proxy node check ($SITE) -- from this client, against ${LISTEN}:${PROXY_PORT}" + + if ! command -v curl >/dev/null 2>&1; then + nrefuse "curl is not present on this client -- cannot evaluate proxy reachability" + else + # (1) The client can actually CONNECT through the proxy to the store, and what + # comes back is the store's answer -- not a 200 from something else. + body="$(mktemp)"; cerr="$(mktemp)" + code="$(curl -sS -o "$body" -w '%{http_code}' -x "http://${LISTEN}:${PROXY_PORT}" \ + -m 30 -H 'Snap-Device-Series: 16' "$STORE_PROBE_URL" 2>"$cerr")"; crc=$? + if [ "$crc" -ne 0 ]; then + if grep -q 'CONNECT tunnel failed, response 403' "$cerr"; then + nno "the proxy DENIED this client (403 at CONNECT) -- this client's source address is outside ${CLIENT_CIDR}; measure it with 'ip route get ${LISTEN}' and fix the site row, do not widen the ACL blind" + else + nrefuse "could not reach $STORE_PROBE_URL THROUGH http://${LISTEN}:${PROXY_PORT} (curl exit $crc: $(head -1 "$cerr")) -- unreachable is not 'nothing there'" + fi + elif [ "$code" != "200" ]; then + nno "store API returned HTTP $code through the proxy (want 200)" + elif ! grep -q '"channel-map"' "$body"; then + nno "body served for the store API has no \"channel-map\" -- not the snap store's answer, whatever its status line said" + elif ! grep -q 'sha3-384' "$body"; then + nno "store API body carries no sha3-384 download digest -- not a usable snap info response" + else + nok "CONNECT to the store API through the proxy: curl exit 0, HTTP 200, channel-map + sha3-384 present" + fi + rm -f "$body" "$cerr" + + # (2) The allowlist must be in force FROM HERE too. Asserting it only on the + # rack would miss a client-specific allow rule, and an open proxy on the node + # plane is a bigger problem than a broken snap install. + dout="$(curl -sS -o /dev/null -w '%{http_code}' -x "http://${LISTEN}:${PROXY_PORT}" \ + -m 20 "$DENY_PROBE_URL" 2>&1)"; drc=$? + if grep -q 'CONNECT tunnel failed, response 403' <<<"$dout"; then + nok "non-allowlisted destination is DENIED from this client -- the allowlist is in force" + elif [ "$drc" -eq 0 ]; then + nno "the proxy SERVED a non-allowlisted destination from this client ($DENY_PROBE_URL, result '$dout') -- OPEN forward proxy on the node plane" + else + nrefuse "deny probe gave curl exit $drc with '$dout' -- neither a 403 denial nor a success; the allowlist state is UNKNOWN from here" + fi + fi + + # (3) snapd's own view of the proxy, WITHOUT root. juju's proxyupdater shells + # out `snap set system proxy.http/https`, and snapd's configcore writes those + # into /etc/environment (envFilePath = "/etc/environment"), which is + # world-readable -- whereas `snap get system proxy` needs root and would break + # this subcommand's no-root property. + if [ ! -r "$ENV_FILE" ]; then + nrefuse "$ENV_FILE is not readable -- cannot evaluate whether the juju snap proxy keys reached this client" + else + ENVLINE="$(grep -E '^[[:space:]]*https_proxy=' "$ENV_FILE" 2>/dev/null | tail -1)" + if [ -z "$ENVLINE" ]; then + nno "no https_proxy in $ENV_FILE -- the juju keys have not reached this client; run: juju model-config -m snap-http-proxy=http://${LISTEN}:${PROXY_PORT} snap-https-proxy=http://${LISTEN}:${PROXY_PORT}" + elif grep -q "http://${LISTEN}:${PROXY_PORT}" <<<"$ENVLINE"; then + nok "snapd's proxy is this DC's proxy ($ENV_FILE: $ENVLINE)" + else + nno "https_proxy in $ENV_FILE points somewhere ELSE: $ENVLINE -- expected http://${LISTEN}:${PROXY_PORT}" + fi + fi + + if [ "$NFAILED" -ne 0 ]; then echo "dc-snap-proxy node ($SITE): FAIL"; exit 1; fi + if [ "$NREFUSED" -ne 0 ]; then + echo "dc-snap-proxy node ($SITE): REFUSE -- could not evaluate; this is NOT a pass"; exit 3 + fi + echo "dc-snap-proxy node ($SITE): PASS"; exit 0 +} + +case "$MODE" in + check) do_check ;; + install) do_install ;; + node) do_node ;; +esac diff --git a/tests/HARNESS-MANIFEST b/tests/HARNESS-MANIFEST index 78aebbe..4248d1c 100644 --- a/tests/HARNESS-MANIFEST +++ b/tests/HARNESS-MANIFEST @@ -30,6 +30,7 @@ dc-rack-net dc-region-topology dc-selector +dc-snap-proxy juju-spaces-check keystone-policy-drift ledger-scan diff --git a/tests/dc-snap-proxy/run-tests.sh b/tests/dc-snap-proxy/run-tests.sh new file mode 100644 index 0000000..fa14afd --- /dev/null +++ b/tests/dc-snap-proxy/run-tests.sh @@ -0,0 +1,486 @@ +#!/usr/bin/env bash +# tests/dc-snap-proxy/run-tests.sh -- guard for scripts/dc-snap-proxy.sh +# (D-135/D-107 RULING 2, 2026-07-31: the per-DC snap FORWARD PROXY, squid, +# CONNECT-only, destination-restricted). +# +# TWO HALVES, DELIBERATELY. +# STATIC (T1-T22): the identity values a silent edit would break -- both sites' +# utility .4, ruled edge gateways and MEASURED client CIDRs, the port that +# is NOT MAAS's 3128, stable-identity keying, service-owned net artifacts, +# the consumption keys, the store/CDN allowlist, and arg/site validation. +# BEHAVIORAL (T23-T49): `check` and `node` are DRIVEN, against a mktemp +# SYSROOT and fake curl/ss/systemctl/virsh/ip/dpkg on PATH. A grep cannot +# tell a gate that fails from one that merely cannot fail, and this script's +# whole point is being a GA-R6 gate: every FAIL path and every REFUSE path +# below has its own failing fixture. +# +# WHAT THE GREEN BELOW DOES *NOT* PROVE (say it plainly, GA-R1): no proxy has +# ever been installed, so no snap has been installed through one. This is a +# FIXTURE green. The real gates are `check ` on the rack and `node ` +# from a client, at apply time. +# +# Exit: 0 all pass | 1 any case failed. ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SCRIPT="$(cd "$HERE/../.." && pwd)/scripts/dc-snap-proxy.sh" +PASS=0; FAIL=0 +ok(){ echo " PASS $1"; PASS=$((PASS+1)); } +no(){ echo " FAIL $1"; FAIL=$((FAIL+1)); } + +[ -f "$SCRIPT" ] && ok "T1 script present" || { no "T1 script present"; echo; echo "dc-snap-proxy: $PASS passed, $FAIL failed"; exit 1; } +bash -n "$SCRIPT" 2>/dev/null && ok "T2 bash -n clean" || no "T2 bash -n clean" + +# =========================================================================== +# STATIC -- identity values and structural invariants. +# =========================================================================== + +# T3-T5: dc0 row. The utility .4 is the address the deploy will be pointed at; +# the edge gw is the proxy's only upstream egress; CLIENT_CIDR is the MEASURED +# source the charm CONTAINERS use toward .4 (metal-admin, NOT the metal-internal +# address `juju status` prints -- that misreading would deny every client). +D0="$(awk '/^ dc0\)/{f=1} f{print} f&&/^ ;;/{exit}' "$SCRIPT")" +grep -q 'LISTEN_CIDR="10.12.8.4/22"' <<<"$D0" \ + && grep -q 'UTIL_NET="vr1-dc0-metal-admin"' <<<"$D0" \ + && ok "T3 dc0 LISTEN 10.12.8.4/22 on metal-admin (D-134 utility .4)" \ + || no "T3 dc0 LISTEN 10.12.8.4/22 on metal-admin (D-134 utility .4)" +grep -q 'EDGE_GW="10.12.4.1"' <<<"$D0" \ + && ok "T4 dc0 edge gw 10.12.4.1" || no "T4 dc0 edge gw 10.12.4.1" +grep -q 'CLIENT_CIDR="10.12.8.0/22"' <<<"$D0" \ + && ok "T5 dc0 client ACL is metal-admin 10.12.8.0/22 (measured container source)" \ + || no "T5 dc0 client ACL is metal-admin 10.12.8.0/22 (measured container source)" + +# T6-T8: dc1 row -- values from lib-net.sh's own vr1-dc1 arm, never mirrored +# from dc0 by symmetry (the lib-hosts.sh scar). +D1="$(awk '/^ dc1\)/{f=1} f{print} f&&/^ ;;/{exit}' "$SCRIPT")" +grep -q 'LISTEN_CIDR="10.12.68.4/22"' <<<"$D1" \ + && grep -q 'UTIL_NET="vr1-dc1-metal-admin"' <<<"$D1" \ + && ok "T6 dc1 LISTEN 10.12.68.4/22 on metal-admin (utility .4)" \ + || no "T6 dc1 LISTEN 10.12.68.4/22 on metal-admin (utility .4)" +grep -q 'EDGE_GW="10.12.64.1"' <<<"$D1" \ + && ok "T7 dc1 edge gw 10.12.64.1" || no "T7 dc1 edge gw 10.12.64.1" +grep -q 'CLIENT_CIDR="10.12.68.0/22"' <<<"$D1" \ + && ok "T8 dc1 client ACL is metal-admin 10.12.68.0/22" \ + || no "T8 dc1 client ACL is metal-admin 10.12.68.0/22" + +# T9: the port. 3128 AND 8000 are MAAS's squid, WILDCARD-bound on BOTH racks +# (measured); 3142 is dc1's apt-cacher-ng; 80 is dc0's mirror nginx. A silent +# move to 3128 would produce a squid that cannot bind, on both DCs. +grep -q 'PROXY_PORT="3129"' "$SCRIPT" \ + && ! grep -qE 'PROXY_PORT="(3128|8000|3142|80)"' "$SCRIPT" \ + && ok "T9 proxy port pinned 3129 (3128/8000 are MAAS's squid, 3142 is acng)" \ + || no "T9 proxy port pinned 3129 (3128/8000 are MAAS's squid, 3142 is acng)" + +# T10: stable-identity keying -- bridge from the libvirt net NAME, no virbrN. +# Scoped to NON-COMMENT lines on purpose: the site rows quote the MEASURED +# `ip -4 -o addr` and `ip route show default` output, which names virbr2/virbr5/ +# virbr6, and deleting that evidence to satisfy a grep would destroy the +# provenance the values rest on. What must not exist is an EXECUTED literal. +CODE="$(grep -vE '^[[:space:]]*#' "$SCRIPT")" +grep -q 'net-dumpxml' <<<"$CODE" && ! grep -qE 'virbr[0-9]' <<<"$CODE" \ + && ok "T10 bridge resolved from net name; no EXECUTED virbrN literal" \ + || no "T10 bridge resolved from net name; no EXECUTED virbrN literal" + +# T11: this service OWNS its net layer and NAMES it for itself. The D-135 +# lesson: on a DC running only the mirror (dc0) or only the cache proxy (dc1), +# that one's documented teardown must not be able to kill this proxy. +grep -q 'snap-proxy-net.service' "$SCRIPT" \ + && grep -q 'snap-proxy-net-apply' "$SCRIPT" \ + && ! grep -qE '(NET_UNIT|NET_HELPER|RESOLVED_DROPIN)=.*(mirror|cache-proxy)' "$SCRIPT" \ + && grep -qE 'ip addr replace' "$SCRIPT" && grep -qE 'ip route replace default' "$SCRIPT" \ + && ok "T11 net layer OWNED here and named for THIS service" \ + || no "T11 net layer OWNED here and named for THIS service" + +# T11b: no OPERATIVE referral to the other two services' installers. Scoped to +# NON-COMMENT lines: the header's account of WHY the coupling is forbidden cites +# them, and deleting that history to satisfy a grep would lose the reason. +grep -vE '^[[:space:]]*#' "$SCRIPT" | grep -qE '(dc-mirror|dc-cache-proxy)\.sh install' \ + && no "T11b no OPERATIVE dc-mirror/dc-cache-proxy referral (header history exempt)" \ + || ok "T11b no OPERATIVE dc-mirror/dc-cache-proxy referral (header history exempt)" + +# T12: idempotent verbs only, so this net unit and the mirror's / cache proxy's +# coexist on a rack running two of them instead of fighting. +! grep -qE 'ip (addr|route) add ' "$SCRIPT" \ + && ok "T12 net writes use idempotent replace (safe to coexist)" \ + || no "T12 net writes use idempotent replace (safe to coexist)" + +# T13: MEASURED/RULED tag discipline on every site row (hard rule 2). +MR="$(grep -cE 'MEASURED/RULED' "$SCRIPT")" +[ "$MR" -ge 2 ] && ok "T13 both site rows carry a MEASURED/RULED tag" \ + || no "T13 both site rows carry a MEASURED/RULED tag (found $MR)" + +# T14: install requires root. +grep -q 'install requires root' "$SCRIPT" && ok "T14 install requires root" \ + || no "T14 install requires root" + +# T15: consumption documents BOTH snap keys, and warns that the store-proxy trio +# is a DIFFERENT product (snapd rejects proxy.store without a signed assertion). +grep -q 'snap-http-proxy=http://' "$SCRIPT" \ + && grep -q 'snap-https-proxy=http://' "$SCRIPT" \ + && grep -qE 'snap-store-proxy.*EMPTY|EMPTY.*snap-store' "$SCRIPT" \ + && ok "T15 documents both snap keys + leaves the store-proxy trio empty" \ + || no "T15 documents both snap keys + leaves the store-proxy trio empty" + +# T16: the destination allowlist must carry the API host AND a CDN wildcard. +# snapd uses ONE http client for the API and the DOWNLOAD, so an allowlist with +# only api.snapcraft.io authenticates and then fails every download. +# Scoped to the STORE_DOMAINS VALUE, not to the file: the header discusses every +# one of these hosts in prose, so a file-wide grep would stay green after the +# allowlist itself was narrowed. That near-miss is the "citation is an existence +# claim; only its content is evidence" class. +SD="$(grep -E '^STORE_DOMAINS=' "$SCRIPT")" +grep -q 'api\.snapcraft\.io' <<<"$SD" \ + && grep -q '\.snapcraftcontent\.com' <<<"$SD" \ + && grep -q 'login\.ubuntu\.com' <<<"$SD" \ + && ok "T16 allowlist VALUE carries the store API, login and the CDN wildcard" \ + || no "T16 allowlist VALUE carries the store API, login and the CDN wildcard" + +# T17: the generated squid config binds the LISTEN address SPECIFICALLY. A +# wildcard bind would collide with MAAS's squid and would also expose the proxy +# on the transit and provider legs. +grep -q 'http_port ${LISTEN}:${PROXY_PORT}' "$SCRIPT" \ + && ok "T17 squid binds LISTEN:PORT specifically, not wildcard" \ + || no "T17 squid binds LISTEN:PORT specifically, not wildcard" + +# T18: the PACKAGED squid.service is stood down at install AND asserted by check. +# Its squid.conf pins http_port 3128 (MAAS holds it), so it can only crash-loop. +# The install half is scoped to do_install's BODY: `check`'s failure message also +# tells the operator to run that command, so a file-wide grep would stay green +# after install stopped doing it. +INSTALL_BODY="$(awk '/^do_install\(\) \{/{f=1} f{print} f&&/^\}/{exit}' "$SCRIPT")" +grep -q 'systemctl disable --now squid' <<<"$INSTALL_BODY" \ + && grep -q 'PACKAGED squid.service is ACTIVE' "$SCRIPT" \ + && ok "T18 packaged squid.service disabled BY install + asserted by check" \ + || no "T18 packaged squid.service disabled BY install + asserted by check" + +# T19: install PARSES the config before enabling anything. +grep -q 'squid -k parse -f' "$SCRIPT" \ + && ok "T19 install parses the squid config before enabling" \ + || no "T19 install parses the squid config before enabling" + +# T20: install never apt-installs -- it fails loud with the exact line instead, +# so a package install stays operator-visible (the dc-mirror/dc-cache-proxy +# pattern). The first grep must match an INVOCATION (a command at the start of a +# statement), never the guidance string, which contains the same words. +# Double-quoted strings are blanked first, for the same reason as T50: the +# guidance message legitimately CONTAINS `apt-get update && apt-get install -y +# squid`, and a naive invocation regex matches inside it. +if grep -vE '^[[:space:]]*#' "$SCRIPT" | sed 's/"[^"]*"/""/g' \ + | grep -qE '(^|;|&&|\|\|)[[:space:]]*(sudo[[:space:]]+)?apt(-get)?[[:space:]]+install'; then + no "T20 install never apt-installs; it prints the line for the operator" +elif grep -q 'run: apt-get update && apt-get install -y squid' "$SCRIPT"; then + ok "T20 install never apt-installs; it prints the line for the operator" +else + no "T20 install never apt-installs; it prints the line for the operator" +fi + +# T21-T22: arg / site validation. +out="$(bash "$SCRIPT" bogus dc0 2>&1)"; rc=$? +[ "$rc" = "2" ] && grep -q 'usage:' <<<"$out" \ + && ok "T21 bad mode -> usage, exit 2" || no "T21 bad mode -> usage, exit 2" +bash "$SCRIPT" check zz >/dev/null 2>&1; [ "$?" = "2" ] \ + && ok "T22 unknown site -> exit 2" || no "T22 unknown site -> exit 2" + +# =========================================================================== +# BEHAVIORAL -- drive `check dc0` and `node dc0` against a fake world. +# =========================================================================== +TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT +FB="$TMP/bin"; mkdir -p "$FB" +SR="$TMP/sysroot" + +cat > "$FB/curl" <<'FAKE' +#!/usr/bin/env bash +# fake curl: honours -o , writes %{http_code} to stdout and any error text +# to stderr, and exits like the real one. Behaviour is selected per TARGET URL, +# because check makes three probes with three different meanings. +out="/dev/null"; url="" +while [ $# -gt 0 ]; do + case "$1" in + -o) out="$2"; shift 2 ;; + -w|-x|-m|-H) shift 2 ;; + -*) shift ;; + *) url="$1"; shift ;; + esac +done +store_body() { # $1 = which fields to include + case "$1" in + full) printf '{"channel-map":[{"download":{"sha3-384":"deadbeef","size":1}}]}\n' > "$out" ;; + nochan) printf '{"result":"error","message":"nope"}\n' > "$out" ;; + nosha) printf '{"channel-map":[{"download":{"size":1}}]}\n' > "$out" ;; + esac +} +deny_403() { echo "curl: (56) CONNECT tunnel failed, response 403" >&2; printf '000'; exit 56; } +case "$url" in + *api.snapcraft.io*) + case "$(cat "$FAKEDIR/curl.store")" in + ok) store_body full; printf '200' ;; + nochan) store_body nochan; printf '200' ;; + nosha) store_body nosha; printf '200' ;; + http404) printf 'not found\n' > "$out"; printf '404' ;; + unreach) echo "curl: (7) Failed to connect to 10.12.8.4 port 3129" >&2; printf '000'; exit 7 ;; + denied) deny_403 ;; + esac ;; + *snapcraftcontent.com*) + case "$(cat "$FAKEDIR/curl.cdn")" in + ok) printf '403' ;; + refused) deny_403 ;; + weird) echo "curl: (35) TLS connect error" >&2; printf '000'; exit 35 ;; + esac ;; + *) + case "$(cat "$FAKEDIR/curl.deny")" in + denied) deny_403 ;; + served) printf '200' ;; + weird) echo "curl: (28) Operation timed out" >&2; printf '000'; exit 28 ;; + esac ;; +esac +FAKE + +cat > "$FB/ss" <<'FAKE' +#!/usr/bin/env bash +case "$(cat "$FAKEDIR/ss.mode")" in + listen) printf 'State Recv-Q Send-Q Local Address:Port Peer Address:Port\nLISTEN 0 256 10.12.8.4:3129 0.0.0.0:* \n' ;; + wildcard) printf 'State Recv-Q Send-Q Local Address:Port Peer Address:Port\nLISTEN 0 256 *:3129 0.0.0.0:* \n' ;; + none) printf 'State Recv-Q Send-Q Local Address:Port Peer Address:Port\n' ;; +esac +FAKE + +cat > "$FB/systemctl" <<'FAKE' +#!/usr/bin/env bash +# modes: units.mode = enabled|disabled (our two units) +# pkgsquid.mode = off|active|enabled +verb="$1"; shift +quiet=0; [ "${1:-}" = "--quiet" ] && { quiet=1; shift; } +unit="${1:-}" +u="$(cat "$FAKEDIR/units.mode")"; p="$(cat "$FAKEDIR/pkgsquid.mode")" +case "$unit" in + squid|squid.service) + case "$verb" in + is-enabled) [ "$p" = "enabled" ] && { echo enabled; exit 0; }; echo disabled; exit 1 ;; + is-active) [ "$p" = "active" ] && exit 0 || exit 3 ;; + esac ;; + *snap-proxy.service|*snap-proxy-net.service) + case "$verb" in + is-enabled) [ "$u" = "enabled" ] && { echo enabled; exit 0; }; echo disabled; exit 1 ;; + is-active) [ "$u" = "enabled" ] && exit 0 || exit 3 ;; + esac ;; +esac +exit 0 +FAKE + +cat > "$FB/virsh" <<'FAKE' +#!/usr/bin/env bash +case "$(cat "$FAKEDIR/virsh.mode")" in + ok) printf "\n \n\n" ;; + missing) exit 1 ;; +esac +FAKE + +cat > "$FB/ip" <<'FAKE' +#!/usr/bin/env bash +args="$*" +case "$args" in + *"addr show dev"*) + case "$(cat "$FAKEDIR/ip.addr")" in + ok) printf '7: virbrTEST inet 10.12.8.4/22 brd 10.12.11.255 scope global secondary virbrTEST\n' ;; + missing) : ;; + esac ;; + *"route show default"*) + case "$(cat "$FAKEDIR/ip.route")" in + ok) printf 'default via 10.12.4.1 dev virbrX proto static\n' ;; + missing) : ;; + esac ;; +esac +FAKE + +cat > "$FB/dpkg" <<'FAKE' +#!/usr/bin/env bash +[ "$(cat "$FAKEDIR/dpkg.mode")" = "installed" ] && exit 0 || exit 1 +FAKE +chmod +x "$FB"/* + +# Lay down a SYSROOT whose five artifacts are byte-identical to what the script +# generates -- by asking THE SCRIPT ITSELF to render them, never by copying its +# text into this file. A hand-copied expectation drifts silently; this cannot. +# Everything above the final MODE dispatch is the script's library half: sourcing +# it with `set -- check dc0` yields the real site row and the real generators. +LIB="$TMP/lib.sh" +LASTCASE="$(grep -n '^case "\$MODE" in$' "$SCRIPT" | tail -1 | cut -d: -f1)" +if [ -z "$LASTCASE" ]; then + no "T-lib could not locate the MODE dispatch -- behavioral cases cannot run" + echo; echo "dc-snap-proxy: $PASS passed, $FAIL failed"; exit 1 +fi +head -n "$((LASTCASE-1))" "$SCRIPT" > "$LIB" +render() { # render + env DC_SNAP_PROXY_SYSROOT="$SR" LIB="$LIB" GENFN="$1" bash -c ' + set -- check dc0 + . "$LIB" + "$GENFN" + ' _ +} +seed_sysroot() { + rm -rf "$SR" + mkdir -p "$SR/etc/squid" "$SR/etc/systemd/system" "$SR/etc/systemd/resolved.conf.d" \ + "$SR/usr/local/sbin" "$SR/etc" + render gen_squid_conf > "$SR/etc/squid/dc0-snap-proxy.conf" + render gen_proxy_unit > "$SR/etc/systemd/system/dc0-snap-proxy.service" + render gen_net_helper > "$SR/usr/local/sbin/dc0-snap-proxy-net-apply" + render gen_net_unit > "$SR/etc/systemd/system/dc0-snap-proxy-net.service" + render gen_resolved_dropin > "$SR/etc/systemd/resolved.conf.d/dc0-snap-proxy.conf" + printf 'PATH="/usr/bin:/bin"\nhttps_proxy=http://10.12.8.4:3129\nhttp_proxy=http://10.12.8.4:3129\n' \ + > "$SR/etc/environment" +} +set_modes() { # all-green defaults + printf 'ok' > "$TMP/curl.store" + printf 'ok' > "$TMP/curl.cdn" + printf 'denied' > "$TMP/curl.deny" + printf 'listen' > "$TMP/ss.mode" + printf 'enabled' > "$TMP/units.mode" + printf 'off' > "$TMP/pkgsquid.mode" + printf 'ok' > "$TMP/virsh.mode" + printf 'ok' > "$TMP/ip.addr" + printf 'ok' > "$TMP/ip.route" + printf 'installed' > "$TMP/dpkg.mode" +} +run_mode() { # run_mode