dc0 SNAP FORWARD PROXY -- FIRST LIVE INSTALL + VERIFICATION CAPTURE
Date:     2026-07-31
Site:     dc0 (vr1-dc0), rack host vr1-dc0-rack = 172.31.0.2 (via voffice1)
Tool:     scripts/dc-snap-proxy.sh  (D-135/D-107 RULING 2, 2026-07-31)
Operator: jessea123 @ jumphost vcloud, branch dc-dc-stage5-preconditions

PURPOSE. The tool had NEVER been applied. Its harness green was FIXTURE green.
This capture is the first end-to-end application on a real rack, and it is the
run that resolves the BUG-3 open question (section 6).

All output below is VERBATIM. Nothing is paraphrased.

===========================================================================
SECTION 0 -- BASE CONFIRMATION (jumphost, before touching the rack)
===========================================================================

$ bash tests/dc-snap-proxy/run-tests.sh
  [...]
  PASS  T50 node body is client-satisfiable only (no rack-side calls)

dc-snap-proxy: 55 passed, 0 failed

$ bash scripts/repo-lint.sh
  [WARN] L1 docs/design-decisions.md: 239 non-ASCII byte(s) (legacy D-001..018 carve-out; NEW entries must be ASCII)

WARN: repo lint (0 fail, 1 warn, 641 files scanned)

Both match the expected base (55/55; 0 fail, 1 warn).

$ sha256sum scripts/dc-snap-proxy.sh tests/dc-snap-proxy/run-tests.sh
318cfb4190eccdfe1d760ac3a7f8f1a23ac2b35e78d07167d730bef7150439f0  scripts/dc-snap-proxy.sh
aaa2a2060e770720f25284f9413b8c222e2d3b073a6f5ab2c6daad642fc0f8ff  tests/dc-snap-proxy/run-tests.sh

Recorded as the pre-edit baseline so any restore-verify is measured, not
reconstructed. Both files were already MODIFIED-UNCOMMITTED in the working tree
on arrival (the earlier BUG-1/BUG-3/BUG-4 fix work); HEAD is 2208141.

TRANSIT VALUE PROVENANCE (hard rule 2 -- not from memory):
  scripts/lib-hosts.sh, vr1-dc0 arm:
    VIRSH_POWER_ADDRESS_FROM_OFFICE1="qemu+ssh://jessea123@172.31.0.2/system"
  ~/.ssh/config.d/vr1-sites:
    Host vr1-dc0-rack 172.31.0.2    # dc0 rack host (libvirt + MAAS rack + D-135 mirror)
      HostName 172.31.0.2
      User jessea123
      ProxyJump voffice1
  The two agree. 172.31.0.2 is used below only via the alias.

===========================================================================
SECTION 1 -- BEFORE-STATE: read-only `check dc0` with NOTHING built
===========================================================================

This proves the gate can FAIL on a real rack and not merely on a fixture.

$ ssh vr1-dc0-rack 'sudo bash -s' -- check dc0 < scripts/dc-snap-proxy.sh; echo "EXIT=$?"
dc-snap-proxy check (dc0) -- 10.12.8.4:3129, clients 10.12.8.0/22
  MISS    squid binary absent at /usr/sbin/squid -- run: apt-get update && apt-get install -y squid
  MISS    /etc/squid/dc0-snap-proxy.conf absent
  MISS    /etc/systemd/system/dc0-snap-proxy.service absent
  MISS    /usr/local/sbin/dc0-snap-proxy-net-apply absent
  MISS    /etc/systemd/system/dc0-snap-proxy-net.service absent
  MISS    /etc/systemd/resolved.conf.d/dc0-snap-proxy.conf absent
  MISS    dc0-snap-proxy.service not enabled (it would not survive a reboot)
  MISS    dc0-snap-proxy-net.service not enabled (it would not survive a reboot)
  MISS    dc0-snap-proxy.service not active
  OK      packaged squid.service neither active nor enabled
  MISS    nothing listening on 10.12.8.4:3129
  OK      10.12.8.4/22 on virbr2 (vr1-dc0-metal-admin)
  OK      default route via edge 10.12.4.1
  REFUSE  could not reach https://api.snapcraft.io/v2/snaps/info/core THROUGH http://10.12.8.4:3129 (curl exit 7: curl: (7) Failed to connect to 10.12.8.4 port 3129 after 0 ms: Couldn't connect to server) -- unreachable is not 'nothing there'
  REFUSE  CDN probe gave curl exit 7 with 'curl: (7) Failed to connect to 10.12.8.4 port 3129 after 0 ms: Couldn't connect to server
000' -- neither an allowed tunnel nor a proxy denial; refusing to interpret it
  REFUSE  deny probe gave curl exit 7 with 'curl: (7) Failed to connect to 10.12.8.4 port 3129 after 0 ms: Couldn't connect to server
000' -- neither a 403 denial nor a success; the allowlist state is UNKNOWN
dc-snap-proxy check (dc0): FAIL
EXIT=1

WHY THIS IS NOT THE "WRONG HOST" TRAP (the recorded per-DC-script failure mode
where running from the jumphost measures the wrong filesystem and reports
uniform total failure). The result is NOT uniform. Three assertions came back
OK, and they are exactly the ones only the real rack can satisfy:
  - "10.12.8.4/22 on virbr2 (vr1-dc0-metal-admin)" -- requires virsh to resolve
    the libvirt network NAME to a bridge; on the jumphost this would REFUSE.
  - "default route via edge 10.12.4.1" -- the rack's route table, not ours.
  - "packaged squid.service neither active nor enabled".
An implausibly uniform result would itself have been the tell. It was not
uniform, so the instrument was current and pointed at the right host.

===========================================================================
SECTION 2 -- PRE-MUTATION STATE OF THE NET LAYER AND THE CLAIMED PORT
===========================================================================

Taken BEFORE install so the header's "the three net units set IDENTICAL
MEASURED values, so they are no-ops for each other" claim becomes testable
rather than merely asserted.

$ ssh vr1-dc0-rack 'ip -4 -o addr show dev virbr2; ip route show default; systemctl is-active dc0-mirror-net.service dc0-cache-proxy-net.service; sudo ss -ltnp | grep -E ":(3128|3129|8000) "; ls -l /usr/sbin/squid'
=== ip -4 -o addr show dev virbr2 ===
7: virbr2    inet 10.12.8.2/22 scope global virbr2\       valid_lft forever preferred_lft forever
7: virbr2    inet 10.12.8.3/22 scope global secondary virbr2\       valid_lft forever preferred_lft forever
7: virbr2    inet 10.12.8.4/22 scope global secondary virbr2\       valid_lft forever preferred_lft forever
=== ip route show default ===
default via 10.12.4.1 dev virbr5
=== mirror/cache net units ===
active
inactive
=== ss -ltnp ports 3128/3129/8000 ===
LISTEN 0      256                *:8000            *:*    users:(("squid",pid=1871633,fd=13))
LISTEN 0      256                *:3128            *:*    users:(("squid",pid=1871633,fd=12))
=== squid binary ===
ls: cannot access '/usr/sbin/squid': No such file or directory

READS:
  - The utility .4 (10.12.8.4/22) was ALREADY present on virbr2 as a secondary,
    placed by dc0-mirror-net.service (active). dc0-cache-proxy-net.service is
    inactive on this rack.
  - MAAS's squid holds *:3128 AND *:8000 WILDCARD-bound, both fds of ONE process
    (pid 1871633). Confirms the header's measured claim on this rack, today.
  - PORT 3129 IS FREE -- nothing listening. The port this build claims is
    unclaimed, verified before mutating.
  - squid binary genuinely absent (so section 1's MISS was an OBSERVED absence).

===========================================================================
SECTION 3 -- THE PACKAGE (installed by hand; `install` deliberately never does)
===========================================================================

$ ssh vr1-dc0-rack 'sudo apt-get update && sudo apt-get install -y squid'
[...]
Restarting services...

Service restarts being deferred:
 /etc/needrestart/restart.d/dbus.service
 systemctl restart getty@tty1.service
 systemctl restart libvirtd.service
 systemctl restart systemd-logind.service
 systemctl restart unattended-upgrades.service
 systemctl restart virtlockd.service
 systemctl restart virtlogd.service

No containers need to be restarted.

VM guests are running outdated hypervisor (qemu) binaries on this host:
 'vr1-dc0-opnsense' with pid 44740
Pending kernel upgrade
----------------------

Newer kernel available

The currently running kernel version is 6.8.0-134-generic which is not the
expected kernel version 6.8.0-136-generic.

Restarting the system to load the new kernel will not be handled automatically,
so you should consider rebooting.

VERSION ACTUALLY INSTALLED:

$ ssh vr1-dc0-rack 'dpkg-query -W -f="${Version}\n" squid; /usr/sbin/squid -v | head -1; ls -l /usr/sbin/squid'
=== dpkg-query ===
6.14-0ubuntu0.24.04.4
=== squid -v ===
Squid Cache: Version 6.14
=== binary ===
-rwxr-xr-x 1 root root 6989912 Jun 13 14:31 /usr/sbin/squid

This is EXACTLY the version the script header names as noble MAIN and
security-supported ("6.14-0ubuntu0.24.04.4"). The header's package claim is now
MEASURED on this rack, not just cited.

--- FINDING 3a: the packaged unit's FATAL was OBSERVED, not just predicted ----

The header argues from reasoning that the packaged squid.service "would FATAL on
bind" because Ubuntu noble pins http_port 3128 and MAAS's squid holds it
wildcard-bound. Installing the package ran that experiment for real:

$ ssh vr1-dc0-rack 'systemctl is-active squid; systemctl is-enabled squid'
failed
enabled

$ ssh vr1-dc0-rack 'sudo journalctl -u squid --no-pager -n 20'
Jul 31 23:01:31 vvr1-dc0 squid[2914603]: storeDirWriteCleanLogs: Starting...
                                             listening port: 3128
Jul 31 23:01:31 vvr1-dc0 squid[2914603]:   Finished.  Wrote 0 entries.
                                             listening port: 3128
Jul 31 23:01:31 vvr1-dc0 squid[2914603]:   Took 0.00 seconds (  0.00 entries/sec).
                                             listening port: 3128
Jul 31 23:01:31 vvr1-dc0 squid[2914603]: FATAL: Unable to open HTTP Socket
                                             listening port: 3128
Jul 31 23:01:31 vvr1-dc0 squid[2914603]: Squid Cache (Version 6.14): Terminated abnormally.
                                             listening port: 3128
Jul 31 23:01:31 vvr1-dc0 squid[2914603]: Closing Pinger socket on FD 14
                                             listening port: 3128
Jul 31 23:01:31 vvr1-dc0 squid[2914584]: Squid Parent: squid-1 process 2914603 exited with status 1
Jul 31 23:01:31 vvr1-dc0 squid[2914584]: Squid Parent: squid-1 process 2914603 will not be restarted for 3600 seconds due to repeated, frequent failures
Jul 31 23:01:31 vvr1-dc0 squid[2914584]: Exiting due to repeated, frequent failures
Jul 31 23:01:31 vvr1-dc0 squid[2914584]: Removing PID file (/run/squid.pid)
Jul 31 23:01:31 vvr1-dc0 systemd[1]: squid.service: Main process exited, code=exited, status=1/FAILURE
Jul 31 23:01:31 vvr1-dc0 systemd[1]: squid.service: Killing process 2914605 (pinger) with signal SIGKILL.
Jul 31 23:01:31 vvr1-dc0 systemd[1]: squid.service: Failed with result 'exit-code'.
Jul 31 23:01:31 vvr1-dc0 systemd[1]: Failed to start squid.service - Squid Web Proxy Server.

"FATAL: Unable to open HTTP Socket / listening port: 3128" is the header's
predicted failure, observed verbatim. The dedicated-instance design decision is
therefore MEASURED-CORRECT on this rack and not merely well-argued. It also
shows the window is real: between apt install and `install dc0`, the rack DID
carry a failed, still-ENABLED squid.service that would fight for 3128 at the
next boot -- which is precisely what `install`'s `systemctl disable --now squid`
and check's packaged-unit assertion exist to close.

===========================================================================
SECTION 4 -- `install dc0` (the first ever application of this tool)
===========================================================================

$ ssh vr1-dc0-rack 'sudo bash -s' -- install dc0 < scripts/dc-snap-proxy.sh; echo "EXIT=$?"
dc-snap-proxy install (dc0): done -- running check:
dc-snap-proxy check (dc0) -- 10.12.8.4:3129, clients 10.12.8.0/22
  OK      squid binary present (/usr/sbin/squid)
  OK      /etc/squid/dc0-snap-proxy.conf matches generated
  OK      /etc/systemd/system/dc0-snap-proxy.service matches generated
  OK      /usr/local/sbin/dc0-snap-proxy-net-apply matches generated
  OK      /etc/systemd/system/dc0-snap-proxy-net.service matches generated
  OK      /etc/systemd/resolved.conf.d/dc0-snap-proxy.conf matches generated
  OK      dc0-snap-proxy.service enabled
  OK      dc0-snap-proxy-net.service enabled
  OK      dc0-snap-proxy.service active
  OK      packaged squid.service neither active nor enabled
  OK      listening on 10.12.8.4:3129
  OK      10.12.8.4/22 on virbr2 (vr1-dc0-metal-admin)
  OK      default route via edge 10.12.4.1
  OK      CONNECT to the store API: curl exit 0, HTTP 200, channel-map + sha3-384 present
  OK      CDN tunnel allowed (curl exit 0; the CDN's own status was '403')
  OK      non-allowlisted destination is DENIED (403 at CONNECT) -- the allowlist is in force
dc-snap-proxy check (dc0): PASS
EXIT=0

Install succeeded on the first attempt with no intervention, and its trailing
self-check PASSED. Note the packaged-unit line flipped from the section-3
"failed + enabled" state to "neither active nor enabled" -- install's
`systemctl disable --now squid` did its job, observed rather than assumed.

--- POST-INSTALL NET-LAYER COMPARISON (the header's no-op claim, tested) ------

The header claims the three per-DC net units "set IDENTICAL MEASURED values with
`ip addr replace` / `ip route replace`, so they are no-ops for each other". dc0
already ran dc0-mirror-net.service, which owns the same .4 alias and the same
default route, so `install` was a live test of that claim.

$ ssh vr1-dc0-rack 'ip -4 -o addr show dev virbr2; ip route show default; systemctl is-active dc0-mirror-net.service dc0-snap-proxy-net.service; sudo ss -ltnp | grep -E ":(3128|3129|8000) "'
=== POST-INSTALL ip -4 -o addr show dev virbr2 ===
7: virbr2    inet 10.12.8.2/22 scope global virbr2\       valid_lft forever preferred_lft forever
7: virbr2    inet 10.12.8.3/22 scope global secondary virbr2\       valid_lft forever preferred_lft forever
7: virbr2    inet 10.12.8.4/22 scope global secondary virbr2\       valid_lft forever preferred_lft forever
=== POST-INSTALL ip route show default ===
default via 10.12.4.1 dev virbr5
=== net units ===
active
active
=== ports ===
LISTEN 0      256        10.12.8.4:3129      0.0.0.0:*    users:(("squid",pid=2916382,fd=10))
LISTEN 0      256                *:8000            *:*    users:(("squid",pid=1871633,fd=13))
LISTEN 0      256                *:3128            *:*    users:(("squid",pid=1871633,fd=12))

RESULT -- the header's claim HOLDS, measured:
  - virbr2's address list is BYTE-IDENTICAL to section 2. No address added,
    removed or reordered. The .4 alias was already there and stayed there.
  - The default route is unchanged (`via 10.12.4.1 dev virbr5`).
  - dc0-mirror-net.service is STILL active alongside dc0-snap-proxy-net.service.
    The two coexist; neither displaced the other.
  - MAAS's squid is STILL pid 1871633 -- the SAME pid as before install. It was
    not restarted, not disturbed, and still holds *:3128 and *:8000. The
    coexistence claim is measured, not reasoned.
  - Our squid is a SEPARATE process (pid 2916382) bound to 10.12.8.4:3129
    SPECIFICALLY, not wildcard, exactly as generated.

===========================================================================
SECTION 5 -- standalone `check dc0` (a separate invocation, not install's)
===========================================================================

$ ssh vr1-dc0-rack 'sudo bash -s' -- check dc0 < scripts/dc-snap-proxy.sh; echo "EXIT=$?"
dc-snap-proxy check (dc0) -- 10.12.8.4:3129, clients 10.12.8.0/22
  OK      squid binary present (/usr/sbin/squid)
  OK      /etc/squid/dc0-snap-proxy.conf matches generated
  OK      /etc/systemd/system/dc0-snap-proxy.service matches generated
  OK      /usr/local/sbin/dc0-snap-proxy-net-apply matches generated
  OK      /etc/systemd/system/dc0-snap-proxy-net.service matches generated
  OK      /etc/systemd/resolved.conf.d/dc0-snap-proxy.conf matches generated
  OK      dc0-snap-proxy.service enabled
  OK      dc0-snap-proxy-net.service enabled
  OK      dc0-snap-proxy.service active
  OK      packaged squid.service neither active nor enabled
  OK      listening on 10.12.8.4:3129
  OK      10.12.8.4/22 on virbr2 (vr1-dc0-metal-admin)
  OK      default route via edge 10.12.4.1
  OK      CONNECT to the store API: curl exit 0, HTTP 200, channel-map + sha3-384 present
  OK      CDN tunnel allowed (curl exit 0; the CDN's own status was '403')
  OK      non-allowlisted destination is DENIED (403 at CONNECT) -- the allowlist is in force
dc-snap-proxy check (dc0): PASS
EXIT=0

PASS, exit 0. This gate has now been observed to do BOTH things GA-R6 requires:
it FAILED on a real rack with nothing built (section 1, exit 1) and it PASSED on
a real rack with everything built (here, exit 0). It is no longer fixture-only.

===========================================================================
SECTION 6 -- BUG-3 RESOLVED: the dstdomain deny shape, MEASURED
===========================================================================

THE OPEN QUESTION (script header, lines 110-120). check's DENY probe asserts the
literal 'CONNECT tunnel failed, response 403'. That shape was measured from a
PORT deny (`deny CONNECT !SSL_ports`) and had NEVER been measured from the
DSTDOMAIN deny (`deny !snap_store`) that the probe actually exercises. If squid
answered a dstdomain denial differently, the branch would fall to REFUSE and
`check` could never PASS -- which under GA-R6 blocks a stage close as hard as a
gate that cannot fail.

The say-line in sections 4 and 5 already indicates the expected branch was hit,
but a say-line is the script's interpretation, not the measurement. So the probe
was ALSO run BY HAND, with stderr unmerged and then merged, to record the raw
shape:

$ ssh vr1-dc0-rack 'curl -sS -o /dev/null -w "HTTPCODE=%{http_code}\n" -x http://10.12.8.4:3129 -m 20 https://example.com/ ; echo "RC=$?"'
=== deny probe, stderr UNMERGED ===
HTTPCODE=000
RC=56
curl: (56) CONNECT tunnel failed, response 403

$ ssh vr1-dc0-rack 'out="$(curl -sS -o /dev/null -w "%{http_code}" -x http://10.12.8.4:3129 -m 20 https://example.com/ 2>&1)"; rc=$?; echo "RC=$rc"; printf "%s\n" "$out"'
=== deny probe, 2>&1 merged (the blob the script matches) ===
RC=56
BLOB_BEGIN
curl: (56) CONNECT tunnel failed, response 403
000
BLOB_END

SQUID'S OWN VIEW of the same events:

$ ssh vr1-dc0-rack 'sudo grep DENIED /var/log/squid/dc0-snap-proxy.access.log'
1785539007.465      0 10.12.8.2 TCP_DENIED/403 3400 CONNECT example.com:443 - HIER_NONE/- text/html
1785539016.760      0 10.12.8.2 TCP_DENIED/403 3400 CONNECT example.com:443 - HIER_NONE/- text/html
1785539016.767      0 10.12.8.2 TCP_DENIED/403 3400 CONNECT example.com:443 - HIER_NONE/- text/html

>>> BUG-3 RULING: CONFIRMED AS EXPECTED. NO CODE CHANGE MADE. <<<

The DSTDOMAIN deny produces EXACTLY the same client-visible shape as the PORT
deny from which the assertion was originally written:
    curl exit code 56
    message  "CONNECT tunnel failed, response 403"
    http_code 000
and squid records it as TCP_DENIED/403 ... HIER_NONE. The assertion in
scripts/dc-snap-proxy.sh is therefore CORRECT AS WRITTEN and was left untouched.

This is the result BUG-3 was waiting for. The reason the shapes coincide is that
curl reports on the CONNECT RESPONSE STATUS, which squid renders as 403 for any
`http_access deny` regardless of which ACL fired; the deny reason is squid-side
(X-Squid-Error / the access log), not encoded in the status line. The BUG-3
protocol's step (b)/(c) -- re-point the assertion, add a harness case, run a
mutation -- were therefore NOT triggered and were correctly not performed.
scripts/dc-snap-proxy.sh is byte-identical to its pre-run state; see section 9.

===========================================================================
SECTION 7 -- CLIENT-SIDE (`node`) VERIFICATION, from a real remote client
===========================================================================

WHICH CLIENT, AND WHY -- the honest limit stated up front. The IDEAL client is a
charm LXD container, because the header's whole CLIENT_CIDR argument rests on
the finding that containers source from metal-admin. That client DOES NOT EXIST
right now, and this is MEASURED, not assumed:

$ ssh vr1-dc0-rack 'sudo virsh -c qemu:///system list --all'
 Id    Name                 State
--------------------------------------
 16    vr1-dc0-opnsense     running
 82    vr1-dc0-maas-01      running
 113   vr1-dc0-juju-01      running
 -     vr1-dc0-compute-01   shut off
 -     vr1-dc0-compute-02   shut off
 -     vr1-dc0-control-01   shut off
 -     vr1-dc0-control-02   shut off
 -     vr1-dc0-control-03   shut off
 -     vr1-dc0-storage-01   shut off
 -     vr1-dc0-storage-02   shut off
 -     vr1-dc0-storage-03   shut off
 -     vr1-dc0-storage-04   shut off

All NINE dc0 OpenStack nodes are SHUT OFF (the prior session's Stage 5 teardown,
commit 2208141). With no node powered there is no juju model and no charm
container, so the container-sourced proof is UNAVAILABLE this session -- not
"absent", UNAVAILABLE. Powering a node is outside this mandate.

The BEST AVAILABLE real client is the dc0 MAAS region VM, which sits ON the
metal-admin plane (10.12.8.0/22 = CLIENT_CIDR) and reaches the proxy ACROSS the
bridge, exactly as a charm container would -- unlike the rack, which reaches the
LISTEN address locally. Its source address was measured first, per the script's
own instruction:

$ ssh vr1-dc0-maas 'hostname; ip route get 10.12.8.4; command -v curl'
=== hostname / client identity ===
hot-kid
=== ip route get 10.12.8.4 ===
10.12.8.4 dev enp1s0 src 10.12.8.6 uid 1000
    cache
=== curl present? ===
/usr/bin/curl

src 10.12.8.6, via enp1s0, inside CLIENT_CIDR 10.12.8.0/22. A genuine remote
client on the client plane.

$ ssh vr1-dc0-maas 'bash -s' -- node dc0 < scripts/dc-snap-proxy.sh; echo "EXIT=$?"
dc-snap-proxy node check (dc0) -- from this client, against 10.12.8.4:3129
  OK      CONNECT to the store API through the proxy: curl exit 0, HTTP 200, channel-map + sha3-384 present
  OK      non-allowlisted destination is DENIED from this client -- the allowlist is in force
  FAIL    no https_proxy in /etc/environment -- the juju keys have not reached this client; run: juju model-config -m <model> snap-http-proxy=http://10.12.8.4:3129 snap-https-proxy=http://10.12.8.4:3129
dc-snap-proxy node (dc0): FAIL
EXIT=1

READING THE THREE PROBES SEPARATELY, as the mandate requires:
  (1) STORE REACHABILITY  -- OK. REAL RESULT. A remote client on the metal-admin
      plane is AUTHORIZED by `acl snap_clients src 10.12.8.0/22` and can open a
      CONNECT tunnel to the store, and the body is the store's own answer
      (channel-map + sha3-384), not a 200 from something else.
  (2) ALLOWLIST IN FORCE  -- OK. REAL RESULT. The deny holds from the client
      side too, so this is not an open forward proxy on the node plane.
  (3) /etc/environment    -- FAIL. EXPECTED AND OUT OF SCOPE. The juju
      snap-http-proxy / snap-https-proxy model-config keys have NOT been set;
      setting them is the next phase and is explicitly excluded from this
      mandate. This failure is the script correctly reporting a true state of
      the world. It is NOT a defect in the proxy and NOT a defect in the tool.

So the node-side exit 1 is driven ENTIRELY by the out-of-scope probe (3). The
two probes that test the proxy itself both passed from a real remote client.

CROSS-CHECK AGAINST SQUID'S OWN LOG -- a 200-shaped answer is not evidence by
itself, so the client identity was read back from the server side:

$ ssh vr1-dc0-rack 'sudo grep " 10.12.8.6 " /var/log/squid/dc0-snap-proxy.access.log'
1785539195.681   1240 10.12.8.6 TCP_TUNNEL/200 16426 CONNECT api.snapcraft.io:443 - HIER_DIRECT/185.125.188.55 -
1785539195.707      0 10.12.8.6 TCP_DENIED/403 3400 CONNECT example.com:443 - HIER_NONE/- text/html

squid independently confirms both outcomes and attributes them to 10.12.8.6.
The client-authorization path is proven from the server's own records.

===========================================================================
SECTION 8 -- REAL SNAP FETCH PROVEN END TO END (non-destructively)
===========================================================================

`check` proves a tunnel opens. It does NOT prove a snap can actually be fetched.
That was proven separately, WITHOUT installing any snap and without changing any
node state, by asking the store for `core`'s real download URL through the proxy
and then range-fetching the first four bytes of the actual payload.

STEP 1 -- the advertised download URL, obtained THROUGH the proxy:

$ ssh vr1-dc0-rack 'curl -sS -x http://10.12.8.4:3129 -m 30 -H "Snap-Device-Series: 16" "https://api.snapcraft.io/v2/snaps/info/core" | python3 -c ...'
download.url = https://api.snapcraft.io/api/v1/snaps/download/99T7MUlRhtI3U0QFgl5mXXESAiSwt776_17292.snap
download.size = 110305280
sha3-384[:32] = 13c728f16430310ba467b5768bf34846

STEP 2 -- range-fetch that URL through the proxy, WITHOUT following redirects:

$ ssh vr1-dc0-rack 'curl -sS -x http://10.12.8.4:3129 -m 60 -r 0-3 -o /tmp/snapmagic.bin -w "%{http_code} size=%{size_download}" "<download.url>"; xxd /tmp/snapmagic.bin'
curl rc=0 http=302 size=551
00000000: 3c21 646f 6374 7970 6520 6874 6d6c 3e0a  <!doctype html>.
...
000000a0: 3a2f 2f63 616e 6f6e 6963 616c 2d62 6f73  ://canonical-bos
000000b0: 3031 2e63 646e 2e73 6e61 7063 7261 6674  01.cdn.snapcraft
000000c0: 636f 6e74 656e 742e 636f 6d2f 646f 776e  content.com/down
000000d0: 6c6f 6164 2d6f 7269 6769 6e2f 6361 6e6f  load-origin/cano
000000e0: 6e69 6361 6c2d 6c67 7730 312f 3939 5437  nical-lgw01/99T7

>>> FINDING 8a -- THIS CORRECTS A MEASURED ASIDE IN THE SCRIPT HEADER. <<<
The header records (lines 124-127): "Measured aside worth knowing: `core`'s own
download URL came back on api.snapcraft.io, not a CDN host -- so for THIS snap
the CDN entry may be unused."
That aside measured only the FIRST HOP. The api.snapcraft.io download URL is a
302 REDIRECT to canonical-bos01.cdn.snapcraftcontent.com. The CDN allowlist
entry is therefore NOT "possibly unused" -- it is LOAD-BEARING for `core`, the
very snap the header used as its example. Had `.snapcraftcontent.com` been left
out of STORE_DOMAINS on the strength of that aside, every snap DOWNLOAD would
have failed after a successful API call -- precisely the failure mode the header
warns about two paragraphs earlier. The allowlist as built is correct; the
aside's conclusion was too weak, and the header should be corrected to say the
CDN entry is PROVEN REQUIRED. (LOGGED, not edited -- hard rule 1.)

STEP 3 -- follow the redirect through the proxy and read the real payload bytes:

$ ssh vr1-dc0-rack 'curl -sS -L -x http://10.12.8.4:3129 -m 90 -r 0-3 -o /tmp/snapmagic.bin -w "http=%{http_code} size=%{size_download} final=%{url_effective}" "<download.url>"; xxd /tmp/snapmagic.bin'
curl rc=0 http=206 size=4 final=https://canonical-bos01.cdn.snapcraftcontent.com/download-origin/canonical-lgw01/99T7MUlRhtI3U0QFgl5mXXESAiSwt776_17292.snap?token=1785553200_d069e2ced1d5f8cdfa97475090ced71bf2bb5838
=== bytes (squashfs magic = hsqs) ===
00000000: 6873 7173                                hsqs

HTTP 206 Partial Content, 4 bytes, and those bytes are 68 73 71 73 = "hsqs" --
the SQUASHFS MAGIC NUMBER, i.e. the first bytes of the genuine core snap image.
This is a REAL SNAP PAYLOAD retrieved through this proxy. Nothing was installed;
4 bytes of a 110 MB artifact were read and discarded.

CONFIRMED SERVER-SIDE in squid's access log (the CDN CONNECT is recorded):
1785539231.285    362 10.12.8.2 TCP_TUNNEL/200 4931 CONNECT canonical-bos01.cdn.snapcraftcontent.com:443 - HIER_DIRECT/91.189.91.43 -

>>> FINDING 8b -- THE DOT-PREFIX WILDCARD IS NOW PROVEN ON A NEW SUBDOMAIN. <<<
The header notes the CDN probe is a liveness-coupled assertion because "only a
real tunnel proves squid's dot-prefix matching actually ADMITS a subdomain". The
probe only ever exercises canonical-lgw01. This fetch admitted
canonical-bos01.cdn.snapcraftcontent.com -- a DIFFERENT subdomain that appears
in no probe and no config line, matched purely by the `.snapcraftcontent.com`
leading-dot rule. The wildcard is proven to generalise, not merely to match the
one host that happens to be probed.

===========================================================================
SECTION 9 -- FINDINGS LOGGED, NOT EXECUTED (hard rule 1)
===========================================================================

--- FINDING 9a: `acl snap_probe src ${LISTEN}/32` IS STILL A DEAD RULE. -------
    Severity: LOW for correctness today, MEDIUM for the guarantee it claims.
    NOT FIXED. Logged only. Recommend a D-NNN / changelog entry next session.

BUG-1 earlier today replaced a dead loopback probe ACL with `${LISTEN}/32`, i.e.
`acl snap_probe src 10.12.8.4/32`. The generated config's comment states:
  "The rack connecting to a LOCAL address selects that address as its source, so
   ${LISTEN}/32 is the expected match; if it is not, the probe still falls
   through to snap_clients because ${LISTEN} lies inside ${CLIENT_CIDR}."

MEASURED THIS SESSION -- the first half of that sentence is FALSE on this rack:

$ ssh vr1-dc0-rack 'ip route get 10.12.8.4'
local 10.12.8.4 dev lo src 10.12.8.2 uid 1000
    cache <local>

The kernel selects 10.12.8.2 -- the PRIMARY address on virbr2 -- as the source,
NOT the 10.12.8.4 secondary alias. Squid's access log agrees: every rack-side
probe in section 8's log extract is attributed to client 10.12.8.2, never
10.12.8.4. So `acl snap_probe src 10.12.8.4/32` matches NOTHING and is still
decoration after the BUG-1 fix. The probes are admitted solely by
`http_access allow snap_clients` (10.12.8.2 is inside 10.12.8.0/22).

WHY IT MATTERS. Nothing is broken today -- check PASSES, and it passes for a
sound reason. But the rule's STATED PURPOSE is not achieved: the comment says it
exists "so that NARROWING snap_clients cannot silently break `check`". Since the
probe actually matches via snap_clients, narrowing snap_clients WOULD break
`check`, silently, exactly as the rule was written to prevent. The safety net
is not connected.

CORRECT FIX (for a future step, NOT applied here): make the ACL the address the
rack actually sources from, or add it alongside -- e.g. derive the primary
address of the utility bridge and use that, or simply widen snap_probe to the
bridge's primary. Do NOT just delete the rule without deciding whether the
narrowing guarantee is wanted. This is a config-content change, so it must go
through the generator + a harness case that can turn it red.

NOTE: this is the SAME CLASS as BUG-1 (an ACL that matches nothing and is
believed to match something). BUG-1 fixed the symptom on the wrong address. The
detector that would have caught both is the one used here: read the CLIENT field
of squid's own access log, rather than reasoning about source selection.

--- FINDING 9b: the header's CDN aside understates the CDN entry. ------------
    See FINDING 8a. The `.snapcraftcontent.com` entry is PROVEN REQUIRED for
    `core`, not "possibly unused". Header comment lines 124-127 should be
    corrected. NOT EDITED this session (hard rule 1).

--- FINDING 9c: apt-installing squid leaves a FAILED, ENABLED unit. ----------
    See FINDING 3a. Between `apt-get install -y squid` and `dc-snap-proxy.sh
    install`, the rack carries a crash-looping-then-failed squid.service that is
    still ENABLED and would fight for 3128 at the next boot. `install` closes
    this, and `check` would catch it. Worth a sentence in any runbook that
    orders these two steps, because an operator who installs the package and
    then stops has left the rack in that state. NOT EDITED this session.

--- FINDING 9d: dc0 juju controller VM is RUNNING but rejected our key. ------
    Observed while looking for a client. vr1-dc0-juju-01 is `running` in virsh,
    and 10.12.8.5:22 answers, but:
      $ ssh vr1-dc0-juju 'which juju'
      ubuntu@10.12.8.5: Permission denied (publickey).
    with the configured IdentityFile
    /home/jessea123/vr1-office1-creds/office1_svc_ed25519. Also noted: there is
    NO juju client binary on the jumphost at all (`juju: No such file or
    directory`, /snap/bin is empty). NOT INVESTIGATED -- out of mandate scope,
    and no key hunting was performed. Flagged because the next phase (setting
    the juju snap-http-proxy / snap-https-proxy model-config keys) NEEDS a
    working juju client, and it is better to learn this now than mid-deploy.

===========================================================================
SECTION 10 -- FINAL STATE OF THE TOOLING (nothing was changed)
===========================================================================

BUG-3 resolved as "confirmed as expected", so no edit to the script or harness
was required or made. Verified by hash against the section-0 baseline:

$ sha256sum scripts/dc-snap-proxy.sh tests/dc-snap-proxy/run-tests.sh
318cfb4190eccdfe1d760ac3a7f8f1a23ac2b35e78d07167d730bef7150439f0  scripts/dc-snap-proxy.sh
aaa2a2060e770720f25284f9413b8c222e2d3b073a6f5ab2c6daad642fc0f8ff  tests/dc-snap-proxy/run-tests.sh

IDENTICAL to the section-0 baseline. No byte of the tool or its harness changed
during this session. The only working-tree change made by this session is the
creation of THIS capture file.

FINAL HARNESS AND LINT (re-run after the live application):

$ bash tests/dc-snap-proxy/run-tests.sh
dc-snap-proxy: 55 passed, 0 failed

Still 55/55 after the live application, as expected -- nothing was edited.

$ bash scripts/repo-lint.sh
  [FAIL] L10 pending change set modifies state-bearing surface(s) [docs/audit/: docs/audit/apex-ipv6-plan-20260731.txt, docs/audit/dc0-snap-proxy-install-20260731.txt] without touching docs/CURRENT-STATE.md -- GA-R1 rule 8 (C1): a status-changing commit updates CURRENT-STATE.md in the SAME commit

FAIL: repo lint (1 fail, 1 warn, 642 files scanned)

>>> OPEN ITEM FOR THE OPERATOR -- repo-lint L10 is RED and was NOT resolved. <<<

This is DECLARED, not hidden, and it was NOT worked around.

CAUSE, confirmed by reading the rule (scripts/repo_lint.py L10, ~line 450):
    audit_hits = sorted(p for p in changed if p.startswith("docs/audit/"))
    ...
    if triggers and "docs/CURRENT-STATE.md" not in changed:
ANY path under docs/audit/ in the pending change set trips L10 unless
docs/CURRENT-STATE.md is changed in the SAME commit. So this capture file ALONE
trips it; it is a structural consequence of producing a docs/audit/ deliverable,
not an accident.

WHY IT WAS LEFT RED. The mandate for this session explicitly forbids editing any
file under docs/ except this capture file, and explicitly forbids `git add` /
`git commit` / `git push`. Updating docs/CURRENT-STATE.md is the ONLY thing that
clears L10, and doing so would have violated that instruction. Deliberately left
for the operator to resolve at commit time, when CURRENT-STATE.md is updated for
the Stage 5 snap-proxy milestone anyway.

CONCURRENT-SESSION NOTE. Two files appeared in this working tree DURING this
session that this session did NOT create:
    ?? docs/audit/apex-ipv6-plan-20260731.txt
    ?? docs/changelog-20260731-snap-proxy-apply-ipv6.md
They were absent from `git status` at session start and present at the end.
Another session is working in this same clone concurrently. This is exactly the
condition behind the earlier commit-contamination incident (e57ad09), so NO git
staging or commit operation was performed by this session, and a blanket
`git add -A` here would sweep another session's in-progress work into a commit.
Whoever commits must stage PATHS EXPLICITLY.

===========================================================================
SUMMARY
===========================================================================

PROVEN END TO END ON REAL HARDWARE THIS SESSION:
  1. The gate FAILS on a real rack with nothing built (exit 1) and PASSES with
     everything built (exit 0). No longer fixture-only in either direction.
  2. `install dc0` works first-try, is self-checking, and its artifacts match
     their generators by content diff.
  3. The proxy RUNS: squid 6.14 (pid 2916382) bound to 10.12.8.4:3129
     specifically, both units enabled, packaged squid.service disabled.
  4. Coexistence with MAAS's squid: MAAS's pid 1871633 untouched across the
     whole install, still holding *:3128 and *:8000.
  5. Net-layer idempotence with dc0-mirror-net.service: virbr2's addresses and
     the default route are byte-identical before and after install.
  6. BUG-3: the dstdomain deny shape is IDENTICAL to the port deny shape
     (curl 56 / "CONNECT tunnel failed, response 403" / TCP_DENIED/403). The
     assertion is correct as written; NO code change was made.
  7. Client-side authorization from a REAL remote client on the metal-admin
     plane (10.12.8.6, across the bridge): allowed to the store, denied to a
     non-allowlisted host, both corroborated in squid's own access log.
  8. A REAL SNAP PAYLOAD was fetched through the proxy: HTTP 206 and the bytes
     "hsqs" (squashfs magic) of the genuine core snap, via the CDN.
  9. The `.snapcraftcontent.com` wildcard admits a subdomain that appears in no
     probe and no config line (canonical-bos01).

NOT PROVEN / STILL OPEN:
  A. NO charm LXD container was tested -- all 9 dc0 nodes are shut off, so the
     "containers source from metal-admin" premise was NOT re-proven this
     session. It remains carried from the prior capture. `node dc0` from a
     container is the outstanding client-side proof.
  B. snapd's own consumption of the proxy is UNPROVEN and OUT OF SCOPE: the
     juju snap-http-proxy / snap-https-proxy keys are not set, /etc/environment
     carries no https_proxy, and no snap was installed. The header's carried
     unknown -- whether a RUNNING snapd re-reads the value without a restart --
     is UNTOUCHED by this session.
  C. dc1 was NOT touched at all. Everything above is dc0 only.
  D. The `snap_probe` ACL is still dead (FINDING 9a). check passes for a sound
     reason, but not the reason the config comment claims.
  E. No juju client exists on the jumphost and vr1-dc0-juju rejects the
     configured key (FINDING 9d) -- the next phase needs one.

END OF CAPTURE.

===========================================================================
ADDENDUM A -- attribution of the .4 alias, MEASURED (closing a soft claim)
===========================================================================

Section 2 asserted the 10.12.8.4/22 alias was "placed by dc0-mirror-net.service".
That was the one INFERENCE in this capture: the alias was measured present and
the unit measured active, but the causal link was not measured. Hard rule 2 says
a would-be inference stops and gets measured, so it was:

$ ssh vr1-dc0-rack 'systemctl cat dc0-mirror-net.service | grep ExecStart; sudo grep -n "10.12.8.4" /usr/local/sbin/dc0-mirror-net-apply'
=== dc0-mirror-net.service ExecStart ===
ExecStart=/usr/local/sbin/dc0-mirror-net-apply
=== does the mirror helper place 10.12.8.4? ===
10:ip addr replace 10.12.8.4/22 dev "$(br_of vr1-dc0-metal-admin)"

CONFIRMED. dc0-mirror-net.service runs /usr/local/sbin/dc0-mirror-net-apply,
whose line 10 places 10.12.8.4/22 on the vr1-dc0-metal-admin bridge -- the SAME
address, the SAME bridge, and the SAME `ip addr replace` verb that
dc0-snap-proxy-net-apply uses. The attribution in section 2 is measured, and the
header's "identical MEASURED values, so they are no-ops for each other" claim is
now confirmed at the level of the generated helper source, not just by observing
that nothing changed.

===========================================================================
ADDENDUM B -- FINDING 9e (logged, not applied)
===========================================================================

--- FINDING 9e: the packaged squid.service rests FAILED, not INACTIVE. -------
    Severity: COSMETIC, but it is a triage-poisoning surface. NOT APPLIED.

Final measured state after install:

$ ssh vr1-dc0-rack 'systemctl is-active squid; systemctl is-enabled squid'
failed
disabled

`check` reports "OK packaged squid.service neither active nor enabled", and that
verdict is CORRECT: `is-active --quiet` returns non-zero for `failed`, and
`is-enabled` is `disabled`, so the unit cannot return at the next boot. The gate
is right.

The gap is narrower than a wrong verdict: `check` cannot DISTINGUISH `inactive`
from `failed`. The packaged unit will sit in `failed` indefinitely (its parent
logged "will not be restarted for 3600 seconds due to repeated, frequent
failures" before being disabled). So a future operator running `systemctl status
squid` -- or any red-unit sweep, or `systemctl --failed` -- sees a RED squid unit
on a rack whose snap-proxy gate is GREEN, and has to rediscover from scratch that
this is expected residue from the package install and not a fault. That is
precisely the "crash-looping squid poisons every later triage" hazard the script
header set out to avoid, surviving in a quieter form.

REMEDY (for a future step, NOT applied here): `systemctl reset-failed squid`
after `systemctl disable --now squid`, which clears the residue without changing
any enablement or binding. If adopted it belongs in do_install's disable line,
and `check` could then assert the packaged unit is `inactive`+`disabled` rather
than merely not-active. Both are content changes to a built surface, so they
need the generator + a harness case that can turn red, and a D-NNN/changelog
entry. LOGGED ONLY -- hard rule 1.

END OF ADDENDA.
