Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260731-snap-proxy-apply-ipv6.md

Session changelog 2026-07-31 (part 2) -- snap proxy apply + IPv6 scoping

GA-R2: ONE changelog per session. This is a SEPARATE session from docs/changelog-20260731-prefer-ipv6-research.md (same date, different session); that file is closed and is not appended to here.

Branch dc-dc-stage5-preconditions. Status claims live ONLY in docs/CURRENT-STATE.md; nothing here is a status claim.


Item 1 -- FINDING: the dc-snap-proxy.sh review's SIMPLIFICATION PLAN was never

landed on any surface and is LOST

What. No artifact change. A finding, recorded because the next action in the prior session's close-out cannot be executed as written.

docs/session-ledger.md:296 gives the next step as "apply the review cuts, re-run the mutation pass (S8 invalidates it)", and docs/audit/queued-findings-20260731-stage5-deploy.txt:145-146 (N2) says "The dc-snap-proxy simplification plan is NOT applied. Applying it REQUIRES re-running the mutation pass -- S8 invalidates M32-M39, M42-M47, M49-M52 as evidence."

MEASURED: the plan those S-numbers refer to exists nowhere in the repo. A repo-wide grep for \bS8\b over *.md / *.txt / *.sh returns exactly two live hits -- the ledger line and the N2 line above, both of which CITE the plan without containing it -- plus three unrelated 2026-07-07 archive hits (docs/archive/script-quality-findings-20260707.md, an S1-S8 consolidation proposal set about KEYSTONE_VIP and D-003 FIP bounds, a different subject entirely). The mutation IDs it invalidates ARE recorded (docs/audit/stage5-snap-proxy-measurements-20260731.txt:505,516,523); the simplification items themselves are not.

So N2's wording understates it. "NOT applied" reads as deferred work that is waiting somewhere. It is not deferred -- it lived only in the review agent's transcript and was lost at the session boundary. This is the repo's own top-tier failure mode (a finding that exists only in a transcript), landing on a finding ABOUT losing findings.

DECISION, and the reasoning stated so it can be overturned: the simplification is NOT re-derived. It was quality-only work on a tool that is about to be applied for the first time; re-deriving a lost plan and then re-running a 53-mutation pass to restore evidence for it is precisely the record-churn the 2026-07-30 standing operator directive shuts down ("Further hardening, auditing, or record repair is OUT OF SCOPE next session unless it directly blocks the deploy"), and it is the GA-F06 circuit-breaker case. Consequence that follows and is NOT a loss: because S8 is never applied, its invalidation of M32-M39/M42-M47/M49-M52 never fires, so the prior session's 53/53 mutation evidence STANDS unchanged for every assertion this session did not touch.

Revert. Nothing to revert (no artifact changed). To reverse the decision, re-derive the plan from scratch against the current file and re-run the full mutation pass -- the old S-numbers are not recoverable and must not be cited.


Item 2 -- FIX BUG-4: dpkg -s squid gates replaced by a BINARY assertion, at

BOTH sites (check AND install)

What. scripts/dc-snap-proxy.sh (the check package gate and do_install's prereq) + tests/dc-snap-proxy/run-tests.sh (the dpkg PATH fake removed, a real binary seam added, T41 re-pointed, T41b and T20b NEW).

Why. BUG-4 from the independent review (docs/audit/queued-findings-20260731-stage5-deploy.txt:64-66): dpkg -s squid exits 0 for Status: deinstall ok config-files, so a PURGED squid reports as present. The existence-vs-content class this repo already has a scar from. [ -x /usr/sbin/squid ] is shorter and strictly stronger.

The review named ONE site; there are TWO. The same gate sat at do_check AND at do_install's prereq. Fixing only check would have left install false-passing its prereq on a purged package and then dying confusingly at squid -k parse or at unit start -- a worse failure than the one being fixed, because it surfaces further from its cause. Both are fixed.

Written through the EXISTING test seam, not around it. The assertion is [ -x "${SYSROOT}/usr/sbin/squid" ]. SYSROOT is empty in every production invocation, so the live path is byte-identical to /usr/sbin/squid; the harness points it at its mktemp tree. This is the script's own documented idiom (DC_SNAP_PROXY_SYSROOT, with in-repo precedent in maas-region-power-key.sh, creds-audit.sh, cloud-snapshot.sh) and it keeps the assertion drivable in BOTH directions, which the dpkg fake also was -- the fix does not cost a failure direction.

Test cases, and why each exists.

  • T41 (re-pointed): binary absent -> FAIL. The old case drove dpkg.mode.
  • T41b (NEW): the path is present but NOT executable -- the exact deinstall ok config-files residue shape. This is the case the OLD assertion could not fail on, i.e. the one that makes the fix mean something. Without it the fix would be untested in its own direction.
  • T20b (NEW): do_install is never EXECUTED by this harness (it needs root and a real rack), so its half of the fix would otherwise carry NO assertion at all. Asserted structurally, following T18's established precedent of scoping a grep to a function body: zero dpkg -s gates anywhere in the file, AND INSTALL_BODY carries the [ ! -x ... ] form.

Mutation-proven (this repo's rule: an assertion nobody can turn red is decoration). Three mutations, each neutering a PREDICATE, script restored and sha256-verified byte-identical after each:

  • check's binary predicate -> if true : T41 AND T41b both red (53/2).
  • install's prereq -> if ! dpkg -s squid ... : T20b red (53/1).
  • install's prereq -> if false : T20b red on its second branch (53/1).

Revert. git checkout -- scripts/dc-snap-proxy.sh tests/dc-snap-proxy/run-tests.sh (reverts items 2-4 together; they are edits to the same two files).


Item 3 -- FIX BUG-1: the probe ACL was DEAD, and it is now asserted

What. gen_squid_conf's acl snap_probe src 127.0.0.1/32 -> src ${LISTEN}/32, its comment block rewritten to describe the real mechanism, + T23b NEW.

Why. BUG-1: squid binds ${LISTEN}:${PROXY_PORT} only, so nothing can ever arrive on loopback -- the rule matched nothing. The probes were actually admitted by snap_clients (because LISTEN lies inside CLIENT_CIDR), and the comment misdescribed its own mechanism. The hazard was live, not cosmetic: anyone narrowing CLIENT_CIDR would break check, and the rule they would reach for to fix it would not have helped.

Why the stronger fix was taken rather than a comment-only fix. The obvious minimal action was to correct the comment and leave the dead rule, on the grounds that changing generated content pulls fixtures with it. Measured, it does not: the harness renders its fixtures by calling the script's OWN generators (render gen_squid_conf, run-tests.sh:312-318), never by copying config text into the test file, so generated-content changes cost nothing in fixture maintenance. That removed the only argument for the weaker fix.

Safe in both directions by construction -- stated because the source-selection step is REASONED, not yet measured on the rack (hard rule 2). A host connecting to one of its own local addresses is expected to select that address as the source, so ${LISTEN}/32 is the expected match; if that is wrong, the probe still falls through to snap_clients, because LISTEN is inside CLIENT_CIDR. So the change cannot regress check, and the first live run's access log will say which rule actually admitted the probe.

T23b (NEW) asserts the RENDERED artifact -- the probe ACL is the LISTEN address and is NOT a loopback rule -- never the comment that describes it. BUG-1's real defect was that a comment and its code disagreed, so an assertion on prose would repeat the defect.

Mutation-proven. Reverting the ACL to 127.0.0.1/32 turns T23b red (54/1); script restored sha256-identical.


Item 4 -- BUG-3 DOCUMENTED IN THE SCRIPT, deliberately NOT "fixed"

What. A new block in scripts/dc-snap-proxy.sh's header, under the existing "READ THIS BEFORE TRIAGING A REFUSE" section. No behavior change.

Why not fixed. BUG-3 is that the DENY probe's expected string CONNECT tunnel failed, response 403 was MEASURED from a port deny (deny CONNECT !SSL_ports), never from the dstdomain deny (deny !snap_store) that the probe actually exercises. It cannot be fixed in the repo, because the correct value is a fact about a live squid that nobody has yet observed -- inventing one would be exactly the fabricated-value failure this repo calls the cardinal sin. Guessing a second plausible string would be worse than the bug.

Why it is not merely cosmetic. If squid answers a dstdomain denial differently, that branch falls to refuse -> exit 3, and check can then NEVER PASS. Under GA-R6 a gate that cannot PASS blocks a stage close exactly as hard as one that cannot fail, so this is a real gate defect and not a probe nicety.

The protocol is now IN THE FILE rather than in a queued-findings entry the next reader will not open: a REFUSE on the deny probe at first run is the EXPECTED, HANDLED path; read the actual output; re-point the assertion to the MEASURED shape; keep the three-outcome structure so an unrecognised result still REFUSES; prove the new assertion can fail. This is also the answer to the close-out's "confirm BUG-3's deny shape once" -- it is a byproduct of the first real check, not separate work.

BUG-2 NOT FIXED, unchanged from N4. | grep -q under set -o pipefail is INHERITED and present identically in dc-mirror.sh:278,281 and dc-cache-proxy.sh:234,252,255. A three-script sweep is out of scope under hard rule 1 and under the standing directive. Its direction is fail-closed (a false MISS, never a false OK), which is why it can wait.


Item 5 -- OPERATOR APPROVAL RECORDED: the agent's rack permissions

What. No artifact change. Recorded because it relaxes the scope of hard rule 3 for one task and must not be inferred later from silence.

Presented to the operator, verbatim as put: .claude/settings.local.json carries Bash(ssh vr1-dc0-rack *) in allow -- an unrestricted wildcard onto the dc0 rack, auto-added by an approval on 2026-07-30 and flagged in that session's sweep (F1) as "worth review". Its consequence, stated at the time of asking: a background agent can mutate the dc0 rack (including sudo bash -s -- install dc0) with NO permission prompt reaching the operator, so hard rule 3's individual gating does not fire for rack commands. The operator was offered a narrowing.

Operator answer, exact utterance: "The agent permissions are approved".

CONSEQUENCE, scoped deliberately narrowly. This authorises the snap-proxy install agent to operate on the dc0 rack under the existing allow rule. It is NOT a general standing relaxation of hard rule 3, it does not extend to dc1, and it does not cover the bundle deploy -- the agent's mandate explicitly stops before add-model / deploy / any juju model-config. A future session must not read this as blanket approval for unsupervised cloud mutation.

Not a GA-R5 ruling and deliberately not recorded as one: it is a harness permission decision, OPS under GA-R3 (no architectural consequence beyond this stage, no Roosevelt delta), so it takes no D-number. The gitignored settings.local.json is unchanged by this session.

Revert. Remove Bash(ssh vr1-dc0-rack *) from allow in .claude/settings.local.json and replace it with a targeted ask rule pinned to the invocation shapes actually needed. Note the 2026-07-30 lesson that applies directly here: quoting style and the profile token are BOTH part of the match, so a narrowing rule that pins the wrong quote form silently fails to match and falls through to the classifier.


Gate results at time of writing

  • bash tests/dc-snap-proxy/run-tests.sh -- 55 passed, 0 failed (52 -> 55: T20b, T23b, T41b added; T41 re-pointed).
  • bash scripts/run-tests-all.sh -- GAUNTLET: ALL GREEN (94 harnesses). HARNESS-MANIFEST NOT re-recorded and deliberately so: no harness was added or removed, only cases within an existing one, so the manifest is unchanged and the R15(2) re-record reason does not apply.
  • bash scripts/repo-lint.sh -- 0 fail, 1 warn (the expected L1 legacy non-ASCII carve-out in docs/design-decisions.md).
  • Four mutations run across items 2-3; all four killed; script restored and sha256-verified byte-identical after each.

Item 6 -- THE dc0 SNAP PROXY IS INSTALLED, RUNNING AND GATE-VERIFIED

What. No repo artifact changed. A LIVE MUTATION on the dc0 rack, executed by a delegated agent under the operator approval recorded in item 5. Capture: docs/audit/dc0-snap-proxy-install-20260731.txt (739 lines, ASCII/LF).

NAMED GATE, GA-R6: dc-snap-proxy.sh check dc0 = PASS, EXIT 0, 16 assertions. RE-RUN INDEPENDENTLY from the main session after the agent reported, rather than taken on its word: squid 6.14 bound to 10.12.8.4:3129 SPECIFICALLY (not wildcard), both units active AND enabled, packaged squid.service neither active nor enabled, all five owned artifacts matching their generators BY CONTENT, and all three behavioral probes green (store API with channel-map + sha3-384, CDN tunnel allowed, non-allowlisted destination DENIED).

The before-state was captured and it FAILED (exit 1) on a real rack -- so the gate is proven able to fail somewhere other than a fixture. It was NOT uniformly failed: three OK lines (virsh-resolved bridge, rack route table) prove the instrument was on the RIGHT HOST. That distinction is the recorded wrong-host trap for these per-DC scripts, where a jumphost run reports every item MISS.

MAAS's squid was untouched throughout -- same pid (1871633) before and after, still holding *:3128 / *:8000. Net-layer idempotence with dc0-mirror-net.service verified: virbr2 addresses and the default route are byte-identical before and after, which is the D-135 coexistence property this script was built to honour.

A REAL SNAP PAYLOAD WAS FETCHED THROUGH THE PROXY -- HTTP 206, and the first bytes are hsqs, the squashfs magic of the genuine core snap. 4 bytes of ~110 MB read and discarded; nothing installed. This is what moves the tool from FIXTURE-green to end-to-end proven, and it is the specific claim the prior session was careful to say it could not make.

STILL NOT PROVEN, stated because the deploy depends on it: no charm LXD container was tested. All nine dc0 nodes are shut off (measured via virsh), so the "containers source from metal-admin" premise underpinning CLIENT_CIDR was NOT re-proven and is carried forward from the prior session's measurement. snapd's own consumption is unproven and out of scope (the juju keys are unset, so node's /etc/environment assertion FAILS as expected). dc1 untouched.

Revert. ssh vr1-dc0-rack 'sudo systemctl disable --now dc0-snap-proxy.service dc0-snap-proxy-net.service', then remove the five generated artifacts named in the script header. The packaged squid remains installed and disabled; MAAS's squid is unaffected either way.


Item 7 -- BUG-3 RESOLVED BY MEASUREMENT: the deny shape is CONFIRMED, no

assertion changed

What. No change. The confirmation BUG-3 was waiting for.

The dstdomain deny (deny !snap_store) produces a shape BYTE-IDENTICAL to the port deny the assertion was originally written from: curl exit 56, CONNECT tunnel failed, response 403, http_code 000, with squid logging TCP_DENIED/403 ... HIER_NONE. Measured by hand with stderr both merged and unmerged, not inferred from check's own say-line -- which matters, because reading the gate's verdict to confirm the gate's assumption is circular.

MECHANISM, so this is not re-opened later: curl reports the CONNECT response status, and squid renders 403 for ANY http_access deny regardless of which ACL fired. The two shapes coincide by construction, not by luck.

CONSEQUENCE: BUG-3's protocol steps (b) and (c) were correctly NOT performed, and scripts/dc-snap-proxy.sh + its harness are sha256-identical to their pre-install state. The header block added in item 4 stays as the record of why the question existed and how it was settled. check dc0 can now be cited as a gate.


Item 8 -- MY BUG-1 FIX WAS WRONG, AND THE MEASUREMENT REFUTES IT (FINDING 9a)

What. tests/dc-snap-proxy/run-tests.sh T23b gains a prominent in-file annotation. scripts/dc-snap-proxy.sh deliberately NOT changed.

Item 3 changed acl snap_probe src 127.0.0.1/32 to src ${LISTEN}/32 and stated the source-selection step as REASONED, NOT MEASURED. The measurement says it is wrong. Measured on the dc0 rack, and re-run independently from the main session:

$ ip route get 10.12.8.4
local 10.12.8.4 dev lo src 10.12.8.2

.4 is a SECONDARY on virbr2 (primary .2), and Linux does not auto-select a secondary address as a source. Squid's own access log attributes every rack-side probe to 10.12.8.2. So the rule is STILL DEAD -- BUG-1's fix corrected the mechanism DESCRIPTION and landed on the WRONG ADDRESS.

Nothing is broken. The probes are admitted by snap_clients exactly as before, and check dc0 passes. What remains unconnected is the rule's stated PURPOSE: that narrowing snap_clients must not silently break check.

NOT FIXED NOW, and the reason is not laziness. check diffs the generated config against the rack's LIVE file, so any edit to gen_squid_conf turns the live dc0 gate RED until install is re-run. That makes it a LIVE MUTATION on a just-verified surface, out of scope under hard rule 1 and against the standing directive. The correct value is the rack's primary metal-admin leg (dc0 10.12.8.2; dc1's 10.12.68.2 is already recorded MEASURED in lib-hosts.sh), and it wants a site-keyed row. Fix the script and T23b TOGETHER at the next install -- dc1 is the natural point, so both DCs land in one pass.

T23b is annotated rather than deleted or re-pointed, per the standing rule that a test asserting a finding must be REPLACED with the new invariant and never deleted to go green. Editing its expected string to 10.12.8.2 today would turn the harness red against the live config; leaving it unannotated would let a future session "fix" the test to match a value that is itself wrong. The annotation states the measurement, the correct value, and why the change is deferred.

GENERALISABLE, and it is the reusable half: a secondary IPv4 alias is never the kernel's chosen source address. Any ACL, firewall rule or allowlist keyed to a service's ALIAS address -- which is exactly what the D-134 utility .4 band is on both racks -- will not match that host's own outbound traffic. This applies equally to dc-mirror.sh and dc-cache-proxy.sh, which share the .4 alias pattern.

Revert. git checkout -- tests/dc-snap-proxy/run-tests.sh.


Item 9 -- FOUR FURTHER FINDINGS FROM THE INSTALL, logged not fixed

  1. The script header's CDN aside is WRONG (finding 8a). It records that core's download URL "came back on api.snapcraft.io, not a CDN host -- so the CDN entry may be unused." That measured only the FIRST HOP: the URL 302-redirects to canonical-bos01.cdn.snapcraftcontent.com. The .snapcraftcontent.com entry is LOAD-BEARING, not speculative, and the same run proves squid's dot-prefix wildcard admits a subdomain appearing in NO probe and NO config line. DOCFIX material against the header.
  2. apt-get install squid leaves a FAILED but still ENABLED unit (9c) that would fight for 3128 at the next boot -- the header's predicted FATAL: Unable to open HTTP Socket ... listening port: 3128 was observed VERBATIM. install closes it; an operator who installs the package and stops there does not.
  3. The packaged unit rests failed+disabled, not inactive (9e). check is right to pass, but systemctl --failed shows red on a healthy rack, which is quiet triage-poisoning. systemctl reset-failed squid is the remedy; NOT applied (a live mutation outside the mandate).
  4. >>> NO JUJU CLIENT EXISTS ON THE JUMPHOST, AND THE dc0 JUJU VM REJECTS THE CONFIGURED KEY (9d). <<< ssh vr1-dc0-juju -> Permission denied (publickey). Not a permission-harness refusal and NOT retried in an altered shape. This blocks the next phase, which is add-model + spaces + deploy under D-138 (cloud-facing clients run from INSIDE the DC). Owed before that phase starts.