diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 2833e65..d49a0bb 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1686,6 +1686,170 @@ runbook reads would leave ovn-chassis with no provider bridge mapping, surfacing later as tenant networks with no external path rather than as a deploy error. + **>>> THE dc0 SNAP FORWARD PROXY IS INSTALLED, RUNNING AND GATE-VERIFIED 2026-07-31. + THE LAST BUILD BETWEEN HERE AND `add-model` IS DONE. <<<** Capture + `docs/audit/dc0-snap-proxy-install-20260731.txt` (739 lines). **NAMED GATE (GA-R6): + `dc-snap-proxy.sh check dc0` = PASS, EXIT 0, 16 assertions** -- squid 6.14 bound to + `10.12.8.4:3129` SPECIFICALLY (not wildcard), both units active AND enabled, the packaged + `squid.service` neither active nor enabled, all five owned artifacts matching their + generators BY CONTENT, and all three behavioral probes green. **Re-run INDEPENDENTLY from + the main session after the delegating agent reported, rather than accepted on its word.** + The before-state `check` FAILED (exit 1) on the real rack, so the gate is proven able to + fail somewhere other than a fixture -- and it was NOT uniformly failed (the virsh-resolved + bridge and rack route-table lines read OK), which is what distinguishes a real red from the + recorded wrong-host trap where these per-DC scripts report every item MISS. + **END-TO-END, the claim the prior session could NOT make: A REAL SNAP PAYLOAD WAS FETCHED + THROUGH THE PROXY** -- HTTP 206 with the first bytes `hsqs`, the squashfs magic of the + genuine `core` snap (4 bytes of ~110 MB read and discarded; nothing installed). The tool is + no longer FIXTURE-green. **MAAS's own squid was untouched** (same pid before and after, + still holding `*:3128`/`*:8000`), and net-layer idempotence with `dc0-mirror-net.service` + was verified byte-identical before and after -- the D-135 coexistence property this script + exists to honour. **NOT PROVEN and carried forward:** no charm LXD container was tested + (all nine dc0 nodes are shut off from the teardown), so the containers-source-from-metal-admin + premise behind `CLIENT_CIDR` rests on the prior session's measurement; snapd's own + consumption is unproven (juju keys unset); dc1 untouched. + **BUG-3 IS RESOLVED BY MEASUREMENT AND NO ASSERTION CHANGED.** The dstdomain deny produces + a shape byte-identical to the port deny the assertion was written from -- curl exit 56, + `CONNECT tunnel failed, response 403`, http_code 000, squid logging `TCP_DENIED/403`. + Mechanism, so it is not re-opened: curl reports the CONNECT RESPONSE STATUS, and squid + renders 403 for ANY `http_access deny` regardless of which ACL fired, so the two shapes + coincide BY CONSTRUCTION. Measured by hand rather than read off `check`'s own verdict, which + would have been circular. `check dc0` may now be cited as a gate. + **BUG-1's FIX WAS WRONG AND THE MEASUREMENT REFUTES IT -- LOGGED, NOT FIXED.** This session + changed the dead `acl snap_probe src 127.0.0.1/32` to `src ${LISTEN}/32` and recorded the + source-selection step as REASONED, NOT MEASURED. Measured on the rack: + `ip route get 10.12.8.4` -> `local 10.12.8.4 dev lo src 10.12.8.2`. **`.4` is a SECONDARY on + virbr2 and Linux does not auto-select a secondary as a source address**, so the rule is + STILL DEAD and squid's access log attributes every rack probe to `10.12.8.2`. Nothing is + broken (`snap_clients` admits the probes, and `check` passes); what is unconnected is the + rule's stated PURPOSE, that narrowing `snap_clients` must not silently break `check`. NOT + fixed now because `check` diffs the generated config against the rack's LIVE file, so + editing the generator turns the live gate RED until `install` is re-run -- a live mutation + on a just-verified surface. Harness case T23b is ANNOTATED with the measurement and the + correct value (dc0 `10.12.8.2`; dc1's `10.12.68.2` already MEASURED in `lib-hosts.sh`) so a + later session cannot "fix" the test to match a wrong value. **GENERALISABLE: a secondary + IPv4 alias is never the kernel's chosen source, so any ACL keyed to a service's ALIAS -- + which the D-134 utility `.4` band is on both racks -- will not match that host's own + outbound traffic. Applies equally to `dc-mirror.sh` and `dc-cache-proxy.sh`.** + **BLOCKER FOR THE NEXT PHASE, found in passing and NOT retried in an altered shape: there is + NO juju client on the jumphost, and the dc0 juju VM REJECTS the configured key** + (`ssh vr1-dc0-juju` -> `Permission denied (publickey)`). D-138 puts cloud-facing clients + INSIDE the DC, so this is owed before `add-model`. Three further findings logged not fixed: + the script header's CDN aside is WRONG (`core`'s download 302-redirects to + `canonical-bos01.cdn.snapcraftcontent.com`, so `.snapcraftcontent.com` is LOAD-BEARING, not + speculative -- DOCFIX material); `apt-get install squid` leaves a FAILED but still ENABLED + unit that would fight for 3128 at the next boot; and the packaged unit rests + `failed`+`disabled`, so `systemctl --failed` shows red on a healthy rack. + **APEX IPv6 PLANNING RECORD PULLED LIVE 2026-07-31** (read-only; capture + `docs/audit/apex-ipv6-plan-20260731.txt`). Source is the VR1 WORKING apex + `office1-netbox` 10.10.1.10:8000 (DOCFIX-195), NOT the baldurkeep v1 reference -- the two + credential files were distinguished by URL and token length before use, per the repo's + one-script-per-source rule. **INSTRUMENT NOTE: the repo-carried export + `netbox/draft/vr1-office1-current-20260725.json` PREDATES the 2026-07-27 apex load (IPv6 + 0 -> 78) and is NOT a valid source for any v6 question.** THREE MEASURED RESULTS. + **(i) The ULA/GUA split is the RECORDED PLAN, not an open decision** -- every VR1 plane at + both DCs carries an explicit prefix citing `D-101/D-111`: provider-public GUA + (`2602:f3e2:f02:10::/64` dc0, `f03:10::/64` dc1) plus a DEDICATED GUA VIP `/64` (`:11::`), + and ULA for metal-admin `:220`/`:320`, metal-internal `:221`/`:321`, data-tenant + `:230`/`:330`, storage `:240`/`:340`, replication `:250`/`:350`, each with a `/60` parent + reserved above the active `/64`. **(ii) THE APEX CARRIES ZERO IPv6 ip-ranges** -- all 27 + are v4 D-134 utility/VIP bands covering all twelve plane/DC combinations, so the missing + MAAS v6 ranges are NOT a build gap against a plan: **the plan itself has no v6 bands**, and + that is the concrete IPv6 planning work that does not exist. Note the two families are + shaped differently at the top (GUA gets a dedicated VIP `/64`; ULA planes carry VIPs inside + the plane `/64`), so a v6 band model cannot simply mirror v4's. **(iii) The apex's v6 + record is VIP-ONLY** -- 78 addresses, exactly 13 per prefix across six prefixes (13 + VIP-carrying apps x 3 legs x 2 DCs), while the 54 node v6 statics that DO exist in MAAS are + absent from the apex. Apex and MAAS therefore diverge on v6 in BOTH directions. + **A ROOSEVELT-DELTA QUESTION THIS SURFACES, NOT YET PUT TO THE OPERATOR:** the apex shows + **Willamette** (a real site: Psi/Alpha/Beta/Omega clouds) and **VR0** both using **GUA on + every plane** (`102:20/30/40/50/80/f0::` and `e02:20/30/40/50/80/f0::`), while VR1 uses ULA + internally -- and **Roosevelt (`2602:f3e2:103::/48`) has NO plane carve yet**, so whichever + pattern VR1 proves is the one it inherits. Under MINIMIZE DELTA TO ROOSEVELT, VR1 is the + LARGER delta against both the prior rehearsal and the live site. + **NAT64/NAT46 RAISED BY THE OPERATOR 2026-07-31 AS A WAY TO TEST THE v6 LEGS; PRIOR ART + FOUND AND NOT YET RULED.** NAT64/DNS64 was CONSIDERED AND REJECTED 2026-07-27 (recorded on + D-101; listed out-of-scope in `docs/audit/node-v6-carve-scope-20260727.md`) -- but that + rejection was scoped to simulating v6 EGRESS, which is a different purpose, so it does not + automatically bind. **Analysis recorded because it is the substance: D-101's own family + matrix rules data-tenant, storage, replication, lb-mgmt and P2P links IPv6-ONLY, and every + one of those is EAST-WEST between nodes that already hold v6 statics -- so there is nothing + for a translator to translate.** Separately measured from the family matrix: **three planes + D-101 rules v6-only are in fact DUAL-STACKED in the build** (data-tenant `10.12.16.0/22`, + storage `10.12.32.0/22`, replication `10.12.36.0/22`), which is the deliberate v4 safety + net rather than a defect -- but it means the untested v6 objective is REMOVING v4 from a + ruled-v6-only plane. Also recorded: "NAT46" is not the symmetric twin of NAT64 (stateful + NAT64 is v6->v4 only; the reverse needs SIIT with a mapped v4 address PER SERVER, which + spends the very IPv4 that D-101's sizing rationale exists to conserve). + **>>> D-139 ADOPTED 2026-07-31 (GA-R5) -- VR1 GOES IPv6-ONLY ON THE EAST-WEST PLANES AND + THE WHOLE CARVE MOVES TO GUA. <<<** TWO rulings taken in SEPARATE exchanges and recorded + separately (neither is a batch adoption). **Ruling A, family matrix -- operator utterance: + "IPv6 on all planes except for metal-admin and provider-public which will remain dual + stack".** Per DC: `provider-public` and `metal-admin` DUAL-STACK; `metal-internal`, + `data-tenant`, `storage`, `replication` and `lb-mgmt` **IPv6-ONLY**. AMENDS D-101's family + matrix in two places -- metal-internal's "datastore east-west stays v4-bound" is + SUPERSEDED, and `lb-mgmt` (already ruled v6-only by D-101 but **never carved anywhere**) + becomes a first-class plane. **Ruling B, addressing model -- operator utterance: "Full GUA + on every plane (Recommended)".** Every plane carves from its DC's GUA `/48` + (`2602:f3e2:f02::/48` dc0, `f03::/48` dc1) on the `:10/:11/:20/:21/:30/:40/:50/:80` octet + map **VR0 DC0 and the Willamette site ALREADY use**, so this conforms to an existing org + standard rather than inventing one. **The ULA `/48` `fd50:840e:74e2::/48` is RETIRED FOR + VR1** (it stays a valid org aggregate, it just carries no VR1 plane) -- AMENDS D-101 and + D-111. Deciding reason is measurable, not aesthetic: RFC 6724's default policy table ranks + IPv4-mapped at precedence 35 and ULA at 3 while GUA falls under `::/0` at 40, so on a + DUAL-STACK plane a ULA leg LOSES address selection to IPv4 and is decorative, whereas GUA + WINS -- full GUA delivers D-101's "IPv6 unless IPv4 is necessary" with no per-host + `gai.conf` tuning. **SEC-010, D-052, D-125 and D-107 are UNCHANGED and nothing is + punctured** -- containment lives at the FORWARDING layer and the DC edge carries no v6 + transport, so GUA addressing creates no reachability without a route. + **THE RECORDED ROOT CAUSE OF THE v4-ONLY CONTAINERS WAS WRONG, AND THIS DOCUMENT CARRIED + IT (GA-R1 C2 -- measurement wins and corrects it here).** The entry above at ~1359-1373 + states that MAAS "has nothing to give for v6" and that "MAAS answers v4 from the dynamic + range". **MEASURED, and re-verified independently from the main session: + `subnet statistics` on `fd50:840e:74e2:220::/64` returns `available_string "100%"`, + `num_available 18446744069414584320`, `gateway_ip=None`.** Zero explicit `ipranges` rows + means zero RESTRICTIONS, not zero availability -- and the v4 container address + `10.12.8.122` sits OUTSIDE the dynamic range `.201-.254`, so the v4 half of the claim is + wrong too. **`scripts/dc-plane-ipam.sh:368-372` has carried the correct MAAS behaviour + since 2026-07-27**, in the tree, contradicting the authoritative document the whole time. + The apparent internal contradiction with ~551-553 is NOT a contradiction: the two entries + read DIFFERENT endpoints (`ipranges read` = explicit table rows, all v4; `subnet + reserved-ip-ranges` = MAAS's COMPUTED in-use set). The real defect is a SCOPE PROMOTION -- + "nothing to hand out" was true of `dhcpd6` and was silently carried to static/AUTO + allocation, where it is false. **CONSEQUENCE: adding v6 ranges is NEITHER NECESSARY NOR + SUFFICIENT, and the instruction "must be measured before any range is created" is + DISCHARGED -- the answer retires the range-creation idea entirely.** + **THE REAL MECHANISM IS JUJU-SIDE AND THERE IS NO KNOB.** `state/linklayerdevices.go` + (`EthernetDeviceForBridge`, pinned at tag `v3.6.27`) takes `addrs[0]` -- ONE address from an + UNSORTED mongo query -- and derives exactly one CIDR, which the MAAS provider turns into a + single `LinkSubnetArgs`. Zero family-awareness on the path; zero matching config keys. + gomaasapi's own doc says "Any number of STATIC links can exist on an interface", so **MAAS + would accept both families and the limit is juju's**. LP #1723240 (`Triaged`/`Low`, open + since 2017-10-12) is this exact symptom and the theory this document recorded is the one + its reporter rebutted in-thread. **DO NOT cite LP #1590598 as evidence dual-stack works -- + it is `Fix Released` about a different thing (v6-ONLY hosts), and citing it here would be + the inverted-citation class again.** CONSEQUENCE: **on a container plane, DUAL-STACK IS NOT + EXPRESSIBLE** (juju picks one family non-deterministically) while v6-ONLY is -- which makes + ruling A the achievable configuration, not merely the desired one. **CARRIED RISK, + reaffirmed by the operator after being put to them: `metal-admin` stays dual-stack AND + carries containers**, so its leg lands on one family by mongo order (today v4); a flip to + v6 would cut the container off from the v4-only mirror, snap proxy and MAAS region. + Mitigation is DETECTION, not config -- a Stage-5 gate asserting every container's + metal-admin leg is IPv4. **OWED, NOT BUILT.** + **BUILD CONSTRAINTS MEASURED BEFORE THE CARVE IS DRAFTED.** MAAS auto-reserves + `::1`-`::ffff:ffff` on EVERY IPv6 `/64`, so explicit STATICs inside it work (that is how the + 54 node statics live at `::100`) while AUTO draws from OUTSIDE -- **juju containers on + v6-only planes will get `:0:1::`-shaped addresses, NOT the D-134 textual mirror**, + and no v6 `ip-range` rows are needed or possible. All six v6 subnets carry + `gateway_ip=None`, correct for on-link east-west; the two exceptions are OPEN and NOT ruled + by D-139 -- `provider-public` (no v6 edge transport) and `replication`'s CROSS-DC leg. + **`ceph-mon`, `ceph-osd` and `mysql-innodb-cluster` are all in `PREFER_IPV6_CHARMS`**, so + storage/replication/metal-internal going v6-only REOPENS the 2026-07-31 D-101 ruling note + that set `prefer-ipv6` false everywhere -- per-app, as each plane converts. + **NOTHING OF D-139 IS EXECUTED. NO APEX PUSH, NO MAAS CARVE, NO NODE RE-CARVE.** The + ordered, individually-gated execution list is in D-139's final section. **UNVERIFIED AND + OWED BEFORE THE CARVE RUNS: the deployed jammy image's own `/etc/gai.conf`**, which can + override the RFC 6724 default table that ruling B's deciding reason rests on. - Project: Omega Cloud, VR1 DC-DC rehearsal -- a two-DC + Office1-headend virtual rehearsal on KVM (vcloud host), rehearsing the future bare-metal Roosevelt deployment (D-100, `docs/design-decisions.md:1946`). diff --git a/docs/audit/apex-ipv6-plan-20260731.txt b/docs/audit/apex-ipv6-plan-20260731.txt new file mode 100644 index 0000000..14127eb --- /dev/null +++ b/docs/audit/apex-ipv6-plan-20260731.txt @@ -0,0 +1,169 @@ +apex-ipv6-plan-20260731.txt +=========================== +READ-ONLY pull of the VR1 working IPAM apex (office1-netbox 10.10.1.10:8000, +DOCFIX-195). NOT the baldurkeep v1 reference. No write of any kind was made. +Puller: $CLAUDE_JOB_DIR/tmp/netbox-v6-plan.py (UA pinned curl/8.5.0 per the +documented 403 User-Agent trap). Host: vcloud. Date: 2026-07-31T23:08:39Z + +NOTE ON INSTRUMENT CURRENCY: the repo-carried export +netbox/draft/vr1-office1-current-20260725.json PREDATES the 2026-07-27 apex load +(IPv6 0 -> 78), so it is NOT a valid source for v6 questions. This is live. + +======================================================================= +APEX: http://10.10.1.10:8000 +============================================================================== + +--- AGGREGATES (5) --- + 10.0.0.0/8 rir=RFC 1918 RFC1918 private IPv4. + 23.157.124.0/24 rir=ARIN ARIN IPv4 allocation. + 172.16.0.0/12 rir=RFC 1918 RFC1918 private IPv4 (OOB space). + 2602:f3e2::/36 rir=ARIN ARIN IPv6 allocation: the HELD /36 = region- + fd50:840e:74e2::/48 rir=RFC 4193 ULA Org ULA /48 (internal-only; D-101 literal). + +--- PREFIXES: 139 total (36 v4, 103 v6) --- + + IPv6 prefixes: + 100::/64 site=- role=Discard vlan=- Discard Prefix + 2602:f3e2:200::/40 site=- role=Region vlan=- Second Region + 2602:f3e2:300::/40 site=- role=Region vlan=- Third Region + 2602:f3e2::/36 site=- role=Public vlan=- Public ARIN assignment to Abysius Inc + 2602:f3e2::/40 site=- role=Infrastructure vlan=- Inter-Regional Infrastructure + 2602:f3e2::/48 site=- role=Infrastructure vlan=- Inter-Regional Site to Site Links + 2602:f3e2:fe::/48 site=- role=- vlan=- Edge Networks + 2602:f3e2:ff::/48 site=- role=Openstack Tenant vlan=- Private cloud tenant reservation + ::1/128 site=- role=Localhost vlan=- Localhost + fc00::/8 site=- role=Private vlan=- Centrally assigned ULA (Not Active) + fd00::/8 site=- role=Private vlan=- Unique Local Addresses (ULA) + fd7a:115c:a1e0::/48 site=- role=VPN vlan=- Tailscale Addresses + fe80::/10 site=- role=Link-Local vlan=- Link-Local + ff00::/8 site=- role=Multicast vlan=- Public Multicast + 2602:f3e2:101:100::/56 site=Charnelton role=Office vlan=- Charnelton + 2602:f3e2:100::/40 site=Eugene role=Region vlan=- Eugene Region; Sites receive /24 allocat + 2602:f3e2:100::/48 site=Eugene role=Infrastructure vlan=- Eugene Regional Infrastructure + 2602:f3e2:100::/56 site=Eugene role=Infrastructure vlan=- Eugene Site to Site Links + 2602:f3e2:101::/48 site=Eugene role=Office vlan=- Eugene Regional Offices; Offices are all + 2602:f3e2:101::/56 site=Eugene role=Office vlan=- Reserved for fun and profit + 2602:f3e2:103::/48 site=Roosevelt role=Datacenter vlan=- Roosevelt Site + 2602:f3e2:e00:1::/127 site=VR0 role=Infrastructure vlan=- VR0 DC1 to DC0 Site to Site Link + 2602:f3e2:e00:2::/127 site=VR0 role=Infrastructure vlan=- VR0 Off0 to DC0 Site to Site Link + 2602:f3e2:e00::/40 site=VR0 role=Region vlan=- Virtual Region 0 (VR0) + 2602:f3e2:e00::/48 site=VR0 role=Infrastructure vlan=- Virtual Region 0 (VR0) Infrastructure + 2602:f3e2:e00::/56 site=VR0 role=Infrastructure vlan=- VR0 Site to Site Links + 2602:f3e2:e01::/48 site=VR0 role=Office vlan=- Virtual Region 0 (VR0) Offices + 2602:f3e2:e01::/56 site=VR0 role=Office vlan=- Virtual Region 0 (VR0) Offices; Reserved + 2602:f3e2:e02:10::/60 site=VR0 DC0 role=Provider vlan=- VR0 DC0 Omega Cloud Provider Prefix + 2602:f3e2:e02:10::/64 site=VR0 DC0 role=Provider vlan=- VR0 DC0 Omega Cloud Provider Subnet + 2602:f3e2:e02:11::/64 site=VR0 DC0 role=Provider vlan=- VR0 DC0 Omega Cloud — Provider API VIPs + 2602:f3e2:e02:20::/60 site=VR0 DC0 role=Metal vlan=- VR0 DC0 Omega Cloud Metal Prefix + 2602:f3e2:e02:20::/64 site=VR0 DC0 role=Metal vlan=- VR0 DC0 Omega Cloud Metal Subnet + 2602:f3e2:e02:30::/60 site=VR0 DC0 role=Data vlan=- VR0 DC0 Omega Cloud Data Prefix + 2602:f3e2:e02:30::/64 site=VR0 DC0 role=Data vlan=- VR0 DC0 Omega Cloud Data Subnet + 2602:f3e2:e02:40::/60 site=VR0 DC0 role=Storage vlan=- VR0 DC0 Omega Cloud Storage Prefix + 2602:f3e2:e02:40::/64 site=VR0 DC0 role=Storage vlan=- VR0 DC0 Omega Cloud Storage Subnet + 2602:f3e2:e02:50::/60 site=VR0 DC0 role=Replication vlan=- VR0 DC0 Omega Cloud Replication Prefix + 2602:f3e2:e02:50::/64 site=VR0 DC0 role=Replication vlan=- VR0 DC0 Omega Cloud Replication Subnet + 2602:f3e2:e02:80::/60 site=VR0 DC0 role=LBaaS Management vlan=- VR0 DC0 Omega Cloud LBaaS Management Pre + 2602:f3e2:e02:80::/64 site=VR0 DC0 role=LBaaS Management vlan=- VR0 DC0 Omega Cloud LBaaS Management Sub + 2602:f3e2:e02::/48 site=VR0 DC0 role=Datacenter vlan=- Virtual Region 0 (VR0) Datacenter 0 (DC0 + 2602:f3e2:e02::/56 site=VR0 DC0 role=Cloud vlan=- VR0 DC0 Omega Cloud + 2602:f3e2:e02:e0::/60 site=VR0 DC0 role=VPN vlan=- VR0 DC0 Omega Cloud VPN Prefix + 2602:f3e2:e02:f0::/60 site=VR0 DC0 role=OOB vlan=- VR0 DC0 Omega Cloud OOB Prefix + 2602:f3e2:e02:f0::/64 site=VR0 DC0 role=OOB vlan=- VR0 DC0 Omega Cloud OOB Subnet + 2602:f3e2:e03::/48 site=VR0 DC1 role=Datacenter vlan=- Virtual Region 0 (VR0) Datacenter 1 (DC1 + 2602:f3e2:e01:100::/56 site=VR0 Off0 role=Office vlan=- Virtual Region 0 (VR0) Office 0 (Off0) + 2602:f3e2:e01:100::/64 site=VR0 Off0 role=Office vlan=- VR0 Off0 Office Subnet + 2602:f3e2:f00::/40 site=VR1 role=Region vlan=- Virtual Region 1 (VR1) + 2602:f3e2:f00::/48 site=VR1 role=Infrastructure vlan=- Virtual Region 1 (VR1) Infrastructure + 2602:f3e2:f00::/56 site=VR1 role=Infrastructure vlan=- VR1 Site to Site Links + 2602:f3e2:f01::/48 site=VR1 role=Office vlan=- Virtual Region 1 (VR1) Offices + 2602:f3e2:f01::/56 site=VR1 role=Office vlan=- Virtual Region 1 (VR1) Offices; Reserved + 2602:f3e2:f02:10::/60 site=VR1 DC0 role=Provider Public vlan=- VR1 DC0 provider-public (GUA /60; D-101/ + 2602:f3e2:f02:10::/64 site=VR1 DC0 role=Provider Public vlan=- VR1 DC0 provider-public (GUA /64 active; + 2602:f3e2:f02:11::/64 site=VR1 DC0 role=Provider Public vlan=- VR1 DC0 provider-public (GUA VIP /64; D- + 2602:f3e2:f02::/48 site=VR1 DC0 role=Datacenter vlan=- Virtual Region 1 (VR1) Datacenter 0 (DC0 + fd50:840e:74e2:220::/60 site=VR1 DC0 role=Metal Admin vlan=- VR1 DC0 metal-admin (ULA metal /60 (admi + fd50:840e:74e2:220::/64 site=VR1 DC0 role=Metal Admin vlan=- VR1 DC0 metal-admin (ULA /64 active; D-1 + fd50:840e:74e2:221::/64 site=VR1 DC0 role=Metal Internal vlan=- VR1 DC0 metal-internal (ULA /64 active ( + fd50:840e:74e2:230::/60 site=VR1 DC0 role=Data Tenant vlan=- VR1 DC0 data-tenant (ULA /60; D-101/D-11 + fd50:840e:74e2:230::/64 site=VR1 DC0 role=Data Tenant vlan=- VR1 DC0 data-tenant (ULA /64 active; D-1 + fd50:840e:74e2:240::/60 site=VR1 DC0 role=Storage vlan=- VR1 DC0 storage (ULA /60; D-101/D-111) + fd50:840e:74e2:240::/64 site=VR1 DC0 role=Storage vlan=- VR1 DC0 storage (ULA /64 active; D-101/D + fd50:840e:74e2:250::/60 site=VR1 DC0 role=Replication Plane vlan=- VR1 DC0 replication (ULA /60; D-101/D-11 + fd50:840e:74e2:250::/64 site=VR1 DC0 role=Replication Plane vlan=- VR1 DC0 replication (ULA /64 active; D-1 + 2602:f3e2:f03:10::/60 site=VR1 DC1 role=Provider Public vlan=- VR1 DC1 provider-public (GUA /60; D-101/ + 2602:f3e2:f03:10::/64 site=VR1 DC1 role=Provider Public vlan=- VR1 DC1 provider-public (GUA /64 active; + 2602:f3e2:f03:11::/64 site=VR1 DC1 role=Provider Public vlan=- VR1 DC1 provider-public (GUA VIP /64; D- + 2602:f3e2:f03::/48 site=VR1 DC1 role=Datacenter vlan=- Virtual Region 1 (VR1) Datacenter 1 (DC1 + fd50:840e:74e2:320::/60 site=VR1 DC1 role=Metal Admin vlan=- VR1 DC1 metal-admin (ULA metal /60 (admi + fd50:840e:74e2:320::/64 site=VR1 DC1 role=Metal Admin vlan=- VR1 DC1 metal-admin (ULA /64 active; D-1 + fd50:840e:74e2:321::/64 site=VR1 DC1 role=Metal Internal vlan=- VR1 DC1 metal-internal (ULA /64 active ( + fd50:840e:74e2:330::/60 site=VR1 DC1 role=Data Tenant vlan=- VR1 DC1 data-tenant (ULA /60; D-101/D-11 + fd50:840e:74e2:330::/64 site=VR1 DC1 role=Data Tenant vlan=- VR1 DC1 data-tenant (ULA /64 active; D-1 + fd50:840e:74e2:340::/60 site=VR1 DC1 role=Storage vlan=- VR1 DC1 storage (ULA /60; D-101/D-111) + fd50:840e:74e2:340::/64 site=VR1 DC1 role=Storage vlan=- VR1 DC1 storage (ULA /64 active; D-101/D + fd50:840e:74e2:350::/60 site=VR1 DC1 role=Replication Plane vlan=- VR1 DC1 replication (ULA /60; D-101/D-11 + fd50:840e:74e2:350::/64 site=VR1 DC1 role=Replication Plane vlan=- VR1 DC1 replication (ULA /64 active; D-1 + 2602:f3e2:f01:100::/56 site=VR1 Off1 role=Office vlan=- VR1 Off1 (D-115) -- mirrors VR0 Off0 e01 + 2602:f3e2:f01:100::/64 site=VR1 Off1 role=Office vlan=- VR1 Off1 office subnet -- NOT YET DEPLOY + 2602:f3e2:102:100::/56 site=Willamette role=Cloud vlan=- Psi Cloud + 2602:f3e2:102:10::/60 site=Willamette role=Provider vlan=- + 2602:f3e2:102:10::/64 site=Willamette role=Provider vlan=- + 2602:f3e2:102:11::/64 site=Willamette role=Provider vlan=- Willamette Omega Cloud — Provider API VI + 2602:f3e2:102:1600::/56 site=Willamette role=Cloud vlan=- Beta Cloud + 2602:f3e2:102:1700::/56 site=Willamette role=Cloud vlan=- Alpha Cloud + 2602:f3e2:102:20::/60 site=Willamette role=Metal vlan=- + 2602:f3e2:102:20::/64 site=Willamette role=Metal vlan=- + 2602:f3e2:102:30::/60 site=Willamette role=Data vlan=- + 2602:f3e2:102:30::/64 site=Willamette role=Data vlan=- + 2602:f3e2:102:40::/60 site=Willamette role=Storage vlan=- + 2602:f3e2:102:40::/64 site=Willamette role=Storage vlan=- + 2602:f3e2:102:50::/60 site=Willamette role=Replication vlan=- + 2602:f3e2:102:50::/64 site=Willamette role=Replication vlan=- + 2602:f3e2:102:80::/60 site=Willamette role=LBaaS Management vlan=- + 2602:f3e2:102:80::/64 site=Willamette role=LBaaS Management vlan=- + 2602:f3e2:102::/48 site=Willamette role=Datacenter vlan=- Willamette Site + 2602:f3e2:102::/56 site=Willamette role=Cloud vlan=- Omega Cloud + 2602:f3e2:102:e0::/60 site=Willamette role=VPN vlan=- + 2602:f3e2:102:f0::/60 site=Willamette role=OOB vlan=- + 2602:f3e2:102:f0::/64 site=Willamette role=OOB vlan=- + +--- IP-RANGES: 27 total (27 v4, 0 v6) --- + (NO IPv6 ip-ranges in the apex) + + IPv4 ranges (for contrast, first 30): + 10.10.1.2/24 -> 10.10.1.49/24 role=- D-120 static site-services band (.2-.49) + 10.10.1.100/24 -> 10.10.1.200/24 role=- D-120 MAAS dynamic band (.100-.200) -- e + 10.10.1.201/24 -> 10.10.1.254/24 role=- D-120 deployed-node band (.201-.254) -- + 10.12.4.4/22 -> 10.12.4.49/22 role=- D-134 utility band (vr1-dc0 provider-pub + 10.12.4.50/22 -> 10.12.4.99/22 role=- D-134 VIP band (vr1-dc0 provider-public) + 10.12.8.4/22 -> 10.12.8.49/22 role=- D-134 utility band (vr1-dc0 metal-admin) + 10.12.8.50/22 -> 10.12.8.99/22 role=- D-134 VIP band (vr1-dc0 metal-admin) -- + 10.12.12.4/22 -> 10.12.12.49/22 role=- D-134 utility band (vr1-dc0 metal-intern + 10.12.12.50/22 -> 10.12.12.99/22 role=- D-134 VIP band (vr1-dc0 metal-internal) + 10.12.16.4/22 -> 10.12.16.49/22 role=- D-134 utility band (vr1-dc0 data-tenant) + 10.12.16.50/22 -> 10.12.16.99/22 role=- D-134 VIP band (vr1-dc0 data-tenant) -- + 10.12.32.4/22 -> 10.12.32.49/22 role=- D-134 utility band (vr1-dc0 storage) -- + 10.12.32.50/22 -> 10.12.32.99/22 role=- D-134 VIP band (vr1-dc0 storage) -- .50- + 10.12.36.4/22 -> 10.12.36.49/22 role=- D-134 utility band (vr1-dc0 replication) + 10.12.36.50/22 -> 10.12.36.99/22 role=- D-134 VIP band (vr1-dc0 replication) -- + 10.12.64.4/22 -> 10.12.64.49/22 role=- D-134 utility band (vr1-dc1 provider-pub + 10.12.64.50/22 -> 10.12.64.99/22 role=- D-134 VIP band (vr1-dc1 provider-public) + 10.12.68.4/22 -> 10.12.68.49/22 role=- D-134 utility band (vr1-dc1 metal-admin) + 10.12.68.50/22 -> 10.12.68.99/22 role=- D-134 VIP band (vr1-dc1 metal-admin) -- + 10.12.72.4/22 -> 10.12.72.49/22 role=- D-134 utility band (vr1-dc1 metal-intern + 10.12.72.50/22 -> 10.12.72.99/22 role=- D-134 VIP band (vr1-dc1 metal-internal) + 10.12.76.4/22 -> 10.12.76.49/22 role=- D-134 utility band (vr1-dc1 data-tenant) + 10.12.76.50/22 -> 10.12.76.99/22 role=- D-134 VIP band (vr1-dc1 data-tenant) -- + 10.12.80.4/22 -> 10.12.80.49/22 role=- D-134 utility band (vr1-dc1 storage) -- + 10.12.80.50/22 -> 10.12.80.99/22 role=- D-134 VIP band (vr1-dc1 storage) -- .50- + 10.12.84.4/22 -> 10.12.84.49/22 role=- D-134 utility band (vr1-dc1 replication) + 10.12.84.50/22 -> 10.12.84.99/22 role=- D-134 VIP band (vr1-dc1 replication) -- + +--- IP-ADDRESSES: 160 total, 78 IPv6 --- + IPv6 grouped by prefix head: + 2602:f3e2:f02:11::/64 13 + 2602:f3e2:f03:11::/64 13 + fd50:840e:74e2:220::/64 13 + fd50:840e:74e2:221::/64 13 + fd50:840e:74e2:320::/64 13 + fd50:840e:74e2:321::/64 13 diff --git a/docs/audit/dc0-snap-proxy-install-20260731.txt b/docs/audit/dc0-snap-proxy-install-20260731.txt new file mode 100644 index 0000000..9b96807 --- /dev/null +++ b/docs/audit/dc0-snap-proxy-install-20260731.txt @@ -0,0 +1,739 @@ +dc0 SNAP FORWARD PROXY -- FIRST LIVE INSTALL + VERIFICATION CAPTURE +Date: 2026-07-31 +Site: dc0 (vr1-dc0), rack host vr1-dc0-rack = 172.31.0.2 (via voffice1) +Tool: scripts/dc-snap-proxy.sh (D-135/D-107 RULING 2, 2026-07-31) +Operator: jessea123 @ jumphost vcloud, branch dc-dc-stage5-preconditions + +PURPOSE. The tool had NEVER been applied. Its harness green was FIXTURE green. +This capture is the first end-to-end application on a real rack, and it is the +run that resolves the BUG-3 open question (section 6). + +All output below is VERBATIM. Nothing is paraphrased. + +=========================================================================== +SECTION 0 -- BASE CONFIRMATION (jumphost, before touching the rack) +=========================================================================== + +$ bash tests/dc-snap-proxy/run-tests.sh + [...] + PASS T50 node body is client-satisfiable only (no rack-side calls) + +dc-snap-proxy: 55 passed, 0 failed + +$ bash scripts/repo-lint.sh + [WARN] L1 docs/design-decisions.md: 239 non-ASCII byte(s) (legacy D-001..018 carve-out; NEW entries must be ASCII) + +WARN: repo lint (0 fail, 1 warn, 641 files scanned) + +Both match the expected base (55/55; 0 fail, 1 warn). + +$ sha256sum scripts/dc-snap-proxy.sh tests/dc-snap-proxy/run-tests.sh +318cfb4190eccdfe1d760ac3a7f8f1a23ac2b35e78d07167d730bef7150439f0 scripts/dc-snap-proxy.sh +aaa2a2060e770720f25284f9413b8c222e2d3b073a6f5ab2c6daad642fc0f8ff tests/dc-snap-proxy/run-tests.sh + +Recorded as the pre-edit baseline so any restore-verify is measured, not +reconstructed. Both files were already MODIFIED-UNCOMMITTED in the working tree +on arrival (the earlier BUG-1/BUG-3/BUG-4 fix work); HEAD is 2208141. + +TRANSIT VALUE PROVENANCE (hard rule 2 -- not from memory): + scripts/lib-hosts.sh, vr1-dc0 arm: + VIRSH_POWER_ADDRESS_FROM_OFFICE1="qemu+ssh://jessea123@172.31.0.2/system" + ~/.ssh/config.d/vr1-sites: + Host vr1-dc0-rack 172.31.0.2 # dc0 rack host (libvirt + MAAS rack + D-135 mirror) + HostName 172.31.0.2 + User jessea123 + ProxyJump voffice1 + The two agree. 172.31.0.2 is used below only via the alias. + +=========================================================================== +SECTION 1 -- BEFORE-STATE: read-only `check dc0` with NOTHING built +=========================================================================== + +This proves the gate can FAIL on a real rack and not merely on a fixture. + +$ ssh vr1-dc0-rack 'sudo bash -s' -- check dc0 < scripts/dc-snap-proxy.sh; echo "EXIT=$?" +dc-snap-proxy check (dc0) -- 10.12.8.4:3129, clients 10.12.8.0/22 + MISS squid binary absent at /usr/sbin/squid -- run: apt-get update && apt-get install -y squid + MISS /etc/squid/dc0-snap-proxy.conf absent + MISS /etc/systemd/system/dc0-snap-proxy.service absent + MISS /usr/local/sbin/dc0-snap-proxy-net-apply absent + MISS /etc/systemd/system/dc0-snap-proxy-net.service absent + MISS /etc/systemd/resolved.conf.d/dc0-snap-proxy.conf absent + MISS dc0-snap-proxy.service not enabled (it would not survive a reboot) + MISS dc0-snap-proxy-net.service not enabled (it would not survive a reboot) + MISS dc0-snap-proxy.service not active + OK packaged squid.service neither active nor enabled + MISS nothing listening on 10.12.8.4:3129 + OK 10.12.8.4/22 on virbr2 (vr1-dc0-metal-admin) + OK default route via edge 10.12.4.1 + REFUSE could not reach https://api.snapcraft.io/v2/snaps/info/core THROUGH http://10.12.8.4:3129 (curl exit 7: curl: (7) Failed to connect to 10.12.8.4 port 3129 after 0 ms: Couldn't connect to server) -- unreachable is not 'nothing there' + REFUSE CDN probe gave curl exit 7 with 'curl: (7) Failed to connect to 10.12.8.4 port 3129 after 0 ms: Couldn't connect to server +000' -- neither an allowed tunnel nor a proxy denial; refusing to interpret it + REFUSE deny probe gave curl exit 7 with 'curl: (7) Failed to connect to 10.12.8.4 port 3129 after 0 ms: Couldn't connect to server +000' -- neither a 403 denial nor a success; the allowlist state is UNKNOWN +dc-snap-proxy check (dc0): FAIL +EXIT=1 + +WHY THIS IS NOT THE "WRONG HOST" TRAP (the recorded per-DC-script failure mode +where running from the jumphost measures the wrong filesystem and reports +uniform total failure). The result is NOT uniform. Three assertions came back +OK, and they are exactly the ones only the real rack can satisfy: + - "10.12.8.4/22 on virbr2 (vr1-dc0-metal-admin)" -- requires virsh to resolve + the libvirt network NAME to a bridge; on the jumphost this would REFUSE. + - "default route via edge 10.12.4.1" -- the rack's route table, not ours. + - "packaged squid.service neither active nor enabled". +An implausibly uniform result would itself have been the tell. It was not +uniform, so the instrument was current and pointed at the right host. + +=========================================================================== +SECTION 2 -- PRE-MUTATION STATE OF THE NET LAYER AND THE CLAIMED PORT +=========================================================================== + +Taken BEFORE install so the header's "the three net units set IDENTICAL +MEASURED values, so they are no-ops for each other" claim becomes testable +rather than merely asserted. + +$ ssh vr1-dc0-rack 'ip -4 -o addr show dev virbr2; ip route show default; systemctl is-active dc0-mirror-net.service dc0-cache-proxy-net.service; sudo ss -ltnp | grep -E ":(3128|3129|8000) "; ls -l /usr/sbin/squid' +=== ip -4 -o addr show dev virbr2 === +7: virbr2 inet 10.12.8.2/22 scope global virbr2\ valid_lft forever preferred_lft forever +7: virbr2 inet 10.12.8.3/22 scope global secondary virbr2\ valid_lft forever preferred_lft forever +7: virbr2 inet 10.12.8.4/22 scope global secondary virbr2\ valid_lft forever preferred_lft forever +=== ip route show default === +default via 10.12.4.1 dev virbr5 +=== mirror/cache net units === +active +inactive +=== ss -ltnp ports 3128/3129/8000 === +LISTEN 0 256 *:8000 *:* users:(("squid",pid=1871633,fd=13)) +LISTEN 0 256 *:3128 *:* users:(("squid",pid=1871633,fd=12)) +=== squid binary === +ls: cannot access '/usr/sbin/squid': No such file or directory + +READS: + - The utility .4 (10.12.8.4/22) was ALREADY present on virbr2 as a secondary, + placed by dc0-mirror-net.service (active). dc0-cache-proxy-net.service is + inactive on this rack. + - MAAS's squid holds *:3128 AND *:8000 WILDCARD-bound, both fds of ONE process + (pid 1871633). Confirms the header's measured claim on this rack, today. + - PORT 3129 IS FREE -- nothing listening. The port this build claims is + unclaimed, verified before mutating. + - squid binary genuinely absent (so section 1's MISS was an OBSERVED absence). + +=========================================================================== +SECTION 3 -- THE PACKAGE (installed by hand; `install` deliberately never does) +=========================================================================== + +$ ssh vr1-dc0-rack 'sudo apt-get update && sudo apt-get install -y squid' +[...] +Restarting services... + +Service restarts being deferred: + /etc/needrestart/restart.d/dbus.service + systemctl restart getty@tty1.service + systemctl restart libvirtd.service + systemctl restart systemd-logind.service + systemctl restart unattended-upgrades.service + systemctl restart virtlockd.service + systemctl restart virtlogd.service + +No containers need to be restarted. + +VM guests are running outdated hypervisor (qemu) binaries on this host: + 'vr1-dc0-opnsense' with pid 44740 +Pending kernel upgrade +---------------------- + +Newer kernel available + +The currently running kernel version is 6.8.0-134-generic which is not the +expected kernel version 6.8.0-136-generic. + +Restarting the system to load the new kernel will not be handled automatically, +so you should consider rebooting. + +VERSION ACTUALLY INSTALLED: + +$ ssh vr1-dc0-rack 'dpkg-query -W -f="${Version}\n" squid; /usr/sbin/squid -v | head -1; ls -l /usr/sbin/squid' +=== dpkg-query === +6.14-0ubuntu0.24.04.4 +=== squid -v === +Squid Cache: Version 6.14 +=== binary === +-rwxr-xr-x 1 root root 6989912 Jun 13 14:31 /usr/sbin/squid + +This is EXACTLY the version the script header names as noble MAIN and +security-supported ("6.14-0ubuntu0.24.04.4"). The header's package claim is now +MEASURED on this rack, not just cited. + +--- FINDING 3a: the packaged unit's FATAL was OBSERVED, not just predicted ---- + +The header argues from reasoning that the packaged squid.service "would FATAL on +bind" because Ubuntu noble pins http_port 3128 and MAAS's squid holds it +wildcard-bound. Installing the package ran that experiment for real: + +$ ssh vr1-dc0-rack 'systemctl is-active squid; systemctl is-enabled squid' +failed +enabled + +$ ssh vr1-dc0-rack 'sudo journalctl -u squid --no-pager -n 20' +Jul 31 23:01:31 vvr1-dc0 squid[2914603]: storeDirWriteCleanLogs: Starting... + listening port: 3128 +Jul 31 23:01:31 vvr1-dc0 squid[2914603]: Finished. Wrote 0 entries. + listening port: 3128 +Jul 31 23:01:31 vvr1-dc0 squid[2914603]: Took 0.00 seconds ( 0.00 entries/sec). + listening port: 3128 +Jul 31 23:01:31 vvr1-dc0 squid[2914603]: FATAL: Unable to open HTTP Socket + listening port: 3128 +Jul 31 23:01:31 vvr1-dc0 squid[2914603]: Squid Cache (Version 6.14): Terminated abnormally. + listening port: 3128 +Jul 31 23:01:31 vvr1-dc0 squid[2914603]: Closing Pinger socket on FD 14 + listening port: 3128 +Jul 31 23:01:31 vvr1-dc0 squid[2914584]: Squid Parent: squid-1 process 2914603 exited with status 1 +Jul 31 23:01:31 vvr1-dc0 squid[2914584]: Squid Parent: squid-1 process 2914603 will not be restarted for 3600 seconds due to repeated, frequent failures +Jul 31 23:01:31 vvr1-dc0 squid[2914584]: Exiting due to repeated, frequent failures +Jul 31 23:01:31 vvr1-dc0 squid[2914584]: Removing PID file (/run/squid.pid) +Jul 31 23:01:31 vvr1-dc0 systemd[1]: squid.service: Main process exited, code=exited, status=1/FAILURE +Jul 31 23:01:31 vvr1-dc0 systemd[1]: squid.service: Killing process 2914605 (pinger) with signal SIGKILL. +Jul 31 23:01:31 vvr1-dc0 systemd[1]: squid.service: Failed with result 'exit-code'. +Jul 31 23:01:31 vvr1-dc0 systemd[1]: Failed to start squid.service - Squid Web Proxy Server. + +"FATAL: Unable to open HTTP Socket / listening port: 3128" is the header's +predicted failure, observed verbatim. The dedicated-instance design decision is +therefore MEASURED-CORRECT on this rack and not merely well-argued. It also +shows the window is real: between apt install and `install dc0`, the rack DID +carry a failed, still-ENABLED squid.service that would fight for 3128 at the +next boot -- which is precisely what `install`'s `systemctl disable --now squid` +and check's packaged-unit assertion exist to close. + +=========================================================================== +SECTION 4 -- `install dc0` (the first ever application of this tool) +=========================================================================== + +$ ssh vr1-dc0-rack 'sudo bash -s' -- install dc0 < scripts/dc-snap-proxy.sh; echo "EXIT=$?" +dc-snap-proxy install (dc0): done -- running check: +dc-snap-proxy check (dc0) -- 10.12.8.4:3129, clients 10.12.8.0/22 + OK squid binary present (/usr/sbin/squid) + OK /etc/squid/dc0-snap-proxy.conf matches generated + OK /etc/systemd/system/dc0-snap-proxy.service matches generated + OK /usr/local/sbin/dc0-snap-proxy-net-apply matches generated + OK /etc/systemd/system/dc0-snap-proxy-net.service matches generated + OK /etc/systemd/resolved.conf.d/dc0-snap-proxy.conf matches generated + OK dc0-snap-proxy.service enabled + OK dc0-snap-proxy-net.service enabled + OK dc0-snap-proxy.service active + OK packaged squid.service neither active nor enabled + OK listening on 10.12.8.4:3129 + OK 10.12.8.4/22 on virbr2 (vr1-dc0-metal-admin) + OK default route via edge 10.12.4.1 + OK CONNECT to the store API: curl exit 0, HTTP 200, channel-map + sha3-384 present + OK CDN tunnel allowed (curl exit 0; the CDN's own status was '403') + OK non-allowlisted destination is DENIED (403 at CONNECT) -- the allowlist is in force +dc-snap-proxy check (dc0): PASS +EXIT=0 + +Install succeeded on the first attempt with no intervention, and its trailing +self-check PASSED. Note the packaged-unit line flipped from the section-3 +"failed + enabled" state to "neither active nor enabled" -- install's +`systemctl disable --now squid` did its job, observed rather than assumed. + +--- POST-INSTALL NET-LAYER COMPARISON (the header's no-op claim, tested) ------ + +The header claims the three per-DC net units "set IDENTICAL MEASURED values with +`ip addr replace` / `ip route replace`, so they are no-ops for each other". dc0 +already ran dc0-mirror-net.service, which owns the same .4 alias and the same +default route, so `install` was a live test of that claim. + +$ ssh vr1-dc0-rack 'ip -4 -o addr show dev virbr2; ip route show default; systemctl is-active dc0-mirror-net.service dc0-snap-proxy-net.service; sudo ss -ltnp | grep -E ":(3128|3129|8000) "' +=== POST-INSTALL ip -4 -o addr show dev virbr2 === +7: virbr2 inet 10.12.8.2/22 scope global virbr2\ valid_lft forever preferred_lft forever +7: virbr2 inet 10.12.8.3/22 scope global secondary virbr2\ valid_lft forever preferred_lft forever +7: virbr2 inet 10.12.8.4/22 scope global secondary virbr2\ valid_lft forever preferred_lft forever +=== POST-INSTALL ip route show default === +default via 10.12.4.1 dev virbr5 +=== net units === +active +active +=== ports === +LISTEN 0 256 10.12.8.4:3129 0.0.0.0:* users:(("squid",pid=2916382,fd=10)) +LISTEN 0 256 *:8000 *:* users:(("squid",pid=1871633,fd=13)) +LISTEN 0 256 *:3128 *:* users:(("squid",pid=1871633,fd=12)) + +RESULT -- the header's claim HOLDS, measured: + - virbr2's address list is BYTE-IDENTICAL to section 2. No address added, + removed or reordered. The .4 alias was already there and stayed there. + - The default route is unchanged (`via 10.12.4.1 dev virbr5`). + - dc0-mirror-net.service is STILL active alongside dc0-snap-proxy-net.service. + The two coexist; neither displaced the other. + - MAAS's squid is STILL pid 1871633 -- the SAME pid as before install. It was + not restarted, not disturbed, and still holds *:3128 and *:8000. The + coexistence claim is measured, not reasoned. + - Our squid is a SEPARATE process (pid 2916382) bound to 10.12.8.4:3129 + SPECIFICALLY, not wildcard, exactly as generated. + +=========================================================================== +SECTION 5 -- standalone `check dc0` (a separate invocation, not install's) +=========================================================================== + +$ ssh vr1-dc0-rack 'sudo bash -s' -- check dc0 < scripts/dc-snap-proxy.sh; echo "EXIT=$?" +dc-snap-proxy check (dc0) -- 10.12.8.4:3129, clients 10.12.8.0/22 + OK squid binary present (/usr/sbin/squid) + OK /etc/squid/dc0-snap-proxy.conf matches generated + OK /etc/systemd/system/dc0-snap-proxy.service matches generated + OK /usr/local/sbin/dc0-snap-proxy-net-apply matches generated + OK /etc/systemd/system/dc0-snap-proxy-net.service matches generated + OK /etc/systemd/resolved.conf.d/dc0-snap-proxy.conf matches generated + OK dc0-snap-proxy.service enabled + OK dc0-snap-proxy-net.service enabled + OK dc0-snap-proxy.service active + OK packaged squid.service neither active nor enabled + OK listening on 10.12.8.4:3129 + OK 10.12.8.4/22 on virbr2 (vr1-dc0-metal-admin) + OK default route via edge 10.12.4.1 + OK CONNECT to the store API: curl exit 0, HTTP 200, channel-map + sha3-384 present + OK CDN tunnel allowed (curl exit 0; the CDN's own status was '403') + OK non-allowlisted destination is DENIED (403 at CONNECT) -- the allowlist is in force +dc-snap-proxy check (dc0): PASS +EXIT=0 + +PASS, exit 0. This gate has now been observed to do BOTH things GA-R6 requires: +it FAILED on a real rack with nothing built (section 1, exit 1) and it PASSED on +a real rack with everything built (here, exit 0). It is no longer fixture-only. + +=========================================================================== +SECTION 6 -- BUG-3 RESOLVED: the dstdomain deny shape, MEASURED +=========================================================================== + +THE OPEN QUESTION (script header, lines 110-120). check's DENY probe asserts the +literal 'CONNECT tunnel failed, response 403'. That shape was measured from a +PORT deny (`deny CONNECT !SSL_ports`) and had NEVER been measured from the +DSTDOMAIN deny (`deny !snap_store`) that the probe actually exercises. If squid +answered a dstdomain denial differently, the branch would fall to REFUSE and +`check` could never PASS -- which under GA-R6 blocks a stage close as hard as a +gate that cannot fail. + +The say-line in sections 4 and 5 already indicates the expected branch was hit, +but a say-line is the script's interpretation, not the measurement. So the probe +was ALSO run BY HAND, with stderr unmerged and then merged, to record the raw +shape: + +$ ssh vr1-dc0-rack 'curl -sS -o /dev/null -w "HTTPCODE=%{http_code}\n" -x http://10.12.8.4:3129 -m 20 https://example.com/ ; echo "RC=$?"' +=== deny probe, stderr UNMERGED === +HTTPCODE=000 +RC=56 +curl: (56) CONNECT tunnel failed, response 403 + +$ ssh vr1-dc0-rack 'out="$(curl -sS -o /dev/null -w "%{http_code}" -x http://10.12.8.4:3129 -m 20 https://example.com/ 2>&1)"; rc=$?; echo "RC=$rc"; printf "%s\n" "$out"' +=== deny probe, 2>&1 merged (the blob the script matches) === +RC=56 +BLOB_BEGIN +curl: (56) CONNECT tunnel failed, response 403 +000 +BLOB_END + +SQUID'S OWN VIEW of the same events: + +$ ssh vr1-dc0-rack 'sudo grep DENIED /var/log/squid/dc0-snap-proxy.access.log' +1785539007.465 0 10.12.8.2 TCP_DENIED/403 3400 CONNECT example.com:443 - HIER_NONE/- text/html +1785539016.760 0 10.12.8.2 TCP_DENIED/403 3400 CONNECT example.com:443 - HIER_NONE/- text/html +1785539016.767 0 10.12.8.2 TCP_DENIED/403 3400 CONNECT example.com:443 - HIER_NONE/- text/html + +>>> BUG-3 RULING: CONFIRMED AS EXPECTED. NO CODE CHANGE MADE. <<< + +The DSTDOMAIN deny produces EXACTLY the same client-visible shape as the PORT +deny from which the assertion was originally written: + curl exit code 56 + message "CONNECT tunnel failed, response 403" + http_code 000 +and squid records it as TCP_DENIED/403 ... HIER_NONE. The assertion in +scripts/dc-snap-proxy.sh is therefore CORRECT AS WRITTEN and was left untouched. + +This is the result BUG-3 was waiting for. The reason the shapes coincide is that +curl reports on the CONNECT RESPONSE STATUS, which squid renders as 403 for any +`http_access deny` regardless of which ACL fired; the deny reason is squid-side +(X-Squid-Error / the access log), not encoded in the status line. The BUG-3 +protocol's step (b)/(c) -- re-point the assertion, add a harness case, run a +mutation -- were therefore NOT triggered and were correctly not performed. +scripts/dc-snap-proxy.sh is byte-identical to its pre-run state; see section 9. + +=========================================================================== +SECTION 7 -- CLIENT-SIDE (`node`) VERIFICATION, from a real remote client +=========================================================================== + +WHICH CLIENT, AND WHY -- the honest limit stated up front. The IDEAL client is a +charm LXD container, because the header's whole CLIENT_CIDR argument rests on +the finding that containers source from metal-admin. That client DOES NOT EXIST +right now, and this is MEASURED, not assumed: + +$ ssh vr1-dc0-rack 'sudo virsh -c qemu:///system list --all' + Id Name State +-------------------------------------- + 16 vr1-dc0-opnsense running + 82 vr1-dc0-maas-01 running + 113 vr1-dc0-juju-01 running + - vr1-dc0-compute-01 shut off + - vr1-dc0-compute-02 shut off + - vr1-dc0-control-01 shut off + - vr1-dc0-control-02 shut off + - vr1-dc0-control-03 shut off + - vr1-dc0-storage-01 shut off + - vr1-dc0-storage-02 shut off + - vr1-dc0-storage-03 shut off + - vr1-dc0-storage-04 shut off + +All NINE dc0 OpenStack nodes are SHUT OFF (the prior session's Stage 5 teardown, +commit 2208141). With no node powered there is no juju model and no charm +container, so the container-sourced proof is UNAVAILABLE this session -- not +"absent", UNAVAILABLE. Powering a node is outside this mandate. + +The BEST AVAILABLE real client is the dc0 MAAS region VM, which sits ON the +metal-admin plane (10.12.8.0/22 = CLIENT_CIDR) and reaches the proxy ACROSS the +bridge, exactly as a charm container would -- unlike the rack, which reaches the +LISTEN address locally. Its source address was measured first, per the script's +own instruction: + +$ ssh vr1-dc0-maas 'hostname; ip route get 10.12.8.4; command -v curl' +=== hostname / client identity === +hot-kid +=== ip route get 10.12.8.4 === +10.12.8.4 dev enp1s0 src 10.12.8.6 uid 1000 + cache +=== curl present? === +/usr/bin/curl + +src 10.12.8.6, via enp1s0, inside CLIENT_CIDR 10.12.8.0/22. A genuine remote +client on the client plane. + +$ ssh vr1-dc0-maas 'bash -s' -- node dc0 < scripts/dc-snap-proxy.sh; echo "EXIT=$?" +dc-snap-proxy node check (dc0) -- from this client, against 10.12.8.4:3129 + OK CONNECT to the store API through the proxy: curl exit 0, HTTP 200, channel-map + sha3-384 present + OK non-allowlisted destination is DENIED from this client -- the allowlist is in force + FAIL no https_proxy in /etc/environment -- the juju keys have not reached this client; run: juju model-config -m snap-http-proxy=http://10.12.8.4:3129 snap-https-proxy=http://10.12.8.4:3129 +dc-snap-proxy node (dc0): FAIL +EXIT=1 + +READING THE THREE PROBES SEPARATELY, as the mandate requires: + (1) STORE REACHABILITY -- OK. REAL RESULT. A remote client on the metal-admin + plane is AUTHORIZED by `acl snap_clients src 10.12.8.0/22` and can open a + CONNECT tunnel to the store, and the body is the store's own answer + (channel-map + sha3-384), not a 200 from something else. + (2) ALLOWLIST IN FORCE -- OK. REAL RESULT. The deny holds from the client + side too, so this is not an open forward proxy on the node plane. + (3) /etc/environment -- FAIL. EXPECTED AND OUT OF SCOPE. The juju + snap-http-proxy / snap-https-proxy model-config keys have NOT been set; + setting them is the next phase and is explicitly excluded from this + mandate. This failure is the script correctly reporting a true state of + the world. It is NOT a defect in the proxy and NOT a defect in the tool. + +So the node-side exit 1 is driven ENTIRELY by the out-of-scope probe (3). The +two probes that test the proxy itself both passed from a real remote client. + +CROSS-CHECK AGAINST SQUID'S OWN LOG -- a 200-shaped answer is not evidence by +itself, so the client identity was read back from the server side: + +$ ssh vr1-dc0-rack 'sudo grep " 10.12.8.6 " /var/log/squid/dc0-snap-proxy.access.log' +1785539195.681 1240 10.12.8.6 TCP_TUNNEL/200 16426 CONNECT api.snapcraft.io:443 - HIER_DIRECT/185.125.188.55 - +1785539195.707 0 10.12.8.6 TCP_DENIED/403 3400 CONNECT example.com:443 - HIER_NONE/- text/html + +squid independently confirms both outcomes and attributes them to 10.12.8.6. +The client-authorization path is proven from the server's own records. + +=========================================================================== +SECTION 8 -- REAL SNAP FETCH PROVEN END TO END (non-destructively) +=========================================================================== + +`check` proves a tunnel opens. It does NOT prove a snap can actually be fetched. +That was proven separately, WITHOUT installing any snap and without changing any +node state, by asking the store for `core`'s real download URL through the proxy +and then range-fetching the first four bytes of the actual payload. + +STEP 1 -- the advertised download URL, obtained THROUGH the proxy: + +$ ssh vr1-dc0-rack 'curl -sS -x http://10.12.8.4:3129 -m 30 -H "Snap-Device-Series: 16" "https://api.snapcraft.io/v2/snaps/info/core" | python3 -c ...' +download.url = https://api.snapcraft.io/api/v1/snaps/download/99T7MUlRhtI3U0QFgl5mXXESAiSwt776_17292.snap +download.size = 110305280 +sha3-384[:32] = 13c728f16430310ba467b5768bf34846 + +STEP 2 -- range-fetch that URL through the proxy, WITHOUT following redirects: + +$ ssh vr1-dc0-rack 'curl -sS -x http://10.12.8.4:3129 -m 60 -r 0-3 -o /tmp/snapmagic.bin -w "%{http_code} size=%{size_download}" ""; xxd /tmp/snapmagic.bin' +curl rc=0 http=302 size=551 +00000000: 3c21 646f 6374 7970 6520 6874 6d6c 3e0a . +... +000000a0: 3a2f 2f63 616e 6f6e 6963 616c 2d62 6f73 ://canonical-bos +000000b0: 3031 2e63 646e 2e73 6e61 7063 7261 6674 01.cdn.snapcraft +000000c0: 636f 6e74 656e 742e 636f 6d2f 646f 776e content.com/down +000000d0: 6c6f 6164 2d6f 7269 6769 6e2f 6361 6e6f load-origin/cano +000000e0: 6e69 6361 6c2d 6c67 7730 312f 3939 5437 nical-lgw01/99T7 + +>>> FINDING 8a -- THIS CORRECTS A MEASURED ASIDE IN THE SCRIPT HEADER. <<< +The header records (lines 124-127): "Measured aside worth knowing: `core`'s own +download URL came back on api.snapcraft.io, not a CDN host -- so for THIS snap +the CDN entry may be unused." +That aside measured only the FIRST HOP. The api.snapcraft.io download URL is a +302 REDIRECT to canonical-bos01.cdn.snapcraftcontent.com. The CDN allowlist +entry is therefore NOT "possibly unused" -- it is LOAD-BEARING for `core`, the +very snap the header used as its example. Had `.snapcraftcontent.com` been left +out of STORE_DOMAINS on the strength of that aside, every snap DOWNLOAD would +have failed after a successful API call -- precisely the failure mode the header +warns about two paragraphs earlier. The allowlist as built is correct; the +aside's conclusion was too weak, and the header should be corrected to say the +CDN entry is PROVEN REQUIRED. (LOGGED, not edited -- hard rule 1.) + +STEP 3 -- follow the redirect through the proxy and read the real payload bytes: + +$ ssh vr1-dc0-rack 'curl -sS -L -x http://10.12.8.4:3129 -m 90 -r 0-3 -o /tmp/snapmagic.bin -w "http=%{http_code} size=%{size_download} final=%{url_effective}" ""; xxd /tmp/snapmagic.bin' +curl rc=0 http=206 size=4 final=https://canonical-bos01.cdn.snapcraftcontent.com/download-origin/canonical-lgw01/99T7MUlRhtI3U0QFgl5mXXESAiSwt776_17292.snap?token=1785553200_d069e2ced1d5f8cdfa97475090ced71bf2bb5838 +=== bytes (squashfs magic = hsqs) === +00000000: 6873 7173 hsqs + +HTTP 206 Partial Content, 4 bytes, and those bytes are 68 73 71 73 = "hsqs" -- +the SQUASHFS MAGIC NUMBER, i.e. the first bytes of the genuine core snap image. +This is a REAL SNAP PAYLOAD retrieved through this proxy. Nothing was installed; +4 bytes of a 110 MB artifact were read and discarded. + +CONFIRMED SERVER-SIDE in squid's access log (the CDN CONNECT is recorded): +1785539231.285 362 10.12.8.2 TCP_TUNNEL/200 4931 CONNECT canonical-bos01.cdn.snapcraftcontent.com:443 - HIER_DIRECT/91.189.91.43 - + +>>> FINDING 8b -- THE DOT-PREFIX WILDCARD IS NOW PROVEN ON A NEW SUBDOMAIN. <<< +The header notes the CDN probe is a liveness-coupled assertion because "only a +real tunnel proves squid's dot-prefix matching actually ADMITS a subdomain". The +probe only ever exercises canonical-lgw01. This fetch admitted +canonical-bos01.cdn.snapcraftcontent.com -- a DIFFERENT subdomain that appears +in no probe and no config line, matched purely by the `.snapcraftcontent.com` +leading-dot rule. The wildcard is proven to generalise, not merely to match the +one host that happens to be probed. + +=========================================================================== +SECTION 9 -- FINDINGS LOGGED, NOT EXECUTED (hard rule 1) +=========================================================================== + +--- FINDING 9a: `acl snap_probe src ${LISTEN}/32` IS STILL A DEAD RULE. ------- + Severity: LOW for correctness today, MEDIUM for the guarantee it claims. + NOT FIXED. Logged only. Recommend a D-NNN / changelog entry next session. + +BUG-1 earlier today replaced a dead loopback probe ACL with `${LISTEN}/32`, i.e. +`acl snap_probe src 10.12.8.4/32`. The generated config's comment states: + "The rack connecting to a LOCAL address selects that address as its source, so + ${LISTEN}/32 is the expected match; if it is not, the probe still falls + through to snap_clients because ${LISTEN} lies inside ${CLIENT_CIDR}." + +MEASURED THIS SESSION -- the first half of that sentence is FALSE on this rack: + +$ ssh vr1-dc0-rack 'ip route get 10.12.8.4' +local 10.12.8.4 dev lo src 10.12.8.2 uid 1000 + cache + +The kernel selects 10.12.8.2 -- the PRIMARY address on virbr2 -- as the source, +NOT the 10.12.8.4 secondary alias. Squid's access log agrees: every rack-side +probe in section 8's log extract is attributed to client 10.12.8.2, never +10.12.8.4. So `acl snap_probe src 10.12.8.4/32` matches NOTHING and is still +decoration after the BUG-1 fix. The probes are admitted solely by +`http_access allow snap_clients` (10.12.8.2 is inside 10.12.8.0/22). + +WHY IT MATTERS. Nothing is broken today -- check PASSES, and it passes for a +sound reason. But the rule's STATED PURPOSE is not achieved: the comment says it +exists "so that NARROWING snap_clients cannot silently break `check`". Since the +probe actually matches via snap_clients, narrowing snap_clients WOULD break +`check`, silently, exactly as the rule was written to prevent. The safety net +is not connected. + +CORRECT FIX (for a future step, NOT applied here): make the ACL the address the +rack actually sources from, or add it alongside -- e.g. derive the primary +address of the utility bridge and use that, or simply widen snap_probe to the +bridge's primary. Do NOT just delete the rule without deciding whether the +narrowing guarantee is wanted. This is a config-content change, so it must go +through the generator + a harness case that can turn it red. + +NOTE: this is the SAME CLASS as BUG-1 (an ACL that matches nothing and is +believed to match something). BUG-1 fixed the symptom on the wrong address. The +detector that would have caught both is the one used here: read the CLIENT field +of squid's own access log, rather than reasoning about source selection. + +--- FINDING 9b: the header's CDN aside understates the CDN entry. ------------ + See FINDING 8a. The `.snapcraftcontent.com` entry is PROVEN REQUIRED for + `core`, not "possibly unused". Header comment lines 124-127 should be + corrected. NOT EDITED this session (hard rule 1). + +--- FINDING 9c: apt-installing squid leaves a FAILED, ENABLED unit. ---------- + See FINDING 3a. Between `apt-get install -y squid` and `dc-snap-proxy.sh + install`, the rack carries a crash-looping-then-failed squid.service that is + still ENABLED and would fight for 3128 at the next boot. `install` closes + this, and `check` would catch it. Worth a sentence in any runbook that + orders these two steps, because an operator who installs the package and + then stops has left the rack in that state. NOT EDITED this session. + +--- FINDING 9d: dc0 juju controller VM is RUNNING but rejected our key. ------ + Observed while looking for a client. vr1-dc0-juju-01 is `running` in virsh, + and 10.12.8.5:22 answers, but: + $ ssh vr1-dc0-juju 'which juju' + ubuntu@10.12.8.5: Permission denied (publickey). + with the configured IdentityFile + /home/jessea123/vr1-office1-creds/office1_svc_ed25519. Also noted: there is + NO juju client binary on the jumphost at all (`juju: No such file or + directory`, /snap/bin is empty). NOT INVESTIGATED -- out of mandate scope, + and no key hunting was performed. Flagged because the next phase (setting + the juju snap-http-proxy / snap-https-proxy model-config keys) NEEDS a + working juju client, and it is better to learn this now than mid-deploy. + +=========================================================================== +SECTION 10 -- FINAL STATE OF THE TOOLING (nothing was changed) +=========================================================================== + +BUG-3 resolved as "confirmed as expected", so no edit to the script or harness +was required or made. Verified by hash against the section-0 baseline: + +$ sha256sum scripts/dc-snap-proxy.sh tests/dc-snap-proxy/run-tests.sh +318cfb4190eccdfe1d760ac3a7f8f1a23ac2b35e78d07167d730bef7150439f0 scripts/dc-snap-proxy.sh +aaa2a2060e770720f25284f9413b8c222e2d3b073a6f5ab2c6daad642fc0f8ff tests/dc-snap-proxy/run-tests.sh + +IDENTICAL to the section-0 baseline. No byte of the tool or its harness changed +during this session. The only working-tree change made by this session is the +creation of THIS capture file. + +FINAL HARNESS AND LINT (re-run after the live application): + +$ bash tests/dc-snap-proxy/run-tests.sh +dc-snap-proxy: 55 passed, 0 failed + +Still 55/55 after the live application, as expected -- nothing was edited. + +$ bash scripts/repo-lint.sh + [FAIL] L10 pending change set modifies state-bearing surface(s) [docs/audit/: docs/audit/apex-ipv6-plan-20260731.txt, docs/audit/dc0-snap-proxy-install-20260731.txt] without touching docs/CURRENT-STATE.md -- GA-R1 rule 8 (C1): a status-changing commit updates CURRENT-STATE.md in the SAME commit + +FAIL: repo lint (1 fail, 1 warn, 642 files scanned) + +>>> OPEN ITEM FOR THE OPERATOR -- repo-lint L10 is RED and was NOT resolved. <<< + +This is DECLARED, not hidden, and it was NOT worked around. + +CAUSE, confirmed by reading the rule (scripts/repo_lint.py L10, ~line 450): + audit_hits = sorted(p for p in changed if p.startswith("docs/audit/")) + ... + if triggers and "docs/CURRENT-STATE.md" not in changed: +ANY path under docs/audit/ in the pending change set trips L10 unless +docs/CURRENT-STATE.md is changed in the SAME commit. So this capture file ALONE +trips it; it is a structural consequence of producing a docs/audit/ deliverable, +not an accident. + +WHY IT WAS LEFT RED. The mandate for this session explicitly forbids editing any +file under docs/ except this capture file, and explicitly forbids `git add` / +`git commit` / `git push`. Updating docs/CURRENT-STATE.md is the ONLY thing that +clears L10, and doing so would have violated that instruction. Deliberately left +for the operator to resolve at commit time, when CURRENT-STATE.md is updated for +the Stage 5 snap-proxy milestone anyway. + +CONCURRENT-SESSION NOTE. Two files appeared in this working tree DURING this +session that this session did NOT create: + ?? docs/audit/apex-ipv6-plan-20260731.txt + ?? docs/changelog-20260731-snap-proxy-apply-ipv6.md +They were absent from `git status` at session start and present at the end. +Another session is working in this same clone concurrently. This is exactly the +condition behind the earlier commit-contamination incident (e57ad09), so NO git +staging or commit operation was performed by this session, and a blanket +`git add -A` here would sweep another session's in-progress work into a commit. +Whoever commits must stage PATHS EXPLICITLY. + +=========================================================================== +SUMMARY +=========================================================================== + +PROVEN END TO END ON REAL HARDWARE THIS SESSION: + 1. The gate FAILS on a real rack with nothing built (exit 1) and PASSES with + everything built (exit 0). No longer fixture-only in either direction. + 2. `install dc0` works first-try, is self-checking, and its artifacts match + their generators by content diff. + 3. The proxy RUNS: squid 6.14 (pid 2916382) bound to 10.12.8.4:3129 + specifically, both units enabled, packaged squid.service disabled. + 4. Coexistence with MAAS's squid: MAAS's pid 1871633 untouched across the + whole install, still holding *:3128 and *:8000. + 5. Net-layer idempotence with dc0-mirror-net.service: virbr2's addresses and + the default route are byte-identical before and after install. + 6. BUG-3: the dstdomain deny shape is IDENTICAL to the port deny shape + (curl 56 / "CONNECT tunnel failed, response 403" / TCP_DENIED/403). The + assertion is correct as written; NO code change was made. + 7. Client-side authorization from a REAL remote client on the metal-admin + plane (10.12.8.6, across the bridge): allowed to the store, denied to a + non-allowlisted host, both corroborated in squid's own access log. + 8. A REAL SNAP PAYLOAD was fetched through the proxy: HTTP 206 and the bytes + "hsqs" (squashfs magic) of the genuine core snap, via the CDN. + 9. The `.snapcraftcontent.com` wildcard admits a subdomain that appears in no + probe and no config line (canonical-bos01). + +NOT PROVEN / STILL OPEN: + A. NO charm LXD container was tested -- all 9 dc0 nodes are shut off, so the + "containers source from metal-admin" premise was NOT re-proven this + session. It remains carried from the prior capture. `node dc0` from a + container is the outstanding client-side proof. + B. snapd's own consumption of the proxy is UNPROVEN and OUT OF SCOPE: the + juju snap-http-proxy / snap-https-proxy keys are not set, /etc/environment + carries no https_proxy, and no snap was installed. The header's carried + unknown -- whether a RUNNING snapd re-reads the value without a restart -- + is UNTOUCHED by this session. + C. dc1 was NOT touched at all. Everything above is dc0 only. + D. The `snap_probe` ACL is still dead (FINDING 9a). check passes for a sound + reason, but not the reason the config comment claims. + E. No juju client exists on the jumphost and vr1-dc0-juju rejects the + configured key (FINDING 9d) -- the next phase needs one. + +END OF CAPTURE. + +=========================================================================== +ADDENDUM A -- attribution of the .4 alias, MEASURED (closing a soft claim) +=========================================================================== + +Section 2 asserted the 10.12.8.4/22 alias was "placed by dc0-mirror-net.service". +That was the one INFERENCE in this capture: the alias was measured present and +the unit measured active, but the causal link was not measured. Hard rule 2 says +a would-be inference stops and gets measured, so it was: + +$ ssh vr1-dc0-rack 'systemctl cat dc0-mirror-net.service | grep ExecStart; sudo grep -n "10.12.8.4" /usr/local/sbin/dc0-mirror-net-apply' +=== dc0-mirror-net.service ExecStart === +ExecStart=/usr/local/sbin/dc0-mirror-net-apply +=== does the mirror helper place 10.12.8.4? === +10:ip addr replace 10.12.8.4/22 dev "$(br_of vr1-dc0-metal-admin)" + +CONFIRMED. dc0-mirror-net.service runs /usr/local/sbin/dc0-mirror-net-apply, +whose line 10 places 10.12.8.4/22 on the vr1-dc0-metal-admin bridge -- the SAME +address, the SAME bridge, and the SAME `ip addr replace` verb that +dc0-snap-proxy-net-apply uses. The attribution in section 2 is measured, and the +header's "identical MEASURED values, so they are no-ops for each other" claim is +now confirmed at the level of the generated helper source, not just by observing +that nothing changed. + +=========================================================================== +ADDENDUM B -- FINDING 9e (logged, not applied) +=========================================================================== + +--- FINDING 9e: the packaged squid.service rests FAILED, not INACTIVE. ------- + Severity: COSMETIC, but it is a triage-poisoning surface. NOT APPLIED. + +Final measured state after install: + +$ ssh vr1-dc0-rack 'systemctl is-active squid; systemctl is-enabled squid' +failed +disabled + +`check` reports "OK packaged squid.service neither active nor enabled", and that +verdict is CORRECT: `is-active --quiet` returns non-zero for `failed`, and +`is-enabled` is `disabled`, so the unit cannot return at the next boot. The gate +is right. + +The gap is narrower than a wrong verdict: `check` cannot DISTINGUISH `inactive` +from `failed`. The packaged unit will sit in `failed` indefinitely (its parent +logged "will not be restarted for 3600 seconds due to repeated, frequent +failures" before being disabled). So a future operator running `systemctl status +squid` -- or any red-unit sweep, or `systemctl --failed` -- sees a RED squid unit +on a rack whose snap-proxy gate is GREEN, and has to rediscover from scratch that +this is expected residue from the package install and not a fault. That is +precisely the "crash-looping squid poisons every later triage" hazard the script +header set out to avoid, surviving in a quieter form. + +REMEDY (for a future step, NOT applied here): `systemctl reset-failed squid` +after `systemctl disable --now squid`, which clears the residue without changing +any enablement or binding. If adopted it belongs in do_install's disable line, +and `check` could then assert the packaged unit is `inactive`+`disabled` rather +than merely not-active. Both are content changes to a built surface, so they +need the generator + a harness case that can turn red, and a D-NNN/changelog +entry. LOGGED ONLY -- hard rule 1. + +END OF ADDENDA. diff --git a/docs/changelog-20260731-snap-proxy-apply-ipv6.md b/docs/changelog-20260731-snap-proxy-apply-ipv6.md new file mode 100644 index 0000000..3471053 --- /dev/null +++ b/docs/changelog-20260731-snap-proxy-apply-ipv6.md @@ -0,0 +1,371 @@ +# Session changelog 2026-07-31 (part 2) -- snap proxy apply + IPv6 scoping + +GA-R2: ONE changelog per session. This is a SEPARATE session from +`docs/changelog-20260731-prefer-ipv6-research.md` (same date, different session); +that file is closed and is not appended to here. + +Branch `dc-dc-stage5-preconditions`. Status claims live ONLY in +`docs/CURRENT-STATE.md`; nothing here is a status claim. + +--- + +## Item 1 -- FINDING: the `dc-snap-proxy.sh` review's SIMPLIFICATION PLAN was never +## landed on any surface and is LOST + +**What.** No artifact change. A finding, recorded because the next action in the +prior session's close-out cannot be executed as written. + +`docs/session-ledger.md:296` gives the next step as "apply the review cuts, +re-run the mutation pass (S8 invalidates it)", and +`docs/audit/queued-findings-20260731-stage5-deploy.txt:145-146` (N2) says "The +dc-snap-proxy simplification plan is NOT applied. Applying it REQUIRES re-running +the mutation pass -- S8 invalidates M32-M39, M42-M47, M49-M52 as evidence." + +**MEASURED: the plan those `S`-numbers refer to exists nowhere in the repo.** A +repo-wide grep for `\bS8\b` over `*.md` / `*.txt` / `*.sh` returns exactly two +live hits -- the ledger line and the N2 line above, both of which CITE the plan +without containing it -- plus three unrelated 2026-07-07 archive hits +(`docs/archive/script-quality-findings-20260707.md`, an S1-S8 consolidation +proposal set about `KEYSTONE_VIP` and D-003 FIP bounds, a different subject +entirely). The mutation IDs it invalidates ARE recorded +(`docs/audit/stage5-snap-proxy-measurements-20260731.txt:505,516,523`); the +simplification items themselves are not. + +**So N2's wording understates it.** "NOT applied" reads as deferred work that is +waiting somewhere. It is not deferred -- it lived only in the review agent's +transcript and was lost at the session boundary. This is the repo's own +top-tier failure mode (a finding that exists only in a transcript), landing on a +finding ABOUT losing findings. + +**DECISION, and the reasoning stated so it can be overturned:** the +simplification is NOT re-derived. It was quality-only work on a tool that is +about to be applied for the first time; re-deriving a lost plan and then re-running +a 53-mutation pass to restore evidence for it is precisely the record-churn the +2026-07-30 standing operator directive shuts down ("Further hardening, auditing, +or record repair is OUT OF SCOPE next session unless it directly blocks the +deploy"), and it is the GA-F06 circuit-breaker case. **Consequence that follows +and is NOT a loss:** because S8 is never applied, its invalidation of +M32-M39/M42-M47/M49-M52 never fires, so the prior session's 53/53 mutation +evidence STANDS unchanged for every assertion this session did not touch. + +**Revert.** Nothing to revert (no artifact changed). To reverse the decision, +re-derive the plan from scratch against the current file and re-run the full +mutation pass -- the old S-numbers are not recoverable and must not be cited. + +--- + +## Item 2 -- FIX BUG-4: `dpkg -s squid` gates replaced by a BINARY assertion, at +## BOTH sites (`check` AND `install`) + +**What.** `scripts/dc-snap-proxy.sh` (the `check` package gate and `do_install`'s +prereq) + `tests/dc-snap-proxy/run-tests.sh` (the `dpkg` PATH fake removed, a real +binary seam added, T41 re-pointed, T41b and T20b NEW). + +**Why.** BUG-4 from the independent review +(`docs/audit/queued-findings-20260731-stage5-deploy.txt:64-66`): `dpkg -s squid` +exits 0 for `Status: deinstall ok config-files`, so a PURGED squid reports as +present. The existence-vs-content class this repo already has a scar from. +`[ -x /usr/sbin/squid ]` is shorter and strictly stronger. + +**The review named ONE site; there are TWO.** The same gate sat at `do_check` +AND at `do_install`'s prereq. Fixing only `check` would have left `install` +false-passing its prereq on a purged package and then dying confusingly at +`squid -k parse` or at unit start -- a worse failure than the one being fixed, +because it surfaces further from its cause. Both are fixed. + +**Written through the EXISTING test seam, not around it.** The assertion is +`[ -x "${SYSROOT}/usr/sbin/squid" ]`. `SYSROOT` is empty in every production +invocation, so the live path is byte-identical to `/usr/sbin/squid`; the harness +points it at its mktemp tree. This is the script's own documented idiom +(`DC_SNAP_PROXY_SYSROOT`, with in-repo precedent in `maas-region-power-key.sh`, +`creds-audit.sh`, `cloud-snapshot.sh`) and it keeps the assertion drivable in +BOTH directions, which the `dpkg` fake also was -- the fix does not cost a +failure direction. + +**Test cases, and why each exists.** +- **T41** (re-pointed): binary absent -> FAIL. The old case drove `dpkg.mode`. +- **T41b** (NEW): the path is present but NOT executable -- the exact `deinstall + ok config-files` residue shape. **This is the case the OLD assertion could not + fail on**, i.e. the one that makes the fix mean something. Without it the fix + would be untested in its own direction. +- **T20b** (NEW): `do_install` is never EXECUTED by this harness (it needs root + and a real rack), so its half of the fix would otherwise carry NO assertion at + all. Asserted structurally, following T18's established precedent of scoping a + grep to a function body: zero `dpkg -s` gates anywhere in the file, AND + `INSTALL_BODY` carries the `[ ! -x ... ]` form. + +**Mutation-proven (this repo's rule: an assertion nobody can turn red is +decoration).** Three mutations, each neutering a PREDICATE, script restored and +sha256-verified byte-identical after each: +- `check`'s binary predicate -> `if true` : **T41 AND T41b both red** (53/2). +- `install`'s prereq -> `if ! dpkg -s squid ...` : **T20b red** (53/1). +- `install`'s prereq -> `if false` : **T20b red** on its second branch (53/1). + +**Revert.** `git checkout -- scripts/dc-snap-proxy.sh tests/dc-snap-proxy/run-tests.sh` +(reverts items 2-4 together; they are edits to the same two files). + +--- + +## Item 3 -- FIX BUG-1: the probe ACL was DEAD, and it is now asserted + +**What.** `gen_squid_conf`'s `acl snap_probe src 127.0.0.1/32` -> `src ${LISTEN}/32`, +its comment block rewritten to describe the real mechanism, + T23b NEW. + +**Why.** BUG-1: squid binds `${LISTEN}:${PROXY_PORT}` only, so nothing can ever +arrive on loopback -- the rule matched nothing. The probes were actually admitted +by `snap_clients` (because `LISTEN` lies inside `CLIENT_CIDR`), and the comment +misdescribed its own mechanism. **The hazard was live, not cosmetic:** anyone +narrowing `CLIENT_CIDR` would break `check`, and the rule they would reach for to +fix it would not have helped. + +**Why the stronger fix was taken rather than a comment-only fix.** The obvious +minimal action was to correct the comment and leave the dead rule, on the grounds +that changing generated content pulls fixtures with it. **Measured, it does not:** +the harness renders its fixtures by calling the script's OWN generators +(`render gen_squid_conf`, `run-tests.sh:312-318`), never by copying config text +into the test file, so generated-content changes cost nothing in fixture +maintenance. That removed the only argument for the weaker fix. + +**Safe in both directions by construction** -- stated because the source-selection +step is REASONED, not yet measured on the rack (hard rule 2). A host connecting to +one of its own local addresses is expected to select that address as the source, +so `${LISTEN}/32` is the expected match; if that is wrong, the probe still falls +through to `snap_clients`, because `LISTEN` is inside `CLIENT_CIDR`. So the change +cannot regress `check`, and the first live run's access log will say which rule +actually admitted the probe. + +**T23b (NEW)** asserts the RENDERED artifact -- the probe ACL is the LISTEN +address and is NOT a loopback rule -- never the comment that describes it. +BUG-1's real defect was that a comment and its code disagreed, so an assertion on +prose would repeat the defect. + +**Mutation-proven.** Reverting the ACL to `127.0.0.1/32` turns **T23b red** (54/1); +script restored sha256-identical. + +--- + +## Item 4 -- BUG-3 DOCUMENTED IN THE SCRIPT, deliberately NOT "fixed" + +**What.** A new block in `scripts/dc-snap-proxy.sh`'s header, under the existing +"READ THIS BEFORE TRIAGING A REFUSE" section. No behavior change. + +**Why not fixed.** BUG-3 is that the DENY probe's expected string +`CONNECT tunnel failed, response 403` was MEASURED from a **port** deny +(`deny CONNECT !SSL_ports`), never from the **dstdomain** deny (`deny !snap_store`) +that the probe actually exercises. It cannot be fixed in the repo, because the +correct value is a fact about a live squid that nobody has yet observed -- +inventing one would be exactly the fabricated-value failure this repo calls the +cardinal sin. Guessing a second plausible string would be worse than the bug. + +**Why it is not merely cosmetic.** If squid answers a dstdomain denial +differently, that branch falls to `refuse` -> exit 3, and `check` can then NEVER +PASS. Under GA-R6 a gate that cannot PASS blocks a stage close exactly as hard as +one that cannot fail, so this is a real gate defect and not a probe nicety. + +**The protocol is now IN THE FILE** rather than in a queued-findings entry the +next reader will not open: a REFUSE on the deny probe at first run is the +EXPECTED, HANDLED path; read the actual output; re-point the assertion to the +MEASURED shape; keep the three-outcome structure so an unrecognised result still +REFUSES; prove the new assertion can fail. This is also the answer to the +close-out's "confirm BUG-3's deny shape once" -- it is a byproduct of the first +real `check`, not separate work. + +**BUG-2 NOT FIXED, unchanged from N4.** `| grep -q` under `set -o pipefail` is +INHERITED and present identically in `dc-mirror.sh:278,281` and +`dc-cache-proxy.sh:234,252,255`. A three-script sweep is out of scope under hard +rule 1 and under the standing directive. Its direction is fail-closed (a false +MISS, never a false OK), which is why it can wait. + +--- + +## Item 5 -- OPERATOR APPROVAL RECORDED: the agent's rack permissions + +**What.** No artifact change. Recorded because it relaxes the scope of hard rule +3 for one task and must not be inferred later from silence. + +**Presented to the operator, verbatim as put:** `.claude/settings.local.json` +carries `Bash(ssh vr1-dc0-rack *)` in **allow** -- an unrestricted wildcard onto +the dc0 rack, auto-added by an approval on 2026-07-30 and flagged in that +session's sweep (F1) as "worth review". Its consequence, stated at the time of +asking: a background agent can mutate the dc0 rack (including +`sudo bash -s -- install dc0`) with NO permission prompt reaching the operator, +so hard rule 3's individual gating does not fire for rack commands. The operator +was offered a narrowing. + +**Operator answer, exact utterance: "The agent permissions are approved".** + +**CONSEQUENCE, scoped deliberately narrowly.** This authorises the snap-proxy +install agent to operate on the dc0 rack under the existing allow rule. It is +NOT a general standing relaxation of hard rule 3, it does not extend to dc1, and +it does not cover the bundle deploy -- the agent's mandate explicitly stops +before `add-model` / `deploy` / any `juju model-config`. A future session must +not read this as blanket approval for unsupervised cloud mutation. + +**Not a GA-R5 ruling and deliberately not recorded as one:** it is a harness +permission decision, OPS under GA-R3 (no architectural consequence beyond this +stage, no Roosevelt delta), so it takes no D-number. The gitignored +`settings.local.json` is unchanged by this session. + +**Revert.** Remove `Bash(ssh vr1-dc0-rack *)` from `allow` in +`.claude/settings.local.json` and replace it with a targeted `ask` rule pinned to +the invocation shapes actually needed. Note the 2026-07-30 lesson that applies +directly here: quoting style and the profile token are BOTH part of the match, so +a narrowing rule that pins the wrong quote form silently fails to match and falls +through to the classifier. + +--- + +## Gate results at time of writing + +- `bash tests/dc-snap-proxy/run-tests.sh` -- **55 passed, 0 failed** (52 -> 55: + T20b, T23b, T41b added; T41 re-pointed). +- `bash scripts/run-tests-all.sh` -- **GAUNTLET: ALL GREEN (94 harnesses)**. + HARNESS-MANIFEST NOT re-recorded and deliberately so: no harness was added or + removed, only cases within an existing one, so the manifest is unchanged and + the R15(2) re-record reason does not apply. +- `bash scripts/repo-lint.sh` -- **0 fail, 1 warn** (the expected L1 legacy + non-ASCII carve-out in `docs/design-decisions.md`). +- Four mutations run across items 2-3; **all four killed**; script restored and + sha256-verified byte-identical after each. + +--- + +## Item 6 -- THE dc0 SNAP PROXY IS INSTALLED, RUNNING AND GATE-VERIFIED + +**What.** No repo artifact changed. A LIVE MUTATION on the dc0 rack, executed by a +delegated agent under the operator approval recorded in item 5. Capture: +`docs/audit/dc0-snap-proxy-install-20260731.txt` (739 lines, ASCII/LF). + +**NAMED GATE, GA-R6:** `dc-snap-proxy.sh check dc0` = **PASS, EXIT 0, 16 assertions**. +**RE-RUN INDEPENDENTLY** from the main session after the agent reported, rather than +taken on its word: squid 6.14 bound to `10.12.8.4:3129` SPECIFICALLY (not wildcard), +both units active AND enabled, packaged `squid.service` neither active nor enabled, +all five owned artifacts matching their generators BY CONTENT, and all three +behavioral probes green (store API with `channel-map` + `sha3-384`, CDN tunnel +allowed, non-allowlisted destination DENIED). + +**The before-state was captured and it FAILED (exit 1) on a real rack** -- so the +gate is proven able to fail somewhere other than a fixture. It was NOT uniformly +failed: three OK lines (virsh-resolved bridge, rack route table) prove the +instrument was on the RIGHT HOST. That distinction is the recorded wrong-host trap +for these per-DC scripts, where a jumphost run reports every item MISS. + +**MAAS's squid was untouched throughout** -- same pid (1871633) before and after, +still holding `*:3128` / `*:8000`. Net-layer idempotence with +`dc0-mirror-net.service` verified: virbr2 addresses and the default route are +byte-identical before and after, which is the D-135 coexistence property this +script was built to honour. + +**A REAL SNAP PAYLOAD WAS FETCHED THROUGH THE PROXY** -- HTTP 206, and the first +bytes are `hsqs`, the squashfs magic of the genuine `core` snap. 4 bytes of ~110 MB +read and discarded; nothing installed. **This is what moves the tool from +FIXTURE-green to end-to-end proven**, and it is the specific claim the prior +session was careful to say it could not make. + +**STILL NOT PROVEN, stated because the deploy depends on it:** no charm LXD +container was tested. All nine dc0 nodes are shut off (measured via virsh), so the +"containers source from metal-admin" premise underpinning `CLIENT_CIDR` was NOT +re-proven and is carried forward from the prior session's measurement. snapd's own +consumption is unproven and out of scope (the juju keys are unset, so `node`'s +`/etc/environment` assertion FAILS as expected). dc1 untouched. + +**Revert.** `ssh vr1-dc0-rack 'sudo systemctl disable --now dc0-snap-proxy.service +dc0-snap-proxy-net.service'`, then remove the five generated artifacts named in the +script header. The packaged `squid` remains installed and disabled; MAAS's squid is +unaffected either way. + +--- + +## Item 7 -- BUG-3 RESOLVED BY MEASUREMENT: the deny shape is CONFIRMED, no +## assertion changed + +**What.** No change. The confirmation BUG-3 was waiting for. + +The dstdomain deny (`deny !snap_store`) produces a shape BYTE-IDENTICAL to the port +deny the assertion was originally written from: **curl exit 56, +`CONNECT tunnel failed, response 403`, http_code 000**, with squid logging +`TCP_DENIED/403 ... HIER_NONE`. Measured by hand with stderr both merged and +unmerged, not inferred from `check`'s own say-line -- which matters, because +reading the gate's verdict to confirm the gate's assumption is circular. + +**MECHANISM, so this is not re-opened later:** curl reports the CONNECT +**response status**, and squid renders 403 for ANY `http_access deny` regardless of +which ACL fired. The two shapes coincide by construction, not by luck. + +**CONSEQUENCE:** BUG-3's protocol steps (b) and (c) were correctly NOT performed, +and `scripts/dc-snap-proxy.sh` + its harness are sha256-identical to their +pre-install state. The header block added in item 4 stays as the record of why the +question existed and how it was settled. `check dc0` can now be cited as a gate. + +--- + +## Item 8 -- MY BUG-1 FIX WAS WRONG, AND THE MEASUREMENT REFUTES IT (FINDING 9a) + +**What.** `tests/dc-snap-proxy/run-tests.sh` T23b gains a prominent in-file +annotation. `scripts/dc-snap-proxy.sh` deliberately NOT changed. + +**Item 3 changed `acl snap_probe src 127.0.0.1/32` to `src ${LISTEN}/32` and stated +the source-selection step as REASONED, NOT MEASURED. The measurement says it is +wrong.** Measured on the dc0 rack, and re-run independently from the main session: + + $ ip route get 10.12.8.4 + local 10.12.8.4 dev lo src 10.12.8.2 + +`.4` is a **SECONDARY** on virbr2 (primary `.2`), and **Linux does not auto-select a +secondary address as a source**. Squid's own access log attributes every rack-side +probe to `10.12.8.2`. So the rule is STILL DEAD -- BUG-1's fix corrected the +mechanism DESCRIPTION and landed on the WRONG ADDRESS. + +**Nothing is broken.** The probes are admitted by `snap_clients` exactly as before, +and `check dc0` passes. What remains unconnected is the rule's stated PURPOSE: that +narrowing `snap_clients` must not silently break `check`. + +**NOT FIXED NOW, and the reason is not laziness.** `check` diffs the generated +config against the rack's LIVE file, so any edit to `gen_squid_conf` turns the live +dc0 gate RED until `install` is re-run. That makes it a LIVE MUTATION on a +just-verified surface, out of scope under hard rule 1 and against the standing +directive. The correct value is the rack's primary metal-admin leg (dc0 +`10.12.8.2`; dc1's `10.12.68.2` is already recorded MEASURED in `lib-hosts.sh`), +and it wants a site-keyed row. **Fix the script and T23b TOGETHER at the next +install -- dc1 is the natural point, so both DCs land in one pass.** + +**T23b is annotated rather than deleted or re-pointed**, per the standing rule that +a test asserting a finding must be REPLACED with the new invariant and never +deleted to go green. Editing its expected string to `10.12.8.2` today would turn +the harness red against the live config; leaving it unannotated would let a future +session "fix" the test to match a value that is itself wrong. The annotation states +the measurement, the correct value, and why the change is deferred. + +**GENERALISABLE, and it is the reusable half:** a secondary IPv4 alias is never the +kernel's chosen source address. Any ACL, firewall rule or allowlist keyed to a +service's ALIAS address -- which is exactly what the D-134 utility `.4` band is on +both racks -- will not match that host's own outbound traffic. This applies equally +to `dc-mirror.sh` and `dc-cache-proxy.sh`, which share the `.4` alias pattern. + +**Revert.** `git checkout -- tests/dc-snap-proxy/run-tests.sh`. + +--- + +## Item 9 -- FOUR FURTHER FINDINGS FROM THE INSTALL, logged not fixed + +1. **The script header's CDN aside is WRONG (finding 8a).** It records that `core`'s + download URL "came back on api.snapcraft.io, not a CDN host -- so the CDN entry + may be unused." That measured only the FIRST HOP: the URL **302-redirects to + `canonical-bos01.cdn.snapcraftcontent.com`**. The `.snapcraftcontent.com` entry + is **LOAD-BEARING, not speculative**, and the same run proves squid's dot-prefix + wildcard admits a subdomain appearing in NO probe and NO config line. DOCFIX + material against the header. +2. **`apt-get install squid` leaves a FAILED but still ENABLED unit (9c)** that + would fight for 3128 at the next boot -- the header's predicted + `FATAL: Unable to open HTTP Socket ... listening port: 3128` was observed + VERBATIM. `install` closes it; an operator who installs the package and stops + there does not. +3. **The packaged unit rests `failed`+`disabled`, not `inactive` (9e).** `check` is + right to pass, but `systemctl --failed` shows red on a healthy rack, which is + quiet triage-poisoning. `systemctl reset-failed squid` is the remedy; NOT + applied (a live mutation outside the mandate). +4. **>>> NO JUJU CLIENT EXISTS ON THE JUMPHOST, AND THE dc0 JUJU VM REJECTS THE + CONFIGURED KEY (9d). <<<** `ssh vr1-dc0-juju` -> `Permission denied (publickey)`. + Not a permission-harness refusal and NOT retried in an altered shape. **This + blocks the next phase**, which is `add-model` + spaces + `deploy` under D-138 + (cloud-facing clients run from INSIDE the DC). Owed before that phase starts. diff --git a/docs/design-decisions.md b/docs/design-decisions.md index eb295a7..727bdc3 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -6853,3 +6853,148 @@ before applying: the D-104 amendment's gate passed at RAM 83% for the 10-node shape and this adds a 12th domain per rack. The substrate caller is `for_each`-keyed, so the addition plans as `1 add / 0 change / 0 destroy`. + +## D-139: ADOPTED -- VR1 goes IPv6-ONLY on the east-west planes, and the whole plane carve moves to GUA [ARCH] + +**Status: TWO RULINGS, BOTH 2026-07-31** (operator, GA-R5), taken in SEPARATE exchanges and +recorded separately -- neither is a batch adoption. Ruling A sets the FAMILY per plane; +ruling B sets the ADDRESSING MODEL. A is independent of B and would stand if B were +reversed. + +Raised by a live measurement session that overturned the recorded root cause of the +IPv4-only-container problem (see "What forced this" below). Admitted under GA-R3's A1 test: +a Roosevelt build session would grep this before carving a plane, and Roosevelt's +`2602:f3e2:103::/48` currently has NO plane carve at all -- whatever VR1 proves, Roosevelt +inherits. + +### Ruling A -- the family matrix + +**Question as presented.** Which planes keep IPv4, given the standing D-101 principle +"IPv6 unless IPv4 is NECESSARY"? A per-plane qualification was put to the operator naming, +for each plane, what actually forces v4: `metal-admin` (PXE/commissioning, the MAAS region +API `jujud` dials for life, the node-DNS forwarder, the entire v4-only D-134 utility band, +and a rack with NO global v6 at all); `provider-public` (tenant FIPs against a +still-substantially-v4 internet, external API VIPs, Octavia tenant LB VIPs, the edge default +route); `metal-internal` (D-101 keeps datastore east-west v4-bound, but as a JUDGEMENT -- +"marginal benefit, real cert cost" -- not a protocol requirement); and `data-tenant`, +`storage`, `replication`, `lb-mgmt` (nothing found forcing v4 -- all east-west between hosts +that already hold v6, no external clients, no gateway needed on-link). + +**Operator answer, exact utterance: "IPv6 on all planes except for metal-admin and +provider-public which will remain dual stack".** (Corrects a mistyped earlier form; the +operator issued the correction unprompted after the ambiguity was put to them rather than +guessed at.) + +**RULED.** Per DC: + +| Plane | IPv4 | IPv6 | Family | +|---|---|---|---| +| `provider-public` | RETAINED | yes | dual-stack | +| `metal-admin` | RETAINED | yes | dual-stack | +| `metal-internal` | REMOVED | yes | **IPv6-only** | +| `data-tenant` | REMOVED | yes | **IPv6-only** | +| `storage` | REMOVED | yes | **IPv6-only** | +| `replication` | REMOVED | yes | **IPv6-only** | +| `lb-mgmt` | n/a | yes | **IPv6-only** (NEW plane) | + +**This AMENDS D-101's family matrix in two places** and D-101 is otherwise unchanged: +(i) `metal-internal`'s "IPv4 retained ... datastore east-west (MySQL / RabbitMQ stay +v4-bound)" is SUPERSEDED -- that plane is now v6-only; (ii) `lb-mgmt` was already ruled +IPv6-only by D-101 but had never been carved ANYWHERE, and is now a first-class plane. +D-101's governing principle is unchanged and this is its application, not its revision. + +### Ruling B -- the addressing model + +**Question as presented.** Three schemes were put with their full costs: (1) full GUA on +every plane; (2) ULA/GUA by plane allocation -- GUA on the two dual-stack planes, ULA on the +five v6-only ones, on the reasoning that RFC 6724 precedence only bites where both families +compete; (3) keep the split exactly as carved, GUA on `provider-public` only. + +**Operator answer, exact utterance: "Full GUA on every plane (Recommended)".** + +**RULED.** Every VR1 plane is carved from its DC's GUA `/48` -- `2602:f3e2:f02::/48` (dc0), +`2602:f3e2:f03::/48` (dc1) -- following the octet map VR0 DC0 and the Willamette site +ALREADY use, so this conforms to an existing org standard rather than inventing one: + +| Plane | `/60` parent | dc0 `/64` | dc1 `/64` | +|---|---|---|---| +| `provider-public` | `f0X:10::/60` | `2602:f3e2:f02:10::/64` | `2602:f3e2:f03:10::/64` | +| `provider-public` VIP | (in `:10::/60`) | `2602:f3e2:f02:11::/64` | `2602:f3e2:f03:11::/64` | +| `metal-admin` | `f0X:20::/60` | `2602:f3e2:f02:20::/64` | `2602:f3e2:f03:20::/64` | +| `metal-internal` | (in `:20::/60`) | `2602:f3e2:f02:21::/64` | `2602:f3e2:f03:21::/64` | +| `data-tenant` | `f0X:30::/60` | `2602:f3e2:f02:30::/64` | `2602:f3e2:f03:30::/64` | +| `storage` | `f0X:40::/60` | `2602:f3e2:f02:40::/64` | `2602:f3e2:f03:40::/64` | +| `replication` | `f0X:50::/60` | `2602:f3e2:f02:50::/64` | `2602:f3e2:f03:50::/64` | +| `lb-mgmt` | `f0X:80::/60` | `2602:f3e2:f02:80::/64` | `2602:f3e2:f03:80::/64` | + +**THE ULA `/48` `fd50:840e:74e2::/48` IS RETIRED FOR VR1.** It remains a valid org aggregate +in the apex; it simply carries no VR1 plane. **This AMENDS D-101**, whose text names that +ULA prefix as a literal, and D-111. + +**Deciding reason, and it is measurable rather than aesthetic:** RFC 6724's default policy +table ranks IPv4-mapped (`::ffff:0:0/96`) at precedence **35** and ULA (`fc00::/7`) at **3**, +while GUA falls under `::/0` at **40**. On a DUAL-STACK plane a ULA leg therefore LOSES +address selection to IPv4 and is largely decorative, whereas GUA WINS -- so full GUA +delivers D-101's "IPv6 unless IPv4 is necessary" by default, with no per-host `gai.conf` +tuning. **UNVERIFIED and owed before the carve is executed: the deployed jammy image's own +`/etc/gai.conf`, which can override the default table.** Secondary reasons: conformance with +Willamette (a REAL site) and VR0, both full GUA; and Roosevelt holds a GUA `/48` with no ULA. + +**Isolation is NOT weakened, and this was checked rather than assumed.** Containment lives +at the FORWARDING layer -- SEC-010's transit drop, D-125's proven egress isolation, D-107's +airgap posture -- and the DC edge carries no v6 transport at all. GUA addressing creates no +reachability without a route. SEC-010, D-052, D-125 and D-107 are all UNCHANGED. + +### What forced this -- the measurement that overturned the recorded cause + +`docs/CURRENT-STATE.md` recorded that containers came up v4-only because "MAAS ... has +nothing to give for v6", the v6 subnets carrying no IP ranges. **MEASURED and WRONG:** +`subnet statistics` on `fd50:840e:74e2:220::/64` returns `available_string "100%"`, +`num_available 18446744069414584320`. Zero explicit `ipranges` rows means zero +RESTRICTIONS, not zero availability -- and `scripts/dc-plane-ipam.sh:368-372` had recorded +the correct MAAS behaviour since 2026-07-27, in the tree, contradicting the authoritative +document. + +**The real mechanism is juju-side and there is NO knob.** `state/linklayerdevices.go` +(`EthernetDeviceForBridge`, pinned at tag `v3.6.27`) takes `addrs[0]` -- ONE address, from an +UNSORTED mongo query -- and derives exactly one subnet CIDR, which the MAAS provider turns +into a single `LinkSubnetArgs`. There is zero family-awareness anywhere on that path and no +config, constraint or binding that changes it. gomaasapi's own doc states "Any number of +STATIC links can exist on an interface", so **MAAS would accept both families; the limit is +juju's.** LP #1723240 (`Triaged`/`Low`, open since 2017-10-12) is this exact symptom, and the +theory recorded in CURRENT-STATE is the one its reporter rebutted in-thread. + +**CONSEQUENCE that makes ruling A the achievable configuration rather than merely the +desired one: on a plane consumed by LXD containers, DUAL-STACK IS NOT EXPRESSIBLE** -- juju +picks one family non-deterministically. A v6-ONLY plane IS expressible. **CARRIED RISK, +reaffirmed by the operator after being put to them:** `metal-admin` stays dual-stack AND +carries containers, so each container's leg lands on one family by mongo order (today v4). +If it ever flipped to v6 the container would lose its path to the v4-only mirror, snap proxy +and MAAS region. No knob exists, so the mitigation is DETECTION -- a Stage-5 gate asserting +every container's `metal-admin` leg is IPv4. Owed, not built. + +### Build constraints measured before this was drafted + +- **MAAS auto-reserves `::1`-`::ffff:ffff` on EVERY IPv6 `/64`** (MAAS 3.7 + `subnet_utilization.py`; recorded at `dc-plane-ipam.sh:368-372` since 2026-07-27). Explicit + STATIC assignment inside that block works -- it is how the existing 54 node statics live at + `::100` etc. AUTO allocation draws from OUTSIDE it. **So juju containers on v6-only planes + will receive `:0:1::`-shaped addresses, NOT the D-134 textual mirror.** No v6 + `ip-range` rows are needed or possible; MAAS refuses ranges inside its own reserved block + and the band is already protected more broadly than the range would be. +- **All six existing v6 subnets carry `gateway_ip=None`** (measured). Correct for the + on-link east-west planes. Two exceptions are OPEN, not ruled here: `provider-public` (the + edge has no v6 transport; D-101 defers external v6 routing to the next deployment) and + `replication`'s CROSS-DC leg, which has no v6 route. +- `ceph-mon`, `ceph-osd` and `mysql-innodb-cluster` are all in `PREFER_IPV6_CHARMS`, so + `storage`, `replication` and `metal-internal` going v6-only REOPENS the 2026-07-31 D-101 + ruling note that set `prefer-ipv6` false on every application. That note was correct for a + cloud with no operational v6; it must be revisited per-app as each plane converts. + +### Execution -- NOT part of this ruling, and none of it is done + +Ordered, each individually gated: apex push (prefixes + retire the VR1 ULA rows); MAAS carve +per DC; re-carve 54 node v6 statics (`dc-node-v6-carve.py`); reissue the Octavia v6 IP SANs +(`octavia-pki.sh reissue` -- the rotation tool this is exactly the case for); update +`lib-net.sh`'s v6 arm and its harness; carve `lb-mgmt` a VLAN/space/subnet, which it has +never had; remove the v4 subnets from the five v6-only planes LAST, after each is proven. diff --git a/scripts/dc-snap-proxy.sh b/scripts/dc-snap-proxy.sh index 07900e1..2ceb807 100644 --- a/scripts/dc-snap-proxy.sh +++ b/scripts/dc-snap-proxy.sh @@ -106,6 +106,18 @@ # proxy is broken": the probe cannot tell "the CDN is down" from "the proxy is # misconfigured", and guessing either way is the failure this repo keeps meeting. # REFUSE means EVALUATE AGAIN, WITH MORE INFORMATION -- it is never a pass. +# +# >>> ONE ASSERTION IS NOT YET CONFIRMED AGAINST A LIVE SQUID -- READ BEFORE +# CITING `check` AS A GATE. <<< The DENY probe (3) expects the literal +# 'CONNECT tunnel failed, response 403'. That shape was MEASURED from a PORT +# deny (`deny CONNECT !SSL_ports`), never from the DSTDOMAIN deny +# (`deny !snap_store`) that is the rule actually being proven. If squid answers a +# dstdomain denial differently, that branch falls to REFUSE and `check` can then +# never PASS -- and under GA-R6 a gate that cannot PASS blocks a stage close as +# hard as one that cannot fail. So the FIRST live run is the confirmation: read +# the deny probe's actual output, and if it is not this string, re-point the +# assertion to the MEASURED shape and prove the new assertion can fail. A REFUSE +# on the first run is the expected-and-handled path, not a defect. # The CDN probe in particular is a LIVENESS-COUPLED assertion: it exists because # a config diff proves the wildcard `.snapcraftcontent.com` is WRITTEN, and only # a real tunnel proves squid's dot-prefix matching actually ADMITS a subdomain. @@ -237,10 +249,17 @@ # Clients: the metal-admin plane. MEASURED as the source the charm CONTAINERS # select toward this address, not the address juju status displays. acl snap_clients src ${CLIENT_CIDR} -# The rack's own loopback, so this script's behavioral smoke test can run. It -# proves upstream reach, CONNECT capability and that the destination rule is in -# force; CLIENT AUTHORIZATION is what \`node\` proves, from a real client. -acl snap_probe src 127.0.0.1/32 +# The rack's own probe source, so this script's behavioral smoke test can run. +# NOTE THE MECHANISM, because an earlier version of this rule misdescribed it and +# was DEAD: squid binds ${LISTEN}:${PROXY_PORT} ONLY, so a probe can never arrive +# on 127.0.0.1 and a loopback ACL matched nothing. The rack connecting to a +# LOCAL address selects that address as its source, so ${LISTEN}/32 is the +# expected match; if it is not, the probe still falls through to snap_clients +# because ${LISTEN} lies inside ${CLIENT_CIDR}. Both paths are admitted on +# purpose: this rule exists so that NARROWING snap_clients cannot silently break +# \`check\`. It proves upstream reach, CONNECT capability and that the destination +# rule is in force; CLIENT AUTHORIZATION is what \`node\` proves, from a real client. +acl snap_probe src ${LISTEN}/32 acl SSL_ports port 443 acl Safe_ports port 443 acl CONNECT method CONNECT @@ -367,10 +386,15 @@ do_check() { echo "dc-snap-proxy check ($SITE) -- ${LISTEN}:${PROXY_PORT}, clients ${CLIENT_CIDR}" - # -- package. An absence here is an OBSERVED absence, so it FAILS; it does not - # refuse. Fail-loud with the exact apt line, like dc-mirror/dc-cache-proxy. - if dpkg -s squid >/dev/null 2>&1; then say "squid installed" - else miss "squid not installed -- run: apt-get update && apt-get install -y squid"; fi + # -- the BINARY, not the package record. An absence here is an OBSERVED absence, + # so it FAILS; it does not refuse. Fail-loud with the exact apt line, like + # dc-mirror/dc-cache-proxy. + # WHY NOT `dpkg -s squid`: it exits 0 for `Status: deinstall ok config-files`, + # so a PURGED squid reports as present and the failure resurfaces later as a + # mysterious unit that will not start. The existence-vs-content class again -- + # assert the thing that has to be there, which is the executable. + if [ -x "${SYSROOT}/usr/sbin/squid" ]; then say "squid binary present (${SYSROOT}/usr/sbin/squid)" + else miss "squid binary absent at ${SYSROOT}/usr/sbin/squid -- run: apt-get update && apt-get install -y squid"; fi # -- the five artifacts this script owns, asserted by CONTENT. check_file "$SQUID_CONF" gen_squid_conf @@ -522,10 +546,13 @@ || { echo "FAIL: ${SITE}-snap-proxy-net.service did not apply" >&2; exit 4; } systemctl restart systemd-resolved >/dev/null 2>&1 || true - # Prereq 2: the package. NEVER apt-installed from here -- the repo pattern is + # Prereq 2: the BINARY. NEVER apt-installed from here -- the repo pattern is # fail-loud with the exact line, so package installs stay operator-visible. - if ! dpkg -s squid >/dev/null 2>&1; then - echo "FAIL: squid not installed" >&2 + # Same reason as `check`: `dpkg -s` exits 0 on a purged `deinstall ok + # config-files` package, which would let install sail past this prereq and die + # confusingly at `squid -k parse` or at unit start instead. + if [ ! -x "${SYSROOT}/usr/sbin/squid" ]; then + echo "FAIL: squid binary absent at ${SYSROOT}/usr/sbin/squid" >&2 echo " net layer is up -- run: apt-get update && apt-get install -y squid" >&2 echo " then re-run: dc-snap-proxy.sh install $SITE" >&2 exit 4 diff --git a/tests/dc-snap-proxy/run-tests.sh b/tests/dc-snap-proxy/run-tests.sh index fa14afd..9075736 100644 --- a/tests/dc-snap-proxy/run-tests.sh +++ b/tests/dc-snap-proxy/run-tests.sh @@ -173,6 +173,22 @@ no "T20 install never apt-installs; it prints the line for the operator" fi +# T20b: NEITHER `check` NOR `install` may gate on `dpkg -s squid` (BUG-4, +# 2026-07-31). dpkg exits 0 for `Status: deinstall ok config-files`, so a PURGED +# squid reports as present -- in `check` that is a false green, and in +# `install`'s prereq it sails past and dies confusingly at `squid -k parse` or at +# unit start instead. `check`'s half is proven behaviorally by T41/T41b; this +# case exists because `do_install` is never EXECUTED by this harness (it needs +# root and a real rack), so its half would otherwise have no assertion at all. +# Asserted on the whole file: the correct number of `dpkg -s` gates is zero. +if grep -vE '^[[:space:]]*#' "$SCRIPT" | grep -q 'dpkg -s'; then + no "T20b neither check nor install gates on 'dpkg -s' (a purged package exits 0)" +elif grep -qE '^[[:space:]]*if \[ ! -x "\$\{SYSROOT\}/usr/sbin/squid" \]' <<<"$INSTALL_BODY"; then + ok "T20b neither check nor install gates on 'dpkg -s' (a purged package exits 0)" +else + no "T20b install's package prereq is not the binary test -- it must be [ ! -x ...squid ]" +fi + # T21-T22: arg / site validation. out="$(bash "$SCRIPT" bogus dc0 2>&1)"; rc=$? [ "$rc" = "2" ] && grep -q 'usage:' <<<"$out" \ @@ -291,10 +307,11 @@ esac FAKE -cat > "$FB/dpkg" <<'FAKE' -#!/usr/bin/env bash -[ "$(cat "$FAKEDIR/dpkg.mode")" = "installed" ] && exit 0 || exit 1 -FAKE +# NO dpkg fake any more. The script asserts the squid BINARY under SYSROOT rather +# than `dpkg -s squid`, because dpkg exits 0 for `Status: deinstall ok +# config-files` -- a PURGED squid reported as present (BUG-4, 2026-07-31). The +# seam is therefore a real file under $SR, seeded by seed_sysroot and removed by +# T41, which is drivable in both directions exactly as the dpkg fake was. chmod +x "$FB"/* # Lay down a SYSROOT whose five artifacts are byte-identical to what the script @@ -319,7 +336,9 @@ seed_sysroot() { rm -rf "$SR" mkdir -p "$SR/etc/squid" "$SR/etc/systemd/system" "$SR/etc/systemd/resolved.conf.d" \ - "$SR/usr/local/sbin" "$SR/etc" + "$SR/usr/local/sbin" "$SR/usr/sbin" "$SR/etc" + # The squid BINARY seam (BUG-4). Executable and present = the green baseline. + install -m 0755 /dev/null "$SR/usr/sbin/squid" render gen_squid_conf > "$SR/etc/squid/dc0-snap-proxy.conf" render gen_proxy_unit > "$SR/etc/systemd/system/dc0-snap-proxy.service" render gen_net_helper > "$SR/usr/local/sbin/dc0-snap-proxy-net-apply" @@ -338,7 +357,6 @@ printf 'ok' > "$TMP/virsh.mode" printf 'ok' > "$TMP/ip.addr" printf 'ok' > "$TMP/ip.route" - printf 'installed' > "$TMP/dpkg.mode" } run_mode() { # run_mode