diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 36602fe..0010d8d 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -106,7 +106,11 @@ after pausing dc1; kept dormant as the fallback) and switched to an INTERIM apt CACHING PROXY (apt-cacher-ng, proxy mode at utility .4:3142) to unblock the DC1 build now, consumed via juju apt-http-proxy; DC1 - proxy REPLACED by a full mirror afterward. scripts/dc-cache-proxy.sh + proxy REPLACED by a full mirror afterward -- **that clause is SUPERSEDED by the + D-135 AMENDMENT 2026-07-27: the per-DC split is the DELIBERATE EXPERIMENT (dc0 + tests the full mirror, dc1 tests the proxy), there is no fallback and no later + convergence, and dc1's partial mirror + apparatus were REMOVED that day.** + scripts/dc-cache-proxy.sh SHIPPED (site-keyed check/install; harness 15/15) + INSTALLED + verified on the DC1 rack 2026-07-24 (check PASS 8/8 -- apt-cacher-ng 3.7.4-1ubuntu5.24.04.1 active on .4:3142; proxy serves archive + UCA @@ -275,8 +279,29 @@ READY-handoff ruling, so no node-side probe can run inside Stage 4; G17 carries it to Stage 5 first boot. What REMAINS in Stage 4 for bullet 5 is the RACK-side half only: the artifact source answers on its own address with an attested-current sync. dc1's proxy - already satisfies that (PASS). dc0 does NOT yet -- its sync must be re-run and read OK - before bullet 5 can be marked met. + already satisfies that (PASS). **dc0's sync was RE-RUN 2026-07-27 (gated) and completed in + 20s: `OK 2026-07-27T08:43:46Z ubuntu=0 uca=0`; the fixed `dc-mirror.sh check dc0` now + reports PASS -- a PASS that is trustworthy precisely because the same check FAILED on the + same rack twenty minutes earlier. BULLET 5 IS THEREFORE MET FOR BOTH DCs** (dc0 mirror + PASS, dc1 proxy PASS), with the node-side half at G17. + - **dc1's full mirror REMOVED 2026-07-27, not paused** (D-135 AMENDMENT that day; operator: + "DC0 is the test of a full mirror. DC1 is the test of the mirror proxy."). The earlier + "dormant as the fallback" record was wrong about intent. Sequenced per operator direction + "Do the net layer first, then rip it all down", because a coupling trap made the naive + order destructive: `dc-cache-proxy.sh` did not own its utility net layer, so three files + named `dc1-mirror-*` were load-bearing FOR THE PROXY -- the recorded removal procedure + would have killed the dc1 apt path, and the proxy could not be rebuilt afterward without + reinstalling the whole mirror apparatus (enabled daily timer included), so + teardown-and-rebuild did not converge. FIXED FIRST: the net layer moved into + `dc-cache-proxy.sh` as `-cache-proxy-net.service` + apply helper + its own resolved + drop-in (harness **20/20**), installed on the dc1 rack, and INDEPENDENCE PROVEN by + deleting the live address and default route outright and applying the proxy-owned unit + ALONE -- `check dc1` PASS with `dc1-mirror-net.service` disabled. THEN removed wholesale: + all mirror units, the timer, the sync helper, the nginx vhost, the resolved drop-in, the + `debmirror` package, and the 330G of partial mirror (33,037 files, `rm -rf` 47.6s; rack + disk 339G -> 9.0G). Zero mirror-named residue; proxy PASS throughout, never down. nginx + LEFT installed deliberately (generic, now serving only its stock vhost). Capture + `docs/audit/dc1-mirror-teardown-20260727.txt`. - **Bullet 6 "NTP from the DC's own OPNsense edge working" is STALE and unsatisfiable as written** -- SUPERSEDED by D-129(iv), RULED 2026-07-21 ("Keep MAAS hierarchy (Recommended)", no NTP role on the edge). The bullet survives in FOUR surfaces diff --git a/docs/audit/dc1-mirror-teardown-20260727.txt b/docs/audit/dc1-mirror-teardown-20260727.txt new file mode 100644 index 0000000..32c1a19 --- /dev/null +++ b/docs/audit/dc1-mirror-teardown-20260727.txt @@ -0,0 +1,82 @@ +=== dc1 full-mirror teardown + proxy net-layer ownership -- 2026-07-27 === +repo HEAD: fbda433d28df80a47caa9480536eb350c85abcef branch: dc-dc-stage4-phase3-maas-deploy + +OPERATOR INTENT (corrects the record): DC0 tests the FULL MIRROR; DC1 tests the +MIRROR PROXY. dc1's partially-built full mirror and its script/files/settings were to be +REMOVED. CURRENT-STATE had recorded dc1's debmirror as 'PAUSED ... kept dormant as the +fallback' -- there is no fallback, and that claim is corrected in this change. +Operator direction on sequencing: "Do the net layer first, then rip it all down." + +--- WHY THE NET LAYER HAD TO MOVE FIRST (the trap that was found) --- +dc-cache-proxy.sh did NOT own its utility net layer -- its header said it REUSES +dc-mirror.sh's -mirror-net.service and 'fails loud if absent'. So on dc1, whose +ruled artifact path is the PROXY, three files named dc1-mirror-* were load-bearing for +the proxy: the net unit, its apply helper, and the resolved DNS drop-in. Consequences: + (a) the 07-23 changelog's recorded mirror-removal ('stop/disable ... -net.service, rm + the generated files') would have SILENTLY KILLED the proxy, taken down the Stage 4 + bullet-5 PASS, and blocked the Stage-5 dc1 deploy that consumes + apt-http-proxy=http://10.12.68.4:3142; + (b) the proxy could not be REBUILT after such a teardown -- nothing but + 'dc-mirror.sh install' created the net layer, so rebuilding a PROXY required + reinstalling the entire MIRROR apparatus (sync unit, ENABLED daily timer, nginx + vhost, debmirror package). A teardown-and-rebuild therefore did not converge. + +--- 1. net layer moved into dc-cache-proxy.sh, then INSTALLED on the dc1 rack --- + OK apt-cacher-ng installed + OK /etc/apt-cacher-ng/acng.conf.d/dc1-cache-proxy.conf matches + OK apt-cacher-ng active + OK listening on :3142 + OK /usr/local/sbin/dc1-cache-proxy-net-apply matches + OK /etc/systemd/system/dc1-cache-proxy-net.service matches + OK /etc/systemd/resolved.conf.d/dc1-cache-proxy.conf matches + OK dc1-cache-proxy-net.service enabled + OK 10.12.68.4/22 on virbr6 (vr1-dc1-metal-admin) + OK default route via edge 10.12.64.1 + OK proxy serves http://archive.ubuntu.com/ubuntu/dists/jammy/Release (200) + OK proxy serves http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/jammy-updates/caracal/Release (200) +dc-cache-proxy check (dc1): PASS + +--- 2. INDEPENDENCE PROOF (the step that made the teardown safe) --- +A oneshot with RemainAfterExit=yes does not undo its work on stop, so merely stopping +dc1-mirror-net.service would have proven NOTHING. The live state was therefore REMOVED +and the proxy-owned unit applied ALONE: + systemctl disable --now dc1-mirror-net.service -> unit gone from multi-user.target + ip addr del 10.12.68.4/22 dev virbr6 -> addr count 0 + ip route del default via 10.12.64.1 -> default route count 0 + systemctl restart dc1-cache-proxy-net.service -> applied ok + re-verified: 10.12.68.4/22 on virbr6 + default via 10.12.64.1 RESTORED + dc-cache-proxy check (dc1): PASS with the mirror net unit DISABLED + +--- 3. post-teardown state of the rack --- +mirror-named units remaining: + NONE +mirror files remaining: + gone /usr/local/sbin/dc1-mirror-sync + gone /usr/local/sbin/dc1-mirror-net-apply + gone /etc/systemd/system/dc1-mirror-sync.service + gone /etc/systemd/system/dc1-mirror-sync.timer + gone /etc/systemd/system/dc1-mirror-net.service + gone /etc/nginx/sites-available/dc1-mirror + gone /etc/nginx/sites-enabled/dc1-mirror + gone /etc/systemd/resolved.conf.d/dc1-mirror.conf + gone /var/lib/dc-mirror +debmirror package: + removed +proxy-owned artifacts (must all be present): + PRESENT /usr/local/sbin/dc1-cache-proxy-net-apply + PRESENT /etc/systemd/system/dc1-cache-proxy-net.service + PRESENT /etc/systemd/resolved.conf.d/dc1-cache-proxy.conf + PRESENT /etc/apt-cacher-ng/acng.conf.d/dc1-cache-proxy.conf +nginx: DELIBERATELY LEFT (see note) -- enabled vhosts: + default +disk: +/dev/vda1 2.9T 9.0G 2.9T 1% / + +NOTE ON nginx: LEFT INSTALLED, deliberately. It was a MIRROR prereq, but once the +dc1-mirror vhost is gone it is a generic web server serving only its stock 'default' +vhost -- not mirror-specific. Removing the metapackage while nginx-core keeps running is +a half-measure worse than either extreme, and a full purge is scope creep on a rack host. +If dc1 is ever rebuilt as a mirror, dc-mirror.sh install fail-louds with the exact apt line. + +DELETION COST, measured (the question that decided surgical-vs-rebuild): 330G but only +33,037 files (~10MB average), so rm -rf took 47.6s wall. Disk 339G used -> 9.0G. diff --git a/docs/changelog-20260727-creds-consolidation.md b/docs/changelog-20260727-creds-consolidation.md index 634342c..69b4d0e 100644 --- a/docs/changelog-20260727-creds-consolidation.md +++ b/docs/changelog-20260727-creds-consolidation.md @@ -275,6 +275,76 @@ Revert: `git revert` this commit's doc/runbook changes; the G17 row and the DOCFIX-204 edits are text-only, and no live state was touched. +## 9. dc0 sync re-run; dc1 full mirror REMOVED after moving the proxy's net layer (D-135 amendment) + +**dc0 (gated).** `systemctl start dc0-mirror-sync.service` -> `OK 2026-07-27T08:43:46Z +ubuntu=0 uca=0` in **20 seconds**, confirming the transient-`500` diagnosis exactly. Fixed +`dc-mirror.sh check dc0` now reports **PASS** -- trustworthy precisely because the same check +FAILED on the same rack twenty minutes earlier. **Stage 4 DoD bullet 5 is now MET for both DCs.** + +**dc1 -- the record was wrong about intent.** CURRENT-STATE said the debmirror was "PAUSED ... +kept dormant as the fallback". Operator, exact utterance: "DC0 is the test of a full mirror. +DC1 is the test of the mirror proxy. DC1 needed the partially created full mirror and +associated script/files/settings removed." Recorded as a **D-135 AMENDMENT 2026-07-27**: the +per-DC split is the DELIBERATE EXPERIMENT, not an interim -- no fallback, no later +convergence. That also supersedes the "DC1's proxy is REPLACED by a full mirror afterward" +claim, which was in THREE places (D-135, CURRENT-STATE:109, `dc-cache-proxy.sh:6`); all three +corrected. + +**The coupling trap, and why sequencing mattered.** Asked whether to rip down and rebuild +rather than pick and choose, I found the rebuild path was BLOCKED, not merely slower: +`dc-cache-proxy.sh` did not own its utility net layer -- its header said it REUSES +`dc-mirror.sh`'s `-mirror-net.service` and "fails loud if absent". So on dc1 three files +named `dc1-mirror-*` (net unit, apply helper, resolved DNS drop-in) were load-bearing FOR THE +PROXY. Therefore (a) the 07-23 changelog's recorded removal ("stop/disable ... -net.service, +rm the generated files") would have silently killed the proxy, the Stage-4 bullet-5 PASS, and +the Stage-5 dc1 apt path; and (b) the proxy could not be REBUILT afterward, since nothing but +`dc-mirror.sh install` created the net layer -- rebuilding a PROXY meant reinstalling the whole +MIRROR apparatus, enabled daily timer included. Operator direction: "Do the net layer first, +then rip it all down." + +**Step 1 -- net layer moved (script change).** `dc-cache-proxy.sh` now owns +`-cache-proxy-net.service` + `-cache-proxy-net-apply` + its own +`resolved.conf.d/-cache-proxy.conf`, reusing dc-mirror's proven mechanism (bridge +resolved from the libvirt net NAME at runtime, hard rule 3). Neutrally named so a proxy-only +DC carries NO mirror-named dependency. Idempotent `ip addr replace` / `ip route replace` on +identical MEASURED values, so the two net units coexist safely where a site runs both -- dc0 +is unaffected. `check` asserts the ARTIFACTS, not just the live address/route, because +asserting only the live effect would PASS while the layer was really the mirror's: it would +look independent and break on teardown, the exact failure being removed. + +Harness **20/20** (was 15). **T12 was REPLACED, not deleted** -- it asserted the superseded +design ("net layer reused ... no ip addr/route writes; points at dc-mirror.sh"), so it failed +by design once the ruling landed. It is now INVERTED (T12: owns the layer) plus T12b-T12f +(no operative dc-mirror referral; proxy-named artifacts; enabled-not-just-started; check +asserts artifacts; idempotent verbs only). T12b is deliberately scoped to NON-COMMENT lines: +the header's account of WHY the coupling was removed cites `dc-mirror.sh install`, and deleting +that history to satisfy a grep would lose the reason the trap existed. + +**Step 2 -- independence PROVEN before deleting anything.** A `oneshot` with +`RemainAfterExit=yes` does not undo its work on stop, so merely stopping +`dc1-mirror-net.service` would have proven nothing. The live state was removed outright -- +`ip addr del 10.12.68.4/22` (count -> 0) and `ip route del default via 10.12.64.1` (count -> +0) -- then the proxy-owned unit applied ALONE restored both, and `check dc1` returned **PASS +with the mirror net unit disabled**. + +**Step 3 -- wholesale removal, no picking and choosing.** All mirror units, the timer, the sync +helper, the nginx vhost, the resolved drop-in, the `debmirror` package, and the **330G** of +partial mirror (33,037 files; `rm -rf` **47.6s**; rack disk 339G -> 9.0G). Zero mirror-named +residue; every proxy artifact present; proxy PASS throughout and never down. Capture +`docs/audit/dc1-mirror-teardown-20260727.txt`. + +nginx **left installed, deliberately**: it was a mirror prereq, but with the vhost gone it is a +generic server on its stock `default` vhost. Removing the metapackage while `nginx-core` keeps +running is a half-measure worse than either extreme, and a full purge is scope creep on a rack +host. If dc1 is ever rebuilt as a mirror, `dc-mirror.sh install` fail-louds with the exact apt line. + +Revert: reinstall via `dc-mirror.sh install dc1` (recreates units/vhost/timer; needs the +`debmirror` package reinstalled first) then `sync dc1` to re-pull ~950G. **The 330G partial is +NOT recoverable** -- deliberately, per the amendment. The proxy net layer reverts by +`git revert` of `scripts/dc-cache-proxy.sh` + its harness, then re-running +`dc-mirror.sh install dc1` to restore the mirror-owned layer. + --- ## Findings LOGGED, not actioned (hard rule 1) diff --git a/docs/design-decisions.md b/docs/design-decisions.md index d06c63e..f4bbf3f 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -4725,9 +4725,38 @@ later D-129(iv) ruling (MAAS hierarchy stays the time authority); the edge remains the SITE's upstream sync point. +**AMENDMENT 2026-07-27 -- the two DCs test two STRATEGIES; dc1's full mirror is REMOVED, +not paused.** The 2026-07-24 amendment recorded dc1's debmirror as "PAUSED ... kept dormant +as the fallback" and said "DC1's proxy is REPLACED by a full mirror afterward", i.e. a +temporary state converging on two full mirrors. That is SUPERSEDED. Operator statement, +exact utterance (2026-07-27): **"DC0 is the test of a full mirror. DC1 is the test of the +mirror proxy. DC1 needed the partially created full mirror and associated +script/files/settings removed."** So the per-DC split is the DELIBERATE EXPERIMENT, not an +interim: dc0 = full mirror, dc1 = caching proxy, each proven on its own DC. There is no +fallback and no later convergence. EXECUTED the same day: dc1's 330G partial mirror, its +units, timer, sync helper, nginx vhost, resolved drop-in and the `debmirror` package are all +removed (capture `docs/audit/dc1-mirror-teardown-20260727.txt`). + +**Consequence that had to be fixed FIRST (a measured coupling trap).** `dc-cache-proxy.sh` +did not own its utility net layer -- it REUSED `dc-mirror.sh`'s `-mirror-net.service` +and refused to install without it. On dc1, whose ruled path is the PROXY, three files named +`dc1-mirror-*` (net unit, apply helper, resolved DNS drop-in) were therefore load-bearing +FOR THE PROXY. Two consequences, both real: the recorded mirror-removal procedure would have +silently killed the proxy (and with it the Stage-5 dc1 apt path), and the proxy could not be +REBUILT after such a teardown, because nothing but `dc-mirror.sh install` created the net +layer -- so rebuilding a proxy meant reinstalling the whole mirror apparatus including an +ENABLED daily debmirror timer. Teardown-and-rebuild did not converge. The net layer now +lives in `dc-cache-proxy.sh` as `-cache-proxy-net.service` (+ apply helper + its own +resolved drop-in), neutrally named so a proxy-only DC carries NO mirror-named dependency; +the two net units are safe to coexist where a site runs both (idempotent `ip addr replace` / +`ip route replace` on identical MEASURED values), so dc0 is unaffected. Proven by removing +the live address and route outright and applying the proxy-owned unit ALONE. **Roosevelt: a +DC's artifact-delivery strategy must be changeable without rebuilding the other strategy's +apparatus -- per-DC utility-node services own their own network prerequisites.** + **A1 test:** Roosevelt greps this for the mirror service shape, the utility-address -convention, the staged class order, and the measure-then-rule posture on -charmhub/snaps. +convention, the staged class order, the measure-then-rule posture on charmhub/snaps, and +the per-DC-strategy rule that a utility service owns its own net prerequisites. ## D-136: NetBox-coupled per-DC render pipeline for bundle overlays and tfvars [ARCH] diff --git a/scripts/dc-cache-proxy.sh b/scripts/dc-cache-proxy.sh index d37fec6..503d687 100644 --- a/scripts/dc-cache-proxy.sh +++ b/scripts/dc-cache-proxy.sh @@ -3,8 +3,12 @@ # (apt-cacher-ng). D-135 amendment 2026-07-24 (operator-ruled): the INTERIM DC1 # apt path so the DC1 build starts WITHOUT waiting on the full ~950 GiB # debmirror. DC0 stays the full-mirror path (D-135 item 1); DC1's proxy is -# REPLACED by a full mirror afterward. Roosevelt: cache-proxy vs full-mirror is -# a per-DC utility-node artifact-delivery choice (D-132/D-133). +# the DC1 STRATEGY, not an interim step (D-135 AMENDMENT 2026-07-27: "DC0 is the +# test of a full mirror. DC1 is the test of the mirror proxy."). The earlier +# "DC1's proxy is REPLACED by a full mirror afterward" plan is SUPERSEDED -- dc1's +# partial mirror and its whole apparatus were REMOVED 2026-07-27. Roosevelt: +# cache-proxy vs full-mirror is a per-DC utility-node artifact-delivery choice +# (D-132/D-133), and each DC proves one of them. # # RUNS ON THE DC RACK HOST, invoked over ssh like dc-mirror.sh (no clone there): # check: ssh -i -J voffice1 @ \ @@ -12,12 +16,31 @@ # install: same with 'install dc1' (idempotent; safe to re-run). # # PREREQS on the rack: -# - the utility NET LAYER (the .4 alias on metal-admin + the default route via -# the DC edge) must already be up -- it is provided by dc-mirror.sh's -# -mirror-net.service, present on any site that ran the mirror install. -# This script does NOT re-create it (it REUSES it); check fails loud if absent. # - apt-cacher-ng must be apt-installed first (fail-loud with the exact line). # +# THIS SCRIPT NOW OWNS ITS OWN UTILITY NET LAYER (changed 2026-07-27). +# It previously REUSED dc-mirror.sh's -mirror-net.service for the .4 +# alias + the edge default route, and refused to install without it. That +# coupling was a measured trap: on dc1 -- whose RULED artifact path is this +# proxy, NOT a mirror -- three files named `dc1-mirror-*` (the net unit, its +# apply helper, and the resolved drop-in) were load-bearing for the PROXY. So +# the recorded mirror-removal procedure (07-23 changelog: "stop/disable +# -mirror-sync.timer + -net.service, rm the generated files") would have +# silently killed the proxy, and anyone tidying up "leftover mirror stuff" on a +# proxy-only DC would have done the same. Worse, the proxy could not be REBUILT +# after such a teardown: nothing but `dc-mirror.sh install` created the net +# layer, so rebuilding a proxy meant reinstalling the whole mirror apparatus +# (sync unit, enabled daily timer, nginx vhost, debmirror package). +# +# Now: -cache-proxy-net.service + -cache-proxy-net-apply + +# /etc/systemd/resolved.conf.d/-cache-proxy.conf. Neutrally named, owned +# here, so a proxy-only DC has NO mirror-named dependency and is independently +# rebuildable. The two net units are SAFE TO COEXIST on a site running both +# (`ip addr replace` / `ip route replace` are idempotent and both set the same +# MEASURED values), so this is additive for dc0, which keeps its mirror. +# Roosevelt: a DC's artifact-delivery strategy is a per-DC choice that must be +# changeable without a rebuild of the other strategy's apparatus. +# # CONSUMPTION (Stage 5, PROXY mode -- all http apt cached, incl. UCA, no remap): # juju model-config apt-http-proxy=http://:3142 # (or per-node /etc/apt/apt.conf.d: Acquire::http::Proxy "http://:3142";) @@ -58,6 +81,10 @@ LISTEN="${LISTEN_CIDR%%/*}" PROXY_PORT="3142" ACNG_DROPIN="/etc/apt-cacher-ng/acng.conf.d/${SITE}-cache-proxy.conf" +# Net layer owned HERE (see the header note). Named for the PROXY, never "mirror". +NET_HELPER="/usr/local/sbin/${SITE}-cache-proxy-net-apply" +NET_UNIT="/etc/systemd/system/${SITE}-cache-proxy-net.service" +RESOLVED_DROPIN="/etc/systemd/resolved.conf.d/${SITE}-cache-proxy.conf" # --------------------------------------------------------------------------- # Generated config (single source of truth for check AND install). Minimal: @@ -75,6 +102,55 @@ EOF } +# --- the utility net layer this script OWNS --------------------------------- +# Same MEASURED mechanism dc-mirror.sh proved on both racks 2026-07-23: resolve +# the bridge from the libvirt network NAME at runtime, then idempotently place +# the .4 alias and the edge default route. Deliberately identical values, so the +# two units coexist without fighting where a site runs both. +gen_net_helper() { + cat </dev/null | grep -q ":${PROXY_PORT} " \ && say "OK listening on :${PROXY_PORT}" || miss "not listening on :${PROXY_PORT}" - # Utility net layer (REUSED from -mirror-net.service, not owned here). + # Utility net layer -- OWNED HERE since 2026-07-27, so assert the ARTIFACTS too, + # not just the live effect. Asserting only the live address/route would pass + # while the layer was actually provided by -mirror-net.service, which is + # exactly the hidden coupling this change removes: it would look independent and + # break the moment the mirror was torn down. + for pair in "$NET_HELPER:gen_net_helper" "$NET_UNIT:gen_net_unit" \ + "$RESOLVED_DROPIN:gen_resolved_dropin"; do + f="${pair%%:*}"; g="${pair##*:}" + if [ ! -f "$f" ]; then miss "$f absent (this script owns the net layer now)" + elif diff -q <("$g") "$f" >/dev/null 2>&1; then say "OK $f matches" + else miss "$f DIFFERS from generated"; fi + done + systemctl is-enabled --quiet "${SITE}-cache-proxy-net.service" 2>/dev/null \ + && say "OK ${SITE}-cache-proxy-net.service enabled" \ + || miss "${SITE}-cache-proxy-net.service not enabled (the layer would not survive a reboot)" br="$(util_bridge)" if [ -n "$br" ] && ip -4 -o addr show dev "$br" 2>/dev/null | grep -q " ${LISTEN}/"; then say "OK ${LISTEN_CIDR} on $br ($UTIL_NET)" - else miss "${LISTEN_CIDR} absent on metal-admin bridge (run dc-mirror.sh install $SITE -- it owns the net layer)"; fi + else miss "${LISTEN_CIDR} absent on metal-admin bridge (run: dc-cache-proxy.sh install $SITE)"; fi ip route show default 2>/dev/null | grep -q "via ${EDGE_GW}" \ && say "OK default route via edge ${EDGE_GW}" \ || miss "default route via ${EDGE_GW} absent" @@ -131,14 +221,28 @@ do_install() { [ "$(id -u)" = "0" ] || { echo "FAIL: install requires root" >&2; exit 4; } umask 022 - # Prereq 1: the utility net layer must be up (we reuse it, never re-create it). + # Prereq: the libvirt utility net must EXIST (we resolve its bridge). This is a + # rack-standup artifact (dc-rack-net.sh), not a mirror artifact -- so a proxy-only + # DC has no dependency on the mirror apparatus at any point. br="$(util_bridge)" - if [ -z "$br" ] || ! ip -4 -o addr show dev "$br" 2>/dev/null | grep -q " ${LISTEN}/" \ - || ! ip route show default 2>/dev/null | grep -q "via ${EDGE_GW}"; then - echo "FAIL: utility net layer absent (${LISTEN} on ${UTIL_NET} + default via ${EDGE_GW})" >&2 - echo " it is owned by dc-mirror.sh -- run: dc-mirror.sh install $SITE first" >&2 + if [ -z "$br" ]; then + echo "FAIL: libvirt network '${UTIL_NET}' not found -- cannot resolve its bridge" >&2 + echo " the rack net layer comes first: dc-rack-net.sh install $SITE" >&2 exit 4 fi + # NET LAYER (owned here since 2026-07-27). Written and started BEFORE the proxy, + # so the LISTEN address and upstream egress exist independently of any mirror. + install -m 0755 /dev/null "$NET_HELPER" || exit 4 + gen_net_helper > "$NET_HELPER" || exit 4 + chmod 0755 "$NET_HELPER" || exit 4 + gen_net_unit > "$NET_UNIT" || exit 4 + mkdir -p "$(dirname "$RESOLVED_DROPIN")" || exit 4 + gen_resolved_dropin > "$RESOLVED_DROPIN" || exit 4 + systemctl daemon-reload || exit 4 + systemctl enable "${SITE}-cache-proxy-net.service" >/dev/null 2>&1 || exit 4 + systemctl restart "${SITE}-cache-proxy-net.service" \ + || { echo "FAIL: ${SITE}-cache-proxy-net.service did not apply" >&2; exit 4; } + systemctl restart systemd-resolved >/dev/null 2>&1 || true # Prereq 2: the package (fail-loud, like dc-mirror.sh does for debmirror/nginx). if ! dpkg -s apt-cacher-ng >/dev/null 2>&1; then echo "FAIL: apt-cacher-ng not installed" >&2 diff --git a/tests/dc-cache-proxy/run-tests.sh b/tests/dc-cache-proxy/run-tests.sh index 9eb23f5..19f07ee 100644 --- a/tests/dc-cache-proxy/run-tests.sh +++ b/tests/dc-cache-proxy/run-tests.sh @@ -55,11 +55,59 @@ # T11: install requires root grep -q 'install requires root' "$SCRIPT" && ok "T11 install requires root" || no "T11 install requires root" -# T12: net layer is REUSED, not re-created -- install fails loud if absent and -# points at dc-mirror.sh (the owner); no 'ip addr'/'ip route replace' writes here -grep -q 'dc-mirror.sh install' "$SCRIPT" && ! grep -qE 'ip (addr|route) (replace|add)' "$SCRIPT" \ - && ok "T12 net layer reused (no ip addr/route writes; points at dc-mirror.sh)" \ - || no "T12 net layer reused (no ip addr/route writes; points at dc-mirror.sh)" +# T12 (REPLACED 2026-07-27 -- it asserted the SUPERSEDED design). It used to require +# that this script REUSE dc-mirror.sh's net layer and write no ip addr/route itself. +# That coupling was a measured trap: on a proxy-only DC, three `*-mirror-*` files were +# load-bearing for the PROXY, so the recorded mirror-removal procedure would have killed +# it, and the proxy could not be rebuilt without reinstalling the whole mirror apparatus. +# Operator ruled the net layer moves here (2026-07-27), so the test is INVERTED rather +# than dropped -- the old assertion is now the defect it protected against. +# T12: this script OWNS the net layer -- it writes the address and route itself... +grep -qE 'ip addr replace' "$SCRIPT" && grep -qE 'ip route replace default' "$SCRIPT" \ + && ok "T12 net layer is OWNED here (writes its own addr + default route)" \ + || no "T12 net layer is OWNED here (writes its own addr + default route)" + +# T12b: ...and no longer defers to dc-mirror.sh for it. A lingering "run dc-mirror.sh +# install" instruction would send an operator to re-create the very apparatus a +# proxy-only DC is meant not to have. Scoped to NON-COMMENT lines on purpose: the +# header's account of WHY the coupling was removed cites that command, and deleting +# the history to satisfy a grep would lose the reason the trap existed. What must not +# survive is an OPERATIVE referral -- one reachable at runtime in an echo/miss string. +grep -vE '^[[:space:]]*#' "$SCRIPT" | grep -q 'dc-mirror.sh install' \ + && no "T12b no OPERATIVE dc-mirror.sh referral remains (header history is exempt)" \ + || ok "T12b no OPERATIVE dc-mirror.sh referral remains (header history is exempt)" + +# T12c: the owned artifacts are named for the PROXY, never "mirror". The whole point +# is that a proxy-only DC carries no mirror-named dependency for someone to tidy away. +grep -q 'cache-proxy-net.service' "$SCRIPT" \ + && grep -q 'cache-proxy-net-apply' "$SCRIPT" \ + && ! grep -qE '(NET_UNIT|NET_HELPER|RESOLVED_DROPIN)=.*mirror' "$SCRIPT" \ + && ok "T12c owned net artifacts are proxy-named, not mirror-named" \ + || no "T12c owned net artifacts are proxy-named, not mirror-named" + +# T12d: the net layer must be ENABLED, not merely started -- otherwise the proxy's +# LISTEN address and egress vanish on the next rack reboot. This rack has rebooted +# before (the D-126/D-128 base-leg machinery exists for exactly that reason). +grep -qE 'systemctl enable "\$\{SITE\}-cache-proxy-net.service"' "$SCRIPT" \ + && grep -q 'cache-proxy-net.service enabled' "$SCRIPT" \ + && ok "T12d net unit is enabled at install and asserted by check (reboot-persistent)" \ + || no "T12d net unit is enabled at install and asserted by check (reboot-persistent)" + +# T12e: check must assert the ARTIFACTS, not just the live address/route. Asserting +# only the live effect would PASS while the layer was really provided by +# -mirror-net.service -- it would look independent and break on mirror teardown, +# which is the exact failure this change exists to prevent. +grep -q 'gen_net_helper' "$SCRIPT" && grep -q 'gen_net_unit' "$SCRIPT" \ + && grep -q 'this script owns the net layer now' "$SCRIPT" \ + && ok "T12e check asserts the owned net artifacts, not only their live effect" \ + || no "T12e check asserts the owned net artifacts, not only their live effect" + +# T12f: idempotent verbs only. `ip addr replace`/`ip route replace` let this unit and +# a surviving -mirror-net.service coexist on a site running both (dc0 keeps its +# mirror), setting identical MEASURED values instead of fighting. +! grep -qE 'ip (addr|route) add ' "$SCRIPT" \ + && ok "T12f net writes use idempotent replace (safe to coexist with the mirror unit)" \ + || no "T12f net writes use idempotent replace (safe to coexist with the mirror unit)" # T13: MEASURED/RULED-tag discipline on every site row (hard rule 2) MR="$(grep -cE 'MEASURED/RULED' "$SCRIPT")"