diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index f8d1323..e3e80b0 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -98,8 +98,18 @@ lib-net VID caveat, appendix-A edge-pf entry, DoD items 8-9 + item-4 range correction; gauntlet ALL GREEN (77). Still QUEUED to stage close: set-interface-v4 reload amendment (option, unruled); - permission-rules prune. NEXT: syncs complete (last-sync.status OK - both) -> mirror node-side reachability gate -> stage close-out set. + permission-rules prune. **D-135 AMENDED 2026-07-24 (operator-ruled -- + DC0/DC1 artifact-path split):** DC0 = the full-mirror deployment under + test (debmirror RUNNING, measured ~70% / 665 GiB of ~948 at + 2026-07-24T15:35Z); DC1 debmirror PAUSED at ~330 GiB (measured: the + vcloud uplink was NOT the shared bottleneck -- solo dc0 did not speed up + after pausing dc1; kept dormant as the fallback) and switched to an + INTERIM apt CACHING PROXY (apt-cacher-ng, proxy mode at utility .4:3142) + to unblock the DC1 build now, consumed via juju apt-http-proxy; DC1 + proxy REPLACED by a full mirror afterward. scripts/dc-cache-proxy.sh + SHIPPED (site-keyed check/install; harness 15/15), install on the DC1 + rack GATED. NEXT: DC0 full-mirror completes (last-sync.status OK) -> + reachability gate -> stage close-out; DC1 cache-proxy install + build. - Position inside Stage 3: deploy step A EXECUTED 2026-07-19 (6/0/6 exact; convergence zero -- `docs/audit/outer-plan-20260719-postA-converged.txt`). **Deploy step B diff --git a/docs/changelog-20260724-caveman-disable.md b/docs/changelog-20260724-caveman-disable.md index 2ccdb99..df87bd9 100644 --- a/docs/changelog-20260724-caveman-disable.md +++ b/docs/changelog-20260724-caveman-disable.md @@ -100,3 +100,53 @@ (77 harnesses)**. - **Revert:** `git revert ` -- removes L11 (+ its 3 harness cases), the two `deny` rules, and the CLAUDE.md section together. + +--- + +(Session continued into the VR1 DC-mirror work -- distinct topic, same session, so +per GA-R2 it lands in this one changelog.) + +## Item 4 -- DC1 debmirror PAUSED; bandwidth measured NOT the bottleneck + +- Operator asked to pause one sync so a single mirror completes before tomorrow, + on the premise the two syncs share the vcloud uplink. PAUSED dc1 (live): + `systemctl stop dc1-mirror-sync.service dc1-mirror-sync.timer` on the dc1 rack; + debmirror killed (0 procs), held at **329.9 GiB** (resumable; status left stale + "RUNNING"). Timer stopped so it cannot auto-restart and re-contend. +- MEASURED result: pausing dc1 did NOT speed dc0 up -- solo dc0 averaged ~37 GiB/h + vs ~35 GiB/h while contended. So the vcloud uplink is NOT the binding + constraint (it is per-connection/per-file, dominated by universe's small-package + tail). The pause neither helped nor hurt dc0; dc0 is on track (~70% at + 2026-07-24T15:35Z, ETA ~this evening UTC). This finding fed the DC1 decision + (item 5): rather than idle dc1, switch it to a caching proxy. +- **Revert:** resume dc1's full sync -- on the dc1 rack: + `systemctl daemon-reload && systemctl start dc1-mirror-sync.service dc1-mirror-sync.timer` + (debmirror resumes from 329.9 GiB, skipping downloaded files). + +## Item 5 -- DC1 interim apt caching proxy: D-135 amendment + dc-cache-proxy.sh + +- **D-135 AMENDED 2026-07-24 (GA-R5, operator utterance quoted in + design-decisions.md):** DC1 gets an INTERIM apt caching proxy to unblock its + build now; DC0 stays the full-mirror deployment under test; DC1 proxy REPLACED + by a full mirror afterward. Both artifact-delivery models exercised this + rehearsal (Roosevelt input, D-132/D-133). The paused dc1 debmirror (~330 GiB) + is KEPT dormant as the fallback, not deleted. +- **Shipped `scripts/dc-cache-proxy.sh`** (site-keyed check/install; apt-cacher-ng + in proxy mode at the D-134 utility `.4:3142`, consumed via + `juju model-config apt-http-proxy=http://:3142`; caches archive + + security + UCA with no per-repo remap). REUSES the `-mirror-net.service` + net layer (does not re-create it; fails loud if absent, pointing at dc-mirror.sh). + Feasibility recon confirmed read-only first: apt-cacher-ng installable + (`3.7.4-1ubuntu5.24.04.1`), all three upstreams 200 via the edge, port 3142 free. +- **Harness `tests/dc-cache-proxy/run-tests.sh`** (15 static cases: syntax, both + sites' utility .4 + ruled edge gw, stable-identity keying, port pin, proxy-mode + consumption, archive+UCA smoke coverage, net-layer reuse, MEASURED-tag + discipline, arg/site validation) -- **15/15 PASS**. +- CURRENT-STATE Stage-4 narrative updated (DC0/DC1 artifact-path split). repo-lint + 0 fail / 1 legacy warn; `bash scripts/run-tests-all.sh` -> **GAUNTLET: ALL GREEN + (78 harnesses)** (was 77; +tests/dc-cache-proxy). +- Live install on the DC1 rack (`apt-get install apt-cacher-ng` + `dc-cache-proxy.sh + install dc1`) is the GATED mutation, done separately after this commit. +- **Revert:** `git revert ` (removes the script, harness, D-135 amendment + paragraph, and CURRENT-STATE split); if installed live, on the dc1 rack: + `systemctl disable --now apt-cacher-ng; rm -f /etc/apt-cacher-ng/acng.conf.d/dc1-cache-proxy.conf`. diff --git a/docs/design-decisions.md b/docs/design-decisions.md index 9e0603d..621e94a 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -4593,7 +4593,7 @@ --- -## D-135: ADOPTED -- VR1 per-DC artifact mirror realization (D-107 build-out, staged) [ARCH] +## D-135: ADOPTED (AMENDED 2026-07-24) -- VR1 per-DC artifact mirror realization (D-107 build-out, staged) [ARCH] **Status:** ADOPTED 2026-07-23 (Stage 4, branch dc-dc-stage4-phase3-maas-deploy). Two GA-R5 exchanges, both quoted. (1) The Stage-4 mirror gate: a rehearsal-scoped D-107 @@ -4603,6 +4603,23 @@ Ubuntu Cloud Archive mirror, debmirror+nginx, both DCs -- now?"; operator answer: "Adopt + build item 1 (Recommended)". +**AMENDED 2026-07-24 (GA-R5, third exchange -- Stage 4): the DC1 apt path.** +Question: "DC1 is blocked waiting on its full ~950 GiB debmirror (~35% done); +adopt a caching proxy (apt-cacher-ng) on DC1 as the INTERIM apt path to unblock +the DC1 build now, keeping DC0 on the full mirror?" Operator answer, exact +utterance: "We will use it on DC1 to keep the build moving forward. We will +replace DC1 cache with a full mirror afterwards and test the full mirror +deployment on DC0." CONSEQUENCE: DC1 = interim caching proxy +(`scripts/dc-cache-proxy.sh`, apt-cacher-ng proxy mode at the utility .4:3142, +consumed via `juju model-config apt-http-proxy`), to be REPLACED by a full +mirror afterward; DC0 stays the full-mirror deployment under test. Both +artifact-delivery models thus get exercised this rehearsal -- direct Roosevelt +input for the D-132/D-133 utility-node design. SCOPE: the proxy caches apt only +(archive/security/UCA); charmhub is controller-side (Office1), snaps ride the +still-open edge egress or await D-135 item 3 (egress narrowing remains the +deferred closing mutation). The paused DC1 debmirror (~330 GiB) is KEPT dormant, +not deleted, as the fallback for the later full-mirror completion. + **Decision.** D-107's per-DC mirror is BUILT (not excepted) for VR1, staged by what Stage 5 consumes: - **Placement (all classes):** ON each DC's rack host (vvr1-dc0 / vvr1-dc1; measured diff --git a/scripts/dc-cache-proxy.sh b/scripts/dc-cache-proxy.sh new file mode 100644 index 0000000..d37fec6 --- /dev/null +++ b/scripts/dc-cache-proxy.sh @@ -0,0 +1,160 @@ +#!/usr/bin/env bash +# scripts/dc-cache-proxy.sh -- per-DC apt CACHING PROXY +# (apt-cacher-ng). D-135 amendment 2026-07-24 (operator-ruled): the INTERIM DC1 +# apt path so the DC1 build starts WITHOUT waiting on the full ~950 GiB +# debmirror. DC0 stays the full-mirror path (D-135 item 1); DC1's proxy is +# REPLACED by a full mirror afterward. Roosevelt: cache-proxy vs full-mirror is +# a per-DC utility-node artifact-delivery choice (D-132/D-133). +# +# RUNS ON THE DC RACK HOST, invoked over ssh like dc-mirror.sh (no clone there): +# check: ssh -i -J voffice1 @ \ +# 'sudo bash -s' -- check dc1 < scripts/dc-cache-proxy.sh +# install: same with 'install dc1' (idempotent; safe to re-run). +# +# PREREQS on the rack: +# - the utility NET LAYER (the .4 alias on metal-admin + the default route via +# the DC edge) must already be up -- it is provided by dc-mirror.sh's +# -mirror-net.service, present on any site that ran the mirror install. +# This script does NOT re-create it (it REUSES it); check fails loud if absent. +# - apt-cacher-ng must be apt-installed first (fail-loud with the exact line). +# +# CONSUMPTION (Stage 5, PROXY mode -- all http apt cached, incl. UCA, no remap): +# juju model-config apt-http-proxy=http://:3142 +# (or per-node /etc/apt/apt.conf.d: Acquire::http::Proxy "http://:3142";) +# +# EXIT: 0 ok | 1 check failed | 2 bad args/unknown site | 4 install failed. +# ASCII + LF only. +set -uo pipefail + +MODE="${1:-}"; SITE="${2:-}" +case "$MODE" in check|install) ;; *) + echo "usage: dc-cache-proxy.sh " >&2; exit 2 ;; esac + +# --------------------------------------------------------------------------- +# Site table. Same MEASURED/RULED utility .4 + edge gateway per DC as +# dc-mirror.sh (the proxy listens at the SAME utility address the deploy +# already expects). ADD A SITE ONLY WITH MEASURED/RULED VALUES (hard rule 2); +# the harness rejects rows without a MEASURED tag. +# --------------------------------------------------------------------------- +case "$SITE" in + dc0) + # MEASURED/RULED 2026-07-23 (D-134 amendment + D-135). NOTE: dc0's RULED + # path is the FULL MIRROR (D-135 item 1); this row exists for tooling + # parity only -- dc0 is not deployed via the proxy. + UTIL_NET="vr1-dc0-metal-admin" + LISTEN_CIDR="10.12.8.4/22" + EDGE_GW="10.12.4.1" + ;; + dc1) + # MEASURED/RULED 2026-07-24 (D-135 amendment -- interim DC1 apt path): + # LISTEN 10.12.68.4 -- the D-134 utility .4, reused from the mirror-net + # layer; EDGE_GW 10.12.64.1 -- dc1 edge LAN gw (upstream egress). + UTIL_NET="vr1-dc1-metal-admin" + LISTEN_CIDR="10.12.68.4/22" + EDGE_GW="10.12.64.1" + ;; + *) echo "FAIL: unknown site '$SITE' -- add a MEASURED row block first" >&2; exit 2 ;; +esac +LISTEN="${LISTEN_CIDR%%/*}" +PROXY_PORT="3142" +ACNG_DROPIN="/etc/apt-cacher-ng/acng.conf.d/${SITE}-cache-proxy.conf" + +# --------------------------------------------------------------------------- +# Generated config (single source of truth for check AND install). Minimal: +# apt-cacher-ng in proxy mode caches all http apt (archive/security/UCA) with +# no per-repo remap; we only pin the port so a package-default change cannot +# move it. Default CacheDir (/var/cache/apt-cacher-ng) is kept (owner-correct). +# --------------------------------------------------------------------------- +gen_acng_dropin() { + cat </dev/null \ + | sed -n "s/.*bridge name='\([^']*\)'.*/\1/p" +} + +# --------------------------------------------------------------------------- +# check -- mutate no CONFIG/service state; report and exit 0 ok / 1 incomplete. +# (The behavioral smoke test may populate the proxy cache with two Release +# files -- benign and self-managing; it is the proof the proxy actually serves.) +# --------------------------------------------------------------------------- +FAILED=0 +say(){ echo " $1"; } +miss(){ echo " MISS $1"; FAILED=1; } + +do_check() { + command -v apt-cacher-ng >/dev/null 2>&1 || dpkg -s apt-cacher-ng >/dev/null 2>&1 \ + && say "OK apt-cacher-ng installed" || miss "apt-cacher-ng not installed" + if [ ! -f "$ACNG_DROPIN" ]; then miss "$ACNG_DROPIN absent"; else + if diff -q <(gen_acng_dropin) "$ACNG_DROPIN" >/dev/null 2>&1; then + say "OK $ACNG_DROPIN matches"; else miss "$ACNG_DROPIN DIFFERS from generated"; fi + fi + systemctl is-active --quiet apt-cacher-ng \ + && say "OK apt-cacher-ng active" || miss "apt-cacher-ng not active" + ss -ltn 2>/dev/null | grep -q ":${PROXY_PORT} " \ + && say "OK listening on :${PROXY_PORT}" || miss "not listening on :${PROXY_PORT}" + # Utility net layer (REUSED from -mirror-net.service, not owned here). + br="$(util_bridge)" + if [ -n "$br" ] && ip -4 -o addr show dev "$br" 2>/dev/null | grep -q " ${LISTEN}/"; then + say "OK ${LISTEN_CIDR} on $br ($UTIL_NET)" + else miss "${LISTEN_CIDR} absent on metal-admin bridge (run dc-mirror.sh install $SITE -- it owns the net layer)"; fi + ip route show default 2>/dev/null | grep -q "via ${EDGE_GW}" \ + && say "OK default route via edge ${EDGE_GW}" \ + || miss "default route via ${EDGE_GW} absent" + # Behavioral smoke test: fetch through the proxy (proxy mode) -- archive AND + # UCA, the two the deploy needs; proves cache-miss passthrough + upstream reach. + if command -v curl >/dev/null 2>&1; then + for u in "http://archive.ubuntu.com/ubuntu/dists/jammy/Release" \ + "http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/jammy-updates/caracal/Release"; do + HTTP="$(curl -s -o /dev/null -w '%{http_code}' -x "http://${LISTEN}:${PROXY_PORT}" -m 20 "$u" 2>/dev/null)" + [ "$HTTP" = "200" ] && say "OK proxy serves $u (200)" \ + || miss "proxy did NOT serve $u (got '$HTTP')" + done + fi + [ "$FAILED" -eq 0 ] && { echo "dc-cache-proxy check ($SITE): PASS"; exit 0; } + echo "dc-cache-proxy check ($SITE): FAIL"; exit 1 +} + +# --------------------------------------------------------------------------- +# install -- idempotent; requires root. Net layer is a PREREQ (owned by +# dc-mirror.sh), verified fail-loud; apt-cacher-ng package is fail-loud too. +# --------------------------------------------------------------------------- +do_install() { + [ "$(id -u)" = "0" ] || { echo "FAIL: install requires root" >&2; exit 4; } + umask 022 + # Prereq 1: the utility net layer must be up (we reuse it, never re-create it). + br="$(util_bridge)" + if [ -z "$br" ] || ! ip -4 -o addr show dev "$br" 2>/dev/null | grep -q " ${LISTEN}/" \ + || ! ip route show default 2>/dev/null | grep -q "via ${EDGE_GW}"; then + echo "FAIL: utility net layer absent (${LISTEN} on ${UTIL_NET} + default via ${EDGE_GW})" >&2 + echo " it is owned by dc-mirror.sh -- run: dc-mirror.sh install $SITE first" >&2 + exit 4 + fi + # Prereq 2: the package (fail-loud, like dc-mirror.sh does for debmirror/nginx). + if ! dpkg -s apt-cacher-ng >/dev/null 2>&1; then + echo "FAIL: apt-cacher-ng not installed" >&2 + echo " net layer is up -- run: apt-get update && apt-get install -y apt-cacher-ng" >&2 + echo " then re-run: dc-cache-proxy.sh install $SITE" >&2 + exit 4 + fi + mkdir -p "$(dirname "$ACNG_DROPIN")" || exit 4 + gen_acng_dropin > "$ACNG_DROPIN" || exit 4 + systemctl enable apt-cacher-ng >/dev/null 2>&1 || exit 4 + systemctl restart apt-cacher-ng || { echo "FAIL: apt-cacher-ng restart" >&2; exit 4; } + echo "dc-cache-proxy install ($SITE): done -- running check:" + do_check +} + +case "$MODE" in + check) do_check ;; + install) do_install ;; +esac diff --git a/tests/dc-cache-proxy/run-tests.sh b/tests/dc-cache-proxy/run-tests.sh new file mode 100644 index 0000000..9eb23f5 --- /dev/null +++ b/tests/dc-cache-proxy/run-tests.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# tests/dc-cache-proxy/run-tests.sh -- guard for scripts/dc-cache-proxy.sh +# (D-135 amendment 2026-07-24: the INTERIM per-DC apt CACHING PROXY, apt-cacher-ng, +# the DC1-unblock path). Static: syntax, both sites' identity values (the utility +# .4 the deploy expects + the ruled edge gateways -- changing one silently breaks +# the proxy address or upstream egress), stable-identity keying (libvirt net +# names, never virbrN), proxy-mode port pin, the MEASURED-tag discipline on site +# rows, arg/site validation, and that the net layer is REUSED (not re-created). +# Exit: 0 all pass | 1 any case failed. ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SCRIPT="$(cd "$HERE/../.." && pwd)/scripts/dc-cache-proxy.sh" +PASS=0; FAIL=0 +ok(){ echo " PASS $1"; PASS=$((PASS+1)); } +no(){ echo " FAIL $1"; FAIL=$((FAIL+1)); } + +[ -f "$SCRIPT" ] && ok "T1 script present" || { no "T1 script present"; echo; echo "dc-cache-proxy: $PASS passed, $FAIL failed"; exit 1; } +bash -n "$SCRIPT" 2>/dev/null && ok "T2 bash -n clean" || no "T2 bash -n clean" + +# T3-T4: dc0 identity values (utility .4 + ruled edge gw) -- parity row +grep -q 'LISTEN_CIDR="10.12.8.4/22"' "$SCRIPT" \ + && grep -q 'UTIL_NET="vr1-dc0-metal-admin"' "$SCRIPT" \ + && ok "T3 dc0 LISTEN 10.12.8.4/22 on metal-admin (utility .4)" \ + || no "T3 dc0 LISTEN 10.12.8.4/22 on metal-admin (utility .4)" +awk '/^ dc0\)/{f=1} f{print} f&&/;;/{exit}' "$SCRIPT" | grep -q 'EDGE_GW="10.12.4.1"' \ + && ok "T4 dc0 edge gw 10.12.4.1" || no "T4 dc0 edge gw 10.12.4.1" + +# T5-T6: dc1 identity values (the INTERIM cache-proxy site) +grep -q 'LISTEN_CIDR="10.12.68.4/22"' "$SCRIPT" \ + && grep -q 'UTIL_NET="vr1-dc1-metal-admin"' "$SCRIPT" \ + && ok "T5 dc1 LISTEN 10.12.68.4/22 on metal-admin (utility .4)" \ + || no "T5 dc1 LISTEN 10.12.68.4/22 on metal-admin (utility .4)" +awk '/^ dc1\)/{f=1} f{print} f&&/;;/{exit}' "$SCRIPT" | grep -q 'EDGE_GW="10.12.64.1"' \ + && ok "T6 dc1 edge gw 10.12.64.1" || no "T6 dc1 edge gw 10.12.64.1" + +# T7: stable-identity keying -- no virbrN literal; bridge from the net name +grep -q 'net-dumpxml' "$SCRIPT" && ! grep -qE 'virbr[0-9]' "$SCRIPT" \ + && ok "T7 bridge resolved from net name; no virbrN literal" \ + || no "T7 bridge resolved from net name; no virbrN literal" + +# T8: proxy-mode port pinned to 3142 +grep -q 'PROXY_PORT="3142"' "$SCRIPT" && grep -q 'Port: ${PROXY_PORT}' "$SCRIPT" \ + && ok "T8 apt-cacher-ng port pinned 3142" || no "T8 apt-cacher-ng port pinned 3142" + +# T9: consumption documented as proxy mode (apt-http-proxy), not apt-mirror +grep -q 'apt-http-proxy=http://' "$SCRIPT" \ + && ok "T9 documents proxy-mode consumption (apt-http-proxy)" \ + || no "T9 documents proxy-mode consumption (apt-http-proxy)" + +# T10: behavioral smoke test covers BOTH archive and UCA through the proxy +grep -q 'archive.ubuntu.com' "$SCRIPT" && grep -q 'ubuntu-cloud.archive.canonical.com' "$SCRIPT" \ + && grep -q -- '-x "http://${LISTEN}:${PROXY_PORT}"' "$SCRIPT" \ + && ok "T10 smoke test proxies archive + UCA" || no "T10 smoke test proxies archive + UCA" + +# T11: install requires root +grep -q 'install requires root' "$SCRIPT" && ok "T11 install requires root" || no "T11 install requires root" + +# T12: net layer is REUSED, not re-created -- install fails loud if absent and +# points at dc-mirror.sh (the owner); no 'ip addr'/'ip route replace' writes here +grep -q 'dc-mirror.sh install' "$SCRIPT" && ! grep -qE 'ip (addr|route) (replace|add)' "$SCRIPT" \ + && ok "T12 net layer reused (no ip addr/route writes; points at dc-mirror.sh)" \ + || no "T12 net layer reused (no ip addr/route writes; points at dc-mirror.sh)" + +# T13: MEASURED/RULED-tag discipline on every site row (hard rule 2) +MR="$(grep -cE 'MEASURED/RULED' "$SCRIPT")" +[ "$MR" -ge 2 ] && ok "T13 both site rows carry a MEASURED/RULED tag" \ + || no "T13 both site rows carry a MEASURED/RULED tag (found $MR)" + +# T14: bad args / unknown site -> exit 2 +out="$(bash "$SCRIPT" bogus dc1 2>&1)"; [ "$?" = "2" ] || out="$out RC!=2" +grep -q 'usage:' <<<"$out" && ok "T14 bad mode -> usage, exit 2" || no "T14 bad mode -> usage, exit 2" +bash "$SCRIPT" check zz >/dev/null 2>&1; [ "$?" = "2" ] \ + && ok "T15 unknown site -> exit 2" || no "T15 unknown site -> exit 2" + +echo; echo "dc-cache-proxy: $PASS passed, $FAIL failed" +[ "$FAIL" -eq 0 ] && exit 0 || exit 1