Newer
Older
openstack-caracal-dc-dc / docs / audit / dc1-mirror-teardown-20260727.txt
=== dc1 full-mirror teardown + proxy net-layer ownership -- 2026-07-27 ===
repo HEAD: fbda433d28df80a47caa9480536eb350c85abcef  branch: dc-dc-stage4-phase3-maas-deploy

OPERATOR INTENT (corrects the record): DC0 tests the FULL MIRROR; DC1 tests the
MIRROR PROXY. dc1's partially-built full mirror and its script/files/settings were to be
REMOVED. CURRENT-STATE had recorded dc1's debmirror as 'PAUSED ... kept dormant as the
fallback' -- there is no fallback, and that claim is corrected in this change.
Operator direction on sequencing: "Do the net layer first, then rip it all down."

--- WHY THE NET LAYER HAD TO MOVE FIRST (the trap that was found) ---
dc-cache-proxy.sh did NOT own its utility net layer -- its header said it REUSES
dc-mirror.sh's <site>-mirror-net.service and 'fails loud if absent'. So on dc1, whose
ruled artifact path is the PROXY, three files named dc1-mirror-* were load-bearing for
the proxy: the net unit, its apply helper, and the resolved DNS drop-in. Consequences:
  (a) the 07-23 changelog's recorded mirror-removal ('stop/disable ... -net.service, rm
      the generated files') would have SILENTLY KILLED the proxy, taken down the Stage 4
      bullet-5 PASS, and blocked the Stage-5 dc1 deploy that consumes
      apt-http-proxy=http://10.12.68.4:3142;
  (b) the proxy could not be REBUILT after such a teardown -- nothing but
      'dc-mirror.sh install' created the net layer, so rebuilding a PROXY required
      reinstalling the entire MIRROR apparatus (sync unit, ENABLED daily timer, nginx
      vhost, debmirror package). A teardown-and-rebuild therefore did not converge.

--- 1. net layer moved into dc-cache-proxy.sh, then INSTALLED on the dc1 rack ---
  OK   apt-cacher-ng installed
  OK   /etc/apt-cacher-ng/acng.conf.d/dc1-cache-proxy.conf matches
  OK   apt-cacher-ng active
  OK   listening on :3142
  OK   /usr/local/sbin/dc1-cache-proxy-net-apply matches
  OK   /etc/systemd/system/dc1-cache-proxy-net.service matches
  OK   /etc/systemd/resolved.conf.d/dc1-cache-proxy.conf matches
  OK   dc1-cache-proxy-net.service enabled
  OK   10.12.68.4/22 on virbr6 (vr1-dc1-metal-admin)
  OK   default route via edge 10.12.64.1
  OK   proxy serves http://archive.ubuntu.com/ubuntu/dists/jammy/Release (200)
  OK   proxy serves http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/jammy-updates/caracal/Release (200)
dc-cache-proxy check (dc1): PASS

--- 2. INDEPENDENCE PROOF (the step that made the teardown safe) ---
A oneshot with RemainAfterExit=yes does not undo its work on stop, so merely stopping
dc1-mirror-net.service would have proven NOTHING. The live state was therefore REMOVED
and the proxy-owned unit applied ALONE:
  systemctl disable --now dc1-mirror-net.service   -> unit gone from multi-user.target
  ip addr del 10.12.68.4/22 dev virbr6             -> addr count 0
  ip route del default via 10.12.64.1              -> default route count 0
  systemctl restart dc1-cache-proxy-net.service    -> applied ok
  re-verified: 10.12.68.4/22 on virbr6 + default via 10.12.64.1 RESTORED
  dc-cache-proxy check (dc1): PASS with the mirror net unit DISABLED

--- 3. post-teardown state of the rack ---
mirror-named units remaining:
  NONE
mirror files remaining:
  gone  /usr/local/sbin/dc1-mirror-sync
  gone  /usr/local/sbin/dc1-mirror-net-apply
  gone  /etc/systemd/system/dc1-mirror-sync.service
  gone  /etc/systemd/system/dc1-mirror-sync.timer
  gone  /etc/systemd/system/dc1-mirror-net.service
  gone  /etc/nginx/sites-available/dc1-mirror
  gone  /etc/nginx/sites-enabled/dc1-mirror
  gone  /etc/systemd/resolved.conf.d/dc1-mirror.conf
  gone  /var/lib/dc-mirror
debmirror package:
  removed
proxy-owned artifacts (must all be present):
  PRESENT /usr/local/sbin/dc1-cache-proxy-net-apply
  PRESENT /etc/systemd/system/dc1-cache-proxy-net.service
  PRESENT /etc/systemd/resolved.conf.d/dc1-cache-proxy.conf
  PRESENT /etc/apt-cacher-ng/acng.conf.d/dc1-cache-proxy.conf
nginx: DELIBERATELY LEFT (see note) -- enabled vhosts:
    default
disk:
/dev/vda1       2.9T  9.0G  2.9T   1% /

NOTE ON nginx: LEFT INSTALLED, deliberately. It was a MIRROR prereq, but once the
dc1-mirror vhost is gone it is a generic web server serving only its stock 'default'
vhost -- not mirror-specific. Removing the metapackage while nginx-core keeps running is
a half-measure worse than either extreme, and a full purge is scope creep on a rack host.
If dc1 is ever rebuilt as a mirror, dc-mirror.sh install fail-louds with the exact apt line.

DELETION COST, measured (the question that decided surgical-vs-rebuild): 330G but only
33,037 files (~10MB average), so rm -rf took 47.6s wall. Disk 339G used -> 9.0G.