=== Stage 4 mirror gate -- measured state 2026-07-27 (read-only) ===
repo HEAD: 4b192a19ee8142313d0f3788399075fd067a01c3 branch: dc-dc-stage4-phase3-maas-deploy
WHY THIS CAPTURE EXISTS. The Stage 4 DoD (runbooks/dc-dc-phase3-maas-enlist-deploy.md:472-485)
requires 'per-DC mirror reachable from nodes'. GA-R6 requires a stage to close on a NAMED
executable check whose output the closing commit cites. That check -- dc-mirror.sh check --
was FALSE-GREENING on the single most important line: it tested that last-sync.status EXISTS
and printed the contents behind an unconditional 'OK'. Both racks read PASS while neither
had a good sync. Fixed this session (harness 19 -> 24); below is the state the FIXED check
reports. The mirror gate is therefore NOT MET as of this capture -- honestly, for the first
time.
--- 1. dc0 (full mirror, D-135 item 1) : dc-mirror.sh check ---
OK /usr/local/sbin/dc0-mirror-net-apply matches
OK /etc/systemd/system/dc0-mirror-net.service matches
OK /etc/systemd/resolved.conf.d/dc0-mirror.conf matches
OK /usr/local/sbin/dc0-mirror-sync matches
OK /etc/systemd/system/dc0-mirror-sync.service matches
OK /etc/systemd/system/dc0-mirror-sync.timer matches
OK /etc/nginx/sites-available/dc0-mirror matches
OK /etc/nginx/sites-enabled/dc0-mirror enabled
OK dc0-mirror-net.service enabled
OK dc0-mirror-sync.timer enabled
OK 10.12.8.4/22 on virbr2 (vr1-dc0-metal-admin)
OK default route via edge 10.12.4.1
OK nginx active
OK mirror answers http://10.12.8.4/ (200)
MISS last-sync FAILED: FAIL 2026-07-27T00:54:54Z ubuntu=255 uca=0 -- the mirror content is NOT attested current; investigate before any gate cites this check
dc-mirror check (dc0): FAIL
--- 2. dc0 root cause: transient upstream fetch failure, content intact ---
FAIL 2026-07-27T00:54:54Z ubuntu=255 uca=0
949G /var/lib/dc-mirror/ubuntu
342M /var/lib/dc-mirror/cloud-archive
jammy
jammy-security
jammy-updates
Result=exit-code
ExecMainStatus=1
ActiveState=failed
NextElapseUSecRealtime=Tue 2026-07-28 00:14:48 UTC
LastTriggerUSec=Mon 2026-07-27 00:49:42 UTC
Jul 27 00:54:49 vvr1-dc0 dc0-mirror-sync[1560492]: Download of dists/jammy/Release failed: 500 read timeout
Jul 27 00:54:50 vvr1-dc0 dc0-mirror-sync[1560492]: Errors:
Jul 27 00:54:50 vvr1-dc0 dc0-mirror-sync[1560492]: Download of dists/jammy/Release failed: 500 read timeout
Jul 27 00:54:50 vvr1-dc0 dc0-mirror-sync[1560492]: Failed to download some Release, Release.gpg or InRelease files!
Jul 27 00:54:54 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Failed with result 'exit-code'.
Jul 27 00:54:54 vvr1-dc0 systemd[1]: Failed to start dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA).
--- 3. dc1 (apt caching proxy is the RULED artifact path, D-135 amendment) ---
OK apt-cacher-ng installed
OK /etc/apt-cacher-ng/acng.conf.d/dc1-cache-proxy.conf matches
OK apt-cacher-ng active
OK listening on :3142
OK 10.12.68.4/22 on virbr6 (vr1-dc1-metal-admin)
OK default route via edge 10.12.64.1
OK proxy serves http://archive.ubuntu.com/ubuntu/dists/jammy/Release (200)
OK proxy serves http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/jammy-updates/caracal/Release (200)
dc-cache-proxy check (dc1): PASS
--- 4. dc1 dormant debmirror: the fallback, and how fragile its dormancy is ---
OK /usr/local/sbin/dc1-mirror-net-apply matches
OK /etc/systemd/system/dc1-mirror-net.service matches
OK /etc/systemd/resolved.conf.d/dc1-mirror.conf matches
OK /usr/local/sbin/dc1-mirror-sync matches
OK /etc/systemd/system/dc1-mirror-sync.service matches
OK /etc/systemd/system/dc1-mirror-sync.timer matches
OK /etc/nginx/sites-available/dc1-mirror matches
OK /etc/nginx/sites-enabled/dc1-mirror enabled
OK dc1-mirror-net.service enabled
OK dc1-mirror-sync.timer enabled
OK 10.12.68.4/22 on virbr6 (vr1-dc1-metal-admin)
OK default route via edge 10.12.64.1
OK nginx active
OK mirror answers http://10.12.68.4/ (200)
MISS last-sync UNKNOWN: status says RUNNING (RUNNING 2026-07-23T21:49:35Z) but dc1-mirror-sync.service is NOT active -- the sync died without writing a terminal state (stale marker, not a success)
dc-mirror check (dc1): FAIL
RUNNING 2026-07-23T21:49:35Z
330G /var/lib/dc-mirror/ubuntu
enabled
NextElapseUSecRealtime=
LastTriggerUSec=Fri 2026-07-24 00:52:13 UTC
Result=signal
ActiveState=failed
READING OF 4: the timer is ENABLED with an EMPTY next-elapse because the service sits in
failed/Result=signal from the 2026-07-24 pause-kill. D-135-amended ruling says dc1's
debmirror is PAUSED and 'kept dormant as the fallback' -- but nothing ENFORCES that. The
dormancy is a side effect of a failed unit, not a declared state; a systemctl reset-failed
would re-arm a 330G -> ~949G pull on a rack whose artifact path is now the proxy.
================================================================================
CORRECTION APPENDED 2026-07-27 (same day, after further measurement). Appended
rather than edited in place: this file is dated evidence, so a wrong analytical
claim inside it is superseded openly, not quietly rewritten.
THE CLAIM ABOVE (section 4 reading, line ~89) IS WRONG:
"a systemctl reset-failed would re-arm a 330G -> ~949G pull"
MEASURED afterwards, the re-arm vector is a REBOOT, not reset-failed:
dc1-mirror-sync.timer is-enabled = enabled is-active = INACTIVE (dead)
NextElapseUSecRealtime = (empty) Persistent = yes
LastTriggerUSec = Fri 2026-07-24 00:52:13 UTC
The timer was STOPPED (not disabled) at the 07-24 pause. Nothing is scheduled while
it is inactive, so `reset-failed` on the SERVICE does not start the TIMER and would
NOT have re-armed anything. What re-arms it is a reboot: `enabled` means systemd
starts the timer at boot, and `Persistent=yes` means a timer whose scheduled time has
already passed fires IMMEDIATELY on activation to catch up -- so the pull would have
begun at once, not at the next 00:xx window. That vector was real and recurring: the
rack lives inside vvr1-dc1 on the vcloud host, so a vcloud reboot reboots the rack.
MOOT AS OF THE SAME DAY: dc1's mirror was REMOVED entirely
(docs/audit/dc1-mirror-teardown-20260727.txt), so no dc1 timer exists to re-arm. The
correction is recorded because the CAUSAL LESSON generalises and dc0's timer is still
enabled with Persistent=yes -- see the systemd-timer entry in the operating skill's
platform-traps reference.
