diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index fcb64f1..9480faa 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -248,6 +248,44 @@ author + apply the 10th controller VM (egress OPEN); resolve the vault-VIP + phasing; fix the deploy-gate scripts; `juju bootstrap` (controller tag, egress OPEN) -> deploy. Separately: DC0 full-mirror completes -> reachability gate -> stage close-out. +- **STAGE 4 CLOSE-OUT OPENED 2026-07-27** (operator-directed, ahead of resuming the DC1 + Stage-5 chain -- the two are SEPARATE tracks sharing one branch by the 07-24 one-branch + ruling, and no Stage-5 item is a Stage-4 close precondition). Of the DoD's six bullets + (`runbooks/dc-dc-phase3-maas-enlist-deploy.md:472-485`), four are met and captured (nodes + READY+carved+tagged, six planes per node, provider NIC raw, PXE v4). The remaining two are + BOTH defective as written: + - **Bullet 5 "per-DC mirror reachable" -- NOT MET, and the check it closes on was + FALSE-GREENING.** `dc-mirror.sh check` asserted only that `last-sync.status` EXISTS, + printing its contents behind an unconditional `OK`, so it could not fail. MEASURED both + racks 2026-07-27: dc0 read `FAIL ... ubuntu=255`, dc1 read a FOUR-DAY-STALE + `RUNNING 2026-07-23T21:49:35Z` left by the debmirror the D-135 amendment killed -- both + printed OK and PASSED. FIXED (status word now case-analysed; RUNNING is an explicit + UNKNOWN cross-checked against the unit; absent and unrecognised both refuse; harness + **24/24**, was 19). Re-run now reports the true state: **dc0 FAIL / dc1 FAIL**, capture + `docs/audit/stage4-mirror-gate-20260727.txt`. Substance: dc0's mirror CONTENT is intact + (949G+342M, all three dists + pool) and only the overnight INCREMENTAL failed, on a + transient upstream `500 read timeout` for `dists/jammy/Release`; dc1's PROXY -- its RULED + artifact path per the D-135 amendment -- checks PASS genuinely (apt-cacher-ng on .4:3142 + serving archive + UCA 200), while its dormant fallback debmirror is dormant only + ACCIDENTALLY (timer `enabled` with EMPTY next-elapse because the unit sits in + failed/Result=signal, so a `reset-failed` re-arms a 330G->949G pull; the ruling is not + enforced by anything). STILL OPEN: the node-side half of bullet 5 -- nodes are READY + (powered off) by the READY-handoff ruling, so a node-side probe needs either a gated + rescue-boot check or, per GA-R6 E3, its own gate row targeted at Stage 5 first boot. That + is an operator decision; there is NO conditional close. + - **Bullet 6 "NTP from the DC's own OPNsense edge working" is STALE and unsatisfiable as + written** -- SUPERSEDED by D-129(iv), RULED 2026-07-21 ("Keep MAAS hierarchy + (Recommended)", no NTP role on the edge). The bullet survives in FOUR surfaces + (`docs/dc-dc-deployment-workflow.md:206`, `docs/dc-dc-buildout-design.md:120`, + `runbooks/dc-dc-phase3-maas-enlist-deploy.md:412,484`, + `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:26`) and needs a DOCFIX re-expressing it as + MAAS-hierarchy time verification before it can be checked at all. + Also queued to this close and NOT yet done: carve residue cleanup (192.168.1.0/24, emptied + fabric-5, ~90 empty auto-created fabrics), the SEC-024 retention question on the two + `pre-*` state-surgery snapshots (irreversible, operator-deferred), the `set-interface-v4` + reload amendment (an unruled OPTION), and the GA-R6 close-out set itself (GA-R2 + consolidation, skill sweep + snapshot regeneration, GA-R7 memory review, final gauntlet, + operator-gated merge to `main` as a merge commit, branch retirement). - Position inside Stage 3: deploy step A EXECUTED 2026-07-19 (6/0/6 exact; convergence zero -- `docs/audit/outer-plan-20260719-postA-converged.txt`). **Deploy step B diff --git a/docs/audit/stage4-mirror-gate-20260727.txt b/docs/audit/stage4-mirror-gate-20260727.txt new file mode 100644 index 0000000..c1e9632 --- /dev/null +++ b/docs/audit/stage4-mirror-gate-20260727.txt @@ -0,0 +1,90 @@ +=== Stage 4 mirror gate -- measured state 2026-07-27 (read-only) === +repo HEAD: 4b192a19ee8142313d0f3788399075fd067a01c3 branch: dc-dc-stage4-phase3-maas-deploy + +WHY THIS CAPTURE EXISTS. The Stage 4 DoD (runbooks/dc-dc-phase3-maas-enlist-deploy.md:472-485) +requires 'per-DC mirror reachable from nodes'. GA-R6 requires a stage to close on a NAMED +executable check whose output the closing commit cites. That check -- dc-mirror.sh check -- +was FALSE-GREENING on the single most important line: it tested that last-sync.status EXISTS +and printed the contents behind an unconditional 'OK'. Both racks read PASS while neither +had a good sync. Fixed this session (harness 19 -> 24); below is the state the FIXED check +reports. The mirror gate is therefore NOT MET as of this capture -- honestly, for the first +time. + +--- 1. dc0 (full mirror, D-135 item 1) : dc-mirror.sh check --- + OK /usr/local/sbin/dc0-mirror-net-apply matches + OK /etc/systemd/system/dc0-mirror-net.service matches + OK /etc/systemd/resolved.conf.d/dc0-mirror.conf matches + OK /usr/local/sbin/dc0-mirror-sync matches + OK /etc/systemd/system/dc0-mirror-sync.service matches + OK /etc/systemd/system/dc0-mirror-sync.timer matches + OK /etc/nginx/sites-available/dc0-mirror matches + OK /etc/nginx/sites-enabled/dc0-mirror enabled + OK dc0-mirror-net.service enabled + OK dc0-mirror-sync.timer enabled + OK 10.12.8.4/22 on virbr2 (vr1-dc0-metal-admin) + OK default route via edge 10.12.4.1 + OK nginx active + OK mirror answers http://10.12.8.4/ (200) + MISS last-sync FAILED: FAIL 2026-07-27T00:54:54Z ubuntu=255 uca=0 -- the mirror content is NOT attested current; investigate before any gate cites this check +dc-mirror check (dc0): FAIL + +--- 2. dc0 root cause: transient upstream fetch failure, content intact --- +FAIL 2026-07-27T00:54:54Z ubuntu=255 uca=0 +949G /var/lib/dc-mirror/ubuntu +342M /var/lib/dc-mirror/cloud-archive +jammy +jammy-security +jammy-updates +Result=exit-code +ExecMainStatus=1 +ActiveState=failed +NextElapseUSecRealtime=Tue 2026-07-28 00:14:48 UTC +LastTriggerUSec=Mon 2026-07-27 00:49:42 UTC +Jul 27 00:54:49 vvr1-dc0 dc0-mirror-sync[1560492]: Download of dists/jammy/Release failed: 500 read timeout +Jul 27 00:54:50 vvr1-dc0 dc0-mirror-sync[1560492]: Errors: +Jul 27 00:54:50 vvr1-dc0 dc0-mirror-sync[1560492]: Download of dists/jammy/Release failed: 500 read timeout +Jul 27 00:54:50 vvr1-dc0 dc0-mirror-sync[1560492]: Failed to download some Release, Release.gpg or InRelease files! +Jul 27 00:54:54 vvr1-dc0 systemd[1]: dc0-mirror-sync.service: Failed with result 'exit-code'. +Jul 27 00:54:54 vvr1-dc0 systemd[1]: Failed to start dc0-mirror-sync.service - dc0 mirror upstream sync (dc-mirror.sh; D-135 item 1: apt+UCA). + +--- 3. dc1 (apt caching proxy is the RULED artifact path, D-135 amendment) --- + OK apt-cacher-ng installed + OK /etc/apt-cacher-ng/acng.conf.d/dc1-cache-proxy.conf matches + OK apt-cacher-ng active + OK listening on :3142 + OK 10.12.68.4/22 on virbr6 (vr1-dc1-metal-admin) + OK default route via edge 10.12.64.1 + OK proxy serves http://archive.ubuntu.com/ubuntu/dists/jammy/Release (200) + OK proxy serves http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/jammy-updates/caracal/Release (200) +dc-cache-proxy check (dc1): PASS + +--- 4. dc1 dormant debmirror: the fallback, and how fragile its dormancy is --- + OK /usr/local/sbin/dc1-mirror-net-apply matches + OK /etc/systemd/system/dc1-mirror-net.service matches + OK /etc/systemd/resolved.conf.d/dc1-mirror.conf matches + OK /usr/local/sbin/dc1-mirror-sync matches + OK /etc/systemd/system/dc1-mirror-sync.service matches + OK /etc/systemd/system/dc1-mirror-sync.timer matches + OK /etc/nginx/sites-available/dc1-mirror matches + OK /etc/nginx/sites-enabled/dc1-mirror enabled + OK dc1-mirror-net.service enabled + OK dc1-mirror-sync.timer enabled + OK 10.12.68.4/22 on virbr6 (vr1-dc1-metal-admin) + OK default route via edge 10.12.64.1 + OK nginx active + OK mirror answers http://10.12.68.4/ (200) + MISS last-sync UNKNOWN: status says RUNNING (RUNNING 2026-07-23T21:49:35Z) but dc1-mirror-sync.service is NOT active -- the sync died without writing a terminal state (stale marker, not a success) +dc-mirror check (dc1): FAIL +RUNNING 2026-07-23T21:49:35Z +330G /var/lib/dc-mirror/ubuntu +enabled +NextElapseUSecRealtime= +LastTriggerUSec=Fri 2026-07-24 00:52:13 UTC +Result=signal +ActiveState=failed + +READING OF 4: the timer is ENABLED with an EMPTY next-elapse because the service sits in +failed/Result=signal from the 2026-07-24 pause-kill. D-135-amended ruling says dc1's +debmirror is PAUSED and 'kept dormant as the fallback' -- but nothing ENFORCES that. The +dormancy is a side effect of a failed unit, not a declared state; a systemctl reset-failed +would re-arm a 330G -> ~949G pull on a rack whose artifact path is now the proxy. diff --git a/docs/changelog-20260727-creds-consolidation.md b/docs/changelog-20260727-creds-consolidation.md index fcc6603..a47b9fd 100644 --- a/docs/changelog-20260727-creds-consolidation.md +++ b/docs/changelog-20260727-creds-consolidation.md @@ -179,6 +179,54 @@ Revert: drop the 13 appended lines from `permissions.allow` in `.claude/settings.json`. No other field was touched, so the revert is a contiguous deletion. +## 7. Stage 4 close-out opened: `dc-mirror.sh check` was FALSE-GREENING the mirror gate + +Operator direction: close Stage 4 before resuming the DC1 Stage-5 chain. The Stage 4 DoD +(`runbooks/dc-dc-phase3-maas-enlist-deploy.md:472-485`) has six bullets; four are already met +and captured. Verifying the fifth -- "per-DC mirror reachable" -- found that **the check the +gate would close on could not fail.** + +`scripts/dc-mirror.sh:271-273` tested `[ -f last-sync.status ]` and printed the file's contents +behind an unconditional `OK`. So the status word was displayed but never asserted on, and the +check exited PASS regardless. MEASURED on both racks, 2026-07-27: + +- **dc0** read `FAIL 2026-07-27T00:54:54Z ubuntu=255 uca=0` -> printed `OK`, check PASS. +- **dc1** read `RUNNING 2026-07-23T21:49:35Z` -- a marker FOUR DAYS stale, left by the + debmirror the D-135 amendment killed with a signal, so it never wrote a terminal state -> + printed `OK`, check PASS. + +FIXED: the status word is now case-analysed. `OK*` passes; `FAIL*` misses; an ABSENT file +misses (it was a passing `note`, but nothing has attested the content); an unrecognised word +REFUSES rather than defaulting to success. `RUNNING*` is deliberately neither -- in-flight and +died-mid-run are indistinguishable from the file alone, so it reports an explicit UNKNOWN and +cross-checks the unit: a RUNNING marker with no active unit is named as a corpse, not a +success. Harness `tests/dc-mirror/run-tests.sh` **24/24** (was 19); T20-T24 lock each status +word, T24 specifically locks out the return of the exact defective shape. + +PROOF THE FIX WORKS -- re-run against both live racks, and the gate is now honestly NOT MET: +`dc-mirror check (dc0): FAIL` on the failed sync, `dc-mirror check (dc1): FAIL` on the stale +marker. Capture `docs/audit/stage4-mirror-gate-20260727.txt`. + +SUBSTANCE BEHIND THE TWO FAILURES (measured, not inferred): + +- **dc0's mirror CONTENT is complete and intact** -- 949G ubuntu + 342M cloud-archive, all + three dists (`jammy`, `jammy-updates`, `jammy-security`) plus `pool` and `project`. What + failed is the overnight INCREMENTAL: `Download of dists/jammy/Release failed: 500 read + timeout` -> "Failed to download some Release, Release.gpg or InRelease files!" -> debmirror + aborted the ubuntu leg with 255. The UCA leg succeeded (`uca=0`). A transient upstream fetch + failure, not mirror damage. The daily timer re-arms for 2026-07-28 00:14:48. +- **dc1's debmirror is dormant only ACCIDENTALLY.** `dc1-mirror-sync.timer` is `enabled` with + an EMPTY `NextElapseUSecRealtime`, because the service sits in `failed`/`Result=signal` from + the 07-24 pause. The D-135 amendment RULED it paused and "kept dormant as the fallback", but + nothing enforces that -- the dormancy is a side effect of a failed unit, not a declared + state, so a `systemctl reset-failed` (or possibly a reboot) re-arms a 330G -> ~949G pull on + a rack whose ruled artifact path is now the proxy. dc1's PROXY meanwhile checks **PASS** + genuinely: apt-cacher-ng active on `.4:3142`, serving both archive and UCA `Release` with 200. + +Revert: `git revert` this commit's `scripts/dc-mirror.sh` + `tests/dc-mirror/run-tests.sh` +changes -- the check returns to passing unconditionally on last-sync. No live state was +altered by this item; both racks were read only. + --- ## Findings LOGGED, not actioned (hard rule 1) diff --git a/scripts/dc-mirror.sh b/scripts/dc-mirror.sh index 81db53c..e409a76 100755 --- a/scripts/dc-mirror.sh +++ b/scripts/dc-mirror.sh @@ -268,9 +268,40 @@ [ "$HTTP" = "200" ] && say "OK mirror answers http://${LISTEN}/ (200)" \ || miss "mirror not answering on http://${LISTEN}/ (got '$HTTP')" fi - [ -f "${MIRROR_ROOT}/last-sync.status" ] \ - && say "OK last-sync: $(cat "${MIRROR_ROOT}/last-sync.status")" \ - || say "note last-sync.status absent (no sync attempted yet)" + # ASSERT ON THE STATUS CONTENT, not merely on the file existing. The original form here + # tested `[ -f ]` and printed whatever the file said behind an unconditional "OK", so a + # sync that FAILED and a sync that never finished both read as green and the check still + # exited PASS. MEASURED 2026-07-27, both racks, at the Stage 4 mirror gate: dc0 read + # "FAIL 2026-07-27T00:54:54Z ubuntu=255 uca=0" (a transient upstream 500 on + # dists/jammy/Release) and dc1 read "RUNNING 2026-07-23T21:49:35Z" -- a marker four days + # stale, left by the debmirror the D-135 amendment PAUSED with a signal, so it never wrote + # a terminal state. Both printed OK. A gate that cannot distinguish "synced" from "failed" + # or "died mid-run" is not a gate, and this is the check the stage closes on (GA-R6). + # + # RUNNING is deliberately NOT a pass and NOT a plain failure: it is either a sync genuinely + # in flight or a corpse, and the two are indistinguishable from the file alone -- so it is + # reported as an explicit UNKNOWN that the caller must resolve, never silently accepted. + # The cross-check is the unit: a RUNNING marker with no active unit is a corpse. + LSS="${MIRROR_ROOT}/last-sync.status" + if [ ! -f "$LSS" ]; then + miss "last-sync.status absent -- no sync has ever completed, so the mirror content is not attested" + else + LSTXT="$(tr -d '\r' < "$LSS" | head -1)" + case "$LSTXT" in + OK*) + say "OK last-sync: $LSTXT" ;; + RUNNING*) + if systemctl is-active --quiet "${SITE}-mirror-sync.service" 2>/dev/null; then + miss "last-sync UNKNOWN: a sync is IN FLIGHT ($LSTXT) -- re-run this check once it finishes; the mirror is not yet attested complete" + else + miss "last-sync UNKNOWN: status says RUNNING ($LSTXT) but ${SITE}-mirror-sync.service is NOT active -- the sync died without writing a terminal state (stale marker, not a success)" + fi ;; + FAIL*) + miss "last-sync FAILED: $LSTXT -- the mirror content is NOT attested current; investigate before any gate cites this check" ;; + *) + miss "last-sync UNRECOGNISED: '$LSTXT' -- refusing to interpret an unknown status word as success" ;; + esac + fi [ "$FAILED" -eq 0 ] && { echo "dc-mirror check ($SITE): PASS"; exit 0; } echo "dc-mirror check ($SITE): FAIL"; exit 1 } diff --git a/tests/dc-mirror/run-tests.sh b/tests/dc-mirror/run-tests.sh index ffbe242..9200171 100755 --- a/tests/dc-mirror/run-tests.sh +++ b/tests/dc-mirror/run-tests.sh @@ -108,5 +108,41 @@ && ok "T19 sync unit sets HOME (debmirror GPG home)" \ || no "T19 sync unit sets HOME (debmirror GPG home)" +# T20-T24: the last-sync verdict. MEASURED false green, 2026-07-27, at the Stage 4 mirror +# gate: the check tested `[ -f last-sync.status ]` and printed the contents behind an +# unconditional "OK", so dc0's "FAIL ... ubuntu=255" and dc1's four-day-stale "RUNNING" +# (left by the debmirror the D-135 amendment killed with a signal) BOTH read green and the +# check exited PASS. This is the check a stage gate closes on (GA-R6), so each status word +# gets its own regression lock. + +# T20: a FAILED sync must fail the check, not decorate it. +awk '/FAIL\*\)/{f=1} f&&/miss /{print "hit"; exit}' "$SCRIPT" | grep -q hit \ + && ok "T20 last-sync FAIL routes to miss (the measured dc0 false green)" \ + || no "T20 last-sync FAIL routes to miss (the measured dc0 false green)" + +# T21: RUNNING is never a pass -- in flight and died-mid-run are indistinguishable from the +# file alone, so BOTH branches must be misses (the measured dc1 stale-marker false green). +[ "$(awk '/RUNNING\*\)/{f=1} f&&/^[[:space:]]*(esac|FAIL\*\))/{exit} f&&/miss /{n++} END{print n+0}' "$SCRIPT")" -ge 2 ] \ + && ok "T21 last-sync RUNNING is never a pass, in-flight AND stale-marker both miss" \ + || no "T21 last-sync RUNNING is never a pass, in-flight AND stale-marker both miss" + +# T22: an ABSENT status file means no sync ever completed -- previously a `say note`, i.e. a +# pass. Nothing has attested the mirror content, so it must fail. +awk '/! -f "\$LSS"/{f=1} f&&/miss /{print "hit"; exit}' "$SCRIPT" | grep -q hit \ + && ok "T22 absent last-sync.status is a miss (was a passing note)" \ + || no "T22 absent last-sync.status is a miss (was a passing note)" + +# T23: no silent default. An unrecognised status word must refuse rather than be read as OK +# -- the same no-silent-default discipline the D-137 checker enforces on stage tokens. +awk '/UNRECOGNISED/{print "hit"; exit}' "$SCRIPT" | grep -q hit \ + && ok "T23 an unrecognised status word refuses (no silent default to success)" \ + || no "T23 an unrecognised status word refuses (no silent default to success)" + +# T24: the exact defective shape must not come back -- an OK printed straight from a cat of +# the status file, with no case analysis in between. +grep -q 'say "OK last-sync: \$(cat' "$SCRIPT" \ + && no "T24 the existence-only last-sync shape is gone" \ + || ok "T24 the existence-only last-sync shape is gone" + echo; echo "dc-mirror: $PASS passed, $FAIL failed" [ "$FAIL" -eq 0 ]