diff --git a/.claude/skills/openstack-cloud-ops/references/platform-traps.md b/.claude/skills/openstack-cloud-ops/references/platform-traps.md index 0aa8049..28e44ec 100644 --- a/.claude/skills/openstack-cloud-ops/references/platform-traps.md +++ b/.claude/skills/openstack-cloud-ops/references/platform-traps.md @@ -421,6 +421,51 @@ --- +## 5. systemd timers -- "stopped" is not "dormant" across a reboot + +Measured on the dc1 rack, 2026-07-27, while auditing a debmirror that had been paused +by ruling three days earlier. + +**5a. A STOPPED-but-ENABLED timer re-arms at boot, and `Persistent=yes` makes it fire +IMMEDIATELY.** The state that looks dormant: + +``` +systemctl is-enabled .timer -> enabled +systemctl is-active .timer -> inactive (dead) +systemctl show .timer -p NextElapseUSecRealtime -> (EMPTY) +systemctl show .timer -p Persistent -> yes +``` + +An empty `NextElapseUSecRealtime` is the trap: nothing is scheduled *right now*, so the +unit reads harmless and `systemctl list-timers` shows `NEXT: -`. But `enabled` means +systemd STARTS the timer at the next boot, and `Persistent=yes` means a timer whose +scheduled time has already elapsed triggers **at once** on activation to catch up. So a +job paused days ago does not wait for its next window -- it runs seconds after boot. On +this project that meant a multi-hundred-GB debmirror on a rack whose ruled artifact path +had changed, and the vector is recurring: DC racks live inside containment VMs on the +vcloud host, so a vcloud reboot reboots every rack. + +Corollaries worth keeping: +- **`systemctl reset-failed` on the SERVICE does not re-arm the TIMER.** If the timer is + inactive, clearing the service's failed state changes nothing schedulable. An earlier + reading of this same incident blamed `reset-failed` and was wrong; the vector is the + reboot (correction appended to `docs/audit/stage4-mirror-gate-20260727.txt`). +- **`Result=signal` + `ExecMainStatus=15` on a oneshot means someone SIGTERM'd it**, i.e. + it was killed mid-run and never wrote a terminal state -- which is also why its status + file can sit at `RUNNING` indefinitely. Do not read a stale `RUNNING` as in-flight; + cross-check `systemctl is-active`. +- To make a pause DECLARED rather than incidental: `systemctl disable --now .timer` + (survives reboot) or `mask` (also survives a later `enable`, e.g. from an idempotent + installer re-running). Leaving it merely stopped encodes the intent nowhere. + +**5b. A `oneshot` with `RemainAfterExit=yes` does not undo its work when stopped.** So +`systemctl stop ` proves NOTHING about whether something else still depends on +the addresses/routes it placed. To test a dependency for real, remove the live state +(`ip addr del`, `ip route del`) and re-apply from the unit you believe owns it. Stopping +the unit and seeing everything still work is a false green. + +--- + ## Verbatim error -> cause index (grep this first) | Exact string you see | Cause | Where | @@ -437,6 +482,9 @@ | a Kea/firewall API `set` returns success but the box's behaviour does not change | no `service/reconfigure` (or no firewall `apply`) call | 4c | | MAAS cannot talk to an LXD VM host after an LXD update | LXD auto-refreshed past 6.6; MAAS 3.6/3.7 needs LXD <= 6.6 or 5.21 LTS | 3a | | a MAAS-composed VM gets a lease MAAS did not issue, or PXEs on the wrong L2 | `lxdbr0` runs its own dnsmasq (verified); the composing LXD profile's bridge is the other candidate (unconfirmed -- measure it) | 3b, 3c | +| a timer reads `is-enabled=enabled` but `is-active=inactive` and `NextElapseUSecRealtime` is EMPTY, and `list-timers` shows `NEXT: -` | it is stopped, NOT dormant -- `enabled` restarts it at boot and `Persistent=yes` then fires it immediately | 5a | +| a `last-sync`/status file sits at `RUNNING` for days | the oneshot was SIGTERM'd (`Result=signal`, `ExecMainStatus=15`) and never wrote a terminal state; cross-check `is-active` before believing it | 5a | +| stopping a `-net.service` changes nothing, so "nothing depends on it" | a `oneshot` with `RemainAfterExit=yes` does not undo its work on stop -- delete the live addr/route to test dependency for real | 5b | --- diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 6ca8746..ece3a09 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -52,8 +52,29 @@ (DOCFIX-204, unsatisfiable under D-129(iv)); node-side half split to gate **G17**. TRAVELLING FORWARD by design: G17, 21 open SEC rows, and 7 residual credential-register findings incl. the dc0 edge-API re-mint. **The NEXT stage - branches off post-merge `main`.** History of the stage while it was open - follows. + branches off post-merge `main`.** + **POST-CLOSE DURABILITY SWEEP 2026-07-27** (operator-requested before clearing the + session; capture `docs/audit/queued-findings-20260727.txt`, precedent + `queued-findings-20260726.txt`). Four transcript-only items were landed on surfaces, of + which one was consequential: **`dc-mirror.sh`'s dc1 site row carried no warning that dc1 + is no longer a mirror site**, so `dc-mirror.sh install dc1` would have silently rebuilt + the whole removed apparatus -- enabled daily debmirror timer included, and a fresh ~950G + pull -- which is exactly what someone would reach for on treating a failing `check dc1` + as a regression. The row now states dc1 is proxy-only by the D-135 amendment, that + `check dc1` FAILS BY DESIGN, and that `install dc1` is a deliberate strategy change and + never a repair; a RUNTIME guard is queued, not built (hard rule 1). Also: a WRONG causal + claim in `stage4-mirror-gate-20260727.txt` was superseded by an appended correction + (`reset-failed` cannot re-arm an inactive timer -- the vector was a REBOOT, via + `Persistent=yes` firing immediately on a missed window); the generalisable lesson is now + platform-traps section 5 ("stopped is not dormant across a reboot"; and a `oneshot` with + `RemainAfterExit=yes` does not undo its work on stop, so stopping it proves NOTHING about + dependents -- the reason the dc1 teardown test deleted the live addr/route instead); and + `dc-cache-proxy.sh`'s two-net-units-coexist claim is now marked REASONED, NOT MEASURED + (hard rule 2 -- nobody has run both on one host). Dangling-reference sweep: every path + this session introduced or cited RESOLVES; the pre-existing dangles are all legitimate + (deleted-as-history, not-yet-built, or the deliberately-absent octavia PKI overlay that + preflight P4 fails on). ledger-scan reconciled: 21 open SEC, next-free D 138 / DOCFIX 205 + / BUNDLEFIX 053. History of the stage while it was open follows. Precondition PASS + discovery CAPTURED read-only (`docs/audit/stage4-discovery-20260723.txt`): both DC racks enrolled (vvr1-dc0 `7chphy`, vvr1-dc1 `nmpcq4`); 9 nodes per DC ALL Ready, diff --git a/docs/audit/queued-findings-20260727.txt b/docs/audit/queued-findings-20260727.txt new file mode 100644 index 0000000..83b2093 --- /dev/null +++ b/docs/audit/queued-findings-20260727.txt @@ -0,0 +1,92 @@ +=== 2026-07-27 close-out sweep: content that existed ONLY in the session transcript === +repo HEAD: 93ce463689d6aff6b1a8d02734a29ba3040ed9e8 branch: main +Precedent: docs/audit/queued-findings-20260726.txt. Operator asked for a durability sweep +before clearing the session. NOTHING here is ruled or built; each item names its status. + +--- PART A: FIXED BY THIS SWEEP (were transcript-only, now on a surface) --- + +A1. dc-mirror.sh's dc1 site row carried NO warning that dc1 is no longer a mirror site. + THE MOST CONSEQUENTIAL MISS OF THE SWEEP: `dc-mirror.sh install dc1` would have + silently rebuilt everything the 2026-07-27 teardown removed -- units, nginx vhost, + sync helper, AND an ENABLED daily debmirror timer that would start a fresh ~950G pull + on a rack whose ruled artifact path is the proxy. Someone treating a failing + `check dc1` as a regression would reach for exactly that command. FIXED: the row now + states dc1 is proxy-only by the D-135 amendment, that `check dc1` FAILS BY DESIGN, + and that `install dc1` is a deliberate strategy change, never a repair. The row is + RETAINED so the measured values survive for a legitimate rebuild. + QUEUED (not built, hard rule 1): a RUNTIME guard in do_install for a site whose ruled + path is not the mirror -- today's own lesson is that prose-only prevention does not + fire (DoD item 8 existed and missed dc1; D-137's wiring sat as prose and missed BOTH + DC standups). A comment is strictly weaker than a guard and is an interim. + +A2. docs/audit/stage4-mirror-gate-20260727.txt carried a WRONG causal claim -- that + `systemctl reset-failed` would re-arm the paused dc1 debmirror. Measured afterwards: + reset-failed on the SERVICE cannot start an INACTIVE TIMER. The real vector was a + REBOOT (enabled -> timer starts at boot; Persistent=yes -> fires immediately because + the window had passed). CORRECTION APPENDED to that capture rather than edited in + place -- it is dated evidence, so a wrong claim is superseded openly. + +A3. The generalisable lesson behind A2 was unrecorded. Now platform-traps section 5: + 'stopped is not dormant across a reboot' (5a) plus 'a oneshot with RemainAfterExit=yes + does not undo its work on stop, so stopping it proves NOTHING about dependents' (5b), + with three new rows in the verbatim-error index. 5b is the reason the dc1 teardown was + safe: the independence test deleted the live addr/route instead of stopping the unit. + STILL LIVE: dc0-mirror-sync.timer is enabled with Persistent=yes, so the 5a shape + applies to dc0 today -- benign while its syncs succeed, but it is the same shape. + +A4. dc-cache-proxy.sh stated coexistence of the two net units on one host as FACT. + It is REASONED, not MEASURED (hard rule 2): the proxy has only ever been installed on + dc1, and the mirror net unit was removed there, so nobody has run both on one host. + The header now says so and tells a future session to verify rather than trust the + sentence if dc0 is ever given the proxy. + +--- PART B: MEASURED THIS SESSION, recorded here because they had no other home --- + +B1. The region MAAS snap's ssh config contains exactly ONE Host block (dc1's transit + 172.31.0.6 -> id_dc1_power, IdentitiesOnly yes). dc0 has NO block, so dc0 power ops + use the snap's DEFAULT identity id_ed25519. This MATCHES SEC-016's stated design + ('172.31.0.2/dc0 keeps SEC-012'), so it is confirmation, not drift -- but it is the + measured basis for SEC-016's snap-refresh fragility note: a refresh that wipes + /var/snap/maas/current/root/.ssh/ takes dc1's Host block with it, and dc0 would keep + working on the default identity while dc1 silently lost power control. Asymmetric + failure, so verify BOTH DCs after any snap refresh, not just one. + +B2. MAAS fabric-3 holds 192.168.122.0/24 -- the libvirt DEFAULT network, discovered by + MAAS. Load-bearing for nothing in VR1 and outside the flagged carve residue, so it + was deliberately KEPT. Noted as a possible future cleanup, NOT proposed: it is a + MAAS-discovered fact about the host, and removing discovered state has no ruling. + +B3. dc0's mirror sync logs 'cannot delete non-empty directory: project/trace' on every + run. Cosmetic debmirror cleanup noise, present on successful runs too (verified on the + 2026-07-27 08:43 run that exited 0). NOT a failure signal -- recorded so a future + reader of the journal does not chase it. + +--- PART C: DEFERRED BY RULING (recorded so they are not later read as oversights) --- + +C1. SEC-024's two pre-* state-surgery snapshots: RETAINED by ruling 2026-07-27 ('Keep + both'). SEC-024 stays OPEN as a standing WATCH, not an open remediation; the umask + CAUSE is unfixed and preflight P5 is the detection. +C2. dc0 edge API credential re-mint (sole remaining SEC-021(a) item): deliberately + EXCLUDED from the consolidation batch as a live edge mutation. Accounts for 2 of the + 7 residual credential-register findings. +C3. The 3 residual S5 power-key asymmetries are RULED BY SEC-016, not defects. The + register has no way to record a ruled exception; suppressing them silently would be + wrong and renaming live files to satisfy a checker would be worse. Needs an operator + decision on mechanism. +C4. tests/creds-matrix T24's finding-class baseline covers TIER 1 ONLY -- tier-2/3 classes + (E1/E3/E4/V1/V2) have no baselined red state. Also in the session-ledger bookend. +C5. scripts/creds-mint.sh: unbuilt, unruled advice from the D-137 close. Stage 5 is the + largest credential-minting event in the deployment, so ruling it BEFORE the bundle + deploy is worth more than after. +C6. allow-vs-ask permission precedence is UNVERIFIED here. A useful allow entry for + creds-matrix.py was deliberately SKIPPED rather than risk a broad prefix silently + downgrading the --privileged ask rule. Measuring it unblocks that entry. + +--- PART D: verification run by this sweep --- +D1. Dangling-reference sweep over docs/*.md, runbooks/, scripts/, and the skill: every + path introduced or cited by this session RESOLVES. The pre-existing dangles are all + legitimate -- deliberately deleted files cited as history (the D-112 config.xml + renderers), not-yet-built planned files (overlays/vr1-dc0-vips.yaml, creds-mint.sh), + and the deliberately-absent overlays/octavia-pki.yaml that preflight P4 fails on. +D2. ledger-scan values reconciled against the machine-derived block: 21 open SEC rows, + D next-free 138, DOCFIX 205, BUNDLEFIX 053 -- all match. diff --git a/docs/audit/stage4-mirror-gate-20260727.txt b/docs/audit/stage4-mirror-gate-20260727.txt index c1e9632..40bbd65 100644 --- a/docs/audit/stage4-mirror-gate-20260727.txt +++ b/docs/audit/stage4-mirror-gate-20260727.txt @@ -88,3 +88,30 @@ debmirror is PAUSED and 'kept dormant as the fallback' -- but nothing ENFORCES that. The dormancy is a side effect of a failed unit, not a declared state; a systemctl reset-failed would re-arm a 330G -> ~949G pull on a rack whose artifact path is now the proxy. + +================================================================================ +CORRECTION APPENDED 2026-07-27 (same day, after further measurement). Appended +rather than edited in place: this file is dated evidence, so a wrong analytical +claim inside it is superseded openly, not quietly rewritten. + +THE CLAIM ABOVE (section 4 reading, line ~89) IS WRONG: + "a systemctl reset-failed would re-arm a 330G -> ~949G pull" + +MEASURED afterwards, the re-arm vector is a REBOOT, not reset-failed: + dc1-mirror-sync.timer is-enabled = enabled is-active = INACTIVE (dead) + NextElapseUSecRealtime = (empty) Persistent = yes + LastTriggerUSec = Fri 2026-07-24 00:52:13 UTC + +The timer was STOPPED (not disabled) at the 07-24 pause. Nothing is scheduled while +it is inactive, so `reset-failed` on the SERVICE does not start the TIMER and would +NOT have re-armed anything. What re-arms it is a reboot: `enabled` means systemd +starts the timer at boot, and `Persistent=yes` means a timer whose scheduled time has +already passed fires IMMEDIATELY on activation to catch up -- so the pull would have +begun at once, not at the next 00:xx window. That vector was real and recurring: the +rack lives inside vvr1-dc1 on the vcloud host, so a vcloud reboot reboots the rack. + +MOOT AS OF THE SAME DAY: dc1's mirror was REMOVED entirely +(docs/audit/dc1-mirror-teardown-20260727.txt), so no dc1 timer exists to re-arm. The +correction is recorded because the CAUSAL LESSON generalises and dc0's timer is still +enabled with Persistent=yes -- see the systemd-timer entry in the operating skill's +platform-traps reference. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index fd72f41..5cdfe30 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -256,5 +256,14 @@ the gauntlet red. Same class as the two false greens this session fixed. Also still open from the D-137 close: `creds-mint.sh` is unbuilt/unruled advice and Stage 5 is the largest minting event, so it is worth ruling BEFORE the bundle deploy rather than after. +- POST-CLOSE DURABILITY SWEEP (operator-requested): 4 transcript-only items landed on surfaces + -- worst was `dc-mirror.sh`'s dc1 row carrying NO warning that dc1 is proxy-only, so + `install dc1` would silently rebuild the removed apparatus + enabled timer + ~950G pull. + Also: a WRONG causal claim in the mirror-gate capture superseded by appended correction + (reset-failed cannot re-arm an inactive timer; the vector was a REBOOT via Persistent=yes); + platform-traps section 5 added; the two-net-units coexistence claim marked REASONED-NOT- + MEASURED. Capture `docs/audit/queued-findings-20260727.txt`. QUEUED: a runtime install + guard for a non-mirror site (a comment is strictly weaker and prose-only prevention has + already failed twice here). - Gauntlet ALL GREEN (81), repo-lint 0-fail. Full body: `docs/archive/changelogs/changelog-20260727-creds-consolidation.md`; stage record `docs/archive/stage-records/vr1-stage4-record.md`. Status ONLY in CURRENT-STATE.md. diff --git a/scripts/dc-cache-proxy.sh b/scripts/dc-cache-proxy.sh index 503d687..040ee37 100644 --- a/scripts/dc-cache-proxy.sh +++ b/scripts/dc-cache-proxy.sh @@ -37,7 +37,12 @@ # here, so a proxy-only DC has NO mirror-named dependency and is independently # rebuildable. The two net units are SAFE TO COEXIST on a site running both # (`ip addr replace` / `ip route replace` are idempotent and both set the same -# MEASURED values), so this is additive for dc0, which keeps its mirror. +# MEASURED values), so this should be additive for dc0, which keeps its mirror. +# COEXISTENCE IS REASONED, NOT MEASURED (hard rule 2): the proxy has only ever been +# installed on dc1, where the mirror net unit was then removed. Nobody has run both +# units on one host. The reasoning is that `ip addr replace` / `ip route replace` are +# idempotent and both units write the SAME measured values, so the second is a no-op -- +# but if dc0 is ever given the proxy, VERIFY that rather than trusting this sentence. # Roosevelt: a DC's artifact-delivery strategy is a per-DC choice that must be # changeable without a rebuild of the other strategy's apparatus. # diff --git a/scripts/dc-mirror.sh b/scripts/dc-mirror.sh index e409a76..1964221 100755 --- a/scripts/dc-mirror.sh +++ b/scripts/dc-mirror.sh @@ -80,6 +80,20 @@ EDGE_GW="10.12.4.1" ;; dc1) + # !! dc1 IS NOT A MIRROR SITE ANY MORE (D-135 AMENDMENT 2026-07-27). Operator: + # "DC0 is the test of a full mirror. DC1 is the test of the mirror proxy." dc1's + # partial mirror, its units, timer, sync helper, nginx vhost, resolved drop-in and + # the debmirror package were ALL REMOVED that day + # (docs/audit/dc1-mirror-teardown-20260727.txt). dc1's artifact path is + # scripts/dc-cache-proxy.sh (apt-cacher-ng on .4:3142). + # -> `check dc1` FAILS BY DESIGN. That is not a regression; use dc-cache-proxy.sh. + # -> `install dc1` would SILENTLY REBUILD the removed apparatus, including an + # ENABLED daily debmirror timer that would start a fresh ~950G pull on a rack + # whose ruled path is the proxy. Only do that as a deliberate, ruled decision + # to change dc1's strategy back -- not as a "repair". + # The row is RETAINED (not deleted) so the measured values survive for exactly that + # deliberate rebuild, and so this warning has somewhere to live. A runtime guard is a + # QUEUED finding, not built here (hard rule 1). # MEASURED/RULED 2026-07-23 (same session; dc1 values from the D-124 # amendment plane map + lib-net vr1-dc1 arm): # LISTEN 10.12.68.4/22 on vr1-dc1-metal-admin -- utility-band .4.