diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 4c0d451..815f588 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -142,9 +142,16 @@ observable (correction queued). **Two of the agent's own isolation experiments were INVALID and must not be cited** (other nodes were still running; and a re-commission did not restart MAAS's timer) -- so contention - remains a LIVE hypothesis, not a refuted one. Failed commissioning is - re-runnable; nothing is lost. Session changelog item 19 records the clean - experiment to run next. REMAINING IN G10: this diagnosis, then netem (step E, + remains a LIVE hypothesis, not a refuted one. **CLEAN experiment now RUN + (item 20): a genuinely isolated node still failed at 1770s (~29.5 of 30 + min)** -- that refutes CONTENTION but is consistent with INHERENTLY SLOW, + and the batch pattern 3-pass/6-fail-at-the-mark is the signature of a + MARGINAL 30-min timeout over slow depth-4 nested I/O. 3 nodes reached Ready + on this exact rack/subnet/metadata path, so metadata is NOT globally broken + (rack :5248 up, rack->region 301). LEADING HYPOTHESIS + cheap decisive test, + needing an operator decision (MAAS-wide config): raise `node_timeout` and + commission one node. Failed commissioning is re-runnable; nothing is lost. + REMAINING IN G10: this diagnosis, then netem (step E, NOT started -- target is the dc0<->dc1 mesh = **virbr5 on vcloud**, measured; `modules/netem-link` assumes PASSWORDLESS SUDO which vcloud's operator account lacks). Session changelog item 18. diff --git a/docs/changelog-20260719-dc0-deploy-stepB.md b/docs/changelog-20260719-dc0-deploy-stepB.md index e994ff0..c039ff1 100644 --- a/docs/changelog-20260719-dc0-deploy-stepB.md +++ b/docs/changelog-20260719-dc0-deploy-stepB.md @@ -760,6 +760,50 @@ `counter` to the SEC-010 rules so the node->region hypothesis becomes readable, and check the kernel cmdline MAAS actually serves. +## 20. CLEAN commissioning experiment: contention REFUTED (properly this time) + +- Ran the isolated experiment the earlier invalid ones only pretended to: + ALL node domains OFF (verified 0 running), 397 GiB free, load 0.05, canary + in a TERMINAL `Failed commissioning` state (so the commission restarts + MAAS's 30-min timer), commission EXACTLY ONE, poll READ-ONLY only, touch + nothing. +- RESULT: `Failed commissioning after 1770s` (~29.5 min = the 30-min + timeout), a SINGLE node entirely alone. **Contention is refuted** -- it is + not I/O bandwidth, not memory, not the metadata service being swamped. The + failure is systemic and per-node. +- This CORRECTS changelog item 18-19's contention hypothesis (which rested on + contaminated runs). The event trace is consistent every time: `Commissioning` + -> `Performing PXE boot` -> `Loading ephemeral` (~40s later) -> 30 min of + silence -> timeout. The node gets the ephemeral image handoff and then never + reports back. +- REFRAME after further measurement: "contention refuted" is right but too + narrow. The single-node fail refutes CONTENTION (competing for resources) + but is fully CONSISTENT with INHERENTLY SLOW -- the node may have still been + working at 30 min, just not done. And the batch result **3 pass / 6 fail + exactly at the 30-min mark** is the classic signature of a MARGINAL TIMEOUT + over a slow, variable operation: the faster-scheduled nodes finish under the + wire, the rest are still going when the timer fires. 1770s is 29.5 of 30. +- KEY CLUE against a network/metadata block: **3 nodes DID reach Ready on this + exact rack, subnet, and metadata proxy.** So the metadata path is NOT + globally broken; a "node never reaches :5248" theory has to explain why it + works for 3 identical nodes. Infra confirmed UP: rack `:5248` listening (2 + sockets), rack->region 5240 = 301. A `:5248` tcpdump returned 0 packets but + its 220s window likely closed before the ephemeral OS reached the + metadata-fetch stage -- INCONCLUSIVE (a timing race), not evidence. +- **LEADING HYPOTHESIS (operator-facing): MAAS's default 30-min commissioning + timeout is too short for depth-4 nested commissioning I/O.** Cheap, decisive + test that needs no stall-pinpointing: raise `node_timeout` + (`maas admin maas set-config name=node_timeout value=`, e.g. 60), + commission ONE node, see if it completes. If it does, the fix is a config + value (and a Roosevelt note: bare metal commissions faster, so this is a + VR1-nested tuning, not a Roosevelt default). If it still fails well past + 30 min, the timeout is exonerated and the next probe is the metadata-proxy + access log / a console screenshot (needs the graphics-device correction to + the node-vm console -- item 19). +- STOPPED the live diagnosis here per the reconciliation circuit-breaker: + many diagnostic cycles, and the remaining hypotheses need an operator + decision (change a MAAS-wide config) rather than more agent spelunking. + ## 8. STAGE-CLOSE QUEUE ITEM (operator directive, 2026-07-20): OPNsense 26.7 review - Operator directive (verbatim intent): at the proper time/place, review the