diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 815f588..88b1d95 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -150,7 +150,18 @@ on this exact rack/subnet/metadata path, so metadata is NOT globally broken (rack :5248 up, rack->region 301). LEADING HYPOTHESIS + cheap decisive test, needing an operator decision (MAAS-wide config): raise `node_timeout` and - commission one node. Failed commissioning is re-runnable; nothing is lost. + commission one node. **DIAGNOSTIC COMMITTEE run 2026-07-20 (4 independent + reviewers, `docs/audit/commissioning-committee-20260720.md`) REFUTED that + hypothesis 4/4** -- 30 min of SILENCE is a hang, not slow progress; a longer + clock cannot fix a hang, and the proposed one-node test was CONFOUNDED + (changed timeout + concurrency together). Post-committee reads: **MTU branch + EXONERATED** (metal-admin MAAS VLAN MTU is 1500, so the guest never goes + jumbo); region healthy at rest. STILL-LIVE causes, both needing observation + DURING a run: region Temporal starvation, and a commissioning-only script + hang on nested-virt hardware. Decisive gated test (supersedes node_timeout): + one commission with `console=ttyS0` on the kernel + a full-window, + lease-IP-keyed capture on virbr2 + enp1s0. Failed commissioning is + re-runnable; nothing is lost. REMAINING IN G10: this diagnosis, then netem (step E, NOT started -- target is the dc0<->dc1 mesh = **virbr5 on vcloud**, measured; `modules/netem-link` assumes PASSWORDLESS SUDO which vcloud's diff --git a/docs/audit/commissioning-committee-20260720.md b/docs/audit/commissioning-committee-20260720.md new file mode 100644 index 0000000..bab0302 --- /dev/null +++ b/docs/audit/commissioning-committee-20260720.md @@ -0,0 +1,141 @@ +# Diagnostic committee — MAAS commissioning failure (2026-07-20) + +Four independent reviewers (paper review, no live-cloud access) over one shared +evidence packet. Lenses: MAAS commissioning internals; nested-virt/boot +performance; network/MTU; adversarial assumption-breaker. Evidence packet: +scratchpad `commissioning-evidence.md`. + +## CONSENSUS (all four agree) + +1. **The agent's leading hypothesis -- "marginal 30-min timeout over slow + nested I/O" -- is WRONG.** Refuted 4/4 (confidences: HIGH, ~75-80%, high, + ~90%). *Slow is not silent.* A slow-but-progressing commission still emits a + trickle of script-result events, just late; the symptom is ZERO events for + 30 minutes = the node never phoned home = a **hang/stall**, not slow + progress. A longer clock does not fix a hang. +2. **The failure is in the POST-"Loading ephemeral" phase.** Not slow boot, not + the CPU model, not nested-virt perf (the boots are genuinely KVM-accelerated + -- a `type=kvm` domain cannot silently fall back to TCG, and all 9 enlisted), + not memory, not contention (the isolated node failed at the full ~29.5 min + with 397 GiB free). +3. **The decisive next step is to OBSERVE, not to change the clock.** Two + instruments, both prescribed by multiple reviewers: (a) a FULL-WINDOW packet + capture keyed on a failing node's **lease IP** (NOT port-scoped) across the + entire 30 min, on BOTH `virbr2` (metal-admin) and `enp1s0` (transit); (b) + `console=ttyS0,115200` on the MAAS commissioning kernel cmdline so the + (currently VGA-only) boot becomes legible in the serial log the node-vm + module already writes. +4. **The agent's proposed test is CONFOUNDED and must not be run as designed:** + "raise node_timeout AND commission ONE node" changes TWO variables at once + (timeout + concurrency 9->1). A one-node pass is equally predicted by + concurrency-relief and by a timing race -- it would "confirm" the timeout + theory while proving nothing. + +## PIVOTAL ZERO-COST UNKNOWNS (flagged by 3 of 4) + +- **The MAAS SUBNET MTU for metal-admin (DHCP option 26)** -- a MAAS setting + INDEPENDENT of the libvirt network's 9000. It decides whether the guest + ephemeral NIC even comes up jumbo. If 1500 -> the whole MTU branch largely + collapses and attention shifts to Temporal/script-hang. If 9000 -> the MTU + race becomes prime. Single read, reshapes the whole recommendation. +- **Same-3 vs rotating-3 on a re-run** -- from event history / a second batch. + Same 3 pass every time => deterministic per-node cause; a rotating 3 => a race + or region-side starvation. Splits the hypothesis space for free. + +## DISAGREEMENT ON MECHANISM (ranked candidates, pooled) + +- **Region-side Temporal starvation / partial wedge** (adversarial #1). MAAS + 3.5+ drives commissioning via Temporal workflows ON THE REGION; Temporal was + region-wide WEDGED and only `snap restart`-recovered immediately before these + runs. Explains every hard fact: total silence (workflow never dispatches the + next activity), isolated-node-still-fails (isolating the NODE cannot relieve a + REGION bottleneck), 3/6 with no shape correlation (worker/task-queue + starvation lets a few through), all-9-enlist (enlistment is a lighter/earlier + workflow). **The agent measured only RACK resources; region Temporal health + during a run is UNMEASURED** -- and it is the box that was just unwedged. + Refs: LP #2084788 (3.5.1 stuck-forever-commissioning), MAAS 3.7 release notes + (Temporal heartbeat-timeout failures). +- **Commissioning-phase-only script hang on nested-virt hardware** (adversarial + #1 co-lead). Commissioning runs scripts enlistment does NOT (storage + `smartctl`, `lldpd` waiting for a neighbor an isolated libvirt switch never + sends, `bmc-config` probing a BMC that doesn't exist on a VM). MAAS logs + script COMPLETION, not start -- so a hung script produces IDENTICAL silence, + cleanly resolving the enlistment paradox. Per-attempt-invariant -> survives + the isolated-node test. +- **Jumbo/squashfs bulk-transfer black hole** (MAAS-internals #1; network LOW; + adversarial #3). Near-exact upstream match LP #1807252: hang right after + "Loading ephemeral" downloading the squashfs, NIC MTU flips 1500->9000 ~20s + after boot mid-transfer, large frames black-hole, fixed by MTU 1500. The + OBVIOUS version is REFUTED (enlistment pulls the same squashfs over the same + node<->rack path and all 9 enlisted; and the rack PROXIES metadata, so the + node never PMTUs across the 1500 transit). Stays alive only via the narrow + path: guest vNIC MTU is never set in node-vm (main.tf:115-129) while the plane + is 9000, and the ~20s post-boot MTU flip is a documented RACE that would + explain 3/6. Network reviewer's strong counter: config is UNIFORMLY jumbo (not + mismatched) and the squashfs is a single LOCAL hop (node->tap->virbr2->rack + httpd on the same host), so a static mismatch would fail all 9, not 3/6. +- **SEC-010 FORWARD-drop catching non-proxied return traffic** (adversarial #4, + low). Only bites if a commissioning step makes the node talk region-DIRECT + (forwarded, not proxied) with no established/related accept above the drop. + Check only if the capture shows region-direct attempts. + +## PROCESS CORRECTIONS THE COMMITTEE CAUGHT (agent's own errors) + +1. Leading hypothesis refuted 4/4. +2. Proposed test confounded (timeout + concurrency changed together). +3. Measured the WRONG BOX -- rack resources, never region Temporal during a run. +4. The earlier tcpdump was doubly mis-scoped: port 5248 AND a 220s window; must + be lease-IP-keyed, full-window, both interfaces. +5. The "enlistment paradox" has a clean resolution the agent missed: MAAS logs + script completion not start, so a commissioning-only script hang = identical + silence. + +## RECOMMENDED SEQUENCE (supersedes "raise node_timeout") + +1. Zero-cost reads FIRST (read-only): metal-admin MAAS **subnet MTU**; region + **Temporal** worker health + region resource headroom; current machine + states. +2. Then the decisive instrument (gated): full-window, lease-IP-keyed capture on + virbr2 + enp1s0 during one commission, PLUS `console=ttyS0` on the + commissioning kernel to read the boot. Four mutually-exclusive capture + outcomes each kill whole branches (squashfs stalls mid-transfer -> MTU; + squashfs completes then silence -> script/cloud-init hang; metadata reaches + region but no/slow answer -> Temporal/proxy; scripts fetch then silent -> + specific script hang). +3. Only AFTER localization: the targeted fix (MTU clamp to 1500 / node_timeout / + Temporal restart / script exclusion), each reversible. + +## POST-COMMITTEE ZERO-COST READS (agent, read-only, 2026-07-20) + +- **metal-admin MAAS VLAN MTU = 1500** (not 9000). The libvirt NETWORK is 9000 + but MAAS serves DHCP option 26 = 1500, so the guest ephemeral NIC comes up + 1500 and NEVER emits a jumbo frame. **The MTU/squashfs-black-hole branch is + EXONERATED by measurement** -- the exact "if 1500, MTU is off the table" fork + three reviewers named. (10.10.1.0/24 is likewise 1500.) +- **Region resources ample:** voffice1 = 16 vCPU, 31 GiB RAM (23 GiB + available), load ~0.95 at a quiet moment. Not starved at rest. +- **Temporal:** `"Not enough hosts to serve the request"` = **175,201 since + 04:00** but **0 in the last 15 min**. Temporal was catastrophically wedged + this morning (the DHCP incident), recovered after `snap restart maas`, and is + quiet AT REST now. This neither confirms nor clears the "Temporal starvation + DURING a commission" hypothesis -- the 175k is mostly pre-restart, and no + commission is running now. Only a capture/log DURING an active commission + settles it. + +## NET EFFECT ON THE HYPOTHESIS RANKING + +- OUT (refuted 4/4): marginal 30-min timeout -> do NOT raise node_timeout as the + first move. +- OUT (measured): jumbo/MTU black hole -> subnet MTU is 1500. +- STILL LIVE, need observation DURING a run: + 1. Region Temporal starvation during commissioning (was wedged this AM; + healthy at rest; unmeasured during a run). + 2. Commissioning-only script hang on nested-virt hardware (lldpd/smartctl/ + bmc-config probing hardware absent in a VM; MAAS logs script COMPLETION so + a hang = silence). +- The decisive gated instrument is unchanged: one commission with + `console=ttyS0` on the kernel + a full-window, lease-IP-keyed capture on + virbr2 + enp1s0, watched live for Temporal errors on the region. + +Web refs: LP #1807252, LP #2084788, LP #1908452, LP #1403955, MAAS 3.7 release +notes, maas.io controller-communication docs.