diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 52836be..d135e67 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -132,13 +132,19 @@ route is refuted); retire-or-keep is a stage-close question, as is the D-103/D-123 amendment text. Session changelog items 15-17. **COMMISSIONING PARTIAL (open, 2026-07-20): 3 nodes Ready, 6 timed out.** - Contention REFUTED by test (a single node alone sat in `Loading ephemeral` - 20+ min with 385 GiB free, load 0.04); rack boot-image sync and DHCP also - ruled out by measurement. Diagnosis is BLOCKED because `modules/node-vm` - defines NO serial console -- the same sealed-box gap already logged for - `modules/cloudinit-vm`, and the fix is the opt-in serial+log pattern - `modules/opnsense-edge` already has. Failed commissioning is re-runnable; - nothing is lost. REMAINING IN G10: this diagnosis, then netem (step E, + Established: PXE and the ephemeral handoff WORK, and the ephemeral OS boots + with working networking (nodes hold leases and do NTP to the rack) -- it + simply never completes. Ruled out by measurement: memory, rack boot-image + sync, DHCP, and node shape. The node->region path (SEC-010) is SUSPECTED + but UNCONFIRMED (those rules carry no counters). A serial console was added + to `modules/node-vm` and applied in-place to all 9, but the logs stay empty + -- firmware writes to VGA, so serial alone does NOT make a PXE-booting node + observable (correction queued). **Two of the agent's own isolation + experiments were INVALID and must not be cited** (other nodes were still + running; and a re-commission did not restart MAAS's timer) -- so contention + remains a LIVE hypothesis, not a refuted one. Failed commissioning is + re-runnable; nothing is lost. Session changelog item 19 records the clean + experiment to run next. REMAINING IN G10: this diagnosis, then netem (step E, NOT started -- target is the dc0<->dc1 mesh = **virbr5 on vcloud**, measured; `modules/netem-link` assumes PASSWORDLESS SUDO which vcloud's operator account lacks). Session changelog item 18. diff --git a/docs/changelog-20260719-dc0-deploy-stepB.md b/docs/changelog-20260719-dc0-deploy-stepB.md index be4ea24..e994ff0 100644 --- a/docs/changelog-20260719-dc0-deploy-stepB.md +++ b/docs/changelog-20260719-dc0-deploy-stepB.md @@ -707,6 +707,59 @@ PASSWORDLESS SUDO, which vcloud's operator account does not have** -- a documented module assumption that this host violates. +## 19. node-vm console SHIPPED; commissioning narrowed but UNRESOLVED -- and two +## of my own experiments were INVALID (disclosed) + +- SHIPPED (operator-ruled): `modules/node-vm` now defines a serial console + + boot log, with the log directory PARAMETERIZED (`serial_log_dir`) rather + than opnsense-edge's hardcoded vcloud literal, which is wrong on any other + libvirt host. Applied to all 9 domains: **0 add / 9 change / 0 destroy**, + in-place, nothing destroyed. validate PASS, repo-lint 0 fail. +- **The console did NOT yield the hoped-for evidence, and that is itself a + finding:** the logs stay 0 bytes even after a clean domain restart. Cause: + SeaBIOS/PXE firmware writes to the VGA console, not serial, and MAAS's + ephemeral kernel is not told `console=ttyS0`. **A serial console alone does + not make a PXE-booting node observable** -- seeing the firmware/ephemeral + stage needs a graphics device + `virsh screenshot` (or a MAAS-side kernel + cmdline change). Queue this as the correction to the node-vm console work; + do not assume the console added today closes the observability gap. + +WHAT IS SOLIDLY ESTABLISHED (all measured): +- PXE works and the ephemeral handoff happens: event log shows `Performing + PXE boot` 18:17:03 -> `Loading ephemeral` 18:17:45 -> silence -> timeout. +- The ephemeral OS BOOTS AND HAS WORKING NETWORKING: tcpdump on metal-admin + caught 5-6 nodes holding DHCP leases (10.12.8.112-.117) and exchanging NTP + with the rack. So node->rack reachability is fine. +- No traffic toward the REGION (10.10.0.20) or the metadata ports (5248/5240) + was seen in the sampled windows. SEC-010 was suspected of blocking the + node->region path, but its rules carry no `counter`, so drops cannot be + read directly -- **hypothesis UNCONFIRMED, not refuted.** +- Ruled out by measurement: memory (385 GiB free), rack boot-image sync + (`synced`, 10 images), DHCP (leases work), and node SHAPE (Ready and + Failed both span 16/64, 12/48 and 8/24). + +**INVALID EXPERIMENTS -- MINE, RECORDED SO NOBODY TRUSTS THEM:** +1. The first "commission a single node ALONE" test was NOT alone: tcpdump + later proved 5-6 other nodes were still up and running ephemeral images. + My conclusion "contention is refuted" was therefore UNSOUND. Contention + remains a live hypothesis. +2. The second attempt was also invalid: MAAS had the machine in + `Commissioning` already, so the new request did not restart the 30-minute + timer (the failure at 19:24:51 belongs to the 18:54 attempt), and I then + destroyed domains mid-commission, killing the in-flight run. + STOPPED HERE rather than iterate again -- this is the reconciliation + circuit-breaker: repeated cycles without convergence, and contaminated + evidence is worse than none. + +CLEANEST NEXT EXPERIMENT (not run): with ALL node domains powered off and +nothing else touching the fleet, commission EXACTLY ONE machine from a clean +`Ready`/`New` state and leave it strictly untouched for 35+ minutes. Only +that isolates contention from a genuine per-node failure. Candidate fixes to +test after it: raise MAAS `node_timeout` (nested I/O is slow and 30 min may +simply be too short), add a graphics device for `virsh screenshot`, add +`counter` to the SEC-010 rules so the node->region hypothesis becomes +readable, and check the kernel cmdline MAAS actually serves. + ## 8. STAGE-CLOSE QUEUE ITEM (operator directive, 2026-07-20): OPNsense 26.7 review - Operator directive (verbatim intent): at the proper time/place, review the