diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 2bce5bf..a0977ef 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -571,15 +571,36 @@ **IDENTITY NOTE: system_id and hostname are BOTH re-minted** (`xqqwdq` -> `6q4syf`, `moral-salmon` -> `civil-bug`); the BOOT MAC is the only stable key, which is why the snapshot records it as the identity. - **THE RE-ENROLMENT ORDERING IS NOT THE OBVIOUS ONE -- MEASURED, and it is a standing - DC-migration invariant.** A freshly-enlisted machine in a new region has NO power - configuration, so MAAS cannot power-cycle it and therefore CANNOT drive commissioning. - Working order: (1) `machine delete` on the OLD region (the libvirt domain is untouched -- - all 12 dc0 domains stayed defined); (2) **power the VM on BY HAND** (`virsh -c + **RE-ENROLMENT ORDERING -- WHAT IS PROVEN, AND WHAT IS NOT YET.** **PROVEN by measurement:** + (1) `machine delete` on the OLD region leaves the libvirt domain untouched (all 12 dc0 + domains stayed defined); (2) the VM must be **powered on BY HAND** (`virsh -c qemu:///system start ` on the rack) -- nothing else can start a machine the new - region has never seen; (3) it PXEs, leases from the dynamic range and SELF-ENLISTS; - (4) **set power config BEFORE expecting commissioning to progress**; (5) `machine abort` - -> `New`, then `machine commission`, at which point MAAS powers it on itself. + region has never seen; (3) it then PXEs, leases from the dynamic range and SELF-ENLISTS + with its pinned boot MAC; (4) a freshly-enlisted machine has NO power configuration, so + **setting power config is a PREREQUISITE, not an afterthought** -- without it MAAS cannot + power-cycle the node, the enlistment ephemeral powers itself off, and the machine WEDGES in + `Commissioning` emitting ZERO events (observed directly here); (5) `machine abort` returns + it to `New` and `machine commission` then has MAAS power it on itself (observed: `vm=running`). + **NOT YET PROVEN: that this sequence reaches `Ready`.** It has not. See the open item below; + the ordering above must NOT be treated as a closed standing invariant until a machine lands + `Ready`, and it may yet need another step. + **OPEN -- COMMISSIONING PASSES BUT THE MACHINE RETURNS TO `New`, NOT `Ready`.** MEASURED + from `node-script-results read`, which is the only direct evidence and had to be read + rather than inferred: result set id=3 (Commissioning) **`Passed`** with all 13 scripts + exit 0 (`30-maas-01-bmc-config` Skipped, normal for a VM), and result set id=4 (Testing) + **`Passed`** with `smartctl-validate` **Skipped** on both virtio disks. Hardware discovery + populated correctly -- `cpu_count=8`, `memory=24576`, storage present, matching the D-121 + Option C storage-node shape exactly. So **neither commissioning nor testing failed**, and + the earlier suspicion that smartctl against virtio disks was the culprit is REFUTED. The + machine nonetheless transitioned to `New` at 21:53:08, the same second the testing set + completed. Root cause NOT established. A third commission is in flight under identical + conditions purely to see whether the second was a one-off; if it also lands `New`, the + fault is systematic and the next probes are the Office1 region's commissioning/testing + config defaults (the two regions were configured independently, and Office1 commissioned + these same ten nodes to `Ready` on 2026-07-21). + **THE OTHER NINE MACHINES ARE DELIBERATELY UNTOUCHED** until the canary reaches `Ready` -- + deleting more records would multiply an undiagnosed fault, and the return path (re-enable + Office1 DHCP, re-enlist) gets more expensive per machine. **SKIPPING (4) WEDGES THE MACHINE IN `Commissioning` PRODUCING ZERO EVENTS** -- observed here, and it looks alarming until named: the enlistment ephemeral powers the node off when it finishes, and with no power driver MAAS can never bring it back. diff --git a/docs/changelog-20260730-dc0-region-migration.md b/docs/changelog-20260730-dc0-region-migration.md index 3bd7f85..efed959 100644 --- a/docs/changelog-20260730-dc0-region-migration.md +++ b/docs/changelog-20260730-dc0-region-migration.md @@ -585,7 +585,9 @@ **MEASURED SEQUENCE, and it is NOT the obvious one.** A freshly-enlisted machine in a new region has NO power configuration, so MAAS cannot power-cycle it, so it CANNOT drive -commissioning. The working order is: +commissioning. Steps 1-5 below are each individually MEASURED, but see item 18: **the +sequence has NOT yet produced a `Ready` machine, so it is not a closed invariant and may +need a further step.** The order so far: 1. `machine delete ` on the OLD region (the libvirt domain is untouched -- all 12 dc0 domains stayed defined throughout). @@ -655,3 +657,48 @@ staging dir absent on dc1; queued to standup DoD"). Worth folding into that same DoD item: a serial log configured but not capturing is a standup defect, and it should be verified by mtime at each DC standup rather than assumed from the domain XML. + + +--- + +## Item 18 -- OPEN: commissioning PASSES but the canary returns to `New`, not `Ready` + +**Do not read items 16's ordering as complete.** The canary has not reached `Ready`. + +**MEASURED, from the surface I initially skipped.** My first look at +`node-script-results read` returned unparseable output and I moved on -- which is the +"could not look is not nothing there" failure, committed against my own rule. Read +properly, it is the only direct evidence and it is unambiguous: + +| result set | type | status | detail | +|---|---|---|---| +| id=2 | Commissioning | Aborted | the enlistment-time run I aborted deliberately | +| id=3 | Commissioning | **Passed** | all 13 scripts exit 0; `30-maas-01-bmc-config` Skipped (normal for a VM) | +| id=4 | Testing | **Passed** | `smartctl-validate` **Skipped** on both virtio disks | + +Hardware discovery populated correctly: `cpu_count=8`, `memory=24576`, storage present -- +exactly the D-121 Option C storage-node shape. **So neither commissioning nor testing +failed.** The machine transitioned to `New` at 21:53:08, the same second the testing set +completed. + +**REFUTED:** the plausible theory that `smartctl-validate` against virtio disks was the +fault. It did not fail -- it was Skipped, and its result set Passed. Had I re-run with +`testing_scripts=none` as the first move, it would have "fixed" nothing and I would have +learned nothing. + +**Root cause NOT established.** A third commission is in flight under conditions identical +to the second, purely to distinguish a one-off from a systematic fault. That is the only +reason to repeat it; a bare re-run is otherwise the same-variation retry pattern this repo +warns against. **If it also lands `New`,** the next probe is the two regions' commissioning +and testing config defaults -- they were configured independently, and the Office1 region +commissioned these same ten nodes to `Ready` on 2026-07-21, so a config delta is the +leading candidate. + +**THE OTHER NINE ARE DELIBERATELY UNTOUCHED.** Canary-first exists precisely so an +undiagnosed fault is not multiplied by ten, and the return path (re-enable Office1 DHCP, +re-enlist) grows more expensive per machine. + +**`dc-node-carve.sh` is also correctly still unwritten:** a `New` machine's interfaces are +rediscovered on the next commission, so anything built against the current +`fabric-7..11` placement would be built against state that changes underneath it. `Ready` +is when the interface set is stable enough to carve.