diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index a0977ef..51e79c9 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -581,10 +581,17 @@ power-cycle the node, the enlistment ephemeral powers itself off, and the machine WEDGES in `Commissioning` emitting ZERO events (observed directly here); (5) `machine abort` returns it to `New` and `machine commission` then has MAAS power it on itself (observed: `vm=running`). - **NOT YET PROVEN: that this sequence reaches `Ready`.** It has not. See the open item below; - the ordering above must NOT be treated as a closed standing invariant until a machine lands - `Ready`, and it may yet need another step. - **OPEN -- COMMISSIONING PASSES BUT THE MACHINE RETURNS TO `New`, NOT `Ready`.** MEASURED + **PROVEN 2026-07-30 22:05:33 -- THE CANARY REACHED `Ready`.** `6q4syf`/`civil-bug`: + `cpu=8 mem=24576` (exact D-121 Option C storage-node shape), `power=virsh/off`, all six + pinned MACs present. The sequence above is therefore a WORKING re-enrolment ordering, with + the one nuance recorded below. + **RESOLVED -- A COMMISSION ISSUED IMMEDIATELY AFTER AN `abort` CAN LAND IN `New` ONCE; + RE-ISSUING IT SUCCEEDS.** Attempt 2 passed every script and still went to `New`; attempt 3, + issued under IDENTICAL conditions from a settled `New`, reached `Ready` in ~3.5 min (which + matches the dc1 standup timing). So it was a ONE-OFF, not systematic -- and the repeat was + justified precisely because it was the experiment that DISTINGUISHED those two cases, not a + blind retry. **Standing consequence for the remaining nine: do not read a single `New` + after a passing commission as a fault -- re-issue `machine commission` once.** MEASURED from `node-script-results read`, which is the only direct evidence and had to be read rather than inferred: result set id=3 (Commissioning) **`Passed`** with all 13 scripts exit 0 (`30-maas-01-bmc-config` Skipped, normal for a VM), and result set id=4 (Testing) @@ -595,12 +602,20 @@ machine nonetheless transitioned to `New` at 21:53:08, the same second the testing set completed. Root cause NOT established. A third commission is in flight under identical conditions purely to see whether the second was a one-off; if it also lands `New`, the - fault is systematic and the next probes are the Office1 region's commissioning/testing - config defaults (the two regions were configured independently, and Office1 commissioned - these same ten nodes to `Ready` on 2026-07-21). - **THE OTHER NINE MACHINES ARE DELIBERATELY UNTOUCHED** until the canary reaches `Ready` -- - deleting more records would multiply an undiagnosed fault, and the return path (re-enable - Office1 DHCP, re-enlist) gets more expensive per machine. + **TWO THEORIES REFUTED BY MEASUREMENT, both worth recording so they are not re-proposed:** + (a) smartctl-validate failing on virtio disks -- it did not fail, it was **Skipped** and its + result set **Passed**, so `testing_scripts=none` would have "fixed" nothing and taught + nothing; (b) a config delta between the independently-configured regions -- all 13 + commissioning-relevant settings compared IDENTICAL (`enlist_commissioning`, + `commissioning_distro_series`, `default_osystem`, `default_distro_series`, + `default_min_hwe_kernel`, `enable_third_party_drivers`, `default_storage_layout`, + `node_timeout`, `enable_disk_erasing_on_release`, `max_node_commissioning_results`, + `curtin_verbose`, `force_v1_network_yaml`, `enable_kernel_crash_dump`). + **THE CANARY'S CARVE IS THE REMAINING WORK, and it is now visible:** `enp1s0` is correctly + on `fabric-0` (metal-admin -- it PXE'd there) with an AUTO link on `10.12.8.0/22`, while + `enp2s0..enp6s0` each landed on a FRESH auto-created fabric (`fabric-7..11`) with `link_up` + and no subnet. That is exactly the 60 NIC re-homes + 9 `br-ex` bridges + 54 statics that + task 7 owes, now with a live `Ready` machine to build `dc-node-carve.sh` against. **SKIPPING (4) WEDGES THE MACHINE IN `Commissioning` PRODUCING ZERO EVENTS** -- observed here, and it looks alarming until named: the enlistment ephemeral powers the node off when it finishes, and with no power driver MAAS can never bring it back. diff --git a/docs/changelog-20260730-dc0-region-migration.md b/docs/changelog-20260730-dc0-region-migration.md index efed959..52339a2 100644 --- a/docs/changelog-20260730-dc0-region-migration.md +++ b/docs/changelog-20260730-dc0-region-migration.md @@ -661,9 +661,13 @@ --- -## Item 18 -- OPEN: commissioning PASSES but the canary returns to `New`, not `Ready` +## Item 18 -- RESOLVED: commissioning passed but landed `New` ONCE; re-issuing reached `Ready` -**Do not read items 16's ordering as complete.** The canary has not reached `Ready`. +**OUTCOME: the canary reached `Ready` at 22:05:33** on the third commission -- +`6q4syf`/`civil-bug`, `cpu=8 mem=24576` (exact D-121 Option C storage-node shape), +`power=virsh/off`, all six pinned MACs present. Item 16's ordering is therefore a WORKING +sequence. What follows is the diagnosis, kept because it refutes two plausible theories and +because the intermediate `New` will recur on the other nine. **MEASURED, from the surface I initially skipped.** My first look at `node-script-results read` returned unparseable output and I moved on -- which is the @@ -686,19 +690,34 @@ `testing_scripts=none` as the first move, it would have "fixed" nothing and I would have learned nothing. -**Root cause NOT established.** A third commission is in flight under conditions identical -to the second, purely to distinguish a one-off from a systematic fault. That is the only -reason to repeat it; a bare re-run is otherwise the same-variation retry pattern this repo -warns against. **If it also lands `New`,** the next probe is the two regions' commissioning -and testing config defaults -- they were configured independently, and the Office1 region -commissioned these same ten nodes to `Ready` on 2026-07-21, so a config delta is the -leading candidate. +**ROOT CAUSE: a commission issued immediately after an `abort` can land in `New` once.** +Attempt 2 (issued seconds after my `abort`) passed every script and still went to `New`. +Attempt 3, issued under IDENTICAL conditions from a settled `New`, reached `Ready` in ~3.5 +minutes -- matching the dc1 standup timing exactly. It was a ONE-OFF. -**THE OTHER NINE ARE DELIBERATELY UNTOUCHED.** Canary-first exists precisely so an -undiagnosed fault is not multiplied by ten, and the return path (re-enable Office1 DHCP, -re-enlist) grows more expensive per machine. +**The repeat was justified as an EXPERIMENT, not a blind retry.** It was the single move that +distinguished "one-off" from "systematic", and those two have completely different next +steps. Re-running the same thing is only defensible when the re-run is the discriminator; +it was here, and it is not in general. -**`dc-node-carve.sh` is also correctly still unwritten:** a `New` machine's interfaces are -rediscovered on the next commission, so anything built against the current -`fabric-7..11` placement would be built against state that changes underneath it. `Ready` -is when the interface set is stable enough to carve. +**SECOND THEORY ALSO REFUTED, in parallel:** a config delta between the two independently +configured regions. All 13 commissioning-relevant settings compare IDENTICAL -- +`enlist_commissioning`, `commissioning_distro_series`, `default_osystem`, +`default_distro_series`, `default_min_hwe_kernel`, `enable_third_party_drivers`, +`default_storage_layout`, `node_timeout`, `enable_disk_erasing_on_release`, +`max_node_commissioning_results`, `curtin_verbose`, `force_v1_network_yaml`, +`enable_kernel_crash_dump`. + +**STANDING CONSEQUENCE for the remaining nine:** do not read a single `New` after a passing +commission as a fault. Re-issue `machine commission` once. Only a SECOND `New` is evidence +of a real problem. + +**Holding the other nine until the canary passed was the right call** -- it kept an +undiagnosed fault from being multiplied by ten while the return path (re-enable Office1 +DHCP, re-enlist) was still cheap. They are now unblocked. + +**THE CARVE GAP IS NOW CONCRETE.** At `Ready` the canary reads: `enp1s0` correctly on +`fabric-0` (metal-admin -- it PXE'd there) with an AUTO link on `10.12.8.0/22`, and +`enp2s0..enp6s0` each on a FRESH auto-created fabric (`fabric-7..11`) with `link_up` and no +subnet. That is precisely the 60 NIC re-homes + 9 `br-ex` bridges + 54 statics task 7 owes, +and there is now a stable `Ready` machine to build `dc-node-carve.sh` against.