diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 8f2a276..2bce5bf 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -562,6 +562,44 @@ regions match. Read-only `pgrep`/`ps -ef` over ssh were added to **allow**. This is the standing lesson restated: CHECK WHETHER A RULE FAILED TO MATCH BEFORE BLAMING THE CLASSIFIER, and remember that quoting style and the profile token are both part of the match. + **MACHINE RE-ENROLMENT STARTED 2026-07-30, CANARY FIRST (dc1 precedent).** Pre-delete + identity snapshot `docs/audit/dc0-premigration-machine-identity-20260730.txt` -- all 10 + boot MACs cross-checked against `lib-hosts.sh HOST_BOOT_MAC` and matching exactly. Canary + `xqqwdq`/moral-salmon = `vr1-dc0-storage-04` (a storage node -- least central of the ten). + Office1 23 -> 22 machines; the canary re-enlisted into the new region carrying boot MAC + `52:54:00:2b:ed:ab`, i.e. **the machine that arrived is the machine that left**. + **IDENTITY NOTE: system_id and hostname are BOTH re-minted** (`xqqwdq` -> `6q4syf`, + `moral-salmon` -> `civil-bug`); the BOOT MAC is the only stable key, which is why the + snapshot records it as the identity. + **THE RE-ENROLMENT ORDERING IS NOT THE OBVIOUS ONE -- MEASURED, and it is a standing + DC-migration invariant.** A freshly-enlisted machine in a new region has NO power + configuration, so MAAS cannot power-cycle it and therefore CANNOT drive commissioning. + Working order: (1) `machine delete` on the OLD region (the libvirt domain is untouched -- + all 12 dc0 domains stayed defined); (2) **power the VM on BY HAND** (`virsh -c + qemu:///system start ` on the rack) -- nothing else can start a machine the new + region has never seen; (3) it PXEs, leases from the dynamic range and SELF-ENLISTS; + (4) **set power config BEFORE expecting commissioning to progress**; (5) `machine abort` + -> `New`, then `machine commission`, at which point MAAS powers it on itself. + **SKIPPING (4) WEDGES THE MACHINE IN `Commissioning` PRODUCING ZERO EVENTS** -- observed + here, and it looks alarming until named: the enlistment ephemeral powers the node off when + it finishes, and with no power driver MAAS can never bring it back. + **The power command needs BOTH URIs and `maas-node-power.sh` already supports it:** + `VIRSH_URI` (env) enumerates domains from voffice1 over the TRANSIT /30, while the + positional power address stores what the REGION dials over METAL-ADMIN. Result + `[ok] civil-bug -> vr1-dc0-storage-04 (power state: off)` -- a REAL `query-power-state`, + proving the region drives the rack's libvirt over metal-admin with the SEC-012 key + installed in its snap. Power path verified end to end, not inferred. + **DIAGNOSTIC BLIND SPOT FOUND: the per-node serial log stopped capturing on 2026-07-21.** + `/var/lib/libvirt/vr1/staging/vr1-dc0-storage-04-serial.log` has mtime 2026-07-21 08:54:43 + although the domain XML configures `` on both `` and + ``. It still CONTAINS a complete, plausible boot ending in a clean + `reboot: Power down`, and **that stale content was read as current and reported as fact** + before the mtime was checked -- the giveaway was `DataSourceMAASLocal + [http://10.12.8.2:5248/...]`, the OLD region's metadata path. An append-mode log that has + silently stopped appending is worse than an absent one, because `tail` returns confident, + well-formed, WRONG evidence. LOGGED NOT ACTIONED; folds into the existing dc1 standup-DoD + item for serial logging, with the added requirement to verify by MTIME rather than from + the domain XML. **>>> PREVIOUSLY BLOCKED ON A PERMISSION WALL (now cleared -- kept as history) <<<** The next step is the one-way DHCP cutover -- `vlan update 4 0 dhcp_on=false` at Office1, verify `dhcpd` STOPPED **by process** (MAAS self-report is not evidence -- the 2026-07-20 diff --git a/docs/audit/dc0-premigration-machine-identity-20260730.txt b/docs/audit/dc0-premigration-machine-identity-20260730.txt new file mode 100644 index 0000000..ec27fae --- /dev/null +++ b/docs/audit/dc0-premigration-machine-identity-20260730.txt @@ -0,0 +1,28 @@ +dc0 MACHINE IDENTITY SNAPSHOT -- taken immediately before the delete/re-enrol +2026-07-30. Office1 region. The rebuild reference for the 10 records being removed. +Boot MAC is the STABLE identity: system_ids are re-minted on re-enlistment and +hostnames are MAAS auto-generated. Matched against lib-hosts.sh HOST_BOOT_MAC. +============================================================================== +count=10 +ssyexn ace-robin Ready boot_mac=52:54:00:02:ff:57 tags=control,openstack-vr1-dc0,virtual +sn6qda alert-cub Ready boot_mac=52:54:00:b1:94:d1 tags=openstack-vr1-dc0,storage,virtual +kghggm first-oryx Ready boot_mac=52:54:00:5f:8d:42 tags=openstack-vr1-dc0,storage,virtual +dbcq8h keen-dove Ready boot_mac=52:54:00:1b:19:e6 tags=compute,openstack-vr1-dc0,virtual +xqqwdq moral-salmon Ready boot_mac=52:54:00:2b:ed:ab tags=openstack-vr1-dc0,storage,virtual +7n87bt moved-troll Ready boot_mac=52:54:00:48:e7:1e tags=juju-controller-vr1-dc0,virtual +sk8c4d real-filly Ready boot_mac=52:54:00:4f:de:a9 tags=control,openstack-vr1-dc0,virtual +sgwfnb superb-piglet Ready boot_mac=52:54:00:18:ab:b4 tags=compute,openstack-vr1-dc0,virtual +nhg3nf wired-thrush Ready boot_mac=52:54:00:be:69:c5 tags=control,openstack-vr1-dc0,virtual +8mtpxq wise-stud Ready boot_mac=52:54:00:48:86:2c tags=openstack-vr1-dc0,storage,virtual + +-- cross-check against the repo pin (scripts/lib-hosts.sh HOST_BOOT_MAC) -- + vr1-dc0-control-01 52:54:00:be:69:c5 + vr1-dc0-control-02 52:54:00:02:ff:57 + vr1-dc0-control-03 52:54:00:4f:de:a9 + vr1-dc0-compute-01 52:54:00:1b:19:e6 + vr1-dc0-compute-02 52:54:00:18:ab:b4 + vr1-dc0-storage-01 52:54:00:5f:8d:42 + vr1-dc0-storage-02 52:54:00:48:86:2c + vr1-dc0-storage-03 52:54:00:b1:94:d1 + vr1-dc0-storage-04 52:54:00:2b:ed:ab + vr1-dc0-juju-01 52:54:00:48:e7:1e diff --git a/docs/changelog-20260730-dc0-region-migration.md b/docs/changelog-20260730-dc0-region-migration.md index cb16228..3bd7f85 100644 --- a/docs/changelog-20260730-dc0-region-migration.md +++ b/docs/changelog-20260730-dc0-region-migration.md @@ -574,3 +574,84 @@ **Revert.** `ssh voffice1 'maas vr1-dc0-region vlan update 0 0 dhcp_on=false'` then `ssh voffice1 'maas admin vlan update 4 0 dhcp_on=true primary_rack=7chphy'`, verifying by process at each step. Remove the added rules from `.claude/settings.local.json`. + +--- + +## Item 16 -- CANARY re-enrolment, and the ORDERING the migration actually needs + +**Canary chosen deliberately:** `xqqwdq` / moral-salmon = `vr1-dc0-storage-04`, a storage +node -- the least central of the ten if anything went wrong. The dc1 standup set the +precedent of proving enlistment with one machine before touching the rest. + +**MEASURED SEQUENCE, and it is NOT the obvious one.** A freshly-enlisted machine in a new +region has NO power configuration, so MAAS cannot power-cycle it, so it CANNOT drive +commissioning. The working order is: + +1. `machine delete ` on the OLD region (the libvirt domain is untouched -- all 12 dc0 + domains stayed defined throughout). +2. **Power the VM on by hand** (`virsh -c qemu:///system start ` on the rack). This + step exists only because the new region has no power driver for a machine it has never + seen. Nothing else can start it. +3. It PXEs, leases from the new region's dynamic range, and SELF-ENLISTS. +4. **Set power config BEFORE expecting commissioning to progress.** +5. `machine abort` (-> `New`), then `machine commission`. MAAS now powers it on itself. + +**Skipping step 4 leaves the machine wedged in `Commissioning` producing ZERO events** -- +which is exactly what happened here and looks alarming until the cause is named. The +enlistment ephemeral powers the node off when it finishes; with no power driver MAAS can +never bring it back, so it waits forever. `machine abort` is the documented way out. + +**The power command needs BOTH URIs, and `maas-node-power.sh` already supports it:** + + MAAS_PROFILE=vr1-dc0-region \ + VIRSH_URI=qemu+ssh://jessea123@172.31.0.2/system \ # ENUMERATE from voffice1 (transit) + bash scripts/maas-node-power.sh --commit \ + qemu+ssh://jessea123@10.12.8.2/system vr1-dc0 # STORE what the REGION dials + +`VIRSH_URI` defaults to the power address but is env-overridable, which is precisely the +separation this migration needs: voffice1 can reach the rack only over the transit /30, +while the region reaches it only over metal-admin. Conflating them would either fail to +enumerate or store an address the region cannot dial. + +**Result:** `[ok] civil-bug -> vr1-dc0-storage-04 (power state: off)` -- a REAL +`query-power-state`, which proves the region can drive the rack's libvirt over metal-admin +using the SEC-012 key installed in its snap (item 5). Power path verified end to end, not +inferred. MAAS then powered the VM on for commissioning by itself (`vm=running`). + +**IDENTITY PRESERVED, verified by the stable key:** the re-enlisted machine carries boot MAC +`52:54:00:2b:ed:ab`, matching `lib-hosts.sh HOST_BOOT_MAC[vr1-dc0-storage-04]` exactly. Its +system_id changed (`xqqwdq` -> `6q4syf`) and its hostname changed (`moral-salmon` -> +`civil-bug`), which is why the pre-delete snapshot +(`docs/audit/dc0-premigration-machine-identity-20260730.txt`) records the BOOT MAC as the +identity and treats system_id and hostname as disposable. + +**Revert.** Delete the record from the new region and re-enlist into Office1 by re-enabling +its DHCP -- possible only while the domain and its pinned MACs survive, which they do. + +--- + +## Item 17 -- DIAGNOSTIC BLIND SPOT: the per-node serial log stopped capturing on 2026-07-21 + +**Measured.** `/var/lib/libvirt/vr1/staging/vr1-dc0-storage-04-serial.log` has mtime +**2026-07-21 08:54:43** -- nine days stale. Today's boot wrote NOTHING to it, although the +domain XML configures both `` and `` with +``. + +**Why this matters more than it looks.** The file still CONTAINS a complete, plausible boot: +kernel command line, cloud-init stages, "All scripts successfully ran", and a clean +`reboot: Power down` at 138s. **I read that as today's boot and reported it as fact** before +checking the mtime. It was the Stage-4 commissioning of 2026-07-21, betrayed only by its +internal dates and by `DataSourceMAASLocal [http://10.12.8.2:5248/...]` -- the OLD region's +metadata path. An append-mode log that has silently stopped appending is worse than an +absent one: `tail` returns confident, well-formed, wrong evidence. + +**Same class as two other misreadings this session** (a `+time=3 +tries=1` DNS probe called +a capability gap; a `head -5`-truncated listener list called a missing TFTP bind). All three +were instrument errors that produced a clean-looking answer. The rule that catches this one: +**check the mtime before trusting a log file as current.** + +**LOGGED, NOT ACTIONED (hard rule 1).** Not fixed here -- it is a diagnostic aid, not on the +deploy path, and the 2026-07-22 dc1 build already recorded a related fault ("serial-log +staging dir absent on dc1; queued to standup DoD"). Worth folding into that same DoD item: +a serial log configured but not capturing is a standup defect, and it should be verified by +mtime at each DC standup rather than assumed from the domain XML.