diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 51e79c9..771ac83 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -611,6 +611,33 @@ `default_min_hwe_kernel`, `enable_third_party_drivers`, `default_storage_layout`, `node_timeout`, `enable_disk_erasing_on_release`, `max_node_commissioning_results`, `curtin_verbose`, `force_v1_network_yaml`, `enable_kernel_crash_dump`). + **ALL NINE REMAINING OFFICE1 RECORDS DELETED 2026-07-30, individually (never batched -- + hard rule 3; a scripted loop over the nine was correctly REFUSED by the harness guard and + was not retried in that shape).** Office1 is down to **13 machines with ZERO dc0-tagged**: + the two office1 VMs, dc1's nine role nodes, dc1's juju controller, and `hot-kid`. + **TWO FINDINGS THAT CHANGE THE STANDUP PROCEDURE.** + **(1) A NEW REGION HAS *ZERO* SSH KEYS, AND THAT WOULD HAVE BROKEN JUJU BOOTSTRAP.** + `maas vr1-dc0-region sshkeys read` -> **count 0**; Office1 has one + (`SHA256:iUYex2kmvtdGTPiMKRZ2Nfywqa8pWacVV1GEEx6sscI`, `vr1-office1-svc`). MAAS injects + its registered keys into every machine it DEPLOYS, and `juju bootstrap` needs direct + client->machine SSH -- so this is **exactly the signature that consumed three bootstrap + attempts earlier today** and it would have recurred at task 8 looking like a network fault. + FIXED by importing the SAME key Office1 carries (deliberately, not a new mint: deployed + nodes are recorded as carrying `vr1-office1-svc`, and this session's `vr1-dc0-juju` / node + ssh aliases depend on that identity; a per-DC node key is a defensible SEC improvement but + is LOGGED, not done mid-migration). **Standing DC-standup item, belongs beside the power-key + prerequisite: a fresh region's SSH key set is EMPTY until someone fills it.** + **(2) ENLISTMENT DOES NOT SCALE TO NINE CONCURRENT NODES.** The canary alone enlisted and + commissioned in ~2 min. Nine started together: all leasing as `maas-enlisting-node`, all + alive (ping + sshd open, CPU climbing), cloud-init output flowing until 22:12:46 and then + NOTHING for 15 minutes while DHCP renewals continued -- **zero machine records created**. + Everything downstream RULED OUT by measurement: DC edge egress healthy (rack -> + `security.ubuntu.com` 200 in 0.9s); the MAAS proxy healthy (200 in 1.3s via + `10.12.8.6:8000` -- the ONLY path nodes have, since metal-admin carries no gateway); region + VM idle (load 0.29, 5 GB free); nodes alive. Restarted in BATCHES OF THREE. Root cause + remains uncharacterised beyond concurrency; the canary is direct evidence that low + concurrency works and the migration does not need the root cause to proceed. **LOGGED for + the dc1 standup and Roosevelt: do not power on a whole rack for enlistment at once.** **THE CANARY'S CARVE IS THE REMAINING WORK, and it is now visible:** `enp1s0` is correctly on `fabric-0` (metal-admin -- it PXE'd there) with an AUTO link on `10.12.8.0/22`, while `enp2s0..enp6s0` each landed on a FRESH auto-created fabric (`fabric-7..11`) with `link_up` diff --git a/docs/changelog-20260730-dc0-region-migration.md b/docs/changelog-20260730-dc0-region-migration.md index 52339a2..f3ae4cc 100644 --- a/docs/changelog-20260730-dc0-region-migration.md +++ b/docs/changelog-20260730-dc0-region-migration.md @@ -721,3 +721,65 @@ `enp2s0..enp6s0` each on a FRESH auto-created fabric (`fabric-7..11`) with `link_up` and no subnet. That is precisely the 60 NIC re-homes + 9 `br-ex` bridges + 54 statics task 7 owes, and there is now a stable `Ready` machine to build `dc-node-carve.sh` against. + +--- + +## Item 19 -- THE NEW REGION HAD ZERO SSH KEYS (would have broken juju bootstrap) + +**MEASURED:** `maas vr1-dc0-region sshkeys read` -> **count 0**, while +`maas admin sshkeys read` -> 1 (`SHA256:iUYex2kmvtdGTPiMKRZ2Nfywqa8pWacVV1GEEx6sscI`, +`vr1-office1-svc`). A fresh `maas init` registers no keys, and nothing in the migration +sequence adds them. + +**Why this matters far beyond convenience.** MAAS injects its registered SSH keys into every +machine it deploys. With none registered, deployed nodes would come up with NO authorized +key -- and `juju bootstrap` needs direct client->machine SSH. That is **exactly the failure +signature that consumed three bootstrap attempts on 2026-07-30** ("juju then spent 15 +minutes unable to SSH the machine"). Task 8 would have hit it again, and the symptom would +have looked like a network fault rather than a missing key. + +**Found by accident, which is worth admitting:** I was probing the stalled enlistment nodes +and noticed sshd was open but I had no way in. The gap was never on my checklist. It should +be a standing DC-standup item -- **a new region's SSH key set is empty until someone fills +it** -- and it belongs next to the power-key item (prerequisite 5) in the phase-2 runbook. + +**FIXED:** the SAME key Office1 carries was imported, deliberately rather than minting a new +one -- `docs/CURRENT-STATE.md` records that deployed nodes carry the MAAS-registered +`vr1-office1-svc` key, and the `vr1-dc0-juju` / node ssh aliases built earlier this session +depend on that identity. Minting a per-DC node key is a defensible SEC-side improvement but +would break the existing aliases and tooling, so it is LOGGED as a follow-up rather than +done mid-migration. + +**Revert.** `maas vr1-dc0-region sshkeys delete `. + +--- + +## Item 20 -- enlistment does NOT scale to nine concurrent nodes; batch it + +**Measured sequence.** The canary alone enlisted and commissioned in ~2 min. Nine started +together produced: all 9 leasing as `maas-enlisting-node`, all 9 alive (ping + sshd open, +CPU climbing), cloud-init output flowing until **22:12:46** -- and then nothing for 15 +minutes while DHCP renewals continued. **Zero** machine records were created. + +**Everything downstream was RULED OUT by measurement, not assumed:** + +| suspect | measurement | verdict | +|---|---|---| +| DC edge egress | rack -> `security.ubuntu.com` 200 in **0.9s** | healthy | +| MAAS proxy (the only path nodes have -- metal-admin carries NO gateway) | rack -> via `10.12.8.6:8000` 200 in **1.3s** | healthy | +| region VM overload | load **0.29**, 5 GB free | idle | +| nodes dead | ping OK, ssh port OPEN, CPU climbing | alive | + +**A NOTE ON MY OWN INSTRUMENT, because I got this wrong twice in five minutes.** I first +called it stalled off a bad elapsed-time estimate (the last line was 90 seconds old, not +minutes). Then, correcting too fast, I counted "104 maas-enlist lines in 3 minutes" and +called it recovered -- but those were **DHCP lease renewals**, not cloud-init progress; +filtering for `cloud-init` specifically returned **0**. The stall was real, my first two +readings of it were not. Counting lines from a broad grep is not the same as measuring the +thing you care about. + +**ACTION:** all nine stopped and restarted in batches of three. Not a fix for the root +cause -- which is still uncharacterised beyond "concurrency" -- but the canary is direct +evidence that low concurrency works, and the migration does not need the root cause to +proceed. **LOGGED for the dc1 standup and for Roosevelt: do not power on a whole rack of +nodes for enlistment at once.**