diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index b9f65bb..ec62c39 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -83,7 +83,14 @@ registered (7chphy, rackd running), metal-admin dynamic range 10.12.8.100-.200 created (operator-ruled D-120 inheritance), VLAN 5005 `dhcp_on=true primary_rack=7chphy` verified by read-back. - **OPEN INCIDENT (blocks node enlistment): MAAS DHCP is DOWN REGION-WIDE.** + **INCIDENT RESOLVED 2026-07-20** (operator-approved region restart): + dhcpd now RUNNING on both controllers (verified by process, not service + status), and **all 9 DC0 nodes ENLISTED in MAAS** with shapes exactly + matching D-121 Option C (3x16cpu/64GiB + 2x12cpu/48GiB + 4x8cpu/24GiB) -- + `docs/audit/stepD-enlistment-20260720.txt`. **The G10 depth-4 nested boot + gate is therefore PASS**: node VMs inside vvr1-dc0 PXE-booted from the + Office1 region across the transit and run MAAS's ephemeral kernel. + The incident as originally found: MAAS 3.7 drives DHCP via Temporal, and Temporal is wedged on the region ("Not enough hosts to serve the request", 2807 retries), so no dhcpd runs on EITHER controller -- including `voffice1` itself, whose compose net @@ -91,10 +98,13 @@ this deploy (almost certainly since the 2026-07-17 host reboot); unnoticed because both Office1 VMs were already Deployed. Any "Office1 MAAS DHCP working" claim is currently FALSE. Proposed gated remedy: - restart MAAS on the region, then re-verify dhcpd behaviorally on both. - Details + detection-gap finding: session changelog item 13. - REMAINING after the incident: node enlistment, `TF_VAR_maas_api_key` + - vm-host registration (step D part 2), and netem (step E). + restart MAAS on the region -- DONE, and it fixed BOTH sites, confirming a + single root cause. Details + the queued detection-gap finding + (cloud-assert trusts MAAS's self-report and missed a dead DHCP server): + session changelog items 13-14; appendix-A entry queued. + REMAINING: `TF_VAR_maas_api_key` + `maas-vm-host` registration so MAAS has + power control (commissioning ends in a power-off; the module is BUILT but + UNVALIDATED) -- step D part 2 -- then netem (step E). - The grounding audit is COMPLETE and EXITED (2026-07-19): Phases 1-6 all closed (charter `148dcef`; rulings `docs/audit/ga-rulings.md`; the Phase-5 sweep ran as six operator-gated batches in one session; exit diff --git a/docs/audit/stepD-enlistment-20260720.txt b/docs/audit/stepD-enlistment-20260720.txt new file mode 100644 index 0000000..02ea4e8 --- /dev/null +++ b/docs/audit/stepD-enlistment-20260720.txt @@ -0,0 +1,3 @@ +count: 9 +status: {'New': 1, 'Commissioning': 8} +shapes: {'16cpu/64GiB': 3, '8cpu/24GiB': 4, '12cpu/48GiB': 2} diff --git a/docs/changelog-20260719-dc0-deploy-stepB.md b/docs/changelog-20260719-dc0-deploy-stepB.md index 6eeafc4..ba63ed2 100644 --- a/docs/changelog-20260719-dc0-deploy-stepB.md +++ b/docs/changelog-20260719-dc0-deploy-stepB.md @@ -504,7 +504,42 @@ the repo already applies elsewhere. - No prior art: appendix-A has no entry for this symptom. NEW incident material once the fix is known. -- PROPOSED REMEDY (gated, NOT run): restart MAAS on the region +**RESOLVED 2026-07-20 (operator-approved: "Go ahead with the restart").** +- `sudo snap restart maas` on voffice1 -> region back (HTTP 301 on + localhost:5240). Recovery verified BEHAVIORALLY, not by service status: + `pgrep -c dhcpd` on voffice1 0 -> **1**, and on the DC rack 0 -> **1**; + MAAS then reported `dhcpd: running` on BOTH controllers (was `off` on the + rack, absent entirely on the region). One restart fixed both sites, which + confirms the single wedged-Temporal root cause rather than two faults. +- **PROOF IT WAS THE REAL BLOCKER:** the identical canary that had failed + before (`virsh reset vr1-dc0-control-01`) enlisted immediately after. +- APPENDIX-A MATERIAL (queued, new incident class): symptom "MAAS says + dhcp_on=True but no dhcpd runs and nothing PXEs"; check `pgrep -c dhcpd` + on each controller and the region journal for Temporal + `"Not enough hosts to serve the request"`; remedy `snap restart maas`; + verify behaviorally on every controller, never by service_set. + +## 14. G10 DEPTH-4 NESTED BOOT GATE: **PASS** -- all 9 DC0 nodes enlisted in MAAS + +- After the DHCP fix, all nine node VMs were reset and **all nine enlisted** + (`docs/audit/stepD-enlistment-20260720.txt`): count 9, status 8 + Commissioning + 1 New, shapes `{16cpu/64GiB: 3, 12cpu/48GiB: 2, + 8cpu/24GiB: 4}` -- an EXACT match to the D-121 Option C / R-3 layout + (3 control + 2 compute + 4 storage). +- **This closes the D-114/D-123 depth-4 boot gate, the flagged-risk item of + Model B.** A node VM inside `vvr1-dc0` (itself a VM on vcloud) PXE-booted + from the Office1 region across the D-124 transit and is running MAAS's + ephemeral commissioning kernel -- i.e. real nested KVM at depth 4, proven + by behavior rather than by inference from `svm` being present. +- Still open in step D: commissioning completes with a power-off, and MAAS + has no power driver for these VMs yet -- the `maas-vm-host` registration + (module BUILT 2026-07-09, still UNVALIDATED) is the next piece, and it + needs `TF_VAR_maas_api_key`. +- **Revert:** `maas admin machines delete` per machine (or delete the pod + once registered); the VMs themselves are inner-root state. + +- PROPOSED REMEDY (superseded by the RESOLVED block above; kept for the + audit trail): restart MAAS on the region (`sudo snap restart maas` on voffice1) to re-establish Temporal workers, then re-verify dhcpd on BOTH controllers behaviorally. Blast radius is low right now (no OpenStack workloads; both Office1 VMs already deployed