diff --git a/runbooks/appendix-A-troubleshooting.md b/runbooks/appendix-A-troubleshooting.md index 2893939..59ca895 100644 --- a/runbooks/appendix-A-troubleshooting.md +++ b/runbooks/appendix-A-troubleshooting.md @@ -667,3 +667,46 @@ or yaml/json formats when the workload MESSAGE matters. (Rotated here from the session ledger's standing-lessons block, 2026-07-19 -- this is its durable home.) + +## Symptom: MAAS reports `dhcp_on=True` but NOTHING PXEs -- no dhcpd is running anywhere [2026-07-20] + +-------------------------------------------------------------------------------- +SYMPTOM: A VLAN reads `dhcp_on=True` with the right `primary_rack`, the rack's + interface is correctly linked to the subnet, MAAS's own `service_set` + reports the controllers healthy -- and yet nothing PXE-boots. On the + rack: `dhcpd: off`, no dhcpd process, and no generated + `/var/snap/maas/common/maas/dhcpd.conf`. Config state and RUNNING + state disagree. +-------------------------------------------------------------------------------- +CAUSE: MAAS 3.6+/3.7 drives DHCP configuration through TEMPORAL workflows. If + Temporal is wedged, the DB happily records `dhcp_on=True` while the + workflow that pushes config to a rack can NEVER execute. Confirm from + the region's journal: + + sudo journalctl -u snap.maas.pebble --since "20 min ago" \ + | grep -iE "Not enough hosts to serve the request" + + ("Not enough hosts" = no worker is polling the task queue. Seen with + retry counts in the thousands.) + +CHECK: Trust the PROCESS, never MAAS's service_set -- which reported the + region healthy throughout: + + pgrep -c dhcpd # run on EVERY controller, region included + +FIX: Restart MAAS on the REGION (operator-gated): + + sudo snap restart maas + + Then re-verify BEHAVIORALLY on every controller (`pgrep -c dhcpd` + 0 -> 1), not by service status. One region restart fixed both the + region and a DC rack in the 2026-07-20 incident, which is itself the + confirmation of a single wedged-Temporal root cause rather than two + independent faults. + +SCOPE TRAP: the failure is SILENT on an already-built site. In the 2026-07-20 + case the Office1 region had had NO dhcpd for ~3 days (since a host + reboot) and nobody noticed, because every existing service VM was + already Deployed and holding its address. It only surfaced when a new + DC's nodes tried to PXE. After ANY host reboot, check `pgrep -c dhcpd` + on each controller rather than assuming DHCP came back with the VMs.