diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 70560d2..b9f65bb 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -79,9 +79,22 @@ `domifaddr --source agent` reports both legs. Note this run also exposed and fixed a false-success bug: `opnsense-plugins.sh apply` had ALWAYS dry-run (see session changelog item 12), so any prior "applied" claim - from that script is void. REMAINING: MAAS reach + - `TF_VAR_maas_api_key` + region-side metal-admin DHCP naming this rack - primary_rack (step D), and netem (step E). + from that script is void. **Step D part 1 DONE 2026-07-20**: rack + registered (7chphy, rackd running), metal-admin dynamic range + 10.12.8.100-.200 created (operator-ruled D-120 inheritance), VLAN 5005 + `dhcp_on=true primary_rack=7chphy` verified by read-back. + **OPEN INCIDENT (blocks node enlistment): MAAS DHCP is DOWN REGION-WIDE.** + MAAS 3.7 drives DHCP via Temporal, and Temporal is wedged on the region + ("Not enough hosts to serve the request", 2807 retries), so no dhcpd runs + on EITHER controller -- including `voffice1` itself, whose compose net + reads `dhcp=True` with no dhcpd process. Predates and is NOT caused by + this deploy (almost certainly since the 2026-07-17 host reboot); + unnoticed because both Office1 VMs were already Deployed. Any "Office1 + MAAS DHCP working" claim is currently FALSE. Proposed gated remedy: + restart MAAS on the region, then re-verify dhcpd behaviorally on both. + Details + detection-gap finding: session changelog item 13. + REMAINING after the incident: node enlistment, `TF_VAR_maas_api_key` + + vm-host registration (step D part 2), and netem (step E). - The grounding audit is COMPLETE and EXITED (2026-07-19): Phases 1-6 all closed (charter `148dcef`; rulings `docs/audit/ga-rulings.md`; the Phase-5 sweep ran as six operator-gated batches in one session; exit diff --git a/docs/changelog-20260719-dc0-deploy-stepB.md b/docs/changelog-20260719-dc0-deploy-stepB.md index 956933a..6eeafc4 100644 --- a/docs/changelog-20260719-dc0-deploy-stepB.md +++ b/docs/changelog-20260719-dc0-deploy-stepB.md @@ -459,6 +459,58 @@ `core/firmware/remove/`; set `expose_qga_channel = false` and re-apply (in-place domain update again). +## 13. Step D part 1 DONE (DHCP configured) -- then INCIDENT: MAAS DHCP is DOWN +## REGION-WIDE (Temporal wedged). Not caused by this deploy. + +Step-D work completed before the incident surfaced: +- Both controllers confirmed registered: `voffice1` (region, mtstwf) and + **`vvr1-dc0` (rack, 7chphy, rackd running)**. +- MAAS auto-discovered the inner planes from the rack's interfaces: + metal-admin `10.12.8.0/22` = VLAN 5005/fabric-4, provider-public + `10.12.4.0/22` = VLAN 5006/fabric-5. (Stale `192.168.1.0/24` also sits on + fabric-5 -- the edge's old factory LAN, now vacated; cleanup queued.) +- **RULING RECORD (GA-R5), 2026-07-20.** Question: metal-admin is a /22 but + D-120's band layout is written for a compose /24; what dynamic range? + Operator answer, exact utterance (option selected): "Inherit D-120 band: + 10.12.8.100-10.12.8.200 (Recommended)". Range CREATED and verified on the + correct subnet; VLAN 5005 updated `dhcp_on=true primary_rack=7chphy`, + verified by read-back (`10.12.8.0/22 dhcp=True primary=7chphy`). + +**INCIDENT (found by verification, not by a self-report):** +- Canary `virsh reset vr1-dc0-control-01` did NOT enlist. MEASURED cause + chain: rack service_set shows `dhcpd: off`; no dhcpd process on the rack; + no `/var/snap/maas/common/maas/dhcpd.conf` generated -- while MAAS's own + API reports the VLAN as `dhcp_on=True` with the correct primary rack and + the interface correctly linked (VLAN 5005, static 10.12.8.2). Config + state and running state DISAGREE. +- Root cause, from the region's journal: MAAS 3.7 drives DHCP through + **Temporal workflows**, and Temporal is wedged -- + `"Not enough hosts to serve the request"`, `poll_workflow_task_queue + retried 2807 times`, task queue `mtstwf@region:...`. No worker is polling, + so the workflow that pushes DHCP config to a rack can never run. +- **SCOPE IS WIDER THAN THIS DEPLOY:** `voffice1` itself has **NO dhcpd + process** (`pgrep -c dhcpd` -> 0) and reports no dhcpd service, while + Office1's compose net `10.10.1.0/24` reads `dhcp=True`. So MAAS DHCP has + been DOWN REGION-WIDE -- almost certainly since the 2026-07-17 host reboot + -- and went unnoticed because office1-netbox/office1-tailscale were + already Deployed and kept their addresses. **This is NOT caused by the DC0 + deploy; the deploy is what exposed it.** +- Consequence for the as-built record: any claim that Office1 MAAS DHCP is + "working" is currently FALSE (last positive evidence predates the reboot). +- **Detection gap (queued):** `cloud-assert.sh` and the Office1 health + checks did not catch a dead DHCP server, because MAAS's own service_set + reported the region healthy. A behavioral DHCP assertion belongs in the + health sweep -- this is precisely the "service's own verdict" principle + the repo already applies elsewhere. +- No prior art: appendix-A has no entry for this symptom. NEW incident + material once the fix is known. +- PROPOSED REMEDY (gated, NOT run): restart MAAS on the region + (`sudo snap restart maas` on voffice1) to re-establish Temporal workers, + then re-verify dhcpd on BOTH controllers behaviorally. Blast radius is + low right now (no OpenStack workloads; both Office1 VMs already deployed + and unaffected by a brief DHCP/API outage), which makes this the cheapest + moment to fix it. + ## 8. STAGE-CLOSE QUEUE ITEM (operator directive, 2026-07-20): OPNsense 26.7 review - Operator directive (verbatim intent): at the proper time/place, review the