diff --git a/.claude/skills/openstack-cloud-ops/references/platform-traps.md b/.claude/skills/openstack-cloud-ops/references/platform-traps.md index 8612fdc..0aa8049 100644 --- a/.claude/skills/openstack-cloud-ops/references/platform-traps.md +++ b/.claude/skills/openstack-cloud-ops/references/platform-traps.md @@ -167,6 +167,18 @@ in `virsh dominfo`, `destroy` + `undefine` the domain and re-apply (volumes survive an `undefine`). +Second corollary (measured 2026-07-20/21): the undefine/redefine also +**REGENERATES every UNPINNED NIC MAC**. The plan reads `0 to add, N to change, +0 to destroy`, the apply succeeds -- and every touched domain now has brand-new +MACs. Anything that keys on those MACs is silently stranded: in the 2026-07-21 +commissioning incident all 9 MAAS-enlisted nodes became UNKNOWN at PXE (no +"Performing PXE boot" event, silent 30-min timeouts), because a serial-console +retrofit apply had drifted every boot-interface MAC. **Rule: any domain whose +MACs are load-bearing (MAAS-enlisted nodes above all) carries config-pinned +MACs** -- `mac = { address = ... }` per interface; see `modules/node-vm`'s +`interface_macs` + `tests/node-vm`. VR-only trap: Roosevelt metal has fixed +MACs. + ### 1f. Two more upstream behaviours worth knowing (from the release notes) - `capacity_unit = "GiB"` on a volume caused **"Provider produced inconsistent @@ -419,6 +431,7 @@ | `Could not open '': Permission denied` (domain defines, then fails to START; nothing names AppArmor) -- check `dmesg` for `apparmor="DENIED" operation="open" profile="libvirt-"` | non-default libvirt pool path not granted in AppArmor | 2 | | `Provider produced inconsistent result after apply` on a volume with `capacity_unit` | provider < 0.9.4 | 1f | | plan says `will be updated in-place`, guest uptime resets anyway | provider undefines + redefines the domain | 1e | +| MAAS commissioning times out at 30 min, NO PXE event for a known machine | an in-place apply regenerated unpinned NIC MACs; MAAS records stale | 1e | | `Illegal variable name.` from an `ssh root@ '...'` | root's shell is tcsh; `$(...)` unsupported | 4d | | OPNsense boots on FACTORY DEFAULTS with a correct config ISO attached; console shows `>>> Invoking import script 'importer'` and nothing more | the Configuration Importer can never fire on a nano image | 4a | | a Kea/firewall API `set` returns success but the box's behaviour does not change | no `service/reconfigure` (or no firewall `apply`) call | 4c | diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index b5a5f04..c47aee9 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -160,8 +160,12 @@ pattern, repo-carried + part of DC standup definition-of-done; sub-2..4 OPEN -- status line in design-decisions.md is the authority). SEC-014 OPENED (rack cluster - secret exposure during diagnosis). Queued delivery: appendix-A - entries, LP report, stale pod object cleanup. Forwarder + rack-legs + secret exposure during diagnosis). Queued delivery: incident docs + SHIPPED 2026-07-21 (two appendix-A entries, platform-traps 1e second + corollary + index row, LP draft + `docs/audit/lp-draft-20260721-maas-agent-resolver.md` -- operator to + file). Still queued: stale pod object cleanup (stage close, with + SEC-013). Forwarder + rack-legs persistence SHIPPED 2026-07-21 as `scripts/dc-rack-net.sh` (D-131 sub-1 delivery; harness 14 cases; gauntlet 74 ALL GREEN) and **INSTALLED on the rack 2026-07-21 (operator-approved)**: install diff --git a/docs/audit/lp-draft-20260721-maas-agent-resolver.md b/docs/audit/lp-draft-20260721-maas-agent-resolver.md new file mode 100644 index 0000000..2b1d25c --- /dev/null +++ b/docs/audit/lp-draft-20260721-maas-agent-resolver.md @@ -0,0 +1,61 @@ +# LP report DRAFT (operator to file against `maas`) -- 2026-07-21 + +Copy-paste body below. Evidence pointers at the end are repo-internal -- +do not paste those into the public report; attach sanitized excerpts +instead (agent journal lines, dig outputs -- NO cluster secrets, the +2026-07-21 as-executed log contains SEC-014 material). + +--- + +**Title:** maas-agent resolver on rack-only controller SERVFAILs +authoritative maas-internal queries on internet-isolated racks (walks +root hints; ignores resolv.conf); breaks commissioning cloud-config-url + +**Affects:** MAAS 3.7.2 (snap 3.7/stable, rev 41649), rack-only +controller. + +**Environment:** one region+rack controller (site A) and one RACK-ONLY +controller (site B) enrolled to it over a routed point-to-point link. +The rack site is internet-isolated by design (no outbound DNS/IP reach +from the rack or its nodes; the rack reaches the region over the +point-to-point link only). Nodes at site B are KVM VMs PXE-booting from +the rack. + +**What happens:** commissioning nodes at the rack-only site PXE fine, +boot the ephemeral image fine, then cloud-init finds no datasource and +the node idles at a login prompt until the 30-minute node_timeout. The +commissioning cloud-config-url uses a `.maas-internal` name. +Node-facing DNS at a rack-only site is served by the maas-agent +resolver (the rack's dhcpd hands out the rack IP as DNS). That resolver +answers SERVFAIL to EVERY query -- including names in its own +authoritative `maas-internal` zone -- because it attempts full recursive +resolution from the public root hints (unreachable on an isolated +rack), and it ignores /etc/resolv.conf. There appears to be no on-disk +configuration surface for it (config is Temporal-pushed). + +**Expected:** the agent resolver should answer authoritatively for +maas-internal without recursion (it owns the zone), and/or fall back to +the host resolver configuration for everything else -- an isolated rack +with a healthy region link should be able to commission nodes. + +**Why co-located deployments never see this:** with region+rack on one +host, node DNS is served by the region's BIND (authoritative, +recursion via forwarders), so the defective path is only exercised by +rack-only controllers -- i.e. exactly the multi-site topology racks +exist for. + +**Workaround (proven):** dnsmasq on the rack bound to an alias on the +node-facing bridge, `no-resolv`, `server=` (reachable +over the rack->region link); point the subnet's dns_servers at the +alias with allow_dns=false. Commissioning then completes in minutes. +Note: dhcpd still PREPENDS the rack IP to DHCP option 6 despite +allow_dns=false, so clients succeed only via SERVFAIL failover to the +second server -- possibly a separate template bug worth its own look. + +--- + +Repo-internal evidence (do NOT paste): the agent journal excerpts and +dig outputs are in ~/as-executed/2026-07-21-ops-commissioning-diag.log +(0600; contains SEC-014 material -- sanitize before attaching); +adjudication chain docs/audit/commissioning-diag-20260721.txt; D-131 +context docs/design-decisions.md. diff --git a/docs/changelog-20260721-close-and-delivery.md b/docs/changelog-20260721-close-and-delivery.md index c64377d..2ce20b4 100644 --- a/docs/changelog-20260721-close-and-delivery.md +++ b/docs/changelog-20260721-close-and-delivery.md @@ -107,3 +107,16 @@ systemctl disable --now dc0-rack-legs dc0-node-dns; rm the four installed files; re-add the legs + old unit per changelog-20260721- commissioning-diag item 4's revert. + +## 6. Incident documentation (queued items 9b/9c/10 of the predecessor) + +- appendix-A: two new entries -- (a) 30-min timeout with NO PXE event + for a known machine -> MAC-record drift (diff domiflist vs MAAS + interfaces; repair + pin); (b) cloud-init "no datasource" on a + rack-only DC -> agent-resolver SERVFAIL (dig SOA check; D-131 + forwarder fix via dc-rack-net.sh). +- platform-traps 1e: second corollary (in-place apply REGENERATES + unpinned NIC MACs; pin rule) + verbatim-index row. +- LP draft authored: docs/audit/lp-draft-20260721-maas-agent-resolver.md + (operator to file; sanitization warning re SEC-014 material included). +- **Revert:** git revert this commit (docs only). diff --git a/runbooks/appendix-A-troubleshooting.md b/runbooks/appendix-A-troubleshooting.md index 59ca895..e19add3 100644 --- a/runbooks/appendix-A-troubleshooting.md +++ b/runbooks/appendix-A-troubleshooting.md @@ -710,3 +710,70 @@ already Deployed and holding its address. It only surfaced when a new DC's nodes tried to PXE. After ANY host reboot, check `pgrep -c dhcpd` on each controller rather than assuming DHCP came back with the VMs. + +## Symptom: commissioning times out at exactly 30 min; NO "Performing PXE boot" event for a known machine [2026-07-21] + +-------------------------------------------------------------------------------- +SYMPTOM: A previously-enlisted machine is commissioned; the node VM powers on + and PXE/DHCP traffic is visible on the boot bridge (tcpdump), but the + machine's MAAS event log records NO new PXE event, nothing tagged for + it happens, and commissioning dies silently at the 30-minute + node_timeout. A serial log, if configured, may hold 0 bytes (node + firmware writes to VGA, not serial -- serial alone does not make a + PXE-booting node observable). +-------------------------------------------------------------------------------- +CAUSE: The machine's recorded boot-interface MAC no longer matches the live + NIC. An OpenTofu "in-place" domain apply REGENERATES every unpinned + NIC MAC (plan 0/N/0 -- platform-traps 1e, second corollary), so every + boot arrives at MAAS as an UNKNOWN node: no event against the known + machine, no per-machine kernel opts, silent timeout. + +CHECK: Diff live against records, per machine: + + virsh domiflist # live MACs, on the VM host + maas interfaces read # recorded MACs + + Or capture on the boot bridge: DHCP Discover from a MAC absent from + MAAS's records is the whole story in one packet. + +FIX: (1) Repair records: per-machine boot-interface MAC update to the + measured live value (machines in Failed states need + mark-broken -> update -> mark-fixed); read-back verify every one. + (2) Close the class: PIN the MACs in config so no apply can drift + them again -- modules/node-vm `interface_macs` (all-or-nothing + validation; harness tests/node-vm). Full chain: + docs/audit/commissioning-diag-20260721.txt. + +## Symptom: ephemeral OS boots to a login prompt; cloud-init "no datasource found, searched classes ()" -- rack-only DC controller [2026-07-21] + +-------------------------------------------------------------------------------- +SYMPTOM: PXE and the ephemeral kernel handoff WORK (node holds a lease, does + NTP against the rack), but commissioning scripts never run: serial + shows cloud-init finding no datasource and the node sits at a login + prompt until the 30-minute timeout. Office1/VR0 never show this. +-------------------------------------------------------------------------------- +CAUSE: MAAS 3.7 RACK-ONLY controllers serve node-facing DNS from the + maas-agent resolver, which on an internet-isolated rack SERVFAILs + EVERY query -- including its own authoritative `maas-internal` zone -- + because it resolves by walking public root hints and ignores + /etc/resolv.conf (no on-disk config surface; Temporal-pushed). + cloud-init's commissioning cloud-config-url uses a maas-internal + name, so it never resolves. Co-located region+rack sites are immune + (region BIND owns node DNS) -- this surface only exists at a + rack-only DC. Upstream defect; LP report queued (changelog + 2026-07-21). + +CHECK: From the rack, query the node-facing resolver for its OWN zone: + + dig @ maas-internal SOA # SERVFAIL = this + +FIX: D-131 sub-1 STANDING pattern (ruled 2026-07-21): the repo-carried + forwarder -- + + scripts/dc-rack-net.sh install # on the rack, via ssh + + (dnsmasq on a metal-admin alias, no-resolv, forwards to region BIND + over the rack's own transit connection; SEC-010 untouched) plus the + region-side runbook step: metal-admin subnet dns_servers=, + allow_dns=false. Verify: `dig @ maas-internal SOA` answers; + a canary commission goes Ready in minutes instead of timing out.