commissioning-diag-20260721 -- instrumented diagnosis adjudication Logged window: ~/as-executed/2026-07-21-ops-commissioning-diag.log (all commands + outputs verbatim there; this file is the citable summary). Committee charter run: docs/audit/commissioning-committee-20260720.md. VERDICT: two stacked faults, both measured, both fixed/worked-around. FAULT 1 -- MAC drift (all 9 nodes) - Instrumented run (07:17:49 -> 07:48:51 timeout): domain restarted (libvirt Id 39 -> 40, power-cycle works) but PXE DHCP came from 52:54:00:2b:ed:ab -- absent from MAAS's record for hardy-dove (52:54:00:c7:da:bc). MAAS saw an unknown node: no "Performing PXE boot" event, no diag-serial tag kernel_opts served (serial log 0 bytes all window), no association -- 30-min timer expired on a machine record whose real VM booted normally. - Event-timeline boundary: 2026-07-20 17:40 + 18:16 runs show PXE events; every run after the in-place serial-console apply to all 9 node-vm domains (session changelog 2026-07-20 item 19; tofu plan said 0/9/0 in-place) shows none. The apply regenerated every metal-admin NIC MAC. - Full-fleet audit: 9/9 live MACs differed from MAAS boot-interface records (pairing per docs/audit/stepD-power-20260720.txt). - Re-frames: "3 pass / 6 fail" = the 3 finished BEFORE the apply; item-20 "isolated 1770s failure" = first post-apply run. Committee's timeout-refutation stands; its ranked mechanisms do not (Temporal: 136 journal lines, 0 errors during the full window). - REPAIR (ruling 2026-07-21, "Update MACs in place (Recommended)"): boot-interface MAC updated per machine to the measured live value; Failed-state machines via mark-broken -> interface update -> mark-fixed (MAAS requires New/Ready/Allocated/Broken). Read-back verified 9/9. FAULT 2 -- MAAS 3.7 rack-only agent resolver defect (surfaced by fixing 1) - Post-repair proof run: PXE recognized t+14s; Ubuntu 24.04.4 boots to login in ~54s (squashfs URL is IP-based, transfers fine); then cloud-init: "Did not find any data source, searched classes: ()". cloud-config-url uses http://10-12-8-0--22.maas-internal:5248/...; DHCP option 6 pointed nodes at the rack agent resolver (10.12.8.2:53). - Agent SERVFAILs EVERY query (incl. its own authoritative zone), correlated per-query in its journal: "Failed to handle authoritative query error=dial udp :53: connect: network is unreachable". Rack has no internet (by design). Agent ignores /etc/resolv.conf (proven: symlink to real upstream list + snap restart -> unchanged); resolver config is Temporal-pushed, nothing on disk. - Office1 immune: agent binds no :53 there; region BIND owns node DNS on co-located controllers. The rack-only resolver is FIRST EXERCISED in VR1 and is defective on isolated racks. LP report queued. - Region BIND (10.10.0.20, reachable over transit) answers 10-12-8-0--22.maas-internal -> 10.12.8.2 and hardy-dove.maas correctly. - SEC-010 re-verified ENFORCED mid-diagnosis: dedicated nft chain, priority filter-10, blanket enp1s0 FORWARD drop both directions. This rules out (by design) any remedy that forwards node traffic to the region; rulings below stay inside the boundary. - Interim ruling "Region upstream, runtime": resolvectl on rack enp1s0 (kept, harmless for the rack's own resolution) -- did NOT affect the agent. Superseded by: - WORKAROUND (ruling 2026-07-21, "Rack dnsmasq forwarder (Recommended)"): dc0-node-dns.service on the rack -- dnsmasq bound to virbr2 alias 10.12.8.3 (ip addr replace in ExecStartPre; After=libvirtd), no-resolv, server=10.10.0.20; rack OUTPUT traffic over transit, SEC-010 untouched. metal-admin subnet (id 6): dns_servers=10.12.8.3, allow_dns=false. NOTE: dhcpd still templates option 6 as "10.12.8.2, 10.12.8.3" (rack IP prepended despite allow_dns=false); acceptable -- stub resolvers fail over on SERVFAIL, proven by the green run below. Template behavior to be investigated at proper delivery. PROOF - Canary hardy-dove: Ready in ~3 minutes (after seven consecutive 30-min failures); commissioning scripts visible on serial (maas-lshw etc., all rc=0). Serial observability itself was enabled by the diag-serial tag (kernel_opts console=ttyS0,115200) -- the committee's instrument. - Fleet: remaining 8 re-commissioned (5 had never truly commissioned -- mark-fixed does not commission; 3 carried stale interface data). RESULT: see CURRENT-STATE + session changelog (recorded at terminal state of the fleet run). OPEN / HONEST UNKNOWNS - How the 3 nodes passed on 2026-07-20 evening: dhcpd.conf has been regenerated since; that day's option-6 value is unrecoverable. Does not affect the fixes; noted for completeness. - Stale MAAS pod object "vr1-dc0-inner" (id 4) still registered (pod route was refuted) -- stage-close cleanup question with SEC-013. SEC - SEC-014 OPENED: /var/snap/maas//agent.yaml read into session context (and as-executed log) included the rack cluster secret. Rotation = rack re-enrollment; see docs/security-ledger.md row. Artifacts (host-side, not committed): rack /home/jessea123/ diag-{metal,transit}-20260721.pcap, region /home/jessea123/ diag-temporal-20260721.log, rack serial log /var/lib/libvirt/vr1/staging/vr1-dc0-storage-04-serial.log.