commissioning-diag-20260721 -- instrumented diagnosis adjudication
Logged window: ~/as-executed/2026-07-21-ops-commissioning-diag.log (all
commands + outputs verbatim there; this file is the citable summary).
Committee charter run: docs/audit/commissioning-committee-20260720.md.

VERDICT: two stacked faults, both measured, both fixed/worked-around.

FAULT 1 -- MAC drift (all 9 nodes)
- Instrumented run (07:17:49 -> 07:48:51 timeout): domain restarted
  (libvirt Id 39 -> 40, power-cycle works) but PXE DHCP came from
  52:54:00:2b:ed:ab -- absent from MAAS's record for hardy-dove
  (52:54:00:c7:da:bc). MAAS saw an unknown node: no "Performing PXE boot"
  event, no diag-serial tag kernel_opts served (serial log 0 bytes all
  window), no association -- 30-min timer expired on a machine record
  whose real VM booted normally.
- Event-timeline boundary: 2026-07-20 17:40 + 18:16 runs show PXE events;
  every run after the in-place serial-console apply to all 9 node-vm
  domains (session changelog 2026-07-20 item 19; tofu plan said 0/9/0
  in-place) shows none. The apply regenerated every metal-admin NIC MAC.
- Full-fleet audit: 9/9 live MACs differed from MAAS boot-interface
  records (pairing per docs/audit/stepD-power-20260720.txt).
- Re-frames: "3 pass / 6 fail" = the 3 finished BEFORE the apply; item-20
  "isolated 1770s failure" = first post-apply run. Committee's
  timeout-refutation stands; its ranked mechanisms do not (Temporal: 136
  journal lines, 0 errors during the full window).
- REPAIR (ruling 2026-07-21, "Update MACs in place (Recommended)"):
  boot-interface MAC updated per machine to the measured live value;
  Failed-state machines via mark-broken -> interface update -> mark-fixed
  (MAAS requires New/Ready/Allocated/Broken). Read-back verified 9/9.

FAULT 2 -- MAAS 3.7 rack-only agent resolver defect (surfaced by fixing 1)
- Post-repair proof run: PXE recognized t+14s; Ubuntu 24.04.4 boots to
  login in ~54s (squashfs URL is IP-based, transfers fine); then
  cloud-init: "Did not find any data source, searched classes: ()".
  cloud-config-url uses http://10-12-8-0--22.maas-internal:5248/...;
  DHCP option 6 pointed nodes at the rack agent resolver (10.12.8.2:53).
- Agent SERVFAILs EVERY query (incl. its own authoritative zone),
  correlated per-query in its journal: "Failed to handle authoritative
  query error=dial udp <public-root-server>:53: connect: network is
  unreachable". Rack has no internet (by design). Agent ignores
  /etc/resolv.conf (proven: symlink to real upstream list + snap restart
  -> unchanged); resolver config is Temporal-pushed, nothing on disk.
- Office1 immune: agent binds no :53 there; region BIND owns node DNS on
  co-located controllers. The rack-only resolver is FIRST EXERCISED in
  VR1 and is defective on isolated racks. LP report queued.
- Region BIND (10.10.0.20, reachable over transit) answers
  10-12-8-0--22.maas-internal -> 10.12.8.2 and hardy-dove.maas correctly.
- SEC-010 re-verified ENFORCED mid-diagnosis: dedicated nft chain,
  priority filter-10, blanket enp1s0 FORWARD drop both directions. This
  rules out (by design) any remedy that forwards node traffic to the
  region; rulings below stay inside the boundary.
- Interim ruling "Region upstream, runtime": resolvectl on rack enp1s0
  (kept, harmless for the rack's own resolution) -- did NOT affect the
  agent. Superseded by:
- WORKAROUND (ruling 2026-07-21, "Rack dnsmasq forwarder (Recommended)"):
  dc0-node-dns.service on the rack -- dnsmasq bound to virbr2 alias
  10.12.8.3 (ip addr replace in ExecStartPre; After=libvirtd), no-resolv,
  server=10.10.0.20; rack OUTPUT traffic over transit, SEC-010 untouched.
  metal-admin subnet (id 6): dns_servers=10.12.8.3, allow_dns=false.
  NOTE: dhcpd still templates option 6 as "10.12.8.2, 10.12.8.3" (rack IP
  prepended despite allow_dns=false); acceptable -- stub resolvers fail
  over on SERVFAIL, proven by the green run below. Template behavior to
  be investigated at proper delivery.

PROOF
- Canary hardy-dove: Ready in ~3 minutes (after seven consecutive 30-min
  failures); commissioning scripts visible on serial (maas-lshw etc., all
  rc=0). Serial observability itself was enabled by the diag-serial tag
  (kernel_opts console=ttyS0,115200) -- the committee's instrument.
- Fleet: remaining 8 re-commissioned (5 had never truly commissioned --
  mark-fixed does not commission; 3 carried stale interface data).
  RESULT: see CURRENT-STATE + session changelog (recorded at terminal
  state of the fleet run).

OPEN / HONEST UNKNOWNS
- How the 3 nodes passed on 2026-07-20 evening: dhcpd.conf has been
  regenerated since; that day's option-6 value is unrecoverable. Does not
  affect the fixes; noted for completeness.
- Stale MAAS pod object "vr1-dc0-inner" (id 4) still registered (pod
  route was refuted) -- stage-close cleanup question with SEC-013.

SEC
- SEC-014 OPENED: /var/snap/maas/<rev>/agent.yaml read into session
  context (and as-executed log) included the rack cluster secret.
  Rotation = rack re-enrollment; see docs/security-ledger.md row.

Artifacts (host-side, not committed): rack /home/jessea123/
diag-{metal,transit}-20260721.pcap, region /home/jessea123/
diag-temporal-20260721.log, rack serial log
/var/lib/libvirt/vr1/staging/vr1-dc0-storage-04-serial.log.
