# Session changelog 2026-07-21 -- commissioning diagnosis + repair (logged: ops-commissioning-diag)

One changelog per session (GA-R2/D1). Status lives ONLY in
docs/CURRENT-STATE.md; this file is narrative + reverts. Full adjudication:
docs/audit/commissioning-diag-20260721.txt. As-executed log:
~/as-executed/2026-07-21-ops-commissioning-diag.log.

## 1. Instrumented commissioning run (committee-prescribed) EXECUTED

- Phase 1 zero-cost reads, then gated: diag-serial MAAS tag
  (kernel_opts console=ttyS0,115200) on canary sw6wqe/hardy-dove only;
  serial-console tag temporarily DETACHED from the canary (alphabetical
  tag concatenation would have left console=tty0 last, sending userspace
  output to VGA). Instruments: MAC+broadcast tcpdump on virbr2, full
  tcpdump on enp1s0, live Temporal journal follow -- all 45-min capped.
- One commission, hands-off, full window. Result: timeout at 30 min,
  serial 0 bytes -- and the pcap held the answer (item 2).
- FINDING (record divergence): a serial-console tag with console
  kernel_opts ALREADY existed (comment cites D-114 debug) contradicting
  the committed "ephemeral kernel is not told console=ttyS0" claim.
- **Revert:** diag-serial tag delete + re-attach serial-console to sw6wqe
  (Phase 7, item 8).

## 2. FAULT 1 FOUND + REPAIRED: all 9 node MACs drifted (in-place apply)

- PXE DHCP in the capture came from a MAC absent from MAAS's records;
  fleet audit: 9/9 live metal-admin MACs differ from MAAS boot-interface
  records. Boundary: the 2026-07-20 in-place serial-console apply
  (tofu 0/9/0) regenerated NIC MACs. Full chain in the adjudication file.
- RULING (GA-R5, 2026-07-21): question = how to repair the records;
  operator selection = "Update MACs in place (Recommended)".
- Executed: per-machine boot-interface MAC -> measured live value;
  Failed-state machines needed mark-broken -> update -> mark-fixed.
  Read-back verified 9/9.
- **Revert:** maas admin interface update <system_id> <if_id>
  mac_address=<old value from this file's adjudication capture> per
  machine (old values in the as-executed log); machines return to their
  pre-session states via mark-broken/mark-fixed as needed.

## 3. FAULT 2 FOUND: MAAS 3.7 rack-only agent resolver defect

- Post-repair proof run booted fully; cloud-init found no datasource:
  the agent resolver (node DNS via DHCP) SERVFAILs everything on an
  isolated rack -- walks public root hints even for authoritative
  maas-internal; ignores resolv.conf. Office1 immune (region BIND owns
  node DNS there). First-exercised surface in VR1. LP REPORT QUEUED
  (item 10). SEC-010 re-verified enforced during diagnosis.
- Interim RULING "Region upstream, runtime": resolvectl dns enp1s0
  10.10.0.20 + domain ~. on the rack + /etc/resolv.conf relinked to
  /run/systemd/resolve/resolv.conf. Did NOT affect the agent (kept --
  gives the rack itself working resolution via region).
- **Revert:** resolvectl revert enp1s0; ln -sf
  ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf.

## 4. FAULT 2 WORKAROUND: dc0-node-dns forwarder (operator-ruled)

- RULING (GA-R5, 2026-07-21): question = node DNS remedy; operator
  selection = "Rack dnsmasq forwarder (Recommended)".
- Shipped to the rack (NOT yet repo-carried -- item 9):
  /etc/dnsmasq-dc0-node.conf + /etc/systemd/system/dc0-node-dns.service
  (dnsmasq on virbr2 alias 10.12.8.3, no-resolv, server=10.10.0.20;
  alias via ExecStartPre ip addr replace; After=libvirtd). Enabled.
- MAAS: subnet 6 (metal-admin) dns_servers=10.12.8.3 + allow_dns=false.
  dhcpd still prepends the rack IP (option 6 = "10.12.8.2, 10.12.8.3")
  -- works via SERVFAIL failover (proven); template behavior to
  investigate at delivery.
- **Revert:** maas admin subnet update 6 dns_servers= allow_dns=true;
  on rack: systemctl disable --now dc0-node-dns; rm
  /etc/systemd/system/dc0-node-dns.service /etc/dnsmasq-dc0-node.conf;
  ip addr del 10.12.8.3/22 dev virbr2.

## 5. PROOF + fleet re-commission

- Canary hardy-dove: Ready in ~3 min after seven consecutive 30-min
  failures; commissioning scripts visible on serial, rc=0.
- Fleet: remaining 8 commissioned (5 never truly commissioned --
  mark-fixed does not commission; 3 carried stale interface data).
  Outcome recorded in CURRENT-STATE (same commit).
- **Revert:** n/a (commissioning is re-runnable).

## 6. SEC-014 OPENED: rack cluster secret exposed to session context

- agent.yaml read (diagnosing the resolver) included the `secret` field.
  Exposure scope: Claude session context, operator terminal, as-executed
  log (0600, jumphost). Row + disposition: docs/security-ledger.md.
  Rotation = rack re-enrollment; schedule at operator convenience.
- **Revert:** n/a (exposure event; row tracks rotation).

## 7. Session tooling: targeted ask rules (classifier walls)

- .claude/settings.local.json (untracked, global-gitignored): ask rules
  added for wrapped/unwrapped `maas admin tags|tag|machine|interface|
  subnet` via voffice1 and rack-sudo via the dc0 key -- per the standing
  multi-workstation pattern (operator-approved this session). Lesson:
  a `sleep N;` prefix breaks prefix-matching -- keep wrapped commands
  rule-shaped.
- **Revert:** remove the ask entries from settings.local.json.

## 8. Phase 7 cleanup (pending fleet-green)

- diag-serial tag: detach from sw6wqe + delete; re-attach serial-console.
  QUEUED OBSERVATION: serial-console's opts end in console=tty0, which
  claims /dev/console for VGA -- fine for kernel logs, hides userspace;
  revisit the tag's opts when serial observability next matters.
- **Revert:** n/a (is itself the revert).

## 9. QUEUED delivery work (change-delivery loop, not this session)

- modules/node-vm: PIN NIC MACs (set to current live values) so no apply
  can drift them again; harness case for MAC stability. Roosevelt note:
  metal has fixed MACs; the trap is VR-only.
- platform-traps entry: tofu/libvirt "in-place" 0/N/0 domain change can
  REGENERATE NIC MACs (measured 2026-07-20/21).
- appendix-A entries: (a) "commissioning times out at exactly 30 min,
  NO 'Performing PXE boot' event" -> diff live domain MACs vs MAAS
  records; (b) "cloud-init: no datasource, searched classes ()" on
  rack-only controllers -> query the rack agent resolver for
  maas-internal; if SERVFAIL see D-131/forwarder.
- dc0-node-dns forwarder + rack-legs persistence: repo-carried
  script/module + tests (site-keyed, per D-128 register item 20).
- D-131 PROPOSED (authored this session, design-decisions.md): node-facing
  DNS strategy for rack-only controllers (Roosevelt-relevant).
- Committee record addendum: authored this session (outcome section).
- Stale MAAS pod vr1-dc0-inner (id 4): stage-close cleanup with SEC-013.

## 10. LP report (queued for operator)

- Title: "maas-agent resolver on rack-only controller SERVFAILs
  authoritative maas-internal queries on internet-isolated racks (walks
  root hints; ignores resolv.conf); breaks commissioning cloud-config-url"
  MAAS 3.7.2 / snap 41649. Evidence: adjudication file + agent journal
  excerpts in the as-executed log.