Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260721-commissioning-diag.md
@JANeumatrix JANeumatrix 6 hours ago 6 KB Session close 2026-07-21: commissioning RESOLVED 9/9 Ready; D-131 proposed; SEC-014 opened
Raw Blame History

Session changelog 2026-07-21 -- commissioning diagnosis + repair (logged: ops-commissioning-diag)

One changelog per session (GA-R2/D1). Status lives ONLY in docs/CURRENT-STATE.md; this file is narrative + reverts. Full adjudication: docs/audit/commissioning-diag-20260721.txt. As-executed log: ~/as-executed/2026-07-21-ops-commissioning-diag.log.

1. Instrumented commissioning run (committee-prescribed) EXECUTED

  • Phase 1 zero-cost reads, then gated: diag-serial MAAS tag (kernel_opts console=ttyS0,115200) on canary sw6wqe/hardy-dove only; serial-console tag temporarily DETACHED from the canary (alphabetical tag concatenation would have left console=tty0 last, sending userspace output to VGA). Instruments: MAC+broadcast tcpdump on virbr2, full tcpdump on enp1s0, live Temporal journal follow -- all 45-min capped.
  • One commission, hands-off, full window. Result: timeout at 30 min, serial 0 bytes -- and the pcap held the answer (item 2).
  • FINDING (record divergence): a serial-console tag with console kernel_opts ALREADY existed (comment cites D-114 debug) contradicting the committed "ephemeral kernel is not told console=ttyS0" claim.
  • Revert: diag-serial tag delete + re-attach serial-console to sw6wqe (Phase 7, item 8).

2. FAULT 1 FOUND + REPAIRED: all 9 node MACs drifted (in-place apply)

  • PXE DHCP in the capture came from a MAC absent from MAAS's records; fleet audit: 9/9 live metal-admin MACs differ from MAAS boot-interface records. Boundary: the 2026-07-20 in-place serial-console apply (tofu 0/9/0) regenerated NIC MACs. Full chain in the adjudication file.
  • RULING (GA-R5, 2026-07-21): question = how to repair the records; operator selection = "Update MACs in place (Recommended)".
  • Executed: per-machine boot-interface MAC -> measured live value; Failed-state machines needed mark-broken -> update -> mark-fixed. Read-back verified 9/9.
  • Revert: maas admin interface update mac_address=<old value from this file's adjudication capture> per machine (old values in the as-executed log); machines return to their pre-session states via mark-broken/mark-fixed as needed.

3. FAULT 2 FOUND: MAAS 3.7 rack-only agent resolver defect

  • Post-repair proof run booted fully; cloud-init found no datasource: the agent resolver (node DNS via DHCP) SERVFAILs everything on an isolated rack -- walks public root hints even for authoritative maas-internal; ignores resolv.conf. Office1 immune (region BIND owns node DNS there). First-exercised surface in VR1. LP REPORT QUEUED (item 10). SEC-010 re-verified enforced during diagnosis.
  • Interim RULING "Region upstream, runtime": resolvectl dns enp1s0 10.10.0.20 + domain ~. on the rack + /etc/resolv.conf relinked to /run/systemd/resolve/resolv.conf. Did NOT affect the agent (kept -- gives the rack itself working resolution via region).
  • Revert: resolvectl revert enp1s0; ln -sf ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf.

4. FAULT 2 WORKAROUND: dc0-node-dns forwarder (operator-ruled)

  • RULING (GA-R5, 2026-07-21): question = node DNS remedy; operator selection = "Rack dnsmasq forwarder (Recommended)".
  • Shipped to the rack (NOT yet repo-carried -- item 9): /etc/dnsmasq-dc0-node.conf + /etc/systemd/system/dc0-node-dns.service (dnsmasq on virbr2 alias 10.12.8.3, no-resolv, server=10.10.0.20; alias via ExecStartPre ip addr replace; After=libvirtd). Enabled.
  • MAAS: subnet 6 (metal-admin) dns_servers=10.12.8.3 + allow_dns=false. dhcpd still prepends the rack IP (option 6 = "10.12.8.2, 10.12.8.3") -- works via SERVFAIL failover (proven); template behavior to investigate at delivery.
  • Revert: maas admin subnet update 6 dns_servers= allow_dns=true; on rack: systemctl disable --now dc0-node-dns; rm /etc/systemd/system/dc0-node-dns.service /etc/dnsmasq-dc0-node.conf; ip addr del 10.12.8.3/22 dev virbr2.

5. PROOF + fleet re-commission

  • Canary hardy-dove: Ready in ~3 min after seven consecutive 30-min failures; commissioning scripts visible on serial, rc=0.
  • Fleet: remaining 8 commissioned (5 never truly commissioned -- mark-fixed does not commission; 3 carried stale interface data). Outcome recorded in CURRENT-STATE (same commit).
  • Revert: n/a (commissioning is re-runnable).

6. SEC-014 OPENED: rack cluster secret exposed to session context

  • agent.yaml read (diagnosing the resolver) included the secret field. Exposure scope: Claude session context, operator terminal, as-executed log (0600, jumphost). Row + disposition: docs/security-ledger.md. Rotation = rack re-enrollment; schedule at operator convenience.
  • Revert: n/a (exposure event; row tracks rotation).

7. Session tooling: targeted ask rules (classifier walls)

  • .claude/settings.local.json (untracked, global-gitignored): ask rules added for wrapped/unwrapped maas admin tags|tag|machine|interface| subnet via voffice1 and rack-sudo via the dc0 key -- per the standing multi-workstation pattern (operator-approved this session). Lesson: a sleep N; prefix breaks prefix-matching -- keep wrapped commands rule-shaped.
  • Revert: remove the ask entries from settings.local.json.

8. Phase 7 cleanup (pending fleet-green)

  • diag-serial tag: detach from sw6wqe + delete; re-attach serial-console. QUEUED OBSERVATION: serial-console's opts end in console=tty0, which claims /dev/console for VGA -- fine for kernel logs, hides userspace; revisit the tag's opts when serial observability next matters.
  • Revert: n/a (is itself the revert).

9. QUEUED delivery work (change-delivery loop, not this session)

  • modules/node-vm: PIN NIC MACs (set to current live values) so no apply can drift them again; harness case for MAC stability. Roosevelt note: metal has fixed MACs; the trap is VR-only.
  • platform-traps entry: tofu/libvirt "in-place" 0/N/0 domain change can REGENERATE NIC MACs (measured 2026-07-20/21).
  • appendix-A entries: (a) "commissioning times out at exactly 30 min, NO 'Performing PXE boot' event" -> diff live domain MACs vs MAAS records; (b) "cloud-init: no datasource, searched classes ()" on rack-only controllers -> query the rack agent resolver for maas-internal; if SERVFAIL see D-131/forwarder.
  • dc0-node-dns forwarder + rack-legs persistence: repo-carried script/module + tests (site-keyed, per D-128 register item 20).
  • D-131 PROPOSED (authored this session, design-decisions.md): node-facing DNS strategy for rack-only controllers (Roosevelt-relevant).
  • Committee record addendum: authored this session (outcome section).
  • Stale MAAS pod vr1-dc0-inner (id 4): stage-close cleanup with SEC-013.

10. LP report (queued for operator)

  • Title: "maas-agent resolver on rack-only controller SERVFAILs authoritative maas-internal queries on internet-isolated racks (walks root hints; ignores resolv.conf); breaks commissioning cloud-config-url" MAAS 3.7.2 / snap 41649. Evidence: adjudication file + agent journal excerpts in the as-executed log.