# INCIDENT 2026-07-21: stale-pod deletion cascaded to the 9 DC0 machine records # # Context: operator-ruled "Retire fully" for opentofu/vr1-dc0-maas (stage # close). Cleanup step 2 was deleting the stale pod object the failed # 2026-07-20 pod apply left on the region. # # WHAT WAS RUN (gated, in sequence): $ ssh voffice1 'maas admin vm-host delete 4' # stale pod vr1-dc0-inner (virsh) DELETE-EXIT=0 $ maas admin vm-hosts read # read-back id=1 name=office1-lxd type=lxd # pod gone, as intended $ maas admin machines read # cascade check -- RUN TOO LATE (see ERROR) 2 machines Counter({'Deployed': 2}) # office1-netbox, office1-tailscale ONLY # => the 9 DC0 machine records (previously ALL READY, shapes exact to D-121 # Option C, per-machine virsh power) were DELETED with the pod. # # ERROR (agent process failure, owned): the association check was performed # AFTER the delete. The correct order -- read the pod's machine list BEFORE # deleting -- was skipped on the reasoning "the pod registration failed, so # no machines can be associated". That reasoning was WRONG: the 2026-07-20 # pod apply created the pod object and its refresh got far enough (it died # on domblkinfo DURING enumeration) to link the nine existing machines to # pod id=4. MAAS deletes pod-linked machine RECORDS on vm-host delete even # without decompose. # # BLAST RADIUS (measured): # - Substrate INTACT: inside vvr1-dc0, virsh -c qemu:///system list --all # shows all 10 domains (vr1-dc0-opnsense RUNNING id 16; the 9 node VMs # shut off) -- no decompose reached libvirt (the pod's virsh path was # broken; no decompose was requested). # - MAC pins INTACT (config-pinned in the inner root since 2026-07-21). # - Rack services INTACT: dc0-node-dns / rack legs / dhcpd untouched. # - LOST: the 9 MAAS machine records -- enlistment identity, commissioning # results (Ready state), per-machine power_type=virsh wiring. # # RECOVERY PATH (the 2026-07-21 proven flow, all steps operator-gated): # 1. Power on the 9 node VMs on the rack (virsh start) -> they PXE-boot # to the region across the transit and ENLIST as New (pinned MACs). # 2. scripts/maas-node-power.sh (MAC-matched, dry-run first) re-wires # per-machine virsh power; verify by query-power-state. # 3. Commission the fleet -> Ready expected in minutes (07-21 proof: # canary ~3 min, fleet of 8 concurrent ~4 min, forwarder path live). # 4. Verify shapes against D-121 Option C, capture, correct CURRENT-STATE.