dc0 SUBSTRATE APPLY -- R1 OSD volumes + the D-104 controller VM, 2026-07-29 ========================================================================== Operator approval: "Apply dc0 first". Executed FROM voffice1 (D-128 Plane 2 -- the inner tfstate lives there). No run-logged.sh window was open; this capture plus the session changelog are therefore the ENTIRE as-executed record. -- PRE-APPLY (same-session re-verify, the G8 precedent) ------------------------------ Clone at c9cc79f (the commit that authored the change; the two later commits are docs-only). Plan SAVED to /tmp/dc0-osd-juju.tfplan and applied from the saved plan, so what was applied is exactly what was reviewed. 6 to add = 4 x libvirt_volume.osd + juju-01 domain + juju-01 disk volume 4 to change = the 4 storage domains, IN-PLACE (devices.disks gaining vdb) 0 to destroy MAC-line audit of the saved plan: ONE grep hit, and it was a FALSE POSITIVE -- `type_machine = "q35"` inside the juju-01 CREATE block ("mac" is a substring of "machine"). ZERO actual MAC-address lines in the diff. Recorded because the grep looked alarming and the resolution is the kind of thing that should not have to be rediscovered. -- APPLY ----------------------------------------------------------------------------- Apply complete! Resources: 6 added, 4 changed, 0 destroyed. -- THE CHECKPOINT: MACs verified BEFORE anything touched MAAS ------------------------- This is the step that makes the 2026-07-20 incident non-repeatable. That event became a fleet-wide outage because MAAS was told to re-commission while its records were already stale; a 0/9/0 in-place apply had silently regenerated every NIC MAC and the PLAN DID NOT SHOW IT. So a clean plan was treated as necessary, never sufficient. virsh -c qemu+ssh://jessea123@172.31.0.2/system domiflist vs scripts/lib-hosts.sh HOST_BOOT_MAC (lib_hosts_select_dc vr1-dc0) all 9 nodes: pinned == live. MAC DRIFT COUNT: 0 all 9 nodes: 6 NICs each (a regenerated set would also have shown here) MAC pinning (shipped 2026-07-21 precisely because the plan was untrustworthy on this point) HELD through an in-place domain update. First live evidence of that. -- POST-STATE ------------------------------------------------------------------------ domblklist, all four dc0 storage nodes: vda vdb domblklist, vr1-dc0-juju-01: vda -> vr1-dc0-juju-01-disk.qcow2 CONVERGENCE RE-PLAN (precondition 1): "no differences, so no changes are needed" -> the inner root is back to ZERO DIFF. The property the amendment said this apply would deliberately spend has been restored and captured. MAAS, unchanged and undisturbed: 20 machines: 18 Ready + 2 Deployed 18 VR1 role nodes, 216 links total, EVERY link still mode=static dc0 nodes by block-device count: {1: 9} That last line is the important one: MAAS still sees ONE block device per dc0 node. The new vdb exists in libvirt but NOT in MAAS's inventory, which is exactly the expected state and is why the re-commission is genuinely required rather than optional. -- NEXT, GATED SEPARATELY ------------------------------------------------------------- maas admin machine commission skip_networking=1 (4 dc0 storage nodes) skip_networking=1 preserves the 12 static links per node; skip_storage is left UNSET so storage IS re-scanned and vdb enters inventory. Verify afterwards that block-device count goes 1 -> 2 and that links stay at 216 / all static. dc1 has NOT been applied. Its plan is identical in shape (6/4/0). -- STILL FLAGGED, not blocking -------------------------------------------------------- dc0 /var/lib/libvirt: 2.0T available vs 4 x 500 GiB = 2.0 TiB nominal. Thin-provisioned, so the volumes cost ~nothing at creation (all four created in 0s), but there is no room for those OSDs to FILL. dc1 has 2.9T. A capacity decision is owed before the cloud carries real data.