diff --git a/docs/audit/r1-precondition3-verify-20260729.txt b/docs/audit/r1-precondition3-verify-20260729.txt new file mode 100644 index 0000000..5d9d5c7 --- /dev/null +++ b/docs/audit/r1-precondition3-verify-20260729.txt @@ -0,0 +1,86 @@ +R1 PRECONDITION 3 -- VERIFICATION CAPTURE, 2026-07-29 +===================================================== +Read-only. Run from voffice1 (the D-128 Plane-2 host; `maas` and `juju` live there, +both ABSENT on vcloud). No mutation performed. + +THE PRECONDITION, verbatim from the D-121 AMENDMENT 2026-07-27: + "MAAS must RE-COMMISSION the four storage nodes per DC afterwards for the new device + to appear in its inventory, and the audit did NOT verify whether re-commissioning + preserves the D-134 statics and the pinned MACs. Verify that before the apply, not + after -- the 2026-07-20 MAC-regeneration incident is the precedent." + +Two independent halves. Both now answered. + +-- HALF 1: does re-commissioning reset the network config? --------------------------- +BASELINE MEASURED (maas admin machines read): + 20 machines: 18 Ready + 2 Deployed (office1 guests) + all 18 VR1 role nodes: 7 interfaces, 12 links, mode=static on EVERY link + (12 = 6 planes x 2 families: the D-134 v4 carve + the 2026-07-27 v6 carve) + +MECHANISM, from `maas admin machine commission --help` on THIS MAAS (3.7.2): + :param skip_networking: Optional. Whether to skip + re-configuring the networking on the machine after the commissioning + has completed. '1' == True, '0' == False. + :param skip_storage: Optional. Whether to skip + re-configuring the storage on the machine after the commissioning has ... + +READING: the option exists BECAUSE the default re-configures networking after +commissioning. That is exactly the risk the amendment flagged. The combination R1 needs +is therefore: + + maas admin machine commission skip_networking=1 + + - skip_networking=1 -> the 12 static links and their addresses are NOT re-derived + - skip_storage UNSET -> storage IS re-scanned, which is the whole point (the new + /dev/vdb must enter MAAS's inventory) + +NOT ASSERTED: that skip_networking=1 is bit-for-bit lossless. It is the vendor's own +control for this exact case on this exact version, which is the strongest evidence +available without mutating. The post-apply verification below is what closes it. + +-- HALF 2: does the tofu change itself touch the NICs? ------------------------------- +The 2026-07-20 incident is the precedent BECAUSE the plan looked innocent: it reported +0/9/0 in-place and regenerated all 9 nodes' MACs anyway. So a clean plan is necessary +but NOT sufficient, and is not treated as sufficient here. + +MEASURED PLAN, both inner roots (tofu plan, -lock=false, read-only): + vr1-dc0-substrate: Plan: 6 to add, 4 to change, 0 to destroy + vr1-dc1-substrate: Plan: 6 to add, 4 to change, 0 to destroy + adds = 4 x libvirt_volume.osd + the juju-01 domain + its disk volume + changes = the 4 storage domains, IN-PLACE + +DIFF CONTENT on an in-place domain -- the ONLY change is devices.disks gaining vdb: + ~ resource "libvirt_domain" "node" { + ~ devices = { + ~ disks = [ + + { driver={type=qcow2} + source={volume={pool=vr1-dc1-inner-pool + volume=vr1-dc1-storage-01-osd.qcow2}} + target={bus=virtio dev=vdb} } + + grep for mac/interface/network_name across the whole diff returns exactly ONE hit, + `+ interfaces = [`, and it is inside the NEW juju-01 CREATE block -- not a change to + any existing domain. NO existing NIC is touched. + +WHY THE RISK IS MATERIALLY LOWER THAN 2026-07-20: the MACs are now CONFIG-PINNED +(`interface_macs`, shipped 2026-07-21 precisely because the plan was untrustworthy on +this point). The provider asserts them rather than leaving them to libvirt. + +-- CONCLUSION ----------------------------------------------------------------------- +Precondition 3 is SATISFIABLE, and the safe sequence is: + + 1. apply (creates the 4 volumes per DC, attaches vdb in-place) + 2. *** BEFORE re-commissioning ***, verify MACs are unchanged: + virsh -c qemu+ssh:///system domiflist vs lib-hosts HOST_*_MAC + This is the checkpoint that makes 2026-07-20 non-repeatable. That incident became + a fleet-wide outage because MAAS was told to re-commission while its records were + already stale. Checking here costs one command and removes the whole class. + 3. re-commission the 4 storage nodes per DC WITH skip_networking=1 + 4. verify 12 static links still present per node, and that /dev/vdb now appears in + physicalblockdevice_set + 5. re-converge both inner roots to ZERO DIFF and capture (precondition 1) + +STILL OPEN, unrelated to this precondition: + dc0 /var/lib/libvirt has 2.0T available against 4 x 500 GiB = 2.0 TiB nominal. + Thin-provisioned so creation is nearly free, but there is no room for those OSDs to + fill. dc1 has 2.9T. Flagged, not blocking the apply.