diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index e73ce22..90c5b28 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1741,6 +1741,28 @@ **FLAGGED, not blocking: dc0 `/var/lib/libvirt` has 2.0T available against 4 x 500 GiB = 2.0 TiB nominal.** Thin-provisioned so creation is nearly free, but there is no room for those OSDs to FILL. dc1 has 2.9T. +- **dc0 SUBSTRATE APPLIED 2026-07-29 (operator: "Apply dc0 first") -- R1's OSD volumes and + the D-104 controller VM are BUILT on dc0.** Capture `docs/audit/dc0-osd-juju-apply-20260729.txt`. + Executed from voffice1 via a SAVED PLAN (what was applied is exactly what was reviewed); + same-session pre-apply re-verify per the G8 precedent. **Apply complete: 6 added, 4 changed, + 0 destroyed** -- 4 OSD volumes + the juju-01 domain and its disk, and the 4 storage domains + updated in-place to attach `vdb`. + **THE CHECKPOINT HELD: MAC DRIFT COUNT 0.** Verified by `virsh domiflist` against + `lib-hosts` BEFORE anything touched MAAS -- all 9 nodes pinned==live, 6 NICs each. This is + the step that makes 2026-07-20 non-repeatable: that event became a fleet-wide outage because + MAAS was told to re-commission while its records were already stale, after a 0/9/0 in-place + apply had silently regenerated every MAC **without the plan showing it**. So the clean plan + was treated as necessary and never sufficient. **First live evidence that the 2026-07-21 MAC + pinning holds through an in-place domain update.** + **CONVERGENCE RESTORED (precondition 1): the dc0 inner root re-plans ZERO DIFF.** MAAS + undisturbed -- 20 machines (18 Ready + 2 Deployed), 216 links, every one still `mode=static`. + **MAAS still sees ONE block device per dc0 node**, which is the expected state and is why the + re-commission is genuinely required. **NEXT, GATED SEPARATELY: `maas admin machine commission + skip_networking=1` on the four dc0 storage nodes** (storage re-scanned, network + preserved), then verify block-device count 1 -> 2 with links still 216/static. + **dc1 is NOT applied**; its plan is identical in shape (6/4/0). + **STILL FLAGGED: dc0 `/var/lib/libvirt` 2.0T available vs 2.0 TiB nominal** -- all four + volumes created in 0s because qcow2 is thin, but there is no room for those OSDs to FILL. - **THREE PLATFORM BEHAVIOURS GRADUATED to `references/platform-traps.md`** at session close, having been recorded only in this status document (which is consolidated over time, so a durable trap does not belong here alone): MAAS auto-reserves `::1`-`::ffff:ffff` on EVERY diff --git a/docs/audit/dc0-osd-juju-apply-20260729.txt b/docs/audit/dc0-osd-juju-apply-20260729.txt new file mode 100644 index 0000000..1fa19ba --- /dev/null +++ b/docs/audit/dc0-osd-juju-apply-20260729.txt @@ -0,0 +1,69 @@ +dc0 SUBSTRATE APPLY -- R1 OSD volumes + the D-104 controller VM, 2026-07-29 +========================================================================== +Operator approval: "Apply dc0 first". Executed FROM voffice1 (D-128 Plane 2 -- the inner +tfstate lives there). No run-logged.sh window was open; this capture plus the session +changelog are therefore the ENTIRE as-executed record. + +-- PRE-APPLY (same-session re-verify, the G8 precedent) ------------------------------ +Clone at c9cc79f (the commit that authored the change; the two later commits are +docs-only). Plan SAVED to /tmp/dc0-osd-juju.tfplan and applied from the saved plan, so +what was applied is exactly what was reviewed. + + 6 to add = 4 x libvirt_volume.osd + juju-01 domain + juju-01 disk volume + 4 to change = the 4 storage domains, IN-PLACE (devices.disks gaining vdb) + 0 to destroy + +MAC-line audit of the saved plan: ONE grep hit, and it was a FALSE POSITIVE -- +`type_machine = "q35"` inside the juju-01 CREATE block ("mac" is a substring of +"machine"). ZERO actual MAC-address lines in the diff. Recorded because the grep looked +alarming and the resolution is the kind of thing that should not have to be rediscovered. + +-- APPLY ----------------------------------------------------------------------------- + Apply complete! Resources: 6 added, 4 changed, 0 destroyed. + +-- THE CHECKPOINT: MACs verified BEFORE anything touched MAAS ------------------------- +This is the step that makes the 2026-07-20 incident non-repeatable. That event became a +fleet-wide outage because MAAS was told to re-commission while its records were already +stale; a 0/9/0 in-place apply had silently regenerated every NIC MAC and the PLAN DID NOT +SHOW IT. So a clean plan was treated as necessary, never sufficient. + + virsh -c qemu+ssh://jessea123@172.31.0.2/system domiflist + vs scripts/lib-hosts.sh HOST_BOOT_MAC (lib_hosts_select_dc vr1-dc0) + + all 9 nodes: pinned == live. MAC DRIFT COUNT: 0 + all 9 nodes: 6 NICs each (a regenerated set would also have shown here) + + MAC pinning (shipped 2026-07-21 precisely because the plan was untrustworthy on this + point) HELD through an in-place domain update. First live evidence of that. + +-- POST-STATE ------------------------------------------------------------------------ + domblklist, all four dc0 storage nodes: vda vdb + domblklist, vr1-dc0-juju-01: vda -> vr1-dc0-juju-01-disk.qcow2 + + CONVERGENCE RE-PLAN (precondition 1): "no differences, so no changes are needed" + -> the inner root is back to ZERO DIFF. The property the amendment said this apply + would deliberately spend has been restored and captured. + + MAAS, unchanged and undisturbed: + 20 machines: 18 Ready + 2 Deployed + 18 VR1 role nodes, 216 links total, EVERY link still mode=static + dc0 nodes by block-device count: {1: 9} + + That last line is the important one: MAAS still sees ONE block device per dc0 node. The + new vdb exists in libvirt but NOT in MAAS's inventory, which is exactly the expected + state and is why the re-commission is genuinely required rather than optional. + +-- NEXT, GATED SEPARATELY ------------------------------------------------------------- + maas admin machine commission skip_networking=1 (4 dc0 storage nodes) + + skip_networking=1 preserves the 12 static links per node; skip_storage is left UNSET so + storage IS re-scanned and vdb enters inventory. Verify afterwards that block-device + count goes 1 -> 2 and that links stay at 216 / all static. + + dc1 has NOT been applied. Its plan is identical in shape (6/4/0). + +-- STILL FLAGGED, not blocking -------------------------------------------------------- + dc0 /var/lib/libvirt: 2.0T available vs 4 x 500 GiB = 2.0 TiB nominal. Thin-provisioned, + so the volumes cost ~nothing at creation (all four created in 0s), but there is no room + for those OSDs to FILL. dc1 has 2.9T. A capacity decision is owed before the cloud + carries real data. diff --git a/docs/changelog-20260728-vip-arity-gate.md b/docs/changelog-20260728-vip-arity-gate.md index ae56587..52b4272 100644 --- a/docs/changelog-20260728-vip-arity-gate.md +++ b/docs/changelog-20260728-vip-arity-gate.md @@ -872,3 +872,44 @@ statics and the pinned MACs -- and it names the 2026-07-20 MAC-regeneration incident as the precedent for exactly this class. That verification needs live MAAS, which is absent on this host. **I am not treating it as satisfied.** + +--- + +# 2026-07-29 -- dc0 SUBSTRATE APPLIED (live mutation, operator-gated) + +## 35. R1's OSD volumes and the D-104 controller VM are BUILT on dc0 + +Operator approval, exact utterance: **"Apply dc0 first"**. Executed from voffice1 (D-128 +Plane 2 -- the inner tfstate lives there) via a SAVED PLAN, with a same-session pre-apply +re-verify per the G8 precedent, so what was applied is exactly what was reviewed. + +**Apply complete: 6 added, 4 changed, 0 destroyed.** Four `libvirt_volume.osd` + the +`vr1-dc0-juju-01` domain and its disk; the four storage domains updated IN-PLACE to attach +`vdb`. + +**THE CHECKPOINT HELD -- MAC DRIFT COUNT 0.** Verified by `virsh domiflist` against +`lib-hosts`' pinned MACs BEFORE anything touched MAAS: all 9 nodes pinned==live, 6 NICs +each. This is the whole point of sequencing the check there. 2026-07-20 became a +fleet-wide outage because MAAS was told to re-commission while its records were already +stale -- a 0/9/0 in-place apply had regenerated every MAC and **the plan did not show it**. +So the clean plan was necessary and explicitly not sufficient. **First live evidence that +the 2026-07-21 MAC pinning survives an in-place domain update.** + +**A grep that looked alarming and was not:** the saved plan's MAC audit returned one hit, +`type_machine = "q35"` inside the juju-01 CREATE block -- "mac" is a substring of +"machine". Zero actual MAC lines. Recorded so nobody has to rediscover it under pressure. + +**Convergence restored (precondition 1):** the dc0 inner root re-plans **ZERO DIFF**. The +property the amendment said this apply would deliberately spend is back and captured. + +**MAAS undisturbed:** 20 machines (18 Ready + 2 Deployed), 216 links, every one still +`mode=static`. And **MAAS still sees ONE block device per dc0 node** -- the new `vdb` is in +libvirt but not yet in MAAS's inventory, which is the expected state and confirms the +re-commission is genuinely required rather than a formality. + +**Revert.** `tofu destroy -target` on the five new resources would remove the volumes and +the controller domain; the four in-place disk attachments revert by setting `osd_gib` back +out of the storage entries and re-applying. Nothing was destroyed, so nothing is lost. + +**NOT DONE, gated separately:** the re-commission +(`maas admin machine commission skip_networking=1`), and dc1 entirely.