Newer
Older
openstack-caracal-dc-dc / docs / audit / dc0-osd-juju-apply-20260729.txt
dc0 SUBSTRATE APPLY -- R1 OSD volumes + the D-104 controller VM, 2026-07-29
==========================================================================
Operator approval: "Apply dc0 first". Executed FROM voffice1 (D-128 Plane 2 -- the inner
tfstate lives there). No run-logged.sh window was open; this capture plus the session
changelog are therefore the ENTIRE as-executed record.

-- PRE-APPLY (same-session re-verify, the G8 precedent) ------------------------------
Clone at c9cc79f (the commit that authored the change; the two later commits are
docs-only). Plan SAVED to /tmp/dc0-osd-juju.tfplan and applied from the saved plan, so
what was applied is exactly what was reviewed.

  6 to add    = 4 x libvirt_volume.osd + juju-01 domain + juju-01 disk volume
  4 to change = the 4 storage domains, IN-PLACE (devices.disks gaining vdb)
  0 to destroy

MAC-line audit of the saved plan: ONE grep hit, and it was a FALSE POSITIVE --
`type_machine = "q35"` inside the juju-01 CREATE block ("mac" is a substring of
"machine"). ZERO actual MAC-address lines in the diff. Recorded because the grep looked
alarming and the resolution is the kind of thing that should not have to be rediscovered.

-- APPLY -----------------------------------------------------------------------------
  Apply complete! Resources: 6 added, 4 changed, 0 destroyed.

-- THE CHECKPOINT: MACs verified BEFORE anything touched MAAS -------------------------
This is the step that makes the 2026-07-20 incident non-repeatable. That event became a
fleet-wide outage because MAAS was told to re-commission while its records were already
stale; a 0/9/0 in-place apply had silently regenerated every NIC MAC and the PLAN DID NOT
SHOW IT. So a clean plan was treated as necessary, never sufficient.

  virsh -c qemu+ssh://jessea123@172.31.0.2/system domiflist <domain>
  vs scripts/lib-hosts.sh HOST_BOOT_MAC (lib_hosts_select_dc vr1-dc0)

  all 9 nodes: pinned == live.   MAC DRIFT COUNT: 0
  all 9 nodes: 6 NICs each (a regenerated set would also have shown here)

  MAC pinning (shipped 2026-07-21 precisely because the plan was untrustworthy on this
  point) HELD through an in-place domain update. First live evidence of that.

-- POST-STATE ------------------------------------------------------------------------
  domblklist, all four dc0 storage nodes:  vda vdb
  domblklist, vr1-dc0-juju-01:             vda -> vr1-dc0-juju-01-disk.qcow2

  CONVERGENCE RE-PLAN (precondition 1): "no differences, so no changes are needed"
    -> the inner root is back to ZERO DIFF. The property the amendment said this apply
       would deliberately spend has been restored and captured.

  MAAS, unchanged and undisturbed:
    20 machines: 18 Ready + 2 Deployed
    18 VR1 role nodes, 216 links total, EVERY link still mode=static
    dc0 nodes by block-device count: {1: 9}

  That last line is the important one: MAAS still sees ONE block device per dc0 node. The
  new vdb exists in libvirt but NOT in MAAS's inventory, which is exactly the expected
  state and is why the re-commission is genuinely required rather than optional.

-- NEXT, GATED SEPARATELY -------------------------------------------------------------
  maas admin machine commission <sysid> skip_networking=1     (4 dc0 storage nodes)

  skip_networking=1 preserves the 12 static links per node; skip_storage is left UNSET so
  storage IS re-scanned and vdb enters inventory. Verify afterwards that block-device
  count goes 1 -> 2 and that links stay at 216 / all static.

  dc1 has NOT been applied. Its plan is identical in shape (6/4/0).

-- STILL FLAGGED, not blocking --------------------------------------------------------
  dc0 /var/lib/libvirt: 2.0T available vs 4 x 500 GiB = 2.0 TiB nominal. Thin-provisioned,
  so the volumes cost ~nothing at creation (all four created in 0s), but there is no room
  for those OSDs to FILL. dc1 has 2.9T. A capacity decision is owed before the cloud
  carries real data.