Newer
Older
openstack-caracal-dc-dc / docs / audit / osd-carve-20260729 / README.txt
OSD CARVE + CONTROLLER VM -- raw captures, 2026-07-29
=====================================================
Before/after MAAS snapshots for every re-commissioned node, plus dc1's plan and apply
output. Promoted here from voffice1:/tmp, which does not survive a reboot. Two background
agents produced these under operator direction; both were barred from repo writes, so
without this promotion dc1 would have had NO audit record at all (dc0 has
docs/audit/dc0-osd-juju-apply-20260729.txt).

  before.json / after.json                 dc0 storage-01 (the canary, run by hand)
  before-storage-0{2,3,4}.json / after-*   dc0 storage-02/03/04  (agent A)
  s0{1,2,3,4}.{before,after}.json          dc1 storage-01..04    (agent B)
  dc1-phase1-{plan,apply}.txt              dc1 substrate plan + apply output

WHAT THE SNAPSHOTS SHOW. Every before/after pair differs by exactly ONE substantive change:
a second block device `vdb` of 536870912000 bytes (500 GiB) appearing. links stay 12 -> 12,
per-interface MACs identical, per-interface link list (mode/ip/cidr) identical.

THE ONE EXTRA DIFFERENCE, disclosed rather than hidden: three of dc1's four pairs also show
`power_state` moving between `off` and `error`. Agent B initially declared a STOP on this,
re-classified it on measurement, and excluded `power_state` from its verdict for the
remaining nodes -- printing it as an observation. Its stated basis: 8 of 9 dc1 nodes already
read `error` BEFORE any mutation, and storage-01 re-read `off` three times over ~40s.

THAT JUDGMENT IS VINDICATED BY LATER MEASUREMENT, verified independently after both agents
finished: all 18 role nodes across both DCs read power_state `off`, including the five dc1
nodes never commissioned. So it was a transient MAAS power-query flap, not a credential or
power-path defect. SEC-016 (per-DC MAAS->libvirt power keys) is the plausible governing item
if it recurs; it is not live now.

INDEPENDENT VERIFICATION of both agents' reports (run by the orchestrating session, from
MAAS directly, NOT from the agents' output):
  fleet 22 = 18 Ready + 2 Deployed + 2 New
  18 VR1 role nodes | 216 links | 216 of 216 mode=static
  vr1-dc0 blockdevs {1:5, 2:4}   power {off:9}   tagcount {2:9}
  vr1-dc1 blockdevs {1:5, 2:4}   power {off:9}   tagcount {2:9}
  NEW 7n87bt moved-troll   4 cpu / 8192 MiB / power_type '' / tags [virtual]
  NEW p8tdwg square-ferret 4 cpu / 8192 MiB / power_type '' / tags [virtual]

Every claim in both agent reports matched. MAAS TAGS SURVIVED re-commissioning on all 18
nodes (tagcount 2 everywhere) -- agent A checked this unprompted because its snapshot code
structurally could not see tags. That removes a would-be Stage-5 Juju placement blocker.

STILL TRUE AND STILL BLOCKING for the two controllers: power_type is EMPTY on both, which is
the state that stalled all nine role nodes on 2026-07-20. They cannot be commissioned until
maas-node-power.sh sets power_type=virsh, which is now possible because both are in
lib-hosts as of the D-134 amendment (octet .5) with their MACs pinned.