Newer
Older
openstack-caracal-dc-dc / docs / audit / queued-findings-20260729.txt
QUEUED FINDINGS -- 2026-07-29 session close sweep
=================================================
Content that existed ONLY in this session transcript. Captured before compaction, per the
precedent of queued-findings-20260726/-20260727. NONE of this is ruled or built; it is
material for the Stage-5 close skill sweep (deferred by operator direction 2026-07-27).

-- A. THE STRICT-BASH QUOTING TRAP FAMILY (three distinct instances, one session) ------
These gate scripts run `set -euo pipefail` + `inherit_errexit` + `IFS=$'\n\t'`. That
combination turns three ordinary-looking constructs into silent or confusing failures.
Each was MEASURED here, not theorised:

  A1. A SPACE-SEPARATED STRING DOES NOT WORD-SPLIT under IFS=$'\n\t'.
      `VIP_SRC="a b"; grep ... $VIP_SRC` passed BOTH names to grep as ONE filename; grep
      exited 2, pipefail propagated, and pre-flight-checks DIED SILENTLY mid-CHECK-1.
      Same root cause made maas-role-tags report a single absurd tag literally named
      'control compute storage'. ARRAYS ARE IMMUNE -- use `X=(a b)` / `"${X[@]}"`.

  A2. A BARE `$( )` ABORTS THE SCRIPT rather than falling through to a refusal branch.
      `LBNET_MTU="$([ -n "$X" ] && openstack ...)"` exits non-zero when X is empty; with
      inherit_errexit the whole run dies instead of reaching the "could not look" REFUSE.
      Every capture in a refusal path needs an explicit `|| true`.

  A3. AN APOSTROPHE INSIDE A SINGLE-QUOTED SHELL STRING that wraps embedded python
      terminates the string, and bash then parses python. One word ("host's") in a comment
      failed all 11 harness cases at once with a shell syntax error pointing at a python
      line. Embedded-python blocks in these scripts must contain NO apostrophes.

-- B. TWO GREPS THAT LOOKED LIKE FINDINGS AND WERE NOT ---------------------------------
  B1. `grep mac` MATCHES `type_machine`. A MAC-safety audit of a saved tofu plan -- run
      specifically because 2026-07-20's in-place apply regenerated every MAC -- returned
      one hit that was `type_machine = "q35"`. Under time pressure that reads as a live
      MAC change on a domain about to be applied. Grep `mac_address`, not `mac`.
  B2. `grep -c 'macs = \['` COUNTS COMMENTS. A config comment explaining a deliberate
      `macs = []` inflated a node-count assertion from 10 to 11 -- indistinguishable from a
      spurious extra node. Count CONFIG: strip comment lines first. Same class as the
      ledger-scan defect where a doc merely QUOTING a token inflated the next-free counter.

-- C. MAAS BEHAVIOURS MEASURED THIS SESSION --------------------------------------------
  C1. ENLISTMENT IS NOT COMMISSIONING. `New` is post-enlistment, pre-commissioning. A VM
      that PXE-boots into MAAS enlists on its own and sits in New; nothing has probed it.
  C2. MAAS AUTO-CONFIGURES POWER ONLY FOR IPMI MACHINES. Evidenced by the commission API
      help: "skip_bmc_config ... for IPMI based machines". A KVM guest has NO BMC, and the
      virsh parameters (qemu+ssh URI, credential, libvirt DOMAIN NAME) are HYPERVISOR facts
      nothing inside the guest can discover. Hence the chicken-and-egg that caused
      2026-07-20: commissioning needs power control, but for a virsh VM power config cannot
      be discovered BY commissioning. It MUST be set out-of-band first.
  C3. `skip_networking=1` PRESERVES NETWORK CONFIG EXACTLY while storage is still
      re-scanned. Measured on NINE nodes (1 canary + 8): every before/after diff was
      exactly one hunk, the new block device. Links, modes, addresses, subnets and
      per-interface MACs identical. This is the vendor control for "add a disk to an
      already-configured node".
  C4. MAAS TAGS SURVIVE re-commissioning (18/18 nodes verified).
  C5. The MAC-ADOPTION PLAN SHAPE: pinning MACs on an already-created domain plans
      `0 add / 1 change / 0 destroy`, adding `mac = { address = ... }` blocks whose values
      are IDENTICAL to live. That is ADOPTION, not drift -- tofu state simply has no `mac`
      block for a domain created unpinned. Until it is applied the pins protect NOTHING.

-- D. TWO GUARDS THAT CAN BE DEFEATED BY ACCIDENT --------------------------------------
  D1. `lib_hosts_select_dc` refuses a second DC in one shell (deliberate). Muting it with
      `2>&1` yields ONE DC's data silently labelled as the other's. Found by an agent that
      caught itself. Anyone scripting cross-DC lookups must re-source per DC.
  D2. NetBox's `?site=` API filter SILENTLY DOES NOT FILTER -- `/ipam/ip-addresses/?site=X`
      returned the full unfiltered set for both sites. A per-site count taken that way is
      wrong and looks right.

-- E. PROCESS NOTE: two live-ops agents in parallel ------------------------------------
  What made it safe, and is worth repeating: the AGENTS operated on the voffice1 clone
  while the orchestrator edited the vcloud clone, so repo edits could not reach their
  working tree mid-run and there was no git race. Agents were explicitly barred from
  `git commit`/`push`, so all recording stayed with the orchestrator.
  Both agents were given hard STOP conditions and both honoured them; one declared a STOP,
  re-measured, disclosed a reclassification and invited reversal -- which is the behaviour
  that made its result verifiable. Their reports were checked against an INDEPENDENT read
  of MAAS rather than accepted.
  COST: agents barred from repo writes cannot leave captures. dc1's evidence sat in
  voffice1:/tmp and had to be promoted by hand; without that it would have been lost to a
  reboot, and dc1 would have had no audit record while dc0 had one.