diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index d8a29d3..b2fb863 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1902,6 +1902,23 @@ DCs. **Every tag the bundle machines block constrains on now exists and is applied**, which was the loud-but-blocking allocation failure the chain audit measured. Gauntlet **ALL GREEN (87) on vcloud**; repo-lint 0 fail. +- **SESSION CLOSE SWEEP 2026-07-29** -- capture `docs/audit/queued-findings-20260729.txt`, per the + `queued-findings-20260726/-20260727` precedent. Transcript-only material preserved before + compaction: a **strict-bash quoting trap family** with THREE measured instances this session + (`IFS=$'\n\t'` defeats word-splitting on a space-separated string; a bare `$( )` under + `inherit_errexit` ABORTS instead of reaching a refusal branch; an apostrophe inside the + single-quoted shell string wrapping embedded python makes bash parse python); **two greps that + looked like findings and were not** (`grep mac` matches `type_machine`; `grep -c 'macs = ['` + counts COMMENTS -- the ledger-scan defect class); **five MAAS behaviours** including that + enlistment is NOT commissioning and that MAAS auto-configures power only for IPMI machines, so a + virsh VM must be power-configured OUT OF BAND first; **two guards defeatable by accident** + (`lib_hosts_select_dc`'s one-DC-per-shell guard muted by `2>&1`; NetBox's `?site=` filter + silently not filtering); and a **process note on running two live-ops agents in parallel** -- what + made it safe was that agents worked the voffice1 clone while the orchestrator edited vcloud's, so + no git race was possible, at the cost that agents barred from repo writes cannot leave captures + (dc1's evidence had to be promoted by hand from `/tmp` or it would have been lost). + **NOT graduated to the skill: the skill sweep stays DEFERRED to the Stage-5 close** by the + 2026-07-27 operator direction; this capture is its input. - **THREE PLATFORM BEHAVIOURS GRADUATED to `references/platform-traps.md`** at session close, having been recorded only in this status document (which is consolidated over time, so a durable trap does not belong here alone): MAAS auto-reserves `::1`-`::ffff:ffff` on EVERY diff --git a/docs/audit/queued-findings-20260729.txt b/docs/audit/queued-findings-20260729.txt new file mode 100644 index 0000000..3993744 --- /dev/null +++ b/docs/audit/queued-findings-20260729.txt @@ -0,0 +1,77 @@ +QUEUED FINDINGS -- 2026-07-29 session close sweep +================================================= +Content that existed ONLY in this session transcript. Captured before compaction, per the +precedent of queued-findings-20260726/-20260727. NONE of this is ruled or built; it is +material for the Stage-5 close skill sweep (deferred by operator direction 2026-07-27). + +-- A. THE STRICT-BASH QUOTING TRAP FAMILY (three distinct instances, one session) ------ +These gate scripts run `set -euo pipefail` + `inherit_errexit` + `IFS=$'\n\t'`. That +combination turns three ordinary-looking constructs into silent or confusing failures. +Each was MEASURED here, not theorised: + + A1. A SPACE-SEPARATED STRING DOES NOT WORD-SPLIT under IFS=$'\n\t'. + `VIP_SRC="a b"; grep ... $VIP_SRC` passed BOTH names to grep as ONE filename; grep + exited 2, pipefail propagated, and pre-flight-checks DIED SILENTLY mid-CHECK-1. + Same root cause made maas-role-tags report a single absurd tag literally named + 'control compute storage'. ARRAYS ARE IMMUNE -- use `X=(a b)` / `"${X[@]}"`. + + A2. A BARE `$( )` ABORTS THE SCRIPT rather than falling through to a refusal branch. + `LBNET_MTU="$([ -n "$X" ] && openstack ...)"` exits non-zero when X is empty; with + inherit_errexit the whole run dies instead of reaching the "could not look" REFUSE. + Every capture in a refusal path needs an explicit `|| true`. + + A3. AN APOSTROPHE INSIDE A SINGLE-QUOTED SHELL STRING that wraps embedded python + terminates the string, and bash then parses python. One word ("host's") in a comment + failed all 11 harness cases at once with a shell syntax error pointing at a python + line. Embedded-python blocks in these scripts must contain NO apostrophes. + +-- B. TWO GREPS THAT LOOKED LIKE FINDINGS AND WERE NOT --------------------------------- + B1. `grep mac` MATCHES `type_machine`. A MAC-safety audit of a saved tofu plan -- run + specifically because 2026-07-20's in-place apply regenerated every MAC -- returned + one hit that was `type_machine = "q35"`. Under time pressure that reads as a live + MAC change on a domain about to be applied. Grep `mac_address`, not `mac`. + B2. `grep -c 'macs = \['` COUNTS COMMENTS. A config comment explaining a deliberate + `macs = []` inflated a node-count assertion from 10 to 11 -- indistinguishable from a + spurious extra node. Count CONFIG: strip comment lines first. Same class as the + ledger-scan defect where a doc merely QUOTING a token inflated the next-free counter. + +-- C. MAAS BEHAVIOURS MEASURED THIS SESSION -------------------------------------------- + C1. ENLISTMENT IS NOT COMMISSIONING. `New` is post-enlistment, pre-commissioning. A VM + that PXE-boots into MAAS enlists on its own and sits in New; nothing has probed it. + C2. MAAS AUTO-CONFIGURES POWER ONLY FOR IPMI MACHINES. Evidenced by the commission API + help: "skip_bmc_config ... for IPMI based machines". A KVM guest has NO BMC, and the + virsh parameters (qemu+ssh URI, credential, libvirt DOMAIN NAME) are HYPERVISOR facts + nothing inside the guest can discover. Hence the chicken-and-egg that caused + 2026-07-20: commissioning needs power control, but for a virsh VM power config cannot + be discovered BY commissioning. It MUST be set out-of-band first. + C3. `skip_networking=1` PRESERVES NETWORK CONFIG EXACTLY while storage is still + re-scanned. Measured on NINE nodes (1 canary + 8): every before/after diff was + exactly one hunk, the new block device. Links, modes, addresses, subnets and + per-interface MACs identical. This is the vendor control for "add a disk to an + already-configured node". + C4. MAAS TAGS SURVIVE re-commissioning (18/18 nodes verified). + C5. The MAC-ADOPTION PLAN SHAPE: pinning MACs on an already-created domain plans + `0 add / 1 change / 0 destroy`, adding `mac = { address = ... }` blocks whose values + are IDENTICAL to live. That is ADOPTION, not drift -- tofu state simply has no `mac` + block for a domain created unpinned. Until it is applied the pins protect NOTHING. + +-- D. TWO GUARDS THAT CAN BE DEFEATED BY ACCIDENT -------------------------------------- + D1. `lib_hosts_select_dc` refuses a second DC in one shell (deliberate). Muting it with + `2>&1` yields ONE DC's data silently labelled as the other's. Found by an agent that + caught itself. Anyone scripting cross-DC lookups must re-source per DC. + D2. NetBox's `?site=` API filter SILENTLY DOES NOT FILTER -- `/ipam/ip-addresses/?site=X` + returned the full unfiltered set for both sites. A per-site count taken that way is + wrong and looks right. + +-- E. PROCESS NOTE: two live-ops agents in parallel ------------------------------------ + What made it safe, and is worth repeating: the AGENTS operated on the voffice1 clone + while the orchestrator edited the vcloud clone, so repo edits could not reach their + working tree mid-run and there was no git race. Agents were explicitly barred from + `git commit`/`push`, so all recording stayed with the orchestrator. + Both agents were given hard STOP conditions and both honoured them; one declared a STOP, + re-measured, disclosed a reclassification and invited reversal -- which is the behaviour + that made its result verifiable. Their reports were checked against an INDEPENDENT read + of MAAS rather than accepted. + COST: agents barred from repo writes cannot leave captures. dc1's evidence sat in + voffice1:/tmp and had to be promoted by hand; without that it would have been lost to a + reboot, and dc1 would have had no audit record while dc0 had one. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 0b94f8f..81868bd 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -245,20 +245,19 @@ - NEXT: gate host-authority; `provider-bundle-check` ARITY gap (imminent); **voffice1 back to `main` at merge**; then the renderer. Body: `changelog-20260727-stage5-phase0.md`. -## SESSION CLOSE 2026-07-29 -- VIP arity gate, the renderer, ruling-3 commit 1 (bounded, GA-R4) +## SESSION CLOSE 2026-07-29 -- arity gate, renderer, ruling 3, chain audit, OSD carve LIVE (bounded, GA-R4) -- Branch `dc-dc-stage5-preconditions`, 6 commits `d2460d0..b952683`. NO stage opened/closed. - Scan re-verified: 3 decisions, SEC 21, D 138 / DOCFIX 205 / BUNDLEFIX 053 -- all unchanged. -- **ARITY GAP CLOSED** + R11's three ruled gate changes (band 50->99, `VIP_COUNT_EXPECT` 11->13, - `EXPECT_PUBLIC_VIP` STAYS 11, hacluster-principal-without-VIP now FAILS). Checker takes a v4 - triple OR a dual-family sextet; v6 legs validated against APEX-derived /64s; `prefer-ipv6` - COUPLED to arity BOTH ways -- the L3-9 order that drops the v6 legs is the one that exits 0. -- **RENDERER SHIPPED** (D-136 (D)): `derive`/`render` split where F2 froze the interface; - reproduces `overlays/vr1-dc1-vips.yaml` BYTE-FOR-BYTE, proven able to fail on 3 seeded faults. -- ORDER INVERTED BY MEASUREMENT -- renderer BEFORE the extraction, since the reproduction window - closes when the reconciliation lands. It then caught a real renderer bug within the hour: the - first dual-family v6 leg joined with ONE colon, malformed but plausible (T13/T13b). -- **RULING 3 COMMIT 1:** `bundle.yaml` VIP-FREE, dc0's VIPs a RENDERED overlay, neutrality proven - by identical checker output. phase-01's RUN block would have ABORTED THE DEPLOY (0/0/0 vs - 11/11/0); Octavia SAN KeyError -> empty VIP; CHECK 1 died on `IFS` splitting. -- NEXT: commit 2 (dual-stack + `.61`/`.62`). Gauntlet ALL GREEN (85) vcloud; lint 0 fail. Body: `docs/changelog-20260728-vip-arity-gate.md`. +- Branch `dc-dc-stage5-preconditions`, 22 commits pushed. NO stage opened/closed. Scan unchanged: 3 decisions, SEC 21, D 138 / DOCFIX 205 / BUNDLEFIX 053. +- **VIP arity gap CLOSED + R11's three ruled gate changes.** `prefer-ipv6` is coupled to arity BOTH ways, closing the L3-9 merge order in which dropping the v6 legs exited 0. +- **RENDERER SHIPPED** (D-136 (D)), reproducing a reviewed artifact BYTE-FOR-BYTE. **RULING 3, both commits:** `bundle.yaml` is VIP-FREE; both DCs dual-family, vault `.61` / designate `.62` BUILT. +- **CHAIN AUDIT (step 6):** the adversarial lens made the chain go wrong-but-GREEN three times -- R9's ruled drift gate had never been built and dc0's overlay was gated by nothing. + Built `tests/render-drift`, added 3 refusals, fixed 2 harness cases that could not fail. +- **LIVE, each gated and independently verified:** 8 OSD volumes + 2 controller VMs applied both DCs; 9 re-commissions with `skip_networking=1`, every diff EXACTLY the new device; + MAC adoption applied (drift 0 across 20 nodes, both roots ZERO DIFF); both controllers commissioned; all ruled MAAS tags created. **THE STAGE-5 ALLOCATION BLOCKER IS CLOSED.** +- **R1 precondition 3 CLOSED BY MEASUREMENT** -- `skip_networking=1` is the vendor control, and the MAC check sits BETWEEN apply and re-commission, which is what makes 2026-07-20 non-repeatable. +- **RULED (GA-R5):** v6 IP SANs on the Octavia controller cert (D-109 note); juju controller at utility `.5` AND the octet map is a STANDING CROSS-DC STANDARD (D-134 amendment) -- + divergence between DCs at the same octet is now a DEFECT, not a local choice. +- **Octavia: clean negative.** juju 3.6 DEEP-MERGES overlay `options` (charm v12.1.1 source), so octavia-pki cannot clobber the VIP; and nothing inside Octavia requires IPv4. +- **OWNED:** three strict-bash quoting traps, two false-positive greps, one fabricated flag, one red-lint push. All corrected on-surface; traps in `docs/audit/queued-findings-20260729.txt`. +- **NEXT:** octavia-pki generation -- the last item hard-failing preflight, and the generator is now per-DC. Gauntlet ALL GREEN (87) vcloud; repo-lint 0 fail. +- Body: `docs/changelog-20260728-vip-arity-gate.md`. Status ONLY in CURRENT-STATE.md.