diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index dfcc826..8945165 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1347,16 +1347,67 @@ reach P0-1 (the gauntlet's `tofu fmt` walks the filesystem, so voffice1 stays red on a gitignored file only it has). Needs one GA-R5 exchange; no D-number self-assigned, GA-R3 resolves doubt DOWN to OPS. - 2. **THE `provider-bundle-check` ARITY GAP** -- `:137` requires exactly 3 addresses per - `vip`, a dual-family VIP is 6, so under R2 the checker fails EVERY application; `:149`'s - octet extraction also returns the whole string on a v6 literal. R11 ruled three gate - changes and NOT this one. **Now imminent rather than theoretical: the apex holds the - dual-family set the renderer will emit, so the gate cannot validate what is about to be - produced.** Also carried: `EXPECT_PUBLIC_VIP` must STAY 11 while `VIP_COUNT_EXPECT` goes - to 13, because neither vault nor designate has a `public` binding. + 2. **THE `provider-bundle-check` ARITY GAP -- CLOSED 2026-07-28** (see the entry below). + As raised: `:137` required exactly 3 addresses per `vip`, a dual-family VIP is 6, so + under R2 the checker failed EVERY application; `:149`'s octet extraction returned the + whole string on a v6 literal. R11 ruled three gate changes and NOT this one. 3. **voffice1 TRACKS `dc-dc-stage5-preconditions`, not `main`** -- required, since the repo-carried tooling runs where `maas` lives. **Return it to `main` at merge**; a working host left on a retired branch is exactly the Phase-0 defect this session opened by fixing. +- **VIP ARITY GAP CLOSED + R11's THREE RULED GATE CHANGES EXECUTED 2026-07-28.** Operator + direction, exact utterance: **"Fix the arity gap first, then start the renderer"**; the + scope fork (arity alone vs arity plus R11's ruled changes) was put separately and answered + **"Arity gap + R11's three ruled changes (Recommended)"**. OPS under GA-R3 -- the arity fix + makes an ALREADY-RULED end state expressible, so doubt resolves DOWN; no D-number assigned. + Changelog `docs/changelog-20260728-vip-arity-gate.md`. + **THE GATE CAN NOW EXPRESS WHAT R2 AND R11 RULED.** `provider-bundle-check.py` accepts a v4 + triple OR a dual-family sextet, validates the three v6 legs against the per-DC v6 `/64`s + **read from the NetBox apex record** (D-136 option (D) -- not hardcoded; same `(role, kind)` + keying as `dc-plane-ipam.sh` and `dc-plane-apex-import.py`, and the provider leg correctly + takes the DEDICATED GUA VIP `/64`), requires the v6 host part to MIRROR the v4 octet + textually, and **COUPLES `prefer-ipv6` to the arity in BOTH directions** -- which is the + substance, because the measured L3-9 finding is that the merge order keeping `prefer-ipv6` + while dropping the v6 legs is the one that EXITS 0. A dual-family vip with an unreadable + apex now REFUSES at exit 2 rather than passing; a v4-only bundle needs no apex at all. + R11's ruled three, executed in the same pass: band `50-60` -> **`50-99`** (checker + `OCTET_LO/HI` + `lib-net.sh:VIP_OCTET_MAX`), `VIP_COUNT_EXPECT` **11 -> 13**, + `EXPECT_PUBLIC_VIP` **deliberately STAYS 11** (measured: neither vault nor designate carries + a `public` binding), and the NEW invariant that an **hacluster principal with no `vip` FAILS** + -- the ruled hardening, since `cluster_count` is asserted nowhere and a 3->1 rewrite of all + 20 values still produces a byte-identical PASS. + **MEASURED CONSEQUENCE, stated precisely rather than glossed: two sub-checks flip PASS -> + FAIL.** `provider-bundle-check` on the base bundle now FAILS with `hacluster relation but no + vip: designate`, and `pre-flight-checks` CHECK 1 reports `OK=11 (want OK=13)`. **`preflight.sh` + was ALREADY exit 1 before this change** (octavia-pki absent, MAAS unreachable from the + jumphost, P5's 7 findings); measured after, still exit 1 at `3 fatal, 2 warning`. It did NOT + flip preflight pass -> fail and **no deploy path that was open is closed** -- the P5 precedent + above. Both reds are the ruled gate reporting real work owed: R11's vault `.61` / designate + `.62` are RULED-BUT-NOT-BUILT, and per R6 they land BEFORE the HA overlay. + Harness **28/28** (was 15), gauntlet **ALL GREEN (84) ON vcloud** (host named -- the gauntlet + is measured host-dependent), repo-lint 0 fail / 604 files scanned. **THREE HARNESS CASES WERE + RE-POINTED, NOT DELETED** (the standing rule): the fixture base split into a pristine repo + bundle and one carrying designate's ruled `.62`, so the rc=0 cases still assert their own + invariant while NEW case T16 keeps the real tree honest by asserting the pristine bundle DOES + trip invariant 8. **When `.62` lands in `bundle.yaml`, T16 must be re-pointed, not deleted.** + The gate was proven able to BOTH fail and pass (T16 red / T18 green on the same check) -- + the complement invariant this branch established. + **APEX RE-VERIFIED LIVE the same session** (operator question: was the IPv6 actually pushed + last session): `http://10.10.1.10:8000` reports ip-addresses **160**, IPv6 **78**, VIP-described + **156** (78 v4 / 78 v6, 78 dc0 / 78 dc1), ip-ranges **27**, prefixes **139 / 103 IPv6** -- matching + this document's recorded figures exactly. Spot-checked rather than counted: keystone `.50` is + present on all six legs in both DCs with the v6 host part mirroring the octet, and vault `.61` / + designate `.62` are reserved in both families. `ip-ranges` are all v4, which is CORRECT, not a + gap (MAAS's `::1`-`::ffff:ffff` default reservation). **Measured caveat: the NetBox `?site=` + filter silently does NOT filter** -- it returned the full 160 for both sites, so the per-DC + split was re-derived from the addresses themselves. + **LOGGED NOT ACTIONED:** `pre-flight-checks` CHECK 1's awk parse is triple-shaped and will + mis-read a sextet (same class, one gate over -- interlocks with BLOCKER-1); `cluster_count` + coherence is still unasserted; and a NetBox target-drift finding (four `netbox/*.py` tools + document `netbox.baldurkeep.com` in their usage examples, TWO of which -- `ipv6-mark-reserved.py` + and `ipv4-prefixes-import.py` -- carry write paths with NO `SANDBOX_HOSTS` guard, while the + guarded tools expose a deliberate `--yes-write-upstream` override; and + `~/vr1-office1-creds/vr1-netbox.env` points at the v1 reference while `vr1-netbox-sandbox.env` + points at the LIVE apex, i.e. the filenames are inverted). - **THREE PLATFORM BEHAVIOURS GRADUATED to `references/platform-traps.md`** at session close, having been recorded only in this status document (which is consolidated over time, so a durable trap does not belong here alone): MAAS auto-reserves `::1`-`::ffff:ffff` on EVERY diff --git a/docs/changelog-20260728-vip-arity-gate.md b/docs/changelog-20260728-vip-arity-gate.md new file mode 100644 index 0000000..ecd6052 --- /dev/null +++ b/docs/changelog-20260728-vip-arity-gate.md @@ -0,0 +1,159 @@ +# Session changelog 2026-07-28 -- the VIP arity gap + R11's ruled gate changes + +ONE changelog per session (GA-R2/D1). Under blanket approval this is the review +surface: every item states WHAT, WHY (evidence), and HOW TO REVERT. + +**No live mutation in this session.** Everything below is a repo change; the only +live access was READ-ONLY (a NetBox apex poll and a `pre-flight-checks` run). No +logged window was opened because nothing was mutated. + +Status authority for everything here is `docs/CURRENT-STATE.md` (GA-R1). This +file is session-scoped scratch and is NOT citable as status or decision +authority. + +--- + +## 1. `provider-bundle-check.py`: VIP arity, family coupling, and the apex-derived v6 bands + +**What.** The `vip` validator now accepts a v4 TRIPLE (3 addresses) or a +dual-family SEXTET (6), validates the three v6 legs against the per-DC v6 /64s, +requires the v6 host part to MIRROR the v4 octet textually, and couples +`prefer-ipv6` to the arity in both directions. The v6 bands are READ FROM THE +NETBOX APEX RECORD, not hardcoded. The standalone octet-uniqueness block (old +item 6) folded into the same loop. + +**Why.** Measured at `:137`: `if len(parts) != 3` fails EVERY application once R2's +ruled dual-stack lands, and `:149`'s `p.split(".")[-1]` returns the whole address +on a v6 literal, so every v6 leg was trivially "unique" and unchecked. The apex now +holds the dual-family set a renderer will emit (verified live this session: 160 +ip-addresses, 78 IPv6, 156 VIP objects), so the only gate that validates overlays +could not validate what is about to be produced. + +The `prefer-ipv6` coupling is the substance, not a bonus. `prefer-ipv6` makes +HAProxy bind `:::port` in ADDITION to `*:port`; the measured L3-9 finding is that +the overlay merge order which keeps `prefer-ipv6` while dropping the v6 VIP legs +is the one that EXITS 0 -- the dangerous order is the green one. Both directions +now fail. + +**Authority.** The arity gap was carried in `docs/CURRENT-STATE.md` as an open +question, explicitly NOT covered by R11. Operator direction 2026-07-28, exact +utterance: **"Fix the arity gap first, then start the renderer"**, with the scope +fork (arity alone vs arity + R11's three ruled changes) put separately and +answered **"Arity gap + R11's three ruled changes (Recommended)"**. OPS under +GA-R3 -- a script change making an already-RULED end state expressible; doubt +resolves DOWN, no D-number assigned. + +**Prior art followed, not reinvented.** The `(role, kind)` apex keying and the +"provider leg lives in the DEDICATED GUA VIP /64, admin/internal in their plane +/64s" rule are copied from `scripts/dc-plane-ipam.sh:102-131` and +`netbox/dc-plane-apex-import.py`. Both live-API and repo-dump shapes are handled +(nested `scope.slug` vs flat `scope_site`) -- writing against one and matching +zero objects on the other is a trap this repo already hit once. + +**Refuses rather than passes.** A dual-family vip with an unreadable or absent +apex record exits **2**, never 0: "could not look" is never "nothing there". A +v4-only bundle needs no apex at all, so this introduces no new dependency for +the current tree (harness T25/T26 lock both halves). + +**Revert.** `git revert` this commit. The checker returns to triple-only; nothing +live depends on it. + +## 2. R11's three ruled gate changes (D-020 AMENDMENT, RULED 2026-07-27) + +**What.** +- `provider-bundle-check.py`: `OCTET_LO, OCTET_HI` 50-60 -> **50-99**. +- `scripts/lib-net.sh`: `VIP_OCTET_MAX` 60 -> **99**, `VIP_COUNT_EXPECT` 11 -> **13**. +- `provider-bundle-check.py`: NEW invariant 8 -- an hacluster principal with no + `vip` is a FAIL. +- `EXPECT_PUBLIC_VIP` deliberately **STAYS 11**. + +**Why.** R11 ruled vault `.61` and designate `.62` as full dual-family triples and +ruled the gate hardening **IN SCOPE, not deferred**. `cluster_count` is asserted +NOWHERE in `scripts/` or `tests/`, so a 3->1 rewrite of all 20 values produces a +byte-identical PASS -- decorative HA was found by audit rather than by gate. + +`EXPECT_PUBLIC_VIP` stays 11 because -- measured -- NEITHER vault nor designate +carries a `public` binding, so they do not join that count. Bumping both constants +together breaks the gate. The two band constants are SEPARATELY NAMED in two files; +this is a two-file change by construction, and both sites carry a comment saying so. + +**MEASURED CONSEQUENCE, stated precisely.** Two sub-checks flip PASS -> FAIL: +- `provider-bundle-check` on the base bundle now FAILS with + `hacluster relation but no vip: designate`. +- `pre-flight-checks` CHECK 1 now reports `VIP validation OK=11 bad=0 (want OK=13)`. + +**`preflight.sh` was ALREADY exit 1 before this change** (octavia-pki overlay +absent, MAAS unreachable from the jumphost, P5's 7 credential findings). Measured +after: still exit 1, now `3 fatal, 2 warning`. This did NOT flip preflight from +pass to fail and **no deploy path that was open is closed by it** -- the same +precedent this document records for P5. The reds are the ruled gate reporting real +work owed: R11's `.61`/`.62` VIPs are RULED-BUT-NOT-BUILT, and per R6 they land +BEFORE the HA overlay. + +**Revert.** Revert this commit; the constants and the invariant-8 block are +contiguous and independent of item 1. + +## 3. Harness: 15 -> 28 cases, with three cases RE-POINTED rather than deleted + +**What.** `tests/provider-bundle-check/run-tests.sh` gains 13 cases. The fixture +base splits: `pristine.yaml` is the repo bundle untouched; `good.yaml` is that +bundle PLUS designate's ruled `.62` VIP. + +**Why the split.** Invariant 8 legitimately fails the real repo bundle. Left alone, +every rc=0 case (T1/T9/T14) would have been asserting "fails for the reason we +already know" instead of its own invariant. Per the standing rule -- *when +remediating a finding turns the harness red, REPLACE the assertion with the new +invariant and say so, never delete it to go green* -- the real tree's state is +kept honest by a NEW case (T16) that asserts `pristine.yaml` DOES trip it. +**When `.62` lands in `bundle.yaml`, T16 must be re-pointed, not deleted.** + +**Proven able to fail AND to pass** -- the complement this project recorded on +2026-07-27 after building three gates whose every live run failed: +- T16 the pristine bundle FAILS on designate; T18 the same check goes GREEN once + the ruled VIP lands. +- T17 vault FAILS under `dc-ha-scaleup.yaml` -- R6's "VIPs before the overlay" + ordering, made executable. +- T19 a valid dual-family VIP PASSES and is counted as dual; T20/T21 both + directions of the `prefer-ipv6` coupling FAIL; T22 a v6 leg in the node `/64` + instead of the VIP `/64` FAILS; T23 the `::32` hex trap FAILS; T24 a 2-address + vip FAILS; T25 an unreadable apex REFUSES at rc=2; T26 a v4-only bundle needs + no apex; T27/T28 the band is widened to 99 but still bounded. + +T8's assertion string moved `outside 50-60` -> `outside 50-99` (the same +re-pointing rule; the case still guards the band, at its ruled bounds). + +**Evidence.** Harness **28/28** (was 15). Gauntlet **ALL GREEN (84 harnesses) on +vcloud** -- the host is named because the gauntlet is measured HOST-DEPENDENT +(2/81 fail on voffice1 on an identical commit). `repo-lint` 0 fail / 1 standing +warn / 604 files scanned. No harness was added or renamed, so +`tests/HARNESS-MANIFEST` is unchanged and needed no re-record. + +**Revert.** Revert this commit; the harness returns to 15 cases. + +--- + +## Findings LOGGED, NOT ACTIONED (hard rule 1) + +1. **`pre-flight-checks.sh` CHECK 1 is triple-shaped.** Its awk parse assumes three + columns, so it will mis-read a dual-family sextet when those artifacts land. Same + class as the arity gap just fixed, one gate over. Not touched -- it interlocks + with BLOCKER-1 (the ruling-3 VIP extraction makes CHECK 1 see 0 VIPs). +2. **`cluster_count` coherence is still unasserted.** Invariant 8 catches an + hacluster principal with NO vip, which is what R11 ruled. It does NOT check that + `cluster_count` matches the principal's `num_units` -- a 3->1 rewrite of all 20 + values still produces a PASS. Out of ruled scope; worth its own exchange. +3. **NetBox instance/target drift (raised by the operator this session).** Four + `netbox/*.py` tools carry in-file usage examples pointing at + `https://netbox.baldurkeep.com` -- the FROZEN v1 reference. Two of them + (`ipv6-mark-reserved.py`, `ipv4-prefixes-import.py`) have **NO `SANDBOX_HOSTS` + write guard at all** while carrying write paths (`update_status`, + `create_or_report_prefix`, `create_or_report_iprange`). The guarded tools also + expose a deliberate `--yes-write-upstream` escape hatch, so "writes only reach + 10.10.1.10" is a convention with an override, not a hard invariant. +4. **Credential env-file naming is inverted.** `~/vr1-office1-creds/vr1-netbox.env` + points at `netbox.baldurkeep.com` (the read-only v1 reference) while + `vr1-netbox-sandbox.env` points at the LIVE VR1 apex `http://10.10.1.10:8000`. + The filename says the opposite of what each is. +5. **The NetBox `?site=` API filter silently does not filter.** Measured this + session: `/api/ipam/ip-addresses/?site=vr1-dc0` and `?site=vr1-dc1` each returned + the full 160. A per-site apex count taken that way would be wrong and look right. diff --git a/scripts/lib-net.sh b/scripts/lib-net.sh index 1a66055..a8d3545 100644 --- a/scripts/lib-net.sh +++ b/scripts/lib-net.sh @@ -55,8 +55,16 @@ VIP_PREFIX_ADMIN="10.12.8" VIP_PREFIX_INTERNAL="10.12.12" VIP_OCTET_MIN=50 -VIP_OCTET_MAX=60 -VIP_COUNT_EXPECT=11 +# D-020 AMENDMENT / R11 (RULED 2026-07-27): band widened 60 -> 99 and the expected +# count 11 -> 13 to admit vault .61 and designate .62, both RULED-BUT-NOT-BUILT -- +# so pre-flight-checks CHECK 1 reports 11 against an expected 13 until those VIPs +# land in the deploy artifacts. That is the ruled gate reporting real work owed, not +# a regression. NOTE these two constants are SEPARATELY NAMED from +# provider-bundle-check.py's own OCTET_LO/OCTET_HI + EXPECT_PUBLIC_VIP -- a two-file +# change, and EXPECT_PUBLIC_VIP deliberately STAYS 11 (neither vault nor designate +# carries a `public` binding, so they do not join that count). +VIP_OCTET_MAX=99 +VIP_COUNT_EXPECT=13 # D-003 provider FIP allocation pool: a RESERVED iprange on the provider-ext subnet # so neutron owns it without colliding with a MAAS auto-static primary (KI-P3-001). diff --git a/scripts/provider-bundle-check.py b/scripts/provider-bundle-check.py index 2d029d0..1f124d7 100644 --- a/scripts/provider-bundle-check.py +++ b/scripts/provider-bundle-check.py @@ -5,8 +5,12 @@ Asserts ONLY the post-revert (D-052/D-053 + Pattern A) provider invariants on a Charmed-OpenStack bundle: 1. exactly 11 API charms bind public -> provider-public; none remain on provider-vip - 2. every clustered VIP is a triple: provider-public(10.12.4/22) admin(10.12.8/22) - internal(10.12.12/22), all sharing one last octet in 50-60 + 2. every clustered VIP is EITHER a v4 triple -- provider-public(10.12.4/22) + admin(10.12.8/22) internal(10.12.12/22) -- OR a dual-family sextet appending + the three v6 legs (R2, RULED 2026-07-27: dual-stack for both DCs). All legs + share ONE host part, in 50-99 (D-020 amendment / R11, 2026-07-27), and the v6 + host part MIRRORS the v4 octet textually (D-136 ruling, same date). + prefer-ipv6 and the v6 legs must land TOGETHER (see invariant 9). 3. ovn-chassis bridge-interface-mappings carries ALL FOUR chassis MACs, INCLUDING openstack0's -- the Pattern A revert re-adds the openstack0 MAC that D-057 trimmed for the now-dead provider-vip plane. @@ -20,6 +24,15 @@ use-policyd-override true), and the committed zip content matches policies/domain-manager-policy.yaml (DOCFIX-071 drift guard; content compare, not byte compare, so zip mtimes cannot false-fail it) + 8. R11 (D-020 AMENDMENT, 2026-07-27): a principal wired to an hacluster subordinate + MUST carry a vip. Without one it binds a UNIT address and pacemaker has nothing + to fail over to -- decorative HA. `cluster_count` is asserted NOWHERE in this + repo, so a 3->1 rewrite of every value produces a byte-identical PASS; this is + the ruled gate hardening that makes the shape checkable at all. + 9. prefer-ipv6 and dual-family VIP arity are COUPLED. prefer-ipv6 makes HAProxy bind + :::port in ADDITION to *:port, so the two must travel together. Measured (L3-9): + the overlay merge order that keeps prefer-ipv6 while silently dropping the v6 VIP + legs is the one that exits 0 -- the dangerous order is the GREEN one. It is the single deploy-gate: it REPLACED the retired scripts/d057-bundle-check.py (D-060) and now also scripts/review-bundle.py (DOCFIX-070). FAIL -> exit 1. ASCII-only output. @@ -38,8 +51,18 @@ "vr1-dc0": ("10.12.4.0/22", "10.12.8.0/22", "10.12.12.0/22"), "vr1-dc1": ("10.12.64.0/22", "10.12.68.0/22", "10.12.72.0/22"), } -OCTET_LO, OCTET_HI = 50, 60 +# D-020 AMENDMENT / R11 (RULED 2026-07-27): the band widens 50-60 -> 50-99 to admit +# vault .61 and designate .62. EXPECT_PUBLIC_VIP deliberately STAYS 11 -- measured, +# NEITHER vault nor designate carries a `public` binding, so they do not join that +# count and bumping both constants together breaks the gate. +OCTET_LO, OCTET_HI = 50, 99 EXPECT_PUBLIC_VIP = 11 + +# v6 VIP bands are NOT hardcoded: D-136 option (D) (ADOPTED 2026-07-27) makes the +# NetBox apex the source for v6 prefixes. Same record and same (role, kind) keying +# as scripts/dc-plane-ipam.sh:102-131 and netbox/dc-plane-apex-import.py. +APEX_ENV = "APEX_RECORD" +APEX_GLOB = "netbox/draft/vr1-office1-current-*.json" EXPECT_CHASSIS_MACS = { "52:54:00:3d:fd:54", # openstack0 -- re-added by the Pattern A revert (D-057 had trimmed it) "52:54:00:9d:63:77", # openstack1 @@ -84,6 +107,71 @@ out[k] = v return out +def _apex_v6_bands(dc): + """The three v6 VIP-leg /64s for `dc`, read from the NetBox apex record. + + Returns (bands, None) on success or (None, reason) -- a reason is a REFUSAL + (exit 2), never a pass: "could not look" is never "nothing there". + + The PROVIDER leg lives in the DEDICATED GUA VIP /64 (the apex marks it with + "VIP" in the description), NOT the provider node /64; admin and internal use + their own plane /64s. The record's shape differs between the live API and the + repo dumps (live returns nested scope.slug/scope.name, dumps flatten to + scope_site + a bare role string), so both are handled -- writing against one + shape and matching zero objects on the other is a measured trap. + """ + import glob, json, os + rec = os.environ.get(APEX_ENV) + if not rec: + root = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + cand = sorted(glob.glob(os.path.join(root, APEX_GLOB))) + rec = cand[-1] if cand else None + if not rec or not os.path.isfile(rec): + return None, "no readable apex record (looked for %s; set %s)" % (APEX_GLOB, APEX_ENV) + try: + doc = json.load(open(rec, encoding="utf-8")) + except Exception as e: + return None, "apex record %s unreadable: %s" % (rec, e) + found = {} + for p in (doc.get("ipam/prefixes") or []): + pre = str(p.get("prefix", "")) + if ":" not in pre or not pre.endswith("/64"): + continue # /60 and /48 are parent blocks, not planes + sc = p.get("scope") or {} + if (sc.get("slug") or sc.get("name") or p.get("scope_site") or "") != dc: + continue + role = p.get("role") or "" + role = role.get("slug") if isinstance(role, dict) else role + kind = "vip" if "VIP" in str(p.get("description") or "") else "plane" + found[(role, kind)] = pre + want = (("provider", ("provider-public", "vip")), + ("admin", ("metal-admin", "plane")), + ("internal", ("metal-internal", "plane"))) + bands, missing = {}, [] + for leg, key in want: + if key in found: + bands[leg] = ipaddress.ip_network(found[key]) + else: + missing.append("%s/%s" % key) + if missing: + return None, ("apex record %s carries no v6 prefix for %s in %s -- refusing to " + "guess a VIP band" % (rec, ", ".join(missing), dc)) + return bands, None + + +def _host_part(addr): + """The trailing label of an address: the v4 last octet, or the final v6 group. + + The 2026-07-27 ruling mirrors the v4 octet DIGITS into the v6 host part + (keystone .50 -> ...::50), so the two are compared as TEXT and never + numerically -- `printf '%x' 50` is 32, which is exactly the plausible-looking + wrong band that ruling exists to avoid. + """ + ip = ipaddress.ip_address(addr) + s = str(ip) + return s.rsplit(".", 1)[-1] if ip.version == 4 else s.rsplit(":", 1)[-1] + + def _role_of(spec): """Extract control/compute/storage from a machine spec's `constraints: tags=...`.""" m = re.search(r"tags=(\S+)", str((spec or {}).get("constraints", ""))) @@ -128,33 +216,74 @@ else: oks.append("%d charms bind public->provider-public; none on provider-vip" % len(on_public)) - vip_ok = 0 - for n, s in apps.items(): - vip = ((s or {}).get("options", {}) or {}).get("vip") + # -- 2 + 9. VIP legs, arity, family coupling, band, and octet uniqueness. + # A vip is a v4 TRIPLE (3) or, under R2's ruled dual-stack, a dual-family + # SEXTET (6) appending the v6 provider/admin/internal legs. + v6_bands, v6_refusal = None, None + vip_ok = vip_dual = 0 + octet_owner = {} + for n in sorted(apps): + opts = ((apps[n] or {}).get("options") or {}) + vip = opts.get("vip") if not vip: continue + prefer6 = bool(opts.get("prefer-ipv6")) parts = str(vip).split() - if len(parts) != 3: - fails.append("%s vip not a triple: %r" % (n, vip)); continue - prov, adm, intr = parts - try: - okp = ipaddress.ip_address(prov) in PROVIDER - oka = ipaddress.ip_address(adm) in ADMIN - oki = ipaddress.ip_address(intr) in INTERNAL - except ValueError as e: - fails.append("%s bad vip ip: %s" % (n, e)); continue - if not okp: fails.append("%s provider leg %s not in %s" % (n, prov, PROVIDER)); continue - if not oka: fails.append("%s admin leg %s not in %s" % (n, adm, ADMIN)); continue - if not oki: fails.append("%s internal leg %s not in %s" % (n, intr, INTERNAL)); continue - octs = {p.split(".")[-1] for p in parts} - if len(octs) != 1: - fails.append("%s vip octets differ: %r" % (n, vip)); continue - o = int(octs.pop()) + if len(parts) not in (3, 6): + fails.append("%s vip is neither a v4 triple (3) nor a dual-family sextet (6) " + "-- got %d address(es): %r" % (n, len(parts), vip)); continue + dual = (len(parts) == 6) + # prefer-ipv6 binds :::port in ADDITION to *:port, so it and the v6 legs must + # land together. Measured (L3-9): the overlay merge order that keeps + # prefer-ipv6 while dropping the v6 legs is the one that exits 0 -- pacemaker + # would then have no v6 VIP to manage behind a socket that is already bound. + if prefer6 != dual: + fails.append("%s prefer-ipv6=%s but its vip carries %d address(es) -- " + "prefer-ipv6 and the v6 VIP legs must land TOGETHER" + % (n, str(prefer6).lower(), len(parts))); continue + if dual and v6_bands is None and v6_refusal is None: + v6_bands, v6_refusal = _apex_v6_bands(args.dc) + if dual and v6_bands is None: + sys.stderr.write("ERROR: cannot evaluate %s's dual-family vip: %s\n" % (n, v6_refusal)) + return 2 + legs = [("provider", parts[0], PROVIDER), + ("admin", parts[1], ADMIN), + ("internal", parts[2], INTERNAL)] + if dual: + legs += [("provider v6", parts[3], v6_bands["provider"]), + ("admin v6", parts[4], v6_bands["admin"]), + ("internal v6", parts[5], v6_bands["internal"])] + bad = False + for lbl, addr, net in legs: + try: + ip = ipaddress.ip_address(addr) + except ValueError as e: + fails.append("%s bad vip ip: %s" % (n, e)); bad = True; break + if ip.version != net.version: + fails.append("%s %s leg %s is IPv%d, expected IPv%d" + % (n, lbl, addr, ip.version, net.version)); bad = True; break + if ip not in net: + fails.append("%s %s leg %s not in %s" % (n, lbl, addr, net)); bad = True; break + if bad: + continue + hosts = {_host_part(p) for p in parts} + if len(hosts) != 1: + fails.append("%s vip host parts differ across legs -- the v6 host part must " + "MIRROR the v4 octet (2026-07-27 ruling): %r" % (n, vip)); continue + h = hosts.pop() + if not h.isdigit(): + fails.append("%s vip host part %r is not the decimal v4-octet mirror" % (n, h)); continue + o = int(h) if not (OCTET_LO <= o <= OCTET_HI): fails.append("%s vip octet .%d outside %d-%d" % (n, o, OCTET_LO, OCTET_HI)); continue + if h in octet_owner: + fails.append("VIP last octet .%s shared by %s and %s" % (h, octet_owner[h], n)); continue + octet_owner[h] = n vip_ok += 1 + vip_dual += 1 if dual else 0 if vip_ok: - oks.append("%d clustered VIP(s) are provider-public/admin/internal triples, octet 50-60" % vip_ok) + oks.append("%d clustered VIP(s) are provider/admin/internal, octet %d-%d " + "(%d dual-family)" % (vip_ok, OCTET_LO, OCTET_HI, vip_dual)) for n, s in apps.items(): if (s or {}).get("charm") != "ovn-chassis": @@ -199,16 +328,34 @@ else: oks.append("mysql-innodb-cluster num_units=3 (D-062)") - # -- 6. VIP octet uniqueness -- - seen_oct = {} - for n, s in apps.items(): - vip = ((s or {}).get("options", {}) or {}).get("vip") - if not vip: continue - for part in str(vip).split(): - o = part.rsplit(".", 1)[-1] - if o in seen_oct and seen_oct[o] != n: - fails.append("VIP last octet .%s shared by %s and %s" % (o, seen_oct[o], n)) - seen_oct.setdefault(o, n) + # -- 6. VIP octet uniqueness: folded into the family-aware loop above, which + # compares the HOST PART (v4 octet or final v6 group) rather than + # rsplit(".")-ing every token -- on a v6 literal that returned the whole + # address, so every v6 leg was trivially "unique" and unchecked. + + # -- 8. R11 (D-020 AMENDMENT, 2026-07-27): an hacluster principal with no VIP. + # Decorative HA -- the principal binds a UNIT address and pacemaker has + # nothing to fail over to. Nothing in scripts/ or tests/ asserts + # cluster_count, so this shape passed every gate until now. + ha_principals = set() + for r in rels: + if not isinstance(r, list) or len(r) != 2: + continue + sides = [str(x) for x in r] + if not all(x.endswith(":ha") for x in sides): + continue + for i, side in enumerate(sides): + app, peer = side.split(":")[0], sides[1 - i].split(":")[0] + if app in apps and (apps.get(peer) or {}).get("charm") == "hacluster": + ha_principals.add(app) + no_vip = sorted(a for a in ha_principals + if not ((apps.get(a) or {}).get("options") or {}).get("vip")) + if no_vip: + fails.append("hacluster relation but no vip: %s -- clustered principal binds a unit " + "address with nothing to fail over to (R11 / D-020 amendment)" + % ", ".join(no_vip)) + elif ha_principals: + oks.append("%d hacluster principal(s) all carry a VIP (R11)" % len(ha_principals)) # -- 7. DOCFIX-071: keystone policy ships in-bundle, zip content matches source -- import os, zipfile diff --git a/tests/provider-bundle-check/run-tests.sh b/tests/provider-bundle-check/run-tests.sh index 569248e..7ff5727 100644 --- a/tests/provider-bundle-check/run-tests.sh +++ b/tests/provider-bundle-check/run-tests.sh @@ -13,10 +13,35 @@ TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT PASS=0; FAIL=0 -# fixture base: real bundle + the policies dir the DOCFIX-071 check resolves +# fixture base: real bundle + the policies dir the DOCFIX-071 check resolves. +# +# `pristine.yaml` is the repo bundle UNTOUCHED. `good.yaml` is that bundle plus the +# ONE thing R11 (D-020 amendment, RULED 2026-07-27) ruled and that is not built yet: +# designate's VIP at the ruled octet .62. Without it the new hacluster-principal +# check fails the repo bundle, and every rc=0 case below would be asserting "fails +# for the reason we already know" instead of its own invariant. T16 keeps the real +# tree's state honest by asserting that pristine.yaml DOES trip it. mkdir -p "$TMP/policies" -cp "$REPO/bundle.yaml" "$TMP/good.yaml" +cp "$REPO/bundle.yaml" "$TMP/pristine.yaml" cp "$REPO/policies/domain-manager-policy.yaml" "$REPO/policies/overrides.zip" "$TMP/policies/" +python3 - "$TMP/pristine.yaml" "$TMP/good.yaml" <<'PY' +import sys, yaml +b = yaml.safe_load(open(sys.argv[1])) +b["applications"]["designate"].setdefault("options", {})["vip"] = "10.12.4.62 10.12.8.62 10.12.12.62" +yaml.safe_dump(b, open(sys.argv[2], "w")) +PY +# dc1 twin of that ruled VIP, so the dc1 overlay case (T14) validates the same shape. +cat > "$TMP/dc1-designate.yaml" <<'YAML' +applications: + designate: + options: + vip: "10.12.64.62 10.12.68.62 10.12.72.62" +YAML + +# A valid dual-family (R2 dual-stack) VIP for keystone: the three v4 legs plus the +# three v6 legs, host part mirroring the v4 octet per the 2026-07-27 ruling. The +# provider v6 leg is the DEDICATED GUA VIP /64 (f02:11::), not the node /64. +DUAL6="10.12.4.50 10.12.8.50 10.12.12.50 2602:f3e2:f02:11::50 fd50:840e:74e2:220::50 fd50:840e:74e2:221::50" mutate() { # mutate python3 - "$TMP/good.yaml" "$TMP/$1" "$2" <<'PY' @@ -68,9 +93,9 @@ run 1 'DIFFERS from policies' "T7 zip/source drift FAILS" "$TMP/t7.yaml" cp "$REPO/policies/domain-manager-policy.yaml" "$TMP/policies/" # restore -# T8 original invariant still guarded: octavia VIP off-band +# T8 original invariant still guarded: octavia VIP off-band (band widened to 50-99 by R11) mutate t8.yaml 'b["applications"]["octavia"]["options"]["vip"]="10.12.4.233 10.12.8.233 10.12.12.233"' -run 1 'outside 50-60' "T8 off-band VIP octet FAILS" "$TMP/t8.yaml" +run 1 'outside 50-99' "T8 off-band VIP octet FAILS" "$TMP/t8.yaml" # ---- Placement / anti-affinity checks (only fire on a role-separated bundle) ---- # Build a valid role-separated fixture from the base: 3 control / 2 compute / 4 storage, @@ -127,12 +152,75 @@ # T14 overlay merge + DC bands: base + dc1 VIPs, --dc vr1-dc1 PASSES runargs 0 'PASS: Pattern A' "T14 dc1-vips overlay + --dc vr1-dc1 passes" \ - "$TMP/good.yaml" --overlay "$REPO/overlays/vr1-dc1-vips.yaml" --dc vr1-dc1 + "$TMP/good.yaml" --overlay "$REPO/overlays/vr1-dc1-vips.yaml" \ + --overlay "$TMP/dc1-designate.yaml" --dc vr1-dc1 # T15 same overlay but default (dc0) bands -> dc1 VIPs off-band FAIL runargs 1 'not in 10.12.4' "T15 dc1 VIPs vs dc0 bands FAILS" \ "$TMP/good.yaml" --overlay "$REPO/overlays/vr1-dc1-vips.yaml" +# ---- R11 (D-020 amendment): an hacluster principal with no VIP is decorative HA ---- +# T16 the REAL repo bundle trips it. This is the ruled-but-not-built designate VIP; +# when .62 lands in bundle.yaml this case must be re-pointed, NOT deleted. +run 1 'hacluster relation but no vip: designate' \ + "T16 pristine repo bundle FAILS on the R11 designate VIP gap" "$TMP/pristine.yaml" + +# T17 vault is the second R11 gap and it only appears under the HA overlay -- R6 ruled +# the VIPs land BEFORE that overlay, so this is the ordering made executable. +runargs 1 'hacluster relation but no vip: .*vault' "T17 vault under dc-ha-scaleup FAILS (R6 ordering)" \ + "$TMP/good.yaml" --overlay "$REPO/overlays/dc-ha-scaleup.yaml" + +# T18 the check can also PASS, not only fail: give designate its ruled VIP -> green. +run 0 'hacluster principal\(s\) all carry a VIP' \ + "T18 hacluster check goes GREEN once the ruled VIP lands" "$TMP/good.yaml" + +# ---- R2 dual-stack: VIP arity, family coupling, and the apex-derived v6 bands ---- +# T19 a valid dual-family sextet PASSES and is counted as dual +mutate t19.yaml "b['applications']['keystone']['options'].update({'vip':'$DUAL6','prefer-ipv6':True})" +run 0 '1 dual-family' "T19 valid dual-family VIP passes" "$TMP/t19.yaml" + +# T20 prefer-ipv6 kept while the v6 legs are dropped -- the L3-9 merge order that +# exits 0 today. This is the whole reason the coupling exists. +mutate t20.yaml 'b["applications"]["keystone"]["options"]["prefer-ipv6"]=True' +run 1 'prefer-ipv6 and the v6 VIP legs must land TOGETHER' \ + "T20 prefer-ipv6 without v6 legs FAILS" "$TMP/t20.yaml" + +# T21 the inverse: v6 legs present but nothing binds v6 +mutate t21.yaml "b['applications']['keystone']['options']['vip']='$DUAL6'" +run 1 'prefer-ipv6 and the v6 VIP legs must land TOGETHER' \ + "T21 v6 legs without prefer-ipv6 FAILS" "$TMP/t21.yaml" + +# T22 v6 provider leg in the NODE /64 instead of the dedicated GUA VIP /64 +mutate t22.yaml "b['applications']['keystone']['options'].update({'vip':'${DUAL6/f02:11::50/f02:10::50}','prefer-ipv6':True})" +run 1 'provider v6 leg .* not in 2602:f3e2:f02:11::/64' \ + "T22 v6 leg in the wrong /64 FAILS" "$TMP/t22.yaml" + +# T23 v6 host part not mirroring the v4 octet -- `printf '%x' 50` = 32 is the exact +# plausible-looking wrong value the 2026-07-27 ruling exists to prevent. +mutate t23.yaml "b['applications']['keystone']['options'].update({'vip':'${DUAL6/f02:11::50/f02:11::32}','prefer-ipv6':True})" +run 1 'must MIRROR the v4 octet' "T23 v6 host part not mirroring the v4 octet FAILS" "$TMP/t23.yaml" + +# T24 arity that is neither a triple nor a sextet +mutate t24.yaml 'b["applications"]["keystone"]["options"]["vip"]="10.12.4.50 10.12.8.50"' +run 1 'neither a v4 triple .3. nor a dual-family sextet' "T24 2-address vip FAILS" "$TMP/t24.yaml" + +# T25 an unreadable apex REFUSES (rc=2) rather than passing -- "could not look" is +# never "nothing there". Only fires when a dual-family vip is actually present. +APEX_RECORD=/nonexistent/apex.json runargs 2 'cannot evaluate .* no readable apex record' \ + "T25 dual-family + unreadable apex REFUSES (rc=2)" "$TMP/t19.yaml" + +# T26 ...and a v4-only bundle does NOT need the apex at all (no spurious refusal) +APEX_RECORD=/nonexistent/apex.json runargs 0 'PASS: Pattern A' \ + "T26 v4-only bundle needs no apex record" "$TMP/good.yaml" + +# ---- R11 band widen 50-60 -> 50-99 ---- +# T27 octet .62 (designate's ruled octet) is now IN band -- proven by the band message +run 0 'octet 50-99' "T27 R11 band widened to 50-99" "$TMP/good.yaml" + +# T28 ...but the band is still bounded: .100 is still rejected +mutate t28.yaml 'b["applications"]["glance"]["options"]["vip"]="10.12.4.100 10.12.8.100 10.12.12.100"' +run 1 'outside 50-99' "T28 band still bounded (.100 rejected)" "$TMP/t28.yaml" + echo echo "RESULT: PASS=$PASS FAIL=$FAIL" [[ "$FAIL" -eq 0 ]] && { echo "ALL PASS"; exit 0; } || exit 1