Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260728-vip-arity-gate.md
@JANeumatrix JANeumatrix 20 hours ago 17 KB Render-pipeline step 4: the renderer, reproducing a reviewed artifact byte-for-byte
Raw Blame History

Session changelog 2026-07-28 -- the VIP arity gap + R11's ruled gate changes

ONE changelog per session (GA-R2/D1). Under blanket approval this is the review surface: every item states WHAT, WHY (evidence), and HOW TO REVERT.

No live mutation in this session. Everything below is a repo change; the only live access was READ-ONLY (a NetBox apex poll and a pre-flight-checks run). No logged window was opened because nothing was mutated.

Status authority for everything here is docs/CURRENT-STATE.md (GA-R1). This file is session-scoped scratch and is NOT citable as status or decision authority.

1. provider-bundle-check.py: VIP arity, family coupling, and the apex-derived v6 bands

What. The vip validator now accepts a v4 TRIPLE (3 addresses) or a dual-family SEXTET (6), validates the three v6 legs against the per-DC v6 /64s, requires the v6 host part to MIRROR the v4 octet textually, and couples prefer-ipv6 to the arity in both directions. The v6 bands are READ FROM THE NETBOX APEX RECORD, not hardcoded. The standalone octet-uniqueness block (old item 6) folded into the same loop.

Why. Measured at :137: if len(parts) != 3 fails EVERY application once R2's ruled dual-stack lands, and :149's p.split(".")[-1] returns the whole address on a v6 literal, so every v6 leg was trivially "unique" and unchecked. The apex now holds the dual-family set a renderer will emit (verified live this session: 160 ip-addresses, 78 IPv6, 156 VIP objects), so the only gate that validates overlays could not validate what is about to be produced.

The prefer-ipv6 coupling is the substance, not a bonus. prefer-ipv6 makes HAProxy bind :::port in ADDITION to *:port; the measured L3-9 finding is that the overlay merge order which keeps prefer-ipv6 while dropping the v6 VIP legs is the one that EXITS 0 -- the dangerous order is the green one. Both directions now fail.

Authority. The arity gap was carried in docs/CURRENT-STATE.md as an open question, explicitly NOT covered by R11. Operator direction 2026-07-28, exact utterance: "Fix the arity gap first, then start the renderer", with the scope fork (arity alone vs arity + R11's three ruled changes) put separately and answered "Arity gap + R11's three ruled changes (Recommended)". OPS under GA-R3 -- a script change making an already-RULED end state expressible; doubt resolves DOWN, no D-number assigned.

Prior art followed, not reinvented. The (role, kind) apex keying and the "provider leg lives in the DEDICATED GUA VIP /64, admin/internal in their plane /64s" rule are copied from scripts/dc-plane-ipam.sh:102-131 and netbox/dc-plane-apex-import.py. Both live-API and repo-dump shapes are handled (nested scope.slug vs flat scope_site) -- writing against one and matching zero objects on the other is a trap this repo already hit once.

Refuses rather than passes. A dual-family vip with an unreadable or absent apex record exits 2, never 0: "could not look" is never "nothing there". A v4-only bundle needs no apex at all, so this introduces no new dependency for the current tree (harness T25/T26 lock both halves).

Revert. git revert this commit. The checker returns to triple-only; nothing live depends on it.

2. R11's three ruled gate changes (D-020 AMENDMENT, RULED 2026-07-27)

What.

  • provider-bundle-check.py: OCTET_LO, OCTET_HI 50-60 -> 50-99.
  • scripts/lib-net.sh: VIP_OCTET_MAX 60 -> 99, VIP_COUNT_EXPECT 11 -> 13.
  • provider-bundle-check.py: NEW invariant 8 -- an hacluster principal with no vip is a FAIL.
  • EXPECT_PUBLIC_VIP deliberately STAYS 11.

Why. R11 ruled vault .61 and designate .62 as full dual-family triples and ruled the gate hardening IN SCOPE, not deferred. cluster_count is asserted NOWHERE in scripts/ or tests/, so a 3->1 rewrite of all 20 values produces a byte-identical PASS -- decorative HA was found by audit rather than by gate.

EXPECT_PUBLIC_VIP stays 11 because -- measured -- NEITHER vault nor designate carries a public binding, so they do not join that count. Bumping both constants together breaks the gate. The two band constants are SEPARATELY NAMED in two files; this is a two-file change by construction, and both sites carry a comment saying so.

MEASURED CONSEQUENCE, stated precisely. Two sub-checks flip PASS -> FAIL:

  • provider-bundle-check on the base bundle now FAILS with hacluster relation but no vip: designate.
  • pre-flight-checks CHECK 1 now reports VIP validation OK=11 bad=0 (want OK=13).

preflight.sh was ALREADY exit 1 before this change (octavia-pki overlay absent, MAAS unreachable from the jumphost, P5's 7 credential findings). Measured after: still exit 1, now 3 fatal, 2 warning. This did NOT flip preflight from pass to fail and no deploy path that was open is closed by it -- the same precedent this document records for P5. The reds are the ruled gate reporting real work owed: R11's .61/.62 VIPs are RULED-BUT-NOT-BUILT, and per R6 they land BEFORE the HA overlay.

Revert. Revert this commit; the constants and the invariant-8 block are contiguous and independent of item 1.

3. Harness: 15 -> 30 cases, with three cases RE-POINTED rather than deleted

What. tests/provider-bundle-check/run-tests.sh gains 15 cases. The fixture base splits: pristine.yaml is the repo bundle untouched; good.yaml is that bundle PLUS designate's ruled .62 VIP.

Why the split. Invariant 8 legitimately fails the real repo bundle. Left alone, every rc=0 case (T1/T9/T14) would have been asserting "fails for the reason we already know" instead of its own invariant. Per the standing rule -- when remediating a finding turns the harness red, REPLACE the assertion with the new invariant and say so, never delete it to go green -- the real tree's state is kept honest by a NEW case (T16) that asserts pristine.yaml DOES trip it. When .62 lands in bundle.yaml, T16 must be re-pointed, not deleted.

Proven able to fail AND to pass -- the complement this project recorded on 2026-07-27 after building three gates whose every live run failed:

  • T16 the pristine bundle FAILS on designate; T18 the same check goes GREEN once the ruled VIP lands.
  • T17 vault FAILS under dc-ha-scaleup.yaml -- R6's "VIPs before the overlay" ordering, made executable.
  • T19 a valid dual-family VIP PASSES and is counted as dual; T20/T21 both directions of the prefer-ipv6 coupling FAIL; T22 a v6 leg in the node /64 instead of the VIP /64 FAILS; T23 the ::32 hex trap FAILS; T24 a 2-address vip FAILS; T25 an unreadable apex REFUSES at rc=2; T26 a v4-only bundle needs no apex; T27/T28 the band is widened to 99 but still bounded.

T29/T30 exercise the dc1 arm of the apex lookup, which T19-T23 never reached (they all run at the default --dc vr1-dc0): T29 resolves dc1's f03:11::/64 / :320::/64 / :321::/64 and passes, T30 proves a dc0 v6 leg under --dc vr1-dc1 FAILS, so a cross-DC copy-paste of a rendered overlay cannot pass. The renderer's first real output is per-DC overlays, and a gate observed in only one DC mode is the same untrustworthiness T16/T18 exist to rule out.

T8's assertion string moved outside 50-60 -> outside 50-99 (the same re-pointing rule; the case still guards the band, at its ruled bounds).

Evidence. Harness 30/30 (was 15). Gauntlet ALL GREEN (84 harnesses) on vcloud -- the host is named because the gauntlet is measured HOST-DEPENDENT (2/81 fail on voffice1 on an identical commit). repo-lint 0 fail / 1 standing warn / 605 files scanned. No harness was added or renamed, so tests/HARNESS-MANIFEST is unchanged and needed no re-record.

Revert. Revert this commit; the harness returns to 15 cases.

Findings LOGGED, NOT ACTIONED (hard rule 1)

  1. pre-flight-checks.sh CHECK 1 is triple-shaped. Its awk parse assumes three columns, so it will mis-read a dual-family sextet when those artifacts land. Same class as the arity gap just fixed, one gate over. Not touched -- it interlocks with BLOCKER-1 (the ruling-3 VIP extraction makes CHECK 1 see 0 VIPs).
  2. cluster_count coherence is still unasserted. Invariant 8 catches an hacluster principal with NO vip, which is what R11 ruled. It does NOT check that cluster_count matches the principal's num_units -- a 3->1 rewrite of all 20 values still produces a PASS. Out of ruled scope; worth its own exchange.
  3. NetBox instance/target drift (raised by the operator this session). Four netbox/*.py tools carry in-file usage examples pointing at https://netbox.baldurkeep.com -- the FROZEN v1 reference. Two of them (ipv6-mark-reserved.py, ipv4-prefixes-import.py) have NO SANDBOX_HOSTS write guard at all while carrying write paths (update_status, create_or_report_prefix, create_or_report_iprange). The guarded tools also expose a deliberate --yes-write-upstream escape hatch, so "writes only reach 10.10.1.10" is a convention with an override, not a hard invariant.
  4. Credential env-file naming is inverted. ~/vr1-office1-creds/vr1-netbox.env points at netbox.baldurkeep.com (the read-only v1 reference) while vr1-netbox-sandbox.env points at the LIVE VR1 apex http://10.10.1.10:8000. The filename says the opposite of what each is.
  5. The NetBox ?site= API filter silently does not filter. Measured this session: /api/ipam/ip-addresses/?site=vr1-dc0 and ?site=vr1-dc1 each returned the full 160. A per-site apex count taken that way would be wrong and look right.

2026-07-29 -- render-pipeline STEP 4: the renderer

Same branch, continued. Operator direction: "You can order the steps in whatever order you want. Deploy agents to help if needed."

4. Ordering decision, and why it INVERTED after measurement

I first proposed doing ruling-3's VIP extraction before the renderer, then reversed it once three read-only agents reported. Recording the reversal rather than the conclusion, because the reason is the reusable part:

  • The byte-for-byte reproduction WINDOW IS STILL OPEN and closes on its own. overlays/vr1-dc1-vips.yaml is still byte-identical to tests/render-baseline/fixtures/vr1-dc1-vips.v4-11app.yaml (3f93ecb3...) -- verified this session. The reconciliation (11->13 apps, dual-family) destroys it.
  • The extraction's blast radius is FAR larger than ruling 3's "2 commits". Measured: it breaks a DEPLOY GATE (runbooks/phase-01-bundle-deploy.md:170-176, where juju deploy runs only on 11/11/0), the Octavia SAN derivation (:371), preflight in two places, and 8 of 30 harness cases.
  • So the extraction is better done BY the renderer, once the renderer is proven -- which makes dc0's overlay a validated output rather than a hand edit.

5. scripts/render-dc-overlays.py -- two stages, split where F2 froze the interface

What. derive (apex + lib-net.sh -> a per-DC VALUES file) and render (VALUES -> overlay TEXT). Plus check, which renders and byte-compares.

Why split. Forward item F2 froze the interface: "the renderer is a deterministic command-line tool whose inputs are the per-DC values file and whose outputs are committed, reviewable artifacts. Nothing about automating it should require changing the renderer." render is therefore PURE -- no apex, no network, no clock -- so it is offline-testable and a future CI job only re-runs derive.

Why a text emitter, not yaml.dump. The target is byte-compared. A round-tripper destroys the comment header, normalises vip: "a b c" to a plain scalar, and imposes its own key order -- all three are load-bearing. PyYAML reads the values file and never writes an overlay.

Two things that are INPUT, not output, and would otherwise be invented:

  • The 12-line comment header cites one-off measurements and a design-doc fork; none of it is computable from the apex. derive --from-overlay lifts it VERBATIM from the artifact being reproduced, so a reproduction run is honest about which bytes the tool actually generates.
  • Emission order is ASCENDING OCTET, not alphabetical (keystone .50 before barbican .51). Sorting by name reproduces nothing.

The ruled app->octet map is READ, not restated -- parsed with ast out of netbox/dc-plane-apex-import.py's APP_OCTET, the tool that wrote those values into the apex. A second hand-maintained copy is a second thing to drift.

Revert. Delete scripts/render-dc-overlays.py, tests/render-dc-overlays/, render/values/, and revert the render-baseline T6 hunk + the manifest line. Nothing else consumes any of it; no committed artifact was regenerated by it yet.

6. RESULT: byte-for-byte reproduction achieved, and PROVEN ABLE TO FAIL

render output hashes to 3f93ecb3205cea8b303f5b2cbbbde343a2762156575277ca84684369da8bf896 -- identical to both the committed overlay (1634 bytes) and the frozen fixture.

A byte-compare only ever observed PASSING is as untrustworthy as a gate only ever observed failing, so three faults were seeded and all three fail it: a wrong octet, a dropped header, and dc0 prefixes against the dc1 artifact. render-baseline's new T6 was separately proven by making the renderer sort alphabetically -- hash changed, case went red.

7. THE ARITY WORK PAID FOR ITSELF WITHIN THE HOUR -- a real renderer bug, caught by the gate

The renderer's FIRST dual-family output was malformed. The values file stores each v6 /64 base with trailing colons stripped (2602:f3e2:f03:11), and the emitter joined with a SINGLE colon -- yielding 2602:f3e2:f03:11:50. That is not a valid address, and it still looks like one.

provider-bundle-check.py -- fixed yesterday for exactly this class -- rejected all 13 applications with bad vip ip. Fixed to ::%d and locked by harness cases T13/T13b, which assert both that the correct form is present AND that no single-colon leg is emitted. This is the acceptance criterion the whole arity fix existed to provide, working on its first real use.

8. End-to-end chain now demonstrated

apex -> values -> renderer -> overlay -> gate. The dual-family dc1 render with R11's ruled-but-unbuilt apps included produces 13 clustered VIPs, all dual-family, and provider-bundle-check --dc vr1-dc1 reports PASS including the new hacluster invariant (12 principals, all carrying a VIP).

Evidence. New harness tests/render-dc-overlays 17/17; render-baseline 10/10 (was 9, T6 added -- ADDED per its README, existing assertions NOT repointed). Gauntlet ALL GREEN (85 harnesses) on vcloud (was 84); tests/HARNESS-MANIFEST re-recorded deliberately after the drift gate caught the addition -- which is the gate working. repo-lint 0 fail / 608 files scanned.

Findings from the three read-only agents -- LOGGED, NOT ACTIONED (hard rule 1)

  1. F3's MAC sourcing claim is FALSIFIED. docs/dc-dc-deployment-workflow.md:361 says ovn-chassis MACs "stay sourced from scripts/lib-hosts.sh". Measured: lib-hosts carries only HOST_BOOT_MAC (the PXE NIC, ...:NN:01); ovn-chassis needs the provider-public NIC (...:NN:02), which is not in lib-hosts at all. The real source is opentofu/vr1-dcN-substrate/main.tf, macs[1] of each compute node -- and it exists for BOTH DCs. This also narrows CURRENT-STATE's claim that dc0's MACs "cannot be derived": not from a PATTERN, but they are pinned literals in tofu and are perfectly sourceable. F3's conclusion (MACs are renderer output, not apex input) still stands; only its sourcing sentence is wrong.
  2. scripts/dc-plane-ipam.sh:120 has a latent scope-matcher bug. It matches scope.name or scope_site but NOT scope.slug, so it would match ZERO prefixes against a live-shaped record. Latent only because it defaults to a repo dump. The other two tools use the three-way matcher; this one is the odd one out.
  3. pre-flight-checks.sh CHECK 1 is triple-only AND dc0-only. It hardcodes if(n!=3) (:86), reads bundle.yaml as RAW TEXT (never a merged input), and never calls lib_net_select_dc -- and lib-net.sh deliberately UNSETS VIP_PREFIX_* for dc1, so CHECK 1 cannot evaluate dc1 at all. The renderer's dual-family output will pass provider-bundle-check and FAIL preflight P4.
  4. CURRENT-STATE is stale on dc1's FIP pool -- an early paragraph still says it "needs a ruling"; a later entry records it RULED and applied, and lib-net.sh carries the values. Same in-document supersession class this doc keeps finding.
  5. render-baseline T3 compares COUNTS, not SETS. A deleted pinned fixture plus an added unpinned one keeps the count equal; a # comment line in SHA256SUMS inflates the pin count and masks one unpinned file; and -maxdepth 1 cannot see a subdirectory. It catches the fault it was seeded against; a set-difference would close all three.
  6. The B1/B5 comment tokens are defined in bundle.yaml's own header (:11-13, :19-24). Ten of the eleven inline VIP comments are the bare token # B1. Migrating them into an overlay STRANDS them from their definitions -- appendix-A:652-661 records the repo's rule for exactly this identifier class. This must be handled when the extraction lands, not discovered then.