Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260728-vip-arity-gate.md

Session changelog 2026-07-28 -- the VIP arity gap + R11's ruled gate changes

ONE changelog per session (GA-R2/D1). Under blanket approval this is the review surface: every item states WHAT, WHY (evidence), and HOW TO REVERT.

No live mutation in this session. Everything below is a repo change; the only live access was READ-ONLY (a NetBox apex poll and a pre-flight-checks run). No logged window was opened because nothing was mutated.

Status authority for everything here is docs/CURRENT-STATE.md (GA-R1). This file is session-scoped scratch and is NOT citable as status or decision authority.


1. provider-bundle-check.py: VIP arity, family coupling, and the apex-derived v6 bands

What. The vip validator now accepts a v4 TRIPLE (3 addresses) or a dual-family SEXTET (6), validates the three v6 legs against the per-DC v6 /64s, requires the v6 host part to MIRROR the v4 octet textually, and couples prefer-ipv6 to the arity in both directions. The v6 bands are READ FROM THE NETBOX APEX RECORD, not hardcoded. The standalone octet-uniqueness block (old item 6) folded into the same loop.

Why. Measured at :137: if len(parts) != 3 fails EVERY application once R2's ruled dual-stack lands, and :149's p.split(".")[-1] returns the whole address on a v6 literal, so every v6 leg was trivially "unique" and unchecked. The apex now holds the dual-family set a renderer will emit (verified live this session: 160 ip-addresses, 78 IPv6, 156 VIP objects), so the only gate that validates overlays could not validate what is about to be produced.

The prefer-ipv6 coupling is the substance, not a bonus. prefer-ipv6 makes HAProxy bind :::port in ADDITION to *:port; the measured L3-9 finding is that the overlay merge order which keeps prefer-ipv6 while dropping the v6 VIP legs is the one that EXITS 0 -- the dangerous order is the green one. Both directions now fail.

Authority. The arity gap was carried in docs/CURRENT-STATE.md as an open question, explicitly NOT covered by R11. Operator direction 2026-07-28, exact utterance: "Fix the arity gap first, then start the renderer", with the scope fork (arity alone vs arity + R11's three ruled changes) put separately and answered "Arity gap + R11's three ruled changes (Recommended)". OPS under GA-R3 -- a script change making an already-RULED end state expressible; doubt resolves DOWN, no D-number assigned.

Prior art followed, not reinvented. The (role, kind) apex keying and the "provider leg lives in the DEDICATED GUA VIP /64, admin/internal in their plane /64s" rule are copied from scripts/dc-plane-ipam.sh:102-131 and netbox/dc-plane-apex-import.py. Both live-API and repo-dump shapes are handled (nested scope.slug vs flat scope_site) -- writing against one and matching zero objects on the other is a trap this repo already hit once.

Refuses rather than passes. A dual-family vip with an unreadable or absent apex record exits 2, never 0: "could not look" is never "nothing there". A v4-only bundle needs no apex at all, so this introduces no new dependency for the current tree (harness T25/T26 lock both halves).

Revert. git revert this commit. The checker returns to triple-only; nothing live depends on it.

2. R11's three ruled gate changes (D-020 AMENDMENT, RULED 2026-07-27)

What.

  • provider-bundle-check.py: OCTET_LO, OCTET_HI 50-60 -> 50-99.
  • scripts/lib-net.sh: VIP_OCTET_MAX 60 -> 99, VIP_COUNT_EXPECT 11 -> 13.
  • provider-bundle-check.py: NEW invariant 8 -- an hacluster principal with no vip is a FAIL.
  • EXPECT_PUBLIC_VIP deliberately STAYS 11.

Why. R11 ruled vault .61 and designate .62 as full dual-family triples and ruled the gate hardening IN SCOPE, not deferred. cluster_count is asserted NOWHERE in scripts/ or tests/, so a 3->1 rewrite of all 20 values produces a byte-identical PASS -- decorative HA was found by audit rather than by gate.

EXPECT_PUBLIC_VIP stays 11 because -- measured -- NEITHER vault nor designate carries a public binding, so they do not join that count. Bumping both constants together breaks the gate. The two band constants are SEPARATELY NAMED in two files; this is a two-file change by construction, and both sites carry a comment saying so.

MEASURED CONSEQUENCE, stated precisely. Two sub-checks flip PASS -> FAIL:

  • provider-bundle-check on the base bundle now FAILS with hacluster relation but no vip: designate.
  • pre-flight-checks CHECK 1 now reports VIP validation OK=11 bad=0 (want OK=13).

preflight.sh was ALREADY exit 1 before this change (octavia-pki overlay absent, MAAS unreachable from the jumphost, P5's 7 credential findings). Measured after: still exit 1, now 3 fatal, 2 warning. This did NOT flip preflight from pass to fail and no deploy path that was open is closed by it -- the same precedent this document records for P5. The reds are the ruled gate reporting real work owed: R11's .61/.62 VIPs are RULED-BUT-NOT-BUILT, and per R6 they land BEFORE the HA overlay.

Revert. Revert this commit; the constants and the invariant-8 block are contiguous and independent of item 1.

3. Harness: 15 -> 30 cases, with three cases RE-POINTED rather than deleted

What. tests/provider-bundle-check/run-tests.sh gains 15 cases. The fixture base splits: pristine.yaml is the repo bundle untouched; good.yaml is that bundle PLUS designate's ruled .62 VIP.

Why the split. Invariant 8 legitimately fails the real repo bundle. Left alone, every rc=0 case (T1/T9/T14) would have been asserting "fails for the reason we already know" instead of its own invariant. Per the standing rule -- when remediating a finding turns the harness red, REPLACE the assertion with the new invariant and say so, never delete it to go green -- the real tree's state is kept honest by a NEW case (T16) that asserts pristine.yaml DOES trip it. When .62 lands in bundle.yaml, T16 must be re-pointed, not deleted.

Proven able to fail AND to pass -- the complement this project recorded on 2026-07-27 after building three gates whose every live run failed:

  • T16 the pristine bundle FAILS on designate; T18 the same check goes GREEN once the ruled VIP lands.
  • T17 vault FAILS under dc-ha-scaleup.yaml -- R6's "VIPs before the overlay" ordering, made executable.
  • T19 a valid dual-family VIP PASSES and is counted as dual; T20/T21 both directions of the prefer-ipv6 coupling FAIL; T22 a v6 leg in the node /64 instead of the VIP /64 FAILS; T23 the ::32 hex trap FAILS; T24 a 2-address vip FAILS; T25 an unreadable apex REFUSES at rc=2; T26 a v4-only bundle needs no apex; T27/T28 the band is widened to 99 but still bounded.

T29/T30 exercise the dc1 arm of the apex lookup, which T19-T23 never reached (they all run at the default --dc vr1-dc0): T29 resolves dc1's f03:11::/64 / :320::/64 / :321::/64 and passes, T30 proves a dc0 v6 leg under --dc vr1-dc1 FAILS, so a cross-DC copy-paste of a rendered overlay cannot pass. The renderer's first real output is per-DC overlays, and a gate observed in only one DC mode is the same untrustworthiness T16/T18 exist to rule out.

T8's assertion string moved outside 50-60 -> outside 50-99 (the same re-pointing rule; the case still guards the band, at its ruled bounds).

Evidence. Harness 30/30 (was 15). Gauntlet ALL GREEN (84 harnesses) on vcloud -- the host is named because the gauntlet is measured HOST-DEPENDENT (2/81 fail on voffice1 on an identical commit). repo-lint 0 fail / 1 standing warn / 605 files scanned. No harness was added or renamed, so tests/HARNESS-MANIFEST is unchanged and needed no re-record.

Revert. Revert this commit; the harness returns to 15 cases.


Findings LOGGED, NOT ACTIONED (hard rule 1)

  1. pre-flight-checks.sh CHECK 1 is triple-shaped. Its awk parse assumes three columns, so it will mis-read a dual-family sextet when those artifacts land. Same class as the arity gap just fixed, one gate over. Not touched -- it interlocks with BLOCKER-1 (the ruling-3 VIP extraction makes CHECK 1 see 0 VIPs).
  2. cluster_count coherence is still unasserted. Invariant 8 catches an hacluster principal with NO vip, which is what R11 ruled. It does NOT check that cluster_count matches the principal's num_units -- a 3->1 rewrite of all 20 values still produces a PASS. Out of ruled scope; worth its own exchange.
  3. NetBox instance/target drift (raised by the operator this session). Four netbox/*.py tools carry in-file usage examples pointing at https://netbox.baldurkeep.com -- the FROZEN v1 reference. Two of them (ipv6-mark-reserved.py, ipv4-prefixes-import.py) have NO SANDBOX_HOSTS write guard at all while carrying write paths (update_status, create_or_report_prefix, create_or_report_iprange). The guarded tools also expose a deliberate --yes-write-upstream escape hatch, so "writes only reach 10.10.1.10" is a convention with an override, not a hard invariant.
  4. Credential env-file naming is inverted. ~/vr1-office1-creds/vr1-netbox.env points at netbox.baldurkeep.com (the read-only v1 reference) while vr1-netbox-sandbox.env points at the LIVE VR1 apex http://10.10.1.10:8000. The filename says the opposite of what each is.
  5. The NetBox ?site= API filter silently does not filter. Measured this session: /api/ipam/ip-addresses/?site=vr1-dc0 and ?site=vr1-dc1 each returned the full 160. A per-site apex count taken that way would be wrong and look right.

2026-07-29 -- render-pipeline STEP 4: the renderer

Same branch, continued. Operator direction: "You can order the steps in whatever order you want. Deploy agents to help if needed."

4. Ordering decision, and why it INVERTED after measurement

I first proposed doing ruling-3's VIP extraction before the renderer, then reversed it once three read-only agents reported. Recording the reversal rather than the conclusion, because the reason is the reusable part:

  • The byte-for-byte reproduction WINDOW IS STILL OPEN and closes on its own. overlays/vr1-dc1-vips.yaml is still byte-identical to tests/render-baseline/fixtures/vr1-dc1-vips.v4-11app.yaml (3f93ecb3...) -- verified this session. The reconciliation (11->13 apps, dual-family) destroys it.
  • The extraction's blast radius is FAR larger than ruling 3's "2 commits". Measured: it breaks a DEPLOY GATE (runbooks/phase-01-bundle-deploy.md:170-176, where juju deploy runs only on 11/11/0), the Octavia SAN derivation (:371), preflight in two places, and 8 of 30 harness cases.
  • So the extraction is better done BY the renderer, once the renderer is proven -- which makes dc0's overlay a validated output rather than a hand edit.

5. scripts/render-dc-overlays.py -- two stages, split where F2 froze the interface

What. derive (apex + lib-net.sh -> a per-DC VALUES file) and render (VALUES -> overlay TEXT). Plus check, which renders and byte-compares.

Why split. Forward item F2 froze the interface: "the renderer is a deterministic command-line tool whose inputs are the per-DC values file and whose outputs are committed, reviewable artifacts. Nothing about automating it should require changing the renderer." render is therefore PURE -- no apex, no network, no clock -- so it is offline-testable and a future CI job only re-runs derive.

Why a text emitter, not yaml.dump. The target is byte-compared. A round-tripper destroys the comment header, normalises vip: "a b c" to a plain scalar, and imposes its own key order -- all three are load-bearing. PyYAML reads the values file and never writes an overlay.

Two things that are INPUT, not output, and would otherwise be invented:

  • The 12-line comment header cites one-off measurements and a design-doc fork; none of it is computable from the apex. derive --from-overlay lifts it VERBATIM from the artifact being reproduced, so a reproduction run is honest about which bytes the tool actually generates.
  • Emission order is ASCENDING OCTET, not alphabetical (keystone .50 before barbican .51). Sorting by name reproduces nothing.

The ruled app->octet map is READ, not restated -- parsed with ast out of netbox/dc-plane-apex-import.py's APP_OCTET, the tool that wrote those values into the apex. A second hand-maintained copy is a second thing to drift.

Revert. Delete scripts/render-dc-overlays.py, tests/render-dc-overlays/, render/values/, and revert the render-baseline T6 hunk + the manifest line. Nothing else consumes any of it; no committed artifact was regenerated by it yet.

6. RESULT: byte-for-byte reproduction achieved, and PROVEN ABLE TO FAIL

render output hashes to 3f93ecb3205cea8b303f5b2cbbbde343a2762156575277ca84684369da8bf896 -- identical to both the committed overlay (1634 bytes) and the frozen fixture.

A byte-compare only ever observed PASSING is as untrustworthy as a gate only ever observed failing, so three faults were seeded and all three fail it: a wrong octet, a dropped header, and dc0 prefixes against the dc1 artifact. render-baseline's new T6 was separately proven by making the renderer sort alphabetically -- hash changed, case went red.

7. THE ARITY WORK PAID FOR ITSELF WITHIN THE HOUR -- a real renderer bug, caught by the gate

The renderer's FIRST dual-family output was malformed. The values file stores each v6 /64 base with trailing colons stripped (2602:f3e2:f03:11), and the emitter joined with a SINGLE colon -- yielding 2602:f3e2:f03:11:50. That is not a valid address, and it still looks like one.

provider-bundle-check.py -- fixed yesterday for exactly this class -- rejected all 13 applications with bad vip ip. Fixed to ::%d and locked by harness cases T13/T13b, which assert both that the correct form is present AND that no single-colon leg is emitted. This is the acceptance criterion the whole arity fix existed to provide, working on its first real use.

8. End-to-end chain now demonstrated

apex -> values -> renderer -> overlay -> gate. The dual-family dc1 render with R11's ruled-but-unbuilt apps included produces 13 clustered VIPs, all dual-family, and provider-bundle-check --dc vr1-dc1 reports PASS including the new hacluster invariant (12 principals, all carrying a VIP).

Evidence. New harness tests/render-dc-overlays 17/17; render-baseline 10/10 (was 9, T6 added -- ADDED per its README, existing assertions NOT repointed). Gauntlet ALL GREEN (85 harnesses) on vcloud (was 84); tests/HARNESS-MANIFEST re-recorded deliberately after the drift gate caught the addition -- which is the gate working. repo-lint 0 fail / 608 files scanned.


Findings from the three read-only agents -- LOGGED, NOT ACTIONED (hard rule 1)

  1. F3's MAC sourcing claim is FALSIFIED. docs/dc-dc-deployment-workflow.md:361 says ovn-chassis MACs "stay sourced from scripts/lib-hosts.sh". Measured: lib-hosts carries only HOST_BOOT_MAC (the PXE NIC, ...:NN:01); ovn-chassis needs the provider-public NIC (...:NN:02), which is not in lib-hosts at all. The real source is opentofu/vr1-dcN-substrate/main.tf, macs[1] of each compute node -- and it exists for BOTH DCs. This also narrows CURRENT-STATE's claim that dc0's MACs "cannot be derived": not from a PATTERN, but they are pinned literals in tofu and are perfectly sourceable. F3's conclusion (MACs are renderer output, not apex input) still stands; only its sourcing sentence is wrong.
  2. scripts/dc-plane-ipam.sh:120 has a latent scope-matcher bug. It matches scope.name or scope_site but NOT scope.slug, so it would match ZERO prefixes against a live-shaped record. Latent only because it defaults to a repo dump. The other two tools use the three-way matcher; this one is the odd one out.
  3. pre-flight-checks.sh CHECK 1 is triple-only AND dc0-only. It hardcodes if(n!=3) (:86), reads bundle.yaml as RAW TEXT (never a merged input), and never calls lib_net_select_dc -- and lib-net.sh deliberately UNSETS VIP_PREFIX_* for dc1, so CHECK 1 cannot evaluate dc1 at all. The renderer's dual-family output will pass provider-bundle-check and FAIL preflight P4.
  4. CURRENT-STATE is stale on dc1's FIP pool -- an early paragraph still says it "needs a ruling"; a later entry records it RULED and applied, and lib-net.sh carries the values. Same in-document supersession class this doc keeps finding.
  5. render-baseline T3 compares COUNTS, not SETS. A deleted pinned fixture plus an added unpinned one keeps the count equal; a # comment line in SHA256SUMS inflates the pin count and masks one unpinned file; and -maxdepth 1 cannot see a subdirectory. It catches the fault it was seeded against; a set-difference would close all three.
  6. The B1/B5 comment tokens are defined in bundle.yaml's own header (:11-13, :19-24). Ten of the eleven inline VIP comments are the bare token # B1. Migrating them into an overlay STRANDS them from their definitions -- appendix-A:652-661 records the repo's rule for exactly this identifier class. This must be handled when the extraction lands, not discovered then.

2026-07-29 -- RULING 3, COMMIT 1: the neutral VIP MOVE (+ blast-radius repair)

Operator: "Approved, continue". This is commit 1 of the two ruling 3 (2026-07-25) prescribes. The dual-stack ADD is commit 2 and is deliberately NOT in here.

9. bundle.yaml is now VIP-FREE; overlays/vr1-dc0-vips.yaml is GENERATED

What. All 11 inline vip: lines removed from bundle.yaml; placement's then-empty options: block dropped rather than left parsing as null. The values moved to overlays/vr1-dc0-vips.yaml, rendered by scripts/render-dc-overlays.py from render/values/vr1-dc0-vips.yaml -- the renderer's first real output.

The 11 inline comments were carried, not dropped. Ten are the bare token # B1 and three carry real operational content (keystone's catalog-endpoint note, ceph-radosgw's un-defer note, openstack-dashboard's ALLOWED_HOSTS warning). They were extracted VERBATIM from bundle.yaml by a parser, not retyped, and the renderer grew optional per-app comment support to carry them. An app with no comment emits none, so the dc1 artifact still reproduces byte-for-byte (re-verified: 1634 bytes, unchanged).

The B1/B5 tokens stay DEFINED in bundle.yaml's header, where they always were; the overlay header names that explicitly so the references do not dangle (appendix-A:652-661 is the repo's rule for this identifier class).

PROVEN NEUTRAL by the ruling's own named check. provider-bundle-check on the pre-extraction bundle and on bundle.yaml --overlay overlays/vr1-dc0-vips.yaml --dc vr1-dc0 produce line-for-line identical output, including the same known hacluster relation but no vip: designate failure. Separately, a structural diff confirmed only vip changed: applications 56 -> 56, relations 108 -> 108, every non-VIP option and every non-option key byte-identical.

Revert. git revert; the values file and overlay are additive and the bundle edit is a pure line removal.

10. Blast-radius repair -- the part that made this bigger than "2 commits"

Every item below would have been BROKEN by the move. All were found by the read-only agent sweep BEFORE the edit, not after.

  • scripts/preflight.sh P2 validated the BARE base, which post-move reports 11 phantom "no vip" failures. Now validates the MERGED dc0 input; the overlay is passed only when present so a tree without it still runs.
  • scripts/pre-flight-checks.sh CHECK 1 read bundle.yaml as raw text and would have seen ZERO VIPs -> hard-fail. Now reads the merged source. This is BLOCKER-1, fixed the ruled way (point it at the merged bundle) and explicitly NOT by bumping the expected count. A measured trap on the way: the file sets IFS=$'\n\t', so a space-joined "$VIP_SRC" reached grep as ONE filename, grep exited 2, pipefail propagated and the gate died silently mid-check. Fixed with a bash ARRAY.
  • runbooks/phase-01-bundle-deploy.md -- the real hazard. Its RUN block gates juju deploy on TOT/HI/LO == 11/11/0; post-move that reads 0/0/0 and aborts the deploy. Both guard blocks now span base + overlay. A second bug had to be fixed in the same edit: grep -c over TWO files prints one count PER FILE, so the bare $( ) would have captured a two-line string and every numeric test would have failed anyway -- the counts are now awk-summed. Verified end to end: the gate reads 11/11/0 -> DEPLOY. The deploy line itself now carries --overlay overlays/vr1-dc0-vips.yaml.
  • phase-01's Octavia SAN derivation (DOCFIX-067) read bundle['applications']['octavia']['options']['vip'], which post-move raises KeyError -- swallowed by || true into an EMPTY VIP, then a "implausible VIP" abort. Now reads the overlay. Verified: derives 10.12.4.57.
  • runbooks/phase-00-teardown-maas-reset.md:233 bare checker call -> merged input.
  • runbooks/dc-dc-phase4-juju-bundle-per-dc.md said "DC1 needs NO VIP/CIDR changes at all" and "deploy the EXISTING bundle.yaml with NO VIP edits". Followed verbatim post-move that deploys a VIP-free DC1. Superseded in place (the D-101 plane reasoning is kept, because it is still correct); the DC2 instruction to "edit bundle.yaml's VIP lines" now points at rendering a new overlay instead.
  • runbooks/appendix-A-troubleshooting.md L3 told an operator to grep the bundle for the metal-side VIP token -- which now returns zero, and the block's || true makes zero look like a clean result. Note added.
  • tests/render-baseline/README.md records a capture-time bundle.yaml sha256 that no longer re-verifies. Marked HISTORY with what changed and why, rather than re-pinned -- the fixture hashes themselves are untouched.

11. Harness: 30 -> 31, with the fixture BASE re-pointed

tests/provider-bundle-check now builds its fixture base by MERGING overlays/vr1-dc0-vips.yaml into bundle.yaml, because that pair is the deploy input. T16's label changed from "pristine repo bundle" to "the real dc0 deploy input" -- same assertion, honest name. NEW T31 asserts ruling 3's new invariant directly: the BARE base fails for every clustered principal, so "deployed without its overlay" is a tested state rather than an assumption. 31/31.

Evidence. Gauntlet ALL GREEN (85) on vcloud; repo-lint 0 fail / 610 files; render-dc-overlays 17/17; render-baseline 10/10 (dc1 reproduction still byte-exact after the comment-support change); preflight 16/16.

Still red, unchanged and expected: pre-flight-checks CHECK 1 reports OK=11 want 13 and provider-bundle-check reports designate -- both are R11's .61/.62 VIPs being RULED-BUT-NOT-BUILT, which is commit 2's job, not this one's.


2026-07-29 -- RULING 3, COMMIT 2: the dual-stack ADD (R2 + R11 built)

Operator: "Continue as autonomously as possible. Deploy agents as needed." Two read-only agents mapped the blast radius before any edit -- the same move that caught the deploy-gate break in commit 1, and it caught another one here.

12. Both per-DC VIP overlays are now DUAL-FAMILY, 13 apps

What. overlays/vr1-dc0-vips.yaml and overlays/vr1-dc1-vips.yaml re-rendered: each app carries prefer-ipv6: true and a six-address vip (3 v4 legs + 3 v6), and vault .61 and designate .62 are now BUILT (R11, a D-020 amendment). v6 prefixes come from the apex; the host part mirrors the v4 octet textually.

RESULT, measured both DCs: 13 clustered VIP(s) ... (13 dual-family) and 12 hacluster principal(s) all carry a VIP -> PASS. With dc-ha-scaleup.yaml stacked on top: 13 principals, PASS -- so R6's ruled ordering (VIPs before the HA overlay) is now satisfied and executable, not just stated.

Two decisions that were RULED-BUT-NEVER-BUILT are now artifacts: R11's vault/designate VIPs, and R2's dual-stack for the API VIP layer.

Revert. Re-render both overlays from the values files without --dual-family --include-unbuilt, and revert the gate/harness hunks below.

13. THE L3-9 COLLISION IS RESOLVED -- and the recorded description of it was half stale

overlays/dc-dc-ipv6-family-matrix.yaml carried a vip + prefer-ipv6 pair for ten API charms, every value an unrendered {{TOKEN}}. Those duplicated what the per-DC overlays now own, and take-last merge between them is the collision. Narrowed to ceph-mon only.

The vip and prefer-ipv6 keys were removed TOGETHER. Removing the vips alone would have recreated the exact defect on purpose -- which is why this is worth stating.

MEASURED, and it corrects this repo's own record: CURRENT-STATE described L3-9 as "the dangerous merge order is the one that PASSES". That has not been true since 2026-07-28. Invariant 9 (added by the arity commit, precisely because of L3-9) makes vips-last fail on the coupling, while matrix-last already failed on the placeholders -- so NEITHER order was green. The gate closed the SILENT half; this commit closes the collision itself. Both merge orders now PASS, i.e. the result is order-independent, which is the real fix.

What survives, because it exists in NO other file: ceph-mon's ceph-public-network / ceph-cluster-network ULA-only config and its privacy-extension caveat. ceph-mon is in neither VIP overlay and never will be -- it has no VIP. Also removed: the two Launchpad citations arguing octavia lb-mgmt IPv6 was an open risk. Both were read in full and BOTH failed (one a different installer's bug, Fix Released 2021; one not an IPv6 defect at all), and R8 RULED the question closed on 2026-07-27 -- leaving them standing in a live artifact was a refuted citation presented as current.

14. Gate repairs the agents caught BEFORE the edit

  • runbooks/phase-01-bundle-deploy.md -- the deploy guard would have ABORTED AGAIN. Its HI regex is 10\.12\.4\.(5[0-9]|60), which excludes .61 and .62, so the counts would have read 13/11/0 against a required 11/11/0 and juju deploy would not have run. Widened to (5[0-9]|6[0-2]) and the expected counts moved to 13/13/0. Verified end to end: 13/13/0 -> DEPLOY. This is the second time this one guard has nearly blocked the deploy across two commits; it is worth knowing it is that brittle.
  • scripts/pre-flight-checks.sh CHECK 1 hardcoded if(n!=3) and would have printed MALFORMED(not 3 IPs) for all 13 and failed OK=0 bad=13. Now accepts a v4 triple OR an R2 sextet, and for a sextet asserts the last three legs are actually v6 literals. Deep v6 prefix/mirror checking is deliberately NOT duplicated here -- provider-bundle-check does it against the apex. Proven able to fail: a 4-address vip -> MALFORMED(not 3 or 6), a v4 in a v6 slot -> NOT-IPv6 in v6 leg 1, and a v4 triple still passes.

15. Harness: 31 -> 33, and 18 for the renderer, with everything RE-POINTED not deleted

  • T16 and T17 both asserted R11's gaps were OPEN. R11 is now built, so both were inverted to the surviving invariant (the deploy input is CLEAN; R6's ordering is satisfied at 13 principals) and paired with NEW T16b/T17b that remove a VIP and demand the check still fires -- otherwise inverting them would have removed the teeth.
  • T20/T21, the coupling cases, had become NO-OPS (the overlay already sets what they were setting) -- they would have gone GREEN while testing nothing, which an agent caught. Inverted: T20 now truncates the vip while keeping prefer-ipv6, T21 drops prefer-ipv6 while keeping six legs.
  • A v4-only twin fixture was added so the pre-dual-stack cases (T5/T8/T26/T28) still exercise band, octet-uniqueness and apex-independence instead of tripping the coupling check first and never reaching their own assertion.
  • The renderer harness now proves reproduction on BOTH shapes: T3 against the FROZEN v4 fixture (immutable, a stable target) and new T3b against the LIVE dual-family overlay (whose own header mandates it stay renderer-reproducible).

Evidence. provider-bundle-check 33/33, render-dc-overlays 18/18, render-baseline 10/10, preflight 16/16. Gauntlet ALL GREEN (85) on vcloud; repo-lint 0 fail / 611 files.

PREFLIGHT: BOTH STANDING REDS CLEARED. It was 3 fatal with P2 and CHECK 1 among them; it is now 2 fatal -- the deliberately-absent octavia-pki overlay and MAAS unreachable from vcloud (a missing-binary artifact that R10 measured clears on voffice1). Nothing this work introduced is red.


Findings LOGGED, NOT ACTIONED

  1. ceph-mon's two CIDRs are gated by NOTHING. provider-bundle-check skips any app without a vip, so {{DC_STORAGE_ULA_CIDR}} / {{DC_REPLICATION_ULA_CIDR}} are validated by no gate at all -- and they are now the ONLY content of that overlay. A prominent header warning was added; rejecting {{...}} in any option value would be the real fix and is a gate change out of this commit's scope.
  2. octavia's dual-family API VIP is an INFERENCE, flagged not hidden. R2 ruled dual-stack and R8 ruled octavia's lb-mgmt network, but nothing explicitly rules that octavia's own API VIP goes dual-family. It is in both VIP overlays, so a uniform re-render gave it prefer-ipv6: true. That follows R2 applied evenly and matches the old matrix file's own reasoning ("no different from the other charms"), but it is not a quoted ruling -- worth one operator confirmation.
  3. Nothing checks vip-WITHOUT-hacluster. Invariant 8 catches a clustered principal with no VIP; the inverse is unguarded. Relevant because vault-hacluster exists only in dc-ha-scaleup.yaml, so a base+dc0-vips deploy gives vault a VIP and no HA relation.
  4. runbooks/dc-dc-phase4-juju-bundle-per-dc.md Step 6 still instructs passing the matrix overlay alongside two overlays that do not exist on disk and no VIP overlay. Wrong before this change; now wrong differently.