Newer
Older
openstack-caracal-dc-dc / docs / audit / vip-reconciliation-20260727.md

VIP / band value reconciliation -- read-only measurement, 2026-07-27

Produced for step 4 of the render-pipeline order of operations (settle the values before populating any authority). A read-only agent enumerated the four disagreeing surfaces; every consequential claim below was then re-verified directly before being recorded, per this project's standing rule that a lens finding is an OBSERVATION, not a conclusion. One of its conclusions did not survive that check -- see "Corrected" below.

Nothing here is adopted. Two items need an operator ruling.


1. NEW GAP -- R11 ruled three gate changes, but NOT the one R2 forces

VERIFIED DIRECTLY. scripts/provider-bundle-check.py:137:

        parts = str(vip).split()
        if len(parts) != 3:
            fails.append("%s vip not a triple: %r" % (n, vip)); continue

A DUAL-FAMILY vip is six addresses, not three. So under R2 the checker fails every application. R11's ruled gate changes were: band .60 -> .99, VIP_COUNT_EXPECT 11 -> 13, and FAIL on an hacluster relation with no VIP. Arity is not among them.

It does not stop at arity. Two further v4 assumptions in the same block:

  • :149 octs = {p.split(".")[-1] for p in parts} -- on a v6 literal this returns the ENTIRE address string, so the "all legs share one octet" invariant silently stops meaning anything.
  • ipaddress.ip_address('2602:f3e2:f02:11::50') in IPv4Network('10.12.4.0/22') returns False without raising, so a naive arity fix relocates the failure to the provider-leg check at :141-146 rather than solving it.

This is ruled-work-incomplete, not a code nit: R2 and R11 are both RULED, and the gate cannot express their combined end state. Whether the arity change rides along as OPS execution of R11 or needs its own exchange is an operator call.

2. TRAP -- EXPECT_PUBLIC_VIP must STAY 11 while VIP_COUNT_EXPECT goes to 13

VERIFIED DIRECTLY. provider-bundle-check.py:41-42 holds OCTET_LO, OCTET_HI = 50, 60 and EXPECT_PUBLIC_VIP = 11. :121-123 computes on_public from bindings["public"]. Measured against bundle.yaml:

app has a public binding?
keystone YES
vault NO ('', access, certificates, cluster, external, ha, secrets, shared-db)
designate NO ('', certificates, coordinator-memcached, dns-backend, dnsaas, ha, identity-service, shared-db)

So the two applications R11 adds do NOT join the public-VIP count. Anyone reading the ruling's "11 -> 13" and bumping both constants breaks the gate. Same class as the two-separately-named-band-constants trap the rulings already flag, in the opposite direction. designate's provider leg is dnsaas: provider-public, which is why R11 could still call a provider leg coherent for it.

3. The L3-9 collision is worse than recorded -- and the DANGEROUS order is the GREEN one

Ten applications are assigned vip by BOTH overlays/vr1-dc1-vips.yaml and overlays/dc-dc-ipv6-family-matrix.yaml. options.vip is a scalar, so per-key merge is last-wins.

  • ipv6-matrix LAST -> exit 1. Ten [FAIL] <app> vip not a triple because the matrix's values are unrendered {{TOKEN}} placeholders. Loud, safe.
  • vips LAST -> exit 0, "[ok] 11 clustered VIP(s) ... octet 50-60". Every IPv6 leg is silently dropped.

Beyond what L3-9 recorded: prefer-ipv6: true is a SEPARATE key and therefore SURVIVES the merge in both orders. So the green order would deploy charms configured to bind :::port with no v6 VIP for pacemaker to manage -- a green gate over a lost ruling AND a half-configured dual-stack.

CAVEAT ON METHOD: measured through provider-bundle-check.py's merge_overlay, whose own docstring calls it a MIRROR of juju's documented semantics. juju is absent from this host's PATH. Confirm with juju deploy --dry-run before relying on the merge result -- this is an unverified-mirror claim, not a juju measurement.

4. CORRECTED -- the dc1 band-bounds "inheritance" is CORRECT, not a defect

The agent reported that VIP_OCTET_MIN/MAX are absent from the dc1 unset list, so a dc1 selector-caller "silently inherits dc0's band bounds", framing it as a defect.

The observation is accurate and the conclusion is wrong. Measured: lib_net_select_dc vr1-dc1 -> VIP_OCTET_MIN=[50] VIP_OCTET_MAX=[60] VIP_COUNT_EXPECT=[UNSET] VIP_PREFIX_PROVIDER=[UNSET].

But the octet band is DC-INVARIANT by design: dc1's VIPs are .50-.60 within its own prefixes (10.12.64.50 10.12.68.50 10.12.72.50), exactly as dc0's are within 10.12.4/8/12. What differs per DC is the PREFIX, and the prefixes ARE unset. So the asymmetry at lib-net.sh:157 is deliberate and right: unset what differs, keep what does not. A useful consequence -- R11's widening of the band to .99 applies to both DCs automatically.

Recorded because a future session acting on the uncorrected framing would "fix" a non-defect and, in doing so, would have to invent per-DC band bounds that do not exist.

5. Numeric inconsistency between two same-day rulings

R4's text refers to "33 per-DC VIPs" (11 x 3). Post-R11 the count is 13 x 3 = 39 per family, and under R2's dual-family shape 78 addresses per DC. Both rulings are dated 2026-07-27. Flagged as an inconsistency to reconcile when the values are populated; neither is asserted to supersede the other.

6. Where dc0's values must land -- a FIFTH surface, and it does not exist

overlays/vr1-dc0-vips.yaml is RULED (2026-07-25 symmetric-overlay ruling: base becomes VIP-free, every per-DC value arrives via overlay, dc0 included) and measured absent. It is not one of the four surfaces enumerated, and it is what the renderer must write.

7. What is NOT measured, and what would measure it

  • Every IPv6 host part -- all 26 v6 triples. The apex carries the v6 PREFIXES for both DCs (GUA provider VIP /64 2602:f3e2:f02:11::/64 dc0 / f03:11::/64 dc1; ULA metal-admin fd50:840e:74e2:220::/64 / :320::/64; metal-internal :221::/64 / :321::/64) but zero per-address IPv6 objects. R4 left the v6 host-part mapping explicitly unruled. Would measure it: an operator ruling on the host-part convention.
  • A structural asymmetry the renderer will hit: the GUA provider leg has a DEDICATED VIP /64 separate from its node /64, while each ULA metal leg has only ONE /64 shared with node statics. So "mirror the v4 host part" is well-defined for the ULA legs (::50-::62 inside the plane /64) but genuinely ambiguous for the GUA leg: f02:11::50 (mirror the octet) or f02:11::1 (first in a dedicated block)? Both readings are defensible. Needs a ruling.
  • Charm-side acceptance that vault and designate take a three-address space-separated vip, and per-charm prefer-ipv6 support. Would measure it: juju config <app> post-deploy, or charm source / Charmhub.
  • Juju's real overlay merge semantics (see the caveat in section 3).

8. Constants inventory (for the execution pass, not adopted here)

file:line constant now note
lib-net.sh:58 VIP_OCTET_MAX 60 -> 99 (R11)
lib-net.sh:59 VIP_COUNT_EXPECT 11 -> 13 (R11)
provider-bundle-check.py:41 OCTET_LO, OCTET_HI 50, 60 -> 99; separately named from lib-net
provider-bundle-check.py:42 EXPECT_PUBLIC_VIP 11 STAYS 11 -- section 2
provider-bundle-check.py:137 arity literal 3 UNRULED -- section 1
pre-flight-checks.sh:78 VIPLINES greps bundle.yaml, counts LINES already-known VIP-free-base break; also arity-blind, so it could never detect the section-3 loss

scripts/checks/ was swept and hardcodes no VIP octet or count -- d011-02/d011-03 derive VIPs dynamically from endpoints.