Newer
Older
openstack-caracal-dc-dc / docs / audit / gua-carve-completion-proposal-20260809.md

GUA carve completion -- proposal + missing-assignment scan (2026-08-09)

Status: DRAFT PROPOSAL. Nothing here is a ruling. I propose; the operator rules (GA-R5). Not a status surface (GA-R1) -- stage/gate status lives in docs/CURRENT-STATE.md only. Origin: the geneve-over-v6 LB failure (docs/audit/geneve-over-v6-rootcause-20260808.md) -> a NetBox apex check -> operator direction: "assign the carves ... propose the missing gua carve, scan for any additional missing assignments, and provide a proposed matrix." Rev 2 -- incorporates a fable-level adversarial review (findings folded in below; the review's evidence is cited where it changed a claim).

Evidence base. Current-state figures are from a fresh READ-ONLY apex dump this session: netbox/draft/vr1-office1-current-20260809.json (office1-netbox 10.10.1.10, DOCFIX-195 VR1 working apex -- 152 prefixes / 27 ip-ranges / 194 ip-addresses; every number below independently re-derived in the fable review). Rulings cited inline. No mutation performed.


1. Governing frame (cited, not re-argued)

  • D-139 ruling A -- family matrix per DC: provider-public + metal-admin dual-stack; metal-internal / data-tenant / storage / replication IPv6-only (target); lb-mgmt IPv6-only (new plane).
  • D-139 ruling B -- Full GUA on every plane, from each DC's /48 (2602:f3e2:f02::/48 dc0, 2602:f3e2:f03::/48 dc1). ULA /48 fd50:840e:74e2::/48 RETIRED for VR1. (Deciding RFC-6724 reason struck 2026-08-01; ruling stands on Willamette/VR0 conformance + Roosevelt-delta. Ruling B is orthogonal to the v4-keep question -- a GUA dual-stack carve is achievable today.)
  • D-139 "B plus C" narrowing (changelog-20260731 Items 14/15/16): the v6-only conversion is a bounded experiment on storage + replication TOGETHER (coupled by ceph-osd's global ms_bind_ipv4=False). metal-internal, data-tenant, lb-mgmt stay dual-stack. Item 14 (measured by charm/doc reading) records the blockers are upstream charm defects, not deployment faults: metal-internal (mysql URIs unbracketed-v6-invalid; corosync ip_version: ipv4), and -- the one that bears on data-tenant -- "OVN documenting the encap column as 'The IPv4 address of the encapsulation tunnel endpoint' with zero v6 encap values anywhere in OVN 24.03's test suite" (flagged there as a sourced observation, NOT re-verified). See section 5 for how this is engaged.
  • D-139 OOB amendments (2026-08-01 a/b) -- oob dual-stack: v6 f0X:f0::/60+/64; v4 dc0 10.12.40.0/22, dc1 10.12.88.0/22 (RULED, not built). VPN :e0 deferred to Roosevelt.
  • D-141 -- IPAM allocations authored dual-stack, status-distinguished: v4 active, GUA v6 reserved (until the consuming layer is capable; gate = docs/charm-ip-family-compatibility.md), superseded ULA deprecated; v6 host-numbering mirrors v4.
  • D-134 -- per-plane /22 v4 octet bands (utility .4-.49, VIP .50-.99, nodes .100-.200); its 2026-07-27 amendment defines the v6 bands -- do not invent v6 host values, read them there.
  • D-139 build constraints (measured): zero v6 ip-range rows are needed/possible (MAAS auto-reserves ::1-::ffff:ffff per /64; node statics live at ::100...; containers auto-get <prefix>:0:1::). A v6-only plane is expressible; dual-stack on a container plane is not (juju takes one family via addrs[0], non-deterministic -- LP #1723240; no knob).

Re-IP interaction: the 10.12->10.13 re-IP pivot is v4-only (10.12 collides with the live cloud). The GUA v6 carve is unaffected (f02/f03 unchanged) -> completing the v6 GUA apex is durable, not throwaway, and is the apex-drives-deploy direction D-141 requires. Only the v4 OOB /22s move under the re-IP.


2. Current state -- measured (2026-08-09 apex dump; re-derived in review)

DC0 (f02) -- GUA carve structurally complete per convention.

  • 9 planes carry GUA prefixes (:10 :11 :20 :21 :30 :40 :50 :80 :f0), status active (16 rows).
  • GUA VIP host-addrs: 39, all reserved (13 each on :11/:20/:21) ok D-141.
  • ULA prefixes + ULA VIPs (26): all deprecated ok.
  • Node v6 statics: 0 apex records -- correct by convention (v4 side records no .100-.200 nodes either).

DC1 (f03) -- GUA carve INCOMPLETE (headline gap).

  • GUA prefixes: provider-public only (:10 :11). MISSING 7 planes / 13 prefix rows (:20 /60+/64, :21 /64, :30 /60+/64, :40 /60+/64, :50 /60+/64, :80 /60+/64, :f0 /60+/64).
  • GUA VIPs: 13 reserved (provider-public only). metal-admin/internal GUA VIPs MISSING (26).
  • ULA prefixes still active (5 planes / 9 rows: 320/321/330/340/350). ULA VIPs (26) still reserved on :320/:321 -- DC1's metal-admin/internal VIPs live only on retired-family ULA.

Cross-cutting findings (some are ruled-vs-built divergences, not just gaps):

  • v6 utility host records = 0 on both DCs -- juju .5/maas .6/tailscale .7/rack .2 are v4-only. D-141 rule 1 (author dual-stack) -> v6 reserved twins owed. Low consequence (utility is v4-reached).
  • lb-mgmt :80 prefix active on dc0 -- but D-139's G18 annotation (2026-08-08) rules the apex GUA lb-mgmt reserved (no charm consumer). Two rows exist (f02:80::/60 AND ::/64); the G18 text names only the /64 -- the /60's status is unspecified and must be asked, not assumed.
  • All 78 v4 VIP host-addrs are reserved on BOTH DCs (not active). If dc0's API VIPs are live (dc0 is deployed, core services active), the apex is under-promoted vs D-141 rule 2 -- a divergence to confirm/correct. If they are not yet verified-live, reserved is correct. Not asserted either way here.
  • Utility .4 (apt-cacher proxy, D-135 + 2026-08-02(b) convergence ruling) has no apex host row on any plane, either DC -- a candidate genuinely-missing v4 assignment (verify proxy build state before asserting).

3. Proposed GUA assignment matrix (both DCs)

Prefix active = the plane object exists (D-139 carve convention). VIP-allocation reserved per D-141. lb-mgmt prefix reserved per G18. Host-numbering mirrors the v4 octet (D-134 + its v6-band amendment).

Plane v6 /60 parent dc0 /64 dc1 /64 v4 (family) Prefix status VIP-alloc status
provider-public f0X:10::/60 f02:10::/64 f03:10::/64 10.12.4 / 64 (dual) active reserved
metal-admin f0X:20::/60 f02:20::/64 f03:20::/64 10.12.8 / 68 (dual) active reserved
metal-internal (in :20::/60) f02:21::/64 f03:21::/64 10.12.12 / 72 (dual; v6-only target, blocked section 5) active reserved
data-tenant f0X:30::/60 f02:30::/64 f03:30::/64 10.12.16 / 76 (dual; v6-only decision section 5) active --
storage f0X:40::/60 f02:40::/64 f03:40::/64 10.12.32 / 80 (v6-only experiment, gated) active --
replication f0X:50::/60 f02:50::/64 f03:50::/64 10.12.36 / 84 (v6-only experiment, gated) active --
lb-mgmt f0X:80::/60 f02:80::/64 f03:80::/64 none (v6-only, new) reserved (G18) --
oob f0X:f0::/60 f02:f0::/64 f03:f0::/64 dc0 10.12.40 / dc1 10.12.88 (dual) active --

Prefix rows are authored now regardless of family conversion (both families coexist in the apex). "v6-only experiment" (storage/replication) is a bounded experiment gated on the still-untaken network-get measurement, not greenlit deployment.


4. Missing-assignment scan -- enumerated

# Missing / divergent assignment Layer Tool / step Note
M1 DC1 GUA plane prefixes (7 planes / 13 rows) apex v6 netbox/d139-gua-carve.py --dc vr1-dc1 (CREATE-only; D-139 step 1) executes ruled D-139
M2 DC1 metal-admin + metal-internal GUA VIPs (26, reserved) apex v6 D-139 step 6 re-home needs its own GA-R5 (section 5.2) -- D-139 ORDERING RULING reserved step 6
M3 DC1 ULA retirement -- 9 prefixes + 26 VIPs -> deprecated apex v6 D-139 step 6, AFTER M2 (DEFECT 3) same GA-R5 as M2
M4 OOB v4 /22 (dc0 10.12.40, dc1 10.12.88) apex+MAAS v4 RULED not built; carve tool is v6-only -> tool gap moves under re-IP -> defer (section 5.2)
M5 lb-mgmt status -- dc0 :80 /64 (and /60?) active->reserved; author dc1 :80 reserved apex v6 status edit(s) two rows; /60 status is an operator question (G18 named only /64)
M6 v6 utility host records on dual-stack planes apex v6 D-141 rule 1; v6 values from D-134 amendment, not invented low consequence
M7 DC1 v4 utility host records (thinner than dc0) apex v4 at DC1 standup expected while DC1 held
M8 .4 apt-cacher proxy host record (both DCs) apex v4 verify proxy build first candidate real gap (D-135 / 2026-08-02(b))
S2 MAAS GUA subnet carve per DC (distinct from node statics) MAAS D-139 step 2 out of apex-only scope; rack-side; listed so it is not lost
S4 Octavia v6 IP-SAN reissue + lib-net.sh v6 arm PKI + repo D-139 step 4 (octavia-pki.sh reissue) out of apex-only scope
S3 Node v6 statics in MAAS (::100.../node) MAAS dc-node-v6-carve.py (D-139 step 3) NOT asserted absent -- needs rack access; dc0 data-tenant confirmed carved (::120/::121), completeness UNMEASURED
S5 lb-mgmt VLAN/space/subnet MAAS D-139 step 5 never carved either family

Overlay/render coupling (from review -- must not be missed): M2/M3 touch only the apex, but the D-136 rendered per-DC VIP overlays carry DC1's v6 VIPs in the ULA range. Retiring/ re-homing in the apex without regenerating the overlay consumers would leave a later DC1 deploy pointing at deprecated ULA VIPs (memory #16: enumerate a change's consumers). Enumerate the D-136 consumers as part of M2/M3.

Classification: M1, M5, M6 execute existing D-139/D-141 rulings (gated mutations). M2/M3 do NOT -- see section 5.2. S2-S5 are MAAS/PKI/repo steps outside this apex-only proposal, listed for completeness.


5. Decisions

5.1 RESOLVED 2026-08-09 -- geneve-over-v6 CONFIRMED working; v6-only data-tenant stands (D-number owed)

Governing preference (operator, 2026-08-09; = the standing D-101/D-139 posture): "v6-only, when not possible dual-stack v4/v6, and only if that is not possible v4."

The prerequisite this section used to gate on is now MEASURED, and it PASSED. geneve-over-IPv6 works on this exact stack (OVS 3.3.0 / OVN 24.03.2 / kernel 5.15.0-186) -- LIVE-CONFIRMED on vr1-dc0: a real VM->VM cross-compute ping over a v6 geneve tunnel = 8/8, 0% loss (evidence + tested sequence: docs/audit/geneve-over-v6-rootcause-20260808.md, LIVE-CONFIRMED section). So v6-only data-tenant is viable and v4-forced is OFF the table -- the posture's first rung holds. (A prior same-day test wrongly concluded "v6 broken"; it sourced pings from an OVN localport that never tunnels by design -- retracted.)

The v6-only fix is THREE parts, all now understood (it is NOT "just make the plane v6-only"):

  1. Carve v6 on the containerized (octavia LXD) chassis' data-tenant leg -- they auto-pick v4 (D-134); the original 2026-08-08 family split. Carve/assign, not auto-pick.
  2. Deliver ovn-encap-ip UNbracketed -- ovn-chassis 24.03 emits it bracketed ("[2602:...]"), which OVS geneve rejects (tunnel ofport -1, bad remote_ip). Fixed charm revision OR a persistent post-deploy override.
  3. neutron overlay_ip_version=6 -- tenant MTU ~1422 (v6 geneve overhead), else a later large-packet bug.

dual-stack is still NOT a viable middle for this plane (metal v6 + container v4 via addrs[0] = the split); the choice was only ever v6-only vs v4-only, and v6-only is now proven achievable.

Ordering hazard (D-139 DEFECT 2, unchanged): dc-node-v6-carve.py needs v4 present (if not v4: continue; host part = v4 last octet). Carve node v6 statics WHILE v4 exists, THEN remove v4 -- or rewrite the script to pivot on the plane/space.

The gate is now BUILT: scripts/geneve-encap-assert.sh (harness 16/16) asserts encap-family consistency AND every tunnel ofport>=0 (the family-only check missed the bracket bug); wired into phase-04 Step 12.2. Run at Stage-5 close before any workload smoke.

DECISION for ruling (GA-R5): ratify v6-only data-tenant (the fix is understood and proven) and settle the D-number -- the rootcause doc PROPOSED a new number; I lean D-139 amendment (parent owns the family matrix; avoids the mint-a-number trap). The finding is broader than one plane (any LXD-hosted ovn-chassis + the bracket bug) -- the Roosevelt-delta that argues either way. Operator's call.

5.2 Enumerated, NOT presented for ruling this exchange (recorded so they are not lost)

  1. DC1 step-6 (M2/M3) is reserved by D-139's ORDERING RULING for its own GA-R5 exchange (steps 4-6 "were NOT put"; step 6 "needs its own GA-R5 exchange ... NOT inferred"). So M2/M3 are a future decision, not mere execution -- do not run them on this proposal's authority.
  2. OOB v4 timing (M4): values move to 10.13.* under the owed re-IP ruling -> defer to it; don't carve v4 now to renumber later. (OPS/ordering.)
  3. DC1 apex prefixes (M1) now vs fold into rebuild: recommend now -- v6 GUA is re-IP-independent, the apex is the design source of truth (D-141), and M1 executes D-139 already ruled (gated mutation).

6. Geneve encap gate -- BUILT 2026-08-09 (was ruled-not-built)

D-139/D-101 named geneve-over-v6 a verification gate; nothing executable asserted it. Now delivered: scripts/geneve-encap-assert.sh (+ tests/geneve-encap-assert/run-tests.sh, 16/16) enumerates every chassis Encap IP and asserts single-family, == the ruled family (--expect-family v6, parameterized -- not a hardcoded v6) AND every geneve tunnel ofport>=0 (the bracket-bug check the family-only version missed); proven in both failing directions. Wired into phase-04 Step 12.2. Still owed -- D-139's OTHER gate: every container's metal-admin leg is IPv4 (the D-139 CARRIED RISK; more critical once data-tenant goes v6-only, leaving metal-admin the last dual-stack container plane) -- not yet built.


7. Limits / what I did NOT do

  • Apex-only proposal. No MAAS-side measurement (S2/S3/S5 need rack access; not asserted absent).
  • The OVN-v6-geneve capability is established generically (web sources); the deployment-specific datapath is unproven and is the section 5.1 gate. The addrs[0]<->D-134-auto-pick link is an inference joining two records, not this incident's independently measured mechanism.
  • "0 addresses in the apex" corroborates the 2026-07-31 live "zero ranges" finding (same direction); it is a design-level read, not a fresh live re-measure. D-139 established zero-ranges != can't-allocate.
  • No mutation performed. Every M-item is a gated action awaiting approval; M2/M3 additionally await their own GA-R5.