geneve-over-v6 overlay FAILURE -- root cause (dc0 activation checkpoint, 2026-08-08)

Status authority is docs/CURRENT-STATE.md; this is the root-cause record for the checkpoint "verify-live geneve-over-v6" wrap-gate item. Surfaced by the 1-test-LB smoke test (D-021 amphora build had just been fixed). All claims below are MEASURED this session.

Symptom

• 1 test LB 24bd326f-611c-4a8c-a348-e9c67f5d156c (VIP 198.51.100.21) stuck PENDING_CREATE.
• Amphora nova instance 66542b9f-... is ACTIVE (image 775ebeba-... boots -> image validated), on lb-mgmt-net at fc00:5b7a:7bdc:bd86:f816:3eff:fe2b:9473, but Octavia amphora = BOOTING.
• octavia health-manager log: repeated ConnectTimeout to [fc00:...:9473]:9443 (NOT ssl/cert/ refused -> not a version/agent problem).
• lb-mgmt-sec-grp correctly allows tcp IPv6 ::/0 9443 + ipv6-icmp (SG is NOT the cause).
• ping6 from o-hm0 (fc00:5b7a:7bdc:bd86:f816:3eff:fef9:2dc8, same /64) -> amphora = 100% loss (even ICMPv6, which the SG allows) -> a same-subnet OVERLAY delivery failure, cross-node.

Root cause (measured)

OVN geneve encap IPs are split across address families between containerized control chassis and metal compute chassis, so no control<->compute geneve tunnel can form:

Corroboration on BOTH compute chassis (ovs-vsctl list interface) -- INCLUDING vr1-dc0-compute-01, the amphora's actual host (ovn-chassis/1): geneve tunnels to the three control chassis carry IPv4 remote_ip=10.12.16.{1,2,3} while the local endpoint is IPv6 -> cross-family, cannot encapsulate. Only the compute<->compute tunnel ([2602:f3e2:f02:30::12x], v6<->v6) is family-consistent. NOTE on evidence weight: bfd_status read empty on all tunnels, but that is AMBIGUOUS (OVN geneve does not necessarily populate BFD) and is NOT relied on -- the DECISIVE evidence is the encap family split + the 100% ICMPv6 loss on the tested control->compute path. The v6<->v6 compute<->compute data path is UNTESTED (no second instance existed to ping), so it is NOT claimed healthy.

Mechanism -- why control got v4, compute got v6

The octavia control chassis is an LXD container whose data/geneve-plane interface (eth2) has ONLY 10.12.16.1/22 (IPv4); it has NO v6 address on the data-tenant plane (its only v6 is o-hm0, the lb-mgmt OVERLAY it serves, not the underlay). ovn-chassis therefore derives an IPv4 ovn-encap-ip. Metal compute nodes carry IPv6 on the same plane and derive a v6 encap. Governing prior art: D-134 records LXD container addresses are AUTO-PICKED (not carved) -> the auto-pick landed IPv4-only for the containerized chassis, diverging from the carved metal's IPv6. D-101 intends geneve-over-v6 on the data-tenant underlay and names it a Phase-0 verification gate; this is the live proof that gate is not clean.

SAME PLANE, family-selection defect (NOT a binding error): lib-net.sh PLANE_CIDRS confirms 10.12.16.0/22 = the data-tenant plane, and data-tenant is in SPACES6 (v6-capable); the v6 side is the GUA 2602:f3e2:f02:30::/64 seen on metal. Both node types bind the same data-tenant space -- the divergence is address-family SELECTION on one dual-stack plane (containers got a v4-only address there, metal got v6), not a mis-binding. Note also that neither side matches D-101's "IPv6-only ULA data-tenant" literally (compute uses a GUA, control v4).

Roosevelt-delta (why this outlives throwaway dc0)

Any v6 build with CONTAINERIZED OVN chassis (octavia/ovn-chassis-octavia, and any LXD-hosted ovn-chassis) must give those containers a v6 data-tenant address so geneve encap is v6 and matches metal. The auto-picked LXD addressing (D-134) does not currently guarantee this. This is the containerized-chassis-v6-underlay problem; it will recur on the 10.13 rebuild. Warrants a new D-number (GA-R3: architectural + Roosevelt-delta) -- PROPOSED; number to be assigned by grep-for-next-free AT ruling time (operator ruling owed), not pre-allocated here.

Fix landscape (all gated; none applied)

• Durable (rebuild): ensure containerized OVN chassis get a v6 address on the data-tenant plane (carve/assign, not auto-pick) so ovn-encap-ip is v6 cloud-wide; OR pin the geneve encap family consistently. This is the real fix and belongs in the rebuild bundle/addressing.
• Live dc0 workaround (to complete the smoke test only): add a v6 data-tenant address to the 3 control-node OVN-chassis containers and set ovn-encap-ip to it (family-consistent with compute). Non-trivial (the containers have no v6 there today); throwaway-only.
• Alternative: flip compute encap to v4 to match control -- REJECTED direction (inverts the geneve-over-v6 design intent, D-101).

2026-08-09 -- v6<->v6 datapath: FIRST TEST RETRACTED (invalid instrument); STATUS = STILL UNTESTED

RETRACTION. A first 2026-08-09 attempt concluded "v6<->v6 geneve PROVEN NON-FUNCTIONAL" and inferred v4-forced. That conclusion is WITHDRAWN -- the test used an invalid packet source (fable-advisory catch; instrument-currency #23 sibling: the test did not bypass a layer that independently produces the same negative). The raw measurements were real but do not support the conclusion drawn:

Why T/T' prove nothing about v6 geneve. The ovnmeta-<net> namespace port is an OVN localport (how ML2/OVN implements metadata -- the same port exists on EVERY chassis). Per OVN architecture, traffic to/from a localport is NEVER forwarded over a tunnel -- ovn-controller installs an explicit drop for localport-sourced packets destined to a remote chassis. So T=100% loss and T'=0-geneve-emitted are the by-design result for ANY encap family; a fully healthy v4 overlay would show the identical T/T'. C2 (underlay) and C1 (same-chassis local delivery) are both consistent with a localport source and do not compensate. The distinguishing test must source from a real VM VIF port, not a localport.

On the "mixed-family mesh" worry: NOT a real mechanism -- ovn-controller builds tunnels pairwise, one per remote-chassis encap, as independent state; a v4 chassis in the SB does not poison a v6<->v6 pair's flows. The only casualties of the family split are the cross-family (control<->compute) pairs -- that part of the 2026-08-08 root cause STANDS UNCHANGED.

CURRENT STATUS: v6<->v6 geneve datapath = STILL UNTESTED. The "Fix landscape" below is NOT overturned. Upstream signal (fable + general OVN knowledge): geneve-over-IPv6 is a supported, production-exercised capability (OVS v6 tunnel endpoints since 2.6/2016; neutron overlay_ip_version=6; ovn-kubernetes runs single-stack v6 in production). Item 14 is stale man-page wording + thin test coverage, not a capability denial. Expectation: the corrected test PASSES and the v6-only direction (section 5.1 of the carve proposal) holds -- but it must be MEASURED, not assumed.

Corrected diagnostic sequence (bounded, ~30 min, dc0 throwaway -- do not exceed):

1. Tunnel-port health on compute-01: ovs-vsctl --columns=name,ofport,error list interface ovn0-vr1-d-0
   • ovs-appctl dpif/show | grep geneve. ofport>=0/no error -> kernel made the v6 tunnel; proceed. ofport:-1/error -> real kernel/OVS v6-geneve defect; STOP and rule v4 for the rebuild.
2. THE test: VM->VM cross-chassis ping from a REAL VM (console into instance-A, ping instance-B), tcpdump udp 6081 on both computes. Pass + geneve both ways -> VERDICT v6 geneve WORKS; family-split is the sole LB cause; v6-only fix reinstated. 0 emitted from a real VIF -> go to 3. Emitted, no reply -> underlay UDP-6081 filtering, check + stop.
3. Only if 2 emits nothing: ovs-appctl ofproto/trace from instance-A's in_port (real MAC/IP) -> instance-B. Ends in output:<tunnel ofport> -> flows fine, kernel encap failing (record, rule v4). Ends in drop -> capture the drop flow (that IS the transferable answer). STOP -- no kernel bisecting on a throwaway cloud.

MTU note for the rebuild: v6 geneve overhead is 20 bytes larger; neutron overlay_ip_version=6 must be set so tenant MTU computes to ~1422, else a real large-packet failure appears later (different bug).

RESOLVED 2026-08-09 -- ROOT CAUSE = BRACKETED v6 ovn-encap-ip (charm/OVN format bug); geneve-over-v6 IS viable

Ran the corrected bounded diagnostic (Step 1 + a self-contained distinguishing test). geneve-over-v6 is NOT broken in this stack; a bracket-format bug prevents the tunnel from instantiating. Measured on vr1-dc0-compute-01, OVS 3.3.0 / OVN 24.03.2 / kernel 5.15.0-186 / geneve.ko 0.6 (kernel datapath):

• Step 1 -- v6 tunnel port health: ovn0-vr1-d-0 (remote_ip=[2602:f3e2:f02:30::121]) -> ofport: -1, error: "bad geneve 'remote_ip'". The v6 tunnel is NOT in the datapath (dpif/show shows only the 3 IPv4 tunnels). This -- not the retracted localport test -- is the valid evidence that cross-compute v6 geneve does not forward today.
• Distinguishing test (throwaway bridge, cleaned up): a geneve interface with an UNbracketed v6 remote_ip=2602:f3e2:f02:30::121 -> ofport: 1, no error; the bracketed [2602:...::121] -> ofport -1, same "bad remote_ip"; v4 control -> ofport 2, no error. So kernel 5.15 + OVS 3.3 DO support IPv6 geneve endpoints; the brackets are the sole blocker.
• Pinpoint: ovs-vsctl get open_vswitch . external_ids:ovn-encap-ip on both computes returns the value bracketed ("[2602:f3e2:f02:30::120]" / "...::121]"). The ovn-chassis 24.03 charm sets the v6 encap-ip wrapped in [...]; that bracketed literal propagates to the OVS geneve remote_ip, which OVS rejects. juju config ovn-chassis exposes no encap/ip-family/source-ip lever to override it.

VERDICT: geneve-over-IPv6 is achievable on this exact kernel/OVS/OVN; the blocker is the bracketed v6 ovn-encap-ip emitted by ovn-chassis 24.03. This is a config/charm defect, NOT a kernel or capability limit and NOT a reason to force v4.

Consequence for the fix landscape (below) -- REINSTATED with a correction. The original durable fix (give containerized OVN chassis a v6 data-tenant address so encap is v6 cloud-wide) is correct AND NECESSARY but NOT SUFFICIENT: the v6 encap-ip must ALSO be delivered to OVS unbracketed, or every chassis (metal included) gets ofport -1 and no v6 geneve forms. So the rebuild needs BOTH: (i) v6 on the LXD chassis' data-tenant leg (carve, not auto-pick), and (ii) an unbracketed v6 ovn-encap-ip -- via a fixed/newer ovn-chassis charm revision, or a verified post-config override. Owed rebuild investigation: is the bracket a known upstream ovn-chassis/OVN bug with a fix revision? (not yet searched).

Roosevelt-delta (corrected): a v6 build must (a) give containerized chassis a carved v6 data-tenant address, (b) ensure ovn-encap-ip reaches OVS unbracketed, and (c) gate on BOTH the encap-family consistency AND every v6 tunnel port having ofport >= 0 / empty error (the bracket bug is invisible to a family-only check -- all chassis can be "v6" and still have ofport -1). v4-forced is OFF the table.

LIVE-CONFIRMED 2026-08-09 -- unbracket fix proven end-to-end (real VM->VM, 0% loss). TESTED SEQUENCE for the rebuild.

The fix was proven live on dc0 (operator-directed: capture tested settings/steps for the 10.13 rebuild). Exact tested sequence, run from the dc0 rack (172.31.0.2, D-138), all steps captured:

1. Diagnose (per compute): sudo ovs-vsctl --columns=ofport,error list interface <geneve-port> -> ofport -1, error "bad geneve 'remote_ip'"; sudo ovs-vsctl get open_vswitch . external_ids:ovn-encap-ip -> "[2602:f3e2:f02:30::120]" (bracketed = the defect).
2. Fix (per compute, unbracketed): sudo ovs-vsctl set open_vswitch . external_ids:ovn-encap-ip=2602:f3e2:f02:30::120 (compute-01) and ...::121 (compute-02). ovn-controller rebuilt the tunnels within ~12 s.
3. Verify tunnel: ovn0-vr1-d-0 -> ofport 14, error [], present in dpif/show as geneve ... remote_ip=2602:f3e2:f02:30::121 (unbracketed).
4. Verify forwarding (ofproto/trace from a real VM VIF): datapath action set(tunnel(ipv6_dst=2602:f3e2:f02:30::121, tp_dst=6081, geneve(...))) -> output to kernel tunnel.
5. Live datapath proof: VM geneve-proof-c (compute-01) -> VM geneve-proof-b (compute-02, 192.0.2.20): 8/8 received, 0% loss, rtt ~3 ms. Real VM->VM over geneve-over-IPv6.

CONCLUSION (measured, not inferred): geneve-over-IPv6 works on OVS 3.3.0 / OVN 24.03.2 / kernel 5.15 once ovn-encap-ip is delivered to OVS UNBRACKETED. The ovn-chassis 24.03 charm's bracketed v6 ovn-encap-ip is the sole defect. step 2 is a live WORKAROUND (transient -- a charm hook may re-assert the bracketed value); the DURABLE rebuild fix is a fixed/newer charm revision OR a persistent post-deploy override. Owed for the rebuild plan (change-delivery loop, not this session): (a) confirm the fixed charm revision (charm-ovn-chassis IPv6 bracket bug family -- LP #1968355 area; not yet pinned to a fix rev); (b) add the encap unbracketing + the two verification gates below to the phase-04/phase-01 deploy runbook; (c) the gate must assert BOTH encap-family consistency AND every tunnel ofport>=0/empty error (a family-only check passes while all tunnels sit at ofport -1).

Also required for a v6 overlay (not re-tested here, carry forward): the containerized control-plane chassis (octavia/ovn-chassis-octavia LXD) still need a CARVED v6 data-tenant address (they are v4-only today -- the original 2026-08-08 family split), AND neutron overlay_ip_version=6 for correct tenant MTU (~1422). The unbracket fix is necessary but only sufficient for the metal<->metal path proven here.

Teardown of geneve-proof-* + lb-smoke proceeds now (reproduction captured above). The manual step-2 encap override is left as-is on throwaway dc0 (harmless; not persisted to git/charm).

REBUILD RECIPE -- geneve-over-v6 on 10.13 (consolidated; ruled D-139 AMENDMENT 2026-08-09)

One place for the rebuild session. Three fixes + one gate; order matters (DEFECT 2).

1. Container v6 carve (fixes the 2026-08-08 family split). The LXD ovn-chassis chassis (octavia / ovn-chassis-octavia) must hold a v6 data-tenant address, not the D-134 auto-picked v4. Extend the carve so LXD-container data-tenant legs get v6 (parity with metal ::12x). Verify with juju exec --unit ovn-central/leader -- ovn-sbctl list encap -> all Encap.ip v6.
2. Unbracket the encap (fixes the 2026-08-09 bracket bug). ovn-chassis 24.03 sets external_ids:ovn-encap-ip BRACKETED; OVS geneve -> ofport -1. No fixed charm revision was pinned by search 2026-08-09 (charm-ovn-chassis LP #1968355 IPv6 family; inconclusive) -- so do NOT assume a channel bump fixes it. RELIABLE fix = a persistent post-deploy override on every chassis host: sudo ovs-vsctl set open_vswitch . external_ids:ovn-encap-ip=<unbracketed v6> (idempotent; re-assert after any ovn-chassis config-changed hook -- wire it as a oneshot/subordinate, not a one-time command). Re-test a NEWER ovn-chassis channel at rebuild; if it emits unbracketed, the override is unneeded.
3. neutron overlay_ip_version=6 for tenant MTU ~1422 (v6 geneve +20B), else large-packet failures later.
4. GATE at Stage-5 close, before any workload smoke: bash scripts/geneve-encap-assert.sh --live "$DC_MODEL" --expect-family v6 (C1 family + C2 tunnel ofport). A real VM->VM cross-compute ping is the functional second half -- source from a VM VIF, NOT an ovnmeta/localport (localport traffic never tunnels; that was the retracted test).

Ordering (D-139 DEFECT 2): carve node/container v6 statics WHILE the v4 twin still exists (dc-node-v6-carve keys off the v4 last octet), THEN remove data-tenant v4. Still OWED (build at rebuild, needs the live cloud): D-139's metal-admin-leg-IPv4 gate (CARRIED RISK -- assert every container's metal-admin leg is v4; same offline+--live shape as geneve-encap-assert; specced but not built, to avoid building against an unmeasured target).

Evidence

This session's captured output (transcript): ovn-sbctl show, ovs-vsctl list interface (bfd_status empty), octavia health-manager log ConnectTimeout, ping6 100% loss, SG rule list, ip addr on octavia/leader (eth2 v4-only on data). Full changelog at session close.