Newer
Older
openstack-caracal-dc-dc / docs / audit / geneve-over-v6-rootcause-20260808.md

geneve-over-v6 overlay FAILURE -- root cause (dc0 activation checkpoint, 2026-08-08)

Status authority is docs/CURRENT-STATE.md; this is the root-cause record for the checkpoint "verify-live geneve-over-v6" wrap-gate item. Surfaced by the 1-test-LB smoke test (D-021 amphora build had just been fixed). All claims below are MEASURED this session.

Symptom

  • 1 test LB 24bd326f-611c-4a8c-a348-e9c67f5d156c (VIP 198.51.100.21) stuck PENDING_CREATE.
  • Amphora nova instance 66542b9f-... is ACTIVE (image 775ebeba-... boots -> image validated), on lb-mgmt-net at fc00:5b7a:7bdc:bd86:f816:3eff:fe2b:9473, but Octavia amphora = BOOTING.
  • octavia health-manager log: repeated ConnectTimeout to [fc00:...:9473]:9443 (NOT ssl/cert/ refused -> not a version/agent problem).
  • lb-mgmt-sec-grp correctly allows tcp IPv6 ::/0 9443 + ipv6-icmp (SG is NOT the cause).
  • ping6 from o-hm0 (fc00:5b7a:7bdc:bd86:f816:3eff:fef9:2dc8, same /64) -> amphora = 100% loss (even ICMPv6, which the SG allows) -> a same-subnet OVERLAY delivery failure, cross-node.

Root cause (measured)

OVN geneve encap IPs are split across address families between containerized control chassis and metal compute chassis, so no control<->compute geneve tunnel can form:

Chassis Role ovn-sbctl Encap geneve ip Family
juju-d4f45a-0-lxd-13 octavia/o-hm0 (control, LXD) 10.12.16.1 IPv4
juju-d4f45a-1-lxd-15 control (LXD) 10.12.16.2 IPv4
juju-d4f45a-2-lxd-13 control (LXD) 10.12.16.3 IPv4
vr1-dc0-compute-01 amphora host (metal) [2602:f3e2:f02:30::120] IPv6
vr1-dc0-compute-02 compute (metal) [2602:f3e2:f02:30::121] IPv6

Corroboration on compute-02 (ovs-vsctl list interface): geneve tunnels to the control chassis carry IPv4 remote_ip=10.12.16.{1,2,3} while the local endpoint is IPv6; every tunnel bfd_status is empty (none established). Only the compute<->compute tunnel ([2602:f3e2:f02:30::120], v6<->v6) is family-consistent.

Mechanism -- why control got v4, compute got v6

The octavia control chassis is an LXD container whose data/geneve-plane interface (eth2) has ONLY 10.12.16.1/22 (IPv4); it has NO v6 address on the data-tenant plane (its only v6 is o-hm0, the lb-mgmt OVERLAY it serves, not the underlay). ovn-chassis therefore derives an IPv4 ovn-encap-ip. Metal compute nodes carry IPv6 on the same plane and derive a v6 encap. Governing prior art: D-134 records LXD container addresses are AUTO-PICKED (not carved) -> the auto-pick landed IPv4-only for the containerized chassis, diverging from the carved metal's IPv6. D-101 intends geneve-over-v6 on the data-tenant underlay and names it a Phase-0 verification gate; this is the live proof that gate is not clean.

Note: neither side matches D-101's "IPv6-only ULA data-tenant" literally -- compute uses a GUA 2602:f3e2:f02:30::/64 and control uses v4 10.12.16.0/22. The data-tenant plane is effectively dual-stack and encap selection diverges by node type (container vs metal).

Roosevelt-delta (why this outlives throwaway dc0)

Any v6 build with CONTAINERIZED OVN chassis (octavia/ovn-chassis-octavia, and any LXD-hosted ovn-chassis) must give those containers a v6 data-tenant address so geneve encap is v6 and matches metal. The auto-picked LXD addressing (D-134) does not currently guarantee this. This is the containerized-chassis-v6-underlay problem; it will recur on the 10.13 rebuild. Candidate for a new D-number (GA-R3: architectural + Roosevelt-delta; next-free D-143) -- PROPOSED, not minted (operator ruling owed).

Fix landscape (all gated; none applied)

  • Durable (rebuild): ensure containerized OVN chassis get a v6 address on the data-tenant plane (carve/assign, not auto-pick) so ovn-encap-ip is v6 cloud-wide; OR pin the geneve encap family consistently. This is the real fix and belongs in the rebuild bundle/addressing.
  • Live dc0 workaround (to complete the smoke test only): add a v6 data-tenant address to the 3 control-node OVN-chassis containers and set ovn-encap-ip to it (family-consistent with compute). Non-trivial (the containers have no v6 there today); throwaway-only.
  • Alternative: flip compute encap to v4 to match control -- REJECTED direction (inverts the geneve-over-v6 design intent, D-101).

Evidence

This session's captured output (transcript): ovn-sbctl show, ovs-vsctl list interface (bfd_status empty), octavia health-manager log ConnectTimeout, ping6 100% loss, SG rule list, ip addr on octavia/leader (eth2 v4-only on data). Full changelog at session close.