diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index fd76dbc..07f129d 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -63,8 +63,14 @@ > image validated) but stays PENDING_CREATE: **o-hm0 -> amphora over lb-mgmt (v6-ULA) = 100% > ICMPv6 loss** (SG correctly allows v6 tcp/9443) -- a **geneve-over-v6 overlay** delivery failure > between the control-node octavia unit and the compute-node amphora. This IS the checkpoint's -> "verify-live geneve-over-v6" wrap-gate item; UNDER INVESTIGATION. Evidence: this session's -> captured output (transcript); full changelog at session close. +> "verify-live geneve-over-v6" wrap-gate item -- **ROOT-CAUSED (gate result: FAIL):** OVN geneve +> encap IPs are SPLIT ACROSS FAMILIES -- the containerized control-node chassis (octavia LXD) +> have IPv4-only data-plane addresses (`10.12.16.x`; D-134 auto-picked, not carved) so derive +> IPv4 encap, while the carved compute METAL uses IPv6 (`2602:f3e2:f02:30::x`); cross-family +> geneve tunnels never form (`bfd_status` empty), 100% cross-node loss. Roosevelt-delta: v6 +> builds must give containerized OVN chassis a v6 data-tenant address (candidate D-143, PROPOSED). +> Fix is substantial + rebuild-relevant; NOT applied. Root-cause record: +> `docs/audit/geneve-over-v6-rootcause-20260808.md`. Full changelog at session close. > ## >>> STANDING OPERATOR DIRECTIVE, 2026-07-30: THE NEXT SESSION PROCEEDS TO THE JUJU DEPLOYMENT (STAGE 5). NO MATTER WHAT. <<< > diff --git a/docs/audit/geneve-over-v6-rootcause-20260808.md b/docs/audit/geneve-over-v6-rootcause-20260808.md new file mode 100644 index 0000000..2299efb --- /dev/null +++ b/docs/audit/geneve-over-v6-rootcause-20260808.md @@ -0,0 +1,70 @@ +# geneve-over-v6 overlay FAILURE -- root cause (dc0 activation checkpoint, 2026-08-08) + +Status authority is `docs/CURRENT-STATE.md`; this is the root-cause record for the +checkpoint "verify-live geneve-over-v6" wrap-gate item. Surfaced by the 1-test-LB smoke +test (D-021 amphora build had just been fixed). All claims below are MEASURED this session. + +## Symptom +- 1 test LB `24bd326f-611c-4a8c-a348-e9c67f5d156c` (VIP 198.51.100.21) stuck `PENDING_CREATE`. +- Amphora nova instance `66542b9f-...` is ACTIVE (image `775ebeba-...` boots -> image validated), + on lb-mgmt-net at `fc00:5b7a:7bdc:bd86:f816:3eff:fe2b:9473`, but Octavia amphora = `BOOTING`. +- octavia health-manager log: repeated `ConnectTimeout` to `[fc00:...:9473]:9443` (NOT ssl/cert/ + refused -> not a version/agent problem). +- `lb-mgmt-sec-grp` correctly allows `tcp IPv6 ::/0 9443` + `ipv6-icmp` (SG is NOT the cause). +- ping6 from o-hm0 (`fc00:5b7a:7bdc:bd86:f816:3eff:fef9:2dc8`, same /64) -> amphora = **100% loss** + (even ICMPv6, which the SG allows) -> a same-subnet OVERLAY delivery failure, cross-node. + +## Root cause (measured) +OVN geneve **encap IPs are split across address families** between containerized control +chassis and metal compute chassis, so no control<->compute geneve tunnel can form: + +| Chassis | Role | `ovn-sbctl` Encap geneve ip | Family | +|---|---|---|---| +| `juju-d4f45a-0-lxd-13` | octavia/o-hm0 (control, LXD) | `10.12.16.1` | IPv4 | +| `juju-d4f45a-1-lxd-15` | control (LXD) | `10.12.16.2` | IPv4 | +| `juju-d4f45a-2-lxd-13` | control (LXD) | `10.12.16.3` | IPv4 | +| `vr1-dc0-compute-01` | amphora host (metal) | `[2602:f3e2:f02:30::120]` | IPv6 | +| `vr1-dc0-compute-02` | compute (metal) | `[2602:f3e2:f02:30::121]` | IPv6 | + +Corroboration on compute-02 (`ovs-vsctl list interface`): geneve tunnels to the control +chassis carry IPv4 `remote_ip=10.12.16.{1,2,3}` while the local endpoint is IPv6; **every +tunnel `bfd_status` is empty** (none established). Only the compute<->compute tunnel +(`[2602:f3e2:f02:30::120]`, v6<->v6) is family-consistent. + +## Mechanism -- why control got v4, compute got v6 +The octavia control chassis is an **LXD container** whose data/geneve-plane interface +(`eth2`) has ONLY `10.12.16.1/22` (IPv4); it has NO v6 address on the data-tenant plane +(its only v6 is `o-hm0`, the lb-mgmt OVERLAY it serves, not the underlay). ovn-chassis +therefore derives an IPv4 `ovn-encap-ip`. Metal compute nodes carry IPv6 on the same plane +and derive a v6 encap. Governing prior art: **D-134 records LXD container addresses are +AUTO-PICKED (not carved)** -> the auto-pick landed IPv4-only for the containerized chassis, +diverging from the carved metal's IPv6. D-101 intends geneve-over-v6 on the data-tenant +underlay and names it a Phase-0 verification gate; this is the live proof that gate is not +clean. + +Note: neither side matches D-101's "IPv6-only **ULA** data-tenant" literally -- compute uses +a **GUA** `2602:f3e2:f02:30::/64` and control uses v4 `10.12.16.0/22`. The data-tenant plane +is effectively dual-stack and encap selection diverges by node type (container vs metal). + +## Roosevelt-delta (why this outlives throwaway dc0) +Any v6 build with CONTAINERIZED OVN chassis (octavia/ovn-chassis-octavia, and any LXD-hosted +ovn-chassis) must give those containers a v6 data-tenant address so geneve encap is v6 and +matches metal. The auto-picked LXD addressing (D-134) does not currently guarantee this. This +is the containerized-chassis-v6-underlay problem; it will recur on the 10.13 rebuild. Candidate +for a new D-number (GA-R3: architectural + Roosevelt-delta; next-free D-143) -- PROPOSED, not +minted (operator ruling owed). + +## Fix landscape (all gated; none applied) +- **Durable (rebuild):** ensure containerized OVN chassis get a v6 address on the data-tenant + plane (carve/assign, not auto-pick) so `ovn-encap-ip` is v6 cloud-wide; OR pin the geneve + encap family consistently. This is the real fix and belongs in the rebuild bundle/addressing. +- **Live dc0 workaround (to complete the smoke test only):** add a v6 data-tenant address to + the 3 control-node OVN-chassis containers and set `ovn-encap-ip` to it (family-consistent + with compute). Non-trivial (the containers have no v6 there today); throwaway-only. +- **Alternative:** flip compute encap to v4 to match control -- REJECTED direction (inverts the + geneve-over-v6 design intent, D-101). + +## Evidence +This session's captured output (transcript): `ovn-sbctl show`, `ovs-vsctl list interface` +(bfd_status empty), octavia health-manager log ConnectTimeout, ping6 100% loss, SG rule list, +`ip addr` on octavia/leader (eth2 v4-only on data). Full changelog at session close.