# geneve-over-v6 overlay FAILURE -- root cause (dc0 activation checkpoint, 2026-08-08)

Status authority is `docs/CURRENT-STATE.md`; this is the root-cause record for the
checkpoint "verify-live geneve-over-v6" wrap-gate item. Surfaced by the 1-test-LB smoke
test (D-021 amphora build had just been fixed). All claims below are MEASURED this session.

## Symptom
- 1 test LB `24bd326f-611c-4a8c-a348-e9c67f5d156c` (VIP 198.51.100.21) stuck `PENDING_CREATE`.
- Amphora nova instance `66542b9f-...` is ACTIVE (image `775ebeba-...` boots -> image validated),
  on lb-mgmt-net at `fc00:5b7a:7bdc:bd86:f816:3eff:fe2b:9473`, but Octavia amphora = `BOOTING`.
- octavia health-manager log: repeated `ConnectTimeout` to `[fc00:...:9473]:9443` (NOT ssl/cert/
  refused -> not a version/agent problem).
- `lb-mgmt-sec-grp` correctly allows `tcp IPv6 ::/0 9443` + `ipv6-icmp` (SG is NOT the cause).
- ping6 from o-hm0 (`fc00:5b7a:7bdc:bd86:f816:3eff:fef9:2dc8`, same /64) -> amphora = **100% loss**
  (even ICMPv6, which the SG allows) -> a same-subnet OVERLAY delivery failure, cross-node.

## Root cause (measured)
OVN geneve **encap IPs are split across address families** between containerized control
chassis and metal compute chassis, so no control<->compute geneve tunnel can form:

| Chassis | Role | `ovn-sbctl` Encap geneve ip | Family |
|---|---|---|---|
| `juju-d4f45a-0-lxd-13` | octavia/o-hm0 (control, LXD) | `10.12.16.1` | IPv4 |
| `juju-d4f45a-1-lxd-15` | control (LXD) | `10.12.16.2` | IPv4 |
| `juju-d4f45a-2-lxd-13` | control (LXD) | `10.12.16.3` | IPv4 |
| `vr1-dc0-compute-01`   | amphora host (metal) | `[2602:f3e2:f02:30::120]` | IPv6 |
| `vr1-dc0-compute-02`   | compute (metal) | `[2602:f3e2:f02:30::121]` | IPv6 |

Corroboration on BOTH compute chassis (`ovs-vsctl list interface`) -- INCLUDING
`vr1-dc0-compute-01`, the amphora's actual host (`ovn-chassis/1`): geneve tunnels to the three
control chassis carry IPv4 `remote_ip=10.12.16.{1,2,3}` while the local endpoint is IPv6 ->
cross-family, cannot encapsulate. Only the compute<->compute tunnel (`[2602:f3e2:f02:30::12x]`,
v6<->v6) is family-consistent. NOTE on evidence weight: `bfd_status` read empty on all tunnels,
but that is AMBIGUOUS (OVN geneve does not necessarily populate BFD) and is NOT relied on -- the
DECISIVE evidence is the encap family split + the 100% ICMPv6 loss on the tested control->compute
path. The v6<->v6 compute<->compute data path is UNTESTED (no second instance existed to ping),
so it is NOT claimed healthy.

## Mechanism -- why control got v4, compute got v6
The octavia control chassis is an **LXD container** whose data/geneve-plane interface
(`eth2`) has ONLY `10.12.16.1/22` (IPv4); it has NO v6 address on the data-tenant plane
(its only v6 is `o-hm0`, the lb-mgmt OVERLAY it serves, not the underlay). ovn-chassis
therefore derives an IPv4 `ovn-encap-ip`. Metal compute nodes carry IPv6 on the same plane
and derive a v6 encap. Governing prior art: **D-134 records LXD container addresses are
AUTO-PICKED (not carved)** -> the auto-pick landed IPv4-only for the containerized chassis,
diverging from the carved metal's IPv6. D-101 intends geneve-over-v6 on the data-tenant
underlay and names it a Phase-0 verification gate; this is the live proof that gate is not
clean.

SAME PLANE, family-selection defect (NOT a binding error): `lib-net.sh` PLANE_CIDRS confirms
`10.12.16.0/22` = the `data-tenant` plane, and `data-tenant` is in `SPACES6` (v6-capable); the
v6 side is the GUA `2602:f3e2:f02:30::/64` seen on metal. Both node types bind the same
`data-tenant` space -- the divergence is address-family SELECTION on one dual-stack plane
(containers got a v4-only address there, metal got v6), not a mis-binding. Note also that neither
side matches D-101's "IPv6-only **ULA** data-tenant" literally (compute uses a GUA, control v4).

## Roosevelt-delta (why this outlives throwaway dc0)
Any v6 build with CONTAINERIZED OVN chassis (octavia/ovn-chassis-octavia, and any LXD-hosted
ovn-chassis) must give those containers a v6 data-tenant address so geneve encap is v6 and
matches metal. The auto-picked LXD addressing (D-134) does not currently guarantee this. This
is the containerized-chassis-v6-underlay problem; it will recur on the 10.13 rebuild. Warrants a
new D-number (GA-R3: architectural + Roosevelt-delta) -- PROPOSED; number to be assigned by
grep-for-next-free AT ruling time (operator ruling owed), not pre-allocated here.

## Fix landscape (all gated; none applied)
- **Durable (rebuild):** ensure containerized OVN chassis get a v6 address on the data-tenant
  plane (carve/assign, not auto-pick) so `ovn-encap-ip` is v6 cloud-wide; OR pin the geneve
  encap family consistently. This is the real fix and belongs in the rebuild bundle/addressing.
- **Live dc0 workaround (to complete the smoke test only):** add a v6 data-tenant address to
  the 3 control-node OVN-chassis containers and set `ovn-encap-ip` to it (family-consistent
  with compute). Non-trivial (the containers have no v6 there today); throwaway-only.
- **Alternative:** flip compute encap to v4 to match control -- REJECTED direction (inverts the
  geneve-over-v6 design intent, D-101).

## 2026-08-09 -- v6<->v6 datapath: FIRST TEST RETRACTED (invalid instrument); STATUS = STILL UNTESTED

**RETRACTION.** A first 2026-08-09 attempt concluded "v6<->v6 geneve PROVEN NON-FUNCTIONAL" and inferred
v4-forced. **That conclusion is WITHDRAWN -- the test used an invalid packet source** (fable-advisory
catch; instrument-currency #23 sibling: the test did not bypass a layer that independently produces the
same negative). The raw measurements were real but do not support the conclusion drawn:

| # | Test | Path | Result |
|---|---|---|---|
| C1 | ping `192.0.2.10` (instance-A) | SAME chassis (compute-01), no geneve | 0% loss (4/4) |
| C2 | ping6 `2602:f3e2:f02:30::121` | v6 UNDERLAY compute-01->compute-02 | 0% loss (4/4) |
| T  | ping `192.0.2.20` (instance-B) from `ovnmeta-<net>` namespace | CROSS chassis, v6 geneve | 100% loss |
| T' | `tcpdump -ni enp4s0 udp port 6081` during T | geneve on the wire | 0 packets |

**Why T/T' prove nothing about v6 geneve.** The `ovnmeta-<net>` namespace port is an OVN **`localport`**
(how ML2/OVN implements metadata -- the same port exists on EVERY chassis). Per OVN architecture,
**traffic to/from a localport is NEVER forwarded over a tunnel** -- ovn-controller installs an explicit
drop for localport-sourced packets destined to a remote chassis. So T=100% loss and T'=0-geneve-emitted
are the **by-design** result for ANY encap family; a fully healthy v4 overlay would show the identical
T/T'. C2 (underlay) and C1 (same-chassis local delivery) are both consistent with a localport source and
do not compensate. The distinguishing test must source from a **real VM VIF port**, not a localport.

**On the "mixed-family mesh" worry:** NOT a real mechanism -- ovn-controller builds tunnels pairwise, one
per remote-chassis encap, as independent state; a v4 chassis in the SB does not poison a v6<->v6 pair's
flows. The only casualties of the family split are the cross-family (control<->compute) pairs -- that part of
the 2026-08-08 root cause **STANDS UNCHANGED**.

**CURRENT STATUS: v6<->v6 geneve datapath = STILL UNTESTED.** The "Fix landscape" below is NOT overturned.
Upstream signal (fable + general OVN knowledge): geneve-over-IPv6 is a **supported, production-exercised**
capability (OVS v6 tunnel endpoints since 2.6/2016; neutron `overlay_ip_version=6`; ovn-kubernetes runs
single-stack v6 in production). Item 14 is stale man-page wording + thin test coverage, **not** a
capability denial. Expectation: the corrected test PASSES and the v6-only direction (section 5.1 of the carve
proposal) holds -- but it must be MEASURED, not assumed.

**Corrected diagnostic sequence (bounded, ~30 min, dc0 throwaway -- do not exceed):**
1. Tunnel-port health on compute-01: `ovs-vsctl --columns=name,ofport,error list interface ovn0-vr1-d-0`
   + `ovs-appctl dpif/show | grep geneve`. `ofport>=0`/no error -> kernel made the v6 tunnel; proceed.
   `ofport:-1`/error -> real kernel/OVS v6-geneve defect; STOP and rule v4 for the rebuild.
2. THE test: VM->VM cross-chassis ping from a REAL VM (console into instance-A, ping instance-B), tcpdump
   `udp 6081` on both computes. Pass + geneve both ways -> VERDICT v6 geneve WORKS; family-split is the sole
   LB cause; v6-only fix reinstated. 0 emitted from a real VIF -> go to 3. Emitted, no reply -> underlay
   UDP-6081 filtering, check + stop.
3. Only if 2 emits nothing: `ovs-appctl ofproto/trace` from instance-A's in_port (real MAC/IP) -> instance-B.
   Ends in `output:<tunnel ofport>` -> flows fine, kernel encap failing (record, rule v4). Ends in drop ->
   capture the drop flow (that IS the transferable answer). STOP -- no kernel bisecting on a throwaway cloud.

**MTU note for the rebuild:** v6 geneve overhead is 20 bytes larger; neutron `overlay_ip_version=6` must be
set so tenant MTU computes to ~1422, else a real *large-packet* failure appears later (different bug).

### RESOLVED 2026-08-09 -- ROOT CAUSE = BRACKETED v6 `ovn-encap-ip` (charm/OVN format bug); geneve-over-v6 IS viable

Ran the corrected bounded diagnostic (Step 1 + a self-contained distinguishing test). **geneve-over-v6 is
NOT broken in this stack; a bracket-format bug prevents the tunnel from instantiating.** Measured on
`vr1-dc0-compute-01`, OVS 3.3.0 / OVN 24.03.2 / kernel `5.15.0-186` / geneve.ko 0.6 (kernel datapath):

- **Step 1 -- v6 tunnel port health:** `ovn0-vr1-d-0` (`remote_ip=[2602:f3e2:f02:30::121]`) ->
  **`ofport: -1`, `error: "bad geneve 'remote_ip'"`**. The v6 tunnel is NOT in the datapath
  (`dpif/show` shows only the 3 IPv4 tunnels). This -- not the retracted localport test -- is the valid
  evidence that cross-compute v6 geneve does not forward today.
- **Distinguishing test (throwaway bridge, cleaned up):** a geneve interface with an **UNbracketed** v6
  `remote_ip=2602:f3e2:f02:30::121` -> **`ofport: 1`, no error**; the **bracketed** `[2602:...::121]` ->
  `ofport -1`, same "bad remote_ip"; v4 control -> ofport 2, no error. **So kernel 5.15 + OVS 3.3 DO support
  IPv6 geneve endpoints; the brackets are the sole blocker.**
- **Pinpoint:** `ovs-vsctl get open_vswitch . external_ids:ovn-encap-ip` on both computes returns the value
  **bracketed** (`"[2602:f3e2:f02:30::120]"` / `"...::121]"`). The `ovn-chassis` 24.03 charm sets the v6
  encap-ip wrapped in `[...]`; that bracketed literal propagates to the OVS geneve `remote_ip`, which OVS
  rejects. `juju config ovn-chassis` exposes no encap/ip-family/source-ip lever to override it.

**VERDICT: geneve-over-IPv6 is achievable on this exact kernel/OVS/OVN; the blocker is the bracketed v6
`ovn-encap-ip` emitted by ovn-chassis 24.03.** This is a config/charm defect, NOT a kernel or capability
limit and NOT a reason to force v4.

**Consequence for the fix landscape (below) -- REINSTATED with a correction.** The original durable fix
(give containerized OVN chassis a v6 data-tenant address so encap is v6 cloud-wide) is correct AND
NECESSARY but **NOT SUFFICIENT**: the v6 encap-ip must ALSO be delivered to OVS **unbracketed**, or every
chassis (metal included) gets `ofport -1` and no v6 geneve forms. So the rebuild needs BOTH: (i) v6 on the
LXD chassis' data-tenant leg (carve, not auto-pick), and (ii) an unbracketed v6 `ovn-encap-ip` -- via a
fixed/newer ovn-chassis charm revision, or a verified post-config override. **Owed rebuild investigation:**
is the bracket a known upstream ovn-chassis/OVN bug with a fix revision? (not yet searched).

**Roosevelt-delta (corrected):** a v6 build must (a) give containerized chassis a carved v6 data-tenant
address, (b) ensure `ovn-encap-ip` reaches OVS unbracketed, and (c) gate on BOTH the encap-family
consistency AND every v6 tunnel port having `ofport >= 0` / empty `error` (the bracket bug is invisible to
a family-only check -- all chassis can be "v6" and still have `ofport -1`). v4-forced is OFF the table.

### LIVE-CONFIRMED 2026-08-09 -- unbracket fix proven end-to-end (real VM->VM, 0% loss). TESTED SEQUENCE for the rebuild.

The fix was proven live on dc0 (operator-directed: capture tested settings/steps for the 10.13 rebuild).
Exact tested sequence, run from the dc0 rack (`172.31.0.2`, D-138), all steps captured:

1. **Diagnose** (per compute): `sudo ovs-vsctl --columns=ofport,error list interface <geneve-port>` ->
   `ofport -1`, `error "bad geneve 'remote_ip'"`; `sudo ovs-vsctl get open_vswitch . external_ids:ovn-encap-ip`
   -> `"[2602:f3e2:f02:30::120]"` (bracketed = the defect).
2. **Fix** (per compute, unbracketed): `sudo ovs-vsctl set open_vswitch . external_ids:ovn-encap-ip=2602:f3e2:f02:30::120`
   (compute-01) and `...::121` (compute-02). ovn-controller rebuilt the tunnels within ~12 s.
3. **Verify tunnel:** `ovn0-vr1-d-0` -> **`ofport 14`, `error []`**, present in `dpif/show` as
   `geneve ... remote_ip=2602:f3e2:f02:30::121` (unbracketed).
4. **Verify forwarding** (`ofproto/trace` from a real VM VIF): datapath action
   `set(tunnel(ipv6_dst=2602:f3e2:f02:30::121, tp_dst=6081, geneve(...))) -> output to kernel tunnel`.
5. **Live datapath proof:** VM `geneve-proof-c` (compute-01) -> VM `geneve-proof-b` (compute-02, `192.0.2.20`):
   **8/8 received, 0% loss, rtt ~3 ms.** Real VM->VM over geneve-over-IPv6.

**CONCLUSION (measured, not inferred): geneve-over-IPv6 works on OVS 3.3.0 / OVN 24.03.2 / kernel 5.15
once `ovn-encap-ip` is delivered to OVS UNBRACKETED.** The `ovn-chassis` 24.03 charm's bracketed v6
`ovn-encap-ip` is the sole defect. `step 2` is a live WORKAROUND (transient -- a charm hook may re-assert
the bracketed value); the DURABLE rebuild fix is a fixed/newer charm revision OR a persistent post-deploy
override. Owed for the rebuild plan (change-delivery loop, not this session): (a) confirm the fixed charm
revision (charm-ovn-chassis IPv6 bracket bug family -- LP #1968355 area; not yet pinned to a fix rev);
(b) add the encap unbracketing + the two verification gates below to the phase-04/phase-01 deploy runbook;
(c) the gate must assert BOTH encap-family consistency AND every tunnel `ofport>=0`/empty `error` (a
family-only check passes while all tunnels sit at `ofport -1`).

**Also required for a v6 overlay (not re-tested here, carry forward):** the containerized control-plane
chassis (octavia/ovn-chassis-octavia LXD) still need a CARVED v6 data-tenant address (they are v4-only
today -- the original 2026-08-08 family split), AND neutron `overlay_ip_version=6` for correct tenant MTU
(~1422). The unbracket fix is necessary but only sufficient for the metal<->metal path proven here.

Teardown of `geneve-proof-*` + `lb-smoke` proceeds now (reproduction captured above). The manual step-2
encap override is left as-is on throwaway dc0 (harmless; not persisted to git/charm).

## Evidence
This session's captured output (transcript): `ovn-sbctl show`, `ovs-vsctl list interface`
(bfd_status empty), octavia health-manager log ConnectTimeout, ping6 100% loss, SG rule list,
`ip addr` on octavia/leader (eth2 v4-only on data). Full changelog at session close.