Newer
Older
openstack-caracal-dc-dc / docs / audit / octavia-ipv6-research-20260727.md

Octavia lb-mgmt-net IPv6 -- UPSTREAM/VENDOR RESEARCH (2026-07-27)

Why this exists. R8 was first presented to the operator using the evidence overlays/dc-dc-ipv6-family-matrix.yaml carries in its own comments -- two Launchpad bug numbers cited as showing "upstream IPv6 lb-mgmt-net breakage". The operator declined to rule on that basis: "Check online for information about lb-mgmt and IPv6. I would like documentation and vendor specific information rather than conjecture. Research is cheap, guesswork is expensive."

That was the correct call. Both citations turn out not to support the claim, and the conclusion inverts. This file records what the primary sources actually say.


1. The two cited bugs do not say what the overlay says they say

LP #1911788 -- "IPv6 mgmt network not working, octavia can't talk to amphora"

  • Project: OpenStack Octavia Charm
  • Status: Incomplete, marked a DUPLICATE of LP #1896630 ("Need for managing /etc/hosts for containers")
  • Last activity: 15 January 2021
  • ROOT CAUSE AS DIAGNOSED IN THE BUG -- it is not an IPv6 defect. An OVN port-binding hostname mismatch: binding_host_id carried the SHORTNAME (juju-37c2ba-2-lxd-16) instead of the FQDN (...-16.maas), so binding_vif_type read binding_failed rather than ovs. OVN controller logged repeatedly: "Not claiming lport ..., chassis [FQDN] requested-chassis [shortname]". The IPv6 address in the timeout message was incidental -- the port never bound, so nothing would have worked over ANY address family.
  • THIS ROOT CAUSE IS DIRECTLY RELEVANT TO THIS DEPLOYMENT, and not as an IPv6 question. #1896630 is about OVS/OVN in LXD containers on MAAS failing FQDN resolution at deploy or reboot. This repo deploys OpenStack into LXD containers on MAAS -- 27 LXD units in the base bundle, ~55 with the HA overlay. The bug the overlay cites as an IPv6 problem is really a container-FQDN problem that this topology is exposed to regardless of address family.

LP #1913409 -- "octavia_amp_network does not support IPv6"

  • Project: kolla-ansible -- a DIFFERENT DEPLOYMENT TOOL, not Octavia and not the charm
  • Status: Fix Released
  • Fixed in kolla-ansible 12.0.0.0rc1 (2021-04-23), 11.1.0 (2021-07-29), stable/victoria (2021-05-06); status set Fix Released 2021-10-27
  • Substance: kolla-ansible was not honouring ip_version from the os_subnet module, so it always built the mgmt network as IPv4. The fix passes ip_version, ipv6_address_mode and ipv6_ra_mode through.
  • Bearing on a Juju/charm deployment: NONE. It was a bug in another installer, fixed five years ago.

2. What upstream Octavia actually says

docs.openstack.org/octavia/latest/contributor/guides/dev-quick-start.html, verbatim:

"At the present time, though IPv4 subnets are used by default for the LB Network (for example: 172.16.0.0/12), IPv6 subnets can be used for the LB Network."

Documented as supported. The docs give no IPv6-specific configuration guidance, version floor, or testing-status statement, so this is a statement of support without a statement of coverage -- worth knowing, but it is not a warning.

3. THE FINDING THAT INVERTS R8 -- the charm's DEFAULT is IPv6

LP #1897418 ("feature request: have option to use ipv4 when setting up lb-mgmt-net"), project OpenStack Octavia Charm, status Opinion, unassigned. Its description, verbatim:

"By default, Octavia charm uses ipv6 for its lb-mgmt-subnet. However, it would be nice to have the option to choose an ipv4 network from the start instead of deleting the ipv6 network and recreating the ipv4 subnet."

Two consequences, and they run opposite to how R8 was first framed:

  1. IPv6 lb-mgmt is the charm's DEFAULT. A v4-only lb-mgmt-net is therefore the DEPARTURE -- it requires deleting the charm-created IPv6 network and recreating an IPv4 subnet, which is exactly the friction this feature request exists to remove.
  2. D-101's placement of lb-mgmt in the IPv6-only ULA group is ALIGNED with the charm's own default behaviour, not in tension with it.

The maintainer response records the supported escape hatch: set create-mgmt-network: false and manage the network yourself. And a reporter on the same bug records positive operational evidence -- success "using the charm default (ipv6) network but using ipv4 everywhere else."

4. Incidental finding, relevant to the R3 MTU ruling

Surfaced while searching the charm's bug list: LP #2018998 -- "MTU mismatch between o-hm0 and lb-mgmt-net", against OpenStack Octavia Charm. Not read in full this session. It concerns the health-manager interface o-hm0 versus the lb-mgmt-net MTU, which is precisely the surface R3's jumbo-underlay ruling touches. Queued for reading before the Octavia step of Stage 5, not actioned here.

5. What this means for the audit's own evidence discipline

The overlay's comment block is careful, well-written and reaches a defensible-sounding conclusion -- and its two supporting citations do not support it. One is a mis-attributed container-FQDN bug, the other is a fixed bug in a different installer. Nothing in the repo could have caught this, because the citation is prose and no gate reads prose.

This is the same class as the two ruled-but-never-built decisions this audit found (D-134's bands, D-020's vault VIP): a plausible in-repo statement that nothing verifies. It generalises the audit's own rule -- assert on content, not existence -- to citations: a bug NUMBER in a comment is an existence claim; what the bug SAYS is the content, and only the content is evidence.


6. FULL READ + VERSION WEIGHING (operator-directed: "Read both bugs in full. Weigh the bug report dates and versioning against the current version and current installation/configuration instructions.")

Our version set, for reference: OpenStack Caracal 2024.1, octavia charm 2024.1/stable, hacluster 2.4/stable, ceph charms squid/stable, Juju 3.6.27, MAAS 3.7.2, node OS jammy 22.04.

6a. LP #1896630 -- container FQDN / /etc/hosts -- RESOLVED FOR OUR VERSION SET

Full read: reported 2020-09-22 by Drew Freiberger, affects 14 people. Task status: charm-layer-ovn Invalid (2021-05-01), Canonical Juju Triaged/Low, charm-nova-compute Fix Released, OpenStack Charm Guide Fix Released.

Mechanism, quoted from the bug: "When deploying on metal with MAAS, MAAS will add the FQDN to the localhost record in /etc/hosts so that issuing the hostname -f command will always succeed regardless of availability of the network. When deploying on the other provider combinations it is Juju that does the host initialization and Juju does not add the FQDN to the localhost record in /etc/hosts." Open vSwitch's init calls hostname -f before the network is up, so it records a SHORT hostname; a race between OVS startup and DNS resolution, with a cyclic ordering dependency (OVS before netplan, but OVS needs resolution).

Primary fix: ovs-record-hostname.service, which records the hostname on first start into the OVS database so it is stable thereafter. The bug states the fix landed in OVS 2.15 (Focal shipped 2.13). Secondary fix: charm-nova-compute stores the initial FQDN in the unit KV database on install ("Use a stable hostname to render nova.conf", February 2023).

VERSION VERDICT -- FIXED, and verified against what OUR nodes would actually install. Queried from this deployment's own dc0 mirror rather than assumed:

jammy         : openvswitch-switch 2.17.0-0ubuntu1
jammy-updates : openvswitch-switch 2.17.9-0ubuntu0.22.04.2

2.17.9 >> 2.15, so ovs-record-hostname.service is present on every node this deployment will build. The bug's era was Juju 2.7.8, charms 20.08 Ussuri, Focal/OVS 2.13; we are Juju 3.6.27, charms 2024.1, jammy/OVS 2.17.9. The charm-layer-ovn task -- the one closest to this repo's OVN usage -- was closed Invalid, i.e. the OVN charm was judged not to be where the fix belonged.

Consequence for R8: LP #1911788 (the overlay's IPv6 citation) is a duplicate of this bug. Its mechanism is fixed in the OVS our nodes install. So #1911788 has no live basis in this deployment in EITHER address family -- it cannot be used as evidence against IPv6 lb-mgmt-net here.

6b. LP #2018998 -- o-hm0 vs lb-mgmt-net MTU -- LIVE RISK ON OUR EXACT PIN, and it is COUPLED TO R3

Full read: High importance, assigned to Edward Hope-Morley, affects 6 people across multiple charm projects. Status Fix Released for master, 2023.1, Yoga and Zed; Fix Committed for Ussuri, Victoria, Wallaby, Xena (as of November 2023). Versions affected recorded as "Ussuri through 2024.1". Related: OpenStack Storyboard #2009734.

Mechanism: lb-mgmt-net was created at MTU 8942 while the health-manager interface o-hm0 stayed at 1500. Health-status messages from load balancers with many members/listeners/pools exceed 1500 bytes and are silently dropped, which triggers spurious load-balancer failovers. Evidence quoted in the bug: "UDP, length 1534 > 1500 and o-hm0 never receives them" and kernel logs "o-hm0: dropped over-mtu packet: 1744 > 1500".

Fix: commit f326ef267aa3e671b608b823bfcaf938868cba45 -- "will ensure that mtu is set whenever setup_hm_port() is called regardless of if the port has just been created or not. Also checks mtu in update-status hook". Workaround: ovs-vsctl set interface o-hm0 mtu_request=8942.

VERSION VERDICT -- NOT SAFELY CLOSED FOR US. A recurrence was reported 2025-12-31 against octavia 14.0.0 / 2024.1 stable -- which is precisely the channel bundle.yaml pins (octavia: channel: 2024.1/stable). So the fix is released in our lineage but a regression is reported on our exact version, recently.

THIS INTERACTS DIRECTLY WITH THE R3 RULING. R3 ruled the underlay be finished to jumbo (9000). This bug is exactly the jumbo-underlay failure mode for Octavia: a jumbo lb-mgmt-net beside a 1500 o-hm0 produces dropped health messages and phantom failovers -- and it fails SILENTLY, as a stability problem rather than an error. Owed at the Octavia step of Stage 5: verify o-hm0's MTU matches lb-mgmt-net's after deploy, not merely that the charm claims to set it. That is the assert-on-content rule applied to this bug's own history. Queued; not actioned here.

6c. Net effect on R8's evidence base

The overlay's two citations do not survive the full read:

  • #1913409 -- different installer (kolla-ansible), Fix Released 2021, irrelevant to a charm deployment.
  • #1911788 -- not an IPv6 defect; duplicate of #1896630, whose mechanism is fixed in OVS 2.15 and our nodes install 2.17.9. Against that, the charm's own DEFAULT for lb-mgmt-subnet is IPv6 (LP #1897418), and upstream Octavia documents IPv6 LB Networks as usable. There is no evidenced upstream barrier to IPv6 lb-mgmt-net at our version set. The one genuine, version-current Octavia network risk found by this research is the MTU coupling in 6b -- which is about MTU, not address family.

Sources