diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 6a59c22..f8b54cf 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -652,7 +652,30 @@ `bundle.yaml`, or it breaks again the moment ruling-3's VIP extraction and R11's `.61`/`.62` land.** Roosevelt analog: same "no cross-DC shared secret" principle as the per-DC MAAS power keys (SEC-012/-016). - Remaining: R8-R10 blocking, R12-R15 standing -- all measured. + **R8 RESEARCHED, NOT YET RULED -- and the evidence base INVERTED.** The operator declined + to rule on the in-repo citations ("Research is cheap, guesswork is expensive") and directed + upstream/vendor research, then a full read of two bugs weighed against current versions. + Capture: `docs/audit/octavia-ipv6-research-20260727.md`. **BOTH citations in + `overlays/dc-dc-ipv6-family-matrix.yaml` fail on inspection.** LP #1913409 is **Fix + Released (2021)** against **kolla-ansible**, a different installer -- no bearing on a charm + deploy. LP #1911788 is **Incomplete, a duplicate of LP #1896630**, and is **NOT an IPv6 + defect**: its diagnosed cause is an OVN port-binding hostname mismatch (shortname vs FQDN) + for LXD containers on MAAS, fixed by `ovs-record-hostname.service` **in OVS 2.15** -- + and this deployment's own dc0 mirror was queried to confirm our nodes install **OVS + 2.17.0 / 2.17.9-0ubuntu0.22.04.2** on jammy, so the mechanism is fixed here. Meanwhile + **the octavia charm's DEFAULT for `lb-mgmt-subnet` is IPv6** (LP #1897418, verbatim: "By + default, Octavia charm uses ipv6 for its lb-mgmt-subnet"), and upstream Octavia documents + IPv6 LB Networks as usable. **So D-101's IPv6-only placement of lb-mgmt is ALIGNED with + the charm default, and a v4-only lb-mgmt would be the DEPARTURE.** + **NEW LIVE RISK FOUND, and it is COUPLED TO R3: LP #2018998 "MTU mismatch between o-hm0 + and lb-mgmt-net"** (charm-octavia, High). A jumbo `lb-mgmt-net` (8942 in the bug) beside a + 1500 `o-hm0` silently drops health messages >1500B and triggers SPURIOUS load-balancer + failovers. Fix Released for our lineage, **but a recurrence was reported 2025-12-31 + against octavia 14.0.0 / 2024.1 stable -- the exact channel `bundle.yaml` pins.** R3 ruled + the underlay be finished to jumbo, which is precisely this bug's precondition. **OWED at + the Octavia step of Stage 5: verify `o-hm0`'s MTU MATCHES `lb-mgmt-net`'s after deploy, + not merely that the charm claims to set it.** Queued, not actioned. + Remaining: R8 (to re-present on corrected evidence), R9-R10 blocking, R12-R15 standing. - Position inside Stage 3: deploy step A EXECUTED 2026-07-19 (6/0/6 exact; convergence zero -- `docs/audit/outer-plan-20260719-postA-converged.txt`). **Deploy step B diff --git a/docs/audit/octavia-ipv6-research-20260727.md b/docs/audit/octavia-ipv6-research-20260727.md new file mode 100644 index 0000000..f7950f8 --- /dev/null +++ b/docs/audit/octavia-ipv6-research-20260727.md @@ -0,0 +1,200 @@ +# Octavia lb-mgmt-net IPv6 -- UPSTREAM/VENDOR RESEARCH (2026-07-27) + +**Why this exists.** R8 was first presented to the operator using the evidence +`overlays/dc-dc-ipv6-family-matrix.yaml` carries in its own comments -- two Launchpad +bug numbers cited as showing "upstream IPv6 lb-mgmt-net breakage". The operator +declined to rule on that basis: *"Check online for information about lb-mgmt and IPv6. +I would like documentation and vendor specific information rather than conjecture. +Research is cheap, guesswork is expensive."* + +That was the correct call. **Both citations turn out not to support the claim, and the +conclusion inverts.** This file records what the primary sources actually say. + +--- + +## 1. The two cited bugs do not say what the overlay says they say + +### LP #1911788 -- "IPv6 mgmt network not working, octavia can't talk to amphora" +- Project: **OpenStack Octavia Charm** +- Status: **Incomplete**, marked a **DUPLICATE of LP #1896630** ("Need for managing + /etc/hosts for containers") +- Last activity: **15 January 2021** +- **ROOT CAUSE AS DIAGNOSED IN THE BUG -- it is not an IPv6 defect.** An OVN + port-binding hostname mismatch: `binding_host_id` carried the SHORTNAME + (`juju-37c2ba-2-lxd-16`) instead of the FQDN (`...-16.maas`), so `binding_vif_type` + read `binding_failed` rather than `ovs`. OVN controller logged repeatedly: "Not + claiming lport ..., chassis [FQDN] requested-chassis [shortname]". + The IPv6 address in the timeout message was incidental -- the port never bound, so + nothing would have worked over ANY address family. +- **THIS ROOT CAUSE IS DIRECTLY RELEVANT TO THIS DEPLOYMENT, and not as an IPv6 + question.** #1896630 is about OVS/OVN in **LXD containers on MAAS** failing FQDN + resolution at deploy or reboot. This repo deploys OpenStack into LXD containers on + MAAS -- 27 LXD units in the base bundle, ~55 with the HA overlay. The bug the + overlay cites as an IPv6 problem is really a container-FQDN problem that this + topology is exposed to regardless of address family. + +### LP #1913409 -- "octavia_amp_network does not support IPv6" +- Project: **kolla-ansible** -- a DIFFERENT DEPLOYMENT TOOL, not Octavia and not the + charm +- Status: **Fix Released** +- Fixed in kolla-ansible 12.0.0.0rc1 (2021-04-23), 11.1.0 (2021-07-29), stable/victoria + (2021-05-06); status set Fix Released 2021-10-27 +- Substance: kolla-ansible was not honouring `ip_version` from the `os_subnet` module, + so it always built the mgmt network as IPv4. The fix passes `ip_version`, + `ipv6_address_mode` and `ipv6_ra_mode` through. +- **Bearing on a Juju/charm deployment: NONE.** It was a bug in another installer, + fixed five years ago. + +## 2. What upstream Octavia actually says + +`docs.openstack.org/octavia/latest/contributor/guides/dev-quick-start.html`, verbatim: + +> "At the present time, though IPv4 subnets are used by default for the LB Network +> (for example: 172.16.0.0/12), IPv6 subnets can be used for the LB Network." + +Documented as supported. The docs give no IPv6-specific configuration guidance, +version floor, or testing-status statement, so this is a statement of support without +a statement of coverage -- worth knowing, but it is not a warning. + +## 3. THE FINDING THAT INVERTS R8 -- the charm's DEFAULT is IPv6 + +LP #1897418 ("feature request: have option to use ipv4 when setting up lb-mgmt-net"), +project **OpenStack Octavia Charm**, status **Opinion**, unassigned. Its description, +verbatim: + +> "By default, Octavia charm uses ipv6 for its lb-mgmt-subnet. However, it would be +> nice to have the option to choose an ipv4 network from the start instead of deleting +> the ipv6 network and recreating the ipv4 subnet." + +Two consequences, and they run opposite to how R8 was first framed: +1. **IPv6 lb-mgmt is the charm's DEFAULT.** A v4-only lb-mgmt-net is therefore the + DEPARTURE -- it requires deleting the charm-created IPv6 network and recreating an + IPv4 subnet, which is exactly the friction this feature request exists to remove. +2. **D-101's placement of `lb-mgmt` in the IPv6-only ULA group is ALIGNED with the + charm's own default behaviour**, not in tension with it. + +The maintainer response records the supported escape hatch: set +`create-mgmt-network: false` and manage the network yourself. And a reporter on the +same bug records **positive operational evidence** -- success "using the charm default +(ipv6) network but using ipv4 everywhere else." + +## 4. Incidental finding, relevant to the R3 MTU ruling + +Surfaced while searching the charm's bug list: **LP #2018998 -- "MTU mismatch between +o-hm0 and lb-mgmt-net"**, against **OpenStack Octavia Charm**. Not read in full this +session. It concerns the health-manager interface `o-hm0` versus the lb-mgmt-net MTU, +which is precisely the surface R3's jumbo-underlay ruling touches. **Queued for reading +before the Octavia step of Stage 5**, not actioned here. + +## 5. What this means for the audit's own evidence discipline + +The overlay's comment block is careful, well-written and reaches a defensible-sounding +conclusion -- and its two supporting citations do not support it. One is a +mis-attributed container-FQDN bug, the other is a fixed bug in a different installer. +Nothing in the repo could have caught this, because the citation is prose and no gate +reads prose. + +This is the same class as the two ruled-but-never-built decisions this audit found +(D-134's bands, D-020's vault VIP): **a plausible in-repo statement that nothing +verifies.** It generalises the audit's own rule -- assert on content, not existence -- +to citations: a bug NUMBER in a comment is an existence claim; what the bug SAYS is the +content, and only the content is evidence. + +--- + +## 6. FULL READ + VERSION WEIGHING (operator-directed: "Read both bugs in full. Weigh the bug report dates and versioning against the current version and current installation/configuration instructions.") + +Our version set, for reference: OpenStack **Caracal 2024.1**, octavia charm **2024.1/stable**, +hacluster **2.4/stable**, ceph charms **squid/stable**, Juju **3.6.27**, MAAS **3.7.2**, +node OS **jammy 22.04**. + +### 6a. LP #1896630 -- container FQDN / `/etc/hosts` -- RESOLVED FOR OUR VERSION SET + +Full read: reported 2020-09-22 by Drew Freiberger, affects 14 people. Task status: +`charm-layer-ovn` **Invalid** (2021-05-01), Canonical Juju **Triaged/Low**, +`charm-nova-compute` **Fix Released**, OpenStack Charm Guide **Fix Released**. + +Mechanism, quoted from the bug: *"When deploying on metal with MAAS, MAAS will add the +FQDN to the localhost record in /etc/hosts so that issuing the `hostname -f` command will +always succeed regardless of availability of the network. When deploying on the other +provider combinations it is Juju that does the host initialization and Juju does not add +the FQDN to the localhost record in /etc/hosts."* Open vSwitch's init calls `hostname -f` +before the network is up, so it records a SHORT hostname; a race between OVS startup and +DNS resolution, with a cyclic ordering dependency (OVS before netplan, but OVS needs +resolution). + +Primary fix: **`ovs-record-hostname.service`**, which records the hostname on first start +into the OVS database so it is stable thereafter. **The bug states the fix landed in OVS +2.15** (Focal shipped 2.13). Secondary fix: `charm-nova-compute` stores the initial FQDN +in the unit KV database on install ("Use a stable hostname to render nova.conf", +February 2023). + +**VERSION VERDICT -- FIXED, and verified against what OUR nodes would actually install.** +Queried from this deployment's own dc0 mirror rather than assumed: +``` +jammy : openvswitch-switch 2.17.0-0ubuntu1 +jammy-updates : openvswitch-switch 2.17.9-0ubuntu0.22.04.2 +``` +2.17.9 >> 2.15, so `ovs-record-hostname.service` is present on every node this deployment +will build. The bug's era was Juju 2.7.8, charms 20.08 Ussuri, Focal/OVS 2.13; we are Juju +3.6.27, charms 2024.1, jammy/OVS 2.17.9. The `charm-layer-ovn` task -- the one closest to +this repo's OVN usage -- was closed **Invalid**, i.e. the OVN charm was judged not to be +where the fix belonged. + +**Consequence for R8:** LP #1911788 (the overlay's IPv6 citation) is a duplicate of this +bug. Its mechanism is fixed in the OVS our nodes install. **So #1911788 has no live basis +in this deployment in EITHER address family** -- it cannot be used as evidence against +IPv6 lb-mgmt-net here. + +### 6b. LP #2018998 -- o-hm0 vs lb-mgmt-net MTU -- LIVE RISK ON OUR EXACT PIN, and it is COUPLED TO R3 + +Full read: **High** importance, assigned to Edward Hope-Morley, affects 6 people across +multiple charm projects. Status **Fix Released** for master, 2023.1, Yoga and Zed; +**Fix Committed** for Ussuri, Victoria, Wallaby, Xena (as of November 2023). Versions +affected recorded as **"Ussuri through 2024.1"**. Related: OpenStack Storyboard #2009734. + +Mechanism: `lb-mgmt-net` was created at **MTU 8942** while the health-manager interface +**`o-hm0` stayed at 1500**. Health-status messages from load balancers with many +members/listeners/pools exceed 1500 bytes and are silently dropped, which triggers +**spurious load-balancer failovers**. Evidence quoted in the bug: *"UDP, length 1534 > 1500 +and o-hm0 never receives them"* and kernel logs *"o-hm0: dropped over-mtu packet: 1744 > +1500"*. + +Fix: commit `f326ef267aa3e671b608b823bfcaf938868cba45` -- *"will ensure that mtu is set +whenever setup_hm_port() is called regardless of if the port has just been created or not. +Also checks mtu in update-status hook"*. Workaround: `ovs-vsctl set interface o-hm0 +mtu_request=8942`. + +**VERSION VERDICT -- NOT SAFELY CLOSED FOR US.** A **recurrence was reported 2025-12-31 +against octavia 14.0.0 / 2024.1 stable** -- which is precisely the channel `bundle.yaml` +pins (`octavia: channel: 2024.1/stable`). So the fix is released in our lineage but a +regression is reported on our exact version, recently. + +**THIS INTERACTS DIRECTLY WITH THE R3 RULING.** R3 ruled the underlay be finished to jumbo +(9000). This bug is exactly the jumbo-underlay failure mode for Octavia: a jumbo +`lb-mgmt-net` beside a 1500 `o-hm0` produces dropped health messages and phantom failovers +-- and it fails SILENTLY, as a stability problem rather than an error. **Owed at the +Octavia step of Stage 5: verify `o-hm0`'s MTU matches `lb-mgmt-net`'s after deploy, not +merely that the charm claims to set it.** That is the assert-on-content rule applied to +this bug's own history. Queued; not actioned here. + +### 6c. Net effect on R8's evidence base + +The overlay's two citations do not survive the full read: +- #1913409 -- different installer (kolla-ansible), **Fix Released 2021**, irrelevant to a + charm deployment. +- #1911788 -- **not an IPv6 defect**; duplicate of #1896630, whose mechanism is **fixed in + OVS 2.15** and our nodes install **2.17.9**. +Against that, the charm's own DEFAULT for `lb-mgmt-subnet` is **IPv6** (LP #1897418), and +upstream Octavia documents IPv6 LB Networks as usable. **There is no evidenced upstream +barrier to IPv6 lb-mgmt-net at our version set.** The one genuine, version-current Octavia +network risk found by this research is the MTU coupling in 6b -- which is about MTU, not +address family. + +## Sources +- https://docs.openstack.org/octavia/latest/contributor/guides/dev-quick-start.html +- https://bugs.launchpad.net/bugs/1911788 +- https://bugs.launchpad.net/bugs/1913409 +- https://bugs.launchpad.net/bugs/1897418 +- https://bugs.launchpad.net/charm-octavia/+bug/2018998 +- https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/xena/app-octavia.html