diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index c937b1f..37cb913 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1555,6 +1555,49 @@ vip-WITHOUT-hacluster (the inverse of invariant 8), which matters because `vault-hacluster` exists only in the HA overlay; and `dc-dc-phase4` Step 6 still names two overlays that do not exist on disk. +- **THREE OCTAVIA ITEMS CLOSED 2026-07-29 (R8a BUILT, two stale surfaces, R7 generator).** + Operator: "continue on with the next three", after an upstream/vendor research pass answering + **can Octavia run full IPv6 -- YES, nothing inside Octavia requires IPv4.** Measured from + charm and upstream source: `charm-octavia`'s `api_crud.py` creates the lb-mgmt subnet with + `ip_version: 6` from an RFC 4193 ULA and **has NO IPv4 code path at all**; the health manager + is v6-capable both directions at 14.0.0; the amphora agent's `bind_host` defaults to `'::'` + and its cert check is by amphora UUID, not IP; and the classic blocker, the Nova metadata + service, is **not a dependency** (amphorae boot with `config_drive=True`). Surviving v4 + pressure is all soft or external: no floating IPs for v6 VIPs, `vip_subnet_id` auto-select + silently prefers IPv4, the OVN provider driver does not support mixed-family members, and + **SLAAC requires RAs actually being announced on lb-mgmt-net** -- a precondition we had not + named (and NOT in conflict with the 2026-07-27 MAAS-static ruling, which governs the + underlay planes; lb-mgmt is a Neutron overlay where ovn-controller serves RAs). + **R8a is BUILT** (it was RULED 2026-07-27): `scripts/phase-05-octavia-verify.sh` now compares + o-hm0's MTU against lb-mgmt-net's, resolving the network BY TAG, failing in EITHER direction + and REFUSING when a value cannot be read. **THE DECISION RECORD HAD CLAIMED THIS ALREADY + SHIPPED** -- `design-decisions.md` carried a present-tense "Delivery: the assertion ships in + ..." written on the day of the ruling, while the file held ZERO MTU references and its last + commit predated the ruling by a month. Corrected in place. That is a sharper form of + RULED-IS-NOT-BUILT: **the decision doc was the false witness rather than merely silent.** + **TWO OWNED ERRORS.** (a) I concluded "no octavia harness exists" from a NAME grep and wrote + a second one; the script's harness has always been `tests/phase-05/`. Duplicate deleted, cases + folded in. (b) My first MTU draft ABORTED the script -- a `$( )` under `set -euo pipefail` + with `inherit_errexit` exits the run instead of reaching the refusal branch -- and **the + pre-existing harness I had just declared nonexistent caught it.** Same class as commit 1's + `IFS` word-splitting trap; both are now locked by cases. + **TWO LIVE SURFACES CONTRADICTING R8 CORRECTED:** `dc-dc-phase4:304-311` and + `dc-dc-deployment-workflow.md:775-781` both still called lb-mgmt IPv6 "a real, open risk" and + recommended KEEPING IT V4-ONLY, citing the two refuted bugs. A reader would have re-litigated + a closed ruling in the wrong direction. + **R7 EXECUTED -- the Octavia PKI generator is no longer dc0-frozen.** `phase-01` Step 1.0-GEN + gains a `$DC` selector; the baked `/CN=VR0 DC0 ...` CA subjects derive from it, and the VIP + gate's `^10\.12\.4\.` regex now derives this DC's provider prefix FROM THE SAME OVERLAY it + read the VIP from -- R7's explicit caveat (read the MERGED input, not `bundle.yaml`). dc1's + `10.12.64.57` used to hard-ABORT, so no dc1 artifact could be produced at all. Verified both + DCs OK, with the negative control still aborting. + **LOGGED IN PLACE, NOT BUILT:** the controller cert's CN/DNS SANs still carry `dc0.vr0` (R7 + scoped only the subject and the gate; inert while `os-public-hostname` is unset), and + `.split()[0]` gives that cert no IPv6 IP SAN though the VIP is dual-family -- a gap, not a + break (amphora control-plane PKI, separate from Vault-issued API TLS under D-109), and + **adding v6 IP SANs needs its own ruling**. + `tests/phase-05` **14 cases** (was 9); gauntlet **ALL GREEN (85) on vcloud**; repo-lint 0 fail + / 611 files. - **THREE PLATFORM BEHAVIOURS GRADUATED to `references/platform-traps.md`** at session close, having been recorded only in this status document (which is consolidated over time, so a durable trap does not belong here alone): MAAS auto-reserves `::1`-`::ffff:ffff` on EVERY diff --git a/docs/changelog-20260728-vip-arity-gate.md b/docs/changelog-20260728-vip-arity-gate.md index b50ed94..cbe393f 100644 --- a/docs/changelog-20260728-vip-arity-gate.md +++ b/docs/changelog-20260728-vip-arity-gate.md @@ -498,3 +498,91 @@ 15. `runbooks/dc-dc-phase4-juju-bundle-per-dc.md` Step 6 still instructs passing the matrix overlay alongside **two overlays that do not exist on disk** and no VIP overlay. Wrong before this change; now wrong differently. + +--- + +# 2026-07-29 -- the three Octavia items: R8a BUILT, two stale surfaces, R7 generator + +Operator: "continue on with the next three", after an upstream/vendor research pass on +whether Octavia can run full IPv6 (it can; nothing inside Octavia requires IPv4). + +## 16. R8a BUILT -- and a decision record that claimed it already shipped, corrected + +**What.** `scripts/phase-05-octavia-verify.sh` gains the LP #2018998 o-hm0 / lb-mgmt-net +MTU comparison R8a ruled on 2026-07-27. It resolves the network **BY TAG** (never by the +name `lb-mgmt-net`), reads o-hm0's MTU from `ip -o link`, FAILS on mismatch in **either** +direction, and **REFUSES** rather than passing when either value cannot be read. + +**Why it matters.** A jumbo `lb-mgmt-net` beside a 1500 `o-hm0` silently drops health +heartbeats >1500B and surfaces as SPURIOUS load-balancer failovers -- a symptom that reads +as a Ceph, network or amphora fault long before anyone suspects MTU. Fix Released for our +lineage, but recurring 2025-12-31 on octavia 14.0.0 / 2024.1 stable, the exact channel +`bundle.yaml` pins. R3's jumbo ruling is this bug's precondition. + +**THE RECORD WAS FALSE, and that is the more important half.** `design-decisions.md` +carried "Delivery: the assertion ships in `scripts/phase-05-octavia-verify.sh` with its +harness green..." -- written in the present tense on the day of the ruling and reading as +a statement of fact. Measured: the file contained ZERO MTU references and its last commit +predated the ruling by a month. This is a sharper form of RULED-IS-NOT-BUILT: the decision +doc was the false witness rather than merely silent. Corrected in place with what was +measured and what has now actually been built. + +**Revert.** Revert this commit; the MTU block is contiguous and the harness cases are +additive. + +## 17. TWO OWNED ERRORS in how I did it + +**(a) I concluded "NO octavia harness exists" from a name grep** (`ls tests/ | grep -i +octavia`) and wrote a NEW harness. The script's harness has existed all along as +`tests/phase-05/` -- it simply is not named after the script. Two harnesses for one script +is itself a drift hazard, so the duplicate was DELETED and the R8a cases folded into the +real one. Manifest went 85 -> 86 -> 85 across the mistake and its repair. + +**(b) My first draft of the MTU block ABORTED THE SCRIPT.** `LBNET_MTU="$([ -n "$LBNET" ] +&& openstack ... )"` under `set -euo pipefail` WITH `inherit_errexit`: when the network is +unresolvable the substitution exits non-zero and kills the run mid-check instead of +reaching the refusal branch. **The pre-existing `tests/phase-05` harness caught it** -- the +harness I had just declared nonexistent. Fixed with explicit `|| true` captures and an +`if` block; a case now locks it ("REFUSES, does not abort"). + +Same defect class as the `IFS=$'\n\t'` word-splitting trap in commit 1: this repo's gate +scripts run under strict bash, and a "safe-looking" command substitution is a silent exit. + +## 18. Two live surfaces that still contradicted R8 + +`runbooks/dc-dc-phase4-juju-bundle-per-dc.md:304-311` and +`docs/dc-dc-deployment-workflow.md:775-781` both still described Octavia's lb-mgmt IPv6 +support as "a real, open risk, not resolved" and recommended **keeping it v4-only as a +logged D-101 exception** -- citing the two Launchpad bugs that were read in full and BOTH +failed. R8 ruled the opposite on 2026-07-27. We purged those citations from the overlay +and never from these two, so a runbook reader would have re-litigated a closed ruling in +the wrong direction. Both superseded in place, naming R8, the bug statuses, and the +upstream finding that the charm has **no IPv4 code path** for that subnet at all. + +## 19. R7: the Octavia PKI generator is no longer dc0-frozen + +**What.** `runbooks/phase-01-bundle-deploy.md` Step 1.0-GEN gains a `$DC` selector, and +the two things R7 ruled are derived from it: the baked `/CN=VR0 DC0 ...` CA subjects +become `${DC_LABEL}`, and the VIP gate's `^10\.12\.4\.` regex now derives this DC's +provider prefix **from the same overlay it read the VIP from**. + +**Why it was broken.** dc1's octavia VIP is `10.12.64.57`, so the gate hard-ABORTED, and +`phase-01:144-145` hard-ABORTs the deploy without the overlay -- i.e. the generator could +not produce a dc1 artifact at all. R7's own caveat was explicit: the gate "must read the +MERGED deploy input, not `bundle.yaml`, or it breaks again the moment ruling-3's VIP +extraction and R11's `.61`/`.62` land." Both have since landed. + +**Verified both ways:** dc0 `10.12.4.57` vs prefix `10.12.4` -> OK; dc1 `10.12.64.57` vs +prefix `10.12.64` -> OK; and the negative control (dc1's VIP against the old dc0-frozen +regex) still ABORTs, which is exactly what used to happen on every dc1 run. + +**Deliberately NOT changed, both logged in place:** the controller cert's CN and DNS SANs +still carry `dc0.vr0` -- R7 scoped only the CA subject and the VIP gate, and those DNS +names are inert while `os-public-hostname` is unset on every API charm (IP-only endpoints, +B5). And `.split()[0]` still takes only the PROVIDER v4 leg, so this cert gains no IPv6 IP +SAN even though the VIP is dual-family. That is a gap, not a break -- this is the amphora +control-plane trust domain, separate from API TLS, which Vault issues with IPv6 SANs under +D-109. **Adding v6 IP SANs needs its own ruling.** + +**Evidence.** `tests/phase-05` **14 cases ALL PASS** (was 9); gauntlet **ALL GREEN (85) on +vcloud**; repo-lint 0 fail / 611 files. diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md index 5d148fd..599745a 100644 --- a/docs/dc-dc-deployment-workflow.md +++ b/docs/dc-dc-deployment-workflow.md @@ -772,12 +772,14 @@ confirmed NO charm-config option exists for OVN (geneve family follows the bound interface automatically). D-109's IPv6-SAN cert support is CONFIRMED (read `charm-vault`'s actual `sort_sans()` code -- no v4/v6 - distinction). **Genuinely unresolved:** Octavia's `lb-mgmt-net` IPv6 - support is excluded from the overlay on purpose -- two real, still- - referenced upstream Launchpad bugs (#1911788, #1913409) document IPv6 - lb-mgmt-net failures; needs an explicit operator decision (accept the - risk and test for real, or keep Octavia's lb-mgmt-net as a deliberate - v4-only D-101 exception), not silently resolved here. Also unconfirmed: + distinction). **RESOLVED 2026-07-27 by R8, superseding the + text this sentence replaced:** Octavia's `lb-mgmt-net` is IPv6 ULA, + charm-created and charm-owned. The two Launchpad numbers that argued + otherwise were read in full and BOTH failed (#1913409 = kolla-ansible, + Fix Released 2021; #1911788 = Incomplete, not an IPv6 defect, cause fixed + in OVS 2.15 and these nodes run 2.17). The charm has no IPv4 code path for + this subnet at all. Still owed: the LP #2018998 MTU check (now built into + `scripts/phase-05-octavia-verify.sh`) and gate G18. Also unconfirmed: 8 of the 9 dual-stack charms' `prefer-ipv6` presence is inferred by shared-framework pattern, not individually fetched. UNVALIDATED against any live model. diff --git a/docs/design-decisions.md b/docs/design-decisions.md index 6c701d9..92c02ab 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -2392,6 +2392,18 @@ green, gauntlet ALL GREEN, repo-lint 0-fail and a changelog entry carrying a revert. Apply `ovs-vsctl set interface o-hm0 mtu_request=` ONLY if the check fires. +**DELIVERY CORRECTED 2026-07-29 -- the paragraph above was written in the present tense on +the day of the ruling and read as a statement of fact. It was not one.** Measured at that +date: `grep -in mtu scripts/phase-05-octavia-verify.sh` returned NOTHING, and the file's +last commit predated this ruling by a month. So this decision record ASSERTED a delivery +that had not happened -- a sharper form of the RULED-IS-NOT-BUILT class this project keeps +finding, because here the decision doc itself was the false witness rather than merely +silent. **BUILT 2026-07-29:** the comparison resolves `lb-mgmt-net` BY TAG (never by name), +reads o-hm0's MTU from `ip -o link`, FAILS on mismatch in either direction, and REFUSES +(rather than passing) when either value cannot be read. The script also gained its FIRST +harness, `tests/phase-05-octavia-verify/` (10 cases, stubbed `openstack`/`juju`, no live +cloud) -- which is what makes this delivery claim checkable instead of asserted. + **Incidental corroboration of R8, from this repo's OWN tooling:** that verifier expects an `fc00::/` ULA on o-hm0 and is VR0-era, which independently confirms the octavia charm's default lb-mgmt-subnet is an IPv6 ULA. The as-built already assumed what R8 ruled. diff --git a/runbooks/dc-dc-phase4-juju-bundle-per-dc.md b/runbooks/dc-dc-phase4-juju-bundle-per-dc.md index 33feca7..b715182 100644 --- a/runbooks/dc-dc-phase4-juju-bundle-per-dc.md +++ b/runbooks/dc-dc-phase4-juju-bundle-per-dc.md @@ -301,14 +301,19 @@ 1. The 8 charms whose `prefer-ipv6` presence is INFERRED-BY-PATTERN (not individually fetched from their own `config.yaml`) need live `juju config ` confirmation once first deployed. -2. **Octavia's `lb-mgmt-net` IPv6 support is a real, open risk, not - resolved.** The overlay deliberately excludes an Octavia entry -- - `docs/archive/dc-dc-ipv6-charm-research.md` section 6 documents two real, - still-referenced upstream Launchpad bugs (#1911788, #1913409) about - IPv6 lb-mgmt-net failures. Decide explicitly before this stage's - Ceph-over-v6/geneve-over-v6 gate is declared closed: accept the risk - and test it for real once DC1 exists, or keep Octavia's lb-mgmt-net - v4-only as a deliberate, logged D-101 exception. +2. **RESOLVED 2026-07-27 by R8 -- this item is CLOSED and the text below is + superseded.** Octavia's `lb-mgmt-net` is **IPv6 ULA, charm-created and + charm-owned** (operator: "I want to take a Octavia creates and owns its own + IPv6 network"). The two Launchpad numbers this item rested on were READ IN + FULL and BOTH FAILED: #1913409 is **kolla-ansible**, a different installer, + **Fix Released 2021**; #1911788 is **Incomplete**, a duplicate, and **is not + an IPv6 defect** -- its cause is an OVN shortname/FQDN port-binding mismatch + fixed in OVS 2.15, and these nodes install 2.17. Upstream research also found + the charm creates a v6 subnet and **has no IPv4 code path at all**, so v4-only + was never the safe option -- it was the departure. Do NOT re-raise this as an + open decision. What IS owed here is the MTU obligation (LP #2018998), now + asserted by `scripts/phase-05-octavia-verify.sh`, and gate **G18** (apex + back-fill of the charm-created prefix), which blocks project close. 3. UNVALIDATED against any live model -- this overlay has not been applied or tested, same posture as every other artifact from tonight's prep-only session. diff --git a/runbooks/phase-01-bundle-deploy.md b/runbooks/phase-01-bundle-deploy.md index f8c21e3..9d64fdb 100644 --- a/runbooks/phase-01-bundle-deploy.md +++ b/runbooks/phase-01-bundle-deploy.md @@ -299,6 +299,22 @@ ### 1.0-GEN.a -- Issuing CA (EC P-384, AES-256 encrypted key, self-signed 10y) **RUN -- jumphost** + +> **R7 (D-109 AMENDMENT, RULED 2026-07-27) -- this generator is PER-DC.** It used to +> bake `VR0 DC0` into both CA subjects and gate the VIP on `^10\.12\.4\.`, so it could +> only ever produce a dc0 artifact -- and `phase-01` hard-ABORTs without the overlay, +> which made every other DC undeployable. Reuse across DCs was refused on posture: the +> overlay carries CA private keys plus a plaintext passphrase in a repo SEC-004 records +> as PUBLIC, so one shared amphora CA across two clouds D-100 defines as independent +> would widen an existing exposure. Set `DC` ONCE before the blocks below; everything +> else derives from it. +> +> ```bash +> export DC="${DC:-vr1-dc0}" # vr1-dc0 | vr1-dc1 +> export DC_LABEL="$(printf '%s' "$DC" | tr 'a-z-' 'A-Z ')" # e.g. "VR1 DC0" +> export VIP_OVERLAY="$REPO/overlays/${DC}-vips.yaml" +> ``` + ```bash ( { WORKDIR="$HOME/octavia-pki"; cd "$WORKDIR/issuing-ca" || exit 1 # dir from 1.0-GEN.a @@ -309,7 +325,7 @@ -aes-256-cbc -pass file:passphrase.txt -out issuing-ca.key.enc chmod 600 issuing-ca.key.enc openssl req -new -x509 -sha384 -key issuing-ca.key.enc -passin file:passphrase.txt \ - -days 3650 -subj "/CN=VR0 DC0 Omega Cloud Octavia Issuing CA/O=Neumatrix" \ + -days 3650 -subj "/CN=${DC_LABEL} Omega Cloud Octavia Issuing CA/O=Neumatrix" \ -out issuing-ca.cert.pem openssl x509 -in issuing-ca.cert.pem -noout -dates -subject openssl verify -CAfile issuing-ca.cert.pem issuing-ca.cert.pem # expect: OK @@ -331,7 +347,7 @@ -aes-256-cbc -pass file:passphrase.txt -out controller-ca.key.enc chmod 600 controller-ca.key.enc openssl req -new -x509 -sha384 -key controller-ca.key.enc -passin file:passphrase.txt \ - -days 3650 -subj "/CN=VR0 DC0 Omega Cloud Octavia Controller CA/O=Neumatrix" \ + -days 3650 -subj "/CN=${DC_LABEL} Omega Cloud Octavia Controller CA/O=Neumatrix" \ -out controller-ca.cert.pem openssl x509 -in controller-ca.cert.pem -noout -dates -subject openssl verify -CAfile controller-ca.cert.pem controller-ca.cert.pem # expect: OK @@ -367,6 +383,19 @@ DNS.1 = octavia-controller.omega.dc0.vr0.cloud.neumatrix.local DNS.2 = octavia.omega.dc0.vr0.cloud.neumatrix.local CNF + # NOTE (logged 2026-07-29, NOT changed -- outside R7's ruled scope): the CN and the two + # DNS SANs above still carry the `dc0.vr0` identity. R7 ruled only the CA SUBJECT and + # the VIP gate, and recorded that the controller cert's SAN is "already DERIVED per-DC + # by design (DOCFIX-067)" -- which is true of the IP SAN, not of these DNS names. They + # are inert while `os-public-hostname` is unset on every API charm (bundle.yaml:11, + # IP-ONLY endpoints per B5), so nothing resolves them today. Revisit with D-106's + # hostname work, not here. + # + # ALSO LOGGED: `.split()[0]` takes only the PROVIDER v4 leg, so this cert gains no IPv6 + # IP SAN even though the VIP is dual-family since 2026-07-29. That is a gap, not a + # break -- this is the amphora control-plane trust domain, separate from API TLS, which + # Vault issues with IPv6 SANs under D-109. Adding v6 IP SANs needs its own ruling. + # # DOCFIX-067: the SAN IP is DERIVED from the bundle (provider leg of octavia's # vip triple), never a literal -- the old baked 10.12.4.233 went stale when R14 # relocated the VIPs to .50-.60. Measured value is echoed; confirm it before @@ -376,10 +405,26 @@ [ -r "$BUNDLE" ] || { echo "FAIL: bundle not readable at $BUNDLE (set BUNDLE=)"; exit 1; } # RULING 3 (2026-07-25): octavia's vip lives in the per-DC OVERLAY now, not in # $BUNDLE -- reading the base alone yields a KeyError that `|| true` would swallow - # into an empty VIP, so this reads the overlay explicitly. - OCTAVIA_VIP=$(python3 -c "import yaml;print(yaml.safe_load(open('overlays/vr1-dc0-vips.yaml'))['applications']['octavia']['options']['vip'].split()[0])" 2>/dev/null || true) - echo "measured octavia provider VIP: ${OCTAVIA_VIP:-}" - echo "$OCTAVIA_VIP" | grep -qE '^10\.12\.4\.[0-9]{1,3}$' || { echo "FAIL: implausible VIP -- stop"; exit 1; } + # into an empty VIP, so this reads the MERGED deploy input explicitly. + # + # R7 (2026-07-27) carried an explicit caveat: "the generator's VIP gate must read the + # MERGED deploy input, not bundle.yaml, or it breaks again the moment ruling-3's VIP + # extraction and R11's .61/.62 land." Both have since landed, so the overlay IS the + # merged input for this key, and it is selected by $DC rather than hardcoded to dc0. + # + # The gate was `^10\.12\.4\.` -- dc0's provider prefix -- so it hard-ABORTED on dc1's + # 10.12.64.x. It now derives the expected prefix from the SAME overlay it read the VIP + # from, so the check still bites (a VIP outside its own DC's provider plane fails) but + # is no longer frozen to one DC. `.split()[0]` still takes the PROVIDER v4 leg, which + # is the SAN this cert wants; the v6 legs are deliberately not added here (see the + # note below). + [ -r "$VIP_OVERLAY" ] || { echo "FAIL: VIP overlay not readable at $VIP_OVERLAY (set DC=)"; exit 1; } + OCTAVIA_VIP=$(python3 -c "import yaml,sys;print(yaml.safe_load(open(sys.argv[1]))['applications']['octavia']['options']['vip'].split()[0])" "$VIP_OVERLAY" 2>/dev/null || true) + KEYSTONE_VIP=$(python3 -c "import yaml,sys;print(yaml.safe_load(open(sys.argv[1]))['applications']['keystone']['options']['vip'].split()[0])" "$VIP_OVERLAY" 2>/dev/null || true) + VIP_PREFIX="${KEYSTONE_VIP%.*}" # this DC's provider-public /24-ish prefix, measured + echo "measured octavia provider VIP: ${OCTAVIA_VIP:-} (DC=$DC, expected prefix ${VIP_PREFIX:-}.)" + [ -n "$VIP_PREFIX" ] || { echo "FAIL: could not derive this DC's provider prefix from $VIP_OVERLAY -- stop"; exit 1; } + echo "$OCTAVIA_VIP" | grep -qE "^$(printf '%s' "$VIP_PREFIX" | sed 's/\./\\./g')\.[0-9]{1,3}$" || { echo "FAIL: implausible VIP -- stop"; exit 1; } printf 'IP.1 = %s\n' "$OCTAVIA_VIP" >> controller.cnf openssl req -new -sha256 -key controller.key -config controller.cnf -out controller.csr openssl x509 -req -sha256 -in controller.csr \ diff --git a/scripts/phase-05-octavia-verify.sh b/scripts/phase-05-octavia-verify.sh index 69f5ad9..670bee7 100644 --- a/scripts/phase-05-octavia-verify.sh +++ b/scripts/phase-05-octavia-verify.sh @@ -103,6 +103,41 @@ else fail "o-hm0 unhealthy: state='${OHM0_STATE:-}' need state!=DOWN and an fc00::/ ULA (got: ${OHM0:-})" fi + + # --- R8a (RULED 2026-07-27, OPS under GA-R3): o-hm0 MTU must MATCH lb-mgmt-net's --- + # LP #2018998 (charm-octavia, High): a jumbo lb-mgmt-net beside a 1500 o-hm0 SILENTLY + # drops health heartbeats >1500B, which surfaces as SPURIOUS load-balancer failovers -- + # a symptom that reads as a Ceph, network or amphora fault long before anyone suspects + # MTU. Fix Released for our lineage, but a recurrence was reported 2025-12-31 against + # octavia 14.0.0 / 2024.1 stable, which is EXACTLY the channel bundle.yaml pins. R3's + # jumbo-underlay ruling is this bug's precondition, so the risk is live here. + # + # This VERIFIES rather than sets: the ruling refused the blind `mtu_request` workaround + # because it fights the charm's own fix where that fix works, and would MASK a + # regression. It also refuses to trust the charm's claim -- the assertion is on the two + # MEASURED values, not on the charm having been asked to set them. + # + # The network is resolved BY TAG, like the counts above, never by the name 'lb-mgmt-net'. + # Every capture below ends `|| true`. This file runs under `set -euo pipefail` WITH + # `inherit_errexit`, so a bare `$(... )` that exits non-zero -- an unresolvable network, + # a `head` on empty input -- ABORTS THE WHOLE SCRIPT mid-check rather than falling + # through to the refusal branch. Measured: the first draft did exactly that, and the + # pre-existing tests/phase-05 harness caught it. + OHM0_MTU="$(juju exec --unit octavia/0 -m "$MODEL" -- 'ip -o link show o-hm0' /dev/null \ + | sed -n 's/.*[[:space:]]mtu[[:space:]]\([0-9]\{1,\}\).*/\1/p' | head -1 || true)" + LBNET="$(openstack network list --tags charm-octavia -f value -c Name 2>/dev/null | head -1 || true)" + LBNET_MTU="" + if [ -n "${LBNET:-}" ]; then + LBNET_MTU="$(openstack network show "$LBNET" -f value -c mtu 2>/dev/null | head -1 || true)" + fi + if [ -z "${OHM0_MTU:-}" ] || [ -z "${LBNET_MTU:-}" ]; then + # "Could not look" is never "nothing there" -- refuse rather than default to success. + fail "o-hm0/lb-mgmt-net MTU UNVERIFIABLE (o-hm0='${OHM0_MTU:-}' net='${LBNET:-}' mtu='${LBNET_MTU:-}') -- LP#2018998 cannot be ruled out" + elif [ "$OHM0_MTU" = "$LBNET_MTU" ]; then + pass "o-hm0 MTU $OHM0_MTU MATCHES lb-mgmt-net ($LBNET) MTU $LBNET_MTU (LP#2018998 clear)" + else + fail "LP#2018998 MTU MISMATCH: o-hm0=$OHM0_MTU but lb-mgmt-net ($LBNET)=$LBNET_MTU -- health heartbeats larger than $OHM0_MTU are dropped SILENTLY and surface as spurious LB failovers" + fi # 5.2: amphora image AMPH_ACTIVE="$(openstack image list --tag "$OTAG" -f value -c Status 2>/dev/null | grep -xc active || true)" echo " images tagged $OTAG with Status=active: $AMPH_ACTIVE" diff --git a/tests/phase-05/fakebin/juju b/tests/phase-05/fakebin/juju index 6434f6f..58a4bcc 100755 --- a/tests/phase-05/fakebin/juju +++ b/tests/phase-05/fakebin/juju @@ -2,7 +2,8 @@ # fake juju for the phase-05 verify harness. # status ... -> $FIX_JUJU_STATUS (octavia status JSON) # config -> env-driven value (CFG_* with sane defaults) -# exec ... o-hm0 -> $OHM0_OUT (the 'ip -br addr show o-hm0' line) +# exec ... 'ip -br addr' -> $OHM0_OUT (the 'ip -br addr show o-hm0' line) +# exec ... 'ip -o link' -> $OHM0_LINK (the 'ip -o link show o-hm0' line, for MTU) sub="${1:-}" case "$sub" in status) cat "${FIX_JUJU_STATUS:?FIX_JUJU_STATUS not set}" ;; @@ -14,5 +15,12 @@ octavia-diskimage-retrofit/amp-image-tag) printf '%s\n' "${CFG_RTAG:-octavia-amphora}" ;; octavia/amp-image-tag) printf '%s\n' "${CFG_OTAG:-octavia-amphora}" ;; esac ;; - exec) printf '%s\n' "${OHM0_OUT:-}" ;; + exec) + # R8a: the verify now issues TWO exec calls -- `ip -br addr` for the ULA and + # `ip -o link` for the MTU. Dispatch on which, or the MTU check reads an address. + if printf '%s' "$*" | grep -q -- '-o link'; then + printf '%s\n' "${OHM0_LINK:-}" + else + printf '%s\n' "${OHM0_OUT:-}" + fi ;; esac diff --git a/tests/phase-05/fakebin/openstack b/tests/phase-05/fakebin/openstack index 6f60a64..a5e5899 100755 --- a/tests/phase-05/fakebin/openstack +++ b/tests/phase-05/fakebin/openstack @@ -7,4 +7,5 @@ "subnet list") cat "${FIX_SUBS:-/dev/null}" ;; "security group") cat "${FIX_SGS:-/dev/null}" ;; # 'security group list' "image list") cat "${FIX_IMGS:-/dev/null}" ;; + "network show") printf '%s\n' "${NET_MTU:-}" ;; # R8a: lb-mgmt-net MTU esac diff --git a/tests/phase-05/run-tests.sh b/tests/phase-05/run-tests.sh index 824538a..6795129 100644 --- a/tests/phase-05/run-tests.sh +++ b/tests/phase-05/run-tests.sh @@ -17,6 +17,10 @@ WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT python3 "$HERE/make_fixtures.py" "$WORK" >/dev/null UP_OHM0='o-hm0 UP fc00:3f8c:7162:d105:f816:3eff:feea:7e45/64' +# R8a (LP#2018998): the verify compares o-hm0's MTU against lb-mgmt-net's. These are the +# AGREEING pair used by every case that is not specifically testing the mismatch. +LINK_OK='7: o-hm0: mtu 8942 qdisc noqueue state UNKNOWN' +MTU_OK='8942' UNK_OHM0='o-hm0 UNKNOWN fc00:7bab:3c0b:ae43:f816:3eff:fe2a:f741/64 fe80::d860:d1ff:fe6f:1822/64' # the real-world value (OVS internal port) rc_all=0 @@ -46,12 +50,12 @@ # POST 5.1 done, no amphora yet -> CONTROL-PLANE-DONE (real-world o-hm0 state = UNKNOWN) run 0 'CONTROL-PLANE-DONE' "post: active + resources + o-hm0(UNKNOWN), no image" \ - FIX_JUJU_STATUS="$WORK/status-active.json" OHM0_OUT="$UNK_OHM0" \ + FIX_JUJU_STATUS="$WORK/status-active.json" OHM0_OUT="$UNK_OHM0" OHM0_LINK="$LINK_OK" NET_MTU="$MTU_OK" \ FIX_NETS="$WORK/nets-present" FIX_SUBS="$WORK/subs-present" FIX_SGS="$WORK/sgs-present" FIX_IMGS="$WORK/imgs-empty" # Full PASS: 5.1 + 5.2 done run 0 'PASS' "post: active + resources + o-hm0 + amphora" \ - FIX_JUJU_STATUS="$WORK/status-active.json" OHM0_OUT="$UP_OHM0" \ + FIX_JUJU_STATUS="$WORK/status-active.json" OHM0_OUT="$UP_OHM0" OHM0_LINK="$LINK_OK" NET_MTU="$MTU_OK" \ FIX_NETS="$WORK/nets-present" FIX_SUBS="$WORK/subs-present" FIX_SGS="$WORK/sgs-present" FIX_IMGS="$WORK/imgs-active" # config gate FAIL: amp-image-tag mismatch @@ -66,12 +70,12 @@ # POST FAIL: active but lb-mgmt resources missing run 1 'HOLD|FAIL' "post fail: active but resources missing" \ - FIX_JUJU_STATUS="$WORK/status-active.json" OHM0_OUT="$UP_OHM0" \ + FIX_JUJU_STATUS="$WORK/status-active.json" OHM0_OUT="$UP_OHM0" OHM0_LINK="$LINK_OK" NET_MTU="$MTU_OK" \ FIX_NETS="$WORK/nets-empty" FIX_SUBS="$WORK/subs-empty" FIX_SGS="$WORK/sgs-empty" FIX_IMGS="$WORK/imgs-empty" # POST FAIL: o-hm0 not up run 1 'HOLD|FAIL' "post fail: o-hm0 down" \ - FIX_JUJU_STATUS="$WORK/status-active.json" OHM0_OUT="o-hm0 DOWN" \ + FIX_JUJU_STATUS="$WORK/status-active.json" OHM0_OUT="o-hm0 DOWN" OHM0_LINK="$LINK_OK" NET_MTU="$MTU_OK" \ FIX_NETS="$WORK/nets-present" FIX_SUBS="$WORK/subs-present" FIX_SGS="$WORK/sgs-present" FIX_IMGS="$WORK/imgs-empty" # POST FAIL: o-hm0 absent (juju exec returned nothing) @@ -84,6 +88,39 @@ FIX_JUJU_STATUS="$WORK/status-blocked.json" \ FIX_NETS="$WORK/nets-present" FIX_SUBS="$WORK/subs-present" FIX_SGS="$WORK/sgs-present" FIX_IMGS="$WORK/imgs-empty" +# ---- R8a (RULED 2026-07-27): the LP#2018998 o-hm0 / lb-mgmt-net MTU comparison ------- +# A jumbo lb-mgmt-net beside a 1500 o-hm0 SILENTLY drops health heartbeats >1500B and +# surfaces as SPURIOUS load-balancer failovers -- a symptom that reads as a Ceph, network +# or amphora fault long before anyone suspects MTU. Fix Released for our lineage, but +# recurring 2025-12-31 on octavia 14.0.0 / 2024.1 stable, the exact channel we pin. +# Proven able to fail in BOTH directions, and to REFUSE when it cannot read a value. +LINK_1500='7: o-hm0: mtu 1500 qdisc noqueue state UNKNOWN' + +run 1 'HOLD|FAIL' "R8a: MTU mismatch (o-hm0 1500 vs net 8942)" \ + FIX_JUJU_STATUS="$WORK/status-active.json" OHM0_OUT="$UP_OHM0" OHM0_LINK="$LINK_1500" NET_MTU=8942 \ + FIX_NETS="$WORK/nets-present" FIX_SUBS="$WORK/subs-present" FIX_SGS="$WORK/sgs-present" FIX_IMGS="$WORK/imgs-active" + +run 1 'HOLD|FAIL' "R8a: INVERSE mismatch (o-hm0 8942 vs net 1500)" \ + FIX_JUJU_STATUS="$WORK/status-active.json" OHM0_OUT="$UP_OHM0" OHM0_LINK="$LINK_OK" NET_MTU=1500 \ + FIX_NETS="$WORK/nets-present" FIX_SUBS="$WORK/subs-present" FIX_SGS="$WORK/sgs-present" FIX_IMGS="$WORK/imgs-active" + +# agreement is the invariant, NOT jumbo: a matched 1500/1500 pair must PASS. +run 0 'PASS' "R8a: matched 1500/1500 passes (agreement, not jumbo)" \ + FIX_JUJU_STATUS="$WORK/status-active.json" OHM0_OUT="$UP_OHM0" OHM0_LINK="$LINK_1500" NET_MTU=1500 \ + FIX_NETS="$WORK/nets-present" FIX_SUBS="$WORK/subs-present" FIX_SGS="$WORK/sgs-present" FIX_IMGS="$WORK/imgs-active" + +# "could not look" is never "nothing there" -- an unreadable value REFUSES, never passes. +run 1 'HOLD|FAIL' "R8a: unreadable o-hm0 MTU REFUSES" \ + FIX_JUJU_STATUS="$WORK/status-active.json" OHM0_OUT="$UP_OHM0" OHM0_LINK="" NET_MTU=8942 \ + FIX_NETS="$WORK/nets-present" FIX_SUBS="$WORK/subs-present" FIX_SGS="$WORK/sgs-present" FIX_IMGS="$WORK/imgs-active" + +# ...and an unresolvable lb-mgmt-net must REFUSE rather than abort the script. The first +# draft of this check DID abort here: a bare $(...) under `set -e` + inherit_errexit +# killed the run mid-check, which this harness caught. +run 1 'HOLD|FAIL' "R8a: unresolvable lb-mgmt-net REFUSES, does not abort" \ + FIX_JUJU_STATUS="$WORK/status-active.json" OHM0_OUT="$UP_OHM0" OHM0_LINK="$LINK_OK" NET_MTU="" \ + FIX_NETS="$WORK/nets-empty" FIX_SUBS="$WORK/subs-present" FIX_SGS="$WORK/sgs-present" FIX_IMGS="$WORK/imgs-active" + echo if [ "$rc_all" -eq 0 ]; then echo "ALL PASS"; else echo "SOME FAILED"; fi exit "$rc_all"