diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index e793cd4..143ca49 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -518,6 +518,23 @@ question from the rack side. `repo-lint`/gauntlet ON voffice1 remain deliberately deferred until precondition 0.1 advances that 105-commit-stale clone -- running them today would measure a stale tree. + **R3 RULED 2026-07-27 -- exact utterance "Raise the two lagging segments to 9000 + (Recommended)"**, recorded as a D-101 RULING NOTE (D-102 is merged into D-101 and directs + amendments there). **The question was re-framed by measurement before it was put:** + `scripts/dc-dc-mtu-geneve-budget.sh` had NEVER been run to a recorded verdict despite + D-101 calling the measured underlay MTU a Phase-0 gate. Run both ways this session + (capture `docs/audit/mtu-budget-20260727.txt`): underlay 9000 -> tenant MTU stays 1500; + underlay 1500 -> tenant MTU 1444 requiring ovn geneve + tenant-network + amphora to agree + permanently, which NOTHING in this repo checks. Measured underlay: every vcloud MESH leg + is ALREADY 9000 including the inter-DC `virbr5`, as are all six plane bridges on both + racks; the four 1500 legs are the D-125 SIMULATED-ISP uplinks and must STAY 1500. Exactly + TWO segments lag -- the rack transit NIC `enp1s0` in both containment VMs, and all 17 + MAAS VLAN records (the silent one: MAAS renders VLAN MTU into node netplan, so a jumbo + bridge under a 1500 record still yields 1500 node interfaces). Coupled to R2: the 56-byte + budget overhead is the IPv6 figure and applies BECAUSE dual-stack was ruled (v4-only would + have been 42 / 1458). Execution is a SEPARATE gated step; the verification owed is a + BEHAVIOURAL large-frame test with DF set across the inter-DC path, not a reading of + interface MTUs. - Position inside Stage 3: deploy step A EXECUTED 2026-07-19 (6/0/6 exact; convergence zero -- `docs/audit/outer-plan-20260719-postA-converged.txt`). **Deploy step B diff --git a/docs/audit/mtu-budget-20260727.txt b/docs/audit/mtu-budget-20260727.txt new file mode 100644 index 0000000..cb30e8f --- /dev/null +++ b/docs/audit/mtu-budget-20260727.txt @@ -0,0 +1,94 @@ +===================================================================== +D-101 MTU / geneve-over-v6 BUDGET -- first recorded verdict +Date: 2026-07-27 Branch: dc-dc-stage5-grounding-audit +Context: Stage-5 grounding audit, ruling R3. +===================================================================== + +`scripts/dc-dc-mtu-geneve-budget.sh` has existed since it was written but had +NEVER been run to a recorded verdict (audit finding L2-5). D-101 states "The +measured underlay MTU is a Phase-0 gate -- do not assume jumbo." This capture is +that measurement. + +--------------------------------------------------------------------- +1. MEASURED UNDERLAY -- vcloud host bridges +--------------------------------------------------------------------- + $ virsh net-list --name | while read n; do br=$(virsh net-info "$n" | awk '/Bridge:/{print $2}'); + ip -o link show "$br"; done + + mesh-vr1-dc0-office1 virbr7 mtu=9000 + mesh-vr1-dc0-vr1-dc1 virbr5 mtu=9000 <- THE INTER-DC PATH + mesh-vr1-dc1-office1 virbr3 mtu=9000 + office1-local virbr2 mtu=9000 + office1-wan virbr11 mtu=1500 <- simulated ISP, CORRECT at 1500 + vr1-dc0-uplink virbr4 mtu=1500 <- simulated ISP, CORRECT at 1500 + vr1-dc1-uplink virbr1 mtu=1500 <- simulated ISP, CORRECT at 1500 + wan virbr0 mtu=1500 <- simulated ISP, CORRECT at 1500 + +IMPORTANT: the four 1500 legs are the D-125 simulated-ISP uplinks. They MODEL +THE INTERNET and must STAY 1500 -- D-125's egress isolation gate is defined +against them. They are not part of the jumbo underlay and are not a defect. + +--------------------------------------------------------------------- +2. MEASURED UNDERLAY -- rack side (both DCs) +--------------------------------------------------------------------- + Six plane bridges per rack (virbr1..virbr6): mtu=9000 [audit finding L2-5] + Rack transit leg enp1s0: mtu=1500 <- LAGGING SEGMENT 1 + +--------------------------------------------------------------------- +3. MEASURED -- MAAS VLAN records +--------------------------------------------------------------------- + All 17 VLANs report mtu 1500, including all ten named plane fabrics and both + metal-admin/boot fabrics. <- LAGGING SEGMENT 2 + + This is the one that bites silently: MAAS renders the VLAN MTU into node + netplan, so a jumbo BRIDGE under a 1500 MAAS record still yields 1500 node + interfaces. Jumbo bridges alone do not deliver a jumbo underlay. + +--------------------------------------------------------------------- +4. BUDGET SCRIPT VERDICTS -- both branches, run this session +--------------------------------------------------------------------- + $ bash scripts/dc-dc-mtu-geneve-budget.sh --underlay-mtu 1500 + geneve-over-v6 overhead (D-101: 40 IPv6 + 8 UDP + 8 Geneve base): 56 bytes + Underlay is NOT jumbo (< 9000). + Per D-101: tenant_mtu = underlay_mtu - 56 = 1500 - 56 = 1444 + Recorded decision: underlay_mtu = 1500; tenant MTU = 1444 (v6-geneve), + MUST be set consistently across ovn geneve, tenant-network MTU, and amphora + VERDICT: PASS + MTU-BUDGET: underlay=1500 jumbo=no tenant_mtu=1444 verdict=PASS + + $ bash scripts/dc-dc-mtu-geneve-budget.sh --underlay-mtu 9000 + Underlay is jumbo (>= 9000). + Per D-101: tenant MTU stays 1500; geneve/amphora headroom is generous. + VERDICT: PASS + MTU-BUDGET: underlay=9000 jumbo=yes tenant_mtu=1500 verdict=PASS + + NOTE: the 56-byte overhead is the IPv6 figure, and it applies BECAUSE R2 ruled + dual-stack. Under a v4-only deployment the overhead would have been 42 and the + 1500-underlay tenant MTU would have been 1458. R2 and R3 are coupled. + + NOTE ON THE SCRIPT: invoked bare it correctly REFUSES -- "FAIL: --underlay-mtu + is REQUIRED (no default -- measure it this session)". It does not guess. That + is the assert-on-content discipline working. + +--------------------------------------------------------------------- +5. VERDICT AS RULED (R3, 2026-07-27) +--------------------------------------------------------------------- +Operator selection, exact utterance: "Raise the two lagging segments to 9000 +(Recommended)". + +Target state: underlay jumbo END TO END, tenant MTU stays 1500, no per-charm MTU +coordination required. + +Work implied (each a gated mutation, NOT authorised by the ruling): + a. Raise `enp1s0` MTU to 9000 inside BOTH containment VMs (vvr1-dc0, vvr1-dc1), + persistently -- a netplan/config change, not just `ip link set`. + b. Raise the MAAS VLAN MTU records to 9000 on the plane fabrics, so MAAS renders + jumbo into node netplan. + c. LEAVE the four simulated-ISP uplink legs at 1500 (see section 1). + d. Re-run this budget script with `--underlay-mtu 9000` and capture the verdict + as the closing evidence. + +Verification owed after the change: an end-to-end large-frame test across the +inter-DC path (dc0 rack -> virbr5 -> dc1 rack) with DF set, not merely a reading +of the interface MTUs. An interface claiming 9000 is not proof a 9000-byte frame +survives the path. diff --git a/docs/audit/queued-rulings-20260727.md b/docs/audit/queued-rulings-20260727.md index f195f6f..18e1e27 100644 --- a/docs/audit/queued-rulings-20260727.md +++ b/docs/audit/queued-rulings-20260727.md @@ -196,7 +196,19 @@ - **(b)** Accept a 1500 underlay and pin ~1444 consistently across ovn geneve, tenant-network MTU and amphora, set explicitly in the bundle. -OPERATOR UTTERANCE: +OPERATOR UTTERANCE: **"Raise the two lagging segments to 9000 (Recommended)"** -- +RULED 2026-07-27. **R3 IS CLOSED.** Recorded as a D-101 RULING NOTE (D-102, the +original MTU sub-policy, is merged into D-101 and directs amendments there). +**The question was re-framed by measurement before it was put:** the budget script +had never been run to a verdict, and running it showed the jumbo branch is nearly +complete already -- every vcloud MESH leg including the inter-DC `virbr5` is 9000, +as are all six plane bridges on both racks. Only TWO segments lag: the rack transit +NIC `enp1s0` inside both containment VMs, and the 17 MAAS VLAN records. The four +1500 legs are the D-125 simulated-ISP uplinks and must STAY 1500. Capture: +`docs/audit/mtu-budget-20260727.txt`. Coupled to R2: the 56-byte overhead is the +IPv6 figure and applies because dual-stack was ruled. Execution is a separate gated +step; the verification owed is a behavioural large-frame test with DF set across the +inter-DC path, not a reading of interface MTUs. --- diff --git a/docs/design-decisions.md b/docs/design-decisions.md index 24ead90..3a21eae 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -2140,6 +2140,62 @@ them; see the corrected Effect above. Nothing further is owed from the operator on the address values. +### RULING NOTE 2026-07-27 -- D-101 MTU sub-policy RESOLVED to the jumbo branch + +**Status:** RULED 2026-07-27 (operator, GA-R5). D-102 (the original MTU sub-policy) is +MERGED INTO D-101 and its body says "amend D-101 instead", so this note lands here. + +Question as presented, verbatim: "R3 -- MTU budget. The underlay is already jumbo on every +mesh and plane bridge; only two segments lag (the rack transit NIC inside both containment +VMs at 1500, and all 17 MAAS VLAN records at 1500). The budget script verdicts: underlay +9000 -> tenant MTU stays 1500; underlay 1500 -> tenant MTU 1444, which must then be set +consistently across ovn geneve, tenant-network and amphora. Which branch?" Options +presented: (a) raise the two lagging segments to 9000; (b) accept 1500 and pin tenant MTU +to 1444; (c) raise the MAAS VLANs only, leaving the transit at 1500. Operator selection, +exact utterance: **"Raise the two lagging segments to 9000 (Recommended)"**. + +**MEASURED FIRST -- capture `docs/audit/mtu-budget-20260727.txt`.** D-101 requires that +"the measured underlay MTU is a Phase-0 gate -- do not assume jumbo", and +`scripts/dc-dc-mtu-geneve-budget.sh` had existed without ever being run to a recorded +verdict (audit finding L2-5). It has now been run both ways. The audit's original framing +("jumbo bridges, 1500 VLANs, neither branch") was correct but understated how nearly done +the jumbo branch already is: +- Every vcloud MESH leg is already 9000, including `mesh-vr1-dc0-vr1-dc1` (virbr5), the + inter-DC path itself. `office1-local` is 9000. All six plane bridges on both racks are 9000. +- The four 1500 legs are the D-125 SIMULATED-ISP uplinks (`vr1-dc0-uplink`, + `vr1-dc1-uplink`, `office1-wan`, `wan`). They MODEL THE INTERNET, D-125's egress + isolation gate is defined against them, and **they must STAY 1500**. They are not a defect. +- Exactly TWO segments lag: the rack transit NIC `enp1s0` inside both containment VMs, and + the MAAS VLAN MTU records (all 17 read 1500). + +**Why the MAAS record is the one that bites silently:** MAAS renders the VLAN MTU into node +netplan, so a jumbo BRIDGE beneath a 1500 MAAS record still yields 1500 node interfaces. +Jumbo bridges alone do not deliver a jumbo underlay. + +**Rationale for (a).** Tenant MTU stays 1500 -- the ordinary case -- so no per-charm MTU +coordination is needed at all. Option (b) required ovn geneve, tenant-network MTU and +amphora to agree exactly and permanently across both DCs, and NOTHING in this repo checks +MTU (neither `cloud-assert.sh` nor `provider-bundle-check.py` has any MTU assertion), so a +drift between those three would be silent -- D-101 itself names this "the classic +nested-OpenStack failure mode". Option (c) was refused as the worst of the three: it looks +jumbo at the plane layer while the transit remains capped, so D-108 `ceph-rbd-mirror` +replication and radosgw multisite are throttled by a chokepoint that is invisible where an +operator would look for it. + +**COUPLING TO R2, stated because it changes the arithmetic:** the 56-byte overhead in the +budget is the IPv6 figure, and it applies BECAUSE R2 ruled dual-stack. Under a v4-only +deployment the overhead would have been 42 and the 1500-underlay tenant MTU 1458. R2 and +R3 are not independent. + +**Execution is a SEPARATE gated step, NOT authorised by this ruling.** Four items, in +`docs/audit/mtu-budget-20260727.txt` section 5: raise `enp1s0` persistently in both +containment VMs; raise the MAAS VLAN MTU records on the plane fabrics; LEAVE the ISP uplink +legs at 1500; re-run the budget with `--underlay-mtu 9000` and capture the closing verdict. +**Verification owed is behavioural, not declarative:** an end-to-end large-frame test with +DF set across the inter-DC path (dc0 rack -> virbr5 -> dc1 rack). An interface claiming +9000 is not proof a 9000-byte frame survives the path -- that is the assert-on-content rule +applied to MTU. + **What IS owed, all mechanical, no ruling required:** propagate the ratified v6 carve into `scripts/lib-net.sh` (a v6 arm per DC, keyed the same way the v4 `PLANE_CIDRS` arms are), and carve the twelve plane prefixes into MAAS as subnets on their existing fabrics. Both