diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index f53f356..b375bb8 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -89,6 +89,13 @@ > (a) control + SEC-010 successor; DEC-08/09 rack / D-131 retirement (live-re-measure-gated); DEC-24 the > dc0<->dc1 mesh + cross-DC Ceph path; the 13 owed artifacts + 9 owed live measurements. Next-free now > D-145. Scope handoff (history): `docs/audit/container-elim-pass/SCOPE-AND-EXECUTION-PLAN.md`. +> **IPv6-UNLOCK DESIGN NOTE (2026-08-10, read-only pass5, recorded under D-144):** the collapse +> unlocks little NEW IPv6 -- of a 19-surface family matrix only the DEC-15 power-dial reach is a +> clean collapse-caused v6 unlock (UNVERIFIED, DEC-15-gated); folded into D-144's owed DECs (DEC-15 +> prefer-v6-reach; DEC-14/16 the unruled D-124 transit family; DEC-24 = replication carrier + its +> open D-139 v6-route, same leg). D-139's plane matrix + geneve-over-v6 + LP#1723240 (D-141) are +> NOT unlocked by the collapse (different layers). Note: +> `docs/audit/container-elim-pass/ipv6-unlock-design-note-20260810.md`. > > **dc0 checkpoint scope (operator 2026-08-08): "activate + smoke-test"** -- networks + Octavia (1 test > LB) + Designate (1 test zone) + wrap gates (cloud-assert BOM, controller backup, verify-live diff --git a/docs/audit/container-elim-pass/ipv6-unlock-design-note-20260810.md b/docs/audit/container-elim-pass/ipv6-unlock-design-note-20260810.md new file mode 100644 index 0000000..d747990 --- /dev/null +++ b/docs/audit/container-elim-pass/ipv6-unlock-design-note-20260810.md @@ -0,0 +1,92 @@ +# IPv6-unlock design note -- what the container-layer collapse (D-144) does and does NOT open for IPv6 + +**Date:** 2026-08-10. **Status:** DESIGN NOTE (findings only, GA-R5 -- nothing ruled, nothing built). +Owed design input recorded under **D-144**. Read-only follow-on pass (pass5, 3 workers): full +surface enumeration + classification matrix in `pass5-w1-mgmt-v6.md`, `pass5-w2-metal-data-v6.md`, +`pass5-w3-reconciliation.md`. Governing posture: **D-101 -- "IPv6 unless IPv4 is NECESSARY"** (every +v4 leg is a justified exception, never a neutral default). + +## The honest headline + +**The collapse itself unlocks very little NEW IPv6.** Of a 19-row family matrix spanning every +surface the flatten touches or is adjacent to, **exactly ONE row is a clean "the collapse newly makes +this v6-possible" unlock** (the MAAS power-dial reach path) -- and even that is UNVERIFIED pending a +live measurement (DEC-15). The near-empty FLIP-TO-v6 class IS the finding: most of the cloud's IPv6 +story was already ruled by **D-139** (a week before D-144) and is independent of the containment +layer; the rest is forced-v4 by ruling. No proposed v6 flip conflicts with any standing ruling. + +## (a) Genuine unlocks the collapse creates -- the actionable design inputs + +1. **The MAAS power-dial reach path can go v6 (DEC-15).** Flattening removes the old force (dialing a + plane-bridge-bound libvirt address). The new target is vcloud's own sshd over `qemu+ssh` (a + family-agnostic transport) from `metal-admin`, which is already dual-stack. **UNVERIFIED:** whether + vcloud's sshd is v6-reachable from metal-admin at an address NOT on a plane bridge (the (a) + control's own constraint) -- this is precisely what DEC-15's reach half must measure and decide. + Design input: DEC-15 should prefer a v6 reach, consistent with D-101. +2. **The two NEW controls the collapse creates are v6-native by construction, at zero extra cost:** + the (a) cross-DC isolation control (DEC-14) and the SEC-010 transit-drop successor (DEC-16). Both + inherit SEC-010's live `table inet sec010` nftables pattern -- `inet` matches v4 AND v6 on one + interface-scoped ruleset. No v6-specific work needed; just don't regress to an `ip`-family table. +3. **DEC-24 is the ONE place the plane-family ruling and the collapse genuinely intersect.** The + `replication` plane is ruled IPv6-only (D-139) AND its cross-DC leg has **no v6 route today** + (D-139 build-constraint, still OPEN) AND D-144's DEC-24 leaves that same leg's post-flatten + `virbr5`/netem attachment UNDEFINED. So DEC-24's design must resolve BOTH the carrier attachment + AND the v6 route for cross-DC Ceph replication -- they are the same leg. Fold the D-139 v6-route + gap into DEC-24 so it is not solved twice. +4. **A pre-existing v6 GAP surfaced, now flag it to its owning DEC:** the **D-124 region<->rack + transit overlay** (`172.31.0.0/24`, and the client VM's transit leg) has **no ruling on its + family** -- it is v4 by convention/history, not by necessity. The tooling does not block v6 + (mesh-link is bare L2; SSH is family-agnostic), and a candidate v6 home is already RESERVED in the + apex model (`f00::/40` "Infrastructure / site-to-site links", D-115) but carved nowhere. This is a + pre-existing gap (predates D-144), but DEC-14/DEC-16 already govern this same leg -- so it should + be decided WITH them, not rediscovered later. Candidate DEC material; NOT resolved here (GA-R5). + +## What stays FORCED-v4 (so the unlock is not oversold) + +- **metal-admin PXE / node commissioning** -- v4 by RULING (D-101/D-139 "PXE is v4-first"; `dhcpd6` + deliberately OFF; v4 commissioning range; rack has zero global v6). IMPORTANT: this is a *ruling*, + not a measured platform ceiling -- **whether MAAS 3.7 can commission v6-only is UNVERIFIED** (no + test/capture/vendor citation on disk). Do not assert PXE-over-v6 is impossible; assert only that + this repo built and ruled a v4-first path and never tested an alternative. +- **The simulated-ISP uplink NAT** (`172.30.x`, `modules/site-wan`) -- single-stack v4 by module + construction; a deliberate VR1 rehearsal simplification (D-101 rejected NAT64/DNS64, defers + external v6 to the next full deployment). D-144 removes the WAN-bridge INDIRECTION (D-125) around + it but leaves the NAT and its family unchanged. +- **provider-public** and **metal-admin (base plane)** -- ruled DUAL-STACK (D-139): FIPs/API-VIPs + against a still-v4 internet; PXE. Both families load-bearing by ruling. + +## What is NOT unlocked by the collapse (different layers -- do not credit D-144) + +- **D-139's entire plane family matrix** (the six planes' v6-only/dual rulings, incl. the four + east-west planes ruled v6-only but still dual-carved live -- v4 removed LAST "after each is + proven", not yet done). Ruled 2026-07-31; independent of the containment layer. +- **The geneve-over-v6 rebuild fix** (unbracket `ovn-encap-ip` + carve v6 on the containerized OVN + chassis) -- an OVN/OVS build-mechanics fix on the NODE layer; gated by `scripts/geneve-encap-assert.sh`. + Its relationship to D-144 is NONE. +- **The LXD API-charm container v6 gap (juju LP #1723240 / D-141)** -- v4-active/v6-reserved, gated on + a juju upstream defect (`EthernetDeviceForBridge` picking `addrs[0]` from an unsorted query) + per-charm + fixes. **"container-elim" here means the `vvr1-dcN` CONTAINMENT VM (D-122/D-123), NOT juju's LXD + control-plane containers on the node VMs.** D-144 does not touch, unblock, or moot LP #1723240 -- + an operator might assume flattening the substrate flattens this defect away; it does not (juju, not + the hypervisor, is the constraint). + +## Roosevelt-delta + +Transfers to bare metal (v4 by protocol/juju-layer, not virtualization): the **PXE v4-first ruling** +(though a v6-only capability remains unmeasured) and **LP #1723240** (juju-layer, substrate-independent). +Does NOT transfer (virtual-only): the `qemu+ssh` dial + D-125 bridge-in (removed), the (a) control + +SEC-010 successor + the power-key mitigation (artifacts of single-hypervisor co-residency -- bare metal +has per-host BMC/IPMI + real NICs), and the simulated-ISP v4 NAT (a rehearsal simplification; Roosevelt +gets full v4+v6 edge transport per D-101). The mesh/netem carrier (DEC-24's leg) is a virtualization +shim with no bare-metal analog -- its answer must be found in VR1. + +## Bottom line for D-144's owed work + +Fold three things into the already-owed DEC design: +- **DEC-15:** prefer a v6 reach for the power dial (verify the vcloud-sshd address is v6-reachable off + any plane bridge). +- **DEC-14/DEC-16:** decide the D-124 transit-overlay family (v6 candidate `f00::/40` reserved) on the + same leg these DECs already govern; keep the controls' nftables `inet`-family (v6-native) shape. +- **DEC-24:** resolve the cross-DC replication carrier AND its v6 route together (same leg; D-139's + open v6-route gap belongs here). +Everything else in the IPv6 story is D-139/D-141 work, independent of the collapse. diff --git a/docs/audit/container-elim-pass/pass5-w1-mgmt-v6.md b/docs/audit/container-elim-pass/pass5-w1-mgmt-v6.md new file mode 100644 index 0000000..b9183d7 --- /dev/null +++ b/docs/audit/container-elim-pass/pass5-w1-mgmt-v6.md @@ -0,0 +1,88 @@ +# Pass 5 (W1) -- substrate/management-plane address-family surfaces under the flat +container-elim topology (D-144) + +**Author:** W1 worker, follow-on to the container-elim pass (D-144, RULED 2026-08-10). +**Scope:** ONLY the substrate/management-plane surfaces the flat topology REDESIGNS -- +where the collapse of `vcloud -> vvr1-dcN -> node VMs` to `vcloud -> node VMs` changes +what address family a path can carry. READ-ONLY; nothing built or executed. + +**Session-measured inputs (per tasking, used as given, not re-derived here):** (1) the +MAAS node power-dial (`qemu+ssh`) from a region VM connects to **vcloud's sshd**, not to +a libvirt endpoint bound on a plane bridge; (2) vcloud's libvirt access control is +**group-based with fine-grained polkit actions available**. Both feed DEC-15 (the +power-key mitigation, OWED, `docs/design-decisions.md:8299-8304`). + +**Governing posture (read before any family claim, per memory caveat):** D-101 +(`docs/design-decisions.md:2243`) -- "IPv6 unless IPv4 is NECESSARY", v4 is a justified +exception, never a neutral default. D-139 Ruling A (`docs/design-decisions.md:7280`, +RULED 2026-07-31) -- per DC, `provider-public` and `metal-admin` are **dual-stack** +(v4 retained + v6), the other five planes are **IPv6-only**. Neither rules on the D-124 +transit overlay or the MAAS power-dial reach path -- those are OUTSIDE the six +plane/space family matrix, which is exactly why this pass exists. + +--- + +## Table + +| # | Surface | Current family + cite | Flat-design option | What forces v4 | Tooling-support status | Governing D | +|---|---|---|---|---|---|---| +| 1 | **MAAS node power-dial reach path** (`vr1-dcN-maas-01` -> target libvirtd, `qemu+ssh`) | **v4 only, both forms.** `VIRSH_POWER_ADDRESS_FROM_OFFICE1="qemu+ssh://jessea123@172.31.0.2/system"` (`scripts/lib-hosts.sh:212`, dc0; `:246`, dc1 `172.31.0.6`) and `VIRSH_POWER_ADDRESS_FROM_DCREGION="qemu+ssh://jessea123@10.12.8.2/system"` (`:213`, dc0; `:250`, dc1 `10.12.68.2`) -- both dial the (soon-eliminated) rack/containment VM's libvirtd on a plane-bridge address. | **v6-possible, genuinely structural.** Under D-144 the target is no longer a plane-bridge libvirt endpoint at all -- it becomes vcloud's own sshd (session-measured fact above), and `vr1-dcN-maas-01` already holds a v6 address on `metal-admin` (dual-stack, D-139 Ruling A; `lib-hosts.sh:199,233` carve the region VM 2-plane, no br-ex, per `:95-100`). SSH transport is family-agnostic; `qemu+ssh://` accepts a bracketed IPv6 literal or an AAAA-resolving hostname (OpenSSH + libvirt vendor capability, not repo-specific). | **Nothing structural any more.** The historical force was "dial the address the target libvirtd is bound to on a plane bridge" -- removed by flattening (there is no more plane-bridge libvirt host to dial; vcloud's libvirtd is not itself plane-bound). | **UNVERIFIED (live).** Whether vcloud's sshd is actually reachable from the `metal-admin` plane over v6 -- and at what address, since the (a) control forbids a host address ON a plane bridge -- is exactly what DEC-15 is GATED on: *"GATED on vcloud's live polkit/libvirt config"* (`docs/design-decisions.md:8302`); pass2 states directly *"a read of vcloud's LIVE polkit/libvirt config is delivery work, not asserted here"* (`pass2-admin-report.md:166`). The qemu+ssh/OpenSSH v6-literal mechanism itself is vendor-verified; the concrete reachable address is not. | **DEC-15** (D-144 body, `docs/design-decisions.md:8299-8304`) -- "must decide REACH ... AND authorization together." | +| 2 | **D-124 region<->rack TRANSIT overlay** (`172.31.0.0/24`, carved supernet) | **v4 only.** Dedicated management/transit supernet `172.31.0.0/24`, NOT under Cloud, ratified 2026-07-16 (`docs/design-decisions.md:5044-5054`); dc0 `/30` = `172.31.0.0/30`, dc1 = a sibling `/30` per the same pattern. Carried by `opentofu/modules/mesh-link` (the D-100 dark-fiber triangle). | **v6-possible, but an OPEN/UNRULED question, not a tooling gap.** `modules/mesh-link` is a bare L2 segment with NO address family baked in (`opentofu/modules/mesh-link/main.tf`, `variables.tf` -- only `link_name`/`mtu`; addressing is entirely the consumer's, i.e. D-124's). The IPv6 IPAM apex already reserves the conceptually-matching slot: `X00::/48` = **"Infrastructure / site-to-site links"**, VR1's is `f00::/40` -> per-DC `/48`s (`docs/design-decisions.md:3751-3753`) -- structurally the right home for a transit link, but **no `f00::...` allocation has actually been carved or cited anywhere in the repo** (grep for `f00::` returns only the table definition, `docs/design-decisions.md:3753`). Under D-144 this leg's consumer moves to the client VM (D-124 AMENDS, `docs/design-decisions.md:8279-8280`); the amendment says addressing "survives" unchanged, not that family was reconsidered. | **Nothing cited.** No D-NNN states PXE, DHCP, or any protocol on this leg requires v4; it was carved v4 by convention (mirroring the v4 Edge role, D-115) at a time (2026-07-16) that predates D-139's v6-only-east-west posture. Its live traffic (per FINAL-PLAN concern-(ii), `FINAL-PLAN.md:132-147`) is operator `ssh -J` + Office1-originated flows -- SSH, again family-agnostic. | **UNVERIFIED (design), not tooling-blocked.** mesh-link (L2) and SSH (transport) both support v6 today; what's missing is a ruling to carve a v6 prefix for the `transit` NetBox role and amend D-124 to dual-stack or v6-only. Not part of any current DEC row -- a genuine NEW gap this pass surfaces. | **D-124 AMENDS** (D-144 ledger, `docs/design-decisions.md:8279-8280`); no DEC row currently owns the family question. | +| 3 | **Client-VM metal-admin leg** | **Dual-stack already, structurally.** The client VM lands on utility-band `.8` on `metal-admin` (D-134 AMENDMENT 2026-08-10, `docs/design-decisions.md:6240-6248`: `10.13.8.8` dc0 / `10.13.68.8` dc1) -- `metal-admin` is D-139 Ruling-A dual-stack (v4 retained + v6) for EVERY host on that plane, the client VM included; no plane-level exception is named for it. | **Already v6 (no flip needed).** This is not an "unlock" from the collapse -- it is inherited automatically from the standing D-139 plane-family ruling the moment the client VM is placed on `metal-admin`. | PXE/commissioning-class forces that keep `metal-admin` dual-stack (not v6-only) are plane-wide (D-139 Ruling A rationale: MAAS region API, node-DNS forwarder, the v4-only D-134 utility band, no rack with v6-only reachability) -- they justify the plane staying dual-stack, they do not force the client VM specifically off v6. | **Verified (ruled).** D-139 Ruling A is an ADOPTED, RULED family assignment for the whole plane -- no live measurement owed for the family itself (carve mechanics at build time are a separate, ordinary Stage-3/4 concern). | **D-139 Ruling A**; **D-134 AMENDMENT 2026-08-10** (octet). | +| 4 | **Client-VM transit leg + how MAAS reaches/powers via it** | v4, same `172.31.0.x/30` scheme as row 2 (D-124 AMENDS: "Scheme-A addressing survives on the client VM," `docs/design-decisions.md:8279-8280`). | Same as row 2 (v6-possible, unruled) -- **BUT do not conflate this leg with row 1.** The client VM's transit leg carries operator `ssh -J` / Office1-originated flows and is the concern-(ii) SEC-010-successor endpoint (`FINAL-PLAN.md:132-147`); **MAAS's own power-dial (row 1) does NOT route through the client VM at all** in the pass's design -- `vr1-dcN-maas-01` and vcloud's libvirtd are both flat vcloud-resident objects, so DEC-15's reach question is `maas-01 -> vcloud`, never `maas-01 -> client-VM -> vcloud`. Treating the client VM as if it sits in the power-dial path would misattribute a DEC-15 finding to the wrong surface. | Same as row 2. | Same as row 2 (design-open, not tooling-blocked). | **D-124 AMENDS**; concern-(ii) endpoint disposition is **DEC-16** (D-144 body). | +| 5 | **(a) cross-DC host isolation control** (vcloud-level nftables, OWED, DEC-14) | N/A -- not yet built. Modeled on the EXISTING SEC-010 mechanism (below), which is already family-agnostic. | **v6-native by construction, VERIFIED via precedent.** SEC-010's live artifact uses `table inet sec010` (`scripts/site-headend-install.sh:299`, confirmed `nft list table inet sec010` check at `:221`) -- nftables' `inet` family is a SINGLE ruleset that matches BOTH IPv4 and IPv6 traffic, and the rule matches by `oifname`/`iifname` (interface), never by IP literal or family. Building the (a) control on the same `table inet` pattern gets v4+v6 coverage for free, with no new capability needed. | Nothing -- this is a forwarding/interface-scoped control, not an address-family-scoped one. | **Verified (repo precedent, mechanism-identical).** The `table inet` fact is a direct read of the live SEC-010 artifact; the (a) control's own concrete ruleset is still OWED (DEC-14), but the FAMILY question is answered by the pattern it will reuse. | **DEC-14** (D-144 body, `docs/design-decisions.md:8305-8308`). | +| 6 | **SEC-010 transit-drop successor** (endpoints: client VM + voffice1, OWED, DEC-16) | v4-and-v6-agnostic already -- see row 5; SEC-010's CURRENT closed artifact (`docs/security-ledger.md:21`) is `table inet sec010`, i.e. already dual-family on its existing (v4-only-addressed) interface. | **v6-native by construction, same evidence as row 5.** The planned extraction (pulling the writer out of `site-headend-install.sh`'s `node_host_setup()` into a role-agnostic subcommand installing both ends, `pass2-admin-report.md:142-147`) carries the `table inet` shape forward -- no redesign needed for the successor to cover a v6 transit leg IF/WHEN row 2/4's transit ever carries v6 traffic. | Nothing -- same interface-scoped reasoning as row 5. | **Verified (repo precedent).** Same citation as row 5. | **DEC-16** (D-144 body). | +| 7 | **MAAS region<->rack control path in the flat topology** | **Conditionally MOOT, pending DEC-08.** `vr1-dcN-maas-01` already runs COMBINED region+rack (pass2 adversarial check 2, `pass2-admin-report.md:26-46`, confirmed by direct read of both DC changelogs); D-131's own text independently states *"Co-located region+rack controllers are immune"* to the MAAS 3.7 rack-only-agent DNS defect (`docs/design-decisions.md:5705`), corroborating that region+rack already run as ONE co-located service, not two ends of a network hop. Standalone rack-controller RETIREMENT is pass2's recommendation (`pass2-admin-report.md:267-277`) but is **GATED, not yet ratified** -- DEC-08 (D-144 body) needs a current-day live re-measure of `primary_rack`/rackd state at both DCs first. | **If DEC-08 ratifies retirement: the question dissolves** -- no separate rack agent, no network link, so no address-family question to answer for VR1. **If retirement is REJECTED** (a standalone rack persists somewhere): whether **MAAS 3.7 supports a v6-only or dual-stack region<->rack RPC link** is a genuinely open, UNVERIFIED question. | N/A pending the DEC-08 branch. | **UNVERIFIED, no repo citation found either way.** Grepped `docs/design-decisions.md` + `docs/CURRENT-STATE.md` for MAAS-3.7-and-IPv6: three unrelated 3.7 facts turned up (LXD >=6.7 incompatibility, `site-headend-install.sh:60`; the rack-only-agent DNS SERVFAIL bug, D-131 / `dc-rack-net.sh:26,134`; MAAS auto-reserving `::1`-`::ffff:ffff` on every v6 `/64`, `docs/design-decisions.md:7421`) -- NONE address rack<->region transport family. **To confirm:** Canonical's MAAS 3.7 release/architecture docs (rack-registration protocol), or a live test enrolling a rack-only agent against a v6-only region address, before this branch is ever exercised. | **DEC-08** (D-144 body, `docs/design-decisions.md:8309-8311`). | + +--- + +## Summary -- genuine unlocks vs needs-verification + +**Genuine, evidence-backed unlocks (structural, caused by the collapse or by an +already-ruled ADOPTED decision -- not guesses):** +- **Row 3 -- client-VM metal-admin leg is dual-stack today**, inherited automatically + from D-139 Ruling A; nothing to build or verify for the family itself. +- **Rows 5/6 -- the (a) control and the SEC-010 successor are v6-native by construction**, + verified by direct citation of the LIVE SEC-010 artifact's `table inet` shape + (`site-headend-install.sh:299`). Building them on the same pattern costs nothing extra + for v6 coverage. +- **Row 1 -- the MAAS power-dial reach path structurally CAN become v6**, because + flattening removes the old force (dialing a plane-bridge-bound libvirt address) and + replaces it with an SSH dial to vcloud itself, over a transport (OpenSSH/qemu+ssh) that + is vendor-verified family-agnostic, from a source (`metal-admin`) that already has v6. + +**Needs live/design verification before being called real (do not build against these +yet):** +- **Row 1's concrete answer** -- whether vcloud's sshd is v6-reachable from `metal-admin` + at an address that is NOT itself on a plane bridge (satisfying the (a) control) -- is + explicitly what DEC-15 is gated on; this pass did not and could not resolve it + read-only. +- **Rows 2/4 -- the D-124 transit overlay's family is an open ruling, not a tooling + gap.** mesh-link (L2) and SSH (transport) both already support v6; what's missing is a + NetBox carve (candidate slot: VR1's `f00::/40` "Infrastructure / site-to-site links", + currently unassigned anywhere in the repo) and a D-124 amendment. This is a NEW gap this + pass surfaces -- it is not currently owned by any DEC row in the D-144 body. +- **Row 7 -- moot if DEC-08 ratifies rack retirement (the likely outcome per pass2's own + recommendation), otherwise genuinely UNVERIFIED against MAAS 3.7's own capabilities**, + with no repo or vendor citation found either way. + +**Top risks carried forward:** +1. Do not let DEC-15 get ruled assuming v6 reach "just works" because SSH is + family-agnostic -- the actual blocker is vcloud's live reachable address, unmeasured. +2. Do not let the D-124/transit v6 question get silently dropped because it isn't a DEC + row today -- flag it explicitly when DEC-14/DEC-16 (which touch the same leg) are + ruled, so a future session doesn't have to rediscover the `f00::/40` slot from scratch. +3. Row 7's MAAS-3.7-v6 claim must never be asserted from this document alone -- it is + UNVERIFIED by design; the DEC-08 retirement path is the one this pass recommends + relying on instead. + +**Verification note.** Direct reads this session: `docs/audit/container-elim-pass/FINAL-PLAN.md` +(full), `pass2-admin-report.md` (full), `scripts/lib-net.sh` (full), `scripts/lib-hosts.sh` +(full), `docs/tool-index.md` (full); `docs/design-decisions.md` D-144 (`:8210-8330`), D-124 +(`:4978-5070`), D-101 (`:2243-2340`), D-139 (`:7280-7340`), D-134 amendment 2026-08-10 +(`:6240-6248`), D-128 (`:5353-5397`, incl. its D-144 amendment header), D-115 (`:3723-3810`), +D-126 (`:5172-5250`), D-131 co-location note (`:5700-5710`); `docs/security-ledger.md` SEC-010 +row (`:21`); `scripts/site-headend-install.sh` (`nft`/`inet` greps, `:219-314`); +`opentofu/modules/mesh-link/*.tf` (full); repo-wide greps for `f00::`, `MAAS 3.7`+ipv6, +`region+rack`, `polkit`, `vcloud.*ssh`. READ-ONLY; no mutation; nothing executed against the +cloud. diff --git a/docs/audit/container-elim-pass/pass5-w2-metal-data-v6.md b/docs/audit/container-elim-pass/pass5-w2-metal-data-v6.md new file mode 100644 index 0000000..5fe5a5c --- /dev/null +++ b/docs/audit/container-elim-pass/pass5-w2-metal-data-v6.md @@ -0,0 +1,181 @@ +# Pass-5 W2 -- METAL + TENANT/DATA plane address families under the flat Option-1 topology + +**READ-ONLY planning follow-on to D-144 (container-layer elimination).** Scope: the METAL +(metal-admin/PXE) and TENANT/DATA planes (the six D-052/D-101/D-139 planes + the uplink NAT + +lb-mgmt) -- the surfaces the D-144 flatten does **NOT** redesign. D-144 re-homes these planes +onto vcloud libvirt with **zero body changes to family/CIDR/MTU** +(`FINAL-PLAN.md:29-31`: *"plane CIDRs/families/MTU (D-139/D-143 own the values -- the removal +changes NO byte budget)"*). This document enumerates what family each plane carries TODAY (built) +and under RULING (target), what forces v4 where it persists, and separates two items that are +**NOT touched or unlocked by the collapse**: the LXD API-charm container v6 gap, and the +geneve-over-v6 rebuild items. + +**A note on "current family" citations, honestly stated up front:** `scripts/lib-net.sh` +(`PLANE_CIDRS`/`PLANE_NAME`, lines 24-32) carries **v4 literals only** -- there is no v6 arm in +the file today. D-139's own execution list (design-decisions.md:7442) still owes *"update +`lib-net.sh`'s v6 arm and its harness"*. So every v4 cell below cites `lib-net.sh` directly; every +v6 cell cites the D-139 ruling tables / CURRENT-STATE's measured build state instead, because +there is nothing else on disk to cite. + +**A note on current-vs-ruled, also stated up front:** D-139 ruling A's four now-v6-only planes +(`metal-internal`, `data-tenant`, `storage`, `replication`) are **currently dual-carved in MAAS** +-- the v4 subnets are removed **LAST**, "after each is proven" (design-decisions.md:7443). So a +live `maas admin subnets read` today will show v4 present on all six planes; the family column +below is the **ruled target**, with a build-status note distinguishing it from the live state. + +--- + +## 1. metal-admin -- the PXE/commissioning + DHCP plane + +| Item | Value | +|---|---| +| Current family (lib-net) | `10.12.8.0/22` (`lib-net.sh:26`), gw `10.12.8.1` at vr0-dc0 only -- **VR1 has NO router on this plane, measured** (`lib-net.sh:135-146`: `maas admin subnets read` returns gateway `none`, `ping 10.12.8.1` 100% loss / incomplete ARP, both DCs) | +| Ruled family | **dual-stack** (D-139 ruling A, design-decisions.md:7327) -- v4 RETAINED + v6 GUA added (`f0X:20::/60` parent, ruling B, :7357) | +| forced-v4 / already-v6 / dual | **dual, by ruling** -- but the PXE/DHCP sub-function inside it is **v4-only by design**, not merely v4-heavy | +| What forces the v4 half | Three independent, repo-stated reasons (D-139 ruling A's own framing, :7308-7309): (1) PXE/commissioning; (2) the MAAS region API `jujud` dials for life; (3) the entire v4-only D-134 utility band (`.4` mirror / `.5` juju / `.6` MAAS region / `.7` tailscale / `.8` client -- D-134 AMENDMENT 2026-08-10, :6240); (4) "a rack with NO global v6 at all" -- confirmed by measurement, not assumption: CURRENT-STATE's 2026-07-31 sweep found `ip -6 -o addr show scope global` returns EMPTY on the rack, so the D-135 mirror, D-131 node-DNS forwarder and MAAS rack agent are v4-only by construction | +| **PXE-over-v6 capability -- VERIFY or mark UNVERIFIED (the anchor question)** | **UNVERIFIED as a platform-capability claim; RULED+BUILT as a posture.** Nothing read in this repo *measures* whether MAAS 3.7 can commission a node IPv6-only -- there is no such test, capture, or MAAS-source citation on disk. What IS on record: (a) D-101 **rules** "PXE is v4-first" (design-decisions.md:2307-2308) -- a ruling, not a measurement; (b) the D-134 node-v6-acquisition ruling (:6138-6144, operator: "MAAS static assignment is correct") leaves `dhcpd6` **off by design** -- "MAAS starts it only for a v6 dynamic range, and under this ruling none should exist" (:6151); (c) commissioning today allocates from the v4 `dynamic .201-.254` range, and every v6 `/64` carries **zero ip ranges** (CURRENT-STATE measured sweep, "ABSENT on v6" item i); (d) the rack itself has no global v6 (item ii above). **Net: this repo has built and ruled a v4-first PXE path and never built or tested a v6-only alternative to compare against -- do not import outside knowledge of MAAS's IPv6 PXE support; state the capability question as open.** | +| Tooling status | v4 carve/verify: `scripts/dc-node-carve.sh`, `scripts/carve-host-interfaces.sh` (BUILT, live). v6 carve/verify: `scripts/dc-node-v6-carve.py`, `scripts/dc-node-v6-verify.sh` (gate **G19**, header cites D-139/D-101/D-134) -- **BUILT, on disk**, but the v6 execution list items ("MAAS carve per DC", "re-carve 54 node v6 statics") are marked NOT DONE at D-139 (:7439-7443). A Stage-5-first-boot G17 v6-static assertion is PROPOSED, not adopted (:7157-7163 style, D-101 node-IPv6 note). | +| Governing D | D-101 (family + PXE v4-first rule, :2307-2308), D-139 ruling A (dual-stack retained, :7327), D-134 (utility octet band, v4-only by construction) | + +--- + +## 2. The six D-052/D-101/D-139 planes -- ruled target vs. built state + +| Plane | Current CIDR (lib-net, v4) | Ruled family (D-139 ruling A) | forced-v4/already-v6/dual | What forces it | Governing D | +|---|---|---|---|---|---| +| `provider-public` | `10.12.4.0/22` (`lib-net.sh:25`), gw `10.12.4.1` (the ONE routed VR1 plane, measured) | **dual-stack** (:7326); GUA `f0X:10::/60`, :7355 | dual | Tenant FIPs against "a still-substantially-v4 internet", external API VIPs, Octavia tenant LB VIPs, the edge default route (:7310-7311) | D-101, D-139 ruling A | +| `metal-admin` | `10.12.8.0/22` (`lib-net.sh:26`) | dual-stack (:7327) | dual (PXE sub-forces v4; see Sec 1) | see Sec 1 | D-101, D-139 ruling A | +| `metal-internal` | `10.12.12.0/22` (`lib-net.sh:27`), gw `none` (VR1) | **IPv6-only** (:7328) -- SUPERSEDES D-101's earlier "datastore east-west v4-bound" clause (:7335-7336) | already-v6 (ruled); **v4 still present live** -- removed LAST after proof (:7443) | Nothing forces v4 here per D-139's own audit: D-101's v4 lean was "a JUDGEMENT... not a protocol requirement" (:7312-7313), reversed by D-139 | D-101 (superseded clause), D-139 ruling A | +| `data-tenant` | `10.12.16.0/22` (`lib-net.sh:28`) | **IPv6-only** (:7329) -- geneve underlay | already-v6 (ruled); v4 still present live | East-west only, no external clients, on-link, no gateway needed (:7314-7315) | D-101, D-139 ruling A | +| `storage` | `10.12.32.0/22` (`lib-net.sh:29`) | **IPv6-only** (:7330) -- Ceph public | already-v6 (ruled); v4 still present live | Same as above; `ceph-mon`/`ceph-osd` are in `PREFER_IPV6_CHARMS` (:7432) | D-101, D-139 ruling A | +| `replication` | `10.12.36.0/22` (`lib-net.sh:30`) | **IPv6-only** (:7331) -- Ceph cluster incl. cross-DC leg | already-v6 (ruled); v4 still present live; **cross-DC leg has no v6 route at all** (build-constraints bullet, :7431, OPEN) | Same on-link reasoning; the cross-DC hop is the one exception D-139 itself leaves open | D-101, D-139 ruling A, D-108 (replication) | +| `lb-mgmt` | n/a (never had a v4 carve) | **IPv6-only, NEW plane** (:7332) | **two distinct v6 objects -- do not conflate (G18 warning)**: (1) octavia's **charm-created ULA `fc00::/64`** (R8-ruled, live once Octavia deploys); (2) the **D-139 apex GUA `f0X:80::/64`** (dc0 `2602:f3e2:f02:80::/64`) carved as a MAAS-underlay plane with **NO charm consumer** -- octavia binds no `lb-mgmt` space -- kept **`reserved`** per the G18 ruling (option b, CURRENT-STATE:7825 cell, RULED 2026-08-08) | Object (1): charm default, no CIDR/family option exposed by the charm. Object (2): an architectural allocation with intent recorded, deliberately unconsumed | D-101 (original ULA rule), D-139 (apex GUA carve + annotation, :7287-7296), G18 ruling (CURRENT-STATE:7825) | + +**Tooling status (all six planes):** carve/verify built (`scripts/dc-node-v6-carve.py`, +`scripts/dc-node-v6-verify.sh`, gate G19); IPAM incl. ULA-retirement bookkeeping in +`scripts/dc-plane-ipam.sh` (D-139 ruling B retires the VR1 ULA `/48` in favor of full GUA, so this +script's ULA-retirement arm is directly relevant, `dc-plane-ipam.sh:368-372` also holds the MAAS +"zero ip-ranges != zero availability" fact that overturned the original mis-diagnosed root cause). +D-139's own execution list (apex push, MAAS carve, re-carve 54 node statics, reissue Octavia v6 +SANs, `lib-net.sh` v6 arm, `lb-mgmt` VLAN/space/subnet carve, v4 subnet removal LAST) is **entered +nowhere as done** -- design-decisions.md:7437 states plainly "none of it is done." + +--- + +## 3. The uplink/ISP NAT (`172.30.x`) -- persists through the collapse, stays v4 + +| Item | Value | +|---|---| +| Module evidence | `opentofu/modules/site-wan/main.tf` -- `resource "libvirt_network" "site_wan"` takes **one** `var.cidr`, one `forward.mode="nat"`, and a **single-entry** `ips = [{ address = cidrhost(var.cidr, 1), prefix = ... }]` block. No v6 CIDR variable, no second `ips` entry, no v6 anywhere in the module -- single-stack v4 by construction, not by omission of a flag. | +| Live values | `172.30.2.0/24` (vr1-dc0, `opentofu/main.tf:382`), `172.30.3.0/24` (vr1-dc1, `:394`) -- D-115-assigned, RULED literals (D-125 addressing note, design-decisions.md:5151-5159) | +| forced-v4 / already-v6 / dual | **forced-v4**, single-stack | +| What forces it | It simulates a v4 ISP hand-off: OPNsense WAN keeps its static v4 `.2` (D-113 edge config, unchanged by D-125), and D-101's own governing rationale explicitly defers external v6: *"External IPv6 routing is deliberately NOT added this deployment; it arrives with the next full deployment"* (design-decisions.md:2300-2301) -- **NAT64/DNS64 was considered as a v6-egress simulant and REJECTED** ("a shim with NO Roosevelt analog... testing a translation path that will never run in production", :2295-2299). Roosevelt itself will have "full IPv4 and IPv6 edge transport" (operator, :2274-2276) -- VR1's v4-only edge is a deliberate rehearsal simplification, not a capability gap. | +| Effect of D-144 | **Persists unchanged in family.** D-144's own text: *"edge WAN attaches directly to the per-DC `site-wan` NAT"* (:8256-8257) -- the module and its cidr are UNCHANGED; only the WAN bridge indirection (`modules/wan-bridge`, D-125's bridge-in plumbing that existed solely to give the now-eliminated containment VM egress) is removed. D-125 itself is marked **TERMINATED by D-144** (design-decisions.md:5091-5094) but the uplink NAT it wired through is explicitly retained. | +| Governing D | D-101 (external-v6-deferred rationale), D-113 (OPNsense static WAN), D-115 (the `/24` literals), D-125 (TERMINATED by D-144, history only), D-144 (persistence + re-wire) | + +--- + +## 4. lb-mgmt (already v6) -- summary + +See Section 2 row above for the full two-object treatment. Short form: **already v6**, both +objects, but the live/consumed one is the charm-created ULA `fc00::/64` (R8); the D-139 apex GUA +`/64` is a reserved architectural placeholder with no consumer and must stay that way per the G18 +ruling -- pointing octavia at it would reopen R8. + +--- + +## 5. NOT unlocked by the collapse -- two SEPARATE layers, explicitly out of D-144's scope + +**Primary source for this separation, cited verbatim:** `docs/audit/container-elim-pass/pass0-admin-report.md:57-60` -- + +> "The `vvr1-dcN` VMs + their inner libvirtd + the inner roots + the bootstrap gate's node-host +> mode + the D-125 bridge-in plumbing... and the qemu+ssh provider dial and its D-126 keys. **It is +> NOT: the six planes, the node VMs, the DC edge, the mesh triangle, the uplink NATs, or the LXD +> API-charm containers on deployed nodes** (all of which persist; the first three re-home)." + +### 5a. The LXD API-charm container v6 gap (CURRENT-STATE ~1456-1730) + +This is a **juju-side container-provisioning defect on the deployed OpenStack NODE VMs** -- +completely different hardware/software layer from the `vvr1-dcN` containment VM D-144 eliminates. +Summary of the mechanism, as measured and recorded (CURRENT-STATE.md, the 2026-07-31 D3 block): + +- The HOST (a node VM) is fully dual-stacked: every plane bridge on it carries a global v6 + address (`br-ex 2602:f3e2:f02:10::100/64`, etc.). +- The LXD **containers** juju provisions on that host for the API charms (keystone, cinder, + glance, neutron-api, nova-cloud-controller, openstack-dashboard, ceph-radosgw, ...) come up + **v4-only**, because juju's `EthernetDeviceForBridge` (`state/linklayerdevices.go`, pinned + `v3.6.27`) takes `addrs[0]` from an **unsorted** mongo query and derives exactly ONE subnet -- + "zero family-awareness anywhere on that path" (D-139, design-decisions.md:7401-7406). This is + **LP #1723240** (`Triaged`/`Low`, open since 2017), not a VR1-specific defect. +- D-139's own ruling A response to this measurement was to make the four internal planes + **IPv6-only** (dual-stack is "NOT EXPRESSIBLE" for juju containers, so make v6 the only option + where possible, :7410-7412). `metal-admin` **stays dual-stack and carries containers**, so its + containers land on ONE family by mongo order (today v4) -- a **CARRIED RISK**, with a detection + gate ("every container's `metal-admin` leg is IPv4") explicitly **owed, not built** (:7413-7417). +- Separately, `prefer-ipv6` is currently **set false on all seven charms that declare it** + (D-101 ruling note (b), 2026-07-31) until IPv6 is proven operational end-to-end; every dual-family + VIP leg (13 apps x v6) is kept regardless. + +**D-144's relationship to this gap: NONE.** The client VM / flat-topology change touches +substrate provisioning (tofu roots, the qemu+ssh dial, the WAN bridge); it does not touch juju's +container-network-allocation code path, MAAS's v6 ip-range population, or the `prefer-ipv6` charm +config. Nothing in D-144 changes any input to LP #1723240's failure mode. + +### 5b. The geneve-over-v6 rebuild items (D-139, 2026-08-09 amendment) + +Also a **different layer** -- OVN/OVS chassis config on the deployed compute/network nodes, not +the containment VM. Confirmed viable 2026-08-09 (real VM-to-VM cross-compute ping over v6 geneve, +8/8, 0% loss on OVS 3.3.0 / OVN 24.03.2 / kernel 5.15.0-186; design-decisions.md:7960-8001). Two +conditions, both still owed at rebuild time and neither touched by D-144: + +1. Containerized OVN chassis (the octavia / ovn-chassis-octavia LXD units -- itself an instance of + the 5a auto-pick problem) must take a **carved**, not auto-picked, v6 data-tenant address. +2. `ovn-encap-ip` must reach OVS **unbracketed** -- ovn-chassis 24.03 sets it bracketed + (`"[2602:...]"`), which OVS's geneve rejects (`ofport -1`); fix is a fixed charm revision or a + persistent `ovs-vsctl` override. + +Plus `neutron overlay_ip_version=6` for the correct tenant MTU (~1422 vs 1500, v6 geneve being 20 +bytes heavier than v4 geneve). **Gate:** `scripts/geneve-encap-assert.sh` (built 2026-08-09, +harness 16/16) checks both encap-family AND per-tunnel `ofport>=0`, wired into +`runbooks/dc-dc-phase4-juju-bundle-per-dc.md` Step 12.2. **D-144's relationship: NONE** -- these +are OVN chassis / neutron config items that apply identically whether the node VMs sit under a +containment VM or flat on vcloud libvirt; the fix targets the chassis-carve and the charm +revision, not the virtualization depth. + +**Why this separation matters for the redeploy:** D-144's `FINAL-PLAN.md` two-axis discipline +(`[D-143]` value-substitution vs `[CE]` shape-change) explicitly excludes both 5a and 5b from +either axis -- `lib-net.sh carries ZERO container-elim edits` (:8323) and neither gap is named in +D-144's reconciliation ledger (:8270-8294). Crediting the flatten with fixing (or being blocked +by) either gap would be a manufactured link between two independently-tracked defects. + +--- + +## 6. Risk notes carried forward (not new findings -- surfaced because they sit at the W2 boundary) + +- **Replication's cross-DC leg** is ruled IPv6-only (Sec 2) but has **no v6 route today** + (D-139 build-constraints bullet, OPEN) **and** D-144's DEC-24 leaves the post-flatten mesh-leg / + netem attachment for that same carrier **undefined** (`FINAL-PLAN.md` / D-144 :8312-8317, + "the pass reassigned the Office1-transit legs to the client VM cleanly but left this leg's + post-flatten attachment + netem routing undefined"). This is the one point where the W2 family + ruling and the D-144 collapse genuinely intersect -- an unresolved carrier for an unresolved + route. +- **PREFER_IPV6_CHARMS reopen-per-app:** `ceph-mon`, `ceph-osd`, `mysql-innodb-cluster` are all in + this set; converting `storage`/`replication`/`metal-internal` to v6-only (Sec 2) REOPENS the + 2026-07-31 "set false on all seven" ruling note per-app as each plane converts (D-139 + build-constraints bullet, :7432-7435) -- not yet re-ruled. +- **metal-admin container-family-flip risk** (Sec 5a) has a named, owed, unbuilt detection gate. + +--- + +## Sources read in full for this pass + +`scripts/lib-net.sh` (all 264 lines); `docs/design-decisions.md` D-101 (:2243-2510ish, incl. the +2026-07-25/07-27 ruling notes), D-125 (:5089-5170, incl. the D-144 termination annotation), D-134 +(all amendments through the 2026-08-10 `.8` client-VM entry), D-139 (:7280-8001, both rulings + the +2026-08-01 reconsideration note + the 2026-08-09 geneve amendment), D-141 (header only, cross-ref), +D-144 (:8210-8329, full entry); `docs/CURRENT-STATE.md` :1440-1740 (the LXD container v6 gap, D1-D3 ++ the systematic v6 sweep + both rulings) and the G18 gate cell (:7825); `docs/tool-index.md` (full, +for artifact lookups); `docs/audit/container-elim-pass/FINAL-PLAN.md` (Sections 1-2); +`docs/audit/container-elim-pass/pass0-admin-report.md` (:1-90, the container-layer scope +definition); `opentofu/modules/site-wan/main.tf` (full); `opentofu/main.tf` (uplink module +instantiations, :369-394). diff --git a/docs/audit/container-elim-pass/pass5-w3-reconciliation.md b/docs/audit/container-elim-pass/pass5-w3-reconciliation.md new file mode 100644 index 0000000..c62ccdf --- /dev/null +++ b/docs/audit/container-elim-pass/pass5-w3-reconciliation.md @@ -0,0 +1,255 @@ +# Pass 5 (W3) -- IPv6-unlock posture reconciliation, classification matrix, Roosevelt-delta + +**Author:** W3 worker, follow-on to the container-elim pass (D-144, RULED 2026-08-10). +**Scope:** posture reconciliation against the standing IPv6 rulings, the CLASSIFICATION +MATRIX across every family-bearing surface touched or adjacent to the collapse, the +Roosevelt-delta, and the honest (a)-genuine-unlocks-vs-(b)-independent-work framing. +READ-ONLY; nothing built, nothing executed, no D-number minted or amended here (GA-R5 -- +findings are LOGGED, never ruled). + +**Inputs read in full this session:** `docs/audit/container-elim-pass/FINAL-PLAN.md`; +`docs/design-decisions.md` D-101 (`:2243-2412`), D-124 + its four amendments +(`:4978-5170`), D-134 + all amendments incl. the 2026-08-10 `.8` client-VM one +(`:5885-6250`), D-138 (`:7091-7159`), D-139 both rulings (`:7280-7409`), D-141 +(`:8003-8055`), D-143 (`:8110-8208`), D-144 including its DEC package +(`:8210-8330`); `docs/audit/container-elim-pass/pass5-w1-mgmt-v6.md` (full) and +`docs/audit/container-elim-pass/pass5-w2-metal-data-v6.md` (full) -- both landed +mid-session; their precise, line-cited rows are IMPORTED into this matrix rather than +re-derived, per the tasking's "pull from W1/W2 once written" instruction. Every row +below sourced from W1/W2/FINAL-PLAN/design-decisions.md is cited to its origin; none +are re-derived independently unless marked `[W3]`. + +--- + +## 1. Posture reconciliation -- the standing rule, and where each surface class sits against it + +**The standing principle (D-101, operator verbatim, `:2274-2280`): "We are building as +close to full IPv6 as possible on every layer possible... using IPv4 where necessary +only because it is necessary."** Restated as the rule (`:2282`): **IPv6 unless IPv4 is +NECESSARY.** Every v4 leg is a **justified exception**, never a neutral default choice +(`:2282-2290`). D-139 Ruling A (`:7280-7338`, RULED 2026-07-31) is D-101's own +APPLICATION, not a revision: it names, per plane, exactly what forces v4 -- +`metal-admin` (PXE, the MAAS region API dial, the node-DNS forwarder, the v4-only +D-134 utility band, a rack with no global v6) and `provider-public` (tenant FIPs +against a still-v4 internet, external API VIPs, the edge default route) stay +**dual-stack**; the other five planes (`metal-internal`, `data-tenant`, `storage`, +`replication`, `lb-mgmt`) are **IPv6-only**. D-141 (`:8003-8055`) narrows the posture +ONE further notch for a specific, named case: container-hosted API-charm VIPs are +**dual-stack-AUTHORED but v4-`active`/v6-`reserved`**, gated on juju LP #1723240 + +per-charm fixes -- **this is not a cloud-wide v4 lean, it is the one place D-101's +default is provably blocked today** (memory pointer confirmed against the primary +source this session). + +**Reconciliation test applied to every surface below:** does a proposed v6 change +(a) apply an ALREADY-ruled family (mechanical, no new decision), (b) touch a +NAMED v4-necessity (D-141's LP #1723240 case; PXE), or (c) fall into a gap no ruling +has adjudicated? Class (c) is the one this pass must not silently resolve -- GA-R5 +forbids picking for the operator, so gaps are logged as candidate DEC material, not +flipped. + +**Ruled-vs-built distinction, imported from W2 (important -- do not conflate the +two):** for the four now-IPv6-only planes, the v4 subnets are ruled REMOVED but +**still live in MAAS today** -- D-139's own execution list states plainly "none of +it is done" (`design-decisions.md:7437`), and v4 is removed LAST, "after each is +proven" (`:7443`, per pass5-w2-metal-data-v6.md Section 2). A row marked ALREADY-v6 +below means the FAMILY RULING is settled, not that the v4 side is already gone from +the build. + +**D-143 axis discipline (`:8153-8158`, `:8321-8323`):** the re-IP is explicitly +**"a v4-only re-IP"** -- "The IPv6 family matrix, the 'IPv6 wherever possible' +principle... and the dual-stack ruling are UNTOUCHED." No v6 literal changes value +because of D-143; only v4 second/third octets shift 12->13 (D-134's octet-preserving +map, `:8169-8173`). Every row below that touches a v6 family is therefore `[D-139- +independent]` with respect to the `[D-143]` axis, on top of whatever it is with +respect to `[CE]`. + +--- + +## 2. The classification matrix + +**Classes used** (the tasking specified four -- ALREADY-v6, FLIP-TO-v6-NOW-POSSIBLE, +DUAL-STACK-REQUIRED, FORCED-v4 -- this pass's evidence required five more to avoid +mis-stating an unruled or in-flight surface as one of the four): + +| Class | Meaning | +|---|---| +| **ALREADY-v6** | Family ruled and, where checkable, already the standing state; applying it to a new object (e.g. the client VM) is mechanical, not a new decision. May still have v4 physically present in the live build (see ruled-vs-built note above). | +| **DUAL-STACK-REQUIRED** | Ruled dual-stack; both families are load-bearing by explicit ruling, not a default. | +| **FLIP-TO-v6-NOW-POSSIBLE** | The collapse structurally removes the v4 force; v6 becomes technically reachable, but the concrete mechanism/measurement is still OWED. | +| **v6-CAPABLE-BY-DESIGN (new, collapse-created)** | A NEW control/surface the collapse itself creates, built on a family-agnostic mechanism (verified by precedent), so it costs nothing extra for v6 day one. | +| **FORCED-v4** | A cited, ruled necessity for the specific sub-function. Distinguish the RULING ("v4-first") from an unmeasured universal capability claim -- see row 15. | +| **UNRULED-GAP** | No D-number adjudicates the family; v4 today by convention/history, not by ruling. Logged, not flipped. | +| **DUAL-AUTHORED-STATUS-GATED** | D-141's specific pattern: v4 `active`, v6 `reserved`, promotion gated on a named capability. | +| **CONDITIONAL/OPEN** | Resolution depends on a separate, not-yet-ruled fork (here, DEC-08); no family answer is assertable until the fork resolves. | +| **RETIRED / N/A** | The surface is eliminated by D-144 outright; no family question survives it. | + +| # | Surface | Class | Reason / governing D | Collapse-relation | +|---|---|---|---|---| +| 1 | MAAS node power-dial reach path (`vr1-dcN-maas-01` -> vcloud libvirtd) | **FLIP-TO-v6-NOW-POSSIBLE** | Flattening removes the old force (dialing a plane-bridge-bound libvirt address); the new target is vcloud's own sshd over a family-agnostic transport, from a source (`metal-admin`) that already has v6. Concrete reachable address is UNVERIFIED, live-gated on DEC-15. [pass5-w1 row 1, `docs/design-decisions.md:8299-8304`] | **(a) genuine unlock** -- did not exist as a question under Model B | +| 2 | D-124 region<->rack transit overlay (`172.31.0.0/24`) | **UNRULED-GAP** | No D-number rules this leg's family. `mesh-link` (L2) and SSH (transport) both already support v6; a candidate v6 slot exists in the apex (`f00::/40` "Infrastructure / site-to-site links", D-115 `:3751-3753`) but nothing has carved it. This is a PRE-EXISTING gap (predates D-144), not created by the collapse. [pass5-w1 row 2] | **(b) independent** -- collapse only re-homes the leg's endpoint, does not touch family | +| 3 | Client-VM `metal-admin` leg (`.8`, D-134 2026-08-10 amendment) | **ALREADY-v6** | `metal-admin` is D-139 Ruling-A dual-stack for every host on the plane; the client VM inherits it mechanically the moment it is placed there. No new v6 decision -- applying an existing rule to a new object. [pass5-w1 row 3; `docs/design-decisions.md:6240-6248`] | **(b) independent of collapse's v6 credit** -- the collapse creates the OBJECT, not the rule | +| 4 | Client-VM transit leg (same `172.31.0.x/30` scheme) | **UNRULED-GAP** | Same open question as row 2; the D-144 ledger's "D-124 AMENDS -- Scheme-A addressing survives" (`:8279-8280`) preserves the ADDRESSING, not a family decision. Do not conflate with row 1 (MAAS's power-dial does not route through the client VM). [pass5-w1 row 4] | **(b) independent**, re-homed not re-familied | +| 5 | (a) cross-DC host isolation control (OWED, DEC-14) | **v6-CAPABLE-BY-DESIGN** | Modeled on SEC-010's live `table inet sec010` artifact (`scripts/site-headend-install.sh:299`) -- nftables' `inet` family matches v4 AND v6 in one ruleset, scoped by interface not IP literal. Costs nothing extra for v6 day one. [pass5-w1 row 5] | **(a) genuine unlock** -- the control itself is NEW, created only because co-residency is new | +| 6 | SEC-010 transit-drop successor (endpoints: client VM + voffice1, DEC-16) | **v6-CAPABLE-BY-DESIGN** | Same `table inet` precedent as row 5; the planned extraction carries the shape forward. Covers a v6 transit leg IF/WHEN row 2/4 is ever ruled onto v6. [pass5-w1 row 6] | **(a) genuine unlock** in mechanism; **(b)** the traffic it protects (row 2/4) is still v4 pending its own ruling | +| 7 | MAAS region<->rack control path (retirement pending DEC-08) | **CONDITIONAL/OPEN** | Moot if DEC-08 ratifies rack-controller retirement (pass2's own recommendation); if rejected, whether MAAS 3.7's rack<->region RPC supports v6 is UNVERIFIED -- no repo or vendor citation found either way. [pass5-w1 row 7] | **(b) independent** of the collapse itself (D-132 already put region+rack co-located per DC) | +| 8 | `provider-public` plane | **DUAL-STACK-REQUIRED** | D-139 Ruling A: FIPs against a still-v4 internet, external API VIPs, Octavia tenant LB VIPs, edge default route. [pass5-w2 Section 2; `docs/design-decisions.md:7310-7311,7326`] | **(b) independent** -- D-139-ruled 2026-07-31, predates D-144 | +| 9 | `metal-admin` plane (base plane) | **DUAL-STACK-REQUIRED**, with an embedded **FORCED-v4** sub-function | Dual by ruling (D-139 `:7327`), but the PXE/DHCP sub-function inside it is v4-only **by design**, not merely v4-heavy: MAAS's own v6-static-only node-acquisition ruling (D-134 2026-07-27 amendment, `:6138-6144`) leaves `dhcpd6` deliberately OFF, and the rack itself measures zero global v6 (`ip -6 -o addr show scope global` empty, per pass5-w2 Section 1). **PXE-over-v6 platform CAPABILITY is UNVERIFIED** -- no test/capture/vendor citation on disk states whether MAAS 3.7 even supports v6-only commissioning; do not import outside knowledge. [pass5-w2 Section 1, full] | **(b) independent** | +| 10 | `metal-internal` plane | **ALREADY-v6** (ruled IPv6-only) | D-139 Ruling A explicitly **SUPERSEDES** D-101's "datastore east-west v4-bound" clause (`:7334-7336`) -- "nothing forces v4 here... a JUDGEMENT, not a protocol requirement" reversed. **Build status: v4 still live, removed LAST after proof** (pass5-w2 Section 2). A reader citing D-101's stale text alone would mis-state this plane's CURRENT family -- cite D-139. | **(b) independent** -- ruled 2026-07-31 | +| 11 | `data-tenant` plane (geneve overlay) | **ALREADY-v6** (ruled IPv6-only) | D-139 Ruling A. The 2026-08-09 geneve-over-v6 rebuild fix is recorded at `docs/audit/geneve-over-v6-rootcause-20260808.md` (confirmed present on disk) and is **orthogonal to D-144** per pass5-w2 Section 5b's explicit statement -- it fixes HOW v6 geneve works (a bracketed `ovn-encap-ip` OVS rejection), not WHETHER the plane is v6; do not credit the collapse for this. | **(b) independent, and independent of D-139 too** (a build-mechanics fix, not a family ruling) | +| 12 | `storage` plane (Ceph public) | **ALREADY-v6** (ruled IPv6-only) | D-139 Ruling A. Build status: v4 still live, removed last. `ceph-mon`/`ceph-osd` are in `PREFER_IPV6_CHARMS` (`:7432`) -- converting this plane REOPENS the 2026-07-31 "prefer-ipv6 false on all seven" ruling note per-app, not yet re-ruled (pass5-w2 Section 6 risk note). | **(b) independent** | +| 13 | `replication` plane (Ceph cluster incl. cross-DC leg) | **ALREADY-v6** (family settled) but **carrier path OPEN (DEC-24)** | D-139 Ruling A settles the FAMILY. Two open items compound on the SAME carrier: (i) the cross-DC leg has **no v6 route at all today** (D-139's own build-constraints bullet, `:7431`, pre-existing and OPEN); (ii) D-144's DEC-24 (`:8312-8317`) leaves the post-flatten `virbr5`/netem attachment for that same leg UNDEFINED. pass5-w2 Section 6 names this explicitly: **"the one point where the W2 family ruling and the D-144 collapse genuinely intersect -- an unresolved carrier for an unresolved route."** | **Mixed: (b) family independent; (a) the carrier-attachment QUESTION is collapse-raised** (DEC-24 exists because of the flatten) | +| 14 | `lb-mgmt` plane (Octavia) | **ALREADY-v6**, but **two distinct objects, do not conflate (G18 warning)** | (1) the octavia charm's own live/consumed **ULA `fc00::/64`** (R8-ruled); (2) D-139's apex **GUA `f0X:80::/64`** carved as a MAAS-underlay plane with NO charm consumer, kept `reserved` per the G18 ruling (option b). Pointing octavia at object (2) would reopen R8. [pass5-w2 Sections 2 and 4] | **(b) independent** -- G18/D-141 governed, pre-dates D-144 | +| 15 | MAAS PXE / node provisioning (sub-function of row 9) | **FORCED-v4** | D-101 (`:2307-2308`) and D-139 Ruling A both **rule** "PXE is v4-first" -- but this is a ruling, not a measured platform-capability ceiling. **Do not assert PXE-over-v6 is impossible**: no test, capture, or MAAS-source citation on disk measures whether MAAS 3.7 could commission v6-only (pass5-w2 Section 1, the "anchor question", explicit UNVERIFIED verdict). What IS settled: the ruling is unscoped to VR1 specifically and nothing in D-101/D-139 limits the v4-first posture to this deployment. | **(b) independent** -- and see Roosevelt-delta below | +| 16 | LXD-container / API-charm VIPs (juju LP #1723240) | **DUAL-AUTHORED-STATUS-GATED** | D-141: v4 `active`, v6 `reserved`, promotion gated on the two-layer capability set (LP #1723240 + per-charm fixes, `docs/charm-ip-family-compatibility.md`). Carried risk (pass5-w2 Section 5a): on `metal-admin` (dual-stack), container addresses land on ONE family by unsorted mongo order (today v4) -- a detection gate ("every container's metal-admin leg is IPv4") is OWED, not built. **DISAMBIGUATION (do not overstate D-144's reach):** "container-elim" in this pass's title means the `vvr1-dcN` **containment VM** (D-122/D-123 Model B) -- it has nothing to do with juju's LXD-container control-plane units on the node VMs. D-144 eliminating the containment VM does **not** touch, unblock, or moot LP #1723240 in any way (pass5-w2 Section 5a: "D-144's relationship to this gap: NONE"). | **(b) fully independent** -- different container concept entirely | +| 17 | Uplink/ISP NAT (`172.30.x/24`, `modules/site-wan`) | **FORCED-v4**, single-stack by construction | `opentofu/modules/site-wan/main.tf` takes ONE `var.cidr`, one `forward.mode="nat"`, one `ips` entry -- no v6 CIDR variable exists in the module at all (pass5-w2 Section 3). D-101's own rationale defers external v6 deliberately: NAT64/DNS64 was considered and REJECTED as a v6-egress simulant (`:2295-2299`); VR1's v4-only edge is a rehearsal simplification, not a capability gap. **D-144 persists this UNCHANGED in family** -- "edge WAN attaches directly to the per-DC `site-wan` NAT" (`:8256-8257`); only the WAN-bridge INDIRECTION (row 19, D-125's plumbing) is removed, not the NAT module or its family. | **(b) independent** -- do not conflate with row 19's retirement | +| 18 | `qemu+ssh` provider dial (voffice1 -> `vvr1-dcN`, D-126 per-env keys) | **RETIRED / N/A** | D-144: "the qemu+ssh provider dial + its D-126 per-env keys (no successor)" (`:8254-8255`). The surface is DELETED, not flipped -- there is no family question left to classify. | **(a) genuine removal** (not a v6 unlock, a surface elimination) | +| 19 | D-125 bridge-in `modules/wan-bridge` (WAN indirection only, NOT the uplink NAT) | **RETIRED / N/A** | D-125 header: "TERMINATED 2026-08-10 by D-144" (`:5091-5094`); the DC edge WAN attaches directly to row 17's `site-wan` NAT instead of bridging through the (now-eliminated) containment VM. Retiring the bridge module removes an indirection layer, not a family -- row 17's v4-only NAT survives this row's retirement unchanged. | **(a) genuine removal** | + +**Matrix shape (19 rows):** ALREADY-v6 = 6 (rows 3, 10, 11, 12, 13-family-half, 14); +DUAL-STACK-REQUIRED = 2 (rows 8, 9); FLIP-TO-v6-NOW-POSSIBLE = 1 (row 1); +v6-CAPABLE-BY-DESIGN(new) = 2 (rows 5, 6); FORCED-v4 = 2 (rows 15, 17); +UNRULED-GAP = 2 (rows 2, 4); DUAL-AUTHORED-STATUS-GATED = 1 (row 16); +CONDITIONAL/OPEN = 1 (row 7); RETIRED/N/A = 2 (rows 18, 19). +**6+2+1+2+2+2+1+1+2 = 19.** (Row 13 counted once, under ALREADY-v6, with its +carrier-open half called out in the reason column and Section 4 below -- not +double-counted.) + +**The honest headline: only ONE row (row 1) is a clean, single "the collapse newly +makes this v6-possible" unlock, and even it is UNVERIFIED pending DEC-15.** Rows 5/6 +are collapse-created SURFACES that happen to be v6-capable by construction, not +existing v4 surfaces flipped to v6. An empty-or-near-empty FLIP-TO-v6-NOW-POSSIBLE +class is itself the finding, not a gap in this pass's search. + +--- + +## 3. Roosevelt-delta -- bare metal has no virtualization substrate + +Judged surface-by-surface against the operator's "module deployment project" lens +(FINAL-PLAN Section 3, `:135-142`, imported rather than re-derived): + +**Transfers verbatim (v4-forced on bare metal too -- not a virtualization artifact):** +- **Row 15, PXE ruling.** The v4-first RULING is unscoped to VR1 and nothing in + D-101/D-139 limits it to this deployment, so it transfers as a posture. **What does + NOT transfer as a settled fact is a capability claim** -- whether Roosevelt's actual + firmware/MAAS stack could do v6-only PXE/HTTP-boot is genuinely unmeasured here (per + row 15/pass5-w2's explicit UNVERIFIED verdict); this pass does not assert PXE is + protocol-impossible over v6 in general, only that this repo has built and ruled a + v4-first path and never tested an alternative to compare against. +- **Row 16, LP #1723240.** This is a **juju-layer defect** (`state/linklayerdevices.go`, + `EthernetDeviceForBridge` taking `addrs[0]` from an unsorted query, per D-139's "What + forced this" section `:7401-7409`) -- substrate-independent. Bare metal runs the + same juju version against the same bug; it transfers UNCHANGED. This is the + non-obvious one: an operator could assume flattening the substrate also flattens + this defect away. It does not -- juju, not the hypervisor, is the constraint. + +**Does NOT transfer (virtual-only artifacts, no bare-metal analog in the same shape):** +- **Rows 18/19, `qemu+ssh` dial + D-125 bridge-in module.** Both exist ONLY because of + nested libvirt (Model B); Roosevelt has no containment layer to dial into or bridge + WAN through, so neither the surface nor its retirement has a bare-metal counterpart. +- **Rows 5/6, the (a) control + SEC-010 successor.** FINAL-PLAN Section 3 (`:139-142`) + flags these explicitly as "artifacts of vcloud's single-hypervisor co-residency with + no bare-metal analog in the same shape" -- a Roosevelt DC has its own physical NICs + and switches, not a shared kernel with another DC's plane bridges. +- **Row 1's power-key mitigation (DEC-15).** Same reasoning: the blast-radius problem + it mitigates is specifically "one vcloud libvirtd controls everything"; bare metal's + power control is BMC/IPMI per physical host, already isolated by hardware. +- **Row 17's simulated-ISP v4 NAT.** D-101 consequence 3 (`:2294-2301`): "Roosevelt has + FULL v4 AND v6 EDGE TRANSPORT... External IPv6 routing is deliberately NOT added this + deployment; it arrives with the next full deployment." VR1's v4-only simulated-ISP + hop is a REHEARSAL SIMPLIFICATION, not a Roosevelt requirement -- it does not + transfer. +- **Row 2/4's D-124 transit `/30`s.** The transit-numbered-mesh SCHEME (region<->rack + P2P addressing) is a virtualization-era answer to "how does an off-site controller + reach an isolated rack"; whether it has a Roosevelt analog depends on Roosevelt's own + physical topology, which is out of this pass's scope to assert. + +**Mesh triangle / netem (row 13's carrier, `virbr5` + siblings):** FINAL-PLAN's own L0 +judgment (`:135-136`) already rules these "virtualization shims" that do NOT transfer; +DEC-24's open carrier-attachment question is a VR1-internal problem with no Roosevelt +equivalent to inherit an answer from. + +--- + +## 4. (a) genuine unlocks vs (b) independent work -- the key honest framing + +**(a) Genuine unlocks the collapse creates** (would not exist, or would not be +answerable this way, under Model B): +1. Row 1 -- the MAAS power-dial reach path becomes structurally v6-answerable (still + unverified live). +2. Rows 5/6 -- two BRAND NEW controls (the (a) isolation control, the SEC-010 + successor) that exist only because flattening creates co-residency; both happen to + be v6-capable by construction at zero extra cost. +3. Row 13's carrier-ATTACHMENT question (DEC-24) -- raised specifically because the + flatten reassigns every other cross-DC/cross-site leg and left this one undefined; + its FAMILY (already-v6) was settled by D-139 well before D-144 existed. +4. Rows 18/19 -- outright REMOVAL of two v4-only virtualization artifacts (not v6 work, + but real simplification the collapse buys). + +**(b) v6 work that is INDEPENDENT of the collapse and must not be credited to it:** +1. D-139's entire plane family matrix (rows 8-14) -- ruled 2026-07-31, over a week + before D-144 (2026-08-10). None of it required, or was blocked by, the containment + layer's existence. +2. The geneve-over-v6 rebuild fix (row 11) -- a build-mechanics defect (bracketed + `ovn-encap-ip`) fixed independently of any topology change; the plane was already + ruled v6-only under Model B. pass5-w2 Section 5b states its D-144 relationship as + explicitly "NONE". +3. D-141's LXD-container dual-stack-authored/status-gated pattern (row 16) -- gated on + a juju upstream defect that the collapse cannot touch, per the Roosevelt-delta + section above and pass5-w2 Section 5a's explicit "NONE" verdict. **This is the + pattern the tasking's "do not overstate" caveat targets directly: D-144 flattens + the containment VM, full stop; it does not, and cannot, unblock LP #1723240.** +4. Row 3 (client-VM metal-admin leg) -- v6 arrives via a standing D-139 rule applied to + a new object, not via a new decision the collapse forced. +5. Row 15 (PXE forced-v4) -- unaffected by topology; still forced under the flat shape + exactly as it was under Model B. +6. Row 17 (uplink NAT) -- persists in family completely unchanged through D-144; only + an indirection layer (row 19) is removed around it. + +**No v6 flip identified in this matrix conflicts with a standing ruling.** The two +UNRULED-GAP rows (2, 4 -- the D-124 transit family) are pre-existing gaps this pass +surfaces, not proposed flips; per GA-R5 they are logged as **candidate DEC material** +(a future DEC-14/DEC-16 ruling session should decide whether to close them, since both +touch the same leg those DECs already govern) and are explicitly NOT resolved here. + +--- + +## 5. Open items carried forward (not ruled, not built) + +- **The D-124 transit-overlay family gap (rows 2, 4)** has no owning DEC row today. + Flag it when DEC-14/DEC-16 are ruled (pass5-w1's own top risk #2) so it is not + rediscovered from scratch. +- **DEC-24's carrier-attachment question (row 13)** is already an OWED open item in + D-144's body (`:8312-8317`); pass5-w2 independently names it the single point where + its plane-family work and the D-144 collapse genuinely intersect (Section 6). +- **Row 7's MAAS-3.7 v6 RPC capability** is UNVERIFIED either way in the repo; do not + assert it in either direction until DEC-08 resolves (retirement likely moots it). +- **Row 9/15's PXE-over-v6 platform capability** is UNVERIFIED (pass5-w2's own explicit + verdict) -- carried forward as a standing caveat, not something this pass or a future + one should assert either way without a measured test. +- **pass5-w1 and pass5-w2 both landed during this session** and are fully reconciled + into the matrix above (rows 1-7 from W1, rows 8-17 informed by W2, row-for-row, no + contradiction found between W2's findings and this pass's earlier draft rows -- the + earlier draft's PXE framing (row 15) and its D-125/uplink-NAT conflation (rows 17/19) + were corrected against W2's more precise citations before this version was written). + +--- + +## Verification note + +Direct reads this session: `docs/audit/container-elim-pass/FINAL-PLAN.md` (full), +`docs/audit/container-elim-pass/pass5-w1-mgmt-v6.md` (full, rows 1-7 imported +verbatim with original citations preserved), `docs/audit/container-elim-pass/ +pass5-w2-metal-data-v6.md` (full, rows 8-17 informed by its precise findings), +`docs/design-decisions.md` D-101 (`:2243-2412`, incl. the 2026-07-27 ruling-note +correction), D-124 + amendments (`:4978-5170`, incl. the full D-125 bridge-in text), +D-134 + all amendments incl. 2026-08-10 (`:5885-6250`), D-138 (`:7091-7159`), D-139 +both rulings (`:7280-7409`), D-141 (`:8003-8055`), D-143 (`:8110-8208`), D-144 incl. +DEC package (`:8210-8330`); `docs/tool-index.md` (header + teardown table); +`ls scripts/ | grep geneve` confirmed `geneve-encap-assert.sh` and +`dc-dc-mtu-geneve-budget.sh` exist on disk; `ls docs/audit/ | grep geneve` confirmed +`geneve-over-v6-rootcause-20260808.md` exists on disk (cited by path, not memory); +`ls docs/audit/container-elim-pass/ | grep pass5` re-checked twice this session -- +`pass5-w1-mgmt-v6.md` and `pass5-w2-metal-data-v6.md` both present at write time, +fully reconciled into this document (Section 5). READ-ONLY throughout; no mutation; +nothing executed against the cloud; no D-number ruled, amended, or proposed as a +ruling by this pass (findings only, GA-R5). diff --git a/docs/design-decisions.md b/docs/design-decisions.md index 6b3c252..1f1c5c1 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -8321,6 +8321,17 @@ - **Two-axis discipline:** every redeploy change is attributable to `[D-143]` (10.12->10.13 value substitution) or `[CE]` (this shape change); four dual-cause items carry `[both]`. `lib-net.sh` carries ZERO container-elim edits. +- **IPv6-unlock design input (added 2026-08-10, read-only pass5):** the collapse unlocks little NEW + IPv6 -- of a 19-surface family matrix, only the MAAS power-dial reach path is a clean collapse-caused + v6 unlock (and it is UNVERIFIED, DEC-15-gated). Actionable inputs folded into the owed DECs: (i) + DEC-15 should prefer a v6 reach (vcloud sshd, off any plane bridge); (ii) DEC-14/16 should decide the + UNRULED D-124 transit-overlay family (v6 candidate `f00::/40` reserved in the apex) on the same leg + they govern, and keep the (a)/SEC-010-successor controls' nftables `inet` (v6-native) shape; (iii) + DEC-24 must resolve the cross-DC replication carrier AND its still-open D-139 v6-route together (same + leg). NOT unlocked by the collapse (different layers, do NOT credit D-144): D-139's plane family + matrix, the geneve-over-v6 fix, and the LXD-container/juju LP#1723240 gap (D-141). metal-admin PXE + stays v4 by ruling (v6-only MAAS commissioning is UNVERIFIED, not asserted impossible). Full note: + `docs/audit/container-elim-pass/ipv6-unlock-design-note-20260810.md`. **A1 TEST (GA-R3):** admitted -- SUPERSEDES D-123's core Model-B ruling; a Roosevelt / pre-Roosevelt build session greps this before laying out substrate shape; and the layered module workflow (L0-L5,