diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 67a097f..2a434c2 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1288,7 +1288,42 @@ **G17** exists for. PROPOSED, not adopted (amending a gate row is the operator's): fold a fourth G17 assertion that a booted node carries a global v6 address from its plane's `/64` in the ruled `::100`-`::200` band. G17's current three assertions are unchanged. -- **RENDER-PIPELINE STEP 3 IS COMPLETE 2026-07-27.** Every authoritative source now carries the +- **TWO CORRECTIONS TO THIS DOCUMENT'S OWN STEP-3 CLAIMS, both surfaced by an operator + question ("Are we mirroring the assigned IPv6 octet with the last IPv4 octet like we did + before?").** (i) **NODE v6 IS NOT ASSIGNED AND WILL NOT ARRIVE WITH THE SUBNET.** This + document said MAAS static assignment "needs only the subnet, which now exists". WRONG: + MAAS `mode=static` means an EXPLICITLY CONFIGURED address -- the 90 v4 node links were set + that way by the Stage-4 carve. Measured: **all 18 Ready nodes carry ZERO IPv6 links**, while + their v4 side is correctly octet-mirrored (`superb-piglet` holds `.121` on metal-admin, + metal-internal and data-tenant alike). So the VIPs are mirrored (156 apex objects) but the + NODES are not, and 108 explicit assignments are owed. (ii) **v6 `gateway_ip` and + `dns_servers` are UNSET on all 12 v6 subnets** (measured), against v4 which carries the edge + gateway on provider-public and the D-131 forwarder on metal-admin. Consistent with the + no-external-v6-routing posture, but it was not named before this document called step 3 + complete. **STEP 3's MAAS/apex/lib-net population stands as recorded; "complete" was + premature and is withdrawn** -- the node layer is the remaining half. +- **D-101 GOVERNING RATIONALE RECORDED 2026-07-27** (operator, verbatim; it existed in NO repo + surface). Establishes the standing principle **IPv6 unless IPv4 is NECESSARY**, driven by + real IPv4 SIZING constraints in future expansion -- a commercial requirement, not a protocol + preference. Three things a future session must not misread: the v4-first sequencing was + DELIBERATE RISK REDUCTION (bringing the stack up on v4 first rather than debugging the cloud + and IPv6-in-charms at once), so v4 surfaces are not oversights to "correct"; Roosevelt will + have FULL v4 AND v6 EDGE TRANSPORT; and consequently **NAT64/DNS64 was CONSIDERED AND + REJECTED** as a way to simulate v6 egress -- with native v6 transport at Roosevelt it is a + shim with no Roosevelt analog, and the ULA planes are internal by design so nothing here + needs v6 egress. **External IPv6 routing is deliberately NOT added this deployment.** +- **NODE v6 CARVE SCOPED, NOT EXECUTED** -- `docs/audit/node-v6-carve-scope-20260727.md`. + **108 assignments (18 nodes x 6 planes)**, each a v6 static whose host part equals that + node's EXISTING v4 octet (read live, never from a table) inside that plane's `/64`; the + provider leg uses the node `/64` `f0X:10::`, not the VIP `/64`. Prior art measured: + `carve-host-interfaces.sh` already makes the exact `interface link-subnet ... mode=STATIC` + call but is **v4-only (`grep -ciE 'ipv6|::|inet6'` -> 0)** with hardcoded v4 bases, so a v6 + arm or a companion on the proven `dc-plane-ipam.sh` pattern is needed. **THE OPERATOR'S OWN + RATIONALE MAKES THE CASE FOR DOING IT BEFORE THE DEPLOY:** v4-first was chosen partly to + avoid IPv6-in-charms trouble, and a node carrying v6 on six planes is exactly what surfaces + such modules -- far cheaper on a static read-back than mid-bundle. Explicitly out of scope: + v6 gateways/DNS, rack legs (not needed under the static ruling), NAT64, tenant addressing. +- **RENDER-PIPELINE STEP 3 -- MAAS, lib-net AND APEX HALVES COMPLETE 2026-07-27.** Every authoritative source now carries the ruled values: **MAAS** (12 v6 plane subnets carved; 24 D-134 bands + both FIP pools reserved; `dc-plane-ipam check` pass=24 fail=0 and `reserve` planned=0 on BOTH DCs), **`lib-net.sh`** (dc1 FIP pool set, with the R9/D-133 unsets now guarded by new harness cases), and the diff --git a/docs/audit/node-v6-carve-scope-20260727.md b/docs/audit/node-v6-carve-scope-20260727.md new file mode 100644 index 0000000..a3cae0e --- /dev/null +++ b/docs/audit/node-v6-carve-scope-20260727.md @@ -0,0 +1,121 @@ +# Node IPv6 carve -- SCOPE (2026-07-27). Not executed. + +Operator-directed scoping ("scope the node v6 carve"). This is a plan, not a status +surface: stage/gate status lives only in `docs/CURRENT-STATE.md` (GA-R1). + +## Why this exists, and the correction behind it + +When the node-v6-acquisition question was ruled ("MAAS static assignment is correct"), +this session told the operator that static assignment "needs only the subnet, which now +exists". **That was wrong, and the operator's own question exposed it** ("Are we mirroring +the assigned IPv6 octet with the last IPv4 octet like we did before?"). + +Measured: MAAS `mode=static` means an EXPLICITLY CONFIGURED address. The 90 v4 node links +were set that way by the Stage-4 carve, not auto-allocated. So a v6 address only appears +on a node if it is assigned, exactly as the v4 statics were. + +``` +superb-piglet enp1s0 10.12.8.121 mode=static <- explicitly set, Stage-4 carve + enp3s0 10.12.12.121 mode=static + enp4s0 10.12.16.121 mode=static + v4 links = 6 v6 links = 0 +ACROSS ALL 18 Ready nodes: IPv6 links = 0 +``` + +The v4 side already demonstrates the octet mirror this carve must reproduce in v6: one +stable octet per node across every plane (`.121` above), per D-134. + +## Governing decisions + +- **D-101** family matrix + its 2026-07-27 GOVERNING RATIONALE: *IPv6 unless IPv4 is + necessary*. This carve is that principle applied at the node layer -- the layer where + the IPv4 sizing constraint actually bites. +- **D-134** (2026-07-23 amendment): node band `.100-.200`, one stable octet per node + across all six planes; control `.100-.119` / compute `.120-.149` / storage `.150-.200`. +- **v6 host-part convention RULED 2026-07-27**: mirror the v4 octet TEXTUALLY. Node `.121` + becomes `::121`, NOT the hex conversion. (The same trap already caught the band work: + `printf '%x' 50` yields `32`.) +- **R2 / D-101**: dual-stack where IPv4 is required, IPv6-only where sound, BOTH DCs. + +## Size and shape + +**108 link assignments: 18 nodes x 6 planes.** Per node, per plane, one v6 static whose +host part equals that node's existing v4 octet, inside that plane's `/64`. + +| plane | dc0 v6 `/64` | dc1 v6 `/64` | +|---|---|---| +| provider-public (GUA) | `2602:f3e2:f02:10::/64` | `2602:f3e2:f03:10::/64` | +| metal-admin (ULA) | `fd50:840e:74e2:220::/64` | `fd50:840e:74e2:320::/64` | +| metal-internal (ULA) | `:221::/64` | `:321::/64` | +| data-tenant (ULA) | `:230::/64` | `:330::/64` | +| storage (ULA) | `:240::/64` | `:340::/64` | +| replication (ULA) | `:250::/64` | `:350::/64` | + +NOTE the provider leg uses the **node** `/64` (`f0X:10::`), NOT the VIP `/64` (`f0X:11::`) +which holds the hacluster-managed API VIPs already written to the apex. + +All 12 subnets EXIST in MAAS (carved earlier this session) and each sits on the SAME vlan +as its v4 twin, so this is addressing on established L2, not new fabric. + +## Mechanism -- prior art exists and is v4-only + +`scripts/carve-host-interfaces.sh` performed the v4 carve and already makes the exact call +needed: + +``` +maas interface link-subnet \ + mode=STATIC subnet= ip_address= +``` + +Measured: `grep -ciE 'ipv6|::|inet6'` over that script returns **0** -- it is v4-only, and +its addresses are built as hardcoded v4 bases plus the octet (e.g. `:263` +`ip_address="10.12.8.$OCTET"`). So this carve needs either a v6 arm in that script or a +companion tool. RECOMMEND a companion on the `dc-plane-ipam.sh` pattern already proven this +session: site-keyed, **dry by default**, idempotent, and READ BACK per write. + +Precedent that the operation is safe on `Ready` nodes: the Stage-4 carve did 90 of these +on Ready nodes with a read-back each, and all 18 stayed Ready. + +## Preconditions + +1. Nodes stay in `Ready` (powered off). `link-subnet` does not require a boot, and MAAS + renders these into node netplan at deploy -- which is why doing it BEFORE Stage 5 makes + them land at first boot rather than needing a re-deploy. +2. Octet source is the node's EXISTING v4 assignment, read live -- never a table. The + mirror must be derived from what MAAS actually holds, so a node whose octet ever drifted + cannot silently get a mismatched v6 address. +3. The v6 subnets' `gateway_ip` and `dns_servers` are UNSET (measured). That is consistent + with the no-external-v6-routing posture, but see the open item below -- it is a separate + decision from this carve and does NOT block it. + +## Risks, and the one the operator named + +- **Charm/module IPv6 support is the real risk.** The operator's own rationale records that + v4-first was chosen partly to avoid "troubleshooting IPv6 configurations around modules + that may not have supported it as default". A node carrying v6 on all six planes is + exactly what surfaces those modules. **That argues for carving BEFORE the deploy, not + after** -- discovering a charm that mishandles a v6 leg during the bundle deploy is far + more expensive than discovering it on a static-assignment read-back. +- **Blast radius is per-link and reversible**: `interface unlink-subnet` removes one. No + node is powered on; no tfstate, no running service. +- **Do NOT re-commission to apply these.** The 2026-07-20 incident regenerated all 9 node + NIC MACs on an in-place apply and stranded MAAS's records; a carve must touch links only. + +## Verification owed + +1. **Per-write read-back**, as every mutation this session has done. +2. **A gate assertion**: extend `dc-plane-ipam.sh check` so 18/18 nodes carrying their + mirrored v6 octet on all six planes is a NAMED executable check rather than a count in + prose -- the same reason the bands got a gate. +3. **At Stage-5 first boot (G17)**: that the node actually brings the address up. MAAS + holding a link is not the same as an interface carrying it -- this session has already + been bitten by asserting existence instead of content. + +## Explicitly OUT of scope + +- v6 `gateway_ip` on any plane, and v6 `dns_servers` -- open, and deliberate under the + no-external-v6-routing posture. +- Rack-bridge v6 legs -- NOT needed under the static-assignment ruling. +- NAT64/DNS64 -- REJECTED 2026-07-27 (Roosevelt has native v6 edge transport, so it has no + Roosevelt analog). Recorded on D-101. +- Tenant addressing -- Neutron-managed, never in the apex (D-016/D-074). diff --git a/docs/design-decisions.md b/docs/design-decisions.md index d376359..6c701d9 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -2068,6 +2068,41 @@ **Context:** v1 / DC0 is IPv4-only on 10.12.0.0/16 (six planes in `scripts/lib-net.sh`). The developed IPv6 plan (in-project, not yet committed) classifies planes by reachability: GUA for external clients, ULA for internal-only, IPv4 where forced. DC-DC is the phase that activates it. +**GOVERNING RATIONALE -- recorded 2026-07-27 from the operator, verbatim, because it +existed in NO repo surface and is decision-shaping beyond this entry.** D-101 rules WHICH +family each plane carries; it recorded nothing about WHY, so a reader could not tell a +deliberate posture from an accident, and a v4-heavy deployment reads as IPv6 neglect. +Operator, exact words: + +> "The ULA only networks are fully internal, yes I do understand that. Roosevelt will have +> full ipv4 and ipv6 edge transport. There are real IPv4 sizing constraints that we are +> running up against in future expansion plans. We are building as close to full IPv6 as +> possible on every layer possible. We are using IPv4 where necessary only because it is +> necessary on same layers at this point in time. We are also using IPv4 because bringing +> the full stack live on IPv4 was easier in earlier stages then also troubleshooting IPv6 +> configurations around modules that may not have supported it as default." + +**THE STANDING PRINCIPLE this establishes: IPv6 unless IPv4 is NECESSARY.** It answers +future per-plane and per-service questions without a fresh ruling each time -- the default +is v6, and every v4 leg is a justified exception, not a neutral choice. + +**THREE CONSEQUENCES a future session must not misread.** +1. **The v4-first sequencing was DELIBERATE RISK REDUCTION, not neglect.** Bringing the + stack up on v4 first avoided debugging the cloud and IPv6-in-charms simultaneously. + Do NOT "correct" the v4 surfaces as though they were oversights, and do NOT read the + absence of v6 somewhere as evidence it was not wanted. +2. **The driver is IPv4 SIZING in future expansion**, i.e. commercial capacity, not + protocol preference. That makes v6 coverage a business requirement, which is why it is + worth cost here rather than deferring wholesale to Roosevelt. +3. **Roosevelt has FULL v4 AND v6 EDGE TRANSPORT.** Consequence measured 2026-07-27: + **NAT64/DNS64 was considered as a way to simulate v6 egress in VR1 and REJECTED** -- + with native v6 transport at Roosevelt it is a shim with NO Roosevelt analog, testing a + translation path that will never run in production, while adding a component to the + OPNsense edges (the most fragile surface here) and re-coupling the rack v6 leg through + DNS64's dependence on the D-131 forwarder. The ULA planes are internal BY DESIGN, so + nothing in VR1 needs v6 egress. **External IPv6 routing is deliberately NOT added this + deployment; it arrives with the next full deployment.** + **Decision:** - Per-DC distinct v4: DC1 INHERITS the DC0 six-plane v4 layout (provider-public 10.12.4.0/22 ... replication 10.12.36.0/22) unchanged, so DC1 equals the validated template. DC2 gets a DISTINCT, non-overlapping parallel supernet (same six-plane structure; exact prefixes assigned in NetBox, non-colliding with DC1, operator infrastructure, and client VPN ranges per D-074). Distinct ranges keep DR / backup endpoints unambiguous (no NAT hairpin). - Family matrix (per DC), family follows reachability: