diff --git a/docs/changelog-20260716-review-sweep-phaseC2-D125.md b/docs/changelog-20260716-review-sweep-phaseC2-D125.md index 02c69ca..b4e349c 100644 --- a/docs/changelog-20260716-review-sweep-phaseC2-D125.md +++ b/docs/changelog-20260716-review-sweep-phaseC2-D125.md @@ -21,15 +21,17 @@ **OUTER root (`opentofu/main.tf` + `variables.tf`, vcloud).** - New `module "vr1_dc0_uplink"` (`site-wan` NAT) = the SINGLE ISP NAT at vcloud level (egresses via - vcloud's uplink). CIDR = new HELD `var.vr1_dc0_uplink_cidr` (no default; NetBox office1-netbox apex). + vcloud's uplink). CIDR = `172.30.2.0/24` inlined -- the SAME /24 D-115 already assigned vr1-dc0's edge + WAN (bridge-in is single-NAT; RULED, not HELD). [CORRECTED 2026-07-16: an earlier draft added a HELD + `var.vr1_dc0_uplink_cidr` + a NetBox assignment; both removed -- D-125 adds no new NetBox object.] - `vvr1_dc0` gains a 2nd NIC (order `[transit, uplink]` -- transit stays first so SEC-010/`--transit-if` keep keying on it). Netplan now declares the IP-less `uplink` port + the `br-vr1-dc0-wan` bridge (STP off), so the bridge exists before the inner apply. `tofu validate` Success. **INNER root (`opentofu/vr1-dc0-substrate/main.tf`).** `module "vr1_dc0_wan"` switched `site-wan` (NAT, `172.30.2.0/24`) -> `wan-bridge` bound to `br-vr1-dc0-wan`. OPNsense `wan_network_name` -wiring unchanged (same output). OPNsense WAN re-addresses into the vcloud ISP /24 (edge config, D-113 -- -HELD). `tofu validate` Success. +wiring unchanged (same output). The `/24` and the OPNsense WAN `.2` are UNCHANGED (same 172.30.2.0/24, +D-115); only the libvirt realization moved to the vcloud NAT. `tofu validate` Success. **Bootstrap (`scripts/site-headend-install.sh` node-host mode).** - New `--uplink-if` (default `uplink`) + `--wan-bridge` (default `br-vr1-dc0-wan`). @@ -56,15 +58,16 @@ 1. **L2-NAT of foreign MAC across the nested bridge** -- unprovable pre-apply (bridge-in's L4-boot-gate equivalent). Isolation-test first; FAIL => revert to double-NAT (D-125). -2. **HELD:** assign `vr1_dc0_uplink_cidr` (vcloud ISP /24) in office1-netbox; set OPNsense WAN static in - that /24 (edge config). `tofu plan` requires the tfvar. -3. Confirm the `uplink`/`br-vr1-dc0-wan` interface names on first boot (enp2s0-vs-ensX naming trap). +2. Confirm the `uplink`/`br-vr1-dc0-wan` interface names on first boot (enp2s0-vs-ensX naming trap). + +(No new HELD addressing: the `/24` is the ruled `172.30.2.0/24` and the OPNsense WAN stays `.2`; D-125 +adds no NetBox object. The only NetBox work for the DC substrate is D-124's transit /30 + rack IP.) ## Revert (per item) -- `git rm -r opentofu/modules/wan-bridge`; revert `opentofu/main.tf` + `variables.tf` (drop - `vr1_dc0_uplink` + `vr1_dc0_uplink_cidr` + the 2nd NIC/netplan bridge); revert - `opentofu/vr1-dc0-substrate/main.tf` `vr1_dc0_wan` back to `site-wan` NAT `172.30.2.0/24`; revert the - `site-headend-install.sh` `--uplink-if`/`--wan-bridge` block + harness; revert the doc records. This +- `git rm -r opentofu/modules/wan-bridge`; revert `opentofu/main.tf` (drop `vr1_dc0_uplink` + the 2nd + NIC/netplan bridge); revert `opentofu/vr1-dc0-substrate/main.tf` `vr1_dc0_wan` back to `site-wan` NAT + `172.30.2.0/24`; revert the `site-headend-install.sh` `--uplink-if`/`--wan-bridge` block + harness; + revert the doc records. (No tfvar to drop -- the /24 is an inlined ruled literal.) This reverts D-125's bridge-in to the (dead-egress) committed Model B state -- the full B->A revert is `docs/model-a-fallback-plan.md`. diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md index 067e39e..9530ba6 100644 --- a/docs/dc-dc-deployment-workflow.md +++ b/docs/dc-dc-deployment-workflow.md @@ -184,9 +184,10 @@ inner applies. The stale `vr1-dc1` "supernet not assigned" gate is corrected (D-115 assigned `10.12.64.0/19`); vr1-dc1 stays out of scope by sequencing. -**Blocking to EXECUTE (all operator-gated):** the HELD NetBox addressing -- rack transit/30 + rack IP -(D-124), `vr1_dc0_uplink_cidr` + OPNsense WAN (D-125) -- via office1-netbox (apex token); then the gated -applies; then the deploy-time gates (depth-4 nova boot D-114/D-123; D-125 foreign-MAC egress isolation +**Blocking to EXECUTE (all operator-gated):** the HELD NetBox addressing -- **D-124's rack transit/30 + +rack metal-admin IP** (via office1-netbox, apex token; D-125 needs NO new NetBox object -- its /24 is the +ruled `172.30.2.0/24`, OPNsense WAN stays `.2`); then the gated applies; then the deploy-time gates +(depth-4 nova boot D-114/D-123; D-125 foreign-MAC egress isolation test; SEC-010 packet-drop). --- diff --git a/docs/design-decisions.md b/docs/design-decisions.md index ce9885a..d018675 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -3528,14 +3528,17 @@ DCs; the operator's 2026-07-16 Model B ruling (D-123) nests the DC nodes inside `vvr1-dc0`, so a single `virsh destroy vvr1-dc0` IS a true DC site-down -- D-122's original intent is vindicated, not amended. -- **(D-125) The WAN-uplink bullet's REALIZATION is superseded by D-125** (bridge-in). This entry's - "WAN = the dedicated per-site ISP uplink `vr1-dc0-wan` (`172.30.2.0/24`, D-115)... attach the DC edge's - WAN there" describes a NAT'd `/24` at the site level -- which under D-123 Model B nesting would have no - egress (OBS-3). D-125 keeps D-122's **intent** (each DC has its own dedicated ISP uplink; the DC edge's - WAN attaches there) but changes the MECHANISM: `vr1-dc0-wan` is now a libvirt BRIDGE - (`modules/wan-bridge`) and the per-DC simulated-ISP `/24` moves to a **vcloud-level** NAT - (`var.vr1_dc0_uplink_cidr`) that `vvr1-dc0` bridges out to. The old `172.30.2.0/24` (D-115) is UNUSED in - the Model B state (it returns only on a Model A revert). Intent stands; realization see D-125. +- **(D-125) The WAN-uplink bullet's REALIZATION is superseded by D-125** (bridge-in) -- but the + ADDRESSING is unchanged. This entry's "WAN = the dedicated per-site ISP uplink `vr1-dc0-wan` + (`172.30.2.0/24`, D-115)... attach the DC edge's WAN there" describes a NAT'd `/24` at the site level -- + which under D-123 Model B nesting would have no egress (OBS-3). D-125 keeps D-122's **intent** (each DC + has its own dedicated ISP uplink; the DC edge's WAN attaches there) AND the same `/24` + (`172.30.2.0/24`, D-115): bridge-in is single-NAT, so there is exactly ONE simulated-ISP /24 and its + IPAM identity is UNCHANGED from Model A. Only the MECHANISM moved -- `vr1-dc0-wan` is now a libvirt + BRIDGE (`modules/wan-bridge`) onto a **vcloud-level** NAT (`module "vr1_dc0_uplink"`, cidr `172.30.2.0/24` + inlined) that `vvr1-dc0` bridges out to; the OPNsense WAN keeps its static `.2`. No new /24, no + re-address, no new NetBox object. (An earlier D-125 draft wrongly said the /24 "moves to a new + vcloud-level NAT" and "the old 172.30.2.0/24 is UNUSED" -- corrected 2026-07-16.) ## D-123: VR1 DC site-down DR model -- node-VM placement + the per-DC MAAS headend (resolves a D-122/D-103 tension) @@ -3707,8 +3710,9 @@ WAN segment becomes a **bridge**, not a NAT, so the inner OPNsense edge's WAN attaches through `vvr1-dc0` to that ISP network and there is exactly ONE NAT (at vcloud = the simulated ISP hand-off). Concretely: -- **Outer (vcloud):** a new `site-wan` NAT instance `vr1-dc0-uplink` on its own `/24` (the simulated ISP; - NetBox-assigned, HELD -- see below), plus a second NIC on `vvr1-dc0` attached to it. `vvr1-dc0` NIC order +- **Outer (vcloud):** a new `site-wan` NAT instance `vr1-dc0-uplink` on `172.30.2.0/24` (the simulated + ISP -- the SAME /24 D-115 assigned vr1-dc0; a RULED literal, see Addressing below), plus a second NIC on + `vvr1-dc0` attached to it. `vvr1-dc0` NIC order = **[transit, uplink]** -- transit STAYS first so SEC-010/`--transit-if` keep keying on it; the uplink is an **IP-less bridge port** on `vvr1-dc0`. - **`vvr1-dc0` netplan (cloud-init, declarative):** the uplink NIC (`enp2s0` -> `uplink`, IP-less) @@ -3743,11 +3747,14 @@ `ip_forward=1` (there is no inner WAN NAT to forward), but SEC-010's scoped FORWARD-drop is RETAINED as-is (correct and harmless either way); see the security ledger and `references/platform-traps.md`. -**NEW HELD gates (hard rule 2 -- no invented values):** -- `vr1_dc0_uplink_cidr` -- the vcloud ISP `/24` (NetBox `office1-netbox`, apex-held). Variable has NO - default; `tofu plan` requires the tfvars, mirroring the D-124 rack-IP pattern. -- **OPNsense WAN re-address** -- moves from `vr1-dc0-wan`'s old `172.30.2.2` (D-115) into the vcloud ISP - `/24`; set in the edge config (D-113), not tofu. HELD until the `/24` is assigned. +**Addressing: RULED, not HELD (corrected 2026-07-16).** The vcloud ISP `/24` is the SAME `172.30.2.0/24` +D-115 already assigned vr1-dc0's edge WAN (role edge, site vr1-dc0) -- bridge-in is single-NAT, so there +is one /24 and its IPAM identity is unchanged from Model A. It is therefore a RULED literal inlined in +`module "vr1_dc0_uplink"` (cidr `172.30.2.0/24`), NOT a NetBox-assigned HELD tfvar. The OPNsense WAN keeps +its static `.2` (D-113 edge config) -- no re-address. **D-125 adds NO new NetBox object** (the `/24` is +already registered by `netbox/dc-edge-wan-import.py`), and it introduces NO new HELD gate. (An earlier +draft of this entry wrongly modeled a `var.vr1_dc0_uplink_cidr` HELD value + an OPNsense WAN re-address; +both removed 2026-07-16.) **Module-packaging note.** Implemented as a NEW sibling `modules/wan-bridge` rather than a `mode` flag on `modules/site-wan` (the option preview's phrasing): same ruled topology, but it keeps every existing diff --git a/docs/model-a-fallback-plan.md b/docs/model-a-fallback-plan.md index 5aa67ca..fe843dd 100644 --- a/docs/model-a-fallback-plan.md +++ b/docs/model-a-fallback-plan.md @@ -65,24 +65,25 @@ 7. **Inner root:** B adds `opentofu/vr1-dc0-substrate/` (a new root dir + its own state) and a `site-headend-install.sh --host-nodes` node-host bootstrap; A has neither. Revert removes them. 8. **Per-DC ISP egress (D-125 bridge-in):** because B nests `vr1-dc0-wan` inside `vvr1-dc0`, B adds a - vcloud-level ISP NAT (`module "vr1_dc0_uplink"`, `site-wan`), a 2nd IP-less uplink NIC + the - `br-vr1-dc0-wan` netplan bridge on `vvr1-dc0`, the new `modules/wan-bridge`, the bootstrap's - `--uplink-if`/`--wan-bridge` verify, and re-addresses the OPNsense WAN into the vcloud ISP `/24`. - **In Model A none of this exists** -- `vr1-dc0-wan` is a vcloud-level NAT the vcloud-level edge - attaches to directly, egressing via vcloud with no uplink NIC, no bridge, no `wan-bridge` module. - Revert removes all of it and restores the OPNsense WAN to `172.30.2.2` on the `vr1-dc0-wan` NAT. - (New HELD gates that vanish on revert: `vr1_dc0_uplink_cidr`, the OPNsense WAN re-address.) + vcloud-level ISP NAT (`module "vr1_dc0_uplink"`, `site-wan`, cidr `172.30.2.0/24`), a 2nd IP-less + uplink NIC + the `br-vr1-dc0-wan` netplan bridge on `vvr1-dc0`, the new `modules/wan-bridge`, and the + bootstrap's `--uplink-if`/`--wan-bridge` verify. **The ADDRESSING is identical to Model A** -- same + `172.30.2.0/24` (D-115), OPNsense WAN still `.2`; only the libvirt realization differs (bridge through + `vvr1-dc0` vs a direct vcloud-level NAT). **In Model A the extra plumbing does not exist** -- `vr1-dc0-wan` + is a vcloud-level NAT the vcloud-level edge attaches to directly, no uplink NIC, no bridge, no + `wan-bridge` module. Revert removes that plumbing; NO re-address is needed (the address never changed). + (No HELD gate here: the /24 is a ruled literal, not a tfvar.) ## 4. Revert procedure (if Model B deployment fails) 1. STOP -- do not attempt to fix Model B in place if nested-virt (depth-4) is the failure mode; that is the known risk this fallback exists for. 2. `git checkout model-a-fallback -- opentofu/main.tf opentofu/variables.tf opentofu/modules/` - (restores the Model A substrate verbatim -- this also drops `module "vr1_dc0_uplink"` and the - `vr1_dc0_uplink_cidr` var, since Model A has neither), then `git rm -r opentofu/vr1-dc0-substrate` - (the inner root does not exist in Model A) and `git rm -r opentofu/modules/wan-bridge` (D-125, - also absent in Model A), and revert the `site-headend-install.sh` node-host mode (incl. the D-125 - `--uplink-if`/`--wan-bridge` WAN-bridge verify). Restore the OPNsense WAN address to `172.30.2.2`. + (restores the Model A substrate verbatim -- this also drops `module "vr1_dc0_uplink"`, since Model A + has no vcloud uplink), then `git rm -r opentofu/vr1-dc0-substrate` (the inner root does not exist in + Model A) and `git rm -r opentofu/modules/wan-bridge` (D-125, also absent in Model A), and revert the + `site-headend-install.sh` node-host mode (incl. the D-125 `--uplink-if`/`--wan-bridge` WAN-bridge + verify). The OPNsense WAN address is UNCHANGED (`.2` on `172.30.2.0/24`) -- nothing to restore. 3. `bash scripts/opentofu-validate.sh` -> expect 11/11 PASS (Model A already validates). 4. Re-instate D-103/D-114 as ADOPTED (they were only annotated superseded, not deleted -- see the sweep's supersession notes; revert removes those annotations). diff --git a/docs/session-ledger.md b/docs/session-ledger.md index ea7b369..70d3c5c 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -259,8 +259,11 @@ `modules/wan-bridge`) so OPNsense WAN rides to the single vcloud NAT. **Authored present-only; both roots validate; NOT closed -- the L2-NAT-of-foreign-MAC path is a DEPLOY-TIME gate (bridge-in equivalent of the L4 boot gate), fallback = double-NAT.** See D-125 + platform-traps 1g. - - **New HELD gates (hard rule 2):** `vr1_dc0_uplink_cidr` (vcloud ISP /24, office1-netbox apex) + the - OPNsense WAN re-address from `172.30.2.2` into that /24 (edge config, D-113). + - **No new HELD gate (CORRECTED 2026-07-16):** the vcloud ISP /24 is the SAME `172.30.2.0/24` D-115 + already ruled for vr1-dc0 (bridge-in is single-NAT; IPAM identity unchanged from Model A), inlined as + a ruled literal in `module "vr1_dc0_uplink"` -- NOT a NetBox-assigned tfvar. OPNsense WAN stays `.2`; + no re-address. D-125 adds NO NetBox object. (An earlier draft's `vr1_dc0_uplink_cidr` HELD var + + re-address were removed -- see the correction commit below.) - **br_netfilter constraint:** SEC-010 FORWARD-drop stays interface-scoped, NEVER globalized. `--check` hardened: fail-open guard now covers the transit-iface existence AND the WAN bridge/uplink enslavement. - Harness: `site-headend-install` 49->54; `wan-bridge` covered by opentofu-validate S3. Records: D-125, diff --git a/opentofu/main.tf b/opentofu/main.tf index e7acc48..4170da7 100644 --- a/opentofu/main.tf +++ b/opentofu/main.tf @@ -326,8 +326,12 @@ # FORWARD-dropped). vvr1-dc0 takes an IP-less uplink NIC on this network (below); the # inner vr1-dc0-wan is a BRIDGE (modules/wan-bridge) onto vvr1-dc0's br-vr1-dc0-wan, # so the inner OPNsense WAN lands directly on THIS /24 and there is one NAT, here. -# The /24 is NetBox-assigned (HELD, hard rule 2). Same site-wan NAT module as -# office1-wan -- 1500 MTU, no DHCP (edge WAN is static, D-113). +# The /24 is the SAME simulated-ISP WAN D-115 already assigned vr1-dc0 (172.30.2.0/24, +# role edge) -- bridge-in is single-NAT, so there is exactly ONE /24 and its IPAM +# identity is UNCHANGED from Model A; only the libvirt realization moved to vcloud. +# So this is a RULED literal, NOT a NetBox-assigned HELD value. OPNsense WAN stays .2 +# (D-113 edge config, unchanged). Same site-wan NAT module as office1-wan -- 1500 MTU, +# no DHCP (edge WAN is static). # DEPLOY-TIME GATE (D-125): that vcloud's virbr actually NATs OPNsense's foreign # MAC across the doubly-nested bridge is unprovable pre-apply -- the bridge-in # equivalent of the depth-4 boot gate. Isolation-test first (bootstrap prints it); @@ -335,7 +339,7 @@ module "vr1_dc0_uplink" { source = "./modules/site-wan" network_name = "vr1-dc0-uplink" - cidr = var.vr1_dc0_uplink_cidr + cidr = "172.30.2.0/24" # D-115 vr1-dc0 sim-ISP edge WAN; realized at vcloud under D-125 (ruled, not HELD) # mtu defaults to 1500 (the ISP-uplink domain; never jumbo). } diff --git a/opentofu/variables.tf b/opentofu/variables.tf index 12e924d..e688393 100644 --- a/opentofu/variables.tf +++ b/opentofu/variables.tf @@ -146,14 +146,9 @@ type = string } -# ---- D-125 (Model B bridge-in, closes OBS-3): the vcloud-level per-DC simulated -# ISP /24. This is the SINGLE NAT (the ISP hand-off), created at vcloud so it -# egresses via vcloud's uplink -- restoring the independent per-DC egress that -# Model B's inner nesting severed. vvr1-dc0 gets an IP-less uplink NIC on this -# network; the inner OPNsense WAN takes a STATIC address in this /24 (edge config, -# D-113). NetBox-assigned (office1-netbox, apex-held) -- NO default here, hard -# rule 2: `tofu plan` requires the tfvar, mirroring the D-124 rack IPs. ---- -variable "vr1_dc0_uplink_cidr" { - description = "vcloud-level per-DC simulated-ISP uplink /24 for vr1-dc0 (D-125). NetBox-assigned; gateway is .1, OPNsense WAN takes a static .2-band address here. No default (HELD, hard rule 2)." - type = string -} +# ---- D-125 (Model B bridge-in): the vr1-dc0 vcloud-level simulated-ISP uplink /24 +# is NOT a HELD variable -- it is the SAME /24 D-115 already ruled for vr1-dc0's edge +# WAN (172.30.2.0/24), since bridge-in is single-NAT and the /24's IPAM identity is +# unchanged from Model A. It is therefore a RULED literal inlined in main.tf's +# `module "vr1_dc0_uplink"`, not a tfvar. (An earlier D-125 draft wrongly modeled it +# as a new NetBox-assigned HELD value; corrected 2026-07-16.) ---- diff --git a/opentofu/vr1-dc0-substrate/main.tf b/opentofu/vr1-dc0-substrate/main.tf index bf517cc..8b4386c 100644 --- a/opentofu/vr1-dc0-substrate/main.tf +++ b/opentofu/vr1-dc0-substrate/main.tf @@ -37,10 +37,11 @@ # Under D-123 Model B this network is created INSIDE vvr1-dc0, whose only routed leg is # the East-West-only transit (SEC-010 FORWARD-dropped), so a NAT here would have no egress # (that was OBS-3). Instead it bridges onto vvr1-dc0's br-vr1-dc0-wan (an IP-less uplink -# NIC on the vcloud ISP NAT, declared in the outer root's vvr1-dc0 netplan). The OPNsense -# WAN then takes a STATIC address on the vcloud ISP /24 (var.vr1_dc0_uplink_cidr, set in the -# edge config per D-113 -- HELD until that /24 is NetBox-assigned) and there is ONE NAT, at -# vcloud. host_bridge MUST match the netplan bridge name in opentofu/main.tf. +# NIC on the vcloud ISP NAT `vr1-dc0-uplink` = 172.30.2.0/24, declared in the outer root's +# vvr1-dc0 netplan). The OPNsense WAN keeps its static .2 on that SAME /24 D-115 already +# assigned vr1-dc0 (bridge-in is single-NAT; the /24 is UNCHANGED from Model A, only the +# libvirt host moved) -- set in the edge config (D-113), no re-address, no new NetBox object. +# There is ONE NAT, at vcloud. host_bridge MUST match the netplan bridge name in opentofu/main.tf. module "vr1_dc0_wan" { source = "../modules/wan-bridge" network_name = "vr1-dc0-wan" diff --git a/runbooks/dc-dc-phase2-tofu-dc-substrate.md b/runbooks/dc-dc-phase2-tofu-dc-substrate.md index 1bcae28..23d397d 100644 --- a/runbooks/dc-dc-phase2-tofu-dc-substrate.md +++ b/runbooks/dc-dc-phase2-tofu-dc-substrate.md @@ -172,12 +172,12 @@ **D-125 (bridge-in per-DC ISP egress -- closes OBS-3).** Under Model B the inner `vr1-dc0-wan` was a NAT with no egress (its host `vvr1-dc0` is transit-only). Fix, folded into the roots above: -- **A (OUTER)** ALSO creates the vcloud ISP NAT `vr1-dc0-uplink` (HELD `var.vr1_dc0_uplink_cidr` -- assign - the /24 in office1-netbox before `tofu plan`) + a 2nd IP-less uplink NIC + the `br-vr1-dc0-wan` netplan - bridge on `vvr1-dc0`. +- **A (OUTER)** ALSO creates the vcloud ISP NAT `vr1-dc0-uplink` on `172.30.2.0/24` (the SAME /24 D-115 + ruled for vr1-dc0's edge WAN -- a ruled literal, NOT a HELD tfvar; no new NetBox object) + a 2nd IP-less + uplink NIC + the `br-vr1-dc0-wan` netplan bridge on `vvr1-dc0`. - **B (BOOTSTRAP)** `--check` ALSO verifies `br-vr1-dc0-wan` exists with `uplink` enslaved (fail-open guard). - **C (INNER)** `vr1-dc0-wan` is now a BRIDGE (`modules/wan-bridge`) onto `br-vr1-dc0-wan`; the OPNsense - WAN takes a static address in the vcloud ISP /24 (edge config, D-113 -- HELD). + WAN keeps its static `.2` on that same /24 (edge config, D-113 -- unchanged, no re-address). - **DEPLOY-TIME GATE (Step 12):** before trusting the OPNsense chain, isolation-test egress -- a throwaway guest on `br-vr1-dc0-wan` must get a vcloud-ISP address + ping out. FAIL => revert to the D-125 double-NAT fallback (not a redesign). See D-125 + `docs/changelog-20260716-review-sweep-phaseC2-D125.md`.