diff --git a/.claude/skills/openstack-cloud-ops/references/platform-traps.md b/.claude/skills/openstack-cloud-ops/references/platform-traps.md index 44419f9..b8f2910 100644 --- a/.claude/skills/openstack-cloud-ops/references/platform-traps.md +++ b/.claude/skills/openstack-cloud-ops/references/platform-traps.md @@ -184,6 +184,35 @@ Before relying on either, read the real schema: `tofu providers schema -json | jq '.provider_schemas[].resource_schemas.libvirt_domain'`. +### 1g. `forward=bridge` networks + `br_netfilter`: a global FORWARD-drop silently kills bridged frames (D-125) + +A `libvirt_network` with `forward = { mode = "bridge" }` (dmacvicar schema: +`bridge = { name = "" }`, no `ips`, no NAT) attaches guest taps to a +**pre-existing** host bridge -- libvirt does NOT create the bridge, so it must +already exist (declare it in the host's netplan, not the tofu) or the network +fails to start. This is the Model B "bridge-in" WAN path (D-125): the inner +OPNsense WAN rides `br-vr1-dc0-wan` out through `vvr1-dc0`'s IP-less uplink NIC to +the single vcloud ISP NAT. + +Two silent-failure traps ride along: + +- **`br_netfilter` makes bridged frames traverse the L3 `FORWARD` chain.** So a + GLOBAL forward-drop -- `iptables -P FORWARD DROP`, or an nftables forward hook + with `policy drop` and no interface match -- silently kills the WAN bridge even + though it is "just L2". Keep any DC-LOCAL forward-drop (SEC-010) **scoped to the + transit interface** (`oifname/iifname ""`), never global. Note + `net.ipv4.ip_forward=0` is L3-only and does NOT affect pure bridging -- do not + reach for it expecting to gate a bridge, and do not assume `ip_forward=1` is + required just because an older NAT-based design needed it. +- **Foreign-MAC egress across a nested bridge is unprovable until apply.** When the + bridged guest is itself inside another guest (OPNsense inside `vvr1-dc0`), the + outer NAT's `virbr` sees a foreign source MAC arriving on the containment VM's + port; an anti-spoof/nwfilter drop there blackholes egress. `tofu validate` is + clean regardless. TEST IN ISOLATION FIRST: attach a throwaway guest to the + bridge, confirm it gets an address and pings out, BEFORE the real chain rides + it. (D-125's fallback for this failure is double-NAT: masquerade at `vvr1-dc0` + instead of relying on nested L2.) + ## 2. AppArmor blocks qemu on any non-default pool path libvirt's stock `abstractions/libvirt-qemu` grants qemu the DEFAULT pool path diff --git a/docs/changelog-20260716-review-sweep-phaseC2-D125.md b/docs/changelog-20260716-review-sweep-phaseC2-D125.md new file mode 100644 index 0000000..02c69ca --- /dev/null +++ b/docs/changelog-20260716-review-sweep-phaseC2-D125.md @@ -0,0 +1,70 @@ +# 2026-07-16 -- D-125: Model B per-DC ISP egress (bridge-in, single-NAT) -- resolves OBS-3's design gap; egress efficacy is a deploy-time gate (present-only) + +## Context + +Post-commit `5175365` (Stage-3 sweep C+D), the adversarial re-run + advisor confirmed **OBS-3** as +material once the operator ruled **"DC0 and DC1 have their own ISP connections."** Model B nesting had +severed the per-DC simulated-ISP egress: `modules/site-wan` makes `vr1-dc0-wan` a libvirt NAT that +egresses via ITS HOST's default route; under Model B that host is `vvr1-dc0` (transit-only, East-West, +SEC-010 FORWARD-dropped), so the inner NAT had nowhere to go. Operator ruled **bridge-in (single-NAT)** +over the double-NAT alternative. Governing record: **D-125** (cross-ref D-122/D-123). NO cloud mutation; +authored present-only; both roots validate. **NOT applied, NOT committed, NOT pushed.** + +## What changed + +**New module `opentofu/modules/wan-bridge/` (versions/variables/main.tf/outputs.tf).** A libvirt +`forward = { mode = "bridge" }` network bound to a pre-existing host bridge (`bridge = { name = ... }`), +no subnet/NAT. Sibling to `site-wan` (chosen over a `site-wan` `mode` flag so every existing NAT user -- +`office1-wan`, the new `vr1-dc0-uplink` -- stays literally untouched; noted vs. the option preview's +"mode param" wording). Schema (`bridge` is an attribute-OBJECT, not a string) measured from the installed +provider, not inferred. Covered by `opentofu-validate` S3 (standalone `[PASS]`). + +**OUTER root (`opentofu/main.tf` + `variables.tf`, vcloud).** +- New `module "vr1_dc0_uplink"` (`site-wan` NAT) = the SINGLE ISP NAT at vcloud level (egresses via + vcloud's uplink). CIDR = new HELD `var.vr1_dc0_uplink_cidr` (no default; NetBox office1-netbox apex). +- `vvr1_dc0` gains a 2nd NIC (order `[transit, uplink]` -- transit stays first so SEC-010/`--transit-if` + keep keying on it). Netplan now declares the IP-less `uplink` port + the `br-vr1-dc0-wan` bridge + (STP off), so the bridge exists before the inner apply. `tofu validate` Success. + +**INNER root (`opentofu/vr1-dc0-substrate/main.tf`).** `module "vr1_dc0_wan"` switched +`site-wan` (NAT, `172.30.2.0/24`) -> `wan-bridge` bound to `br-vr1-dc0-wan`. OPNsense `wan_network_name` +wiring unchanged (same output). OPNsense WAN re-addresses into the vcloud ISP /24 (edge config, D-113 -- +HELD). `tofu validate` Success. + +**Bootstrap (`scripts/site-headend-install.sh` node-host mode).** +- New `--uplink-if` (default `uplink`) + `--wan-bridge` (default `br-vr1-dc0-wan`). +- `node_host_check`: WAN-egress guard -- verifies the bridge exists AND the uplink is ENSLAVED (`master + `); same fail-open class as the SEC-010 transit-iface check. Prints the DEPLOY-TIME isolation + gate (throwaway guest on the bridge must get a vcloud-ISP addr + ping out; FAIL => double-NAT fallback). +- `node_host_setup`: SEC-010 rationale corrected (bridge-in has no inner NAT, so the old ip_forward + forcing no longer applies); **br_netfilter CONSTRAINT** -- keep the FORWARD-drop interface-scoped, + never global. Verifies the WAN bridge landed (does NOT create it; netplan owns L2). Harness 49 -> 54. + +**Records.** D-125 (new); `security-ledger` SEC-010 (bridge-in + br_netfilter constraint + hardened +`--check`); `model-a-fallback-plan` B->A delta item 8 + revert step + OPNsense WAN restore; `session-ledger` +(commit `5175365` + OBS-3 RESOLVED-by-D-125 + new HELD gates); `references/platform-traps.md` 1g (bridge ++ br_netfilter + nested-foreign-MAC trap). + +## Verification + +- `tofu validate` OUTER = Success; INNER = Success; `opentofu-validate` S3 `[PASS] modules/wan-bridge` + (+ `site-wan` still `[PASS]`, untouched). +- `tests/site-headend-install` = 54/54. +- `repo-lint` = 0 fail (1 documented legacy WARN). All edited files ASCII/LF. + +## Deploy-time gates opened (NOT closed by this change) + +1. **L2-NAT of foreign MAC across the nested bridge** -- unprovable pre-apply (bridge-in's L4-boot-gate + equivalent). Isolation-test first; FAIL => revert to double-NAT (D-125). +2. **HELD:** assign `vr1_dc0_uplink_cidr` (vcloud ISP /24) in office1-netbox; set OPNsense WAN static in + that /24 (edge config). `tofu plan` requires the tfvar. +3. Confirm the `uplink`/`br-vr1-dc0-wan` interface names on first boot (enp2s0-vs-ensX naming trap). + +## Revert (per item) + +- `git rm -r opentofu/modules/wan-bridge`; revert `opentofu/main.tf` + `variables.tf` (drop + `vr1_dc0_uplink` + `vr1_dc0_uplink_cidr` + the 2nd NIC/netplan bridge); revert + `opentofu/vr1-dc0-substrate/main.tf` `vr1_dc0_wan` back to `site-wan` NAT `172.30.2.0/24`; revert the + `site-headend-install.sh` `--uplink-if`/`--wan-bridge` block + harness; revert the doc records. This + reverts D-125's bridge-in to the (dead-egress) committed Model B state -- the full B->A revert is + `docs/model-a-fallback-plan.md`. diff --git a/docs/design-decisions.md b/docs/design-decisions.md index 33272a4..ce9885a 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -3516,7 +3516,7 @@ core-service provider), DOCFIX-185 (edge is a real-ISP router, not an egress airgap), D-113(a2) (edge config over REST). DOCFIX-185's D-100/D-107 amendment note is subsumed here. -### D-122 -- AMENDMENT (2026-07-16): MAAS-controller bullet superseded (C1); single-object site-down vindicated by D-123 Model B (C3) +### D-122 -- AMENDMENT (2026-07-16): MAAS-controller bullet superseded (C1); single-object site-down vindicated by D-123 Model B (C3); WAN-uplink realization superseded by D-125 Two record corrections from the review (R3-F04): - **(C1) The "Each site runs its own MAAS controller (as voffice1 does)" bullet is SUPERSEDED** by @@ -3528,6 +3528,14 @@ DCs; the operator's 2026-07-16 Model B ruling (D-123) nests the DC nodes inside `vvr1-dc0`, so a single `virsh destroy vvr1-dc0` IS a true DC site-down -- D-122's original intent is vindicated, not amended. +- **(D-125) The WAN-uplink bullet's REALIZATION is superseded by D-125** (bridge-in). This entry's + "WAN = the dedicated per-site ISP uplink `vr1-dc0-wan` (`172.30.2.0/24`, D-115)... attach the DC edge's + WAN there" describes a NAT'd `/24` at the site level -- which under D-123 Model B nesting would have no + egress (OBS-3). D-125 keeps D-122's **intent** (each DC has its own dedicated ISP uplink; the DC edge's + WAN attaches there) but changes the MECHANISM: `vr1-dc0-wan` is now a libvirt BRIDGE + (`modules/wan-bridge`) and the per-DC simulated-ISP `/24` moves to a **vcloud-level** NAT + (`var.vr1_dc0_uplink_cidr`) that `vvr1-dc0` bridges out to. The old `172.30.2.0/24` (D-115) is UNUSED in + the Model B state (it returns only on a Model A revert). Intent stands; realization see D-125. ## D-123: VR1 DC site-down DR model -- node-VM placement + the per-DC MAAS headend (resolves a D-122/D-103 tension) @@ -3678,3 +3686,76 @@ GiB VM) with `expose_nested_virt = true`. **What still stands:** the transit addressing (Scheme A -- office1<->dc0 mesh transit /30 + rack metal-admin IP) is unchanged; the containment VM still straddles metal-admin (node-facing) + the transit (region-facing), now ALSO hosting the nodes. + +## D-125: VR1 Model B per-DC ISP egress -- bridge-in single-NAT (resolves OBS-3's design gap; egress efficacy is a deploy-time gate) + +**Status:** ADOPTED (operator ruling 2026-07-16 -- "DC0 and DC1 are supposed to have ISP connections", +then chose bridge-in over double-NAT). Governs how each DC's simulated ISP uplink (`vr1-dc0-wan`, D-122) +egresses under D-123 **Model B**. Authored present-only; the L2-NAT path is a DEPLOY-TIME gate (see below) +-- this entry does NOT assert the egress works, only how it is wired. + +**The defect this fixes (OBS-3).** `modules/site-wan` builds `vr1-dc0-wan` as a libvirt **NAT** network, +which egresses via ITS HOST's default route. Under Model A that host was **vcloud** (masquerade out vcloud's +uplink -> real egress; D-122's dedicated per-site L3 ISP satisfied). D-123 Model B moved `vr1-dc0-wan` to +the **inner** provider -- it is now created INSIDE `vvr1-dc0`, whose only outer NIC is the office1<->dc0 +transit leg (East-West-only, D-122; and SEC-010 FORWARD-drops it). So the inner NAT had no egress: the DC +edge's ISP was dead. This was logged as OBS-3 in the Stage-3 review and confirmed material by the operator. + +**Decision (operator-ruled): bridge-in, single NAT.** Restore independent per-DC egress by giving `vvr1-dc0` +its own **WAN uplink NIC** (a second leg -- one that never existed in Model A, where the vcloud-level edge +attached to the vcloud-level WAN directly) onto a **vcloud-level per-DC ISP NAT** network. The DC's inner +WAN segment becomes a **bridge**, not a NAT, so the inner OPNsense edge's WAN attaches through `vvr1-dc0` to +that ISP network and there is exactly ONE NAT (at vcloud = the simulated ISP hand-off). Concretely: + +- **Outer (vcloud):** a new `site-wan` NAT instance `vr1-dc0-uplink` on its own `/24` (the simulated ISP; + NetBox-assigned, HELD -- see below), plus a second NIC on `vvr1-dc0` attached to it. `vvr1-dc0` NIC order + = **[transit, uplink]** -- transit STAYS first so SEC-010/`--transit-if` keep keying on it; the uplink is + an **IP-less bridge port** on `vvr1-dc0`. +- **`vvr1-dc0` netplan (cloud-init, declarative):** the uplink NIC (`enp2s0` -> `uplink`, IP-less) + AND the bridge `br-vr1-dc0-wan` enslaving it are declared in `vvr1-dc0`'s cloud-init netplan (no IP + on the `vvr1-dc0` side), so the bridge exists before the inner apply. The node-host bootstrap does + NOT create it -- it only VERIFIES the bridge + uplink exist, grants libvirt/AppArmor access, and + isolation-tests egress. (Declarative netplan avoids a networkd-vs-script fight and maps cleanly to + "the DC host's netplan defines the WAN bridge to its ISP NIC" on Roosevelt.) +- **Inner (`vvr1-dc0`):** `vr1-dc0-wan` is realized by the new `modules/wan-bridge` (libvirt + `forward = { mode = "bridge" }`, bound to `br-vr1-dc0-wan`, no subnet/NAT) instead of `site-wan`. OPNsense + WAN then gets a static address in the **vcloud ISP `/24`** and default-routes to that `.1` gateway. + +**Rejected alternative -- double-NAT (kept as the deploy-time fallback).** Inner `vr1-dc0-wan` stays a NAT +and `vvr1-dc0` masquerades it out the uplink NIC into the vcloud ISP NAT. Robust (routes/masquerades AT +`vvr1-dc0`, no reliance on nested L2) but adds a third NAT hop and lower ISP-handoff fidelity. If the +bridge-in L2 path fails at apply, revert to this -- it is the safe fallback and needs only a mode change on +the inner WAN + `ip_forward`/masquerade on `vvr1-dc0`. + +**DEPLOY-TIME GATE (unprovable pre-apply -- the bridge-in equivalent of the D-114 depth-4 boot gate).** +`tofu validate` is clean, but whether **vcloud's libvirt NAT actually NATs OPNsense's foreign-MAC frames +across the doubly-nested bridge** (OPNsense MAC on `vvr1-dc0`'s uplink tap -> `br-vr1-dc0-wan` -> vcloud +`virbr`) is only knowable at apply: vcloud's `virbr` must accept the foreign source MAC (no nwfilter/ +anti-spoof drop), let OPNsense ARP-resolve `.1`, and masquerade the `/24`. **Validate in isolation FIRST** +(the bootstrap prints the gate): a throwaway guest on `br-vr1-dc0-wan` must get a vcloud-ISP address and +ping out BEFORE the OPNsense chain rides on it. Failure here => revert to double-NAT (above), not a redesign. + +**CONSTRAINT -- `br_netfilter`: do NOT globalize the FORWARD drop.** With `br_netfilter` loaded, bridged WAN +frames traverse the L3 `FORWARD` chain, so a **global** FORWARD policy-drop (or a blanket `iptables -P +FORWARD DROP`) would silently kill the WAN bridge. SEC-010 therefore MUST stay **interface-scoped to the +transit leg** (its current form) -- never globalized. Note `ip_forward=0` is L3-only and does NOT affect +pure L2 bridging; the two are easy to conflate. Corollary: bridge-in removes Model B's earlier reliance on +`ip_forward=1` (there is no inner WAN NAT to forward), but SEC-010's scoped FORWARD-drop is RETAINED as-is +(correct and harmless either way); see the security ledger and `references/platform-traps.md`. + +**NEW HELD gates (hard rule 2 -- no invented values):** +- `vr1_dc0_uplink_cidr` -- the vcloud ISP `/24` (NetBox `office1-netbox`, apex-held). Variable has NO + default; `tofu plan` requires the tfvars, mirroring the D-124 rack-IP pattern. +- **OPNsense WAN re-address** -- moves from `vr1-dc0-wan`'s old `172.30.2.2` (D-115) into the vcloud ISP + `/24`; set in the edge config (D-113), not tofu. HELD until the `/24` is assigned. + +**Module-packaging note.** Implemented as a NEW sibling `modules/wan-bridge` rather than a `mode` flag on +`modules/site-wan` (the option preview's phrasing): same ruled topology, but it keeps every existing +`site-wan` NAT user (`office1-wan`, the new `vr1-dc0-uplink`) literally untouched and avoids inverting +`site-wan`'s NAT-specific name/doc. See the phase-C2 changelog. + +**Related:** D-122 (dedicated per-site L3 ISP -- this is its Model B realization), D-123 (Model B nesting, +which created the defect), D-115 (`vr1-dc0-wan` 172.30.2.0/24 -- the old inner subnet, now the bridge), +D-113 (edge config over REST -- OPNsense WAN static addr), SEC-010 (transit FORWARD-drop, scoped), +DOCFIX-185 (edge is a real-ISP router, not an egress airgap). **Fallback:** `docs/model-a-fallback-plan.md` +section 3 (revert removes the uplink NIC/network/bridge + `wan-bridge` and restores the OPNsense WAN addr). diff --git a/docs/model-a-fallback-plan.md b/docs/model-a-fallback-plan.md index c9a0ab6..5aa67ca 100644 --- a/docs/model-a-fallback-plan.md +++ b/docs/model-a-fallback-plan.md @@ -64,14 +64,25 @@ 6. **Site-down primitive:** B = one `virsh destroy vvr1-dc0`; A = scripted group-destroy. 7. **Inner root:** B adds `opentofu/vr1-dc0-substrate/` (a new root dir + its own state) and a `site-headend-install.sh --host-nodes` node-host bootstrap; A has neither. Revert removes them. +8. **Per-DC ISP egress (D-125 bridge-in):** because B nests `vr1-dc0-wan` inside `vvr1-dc0`, B adds a + vcloud-level ISP NAT (`module "vr1_dc0_uplink"`, `site-wan`), a 2nd IP-less uplink NIC + the + `br-vr1-dc0-wan` netplan bridge on `vvr1-dc0`, the new `modules/wan-bridge`, the bootstrap's + `--uplink-if`/`--wan-bridge` verify, and re-addresses the OPNsense WAN into the vcloud ISP `/24`. + **In Model A none of this exists** -- `vr1-dc0-wan` is a vcloud-level NAT the vcloud-level edge + attaches to directly, egressing via vcloud with no uplink NIC, no bridge, no `wan-bridge` module. + Revert removes all of it and restores the OPNsense WAN to `172.30.2.2` on the `vr1-dc0-wan` NAT. + (New HELD gates that vanish on revert: `vr1_dc0_uplink_cidr`, the OPNsense WAN re-address.) ## 4. Revert procedure (if Model B deployment fails) 1. STOP -- do not attempt to fix Model B in place if nested-virt (depth-4) is the failure mode; that is the known risk this fallback exists for. 2. `git checkout model-a-fallback -- opentofu/main.tf opentofu/variables.tf opentofu/modules/` - (restores the Model A substrate verbatim), then `git rm -r opentofu/vr1-dc0-substrate` (the inner - root does not exist in Model A) and revert the `site-headend-install.sh` node-host mode. + (restores the Model A substrate verbatim -- this also drops `module "vr1_dc0_uplink"` and the + `vr1_dc0_uplink_cidr` var, since Model A has neither), then `git rm -r opentofu/vr1-dc0-substrate` + (the inner root does not exist in Model A) and `git rm -r opentofu/modules/wan-bridge` (D-125, + also absent in Model A), and revert the `site-headend-install.sh` node-host mode (incl. the D-125 + `--uplink-if`/`--wan-bridge` WAN-bridge verify). Restore the OPNsense WAN address to `172.30.2.2`. 3. `bash scripts/opentofu-validate.sh` -> expect 11/11 PASS (Model A already validates). 4. Re-instate D-103/D-114 as ADOPTED (they were only annotated superseded, not deleted -- see the sweep's supersession notes; revert removes those annotations). diff --git a/docs/security-ledger.md b/docs/security-ledger.md index e7fabef..1aedfe8 100644 --- a/docs/security-ledger.md +++ b/docs/security-ledger.md @@ -18,7 +18,7 @@ | SEC-007 | 2026-07-12 | `~/vr1-office1-creds/` on the jumphost holds the Office1 edge root password + its bcrypt hash + the `office1_svc` SSH PRIVATE key + (added 2026-07-13) the OPNsense **API key/secret** (`opnsense-api.txt`, 0600), all in plaintext (dir 0700, files 0600) | 2026-07-12 edge build; custody detail deliberately not recorded here per D-069 | operator | OPEN -- required for edge management (D-112(c) makes SSH the only management path), so this is a ROTATION obligation, not a delete-me. Rotate at v1 close / if the jumphost is rebuilt or shared | | SEC-008 | 2026-07-13 | `~/vr1-office1-creds/tailscale-authkey.txt` (vcloud, 0600) -- the operator's 48-char Tailscale auth key for the SELF-HOSTED control plane `tailscale.baldurkeep.com`. Used to enrol `office1-tailscale`. Consumed BY PATH and never printed into a session; the copy shipped to the node was SHREDDED after `tailscale up` (tailscaled holds its own node key now). | 2026-07-13 Office1 Tailscale enrolment; docs/vr1-office1-as-built.md | operator | OPEN -- ROTATION obligation, not a delete-me: the key is still on vcloud for re-enrolment. Revoke/reissue on the control server at v1 close, or immediately if vcloud is rebuilt or shared. | | SEC-009 | 2026-07-15 | Credential/env SPRAWL + world-readable exposure on vcloud: three env files sat loose in `~` outside the consolidated `~/vr1-office1-creds/` -- `.vr1-netbox.env` (upstream NetBox token), `vr1-office1.env` (edge root-password hash + SSH key path), and `vr1-stage1.env` which held `TF_VAR_maas_api_key` (a MAAS API secret) at mode **0664 (group/world-readable)**; `tailscale-authkey.txt` in the creds dir was also 0664. | docs/changelog-20260715-creds-consolidation.md | operator | **REMEDIATED 2026-07-15** -- all three moved into `~/vr1-office1-creds/`, every sensitive file `chmod 600`. STANDING CONVENTION established (below). Note: this is a CLOSED in-house test; the rotation obligations of SEC-005/006/007 still apply at v1 close, this row only closes the SPRAWL + PERMS exposure. | -| SEC-010 | 2026-07-16 | **metal-admin DC-LOCAL invariant (D-052/D-100) is PRESERVED but NOT ENFORCED in the committed Stage-3 config.** The `vvr1_dc0` rack straddles metal-admin (10.12.8.0/22, DC-local) + the office1<->dc0 transit (crosses fiber). Its committed cloud-init pins static IPs only -- **no `net.ipv4.ip_forward=0` sysctl, no host firewall on the transit leg.** Cross-plane routing (metal-admin <-> the whole Office1 /22, bidirectional) is blocked ONLY by Ubuntu's distro default; the deferred MAAS-rack install could silently flip it. A MAAS rack proxies at the application layer and needs no kernel forwarding, so pinning is free. **MODEL B UPDATE (2026-07-16):** after the D-123 Model B reshape, the OUTER `vvr1-dc0` is single-leg (transit only); metal-admin + the other 5 planes are now INNER bridges on `vvr1-dc0`'s own libvirtd. The forwarding hardening is MORE critical (vvr1-dc0 now bridges ALL 6 inner planes + the transit) and belongs on the inner libvirt host, not just a 2-leg rack -- wire it in the C3 bootstrap (`site-headend-install.sh` node-host mode). | 2026-07-16 plane-segregation review + Model B reshape cross-check; `opentofu/main.tf` `module "vvr1_dc0"` | operator | **OPEN -- artifact COMMITTED (Phase D 2026-07-16), gated on apply+verify.** Enforced via a FORWARD-drop across the transit leg, NOT a global `ip_forward=0` (the inner `vr1-dc0-wan` NAT forces `ip_forward=1` on vvr1-dc0, so a global sysctl is unusable under Model B). `scripts/site-headend-install.sh --host-nodes` writes `/etc/nftables-sec010.nft` (drop FORWARD in+out the transit interface, default `mgmt`) + a boot-persistent `sec010-fw.service`; `--host-nodes --check` is the MECHANICAL pre-apply gate (fails if the rule is absent). Nothing routes across the fiber THROUGH vvr1-dc0; the rack proxies MAAS at the app layer (originated/terminated, not forwarded). Same pin follows onto `voffice1` when its transit leg is wired. Region route must target only the rack transit /30, never 10.12.8.0/22. Stays OPEN until applied + verified on vvr1-dc0. | +| SEC-010 | 2026-07-16 | **metal-admin DC-LOCAL invariant (D-052/D-100) is PRESERVED but NOT ENFORCED in the committed Stage-3 config.** The `vvr1_dc0` rack straddles metal-admin (10.12.8.0/22, DC-local) + the office1<->dc0 transit (crosses fiber). Its committed cloud-init pins static IPs only -- **no `net.ipv4.ip_forward=0` sysctl, no host firewall on the transit leg.** Cross-plane routing (metal-admin <-> the whole Office1 /22, bidirectional) is blocked ONLY by Ubuntu's distro default; the deferred MAAS-rack install could silently flip it. A MAAS rack proxies at the application layer and needs no kernel forwarding, so pinning is free. **MODEL B UPDATE (2026-07-16):** after the D-123 Model B reshape, the OUTER `vvr1-dc0` is single-leg (transit only); metal-admin + the other 5 planes are now INNER bridges on `vvr1-dc0`'s own libvirtd. The forwarding hardening is MORE critical (vvr1-dc0 now bridges ALL 6 inner planes + the transit) and belongs on the inner libvirt host, not just a 2-leg rack -- wire it in the C3 bootstrap (`site-headend-install.sh` node-host mode). | 2026-07-16 plane-segregation review + Model B reshape cross-check; `opentofu/main.tf` `module "vvr1_dc0"` | operator | **OPEN -- artifact COMMITTED (Phase D 2026-07-16), gated on apply+verify.** Enforced via a FORWARD-drop across the transit leg, NOT a global `ip_forward=0`. `scripts/site-headend-install.sh --host-nodes` writes `/etc/nftables-sec010.nft` (drop FORWARD in+out the transit interface, default `mgmt`) + a boot-persistent `sec010-fw.service`; `--host-nodes --check` is the MECHANICAL pre-apply gate (fails if the rule is absent, AND -- hardened 2026-07-16 -- if the keyed transit interface does not EXIST, since an nftables oifname on an absent iface loads clean but matches nothing = fail-open). **D-125 UPDATE (2026-07-16):** the earlier rationale (a global `ip_forward=0` being unusable because the inner `vr1-dc0-wan` NAT forced `ip_forward=1`) NO LONGER applies -- under bridge-in the inner `vr1-dc0-wan` is a BRIDGE (no inner NAT). The scoped FORWARD-drop is RETAINED as-is (correct either way). **br_netfilter CONSTRAINT:** it MUST stay interface-scoped and NEVER be globalized -- with `br_netfilter` loaded, bridged WAN frames traverse the L3 FORWARD chain, so a global forward-drop would silently kill the `br-vr1-dc0-wan` WAN path. `--check` now also verifies the WAN bridge exists with its uplink enslaved (same fail-open class). Nothing routes across the fiber THROUGH vvr1-dc0; the rack proxies MAAS at the app layer (originated/terminated, not forwarded). Same pin follows onto `voffice1` when its transit leg is wired. Region route must target only the rack transit /30, never 10.12.8.0/22. Stays OPEN until applied + verified on vvr1-dc0. | | SEC-011 | 2026-07-16 | **Node least-connectivity gap (not an L2 breach).** Under D-121 Option C role separation, all nodes get a uniform 6-plane NIC set, so a ceph-osd STORAGE node has a leg on provider-public (external/FIP) + data-tenant (tenant geneve) -- planes it never binds per D-052. Planes stay isolated L2 (no crosstalk). | 2026-07-16 plane-segregation review; `opentofu/main.tf` `local.vr1_dc0_node_nics` | operator | **CLOSED 2026-07-16 (operator ruling -- keep uniform 6-NIC).** Review R3-F10: A2's cross-examination refuted the attack-surface concern -- in the isolated-L2 sim the unbound vNICs have no reachability out, and pruning would INCREASE Roosevelt-delta (baremetal trunks all VLANs to every node on bonded NICs, so all planes are present regardless of L3 binding). Uniform 6-NIC is the more Roosevelt-faithful model. Accepted non-issue; no code change. | **STANDING CONVENTION (SEC-009, 2026-07-15): per-site credential/env consolidation.** ALL sensitive diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 02485e7..5a18fbb 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -245,11 +245,26 @@ assignment (Phase E, L1-07) + the depth-4 nested-virt L4 boot gate. - **Model B cross-check (agent) = FAITHFUL, no BLOCKING**; fixes applied: 32-GiB-overhead RATIFIED in the record (870/154, vvr1-dc0 416 GiB), false metal-admin-IP comment corrected, SEC-010 updated for the - single-leg-outer/bridges-6-inner-planes reality. **OBS-3 (logged):** under Model B the inner - `vr1-dc0-wan` NAT has NO upstream (vvr1-dc0 has only the transit leg + no default route per D-107); the - simulated-ISP uplink is inert unless vvr1-dc0 is given an egress path -- confirm this is intended (closed - test, edge-boundary-only) or wire a controlled egress. Transit IP is duplicated across roots + single-leg-outer/bridges-6-inner-planes reality. **OBS-3: design gap RESOLVED by D-125; egress efficacy + is a deploy-time gate (see the C+D-commit entry below).** Transit IP is duplicated across roots (`vr1_dc0_rack_transit_ip` outer / `vvr1_dc0_transit_ip` inner) -- keep equal in tfvars. + +- **Stage-3 sweep C+D COMMITTED as `5175365`** (operator "commit but don't push"; NOT pushed). The + adversarial re-run then confirmed **OBS-3 material** once the operator ruled **"DC0 and DC1 have their + own ISP connections"**, resolved by **D-125 (bridge-in, single-NAT)**: + - Model B nesting had severed per-DC ISP egress: `modules/site-wan` makes `vr1-dc0-wan` a libvirt NAT + that egresses via ITS HOST's default route; under Model B that host is `vvr1-dc0` (transit-only, + East-West, SEC-010 FORWARD-dropped) -> dead. Fix: vcloud ISP NAT `vr1-dc0-uplink` + an IP-less uplink + NIC + netplan bridge `br-vr1-dc0-wan` on `vvr1-dc0`; inner `vr1-dc0-wan` becomes a BRIDGE (new + `modules/wan-bridge`) so OPNsense WAN rides to the single vcloud NAT. **Authored present-only; both + roots validate; NOT closed -- the L2-NAT-of-foreign-MAC path is a DEPLOY-TIME gate (bridge-in + equivalent of the L4 boot gate), fallback = double-NAT.** See D-125 + platform-traps 1g. + - **New HELD gates (hard rule 2):** `vr1_dc0_uplink_cidr` (vcloud ISP /24, office1-netbox apex) + the + OPNsense WAN re-address from `172.30.2.2` into that /24 (edge config, D-113). + - **br_netfilter constraint:** SEC-010 FORWARD-drop stays interface-scoped, NEVER globalized. `--check` + hardened: fail-open guard now covers the transit-iface existence AND the WAN bridge/uplink enslavement. + - Harness: `site-headend-install` 49->54; `wan-bridge` covered by opentofu-validate S3. Records: D-125, + SEC-010, model-a-fallback B->A delta (item 8 + revert), platform-traps 1g. - **Phase D NOT STARTED** -- SEC-010 rack `ip_forward=0` + firewall artifact + mechanical pre-apply gate. - **Phase E NOT STARTED** -- operator assigns rack transit/30 + rack IP in office1-netbox (apex token), fill tfvars -> `tofu plan` renders. diff --git a/opentofu/main.tf b/opentofu/main.tf index 5ca47fe..e7acc48 100644 --- a/opentofu/main.tf +++ b/opentofu/main.tf @@ -318,6 +318,28 @@ # } # ===================================================================== +# D-125 (Model B bridge-in, closes OBS-3): the vr1-dc0 simulated-ISP uplink, at +# VCLOUD level. This is the SINGLE NAT = the ISP hand-off. It egresses via vcloud's +# own uplink (a libvirt NAT network masquerades out ITS host's default route), which +# is exactly the per-DC egress that D-123 Model B severed by nesting vr1-dc0-wan +# inside vvr1-dc0 (whose only routed leg is the East-West-only transit, SEC-010 +# FORWARD-dropped). vvr1-dc0 takes an IP-less uplink NIC on this network (below); the +# inner vr1-dc0-wan is a BRIDGE (modules/wan-bridge) onto vvr1-dc0's br-vr1-dc0-wan, +# so the inner OPNsense WAN lands directly on THIS /24 and there is one NAT, here. +# The /24 is NetBox-assigned (HELD, hard rule 2). Same site-wan NAT module as +# office1-wan -- 1500 MTU, no DHCP (edge WAN is static, D-113). +# DEPLOY-TIME GATE (D-125): that vcloud's virbr actually NATs OPNsense's foreign +# MAC across the doubly-nested bridge is unprovable pre-apply -- the bridge-in +# equivalent of the depth-4 boot gate. Isolation-test first (bootstrap prints it); +# fallback = double-NAT. See D-125. +module "vr1_dc0_uplink" { + source = "./modules/site-wan" + network_name = "vr1-dc0-uplink" + cidr = var.vr1_dc0_uplink_cidr + # mtu defaults to 1500 (the ISP-uplink domain; never jumbo). +} + +# ===================================================================== # D-123 MODEL B / D-124 AMENDMENT: vvr1-dc0 -- the vr1-dc0 SITE CONTAINMENT VM. Under # Model B it is BOTH the MAAS rack controller (enrolled to the Office1 region) AND the # nested libvirt HOST for the DC's entire node fleet (9 nodes + 6 planes + wan + edge, @@ -340,10 +362,16 @@ expose_nested_virt = true # D-123 Model B: LOAD-BEARING -- inner node VMs need svm passthrough or no inner KVM / nova boots # D-123 Model B: metal-admin is now an INNER bridge (created by the inner root's - # dc-planes INSIDE vvr1-dc0; the rack serves PXE/DHCP there). So the OUTER vvr1-dc0 has - # ONE leg -- the office1<->dc0 transit that reaches the Office1 region (D-124 Scheme A). + # dc-planes INSIDE vvr1-dc0; the rack serves PXE/DHCP there). TWO outer legs: + # NIC1 (enp1s0 -> mgmt) = office1<->dc0 transit, region-facing (D-124 Scheme A). + # SEC-010/--transit-if key on THIS interface -- keep it first. + # NIC2 (enp2s0 -> uplink) = D-125 bridge-in: an IP-LESS port enslaved into the netplan + # bridge br-vr1-dc0-wan below, attached to the vcloud ISP NAT + # (vr1-dc0-uplink). The inner OPNsense WAN rides this bridge to + # the single vcloud NAT -- restoring per-DC egress (OBS-3). network_names = [ - module.mesh_vr1_dc0_office1.network_name, # office1<->dc0 transit (region-facing) + module.mesh_vr1_dc0_office1.network_name, # NIC1 enp1s0 -> mgmt (transit; SEC-010 keys here) + module.vr1_dc0_uplink.network_name, # NIC2 enp2s0 -> uplink (IP-less port of br-vr1-dc0-wan) ] user_data = <<-EOT @@ -377,12 +405,22 @@ # this (documented trap; modules/opnsense-edge + the runbook). metal-admin gives # the rack no default route; the region route rides the transit toward Office1 # (10.10.0.0/22, as-built D-115). - # Single OUTER leg = the transit (first/only NIC = enp1s0; interface-naming trap still - # applies -- CONFIRM on first boot). metal-admin is configured on the INNER bridge by the - # BOOTSTRAP step (site-headend-install.sh node-host mode) using var.vr1_dc0_rack_metal_admin_ip - # -- that var is consumed by the bootstrap script, NOT by this outer root nor the inner - # OpenTofu root (it remains a required tfvar for the L1-07 rack-addressing gate). The region - # route rides the transit toward Office1 (10.10.0.0/22, as-built D-115). + # OUTER legs = transit (NIC1 enp1s0 -> mgmt) + D-125 uplink (NIC2 enp2s0 -> uplink). + # INTERFACE-NAMING TRAP applies to BOTH (enp1s0/enp2s0 vs ensX depend on machine type, + # unknown until first boot -- CONFIRM on boot and adjust each `match` before relying on + # this). metal-admin is configured on the INNER bridge by the BOOTSTRAP step + # (site-headend-install.sh node-host mode) using var.vr1_dc0_rack_metal_admin_ip -- that + # var is consumed by the bootstrap script, NOT by this outer root nor the inner OpenTofu + # root (it remains a required tfvar for the L1-07 rack-addressing gate). The region route + # rides the transit toward Office1 (10.10.0.0/22, as-built D-115). + # + # D-125 bridge-in (closes OBS-3): NIC2 `uplink` is an IP-LESS bridge port; the netplan + # bridge `br-vr1-dc0-wan` (no IP on the vvr1-dc0 side -- only the inner OPNsense WAN gets a + # vcloud-ISP /24 address across it) is declared HERE so it exists before the inner apply. + # STP off (single port). The bootstrap only VERIFIES the bridge + grants libvirt/AppArmor + # access + isolation-tests egress; it does not create the bridge. br_netfilter CONSTRAINT + # (D-125): SEC-010's FORWARD-drop MUST stay scoped to `mgmt` -- a GLOBAL forward drop would + # silently kill bridged WAN frames if br_netfilter is loaded. network_config = <<-EOT version: 2 ethernets: @@ -394,5 +432,19 @@ routes: - to: "10.10.0.0/22" via: "${var.vr1_dc0_rack_transit_peer_ip}" + uplink: + match: + name: "enp2s0" + set-name: uplink + dhcp4: false + dhcp6: false + bridges: + br-vr1-dc0-wan: + interfaces: [uplink] + dhcp4: false + dhcp6: false + parameters: + stp: false + forward-delay: 0 EOT } diff --git a/opentofu/modules/wan-bridge/main.tf b/opentofu/modules/wan-bridge/main.tf new file mode 100644 index 0000000..699a1b1 --- /dev/null +++ b/opentofu/modules/wan-bridge/main.tf @@ -0,0 +1,31 @@ +# wan-bridge: the DC WAN segment realized as a libvirt BRIDGE onto a pre-existing +# host bridge (D-125, Model B bridge-in). NOT a NAT and NOT a subnet owner -- +# see variables.tf for why this exists (OBS-3) and how it differs from site-wan. +# +# SCHEMA NOTE: same provider (registry.opentofu.org/dmacvicar/libvirt) and same +# attribute-style nesting confirmed by modules/site-wan (via +# `tofu providers schema -json`): +# forward = { mode = "bridge" } (nesting=single; no top-level `mode`) +# bridge = { name = "" } (attribute-object; the existing Linux bridge) +# mtu = { size = N } (nesting=single; matches site-wan/dc-planes) +# No `ips` and no NAT: addressing lives on the vcloud ISP /24 that the host +# bridge's enslaved uplink NIC reaches; the edge WAN is static there (D-113). + +resource "libvirt_network" "wan_bridge" { + name = var.network_name + autostart = true + + # forward mode = bridge: hand traffic straight onto an existing host bridge; + # libvirt neither creates the bridge nor manages L3 on it (the bootstrap does). + forward = { + mode = "bridge" + } + + bridge = { + name = var.host_bridge + } + + mtu = { + size = var.mtu + } +} diff --git a/opentofu/modules/wan-bridge/outputs.tf b/opentofu/modules/wan-bridge/outputs.tf new file mode 100644 index 0000000..fa973ed --- /dev/null +++ b/opentofu/modules/wan-bridge/outputs.tf @@ -0,0 +1,4 @@ +output "network_name" { + description = "The libvirt network name, for a consumer's wan_network_name." + value = libvirt_network.wan_bridge.name +} diff --git a/opentofu/modules/wan-bridge/variables.tf b/opentofu/modules/wan-bridge/variables.tf new file mode 100644 index 0000000..30e0541 --- /dev/null +++ b/opentofu/modules/wan-bridge/variables.tf @@ -0,0 +1,40 @@ +# wan-bridge: the DC WAN segment as a libvirt BRIDGE onto an existing host +# bridge (D-125, Model B bridge-in), NOT a NAT. Sibling to modules/site-wan: +# - site-wan = NAT'd /24 simulated ISP, egresses via ITS HOST's default route +# (correct at vcloud level; the `office1-wan` / `vr1-dc0-uplink` +# pattern -- there is exactly ONE such NAT = the simulated ISP). +# - wan-bridge = no subnet, no NAT: it hands the site edge's WAN leg straight +# onto a pre-existing Linux bridge (`br-vr1-dc0-wan`) that the +# node-host bootstrap builds by enslaving vvr1-dc0's IP-less +# uplink NIC. The edge WAN then lives on the vcloud ISP /24 and +# the single NAT is at vcloud (D-125). Use this INSIDE a +# containment VM whose real egress is a bridged uplink port. +# +# Why a bridge and not a NAT here: under D-123 Model B this network is created +# INSIDE vvr1-dc0, whose only routed leg is the East-West-only transit (SEC-010 +# FORWARD-drops it). A NAT here would have no egress (OBS-3). Bridging the edge +# WAN through to the vcloud ISP NAT restores independent per-DC egress with a +# single NAT hop. + +variable "network_name" { + description = "libvirt network name, e.g. \"vr1-dc0-wan\" (the edge's wan_network_name)." + type = string +} + +variable "host_bridge" { + description = <<-EOT + Name of a PRE-EXISTING Linux host bridge this libvirt network attaches to, + e.g. "br-vr1-dc0-wan". The bootstrap (site-headend-install.sh node-host + mode) MUST create this bridge and enslave the IP-less uplink NIC BEFORE the + inner tofu apply -- libvirt does not create it in bridge forward mode. The + edge WAN interface then takes a STATIC address on the vcloud ISP /24 (set in + the edge config, D-113), reaching the vcloud NAT gateway across this bridge. + EOT + type = string +} + +variable "mtu" { + description = "WAN-segment MTU. Default 1500 (the ISP-uplink domain; NEVER jumbo -- matches site-wan)." + type = number + default = 1500 +} diff --git a/opentofu/modules/wan-bridge/versions.tf b/opentofu/modules/wan-bridge/versions.tf new file mode 100644 index 0000000..4685df1 --- /dev/null +++ b/opentofu/modules/wan-bridge/versions.tf @@ -0,0 +1,11 @@ +# Provider source mapping for this module (DOCFIX-179). Child modules MUST +# declare their own required_providers so OpenTofu maps the local name +# `libvirt` to dmacvicar/libvirt; version is pinned once at the ROOT +# (the consuming root's versions.tf), child modules declare source only. +terraform { + required_providers { + libvirt = { + source = "dmacvicar/libvirt" + } + } +} diff --git a/opentofu/variables.tf b/opentofu/variables.tf index 877d440..12e924d 100644 --- a/opentofu/variables.tf +++ b/opentofu/variables.tf @@ -145,3 +145,15 @@ description = "Office1 region's IP on the transit link -- the rack's next-hop toward the MAAS region (10.10.0.0/22). NetBox-assigned (D-124)." type = string } + +# ---- D-125 (Model B bridge-in, closes OBS-3): the vcloud-level per-DC simulated +# ISP /24. This is the SINGLE NAT (the ISP hand-off), created at vcloud so it +# egresses via vcloud's uplink -- restoring the independent per-DC egress that +# Model B's inner nesting severed. vvr1-dc0 gets an IP-less uplink NIC on this +# network; the inner OPNsense WAN takes a STATIC address in this /24 (edge config, +# D-113). NetBox-assigned (office1-netbox, apex-held) -- NO default here, hard +# rule 2: `tofu plan` requires the tfvar, mirroring the D-124 rack IPs. ---- +variable "vr1_dc0_uplink_cidr" { + description = "vcloud-level per-DC simulated-ISP uplink /24 for vr1-dc0 (D-125). NetBox-assigned; gateway is .1, OPNsense WAN takes a static .2-band address here. No default (HELD, hard rule 2)." + type = string +} diff --git a/opentofu/vr1-dc0-substrate/main.tf b/opentofu/vr1-dc0-substrate/main.tf index 72a1a8e..bf517cc 100644 --- a/opentofu/vr1-dc0-substrate/main.tf +++ b/opentofu/vr1-dc0-substrate/main.tf @@ -33,11 +33,18 @@ planes = var.vr1_dc0_planes } -# D-122: the dedicated per-site simulated-ISP uplink (172.30.2.0/24, D-115). +# D-125 (Model B bridge-in, closes OBS-3): the DC WAN segment is a BRIDGE, not a NAT. +# Under D-123 Model B this network is created INSIDE vvr1-dc0, whose only routed leg is +# the East-West-only transit (SEC-010 FORWARD-dropped), so a NAT here would have no egress +# (that was OBS-3). Instead it bridges onto vvr1-dc0's br-vr1-dc0-wan (an IP-less uplink +# NIC on the vcloud ISP NAT, declared in the outer root's vvr1-dc0 netplan). The OPNsense +# WAN then takes a STATIC address on the vcloud ISP /24 (var.vr1_dc0_uplink_cidr, set in the +# edge config per D-113 -- HELD until that /24 is NetBox-assigned) and there is ONE NAT, at +# vcloud. host_bridge MUST match the netplan bridge name in opentofu/main.tf. module "vr1_dc0_wan" { - source = "../modules/site-wan" + source = "../modules/wan-bridge" network_name = "vr1-dc0-wan" - cidr = "172.30.2.0/24" + host_bridge = "br-vr1-dc0-wan" # mtu defaults to 1500 (ISP-uplink domain; NOT the jumbo planes/mesh). } diff --git a/runbooks/dc-dc-phase2-tofu-dc-substrate.md b/runbooks/dc-dc-phase2-tofu-dc-substrate.md index 9f281cf..1bcae28 100644 --- a/runbooks/dc-dc-phase2-tofu-dc-substrate.md +++ b/runbooks/dc-dc-phase2-tofu-dc-substrate.md @@ -166,8 +166,21 @@ Node count is R-3 = 3 control + 2 compute + **4 storage = 9/DC** (Step 6's "8" is superseded). The Model A single-root layout is preserved as a revert (git tag `model-a-fallback` + `docs/model-a-fallback-plan.md`). Sizing derived by `scripts/dc-dc-whole-host-budget.py` (870/1024 GiB, FIT). SEC-010 DC-LOCAL is enforced -by the bootstrap's transit FORWARD-drop (NOT a global `ip_forward=0` -- the inner WAN NAT forces -`ip_forward=1`). See `docs/changelog-20260716-review-sweep-phaseC.md`. +by the bootstrap's transit FORWARD-drop (NOT a global `ip_forward=0`), kept interface-scoped -- **never +globalize it** (D-125 br_netfilter constraint: bridged WAN frames traverse L3 FORWARD). See +`docs/changelog-20260716-review-sweep-phaseC.md`. + +**D-125 (bridge-in per-DC ISP egress -- closes OBS-3).** Under Model B the inner `vr1-dc0-wan` was a NAT +with no egress (its host `vvr1-dc0` is transit-only). Fix, folded into the roots above: +- **A (OUTER)** ALSO creates the vcloud ISP NAT `vr1-dc0-uplink` (HELD `var.vr1_dc0_uplink_cidr` -- assign + the /24 in office1-netbox before `tofu plan`) + a 2nd IP-less uplink NIC + the `br-vr1-dc0-wan` netplan + bridge on `vvr1-dc0`. +- **B (BOOTSTRAP)** `--check` ALSO verifies `br-vr1-dc0-wan` exists with `uplink` enslaved (fail-open guard). +- **C (INNER)** `vr1-dc0-wan` is now a BRIDGE (`modules/wan-bridge`) onto `br-vr1-dc0-wan`; the OPNsense + WAN takes a static address in the vcloud ISP /24 (edge config, D-113 -- HELD). +- **DEPLOY-TIME GATE (Step 12):** before trusting the OPNsense chain, isolation-test egress -- a throwaway + guest on `br-vr1-dc0-wan` must get a vcloud-ISP address + ping out. FAIL => revert to the D-125 double-NAT + fallback (not a redesign). See D-125 + `docs/changelog-20260716-review-sweep-phaseC2-D125.md`. --- diff --git a/scripts/site-headend-install.sh b/scripts/site-headend-install.sh index 160664f..18ddb68 100644 --- a/scripts/site-headend-install.sh +++ b/scripts/site-headend-install.sh @@ -87,6 +87,11 @@ # SEC-010: the transit-leg interface name on vvr1-dc0 (the outer NIC; netplan set-name in the # outer cloud-init is "mgmt"). The DC-LOCAL FORWARD-drop is keyed on it. TRANSIT_IF="mgmt" +# D-125 bridge-in (closes OBS-3): the WAN uplink port + host bridge on vvr1-dc0, both DECLARED +# in the outer cloud-init netplan (set-name "uplink"; bridge "br-vr1-dc0-wan"). The bootstrap +# does NOT create them -- it VERIFIES they exist (fail-open class, like the SEC-010 transit +# check) so the inner OPNsense WAN actually has egress. +UPLINK_IF="uplink"; WAN_BRIDGE="br-vr1-dc0-wan" while [ $# -gt 0 ]; do case "$1" in --check) CHECK=1 ;; @@ -101,6 +106,8 @@ --inner-pool-path) shift; INNER_POOL_PATH="${1:-}" ;; --opnsense-base) shift; OPNSENSE_BASE="${1:-}" ;; --transit-if) shift; TRANSIT_IF="${1:-}" ;; + --uplink-if) shift; UPLINK_IF="${1:-}" ;; + --wan-bridge) shift; WAN_BRIDGE="${1:-}" ;; -h|--help) echo "usage: site-headend-install.sh [--check|--dry-run] [--role ]" echo " region+rack (default): --compose-cidr [--maas-url-ip ]" @@ -217,6 +224,21 @@ else echo " [ok] SEC-010 transit FORWARD-drop present + transit interface '$TRANSIT_IF' exists" fi + # D-125 bridge-in WAN egress: the host bridge '$WAN_BRIDGE' must EXIST and its ISP-facing + # uplink port '$UPLINK_IF' must be ENSLAVED to it -- same fail-open class as the SEC-010 + # transit check: a missing bridge/unenslaved port means the inner OPNsense WAN silently has + # NO egress, yet `tofu validate` is clean. Both are declared in vvr1-dc0's outer netplan. + if ! ip link show "$WAN_BRIDGE" >/dev/null 2>&1; then + echo " [--] D-125 WAN bridge '$WAN_BRIDGE' MISSING -- inner OPNsense WAN has no egress (check vvr1-dc0 netplan)"; ok=1 + elif ! ip link show "$UPLINK_IF" 2>/dev/null | grep -q "master $WAN_BRIDGE"; then + echo " [--] D-125 uplink '$UPLINK_IF' not enslaved to '$WAN_BRIDGE' (missing/naming trap?) -- confirm --uplink-if / netplan"; ok=1 + else + echo " [ok] D-125 WAN bridge '$WAN_BRIDGE' present + uplink '$UPLINK_IF' enslaved" + echo " [gate] DEPLOY-TIME (unprovable here, like the depth-4 boot gate): before the OPNsense chain" + echo " rides it, prove egress in ISOLATION -- attach a throwaway guest to '$WAN_BRIDGE'," + echo " confirm it gets a vcloud-ISP address and pings out. FAIL => revert to the D-125" + echo " double-NAT fallback, NOT a redesign." + fi return $ok } @@ -242,11 +264,19 @@ # yields qemu 'Permission denied' define-then-fail on the inner domains -- a silent dead guest). run "create inner pool dir $INNER_POOL_PATH" -- mkdir -p "$INNER_POOL_PATH" run "AppArmor: allow inner pool path" -- bash -c "install -d /etc/apparmor.d/local/abstractions 2>/dev/null; grep -qF '$INNER_POOL_PATH' /etc/apparmor.d/local/abstractions/libvirt-qemu 2>/dev/null || echo ' \"$INNER_POOL_PATH/**\" rwk,' >> /etc/apparmor.d/local/abstractions/libvirt-qemu; systemctl reload apparmor 2>/dev/null || true" - # SEC-010 (D-052/D-100 DC-LOCAL under Model B): ip_forward is forced ON by the inner vr1-dc0-wan - # NAT, so a global ip_forward=0 is unusable. Enforce DC-local with a FORWARD-drop across the TRANSIT - # leg -- nothing routes ACROSS the fiber THROUGH vvr1-dc0 (the rack proxies MAAS at the app layer = - # originated/terminated, NOT forwarded). vvr1-dc0's own region traffic (OUTPUT/INPUT) is unaffected. - echo "== SEC-010 (DC-LOCAL): FORWARD-drop across the transit leg '$TRANSIT_IF' ==" + # SEC-010 (D-052/D-100 DC-LOCAL under Model B): enforce DC-local with a FORWARD-drop across the + # TRANSIT leg -- nothing routes ACROSS the fiber THROUGH vvr1-dc0 (the rack proxies MAAS at the app + # layer = originated/terminated, NOT forwarded). vvr1-dc0's own region traffic (OUTPUT/INPUT) is + # unaffected. + # D-125 CONSTRAINT -- keep this drop SCOPED to '$TRANSIT_IF'; do NOT globalize it. Under bridge-in + # the inner vr1-dc0-wan is a BRIDGE (not a NAT), so the pre-bridge-in rationale (an inner WAN NAT + # requiring host ip_forward, which once ruled out a global ip_forward=0) no longer applies -- but + # with br_netfilter loaded, bridged WAN frames traverse the L3 FORWARD chain, so a GLOBAL + # forward-drop (or `iptables -P FORWARD DROP`) would silently kill the WAN bridge. The + # interface-scoped drop below leaves the '$WAN_BRIDGE'/'$UPLINK_IF' path untouched (policy accept) + # while still blocking transit forwarding. (ip_forward=0 is L3-only and does not affect pure L2 + # bridging; do not conflate the two.) + echo "== SEC-010 (DC-LOCAL): FORWARD-drop across the transit leg '$TRANSIT_IF' (scoped; D-125: never global) ==" run "install nftables" -- apt-get install -y nftables if [ "$DRYRUN" = "1" ]; then echo " [dry-run] write /etc/nftables-sec010.nft (FORWARD-drop in+out '$TRANSIT_IF') + enable sec010-fw.service (boot-persistent)" @@ -276,6 +306,21 @@ systemctl daemon-reload; systemctl enable sec010-fw.service >/dev/null 2>&1 && echo " -> sec010-fw.service enabled (boot-persistent)" fi + # D-125 bridge-in WAN egress: the host bridge + uplink port are DECLARED in vvr1-dc0's outer + # netplan (this script does not create them). Verify they landed so the inner OPNsense WAN has + # a real path out; libvirt's forward=bridge network (modules/wan-bridge) attaches taps to this + # pre-existing bridge -- it must exist BEFORE the inner apply. + echo "== D-125 WAN bridge (verify only; declared in the outer netplan) ==" + if ip link show "$WAN_BRIDGE" >/dev/null 2>&1 && ip link show "$UPLINK_IF" 2>/dev/null | grep -q "master $WAN_BRIDGE"; then + echo " -> '$WAN_BRIDGE' present with '$UPLINK_IF' enslaved" + echo " GATE (deploy-time, unprovable pre-apply): before the OPNsense chain rides it, attach a" + echo " throwaway guest to '$WAN_BRIDGE' and confirm a vcloud-ISP address + ping-out. FAIL =>" + echo " revert to the D-125 double-NAT fallback, not a redesign." + else + echo " WARN: '$WAN_BRIDGE'/'$UPLINK_IF' not ready -- inner OPNsense WAN will have NO egress (OBS-3)." + echo " Check vvr1-dc0's netplan (interface-naming trap: enp2s0 vs ensX) BEFORE the inner apply." + fi + # OPNsense base for the inner edge (opnsense-prep-image.sh needs network; run on this host). # Verify, do NOT fetch inline (the prep script is not guaranteed present here). if [ -s "$OPNSENSE_BASE" ]; then echo " -> opnsense base present: $OPNSENSE_BASE" diff --git a/tests/site-headend-install/run-tests.sh b/tests/site-headend-install/run-tests.sh index 68b5221..22dd60b 100644 --- a/tests/site-headend-install/run-tests.sh +++ b/tests/site-headend-install/run-tests.sh @@ -114,6 +114,20 @@ # SEC-010 --check must not fail-open: it must verify the keyed transit interface ACTUALLY EXISTS, # because nftables oifname/iifname with an absent name loads clean but matches NOTHING (advisor catch) grep -q 'ip link show "$TRANSIT_IF"' "$S" && ok || bad "SEC-010 --check must verify the transit interface exists (nftables fail-open otherwise)" +# D-125 bridge-in (OBS-3): node-host must VERIFY the WAN bridge + uplink so the inner OPNsense WAN +# has real egress. The dry-run must surface the D-125 WAN-bridge section. +printf '%s' "$nhout" | grep -qi 'WAN bridge' && ok || bad "node-host --dry-run missing the D-125 WAN-bridge verify section" +# the uplink guard is the SAME fail-open class as the transit check: it must confirm the port is +# ENSLAVED to the bridge (not merely that a name exists) +grep -q 'master $WAN_BRIDGE' "$S" && ok || bad "D-125 WAN check must verify the uplink is enslaved to the bridge (fail-open otherwise)" +# --uplink-if must be overridable and reflected in the output +upout="$(bash "$S" --role rack --host-nodes --uplink-if wanup --dry-run --region-url "$RURL" --enroll-secret-file "$SECFILE" 2>&1)" +printf '%s' "$upout" | grep -q 'wanup' && ok || bad "--uplink-if override not reflected in the D-125 WAN-bridge check" +# br_netfilter CONSTRAINT: SEC-010's FORWARD-drop must stay interface-scoped; the script must warn +# against globalizing it (else bridged WAN frames die silently under br_netfilter) +grep -qiE 'never global|do NOT globalize|br_netfilter' "$S" && ok || bad "D-125 br_netfilter constraint (keep SEC-010 scoped, never global) must be recorded in the script" +# the stale "ip_forward forced ON by the inner NAT" rationale must be gone (bridge-in has no inner NAT) +grep -q 'forced ON by the inner' "$S" && bad "stale pre-D-125 rationale (ip_forward forced by inner NAT) still present -- bridge-in removed the inner NAT" || ok # it must still enroll the rack AND must not leak the secret printf '%s' "$nhout" | grep -q 'maas init rack' && ok || bad "node-host mode dropped the rack enrollment" printf '%s' "$nhout" | grep -q "$SENTINEL" && bad "node-host --dry-run LEAKED the enrollment secret" || ok