diff --git a/docs/changelog-20260716-review-sweep-phaseC.md b/docs/changelog-20260716-review-sweep-phaseC.md new file mode 100644 index 0000000..0162e5e --- /dev/null +++ b/docs/changelog-20260716-review-sweep-phaseC.md @@ -0,0 +1,64 @@ +# 2026-07-16 -- Stage-3 review sweep, Phase C (D-123 Model B OpenTofu reshape + node-host bootstrap) + +## Context + +Phase C of the review sweep (`docs/stage3-adversarial-review-20260716.md`), implementing the operator's +D-123 = **Model B** ruling (R-1): the vr1-dc0 node fleet nests INSIDE the containment VM `vvr1-dc0`. +Design was cross-checked by a 4-charter agent workflow (mechanism/compatibility/best-practice/structure); +the two-root implementation was cross-checked by a second agent (verdict: FAITHFUL, no BLOCKING). Inner +transport = **remote qemu+ssh from Office1** (R-5). NO cloud mutation; nothing applied. Remaining: +maas-vm-host retarget, C5 runbook reorg, Phase D SEC-010. + +## What changed + +**Mechanism (forced, cross-checked): TWO-ROOT / TWO-STATE split.** A libvirt provider cannot be +configured from a resource created in the same apply, so the substrate splits into an OUTER root +(vcloud) that boots vvr1-dc0 and an INNER root that builds the fleet on vvr1-dc0's own libvirt, with a +BOOTSTRAP GATE between. Modules are reused VERBATIM -- the shim stays wholesale-deletable, Roosevelt-collapsing. + +**C1 -- OUTER root (`opentofu/main.tf`, vcloud).** +- `vvr1_dc0` resized 4/8192/80 -> **108 vCPU / 416 GiB / ~3000 GiB** via new `var.vvr1_dc0_*` (derived + defaults, `variables.tf`; hard rule 2); `expose_nested_virt` false -> **true** (load-bearing). + Collapsed to a SINGLE outer leg (the transit); metal-admin + the other 5 planes become INNER bridges. +- Removed the inner-substrate blocks (planes / wan / edge / 9-node layout) -> MOVED markers. Deleted the + now-dangling `moved{dc1_planes->vr1_dc0_planes}` block (kept the two storage `moved` blocks). +- `tofu validate` Success; no dangling references. + +**C2 -- INNER root (`opentofu/vr1-dc0-substrate/`, NEW: versions/variables/main.tf).** Provider = +`qemu+ssh://@/system` (R-5, key auth, no `no_verify`). Recreates the inner +storage pool + 6 dc-planes + site-wan + opnsense-edge + 9 node-VMs (3+2+4, R-3), reusing `../modules/*` +verbatim. `init` + `validate` Success. `.terraform/` gitignored; `.terraform.lock.hcl` committed. + +**C3 -- Bootstrap node-host mode (`scripts/site-headend-install.sh` + harness).** New `--host-nodes` +(valid only with `--role rack`) makes the DC rack a nested libvirt host so the inner provider has a +target: installs qemu-kvm + libvirt, persists + verifies `kvm_amd/intel nested=1`, adds the ssh user to +the libvirt group, creates the inner pool dir + an AppArmor grant (platform-traps: non-default pool path +otherwise fails qemu with Permission denied), and verifies/instructs the OPNsense base image (delegated +to `opnsense-prep-image.sh`, not fetched inline). `--check`/`--dry-run` supported; secret never leaked; +plain-rack backward-compatible. Harness 32 -> **45/45**. + +**Sizing DERIVED + RATIFIED.** `scripts/dc-dc-whole-host-budget.py` (Phase B) derives vvr1-dc0 = one DC +fleet (104 vCPU / 384 GiB / 2850 GiB) + **32 GiB** containment overhead (ratified for Model B: the VM +also carries the inner edge + libvirtd + nested-KVM paging the rack-only sizing lacked). Whole-host 2-DC += **870 / 1024 GiB (85%), FIT, 154 GiB headroom**. The design-decisions D-121/123/124 amendments were +reconciled from the earlier 16-GiB figures (838/404/186) to the ratified 32-GiB figures (870/416/154). + +**Cross-check fixes applied:** false metal-admin-IP comment corrected (it flows to the bootstrap script, +not the inner root); SEC-010 updated for the single-leg-outer / bridges-6-inner-planes reality (hardening +now belongs on the inner libvirt host); OBS-3 logged (nested `vr1-dc0-wan` NAT has no upstream under +Model B -- confirm intended or wire a controlled egress). + +## Tests / verification + +- `cd opentofu && tofu validate` -> Success (outer); `cd opentofu/vr1-dc0-substrate && tofu init+validate` -> Success (inner). +- `bash tests/site-headend-install/run-tests.sh` -> **45/45 PASS**. +- `bash scripts/repo-lint.sh` -> 0 fail, 1 legacy WARN. Byte hygiene: 0 non-ASCII / 0 CR added (edited regions). +- `python3 scripts/dc-dc-whole-host-budget.py --model B --containment-overhead-mem-gib 32` -> FIT 870/1024, 154 headroom. +- Agent cross-check of the two-root reshape: FAITHFUL, no BLOCKING. + +## Revert + +- Model A fallback: `git checkout model-a-fallback -- opentofu/` (tag on `114d392`, R-3-compliant) then + `bash scripts/opentofu-validate.sh`. See `docs/model-a-fallback-plan.md`. +- Node-host mode: `git checkout -- scripts/site-headend-install.sh tests/site-headend-install/`. +- Inner root: `git rm -r opentofu/vr1-dc0-substrate`. No cloud state touched. diff --git a/docs/design-decisions.md b/docs/design-decisions.md index cfa361c..33272a4 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -2538,7 +2538,7 @@ `vvr1-dc0`/`vvr1-dc1` now ALSO hold the OpenStack node VMs (this entry previously scoped them to the MAAS/LXD headend + the non-stack LXD-composed machines ONLY, NOT the OpenStack nodes). This makes a single-object `virsh destroy` a true site-down primitive for DCs, matching Office1. Consequence: the DC -containment VM is resized to hold the node fleet (~404 GiB) with `expose_nested_virt = true` (D-124 +containment VM is resized to hold the node fleet (~416 GiB, derived) with `expose_nested_virt = true` (D-124 rack sizing void). Office1's `voffice1` scope is unchanged. **Context.** Stage 2 was about to build Office1 per D-103: three peer VMs (MAAS-region, NetBox, @@ -3448,8 +3448,8 @@ headroom (a single storage-node loss cannot re-replicate until the host returns -- the property D-121 flagged for the REJECTED Option A but not, originally, for the ADOPTED Option C; review R3-F08). 4 OSD hosts restore self-heal (CRUSH can still place 3 distinct replicas on host loss). - Ceph disk re-checked for 4x500Gi/DC = PASS (5.31 TiB margin). Whole-host RAM hand-estimate - ~838 GiB/1024 (81%). + Ceph disk re-checked for 4x500Gi/DC = PASS (5.31 TiB margin). Whole-host RAM hand-estimate ~870 GiB/1024 (85%) at the ratified 32 GiB Model B + containment overhead (was ~838/81% at 16 GiB -- see the vault/re-validation amendment below). 2. **Vault-HA backend = (v-a) MySQL-backed, operator-ruled 2026-07-16.** The 2026-07-15 "RESOLVED" was agent-authored (no operator vault utterance; the only AskUserQuestion asked node-layout + a delivery "Proceed", never vault -- R3-F01). The operator confirmed v-a on 2026-07-16 after (v-b) @@ -3459,11 +3459,15 @@ 3. **Re-validation OWED (R3-F06):** the whole-host capacity model `scratchpad/optc-calc.py` is UNCOMMITTED and absent, so the changelog's "reproducible" claim is currently false. It must be committed as a harnessed calculator and RE-RUN for the 3+2+4 shape AND Model B nesting overhead - before the 838 GiB sizing is treated as measured. The disk validation originally recorded in this + before the ~870 GiB sizing is treated as measured. The disk validation originally recorded in this entry was for the runner-up Option B (4+4) -- re-record against the ruled layout. **DONE 2026-07-16:** replaced by the committed `scripts/dc-dc-whole-host-budget.py` - (`tests/dc-dc-whole-host-budget/`, 13/13). Re-run for the ruled Model B 3+2+4 shape = **838 GiB / - 1024, FIT, 186 GiB headroom** (vCPU 238/256 overcommit; disk 62% thin). Ceph disk re-run for 4 + (`tests/dc-dc-whole-host-budget/`, 13/13). Re-run for the ruled Model B 3+2+4 shape with the + **32 GiB containment overhead RATIFIED** (Model B: vvr1-dc0 also carries the inner OPNsense edge + + nested libvirtd + inner-pool management + nested-KVM paging the rack-only sizing never had; the + earlier 16 GiB figure gave 838/186) = **870 GiB / 1024 (85%), FIT, 154 GiB headroom** (per-VM + vvr1-dc0 = 108 vCPU / 416 GiB / ~3000 GiB, `opentofu/variables.tf` `vvr1_dc0_*` defaults; vCPU + 238/256 overcommit; disk 62% thin). Ceph disk re-run for 4 storage/DC = PASS 5.31 TiB. The lost `scratchpad/optc-calc.py` and its "reproducible" changelog claim are retired. @@ -3607,7 +3611,7 @@ the "containment holds headend + non-stack LXD only, NOT the OpenStack nodes" scope. 2. **D-124 rack sizing (4/8192/80) is VOID** -- `vvr1-dc0` must contain one DC's full node fleet (3 control + 2 compute + 4 storage per the D-121 R-3 amendment = ~384 GiB node RAM + overhead, - ~404 GiB VM) and set `expose_nested_virt = true`. D-124's transit-addressing (Scheme A) still stands. + ~416 GiB VM) and set `expose_nested_virt = true`. D-124's transit-addressing (Scheme A) still stands. 3. **Depth-4 nested virt ACCEPTED** as a rehearsal cost (VR0 proved depth 2). Functional risk to the Phase-5 workload boot flagged and accepted; `docs/model-a-fallback-plan.md` + git tag `model-a-fallback` preserve Model A as a low-cost revert if Model B fails to deploy. @@ -3670,7 +3674,7 @@ Under D-123 Model B (operator ruling), `vvr1-dc0` no longer holds ONLY the MAAS rack -- it now contains the DC's entire node fleet. So the **4 vCPU / 8192 MiB / 80 GiB** rack sizing in this entry is VOID: -`vvr1-dc0` must be sized to hold 3 control + 2 compute + 4 storage (~384 GiB node RAM + overhead, ~404 +`vvr1-dc0` must be sized to hold 3 control + 2 compute + 4 storage (~384 GiB node RAM + 32 overhead, ~416 GiB VM) with `expose_nested_virt = true`. **What still stands:** the transit addressing (Scheme A -- office1<->dc0 mesh transit /30 + rack metal-admin IP) is unchanged; the containment VM still straddles metal-admin (node-facing) + the transit (region-facing), now ALSO hosting the nodes. diff --git a/docs/model-a-fallback-plan.md b/docs/model-a-fallback-plan.md index 66db470..c9a0ab6 100644 --- a/docs/model-a-fallback-plan.md +++ b/docs/model-a-fallback-plan.md @@ -2,12 +2,14 @@ **Purpose.** The operator ruled **Model B** for D-123 (nodes nested inside `vvr1-dc0`, single-object `virsh destroy` site-down) -- the heavier, higher-risk path (depth-4 nested virt, supersedes -D-103/D-114, ~380 GiB containment VM). This document preserves **Model A** as a fully-specified, +D-103/D-114, ~416 GiB containment VM). This document preserves **Model A** as a fully-specified, already-implemented fallback so that if Model B fails to deploy, we revert WITHOUT re-engineering. **Revert anchor (git).** Model A is not theoretical -- it is the CURRENTLY COMMITTED substrate. The -last commit before any Model B reshape is tagged **`model-a-fallback`** (on `87a7a8a`, branch -`dc-dc-stage3-phase2-dc-substrate`). To restore Model A: `git checkout model-a-fallback -- opentofu/` +last commit before any Model B reshape is tagged **`model-a-fallback`** (re-cut 2026-07-16 onto +`114d392`, the **R-3-compliant** Model A layout with 4 storage nodes/DC -- the prior anchor at +`87a7a8a` was R-3-stale, flagged by the Model B design cross-check). To restore Model A: +`git checkout model-a-fallback -- opentofu/` (or cherry-pick the substrate files), then re-run `bash scripts/opentofu-validate.sh`. No file needs to be re-authored -- Model A already validates (`tofu validate` Success; 11/11 modules). @@ -53,20 +55,23 @@ 1. **Node placement:** B retargets `module "vr1_dc0_node"` (and the 6 planes + `vr1-dc0-wan`) from vcloud's libvirt to **`vvr1-dc0`'s inner libvirt**. A restores them to vcloud level. -2. **Headend sizing:** B resizes `vvr1-dc0` from 4/8192/80 to ~380 GiB (must hold one DC's full node +2. **Headend sizing:** B resizes `vvr1-dc0` from 4/8192/80 to ~416 GiB (must hold one DC's full node fleet) and sets `expose_nested_virt = true`. A restores D-124's 4/8192/80. 3. **maas-vm-host target:** B registers `vvr1-dc0`'s inner virsh; A registers vcloud's virsh. 4. **Governance:** B supersedes D-103/D-114; A keeps them ADOPTED as-is. On revert, the D-103/D-114 supersession is withdrawn. 5. **Nesting depth:** B = 4 (unproven); A = 2 (VR0-proven). 6. **Site-down primitive:** B = one `virsh destroy vvr1-dc0`; A = scripted group-destroy. +7. **Inner root:** B adds `opentofu/vr1-dc0-substrate/` (a new root dir + its own state) and a + `site-headend-install.sh --host-nodes` node-host bootstrap; A has neither. Revert removes them. ## 4. Revert procedure (if Model B deployment fails) 1. STOP -- do not attempt to fix Model B in place if nested-virt (depth-4) is the failure mode; that is the known risk this fallback exists for. 2. `git checkout model-a-fallback -- opentofu/main.tf opentofu/variables.tf opentofu/modules/` - (restores the Model A substrate verbatim). + (restores the Model A substrate verbatim), then `git rm -r opentofu/vr1-dc0-substrate` (the inner + root does not exist in Model A) and revert the `site-headend-install.sh` node-host mode. 3. `bash scripts/opentofu-validate.sh` -> expect 11/11 PASS (Model A already validates). 4. Re-instate D-103/D-114 as ADOPTED (they were only annotated superseded, not deleted -- see the sweep's supersession notes; revert removes those annotations). @@ -78,7 +83,7 @@ ## 5. Failure signals that should trigger the revert - nova-compute guests fail to boot or are unusably slow at 3x-nested KVM (the depth-4 risk). -- `vvr1-dc0` cannot be allocated ~380 GiB on the host alongside the other layers. +- `vvr1-dc0` cannot be allocated ~416 GiB on the host alongside the other layers. - MAAS enrolment/commissioning breaks because the node domains are no longer vcloud-visible. - `expose_nested_virt = true` on `vvr1-dc0` destabilises the headend/rack. diff --git a/docs/security-ledger.md b/docs/security-ledger.md index b3d67c6..e7fabef 100644 --- a/docs/security-ledger.md +++ b/docs/security-ledger.md @@ -18,7 +18,7 @@ | SEC-007 | 2026-07-12 | `~/vr1-office1-creds/` on the jumphost holds the Office1 edge root password + its bcrypt hash + the `office1_svc` SSH PRIVATE key + (added 2026-07-13) the OPNsense **API key/secret** (`opnsense-api.txt`, 0600), all in plaintext (dir 0700, files 0600) | 2026-07-12 edge build; custody detail deliberately not recorded here per D-069 | operator | OPEN -- required for edge management (D-112(c) makes SSH the only management path), so this is a ROTATION obligation, not a delete-me. Rotate at v1 close / if the jumphost is rebuilt or shared | | SEC-008 | 2026-07-13 | `~/vr1-office1-creds/tailscale-authkey.txt` (vcloud, 0600) -- the operator's 48-char Tailscale auth key for the SELF-HOSTED control plane `tailscale.baldurkeep.com`. Used to enrol `office1-tailscale`. Consumed BY PATH and never printed into a session; the copy shipped to the node was SHREDDED after `tailscale up` (tailscaled holds its own node key now). | 2026-07-13 Office1 Tailscale enrolment; docs/vr1-office1-as-built.md | operator | OPEN -- ROTATION obligation, not a delete-me: the key is still on vcloud for re-enrolment. Revoke/reissue on the control server at v1 close, or immediately if vcloud is rebuilt or shared. | | SEC-009 | 2026-07-15 | Credential/env SPRAWL + world-readable exposure on vcloud: three env files sat loose in `~` outside the consolidated `~/vr1-office1-creds/` -- `.vr1-netbox.env` (upstream NetBox token), `vr1-office1.env` (edge root-password hash + SSH key path), and `vr1-stage1.env` which held `TF_VAR_maas_api_key` (a MAAS API secret) at mode **0664 (group/world-readable)**; `tailscale-authkey.txt` in the creds dir was also 0664. | docs/changelog-20260715-creds-consolidation.md | operator | **REMEDIATED 2026-07-15** -- all three moved into `~/vr1-office1-creds/`, every sensitive file `chmod 600`. STANDING CONVENTION established (below). Note: this is a CLOSED in-house test; the rotation obligations of SEC-005/006/007 still apply at v1 close, this row only closes the SPRAWL + PERMS exposure. | -| SEC-010 | 2026-07-16 | **metal-admin DC-LOCAL invariant (D-052/D-100) is PRESERVED but NOT ENFORCED in the committed Stage-3 config.** The `vvr1_dc0` rack straddles metal-admin (10.12.8.0/22, DC-local) + the office1<->dc0 transit (crosses fiber). Its committed cloud-init pins static IPs only -- **no `net.ipv4.ip_forward=0` sysctl, no host firewall on the transit leg.** Cross-plane routing (metal-admin <-> the whole Office1 /22, bidirectional) is blocked ONLY by Ubuntu's distro default; the deferred MAAS-rack install could silently flip it. A MAAS rack proxies at the application layer and needs no kernel forwarding, so pinning is free. | 2026-07-16 plane-segregation review (advisor + 2 read-only agents); `opentofu/main.tf` vvr1_dc0 :376-439 | operator | **OPEN -- PRE-APPLY hardening.** Add `net.ipv4.ip_forward=0` (+ v6) via cloud-init `write_files`/`bootcmd` sysctl on the rack, and an nft/ufw rule pinning the transit leg to MAAS region ports only. Same pin must follow onto `voffice1` when its transit leg is wired (currently single-homed, so F1 is latent). Region route must target only the rack transit /30, never 10.12.8.0/22 (NetBox/region config, off-repo). Close BEFORE `tofu apply`. | +| SEC-010 | 2026-07-16 | **metal-admin DC-LOCAL invariant (D-052/D-100) is PRESERVED but NOT ENFORCED in the committed Stage-3 config.** The `vvr1_dc0` rack straddles metal-admin (10.12.8.0/22, DC-local) + the office1<->dc0 transit (crosses fiber). Its committed cloud-init pins static IPs only -- **no `net.ipv4.ip_forward=0` sysctl, no host firewall on the transit leg.** Cross-plane routing (metal-admin <-> the whole Office1 /22, bidirectional) is blocked ONLY by Ubuntu's distro default; the deferred MAAS-rack install could silently flip it. A MAAS rack proxies at the application layer and needs no kernel forwarding, so pinning is free. **MODEL B UPDATE (2026-07-16):** after the D-123 Model B reshape, the OUTER `vvr1-dc0` is single-leg (transit only); metal-admin + the other 5 planes are now INNER bridges on `vvr1-dc0`'s own libvirtd. The forwarding hardening is MORE critical (vvr1-dc0 now bridges ALL 6 inner planes + the transit) and belongs on the inner libvirt host, not just a 2-leg rack -- wire it in the C3 bootstrap (`site-headend-install.sh` node-host mode). | 2026-07-16 plane-segregation review + Model B reshape cross-check; `opentofu/main.tf` `module "vvr1_dc0"` | operator | **OPEN -- artifact COMMITTED (Phase D 2026-07-16), gated on apply+verify.** Enforced via a FORWARD-drop across the transit leg, NOT a global `ip_forward=0` (the inner `vr1-dc0-wan` NAT forces `ip_forward=1` on vvr1-dc0, so a global sysctl is unusable under Model B). `scripts/site-headend-install.sh --host-nodes` writes `/etc/nftables-sec010.nft` (drop FORWARD in+out the transit interface, default `mgmt`) + a boot-persistent `sec010-fw.service`; `--host-nodes --check` is the MECHANICAL pre-apply gate (fails if the rule is absent). Nothing routes across the fiber THROUGH vvr1-dc0; the rack proxies MAAS at the app layer (originated/terminated, not forwarded). Same pin follows onto `voffice1` when its transit leg is wired. Region route must target only the rack transit /30, never 10.12.8.0/22. Stays OPEN until applied + verified on vvr1-dc0. | | SEC-011 | 2026-07-16 | **Node least-connectivity gap (not an L2 breach).** Under D-121 Option C role separation, all nodes get a uniform 6-plane NIC set, so a ceph-osd STORAGE node has a leg on provider-public (external/FIP) + data-tenant (tenant geneve) -- planes it never binds per D-052. Planes stay isolated L2 (no crosstalk). | 2026-07-16 plane-segregation review; `opentofu/main.tf` `local.vr1_dc0_node_nics` | operator | **CLOSED 2026-07-16 (operator ruling -- keep uniform 6-NIC).** Review R3-F10: A2's cross-examination refuted the attack-surface concern -- in the isolated-L2 sim the unbound vNICs have no reachability out, and pruning would INCREASE Roosevelt-delta (baremetal trunks all VLANs to every node on bonded NICs, so all planes are present regardless of L3 binding). Uniform 6-NIC is the more Roosevelt-faithful model. Accepted non-issue; no code change. | **STANDING CONVENTION (SEC-009, 2026-07-15): per-site credential/env consolidation.** ALL sensitive diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 25b352f..02485e7 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -203,9 +203,53 @@ - **Phase B DONE** -- `scripts/dc-dc-whole-host-budget.py` + `tests/dc-dc-whole-host-budget/` (13/13) replace the lost `optc-calc.py` (R3-F06). Re-validation: Model B 3+2+4 = 838 GiB/1024 FIT (186 headroom). Gauntlet 63 ALL GREEN. -- **Phase C NOT STARTED** -- the Model B OpenTofu reshape (retarget node-vm/planes/wan to `vvr1-dc0`'s - inner libvirt; resize `vvr1-dc0` ~404 GiB + `expose_nested_virt=true`; add `storage-04`; retarget - maas-vm-host). Heaviest/riskiest phase; checkpoint before starting. `model-a-fallback` tag preserved. +- **Phase C IN PROGRESS** -- Model B OpenTofu reshape. Design cross-checked by a 3-charter agent + workflow (compatibility/best-practice/structure): mechanism = **TWO-ROOT / TWO-STATE split** (forced + by the provider-cannot-depend-on-same-apply-resource invariant; -target + provider-with-unknown-values + are anti-patterns). OUTER root (main.tf, vcloud) = vvr1-dc0 resized + transit + office1; INNER root + (NEW `opentofu/vr1-dc0-substrate/`, own state) = 6 planes + wan + edge + inner pool + 9 nodes, modules + reused VERBATIM. BOOTSTRAP GATE between (extend site-headend-install.sh: nested libvirtd/pool/nested=1/ + base image). Verdict GO-WITH-MITIGATION on depth-4. **[R-5] inner transport = REMOTE qemu+ssh from + Office1 (operator ruling 2026-07-16)** -- tofu on Office1, dials vvr1-dc0 over the D-124 transit; both + states on Office1. **vvr1-dc0 sizing DERIVED (not invented) = 108 vCPU / 416 GiB / ~3000 GiB** + (104 node vCPU + 4 oh; 384 node GiB + 32 oh; 2850 inner pool + OS), whole-host 870/1024 (85%) FIT. + DONE so far: storage-04 (R-3), fallback re-cut R-3-compliant (114d392); **C1 OUTER root** (main.tf: + vvr1_dc0 resized via var.vvr1_dc0_* [108/425984/3221225472000, derived], expose_nested_virt=true, + single transit leg; removed planes/wan/edge/node blocks + the dangling moved block; tofu validate + Success); **C2 INNER root** `opentofu/vr1-dc0-substrate/` (versions/variables/main.tf: qemu+ssh + provider + inner pool + 6 planes + wan + edge + 9 node-vm, modules reused verbatim; init+validate + Success). Both roots validate; repo-lint 0-fail. Cross-check agent running on the two-root impl. + **C3 DONE + REVIEWED** -- `site-headend-install.sh` `--host-nodes` (rack-only) node-host mode: installs + qemu-kvm+libvirt, persists+verifies kvm nested=1, adds ssh user to libvirt group, inner pool dir + + AppArmor grant, verifies/instructs the OPNsense base (delegated to opnsense-prep-image.sh). Harness + 32->45/45. Review-after-step (per operator discipline): gauntlet ALL GREEN (63), repo-lint 0-fail, + ASCII/CR clean, grounding OK (implements the C3 bootstrap the inner-root header names), drift sweep + clean (record/docs/code all reference "node-host mode" consistently). `docs/changelog-20260716-review- + sweep-phaseC.md`. + **PHASE D + REMAINDER DONE + REVIEWED (2026-07-16):** + - **SEC-010 (Phase D)** -- artifact COMMITTED in the node-host bootstrap: a FORWARD-drop across the + transit leg (NOT a global ip_forward=0 -- the inner vr1-dc0-wan NAT forces ip_forward=1), written to + /etc/nftables-sec010.nft + boot-persistent sec010-fw.service; `--host-nodes --check` is the mechanical + pre-apply gate. Harness 45->48/48. SEC-010 ledger row reconciled to the code. Stays OPEN until + applied+verified on vvr1-dc0. + - **maas-vm-host** -- header updated for Model B (register vvr1-dc0's INNER/local virsh to the Office1 + region; no cross-fiber dial). Still DEFERRED (DOCFIX-179; not instantiated). + - **C5 runbook reorg** -- `dc-dc-phase2-tofu-dc-substrate.md` gains the "D-123 MODEL B RESHAPE" section: + outer apply -> bootstrap gate (site-headend --host-nodes) -> inner apply -> maas-register; Step 6's + "8 nodes" superseded by R-3's 9. + - **model-a-fallback-plan.md** synced (~416 GiB; inner-root + node-host mode added to the B->A delta + + revert procedure). + Review-after-each-step applied throughout (harness/validate/lint/ASCII/drift per step). All Phase C+D + items COMPLETE. NOTHING applied/committed since 114d392; Phase C/D edits present-only in the working + tree. REMAINING (operator-gated, deploy-time): the actual outer/bootstrap/inner APPLIES + rack-IP + assignment (Phase E, L1-07) + the depth-4 nested-virt L4 boot gate. + - **Model B cross-check (agent) = FAITHFUL, no BLOCKING**; fixes applied: 32-GiB-overhead RATIFIED in + the record (870/154, vvr1-dc0 416 GiB), false metal-admin-IP comment corrected, SEC-010 updated for the + single-leg-outer/bridges-6-inner-planes reality. **OBS-3 (logged):** under Model B the inner + `vr1-dc0-wan` NAT has NO upstream (vvr1-dc0 has only the transit leg + no default route per D-107); the + simulated-ISP uplink is inert unless vvr1-dc0 is given an egress path -- confirm this is intended (closed + test, edge-boundary-only) or wire a controlled egress. Transit IP is duplicated across roots + (`vr1_dc0_rack_transit_ip` outer / `vvr1_dc0_transit_ip` inner) -- keep equal in tfvars. - **Phase D NOT STARTED** -- SEC-010 rack `ip_forward=0` + firewall artifact + mechanical pre-apply gate. - **Phase E NOT STARTED** -- operator assigns rack transit/30 + rack IP in office1-netbox (apex token), fill tfvars -> `tofu plan` renders. diff --git a/opentofu/main.tf b/opentofu/main.tf index 36a72de..5ca47fe 100644 --- a/opentofu/main.tf +++ b/opentofu/main.tf @@ -21,13 +21,16 @@ # ---- DC1: inherits the DC0 six-plane v4 layout unchanged (D-101) ---- -module "vr1_dc0_planes" { - source = "./modules/dc-planes" - dc_name = "vr1-dc0" - domain_suffix = var.domain_suffix - mtu = var.underlay_mtu - planes = var.vr1_dc0_planes -} +# D-123 MODEL B (2026-07-16): the 6 vr1-dc0 planes MOVED to the INNER root +# `opentofu/vr1-dc0-substrate/` -- under Model B the nodes live INSIDE vvr1-dc0, so +# their planes are created on vvr1-dc0's OWN libvirt (via the qemu+ssh inner provider, +# R-5), not on vcloud. A libvirt provider cannot be configured from a resource created +# in the same apply, so the inner substrate is a separate root/state applied AFTER +# vvr1-dc0 is up (the bootstrap gate). See docs/stage3-adversarial-review-20260716.md +# (R3-F01 / Phase C) + docs/model-a-fallback-plan.md. +# NOTE: module.vr1_dc0_planes was applied at vcloud (state serial 21, empty); the outer +# apply will DESTROY those 6 empty networks (gated, verify with `tofu plan`) and the +# inner apply recreates them inside vvr1-dc0. module "vr1_dc0_storage" { source = "./modules/dc-storage-pool" @@ -240,11 +243,9 @@ # STOP -- standing lesson 2: an apply touching a libvirt_domain's devices is an # OUTAGE, and "updated in-place" does NOT mean "no restart". # ----------------------------------------------------------------------------- -moved { - from = module.dc1_planes - to = module.vr1_dc0_planes -} - +# D-123 MODEL B: the moved{dc1_planes -> vr1_dc0_planes} block was DELETED here -- +# vr1_dc0_planes is no longer in this (outer) root (it moved to the inner root), and a +# `moved` block cannot target an address absent from the config / cross state files. moved { from = module.dc1_storage to = module.vr1_dc0_storage @@ -289,69 +290,19 @@ # below are the D-123-independent substrate (D-103/D-114 as-built) plus the rack. # ===================================================================== -# D-122: the dedicated per-site simulated-ISP uplink (172.30.2.0/24, D-115). -module "vr1_dc0_wan" { - source = "./modules/site-wan" - network_name = "vr1-dc0-wan" - cidr = "172.30.2.0/24" - # mtu defaults to 1500 (ISP-uplink domain; NOT the jumbo planes/mesh). -} - -# D-122: the DC edge -- 2-NIC (WAN + LAN), Office1 pattern (2048/2/nano). The six -# planes are fabric-routed (OVN/OpenStack), so the edge is only the external -# boundary: LAN = provider-public (edge is its external gateway, D-100), WAN = -# the dedicated uplink above. -module "vr1_dc0_opnsense" { - source = "./modules/opnsense-edge" - vm_name = "vr1-dc0-opnsense" - memory_mib = 2048 - vcpu = 2 - pool_name = module.vr1_dc0_storage.pool_name - # No disk_size_bytes: nano direct-copy, sized by opnsense-prep-image.sh (DOCFIX-189). - # PREREQUISITE (runbook Step 4): the prepped nano image must exist at this path. - base_volume_path = "/var/lib/libvirt/vr1/vr1-dc0/opnsense-26.1-nano.qcow2" - lan_network_name = module.vr1_dc0_planes.network_names["provider-public"] - wan_network_name = module.vr1_dc0_wan.network_name -} - -# D-121 Option C node layout (R-3 AMENDMENT 2026-07-16): 3 control + 2 compute + 4 storage -# (16/12/8 vCPU, 64/48/24 GiB, 150/100/550 GiB) = 9 nodes/DC. Storage 3->4 restores Ceph -# rebuild headroom (R3-F08). Six NICs each (one per plane, baremetal-matched D-122); -# metal-admin FIRST = the PXE/boot plane (D-052 default binding). -locals { - vr1_dc0_node_nics = [ - module.vr1_dc0_planes.network_names["metal-admin"], # PXE / boot fabric first - module.vr1_dc0_planes.network_names["provider-public"], - module.vr1_dc0_planes.network_names["metal-internal"], - module.vr1_dc0_planes.network_names["data-tenant"], - module.vr1_dc0_planes.network_names["storage"], - module.vr1_dc0_planes.network_names["replication"], - ] - - vr1_dc0_nodes = { - "vr1-dc0-control-01" = { vcpu = 16, mem = 65536, disk_gib = 150 } - "vr1-dc0-control-02" = { vcpu = 16, mem = 65536, disk_gib = 150 } - "vr1-dc0-control-03" = { vcpu = 16, mem = 65536, disk_gib = 150 } - "vr1-dc0-compute-01" = { vcpu = 12, mem = 49152, disk_gib = 100 } - "vr1-dc0-compute-02" = { vcpu = 12, mem = 49152, disk_gib = 100 } - "vr1-dc0-storage-01" = { vcpu = 8, mem = 24576, disk_gib = 550 } - "vr1-dc0-storage-02" = { vcpu = 8, mem = 24576, disk_gib = 550 } - "vr1-dc0-storage-03" = { vcpu = 8, mem = 24576, disk_gib = 550 } - "vr1-dc0-storage-04" = { vcpu = 8, mem = 24576, disk_gib = 550 } # R-3 (2026-07-16): 4th OSD host - } -} - -module "vr1_dc0_node" { - for_each = local.vr1_dc0_nodes - - source = "./modules/node-vm" - vm_name = each.key - vcpu = each.value.vcpu - memory_mib = each.value.mem - disk_size_bytes = each.value.disk_gib * 1024 * 1024 * 1024 - pool_name = module.vr1_dc0_storage.pool_name - network_names = local.vr1_dc0_node_nics -} +# ===================================================================== +# D-123 MODEL B (2026-07-16): the vr1-dc0 INNER SUBSTRATE moved to the inner root. +# The site-wan uplink, the 2-NIC OPNsense edge, the R-3 Option-C node layout (3 control +# + 2 compute + 4 storage = 9 nodes/DC) and the 9 node VMs are created INSIDE vvr1-dc0 +# by the inner root `opentofu/vr1-dc0-substrate/` (provider = qemu+ssh to vvr1-dc0 over +# the D-124 transit, R-5). Their HCL is UNCHANGED -- the same node-vm / site-wan / +# opnsense-edge / dc-planes modules are reused verbatim; only the provider they run +# against moved from vcloud to vvr1-dc0's inner libvirt. This keeps the whole nested +# substrate one wholesale-deletable Section-9 shim (D-103) that collapses to a single +# root at Roosevelt. Apply order: OUTER (this root -- boots + sizes vvr1-dc0) -> +# BOOTSTRAP GATE (site-headend-install.sh: install nested libvirtd + inner pool + +# kvm nested=1 + stage the opnsense base image) -> INNER root (planes/wan/edge/nodes). +# ===================================================================== # netem (runbook Step 11) -- HELD as a comment. It is a POST-apply step that runs # SSH-wrapped provisioners from Office1 to the vcloud host, so it needs a real @@ -367,27 +318,32 @@ # } # ===================================================================== -# D-123 / D-124: the vr1-dc0 MAAS RACK CONTROLLER (vvr1-dc0). An Ubuntu VM (reusing -# the noble base) that will run a MAAS rack controller enrolled to the Office1 MAAS -# REGION. Two legs: metal-admin (serves PXE/DHCP to the vr1-dc0 nodes) + the -# office1<->dc0 mesh transit (reaches the region, D-124 Scheme A). It PROXIES OS -# images from the region, so it needs no direct DC egress (D-107). NOT YET APPLIED. -# The MAAS rack INSTALL itself is a gated post-boot step (an adapted -# site-headend-install.sh rack-only mode -- follow-up), like voffice1's install. +# D-123 MODEL B / D-124 AMENDMENT: vvr1-dc0 -- the vr1-dc0 SITE CONTAINMENT VM. Under +# Model B it is BOTH the MAAS rack controller (enrolled to the Office1 region) AND the +# nested libvirt HOST for the DC's entire node fleet (9 nodes + 6 planes + wan + edge, +# all created by the inner root). Sized to hold that fleet (~416 GiB, derived) with +# expose_nested_virt=true. ONE outer leg: the office1<->dc0 transit (region, D-124 +# Scheme A); metal-admin + the other planes are INNER bridges. It PROXIES OS images from +# the region (D-107). Site-down = a single `virsh destroy vvr1-dc0` (the D-122 intent). +# NOT YET APPLIED. The gated post-boot BOOTSTRAP (site-headend-install.sh node-host mode: +# rack enroll + nested libvirtd + inner pool + kvm nested=1 + base image) precedes the +# inner-root apply. # ===================================================================== module "vvr1_dc0" { source = "./modules/cloudinit-vm" vm_name = "vvr1-dc0" - vcpu = 4 # D-124 (rack controller: lightweight) - memory_mib = 8192 # D-124 - disk_size_bytes = 80 * 1024 * 1024 * 1024 # D-124: 80 GiB + vcpu = var.vvr1_dc0_vcpu # D-123 Model B / D-124 AMENDMENT: hosts the DC's node fleet + memory_mib = var.vvr1_dc0_memory_mib # derived 416 GiB (384 node + 32 overhead) + disk_size_bytes = var.vvr1_dc0_disk_bytes # derived ~3000 GiB (holds the inner libvirt pool) pool_name = module.vr1_dc0_storage.pool_name base_volume_path = module.ubuntu_noble_base.path - expose_nested_virt = false # rack-only: the region + its LXD host compose VMs, not the rack + expose_nested_virt = true # D-123 Model B: LOAD-BEARING -- inner node VMs need svm passthrough or no inner KVM / nova boots + # D-123 Model B: metal-admin is now an INNER bridge (created by the inner root's + # dc-planes INSIDE vvr1-dc0; the rack serves PXE/DHCP there). So the OUTER vvr1-dc0 has + # ONE leg -- the office1<->dc0 transit that reaches the Office1 region (D-124 Scheme A). network_names = [ - module.vr1_dc0_planes.network_names["metal-admin"], # leg 1: node-facing PXE/DHCP - module.mesh_vr1_dc0_office1.network_name, # leg 2: office1<->dc0 transit (region) + module.mesh_vr1_dc0_office1.network_name, # office1<->dc0 transit (region-facing) ] user_data = <<-EOT @@ -421,17 +377,18 @@ # this (documented trap; modules/opnsense-edge + the runbook). metal-admin gives # the rack no default route; the region route rides the transit toward Office1 # (10.10.0.0/22, as-built D-115). + # Single OUTER leg = the transit (first/only NIC = enp1s0; interface-naming trap still + # applies -- CONFIRM on first boot). metal-admin is configured on the INNER bridge by the + # BOOTSTRAP step (site-headend-install.sh node-host mode) using var.vr1_dc0_rack_metal_admin_ip + # -- that var is consumed by the bootstrap script, NOT by this outer root nor the inner + # OpenTofu root (it remains a required tfvar for the L1-07 rack-addressing gate). The region + # route rides the transit toward Office1 (10.10.0.0/22, as-built D-115). network_config = <<-EOT version: 2 ethernets: - admin: - match: - name: "enp1s0" - set-name: admin - addresses: ["${var.vr1_dc0_rack_metal_admin_ip}/22"] mgmt: match: - name: "enp2s0" + name: "enp1s0" set-name: mgmt addresses: ["${var.vr1_dc0_rack_transit_ip}/${var.vr1_dc0_rack_transit_prefix}"] routes: diff --git a/opentofu/modules/maas-vm-host/main.tf b/opentofu/modules/maas-vm-host/main.tf index de99dad..7d1931d 100644 --- a/opentofu/modules/maas-vm-host/main.tf +++ b/opentofu/modules/maas-vm-host/main.tf @@ -3,6 +3,13 @@ # that DC's MAAS rack controller as a virsh VM-host, so MAAS DISCOVERS the # OpenTofu-created node VMs"). # +# D-123 MODEL B (2026-07-16): "the vcloud host's libvirt" above is SUPERSEDED for the DCs -- +# under Model B the node VMs live INSIDE vvr1-dc0, so the vm-host to register is vvr1-dc0's OWN +# (INNER) virsh -- power_address = qemu:///system LOCAL to the rack (vvr1-dc0 is BOTH the MAAS +# rack AND the libvirt host, so no cross-fiber virsh dial). Still DEFERRED (DOCFIX-179: a maas +# provider block forces every plan to demand creds); wire in its own gated step after the region +# + inner nodes exist, registering to the Office1 REGION. +# # DELIBERATELY uses `maas_vm_host`, NOT `maas_vm_host_machine` -- confirmed # by reading both resources' real schemas (canonical/terraform-provider-maas, # fetched 2026-07-09): `maas_vm_host_machine` takes `cores`/`memory`/ diff --git a/opentofu/variables.tf b/opentofu/variables.tf index 9718811..877d440 100644 --- a/opentofu/variables.tf +++ b/opentofu/variables.tf @@ -97,6 +97,30 @@ type = number } +# ---- D-123 MODEL B (2026-07-16) + D-124 AMENDMENT: vvr1-dc0 is now the DC's node HOST, +# not a 4/8/80 rack. Sizing DERIVED (not invented, hard rule 2) from the ruled R-3 layout +# via scripts/dc-dc-whole-host-budget.py: one DC fleet (3 control + 2 compute + 4 storage) +# = 104 vCPU / 384 GiB / 2850 GiB thin, + containment overhead (4 vCPU / 32 GiB / ~150 GiB +# OS). Whole-host 2-DC total = 870/1024 GiB (85%), FIT. Defaults encode the derivation; +# override in tfvars only with a re-derived value. ---- +variable "vvr1_dc0_vcpu" { + description = "vvr1-dc0 vCPU (Model B: hosts one DC's 9 node VMs). Derived: 104 node + 4 overhead." + type = number + default = 108 +} + +variable "vvr1_dc0_memory_mib" { + description = "vvr1-dc0 RAM in MiB (Model B). Derived: 384 GiB node fleet + 32 GiB overhead = 416 GiB." + type = number + default = 425984 +} + +variable "vvr1_dc0_disk_bytes" { + description = "vvr1-dc0 boot disk in bytes (Model B: holds the inner libvirt pool). Derived: ~2850 GiB node disks (thin) + ~150 GiB OS = ~3000 GiB." + type = number + default = 3221225472000 +} + # ---- D-124: vr1-dc0 MAAS rack controller (vvr1-dc0) addressing. NO defaults -- # these come from office1-netbox (the IPAM apex) via the D-124 importer, then # into a tfvars entry. Do NOT invent them here (hard rule 2). The rack straddles diff --git a/opentofu/vr1-dc0-substrate/.terraform.lock.hcl b/opentofu/vr1-dc0-substrate/.terraform.lock.hcl new file mode 100644 index 0000000..f6d738b --- /dev/null +++ b/opentofu/vr1-dc0-substrate/.terraform.lock.hcl @@ -0,0 +1,35 @@ +# This file is maintained automatically by "tofu init". +# Manual edits may be lost in future updates. + +provider "registry.opentofu.org/dmacvicar/libvirt" { + version = "0.9.8" + constraints = "0.9.8" + hashes = [ + "h1:/9TSq9ibeojseTyT6Jx3z/dbvT5uw8ka3/18NKonrQc=", + "h1:BN5hy1RaHlpcV0BDJYrSOEklK4YNdZCpiXOOGn0ce9k=", + "h1:MNJ6dcFru4DwQM7maIUAADJIHn3aF+L8XsnRJlleNaM=", + "h1:P1Cx0VZGTe6cpWpQ4JFOPJJb+Z+N50t4e1VA76T1tAA=", + "h1:Pm9du7Rzsg1bhLq+XnW9VaLSjTu0Yytr9AOdP9FLQvM=", + "h1:Q/JvNhRskJqDKPJoO9DW3+Ons3mk8kc++zlT4TucyFw=", + "h1:Tzk+wRzhboY3F3wB1u0y3ySMMd4ZPtJbOYlTbbkxZc0=", + "h1:Uuk8gkDEzmXfOTW/nm8GMG2aen7n/+X5GsTAf7eVRJc=", + "h1:cRBlnB/TG4+l0y0i42QLU0JoLDBxxJEwCguLXS9Rv8A=", + "h1:jn+hA9lIJzH5T0550IaKNsHxEAbyTGtvYWm02JCxexM=", + "h1:pKAwAcCGEwsKvVy5eIxdmb1gLhTTxB0xGVi8/jqj0uI=", + "h1:wS5VlGinhpnMCva0tKwLRPRQxmMUl80L05AueWshzNg=", + "h1:yqZeKoJ+EZc3687/+ZBqBmtwzvBPLNwaEHW74+bSc6Y=", + "zh:061e5187853729e1d8ba20938402ad6e778b4097436925d0bef7741c8aa26ee1", + "zh:69a2ac8ee0cb0c1581bdc9a1296d89abcd1808229ff0a2fa20678ba4f7472944", + "zh:8b02bf349191b4e7d9529d620da9e9265de5bd01d6983bb079589acdf80dda7a", + "zh:9620872ed1da1ff421e85dd4ba2a110c284f3f0d06369c36a9aa75d501155fad", + "zh:9ae7c932b7fb6f62ff20b863a15a7cd713415faa4c789472dc403062808366cb", + "zh:a369c3e271b49e085940e6181fe6c1086c2126eebf5c7ad2b1db8fc8792dd30c", + "zh:a45ad97d7192bea104b32521839dd705a330f13364cf1007dc912b6a0b6be3c8", + "zh:b69f2a13ff688ad4b79e14deb6ef6341187864f50960bc0fd9c682f6077b4318", + "zh:c20b467ff0ac76431a7bdb99662b9414b922ea603327509518c0179416cdfea3", + "zh:c22d45913bd1f00a33d1a4b80547e4a2854409a298a0868c1730124f03a210ad", + "zh:cf82bb14884fcbdcd57620cee248994b1633ea246fa49fd668d2f1739abb982d", + "zh:ddc1cb4417206dd46eb22c3b81204b3a5edda03c4436e6a5914d541fcfd4c27f", + "zh:e6b1983b8f1b9658448d0d09635619cc75dde314e6b4d7fd16d827f9e4bdb767", + ] +} diff --git a/opentofu/vr1-dc0-substrate/main.tf b/opentofu/vr1-dc0-substrate/main.tf new file mode 100644 index 0000000..72a1a8e --- /dev/null +++ b/opentofu/vr1-dc0-substrate/main.tf @@ -0,0 +1,92 @@ +# ===================================================================== +# D-123 MODEL B (2026-07-16) -- vr1-dc0 INNER SUBSTRATE, created inside vvr1-dc0. +# Run this root AFTER: (1) the outer opentofu/ root has booted + sized vvr1-dc0 +# (expose_nested_virt=true), and (2) the bootstrap gate (site-headend-install.sh +# node-host mode) has installed libvirtd + the inner pool + kvm nested=1 + the +# OPNsense base image on vvr1-dc0. The provider dials vvr1-dc0 over the D-124 transit +# (R-5: remote qemu+ssh from Office1). All module bodies are UNCHANGED from the outer +# root -- only the provider they run against moved (vcloud -> vvr1-dc0). See +# docs/model-a-fallback-plan.md for the Model A revert. +# ===================================================================== + +provider "libvirt" { + # qemu+ssh to vvr1-dc0 over the transit. Key auth + known_hosts verified out of band + # (NO no_verify). The IP is MEASURED after the outer apply (hard rule 2). + uri = "qemu+ssh://${var.vvr1_dc0_ssh_user}@${var.vvr1_dc0_transit_ip}/system" +} + +# Inner storage pool (backs the node + edge disks) -- created inside vvr1-dc0 at the +# path the bootstrap step provisioned. +module "inner_storage" { + source = "../modules/dc-storage-pool" + dc_name = "vr1-dc0-inner" + target_path = var.inner_pool_path +} + +# The six vr1-dc0 planes (D-101 layout, unchanged) -- now isolated-L2 networks INSIDE +# vvr1-dc0. metal-admin is where the rack serves PXE/DHCP to the inner nodes. +module "vr1_dc0_planes" { + source = "../modules/dc-planes" + dc_name = "vr1-dc0" + domain_suffix = var.domain_suffix + mtu = var.underlay_mtu + planes = var.vr1_dc0_planes +} + +# D-122: the dedicated per-site simulated-ISP uplink (172.30.2.0/24, D-115). +module "vr1_dc0_wan" { + source = "../modules/site-wan" + network_name = "vr1-dc0-wan" + cidr = "172.30.2.0/24" + # mtu defaults to 1500 (ISP-uplink domain; NOT the jumbo planes/mesh). +} + +# D-122: the DC edge -- 2-NIC (WAN + LAN), Office1 pattern (2048/2/nano). LAN = +# provider-public (external gateway, D-100); WAN = the dedicated uplink above. +module "vr1_dc0_opnsense" { + source = "../modules/opnsense-edge" + vm_name = "vr1-dc0-opnsense" + memory_mib = 2048 + vcpu = 2 + pool_name = module.inner_storage.pool_name + base_volume_path = var.opnsense_base_path + lan_network_name = module.vr1_dc0_planes.network_names["provider-public"] + wan_network_name = module.vr1_dc0_wan.network_name +} + +# D-121 Option C layout (R-3 2026-07-16): 3 control + 2 compute + 4 storage = 9 nodes/DC. +# Six NICs each (one per plane); metal-admin FIRST = PXE/boot plane (D-052 default binding). +locals { + vr1_dc0_node_nics = [ + module.vr1_dc0_planes.network_names["metal-admin"], # PXE / boot fabric first + module.vr1_dc0_planes.network_names["provider-public"], + module.vr1_dc0_planes.network_names["metal-internal"], + module.vr1_dc0_planes.network_names["data-tenant"], + module.vr1_dc0_planes.network_names["storage"], + module.vr1_dc0_planes.network_names["replication"], + ] + + vr1_dc0_nodes = { + "vr1-dc0-control-01" = { vcpu = 16, mem = 65536, disk_gib = 150 } + "vr1-dc0-control-02" = { vcpu = 16, mem = 65536, disk_gib = 150 } + "vr1-dc0-control-03" = { vcpu = 16, mem = 65536, disk_gib = 150 } + "vr1-dc0-compute-01" = { vcpu = 12, mem = 49152, disk_gib = 100 } + "vr1-dc0-compute-02" = { vcpu = 12, mem = 49152, disk_gib = 100 } + "vr1-dc0-storage-01" = { vcpu = 8, mem = 24576, disk_gib = 550 } + "vr1-dc0-storage-02" = { vcpu = 8, mem = 24576, disk_gib = 550 } + "vr1-dc0-storage-03" = { vcpu = 8, mem = 24576, disk_gib = 550 } + "vr1-dc0-storage-04" = { vcpu = 8, mem = 24576, disk_gib = 550 } # R-3: 4th OSD host + } +} + +module "vr1_dc0_node" { + for_each = local.vr1_dc0_nodes + + source = "../modules/node-vm" + vm_name = each.key + vcpu = each.value.vcpu + memory_mib = each.value.mem + disk_size_bytes = each.value.disk_gib * 1024 * 1024 * 1024 + pool_name = module.inner_storage.pool_name + network_names = local.vr1_dc0_node_nics +} diff --git a/opentofu/vr1-dc0-substrate/variables.tf b/opentofu/vr1-dc0-substrate/variables.tf new file mode 100644 index 0000000..8796726 --- /dev/null +++ b/opentofu/vr1-dc0-substrate/variables.tf @@ -0,0 +1,40 @@ +# Inner-root inputs. The two connection values (transit IP + ssh user) are MEASURED +# after vvr1-dc0 boots (hard rule 2 -- not invented); the rest mirror the outer root. + +variable "vvr1_dc0_transit_ip" { + description = "vvr1-dc0's static transit IP (D-124 Scheme A, NetBox-assigned) -- where this root's qemu+ssh inner provider connects. MEASURED after the outer apply boots vvr1-dc0." + type = string +} + +variable "vvr1_dc0_ssh_user" { + description = "SSH user on vvr1-dc0 for the qemu+ssh inner provider (the cloud-init user, e.g. jessea123). Key auth + known_hosts verified out of band (NO no_verify)." + type = string + default = "jessea123" +} + +variable "inner_pool_path" { + description = "Directory path INSIDE vvr1-dc0 for the inner libvirt storage pool that backs the node/edge disks (created by the bootstrap step)." + type = string + default = "/var/lib/libvirt/vr1-dc0-inner" +} + +variable "opnsense_base_path" { + description = "Path INSIDE vvr1-dc0 to the prepped OPNsense nano qcow2 (staged by the bootstrap step, opnsense-prep-image.sh run on vvr1-dc0)." + type = string + default = "/var/lib/libvirt/vr1-dc0-inner/opnsense-26.1-nano.qcow2" +} + +variable "domain_suffix" { + description = "DNS domain suffix for the planes (mirrors the outer root)." + type = string +} + +variable "underlay_mtu" { + description = "Jumbo MTU for the internal fabric planes/mesh (9000). The site-wan uplink stays 1500." + type = number +} + +variable "vr1_dc0_planes" { + description = "The six vr1-dc0 planes (CIDR map), copied from the outer root's var of the same name -- the D-101 layout, unchanged. dc-planes creates them as isolated-L2 networks inside vvr1-dc0." + type = any +} diff --git a/opentofu/vr1-dc0-substrate/versions.tf b/opentofu/vr1-dc0-substrate/versions.tf new file mode 100644 index 0000000..9ef5a8c --- /dev/null +++ b/opentofu/vr1-dc0-substrate/versions.tf @@ -0,0 +1,20 @@ +# INNER root for D-123 MODEL B (2026-07-16). The vr1-dc0 substrate -- 6 dc-planes, +# the site-wan uplink, the OPNsense edge, an inner storage pool, and the 9 node VMs +# -- created INSIDE vvr1-dc0 via the qemu+ssh inner provider (R-5), NOT on vcloud. +# This is a SEPARATE root + state from opentofu/ (the outer root) because a libvirt +# provider cannot be configured from a resource created in the same apply: vvr1-dc0 +# must exist, be booted, and have libvirtd reachable before this root can plan. Apply +# order: outer (opentofu/) boots vvr1-dc0 -> bootstrap gate (site-headend-install.sh +# installs nested libvirtd/pool/kvm-nested/base-image) -> THIS root. The modules are +# reused VERBATIM from ../modules/ so the whole shim collapses to a single root at +# Roosevelt. Provider pin matches the outer root's versions.tf (0.9.8, confirmed). +terraform { + required_version = ">= 1.6.0" + + required_providers { + libvirt = { + source = "dmacvicar/libvirt" + version = "0.9.8" + } + } +} diff --git a/runbooks/dc-dc-phase2-tofu-dc-substrate.md b/runbooks/dc-dc-phase2-tofu-dc-substrate.md index fa61feb..9f281cf 100644 --- a/runbooks/dc-dc-phase2-tofu-dc-substrate.md +++ b/runbooks/dc-dc-phase2-tofu-dc-substrate.md @@ -141,6 +141,34 @@ -> EXIT GATE -> Stage 4 (per-DC, once vr1-dc1 clears its own gate) ``` +### D-123 MODEL B RESHAPE (2026-07-16) -- the single-root sequence above is REGROUPED; read this + +The operator ruled D-123 = **Model B**: the vr1-dc0 node fleet nests INSIDE the containment VM +`vvr1-dc0` (single-object `virsh destroy vvr1-dc0` = site-down). A libvirt provider cannot target a VM +created in the same apply, so the substrate is now TWO ROOTS with a bootstrap gate between. The Steps +above still describe the WORK; they regroup under the new roots + a new apply order: + +- **A. OUTER apply** (`opentofu/`, vcloud) -- creates + sizes `vvr1-dc0` (~416 GiB, `expose_nested_virt=true`, + single transit leg) + the transit. Steps 1-3 (preflight/decisions) apply here. `[MUTATION: gated]` +- **B. BOOTSTRAP GATE** on vvr1-dc0 (after it boots): `scripts/site-headend-install.sh --role rack + --host-nodes --region-url ... --enroll-secret-file ...` -- enroll the rack to the Office1 region AND + make vvr1-dc0 a nested libvirt host (qemu/libvirt, `kvm nested=1`, inner pool, AppArmor, the SEC-010 + transit FORWARD-drop). Prep the OPNsense base ON vvr1-dc0 (Step 4 -- run `opnsense-prep-image.sh` + there). GATE: `site-headend-install.sh --role rack --host-nodes --check` must pass (incl. SEC-010). +- **C. INNER apply** (`opentofu/vr1-dc0-substrate/`, provider = qemu+ssh to vvr1-dc0 over the transit, + R-5) -- the 6 planes + `vr1-dc0-wan` + edge + inner pool + 9 node VMs. Steps 5-6 (wire edge/nodes) are + ALREADY authored in this inner root; Step 7 (init/validate/plan) + Step 8 (apply) run HERE, against + vvr1-dc0's libvirt, NOT vcloud. `[MUTATION: gated]` +- **D. maas-vm-host** (Step 9) -- register vvr1-dc0's OWN (inner, local `qemu:///system`) virsh to the + Office1 REGION; MAAS discovers the inner node VMs (Step 10). Still DEFERRED (DOCFIX-179). +- **E. netem** (Step 11) -- UNCHANGED: runs on the vcloud-level mesh bridges (virbr7), outer. + +Node count is R-3 = 3 control + 2 compute + **4 storage = 9/DC** (Step 6's "8" is superseded). The Model +A single-root layout is preserved as a revert (git tag `model-a-fallback` + `docs/model-a-fallback-plan.md`). +Sizing derived by `scripts/dc-dc-whole-host-budget.py` (870/1024 GiB, FIT). SEC-010 DC-LOCAL is enforced +by the bootstrap's transit FORWARD-drop (NOT a global `ip_forward=0` -- the inner WAN NAT forces +`ip_forward=1`). See `docs/changelog-20260716-review-sweep-phaseC.md`. + --- ## Step 1 -- Pre-flight: confirm Stage 1 + Stage 2 gates closed (READ-ONLY) diff --git a/scripts/site-headend-install.sh b/scripts/site-headend-install.sh index 979503c..160664f 100644 --- a/scripts/site-headend-install.sh +++ b/scripts/site-headend-install.sh @@ -79,6 +79,14 @@ CHECK=0; DRYRUN=0; COMPOSE_CIDR=""; MAAS_IP="" ROLE="region+rack"; REGION_URL=""; ENROLL_SECRET_FILE="" +# D-123 MODEL B: a DC rack (vvr1-dc0) is ALSO the nested libvirt/KVM host for the INNER +# OpenTofu root that builds the node fleet (opentofu/vr1-dc0-substrate/). --host-nodes adds +# that setup to the rack install (make it a libvirt host + nested KVM + inner pool + base +# image) so the qemu+ssh inner provider (R-5) has something to connect to. +HOST_NODES=0; HOST_USER="jessea123"; INNER_POOL_PATH="/var/lib/libvirt/vr1-dc0-inner"; OPNSENSE_BASE="" +# SEC-010: the transit-leg interface name on vvr1-dc0 (the outer NIC; netplan set-name in the +# outer cloud-init is "mgmt"). The DC-LOCAL FORWARD-drop is keyed on it. +TRANSIT_IF="mgmt" while [ $# -gt 0 ]; do case "$1" in --check) CHECK=1 ;; @@ -88,15 +96,25 @@ --maas-url-ip) shift; MAAS_IP="${1:-}" ;; --region-url) shift; REGION_URL="${1:-}" ;; --enroll-secret-file) shift; ENROLL_SECRET_FILE="${1:-}" ;; + --host-nodes) HOST_NODES=1 ;; + --host-user) shift; HOST_USER="${1:-}" ;; + --inner-pool-path) shift; INNER_POOL_PATH="${1:-}" ;; + --opnsense-base) shift; OPNSENSE_BASE="${1:-}" ;; + --transit-if) shift; TRANSIT_IF="${1:-}" ;; -h|--help) echo "usage: site-headend-install.sh [--check|--dry-run] [--role ]" echo " region+rack (default): --compose-cidr [--maas-url-ip ]" echo " rack: --region-url --enroll-secret-file " + echo " rack + Model B node host (vvr1-dc0): add --host-nodes [--host-user U]" + echo " [--inner-pool-path P] [--opnsense-base QCOW2]" exit 0 ;; *) echo "FAIL: unknown arg '$1'" >&2; exit 2 ;; esac shift done +# --host-nodes only makes sense on a DC rack (the Model B node host); reject on region+rack. +[ "$HOST_NODES" = "1" ] && [ "$ROLE" != "rack" ] && { echo "FAIL: --host-nodes requires --role rack (it is the D-123 Model B DC node host)" >&2; exit 2; } +[ -n "$OPNSENSE_BASE" ] || OPNSENSE_BASE="$INNER_POOL_PATH/opnsense-26.1-nano.qcow2" case "$ROLE" in region+rack|rack) ;; @@ -177,7 +195,101 @@ } # Dispatch check/precheck to the role-appropriate reporter. -do_report() { if [ "$ROLE" = "rack" ]; then report_rack; else report; fi; } +# D-123 Model B node-host: report readiness of vvr1-dc0 as a nested libvirt host (--check). +node_host_check() { + local ok=0 + echo "== node-host (Model B) readiness ==" + have virsh && echo " [ok] libvirt/qemu installed" || { echo " [--] libvirt/qemu NOT installed"; ok=1; } + local kmod=""; [ -d /sys/module/kvm_amd ] && kmod=kvm_amd; [ -d /sys/module/kvm_intel ] && kmod=kvm_intel + if [ -n "$kmod" ] && grep -qiE '^(Y|1)$' "/sys/module/$kmod/parameters/nested" 2>/dev/null; then + echo " [ok] nested KVM ON ($kmod)" + else echo " [--] nested KVM OFF (kmod=${kmod:-none}) -- inner node VMs get no /dev/kvm"; ok=1; fi + id -nG "$HOST_USER" 2>/dev/null | grep -qw libvirt && echo " [ok] $HOST_USER in libvirt group" || { echo " [--] $HOST_USER NOT in libvirt group"; ok=1; } + [ -d "$INNER_POOL_PATH" ] && echo " [ok] inner pool dir $INNER_POOL_PATH" || { echo " [--] inner pool dir $INNER_POOL_PATH missing"; ok=1; } + [ -s "$OPNSENSE_BASE" ] && echo " [ok] opnsense base $OPNSENSE_BASE" || { echo " [--] opnsense base missing -- run: opnsense-prep-image.sh 26.1 $OPNSENSE_BASE"; ok=1; } + # SEC-010 mechanical pre-apply gate: the transit FORWARD-drop must be present (DC-LOCAL, D-052/D-100) + # AND must key on an interface that ACTUALLY EXISTS -- nftables oifname/iifname with an absent name + # loads clean but matches NOTHING (fail-open), which would silently leave DC-LOCAL unenforced. + if ! nft list table inet sec010 >/dev/null 2>&1; then + echo " [--] SEC-010 FORWARD-drop MISSING -- DC-LOCAL NOT enforced (metal-admin could route across the fiber)"; ok=1 + elif ! ip link show "$TRANSIT_IF" >/dev/null 2>&1; then + echo " [--] SEC-010 rule keys on '$TRANSIT_IF' but NO such interface -- rule matches NOTHING (FAIL-OPEN); confirm the transit interface name (--transit-if)"; ok=1 + else + echo " [ok] SEC-010 transit FORWARD-drop present + transit interface '$TRANSIT_IF' exists" + fi + return $ok +} + +# D-123 Model B node-host: make this rack (vvr1-dc0) a nested libvirt/KVM host so the inner +# OpenTofu root (opentofu/vr1-dc0-substrate/, qemu+ssh from Office1, R-5) can build the fleet. +node_host_setup() { + echo "== node-host (D-123 Model B): make this rack a nested libvirt/KVM host for the inner tofu ==" + run "install qemu-kvm + libvirt + qemu-utils" -- apt-get install -y qemu-kvm libvirt-daemon-system qemu-utils || exit 4 + run "enable libvirtd" -- systemctl enable --now libvirtd + # nested KVM (persist + verify): inner guests need /dev/kvm, which needs the host module's + # nested=1 AND the outer tofu's expose_nested_virt=true on vvr1-dc0. + local kmod=""; [ -d /sys/module/kvm_amd ] && kmod=kvm_amd; [ -d /sys/module/kvm_intel ] && kmod=kvm_intel + [ -n "$kmod" ] || { echo "FAIL: no kvm_amd/kvm_intel -- is expose_nested_virt=true on vvr1-dc0? (outer tofu)" >&2; exit 4; } + run "persist nested=1 for $kmod" -- bash -c "echo 'options $kmod nested=1' > /etc/modprobe.d/kvm-nested.conf" + if ! grep -qiE '^(Y|1)$' "/sys/module/$kmod/parameters/nested" 2>/dev/null; then + run "reload $kmod for nested=1" -- bash -c "modprobe -r $kmod 2>/dev/null; modprobe $kmod" \ + || echo " WARN: could not reload $kmod (VMs running?) -- REBOOT vvr1-dc0 to apply nested=1" + fi + grep -qiE '^(Y|1)$' "/sys/module/$kmod/parameters/nested" 2>/dev/null \ + && echo " -> nested KVM ON ($kmod)" || echo " WARN: nested KVM still OFF -- reboot vvr1-dc0 (inner nova will not boot until ON)" + run "add $HOST_USER to libvirt group" -- usermod -aG libvirt "$HOST_USER" + # inner pool dir + AppArmor grant (platform-traps: a non-default libvirt pool path otherwise + # yields qemu 'Permission denied' define-then-fail on the inner domains -- a silent dead guest). + run "create inner pool dir $INNER_POOL_PATH" -- mkdir -p "$INNER_POOL_PATH" + run "AppArmor: allow inner pool path" -- bash -c "install -d /etc/apparmor.d/local/abstractions 2>/dev/null; grep -qF '$INNER_POOL_PATH' /etc/apparmor.d/local/abstractions/libvirt-qemu 2>/dev/null || echo ' \"$INNER_POOL_PATH/**\" rwk,' >> /etc/apparmor.d/local/abstractions/libvirt-qemu; systemctl reload apparmor 2>/dev/null || true" + # SEC-010 (D-052/D-100 DC-LOCAL under Model B): ip_forward is forced ON by the inner vr1-dc0-wan + # NAT, so a global ip_forward=0 is unusable. Enforce DC-local with a FORWARD-drop across the TRANSIT + # leg -- nothing routes ACROSS the fiber THROUGH vvr1-dc0 (the rack proxies MAAS at the app layer = + # originated/terminated, NOT forwarded). vvr1-dc0's own region traffic (OUTPUT/INPUT) is unaffected. + echo "== SEC-010 (DC-LOCAL): FORWARD-drop across the transit leg '$TRANSIT_IF' ==" + run "install nftables" -- apt-get install -y nftables + if [ "$DRYRUN" = "1" ]; then + echo " [dry-run] write /etc/nftables-sec010.nft (FORWARD-drop in+out '$TRANSIT_IF') + enable sec010-fw.service (boot-persistent)" + else + cat > /etc/nftables-sec010.nft < SEC-010 FORWARD-drop loaded (transit leg '$TRANSIT_IF')" + cat > /etc/systemd/system/sec010-fw.service </dev/null 2>&1 && echo " -> sec010-fw.service enabled (boot-persistent)" + fi + + # OPNsense base for the inner edge (opnsense-prep-image.sh needs network; run on this host). + # Verify, do NOT fetch inline (the prep script is not guaranteed present here). + if [ -s "$OPNSENSE_BASE" ]; then echo " -> opnsense base present: $OPNSENSE_BASE" + else echo " NOTE: opnsense base MISSING at $OPNSENSE_BASE -- run on this host BEFORE the inner apply:"; echo " bash scripts/opnsense-prep-image.sh 26.1 $OPNSENSE_BASE"; fi + echo " -> node-host ready. Inner apply next (from Office1): cd opentofu/vr1-dc0-substrate && tofu apply." +} + +do_report() { + if [ "$ROLE" = "rack" ]; then + report_rack; local r=$? + if [ "$HOST_NODES" = "1" ]; then node_host_check || r=1; fi + return $r + else report; fi +} report_st=0 if [ "$CHECK" = "1" ]; then do_report; exit $?; fi @@ -229,6 +341,8 @@ # SKIPPED IN RACK MODE (region concerns): PostgreSQL, maas init region+rack, # createadmin/apikey/login, LXD (5.21 pin), LXD vm-host registration, compose-network DHCP. # metal-admin DHCP (this rack as primary_rack) is enabled REGION-side as a runbook step. + # D-123 MODEL B: if this DC rack is ALSO the node host, set up nested libvirt (respects --dry-run). + [ "$HOST_NODES" = "1" ] && node_host_setup if [ "$DRYRUN" = "1" ]; then echo " (dry-run: no changes made)"; exit 0; fi echo echo "RACK ENROLLED to $REGION_URL. Next (REGION-side runbook, NOT this script): enable" diff --git a/tests/site-headend-install/run-tests.sh b/tests/site-headend-install/run-tests.sh index c954a6a..68b5221 100644 --- a/tests/site-headend-install/run-tests.sh +++ b/tests/site-headend-install/run-tests.sh @@ -98,6 +98,32 @@ defout="$(bash "$S" --dry-run --compose-cidr 10.10.1.0/24 2>&1)" printf '%s' "$defout" | grep -q 'region+rack' && ok || bad "default (no --role) must still run the region+rack flow" +# 8. D-123 MODEL B node-host (--host-nodes): a DC rack that is ALSO the nested libvirt host for +# the inner OpenTofu root. Must attach to rack mode only, set up nested KVM + inner pool, and +# NOT leak the secret or regress the plain-rack contract. +t "--host-nodes without --role rack -> 2" 2 bash "$S" --host-nodes --dry-run +t "rack + --host-nodes --dry-run -> 0" 0 bash "$S" --role rack --host-nodes --dry-run --region-url "$RURL" --enroll-secret-file "$SECFILE" +nhout="$(bash "$S" --role rack --host-nodes --dry-run --region-url "$RURL" --enroll-secret-file "$SECFILE" 2>&1)" +# the node-host setup steps must appear (incl. the SEC-010 DC-LOCAL forward-drop, R3-F02/Phase D) +for step in 'node-host' 'nested=1' 'qemu-kvm' 'inner pool' 'AppArmor' 'libvirt group' 'SEC-010' 'FORWARD-drop'; do + printf '%s' "$nhout" | grep -qi "$step" && ok || bad "node-host --dry-run missing step: '$step'" +done +# SEC-010 forward-drop must key on the transit interface, overridable via --transit-if +tiout="$(bash "$S" --role rack --host-nodes --transit-if wan0 --dry-run --region-url "$RURL" --enroll-secret-file "$SECFILE" 2>&1)" +printf '%s' "$tiout" | grep -q "wan0" && ok || bad "--transit-if override not reflected in the SEC-010 forward-drop" +# SEC-010 --check must not fail-open: it must verify the keyed transit interface ACTUALLY EXISTS, +# because nftables oifname/iifname with an absent name loads clean but matches NOTHING (advisor catch) +grep -q 'ip link show "$TRANSIT_IF"' "$S" && ok || bad "SEC-010 --check must verify the transit interface exists (nftables fail-open otherwise)" +# it must still enroll the rack AND must not leak the secret +printf '%s' "$nhout" | grep -q 'maas init rack' && ok || bad "node-host mode dropped the rack enrollment" +printf '%s' "$nhout" | grep -q "$SENTINEL" && bad "node-host --dry-run LEAKED the enrollment secret" || ok +# --dry-run must MUTATE NOTHING (the real "-> install ..." line only prints in a live run) +printf '%s' "$nhout" | grep -qE '^\s+->\s+install qemu' && bad "node-host --dry-run appears to have INSTALLED packages" || ok +# plain rack (no --host-nodes) must NOT print node-host steps (backward compat) +printf '%s' "$rackout" | grep -qi 'node-host' && bad "plain rack regressed: printed node-host steps without --host-nodes" || ok +# opnsense base is staged by opnsense-prep-image.sh, not fetched inline by this script +printf '%s' "$nhout" | grep -q 'opnsense-prep-image.sh' && ok || bad "node-host must delegate the opnsense base to opnsense-prep-image.sh" + echo total=$((pass+fail)) if [ "$fail" -eq 0 ]; then echo "site-headend-install: $pass/$total PASS"; exit 0; fi