# 2026-07-16 -- Stage-3 review sweep, Phase C (D-123 Model B OpenTofu reshape + node-host bootstrap)

## Context

Phase C of the review sweep (`docs/stage3-adversarial-review-20260716.md`), implementing the operator's
D-123 = **Model B** ruling (R-1): the vr1-dc0 node fleet nests INSIDE the containment VM `vvr1-dc0`.
Design was cross-checked by a 4-charter agent workflow (mechanism/compatibility/best-practice/structure);
the two-root implementation was cross-checked by a second agent (verdict: FAITHFUL, no BLOCKING). Inner
transport = **remote qemu+ssh from Office1** (R-5). NO cloud mutation; nothing applied. Remaining:
maas-vm-host retarget, C5 runbook reorg, Phase D SEC-010.

## What changed

**Mechanism (forced, cross-checked): TWO-ROOT / TWO-STATE split.** A libvirt provider cannot be
configured from a resource created in the same apply, so the substrate splits into an OUTER root
(vcloud) that boots vvr1-dc0 and an INNER root that builds the fleet on vvr1-dc0's own libvirt, with a
BOOTSTRAP GATE between. Modules are reused VERBATIM -- the shim stays wholesale-deletable, Roosevelt-collapsing.

**C1 -- OUTER root (`opentofu/main.tf`, vcloud).**
- `vvr1_dc0` resized 4/8192/80 -> **108 vCPU / 416 GiB / ~3000 GiB** via new `var.vvr1_dc0_*` (derived
  defaults, `variables.tf`; hard rule 2); `expose_nested_virt` false -> **true** (load-bearing).
  Collapsed to a SINGLE outer leg (the transit); metal-admin + the other 5 planes become INNER bridges.
- Removed the inner-substrate blocks (planes / wan / edge / 9-node layout) -> MOVED markers. Deleted the
  now-dangling `moved{dc1_planes->vr1_dc0_planes}` block (kept the two storage `moved` blocks).
- `tofu validate` Success; no dangling references.

**C2 -- INNER root (`opentofu/vr1-dc0-substrate/`, NEW: versions/variables/main.tf).** Provider =
`qemu+ssh://<user>@<vvr1-dc0-transit-ip>/system` (R-5, key auth, no `no_verify`). Recreates the inner
storage pool + 6 dc-planes + site-wan + opnsense-edge + 9 node-VMs (3+2+4, R-3), reusing `../modules/*`
verbatim. `init` + `validate` Success. `.terraform/` gitignored; `.terraform.lock.hcl` committed.

**C3 -- Bootstrap node-host mode (`scripts/site-headend-install.sh` + harness).** New `--host-nodes`
(valid only with `--role rack`) makes the DC rack a nested libvirt host so the inner provider has a
target: installs qemu-kvm + libvirt, persists + verifies `kvm_amd/intel nested=1`, adds the ssh user to
the libvirt group, creates the inner pool dir + an AppArmor grant (platform-traps: non-default pool path
otherwise fails qemu with Permission denied), and verifies/instructs the OPNsense base image (delegated
to `opnsense-prep-image.sh`, not fetched inline). `--check`/`--dry-run` supported; secret never leaked;
plain-rack backward-compatible. Harness 32 -> **45/45**.

**Sizing DERIVED + RATIFIED.** `scripts/dc-dc-whole-host-budget.py` (Phase B) derives vvr1-dc0 = one DC
fleet (104 vCPU / 384 GiB / 2850 GiB) + **32 GiB** containment overhead (ratified for Model B: the VM
also carries the inner edge + libvirtd + nested-KVM paging the rack-only sizing lacked). Whole-host 2-DC
= **870 / 1024 GiB (85%), FIT, 154 GiB headroom**. The design-decisions D-121/123/124 amendments were
reconciled from the earlier 16-GiB figures (838/404/186) to the ratified 32-GiB figures (870/416/154).

**Cross-check fixes applied:** false metal-admin-IP comment corrected (it flows to the bootstrap script,
not the inner root); SEC-010 updated for the single-leg-outer / bridges-6-inner-planes reality (hardening
now belongs on the inner libvirt host); OBS-3 logged (nested `vr1-dc0-wan` NAT has no upstream under
Model B -- confirm intended or wire a controlled egress).

## Tests / verification

- `cd opentofu && tofu validate` -> Success (outer); `cd opentofu/vr1-dc0-substrate && tofu init+validate` -> Success (inner).
- `bash tests/site-headend-install/run-tests.sh` -> **45/45 PASS**.
- `bash scripts/repo-lint.sh` -> 0 fail, 1 legacy WARN. Byte hygiene: 0 non-ASCII / 0 CR added (edited regions).
- `python3 scripts/dc-dc-whole-host-budget.py --model B --containment-overhead-mem-gib 32` -> FIT 870/1024, 154 headroom.
- Agent cross-check of the two-root reshape: FAITHFUL, no BLOCKING.

## Revert

- Model A fallback: `git checkout model-a-fallback -- opentofu/` (tag on `114d392`, R-3-compliant) then
  `bash scripts/opentofu-validate.sh`. See `docs/model-a-fallback-plan.md`.
- Node-host mode: `git checkout <pre-C3> -- scripts/site-headend-install.sh tests/site-headend-install/`.
- Inner root: `git rm -r opentofu/vr1-dc0-substrate`. No cloud state touched.
