diff --git a/docs/changelog-20260717-d127-iac-autostart-and-guest-updates.md b/docs/changelog-20260717-d127-iac-autostart-and-guest-updates.md new file mode 100644 index 0000000..6b0265e --- /dev/null +++ b/docs/changelog-20260717-d127-iac-autostart-and-guest-updates.md @@ -0,0 +1,60 @@ +# 2026-07-17 -- D-127 IaC autostart matrix (baked into the deploy tooling) + Office1 guest updates + +## Guest OS updates (agent-run via passwordless sudo INSIDE the guests) + +The vcloud HOST + office1-opnsense stay in the operator's bookend restart; the already-deployed guests I +have sudo inside were updated now: +- **office1-netbox** (LXD): `apt upgrade` (9 pkgs, userland, no kernel) -- no reboot; netbox-docker + stack unaffected + still healthy (HTTP 302). +- **office1-tailscale** (LXD): `apt full-upgrade` (kernel 6.8.0-134 -> 6.8.0-136) + reboot; came back on + the new kernel, tailscale online (`100.64.0.53`) re-advertising `10.10.0.0/22`. Not the ProxyJump path, + so safe. +- **voffice1**: 0 upgradable -- nothing to do (so its load-bearing reboot risk was moot this round). +- **LXD `boot.autostart=true`** set on both office1-netbox + office1-tailscale (D-127 non-host layer). + +## D-127 IaC autostart matrix (operator: bake it into the tooling so DC0/DC1 aren't missed at install) + +Advisor-gated. Added an `autostart` bool var (default false) to `modules/cloudinit-vm`, +`modules/opnsense-edge`, `modules/node-vm`, wired to `libvirt_domain.autostart`; set EXPLICITLY per +instance (shared modules => IaC is the source of truth, else it fights the live layer): + +| instance | root | autostart | why | +|---|---|---|---| +| voffice1 | outer | **true** | foundational Office1 host | +| office1-opnsense | outer | **true** | foundational edge (post-DOCFIX-188) | +| vvr1-dc0 | outer | **false** | DC containment VM -- MANUAL, gated bring-up | +| vr1-dc0-opnsense | inner | **true** | DC edge comes up with its container | +| node-vm (x9) | inner | **false** | MAAS-power-controlled | + +**HARD GATE verified:** `tofu plan` shows the two EXISTING Office1 domains (voffice1, office1-opnsense) as +`autostart false->true` **update-in-place** -- NOT replacement (replacement would destroy the live +host/edge). vvr1-dc0 is a new create with autostart=false baked in. Outer plan is now +`5 add / 2 in-place / 6 destroy`; the 2 in-place collapse to 0 once the bookend sets those two autostart +live before the fresh-session apply. + +**Drift guard:** `tests/opentofu-validate/run-tests.sh` T11-T15 pin each matrix value (flipping +vvr1-dc0/nodes to true -- auto-booting the 416 GiB nested container / MAAS nodes on host boot -- is the +exact defect guarded). opentofu-validate clean; harness ALL PASS. + +**Standup runbook** updated: the composed service VMs' `boot.autostart` is now a recorded install step +(not a manual afterthought). **DC0/DC1 create NO LXD instances** (inner root = node-vm + opnsense-edge + +planes/pool/wan-bridge), so the DC side is purely this VM matrix -- no DC-LXD gap. + +## Records + +D-127 implementation updated (IaC layer DONE); readiness-doc plan line updated (5/2/6, collapses to 0); +D-127 as-built already notes the netbox restart policy. + +## Verification + +- `bash tests/opentofu-validate/run-tests.sh` -> ALL PASS (incl. T11-T15 matrix pins). +- `bash scripts/opentofu-validate.sh` -> clean (both roots + 12 modules). +- `tofu -chdir=opentofu plan` -> `5 add / 2 in-place / 6 destroy`; no forces-replacement on any existing. +- `bash scripts/repo-lint.sh` -> 0 fail (1 legacy WARN). + +## Revert + +- `git checkout opentofu/modules/{cloudinit-vm,opnsense-edge,node-vm}/{main,variables}.tf opentofu/main.tf + opentofu/vr1-dc0-substrate/main.tf tests/opentofu-validate/run-tests.sh runbooks/dc-dc-phase1-office1-standup.md`; + drop the D-127 IaC-layer text. Live: `sudo lxc config unset boot.autostart` on voffice1. The guest + apt upgrades are not repo state (OS state). diff --git a/docs/dc0-deploy-readiness.md b/docs/dc0-deploy-readiness.md index ff6d233..ac91ea0 100644 --- a/docs/dc0-deploy-readiness.md +++ b/docs/dc0-deploy-readiness.md @@ -34,7 +34,10 @@ ## B. The outer-root plan -- VERIFIED SAFE (this is the apply gate) -`tofu -chdir=opentofu plan` => **5 add / 0 change / 6 destroy**, decomposed and verified: +`tofu -chdir=opentofu plan` => **5 add / 2 in-place / 6 destroy**, decomposed and verified: +- **2 in-place (D-127 autostart, 2026-07-17):** voffice1 + office1-opnsense get `autostart false->true` + -- an **update-in-place** (verified: NOT replacement; replacement would destroy the live host/edge). + These COLLAPSE to 0-change once the bookend sets them autostart=true live before the fresh-session apply. - **5 create:** `vvr1_dc0` containment VM (**425984 MiB = 416 GiB / 108 vCPU** -- full-fleet D-121/R-3 sizing, expose_nested_virt) + disk/seed/cloudinit; `vr1_dc0_uplink` (D-125 simulated-ISP NAT). - **6 destroy:** the six `vr1-dc0` plane networks -- INTENDED Model B relocation from vcloud (outer) @@ -42,7 +45,7 @@ EMPTY (0 leases, 0 attached domains). No live/consumed infra touched. - **RAM gate:** host 980 GiB available >= 416 GiB needed (only voffice1 32G + opnsense 2G committed). -> A fresh session MUST re-run the plan and re-confirm 5-add/0-change/6-destroy-empty before apply +> A fresh session MUST re-run the plan and re-confirm 5-add/(0-or-2)-change/6-destroy-empty (autostart in-place collapses to 0 post-bookend) before apply > (live state can drift; verify, don't assume). --- @@ -108,7 +111,7 @@ 3. Read THIS doc + `runbooks/dc-dc-phase2-tofu-dc-substrate.md` (Model B reshape section, ~line 144). 4. **Start logging (cloud-mutating session):** `bash scripts/run-logged.sh dc0-deploy`. 5. Settle **C1.1** (the per-env-key ruling) first. -6. Re-run the outer plan; confirm 5-add/0-change/6-destroy-empty; then the **gated outer apply**. +6. Re-run the outer plan; confirm 5-add/(0-or-2)-change/6-destroy-empty (autostart in-place collapses to 0 post-bookend); then the **gated outer apply**. 7. Proceed A -> B -> C -> D -> E, each mutation individually approved; review after each step. ## E. Gated deploy sequence (Model B, from the runbook) diff --git a/docs/design-decisions.md b/docs/design-decisions.md index 1b79cc4..743381a 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -3911,8 +3911,20 @@ **Implementation (two layers):** (1) LIVE, immediate, no guest bounce -- `virsh autostart voffice1` (and `office1-opnsense` after its verified cold boot); runs as jessea123 via the libvirt group (no sudo). (2) -REPRODUCIBLE (IaC) -- add an `autostart` argument to the `cloudinit-vm` + `opnsense-edge` modules (with its -harness), leaving the `vvr1_dc0`/dc-containment instances default-off, so a rebuild re-applies the policy. +REPRODUCIBLE (IaC) -- **DONE 2026-07-17** (operator: bake the matrix into the deploy tooling so DC0/DC1 are +configured at installation, not missed). Added an `autostart` bool var (default false) to `modules/cloudinit-vm`, +`modules/opnsense-edge`, `modules/node-vm` (wired to `libvirt_domain.autostart`); set EXPLICITLY per instance +in the roots -- voffice1=true, office1-opnsense=true, vvr1-dc0=FALSE (outer); vr1-dc0-opnsense=true, +node-vm=FALSE (inner). Shared modules mean autostart is managed for EVERY instance, so IaC is now the source +of truth (omitting it would fight the live layer). `tests/opentofu-validate/run-tests.sh` T11-T15 pin the +matrix against drift (auto-booting the 416 GiB container / MAAS nodes is the exact defect guarded). VERIFIED +by `tofu plan`: the two EXISTING Office1 domains (voffice1, office1-opnsense) show `autostart false->true` +**update-in-place** (NOT replacement -- confirmed; replacement would destroy the live host/edge); vvr1-dc0 +is a new create with autostart=false baked in. Outer plan is now `5 add / 2 in-place / 6 destroy`, which +collapses back to `0 change` on those two once the bookend has set them autostart=true live before the +fresh-session apply. LXD half: `boot.autostart=true` set live on office1-netbox/tailscale + captured in the +Office1 standup runbook. DC0/DC1 create NO LXD instances (inner root = node-vm + opnsense-edge + planes/pool/ +wan-bridge), so the DC side is purely this VM matrix -- no DC-LXD gap. **Sequencing for a host reboot:** `virsh autostart voffice1` (returns on its own) -> reboot -> manually `virsh start office1-opnsense` and WATCH it cold-boot (the live-boot verification) -> then `virsh autostart diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 705412c..6091ae3 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -150,6 +150,16 @@ D-124/edge-WAN IPAM is now in the apex. See `docs/changelog-20260717-netbox-apex-recovery-edge-wan-import.md`. +**GUEST UPDATES + D-127 IaC AUTOSTART done (2026-07-17).** Office1 guests patched via agent sudo INSIDE +them: office1-netbox apt upgrade (no reboot, netbox healthy); office1-tailscale full-upgrade -> kernel +6.8.0-136 + reboot (back, re-advertising 10.10.0.0/22); voffice1 0-upgradable. LXD boot.autostart=true set +on both. **D-127 IaC layer DONE** (operator: bake the matrix so DC0/DC1 not missed): `autostart` var added +to cloudinit-vm/opnsense-edge/node-vm, set per matrix (voffice1/office1-opnsense/DC-edge=true; +vvr1-dc0/nodes=FALSE); opentofu T11-T15 pin it; HARD GATE verified (voffice1/office1-opnsense = autostart +in-place, NOT replacement). Outer plan now 5 add / 2 in-place / 6 destroy (collapses to 0-change once the +bookend sets those two autostart live pre-apply). Standup runbook records the LXD boot.autostart step. See +`docs/changelog-20260717-d127-iac-autostart-and-guest-updates.md`. STILL bookend: vcloud + opnsense patch/reboot. + **vcloud OS-UPDATE + REBOOT window (2026-07-16) -- IN FLIGHT; D-127 autostart policy recorded.** Operator patching the vcloud host during the open restart window (only Office1 VMs live, before the DC0 build). D-127 RULED: voffice1 autostarts; office1-opnsense autostarts AFTER one verified post-fix cold boot diff --git a/opentofu/main.tf b/opentofu/main.tf index dab5976..2dc56d6 100644 --- a/opentofu/main.tf +++ b/opentofu/main.tf @@ -108,6 +108,7 @@ module "office1_opnsense" { source = "./modules/opnsense-edge" vm_name = "office1-opnsense" + autostart = true # D-127: foundational edge (safe to autostart post-DOCFIX-188; verify 1 cold boot) memory_mib = 2048 vcpu = 2 pool_name = module.office1_storage.pool_name @@ -179,6 +180,7 @@ module "voffice1" { source = "./modules/cloudinit-vm" vm_name = "voffice1" + autostart = true # D-127: foundational Office1 host (MAAS region + LXD service host) vcpu = var.voffice1_vcpu memory_mib = var.voffice1_memory_mib disk_size_bytes = var.voffice1_disk_bytes @@ -358,6 +360,7 @@ module "vvr1_dc0" { source = "./modules/cloudinit-vm" vm_name = "vvr1-dc0" + autostart = false # D-127: DC containment VM -- MANUAL (gated depth-4/site-down bring-up), never on host boot vcpu = var.vvr1_dc0_vcpu # D-123 Model B / D-124 AMENDMENT: hosts the DC's node fleet memory_mib = var.vvr1_dc0_memory_mib # derived 416 GiB (384 node + 32 overhead) disk_size_bytes = var.vvr1_dc0_disk_bytes # derived ~3000 GiB (holds the inner libvirt pool) diff --git a/opentofu/modules/cloudinit-vm/main.tf b/opentofu/modules/cloudinit-vm/main.tf index 8f9cc8d..43fed3a 100644 --- a/opentofu/modules/cloudinit-vm/main.tf +++ b/opentofu/modules/cloudinit-vm/main.tf @@ -76,6 +76,7 @@ vcpu = var.vcpu type = "kvm" running = true + autostart = var.autostart # D-127 boot matrix -- set per instance in the root (never defaulted-by-accident) # ACPI/APIC: with no `features` block libvirt renders `acpi=off` -- see the # opnsense-edge module for the full write-up (a FreeBSD guest PANICS outright). diff --git a/opentofu/modules/cloudinit-vm/variables.tf b/opentofu/modules/cloudinit-vm/variables.tf index 9fcd1c1..dfcb498 100644 --- a/opentofu/modules/cloudinit-vm/variables.tf +++ b/opentofu/modules/cloudinit-vm/variables.tf @@ -86,3 +86,9 @@ EOT type = bool } + +variable "autostart" { + description = "Start this domain automatically when the libvirt host boots (D-127 boot matrix). MANUAL (false) for DC containment + node VMs; true for foundational service VMs + edges. Set explicitly per instance in the root -- do not rely on the default." + type = bool + default = false +} diff --git a/opentofu/modules/node-vm/main.tf b/opentofu/modules/node-vm/main.tf index a915675..5cb59bc 100644 --- a/opentofu/modules/node-vm/main.tf +++ b/opentofu/modules/node-vm/main.tf @@ -54,6 +54,7 @@ vcpu = var.vcpu type = "kvm" running = true + autostart = var.autostart # D-127 boot matrix -- node VMs are MAAS-power-controlled: MANUAL (false) # ACPI/APIC: with no `features` block libvirt renders `acpi=off` -- see the # opnsense-edge module for the full write-up. Especially important here: these are diff --git a/opentofu/modules/node-vm/variables.tf b/opentofu/modules/node-vm/variables.tf index 1e71a17..5451fe8 100644 --- a/opentofu/modules/node-vm/variables.tf +++ b/opentofu/modules/node-vm/variables.tf @@ -32,3 +32,9 @@ EOT type = list(string) } + +variable "autostart" { + description = "Start this domain automatically when the libvirt host boots (D-127 boot matrix). MANUAL (false) for DC containment + node VMs; true for foundational service VMs + edges. Set explicitly per instance in the root -- do not rely on the default." + type = bool + default = false +} diff --git a/opentofu/modules/opnsense-edge/main.tf b/opentofu/modules/opnsense-edge/main.tf index 6504b98..54828d5 100644 --- a/opentofu/modules/opnsense-edge/main.tf +++ b/opentofu/modules/opnsense-edge/main.tf @@ -97,6 +97,7 @@ vcpu = var.vcpu type = "kvm" running = true + autostart = var.autostart # D-127 boot matrix -- set per instance in the root # ACPI IS REQUIRED. With no `features` block, libvirt renders the machine with # `acpi=off`, and FreeBSD/OPNsense then panics on the FIRST interrupt setup: diff --git a/opentofu/modules/opnsense-edge/variables.tf b/opentofu/modules/opnsense-edge/variables.tf index 220fb75..8a91cba 100644 --- a/opentofu/modules/opnsense-edge/variables.tf +++ b/opentofu/modules/opnsense-edge/variables.tf @@ -69,3 +69,9 @@ EOT type = string } + +variable "autostart" { + description = "Start this domain automatically when the libvirt host boots (D-127 boot matrix). MANUAL (false) for DC containment + node VMs; true for foundational service VMs + edges. Set explicitly per instance in the root -- do not rely on the default." + type = bool + default = false +} diff --git a/opentofu/vr1-dc0-substrate/main.tf b/opentofu/vr1-dc0-substrate/main.tf index 8b4386c..f0cf6f9 100644 --- a/opentofu/vr1-dc0-substrate/main.tf +++ b/opentofu/vr1-dc0-substrate/main.tf @@ -54,6 +54,7 @@ module "vr1_dc0_opnsense" { source = "../modules/opnsense-edge" vm_name = "vr1-dc0-opnsense" + autostart = true # D-127: the DC edge comes up with its containment VM (routing for the site) memory_mib = 2048 vcpu = 2 pool_name = module.inner_storage.pool_name @@ -92,6 +93,7 @@ source = "../modules/node-vm" vm_name = each.key + autostart = false # D-127: node VMs are MAAS-power-controlled -- MANUAL, never on host boot vcpu = each.value.vcpu memory_mib = each.value.mem disk_size_bytes = each.value.disk_gib * 1024 * 1024 * 1024 diff --git a/runbooks/dc-dc-phase1-office1-standup.md b/runbooks/dc-dc-phase1-office1-standup.md index 2cd2f9a..b9ed878 100644 --- a/runbooks/dc-dc-phase1-office1-standup.md +++ b/runbooks/dc-dc-phase1-office1-standup.md @@ -221,6 +221,16 @@ -> EXIT GATE -> Stage 3 (DC1), which D-114 GATES behind this ``` +**D-127 boot matrix -- set LXD autostart on the composed service VMs (do NOT skip; a rebuild misses it +otherwise).** After composing NetBox + Tailscale (Steps 8/9/12), set them to auto-start when voffice1 boots +(they are FOUNDATIONAL service VMs -> autostart per D-127). On voffice1: +``` +sudo lxc config set office1-netbox boot.autostart true +sudo lxc config set office1-tailscale boot.autostart true +``` +(Set live 2026-07-17; recorded here so the installation sets it, not a manual afterthought. Their internal +services already survive a reboot: NetBox's netbox-docker stack carries `restart: unless-stopped`.) + --- ## Step 1 -- Confirm Stage 1 exit gate (READ-ONLY) diff --git a/tests/opentofu-validate/run-tests.sh b/tests/opentofu-validate/run-tests.sh index 222be17..fdd657c 100644 --- a/tests/opentofu-validate/run-tests.sh +++ b/tests/opentofu-validate/run-tests.sh @@ -87,5 +87,19 @@ echo " SKIP T8-T10 S3 module-standalone cases (no tofu binary on PATH)" fi +# --- D-127 autostart boot matrix -- pin per-instance values against drift --- +# Foundational service VMs + edges autostart; DC containment + node VMs MANUAL. +# Flipping vvr1-dc0/nodes to true (auto-booting the 416 GiB nested container / +# MAAS-controlled nodes on host boot) is exactly the defect this guards. +pin() { # pin