diff --git a/docs/design-decisions.md b/docs/design-decisions.md index 18db469..1b79cc4 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -3876,3 +3876,49 @@ their `tests/` dirs; on any host where `install` ran, `bash scripts/site-forward.sh remove office1-netbox` (rootless) + optionally `loginctl disable-linger`, and `bash scripts/site-ssh-config.sh remove` (leaves the Include line + `~/.ssh/config`). + +## D-127: VR1 host-level VM autostart policy -- foundational service VMs autostart, DC containment VMs manual + +**Status:** RULED 2026-07-16 (operator: "set the autostart for the VMs so we don't run into problems if +the hosts restart"). Recorded before an OS-update/reboot of the vcloud host; the live `virsh autostart` +settings are applied AFTER a verified clean cold boot (operator sequencing). + +**Problem.** All vcloud host-level libvirt domains were `autostart=disable`, so a host reboot leaves the +Office1 VMs down until manually started -- a footgun as the buildout adds restart events. Need a standing +policy for which VMs auto-boot with the host vs which require a deliberate manual bring-up. + +**Policy (the rule of thumb).** Autostart = YES for FOUNDATIONAL service VMs with no fragile or gated boot. +MANUAL = the DC CONTAINMENT VMs and anything whose bring-up is a deliberate, gated, resource-heavy, or +ordering-sensitive act. + +**Classification (host-level libvirt domains on vcloud):** +- **voffice1 -> AUTOSTART.** Office1 host (MAAS region, LXD host for netbox/tailscale, libvirt). Foundational; + no boot fragility; must return after any host restart. +- **office1-opnsense -> AUTOSTART, but enable only AFTER one verified post-fix cold boot.** The triple-fault + (`incident-20260712-opnsense-edge-boot-triplefault.md`) was a FIXED config bug (DOCFIX-188: `memory_unit` + KiB->MiB starving the bootloader), not live fragility -- the running edge proves the fix; only the cold + boot post-fix is unverified ("live boot verification is the remaining gated step"). Watch it boot once, + then `virsh autostart`. +- **vvr1-dc0 (and future vr1-dc1 containment) -> MANUAL.** The 416 GiB/108-vCPU nested-KVM containment VM: + bring-up is a gated act (depth-4 nested-virt boot gate, D-123 site-down semantics, resource-heavy). Do NOT + auto-boot on host restart. Already default-off (the `cloudinit-vm` module sets `running=true`, no + autostart) -- keep it. + +**Non-host layers (so "Office1 fully returns" holds):** LXD service VMs INSIDE voffice1 (office1-netbox, +office1-tailscale) set autostart WITHIN voffice1 (`lxc config set boot.autostart true`); inner node +VMs inside vvr1-dc0 stay MAAS-power-controlled (inner libvirt autostart off); inner OPNsense edge is +manual/gated like the host edge. + +**Implementation (two layers):** (1) LIVE, immediate, no guest bounce -- `virsh autostart voffice1` (and +`office1-opnsense` after its verified cold boot); runs as jessea123 via the libvirt group (no sudo). (2) +REPRODUCIBLE (IaC) -- add an `autostart` argument to the `cloudinit-vm` + `opnsense-edge` modules (with its +harness), leaving the `vvr1_dc0`/dc-containment instances default-off, so a rebuild re-applies the policy. + +**Sequencing for a host reboot:** `virsh autostart voffice1` (returns on its own) -> reboot -> manually +`virsh start office1-opnsense` and WATCH it cold-boot (the live-boot verification) -> then `virsh autostart +office1-opnsense`. Bring DC containment VMs up manually per their gated procedure, never on host boot. + +**Related:** D-123 (Model B site-down = the reason vvr1-dc0 is manual), DOCFIX-188 (the opnsense mem-unit +fix that makes the edge safe to autostart), `runbooks/ops-restart-procedure.md` (the cloud-side analogue: +`resume-guests-state-on-host-boot`). **Revert:** `virsh autostart --disable `; drop the IaC +`autostart` args; delete this entry. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 1ed3ca4..e005b22 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -139,6 +139,17 @@ `office1-opnsense` all return live from this session -- so the agent now HAS durable shell into Office1 (load-bearing assumption confirmed: key read from config `IdentityFile`, no creds path in command, passes the guard/classifier). The systemd FORWARD layer stays reviewed-but-unexercised (not installed). +**vcloud OS-UPDATE + REBOOT window (2026-07-16) -- IN FLIGHT; D-127 autostart policy recorded.** +Operator patching the vcloud host during the open restart window (only Office1 VMs live, before the DC0 +build). D-127 RULED: voffice1 autostarts; office1-opnsense autostarts AFTER one verified post-fix cold boot +(DOCFIX-188 mem-unit fix); vvr1-dc0/dc1 containment stay MANUAL. **A host reboot ENDS the running agent +session** -- so this is recorded before it. **POST-REBOOT TODO (next session):** (1) `virsh start voffice1` ++ `virsh start office1-opnsense`, WATCH the edge cold-boot clean (live-boot verification); (2) verify `ssh +voffice1` / `ssh office1-netbox` answer (D-126 ProxyJump; the direct L3 route to 10.10.1.10 is already gone +-- use SSH); (3) THEN `virsh autostart voffice1` + `virsh autostart office1-opnsense` (operator sequencing: +autostart only AFTER clean-startup confirmed); (4) optionally set `boot.autostart` on the LXD service VMs +inside voffice1. Then resume DC0 readiness (`docs/dc0-deploy-readiness.md`). + **DC0 DEPLOY READINESS (2026-07-16) -- at the outer-apply gate; see `docs/dc0-deploy-readiness.md`.** Stage-3 Phase-2 DC0 is EXECUTABLE (D-121..125 ADOPTED, R-1/R-2/R-3 ruled, tfvars in, opentofu-validate clean). Outer-root `tofu plan` RUN + VERIFIED SAFE: 5 add (`vvr1-dc0` 416GiB/108vcpu + D-125 uplink) / diff --git a/runbooks/ops-restart-procedure.md b/runbooks/ops-restart-procedure.md index 6688018..025fd46 100644 --- a/runbooks/ops-restart-procedure.md +++ b/runbooks/ops-restart-procedure.md @@ -47,6 +47,11 @@ tenant VM; it WILL auto-resume with the rest. Its manual-start policy (D-041) now governs deliberate stops only. +**VR1 host-VM autostart policy (D-127):** the vcloud host-level libvirt domains follow a separate rule from +the cloud guests above. FOUNDATIONAL service VMs autostart (`voffice1`; `office1-opnsense` after one verified +post-fix cold boot -- DOCFIX-188); DC CONTAINMENT VMs (`vvr1-dc0`, future `vr1-dc1`) stay MANUAL (gated +depth-4/site-down bring-up, never on host boot). Set live with `virsh autostart `; see D-127. + ## Shutdown sequence 1. (Optional) stop guests you want held down post-maintenance.