diff --git a/docs/changelog-20260712-opnsense-edge-boot-fixes.md b/docs/changelog-20260712-opnsense-edge-boot-fixes.md new file mode 100644 index 0000000..2caa6d9 --- /dev/null +++ b/docs/changelog-20260712-opnsense-edge-boot-fixes.md @@ -0,0 +1,36 @@ +# Changelog 2026-07-12 -- DOCFIX-187: opnsense-edge module first-boot fixes (boot still OPEN) + +**Context.** First real boot of the Office1 OPNsense edge. Several correct-but-necessary +module fixes were made while debugging; the boot is NOT yet resolved (a BTX-loader +triple-fault remains -- see `docs/incident-20260712-opnsense-edge-boot-triplefault.md`). +These changes are kept because a working config needs them regardless. + +## `opentofu/modules/opnsense-edge/main.tf` +1. **Serial console added** (`devices.serials`, file-backed to + `/var/lib/libvirt/vr1/staging/${vm_name}-serial.log`). OPNsense nano is serial-only; the + module had no console, so the guest had no output and faulted with nothing to show. This + got the boot from a blank early fault to the loader stage (and gives boot-log capture). +2. **`machine` q35 -> `pc` (i440fx).** FreeBSD's legacy BTX loader is far more commonly + booted on i440fx; q35 is a known early-fault source. (Did not fix this incident, but is + the safer, documented choice.) +3. **Disk: COW overlay -> direct per-VM copy** (`create.content.url = base_volume_path`, + no `backing_store`). Matches the documented `virt-install --import ` flow; + the nano is designed to boot + auto-expand in place, not through a read-only backing. + `var.disk_size_bytes` is now unused (nano is pre-grown by `opnsense-prep-image.sh`). +4. **CPU `host-passthrough` with AMD `svm` disabled** (`cpu.features = [{name=svm, + policy=disable}]`). The documented AMD nested-virt fix (`-cpu host,-svm`). NOTE: the + provider key is `features` (plural) -- `feature` validates but is silently dropped. + +## Important operational note discovered +- `machine` and `cpu` are create-time; the dmacvicar provider plans them as an in-place + "change" that libvirt does NOT apply to an existing domain. To actually change them: + `virsh destroy ; virsh undefine ; tofu apply` (recreate). `tofu apply -replace` + hit "domain already exists" mid-replace this session. + +## Status +- Module `tofu validate` -> Success. NOT booting: OPNsense triple-faults at the BTX loader + regardless of the above (incident report has the full trail + ranked next steps). + +## Revert +- `git checkout opentofu/modules/opnsense-edge/main.tf` to the pre-DOCFIX-187 revision. + (Reverts the serial console + i440fx + direct-disk + svm-disable together.) diff --git a/docs/incident-20260712-opnsense-edge-boot-triplefault.md b/docs/incident-20260712-opnsense-edge-boot-triplefault.md new file mode 100644 index 0000000..057342f --- /dev/null +++ b/docs/incident-20260712-opnsense-edge-boot-triplefault.md @@ -0,0 +1,74 @@ +# Incident 2026-07-12: Office1 OPNsense edge triple-faults at the BTX loader (UNRESOLVED) + +**Status:** OPEN. The Office1 OPNsense edge VM is built and *starts*, but OPNsense +triple-faults ~262 bytes into boot. Blocks the Office1 headend (router+DHCP for +`office1-local`) and therefore the Office1 NetBox VM. Written at a context limit as a +resume artifact -- a fresh session should read this + `docs/session-ledger.md` first. + +## Symptom (confirmed, reproducible) +- Domain `office1-opnsense` (libvirt, `qemu:///system`) enters `running (booted)` then + goes to `paused (unknown)`. `virsh resume` fails with *"cont: Resetting the Virtual + Machine is required"* -> the guest **triple-faulted**. +- Serial capture (`/var/lib/libvirt/vr1/staging/office1-opnsense-serial.log`, `root:600`, + read with `sudo cat`) contains exactly, every boot: + ``` + /boot.config: -S115200 -h -D + ``` + i.e. FreeBSD `boot2` echoes `/boot.config` (serial 115200, `-h` serial console, `-D` + dual console) and then triple-faults **handing off to the BTX `/boot/loader`**. The log + mtime updates on each boot (confirmed fresh, not stale); it is deterministically 262 bytes. + +## Environment +- **Double-nested virt:** OPNsense guest -> `vcloud` host (itself a VM: virtio NIC/disk) + -> outer hypervisor. Host CPU model reported by libvirt: **AMD `Opteron_G3`**. Nested + KVM on (`kvm_amd/parameters/nested = 1`). +- Image: OPNsense **26.1 nano** amd64 (`scripts/opnsense-prep-image.sh 26.1`), prepped to + `/var/lib/libvirt/vr1/office1/opnsense-26.1-nano.qcow2` (11 GiB virtual). +- Module: `opentofu/modules/opnsense-edge`, instantiated as `module "office1_opnsense"` in + `opentofu/main.tf`. LAN=`office1-local` (`vtnet0`), WAN=`office1-wan` (`vtnet1`, NAT + `172.30.1.0/24`). Config ISO (real-ISP-router config, DOCFIX-185) at + `/var/lib/libvirt/vr1/staging/office1-opnsense-config.iso`. + +## What was tried -- ALL applied and verified in the domain XML, NONE resolved it +1. **Serial console added** (module gap -- nano is serial-only). Got the boot *to* the + loader stage (from a no-console early fault to the 262-byte `/boot.config` point). +2. **`machine` q35 -> i440fx** (`pc-i440fx-noble`, verified). No change. (Note: machine + + cpu are create-time; the provider does an in-place "change" that does NOT apply -- + must recreate the domain: `virsh destroy && virsh undefine`, then `tofu apply`.) +3. **Disk: COW overlay -> direct per-VM copy** of the nano (verified 11 GiB / 2.14 GiB + allocated, no backing). No change. +4. **CPU `host-passthrough`** (verified). No change. +5. **Disable AMD `svm`** (`` in XML -- the documented + AMD nested-virt fix, forum: `-cpu host,-svm`). **Still 262 bytes.** NOTE: the provider's + CPU-feature key is `features` (plural); `feature` validates but is silently dropped. + +## Ranked next steps for a fresh session +1. **Full forum CPU flag set**, not just `-svm`: add `+kvm_pv_eoi,+kvm_pv_unhalt` (and try + without `svm` masking too). Likely via `libvirt_domain.qemu_commandline` since the + provider's CPU `features` may not fully translate under `host-passthrough`. Confirm the + masking actually reaches the guest CPUID. +2. **Video device (`-D` dual console).** `/boot.config` requests dual console but the domain + has **no video/graphics device** -- `boot2`'s VGA init may fault. Add a `graphics` + + `video` (VGA/std) device and retry. (Cheap, plausible, not yet tried.) +3. **Memory 2 GB -> 4 GB** (below OPNsense's 3 GB min; guides use 4096). Recreate to apply. +4. **UEFI boot (OVMF)** instead of legacy BIOS/BTX -- sidesteps BTX entirely, but the nano + is a BIOS/MBR image, so this needs care (or a different image build). +5. **Outer-hypervisor CPU:** under double-nesting, vcloud's exposed CPU (`Opteron_G3`) may + not provide what FreeBSD's BTX needs. Consider testing a minimal FreeBSD/OPNsense boot + directly on vcloud to isolate whether it's nesting-depth-specific. + +## Reproduce / operate +- Recreate + boot: `cd opentofu && source ~/vr1-stage1.env && virsh -c qemu:///system + destroy office1-opnsense; virsh -c qemu:///system undefine office1-opnsense; tofu apply`. +- Watch: `virsh domstate --reason office1-opnsense`; serial via `sudo cat` the log above. +- Provider CPU/serial schema is attribute-style + nested under `devices`/`cpu`; introspect + with `tofu providers schema -json` (see how `serials`/`features` were found this session). +- As-executed log: `~/as-executed/2026-07-12-dc-dc-phase1-office1.log`. +- Creds (jumphost-only, 0600, `~/vr1-office1-creds/`): SSH key + OPNsense root pw/hash. + **Operator: revoke the pasted NetBox token + these at close.** + +## Cross-refs +- `docs/changelog-20260712-opnsense-edge-boot-fixes.md` (DOCFIX-187, the module changes). +- `docs/changelog-20260712-office1-opnsense-edge-build.md` (DOCFIX-186, the build + the + apparmor/config-iso-staging findings). +- `docs/changelog-20260712-opnsense-edge-real-isp-router.md` (DOCFIX-185, config posture). diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 6128a30..8a349ca 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -1692,6 +1692,28 @@ then the Office1 NetBox VM. - **Numbers:** D next-free 112, DOCFIX next-free 187 (184/185/186 committed this session). +## Jumphost stream -- session 2026-07-12 (cont.): OPNsense edge BOOT INCIDENT (OPEN) + context-limit checkpoint + +- **Office1 OPNsense edge boots but OPNsense TRIPLE-FAULTS at the BTX loader** (262 bytes: + `/boot.config: -S115200 -h -D` then dies loading /boot/loader). Double-nested virt + (OPNsense -> vcloud VM -> outer hypervisor), AMD Opteron_G3. **INCIDENT REPORT (read + first on resume): `docs/incident-20260712-opnsense-edge-boot-triplefault.md`.** +- **DOCFIX-187 (committed): opnsense-edge module fixes** -- serial console (nano is + serial-only; was the module's biggest gap), q35->i440fx, COW-overlay->direct-copy disk, + host-passthrough CPU + AMD `svm` disabled. All correct + kept; NONE resolved the boot. +- **KEY GOTCHA:** `machine`/`cpu` are create-time; the provider plans them in-place and + libvirt ignores that -- must `virsh destroy; virsh undefine; tofu apply` to recreate. + Provider CPU-feature key is `features` (plural); `feature` silently drops. +- **NEXT (ranked, in the incident report):** full CPU flags (+kvm_pv_eoi,+kvm_pv_unhalt via + qemu_commandline), add a video device (`-D` dual-console + no VGA -> boot2 may fault), + mem 2G->4G, UEFI/OVMF, or outer-hypervisor CPU investigation. +- **Live state:** `office1-opnsense` domain paused/faulted; `office1-wan` NAT net up; + Stage-1 13 objects + edge disk/config-seed volumes exist. Creds `~/vr1-office1-creds/` + (revoke at close, + the pasted NetBox token). apparmor rule for /var/lib/libvirt/vr1/** + applied by operator (needed for any VR1 VM). +- **CONTEXT-LIMIT CHECKPOINT.** Session near max; resume in a fresh session from the incident + report + this ledger. **Numbers:** D next-free 112, DOCFIX next-free 188. + diff --git a/opentofu/modules/opnsense-edge/main.tf b/opentofu/modules/opnsense-edge/main.tf index db213da..c9eeea6 100644 --- a/opentofu/modules/opnsense-edge/main.tf +++ b/opentofu/modules/opnsense-edge/main.tf @@ -42,18 +42,24 @@ # touching for this module at all. resource "libvirt_volume" "disk" { - name = "${var.vm_name}-disk.qcow2" - pool = var.pool_name - capacity = var.disk_size_bytes + name = "${var.vm_name}-disk.qcow2" + pool = var.pool_name - target = { - format = { - type = "qcow2" + # Direct per-VM COPY of the prepared nano image, NOT a backing/overlay. + # OPNsense nano is designed to be booted + auto-expanded IN PLACE; booting it + # through a read-only COW backing (with an overlay capacity larger than the + # base) made boot2 fault while loading /boot/loader (first-boot finding + # 2026-07-12 -- q35->i440fx did not help, the disk shape did). Matches the + # documented `virt-install --import ` workflow. The nano is already + # grown to its target size by scripts/opnsense-prep-image.sh, so var.disk_size_bytes + # is no longer used here. + create = { + content = { + url = var.base_volume_path } } - backing_store = { - path = var.base_volume_path + target = { format = { type = "qcow2" } @@ -78,10 +84,29 @@ type = "kvm" running = true + # host-passthrough, but DISABLE the AMD `svm` (nested-virt) feature in the + # guest. ROOT CAUSE of the first-boot triple-fault (2026-07-12): on an AMD host + # (this one is Opteron_G3), passing `svm` to a FreeBSD/OPNsense guest makes its + # loader triple-fault -- a well-documented KVM bug. Equivalent to the forum's + # `-cpu host,-svm` fix. (qemu64 default also lacks features, so host-passthrough + # is still right; we just drop svm.) + cpu = { + mode = "host-passthrough" + features = [ + { + name = "svm" + policy = "disable" + } + ] + } + os = { - type = "hvm" - type_arch = "x86_64" - type_machine = "q35" + type = "hvm" + type_arch = "x86_64" + # i440fx (pc), NOT q35: FreeBSD/OPNsense's legacy BTX `/boot/loader` triple- + # faults on q35 in qemu -- the boot dies right after boot2 echoes + # /boot.config (confirmed via the serial log, first-boot finding 2026-07-12). + type_machine = "pc" } devices = { @@ -144,5 +169,17 @@ } } ] + + # OPNsense nano images are SERIAL-console only. This file-backed serial gives + # the guest a serial device (its comconsole loader needs one) AND captures the + # boot output to a file, so the first boot can be read without an interactive + # console (first-boot finding 2026-07-12: with no console the domain + # triple-faulted). + serials = [ + { + source = { file = { path = "/var/lib/libvirt/vr1/staging/${var.vm_name}-serial.log" } } + target = { port = "0" } + } + ] } }