diff --git a/docs/changelog-20260712-docfix189-max-context-session-audit.md b/docs/changelog-20260712-docfix189-max-context-session-audit.md new file mode 100644 index 0000000..88272a1 --- /dev/null +++ b/docs/changelog-20260712-docfix189-max-context-session-audit.md @@ -0,0 +1,92 @@ +# Changelog 2026-07-12 -- DOCFIX-189: re-audit of the max-context session's troubleshooting + docs + +**Why.** The 2026-07-12 OPNsense boot session ran to a context limit *while working from a +wrong diagnosis*. DOCFIX-188 fixed the actual bug (a 2 MiB guest -- missing `memory_unit`). +This pass re-audits everything that session **wrote**, on the operator's instruction, because +work authored under a false theory tends to encode that theory as fact. + +**Headline:** the settings it changed are mostly fine and are RETAINED. The *reasoning* it +recorded was not -- it attributed the fault to whatever knob was being turned at the time, +on no evidence, four separate times. It also left one silently-dead config knob. + +## Class 1 -- false causal claims (the real damage) + +Each change was written up as a contributing cause. **None of them changed the symptom**: the +fault stayed at a deterministic 262 bytes through every one. Corrected in place, at the +source, so the next reader cannot inherit the theory: + +| Claim as written | Truth | +|---|---| +| `opnsense-edge/main.tf` -- COW backing "made boot2 fault ... **the disk shape did** [help]" | FALSE. Fault persisted identically. | +| `opnsense-edge/main.tf` -- q35 "triple-faults ... **confirmed via the serial log**" | FALSE. Serial log showed the SAME fault before and after. q35 was never shown to be a problem. | +| `opnsense-edge/main.tf` -- "with no console the domain triple-faulted" | MISLEADING. It faulted with AND without a console; the console *revealed* the fault, it did not change it. | +| `opnsense-edge/main.tf` -- `svm` disable is "**ROOT CAUSE** of the first-boot triple-fault" | FALSE. (Already corrected in DOCFIX-188.) | + +All four settings are **RETAINED** -- each is defensible on its own merits (serial is a +genuine nano requirement; i440fx is the conventional FreeBSD machine type; direct-copy is the +documented `virt-install --import` flow; `svm`-disable is reasonable hardening). Only the +false *justifications* are gone. The serial console genuinely earned its keep: it is what made +the fault legible at all. + +## Class 2 -- a silently dead knob (a real functional defect) + +`disk_size_bytes = 17179869184` (16 GiB) in `main.tf` **did nothing.** When DOCFIX-187 changed +the disk from a sized COW overlay to a direct copy of the prepped nano, nothing consumed the +variable any more -- but an unused variable is legal HCL, so neither `tofu validate` nor the +plan said a word. + +**Measured:** `virsh vol-info` -> the live disk is **11.00 GiB**, not the 16 GiB `main.tf` +declares. Real sizing comes from `scripts/opnsense-prep-image.sh`'s `GROW` (default `+8G`). + +- Removed `disk_size_bytes` from the `office1_opnsense` instantiation **and** from + `modules/opnsense-edge/variables.tf` (must be removed together or tofu errors). +- Left an explicit note in both places saying sizing is prep-image's job, so nobody re-adds a + sizing input the module does not consume. +- **Proof it was dead:** `tofu plan` is byte-identical before and after removal + (`0 to add, 1 to change, 0 to destroy`). Removing it changes no resource. +- Consequence to know: every VM built from the same prepped base image gets the same disk + size. If the edge needs more than 11 GiB, re-run prep-image with a bigger `GROW`. + +## Class 3 -- stale/unsourced facts + +- **`Opteron_G3` is a RED HERRING** (incident report). The host CPU is really an **AMD EPYC + 9965 (Zen 5, family 26)**; libvirt's CPU-model DB does not know family 26 and falls back to + the oldest matching name. The prior session partly built its nested-virt/old-CPU theory on + this artifact. Corrected + annotated: read `/proc/cpuinfo`, not libvirt's model guess. +- **"OPNsense's 3 GB min" is UNVERIFIED** (incident report, ranked step 3). No source given. + Annotated: do not propagate. That step was right-for-the-wrong-reason -- memory *was* the + problem, but it was 2 MiB, not "2 GB but slightly small", and nobody measured it. +- **DOCFIX-186's "the disk uses `backing_store` -- a reference, not a copy"** is now STALE + (DOCFIX-187 made it a copy). Conclusion still holds -- no target-path collision -- but for + a *different* mechanism (source and target volume names differ). Annotated, because the + original reasoning no longer supports the conclusion. +- **DOCFIX-187's changelog** gains a SUPERSEDED-IN-PART header pointing at DOCFIX-188/189. + +## Kept as-is (audited, no change needed) + +- DOCFIX-186's two infra findings are REAL and measured -- the `config_iso_path`-outside-the-pool + requirement, and the **apparmor** rule for `/var/lib/libvirt/vr1/**` (correctly flagged as + foundational; it gates every VR1 VM). Both stand. +- DOCFIX-185's egress-airgap strip was an operator ruling on transport model, not a + troubleshooting artifact. Untouched. + +## Verification + +- `tofu fmt` clean; `tofu validate` **Success**; `tofu plan` unchanged (proves the knob was dead). +- `scripts/opentofu-validate.sh`: S1 PASS + fmt/init/validate PASS. +- `tests/opentofu-validate/run-tests.sh`: **4 PASS / 0 FAIL**. +- `scripts/repo-lint.sh`: **0 fail** (1 documented legacy warn). + +## Standing lesson (the one worth keeping) + +**When a change does not fix the symptom, do not write it up as though it did.** Four +successive non-fixes were each recorded as a contributing cause, and the accumulated fiction +is what a fresh session would have inherited and continued. A fault that is *deterministic and +immune to every knob you turn* is evidence that you have not yet touched the cause -- it is +not evidence that every knob you turned was load-bearing. + +## Revert + +- `git revert `. Comment-only + one dead-variable removal; no live-state dependency. +- To restore the dead knob (do not): re-add `disk_size_bytes` to `modules/opnsense-edge/ + variables.tf` and pass it in `main.tf`. It will still do nothing. diff --git a/docs/changelog-20260712-office1-opnsense-edge-build.md b/docs/changelog-20260712-office1-opnsense-edge-build.md index 7256a7e..8ddab6a 100644 --- a/docs/changelog-20260712-office1-opnsense-edge-build.md +++ b/docs/changelog-20260712-office1-opnsense-edge-build.md @@ -26,6 +26,12 @@ errors "volume target path ... already exists". Fixed by staging the ISO at `/var/lib/libvirt/vr1/staging/` and pointing `config_iso_path` there. (The disk uses `backing_store` -- a reference, not a copy -- so the in-pool nano image is fine.) + **CORRECTION 2026-07-12 (DOCFIX-189):** that parenthetical is now STALE -- DOCFIX-187 + changed the disk to a direct copy (`create.content.url`), so it no longer uses + `backing_store`. The in-pool nano image is still fine, but for a DIFFERENT reason: the + copy's target volume name (`-disk.qcow2`) differs from the source + (`opnsense-26.1-nano.qcow2`), so there is no target-path collision. Same conclusion, + different mechanism -- do not rely on the original reasoning. 2. **apparmor blocks the custom pool path `/var/lib/libvirt/vr1/`.** The domain was defined but failed to start: qemu (`libvirt-qemu`) got "Permission denied" on the backing image despite correct POSIX perms. libvirt's apparmor abstraction only whitelists diff --git a/docs/changelog-20260712-opnsense-edge-boot-fixes.md b/docs/changelog-20260712-opnsense-edge-boot-fixes.md index 2caa6d9..983f062 100644 --- a/docs/changelog-20260712-opnsense-edge-boot-fixes.md +++ b/docs/changelog-20260712-opnsense-edge-boot-fixes.md @@ -1,5 +1,24 @@ # Changelog 2026-07-12 -- DOCFIX-187: opnsense-edge module first-boot fixes (boot still OPEN) +> **SUPERSEDED IN PART -- read `docs/changelog-20260712-libvirt-memory-unit-rootcause.md` +> (DOCFIX-188) and the DOCFIX-189 audit first.** +> +> This changelog was written under a WRONG diagnosis, at a context limit. The boot fault was +> a **2 MiB guest** (missing `memory_unit`), not a CPU/machine/disk/console problem. +> +> The CHANGES below are retained (each is defensible on its own merits), but the +> **CAUSAL CLAIMS attached to items 2, 3 and 4 are FALSE** and were corrected in the module +> source by DOCFIX-189: +> - item 2 (q35 -> i440fx): did NOT fix the fault. q35 was never shown to be a problem. +> - item 3 (COW overlay -> direct copy): did NOT fix the fault. It also silently orphaned +> `disk_size_bytes`, so the edge got 11 GiB while `main.tf` claimed 16 GiB (DOCFIX-189). +> - item 4 (`svm` disable): did NOT fix the fault. +> - item 1 (serial console) is the one that earned its keep -- but it *revealed* the fault, +> it did not change it. +> +> Standing lesson: when a change does not fix the symptom, do not write it up as though it +> did. Every one of these was recorded as a contributing cause on no evidence. + **Context.** First real boot of the Office1 OPNsense edge. Several correct-but-necessary module fixes were made while debugging; the boot is NOT yet resolved (a BTX-loader triple-fault remains -- see `docs/incident-20260712-opnsense-edge-boot-triplefault.md`). diff --git a/docs/incident-20260712-opnsense-edge-boot-triplefault.md b/docs/incident-20260712-opnsense-edge-boot-triplefault.md index f98cacb..48060e6 100644 --- a/docs/incident-20260712-opnsense-edge-boot-triplefault.md +++ b/docs/incident-20260712-opnsense-edge-boot-triplefault.md @@ -51,6 +51,11 @@ - **Double-nested virt:** OPNsense guest -> `vcloud` host (itself a VM: virtio NIC/disk) -> outer hypervisor. Host CPU model reported by libvirt: **AMD `Opteron_G3`**. Nested KVM on (`kvm_amd/parameters/nested = 1`). + - **CORRECTION 2026-07-12 (DOCFIX-189): the `Opteron_G3` label is a RED HERRING.** The + real CPU is an **AMD EPYC 9965 (Zen 5, family 26)** (`/proc/cpuinfo`). libvirt's CPU + model database does not know family 26, so it falls back to the oldest matching model + name. The prior session built a nested-virt/ancient-CPU theory partly on this artifact. + Read `/proc/cpuinfo`, not libvirt's model guess. - Image: OPNsense **26.1 nano** amd64 (`scripts/opnsense-prep-image.sh 26.1`), prepped to `/var/lib/libvirt/vr1/office1/opnsense-26.1-nano.qcow2` (11 GiB virtual). - Module: `opentofu/modules/opnsense-edge`, instantiated as `module "office1_opnsense"` in @@ -80,6 +85,11 @@ has **no video/graphics device** -- `boot2`'s VGA init may fault. Add a `graphics` + `video` (VGA/std) device and retry. (Cheap, plausible, not yet tried.) 3. **Memory 2 GB -> 4 GB** (below OPNsense's 3 GB min; guides use 4096). Recreate to apply. + - **CORRECTION (DOCFIX-189): this step was RIGHT FOR THE WRONG REASON, and its premise is + unsourced.** The memory *was* the problem -- but it was 2 MiB, not "2 GB but a bit + small", and no measurement was taken to notice that. The "3 GB min" figure is + UNVERIFIED (no source given); do not propagate it. If 2 GiB proves insufficient after + the real fix, check OPNsense 26.1's documented minimum before picking a number. 4. **UEFI boot (OVMF)** instead of legacy BIOS/BTX -- sidesteps BTX entirely, but the nano is a BIOS/MBR image, so this needs care (or a different image build). 5. **Outer-hypervisor CPU:** under double-nesting, vcloud's exposed CPU (`Opteron_G3`) may diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 4de33fb..05b8396 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -1761,7 +1761,33 @@ produces **37 spurious `100644 -> 100755` mode-change diffs** (zero content change). Had to `git checkout` them out of this session's commit. Fix is either commit the stubs 0755 or have the harness restore the mode; either way a gauntlet run should not dirty the tree. -- **Numbers:** D next-free 112, DOCFIX next-free 189. +- **RE-AUDIT of the max-context session (DOCFIX-189, operator-instructed).** That session + authored docs+module comments *while working from a wrong diagnosis*, so its reasoning was + re-checked, not just its one-line bug. Findings: + - **4 FALSE CAUSAL CLAIMS corrected at the source** (opnsense-edge/main.tf): the disk shape, + q35->i440fx, the serial console, and svm-disable were EACH written up as a contributing + cause. **None changed the symptom** -- the fault sat at a deterministic 262 bytes through + all of them. All four SETTINGS are RETAINED (each defensible on its own merits); only the + false justifications are gone. The serial console genuinely earned its keep -- it made the + fault legible -- but it revealed the fault, it did not change it. + - **ONE SILENTLY DEAD KNOB (real functional defect):** `disk_size_bytes = 16 GiB` in main.tf + did NOTHING once DOCFIX-187 made the disk a direct copy -- unused variables are legal HCL, + so validate/plan never complained. MEASURED: live disk is **11.00 GiB**, not 16. Removed + from main.tf + module variables; sizing is really `opnsense-prep-image.sh`'s GROW (+8G). + Proof it was dead: `tofu plan` is byte-identical before/after removal. + - **`Opteron_G3` is a RED HERRING:** the host is really an **AMD EPYC 9965 (Zen 5, fam 26)**; + libvirt's CPU-model DB doesn't know fam 26 and falls back to the oldest name. The prior + session partly built its nested-virt theory on this artifact. Read /proc/cpuinfo. + - **"OPNsense 3 GB min" is UNVERIFIED** (no source) -- annotated, do not propagate. + - DOCFIX-186's `backing_store` parenthetical is STALE (disk is a copy now) -- same + conclusion, different mechanism; annotated. DOCFIX-187 changelog gains a + SUPERSEDED-IN-PART header. DOCFIX-186's apparmor + config-iso-staging findings are REAL + and STAND. DOCFIX-185 (operator transport ruling) untouched. + - **STANDING LESSON:** when a change does NOT fix the symptom, do not write it up as though + it did. Four successive non-fixes were each recorded as a cause, and that accumulated + fiction is what a fresh session would have inherited. A fault that is deterministic and + immune to every knob means you have not yet touched the cause. +- **Numbers:** D next-free 112, DOCFIX next-free 190. diff --git a/opentofu/main.tf b/opentofu/main.tf index 39cd111..de84b98 100644 --- a/opentofu/main.tf +++ b/opentofu/main.tf @@ -105,8 +105,11 @@ memory_mib = 2048 vcpu = 2 pool_name = module.office1_storage.pool_name - disk_size_bytes = 17179869184 # 16 GiB (base OPNsense 26.1 nano is 11 GiB virtual) base_volume_path = "/var/lib/libvirt/vr1/office1/opnsense-26.1-nano.qcow2" + # No disk_size_bytes: the disk is a direct copy of the prepped nano, so its size comes + # from scripts/opnsense-prep-image.sh's GROW (default +8G -> 11 GiB). The old + # `disk_size_bytes = 16 GiB` here was SILENTLY IGNORED -- measured: the live volume is + # 11 GiB. Removed in DOCFIX-189. config_iso_path = "/var/lib/libvirt/vr1/staging/office1-opnsense-config.iso" # staging, NOT the pool dir (create.content.url collides with its own target volume there -- first-boot finding 2026-07-12) lan_network_name = module.office1_network.network_name wan_network_name = "office1-wan" diff --git a/opentofu/modules/opnsense-edge/main.tf b/opentofu/modules/opnsense-edge/main.tf index 4ee85fc..6fb67c8 100644 --- a/opentofu/modules/opnsense-edge/main.tf +++ b/opentofu/modules/opnsense-edge/main.tf @@ -45,14 +45,21 @@ name = "${var.vm_name}-disk.qcow2" pool = var.pool_name - # Direct per-VM COPY of the prepared nano image, NOT a backing/overlay. - # OPNsense nano is designed to be booted + auto-expanded IN PLACE; booting it - # through a read-only COW backing (with an overlay capacity larger than the - # base) made boot2 fault while loading /boot/loader (first-boot finding - # 2026-07-12 -- q35->i440fx did not help, the disk shape did). Matches the - # documented `virt-install --import ` workflow. The nano is already - # grown to its target size by scripts/opnsense-prep-image.sh, so var.disk_size_bytes - # is no longer used here. + # Direct per-VM COPY of the prepared nano image, NOT a backing/overlay. Matches the + # documented `virt-install --import ` workflow; OPNsense nano is designed + # to be booted + auto-expanded in place. + # + # CORRECTION 2026-07-12 (DOCFIX-189): the original version of this comment claimed the + # COW backing "made boot2 fault ... the disk shape did [help]". That is FALSE -- the + # fault persisted identically after this change. The real cause was the memory unit + # (see the domain below). This shape is RETAINED because it is the documented import + # workflow, not because it fixed anything. + # + # DISK SIZING IS NOT OWNED HERE. The copy inherits the prepped image's capacity, so the + # size lever is scripts/opnsense-prep-image.sh's GROW arg (default +8G -> an 11 GiB + # nano). There is deliberately no `capacity`/`disk_size_bytes` input: setting one here + # would be silently ignored, which is exactly the trap DOCFIX-189 removed. Consequence + # to know: every VM built from the same prepped base gets the same disk size. create = { content = { url = var.base_volume_path @@ -108,9 +115,12 @@ os = { type = "hvm" type_arch = "x86_64" - # i440fx (pc), NOT q35: FreeBSD/OPNsense's legacy BTX `/boot/loader` triple- - # faults on q35 in qemu -- the boot dies right after boot2 echoes - # /boot.config (confirmed via the serial log, first-boot finding 2026-07-12). + # i440fx (pc) rather than q35. CORRECTION 2026-07-12 (DOCFIX-189): the original + # comment here claimed q35 "triple-faults ... confirmed via the serial log". That is + # FALSE -- the serial log showed the SAME 262-byte fault before AND after this change; + # the real cause was the memory unit (see above). i440fx is RETAINED only because it + # is the conventional, widely-exercised machine type for FreeBSD/OPNsense guests -- + # NOT because it fixed this incident. q35 was never actually shown to be a problem. type_machine = "pc" } @@ -175,11 +185,15 @@ } ] - # OPNsense nano images are SERIAL-console only. This file-backed serial gives - # the guest a serial device (its comconsole loader needs one) AND captures the - # boot output to a file, so the first boot can be read without an interactive - # console (first-boot finding 2026-07-12: with no console the domain - # triple-faulted). + # OPNsense nano images are SERIAL-console only -- the image ships /boot.config with + # `-S115200 -h`, so its loader expects a serial device. This file-backed serial gives + # the guest that device AND captures boot output to a file, so a first boot can be + # read without an interactive console. + # + # This is a GENUINE requirement and the single most useful thing DOCFIX-187 added -- + # it is what made the 2026-07-12 fault legible at all. But note the correction + # (DOCFIX-189): adding it did not *change* the fault, it only revealed it. The domain + # triple-faulted both with and without a console. serials = [ { source = { file = { path = "/var/lib/libvirt/vr1/staging/${var.vm_name}-serial.log" } } diff --git a/opentofu/modules/opnsense-edge/variables.tf b/opentofu/modules/opnsense-edge/variables.tf index 31abce0..ccd545a 100644 --- a/opentofu/modules/opnsense-edge/variables.tf +++ b/opentofu/modules/opnsense-edge/variables.tf @@ -18,10 +18,13 @@ type = string } -variable "disk_size_bytes" { - description = "Boot disk size in bytes (the backing_store overlay's capacity, not the base nano image's own size). No default -- pass explicitly." - type = number -} +# NOTE: there is deliberately NO `disk_size_bytes` here. It was REMOVED 2026-07-12 +# (DOCFIX-189) because it was DEAD: once the disk became a direct copy of the prepped +# nano (DOCFIX-187), nothing consumed it, so `disk_size_bytes = 16 GiB` was silently +# ignored and the edge got the base image's 11 GiB instead. An unused variable is legal +# HCL, so neither `tofu validate` nor the plan flagged the lie. +# Disk size is owned by scripts/opnsense-prep-image.sh's GROW arg. Do not re-add a sizing +# input here unless the module actually consumes it. variable "base_volume_path" { description = <<-EOT