Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260712-docfix189-max-context-session-audit.md

Changelog 2026-07-12 -- DOCFIX-189: re-audit of the max-context session's troubleshooting + docs

Why. The 2026-07-12 OPNsense boot session ran to a context limit while working from a wrong diagnosis. DOCFIX-188 fixed the actual bug (a 2 MiB guest -- missing memory_unit). This pass re-audits everything that session wrote, on the operator's instruction, because work authored under a false theory tends to encode that theory as fact.

Headline: the settings it changed are mostly fine and are RETAINED. The reasoning it recorded was not -- it attributed the fault to whatever knob was being turned at the time, on no evidence, four separate times. It also left one silently-dead config knob.

Class 1 -- false causal claims (the real damage)

Each change was written up as a contributing cause. None of them changed the symptom: the fault stayed at a deterministic 262 bytes through every one. Corrected in place, at the source, so the next reader cannot inherit the theory:

Claim as written Truth
opnsense-edge/main.tf -- COW backing "made boot2 fault ... the disk shape did [help]" FALSE. Fault persisted identically.
opnsense-edge/main.tf -- q35 "triple-faults ... confirmed via the serial log" FALSE. Serial log showed the SAME fault before and after. q35 was never shown to be a problem.
opnsense-edge/main.tf -- "with no console the domain triple-faulted" MISLEADING. It faulted with AND without a console; the console revealed the fault, it did not change it.
opnsense-edge/main.tf -- svm disable is "ROOT CAUSE of the first-boot triple-fault" FALSE. (Already corrected in DOCFIX-188.)

All four settings are RETAINED -- each is defensible on its own merits (serial is a genuine nano requirement; i440fx is the conventional FreeBSD machine type; direct-copy is the documented virt-install --import flow; svm-disable is reasonable hardening). Only the false justifications are gone. The serial console genuinely earned its keep: it is what made the fault legible at all.

Class 2 -- a silently dead knob (a real functional defect)

disk_size_bytes = 17179869184 (16 GiB) in main.tf did nothing. When DOCFIX-187 changed the disk from a sized COW overlay to a direct copy of the prepped nano, nothing consumed the variable any more -- but an unused variable is legal HCL, so neither tofu validate nor the plan said a word.

Measured: virsh vol-info -> the live disk is 11.00 GiB, not the 16 GiB main.tf declares. Real sizing comes from scripts/opnsense-prep-image.sh's GROW (default +8G).

  • Removed disk_size_bytes from the office1_opnsense instantiation and from modules/opnsense-edge/variables.tf (must be removed together or tofu errors).
  • Left an explicit note in both places saying sizing is prep-image's job, so nobody re-adds a sizing input the module does not consume.
  • Proof it was dead: tofu plan is byte-identical before and after removal (0 to add, 1 to change, 0 to destroy). Removing it changes no resource.
  • Consequence to know: every VM built from the same prepped base image gets the same disk size. If the edge needs more than 11 GiB, re-run prep-image with a bigger GROW.

Class 3 -- stale/unsourced facts

  • Opteron_G3 is a RED HERRING (incident report). The host CPU is really an AMD EPYC 9965 (Zen 5, family 26); libvirt's CPU-model DB does not know family 26 and falls back to the oldest matching name. The prior session partly built its nested-virt/old-CPU theory on this artifact. Corrected + annotated: read /proc/cpuinfo, not libvirt's model guess.
  • "OPNsense's 3 GB min" is UNVERIFIED (incident report, ranked step 3). No source given. Annotated: do not propagate. That step was right-for-the-wrong-reason -- memory was the problem, but it was 2 MiB, not "2 GB but slightly small", and nobody measured it.
  • DOCFIX-186's "the disk uses backing_store -- a reference, not a copy" is now STALE (DOCFIX-187 made it a copy). Conclusion still holds -- no target-path collision -- but for a different mechanism (source and target volume names differ). Annotated, because the original reasoning no longer supports the conclusion.
  • DOCFIX-187's changelog gains a SUPERSEDED-IN-PART header pointing at DOCFIX-188/189.

Kept as-is (audited, no change needed)

  • DOCFIX-186's two infra findings are REAL and measured -- the config_iso_path-outside-the-pool requirement, and the apparmor rule for /var/lib/libvirt/vr1/** (correctly flagged as foundational; it gates every VR1 VM). Both stand.
  • DOCFIX-185's egress-airgap strip was an operator ruling on transport model, not a troubleshooting artifact. Untouched.

Verification

  • tofu fmt clean; tofu validate Success; tofu plan unchanged (proves the knob was dead).
  • scripts/opentofu-validate.sh: S1 PASS + fmt/init/validate PASS.
  • tests/opentofu-validate/run-tests.sh: 4 PASS / 0 FAIL.
  • scripts/repo-lint.sh: 0 fail (1 documented legacy warn).

Standing lesson (the one worth keeping)

When a change does not fix the symptom, do not write it up as though it did. Four successive non-fixes were each recorded as a contributing cause, and the accumulated fiction is what a fresh session would have inherited and continued. A fault that is deterministic and immune to every knob you turn is evidence that you have not yet touched the cause -- it is not evidence that every knob you turned was load-bearing.

Revert

  • git revert <sha>. Comment-only + one dead-variable removal; no live-state dependency.
  • To restore the dead knob (do not): re-add disk_size_bytes to modules/opnsense-edge/ variables.tf and pass it in main.tf. It will still do nothing.