| 2026-07-12 |

DOCFIX-190: second boot bug -- libvirt domains had ACPI disabled (FreeBSD kernel panic)
...
With DOCFIX-188's memory fix in place the guest finally got its 2 GiB, the kernel
reached interrupt initialisation, and hit a SECOND defect the first bug had masked:
panic: running without device atpic requires a local APIC
apic_init() ... mi_startup() ... db>
None of the three VM modules emitted a `features` block, so libvirt rendered
`-machine pc-i440fx-noble,...,acpi=off`. FreeBSD gets its local APIC from ACPI's
MADT table and the OPNsense kernel has no atpic fallback, so with ACPI off it has
no usable interrupt controller. The guest parks in the ddb debugger -- which is
the 100%-of-one-core spin, while virsh still reports the domain as "running".
Fix: features = { acpi = true, apic = {} } in opnsense-edge, cloudinit-vm and
node-vm. node-vm matters most long term: MAAS drives power via ACPI signalling,
so without it graceful shutdown / MAAS power-off would never have worked.
Guard S2 added. Like memory_unit, `tofu validate` cannot catch this -- the whole
features block is optional, so its absence is schema-valid. Flag renamed
--static-only (--check-memory-unit kept as a back-compat alias). Harness 4 -> 6
PASS; fixtures/s2-bad PASSES S1 and still FAILS S2, proving the guards catch
independent classes rather than one masking the other.
LIVE RESULT (measured): acpi=on; kernel boots to userland; CPU idle, no spin;
DHCP lease 172.30.1.126 hostname "OPNsense" on office1-wan. Both boot bugs closed.
STILL OPEN: the Configuration Importer did not apply -- OPNsense came up on
factory defaults. The ISO is verified well-formed (ISO9660 OPNSENSE_CFG containing
CONF/CONFIG.XML with our 10.10.0.1). This is the mechanism the module header has
always flagged UNVERIFIED. Do not guess the fix; read the Importer's console
output. The unanswered WAN ping is NOT a fault -- OPNsense blocks inbound on WAN.
Gauntlet-relevant suites green; repo-lint 0 fail; tofu fmt/validate clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|

DOCFIX-188: root-cause the OPNsense boot triple-fault -- libvirt memory_unit (guest had 2 MiB)
...
The 2026-07-12 triple-fault was never a CPU/nesting/machine/console problem.
dmacvicar/libvirt >=0.9 reinterprets `memory` as raw libvirt units (KiB), not
MiB (the 0.8-era meaning the modules were authored against). With no
`memory_unit`, `memory = 2048` rendered `<memory unit='KiB'>2048</memory>` ->
QEMU `-m size=2048k` = 2 MiB. boot2 fits in 2 MiB and echoes /boot.config;
/boot/loader does not -- hence the deterministic 262-byte fault, immune to every
knob the prior window turned.
Measured every hop: module input -> tofu state -> domain XML -> virsh dominfo
(Max memory: 2048 KiB) -> live QEMU cmdline.
FOUNDATIONAL: the identical defect was latent in all three VM modules --
opnsense-edge (live blocker), cloudinit-vm (MAAS/NetBox/GitBucket, the next
step), node-vm (Stage 3). Every VR1 VM would have gotten 1024x too little RAM.
- memory_unit = "MiB" on all three libvirt_domain resources (memory_mib = 2048
value untouched -- it now means the 2 GiB always intended).
- Corrected the disproven "svm is ROOT CAUSE" comment in opnsense-edge; the
setting is retained as legitimate hardening, the false claim is not.
- Guard: opentofu-validate.sh S1 + --check-memory-unit flag. `tofu validate`
cannot catch this (memory_unit is optional => omitting it is schema-valid,
which is exactly how it shipped). Harness T3 is a true negative test: the
fixture IS the shipped defect. Harness 1 -> 4 PASS.
- Incident report gains a READ-THIS-FIRST box so nobody re-chases the CPU theories.
Gauntlet ALL GREEN (52 harnesses); repo-lint 0 fail; tofu fmt/validate clean.
Live boot verification remains a separately-gated operator step.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
| 2026-07-10 |

DC-DC Stage 1 (dc-dc-phase0): first real execution + scaffold/tooling fixes
...
First end-to-end run of a DC-DC runbook against real infrastructure (vcloud host,
OpenTofu v1.12.3). Applied 13 libvirt objects (6 DC1 planes, dc1/office1/dc2
pools, office1-local, 3 mesh legs, all MTU 9000); prior VR0 nets torn down (wan
kept as a gap-#17 model); DC2 planes deferred (NetBox supernet), DC2 storage wired.
DOCFIX-178: scripts/prereqs/ idempotent workstation prereq installers + a
Prerequisites section in the runbook + tests/prereqs/ (25/25). Runbooks must
not assume prereq runtimes.
DOCFIX-179: OpenTofu scaffold first validation + apply. Per-module
required_providers (child modules do NOT inherit provider SOURCE -> OpenTofu
inferred nonexistent hashicorp/libvirt); tofu fmt -recursive; provider "maas"
deferred to Stage 3 (kept a sensitive key out of Stage-1 plan/state);
repo-lint now skips the .terraform/ provider cache; .terraform.lock.hcl added.
DOCFIX-180: dc-dc-phase0 runbook as-executed corrections folded in at phase
close (two MTU domains, non-pristine-host handling, Step-9 13-resource plan,
-input=false, Known-Gap resolved).
repo-lint 0 fail; prereqs harness 25/25; repo-lint harness 34/34.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|

DOCFIX-148: OpenTofu module audit pass + provider-docs skill reference
...
Audited every "UNVERIFIED"/"inferred"/"assumed" marker across all 9 modules
built this session rather than leaving them as-is. Found and fixed a real
documentation error: opnsense-edge's comments claimed devices.interfaces
list order directly sets OPNsense's LAN/WAN role. Researching OPNsense's own
interface-assignment model showed this conflates two things -- list order
plausibly controls which vtnetN device number a NIC gets (libvirt's own
docs: auto-assigned PCI addresses "usually match" XML order for a simple
topology), but the actual LAN/WAN role is a separate, explicit mapping set
inside config.xml itself ("vtnet0=WAN" is a convention some guides choose,
not an enforced rule). Corrected main.tf's comments and both
lan_network_name/wan_network_name variable descriptions -- no config.xml
exists in this repo yet, so this fixes documentation accuracy for whoever
writes it next, not current behavior.
Confidence upgraded (not just re-hedged) on three other items: node-vm's
boot={order=N} shape (confirmed the provider 1:1-mirrors libvirt's own XML,
and libvirt's own docs confirm the single-attribute shape), create.content.url
accepting local paths (second independent source), and the genisoimage
flags (confirmed standard, cleanly separated from the still-open question of
whether OPNsense's Importer actually reads the result). Confirmed safe:
maas_vm_host's zone/pool computed-if-unset behavior.
Added .claude/skills/openstack-cloud-ops/references/opentofu-provider-docs.md:
indexes every provider/doc source used this session with exact URLs and
confirmed versions, plus the fetch methodology that actually worked
(Registry doc pages are JS-rendered, use GitHub instead; branch names vary,
check default_branch via the API; real example .tf files beat doc-summarized
prose for syntax questions). Wired into SKILL.md's routing table,
cross-referenced from opentofu/README.md in both directions without
duplicating content.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|

DOCFIX-144: opentofu/modules/node-vm + fix real syntax bug in DOCFIX-142
...
Built the D-103 node-VM pattern (blank disk, PXE-boot, MAAS-managed) for
Stage 3 -- this time from the provider's own real example .tf files
(examples/domain_with_network.tf, examples/alpine_cloudinit.tf), not
doc-summarization alone, since that confidence gap is exactly why the domain
module was deferred last time.
That surfaced a real bug in the already-pushed dc-planes/mesh-link/
dc-storage-pool modules (14d7382): they used classic HCL block syntax
(domain { ... }) where this provider's current schema needs attribute-style
objects (domain = { ... }) -- confirmed once real examples showed the actual
convention. All three fixed; opentofu/README.md's schema notes now state this
as a provider-wide assumption for any field not personally confirmed
otherwise.
One inference flagged, not presented as fact: the per-device boot-order
attribute's exact internal shape ({ order = N }) -- the field's existence is
confirmed, its shape is not. Not instantiated in root main.tf: node
count/sizing is a pending Phase-0 decision, not invented here. Still
deferred: the cloud-init/base-image VM pattern (Office1 services + OPNsense),
DC2 planes, netem application.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|