| 2026-07-12 |
DOCFIX-190: second boot bug -- libvirt domains had ACPI disabled (FreeBSD kernel panic)
...
With DOCFIX-188's memory fix in place the guest finally got its 2 GiB, the kernel
reached interrupt initialisation, and hit a SECOND defect the first bug had masked:
panic: running without device atpic requires a local APIC
apic_init() ... mi_startup() ... db>
None of the three VM modules emitted a `features` block, so libvirt rendered
`-machine pc-i440fx-noble,...,acpi=off`. FreeBSD gets its local APIC from ACPI's
MADT table and the OPNsense kernel has no atpic fallback, so with ACPI off it has
no usable interrupt controller. The guest parks in the ddb debugger -- which is
the 100%-of-one-core spin, while virsh still reports the domain as "running".
Fix: features = { acpi = true, apic = {} } in opnsense-edge, cloudinit-vm and
node-vm. node-vm matters most long term: MAAS drives power via ACPI signalling,
so without it graceful shutdown / MAAS power-off would never have worked.
Guard S2 added. Like memory_unit, `tofu validate` cannot catch this -- the whole
features block is optional, so its absence is schema-valid. Flag renamed
--static-only (--check-memory-unit kept as a back-compat alias). Harness 4 -> 6
PASS; fixtures/s2-bad PASSES S1 and still FAILS S2, proving the guards catch
independent classes rather than one masking the other.
LIVE RESULT (measured): acpi=on; kernel boots to userland; CPU idle, no spin;
DHCP lease 172.30.1.126 hostname "OPNsense" on office1-wan. Both boot bugs closed.
STILL OPEN: the Configuration Importer did not apply -- OPNsense came up on
factory defaults. The ISO is verified well-formed (ISO9660 OPNSENSE_CFG containing
CONF/CONFIG.XML with our 10.10.0.1). This is the mechanism the module header has
always flagged UNVERIFIED. Do not guess the fix; read the Importer's console
output. The unanswered WAN ping is NOT a fault -- OPNsense blocks inbound on WAN.
Gauntlet-relevant suites green; repo-lint 0 fail; tofu fmt/validate clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-189: re-audit the max-context session's troubleshooting + docs
...
Operator-instructed re-evaluation. The 2026-07-12 session authored its docs and
module comments WHILE WORKING FROM A WRONG DIAGNOSIS, at a context limit, so its
reasoning was re-checked rather than just its one-line bug (DOCFIX-188).
Headline: the settings it changed are mostly fine and are RETAINED. The reasoning
it recorded was not -- it attributed the fault to whatever knob was being turned,
on no evidence, four separate times. It also left one silently-dead knob.
FALSE CAUSAL CLAIMS (corrected at the source, opnsense-edge/main.tf). The disk
shape, q35->i440fx, the serial console and svm-disable were EACH written up as a
contributing cause. None changed the symptom -- the fault sat at a deterministic
262 bytes through all of them. All four settings RETAINED (each defensible on its
own merits); only the false justifications are gone. The serial console earned its
keep, but it REVEALED the fault, it did not change it.
DEAD KNOB (real functional defect). `disk_size_bytes = 16 GiB` did nothing once
DOCFIX-187 made the disk a direct copy; unused variables are legal HCL so neither
validate nor plan complained. MEASURED: the live disk is 11.00 GiB, not 16.
Removed from main.tf + module variables; real sizing is opnsense-prep-image.sh's
GROW (+8G). Proof it was dead: tofu plan is byte-identical before/after removal.
STALE/UNSOURCED FACTS. "Opteron_G3" is a red herring -- the host is really an AMD
EPYC 9965 (Zen 5, family 26); libvirt's CPU-model DB doesn't know family 26 and
falls back to the oldest name, and the prior session partly built its nested-virt
theory on that artifact. "OPNsense's 3 GB min" is unsourced -- annotated, do not
propagate. DOCFIX-186's backing_store parenthetical is stale (same conclusion,
different mechanism). DOCFIX-187's changelog gains a SUPERSEDED-IN-PART header.
KEPT: DOCFIX-186's apparmor + config-iso-staging findings are real and measured.
DOCFIX-185 (operator transport ruling) untouched.
STANDING LESSON: when a change does not fix the symptom, do not write it up as
though it did. A fault that is deterministic and immune to every knob you turn
means you have not yet touched the cause.
Gauntlet ALL GREEN (52); repo-lint 0 fail; tofu fmt/validate/plan clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-188: root-cause the OPNsense boot triple-fault -- libvirt memory_unit (guest had 2 MiB)
...
The 2026-07-12 triple-fault was never a CPU/nesting/machine/console problem.
dmacvicar/libvirt >=0.9 reinterprets `memory` as raw libvirt units (KiB), not
MiB (the 0.8-era meaning the modules were authored against). With no
`memory_unit`, `memory = 2048` rendered `<memory unit='KiB'>2048</memory>` ->
QEMU `-m size=2048k` = 2 MiB. boot2 fits in 2 MiB and echoes /boot.config;
/boot/loader does not -- hence the deterministic 262-byte fault, immune to every
knob the prior window turned.
Measured every hop: module input -> tofu state -> domain XML -> virsh dominfo
(Max memory: 2048 KiB) -> live QEMU cmdline.
FOUNDATIONAL: the identical defect was latent in all three VM modules --
opnsense-edge (live blocker), cloudinit-vm (MAAS/NetBox/GitBucket, the next
step), node-vm (Stage 3). Every VR1 VM would have gotten 1024x too little RAM.
- memory_unit = "MiB" on all three libvirt_domain resources (memory_mib = 2048
value untouched -- it now means the 2 GiB always intended).
- Corrected the disproven "svm is ROOT CAUSE" comment in opnsense-edge; the
setting is retained as legitimate hardening, the false claim is not.
- Guard: opentofu-validate.sh S1 + --check-memory-unit flag. `tofu validate`
cannot catch this (memory_unit is optional => omitting it is schema-valid,
which is exactly how it shipped). Harness T3 is a true negative test: the
fixture IS the shipped defect. Harness 1 -> 4 PASS.
- Incident report gains a READ-THIS-FIRST box so nobody re-chases the CPU theories.
Gauntlet ALL GREEN (52 harnesses); repo-lint 0 fail; tofu fmt/validate clean.
Live boot verification remains a separately-gated operator step.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-187: opnsense-edge module boot fixes + boot-triple-fault incident report
...
Office1 OPNsense edge first-boot debugging. Several correct module fixes made;
the boot is NOT yet resolved (BTX-loader triple-fault under double-nested virt),
documented as an OPEN incident for the next session.
Module (opentofu/modules/opnsense-edge/main.tf), all kept (a working config
needs them, none resolved the fault):
- serial console (nano is serial-only; the module had none) + boot-log capture
- machine q35 -> i440fx (pc)
- disk COW-overlay -> direct per-VM copy (matches virt-install --import)
- CPU host-passthrough with AMD svm disabled (documented AMD nested-virt fix;
note the provider key is `features` plural, not `feature`)
Incident report docs/incident-20260712-opnsense-edge-boot-triplefault.md: full
symptom (boot2 echoes /boot.config then triple-faults loading /boot/loader, a
consistent 262-byte fault confirmed fresh each boot), environment (double-nested
AMD Opteron_G3), everything tried, and ranked next steps (full CPU flags
+kvm_pv_eoi/+kvm_pv_unhalt, add a video device for the -D dual console, mem 4G,
UEFI/OVMF, outer-hypervisor CPU). Ledger checkpoint added; session at context
limit.
repo-lint 0 fail; module tofu validate Success.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
| 2026-07-10 |
DC-DC Stage 1 (dc-dc-phase0): first real execution + scaffold/tooling fixes
...
First end-to-end run of a DC-DC runbook against real infrastructure (vcloud host,
OpenTofu v1.12.3). Applied 13 libvirt objects (6 DC1 planes, dc1/office1/dc2
pools, office1-local, 3 mesh legs, all MTU 9000); prior VR0 nets torn down (wan
kept as a gap-#17 model); DC2 planes deferred (NetBox supernet), DC2 storage wired.
DOCFIX-178: scripts/prereqs/ idempotent workstation prereq installers + a
Prerequisites section in the runbook + tests/prereqs/ (25/25). Runbooks must
not assume prereq runtimes.
DOCFIX-179: OpenTofu scaffold first validation + apply. Per-module
required_providers (child modules do NOT inherit provider SOURCE -> OpenTofu
inferred nonexistent hashicorp/libvirt); tofu fmt -recursive; provider "maas"
deferred to Stage 3 (kept a sensitive key out of Stage-1 plan/state);
repo-lint now skips the .terraform/ provider cache; .terraform.lock.hcl added.
DOCFIX-180: dc-dc-phase0 runbook as-executed corrections folded in at phase
close (two MTU domains, non-pristine-host handling, Step-9 13-resource plan,
-input=false, Known-Gap resolved).
repo-lint 0 fail; prereqs harness 25/25; repo-lint harness 34/34.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
DOCFIX-170: full-project sweep -- VR0 runbook + OpenTofu doc fixes
...
Part of an 8-dimension parallel sweep for errors/architectural problems
across the whole repo, requested after DOCFIX-169. phase-08-workload-
cluster-acceptance.md had the same cascading-staleness class as DOCFIX-169
(Designate framing never updated after DOCFIX-167's bundle.yaml edit) --
fixed with a nuanced correction distinguishing "charms now active/idle"
from "D-011.8's real acceptance criterion still needs Stage 7's overlay."
Also fixed a stale opentofu/modules/base-image/outputs.tf comment
referencing nonexistent cloudinit-vm variables. Session ledger updated
with the full 8-dimension sweep summary and two orphaned DOCFIX-number
closures (164, 167) caught as integration misses.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-163: Office1-local network module + OPNsense edge ownership (gaps #12+16)
...
New opentofu/modules/office1-network (dc-planes-shaped, one shared network
instead of six), instantiated for real in main.tf -- deliberately not a
reused host bridge, since that would live outside OpenTofu's D-103
inventory and break the virsh net-list --all completeness check used
throughout the runbooks. Decided gap #16: Office1 gets its own OPNsense
edge (a fourth site), owned by Stage 2's runbook (new Step 4b). Found a
deeper cross-site gap while wiring the edge's WAN side (no per-site
ISP-uplink network exists for ANY site yet, DC1/DC2 included) -- logged as
new gap #17 rather than defaulted to a mesh-link, which would contradict
D-100's management-only ruling for those legs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-148: OpenTofu module audit pass + provider-docs skill reference
...
Audited every "UNVERIFIED"/"inferred"/"assumed" marker across all 9 modules
built this session rather than leaving them as-is. Found and fixed a real
documentation error: opnsense-edge's comments claimed devices.interfaces
list order directly sets OPNsense's LAN/WAN role. Researching OPNsense's own
interface-assignment model showed this conflates two things -- list order
plausibly controls which vtnetN device number a NIC gets (libvirt's own
docs: auto-assigned PCI addresses "usually match" XML order for a simple
topology), but the actual LAN/WAN role is a separate, explicit mapping set
inside config.xml itself ("vtnet0=WAN" is a convention some guides choose,
not an enforced rule). Corrected main.tf's comments and both
lan_network_name/wan_network_name variable descriptions -- no config.xml
exists in this repo yet, so this fixes documentation accuracy for whoever
writes it next, not current behavior.
Confidence upgraded (not just re-hedged) on three other items: node-vm's
boot={order=N} shape (confirmed the provider 1:1-mirrors libvirt's own XML,
and libvirt's own docs confirm the single-attribute shape), create.content.url
accepting local paths (second independent source), and the genisoimage
flags (confirmed standard, cleanly separated from the still-open question of
whether OPNsense's Importer actually reads the result). Confirmed safe:
maas_vm_host's zone/pool computed-if-unset behavior.
Added .claude/skills/openstack-cloud-ops/references/opentofu-provider-docs.md:
indexes every provider/doc source used this session with exact URLs and
confirmed versions, plus the fetch methodology that actually worked
(Registry doc pages are JS-rendered, use GitHub instead; branch names vary,
check default_branch via the API; real example .tf files beat doc-summarized
prose for syntax questions). Wired into SKILL.md's routing table,
cross-referenced from opentofu/README.md in both directions without
duplicating content.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-147: opentofu/modules/maas-vm-host + modules/netem-link
...
Registers the vcloud host with MAAS as a virsh VM host, and applies tc netem
WAN-simulation parameters -- the two remaining unbuilt mechanisms from the
OpenTofu scope, researched before drafting per instruction.
maas-vm-host uses canonical/maas's maas_vm_host (register the chassis so
MAAS discovers already-existing VMs), deliberately not maas_vm_host_machine
(composes new VMs itself -- takes cores/memory/disks as inputs). Read both
schemas and cross-checked against D-103's explicit "MAAS does NOT compose
new ones" before picking one.
netem-link uses terraform_data (OpenTofu's own recommended replacement for
null_resource) with an SSH-wrapped local-exec provisioner -- confirmed
local-exec runs on the machine invoking tofu apply (Office1 per D-103), not
the vcloud host where the bridges live, so a bare local command would do
nothing useful.
Neither module instantiated: needs a real MAAS zone/pool, the vcloud host's
SSH target, and real netem parameters, none of which exist yet.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-146: opnsense-edge module + nano-image/config-ISO prep scripts
...
Implements the mechanism identified in today's OPNsense deployment research
(cloud-init confirmed unreliable on FreeBSD; the real native mechanism is
OPNsense's Configuration Importer, which scans an attached volume for
/conf/config.xml and gained ISO9660 support specifically for VM/cloud
automation).
scripts/opnsense-prep-image.sh: downloads the nano image, decompresses,
converts raw->qcow2, resizes -- work create.content.url almost certainly
can't do itself (a plain fetch, no decompression). No hardcoded mirror URL;
OPNSENSE_MIRROR_BASE is required explicitly.
scripts/opnsense-build-config-iso.sh: builds the ISO9660 image containing
/conf/config.xml that the Configuration Importer picks up.
opentofu/modules/opnsense-edge: wires both together, mechanically identical
to modules/cloudinit-vm's cdrom-attach shape but with no libvirt_cloudinit_disk
resource (wrong format for this). Explicit lan_network_name/wan_network_name
variables rather than an ordered list, since OPNsense's interface-1-is-LAN
convention makes list-ordering a real place for a silent mistake to hide. No
PXE boot-order attribute used -- this VM boots from its own disk, not the
network, so node-vm's flagged-unverified boot field doesn't apply here.
Both new scripts' harnesses (4/4, 3/3) exercise REAL missing-tool guards --
qemu-img/genisoimage/xorriso are all genuinely absent from this environment.
Two things flagged, not presented as fact: whether the Importer's ISO9660
behavior holds on a real boot, and whether interface list-order reliably
maps to LAN/WAN NIC enumeration for this guest. config_iso_path has no
default -- real config.xml content per site is still undesigned. Not
instantiated in root main.tf.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-145 + OPNsense deployment research
...
opentofu/modules/base-image + modules/cloudinit-vm: the cloud-init/pre-built-
image VM pattern for Office1's own service VMs (MAAS-region, NetBox,
GitBucket), verified against examples/alpine_cloudinit.tf. The base image's
.path is threaded across the module boundary as a real attribute reference
rather than reconstructed from pool+name strings. user_data/meta_data/
network_config are required inputs with no default -- the real Office1 VM
configs aren't designed yet, and a plausible DHCP default would silently
fail anyway since these planes carry no libvirt DHCP.
OPNsense research (no DOCFIX -- research delivery, not code): checked
OPNsense's own docs/forum/GitHub. Confirmed cloud-init is genuinely
unreliable on OPNsense (FreeBSD; OPNsense's own forum consensus), closing
the open question cloudinit-vm's build left. Found the real mechanism: the
Configuration Importer scans an attached volume for /conf/config.xml in a
2-3s boot window; ISO9660 support was added to it specifically for VM/cloud
automation -- mechanically identical to cloudinit-vm's cdrom-attach shape,
different payload. Use the nano image for KVM, not the installer images.
Fully sourced in opentofu/README.md. Not built yet, itemized: a decompress/
convert preprocessing script, an ISO9660 config.xml-seed builder, and the
real config.xml content.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-144: opentofu/modules/node-vm + fix real syntax bug in DOCFIX-142
...
Built the D-103 node-VM pattern (blank disk, PXE-boot, MAAS-managed) for
Stage 3 -- this time from the provider's own real example .tf files
(examples/domain_with_network.tf, examples/alpine_cloudinit.tf), not
doc-summarization alone, since that confidence gap is exactly why the domain
module was deferred last time.
That surfaced a real bug in the already-pushed dc-planes/mesh-link/
dc-storage-pool modules (14d7382): they used classic HCL block syntax
(domain { ... }) where this provider's current schema needs attribute-style
objects (domain = { ... }) -- confirmed once real examples showed the actual
convention. All three fixed; opentofu/README.md's schema notes now state this
as a provider-wide assumption for any field not personally confirmed
otherwise.
One inference flagged, not presented as fact: the per-device boot-order
attribute's exact internal shape ({ order = N }) -- the field's existence is
confirmed, its shape is not. Not instantiated in root main.tf: node
count/sizing is a pending Phase-0 decision, not invented here. Still
deferred: the cloud-init/base-image VM pattern (Office1 services + OPNsense),
DC2 planes, netem application.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-142: opentofu/ network + storage-pool scaffold (VR1 IaC, gap #2)
...
First OpenTofu content in the repo. Network/pool layer only: modules/dc-planes
(the six per-DC planes, D-052/D-100), modules/mesh-link (the D-100 dark-fiber
triangle), modules/dc-storage-pool, wired for DC1 (inherits DC0's CIDRs per
D-101) and Office1. DC2 deliberately not wired -- its CIDRs aren't assigned
yet (D-101 open sub-item).
Provider schema (dmacvicar/libvirt v0.9.8) verified against its actual current
docs this session, not memory -- it has diverged materially from older common
examples (network isolation is a nested forward.mode, not top-level mode;
libvirt_domain restructured to ~40 args mirroring raw libvirt XML). Node-VM/
volume resources, OPNsense config, and tc netem application are deliberately
deferred: that schema is too large/unfamiliar to author safely without a real
`tofu providers schema` pass on a connected machine -- see opentofu/README.md
for the full account and the concrete next step.
Added scripts/opentofu-validate.sh (fmt/init/validate wrapper) + a harness
testing what's actually exercisable without a tofu binary (none is available
anywhere this repo has been worked in this session -- logged, not hidden).
Extended repo_lint.py's L1 check to cover .tf files (uncovered before since
none existed). repo-lint 0 fail / 1 documented warn; both harnesses green.
Workflow doc + its companion visual tracker updated to match.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
