| 2026-07-17 |

D-127 IaC autostart matrix baked into deploy tooling + Office1 guest updates
...
Operator directive: bake the autoboot-vs-manual matrix into the IaC so DC0/DC1 get
correct autostart at creation, not by hand. Added an `autostart` bool var to modules
cloudinit-vm/opnsense-edge/node-vm (wired to libvirt_domain.autostart); set explicitly
per instance (shared modules => IaC is source of truth): voffice1=true,
office1-opnsense=true, vvr1-dc0=FALSE, vr1-dc0-opnsense=true, node-vm=FALSE.
HARD GATE verified: the two existing Office1 domains (voffice1, office1-opnsense) plan
as autostart false->true UPDATE-IN-PLACE, NOT replacement (replacement would destroy the
live host/edge). vvr1-dc0 is a new create with autostart=false baked in. Outer plan now
5 add / 2 in-place / 6 destroy; the 2 collapse to 0 once the bookend sets those two
autostart live pre-apply. tests/opentofu-validate T11-T15 pin the matrix against drift.
Standup runbook records the composed service VMs' LXD boot.autostart step. Confirmed
DC0/DC1 create NO LXD instances (inner root = node-vm + opnsense-edge + planes/pool/
wan-bridge) -- DC side is purely this VM matrix.
Office1 guest updates (agent sudo inside the guests): office1-netbox apt upgrade (no
reboot, netbox healthy); office1-tailscale full-upgrade -> kernel 6.8.0-136 + reboot
(back, re-advertising 10.10.0.0/22); voffice1 0-upgradable; LXD boot.autostart=true set
on both. vcloud + office1-opnsense stay in the operator's bookend.
Gauntlet ALL GREEN (66); opentofu-validate clean; repo-lint 0-fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
| 2026-07-13 |

DOCFIX-194: the OpenTofu gate was GREEN over two BROKEN modules
...
scripts/opentofu-validate.sh printed PASS on 2026-07-13 while node-vm and netem-link were flatly
broken -- neither could have applied, and one could not even init. Both are modules Stage 3 and
Stage 4 are built on.
ROOT CAUSE OF THE MISS: root-only validation. `tofu validate` against the ROOT module only parses
modules the root actually INSTANTIATES. A module that is not called -- or is called only from a
commented-out block -- is NEVER PARSED. Root calls dc-planes, dc-storage-pool, mesh-link,
office1-network, opnsense-edge. It does NOT call base-image, cloudinit-vm, maas-vm-host,
netem-link, node-vm -- exactly the set Stage 2 (cloudinit-vm, base-image) and Stage 3 (node-vm,
maas-vm-host, netem-link) depend on. They had never been parsed by any tool.
Only findable because a tofu binary now exists on the jumphost (OpenTofu v1.12.3). The tree was
authored in sessions with no binary to self-check -- exactly the risk opentofu/README.md flagged.
DEFECT 1 -- node-vm: libvirt_volume had a top-level `format = {...}`, which is not a real
attribute ("An argument named format is not expected here"). Schema-verified via
`tofu providers schema -json` (dmacvicar/libvirt v0.9.8): libvirt_volume has NO top-level
`format` -- it nests under `target`. modules/base-image had the correct nesting all along, which
is why it passed and node-vm did not. node-vm builds EVERY DC node (PXE-boot blanks); Stage 3/4
would have failed on first use. The module's own header said "VERIFY with tofu providers schema
-json before the first real apply" -- nobody could, until now.
DEFECT 2 -- netem-link: a destroy-time provisioner referenced var.*, which OpenTofu rejects AT
INIT ("Destroy-time provisioners ... may only reference attributes of the related resource, via
'self', 'count.index', or 'each.key'"). The module could not even initialize. It is the
DR/latency mechanism for Stage 3 and the Stage 6 failover drill. Fixed with the canonical
pattern: stash values in `input`, read back as self.input.*.
THE DURABLE FIX -- S3: the gate now validates EVERY module STANDALONE, in a temp copy (so it
never writes .terraform/ or a module-level lock file into the repo; only the ROOT lock file is
tracked, deliberately). The two bugs are one-line fixes; the real defect was the gate. A gate
that reports green over unparsed code is worse than no gate -- it manufactures confidence.
VERIFICATION: tests/opentofu-validate 9 PASS (T2 skipped -- cannot exercise the missing-binary
guard on a box that has the binary). T8 is the load-bearing case: a fixture whose ROOT IS VALID
and does NOT call a BROKEN module -- root-only validation reports PASS on it, so S3 must FAIL.
If T8 ever goes green, the blind spot is back. T9 proves T8 fails on the defect, not the fixture
shape. T10 covers all 10 real modules. Fixtures use terraform_data (a builtin), so no provider
download is needed.
Gate against the real tree: S1 PASS, S2 PASS, fmt clean, root validate clean, 10/10 modules PASS
standalone. Gauntlet ALL GREEN (53 harnesses); repo-lint 0 fail.
Nothing was instantiated; no live system is affected by either the bug or the fix.
Revert: git revert this commit (restores the two broken modules and the root-only gate).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
| 2026-07-12 |

DOCFIX-190: second boot bug -- libvirt domains had ACPI disabled (FreeBSD kernel panic)
...
With DOCFIX-188's memory fix in place the guest finally got its 2 GiB, the kernel
reached interrupt initialisation, and hit a SECOND defect the first bug had masked:
panic: running without device atpic requires a local APIC
apic_init() ... mi_startup() ... db>
None of the three VM modules emitted a `features` block, so libvirt rendered
`-machine pc-i440fx-noble,...,acpi=off`. FreeBSD gets its local APIC from ACPI's
MADT table and the OPNsense kernel has no atpic fallback, so with ACPI off it has
no usable interrupt controller. The guest parks in the ddb debugger -- which is
the 100%-of-one-core spin, while virsh still reports the domain as "running".
Fix: features = { acpi = true, apic = {} } in opnsense-edge, cloudinit-vm and
node-vm. node-vm matters most long term: MAAS drives power via ACPI signalling,
so without it graceful shutdown / MAAS power-off would never have worked.
Guard S2 added. Like memory_unit, `tofu validate` cannot catch this -- the whole
features block is optional, so its absence is schema-valid. Flag renamed
--static-only (--check-memory-unit kept as a back-compat alias). Harness 4 -> 6
PASS; fixtures/s2-bad PASSES S1 and still FAILS S2, proving the guards catch
independent classes rather than one masking the other.
LIVE RESULT (measured): acpi=on; kernel boots to userland; CPU idle, no spin;
DHCP lease 172.30.1.126 hostname "OPNsense" on office1-wan. Both boot bugs closed.
STILL OPEN: the Configuration Importer did not apply -- OPNsense came up on
factory defaults. The ISO is verified well-formed (ISO9660 OPNSENSE_CFG containing
CONF/CONFIG.XML with our 10.10.0.1). This is the mechanism the module header has
always flagged UNVERIFIED. Do not guess the fix; read the Importer's console
output. The unanswered WAN ping is NOT a fault -- OPNsense blocks inbound on WAN.
Gauntlet-relevant suites green; repo-lint 0 fail; tofu fmt/validate clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|

DOCFIX-188: root-cause the OPNsense boot triple-fault -- libvirt memory_unit (guest had 2 MiB)
...
The 2026-07-12 triple-fault was never a CPU/nesting/machine/console problem.
dmacvicar/libvirt >=0.9 reinterprets `memory` as raw libvirt units (KiB), not
MiB (the 0.8-era meaning the modules were authored against). With no
`memory_unit`, `memory = 2048` rendered `<memory unit='KiB'>2048</memory>` ->
QEMU `-m size=2048k` = 2 MiB. boot2 fits in 2 MiB and echoes /boot.config;
/boot/loader does not -- hence the deterministic 262-byte fault, immune to every
knob the prior window turned.
Measured every hop: module input -> tofu state -> domain XML -> virsh dominfo
(Max memory: 2048 KiB) -> live QEMU cmdline.
FOUNDATIONAL: the identical defect was latent in all three VM modules --
opnsense-edge (live blocker), cloudinit-vm (MAAS/NetBox/GitBucket, the next
step), node-vm (Stage 3). Every VR1 VM would have gotten 1024x too little RAM.
- memory_unit = "MiB" on all three libvirt_domain resources (memory_mib = 2048
value untouched -- it now means the 2 GiB always intended).
- Corrected the disproven "svm is ROOT CAUSE" comment in opnsense-edge; the
setting is retained as legitimate hardening, the false claim is not.
- Guard: opentofu-validate.sh S1 + --check-memory-unit flag. `tofu validate`
cannot catch this (memory_unit is optional => omitting it is schema-valid,
which is exactly how it shipped). Harness T3 is a true negative test: the
fixture IS the shipped defect. Harness 1 -> 4 PASS.
- Incident report gains a READ-THIS-FIRST box so nobody re-chases the CPU theories.
Gauntlet ALL GREEN (52 harnesses); repo-lint 0 fail; tofu fmt/validate clean.
Live boot verification remains a separately-gated operator step.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
| 2026-07-10 |

DOCFIX-142: opentofu/ network + storage-pool scaffold (VR1 IaC, gap #2)
...
First OpenTofu content in the repo. Network/pool layer only: modules/dc-planes
(the six per-DC planes, D-052/D-100), modules/mesh-link (the D-100 dark-fiber
triangle), modules/dc-storage-pool, wired for DC1 (inherits DC0's CIDRs per
D-101) and Office1. DC2 deliberately not wired -- its CIDRs aren't assigned
yet (D-101 open sub-item).
Provider schema (dmacvicar/libvirt v0.9.8) verified against its actual current
docs this session, not memory -- it has diverged materially from older common
examples (network isolation is a nested forward.mode, not top-level mode;
libvirt_domain restructured to ~40 args mirroring raw libvirt XML). Node-VM/
volume resources, OPNsense config, and tc netem application are deliberately
deferred: that schema is too large/unfamiliar to author safely without a real
`tofu providers schema` pass on a connected machine -- see opentofu/README.md
for the full account and the concrete next step.
Added scripts/opentofu-validate.sh (fmt/init/validate wrapper) + a harness
testing what's actually exercisable without a tofu binary (none is available
anywhere this repo has been worked in this session -- logged, not hidden).
Extended repo_lint.py's L1 check to cover .tf files (uncovered before since
none existed). repo-lint 0 fail / 1 documented warn; both harnesses green.
Workflow doc + its companion visual tracker updated to match.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|