node-vm: pin NIC MACs (54 measured) -- in-place applies can no longer drift them ...

Queued delivery item from the 2026-07-21 commissioning diagnosis: an
"in-place" apply (0/9/0) regenerated all 9 unpinned boot MACs on
2026-07-20 and stranded the fleet in MAAS.

- modules/node-vm: optional interface_macs (all-or-nothing + format
  validations); mac = { address = ... } shape verified against the
  dmacvicar/libvirt 0.9.8 schema dump this session.
- vr1-dc0-substrate: all 9 nodes x 6 planes pinned to values MEASURED
  this session (virsh domiflist on vvr1-dc0); metal-admin entries
  cross-checked vs MAAS boot-interface records 9/9.
- NEW tests/node-vm harness (12 cases). Gauntlet 72 -> 73 ALL GREEN;
  repo-lint 0 fail; opentofu-validate PASS.

NOT YET APPLIED on voffice1 (inner root state) -- the gated apply's plan
must show NO replace; any replace is a STOP.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01STHXiHfoxHqq8fGRVvb66G

modules/node-vm: add serial console + boot log (parameterized dir) ...

Its absence blocked a live diagnosis: 6/9 dc0 nodes stall in MAAS 'Loading
ephemeral' and, with no console and no agent, a stuck node is unobservable.
Same pattern opnsense-edge already carries, for the same reason -- a
PXE-booting node emits everything interesting before anything sshable
exists. Unconditional (MAAS-managed nodes are rebuilt at will; no
live-router bounce risk like the D-129 qga channel). Log dir is a variable
rather than opnsense-edge's hardcoded vcloud literal, which is wrong on any
other libvirt host -- backport queued.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

D-127 IaC autostart matrix baked into deploy tooling + Office1 guest updates ...

Operator directive: bake the autoboot-vs-manual matrix into the IaC so DC0/DC1 get
correct autostart at creation, not by hand. Added an `autostart` bool var to modules
cloudinit-vm/opnsense-edge/node-vm (wired to libvirt_domain.autostart); set explicitly
per instance (shared modules => IaC is source of truth): voffice1=true,
office1-opnsense=true, vvr1-dc0=FALSE, vr1-dc0-opnsense=true, node-vm=FALSE.

HARD GATE verified: the two existing Office1 domains (voffice1, office1-opnsense) plan
as autostart false->true UPDATE-IN-PLACE, NOT replacement (replacement would destroy the
live host/edge). vvr1-dc0 is a new create with autostart=false baked in. Outer plan now
5 add / 2 in-place / 6 destroy; the 2 collapse to 0 once the bookend sets those two
autostart live pre-apply. tests/opentofu-validate T11-T15 pin the matrix against drift.

Standup runbook records the composed service VMs' LXD boot.autostart step. Confirmed
DC0/DC1 create NO LXD instances (inner root = node-vm + opnsense-edge + planes/pool/
wan-bridge) -- DC side is purely this VM matrix.

Office1 guest updates (agent sudo inside the guests): office1-netbox apt upgrade (no
reboot, netbox healthy); office1-tailscale full-upgrade -> kernel 6.8.0-136 + reboot
(back, re-advertising 10.10.0.0/22); voffice1 0-upgradable; LXD boot.autostart=true set
on both. vcloud + office1-opnsense stay in the operator's bookend.

Gauntlet ALL GREEN (66); opentofu-validate clean; repo-lint 0-fail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH

D-114 follow-through: fix node-vm's nested-virt defect NOW; voffice1 pinned to a reserved address ...

Operator ruling: fixes land as we find them -- only DC1 DEPLOYMENT is gated, not DC1 fixes.

1. modules/node-vm: UNCONDITIONAL `cpu = { mode = "host-passthrough" }`. Same defect just found in
   cloudinit-vm: with no cpu block libvirt hands the guest a generic emulated CPU with NO svm. A DC
   node runs nova-compute, so it MUST run KVM guests -- there is no correct value other than this,
   hence no knob that could only ever be set wrong (contrast cloudinit-vm, where it IS a real per-VM
   decision, and opnsense-edge, which disables svm as correct hardening for a router). Not yet
   instantiated, so no live call breaks. Without it the DC nodes would have come up looking healthy
   and been unable to launch a single instance -- surfacing in Stage 5 as an inscrutable scheduler error.

2. voffice1 moved off the dynamic pool onto a RESERVED address. It first took 10.10.0.100 -- the
   first address of Kea's pool (.100-.199). A MAAS region controller must not hold an address that
   can move. Added a Kea host reservation via the D-113(a2) REST API (a second live exercise of that
   write path): 52:54:00:6a:87:e5 -> 10.10.0.20, deliberately OUTSIDE the pool. add_reservation ->
   "saved", service/reconfigure -> "ok". Rebooted; voffice1 came back on 10.10.0.20, .100 released,
   /dev/kvm still present. Done now because nothing depends on the address yet.

3. runbooks/dc-dc-phase1-office1-standup.md rewritten to the D-114 model (was: three sibling service
   VMs + an unruled Option A/B fork -- it would have actively misdirected whoever ran it). New
   sequence: voffice1 via tofu -> first contact (lease/SSH/nested-KVM) -> MAAS on voffice1 -> LXD
   (5.21 LTS) -> register LXD as a MAAS VM host -> MAAS-COMPOSE NetBox/GitBucket/Tailscale. The
   MAAS/LXD command lines carry explicit PENDING VERIFICATION markers rather than fabricated flags.
   docs/dc-dc-deployment-workflow.md Stage 2 + gap register updated to match.

repo-lint 0 fail. opentofu-validate PASS.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

DOCFIX-194: the OpenTofu gate was GREEN over two BROKEN modules ...

scripts/opentofu-validate.sh printed PASS on 2026-07-13 while node-vm and netem-link were flatly
broken -- neither could have applied, and one could not even init. Both are modules Stage 3 and
Stage 4 are built on.

ROOT CAUSE OF THE MISS: root-only validation. `tofu validate` against the ROOT module only parses
modules the root actually INSTANTIATES. A module that is not called -- or is called only from a
commented-out block -- is NEVER PARSED. Root calls dc-planes, dc-storage-pool, mesh-link,
office1-network, opnsense-edge. It does NOT call base-image, cloudinit-vm, maas-vm-host,
netem-link, node-vm -- exactly the set Stage 2 (cloudinit-vm, base-image) and Stage 3 (node-vm,
maas-vm-host, netem-link) depend on. They had never been parsed by any tool.

Only findable because a tofu binary now exists on the jumphost (OpenTofu v1.12.3). The tree was
authored in sessions with no binary to self-check -- exactly the risk opentofu/README.md flagged.

DEFECT 1 -- node-vm: libvirt_volume had a top-level `format = {...}`, which is not a real
attribute ("An argument named format is not expected here"). Schema-verified via
`tofu providers schema -json` (dmacvicar/libvirt v0.9.8): libvirt_volume has NO top-level
`format` -- it nests under `target`. modules/base-image had the correct nesting all along, which
is why it passed and node-vm did not. node-vm builds EVERY DC node (PXE-boot blanks); Stage 3/4
would have failed on first use. The module's own header said "VERIFY with tofu providers schema
-json before the first real apply" -- nobody could, until now.

DEFECT 2 -- netem-link: a destroy-time provisioner referenced var.*, which OpenTofu rejects AT
INIT ("Destroy-time provisioners ... may only reference attributes of the related resource, via
'self', 'count.index', or 'each.key'"). The module could not even initialize. It is the
DR/latency mechanism for Stage 3 and the Stage 6 failover drill. Fixed with the canonical
pattern: stash values in `input`, read back as self.input.*.

THE DURABLE FIX -- S3: the gate now validates EVERY module STANDALONE, in a temp copy (so it
never writes .terraform/ or a module-level lock file into the repo; only the ROOT lock file is
tracked, deliberately). The two bugs are one-line fixes; the real defect was the gate. A gate
that reports green over unparsed code is worse than no gate -- it manufactures confidence.

VERIFICATION: tests/opentofu-validate 9 PASS (T2 skipped -- cannot exercise the missing-binary
guard on a box that has the binary). T8 is the load-bearing case: a fixture whose ROOT IS VALID
and does NOT call a BROKEN module -- root-only validation reports PASS on it, so S3 must FAIL.
If T8 ever goes green, the blind spot is back. T9 proves T8 fails on the defect, not the fixture
shape. T10 covers all 10 real modules. Fixtures use terraform_data (a builtin), so no provider
download is needed.

Gate against the real tree: S1 PASS, S2 PASS, fmt clean, root validate clean, 10/10 modules PASS
standalone. Gauntlet ALL GREEN (53 harnesses); repo-lint 0 fail.

Nothing was instantiated; no live system is affected by either the bug or the fix.

Revert: git revert this commit (restores the two broken modules and the root-only gate).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57

DOCFIX-190: second boot bug -- libvirt domains had ACPI disabled (FreeBSD kernel panic) ...

With DOCFIX-188's memory fix in place the guest finally got its 2 GiB, the kernel
reached interrupt initialisation, and hit a SECOND defect the first bug had masked:

  panic: running without device atpic requires a local APIC
  apic_init() ... mi_startup() ... db>

None of the three VM modules emitted a `features` block, so libvirt rendered
`-machine pc-i440fx-noble,...,acpi=off`. FreeBSD gets its local APIC from ACPI's
MADT table and the OPNsense kernel has no atpic fallback, so with ACPI off it has
no usable interrupt controller. The guest parks in the ddb debugger -- which is
the 100%-of-one-core spin, while virsh still reports the domain as "running".

Fix: features = { acpi = true, apic = {} } in opnsense-edge, cloudinit-vm and
node-vm. node-vm matters most long term: MAAS drives power via ACPI signalling,
so without it graceful shutdown / MAAS power-off would never have worked.

Guard S2 added. Like memory_unit, `tofu validate` cannot catch this -- the whole
features block is optional, so its absence is schema-valid. Flag renamed
--static-only (--check-memory-unit kept as a back-compat alias). Harness 4 -> 6
PASS; fixtures/s2-bad PASSES S1 and still FAILS S2, proving the guards catch
independent classes rather than one masking the other.

LIVE RESULT (measured): acpi=on; kernel boots to userland; CPU idle, no spin;
DHCP lease 172.30.1.126 hostname "OPNsense" on office1-wan. Both boot bugs closed.

STILL OPEN: the Configuration Importer did not apply -- OPNsense came up on
factory defaults. The ISO is verified well-formed (ISO9660 OPNSENSE_CFG containing
CONF/CONFIG.XML with our 10.10.0.1). This is the mechanism the module header has
always flagged UNVERIFIED. Do not guess the fix; read the Importer's console
output. The unanswered WAN ping is NOT a fault -- OPNsense blocks inbound on WAN.

Gauntlet-relevant suites green; repo-lint 0 fail; tofu fmt/validate clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk

DOCFIX-188: root-cause the OPNsense boot triple-fault -- libvirt memory_unit (guest had 2 MiB) ...

The 2026-07-12 triple-fault was never a CPU/nesting/machine/console problem.
dmacvicar/libvirt >=0.9 reinterprets `memory` as raw libvirt units (KiB), not
MiB (the 0.8-era meaning the modules were authored against). With no
`memory_unit`, `memory = 2048` rendered `<memory unit='KiB'>2048</memory>` ->
QEMU `-m size=2048k` = 2 MiB. boot2 fits in 2 MiB and echoes /boot.config;
/boot/loader does not -- hence the deterministic 262-byte fault, immune to every
knob the prior window turned.

Measured every hop: module input -> tofu state -> domain XML -> virsh dominfo
(Max memory: 2048 KiB) -> live QEMU cmdline.

FOUNDATIONAL: the identical defect was latent in all three VM modules --
opnsense-edge (live blocker), cloudinit-vm (MAAS/NetBox/GitBucket, the next
step), node-vm (Stage 3). Every VR1 VM would have gotten 1024x too little RAM.

- memory_unit = "MiB" on all three libvirt_domain resources (memory_mib = 2048
  value untouched -- it now means the 2 GiB always intended).
- Corrected the disproven "svm is ROOT CAUSE" comment in opnsense-edge; the
  setting is retained as legitimate hardening, the false claim is not.
- Guard: opentofu-validate.sh S1 + --check-memory-unit flag. `tofu validate`
  cannot catch this (memory_unit is optional => omitting it is schema-valid,
  which is exactly how it shipped). Harness T3 is a true negative test: the
  fixture IS the shipped defect. Harness 1 -> 4 PASS.
- Incident report gains a READ-THIS-FIRST box so nobody re-chases the CPU theories.

Gauntlet ALL GREEN (52 harnesses); repo-lint 0 fail; tofu fmt/validate clean.
Live boot verification remains a separately-gated operator step.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk

DC-DC Stage 1 (dc-dc-phase0): first real execution + scaffold/tooling fixes ...

First end-to-end run of a DC-DC runbook against real infrastructure (vcloud host,
OpenTofu v1.12.3). Applied 13 libvirt objects (6 DC1 planes, dc1/office1/dc2
pools, office1-local, 3 mesh legs, all MTU 9000); prior VR0 nets torn down (wan
kept as a gap-#17 model); DC2 planes deferred (NetBox supernet), DC2 storage wired.

DOCFIX-178: scripts/prereqs/ idempotent workstation prereq installers + a
  Prerequisites section in the runbook + tests/prereqs/ (25/25). Runbooks must
  not assume prereq runtimes.
DOCFIX-179: OpenTofu scaffold first validation + apply. Per-module
  required_providers (child modules do NOT inherit provider SOURCE -> OpenTofu
  inferred nonexistent hashicorp/libvirt); tofu fmt -recursive; provider "maas"
  deferred to Stage 3 (kept a sensitive key out of Stage-1 plan/state);
  repo-lint now skips the .terraform/ provider cache; .terraform.lock.hcl added.
DOCFIX-180: dc-dc-phase0 runbook as-executed corrections folded in at phase
  close (two MTU domains, non-pristine-host handling, Step-9 13-resource plan,
  -input=false, Known-Gap resolved).

repo-lint 0 fail; prereqs harness 25/25; repo-lint harness 34/34.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

DOCFIX-148: OpenTofu module audit pass + provider-docs skill reference ...

Audited every "UNVERIFIED"/"inferred"/"assumed" marker across all 9 modules
built this session rather than leaving them as-is. Found and fixed a real
documentation error: opnsense-edge's comments claimed devices.interfaces
list order directly sets OPNsense's LAN/WAN role. Researching OPNsense's own
interface-assignment model showed this conflates two things -- list order
plausibly controls which vtnetN device number a NIC gets (libvirt's own
docs: auto-assigned PCI addresses "usually match" XML order for a simple
topology), but the actual LAN/WAN role is a separate, explicit mapping set
inside config.xml itself ("vtnet0=WAN" is a convention some guides choose,
not an enforced rule). Corrected main.tf's comments and both
lan_network_name/wan_network_name variable descriptions -- no config.xml
exists in this repo yet, so this fixes documentation accuracy for whoever
writes it next, not current behavior.

Confidence upgraded (not just re-hedged) on three other items: node-vm's
boot={order=N} shape (confirmed the provider 1:1-mirrors libvirt's own XML,
and libvirt's own docs confirm the single-attribute shape), create.content.url
accepting local paths (second independent source), and the genisoimage
flags (confirmed standard, cleanly separated from the still-open question of
whether OPNsense's Importer actually reads the result). Confirmed safe:
maas_vm_host's zone/pool computed-if-unset behavior.

Added .claude/skills/openstack-cloud-ops/references/opentofu-provider-docs.md:
indexes every provider/doc source used this session with exact URLs and
confirmed versions, plus the fetch methodology that actually worked
(Registry doc pages are JS-rendered, use GitHub instead; branch names vary,
check default_branch via the API; real example .tf files beat doc-summarized
prose for syntax questions). Wired into SKILL.md's routing table,
cross-referenced from opentofu/README.md in both directions without
duplicating content.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A

DOCFIX-144: opentofu/modules/node-vm + fix real syntax bug in DOCFIX-142 ...

Built the D-103 node-VM pattern (blank disk, PXE-boot, MAAS-managed) for
Stage 3 -- this time from the provider's own real example .tf files
(examples/domain_with_network.tf, examples/alpine_cloudinit.tf), not
doc-summarization alone, since that confidence gap is exactly why the domain
module was deferred last time.

That surfaced a real bug in the already-pushed dc-planes/mesh-link/
dc-storage-pool modules (14d7382): they used classic HCL block syntax
(domain { ... }) where this provider's current schema needs attribute-style
objects (domain = { ... }) -- confirmed once real examples showed the actual
convention. All three fixed; opentofu/README.md's schema notes now state this
as a provider-wide assumption for any field not personally confirmed
otherwise.

One inference flagged, not presented as fact: the per-device boot-order
attribute's exact internal shape ({ order = N }) -- the field's existence is
confirmed, its shape is not. Not instantiated in root main.tf: node
count/sizing is a pending Phase-0 decision, not invented here. Still
deferred: the cloud-init/base-image VM pattern (Office1 services + OPNsense),
DC2 planes, netem application.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A