opentofu/ -- VR1 (DC-DC) infrastructure-as-code

Owns create/destroy of the "physical" layer per D-103: libvirt domains (node VMs), all virtual networks (dark-fiber mesh, per-DC planes, ISP-edge), the OPNsense edge VMs, and the Office1 service VMs. Runs from the Office1 operator VM against the vcloud host libvirt. See docs/dc-dc-buildout-design.md and docs/dc-dc-deployment-workflow.md (Stage 1-2, tooling gap register item 2) for the surrounding plan.

Status: SCAFFOLD, UNVALIDATED. No tofu/terraform binary was available in the session that authored this -- nothing here has been run through tofu init/validate/plan, let alone apply. Before trusting any of it:

bash scripts/opentofu-validate.sh

on a machine with the binary and network access to the provider registry, and read the result before proceeding.

Scope of this delivery

Built:

• modules/dc-planes -- the six per-DC plane segments (D-052/D-100) as isolated libvirt_network resources.
• modules/mesh-link -- one inter-site L2 segment per leg of the D-100 dark-fiber triangle (DC1<->DC2, DC1<->Office1, DC2<->Office1).
• modules/dc-storage-pool -- a directory-backed libvirt_pool per DC/Office1 for future node-VM disk images.
• modules/node-vm (2026-07-09) -- one MAAS-managed OpenStack node VM per call: a BLANK libvirt_volume boot disk + a libvirt_domain with PXE-boot priority on its first network interface. This is the D-103 "OpenTofu creates the node-VM libvirt domains (SHIM)" need for Stage 3 -- deliberately NOT the cloud-init/pre-built-image pattern (see below), since MAAS images these VMs itself after enlistment. NOT instantiated in root main.tf yet: node count/ memory/vcpu/disk sizing is a Phase-0 host/disk-budget decision (buildout-design Section 3) that hasn't been made -- call the module once real values exist, don't invent placeholder specs to wire it in sooner.
• modules/base-image + modules/cloudinit-vm (2026-07-09) -- the cloud-init/ pre-built-image VM pattern, for Office1's own service VMs (MAAS-region, NetBox, GitBucket). Split into two, matching how examples/alpine_cloudinit.tf itself separates concerns: base-image downloads one shared cloud image ONCE (call it once per distinct image); cloudinit-vm creates a per-VM copy-on-write overlay disk + a libvirt_cloudinit_disk NoCloud seed + the domain (call it once per VM instance, passing module.<base-image-call>.path straight through for the overlay's backing_store.path -- a real attribute reference, not a reconstructed guess). user_data/meta_data/network_config are REQUIRED inputs with no default and no generic fallback -- what Office1's VMs actually need configured is real design work not done yet, and a plausible- looking default (e.g. dhcp4: true) would silently fail anyway since this repo's planes carry no libvirt-managed DHCP. NOT confirmed for OPNsense (FreeBSD-based; cloud-init/NoCloud support is not the same guarantee as for Linux cloud images) -- check that specifically before assuming this module covers OPNsense too. Neither module instantiated in root main.tf yet: no image source has been chosen for any Office1 service VM.
• Root wiring for DC1 (planes + pool, using DC1's inherited DC0 CIDRs) and Office1 (pool only), plus all three mesh links.

Syntax bug fixed 2026-07-09: dc-planes, mesh-link, and dc-storage-pool originally used classic HCL block syntax (domain { name = ... }) for libvirt_network's domain/mtu and libvirt_pool's target. Building the domain module surfaced real example .tf files from the provider's own repo (examples/domain_with_network.tf, examples/alpine_cloudinit.tf) that consistently use attribute-style nested objects instead (os = { ... }, devices = { ... }) -- this provider has moved to a schema style where nested structures are object-typed attributes, not blocks. All three modules were corrected to match once this was confirmed; see each file's syntax-note comment. This was not previously verifiable from the doc-summarization passes alone (the docs describe fields as "nested blocks" without showing the actual call syntax) -- real example code is what settled it.

Deliberately NOT built yet, and why:

• DC2's plane networks. D-101 has not assigned DC2's supernet yet (it's an explicit open sub-item). main.tf has the DC2 module block written and commented out, ready to uncomment once NetBox assigns real CIDRs -- do not fill in a guessed default to make it "work" sooner.
• tc netem WAN simulation. libvirt_network has no netem-equivalent argument; this is an OS-level tc qdisc setting applied to the bridge interface a mesh-link module creates, which needs either a null_resource provisioner or a small companion script -- not yet written. The exact parameters are themselves a Phase-0 open item (D-100), so this was going to need a second pass regardless.
• OPNsense edges -- four separable pieces, not one; (b) and (c) are now RESEARCHED (2026-07-09, see the dedicated section below), not blocked: (a) the network substrate is done (modules/mesh-link, above); (b)+(c) the image AND its config are a single confirmed pattern now (nano image + a config.xml-seed ISO attached as cdrom, mechanically identical to modules/cloudinit-vm's shape) -- not yet built as a module, but no longer an open research question, just implementation work; (d) tc netem WAN simulation is a separate OS-level tc qdisc setting with no Terraform-native equivalent at all, needing its own provisioner/script, and whose parameters are still an unruled D-100 open item -- a decision, not just scripting, is needed first. Full breakdown in docs/dc-dc-deployment-workflow.md gap register item 4.
• MAAS-region / NetBox / GitBucket service VMs -- module exists (cloudinit-vm), not instantiated. No image source has been chosen for any of them, and their actual user_data/meta_data/network_config content (what packages, what config, what static address) is real design work that hasn't been done -- required inputs, no invented placeholders.

OPNsense deployment research (2026-07-09) -- read before building the OPNsense module

Researched directly against OPNsense's own docs/forum/GitHub, not inferred -- confirms modules/cloudinit-vm genuinely does NOT apply, AND identifies the real mechanism, which turns out to need the same mechanical shape.

1. Cloud-init is confirmed unreliable on OPNsense, not just unconfirmed. Community consensus on OPNsense's own forum: "FreeBSD cloud-init support, which is not great yet" -- an experienced user's conclusion after investigating it directly. (https://forum.opnsense.org/index.php?topic=42517.0)
2. The real native mechanism is the Configuration Importer, official docs: runs very early in boot (a 2-3 second window), scans attached volumes for /conf/config.xml. Supported filesystems: GPT, MBR, ZFS pool, msdosfs (FAT). (https://docs.opnsense.org/manual/backups.html, https://docs.opnsense.org/development/backend/autorun.html)
3. ISO9660 was added to the Importer specifically for VM/cloud automation -- a closed, milestone-targeted, core-developer-assigned GitHub issue requested it explicitly so a config ISO could be attached as a secondary CD-ROM ("shared one to many," vs. a disk image needing a copy per VM). A later bug report confirms ISO9660 detection is live in current code. This means the exact SAME mechanical pattern as modules/cloudinit-vm (a secondary volume attached as a cdrom disk) applies to OPNsense too -- just with a different payload: a plain ISO9660 image containing /conf/config.xml, not a NoCloud-format seed. (https://github.com/opnsense/core/issues/5733, https://github.com/opnsense/core/issues/10017 -- could not pin the exact shipped version from the 22.7 changelog text directly; the feature's existence is corroborated by the later bug report describing its real-world behavior, which is a stronger signal than the milestone tag alone. Confirm on a real boot before relying on it.)
4. There's also a deeper autorun hook for more advanced provisioning: scripts in /usr/local/etc/rc.syshook.d/import/ can override or extend the default import, running before standard network startup. (https://docs.opnsense.org/development/backend/autorun.html)
5. Use the nano image for KVM, not the vga/serial/dvd installer images. Nano is pre-installed (serial console pre-configured, auto-expands to fill the disk on first boot) -- the installer images need an interactive install wizard, which defeats automation. Confirmed real-world workflow: download the .img.bz2 -> bunzip2 -> qemu-img convert -f raw -O qcow2 -> qemu-img resize (grow) -> boot via libvirt with a FreeBSD os-variant, serial console, no graphics. (https://nickcharlton.net/posts/installing-opnsense-virt-install-kvm-serial, https://opnsense.org/download/)
6. A dedicated "build KVM-ready OPNsense images" community project confirms there is no built-in cloud-init/config-injection in official images -- the pattern is always "deploy the immutable base image, then provision separately," matching finding 2 above (seed minimal config via the Importer to get SSH/API reachable, then drive the rest over the network). (https://github.com/maurice-w/opnsense-vm-images)
7. Network interface convention: interface 1 = LAN (default 192.168.1.1), interface 2 = WAN. Order matters when wiring network_names, the same way modules/node-vm's first-interface-gets-PXE-boot-priority does. (https://docs.opnsense.org/manual/install.html)

What building the module still needs, not yet done: (a) a small preprocessing script for step 5's decompress+convert -- create.content.url almost certainly does a plain HTTP fetch with no decompression, so it can't consume the .img.bz2 directly; the qcow2 needs to exist BEFORE OpenTofu sees it. (b) a way to build a plain ISO9660 image containing /conf/config.xml (e.g. genisoimage/xorriso -- exact tool/flags not yet verified against a real build) to feed into a libvirt_volume's create.content.url, the same way libvirt_cloudinit_disk.path feeds cloudinit-vm's seed volume. (c) an actual config.xml for OPNsense's own LAN/WAN/routing role -- real design work, not yet done.

Schema notes (read before extending)

Verified this session against the provider's actual current docs (docs/resources/*.md) AND real example .tf files (examples/*.tf) in dmacvicar/terraform-provider-libvirt, tag v0.9.8, fetched 2026-07-09 -- NOT from training-data memory of older provider versions, which would have been wrong (and initially WAS wrong here once, see the syntax-bug note above):

• Nested structures are attribute-style objects (key = { ... }), everywhere, not classic HCL blocks (key { ... }). Confirmed directly from examples/domain_with_network.tf and examples/alpine_cloudinit.tf (os = { ... }, devices = { ... }, target = { format = { ... } }). This is a provider-wide convention, not resource-specific -- assume it applies to any nested field you haven't personally confirmed otherwise.
• libvirt_network's isolation control lives at forward.mode (a nested attribute), not a top-level mode argument as older examples show. This scaffold OMITS the forward attribute entirely for isolated planes/links (documented native-libvirt behavior for a private, unforwarded switch) rather than guess a mode = "none" value this session could not confirm is still valid. Re-check this specific choice first if tofu validate objects.
• libvirt_domain.devices.disks[].source.volume takes { pool = ..., volume = ... } (pool name + volume name, both strings); .target takes { dev = "vda", bus = "virtio" }; .driver takes { type = "qcow2" }. Confirmed directly from examples/alpine_cloudinit.tf.
• libvirt_domain.devices.interfaces[].source.network takes { network = <libvirt-network-name> } (attach to a libvirt-managed network, what this repo's planes/mesh-links are); .source.bridge takes { bridge = <os-bridge-name> } (attach to a plain OS bridge instead) -- confirmed from examples/domain_with_network.tf. Not used here (planes/mesh-links are libvirt-managed networks), but relevant if OPNsense ends up needing a direct bridge attach to a host NIC.
• libvirt_volume.target.format.type (nested twice: target = { format = { type = "qcow2" } }) and libvirt_volume.backing_store = { path = ..., format = { type = "qcow2" } } (copy-on-write from another volume's path) -- confirmed from examples/alpine_cloudinit.tf.
• UNVERIFIED, flagged in modules/node-vm/main.tf: the per-device boot attribute's internal shape (used to set PXE-first boot order). Confirmed to EXIST as a field name on both interfaces[] and disks[] entries (matching native libvirt's per-device <boot order='N'/>), but { order = N } is an inference by pattern, not confirmed from a real example. Check this first with tofu providers schema -json before relying on PXE-first boot working as written.
• libvirt_pool's target is likewise attribute-style (target = { path = ... }); types like dir are native libvirt storage-pool vocabulary (stable, long-standing, not provider-specific) -- reasonably high confidence.
• libvirt_cloudinit_disk takes flat string arguments name, user_data, meta_data, network_config (cloud-init YAML content, no nested attributes) and exposes a .path -- feed that .path into a libvirt_volume's create.content.url to register it as an attachable cdrom volume. Confirmed directly from examples/alpine_cloudinit.tf.
• A base image created in one module call and consumed via backing_store in a DIFFERENT module call: pass the base module's path OUTPUT straight through as a variable to the consuming module (modules/cloudinit-vm does this). Do not reconstruct a volume's path from its pool name + volume name -- libvirt's actual pool-relative path convention was not independently confirmed this session, and a real .path attribute reference is always available and unambiguous instead.

Conventions carried from the rest of this repo

• No hardcoded repo name (repo-lint L9 would need extending to cover .tf files if this becomes a recurring class of finding -- not yet done, flag if it comes up).
• Every variable without a safe, already-ratified default requires explicit input (no invented fallbacks) -- same "never use an inferred value" discipline as everywhere else in this repo.
• DC1's plane CIDRs are a second, hand-maintained copy of scripts/lib-net.sh's PLANE_CIDRS. Keep them in sync manually until a generator exists to derive one from the other (DOCFIX candidate, not actioned).