opentofu/ -- VR1 (DC-DC) infrastructure-as-code

Owns create/destroy of the "physical" layer per D-103: libvirt domains (node VMs), all virtual networks (dark-fiber mesh, per-DC planes, ISP-edge), the OPNsense edge VMs, and the Office1 service VMs. Runs from the Office1 operator VM against the vcloud host libvirt. See docs/dc-dc-buildout-design.md and docs/dc-dc-deployment-workflow.md (Stage 1-2, tooling gap register item 2) for the surrounding plan.

Status: SCAFFOLD, UNVALIDATED. No tofu/terraform binary was available in the session that authored this -- nothing here has been run through tofu init/validate/plan, let alone apply. Before trusting any of it:

bash scripts/opentofu-validate.sh

on a machine with the binary and network access to the provider registry, and read the result before proceeding.

Scope of this delivery

Built:

• modules/dc-planes -- the six per-DC plane segments (D-052/D-100) as isolated libvirt_network resources.
• modules/mesh-link -- one inter-site L2 segment per leg of the D-100 dark-fiber triangle (DC1<->DC2, DC1<->Office1, DC2<->Office1).
• modules/dc-storage-pool -- a directory-backed libvirt_pool per DC/Office1 for future node-VM disk images.
• modules/node-vm (2026-07-09) -- one MAAS-managed OpenStack node VM per call: a BLANK libvirt_volume boot disk + a libvirt_domain with PXE-boot priority on its first network interface. This is the D-103 "OpenTofu creates the node-VM libvirt domains (SHIM)" need for Stage 3 -- deliberately NOT the cloud-init/pre-built-image pattern (see below), since MAAS images these VMs itself after enlistment. NOT instantiated in root main.tf yet: node count/ memory/vcpu/disk sizing is a Phase-0 host/disk-budget decision (buildout-design Section 3) that hasn't been made -- call the module once real values exist, don't invent placeholder specs to wire it in sooner.
• modules/base-image + modules/cloudinit-vm (2026-07-09) -- the cloud-init/ pre-built-image VM pattern, for Office1's own service VMs (MAAS-region, NetBox, GitBucket). Split into two, matching how examples/alpine_cloudinit.tf itself separates concerns: base-image downloads one shared cloud image ONCE (call it once per distinct image); cloudinit-vm creates a per-VM copy-on-write overlay disk + a libvirt_cloudinit_disk NoCloud seed + the domain (call it once per VM instance, passing module.<base-image-call>.path straight through for the overlay's backing_store.path -- a real attribute reference, not a reconstructed guess). user_data/meta_data/network_config are REQUIRED inputs with no default and no generic fallback -- what Office1's VMs actually need configured is real design work not done yet, and a plausible- looking default (e.g. dhcp4: true) would silently fail anyway since this repo's planes carry no libvirt-managed DHCP. NOT confirmed for OPNsense (FreeBSD-based; cloud-init/NoCloud support is not the same guarantee as for Linux cloud images) -- check that specifically before assuming this module covers OPNsense too. Neither module instantiated in root main.tf yet: no image source has been chosen for any Office1 service VM.
• modules/opnsense-edge (2026-07-09) -- one OPNsense edge VM per call, using OPNsense's own Configuration Importer (NOT cloud-init -- see the dedicated research section below) via a plain ISO9660 volume containing /conf/config.xml, attached as a secondary cdrom disk -- mechanically identical to cloudinit-vm's seed-volume shape, no libvirt_cloudinit_disk resource involved (wrong format for this). Paired with two new scripts that do the OUT-OF-OPENTOFU preparation work: scripts/opnsense-prep-image.sh (download the nano image, decompress, convert to qcow2, resize) and scripts/opnsense-build-config-iso.sh (build the ISO9660 config-seed from a real config.xml). Both scripts require external tools not present in the session that wrote them (qemu-img; genisoimage/xorriso) -- their harnesses only test the guard-clause paths, not the real download/convert/build behavior; see each script's header. config_iso_path has no default: real config.xml content per site is design work not done. Not instantiated in root main.tf.
• Root wiring for DC1 (planes + pool, using DC1's inherited DC0 CIDRs) and Office1 (pool only), plus all three mesh links.

Syntax bug fixed 2026-07-09: dc-planes, mesh-link, and dc-storage-pool originally used classic HCL block syntax (domain { name = ... }) for libvirt_network's domain/mtu and libvirt_pool's target. Building the domain module surfaced real example .tf files from the provider's own repo (examples/domain_with_network.tf, examples/alpine_cloudinit.tf) that consistently use attribute-style nested objects instead (os = { ... }, devices = { ... }) -- this provider has moved to a schema style where nested structures are object-typed attributes, not blocks. All three modules were corrected to match once this was confirmed; see each file's syntax-note comment. This was not previously verifiable from the doc-summarization passes alone (the docs describe fields as "nested blocks" without showing the actual call syntax) -- real example code is what settled it.

Deliberately NOT built yet, and why:

• DC2's plane networks. D-101 has not assigned DC2's supernet yet (it's an explicit open sub-item). main.tf has the DC2 module block written and commented out, ready to uncomment once NetBox assigns real CIDRs -- do not fill in a guessed default to make it "work" sooner.
• tc netem WAN simulation. libvirt_network has no netem-equivalent argument; this is an OS-level tc qdisc setting applied to the bridge interface a mesh-link module creates, which needs either a null_resource provisioner or a small companion script -- not yet written. The exact parameters are themselves a Phase-0 open item (D-100), so this was going to need a second pass regardless.
• OPNsense edges -- (a)-(c) DONE 2026-07-09, only (d) remains: (a) network substrate (modules/mesh-link); (b)+(c) the image AND its config, built as modules/opnsense-edge + the two prep scripts above -- UNVALIDATED (no real boot, no tofu binary, no ISO-building tool available this session; see each file's header for exactly what's confirmed vs. assumed); real config.xml content per site is still design work, not yet done, and the module has no fallback for it. (d) tc netem WAN simulation is a separate OS-level tc qdisc setting with no Terraform-native equivalent at all, needing its own provisioner/script, and whose parameters are still an unruled D-100 open item -- a decision, not just scripting, is needed first. Full breakdown in docs/dc-dc-deployment-workflow.md gap register item 4.
• MAAS-region / NetBox / GitBucket service VMs -- module exists (cloudinit-vm), not instantiated. No image source has been chosen for any of them, and their actual user_data/meta_data/network_config content (what packages, what config, what static address) is real design work that hasn't been done -- required inputs, no invented placeholders.

OPNsense deployment research (2026-07-09) -- the basis for modules/opnsense-edge

Researched directly against OPNsense's own docs/forum/GitHub, not inferred -- confirms modules/cloudinit-vm genuinely does NOT apply, identifies the real mechanism (which turns out to need the same mechanical shape), and is now implemented as modules/opnsense-edge above. Read this before changing that module -- it's the "why," not just background.

1. Cloud-init is confirmed unreliable on OPNsense, not just unconfirmed. Community consensus on OPNsense's own forum: "FreeBSD cloud-init support, which is not great yet" -- an experienced user's conclusion after investigating it directly. (https://forum.opnsense.org/index.php?topic=42517.0)
2. The real native mechanism is the Configuration Importer, official docs: runs very early in boot (a 2-3 second window), scans attached volumes for /conf/config.xml. Supported filesystems: GPT, MBR, ZFS pool, msdosfs (FAT). (https://docs.opnsense.org/manual/backups.html, https://docs.opnsense.org/development/backend/autorun.html)
3. ISO9660 was added to the Importer specifically for VM/cloud automation -- a closed, milestone-targeted, core-developer-assigned GitHub issue requested it explicitly so a config ISO could be attached as a secondary CD-ROM ("shared one to many," vs. a disk image needing a copy per VM). A later bug report confirms ISO9660 detection is live in current code. This means the exact SAME mechanical pattern as modules/cloudinit-vm (a secondary volume attached as a cdrom disk) applies to OPNsense too -- just with a different payload: a plain ISO9660 image containing /conf/config.xml, not a NoCloud-format seed. (https://github.com/opnsense/core/issues/5733, https://github.com/opnsense/core/issues/10017 -- could not pin the exact shipped version from the 22.7 changelog text directly; the feature's existence is corroborated by the later bug report describing its real-world behavior, which is a stronger signal than the milestone tag alone. Confirm on a real boot before relying on it.)
4. There's also a deeper autorun hook for more advanced provisioning: scripts in /usr/local/etc/rc.syshook.d/import/ can override or extend the default import, running before standard network startup. (https://docs.opnsense.org/development/backend/autorun.html)
5. Use the nano image for KVM, not the vga/serial/dvd installer images. Nano is pre-installed (serial console pre-configured, auto-expands to fill the disk on first boot) -- the installer images need an interactive install wizard, which defeats automation. Confirmed real-world workflow: download the .img.bz2 -> bunzip2 -> qemu-img convert -f raw -O qcow2 -> qemu-img resize (grow) -> boot via libvirt with a FreeBSD os-variant, serial console, no graphics. (https://nickcharlton.net/posts/installing-opnsense-virt-install-kvm-serial, https://opnsense.org/download/)
6. A dedicated "build KVM-ready OPNsense images" community project confirms there is no built-in cloud-init/config-injection in official images -- the pattern is always "deploy the immutable base image, then provision separately," matching finding 2 above (seed minimal config via the Importer to get SSH/API reachable, then drive the rest over the network). (https://github.com/maurice-w/opnsense-vm-images)
7. Network interface convention: interface 1 = LAN (default 192.168.1.1), interface 2 = WAN. Order matters when wiring network_names, the same way modules/node-vm's first-interface-gets-PXE-boot-priority does. (https://docs.opnsense.org/manual/install.html)

Built from this research (2026-07-09): scripts/opnsense-prep-image.sh (step 5's decompress+convert, since create.content.url almost certainly does a plain HTTP fetch with no decompression and can't consume the .img.bz2 directly), scripts/opnsense-build-config-iso.sh (a plain ISO9660 image containing /conf/config.xml, to feed a libvirt_volume's create.content.url the same way libvirt_cloudinit_disk.path feeds cloudinit-vm's seed volume), and modules/opnsense-edge wiring both together. Still not done: the actual config.xml content for OPNsense's own LAN/WAN/routing role at each site -- real design work, no fallback provided; and real-world verification of the whole chain (no tofu binary, no ISO-building tool, and no real OPNsense boot were available this session to confirm any of it end-to-end).

Schema notes (read before extending)

Verified this session against the provider's actual current docs (docs/resources/*.md) AND real example .tf files (examples/*.tf) in dmacvicar/terraform-provider-libvirt, tag v0.9.8, fetched 2026-07-09 -- NOT from training-data memory of older provider versions, which would have been wrong (and initially WAS wrong here once, see the syntax-bug note above):

• Nested structures are attribute-style objects (key = { ... }), everywhere, not classic HCL blocks (key { ... }). Confirmed directly from examples/domain_with_network.tf and examples/alpine_cloudinit.tf (os = { ... }, devices = { ... }, target = { format = { ... } }). This is a provider-wide convention, not resource-specific -- assume it applies to any nested field you haven't personally confirmed otherwise.
• libvirt_network's isolation control lives at forward.mode (a nested attribute), not a top-level mode argument as older examples show. This scaffold OMITS the forward attribute entirely for isolated planes/links (documented native-libvirt behavior for a private, unforwarded switch) rather than guess a mode = "none" value this session could not confirm is still valid. Re-check this specific choice first if tofu validate objects.
• libvirt_domain.devices.disks[].source.volume takes { pool = ..., volume = ... } (pool name + volume name, both strings); .target takes { dev = "vda", bus = "virtio" }; .driver takes { type = "qcow2" }. Confirmed directly from examples/alpine_cloudinit.tf.
• libvirt_domain.devices.interfaces[].source.network takes { network = <libvirt-network-name> } (attach to a libvirt-managed network, what this repo's planes/mesh-links are); .source.bridge takes { bridge = <os-bridge-name> } (attach to a plain OS bridge instead) -- confirmed from examples/domain_with_network.tf. Not used here (planes/mesh-links are libvirt-managed networks), but relevant if OPNsense ends up needing a direct bridge attach to a host NIC.
• libvirt_volume.target.format.type (nested twice: target = { format = { type = "qcow2" } }) and libvirt_volume.backing_store = { path = ..., format = { type = "qcow2" } } (copy-on-write from another volume's path) -- confirmed from examples/alpine_cloudinit.tf.
• UNVERIFIED, flagged in modules/node-vm/main.tf: the per-device boot attribute's internal shape (used to set PXE-first boot order). Confirmed to EXIST as a field name on both interfaces[] and disks[] entries (matching native libvirt's per-device <boot order='N'/>), but { order = N } is an inference by pattern, not confirmed from a real example. Check this first with tofu providers schema -json before relying on PXE-first boot working as written.
• libvirt_pool's target is likewise attribute-style (target = { path = ... }); types like dir are native libvirt storage-pool vocabulary (stable, long-standing, not provider-specific) -- reasonably high confidence.
• libvirt_cloudinit_disk takes flat string arguments name, user_data, meta_data, network_config (cloud-init YAML content, no nested attributes) and exposes a .path -- feed that .path into a libvirt_volume's create.content.url to register it as an attachable cdrom volume. Confirmed directly from examples/alpine_cloudinit.tf.
• A base image created in one module call and consumed via backing_store in a DIFFERENT module call: pass the base module's path OUTPUT straight through as a variable to the consuming module (modules/cloudinit-vm does this). Do not reconstruct a volume's path from its pool name + volume name -- libvirt's actual pool-relative path convention was not independently confirmed this session, and a real .path attribute reference is always available and unambiguous instead.
• create.content.url accepts plain LOCAL filesystem paths, not just http(s) URLs -- inferred from examples/alpine_cloudinit.tf assigning libvirt_cloudinit_disk.alpine_seed.path (a local path, not a URL) directly to create.content.url. modules/opnsense-edge relies on this for BOTH its base image (scripts/opnsense-prep-image.sh's output path) and its config-seed ISO (scripts/opnsense-build-config-iso.sh's output path) -- neither is fetched from a real URL. Reasonably high confidence (it's exactly what the real example does), but the mechanism wasn't named explicitly in the docs pages fetched this session -- confirm with tofu plan if it errors on a file://-less bare path.

Conventions carried from the rest of this repo

• No hardcoded repo name (repo-lint L9 would need extending to cover .tf files if this becomes a recurring class of finding -- not yet done, flag if it comes up).
• Every variable without a safe, already-ratified default requires explicit input (no invented fallbacks) -- same "never use an inferred value" discipline as everywhere else in this repo.
• DC1's plane CIDRs are a second, hand-maintained copy of scripts/lib-net.sh's PLANE_CIDRS. Keep them in sync manually until a generator exists to derive one from the other (DOCFIX candidate, not actioned).