opentofu/ -- VR1 (DC-DC) infrastructure-as-code

Owns create/destroy of the "physical" layer per D-103: libvirt domains (node VMs), all virtual networks (dark-fiber mesh, per-DC planes, ISP-edge), the OPNsense edge VMs, and the Office1 service VMs. Runs from the Office1 operator VM against the vcloud host libvirt. See docs/dc-dc-buildout-design.md and docs/dc-dc-deployment-workflow.md (Stage 1-2, tooling gap register item 2) for the surrounding plan.

Status: SCAFFOLD, UNVALIDATED. No tofu/terraform binary was available in the session that authored this -- nothing here has been run through tofu init/validate/plan, let alone apply. Before trusting any of it:

bash scripts/opentofu-validate.sh

on a machine with the binary and network access to the provider registry, and read the result before proceeding.

Scope of this delivery

Built:

• modules/dc-planes -- the six per-DC plane segments (D-052/D-100) as isolated libvirt_network resources.
• modules/mesh-link -- one inter-site L2 segment per leg of the D-100 dark-fiber triangle (DC1<->DC2, DC1<->Office1, DC2<->Office1).
• modules/dc-storage-pool -- a directory-backed libvirt_pool per DC/Office1 for future node-VM disk images.
• modules/node-vm (2026-07-09) -- one MAAS-managed OpenStack node VM per call: a BLANK libvirt_volume boot disk + a libvirt_domain with PXE-boot priority on its first network interface. This is the D-103 "OpenTofu creates the node-VM libvirt domains (SHIM)" need for Stage 3 -- deliberately NOT the cloud-init/pre-built-image pattern (see below), since MAAS images these VMs itself after enlistment. NOT instantiated in root main.tf yet: node count/ memory/vcpu/disk sizing is a Phase-0 host/disk-budget decision (buildout-design Section 3) that hasn't been made -- call the module once real values exist, don't invent placeholder specs to wire it in sooner.
• modules/base-image + modules/cloudinit-vm (2026-07-09) -- the cloud-init/ pre-built-image VM pattern, for Office1's own service VMs (MAAS-region, NetBox, GitBucket). Split into two, matching how examples/alpine_cloudinit.tf itself separates concerns: base-image downloads one shared cloud image ONCE (call it once per distinct image); cloudinit-vm creates a per-VM copy-on-write overlay disk + a libvirt_cloudinit_disk NoCloud seed + the domain (call it once per VM instance, passing module.<base-image-call>.path straight through for the overlay's backing_store.path -- a real attribute reference, not a reconstructed guess). user_data/meta_data/network_config are REQUIRED inputs with no default and no generic fallback -- what Office1's VMs actually need configured is real design work not done yet, and a plausible- looking default (e.g. dhcp4: true) would silently fail anyway since this repo's planes carry no libvirt-managed DHCP. NOT confirmed for OPNsense (FreeBSD-based; cloud-init/NoCloud support is not the same guarantee as for Linux cloud images) -- check that specifically before assuming this module covers OPNsense too. Neither module instantiated in root main.tf yet: no image source has been chosen for any Office1 service VM.
• modules/opnsense-edge (2026-07-09) -- one OPNsense edge VM per call, using OPNsense's own Configuration Importer (NOT cloud-init -- see the dedicated research section below) via a plain ISO9660 volume containing /conf/config.xml, attached as a secondary cdrom disk -- mechanically identical to cloudinit-vm's seed-volume shape, no libvirt_cloudinit_disk resource involved (wrong format for this). Paired with two new scripts that do the OUT-OF-OPENTOFU preparation work: scripts/opnsense-prep-image.sh (download the nano image, decompress, convert to qcow2, resize) and scripts/opnsense-build-config-iso.sh (build the ISO9660 config-seed from a real config.xml). Both scripts require external tools not present in the session that wrote them (qemu-img; genisoimage/xorriso) -- their harnesses only test the guard-clause paths, not the real download/convert/build behavior; see each script's header. config_iso_path has no default: real config.xml content per site is design work not done. Not instantiated in root main.tf.
• modules/maas-vm-host (2026-07-09) -- registers the vcloud host's virsh/ libvirt connection with MAAS as a VM host, via the official canonical/maas provider's maas_vm_host resource (D-103: "OpenTofu registers each DC's libvirt host to that DC's MAAS rack controller as a virsh VM-host, so MAAS DISCOVERS the OpenTofu-created node VMs"). Deliberately does NOT use maas_vm_host_machine -- see the dedicated research section below for why that resource is the wrong one (it composes new VMs; D-103 explicitly rules that out). Not instantiated: needs a real MAAS zone/pool and the vcloud host's real power_address.
• modules/netem-link (2026-07-09) -- applies tc qdisc ... netem WAN-simulation parameters (D-100) to a mesh-link bridge, via a terraform_data resource with local-exec provisioners run over SSH (NOT a bare local command -- OpenTofu runs from Office1, not the vcloud host where the bridges live). See the dedicated research section below. Not instantiated: needs the real bridge name, the vcloud host's SSH target, and the actual netem parameters (still an unruled D-100 decision).
• Root wiring for DC1 (planes + pool, using DC1's inherited DC0 CIDRs) and Office1 (pool only), plus all three mesh links, plus a provider "maas" {} block (api_url/api_key as new required variables, maas_api_key marked sensitive = true).

Syntax bug fixed 2026-07-09: dc-planes, mesh-link, and dc-storage-pool originally used classic HCL block syntax (domain { name = ... }) for libvirt_network's domain/mtu and libvirt_pool's target. Building the domain module surfaced real example .tf files from the provider's own repo (examples/domain_with_network.tf, examples/alpine_cloudinit.tf) that consistently use attribute-style nested objects instead (os = { ... }, devices = { ... }) -- this provider has moved to a schema style where nested structures are object-typed attributes, not blocks. All three modules were corrected to match once this was confirmed; see each file's syntax-note comment. This was not previously verifiable from the doc-summarization passes alone (the docs describe fields as "nested blocks" without showing the actual call syntax) -- real example code is what settled it.

Deliberately NOT built yet, and why:

• DC2's plane networks. D-101 has not assigned DC2's supernet yet (it's an explicit open sub-item). main.tf has the DC2 module block written and commented out, ready to uncomment once NetBox assigns real CIDRs -- do not fill in a guessed default to make it "work" sooner.
• OPNsense edges -- (a)-(c) DONE 2026-07-09, (d) mechanism now also DONE, only real parameters/instantiation remain: (a) network substrate (modules/mesh-link); (b)+(c) the image AND its config (modules/opnsense-edge + two prep scripts) -- UNVALIDATED (no real boot, no tofu binary, no ISO-building tool available this session); real config.xml content per site is still design work, not yet done, and the module has no fallback for it. (d) tc netem -- the injection MECHANISM is now built (modules/netem-link), but the actual latency/jitter/loss/rate values are still an unruled D-100 open item, so nothing is instantiated. Full breakdown in docs/dc-dc-deployment-workflow.md gap register item 4.
• MAAS VM-host registration -- mechanism DONE (modules/maas-vm-host), not instantiated. Needs a real MAAS zone/pool and the vcloud host's power address; neither exists yet.
• MAAS-region / NetBox / GitBucket service VMs -- module exists (cloudinit-vm), not instantiated. No image source has been chosen for any of them, and their actual user_data/meta_data/network_config content (what packages, what config, what static address) is real design work that hasn't been done -- required inputs, no invented placeholders.

OPNsense deployment research (2026-07-09) -- the basis for modules/opnsense-edge

Researched directly against OPNsense's own docs/forum/GitHub, not inferred -- confirms modules/cloudinit-vm genuinely does NOT apply, identifies the real mechanism (which turns out to need the same mechanical shape), and is now implemented as modules/opnsense-edge above. Read this before changing that module -- it's the "why," not just background.

1. Cloud-init is confirmed unreliable on OPNsense, not just unconfirmed. Community consensus on OPNsense's own forum: "FreeBSD cloud-init support, which is not great yet" -- an experienced user's conclusion after investigating it directly. (https://forum.opnsense.org/index.php?topic=42517.0)
2. The real native mechanism is the Configuration Importer, official docs: runs very early in boot (a 2-3 second window), scans attached volumes for /conf/config.xml. Supported filesystems: GPT, MBR, ZFS pool, msdosfs (FAT). (https://docs.opnsense.org/manual/backups.html, https://docs.opnsense.org/development/backend/autorun.html)
3. ISO9660 was added to the Importer specifically for VM/cloud automation -- a closed, milestone-targeted, core-developer-assigned GitHub issue requested it explicitly so a config ISO could be attached as a secondary CD-ROM ("shared one to many," vs. a disk image needing a copy per VM). A later bug report confirms ISO9660 detection is live in current code. This means the exact SAME mechanical pattern as modules/cloudinit-vm (a secondary volume attached as a cdrom disk) applies to OPNsense too -- just with a different payload: a plain ISO9660 image containing /conf/config.xml, not a NoCloud-format seed. (https://github.com/opnsense/core/issues/5733, https://github.com/opnsense/core/issues/10017 -- could not pin the exact shipped version from the 22.7 changelog text directly; the feature's existence is corroborated by the later bug report describing its real-world behavior, which is a stronger signal than the milestone tag alone. Confirm on a real boot before relying on it.)
4. There's also a deeper autorun hook for more advanced provisioning: scripts in /usr/local/etc/rc.syshook.d/import/ can override or extend the default import, running before standard network startup. (https://docs.opnsense.org/development/backend/autorun.html)
5. Use the nano image for KVM, not the vga/serial/dvd installer images. Nano is pre-installed (serial console pre-configured, auto-expands to fill the disk on first boot) -- the installer images need an interactive install wizard, which defeats automation. Confirmed real-world workflow: download the .img.bz2 -> bunzip2 -> qemu-img convert -f raw -O qcow2 -> qemu-img resize (grow) -> boot via libvirt with a FreeBSD os-variant, serial console, no graphics. (https://nickcharlton.net/posts/installing-opnsense-virt-install-kvm-serial, https://opnsense.org/download/)
6. A dedicated "build KVM-ready OPNsense images" community project confirms there is no built-in cloud-init/config-injection in official images -- the pattern is always "deploy the immutable base image, then provision separately," matching finding 2 above (seed minimal config via the Importer to get SSH/API reachable, then drive the rest over the network). (https://github.com/maurice-w/opnsense-vm-images)
7. Network interface convention: interface 1 = LAN (default 192.168.1.1), interface 2 = WAN. Order matters when wiring network_names, the same way modules/node-vm's first-interface-gets-PXE-boot-priority does. (https://docs.opnsense.org/manual/install.html)

Built from this research (2026-07-09): scripts/opnsense-prep-image.sh (step 5's decompress+convert, since create.content.url almost certainly does a plain HTTP fetch with no decompression and can't consume the .img.bz2 directly), scripts/opnsense-build-config-iso.sh (a plain ISO9660 image containing /conf/config.xml, to feed a libvirt_volume's create.content.url the same way libvirt_cloudinit_disk.path feeds cloudinit-vm's seed volume), and modules/opnsense-edge wiring both together. Still not done: the actual config.xml content for OPNsense's own LAN/WAN/routing role at each site -- real design work, no fallback provided; and real-world verification of the whole chain (no tofu binary, no ISO-building tool, and no real OPNsense boot were available this session to confirm any of it end-to-end).

MAAS registration + tc netem research (2026-07-09) -- the basis for modules/maas-vm-host and modules/netem-link

Researched directly against the canonical/terraform-provider-maas provider's own docs and OpenTofu's own official docs, not inferred.

1. A real, official MAAS provider exists: canonical/maas, v2.7.2 (confirmed from the registry page metadata, published 2026-01-30) -- not a community/unofficial one. Moved from an older maas/maas namespace; use canonical/maas. (https://registry.terraform.io/providers/canonical/maas/latest)
2. maas_vm_host (register) vs. maas_vm_host_machine (compose) are DIFFERENT operations -- confirmed by reading both resources' real schemas, not just their names. maas_vm_host_machine's arguments (cores, memory, storage_disks, network_interfaces as INPUTS) are MAAS's own "Compose machine" pod feature -- you specify desired specs and MAAS creates a new VM via the virsh connection itself. D-103 explicitly rules this out: "MAAS owns commission / deploy / power / release of those node VMs; it does NOT compose new ones." maas_vm_host (register the virsh chassis; MAAS's own discovery then enlists whatever domains already exist there) is the one that matches D-103's actual described flow -- modules/node-vm pre-creates the VMs, registering the host is what makes MAAS find them. Using vm_host_machine here would have MAAS and OpenTofu fighting over VM creation. (https://raw.githubusercontent.com/canonical/terraform-provider-maas/master/docs/resources/vm_host.md, .../vm_host_machine.md)
3. power_address confirmed in the exact virsh URI format already used elsewhere in this repo -- the provider's own docs example is "qemu+ssh://172.16.99.2/system", the same shape as this module's own libvirt_uri variable. In practice these should point at the same vcloud host, kept as independent inputs rather than silently assumed identical.
4. local-exec runs on the machine invoking tofu apply (Office1), NOT on the resource or the vcloud host -- confirmed from OpenTofu's own official docs. This matters directly: D-103 says OpenTofu itself runs from the Office1 operator VM, so a bare local-exec command applying tc qdisc would run on Office1, not touch the vcloud host's bridge interfaces at all. modules/netem-link wraps the command in an explicit SSH hop to the vcloud host because of this. (https://opentofu.org/docs/language/resources/provisioners/local-exec/)
5. OpenTofu's own docs recommend terraform_data over the older null_resource for exactly the "run a provisioner with no logical resource to attach it to" case -- a built-in resource, no provider needed. Used in modules/netem-link, not null_resource. (https://opentofu.org/docs/language/resources/tf-data/)

Built from this research (2026-07-09): modules/maas-vm-host (registration only, deliberately no composition), modules/netem-link (terraform_data + SSH-wrapped local-exec, with a destroy-time provisioner to remove the qdisc on teardown). Neither instantiated: both need real inputs (a MAAS zone/pool, the vcloud host's SSH target, real netem parameters) that don't exist yet.

Schema notes (read before extending)

Verified this session against the provider's actual current docs (docs/resources/*.md) AND real example .tf files (examples/*.tf) in dmacvicar/terraform-provider-libvirt, tag v0.9.8, fetched 2026-07-09 -- NOT from training-data memory of older provider versions, which would have been wrong (and initially WAS wrong here once, see the syntax-bug note above):

• Nested structures are attribute-style objects (key = { ... }), everywhere, not classic HCL blocks (key { ... }). Confirmed directly from examples/domain_with_network.tf and examples/alpine_cloudinit.tf (os = { ... }, devices = { ... }, target = { format = { ... } }). This is a provider-wide convention, not resource-specific -- assume it applies to any nested field you haven't personally confirmed otherwise.
• libvirt_network's isolation control lives at forward.mode (a nested attribute), not a top-level mode argument as older examples show. This scaffold OMITS the forward attribute entirely for isolated planes/links (documented native-libvirt behavior for a private, unforwarded switch) rather than guess a mode = "none" value this session could not confirm is still valid. Re-check this specific choice first if tofu validate objects.
• libvirt_domain.devices.disks[].source.volume takes { pool = ..., volume = ... } (pool name + volume name, both strings); .target takes { dev = "vda", bus = "virtio" }; .driver takes { type = "qcow2" }. Confirmed directly from examples/alpine_cloudinit.tf.
• libvirt_domain.devices.interfaces[].source.network takes { network = <libvirt-network-name> } (attach to a libvirt-managed network, what this repo's planes/mesh-links are); .source.bridge takes { bridge = <os-bridge-name> } (attach to a plain OS bridge instead) -- confirmed from examples/domain_with_network.tf. Not used here (planes/mesh-links are libvirt-managed networks), but relevant if OPNsense ends up needing a direct bridge attach to a host NIC.
• libvirt_volume.target.format.type (nested twice: target = { format = { type = "qcow2" } }) and libvirt_volume.backing_store = { path = ..., format = { type = "qcow2" } } (copy-on-write from another volume's path) -- confirmed from examples/alpine_cloudinit.tf.
• UNVERIFIED, flagged in modules/node-vm/main.tf: the per-device boot attribute's internal shape (used to set PXE-first boot order). Confirmed to EXIST as a field name on both interfaces[] and disks[] entries (matching native libvirt's per-device <boot order='N'/>), but { order = N } is an inference by pattern, not confirmed from a real example. Check this first with tofu providers schema -json before relying on PXE-first boot working as written.
• libvirt_pool's target is likewise attribute-style (target = { path = ... }); types like dir are native libvirt storage-pool vocabulary (stable, long-standing, not provider-specific) -- reasonably high confidence.
• libvirt_cloudinit_disk takes flat string arguments name, user_data, meta_data, network_config (cloud-init YAML content, no nested attributes) and exposes a .path -- feed that .path into a libvirt_volume's create.content.url to register it as an attachable cdrom volume. Confirmed directly from examples/alpine_cloudinit.tf.
• A base image created in one module call and consumed via backing_store in a DIFFERENT module call: pass the base module's path OUTPUT straight through as a variable to the consuming module (modules/cloudinit-vm does this). Do not reconstruct a volume's path from its pool name + volume name -- libvirt's actual pool-relative path convention was not independently confirmed this session, and a real .path attribute reference is always available and unambiguous instead.
• create.content.url accepts plain LOCAL filesystem paths, not just http(s) URLs -- inferred from examples/alpine_cloudinit.tf assigning libvirt_cloudinit_disk.alpine_seed.path (a local path, not a URL) directly to create.content.url. modules/opnsense-edge relies on this for BOTH its base image (scripts/opnsense-prep-image.sh's output path) and its config-seed ISO (scripts/opnsense-build-config-iso.sh's output path) -- neither is fetched from a real URL. Reasonably high confidence (it's exactly what the real example does), but the mechanism wasn't named explicitly in the docs pages fetched this session -- confirm with tofu plan if it errors on a file://-less bare path.
• maas_vm_host's type argument accepts "virsh" or "lxd" (confirmed from the provider's own docs); power_address is a flat string, no nested attributes; zone/pool are flat optional strings, not objects. Simpler, flatter schema than the libvirt provider's -- confirmed directly from docs/resources/vm_host.md in canonical/terraform-provider-maas.
• provider "maas" { ... } takes flat top-level arguments (api_url, api_key, api_version [defaults `"2.0"`], installation_method [defaults `"snap"`], tls_ca_cert_path, tls_insecure_skip_verify) -- confirmed from the provider's own docs/index.md, including its exact example block.
• terraform_data's schema is input (optional, stored + reflected in output) and triggers_replace (optional, forces replacement when changed) -- confirmed from OpenTofu's own official docs, including a complete example using it with a local-exec provisioner keyed on triggers_replace. modules/netem-link uses triggers_replace (not input) since nothing needs to be stored/reflected, just re-triggered when the bridge or netem args change.

Conventions carried from the rest of this repo

• No hardcoded repo name (repo-lint L9 would need extending to cover .tf files if this becomes a recurring class of finding -- not yet done, flag if it comes up).
• Every variable without a safe, already-ratified default requires explicit input (no invented fallbacks) -- same "never use an inferred value" discipline as everywhere else in this repo.
• DC1's plane CIDRs are a second, hand-maintained copy of scripts/lib-net.sh's PLANE_CIDRS. Keep them in sync manually until a generator exists to derive one from the other (DOCFIX candidate, not actioned).