# opentofu/ -- VR1 (DC-DC) infrastructure-as-code Owns create/destroy of the "physical" layer per D-103: libvirt domains (node VMs), all virtual networks (dark-fiber mesh, per-DC planes, ISP-edge), the OPNsense edge VMs, and the Office1 service VMs. Runs from the Office1 operator VM against the vcloud host libvirt. See `docs/dc-dc-buildout-design.md` and `docs/dc-dc-deployment-workflow.md` (Stage 1-2, tooling gap register item 2) for the surrounding plan. **Status: SCAFFOLD, UNVALIDATED.** No `tofu`/`terraform` binary was available in the session that authored this -- nothing here has been run through `tofu init`/`validate`/`plan`, let alone `apply`. Before trusting any of it: ``` bash scripts/opentofu-validate.sh ``` on a machine with the binary and network access to the provider registry, and read the result before proceeding. ## Scope of this delivery Built: - `modules/dc-planes` -- the six per-DC plane segments (D-052/D-100) as isolated `libvirt_network` resources. - `modules/mesh-link` -- one inter-site L2 segment per leg of the D-100 dark-fiber triangle (DC1<->DC2, DC1<->Office1, DC2<->Office1). - `modules/dc-storage-pool` -- a directory-backed `libvirt_pool` per DC/Office1 for future node-VM disk images. - `modules/node-vm` (2026-07-09) -- one MAAS-managed OpenStack node VM per call: a BLANK `libvirt_volume` boot disk + a `libvirt_domain` with PXE-boot priority on its first network interface. This is the D-103 "OpenTofu creates the node-VM libvirt domains (SHIM)" need for Stage 3 -- deliberately NOT the cloud-init/pre-built-image pattern (see below), since MAAS images these VMs itself after enlistment. NOT instantiated in root `main.tf` yet: node count/ memory/vcpu/disk sizing is a Phase-0 host/disk-budget decision (buildout-design Section 3) that hasn't been made -- call the module once real values exist, don't invent placeholder specs to wire it in sooner. - `modules/base-image` + `modules/cloudinit-vm` (2026-07-09) -- the cloud-init/ pre-built-image VM pattern, for Office1's own service VMs (MAAS-region, NetBox, GitBucket). Split into two, matching how `examples/alpine_cloudinit.tf` itself separates concerns: `base-image` downloads one shared cloud image ONCE (call it once per distinct image); `cloudinit-vm` creates a per-VM copy-on-write overlay disk + a `libvirt_cloudinit_disk` NoCloud seed + the domain (call it once per VM instance, passing `module..path` straight through for the overlay's `backing_store.path` -- a real attribute reference, not a reconstructed guess). `user_data`/`meta_data`/`network_config` are REQUIRED inputs with no default and no generic fallback -- what Office1's VMs actually need configured is real design work not done yet, and a plausible- looking default (e.g. `dhcp4: true`) would silently fail anyway since this repo's planes carry no libvirt-managed DHCP. **NOT confirmed for OPNsense** (FreeBSD-based; cloud-init/NoCloud support is not the same guarantee as for Linux cloud images) -- check that specifically before assuming this module covers OPNsense too. Neither module instantiated in root `main.tf` yet: no image source has been chosen for any Office1 service VM. - `modules/opnsense-edge` (2026-07-09) -- one OPNsense edge VM per call, using OPNsense's own Configuration Importer (NOT cloud-init -- see the dedicated research section below) via a plain ISO9660 volume containing `/conf/config.xml`, attached as a secondary cdrom disk -- mechanically identical to `cloudinit-vm`'s seed-volume shape, no `libvirt_cloudinit_disk` resource involved (wrong format for this). Paired with two new scripts that do the OUT-OF-OPENTOFU preparation work: `scripts/opnsense-prep-image.sh` (download the nano image, decompress, convert to qcow2, resize) and `scripts/opnsense-build-config-iso.sh` (build the ISO9660 config-seed from a real `config.xml`). Both scripts require external tools not present in the session that wrote them (`qemu-img`; `genisoimage`/`xorriso`) -- their harnesses only test the guard-clause paths, not the real download/convert/build behavior; see each script's header. `config_iso_path` has no default: real `config.xml` content per site is design work not done. Not instantiated in root `main.tf`. - `modules/maas-vm-host` (2026-07-09) -- registers the vcloud host's virsh/ libvirt connection with MAAS as a VM host, via the official `canonical/maas` provider's `maas_vm_host` resource (D-103: "OpenTofu registers each DC's libvirt host to that DC's MAAS rack controller as a virsh VM-host, so MAAS DISCOVERS the OpenTofu-created node VMs"). Deliberately does NOT use `maas_vm_host_machine` -- see the dedicated research section below for why that resource is the wrong one (it composes new VMs; D-103 explicitly rules that out). Not instantiated: needs a real MAAS zone/pool and the vcloud host's real power_address. - `modules/netem-link` (2026-07-09) -- applies `tc qdisc ... netem` WAN-simulation parameters (D-100) to a `mesh-link` bridge, via a `terraform_data` resource with `local-exec` provisioners run over SSH (NOT a bare local command -- OpenTofu runs from Office1, not the vcloud host where the bridges live). See the dedicated research section below. Not instantiated: needs the real bridge name, the vcloud host's SSH target, and the actual netem parameters (still an unruled D-100 decision). - Root wiring for DC1 (planes + pool, using DC1's inherited DC0 CIDRs) and Office1 (pool only), plus all three mesh links, plus a `provider "maas" {}` block (api_url/api_key as new required variables, `maas_api_key` marked `sensitive = true`). **Syntax bug fixed 2026-07-09:** `dc-planes`, `mesh-link`, and `dc-storage-pool` originally used classic HCL block syntax (`domain { name = ... }`) for `libvirt_network`'s `domain`/`mtu` and `libvirt_pool`'s `target`. Building the domain module surfaced real example `.tf` files from the provider's own repo (`examples/domain_with_network.tf`, `examples/alpine_cloudinit.tf`) that consistently use attribute-style nested objects instead (`os = { ... }`, `devices = { ... }`) -- this provider has moved to a schema style where nested structures are object-typed attributes, not blocks. All three modules were corrected to match once this was confirmed; see each file's syntax-note comment. This was not previously verifiable from the doc-summarization passes alone (the docs describe fields as "nested blocks" without showing the actual call syntax) -- real example code is what settled it. Deliberately NOT built yet, and why: - **DC2's plane networks.** D-101 has not assigned DC2's supernet yet (it's an explicit open sub-item). `main.tf` has the DC2 module block written and commented out, ready to uncomment once NetBox assigns real CIDRs -- do not fill in a guessed default to make it "work" sooner. - **OPNsense edges -- (a)-(c) DONE 2026-07-09, (d) mechanism now also DONE, only real parameters/instantiation remain:** (a) network substrate (`modules/mesh-link`); (b)+(c) the image AND its config (`modules/opnsense-edge` + two prep scripts) -- UNVALIDATED (no real boot, no `tofu` binary, no ISO-building tool available this session); real `config.xml` content per site is still design work, not yet done, and the module has no fallback for it. (d) `tc netem` -- the injection MECHANISM is now built (`modules/netem-link`), but the actual latency/jitter/loss/rate values are still an unruled D-100 open item, so nothing is instantiated. Full breakdown in `docs/dc-dc-deployment-workflow.md` gap register item 4. - **MAAS VM-host registration -- mechanism DONE (`modules/maas-vm-host`), not instantiated.** Needs a real MAAS zone/pool and the vcloud host's power address; neither exists yet. - **MAAS-region / NetBox / GitBucket service VMs -- module exists (`cloudinit-vm`), not instantiated.** No image source has been chosen for any of them, and their actual `user_data`/`meta_data`/`network_config` content (what packages, what config, what static address) is real design work that hasn't been done -- required inputs, no invented placeholders. ## OPNsense deployment research (2026-07-09) -- the basis for `modules/opnsense-edge` Researched directly against OPNsense's own docs/forum/GitHub, not inferred -- confirms `modules/cloudinit-vm` genuinely does NOT apply, identifies the real mechanism (which turns out to need the same mechanical shape), and is now implemented as `modules/opnsense-edge` above. Read this before changing that module -- it's the "why," not just background. 1. **Cloud-init is confirmed unreliable on OPNsense, not just unconfirmed.** Community consensus on OPNsense's own forum: "FreeBSD cloud-init support, which is not great yet" -- an experienced user's conclusion after investigating it directly. (https://forum.opnsense.org/index.php?topic=42517.0) 2. **The real native mechanism is the Configuration Importer**, official docs: runs very early in boot (a 2-3 second window), scans attached volumes for `/conf/config.xml`. Supported filesystems: GPT, MBR, ZFS pool, msdosfs (FAT). (https://docs.opnsense.org/manual/backups.html, https://docs.opnsense.org/development/backend/autorun.html) 3. **ISO9660 was added to the Importer specifically for VM/cloud automation** -- a closed, milestone-targeted, core-developer-assigned GitHub issue requested it explicitly so a config ISO could be attached as a secondary CD-ROM ("shared one to many," vs. a disk image needing a copy per VM). A later bug report confirms ISO9660 detection is live in current code. This means the exact SAME mechanical pattern as `modules/cloudinit-vm` (a secondary volume attached as a `cdrom` disk) applies to OPNsense too -- just with a different payload: a plain ISO9660 image containing `/conf/config.xml`, not a NoCloud-format seed. (https://github.com/opnsense/core/issues/5733, https://github.com/opnsense/core/issues/10017 -- could not pin the exact shipped version from the 22.7 changelog text directly; the feature's existence is corroborated by the later bug report describing its real-world behavior, which is a stronger signal than the milestone tag alone. Confirm on a real boot before relying on it.) 4. **There's also a deeper `autorun` hook** for more advanced provisioning: scripts in `/usr/local/etc/rc.syshook.d/import/` can override or extend the default import, running before standard network startup. (https://docs.opnsense.org/development/backend/autorun.html) 5. **Use the nano image for KVM, not the vga/serial/dvd installer images.** Nano is pre-installed (serial console pre-configured, auto-expands to fill the disk on first boot) -- the installer images need an interactive install wizard, which defeats automation. Confirmed real-world workflow: download the `.img.bz2` -> `bunzip2` -> `qemu-img convert -f raw -O qcow2` -> `qemu-img resize` (grow) -> boot via libvirt with a FreeBSD os-variant, serial console, no graphics. (https://nickcharlton.net/posts/installing-opnsense-virt-install-kvm-serial, https://opnsense.org/download/) 6. **A dedicated "build KVM-ready OPNsense images" community project confirms there is no built-in cloud-init/config-injection in official images** -- the pattern is always "deploy the immutable base image, then provision separately," matching finding 2 above (seed minimal config via the Importer to get SSH/API reachable, then drive the rest over the network). (https://github.com/maurice-w/opnsense-vm-images) 7. **Network interface convention: interface 1 = LAN (default 192.168.1.1), interface 2 = WAN.** Order matters when wiring `network_names`, the same way `modules/node-vm`'s first-interface-gets-PXE-boot-priority does. (https://docs.opnsense.org/manual/install.html) **Built from this research (2026-07-09):** `scripts/opnsense-prep-image.sh` (step 5's decompress+convert, since `create.content.url` almost certainly does a plain HTTP fetch with no decompression and can't consume the `.img.bz2` directly), `scripts/opnsense-build-config-iso.sh` (a plain ISO9660 image containing `/conf/config.xml`, to feed a `libvirt_volume`'s `create.content.url` the same way `libvirt_cloudinit_disk.path` feeds `cloudinit-vm`'s seed volume), and `modules/opnsense-edge` wiring both together. **Still not done:** the actual `config.xml` content for OPNsense's own LAN/WAN/routing role at each site -- real design work, no fallback provided; and real-world verification of the whole chain (no `tofu` binary, no ISO-building tool, and no real OPNsense boot were available this session to confirm any of it end-to-end). ## MAAS registration + tc netem research (2026-07-09) -- the basis for `modules/maas-vm-host` and `modules/netem-link` Researched directly against the `canonical/terraform-provider-maas` provider's own docs and OpenTofu's own official docs, not inferred. 1. **A real, official MAAS provider exists: `canonical/maas`**, v2.7.2 (confirmed from the registry page metadata, published 2026-01-30) -- not a community/unofficial one. Moved from an older `maas/maas` namespace; use `canonical/maas`. (https://registry.terraform.io/providers/canonical/maas/latest) 2. **`maas_vm_host` (register) vs. `maas_vm_host_machine` (compose) are DIFFERENT operations -- confirmed by reading both resources' real schemas, not just their names.** `maas_vm_host_machine`'s arguments (`cores`, `memory`, `storage_disks`, `network_interfaces` as INPUTS) are MAAS's own "Compose machine" pod feature -- you specify desired specs and MAAS creates a new VM via the virsh connection itself. D-103 explicitly rules this out: "MAAS owns commission / deploy / power / release of those node VMs; it does NOT compose new ones." `maas_vm_host` (register the virsh chassis; MAAS's own discovery then enlists whatever domains already exist there) is the one that matches D-103's actual described flow -- `modules/node-vm` pre-creates the VMs, registering the host is what makes MAAS find them. Using `vm_host_machine` here would have MAAS and OpenTofu fighting over VM creation. (https://raw.githubusercontent.com/canonical/terraform-provider-maas/master/docs/resources/vm_host.md, .../vm_host_machine.md) 3. **`power_address` confirmed in the exact virsh URI format already used elsewhere in this repo** -- the provider's own docs example is `"qemu+ssh://172.16.99.2/system"`, the same shape as this module's own `libvirt_uri` variable. In practice these should point at the same vcloud host, kept as independent inputs rather than silently assumed identical. 4. **`local-exec` runs on the machine invoking `tofu apply` (Office1), NOT on the resource or the vcloud host** -- confirmed from OpenTofu's own official docs. This matters directly: D-103 says OpenTofu itself runs from the Office1 operator VM, so a bare `local-exec` command applying `tc qdisc` would run on Office1, not touch the vcloud host's bridge interfaces at all. `modules/netem-link` wraps the command in an explicit SSH hop to the vcloud host because of this. (https://opentofu.org/docs/language/resources/provisioners/local-exec/) 5. **OpenTofu's own docs recommend `terraform_data` over the older `null_resource`** for exactly the "run a provisioner with no logical resource to attach it to" case -- a built-in resource, no provider needed. Used in `modules/netem-link`, not `null_resource`. (https://opentofu.org/docs/language/resources/tf-data/) **Built from this research (2026-07-09):** `modules/maas-vm-host` (registration only, deliberately no composition), `modules/netem-link` (`terraform_data` + SSH-wrapped `local-exec`, with a destroy-time provisioner to remove the qdisc on teardown). Neither instantiated: both need real inputs (a MAAS zone/pool, the vcloud host's SSH target, real netem parameters) that don't exist yet. ## Schema notes (read before extending) Verified this session against the provider's actual current docs (`docs/resources/*.md`) AND real example `.tf` files (`examples/*.tf`) in `dmacvicar/terraform-provider-libvirt`, tag `v0.9.8`, fetched 2026-07-09 -- NOT from training-data memory of older provider versions, which would have been wrong (and initially WAS wrong here once, see the syntax-bug note above): - **Nested structures are attribute-style objects (`key = { ... }`), everywhere, not classic HCL blocks (`key { ... }`).** Confirmed directly from `examples/domain_with_network.tf` and `examples/alpine_cloudinit.tf` (`os = { ... }`, `devices = { ... }`, `target = { format = { ... } }`). This is a provider-wide convention, not resource-specific -- assume it applies to any nested field you haven't personally confirmed otherwise. - `libvirt_network`'s isolation control lives at `forward.mode` (a nested attribute), not a top-level `mode` argument as older examples show. This scaffold OMITS the `forward` attribute entirely for isolated planes/links (documented native-libvirt behavior for a private, unforwarded switch) rather than guess a `mode = "none"` value this session could not confirm is still valid. Re-check this specific choice first if `tofu validate` objects. - `libvirt_domain.devices.disks[].source.volume` takes `{ pool = ..., volume = ... }` (pool name + volume name, both strings); `.target` takes `{ dev = "vda", bus = "virtio" }`; `.driver` takes `{ type = "qcow2" }`. Confirmed directly from `examples/alpine_cloudinit.tf`. - `libvirt_domain.devices.interfaces[].source.network` takes `{ network = }` (attach to a libvirt-managed network, what this repo's planes/mesh-links are); `.source.bridge` takes `{ bridge = }` (attach to a plain OS bridge instead) -- confirmed from `examples/domain_with_network.tf`. Not used here (planes/mesh-links are libvirt-managed networks), but relevant if OPNsense ends up needing a direct bridge attach to a host NIC. - `libvirt_volume.target.format.type` (nested twice: `target = { format = { type = "qcow2" } }`) and `libvirt_volume.backing_store = { path = ..., format = { type = "qcow2" } }` (copy-on-write from another volume's path) -- confirmed from `examples/alpine_cloudinit.tf`. - **UNVERIFIED, flagged in `modules/node-vm/main.tf`:** the per-device `boot` attribute's internal shape (used to set PXE-first boot order). Confirmed to EXIST as a field name on both `interfaces[]` and `disks[]` entries (matching native libvirt's per-device ``), but `{ order = N }` is an inference by pattern, not confirmed from a real example. Check this first with `tofu providers schema -json` before relying on PXE-first boot working as written. - `libvirt_pool`'s `target` is likewise attribute-style (`target = { path = ... }`); types like `dir` are native libvirt storage-pool vocabulary (stable, long-standing, not provider-specific) -- reasonably high confidence. - `libvirt_cloudinit_disk` takes flat string arguments `name`, `user_data`, `meta_data`, `network_config` (cloud-init YAML content, no nested attributes) and exposes a `.path` -- feed that `.path` into a `libvirt_volume`'s `create.content.url` to register it as an attachable cdrom volume. Confirmed directly from `examples/alpine_cloudinit.tf`. - A base image created in one module call and consumed via `backing_store` in a DIFFERENT module call: pass the base module's `path` OUTPUT straight through as a variable to the consuming module (`modules/cloudinit-vm` does this). Do not reconstruct a volume's path from its pool name + volume name -- libvirt's actual pool-relative path convention was not independently confirmed this session, and a real `.path` attribute reference is always available and unambiguous instead. - `create.content.url` accepts plain LOCAL filesystem paths, not just http(s) URLs -- inferred from `examples/alpine_cloudinit.tf` assigning `libvirt_cloudinit_disk.alpine_seed.path` (a local path, not a URL) directly to `create.content.url`. `modules/opnsense-edge` relies on this for BOTH its base image (`scripts/opnsense-prep-image.sh`'s output path) and its config-seed ISO (`scripts/opnsense-build-config-iso.sh`'s output path) -- neither is fetched from a real URL. Reasonably high confidence (it's exactly what the real example does), but the mechanism wasn't named explicitly in the docs pages fetched this session -- confirm with `tofu plan` if it errors on a `file://`-less bare path. - `maas_vm_host`'s `type` argument accepts `"virsh"` or `"lxd"` (confirmed from the provider's own docs); `power_address` is a flat string, no nested attributes; `zone`/`pool` are flat optional strings, not objects. Simpler, flatter schema than the libvirt provider's -- confirmed directly from `docs/resources/vm_host.md` in `canonical/terraform-provider-maas`. - `provider "maas" { ... }` takes flat top-level arguments (`api_url`, `api_key`, `api_version` [defaults `"2.0"`], `installation_method` [defaults `"snap"`], `tls_ca_cert_path`, `tls_insecure_skip_verify`) -- confirmed from the provider's own `docs/index.md`, including its exact example block. - `terraform_data`'s schema is `input` (optional, stored + reflected in `output`) and `triggers_replace` (optional, forces replacement when changed) -- confirmed from OpenTofu's own official docs, including a complete example using it with a `local-exec` provisioner keyed on `triggers_replace`. `modules/netem-link` uses `triggers_replace` (not `input`) since nothing needs to be stored/reflected, just re-triggered when the bridge or netem args change. ## Conventions carried from the rest of this repo - No hardcoded repo name (repo-lint L9 would need extending to cover `.tf` files if this becomes a recurring class of finding -- not yet done, flag if it comes up). - Every variable without a safe, already-ratified default requires explicit input (no invented fallbacks) -- same "never use an inferred value" discipline as everywhere else in this repo. - DC1's plane CIDRs are a second, hand-maintained copy of `scripts/lib-net.sh`'s `PLANE_CIDRS`. Keep them in sync manually until a generator exists to derive one from the other (DOCFIX candidate, not actioned).