Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / opentofu / README.md
@JANeumatrix JANeumatrix 2 hours ago 38 KB DC-DC Stage 1 (dc-dc-phase0): first real execution + scaffold/tooling fixes
Raw Blame History

opentofu/ -- VR1 (DC-DC) infrastructure-as-code

Owns create/destroy of the "physical" layer per D-103: libvirt domains (node VMs), all virtual networks (dark-fiber mesh, per-DC planes, ISP-edge), the OPNsense edge VMs, and the Office1 service VMs. Runs from the Office1 operator VM against the vcloud host libvirt. See docs/dc-dc-buildout-design.md and docs/dc-dc-deployment-workflow.md (Stage 1-2, tooling gap register item 2) for the surrounding plan. See the skill's .claude/skills/openstack-cloud-ops/references/opentofu-provider-docs.md for where every provider fact below was sourced from and the fetch methodology that worked -- read that first if extending any module here.

Status: STAGE-1 VALIDATED + APPLIED (2026-07-10); Stage 2/3 modules still UNVALIDATED. First real tofu run: 2026-07-10 on the vcloud host (OpenTofu v1.12.3), executing runbooks/dc-dc-phase0-vcloud-prep.md. The root module + the four Stage-1 module types (dc-planes, dc-storage-pool, mesh-link, office1-network) passed tofu init/validate/plan and were apply-ed clean (13 libvirt objects, all MTU 9000). Two fixes were required (DOCFIX-179): per-module required_providers (child modules do NOT inherit provider SOURCE mapping -- without their own versions.tf, OpenTofu infers hashicorp/libvirt, which does not exist), and tofu fmt -recursive. The root provider "maas" block was also deferred to Stage 3 (it forced Stage 1 to supply a sensitive key for zero MAAS resources). STILL UNVALIDATED -- the modules not yet instantiated from root: node-vm, base-image, cloudinit-vm, opnsense-edge, maas-vm-host, netem-link (Stage 2/3). Re-run before trusting those, on a machine with the binary + provider-registry access:

bash scripts/opentofu-validate.sh

Scope of this delivery

Built:

  • modules/dc-planes -- the six per-DC plane segments (D-052/D-100) as isolated libvirt_network resources.
  • modules/mesh-link -- one inter-site L2 segment per leg of the D-100 dark-fiber triangle (DC1<->DC2, DC1<->Office1, DC2<->Office1).
  • modules/dc-storage-pool -- a directory-backed libvirt_pool per DC/Office1 for future node-VM disk images.
  • modules/node-vm (2026-07-09) -- one MAAS-managed OpenStack node VM per call: a BLANK libvirt_volume boot disk + a libvirt_domain with PXE-boot priority on its first network interface. This is the D-103 "OpenTofu creates the node-VM libvirt domains (SHIM)" need for Stage 3 -- deliberately NOT the cloud-init/pre-built-image pattern (see below), since MAAS images these VMs itself after enlistment. NOT instantiated in root main.tf yet: node count/ memory/vcpu/disk sizing is a Phase-0 host/disk-budget decision (buildout-design Section 3) that hasn't been made -- call the module once real values exist, don't invent placeholder specs to wire it in sooner.
  • modules/base-image + modules/cloudinit-vm (2026-07-09) -- the cloud-init/ pre-built-image VM pattern, for Office1's own service VMs (MAAS-region, NetBox, GitBucket). Split into two, matching how examples/alpine_cloudinit.tf itself separates concerns: base-image downloads one shared cloud image ONCE (call it once per distinct image); cloudinit-vm creates a per-VM copy-on-write overlay disk + a libvirt_cloudinit_disk NoCloud seed + the domain (call it once per VM instance, passing module.<base-image-call>.path straight through for the overlay's backing_store.path -- a real attribute reference, not a reconstructed guess). user_data/meta_data/network_config are REQUIRED inputs with no default and no generic fallback -- what Office1's VMs actually need configured is real design work not done yet, and a plausible- looking default (e.g. dhcp4: true) would silently fail anyway since this repo's planes carry no libvirt-managed DHCP. NOT confirmed for OPNsense (FreeBSD-based; cloud-init/NoCloud support is not the same guarantee as for Linux cloud images) -- check that specifically before assuming this module covers OPNsense too. Neither module instantiated in root main.tf yet: no image source has been chosen for any Office1 service VM.
  • modules/opnsense-edge (2026-07-09) -- one OPNsense edge VM per call, using OPNsense's own Configuration Importer (NOT cloud-init -- see the dedicated research section below) via a plain ISO9660 volume containing /conf/config.xml, attached as a secondary cdrom disk -- mechanically identical to cloudinit-vm's seed-volume shape, no libvirt_cloudinit_disk resource involved (wrong format for this). Paired with two new scripts that do the OUT-OF-OPENTOFU preparation work: scripts/opnsense-prep-image.sh (download the nano image, decompress, convert to qcow2, resize) and scripts/opnsense-build-config-iso.sh (build the ISO9660 config-seed from a real config.xml). Both scripts require external tools not present in the session that wrote them (qemu-img; genisoimage/xorriso) -- their harnesses only test the guard-clause paths, not the real download/convert/build behavior; see each script's header. config_iso_path has no default. Not instantiated in root main.tf.
  • templates/opnsense-config.xml.tmpl + scripts/opnsense-render-config.sh (2026-07-09) -- the actual config.xml DESIGN, not just the mechanism: a {{TOKEN}}-parameterized template built from OPNsense's own real, currently-shipped config.xml.sample (confirms the LAN/WAN role is set via each interface block's own <if> device name, not declaration order -- see the "Audit pass" section below) plus the confirmed staticroutes schema, covering system/interfaces/gateway/staticroutes/NAT/DNS/NTP and D-107-shaped firewall rules (default-deny WAN egress except NTP + mirror sync). The renderer is FULLY TESTED end-to-end (needs no external tool, unlike the other opnsense-* scripts) -- 8/8 harness green, including a real bug the harness itself caught: the token HOSTNAME collides with bash's own built-in $HOSTNAME variable (renamed to OPNSENSE_HOSTNAME). Full token legend, what's a real value vs. a still-pending decision (D-100/D-101/D-107, or something only measurable on a real boot), in templates/README.md. No per-site config.xml written yet -- several tokens genuinely cannot be filled until Stage 0 ratifies the decisions they depend on.
  • modules/maas-vm-host (2026-07-09) -- registers the vcloud host's virsh/ libvirt connection with MAAS as a VM host, via the official canonical/maas provider's maas_vm_host resource (D-103: "OpenTofu registers each DC's libvirt host to that DC's MAAS rack controller as a virsh VM-host, so MAAS DISCOVERS the OpenTofu-created node VMs"). Deliberately does NOT use maas_vm_host_machine -- see the dedicated research section below for why that resource is the wrong one (it composes new VMs; D-103 explicitly rules that out). Not instantiated: needs a real MAAS zone/pool and the vcloud host's real power_address.
  • modules/office1-network (2026-07-09) -- closes tooling gap register item #12: Office1's own LOCAL virtual network, the shared L2 segment the three Office1 service VMs (MAAS-region, NetBox, GitBucket), the vcloud host's own management interface, and Office1's own OPNsense edge's LAN side (gap #16) all attach to. Shaped exactly like modules/dc-planes (isolated libvirt_network, no forward/DHCP, attribute-style syntax) but for ONE network, not six planes -- Office1 is a headend, not an OpenStack DC, so the six-plane template does not apply to it. Design decision (resolves gap #12's own "new module vs. host-bridge reuse" fork): a new module, NOT a reused host bridge -- see the module's own header comment for the full reasoning (in short: a hand-configured bridge would be the one network in this topology living outside OpenTofu's D-103-mandated inventory, and would silently break this repo's own virsh net-list --all completeness assumption used as a CHECK step throughout the runbooks). INSTANTIATED for real in root main.tf (needs no value beyond domain_suffix/ underlay_mtu, both already real required inputs elsewhere in this file -- the same reasoning that already lets the three mesh-link legs be instantiated today).
  • modules/opnsense-edge now has a FOURTH intended caller (2026-07-09, closes tooling gap register item #16): Office1 gets its own OPNsense edge alongside DC1/DC2 (the buildout design's "per-site" wording is explicit -- three sites, three edges), and Stage 2's runbook (runbooks/dc-dc-phase1-office1-standup.md) owns creating it, since Stage 2's whole scope is already "Office1 headend standup" and Stage 1 already creates the mesh-link networks this edge's neighbors depend on. A module "office1_opnsense" skeleton is written in root main.tf, commented out -- mirroring the identical NOT-YET-REAL status of DC1/DC2's own edges (neither is instantiated either; see runbooks/dc-dc-phase2-tofu-dc-substrate.md Step 5, which is itself still all placeholders). Its lan_network_name resolves cleanly to module.office1_network.network_name; its wan_network_name does NOT -- see the new gap register item #17 below, a DEEPER, cross-site gap this delivery found rather than invented past.
  • modules/netem-link (2026-07-09) -- applies tc qdisc ... netem WAN-simulation parameters (D-100) to a mesh-link bridge, via a terraform_data resource with local-exec provisioners run over SSH (NOT a bare local command -- OpenTofu runs from Office1, not the vcloud host where the bridges live). See the dedicated research section below. Not instantiated: needs the real bridge name, the vcloud host's SSH target, and the actual netem parameters (still an unruled D-100 decision).
  • Root wiring for DC1 (planes + pool, using DC1's inherited DC0 CIDRs) and Office1 (pool + local network, 2026-07-09), plus all three mesh links, plus a provider "maas" {} block (api_url/api_key as new required variables, maas_api_key marked sensitive = true).

Syntax bug fixed 2026-07-09: dc-planes, mesh-link, and dc-storage-pool originally used classic HCL block syntax (domain { name = ... }) for libvirt_network's domain/mtu and libvirt_pool's target. Building the domain module surfaced real example .tf files from the provider's own repo (examples/domain_with_network.tf, examples/alpine_cloudinit.tf) that consistently use attribute-style nested objects instead (os = { ... }, devices = { ... }) -- this provider has moved to a schema style where nested structures are object-typed attributes, not blocks. All three modules were corrected to match once this was confirmed; see each file's syntax-note comment. This was not previously verifiable from the doc-summarization passes alone (the docs describe fields as "nested blocks" without showing the actual call syntax) -- real example code is what settled it.

Deliberately NOT built yet, and why:

  • DC2's plane networks. D-101 has not assigned DC2's supernet yet (it's an explicit open sub-item). main.tf has the DC2 module block written and commented out, ready to uncomment once NetBox assigns real CIDRs -- do not fill in a guessed default to make it "work" sooner.
  • OPNsense edges -- (a)-(c) DONE 2026-07-09, (d) mechanism now also DONE, only real parameters/instantiation remain: (a) network substrate (modules/mesh-link); (b)+(c) the image AND its config (modules/opnsense-edge + two prep scripts) -- UNVALIDATED (no real boot, no tofu binary, no ISO-building tool available this session); real config.xml content per site is still design work, not yet done, and the module has no fallback for it. (d) tc netem -- the injection MECHANISM is now built (modules/netem-link), but the actual latency/jitter/loss/rate values are still an unruled D-100 open item, so nothing is instantiated. Full breakdown in docs/dc-dc-deployment-workflow.md gap register item 4.
  • Office1's own OPNsense edge -- ownership DECIDED 2026-07-09 (gap #16 CLOSED), instantiation still blocked, same as DC1/DC2's edges. A module "office1_opnsense" skeleton exists in root main.tf, commented out: its memory_mib/vcpu/disk_size_bytes/base_volume_path/config_iso_path have the exact same "no invented specs" blockers as DC1/DC2's own not-yet-instantiated edges. Its LAN side is now resolvable (module.office1_network.network_name), but its WAN side is NOT -- see the new gap register item #17 immediately below, which is NOT Office1-specific: it blocks DC1's and DC2's own wan_network_name values identically (confirmed by reading runbooks/dc-dc-phase2-tofu-dc-substrate.md Step 5's own DC1 template, which carries the identical unresolved placeholder).
  • No dedicated per-site ISP-uplink/WAN-side network exists anywhere in this repo, for ANY site (gap register item #17, NEW 2026-07-09). Every modules/opnsense-edge caller needs a wan_network_name distinct from both the LAN-side network (office1-network / a dc-planes plane) and the three mesh-link legs -- the mesh legs are confirmed MANAGEMENT-TRAFFIC-ONLY by D-100's own sub-item ruling (Office1<->DC fiber = MAAS/Juju/operator), not a simulated ISP uplink, so wiring an edge's WAN side to a mesh leg would misrepresent the topology rather than merely placeholder it. This was found while resolving gap #16 for Office1 and confirmed to be a pre-existing, unresolved gap in Stage 3's own DC1/DC2 wiring too (its own runbook's wan_network_name line is still a literal "confirm which, do not assume" placeholder) -- not invented here for Office1 alone. Needs a decision: a fourth network shape per site (dc-planes/mesh-link/ office1-network-style, one dedicated libvirt_network per site standing in for its "internet" uplink), or some other mechanism not yet considered. Blocks instantiating ANY of the four opnsense-edge calls (dc1_opnsense, dc2_opnsense, office1_opnsense) end-to-end.
  • MAAS VM-host registration -- mechanism DONE (modules/maas-vm-host), not instantiated. Needs a real MAAS zone/pool and the vcloud host's power address; neither exists yet.
  • MAAS-region / NetBox / GitBucket service VMs -- module exists (cloudinit-vm), not instantiated. No image source has been chosen for any of them, and their actual user_data/meta_data/network_config content (what packages, what config, what static address) is real design work that hasn't been done -- required inputs, no invented placeholders.

OPNsense deployment research (2026-07-09) -- the basis for modules/opnsense-edge

Researched directly against OPNsense's own docs/forum/GitHub, not inferred -- confirms modules/cloudinit-vm genuinely does NOT apply, identifies the real mechanism (which turns out to need the same mechanical shape), and is now implemented as modules/opnsense-edge above. Read this before changing that module -- it's the "why," not just background.

  1. Cloud-init is confirmed unreliable on OPNsense, not just unconfirmed. Community consensus on OPNsense's own forum: "FreeBSD cloud-init support, which is not great yet" -- an experienced user's conclusion after investigating it directly. (https://forum.opnsense.org/index.php?topic=42517.0)
  2. The real native mechanism is the Configuration Importer, official docs: runs very early in boot (a 2-3 second window), scans attached volumes for /conf/config.xml. Supported filesystems: GPT, MBR, ZFS pool, msdosfs (FAT). (https://docs.opnsense.org/manual/backups.html, https://docs.opnsense.org/development/backend/autorun.html)
  3. ISO9660 was added to the Importer specifically for VM/cloud automation -- a closed, milestone-targeted, core-developer-assigned GitHub issue requested it explicitly so a config ISO could be attached as a secondary CD-ROM ("shared one to many," vs. a disk image needing a copy per VM). A later bug report confirms ISO9660 detection is live in current code. This means the exact SAME mechanical pattern as modules/cloudinit-vm (a secondary volume attached as a cdrom disk) applies to OPNsense too -- just with a different payload: a plain ISO9660 image containing /conf/config.xml, not a NoCloud-format seed. (https://github.com/opnsense/core/issues/5733, https://github.com/opnsense/core/issues/10017 -- could not pin the exact shipped version from the 22.7 changelog text directly; the feature's existence is corroborated by the later bug report describing its real-world behavior, which is a stronger signal than the milestone tag alone. Confirm on a real boot before relying on it.)
  4. There's also a deeper autorun hook for more advanced provisioning: scripts in /usr/local/etc/rc.syshook.d/import/ can override or extend the default import, running before standard network startup. (https://docs.opnsense.org/development/backend/autorun.html)
  5. Use the nano image for KVM, not the vga/serial/dvd installer images. Nano is pre-installed (serial console pre-configured, auto-expands to fill the disk on first boot) -- the installer images need an interactive install wizard, which defeats automation. Confirmed real-world workflow: download the .img.bz2 -> bunzip2 -> qemu-img convert -f raw -O qcow2 -> qemu-img resize (grow) -> boot via libvirt with a FreeBSD os-variant, serial console, no graphics. (https://nickcharlton.net/posts/installing-opnsense-virt-install-kvm-serial, https://opnsense.org/download/)
  6. A dedicated "build KVM-ready OPNsense images" community project confirms there is no built-in cloud-init/config-injection in official images -- the pattern is always "deploy the immutable base image, then provision separately," matching finding 2 above (seed minimal config via the Importer to get SSH/API reachable, then drive the rest over the network). (https://github.com/maurice-w/opnsense-vm-images)
  7. Network interface convention: interface 1 = LAN (default 192.168.1.1), interface 2 = WAN. Order matters when wiring network_names, the same way modules/node-vm's first-interface-gets-PXE-boot-priority does. (https://docs.opnsense.org/manual/install.html)

Built from this research (2026-07-09): scripts/opnsense-prep-image.sh (step 5's decompress+convert, since create.content.url almost certainly does a plain HTTP fetch with no decompression and can't consume the .img.bz2 directly), scripts/opnsense-build-config-iso.sh (a plain ISO9660 image containing /conf/config.xml, to feed a libvirt_volume's create.content.url the same way libvirt_cloudinit_disk.path feeds cloudinit-vm's seed volume), and modules/opnsense-edge wiring both together. Done as of a follow-on 2026-07-09 delivery: the actual config.xml DESIGN -- see templates/opnsense-config.xml.tmpl + templates/README.md for the full, tested template and token legend. Still not done: the final per-site values for several tokens (pending Stage 0 ratification of D-100/D-101/D-107, or only measurable on a real boot); and real-world verification of the whole chain end-to-end (no tofu binary, no ISO-building tool, and no real OPNsense boot were available this session to confirm any of it, though the template renderer itself is fully tested and needs none of those).

MAAS registration + tc netem research (2026-07-09) -- the basis for modules/maas-vm-host and modules/netem-link

Researched directly against the canonical/terraform-provider-maas provider's own docs and OpenTofu's own official docs, not inferred.

  1. A real, official MAAS provider exists: canonical/maas, v2.7.2 (confirmed from the registry page metadata, published 2026-01-30) -- not a community/unofficial one. Moved from an older maas/maas namespace; use canonical/maas. (https://registry.terraform.io/providers/canonical/maas/latest)
  2. maas_vm_host (register) vs. maas_vm_host_machine (compose) are DIFFERENT operations -- confirmed by reading both resources' real schemas, not just their names. maas_vm_host_machine's arguments (cores, memory, storage_disks, network_interfaces as INPUTS) are MAAS's own "Compose machine" pod feature -- you specify desired specs and MAAS creates a new VM via the virsh connection itself. D-103 explicitly rules this out: "MAAS owns commission / deploy / power / release of those node VMs; it does NOT compose new ones." maas_vm_host (register the virsh chassis; MAAS's own discovery then enlists whatever domains already exist there) is the one that matches D-103's actual described flow -- modules/node-vm pre-creates the VMs, registering the host is what makes MAAS find them. Using vm_host_machine here would have MAAS and OpenTofu fighting over VM creation. (https://raw.githubusercontent.com/canonical/terraform-provider-maas/master/docs/resources/vm_host.md, .../vm_host_machine.md)
  3. power_address confirmed in the exact virsh URI format already used elsewhere in this repo -- the provider's own docs example is "qemu+ssh://172.16.99.2/system", the same shape as this module's own libvirt_uri variable. In practice these should point at the same vcloud host, kept as independent inputs rather than silently assumed identical.
  4. local-exec runs on the machine invoking tofu apply (Office1), NOT on the resource or the vcloud host -- confirmed from OpenTofu's own official docs. This matters directly: D-103 says OpenTofu itself runs from the Office1 operator VM, so a bare local-exec command applying tc qdisc would run on Office1, not touch the vcloud host's bridge interfaces at all. modules/netem-link wraps the command in an explicit SSH hop to the vcloud host because of this. (https://opentofu.org/docs/language/resources/provisioners/local-exec/)
  5. OpenTofu's own docs recommend terraform_data over the older null_resource for exactly the "run a provisioner with no logical resource to attach it to" case -- a built-in resource, no provider needed. Used in modules/netem-link, not null_resource. (https://opentofu.org/docs/language/resources/tf-data/)

Built from this research (2026-07-09): modules/maas-vm-host (registration only, deliberately no composition), modules/netem-link (terraform_data + SSH-wrapped local-exec, with a destroy-time provisioner to remove the qdisc on teardown). Neither instantiated: both need real inputs (a MAAS zone/pool, the vcloud host's SSH target, real netem parameters) that don't exist yet.

Schema notes (read before extending)

Verified this session against the provider's actual current docs (docs/resources/*.md) AND real example .tf files (examples/*.tf) in dmacvicar/terraform-provider-libvirt, tag v0.9.8, fetched 2026-07-09 -- NOT from training-data memory of older provider versions, which would have been wrong (and initially WAS wrong here once, see the syntax-bug note above):

  • Nested structures are attribute-style objects (key = { ... }), everywhere, not classic HCL blocks (key { ... }). Confirmed directly from examples/domain_with_network.tf and examples/alpine_cloudinit.tf (os = { ... }, devices = { ... }, target = { format = { ... } }). This is a provider-wide convention, not resource-specific -- assume it applies to any nested field you haven't personally confirmed otherwise.
  • libvirt_network's isolation control lives at forward.mode (a nested attribute), not a top-level mode argument as older examples show. This scaffold OMITS the forward attribute entirely for isolated planes/links (documented native-libvirt behavior for a private, unforwarded switch) rather than guess a mode = "none" value this session could not confirm is still valid. Re-check this specific choice first if tofu validate objects.
  • libvirt_domain.devices.disks[].source.volume takes { pool = ..., volume = ... } (pool name + volume name, both strings); .target takes { dev = "vda", bus = "virtio" }; .driver takes { type = "qcow2" }. Confirmed directly from examples/alpine_cloudinit.tf.
  • libvirt_domain.devices.interfaces[].source.network takes { network = <libvirt-network-name> } (attach to a libvirt-managed network, what this repo's planes/mesh-links are); .source.bridge takes { bridge = <os-bridge-name> } (attach to a plain OS bridge instead) -- confirmed from examples/domain_with_network.tf. Not used here (planes/mesh-links are libvirt-managed networks), but relevant if OPNsense ends up needing a direct bridge attach to a host NIC.
  • libvirt_volume.target.format.type (nested twice: target = { format = { type = "qcow2" } }) and libvirt_volume.backing_store = { path = ..., format = { type = "qcow2" } } (copy-on-write from another volume's path) -- confirmed from examples/alpine_cloudinit.tf.
  • UNVERIFIED, flagged in modules/node-vm/main.tf: the per-device boot attribute's internal shape (used to set PXE-first boot order). Confirmed to EXIST as a field name on both interfaces[] and disks[] entries (matching native libvirt's per-device <boot order='N'/>), but { order = N } is an inference by pattern, not confirmed from a real example. Check this first with tofu providers schema -json before relying on PXE-first boot working as written.
  • libvirt_pool's target is likewise attribute-style (target = { path = ... }); types like dir are native libvirt storage-pool vocabulary (stable, long-standing, not provider-specific) -- reasonably high confidence.
  • libvirt_cloudinit_disk takes flat string arguments name, user_data, meta_data, network_config (cloud-init YAML content, no nested attributes) and exposes a .path -- feed that .path into a libvirt_volume's create.content.url to register it as an attachable cdrom volume. Confirmed directly from examples/alpine_cloudinit.tf.
  • A base image created in one module call and consumed via backing_store in a DIFFERENT module call: pass the base module's path OUTPUT straight through as a variable to the consuming module (modules/cloudinit-vm does this). Do not reconstruct a volume's path from its pool name + volume name -- libvirt's actual pool-relative path convention was not independently confirmed this session, and a real .path attribute reference is always available and unambiguous instead.
  • create.content.url accepts plain LOCAL filesystem paths, not just http(s) URLs -- inferred from examples/alpine_cloudinit.tf assigning libvirt_cloudinit_disk.alpine_seed.path (a local path, not a URL) directly to create.content.url, and corroborated by a second, independent source during the 2026-07-09 audit pass describing the same volume-creation mechanism as supporting "both local paths (relative to the machine running Terraform) and remote images" via an underlying go-getter-style fetch (a well-known library that treats bare/file:// local paths as one of its supported source schemes alongside http/https). modules/opnsense-edge relies on this for BOTH its base image (scripts/opnsense-prep-image.sh's output path) and its config-seed ISO (scripts/opnsense-build-config-iso.sh's output path) -- neither is fetched from a real URL. Reasonably high confidence now (two independent corroborating sources), though still not seen named explicitly in the provider's own prose docs -- confirm with tofu plan if it errors on a file://-less bare path.
  • maas_vm_host's type argument accepts "virsh" or "lxd" (confirmed from the provider's own docs); power_address is a flat string, no nested attributes; zone/pool are flat optional strings, not objects. Simpler, flatter schema than the libvirt provider's -- confirmed directly from docs/resources/vm_host.md in canonical/terraform-provider-maas.
  • provider "maas" { ... } takes flat top-level arguments (api_url, api_key, api_version [defaults `"2.0"`], installation_method [defaults `"snap"`], tls_ca_cert_path, tls_insecure_skip_verify) -- confirmed from the provider's own docs/index.md, including its exact example block.
  • terraform_data's schema is input (optional, stored + reflected in output) and triggers_replace (optional, forces replacement when changed) -- confirmed from OpenTofu's own official docs, including a complete example using it with a local-exec provisioner keyed on triggers_replace. modules/netem-link uses triggers_replace (not input) since nothing needs to be stored/reflected, just re-triggered when the bridge or netem args change.

Audit pass (2026-07-09) -- re-checked every module's flagged assumptions

Operator asked to pull documentation for anything drafted without it, and review every module for errors/assumptions, after 5 modules had already shipped across 4 changelogs. Went through every "UNVERIFIED"/"inferred"/ "assumed" note in the codebase and did further research rather than leaving them as-is. Outcomes:

  • node-vm's boot = { order = N } shape -- confidence UPGRADED, not resolved outright. Confirmed the provider code-generates its schema as a 1:1 mirror of libvirt's own domain XML (an architectural fact, not documented in the resource pages fetched earlier), and confirmed libvirt's own official docs show the native element is exactly <boot order='N'/> -- one attribute, matching the established single-attribute-element pattern already confirmed elsewhere in this schema (mtu.size, network.domain.name). Comment updated to reflect this is now well-supported reasoning, not a blind guess -- still recommend a real schema check before trusting it.
  • opnsense-edge's LAN/WAN interface-ordering claim -- GENUINE ERROR FOUND AND FIXED, not just re-confirmed. The original comment conflated two separate things: which vtnetN device number a NIC gets (plausibly controlled by devices.interfaces list order -- libvirt's own docs on PCI addressing say auto-assigned addresses "usually match" XML order for a simple topology like this one) and OPNsense's actual LAN/WAN ROLE assignment, which is a SEPARATE, explicit mapping made inside config.xml itself (or the interactive/API assignment step) -- confirmed via OPNsense's own interface-configuration model ("maps logical 'friendly' names... to physical or virtual system devices"). "vtnet0 = WAN" is a convention some setup guides choose, not a FreeBSD/OPNsense-enforced rule. Both main.tf's comments and variables.tf's lan_network_name/ wan_network_name descriptions were rewritten to state this correctly: the variables express INTENT for the network layer, but the real LAN/WAN role still has to be set correctly in config.xml against whichever vtnetN is confirmed (via a real boot) to correspond to which network -- this repo does not yet write that config.xml, so the correction changes documentation accuracy, not current behavior, but would have misled whoever writes it next.
  • create.content.url accepting local paths -- corroborated by a second, independent source (a go-getter-style fetch mechanism explicitly described as supporting local paths). Confidence upgraded; schema notes entry above updated.
  • genisoimage/xorriso flags (-V/-J/-R) -- CONFIRMED standard, correct usage, not just "reasonable." scripts/opnsense-build-config-iso.sh's header comment corrected to separate this (now confirmed) from the genuinely still-open question (whether OPNsense's Importer actually reads the result on a real boot).
  • maas_vm_host's zone/pool optionality -- CONFIRMED safe. Both are computed by MAAS when left unset, validating the default = null design choice as a supported no-op rather than an unhandled edge case.

Not re-litigated: items already backed by a primary, authoritative source the first time (OPNsense's Configuration Importer mechanism itself, the provider's syntax-bug fix, maas_vm_host vs. maas_vm_host_machine) -- re-researching well-sourced findings on every pass would be its own kind of waste. The audit targeted specifically the notes marked UNVERIFIED/inferred/ assumed, since those were the actual gaps.

State file handling (2026-07-10, DOCFIX-175)

Found while reviewing what's still missing from the VR1 buildout, before Stage 1's own first real tofu apply (runbooks/dc-dc-phase0-vcloud-prep.md Step 10 -- DC1 planes + Office1 pool + mesh links; confirmed the actual first apply by reading the runbook directly, not assumed from file naming -- an earlier draft of this section wrongly assumed Stage 3 was first). No backend block is configured anywhere in this module tree (see versions.tf, main.tf -- both silent on it, and every stage's own tofu init runs -backend=false), so state defaults to a local terraform.tfstate file in whatever directory tofu apply is actually run from (per this README's own framing, that's opentofu/ on the Office1 operator VM). At Stage 1 time maas_api_key may still be a placeholder (per that runbook's own "Known gap" section -- MAAS doesn't exist for real until Stage 2), but the state-protection discipline below applies from Stage 1 onward regardless, since the file will carry whatever real credential is exported the moment one is used for any later plan/apply.

The real risk: sensitive = true variables ARE written to state (and plan files) in plaintext -- the flag only redacts CLI/UI output. Verified against HashiCorp's own current documentation (not assumed from general Terraform knowledge, same discipline as everywhere else in this file): "Terraform stores values with the sensitive argument in both state and plan files, and anyone who can access those files can access your sensitive values" (developer.hashicorp.com/terraform/language/manage-sensitive-data, fetched 2026-07-10). This applies uniformly regardless of whether the sensitive value is consumed by a resource argument or, as with var.maas_api_key here, only by a provider {} block -- the docs draw no such distinction, and the only mechanism that actually OMITS a value from state entirely is the separate ephemeral argument, not used anywhere in this repo (versions.tf only sets a >= 1.6.0 floor, not a pin, so whether a given real tofu install even supports ephemeral depends on whatever version is actually installed at apply time -- not verified here, and beside the point, since nothing in this tree uses it either way). Once ANY plan/ apply runs with a real (non-placeholder) TF_VAR_maas_api_key exported -- Stage 1 if the operator happens to have real MAAS credentials that early, more likely Stage 2/3 once MAAS genuinely exists -- terraform.tfstate will contain that live credential in the clear, with no protection beyond filesystem permissions.

Mitigations adopted for VR1 (rehearsal scope -- not a full remote-backend migration, which is a Roosevelt/production-grade decision, not decided here):

  1. Never commit state. .gitignore now excludes opentofu/**/*.tfstate*, .terraform/, *.tfplan, *.auto.tfvars*, and the standard Terraform override/crash-log patterns (DOCFIX-175). This was a real gap: state files were NOT previously excluded (harmless only because tofu has never actually been run in a session that had this clone -- see the SCAFFOLD/UNVALIDATED status at the top of this file).
  2. Never pass secrets via -var or -var-file committed to the repo. Use TF_VAR_maas_api_key (an environment variable, set interactively or from the operator's existing out-of-band secret store -- never a shell history entry). variables.tf's own description now says this too.
  3. Restrict the state file's permissions immediately after the first tofu init/apply on the Office1 operator VM: chmod 600 opentofu/terraform.tfstate (and chmod 700 the containing directory if it isn't already operator-only). Not automated by any script here -- there is no script in this repo that runs tofu yet (tofu itself was never available in any session that authored this tree; see scripts/opentofu-validate.sh), so this is a manual runbook step for whoever executes Stage 1 for real -- wired in as an explicit "SECURE THE STATE FILE" callout in runbooks/dc-dc-phase0-vcloud-prep.md Step 10 (the true first real tofu apply), with a standing reminder to repeat the check after every subsequent stage's apply too (a short pointer added to Stage 3 Step 8 specifically, since that stage is the first to actually need a real MAAS credential).
  4. Back up the state file out-of-band, encrypted, same discipline as ~/vault-init/-class material (see this repo's top-level CLAUDE.md secrets norms) -- losing terraform.tfstate with no backend and no backup means OpenTofu loses track of every resource it created; recovery is either a manual tofu import per resource (tedious, error-prone) or accepting a tofu destroy/rebuild from scratch. Not automated here -- real backup cadence/location is an operator decision once Stage 2 is live, not invented in advance.
  5. No state locking exists on a local backend. Two concurrent tofu apply runs against the same state file can corrupt it -- there is no DynamoDB-style lock table or equivalent here. Low practical risk for VR1 (this repo's own operating discipline is single-operator, one runbook step executed at a time, per CLAUDE.md hard rule 1) but a real gap for Roosevelt: flagging, not deciding -- a real remote backend with locking (S3-compatible + lock table, Terraform Cloud, Consul, etc.) is production-grade infrastructure this rehearsal deliberately does not need yet, and choosing one is a D-NNN-worthy decision for whenever Roosevelt planning starts, not something to invent here.

Not fixed by this delivery (documented, not silently deferred): an encrypted or remote backend and real state locking -- both are production-grade infrastructure decisions for Roosevelt, not invented here (item 5).

Teardown / rollback (2026-07-10, DOCFIX-176)

See runbooks/dc-dc-teardown-rollback.md for how to tofu destroy this tree's resources -- scoped to a single DC/Office1, or the full VR1 layer -- in safe dependency order, including the MAAS-side-first sequencing this repo's D-061 already established the principle for at the juju/MAAS-machine layer. Also covers the "a stage's apply failed partway through" decision tree (fix-forward vs. teardown) that gap register item #19 flagged as missing.

Conventions carried from the rest of this repo

  • No hardcoded repo name (repo-lint L9 would need extending to cover .tf files if this becomes a recurring class of finding -- not yet done, flag if it comes up).
  • Every variable without a safe, already-ratified default requires explicit input (no invented fallbacks) -- same "never use an inferred value" discipline as everywhere else in this repo.
  • DC1's plane CIDRs are a second, hand-maintained copy of scripts/lib-net.sh's PLANE_CIDRS. Keep them in sync manually until a generator exists to derive one from the other (DOCFIX candidate, not actioned).