# opentofu/ -- VR1 (DC-DC) infrastructure-as-code

Owns create/destroy of the "physical" layer per D-103: libvirt domains (node
VMs), all virtual networks (dark-fiber mesh, per-DC planes, ISP-edge), the
OPNsense edge VMs, and the Office1 service VMs. Runs from the Office1 operator
VM against the vcloud host libvirt. See `docs/dc-dc-buildout-design.md` and
`docs/dc-dc-deployment-workflow.md` (Stage 1-2, tooling gap register item 2)
for the surrounding plan. See the skill's
`.claude/skills/openstack-cloud-ops/references/opentofu-provider-docs.md`
for where every provider fact below was sourced from and the fetch
methodology that worked -- read that first if extending any module here.

> ## !!! STATUS RECONCILED 2026-07-16 -- READ THIS FIRST; the per-module prose below predates the D-123 Model B reshape + D-113(a2) and parts are STALE or (config.xml/ISO) DANGEROUS !!!
>
> - **TWO ROOTS now (D-123 Model B).** The DC substrate is split: the OUTER root (this dir,
>   `opentofu/`, vcloud provider) boots + sizes the containment VM `vvr1-dc0` + the transit + the
>   **D-125** vcloud ISP uplink (`vr1-dc0-uplink`); the INNER root (`opentofu/vr1-dc0-substrate/`,
>   provider = `qemu+ssh` to `vvr1-dc0`, R-5) builds the DC fleet INSIDE it -- **9 node VMs** (3+2+4,
>   D-121/R-3), the 6 planes, the OPNsense edge, the inner pool, and `vr1-dc0-wan`. A gated BOOTSTRAP
>   (`scripts/site-headend-install.sh --host-nodes`) sits between. **Both roots `tofu validate`;**
>   `scripts/opentofu-validate.sh` (S3) validates EVERY module standalone. NOT YET APPLIED beyond Stage 1.
> - **Modules ADDED since the "Built" list below was written:** `site-wan` (per-site ISP NAT, D-122)
>   and `wan-bridge` (the D-125 bridge-in WAN segment) -- both are MISSING from that list.
> - **INSTANTIATED (contradicts several "not instantiated" claims below):** `cloudinit-vm` + `base-image`
>   (voffice1, vvr1-dc0, `ubuntu_noble_base`), `node-vm` (9/DC, inner root),
>   `dc-planes`/`site-wan`/`wan-bridge`/`opnsense-edge` (inner root). Node sizing is RULED (D-121/R-3),
>   not "a decision that hasn't been made."
> - **DANGER -- the OPNsense `config.xml` + config-ISO path described below is DELETED; do NOT run it.**
>   `templates/opnsense-config.xml.tmpl`, `scripts/opnsense-build-config-iso.sh`, and
>   `scripts/opnsense-render-config.sh` are GONE (D-112: the Importer can never fire on a nano image;
>   D-113(a2): edge config is the REST API, `scripts/opnsense-api.sh`). A full `config.xml` push would
>   CLOBBER the live Office1 edge's API-managed state. The `opnsense-edge` module dropped
>   `config_iso_path`/`config_seed`.
> - **GitBucket is OUT of scope (D-116)** wherever this doc lists it as an Office1 service VM.
>
> Treat the design-decisions ledger (D-103/D-113/D-116/D-121..D-125) + the actual tree as authoritative
> where the per-module prose below disagrees; that prose is kept for its provider-research value, not as
> current instantiation state.

**Stage-1 history (still true):** first real `tofu` run 2026-07-10 on the vcloud host (the tofu
version of that run is historical; the CURRENT pin lives ONLY in `docs/CURRENT-STATE.md` section 7),
executing `runbooks/dc-dc-phase0-vcloud-prep.md`. The root module + the four Stage-1 module
types (`dc-planes`, `dc-storage-pool`, `mesh-link`, `office1-network`) passed `init`/`validate`/`plan`
and were `apply`-ed clean (13 libvirt objects, all MTU 9000). DOCFIX-179 fixes: per-module
`required_providers` (child modules do NOT inherit provider SOURCE mapping -- without their own
`versions.tf`, OpenTofu infers `hashicorp/libvirt`, which does not exist) + `tofu fmt -recursive`.
Re-validate on a machine with the binary + provider-registry access:

```
bash scripts/opentofu-validate.sh
```

## Scope of this delivery

Built:
- `modules/dc-planes` -- the six per-DC plane segments (D-052/D-100) as
  isolated `libvirt_network` resources.
- `modules/mesh-link` -- one inter-site L2 segment per leg of the D-100
  dark-fiber triangle (DC1<->DC2, DC1<->Office1, DC2<->Office1).
- `modules/dc-storage-pool` -- a directory-backed `libvirt_pool` per DC/Office1
  for future node-VM disk images.
- `modules/node-vm` (2026-07-09) -- one MAAS-managed OpenStack node VM per
  call: a BLANK `libvirt_volume` boot disk + a `libvirt_domain` with PXE-boot
  priority on its first network interface. This is the D-103 "OpenTofu creates
  the node-VM libvirt domains (SHIM)" need for Stage 3 -- deliberately NOT the
  cloud-init/pre-built-image pattern (see below), since MAAS images these VMs
  itself after enlistment. NOT instantiated in root `main.tf` yet: node count/
  memory/vcpu/disk sizing is a Phase-0 host/disk-budget decision
  (buildout-design Section 3) that hasn't been made -- call the module once
  real values exist, don't invent placeholder specs to wire it in sooner.
- `modules/base-image` + `modules/cloudinit-vm` (2026-07-09) -- the cloud-init/
  pre-built-image VM pattern, for Office1's own service VMs (MAAS-region,
  NetBox, GitBucket). Split into two, matching how
  `examples/alpine_cloudinit.tf` itself separates concerns: `base-image`
  downloads one shared cloud image ONCE (call it once per distinct image);
  `cloudinit-vm` creates a per-VM copy-on-write overlay disk + a
  `libvirt_cloudinit_disk` NoCloud seed + the domain (call it once per VM
  instance, passing `module.<base-image-call>.path` straight through for the
  overlay's `backing_store.path` -- a real attribute reference, not a
  reconstructed guess). `user_data`/`meta_data`/`network_config` are REQUIRED
  inputs with no default and no generic fallback -- what Office1's VMs
  actually need configured is real design work not done yet, and a plausible-
  looking default (e.g. `dhcp4: true`) would silently fail anyway since this
  repo's planes carry no libvirt-managed DHCP. **NOT confirmed for OPNsense**
  (FreeBSD-based; cloud-init/NoCloud support is not the same guarantee as for
  Linux cloud images) -- check that specifically before assuming this module
  covers OPNsense too. Neither module instantiated in root `main.tf` yet: no
  image source has been chosen for any Office1 service VM.
- **[SUPERSEDED -- see the STATUS banner: the config-ISO/`config.xml` path below is DELETED and DANGEROUS; the module dropped `config_iso_path`, and edge config is the REST API (D-112/D-113(a2)). The paragraph is kept only for the Importer research.]** `modules/opnsense-edge` (2026-07-09) -- one OPNsense edge VM per call, using
  OPNsense's own Configuration Importer (NOT cloud-init -- see the dedicated
  research section below) via a plain ISO9660 volume containing
  `/conf/config.xml`, attached as a secondary cdrom disk -- mechanically
  identical to `cloudinit-vm`'s seed-volume shape, no `libvirt_cloudinit_disk`
  resource involved (wrong format for this). Paired with two new scripts that
  do the OUT-OF-OPENTOFU preparation work: `scripts/opnsense-prep-image.sh`
  (download the nano image, decompress, convert to qcow2, resize) and
  `scripts/opnsense-build-config-iso.sh` (build the ISO9660 config-seed from
  a real `config.xml`). Both scripts require external tools not present in
  the session that wrote them (`qemu-img`; `genisoimage`/`xorriso`) --
  their harnesses only test the guard-clause paths, not the real
  download/convert/build behavior; see each script's header. `config_iso_path`
  has no default. Not instantiated in root `main.tf`.
- **[DELETED FILES (D-113(a2)) -- `templates/opnsense-config.xml.tmpl` and `scripts/opnsense-render-config.sh` no longer exist; retained below as historical design notes only. Do NOT render/push a full config.xml -- it clobbers the live edge.]** `templates/opnsense-config.xml.tmpl` + `scripts/opnsense-render-config.sh`
  (2026-07-09) -- the actual `config.xml` DESIGN, not just the mechanism:
  a `{{TOKEN}}`-parameterized template built from OPNsense's own real,
  currently-shipped `config.xml.sample` (confirms the LAN/WAN role is set
  via each interface block's own `<if>` device name, not declaration order
  -- see the "Audit pass" section below) plus the confirmed `staticroutes`
  schema, covering system/interfaces/gateway/staticroutes/NAT/DNS/NTP and
  D-107-shaped firewall rules (default-deny WAN egress except NTP + mirror
  sync). The renderer is FULLY TESTED end-to-end (needs no external tool,
  unlike the other opnsense-* scripts) -- 8/8 harness green, including a
  real bug the harness itself caught: the token `HOSTNAME` collides with
  bash's own built-in `$HOSTNAME` variable (renamed to
  `OPNSENSE_HOSTNAME`). Full token legend, what's a real value vs. a
  still-pending decision (D-100/D-101/D-107, or something only measurable
  on a real boot), in `templates/README.md`. No per-site config.xml written
  yet -- several tokens genuinely cannot be filled until Stage 0 ratifies
  the decisions they depend on.
- `modules/maas-vm-host` (2026-07-09) -- registers the vcloud host's virsh/
  libvirt connection with MAAS as a VM host, via the official `canonical/maas`
  provider's `maas_vm_host` resource (D-103: "OpenTofu registers each DC's
  libvirt host to that DC's MAAS rack controller as a virsh VM-host, so MAAS
  DISCOVERS the OpenTofu-created node VMs"). Deliberately does NOT use
  `maas_vm_host_machine` -- see the dedicated research section below for why
  that resource is the wrong one (it composes new VMs; D-103 explicitly rules
  that out). Not instantiated: needs a real MAAS zone/pool and the vcloud
  host's real power_address.
- `modules/office1-network` (2026-07-09) -- closes tooling gap register item
  #12: Office1's own LOCAL virtual network, the shared L2 segment the three
  Office1 service VMs (MAAS-region, NetBox, GitBucket), the vcloud host's own
  management interface, and Office1's own OPNsense edge's LAN side (gap #16)
  all attach to. Shaped exactly like `modules/dc-planes` (isolated
  `libvirt_network`, no `forward`/DHCP, attribute-style syntax) but for ONE
  network, not six planes -- Office1 is a headend, not an OpenStack DC, so
  the six-plane template does not apply to it. **Design decision (resolves
  gap #12's own "new module vs. host-bridge reuse" fork): a new module, NOT a
  reused host bridge** -- see the module's own header comment for the full
  reasoning (in short: a hand-configured bridge would be the one network in
  this topology living outside OpenTofu's D-103-mandated inventory, and would
  silently break this repo's own `virsh net-list --all` completeness
  assumption used as a CHECK step throughout the runbooks). INSTANTIATED for
  real in root `main.tf` (needs no value beyond `domain_suffix`/
  `underlay_mtu`, both already real required inputs elsewhere in this file --
  the same reasoning that already lets the three mesh-link legs be
  instantiated today).
- `modules/opnsense-edge` now has a FOURTH intended caller (2026-07-09,
  closes tooling gap register item #16): Office1 gets its own OPNsense edge
  alongside DC1/DC2 (the buildout design's "per-site" wording is explicit --
  three sites, three edges), and Stage 2's runbook
  (`runbooks/dc-dc-phase1-office1-standup.md`) owns creating it, since Stage
  2's whole scope is already "Office1 headend standup" and Stage 1 already
  creates the mesh-link networks this edge's neighbors depend on. A `module
  "office1_opnsense"` skeleton is written in root `main.tf`, commented out --
  mirroring the identical NOT-YET-REAL status of DC1/DC2's own edges (neither
  is instantiated either; see `runbooks/dc-dc-phase2-tofu-dc-substrate.md`
  Step 5, which is itself still all placeholders). Its `lan_network_name`
  resolves cleanly to `module.office1_network.network_name`; its
  `wan_network_name` does NOT -- see the new gap register item #17 below,
  a DEEPER, cross-site gap this delivery found rather than invented past.
- `modules/netem-link` (2026-07-09) -- applies `tc qdisc ... netem`
  WAN-simulation parameters (D-100) to a `mesh-link` bridge, via a
  `terraform_data` resource with `local-exec` provisioners run over SSH (NOT
  a bare local command -- OpenTofu runs from Office1, not the vcloud host
  where the bridges live). See the dedicated research section below. Not
  instantiated: needs the real bridge name, the vcloud host's SSH target,
  and the actual netem parameters (still an unruled D-100 decision).
- Root wiring for DC1 (planes + pool, using DC1's inherited DC0 CIDRs) and
  Office1 (pool + local network, 2026-07-09), plus all three mesh links, plus
  a `provider "maas" {}` block (api_url/api_key as new required variables,
  `maas_api_key` marked `sensitive = true`).

**Syntax bug fixed 2026-07-09:** `dc-planes`, `mesh-link`, and `dc-storage-pool`
originally used classic HCL block syntax (`domain { name = ... }`) for
`libvirt_network`'s `domain`/`mtu` and `libvirt_pool`'s `target`. Building the
domain module surfaced real example `.tf` files from the provider's own repo
(`examples/domain_with_network.tf`, `examples/alpine_cloudinit.tf`) that
consistently use attribute-style nested objects instead (`os = { ... }`,
`devices = { ... }`) -- this provider has moved to a schema style where nested
structures are object-typed attributes, not blocks. All three modules were
corrected to match once this was confirmed; see each file's syntax-note
comment. This was not previously verifiable from the doc-summarization passes
alone (the docs describe fields as "nested blocks" without showing the actual
call syntax) -- real example code is what settled it.

Deliberately NOT built yet, and why:
- **DC2's plane networks.** D-101 has not assigned DC2's supernet yet (it's an
  explicit open sub-item). `main.tf` has the DC2 module block written and
  commented out, ready to uncomment once NetBox assigns real CIDRs -- do not
  fill in a guessed default to make it "work" sooner.
- **OPNsense edges -- (a)-(c) DONE 2026-07-09, (d) mechanism now also DONE,
  only real parameters/instantiation remain:** (a) network substrate
  (`modules/mesh-link`); (b)+(c) the image AND its config
  (`modules/opnsense-edge` + two prep scripts) -- UNVALIDATED (no real boot,
  no `tofu` binary, no ISO-building tool available this session); real
  `config.xml` content per site is still design work, not yet done, and the
  module has no fallback for it. (d) `tc netem` -- the injection MECHANISM is
  now built (`modules/netem-link`), but the actual latency/jitter/loss/rate
  values are still an unruled D-100 open item, so nothing is instantiated.
  Full breakdown in `docs/dc-dc-deployment-workflow.md` gap register item 4.
- **Office1's own OPNsense edge -- ownership DECIDED 2026-07-09 (gap #16
  CLOSED), instantiation still blocked, same as DC1/DC2's edges.** A `module
  "office1_opnsense"` skeleton exists in root `main.tf`, commented out: its
  `memory_mib`/`vcpu`/`disk_size_bytes`/`base_volume_path`/`config_iso_path`
  have the exact same "no invented specs" blockers as DC1/DC2's own
  not-yet-instantiated edges. Its LAN side is now resolvable
  (`module.office1_network.network_name`), but its WAN side is NOT -- see
  the new gap register item #17 immediately below, which is NOT
  Office1-specific: it blocks DC1's and DC2's own `wan_network_name` values
  identically (confirmed by reading `runbooks/dc-dc-phase2-tofu-dc-substrate.md`
  Step 5's own DC1 template, which carries the identical unresolved
  placeholder).
- **No dedicated per-site ISP-uplink/WAN-side network exists anywhere in
  this repo, for ANY site (gap register item #17, NEW 2026-07-09).** Every
  `modules/opnsense-edge` caller needs a `wan_network_name` distinct from
  both the LAN-side network (office1-network / a dc-planes plane) and the
  three mesh-link legs -- the mesh legs are confirmed MANAGEMENT-TRAFFIC-ONLY
  by D-100's own sub-item ruling (Office1<->DC fiber = MAAS/Juju/operator),
  not a simulated ISP uplink, so wiring an edge's WAN side to a mesh leg
  would misrepresent the topology rather than merely placeholder it. This
  was found while resolving gap #16 for Office1 and confirmed to be a
  pre-existing, unresolved gap in Stage 3's own DC1/DC2 wiring too (its own
  runbook's `wan_network_name` line is still a literal "confirm which, do
  not assume" placeholder) -- not invented here for Office1 alone. Needs a
  decision: a fourth network shape per site (dc-planes/mesh-link/
  office1-network-style, one dedicated `libvirt_network` per site standing
  in for its "internet" uplink), or some other mechanism not yet considered.
  Blocks instantiating ANY of the four `opnsense-edge` calls
  (`dc1_opnsense`, `dc2_opnsense`, `office1_opnsense`) end-to-end.
- **MAAS VM-host registration -- mechanism DONE (`modules/maas-vm-host`), not
  instantiated.** Needs a real MAAS zone/pool and the vcloud host's power
  address; neither exists yet.
- **MAAS-region / NetBox / GitBucket service VMs -- module exists
  (`cloudinit-vm`), not instantiated.** No image source has been chosen for
  any of them, and their actual `user_data`/`meta_data`/`network_config`
  content (what packages, what config, what static address) is real design
  work that hasn't been done -- required inputs, no invented placeholders.

## OPNsense deployment research (2026-07-09) -- the basis for `modules/opnsense-edge`

Researched directly against OPNsense's own docs/forum/GitHub, not inferred --
confirms `modules/cloudinit-vm` genuinely does NOT apply, identifies the real
mechanism (which turns out to need the same mechanical shape), and is now
implemented as `modules/opnsense-edge` above. Read this before changing that
module -- it's the "why," not just background.

1. **Cloud-init is confirmed unreliable on OPNsense, not just unconfirmed.**
   Community consensus on OPNsense's own forum: "FreeBSD cloud-init support,
   which is not great yet" -- an experienced user's conclusion after
   investigating it directly.
   (https://forum.opnsense.org/index.php?topic=42517.0)
2. **The real native mechanism is the Configuration Importer**, official docs:
   runs very early in boot (a 2-3 second window), scans attached volumes for
   `/conf/config.xml`. Supported filesystems: GPT, MBR, ZFS pool, msdosfs
   (FAT). (https://docs.opnsense.org/manual/backups.html,
   https://docs.opnsense.org/development/backend/autorun.html)
3. **ISO9660 was added to the Importer specifically for VM/cloud automation**
   -- a closed, milestone-targeted, core-developer-assigned GitHub issue
   requested it explicitly so a config ISO could be attached as a secondary
   CD-ROM ("shared one to many," vs. a disk image needing a copy per VM).
   A later bug report confirms ISO9660 detection is live in current code.
   This means the exact SAME mechanical pattern as `modules/cloudinit-vm`
   (a secondary volume attached as a `cdrom` disk) applies to OPNsense too --
   just with a different payload: a plain ISO9660 image containing
   `/conf/config.xml`, not a NoCloud-format seed.
   (https://github.com/opnsense/core/issues/5733,
   https://github.com/opnsense/core/issues/10017 -- could not pin the exact
   shipped version from the 22.7 changelog text directly; the feature's
   existence is corroborated by the later bug report describing its
   real-world behavior, which is a stronger signal than the milestone tag
   alone. Confirm on a real boot before relying on it.)
4. **There's also a deeper `autorun` hook** for more advanced provisioning:
   scripts in `/usr/local/etc/rc.syshook.d/import/` can override or extend the
   default import, running before standard network startup.
   (https://docs.opnsense.org/development/backend/autorun.html)
5. **Use the nano image for KVM, not the vga/serial/dvd installer images.**
   Nano is pre-installed (serial console pre-configured, auto-expands to fill
   the disk on first boot) -- the installer images need an interactive
   install wizard, which defeats automation. Confirmed real-world workflow:
   download the `.img.bz2` -> `bunzip2` -> `qemu-img convert -f raw -O qcow2`
   -> `qemu-img resize` (grow) -> boot via libvirt with a FreeBSD os-variant,
   serial console, no graphics.
   (https://nickcharlton.net/posts/installing-opnsense-virt-install-kvm-serial,
   https://opnsense.org/download/)
6. **A dedicated "build KVM-ready OPNsense images" community project confirms
   there is no built-in cloud-init/config-injection in official images** --
   the pattern is always "deploy the immutable base image, then provision
   separately," matching finding 2 above (seed minimal config via the
   Importer to get SSH/API reachable, then drive the rest over the network).
   (https://github.com/maurice-w/opnsense-vm-images)
7. **Network interface convention: interface 1 = LAN (default 192.168.1.1),
   interface 2 = WAN.** Order matters when wiring `network_names`, the same
   way `modules/node-vm`'s first-interface-gets-PXE-boot-priority does.
   (https://docs.opnsense.org/manual/install.html)

**Built from this research (2026-07-09):** `scripts/opnsense-prep-image.sh`
(step 5's decompress+convert, since `create.content.url` almost certainly
does a plain HTTP fetch with no decompression and can't consume the
`.img.bz2` directly), `scripts/opnsense-build-config-iso.sh` (a plain ISO9660
image containing `/conf/config.xml`, to feed a `libvirt_volume`'s
`create.content.url` the same way `libvirt_cloudinit_disk.path` feeds
`cloudinit-vm`'s seed volume), and `modules/opnsense-edge` wiring both
together. **Done as of a follow-on 2026-07-09 delivery:** the actual
`config.xml` DESIGN -- see `templates/opnsense-config.xml.tmpl` +
`templates/README.md` for the full, tested template and token legend.
**Still not done:** the final per-site values for several tokens (pending
Stage 0 ratification of D-100/D-101/D-107, or only measurable on a real
boot); and real-world verification of the whole chain end-to-end (no `tofu`
binary, no ISO-building tool, and no real OPNsense boot were available this
session to confirm any of it, though the template renderer itself is fully
tested and needs none of those).

## MAAS registration + tc netem research (2026-07-09) -- the basis for `modules/maas-vm-host` and `modules/netem-link`

Researched directly against the `canonical/terraform-provider-maas` provider's
own docs and OpenTofu's own official docs, not inferred.

1. **A real, official MAAS provider exists: `canonical/maas`**, v2.7.2
   (confirmed from the registry page metadata, published 2026-01-30) -- not
   a community/unofficial one. Moved from an older `maas/maas` namespace;
   use `canonical/maas`.
   (https://registry.terraform.io/providers/canonical/maas/latest)
2. **`maas_vm_host` (register) vs. `maas_vm_host_machine` (compose) are
   DIFFERENT operations -- confirmed by reading both resources' real
   schemas, not just their names.** `maas_vm_host_machine`'s arguments
   (`cores`, `memory`, `storage_disks`, `network_interfaces` as INPUTS) are
   MAAS's own "Compose machine" pod feature -- you specify desired specs and
   MAAS creates a new VM via the virsh connection itself. D-103 explicitly
   rules this out: "MAAS owns commission / deploy / power / release of those
   node VMs; it does NOT compose new ones." `maas_vm_host` (register the
   virsh chassis; MAAS's own discovery then enlists whatever domains already
   exist there) is the one that matches D-103's actual described flow --
   `modules/node-vm` pre-creates the VMs, registering the host is what makes
   MAAS find them. Using `vm_host_machine` here would have MAAS and OpenTofu
   fighting over VM creation.
   (https://raw.githubusercontent.com/canonical/terraform-provider-maas/master/docs/resources/vm_host.md,
   .../vm_host_machine.md)
3. **`power_address` confirmed in the exact virsh URI format already used
   elsewhere in this repo** -- the provider's own docs example is
   `"qemu+ssh://172.16.99.2/system"`, the same shape as this module's own
   `libvirt_uri` variable. In practice these should point at the same vcloud
   host, kept as independent inputs rather than silently assumed identical.
4. **`local-exec` runs on the machine invoking `tofu apply` (Office1), NOT on
   the resource or the vcloud host** -- confirmed from OpenTofu's own
   official docs. This matters directly: D-103 says OpenTofu itself runs
   from the Office1 operator VM, so a bare `local-exec` command applying
   `tc qdisc` would run on Office1, not touch the vcloud host's bridge
   interfaces at all. `modules/netem-link` wraps the command in an explicit
   SSH hop to the vcloud host because of this.
   (https://opentofu.org/docs/language/resources/provisioners/local-exec/)
5. **OpenTofu's own docs recommend `terraform_data` over the older
   `null_resource`** for exactly the "run a provisioner with no logical
   resource to attach it to" case -- a built-in resource, no provider needed.
   Used in `modules/netem-link`, not `null_resource`.
   (https://opentofu.org/docs/language/resources/tf-data/)

**Built from this research (2026-07-09):** `modules/maas-vm-host`
(registration only, deliberately no composition), `modules/netem-link`
(`terraform_data` + SSH-wrapped `local-exec`, with a destroy-time
provisioner to remove the qdisc on teardown). Neither instantiated: both
need real inputs (a MAAS zone/pool, the vcloud host's SSH target, real
netem parameters) that don't exist yet.

## Schema notes (read before extending)

Verified this session against the provider's actual current docs
(`docs/resources/*.md`) AND real example `.tf` files (`examples/*.tf`) in
`dmacvicar/terraform-provider-libvirt`, tag `v0.9.8`, fetched 2026-07-09 --
NOT from training-data memory of older provider versions, which would have
been wrong (and initially WAS wrong here once, see the syntax-bug note above):

- **Nested structures are attribute-style objects (`key = { ... }`),
  everywhere, not classic HCL blocks (`key { ... }`).** Confirmed directly
  from `examples/domain_with_network.tf` and `examples/alpine_cloudinit.tf`
  (`os = { ... }`, `devices = { ... }`, `target = { format = { ... } }`).
  This is a provider-wide convention, not resource-specific -- assume it
  applies to any nested field you haven't personally confirmed otherwise.
- `libvirt_network`'s isolation control lives at `forward.mode` (a nested
  attribute), not a top-level `mode` argument as older examples show. This
  scaffold OMITS the `forward` attribute entirely for isolated planes/links
  (documented native-libvirt behavior for a private, unforwarded switch)
  rather than guess a `mode = "none"` value this session could not confirm is
  still valid. Re-check this specific choice first if `tofu validate` objects.
- `libvirt_domain.devices.disks[].source.volume` takes `{ pool = ..., volume =
  ... }` (pool name + volume name, both strings); `.target` takes `{ dev =
  "vda", bus = "virtio" }`; `.driver` takes `{ type = "qcow2" }`. Confirmed
  directly from `examples/alpine_cloudinit.tf`.
- `libvirt_domain.devices.interfaces[].source.network` takes `{ network =
  <libvirt-network-name> }` (attach to a libvirt-managed network, what this
  repo's planes/mesh-links are); `.source.bridge` takes `{ bridge =
  <os-bridge-name> }` (attach to a plain OS bridge instead) -- confirmed from
  `examples/domain_with_network.tf`. Not used here (planes/mesh-links are
  libvirt-managed networks), but relevant if OPNsense ends up needing a direct
  bridge attach to a host NIC.
- `libvirt_volume.target.format.type` (nested twice: `target = { format = {
  type = "qcow2" } }`) and `libvirt_volume.backing_store = { path = ...,
  format = { type = "qcow2" } }` (copy-on-write from another volume's path)
  -- confirmed from `examples/alpine_cloudinit.tf`.
- **UNVERIFIED, flagged in `modules/node-vm/main.tf`:** the per-device `boot`
  attribute's internal shape (used to set PXE-first boot order). Confirmed to
  EXIST as a field name on both `interfaces[]` and `disks[]` entries (matching
  native libvirt's per-device `<boot order='N'/>`), but `{ order = N }` is an
  inference by pattern, not confirmed from a real example. Check this first
  with `tofu providers schema -json` before relying on PXE-first boot working
  as written.
- `libvirt_pool`'s `target` is likewise attribute-style (`target = { path =
  ... }`); types like `dir` are native libvirt storage-pool vocabulary
  (stable, long-standing, not provider-specific) -- reasonably high
  confidence.
- `libvirt_cloudinit_disk` takes flat string arguments `name`, `user_data`,
  `meta_data`, `network_config` (cloud-init YAML content, no nested
  attributes) and exposes a `.path` -- feed that `.path` into a
  `libvirt_volume`'s `create.content.url` to register it as an attachable
  cdrom volume. Confirmed directly from `examples/alpine_cloudinit.tf`.
- A base image created in one module call and consumed via `backing_store` in
  a DIFFERENT module call: pass the base module's `path` OUTPUT straight
  through as a variable to the consuming module (`modules/cloudinit-vm` does
  this). Do not reconstruct a volume's path from its pool name + volume name
  -- libvirt's actual pool-relative path convention was not independently
  confirmed this session, and a real `.path` attribute reference is always
  available and unambiguous instead.
- `create.content.url` accepts plain LOCAL filesystem paths, not just http(s)
  URLs -- inferred from `examples/alpine_cloudinit.tf` assigning
  `libvirt_cloudinit_disk.alpine_seed.path` (a local path, not a URL)
  directly to `create.content.url`, and corroborated by a second,
  independent source during the 2026-07-09 audit pass describing the same
  volume-creation mechanism as supporting "both local paths (relative to the
  machine running Terraform) and remote images" via an underlying
  go-getter-style fetch (a well-known library that treats bare/`file://`
  local paths as one of its supported source schemes alongside http/https).
  `modules/opnsense-edge` relies on this for BOTH its base image
  (`scripts/opnsense-prep-image.sh`'s output path) and its config-seed ISO
  (`scripts/opnsense-build-config-iso.sh`'s output path) -- neither is
  fetched from a real URL. Reasonably high confidence now (two independent
  corroborating sources), though still not seen named explicitly in the
  provider's own prose docs -- confirm with `tofu plan` if it errors on a
  `file://`-less bare path.
- `maas_vm_host`'s `type` argument accepts `"virsh"` or `"lxd"` (confirmed
  from the provider's own docs); `power_address` is a flat string, no nested
  attributes; `zone`/`pool` are flat optional strings, not objects. Simpler,
  flatter schema than the libvirt provider's -- confirmed directly from
  `docs/resources/vm_host.md` in `canonical/terraform-provider-maas`.
- `provider "maas" { ... }` takes flat top-level arguments (`api_url`,
  `api_key`, `api_version` [defaults `"2.0"`], `installation_method`
  [defaults `"snap"`], `tls_ca_cert_path`, `tls_insecure_skip_verify`) --
  confirmed from the provider's own `docs/index.md`, including its exact
  example block.
- `terraform_data`'s schema is `input` (optional, stored + reflected in
  `output`) and `triggers_replace` (optional, forces replacement when
  changed) -- confirmed from OpenTofu's own official docs, including a
  complete example using it with a `local-exec` provisioner keyed on
  `triggers_replace`. `modules/netem-link` uses `triggers_replace` (not
  `input`) since nothing needs to be stored/reflected, just re-triggered
  when the bridge or netem args change.

## Audit pass (2026-07-09) -- re-checked every module's flagged assumptions

Operator asked to pull documentation for anything drafted without it, and
review every module for errors/assumptions, after 5 modules had already
shipped across 4 changelogs. Went through every "UNVERIFIED"/"inferred"/
"assumed" note in the codebase and did further research rather than leaving
them as-is. Outcomes:

- **`node-vm`'s `boot = { order = N }` shape -- confidence UPGRADED, not
  resolved outright.** Confirmed the provider code-generates its schema as a
  1:1 mirror of libvirt's own domain XML (an architectural fact, not
  documented in the resource pages fetched earlier), and confirmed libvirt's
  own official docs show the native element is exactly `<boot order='N'/>`
  -- one attribute, matching the established single-attribute-element
  pattern already confirmed elsewhere in this schema (`mtu.size`,
  `network.domain.name`). Comment updated to reflect this is now
  well-supported reasoning, not a blind guess -- still recommend a real
  schema check before trusting it.
- **`opnsense-edge`'s LAN/WAN interface-ordering claim -- GENUINE ERROR
  FOUND AND FIXED, not just re-confirmed.** The original comment conflated
  two separate things: which `vtnetN` device number a NIC gets (plausibly
  controlled by `devices.interfaces` list order -- libvirt's own docs on PCI
  addressing say auto-assigned addresses "usually match" XML order for a
  simple topology like this one) and OPNsense's actual LAN/WAN ROLE
  assignment, which is a SEPARATE, explicit mapping made inside `config.xml`
  itself (or the interactive/API assignment step) -- confirmed via OPNsense's
  own interface-configuration model ("maps logical 'friendly' names... to
  physical or virtual system devices"). "`vtnet0` = WAN" is a convention
  some setup guides choose, not a FreeBSD/OPNsense-enforced rule. Both
  `main.tf`'s comments and `variables.tf`'s `lan_network_name`/
  `wan_network_name` descriptions were rewritten to state this correctly:
  the variables express INTENT for the network layer, but the real LAN/WAN
  role still has to be set correctly in `config.xml` against whichever
  vtnetN is confirmed (via a real boot) to correspond to which network --
  this repo does not yet write that config.xml, so the correction changes
  documentation accuracy, not current behavior, but would have misled
  whoever writes it next.
- **`create.content.url` accepting local paths -- corroborated by a second,
  independent source** (a go-getter-style fetch mechanism explicitly
  described as supporting local paths). Confidence upgraded; schema notes
  entry above updated.
- **`genisoimage`/`xorriso` flags (`-V`/`-J`/`-R`) -- CONFIRMED standard,
  correct usage**, not just "reasonable." `scripts/opnsense-build-config-iso.sh`'s
  header comment corrected to separate this (now confirmed) from the
  genuinely still-open question (whether OPNsense's Importer actually reads
  the result on a real boot).
- **`maas_vm_host`'s `zone`/`pool` optionality -- CONFIRMED safe.** Both are
  computed by MAAS when left unset, validating the `default = null` design
  choice as a supported no-op rather than an unhandled edge case.

**Not re-litigated:** items already backed by a primary, authoritative
source the first time (OPNsense's Configuration Importer mechanism itself,
the provider's syntax-bug fix, `maas_vm_host` vs. `maas_vm_host_machine`) --
re-researching well-sourced findings on every pass would be its own kind of
waste. The audit targeted specifically the notes marked UNVERIFIED/inferred/
assumed, since those were the actual gaps.

## State file handling (2026-07-10, DOCFIX-175)

Found while reviewing what's still missing from the VR1 buildout, before
Stage 1's own first real `tofu apply` (`runbooks/dc-dc-phase0-vcloud-prep.md`
Step 10 -- DC1 planes + Office1 pool + mesh links; confirmed the actual
first apply by reading the runbook directly, not assumed from file naming --
an earlier draft of this section wrongly assumed Stage 3 was first). No
`backend` block is configured anywhere in this module tree (see
`versions.tf`, `main.tf` -- both silent on it, and every stage's own `tofu
init` runs `-backend=false`), so state defaults to a local
`terraform.tfstate` file in whatever directory `tofu apply` is actually run
from (per this README's own framing, that's `opentofu/` on the Office1
operator VM). At Stage 1 time `maas_api_key` may still be a placeholder (per
that runbook's own "Known gap" section -- MAAS doesn't exist for real until
Stage 2), but the state-protection discipline below applies from Stage 1
onward regardless, since the file will carry whatever real credential is
exported the moment one is used for any later plan/apply.

**The real risk: `sensitive = true` variables ARE written to state (and plan
files) in plaintext -- the flag only redacts CLI/UI output.** Verified
against HashiCorp's own current documentation (not assumed from general
Terraform knowledge, same discipline as everywhere else in this file):
*"Terraform stores values with the `sensitive` argument in both state and
plan files, and anyone who can access those files can access your sensitive
values"* (developer.hashicorp.com/terraform/language/manage-sensitive-data,
fetched 2026-07-10). This applies uniformly regardless of whether the
sensitive value is consumed by a resource argument or, as with
`var.maas_api_key` here, only by a `provider {}` block -- the docs draw no
such distinction, and the only mechanism that actually OMITS a value from
state entirely is the separate `ephemeral` argument, not used anywhere in
this repo (`versions.tf` only sets a `>= 1.6.0` floor, not a pin, so whether
a given real `tofu` install even supports `ephemeral` depends on whatever
version is actually installed at apply time -- not verified here, and beside
the point, since nothing in this tree uses it either way). Once ANY plan/
apply runs with a real (non-placeholder) `TF_VAR_maas_api_key` exported --
Stage 1 if the operator happens to have real MAAS credentials that early,
more likely Stage 2/3 once MAAS genuinely exists -- `terraform.tfstate` will
contain that live credential in the clear, with no protection beyond
filesystem permissions.

**Mitigations adopted for VR1 (rehearsal scope -- not a full remote-backend
migration, which is a Roosevelt/production-grade decision, not decided
here):**

1. **Never commit state.** `.gitignore` now excludes
   `opentofu/**/*.tfstate*`, `.terraform/`, `*.tfplan`, `*.auto.tfvars*`, and
   the standard Terraform override/crash-log patterns (DOCFIX-175). This was
   a real gap: state files were NOT previously excluded (harmless only
   because `tofu` has never actually been run in a session that had this
   clone -- see the SCAFFOLD/UNVALIDATED status at the top of this file).
2. **Never pass secrets via `-var` or `-var-file` committed to the repo.**
   Use `TF_VAR_maas_api_key` (an environment variable, set interactively or
   from the operator's existing out-of-band secret store -- never a shell
   history entry). `variables.tf`'s own description now says this too.
3. **Restrict the state file's permissions immediately after the first
   `tofu init`/`apply`** on the Office1 operator VM: `chmod 600
   opentofu/terraform.tfstate` (and `chmod 700` the containing directory if
   it isn't already operator-only). Not automated by any script here --
   there is no script in this repo that runs `tofu` yet (`tofu` itself was
   never available in any session that authored this tree; see
   `scripts/opentofu-validate.sh`), so this is a manual runbook step for
   whoever executes Stage 1 for real -- wired in as an explicit "SECURE THE
   STATE FILE" callout in `runbooks/dc-dc-phase0-vcloud-prep.md` Step 10
   (the true first real `tofu apply`), with a standing reminder to repeat the
   check after every subsequent stage's apply too (a short pointer added to
   Stage 3 Step 8 specifically, since that stage is the first to actually
   need a real MAAS credential).
4. **Back up the state file out-of-band, encrypted, same discipline as
   `~/vault-init/`-class material** (see this repo's top-level `CLAUDE.md`
   secrets norms) -- losing `terraform.tfstate` with no backend and no
   backup means OpenTofu loses track of every resource it created; recovery
   is either a manual `tofu import` per resource (tedious, error-prone) or
   accepting a `tofu destroy`/rebuild from scratch. Not automated here --
   real backup cadence/location is an operator decision once Stage 2 is
   live, not invented in advance.
5. **No state locking exists on a local backend.** Two concurrent
   `tofu apply` runs against the same state file can corrupt it -- there is
   no `DynamoDB`-style lock table or equivalent here. Low practical risk for
   VR1 (this repo's own operating discipline is single-operator, one
   runbook step executed at a time, per `CLAUDE.md` hard rule 1) but a real
   gap for Roosevelt: **flagging, not deciding** -- a real remote backend
   with locking (S3-compatible + lock table, Terraform Cloud, Consul, etc.)
   is production-grade infrastructure this rehearsal deliberately does not
   need yet, and choosing one is a D-NNN-worthy decision for whenever
   Roosevelt planning starts, not something to invent here.

**Not fixed by this delivery (documented, not silently deferred):** an
encrypted or remote backend and real state locking -- both are
production-grade infrastructure decisions for Roosevelt, not invented here
(item 5).

## Teardown / rollback (2026-07-10, DOCFIX-176)

See `runbooks/dc-dc-teardown-rollback.md` for how to `tofu destroy` this
tree's resources -- scoped to a single DC/Office1, or the full VR1 layer --
in safe dependency order, including the MAAS-side-first sequencing this
repo's D-061 already established the principle for at the juju/MAAS-machine
layer. Also covers the "a stage's apply failed partway through" decision
tree (fix-forward vs. teardown) that gap register item #19 flagged as
missing.

## Conventions carried from the rest of this repo

- No hardcoded repo name (repo-lint L9 would need extending to cover `.tf`
  files if this becomes a recurring class of finding -- not yet done, flag if
  it comes up).
- Every variable without a safe, already-ratified default requires explicit
  input (no invented fallbacks) -- same "never use an inferred value"
  discipline as everywhere else in this repo.
- DC1's plane CIDRs are a second, hand-maintained copy of
  `scripts/lib-net.sh`'s `PLANE_CIDRS`. Keep them in sync manually until a
  generator exists to derive one from the other (DOCFIX candidate, not
  actioned).
