diff --git a/docs/changelog-20260709-opentofu-cloudinit-vm.md b/docs/changelog-20260709-opentofu-cloudinit-vm.md new file mode 100644 index 0000000..0876602 --- /dev/null +++ b/docs/changelog-20260709-opentofu-cloudinit-vm.md @@ -0,0 +1,67 @@ +# Changelog 2026-07-09 -- OpenTofu cloud-init/base-image VM modules (DOCFIX-145) + +No live infrastructure touched -- unapplied IaC source only. Continues the +DOCFIX-142/144 OpenTofu scaffold. + +## Item + +### 1. DOCFIX-145 -- `opentofu/modules/base-image` + `opentofu/modules/cloudinit-vm` +FILES (new): `opentofu/modules/base-image/{main,variables,outputs}.tf`, +`opentofu/modules/cloudinit-vm/{main,variables,outputs}.tf`. + +WHAT: the second VM-creation pattern this repo needs, for Office1's own +service VMs (MAAS-region, NetBox, GitBucket -- D-103), genuinely distinct +from `modules/node-vm`'s blank-disk PXE pattern (that one is for MAAS-managed +OpenStack nodes MAAS itself images after enlistment; these are self-managed +VMs that need a real bootable OS + cloud-init). Split into two modules +matching how the provider's own real example separates concerns: +`base-image` downloads one shared cloud image once (`libvirt_volume` with +`create.content.url`); `cloudinit-vm` creates a per-instance copy-on-write +overlay disk (`backing_store`), a `libvirt_cloudinit_disk` NoCloud seed, and +the domain referencing both. + +VERIFICATION: every resource/attribute shape used is taken directly from +`examples/alpine_cloudinit.tf` (fetched and read as real, runnable code last +turn while building the node-VM module) -- the same high-confidence source +that surfaced the syntax-bug fix in DOCFIX-144. One design choice improved +during authoring: the base image's `.path` is threaded across the module +boundary as a real output/input (`module..path` -> +`cloudinit-vm`'s `base_volume_path` variable) rather than reconstructed from +a pool name + volume name string, avoiding an unconfirmed path-convention +guess entirely. + +DELIBERATELY NOT DONE: +- **`user_data`/`meta_data`/`network_config` have no defaults and no generic + fallback content.** What Office1's three service VMs actually need + installed/configured is real design work that hasn't happened; a + plausible-looking default (e.g. the common `dhcp4: true` netplan stanza) + would also silently fail here regardless, since this repo's plane networks + carry no libvirt-managed DHCP (NetBox is the IPAM apex). +- **OPNsense applicability is explicitly NOT confirmed.** This whole + cloud-init/NoCloud mechanism is standard for Linux cloud images; OPNsense + is FreeBSD-based, and its support for the same mechanism was not checked + this session. Flagged prominently in both modules' headers and + `opentofu/README.md` -- do not assume `cloudinit-vm` covers OPNsense + without verifying that specifically first. +- **Neither module instantiated in root `main.tf`.** No image source has + been chosen for any Office1 service VM. + +Workflow doc (`docs/dc-dc-deployment-workflow.md`) Stage 2 authoring status, +gap register items 2 and 4, and the status summary table updated to match; +companion visual tracker (same Artifact URL) redeployed. + +HARNESS: none added -- pure `.tf` source, same as the prior OpenTofu +deliveries; `scripts/opentofu-validate.sh` covers all of `opentofu/` +generically once a `tofu` binary is available to run it. + +REVERT: `git rm -r opentofu/modules/base-image opentofu/modules/cloudinit-vm`; +revert the workflow-doc sections listed above if reverting fully. + +## Next actionable step (unchanged, still logged not actioned) + +Same as DOCFIX-144: on a machine with the `tofu` binary and registry network +access, run `bash scripts/opentofu-validate.sh` against all of `opentofu/`, +then `tofu providers schema -json` to confirm the remaining unverified +details (the `boot`-order shape in `node-vm`; whether `libvirt_cloudinit_disk` ++ this whole pattern actually produces a bootable OPNsense image, which needs +a real test boot, not just a schema check). diff --git a/docs/changelog-20260709-opnsense-research.md b/docs/changelog-20260709-opnsense-research.md new file mode 100644 index 0000000..5896173 --- /dev/null +++ b/docs/changelog-20260709-opnsense-research.md @@ -0,0 +1,68 @@ +# Changelog 2026-07-09 -- OPNsense deployment research + +Research delivery, not a script/code change -- no live infrastructure touched, +no DOCFIX number assigned (matches the `changelog-20260709-dc-dc-design.md` +precedent for design/research-only deliveries). + +## Item + +### 1. OPNsense deployment mechanism researched against OPNsense's own docs/forum/GitHub +FILES: `opentofu/README.md` (new dedicated section, sourced), +`docs/dc-dc-deployment-workflow.md` (gap register item 4, summary table). + +WHAT: closes the open question from DOCFIX-145 -- whether `modules/ +cloudinit-vm` applies to OPNsense. It does not (confirmed, not just hedged): +OPNsense's own forum consensus is that FreeBSD's cloud-init support "is not +great yet." But there IS a real, officially documented alternative that uses +the exact same mechanical shape already built: + +- The **Configuration Importer** (official docs) runs in a 2-3 second + boot-time window, scanning an attached volume for `/conf/config.xml`. + Supported filesystems: GPT, MBR, ZFS pool, msdosfs (FAT). +- **ISO9660 support was added to the Importer specifically for VM/cloud + automation** -- a closed, milestone-targeted, core-developer-assigned + GitHub issue requested it explicitly so a config ISO could be attached as + a secondary CD-ROM ("shared one to many"). A later bug report confirms the + behavior is live in current code (exact shipped version not pinned from + the 22.7 changelog text directly -- the later bug report is the stronger + signal; confirm on a real boot before relying on it). + This means: same mechanical pattern as `cloudinit-vm` (a secondary volume + attached as a `cdrom` disk), different payload (a plain ISO9660 image with + `/conf/config.xml`, not a NoCloud seed). +- A deeper `autorun` hook (`/usr/local/etc/rc.syshook.d/import/`) exists for + more advanced provisioning beyond a static config.xml. +- **Use the nano image for KVM**, not the vga/serial/dvd installer images -- + pre-installed, serial-console-ready, auto-expands on first boot. Confirmed + real-world workflow: download `.img.bz2` -> `bunzip2` -> `qemu-img convert + -f raw -O qcow2` -> `qemu-img resize` -> boot via libvirt with a FreeBSD + os-variant, serial console, no graphics. +- Network convention: interface 1 = LAN (default 192.168.1.1), interface 2 = + WAN -- order matters, same instinct as `node-vm`'s PXE-boot-priority design. + +All findings sourced with URLs in `opentofu/README.md`'s dedicated section +(docs.opnsense.org, the OPNsense forum, opnsense/core GitHub issues, and one +independent KVM how-to + one community VM-image-building project used for +cross-corroboration). + +WHAT STILL NEEDS BUILDING (not done this delivery, itemized so it's a +punch-list not a vague "TODO"): +1. A preprocessing script for the nano image's decompress+convert step -- + `libvirt_volume`'s `create.content.url` almost certainly does a plain HTTP + fetch with no decompression, so the `.img.bz2` can't be consumed directly; + the qcow2 needs to exist before OpenTofu sees it. +2. A way to build a plain ISO9660 image containing `/conf/config.xml` + (`genisoimage`/`xorriso` -- exact tool/flags not yet verified against a + real build) to feed a `libvirt_volume`'s `create.content.url`, the same + way `libvirt_cloudinit_disk.path` feeds `cloudinit-vm`'s seed volume today. +3. The actual `config.xml` content for OPNsense's LAN/WAN/routing role at + each site -- real design work, not started. + +REVERT: `git checkout HEAD~ -- opentofu/README.md docs/dc-dc-deployment-workflow.md`. + +## Next actionable step + +Build the OPNsense-specific module (working name: `modules/opnsense-edge`) +once items 1-3 above are ready, following the now-confirmed pattern. Verify +the ISO9660 Importer behavior on a real boot as part of that work, not before +-- this research pins down the mechanism with reasonable confidence but the +module build is where it gets tested for real. diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md index b17fcb1..f67f1ef 100644 --- a/docs/dc-dc-deployment-workflow.md +++ b/docs/dc-dc-deployment-workflow.md @@ -64,7 +64,7 @@ | **Gate** | MAAS region reachable; NetBox authoritative + populated (planes, per-DC v4, ULA/GUA carve); GitBucket serving; OpenTofu reaches vcloud host libvirt; Tailscale confirmed to Office1 only. | | **Owns** | D-103 (Office1 VMs are OpenTofu-created), D-107 (headend is not a core-service provider). | | **Reuse vs new** | NEW infra (Office1 didn't exist for VR0 DC0), but the *practices* transfer directly: NetBox-as-IPAM-apex discipline, the `netbox/*-import.py` pattern, and this repo's own git/GitBucket hosting model are all precedent, just extended. | -| **Authoring status** | No runbook yet. | +| **Authoring status** | Runbook not started. `opentofu/modules/base-image` + `cloudinit-vm` (2026-07-09) give the VM-creation mechanism for the MAAS-region/NetBox/GitBucket service VMs -- neither instantiated yet: no image source chosen, and the real `user_data`/`meta_data`/`network_config` content for any of the three VMs hasn't been designed. See `opentofu/README.md`. | **State:** NOT STARTED. @@ -191,57 +191,64 @@ a `$DC` (or `--dc`) selector, with both libraries becoming DC-keyed lookups instead of flat constants. Do this BEFORE writing the Stage 5 runbook, or the "high reuse" claim above is not actually true operationally. -2. **OpenTofu -- network/pool/node-VM layer scaffolded 2026-07-09 (was: zero - files).** `opentofu/` now has: `modules/dc-planes` (six per-DC planes), - `modules/mesh-link` (the D-100 dark-fiber triangle legs), - `modules/dc-storage-pool`, wired for DC1 + Office1 (DC2 deliberately not - wired -- its CIDRs aren't assigned yet, D-101); `modules/node-vm` (blank - disk, PXE-boot MAAS-managed node VM, the D-103 pattern for Stage 3) -- - not yet instantiated, node count/sizing is a pending Phase-0 decision. - Provider schema was confirmed against the actual current docs AND real - example `.tf` files (not memory) -- it differs materially from older - common examples: nested structures are attribute-style objects (`key = - { ... }`), not classic HCL blocks. **A real bug from that first pass was - found and fixed:** `dc-planes`/`mesh-link`/`dc-storage-pool` originally - used block syntax for `libvirt_network`'s `domain`/`mtu` and - `libvirt_pool`'s `target`; corrected once real provider examples surfaced - the actual convention. Still missing, deliberately deferred (see - `opentofu/README.md`): the cloud-init/base-image VM pattern (needed for - Office1 service VMs and OPNsense -- a genuinely different module from - `node-vm`'s PXE pattern), the per-device `boot`-order attribute's exact - internal shape (flagged UNVERIFIED in `node-vm/main.tf`, an inference by - pattern not a confirmed field), and `tc netem` application. **UNVALIDATED** - -- no `tofu` binary was available to run `scripts/opentofu-validate.sh` - this session; run it on a connected machine before trusting any of this. +2. **OpenTofu -- network/pool/node-VM/cloud-init layer scaffolded 2026-07-09 + (was: zero files).** `opentofu/` now has: `modules/dc-planes` (six per-DC + planes), `modules/mesh-link` (the D-100 dark-fiber triangle legs), + `modules/dc-storage-pool`, `modules/node-vm` (blank disk, PXE-boot + MAAS-managed node VM, the D-103 pattern for Stage 3), and + `modules/base-image` + `modules/cloudinit-vm` (the OTHER VM pattern -- + pre-built image + cloud-init, for Office1's own service VMs). DC1 + Office1 + pool/plane wiring is instantiated in root `main.tf`; `node-vm` and + `cloudinit-vm` are NOT (node sizing and Office1 VM image/config choices are + both pending decisions -- neither invented here). Provider schema was + confirmed against the actual current docs AND real example `.tf` files + (not memory) -- it differs materially from older common examples: nested + structures are attribute-style objects (`key = { ... }`), not classic HCL + blocks. **A real bug from the first pass was found and fixed:** + `dc-planes`/`mesh-link`/`dc-storage-pool` originally used block syntax for + `libvirt_network`'s `domain`/`mtu` and `libvirt_pool`'s `target`; corrected + once real provider examples surfaced the actual convention -- discovered + precisely BECAUSE building the domain module forced fetching real example + code instead of doc-summaries alone. Still missing, deliberately deferred + (see `opentofu/README.md`): OPNsense's own bootability via `cloudinit-vm` + is NOT confirmed (FreeBSD, not the same cloud-init/NoCloud guarantee as + Linux images); the per-device `boot`-order attribute's exact internal + shape (flagged UNVERIFIED in `node-vm/main.tf`, an inference by pattern + not a confirmed field); and `tc netem` application. **UNVALIDATED** -- no + `tofu` binary was available to run `scripts/opentofu-validate.sh` this + session; run it on a connected machine before trusting any of this. 3. **NetBox multi-DC/dual-stack import pipeline.** `netbox/` has exactly two scripts (`ipv4-prefixes-import.py`, `ipv6-mark-reserved.py`) -- single-DC v4 import and v6-reservation-marking only. D-101 requires DC2's supernet, the org ULA /48, and per-DC GUA carves to be NetBox-authoritative via "the extended import pipeline" -- that pipeline is not written. Blocks Stage 0's exit condition (literals must be real, not placeholders) and Stage 2. -4. **OPNsense edge deployment/config, and netem WAN-simulation -- four separable - sub-gaps, not one.** Each site (Office1, DC1, DC2) gets its own independent - OPNsense edge with a simulated ISP uplink (D-100); OpenTofu owns creating - the edge VMs (D-103). Blocks Stage 1 (Office1 edge) and Stage 3 (per-DC - edges) gates, which require netem parameters "applied and measured." +4. **OPNsense edge deployment/config, and netem WAN-simulation -- (b)/(c) were + RESEARCHED 2026-07-09 (`opentofu/README.md`'s dedicated section, sourced + against OPNsense's own docs/forum/GitHub), (d) is still a real gap.** Each + site (Office1, DC1, DC2) gets its own independent OPNsense edge with a + simulated ISP uplink (D-100); OpenTofu owns creating the edge VMs (D-103). + Blocks Stage 1 (Office1 edge) and Stage 3 (per-DC edges) gates, which + require netem parameters "applied and measured." - **(a) Network substrate -- DONE.** `opentofu/modules/mesh-link` creates the inter-site L2 segments (DC1<->DC2, DC1<->Office1, DC2<->Office1) an OPNsense VM's interfaces would eventually attach to. - - **(b) The OPNsense VM/image itself -- needs a DIFFERENT module than - `node-vm` (gap #2's now-built domain module).** `node-vm` is the blank- - disk PXE pattern for MAAS-managed OpenStack nodes; OPNsense needs a real - bootable image + likely cloud-init, confirmed as a distinct, separately - schema'd provider pattern (`libvirt_cloudinit_disk` + a downloaded base - `libvirt_volume`) but not yet built as its own module. No image source - (official OPNsense qcow2/ISO vs a custom-built image) has been chosen - either. - - **(c) OPNsense routing/firewall configuration -- NOT STARTED, no - mechanism chosen.** Once the VM exists, per-site routing + the "simulated - ISP uplink" role need configuring. OPNsense's own config is an XML file - (`config.xml`), not something the libvirt provider touches at all -- - options are a pre-baked config baked into the image, a cloud-init-driven - first-boot script, or OPNsense's own API/CLI driven post-boot by a - provisioner. None evaluated yet. + - **(b)+(c) The OPNsense VM/image AND its config -- RESEARCHED, confirmed + as ONE pattern, not built yet.** Cloud-init is confirmed unreliable on + OPNsense (FreeBSD) -- OPNsense's own forum consensus, not just a hedge. + The real native mechanism is the Configuration Importer (official docs): + scans an attached volume for `/conf/config.xml` in a 2-3 second + boot-time window; ISO9660 support was added to it specifically for + VM/cloud automation (attach a config ISO as a secondary CD-ROM) -- + mechanically IDENTICAL to `modules/cloudinit-vm`'s shape (a secondary + volume attached as `cdrom`), just a different payload (a plain ISO9660 + image with `/conf/config.xml`, not a NoCloud seed). Use the nano image + (pre-installed, serial-ready), not the installer images. Still needed to + actually build it: a preprocessing script for the nano image's + decompress+convert (OpenTofu's volume-download mechanism won't do that + itself), a way to build the config.xml ISO, and the actual config.xml + content (real design work). No image source formally chosen yet either. + Full sourcing in `opentofu/README.md`. - **(d) `tc netem` WAN-simulation -- a wholly separate mechanism, not an OPNsense setting.** No Terraform/OpenTofu-native equivalent exists at all (confirmed against the `libvirt_network` schema, gap #2); it is a @@ -294,8 +301,9 @@ | Decisions (D-100..D-110) | DRAFTED, ALL PROPOSED -- **Stage 0 is the very next action** | | DR mechanism seed | DONE (`dc-dc-replication-DR-seed.md`), superseded-carrier corrected into D-108 | | Stage 0-7 runbooks | NOT YET WRITTEN -- see Tooling gap register above for the itemized breakdown | -| `opentofu/` (networks/pools/node-VM PXE pattern) | SCAFFOLDED 2026-07-09, UNVALIDATED (no `tofu` binary available this session) -- see gap #2 and `opentofu/README.md` | -| `opentofu/` (cloud-init/base-image VM pattern, OPNsense, netem) | NOT STARTED -- deliberately deferred, see `opentofu/README.md` | +| `opentofu/` (networks/pools/node-VM PXE/cloud-init-VM patterns) | SCAFFOLDED 2026-07-09, UNVALIDATED (no `tofu` binary available this session) -- see gap #2 and `opentofu/README.md` | +| OPNsense image+config mechanism | RESEARCHED 2026-07-09 (config.xml ISO via the Configuration Importer, nano image) -- not yet built, see `opentofu/README.md` | +| `opentofu/` (netem, DC2 planes) | NOT DONE -- deliberately deferred, see `opentofu/README.md` | | Reusable as-is, repo-agnostic (2026-07-09 sweep) | `bundle.yaml`, `phase-01..08` VR0 DC0 runbooks (Stage 5/7 template, once DC-parameterized -- gap #1), `preflight.sh`, `cloud-assert.sh`, `repo-lint.sh`, `run-logged.sh`, `ledger-scan.sh` | | `lib-net.sh` / `lib-hosts.sh` | NOT yet DC-agnostic -- single-deployment hardcoded literals; see gap #1 | | Environment/skill docs (`environment.md`, README.md) | Still describe VR0 DC0/v1 single-DC scope wholesale -- flagged, not yet rewritten for VR1; not a blocker for Stage 0 but will need doing before/during Stage 1 | diff --git a/docs/session-ledger.md b/docs/session-ledger.md index f3288f9..57fb607 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -773,8 +773,41 @@ state the attribute-object convention as a provider-wide assumption. UNVERIFIED, flagged: the per-device `boot`-order attribute's exact shape (an inference by pattern). Still deferred: the cloud-init/base-image VM - pattern (Office1 services + OPNsense), DC2 planes, netem. Uncommitted -- - operator has not yet asked to commit this piece. + pattern (Office1 services + OPNsense), DC2 planes, netem. + COMMITTED + PUSHED (3324387). +- **DONE -- DOCFIX-145 (changelog `docs/changelog-20260709-opentofu-cloudinit-vm.md`):** + built `opentofu/modules/base-image` + `modules/cloudinit-vm` (the OTHER VM + pattern, for Office1's own service VMs -- MAAS-region, NetBox, GitBucket). + Same high-confidence source as DOCFIX-144 (`examples/alpine_cloudinit.tf`, + fetched as real code). Threaded the base image's `.path` output across the + module boundary as a real attribute reference rather than reconstructing a + volume path from pool+name strings -- a cleaner design than the earlier + draft, avoiding an unconfirmed-convention guess entirely. + `user_data`/`meta_data`/`network_config` are required inputs with no + default (Office1 VM configs aren't designed yet; a plausible DHCP default + would also silently fail since these planes carry no libvirt DHCP). + **OPNsense applicability explicitly NOT confirmed** -- FreeBSD-based, + cloud-init/NoCloud support not the same guarantee as Linux images, flagged + prominently rather than assumed. Neither module instantiated (no image + source chosen for any service VM). Workflow doc + visual tracker updated. + Uncommitted -- operator has not yet asked to commit this piece. +- **DONE -- OPNsense deployment research (changelog + `docs/changelog-20260709-opnsense-research.md`, no DOCFIX -- research + delivery, matches the design-doc precedent):** operator asked to check + OPNsense's actual documentation. Confirmed cloud-init is genuinely + unreliable on OPNsense (FreeBSD; OPNsense's own forum consensus), closing + the DOCFIX-145 open question. Found the real mechanism instead: the + Configuration Importer (official docs) scans an attached volume for + `/conf/config.xml` in a 2-3s boot window; ISO9660 support was added to it + specifically for VM/cloud automation (a closed, milestone-targeted, + core-dev-assigned GitHub issue) -- mechanically identical to + `cloudinit-vm`'s cdrom-attach shape, different payload. Use the nano image + for KVM (pre-installed, serial-ready), not the installer images. Fully + sourced in `opentofu/README.md`. NOT built yet: a decompress/convert + preprocessing script for the nano image, an ISO9660 config.xml-seed + builder, and the real config.xml content -- itemized as the next + actionable step. Uncommitted -- operator has not yet asked to commit this + piece. diff --git a/opentofu/README.md b/opentofu/README.md index 72bd3ef..6fbb450 100644 --- a/opentofu/README.md +++ b/opentofu/README.md @@ -36,6 +36,24 @@ memory/vcpu/disk sizing is a Phase-0 host/disk-budget decision (buildout-design Section 3) that hasn't been made -- call the module once real values exist, don't invent placeholder specs to wire it in sooner. +- `modules/base-image` + `modules/cloudinit-vm` (2026-07-09) -- the cloud-init/ + pre-built-image VM pattern, for Office1's own service VMs (MAAS-region, + NetBox, GitBucket). Split into two, matching how + `examples/alpine_cloudinit.tf` itself separates concerns: `base-image` + downloads one shared cloud image ONCE (call it once per distinct image); + `cloudinit-vm` creates a per-VM copy-on-write overlay disk + a + `libvirt_cloudinit_disk` NoCloud seed + the domain (call it once per VM + instance, passing `module..path` straight through for the + overlay's `backing_store.path` -- a real attribute reference, not a + reconstructed guess). `user_data`/`meta_data`/`network_config` are REQUIRED + inputs with no default and no generic fallback -- what Office1's VMs + actually need configured is real design work not done yet, and a plausible- + looking default (e.g. `dhcp4: true`) would silently fail anyway since this + repo's planes carry no libvirt-managed DHCP. **NOT confirmed for OPNsense** + (FreeBSD-based; cloud-init/NoCloud support is not the same guarantee as for + Linux cloud images) -- check that specifically before assuming this module + covers OPNsense too. Neither module instantiated in root `main.tf` yet: no + image source has been chosen for any Office1 service VM. - Root wiring for DC1 (planes + pool, using DC1's inherited DC0 CIDRs) and Office1 (pool only), plus all three mesh links. @@ -53,14 +71,6 @@ call syntax) -- real example code is what settled it. Deliberately NOT built yet, and why: -- **The cloud-init/pre-built-image VM pattern** (a `libvirt_cloudinit_disk` + - a downloaded base `libvirt_volume` + a domain referencing both), needed for - Office1's own service VMs and for OPNsense -- confirmed as a real, distinct - provider pattern via `examples/alpine_cloudinit.tf`, but not yet built as a - module here. Different enough from `node-vm` (image source, `backing_store` - copy-on-write, `libvirt_cloudinit_disk`'s `user_data`/`meta_data`/ - `network_config`) to warrant its own module rather than bolting it onto - `node-vm`. - **DC2's plane networks.** D-101 has not assigned DC2's supernet yet (it's an explicit open sub-item). `main.tf` has the DC2 module block written and commented out, ready to uncomment once NetBox assigns real CIDRs -- do not @@ -71,22 +81,89 @@ provisioner or a small companion script -- not yet written. The exact parameters are themselves a Phase-0 open item (D-100), so this was going to need a second pass regardless. -- **OPNsense edges -- four separable pieces, not one, all currently blocked or - unstarted:** (a) the network substrate is done (`modules/mesh-link`, above); - (b) the VM/image itself needs the cloud-init/base-image module (above, still - unbuilt) -- `modules/node-vm`'s blank-PXE-boot pattern does NOT apply to - OPNsense, which needs a real bootable image -- and no image source (official - OPNsense qcow2/ISO vs. a custom build) has been chosen either; (c) OPNsense's - own routing/firewall config is a `config.xml` the - libvirt provider never touches -- pre-baked image vs. cloud-init first-boot - vs. API/CLI-driven provisioner is an open choice, not started; (d) `tc - netem` WAN simulation is a separate OS-level `tc qdisc` setting with no - Terraform-native equivalent at all, needing its own provisioner/script, and - whose parameters are still an unruled D-100 open item -- a decision, not - just scripting, is needed first. Full breakdown in - `docs/dc-dc-deployment-workflow.md` gap register item 4. -- **MAAS-region / NetBox / GitBucket service VMs.** Depend on the deferred - domain module (they're VMs too). +- **OPNsense edges -- four separable pieces, not one; (b) and (c) are now + RESEARCHED (2026-07-09, see the dedicated section below), not blocked:** + (a) the network substrate is done (`modules/mesh-link`, above); + (b)+(c) the image AND its config are a single confirmed pattern now (nano + image + a config.xml-seed ISO attached as cdrom, mechanically identical to + `modules/cloudinit-vm`'s shape) -- not yet built as a module, but no longer + an open research question, just implementation work; (d) `tc netem` WAN + simulation is a separate OS-level `tc qdisc` setting with no Terraform-native + equivalent at all, needing its own provisioner/script, and whose parameters + are still an unruled D-100 open item -- a decision, not just scripting, is + needed first. Full breakdown in `docs/dc-dc-deployment-workflow.md` gap + register item 4. +- **MAAS-region / NetBox / GitBucket service VMs -- module exists + (`cloudinit-vm`), not instantiated.** No image source has been chosen for + any of them, and their actual `user_data`/`meta_data`/`network_config` + content (what packages, what config, what static address) is real design + work that hasn't been done -- required inputs, no invented placeholders. + +## OPNsense deployment research (2026-07-09) -- read before building the OPNsense module + +Researched directly against OPNsense's own docs/forum/GitHub, not inferred -- +confirms `modules/cloudinit-vm` genuinely does NOT apply, AND identifies the +real mechanism, which turns out to need the same mechanical shape. + +1. **Cloud-init is confirmed unreliable on OPNsense, not just unconfirmed.** + Community consensus on OPNsense's own forum: "FreeBSD cloud-init support, + which is not great yet" -- an experienced user's conclusion after + investigating it directly. + (https://forum.opnsense.org/index.php?topic=42517.0) +2. **The real native mechanism is the Configuration Importer**, official docs: + runs very early in boot (a 2-3 second window), scans attached volumes for + `/conf/config.xml`. Supported filesystems: GPT, MBR, ZFS pool, msdosfs + (FAT). (https://docs.opnsense.org/manual/backups.html, + https://docs.opnsense.org/development/backend/autorun.html) +3. **ISO9660 was added to the Importer specifically for VM/cloud automation** + -- a closed, milestone-targeted, core-developer-assigned GitHub issue + requested it explicitly so a config ISO could be attached as a secondary + CD-ROM ("shared one to many," vs. a disk image needing a copy per VM). + A later bug report confirms ISO9660 detection is live in current code. + This means the exact SAME mechanical pattern as `modules/cloudinit-vm` + (a secondary volume attached as a `cdrom` disk) applies to OPNsense too -- + just with a different payload: a plain ISO9660 image containing + `/conf/config.xml`, not a NoCloud-format seed. + (https://github.com/opnsense/core/issues/5733, + https://github.com/opnsense/core/issues/10017 -- could not pin the exact + shipped version from the 22.7 changelog text directly; the feature's + existence is corroborated by the later bug report describing its + real-world behavior, which is a stronger signal than the milestone tag + alone. Confirm on a real boot before relying on it.) +4. **There's also a deeper `autorun` hook** for more advanced provisioning: + scripts in `/usr/local/etc/rc.syshook.d/import/` can override or extend the + default import, running before standard network startup. + (https://docs.opnsense.org/development/backend/autorun.html) +5. **Use the nano image for KVM, not the vga/serial/dvd installer images.** + Nano is pre-installed (serial console pre-configured, auto-expands to fill + the disk on first boot) -- the installer images need an interactive + install wizard, which defeats automation. Confirmed real-world workflow: + download the `.img.bz2` -> `bunzip2` -> `qemu-img convert -f raw -O qcow2` + -> `qemu-img resize` (grow) -> boot via libvirt with a FreeBSD os-variant, + serial console, no graphics. + (https://nickcharlton.net/posts/installing-opnsense-virt-install-kvm-serial, + https://opnsense.org/download/) +6. **A dedicated "build KVM-ready OPNsense images" community project confirms + there is no built-in cloud-init/config-injection in official images** -- + the pattern is always "deploy the immutable base image, then provision + separately," matching finding 2 above (seed minimal config via the + Importer to get SSH/API reachable, then drive the rest over the network). + (https://github.com/maurice-w/opnsense-vm-images) +7. **Network interface convention: interface 1 = LAN (default 192.168.1.1), + interface 2 = WAN.** Order matters when wiring `network_names`, the same + way `modules/node-vm`'s first-interface-gets-PXE-boot-priority does. + (https://docs.opnsense.org/manual/install.html) + +**What building the module still needs, not yet done:** (a) a small +preprocessing script for step 5's decompress+convert -- `create.content.url` +almost certainly does a plain HTTP fetch with no decompression, so it can't +consume the `.img.bz2` directly; the qcow2 needs to exist BEFORE OpenTofu +sees it. (b) a way to build a plain ISO9660 image containing `/conf/config.xml` +(e.g. `genisoimage`/`xorriso` -- exact tool/flags not yet verified against a +real build) to feed into a `libvirt_volume`'s `create.content.url`, the same +way `libvirt_cloudinit_disk.path` feeds `cloudinit-vm`'s seed volume. (c) an +actual `config.xml` for OPNsense's own LAN/WAN/routing role -- real design +work, not yet done. ## Schema notes (read before extending) @@ -134,6 +211,18 @@ ... }`); types like `dir` are native libvirt storage-pool vocabulary (stable, long-standing, not provider-specific) -- reasonably high confidence. +- `libvirt_cloudinit_disk` takes flat string arguments `name`, `user_data`, + `meta_data`, `network_config` (cloud-init YAML content, no nested + attributes) and exposes a `.path` -- feed that `.path` into a + `libvirt_volume`'s `create.content.url` to register it as an attachable + cdrom volume. Confirmed directly from `examples/alpine_cloudinit.tf`. +- A base image created in one module call and consumed via `backing_store` in + a DIFFERENT module call: pass the base module's `path` OUTPUT straight + through as a variable to the consuming module (`modules/cloudinit-vm` does + this). Do not reconstruct a volume's path from its pool name + volume name + -- libvirt's actual pool-relative path convention was not independently + confirmed this session, and a real `.path` attribute reference is always + available and unambiguous instead. ## Conventions carried from the rest of this repo diff --git a/opentofu/modules/base-image/main.tf b/opentofu/modules/base-image/main.tf new file mode 100644 index 0000000..76847e5 --- /dev/null +++ b/opentofu/modules/base-image/main.tf @@ -0,0 +1,25 @@ +# base-image: one downloaded cloud image, shared (via backing_store +# copy-on-write in modules/cloudinit-vm) across every VM that boots from it. +# Downloaded ONCE per image, not per-VM -- call this module once per distinct +# image, `modules/cloudinit-vm` once per VM instance. +# +# Confirmed directly from examples/alpine_cloudinit.tf (fetched and read this +# session, tag v0.9.8) -- the `alpine_base` resource in that file is exactly +# this shape. + +resource "libvirt_volume" "base" { + name = var.image_name + pool = var.pool_name + + target = { + format = { + type = "qcow2" + } + } + + create = { + content = { + url = var.source_url + } + } +} diff --git a/opentofu/modules/base-image/outputs.tf b/opentofu/modules/base-image/outputs.tf new file mode 100644 index 0000000..25172e9 --- /dev/null +++ b/opentofu/modules/base-image/outputs.tf @@ -0,0 +1,14 @@ +output "pool" { + description = "Pool the base image lives in -- for modules/cloudinit-vm's base_volume_pool input." + value = libvirt_volume.base.pool +} + +output "name" { + description = "Base image volume name -- for modules/cloudinit-vm's base_volume_name input." + value = libvirt_volume.base.name +} + +output "path" { + description = "Host filesystem path to the base image volume." + value = libvirt_volume.base.path +} diff --git a/opentofu/modules/base-image/variables.tf b/opentofu/modules/base-image/variables.tf new file mode 100644 index 0000000..33c194e --- /dev/null +++ b/opentofu/modules/base-image/variables.tf @@ -0,0 +1,24 @@ +variable "image_name" { + description = "libvirt_volume name for this base image, e.g. \"ubuntu-24.04-base.qcow2\"." + type = string +} + +variable "pool_name" { + description = "libvirt_pool name this base image is stored in (output of modules/dc-storage-pool)." + type = string +} + +variable "source_url" { + description = <<-EOT + URL to download the cloud image from (a real, chosen, cloud-init-capable + qcow2 -- e.g. an official Ubuntu/Debian cloud image URL). No default -- + the image choice itself hasn't been made for any of Office1's service VMs + (MAAS-region, NetBox, GitBucket) yet; pass it explicitly once it has been. + NOTE: this pattern is confirmed for standard Linux cloud images (verified + against examples/alpine_cloudinit.tf); it is NOT confirmed to work + unmodified for OPNsense (FreeBSD-based -- cloud-init/NoCloud compatibility + is not guaranteed the same way). Do not assume this module covers OPNsense + without checking that first. + EOT + type = string +} diff --git a/opentofu/modules/cloudinit-vm/main.tf b/opentofu/modules/cloudinit-vm/main.tf new file mode 100644 index 0000000..9ebd8e5 --- /dev/null +++ b/opentofu/modules/cloudinit-vm/main.tf @@ -0,0 +1,126 @@ +# cloudinit-vm: one cloud-init-provisioned VM booting from a shared base +# image via copy-on-write. For Office1's own service VMs (MAAS-region, +# NetBox, GitBucket -- D-103) -- genuinely different from +# modules/node-vm's blank-disk PXE pattern, which is for MAAS-managed +# OpenStack nodes that MAAS itself images after enlistment. Call +# modules/base-image once per distinct image, this module once per VM +# instance. +# +# NOT CONFIRMED for OPNsense: this whole pattern (cloud-init/NoCloud via +# libvirt_cloudinit_disk) is standard for Linux cloud images; OPNsense is +# FreeBSD-based and its cloud-init/NoCloud support is not the same guarantee. +# Do not assume this module covers OPNsense without checking that +# specifically first -- see opentofu/README.md. +# +# Every resource/attribute shape below is used exactly as shown in +# examples/alpine_cloudinit.tf (fetched and read directly this session, tag +# v0.9.8) -- attribute-style nested objects throughout, same convention as +# modules/node-vm and the corrected modules/dc-planes etc. No `graphics` +# block: this is a headless server VM pattern (the real example included VNC +# graphics; omitted here as unnecessary for a service VM, not because it's +# confirmed invalid to include -- add it back if console-debug access is +# ever wanted). + +resource "libvirt_volume" "disk" { + name = "${var.vm_name}-disk.qcow2" + pool = var.pool_name + capacity = var.disk_size_bytes + + target = { + format = { + type = "qcow2" + } + } + + backing_store = { + path = var.base_volume_path # the caller passes module..path + # straight through -- a real attribute + # reference, matching + # examples/alpine_cloudinit.tf's + # `libvirt_volume.alpine_base.path` exactly, + # just threaded across a module boundary + # instead of within one file. + format = { + type = "qcow2" + } + } +} + +resource "libvirt_cloudinit_disk" "seed" { + name = "${var.vm_name}-cloudinit" + + user_data = var.user_data + meta_data = var.meta_data + network_config = var.network_config +} + +resource "libvirt_volume" "seed" { + name = "${var.vm_name}-cloudinit.iso" + pool = var.pool_name + + create = { + content = { + url = libvirt_cloudinit_disk.seed.path + } + } +} + +resource "libvirt_domain" "vm" { + name = var.vm_name + memory = var.memory_mib + vcpu = var.vcpu + type = "kvm" + running = true + + os = { + type = "hvm" + type_arch = "x86_64" + type_machine = "q35" + } + + devices = { + disks = [ + { + source = { + volume = { + pool = libvirt_volume.disk.pool + volume = libvirt_volume.disk.name + } + } + target = { + dev = "vda" + bus = "virtio" + } + driver = { + type = "qcow2" + } + }, + { + device = "cdrom" + source = { + volume = { + pool = libvirt_volume.seed.pool + volume = libvirt_volume.seed.name + } + } + target = { + dev = "sda" + bus = "sata" + } + } + ] + + interfaces = [ + for net_name in var.network_names : { + model = { + type = "virtio" + } + source = { + network = { + network = net_name + } + } + } + ] + } +} diff --git a/opentofu/modules/cloudinit-vm/outputs.tf b/opentofu/modules/cloudinit-vm/outputs.tf new file mode 100644 index 0000000..45d3867 --- /dev/null +++ b/opentofu/modules/cloudinit-vm/outputs.tf @@ -0,0 +1,9 @@ +output "domain_id" { + description = "libvirt domain id." + value = libvirt_domain.vm.id +} + +output "domain_name" { + description = "libvirt domain name." + value = libvirt_domain.vm.name +} diff --git a/opentofu/modules/cloudinit-vm/variables.tf b/opentofu/modules/cloudinit-vm/variables.tf new file mode 100644 index 0000000..0815632 --- /dev/null +++ b/opentofu/modules/cloudinit-vm/variables.tf @@ -0,0 +1,68 @@ +variable "vm_name" { + description = "Domain name, e.g. \"office1-netbox\", \"office1-maas-region\"." + type = string +} + +variable "memory_mib" { + description = "Memory in MiB. No default -- pass explicitly." + type = number +} + +variable "vcpu" { + description = "Virtual CPU count. No default -- pass explicitly." + type = number +} + +variable "pool_name" { + description = "libvirt_pool this VM's disk + cloud-init seed volume live in." + type = string +} + +variable "disk_size_bytes" { + description = "Boot disk size in bytes (the backing_store overlay's capacity, not the base image's). No default -- pass explicitly." + type = number +} + +variable "base_volume_path" { + description = <<-EOT + Host filesystem path of the base image this VM boots from -- pass + modules/base-image's `path` OUTPUT directly (a real resource attribute), + e.g. `module.ubuntu_base.path`. Do not reconstruct this from a pool name + + volume name; libvirt's actual pool-relative path convention was not + independently confirmed this session (see main.tf's note on this). + EOT + type = string +} + +variable "network_names" { + description = "Ordered list of libvirt_network names (outputs of modules/dc-planes or modules/mesh-link) this VM attaches to." + type = list(string) +} + +variable "user_data" { + description = <<-EOT + Full #cloud-config YAML content (a real, chosen configuration -- hostname, + packages, users, etc. for THIS specific VM). No default and no generic + fallback content is provided here: what Office1's MAAS-region/NetBox/ + GitBucket VMs actually need installed/configured is real design work that + hasn't been done yet, and inventing placeholder cloud-config content would + misrepresent it as a real decision. Pass the actual config once written. + EOT + type = string +} + +variable "meta_data" { + description = "Cloud-init meta-data YAML (instance-id, local-hostname). No default -- pass explicitly, instance-id must be unique per VM." + type = string +} + +variable "network_config" { + description = <<-EOT + Cloud-init network-config YAML (version 2 netplan-style). No default and + deliberately NOT a "dhcp4: true" fallback: this repo's plane networks + carry no libvirt-managed DHCP (NetBox is the IPAM apex, skill hard rule + 3), so a real static address from NetBox is required here, not a + plausible-looking default that would silently fail to reach anything. + EOT + type = string +}