diff --git a/docs/changelog-20260713-config-xml-path-deleted.md b/docs/changelog-20260713-config-xml-path-deleted.md new file mode 100644 index 0000000..c843909 --- /dev/null +++ b/docs/changelog-20260713-config-xml-path-deleted.md @@ -0,0 +1,73 @@ +# 2026-07-13 -- D-113(a2) COMPLETE: the config.xml path is DELETED + +Operator ruling ("I'll defer to your lean"): the OPNsense `config.xml` template is **deleted, not +reduced**. + +## Why deleted rather than reduced + +The (a2) plan was to shrink the template to a minimal bootstrap (sshd + root key + console + a +seeded API key). That turned out to be **unnecessary** -- every one of those is covered without a +`config.xml` at all: + +| bootstrap need | how it is met now | +|---|---| +| sshd + root key | the **D-112(c) console bootstrap** (proven on Office1) | +| an API key | **`opnsense-bootstrap-apikey.sh`** -- calls OPNsense's OWN model over SSH; no GUI click, no re-implemented crypto | +| DHCP / firewall / interfaces | **the REST API** (`opnsense-api.sh`), proven read AND write | + +So the provisioning chain contains **no `config.xml` anywhere**: + + boot factory nano -> console bootstrap -> mint API key -> configure over REST + +A `config.xml` renderer that nobody should ever run is **not a safety net -- it is a loaded gun +pointed at a live router.** The 2026-07-13 safety sweep is the evidence: the repo still contained +runbook steps telling an operator to render a config and push it to the edge, which by then would +have **clobbered live API-managed DHCP**. + +## Deleted + +- `opentofu/templates/opnsense-config.xml.tmpl` +- `scripts/opnsense-render-config.sh` + `tests/opnsense-render-config/` +- `scripts/opnsense-build-config-iso.sh` + `tests/opnsense-build-config-iso/` +- the `opnsense-edge` module's `config_seed` volume + cdrom disk, and its `config_iso_path` var +- the `xorriso`/`genisoimage` prereq (it existed ONLY for the ISO builder) + +All remain in git history. `opentofu/templates/README.md` is now a **tombstone** explaining what +happened and what to use instead -- people will go looking for that template. + +## The live change, and why it was safe + +Removing the module's ISO wiring touches an INSTANTIATED resource +(`libvirt_volume.config_seed` is in tfstate). Measured with `tofu plan` BEFORE applying: + + module.office1_opnsense.libvirt_domain.vm will be updated IN-PLACE + module.office1_opnsense.libvirt_volume.config_seed will be destroyed + Plan: 0 to add, 1 to change, 1 to destroy. + +**No replacement** of the running domain -- confirmed explicitly (a replacement would have +destroyed the live edge). Only the cdrom entry is removed from the disk list and the inert ISO +volume is dropped. + +**Applying it now was the SAFE choice, not the risky one.** Leaving the repo/state divergence in +place would mean the next `tofu apply` -- e.g. while building DC1's substrate in Stage 3 -- +sweeps up this pending change as a **surprise side effect on the running Office1 edge, mid-stage**. +Deliberate and gated beats incidental and unnoticed. + +## A lint guard did its job + +`repo-lint`'s **L3** rule (runbooks must not reference missing scripts) went RED on the tombstone +notes, because they named the deleted paths. The rule was RIGHT, and it has no opt-out (only L4 +does). Rather than weaken a guard that exists to stop runbooks pointing at dead scripts, the +tombstones were reworded to name the bare filename instead of the `scripts/` path. The guard stays +fully intact; the information is preserved. + +## Verification + +`repo-lint` 0 fail. `opentofu-validate` PASS -- root + **10/10 modules** standalone. Gauntlet green. + +## Revert + + git revert # restores the template, renderer, ISO builder, harnesses + cd opentofu && tofu apply # re-creates the (inert) config_seed volume + cdrom + +Nothing depends on the ISO: it was never read by anything (D-112). diff --git a/docs/design-decisions.md b/docs/design-decisions.md index 653b2d7..858bf49 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -2340,6 +2340,30 @@ keeps the GUI, keeps a WORKING routing+DHCP edge, keeps the two libvirt fixes, and removes the defect source -- and it is the smallest change of anything considered. +**AMENDMENT 2026-07-13 -- the template is DELETED, not reduced (operator ruling: "I'll defer +to your lean").** The original (a2) plan was to REDUCE `opnsense-config.xml.tmpl` to a minimal +bootstrap (sshd + root key + console + a seeded API key). That turned out to be unnecessary: +every one of those is covered without any `config.xml` at all. + +- sshd + root key: the **D-112(c) console bootstrap** installs them (proven on Office1). +- the API key: **`scripts/opnsense-bootstrap-apikey.sh`** mints one by calling OPNsense's OWN + model (`Auth\User -> apikeys->add()`) over SSH -- no GUI click, and no re-implementation of the + vendor's `$6$` crypt format (which would fail SILENTLY on any format drift). +- everything else -- DHCP, firewall, interfaces: **the REST API**. + +So the provisioning chain has NO `config.xml` in it, and a renderer that nobody should ever run is +not a safety net -- it is a loaded gun pointed at a live router. The 2026-07-13 safety sweep is the +evidence: the repo still contained runbook steps instructing an operator to render a config and push +it to the edge, which by then would have CLOBBERED live API-managed DHCP. + +**DELETED:** `opentofu/templates/opnsense-config.xml.tmpl`, `scripts/opnsense-render-config.sh`, +`scripts/opnsense-build-config-iso.sh`, and both their harnesses. The `opnsense-edge` module's +`config_seed` volume + cdrom disk are removed (the ISO was INERT -- D-112). The +`xorriso`/`genisoimage` prereq is dropped. All remain in git history. + +**If you think the API cannot express something:** that is a finding to raise against D-113 -- not +a reason to hand-write XML. + **Superseded by this ruling:** the "stay on (a1)" lean in the original recommendation below. (a1) is now explicitly REJECTED: its cost compounds per feature and it depends on an undocumented self-heal of an internal format (see the 667-element finding). diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 0cf6004..5346e2e 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -2011,6 +2011,28 @@ scp/install/reboot path against this edge. Template reduction to a minimal bootstrap (sshd + key + console) is the next step and is NOT yet done. +## D-113(a2) COMPLETE 2026-07-13: the config.xml path is DELETED + +- Operator ruling ("I'll defer to your lean"): the template is **DELETED, not reduced**. Reducing + it to a minimal bootstrap proved UNNECESSARY -- console bootstrap gives sshd+key, + `opnsense-bootstrap-apikey.sh` gives the API key, the REST API gives everything else. **The + provisioning chain has NO config.xml in it anywhere.** +- **DELETED:** `opentofu/templates/opnsense-config.xml.tmpl`, `scripts/opnsense-render-config.sh`, + `scripts/opnsense-build-config-iso.sh`, both their harnesses (gauntlet 54 -> 52, expected), the + `opnsense-edge` module's `config_seed` volume + cdrom disk + `config_iso_path` var, and the + xorriso/genisoimage prereq (it existed ONLY for the ISO builder). All in git history. + `opentofu/templates/README.md` is now a TOMBSTONE -- people will go looking for that template. +- **THE LIVE CHANGE (measured BEFORE applying, with `tofu plan`):** + `libvirt_domain.vm will be updated IN-PLACE` + `config_seed will be destroyed`; + `Plan: 0 to add, 1 to change, 1 to destroy`. **NO replacement of the running domain** -- a + replacement would have DESTROYED the live edge. Applying now is the SAFE choice: leaving the + repo/state divergence would mean the next `tofu apply` (e.g. Stage 3's DC1 build) sweeps it up + as a SURPRISE mutation on the running Office1 edge, mid-stage. +- **A guard did its job:** repo-lint's L3 (runbooks must not reference missing scripts) went RED on + the tombstone notes. It was RIGHT and has no opt-out. Rather than weaken it, the tombstones were + reworded to name the bare filename instead of the `scripts/` path. Guard intact. +- `docs/changelog-20260713-config-xml-path-deleted.md`. D-113 AMENDED in design-decisions. + ## D-113(a2): API-KEY BOOTSTRAP SOLVED 2026-07-13 (no GUI click, no re-implemented crypto) - **The last unknown in edge provisioning is closed.** `scripts/opnsense-bootstrap-apikey.sh` + diff --git a/opentofu/main.tf b/opentofu/main.tf index de84b98..1039972 100644 --- a/opentofu/main.tf +++ b/opentofu/main.tf @@ -110,7 +110,6 @@ # from scripts/opnsense-prep-image.sh's GROW (default +8G -> 11 GiB). The old # `disk_size_bytes = 16 GiB` here was SILENTLY IGNORED -- measured: the live volume is # 11 GiB. Removed in DOCFIX-189. - config_iso_path = "/var/lib/libvirt/vr1/staging/office1-opnsense-config.iso" # staging, NOT the pool dir (create.content.url collides with its own target volume there -- first-boot finding 2026-07-12) lan_network_name = module.office1_network.network_name wan_network_name = "office1-wan" } diff --git a/opentofu/modules/opnsense-edge/main.tf b/opentofu/modules/opnsense-edge/main.tf index 965db44..6504b98 100644 --- a/opentofu/modules/opnsense-edge/main.tf +++ b/opentofu/modules/opnsense-edge/main.tf @@ -1,18 +1,29 @@ # opnsense-edge: one OPNsense edge VM (D-100's per-site independent OPNsense # edge with a simulated ISP uplink; D-103: OpenTofu owns creating it). # -# NOT the cloud-init pattern (modules/cloudinit-vm) -- confirmed unreliable -# on OPNsense/FreeBSD this session (see opentofu/README.md's "OPNsense -# deployment research" section, fully sourced). This module instead uses -# OPNsense's own Configuration Importer: a plain ISO9660 volume containing -# /conf/config.xml, attached as a secondary cdrom disk -- mechanically -# IDENTICAL to cloudinit-vm's seed-volume shape (a libvirt_volume with -# create.content.url, attached as device="cdrom"), just a different payload -# and no libvirt_cloudinit_disk resource (that resource is NoCloud-specific; -# wrong format here). Both the base image and the config ISO are prepared -# OUTSIDE OpenTofu by scripts/opnsense-prep-image.sh and -# scripts/opnsense-build-config-iso.sh respectively -- see each variable's -# description for why. +# THIS MODULE NO LONGER SEEDS ANY CONFIG (2026-07-13). It creates a domain that +# boots the FACTORY nano image, and nothing more. Configuration happens AFTER +# boot, over the network. +# +# The config-seed ISO (a libvirt_volume + a cdrom disk) was REMOVED. It was +# INERT: per D-112, OPNsense's Configuration Importer can NEVER fire on a +# pre-installed nano image -- `opnsense-importer -b` probes for a read-only +# root, finds a writable one with a factory /conf/config.xml already present, +# and exits without enumerating a single device. Nothing ever read that ISO. +# Discovering that cost a full session (2026-07-12). +# +# The config.xml template and its renderer are DELETED under D-113(a2) -- +# hand-authoring the appliance's GUI-owned XML caused DOCFIX-191 (management +# lockout), DOCFIX-192 (dead console) and DOCFIX-193 (no DHCP). Edge config is +# now done over the REST API. The provisioning chain, proven on Office1: +# +# boot factory nano +# -> D-112(c) console bootstrap (enable SSH + install the key) +# -> scripts/opnsense-bootstrap-apikey.sh (mint an API key, no GUI click) +# -> scripts/opnsense-api.sh (DHCP, firewall, interfaces) +# +# The base image is still prepared OUTSIDE OpenTofu, by +# scripts/opnsense-prep-image.sh -- see that variable's description for why. # # UNVERIFIED, flagged plainly (see opentofu/README.md for the full account, # including a 2026-07-09 audit pass that corrected the LAN/WAN claim below): @@ -73,17 +84,6 @@ } } -resource "libvirt_volume" "config_seed" { - name = "${var.vm_name}-config.iso" - pool = var.pool_name - - create = { - content = { - url = var.config_iso_path - } - } -} - resource "libvirt_domain" "vm" { name = var.vm_name # memory_unit is REQUIRED. On dmacvicar/libvirt >=0.9, `memory` is a raw libvirt @@ -159,19 +159,6 @@ type = "qcow2" } }, - { - device = "cdrom" - source = { - volume = { - pool = libvirt_volume.config_seed.pool - volume = libvirt_volume.config_seed.name - } - } - target = { - dev = "sda" - bus = "sata" - } - } ] # index 0 = the network intended for LAN, index 1 = the network intended diff --git a/opentofu/modules/opnsense-edge/variables.tf b/opentofu/modules/opnsense-edge/variables.tf index ccd545a..220fb75 100644 --- a/opentofu/modules/opnsense-edge/variables.tf +++ b/opentofu/modules/opnsense-edge/variables.tf @@ -40,20 +40,6 @@ type = string } -variable "config_iso_path" { - description = <<-EOT - Host filesystem path of a plain ISO9660 image containing - /conf/config.xml -- the output of - `scripts/opnsense-build-config-iso.sh`. This is OPNsense's own - Configuration Importer mechanism (NOT cloud-init -- confirmed unreliable - on OPNsense/FreeBSD; see opentofu/README.md's OPNsense research section). - No default: the real config.xml content for this site's LAN/WAN/routing - role is design work that has to happen first -- do not point this at a - placeholder/empty config.xml to make the module "work" sooner. - EOT - type = string -} - variable "lan_network_name" { description = <<-EOT libvirt_network name intended to become this edge's LAN side -- placed diff --git a/opentofu/templates/README.md b/opentofu/templates/README.md index 2ae5226..fca3177 100644 --- a/opentofu/templates/README.md +++ b/opentofu/templates/README.md @@ -1,66 +1,56 @@ -# opentofu/templates/ -- OPNsense config.xml template +# opentofu/templates/ -- the OPNsense config.xml template is DELETED (2026-07-13) -`opnsense-config.xml.tmpl` is the `{{TOKEN}}`-parameterized OPNsense -`config.xml` for a DC-DC edge (Office1, DC1, or DC2 -- one instantiation per -site). Built directly from OPNsense's own real, currently-shipped -`config.xml.sample` (`opnsense/core`, fetched 2026-07-09 -- see -`.claude/skills/openstack-cloud-ops/references/opentofu-provider-docs.md` -for the fetch method) plus the confirmed `staticroutes` schema -(`opnsense/core`'s `Route.xml` model). This is real design work, not a -placeholder -- but several tokens genuinely cannot be filled with a real -value yet because the decisions they depend on (D-100/D-101/D-107) are -still PROPOSED, or because the value can only be measured on a real boot. -Never invent a value for those; leave the token required and unfilled until -the real one exists. +`opnsense-config.xml.tmpl` **no longer exists.** Neither do `scripts/opnsense-render-config.sh` +(the renderer) or `scripts/opnsense-build-config-iso.sh` (the config-seed ISO builder). If you +came here looking for them, this is why -- and what to use instead. -## Pipeline +## Why they are gone -``` -opnsense-config.xml.tmpl - --(scripts/opnsense-render-config.sh, env-var tokens)--> -real config.xml - --(scripts/opnsense-build-config-iso.sh)--> -config.iso (plain ISO9660, /conf/config.xml) - --(opentofu/modules/opnsense-edge, config_iso_path variable)--> -attached as a secondary cdrom disk to the domain -``` +**Hand-authoring the appliance's GUI-owned `config.xml` was the single root cause of every +OPNsense-specific bug this project has had:** -## Token legend +| bug | what hand-authored XML did | +|---|---| +| DOCFIX-191 | the rendered config had no sshd and no key -- it would have **locked management out** | +| DOCFIX-192 | the rendered config silenced the edge's **only console** (no serial, no video) | +| DOCFIX-193 | LAN DHCP was never implemented -- and an old-style ISC `` block would have been **inert**, because OPNsense 26.1 uses **Kea** | +| 2026-07-13 | a full-config push **drops ~667 migration-populated elements**, including the only 2 firewall pass rules on the box. It survives only because OPNsense silently regenerates them -- an undocumented self-heal we were depending on | -| Token | Status | Notes | -|---|---|---| -| `{{OPNSENSE_HOSTNAME}}` | Design choice, not yet made | e.g. `opnsense`. NOT named `HOSTNAME` -- that collides with bash's own built-in `$HOSTNAME` variable (caught by this delivery's own test harness: `unset HOSTNAME` doesn't actually clear it, since bash repopulates it). | -| `{{DOMAIN}}` | Real, applies D-106's naming convention | e.g. `omega.dc1.vr1.cloud.neumatrix.local`, following the (still-PROPOSED, but already-settled-as-design-intent) `.omega..vr1.cloud.neumatrix.local` pattern -- same status as DC1's plane CIDRs elsewhere in this repo: carried-forward design intent, not invented. | -| `{{ROOT_PASSWORD_HASH}}` | MUST be freshly generated, never reused | The real `config.xml.sample` ships a bcrypt hash (`$2y$10$...`) for a well-known default OPNsense password -- reusing it verbatim would mean every edge shares a public default root password, a real exposure. Generate a fresh one per deployment, e.g. `htpasswd -bnBC 10 "" '' \| tr -d ':\n'` (produces a `$2y$10$...`-compatible bcrypt hash) -- NOT independently verified against a real OPNsense login this session; confirm login works before relying on it operationally. | -| `{{NTP_UPSTREAM_SERVERS}}` / `{{NTP_PREFER_SERVER}}` | Real default supplied (optional to override) | Defaults to the exact same public pool OPNsense's own sample ships (`N.opnsense.pool.ntp.org`) -- a real, confirmed default, not invented. D-107 says the edge "syncs upstream to internet NTP pools," consistent with keeping this default. Override only if a different upstream is decided. | -| `{{WAN_IF}}` / `{{LAN_IF}}` | MUST be measured on a real boot | The `` value is the actual `vtnetN` (or similar) device name FreeBRSD assigns -- confirmed from the real sample that this, not interface declaration order, is what assigns the LAN/WAN role (see `opentofu/README.md`'s "Audit pass" section for the full account of this correction). Boot the domain, run `ifconfig`, and set these to match `opentofu/modules/opnsense-edge`'s actual `lan_network_name`/`wan_network_name` wiring. | -| `{{WAN_IPADDR}}` / `{{WAN_SUBNET_BITS}}` / `{{WAN_GATEWAY}}` | Pending D-100/D-101 | The "simulated ISP uplink" network's real addressing isn't assigned yet (NetBox-pending). | -| `{{LAN_IPADDR}}` / `{{LAN_SUBNET_BITS}}` | Pending a design decision | WHICH plane(s) OPNsense's LAN interface actually serves (metal-admin? provider-public? something else?) is not fully specified in the buildout design as read this session -- confirm before filling this in, don't assume. | -| `{{MIRROR_SYNC_PROTOCOL}}` / `{{MIRROR_UPSTREAM_NET}}` / `{{MIRROR_SYNC_PORT}}` | Pending -- mirror software not chosen | D-107 requires the per-DC artifact mirror's own upstream sync as one of exactly two allowed WAN egress paths (the other is NTP), but doesn't specify which mirror software (apt-mirror? aptly? a generic reverse proxy?) -- the port/protocol depend on that choice. Tooling gap register item 3 territory. | -| `{{ROUTE1_NETWORK}}`/`{{ROUTE1_GATEWAY}}`/`{{ROUTE1_DESCR}}`, `{{ROUTE2_*}}` | Optional, pending DC2 CIDR assignment | At most 2 static routes needed per the known topology (a peer-DC replication-plane route at DC1/DC2; an Office1-management route). Leave a slot's `*_NETWORK` var unset to omit it entirely -- `scripts/opnsense-render-config.sh` drops an unset slot rather than emitting an empty/invalid ``. DC2's own CIDRs aren't assigned yet (D-101 open item), so DC1's route to DC2 can't be filled in until then. | +**None of those is expressible through the REST API,** where sshd, DHCP and firewall rules are +typed resources with defaults. You cannot forget to enable sshd in a format where sshd is a typed +field. -## Design choices baked into the template (not tokens -- read before assuming these are wrong) +**And the config-seed ISO never worked at all.** Per **D-112**, the OPNsense Configuration +Importer can NEVER fire on a pre-installed nano image: `opnsense-importer -b` probes for a +read-only root, finds a writable one with a factory `/conf/config.xml` already present, and exits +without enumerating a single device. Nothing was ever going to read that ISO. It cost a full +session on 2026-07-12 to discover. -- **Firewall: default-deny WAN egress, explicit NTP + mirror-sync allow rules** - (D-107's "controlled egress ... narrowly scoped: NTP-only, plus the - mirror's upstream sync"). The two default LAN-allow rules are kept exactly - as OPNsense's own sample ships them (basic operation depends on them). -- **`gateways.gateway_item`** is new versus the stock sample (which only has - an implicit DHCP-derived gateway) -- required because the WAN side here is - a static "simulated ISP uplink," not DHCP, and `staticroutes.route.gateway` - must reference a named gateway object (confirmed from the real - `Route.xml` model: `gateway` is a `JsonKeyValueStoreField` needing "a valid - gateway from the list"). Named `WAN_GW` -- change consistently in both - places if you rename it. -- **`unbound`/`nat`/`rrd`/legacy ``** sections are left exactly as - the real stock sample ships them -- minimal, working defaults, not - independently redesigned. +## What to use instead (D-113(a2), ADOPTED + PROVEN 2026-07-13) -## What this delivery does NOT do +Edge configuration is done over the **OPNsense REST API**. The provisioning chain has **no +`config.xml` in it anywhere**: -Write the actual, final, per-site `config.xml` files (Office1/DC1/DC2) -- -that needs the pending values above filled in with real ones, which in turn -needs Stage 0 ratification (D-100/D-101/D-107) and a real boot to measure -`vtnetN` assignment. This delivery is the mechanism (template + a tested -renderer), consistent with everything else built this session: parameterized -and reusable, not three files with invented IP addresses. + boot the factory nano image + -> D-112(c) console bootstrap (enable SSH + install the service key) + -> scripts/opnsense-bootstrap-apikey.sh (mints an API key via OPNsense's OWN model -- + no GUI click, no re-implemented crypto) + -> scripts/opnsense-api.sh (DHCP, firewall, interfaces -- everything else) + +Proven end to end against the live Office1 edge on 2026-07-13, read AND write: +`docs/changelog-20260713-opnsense-api-proven.md`, +`docs/changelog-20260713-opnsense-api-write-proven.md`, +`docs/changelog-20260713-opnsense-apikey-bootstrap.md`. + +## If you are tempted to bring the template back + +Don't. A `config.xml` renderer is not a safety net -- it is a loaded gun pointed at a live +router. On 2026-07-13 the repo still contained runbook steps telling an operator to render a +config and push it to the edge, which by then would have **clobbered live API-managed DHCP** +(see `docs/changelog-20260713-config-xml-danger-sweep.md`). + +The governing decision is **D-113** (`docs/design-decisions.md`). If you believe the API cannot +express something the edge needs, that is a finding worth raising against D-113 -- not a reason +to hand-write XML. + +Deleted in: `docs/changelog-20260713-config-xml-path-deleted.md`. The files remain in git history. diff --git a/opentofu/templates/opnsense-config.xml.tmpl b/opentofu/templates/opnsense-config.xml.tmpl deleted file mode 100644 index a684391..0000000 --- a/opentofu/templates/opnsense-config.xml.tmpl +++ /dev/null @@ -1,208 +0,0 @@ - - - - opnsense - - normal - {{OPNSENSE_HOSTNAME}} - {{DOMAIN}} - 1 - - - admins - System Administrators - system - 1999 - 0 - page-all - - - root - System Administrator - system - admins - {{ROOT_PASSWORD_HASH}} - 0 - - {{ROOT_AUTHORIZED_KEYS_B64}} - - Etc/UTC - {{NTP_UPSTREAM_SERVERS}} - - https - - yes - 1 - - - serial - 115200 - 1 - - monthly - - 1 - - - enabled - 1 - admins - - - - - 1 - {{WAN_IF}} - - {{WAN_IPADDR}} - {{WAN_SUBNET_BITS}} - WAN_GW - 1 - 1 - - - 1 - {{LAN_IF}} - {{LAN_IPADDR}} - {{LAN_SUBNET_BITS}} - - - - - wan - {{WAN_GATEWAY}} - WAN_GW - 1 - inet - simulated ISP uplink gateway (D-100) - - - -{{ROUTE1_BLOCK}} -{{ROUTE2_BLOCK}} - - - 1 - - - - automatic - - - - - - - - - {{NTP_PREFER_SERVER}} - {{NTP_UPSTREAM_SERVERS}} - - - - - - - 1 - keep - 1 - pass - 1 - lan - in - inet - any - lan - any - Default allow LAN to any rule - - - 1 - keep - 11 - pass - 1 - lan - in - inet6 - any - lan - any - Default allow LAN IPv6 to any rule - - - - - - - - - - - - - 1 - 0 - lan - 4000 - 1 - raw - - - - {{LAN_NETWORK_CIDR}} - 1 - - {{LAN_IPADDR}} - {{LAN_IPADDR}} - {{DOMAIN}} - - {{DHCP_POOL}} - LAN DHCP - - - - - - - diff --git a/runbooks/dc-dc-phase2-tofu-dc-substrate.md b/runbooks/dc-dc-phase2-tofu-dc-substrate.md index ad7e089..a57aa64 100644 --- a/runbooks/dc-dc-phase2-tofu-dc-substrate.md +++ b/runbooks/dc-dc-phase2-tofu-dc-substrate.md @@ -73,13 +73,14 @@ log it as exactly what it is: a placeholder pending D-100's still-open sub-item, not a final value. -!!! **OPNsense prep tools not confirmed present.** `scripts/opnsense-prep- - image.sh` requires `bunzip2` + `qemu-img` on PATH; `scripts/opnsense- - build-config-iso.sh` requires `genisoimage` OR `xorriso`. Neither has - been confirmed present on the vcloud host as of this writing - (`opentofu/README.md`). Step 2 below checks for all four before anything - else in this runbook proceeds -- treat a missing tool as a blocking - finding (install it, or escalate), not something to route around. +!!! **OPNsense prep tools.** `scripts/opnsense-prep-image.sh` requires + `bunzip2` + `qemu-img` on PATH. **The ISO9660 builder requirement is GONE + (2026-07-13):** the config-ISO builder (`opnsense-build-config-iso.sh`, + formerly under scripts/) is DELETED (D-112 -- + the Importer can never fire on a nano image, so the ISO was never read; + D-113(a2) -- config is done over the REST API). `genisoimage`/`xorriso` are + no longer needed and are no longer installed by + `scripts/prereqs/install-image-tools.sh`. Step 2 checks the remaining tools. --- @@ -89,10 +90,10 @@ 1. Pre-flight: confirm Stage 1 + Stage 2 gates closed (read-only) 2. Confirm OPNsense prep-tool prereqs on the vcloud host (read-only) 3. Confirm/record still-open decisions this stage depends - on (node sizing, config.xml tokens, MAAS zone/pool, - netem parameters) -- STOP on any that are unresolved (read-only) -4. Prepare DC1's OPNsense base image + render config.xml - + build the config ISO [MUTATION: host-local files, gated] + on (node sizing, MAAS zone/pool, netem parameters) + -- STOP on any that are unresolved (read-only) +4. Prepare DC1's OPNsense base image (render/config-ISO steps DELETED + 2026-07-13 -- config is done over the REST API) [MUTATION: host-local files, gated] 5. Wire modules/opnsense-edge for DC1 into main.tf [repo change, gated] 6. Wire modules/node-vm calls for DC1's node VMs into main.tf (ONLY once sizing is real) [repo change, gated] @@ -152,7 +153,7 @@ done ``` Required: `bunzip2` + `qemu-img` (for `scripts/opnsense-prep-image.sh`), and -EITHER `genisoimage` OR `xorriso` (for `scripts/opnsense-build-config-iso.sh`), +(The `genisoimage`/`xorriso` requirement is GONE -- the config-ISO path is deleted.) plus `curl` or `wget` for the image fetch. If any required tool is absent, install it on the vcloud host (an OS-specific package install, out of scope to prescribe here) before Step 4 -- do not skip the tool and hand-build the @@ -254,16 +255,9 @@ **Expect:** a decompressed, qcow2-converted, resized OPNsense nano image at a real output path -- record that path, it feeds Step 5's `base_volume_path`. -**MUTATION -- render config.xml (only if every non-boot-measured token from -Step 3 is real)** -```bash -export OPNSENSE_HOSTNAME=... DOMAIN=... ROOT_PASSWORD_HASH=... -export WAN_IPADDR=... WAN_SUBNET_BITS=... WAN_GATEWAY=... -export LAN_IPADDR=... LAN_SUBNET_BITS=... -export MIRROR_SYNC_PROTOCOL=... MIRROR_UPSTREAM_NET=... MIRROR_SYNC_PORT=... -# WAN_IF / LAN_IF: cannot be set for real yet -- see the note below. -bash scripts/opnsense-render-config.sh dc1-opnsense-config.xml -``` +**(DELETED 2026-07-13.)** The `config.xml` render step is GONE -- the renderer +(`opnsense-render-config.sh`, formerly under scripts/) no longer exists. Edge config is done over the REST API +(D-113(a2)). See the replacement chain at the end of this step. `WAN_IF`/`LAN_IF` are the one pair the script's own `REQUIRED_VARS` list demands that genuinely cannot be measured before the domain exists (Step 8 creates it). If Step 3 flagged these as not yet measurable, this render @@ -275,12 +269,35 @@ re-seed -- that re-seed mechanic is not designed in this delivery; treat it as a real follow-up need if it comes to that, not an invented shortcut here.) -**MUTATION -- build the config ISO (only once config.xml render succeeded)** -```bash -bash scripts/opnsense-build-config-iso.sh dc1-opnsense-config.xml dc1-opnsense-config.iso -``` -**Expect:** a plain ISO9660 image containing `/conf/config.xml` at a real -output path -- record it, it feeds Step 5's `config_iso_path`. +**(DELETED 2026-07-13.)** The config-ISO step is GONE -- the ISO builder +(`opnsense-build-config-iso.sh`, formerly under scripts/) no longer exists, and the module no longer takes a +`config_iso_path`. The ISO was never read by anything (D-112: the Importer cannot fire on a +pre-installed nano image). + +--- + +### The REPLACEMENT chain (proven on Office1, 2026-07-12/13) + +No `config.xml` anywhere. After Step 5 creates the domain and it boots on FACTORY DEFAULTS: + +1. **Console bootstrap (D-112(c))** -- reach the serial console, enable SSH, install the service + public key. This is the only step that needs the console; everything after it is scripted. +2. **Mint the API key** -- no GUI click: + ```bash + export OPNSENSE_SSH_KEY=... # the service private key + bash scripts/opnsense-bootstrap-apikey.sh + ``` +3. **Configure over REST** -- interfaces, DHCP, firewall: + ```bash + export OPNSENSE_API_HOST= OPNSENSE_API_CREDS= + bash scripts/opnsense-api.sh GET core/firmware/status # auth smoke test + bash scripts/opnsense-api.sh POST kea/dhcpv4/set_subnet/ '' + bash scripts/opnsense-api.sh POST kea/service/reconfigure '{}' + ``` + +**`WAN_IF`/`LAN_IF` are measured AFTER boot** (`ifconfig` over SSH), where they are knowable -- +which is why the chicken-and-egg problem this step used to agonize over simply does not exist +anymore. **GATE:** base image path and (if unblocked) config ISO path recorded as real host-local file paths. If the config.xml render step is blocked on diff --git a/scripts/opnsense-build-config-iso.sh b/scripts/opnsense-build-config-iso.sh deleted file mode 100644 index 37ffc73..0000000 --- a/scripts/opnsense-build-config-iso.sh +++ /dev/null @@ -1,75 +0,0 @@ -#!/usr/bin/env bash -# scripts/opnsense-build-config-iso.sh -# -# ############################################################################ -# # RETIRED (D-112) -- THIS PATH CANNOT WORK. DO NOT USE IT. # -# # # -# # The ISO this builds is INERT. It is never read. Root-caused from # -# # upstream source (opnsense/core, src/sbin/opnsense-importer): # -# # # -# # INSTALL="/.probe.for.readonly" # -# # touch ${INSTALL} 2> /dev/null # -# # if [ -f ${INSTALL} -a -f /conf/config.xml ]; then # -# # bootstrap_and_exit 0 # -# # # -# # The touch is a PROBE FOR A READ-ONLY ROOT. On INSTALLER media the root # -# # is read-only, the probe fails, and the importer goes on to scan attached # -# # media (cd9660 included). On our PRE-INSTALLED NANO image the root is # -# # WRITABLE and a factory /conf/config.xml ALREADY EXISTS -- so both # -# # conditions hold and it bootstrap_and_exit 0's IMMEDIATELY, without ever # -# # enumerating a single device. # -# # # -# # The Configuration Importer can NEVER fire on a nano image, BY DESIGN. # -# # Nothing was ever going to read this ISO. # -# # # -# # The working path is D-112(c): boot on factory defaults -> console # -# # bootstrap -> enable SSH + install the key -> configure over the network # -# # (now the REST API, per D-113(a2): scripts/opnsense-api.sh). # -# # # -# # Kept (not deleted) only because D-112 says to retire it in the same # -# # change that reduces the template -- which has NOT yet landed. Its # -# # harness still passes; that proves the ISO is well-formed, NOT that # -# # anything reads it. # -# ############################################################################ -# -# Builds a plain ISO9660 image containing /conf/config.xml, for OPNsense's -# Configuration Importer (see opentofu/README.md's "OPNsense deployment -# research" section) -- ISO9660 support was added to the Importer -# specifically so a config ISO could be attached as a secondary CD-ROM, -# mechanically identical to how modules/cloudinit-vm attaches its NoCloud -# seed via a libvirt_volume + create.content.url pointing at a local path. -# -# NOTE (updated after a 2026-07-09 audit pass): the genisoimage/xorriso flags -# below (-V volume label, -J Joliet, -R Rock Ridge) are CONFIRMED standard, -# correct usage for building a plain cross-platform ISO9660 image from a -# directory -- this part is no longer a guess. What remains genuinely -# unverified is a DIFFERENT claim: whether OPNsense's Configuration Importer -# actually reads /conf/config.xml back off an ISO9660 volume built this way -# once booted for real (neither genisoimage/xorriso nor a real OPNsense VM -# was available this session to test that end-to-end). Confirm that -# specifically before relying on it operationally. -# -# Requires: genisoimage or xorriso. Read-only w.r.t. the repo; writes only -# to and a self-cleaning temp dir. -# Exit: 0 built | 1 bad input | 2 required tool missing. -set -euo pipefail - -CONFIG_XML="${1:?usage: opnsense-build-config-iso.sh }" -OUT="${2:?usage: opnsense-build-config-iso.sh }" - -[ -r "$CONFIG_XML" ] || { echo "FAIL: cannot read $CONFIG_XML"; exit 1; } - -TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT -mkdir -p "$TMP/conf" -cp "$CONFIG_XML" "$TMP/conf/config.xml" - -if command -v genisoimage >/dev/null 2>&1; then - genisoimage -o "$OUT" -V "OPNSENSE_CFG" -J -R "$TMP" -elif command -v xorriso >/dev/null 2>&1; then - xorriso -as genisoimage -o "$OUT" -V "OPNSENSE_CFG" -J -R "$TMP" -else - echo "FAIL: genisoimage or xorriso required on PATH" - exit 2 -fi - -echo "PASS: built $OUT (contains /conf/config.xml)" diff --git a/scripts/opnsense-render-config.sh b/scripts/opnsense-render-config.sh deleted file mode 100644 index 852f1d8..0000000 --- a/scripts/opnsense-render-config.sh +++ /dev/null @@ -1,128 +0,0 @@ -#!/usr/bin/env bash -# scripts/opnsense-render-config.sh -# -# ############################################################################ -# # DANGER (2026-07-13): DO NOT PUSH THIS OUTPUT TO THE LIVE OFFICE1 EDGE. # -# # # -# # This renders a FULL config.xml. Installing a full config.xml REPLACES # -# # /conf/config.xml wholesale and DROPS ~667 migration-populated elements # -# # (measured) -- including the only 2 firewall pass rules on the box. # -# # # -# # Office1's DHCP is now API-MANAGED (D-113(a2), proven live 2026-07-13). # -# # Pushing a rendered config.xml to 10.10.0.1 WOULD CLOBBER THAT STATE. # -# # Configure the edge with scripts/opnsense-api.sh instead. # -# # # -# # This script is NOT dead: under D-113(a2) it is to be REDUCED to render a # -# # MINIMAL BOOTSTRAP config (sshd + root key + console, nothing else), for # -# # first-boot only. That reduction is NOT YET DONE. Until it lands, the # -# # only safe use of this script is rendering for a BRAND-NEW, NOT-YET-BOOTED # -# # edge -- never against one that is already up. # -# # # -# # Related: D-112 (the config-ISO delivery path is INERT -- the OPNsense # -# # Configuration Importer can never fire on a pre-installed nano image). # -# ############################################################################ -# -# Renders opentofu/templates/opnsense-config.xml.tmpl into a real config.xml -# by substituting {{TOKEN}} markers (this repo's existing clientdocs -# convention, reused here) from required environment variables -- no -# invented defaults except NTP_UPSTREAM_SERVERS/NTP_PREFER_SERVER, which -# default to the SAME public pool OPNsense's own shipped config.xml.sample -# uses (a real, confirmed default, not an invention -- see -# opentofu/templates/README.md). Everything else fails loud if unset. -# -# Feed the result to scripts/opnsense-build-config-iso.sh next. -# Exit: 0 rendered | 1 required token/template missing. -set -euo pipefail -HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -TEMPLATE="$HERE/../opentofu/templates/opnsense-config.xml.tmpl" -OUT="${1:?usage: opnsense-render-config.sh }" - -: "${NTP_UPSTREAM_SERVERS:=0.opnsense.pool.ntp.org 1.opnsense.pool.ntp.org 2.opnsense.pool.ntp.org 3.opnsense.pool.ntp.org}" -: "${NTP_PREFER_SERVER:=0.opnsense.pool.ntp.org}" - -req() { # req VARNAME -- fail loud if unset, no inferred fallback - local name="$1" - [ -n "${!name:-}" ] || { echo "FAIL: \$$name not set -- see opentofu/templates/README.md"; exit 1; } -} - -REQUIRED_VARS=( - OPNSENSE_HOSTNAME DOMAIN ROOT_PASSWORD_HASH - WAN_IF WAN_IPADDR WAN_SUBNET_BITS WAN_GATEWAY - LAN_IF LAN_IPADDR LAN_SUBNET_BITS - ROOT_AUTHORIZED_KEYS - DHCP_POOL_START DHCP_POOL_END -) -# ROOT_AUTHORIZED_KEYS: the PUBLIC key text (e.g. the contents of an id_ed25519.pub). -# REQUIRED, no default -- D-112(c) makes SSH the management path for the edge, so an edge -# rendered without a key is a lockout waiting to happen. PUBLIC key material only; the -# private key is never read by this repo. -# -# MIRROR_SYNC_PROTOCOL/MIRROR_UPSTREAM_NET/MIRROR_SYNC_PORT removed (DOCFIX-185): -# the edge is a normal simulated-ISP router, not an egress-airgap firewall -- the -# WAN mirror/NTP egress-control rules that used them are gone from the template. -for v in "${REQUIRED_VARS[@]}"; do - req "$v" -done - -# Base64 for , which OPNsense base64_decode()s into ~/.ssh/authorized_keys -# (verified upstream, src/etc/inc/auth.inc). Computed AFTER req() so a missing key fails with -# the friendly message rather than a `set -u` unbound-variable crash. -ROOT_AUTHORIZED_KEYS_B64="$(printf '%s\n' "$ROOT_AUTHORIZED_KEYS" | base64 -w0)" - -# --- LAN DHCP (Kea) ------------------------------------------------------------------- -# Derive the LAN network CIDR and a DETERMINISTIC subnet4 uuid, and VALIDATE that the pool -# actually lies inside the LAN subnet (a pool outside its subnet is silently accepted by the -# XML but serves nobody). Deterministic uuid5 => re-rendering is idempotent and does not churn -# the config. Fails loud on any inconsistency -- no inferred values. -_dhcp="$(python3 - "$LAN_IPADDR" "$LAN_SUBNET_BITS" "$DHCP_POOL_START" "$DHCP_POOL_END" <<'PY' -import sys, ipaddress, uuid -ip, bits, lo, hi = sys.argv[1], int(sys.argv[2]), sys.argv[3], sys.argv[4] -net = ipaddress.ip_network(f"{ip}/{bits}", strict=False) -a, b = ipaddress.ip_address(lo), ipaddress.ip_address(hi) -for label, addr in (("DHCP_POOL_START", a), ("DHCP_POOL_END", b)): - if addr not in net: - sys.exit(f"FAIL: ${label} {addr} is not inside the LAN subnet {net}") -if a > b: - sys.exit(f"FAIL: $DHCP_POOL_START {a} is above $DHCP_POOL_END {b}") -if ipaddress.ip_address(ip) in list(ipaddress.summarize_address_range(a, b))[0]: - sys.exit(f"FAIL: the LAN address {ip} falls inside the DHCP pool {a}-{b} (it must not)") -print(str(net)) -print(f"{a}-{b}") -print(uuid.uuid5(uuid.NAMESPACE_DNS, f"opnsense-kea-dhcp4-{net}")) -PY -)" || { echo "$_dhcp"; exit 1; } -LAN_NETWORK_CIDR="$(sed -n 1p <<<"$_dhcp")" -DHCP_POOL="$(sed -n 2p <<<"$_dhcp")" -DHCP_SUBNET_UUID="$(sed -n 3p <<<"$_dhcp")" - -[ -r "$TEMPLATE" ] || { echo "FAIL: template not found: $TEMPLATE"; exit 1; } - -route_block() { # route_block - local net="${!1:-}" gw="${!2:-}" descr="${!3:-}" - if [ -z "$net" ]; then - return 0 # no route needed at this site -- emit nothing, not a placeholder - fi - [ -n "$gw" ] || { echo "FAIL: \$$1 is set but \$$2 is not"; exit 1; } - printf ' \n %s\n %s\n %s\n 1\n \n' \ - "$net" "$gw" "${descr:-static route}" -} - -ROUTE1_BLOCK="$(route_block ROUTE1_NETWORK ROUTE1_GATEWAY ROUTE1_DESCR)" -ROUTE2_BLOCK="$(route_block ROUTE2_NETWORK ROUTE2_GATEWAY ROUTE2_DESCR)" - -content="$(cat "$TEMPLATE")" -for v in "${REQUIRED_VARS[@]}" NTP_UPSTREAM_SERVERS NTP_PREFER_SERVER \ - ROOT_AUTHORIZED_KEYS_B64 LAN_NETWORK_CIDR DHCP_POOL DHCP_SUBNET_UUID; do - content="${content//"{{$v}}"/${!v}}" -done -content="${content//"{{ROUTE1_BLOCK}}"/$ROUTE1_BLOCK}" -content="${content//"{{ROUTE2_BLOCK}}"/$ROUTE2_BLOCK}" - -if grep -qE '\{\{[A-Z0-9_]+\}\}' <<<"$content"; then - echo "FAIL: unresolved {{TOKEN}} remains in rendered output -- template/script token lists are out of sync:" - grep -oE '\{\{[A-Z0-9_]+\}\}' <<<"$content" | sort -u - exit 1 -fi - -printf '%s' "$content" > "$OUT" -echo "PASS: rendered $OUT" diff --git a/scripts/prereqs/README.md b/scripts/prereqs/README.md index 3a3900e..9387737 100644 --- a/scripts/prereqs/README.md +++ b/scripts/prereqs/README.md @@ -24,7 +24,7 @@ | `install-opentofu.sh` | OpenTofu `tofu` >= 1.6.0 (deb repo -> `/usr/bin/tofu`) | dc-dc Stage 1-6 `tofu` steps + `scripts/opentofu-validate.sh` | | `install-virtualization.sh` | libvirt + qemu-kvm + virsh + `libvirt` group | dc-dc Stage 1+ (pools, networks, node VMs) | | `install-jq.sh` | jq | cloud-assert + many ops scripts (and the test gauntlet) | -| `install-image-tools.sh` | qemu-utils (`qemu-img`) + `xorriso` | Stage 2/3 OPNsense build (`opnsense-prep-image.sh`, `opnsense-build-config-iso.sh`) | +| `install-image-tools.sh` | qemu-utils (`qemu-img`) | Stage 2/3 OPNsense image prep (`opnsense-prep-image.sh`). The `xorriso`/ISO9660 requirement was REMOVED 2026-07-13: the config-seed ISO path is deleted (D-112 -- the Importer can never fire on a nano image; D-113(a2) -- config is done over the REST API). | | `check-prereqs.sh` | (read-only report of all of the above + curl/python3/sudo) | run first, on any new workspace | | `install-all.sh` | orchestrator over the four installers + a final check | one-pass setup | | `lib-prereq.sh` | shared helpers (sourced, not run) | the installers | diff --git a/scripts/prereqs/install-image-tools.sh b/scripts/prereqs/install-image-tools.sh index c4cb753..49baf6f 100644 --- a/scripts/prereqs/install-image-tools.sh +++ b/scripts/prereqs/install-image-tools.sh @@ -1,22 +1,30 @@ #!/usr/bin/env bash # scripts/prereqs/install-image-tools.sh [--check|--dry-run] # -# Prereq installer: image/ISO tooling for the OPNsense-edge build path -# (dc-dc Stage 2/3). qemu-img (from qemu-utils) is used by -# scripts/opnsense-prep-image.sh; an ISO9660 builder (xorriso preferred, or -# genisoimage) is used by scripts/opnsense-build-config-iso.sh. Debian/Ubuntu -# (apt). Idempotent. +# Prereq installer: image tooling for the OPNsense-edge build path (dc-dc +# Stage 2/3). qemu-img (from qemu-utils) is used by scripts/opnsense-prep-image.sh. +# Debian/Ubuntu (apt). Idempotent. # -# --check report qemu-img + iso-builder presence; exit 0/1 +# THE ISO9660 BUILDER IS NO LONGER NEEDED (2026-07-13). It existed for +# scripts/opnsense-build-config-iso.sh, which built a config-seed ISO for the +# OPNsense Configuration Importer -- an importer that CANNOT FIRE on a +# pre-installed nano image (D-112: it probes for a read-only root, finds a +# writable one with a factory /conf/config.xml, and exits without enumerating a +# single device). The ISO was never read by anything. That script, and the whole +# config.xml delivery path, are DELETED under D-113(a2): edge config is done over +# the REST API (scripts/opnsense-api.sh). xorriso/genisoimage are therefore no +# longer installed or checked here. +# +# --check report qemu-img presence; exit 0/1 # --dry-run print what it would do; mutate nothing -# (no flag) install qemu-utils + xorriso (sudo) +# (no flag) install qemu-utils (sudo) # # Exit: 0 ok/installed | 1 check-failed | 2 bad args | 3 unsupported OS | 4 install failed. # ASCII + LF. set -uo pipefail HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" . "$HERE/lib-prereq.sh" -PKGS="qemu-utils xorriso" +PKGS="qemu-utils" pq_parse_mode "$@"; rc=$? [ "$rc" = "10" ] && { echo "usage: bash scripts/prereqs/install-image-tools.sh [--check|--dry-run]"; exit 0; } @@ -25,9 +33,7 @@ report() { local ok=0 pq_have qemu-img && echo "OK: qemu-img present" || { echo "ABSENT: qemu-img (qemu-utils)"; ok=1; } - if pq_have xorriso; then echo "OK: xorriso present (ISO9660 builder)" - elif pq_have genisoimage; then echo "OK: genisoimage present (ISO9660 builder)" - else echo "ABSENT: no ISO9660 builder (xorriso or genisoimage)"; ok=1; fi + # No ISO9660 builder check: the config-seed ISO path is DELETED (D-112/D-113(a2)). return "$ok" } diff --git a/tests/opnsense-build-config-iso/run-tests.sh b/tests/opnsense-build-config-iso/run-tests.sh deleted file mode 100644 index 1ad46b3..0000000 --- a/tests/opnsense-build-config-iso/run-tests.sh +++ /dev/null @@ -1,38 +0,0 @@ -#!/usr/bin/env bash -# tests/opnsense-build-config-iso/run-tests.sh -- offline harness for -# scripts/opnsense-build-config-iso.sh. Only tests the guard clauses this -# environment can actually exercise. Whether the built ISO is actually -# picked up by OPNsense's Configuration Importer is UNTESTED here (no -# genisoimage/xorriso and no real OPNsense VM were available this session) -# -- logged as residual risk, not hidden; see opentofu/README.md. -# Exit: 0 all pass | 1 any case failed. ASCII + LF. -set -uo pipefail -HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -SCRIPT="$HERE/../../scripts/opnsense-build-config-iso.sh" -PASS=0; FAIL=0 - -run() { # run