diff --git a/docs/design-decisions.md b/docs/design-decisions.md index 8762c14..d5f1924 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -2164,3 +2164,71 @@ **References:** `netbox/dc-dc-prefixes-import.py` (`carve_v6`), `docs/dc-dc-netbox-buildout-scope.md` section 5.1, D-101 (family matrix), D-052/D-053 (six-plane), the 2026-07-11 live NetBox export. Harness `tests/dc-dc-prefixes-import/` 53/53. Applies to the Office1 simulation NetBox first; flows back to `netbox.baldurkeep.com` as refined draft. --- + +## D-112: OPNsense edge config delivery -- the Configuration Importer cannot work on a nano image + +**Status:** PROPOSED 2026-07-12. Operator ruling required -- this replaces the config-delivery +mechanism of a BUILT surface (`opentofu/modules/opnsense-edge` + `scripts/opnsense-build-config-iso.sh`). +Options are presented; none is implemented. + +**Finding (MEASURED, not inferred).** The Office1 edge boots (DOCFIX-188 memory + DOCFIX-190 ACPI) +but comes up on FACTORY DEFAULTS. The config ISO is well-formed and correctly attached -- verified: +ISO9660 labelled `OPNSENSE_CFG`, containing `CONF/CONFIG.XML` with our `` payload and +`10.10.0.1`; the guest sees it as `cd0` (`QEMU QEMU DVD-ROM`, 186x2048 = exactly our 380928-byte +image). The boot log shows the importer being invoked and then doing NOTHING: + + >>> Invoking import script 'importer' + (no further output) + +**Root cause -- architectural, not a bug.** `rc.syshook.d/import/20-importer` runs +`/usr/local/sbin/opnsense-importer -b`. Upstream source (opnsense/core, `src/sbin/opnsense-importer`): + + INSTALL="/.probe.for.readonly" + ... + if [ -n "${DO_BOOT}" ]; then + touch ${INSTALL} 2> /dev/null + if [ -f ${INSTALL} -a -f /conf/config.xml ]; then + bootstrap_and_exit 0 + fi + +The `touch` is a PROBE FOR A READ-ONLY ROOT. On INSTALLER media the root is read-only, the touch +fails, the marker is absent -- so the importer proceeds to scan attached media (and yes, its scan +path does support cd9660). On a PRE-INSTALLED image (our nano) the root is WRITABLE and a factory +`/conf/config.xml` ALREADY EXISTS, so BOTH conditions hold and it `bootstrap_and_exit 0`s +immediately -- it never enumerates a single device. + +**Therefore the Configuration Importer can NEVER fire on a pre-installed nano image, by design.** +Nothing was ever going to read our ISO. + +**What the prior research got right and wrong** (`opentofu/README.md` "OPNsense deployment +research", items 2-3): it correctly established that the Importer exists and that its scan path +supports ISO9660 for VM automation. It never established that the Importer would RUN AT ALL on a +nano image -- and it does not. The module header has always flagged this exact mechanism UNVERIFIED; +that flag was correct and is now discharged NEGATIVE. + +**Options (operator to rule; none implemented):** + +- **(a) Bake `config.xml` into the image before first boot.** `opnsense-prep-image.sh` writes + `/conf/config.xml` directly into the nano's UFS root. Cleanest and most immutable-infra; no + importer, no ISO, no boot-time interactivity. RISK/UNVERIFIED: writing FreeBSD UFS2 from Linux -- + most distro kernels ship `CONFIG_UFS_FS_WRITE` off, so this may be infeasible without libguestfs + or a FreeBSD helper. FEASIBILITY MUST BE PROVEN BEFORE ADOPTING. +- **(b) Switch from the nano image to the installer image.** On installer media the root IS + read-only, so the Importer runs and the existing ISO mechanism works AS DESIGNED and unchanged. + Cost: an actual install step per edge; needs checking whether it is non-interactive. +- **(c) Boot factory-default, then push config over the network.** Provision post-boot via + OPNsense's API/SSH from Office1 (factory default LAN is `192.168.1.1/24` on `office1-local`, which + we own). Most Roosevelt-transferable -- real edges get provisioned over the network too -- but it + is two-phase and needs a bootstrap path onto the default LAN. +- **(d) Serial-console expect-scripting.** Drive the interactive importer prompt over the serial + console. NOT RECOMMENDED: fragile, timing-dependent (the prompt has a 7-second timeout). + +**Impacts if not (b):** `scripts/opnsense-build-config-iso.sh`, its harness, the module's +`config_seed` volume + cdrom disk, and the README research section all assume the ISO mechanism. + +**References:** `docs/incident-20260712-opnsense-edge-boot-triplefault.md`, +`docs/changelog-20260712-libvirt-acpi-kernel-panic.md` (DOCFIX-190), upstream +`opnsense/core:src/sbin/opnsense-importer` + `src/etc/rc.syshook.d/import/20-importer`. +Also gates Stage 3 (the per-DC OPNsense edges use this same module). + +--- diff --git a/docs/session-ledger.md b/docs/session-ledger.md index baedb0b..11d9b4f 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -1822,7 +1822,27 @@ creation mode sets the ACL **mask** to `---`, nullifying the named-user entry (measured: `user:jessea123:r-- #effective:---`). Proposed (NOT yet verified): pre-create the log 0644 before `tofu apply` so libvirt appends to an already-readable file. -- **Numbers:** D next-free 112, DOCFIX next-free 191. +- **BUG 3 ROOT-CAUSED -> D-112 PROPOSED (operator ruling required; NOT implemented).** The + Configuration Importer **can never fire on a pre-installed nano image, BY DESIGN.** Upstream + `opnsense-importer -b` does `touch /.probe.for.readonly` -- a PROBE FOR A READ-ONLY ROOT -- + and `if [ -f ${INSTALL} -a -f /conf/config.xml ]; then bootstrap_and_exit 0; fi`. On INSTALLER + media root is read-only so the touch fails and it scans media (cd9660 supported). On our nano, + root is WRITABLE and a factory `/conf/config.xml` ALREADY EXISTS -> both true -> it exits + immediately, never enumerating a device. Boot log confirms: `>>> Invoking import script + 'importer'` then NOTHING. Our ISO was fine all along (verified: ISO9660 `OPNSENSE_CFG`, + `CONF/CONFIG.XML` with 10.10.0.1; guest sees it as cd0, exact byte size) -- nothing was ever + going to read it. + - The README research (items 2-3) correctly established the Importer supports ISO9660; it NEVER + established the Importer RUNS on a nano. The module's own UNVERIFIED flag was right, and is + now discharged NEGATIVE. + - **Options in D-112 (operator picks):** (a) bake config.xml into the image in prep-image + [cleanest; UFS2-write-from-Linux feasibility UNPROVEN -- must verify first]; (b) switch nano -> + installer image [the ISO mechanism then works AS DESIGNED, unchanged]; (c) boot factory-default + then push config via API/SSH over office1-local [most Roosevelt-transferable]; (d) serial expect + scripting [NOT recommended -- 7s timeout, fragile]. + - Impacts if not (b): `opnsense-build-config-iso.sh` + harness, the module's config_seed/cdrom, + the README research section. ALSO GATES STAGE 3 (per-DC edges use this same module). +- **Numbers:** D next-free 113, DOCFIX next-free 191.