diff --git a/docs/design-decisions.md b/docs/design-decisions.md index 65dfb3b..fe0b762 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -2309,3 +2309,104 @@ Also gates Stage 3 (the per-DC OPNsense edges use this same module). --- + +## D-113: is OPNsense the right edge platform, or should the simulated edges be VyOS/Linux? + +**Status:** PROPOSED 2026-07-13 -- operator asked, after a five-bug day building the Office1 +edge, whether something is better suited. Options and evidence below; NOT ruled. Do not +migrate, and do not build edges 2/3, until this is ruled. + +**Scope.** The VR1 edges are SIMULATED customer-site / ISP routers (Office1 today; Stage 3 adds +one per DC). They are lab infrastructure, not a product surface. Nothing a tenant touches +depends on this choice; what depends on it is our own automation cost, for three edges. + +### First, an honest cost accounting of 2026-07-12 (five bugs) + +| # | bug | OPNsense's fault? | +|---|---|---| +| DOCFIX-188 | guest got 2 MiB RAM (`memory_unit`) -- triple fault | **NO** -- libvirt module bug; would hit ANY guest | +| DOCFIX-190 | `acpi=off` -> FreeBSD kernel panic | **NO** -- libvirt module bug; generic | +| DOCFIX-191 | rendered config had no sshd + no key -> would have locked management out | yes (hand-authored `config.xml`) | +| DOCFIX-192 | rendered config silenced the only console | yes (hand-authored `config.xml`) | +| DOCFIX-193 | LAN DHCP was never implemented (and ISC `` would have been inert -- 26.1 is Kea) | yes (hand-authored `config.xml`) | + +Two of five were platform-agnostic libvirt bugs whose fixes now protect every VR1 VM +(MAAS/NetBox/GitBucket, the DC nodes). Finding them via OPNsense was lucky, not costly. + +**The other three share ONE root cause: we hand-author the appliance's internal, GUI-owned +`config.xml`.** That is a known-bad idea in any product of this class -- and pfSense would be +identical, since it is the same format lineage. + +**2026-07-13 adds a fourth data point of the same kind.** A full-config push REPLACES +`/conf/config.xml`; the rendered template is 128 elements against a live 796, so it DROPS ~667 +migration-populated elements -- including the only two firewall pass rules on the box +("Default allow LAN to any"), against a pf base policy of block-drop-all. It works only because +OPNsense regenerates them on boot (proven across 4 push+boot cycles, `/conf/backup/` trail). +**We are depending on an undocumented self-heal of an internal format.** That is the ongoing +cost, and it is the thing to weigh -- not the boot bugs. + +### THE CONFOUND (must be resolved before ruling) + +**D-112 ADOPTED option (c) with the explicit rationale "C since it is the way people automate +opnsense" -- i.e. the OPNsense REST API (`opn-cli`, the Ansible OPNsense collection).** + +**What we actually built is NOT that.** We render a full `config.xml` from a Jinja-ish template +and scp it over the wire. That is a config-file bake delivered post-boot -- it satisfies the +letter of (c) (provisioning happens over the network, after boot) but NOT the rationale the +ruling was made on. + +**Every OPNsense-specific bug we are now charging against the platform (DOCFIX-191/192/193, and +the 667-element drop) is an artifact of hand-authoring XML -- exactly what the REST API exists +to avoid.** The API models DHCP, sshd, console, and firewall rules as first-class resources; +none of those three bugs is expressible through it. + +So the fair comparison is not "OPNsense vs VyOS". It is: + +- **(a1) OPNsense + `config.xml` templating** -- the status quo. Cheapest right now; every new + feature (VPN, HA, VLANs) means reverse-engineering another slice of an internal format, and + the self-heal dependency above is permanent. +- **(a2) OPNsense + the REST API** -- what D-112's rationale actually called for. Config becomes + declarative API calls; the internal format stops being our problem. Cost: retire the template, + build/adopt an API client (`opn-cli` is packaged and mainstream), and keep only a MINIMAL + bootstrap config (sshd + key + console) as a file. The boot bugs stay fixed either way. +- **(b) VyOS** -- purpose-built for this. Plain declarative text config, first-class cloud-init, + a real config API; it exists to be a lab/edge router. Friction: LTS images are + subscription-gated (rolling is free), it is less "appliance-like" than a real site router, and + switching discards a WORKING, routing, DHCP-serving edge to re-run discovery on a new platform. +- **(c) Plain Linux router** (Debian/Ubuntu + nftables + Kea + FRR, provisioned by cloud-init). + Maximum fidelity to the rest of our tooling -- same image pipeline, same cloud-init, same + Ansible/OpenTofu idioms as every other VR1 VM, and zero appliance-internal formats. Friction: + we hand-build router behaviour that OPNsense/VyOS ship; least faithful to "a real site runs an + appliance". +- **(d) pfSense** -- **REJECT.** Same config-format lineage as OPNsense; buys nothing, costs a + migration. + +### Recommendation (mine; the operator rules) + +**Stay on OPNsense -- but the real question is (a1) vs (a2), and I lean (a2).** + +Rationale for staying: the hard part is DONE and MEASURED WORKING (boots, routes, NATs, reaches +the internet, serves DHCP, managed over SSH with a verified push pipeline). Edges 2 and 3 reuse +the module and template -- different tokens, same everything -- so the cost paid was one-time, +not per-edge. Switching platforms discards that and re-runs the same class of discovery +elsewhere. VyOS is the defensible blank-sheet pick, but we are not on a blank sheet. + +Rationale for (a2) over (a1): (a1)'s cost COMPOUNDS. Each new feature is another slice of +reverse-engineered XML plus a self-heal we do not control. (a2) pays a bounded, one-time cost to +get onto the path D-112's ruling actually named, and it is the honest reading of that ruling. + +**What would change my mind entirely (i.e. rule (b)/(c)):** if, having moved to (a2), the API +still forces us to reverse-engineer the config format for each new feature -- then OPNsense is +not buying us the appliance abstraction we are paying for, and VyOS's plain-text config wins. + +### Cost of deferring + +Low but not zero: Stage 3 builds two more edges from this same module. Ruling BEFORE Stage 3 +means the choice is made once; ruling after means migrating three edges instead of one. + +**Evidence:** `docs/changelog-20260712-office1-opnsense-edge-build.md`, +`docs/changelog-20260712-opnsense-edge-boot-fixes.md`, +`docs/changelog-20260713-office1-dhcp-apply.md` (the 667-element self-heal, measured), +D-112 (the (c) ruling and its stated rationale). + +--- diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 98f8834..af2ba8b 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -1943,6 +1943,17 @@ (c) `store` DELETES the credential on a 401, so a rejected token silently empties the file. SEC-005 is OPEN: plaintext account-wide token on the jumphost -- revoke/rotate at v1 close. +- **D-113 PROPOSED (2026-07-13, UNRULED):** is OPNsense the right edge platform? Written up + with alternatives (a1 OPNsense+config.xml = status quo, a2 OPNsense+REST API, b VyOS, + c plain-Linux router, d pfSense=reject). **Key finding while drafting: D-112 adopted (c) on + the rationale "the way people automate opnsense" = the REST API, but what we BUILT is + full-config.xml templating over scp -- which satisfies (c)'s letter, not its rationale. + All three OPNsense-specific bugs (DOCFIX-191/192/193) plus the 667-element self-heal + dependency are artifacts of hand-authoring XML -- exactly what the API exists to avoid.** + So the real question is a1 vs a2, not OPNsense vs VyOS. My lean: stay on OPNsense, move to + (a2). NOT RULED -- operator decides. Rule BEFORE Stage 3 (it builds 2 more edges from this + same module; ruling after means migrating 3 edges instead of 1). + ## State facts to remember - beta cluster left at **node_count=2** (deliberate; bonus resize acceptance coverage).