Newer
Older
openstack-caracal-dc-dc / docs / opnsense-edge-addon-review-20260718.md

OPNsense edge -- plugin / add-on review (2026-07-18)

Subject. office1-opnsense, OPNsense 26.7 (amd64), reviewed for plugin/add-on additions that benefit the VR1 build and transfer to the Roosevelt edge. Read-only review; nothing was installed. Adopting any item is a design decision (see D-129, PROPOSED).

Method + the lens that sorts the catalog

  • Measured state (2026-07-18): version 26.7, 0 outdated packages, no pending firmware. Installed plugins: os-git-backup only (and it is UNCONFIGURED -- see Findings). Available catalog: 104 plugins (208 rows incl. -devel).
  • Edge role bounds the review. The edge is the per-site BOUNDARY firewall/router (D-100 / D-122): 2 NICs (WAN + LAN), REST-API managed (D-113). The OVN/OpenStack fabric does inter-plane routing; the edge is NOT an app-services host, NOT an inter-plane router, NOT the cloud's DNS/LB/mail/proxy tier. That excludes most of the catalog by construction.
  • Sorting lens = minimize-delta-to-Roosevelt. "Base deployment" means the per-site edge profile that ultimately lands on Roosevelt baremetal. VR1's edge is a libvirt guest; some items help the virtual rehearsal but do not transfer, and some are the reverse.

Bottom line

Bucket Items Do it? Transfers to Roosevelt?
A. Roosevelt base-edge hardware profile os-smart, os-nut or os-apcupsd, os-cpu-microcode-intel/-amd, os-lldpd Record in the metal edge profile YES (inert in VR1 VMs)
B. VR1 sim scaffolding os-qemu-guest-agent Add on VR1 VM edges now NO -- uninstall at metal (like netem)
C. Buildout diagnostics os-iperf Add on edges during buildout Optional (keep or drop at metal)
D. Observability (COS/Prometheus) os-node_exporter CONDITIONAL -- needs a scope ruling YES if adopted
E. Time (NTP) core ntpd (opt. os-chrony) Confirm site NTP anchor design YES
F. Decisions, not adds os-frr, os-tailscale Ruling required (D-129) Depends on ruling
G. Security hardening Suricata (core) + os-intrusion-detection-content-*, os-crowdsec, os-sunnyvalley, os-maltrail, os-acme-client, os-wazuh-agent DEFER (closed-test posture) YES, at Roosevelt / post-teardown

Actually do soon (low-risk, high-value): wire os-git-backup (Findings #1), os-qemu-guest-agent on the VR1 VM edges, os-iperf for transit validation. Everything else is either "record for the metal profile" or "needs a ruling".

Buckets in detail

A. Roosevelt base-edge hardware profile -- the real "add to base deployments" answer

These belong in the per-site metal edge profile. They cannot be exercised in VR1 (virtual edge), so record them now and enable at Roosevelt; they are no-ops/uninstalls in the sim.

  • os-smart -- SMART disk health/alerting on the edge appliance's physical disks.
  • os-nut OR os-apcupsd -- UPS monitoring + clean shutdown on power loss. Pick one per the actual UPS (NUT is the broader/standard choice).
  • os-cpu-microcode-intel / os-cpu-microcode-amd -- CPU microcode updates (security/stability).
  • os-lldpd -- LLDP neighbor/topology discovery; valuable for mapping the metal edge into the rack fabric. (os-hw-probe / os-dmidecode are optional one-shot hardware-inventory helpers.)

B. VR1 sim scaffolding -- useful NOW, does not transfer

  • os-qemu-guest-agent -- the VR1 edge is a KVM guest (Office1 on vcloud; DC edges nested in vvr1-dc0 under Model B). The guest agent gives the libvirt host clean guest coordination: guest-initiated graceful shutdown, fsfreeze for consistent snapshots, and guest IP reporting to virsh. This directly strengthens the reboot/autostart work already in flight (D-127 autostart matrix, D-126 base-leg boot-race). VM-only -> increases Roosevelt delta; bill it exactly like netem: rehearsal aid, uninstall at metal. (os-virtualbox/os-vmware/os-xen are the wrong- hypervisor equivalents -- we are KVM, so os-qemu-guest-agent is the correct one.)

C. Buildout diagnostics -- going forward

  • os-iperf -- on-edge throughput testing. Genuinely useful for the DC-DC buildout: validate the East-West transit / dark-fiber path (D-122/D-124) and sanity-check the netem rehearsal shaping from a real endpoint. Add to edges during the transit-wiring stages; keep or drop at metal.

D. Observability -- grounded on the actual stack (COS)

The deployment's monitoring stack is COS (Canonical Observability Stack, Prometheus-based). Per D-105 it is per-DC and deployed INSIDE each cloud (Juju, Stage 7) -- it is not an Office1/edge roll-up, and the edge sits OUTSIDE that failure domain (edge LAN faces provider-public, D-100).

  • os-node_exporter is the Prometheus-native fit -- COS can scrape it directly.
  • CONDITIONAL: whether COS should scrape the boundary edge at all is a scope decision (D-105 scoped COS to the cloud, not the edge). If edge-in-COS is wanted, os-node_exporter is the pick.
  • Alternatives for other stacks (NOT recommended over node_exporter given COS): os-telegraf (multi-output), os-net-snmp (SNMP polling), os-zabbix7-agent (Zabbix), os-netdata (local UI). os-collectd/os-munin-node/os-nrpe are legacy monitoring -- skip.

E. Time (NTP)

NTP is core (ntpd); os-chrony is an optional chrony backend. Not an "add" so much as a design point to confirm: a distributed cloud needs tight time sync (Ceph, Keystone tokens, TLS validity, correlated logs). Decide whether the edge anchors site NTP (stratum source for the site network) or whether time comes from elsewhere (MAAS/upstream). Record the answer per site.

F. Decisions, not adds (present, do not implement)

  • os-frr (FRRouting -- OSPF/BGP). Our transit is static (D-124 transit /30, static rack route, single ISP uplink per site under D-125). FRR is unnecessary for VR1 and is a Roosevelt- scale question (dynamic inter-site routing / BGP to ISPs). Defer to a Roosevelt routing ruling.
  • os-tailscale -- native edge Tailscale. We currently run Tailscale as a separate LXD subnet-router VM (office1-tailscale, advertising 10.10.0.0/22, D-107). The plugin could consolidate that onto the edge (one fewer VM) but couples the overlay client to the boundary firewall and changes the D-107 posture. Consolidation is a ruling, not a casual add.

G. Security hardening -- DEFERRED per closed-test posture

Listed so they are not forgotten; relevant at Roosevelt / post-teardown, not in the closed test (project posture: real-world security deferred).

  • Suricata IDS/IPS is CORE (built-in); the plugins provide only rule CONTENT: os-intrusion-detection-content-et-open (free ET Open), -et-pro, -snort-vrt, -ptopen.
  • os-crowdsec -- behavioral IPS / collaborative blocklists.
  • os-sunnyvalley -- Zenarmor NGFW (L7 app control) -- heavyweight, commercial tiers; a real option for the Roosevelt edge but a significant addition. Note, do not adopt lightly.
  • os-maltrail (malicious-traffic detection), os-wazuh-agent/os-beats (SIEM log shipping), os-acme-client (Let's Encrypt real TLS for GUI/API -- self-signed is fine in the closed test; wanted at Roosevelt). os-etpro-telemetry/os-q-feeds-connector -- threat feeds.

H. Reviewed and rejected (with reason) -- so the review is exhaustive, not cherry-picked

  • Modern VPN is CORE, so the plugins are not needed: OPNsense 26.x ships WireGuard, IPsec and OpenVPN in core (the catalog only offers os-openvpn-legacy / os-strongswan-legacy). Inter-site is dark fiber East-West (D-122), not a VPN, so even core VPN is not on the critical path.
  • DHCP: already on the modern core. The edge runs Kea (core); os-isc-dhcp is the LEGACY ISC server -- do not add.
  • App services that are not the edge's role: os-haproxy (Octavia does cloud LB), os-relayd, os-squid/os-OPNProxy/os-web-proxy-sso/os-c-icap/os-ftp-proxy (proxy), os-nginx/os-caddy (web), os-postfix/os-rspamd/os-clamav/os-redis (mail/AV/cache), os-freeradius/ os-radsecproxy (RADIUS), os-bind (Designate owns cloud DNS, D-106; os-ddclient/ os-cloudflared/os-rfc2136 dynamic-DNS paths are likewise not ours).
  • Overlay/tunnel alternatives we do not use (we are committed to Tailscale, D-107): os-netbird, os-zerotier, os-openconnect, os-tinc, os-tor, os-shadowsocks.
  • Config-mgmt / cloud SDKs that are not our tooling (we use Juju + MAAS + OpenTofu + REST): os-puppet-agent, os-google-cloud-sdk.
  • L2 relays / PXE bits that do not cross the edge: os-mdns-repeater, os-igmp-proxy, os-udpbroadcastrelay (our PXE/mDNS stays on the internal planes), os-tftp (MAAS serves TFTP internally, not the edge).
  • Niche / cosmetic / hardware-specific: os-siproxd/os-turnserver (VoIP), os-upnp, os-lcdproc-sdeclcd (LCD panel), os-wol, os-theme-* (cosmetic), os-grid_example/ os-helloworld (dev samples), os-realtek-re/os-dec-hw (specific NIC/appliance drivers), os-ntopng/os-vnstat (optional traffic visibility -- heavier; revisit only if deep flow analytics are wanted).

Findings (already-true state, not recommendations)

  1. os-git-backup is installed but UNCONFIGURED -- there is no config node for it, so no config backup is running. That is a false sense of safety. Recommend wiring it to push the edge config.xml to git.baldurkeep.com (the git service of record) -- this is the single highest-value hygiene item and aligns the edge with the repo-of-record / config-as-code discipline. Alternatively remove the plugin so it does not imply protection it is not providing.
  2. Version drift recorded: the edge moved 26.1.11 -> 26.7. docs/vr1-office1-as-built.md updated (the VR1 infra version home; appendix-B is charm-channels only).
  3. Core features to confirm-configured (not plugins): Kea DHCP (in use), Suricata (core, off -- defer), NTP (core -- see E), WireGuard/IPsec (core, available if ever needed).

Adoption path (discipline)

Any adoption is a D-129 ruling + a provisioning change: bake the chosen plugin install into the edge bootstrap over the REST-API path (D-113) via the OPNsense firmware API (/api/core/firmware/install), delivered with the touched script's test harness and a changelog entry with a revert. This review installs nothing. The recommended near-term set (git-backup wiring, qemu-guest-agent on VR1 edges, iperf for transit validation) can proceed as soon as the operator rules D-129.