diff --git a/docs/design-decisions.md b/docs/design-decisions.md index 5f0b114..7fd97ee 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -4011,3 +4011,45 @@ (inner-root-from-Office1), `docs/dc-dc-deployment-workflow.md` (Cross-cutting: the plane split as doctrine), the operating skill (invariant). **Revert:** delete this entry + the skill invariant; the operating model has no built-surface footprint to unwind. + +## D-129: OPNsense edge plugin / add-on base profile (VR1 -> Roosevelt) + +**Status:** PROPOSED 2026-07-18 (operator asked for a plugin/add-on review of the 26.7 edge for +additions that benefit the stack going forward). Options presented below; NOTHING installed. The +operator rules which buckets are adopted; only then does implementation proceed. + +**Problem.** `office1-opnsense` was updated to OPNsense 26.7 and is a near-stock install (only +`os-git-backup`, itself unconfigured). The edge is the reusable per-site boundary profile that will +land on Roosevelt baremetal, so "what should be on the base edge?" is a real, transferable question. +Full catalog review: `docs/opnsense-edge-addon-review-20260718.md` (104 plugins, all categorized). + +**Recommendation (for the operator to rule, bucket by bucket).** +- **Roosevelt metal edge profile (record now, inert in VR1 VMs):** `os-smart`, `os-nut` OR + `os-apcupsd`, `os-cpu-microcode-intel`/`-amd`, `os-lldpd`. These are the genuine base-deployment + items; they only run on baremetal. +- **VR1 sim scaffolding (add on VR1 VM edges now; uninstall at metal, like netem):** + `os-qemu-guest-agent` -- clean libvirt-guest coordination; aids the D-126/D-127 reboot work. +- **Buildout diagnostics:** `os-iperf` on edges to validate DC-DC transit throughput (D-122/D-124). +- **Observability -- CONDITIONAL:** `os-node_exporter` (Prometheus-native, matches the COS stack) IF + the operator rules the boundary edge in-scope for COS scraping. D-105 scoped COS per-DC INSIDE the + cloud; the edge is outside that failure domain -- so this is a scope decision, not an automatic add. +- **Defer per closed-test posture (Roosevelt / post-teardown):** Suricata (core) + IDS content + plugins, `os-crowdsec`, `os-sunnyvalley` (Zenarmor), `os-maltrail`, `os-acme-client`, SIEM agents. + +**Open decision points.** (1) which buckets to adopt; (2) COS-scrapes-the-edge scope (D-105); (3) +`os-frr` -- dynamic routing is unneeded for VR1's static transit, a Roosevelt-scale question; (4) +`os-tailscale` -- consolidate the separate subnet-router VM onto the edge vs keep D-107 as-is; (5) +whether the edge anchors site NTP. Items 3-5 are their own sub-rulings. + +**Adjacent finding (not part of the ruling):** `os-git-backup` is installed but UNCONFIGURED -- no +config backup is running. Recommend wiring it to `git.baldurkeep.com` (config-as-code) regardless of +D-129's outcome; tracked in the review doc's Findings. + +**Implementation path (once ruled).** Bake the chosen plugin set into the edge bootstrap over the +REST-API path (D-113) via the OPNsense firmware API (`/api/core/firmware/install`), with the touched +script's harness + a changelog entry with a revert. VR1 vs Roosevelt sets differ (sim scaffolding is +VR1-only; the hardware profile is metal-only). + +**Related:** D-100/D-122 (edge = boundary role), D-113 (REST-API config), D-105 (COS scope), D-106 +(Designate owns DNS), D-107 (Tailscale subnet-router VM), D-124/D-125 (static transit), D-127 +(autostart), D-126 (base-leg). **Revert:** delete this entry + the review doc; nothing was installed. diff --git a/docs/opnsense-edge-addon-review-20260718.md b/docs/opnsense-edge-addon-review-20260718.md new file mode 100644 index 0000000..0eb38e7 --- /dev/null +++ b/docs/opnsense-edge-addon-review-20260718.md @@ -0,0 +1,143 @@ +# OPNsense edge -- plugin / add-on review (2026-07-18) + +**Subject.** `office1-opnsense`, **OPNsense 26.7** (amd64), reviewed for plugin/add-on +additions that benefit the VR1 build and transfer to the Roosevelt edge. Read-only review; +**nothing was installed.** Adopting any item is a design decision (see D-129, PROPOSED). + +## Method + the lens that sorts the catalog + +- **Measured state (2026-07-18):** version 26.7, **0 outdated packages**, no pending firmware. + Installed plugins: **`os-git-backup` only** (and it is **UNCONFIGURED** -- see Findings). + Available catalog: **104 plugins** (208 rows incl. `-devel`). +- **Edge role bounds the review.** The edge is the per-site **BOUNDARY** firewall/router + (D-100 / D-122): 2 NICs (WAN + LAN), REST-API managed (D-113). The OVN/OpenStack fabric does + inter-plane routing; the edge is NOT an app-services host, NOT an inter-plane router, NOT the + cloud's DNS/LB/mail/proxy tier. That excludes most of the catalog by construction. +- **Sorting lens = minimize-delta-to-Roosevelt.** "Base deployment" means the per-site edge + profile that ultimately lands on Roosevelt **baremetal**. VR1's edge is a libvirt guest; some + items help the *virtual* rehearsal but do not transfer, and some are the reverse. + +## Bottom line + +| Bucket | Items | Do it? | Transfers to Roosevelt? | +|---|---|---|---| +| A. Roosevelt base-edge **hardware** profile | `os-smart`, `os-nut` or `os-apcupsd`, `os-cpu-microcode-intel`/`-amd`, `os-lldpd` | Record in the metal edge profile | YES (inert in VR1 VMs) | +| B. VR1 **sim scaffolding** | `os-qemu-guest-agent` | Add on VR1 VM edges now | NO -- uninstall at metal (like netem) | +| C. Buildout **diagnostics** | `os-iperf` | Add on edges during buildout | Optional (keep or drop at metal) | +| D. **Observability** (COS/Prometheus) | `os-node_exporter` | CONDITIONAL -- needs a scope ruling | YES if adopted | +| E. **Time** (NTP) | core `ntpd` (opt. `os-chrony`) | Confirm site NTP anchor design | YES | +| F. **Decisions, not adds** | `os-frr`, `os-tailscale` | Ruling required (D-129) | Depends on ruling | +| G. **Security hardening** | Suricata (core) + `os-intrusion-detection-content-*`, `os-crowdsec`, `os-sunnyvalley`, `os-maltrail`, `os-acme-client`, `os-wazuh-agent` | DEFER (closed-test posture) | YES, at Roosevelt / post-teardown | + +**Actually do soon (low-risk, high-value):** wire `os-git-backup` (Findings #1), +`os-qemu-guest-agent` on the VR1 VM edges, `os-iperf` for transit validation. Everything else is +either "record for the metal profile" or "needs a ruling". + +## Buckets in detail + +### A. Roosevelt base-edge hardware profile -- the real "add to base deployments" answer +These belong in the per-site **metal** edge profile. They cannot be exercised in VR1 (virtual +edge), so record them now and enable at Roosevelt; they are no-ops/uninstalls in the sim. +- **`os-smart`** -- SMART disk health/alerting on the edge appliance's physical disks. +- **`os-nut` OR `os-apcupsd`** -- UPS monitoring + clean shutdown on power loss. Pick one per the + actual UPS (NUT is the broader/standard choice). +- **`os-cpu-microcode-intel` / `os-cpu-microcode-amd`** -- CPU microcode updates (security/stability). +- **`os-lldpd`** -- LLDP neighbor/topology discovery; valuable for mapping the metal edge into the + rack fabric. (`os-hw-probe` / `os-dmidecode` are optional one-shot hardware-inventory helpers.) + +### B. VR1 sim scaffolding -- useful NOW, does not transfer +- **`os-qemu-guest-agent`** -- the VR1 edge is a KVM guest (Office1 on vcloud; DC edges nested in + `vvr1-dc0` under Model B). The guest agent gives the libvirt host clean guest coordination: + guest-initiated graceful shutdown, `fsfreeze` for consistent snapshots, and guest IP reporting to + `virsh`. This directly strengthens the reboot/autostart work already in flight (D-127 autostart + matrix, D-126 base-leg boot-race). **VM-only -> increases Roosevelt delta**; bill it exactly like + netem: rehearsal aid, uninstall at metal. (`os-virtualbox`/`os-vmware`/`os-xen` are the wrong- + hypervisor equivalents -- we are KVM, so `os-qemu-guest-agent` is the correct one.) + +### C. Buildout diagnostics -- going forward +- **`os-iperf`** -- on-edge throughput testing. Genuinely useful for the DC-DC buildout: validate the + East-West transit / dark-fiber path (D-122/D-124) and sanity-check the netem rehearsal shaping from + a real endpoint. Add to edges during the transit-wiring stages; keep or drop at metal. + +### D. Observability -- grounded on the actual stack (COS) +The deployment's monitoring stack is **COS** (Canonical Observability Stack, Prometheus-based). Per +**D-105 it is per-DC and deployed INSIDE each cloud** (Juju, Stage 7) -- it is not an Office1/edge +roll-up, and the edge sits OUTSIDE that failure domain (edge LAN faces provider-public, D-100). +- **`os-node_exporter`** is the Prometheus-native fit -- COS can scrape it directly. +- **CONDITIONAL:** whether COS should scrape the boundary edge at all is a **scope decision** (D-105 + scoped COS to the cloud, not the edge). If edge-in-COS is wanted, `os-node_exporter` is the pick. +- Alternatives for other stacks (NOT recommended over node_exporter given COS): `os-telegraf` + (multi-output), `os-net-snmp` (SNMP polling), `os-zabbix7-agent` (Zabbix), `os-netdata` (local UI). + `os-collectd`/`os-munin-node`/`os-nrpe` are legacy monitoring -- skip. + +### E. Time (NTP) +NTP is **core** (`ntpd`); `os-chrony` is an optional chrony backend. Not an "add" so much as a +**design point to confirm**: a distributed cloud needs tight time sync (Ceph, Keystone tokens, TLS +validity, correlated logs). Decide whether the edge anchors **site NTP** (stratum source for the +site network) or whether time comes from elsewhere (MAAS/upstream). Record the answer per site. + +### F. Decisions, not adds (present, do not implement) +- **`os-frr`** (FRRouting -- OSPF/BGP). Our transit is **static** (D-124 transit /30, static rack + route, single ISP uplink per site under D-125). FRR is unnecessary for VR1 and is a Roosevelt- + **scale** question (dynamic inter-site routing / BGP to ISPs). Defer to a Roosevelt routing ruling. +- **`os-tailscale`** -- native edge Tailscale. We currently run Tailscale as a **separate** LXD + subnet-router VM (`office1-tailscale`, advertising `10.10.0.0/22`, D-107). The plugin could + consolidate that onto the edge (one fewer VM) but couples the overlay client to the boundary + firewall and changes the D-107 posture. Consolidation is a **ruling**, not a casual add. + +### G. Security hardening -- DEFERRED per closed-test posture +Listed so they are not forgotten; relevant at **Roosevelt / post-teardown**, not in the closed test +(project posture: real-world security deferred). +- **Suricata IDS/IPS is CORE** (built-in); the plugins provide only rule CONTENT: + `os-intrusion-detection-content-et-open` (free ET Open), `-et-pro`, `-snort-vrt`, `-ptopen`. +- **`os-crowdsec`** -- behavioral IPS / collaborative blocklists. +- **`os-sunnyvalley`** -- Zenarmor NGFW (L7 app control) -- heavyweight, commercial tiers; a real + option for the Roosevelt edge but a significant addition. Note, do not adopt lightly. +- **`os-maltrail`** (malicious-traffic detection), **`os-wazuh-agent`**/`os-beats` (SIEM log + shipping), **`os-acme-client`** (Let's Encrypt real TLS for GUI/API -- self-signed is fine in the + closed test; wanted at Roosevelt). **`os-etpro-telemetry`**/`os-q-feeds-connector` -- threat feeds. + +### H. Reviewed and rejected (with reason) -- so the review is exhaustive, not cherry-picked +- **Modern VPN is CORE, so the plugins are not needed:** OPNsense 26.x ships WireGuard, IPsec and + OpenVPN in core (the catalog only offers `os-openvpn-legacy` / `os-strongswan-legacy`). Inter-site + is dark fiber East-West (D-122), not a VPN, so even core VPN is not on the critical path. +- **DHCP: already on the modern core.** The edge runs **Kea** (core); `os-isc-dhcp` is the LEGACY ISC + server -- do not add. +- **App services that are not the edge's role:** `os-haproxy` (Octavia does cloud LB), `os-relayd`, + `os-squid`/`os-OPNProxy`/`os-web-proxy-sso`/`os-c-icap`/`os-ftp-proxy` (proxy), `os-nginx`/`os-caddy` + (web), `os-postfix`/`os-rspamd`/`os-clamav`/`os-redis` (mail/AV/cache), `os-freeradius`/ + `os-radsecproxy` (RADIUS), `os-bind` (**Designate owns cloud DNS**, D-106; `os-ddclient`/ + `os-cloudflared`/`os-rfc2136` dynamic-DNS paths are likewise not ours). +- **Overlay/tunnel alternatives we do not use** (we are committed to Tailscale, D-107): + `os-netbird`, `os-zerotier`, `os-openconnect`, `os-tinc`, `os-tor`, `os-shadowsocks`. +- **Config-mgmt / cloud SDKs that are not our tooling** (we use Juju + MAAS + OpenTofu + REST): + `os-puppet-agent`, `os-google-cloud-sdk`. +- **L2 relays / PXE bits that do not cross the edge:** `os-mdns-repeater`, `os-igmp-proxy`, + `os-udpbroadcastrelay` (our PXE/mDNS stays on the internal planes), `os-tftp` (MAAS serves TFTP + internally, not the edge). +- **Niche / cosmetic / hardware-specific:** `os-siproxd`/`os-turnserver` (VoIP), `os-upnp`, + `os-lcdproc-sdeclcd` (LCD panel), `os-wol`, `os-theme-*` (cosmetic), `os-grid_example`/ + `os-helloworld` (dev samples), `os-realtek-re`/`os-dec-hw` (specific NIC/appliance drivers), + `os-ntopng`/`os-vnstat` (optional traffic visibility -- heavier; revisit only if deep flow + analytics are wanted). + +## Findings (already-true state, not recommendations) + +1. **`os-git-backup` is installed but UNCONFIGURED** -- there is no config node for it, so **no + config backup is running**. That is a false sense of safety. **Recommend wiring it** to push the + edge `config.xml` to `git.baldurkeep.com` (the git service of record) -- this is the single + highest-value hygiene item and aligns the edge with the repo-of-record / config-as-code + discipline. Alternatively remove the plugin so it does not imply protection it is not providing. +2. **Version drift recorded:** the edge moved 26.1.11 -> **26.7**. `docs/vr1-office1-as-built.md` + updated (the VR1 infra version home; `appendix-B` is charm-channels only). +3. **Core features to confirm-configured (not plugins):** Kea DHCP (in use), Suricata (core, off -- + defer), NTP (core -- see E), WireGuard/IPsec (core, available if ever needed). + +## Adoption path (discipline) + +Any adoption is a **D-129** ruling + a provisioning change: bake the chosen plugin install into the +edge bootstrap over the REST-API path (D-113) via the OPNsense firmware API +(`/api/core/firmware/install`), delivered with the touched script's test harness and a changelog +entry with a revert. **This review installs nothing.** The recommended near-term set (git-backup +wiring, qemu-guest-agent on VR1 edges, iperf for transit validation) can proceed as soon as the +operator rules D-129. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 78c4c27..6d1f4be 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -35,12 +35,13 @@ - **PROPOSED / OPEN decisions:** D-068 (Vault substrate hardening, Roosevelt -- 1.16 ruled OUT per amendment; the off-EOL-1.8.8 path remains OPEN), D-071 (routine update cadence + Juju controller patch policy -- AMENDED, still PROPOSED, awaiting operator ratification; gated on - the pre-DC-DC controller HA/backup session). + the pre-DC-DC controller HA/backup session), D-129 (OPNsense edge plugin/add-on base profile -- + PROPOSED 2026-07-18; review in `docs/opnsense-edge-addon-review-20260718.md`, operator rules buckets). - **OPEN security rows:** SEC-001, SEC-003, SEC-004, SEC-005, SEC-006, SEC-007, SEC-008, **SEC-010**. (New since last re-seed: **SEC-010** -- the rack transit-leg FORWARD-drop artifact -- is COMMITTED but stays OPEN until applied + verified on vvr1-dc0. SEC-008 -- the vcloud re-enrolment key -- carried over.) -- **Next-free numbers:** D = **129**, DOCFIX = 196, BUNDLEFIX = 013. +- **Next-free numbers:** D = **130**, DOCFIX = 196, BUNDLEFIX = 013. (Since the last re-seed D-121..128 all ADOPTED -- D-128 = VR1 operating model (2026-07-17). Load-bearing correction to the prior block: **D-123 = Model B** (nodes nest INSIDE `vvr1-dc0`; site-down = one `virsh destroy vvr1-dc0`) via operator ruling R-1 2026-07-16 -- this SUPERSEDES the earlier "D-123 Model A" this block used to @@ -721,6 +722,39 @@ rule 2; the harness fails on an un-commented DC supernet). At each DC standup, MEASURE the vcloud->DC reach and add a MEASURED leg row + install ONLY if that net is isolated (register item 20 / skill invariant). +**OPERATOR PERSONAL KEY pushed to Office1 service VMs (2026-07-18).** On operator request ("push my +id_ed25519 on the vms"), the operator's GitHub-published personal pubkey (`ssh-ed25519 ...J7Zg +JANeumatrix@github`, gh:JANeumatrix) was APPENDED to `~/.ssh/authorized_keys` on **voffice1** (user +jessea123), **office1-netbox** and **office1-tailscale** (user ubuntu) -- ALONGSIDE the `vr1-office1-svc` +service key (service key untouched; the D-126 ProxyJump path is unchanged). Read-only audit first (key +ABSENT on all three), idempotent grep-guarded append, verified 2 keys each. This is a deliberate EXTENSION +of the D-126 per-site-service-key model: the operator's personal identity now has direct login from any +workstation over the tailnet (`ssh jessea123@10.10.0.20` / `ubuntu@10.10.1.10` / `ubuntu@10.10.1.11`). +Pubkey ONLY -- the personal private key is workstation-held, never on the jumphost; login-with-personal-key +still needs the operator to test from a workstation. **Edge (office1-opnsense): operator installs via the +OPNsense GUI** (System > Access > Users > root > Authorized keys) -- a raw `/root/.ssh/authorized_keys` +append is NON-DURABLE (OPNsense regenerates root's keys from `config.xml`); config store audited, personal +key ABSENT there too. **STANDING (per-DC): at each DC standup, add the same personal pubkey to that DC's +service VMs** ("in case I need to log in directly" -- operator instruction), alongside the base-leg per-DC +discipline above. **REMOVAL/ROTATION OBLIGATION:** strip these personal keys at v1 close / teardown (same +class as the SEC key-rotation rows); track if a security-ledger row is wanted. + +**OPNsense EDGE 26.7 UPDATE + PLUGIN/ADD-ON REVIEW (2026-07-18).** Operator updated `office1-opnsense` +**26.1.11 -> 26.7** (0 outdated pkgs, no pending firmware). As-built version RECORDED in +`docs/vr1-office1-as-built.md:42` (appendix-B is charm-channels only, no edge row). Operator asked for a +forward-looking plugin/add-on review -> `docs/opnsense-edge-addon-review-20260718.md` (all 104 catalog +plugins categorized) + **D-129 (PROPOSED)** captures the base-edge-profile decision. Near-stock edge: +only `os-git-backup` installed, and **UNCONFIGURED** (php-verified no config node) -> **no config backup is +actually running** -- recommend wiring it to `git.baldurkeep.com` (config-as-code) regardless of D-129. +Review recommendations by bucket: metal-hardware profile (`os-smart`/`os-nut`/`os-cpu-microcode`/`os-lldpd`, +record for Roosevelt, inert in VR1 VMs); sim scaffolding (`os-qemu-guest-agent`, VR1 VM edges only, aids +D-126/D-127 reboot work); diagnostics (`os-iperf` for transit validation); observability +(`os-node_exporter` CONDITIONAL on a COS-scope ruling -- D-105 scoped COS per-DC inside the cloud, edge is +outside); DEFER all security hardening per closed-test posture. D-129 OPEN decision points: which buckets; +COS-edge scope; `os-frr` (static transit today); `os-tailscale` (consolidate the D-107 subnet-router VM); +edge-as-site-NTP. **NOTHING installed** -- review only. Docs committed on branch +`dc-dc-stage3-phase2-dc-substrate` (not pushed; operator away from workstations). + ## OPEN WORK -- VR0 / DC0 (the running testcloud) 1. **d011 batch-3 live-validation queue** -- `scripts/checks/d011-04-octavia-lb.sh` and diff --git a/docs/vr1-office1-as-built.md b/docs/vr1-office1-as-built.md index 7ccf2cc..cd4baee 100644 --- a/docs/vr1-office1-as-built.md +++ b/docs/vr1-office1-as-built.md @@ -39,7 +39,7 @@ | Host | Address | What runs on it | Level | |---|---|---|---| | **vcloud** | `10.17.11.248` (lab), `10.10.0.10` on `virbr2`, `172.30.1.1` on `virbr11` | libvirt/KVM hypervisor; OpenTofu runs from here | L1 (itself a KVM guest) | -| **office1-opnsense** | LAN `10.10.0.1` / WAN `172.30.1.2` | OPNsense 26.1 edge router: routing, NAT, **Kea DHCP**, firewall | L2 | +| **office1-opnsense** | LAN `10.10.0.1` / WAN `172.30.1.2` | **OPNsense 26.7** edge router (updated from 26.1.11 on 2026-07-18; 0 outdated pkgs): routing, NAT, **Kea DHCP**, firewall. Plugins: `os-git-backup` only (UNCONFIGURED). See `docs/opnsense-edge-addon-review-20260718.md` + D-129. | L2 | | **voffice1** | `10.10.0.20` (Kea reservation) | **MAAS 3.7.2** region+rack + PostgreSQL 16.14; **LXD 5.21.5** (registered to MAAS as VM host `office1-lxd`) | L2 | | **office1-netbox** | `10.10.1.10` | **NetBox 4.6** (netbox-docker; `/opt/netbox-docker`, `docker compose up -d`) on `:8000`; **`restart: unless-stopped`** on all 5 services (2026-07-17 -- survives instance reboot) | L3 (LXD VM) | | **office1-tailscale** | `10.10.1.11` / tailnet **`100.64.0.53`** (`fd7a:115c:a1e0::35`) | **Tailscale 1.98.8**, subnet router advertising `10.10.0.0/22`. Joined the **SELF-HOSTED** control plane `https://tailscale.baldurkeep.com:443` | L3 (LXD VM) |