diff --git a/docs/changelog-20260718-opnsense-plugins-installer.md b/docs/changelog-20260718-opnsense-plugins-installer.md new file mode 100644 index 0000000..536edaf --- /dev/null +++ b/docs/changelog-20260718-opnsense-plugins-installer.md @@ -0,0 +1,40 @@ +# Changelog 2026-07-18 -- OPNsense edge plugin profile-installer (D-129) + +Autonomous processing of the D-129 "engineering for the rest" that does NOT need +the operator at a keyboard. **Nothing was applied to the cloud** (the live plugin +install remains classifier-gated). Repo-only: one new script + harness + pin +refinements. Gauntlet ALL GREEN (68), repo-lint 0-fail. + +## Items + +1. **NEW `scripts/opnsense-plugins.sh` + `tests/opnsense-plugins/run-tests.sh` (17/17).** + - WHAT: a named-profile plugin installer for an OPNsense edge over the D-113 + firmware API. `list` + `apply `. Thin orchestrator over + `scripts/opnsense-api.sh` (auth/secret handling reused, never re-implemented). + - WHY: closes the D-129 pin "build the reusable profile-installer at the DC-edge + step". Encodes the two 2026-07-18 lessons so they are not re-learned: (a) + install ONE plugin at a time and POLL `core/firmware/upgradestatus` -- back-to- + back `firmware/install` POSTs collide on the OPNsense firmware lock and the + second fails silently (this is why `os-iperf` did not install); (b) loudly warns + that `os-qemu-guest-agent` is a package, NOT a working agent (needs the libvirt + channel + service enable + restart). `--dry-run` is offline (no net/creds) so + the harness proves the shape. + - REVERT: `git rm scripts/opnsense-plugins.sh tests/opnsense-plugins/run-tests.sh`. + +2. **D-129 + `dc-dc-phase2` pin refined (doc-only).** + - WHAT: recorded the installer as BUILT (runbook now says USE it, not build it); + logged the qga-channel design (add to `modules/opnsense-edge` as an OPT-IN var + defaulting OFF, so the shared-module edit does NOT change the live + `office1-opnsense` plan or dirty the DC0 outer plan; mechanism-to-verify on + dmacvicar 0.9.8 -- native `qemu_agent` arg vs `xml{xslt}` injection, not assumed); + logged the exact `os-git-backup` operator input needed (target repo + deploy key). + - WHY: advances the items I cannot execute autonomously to "ready to implement + safely" without touching the cloud or the imminent DC0 deploy. + - REVERT: revert the D-129 Implementation-prep block + the phase2 pin edit. + +## Still operator-only (cannot be processed autonomously) +- `os-iperf` re-install (classifier-gated firmware POST) -- operator-run or add a Bash + permission rule. +- qga functional enablement (module opt-in channel + service enable + edge RESTART). +- `os-git-backup` wiring (choose repo + add the plugin's deploy key on GitBucket). +- The four D-129 sub-decisions (COS-edge scope, os-frr, os-tailscale, edge-NTP). diff --git a/docs/design-decisions.md b/docs/design-decisions.md index 6e65bce..fe3b650 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -4050,7 +4050,26 @@ - `os-iperf`: **first install FAILED** -- two back-to-back `core/firmware/install` POSTs collided on the OPNsense firmware LOCK (the 2nd ran while the 1st held it). **Lesson for the profile-installer: install ONE plugin at a time and POLL `upgradestatus` to completion before the next.** Re-run of `os-iperf` - pending. + pending (still classifier-gated for the agent -- operator-run or a Bash permission rule). + +**Implementation prep (autonomous 2026-07-18; NOTHING applied to the cloud).** +- **Profile-installer BUILT: `scripts/opnsense-plugins.sh` + `tests/opnsense-plugins/` (17/17, gauntlet + 68 GREEN).** `list` + `apply `; installs ONE plugin at a time + polls + `upgradestatus` (encodes the firmware-lock lesson), delegates auth to `opnsense-api.sh`, warns on the qga + channel dependency, offline `--dry-run`. This CLOSES the "build the reusable installer" pin; live `apply` + is operator-gated (firmware POST). Runbook pin updated to USE it. +- **qga libvirt channel (the functional gap) -- design, NOT applied.** Add to `modules/opnsense-edge` as an + **OPT-IN variable defaulting to OFF** (e.g. `enable_qemu_agent = false`) so EXISTING edges get NO plan + change -- CRITICAL: a default-on edit to the shared module would force a change/bounce on the LIVE + `office1-opnsense` and dirty the imminent DC0 outer plan (5/0/6). Set true only for NEW DC-edge + instantiations (born with the channel, no retrofit); `office1-opnsense` opts in + restarts on its own + schedule. **MECHANISM TO VERIFY before writing (not in `references/opentofu-provider-docs.md` -- do NOT + assume):** whether dmacvicar/libvirt **0.9.8** exposes a native `qemu_agent = true` domain arg or needs an + `xml { xslt = ... }` channel injection. Then enable the service + restart. +- **`os-git-backup` wiring (Finding #1) NEEDS OPERATOR INPUT -- cannot be done autonomously:** (a) choose a + target git repo (a repo on `git.baldurkeep.com` for edge-config backups); (b) the plugin generates an SSH + keypair -- add its PUBLIC key as a DEPLOY KEY on that repo; (c) set branch + schedule + enable. Steps + (a)/(b) are operator/GitBucket actions; the write itself is also a gated edge mutation. **Problem.** `office1-opnsense` was updated to OPNsense 26.7 and is a near-stock install (only `os-git-backup`, itself unconfigured). The edge is the reusable per-site boundary profile that will diff --git a/docs/session-ledger.md b/docs/session-ledger.md index a3da478..ecdc096 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -766,8 +766,17 @@ + edge RESTART -> DEFERRED to next scheduled edge restart (new DC edges born with the channel). `os-iperf` did NOT install: two back-to-back firmware POSTs collided on the OPNsense firmware LOCK -> re-run pending (profile-installer must be one-at-a-time + poll). Records reconciled to this reality (D-129 Verification -note, phase2 pin, as-built). Docs committed on branch `dc-dc-stage3-phase2-dc-substrate` (not pushed; -operator away from workstations). +note, phase2 pin, as-built). Docs committed on branch `dc-dc-stage3-phase2-dc-substrate` (pushed +2026-07-18, `83876c3..c1ef7ab`). +**AUTONOMOUS PROCESSING 2026-07-18 (operator: "process what you can without me running anything").** +Boot-race carry-forward RETIRED (D-126 base-leg PROVEN across the forced reboot -- `added leg` in the +journal; jessea123 added to `systemd-journal` group so unit journals read without sudo). Built the D-129 +profile-installer `scripts/opnsense-plugins.sh` + `tests/opnsense-plugins/` (17/17; gauntlet 68 GREEN) -- +closes the "build the installer" pin; encodes one-at-a-time+poll + the qga-channel warning. Logged the qga +channel design (OPT-IN var, default OFF -> no office1/DC0-plan side-effect; mechanism-to-verify on dmacvicar +0.9.8) + the `os-git-backup` operator-input ask. **Still operator-only:** `os-iperf` re-install (classifier- +gated), qga functional enablement (channel+enable+restart), git-backup wiring, the 4 D-129 sub-decisions. +See `docs/changelog-20260718-opnsense-plugins-installer.md`. ## OPEN WORK -- VR0 / DC0 (the running testcloud) diff --git a/runbooks/dc-dc-phase2-tofu-dc-substrate.md b/runbooks/dc-dc-phase2-tofu-dc-substrate.md index 176d03c..685a03c 100644 --- a/runbooks/dc-dc-phase2-tofu-dc-substrate.md +++ b/runbooks/dc-dc-phase2-tofu-dc-substrate.md @@ -168,12 +168,14 @@ D-113 firmware API (`scripts/opnsense-api.sh POST core/firmware/install/`) -- `qemu-guest-agent` gives the inner libvirt host clean guest coordination (graceful shutdown/fsfreeze/IP report; pairs with D-127 autostart), `iperf` validates the transit path. **Verify the `qemu-guest-agent` daemon is - RUNNING and answers `virsh ... guest-ping` -- not just the package present.** BUILD a small idempotent - profile-installer here (thin wrapper over `opnsense-api.sh`, `--check`); it MUST install **one plugin at - a time and poll `core/firmware/upgradestatus` to completion** before the next -- back-to-back installs - collide on the OPNsense firmware lock and the later one fails silently (measured 2026-07-18). **For qga - to actually work, `modules/opnsense-edge` must define the `org.qemu.guest_agent.0` virtio-serial - channel in the domain XML** (verified ABSENT 2026-07-18; NEW DC edges then boot with it, no retrofit). + RUNNING and answers `virsh ... guest-ping` -- not just the package present.** USE the built installer: + `OPNSENSE_API_HOST= bash scripts/opnsense-plugins.sh apply vr1-edge` (BUILT 2026-07-18, + `tests/opnsense-plugins/`) -- it installs **one plugin at a time and polls `core/firmware/upgradestatus` + to completion** before the next (back-to-back installs collide on the OPNsense firmware lock and the later + one fails silently -- measured 2026-07-18). **For qga to actually work, `modules/opnsense-edge` must expose + the `org.qemu.guest_agent.0` virtio-serial channel** (verified ABSENT 2026-07-18; add it as an OPT-IN var + defaulting OFF so existing edges are unchanged, set true for new DC edges -- see D-129 Implementation prep, + mechanism-to-verify on dmacvicar 0.9.8). NEW DC edges then boot with the channel, no retrofit. NOT the Roosevelt-metal profile (baremetal-only) and NOT `os-node_exporter` (gated on the COS-edge scope, D-129). diff --git a/scripts/opnsense-plugins.sh b/scripts/opnsense-plugins.sh new file mode 100755 index 0000000..4ce2a5b --- /dev/null +++ b/scripts/opnsense-plugins.sh @@ -0,0 +1,124 @@ +#!/usr/bin/env bash +# scripts/opnsense-plugins.sh [--dry-run] [profile] +# +# Install a NAMED OPNsense plugin PROFILE on an edge via the D-113 firmware API, +# the reusable tool D-129 pins for DC-edge bring-up (runbooks/dc-dc-phase2-tofu- +# dc-substrate.md, "Edge plugin profile"). It is a thin ORCHESTRATOR over +# scripts/opnsense-api.sh -- that script owns auth + secret handling (creds never +# in argv/context); this one owns the profile table + the install DISCIPLINE. +# +# THE DISCIPLINE (both learned the hard way 2026-07-18, D-129 Verification): +# 1. ONE PLUGIN AT A TIME + POLL. Two back-to-back core/firmware/install POSTs +# collide on the OPNsense firmware LOCK -- the second fails SILENTLY. So we +# install one, poll core/firmware/upgradestatus to completion, THEN the next. +# 2. os-qemu-guest-agent PACKAGE != a working guest agent. It also needs the +# libvirt domain to expose the org.qemu.guest_agent.0 virtio channel (an IaC +# job in modules/opnsense-edge) AND the service enabled AND an edge restart. +# This script installs the package and LOUDLY says so; it cannot do the rest. +# +# Live use needs OPNSENSE_API_HOST (e.g. 10.10.0.1) + the API creds file that +# opnsense-api.sh reads. --dry-run prints the intended API calls and touches no +# network and reads no creds (so the harness can prove the shape offline). +# +# Exit: 0 ok | 1 usage/arg error | 2 an install did not converge. +set -euo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +API="$HERE/opnsense-api.sh" + +# Poll bounds for the firmware action (live only). Generous: a plugin pull over a +# slow lab uplink can take a while; we would rather wait than declare a false fail. +: "${PLUGIN_POLL_MAX:=90}" # iterations +: "${PLUGIN_POLL_SLEEP:=2}" # seconds between polls + +DRY_RUN=0 +if [ "${1:-}" = "--dry-run" ]; then DRY_RUN=1; shift; fi +CMD="${1:-}" +PROFILE="${2:-}" + +usage() { + cat >&2 < [profile] + list print the known profiles and their plugins + apply install the profile's plugins, ONE AT A TIME + poll + +profiles: + vr1-edge the VR1 (virtual) edge base profile -- D-129 + metal-edge Roosevelt baremetal edge profile (site-specific, see notes) + + e.g. OPNSENSE_API_HOST=10.10.0.1 opnsense-plugins.sh apply vr1-edge + opnsense-plugins.sh --dry-run apply vr1-edge +EOF + exit 1 +} + +# --- Profile table (the single source of truth for "what is on an edge"). ------- +# vr1-edge: adopted by D-129 2026-07-18. os-qemu-guest-agent = clean libvirt-guest +# coordination (needs the channel, see warning below); os-iperf = transit +# throughput validation. +profile_plugins() { + case "$1" in + vr1-edge) echo "os-qemu-guest-agent os-iperf" ;; + # metal-edge is DELIBERATELY not a flat list: os-nut vs os-apcupsd and the + # intel-vs-amd microcode package are per-host choices (D-129). Emit the + # unambiguous members; the operator adds the site-specific ones by hand. + metal-edge) echo "os-smart os-lldpd" ;; + *) return 1 ;; + esac +} + +list_profiles() { + local p + for p in vr1-edge metal-edge; do + printf '%-11s %s\n' "$p" "$(profile_plugins "$p")" + done + echo "metal-edge NOTE: also add os-nut|os-apcupsd + os-cpu-microcode-intel|-amd per host (D-129)." +} + +# --- qemu-guest-agent honesty: the package is necessary but NOT sufficient. ----- +qga_warning() { + cat >&2 <<'EOF' +NOTE: os-qemu-guest-agent installs the PACKAGE only. For a WORKING guest agent the + libvirt domain must expose the org.qemu.guest_agent.0 virtio channel (IaC: + modules/opnsense-edge), the service must be enabled, and the edge restarted. + New DC edges should be BORN with the channel; office1-opnsense retrofits at + its next scheduled restart. (D-129 Verification, 2026-07-18.) +EOF +} + +api() { bash "$API" ${DRY_RUN:+--dry-run} "$@"; } + +# Poll core/firmware/upgradestatus until the running action finishes. Live only; +# dry-run returns immediately (nothing was actually kicked off). +poll_until_done() { + [ "$DRY_RUN" = "1" ] && { echo "DRY-RUN would poll core/firmware/upgradestatus"; return 0; } + local i resp + for i in $(seq 1 "$PLUGIN_POLL_MAX"); do + resp="$(bash "$API" GET core/firmware/upgradestatus 2>/dev/null || true)" + # status is "running" while the action is live; anything else = finished. + printf '%s' "$resp" | grep -qE '"status"[[:space:]]*:[[:space:]]*"running"' || return 0 + sleep "$PLUGIN_POLL_SLEEP" + done + echo "FAIL: firmware action did not converge in $((PLUGIN_POLL_MAX*PLUGIN_POLL_SLEEP))s" >&2 + return 2 +} + +apply_profile() { + local prof="$1" plugins pkg + plugins="$(profile_plugins "$prof")" || { echo "FAIL: unknown profile '$prof'" >&2; usage; } + + case " $plugins " in *" os-qemu-guest-agent "*) qga_warning ;; esac + + for pkg in $plugins; do + echo "== install $pkg ==" + api POST "core/firmware/install/$pkg" # kick off (async on the edge) + poll_until_done # ... and WAIT before the next (the lock) + done + echo "OK: profile '$prof' applied ($plugins)" +} + +case "$CMD" in + list) [ -n "$PROFILE" ] && usage; list_profiles ;; + apply) [ -n "$PROFILE" ] || usage; apply_profile "$PROFILE" ;; + *) usage ;; +esac diff --git a/tests/opnsense-plugins/run-tests.sh b/tests/opnsense-plugins/run-tests.sh new file mode 100755 index 0000000..eaca9c1 --- /dev/null +++ b/tests/opnsense-plugins/run-tests.sh @@ -0,0 +1,62 @@ +#!/usr/bin/env bash +# tests/opnsense-plugins/run-tests.sh +# +# Harness for scripts/opnsense-plugins.sh (D-129 edge plugin profile installer). +# OFFLINE -- no network, no creds, no edge. Drives --dry-run (which prints the +# intended opnsense-api.sh calls without touching the wire) + greps the source, +# PINNING the two lessons the script exists to encode: (1) install ONE plugin at +# a time and POLL between (the firmware-lock trap), (2) os-qemu-guest-agent is a +# package, NOT a working agent (the channel/enable/restart caveat). A careless +# edit reddens HERE, not on a live edge. ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +S="$(cd "$HERE/../../scripts" && pwd)/opnsense-plugins.sh" +pass=0; fail=0 +ok() { pass=$((pass+1)); } +bad() { fail=$((fail+1)); echo " FAIL: $1"; } + +# dry-run still needs a host set (opnsense-api.sh checks it before the dry-run +# branch) -- use an invalid one; nothing must reach it. +export OPNSENSE_API_HOST="test.invalid" +export OPNSENSE_API_CREDS="/nonexistent/should-never-be-read" + +# --- parse / usage --- +bash -n "$S" 2>/dev/null && ok || bad "does not parse (bash -n)" +bash "$S" >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "no-arg must be nonzero (usage)" +bash "$S" bogus-cmd >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "bad subcommand must be nonzero" + +# --- list --- +L="$(bash "$S" list 2>&1)" +printf '%s' "$L" | grep -q 'vr1-edge' && ok || bad "list lost vr1-edge" +printf '%s' "$L" | grep -q 'os-qemu-guest-agent' && ok || bad "vr1-edge lost os-qemu-guest-agent" +printf '%s' "$L" | grep -q 'os-iperf' && ok || bad "vr1-edge lost os-iperf" +printf '%s' "$L" | grep -q 'metal-edge' && ok || bad "list lost metal-edge" +printf '%s' "$L" | grep -qE 'microcode|os-nut|os-apcupsd' && ok || bad "metal-edge lost its site-specific note" + +# --- dry-run apply vr1-edge (stdout=calls, stderr=warnings) --- +OUT="$(bash "$S" --dry-run apply vr1-edge 2>/tmp/opl_err.$$)"; rc=$? +ERR="$(cat /tmp/opl_err.$$ 2>/dev/null)"; rm -f /tmp/opl_err.$$ +[ "$rc" -eq 0 ] && ok || bad "--dry-run apply vr1-edge -> nonzero ($rc)" +printf '%s' "$OUT" | grep -q 'core/firmware/install/os-qemu-guest-agent' && ok || bad "dry-run missing qga install call" +printf '%s' "$OUT" | grep -q 'core/firmware/install/os-iperf' && ok || bad "dry-run missing iperf install call" + +# --- DISCIPLINE 1: one-at-a-time + poll. qga install BEFORE iperf, poll BETWEEN. --- +nq="$(printf '%s\n' "$OUT" | grep -n 'install/os-qemu-guest-agent' | head -1 | cut -d: -f1)" +np="$(printf '%s\n' "$OUT" | grep -n 'poll' | head -1 | cut -d: -f1)" +ni="$(printf '%s\n' "$OUT" | grep -n 'install/os-iperf' | head -1 | cut -d: -f1)" +if [ -n "$nq" ] && [ -n "$np" ] && [ -n "$ni" ] && [ "$nq" -lt "$np" ] && [ "$np" -lt "$ni" ]; then ok +else bad "not one-at-a-time: expected qga($nq) < poll($np) < iperf($ni)"; fi +grep -q 'upgradestatus' "$S" && ok || bad "source lost the upgradestatus poll (the firmware-lock fix)" + +# --- DISCIPLINE 2: qemu-guest-agent honesty (channel/enable/restart caveat) --- +printf '%s' "$ERR" | grep -qiE 'channel|guest_agent' && ok || bad "no qga channel/enable warning emitted" + +# --- unknown profile fails loud --- +bash "$S" --dry-run apply no-such-profile >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "unknown profile must be nonzero" + +# --- no baked secret; delegates auth to opnsense-api.sh --- +grep -qiE '^[^#]*secret[[:space:]]*=' "$S" && bad "script bakes a secret (must delegate to opnsense-api.sh)" || ok +grep -q 'opnsense-api.sh' "$S" && ok || bad "does not delegate HTTP to opnsense-api.sh" + +echo "opnsense-plugins: $pass passed, $fail failed" +[ "$fail" -eq 0 ]