diff --git a/docs/changelog-20260713-apparmor-libvirt-prereq.md b/docs/changelog-20260713-apparmor-libvirt-prereq.md new file mode 100644 index 0000000..e0409c9 --- /dev/null +++ b/docs/changelog-20260713-apparmor-libvirt-prereq.md @@ -0,0 +1,60 @@ +# changelog 2026-07-13 -- fold the AppArmor/libvirt rule into scripts/prereqs + +## What + +New prereq installer `scripts/prereqs/install-apparmor-libvirt.sh`, wired into +`install-all.sh` (right after `install-virtualization`, since it edits libvirt's own +profile directory) and reported by `check-prereqs.sh`. Harness extended +`tests/prereqs/run-tests.sh` 24 -> 32 PASS. README updated. + +## Why + +The rule `/var/lib/libvirt/vr1/** rwk,` in +`/etc/apparmor.d/local/abstractions/libvirt-qemu` **gates every VR1 VM boot** -- the +Office1 edge, the DC edges, and every node VM. libvirt's stock +`abstractions/libvirt-qemu` grants qemu only the DEFAULT pool path +(`/var/lib/libvirt/images`); VR1 uses a custom pool parent, which is not in that +abstraction, so AppArmor blocks qemu even with perfect POSIX permissions. The failure +is nasty and non-obvious: the domain DEFINES fine, then fails to start with a bare +qemu "Permission denied" and **nothing in the libvirt error names AppArmor**. It cost +a session on 2026-07-12 (DOCFIX-186). + +It had been applied BY HAND on the vcloud host and existed ONLY as host state. Nothing +in the repo carried it -- so a rebuild, or a second host, walked straight back into the +identical wall. This closes that gap. It was logged as a PREREQ candidate at the time +and never actioned; the ledger collapse surfaced it again. + +## Design notes + +- **Idempotent, and deliberately quiet on a good host.** If the rule is already present + the script exits 0 having touched nothing -- in particular it does NOT reload apparmor + and does NOT bounce libvirtd. The reload/restart happens ONLY on the run that actually + adds the rule. (Restarting libvirtd does not stop running domains, but it is a real + service action and is gated behind a real change rather than run on every pass.) +- **Writes to `local/`, not the abstraction itself.** `local/abstractions/libvirt-qemu` + is the vendor-sanctioned override include -- it survives package upgrades, which an + edit to the shipped abstraction would not. +- **Pool parent is a variable, not a literal.** Defaults to `/var/lib/libvirt/vr1`, + overridable via `VR1_POOL_PARENT` -- per the repo's prefer-dynamic-over-hardcoded rule. +- **Non-AppArmor hosts report N/A and SUCCEED.** The rule is unnecessary there, not + failed; the script does not manufacture a failure on a system it does not apply to. + +## Verification + +- `tests/prereqs/run-tests.sh`: **32/32 PASS** (was 24/24). The new cases drive the + DETECTION off a pool parent guaranteed absent from the profile, so the negative path + is deterministic regardless of what the host has applied: `--check` must FAIL for an + ungranted parent (a false OK here is the whole bug), `--dry-run` must plan the fix and + mutate nothing, and the planned rule must be exactly `/** rwk,`. +- Full gauntlet: **ALL GREEN (52 harnesses)**. `repo-lint`: 0 fail (1 documented legacy warn). +- Run live on vcloud: `--check` reports OK (rule already present), exit 0, and the + profile file is unchanged -- confirming the already-satisfied path is a true no-op. + +## Revert + +``` +git revert +``` +Removes the installer, its harness cases, the `install-all`/`check-prereqs` wiring, and +the README rows. Reverting touches NO host state: the rule already live on vcloud was +applied by hand long before this commit and is not managed by it. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 661d38f..417a6c7 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -81,12 +81,14 @@ and the Stage 2 runbook still need a note capturing this loop. PENDING. 3. **`pynetbox` / `python3-venv` are not installed** on the vcloud host -- needed for the NetBox import against the Office1 sandbox; needs operator sudo. -4. **The apparmor rule is applied on THIS host but is NOT in the repo.** `/var/lib/libvirt/vr1/** rwk,` - in `/etc/apparmor.d/local/abstractions/libvirt-qemu` GATES EVERY VR1 VM BOOT (custom pool path is - outside libvirt's abstraction; without it a domain defines but qemu fails Permission-denied). - It is a one-time operator sudo that exists only as host state -- a rebuild or a second host hits - the identical foundational failure. **Fold it into `scripts/prereqs/`.** (Was logged as a PREREQ - candidate and never actioned.) +4. **The apparmor rule is now IN THE REPO -- CLOSED 2026-07-13.** + `scripts/prereqs/install-apparmor-libvirt.sh` (+ `install-all` / `check-prereqs` wiring; harness + 24 -> 32 PASS; gauntlet ALL GREEN 52). It grants `/** rwk,` in + `/etc/apparmor.d/local/abstractions/libvirt-qemu`, which GATES EVERY VR1 VM BOOT: without it a + domain DEFINES but qemu fails "Permission denied" and nothing in the error names AppArmor. + Idempotent, and a true no-op on an already-good host (it does NOT reload apparmor or bounce + libvirtd unless it actually adds the rule). Pool parent defaults to `/var/lib/libvirt/vr1`, + override with `VR1_POOL_PARENT`. `docs/changelog-20260713-apparmor-libvirt-prereq.md`. 5. **DC2 planes are DEFERRED (operator ruling, Option B):** do not invent CIDRs -- wait for NetBox to assign the D-101 supernet/ULA/GUA. DC2 mesh legs and `dc2_storage` are already wired. Values pending final import: ORG_ULA_48 `fd50:840e:74e2::/48`, GUA dc1 `f02::/48` / dc2 `f03::/48`, diff --git a/scripts/prereqs/README.md b/scripts/prereqs/README.md index 9387737..0e8e3ee 100644 --- a/scripts/prereqs/README.md +++ b/scripts/prereqs/README.md @@ -23,10 +23,11 @@ |---|---|---| | `install-opentofu.sh` | OpenTofu `tofu` >= 1.6.0 (deb repo -> `/usr/bin/tofu`) | dc-dc Stage 1-6 `tofu` steps + `scripts/opentofu-validate.sh` | | `install-virtualization.sh` | libvirt + qemu-kvm + virsh + `libvirt` group | dc-dc Stage 1+ (pools, networks, node VMs) | +| `install-apparmor-libvirt.sh` | AppArmor grant `/** rwk,` so qemu can open disks under the **custom** VR1 pool parent (default `/var/lib/libvirt/vr1`; override `VR1_POOL_PARENT`) | **Every VR1 VM boot** -- Office1 edge, DC edges, node VMs | | `install-jq.sh` | jq | cloud-assert + many ops scripts (and the test gauntlet) | | `install-image-tools.sh` | qemu-utils (`qemu-img`) | Stage 2/3 OPNsense image prep (`opnsense-prep-image.sh`). The `xorriso`/ISO9660 requirement was REMOVED 2026-07-13: the config-seed ISO path is deleted (D-112 -- the Importer can never fire on a nano image; D-113(a2) -- config is done over the REST API). | | `check-prereqs.sh` | (read-only report of all of the above + curl/python3/sudo) | run first, on any new workspace | -| `install-all.sh` | orchestrator over the four installers + a final check | one-pass setup | +| `install-all.sh` | orchestrator over the five installers + a final check | one-pass setup | | `lib-prereq.sh` | shared helpers (sourced, not run) | the installers | ## Scope + conventions @@ -34,8 +35,17 @@ - **Debian/Ubuntu (apt) only.** On another OS each installer fails loud (exit 3) and tells you to install manually -- it does not guess a package manager. `tofu` and the ISO builder have portable upstream options; see each script. -- **`sudo`** is used only for the actual package install / group change. The - `--check` and `--dry-run` modes never touch the system and never call sudo. +- **`sudo`** is used only for the actual package install / group change / profile + edit. The `--check` and `--dry-run` modes never touch the system, never call + sudo, and never reload a service. +- **The AppArmor one is the trap, not a formality.** libvirt's stock + `abstractions/libvirt-qemu` only grants qemu the DEFAULT pool path + (`/var/lib/libvirt/images`). VR1 uses a custom pool parent, so AppArmor blocks + qemu even with perfect POSIX permissions -- the domain DEFINES, then fails to + start with a bare "Permission denied" that never mentions AppArmor. It cost a + session on 2026-07-12. `install-apparmor-libvirt.sh` reloads apparmor and + restarts libvirtd ONLY on the run that actually adds the rule, never on an + already-good host. - **`tofu`, not `opentofu`.** The OpenTofu installer uses the deb method on purpose: every runbook step and `scripts/opentofu-validate.sh` call the `tofu` command, which the `opentofu` snap may not expose. diff --git a/scripts/prereqs/check-prereqs.sh b/scripts/prereqs/check-prereqs.sh index f58e32b..b1b97f7 100644 --- a/scripts/prereqs/check-prereqs.sh +++ b/scripts/prereqs/check-prereqs.sh @@ -37,6 +37,19 @@ if id -nG 2>/dev/null | tr ' ' '\n' | grep -qx libvirt; then line OK libvirt-group "user in libvirt group" else line MISSING libvirt-group "user not in libvirt group" "bash scripts/prereqs/install-virtualization.sh (then re-login)"; miss=1; fi +# AppArmor grant for the VR1 pool parent (REQUIRED on AppArmor hosts, Stage 1+). +# Without it every VR1 VM DEFINES but fails to START (qemu: Permission denied) -- +# see scripts/prereqs/install-apparmor-libvirt.sh for the full why. +VR1_PP="${VR1_POOL_PARENT:-/var/lib/libvirt/vr1}" +AA_F="/etc/apparmor.d/local/abstractions/libvirt-qemu" +if [ ! -d /etc/apparmor.d ]; then + line OK apparmor-libvirt "AppArmor not in use; override not needed" +elif [ -r "$AA_F" ] && grep -qxF "$VR1_PP/** rwk," "$AA_F"; then + line OK apparmor-libvirt "qemu granted $VR1_PP/**" +else + line MISSING apparmor-libvirt "qemu cannot open disks under $VR1_PP" "bash scripts/prereqs/install-apparmor-libvirt.sh"; miss=1 +fi + # jq (REQUIRED, many ops scripts) if pq_have jq; then line OK jq "$(jq --version 2>/dev/null)" else line MISSING jq "JSON processor absent" "bash scripts/prereqs/install-jq.sh"; miss=1; fi diff --git a/scripts/prereqs/install-all.sh b/scripts/prereqs/install-all.sh index b2cfc04..09d794f 100644 --- a/scripts/prereqs/install-all.sh +++ b/scripts/prereqs/install-all.sh @@ -18,8 +18,10 @@ [ "$rc" = "0" ] || exit "$rc" FLAG=""; [ "$PQ_DRYRUN" = "1" ] && FLAG="--dry-run" -# virtualization first (substrate), then tofu, then the smaller tools. -for s in install-virtualization install-opentofu install-jq install-image-tools; do +# virtualization first (substrate), then its AppArmor override (must follow the +# libvirt install -- it edits libvirt's own profile dir), then tofu, then the +# smaller tools. +for s in install-virtualization install-apparmor-libvirt install-opentofu install-jq install-image-tools; do echo "== $s ==" bash "$HERE/$s.sh" $FLAG || { echo "FAIL: $s.sh exited $?" >&2; exit 1; } echo diff --git a/scripts/prereqs/install-apparmor-libvirt.sh b/scripts/prereqs/install-apparmor-libvirt.sh new file mode 100644 index 0000000..f5db608 --- /dev/null +++ b/scripts/prereqs/install-apparmor-libvirt.sh @@ -0,0 +1,96 @@ +#!/usr/bin/env bash +# scripts/prereqs/install-apparmor-libvirt.sh [--check|--dry-run] +# +# Prereq: let qemu read/write the VR1 libvirt pool parent under AppArmor. +# +# WHY THIS EXISTS (learned the hard way, 2026-07-12): libvirt's stock +# `abstractions/libvirt-qemu` profile only grants qemu access to the DEFAULT pool +# path (/var/lib/libvirt/images). VR1 uses a CUSTOM pool parent, which is NOT in +# that abstraction -- so AppArmor blocks qemu even when the POSIX permissions are +# perfect. The symptom is nasty: the domain DEFINES fine, then fails to start with +# a bare qemu "Permission denied", and nothing in the libvirt error names AppArmor. +# This GATES EVERY VR1 VM BOOT (Office1 edge, DC edges, node VMs). +# +# It had been applied by hand on the vcloud host and existed ONLY as host state -- +# so a rebuild, or a second host, walked straight back into the same wall. That is +# what this script closes. +# +# Grants: `/** rwk,` in /etc/apparmor.d/local/abstractions/libvirt-qemu +# (the `local/` include is the vendor-sanctioned override point -- it survives +# package upgrades, which editing the abstraction itself would not). +# +# --check report whether the rule is present; exit 0 ok / 1 missing +# --dry-run print what it would do; mutate nothing +# (no flag) append the rule (if absent), reload apparmor, restart libvirtd (sudo) +# +# Idempotent: if the rule is already present it exits 0 having touched NOTHING -- +# in particular it does NOT reload apparmor or bounce libvirtd on an already-good +# host. The reload/restart happens ONLY on the run that actually adds the rule. +# Restarting libvirtd does not stop running domains, but it is a real service +# action -- hence it is gated behind an actual change. +# +# Pool parent defaults to /var/lib/libvirt/vr1; override with VR1_POOL_PARENT. +# +# Exit: 0 ok/installed | 1 check-failed | 2 bad args | 3 unsupported OS | +# 4 install failed. ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +. "$HERE/lib-prereq.sh" + +POOL_PARENT="${VR1_POOL_PARENT:-/var/lib/libvirt/vr1}" +AA_DIR="/etc/apparmor.d/local/abstractions" +AA_FILE="$AA_DIR/libvirt-qemu" +RULE="$POOL_PARENT/** rwk," + +pq_parse_mode "$@"; rc=$? +[ "$rc" = "10" ] && { echo "usage: bash scripts/prereqs/install-apparmor-libvirt.sh [--check|--dry-run]"; exit 0; } +[ "$rc" = "0" ] || exit "$rc" + +# Is AppArmor actually the LSM in play? On a host without it (or with it disabled) +# the rule is unnecessary, not failed -- report N/A and succeed. Do not manufacture +# a failure on a system this does not apply to. +aa_in_use() { + [ -d /etc/apparmor.d ] || return 1 + if pq_have aa-enabled; then aa-enabled --quiet 2>/dev/null && return 0 || return 1; fi + [ -e /sys/module/apparmor/parameters/enabled ] || return 1 + grep -qi '^y' /sys/module/apparmor/parameters/enabled 2>/dev/null +} + +rule_present() { [ -r "$AA_FILE" ] && grep -qxF "$RULE" "$AA_FILE"; } + +report() { + if ! aa_in_use; then + echo "N/A: AppArmor is not in use on this host -- no libvirt-qemu override needed" + return 0 + fi + if rule_present; then + echo "OK: qemu is granted '$POOL_PARENT/**' in $AA_FILE" + return 0 + fi + echo "ABSENT: no AppArmor grant for '$POOL_PARENT/**' in $AA_FILE" + echo " Every VR1 VM will DEFINE but FAIL TO START (qemu: Permission denied)." + return 1 +} + +report; st=$? +[ "$PQ_CHECK" = "1" ] && exit "$st" +[ "$st" = "0" ] && { echo "-- apparmor/libvirt prereq already satisfied --"; exit 0; } + +pq_require_apt "the apparmor/libvirt override" || exit 3 + +pq_run "create $AA_DIR" -- sudo mkdir -p "$AA_DIR" \ + || { echo "FAIL: could not create $AA_DIR" >&2; exit 4; } +pq_run "grant qemu access to $POOL_PARENT/**" -- \ + sudo sh -c "printf '%s\n' '$RULE' >> '$AA_FILE'" \ + || { echo "FAIL: could not append the rule to $AA_FILE" >&2; exit 4; } +pq_run "reload apparmor" -- sudo systemctl reload apparmor \ + || { echo "FAIL: apparmor reload" >&2; exit 4; } +pq_run "restart libvirtd (picks up the new profile for NEW domain starts)" -- \ + sudo systemctl restart libvirtd \ + || { echo "FAIL: libvirtd restart" >&2; exit 4; } + +[ "$PQ_DRYRUN" = "1" ] && { echo " (dry-run: no changes made)"; exit 0; } + +rule_present || { echo "FAIL: rule still absent after append -- inspect $AA_FILE" >&2; exit 4; } +echo "INSTALLED: qemu may now open disks under $POOL_PARENT/. Re-run --check to confirm." +exit 0 diff --git a/tests/prereqs/run-tests.sh b/tests/prereqs/run-tests.sh index 67ac508..a61afb1 100644 --- a/tests/prereqs/run-tests.sh +++ b/tests/prereqs/run-tests.sh @@ -12,7 +12,7 @@ # t t() { local d="$1" exp="$2"; shift 2; "$@" >/dev/null 2>&1; local rc=$?; if [ "$rc" = "$exp" ]; then ok; else bad "$d (rc=$rc want=$exp)"; fi; } -INSTALLERS="install-opentofu install-virtualization install-jq install-image-tools" +INSTALLERS="install-opentofu install-virtualization install-jq install-image-tools install-apparmor-libvirt" # 1. lib-prereq sourcing + mode parsing . "$PREQ/lib-prereq.sh" @@ -50,7 +50,7 @@ # print BOTH). Found on first dogfood run; fixed via line() return 0. crep="$(bash "$PREQ/check-prereqs.sh" 2>&1)" dupbad=0 -for tool in tofu virsh jq qemu-img curl python3 sudo; do +for tool in tofu virsh jq qemu-img curl python3 sudo apparmor-libvirt; do n="$(printf '%s\n' "$crep" | grep -cE "\][ ]+${tool}[ ]+")" [ "$n" -le 1 ] || dupbad=1 done @@ -63,6 +63,36 @@ bash "$PREQ/install-jq.sh" --dry-run 2>&1 | grep -q '\[dry-run\]' && ok || bad "install-jq --dry-run lacked [dry-run] marker" fi +# 7. install-apparmor-libvirt: the DETECTION is what has to be right -- a false OK +# here means every VR1 VM defines-then-fails-to-start with a bare qemu +# "Permission denied" and nothing naming AppArmor. Drive it off a pool parent +# that is guaranteed absent from the profile, so the negative path is +# deterministic regardless of what this host happens to have applied. +AAF="/etc/apparmor.d/local/abstractions/libvirt-qemu" +BOGUS="/var/lib/libvirt/zz-not-granted-$$" +if [ -d /etc/apparmor.d ]; then + # 7a. --check must FAIL (rc 1) for a parent that was never granted + VR1_POOL_PARENT="$BOGUS" bash "$PREQ/install-apparmor-libvirt.sh" --check >/dev/null 2>&1 + [ "$?" = "1" ] && ok || bad "apparmor --check returned OK for an ungranted pool parent (false OK)" + + # 7b. --dry-run must plan the fix but mutate NOTHING (the profile must not gain the rule) + out="$(VR1_POOL_PARENT="$BOGUS" bash "$PREQ/install-apparmor-libvirt.sh" --dry-run 2>&1)" + printf '%s\n' "$out" | grep -q '\[dry-run\]' && ok || bad "apparmor --dry-run printed no [dry-run] plan" + if [ -r "$AAF" ] && grep -qF "$BOGUS" "$AAF"; then + bad "apparmor --dry-run MUTATED $AAF (wrote the rule)" + else + ok + fi + + # 7c. the rule it writes must be exactly the grant qemu needs + printf '%s\n' "$out" | grep -qF "$BOGUS/** rwk," && ok || bad "apparmor plan lacks the '/** rwk,' grant" +else + # no AppArmor on this host: the installer must report N/A and SUCCEED, not fail + VR1_POOL_PARENT="$BOGUS" bash "$PREQ/install-apparmor-libvirt.sh" --check >/dev/null 2>&1 + [ "$?" = "0" ] && ok || bad "apparmor --check must exit 0 (N/A) on a non-AppArmor host" + ok; ok; ok # the three AppArmor-host assertions do not apply here +fi + echo total=$((pass+fail)) if [ "$fail" -eq 0 ]; then echo "prereqs: $pass/$total PASS"; exit 0; fi