diff --git a/.claude/skills/openstack-cloud-ops/SKILL.md b/.claude/skills/openstack-cloud-ops/SKILL.md index f524cce..19c16de 100644 --- a/.claude/skills/openstack-cloud-ops/SKILL.md +++ b/.claude/skills/openstack-cloud-ops/SKILL.md @@ -95,7 +95,8 @@ | Why is it built this way? / proposing changes | repo `docs/design-decisions.md` (D-NNN); grep before assigning a new number | | Versions, channels, pins | repo `runbooks/appendix-B-asbuilt-version-lock.md` | | Environment facts (hosts, repo, planes) | `references/environment.md` | -| OpenTofu / libvirt / MAAS / OPNsense IaC work (repo `opentofu/`) | `references/opentofu-provider-docs.md` FIRST (sources + fetch method), then repo `opentofu/README.md` (per-module ground truth, schema notes, audit findings) | +| OpenTofu / libvirt / MAAS / OPNsense IaC work (repo `opentofu/`) | `references/opentofu-provider-docs.md` FIRST (sources + fetch method), then repo `opentofu/README.md` (per-module ground truth, schema notes, audit findings -- but its OPNsense section is SUPERSEDED by D-112/D-113) | +| Creating/changing a `libvirt_domain`; anything about nested KVM, a libvirt storage pool, LXD versions, an OPNsense edge, or a VM that boots wrong | `references/platform-traps.md` -- per-tool traps + version pins + a verbatim-error index (KiB-vs-MiB memory, `acpi=off`, missing `cpu` block = no `svm`, "in-place" bounces the guest, AppArmor on non-default pools, MAAS-vs-LXD 6.7, lxdbr0's dnsmasq, tcsh on the edge) | ## Standard loops (repeatable session shapes) diff --git a/.claude/skills/openstack-cloud-ops/references/environment.md b/.claude/skills/openstack-cloud-ops/references/environment.md index 8bcbe80..f73e354 100644 --- a/.claude/skills/openstack-cloud-ops/references/environment.md +++ b/.claude/skills/openstack-cloud-ops/references/environment.md @@ -29,6 +29,11 @@ Charmed OpenStack Caracal 2024.1 - Juju 3.6, MAAS 3.7.2, Vault TLS (charm-pki root CA), OVN 24.03, Ceph Squid, Octavia (amphora), Barbican, mysql-innodb-cluster, RabbitMQ, Magnum + magnum-capi-helm driver + azimuth +(SUBSTRATE below the charms -- pins + traps in `references/platform-traps.md`: +libvirt/KVM via the `dmacvicar/libvirt` OpenTofu provider `0.9.8` on OpenTofu +`v1.12.3`; **LXD pinned to the `5.21` LTS track** -- MAAS 3.6/3.7 cannot talk +to LXD >= 6.7, D-114; OPNsense 26.1 edges driven by REST API, D-113; the vcloud +host is ITSELF a KVM guest, so nesting is a per-level requirement) capi-helm-charts (kubeadm engine), in-cloud single-homed CAPI mgmt VM (capi-mgmt-v2, k8s-snap, D-035). NetBox is the IPAM apex: never hand-edit downstream MAAS or overlays for network values. diff --git a/.claude/skills/openstack-cloud-ops/references/opentofu-provider-docs.md b/.claude/skills/openstack-cloud-ops/references/opentofu-provider-docs.md index a9cfc5e..86c1c6a 100644 --- a/.claude/skills/openstack-cloud-ops/references/opentofu-provider-docs.md +++ b/.claude/skills/openstack-cloud-ops/references/opentofu-provider-docs.md @@ -45,10 +45,25 @@ ## Providers used in `repo opentofu/` ### `dmacvicar/libvirt` (pinned `0.9.8`, confirmed 2026-07-09) + +**Before writing any `libvirt_domain`, read `references/platform-traps.md` +section 1.** Four schema-level traps there have each cost real time and none +of them is a validate-time error: bare `memory` is KiB not MiB; no `features` +block means `acpi=off`; no `cpu` block means no `svm` (nested KVM silently +impossible); and "will be updated in-place" still BOUNCES the guest. + - Registry: https://registry.terraform.io/providers/dmacvicar/libvirt/latest (JS-rendered -- use the GitHub repo for actual content) - GitHub: https://github.com/dmacvicar/terraform-provider-libvirt (default - branch `main`) + branch `main`; there is no `CHANGELOG.md` -- the release NOTES on the + releases page are the changelog, and they are unusually informative: + `.../releases/tag/v0.9.0` is the breaking-rewrite rationale, `v0.9.1` is the + 0.9.0->0.9.1 attribute-rename migration guide) +- **Version boundary that invalidates most examples on the internet:** 0.9.0 + (2025-11-08) is a compatibility-BREAKING rewrite (plugin framework, "HCL maps + almost 1:1 to libvirt XML"); the legacy provider lives on the `v0.8` branch. + 0.9.1 then code-generated the schema and renamed attributes again. Anything + 0.8-era -- including model memory -- is wrong for this pin. - Docs: `docs/resources/*.md`, `docs/data-sources/*.md` (NOT the older `website/docs/r/*.markdown` layout some search results/training data reference -- that's stale) @@ -89,20 +104,46 @@ issue) -- these were the most reliable fetches of the whole session. ### OPNsense (not a Terraform provider -- the guest OS itself) -- Official docs: https://docs.opnsense.org/manual/install.html, - .../manual/backups.html, .../development/backend/autorun.html -- Forum (community consensus, cite specific threads, not the forum - generally): https://forum.opnsense.org/index.php?topic=42517.0 (cloud-init - unreliable on FreeBSD -- the finding that ruled out reusing - `modules/cloudinit-vm` for OPNsense) -- GitHub issues (feature provenance): https://github.com/opnsense/core/issues/5733 - (ISO9660 Configuration-Importer support added specifically for VM/cloud - automation), .../issues/10017 (corroborates it shipped and is live) -- Full sourced writeup: `repo opentofu/README.md`'s "OPNsense deployment - research" section. + +**SUPERSEDED RESEARCH WARNING (2026-07-12/13).** `repo opentofu/README.md`'s +"OPNsense deployment research" section and the 2026-07-09 changelogs describe a +**config-ISO / Configuration-Importer** delivery path. That path is DEAD and its +code is DELETED: +- **D-112:** the Configuration Importer can NEVER fire on a pre-installed nano + image (upstream `opnsense-importer` probes for a read-only root and exits + before scanning media). The ISO was inert; nothing was ever going to read it. +- **D-113 (ADOPTED 2026-07-13, option a2):** OPNsense STAYS, but config moves to + the **REST API**. `opentofu/templates/opnsense-config.xml.tmpl`, + `scripts/opnsense-render-config.sh` and `scripts/opnsense-build-config-iso.sh` + are DELETED, along with the module's `config_seed` volume + cdrom disk. + **A rendered-`config.xml` push would now CLOBBER live API-managed DHCP.** + +Current path: `scripts/opnsense-api.sh` (REST client) + +`scripts/opnsense-bootstrap-apikey.sh` (mints a key via OPNsense's own model +over SSH). Traps, exact endpoints, the mandatory `service/reconfigure` step, the +60-second firewall auto-rollback, and the tcsh trap: `references/platform-traps.md` +section 4. + +- Official docs: https://docs.opnsense.org/manual/install.html; + REST API: https://docs.opnsense.org/development/how-tos/api.html; + 26.1 release notes (ISC-DHCP left core; dnsmasq is the new default): + https://docs.opnsense.org/releases/CE_26.1.html +- Still-valid finding from the 2026-07-09 research: cloud-init is unreliable on + FreeBSD, so `modules/cloudinit-vm` genuinely does not apply to OPNsense + (https://forum.opnsense.org/index.php?topic=42517.0), and the nano image (not + the installer image) is the one to use. +- Historical only (the ISO mechanism these describe cannot fire on nano): + https://github.com/opnsense/core/issues/5733, .../issues/10017. ## Where the full findings live (don't duplicate here, it will drift) +**Caveat first (2026-07-13):** `repo opentofu/README.md` predates D-112/D-113 +and its "OPNsense deployment research" section still describes the DELETED +config-ISO path -- see the superseded-research warning above before acting on +anything it says about OPNsense. Its libvirt schema notes also predate the four +domain traps in `references/platform-traps.md` section 1. Everything else in it +still stands. + `repo opentofu/README.md` is the authoritative, per-module account -- schema notes, what's confirmed vs. assumed per attribute, and a dated "Audit pass" section recording a later re-review (a real documentation diff --git a/.claude/skills/openstack-cloud-ops/references/platform-traps.md b/.claude/skills/openstack-cloud-ops/references/platform-traps.md new file mode 100644 index 0000000..4755be7 --- /dev/null +++ b/.claude/skills/openstack-cloud-ops/references/platform-traps.md @@ -0,0 +1,397 @@ +# Platform traps and version pins -- libvirt/KVM, MAAS+LXD, OPNsense + +The substrate BELOW the cloud (the hypervisor, the provisioner, the edge +routers) has its own version boundaries and its own silent-failure modes. +Every item below is either (a) sourced to a vendor/upstream URL, or (b) +MEASURED on this project and sourced to the repo's own D-NNN / changelog. +Nothing here is generic advice: each entry exists because it cost, or would +have cost, a session. + +Companion files: `references/opentofu-provider-docs.md` (provider schema + +fetch methodology), `references/troubleshooting.md` (method). Start at the +verbatim-error index at the bottom if you have an error string in hand. + +## Version pins that actually matter (and why) + +| Thing | Pin | Why the pin exists | +|---|---|---| +| `dmacvicar/libvirt` provider | `0.9.8` | 0.9.0 was a compatibility-BREAKING rewrite; anything remembered from 0.8-era examples is wrong (see below) | +| OpenTofu | `v1.12.3` (as-run) | the binary that actually executed Stage 1 | +| LXD | **`5.21` LTS track** | MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7 (D-114; see below) | +| MAAS | 3.7.2 (VR0) | the LXD ceiling above is a property of MAAS 3.6/3.7 | +| OPNsense | 26.1 "Witty Woodpecker" | ISC-DHCP left core in 26.1 -- Kea/dnsmasq only (see below) | +| Vault charm | `1.8/stable` | 1.16 is an incompatible charm, deliberately NOT an upgrade (appendix-B, D-068) | + +Charm channels/revisions are appendix-B's job, not this file's. This file +covers only the substrate the charms sit on. + +--- + +## 1. dmacvicar/libvirt provider (pinned 0.9.8) + +### 1a. The 0.8 -> 0.9 break invalidates most examples you will find + +v0.9.0 (2025-11-08) is a deliberate, compatibility-breaking rewrite on the +plugin framework, with the design principle "HCL maps almost 1:1 to libvirt +XML"; the legacy provider stayed on the `v0.8` branch. v0.9.1 (2025-11-30) +then code-generated the schema from libvirt's XML schema, renaming attributes +again (`unit` -> `memory_unit`, `os.arch` -> `os.type_arch`, `os.machine` -> +`os.type_machine`, `os.kernel_args` -> `os.cmdline`, camelCase -> snake_case +throughout). +- https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.0 +- https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.1 + +**Consequence:** blog posts, StackOverflow answers, and model memory almost +all describe 0.8. Nested structures are attribute-style objects +(`os = { ... }`, `devices = { ... }`), not HCL blocks. Confirm against +`docs/resources/*.md` + `examples/*.tf` at the pinned tag, or +`tofu providers schema -json`. + +### 1b. Bare `memory` is KiB, not MiB -- 1024x too little RAM (DOCFIX-188) + +The 0.9.1 migration guide states the rule explicitly: "Value/unit pairs are +explicit -- whenever libvirt exposes a value with a unit attribute the +provider now has two attributes (`memory` + `memory_unit`, `capacity` + +`capacity_unit`, etc.). **Leaving the unit unset lets libvirt use its +default.**" +(https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.1) + +And libvirt's default is KiB: "The units for this value are determined by the +optional attribute `unit`, which defaults to 'KiB'" +(https://libvirt.org/formatdomain.html#memory-allocation). + +So `memory = 2048` with no `memory_unit` is **2 MiB**, not 2 GiB. In 0.8, +`memory` meant MiB -- that is exactly how this shipped here. + + memory = var.memory_mib + memory_unit = "MiB" # REQUIRED. Never omit. + +Signatures (all measured, 2026-07-12): QEMU cmdline `-m size=2048k`; +`virsh dominfo` -> `Max memory: 2048 KiB`; a FreeBSD guest echoes +`/boot.config` (262 bytes of serial output) and then triple-faults handing off +to `/boot/loader`. `tofu validate` CANNOT catch it (the attribute is +optional) -- `scripts/opentofu-validate.sh` check **S1** does. +Evidence: `docs/changelog-20260712-libvirt-memory-unit-rootcause.md`, +`docs/incident-20260712-opnsense-edge-boot-triplefault.md`. + +The provider's own current domain examples now carry `memory_unit = "MiB"` +(`docs/resources/domain.md`) -- a fresh reader of the docs would get this +right; a reader of any 0.8-era example would not. + +### 1c. No `features` block => `acpi=off` => FreeBSD panics (DOCFIX-190) + +The whole `features` attribute is optional, so omitting it is schema-valid -- +and libvirt then renders the machine with `acpi=off` (measured on the QEMU +cmdline: `-machine pc-i440fx-noble,...,acpi=off`). + + features = { + acpi = true + apic = {} # acpi is a Boolean; apic is a nested object + } + +(schema: `docs/resources/domain.md`, `features.acpi` (Boolean), +`features.apic` (Attributes).) + +Three distinct consequences, only one of which is loud: +- FreeBSD/OPNsense **panics** (it finds the local APIC via ACPI's MADT and + ships no `atpic` fallback): `panic: running without device atpic requires a + local APIC` -> `db>` prompt -> domain shows `running` while burning 100% of + one core forever. +- Linux guests boot but degrade: no clean ACPI shutdown/reboot signalling. +- **MAAS drives power off/on via ACPI**, so a MAAS-managed node VM without + ACPI can only ever be hard-stopped. That would have been a nasty Stage-3 + debug. +Evidence: `docs/changelog-20260712-libvirt-acpi-kernel-panic.md`. +Guard: `scripts/opentofu-validate.sh` check **S2**. + +### 1d. No `cpu` block => generic CPU => NO `svm` => nested KVM impossible + +Found 2026-07-13. With no `cpu` attribute, libvirt hands the guest a generic +emulated CPU model with no virtualization flag. Measured: host is an **AMD +EPYC 9965**, the default-CPU guest was handed an **`Opteron_G3`**. Nothing +errors -- the guest just has no `/dev/kvm` and no `svm` in `/proc/cpuinfo`. + +Ubuntu's own nested-virt guide states the guest-side requirement: set the +guest `cpu mode` to `host-model` or `host-passthrough` for the guest to see +`svm`/`vmx` +(https://ubuntu.com/server/docs/how-to/virtualisation/enable-nested-virtualisation/). + + cpu = { + mode = "host-passthrough" + # per-caller: pass svm through, or disable it as router hardening + features = var.expose_nested_virt ? [] : [ + { name = "svm", policy = "disable" } + ] + } + +**The host side is a SEPARATE requirement and bites at every level.** The +vcloud host is ITSELF a KVM guest (`systemd-detect-virt` -> `kvm`), so nesting +must be enabled at each level, not just ours: +- check: `cat /sys/module/kvm_amd/parameters/nested` -> `Y` or `1` + (Intel: `/sys/module/kvm_intel/parameters/nested`) +- persist: `options kvm-amd nested=1` in a file under `/etc/modprobe.d/` + (same source as above) + +Why this matters here (D-114): LXD **virtual machines** are qemu/KVM guests, +so MAAS composing service VMs inside a site containment VM needs working +nested KVM at that depth; and `nova-compute` on the DC nodes needs it one +level deeper still. `modules/node-vm` still has the missing-`cpu`-block defect +-- logged, NOT fixed, because DC1 is gated behind Office1 +(`docs/changelog-20260713-d114-voffice1-nested-virt.md`). + +### 1e. "will be updated in-place" DOES NOT mean "the guest stays up" + +A plan line reading `libvirt_domain.vm will be updated in-place` **bounced a +live guest** (measured 2026-07-13: uptime 8m36s -> 6s; ~30s with no routing and +no DHCP on the Office1 edge). "In-place" is a statement about the Terraform +RESOURCE, not about the domain. + +Upstream confirms the mechanism: the v0.9.4 release notes describe domain +update as "updating a domain (e.g. changing memory) **would undefine and +re-define it**" (the 0.9.4 fix was only to preserve NVRAM/TPM files across +that undefine), and v0.9.6 added "configurable update shutdown behavior so +domain updates can request guest shutdown and wait with a configurable timeout +before forcing a stop." +- https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.4 +- https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.6 + +**Rule: treat ANY `tofu apply` whose plan touches a `libvirt_domain` as an +OUTAGE of that guest. Schedule and gate it as one.** Read the plan's resource +list before applying; `0 to change` on the domains is the only assurance that +a running guest is not about to bounce +(`docs/session-ledger.md`, standing lesson 2). + +Corollary (DOCFIX-187/DOCFIX-188): some domain attributes are CREATE-time to +libvirt (max boot memory, `machine`, `cpu`). The provider will happily plan an +in-place update that libvirt then ignores. If a changed value does not show up +in `virsh dominfo`, `destroy` + `undefine` the domain and re-apply (volumes +survive an `undefine`). + +### 1f. Two more upstream behaviours worth knowing (from the release notes) + +- `capacity_unit = "GiB"` on a volume caused **"Provider produced inconsistent + result after apply"** on every apply before 0.9.4 (libvirt normalizes to + bytes). Fixed in 0.9.4 -- if you see that error string, check the provider + version first. +- `tofu destroy` of a **`dir` pool** used to DELETE the backing directory + (`StoragePoolDelete`). Since 0.9.4 the directory is preserved by default, + with an explicit `destroy.delete = true` to opt back in. Relevant to + `modules/dc-storage-pool`. + (https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.4) +- **UNVERIFIED:** the 0.9.4 notes name a `destroy.shutdown.timeout` option + while `docs/resources/domain.md` documents `destroy = { graceful, timeout }`, + and I could not find the 0.9.6 update-shutdown attribute documented at all. + Before relying on either, read the real schema: + `tofu providers schema -json | jq '.provider_schemas[].resource_schemas.libvirt_domain'`. + +## 2. AppArmor blocks qemu on any non-default pool path + +libvirt's stock `abstractions/libvirt-qemu` grants qemu the DEFAULT pool path +(`/var/lib/libvirt/images`) only. A pool anywhere else is blocked by AppArmor +**even with perfect POSIX permissions**, and the failure names nothing: + +- libvirt/qemu says only: `Could not open '': Permission denied` +- the actual denial is in the kernel log: + `apparmor="DENIED" operation="open" profile="libvirt-" name=""` +- the domain DEFINES fine and fails at START -- so `tofu apply` can "succeed" + into a dead guest. + +Upstream (still open, provider issue #920, with the same workaround): +https://github.com/dmacvicar/terraform-provider-libvirt/issues/920 + +Fix, in the vendor-sanctioned override include (survives package upgrade -- +do NOT edit the shipped abstraction): + + # /etc/apparmor.d/local/abstractions/libvirt-qemu + /var/lib/libvirt/vr1/** rwk, + +This is now installed by `bash scripts/prereqs/install-apparmor-libvirt.sh` +(pool parent overridable via `VR1_POOL_PARENT`), wired into +`scripts/prereqs/install-all.sh` and reported by `check-prereqs.sh`. It cost a +session on 2026-07-12 (DOCFIX-186) because it existed only as hand-applied host +state; `docs/changelog-20260713-apparmor-libvirt-prereq.md`. + +## 3. MAAS + LXD + +### 3a. MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7 (D-114) + +Canonical's own announcement: LXD 6.7 "consolidates some API endpoints" that +MAAS's pinned **pylxd 2.3.5** cannot speak to (fixed in **pylxd >= 2.3.9**, +not yet in a MAAS release). Verbatim guidance: "We recommend using LXD <=6.6 +or the 5.21 LTS release until further notice." +https://discourse.maas.io/t/maas-incompatibility-with-lxd-6-7/15749 + +VR0 runs LXD **5.21.4** and is therefore safe by accident of timing; D-114 +makes it a decision. **LXD 5.21 LTS is supported to June 2029** +(https://canonical.com/blog/lxd_5-21-0_lts). + +**The live hazard is the snap auto-refresh, not the install.** LXD's docs: +"By default, installed snaps update automatically when new releases are +published to the channel they're tracking"; the `latest` track "typically +points to the latest feature release", is "a continuously rolling release +track", and is "not recommended for general use"; a bare `snap install lxd` +uses the most recent LTS track, "which is currently 5.21". +https://canonical.com/lxd/docs/latest/reference/releases-snap/ +=> install/refresh explicitly on `5.21/stable` and verify the tracked channel +(`snap list lxd`, `snap info lxd`) before blaming MAAS for an LXD it can no +longer talk to. + +### 3b. LXD's own bridge runs a dnsmasq DHCP server -- MAAS says turn it off + +"Bridges created by LXD are managed, which means that in addition to creating +the bridge interface itself, LXD also sets up a local `dnsmasq` process to +provide DHCP, IPv6 route announcements and DNS services to the network" +(`ipv4.dhcp` defaults to **true**). +https://canonical.com/lxd/docs/latest/reference/network_bridge/ + +MAAS's own LXD VM-host procedure therefore has you disable exactly that, +because MAAS must be the DHCP/PXE authority for the machines it composes: + + lxc network set lxdbr0 dns.mode=none + lxc network set lxdbr0 ipv4.dhcp=false + lxc network set lxdbr0 ipv6.dhcp=false + +https://canonical.com/maas/docs/how-to-manage-machines + +This is the concrete form of D-114's DHCP warning. On `office1-local`, +**OPNsense/Kea is authoritative** (10.10.0.0/24, pool .100-.199). Two DHCP +servers on one L2 is an intermittent, genuinely unpleasant failure. Never +bridge `lxdbr0` onto the site LAN without disabling its DHCP first. + +### 3c. Two different LXDs in this cloud -- do not conflate them (D-114) + +Conflating these caused a real, recorded error: +- **Juju-created LXD CONTAINERS** on the OpenStack nodes (`lxd:N` placements + in `bundle.yaml`, 21 of them). MAAS never sees these and never needs to. +- **A MAAS-registered LXD KVM host** (the VR0 `lxd` machine) into which MAAS + **COMPOSES VMs** (`maas $PROFILE vm-host compose ...`) -- VR0's `tailscale` + machine is one. These ARE MAAS machines: enlisted, commissioned, deployed, + powered, released. + +D-114 extends the second pattern per site; the first is explicitly out of +scope. Also note: MAAS only sees VMs in the LXD **project** the VM host was +registered with -- VMs in other projects are invisible to it and unaffected +(MAAS LP#1923251, https://bugs.launchpad.net/maas/+bug/1923251). + +**Composed-VM networking -- MEASURE, do not assume.** A MAAS-bundled copy of +the MAAS docs states: "When composing a virtual machine with LXD, MAAS uses +either the 'maas' LXD profile, or (if that doesn't exist) the 'default' LXD +profile. The profile is used to determine which bridge to use." +(served by a MAAS instance's own docs: +https://maas.cloud.cbh.kth.se/MAAS/docs/cli/how-to-use-lxd.html -- **I could +NOT locate this sentence on canonical.com/maas.io today; the docs site has been +reorganized and the old URLs 404. Treat the profile->bridge mechanism as +PLAUSIBLE-BUT-UNCONFIRMED against a primary source.**) Since a composed machine +that PXEs on the wrong L2 is the failure this predicts, do the cheap +measurement instead of trusting either source: `lxc profile show maas` / +`lxc profile show default` on the LXD host, and confirm the NIC's parent bridge +before composing. + +Because LXD **VMs** (not containers) are qemu/KVM guests, this pattern needs +nested KVM at whatever depth the LXD host sits -- see 1d. + +## 4. OPNsense 26.1 edge + +### 4a. `config.xml` is GUI-owned. Do not hand-author it. (D-112, D-113) + +The config-xml path is **DELETED** (template, renderer, ISO builder, and the +module's `config_seed` volume + cdrom disk -- D-113 amendment, 2026-07-13). +Configure the edge over the **REST API** (`scripts/opnsense-api.sh`); mint an +API key with `scripts/opnsense-bootstrap-apikey.sh`. Three separate defects +(no sshd, no console, an inert `` block) all traced to hand-writing +that file, and a full-config push DROPS ~667 migration-populated elements -- +including the box's only two firewall pass rules -- surviving only on an +UNDOCUMENTED self-heal at boot. If you think the API cannot express something, +that is a finding to raise against D-113, not a reason to write XML. + +Two traps that follow: +- **A config ISO can NEVER be read on a nano image** (D-112, root-caused in + upstream `opnsense/core:src/sbin/opnsense-importer`: the importer probes for + a read-only root and `bootstrap_and_exit 0`s on a pre-installed image before + scanning any media). Any instruction to build one is a trap. +- Factory default: LAN `192.168.1.1/24`, **SSH disabled**, **all inbound + blocked on WAN**. An unanswered ping to the WAN address is NOT a fault. + +### 4b. ISC dhcpd is GONE from core in 26.1 -- an `` config is inert + +26.1 release notes: "ISC-DHCP moves to a plugin. It will be automatically +installed during upgrades. **It is not installed on new installations because +it is not being used**, but you can still install and keep using it." Also: +"Dnsmasq is now the default for DHCPv4 and DHCPv6 as well as RA out of the +box." +https://docs.opnsense.org/releases/CE_26.1.html + +So on a fresh 26.1 image: an ISC `` block configures nothing (that was +DOCFIX-193), and the GUI default backend is **dnsmasq**, not Kea. This edge +deliberately runs **Kea** (10.10.0.0/24, pool .100-.199, first real lease +served to `voffice1` 2026-07-13). + +### 4c. REST API: auth, endpoints, and the reconfigure step everyone forgets + +- Auth is **HTTP basic with an API key/secret pair**, not the root password: + "API access is part of the local user authentication system, but uses + key/secret pairs to separate account information from machine to machine + communication." URL pattern: `https:///api///`. + https://docs.opnsense.org/development/how-tos/api.html +- Kea DHCPv4 (exact paths): + `/api/kea/dhcpv4/get`, `/set`, `/add_subnet`, `/set_subnet/$uuid`, + `/del_subnet/$uuid`, `/add_reservation`, `/set_reservation/$uuid`, + `/get_reservation`, `/add_option`; service control: + `/api/kea/service/reconfigure|restart|start|stop|status`. + **A `set*` call only edits config -- it is NOT live until you POST + `/api/kea/service/reconfigure`.** A silently-unapplied change looks exactly + like a change that did not take. + https://docs.opnsense.org/development/api/core/kea.html +- **Firewall rules have a self-rollback safety net -- use it.** The filter + controller exposes `savepoint`, `apply/$rollback_revision`, + `cancel_rollback/$rollback_revision`, `revert/$revision`: "When calling + `savepoint()` a new config revision will be created and the timestamp will be + returned for later use. **If the `cancelRollback(savepoint)` is not called + within 60 seconds, the firewall will rollback to the previous state**." + For any remote rule change on a router you reach THROUGH that router, a + savepoint means a mistake self-reverts instead of locking you out. + https://docs.opnsense.org/development/api/core/firewall.html +- Do NOT re-implement vendor internals to bootstrap. OPNsense stores API + secrets as `crypt(secret,'$6$')`; we deliberately do not mint keys offline -- + a format drift would fail SILENTLY (keys that never authenticate). Call the + vendor's own model instead (`scripts/opnsense-bootstrap-apikey.sh`). + (D-113 amendment.) + +### 4d. root's shell is tcsh -- `$(...)` dies QUIETLY over SSH + +Measured on the Office1 edge: root's shell is **tcsh**. tcsh has no `$(...)` +command substitution (it uses backquotes); a `$(...)` inside a quoted remote +command is rejected with: + + Illegal variable name. + +It already cost a defect: a pre-install snapshot silently no-op'd and the +install then ran with no rollback point behind it +(`docs/changelog-20260713-office1-dhcp-apply.md`). Because the step was +non-fatal, nothing stopped. + +**Always feed remote commands to `sh -s`:** + + ssh -i "$OPNSENSE_SSH_KEY" root@ 'sh -s' <<'SH' + ...POSIX shell here... + SH + +(tcsh's substitution syntax: https://man.freebsd.org/cgi/man.cgi?query=tcsh) + +--- + +## Verbatim error -> cause index (grep this first) + +| Exact string you see | Cause | Where | +|---|---|---| +| guest triple-faults right after echoing `/boot.config`; `virsh dominfo` -> `Max memory: 2048 KiB`; QEMU `-m size=2048k` | `libvirt_domain.memory` with no `memory_unit` -- KiB, not MiB | 1b | +| `panic: running without device atpic requires a local APIC` (then `db>`, 100% of one core, domain "running") | no `features` block -> `acpi=off` | 1c | +| guest has no `/dev/kvm`; `/proc/cpuinfo` has no `svm`; CPU model reads `Opteron_G3` on an EPYC host | no `cpu` block -> generic emulated CPU | 1d | +| `Could not open '': Permission denied` (domain defines, then fails to START; nothing names AppArmor) -- check `dmesg` for `apparmor="DENIED" operation="open" profile="libvirt-"` | non-default libvirt pool path not granted in AppArmor | 2 | +| `Provider produced inconsistent result after apply` on a volume with `capacity_unit` | provider < 0.9.4 | 1f | +| plan says `will be updated in-place`, guest uptime resets anyway | provider undefines + redefines the domain | 1e | +| `Illegal variable name.` from an `ssh root@ '...'` | root's shell is tcsh; `$(...)` unsupported | 4d | +| OPNsense boots on FACTORY DEFAULTS with a correct config ISO attached; console shows `>>> Invoking import script 'importer'` and nothing more | the Configuration Importer can never fire on a nano image | 4a | +| a Kea/firewall API `set` returns success but the box's behaviour does not change | no `service/reconfigure` (or no firewall `apply`) call | 4c | +| MAAS cannot talk to an LXD VM host after an LXD update | LXD auto-refreshed past 6.6; MAAS 3.6/3.7 needs LXD <= 6.6 or 5.21 LTS | 3a | +| a MAAS-composed VM gets a lease MAAS did not issue, or PXEs on the wrong L2 | `lxdbr0` runs its own dnsmasq (verified); the composing LXD profile's bridge is the other candidate (unconfirmed -- measure it) | 3b, 3c | diff --git a/.claude/skills/openstack-cloud-ops/references/troubleshooting.md b/.claude/skills/openstack-cloud-ops/references/troubleshooting.md index c5ba069..9f39241 100644 --- a/.claude/skills/openstack-cloud-ops/references/troubleshooting.md +++ b/.claude/skills/openstack-cloud-ops/references/troubleshooting.md @@ -10,6 +10,13 @@ This file gives you the method and the reflexes that must fire BEFORE and WHILE you read those. +**Substrate failures (libvirt/KVM guest won't boot, AppArmor, MAAS+LXD, +OPNsense edge):** these are NOT in appendix-A. Go to +`references/platform-traps.md` -- it ends in a verbatim-error -> cause index +keyed to exact strings (`panic: running without device atpic requires a local +APIC`, `Illegal variable name.`, a bare `Permission denied` that never names +AppArmor, `-m size=2048k`, ...). Match the string BEFORE hypothesizing. + **VR1 DC-DC track:** appendix-A above is VR0-specific -- nothing in the VR1 DC-DC track (`docs/dc-dc-deployment-workflow.md`) has been executed yet, so there is no equivalent incident index for it (an OpenTofu/MAAS-`vm_host`/ diff --git a/docs/changelog-20260713-office1-headend-maas-lxd.md b/docs/changelog-20260713-office1-headend-maas-lxd.md new file mode 100644 index 0000000..f588968 --- /dev/null +++ b/docs/changelog-20260713-office1-headend-maas-lxd.md @@ -0,0 +1,102 @@ +# changelog 2026-07-13 -- Office1 headend LIVE: MAAS 3.7.2 + LXD 5.21, D-114 proven end to end + +## What + +`voffice1` is a working VR1 site headend. **MAAS 3.7.2** (region+rack, PostgreSQL 16.14) and +**LXD 5.21.5** are installed on it, that LXD is **registered back into MAAS as an LXD VM host**, and +MAAS has **composed, PXE-booted and COMMISSIONED** its first service VM. D-114's model is proven. + +New: `scripts/site-headend-install.sh` + `tests/site-headend-install/run-tests.sh` (18/18 PASS). +The script is the install sequence we actually executed, codified -- not written from docs. Its +`--check` mode was run against the live `voffice1` and reports every item OK. + +## The proof (measured, end to end) + +``` +vcloud (L1 -- itself a KVM guest, measured) + |- voffice1 (L2) MAAS 3.7.2 region+rack + LXD 5.21.5, 10.10.0.20 (Kea reservation) + |- office1-netbox (L3) LXD VIRTUAL-MACHINE, composed by MAAS, PXE-booted by MAAS +``` + +- `office1-netbox`: composed via `maas admin vm-host compose`, leased **10.10.1.100** from MAAS's + own DHCP on `lxdbr0`, PXE-booted, ran commissioning, and reached **status `Ready`** with MAAS + correctly detecting 2 cores / 4096 MiB. It then powered off -- correct for a commissioned machine. +- **This is the DEFINITIVE L3 nested-KVM proof** D-114 asked for. Previously we had only shown + nested KVM was AVAILABLE (`/dev/kvm` + `svm`). Now a guest has actually booted, PXE'd, + commissioned and reported its hardware three levels deep. D-114's Office1 gate is CLEARED. +- It reproduces VR0's `lxd` + `tailscale` pattern exactly: MAAS's unit of visibility is the machine; + Juju's LXD containers (the OpenStack API services) remain invisible to MAAS, as in VR0. + +## The DHCP split -- structurally safe, not safe-by-convention + +| network | authority | MAAS `dhcp_on` | +|---|---|---| +| `10.10.0.0/24` (office1-local, the site LAN) | **Kea**, on the OPNsense edge | **False** | +| `10.10.1.0/24` (`lxdbr0`, the compose network) | **MAAS** | **True** | + +Two DHCP servers, two **physically separate L2s** -- `lxdbr0` is never bridged onto the site LAN, so +they cannot see each other. The script asserts this after configuring DHCP and FAILS if MAAS DHCP is +on for any subnet other than the one it was told about. + +**`10.10.1.0/24` is a NEW allocation that NetBox has not blessed.** It was chosen deliberately (DC1 +is `10.12.x`, DC2 `10.13.0.0/19`, the tenant pool is `10.20.0.0/16` per D-016 -- the obvious-looking +`10.20.x` would have collided with tenant space). It must be registered in NetBox once NetBox +exists. This is the known chicken-and-egg: NetBox is one of the VMs this headend composes. + +## FOUR TRAPS, all hit for real, all now encoded in the script and asserted by the harness + +1. **`lxd init --auto` FAILS on a host that also runs MAAS.** It creates `lxdbr0` with LXD's own + DHCP+DNS on, which starts a dnsmasq; dnsmasq binds the **wildcard** `0.0.0.0:53`, and MAAS's + bind9 (`named`) already holds `:53`. Verbatim: + `Failed starting network: The DNS and DHCP service exited prematurely: exit status 2 + ("dnsmasq: failed to create listening socket for 10.10.1.1: Address already in use")` + The message names the BRIDGE address, which sends you hunting the wrong thing -- the real + conflict is the wildcard bind. **`ipv4.dhcp=false` + `dns.mode=none` are NOT sufficient**: LXD + still spawns dnsmasq for a managed bridge and it still tries to bind `:53`. The fix that works is + `raw.dnsmasq="port=0"`. Verified after the fact: dnsmasq runs but holds **zero** sockets on `:53` + and **zero** on `:67`. +2. **`lxc` reads its config from STDIN.** Pipe this script to `bash -s` over ssh and the first `lxc` + call without a redirect **eats the rest of the script as YAML** (`Error: yaml: unmarshal errors: + line 1: cannot unmarshal !!str 'echo "...' into api.StoragePoolPut`) and everything after it + silently vanishes. Every `lxc` call now ends in `= 6.7 + (pinned pylxd 2.3.5 vs LXD 6.7's consolidated endpoints). Bonus discovered in research: + `core.trust_password` still exists in 5.21 and is GONE in 6.x, so the pin also preserves the + scriptable registration path. https://discourse.maas.io/t/maas-incompatibility-with-lxd-6-7/15749 +4. **MAAS DHCP on the compose network ONLY.** `--compose-cidr` is REQUIRED and has no default, + precisely so it can never be picked by accident, and the script hard-fails if MAAS DHCP ends up + enabled on any other subnet. + +Also encoded: `systemd-timesyncd` must be disabled (MAAS manages time via chrony); a **dynamic +iprange must exist before `dhcp_on=True` leases anything**; `dir` storage driver chosen over zfs +deliberately (no zfs loop file inside an already-nested guest). + +## Discipline notes + +- Every MAAS/LXD flag used here was read from the tool's OWN `--help` **on the box** before use + (`maas init --help`, `maas createadmin --help`, `maas vm-host compose --help`), rather than + trusted from documentation. `--password` on `createadmin` and `storage=label:size` on `compose` + were confirmed this way. +- All secrets (DB password, MAAS admin password, MAAS API key, LXD trust password) are generated ON + the target, stored `0600` under `/root/maas-secrets/`, and were never printed into the session. +- The harness asserts the four traps are still present in the script -- a "simplification" that drops + one goes RED in the gauntlet rather than in production. + +## Verification + +- `tests/site-headend-install/run-tests.sh`: **18/18 PASS**. Includes a self-check that the harness + actually catches a regression (removing the 5.21 pin makes it fail). +- `scripts/site-headend-install.sh --check` against the LIVE `voffice1`: every item OK. +- `repo-lint`: 0 fail. Full gauntlet: ALL GREEN. + +## Revert + +``` +ssh voffice1 'sudo snap remove maas postgresql lxd' # or destroy voffice1 entirely: +tofu -chdir=opentofu destroy -target=module.voffice1 +git revert +``` +Reverting the commit removes the script + harness. The OPNsense edge is untouched. Note the Kea +reservation for voffice1 lives on the edge, not in this commit -- remove it via the REST API +(`kea/dhcpv4/del_reservation`) if voffice1 is destroyed for good. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index b5b6b7b..0221070 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -68,22 +68,28 @@ The authoritative per-stage tracker is `docs/dc-dc-deployment-workflow.md` (stage table + tooling gap register). Read it FIRST. This list is only what that doc does not already carry. -1. **Stage 2 -- IN FLIGHT under D-114 (the three-service-VM model is SUPERSEDED).** `voffice1` (the - Office1 site containment VM) is BUILT AND LIVE: Ubuntu 24.04.4, 16 vCPU / 32 GiB / 600 GiB, - 10.10.0.100 by DHCP, egress through the edge, `/dev/kvm` + `svm` present (D-114's L3 nesting - probe PASSES). The Option A/B fork is RESOLVED -- OpenTofu owns it; `base-image` + `cloudinit-vm` - are instantiated for real. **REMAINING on voffice1, in order:** (a) install MAAS-region; - (b) install LXD **pinned to 5.21 LTS** (MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7); - (c) register that LXD to MAAS as an LXD KVM host; (d) COMPOSE NetBox / GitBucket / Tailscale - into it as MAAS-managed VMs. **The first composed VM is also the DEFINITIVE L3 proof** -- until - one boots, nested KVM is proven AVAILABLE, not proven WORKING. - **HARD CONSTRAINT for step (b):** LXD's `lxdbr0` runs its own dnsmasq DHCP. Kea on the edge is - authoritative on `office1-local`. `lxdbr0` must NEVER be bridged onto that LAN -- two DHCP - servers on one L2 is an intermittent failure that is genuinely unpleasant to diagnose. -1b. **BLOCKS STAGE 3 (logged, not actioned -- DC1 is gated):** `modules/node-vm` has the SAME - missing-`cpu`-block defect `cloudinit-vm` just had. DC nodes run `nova-compute`, which needs - working KVM. Without `host-passthrough` + `svm` they will come up unable to run one instance. - Fix before Stage 3. +1. **Stage 2 -- the Office1 HEADEND IS LIVE and D-114 is PROVEN END TO END (2026-07-13).** + `voffice1`: Ubuntu 24.04.4, 16 vCPU / 32 GiB / 600 GiB, **10.10.0.20** (Kea reservation, outside + the pool), egress through the OPNsense edge. On it: **MAAS 3.7.2** region+rack (PostgreSQL 16.14) + and **LXD 5.21.5**, with that LXD **registered into MAAS as an LXD VM host**. MAAS then + **composed, PXE-booted and COMMISSIONED** `office1-netbox` (LXD VIRTUAL-MACHINE, leased + 10.10.1.100 from MAAS, reached status **Ready**, 2 cores / 4096 MiB detected). + **The L3 nesting proof is now DEFINITIVE** -- a guest booted, PXE'd and commissioned three levels + deep, not merely "`/dev/kvm` exists". VR0's `lxd` + `tailscale` pattern, reproduced. + Codified in `scripts/site-headend-install.sh` (+ harness, 18/18) -- reusable for DC1/DC2. + **THE DHCP SPLIT (structural, not conventional):** `10.10.0.0/24` -> **Kea** on the edge (MAAS + dhcp_on=False); `10.10.1.0/24` (`lxdbr0`) -> **MAAS** (dhcp_on=True). Separate L2s; `lxdbr0` is + NEVER bridged onto the site LAN. The install script hard-fails if MAAS DHCP is on any other subnet. + **REMAINING on Office1:** deploy + install NetBox (composed, Ready, not yet deployed); compose + + install GitBucket and Tailscale; then run the NetBox import. +1b. **`10.10.1.0/24` IS AN UNBLESSED ALLOCATION.** Chosen deliberately (DC1=10.12.x, DC2=10.13.0.0/19, + tenant pool=10.20.0.0/16 per D-016 -- the obvious `10.20.x` would have COLLIDED with tenant space), + but NetBox has not assigned it. Register it once NetBox exists. Known chicken-and-egg: NetBox is + one of the VMs this headend composes. +1c. **`modules/node-vm` nested-virt defect: FIXED 2026-07-13** (operator: fixes land as we find them; + only DC1 *deployment* is gated). Unconditional `cpu = { mode = "host-passthrough" }` -- DC nodes + run `nova-compute` and MUST run KVM guests. Without it they would have come up looking healthy and + been unable to launch a single instance. 2. **NetBox sandbox loop is undocumented.** Operator-clarified architecture: draft-source `netbox.baldurkeep.com` (READ ONLY) -> stand up a NEW NetBox on the Office1 VM (sandbox) -> import draft + planned changes -> simulate VR1 there -> feed validated refinements BACK to diff --git a/scripts/site-headend-install.sh b/scripts/site-headend-install.sh new file mode 100644 index 0000000..44a13db --- /dev/null +++ b/scripts/site-headend-install.sh @@ -0,0 +1,269 @@ +#!/usr/bin/env bash +# scripts/site-headend-install.sh [--check|--dry-run] --compose-cidr [--maas-url-ip ] +# +# Installs a VR1 SITE HEADEND on the host it runs on: MAAS (region+rack) + LXD, with that +# LXD registered back into MAAS as an LXD VM host so MAAS can COMPOSE the site's non-stack +# service machines into it. This is D-114, and it is VR0's proven `lxd` + `tailscale` +# pattern applied per site. +# +# RUN IT ON THE SITE HOST (voffice1 today; vdc1/vdc2 later), as root: +# ssh 'sudo bash -s' -- --compose-cidr 10.10.1.0/24 < scripts/site-headend-install.sh +# +# --check report what is installed/configured; mutate nothing; exit 0 ok / 1 incomplete +# --dry-run print what it would do; mutate nothing +# (no flag) install + configure (idempotent; safe to re-run) +# +# EXIT: 0 ok | 1 check-failed | 2 bad args | 3 unsupported OS | 4 install failed. ASCII + LF. +# +# ===================================================================================== +# THE FOUR TRAPS THIS SCRIPT EXISTS TO ENCODE (all hit for real on 2026-07-13; do not +# "simplify" any of them away): +# +# 1. `lxd init --auto` FAILS OUTRIGHT on a host that also runs MAAS. +# It creates lxdbr0 with LXD's own DHCP+DNS ON, which starts a dnsmasq; dnsmasq binds +# the WILDCARD 0.0.0.0:53, and MAAS's bind9 (`named`) already holds :53. Verbatim: +# Failed starting network: The DNS and DHCP service exited prematurely: exit status 2 +# ("dnsmasq: failed to create listening socket for 10.10.1.1: Address already in use") +# Note the message names the BRIDGE address, which sends you hunting the wrong thing -- +# the real conflict is the wildcard bind. So: NEVER `lxd init --auto` here. Create the +# bridge explicitly with DHCP/DNS off AND `raw.dnsmasq=port=0`, which stops dnsmasq +# binding :53 at all. (`ipv4.dhcp=false` + `dns.mode=none` alone are NOT enough -- LXD +# still spawns dnsmasq for a managed bridge, and it still tries to bind :53.) +# +# 2. `lxc` READS ITS CONFIG FROM STDIN. If you pipe this script to `bash -s` over ssh, the +# FIRST `lxc` call with no stdin redirect EATS THE REST OF THE SCRIPT as YAML and dies +# with `yaml: unmarshal errors`, silently truncating everything after it. Every `lxc` +# call below therefore ends in `= 6.7 (LXD 6.7 consolidated API +# endpoints; MAAS's pinned pylxd 2.3.5 cannot speak to them; fixed in pylxd >= 2.3.9, +# not yet in a MAAS release). Canonical's guidance: LXD <= 6.6, or the 5.21 LTS track. +# https://discourse.maas.io/t/maas-incompatibility-with-lxd-6-7/15749 +# Bonus: `core.trust_password` still EXISTS in 5.21 and is GONE in 6.x -- the pin also +# preserves the scriptable registration path. +# +# 4. MAAS DHCP IS ENABLED ON THE COMPOSE NETWORK ONLY -- NEVER on the site LAN. The site +# LAN already has an authoritative DHCP server (Kea, on the OPNsense edge). Two DHCP +# servers on one L2 is an intermittent failure that is genuinely unpleasant to +# diagnose. --compose-cidr is REQUIRED and has no default precisely so this can never +# be picked by accident, and the script REFUSES to touch DHCP on any other subnet. +# ===================================================================================== +set -uo pipefail + +CHECK=0; DRYRUN=0; COMPOSE_CIDR=""; MAAS_IP="" +while [ $# -gt 0 ]; do + case "$1" in + --check) CHECK=1 ;; + --dry-run) DRYRUN=1 ;; + --compose-cidr) shift; COMPOSE_CIDR="${1:-}" ;; + --maas-url-ip) shift; MAAS_IP="${1:-}" ;; + -h|--help) + echo "usage: site-headend-install.sh [--check|--dry-run] --compose-cidr [--maas-url-ip ]" + exit 0 ;; + *) echo "FAIL: unknown arg '$1'" >&2; exit 2 ;; + esac + shift +done + +[ -n "$COMPOSE_CIDR" ] || { echo "FAIL: --compose-cidr is REQUIRED (e.g. 10.10.1.0/24). It has no default on purpose: MAAS must serve DHCP on the compose network ONLY, never on the site LAN, which the edge's Kea already owns." >&2; exit 2; } +echo "$COMPOSE_CIDR" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+\.0/24$' || { echo "FAIL: --compose-cidr must be a /24 like 10.10.1.0/24 (got '$COMPOSE_CIDR')" >&2; exit 2; } + +BASE="${COMPOSE_CIDR%.0/24}" # 10.10.1 +BRIDGE_IP="${BASE}.1" +RANGE_LO="${BASE}.100" +RANGE_HI="${BASE}.200" +SECRETS=/root/maas-secrets +MAAS_CHANNEL="3.7/stable" +PG_CHANNEL="16/stable" +LXD_CHANNEL="5.21/stable" # TRAP 3 -- do not change without reading it + +run() { # run -- + local d="$1"; shift; [ "${1:-}" = "--" ] && shift + if [ "$DRYRUN" = "1" ]; then printf ' [dry-run] %s\n $ %s\n' "$d" "$*"; return 0; fi + echo " -> $d"; "$@" +} +have() { command -v "$1" >/dev/null 2>&1; } + +# ---------------- check mode (read-only) ---------------- +report() { + local bad=0 + snap list maas >/dev/null 2>&1 && echo "OK: maas snap $(snap list maas 2>/dev/null | awk 'NR>1{print $2" ("$4")"}')" || { echo "ABSENT: maas snap"; bad=1; } + snap list postgresql >/dev/null 2>&1 && echo "OK: postgresql $(snap list postgresql 2>/dev/null | awk 'NR>1{print $2}')" || { echo "ABSENT: postgresql snap"; bad=1; } + if snap list lxd >/dev/null 2>&1; then + local tr; tr="$(snap list lxd 2>/dev/null | awk 'NR>1{print $4}')" + if [ "$tr" = "$LXD_CHANNEL" ]; then echo "OK: lxd on $tr" + else echo "WRONG TRACK: lxd is on '$tr', must be '$LXD_CHANNEL' (MAAS breaks on LXD >= 6.7; track changes are ONE-WAY -- a reinstall is needed)"; bad=1; fi + else echo "ABSENT: lxd snap"; bad=1; fi + + if have lxc && lxc network show lxdbr0 /dev/null 2>&1; then + echo "OK: lxdbr0 exists (ipv4.dhcp=$(lxc network get lxdbr0 ipv4.dhcp /dev/null), dns.mode=$(lxc network get lxdbr0 dns.mode /dev/null))" + local n53 n67 + n53=$(ss -lunp 2>/dev/null | grep dnsmasq | grep -c ':53 ') + n67=$(ss -lunp 2>/dev/null | grep dnsmasq | grep -c ':67 ') + if [ "${n53:-0}" = "0" ] && [ "${n67:-0}" = "0" ]; then + echo "OK: dnsmasq binds NO :53 and NO :67 -- MAAS and the edge's Kea remain sole DHCP/DNS authorities" + else + echo "FAIL: dnsmasq holds :53=$n53 :67=$n67 -- it is FIGHTING MAAS/Kea (see trap 1)"; bad=1 + fi + else echo "ABSENT: lxdbr0"; bad=1; fi + + if have maas && maas status 2>/dev/null | grep -q regiond; then echo "OK: MAAS initialised"; else echo "ABSENT: MAAS not initialised"; bad=1; fi + return "$bad" +} + +report_st=0 +if [ "$CHECK" = "1" ]; then report; exit $?; fi +report >/dev/null 2>&1; report_st=$? +[ "$report_st" = "0" ] && [ "$DRYRUN" = "0" ] && { echo "-- site headend already installed and configured --"; exit 0; } + +have apt-get || { echo "FAIL: Debian/Ubuntu (apt) only." >&2; exit 3; } +# root is required to MUTATE, not to plan: --dry-run stays runnable by anyone (and by the harness). +[ "$DRYRUN" = "1" ] || [ "$(id -u)" = "0" ] || { echo "FAIL: must run as root (use: ssh 'sudo bash -s' -- ... < $0)" >&2; exit 4; } +[ -n "$MAAS_IP" ] || MAAS_IP="$(ip -4 route get 1.1.1.1 2>/dev/null | awk '{print $7; exit}')" +[ -n "$MAAS_IP" ] || { echo "FAIL: could not measure this host's outbound IP; pass --maas-url-ip" >&2; exit 4; } + +install -d -m 0700 "$SECRETS" +newpass() { # newpass -- generate once, 0600, NEVER echoed + [ -s "$1" ] && return 0 + openssl rand -base64 30 | tr -dc 'A-Za-z0-9' | head -c 28 > "$1"; chmod 0600 "$1" +} + +echo "== 1. time: MAAS manages time via chrony; systemd-timesyncd conflicts ==" +run "disable systemd-timesyncd" -- systemctl disable --now systemd-timesyncd >/dev/null 2>&1 + +echo "== 2. MAAS $MAAS_CHANNEL + PostgreSQL $PG_CHANNEL ==" +snap list maas >/dev/null 2>&1 || run "install maas" -- snap install maas --channel="$MAAS_CHANNEL" || exit 4 +snap list postgresql >/dev/null 2>&1 || run "install postgresql" -- snap install postgresql --channel="$PG_CHANNEL" || exit 4 + +echo "== 3. MAAS database (password generated here, 0600, never printed) ==" +if [ "$DRYRUN" = "0" ]; then + newpass "$SECRETS/db.pass"; DBPASS="$(cat "$SECRETS/db.pass")" + postgresql.psql -U postgres -h /tmp -tAc "SELECT 1 FROM pg_roles WHERE rolname='maas'" 2>/dev/null | grep -q 1 \ + || postgresql.psql -U postgres -h /tmp -c "CREATE ROLE maas LOGIN PASSWORD '$DBPASS'" >/dev/null 2>&1 + postgresql.psql -U postgres -h /tmp -tAc "SELECT 1 FROM pg_database WHERE datname='maas'" 2>/dev/null | grep -q 1 \ + || postgresql.createdb -U postgres -h /tmp -O maas maas >/dev/null 2>&1 + echo " -> role + database 'maas' present" +else + echo " [dry-run] create role+db 'maas' with a generated password" +fi + +echo "== 4. maas init region+rack ==" +if [ "$DRYRUN" = "0" ]; then + if maas status 2>/dev/null | grep -q regiond; then + echo " -> already initialised" + else + maas init region+rack --maas-url "http://${MAAS_IP}:5240/MAAS" \ + --database-uri "postgres://maas:$(cat "$SECRETS/db.pass")@localhost/maas" --force >/dev/null 2>&1 \ + || { echo "FAIL: maas init" >&2; exit 4; } + echo " -> initialised" + fi + newpass "$SECRETS/admin.pass" + maas apikey --username admin >/dev/null 2>&1 || \ + maas createadmin --username admin --password "$(cat "$SECRETS/admin.pass")" --email "admin@$(hostname)" >/dev/null 2>&1 + maas apikey --username admin > "$SECRETS/admin.apikey" 2>/dev/null; chmod 0600 "$SECRETS/admin.apikey" + for i in $(seq 1 60); do curl -sf -o /dev/null --max-time 3 "http://${MAAS_IP}:5240/MAAS/" && break; sleep 5; done + maas login admin "http://${MAAS_IP}:5240/MAAS" "$(cat "$SECRETS/admin.apikey")" >/dev/null 2>&1 + echo " -> admin ready, CLI logged in" +else + echo " [dry-run] maas init region+rack + createadmin + login" +fi + +echo "== 5. LXD, DIRECTLY onto $LXD_CHANNEL (trap 3: track changes are ONE-WAY) ==" +if snap list lxd >/dev/null 2>&1; then + tr="$(snap list lxd 2>/dev/null | awk 'NR>1{print $4}')" + [ "$tr" = "$LXD_CHANNEL" ] || { echo "FAIL: lxd already on track '$tr'; you CANNOT downgrade a snap track. Remove it (snap remove lxd) and re-run." >&2; exit 4; } +else + run "install lxd" -- snap install lxd --channel="$LXD_CHANNEL" || exit 4 + run "hold lxd auto-refresh" -- snap refresh --hold lxd >/dev/null 2>&1 +fi +[ "$DRYRUN" = "0" ] && lxd waitready --timeout=90 + +echo "== 6. LXD storage + compose bridge (built explicitly; see trap 1 -- the auto-init path FAILS here) ==" +if [ "$DRYRUN" = "0" ]; then + # dir driver: conservative inside an already-nested guest (no zfs loop file). + lxc storage show default /dev/null 2>&1 || lxc storage create default dir /dev/null 2>&1 + if ! lxc network show lxdbr0 /dev/null 2>&1; then + lxc network create lxdbr0 \ + ipv4.address="${BRIDGE_IP}/24" ipv4.nat=true ipv4.dhcp=false \ + ipv6.address=none dns.mode=none raw.dnsmasq="port=0" /dev/null 2>&1 \ + || { echo "FAIL: could not create lxdbr0 (see trap 1)" >&2; exit 4; } + fi + lxc profile device remove default eth0 /dev/null 2>&1 + lxc profile device add default root disk path=/ pool=default /dev/null 2>&1 + lxc profile device add default eth0 nic network=lxdbr0 name=eth0 /dev/null 2>&1 + lxc config set core.https_address '[::]:8443' lxdbr0 ${BRIDGE_IP}/24, NAT on, LXD DHCP/DNS OFF; API on :8443" + + n53=$(ss -lunp 2>/dev/null | grep dnsmasq | grep -c ':53 ') + n67=$(ss -lunp 2>/dev/null | grep dnsmasq | grep -c ':67 ') + [ "${n53:-0}" = "0" ] && [ "${n67:-0}" = "0" ] \ + || { echo "FAIL: dnsmasq is holding :53=$n53 :67=$n67 -- it will fight MAAS/Kea (trap 1)" >&2; exit 4; } + echo " -> VERIFIED: dnsmasq binds no :53 and no :67" +else + echo " [dry-run] create dir pool + lxdbr0 ${BRIDGE_IP}/24 (dhcp off, dns off, raw.dnsmasq=port=0), expose API" +fi + +echo "== 7. register LXD into MAAS as a VM host ==" +if [ "$DRYRUN" = "0" ]; then + if maas admin vm-hosts read 2>/dev/null | grep -q '"type": "lxd"'; then + echo " -> LXD vm-host already registered" + else + maas admin vm-hosts create type=lxd name="$(hostname)-lxd" \ + power_address="https://${MAAS_IP}:8443" project=default \ + password="$(cat "$SECRETS/lxd-trust.pass")" >/dev/null 2>&1 \ + && echo " -> registered as $(hostname)-lxd" || { echo "FAIL: vm-hosts create" >&2; exit 4; } + fi +else + echo " [dry-run] maas admin vm-hosts create type=lxd power_address=https://${MAAS_IP}:8443" +fi + +echo "== 8. MAAS DHCP on the COMPOSE network ONLY (trap 4) ==" +if [ "$DRYRUN" = "0" ]; then + SUB_JSON="$(maas admin subnets read 2>/dev/null)" + SUB_ID="$(printf '%s' "$SUB_JSON" | python3 -c " +import sys,json +t=sys.argv[1] +for s in json.load(sys.stdin): + if s.get('cidr')==t: print(s['id']); break +" "$COMPOSE_CIDR")" + [ -n "$SUB_ID" ] || { echo "FAIL: MAAS has not discovered $COMPOSE_CIDR (is lxdbr0 up?)" >&2; exit 4; } + FAB_ID="$(printf '%s' "$SUB_JSON" | python3 -c " +import sys,json +t=sys.argv[1] +for s in json.load(sys.stdin): + if s.get('cidr')==t: print(s['vlan']['fabric_id']); break +" "$COMPOSE_CIDR")" + RACK="$(maas admin rack-controllers read 2>/dev/null | python3 -c "import sys,json;print(json.load(sys.stdin)[0]['system_id'])")" + + maas admin ipranges create type=dynamic subnet="$SUB_ID" start_ip="$RANGE_LO" end_ip="$RANGE_HI" \ + comment="MAAS DHCP/PXE for LXD-composed VMs (D-114)" >/dev/null 2>&1 + maas admin vlan update "$FAB_ID" 0 dhcp_on=True primary_rack="$RACK" >/dev/null 2>&1 \ + && echo " -> DHCP ON for $COMPOSE_CIDR ($RANGE_LO-$RANGE_HI), rack=$RACK" + + # HARD GUARD: assert MAAS did not turn DHCP on for any OTHER subnet. + BADSUB="$(maas admin subnets read 2>/dev/null | python3 -c " +import sys,json +t=sys.argv[1] +bad=[s['cidr'] for s in json.load(sys.stdin) if s['vlan'].get('dhcp_on') and s.get('cidr')!=t] +print(','.join(bad)) +" "$COMPOSE_CIDR")" + if [ -n "$BADSUB" ]; then + echo "FAIL: MAAS DHCP is ON for a subnet that is NOT the compose network: $BADSUB" >&2 + echo " The site LAN's DHCP belongs to the edge's Kea. Turn this OFF before proceeding." >&2 + exit 4 + fi + echo " -> VERIFIED: MAAS DHCP is on for $COMPOSE_CIDR and NO other subnet" +else + echo " [dry-run] iprange $RANGE_LO-$RANGE_HI + dhcp_on for $COMPOSE_CIDR only" +fi + +[ "$DRYRUN" = "1" ] && { echo " (dry-run: no changes made)"; exit 0; } +echo +echo "INSTALLED. MAAS UI: http://${MAAS_IP}:5240/MAAS (admin password: $SECRETS/admin.pass, 0600)" +echo "Compose a service VM with:" +echo " maas admin vm-host compose cores=2 memory=4096 storage=root:40 hostname=" +exit 0 diff --git a/tests/site-headend-install/run-tests.sh b/tests/site-headend-install/run-tests.sh new file mode 100644 index 0000000..8880a78 --- /dev/null +++ b/tests/site-headend-install/run-tests.sh @@ -0,0 +1,70 @@ +#!/usr/bin/env bash +# tests/site-headend-install/run-tests.sh +# +# Harness for scripts/site-headend-install.sh. Exercises arg parsing, the SAFETY GUARDS, +# and --dry-run only. It NEVER installs a snap, never touches MAAS/LXD, and never mutates +# the machine (same posture as tests/prereqs and the opnsense-* harnesses). +# +# The guards are the point of this harness. The script's whole reason to exist is that +# four specific traps cost a live session on 2026-07-13; a regression that quietly drops +# one of them would not show up in any deploy until something broke in production. +# ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +S="$(cd "$HERE/../../scripts" && pwd)/site-headend-install.sh" +pass=0; fail=0 +ok() { pass=$((pass+1)); } +bad() { fail=$((fail+1)); echo " FAIL: $1"; } +t() { local d="$1" exp="$2"; shift 2; "$@" >/dev/null 2>&1; local rc=$?; if [ "$rc" = "$exp" ]; then ok; else bad "$d (rc=$rc want=$exp)"; fi; } + +# 1. arg contract +t "--help -> 0" 0 bash "$S" --help +t "unknown arg -> 2" 2 bash "$S" --bogus +t "missing --compose-cidr -> 2" 2 bash "$S" --dry-run +t "non-/24 cidr -> 2" 2 bash "$S" --dry-run --compose-cidr 10.10.1.0/16 +t "garbage cidr -> 2" 2 bash "$S" --dry-run --compose-cidr not-a-cidr +t "valid --dry-run -> 0" 0 bash "$S" --dry-run --compose-cidr 10.10.1.0/24 + +# 2. --compose-cidr must have NO default. A default here is how you end up serving MAAS +# DHCP on the site LAN, where the edge's Kea is authoritative (trap 4). +out="$(bash "$S" --dry-run 2>&1)" +printf '%s' "$out" | grep -q -- "--compose-cidr is REQUIRED" && ok || bad "missing --compose-cidr must fail LOUD and say why" + +# 3. --dry-run must MUTATE NOTHING: no snap install, no lxc, no maas invoked. +out="$(bash "$S" --dry-run --compose-cidr 10.10.1.0/24 2>&1)" +printf '%s' "$out" | grep -q '\[dry-run\]' && ok || bad "--dry-run printed no [dry-run] plan" +if snap list lxd >/dev/null 2>&1 || snap list maas >/dev/null 2>&1; then + bad "--dry-run appears to have INSTALLED a snap on the test host" +else ok; fi + +# 4. the derived addresses must track --compose-cidr, not be hardcoded. +out="$(bash "$S" --dry-run --compose-cidr 10.77.5.0/24 2>&1)" +printf '%s' "$out" | grep -q "10.77.5.1/24" && ok || bad "bridge IP not derived from --compose-cidr" +printf '%s' "$out" | grep -q "10.77.5.100" && ok || bad "range start not derived from --compose-cidr" +printf '%s' "$out" | grep -q "10.77.5.200" && ok || bad "range end not derived from --compose-cidr" + +# 5. THE FOUR TRAPS must remain encoded in the script. Each of these cost a live session; +# if someone "simplifies" one away, this harness goes red rather than the cloud. +grep -q 'raw.dnsmasq' "$S" && ok || bad "trap 1 LOST: raw.dnsmasq=port=0 is what stops dnsmasq binding :53 against MAAS's bind9" +grep -q 'ipv4.dhcp=false' "$S" && ok || bad "trap 1 LOST: lxdbr0 must have LXD's own DHCP off" + +# Fold backslash-continuations first, and strip comments + echo'd strings, so we test what +# the script EXECUTES rather than what it says. (Both of these bit this harness on its +# first run: a multi-line `lxc network create` reads as missing its redirect, and an echo +# that WARNS about `lxd init --auto` reads as calling it.) +CODE="$(sed -e 's/#.*$//' "$S" | sed -e ':a' -e '/\\$/{N; s/\\\n//; ta}' | grep -vE '^[[:space:]]*echo ')" + +printf '%s' "$CODE" | grep -qE '\blxd init --auto\b' \ + && bad "trap 1 LOST: 'lxd init --auto' is EXECUTED -- it FAILS when MAAS's bind9 holds :53" || ok + +n="$(printf '%s' "$CODE" | grep -cE '\blxc\b.*= 6.7)" +grep -q 'is ON for a subnet that is NOT the compose network' "$S" \ + && ok || bad "trap 4 LOST: the guard asserting MAAS DHCP is on NO other subnet is gone" + +echo +total=$((pass+fail)) +if [ "$fail" -eq 0 ]; then echo "site-headend-install: $pass/$total PASS"; exit 0; fi +echo "site-headend-install: $fail/$total FAIL"; exit 1