diff --git a/docs/design-decisions.md b/docs/design-decisions.md index 858bf49..2df87ea 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -2462,3 +2462,112 @@ D-112 (the (c) ruling and its stated rationale). --- + +## D-114: VR1 site containment VMs + MAAS-composed LXD VMs for the non-stack machines + +**Status:** **ADOPTED 2026-07-13 (operator ruling).** AMENDS D-103 (which assigned Office1 three +sibling service VMs and put every VR1 domain directly on the vcloud host). Supersedes nothing else. + +**Context.** Stage 2 was about to build Office1 per D-103: three peer VMs (MAAS-region, NetBox, +GitBucket) created directly on vcloud libvirt, with Tailscale installed on the headend. The +operator described a different shape -- each site (Office1, DC1, DC2) living inside its OWN VM +environment, with MAAS running LXD to hold "the rest of the servers/services." An initial reading +of `bundle.yaml` found only Juju's `lxd:8`-style CONTAINER placements and concluded there was no +precedent for MAAS composing LXD VMs. **That conclusion was WRONG**, and is recorded here because +the error is instructive: the bundle's Juju LXD containers and MAAS's LXD VM host are TWO DIFFERENT +LXD usages in the same cloud, and reading only the bundle collapses them into one. + +**Measured as-built evidence (VR0 / `openstack-caracal-ipv4` MAAS 3.7.2, operator screenshots +2026-07-13).** Seven machines, all Deployed, ALL powered via Virsh: + +- `openstack0-3` -- 8 cores / 32 GiB / 1.1 TB, 6 spaces, tags `openstack, virtual`. Juju machines + 8-11; the OpenStack API services run in JUJU-created LXD CONTAINERS on these (21 `lxd:N` + placements in `bundle.yaml`). MAAS never sees those containers, and never needed to. +- `juju` -- 2 cores / 4 GiB, tags `juju, virtual`. The controller. +- `lxd` -- 8 cores / 16 GiB / 687 GB, tags `lxd, virtual`. A MAAS-DEPLOYED, virsh-powered libvirt + VM that is then REGISTERED BACK INTO MAAS as an LXD KVM host (project `default`, LXD **5.21.4**; + host NICs enp1s0 / enp7s0 / lxdbr0). +- `tailscale` -- 2 cores / 2 GiB / 25 GB, Ubuntu 24.04, tags `pod-console-logging, virtual`. + **It is a VM COMPOSED BY MAAS INSIDE the `lxd` KVM host.** This is the proof of the pattern. + +So the pattern the operator asked for is not new -- it is VR0's own, already proven in production +on this cloud, and it costs 2 cores / 2 GiB per service machine. + +### Ruling + +1. **Non-stack machines become MAAS-composed LXD VMs.** Per site, one MAAS-deployed machine runs + LXD and is registered back into MAAS as an LXD KVM host; the machines that live OUTSIDE the + OpenStack stack (NetBox, GitBucket, Tailscale, and any future service box) are COMPOSED into it + by MAAS. They are therefore MAAS-visible: enlisted, commissioned, deployed, powered, released. + This is what the operator meant by visibility, and it is `lxd` + `tailscale` reproduced per site. + **Explicitly NOT in scope:** the Juju-deployed OpenStack services on the DC nodes. Those stay in + Juju-created LXD CONTAINERS (bundle `lxd:N` placements), invisible to MAAS, exactly as in VR0. + Operator: "I don't care about visibility into the juju deployments into the openstack0-3 + machines as much as the other machines running outside of the stack." + +2. **Each site gets a containment VM** -- `voffice1`, `vdc1`, `vdc2` -- simulating the facility that + the site's servers and services operate within. Rationale (operator): **full-facility snapshots** + and **the ability to simulate total geographical failure of a facility.** The second is the + decisive one: D-108's DR failover drill currently has NO honest "hard-down an entire DC" + primitive, and `virsh destroy vdc1` IS that primitive. This is a capability the flat model + cannot offer at any price. + +3. **Sequencing (operator ruling):** Office1 FIRST, fully up and running, on this model. DC1 is + **GATED** behind it and is not started until Office1 is complete. DC2 follows DC1. + +4. **LXD is version-pinned to the 5.21 LTS track.** MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7: + LXD 6.7 consolidated API endpoints and MAAS's pinned pylxd 2.3.5 cannot speak to them (fixed in + pylxd >= 2.3.9, not yet in a MAAS release). Canonical's own guidance is "use LXD <= 6.6 or the + 5.21 LTS release until further notice." VR0 is already on 5.21.4, i.e. on the safe side by + accident of timing rather than by decision -- this makes it a decision. Goes in appendix-B. + Source: https://discourse.maas.io/t/maas-incompatibility-with-lxd-6-7/15749 + +### What this changes in D-103 + +D-103 said OpenTofu owns creation of "the Office1 service VMs (MAAS-region, NetBox, GitBucket)" as +domains on the vcloud host. Under D-114, OpenTofu owns creation of the SITE CONTAINMENT VMs and the +site's networks; INSIDE a site, the service machines are composed by MAAS into LXD. D-103's core +principle is UNCHANGED and is in fact extended: MAAS still owns the provisioning lifecycle, and +still does not compose the OpenStack NODE VMs (`modules/node-vm` pre-creates those; `maas_vm_host` +registers the virsh host so MAAS DISCOVERS them -- `maas_vm_host_machine` remains ruled out for +nodes). The composition right granted here is scoped to the LXD VM host and the non-stack machines. + +### Known risk, and its entry gate + +**Nesting depth.** `vcloud` is ITSELF a KVM guest (measured: `systemd-detect-virt` -> `kvm`). A site +containment VM therefore puts the site's servers one level deeper than VR0 runs them: DC node VMs +land at L3 and OpenStack TENANT INSTANCES at L4. VR0 proves nesting works to the depth IT uses +(`tailscale` inside `lxd`), but VR0's hosts sit beside vcloud on the same 10.17.11.0/22 lab network +and are thus one level SHALLOWER than a VR1 pod. L4 tenant instances are UNPROVEN, and nova-compute +needs working KVM at the bottom of that stack. The 2026-07-12 OPNsense BTX triple-fault is prior +evidence that this environment is already fragile at depth. + +**Office1 IS the nesting probe -- and that is a feature of the sequencing, not a coincidence.** +(An earlier draft of this entry claimed "Office1 needs no nested KVM." That was WRONG and is +corrected here: LXD **virtual machines** are qemu/KVM guests, so composing NetBox/GitBucket/ +Tailscale as LXD VMs inside `voffice1` requires nested KVM at exactly the same depth the DC nodes +would need it -- L3. Only LXD *containers* would be free, and MAAS does not enlist containers.) +Office1 is therefore the CHEAP place to prove the model: no OpenStack, no Ceph, small VMs. If an +LXD VM boots inside `voffice1`, the pod model is proven to L3 and DC1 inherits that proof. + +**Entry gate for DC1:** (a) Office1 fully up on this model (operator ruling: DC1 is gated behind +it); (b) inside `vdc1`, confirm nested KVM is exposed and a guest actually BOOTS -- which also +proves L4 is reachable for tenant instances, the one depth VR0 has never exercised. If L3 fails at +Office1, the containment-VM model is abandoned early having cost one VM. If L3 holds but L4 fails +at DC1, DC compute stays flat and Office1 keeps its containment VM. Both outcomes are scoped +retreats, not redesigns -- the ruling is deliberately written to make them cheap. + +**DHCP.** OPNsense/Kea is authoritative on `office1-local` (10.10.0.0/24, pool .100-.199). LXD's +`lxdbr0` runs its OWN dnsmasq -- that is FINE as long as it stays on LXD's internal NAT bridge and +is never bridged onto `office1-local`. Two DHCP servers on one L2 is an intermittent failure that +is genuinely unpleasant to diagnose. Any LXD networking change must not put `lxdbr0` on the site LAN. +Corollary bonus: `voffice1` booting on `office1-local` will take the FIRST REAL DHCP LEASE from Kea +-- a path proven only at the daemon level so far, never end to end. + +### Delta to Roosevelt + +The containment VM is simulation-only machinery: in Roosevelt a "facility" is a building full of +real servers, not a VM, and it is torn down by pulling power, not `virsh destroy`. That is an +ACCEPTED delta, bought deliberately: it is what makes a geographic-failure drill executable at all +in a virtual rehearsal. The LXD half carries NO delta -- it is exactly what VR0 runs today and what +Roosevelt would run on real hardware (a service host running LXD, registered to MAAS).