diff --git a/docs/changelog-20260713-d114-voffice1-nested-virt.md b/docs/changelog-20260713-d114-voffice1-nested-virt.md new file mode 100644 index 0000000..b93d355 --- /dev/null +++ b/docs/changelog-20260713-d114-voffice1-nested-virt.md @@ -0,0 +1,78 @@ +# changelog 2026-07-13 -- D-114: voffice1 built; cloudinit-vm gains nested virt + +## What + +1. **`opentofu/modules/cloudinit-vm` gained a `cpu` block and an `expose_nested_virt` variable.** +2. **`opentofu/main.tf` instantiates `modules/base-image` (Ubuntu 24.04 LTS) and `modules/cloudinit-vm` + for `voffice1`** -- the first real instantiation of either module. +3. **APPLIED. `voffice1` is LIVE.** + +## Why -- the module was silently broken for D-114's entire model + +`cloudinit-vm`'s `libvirt_domain` had **no `cpu` block at all**. With none, libvirt renders a +GENERIC EMULATED CPU MODEL that exposes **no `svm` flag** -- so nested KVM is impossible in the +guest. Since LXD **virtual machines** are qemu/KVM guests, that meant **no service VM could ever +have been composed into `voffice1`**, and D-114's model would have failed at its first step, with a +confusing "KVM not available" rather than anything pointing at OpenTofu. + +Measured proof of the masking: the host is an **AMD EPYC 9965**, but a default-CPU guest is handed +an **"Opteron_G3"**. Post-fix, `voffice1` reports `AMD EPYC 9965` and `svm` on all 16 cores. + +`expose_nested_virt` is a per-caller decision with NO default, deliberately: +- **true** -- pass `svm` through. Required for the D-114 site containment VMs. +- **false** -- disable `svm`, matching `modules/opnsense-edge`, which disables it as legitimate + hardening for a router guest that has no business seeing nested virt. (Operator asked whether the + edge's `svm` disable was a leftover of the failed OPNsense deploy: it is NOT. It was *tried* as a + triple-fault fix on 2026-07-12 and did NOT resolve it -- the real cause was the memory unit -- and + was retained on hardening grounds. It stays disabled on the edge.) + +## voffice1 as built (measured, not inferred) + +- Ubuntu 24.04.4 LTS, 16 vCPU / 32 GiB / 600 GiB, `host-passthrough`, single NIC on `office1-local`. +- Reaches the internet THROUGH the OPNsense edge (default gw 10.10.0.1); egress to 1.1.1.1 at 4.6 ms. +- Cloud-init is DELIBERATELY MINIMAL (identity + SSH key + qemu-guest-agent). MAAS and LXD are + installed as separate GATED steps so they are observable and individually approved, not buried in + a first-boot script that either silently works or silently does not. +- The SSH public key is read at plan time with `file(var.office1_ssh_pubkey_path)` -- the key + material never enters a command line, the repo, or an agent's context. The private half is never + read (SEC-007 tracks its rotation). +- `network_config` matches the NIC by **glob** (`en*`), not a guessed kernel name: the name + (ens3/enp1s0/...) depends on machine type and PCI topology and is not knowable before first boot. + Naming it would have been an inferred value. (It came up `enp1s0`.) + +## Two gates CLEARED by this apply + +1. **KEA HAS SERVED ITS FIRST REAL DHCP LEASE.** Standing open item -- the daemon was proven + (bound udp/67, subnet + pool intact) but the SERVICE never was: `office1-local` had no client. + Measured via the D-113(a2) REST API: `10.10.0.100`, hwaddr `52:54:00:6a:87:e5` (matches + voffice1's NIC exactly), hostname `voffice1`, state active. **DHCP is now proven end to end.** +2. **D-114's NESTING PROBE PASSES at L3.** Inside `voffice1`: `/dev/kvm` PRESENT, `svm` on all 16 + cores, real EPYC CPU model. CAVEAT, stated honestly: this proves nested KVM is AVAILABLE. The + DEFINITIVE L3 proof is an actual guest booting inside `voffice1` -- that lands when the first + LXD VM is composed. Do not record L3 as fully proven until then. + +## Verification + +- `scripts/opentofu-validate.sh`: PASS (all modules). +- `tofu plan`: **5 to add, 0 to change, 0 to destroy** -- confirmed BEFORE applying that the running + OPNsense edge was NOT touched (no in-place update -> no repeat of the 2026-07-13 guest bounce). +- `tofu apply`: **5 added, 0 changed, 0 destroyed.** Edge stayed up throughout. +- repo-lint 0 fail. + +## Revert + +``` +tofu -chdir=opentofu destroy -target=module.voffice1 -target=module.ubuntu_noble_base +git revert +``` +Destroys `voffice1`, its disk/seed, and the base image; reverts the module + main.tf changes. The +OPNsense edge and every Stage-1 network/pool are untouched by both halves. NOTE the base image is a +600 MB download -- reverting and re-applying re-fetches it. + +## Follow-on FINDING (logged, NOT actioned -- DC1 is gated) + +`modules/node-vm` has the SAME missing-`cpu`-block defect. The DC OpenStack nodes run +`nova-compute`, which needs working KVM, so they will need `host-passthrough` with `svm` passed +through for the same reason. NOT fixed here: DC1 is explicitly GATED behind Office1 completing +(D-114), and fixing it now would be scope beyond the current step. It must be fixed before Stage 3 +or the DC nodes will come up unable to run a single instance. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 417a6c7..b5b6b7b 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -68,12 +68,22 @@ The authoritative per-stage tracker is `docs/dc-dc-deployment-workflow.md` (stage table + tooling gap register). Read it FIRST. This list is only what that doc does not already carry. -1. **Stage 2 is the critical path and is NOT finished: the three Office1 headend service VMs** - (MAAS-region, NetBox, GitBucket) do not exist. Blocked on two undesigned things, both repo-side: - (a) no base image source chosen; (b) no real `user_data`/`meta_data`/`network_config` written - for the `cloudinit-vm` module. The Stage 2 runbook offers an explicit fork -- Option A (OpenTofu - `cloudinit-vm`) vs Option B (manual `virt-install`, logged as debt). **UNRULED: the operator has - not picked.** Everything downstream (Stage 3+) waits on this. +1. **Stage 2 -- IN FLIGHT under D-114 (the three-service-VM model is SUPERSEDED).** `voffice1` (the + Office1 site containment VM) is BUILT AND LIVE: Ubuntu 24.04.4, 16 vCPU / 32 GiB / 600 GiB, + 10.10.0.100 by DHCP, egress through the edge, `/dev/kvm` + `svm` present (D-114's L3 nesting + probe PASSES). The Option A/B fork is RESOLVED -- OpenTofu owns it; `base-image` + `cloudinit-vm` + are instantiated for real. **REMAINING on voffice1, in order:** (a) install MAAS-region; + (b) install LXD **pinned to 5.21 LTS** (MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7); + (c) register that LXD to MAAS as an LXD KVM host; (d) COMPOSE NetBox / GitBucket / Tailscale + into it as MAAS-managed VMs. **The first composed VM is also the DEFINITIVE L3 proof** -- until + one boots, nested KVM is proven AVAILABLE, not proven WORKING. + **HARD CONSTRAINT for step (b):** LXD's `lxdbr0` runs its own dnsmasq DHCP. Kea on the edge is + authoritative on `office1-local`. `lxdbr0` must NEVER be bridged onto that LAN -- two DHCP + servers on one L2 is an intermittent failure that is genuinely unpleasant to diagnose. +1b. **BLOCKS STAGE 3 (logged, not actioned -- DC1 is gated):** `modules/node-vm` has the SAME + missing-`cpu`-block defect `cloudinit-vm` just had. DC nodes run `nova-compute`, which needs + working KVM. Without `host-passthrough` + `svm` they will come up unable to run one instance. + Fix before Stage 3. 2. **NetBox sandbox loop is undocumented.** Operator-clarified architecture: draft-source `netbox.baldurkeep.com` (READ ONLY) -> stand up a NEW NetBox on the Office1 VM (sandbox) -> import draft + planned changes -> simulate VR1 there -> feed validated refinements BACK to @@ -101,11 +111,10 @@ fiber is East-West/replication ONLY, never an internet path). 8. `docs/dc-dc-deployment-workflow.md`'s status table was corrected 2026-07-13, but re-read it against reality before trusting any row -- it has drifted twice. -9. **DHCP HAS NEVER SERVED A REAL LEASE.** `kea-dhcp4` is running and bound to udp/67 with the - subnet and pool intact -- but `office1-local` has no client host yet, so no client has ever - taken a lease. The daemon is proven; the SERVICE is not proven end-to-end. The first VM on - `office1-local` (i.e. the Stage 2 headend VMs, item 1) is the real test -- treat a successful - lease as that stage's gate, not an assumption. +9. **DHCP PROVEN END TO END -- CLOSED 2026-07-13.** `voffice1` (the first client ever to sit on + `office1-local`) took a real lease from Kea: `10.10.0.100`, hwaddr `52:54:00:6a:87:e5` (matches + its NIC), hostname `voffice1`, state active -- read back through the D-113(a2) REST API. The + service, not just the daemon, now works. 10. **Doc-accuracy finding (measured 2026-07-13, logged not actioned):** `docs/security-ledger.md` SEC-007 says "D-112(c) makes SSH the only management path". MEASURED on the live edge: the web GUI IS listening on the LAN side (10.10.0.1:443 HTTPS 200; :80 301-redirects to it), and is diff --git a/opentofu/main.tf b/opentofu/main.tf index 1039972..424416f 100644 --- a/opentofu/main.tf +++ b/opentofu/main.tf @@ -137,3 +137,86 @@ link_name = "dc2-office1" mtu = var.underlay_mtu } + +# ===================================================================== +# D-114 -- Office1 site containment VM + its base image. +# +# voffice1 IS the Office1 "facility": MAAS-region runs on it, LXD runs on it, and the +# non-stack service machines (NetBox, GitBucket, Tailscale) are VMs MAAS COMPOSES into +# that LXD host -- VR0's proven `lxd` + `tailscale` pattern, applied per site. +# +# expose_nested_virt = true is LOAD-BEARING, not a nicety: LXD virtual machines are +# qemu/KVM guests, so without the host `svm` feature passed through NO service VM can +# ever boot inside voffice1 and D-114's model fails at its first step. (cloudinit-vm had +# NO cpu block at all before 2026-07-13, which silently gave every guest an emulated CPU +# with no svm -- measured: host is an EPYC 9965, a default-CPU guest sees "Opteron_G3".) +# +# Only network is office1-local: voffice1 reaches the internet THROUGH the OPNsense edge, +# like a real server behind a real site router. It takes its address by DHCP from the +# edge's Kea -- which also makes it the FIRST REAL LEASE that path has ever served. +# +# Cloud-init here is DELIBERATELY MINIMAL (identity + access + guest agent). MAAS and LXD +# are installed as separate GATED steps so they are observable and individually approved, +# not buried in a first-boot script that either silently works or silently does not. +# ===================================================================== + +module "ubuntu_noble_base" { + source = "./modules/base-image" + image_name = "ubuntu-24.04-base.qcow2" + pool_name = module.office1_storage.pool_name + # Official Ubuntu 24.04 LTS (noble) cloud image. VERIFIED reachable 2026-07-13 (HTTP 200, + # Last-Modified 2026-07-05). 24.04 matches what VR0 deploys for its lxd/juju/tailscale + # machines, so the image choice carries no new delta. + source_url = "https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img" +} + +module "voffice1" { + source = "./modules/cloudinit-vm" + vm_name = "voffice1" + vcpu = var.voffice1_vcpu + memory_mib = var.voffice1_memory_mib + disk_size_bytes = var.voffice1_disk_bytes + pool_name = module.office1_storage.pool_name + base_volume_path = module.ubuntu_noble_base.path + network_names = [module.office1_network.network_name] + expose_nested_virt = true + + user_data = <<-EOT + #cloud-config + hostname: voffice1 + fqdn: voffice1.${var.domain_suffix} + manage_etc_hosts: true + users: + - name: jessea123 + groups: [adm, sudo] + shell: /bin/bash + sudo: "ALL=(ALL) NOPASSWD:ALL" + ssh_authorized_keys: + - ${trimspace(file(var.office1_ssh_pubkey_path))} + package_update: true + packages: + - qemu-guest-agent + runcmd: + - [systemctl, enable, --now, qemu-guest-agent] + EOT + + meta_data = <<-EOT + instance-id: voffice1-d114 + local-hostname: voffice1 + EOT + + # Match by GLOB, not a guessed interface name: the NIC's kernel name (ens3 / enp1s0 / + # ...) depends on the machine type and PCI topology and is NOT known until first boot. + # Naming it would be an inferred value. dhcp4 is a DELIBERATE choice here, not the + # module's refused default: office1-local genuinely has an authoritative DHCP server + # (Kea on the OPNsense edge, pool .100-.199). A Kea RESERVATION is added afterwards, + # via the OPNsense REST API, to make the address stable. + network_config = <<-EOT + version: 2 + ethernets: + lan: + match: + name: "en*" + dhcp4: true + EOT +} diff --git a/opentofu/modules/cloudinit-vm/main.tf b/opentofu/modules/cloudinit-vm/main.tf index 7e3b52f..8f9cc8d 100644 --- a/opentofu/modules/cloudinit-vm/main.tf +++ b/opentofu/modules/cloudinit-vm/main.tf @@ -87,6 +87,23 @@ apic = {} } + # host-passthrough ALWAYS: with no cpu block at all libvirt renders a generic emulated + # CPU model that exposes NO `svm` flag, so nested KVM is impossible in the guest no + # matter what is asked for. (Measured 2026-07-13: the host is an EPYC 9965 while a + # default-CPU guest is handed an "Opteron_G3".) `svm` itself is then passed through or + # disabled per var.expose_nested_virt -- see modules/opnsense-edge, which disables it as + # hardening for a router guest. D-114's site containment VMs REQUIRE it passed through: + # they run LXD VMs composed by MAAS. + cpu = { + mode = "host-passthrough" + features = var.expose_nested_virt ? [] : [ + { + name = "svm" + policy = "disable" + } + ] + } + os = { type = "hvm" type_arch = "x86_64" diff --git a/opentofu/modules/cloudinit-vm/variables.tf b/opentofu/modules/cloudinit-vm/variables.tf index 9a920ee..83d8ad5 100644 --- a/opentofu/modules/cloudinit-vm/variables.tf +++ b/opentofu/modules/cloudinit-vm/variables.tf @@ -66,3 +66,23 @@ EOT type = string } + +variable "expose_nested_virt" { + description = <<-EOT + Whether this guest may itself run KVM guests (nested virtualisation). + + TRUE -- pass the host's `svm` feature through. REQUIRED for any VM that is + registered to MAAS as an LXD VM host and has VMs composed into it + (D-114: the site containment VMs voffice1/vdc1/vdc2). Without it the + guest gets no svm flag, LXD virtual machines cannot start, and the + D-114 model fails at its first step. + FALSE -- disable `svm`, matching modules/opnsense-edge. Correct hardening for + a guest that has no business seeing nested virt. + + No default: this is a real decision per VM, not a fallback. NOTE the domain + always uses cpu mode "host-passthrough" -- with NO cpu block at all (this + module's state before 2026-07-13) libvirt renders a generic emulated CPU model + that exposes no svm regardless of what this flag says. + EOT + type = bool +} diff --git a/opentofu/variables.tf b/opentofu/variables.tf index 905bbc5..31f16c9 100644 --- a/opentofu/variables.tf +++ b/opentofu/variables.tf @@ -64,3 +64,30 @@ # NetBox assigns it, add a dc2_planes variable shaped exactly like dc1_planes # above and a matching module "dc2_planes" block in main.tf (see the commented # skeleton there). + +# ---- D-114: Office1 site containment VM (voffice1) ---------------------------- +variable "office1_ssh_pubkey_path" { + description = <<-EOT + Path to the PUBLIC half of the Office1 service SSH key, injected into voffice1's + cloud-init. Read with file() at plan time so the key material never has to be pasted + into the repo or a command line. The PRIVATE half stays jumphost-local and is never + read (SEC-007 tracks its rotation). Public keys are not secret; the path is a var so + the location is not hardcoded. + EOT + type = string +} + +variable "voffice1_vcpu" { + description = "voffice1 vCPU count. Hosts MAAS-region + the LXD VM host and the service VMs MAAS composes into it (D-114)." + type = number +} + +variable "voffice1_memory_mib" { + description = "voffice1 memory in MiB." + type = number +} + +variable "voffice1_disk_bytes" { + description = "voffice1 boot disk size in bytes." + type = number +}