Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260713-d114-voffice1-nested-virt.md
@JANeumatrix JANeumatrix 15 hours ago 6 KB D-114 follow-through: fix node-vm's nested-virt defect NOW; voffice1 pinned to a reserved address
Raw Blame History

changelog 2026-07-13 -- D-114: voffice1 built; cloudinit-vm gains nested virt

What

  1. opentofu/modules/cloudinit-vm gained a cpu block and an expose_nested_virt variable.
  2. opentofu/main.tf instantiates modules/base-image (Ubuntu 24.04 LTS) and modules/cloudinit-vm for voffice1 -- the first real instantiation of either module.
  3. APPLIED. voffice1 is LIVE.

Why -- the module was silently broken for D-114's entire model

cloudinit-vm's libvirt_domain had no cpu block at all. With none, libvirt renders a GENERIC EMULATED CPU MODEL that exposes no svm flag -- so nested KVM is impossible in the guest. Since LXD virtual machines are qemu/KVM guests, that meant no service VM could ever have been composed into voffice1, and D-114's model would have failed at its first step, with a confusing "KVM not available" rather than anything pointing at OpenTofu.

Measured proof of the masking: the host is an AMD EPYC 9965, but a default-CPU guest is handed an "Opteron_G3". Post-fix, voffice1 reports AMD EPYC 9965 and svm on all 16 cores.

expose_nested_virt is a per-caller decision with NO default, deliberately:

  • true -- pass svm through. Required for the D-114 site containment VMs.
  • false -- disable svm, matching modules/opnsense-edge, which disables it as legitimate hardening for a router guest that has no business seeing nested virt. (Operator asked whether the edge's svm disable was a leftover of the failed OPNsense deploy: it is NOT. It was tried as a triple-fault fix on 2026-07-12 and did NOT resolve it -- the real cause was the memory unit -- and was retained on hardening grounds. It stays disabled on the edge.)

voffice1 as built (measured, not inferred)

  • Ubuntu 24.04.4 LTS, 16 vCPU / 32 GiB / 600 GiB, host-passthrough, single NIC on office1-local.
  • Reaches the internet THROUGH the OPNsense edge (default gw 10.10.0.1); egress to 1.1.1.1 at 4.6 ms.
  • Cloud-init is DELIBERATELY MINIMAL (identity + SSH key + qemu-guest-agent). MAAS and LXD are installed as separate GATED steps so they are observable and individually approved, not buried in a first-boot script that either silently works or silently does not.
  • The SSH public key is read at plan time with file(var.office1_ssh_pubkey_path) -- the key material never enters a command line, the repo, or an agent's context. The private half is never read (SEC-007 tracks its rotation).
  • network_config matches the NIC by glob (en*), not a guessed kernel name: the name (ens3/enp1s0/...) depends on machine type and PCI topology and is not knowable before first boot. Naming it would have been an inferred value. (It came up enp1s0.)

Two gates CLEARED by this apply

  1. KEA HAS SERVED ITS FIRST REAL DHCP LEASE. Standing open item -- the daemon was proven (bound udp/67, subnet + pool intact) but the SERVICE never was: office1-local had no client. Measured via the D-113(a2) REST API: 10.10.0.100, hwaddr 52:54:00:6a:87:e5 (matches voffice1's NIC exactly), hostname voffice1, state active. DHCP is now proven end to end.
  2. D-114's NESTING PROBE PASSES at L3. Inside voffice1: /dev/kvm PRESENT, svm on all 16 cores, real EPYC CPU model. CAVEAT, stated honestly: this proves nested KVM is AVAILABLE. The DEFINITIVE L3 proof is an actual guest booting inside voffice1 -- that lands when the first LXD VM is composed. Do not record L3 as fully proven until then.

Verification

  • scripts/opentofu-validate.sh: PASS (all modules).
  • tofu plan: 5 to add, 0 to change, 0 to destroy -- confirmed BEFORE applying that the running OPNsense edge was NOT touched (no in-place update -> no repeat of the 2026-07-13 guest bounce).
  • tofu apply: 5 added, 0 changed, 0 destroyed. Edge stayed up throughout.
  • repo-lint 0 fail.

Revert

tofu -chdir=opentofu destroy -target=module.voffice1 -target=module.ubuntu_noble_base
git revert <this commit>

Destroys voffice1, its disk/seed, and the base image; reverts the module + main.tf changes. The OPNsense edge and every Stage-1 network/pool are untouched by both halves. NOTE the base image is a 600 MB download -- reverting and re-applying re-fetches it.

Follow-on FINDING (logged, NOT actioned -- DC1 is gated)

modules/node-vm has the SAME missing-cpu-block defect. The DC OpenStack nodes run nova-compute, which needs working KVM, so they will need host-passthrough with svm passed through for the same reason. NOT fixed here: DC1 is explicitly GATED behind Office1 completing (D-114), and fixing it now would be scope beyond the current step. It must be fixed before Stage 3 or the DC nodes will come up unable to run a single instance.

ADDENDUM (same day): the DC1 fix is APPLIED, not deferred

Operator ruling: "Apply the fixes to DC1 as we find them. Fixes are not blocked, just deployment at this stage." So the modules/node-vm defect logged above as "NOT actioned" is now FIXED.

modules/node-vm gained an UNCONDITIONAL cpu = { mode = "host-passthrough" } -- no expose_nested_virt knob, deliberately. A DC node runs nova-compute; it MUST be able to run KVM guests. There is no correct value other than this one, so no knob that could only ever be set wrong. (Contrast cloudinit-vm, where the flag IS a real per-VM decision, and opnsense-edge, which DISABLES svm as correct hardening for a router.)

modules/node-vm is not yet instantiated, so adding this breaks no live call. Without it the DC nodes would have come up looking healthy while being unable to launch a single instance -- the failure would have surfaced during Stage 5 as an inscrutable scheduler error.

scripts/opentofu-validate.sh: PASS (node-vm, cloudinit-vm, all modules).

ADDENDUM: voffice1 moved to a RESERVED, out-of-pool address

voffice1 first came up on 10.10.0.100 -- the first address of Kea's DYNAMIC pool (.100-.199). Infrastructure, and a MAAS region controller above all, should not hold a pool address that can move.

A Kea host reservation was added through the D-113(a2) REST API (a second live exercise of that write path): 52:54:00:6a:87:e5 -> 10.10.0.20, deliberately OUTSIDE the dynamic pool. kea/dhcpv4/add_reservation -> {"result":"saved"}, then kea/service/reconfigure -> {"status":"ok"}. voffice1 was rebooted and came back on 10.10.0.20; .100 is released back to the pool; /dev/kvm still present. Done NOW because nothing yet depends on the address -- it is the cheapest moment it will ever be.