# FINDING 2026-07-18 -- outer `tofu plan` would BOUNCE the live Office1 headend + edge **Class:** deploy-blocking drift, discovered during pre-apply re-verification. **Status:** OPEN -- operator ruling required. **Nothing applied.** **Blocks:** the Stage-3 Phase-2 DC0 outer `tofu apply` **as currently scoped** (step A of `docs/dc0-deploy-readiness.md`). See "What is NOT blocked" -- DC0 itself has a clean path forward. ## What was measured Post-reboot re-verification of the outer root (read-only `tofu -chdir=opentofu plan -lock=false -input=false`): Plan: 7 to add, 2 to change, 7 to destroy. The committed readiness evidence (`docs/dc0-deploy-readiness.md`) is **5 add / 2 in-place / 6 destroy**. The delta is NOT in DC0 scope -- it is entirely `voffice1`, the live Office1 headend. Decomposition (accounts for the delta exactly): - adds 5 -> 7 = the intended 5 (`vvr1-dc0` domain/disk/seed-volume/cloudinit-seed + `vr1_dc0_uplink`) **+1** `voffice1` cloudinit_disk re-create **+1** `voffice1` seed-volume replace-half. - destroys 6 -> 7 = the intended 6 empty `vr1-dc0` planes **+1** `voffice1` seed-volume destroy-half. - changes stay 2 = `voffice1` and `office1_opnsense` domains, both `+ autostart = true`. > **The OpenTofu 1.12.3 -> 1.12.4 bump is COINCIDENTAL, not causal.** The binary is now > v1.12.4 (README.md:41 and all committed evidence say v1.12.3). Log it as version drift; > it is not why the plan changed. The cause is `/tmp`, below. ## THE DOMINANT RISK: the 2 "in-place" changes bounce both guests **This is the most serious item and an earlier draft of this finding got it wrong by calling it harmless.** `references/platform-traps.md` 1e is unambiguous and MEASURED -- on this very edge: > A plan line reading `libvirt_domain.vm will be updated in-place` **bounced a live > guest** (measured 2026-07-13: uptime 8m36s -> 6s; ~30s with no routing and no DHCP on > the Office1 edge). "In-place" is a statement about the Terraform RESOURCE, not about > the domain. > ... **Rule: treat ANY `tofu apply` whose plan touches a `libvirt_domain` as an OUTAGE > of that guest.** ... `0 to change` on the domains is the only assurance that a running > guest is not about to bounce. Upstream mechanism: domain update **undefines and re-defines** the domain (v0.9.4 notes). Pinned provider here is **dmacvicar/libvirt 0.9.8** (`opentofu/.terraform.lock.hcl`). So the current plan would bounce **`voffice1` AND `office1-opnsense`** -- the headend (NetBox apex 10.10.1.10 + tailscale + the D-128 Plane-2 execution origin + the D-126 ProxyJump path) and the site edge (routing, NAT, Kea DHCP). That is an unplanned Office1 site-down during what is supposed to be a DC0-scoped apply. Bitter irony worth noting: these two in-place writes exist only because `virsh autostart` was set **out-of-band** and the provider does not refresh `autostart` into state. The desired value is **already live on both domains** -- so the plan proposes to bounce two production guests to write a value that is already correct. ## The seed-volume replacement (root cause; danger UNVERIFIED) # module.voffice1.libvirt_cloudinit_disk.seed has been deleted # module.voffice1.libvirt_volume.seed must be replaced ~ create = { # forces replacement ~ content = { ~ url = "/tmp/terraform-provider-libvirt-cloudinit/ cloudinit-775234004c2669d9.iso" -> (known after apply) } } **Root cause (MEASURED):** the provider stages generated cloudinit ISOs under `/tmp/terraform-provider-libvirt-cloudinit/` and records that staging path as the volume's `create.content.url`. **The vcloud host rebooted 2026-07-17 23:38**, clearing `/tmp`: $ ls /tmp/terraform-provider-libvirt-cloudinit/ ls: cannot access ...: No such file or directory The real artifact is intact and in use: $ ls -la /var/lib/libvirt/vr1/office1/voffice1-cloudinit.iso -rw------- 1 libvirt-qemu kvm 45056 Jul 13 19:45 ... $ virsh domblklist voffice1 vda voffice1-disk.qcow2 sda voffice1-cloudinit.iso <-- ATTACHED to the RUNNING domain This is **provider-state drift, not a real configuration change** -- nothing about `voffice1`'s desired config changed. **UNVERIFIED -- do not treat as established:** what provider 0.9.8 actually does when it replaces a volume attached to a running domain it is NOT recreating. The candidates -- error out ("volume in use"), delete/recreate the file under the guest's open fd (guest unaffected until next boot), or force a domain bounce -- have materially different consequences. Trap 1e is about `libvirt_domain` in-place updates and does **not** transfer to volume replacement; citing it here would be misapplied. This question is answerable by observation on a throwaway domain, and should be answered before any path that lets this replacement run. Note the payload is largely inert regardless: cloud-init consumes the seed at first boot and `voffice1` was provisioned 2026-07-13. ## What is NOT blocked **DC0 does not have to wait for the provider question.** The intended DC0 work -- create `vvr1-dc0` + `vr1_dc0_uplink`, destroy the 6 empty `vr1-dc0` planes -- touches **no running guest**. Scoping the apply away from `voffice1` and `office1_opnsense` sidesteps both the bounce and the volume replacement entirely. The unresolved provider behavior only gates whether the `voffice1` churn is EVER allowed to run, not DC0. ## Options (NOT ruled -- operator decides) - **(a) Scope the apply.** `-target` the `vvr1_dc0` / `vr1_dc0_uplink` / `vr1_dc0_planes` addresses so the apply cannot touch either running domain. DC0 moves today with zero Office1 risk. Costs: `-target` is a break-glass tool, and the drift stays outstanding to be dealt with deliberately later. **Recommended for the immediate apply.** - **(b) Reconcile state first, then a clean full apply.** Bring the recorded seed source and the `autostart` attribute back into agreement with reality (re-stage / state surgery / `tofu apply -refresh-only`) so the plan returns to `0 to change` on the domains -- which trap 1e names as the ONLY assurance a guest will not bounce. Slower; restores "the plan matches committed evidence" as a trustworthy gate. **Recommended as the follow-up**, and it may resolve both items without any outage, since both desired values are already live. - **(c) Accept the churn.** Let the apply bounce both guests and replace the seed. **Not recommended** -- it is a deliberate Office1 outage to write already-correct values. - **(d) Durable fix (Roosevelt-transferable; orthogonal -- do regardless).** Prevent a host reboot from ever again turning a live VM's seed into a forced replacement. This will otherwise recur for EVERY cloudinit-backed VM, including all 9 DC0 node VMs and each DC containment VM, so it scales badly. **Mechanism -- partially verified:** there is **NO provider-level knob** for the staging path (VERIFIED via `tofu providers schema -json`: the provider block exposes `uri` ONLY, and `libvirt_cloudinit_disk` exposes only `id/meta_data/name/network_config/path/size/ user_data`). The staging path therefore almost certainly comes from Go's `os.TempDir()`, i.e. **`$TMPDIR` for the tofu process** -- **UNVERIFIED by observation**; confirm that setting `TMPDIR` to a persistent path actually relocates it before writing this up. Warrants **D-130** (next-free confirmed 2026-07-18) plus a `platform-traps.md` entry. ## MECHANISM UPDATE 2026-07-18 -- `-refresh-only` DOES NOT FIX EITHER ITEM Operator selected option (b) (reconcile state first). Investigating the mechanism BEFORE running anything shows **`tofu apply -refresh-only` cannot clear either item.** Both were verified empirically against live state: **1. `autostart` is ABSENT FROM STATE on both domains** -- not stale, absent: $ tofu -chdir=opentofu state show module.voffice1.libvirt_domain.vm | grep -c autostart 0 $ tofu -chdir=opentofu state show module.office1_opnsense.libvirt_domain.vm | grep -c autostart 0 **VERIFIED AGAINST PRIMARY SOURCE** (provider v0.9.8, `internal/provider/domain_resource.go`, fetched and read directly -- not a doc summary, per `references/opentofu-provider-docs.md`'s core lesson). Two facts, both confirmed: **(i) Refresh CANNOT populate `autostart`** -- Read is self-gating (line 938): // Read autostart - always during import, conditionally otherwise if isImport || (!state.Autostart.IsNull() && !state.Autostart.IsUnknown()) { autostart, err := r.client.Libvirt().DomainGetAutostart(domain) State's `Autostart` is NULL, and this is not an import, so the branch is skipped and `autostart` is never read back. The source and the empirical `grep -c` agree exactly. **`-refresh-only` is a guaranteed no-op for this item.** Note `isImport` DOES read it -- `tofu import` is therefore a candidate bounce-free reconcile path. **(ii) An autostart-ONLY change DOES bounce the guest** -- confirmed directly, NOT by the trap-1e analogy (an earlier draft cited trap 1e here; 1e was measured on a *memory* change and does not transfer, so it was the wrong warrant even though the conclusion holds). `Update` (lines 1087-1113): domainConfigUnchanged := reflect.DeepEqual(planData.SanitizedModel, statePlanData.SanitizedModel) runningChanged := !plan.Running.Equal(state.Running) autostartUnchanged := plan.Autostart.Equal(state.Autostart) if domainConfigUnchanged && runningChanged && autostartUnchanged { ... reconcileDomainRunning(...); return // <-- the ONLY no-bounce path } if _, err := r.stopDomainIfRunning(existingDomain, updateOptions); err != nil { ... DomainUndefine / DomainUndefineFlags -> DomainDefineXML -> DomainSetAutostart For our plan: `domainConfigUnchanged` = TRUE, `runningChanged` = **FALSE** (running stays true), `autostartUnchanged` = **FALSE**. The conjunction is FALSE, so the early return is NOT taken and control falls through to **`stopDomainIfRunning`** -> undefine -> redefine. The early-return path is unreachable for an autostart-only diff, because it *requires* `runningChanged` to be true. **The guest is stopped. The bounce is confirmed.** Corollary that makes state reconciliation viable: `Update` only runs when there IS a diff. If state records `autostart = true`, the plan is empty, `Update` is never invoked, and nothing bounces. **2. The seed churn is STRUCTURAL, not drift.** The module chains two resources (`opentofu/modules/cloudinit-vm/main.tf`): resource "libvirt_cloudinit_disk" "seed" { ... } # generates ISO in /tmp resource "libvirt_volume" "seed" { create = { content = { url = libvirt_cloudinit_disk.seed.path } } # copies into pool } `libvirt_cloudinit_disk`'s remote object IS the `/tmp` file. The reboot deleted it, so it plans as re-created -- and re-creation mints a **NEW random `/tmp` filename**, so `libvirt_volume.seed`'s `create.content.url` changes, which **forces replacement**. State still records the vanished path: create = { content = { url = "/tmp/terraform-provider-libvirt-cloudinit/ cloudinit-775234004c2669d9.iso" } } A refresh re-confirms the file is gone; it cannot stop the url from changing, because the url is config-derived and regenerated every plan. **This recurs on every plan after every host reboot, for every cloudinit-backed VM.** **BLAST RADIUS -- CORRECTED 2026-07-18 (MEASURED).** An earlier revision of this finding asserted "all 9 DC0 node VMs and each containment VM" inherit this. **That was WRONG and is retracted.** Measured: $ grep -rln 'libvirt_cloudinit_disk' opentofu/modules/ opentofu/*.tf opentofu/vr1-dc0-substrate/ opentofu/modules/cloudinit-vm/main.tf <-- the ONLY module $ grep -rn 'source.*cloudinit-vm' opentofu/*.tf opentofu/main.tf:181 (module "voffice1") opentofu/main.tf:361 (module "vvr1_dc0") - `modules/node-vm` (the 9 DC0 nodes) is deliberately **blank-disk + PXE-boot, no cloud-init** -- UNAFFECTED. - `modules/opnsense-edge` has no `libvirt_cloudinit_disk` (REST config, D-113(a2)) -- UNAFFECTED. Real exposure: **`voffice1` + `vvr1-dc0` today, plus `vvr1-dc1` later = 3 VMs.** Still worth a durable fix -- it recurs at every reboot for the two most load-bearing VMs in the region -- but it is NOT the fleet-wide defect the earlier text claimed. **Note on redeploy:** on a CLEAN deploy the cloudinit disk and the volume are created in the SAME apply, so the url matches state and there is no diff. This defect only surfaces on a plan taken AFTER a host reboot. It therefore does not break a post-teardown redeploy -- but it does affect ongoing operations, which is why it still needs fixing. ### What option (b) ACTUALLY requires (both avoid any guest bounce) - **Seed volume:** a CONFIG change -- `lifecycle { ignore_changes = [create] }` on `libvirt_volume.seed` in `modules/cloudinit-vm`, so the ephemeral staging path stops forcing replacement of a correct, in-use volume. In-repo and environment-independent. This is a stronger candidate for the D-130 durable fix than the `$TMPDIR` idea in (d) below, which depends on an env var being set identically by every future operator. Needs the module's test harness + changelog per the change-delivery loop. - **`autostart`:** record the already-true value into state WITHOUT invoking `Update`. Two candidates, both bounce-free in principle, both gated: - `tofu import` of the two domains -- the source shows import reads `autostart` back (line 938 `isImport`). Re-importing over an existing state slot is delicate; needs its own verification. - state surgery (`tofu state pull` -> inject `autostart: true` -> `tofu state push`). Touches the state file only, never the guests. The state file has documented protection (gap #18) -- back it up first. - The do-nothing option is also legitimate: **both guests just proved they autostart-recover cleanly from a real host reboot**, so a scheduled apply in a maintenance window is a defensible alternative to hand-editing state. The bounce is ~30s of no routing/DHCP (trap 1e's measurement), not data loss. Neither is `-refresh-only`. All are gated and must be presented individually. ## Recommendation **(a)** for the immediate DC0 apply, **(b)** as the deliberate follow-up to clear the drift **-- but executed via the two mechanisms in the MECHANISM UPDATE above, not via `-refresh-only`**, **(d)** as standing hardening regardless. Do not run the unscoped apply. ## Also required before ANY apply Re-run the readiness doc's **C1 pre-apply checks THIS session** -- specifically re-confirm the 6 `vr1-dc0` planes are still empty (0 leases / 0 attached). That evidence is from a prior session, and this finding is itself proof that committed pre-apply evidence goes stale across a reboot. Do not let the plane-emptiness check inherit that staleness. ## Reproduce / revert Read-only; nothing was mutated. Reproduce with: tofu -chdir=opentofu plan -lock=false -input=false No revert required.