Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / finding-20260718-voffice1-cloudinit-seed-replace.md
@JANeumatrix JANeumatrix 1 day ago 14 KB Findings: post-reboot outer plan would BOUNCE Office1 headend + edge -- DC0 apply STOPPED
Raw Blame History

FINDING 2026-07-18 -- outer tofu plan would BOUNCE the live Office1 headend + edge

Class: deploy-blocking drift, discovered during pre-apply re-verification. Status: OPEN -- operator ruling required. Nothing applied. Blocks: the Stage-3 Phase-2 DC0 outer tofu apply as currently scoped (step A of docs/dc0-deploy-readiness.md). See "What is NOT blocked" -- DC0 itself has a clean path forward.

What was measured

Post-reboot re-verification of the outer root (read-only tofu -chdir=opentofu plan -lock=false -input=false):

Plan: 7 to add, 2 to change, 7 to destroy.

The committed readiness evidence (docs/dc0-deploy-readiness.md) is 5 add / 2 in-place / 6 destroy. The delta is NOT in DC0 scope -- it is entirely voffice1, the live Office1 headend. Decomposition (accounts for the delta exactly):

  • adds 5 -> 7 = the intended 5 (vvr1-dc0 domain/disk/seed-volume/cloudinit-seed + vr1_dc0_uplink) +1 voffice1 cloudinit_disk re-create +1 voffice1 seed-volume replace-half.
  • destroys 6 -> 7 = the intended 6 empty vr1-dc0 planes +1 voffice1 seed-volume destroy-half.
  • changes stay 2 = voffice1 and office1_opnsense domains, both + autostart = true.

The OpenTofu 1.12.3 -> 1.12.4 bump is COINCIDENTAL, not causal. The binary is now v1.12.4 (README.md:41 and all committed evidence say v1.12.3). Log it as version drift; it is not why the plan changed. The cause is /tmp, below.

THE DOMINANT RISK: the 2 "in-place" changes bounce both guests

This is the most serious item and an earlier draft of this finding got it wrong by calling it harmless. references/platform-traps.md 1e is unambiguous and MEASURED -- on this very edge:

A plan line reading libvirt_domain.vm will be updated in-place bounced a live guest (measured 2026-07-13: uptime 8m36s -> 6s; ~30s with no routing and no DHCP on the Office1 edge). "In-place" is a statement about the Terraform RESOURCE, not about the domain. ... Rule: treat ANY tofu apply whose plan touches a libvirt_domain as an OUTAGE of that guest. ... 0 to change on the domains is the only assurance that a running guest is not about to bounce.

Upstream mechanism: domain update undefines and re-defines the domain (v0.9.4 notes). Pinned provider here is dmacvicar/libvirt 0.9.8 (opentofu/.terraform.lock.hcl).

So the current plan would bounce voffice1 AND office1-opnsense -- the headend (NetBox apex 10.10.1.10 + tailscale + the D-128 Plane-2 execution origin + the D-126 ProxyJump path) and the site edge (routing, NAT, Kea DHCP). That is an unplanned Office1 site-down during what is supposed to be a DC0-scoped apply.

Bitter irony worth noting: these two in-place writes exist only because virsh autostart was set out-of-band and the provider does not refresh autostart into state. The desired value is already live on both domains -- so the plan proposes to bounce two production guests to write a value that is already correct.

The seed-volume replacement (root cause; danger UNVERIFIED)

# module.voffice1.libvirt_cloudinit_disk.seed has been deleted
# module.voffice1.libvirt_volume.seed must be replaced
  ~ create = { # forces replacement
      ~ content = { ~ url = "/tmp/terraform-provider-libvirt-cloudinit/
                             cloudinit-775234004c2669d9.iso" -> (known after apply) }
    }

Root cause (MEASURED): the provider stages generated cloudinit ISOs under /tmp/terraform-provider-libvirt-cloudinit/ and records that staging path as the volume's create.content.url. The vcloud host rebooted 2026-07-17 23:38, clearing /tmp:

$ ls /tmp/terraform-provider-libvirt-cloudinit/
ls: cannot access ...: No such file or directory

The real artifact is intact and in use:

$ ls -la /var/lib/libvirt/vr1/office1/voffice1-cloudinit.iso
-rw------- 1 libvirt-qemu kvm 45056 Jul 13 19:45 ...
$ virsh domblklist voffice1
 vda   voffice1-disk.qcow2
 sda   voffice1-cloudinit.iso      <-- ATTACHED to the RUNNING domain

This is provider-state drift, not a real configuration change -- nothing about voffice1's desired config changed.

UNVERIFIED -- do not treat as established: what provider 0.9.8 actually does when it replaces a volume attached to a running domain it is NOT recreating. The candidates -- error out ("volume in use"), delete/recreate the file under the guest's open fd (guest unaffected until next boot), or force a domain bounce -- have materially different consequences. Trap 1e is about libvirt_domain in-place updates and does not transfer to volume replacement; citing it here would be misapplied. This question is answerable by observation on a throwaway domain, and should be answered before any path that lets this replacement run. Note the payload is largely inert regardless: cloud-init consumes the seed at first boot and voffice1 was provisioned 2026-07-13.

What is NOT blocked

DC0 does not have to wait for the provider question. The intended DC0 work -- create vvr1-dc0 + vr1_dc0_uplink, destroy the 6 empty vr1-dc0 planes -- touches no running guest. Scoping the apply away from voffice1 and office1_opnsense sidesteps both the bounce and the volume replacement entirely. The unresolved provider behavior only gates whether the voffice1 churn is EVER allowed to run, not DC0.

Options (NOT ruled -- operator decides)

  • (a) Scope the apply. -target the vvr1_dc0 / vr1_dc0_uplink / vr1_dc0_planes addresses so the apply cannot touch either running domain. DC0 moves today with zero Office1 risk. Costs: -target is a break-glass tool, and the drift stays outstanding to be dealt with deliberately later. Recommended for the immediate apply.
  • (b) Reconcile state first, then a clean full apply. Bring the recorded seed source and the autostart attribute back into agreement with reality (re-stage / state surgery / tofu apply -refresh-only) so the plan returns to 0 to change on the domains -- which trap 1e names as the ONLY assurance a guest will not bounce. Slower; restores "the plan matches committed evidence" as a trustworthy gate. Recommended as the follow-up, and it may resolve both items without any outage, since both desired values are already live.
  • (c) Accept the churn. Let the apply bounce both guests and replace the seed. Not recommended -- it is a deliberate Office1 outage to write already-correct values.
  • (d) Durable fix (Roosevelt-transferable; orthogonal -- do regardless). Prevent a host reboot from ever again turning a live VM's seed into a forced replacement. This will otherwise recur for EVERY cloudinit-backed VM, including all 9 DC0 node VMs and each DC containment VM, so it scales badly. Mechanism -- partially verified: there is NO provider-level knob for the staging path (VERIFIED via tofu providers schema -json: the provider block exposes uri ONLY, and libvirt_cloudinit_disk exposes only id/meta_data/name/network_config/path/size/ user_data). The staging path therefore almost certainly comes from Go's os.TempDir(), i.e. $TMPDIR for the tofu process -- UNVERIFIED by observation; confirm that setting TMPDIR to a persistent path actually relocates it before writing this up. Warrants D-130 (next-free confirmed 2026-07-18) plus a platform-traps.md entry.

MECHANISM UPDATE 2026-07-18 -- -refresh-only DOES NOT FIX EITHER ITEM

Operator selected option (b) (reconcile state first). Investigating the mechanism BEFORE running anything shows tofu apply -refresh-only cannot clear either item. Both were verified empirically against live state:

1. autostart is ABSENT FROM STATE on both domains -- not stale, absent:

$ tofu -chdir=opentofu state show module.voffice1.libvirt_domain.vm | grep -c autostart
0
$ tofu -chdir=opentofu state show module.office1_opnsense.libvirt_domain.vm | grep -c autostart
0

VERIFIED AGAINST PRIMARY SOURCE (provider v0.9.8, internal/provider/domain_resource.go, fetched and read directly -- not a doc summary, per references/opentofu-provider-docs.md's core lesson). Two facts, both confirmed:

(i) Refresh CANNOT populate autostart -- Read is self-gating (line 938):

// Read autostart - always during import, conditionally otherwise
if isImport || (!state.Autostart.IsNull() && !state.Autostart.IsUnknown()) {
    autostart, err := r.client.Libvirt().DomainGetAutostart(domain)

State's Autostart is NULL, and this is not an import, so the branch is skipped and autostart is never read back. The source and the empirical grep -c agree exactly. -refresh-only is a guaranteed no-op for this item. Note isImport DOES read it -- tofu import is therefore a candidate bounce-free reconcile path.

(ii) An autostart-ONLY change DOES bounce the guest -- confirmed directly, NOT by the trap-1e analogy (an earlier draft cited trap 1e here; 1e was measured on a memory change and does not transfer, so it was the wrong warrant even though the conclusion holds). Update (lines 1087-1113):

domainConfigUnchanged := reflect.DeepEqual(planData.SanitizedModel, statePlanData.SanitizedModel)
runningChanged        := !plan.Running.Equal(state.Running)
autostartUnchanged    := plan.Autostart.Equal(state.Autostart)
if domainConfigUnchanged && runningChanged && autostartUnchanged {
    ... reconcileDomainRunning(...); return          // <-- the ONLY no-bounce path
}
if _, err := r.stopDomainIfRunning(existingDomain, updateOptions); err != nil {
... DomainUndefine / DomainUndefineFlags -> DomainDefineXML -> DomainSetAutostart

For our plan: domainConfigUnchanged = TRUE, runningChanged = FALSE (running stays true), autostartUnchanged = FALSE. The conjunction is FALSE, so the early return is NOT taken and control falls through to stopDomainIfRunning -> undefine -> redefine. The early-return path is unreachable for an autostart-only diff, because it requires runningChanged to be true. The guest is stopped. The bounce is confirmed.

Corollary that makes state reconciliation viable: Update only runs when there IS a diff. If state records autostart = true, the plan is empty, Update is never invoked, and nothing bounces.

2. The seed churn is STRUCTURAL, not drift. The module chains two resources (opentofu/modules/cloudinit-vm/main.tf):

resource "libvirt_cloudinit_disk" "seed" { ... }          # generates ISO in /tmp
resource "libvirt_volume" "seed" {
  create = { content = { url = libvirt_cloudinit_disk.seed.path } }   # copies into pool
}

libvirt_cloudinit_disk's remote object IS the /tmp file. The reboot deleted it, so it plans as re-created -- and re-creation mints a NEW random /tmp filename, so libvirt_volume.seed's create.content.url changes, which forces replacement. State still records the vanished path:

create = { content = { url = "/tmp/terraform-provider-libvirt-cloudinit/
                             cloudinit-775234004c2669d9.iso" } }

A refresh re-confirms the file is gone; it cannot stop the url from changing, because the url is config-derived and regenerated every plan. This recurs on every plan after every host reboot, for every cloudinit-backed VM.

BLAST RADIUS -- CORRECTED 2026-07-18 (MEASURED). An earlier revision of this finding asserted "all 9 DC0 node VMs and each containment VM" inherit this. That was WRONG and is retracted. Measured:

$ grep -rln 'libvirt_cloudinit_disk' opentofu/modules/ opentofu/*.tf opentofu/vr1-dc0-substrate/
opentofu/modules/cloudinit-vm/main.tf        <-- the ONLY module
$ grep -rn 'source.*cloudinit-vm' opentofu/*.tf
opentofu/main.tf:181   (module "voffice1")
opentofu/main.tf:361   (module "vvr1_dc0")
  • modules/node-vm (the 9 DC0 nodes) is deliberately blank-disk + PXE-boot, no cloud-init -- UNAFFECTED.
  • modules/opnsense-edge has no libvirt_cloudinit_disk (REST config, D-113(a2)) -- UNAFFECTED.

Real exposure: voffice1 + vvr1-dc0 today, plus vvr1-dc1 later = 3 VMs. Still worth a durable fix -- it recurs at every reboot for the two most load-bearing VMs in the region -- but it is NOT the fleet-wide defect the earlier text claimed.

Note on redeploy: on a CLEAN deploy the cloudinit disk and the volume are created in the SAME apply, so the url matches state and there is no diff. This defect only surfaces on a plan taken AFTER a host reboot. It therefore does not break a post-teardown redeploy -- but it does affect ongoing operations, which is why it still needs fixing.

What option (b) ACTUALLY requires (both avoid any guest bounce)

  • Seed volume: a CONFIG change -- lifecycle { ignore_changes = [create] } on libvirt_volume.seed in modules/cloudinit-vm, so the ephemeral staging path stops forcing replacement of a correct, in-use volume. In-repo and environment-independent. This is a stronger candidate for the D-130 durable fix than the $TMPDIR idea in (d) below, which depends on an env var being set identically by every future operator. Needs the module's test harness + changelog per the change-delivery loop.
  • autostart: record the already-true value into state WITHOUT invoking Update. Two candidates, both bounce-free in principle, both gated:
    • tofu import of the two domains -- the source shows import reads autostart back (line 938 isImport). Re-importing over an existing state slot is delicate; needs its own verification.
    • state surgery (tofu state pull -> inject autostart: true -> tofu state push). Touches the state file only, never the guests. The state file has documented protection (gap #18) -- back it up first.
    • The do-nothing option is also legitimate: both guests just proved they autostart-recover cleanly from a real host reboot, so a scheduled apply in a maintenance window is a defensible alternative to hand-editing state. The bounce is ~30s of no routing/DHCP (trap 1e's measurement), not data loss.

Neither is -refresh-only. All are gated and must be presented individually.

Recommendation

(a) for the immediate DC0 apply, (b) as the deliberate follow-up to clear the drift -- but executed via the two mechanisms in the MECHANISM UPDATE above, not via -refresh-only, (d) as standing hardening regardless. Do not run the unscoped apply.

Also required before ANY apply

Re-run the readiness doc's C1 pre-apply checks THIS session -- specifically re-confirm the 6 vr1-dc0 planes are still empty (0 leases / 0 attached). That evidence is from a prior session, and this finding is itself proof that committed pre-apply evidence goes stale across a reboot. Do not let the plane-emptiness check inherit that staleness.

Reproduce / revert

Read-only; nothing was mutated. Reproduce with:

tofu -chdir=opentofu plan -lock=false -input=false

No revert required.