# FINDING 2026-07-18 -- outer `tofu plan` would BOUNCE the live Office1 headend + edge

**Class:** deploy-blocking drift, discovered during pre-apply re-verification.
**Status:** OPEN -- operator ruling required. **Nothing applied.**
**Blocks:** the Stage-3 Phase-2 DC0 outer `tofu apply` **as currently scoped** (step A
of `docs/dc0-deploy-readiness.md`). See "What is NOT blocked" -- DC0 itself has a clean
path forward.

## What was measured

Post-reboot re-verification of the outer root (read-only
`tofu -chdir=opentofu plan -lock=false -input=false`):

    Plan: 7 to add, 2 to change, 7 to destroy.

The committed readiness evidence (`docs/dc0-deploy-readiness.md`) is
**5 add / 2 in-place / 6 destroy**. The delta is NOT in DC0 scope -- it is entirely
`voffice1`, the live Office1 headend. Decomposition (accounts for the delta exactly):

- adds 5 -> 7 = the intended 5 (`vvr1-dc0` domain/disk/seed-volume/cloudinit-seed +
  `vr1_dc0_uplink`) **+1** `voffice1` cloudinit_disk re-create **+1** `voffice1`
  seed-volume replace-half.
- destroys 6 -> 7 = the intended 6 empty `vr1-dc0` planes **+1** `voffice1` seed-volume
  destroy-half.
- changes stay 2 = `voffice1` and `office1_opnsense` domains, both `+ autostart = true`.

> **The OpenTofu 1.12.3 -> 1.12.4 bump is COINCIDENTAL, not causal.** The binary is now
> v1.12.4 (README.md:41 and all committed evidence say v1.12.3). Log it as version drift;
> it is not why the plan changed. The cause is `/tmp`, below.

## THE DOMINANT RISK: the 2 "in-place" changes bounce both guests

**This is the most serious item and an earlier draft of this finding got it wrong by
calling it harmless.** `references/platform-traps.md` 1e is unambiguous and MEASURED --
on this very edge:

> A plan line reading `libvirt_domain.vm will be updated in-place` **bounced a live
> guest** (measured 2026-07-13: uptime 8m36s -> 6s; ~30s with no routing and no DHCP on
> the Office1 edge). "In-place" is a statement about the Terraform RESOURCE, not about
> the domain.
> ... **Rule: treat ANY `tofu apply` whose plan touches a `libvirt_domain` as an OUTAGE
> of that guest.** ... `0 to change` on the domains is the only assurance that a running
> guest is not about to bounce.

Upstream mechanism: domain update **undefines and re-defines** the domain (v0.9.4 notes).
Pinned provider here is **dmacvicar/libvirt 0.9.8** (`opentofu/.terraform.lock.hcl`).

So the current plan would bounce **`voffice1` AND `office1-opnsense`** -- the headend
(NetBox apex 10.10.1.10 + tailscale + the D-128 Plane-2 execution origin + the D-126
ProxyJump path) and the site edge (routing, NAT, Kea DHCP). That is an unplanned Office1
site-down during what is supposed to be a DC0-scoped apply.

Bitter irony worth noting: these two in-place writes exist only because `virsh autostart`
was set **out-of-band** and the provider does not refresh `autostart` into state. The
desired value is **already live on both domains** -- so the plan proposes to bounce two
production guests to write a value that is already correct.

## The seed-volume replacement (root cause; danger UNVERIFIED)

    # module.voffice1.libvirt_cloudinit_disk.seed has been deleted
    # module.voffice1.libvirt_volume.seed must be replaced
      ~ create = { # forces replacement
          ~ content = { ~ url = "/tmp/terraform-provider-libvirt-cloudinit/
                                 cloudinit-775234004c2669d9.iso" -> (known after apply) }
        }

**Root cause (MEASURED):** the provider stages generated cloudinit ISOs under
`/tmp/terraform-provider-libvirt-cloudinit/` and records that staging path as the
volume's `create.content.url`. **The vcloud host rebooted 2026-07-17 23:38**, clearing
`/tmp`:

    $ ls /tmp/terraform-provider-libvirt-cloudinit/
    ls: cannot access ...: No such file or directory

The real artifact is intact and in use:

    $ ls -la /var/lib/libvirt/vr1/office1/voffice1-cloudinit.iso
    -rw------- 1 libvirt-qemu kvm 45056 Jul 13 19:45 ...
    $ virsh domblklist voffice1
     vda   voffice1-disk.qcow2
     sda   voffice1-cloudinit.iso      <-- ATTACHED to the RUNNING domain

This is **provider-state drift, not a real configuration change** -- nothing about
`voffice1`'s desired config changed.

**UNVERIFIED -- do not treat as established:** what provider 0.9.8 actually does when it
replaces a volume attached to a running domain it is NOT recreating. The candidates --
error out ("volume in use"), delete/recreate the file under the guest's open fd (guest
unaffected until next boot), or force a domain bounce -- have materially different
consequences. Trap 1e is about `libvirt_domain` in-place updates and does **not**
transfer to volume replacement; citing it here would be misapplied. This question is
answerable by observation on a throwaway domain, and should be answered before any path
that lets this replacement run. Note the payload is largely inert regardless: cloud-init
consumes the seed at first boot and `voffice1` was provisioned 2026-07-13.

## What is NOT blocked

**DC0 does not have to wait for the provider question.** The intended DC0 work --
create `vvr1-dc0` + `vr1_dc0_uplink`, destroy the 6 empty `vr1-dc0` planes -- touches
**no running guest**. Scoping the apply away from `voffice1` and `office1_opnsense`
sidesteps both the bounce and the volume replacement entirely. The unresolved provider
behavior only gates whether the `voffice1` churn is EVER allowed to run, not DC0.

## Options (NOT ruled -- operator decides)

- **(a) Scope the apply.** `-target` the `vvr1_dc0` / `vr1_dc0_uplink` /
  `vr1_dc0_planes` addresses so the apply cannot touch either running domain. DC0 moves
  today with zero Office1 risk. Costs: `-target` is a break-glass tool, and the drift
  stays outstanding to be dealt with deliberately later. **Recommended for the immediate
  apply.**
- **(b) Reconcile state first, then a clean full apply.** Bring the recorded seed source
  and the `autostart` attribute back into agreement with reality (re-stage / state
  surgery / `tofu apply -refresh-only`) so the plan returns to `0 to change` on the
  domains -- which trap 1e names as the ONLY assurance a guest will not bounce. Slower;
  restores "the plan matches committed evidence" as a trustworthy gate. **Recommended as
  the follow-up**, and it may resolve both items without any outage, since both desired
  values are already live.
- **(c) Accept the churn.** Let the apply bounce both guests and replace the seed.
  **Not recommended** -- it is a deliberate Office1 outage to write already-correct
  values.
- **(d) Durable fix (Roosevelt-transferable; orthogonal -- do regardless).** Prevent a
  host reboot from ever again turning a live VM's seed into a forced replacement. This
  will otherwise recur for EVERY cloudinit-backed VM, including all 9 DC0 node VMs and
  each DC containment VM, so it scales badly. **Mechanism -- partially verified:** there
  is **NO provider-level knob** for the staging path (VERIFIED via
  `tofu providers schema -json`: the provider block exposes `uri` ONLY, and
  `libvirt_cloudinit_disk` exposes only `id/meta_data/name/network_config/path/size/
  user_data`). The staging path therefore almost certainly comes from Go's
  `os.TempDir()`, i.e. **`$TMPDIR` for the tofu process** -- **UNVERIFIED by
  observation**; confirm that setting `TMPDIR` to a persistent path actually relocates
  it before writing this up. Warrants **D-130** (next-free confirmed 2026-07-18) plus a
  `platform-traps.md` entry.

## MECHANISM UPDATE 2026-07-18 -- `-refresh-only` DOES NOT FIX EITHER ITEM

Operator selected option (b) (reconcile state first). Investigating the mechanism BEFORE
running anything shows **`tofu apply -refresh-only` cannot clear either item.** Both were
verified empirically against live state:

**1. `autostart` is ABSENT FROM STATE on both domains** -- not stale, absent:

    $ tofu -chdir=opentofu state show module.voffice1.libvirt_domain.vm | grep -c autostart
    0
    $ tofu -chdir=opentofu state show module.office1_opnsense.libvirt_domain.vm | grep -c autostart
    0

**VERIFIED AGAINST PRIMARY SOURCE** (provider v0.9.8,
`internal/provider/domain_resource.go`, fetched and read directly -- not a doc summary,
per `references/opentofu-provider-docs.md`'s core lesson). Two facts, both confirmed:

**(i) Refresh CANNOT populate `autostart`** -- Read is self-gating (line 938):

    // Read autostart - always during import, conditionally otherwise
    if isImport || (!state.Autostart.IsNull() && !state.Autostart.IsUnknown()) {
        autostart, err := r.client.Libvirt().DomainGetAutostart(domain)

State's `Autostart` is NULL, and this is not an import, so the branch is skipped and
`autostart` is never read back. The source and the empirical `grep -c` agree exactly.
**`-refresh-only` is a guaranteed no-op for this item.** Note `isImport` DOES read it --
`tofu import` is therefore a candidate bounce-free reconcile path.

**(ii) An autostart-ONLY change DOES bounce the guest** -- confirmed directly, NOT by
the trap-1e analogy (an earlier draft cited trap 1e here; 1e was measured on a *memory*
change and does not transfer, so it was the wrong warrant even though the conclusion
holds). `Update` (lines 1087-1113):

    domainConfigUnchanged := reflect.DeepEqual(planData.SanitizedModel, statePlanData.SanitizedModel)
    runningChanged        := !plan.Running.Equal(state.Running)
    autostartUnchanged    := plan.Autostart.Equal(state.Autostart)
    if domainConfigUnchanged && runningChanged && autostartUnchanged {
        ... reconcileDomainRunning(...); return          // <-- the ONLY no-bounce path
    }
    if _, err := r.stopDomainIfRunning(existingDomain, updateOptions); err != nil {
    ... DomainUndefine / DomainUndefineFlags -> DomainDefineXML -> DomainSetAutostart

For our plan: `domainConfigUnchanged` = TRUE, `runningChanged` = **FALSE** (running stays
true), `autostartUnchanged` = **FALSE**. The conjunction is FALSE, so the early return is
NOT taken and control falls through to **`stopDomainIfRunning`** -> undefine -> redefine.
The early-return path is unreachable for an autostart-only diff, because it *requires*
`runningChanged` to be true. **The guest is stopped. The bounce is confirmed.**

Corollary that makes state reconciliation viable: `Update` only runs when there IS a
diff. If state records `autostart = true`, the plan is empty, `Update` is never invoked,
and nothing bounces.

**2. The seed churn is STRUCTURAL, not drift.** The module chains two resources
(`opentofu/modules/cloudinit-vm/main.tf`):

    resource "libvirt_cloudinit_disk" "seed" { ... }          # generates ISO in /tmp
    resource "libvirt_volume" "seed" {
      create = { content = { url = libvirt_cloudinit_disk.seed.path } }   # copies into pool
    }

`libvirt_cloudinit_disk`'s remote object IS the `/tmp` file. The reboot deleted it, so it
plans as re-created -- and re-creation mints a **NEW random `/tmp` filename**, so
`libvirt_volume.seed`'s `create.content.url` changes, which **forces replacement**. State
still records the vanished path:

    create = { content = { url = "/tmp/terraform-provider-libvirt-cloudinit/
                                 cloudinit-775234004c2669d9.iso" } }

A refresh re-confirms the file is gone; it cannot stop the url from changing, because the
url is config-derived and regenerated every plan. **This recurs on every plan after every
host reboot, for every cloudinit-backed VM.**

**BLAST RADIUS -- CORRECTED 2026-07-18 (MEASURED).** An earlier revision of this finding
asserted "all 9 DC0 node VMs and each containment VM" inherit this. **That was WRONG and
is retracted.** Measured:

    $ grep -rln 'libvirt_cloudinit_disk' opentofu/modules/ opentofu/*.tf opentofu/vr1-dc0-substrate/
    opentofu/modules/cloudinit-vm/main.tf        <-- the ONLY module
    $ grep -rn 'source.*cloudinit-vm' opentofu/*.tf
    opentofu/main.tf:181   (module "voffice1")
    opentofu/main.tf:361   (module "vvr1_dc0")

- `modules/node-vm` (the 9 DC0 nodes) is deliberately **blank-disk + PXE-boot, no
  cloud-init** -- UNAFFECTED.
- `modules/opnsense-edge` has no `libvirt_cloudinit_disk` (REST config, D-113(a2)) --
  UNAFFECTED.

Real exposure: **`voffice1` + `vvr1-dc0` today, plus `vvr1-dc1` later = 3 VMs.** Still
worth a durable fix -- it recurs at every reboot for the two most load-bearing VMs in the
region -- but it is NOT the fleet-wide defect the earlier text claimed.

**Note on redeploy:** on a CLEAN deploy the cloudinit disk and the volume are created in
the SAME apply, so the url matches state and there is no diff. This defect only surfaces
on a plan taken AFTER a host reboot. It therefore does not break a post-teardown
redeploy -- but it does affect ongoing operations, which is why it still needs fixing.

### What option (b) ACTUALLY requires (both avoid any guest bounce)

- **Seed volume:** a CONFIG change -- `lifecycle { ignore_changes = [create] }` on
  `libvirt_volume.seed` in `modules/cloudinit-vm`, so the ephemeral staging path stops
  forcing replacement of a correct, in-use volume. In-repo and environment-independent.
  This is a stronger candidate for the D-130 durable fix than the `$TMPDIR` idea in (d)
  below, which depends on an env var being set identically by every future operator.
  Needs the module's test harness + changelog per the change-delivery loop.
- **`autostart`:** record the already-true value into state WITHOUT invoking `Update`.
  Two candidates, both bounce-free in principle, both gated:
  - `tofu import` of the two domains -- the source shows import reads `autostart` back
    (line 938 `isImport`). Re-importing over an existing state slot is delicate; needs
    its own verification.
  - state surgery (`tofu state pull` -> inject `autostart: true` -> `tofu state push`).
    Touches the state file only, never the guests. The state file has documented
    protection (gap #18) -- back it up first.
  - The do-nothing option is also legitimate: **both guests just proved they
    autostart-recover cleanly from a real host reboot**, so a scheduled apply in a
    maintenance window is a defensible alternative to hand-editing state. The bounce is
    ~30s of no routing/DHCP (trap 1e's measurement), not data loss.

Neither is `-refresh-only`. All are gated and must be presented individually.

## Recommendation

**(a)** for the immediate DC0 apply, **(b)** as the deliberate follow-up to clear the
drift **-- but executed via the two mechanisms in the MECHANISM UPDATE above, not via
`-refresh-only`**, **(d)** as standing hardening regardless. Do not run the unscoped apply.

## Also required before ANY apply

Re-run the readiness doc's **C1 pre-apply checks THIS session** -- specifically
re-confirm the 6 `vr1-dc0` planes are still empty (0 leases / 0 attached). That evidence
is from a prior session, and this finding is itself proof that committed pre-apply
evidence goes stale across a reboot. Do not let the plane-emptiness check inherit that
staleness.

## Reproduce / revert

Read-only; nothing was mutated. Reproduce with:

    tofu -chdir=opentofu plan -lock=false -input=false

No revert required.
