diff --git a/docs/finding-20260718-voffice1-cloudinit-seed-replace.md b/docs/finding-20260718-voffice1-cloudinit-seed-replace.md new file mode 100644 index 0000000..c2e3939 --- /dev/null +++ b/docs/finding-20260718-voffice1-cloudinit-seed-replace.md @@ -0,0 +1,273 @@ +# FINDING 2026-07-18 -- outer `tofu plan` would BOUNCE the live Office1 headend + edge + +**Class:** deploy-blocking drift, discovered during pre-apply re-verification. +**Status:** OPEN -- operator ruling required. **Nothing applied.** +**Blocks:** the Stage-3 Phase-2 DC0 outer `tofu apply` **as currently scoped** (step A +of `docs/dc0-deploy-readiness.md`). See "What is NOT blocked" -- DC0 itself has a clean +path forward. + +## What was measured + +Post-reboot re-verification of the outer root (read-only +`tofu -chdir=opentofu plan -lock=false -input=false`): + + Plan: 7 to add, 2 to change, 7 to destroy. + +The committed readiness evidence (`docs/dc0-deploy-readiness.md`) is +**5 add / 2 in-place / 6 destroy**. The delta is NOT in DC0 scope -- it is entirely +`voffice1`, the live Office1 headend. Decomposition (accounts for the delta exactly): + +- adds 5 -> 7 = the intended 5 (`vvr1-dc0` domain/disk/seed-volume/cloudinit-seed + + `vr1_dc0_uplink`) **+1** `voffice1` cloudinit_disk re-create **+1** `voffice1` + seed-volume replace-half. +- destroys 6 -> 7 = the intended 6 empty `vr1-dc0` planes **+1** `voffice1` seed-volume + destroy-half. +- changes stay 2 = `voffice1` and `office1_opnsense` domains, both `+ autostart = true`. + +> **The OpenTofu 1.12.3 -> 1.12.4 bump is COINCIDENTAL, not causal.** The binary is now +> v1.12.4 (README.md:41 and all committed evidence say v1.12.3). Log it as version drift; +> it is not why the plan changed. The cause is `/tmp`, below. + +## THE DOMINANT RISK: the 2 "in-place" changes bounce both guests + +**This is the most serious item and an earlier draft of this finding got it wrong by +calling it harmless.** `references/platform-traps.md` 1e is unambiguous and MEASURED -- +on this very edge: + +> A plan line reading `libvirt_domain.vm will be updated in-place` **bounced a live +> guest** (measured 2026-07-13: uptime 8m36s -> 6s; ~30s with no routing and no DHCP on +> the Office1 edge). "In-place" is a statement about the Terraform RESOURCE, not about +> the domain. +> ... **Rule: treat ANY `tofu apply` whose plan touches a `libvirt_domain` as an OUTAGE +> of that guest.** ... `0 to change` on the domains is the only assurance that a running +> guest is not about to bounce. + +Upstream mechanism: domain update **undefines and re-defines** the domain (v0.9.4 notes). +Pinned provider here is **dmacvicar/libvirt 0.9.8** (`opentofu/.terraform.lock.hcl`). + +So the current plan would bounce **`voffice1` AND `office1-opnsense`** -- the headend +(NetBox apex 10.10.1.10 + tailscale + the D-128 Plane-2 execution origin + the D-126 +ProxyJump path) and the site edge (routing, NAT, Kea DHCP). That is an unplanned Office1 +site-down during what is supposed to be a DC0-scoped apply. + +Bitter irony worth noting: these two in-place writes exist only because `virsh autostart` +was set **out-of-band** and the provider does not refresh `autostart` into state. The +desired value is **already live on both domains** -- so the plan proposes to bounce two +production guests to write a value that is already correct. + +## The seed-volume replacement (root cause; danger UNVERIFIED) + + # module.voffice1.libvirt_cloudinit_disk.seed has been deleted + # module.voffice1.libvirt_volume.seed must be replaced + ~ create = { # forces replacement + ~ content = { ~ url = "/tmp/terraform-provider-libvirt-cloudinit/ + cloudinit-775234004c2669d9.iso" -> (known after apply) } + } + +**Root cause (MEASURED):** the provider stages generated cloudinit ISOs under +`/tmp/terraform-provider-libvirt-cloudinit/` and records that staging path as the +volume's `create.content.url`. **The vcloud host rebooted 2026-07-17 23:38**, clearing +`/tmp`: + + $ ls /tmp/terraform-provider-libvirt-cloudinit/ + ls: cannot access ...: No such file or directory + +The real artifact is intact and in use: + + $ ls -la /var/lib/libvirt/vr1/office1/voffice1-cloudinit.iso + -rw------- 1 libvirt-qemu kvm 45056 Jul 13 19:45 ... + $ virsh domblklist voffice1 + vda voffice1-disk.qcow2 + sda voffice1-cloudinit.iso <-- ATTACHED to the RUNNING domain + +This is **provider-state drift, not a real configuration change** -- nothing about +`voffice1`'s desired config changed. + +**UNVERIFIED -- do not treat as established:** what provider 0.9.8 actually does when it +replaces a volume attached to a running domain it is NOT recreating. The candidates -- +error out ("volume in use"), delete/recreate the file under the guest's open fd (guest +unaffected until next boot), or force a domain bounce -- have materially different +consequences. Trap 1e is about `libvirt_domain` in-place updates and does **not** +transfer to volume replacement; citing it here would be misapplied. This question is +answerable by observation on a throwaway domain, and should be answered before any path +that lets this replacement run. Note the payload is largely inert regardless: cloud-init +consumes the seed at first boot and `voffice1` was provisioned 2026-07-13. + +## What is NOT blocked + +**DC0 does not have to wait for the provider question.** The intended DC0 work -- +create `vvr1-dc0` + `vr1_dc0_uplink`, destroy the 6 empty `vr1-dc0` planes -- touches +**no running guest**. Scoping the apply away from `voffice1` and `office1_opnsense` +sidesteps both the bounce and the volume replacement entirely. The unresolved provider +behavior only gates whether the `voffice1` churn is EVER allowed to run, not DC0. + +## Options (NOT ruled -- operator decides) + +- **(a) Scope the apply.** `-target` the `vvr1_dc0` / `vr1_dc0_uplink` / + `vr1_dc0_planes` addresses so the apply cannot touch either running domain. DC0 moves + today with zero Office1 risk. Costs: `-target` is a break-glass tool, and the drift + stays outstanding to be dealt with deliberately later. **Recommended for the immediate + apply.** +- **(b) Reconcile state first, then a clean full apply.** Bring the recorded seed source + and the `autostart` attribute back into agreement with reality (re-stage / state + surgery / `tofu apply -refresh-only`) so the plan returns to `0 to change` on the + domains -- which trap 1e names as the ONLY assurance a guest will not bounce. Slower; + restores "the plan matches committed evidence" as a trustworthy gate. **Recommended as + the follow-up**, and it may resolve both items without any outage, since both desired + values are already live. +- **(c) Accept the churn.** Let the apply bounce both guests and replace the seed. + **Not recommended** -- it is a deliberate Office1 outage to write already-correct + values. +- **(d) Durable fix (Roosevelt-transferable; orthogonal -- do regardless).** Prevent a + host reboot from ever again turning a live VM's seed into a forced replacement. This + will otherwise recur for EVERY cloudinit-backed VM, including all 9 DC0 node VMs and + each DC containment VM, so it scales badly. **Mechanism -- partially verified:** there + is **NO provider-level knob** for the staging path (VERIFIED via + `tofu providers schema -json`: the provider block exposes `uri` ONLY, and + `libvirt_cloudinit_disk` exposes only `id/meta_data/name/network_config/path/size/ + user_data`). The staging path therefore almost certainly comes from Go's + `os.TempDir()`, i.e. **`$TMPDIR` for the tofu process** -- **UNVERIFIED by + observation**; confirm that setting `TMPDIR` to a persistent path actually relocates + it before writing this up. Warrants **D-130** (next-free confirmed 2026-07-18) plus a + `platform-traps.md` entry. + +## MECHANISM UPDATE 2026-07-18 -- `-refresh-only` DOES NOT FIX EITHER ITEM + +Operator selected option (b) (reconcile state first). Investigating the mechanism BEFORE +running anything shows **`tofu apply -refresh-only` cannot clear either item.** Both were +verified empirically against live state: + +**1. `autostart` is ABSENT FROM STATE on both domains** -- not stale, absent: + + $ tofu -chdir=opentofu state show module.voffice1.libvirt_domain.vm | grep -c autostart + 0 + $ tofu -chdir=opentofu state show module.office1_opnsense.libvirt_domain.vm | grep -c autostart + 0 + +**VERIFIED AGAINST PRIMARY SOURCE** (provider v0.9.8, +`internal/provider/domain_resource.go`, fetched and read directly -- not a doc summary, +per `references/opentofu-provider-docs.md`'s core lesson). Two facts, both confirmed: + +**(i) Refresh CANNOT populate `autostart`** -- Read is self-gating (line 938): + + // Read autostart - always during import, conditionally otherwise + if isImport || (!state.Autostart.IsNull() && !state.Autostart.IsUnknown()) { + autostart, err := r.client.Libvirt().DomainGetAutostart(domain) + +State's `Autostart` is NULL, and this is not an import, so the branch is skipped and +`autostart` is never read back. The source and the empirical `grep -c` agree exactly. +**`-refresh-only` is a guaranteed no-op for this item.** Note `isImport` DOES read it -- +`tofu import` is therefore a candidate bounce-free reconcile path. + +**(ii) An autostart-ONLY change DOES bounce the guest** -- confirmed directly, NOT by +the trap-1e analogy (an earlier draft cited trap 1e here; 1e was measured on a *memory* +change and does not transfer, so it was the wrong warrant even though the conclusion +holds). `Update` (lines 1087-1113): + + domainConfigUnchanged := reflect.DeepEqual(planData.SanitizedModel, statePlanData.SanitizedModel) + runningChanged := !plan.Running.Equal(state.Running) + autostartUnchanged := plan.Autostart.Equal(state.Autostart) + if domainConfigUnchanged && runningChanged && autostartUnchanged { + ... reconcileDomainRunning(...); return // <-- the ONLY no-bounce path + } + if _, err := r.stopDomainIfRunning(existingDomain, updateOptions); err != nil { + ... DomainUndefine / DomainUndefineFlags -> DomainDefineXML -> DomainSetAutostart + +For our plan: `domainConfigUnchanged` = TRUE, `runningChanged` = **FALSE** (running stays +true), `autostartUnchanged` = **FALSE**. The conjunction is FALSE, so the early return is +NOT taken and control falls through to **`stopDomainIfRunning`** -> undefine -> redefine. +The early-return path is unreachable for an autostart-only diff, because it *requires* +`runningChanged` to be true. **The guest is stopped. The bounce is confirmed.** + +Corollary that makes state reconciliation viable: `Update` only runs when there IS a +diff. If state records `autostart = true`, the plan is empty, `Update` is never invoked, +and nothing bounces. + +**2. The seed churn is STRUCTURAL, not drift.** The module chains two resources +(`opentofu/modules/cloudinit-vm/main.tf`): + + resource "libvirt_cloudinit_disk" "seed" { ... } # generates ISO in /tmp + resource "libvirt_volume" "seed" { + create = { content = { url = libvirt_cloudinit_disk.seed.path } } # copies into pool + } + +`libvirt_cloudinit_disk`'s remote object IS the `/tmp` file. The reboot deleted it, so it +plans as re-created -- and re-creation mints a **NEW random `/tmp` filename**, so +`libvirt_volume.seed`'s `create.content.url` changes, which **forces replacement**. State +still records the vanished path: + + create = { content = { url = "/tmp/terraform-provider-libvirt-cloudinit/ + cloudinit-775234004c2669d9.iso" } } + +A refresh re-confirms the file is gone; it cannot stop the url from changing, because the +url is config-derived and regenerated every plan. **This recurs on every plan after every +host reboot, for every cloudinit-backed VM.** + +**BLAST RADIUS -- CORRECTED 2026-07-18 (MEASURED).** An earlier revision of this finding +asserted "all 9 DC0 node VMs and each containment VM" inherit this. **That was WRONG and +is retracted.** Measured: + + $ grep -rln 'libvirt_cloudinit_disk' opentofu/modules/ opentofu/*.tf opentofu/vr1-dc0-substrate/ + opentofu/modules/cloudinit-vm/main.tf <-- the ONLY module + $ grep -rn 'source.*cloudinit-vm' opentofu/*.tf + opentofu/main.tf:181 (module "voffice1") + opentofu/main.tf:361 (module "vvr1_dc0") + +- `modules/node-vm` (the 9 DC0 nodes) is deliberately **blank-disk + PXE-boot, no + cloud-init** -- UNAFFECTED. +- `modules/opnsense-edge` has no `libvirt_cloudinit_disk` (REST config, D-113(a2)) -- + UNAFFECTED. + +Real exposure: **`voffice1` + `vvr1-dc0` today, plus `vvr1-dc1` later = 3 VMs.** Still +worth a durable fix -- it recurs at every reboot for the two most load-bearing VMs in the +region -- but it is NOT the fleet-wide defect the earlier text claimed. + +**Note on redeploy:** on a CLEAN deploy the cloudinit disk and the volume are created in +the SAME apply, so the url matches state and there is no diff. This defect only surfaces +on a plan taken AFTER a host reboot. It therefore does not break a post-teardown +redeploy -- but it does affect ongoing operations, which is why it still needs fixing. + +### What option (b) ACTUALLY requires (both avoid any guest bounce) + +- **Seed volume:** a CONFIG change -- `lifecycle { ignore_changes = [create] }` on + `libvirt_volume.seed` in `modules/cloudinit-vm`, so the ephemeral staging path stops + forcing replacement of a correct, in-use volume. In-repo and environment-independent. + This is a stronger candidate for the D-130 durable fix than the `$TMPDIR` idea in (d) + below, which depends on an env var being set identically by every future operator. + Needs the module's test harness + changelog per the change-delivery loop. +- **`autostart`:** record the already-true value into state WITHOUT invoking `Update`. + Two candidates, both bounce-free in principle, both gated: + - `tofu import` of the two domains -- the source shows import reads `autostart` back + (line 938 `isImport`). Re-importing over an existing state slot is delicate; needs + its own verification. + - state surgery (`tofu state pull` -> inject `autostart: true` -> `tofu state push`). + Touches the state file only, never the guests. The state file has documented + protection (gap #18) -- back it up first. + - The do-nothing option is also legitimate: **both guests just proved they + autostart-recover cleanly from a real host reboot**, so a scheduled apply in a + maintenance window is a defensible alternative to hand-editing state. The bounce is + ~30s of no routing/DHCP (trap 1e's measurement), not data loss. + +Neither is `-refresh-only`. All are gated and must be presented individually. + +## Recommendation + +**(a)** for the immediate DC0 apply, **(b)** as the deliberate follow-up to clear the +drift **-- but executed via the two mechanisms in the MECHANISM UPDATE above, not via +`-refresh-only`**, **(d)** as standing hardening regardless. Do not run the unscoped apply. + +## Also required before ANY apply + +Re-run the readiness doc's **C1 pre-apply checks THIS session** -- specifically +re-confirm the 6 `vr1-dc0` planes are still empty (0 leases / 0 attached). That evidence +is from a prior session, and this finding is itself proof that committed pre-apply +evidence goes stale across a reboot. Do not let the plane-emptiness check inherit that +staleness. + +## Reproduce / revert + +Read-only; nothing was mutated. Reproduce with: + + tofu -chdir=opentofu plan -lock=false -input=false + +No revert required. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index bb8ded0..77a66dd 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -189,6 +189,78 @@ security pkgs) THEN reboot; (b) office1-opnsense itself is still on 14.3-RELEASE-p16 (guest OS not patched, only cold-booted). +**BOOKEND ITEM (a) CLOSED 2026-07-18 -- MEASURED, and the autostart chain PROVED ITSELF FOR REAL.** +The vcloud host was patched and rebooted between sessions: `uname -r` = **6.8.0-136-generic**, `apt list +--upgradable` EMPTY, no `/var/run/reboot-required`, and `last -x reboot` shows TWO boots on 2026-07-17 +(17:59 and 23:38; current uptime 5h39m from 23:38, ending the -134 run of 8d20h). Item (a) is DONE. Item +**(b) STAYS OPEN** -- a host reboot does not patch the edge guest; `office1-opnsense` is still on +14.3-RELEASE-p16. **The D-127 autostart chain is now PROVEN BY A REAL HOST REBOOT, not merely configured:** +after an unattended boot, `virsh list --all` shows `voffice1` + `office1-opnsense` both running with +autostart, and `ssh voffice1` / `office1-netbox` / `office1-tailscale` all answer live (all up ~5h38m, +matching the host boot). `site-baseleg.sh check office1` = **ok leg + ok route** (the D-126 oneshot re-added +the isolated-net leg through the reboot). NetBox apex HTTP **302** at 10.10.1.10:8000; edge LAN gw +10.10.0.1 ping **0% loss**. Zero manual `virsh start` / `lxc start` was needed. Office1 is fully self-recovered. + +**STAGE-3 WORKFLOW-DOC BLOCKER IS STALE -- reconciled 2026-07-18 (the ledger was right, the doc is wrong).** +`docs/dc-dc-deployment-workflow.md`'s Stage 3 section still reads "**Blocking to EXECUTE:** the HELD NetBox +addressing -- D-124's rack transit/30 + rack metal-admin IP". MEASURED against the live apex +(read-only API GET, token sourced in-shell and never printed): all three objects are PRESENT -- +`172.31.0.0/30` (role Transit, scope **VR1 DC0**), the `172.31.0.0/24` unscoped Transit container, and +`10.12.8.2/22`. This matches the 2026-07-17 apex-recovery entry above. The NetBox addressing is NOT a +blocker; the workflow doc's Stage-3 "Blocking to EXECUTE" line needs a DOCFIX (next-free 196). + +**!! DC0 OUTER APPLY IS BLOCKED ON NEW DRIFT -- do NOT apply. See +`docs/finding-20260718-voffice1-cloudinit-seed-replace.md`.** Post-reboot re-verification of the outer +root (read-only plan, **OpenTofu v1.12.4**) returns **7 add / 2 change / 7 destroy**, NOT the committed +**5 add / 2 in-place / 6 destroy** in `docs/dc0-deploy-readiness.md`. The entire delta is `voffice1`, the +live headend -- NOT DC0 scope: the libvirt provider staged its cloudinit ISO under +`/tmp/terraform-provider-libvirt-cloudinit/`, **the reboot cleared `/tmp`**, so `create.content.url` reads +changed and **forces replacement of `voffice1`'s seed volume -- which is ATTACHED to the RUNNING domain as +`sda`** (`virsh domblklist voffice1`; the real ISO is intact at +`/var/lib/libvirt/vr1/office1/voffice1-cloudinit.iso`). Replacing it risks bouncing the Office1 headend = +NetBox apex + tailscale + the D-128 Plane-2 execution origin + the D-126 ProxyJump path. This is provider- +state drift, not a config change (the volume-replace CONSEQUENCE on a running domain is **UNVERIFIED** -- +do not assert it bounces the guest; trap 1e is about domains, not volumes). +**THE DOMINANT RISK IS THE 2 "in-place" CHANGES, NOT THE VOLUME.** They did NOT collapse to 0 as the D-127 +changelog predicted -- both `voffice1` AND `office1-opnsense` plan as `+ autostart = true`, because `virsh +autostart` was set out-of-band and the provider does not refresh it into state. **The bounce is CONFIRMED +FROM PRIMARY SOURCE** (v0.9.8 `internal/provider/domain_resource.go`, fetched + read directly): `Update`'s +only no-bounce path (L1090) requires `domainConfigUnchanged && runningChanged && autostartUnchanged`, and an +autostart-only diff has `runningChanged`=FALSE, so it falls through to **`stopDomainIfRunning`** -> +undefine -> DefineXML -> SetAutostart. ALSO CONFIRMED (L938): Read only reads `autostart` back +`if isImport || state.Autostart` is non-null -- ours is NULL, so **`-refresh-only` can never populate it** +(matches the empirical `grep -c autostart` = 0 on both domains). NOTE: an earlier draft cited +`platform-traps.md` 1e as the warrant here; 1e was measured on a *memory* change and does NOT transfer to +an autostart-only diff -- the conclusion held but the reasoning was wrong, now replaced by the source above. +1e's operational rule still stands independently: `0 to change` on the domains is the ONLY assurance a guest +will not bounce. So the unscoped apply +would take Office1's headend AND edge down -- to write a value that is ALREADY LIVE on both. An earlier +draft of this finding called these changes "harmless"; that was WRONG and is corrected here and in the +finding doc. Pinned provider = dmacvicar/libvirt **0.9.8**. +Options (a)-(d) are in the finding. **DC0 IS NOT BLOCKED on the unresolved provider question:** the intended +DC0 work (create `vvr1-dc0` + uplink, destroy 6 empty planes) touches NO running guest, so **(a) `-target` +the DC0 addresses** sidesteps both risks and is recommended for the immediate apply, with **(b) reconcile +state first** (`-refresh-only` / re-stage) as the deliberate follow-up to get back to `0 to change`. +**(d) standing hardening:** keep the provider's cloudinit staging OFF `/tmp` so a reboot can never again +force-replace a live VM's seed. **BLAST RADIUS CORRECTED (MEASURED 2026-07-18):** an earlier draft of this +entry claimed this hits "all 9 DC0 node VMs and every DC containment VM" -- **WRONG, retracted.** +`libvirt_cloudinit_disk` appears in exactly ONE module (`modules/cloudinit-vm`), instantiated exactly TWICE +(`main.tf:181` voffice1, `main.tf:361` vvr1_dc0). The 9 nodes use `modules/node-vm` = blank-disk PXE, NO +cloud-init; the edges use REST config (D-113(a2)). Real exposure = **voffice1 + vvr1-dc0 (+vvr1-dc1) = 3**. +Also: on a CLEAN deploy the disk+volume are created in the same apply so there is NO diff -- this only +surfaces on a plan taken after a host reboot, so it does not break a post-teardown redeploy, only ops. +**Mechanism correction:** there is **NO provider knob** for this (VERIFIED via `tofu providers schema +-json` -- provider block exposes `uri` ONLY; `libvirt_cloudinit_disk` has no staging attr); the lever is +almost certainly **`$TMPDIR`** for the tofu process (Go `os.TempDir()`), UNVERIFIED by observation -- +confirm before writing it up. Warrants **D-130** (next-free confirmed 2026-07-18). +**Before ANY apply:** re-run the readiness doc's C1 pre-apply checks THIS session, incl. re-confirming the +6 `vr1-dc0` planes are still empty (0 leases / 0 attached) -- that evidence is from a prior session, and +this finding is itself proof that committed pre-apply evidence goes stale across a reboot. + +**Version drift to log:** OpenTofu on the jumphost is now **v1.12.4**; `README.md:41` and the readiness +evidence say **v1.12.3**. Presumed to have arrived with the host patch (not verified). All committed +plan/validate evidence predates it. + **DC0 DEPLOY READINESS -- PREP COMPLETE 2026-07-17 (deploy-ready). See `docs/dc0-deploy-readiness.md`.** Final gate re-checked clean (validate, tfvars, creds-audit vr1-dc0 CLEAN, gauntlet 65, lint 0-fail). Remaining is NOT prep: the bookend restart (vcloud + voffice1 + opnsense patch/reboot together -- voffice1