# VR1 DC-DC teardown / rollback runbook Scope: tearing down or rolling back the infrastructure `opentofu/` provisions for the VR1 (DC-DC) buildout -- DC1/DC2/Office1 plane and local networks, storage pools, node-VM domains, OPNsense edge VMs, the D-100 mesh-link triangle, and MAAS `vm_host` registrations. Written 2026-07-10 (DOCFIX-175 gap-register item #19) because nothing else in this repo covers this layer: `runbooks/phase-00-teardown-release.sh` / `phase-00-teardown-destroy.sh` (D-061) tear down the VR0 **juju/MAAS-machine** layer, a different pairing of tools and a different failure mode entirely -- see "Relationship to D-061" below for why that mechanism does not just transfer over. **UNVALIDATED, same caveat as everywhere else in `opentofu/`:** no `tofu` binary has been available in any session that authored this tree. Every command below is standard, well-documented OpenTofu/Terraform CLI syntax (verified against HashiCorp's own current docs where cited), not invented -- but run `bash scripts/opentofu-validate.sh` on a machine with the real binary and read the result before trusting any of it against a live state file. --- ## When to use this runbook - **Mid-buildout failure recovery.** A stage's `tofu apply` failed partway through, or created something wrong, and you need to know whether to fix- forward or tear down and retry. See "Rollback decision tree" below -- READ THAT FIRST, since fix-forward is usually the right answer, not a destroy. - **Deliberate re-run from clean.** You want to abandon a DC's (or Office1's, or the whole VR1 layer's) current state and rebuild from Step 1 of its stage runbook. - **NOT for**: tearing down VR0's juju/MAAS-machine layer (use D-061's scripts), or removing a single juju application/relation from an already- deployed OpenStack bundle (that is normal juju operations, not this runbook). --- ## Relationship to D-061 (read before mixing the two layers) D-061's core lesson: destroying/releasing a resource at the WRONG layer, in the wrong order, orphans state at the OTHER layer -- `juju destroy-model` released MAAS-pod-composed machines and MAAS auto-decomposed them, losing the carve; the fix was `remove-machine --keep-instance` FIRST (detach from juju without releasing), verify survival in MAAS, THEN act on the model. The same PRINCIPLE applies here, but the mechanism is different because the layers being coordinated are different: - D-061 coordinates **juju's machine view** against **MAAS's machine view** of hosts MAAS itself pod-composed. - This runbook coordinates **OpenTofu's resource view** (libvirt domains/ networks/pools it created directly) against **MAAS's `vm_host` and (eventually) machine view** of a libvirt host OpenTofu registered via `modules/maas-vm-host` (the `maas_vm_host` resource, canonical/maas provider -- NOT `maas_vm_host_machine`, which composes VMs itself; this repo deliberately uses the plain `maas_vm_host` variant so `modules/ node-vm`'s own domains are the source of truth, not MAAS pod-composition -- see `opentofu/README.md`). Because this repo doesn't use MAAS pod-composition for its OpenTofu-created VMs, the SPECIFIC `--keep-instance` decompose-on-release failure mode D-061 diagnosed does not directly apply here. The GENERAL principle still does: **clean up MAAS's record of a host before destroying the libvirt resource underneath it**, not after -- destroying the libvirt domain/host first would leave MAAS pointing at a `power_address` that no longer answers, an orphaned-but-not-obviously-broken record rather than a clean removal. **Residual open item, flagged not solved here:** this repo has not yet reached Stage 4 (MAAS enlist/commission/deploy) for any VR1 DC, so no VR1 libvirt host has ever had real MAAS-enrolled machines under it. The exact MAAS-side commands to cleanly release machines enrolled under a `vm_host` this runbook is about to tear down (analogous to D-061's `remove-machine --keep-instance` discovery, but for the `vm_host`/enlist-commission-deploy flow instead of pod-composition) have not been worked out or tested. Step 2 below states the PRINCIPLE (MAAS-side first) and points at MAAS's own current docs for the exact commands -- do not invent flags for a scenario this repo has never actually exercised. --- ## Two paths, by intent (mirrors D-061's release-vs-destroy split) - **Path A -- scoped teardown.** Tear down ONE site's resources (a single DC, or Office1), leaving the other sites' infrastructure untouched. Use `-target` to scope both plan and apply. HashiCorp's own docs flag `-target` as an exceptional-circumstances tool, not for routine use, because targeted applies can leave configuration and state silently diverging: *"This targeting capability is provided for exceptional circumstances, such as recovering from mistakes or working around Terraform limitations. It is not recommended to use `-target` for routine operations, since this can lead to undetected configuration drift"* (developer.hashicorp.com/ terraform/cli/commands/plan, fetched 2026-07-10). A scoped teardown IS exactly the kind of exceptional circumstance this describes -- acceptable here, but ALWAYS follow it with a full (untargeted) `tofu plan` afterward (Step 6) to confirm the rest of the configuration didn't drift. - **Path B -- full VR1 teardown.** Abandon everything OpenTofu manages for VR1. Plain `tofu destroy` (no `-target`), letting OpenTofu compute the full safe order from the resource graph itself -- no manual sequencing needed, unlike Path A. Both paths share Steps 1 (state backup) and 6 (final verify). Path A is Steps 2-5; Path B replaces Steps 3-5 with a single plain destroy (see "Path B" section after Step 5). --- ## Step 1 -- Back up the state file before ANY destroy [do this every time] A `destroy` mutates state exactly like an `apply` does -- if it's interrupted partway (network drop, `Ctrl-C`, host reboot), your only way to know what's actually still live is the state file as it stood before you started. ```bash cd opentofu cp -a terraform.tfstate "terraform.tfstate.pre-teardown-$(date +%Y%m%d-%H%M%S)" ``` Store the copy out-of-band per the operator's existing `~/vault-init/`-class secret-handling process (it carries the same plaintext-credential risk as the live file -- see `opentofu/README.md` "State file handling", DOCFIX-175) -- do not just leave it sitting next to the live file as your only copy. --- ## Step 2 -- MAAS-side cleanup FIRST, if any machines were ever enrolled under the site being torn down **CHECK -- does this site have a real MAAS `vm_host` with enrolled machines?** ```bash maas vm-host read # or: maas vm-hosts read | grep -i ``` If this returns nothing (expected for any VR1 site before Stage 4 has run for it): skip to Step 3, there is nothing to clean up at this layer yet. If it DOES return enrolled machines: release/delete them via MAAS's own current CLI/UI per its official docs BEFORE touching the underlying `tofu destroy` in Step 4 -- per the D-061 principle above, do not destroy the libvirt host out from under a MAAS record that still thinks it's live. This repo has not exercised this path for real; treat MAAS's own current documented release/delete flow as authoritative over anything asserted here. **GATE:** either no machines were ever enrolled (common case, pre-Stage-4), or MAAS confirms zero machines remain under this site's `vm_host` before proceeding. --- ## Step 3 (Path A only) -- Plan the scoped destroy, safe order, most-dependent-first OpenTofu's own dependency graph (`dc1_opnsense`/`dc1_node_*` reference `module.dc1_planes`/`module.dc1_storage` outputs) already sequences a single `-target` invocation correctly for everything BELOW it in the graph -- but `maas_vm_host` has no Terraform-expressible dependency on the node-VM domains it was registered against (it references `power_address`, not a domain resource), so it will NOT be destroyed by a plan targeting the storage/plane modules and must be listed explicitly if it exists. For DC1 (adapt names for DC2/Office1 once their own resources are real): ```bash cd opentofu tofu plan -destroy \ -target=module.dc1_maas_vm_host \ -target=module.dc1_opnsense \ -target=module.dc1_node_01 \ -target=module.dc1_node_02 \ ` # -- repeat -target=module.dc1_node_NN for every real node module block ` \ -target=module.dc1_storage \ -target=module.dc1_planes \ -out=teardown-dc1.tfplan ``` Only include `-target=module.dc1_maas_vm_host` if that module is actually wired in `main.tf` (Stage 3 Step 9) -- `tofu plan` errors on a `-target` naming a module that doesn't exist in configuration. **Do NOT target the mesh-link modules** (`mesh_dc1_dc2`, `mesh_dc1_office1`) in a single-DC teardown -- they are SHARED infrastructure with the other site at each leg's far end. Only include a mesh-link leg in the destroy set once BOTH sites it connects are being torn down (see the mesh-link note after Step 5). Review the plan line by line: expect destroys for exactly the resources above (`libvirt_domain`, `libvirt_volume`, `libvirt_network` x6 for the planes, `libvirt_pool`, and `maas_vm_host` if targeted) and NOTHING outside this site's scope -- in particular, confirm no `office1_*`/`dc2_*`/mesh-link resource appears. **GATE:** plan matches this scope exactly; no unexpected destroys outside the targeted site. Do not apply a plan you have not read. --- ## Step 4 (Path A only) -- Apply the scoped destroy [MUTATION: gated] ```bash cd opentofu tofu destroy teardown-dc1.tfplan ``` *(Note: unlike a create/apply plan, a `-destroy`-mode plan file is applied the same way -- `tofu apply ` and `tofu destroy ` are equivalent once the plan itself was generated with `-destroy`; use whichever your `tofu` version's docs confirm for a plan file specifically, and if in doubt, prefer re-running with `tofu destroy` directly rather than guessing.)* Confirm this is the exact reviewed plan file from Step 3 (not re-planned) before running. **VERIFY** ```bash virsh list --all | grep -i dc1 virsh net-list --all | grep -i dc1 virsh pool-list --all | grep -i dc1 ``` Expect: no DC1 domains, networks, or pool remain. --- ## Step 5 (Path A only) -- Confirm no drift outside the targeted scope ```bash cd opentofu tofu plan ``` Run WITHOUT `-target` this time -- this is the check HashiCorp's own targeting warning (Step 3) calls for: confirm the rest of the configuration (Office1, DC2 if wired, mesh links) shows no unexpected changes. Expect either "No changes" for everything outside DC1, or exactly the changes you independently expect for other in-flight work -- nothing attributable to this teardown. **GATE:** untargeted plan shows zero unexpected drift outside the torn-down site. --- ## Path B -- full VR1 teardown (replaces Steps 3-5 above) Do Step 1 (backup) and Step 2 (MAAS-side cleanup, for EVERY site with enrolled machines, not just one) first, then: ```bash cd opentofu tofu plan -destroy -out=teardown-full.tfplan ``` Review: expect a destroy for every resource this tree currently manages (check against `opentofu/main.tf`'s actual uncommented module list at teardown time -- it grows as more stages get executed, so what "everything" means changes over the buildout, unlike Path A's fixed per-site scope). **GATE:** plan matches full current `main.tf` scope exactly. ```bash tofu destroy teardown-full.tfplan ``` **VERIFY** ```bash virsh list --all virsh net-list --all virsh pool-list --all ``` Expect: nothing VR1-related remains (VR0's own DC0/testcloud resources, which OpenTofu does not manage, are untouched by any of this). Then do the equivalent of Step 5 (a final untargeted `tofu plan`) to confirm the destroy converged state to fully empty, not partially. --- ## Mesh-link teardown (either path, handle separately) The three D-100 mesh-link legs (`mesh_dc1_dc2`, `mesh_dc1_office1`, `mesh_dc2_office1`) are shared between two sites each. Destroy a leg only when BOTH its endpoints are permanently going away (e.g., a full VR1 teardown, or abandoning DC2 AND Office1 together) -- never as part of a single-site scoped teardown, and never as a reflex "clean up everything DC1-adjacent" action. `modules/netem-link` (if ever wired against a given leg) already has a correct `destroy`-time provisioner (`provisioner "local-exec" { when = destroy; ... tc qdisc del ... }`, confirmed by reading `opentofu/modules/netem-link/main.tf` directly) -- destroying the corresponding `terraform_data.netem` resource cleans up the real `tc qdisc` rule on the vcloud host automatically; no extra manual step needed for that specific piece. --- ## Rollback decision tree -- "a stage's `tofu apply` failed partway through, now what?" **Read this before reaching for `tofu destroy` at all.** OpenTofu is designed for idempotent convergence -- in most failure modes, fixing the root cause and re-running the SAME `tofu apply` is correct and safer than tearing down and starting over, because it preserves everything that DID apply successfully and only retries what didn't. 1. **`tofu apply` errored before creating anything (e.g., `tofu init`/ `validate`/`plan` failed, or `apply` failed on its very first resource).** Nothing was created. Fix the root cause (bad variable, unreachable provider, syntax error) and re-run from that stage's own `tofu plan` step. No teardown needed. 2. **`tofu apply` created SOME resources, then failed on a later one.** `tofu plan` (no `-destroy`) immediately to see OpenTofu's own view of what's now live vs. what the config wants. Default to FIX-FORWARD: fix whatever caused the failure (a bad node-VM memory size, an unreachable MAAS endpoint, a real infra problem) and re-run `tofu apply` for that same stage -- OpenTofu will create only what's still missing, leaving successfully-created resources alone. Reserve a teardown (Path A, scoped to that site) for when the partial state is itself broken in a way fix-forward can't reach (e.g., a domain stuck in a bad libvirt state that manual `virsh` intervention already tried and failed to fix). 3. **You suspect `terraform.tfstate` itself is corrupted or lost** (this is the scenario `opentofu/README.md`'s "State file handling" section already flags as the real cost of no backend/backup). There is no clean recovery path documented in this repo: either a manual `tofu import` per still-live resource (tedious, must be done for every domain/network/pool by hand) or accepting a full teardown-and-rebuild from Step 1 of the affected stage. This is exactly why Step 1's pre-teardown backup habit matters even outside a deliberate teardown -- back up `terraform.tfstate` after every successful stage apply too, not only before a teardown. 4. **A resource exists in reality but OpenTofu's state doesn't know about it** (e.g., a `virt-install`-created VM from Stage 2's Option B manual path, which OpenTofu never created and cannot destroy). This runbook's `tofu destroy`/`plan -destroy` commands only ever touch what OpenTofu itself manages -- anything created via a manual/Option-B path needs manual teardown (`virsh destroy`/`undefine`, etc.) by the same operator discipline that created it. Do not expect `tofu destroy` to find or remove it. --- ## Verification (of this runbook's own accuracy, not a live run) Every `tofu`/`terraform` flag cited above (`-destroy`, `-target`, the targeting-risk warning) was checked against HashiCorp's own current CLI documentation before being written here (fetched 2026-07-10), not asserted from memory -- same discipline as `opentofu/README.md`'s other research sections. The resource/module names and dependency shapes referenced (`dc1_maas_vm_host`'s lack of a Terraform-expressible dependency on node-VM domains, `netem-link`'s existing destroy-time provisioner) were confirmed by reading the actual `.tf` files in this repo, not assumed. What is NOT verified: this runbook's commands have never been run against a real `tofu` binary or a real state file (same SCAFFOLD/UNVALIDATED status as the rest of `opentofu/`) -- treat every command here as reviewed-but-unexercised until `scripts/opentofu-validate.sh` and a real dry run confirm otherwise.