VR1 DC-DC teardown / rollback runbook

Scope: tearing down or rolling back the infrastructure opentofu/ provisions for the VR1 (DC-DC) buildout -- DC1/DC2/Office1 plane and local networks, storage pools, node-VM domains, OPNsense edge VMs, the D-100 mesh-link triangle, and MAAS vm_host registrations. Written 2026-07-10 (DOCFIX-175 gap-register item #19) because nothing else in this repo covers this layer: runbooks/phase-00-teardown-release.sh / phase-00-teardown-destroy.sh (D-061) tear down the VR0 juju/MAAS-machine layer, a different pairing of tools and a different failure mode entirely -- see "Relationship to D-061" below for why that mechanism does not just transfer over.

UNVALIDATED, same caveat as everywhere else in opentofu/: no tofu binary has been available in any session that authored this tree. Every command below is standard, well-documented OpenTofu/Terraform CLI syntax (verified against HashiCorp's own current docs where cited), not invented -- but run bash scripts/opentofu-validate.sh on a machine with the real binary and read the result before trusting any of it against a live state file.

When to use this runbook

• Mid-buildout failure recovery. A stage's tofu apply failed partway through, or created something wrong, and you need to know whether to fix- forward or tear down and retry. See "Rollback decision tree" below -- READ THAT FIRST, since fix-forward is usually the right answer, not a destroy.
• Deliberate re-run from clean. You want to abandon a DC's (or Office1's, or the whole VR1 layer's) current state and rebuild from Step 1 of its stage runbook.
• NOT for: tearing down VR0's juju/MAAS-machine layer (use D-061's scripts), or removing a single juju application/relation from an already- deployed OpenStack bundle (that is normal juju operations, not this runbook).

Relationship to D-061 (read before mixing the two layers)

D-061's core lesson: destroying/releasing a resource at the WRONG layer, in the wrong order, orphans state at the OTHER layer -- juju destroy-model released MAAS-pod-composed machines and MAAS auto-decomposed them, losing the carve; the fix was remove-machine --keep-instance FIRST (detach from juju without releasing), verify survival in MAAS, THEN act on the model.

The same PRINCIPLE applies here, but the mechanism is different because the layers being coordinated are different:

• D-061 coordinates juju's machine view against MAAS's machine view of hosts MAAS itself pod-composed.
• This runbook coordinates OpenTofu's resource view (libvirt domains/ networks/pools it created directly) against MAAS's vm_host and (eventually) machine view of a libvirt host OpenTofu registered via modules/maas-vm-host (the maas_vm_host resource, canonical/maas provider -- NOT maas_vm_host_machine, which composes VMs itself; this repo deliberately uses the plain maas_vm_host variant so modules/ node-vm's own domains are the source of truth, not MAAS pod-composition -- see opentofu/README.md).

Because this repo doesn't use MAAS pod-composition for its OpenTofu-created VMs, the SPECIFIC --keep-instance decompose-on-release failure mode D-061 diagnosed does not directly apply here. The GENERAL principle still does: clean up MAAS's record of a host before destroying the libvirt resource underneath it, not after -- destroying the libvirt domain/host first would leave MAAS pointing at a power_address that no longer answers, an orphaned-but-not-obviously-broken record rather than a clean removal.

Residual open item, flagged not solved here: this repo has not yet reached Stage 4 (MAAS enlist/commission/deploy) for any VR1 DC, so no VR1 libvirt host has ever had real MAAS-enrolled machines under it. The exact MAAS-side commands to cleanly release machines enrolled under a vm_host this runbook is about to tear down (analogous to D-061's remove-machine --keep-instance discovery, but for the vm_host/enlist-commission-deploy flow instead of pod-composition) have not been worked out or tested. Step 2 below states the PRINCIPLE (MAAS-side first) and points at MAAS's own current docs for the exact commands -- do not invent flags for a scenario this repo has never actually exercised.

Two paths, by intent (mirrors D-061's release-vs-destroy split)

• Path A -- scoped teardown. Tear down ONE site's resources (a single DC, or Office1), leaving the other sites' infrastructure untouched. Use -target to scope both plan and apply. HashiCorp's own docs flag -target as an exceptional-circumstances tool, not for routine use, because targeted applies can leave configuration and state silently diverging: "This targeting capability is provided for exceptional circumstances, such as recovering from mistakes or working around Terraform limitations. It is not recommended to use -target for routine operations, since this can lead to undetected configuration drift" (developer.hashicorp.com/ terraform/cli/commands/plan, fetched 2026-07-10). A scoped teardown IS exactly the kind of exceptional circumstance this describes -- acceptable here, but ALWAYS follow it with a full (untargeted) tofu plan afterward (Step 6) to confirm the rest of the configuration didn't drift.
• Path B -- full VR1 teardown. Abandon everything OpenTofu manages for VR1. Plain tofu destroy (no -target), letting OpenTofu compute the full safe order from the resource graph itself -- no manual sequencing needed, unlike Path A.

Both paths share Steps 1 (state backup) and 6 (final verify). Path A is Steps 2-5; Path B replaces Steps 3-5 with a single plain destroy (see "Path B" section after Step 5).

Step 1 -- Back up the state file before ANY destroy [do this every time]

A destroy mutates state exactly like an apply does -- if it's interrupted partway (network drop, Ctrl-C, host reboot), your only way to know what's actually still live is the state file as it stood before you started.

cd opentofu
cp -a terraform.tfstate "terraform.tfstate.pre-teardown-$(date +%Y%m%d-%H%M%S)"

Store the copy out-of-band per the operator's existing ~/vault-init/-class secret-handling process (it carries the same plaintext-credential risk as the live file -- see opentofu/README.md "State file handling", DOCFIX-175) -- do not just leave it sitting next to the live file as your only copy.

Step 2 -- MAAS-side cleanup FIRST, if any machines were ever enrolled under the site being torn down

CHECK -- does this site have a real MAAS vm_host with enrolled machines?

maas <profile> vm-host read <vm-host-id>   # or: maas <profile> vm-hosts read | grep -i <site>

If this returns nothing (expected for any VR1 site before Stage 4 has run for it): skip to Step 3, there is nothing to clean up at this layer yet.

If it DOES return enrolled machines: release/delete them via MAAS's own current CLI/UI per its official docs BEFORE touching the underlying tofu destroy in Step 4 -- per the D-061 principle above, do not destroy the libvirt host out from under a MAAS record that still thinks it's live. This repo has not exercised this path for real; treat MAAS's own current documented release/delete flow as authoritative over anything asserted here.

GATE: either no machines were ever enrolled (common case, pre-Stage-4), or MAAS confirms zero machines remain under this site's vm_host before proceeding.

Step 3 (Path A only) -- Plan the scoped destroy, safe order, most-dependent-first

OpenTofu's own dependency graph (dc1_opnsense/dc1_node_* reference module.dc1_planes/module.dc1_storage outputs) already sequences a single -target invocation correctly for everything BELOW it in the graph -- but maas_vm_host has no Terraform-expressible dependency on the node-VM domains it was registered against (it references power_address, not a domain resource), so it will NOT be destroyed by a plan targeting the storage/plane modules and must be listed explicitly if it exists.

For DC1 (adapt names for DC2/Office1 once their own resources are real):

cd opentofu
tofu plan -destroy \
  -target=module.dc1_maas_vm_host \
  -target=module.dc1_opnsense \
  -target=module.dc1_node_01 \
  -target=module.dc1_node_02 \
  ` # -- repeat -target=module.dc1_node_NN for every real node module block ` \
  -target=module.dc1_storage \
  -target=module.dc1_planes \
  -out=teardown-dc1.tfplan

Only include -target=module.dc1_maas_vm_host if that module is actually wired in main.tf (Stage 3 Step 9) -- tofu plan errors on a -target naming a module that doesn't exist in configuration.

Do NOT target the mesh-link modules (mesh_dc1_dc2, mesh_dc1_office1) in a single-DC teardown -- they are SHARED infrastructure with the other site at each leg's far end. Only include a mesh-link leg in the destroy set once BOTH sites it connects are being torn down (see the mesh-link note after Step 5).

Review the plan line by line: expect destroys for exactly the resources above (libvirt_domain, libvirt_volume, libvirt_network x6 for the planes, libvirt_pool, and maas_vm_host if targeted) and NOTHING outside this site's scope -- in particular, confirm no office1_*/dc2_*/mesh-link resource appears.

GATE: plan matches this scope exactly; no unexpected destroys outside the targeted site. Do not apply a plan you have not read.

Step 4 (Path A only) -- Apply the scoped destroy [MUTATION: gated]

cd opentofu
tofu destroy teardown-dc1.tfplan

(Note: unlike a create/apply plan, a -destroy-mode plan file is applied the same way -- tofu apply <planfile> and tofu destroy <planfile> are equivalent once the plan itself was generated with -destroy; use whichever your tofu version's docs confirm for a plan file specifically, and if in doubt, prefer re-running with tofu destroy directly rather than guessing.)

Confirm this is the exact reviewed plan file from Step 3 (not re-planned) before running.

VERIFY

virsh list --all | grep -i dc1
virsh net-list --all | grep -i dc1
virsh pool-list --all | grep -i dc1

Expect: no DC1 domains, networks, or pool remain.

Step 5 (Path A only) -- Confirm no drift outside the targeted scope

cd opentofu
tofu plan

Run WITHOUT -target this time -- this is the check HashiCorp's own targeting warning (Step 3) calls for: confirm the rest of the configuration (Office1, DC2 if wired, mesh links) shows no unexpected changes. Expect either "No changes" for everything outside DC1, or exactly the changes you independently expect for other in-flight work -- nothing attributable to this teardown.

GATE: untargeted plan shows zero unexpected drift outside the torn-down site.

Path B -- full VR1 teardown (replaces Steps 3-5 above)

Do Step 1 (backup) and Step 2 (MAAS-side cleanup, for EVERY site with enrolled machines, not just one) first, then:

cd opentofu
tofu plan -destroy -out=teardown-full.tfplan

Review: expect a destroy for every resource this tree currently manages (check against opentofu/main.tf's actual uncommented module list at teardown time -- it grows as more stages get executed, so what "everything" means changes over the buildout, unlike Path A's fixed per-site scope).

GATE: plan matches full current main.tf scope exactly.

tofu destroy teardown-full.tfplan

VERIFY

virsh list --all
virsh net-list --all
virsh pool-list --all

Expect: nothing VR1-related remains (VR0's own DC0/testcloud resources, which OpenTofu does not manage, are untouched by any of this).

Then do the equivalent of Step 5 (a final untargeted tofu plan) to confirm the destroy converged state to fully empty, not partially.

Mesh-link teardown (either path, handle separately)

The three D-100 mesh-link legs (mesh_dc1_dc2, mesh_dc1_office1, mesh_dc2_office1) are shared between two sites each. Destroy a leg only when BOTH its endpoints are permanently going away (e.g., a full VR1 teardown, or abandoning DC2 AND Office1 together) -- never as part of a single-site scoped teardown, and never as a reflex "clean up everything DC1-adjacent" action. modules/netem-link (if ever wired against a given leg) already has a correct destroy-time provisioner (provisioner "local-exec" { when = destroy; ... tc qdisc del ... }, confirmed by reading opentofu/modules/netem-link/main.tf directly) -- destroying the corresponding terraform_data.netem resource cleans up the real tc qdisc rule on the vcloud host automatically; no extra manual step needed for that specific piece.

Rollback decision tree -- "a stage's tofu apply failed partway through, now what?"

Read this before reaching for tofu destroy at all. OpenTofu is designed for idempotent convergence -- in most failure modes, fixing the root cause and re-running the SAME tofu apply is correct and safer than tearing down and starting over, because it preserves everything that DID apply successfully and only retries what didn't.

1. tofu apply errored before creating anything (e.g., tofu init/ validate/plan failed, or apply failed on its very first resource). Nothing was created. Fix the root cause (bad variable, unreachable provider, syntax error) and re-run from that stage's own tofu plan step. No teardown needed.
2. tofu apply created SOME resources, then failed on a later one. tofu plan (no -destroy) immediately to see OpenTofu's own view of what's now live vs. what the config wants. Default to FIX-FORWARD: fix whatever caused the failure (a bad node-VM memory size, an unreachable MAAS endpoint, a real infra problem) and re-run tofu apply for that same stage -- OpenTofu will create only what's still missing, leaving successfully-created resources alone. Reserve a teardown (Path A, scoped to that site) for when the partial state is itself broken in a way fix-forward can't reach (e.g., a domain stuck in a bad libvirt state that manual virsh intervention already tried and failed to fix).
3. You suspect terraform.tfstate itself is corrupted or lost (this is the scenario opentofu/README.md's "State file handling" section already flags as the real cost of no backend/backup). There is no clean recovery path documented in this repo: either a manual tofu import per still-live resource (tedious, must be done for every domain/network/pool by hand) or accepting a full teardown-and-rebuild from Step 1 of the affected stage. This is exactly why Step 1's pre-teardown backup habit matters even outside a deliberate teardown -- back up terraform.tfstate after every successful stage apply too, not only before a teardown.
4. A resource exists in reality but OpenTofu's state doesn't know about it (e.g., a virt-install-created VM from Stage 2's Option B manual path, which OpenTofu never created and cannot destroy). This runbook's tofu destroy/plan -destroy commands only ever touch what OpenTofu itself manages -- anything created via a manual/Option-B path needs manual teardown (virsh destroy/undefine, etc.) by the same operator discipline that created it. Do not expect tofu destroy to find or remove it.

Verification (of this runbook's own accuracy, not a live run)

Every tofu/terraform flag cited above (-destroy, -target, the targeting-risk warning) was checked against HashiCorp's own current CLI documentation before being written here (fetched 2026-07-10), not asserted from memory -- same discipline as opentofu/README.md's other research sections. The resource/module names and dependency shapes referenced (dc1_maas_vm_host's lack of a Terraform-expressible dependency on node-VM domains, netem-link's existing destroy-time provisioner) were confirmed by reading the actual .tf files in this repo, not assumed. What is NOT verified: this runbook's commands have never been run against a real tofu binary or a real state file (same SCAFFOLD/UNVALIDATED status as the rest of opentofu/) -- treat every command here as reviewed-but-unexercised until scripts/opentofu-validate.sh and a real dry run confirm otherwise.