# VR1 DC-DC teardown / rollback runbook

Scope: tearing down or rolling back the infrastructure `opentofu/` provisions
for the VR1 (DC-DC) buildout -- DC1/DC2/Office1 plane and local networks,
storage pools, node-VM domains, OPNsense edge VMs, the D-100 mesh-link
triangle, and MAAS `vm_host` registrations. Written 2026-07-10 (DOCFIX-175
gap-register item #19) because nothing else in this repo covers this layer:
`runbooks/phase-00-teardown-release.sh` / `phase-00-teardown-destroy.sh`
(D-061) tear down the VR0 **juju/MAAS-machine** layer, a different pairing
of tools and a different failure mode entirely -- see "Relationship to
D-061" below for why that mechanism does not just transfer over.

**UNVALIDATED, same caveat as everywhere else in `opentofu/`:** no `tofu`
binary has been available in any session that authored this tree. Every
command below is standard, well-documented OpenTofu/Terraform CLI syntax
(verified against HashiCorp's own current docs where cited), not invented --
but run `bash scripts/opentofu-validate.sh` on a machine with the real
binary and read the result before trusting any of it against a live state
file.

---

> ## !!! MODEL B RESHAPE (D-123, 2026-07-16) -- READ BEFORE ANY DC TEARDOWN; this runbook's body predates it !!!
>
> This runbook was written (2026-07-10) for the SINGLE-ROOT shape (OpenTofu creates the DC node-VM
> domains / planes / edge directly on the **vcloud** host, and `maas-vm-host` registers the vcloud
> host). Under **D-123 Model B** that is no longer how a DC is built, so the resource-by-resource
> mechanics below are partly WRONG for the DCs. Current shape:
>
> - **TWO roots / TWO state files.** The OUTER root (`opentofu/`, vcloud) owns `vvr1-dc0` (the DC
>   containment VM), the office1<->dc0 transit, and the **D-125** vcloud ISP uplink (`vr1-dc0-uplink`
>   NAT + the `vvr1-dc0` uplink NIC/`br-vr1-dc0-wan` netplan bridge). The INNER root
>   (`opentofu/vr1-dc0-substrate/`, provider = `qemu+ssh` to `vvr1-dc0`) owns everything INSIDE the
>   DC -- the 9 node VMs, the 6 planes, `vr1-dc0-wan`, the OPNsense edge, the inner pool. So the
>   "node-VM domains / planes / edge" this runbook talks about live on **`vvr1-dc0`'s libvirt, not
>   vcloud's**, and `maas-vm-host` registers **`vvr1-dc0`'s inner virsh** to the Office1 REGION.
> - **Whole-DC site-down is ONE object: `virsh destroy vvr1-dc0`** (the D-122/D-123 point). Destroying
>   the containment VM instantly takes down the entire inner fleet -- no per-domain group destroy
>   needed. That is the fast path for "abandon this DC and rebuild."
> - **Teardown ORDER (two roots):** tear down the INNER root first (`cd opentofu/vr1-dc0-substrate &&
>   tofu destroy`, or just `virsh destroy vvr1-dc0` if you're discarding the whole DC), THEN the OUTER
>   root. The D-061-style "clean up MAAS's record before destroying the libvirt underneath" principle
>   below still applies -- but the `maas-vm-host` record now points at `vvr1-dc0`'s inner virsh, so
>   remove it BEFORE destroying `vvr1-dc0`.
> - **Reverting Model B -> Model A** (a different operation from teardown) is
>   `docs/archive/model-a-fallback-plan.md` (git tag `model-a-fallback`), not this runbook.
>
> Until the body below is rewritten to the two-root shape, treat its DC-substrate steps as
> **single-root reference** and apply the mapping above. The Office1 (`voffice1`) and mesh/pool steps
> are unaffected.

---

## When to use this runbook

- **Mid-buildout failure recovery.** A stage's `tofu apply` failed partway
  through, or created something wrong, and you need to know whether to fix-
  forward or tear down and retry. See "Rollback decision tree" below --
  READ THAT FIRST, since fix-forward is usually the right answer, not a
  destroy.
- **Deliberate re-run from clean.** You want to abandon a DC's (or Office1's,
  or the whole VR1 layer's) current state and rebuild from Step 1 of its
  stage runbook.
- **NOT for**: tearing down VR0's juju/MAAS-machine layer (use D-061's
  scripts), or removing a single juju application/relation from an already-
  deployed OpenStack bundle (that is normal juju operations, not this
  runbook).

---

## Relationship to D-061 (read before mixing the two layers)

D-061's core lesson: destroying/releasing a resource at the WRONG layer, in
the wrong order, orphans state at the OTHER layer -- `juju destroy-model`
released MAAS-pod-composed machines and MAAS auto-decomposed them, losing
the carve; the fix was `remove-machine --keep-instance` FIRST (detach from
juju without releasing), verify survival in MAAS, THEN act on the model.

The same PRINCIPLE applies here, but the mechanism is different because the
layers being coordinated are different:

- D-061 coordinates **juju's machine view** against **MAAS's machine view**
  of hosts MAAS itself pod-composed.
- This runbook coordinates **OpenTofu's resource view** (libvirt domains/
  networks/pools it created directly) against **MAAS's `vm_host` and
  (eventually) machine view** of a libvirt host OpenTofu registered via
  `modules/maas-vm-host` (the `maas_vm_host` resource, canonical/maas
  provider -- NOT `maas_vm_host_machine`, which composes VMs itself; this
  repo deliberately uses the plain `maas_vm_host` variant so `modules/
  node-vm`'s own domains are the source of truth, not MAAS pod-composition
  -- see `opentofu/README.md`).

Because this repo doesn't use MAAS pod-composition for its OpenTofu-created
VMs, the SPECIFIC `--keep-instance` decompose-on-release failure mode D-061
diagnosed does not directly apply here. The GENERAL principle still does:
**clean up MAAS's record of a host before destroying the libvirt resource
underneath it**, not after -- destroying the libvirt domain/host first would
leave MAAS pointing at a `power_address` that no longer answers, an
orphaned-but-not-obviously-broken record rather than a clean removal.

**Residual open item, flagged not solved here:** this repo has not yet
reached Stage 4 (MAAS enlist/commission/deploy) for any VR1 DC, so no VR1
libvirt host has ever had real MAAS-enrolled machines under it. The exact
MAAS-side commands to cleanly release machines enrolled under a `vm_host`
this runbook is about to tear down (analogous to D-061's `remove-machine
--keep-instance` discovery, but for the `vm_host`/enlist-commission-deploy
flow instead of pod-composition) have not been worked out or tested. Step 2
below states the PRINCIPLE (MAAS-side first) and points at MAAS's own
current docs for the exact commands -- do not invent flags for a scenario
this repo has never actually exercised.

---

## Two paths, by intent (mirrors D-061's release-vs-destroy split)

- **Path A -- scoped teardown.** Tear down ONE site's resources (a single DC,
  or Office1), leaving the other sites' infrastructure untouched. Use
  `-target` to scope both plan and apply. HashiCorp's own docs flag `-target`
  as an exceptional-circumstances tool, not for routine use, because
  targeted applies can leave configuration and state silently diverging:
  *"This targeting capability is provided for exceptional circumstances,
  such as recovering from mistakes or working around Terraform limitations.
  It is not recommended to use `-target` for routine operations, since this
  can lead to undetected configuration drift"* (developer.hashicorp.com/
  terraform/cli/commands/plan, fetched 2026-07-10). A scoped teardown IS
  exactly the kind of exceptional circumstance this describes -- acceptable
  here, but ALWAYS follow it with a full (untargeted) `tofu plan` afterward
  (Step 6) to confirm the rest of the configuration didn't drift.
- **Path B -- full VR1 teardown.** Abandon everything OpenTofu manages for
  VR1. Plain `tofu destroy` (no `-target`), letting OpenTofu compute the
  full safe order from the resource graph itself -- no manual sequencing
  needed, unlike Path A.

Both paths share Steps 1 (state backup) and 6 (final verify). Path A is
Steps 2-5; Path B replaces Steps 3-5 with a single plain destroy (see
"Path B" section after Step 5).

---

## Step 1 -- Back up the state file before ANY destroy [do this every time]

A `destroy` mutates state exactly like an `apply` does -- if it's
interrupted partway (network drop, `Ctrl-C`, host reboot), your only way to
know what's actually still live is the state file as it stood before you
started.

```bash
cd opentofu
cp -a terraform.tfstate "terraform.tfstate.pre-teardown-$(date +%Y%m%d-%H%M%S)"
```
Store the copy out-of-band per the operator's existing `~/vault-init/`-class
secret-handling process (it carries the same plaintext-credential risk as
the live file -- see `opentofu/README.md` "State file handling", DOCFIX-175)
-- do not just leave it sitting next to the live file as your only copy.

---

## Step 2 -- MAAS-side cleanup FIRST, if any machines were ever enrolled under the site being torn down

**CHECK -- does this site have a real MAAS `vm_host` with enrolled machines?**
```bash
maas <profile> vm-host read <vm-host-id>   # or: maas <profile> vm-hosts read | grep -i <site>
```
If this returns nothing (expected for any VR1 site before Stage 4 has run
for it): skip to Step 3, there is nothing to clean up at this layer yet.

If it DOES return enrolled machines: release/delete them via MAAS's own
current CLI/UI per its official docs BEFORE touching the underlying
`tofu destroy` in Step 4 -- per the D-061 principle above, do not destroy
the libvirt host out from under a MAAS record that still thinks it's live.
This repo has not exercised this path for real; treat MAAS's own current
documented release/delete flow as authoritative over anything asserted here.

**GATE:** either no machines were ever enrolled (common case, pre-Stage-4),
or MAAS confirms zero machines remain under this site's `vm_host` before
proceeding.

---

## Step 3 (Path A only) -- Plan the scoped destroy, safe order, most-dependent-first

OpenTofu's own dependency graph (`dc1_opnsense`/`dc1_node_*` reference
`module.dc1_planes`/`module.dc1_storage` outputs) already sequences a single
`-target` invocation correctly for everything BELOW it in the graph -- but
`maas_vm_host` has no Terraform-expressible dependency on the node-VM
domains it was registered against (it references `power_address`, not a
domain resource), so it will NOT be destroyed by a plan targeting the
storage/plane modules and must be listed explicitly if it exists.

For DC1 (adapt names for DC2/Office1 once their own resources are real):

```bash
cd opentofu
tofu plan -destroy \
  -target=module.dc1_maas_vm_host \
  -target=module.dc1_opnsense \
  -target=module.dc1_node_01 \
  -target=module.dc1_node_02 \
  ` # -- repeat -target=module.dc1_node_NN for every real node module block ` \
  -target=module.dc1_storage \
  -target=module.dc1_planes \
  -out=teardown-dc1.tfplan
```
Only include `-target=module.dc1_maas_vm_host` if that module is actually
wired in `main.tf` (Stage 3 Step 9) -- `tofu plan` errors on a `-target`
naming a module that doesn't exist in configuration.

**Do NOT target the mesh-link modules** (`mesh_dc1_dc2`, `mesh_dc1_office1`)
in a single-DC teardown -- they are SHARED infrastructure with the other
site at each leg's far end. Only include a mesh-link leg in the destroy set
once BOTH sites it connects are being torn down (see the mesh-link note
after Step 5).

Review the plan line by line: expect destroys for exactly the resources
above (`libvirt_domain`, `libvirt_volume`, `libvirt_network` x6 for the
planes, `libvirt_pool`, and `maas_vm_host` if targeted) and NOTHING outside
this site's scope -- in particular, confirm no `office1_*`/`dc2_*`/mesh-link
resource appears.

**GATE:** plan matches this scope exactly; no unexpected destroys outside
the targeted site. Do not apply a plan you have not read.

---

## Step 4 (Path A only) -- Apply the scoped destroy [MUTATION: gated]

```bash
cd opentofu
tofu destroy teardown-dc1.tfplan
```
*(Note: unlike a create/apply plan, a `-destroy`-mode plan file is applied
the same way -- `tofu apply <planfile>` and `tofu destroy <planfile>` are
equivalent once the plan itself was generated with `-destroy`; use whichever
your `tofu` version's docs confirm for a plan file specifically, and if in
doubt, prefer re-running with `tofu destroy` directly rather than guessing.)*

Confirm this is the exact reviewed plan file from Step 3 (not re-planned)
before running.

**VERIFY**
```bash
virsh list --all | grep -i dc1
virsh net-list --all | grep -i dc1
virsh pool-list --all | grep -i dc1
```
Expect: no DC1 domains, networks, or pool remain.

---

## Step 5 (Path A only) -- Confirm no drift outside the targeted scope

```bash
cd opentofu
tofu plan
```
Run WITHOUT `-target` this time -- this is the check HashiCorp's own
targeting warning (Step 3) calls for: confirm the rest of the configuration
(Office1, DC2 if wired, mesh links) shows no unexpected changes. Expect
either "No changes" for everything outside DC1, or exactly the changes you
independently expect for other in-flight work -- nothing attributable to
this teardown.

**GATE:** untargeted plan shows zero unexpected drift outside the torn-down
site.

---

## Path B -- full VR1 teardown (replaces Steps 3-5 above)

Do Step 1 (backup) and Step 2 (MAAS-side cleanup, for EVERY site with
enrolled machines, not just one) first, then:

```bash
cd opentofu
tofu plan -destroy -out=teardown-full.tfplan
```
Review: expect a destroy for every resource this tree currently manages
(check against `opentofu/main.tf`'s actual uncommented module list at
teardown time -- it grows as more stages get executed, so what "everything"
means changes over the buildout, unlike Path A's fixed per-site scope).

**GATE:** plan matches full current `main.tf` scope exactly.

```bash
tofu destroy teardown-full.tfplan
```

**VERIFY**
```bash
virsh list --all
virsh net-list --all
virsh pool-list --all
```
Expect: nothing VR1-related remains (VR0's own DC0/testcloud resources,
which OpenTofu does not manage, are untouched by any of this).

Then do the equivalent of Step 5 (a final untargeted `tofu plan`) to confirm
the destroy converged state to fully empty, not partially.

---

## Mesh-link teardown (either path, handle separately)

The three D-100 mesh-link legs (`mesh_dc1_dc2`, `mesh_dc1_office1`,
`mesh_dc2_office1`) are shared between two sites each. Destroy a leg only
when BOTH its endpoints are permanently going away (e.g., a full VR1
teardown, or abandoning DC2 AND Office1 together) -- never as part of a
single-site scoped teardown, and never as a reflex "clean up everything
DC1-adjacent" action. `modules/netem-link` (if ever wired against a given
leg) already has a correct `destroy`-time provisioner
(`provisioner "local-exec" { when = destroy; ... tc qdisc del ... }`,
confirmed by reading `opentofu/modules/netem-link/main.tf` directly) --
destroying the corresponding `terraform_data.netem` resource cleans up the
real `tc qdisc` rule on the vcloud host automatically; no extra manual step
needed for that specific piece.

---

## Rollback decision tree -- "a stage's `tofu apply` failed partway through, now what?"

**Read this before reaching for `tofu destroy` at all.** OpenTofu is
designed for idempotent convergence -- in most failure modes, fixing the
root cause and re-running the SAME `tofu apply` is correct and safer than
tearing down and starting over, because it preserves everything that DID
apply successfully and only retries what didn't.

1. **`tofu apply` errored before creating anything (e.g., `tofu init`/
   `validate`/`plan` failed, or `apply` failed on its very first resource).**
   Nothing was created. Fix the root cause (bad variable, unreachable
   provider, syntax error) and re-run from that stage's own `tofu plan`
   step. No teardown needed.
2. **`tofu apply` created SOME resources, then failed on a later one.**
   `tofu plan` (no `-destroy`) immediately to see OpenTofu's own view of
   what's now live vs. what the config wants. Default to FIX-FORWARD: fix
   whatever caused the failure (a bad node-VM memory size, an
   unreachable MAAS endpoint, a real infra problem) and re-run
   `tofu apply` for that same stage -- OpenTofu will create only what's
   still missing, leaving successfully-created resources alone. Reserve a
   teardown (Path A, scoped to that site) for when the partial state is
   itself broken in a way fix-forward can't reach (e.g., a domain stuck in
   a bad libvirt state that manual `virsh` intervention already tried and
   failed to fix).
3. **You suspect `terraform.tfstate` itself is corrupted or lost** (this is
   the scenario `opentofu/README.md`'s "State file handling" section
   already flags as the real cost of no backend/backup). There is no clean
   recovery path documented in this repo: either a manual `tofu import` per
   still-live resource (tedious, must be done for every domain/network/pool
   by hand) or accepting a full teardown-and-rebuild from Step 1 of the
   affected stage. This is exactly why Step 1's pre-teardown backup habit
   matters even outside a deliberate teardown -- back up
   `terraform.tfstate` after every successful stage apply too, not only
   before a teardown.
4. **A resource exists in reality but OpenTofu's state doesn't know about
   it** (e.g., a `virt-install`-created VM from Stage 2's Option B manual
   path, which OpenTofu never created and cannot destroy). This runbook's
   `tofu destroy`/`plan -destroy` commands only ever touch what OpenTofu
   itself manages -- anything created via a manual/Option-B path needs
   manual teardown (`virsh destroy`/`undefine`, etc.) by the same operator
   discipline that created it. Do not expect `tofu destroy` to find or
   remove it.

---

## Verification (of this runbook's own accuracy, not a live run)

Every `tofu`/`terraform` flag cited above (`-destroy`, `-target`, the
targeting-risk warning) was checked against HashiCorp's own current CLI
documentation before being written here (fetched 2026-07-10), not asserted
from memory -- same discipline as `opentofu/README.md`'s other research
sections. The resource/module names and dependency shapes referenced
(`dc1_maas_vm_host`'s lack of a Terraform-expressible dependency on node-VM
domains, `netem-link`'s existing destroy-time provisioner) were confirmed by
reading the actual `.tf` files in this repo, not assumed. What is NOT
verified: this runbook's commands have never been run against a real `tofu`
binary or a real state file (same SCAFFOLD/UNVALIDATED status as the rest of
`opentofu/`) -- treat every command here as reviewed-but-unexercised until
`scripts/opentofu-validate.sh` and a real dry run confirm otherwise.
