# VR1 DC-DC teardown / rollback runbook Scope: tearing down or rolling back the infrastructure `opentofu/` provisions for the VR1 (DC-DC) buildout -- the `vr1-dc0` / `vr1-dc1` / Office1 plane and local networks, storage pools, node-VM domains, containment VMs, OPNsense edge VMs, and the D-100 mesh-link triangle. (It once also claimed MAAS `vm_host` registrations; there are none -- see Step 2. The `dc1`/`dc2` names it was written with are pre-D-119 and mean different datacenters now.) Written 2026-07-10 (DOCFIX-175 gap-register item #19) because nothing else in this repo covers this layer: `runbooks/phase-00-teardown-release.sh` / `phase-00-teardown-destroy.sh` (D-061) tear down the VR0 **juju/MAAS-machine** layer, a different pairing of tools and a different failure mode entirely -- see "Relationship to D-061" below for why that mechanism does not just transfer over. **UNVALIDATED, same caveat as everywhere else in `opentofu/`:** no `tofu` binary has been available in any session that authored this tree. Every command below is standard, well-documented OpenTofu/Terraform CLI syntax (verified against HashiCorp's own current docs where cited), not invented -- but run `bash scripts/opentofu-validate.sh` on a machine with the real binary and read the result before trusting any of it against a live state file. --- > ## !!! MODEL B RESHAPE (D-123, 2026-07-16) -- READ BEFORE ANY DC TEARDOWN; this runbook's body predates it !!! > > This runbook was written (2026-07-10) for the SINGLE-ROOT shape (OpenTofu creates the DC node-VM > domains / planes / edge directly on the **vcloud** host, and `maas-vm-host` registers the vcloud > host). Under **D-123 Model B** that is no longer how a DC is built, so the resource-by-resource > mechanics below are partly WRONG for the DCs. Current shape: > > - **TWO roots / TWO state files PER DC, and there are now TWO DCs.** The OUTER root (`opentofu/`, > vcloud) owns `vvr1-dc0` and `vvr1-dc1` (the DC containment VMs), the office1<->dcN transits, and > the **D-125** vcloud ISP uplinks (`vr1-dc0-uplink` / `vr1-dc1-uplink` NAT + each containment VM's > uplink NIC / `br-vr1-dcN-wan` netplan bridge). Each INNER root > (`opentofu/vr1-dc0-substrate/`, `opentofu/vr1-dc1-substrate/`; provider = `qemu+ssh` to that DC's > containment VM) owns everything INSIDE its DC -- the node VMs, the 6 planes, `vr1-dcN-wan`, the > OPNsense edge, the inner pool. So the "node-VM domains / planes / edge" this runbook talks about > live on **the containment VM's libvirt, not vcloud's**. > - **There is NO MAAS `vm_host`/pod for any VR1 DC.** MAAS holds per-MACHINE records with > `power_type=virsh` (D-103/D-123 amendments, ruled 2026-07-20) and `modules/maas-vm-host` is > instantiated in NEITHER root -- measured 2026-07-29. Any instruction anywhere to "remove the > `maas-vm-host` record before destroying the containment VM" is VOID: there is no such record, > the read returns nothing on a DC with a full fleet enrolled, and acting on that emptiness > destroys the containment VM out from under live MAAS machine records. **Step 2 is the real > check.** > - **Whole-DC site-down is ONE object: `virsh destroy vvr1-dcN`** (the D-122/D-123 point). Destroying > the containment VM instantly takes down the entire inner fleet -- no per-domain group destroy > needed. That is the fast path for "abandon this DC and rebuild." **Know which verb you are > using:** `virsh destroy` is a forced POWER-OFF and is REVERSIBLE (`virsh start vvr1-dcN`); the > domain, its disk and every MAAS record still exist. `tofu destroy` of `module.vvr1_dcN` UNDEFINES > the domain and deletes its disk -- irreversible, and it takes the whole inner fleet with it. > Under a failed Stage 5 those are not interchangeable. > - **Teardown ORDER (two roots):** tear down the INNER root first (`cd opentofu/vr1-dcN-substrate && > tofu destroy`, or just `virsh destroy vvr1-dcN` if you are only taking the site DOWN), THEN the > OUTER root. The D-061-style "clean up MAAS's record before destroying the libvirt underneath" > principle below still applies -- and with no pod to remove, "MAAS's record" means the per-machine > records enumerated in Step 2. Run Step 2 BEFORE either root's destroy. > - **Reverting Model B -> Model A** (a different operation from teardown) is > `docs/archive/model-a-fallback-plan.md` (git tag `model-a-fallback`), not this runbook. > > Until the body below is rewritten to the two-root shape, treat its DC-substrate steps as > **single-root reference** and apply the mapping above. The Office1 (`voffice1`) and mesh/pool steps > are unaffected. --- ## When to use this runbook - **Mid-buildout failure recovery.** A stage's `tofu apply` failed partway through, or created something wrong, and you need to know whether to fix- forward or tear down and retry. See "Rollback decision tree" below -- READ THAT FIRST, since fix-forward is usually the right answer, not a destroy. - **Deliberate re-run from clean.** You want to abandon a DC's (or Office1's, or the whole VR1 layer's) current state and rebuild from Step 1 of its stage runbook. - **NOT for**: tearing down VR0's juju/MAAS-machine layer (use D-061's scripts), or removing a single juju application/relation from an already- deployed OpenStack bundle (that is normal juju operations, not this runbook). --- ## Relationship to D-061 (read before mixing the two layers) D-061's core lesson: destroying/releasing a resource at the WRONG layer, in the wrong order, orphans state at the OTHER layer -- `juju destroy-model` released MAAS-pod-composed machines and MAAS auto-decomposed them, losing the carve; the fix was `remove-machine --keep-instance` FIRST (detach from juju without releasing), verify survival in MAAS, THEN act on the model. The same PRINCIPLE applies here, but the mechanism is different because the layers being coordinated are different: - D-061 coordinates **juju's machine view** against **MAAS's machine view** of hosts MAAS itself pod-composed. - This runbook coordinates **OpenTofu's resource view** (libvirt domains/ networks/pools it created directly) against **MAAS's per-MACHINE view** of those same domains. MEASURED 2026-07-29: `modules/maas-vm-host` is instantiated in NEITHER root (`grep -n 'maas_vm_host' opentofu/*.tf opentofu/vr1-dc0-substrate/*.tf opentofu/vr1-dc1-substrate/*.tf` returns only comments), and the `provider "maas"` block those modules would need is still deliberately absent (`opentofu/main.tf:9-20`, DOCFIX-179). MAAS knows the DC node VMs because they PXE-ENLISTED, and it powers them per machine with `power_type=virsh` (D-103/D-123 amendments, ruled 2026-07-20 -- `scripts/maas-node-power.sh`'s header carries the two measurements that refuted the pod). Because this repo uses neither MAAS pod-composition NOR a `vm_host` for its OpenTofu-created VMs, the SPECIFIC `--keep-instance` decompose-on-release failure mode D-061 diagnosed does not directly apply here. The GENERAL principle still does: **clean up MAAS's record of a host before destroying the libvirt resource underneath it**, not after -- destroying the libvirt domain/ host first would leave MAAS pointing at a `power_address` that no longer answers, an orphaned-but-not-obviously-broken record rather than a clean removal. **What this section used to say, and why it is now a hazard rather than a caveat:** it stated that no VR1 DC had reached Stage 4, so no VR1 libvirt host had ever had real MAAS-enrolled machines under it. **That is false as of 2026-07-23** (`docs/CURRENT-STATE.md`: 18 nodes READY, 9 per DC, both DCs commissioned; `power_type=virsh` set and proven by a real `query-power-state`), and the per-DC Juju controller VMs were enlisted on top of that in July 2026. Anything downstream of "nothing is enrolled yet" -- including the old Step 2 -- must be treated as REVERSED, not merely stale. The exact MAAS-side release/delete commands for a machine whose libvirt domain is about to disappear are STILL not worked out or tested in this repo: Step 2 states the principle, enumerates the records, and points at MAAS's own current docs for the flags. Do not invent flags for a scenario this repo has never actually exercised. --- ## TWO CLONES, TWO HOSTS -- set these once, before anything below Every block in this runbook runs on one of two hosts (D-128), each with its own clone of this repo. The same `cd` pasted on the wrong one is the foot-gun this section exists to remove, so the two are named differently throughout: - **`$REPO` -- the VCLOUD clone.** Plane 1: the OUTER root (`$REPO/opentofu`, `libvirt_uri = qemu:///system`) and local `virsh`. - **`$O1_REPO` -- the OFFICE1 HEADEND clone, on `voffice1`.** Plane 2: BOTH DC substrate roots (`$O1_REPO/opentofu/vr1-dc0-substrate`, `.../vr1-dc1-substrate`, provider = `qemu+ssh`), their state files, the `maas` CLI, and every `virsh -c qemu+ssh://...` into a containment VM. MEASURE `$O1_REPO` with `ls` on the headend -- do not assume it, and do not assume the inner state lives on vcloud (it does not). --- ## Two paths, by intent (mirrors D-061's release-vs-destroy split) - **Path A -- scoped teardown.** Tear down ONE site's resources (a single DC, or Office1), leaving the other sites' infrastructure untouched. **Scope by choosing the ROOT first** -- each DC substrate root contains exactly one site, so a plain `tofu destroy` there is already scoped and needs no `-target` at all. `-target` is reserved for the OUTER root (which holds all three sites) and for a partial teardown inside one root. Where you do use it, HashiCorp's own docs flag it as an exceptional-circumstances tool, not for routine use, because targeted applies can leave configuration and state silently diverging: *"This targeting capability is provided for exceptional circumstances, such as recovering from mistakes or working around Terraform limitations. It is not recommended to use `-target` for routine operations, since this can lead to undetected configuration drift"* (developer.hashicorp.com/terraform/cli/commands/plan, fetched 2026-07-10). A scoped teardown IS exactly that kind of exceptional circumstance -- acceptable here, but ALWAYS follow a targeted apply with a full (untargeted) `tofu plan` afterward (Step 5) to confirm the rest of the configuration didn't drift. - **Path B -- full VR1 teardown.** Abandon everything OpenTofu manages for VR1. That is now THREE roots across TWO hosts -- both DC substrate roots from the Office1 headend, then the outer root on vcloud -- each a plain `tofu destroy` (no `-target`), letting OpenTofu compute the safe order WITHIN each root from its own resource graph. The order BETWEEN roots is yours to get right: inner before outer, always. Both paths share Steps 1 (state backup), 2 (the MAAS machine census) and 5 (final verify). Path A is Steps 2-5; Path B replaces Steps 3-5 with the per-root destroys described in its own section (see "Path B" section after Step 5). --- ## Step 1 -- Back up the state file before ANY destroy [do this every time] A `destroy` mutates state exactly like an `apply` does -- if it's interrupted partway (network drop, `Ctrl-C`, host reboot), your only way to know what's actually still live is the state file as it stood before you started. **Back up the state of the ROOT YOU ARE ABOUT TO DESTROY, on the host that root executes from.** There is no single state file any more: the OUTER root runs on vcloud (D-128 Plane 1, `libvirt_uri = qemu:///system`) and each DC's INNER root runs FROM THE OFFICE1 HEADEND (D-128 Plane 2, provider = `qemu+ssh`), which is where its state lives -- `docs/CURRENT-STATE.md` records both DCs' inner tfstate as held inside the voffice1 working tree, gitignored, sha256-pinned. A teardown that backed up only `opentofu/terraform.tfstate` on vcloud has NOT backed up the substrate it is about to destroy. ```bash # OUTER root -- on vcloud: cd "$REPO/opentofu" cp -a terraform.tfstate "terraform.tfstate.pre-teardown-$(date +%Y%m%d-%H%M%S)" # INNER root -- from the OFFICE1 HEADEND, for the DC being torn down: cd "$O1_REPO/opentofu/vr1-dc1-substrate" # or vr1-dc0-substrate cp -a terraform.tfstate "terraform.tfstate.pre-teardown-$(date +%Y%m%d-%H%M%S)" ``` Confirm each copy exists and is non-empty before continuing -- a `cp` whose source was absent is the same silence as a clean backup. Store the copy out-of-band per the operator's existing `~/vault-init/`-class secret-handling process (it carries the same plaintext-credential risk as the live file -- see `opentofu/README.md` "State file handling", DOCFIX-175) -- do not just leave it sitting next to the live file as your only copy. --- ## Step 2 -- MAAS-side FIRST: census the MACHINE records this site owns [do this every time] **DO NOT READ A `vm_host` HERE. IT IS A PERMANENT FALSE CLEAR.** The VR1 DCs deliberately have no MAAS pod (see "Relationship to D-061" above): a `maas vm-hosts read` returns nothing on an empty DC and on a DC with a fully commissioned fleet ALIKE. The version of this step that keyed "skip to Step 3, there is nothing to clean up" off that emptiness was routing an operator straight at destroying a containment VM holding live MAAS machine records. **The check is at the MACHINE level.** **Why associations get read BEFORE any destroy, even though the pod mechanism does not apply here.** MAAS deletes every machine record LINKED to a pod when the pod is deleted -- no warning, no decompose flag -- and on 2026-07-21 an association check that ran too late cost 9 machine records (`runbooks/appendix-A-troubleshooting.md`, symptom "`maas ... vm-host delete ` (or pod delete) silently REMOVES enlisted machines"; capture `docs/audit/incident-20260721-pod-delete-cascade.txt`). The mechanism is gone; the discipline it bought is not. **Read the associations first, and a NON-EMPTY result is a STOP.** **WHERE:** from the OFFICE1 HEADEND (D-128 Plane 2 -- the region MAAS and its CLI profile live there). MEASURED on vcloud 2026-07-29: `command -v maas` returns nothing. An ABSENT client is not an unreachable service and is never a clear -- this deployment has misdiagnosed that three times (`scripts/maas-role-tags.sh` makes the same split explicitly). **LENS 1 -- enumerate what EXISTS (never only what is declared).** ```bash maas "${MAAS_PROFILE:-admin}" machines read > /tmp/maas-machines.json \ || { echo "REFUSE: cannot read MAAS machines -- 'could not look' is NOT 'nothing there'" >&2; exit 1; } python3 - /tmp/maas-machines.json <<'PY' import json, sys ms = json.load(open(sys.argv[1])) virsh = [m for m in ms if (m.get("power_type") or "") == "virsh"] for m in ms: print(" %-10s %-18s %-14s power=%s" % ( m["system_id"], m["hostname"], m.get("status_name"), m.get("power_type") or "-")) print("total MAAS machine records: %d" % len(ms)) print("RECORDS REQUIRING ATTRIBUTION (power_type=virsh): %d" % len(virsh)) PY ``` **Write that second number down, and attribute every one of those records before the gate passes.** It is a COUNT, not a sentence, precisely so it cannot be skimmed: each `power_type=virsh` record is powered through SOME containment VM's inner libvirt, and a destroy that has not accounted for all of them is a destroy with an unknown blast radius. (`power_type` does come back from `machines read` -- shipped repo code reads that exact field, `scripts/maas-node-power.sh`.) Attribution needs that machine's power ADDRESS, which is a per-machine parameter: read it with `maas "${MAAS_PROFILE:-admin}" machine read ` and take the FIELD NAME from MAAS's own current docs -- do not invent one. Compare what you find against this site's URI from `scripts/lib-hosts.sh` (`VIRSH_POWER_ADDRESS`), never from memory. **If you cannot obtain the power address for a `virsh`-powered record, that is a REFUSAL, not a clear.** **LENS 2 -- corroborate against this site's pinned roster.** ```bash # on the OFFICE1 HEADEND, standing in that host's clone: O1_REPO="$(git rev-parse --show-toplevel)" \ || { echo "REFUSE: not inside the Office1 clone -- cd there first" >&2; exit 1; } SITE=vr1-dc1 # the site being torn down: vr1-dc0 | vr1-dc1 source "$O1_REPO/scripts/lib-hosts.sh" lib_hosts_select_dc "$SITE" || exit 1 : > /tmp/site-boot-macs.txt for h in "${HOSTS[@]}"; do printf '%s\n' "${HOST_BOOT_MAC[$h]}" >> /tmp/site-boot-macs.txt; done python3 - /tmp/maas-machines.json /tmp/site-boot-macs.txt <<'PY' import json, sys ms = json.load(open(sys.argv[1])) want = {l.strip().lower() for l in open(sys.argv[2]) if l.strip()} if not want: print("REFUSE: no pinned boot MACs for this site -- refusing to read that as zero") sys.exit(1) hits = [] for m in ms: macs = {(i.get("mac_address") or "").lower() for i in m.get("interface_set", [])} if macs & want: hits.append((m["system_id"], m["hostname"], m.get("status_name"), m.get("power_type") or "-")) print("ASSOCIATED MAAS MACHINE RECORDS for %d pinned MACs: %d" % (len(want), len(hits))) for h in hits: print(" %-10s %-18s %-14s power=%s" % h) sys.exit(1 if hits else 0) PY ``` Exit 1 means records EXIST for this site. That is the STOP condition, not an error to work around. Two properties of LENS 2 to hold in mind. `lib_hosts_select_dc` REFUSES a second, DIFFERENT site in the same shell (its stale-value guard) -- tear down one site per shell. And it enumerates only the hosts lib-hosts DECLARES, so it cannot report a record it has no row for (a canary enlistment, a re-enlisted node MAAS re-named, anything added since): that is exactly why LENS 1 runs and why it is not optional. **If records DO exist and you intend to lose them:** that is a deliberate, individually operator-approved decision, never a cleanup side effect. Release or delete them via MAAS's own current documented flow FIRST, then re-run both lenses. This repo has not exercised that path for real; MAAS's own docs are authoritative over anything asserted here. If records are already gone while the domains still exist, appendix-A carries the measured recovery (power the domains on, PXE re-enlist on the pinned MACs, `scripts/maas-node-power.sh` dry then `--commit`, re-commission -- and note MAAS mints NEW hostnames). **GATE (all three, or STOP):** 1. LENS 1 READ SUCCEEDED, and every `power_type=virsh` record in it is attributed to a known containment VM. 2. LENS 2 returns ZERO associated records for the site being torn down. 3. Neither lens was skipped because a command failed. A failure is a refusal. --- ## Step 3 (Path A only) -- Pick the ROOT first, then plan the destroy inside it **Scoping a single DC is now a ROOT choice, not a `-target` choice.** Under D-123 Model B each DC's substrate is its own root with its own state file, and each of those roots contains exactly ONE site -- so a plain `tofu destroy` in `opentofu/vr1-dc1-substrate/` is already scoped to vr1-dc1, with none of the silent-divergence risk HashiCorp's targeting warning describes. Reach for `-target` only for a PARTIAL teardown inside one root, or in the OUTER root, which holds all three sites at once. Module names MEASURED 2026-07-29 (`grep -n '^module' /main.tf`). **There is no `dc1_*` module in either root** -- those names are pre-D-119 and never existed here; a `-target` naming one aborts the plan. | Where it lives | Root | Modules (vr1-dc1 shown; vr1-dc0 is identical with `dc0` substituted) | |---|---|---| | INSIDE the DC | `opentofu/vr1-dc1-substrate/` | `inner_storage`, `vr1_dc1_planes`, `vr1_dc1_wan`, `vr1_dc1_opnsense`, `vr1_dc1_node` | | At vcloud | `opentofu/` (outer) | `vr1_dc1_storage`, `vr1_dc1_uplink`, `vvr1_dc1` | `module.vr1_dc1_node` is a `for_each` module keyed by node name -- targeting the module covers every instance; there are no `_node_01`/`_node_02` siblings to enumerate. There is no `maas_vm_host` module in either root to target (Step 2). `inner_storage` is named identically in BOTH substrate roots -- it is disambiguated by which root you are standing in, which is one more reason to confirm the directory before every command. **Order: INNER root first, then OUTER.** Destroying `module.vvr1_dcN` first leaves the inner state describing objects that no longer exist and removes the only path the inner provider can dial. INNER root -- run FROM THE OFFICE1 HEADEND (D-128 Plane 2; the provider is `qemu+ssh` and the state lives there). For the WHOLE DC substrate, no `-target` at all: ```bash cd "$O1_REPO/opentofu/vr1-dc1-substrate" tofu plan -destroy -out=teardown-vr1-dc1-inner.tfplan ``` For a PARTIAL inner teardown, target the real names: ```bash tofu plan -destroy \ -target=module.vr1_dc1_node \ -target=module.vr1_dc1_opnsense \ -target=module.vr1_dc1_wan \ -target=module.vr1_dc1_planes \ -target=module.inner_storage \ -out=teardown-vr1-dc1-inner.tfplan ``` OUTER root -- on vcloud, and ONLY after the inner half is done: ```bash cd "$REPO/opentofu" tofu plan -destroy \ -target=module.vvr1_dc1 \ -target=module.vr1_dc1_uplink \ -target=module.vr1_dc1_storage \ -out=teardown-vr1-dc1-outer.tfplan ``` **Do NOT target the mesh-link modules** (`mesh_vr1_dc0_vr1_dc1`, `mesh_vr1_dc0_office1`, `mesh_vr1_dc1_office1`) or `netem_vr1_dc0_vr1_dc1` in a single-site teardown -- each leg is SHARED with the site at its far end, and the dc0<->office1 leg additionally CARRIES the live rack<->region transit (MAAS, node DNS, the inner root's own `qemu+ssh` path). Only include a leg once BOTH sites it connects are going away (see the mesh-link note after Step 5). Review the plan line by line: expect destroys for exactly the resources above (`libvirt_domain`, `libvirt_volume`, `libvirt_network` x6 for the planes plus the WAN bridge network, `libvirt_pool`) and NOTHING outside this site's scope -- in particular confirm no `office1_*`, no other DC's `vr1_dc0_*`/`vr1_dc1_*`, and no `mesh_*`/`netem_*` resource appears. **CAPTURE THE PRE-DESTROY BASELINE NOW**, while the containment VM is still up -- Step 4 cannot tell "destroyed cleanly" from "could not look" without it: ```bash source "$O1_REPO/scripts/lib-hosts.sh"; lib_hosts_select_dc vr1-dc1 # -> VIRSH_POWER_ADDRESS virsh -c "$VIRSH_POWER_ADDRESS" list --all > /tmp/pre-teardown-inner-domains.txt virsh -c "$VIRSH_POWER_ADDRESS" net-list --all > /tmp/pre-teardown-inner-nets.txt virsh -c "$VIRSH_POWER_ADDRESS" pool-list --all > /tmp/pre-teardown-inner-pools.txt ``` **GATE:** plan matches this scope exactly; no unexpected destroys outside the targeted site; the baseline capture above is non-empty. Do not apply a plan you have not read. --- ## Step 4 (Path A only) -- Apply the scoped destroy [MUTATION: gated] Apply the INNER plan from the Office1 headend, in the inner root; apply the OUTER plan on vcloud, in `$REPO/opentofu`. Same order as Step 3: inner, then outer. ```bash tofu destroy teardown-vr1-dc1-inner.tfplan # inner root, from the Office1 headend tofu destroy teardown-vr1-dc1-outer.tfplan # outer root, on vcloud -- AFTER the inner verify below ``` *(Note: unlike a create/apply plan, a `-destroy`-mode plan file is applied the same way -- `tofu apply ` and `tofu destroy ` are equivalent once the plan itself was generated with `-destroy`; use whichever your `tofu` version's docs confirm for a plan file specifically, and if in doubt, prefer re-running with `tofu destroy` directly rather than guessing.)* Confirm this is the exact reviewed plan file from Step 3 (not re-planned) before running. **VERIFY -- and note WHICH HOST each half is verified from.** The DC's node, edge and plane objects live inside the containment VM, so a local `virsh` on vcloud can never see them: MEASURED on vcloud 2026-07-29, `virsh list --all` returns exactly `vvr1-dc0`, `vvr1-dc1`, `voffice1`, `office1-opnsense`. A vcloud-local `grep -i dc1` therefore "passes" against a fully intact DC. This is the D-128 split: Plane 1 = the outer `qemu:///system` on vcloud; the inner substrate is reached over `qemu+ssh` from the Office1 headend. INNER half -- from the Office1 headend, BEFORE the outer destroy (afterwards the URI cannot answer BY DESIGN): ```bash source "$O1_REPO/scripts/lib-hosts.sh"; lib_hosts_select_dc vr1-dc1 # -> VIRSH_POWER_ADDRESS virsh -c "$VIRSH_POWER_ADDRESS" version >/dev/null 2>&1 || { echo "REFUSE: cannot reach $VIRSH_POWER_ADDRESS. An empty list from an unreachable" >&2 echo " URI, a stopped containment VM or a bad key is NOT a clean destroy." >&2 exit 1; } virsh -c "$VIRSH_POWER_ADDRESS" list --all virsh -c "$VIRSH_POWER_ADDRESS" net-list --all virsh -c "$VIRSH_POWER_ADDRESS" pool-list --all ``` Take the URI from `scripts/lib-hosts.sh`, never from memory. Diff each against the Step 3 baseline: expect exactly the planned objects gone and NOTHING else changed. An empty list with no baseline to compare it to proves nothing. OUTER half -- on vcloud, after the outer destroy. Assert on the named objects, not on a substring match: ```bash virsh list --all # expect vvr1-dc1 GONE; vvr1-dc0, voffice1, office1-opnsense untouched virsh net-list --all # expect vr1-dc1-uplink GONE; the three mesh-* legs untouched virsh pool-list --all # expect vr1-dc1-pool GONE; vr1-dc0-pool, office1-pool, default untouched ``` (Live names measured 2026-07-29. Never `grep -i dc1` here: it matches `vr1-dc1` AND anything else carrying the string, and it cannot distinguish "absent" from "never visible from this host".) --- ## Step 5 (Path A only) -- Confirm no drift outside the targeted scope ```bash cd "$REPO/opentofu" tofu plan ``` Run WITHOUT `-target` this time -- this is the check HashiCorp's own targeting warning (Step 3) calls for: confirm the rest of the configuration (Office1, the OTHER DC, the mesh legs, netem) shows no unexpected changes. Expect either "No changes" for everything outside the torn-down site, or exactly the changes you independently expect for other in-flight work -- nothing attributable to this teardown. Do the same in the SURVIVING DC's inner root if you used `-target` anywhere; a targeted apply's drift does not announce itself. **GATE:** untargeted plan shows zero unexpected drift outside the torn-down site. --- ## Path B -- full VR1 teardown (replaces Steps 3-5 above) Do Step 1 (back up EVERY root's state -- outer plus BOTH inner roots) and Step 2 (the machine census, run for EVERY site, not just one) first, then destroy BOTH inner roots before the outer one -- "everything" is now three roots on two hosts, not one `cd opentofu`: ```bash # from the OFFICE1 HEADEND, once per DC substrate root: cd "$O1_REPO/opentofu/vr1-dc0-substrate" && tofu plan -destroy -out=teardown-vr1-dc0-inner.tfplan cd "$O1_REPO/opentofu/vr1-dc1-substrate" && tofu plan -destroy -out=teardown-vr1-dc1-inner.tfplan # then, on vcloud: cd "$REPO/opentofu" tofu plan -destroy -out=teardown-full.tfplan ``` Review: expect a destroy for every resource each tree currently manages (check against that root's actual uncommented module list at teardown time -- it grows as more stages get executed, so what "everything" means changes over the buildout, unlike Path A's fixed per-site scope). **GATE:** each plan matches its root's full current scope exactly. ```bash tofu destroy teardown-full.tfplan ``` **VERIFY.** Verify each inner root from the Office1 headend BEFORE the outer destroy, using Step 4's `virsh -c "$VIRSH_POWER_ADDRESS" version` refusal guard -- once the containment VMs are gone the inner URIs cannot answer, and "unreachable" is not evidence of anything. Then, on vcloud: ```bash virsh list --all virsh net-list --all virsh pool-list --all ``` Expect: nothing VR1-related remains (VR0's own testcloud resources, which OpenTofu does not manage, are untouched by any of this). Note that vcloud's `default` pool and `virbr0`/`wan` are not VR1 objects -- their survival is correct, not residue. Then do the equivalent of Step 5 (a final untargeted `tofu plan`) to confirm the destroy converged state to fully empty, not partially. --- ## Mesh-link teardown (either path, handle separately) The three D-100 mesh-link legs live in the OUTER root. Module names and the libvirt networks they create, MEASURED 2026-07-29 (`grep -n '^module' opentofu/main.tf`; `virsh net-list --all`): | Module (outer root) | libvirt network | bridge, measured 2026-07-29 | |---|---|---| | `mesh_vr1_dc0_vr1_dc1` | `mesh-vr1-dc0-vr1-dc1` | `virbr5` | | `mesh_vr1_dc0_office1` | `mesh-vr1-dc0-office1` | `virbr7` | | `mesh_vr1_dc1_office1` | `mesh-vr1-dc1-office1` | `virbr3` | (The pre-D-119 names `mesh_dc1_dc2` / `mesh_dc1_office1` / `mesh_dc2_office1` that this section used to carry exist nowhere -- `opentofu/main.tf` holds `moved` blocks recording exactly that rename. `virbrN` is a drifting libvirt assignment: re-measure with `virsh net-info ` before using one, never carry it from here.) Each leg is shared between two sites. Destroy a leg only when BOTH its endpoints are permanently going away (a full VR1 teardown, or abandoning both of its sites together) -- never as part of a single-site scoped teardown, and never as a reflex "clean up everything DC-adjacent" action. **`mesh-vr1-dc0- office1` deserves extra care:** it carries the live rack<->region transit, so MAAS, node DNS and the inner root's own `qemu+ssh` path all die with it. `modules/netem-link` IS wired now -- `module "netem_vr1_dc0_vr1_dc1"` in the outer root, applied 2026-07-21 against `virbr5` (the zero-traffic dc0<->dc1 leg). It has a correct `destroy`-time provisioner (`provisioner "local-exec" { when = destroy; ... tc qdisc del ... }`, confirmed by reading `opentofu/modules/netem-link/main.tf` directly), so destroying the corresponding `terraform_data.netem` resource cleans up the real `tc qdisc` rule on the vcloud host automatically; no extra manual step for that piece. Destroy it BEFORE the leg it is attached to, and note it is a separate module from the leg -- destroying `mesh_vr1_dc0_vr1_dc1` alone leaves the netem resource pointing at a bridge that no longer exists. --- ## Rollback decision tree -- "a stage's `tofu apply` failed partway through, now what?" **Read this before reaching for `tofu destroy` at all.** OpenTofu is designed for idempotent convergence -- in most failure modes, fixing the root cause and re-running the SAME `tofu apply` is correct and safer than tearing down and starting over, because it preserves everything that DID apply successfully and only retries what didn't. **Question 0 -- WHICH ROOT failed?** There are three: the OUTER root on vcloud, and one INNER substrate root per DC, run from the Office1 headend. The answer decides which state file, which host, and which `virsh` endpoint every branch below is talking about. Answer it before anything else. **Question 0b -- is anything DEPLOYED on top?** From Stage 5 onward a failed `tofu apply` can sit under live MAAS machine records and a live Juju model. **No branch below reaches a destroy without Step 2's machine census passing first.** If Step 2 returns records, the destroy is not the next action -- resolving the records is, and that is an operator-gated decision on its own. **Verb discipline, because these read alike under time pressure:** `virsh destroy vvr1-dcN` is a forced POWER-OFF, reversible with `virsh start`, and it leaves the domain, its disk and every MAAS record intact -- that is the D-122 site-down lever and it is a legitimate first move. `tofu destroy` of `module.vvr1_dcN` UNDEFINES the domain and DELETES its disk, taking the whole inner fleet with it, irreversibly. Reaching for the second when you meant the first is the expensive mistake this runbook exists to prevent. 1. **`tofu apply` errored before creating anything (e.g., `tofu init`/ `validate`/`plan` failed, or `apply` failed on its very first resource).** Nothing was created. Fix the root cause (bad variable, unreachable provider, syntax error) and re-run from that stage's own `tofu plan` step. No teardown needed. 2. **`tofu apply` created SOME resources, then failed on a later one.** `tofu plan` (no `-destroy`) immediately to see OpenTofu's own view of what's now live vs. what the config wants. Default to FIX-FORWARD: fix whatever caused the failure (a bad node-VM memory size, an unreachable MAAS endpoint, a real infra problem) and re-run `tofu apply` for that same stage -- OpenTofu will create only what's still missing, leaving successfully-created resources alone. Reserve a teardown (Path A, scoped to that site) for when the partial state is itself broken in a way fix-forward can't reach (e.g., a domain stuck in a bad libvirt state that manual `virsh` intervention already tried and failed to fix). 3. **You suspect `terraform.tfstate` itself is corrupted or lost** (this is the scenario `opentofu/README.md`'s "State file handling" section already flags as the real cost of no backend/backup). There is no clean recovery path documented in this repo: either a manual `tofu import` per still-live resource (tedious, must be done for every domain/network/pool by hand) or accepting a full teardown-and-rebuild from Step 1 of the affected stage. This is exactly why Step 1's pre-teardown backup habit matters even outside a deliberate teardown -- back up `terraform.tfstate` after every successful stage apply too, not only before a teardown. 4. **A resource exists in reality but OpenTofu's state doesn't know about it** (e.g., a `virt-install`-created VM from Stage 2's Option B manual path, which OpenTofu never created and cannot destroy). This runbook's `tofu destroy`/`plan -destroy` commands only ever touch what OpenTofu itself manages -- anything created via a manual/Option-B path needs manual teardown (`virsh destroy`/`undefine`, etc.) by the same operator discipline that created it. Do not expect `tofu destroy` to find or remove it. 5. **The failure is INSIDE a DC and you are tempted to reach for the containment VM.** Taking the site down (`virsh destroy vvr1-dcN`) is reversible and is often the right move to stop a cascade. Destroying the containment VM through the OUTER root is not a rollback of the inner apply -- it discards the entire inner substrate, leaves the inner state file describing objects that no longer exist, and leaves MAAS holding machine records whose `power_address` will never answer again. Fix-forward in the INNER root first; if you truly need the DC gone, do Step 2, then the inner root, then the outer -- in that order. --- ## Verification (of this runbook's own accuracy, not a live run) Every `tofu`/`terraform` flag cited above (`-destroy`, `-target`, the targeting-risk warning) was checked against HashiCorp's own current CLI documentation before being written here (fetched 2026-07-10), not asserted from memory -- same discipline as `opentofu/README.md`'s other research sections. `netem-link`'s destroy-time provisioner was confirmed by reading the actual `.tf` file, not assumed. **CORRECTION 2026-07-29:** this paragraph previously vouched for "`dc1_maas_vm_host`'s lack of a Terraform-expressible dependency on node-VM domains" as something confirmed by reading the `.tf` files. No such module has ever existed in either root, so that was a verification claim about nothing -- the false-verification class that lets a wrong instruction look checked. All module, network, pool and domain names in this runbook were RE-MEASURED against the tree and the live hosts on 2026-07-29 (`grep -n '^module' /main.tf`; `virsh list/net-list/pool-list --all`; `virsh net-dumpxml`). What is NOT verified: this runbook's commands have never been run against a real `tofu` binary or a real state file (same SCAFFOLD/UNVALIDATED status as the rest of `opentofu/`) -- treat every command here as reviewed-but-unexercised until `scripts/opentofu-validate.sh` and a real dry run confirm otherwise.