VR1 DC-DC teardown / rollback runbook

Scope: tearing down or rolling back the infrastructure opentofu/ provisions for the VR1 (DC-DC) buildout -- the vr1-dc0 / vr1-dc1 / Office1 plane and local networks, storage pools, node-VM domains, containment VMs, OPNsense edge VMs, and the D-100 mesh-link triangle. (It once also claimed MAAS vm_host registrations; there are none -- see Step 2. The dc1/dc2 names it was written with are pre-D-119 and mean different datacenters now.) Written 2026-07-10 (DOCFIX-175 gap-register item #19) because nothing else in this repo covers this layer: runbooks/phase-00-teardown-release.sh / phase-00-teardown-destroy.sh (D-061) tear down the VR0 juju/MAAS-machine layer, a different pairing of tools and a different failure mode entirely -- see "Relationship to D-061" below for why that mechanism does not just transfer over.

UNVALIDATED, same caveat as everywhere else in opentofu/: no tofu binary has been available in any session that authored this tree. Every command below is standard, well-documented OpenTofu/Terraform CLI syntax (verified against HashiCorp's own current docs where cited), not invented -- but run bash scripts/opentofu-validate.sh on a machine with the real binary and read the result before trusting any of it against a live state file.

!!! MODEL B RESHAPE (D-123, 2026-07-16) -- READ BEFORE ANY DC TEARDOWN; this runbook's body predates it !!!

This runbook was written (2026-07-10) for the SINGLE-ROOT shape (OpenTofu creates the DC node-VM domains / planes / edge directly on the vcloud host, and maas-vm-host registers the vcloud host). Under D-123 Model B that is no longer how a DC is built, so the resource-by-resource mechanics below are partly WRONG for the DCs. Current shape:

• TWO roots / TWO state files PER DC, and there are now TWO DCs. The OUTER root (opentofu/, vcloud) owns vvr1-dc0 and vvr1-dc1 (the DC containment VMs), the office1<->dcN transits, and the D-125 vcloud ISP uplinks (vr1-dc0-uplink / vr1-dc1-uplink NAT + each containment VM's uplink NIC / br-vr1-dcN-wan netplan bridge). Each INNER root (opentofu/vr1-dc0-substrate/, opentofu/vr1-dc1-substrate/; provider = qemu+ssh to that DC's containment VM) owns everything INSIDE its DC -- the node VMs, the 6 planes, vr1-dcN-wan, the OPNsense edge, the inner pool. So the "node-VM domains / planes / edge" this runbook talks about live on the containment VM's libvirt, not vcloud's.
• There is NO MAAS vm_host/pod for any VR1 DC. MAAS holds per-MACHINE records with power_type=virsh (D-103/D-123 amendments, ruled 2026-07-20) and modules/maas-vm-host is instantiated in NEITHER root -- measured 2026-07-29. Any instruction anywhere to "remove the maas-vm-host record before destroying the containment VM" is VOID: there is no such record, the read returns nothing on a DC with a full fleet enrolled, and acting on that emptiness destroys the containment VM out from under live MAAS machine records. Step 2 is the real check.
• Whole-DC site-down is ONE object: virsh destroy vvr1-dcN (the D-122/D-123 point). Destroying the containment VM instantly takes down the entire inner fleet -- no per-domain group destroy needed. That is the fast path for "abandon this DC and rebuild." Know which verb you are using: virsh destroy is a forced POWER-OFF and is REVERSIBLE (virsh start vvr1-dcN); the domain, its disk and every MAAS record still exist. tofu destroy of module.vvr1_dcN UNDEFINES the domain and deletes its disk -- irreversible, and it takes the whole inner fleet with it. Under a failed Stage 5 those are not interchangeable.
• Teardown ORDER (two roots): tear down the INNER root first (cd opentofu/vr1-dcN-substrate && tofu destroy, or just virsh destroy vvr1-dcN if you are only taking the site DOWN), THEN the OUTER root. The D-061-style "clean up MAAS's record before destroying the libvirt underneath" principle below still applies -- and with no pod to remove, "MAAS's record" means the per-machine records enumerated in Step 2. Run Step 2 BEFORE either root's destroy.
• Reverting Model B -> Model A (a different operation from teardown) is docs/archive/model-a-fallback-plan.md (git tag model-a-fallback), not this runbook.

Until the body below is rewritten to the two-root shape, treat its DC-substrate steps as single-root reference and apply the mapping above. The Office1 (voffice1) and mesh/pool steps are unaffected.

When to use this runbook

• Mid-buildout failure recovery. A stage's tofu apply failed partway through, or created something wrong, and you need to know whether to fix- forward or tear down and retry. See "Rollback decision tree" below -- READ THAT FIRST, since fix-forward is usually the right answer, not a destroy.
• Deliberate re-run from clean. You want to abandon a DC's (or Office1's, or the whole VR1 layer's) current state and rebuild from Step 1 of its stage runbook.
• NOT for: tearing down VR0's juju/MAAS-machine layer (use D-061's scripts), or removing a single juju application/relation from an already- deployed OpenStack bundle (that is normal juju operations, not this runbook).

Relationship to D-061 (read before mixing the two layers)

D-061's core lesson: destroying/releasing a resource at the WRONG layer, in the wrong order, orphans state at the OTHER layer -- juju destroy-model released MAAS-pod-composed machines and MAAS auto-decomposed them, losing the carve; the fix was remove-machine --keep-instance FIRST (detach from juju without releasing), verify survival in MAAS, THEN act on the model.

The same PRINCIPLE applies here, but the mechanism is different because the layers being coordinated are different:

• D-061 coordinates juju's machine view against MAAS's machine view of hosts MAAS itself pod-composed.
• This runbook coordinates OpenTofu's resource view (libvirt domains/ networks/pools it created directly) against MAAS's per-MACHINE view of those same domains. MEASURED 2026-07-29: modules/maas-vm-host is instantiated in NEITHER root (grep -n 'maas_vm_host' opentofu/*.tf opentofu/vr1-dc0-substrate/*.tf opentofu/vr1-dc1-substrate/*.tf returns only comments), and the provider "maas" block those modules would need is still deliberately absent (opentofu/main.tf:9-20, DOCFIX-179). MAAS knows the DC node VMs because they PXE-ENLISTED, and it powers them per machine with power_type=virsh (D-103/D-123 amendments, ruled 2026-07-20 -- scripts/maas-node-power.sh's header carries the two measurements that refuted the pod).

Because this repo uses neither MAAS pod-composition NOR a vm_host for its OpenTofu-created VMs, the SPECIFIC --keep-instance decompose-on-release failure mode D-061 diagnosed does not directly apply here. The GENERAL principle still does: clean up MAAS's record of a host before destroying the libvirt resource underneath it, not after -- destroying the libvirt domain/ host first would leave MAAS pointing at a power_address that no longer answers, an orphaned-but-not-obviously-broken record rather than a clean removal.

What this section used to say, and why it is now a hazard rather than a caveat: it stated that no VR1 DC had reached Stage 4, so no VR1 libvirt host had ever had real MAAS-enrolled machines under it. That is false as of 2026-07-23 (docs/CURRENT-STATE.md: 18 nodes READY, 9 per DC, both DCs commissioned; power_type=virsh set and proven by a real query-power-state), and the per-DC Juju controller VMs were enlisted on top of that in July 2026. Anything downstream of "nothing is enrolled yet" -- including the old Step 2 -- must be treated as REVERSED, not merely stale. The exact MAAS-side release/delete commands for a machine whose libvirt domain is about to disappear are STILL not worked out or tested in this repo: Step 2 states the principle, enumerates the records, and points at MAAS's own current docs for the flags. Do not invent flags for a scenario this repo has never actually exercised.

TWO CLONES, TWO HOSTS -- set these once, before anything below

Every block in this runbook runs on one of two hosts (D-128), each with its own clone of this repo. The same cd pasted on the wrong one is the foot-gun this section exists to remove, so the two are named differently throughout:

• $REPO -- the VCLOUD clone. Plane 1: the OUTER root ($REPO/opentofu, libvirt_uri = qemu:///system) and local virsh.
• $O1_REPO -- the OFFICE1 HEADEND clone, on voffice1. Plane 2: BOTH DC substrate roots ($O1_REPO/opentofu/vr1-dc0-substrate, .../vr1-dc1-substrate, provider = qemu+ssh), their state files, the maas CLI, and every virsh -c qemu+ssh://... into a containment VM.

MEASURE $O1_REPO with ls on the headend -- do not assume it, and do not assume the inner state lives on vcloud (it does not).

Two paths, by intent (mirrors D-061's release-vs-destroy split)

• Path A -- scoped teardown. Tear down ONE site's resources (a single DC, or Office1), leaving the other sites' infrastructure untouched. Scope by choosing the ROOT first -- each DC substrate root contains exactly one site, so a plain tofu destroy there is already scoped and needs no -target at all. -target is reserved for the OUTER root (which holds all three sites) and for a partial teardown inside one root. Where you do use it, HashiCorp's own docs flag it as an exceptional-circumstances tool, not for routine use, because targeted applies can leave configuration and state silently diverging: "This targeting capability is provided for exceptional circumstances, such as recovering from mistakes or working around Terraform limitations. It is not recommended to use -target for routine operations, since this can lead to undetected configuration drift" (developer.hashicorp.com/terraform/cli/commands/plan, fetched 2026-07-10). A scoped teardown IS exactly that kind of exceptional circumstance -- acceptable here, but ALWAYS follow a targeted apply with a full (untargeted) tofu plan afterward (Step 5) to confirm the rest of the configuration didn't drift.
• Path B -- full VR1 teardown. Abandon everything OpenTofu manages for VR1. That is now THREE roots across TWO hosts -- both DC substrate roots from the Office1 headend, then the outer root on vcloud -- each a plain tofu destroy (no -target), letting OpenTofu compute the safe order WITHIN each root from its own resource graph. The order BETWEEN roots is yours to get right: inner before outer, always.

Both paths share Steps 1 (state backup), 2 (the MAAS machine census) and 5 (final verify). Path A is Steps 2-5; Path B replaces Steps 3-5 with the per-root destroys described in its own section (see "Path B" section after Step 5).

Step 1 -- Back up the state file before ANY destroy [do this every time]

A destroy mutates state exactly like an apply does -- if it's interrupted partway (network drop, Ctrl-C, host reboot), your only way to know what's actually still live is the state file as it stood before you started.

Back up the state of the ROOT YOU ARE ABOUT TO DESTROY, on the host that root executes from. There is no single state file any more: the OUTER root runs on vcloud (D-128 Plane 1, libvirt_uri = qemu:///system) and each DC's INNER root runs FROM THE OFFICE1 HEADEND (D-128 Plane 2, provider = qemu+ssh), which is where its state lives -- docs/CURRENT-STATE.md records both DCs' inner tfstate as held inside the voffice1 working tree, gitignored, sha256-pinned. A teardown that backed up only opentofu/terraform.tfstate on vcloud has NOT backed up the substrate it is about to destroy.

# OUTER root -- on vcloud:
cd "$REPO/opentofu"
cp -a terraform.tfstate "terraform.tfstate.pre-teardown-$(date +%Y%m%d-%H%M%S)"

# INNER root -- from the OFFICE1 HEADEND, for the DC being torn down:
cd "$O1_REPO/opentofu/vr1-dc1-substrate"      # or vr1-dc0-substrate
cp -a terraform.tfstate "terraform.tfstate.pre-teardown-$(date +%Y%m%d-%H%M%S)"

Confirm each copy exists and is non-empty before continuing -- a cp whose source was absent is the same silence as a clean backup.

Store the copy out-of-band per the operator's existing ~/vault-init/-class secret-handling process (it carries the same plaintext-credential risk as the live file -- see opentofu/README.md "State file handling", DOCFIX-175) -- do not just leave it sitting next to the live file as your only copy.

Step 2 -- MAAS-side FIRST: census the MACHINE records this site owns [do this every time]

DO NOT READ A vm_host HERE. IT IS A PERMANENT FALSE CLEAR. The VR1 DCs deliberately have no MAAS pod (see "Relationship to D-061" above): a maas <profile> vm-hosts read returns nothing on an empty DC and on a DC with a fully commissioned fleet ALIKE. The version of this step that keyed "skip to Step 3, there is nothing to clean up" off that emptiness was routing an operator straight at destroying a containment VM holding live MAAS machine records. The check is at the MACHINE level.

Why associations get read BEFORE any destroy, even though the pod mechanism does not apply here. MAAS deletes every machine record LINKED to a pod when the pod is deleted -- no warning, no decompose flag -- and on 2026-07-21 an association check that ran too late cost 9 machine records (runbooks/appendix-A-troubleshooting.md, symptom "`maas ... vm-host delete

(or pod delete) silently REMOVES enlisted machines"; capturedocs/audit/incident-20260721-pod-delete-cascade.txt`). The mechanism is gone; the discipline it bought is not. Read the associations first, and a NON-EMPTY result is a STOP.

WHERE: from the OFFICE1 HEADEND (D-128 Plane 2 -- the region MAAS and its CLI profile live there). MEASURED on vcloud 2026-07-29: command -v maas returns nothing. An ABSENT client is not an unreachable service and is never a clear -- this deployment has misdiagnosed that three times (scripts/maas-role-tags.sh makes the same split explicitly).

LENS 1 -- enumerate what EXISTS (never only what is declared).

maas "${MAAS_PROFILE:-admin}" machines read > /tmp/maas-machines.json \
  || { echo "REFUSE: cannot read MAAS machines -- 'could not look' is NOT 'nothing there'" >&2; exit 1; }

python3 - /tmp/maas-machines.json <<'PY'
import json, sys
ms = json.load(open(sys.argv[1]))
virsh = [m for m in ms if (m.get("power_type") or "") == "virsh"]
for m in ms:
    print("  %-10s %-18s %-14s power=%s" % (
        m["system_id"], m["hostname"], m.get("status_name"),
        m.get("power_type") or "-"))
print("total MAAS machine records: %d" % len(ms))
print("RECORDS REQUIRING ATTRIBUTION (power_type=virsh): %d" % len(virsh))
PY

Write that second number down, and attribute every one of those records before the gate passes. It is a COUNT, not a sentence, precisely so it cannot be skimmed: each power_type=virsh record is powered through SOME containment VM's inner libvirt, and a destroy that has not accounted for all of them is a destroy with an unknown blast radius. (power_type does come back from machines read -- shipped repo code reads that exact field, scripts/maas-node-power.sh.)

Attribution needs that machine's power ADDRESS, which is a per-machine parameter: read it with `maas "${MAAS_PROFILE:-admin}" machine read

and take the FIELD NAME from MAAS's own current docs -- do not invent one. Compare what you find against this site's URI fromscripts/lib-hosts.sh(VIRSH_POWER_ADDRESS), never from memory. **If you cannot obtain the power address for avirsh`-powered record, that is a REFUSAL, not a clear.**

LENS 2 -- corroborate against this site's pinned roster.

# on the OFFICE1 HEADEND, standing in that host's clone:
O1_REPO="$(git rev-parse --show-toplevel)" \
  || { echo "REFUSE: not inside the Office1 clone -- cd there first" >&2; exit 1; }
SITE=vr1-dc1                      # the site being torn down: vr1-dc0 | vr1-dc1
source "$O1_REPO/scripts/lib-hosts.sh"
lib_hosts_select_dc "$SITE" || exit 1
: > /tmp/site-boot-macs.txt
for h in "${HOSTS[@]}"; do printf '%s\n' "${HOST_BOOT_MAC[$h]}" >> /tmp/site-boot-macs.txt; done

python3 - /tmp/maas-machines.json /tmp/site-boot-macs.txt <<'PY'
import json, sys
ms   = json.load(open(sys.argv[1]))
want = {l.strip().lower() for l in open(sys.argv[2]) if l.strip()}
if not want:
    print("REFUSE: no pinned boot MACs for this site -- refusing to read that as zero")
    sys.exit(1)
hits = []
for m in ms:
    macs = {(i.get("mac_address") or "").lower() for i in m.get("interface_set", [])}
    if macs & want:
        hits.append((m["system_id"], m["hostname"], m.get("status_name"),
                     m.get("power_type") or "-"))
print("ASSOCIATED MAAS MACHINE RECORDS for %d pinned MACs: %d" % (len(want), len(hits)))
for h in hits:
    print("  %-10s %-18s %-14s power=%s" % h)
sys.exit(1 if hits else 0)
PY

Exit 1 means records EXIST for this site. That is the STOP condition, not an error to work around.

Two properties of LENS 2 to hold in mind. lib_hosts_select_dc REFUSES a second, DIFFERENT site in the same shell (its stale-value guard) -- tear down one site per shell. And it enumerates only the hosts lib-hosts DECLARES, so it cannot report a record it has no row for (a canary enlistment, a re-enlisted node MAAS re-named, anything added since): that is exactly why LENS 1 runs and why it is not optional.

If records DO exist and you intend to lose them: that is a deliberate, individually operator-approved decision, never a cleanup side effect. Release or delete them via MAAS's own current documented flow FIRST, then re-run both lenses. This repo has not exercised that path for real; MAAS's own docs are authoritative over anything asserted here. If records are already gone while the domains still exist, appendix-A carries the measured recovery (power the domains on, PXE re-enlist on the pinned MACs, scripts/maas-node-power.sh dry then --commit, re-commission -- and note MAAS mints NEW hostnames).

GATE (all three, or STOP):

1. LENS 1 READ SUCCEEDED, and every power_type=virsh record in it is attributed to a known containment VM.
2. LENS 2 returns ZERO associated records for the site being torn down.
3. Neither lens was skipped because a command failed. A failure is a refusal.

Step 3 (Path A only) -- Pick the ROOT first, then plan the destroy inside it

Scoping a single DC is now a ROOT choice, not a -target choice. Under D-123 Model B each DC's substrate is its own root with its own state file, and each of those roots contains exactly ONE site -- so a plain tofu destroy in opentofu/vr1-dc1-substrate/ is already scoped to vr1-dc1, with none of the silent-divergence risk HashiCorp's targeting warning describes. Reach for -target only for a PARTIAL teardown inside one root, or in the OUTER root, which holds all three sites at once.

Module names MEASURED 2026-07-29 (grep -n '^module' <root>/main.tf). There is no dc1_* module in either root -- those names are pre-D-119 and never existed here; a -target naming one aborts the plan.

module.vr1_dc1_node is a for_each module keyed by node name -- targeting the module covers every instance; there are no _node_01/_node_02 siblings to enumerate. There is no maas_vm_host module in either root to target (Step 2). inner_storage is named identically in BOTH substrate roots -- it is disambiguated by which root you are standing in, which is one more reason to confirm the directory before every command.

Order: INNER root first, then OUTER. Destroying module.vvr1_dcN first leaves the inner state describing objects that no longer exist and removes the only path the inner provider can dial.

INNER root -- run FROM THE OFFICE1 HEADEND (D-128 Plane 2; the provider is qemu+ssh and the state lives there). For the WHOLE DC substrate, no -target at all:

cd "$O1_REPO/opentofu/vr1-dc1-substrate"
tofu plan -destroy -out=teardown-vr1-dc1-inner.tfplan

For a PARTIAL inner teardown, target the real names:

tofu plan -destroy \
  -target=module.vr1_dc1_node \
  -target=module.vr1_dc1_opnsense \
  -target=module.vr1_dc1_wan \
  -target=module.vr1_dc1_planes \
  -target=module.inner_storage \
  -out=teardown-vr1-dc1-inner.tfplan

OUTER root -- on vcloud, and ONLY after the inner half is done:

cd "$REPO/opentofu"
tofu plan -destroy \
  -target=module.vvr1_dc1 \
  -target=module.vr1_dc1_uplink \
  -target=module.vr1_dc1_storage \
  -out=teardown-vr1-dc1-outer.tfplan

Do NOT target the mesh-link modules (mesh_vr1_dc0_vr1_dc1, mesh_vr1_dc0_office1, mesh_vr1_dc1_office1) or netem_vr1_dc0_vr1_dc1 in a single-site teardown -- each leg is SHARED with the site at its far end, and the dc0<->office1 leg additionally CARRIES the live rack<->region transit (MAAS, node DNS, the inner root's own qemu+ssh path). Only include a leg once BOTH sites it connects are going away (see the mesh-link note after Step 5).

Review the plan line by line: expect destroys for exactly the resources above (libvirt_domain, libvirt_volume, libvirt_network x6 for the planes plus the WAN bridge network, libvirt_pool) and NOTHING outside this site's scope -- in particular confirm no office1_*, no other DC's vr1_dc0_*/vr1_dc1_*, and no mesh_*/netem_* resource appears.

CAPTURE THE PRE-DESTROY BASELINE NOW, while the containment VM is still up -- Step 4 cannot tell "destroyed cleanly" from "could not look" without it:

source "$O1_REPO/scripts/lib-hosts.sh"; lib_hosts_select_dc vr1-dc1   # -> VIRSH_POWER_ADDRESS
virsh -c "$VIRSH_POWER_ADDRESS" list --all      > /tmp/pre-teardown-inner-domains.txt
virsh -c "$VIRSH_POWER_ADDRESS" net-list --all  > /tmp/pre-teardown-inner-nets.txt
virsh -c "$VIRSH_POWER_ADDRESS" pool-list --all > /tmp/pre-teardown-inner-pools.txt

GATE: plan matches this scope exactly; no unexpected destroys outside the targeted site; the baseline capture above is non-empty. Do not apply a plan you have not read.

Step 4 (Path A only) -- Apply the scoped destroy [MUTATION: gated]

Apply the INNER plan from the Office1 headend, in the inner root; apply the OUTER plan on vcloud, in $REPO/opentofu. Same order as Step 3: inner, then outer.

tofu destroy teardown-vr1-dc1-inner.tfplan     # inner root, from the Office1 headend
tofu destroy teardown-vr1-dc1-outer.tfplan     # outer root, on vcloud -- AFTER the inner verify below

(Note: unlike a create/apply plan, a -destroy-mode plan file is applied the same way -- tofu apply <planfile> and tofu destroy <planfile> are equivalent once the plan itself was generated with -destroy; use whichever your tofu version's docs confirm for a plan file specifically, and if in doubt, prefer re-running with tofu destroy directly rather than guessing.)

Confirm this is the exact reviewed plan file from Step 3 (not re-planned) before running.

VERIFY -- and note WHICH HOST each half is verified from. The DC's node, edge and plane objects live inside the containment VM, so a local virsh on vcloud can never see them: MEASURED on vcloud 2026-07-29, virsh list --all returns exactly vvr1-dc0, vvr1-dc1, voffice1, office1-opnsense. A vcloud-local grep -i dc1 therefore "passes" against a fully intact DC. This is the D-128 split: Plane 1 = the outer qemu:///system on vcloud; the inner substrate is reached over qemu+ssh from the Office1 headend.

INNER half -- from the Office1 headend, BEFORE the outer destroy (afterwards the URI cannot answer BY DESIGN):

source "$O1_REPO/scripts/lib-hosts.sh"; lib_hosts_select_dc vr1-dc1   # -> VIRSH_POWER_ADDRESS
virsh -c "$VIRSH_POWER_ADDRESS" version >/dev/null 2>&1 || {
  echo "REFUSE: cannot reach $VIRSH_POWER_ADDRESS. An empty list from an unreachable" >&2
  echo "        URI, a stopped containment VM or a bad key is NOT a clean destroy." >&2
  exit 1; }
virsh -c "$VIRSH_POWER_ADDRESS" list --all
virsh -c "$VIRSH_POWER_ADDRESS" net-list --all
virsh -c "$VIRSH_POWER_ADDRESS" pool-list --all

Take the URI from scripts/lib-hosts.sh, never from memory. Diff each against the Step 3 baseline: expect exactly the planned objects gone and NOTHING else changed. An empty list with no baseline to compare it to proves nothing.

OUTER half -- on vcloud, after the outer destroy. Assert on the named objects, not on a substring match:

virsh list --all        # expect vvr1-dc1 GONE; vvr1-dc0, voffice1, office1-opnsense untouched
virsh net-list --all    # expect vr1-dc1-uplink GONE; the three mesh-* legs untouched
virsh pool-list --all   # expect vr1-dc1-pool GONE; vr1-dc0-pool, office1-pool, default untouched

(Live names measured 2026-07-29. Never grep -i dc1 here: it matches vr1-dc1 AND anything else carrying the string, and it cannot distinguish "absent" from "never visible from this host".)

Step 5 (Path A only) -- Confirm no drift outside the targeted scope

cd "$REPO/opentofu"
tofu plan

Run WITHOUT -target this time -- this is the check HashiCorp's own targeting warning (Step 3) calls for: confirm the rest of the configuration (Office1, the OTHER DC, the mesh legs, netem) shows no unexpected changes. Expect either "No changes" for everything outside the torn-down site, or exactly the changes you independently expect for other in-flight work -- nothing attributable to this teardown. Do the same in the SURVIVING DC's inner root if you used -target anywhere; a targeted apply's drift does not announce itself.

GATE: untargeted plan shows zero unexpected drift outside the torn-down site.

Path B -- full VR1 teardown (replaces Steps 3-5 above)

Do Step 1 (back up EVERY root's state -- outer plus BOTH inner roots) and Step 2 (the machine census, run for EVERY site, not just one) first, then destroy BOTH inner roots before the outer one -- "everything" is now three roots on two hosts, not one cd opentofu:

# from the OFFICE1 HEADEND, once per DC substrate root:
cd "$O1_REPO/opentofu/vr1-dc0-substrate" && tofu plan -destroy -out=teardown-vr1-dc0-inner.tfplan
cd "$O1_REPO/opentofu/vr1-dc1-substrate" && tofu plan -destroy -out=teardown-vr1-dc1-inner.tfplan

# then, on vcloud:
cd "$REPO/opentofu"
tofu plan -destroy -out=teardown-full.tfplan

Review: expect a destroy for every resource each tree currently manages (check against that root's actual uncommented module list at teardown time -- it grows as more stages get executed, so what "everything" means changes over the buildout, unlike Path A's fixed per-site scope).

GATE: each plan matches its root's full current scope exactly.

tofu destroy teardown-full.tfplan

VERIFY. Verify each inner root from the Office1 headend BEFORE the outer destroy, using Step 4's virsh -c "$VIRSH_POWER_ADDRESS" version refusal guard -- once the containment VMs are gone the inner URIs cannot answer, and "unreachable" is not evidence of anything. Then, on vcloud:

virsh list --all
virsh net-list --all
virsh pool-list --all

Expect: nothing VR1-related remains (VR0's own testcloud resources, which OpenTofu does not manage, are untouched by any of this). Note that vcloud's default pool and virbr0/wan are not VR1 objects -- their survival is correct, not residue.

Then do the equivalent of Step 5 (a final untargeted tofu plan) to confirm the destroy converged state to fully empty, not partially.

Mesh-link teardown (either path, handle separately)

The three D-100 mesh-link legs live in the OUTER root. Module names and the libvirt networks they create, MEASURED 2026-07-29 (grep -n '^module' opentofu/main.tf; virsh net-list --all):

(The pre-D-119 names mesh_dc1_dc2 / mesh_dc1_office1 / mesh_dc2_office1 that this section used to carry exist nowhere -- opentofu/main.tf holds moved blocks recording exactly that rename. virbrN is a drifting libvirt assignment: re-measure with virsh net-info <network> before using one, never carry it from here.)

Each leg is shared between two sites. Destroy a leg only when BOTH its endpoints are permanently going away (a full VR1 teardown, or abandoning both of its sites together) -- never as part of a single-site scoped teardown, and never as a reflex "clean up everything DC-adjacent" action. mesh-vr1-dc0- office1 deserves extra care: it carries the live rack<->region transit, so MAAS, node DNS and the inner root's own qemu+ssh path all die with it.

modules/netem-link IS wired now -- module "netem_vr1_dc0_vr1_dc1" in the outer root, applied 2026-07-21 against virbr5 (the zero-traffic dc0<->dc1 leg). It has a correct destroy-time provisioner (provisioner "local-exec" { when = destroy; ... tc qdisc del ... }, confirmed by reading opentofu/modules/netem-link/main.tf directly), so destroying the corresponding terraform_data.netem resource cleans up the real tc qdisc rule on the vcloud host automatically; no extra manual step for that piece. Destroy it BEFORE the leg it is attached to, and note it is a separate module from the leg -- destroying mesh_vr1_dc0_vr1_dc1 alone leaves the netem resource pointing at a bridge that no longer exists.

Rollback decision tree -- "a stage's tofu apply failed partway through, now what?"

Read this before reaching for tofu destroy at all. OpenTofu is designed for idempotent convergence -- in most failure modes, fixing the root cause and re-running the SAME tofu apply is correct and safer than tearing down and starting over, because it preserves everything that DID apply successfully and only retries what didn't.

Question 0 -- WHICH ROOT failed? There are three: the OUTER root on vcloud, and one INNER substrate root per DC, run from the Office1 headend. The answer decides which state file, which host, and which virsh endpoint every branch below is talking about. Answer it before anything else.

Question 0b -- is anything DEPLOYED on top? From Stage 5 onward a failed tofu apply can sit under live MAAS machine records and a live Juju model. No branch below reaches a destroy without Step 2's machine census passing first. If Step 2 returns records, the destroy is not the next action -- resolving the records is, and that is an operator-gated decision on its own.

Verb discipline, because these read alike under time pressure: virsh destroy vvr1-dcN is a forced POWER-OFF, reversible with virsh start, and it leaves the domain, its disk and every MAAS record intact -- that is the D-122 site-down lever and it is a legitimate first move. tofu destroy of module.vvr1_dcN UNDEFINES the domain and DELETES its disk, taking the whole inner fleet with it, irreversibly. Reaching for the second when you meant the first is the expensive mistake this runbook exists to prevent.

1. tofu apply errored before creating anything (e.g., tofu init/ validate/plan failed, or apply failed on its very first resource). Nothing was created. Fix the root cause (bad variable, unreachable provider, syntax error) and re-run from that stage's own tofu plan step. No teardown needed.
2. tofu apply created SOME resources, then failed on a later one. tofu plan (no -destroy) immediately to see OpenTofu's own view of what's now live vs. what the config wants. Default to FIX-FORWARD: fix whatever caused the failure (a bad node-VM memory size, an unreachable MAAS endpoint, a real infra problem) and re-run tofu apply for that same stage -- OpenTofu will create only what's still missing, leaving successfully-created resources alone. Reserve a teardown (Path A, scoped to that site) for when the partial state is itself broken in a way fix-forward can't reach (e.g., a domain stuck in a bad libvirt state that manual virsh intervention already tried and failed to fix).
3. You suspect terraform.tfstate itself is corrupted or lost (this is the scenario opentofu/README.md's "State file handling" section already flags as the real cost of no backend/backup). There is no clean recovery path documented in this repo: either a manual tofu import per still-live resource (tedious, must be done for every domain/network/pool by hand) or accepting a full teardown-and-rebuild from Step 1 of the affected stage. This is exactly why Step 1's pre-teardown backup habit matters even outside a deliberate teardown -- back up terraform.tfstate after every successful stage apply too, not only before a teardown.
4. A resource exists in reality but OpenTofu's state doesn't know about it (e.g., a virt-install-created VM from Stage 2's Option B manual path, which OpenTofu never created and cannot destroy). This runbook's tofu destroy/plan -destroy commands only ever touch what OpenTofu itself manages -- anything created via a manual/Option-B path needs manual teardown (virsh destroy/undefine, etc.) by the same operator discipline that created it. Do not expect tofu destroy to find or remove it.
5. The failure is INSIDE a DC and you are tempted to reach for the containment VM. Taking the site down (virsh destroy vvr1-dcN) is reversible and is often the right move to stop a cascade. Destroying the containment VM through the OUTER root is not a rollback of the inner apply -- it discards the entire inner substrate, leaves the inner state file describing objects that no longer exist, and leaves MAAS holding machine records whose power_address will never answer again. Fix-forward in the INNER root first; if you truly need the DC gone, do Step 2, then the inner root, then the outer -- in that order.

Verification (of this runbook's own accuracy, not a live run)

Every tofu/terraform flag cited above (-destroy, -target, the targeting-risk warning) was checked against HashiCorp's own current CLI documentation before being written here (fetched 2026-07-10), not asserted from memory -- same discipline as opentofu/README.md's other research sections. netem-link's destroy-time provisioner was confirmed by reading the actual .tf file, not assumed.

CORRECTION 2026-07-29: this paragraph previously vouched for "dc1_maas_vm_host's lack of a Terraform-expressible dependency on node-VM domains" as something confirmed by reading the .tf files. No such module has ever existed in either root, so that was a verification claim about nothing -- the false-verification class that lets a wrong instruction look checked. All module, network, pool and domain names in this runbook were RE-MEASURED against the tree and the live hosts on 2026-07-29 (grep -n '^module' <root>/main.tf; virsh list/net-list/pool-list --all; virsh net-dumpxml). What is NOT verified: this runbook's commands have never been run against a real tofu binary or a real state file (same SCAFFOLD/UNVALIDATED status as the rest of opentofu/) -- treat every command here as reviewed-but-unexercised until scripts/opentofu-validate.sh and a real dry run confirm otherwise.