# VR1 DC-DC teardown / rollback runbook

Scope: tearing down or rolling back the infrastructure `opentofu/` provisions
for the VR1 (DC-DC) buildout -- the `vr1-dc0` / `vr1-dc1` / Office1 plane and
local networks, storage pools, node-VM domains, containment VMs, OPNsense edge
VMs, and the D-100 mesh-link triangle. (It once also claimed MAAS `vm_host`
registrations; there are none -- see Step 2. The `dc1`/`dc2` names it was
written with are pre-D-119 and mean different datacenters now.)
Written 2026-07-10 (DOCFIX-175
gap-register item #19) because nothing else in this repo covers this layer:
`runbooks/phase-00-teardown-release.sh` / `phase-00-teardown-destroy.sh`
(D-061) tear down the VR0 **juju/MAAS-machine** layer, a different pairing
of tools and a different failure mode entirely -- see "Relationship to
D-061" below for why that mechanism does not just transfer over.

**UNVALIDATED, same caveat as everywhere else in `opentofu/`:** no `tofu`
binary has been available in any session that authored this tree. Every
command below is standard, well-documented OpenTofu/Terraform CLI syntax
(verified against HashiCorp's own current docs where cited), not invented --
but run `bash scripts/opentofu-validate.sh` on a machine with the real
binary and read the result before trusting any of it against a live state
file.

---

> ## !!! MODEL B RESHAPE (D-123, 2026-07-16) -- READ BEFORE ANY DC TEARDOWN; this runbook's body predates it !!!
>
> This runbook was written (2026-07-10) for the SINGLE-ROOT shape (OpenTofu creates the DC node-VM
> domains / planes / edge directly on the **vcloud** host, and `maas-vm-host` registers the vcloud
> host). Under **D-123 Model B** that is no longer how a DC is built, so the resource-by-resource
> mechanics below are partly WRONG for the DCs. Current shape:
>
> - **TWO roots / TWO state files PER DC, and there are now TWO DCs.** The OUTER root (`opentofu/`,
>   vcloud) owns `vvr1-dc0` and `vvr1-dc1` (the DC containment VMs), the office1<->dcN transits, and
>   the **D-125** vcloud ISP uplinks (`vr1-dc0-uplink` / `vr1-dc1-uplink` NAT + each containment VM's
>   uplink NIC / `br-vr1-dcN-wan` netplan bridge). Each INNER root
>   (`opentofu/vr1-dc0-substrate/`, `opentofu/vr1-dc1-substrate/`; provider = `qemu+ssh` to that DC's
>   containment VM) owns everything INSIDE its DC -- the node VMs, the 6 planes, `vr1-dcN-wan`, the
>   OPNsense edge, the inner pool. So the "node-VM domains / planes / edge" this runbook talks about
>   live on **the containment VM's libvirt, not vcloud's**.
> - **There is NO MAAS `vm_host`/pod for any VR1 DC.** MAAS holds per-MACHINE records with
>   `power_type=virsh` (D-103/D-123 amendments, ruled 2026-07-20) and `modules/maas-vm-host` is
>   instantiated in NEITHER root -- measured 2026-07-29. Any instruction anywhere to "remove the
>   `maas-vm-host` record before destroying the containment VM" is VOID: there is no such record,
>   the read returns nothing on a DC with a full fleet enrolled, and acting on that emptiness
>   destroys the containment VM out from under live MAAS machine records. **Step 2 is the real
>   check.**
> - **Whole-DC site-down is ONE object: `virsh destroy vvr1-dcN`** (the D-122/D-123 point). Destroying
>   the containment VM instantly takes down the entire inner fleet -- no per-domain group destroy
>   needed. That is the fast path for "abandon this DC and rebuild." **Know which verb you are
>   using:** `virsh destroy` is a forced POWER-OFF and is REVERSIBLE (`virsh start vvr1-dcN`); the
>   domain, its disk and every MAAS record still exist. `tofu destroy` of `module.vvr1_dcN` UNDEFINES
>   the domain and deletes its disk -- irreversible, and it takes the whole inner fleet with it.
>   Under a failed Stage 5 those are not interchangeable.
> - **Teardown ORDER (two roots):** tear down the INNER root first (`cd opentofu/vr1-dcN-substrate &&
>   tofu destroy`, or just `virsh destroy vvr1-dcN` if you are only taking the site DOWN), THEN the
>   OUTER root. The D-061-style "clean up MAAS's record before destroying the libvirt underneath"
>   principle below still applies -- and with no pod to remove, "MAAS's record" means the per-machine
>   records enumerated in Step 2. Run Step 2 BEFORE either root's destroy.
> - **Reverting Model B -> Model A** (a different operation from teardown) is
>   `docs/archive/model-a-fallback-plan.md` (git tag `model-a-fallback`), not this runbook.
>
> Until the body below is rewritten to the two-root shape, treat its DC-substrate steps as
> **single-root reference** and apply the mapping above. The Office1 (`voffice1`) and mesh/pool steps
> are unaffected.

---

## When to use this runbook

- **Mid-buildout failure recovery.** A stage's `tofu apply` failed partway
  through, or created something wrong, and you need to know whether to fix-
  forward or tear down and retry. See "Rollback decision tree" below --
  READ THAT FIRST, since fix-forward is usually the right answer, not a
  destroy.
- **Deliberate re-run from clean.** You want to abandon a DC's (or Office1's,
  or the whole VR1 layer's) current state and rebuild from Step 1 of its
  stage runbook.
- **NOT for**: tearing down VR0's juju/MAAS-machine layer (use D-061's
  scripts), or removing a single juju application/relation from an already-
  deployed OpenStack bundle (that is normal juju operations, not this
  runbook).

---

## Relationship to D-061 (read before mixing the two layers)

D-061's core lesson: destroying/releasing a resource at the WRONG layer, in
the wrong order, orphans state at the OTHER layer -- `juju destroy-model`
released MAAS-pod-composed machines and MAAS auto-decomposed them, losing
the carve; the fix was `remove-machine --keep-instance` FIRST (detach from
juju without releasing), verify survival in MAAS, THEN act on the model.

The same PRINCIPLE applies here, but the mechanism is different because the
layers being coordinated are different:

- D-061 coordinates **juju's machine view** against **MAAS's machine view**
  of hosts MAAS itself pod-composed.
- This runbook coordinates **OpenTofu's resource view** (libvirt domains/
  networks/pools it created directly) against **MAAS's per-MACHINE view** of
  those same domains. MEASURED 2026-07-29: `modules/maas-vm-host` is
  instantiated in NEITHER root (`grep -n 'maas_vm_host' opentofu/*.tf
  opentofu/vr1-dc0-substrate/*.tf opentofu/vr1-dc1-substrate/*.tf` returns
  only comments), and the `provider "maas"` block those modules would need is
  still deliberately absent (`opentofu/main.tf:9-20`, DOCFIX-179). MAAS knows
  the DC node VMs because they PXE-ENLISTED, and it powers them per machine
  with `power_type=virsh` (D-103/D-123 amendments, ruled 2026-07-20 --
  `scripts/maas-node-power.sh`'s header carries the two measurements that
  refuted the pod).

Because this repo uses neither MAAS pod-composition NOR a `vm_host` for its
OpenTofu-created VMs, the SPECIFIC `--keep-instance` decompose-on-release
failure mode D-061 diagnosed does not directly apply here. The GENERAL
principle still does: **clean up MAAS's record of a host before destroying the
libvirt resource underneath it**, not after -- destroying the libvirt domain/
host first would leave MAAS pointing at a `power_address` that no longer
answers, an orphaned-but-not-obviously-broken record rather than a clean
removal.

**What this section used to say, and why it is now a hazard rather than a
caveat:** it stated that no VR1 DC had reached Stage 4, so no VR1 libvirt host
had ever had real MAAS-enrolled machines under it. **That is false as of
2026-07-23** (`docs/CURRENT-STATE.md`: 18 nodes READY, 9 per DC, both DCs
commissioned; `power_type=virsh` set and proven by a real
`query-power-state`), and the per-DC Juju controller VMs were enlisted on top
of that in July 2026. Anything downstream of "nothing is enrolled yet" --
including the old Step 2 -- must be treated as REVERSED, not merely stale. The
exact MAAS-side release/delete commands for a machine whose libvirt domain is
about to disappear are STILL not worked out or tested in this repo: Step 2
states the principle, enumerates the records, and points at MAAS's own current
docs for the flags. Do not invent flags for a scenario this repo has never
actually exercised.

---

## TWO CLONES, TWO HOSTS -- set these once, before anything below

Every block in this runbook runs on one of two hosts (D-128), each with its
own clone of this repo. The same `cd` pasted on the wrong one is the
foot-gun this section exists to remove, so the two are named differently
throughout:

- **`$REPO` -- the VCLOUD clone.** Plane 1: the OUTER root
  (`$REPO/opentofu`, `libvirt_uri = qemu:///system`) and local `virsh`.
- **`$O1_REPO` -- the OFFICE1 HEADEND clone, on `voffice1`.** Plane 2: BOTH
  DC substrate roots (`$O1_REPO/opentofu/vr1-dc0-substrate`,
  `.../vr1-dc1-substrate`, provider = `qemu+ssh`), their state files, the
  `maas` CLI, and every `virsh -c qemu+ssh://...` into a containment VM.

MEASURE `$O1_REPO` with `ls` on the headend -- do not assume it, and do not
assume the inner state lives on vcloud (it does not).

---

## Two paths, by intent (mirrors D-061's release-vs-destroy split)

- **Path A -- scoped teardown.** Tear down ONE site's resources (a single DC,
  or Office1), leaving the other sites' infrastructure untouched. **Scope by
  choosing the ROOT first** -- each DC substrate root contains exactly one
  site, so a plain `tofu destroy` there is already scoped and needs no
  `-target` at all. `-target` is reserved for the OUTER root (which holds all
  three sites) and for a partial teardown inside one root. Where you do use
  it, HashiCorp's own docs flag it as an exceptional-circumstances tool, not
  for routine use, because targeted applies can leave configuration and state
  silently diverging: *"This targeting capability is provided for exceptional
  circumstances, such as recovering from mistakes or working around Terraform
  limitations. It is not recommended to use `-target` for routine operations,
  since this can lead to undetected configuration drift"*
  (developer.hashicorp.com/terraform/cli/commands/plan, fetched 2026-07-10).
  A scoped teardown IS exactly that kind of exceptional circumstance --
  acceptable here, but ALWAYS follow a targeted apply with a full (untargeted)
  `tofu plan` afterward (Step 5) to confirm the rest of the configuration
  didn't drift.
- **Path B -- full VR1 teardown.** Abandon everything OpenTofu manages for
  VR1. That is now THREE roots across TWO hosts -- both DC substrate roots
  from the Office1 headend, then the outer root on vcloud -- each a plain
  `tofu destroy` (no `-target`), letting OpenTofu compute the safe order
  WITHIN each root from its own resource graph. The order BETWEEN roots is
  yours to get right: inner before outer, always.

Both paths share Steps 1 (state backup), 2 (the MAAS machine census) and 5
(final verify). Path A is Steps 2-5; Path B replaces Steps 3-5 with the
per-root destroys described in its own section (see
"Path B" section after Step 5).

---

## Path M -- JUJU MODEL teardown (the innermost layer; NO substrate is touched)

**ADDED 2026-08-02 on operator direction: "this has bit in the past and will be a
reusable and needed set for the future."** Everything here is MEASURED from real runs
on this cloud -- the 2026-07-31 VR1 dc0 teardown (`docs/CURRENT-STATE.md`, "MODEL
TEARDOWN 2026-07-31") and the 2026-08-02 rebuild. It is NOT derived from `juju help`;
where the two disagree, the help text lost, and that is recorded below.

**USE THIS WHEN** you want to re-deploy the bundle into a clean model: a failed or
half-installed deploy, a config defect that reached the charms, a bundle change you want
applied from scratch. **It is NOT a substrate teardown** -- Paths A and B above are a
different, much larger action. If the fix is a deploy input, you almost certainly want
Path M and nothing else.

### M.0 -- What Path M does and does NOT touch

| Touched | NOT touched |
|---|---|
| the `<DC>` model: every unit, container, and machine assignment | the `controller` model and the D-104 controller VM |
| model-level config (see the M.4 trap) | the MAAS region, its database, and the machine RECORDS |
| | the rack host, the OPNsense edge, both proxies, the tofu substrate, the other DC |

`juju destroy-model`'s own summary scopes it: *"Terminate all machines/containers and
resources for a **non-controller model**."* The controller model is out of reach by
design.

### M.1 -- Pre-flight: prove the blast radius, every time

```bash
juju storage -m "$DC_MODEL"        # storage instances -- decides the storage flag
juju offers  -m "$DC_MODEL"        # cross-model offers -- non-empty = another model consumes this one
juju models                        # confirm which models exist and their machine counts
```
- **A storage flag is required ONLY if `juju storage` is non-empty.** MEASURED 2026-08-02
  on `vr1-dc0`: *"No storage to display."* `ceph-osd`'s `osd-devices` are RAW BLOCK
  DEVICES set by charm config, NOT juju storage instances -- so `--destroy-storage`
  destroys nothing here and must not be passed as a reflex. MAAS re-partitions those
  disks at re-image.
- **Non-empty `juju offers` means STOP** and work out what consumes the endpoint first.

### M.2 -- The command [MUTATION: gated]

```bash
juju destroy-model "$DC_MODEL" --force --no-wait --no-prompt
```

**`--no-wait` IS PART OF THE PROVEN PATH. DO NOT DROP IT ON A FAILED DEPLOY.** MEASURED
2026-07-31: a plain `destroy-model` **STALLED and would not self-resolve** -- `attempt 30
to destroy model failed (will retry): model not empty, found 26 machines, 37
applications`, flat for ~19 minutes with the application set BYTE-IDENTICAL across a
12-minute diff. **Mechanism, which makes the stall terminal rather than slow: all 26
machine/container agents were `stopped`, so no teardown hook could execute at all** --
the destroy worker asks and nothing answers. Units already in `error` (11 haclusters at
`hook failed: "stop"`, four at `install`, five at `identity-service-relation-departed`)
could never run their stop hooks, so it was never going to drain. **`--force --no-wait`
cleared it: 18 -> 5 -> 2 machines, then `Model destroyed.`**
Reading `juju help destroy-model` alone would talk you OUT of `--no-wait` ("Rush through
model destruction without waiting for each individual step to complete"). On a model whose
agents are already stopped there is nothing to wait FOR. Trust the measurement.

**>>> BUT `--force` CAN ORPHAN THE MODEL AND TAKE THE WHOLE CONTROLLER DOWN. MEASURED
2026-08-03. READ THIS BEFORE USING IT. <<<** The 2026-08-02 dc0 teardown reported
`Model destroyed.` EXIT 0 with a clean progressive drain and correct MAAS release -- and
still left the model in an INCONSISTENT state: the model document survived while its
STATUS document did not. Consequences, in order:

1. `juju models` shows the model at `life: dead`; **`juju add-model <same-name>` refuses
   with `model "<name>" for admin already exists`.**
2. The controller's `undertaker` worker crash-loops on
   `cannot destroy cloud resources: cannot set status: model not found`, on a growing
   backoff. **It never self-resolves** -- the error is deterministic.
3. **Worst: the `modelcache` worker crash-loops on `failed to initialise backing for
   models:<uuid>: status doc "<uuid>:e" not found, getting new watcher`** -- hundreds of
   iterations per minute (732 logged in ~12 min). The API server DEPENDS on modelcache,
   so **port 17070 goes to `connection refused` and the CONTROLLER IS DOWN.** The VM is
   up and pings; `systemctl is-active jujud-machine-0` reads `active`. Only the log and
   the refused port tell you.

**A `jujud` restart does NOT fix it** (tried; the worker re-enters the same loop). This is
the LP #1737487 class -- "dead model did not get removed, and cannot be removed". Upstream
is explicit that `--force` is a LAST RESORT precisely because it can leave state juju
cannot reconcile.

**MITIGATION, in order of preference:**
- **Try WITHOUT `--force` first** when the model's agents are still ALIVE. The 08-02 run
  drained progressively (36 -> 26 -> 20 -> 16 -> 9 -> 0) with hooks executing, i.e. force
  was not actually needed; it was carried over from the 07-31 stall where agents were
  `stopped` and force genuinely was the only way through.
- **Reserve `--force --no-wait` for the measured stall condition** -- agents `stopped`,
  destroy flat for many minutes, units unable to run teardown hooks.
- **DO NOT re-issue `destroy-model` against a model already at `life: dead`.** Measured
  2026-08-03: it moved the model BACK to `dying` and re-armed the undertaker loop. It did
  not help and it changed state.
- **If the controller is already down on this symptom, REBUILD THE CONTROLLER** rather
  than attempting juju state-DB surgery. A controller hosting only a destroyed model has
  nothing worth preserving, and `phase-4` Step 2's bootstrap is a tested path.

**`--timeout` is the one to leave off:** the help warns that `--force` WITH `--timeout`
continues "without consideration or respect for clean shutdown or resource cleanup".
`--no-prompt` is required non-interactively (the help's Details prose still says `-y`;
that is STALE -- the flag list is authoritative).

### M.3 -- THE RELEASE PATH: what happens to the MAAS machines

**MEASURED, BOTH RUNS -- juju releases the machines and MAAS returns them to `Ready`.
NO `maas machine release` IS NEEDED AND NONE WAS RUN.** After the 2026-07-31 forced
destroy, read back: **all NINE role nodes `Ready / owner=None`, zero stranded.**

- **`enable_disk_erasing_on_release` is `false`** on this region (verify per DC:
  `maas <profile> maas get-config name=enable_disk_erasing_on_release`), so release does
  NOT trigger an erase cycle -- nodes return to `Ready` in minutes and are re-imaged by
  the next deploy. During the 07-31 run the six nodes juju did release reached
  `Ready / owner None` in minutes; **MAAS was never the bottleneck**, juju simply had not
  yet issued a release for the other three.
- **EXPECT the controller VM to stay `Deployed`.** `subtle-grouse` remains `Deployed`
  under `juju-vr1-dc0` because it belongs to the `controller` model, not the destroyed
  one. **The correct post-state is 9 `Ready` + 1 `Deployed`, NOT 10 `Ready`** -- a
  session expecting all ten to release will raise a false alarm.
- **THE CASCADE SIGNATURE TO STOP ON: a DROP in the MAAS machine COUNT.** Records are
  released, never deleted. This repo has a real cascade (2026-07-21, a MAAS pod delete
  taking its linked machine records), which is why the count is checked and not assumed.
- **The VR0 pod warning does NOT apply here.** `runbooks/phase-00-teardown-maas-reset.md`
  records that on VR0's virsh-POD MAAS, `destroy-model` DECOMPOSES pod-composed machines
  (observed 3x) -- there, use the D-061 pair instead. **VR1 uses per-machine
  `power_type=virsh` and NOT pods (D-103/D-123 amendments), so there is nothing to
  decompose**, which both VR1 teardowns confirm. Do not import the VR0 procedure.

### M.4 -- >>> THE TRAP: `destroy-model` TAKES THE MODEL CONFIG WITH IT <<<

**This bit on 2026-07-31 and it will bite again.** Destroying the model destroys its
model-config. Step 3.5's `apt-mirror` and the spaces work vanished with it, **and nothing
in the repo would have caught it** -- the next deploy fails on package fetches and reads
like an artifact-service fault rather than a missing model setting.

**Anything the deploy needs at the MODEL level must be re-established BEFORE the next
`add-model`, or set as a CONTROLLER MODEL-DEFAULT so it survives.**

```bash
juju model-defaults                       # what a NEW model will inherit
juju model-defaults <key>=<value>         # set (per-controller, so per-DC by D-104)
```
**Model-defaults are inherited by NEW models ONLY -- never applied retroactively**, so
they must be right BEFORE `add-model`, not after.

**MEASURED 2026-08-02, and it is a live trap for the very next rebuild:** this
controller's defaults carry **`apt-mirror = http://10.12.8.4/ubuntu` and nothing else**.
After the D-135 amendment (b) convergence, that value is WRONG -- dc0 now consumes the
apt caching proxy -- and the three settings the deploy actually needs
(`apt-http-proxy`, `snap-http-proxy`, `snap-https-proxy`) are **absent from the defaults
and were set at MODEL level**, so the destroy takes them. A rebuild that skips this step
comes back pointing at the retired mirror with no proxies at all.

Per-DC checklist before `add-model`, each read back on CONTENT:

```bash
juju model-defaults --reset apt-mirror                       # convergence: proxy, not mirror
juju model-defaults apt-http-proxy=http://<UTIL>:3142        # dc-cache-proxy.sh LISTEN
juju model-defaults snap-http-proxy=http://<UTIL>:3129       # dc-snap-proxy.sh LISTEN
juju model-defaults snap-https-proxy=http://<UTIL>:3129
# LEAVE snap-store-proxy / -url / -assertions EMPTY (Enterprise Store, different product)
```
Take `<UTIL>` from that DC's own checker output, never from memory (dc0 `10.12.8.4`,
dc1 `10.12.68.4`). **OWED, and still not built: a site-keyed `dc-model-defaults.sh` with
a `check`, so these are VERIFIED at every standup rather than remembered.**

### M.5 -- Post-checks, before rebuilding anything

```bash
juju models                                   # (a) only `controller` remains
maas <profile> machines read | ...            # (b) COUNT unchanged; 9 role nodes Ready, controller VM Deployed
bash scripts/dc-cache-proxy.sh check <site>   # (c) artifact + snap services survived
bash scripts/dc-snap-proxy.sh  check <site>
```
**(b) is the one that catches a cascade** -- a drop in count means STOP, do not rebuild.

**>>> SECOND INSTRUMENT WARNING, 2026-08-03: DO NOT TEST FOR THE MODEL'S ABSENCE WITH A
GREP THAT CANNOT TELL "GONE" FROM "CANNOT LOOK". <<<** A wait-loop polling
`juju models --format=yaml | grep -q "name: admin/<model>"` and breaking when the grep
MISSES reported **"REAPED"** -- while the model still existed and the CONTROLLER API WAS
DOWN. With the API refused, `juju models` prints NOTHING and exits, the grep matches
nothing, and an absence-test reads that as success. It is the "could not look is never
nothing there" rule, in a checker written in the same session that wrote this runbook.
**Poll on a POSITIVE signal instead:** require `juju models` to actually RETURN model
lines, and only then decide whether the target is among them; treat empty output or a
non-zero juju exit as REFUSE (evaluate again), never as "gone". Watch the pipeline too --
`cmd | head -3; rc=$?` captures `head`'s status, not the command's.

**INSTRUMENT WARNING, learned by getting it wrong (2026-07-31): during a teardown,
`juju status -m <model>` is the instrument; `juju models` COUNTS ARE NOT.** The summary
columns ZERO OUT while the model is `destroying` -- they read "0 machines / 0 units"
while `juju status` simultaneously showed 26 machines and 37 applications. The real tell
(three control nodes stuck `Deployed`) was visible and was explained away as "more
containers to work through". Do not read progress from `juju models`.

### M.6 -- Rebuild sequence after a Path M teardown

1. M.5 post-checks all green.
2. **M.4 model-defaults corrected** -- this is the step that gets skipped.
3. `juju add-model <DC>` (inherits the corrected defaults), then re-apply the spaces work.
4. Re-stage the deploy inputs to the DC's `~/repo-stage` and **sha256-verify against repo
   HEAD** before trusting them (the racks carry no git clone).
5. `phase-4` Step 4.2 dry-run gate, then 4.3 deploy, then **4.4 immediately** -- 4.4 is
   the only gate that reads the DEPLOYED config rather than an input to it.

---

## Path C -- JUJU CONTROLLER teardown and rebuild (per DC)

**ADDED 2026-08-03.** Use when the DC's Juju controller itself is unusable -- not the
model. **The trigger that produced this section:** a `--force` model destroy orphaned a
model document without its status document, which crash-looped `modelcache`, which the API
server depends on, so port 17070 went to `connection refused` (Path M.2's warning block).
Also valid for a corrupted controller state DB, a lost controller VM, or a controller that
cannot be upgraded.

**PREFER THIS OVER JUJU STATE-DB SURGERY.** A controller whose only model has been
destroyed holds nothing worth preserving. Mongo surgery on juju's state is unsupported and
riskier than a rebuild along a path this repo has already exercised.

**NOTHING BELOW TOUCHES:** the MAAS region or its database, the machine records, the rack,
the edge, the artifact/snap proxies, the tofu substrate, or the other DC. The nine role
nodes stay `Ready` throughout and are re-imaged only when the bundle is redeployed.

### C.1 -- Census FIRST: write down what must survive

```bash
maas <profile> machines read | jq -r '.[] | "\(.hostname) \(.status_name) \(.owner // "None")"'
bash scripts/dc-cache-proxy.sh check <site>     # via the rack; see the header for transport
bash scripts/dc-snap-proxy.sh  check <site>
```
Record the machine COUNT. **A drop in that count at any later step is the cascade signature
(2026-07-21 incident) and means STOP.** Expected steady state through the whole of Path C:
nine role nodes `Ready / owner=None`, plus the controller VM (`Deployed` before C.2,
`Ready` after).

### C.2 -- Tear the controller down [MUTATION, gated]

**Pick by whether the API answers.** Test it, do not assume:
```bash
timeout 8 bash -c "</dev/tcp/<controller-ip>/17070" && echo API-UP || echo API-DOWN
```

- **API UP** -> `juju destroy-controller <name> --destroy-all-models --no-prompt`
  **This command is in the committed DENY list** (`.claude/settings.json`), so it is an
  OPERATOR-RUN command by policy. That is deliberate: it is the single most destructive
  juju verb.
- **API DOWN** -> **`juju kill-controller` DOES NOT WORK. MEASURED 2026-08-03; DO NOT WASTE
  TEN MINUTES ON IT.** Its help implies a direct-to-provider fallback ("Timeout before
  direct destruction", `-t` default `5m0s`), and the first version of this section said so.
  **It does not reach that fallback.** Run against a controller whose API is refused, with
  `--no-prompt`, it waited ~10 minutes and then exited 1:
  ```
  Unable to open API: open connection timed out
  ERROR cannot connect to model config API: unable to connect to API:
        dial tcp <controller-ip>:17070: connect: connection refused
  ```
  It needs the MODEL CONFIG API to decide what to destroy, so an unreachable API defeats it
  before the timeout matters. **Nothing was destroyed and nothing cascaded** -- the MAAS
  census was identical before and after -- so it is safe, just useless.
  **USE THE MANUAL PAIR INSTEAD (C.2b).**
  **>>> POLICY GAP, LOGGED 2026-08-03: `juju kill-controller` is NOT in the deny list while
  `juju destroy-controller` IS.** `kill-controller` is the STRONGER command. The committed
  policy gates the weaker one and lets the stronger through. Treat `kill-controller` as
  operator-gated regardless of what the rule engine does, and see `docs/security-ledger.md`
  SEC-030 for why the ask/deny rules are not the real gate here -- the presentation
  discipline is.

### C.2b -- API DOWN: the manual pair (juju client, then MAAS)

When `kill-controller` cannot work, the controller is removed in two independent halves.
This is the D-061 coordination principle applied to the controller: **clean up juju's view
first, then MAAS's**, never the reverse.

**(1) Drop the client's registration -- SAFE, no cloud resource touched.**
```bash
juju controllers                              # confirm the name
juju unregister <controller-name> --no-prompt # edits local juju client config ONLY
```
This does not release the VM. After it, MAAS still shows the controller VM `Deployed` and
OWNED by the juju service user -- that is expected, not a leak.

**(2) Release the controller VM in MAAS [MUTATION, gated].** Identify it by TAG, never by
hostname -- MAAS auto-generated hostnames do not match the ruled names, and the tag is the
stable key:
```bash
maas <profile> machines read \
  | jq -r --arg t "juju-controller-$DC" \
      '.[] | select(.tag_names[]? == $t) | "\(.hostname) \(.system_id) \(.status_name) \(.owner // "None")"'
maas <profile> machine release <system_id>
```
**GATE before releasing:** the query returns EXACTLY ONE machine, it is the controller VM,
and it is NOT one of the nine role nodes (cross-check the role nodes are `Ready /
owner=None` and stay that way). **After:** it reads `Ready / owner=None`, the MAAS machine
COUNT is unchanged, and the tag is still on it -- the tag is what C.6/C.7 bootstrap against,
so a release that lost the tag means STOP.

`enable_disk_erasing_on_release=false` on this region, so release is prompt and the disk is
re-imaged at the next bootstrap rather than wiped now.

### C.3 -- Verify the teardown released, and did not cascade

```bash
maas <profile> machines read | jq -r '.[] | "\(.hostname) \(.status_name) \(.owner // "None")"'
```
**GATE:** machine COUNT unchanged from C.1; the controller VM now reads `Ready / owner=None`
alongside the nine role nodes. Records are RELEASED, never deleted -- a count drop is the
cascade signature. Expect NO `maas machine release` to be necessary; juju's own release path
does it (measured, both prior teardowns).

### C.4 -- Clean the client side

```bash
juju controllers                       # is the dead controller still registered?
juju unregister <name> --no-prompt     # client-side ONLY; removes no cloud resource
```
Skip if `kill-controller` already removed it. `unregister` is safe: it edits the local juju
client config and nothing else.

### C.5 -- Rebuild: the credential gate, REGION-SCOPED

**Do not re-mint blindly.** Follow `runbooks/dc-dc-phase4-juju-bundle-per-dc.md` **Step
2.0** verbatim -- it carries the three-outcome gate and DOCFIX-206's correction that the
check must name the REGION explicitly. The trap it exists for: under D-132 q1 each DC has
its own region with its own database, so `juju credentials --client` can list a credential
whose MAAS user lives in a DIFFERENT region. Prove the region with
`scripts/maas-profile-assert.sh` (rack-controller identity -- a machine count is not proof)
before trusting any "already exists".

**A controller rebuild does NOT invalidate the MAAS credential** -- the credential belongs
to the CLOUD definition in the juju CLIENT, not to the controller. If it authenticated
before, it still does. Prove it rather than assume, with Step 2.0's scoped
`maas login <tmp-profile> ... / users read / machines read / rack-controllers read /
maas logout` sequence: *"A credential that EXISTS is not a credential that WORKS."*

Also re-read the cloud ENDPOINT (`juju show-cloud vr1-maas --client`) -- an endpoint still
pointing at the Office1 region is what killed bootstrap attempt 3 on 2026-07-30.

### C.6 -- Rebuild: the controller-tag gate

```bash
bash scripts/maas-role-tags.sh check "$DC"
maas "$MAAS_PROFILE" machines read \
  | jq -r --arg t "juju-controller-$DC" '.[] | select(.tag_names[]? == $t) | "\(.hostname) \(.status_name)"'
```
**GATE (phase-4 Step 2):** the tag exists AND returns EXACTLY ONE machine in `Ready`. Zero
-> bootstrap finds no candidate. More than one -> the target is ambiguous, STOP. A cached
`error` power state is not a fail -- refresh with `machine query-power-state` and re-read.

### C.7 -- Rebuild: bootstrap [MUTATION, gated]

```bash
juju bootstrap <maas-cloud> <controller-name> \
  --bootstrap-constraints "tags=juju-controller-$DC" \
  --constraints "tags=juju-controller-$DC" \
  --bootstrap-base ubuntu@22.04
```
**BOTH constraint flags, per the 2026-07-30 GA-R5 ruling** (operator: *"Use both flags"*) --
`--bootstrap-constraints` is what targets the machine; `--constraints` sets the model
default and honours the runbook's literal text. **`--bootstrap-base ubuntu@22.04` is
CONFIRMED against a live deploy** (the 07-30 MAAS event chain ended `Image Deployed --
deployed ubuntu/jammy/amd64/generic`).

**There is NO `--dry-run` for bootstrap** (checked) -- it is one-shot. That is why C.5 and
C.6 are gates and not formalities.

### C.8 -- >>> IMMEDIATELY AFTER BOOTSTRAP: MODEL-DEFAULTS. A NEW CONTROLLER HAS NONE. <<<

**This is the step that gets skipped, and a fresh controller makes it WORSE than Path M.4:**
model-defaults live ON THE CONTROLLER, so a rebuild loses every one of them. Everything
Path M.4 warns about applies here with nothing carried over.

```bash
juju model-defaults                                          # expect: all empty
juju model-defaults apt-http-proxy=http://<UTIL>:3142        # dc-cache-proxy.sh LISTEN
juju model-defaults snap-http-proxy=http://<UTIL>:3129       # dc-snap-proxy.sh LISTEN
juju model-defaults snap-https-proxy=http://<UTIL>:3129
# LEAVE apt-mirror UNSET -- D-135 amendment (b) converged dc0 onto the proxy
# LEAVE snap-store-proxy / -url / -assertions EMPTY (Enterprise Store, different product)
juju model-defaults                                          # READ BACK on CONTENT
```
Take `<UTIL>` from that DC's own checker output, never memory (dc0 `10.12.8.4`, dc1
`10.12.68.4`). **Defaults are inherited by NEW models only -- set them BEFORE `add-model`.**

### C.9 -- Rebuild: model, spaces, artifact source

`runbooks/dc-dc-phase4-juju-bundle-per-dc.md` **Step 3.5** owns this: `juju add-model
"$DC"`, the spaces gate (`scripts/juju-spaces-check.sh`), and this DC's artifact source.
Then **Step 3** (`preflight.sh` for this DC) and **Step 3.9** (`dc-egress-check.sh`).

### C.10 -- Rebuild: re-stage and redeploy

1. Re-stage the deploy inputs to the DC's `~/repo-stage` and **sha256-verify every one
   against repo HEAD** -- the racks carry no git clone, and a stale staged overlay has bitten
   this project twice.
2. phase-4 **Step 4.2** dry-run gate (keep `--debug`; a bare `--dry-run` prints no
   constraints), then **4.3** deploy, then **4.4 immediately** -- 4.4 is the only gate that
   reads the DEPLOYED config rather than an input to it.

### C.11 -- What a Path C rebuild does NOT restore

- **Model-defaults** -- C.8, and nothing else will remind you.
- **Controller backup** -- phase-4 Step 11 is a separate obligation against the NEW
  controller.
- **The client's known-hosts / juju SSH key** for the old controller VM: `juju` mints a new
  one at bootstrap. If a stale host key blocks you, that is the cause.
- **OWED, still not built:** a site-keyed `dc-model-defaults.sh` with a `check`, so C.8 is
  VERIFIED at every standup instead of remembered. Until it exists, C.8 is the whole control.

---

---

## Step 1 -- Back up the state file before ANY destroy [do this every time]

A `destroy` mutates state exactly like an `apply` does -- if it's
interrupted partway (network drop, `Ctrl-C`, host reboot), your only way to
know what's actually still live is the state file as it stood before you
started.

**Back up the state of the ROOT YOU ARE ABOUT TO DESTROY, on the host that
root executes from.** There is no single state file any more: the OUTER root
runs on vcloud (D-128 Plane 1, `libvirt_uri = qemu:///system`) and each DC's
INNER root runs FROM THE OFFICE1 HEADEND (D-128 Plane 2, provider =
`qemu+ssh`), which is where its state lives -- `docs/CURRENT-STATE.md` records
both DCs' inner tfstate as held inside the voffice1 working tree, gitignored,
sha256-pinned. A teardown that backed up only `opentofu/terraform.tfstate` on
vcloud has NOT backed up the substrate it is about to destroy.

```bash
# OUTER root -- on vcloud:
cd "$REPO/opentofu"
cp -a terraform.tfstate "terraform.tfstate.pre-teardown-$(date +%Y%m%d-%H%M%S)"

# INNER root -- from the OFFICE1 HEADEND, for the DC being torn down:
cd "$O1_REPO/opentofu/vr1-dc1-substrate"      # or vr1-dc0-substrate
cp -a terraform.tfstate "terraform.tfstate.pre-teardown-$(date +%Y%m%d-%H%M%S)"
```
Confirm each copy exists and is non-empty before continuing -- a `cp` whose
source was absent is the same silence as a clean backup.

Store the copy out-of-band per the operator's existing `~/vault-init/`-class
secret-handling process (it carries the same plaintext-credential risk as
the live file -- see `opentofu/README.md` "State file handling", DOCFIX-175)
-- do not just leave it sitting next to the live file as your only copy.

---

## Step 2 -- MAAS-side FIRST: census the MACHINE records this site owns [do this every time]

**DO NOT READ A `vm_host` HERE. IT IS A PERMANENT FALSE CLEAR.** The VR1 DCs
deliberately have no MAAS pod (see "Relationship to D-061" above): a
`maas <profile> vm-hosts read` returns nothing on an empty DC and on a DC with
a fully commissioned fleet ALIKE. The version of this step that keyed "skip to
Step 3, there is nothing to clean up" off that emptiness was routing an
operator straight at destroying a containment VM holding live MAAS machine
records. **The check is at the MACHINE level.**

**Why associations get read BEFORE any destroy, even though the pod mechanism
does not apply here.** MAAS deletes every machine record LINKED to a pod when
the pod is deleted -- no warning, no decompose flag -- and on 2026-07-21 an
association check that ran too late cost 9 machine records
(`runbooks/appendix-A-troubleshooting.md`, symptom "`maas ... vm-host delete
<id>` (or pod delete) silently REMOVES enlisted machines"; capture
`docs/audit/incident-20260721-pod-delete-cascade.txt`). The mechanism is gone;
the discipline it bought is not. **Read the associations first, and a
NON-EMPTY result is a STOP.**

**WHERE:** from the OFFICE1 HEADEND (D-128 Plane 2 -- the region MAAS and its
CLI profile live there). MEASURED on vcloud 2026-07-29: `command -v maas`
returns nothing. An ABSENT client is not an unreachable service and is never a
clear -- this deployment has misdiagnosed that three times
(`scripts/maas-role-tags.sh` makes the same split explicitly).

**LENS 1 -- enumerate what EXISTS (never only what is declared).**
```bash
maas "${MAAS_PROFILE:-admin}" machines read > /tmp/maas-machines.json \
  || { echo "REFUSE: cannot read MAAS machines -- 'could not look' is NOT 'nothing there'" >&2; exit 1; }

python3 - /tmp/maas-machines.json <<'PY'
import json, sys
ms = json.load(open(sys.argv[1]))
virsh = [m for m in ms if (m.get("power_type") or "") == "virsh"]
for m in ms:
    print("  %-10s %-18s %-14s power=%s" % (
        m["system_id"], m["hostname"], m.get("status_name"),
        m.get("power_type") or "-"))
print("total MAAS machine records: %d" % len(ms))
print("RECORDS REQUIRING ATTRIBUTION (power_type=virsh): %d" % len(virsh))
PY
```
**Write that second number down, and attribute every one of those records
before the gate passes.** It is a COUNT, not a sentence, precisely so it cannot
be skimmed: each `power_type=virsh` record is powered through SOME containment
VM's inner libvirt, and a destroy that has not accounted for all of them is a
destroy with an unknown blast radius. (`power_type` does come back from
`machines read` -- shipped repo code reads that exact field,
`scripts/maas-node-power.sh`.)

Attribution needs that machine's power ADDRESS, which is a per-machine
parameter: read it with `maas "${MAAS_PROFILE:-admin}" machine read
<system_id>` and take the FIELD NAME from MAAS's own current docs -- do not
invent one. Compare what you find against this site's URI from
`scripts/lib-hosts.sh` (`VIRSH_POWER_ADDRESS`), never from memory. **If you
cannot obtain the power address for a `virsh`-powered record, that is a
REFUSAL, not a clear.**

**LENS 2 -- corroborate against this site's pinned roster.**
```bash
# on the OFFICE1 HEADEND, standing in that host's clone:
O1_REPO="$(git rev-parse --show-toplevel)" \
  || { echo "REFUSE: not inside the Office1 clone -- cd there first" >&2; exit 1; }
SITE=vr1-dc1                      # the site being torn down: vr1-dc0 | vr1-dc1
source "$O1_REPO/scripts/lib-hosts.sh"
lib_hosts_select_dc "$SITE" || exit 1
: > /tmp/site-boot-macs.txt
for h in "${HOSTS[@]}"; do printf '%s\n' "${HOST_BOOT_MAC[$h]}" >> /tmp/site-boot-macs.txt; done

python3 - /tmp/maas-machines.json /tmp/site-boot-macs.txt <<'PY'
import json, sys
ms   = json.load(open(sys.argv[1]))
want = {l.strip().lower() for l in open(sys.argv[2]) if l.strip()}
if not want:
    print("REFUSE: no pinned boot MACs for this site -- refusing to read that as zero")
    sys.exit(1)
hits = []
for m in ms:
    macs = {(i.get("mac_address") or "").lower() for i in m.get("interface_set", [])}
    if macs & want:
        hits.append((m["system_id"], m["hostname"], m.get("status_name"),
                     m.get("power_type") or "-"))
print("ASSOCIATED MAAS MACHINE RECORDS for %d pinned MACs: %d" % (len(want), len(hits)))
for h in hits:
    print("  %-10s %-18s %-14s power=%s" % h)
sys.exit(1 if hits else 0)
PY
```
Exit 1 means records EXIST for this site. That is the STOP condition, not an
error to work around.

Two properties of LENS 2 to hold in mind. `lib_hosts_select_dc` REFUSES a
second, DIFFERENT site in the same shell (its stale-value guard) -- tear down
one site per shell. And it enumerates only the hosts lib-hosts DECLARES, so it
cannot report a record it has no row for (a canary enlistment, a re-enlisted
node MAAS re-named, anything added since): that is exactly why LENS 1 runs and
why it is not optional.

**If records DO exist and you intend to lose them:** that is a deliberate,
individually operator-approved decision, never a cleanup side effect. Release
or delete them via MAAS's own current documented flow FIRST, then re-run both
lenses. This repo has not exercised that path for real; MAAS's own docs are
authoritative over anything asserted here. If records are already gone while
the domains still exist, appendix-A carries the measured recovery (power the
domains on, PXE re-enlist on the pinned MACs, `scripts/maas-node-power.sh` dry
then `--commit`, re-commission -- and note MAAS mints NEW hostnames).

**GATE (all three, or STOP):**
1. LENS 1 READ SUCCEEDED, and every `power_type=virsh` record in it is
   attributed to a known containment VM.
2. LENS 2 returns ZERO associated records for the site being torn down.
3. Neither lens was skipped because a command failed. A failure is a refusal.

---

## Step 3 (Path A only) -- Pick the ROOT first, then plan the destroy inside it

**Scoping a single DC is now a ROOT choice, not a `-target` choice.** Under
D-123 Model B each DC's substrate is its own root with its own state file, and
each of those roots contains exactly ONE site -- so a plain `tofu destroy` in
`opentofu/vr1-dc1-substrate/` is already scoped to vr1-dc1, with none of the
silent-divergence risk HashiCorp's targeting warning describes. Reach for
`-target` only for a PARTIAL teardown inside one root, or in the OUTER root,
which holds all three sites at once.

Module names MEASURED 2026-07-29 (`grep -n '^module' <root>/main.tf`). **There
is no `dc1_*` module in either root** -- those names are pre-D-119 and never
existed here; a `-target` naming one aborts the plan.

| Where it lives | Root | Modules (vr1-dc1 shown; vr1-dc0 is identical with `dc0` substituted) |
|---|---|---|
| INSIDE the DC | `opentofu/vr1-dc1-substrate/` | `inner_storage`, `vr1_dc1_planes`, `vr1_dc1_wan`, `vr1_dc1_opnsense`, `vr1_dc1_node` |
| At vcloud | `opentofu/` (outer) | `vr1_dc1_storage`, `vr1_dc1_uplink`, `vvr1_dc1` |

`module.vr1_dc1_node` is a `for_each` module keyed by node name -- targeting
the module covers every instance; there are no `_node_01`/`_node_02` siblings
to enumerate. There is no `maas_vm_host` module in either root to target (Step
2). `inner_storage` is named identically in BOTH substrate roots -- it is
disambiguated by which root you are standing in, which is one more reason to
confirm the directory before every command.

**Order: INNER root first, then OUTER.** Destroying `module.vvr1_dcN` first
leaves the inner state describing objects that no longer exist and removes the
only path the inner provider can dial.

INNER root -- run FROM THE OFFICE1 HEADEND (D-128 Plane 2; the provider is
`qemu+ssh` and the state lives there). For the WHOLE DC substrate, no
`-target` at all:
```bash
cd "$O1_REPO/opentofu/vr1-dc1-substrate"
tofu plan -destroy -out=teardown-vr1-dc1-inner.tfplan
```
For a PARTIAL inner teardown, target the real names:
```bash
tofu plan -destroy \
  -target=module.vr1_dc1_node \
  -target=module.vr1_dc1_opnsense \
  -target=module.vr1_dc1_wan \
  -target=module.vr1_dc1_planes \
  -target=module.inner_storage \
  -out=teardown-vr1-dc1-inner.tfplan
```

OUTER root -- on vcloud, and ONLY after the inner half is done:
```bash
cd "$REPO/opentofu"
tofu plan -destroy \
  -target=module.vvr1_dc1 \
  -target=module.vr1_dc1_uplink \
  -target=module.vr1_dc1_storage \
  -out=teardown-vr1-dc1-outer.tfplan
```

**Do NOT target the mesh-link modules** (`mesh_vr1_dc0_vr1_dc1`,
`mesh_vr1_dc0_office1`, `mesh_vr1_dc1_office1`) or `netem_vr1_dc0_vr1_dc1` in
a single-site teardown -- each leg is SHARED with the site at its far end, and
the dc0<->office1 leg additionally CARRIES the live rack<->region transit
(MAAS, node DNS, the inner root's own `qemu+ssh` path). Only include a leg
once BOTH sites it connects are going away (see the mesh-link note after Step
5).

Review the plan line by line: expect destroys for exactly the resources above
(`libvirt_domain`, `libvirt_volume`, `libvirt_network` x6 for the planes plus
the WAN bridge network, `libvirt_pool`) and NOTHING outside this site's scope
-- in particular confirm no `office1_*`, no other DC's `vr1_dc0_*`/`vr1_dc1_*`,
and no `mesh_*`/`netem_*` resource appears.

**CAPTURE THE PRE-DESTROY BASELINE NOW**, while the containment VM is still
up -- Step 4 cannot tell "destroyed cleanly" from "could not look" without it:
```bash
source "$O1_REPO/scripts/lib-hosts.sh"; lib_hosts_select_dc vr1-dc1   # -> VIRSH_POWER_ADDRESS
virsh -c "$VIRSH_POWER_ADDRESS" list --all      > /tmp/pre-teardown-inner-domains.txt
virsh -c "$VIRSH_POWER_ADDRESS" net-list --all  > /tmp/pre-teardown-inner-nets.txt
virsh -c "$VIRSH_POWER_ADDRESS" pool-list --all > /tmp/pre-teardown-inner-pools.txt
```

**GATE:** plan matches this scope exactly; no unexpected destroys outside the
targeted site; the baseline capture above is non-empty. Do not apply a plan
you have not read.

---

## Step 4 (Path A only) -- Apply the scoped destroy [MUTATION: gated]

Apply the INNER plan from the Office1 headend, in the inner root; apply the
OUTER plan on vcloud, in `$REPO/opentofu`. Same order as Step 3: inner, then
outer.

```bash
tofu destroy teardown-vr1-dc1-inner.tfplan     # inner root, from the Office1 headend
tofu destroy teardown-vr1-dc1-outer.tfplan     # outer root, on vcloud -- AFTER the inner verify below
```
*(Note: unlike a create/apply plan, a `-destroy`-mode plan file is applied
the same way -- `tofu apply <planfile>` and `tofu destroy <planfile>` are
equivalent once the plan itself was generated with `-destroy`; use whichever
your `tofu` version's docs confirm for a plan file specifically, and if in
doubt, prefer re-running with `tofu destroy` directly rather than guessing.)*

Confirm this is the exact reviewed plan file from Step 3 (not re-planned)
before running.

**VERIFY -- and note WHICH HOST each half is verified from.** The DC's node,
edge and plane objects live inside the containment VM, so a local `virsh` on
vcloud can never see them: MEASURED on vcloud 2026-07-29, `virsh list --all`
returns exactly `vvr1-dc0`, `vvr1-dc1`, `voffice1`, `office1-opnsense`. A
vcloud-local `grep -i dc1` therefore "passes" against a fully intact DC. This
is the D-128 split: Plane 1 = the outer `qemu:///system` on vcloud; the inner
substrate is reached over `qemu+ssh` from the Office1 headend.

INNER half -- from the Office1 headend, BEFORE the outer destroy (afterwards
the URI cannot answer BY DESIGN):
```bash
source "$O1_REPO/scripts/lib-hosts.sh"; lib_hosts_select_dc vr1-dc1   # -> VIRSH_POWER_ADDRESS
virsh -c "$VIRSH_POWER_ADDRESS" version >/dev/null 2>&1 || {
  echo "REFUSE: cannot reach $VIRSH_POWER_ADDRESS. An empty list from an unreachable" >&2
  echo "        URI, a stopped containment VM or a bad key is NOT a clean destroy." >&2
  exit 1; }
virsh -c "$VIRSH_POWER_ADDRESS" list --all
virsh -c "$VIRSH_POWER_ADDRESS" net-list --all
virsh -c "$VIRSH_POWER_ADDRESS" pool-list --all
```
Take the URI from `scripts/lib-hosts.sh`, never from memory. Diff each against
the Step 3 baseline: expect exactly the planned objects gone and NOTHING else
changed. An empty list with no baseline to compare it to proves nothing.

OUTER half -- on vcloud, after the outer destroy. Assert on the named objects,
not on a substring match:
```bash
virsh list --all        # expect vvr1-dc1 GONE; vvr1-dc0, voffice1, office1-opnsense untouched
virsh net-list --all    # expect vr1-dc1-uplink GONE; the three mesh-* legs untouched
virsh pool-list --all   # expect vr1-dc1-pool GONE; vr1-dc0-pool, office1-pool, default untouched
```
(Live names measured 2026-07-29. Never `grep -i dc1` here: it matches
`vr1-dc1` AND anything else carrying the string, and it cannot distinguish
"absent" from "never visible from this host".)

---

## Step 5 (Path A only) -- Confirm no drift outside the targeted scope

```bash
cd "$REPO/opentofu"
tofu plan
```
Run WITHOUT `-target` this time -- this is the check HashiCorp's own
targeting warning (Step 3) calls for: confirm the rest of the configuration
(Office1, the OTHER DC, the mesh legs, netem) shows no unexpected changes.
Expect either "No changes" for everything outside the torn-down site, or
exactly the changes you independently expect for other in-flight work --
nothing attributable to this teardown. Do the same in the SURVIVING DC's
inner root if you used `-target` anywhere; a targeted apply's drift does not
announce itself.

**GATE:** untargeted plan shows zero unexpected drift outside the torn-down
site.

---

## Path B -- full VR1 teardown (replaces Steps 3-5 above)

Do Step 1 (back up EVERY root's state -- outer plus BOTH inner roots) and Step
2 (the machine census, run for EVERY site, not just one) first, then destroy
BOTH inner roots before the outer one -- "everything" is now three roots on two
hosts, not one `cd opentofu`:

```bash
# from the OFFICE1 HEADEND, once per DC substrate root:
cd "$O1_REPO/opentofu/vr1-dc0-substrate" && tofu plan -destroy -out=teardown-vr1-dc0-inner.tfplan
cd "$O1_REPO/opentofu/vr1-dc1-substrate" && tofu plan -destroy -out=teardown-vr1-dc1-inner.tfplan

# then, on vcloud:
cd "$REPO/opentofu"
tofu plan -destroy -out=teardown-full.tfplan
```
Review: expect a destroy for every resource each tree currently manages (check
against that root's actual uncommented module list at teardown time -- it grows
as more stages get executed, so what "everything" means changes over the
buildout, unlike Path A's fixed per-site scope).

**GATE:** each plan matches its root's full current scope exactly.

```bash
tofu destroy teardown-full.tfplan
```

**VERIFY.** Verify each inner root from the Office1 headend BEFORE the outer
destroy, using Step 4's `virsh -c "$VIRSH_POWER_ADDRESS" version` refusal guard
-- once the containment VMs are gone the inner URIs cannot answer, and
"unreachable" is not evidence of anything. Then, on vcloud:
```bash
virsh list --all
virsh net-list --all
virsh pool-list --all
```
Expect: nothing VR1-related remains (VR0's own testcloud resources, which
OpenTofu does not manage, are untouched by any of this). Note that vcloud's
`default` pool and `virbr0`/`wan` are not VR1 objects -- their survival is
correct, not residue.

Then do the equivalent of Step 5 (a final untargeted `tofu plan`) to confirm
the destroy converged state to fully empty, not partially.

---

## Mesh-link teardown (either path, handle separately)

The three D-100 mesh-link legs live in the OUTER root. Module names and the
libvirt networks they create, MEASURED 2026-07-29 (`grep -n '^module'
opentofu/main.tf`; `virsh net-list --all`):

| Module (outer root) | libvirt network | bridge, measured 2026-07-29 |
|---|---|---|
| `mesh_vr1_dc0_vr1_dc1` | `mesh-vr1-dc0-vr1-dc1` | `virbr5` |
| `mesh_vr1_dc0_office1` | `mesh-vr1-dc0-office1` | `virbr7` |
| `mesh_vr1_dc1_office1` | `mesh-vr1-dc1-office1` | `virbr3` |

(The pre-D-119 names `mesh_dc1_dc2` / `mesh_dc1_office1` / `mesh_dc2_office1`
that this section used to carry exist nowhere -- `opentofu/main.tf` holds
`moved` blocks recording exactly that rename. `virbrN` is a drifting libvirt
assignment: re-measure with `virsh net-info <network>` before using one, never
carry it from here.)

Each leg is shared between two sites. Destroy a leg only when BOTH its
endpoints are permanently going away (a full VR1 teardown, or abandoning both
of its sites together) -- never as part of a single-site scoped teardown, and
never as a reflex "clean up everything DC-adjacent" action. **`mesh-vr1-dc0-
office1` deserves extra care:** it carries the live rack<->region transit, so
MAAS, node DNS and the inner root's own `qemu+ssh` path all die with it.

`modules/netem-link` IS wired now -- `module "netem_vr1_dc0_vr1_dc1"` in the
outer root, applied 2026-07-21 against `virbr5` (the zero-traffic dc0<->dc1
leg). It has a correct `destroy`-time provisioner (`provisioner "local-exec" {
when = destroy; ... tc qdisc del ... }`, confirmed by reading
`opentofu/modules/netem-link/main.tf` directly), so destroying the
corresponding `terraform_data.netem` resource cleans up the real `tc qdisc`
rule on the vcloud host automatically; no extra manual step for that piece.
Destroy it BEFORE the leg it is attached to, and note it is a separate module
from the leg -- destroying `mesh_vr1_dc0_vr1_dc1` alone leaves the netem
resource pointing at a bridge that no longer exists.

---

## Rollback decision tree -- "a stage's `tofu apply` failed partway through, now what?"

**Read this before reaching for `tofu destroy` at all.** OpenTofu is
designed for idempotent convergence -- in most failure modes, fixing the
root cause and re-running the SAME `tofu apply` is correct and safer than
tearing down and starting over, because it preserves everything that DID
apply successfully and only retries what didn't.

**Question 0 -- WHICH ROOT failed?** There are three: the OUTER root on
vcloud, and one INNER substrate root per DC, run from the Office1 headend.
The answer decides which state file, which host, and which `virsh` endpoint
every branch below is talking about. Answer it before anything else.

**Question 0b -- is anything DEPLOYED on top?** From Stage 5 onward a failed
`tofu apply` can sit under live MAAS machine records and a live Juju model.
**No branch below reaches a destroy without Step 2's machine census passing
first.** If Step 2 returns records, the destroy is not the next action --
resolving the records is, and that is an operator-gated decision on its own.

**Verb discipline, because these read alike under time pressure:** `virsh
destroy vvr1-dcN` is a forced POWER-OFF, reversible with `virsh start`, and it
leaves the domain, its disk and every MAAS record intact -- that is the D-122
site-down lever and it is a legitimate first move. `tofu destroy` of
`module.vvr1_dcN` UNDEFINES the domain and DELETES its disk, taking the whole
inner fleet with it, irreversibly. Reaching for the second when you meant the
first is the expensive mistake this runbook exists to prevent.

1. **`tofu apply` errored before creating anything (e.g., `tofu init`/
   `validate`/`plan` failed, or `apply` failed on its very first resource).**
   Nothing was created. Fix the root cause (bad variable, unreachable
   provider, syntax error) and re-run from that stage's own `tofu plan`
   step. No teardown needed.
2. **`tofu apply` created SOME resources, then failed on a later one.**
   `tofu plan` (no `-destroy`) immediately to see OpenTofu's own view of
   what's now live vs. what the config wants. Default to FIX-FORWARD: fix
   whatever caused the failure (a bad node-VM memory size, an
   unreachable MAAS endpoint, a real infra problem) and re-run
   `tofu apply` for that same stage -- OpenTofu will create only what's
   still missing, leaving successfully-created resources alone. Reserve a
   teardown (Path A, scoped to that site) for when the partial state is
   itself broken in a way fix-forward can't reach (e.g., a domain stuck in
   a bad libvirt state that manual `virsh` intervention already tried and
   failed to fix).
3. **You suspect `terraform.tfstate` itself is corrupted or lost** (this is
   the scenario `opentofu/README.md`'s "State file handling" section
   already flags as the real cost of no backend/backup). There is no clean
   recovery path documented in this repo: either a manual `tofu import` per
   still-live resource (tedious, must be done for every domain/network/pool
   by hand) or accepting a full teardown-and-rebuild from Step 1 of the
   affected stage. This is exactly why Step 1's pre-teardown backup habit
   matters even outside a deliberate teardown -- back up
   `terraform.tfstate` after every successful stage apply too, not only
   before a teardown.
4. **A resource exists in reality but OpenTofu's state doesn't know about
   it** (e.g., a `virt-install`-created VM from Stage 2's Option B manual
   path, which OpenTofu never created and cannot destroy). This runbook's
   `tofu destroy`/`plan -destroy` commands only ever touch what OpenTofu
   itself manages -- anything created via a manual/Option-B path needs
   manual teardown (`virsh destroy`/`undefine`, etc.) by the same operator
   discipline that created it. Do not expect `tofu destroy` to find or
   remove it.
5. **The failure is INSIDE a DC and you are tempted to reach for the
   containment VM.** Taking the site down (`virsh destroy vvr1-dcN`) is
   reversible and is often the right move to stop a cascade. Destroying the
   containment VM through the OUTER root is not a rollback of the inner
   apply -- it discards the entire inner substrate, leaves the inner state
   file describing objects that no longer exist, and leaves MAAS holding
   machine records whose `power_address` will never answer again. Fix-forward
   in the INNER root first; if you truly need the DC gone, do Step 2, then the
   inner root, then the outer -- in that order.

---

## Verification (of this runbook's own accuracy, not a live run)

Every `tofu`/`terraform` flag cited above (`-destroy`, `-target`, the
targeting-risk warning) was checked against HashiCorp's own current CLI
documentation before being written here (fetched 2026-07-10), not asserted
from memory -- same discipline as `opentofu/README.md`'s other research
sections. `netem-link`'s destroy-time provisioner was confirmed by reading the
actual `.tf` file, not assumed.

**CORRECTION 2026-07-29:** this paragraph previously vouched for
"`dc1_maas_vm_host`'s lack of a Terraform-expressible dependency on node-VM
domains" as something confirmed by reading the `.tf` files. No such module has
ever existed in either root, so that was a verification claim about nothing --
the false-verification class that lets a wrong instruction look checked. All
module, network, pool and domain names in this runbook were RE-MEASURED against
the tree and the live hosts on 2026-07-29 (`grep -n '^module' <root>/main.tf`;
`virsh list/net-list/pool-list --all`; `virsh net-dumpxml`). What is NOT
verified: this runbook's commands have never been run against a real `tofu`
binary or a real state file (same SCAFFOLD/UNVALIDATED status as the rest of
`opentofu/`) -- treat every command here as reviewed-but-unexercised until
`scripts/opentofu-validate.sh` and a real dry run confirm otherwise.
