# VR1 DC-DC teardown / rollback runbook Scope: tearing down or rolling back the infrastructure `opentofu/` provisions for the VR1 (DC-DC) buildout -- the `vr1-dc0` / `vr1-dc1` / Office1 plane and local networks, storage pools, node-VM domains, containment VMs, OPNsense edge VMs, and the D-100 mesh-link triangle. (It once also claimed MAAS `vm_host` registrations; there are none -- see Step 2. The `dc1`/`dc2` names it was written with are pre-D-119 and mean different datacenters now.) Written 2026-07-10 (DOCFIX-175 gap-register item #19) because nothing else in this repo covers this layer: `runbooks/phase-00-teardown-release.sh` / `phase-00-teardown-destroy.sh` (D-061) tear down the VR0 **juju/MAAS-machine** layer, a different pairing of tools and a different failure mode entirely -- see "Relationship to D-061" below for why that mechanism does not just transfer over. **UNVALIDATED, same caveat as everywhere else in `opentofu/`:** no `tofu` binary has been available in any session that authored this tree. Every command below is standard, well-documented OpenTofu/Terraform CLI syntax (verified against HashiCorp's own current docs where cited), not invented -- but run `bash scripts/opentofu-validate.sh` on a machine with the real binary and read the result before trusting any of it against a live state file. --- > ## !!! MODEL B RESHAPE (D-123, 2026-07-16) -- READ BEFORE ANY DC TEARDOWN; this runbook's body predates it !!! > > This runbook was written (2026-07-10) for the SINGLE-ROOT shape (OpenTofu creates the DC node-VM > domains / planes / edge directly on the **vcloud** host, and `maas-vm-host` registers the vcloud > host). Under **D-123 Model B** that is no longer how a DC is built, so the resource-by-resource > mechanics below are partly WRONG for the DCs. Current shape: > > - **TWO roots / TWO state files PER DC, and there are now TWO DCs.** The OUTER root (`opentofu/`, > vcloud) owns `vvr1-dc0` and `vvr1-dc1` (the DC containment VMs), the office1<->dcN transits, and > the **D-125** vcloud ISP uplinks (`vr1-dc0-uplink` / `vr1-dc1-uplink` NAT + each containment VM's > uplink NIC / `br-vr1-dcN-wan` netplan bridge). Each INNER root > (`opentofu/vr1-dc0-substrate/`, `opentofu/vr1-dc1-substrate/`; provider = `qemu+ssh` to that DC's > containment VM) owns everything INSIDE its DC -- the node VMs, the 6 planes, `vr1-dcN-wan`, the > OPNsense edge, the inner pool. So the "node-VM domains / planes / edge" this runbook talks about > live on **the containment VM's libvirt, not vcloud's**. > - **There is NO MAAS `vm_host`/pod for any VR1 DC.** MAAS holds per-MACHINE records with > `power_type=virsh` (D-103/D-123 amendments, ruled 2026-07-20) and `modules/maas-vm-host` is > instantiated in NEITHER root -- measured 2026-07-29. Any instruction anywhere to "remove the > `maas-vm-host` record before destroying the containment VM" is VOID: there is no such record, > the read returns nothing on a DC with a full fleet enrolled, and acting on that emptiness > destroys the containment VM out from under live MAAS machine records. **Step 2 is the real > check.** > - **Whole-DC site-down is ONE object: `virsh destroy vvr1-dcN`** (the D-122/D-123 point). Destroying > the containment VM instantly takes down the entire inner fleet -- no per-domain group destroy > needed. That is the fast path for "abandon this DC and rebuild." **Know which verb you are > using:** `virsh destroy` is a forced POWER-OFF and is REVERSIBLE (`virsh start vvr1-dcN`); the > domain, its disk and every MAAS record still exist. `tofu destroy` of `module.vvr1_dcN` UNDEFINES > the domain and deletes its disk -- irreversible, and it takes the whole inner fleet with it. > Under a failed Stage 5 those are not interchangeable. > - **Teardown ORDER (two roots):** tear down the INNER root first (`cd opentofu/vr1-dcN-substrate && > tofu destroy`, or just `virsh destroy vvr1-dcN` if you are only taking the site DOWN), THEN the > OUTER root. The D-061-style "clean up MAAS's record before destroying the libvirt underneath" > principle below still applies -- and with no pod to remove, "MAAS's record" means the per-machine > records enumerated in Step 2. Run Step 2 BEFORE either root's destroy. > - **Reverting Model B -> Model A** (a different operation from teardown) is > `docs/archive/model-a-fallback-plan.md` (git tag `model-a-fallback`), not this runbook. > > Until the body below is rewritten to the two-root shape, treat its DC-substrate steps as > **single-root reference** and apply the mapping above. The Office1 (`voffice1`) and mesh/pool steps > are unaffected. --- ## When to use this runbook - **Mid-buildout failure recovery.** A stage's `tofu apply` failed partway through, or created something wrong, and you need to know whether to fix- forward or tear down and retry. See "Rollback decision tree" below -- READ THAT FIRST, since fix-forward is usually the right answer, not a destroy. - **Deliberate re-run from clean.** You want to abandon a DC's (or Office1's, or the whole VR1 layer's) current state and rebuild from Step 1 of its stage runbook. - **NOT for**: tearing down VR0's juju/MAAS-machine layer (use D-061's scripts), or removing a single juju application/relation from an already- deployed OpenStack bundle (that is normal juju operations, not this runbook). --- ## Relationship to D-061 (read before mixing the two layers) D-061's core lesson: destroying/releasing a resource at the WRONG layer, in the wrong order, orphans state at the OTHER layer -- `juju destroy-model` released MAAS-pod-composed machines and MAAS auto-decomposed them, losing the carve; the fix was `remove-machine --keep-instance` FIRST (detach from juju without releasing), verify survival in MAAS, THEN act on the model. The same PRINCIPLE applies here, but the mechanism is different because the layers being coordinated are different: - D-061 coordinates **juju's machine view** against **MAAS's machine view** of hosts MAAS itself pod-composed. - This runbook coordinates **OpenTofu's resource view** (libvirt domains/ networks/pools it created directly) against **MAAS's per-MACHINE view** of those same domains. MEASURED 2026-07-29: `modules/maas-vm-host` is instantiated in NEITHER root (`grep -n 'maas_vm_host' opentofu/*.tf opentofu/vr1-dc0-substrate/*.tf opentofu/vr1-dc1-substrate/*.tf` returns only comments), and the `provider "maas"` block those modules would need is still deliberately absent (`opentofu/main.tf:9-20`, DOCFIX-179). MAAS knows the DC node VMs because they PXE-ENLISTED, and it powers them per machine with `power_type=virsh` (D-103/D-123 amendments, ruled 2026-07-20 -- `scripts/maas-node-power.sh`'s header carries the two measurements that refuted the pod). Because this repo uses neither MAAS pod-composition NOR a `vm_host` for its OpenTofu-created VMs, the SPECIFIC `--keep-instance` decompose-on-release failure mode D-061 diagnosed does not directly apply here. The GENERAL principle still does: **clean up MAAS's record of a host before destroying the libvirt resource underneath it**, not after -- destroying the libvirt domain/ host first would leave MAAS pointing at a `power_address` that no longer answers, an orphaned-but-not-obviously-broken record rather than a clean removal. **What this section used to say, and why it is now a hazard rather than a caveat:** it stated that no VR1 DC had reached Stage 4, so no VR1 libvirt host had ever had real MAAS-enrolled machines under it. **That is false as of 2026-07-23** (`docs/CURRENT-STATE.md`: 18 nodes READY, 9 per DC, both DCs commissioned; `power_type=virsh` set and proven by a real `query-power-state`), and the per-DC Juju controller VMs were enlisted on top of that in July 2026. Anything downstream of "nothing is enrolled yet" -- including the old Step 2 -- must be treated as REVERSED, not merely stale. The exact MAAS-side release/delete commands for a machine whose libvirt domain is about to disappear are STILL not worked out or tested in this repo: Step 2 states the principle, enumerates the records, and points at MAAS's own current docs for the flags. Do not invent flags for a scenario this repo has never actually exercised. --- ## TWO CLONES, TWO HOSTS -- set these once, before anything below Every block in this runbook runs on one of two hosts (D-128), each with its own clone of this repo. The same `cd` pasted on the wrong one is the foot-gun this section exists to remove, so the two are named differently throughout: - **`$REPO` -- the VCLOUD clone.** Plane 1: the OUTER root (`$REPO/opentofu`, `libvirt_uri = qemu:///system`) and local `virsh`. - **`$O1_REPO` -- the OFFICE1 HEADEND clone, on `voffice1`.** Plane 2: BOTH DC substrate roots (`$O1_REPO/opentofu/vr1-dc0-substrate`, `.../vr1-dc1-substrate`, provider = `qemu+ssh`), their state files, the `maas` CLI, and every `virsh -c qemu+ssh://...` into a containment VM. MEASURE `$O1_REPO` with `ls` on the headend -- do not assume it, and do not assume the inner state lives on vcloud (it does not). --- ## Two paths, by intent (mirrors D-061's release-vs-destroy split) - **Path A -- scoped teardown.** Tear down ONE site's resources (a single DC, or Office1), leaving the other sites' infrastructure untouched. **Scope by choosing the ROOT first** -- each DC substrate root contains exactly one site, so a plain `tofu destroy` there is already scoped and needs no `-target` at all. `-target` is reserved for the OUTER root (which holds all three sites) and for a partial teardown inside one root. Where you do use it, HashiCorp's own docs flag it as an exceptional-circumstances tool, not for routine use, because targeted applies can leave configuration and state silently diverging: *"This targeting capability is provided for exceptional circumstances, such as recovering from mistakes or working around Terraform limitations. It is not recommended to use `-target` for routine operations, since this can lead to undetected configuration drift"* (developer.hashicorp.com/terraform/cli/commands/plan, fetched 2026-07-10). A scoped teardown IS exactly that kind of exceptional circumstance -- acceptable here, but ALWAYS follow a targeted apply with a full (untargeted) `tofu plan` afterward (Step 5) to confirm the rest of the configuration didn't drift. - **Path B -- full VR1 teardown.** Abandon everything OpenTofu manages for VR1. That is now THREE roots across TWO hosts -- both DC substrate roots from the Office1 headend, then the outer root on vcloud -- each a plain `tofu destroy` (no `-target`), letting OpenTofu compute the safe order WITHIN each root from its own resource graph. The order BETWEEN roots is yours to get right: inner before outer, always. Both paths share Steps 1 (state backup), 2 (the MAAS machine census) and 5 (final verify). Path A is Steps 2-5; Path B replaces Steps 3-5 with the per-root destroys described in its own section (see "Path B" section after Step 5). --- ## Path M -- JUJU MODEL teardown (the innermost layer; NO substrate is touched) **ADDED 2026-08-02 on operator direction: "this has bit in the past and will be a reusable and needed set for the future."** Everything here is MEASURED from real runs on this cloud -- the 2026-07-31 VR1 dc0 teardown (`docs/CURRENT-STATE.md`, "MODEL TEARDOWN 2026-07-31") and the 2026-08-02 rebuild. It is NOT derived from `juju help`; where the two disagree, the help text lost, and that is recorded below. **USE THIS WHEN** you want to re-deploy the bundle into a clean model: a failed or half-installed deploy, a config defect that reached the charms, a bundle change you want applied from scratch. **It is NOT a substrate teardown** -- Paths A and B above are a different, much larger action. If the fix is a deploy input, you almost certainly want Path M and nothing else. ### M.0 -- What Path M does and does NOT touch | Touched | NOT touched | |---|---| | the `` model: every unit, container, and machine assignment | the `controller` model and the D-104 controller VM | | model-level config (see the M.4 trap) | the MAAS region, its database, and the machine RECORDS | | | the rack host, the OPNsense edge, both proxies, the tofu substrate, the other DC | `juju destroy-model`'s own summary scopes it: *"Terminate all machines/containers and resources for a **non-controller model**."* The controller model is out of reach by design. ### M.1 -- Pre-flight: prove the blast radius, every time ```bash juju storage -m "$DC_MODEL" # storage instances -- decides the storage flag juju offers -m "$DC_MODEL" # cross-model offers -- non-empty = another model consumes this one juju models # confirm which models exist and their machine counts ``` - **A storage flag is required ONLY if `juju storage` is non-empty.** MEASURED 2026-08-02 on `vr1-dc0`: *"No storage to display."* `ceph-osd`'s `osd-devices` are RAW BLOCK DEVICES set by charm config, NOT juju storage instances -- so `--destroy-storage` destroys nothing here and must not be passed as a reflex. MAAS re-partitions those disks at re-image. - **Non-empty `juju offers` means STOP** and work out what consumes the endpoint first. ### M.2 -- The command [MUTATION: gated] ```bash juju destroy-model "$DC_MODEL" --force --no-wait --no-prompt ``` **`--no-wait` IS PART OF THE PROVEN PATH. DO NOT DROP IT ON A FAILED DEPLOY.** MEASURED 2026-07-31: a plain `destroy-model` **STALLED and would not self-resolve** -- `attempt 30 to destroy model failed (will retry): model not empty, found 26 machines, 37 applications`, flat for ~19 minutes with the application set BYTE-IDENTICAL across a 12-minute diff. **Mechanism, which makes the stall terminal rather than slow: all 26 machine/container agents were `stopped`, so no teardown hook could execute at all** -- the destroy worker asks and nothing answers. Units already in `error` (11 haclusters at `hook failed: "stop"`, four at `install`, five at `identity-service-relation-departed`) could never run their stop hooks, so it was never going to drain. **`--force --no-wait` cleared it: 18 -> 5 -> 2 machines, then `Model destroyed.`** Reading `juju help destroy-model` alone would talk you OUT of `--no-wait` ("Rush through model destruction without waiting for each individual step to complete"). On a model whose agents are already stopped there is nothing to wait FOR. Trust the measurement. **>>> BUT `--force` CAN ORPHAN THE MODEL AND TAKE THE WHOLE CONTROLLER DOWN. MEASURED 2026-08-03. READ THIS BEFORE USING IT. <<<** The 2026-08-02 dc0 teardown reported `Model destroyed.` EXIT 0 with a clean progressive drain and correct MAAS release -- and still left the model in an INCONSISTENT state: the model document survived while its STATUS document did not. Consequences, in order: 1. `juju models` shows the model at `life: dead`; **`juju add-model ` refuses with `model "" for admin already exists`.** 2. The controller's `undertaker` worker crash-loops on `cannot destroy cloud resources: cannot set status: model not found`, on a growing backoff. **It never self-resolves** -- the error is deterministic. 3. **Worst: the `modelcache` worker crash-loops on `failed to initialise backing for models:: status doc ":e" not found, getting new watcher`** -- hundreds of iterations per minute (732 logged in ~12 min). The API server DEPENDS on modelcache, so **port 17070 goes to `connection refused` and the CONTROLLER IS DOWN.** The VM is up and pings; `systemctl is-active jujud-machine-0` reads `active`. Only the log and the refused port tell you. **A `jujud` restart does NOT fix it** (tried; the worker re-enters the same loop). This is the LP #1737487 class -- "dead model did not get removed, and cannot be removed". Upstream is explicit that `--force` is a LAST RESORT precisely because it can leave state juju cannot reconcile. **MITIGATION, in order of preference:** - **Try WITHOUT `--force` first** when the model's agents are still ALIVE. The 08-02 run drained progressively (36 -> 26 -> 20 -> 16 -> 9 -> 0) with hooks executing, i.e. force was not actually needed; it was carried over from the 07-31 stall where agents were `stopped` and force genuinely was the only way through. - **Reserve `--force --no-wait` for the measured stall condition** -- agents `stopped`, destroy flat for many minutes, units unable to run teardown hooks. - **DO NOT re-issue `destroy-model` against a model already at `life: dead`.** Measured 2026-08-03: it moved the model BACK to `dying` and re-armed the undertaker loop. It did not help and it changed state. - **If the controller is already down on this symptom, REBUILD THE CONTROLLER** rather than attempting juju state-DB surgery. A controller hosting only a destroyed model has nothing worth preserving, and `phase-4` Step 2's bootstrap is a tested path. **`--timeout` is the one to leave off:** the help warns that `--force` WITH `--timeout` continues "without consideration or respect for clean shutdown or resource cleanup". `--no-prompt` is required non-interactively (the help's Details prose still says `-y`; that is STALE -- the flag list is authoritative). ### M.3 -- THE RELEASE PATH: what happens to the MAAS machines **MEASURED, BOTH RUNS -- juju releases the machines and MAAS returns them to `Ready`. NO `maas machine release` IS NEEDED AND NONE WAS RUN.** After the 2026-07-31 forced destroy, read back: **all NINE role nodes `Ready / owner=None`, zero stranded.** - **`enable_disk_erasing_on_release` is `false`** on this region (verify per DC: `maas maas get-config name=enable_disk_erasing_on_release`), so release does NOT trigger an erase cycle -- nodes return to `Ready` in minutes and are re-imaged by the next deploy. During the 07-31 run the six nodes juju did release reached `Ready / owner None` in minutes; **MAAS was never the bottleneck**, juju simply had not yet issued a release for the other three. - **EXPECT the controller VM to stay `Deployed`.** `subtle-grouse` remains `Deployed` under `juju-vr1-dc0` because it belongs to the `controller` model, not the destroyed one. **The correct post-state is 9 `Ready` + 1 `Deployed`, NOT 10 `Ready`** -- a session expecting all ten to release will raise a false alarm. - **THE CASCADE SIGNATURE TO STOP ON: a DROP in the MAAS machine COUNT.** Records are released, never deleted. This repo has a real cascade (2026-07-21, a MAAS pod delete taking its linked machine records), which is why the count is checked and not assumed. - **The VR0 pod warning does NOT apply here.** `runbooks/phase-00-teardown-maas-reset.md` records that on VR0's virsh-POD MAAS, `destroy-model` DECOMPOSES pod-composed machines (observed 3x) -- there, use the D-061 pair instead. **VR1 uses per-machine `power_type=virsh` and NOT pods (D-103/D-123 amendments), so there is nothing to decompose**, which both VR1 teardowns confirm. Do not import the VR0 procedure. ### M.4 -- >>> THE TRAP: `destroy-model` TAKES THE MODEL CONFIG WITH IT <<< **This bit on 2026-07-31 and it will bite again.** Destroying the model destroys its model-config. Step 3.5's `apt-mirror` and the spaces work vanished with it, **and nothing in the repo would have caught it** -- the next deploy fails on package fetches and reads like an artifact-service fault rather than a missing model setting. **Anything the deploy needs at the MODEL level must be re-established BEFORE the next `add-model`, or set as a CONTROLLER MODEL-DEFAULT so it survives.** ```bash juju model-defaults # what a NEW model will inherit juju model-defaults = # set (per-controller, so per-DC by D-104) ``` **Model-defaults are inherited by NEW models ONLY -- never applied retroactively**, so they must be right BEFORE `add-model`, not after. **MEASURED 2026-08-02, and it is a live trap for the very next rebuild:** this controller's defaults carry **`apt-mirror = http://10.12.8.4/ubuntu` and nothing else**. After the D-135 amendment (b) convergence, that value is WRONG -- dc0 now consumes the apt caching proxy -- and the three settings the deploy actually needs (`apt-http-proxy`, `snap-http-proxy`, `snap-https-proxy`) are **absent from the defaults and were set at MODEL level**, so the destroy takes them. A rebuild that skips this step comes back pointing at the retired mirror with no proxies at all. Per-DC checklist before `add-model`, each read back on CONTENT: ```bash juju model-defaults --reset apt-mirror # convergence: proxy, not mirror juju model-defaults apt-http-proxy=http://:3142 # dc-cache-proxy.sh LISTEN juju model-defaults snap-http-proxy=http://:3129 # dc-snap-proxy.sh LISTEN juju model-defaults snap-https-proxy=http://:3129 # LEAVE snap-store-proxy / -url / -assertions EMPTY (Enterprise Store, different product) ``` Take `` from that DC's own checker output, never from memory (dc0 `10.12.8.4`, dc1 `10.12.68.4`). **OWED, and still not built: a site-keyed `dc-model-defaults.sh` with a `check`, so these are VERIFIED at every standup rather than remembered.** ### M.5 -- Post-checks, before rebuilding anything ```bash juju models # (a) only `controller` remains maas machines read | ... # (b) COUNT unchanged; 9 role nodes Ready, controller VM Deployed bash scripts/dc-cache-proxy.sh check # (c) artifact + snap services survived bash scripts/dc-snap-proxy.sh check ``` **(b) is the one that catches a cascade** -- a drop in count means STOP, do not rebuild. **>>> SECOND INSTRUMENT WARNING, 2026-08-03: DO NOT TEST FOR THE MODEL'S ABSENCE WITH A GREP THAT CANNOT TELL "GONE" FROM "CANNOT LOOK". <<<** A wait-loop polling `juju models --format=yaml | grep -q "name: admin/"` and breaking when the grep MISSES reported **"REAPED"** -- while the model still existed and the CONTROLLER API WAS DOWN. With the API refused, `juju models` prints NOTHING and exits, the grep matches nothing, and an absence-test reads that as success. It is the "could not look is never nothing there" rule, in a checker written in the same session that wrote this runbook. **Poll on a POSITIVE signal instead:** require `juju models` to actually RETURN model lines, and only then decide whether the target is among them; treat empty output or a non-zero juju exit as REFUSE (evaluate again), never as "gone". Watch the pipeline too -- `cmd | head -3; rc=$?` captures `head`'s status, not the command's. **INSTRUMENT WARNING, learned by getting it wrong (2026-07-31): during a teardown, `juju status -m ` is the instrument; `juju models` COUNTS ARE NOT.** The summary columns ZERO OUT while the model is `destroying` -- they read "0 machines / 0 units" while `juju status` simultaneously showed 26 machines and 37 applications. The real tell (three control nodes stuck `Deployed`) was visible and was explained away as "more containers to work through". Do not read progress from `juju models`. ### M.6 -- Rebuild sequence after a Path M teardown 1. M.5 post-checks all green. 2. **M.4 model-defaults corrected** -- this is the step that gets skipped. 3. `juju add-model ` (inherits the corrected defaults), then re-apply the spaces work. 4. Re-stage the deploy inputs to the DC's `~/repo-stage` and **sha256-verify against repo HEAD** before trusting them (the racks carry no git clone). 5. `phase-4` Step 4.2 dry-run gate, then 4.3 deploy, then **4.4 immediately** -- 4.4 is the only gate that reads the DEPLOYED config rather than an input to it. --- ## Path C -- JUJU CONTROLLER teardown and rebuild (per DC) **ADDED 2026-08-03.** Use when the DC's Juju controller itself is unusable -- not the model. **The trigger that produced this section:** a `--force` model destroy orphaned a model document without its status document, which crash-looped `modelcache`, which the API server depends on, so port 17070 went to `connection refused` (Path M.2's warning block). Also valid for a corrupted controller state DB, a lost controller VM, or a controller that cannot be upgraded. **PREFER THIS OVER JUJU STATE-DB SURGERY.** A controller whose only model has been destroyed holds nothing worth preserving. Mongo surgery on juju's state is unsupported and riskier than a rebuild along a path this repo has already exercised. **NOTHING BELOW TOUCHES:** the MAAS region or its database, the machine records, the rack, the edge, the artifact/snap proxies, the tofu substrate, or the other DC. The nine role nodes stay `Ready` throughout and are re-imaged only when the bundle is redeployed. ### C.1 -- Census FIRST: write down what must survive ```bash maas machines read | jq -r '.[] | "\(.hostname) \(.status_name) \(.owner // "None")"' bash scripts/dc-cache-proxy.sh check # via the rack; see the header for transport bash scripts/dc-snap-proxy.sh check ``` Record the machine COUNT. **A drop in that count at any later step is the cascade signature (2026-07-21 incident) and means STOP.** Expected steady state through the whole of Path C: nine role nodes `Ready / owner=None`, plus the controller VM (`Deployed` before C.2, `Ready` after). ### C.2 -- Tear the controller down [MUTATION, gated] **Pick by whether the API answers.** Test it, do not assume: ```bash timeout 8 bash -c "/17070" && echo API-UP || echo API-DOWN ``` - **API UP** -> `juju destroy-controller --destroy-all-models --no-prompt` **This command is in the committed DENY list** (`.claude/settings.json`), so it is an OPERATOR-RUN command by policy. That is deliberate: it is the single most destructive juju verb. - **API DOWN** -> **`juju kill-controller` DOES NOT WORK. MEASURED 2026-08-03; DO NOT WASTE TEN MINUTES ON IT.** Its help implies a direct-to-provider fallback ("Timeout before direct destruction", `-t` default `5m0s`), and the first version of this section said so. **It does not reach that fallback.** Run against a controller whose API is refused, with `--no-prompt`, it waited ~10 minutes and then exited 1: ``` Unable to open API: open connection timed out ERROR cannot connect to model config API: unable to connect to API: dial tcp :17070: connect: connection refused ``` It needs the MODEL CONFIG API to decide what to destroy, so an unreachable API defeats it before the timeout matters. **Nothing was destroyed and nothing cascaded** -- the MAAS census was identical before and after -- so it is safe, just useless. **USE THE MANUAL PAIR INSTEAD (C.2b).** **>>> POLICY GAP, LOGGED 2026-08-03: `juju kill-controller` is NOT in the deny list while `juju destroy-controller` IS.** `kill-controller` is the STRONGER command. The committed policy gates the weaker one and lets the stronger through. Treat `kill-controller` as operator-gated regardless of what the rule engine does, and see `docs/security-ledger.md` SEC-030 for why the ask/deny rules are not the real gate here -- the presentation discipline is. ### C.2b -- API DOWN: the manual pair (juju client, then MAAS) When `kill-controller` cannot work, the controller is removed in two independent halves. This is the D-061 coordination principle applied to the controller: **clean up juju's view first, then MAAS's**, never the reverse. **(1) Drop the client's registration -- SAFE, no cloud resource touched.** ```bash juju controllers # confirm the name juju unregister --no-prompt # edits local juju client config ONLY ``` This does not release the VM. After it, MAAS still shows the controller VM `Deployed` and OWNED by the juju service user -- that is expected, not a leak. **(2) Release the controller VM in MAAS [MUTATION, gated].** Identify it by TAG, never by hostname -- MAAS auto-generated hostnames do not match the ruled names, and the tag is the stable key: ```bash maas machines read \ | jq -r --arg t "juju-controller-$DC" \ '.[] | select(.tag_names[]? == $t) | "\(.hostname) \(.system_id) \(.status_name) \(.owner // "None")"' maas machine release ``` **GATE before releasing:** the query returns EXACTLY ONE machine, it is the controller VM, and it is NOT one of the nine role nodes (cross-check the role nodes are `Ready / owner=None` and stay that way). **After:** it reads `Ready / owner=None`, the MAAS machine COUNT is unchanged, and the tag is still on it -- the tag is what C.6/C.7 bootstrap against, so a release that lost the tag means STOP. `enable_disk_erasing_on_release=false` on this region, so release is prompt and the disk is re-imaged at the next bootstrap rather than wiped now. ### C.3 -- Verify the teardown released, and did not cascade ```bash maas machines read | jq -r '.[] | "\(.hostname) \(.status_name) \(.owner // "None")"' ``` **GATE:** machine COUNT unchanged from C.1; the controller VM now reads `Ready / owner=None` alongside the nine role nodes. Records are RELEASED, never deleted -- a count drop is the cascade signature. Expect NO `maas machine release` to be necessary; juju's own release path does it (measured, both prior teardowns). ### C.4 -- Clean the client side ```bash juju controllers # is the dead controller still registered? juju unregister --no-prompt # client-side ONLY; removes no cloud resource ``` Skip if `kill-controller` already removed it. `unregister` is safe: it edits the local juju client config and nothing else. ### C.5 -- Rebuild: the credential gate, REGION-SCOPED **Do not re-mint blindly.** Follow `runbooks/dc-dc-phase4-juju-bundle-per-dc.md` **Step 2.0** verbatim -- it carries the three-outcome gate and DOCFIX-206's correction that the check must name the REGION explicitly. The trap it exists for: under D-132 q1 each DC has its own region with its own database, so `juju credentials --client` can list a credential whose MAAS user lives in a DIFFERENT region. Prove the region with `scripts/maas-profile-assert.sh` (rack-controller identity -- a machine count is not proof) before trusting any "already exists". **A controller rebuild does NOT invalidate the MAAS credential** -- the credential belongs to the CLOUD definition in the juju CLIENT, not to the controller. If it authenticated before, it still does. Prove it rather than assume, with Step 2.0's scoped `maas login ... / users read / machines read / rack-controllers read / maas logout` sequence: *"A credential that EXISTS is not a credential that WORKS."* Also re-read the cloud ENDPOINT (`juju show-cloud vr1-maas --client`) -- an endpoint still pointing at the Office1 region is what killed bootstrap attempt 3 on 2026-07-30. ### C.6 -- Rebuild: the controller-tag gate ```bash bash scripts/maas-role-tags.sh check "$DC" maas "$MAAS_PROFILE" machines read \ | jq -r --arg t "juju-controller-$DC" '.[] | select(.tag_names[]? == $t) | "\(.hostname) \(.status_name)"' ``` **GATE (phase-4 Step 2):** the tag exists AND returns EXACTLY ONE machine in `Ready`. Zero -> bootstrap finds no candidate. More than one -> the target is ambiguous, STOP. A cached `error` power state is not a fail -- refresh with `machine query-power-state` and re-read. ### C.7 -- Rebuild: bootstrap [MUTATION, gated] ```bash juju bootstrap \ --bootstrap-constraints "tags=juju-controller-$DC" \ --constraints "tags=juju-controller-$DC" \ --bootstrap-base ubuntu@22.04 ``` **BOTH constraint flags, per the 2026-07-30 GA-R5 ruling** (operator: *"Use both flags"*) -- `--bootstrap-constraints` is what targets the machine; `--constraints` sets the model default and honours the runbook's literal text. **`--bootstrap-base ubuntu@22.04` is CONFIRMED against a live deploy** (the 07-30 MAAS event chain ended `Image Deployed -- deployed ubuntu/jammy/amd64/generic`). **There is NO `--dry-run` for bootstrap** (checked) -- it is one-shot. That is why C.5 and C.6 are gates and not formalities. ### C.8 -- >>> IMMEDIATELY AFTER BOOTSTRAP: MODEL-DEFAULTS. A NEW CONTROLLER HAS NONE. <<< **This is the step that gets skipped, and a fresh controller makes it WORSE than Path M.4:** model-defaults live ON THE CONTROLLER, so a rebuild loses every one of them. Everything Path M.4 warns about applies here with nothing carried over. ```bash juju model-defaults # expect: all empty juju model-defaults apt-http-proxy=http://:3142 # dc-cache-proxy.sh LISTEN juju model-defaults snap-http-proxy=http://:3129 # dc-snap-proxy.sh LISTEN juju model-defaults snap-https-proxy=http://:3129 # LEAVE apt-mirror UNSET -- D-135 amendment (b) converged dc0 onto the proxy # LEAVE snap-store-proxy / -url / -assertions EMPTY (Enterprise Store, different product) juju model-defaults # READ BACK on CONTENT ``` Take `` from that DC's own checker output, never memory (dc0 `10.12.8.4`, dc1 `10.12.68.4`). **Defaults are inherited by NEW models only -- set them BEFORE `add-model`.** ### C.9 -- Rebuild: model, spaces, artifact source `runbooks/dc-dc-phase4-juju-bundle-per-dc.md` **Step 3.5** owns this: `juju add-model "$DC"`, the spaces gate (`scripts/juju-spaces-check.sh`), and this DC's artifact source. Then **Step 3** (`preflight.sh` for this DC) and **Step 3.9** (`dc-egress-check.sh`). ### C.10 -- Rebuild: re-stage and redeploy 1. Re-stage the deploy inputs to the DC's `~/repo-stage` and **sha256-verify every one against repo HEAD** -- the racks carry no git clone, and a stale staged overlay has bitten this project twice. 2. phase-4 **Step 4.2** dry-run gate (keep `--debug`; a bare `--dry-run` prints no constraints), then **4.3** deploy, then **4.4 immediately** -- 4.4 is the only gate that reads the DEPLOYED config rather than an input to it. ### C.11 -- What a Path C rebuild does NOT restore - **Model-defaults** -- C.8, and nothing else will remind you. - **Controller backup** -- phase-4 Step 11 is a separate obligation against the NEW controller. - **The client's known-hosts / juju SSH key** for the old controller VM: `juju` mints a new one at bootstrap. If a stale host key blocks you, that is the cause. - **OWED, still not built:** a site-keyed `dc-model-defaults.sh` with a `check`, so C.8 is VERIFIED at every standup instead of remembered. Until it exists, C.8 is the whole control. --- --- ## Step 1 -- Back up the state file before ANY destroy [do this every time] A `destroy` mutates state exactly like an `apply` does -- if it's interrupted partway (network drop, `Ctrl-C`, host reboot), your only way to know what's actually still live is the state file as it stood before you started. **Back up the state of the ROOT YOU ARE ABOUT TO DESTROY, on the host that root executes from.** There is no single state file any more: the OUTER root runs on vcloud (D-128 Plane 1, `libvirt_uri = qemu:///system`) and each DC's INNER root runs FROM THE OFFICE1 HEADEND (D-128 Plane 2, provider = `qemu+ssh`), which is where its state lives -- `docs/CURRENT-STATE.md` records both DCs' inner tfstate as held inside the voffice1 working tree, gitignored, sha256-pinned. A teardown that backed up only `opentofu/terraform.tfstate` on vcloud has NOT backed up the substrate it is about to destroy. ```bash # OUTER root -- on vcloud: cd "$REPO/opentofu" cp -a terraform.tfstate "terraform.tfstate.pre-teardown-$(date +%Y%m%d-%H%M%S)" # INNER root -- from the OFFICE1 HEADEND, for the DC being torn down: cd "$O1_REPO/opentofu/vr1-dc1-substrate" # or vr1-dc0-substrate cp -a terraform.tfstate "terraform.tfstate.pre-teardown-$(date +%Y%m%d-%H%M%S)" ``` Confirm each copy exists and is non-empty before continuing -- a `cp` whose source was absent is the same silence as a clean backup. Store the copy out-of-band per the operator's existing `~/vault-init/`-class secret-handling process (it carries the same plaintext-credential risk as the live file -- see `opentofu/README.md` "State file handling", DOCFIX-175) -- do not just leave it sitting next to the live file as your only copy. --- ## Step 2 -- MAAS-side FIRST: census the MACHINE records this site owns [do this every time] **DO NOT READ A `vm_host` HERE. IT IS A PERMANENT FALSE CLEAR.** The VR1 DCs deliberately have no MAAS pod (see "Relationship to D-061" above): a `maas vm-hosts read` returns nothing on an empty DC and on a DC with a fully commissioned fleet ALIKE. The version of this step that keyed "skip to Step 3, there is nothing to clean up" off that emptiness was routing an operator straight at destroying a containment VM holding live MAAS machine records. **The check is at the MACHINE level.** **Why associations get read BEFORE any destroy, even though the pod mechanism does not apply here.** MAAS deletes every machine record LINKED to a pod when the pod is deleted -- no warning, no decompose flag -- and on 2026-07-21 an association check that ran too late cost 9 machine records (`runbooks/appendix-A-troubleshooting.md`, symptom "`maas ... vm-host delete ` (or pod delete) silently REMOVES enlisted machines"; capture `docs/audit/incident-20260721-pod-delete-cascade.txt`). The mechanism is gone; the discipline it bought is not. **Read the associations first, and a NON-EMPTY result is a STOP.** **WHERE:** from the OFFICE1 HEADEND (D-128 Plane 2 -- the region MAAS and its CLI profile live there). MEASURED on vcloud 2026-07-29: `command -v maas` returns nothing. An ABSENT client is not an unreachable service and is never a clear -- this deployment has misdiagnosed that three times (`scripts/maas-role-tags.sh` makes the same split explicitly). **LENS 1 -- enumerate what EXISTS (never only what is declared).** ```bash maas "${MAAS_PROFILE:-admin}" machines read > /tmp/maas-machines.json \ || { echo "REFUSE: cannot read MAAS machines -- 'could not look' is NOT 'nothing there'" >&2; exit 1; } python3 - /tmp/maas-machines.json <<'PY' import json, sys ms = json.load(open(sys.argv[1])) virsh = [m for m in ms if (m.get("power_type") or "") == "virsh"] for m in ms: print(" %-10s %-18s %-14s power=%s" % ( m["system_id"], m["hostname"], m.get("status_name"), m.get("power_type") or "-")) print("total MAAS machine records: %d" % len(ms)) print("RECORDS REQUIRING ATTRIBUTION (power_type=virsh): %d" % len(virsh)) PY ``` **Write that second number down, and attribute every one of those records before the gate passes.** It is a COUNT, not a sentence, precisely so it cannot be skimmed: each `power_type=virsh` record is powered through SOME containment VM's inner libvirt, and a destroy that has not accounted for all of them is a destroy with an unknown blast radius. (`power_type` does come back from `machines read` -- shipped repo code reads that exact field, `scripts/maas-node-power.sh`.) Attribution needs that machine's power ADDRESS, which is a per-machine parameter: read it with `maas "${MAAS_PROFILE:-admin}" machine read ` and take the FIELD NAME from MAAS's own current docs -- do not invent one. Compare what you find against this site's URI from `scripts/lib-hosts.sh` (`VIRSH_POWER_ADDRESS`), never from memory. **If you cannot obtain the power address for a `virsh`-powered record, that is a REFUSAL, not a clear.** **LENS 2 -- corroborate against this site's pinned roster.** ```bash # on the OFFICE1 HEADEND, standing in that host's clone: O1_REPO="$(git rev-parse --show-toplevel)" \ || { echo "REFUSE: not inside the Office1 clone -- cd there first" >&2; exit 1; } SITE=vr1-dc1 # the site being torn down: vr1-dc0 | vr1-dc1 source "$O1_REPO/scripts/lib-hosts.sh" lib_hosts_select_dc "$SITE" || exit 1 : > /tmp/site-boot-macs.txt for h in "${HOSTS[@]}"; do printf '%s\n' "${HOST_BOOT_MAC[$h]}" >> /tmp/site-boot-macs.txt; done python3 - /tmp/maas-machines.json /tmp/site-boot-macs.txt <<'PY' import json, sys ms = json.load(open(sys.argv[1])) want = {l.strip().lower() for l in open(sys.argv[2]) if l.strip()} if not want: print("REFUSE: no pinned boot MACs for this site -- refusing to read that as zero") sys.exit(1) hits = [] for m in ms: macs = {(i.get("mac_address") or "").lower() for i in m.get("interface_set", [])} if macs & want: hits.append((m["system_id"], m["hostname"], m.get("status_name"), m.get("power_type") or "-")) print("ASSOCIATED MAAS MACHINE RECORDS for %d pinned MACs: %d" % (len(want), len(hits))) for h in hits: print(" %-10s %-18s %-14s power=%s" % h) sys.exit(1 if hits else 0) PY ``` Exit 1 means records EXIST for this site. That is the STOP condition, not an error to work around. Two properties of LENS 2 to hold in mind. `lib_hosts_select_dc` REFUSES a second, DIFFERENT site in the same shell (its stale-value guard) -- tear down one site per shell. And it enumerates only the hosts lib-hosts DECLARES, so it cannot report a record it has no row for (a canary enlistment, a re-enlisted node MAAS re-named, anything added since): that is exactly why LENS 1 runs and why it is not optional. **If records DO exist and you intend to lose them:** that is a deliberate, individually operator-approved decision, never a cleanup side effect. Release or delete them via MAAS's own current documented flow FIRST, then re-run both lenses. This repo has not exercised that path for real; MAAS's own docs are authoritative over anything asserted here. If records are already gone while the domains still exist, appendix-A carries the measured recovery (power the domains on, PXE re-enlist on the pinned MACs, `scripts/maas-node-power.sh` dry then `--commit`, re-commission -- and note MAAS mints NEW hostnames). **GATE (all three, or STOP):** 1. LENS 1 READ SUCCEEDED, and every `power_type=virsh` record in it is attributed to a known containment VM. 2. LENS 2 returns ZERO associated records for the site being torn down. 3. Neither lens was skipped because a command failed. A failure is a refusal. --- ## Step 3 (Path A only) -- Pick the ROOT first, then plan the destroy inside it **Scoping a single DC is now a ROOT choice, not a `-target` choice.** Under D-123 Model B each DC's substrate is its own root with its own state file, and each of those roots contains exactly ONE site -- so a plain `tofu destroy` in `opentofu/vr1-dc1-substrate/` is already scoped to vr1-dc1, with none of the silent-divergence risk HashiCorp's targeting warning describes. Reach for `-target` only for a PARTIAL teardown inside one root, or in the OUTER root, which holds all three sites at once. Module names MEASURED 2026-07-29 (`grep -n '^module' /main.tf`). **There is no `dc1_*` module in either root** -- those names are pre-D-119 and never existed here; a `-target` naming one aborts the plan. | Where it lives | Root | Modules (vr1-dc1 shown; vr1-dc0 is identical with `dc0` substituted) | |---|---|---| | INSIDE the DC | `opentofu/vr1-dc1-substrate/` | `inner_storage`, `vr1_dc1_planes`, `vr1_dc1_wan`, `vr1_dc1_opnsense`, `vr1_dc1_node` | | At vcloud | `opentofu/` (outer) | `vr1_dc1_storage`, `vr1_dc1_uplink`, `vvr1_dc1` | `module.vr1_dc1_node` is a `for_each` module keyed by node name -- targeting the module covers every instance; there are no `_node_01`/`_node_02` siblings to enumerate. There is no `maas_vm_host` module in either root to target (Step 2). `inner_storage` is named identically in BOTH substrate roots -- it is disambiguated by which root you are standing in, which is one more reason to confirm the directory before every command. **Order: INNER root first, then OUTER.** Destroying `module.vvr1_dcN` first leaves the inner state describing objects that no longer exist and removes the only path the inner provider can dial. INNER root -- run FROM THE OFFICE1 HEADEND (D-128 Plane 2; the provider is `qemu+ssh` and the state lives there). For the WHOLE DC substrate, no `-target` at all: ```bash cd "$O1_REPO/opentofu/vr1-dc1-substrate" tofu plan -destroy -out=teardown-vr1-dc1-inner.tfplan ``` For a PARTIAL inner teardown, target the real names: ```bash tofu plan -destroy \ -target=module.vr1_dc1_node \ -target=module.vr1_dc1_opnsense \ -target=module.vr1_dc1_wan \ -target=module.vr1_dc1_planes \ -target=module.inner_storage \ -out=teardown-vr1-dc1-inner.tfplan ``` OUTER root -- on vcloud, and ONLY after the inner half is done: ```bash cd "$REPO/opentofu" tofu plan -destroy \ -target=module.vvr1_dc1 \ -target=module.vr1_dc1_uplink \ -target=module.vr1_dc1_storage \ -out=teardown-vr1-dc1-outer.tfplan ``` **Do NOT target the mesh-link modules** (`mesh_vr1_dc0_vr1_dc1`, `mesh_vr1_dc0_office1`, `mesh_vr1_dc1_office1`) or `netem_vr1_dc0_vr1_dc1` in a single-site teardown -- each leg is SHARED with the site at its far end, and the dc0<->office1 leg additionally CARRIES the live rack<->region transit (MAAS, node DNS, the inner root's own `qemu+ssh` path). Only include a leg once BOTH sites it connects are going away (see the mesh-link note after Step 5). Review the plan line by line: expect destroys for exactly the resources above (`libvirt_domain`, `libvirt_volume`, `libvirt_network` x6 for the planes plus the WAN bridge network, `libvirt_pool`) and NOTHING outside this site's scope -- in particular confirm no `office1_*`, no other DC's `vr1_dc0_*`/`vr1_dc1_*`, and no `mesh_*`/`netem_*` resource appears. **CAPTURE THE PRE-DESTROY BASELINE NOW**, while the containment VM is still up -- Step 4 cannot tell "destroyed cleanly" from "could not look" without it: ```bash source "$O1_REPO/scripts/lib-hosts.sh"; lib_hosts_select_dc vr1-dc1 # -> VIRSH_POWER_ADDRESS virsh -c "$VIRSH_POWER_ADDRESS" list --all > /tmp/pre-teardown-inner-domains.txt virsh -c "$VIRSH_POWER_ADDRESS" net-list --all > /tmp/pre-teardown-inner-nets.txt virsh -c "$VIRSH_POWER_ADDRESS" pool-list --all > /tmp/pre-teardown-inner-pools.txt ``` **GATE:** plan matches this scope exactly; no unexpected destroys outside the targeted site; the baseline capture above is non-empty. Do not apply a plan you have not read. --- ## Step 4 (Path A only) -- Apply the scoped destroy [MUTATION: gated] Apply the INNER plan from the Office1 headend, in the inner root; apply the OUTER plan on vcloud, in `$REPO/opentofu`. Same order as Step 3: inner, then outer. ```bash tofu destroy teardown-vr1-dc1-inner.tfplan # inner root, from the Office1 headend tofu destroy teardown-vr1-dc1-outer.tfplan # outer root, on vcloud -- AFTER the inner verify below ``` *(Note: unlike a create/apply plan, a `-destroy`-mode plan file is applied the same way -- `tofu apply ` and `tofu destroy ` are equivalent once the plan itself was generated with `-destroy`; use whichever your `tofu` version's docs confirm for a plan file specifically, and if in doubt, prefer re-running with `tofu destroy` directly rather than guessing.)* Confirm this is the exact reviewed plan file from Step 3 (not re-planned) before running. **VERIFY -- and note WHICH HOST each half is verified from.** The DC's node, edge and plane objects live inside the containment VM, so a local `virsh` on vcloud can never see them: MEASURED on vcloud 2026-07-29, `virsh list --all` returns exactly `vvr1-dc0`, `vvr1-dc1`, `voffice1`, `office1-opnsense`. A vcloud-local `grep -i dc1` therefore "passes" against a fully intact DC. This is the D-128 split: Plane 1 = the outer `qemu:///system` on vcloud; the inner substrate is reached over `qemu+ssh` from the Office1 headend. INNER half -- from the Office1 headend, BEFORE the outer destroy (afterwards the URI cannot answer BY DESIGN): ```bash source "$O1_REPO/scripts/lib-hosts.sh"; lib_hosts_select_dc vr1-dc1 # -> VIRSH_POWER_ADDRESS virsh -c "$VIRSH_POWER_ADDRESS" version >/dev/null 2>&1 || { echo "REFUSE: cannot reach $VIRSH_POWER_ADDRESS. An empty list from an unreachable" >&2 echo " URI, a stopped containment VM or a bad key is NOT a clean destroy." >&2 exit 1; } virsh -c "$VIRSH_POWER_ADDRESS" list --all virsh -c "$VIRSH_POWER_ADDRESS" net-list --all virsh -c "$VIRSH_POWER_ADDRESS" pool-list --all ``` Take the URI from `scripts/lib-hosts.sh`, never from memory. Diff each against the Step 3 baseline: expect exactly the planned objects gone and NOTHING else changed. An empty list with no baseline to compare it to proves nothing. OUTER half -- on vcloud, after the outer destroy. Assert on the named objects, not on a substring match: ```bash virsh list --all # expect vvr1-dc1 GONE; vvr1-dc0, voffice1, office1-opnsense untouched virsh net-list --all # expect vr1-dc1-uplink GONE; the three mesh-* legs untouched virsh pool-list --all # expect vr1-dc1-pool GONE; vr1-dc0-pool, office1-pool, default untouched ``` (Live names measured 2026-07-29. Never `grep -i dc1` here: it matches `vr1-dc1` AND anything else carrying the string, and it cannot distinguish "absent" from "never visible from this host".) --- ## Step 5 (Path A only) -- Confirm no drift outside the targeted scope ```bash cd "$REPO/opentofu" tofu plan ``` Run WITHOUT `-target` this time -- this is the check HashiCorp's own targeting warning (Step 3) calls for: confirm the rest of the configuration (Office1, the OTHER DC, the mesh legs, netem) shows no unexpected changes. Expect either "No changes" for everything outside the torn-down site, or exactly the changes you independently expect for other in-flight work -- nothing attributable to this teardown. Do the same in the SURVIVING DC's inner root if you used `-target` anywhere; a targeted apply's drift does not announce itself. **GATE:** untargeted plan shows zero unexpected drift outside the torn-down site. --- ## Path B -- full VR1 teardown (replaces Steps 3-5 above) Do Step 1 (back up EVERY root's state -- outer plus BOTH inner roots) and Step 2 (the machine census, run for EVERY site, not just one) first, then destroy BOTH inner roots before the outer one -- "everything" is now three roots on two hosts, not one `cd opentofu`: ```bash # from the OFFICE1 HEADEND, once per DC substrate root: cd "$O1_REPO/opentofu/vr1-dc0-substrate" && tofu plan -destroy -out=teardown-vr1-dc0-inner.tfplan cd "$O1_REPO/opentofu/vr1-dc1-substrate" && tofu plan -destroy -out=teardown-vr1-dc1-inner.tfplan # then, on vcloud: cd "$REPO/opentofu" tofu plan -destroy -out=teardown-full.tfplan ``` Review: expect a destroy for every resource each tree currently manages (check against that root's actual uncommented module list at teardown time -- it grows as more stages get executed, so what "everything" means changes over the buildout, unlike Path A's fixed per-site scope). **GATE:** each plan matches its root's full current scope exactly. ```bash tofu destroy teardown-full.tfplan ``` **VERIFY.** Verify each inner root from the Office1 headend BEFORE the outer destroy, using Step 4's `virsh -c "$VIRSH_POWER_ADDRESS" version` refusal guard -- once the containment VMs are gone the inner URIs cannot answer, and "unreachable" is not evidence of anything. Then, on vcloud: ```bash virsh list --all virsh net-list --all virsh pool-list --all ``` Expect: nothing VR1-related remains (VR0's own testcloud resources, which OpenTofu does not manage, are untouched by any of this). Note that vcloud's `default` pool and `virbr0`/`wan` are not VR1 objects -- their survival is correct, not residue. Then do the equivalent of Step 5 (a final untargeted `tofu plan`) to confirm the destroy converged state to fully empty, not partially. --- ## Mesh-link teardown (either path, handle separately) The three D-100 mesh-link legs live in the OUTER root. Module names and the libvirt networks they create, MEASURED 2026-07-29 (`grep -n '^module' opentofu/main.tf`; `virsh net-list --all`): | Module (outer root) | libvirt network | bridge, measured 2026-07-29 | |---|---|---| | `mesh_vr1_dc0_vr1_dc1` | `mesh-vr1-dc0-vr1-dc1` | `virbr5` | | `mesh_vr1_dc0_office1` | `mesh-vr1-dc0-office1` | `virbr7` | | `mesh_vr1_dc1_office1` | `mesh-vr1-dc1-office1` | `virbr3` | (The pre-D-119 names `mesh_dc1_dc2` / `mesh_dc1_office1` / `mesh_dc2_office1` that this section used to carry exist nowhere -- `opentofu/main.tf` holds `moved` blocks recording exactly that rename. `virbrN` is a drifting libvirt assignment: re-measure with `virsh net-info ` before using one, never carry it from here.) Each leg is shared between two sites. Destroy a leg only when BOTH its endpoints are permanently going away (a full VR1 teardown, or abandoning both of its sites together) -- never as part of a single-site scoped teardown, and never as a reflex "clean up everything DC-adjacent" action. **`mesh-vr1-dc0- office1` deserves extra care:** it carries the live rack<->region transit, so MAAS, node DNS and the inner root's own `qemu+ssh` path all die with it. `modules/netem-link` IS wired now -- `module "netem_vr1_dc0_vr1_dc1"` in the outer root, applied 2026-07-21 against `virbr5` (the zero-traffic dc0<->dc1 leg). It has a correct `destroy`-time provisioner (`provisioner "local-exec" { when = destroy; ... tc qdisc del ... }`, confirmed by reading `opentofu/modules/netem-link/main.tf` directly), so destroying the corresponding `terraform_data.netem` resource cleans up the real `tc qdisc` rule on the vcloud host automatically; no extra manual step for that piece. Destroy it BEFORE the leg it is attached to, and note it is a separate module from the leg -- destroying `mesh_vr1_dc0_vr1_dc1` alone leaves the netem resource pointing at a bridge that no longer exists. --- ## Rollback decision tree -- "a stage's `tofu apply` failed partway through, now what?" **Read this before reaching for `tofu destroy` at all.** OpenTofu is designed for idempotent convergence -- in most failure modes, fixing the root cause and re-running the SAME `tofu apply` is correct and safer than tearing down and starting over, because it preserves everything that DID apply successfully and only retries what didn't. **Question 0 -- WHICH ROOT failed?** There are three: the OUTER root on vcloud, and one INNER substrate root per DC, run from the Office1 headend. The answer decides which state file, which host, and which `virsh` endpoint every branch below is talking about. Answer it before anything else. **Question 0b -- is anything DEPLOYED on top?** From Stage 5 onward a failed `tofu apply` can sit under live MAAS machine records and a live Juju model. **No branch below reaches a destroy without Step 2's machine census passing first.** If Step 2 returns records, the destroy is not the next action -- resolving the records is, and that is an operator-gated decision on its own. **Verb discipline, because these read alike under time pressure:** `virsh destroy vvr1-dcN` is a forced POWER-OFF, reversible with `virsh start`, and it leaves the domain, its disk and every MAAS record intact -- that is the D-122 site-down lever and it is a legitimate first move. `tofu destroy` of `module.vvr1_dcN` UNDEFINES the domain and DELETES its disk, taking the whole inner fleet with it, irreversibly. Reaching for the second when you meant the first is the expensive mistake this runbook exists to prevent. 1. **`tofu apply` errored before creating anything (e.g., `tofu init`/ `validate`/`plan` failed, or `apply` failed on its very first resource).** Nothing was created. Fix the root cause (bad variable, unreachable provider, syntax error) and re-run from that stage's own `tofu plan` step. No teardown needed. 2. **`tofu apply` created SOME resources, then failed on a later one.** `tofu plan` (no `-destroy`) immediately to see OpenTofu's own view of what's now live vs. what the config wants. Default to FIX-FORWARD: fix whatever caused the failure (a bad node-VM memory size, an unreachable MAAS endpoint, a real infra problem) and re-run `tofu apply` for that same stage -- OpenTofu will create only what's still missing, leaving successfully-created resources alone. Reserve a teardown (Path A, scoped to that site) for when the partial state is itself broken in a way fix-forward can't reach (e.g., a domain stuck in a bad libvirt state that manual `virsh` intervention already tried and failed to fix). 3. **You suspect `terraform.tfstate` itself is corrupted or lost** (this is the scenario `opentofu/README.md`'s "State file handling" section already flags as the real cost of no backend/backup). There is no clean recovery path documented in this repo: either a manual `tofu import` per still-live resource (tedious, must be done for every domain/network/pool by hand) or accepting a full teardown-and-rebuild from Step 1 of the affected stage. This is exactly why Step 1's pre-teardown backup habit matters even outside a deliberate teardown -- back up `terraform.tfstate` after every successful stage apply too, not only before a teardown. 4. **A resource exists in reality but OpenTofu's state doesn't know about it** (e.g., a `virt-install`-created VM from Stage 2's Option B manual path, which OpenTofu never created and cannot destroy). This runbook's `tofu destroy`/`plan -destroy` commands only ever touch what OpenTofu itself manages -- anything created via a manual/Option-B path needs manual teardown (`virsh destroy`/`undefine`, etc.) by the same operator discipline that created it. Do not expect `tofu destroy` to find or remove it. 5. **The failure is INSIDE a DC and you are tempted to reach for the containment VM.** Taking the site down (`virsh destroy vvr1-dcN`) is reversible and is often the right move to stop a cascade. Destroying the containment VM through the OUTER root is not a rollback of the inner apply -- it discards the entire inner substrate, leaves the inner state file describing objects that no longer exist, and leaves MAAS holding machine records whose `power_address` will never answer again. Fix-forward in the INNER root first; if you truly need the DC gone, do Step 2, then the inner root, then the outer -- in that order. --- ## Verification (of this runbook's own accuracy, not a live run) Every `tofu`/`terraform` flag cited above (`-destroy`, `-target`, the targeting-risk warning) was checked against HashiCorp's own current CLI documentation before being written here (fetched 2026-07-10), not asserted from memory -- same discipline as `opentofu/README.md`'s other research sections. `netem-link`'s destroy-time provisioner was confirmed by reading the actual `.tf` file, not assumed. **CORRECTION 2026-07-29:** this paragraph previously vouched for "`dc1_maas_vm_host`'s lack of a Terraform-expressible dependency on node-VM domains" as something confirmed by reading the `.tf` files. No such module has ever existed in either root, so that was a verification claim about nothing -- the false-verification class that lets a wrong instruction look checked. All module, network, pool and domain names in this runbook were RE-MEASURED against the tree and the live hosts on 2026-07-29 (`grep -n '^module' /main.tf`; `virsh list/net-list/pool-list --all`; `virsh net-dumpxml`). What is NOT verified: this runbook's commands have never been run against a real `tofu` binary or a real state file (same SCAFFOLD/UNVALIDATED status as the rest of `opentofu/`) -- treat every command here as reviewed-but-unexercised until `scripts/opentofu-validate.sh` and a real dry run confirm otherwise.