diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 1e32fd3..79e4c48 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2052,6 +2052,48 @@ its counts without that sentence would leave it wrong in a way that READS as fixed. **OWED: a human read of the expanded runbook.** It more than doubled in one pass; its individual claims were re-grepped and it is lint-clean, but length is not correctness. +- **ITEM 3.9 (the teardown runbook) AND 3.10 (the gap register) FIXED 2026-07-29.** 3.9 was + the readiness audit's most dangerous item because `runbooks/dc-dc-teardown-rollback.md` is + what an operator reaches for DURING a failed Stage 5, under time pressure. + **THE FALSE CLEAR EXISTED IN THREE PLACES, NOT THE ONE THE AUDIT NAMED.** Besides Step 2's + `vm-host read` -- which can never return, because this repo uses per-machine + `power_type=virsh` and instantiates no `maas_vm_host` module -- the same wrong premise sat + in the "READ BEFORE ANY DC TEARDOWN" header block telling the operator to "remove the + `maas-vm-host` record", and in the "Relationship to D-061" claim that no VR1 DC had reached + Stage 4. **The header one is the consequential discovery: it is read FIRST, so it bypassed + any fix confined to Step 2.** Step 2 is now a two-lens MACHINE census run from the headend + (`maas` is measurably absent on vcloud): lens 1 enumerates what EXISTS and ends in a + countable `RECORDS REQUIRING ATTRIBUTION`, lens 2 corroborates against `lib-hosts` pinned + boot MACs and exits 1 on any hit. **Demonstrated three ways against a fixture -- records + present -> exit 1, genuinely zero -> exit 0, empty roster -> REFUSE** -- so it is a gate, + not a formality. The 2026-07-21 pod-cascade precedent (9 machine records lost to an + association check that ran too late) is retained as the reason associations are read first. + Step 3's six phantom `module.dc1_*` targets are retired: **all 8 targets now resolve**, + re-verified independently here against `^module "X"` across all three roots, and the real + insight recorded is that **scoping a DC is a ROOT choice, not a `-target` choice**, since + each substrate root holds exactly one site (with `inner_storage` ambiguously named + identically in both). Mesh names replaced by a measured module -> network -> bridge table; + Step 4's VERIFY moved to `qemu+ssh` from the headend behind a `virsh version` REFUSAL, since + an unreachable URI, a stopped VM and a bad key all otherwise return the same empty result. + The decision tree gains a "no branch reaches a destroy without Step 2 passing" question -- + it never mentioned the MAAS gate at all -- and the `virsh destroy` (reversible power-off) + versus `tofu destroy` (irreversible) verb distinction. Three further in-file defects fixed: + Step 1 backed up the WRONG state file (inner state lives on voffice1), "Two paths" + contradicted the new Step 3 and pointed twice at a nonexistent Step 6, and `$REPO` silently + meant two different clones (now `$REPO` vcloud vs `$O1_REPO` headend). + **3.10:** item 17 CLOSED with measured evidence, and its own stated fix corrected -- it + closed by D-125 bridge-in, NOT by the "replicate office1-wan per DC" the entry claimed. + Item 19 disambiguated **19a/19b rather than renumbered**, because both are cited BY NUMBER + from outside the file and renumbering would dangle live citations. Item 20 MEASURED rather + than asserted: both DC transits are isolated, vcloud holds no address on virbr7/virbr3, and + the routes resolve via the corporate default -- **verdict no leg required, with the rule + mismatch WRITTEN IN rather than resolved silently** (the skill's criterion has two branches + and the measured state satisfies neither; D-128 breaks the tie), plus an explicit expiry + condition. The voffice1-side transit reboot durability is recorded as UNMEASURED with the + commands that would resolve it. **NEW, logged not fixed:** the teardown header's "no `tofu` + binary" claim is measurably false, register items 15 and 11 are stale in item 17's class, + `CURRENT-STATE.md:2410` carries drifted `main.tf` line numbers, and the runbook cites + DOCFIX-175 where the register says DOCFIX-176. **F5 (MEDIUM) -- preflight's `DC` selector does not reach P4.** `preflight.sh:99` sets `DC` without exporting it, so propagation depends on the caller's invocation form; and it is moot because `pre-flight-checks.sh` never reads `DC` at all (one hit, in a comment). diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md index 599745a..abd5f0a 100644 --- a/docs/dc-dc-deployment-workflow.md +++ b/docs/dc-dc-deployment-workflow.md @@ -854,8 +854,11 @@ internet through it. Office1's edge is also the only one of the three that is live; DC1/DC2's edges are Stage 3's problem and are still blocked on gap #17's DC half. -17. **PARTIALLY CLOSED 2026-07-13 -- the DECISION is made and PROVEN for - Office1; DC1/DC2 still have no uplink network.** +17. **CLOSED -- Office1 2026-07-13, both DCs by 2026-07-22.** (Was carried as + "PARTIALLY CLOSED ... DC1/DC2 still have no uplink network" until + 2026-07-29. That status was STALE-OPEN, and because the skill's VR1 deploy + loop routes every session to read this register FIRST, it produced a FALSE + STOP. Corrected against the tree and the live host, below.) - **Office1: CLOSED.** The answer was the one this gap proposed -- a dedicated per-site ISP-uplink `libvirt_network`, **not** a mesh leg. `office1-wan` exists and is live (measured 2026-07-13: `virbr11`, NAT @@ -863,14 +866,42 @@ `172.30.1.2` with default route `172.30.1.1`, and egress to the internet is verified (0.0% loss). So the topology question is ANSWERED and the pattern is PROVEN in production, not merely designed. - - **DC1/DC2: STILL OPEN.** Neither has an ISP-uplink network. Measured: - `dc1-provider-public` has **no IP and no forward** (isolated), and there - is no `dc1-wan`/`dc2-wan` at all. Stage 3's `dc1_opnsense` still carries - the unresolved `wan_network_name` placeholder described below. - - **The fix is now mechanical, not a decision:** replicate the - `office1-wan` shape per DC. The D-100 reasoning below (mesh legs are - MANAGEMENT-TRAFFIC-ONLY and must NOT carry an edge's WAN side) stands - unchanged and is what the Office1 build honoured. + - **BOTH DCs: CLOSED.** The 2026-07-13 text here said "Neither has an + ISP-uplink network ... there is no `dc1-wan`/`dc2-wan` at all" and that + "Stage 3's `dc1_opnsense` still carries the unresolved + `wan_network_name` placeholder". Every clause is now FALSE. Measured + 2026-07-29: + - Outer root `opentofu/main.tf`: `module "vr1_dc0_uplink"` + (`172.30.2.0/24`) and `module "vr1_dc1_uplink"` (`172.30.3.0/24`), + both `modules/site-wan` NAT networks. (At `:379` and `:391` when + measured; grep the module NAME, line numbers drift.) + - Inner roots: `module "vr1_dc0_wan"` in + `opentofu/vr1-dc0-substrate/main.tf` and `module "vr1_dc1_wan"` in + `opentofu/vr1-dc1-substrate/main.tf`, each a `modules/wan-bridge` + onto that containment VM's `br-vr1-dcN-wan`. Each DC edge's + `wan_network_name` is wired to its own module's output -- no + placeholder remains in either root. + - LIVE on vcloud: `virsh net-list --all` shows `vr1-dc0-uplink` and + `vr1-dc1-uplink` active/autostart; `ip -4 addr` shows `172.30.2.1/24` + and `172.30.3.1/24` on their bridges. + - Gate evidence: D-125 egress isolation PASS for dc0 + (`docs/audit/d125-egress-gate-20260720-matrix.txt`) and for dc1 + (`docs/audit/d125-egress-gate-20260722-dc1.txt`). + - Naming: the searched-for `dc1-wan` does not exist because the DCs were + renamed by D-117/D-119 -- the real names are `vr1-dc0-wan` / + `vr1-dc1-wan`. A grep for the retired token returning nothing is not + evidence of a missing network. + - **How it was actually closed -- NOT by replicating `office1-wan` per + DC.** This gap's own closing note used to say the fix was mechanical + replication of the Office1 shape. It was not: D-125 rules **bridge-in**, + with the single NAT at VCLOUD level (`vr1-dcN-uplink`) and the inner + `vr1-dcN-wan` a BRIDGE onto the containment VM's netplan bridge, so the + inner OPNsense WAN lands directly on that /24 and there is exactly ONE + NAT. That was necessary because D-123 Model B nested the DC WAN inside a + containment VM whose only routed leg is the East-West-only transit. The + D-100 reasoning below (mesh legs are MANAGEMENT-TRAFFIC-ONLY and must + NOT carry an edge's WAN side) stands unchanged and is what all three + builds honoured. *Original 2026-07-09 text follows (the reasoning is still correct; the "for ANY site / nowhere" status claim is superseded by the block above).* @@ -922,7 +953,8 @@ with real locking -- production-grade, Roosevelt-scope, not a rehearsal decision. -19. **No teardown/rollback runbook existed for the VR1 OpenTofu-provisioned +19. **(19a -- "gap #19, DOCFIX-176"; see the numbering note on 19b below.) + No teardown/rollback runbook existed for the VR1 OpenTofu-provisioned infrastructure itself -- CLOSED 2026-07-10 (DOCFIX-176).** `runbooks/phase-00-teardown-*.sh` (D-061) covers tearing down the VR0 juju/MAAS layer, a different pairing of tools; nothing documented how to @@ -941,7 +973,36 @@ has no real machines to test against yet -- the runbook states the principle and points at MAAS's own docs rather than guessing flags. -19. **NEW 2026-07-14 -- THE `dc1`/`dc0` NAME NOW MEANS TWO DIFFERENT DATACENTERS. +19. **(19b) CLOSED 2026-07-14 by D-119 -- THE `dc1`/`dc0` NAME MEANT TWO + DIFFERENT DATACENTERS.** + + **Numbering note (2026-07-29).** This register carried TWO items numbered + 19 -- this one and 19a above -- and this one still declared itself OPEN and + a hard precondition of all Stage 3 OpenTofu work, five months after it was + closed. Neither is renumbered, because both are cited by number from + outside this file: "gap #19" in `docs/design-decisions.md:3746` ("Closes + tooling-gap #19 and Stage 2 close-out item C3") and in + `docs/archive/changelogs/changelog-20260714-d119-region-qualified-dc-namespace.md` + means THIS item (19b); a citation naming DOCFIX-176 means 19a. Item 20 + keeps its number too -- the operating skill cites "register item 20" by + number. + + **Status: CLOSED 2026-07-14 (D-119, ADOPTED -- `docs/design-decisions.md` + D-119, which states in its own status block that it closes this gap and + Stage 2 close-out item C3).** The remap was executed, not merely ruled -- + verified 2026-07-29: `scripts/lib-net.sh` and `scripts/lib-hosts.sh` take + region-qualified selectors (`vr0-dc0` / `vr1-dc0` / `vr1-dc1`) and REFUSE a + bare `dc0`/`dc1`/`dc2` with an explicit RETIRED message, and the OpenTofu + module names are `vr1_dc0_*` / `vr1_dc1_*` / `mesh_vr1_dc0_vr1_dc1` with + `moved` blocks in `opentofu/main.tf` recording the rename from the old + `dc1_*`/`dc2_*` names. **Nothing below is a live blocker; it is kept as the + record of WHY the namespace is region-qualified, which is still load-bearing + when reading any pre-2026-07-14 surface.** + + *Original 2026-07-14 text follows (the reasoning is still correct; its OPEN + status and its "BLOCKS Stage 3" claim are superseded by the block above).* + + **THE `dc1`/`dc0` NAME NOW MEANS TWO DIFFERENT DATACENTERS. This BLOCKS Stage 3 and is the same off-by-one D-117 just fixed, one layer down.** D-117 (ADOPTED, Option B) moved the repo onto the NetBox apex's 0-indexed @@ -970,9 +1031,10 @@ wrong site's prefixes, and the first VR1 DC deploys with the second DC's addressing -- discovered, at best, at `juju deploy`. - **Status: OPEN. It is Stage 2 close-out item C3 and the FIRST step of any - Stage 3 OpenTofu work.** Nothing in `opentofu/` should gain a new DC-keyed - module until it is closed. + *(Its closing line read: "**Status: OPEN. It is Stage 2 close-out item C3 + and the FIRST step of any Stage 3 OpenTofu work.** Nothing in `opentofu/` + should gain a new DC-keyed module until it is closed." Superseded -- see + the CLOSED block at the head of 19b.)* 20. **NEW 2026-07-17 -- durable host->site (isolated-net) reachability had no owning tool -- CLOSED for Office1; DC path templated.** @@ -1002,7 +1064,55 @@ done; if the DC is reached over a routed/NAT net or by `qemu+ssh` to an address the host already holds (D-123 / R-5), no leg is needed. The skill carries this as standing discipline (routing row + operating-model invariant + the VR1 deploy loop). - **Status: Office1 CLOSED; DC rows DEFERRED-until-measured (mechanism in place).** + + **PER-DC VERDICT, MEASURED 2026-07-29 on vcloud (read-only; this replaces the + "DC rows DEFERRED-until-measured" status the entry carried until then -- which + left "no leg needed" as an INFERENCE nobody had checked).** + + What was measured, and how: + - Each DC's containment/service net is its office1<->dcN transit: + `mesh-vr1-dc0-office1` (bridge `virbr7`) and `mesh-vr1-dc1-office1` + (bridge `virbr3`) -- `virsh net-list --all`, `virsh domiflist vvr1-dc0`, + `virsh domiflist vvr1-dc1`, `virsh domiflist voffice1`. Each containment VM + has exactly two NICs: its transit and its D-125 uplink. + - Both transits are ISOLATED: `virsh net-dumpxml` on each shows NO `` and + NO `` element, so libvirt puts no address on `virbr7`/`virbr3`. + - vcloud holds NO L3 presence on either: `ip -4 -o addr show` lists + `virbr11`/`virbr0`/`virbr2`/`virbr4`/`virbr1` and NOT `virbr7`/`virbr3`. + - vcloud has NO path to either rack transit address: `ip route get 172.31.0.2` + and `ip route get 172.31.0.6` (the two `VIRSH_POWER_ADDRESS` values in + `scripts/lib-hosts.sh`) both resolve `via 10.17.8.1 dev enp1s0` -- the + corporate default route, i.e. off-site, not the DC. + + **VERDICT: NO `site-baseleg.sh` row is required for either DC -- but NOT for + the reason the rule above anticipates, and the mismatch is recorded here + deliberately rather than resolved silently.** The rule offers two branches and + the measured state satisfies NEITHER: the net IS an isolated host-local net + (branch 1, "add a leg"), AND vcloud does NOT hold the `qemu+ssh` address + (branch 2, "no leg needed"). The tie is broken by D-128: **no + vcloud-originated DC operation exists.** Plane 2 -- MAAS, the inner `tofu` + roots, the per-DC `virsh -c qemu+ssh://...` power path -- EXECUTES on + `voffice1`, which holds the region end of both transits (its NIC2/NIC3 sit on + `mesh-vr1-dc0-office1` and `mesh-vr1-dc1-office1`, measured above). vcloud + reaches the DCs only THROUGH voffice1, and voffice1 is reached over the + office1 leg this item already closed. A DC leg would be adding host L3 to a + path nothing originates on. **If a future change makes vcloud originate to a + DC directly, this verdict expires and branch 1 applies.** + + **UNMEASURED, explicitly (needs an `ssh voffice1`, outside this session's + read-only scope -- do not read this entry as "the path is fine"):** + (a) that voffice1 currently holds `172.31.0.1/30` and `172.31.0.5/30` on + those NICs, and (b) reboot-durability of that addressing -- the same class of + failure that produced this item for Office1. **Resolves with:** + `ssh voffice1 'ip -4 -o addr show; ip route show'` for (a), plus + `virsh -c "$VIRSH_POWER_ADDRESS" version` per DC as the end-to-end proof; + for (b), the determination is whether that addressing is netplan-persistent + on voffice1 or was added by hand -- if by hand, it is a `site-baseleg`-class + gap on voffice1, not on vcloud, and belongs in this item. + + **Status: Office1 CLOSED (leg installed). Both DC rows: MEASURED 2026-07-29, + verdict NO LEG REQUIRED on vcloud, with the voffice1-side transit addressing + UNMEASURED as above.** --- diff --git a/runbooks/dc-dc-teardown-rollback.md b/runbooks/dc-dc-teardown-rollback.md index a5903c0..feef78a 100644 --- a/runbooks/dc-dc-teardown-rollback.md +++ b/runbooks/dc-dc-teardown-rollback.md @@ -1,9 +1,12 @@ # VR1 DC-DC teardown / rollback runbook Scope: tearing down or rolling back the infrastructure `opentofu/` provisions -for the VR1 (DC-DC) buildout -- DC1/DC2/Office1 plane and local networks, -storage pools, node-VM domains, OPNsense edge VMs, the D-100 mesh-link -triangle, and MAAS `vm_host` registrations. Written 2026-07-10 (DOCFIX-175 +for the VR1 (DC-DC) buildout -- the `vr1-dc0` / `vr1-dc1` / Office1 plane and +local networks, storage pools, node-VM domains, containment VMs, OPNsense edge +VMs, and the D-100 mesh-link triangle. (It once also claimed MAAS `vm_host` +registrations; there are none -- see Step 2. The `dc1`/`dc2` names it was +written with are pre-D-119 and mean different datacenters now.) +Written 2026-07-10 (DOCFIX-175 gap-register item #19) because nothing else in this repo covers this layer: `runbooks/phase-00-teardown-release.sh` / `phase-00-teardown-destroy.sh` (D-061) tear down the VR0 **juju/MAAS-machine** layer, a different pairing @@ -27,21 +30,33 @@ > host). Under **D-123 Model B** that is no longer how a DC is built, so the resource-by-resource > mechanics below are partly WRONG for the DCs. Current shape: > -> - **TWO roots / TWO state files.** The OUTER root (`opentofu/`, vcloud) owns `vvr1-dc0` (the DC -> containment VM), the office1<->dc0 transit, and the **D-125** vcloud ISP uplink (`vr1-dc0-uplink` -> NAT + the `vvr1-dc0` uplink NIC/`br-vr1-dc0-wan` netplan bridge). The INNER root -> (`opentofu/vr1-dc0-substrate/`, provider = `qemu+ssh` to `vvr1-dc0`) owns everything INSIDE the -> DC -- the 9 node VMs, the 6 planes, `vr1-dc0-wan`, the OPNsense edge, the inner pool. So the -> "node-VM domains / planes / edge" this runbook talks about live on **`vvr1-dc0`'s libvirt, not -> vcloud's**, and `maas-vm-host` registers **`vvr1-dc0`'s inner virsh** to the Office1 REGION. -> - **Whole-DC site-down is ONE object: `virsh destroy vvr1-dc0`** (the D-122/D-123 point). Destroying +> - **TWO roots / TWO state files PER DC, and there are now TWO DCs.** The OUTER root (`opentofu/`, +> vcloud) owns `vvr1-dc0` and `vvr1-dc1` (the DC containment VMs), the office1<->dcN transits, and +> the **D-125** vcloud ISP uplinks (`vr1-dc0-uplink` / `vr1-dc1-uplink` NAT + each containment VM's +> uplink NIC / `br-vr1-dcN-wan` netplan bridge). Each INNER root +> (`opentofu/vr1-dc0-substrate/`, `opentofu/vr1-dc1-substrate/`; provider = `qemu+ssh` to that DC's +> containment VM) owns everything INSIDE its DC -- the node VMs, the 6 planes, `vr1-dcN-wan`, the +> OPNsense edge, the inner pool. So the "node-VM domains / planes / edge" this runbook talks about +> live on **the containment VM's libvirt, not vcloud's**. +> - **There is NO MAAS `vm_host`/pod for any VR1 DC.** MAAS holds per-MACHINE records with +> `power_type=virsh` (D-103/D-123 amendments, ruled 2026-07-20) and `modules/maas-vm-host` is +> instantiated in NEITHER root -- measured 2026-07-29. Any instruction anywhere to "remove the +> `maas-vm-host` record before destroying the containment VM" is VOID: there is no such record, +> the read returns nothing on a DC with a full fleet enrolled, and acting on that emptiness +> destroys the containment VM out from under live MAAS machine records. **Step 2 is the real +> check.** +> - **Whole-DC site-down is ONE object: `virsh destroy vvr1-dcN`** (the D-122/D-123 point). Destroying > the containment VM instantly takes down the entire inner fleet -- no per-domain group destroy -> needed. That is the fast path for "abandon this DC and rebuild." -> - **Teardown ORDER (two roots):** tear down the INNER root first (`cd opentofu/vr1-dc0-substrate && -> tofu destroy`, or just `virsh destroy vvr1-dc0` if you're discarding the whole DC), THEN the OUTER -> root. The D-061-style "clean up MAAS's record before destroying the libvirt underneath" principle -> below still applies -- but the `maas-vm-host` record now points at `vvr1-dc0`'s inner virsh, so -> remove it BEFORE destroying `vvr1-dc0`. +> needed. That is the fast path for "abandon this DC and rebuild." **Know which verb you are +> using:** `virsh destroy` is a forced POWER-OFF and is REVERSIBLE (`virsh start vvr1-dcN`); the +> domain, its disk and every MAAS record still exist. `tofu destroy` of `module.vvr1_dcN` UNDEFINES +> the domain and deletes its disk -- irreversible, and it takes the whole inner fleet with it. +> Under a failed Stage 5 those are not interchangeable. +> - **Teardown ORDER (two roots):** tear down the INNER root first (`cd opentofu/vr1-dcN-substrate && +> tofu destroy`, or just `virsh destroy vvr1-dcN` if you are only taking the site DOWN), THEN the +> OUTER root. The D-061-style "clean up MAAS's record before destroying the libvirt underneath" +> principle below still applies -- and with no pod to remove, "MAAS's record" means the per-machine +> records enumerated in Step 2. Run Step 2 BEFORE either root's destroy. > - **Reverting Model B -> Model A** (a different operation from teardown) is > `docs/archive/model-a-fallback-plan.md` (git tag `model-a-fallback`), not this runbook. > @@ -82,57 +97,90 @@ - D-061 coordinates **juju's machine view** against **MAAS's machine view** of hosts MAAS itself pod-composed. - This runbook coordinates **OpenTofu's resource view** (libvirt domains/ - networks/pools it created directly) against **MAAS's `vm_host` and - (eventually) machine view** of a libvirt host OpenTofu registered via - `modules/maas-vm-host` (the `maas_vm_host` resource, canonical/maas - provider -- NOT `maas_vm_host_machine`, which composes VMs itself; this - repo deliberately uses the plain `maas_vm_host` variant so `modules/ - node-vm`'s own domains are the source of truth, not MAAS pod-composition - -- see `opentofu/README.md`). + networks/pools it created directly) against **MAAS's per-MACHINE view** of + those same domains. MEASURED 2026-07-29: `modules/maas-vm-host` is + instantiated in NEITHER root (`grep -n 'maas_vm_host' opentofu/*.tf + opentofu/vr1-dc0-substrate/*.tf opentofu/vr1-dc1-substrate/*.tf` returns + only comments), and the `provider "maas"` block those modules would need is + still deliberately absent (`opentofu/main.tf:9-20`, DOCFIX-179). MAAS knows + the DC node VMs because they PXE-ENLISTED, and it powers them per machine + with `power_type=virsh` (D-103/D-123 amendments, ruled 2026-07-20 -- + `scripts/maas-node-power.sh`'s header carries the two measurements that + refuted the pod). -Because this repo doesn't use MAAS pod-composition for its OpenTofu-created -VMs, the SPECIFIC `--keep-instance` decompose-on-release failure mode D-061 -diagnosed does not directly apply here. The GENERAL principle still does: -**clean up MAAS's record of a host before destroying the libvirt resource -underneath it**, not after -- destroying the libvirt domain/host first would -leave MAAS pointing at a `power_address` that no longer answers, an -orphaned-but-not-obviously-broken record rather than a clean removal. +Because this repo uses neither MAAS pod-composition NOR a `vm_host` for its +OpenTofu-created VMs, the SPECIFIC `--keep-instance` decompose-on-release +failure mode D-061 diagnosed does not directly apply here. The GENERAL +principle still does: **clean up MAAS's record of a host before destroying the +libvirt resource underneath it**, not after -- destroying the libvirt domain/ +host first would leave MAAS pointing at a `power_address` that no longer +answers, an orphaned-but-not-obviously-broken record rather than a clean +removal. -**Residual open item, flagged not solved here:** this repo has not yet -reached Stage 4 (MAAS enlist/commission/deploy) for any VR1 DC, so no VR1 -libvirt host has ever had real MAAS-enrolled machines under it. The exact -MAAS-side commands to cleanly release machines enrolled under a `vm_host` -this runbook is about to tear down (analogous to D-061's `remove-machine ---keep-instance` discovery, but for the `vm_host`/enlist-commission-deploy -flow instead of pod-composition) have not been worked out or tested. Step 2 -below states the PRINCIPLE (MAAS-side first) and points at MAAS's own -current docs for the exact commands -- do not invent flags for a scenario -this repo has never actually exercised. +**What this section used to say, and why it is now a hazard rather than a +caveat:** it stated that no VR1 DC had reached Stage 4, so no VR1 libvirt host +had ever had real MAAS-enrolled machines under it. **That is false as of +2026-07-23** (`docs/CURRENT-STATE.md`: 18 nodes READY, 9 per DC, both DCs +commissioned; `power_type=virsh` set and proven by a real +`query-power-state`), and the per-DC Juju controller VMs were enlisted on top +of that in July 2026. Anything downstream of "nothing is enrolled yet" -- +including the old Step 2 -- must be treated as REVERSED, not merely stale. The +exact MAAS-side release/delete commands for a machine whose libvirt domain is +about to disappear are STILL not worked out or tested in this repo: Step 2 +states the principle, enumerates the records, and points at MAAS's own current +docs for the flags. Do not invent flags for a scenario this repo has never +actually exercised. + +--- + +## TWO CLONES, TWO HOSTS -- set these once, before anything below + +Every block in this runbook runs on one of two hosts (D-128), each with its +own clone of this repo. The same `cd` pasted on the wrong one is the +foot-gun this section exists to remove, so the two are named differently +throughout: + +- **`$REPO` -- the VCLOUD clone.** Plane 1: the OUTER root + (`$REPO/opentofu`, `libvirt_uri = qemu:///system`) and local `virsh`. +- **`$O1_REPO` -- the OFFICE1 HEADEND clone, on `voffice1`.** Plane 2: BOTH + DC substrate roots (`$O1_REPO/opentofu/vr1-dc0-substrate`, + `.../vr1-dc1-substrate`, provider = `qemu+ssh`), their state files, the + `maas` CLI, and every `virsh -c qemu+ssh://...` into a containment VM. + +MEASURE `$O1_REPO` with `ls` on the headend -- do not assume it, and do not +assume the inner state lives on vcloud (it does not). --- ## Two paths, by intent (mirrors D-061's release-vs-destroy split) - **Path A -- scoped teardown.** Tear down ONE site's resources (a single DC, - or Office1), leaving the other sites' infrastructure untouched. Use - `-target` to scope both plan and apply. HashiCorp's own docs flag `-target` - as an exceptional-circumstances tool, not for routine use, because - targeted applies can leave configuration and state silently diverging: - *"This targeting capability is provided for exceptional circumstances, - such as recovering from mistakes or working around Terraform limitations. - It is not recommended to use `-target` for routine operations, since this - can lead to undetected configuration drift"* (developer.hashicorp.com/ - terraform/cli/commands/plan, fetched 2026-07-10). A scoped teardown IS - exactly the kind of exceptional circumstance this describes -- acceptable - here, but ALWAYS follow it with a full (untargeted) `tofu plan` afterward - (Step 6) to confirm the rest of the configuration didn't drift. + or Office1), leaving the other sites' infrastructure untouched. **Scope by + choosing the ROOT first** -- each DC substrate root contains exactly one + site, so a plain `tofu destroy` there is already scoped and needs no + `-target` at all. `-target` is reserved for the OUTER root (which holds all + three sites) and for a partial teardown inside one root. Where you do use + it, HashiCorp's own docs flag it as an exceptional-circumstances tool, not + for routine use, because targeted applies can leave configuration and state + silently diverging: *"This targeting capability is provided for exceptional + circumstances, such as recovering from mistakes or working around Terraform + limitations. It is not recommended to use `-target` for routine operations, + since this can lead to undetected configuration drift"* + (developer.hashicorp.com/terraform/cli/commands/plan, fetched 2026-07-10). + A scoped teardown IS exactly that kind of exceptional circumstance -- + acceptable here, but ALWAYS follow a targeted apply with a full (untargeted) + `tofu plan` afterward (Step 5) to confirm the rest of the configuration + didn't drift. - **Path B -- full VR1 teardown.** Abandon everything OpenTofu manages for - VR1. Plain `tofu destroy` (no `-target`), letting OpenTofu compute the - full safe order from the resource graph itself -- no manual sequencing - needed, unlike Path A. + VR1. That is now THREE roots across TWO hosts -- both DC substrate roots + from the Office1 headend, then the outer root on vcloud -- each a plain + `tofu destroy` (no `-target`), letting OpenTofu compute the safe order + WITHIN each root from its own resource graph. The order BETWEEN roots is + yours to get right: inner before outer, always. -Both paths share Steps 1 (state backup) and 6 (final verify). Path A is -Steps 2-5; Path B replaces Steps 3-5 with a single plain destroy (see +Both paths share Steps 1 (state backup), 2 (the MAAS machine census) and 5 +(final verify). Path A is Steps 2-5; Path B replaces Steps 3-5 with the +per-root destroys described in its own section (see "Path B" section after Step 5). --- @@ -144,10 +192,27 @@ know what's actually still live is the state file as it stood before you started. +**Back up the state of the ROOT YOU ARE ABOUT TO DESTROY, on the host that +root executes from.** There is no single state file any more: the OUTER root +runs on vcloud (D-128 Plane 1, `libvirt_uri = qemu:///system`) and each DC's +INNER root runs FROM THE OFFICE1 HEADEND (D-128 Plane 2, provider = +`qemu+ssh`), which is where its state lives -- `docs/CURRENT-STATE.md` records +both DCs' inner tfstate as held inside the voffice1 working tree, gitignored, +sha256-pinned. A teardown that backed up only `opentofu/terraform.tfstate` on +vcloud has NOT backed up the substrate it is about to destroy. + ```bash -cd opentofu +# OUTER root -- on vcloud: +cd "$REPO/opentofu" +cp -a terraform.tfstate "terraform.tfstate.pre-teardown-$(date +%Y%m%d-%H%M%S)" + +# INNER root -- from the OFFICE1 HEADEND, for the DC being torn down: +cd "$O1_REPO/opentofu/vr1-dc1-substrate" # or vr1-dc0-substrate cp -a terraform.tfstate "terraform.tfstate.pre-teardown-$(date +%Y%m%d-%H%M%S)" ``` +Confirm each copy exists and is non-empty before continuing -- a `cp` whose +source was absent is the same silence as a clean backup. + Store the copy out-of-band per the operator's existing `~/vault-init/`-class secret-handling process (it carries the same plaintext-credential risk as the live file -- see `opentofu/README.md` "State file handling", DOCFIX-175) @@ -155,78 +220,218 @@ --- -## Step 2 -- MAAS-side cleanup FIRST, if any machines were ever enrolled under the site being torn down +## Step 2 -- MAAS-side FIRST: census the MACHINE records this site owns [do this every time] -**CHECK -- does this site have a real MAAS `vm_host` with enrolled machines?** +**DO NOT READ A `vm_host` HERE. IT IS A PERMANENT FALSE CLEAR.** The VR1 DCs +deliberately have no MAAS pod (see "Relationship to D-061" above): a +`maas vm-hosts read` returns nothing on an empty DC and on a DC with +a fully commissioned fleet ALIKE. The version of this step that keyed "skip to +Step 3, there is nothing to clean up" off that emptiness was routing an +operator straight at destroying a containment VM holding live MAAS machine +records. **The check is at the MACHINE level.** + +**Why associations get read BEFORE any destroy, even though the pod mechanism +does not apply here.** MAAS deletes every machine record LINKED to a pod when +the pod is deleted -- no warning, no decompose flag -- and on 2026-07-21 an +association check that ran too late cost 9 machine records +(`runbooks/appendix-A-troubleshooting.md`, symptom "`maas ... vm-host delete +` (or pod delete) silently REMOVES enlisted machines"; capture +`docs/audit/incident-20260721-pod-delete-cascade.txt`). The mechanism is gone; +the discipline it bought is not. **Read the associations first, and a +NON-EMPTY result is a STOP.** + +**WHERE:** from the OFFICE1 HEADEND (D-128 Plane 2 -- the region MAAS and its +CLI profile live there). MEASURED on vcloud 2026-07-29: `command -v maas` +returns nothing. An ABSENT client is not an unreachable service and is never a +clear -- this deployment has misdiagnosed that three times +(`scripts/maas-role-tags.sh` makes the same split explicitly). + +**LENS 1 -- enumerate what EXISTS (never only what is declared).** ```bash -maas vm-host read # or: maas vm-hosts read | grep -i +maas "${MAAS_PROFILE:-admin}" machines read > /tmp/maas-machines.json \ + || { echo "REFUSE: cannot read MAAS machines -- 'could not look' is NOT 'nothing there'" >&2; exit 1; } + +python3 - /tmp/maas-machines.json <<'PY' +import json, sys +ms = json.load(open(sys.argv[1])) +virsh = [m for m in ms if (m.get("power_type") or "") == "virsh"] +for m in ms: + print(" %-10s %-18s %-14s power=%s" % ( + m["system_id"], m["hostname"], m.get("status_name"), + m.get("power_type") or "-")) +print("total MAAS machine records: %d" % len(ms)) +print("RECORDS REQUIRING ATTRIBUTION (power_type=virsh): %d" % len(virsh)) +PY ``` -If this returns nothing (expected for any VR1 site before Stage 4 has run -for it): skip to Step 3, there is nothing to clean up at this layer yet. +**Write that second number down, and attribute every one of those records +before the gate passes.** It is a COUNT, not a sentence, precisely so it cannot +be skimmed: each `power_type=virsh` record is powered through SOME containment +VM's inner libvirt, and a destroy that has not accounted for all of them is a +destroy with an unknown blast radius. (`power_type` does come back from +`machines read` -- shipped repo code reads that exact field, +`scripts/maas-node-power.sh`.) -If it DOES return enrolled machines: release/delete them via MAAS's own -current CLI/UI per its official docs BEFORE touching the underlying -`tofu destroy` in Step 4 -- per the D-061 principle above, do not destroy -the libvirt host out from under a MAAS record that still thinks it's live. -This repo has not exercised this path for real; treat MAAS's own current -documented release/delete flow as authoritative over anything asserted here. +Attribution needs that machine's power ADDRESS, which is a per-machine +parameter: read it with `maas "${MAAS_PROFILE:-admin}" machine read +` and take the FIELD NAME from MAAS's own current docs -- do not +invent one. Compare what you find against this site's URI from +`scripts/lib-hosts.sh` (`VIRSH_POWER_ADDRESS`), never from memory. **If you +cannot obtain the power address for a `virsh`-powered record, that is a +REFUSAL, not a clear.** -**GATE:** either no machines were ever enrolled (common case, pre-Stage-4), -or MAAS confirms zero machines remain under this site's `vm_host` before -proceeding. +**LENS 2 -- corroborate against this site's pinned roster.** +```bash +# on the OFFICE1 HEADEND, standing in that host's clone: +O1_REPO="$(git rev-parse --show-toplevel)" \ + || { echo "REFUSE: not inside the Office1 clone -- cd there first" >&2; exit 1; } +SITE=vr1-dc1 # the site being torn down: vr1-dc0 | vr1-dc1 +source "$O1_REPO/scripts/lib-hosts.sh" +lib_hosts_select_dc "$SITE" || exit 1 +: > /tmp/site-boot-macs.txt +for h in "${HOSTS[@]}"; do printf '%s\n' "${HOST_BOOT_MAC[$h]}" >> /tmp/site-boot-macs.txt; done + +python3 - /tmp/maas-machines.json /tmp/site-boot-macs.txt <<'PY' +import json, sys +ms = json.load(open(sys.argv[1])) +want = {l.strip().lower() for l in open(sys.argv[2]) if l.strip()} +if not want: + print("REFUSE: no pinned boot MACs for this site -- refusing to read that as zero") + sys.exit(1) +hits = [] +for m in ms: + macs = {(i.get("mac_address") or "").lower() for i in m.get("interface_set", [])} + if macs & want: + hits.append((m["system_id"], m["hostname"], m.get("status_name"), + m.get("power_type") or "-")) +print("ASSOCIATED MAAS MACHINE RECORDS for %d pinned MACs: %d" % (len(want), len(hits))) +for h in hits: + print(" %-10s %-18s %-14s power=%s" % h) +sys.exit(1 if hits else 0) +PY +``` +Exit 1 means records EXIST for this site. That is the STOP condition, not an +error to work around. + +Two properties of LENS 2 to hold in mind. `lib_hosts_select_dc` REFUSES a +second, DIFFERENT site in the same shell (its stale-value guard) -- tear down +one site per shell. And it enumerates only the hosts lib-hosts DECLARES, so it +cannot report a record it has no row for (a canary enlistment, a re-enlisted +node MAAS re-named, anything added since): that is exactly why LENS 1 runs and +why it is not optional. + +**If records DO exist and you intend to lose them:** that is a deliberate, +individually operator-approved decision, never a cleanup side effect. Release +or delete them via MAAS's own current documented flow FIRST, then re-run both +lenses. This repo has not exercised that path for real; MAAS's own docs are +authoritative over anything asserted here. If records are already gone while +the domains still exist, appendix-A carries the measured recovery (power the +domains on, PXE re-enlist on the pinned MACs, `scripts/maas-node-power.sh` dry +then `--commit`, re-commission -- and note MAAS mints NEW hostnames). + +**GATE (all three, or STOP):** +1. LENS 1 READ SUCCEEDED, and every `power_type=virsh` record in it is + attributed to a known containment VM. +2. LENS 2 returns ZERO associated records for the site being torn down. +3. Neither lens was skipped because a command failed. A failure is a refusal. --- -## Step 3 (Path A only) -- Plan the scoped destroy, safe order, most-dependent-first +## Step 3 (Path A only) -- Pick the ROOT first, then plan the destroy inside it -OpenTofu's own dependency graph (`dc1_opnsense`/`dc1_node_*` reference -`module.dc1_planes`/`module.dc1_storage` outputs) already sequences a single -`-target` invocation correctly for everything BELOW it in the graph -- but -`maas_vm_host` has no Terraform-expressible dependency on the node-VM -domains it was registered against (it references `power_address`, not a -domain resource), so it will NOT be destroyed by a plan targeting the -storage/plane modules and must be listed explicitly if it exists. +**Scoping a single DC is now a ROOT choice, not a `-target` choice.** Under +D-123 Model B each DC's substrate is its own root with its own state file, and +each of those roots contains exactly ONE site -- so a plain `tofu destroy` in +`opentofu/vr1-dc1-substrate/` is already scoped to vr1-dc1, with none of the +silent-divergence risk HashiCorp's targeting warning describes. Reach for +`-target` only for a PARTIAL teardown inside one root, or in the OUTER root, +which holds all three sites at once. -For DC1 (adapt names for DC2/Office1 once their own resources are real): +Module names MEASURED 2026-07-29 (`grep -n '^module' /main.tf`). **There +is no `dc1_*` module in either root** -- those names are pre-D-119 and never +existed here; a `-target` naming one aborts the plan. +| Where it lives | Root | Modules (vr1-dc1 shown; vr1-dc0 is identical with `dc0` substituted) | +|---|---|---| +| INSIDE the DC | `opentofu/vr1-dc1-substrate/` | `inner_storage`, `vr1_dc1_planes`, `vr1_dc1_wan`, `vr1_dc1_opnsense`, `vr1_dc1_node` | +| At vcloud | `opentofu/` (outer) | `vr1_dc1_storage`, `vr1_dc1_uplink`, `vvr1_dc1` | + +`module.vr1_dc1_node` is a `for_each` module keyed by node name -- targeting +the module covers every instance; there are no `_node_01`/`_node_02` siblings +to enumerate. There is no `maas_vm_host` module in either root to target (Step +2). `inner_storage` is named identically in BOTH substrate roots -- it is +disambiguated by which root you are standing in, which is one more reason to +confirm the directory before every command. + +**Order: INNER root first, then OUTER.** Destroying `module.vvr1_dcN` first +leaves the inner state describing objects that no longer exist and removes the +only path the inner provider can dial. + +INNER root -- run FROM THE OFFICE1 HEADEND (D-128 Plane 2; the provider is +`qemu+ssh` and the state lives there). For the WHOLE DC substrate, no +`-target` at all: ```bash -cd opentofu -tofu plan -destroy \ - -target=module.dc1_maas_vm_host \ - -target=module.dc1_opnsense \ - -target=module.dc1_node_01 \ - -target=module.dc1_node_02 \ - ` # -- repeat -target=module.dc1_node_NN for every real node module block ` \ - -target=module.dc1_storage \ - -target=module.dc1_planes \ - -out=teardown-dc1.tfplan +cd "$O1_REPO/opentofu/vr1-dc1-substrate" +tofu plan -destroy -out=teardown-vr1-dc1-inner.tfplan ``` -Only include `-target=module.dc1_maas_vm_host` if that module is actually -wired in `main.tf` (Stage 3 Step 9) -- `tofu plan` errors on a `-target` -naming a module that doesn't exist in configuration. +For a PARTIAL inner teardown, target the real names: +```bash +tofu plan -destroy \ + -target=module.vr1_dc1_node \ + -target=module.vr1_dc1_opnsense \ + -target=module.vr1_dc1_wan \ + -target=module.vr1_dc1_planes \ + -target=module.inner_storage \ + -out=teardown-vr1-dc1-inner.tfplan +``` -**Do NOT target the mesh-link modules** (`mesh_dc1_dc2`, `mesh_dc1_office1`) -in a single-DC teardown -- they are SHARED infrastructure with the other -site at each leg's far end. Only include a mesh-link leg in the destroy set -once BOTH sites it connects are being torn down (see the mesh-link note -after Step 5). +OUTER root -- on vcloud, and ONLY after the inner half is done: +```bash +cd "$REPO/opentofu" +tofu plan -destroy \ + -target=module.vvr1_dc1 \ + -target=module.vr1_dc1_uplink \ + -target=module.vr1_dc1_storage \ + -out=teardown-vr1-dc1-outer.tfplan +``` -Review the plan line by line: expect destroys for exactly the resources -above (`libvirt_domain`, `libvirt_volume`, `libvirt_network` x6 for the -planes, `libvirt_pool`, and `maas_vm_host` if targeted) and NOTHING outside -this site's scope -- in particular, confirm no `office1_*`/`dc2_*`/mesh-link -resource appears. +**Do NOT target the mesh-link modules** (`mesh_vr1_dc0_vr1_dc1`, +`mesh_vr1_dc0_office1`, `mesh_vr1_dc1_office1`) or `netem_vr1_dc0_vr1_dc1` in +a single-site teardown -- each leg is SHARED with the site at its far end, and +the dc0<->office1 leg additionally CARRIES the live rack<->region transit +(MAAS, node DNS, the inner root's own `qemu+ssh` path). Only include a leg +once BOTH sites it connects are going away (see the mesh-link note after Step +5). -**GATE:** plan matches this scope exactly; no unexpected destroys outside -the targeted site. Do not apply a plan you have not read. +Review the plan line by line: expect destroys for exactly the resources above +(`libvirt_domain`, `libvirt_volume`, `libvirt_network` x6 for the planes plus +the WAN bridge network, `libvirt_pool`) and NOTHING outside this site's scope +-- in particular confirm no `office1_*`, no other DC's `vr1_dc0_*`/`vr1_dc1_*`, +and no `mesh_*`/`netem_*` resource appears. + +**CAPTURE THE PRE-DESTROY BASELINE NOW**, while the containment VM is still +up -- Step 4 cannot tell "destroyed cleanly" from "could not look" without it: +```bash +source "$O1_REPO/scripts/lib-hosts.sh"; lib_hosts_select_dc vr1-dc1 # -> VIRSH_POWER_ADDRESS +virsh -c "$VIRSH_POWER_ADDRESS" list --all > /tmp/pre-teardown-inner-domains.txt +virsh -c "$VIRSH_POWER_ADDRESS" net-list --all > /tmp/pre-teardown-inner-nets.txt +virsh -c "$VIRSH_POWER_ADDRESS" pool-list --all > /tmp/pre-teardown-inner-pools.txt +``` + +**GATE:** plan matches this scope exactly; no unexpected destroys outside the +targeted site; the baseline capture above is non-empty. Do not apply a plan +you have not read. --- ## Step 4 (Path A only) -- Apply the scoped destroy [MUTATION: gated] +Apply the INNER plan from the Office1 headend, in the inner root; apply the +OUTER plan on vcloud, in `$REPO/opentofu`. Same order as Step 3: inner, then +outer. + ```bash -cd opentofu -tofu destroy teardown-dc1.tfplan +tofu destroy teardown-vr1-dc1-inner.tfplan # inner root, from the Office1 headend +tofu destroy teardown-vr1-dc1-outer.tfplan # outer root, on vcloud -- AFTER the inner verify below ``` *(Note: unlike a create/apply plan, a `-destroy`-mode plan file is applied the same way -- `tofu apply ` and `tofu destroy ` are @@ -237,28 +442,57 @@ Confirm this is the exact reviewed plan file from Step 3 (not re-planned) before running. -**VERIFY** +**VERIFY -- and note WHICH HOST each half is verified from.** The DC's node, +edge and plane objects live inside the containment VM, so a local `virsh` on +vcloud can never see them: MEASURED on vcloud 2026-07-29, `virsh list --all` +returns exactly `vvr1-dc0`, `vvr1-dc1`, `voffice1`, `office1-opnsense`. A +vcloud-local `grep -i dc1` therefore "passes" against a fully intact DC. This +is the D-128 split: Plane 1 = the outer `qemu:///system` on vcloud; the inner +substrate is reached over `qemu+ssh` from the Office1 headend. + +INNER half -- from the Office1 headend, BEFORE the outer destroy (afterwards +the URI cannot answer BY DESIGN): ```bash -virsh list --all | grep -i dc1 -virsh net-list --all | grep -i dc1 -virsh pool-list --all | grep -i dc1 +source "$O1_REPO/scripts/lib-hosts.sh"; lib_hosts_select_dc vr1-dc1 # -> VIRSH_POWER_ADDRESS +virsh -c "$VIRSH_POWER_ADDRESS" version >/dev/null 2>&1 || { + echo "REFUSE: cannot reach $VIRSH_POWER_ADDRESS. An empty list from an unreachable" >&2 + echo " URI, a stopped containment VM or a bad key is NOT a clean destroy." >&2 + exit 1; } +virsh -c "$VIRSH_POWER_ADDRESS" list --all +virsh -c "$VIRSH_POWER_ADDRESS" net-list --all +virsh -c "$VIRSH_POWER_ADDRESS" pool-list --all ``` -Expect: no DC1 domains, networks, or pool remain. +Take the URI from `scripts/lib-hosts.sh`, never from memory. Diff each against +the Step 3 baseline: expect exactly the planned objects gone and NOTHING else +changed. An empty list with no baseline to compare it to proves nothing. + +OUTER half -- on vcloud, after the outer destroy. Assert on the named objects, +not on a substring match: +```bash +virsh list --all # expect vvr1-dc1 GONE; vvr1-dc0, voffice1, office1-opnsense untouched +virsh net-list --all # expect vr1-dc1-uplink GONE; the three mesh-* legs untouched +virsh pool-list --all # expect vr1-dc1-pool GONE; vr1-dc0-pool, office1-pool, default untouched +``` +(Live names measured 2026-07-29. Never `grep -i dc1` here: it matches +`vr1-dc1` AND anything else carrying the string, and it cannot distinguish +"absent" from "never visible from this host".) --- ## Step 5 (Path A only) -- Confirm no drift outside the targeted scope ```bash -cd opentofu +cd "$REPO/opentofu" tofu plan ``` Run WITHOUT `-target` this time -- this is the check HashiCorp's own targeting warning (Step 3) calls for: confirm the rest of the configuration -(Office1, DC2 if wired, mesh links) shows no unexpected changes. Expect -either "No changes" for everything outside DC1, or exactly the changes you -independently expect for other in-flight work -- nothing attributable to -this teardown. +(Office1, the OTHER DC, the mesh legs, netem) shows no unexpected changes. +Expect either "No changes" for everything outside the torn-down site, or +exactly the changes you independently expect for other in-flight work -- +nothing attributable to this teardown. Do the same in the SURVIVING DC's +inner root if you used `-target` anywhere; a targeted apply's drift does not +announce itself. **GATE:** untargeted plan shows zero unexpected drift outside the torn-down site. @@ -267,32 +501,44 @@ ## Path B -- full VR1 teardown (replaces Steps 3-5 above) -Do Step 1 (backup) and Step 2 (MAAS-side cleanup, for EVERY site with -enrolled machines, not just one) first, then: +Do Step 1 (back up EVERY root's state -- outer plus BOTH inner roots) and Step +2 (the machine census, run for EVERY site, not just one) first, then destroy +BOTH inner roots before the outer one -- "everything" is now three roots on two +hosts, not one `cd opentofu`: ```bash -cd opentofu +# from the OFFICE1 HEADEND, once per DC substrate root: +cd "$O1_REPO/opentofu/vr1-dc0-substrate" && tofu plan -destroy -out=teardown-vr1-dc0-inner.tfplan +cd "$O1_REPO/opentofu/vr1-dc1-substrate" && tofu plan -destroy -out=teardown-vr1-dc1-inner.tfplan + +# then, on vcloud: +cd "$REPO/opentofu" tofu plan -destroy -out=teardown-full.tfplan ``` -Review: expect a destroy for every resource this tree currently manages -(check against `opentofu/main.tf`'s actual uncommented module list at -teardown time -- it grows as more stages get executed, so what "everything" -means changes over the buildout, unlike Path A's fixed per-site scope). +Review: expect a destroy for every resource each tree currently manages (check +against that root's actual uncommented module list at teardown time -- it grows +as more stages get executed, so what "everything" means changes over the +buildout, unlike Path A's fixed per-site scope). -**GATE:** plan matches full current `main.tf` scope exactly. +**GATE:** each plan matches its root's full current scope exactly. ```bash tofu destroy teardown-full.tfplan ``` -**VERIFY** +**VERIFY.** Verify each inner root from the Office1 headend BEFORE the outer +destroy, using Step 4's `virsh -c "$VIRSH_POWER_ADDRESS" version` refusal guard +-- once the containment VMs are gone the inner URIs cannot answer, and +"unreachable" is not evidence of anything. Then, on vcloud: ```bash virsh list --all virsh net-list --all virsh pool-list --all ``` -Expect: nothing VR1-related remains (VR0's own DC0/testcloud resources, -which OpenTofu does not manage, are untouched by any of this). +Expect: nothing VR1-related remains (VR0's own testcloud resources, which +OpenTofu does not manage, are untouched by any of this). Note that vcloud's +`default` pool and `virbr0`/`wan` are not VR1 objects -- their survival is +correct, not residue. Then do the equivalent of Step 5 (a final untargeted `tofu plan`) to confirm the destroy converged state to fully empty, not partially. @@ -301,18 +547,39 @@ ## Mesh-link teardown (either path, handle separately) -The three D-100 mesh-link legs (`mesh_dc1_dc2`, `mesh_dc1_office1`, -`mesh_dc2_office1`) are shared between two sites each. Destroy a leg only -when BOTH its endpoints are permanently going away (e.g., a full VR1 -teardown, or abandoning DC2 AND Office1 together) -- never as part of a -single-site scoped teardown, and never as a reflex "clean up everything -DC1-adjacent" action. `modules/netem-link` (if ever wired against a given -leg) already has a correct `destroy`-time provisioner -(`provisioner "local-exec" { when = destroy; ... tc qdisc del ... }`, -confirmed by reading `opentofu/modules/netem-link/main.tf` directly) -- -destroying the corresponding `terraform_data.netem` resource cleans up the -real `tc qdisc` rule on the vcloud host automatically; no extra manual step -needed for that specific piece. +The three D-100 mesh-link legs live in the OUTER root. Module names and the +libvirt networks they create, MEASURED 2026-07-29 (`grep -n '^module' +opentofu/main.tf`; `virsh net-list --all`): + +| Module (outer root) | libvirt network | bridge, measured 2026-07-29 | +|---|---|---| +| `mesh_vr1_dc0_vr1_dc1` | `mesh-vr1-dc0-vr1-dc1` | `virbr5` | +| `mesh_vr1_dc0_office1` | `mesh-vr1-dc0-office1` | `virbr7` | +| `mesh_vr1_dc1_office1` | `mesh-vr1-dc1-office1` | `virbr3` | + +(The pre-D-119 names `mesh_dc1_dc2` / `mesh_dc1_office1` / `mesh_dc2_office1` +that this section used to carry exist nowhere -- `opentofu/main.tf` holds +`moved` blocks recording exactly that rename. `virbrN` is a drifting libvirt +assignment: re-measure with `virsh net-info ` before using one, never +carry it from here.) + +Each leg is shared between two sites. Destroy a leg only when BOTH its +endpoints are permanently going away (a full VR1 teardown, or abandoning both +of its sites together) -- never as part of a single-site scoped teardown, and +never as a reflex "clean up everything DC-adjacent" action. **`mesh-vr1-dc0- +office1` deserves extra care:** it carries the live rack<->region transit, so +MAAS, node DNS and the inner root's own `qemu+ssh` path all die with it. + +`modules/netem-link` IS wired now -- `module "netem_vr1_dc0_vr1_dc1"` in the +outer root, applied 2026-07-21 against `virbr5` (the zero-traffic dc0<->dc1 +leg). It has a correct `destroy`-time provisioner (`provisioner "local-exec" { +when = destroy; ... tc qdisc del ... }`, confirmed by reading +`opentofu/modules/netem-link/main.tf` directly), so destroying the +corresponding `terraform_data.netem` resource cleans up the real `tc qdisc` +rule on the vcloud host automatically; no extra manual step for that piece. +Destroy it BEFORE the leg it is attached to, and note it is a separate module +from the leg -- destroying `mesh_vr1_dc0_vr1_dc1` alone leaves the netem +resource pointing at a bridge that no longer exists. --- @@ -324,6 +591,25 @@ tearing down and starting over, because it preserves everything that DID apply successfully and only retries what didn't. +**Question 0 -- WHICH ROOT failed?** There are three: the OUTER root on +vcloud, and one INNER substrate root per DC, run from the Office1 headend. +The answer decides which state file, which host, and which `virsh` endpoint +every branch below is talking about. Answer it before anything else. + +**Question 0b -- is anything DEPLOYED on top?** From Stage 5 onward a failed +`tofu apply` can sit under live MAAS machine records and a live Juju model. +**No branch below reaches a destroy without Step 2's machine census passing +first.** If Step 2 returns records, the destroy is not the next action -- +resolving the records is, and that is an operator-gated decision on its own. + +**Verb discipline, because these read alike under time pressure:** `virsh +destroy vvr1-dcN` is a forced POWER-OFF, reversible with `virsh start`, and it +leaves the domain, its disk and every MAAS record intact -- that is the D-122 +site-down lever and it is a legitimate first move. `tofu destroy` of +`module.vvr1_dcN` UNDEFINES the domain and DELETES its disk, taking the whole +inner fleet with it, irreversibly. Reaching for the second when you meant the +first is the expensive mistake this runbook exists to prevent. + 1. **`tofu apply` errored before creating anything (e.g., `tofu init`/ `validate`/`plan` failed, or `apply` failed on its very first resource).** Nothing was created. Fix the root cause (bad variable, unreachable @@ -358,6 +644,15 @@ manual teardown (`virsh destroy`/`undefine`, etc.) by the same operator discipline that created it. Do not expect `tofu destroy` to find or remove it. +5. **The failure is INSIDE a DC and you are tempted to reach for the + containment VM.** Taking the site down (`virsh destroy vvr1-dcN`) is + reversible and is often the right move to stop a cascade. Destroying the + containment VM through the OUTER root is not a rollback of the inner + apply -- it discards the entire inner substrate, leaves the inner state + file describing objects that no longer exist, and leaves MAAS holding + machine records whose `power_address` will never answer again. Fix-forward + in the INNER root first; if you truly need the DC gone, do Step 2, then the + inner root, then the outer -- in that order. --- @@ -367,10 +662,17 @@ targeting-risk warning) was checked against HashiCorp's own current CLI documentation before being written here (fetched 2026-07-10), not asserted from memory -- same discipline as `opentofu/README.md`'s other research -sections. The resource/module names and dependency shapes referenced -(`dc1_maas_vm_host`'s lack of a Terraform-expressible dependency on node-VM -domains, `netem-link`'s existing destroy-time provisioner) were confirmed by -reading the actual `.tf` files in this repo, not assumed. What is NOT +sections. `netem-link`'s destroy-time provisioner was confirmed by reading the +actual `.tf` file, not assumed. + +**CORRECTION 2026-07-29:** this paragraph previously vouched for +"`dc1_maas_vm_host`'s lack of a Terraform-expressible dependency on node-VM +domains" as something confirmed by reading the `.tf` files. No such module has +ever existed in either root, so that was a verification claim about nothing -- +the false-verification class that lets a wrong instruction look checked. All +module, network, pool and domain names in this runbook were RE-MEASURED against +the tree and the live hosts on 2026-07-29 (`grep -n '^module' /main.tf`; +`virsh list/net-list/pool-list --all`; `virsh net-dumpxml`). What is NOT verified: this runbook's commands have never been run against a real `tofu` binary or a real state file (same SCAFFOLD/UNVALIDATED status as the rest of `opentofu/`) -- treat every command here as reviewed-but-unexercised until