# Changelog 2026-07-10 -- VR1 DC-DC teardown/rollback runbook (DOCFIX-176)

No live infrastructure touched -- this is a new runbook only; nothing in
`opentofu/` has ever been applied against real infrastructure in any session
that authored this repo clone. Closes gap register item #19 (found alongside
DOCFIX-175 while reviewing what's missing from the buildout ahead of
tomorrow's execution).

## The gap

`runbooks/phase-00-teardown-release.sh` / `-destroy.sh` (D-061) cover tearing
down VR0's juju/MAAS-machine layer. Nothing covered the DIFFERENT layer this
buildout also needs to be able to tear down: the OpenTofu-provisioned
libvirt/MAAS-vm_host resources for DC1, DC2, and Office1. If Stage 1 or later
fails partway through a real execution, there was no documented path back to
a clean slate for this specific layer -- and no guidance on whether to even
reach for a teardown vs. fixing forward, which is usually the safer, more
appropriate response to a partial `tofu apply` failure.

## Fix

New `runbooks/dc-dc-teardown-rollback.md`:

- **Relationship to D-061**, stated explicitly rather than assumed: the same
  PRINCIPLE (clean up the higher-layer record before destroying the
  lower-layer resource underneath it) carries over, but the MECHANISM
  differs, because this repo's OpenTofu layer uses the plain `maas_vm_host`
  resource (not MAAS pod-composition), so D-061's specific
  `--keep-instance`/decompose-on-release failure mode doesn't directly
  transfer. Flagged as a residual, untested gap (this repo has never reached
  Stage 4 for any VR1 DC, so no real MAAS-enrolled machine has ever existed
  under a VR1 `vm_host` to test a release procedure against) rather than
  inventing untested MAAS CLI commands.
- **Two paths**: Path A (scoped single-site teardown via `tofu`'s `-target`
  flag, with HashiCorp's own documented targeting-risk warning quoted and
  a mandatory untargeted `tofu plan` afterward to catch drift) and Path B
  (full VR1 teardown, plain `tofu destroy`, letting OpenTofu's own
  dependency graph sequence everything).
- **Explicit dependency ordering for Path A**, since `maas_vm_host` has no
  Terraform-expressible dependency on the node-VM domains it's registered
  against (confirmed by reading `modules/maas-vm-host/main.tf` and
  `modules/node-vm/main.tf` directly -- the resource only references
  `power_address`, not a domain resource) and so would NOT be caught by a
  `-target` scoped to the storage/plane modules alone.
- **Mesh-link handling**: explicit warning never to destroy a D-100 mesh-link
  leg during a single-site teardown (shared infrastructure with the far
  end), only when both endpoints are permanently going away. Also confirmed
  (by reading `modules/netem-link/main.tf` directly) that its
  `destroy`-time `local-exec` provisioner already correctly runs
  `tc qdisc del` -- no extra manual step needed for that piece specifically.
- **Rollback decision tree**: four scenarios (nothing created yet, partial
  apply failure, suspected state corruption/loss, resources created outside
  OpenTofu via Stage 2's Option B manual `virt-install` path) with the
  default bias toward fix-forward over destroy, matching OpenTofu's own
  idempotent-convergence design rather than treating "something went wrong"
  as an automatic teardown trigger.
- **State backup as Step 1 of every path**, cross-referencing DOCFIX-175 --
  a destroy mutates state exactly like an apply does, so the same
  pre-mutation backup discipline applies.

Every `tofu`/`terraform` CLI flag cited (`-destroy`, `-target`, the
targeting-risk warning text) was verified against HashiCorp's own current
documentation before being written (fetched 2026-07-10), not asserted from
memory -- same discipline as `opentofu/README.md`'s other research sections.

Cross-referenced from `opentofu/README.md` (new "Teardown / rollback"
section) and `docs/dc-dc-deployment-workflow.md` (gap register item #19,
CLOSED).

## Self-inflicted issue found and fixed mid-delivery (unrelated to this runbook, but in the same working session)

While preparing this delivery's DOCFIX number, `bash scripts/ledger-scan.sh`
was re-run and caught that the DOCFIX-174 changelog and its
`docs/session-ledger.md` mirror had REINTRODUCED the exact false-positive
class DOCFIX-174 itself had just fixed -- both quoted the illustrative
example number as a literal string while narrating the incident, which
re-inflated `ledger-scan.sh`'s reported next-free value. Caught by re-running
the scan before assigning this delivery's own number rather than trusting
the earlier fix was airtight; fixed by rewording both (see
`docs/changelog-20260710-ledger-scan-self-inflicted-falsepositive-fix.md`'s
own updated "It recurred immediately" section for the full account). No
separate DOCFIX assigned for this -- folded into the DOCFIX-174 changelog
itself since it corrects that same delivery before commit, not a new one.

**Also corrected mid-delivery:** DOCFIX-175's first draft wrongly assumed
Stage 3 ran the first real `tofu apply`; reading the runbooks directly
showed Stage 1 does. Fixed across all of DOCFIX-175's files before this
teardown-runbook delivery, so the "current plan" this runbook's own
cross-references describe is accurate.

## Verification

`bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy warn. No script
logic changed -- one new runbook, two doc cross-references, one README
section. No test harness applies. `bash scripts/ledger-scan.sh`: confirms
next-free is D-111/DOCFIX-176/BUNDLEFIX-013 after this delivery (was
verified clean of the self-inflicted false positive before this number was
assigned).

REVERT: `git checkout HEAD~ -- runbooks/dc-dc-teardown-rollback.md
opentofu/README.md docs/dc-dc-deployment-workflow.md` plus
`git rm runbooks/dc-dc-teardown-rollback.md` if reverting past its
introduction commit (safe -- no live infrastructure depends on this, nothing
here has ever been applied against real state).
