Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260710-vr1-teardown-rollback-runbook.md
@JANeumatrix JANeumatrix 7 hours ago 5 KB DOCFIX-175/176: OpenTofu state file security + VR1 teardown/rollback runbook
Raw Blame History

Changelog 2026-07-10 -- VR1 DC-DC teardown/rollback runbook (DOCFIX-176)

No live infrastructure touched -- this is a new runbook only; nothing in opentofu/ has ever been applied against real infrastructure in any session that authored this repo clone. Closes gap register item #19 (found alongside DOCFIX-175 while reviewing what's missing from the buildout ahead of tomorrow's execution).

The gap

runbooks/phase-00-teardown-release.sh / -destroy.sh (D-061) cover tearing down VR0's juju/MAAS-machine layer. Nothing covered the DIFFERENT layer this buildout also needs to be able to tear down: the OpenTofu-provisioned libvirt/MAAS-vm_host resources for DC1, DC2, and Office1. If Stage 1 or later fails partway through a real execution, there was no documented path back to a clean slate for this specific layer -- and no guidance on whether to even reach for a teardown vs. fixing forward, which is usually the safer, more appropriate response to a partial tofu apply failure.

Fix

New runbooks/dc-dc-teardown-rollback.md:

  • Relationship to D-061, stated explicitly rather than assumed: the same PRINCIPLE (clean up the higher-layer record before destroying the lower-layer resource underneath it) carries over, but the MECHANISM differs, because this repo's OpenTofu layer uses the plain maas_vm_host resource (not MAAS pod-composition), so D-061's specific --keep-instance/decompose-on-release failure mode doesn't directly transfer. Flagged as a residual, untested gap (this repo has never reached Stage 4 for any VR1 DC, so no real MAAS-enrolled machine has ever existed under a VR1 vm_host to test a release procedure against) rather than inventing untested MAAS CLI commands.
  • Two paths: Path A (scoped single-site teardown via tofu's -target flag, with HashiCorp's own documented targeting-risk warning quoted and a mandatory untargeted tofu plan afterward to catch drift) and Path B (full VR1 teardown, plain tofu destroy, letting OpenTofu's own dependency graph sequence everything).
  • Explicit dependency ordering for Path A, since maas_vm_host has no Terraform-expressible dependency on the node-VM domains it's registered against (confirmed by reading modules/maas-vm-host/main.tf and modules/node-vm/main.tf directly -- the resource only references power_address, not a domain resource) and so would NOT be caught by a -target scoped to the storage/plane modules alone.
  • Mesh-link handling: explicit warning never to destroy a D-100 mesh-link leg during a single-site teardown (shared infrastructure with the far end), only when both endpoints are permanently going away. Also confirmed (by reading modules/netem-link/main.tf directly) that its destroy-time local-exec provisioner already correctly runs tc qdisc del -- no extra manual step needed for that piece specifically.
  • Rollback decision tree: four scenarios (nothing created yet, partial apply failure, suspected state corruption/loss, resources created outside OpenTofu via Stage 2's Option B manual virt-install path) with the default bias toward fix-forward over destroy, matching OpenTofu's own idempotent-convergence design rather than treating "something went wrong" as an automatic teardown trigger.
  • State backup as Step 1 of every path, cross-referencing DOCFIX-175 -- a destroy mutates state exactly like an apply does, so the same pre-mutation backup discipline applies.

Every tofu/terraform CLI flag cited (-destroy, -target, the targeting-risk warning text) was verified against HashiCorp's own current documentation before being written (fetched 2026-07-10), not asserted from memory -- same discipline as opentofu/README.md's other research sections.

Cross-referenced from opentofu/README.md (new "Teardown / rollback" section) and docs/dc-dc-deployment-workflow.md (gap register item #19, CLOSED).

Self-inflicted issue found and fixed mid-delivery (unrelated to this runbook, but in the same working session)

While preparing this delivery's DOCFIX number, bash scripts/ledger-scan.sh was re-run and caught that the DOCFIX-174 changelog and its docs/session-ledger.md mirror had REINTRODUCED the exact false-positive class DOCFIX-174 itself had just fixed -- both quoted the illustrative example number as a literal string while narrating the incident, which re-inflated ledger-scan.sh's reported next-free value. Caught by re-running the scan before assigning this delivery's own number rather than trusting the earlier fix was airtight; fixed by rewording both (see docs/changelog-20260710-ledger-scan-self-inflicted-falsepositive-fix.md's own updated "It recurred immediately" section for the full account). No separate DOCFIX assigned for this -- folded into the DOCFIX-174 changelog itself since it corrects that same delivery before commit, not a new one.

Also corrected mid-delivery: DOCFIX-175's first draft wrongly assumed Stage 3 ran the first real tofu apply; reading the runbooks directly showed Stage 1 does. Fixed across all of DOCFIX-175's files before this teardown-runbook delivery, so the "current plan" this runbook's own cross-references describe is accurate.

Verification

bash scripts/repo-lint.sh: 0 fail, 1 documented legacy warn. No script logic changed -- one new runbook, two doc cross-references, one README section. No test harness applies. bash scripts/ledger-scan.sh: confirms next-free is D-111/DOCFIX-176/BUNDLEFIX-013 after this delivery (was verified clean of the self-inflicted false positive before this number was assigned).

REVERT: git checkout HEAD~ -- runbooks/dc-dc-teardown-rollback.md opentofu/README.md docs/dc-dc-deployment-workflow.md plus git rm runbooks/dc-dc-teardown-rollback.md if reverting past its introduction commit (safe -- no live infrastructure depends on this, nothing here has ever been applied against real state).