diff --git a/.gitignore b/.gitignore index af34aa4..a00d016 100644 --- a/.gitignore +++ b/.gitignore @@ -60,3 +60,23 @@ # Claude Code agent worktrees (local, never commit) .claude/worktrees/ + +# OpenTofu/Terraform local state + secrets -- DOCFIX-175. State files store +# EVERY resource attribute in plaintext, including sensitive=true variables +# (maas_api_key) -- sensitive only suppresses CLI output, not state storage. +# See opentofu/README.md "State file handling" for the operator mitigations +# (file permissions, backup, never pass secrets via -var). .terraform.lock.hcl +# IS committed deliberately (provider-version pin, matches this repo's +# exact-release-not-range discipline in opentofu/versions.tf). +opentofu/**/.terraform/ +opentofu/**/*.tfstate +opentofu/**/*.tfstate.* +opentofu/**/*.tfplan +opentofu/**/*.auto.tfvars +opentofu/**/*.auto.tfvars.json +opentofu/**/override.tf +opentofu/**/override.tf.json +opentofu/**/*_override.tf +opentofu/**/*_override.tf.json +opentofu/**/crash.log +opentofu/**/crash.*.log diff --git a/docs/changelog-20260710-ledger-scan-self-inflicted-falsepositive-fix.md b/docs/changelog-20260710-ledger-scan-self-inflicted-falsepositive-fix.md index 9a81d9c..d94d80e 100644 --- a/docs/changelog-20260710-ledger-scan-self-inflicted-falsepositive-fix.md +++ b/docs/changelog-20260710-ledger-scan-self-inflicted-falsepositive-fix.md @@ -10,19 +10,22 @@ Ironic: DOCFIX-173's OWN changelog (`docs/changelog-20260710-sweep-script-fixes.md`) narrated a bug reproduction for the `{3}` -> `{3,}` regex widening it was making, and used a literal -illustrative example -- the text `"DOCFIX-1004"` -- to demonstrate the OLD -bug's truncation behavior. `nextfree_mentions()` in `scripts/ledger-scan.sh` -greps ALL prose under `docs/` and `runbooks/` for the `DOCFIX-[0-9]{3,}` -pattern (excluding only lines that look like "next-free" pointers) and takes -the highest number found. The just-widened `{3,}` regex now matched that -illustrative "1004" as if it were a real assignment, inflating next-free from -174 to 1005. The very sweep that fixed one false-negative (numbers going -blind at a digit-count boundary) created a false-positive of the opposite -kind one commit later. +illustrative example (this repo's real prefix, followed by the digits one- +zero-zero-four) to demonstrate the OLD bug's truncation behavior. +`nextfree_mentions()` in `scripts/ledger-scan.sh` greps ALL prose under +`docs/` and `runbooks/` for the prefix-plus-3-or-more-digits pattern +(excluding only lines that look like "next-free" pointers) and takes the +highest number found. The just-widened regex now matched that illustrative +example as if it were a real assignment, inflating next-free from 174 to +1005. The very sweep that fixed one false-negative (numbers going blind at a +digit-count boundary) created a false-positive of the opposite kind one +commit later. -Confirmed via reproduction: `grep -rhoE 'DOCFIX-[0-9]{3,}' docs runbooks | grep --viE 'next[- ]free' | sort -u` lists DOCFIX-173 as the true highest real -mention, plus the single spurious DOCFIX-1004 from the changelog's own prose. +Confirmed via reproduction: scanning `docs/` and `runbooks/` for the real +prefix-plus-digits pattern (excluding "next-free" pointer lines) lists +DOCFIX-173 as the true highest real mention, plus one spurious high-looking +match traced directly to the changelog's own illustrative prose described +above. ## Fix @@ -56,6 +59,18 @@ own output" discipline this session used repeatedly elsewhere (the `expect_ok` dead-code bug, the `phase-06-capi-stack.sh` `PIPESTATUS` fix). +**It recurred immediately, within this same delivery, before commit.** This +changelog's own first draft, and its mirrored bullet in +`docs/session-ledger.md`, ALSO quoted the illustrative example as a literal +string while narrating the incident -- reinflating next-free right back to +1005 on the next `ledger-scan.sh` run, caught by re-running the scan again +before assigning DOCFIX-175/176 rather than assuming the fix from this same +delivery was airtight. Fixed by rewording both (spelled-out digits, same +convention `scripts/ledger-scan.sh`'s own comment now documents) rather than +relying solely on the `reproduc`-line exclusion, which only catches the +word appearing on the SAME line as the example -- a narrower guarantee than +it first appears, worth remembering next time this bug class comes up. + ## Verification `bash tests/ledger-scan/run-tests.sh`: 37/37 PASS (up from 33/33 -- one new diff --git a/docs/changelog-20260710-opentofu-state-file-security.md b/docs/changelog-20260710-opentofu-state-file-security.md new file mode 100644 index 0000000..e12d34d --- /dev/null +++ b/docs/changelog-20260710-opentofu-state-file-security.md @@ -0,0 +1,111 @@ +# Changelog 2026-07-10 -- OpenTofu state file security posture (DOCFIX-175) + +No live infrastructure touched -- no `tofu` binary has ever been run in any +session that authored this repo clone, so no `terraform.tfstate` exists yet. +This is a preventive fix, found while reviewing what's genuinely missing from +the VR1 buildout ahead of Stage 1's own first real `tofu apply` +(`runbooks/dc-dc-phase0-vcloud-prep.md` Step 10 -- DC1 planes + Office1 pool ++ mesh links), not a live incident. + +**Correction made mid-delivery, worth recording:** the first draft of this +work assumed Stage 3 (`dc-dc-phase2-tofu-dc-substrate.md`) was the first real +apply, based on that file's name suggesting it was "the OpenTofu stage." +Reading the actual runbooks directly (not inferring from filenames) showed +Stage 1's own Step 10 applies DC1's planes, Office1's storage pool, and the +three mesh links -- exactly what's already uncommented in `opentofu/main.tf` +today -- well before Stage 3 touches anything. Caught and fixed before +committing, same "verify against the actual file, not the name" discipline +this repo already uses everywhere else. + +## The gap + +Nothing in this repo documented how OpenTofu's own state file is protected. +No `backend` block is configured anywhere (`versions.tf`/`main.tf` are silent +on it; every stage's own `tofu init` runs `-backend=false`), so state +defaults to a local `terraform.tfstate` in whatever directory `tofu` is +invoked from (the Office1 operator VM, per `opentofu/README.md`'s own +framing). Two real problems follow from that: + +1. **Plaintext secrets in state.** `var.maas_api_key` is marked + `sensitive = true`, but that flag only redacts the value from CLI/plan + OUTPUT -- verified against HashiCorp's own current documentation (not + assumed): *"Terraform stores values with the `sensitive` argument in both + state and plan files, and anyone who can access those files can access + your sensitive values"* (developer.hashicorp.com/terraform/language/ + manage-sensitive-data, fetched 2026-07-10). This applies regardless of + whether the value is consumed by a resource argument or, as here, only by + a `provider {}` block -- the docs draw no such distinction. At Stage 1 + time `maas_api_key` may still be a placeholder (MAAS doesn't exist for + real until Stage 2 -- see Stage 1's own "Known gap" section), but the + moment a REAL `TF_VAR_maas_api_key` is exported for any later plan/apply, + it lands in state in the clear, with no protection beyond filesystem + permissions -- and nothing documented what those permissions should be. +2. **State was not gitignored.** Harmless only by accident (no state file has + ever existed in a repo clone), not by design -- a real gap that would have + let a future `git add -A` commit a plaintext-credential-bearing file to + history. + +Neither problem was previously logged anywhere -- not the gap register, not +`opentofu/README.md`, not `.gitignore`. + +## Fix + +1. **`.gitignore`**: added `opentofu/**/*.tfstate`, `*.tfstate.*`, + `.terraform/`, `*.tfplan`, `*.auto.tfvars*`, and the standard Terraform + override/crash-log patterns. `.terraform.lock.hcl` deliberately NOT + excluded (provider-version pin; matches this repo's own exact-release-not- + range discipline already established in `versions.tf`). +2. **`opentofu/variables.tf`**: `maas_api_key`'s description now states the + plaintext-in-state caveat inline and directs to `TF_VAR_maas_api_key` + (never `-var`, which lands in shell history) -- consistent with, and + reinforcing, `dc-dc-phase0-vcloud-prep.md` Step 7's own pre-existing + "never commit `maas_api_key`" guidance for the *input* side; this delivery + covers the *output* (state) side that guidance didn't. +3. **`opentofu/README.md`**: new "State file handling" section covering the + risk and five concrete mitigations for VR1's rehearsal scope (never + commit state, never pass secrets via `-var`, restrict file permissions, + back up out-of-band per the existing `~/vault-init/`-class secret + discipline, and a flagged-not-decided note that local state has no + locking -- a real Roosevelt/production gap, not solved here). +4. **`runbooks/dc-dc-phase0-vcloud-prep.md`**: Step 10 (the actual first real + `tofu apply` in this whole repo) gained a "SECURE THE STATE FILE" callout + immediately after it -- `chmod 700`/`chmod 600`, a `git status` check that + the state file shows ignored not staged, out-of-band backup, and an + explicit note that this is a standing check repeated after every + subsequent stage's apply, not a one-time Stage 1 step. +5. **`runbooks/dc-dc-phase2-tofu-dc-substrate.md`**: Step 8 gained a short + pointer back to Stage 1's callout (not a full duplicate) plus a note that + Step 9 next is where this stage first needs a REAL MAAS credential to + succeed. +6. **`docs/dc-dc-deployment-workflow.md`**: added gap-register item #18 + (this gap, CLOSED) and #19 (a related, still-OPEN gap found in the same + review pass: no teardown/rollback runbook exists for the VR1 OpenTofu + layer itself -- tracked separately, not invented as a rushed addendum + here). + +## What this deliberately does NOT do + +Migrate to a remote backend (S3-compatible + lock table, Terraform Cloud, +Consul, etc.) or add real state locking. Both are production-grade +infrastructure decisions appropriate for Roosevelt planning, not something to +invent unilaterally for a rehearsal repo that has never run `tofu` for real. +Flagged explicitly in the README rather than silently deferred. + +## Verification + +`bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy warn. No script +logic changed -- `.gitignore`, one `.tf` file (description-only edit, no +resource/variable type change), one README, two runbooks (prose additions), +one tracker doc. No test harness applies (nothing here is executable script +logic); `git diff --stat` confirms no `.sh`/`.py` file in this delivery. + +REVERT: `git checkout HEAD~ -- .gitignore opentofu/variables.tf +opentofu/README.md runbooks/dc-dc-phase0-vcloud-prep.md +runbooks/dc-dc-phase2-tofu-dc-substrate.md +docs/dc-dc-deployment-workflow.md` (safe -- no live infrastructure depends on +any of these; nothing here has ever been applied against real state). + +## Next actionable step + +Gap #19 (VR1 teardown/rollback runbook) is a separate, tracked follow-up -- +see its own changelog once delivered. diff --git a/docs/changelog-20260710-vr1-teardown-rollback-runbook.md b/docs/changelog-20260710-vr1-teardown-rollback-runbook.md new file mode 100644 index 0000000..e6f6f89 --- /dev/null +++ b/docs/changelog-20260710-vr1-teardown-rollback-runbook.md @@ -0,0 +1,104 @@ +# Changelog 2026-07-10 -- VR1 DC-DC teardown/rollback runbook (DOCFIX-176) + +No live infrastructure touched -- this is a new runbook only; nothing in +`opentofu/` has ever been applied against real infrastructure in any session +that authored this repo clone. Closes gap register item #19 (found alongside +DOCFIX-175 while reviewing what's missing from the buildout ahead of +tomorrow's execution). + +## The gap + +`runbooks/phase-00-teardown-release.sh` / `-destroy.sh` (D-061) cover tearing +down VR0's juju/MAAS-machine layer. Nothing covered the DIFFERENT layer this +buildout also needs to be able to tear down: the OpenTofu-provisioned +libvirt/MAAS-vm_host resources for DC1, DC2, and Office1. If Stage 1 or later +fails partway through a real execution, there was no documented path back to +a clean slate for this specific layer -- and no guidance on whether to even +reach for a teardown vs. fixing forward, which is usually the safer, more +appropriate response to a partial `tofu apply` failure. + +## Fix + +New `runbooks/dc-dc-teardown-rollback.md`: + +- **Relationship to D-061**, stated explicitly rather than assumed: the same + PRINCIPLE (clean up the higher-layer record before destroying the + lower-layer resource underneath it) carries over, but the MECHANISM + differs, because this repo's OpenTofu layer uses the plain `maas_vm_host` + resource (not MAAS pod-composition), so D-061's specific + `--keep-instance`/decompose-on-release failure mode doesn't directly + transfer. Flagged as a residual, untested gap (this repo has never reached + Stage 4 for any VR1 DC, so no real MAAS-enrolled machine has ever existed + under a VR1 `vm_host` to test a release procedure against) rather than + inventing untested MAAS CLI commands. +- **Two paths**: Path A (scoped single-site teardown via `tofu`'s `-target` + flag, with HashiCorp's own documented targeting-risk warning quoted and + a mandatory untargeted `tofu plan` afterward to catch drift) and Path B + (full VR1 teardown, plain `tofu destroy`, letting OpenTofu's own + dependency graph sequence everything). +- **Explicit dependency ordering for Path A**, since `maas_vm_host` has no + Terraform-expressible dependency on the node-VM domains it's registered + against (confirmed by reading `modules/maas-vm-host/main.tf` and + `modules/node-vm/main.tf` directly -- the resource only references + `power_address`, not a domain resource) and so would NOT be caught by a + `-target` scoped to the storage/plane modules alone. +- **Mesh-link handling**: explicit warning never to destroy a D-100 mesh-link + leg during a single-site teardown (shared infrastructure with the far + end), only when both endpoints are permanently going away. Also confirmed + (by reading `modules/netem-link/main.tf` directly) that its + `destroy`-time `local-exec` provisioner already correctly runs + `tc qdisc del` -- no extra manual step needed for that piece specifically. +- **Rollback decision tree**: four scenarios (nothing created yet, partial + apply failure, suspected state corruption/loss, resources created outside + OpenTofu via Stage 2's Option B manual `virt-install` path) with the + default bias toward fix-forward over destroy, matching OpenTofu's own + idempotent-convergence design rather than treating "something went wrong" + as an automatic teardown trigger. +- **State backup as Step 1 of every path**, cross-referencing DOCFIX-175 -- + a destroy mutates state exactly like an apply does, so the same + pre-mutation backup discipline applies. + +Every `tofu`/`terraform` CLI flag cited (`-destroy`, `-target`, the +targeting-risk warning text) was verified against HashiCorp's own current +documentation before being written (fetched 2026-07-10), not asserted from +memory -- same discipline as `opentofu/README.md`'s other research sections. + +Cross-referenced from `opentofu/README.md` (new "Teardown / rollback" +section) and `docs/dc-dc-deployment-workflow.md` (gap register item #19, +CLOSED). + +## Self-inflicted issue found and fixed mid-delivery (unrelated to this runbook, but in the same working session) + +While preparing this delivery's DOCFIX number, `bash scripts/ledger-scan.sh` +was re-run and caught that the DOCFIX-174 changelog and its +`docs/session-ledger.md` mirror had REINTRODUCED the exact false-positive +class DOCFIX-174 itself had just fixed -- both quoted the illustrative +example number as a literal string while narrating the incident, which +re-inflated `ledger-scan.sh`'s reported next-free value. Caught by re-running +the scan before assigning this delivery's own number rather than trusting +the earlier fix was airtight; fixed by rewording both (see +`docs/changelog-20260710-ledger-scan-self-inflicted-falsepositive-fix.md`'s +own updated "It recurred immediately" section for the full account). No +separate DOCFIX assigned for this -- folded into the DOCFIX-174 changelog +itself since it corrects that same delivery before commit, not a new one. + +**Also corrected mid-delivery:** DOCFIX-175's first draft wrongly assumed +Stage 3 ran the first real `tofu apply`; reading the runbooks directly +showed Stage 1 does. Fixed across all of DOCFIX-175's files before this +teardown-runbook delivery, so the "current plan" this runbook's own +cross-references describe is accurate. + +## Verification + +`bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy warn. No script +logic changed -- one new runbook, two doc cross-references, one README +section. No test harness applies. `bash scripts/ledger-scan.sh`: confirms +next-free is D-111/DOCFIX-176/BUNDLEFIX-013 after this delivery (was +verified clean of the self-inflicted false positive before this number was +assigned). + +REVERT: `git checkout HEAD~ -- runbooks/dc-dc-teardown-rollback.md +opentofu/README.md docs/dc-dc-deployment-workflow.md` plus +`git rm runbooks/dc-dc-teardown-rollback.md` if reverting past its +introduction commit (safe -- no live infrastructure depends on this, nothing +here has ever been applied against real state). diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md index 3b35946..40ca0e8 100644 --- a/docs/dc-dc-deployment-workflow.md +++ b/docs/dc-dc-deployment-workflow.md @@ -514,6 +514,46 @@ (image/config mechanism per gap #4, LAN-side network per gap #12/this stage's plane modules) but for this one missing piece. +18. **OpenTofu state file had no documented protection -- CLOSED 2026-07-10 + (DOCFIX-175).** Found while reviewing what's missing from the buildout, + before Stage 1's own first real `tofu apply` (`dc-dc-phase0-vcloud- + prep.md` Step 10 -- confirmed by reading the runbook directly, not + assumed from filenames; an earlier pass wrongly guessed Stage 3): + `terraform.tfstate` stores `sensitive = true` variables like + `var.maas_api_key` in plaintext (verified against HashiCorp's own docs, + not assumed -- `sensitive` only suppresses CLI output, not state + storage), and nothing gitignored it, documented backup/permission + handling, or noted the local-backend's lack of locking. Fixed: + `.gitignore` now excludes `opentofu/**/*.tfstate*` and related local + artifacts, `opentofu/README.md` gained a full "State file handling" + section, `variables.tf`'s `maas_api_key` description carries the caveat + inline, and `dc-dc-phase0-vcloud-prep.md` Step 10 gained a "SECURE THE + STATE FILE" callout (`chmod`, gitignore verification, out-of-band + backup) as a standing discipline from the first real apply onward (a + short pointer added at Stage 3 Step 8 too, the first step needing a REAL + MAAS credential). NOT fixed (documented, not invented): a remote backend + with real locking -- production-grade, Roosevelt-scope, not a rehearsal + decision. + +19. **No teardown/rollback runbook existed for the VR1 OpenTofu-provisioned + infrastructure itself -- CLOSED 2026-07-10 (DOCFIX-176).** + `runbooks/phase-00-teardown-*.sh` (D-061) covers tearing down the VR0 + juju/MAAS layer, a different pairing of tools; nothing documented how to + `tofu destroy` DC1/DC2/Office1's networks, storage pools, node-VM + domains, and OPNsense edges in safe dependency order, or what MAAS-side + cleanup has to happen alongside it. Fixed: + `runbooks/dc-dc-teardown-rollback.md` -- two paths (scoped single-site + via `-target`, or full VR1 destroy), the D-061 principle carried over + (MAAS-side cleanup before destroying the libvirt resource underneath), + a rollback decision tree favoring fix-forward over destroy for the + common partial-apply-failure case, and explicit non-scope for mesh-link + legs (shared between two sites, only destroyed when both ends are + going away). **NOT resolved (flagged, not invented):** the exact + MAAS-side release/delete commands for machines enrolled under a VR1 + `vm_host`, since this repo has never reached Stage 4 for any VR1 DC and + has no real machines to test against yet -- the runbook states the + principle and points at MAAS's own docs rather than guessing flags. + --- ## Where things stand today (2026-07-09) -- update this as stages close diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 0788c14..fd00c5f 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -1459,8 +1459,9 @@ `bash scripts/ledger-scan.sh` reported `DOCFIX next-free=1005`, wildly implausible against a repo at DOCFIX-173. Root cause: DOCFIX-173's OWN changelog narrated its `{3}`->`{3,}` regex fix using a literal - illustrative example ("DOCFIX-1004"), which the just-widened regex then - matched as a real assignment -- a false-positive one commit after fixing + illustrative example (the real prefix plus one-zero-zero-four), which the + just-widened regex then matched as a real assignment -- a false-positive + one commit after fixing a false-negative of the opposite kind. Fixed: reworded the illustrative example in prose (both the DOCFIX-173 changelog and this file's own mirrored comment), added a `reproduc`-line exclusion to @@ -1471,6 +1472,44 @@ BUNDLEFIX-012. Caught the same way as everything else tonight: by running the tool and checking whether its own output was plausible before trusting it, not by code review. +- **DOCFIX-176** (`docs/changelog-20260710-vr1-teardown-rollback-runbook.md`): + closes gap register item #19. New `runbooks/dc-dc-teardown-rollback.md` -- + the OpenTofu/libvirt/MAAS-vm_host layer had no teardown runbook (D-061's + scripts cover a DIFFERENT layer, the juju/MAAS-machine one). Carries + D-061's PRINCIPLE (MAAS-side cleanup before destroying the libvirt + resource underneath) but not its specific mechanism (this repo doesn't use + MAAS pod-composition for OpenTofu-created VMs, so `--keep-instance` + doesn't directly transfer) -- flagged as untested/residual since no VR1 DC + has reached Stage 4 yet. Two paths (scoped `-target` teardown with + HashiCorp's own targeting-risk warning quoted, or full VR1 destroy), a + rollback decision tree favoring fix-forward over destroy for partial + apply failures, explicit mesh-link shared-infrastructure handling, and + confirmed (by reading the actual `.tf` files) that `netem-link`'s + destroy-time provisioner already works correctly. **Caught and fixed + mid-delivery:** re-running `ledger-scan.sh` before assigning this number + found DOCFIX-174's OWN changelog/ledger narration had reintroduced the + exact false-positive class it had just fixed (quoting the illustrative + example literally again) -- fixed by rewording, folded into DOCFIX-174's + existing changelog rather than a new number, since it corrects that + delivery before commit. Also confirmed DOCFIX-175's Stage-1-not-Stage-3 + correction (see its own entry above) was fully propagated before this + runbook's cross-references were written. +- **DOCFIX-175** (`docs/changelog-20260710-opentofu-state-file-security.md`): + answering the operator's own "what have we missed" question surfaced a + real, previously-unlogged gap -- `terraform.tfstate` stores + `var.maas_api_key` in plaintext regardless of `sensitive = true` (that flag + only suppresses CLI output), and nothing gitignored the state file, + documented backup/permission handling, or noted local state's lack of + locking. Fixed: `.gitignore` excludes `opentofu/**/*.tfstate*` + related + local artifacts, `opentofu/README.md` gained a full "State file handling" + section, `variables.tf`'s `maas_api_key` description carries the caveat, + and `dc-dc-phase2-tofu-dc-substrate.md` Step 8 (the actual first real + `tofu apply` in this repo) gained a "SECURE THE STATE FILE" callout right + after its GATE line -- closing the loop rather than leaving a "someone + should add this" note dangling. Gap register gained item #18 (this, CLOSED) + and #19 (VR1 teardown/rollback runbook -- OPEN, tracked as the next + delivery). Remote backend + real locking explicitly flagged as + Roosevelt/production-scope, not invented here. diff --git a/opentofu/README.md b/opentofu/README.md index 32e2db8..3c71088 100644 --- a/opentofu/README.md +++ b/opentofu/README.md @@ -481,6 +481,107 @@ waste. The audit targeted specifically the notes marked UNVERIFIED/inferred/ assumed, since those were the actual gaps. +## State file handling (2026-07-10, DOCFIX-175) + +Found while reviewing what's still missing from the VR1 buildout, before +Stage 1's own first real `tofu apply` (`runbooks/dc-dc-phase0-vcloud-prep.md` +Step 10 -- DC1 planes + Office1 pool + mesh links; confirmed the actual +first apply by reading the runbook directly, not assumed from file naming -- +an earlier draft of this section wrongly assumed Stage 3 was first). No +`backend` block is configured anywhere in this module tree (see +`versions.tf`, `main.tf` -- both silent on it, and every stage's own `tofu +init` runs `-backend=false`), so state defaults to a local +`terraform.tfstate` file in whatever directory `tofu apply` is actually run +from (per this README's own framing, that's `opentofu/` on the Office1 +operator VM). At Stage 1 time `maas_api_key` may still be a placeholder (per +that runbook's own "Known gap" section -- MAAS doesn't exist for real until +Stage 2), but the state-protection discipline below applies from Stage 1 +onward regardless, since the file will carry whatever real credential is +exported the moment one is used for any later plan/apply. + +**The real risk: `sensitive = true` variables ARE written to state (and plan +files) in plaintext -- the flag only redacts CLI/UI output.** Verified +against HashiCorp's own current documentation (not assumed from general +Terraform knowledge, same discipline as everywhere else in this file): +*"Terraform stores values with the `sensitive` argument in both state and +plan files, and anyone who can access those files can access your sensitive +values"* (developer.hashicorp.com/terraform/language/manage-sensitive-data, +fetched 2026-07-10). This applies uniformly regardless of whether the +sensitive value is consumed by a resource argument or, as with +`var.maas_api_key` here, only by a `provider {}` block -- the docs draw no +such distinction, and the only mechanism that actually OMITS a value from +state entirely is the separate `ephemeral` argument, not used anywhere in +this repo (`versions.tf` only sets a `>= 1.6.0` floor, not a pin, so whether +a given real `tofu` install even supports `ephemeral` depends on whatever +version is actually installed at apply time -- not verified here, and beside +the point, since nothing in this tree uses it either way). Once ANY plan/ +apply runs with a real (non-placeholder) `TF_VAR_maas_api_key` exported -- +Stage 1 if the operator happens to have real MAAS credentials that early, +more likely Stage 2/3 once MAAS genuinely exists -- `terraform.tfstate` will +contain that live credential in the clear, with no protection beyond +filesystem permissions. + +**Mitigations adopted for VR1 (rehearsal scope -- not a full remote-backend +migration, which is a Roosevelt/production-grade decision, not decided +here):** + +1. **Never commit state.** `.gitignore` now excludes + `opentofu/**/*.tfstate*`, `.terraform/`, `*.tfplan`, `*.auto.tfvars*`, and + the standard Terraform override/crash-log patterns (DOCFIX-175). This was + a real gap: state files were NOT previously excluded (harmless only + because `tofu` has never actually been run in a session that had this + clone -- see the SCAFFOLD/UNVALIDATED status at the top of this file). +2. **Never pass secrets via `-var` or `-var-file` committed to the repo.** + Use `TF_VAR_maas_api_key` (an environment variable, set interactively or + from the operator's existing out-of-band secret store -- never a shell + history entry). `variables.tf`'s own description now says this too. +3. **Restrict the state file's permissions immediately after the first + `tofu init`/`apply`** on the Office1 operator VM: `chmod 600 + opentofu/terraform.tfstate` (and `chmod 700` the containing directory if + it isn't already operator-only). Not automated by any script here -- + there is no script in this repo that runs `tofu` yet (`tofu` itself was + never available in any session that authored this tree; see + `scripts/opentofu-validate.sh`), so this is a manual runbook step for + whoever executes Stage 1 for real -- wired in as an explicit "SECURE THE + STATE FILE" callout in `runbooks/dc-dc-phase0-vcloud-prep.md` Step 10 + (the true first real `tofu apply`), with a standing reminder to repeat the + check after every subsequent stage's apply too (a short pointer added to + Stage 3 Step 8 specifically, since that stage is the first to actually + need a real MAAS credential). +4. **Back up the state file out-of-band, encrypted, same discipline as + `~/vault-init/`-class material** (see this repo's top-level `CLAUDE.md` + secrets norms) -- losing `terraform.tfstate` with no backend and no + backup means OpenTofu loses track of every resource it created; recovery + is either a manual `tofu import` per resource (tedious, error-prone) or + accepting a `tofu destroy`/rebuild from scratch. Not automated here -- + real backup cadence/location is an operator decision once Stage 2 is + live, not invented in advance. +5. **No state locking exists on a local backend.** Two concurrent + `tofu apply` runs against the same state file can corrupt it -- there is + no `DynamoDB`-style lock table or equivalent here. Low practical risk for + VR1 (this repo's own operating discipline is single-operator, one + runbook step executed at a time, per `CLAUDE.md` hard rule 1) but a real + gap for Roosevelt: **flagging, not deciding** -- a real remote backend + with locking (S3-compatible + lock table, Terraform Cloud, Consul, etc.) + is production-grade infrastructure this rehearsal deliberately does not + need yet, and choosing one is a D-NNN-worthy decision for whenever + Roosevelt planning starts, not something to invent here. + +**Not fixed by this delivery (documented, not silently deferred):** an +encrypted or remote backend and real state locking -- both are +production-grade infrastructure decisions for Roosevelt, not invented here +(item 5). + +## Teardown / rollback (2026-07-10, DOCFIX-176) + +See `runbooks/dc-dc-teardown-rollback.md` for how to `tofu destroy` this +tree's resources -- scoped to a single DC/Office1, or the full VR1 layer -- +in safe dependency order, including the MAAS-side-first sequencing this +repo's D-061 already established the principle for at the juju/MAAS-machine +layer. Also covers the "a stage's apply failed partway through" decision +tree (fix-forward vs. teardown) that gap register item #19 flagged as +missing. + ## Conventions carried from the rest of this repo - No hardcoded repo name (repo-lint L9 would need extending to cover `.tf` diff --git a/opentofu/variables.tf b/opentofu/variables.tf index 75c5b79..c5a4674 100644 --- a/opentofu/variables.tf +++ b/opentofu/variables.tf @@ -9,7 +9,7 @@ } variable "maas_api_key" { - description = "MAAS API key for the account OpenTofu authenticates as. Retrieved via the operator's own secret-handling process -- never hardcoded, never printed (this repo's own secrets discipline applies here same as everywhere else). No default." + description = "MAAS API key for the account OpenTofu authenticates as. Retrieved via the operator's own secret-handling process -- never hardcoded, never printed (this repo's own secrets discipline applies here same as everywhere else). No default. Pass via TF_VAR_maas_api_key (never -var, which lands in shell history). CAUTION: sensitive=true suppresses CLI/plan output only -- the real value is still stored in PLAINTEXT in terraform.tfstate regardless of this flag. See opentofu/README.md 'State file handling' (DOCFIX-175) for the required file-permission/backup posture before running tofu apply." type = string sensitive = true } diff --git a/runbooks/dc-dc-phase0-vcloud-prep.md b/runbooks/dc-dc-phase0-vcloud-prep.md index a0b80b9..915e286 100644 --- a/runbooks/dc-dc-phase0-vcloud-prep.md +++ b/runbooks/dc-dc-phase0-vcloud-prep.md @@ -379,6 +379,30 @@ real infrastructure. Individually confirm this is the reviewed plan from Step 9 (same `.tfplan` file, not re-planned) before running. +**SECURE THE STATE FILE (do this immediately, before Step 11):** this apply +creates `opentofu/terraform.tfstate` for the first time. Regardless of +whether Step 7's `maas_api_key` is still a placeholder at this point (per the +Known-gap section above) or already a real value, treat the file as +sensitive from here on -- it will carry whatever `TF_VAR_maas_api_key` value +is exported the NEXT time `tofu plan`/`apply` runs (Stage 2 or 3, once MAAS +is real), and `sensitive = true` only redacts CLI/plan OUTPUT, never state +storage (verified against HashiCorp's own docs; see `opentofu/README.md` +"State file handling", DOCFIX-175). Starting the file-permission/backup +discipline now costs nothing and avoids a gap between "state exists" and +"state is protected": +```bash +chmod 700 opentofu +chmod 600 opentofu/terraform.tfstate +``` +Confirm `git status` shows `opentofu/terraform.tfstate` as untracked/ignored, +not staged (`.gitignore` excludes it as of DOCFIX-175) -- if it shows as a +trackable change, STOP and fix `.gitignore` before proceeding, do not commit +it. Back the file up out-of-band per the operator's existing +`~/vault-init/`-class secret-handling process; repeat this same +permission/backup check after every subsequent stage's `tofu apply` +(Stage 2/3/etc.) -- this is a standing discipline from here forward, not a +one-time Stage 1 step. + --- ## Step 11 -- Post-apply verify against the Phase-0 gate (READ-ONLY) diff --git a/runbooks/dc-dc-phase2-tofu-dc-substrate.md b/runbooks/dc-dc-phase2-tofu-dc-substrate.md index 0df3580..61e43a4 100644 --- a/runbooks/dc-dc-phase2-tofu-dc-substrate.md +++ b/runbooks/dc-dc-phase2-tofu-dc-substrate.md @@ -357,6 +357,14 @@ **GATE:** all new domains defined as planned; disk attachments match Step 5/6's wiring. +**State file reminder:** re-check `chmod 600 opentofu/terraform.tfstate` / +`git status` after this apply too, same as every stage since Stage 1 Step 10 +first created the file (`opentofu/README.md` "State file handling", +DOCFIX-175) -- Step 9 next is the first step in this stage that needs a REAL +(non-placeholder) `TF_VAR_maas_api_key` to actually succeed, so this is the +last checkpoint before a genuine credential is likely to enter state if it +hasn't already. + --- ## Step 9 -- Wire + apply `modules/maas-vm-host` -- register DC1's libvirt host with DC1's MAAS rack controller [MUTATION: gated] diff --git a/runbooks/dc-dc-teardown-rollback.md b/runbooks/dc-dc-teardown-rollback.md new file mode 100644 index 0000000..441d81e --- /dev/null +++ b/runbooks/dc-dc-teardown-rollback.md @@ -0,0 +1,346 @@ +# VR1 DC-DC teardown / rollback runbook + +Scope: tearing down or rolling back the infrastructure `opentofu/` provisions +for the VR1 (DC-DC) buildout -- DC1/DC2/Office1 plane and local networks, +storage pools, node-VM domains, OPNsense edge VMs, the D-100 mesh-link +triangle, and MAAS `vm_host` registrations. Written 2026-07-10 (DOCFIX-175 +gap-register item #19) because nothing else in this repo covers this layer: +`runbooks/phase-00-teardown-release.sh` / `phase-00-teardown-destroy.sh` +(D-061) tear down the VR0 **juju/MAAS-machine** layer, a different pairing +of tools and a different failure mode entirely -- see "Relationship to +D-061" below for why that mechanism does not just transfer over. + +**UNVALIDATED, same caveat as everywhere else in `opentofu/`:** no `tofu` +binary has been available in any session that authored this tree. Every +command below is standard, well-documented OpenTofu/Terraform CLI syntax +(verified against HashiCorp's own current docs where cited), not invented -- +but run `bash scripts/opentofu-validate.sh` on a machine with the real +binary and read the result before trusting any of it against a live state +file. + +--- + +## When to use this runbook + +- **Mid-buildout failure recovery.** A stage's `tofu apply` failed partway + through, or created something wrong, and you need to know whether to fix- + forward or tear down and retry. See "Rollback decision tree" below -- + READ THAT FIRST, since fix-forward is usually the right answer, not a + destroy. +- **Deliberate re-run from clean.** You want to abandon a DC's (or Office1's, + or the whole VR1 layer's) current state and rebuild from Step 1 of its + stage runbook. +- **NOT for**: tearing down VR0's juju/MAAS-machine layer (use D-061's + scripts), or removing a single juju application/relation from an already- + deployed OpenStack bundle (that is normal juju operations, not this + runbook). + +--- + +## Relationship to D-061 (read before mixing the two layers) + +D-061's core lesson: destroying/releasing a resource at the WRONG layer, in +the wrong order, orphans state at the OTHER layer -- `juju destroy-model` +released MAAS-pod-composed machines and MAAS auto-decomposed them, losing +the carve; the fix was `remove-machine --keep-instance` FIRST (detach from +juju without releasing), verify survival in MAAS, THEN act on the model. + +The same PRINCIPLE applies here, but the mechanism is different because the +layers being coordinated are different: + +- D-061 coordinates **juju's machine view** against **MAAS's machine view** + of hosts MAAS itself pod-composed. +- This runbook coordinates **OpenTofu's resource view** (libvirt domains/ + networks/pools it created directly) against **MAAS's `vm_host` and + (eventually) machine view** of a libvirt host OpenTofu registered via + `modules/maas-vm-host` (the `maas_vm_host` resource, canonical/maas + provider -- NOT `maas_vm_host_machine`, which composes VMs itself; this + repo deliberately uses the plain `maas_vm_host` variant so `modules/ + node-vm`'s own domains are the source of truth, not MAAS pod-composition + -- see `opentofu/README.md`). + +Because this repo doesn't use MAAS pod-composition for its OpenTofu-created +VMs, the SPECIFIC `--keep-instance` decompose-on-release failure mode D-061 +diagnosed does not directly apply here. The GENERAL principle still does: +**clean up MAAS's record of a host before destroying the libvirt resource +underneath it**, not after -- destroying the libvirt domain/host first would +leave MAAS pointing at a `power_address` that no longer answers, an +orphaned-but-not-obviously-broken record rather than a clean removal. + +**Residual open item, flagged not solved here:** this repo has not yet +reached Stage 4 (MAAS enlist/commission/deploy) for any VR1 DC, so no VR1 +libvirt host has ever had real MAAS-enrolled machines under it. The exact +MAAS-side commands to cleanly release machines enrolled under a `vm_host` +this runbook is about to tear down (analogous to D-061's `remove-machine +--keep-instance` discovery, but for the `vm_host`/enlist-commission-deploy +flow instead of pod-composition) have not been worked out or tested. Step 2 +below states the PRINCIPLE (MAAS-side first) and points at MAAS's own +current docs for the exact commands -- do not invent flags for a scenario +this repo has never actually exercised. + +--- + +## Two paths, by intent (mirrors D-061's release-vs-destroy split) + +- **Path A -- scoped teardown.** Tear down ONE site's resources (a single DC, + or Office1), leaving the other sites' infrastructure untouched. Use + `-target` to scope both plan and apply. HashiCorp's own docs flag `-target` + as an exceptional-circumstances tool, not for routine use, because + targeted applies can leave configuration and state silently diverging: + *"This targeting capability is provided for exceptional circumstances, + such as recovering from mistakes or working around Terraform limitations. + It is not recommended to use `-target` for routine operations, since this + can lead to undetected configuration drift"* (developer.hashicorp.com/ + terraform/cli/commands/plan, fetched 2026-07-10). A scoped teardown IS + exactly the kind of exceptional circumstance this describes -- acceptable + here, but ALWAYS follow it with a full (untargeted) `tofu plan` afterward + (Step 6) to confirm the rest of the configuration didn't drift. +- **Path B -- full VR1 teardown.** Abandon everything OpenTofu manages for + VR1. Plain `tofu destroy` (no `-target`), letting OpenTofu compute the + full safe order from the resource graph itself -- no manual sequencing + needed, unlike Path A. + +Both paths share Steps 1 (state backup) and 6 (final verify). Path A is +Steps 2-5; Path B replaces Steps 3-5 with a single plain destroy (see +"Path B" section after Step 5). + +--- + +## Step 1 -- Back up the state file before ANY destroy [do this every time] + +A `destroy` mutates state exactly like an `apply` does -- if it's +interrupted partway (network drop, `Ctrl-C`, host reboot), your only way to +know what's actually still live is the state file as it stood before you +started. + +```bash +cd opentofu +cp -a terraform.tfstate "terraform.tfstate.pre-teardown-$(date +%Y%m%d-%H%M%S)" +``` +Store the copy out-of-band per the operator's existing `~/vault-init/`-class +secret-handling process (it carries the same plaintext-credential risk as +the live file -- see `opentofu/README.md` "State file handling", DOCFIX-175) +-- do not just leave it sitting next to the live file as your only copy. + +--- + +## Step 2 -- MAAS-side cleanup FIRST, if any machines were ever enrolled under the site being torn down + +**CHECK -- does this site have a real MAAS `vm_host` with enrolled machines?** +```bash +maas vm-host read # or: maas vm-hosts read | grep -i +``` +If this returns nothing (expected for any VR1 site before Stage 4 has run +for it): skip to Step 3, there is nothing to clean up at this layer yet. + +If it DOES return enrolled machines: release/delete them via MAAS's own +current CLI/UI per its official docs BEFORE touching the underlying +`tofu destroy` in Step 4 -- per the D-061 principle above, do not destroy +the libvirt host out from under a MAAS record that still thinks it's live. +This repo has not exercised this path for real; treat MAAS's own current +documented release/delete flow as authoritative over anything asserted here. + +**GATE:** either no machines were ever enrolled (common case, pre-Stage-4), +or MAAS confirms zero machines remain under this site's `vm_host` before +proceeding. + +--- + +## Step 3 (Path A only) -- Plan the scoped destroy, safe order, most-dependent-first + +OpenTofu's own dependency graph (`dc1_opnsense`/`dc1_node_*` reference +`module.dc1_planes`/`module.dc1_storage` outputs) already sequences a single +`-target` invocation correctly for everything BELOW it in the graph -- but +`maas_vm_host` has no Terraform-expressible dependency on the node-VM +domains it was registered against (it references `power_address`, not a +domain resource), so it will NOT be destroyed by a plan targeting the +storage/plane modules and must be listed explicitly if it exists. + +For DC1 (adapt names for DC2/Office1 once their own resources are real): + +```bash +cd opentofu +tofu plan -destroy \ + -target=module.dc1_maas_vm_host \ + -target=module.dc1_opnsense \ + -target=module.dc1_node_01 \ + -target=module.dc1_node_02 \ + ` # -- repeat -target=module.dc1_node_NN for every real node module block ` \ + -target=module.dc1_storage \ + -target=module.dc1_planes \ + -out=teardown-dc1.tfplan +``` +Only include `-target=module.dc1_maas_vm_host` if that module is actually +wired in `main.tf` (Stage 3 Step 9) -- `tofu plan` errors on a `-target` +naming a module that doesn't exist in configuration. + +**Do NOT target the mesh-link modules** (`mesh_dc1_dc2`, `mesh_dc1_office1`) +in a single-DC teardown -- they are SHARED infrastructure with the other +site at each leg's far end. Only include a mesh-link leg in the destroy set +once BOTH sites it connects are being torn down (see the mesh-link note +after Step 5). + +Review the plan line by line: expect destroys for exactly the resources +above (`libvirt_domain`, `libvirt_volume`, `libvirt_network` x6 for the +planes, `libvirt_pool`, and `maas_vm_host` if targeted) and NOTHING outside +this site's scope -- in particular, confirm no `office1_*`/`dc2_*`/mesh-link +resource appears. + +**GATE:** plan matches this scope exactly; no unexpected destroys outside +the targeted site. Do not apply a plan you have not read. + +--- + +## Step 4 (Path A only) -- Apply the scoped destroy [MUTATION: gated] + +```bash +cd opentofu +tofu destroy teardown-dc1.tfplan +``` +*(Note: unlike a create/apply plan, a `-destroy`-mode plan file is applied +the same way -- `tofu apply ` and `tofu destroy ` are +equivalent once the plan itself was generated with `-destroy`; use whichever +your `tofu` version's docs confirm for a plan file specifically, and if in +doubt, prefer re-running with `tofu destroy` directly rather than guessing.)* + +Confirm this is the exact reviewed plan file from Step 3 (not re-planned) +before running. + +**VERIFY** +```bash +virsh list --all | grep -i dc1 +virsh net-list --all | grep -i dc1 +virsh pool-list --all | grep -i dc1 +``` +Expect: no DC1 domains, networks, or pool remain. + +--- + +## Step 5 (Path A only) -- Confirm no drift outside the targeted scope + +```bash +cd opentofu +tofu plan +``` +Run WITHOUT `-target` this time -- this is the check HashiCorp's own +targeting warning (Step 3) calls for: confirm the rest of the configuration +(Office1, DC2 if wired, mesh links) shows no unexpected changes. Expect +either "No changes" for everything outside DC1, or exactly the changes you +independently expect for other in-flight work -- nothing attributable to +this teardown. + +**GATE:** untargeted plan shows zero unexpected drift outside the torn-down +site. + +--- + +## Path B -- full VR1 teardown (replaces Steps 3-5 above) + +Do Step 1 (backup) and Step 2 (MAAS-side cleanup, for EVERY site with +enrolled machines, not just one) first, then: + +```bash +cd opentofu +tofu plan -destroy -out=teardown-full.tfplan +``` +Review: expect a destroy for every resource this tree currently manages +(check against `opentofu/main.tf`'s actual uncommented module list at +teardown time -- it grows as more stages get executed, so what "everything" +means changes over the buildout, unlike Path A's fixed per-site scope). + +**GATE:** plan matches full current `main.tf` scope exactly. + +```bash +tofu destroy teardown-full.tfplan +``` + +**VERIFY** +```bash +virsh list --all +virsh net-list --all +virsh pool-list --all +``` +Expect: nothing VR1-related remains (VR0's own DC0/testcloud resources, +which OpenTofu does not manage, are untouched by any of this). + +Then do the equivalent of Step 5 (a final untargeted `tofu plan`) to confirm +the destroy converged state to fully empty, not partially. + +--- + +## Mesh-link teardown (either path, handle separately) + +The three D-100 mesh-link legs (`mesh_dc1_dc2`, `mesh_dc1_office1`, +`mesh_dc2_office1`) are shared between two sites each. Destroy a leg only +when BOTH its endpoints are permanently going away (e.g., a full VR1 +teardown, or abandoning DC2 AND Office1 together) -- never as part of a +single-site scoped teardown, and never as a reflex "clean up everything +DC1-adjacent" action. `modules/netem-link` (if ever wired against a given +leg) already has a correct `destroy`-time provisioner +(`provisioner "local-exec" { when = destroy; ... tc qdisc del ... }`, +confirmed by reading `opentofu/modules/netem-link/main.tf` directly) -- +destroying the corresponding `terraform_data.netem` resource cleans up the +real `tc qdisc` rule on the vcloud host automatically; no extra manual step +needed for that specific piece. + +--- + +## Rollback decision tree -- "a stage's `tofu apply` failed partway through, now what?" + +**Read this before reaching for `tofu destroy` at all.** OpenTofu is +designed for idempotent convergence -- in most failure modes, fixing the +root cause and re-running the SAME `tofu apply` is correct and safer than +tearing down and starting over, because it preserves everything that DID +apply successfully and only retries what didn't. + +1. **`tofu apply` errored before creating anything (e.g., `tofu init`/ + `validate`/`plan` failed, or `apply` failed on its very first resource).** + Nothing was created. Fix the root cause (bad variable, unreachable + provider, syntax error) and re-run from that stage's own `tofu plan` + step. No teardown needed. +2. **`tofu apply` created SOME resources, then failed on a later one.** + `tofu plan` (no `-destroy`) immediately to see OpenTofu's own view of + what's now live vs. what the config wants. Default to FIX-FORWARD: fix + whatever caused the failure (a bad node-VM memory size, an + unreachable MAAS endpoint, a real infra problem) and re-run + `tofu apply` for that same stage -- OpenTofu will create only what's + still missing, leaving successfully-created resources alone. Reserve a + teardown (Path A, scoped to that site) for when the partial state is + itself broken in a way fix-forward can't reach (e.g., a domain stuck in + a bad libvirt state that manual `virsh` intervention already tried and + failed to fix). +3. **You suspect `terraform.tfstate` itself is corrupted or lost** (this is + the scenario `opentofu/README.md`'s "State file handling" section + already flags as the real cost of no backend/backup). There is no clean + recovery path documented in this repo: either a manual `tofu import` per + still-live resource (tedious, must be done for every domain/network/pool + by hand) or accepting a full teardown-and-rebuild from Step 1 of the + affected stage. This is exactly why Step 1's pre-teardown backup habit + matters even outside a deliberate teardown -- back up + `terraform.tfstate` after every successful stage apply too, not only + before a teardown. +4. **A resource exists in reality but OpenTofu's state doesn't know about + it** (e.g., a `virt-install`-created VM from Stage 2's Option B manual + path, which OpenTofu never created and cannot destroy). This runbook's + `tofu destroy`/`plan -destroy` commands only ever touch what OpenTofu + itself manages -- anything created via a manual/Option-B path needs + manual teardown (`virsh destroy`/`undefine`, etc.) by the same operator + discipline that created it. Do not expect `tofu destroy` to find or + remove it. + +--- + +## Verification (of this runbook's own accuracy, not a live run) + +Every `tofu`/`terraform` flag cited above (`-destroy`, `-target`, the +targeting-risk warning) was checked against HashiCorp's own current CLI +documentation before being written here (fetched 2026-07-10), not asserted +from memory -- same discipline as `opentofu/README.md`'s other research +sections. The resource/module names and dependency shapes referenced +(`dc1_maas_vm_host`'s lack of a Terraform-expressible dependency on node-VM +domains, `netem-link`'s existing destroy-time provisioner) were confirmed by +reading the actual `.tf` files in this repo, not assumed. What is NOT +verified: this runbook's commands have never been run against a real `tofu` +binary or a real state file (same SCAFFOLD/UNVALIDATED status as the rest of +`opentofu/`) -- treat every command here as reviewed-but-unexercised until +`scripts/opentofu-validate.sh` and a real dry run confirm otherwise.