diff --git a/runbooks/dc-dc-teardown-rollback.md b/runbooks/dc-dc-teardown-rollback.md index c2add67..5218755 100644 --- a/runbooks/dc-dc-teardown-rollback.md +++ b/runbooks/dc-dc-teardown-rollback.md @@ -391,6 +391,171 @@ --- +## Path C -- JUJU CONTROLLER teardown and rebuild (per DC) + +**ADDED 2026-08-03.** Use when the DC's Juju controller itself is unusable -- not the +model. **The trigger that produced this section:** a `--force` model destroy orphaned a +model document without its status document, which crash-looped `modelcache`, which the API +server depends on, so port 17070 went to `connection refused` (Path M.2's warning block). +Also valid for a corrupted controller state DB, a lost controller VM, or a controller that +cannot be upgraded. + +**PREFER THIS OVER JUJU STATE-DB SURGERY.** A controller whose only model has been +destroyed holds nothing worth preserving. Mongo surgery on juju's state is unsupported and +riskier than a rebuild along a path this repo has already exercised. + +**NOTHING BELOW TOUCHES:** the MAAS region or its database, the machine records, the rack, +the edge, the artifact/snap proxies, the tofu substrate, or the other DC. The nine role +nodes stay `Ready` throughout and are re-imaged only when the bundle is redeployed. + +### C.1 -- Census FIRST: write down what must survive + +```bash +maas machines read | jq -r '.[] | "\(.hostname) \(.status_name) \(.owner // "None")"' +bash scripts/dc-cache-proxy.sh check # via the rack; see the header for transport +bash scripts/dc-snap-proxy.sh check +``` +Record the machine COUNT. **A drop in that count at any later step is the cascade signature +(2026-07-21 incident) and means STOP.** Expected steady state through the whole of Path C: +nine role nodes `Ready / owner=None`, plus the controller VM (`Deployed` before C.2, +`Ready` after). + +### C.2 -- Tear the controller down [MUTATION, gated] + +**Pick by whether the API answers.** Test it, do not assume: +```bash +timeout 8 bash -c "/17070" && echo API-UP || echo API-DOWN +``` + +- **API UP** -> `juju destroy-controller --destroy-all-models --no-prompt` + **This command is in the committed DENY list** (`.claude/settings.json`), so it is an + OPERATOR-RUN command by policy. That is deliberate: it is the single most destructive + juju verb. +- **API DOWN** (the case this section was written for) -> `juju kill-controller ` + Its own help: *"Forcibly destroy the specified controller. If the API server is + accessible, this command will attempt to destroy the controller model and all models and + their resources."* When the API is NOT accessible it goes straight to the provider, which + is exactly why it is the tool here. + **>>> POLICY GAP, LOGGED 2026-08-03: `juju kill-controller` is NOT in the deny list while + `juju destroy-controller` IS.** `kill-controller` is the STRONGER command. The committed + policy gates the weaker one and lets the stronger through. Treat `kill-controller` as + operator-gated regardless of what the rule engine does, and see `docs/security-ledger.md` + SEC-030 for why the ask/deny rules are not the real gate here -- the presentation + discipline is. + +### C.3 -- Verify the teardown released, and did not cascade + +```bash +maas machines read | jq -r '.[] | "\(.hostname) \(.status_name) \(.owner // "None")"' +``` +**GATE:** machine COUNT unchanged from C.1; the controller VM now reads `Ready / owner=None` +alongside the nine role nodes. Records are RELEASED, never deleted -- a count drop is the +cascade signature. Expect NO `maas machine release` to be necessary; juju's own release path +does it (measured, both prior teardowns). + +### C.4 -- Clean the client side + +```bash +juju controllers # is the dead controller still registered? +juju unregister --no-prompt # client-side ONLY; removes no cloud resource +``` +Skip if `kill-controller` already removed it. `unregister` is safe: it edits the local juju +client config and nothing else. + +### C.5 -- Rebuild: the credential gate, REGION-SCOPED + +**Do not re-mint blindly.** Follow `runbooks/dc-dc-phase4-juju-bundle-per-dc.md` **Step +2.0** verbatim -- it carries the three-outcome gate and DOCFIX-206's correction that the +check must name the REGION explicitly. The trap it exists for: under D-132 q1 each DC has +its own region with its own database, so `juju credentials --client` can list a credential +whose MAAS user lives in a DIFFERENT region. Prove the region with +`scripts/maas-profile-assert.sh` (rack-controller identity -- a machine count is not proof) +before trusting any "already exists". + +**A controller rebuild does NOT invalidate the MAAS credential** -- the credential belongs +to the CLOUD definition in the juju CLIENT, not to the controller. If it authenticated +before, it still does. Prove it rather than assume, with Step 2.0's scoped +`maas login ... / users read / machines read / rack-controllers read / +maas logout` sequence: *"A credential that EXISTS is not a credential that WORKS."* + +Also re-read the cloud ENDPOINT (`juju show-cloud vr1-maas --client`) -- an endpoint still +pointing at the Office1 region is what killed bootstrap attempt 3 on 2026-07-30. + +### C.6 -- Rebuild: the controller-tag gate + +```bash +bash scripts/maas-role-tags.sh check "$DC" +maas "$MAAS_PROFILE" machines read \ + | jq -r --arg t "juju-controller-$DC" '.[] | select(.tag_names[]? == $t) | "\(.hostname) \(.status_name)"' +``` +**GATE (phase-4 Step 2):** the tag exists AND returns EXACTLY ONE machine in `Ready`. Zero +-> bootstrap finds no candidate. More than one -> the target is ambiguous, STOP. A cached +`error` power state is not a fail -- refresh with `machine query-power-state` and re-read. + +### C.7 -- Rebuild: bootstrap [MUTATION, gated] + +```bash +juju bootstrap \ + --bootstrap-constraints "tags=juju-controller-$DC" \ + --constraints "tags=juju-controller-$DC" \ + --bootstrap-base ubuntu@22.04 +``` +**BOTH constraint flags, per the 2026-07-30 GA-R5 ruling** (operator: *"Use both flags"*) -- +`--bootstrap-constraints` is what targets the machine; `--constraints` sets the model +default and honours the runbook's literal text. **`--bootstrap-base ubuntu@22.04` is +CONFIRMED against a live deploy** (the 07-30 MAAS event chain ended `Image Deployed -- +deployed ubuntu/jammy/amd64/generic`). + +**There is NO `--dry-run` for bootstrap** (checked) -- it is one-shot. That is why C.5 and +C.6 are gates and not formalities. + +### C.8 -- >>> IMMEDIATELY AFTER BOOTSTRAP: MODEL-DEFAULTS. A NEW CONTROLLER HAS NONE. <<< + +**This is the step that gets skipped, and a fresh controller makes it WORSE than Path M.4:** +model-defaults live ON THE CONTROLLER, so a rebuild loses every one of them. Everything +Path M.4 warns about applies here with nothing carried over. + +```bash +juju model-defaults # expect: all empty +juju model-defaults apt-http-proxy=http://:3142 # dc-cache-proxy.sh LISTEN +juju model-defaults snap-http-proxy=http://:3129 # dc-snap-proxy.sh LISTEN +juju model-defaults snap-https-proxy=http://:3129 +# LEAVE apt-mirror UNSET -- D-135 amendment (b) converged dc0 onto the proxy +# LEAVE snap-store-proxy / -url / -assertions EMPTY (Enterprise Store, different product) +juju model-defaults # READ BACK on CONTENT +``` +Take `` from that DC's own checker output, never memory (dc0 `10.12.8.4`, dc1 +`10.12.68.4`). **Defaults are inherited by NEW models only -- set them BEFORE `add-model`.** + +### C.9 -- Rebuild: model, spaces, artifact source + +`runbooks/dc-dc-phase4-juju-bundle-per-dc.md` **Step 3.5** owns this: `juju add-model +"$DC"`, the spaces gate (`scripts/juju-spaces-check.sh`), and this DC's artifact source. +Then **Step 3** (`preflight.sh` for this DC) and **Step 3.9** (`dc-egress-check.sh`). + +### C.10 -- Rebuild: re-stage and redeploy + +1. Re-stage the deploy inputs to the DC's `~/repo-stage` and **sha256-verify every one + against repo HEAD** -- the racks carry no git clone, and a stale staged overlay has bitten + this project twice. +2. phase-4 **Step 4.2** dry-run gate (keep `--debug`; a bare `--dry-run` prints no + constraints), then **4.3** deploy, then **4.4 immediately** -- 4.4 is the only gate that + reads the DEPLOYED config rather than an input to it. + +### C.11 -- What a Path C rebuild does NOT restore + +- **Model-defaults** -- C.8, and nothing else will remind you. +- **Controller backup** -- phase-4 Step 11 is a separate obligation against the NEW + controller. +- **The client's known-hosts / juju SSH key** for the old controller VM: `juju` mints a new + one at bootstrap. If a stale host key blocks you, that is the cause. +- **OWED, still not built:** a site-keyed `dc-model-defaults.sh` with a `check`, so C.8 is + VERIFIED at every standup instead of remembered. Until it exists, C.8 is the whole control. + +--- + +--- + ## Step 1 -- Back up the state file before ANY destroy [do this every time] A `destroy` mutates state exactly like an `apply` does -- if it's