diff --git a/runbooks/dc-dc-teardown-rollback.md b/runbooks/dc-dc-teardown-rollback.md index 5218755..2247707 100644 --- a/runbooks/dc-dc-teardown-rollback.md +++ b/runbooks/dc-dc-teardown-rollback.md @@ -431,11 +431,20 @@ **This command is in the committed DENY list** (`.claude/settings.json`), so it is an OPERATOR-RUN command by policy. That is deliberate: it is the single most destructive juju verb. -- **API DOWN** (the case this section was written for) -> `juju kill-controller ` - Its own help: *"Forcibly destroy the specified controller. If the API server is - accessible, this command will attempt to destroy the controller model and all models and - their resources."* When the API is NOT accessible it goes straight to the provider, which - is exactly why it is the tool here. +- **API DOWN** -> **`juju kill-controller` DOES NOT WORK. MEASURED 2026-08-03; DO NOT WASTE + TEN MINUTES ON IT.** Its help implies a direct-to-provider fallback ("Timeout before + direct destruction", `-t` default `5m0s`), and the first version of this section said so. + **It does not reach that fallback.** Run against a controller whose API is refused, with + `--no-prompt`, it waited ~10 minutes and then exited 1: + ``` + Unable to open API: open connection timed out + ERROR cannot connect to model config API: unable to connect to API: + dial tcp :17070: connect: connection refused + ``` + It needs the MODEL CONFIG API to decide what to destroy, so an unreachable API defeats it + before the timeout matters. **Nothing was destroyed and nothing cascaded** -- the MAAS + census was identical before and after -- so it is safe, just useless. + **USE THE MANUAL PAIR INSTEAD (C.2b).** **>>> POLICY GAP, LOGGED 2026-08-03: `juju kill-controller` is NOT in the deny list while `juju destroy-controller` IS.** `kill-controller` is the STRONGER command. The committed policy gates the weaker one and lets the stronger through. Treat `kill-controller` as @@ -443,6 +452,38 @@ SEC-030 for why the ask/deny rules are not the real gate here -- the presentation discipline is. +### C.2b -- API DOWN: the manual pair (juju client, then MAAS) + +When `kill-controller` cannot work, the controller is removed in two independent halves. +This is the D-061 coordination principle applied to the controller: **clean up juju's view +first, then MAAS's**, never the reverse. + +**(1) Drop the client's registration -- SAFE, no cloud resource touched.** +```bash +juju controllers # confirm the name +juju unregister --no-prompt # edits local juju client config ONLY +``` +This does not release the VM. After it, MAAS still shows the controller VM `Deployed` and +OWNED by the juju service user -- that is expected, not a leak. + +**(2) Release the controller VM in MAAS [MUTATION, gated].** Identify it by TAG, never by +hostname -- MAAS auto-generated hostnames do not match the ruled names, and the tag is the +stable key: +```bash +maas machines read \ + | jq -r --arg t "juju-controller-$DC" \ + '.[] | select(.tag_names[]? == $t) | "\(.hostname) \(.system_id) \(.status_name) \(.owner // "None")"' +maas machine release +``` +**GATE before releasing:** the query returns EXACTLY ONE machine, it is the controller VM, +and it is NOT one of the nine role nodes (cross-check the role nodes are `Ready / +owner=None` and stay that way). **After:** it reads `Ready / owner=None`, the MAAS machine +COUNT is unchanged, and the tag is still on it -- the tag is what C.6/C.7 bootstrap against, +so a release that lost the tag means STOP. + +`enable_disk_erasing_on_release=false` on this region, so release is prompt and the disk is +re-imaged at the next bootstrap rather than wiped now. + ### C.3 -- Verify the teardown released, and did not cascade ```bash