diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 1ffe7d6..28a232f 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1982,6 +1982,43 @@ **MEASURED CORRECTION a future session would otherwise guess wrong: on the dc0 REGION VM the MAAS CLI profile is `dc0`, NOT `vr1-dc0-region`** -- the tool's own error enumerated the valid choices. Profile names are PER HOST; assert, never assume. + **>>> THE MODEL TEARDOWN EXECUTED 2026-08-03, SUCCEEDED, AND THEN TOOK THE CONTROLLER + DOWN. THE dc0 JUJU CONTROLLER IS CURRENTLY UNUSABLE. <<<** + **`juju destroy-model vr1-dc0 --force --no-wait --no-prompt` returned `Model destroyed.` + EXIT 0**, with a clean progressive drain (36 machines/56 apps -> 26 -> 20 -> 16 -> 9 -> 0, + no stall) and a CORRECT MAAS release: **census 10 machines, count UNCHANGED (no cascade), + nine role nodes `Ready / owner=None`, `subtle-grouse` still `Deployed` in the `controller` + model** exactly as predicted. Both proxies re-checked PASS. **M.4 executed:** controller + model-defaults corrected -- `apt-mirror` RESET, `apt-http-proxy=http://10.12.8.4:3142`, + `snap-http-proxy`/`snap-https-proxy=http://10.12.8.4:3129`, all read back on content. + **THEN: `juju add-model vr1-dc0` REFUSED -- `model "vr1-dc0" for admin already exists`.** + The model sat at `life: dead` and was never reaped. Root cause, from the controller log: + the destroy left the model document ALIVE while its STATUS document was GONE. Two workers + crash-loop on it: `undertaker` -> `cannot destroy cloud resources: cannot set status: + model not found` (growing backoff, deterministic, never self-resolves), and -- the + consequential one -- **`modelcache` -> `failed to initialise backing for models:001f6745-...: + status doc "001f6745-...:e" not found, getting new watcher`, 732 iterations logged in ~12 + minutes.** The API server DEPENDS on modelcache, so **port 17070 is `connection refused` + and the controller is DOWN.** The VM is running and pings 0% loss and + `systemctl is-active jujud-machine-0` reads `active` -- only the log and the refused port + reveal it. **A `jujud-machine-0` restart did NOT fix it** (operator-approved, executed + 01:08:59Z; the worker re-entered the same loop). Known upstream class, LP #1737487 "dead + model did not get removed, and cannot be removed"; upstream is explicit that `--force` is + a LAST RESORT because it can leave state juju cannot reconcile. + **OWNED -- TWO OF MY DECISIONS ARE IMPLICATED.** (i) I carried `--force --no-wait` over + from the 07-31 stall, where agents were `stopped` and force genuinely was the only way + through. **This run's agents were ALIVE and the drain was progressive -- force was almost + certainly NOT needed**, and it is the documented cause of exactly this inconsistency. + (ii) When the model would not reap, **I re-issued `destroy-model` against a model already + at `life: dead`** -- that moved it BACK to `dying` and re-armed the undertaker loop. It + did not help and it changed state. Both are now written into Path M as explicit + prohibitions. + **PATH M IS AMENDED** (`runbooks/dc-dc-teardown-rollback.md` M.2): try WITHOUT `--force` + when agents are alive; reserve `--force --no-wait` for the measured stall condition; never + re-issue destroy against a `dead` model; and if the controller is already down on this + symptom, REBUILD THE CONTROLLER rather than attempt juju state-DB surgery. + **NOT DEGRADING WHILE THIS IS DECIDED:** nine nodes `Ready`, MAAS region healthy, both + proxies PASS, model-defaults now controller-level. Nothing is lost by pausing here. **>>> CORRECTED BY PRIOR-RUN EVIDENCE THE REPO ALREADY HELD, ON OPERATOR DIRECTION. THE COMMAND CHANGED. <<<** I wrote that the juju release path was "the one thing I can't prove in advance". **It was already proven and logged** -- this same document, "MODEL TEARDOWN diff --git a/runbooks/dc-dc-teardown-rollback.md b/runbooks/dc-dc-teardown-rollback.md index 63e8e74..311c83d 100644 --- a/runbooks/dc-dc-teardown-rollback.md +++ b/runbooks/dc-dc-teardown-rollback.md @@ -245,6 +245,43 @@ model destruction without waiting for each individual step to complete"). On a model whose agents are already stopped there is nothing to wait FOR. Trust the measurement. +**>>> BUT `--force` CAN ORPHAN THE MODEL AND TAKE THE WHOLE CONTROLLER DOWN. MEASURED +2026-08-03. READ THIS BEFORE USING IT. <<<** The 2026-08-02 dc0 teardown reported +`Model destroyed.` EXIT 0 with a clean progressive drain and correct MAAS release -- and +still left the model in an INCONSISTENT state: the model document survived while its +STATUS document did not. Consequences, in order: + +1. `juju models` shows the model at `life: dead`; **`juju add-model ` refuses + with `model "" for admin already exists`.** +2. The controller's `undertaker` worker crash-loops on + `cannot destroy cloud resources: cannot set status: model not found`, on a growing + backoff. **It never self-resolves** -- the error is deterministic. +3. **Worst: the `modelcache` worker crash-loops on `failed to initialise backing for + models:: status doc ":e" not found, getting new watcher`** -- hundreds of + iterations per minute (732 logged in ~12 min). The API server DEPENDS on modelcache, + so **port 17070 goes to `connection refused` and the CONTROLLER IS DOWN.** The VM is + up and pings; `systemctl is-active jujud-machine-0` reads `active`. Only the log and + the refused port tell you. + +**A `jujud` restart does NOT fix it** (tried; the worker re-enters the same loop). This is +the LP #1737487 class -- "dead model did not get removed, and cannot be removed". Upstream +is explicit that `--force` is a LAST RESORT precisely because it can leave state juju +cannot reconcile. + +**MITIGATION, in order of preference:** +- **Try WITHOUT `--force` first** when the model's agents are still ALIVE. The 08-02 run + drained progressively (36 -> 26 -> 20 -> 16 -> 9 -> 0) with hooks executing, i.e. force + was not actually needed; it was carried over from the 07-31 stall where agents were + `stopped` and force genuinely was the only way through. +- **Reserve `--force --no-wait` for the measured stall condition** -- agents `stopped`, + destroy flat for many minutes, units unable to run teardown hooks. +- **DO NOT re-issue `destroy-model` against a model already at `life: dead`.** Measured + 2026-08-03: it moved the model BACK to `dying` and re-armed the undertaker loop. It did + not help and it changed state. +- **If the controller is already down on this symptom, REBUILD THE CONTROLLER** rather + than attempting juju state-DB surgery. A controller hosting only a destroyed model has + nothing worth preserving, and `phase-4` Step 2's bootstrap is a tested path. + **`--timeout` is the one to leave off:** the help warns that `--force` WITH `--timeout` continues "without consideration or respect for clean shutdown or resource cleanup". `--no-prompt` is required non-interactively (the help's Details prose still says `-y`;