diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 93a0982..1ffe7d6 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1982,7 +1982,41 @@ **MEASURED CORRECTION a future session would otherwise guess wrong: on the dc0 REGION VM the MAAS CLI profile is `dc0`, NOT `vr1-dc0-region`** -- the tool's own error enumerated the valid choices. Profile names are PER HOST; assert, never assume. - **COMMAND AS VERIFIED:** `juju destroy-model vr1-dc0 --force --no-prompt`. Deliberately NOT + **>>> CORRECTED BY PRIOR-RUN EVIDENCE THE REPO ALREADY HELD, ON OPERATOR DIRECTION. THE + COMMAND CHANGED. <<<** I wrote that the juju release path was "the one thing I can't prove + in advance". **It was already proven and logged** -- this same document, "MODEL TEARDOWN + 2026-07-31". Operator: *"You have that information from previous teardowns ... find the + previously logged information in the repo. Log the commands and steps for future + reference."* Two corrections: **(1) `--no-wait` BELONGS IN THE COMMAND and I had excluded + it** -- I reasoned from `juju help` that it was reckless, but MEASURED 2026-07-31 the plain + `destroy-model` STALLED TERMINALLY (`attempt 30 ... model not empty, found 26 machines, 37 + applications`, flat ~19 min, app set byte-identical) because ALL 26 agents were `stopped` + so no teardown hook could execute; `--force --no-wait` cleared it (18 -> 5 -> 2, then + `Model destroyed.`). On a model whose agents are stopped there is nothing to wait FOR -- + the help text talks you out of the flag that works. **(2) THE RELEASE PATH IS PROVEN:** + after that destroy all NINE role nodes read `Ready / owner=None`, zero stranded, and NO + `maas machine release` was needed or run. **Expected post-state is 9 `Ready` + 1 + `Deployed`** -- the controller VM stays `Deployed` in the `controller` model, so a session + expecting 10 Ready raises a false alarm. + **>>> AND THE MODEL-CONFIG TRAP IS LIVE FOR THIS REBUILD. <<<** `destroy-model` TAKES THE + MODEL CONFIG WITH IT -- on 07-31 that silently removed Step 3.5's `apt-mirror` and the + spaces work, and "NOTHING in the repo would have caught it". **MEASURED 2026-08-02: this + controller's model-defaults carry `apt-mirror = http://10.12.8.4/ubuntu` AND NOTHING + ELSE.** Post-convergence that value is WRONG, and the three settings this deploy needs + (`apt-http-proxy`, `snap-http-proxy`, `snap-https-proxy`) were set at MODEL level and will + be destroyed with it. **Defaults are inherited by NEW models only, so they must be + corrected BEFORE `add-model`** -- a rebuild that skips this comes back pointing at the + retired mirror with no proxies. **The VR0 pod warning does NOT apply:** + `phase-00-teardown-maas-reset.md` records that VR0's virsh-POD MAAS DECOMPOSES + pod-composed machines on destroy-model; VR1 uses per-machine `power_type=virsh`, not pods + (D-103/D-123), so there is nothing to decompose -- confirmed by both VR1 teardowns. + **ALL OF IT IS NOW A REUSABLE PROCEDURE:** `runbooks/dc-dc-teardown-rollback.md` gained + **"Path M -- JUJU MODEL teardown"** (M.0 scope table, M.1 blast-radius pre-flight, M.2 the + command and why `--no-wait` stays, M.3 the release path, M.4 the model-config trap with a + per-DC checklist, M.5 post-checks incl. the cascade signature, M.6 rebuild sequence). The + runbook previously documented only SUBSTRATE teardown -- the model layer had no procedure + at all, which is why this kept being re-derived. + **COMMAND AS VERIFIED (superseded below):** `juju destroy-model vr1-dc0 --force --no-prompt`. Deliberately NOT passed: `--destroy-storage` / `--release-storage` (no-ops), `--no-wait` (help says it rushes steps), `--timeout` (help warns `--force` + `--timeout` abandons clean shutdown). OUT OF SCOPE and untouched: the controller model and its VM, the MAAS region and database, diff --git a/docs/audit/destroy-model-blast-radius-20260802.txt b/docs/audit/destroy-model-blast-radius-20260802.txt index fe76b68..38ff422 100644 --- a/docs/audit/destroy-model-blast-radius-20260802.txt +++ b/docs/audit/destroy-model-blast-radius-20260802.txt @@ -79,3 +79,56 @@ (b) MAAS machine COUNT is unchanged and the 9 role nodes read `Ready` -- a DROP in count is the cascade signature from the 2026-07-21 incident and means STOP (c) `dc-cache-proxy.sh check dc0` and `dc-snap-proxy.sh check dc0` still PASS + +-------------------------------------------------------------------------------------- +5. CORRECTED BY PRIOR-RUN EVIDENCE THE REPO ALREADY HELD (operator direction) +-------------------------------------------------------------------------------------- +Operator: "You have that information from previous teardowns for the juju release path +as we have run those commands many times ... find the previously logged information in +the repo. Log the commands and steps for future reference." + +I said in section 4 that the release path was "the one thing I can't prove in advance". +THAT WAS WRONG -- it was already proven and logged, in docs/CURRENT-STATE.md under +"MODEL TEARDOWN 2026-07-31". Two corrections follow from it: + +>>> CORRECTION 1 -- `--no-wait` BELONGS IN THE COMMAND. I had EXCLUDED it. <<< +I reasoned from `juju help` ("Rush through model destruction without waiting for each +individual step") that it was reckless. MEASURED 2026-07-31: the plain `destroy-model` +STALLED terminally -- "attempt 30 to destroy model failed (will retry): model not empty, +found 26 machines, 37 applications", flat ~19 minutes, application set byte-identical. +ALL 26 agents were `stopped`, so NO teardown hook could execute; units already in `error` +could never run their stop hooks. `--force --no-wait` cleared it (18 -> 5 -> 2, then +"Model destroyed."). On a model whose agents are stopped there is nothing to wait FOR. +The help text talks you out of the flag that actually works. + +>>> CORRECTION 2 -- THE RELEASE PATH IS PROVEN, NOT UNKNOWN. <<< +After the 07-31 forced destroy: ALL NINE role nodes read `Ready / owner=None`, ZERO +stranded, and NO `maas machine release` was needed or run. enable_disk_erasing_on_release +=false, so no erase cycle. EXPECTED POST-STATE IS 9 Ready + 1 Deployed -- the controller +VM (`subtle-grouse`) stays Deployed because it lives in the `controller` model. A session +expecting 10 Ready raises a false alarm. + +>>> ALSO RECOVERED: THE MODEL-CONFIG TRAP, WHICH IS LIVE FOR THIS REBUILD. <<< +`destroy-model` takes the model config with it. On 07-31 that silently removed Step 3.5's +`apt-mirror` and the spaces work, and "NOTHING in the repo would have caught it". +MEASURED 2026-08-02: this controller's model-defaults carry `apt-mirror = +http://10.12.8.4/ubuntu` AND NOTHING ELSE. Post-convergence that value is WRONG, and the +three settings this deploy needs -- apt-http-proxy, snap-http-proxy, snap-https-proxy -- +were set at MODEL level and will be destroyed. Defaults are inherited by NEW models only, +so they must be corrected BEFORE `add-model`. + +>>> AND THE VR0 POD WARNING DOES NOT APPLY. <<< +runbooks/phase-00-teardown-maas-reset.md:24-25 records that on VR0's virsh-POD MAAS, +"destroy-model DECOMPOSES the pod-composed machines (observed 3x)". VR1 uses per-machine +power_type=virsh, NOT pods (D-103/D-123), so there is nothing to decompose -- confirmed by +both VR1 teardowns. Do not import the VR0 procedure into a VR1 teardown. + +ALL OF THE ABOVE IS NOW A REUSABLE PROCEDURE: runbooks/dc-dc-teardown-rollback.md, +"Path M -- JUJU MODEL teardown", M.0 through M.6. + +-------------------------------------------------------------------------------------- +6. FINAL COMMAND (supersedes section 4) +-------------------------------------------------------------------------------------- + juju destroy-model vr1-dc0 --force --no-wait --no-prompt + +BLOCKED BY THE PERMISSION LAYER on 2026-08-02 and NOT RUN. No workaround attempted. diff --git a/runbooks/dc-dc-teardown-rollback.md b/runbooks/dc-dc-teardown-rollback.md index feef78a..63e8e74 100644 --- a/runbooks/dc-dc-teardown-rollback.md +++ b/runbooks/dc-dc-teardown-rollback.md @@ -185,6 +185,163 @@ --- +## Path M -- JUJU MODEL teardown (the innermost layer; NO substrate is touched) + +**ADDED 2026-08-02 on operator direction: "this has bit in the past and will be a +reusable and needed set for the future."** Everything here is MEASURED from real runs +on this cloud -- the 2026-07-31 VR1 dc0 teardown (`docs/CURRENT-STATE.md`, "MODEL +TEARDOWN 2026-07-31") and the 2026-08-02 rebuild. It is NOT derived from `juju help`; +where the two disagree, the help text lost, and that is recorded below. + +**USE THIS WHEN** you want to re-deploy the bundle into a clean model: a failed or +half-installed deploy, a config defect that reached the charms, a bundle change you want +applied from scratch. **It is NOT a substrate teardown** -- Paths A and B above are a +different, much larger action. If the fix is a deploy input, you almost certainly want +Path M and nothing else. + +### M.0 -- What Path M does and does NOT touch + +| Touched | NOT touched | +|---|---| +| the `` model: every unit, container, and machine assignment | the `controller` model and the D-104 controller VM | +| model-level config (see the M.4 trap) | the MAAS region, its database, and the machine RECORDS | +| | the rack host, the OPNsense edge, both proxies, the tofu substrate, the other DC | + +`juju destroy-model`'s own summary scopes it: *"Terminate all machines/containers and +resources for a **non-controller model**."* The controller model is out of reach by +design. + +### M.1 -- Pre-flight: prove the blast radius, every time + +```bash +juju storage -m "$DC_MODEL" # storage instances -- decides the storage flag +juju offers -m "$DC_MODEL" # cross-model offers -- non-empty = another model consumes this one +juju models # confirm which models exist and their machine counts +``` +- **A storage flag is required ONLY if `juju storage` is non-empty.** MEASURED 2026-08-02 + on `vr1-dc0`: *"No storage to display."* `ceph-osd`'s `osd-devices` are RAW BLOCK + DEVICES set by charm config, NOT juju storage instances -- so `--destroy-storage` + destroys nothing here and must not be passed as a reflex. MAAS re-partitions those + disks at re-image. +- **Non-empty `juju offers` means STOP** and work out what consumes the endpoint first. + +### M.2 -- The command [MUTATION: gated] + +```bash +juju destroy-model "$DC_MODEL" --force --no-wait --no-prompt +``` + +**`--no-wait` IS PART OF THE PROVEN PATH. DO NOT DROP IT ON A FAILED DEPLOY.** MEASURED +2026-07-31: a plain `destroy-model` **STALLED and would not self-resolve** -- `attempt 30 +to destroy model failed (will retry): model not empty, found 26 machines, 37 +applications`, flat for ~19 minutes with the application set BYTE-IDENTICAL across a +12-minute diff. **Mechanism, which makes the stall terminal rather than slow: all 26 +machine/container agents were `stopped`, so no teardown hook could execute at all** -- +the destroy worker asks and nothing answers. Units already in `error` (11 haclusters at +`hook failed: "stop"`, four at `install`, five at `identity-service-relation-departed`) +could never run their stop hooks, so it was never going to drain. **`--force --no-wait` +cleared it: 18 -> 5 -> 2 machines, then `Model destroyed.`** +Reading `juju help destroy-model` alone would talk you OUT of `--no-wait` ("Rush through +model destruction without waiting for each individual step to complete"). On a model whose +agents are already stopped there is nothing to wait FOR. Trust the measurement. + +**`--timeout` is the one to leave off:** the help warns that `--force` WITH `--timeout` +continues "without consideration or respect for clean shutdown or resource cleanup". +`--no-prompt` is required non-interactively (the help's Details prose still says `-y`; +that is STALE -- the flag list is authoritative). + +### M.3 -- THE RELEASE PATH: what happens to the MAAS machines + +**MEASURED, BOTH RUNS -- juju releases the machines and MAAS returns them to `Ready`. +NO `maas machine release` IS NEEDED AND NONE WAS RUN.** After the 2026-07-31 forced +destroy, read back: **all NINE role nodes `Ready / owner=None`, zero stranded.** + +- **`enable_disk_erasing_on_release` is `false`** on this region (verify per DC: + `maas maas get-config name=enable_disk_erasing_on_release`), so release does + NOT trigger an erase cycle -- nodes return to `Ready` in minutes and are re-imaged by + the next deploy. During the 07-31 run the six nodes juju did release reached + `Ready / owner None` in minutes; **MAAS was never the bottleneck**, juju simply had not + yet issued a release for the other three. +- **EXPECT the controller VM to stay `Deployed`.** `subtle-grouse` remains `Deployed` + under `juju-vr1-dc0` because it belongs to the `controller` model, not the destroyed + one. **The correct post-state is 9 `Ready` + 1 `Deployed`, NOT 10 `Ready`** -- a + session expecting all ten to release will raise a false alarm. +- **THE CASCADE SIGNATURE TO STOP ON: a DROP in the MAAS machine COUNT.** Records are + released, never deleted. This repo has a real cascade (2026-07-21, a MAAS pod delete + taking its linked machine records), which is why the count is checked and not assumed. +- **The VR0 pod warning does NOT apply here.** `runbooks/phase-00-teardown-maas-reset.md` + records that on VR0's virsh-POD MAAS, `destroy-model` DECOMPOSES pod-composed machines + (observed 3x) -- there, use the D-061 pair instead. **VR1 uses per-machine + `power_type=virsh` and NOT pods (D-103/D-123 amendments), so there is nothing to + decompose**, which both VR1 teardowns confirm. Do not import the VR0 procedure. + +### M.4 -- >>> THE TRAP: `destroy-model` TAKES THE MODEL CONFIG WITH IT <<< + +**This bit on 2026-07-31 and it will bite again.** Destroying the model destroys its +model-config. Step 3.5's `apt-mirror` and the spaces work vanished with it, **and nothing +in the repo would have caught it** -- the next deploy fails on package fetches and reads +like an artifact-service fault rather than a missing model setting. + +**Anything the deploy needs at the MODEL level must be re-established BEFORE the next +`add-model`, or set as a CONTROLLER MODEL-DEFAULT so it survives.** + +```bash +juju model-defaults # what a NEW model will inherit +juju model-defaults = # set (per-controller, so per-DC by D-104) +``` +**Model-defaults are inherited by NEW models ONLY -- never applied retroactively**, so +they must be right BEFORE `add-model`, not after. + +**MEASURED 2026-08-02, and it is a live trap for the very next rebuild:** this +controller's defaults carry **`apt-mirror = http://10.12.8.4/ubuntu` and nothing else**. +After the D-135 amendment (b) convergence, that value is WRONG -- dc0 now consumes the +apt caching proxy -- and the three settings the deploy actually needs +(`apt-http-proxy`, `snap-http-proxy`, `snap-https-proxy`) are **absent from the defaults +and were set at MODEL level**, so the destroy takes them. A rebuild that skips this step +comes back pointing at the retired mirror with no proxies at all. + +Per-DC checklist before `add-model`, each read back on CONTENT: + +```bash +juju model-defaults --reset apt-mirror # convergence: proxy, not mirror +juju model-defaults apt-http-proxy=http://:3142 # dc-cache-proxy.sh LISTEN +juju model-defaults snap-http-proxy=http://:3129 # dc-snap-proxy.sh LISTEN +juju model-defaults snap-https-proxy=http://:3129 +# LEAVE snap-store-proxy / -url / -assertions EMPTY (Enterprise Store, different product) +``` +Take `` from that DC's own checker output, never from memory (dc0 `10.12.8.4`, +dc1 `10.12.68.4`). **OWED, and still not built: a site-keyed `dc-model-defaults.sh` with +a `check`, so these are VERIFIED at every standup rather than remembered.** + +### M.5 -- Post-checks, before rebuilding anything + +```bash +juju models # (a) only `controller` remains +maas machines read | ... # (b) COUNT unchanged; 9 role nodes Ready, controller VM Deployed +bash scripts/dc-cache-proxy.sh check # (c) artifact + snap services survived +bash scripts/dc-snap-proxy.sh check +``` +**(b) is the one that catches a cascade** -- a drop in count means STOP, do not rebuild. + +**INSTRUMENT WARNING, learned by getting it wrong (2026-07-31): during a teardown, +`juju status -m ` is the instrument; `juju models` COUNTS ARE NOT.** The summary +columns ZERO OUT while the model is `destroying` -- they read "0 machines / 0 units" +while `juju status` simultaneously showed 26 machines and 37 applications. The real tell +(three control nodes stuck `Deployed`) was visible and was explained away as "more +containers to work through". Do not read progress from `juju models`. + +### M.6 -- Rebuild sequence after a Path M teardown + +1. M.5 post-checks all green. +2. **M.4 model-defaults corrected** -- this is the step that gets skipped. +3. `juju add-model ` (inherits the corrected defaults), then re-apply the spaces work. +4. Re-stage the deploy inputs to the DC's `~/repo-stage` and **sha256-verify against repo + HEAD** before trusting them (the racks carry no git clone). +5. `phase-4` Step 4.2 dry-run gate, then 4.3 deploy, then **4.4 immediately** -- 4.4 is + the only gate that reads the DEPLOYED config rather than an input to it. + +--- + ## Step 1 -- Back up the state file before ANY destroy [do this every time] A `destroy` mutates state exactly like an `apply` does -- if it's