diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 7d445be..84d39a0 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2013,6 +2013,25 @@ at `life: dead`** -- that moved it BACK to `dying` and re-armed the undertaker loop. It did not help and it changed state. Both are now written into Path M as explicit prohibitions. + **>>> DEPLOY STALL ROOT-CAUSED AND CLEARED 2026-08-03: cloud-init `apt-get update` WEDGED ON + THREE UNITS FOR ~13h. <<<** (Operator approved the fix: *"Kill the stuck apt-get update on + the three units"*.) Symptom: machines all MAAS-`Deployed` but juju agents `pending` on + machine 5 (storage-02) and containers `0/lxd/10`+`2/lxd/2` (mysql-innodb-cluster/0 and /2), + which stalled the mysql cluster and the whole pre-vault-init settle. MEASURED cause: on each, + cloud-init `modules --mode=final` ran `apt-get update` at 04:48 and BLOCKED on `archive.ubuntu.com` + (16s CPU over 13h = stuck on network I/O, not working); cloud-init installs jujud AFTER apt, + so the agent was never installed (`inactive`, no log). The other 12 units drew a healthy path; + these three hit the flaky-archive-backend class (the 08-02 F2 finding) and a blocked process + never retries. Both apt proxies (`10.12.8.6:8000` MAAS, `10.12.8.4:3142` DC cacher) tested 200 + at fix time -- the network had recovered; only the wedged processes held. FIX: `pkill` the + stuck `apt-get update` on all three (direct-SSH with the juju key for machine 5; via the LXD + host + `lxc exec` for the containers), clear stale apt/dpkg locks. RESULT: cloud-init resumed + past the update on all three, jujud installed, **all three agents came up `started`** and + mysql/0+/2 went to `maintenance (install)` -- the cluster can now form. + **TWO LATENT ITEMS LOGGED, not fixed (hard rule 1):** (a) the nodes carry DUAL apt proxies + (MAAS's `10.12.8.6:8000` + the DC `10.12.8.4:3142`) -- a config smell; (b) OS-level cloud-init + apt is not pinned to the resilient DC proxy, which is why a flaky upstream backend can wedge a + node at boot at all. Both are DC-standup hardening candidates. **>>> THE v4 VIP REVERT IS EXECUTED AND KEYSTONE'S `Invalid vips` IS CLEARED. <<<** (2026-08-03, operator: *"Yes, drop the v6 problem legs"*.) Done via the RENDER pipeline, not a hand-edit: `render/values/vr1-dc0-vips.yaml` `family: dual -> v4` (the GUA v6 block RETAINED as the D-141