diff --git a/bundle.yaml b/bundle.yaml index 6416645..a6e2388 100644 --- a/bundle.yaml +++ b/bundle.yaml @@ -1040,13 +1040,21 @@ designate-hacluster: {base: ubuntu@22.04/stable, charm: hacluster, channel: 2.4/stable, options: {cluster_count: 3}, bindings: {'': metal-admin, ha: metal-internal, hanode: metal-internal, pacemaker-remote: metal-internal, peer-availability: metal-internal}} # D-106 -- reactivated 2026-07-10, supersedes D-019 deferral vault-hacluster: {base: ubuntu@22.04/stable, charm: hacluster, channel: 2.4/stable, options: {cluster_count: 3}, bindings: {'': metal-admin, ha: metal-internal, hanode: metal-internal, pacemaker-remote: metal-internal, peer-availability: metal-internal}} # D-121 (v-a): vault 3-unit HA on mysql; BUNDLEFIX-002 de-HA reversed 2026-08-05 - # memcached: nova-cloud-controller token/cell caching (BUNDLEFIX-004) + # memcached: nova-cloud-controller token/cell caching (BUNDLEFIX-004). + # Scaled 1->3 (BUNDLEFIX-055, 2026-08-06 -- operator ruling, completing the fold + # BUNDLEFIX-053 missed; the now-retired dc-ha-scaleup.yaml had carried this since + # operator direction 2026-07-31). NO hacluster, NO vip: three INDEPENDENT caches -- + # clients receive the full server list over the relation and hash keys across it, so + # losing one costs that shard's hit rate, not availability (the standard model). One + # per control node. VERIFY AT DEPLOY: the nova-cloud-controller:memcache and + # designate:coordinator-memcached consumers pick up all three servers (three + # memcached_servers entries over the relation, not one). memcached: base: ubuntu@22.04/stable charm: memcached channel: latest/stable - num_units: 1 - to: [lxd:0] + num_units: 3 + to: [lxd:0, lxd:1, lxd:2] bindings: '': metal-admin cache: metal-internal diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 5756446..62d08c5 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2379,6 +2379,13 @@ under D-068.** OWED: D-121 execution runbook (dc1 reuse); post-wave bundle/overlay review vs the live HA installs (pinned task). Body: `docs/changelog-20260805-d121-ha-scaleup.md`. + **memcached CONFIG=3, LIVE=1 (recorded 2026-08-06, BUNDLEFIX-055 / DOCFIX-212).** Operator + ruled 2026-08-06 ("3 units (restore intent)") to restore the 2026-07-31 memcached->3 direction + that BUNDLEFIX-053 never folded and the retired `dc-ha-scaleup.yaml` had carried unrealized. + `bundle.yaml` now sets memcached `num_units:3` + `to:[lxd:0,1,2]` (3 independent caches, no + hacluster/VIP); LIVE is still 1 unit (F4 capture). A teardown+redeploy from `bundle.yaml` brings + it to 3, or a gated live `juju add-unit memcached -n 2 --to lxd:1,lxd:2` -- not done this session + (hard rule 1). This is a CONFIG-of-record vs LIVE divergence, tracked here until reconciled. **NAMED-GATE DEFECT found by measurement -- `phase-03-core-verify.md` Step 3.1 asserts expected non-active/idle = 1 (octavia only); the VR1 roster yields 4 deferred-by-design + gss.** That gate is STALE for VR1 and a DOCFIX is owed (also owed on that runbook: `-m openstack` -> `-m vr1-dc0` diff --git a/docs/changelog-20260806-stage5-dc0-f4-postwave.md b/docs/changelog-20260806-stage5-dc0-f4-postwave.md index ff6a24f..c8e6ead 100644 --- a/docs/changelog-20260806-stage5-dc0-f4-postwave.md +++ b/docs/changelog-20260806-stage5-dc0-f4-postwave.md @@ -60,11 +60,13 @@ ## Item 5 -- Task #1 (post-wave review): dc-ha-scaleup.yaml is now REDUNDANT -- RECOMMENDATION, not executed -- MEASURED: `bundle.yaml` (via BUNDLEFIX-053) now carries the ENTIRE content of - `overlays/dc-ha-scaleup.yaml` -- `num_units:3` + `to:[lxd:0,1,2]` on all 14 apps, +- MEASURED: `bundle.yaml` (via BUNDLEFIX-053) carries the HA content of + `overlays/dc-ha-scaleup.yaml` -- `num_units:3` + `to:[lxd:0,1,2]` on the 14 HA apps, `cluster_count:3` on the 12 hacluster subs, `vault-hacluster` declared (line 1036), - rabbitmq `min-cluster-size:3` (170), memcached scaled, and the `[vault:ha, vault-hacluster:ha]` - relation (1058). Applying dc-ha-scaleup.yaml on top would be an all-keys no-op deep-merge. + rabbitmq `min-cluster-size:3` (170), and the `[vault:ha, vault-hacluster:ha]` relation (1058). + **CORRECTION 2026-08-06: NOT wholly redundant as first stated.** A full merge-diff (Item 7) + found ONE divergence -- `memcached` (overlay `num_units:3`, base `1`), which BUNDLEFIX-053 did + NOT fold. That is resolved by BUNDLEFIX-055 (Item 7); with it, the overlay is genuinely redundant. - The overlay header itself records "NOT YET APPLIED to any live model"; the live 14/14 scale-up was done via per-app `juju add-unit`, never via this overlay. - RECOMMENDATION was put to the operator; RULED 2026-08-06 -- see Item 6. @@ -73,8 +75,11 @@ - **Operator ruling, exact utterance: "Retire the redundancy and archive."** Recorded as a GA-R5 SUPERSESSION note on D-121's R6 ruling (`docs/design-decisions.md`), which retains the - original 2026-07-27 record below it as history. The overlay was WHOLLY REDUNDANT: BUNDLEFIX-053 - folded its entire content into `bundle.yaml`, so applying it was an all-keys no-op deep-merge. + original 2026-07-27 record below it as history. The overlay was redundant EXCEPT for one field: + BUNDLEFIX-053 folded the 14 HA apps + vault-hacluster + rabbitmq + the vault:ha relation (an + all-keys no-op there); a full merge-diff (Item 7) later found it ALSO carried memcached + `num_units:3` which BUNDLEFIX-053 missed (base stayed `1`). BUNDLEFIX-055 (Item 7) folds + memcached, making the archived overlay fully redundant. The retirement stands; the fold completes it. - **Archived:** `git mv overlays/dc-ha-scaleup.yaml docs/archive/dc-ha-scaleup-RETIRED-20260806.yaml` + a RETIRED banner (do-not-deploy). NO prior precedent for archiving an overlay (the only past retirement, `vr0-dc0-testcloud.yaml`, went to a one-time `remove/` dir); `docs/archive/` chosen @@ -107,3 +112,32 @@ 57/0; render-drift 4/0. - Revert: `git mv` the overlay back to overlays/ (strip the banner); restore the 5 harness cases and the doc/comment blocks (git revert this commit); re-render the vips overlays from reverted values. + +## Item 7 -- memcached VALUE drift caught by the retirement; folded to 3 [BUNDLEFIX-055 + DOCFIX-212] + +- **Found while reconciling a stale comment (operator asked for currency ahead of a possible + teardown+redeploy).** A full merge-diff of the archived overlay vs `bundle.yaml` (measured + 2026-08-06) showed the ONLY divergence was `memcached`: overlay `num_units:3` + `to:[lxd:0,1,2]` + vs base `num_units:1` + `to:[lxd:0]`. BUNDLEFIX-053 folded the HA chain but NOT memcached; the + overlay's memcached:3 (operator-directed 2026-07-31) was never folded and never deployed + (git: bundle.yaml memcached==1 at ae06c72 and HEAD; F4 capture: live memcached==1). +- **Owned:** last turn I told the operator "memcached=3 is in bundle.yaml:1044" -- WRONG (it was + `1`); and the retirement changelog/commit overstated BUNDLEFIX-053 as folding memcached / the + overlay as a full no-op. Both corrected above (Items 5, 6). +- **Operator ruling 2026-08-06 (GA-R5): "3 units (restore intent)."** So BUNDLEFIX-055 folds + memcached into `bundle.yaml` (`num_units:1->3`, `to:[lxd:0]->[lxd:0,1,2]`) -- completing the fold + BUNDLEFIX-053 missed and making the archived overlay genuinely, fully redundant. NO hacluster/VIP + (3 independent caches; clients hash across the full server list). The overlay's rationale comment + (previously "the only place the exclusion set is written down") is now carried in bundle.yaml. +- DOCFIX-212: `design-decisions.md` D-121 "Left single (NOT scaled)" list corrected -- memcached + removed from it and recorded as =3. +- **LIVE/CONFIG GAP (recorded in CURRENT-STATE):** live memcached is still 1; config-of-record is + now 3. A teardown+redeploy from `bundle.yaml` brings it up at 3; alternatively a gated live + `juju add-unit memcached -n 2 --to lxd:1,lxd:2`. Not done live this turn (hard rule 1; no + operator instruction to mutate live). +- **Re-stage owed:** this is a FUNCTIONAL bundle.yaml change (not a comment), so the dc0 rack + staged `bundle.yaml` (754f1d9e) is now genuinely stale -- re-stage + sha-verify (gated) OR the + redeploy re-stages. Logged. +- Gates: gauntlet + repo-lint (see close); provider-bundle-check re-run (memcached has no + hacluster/VIP/arity constraint, so the scale is orthogonal to those gates). +- Revert: `bundle.yaml` memcached back to `num_units:1` + `to:[lxd:0]`; revert the D-121 DOCFIX-212 line. diff --git a/docs/design-decisions.md b/docs/design-decisions.md index a5ee5f6..b89197b 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -4447,8 +4447,13 @@ | vault | 1 | 3 | see the backend sub-question below | D-006 (revised by C1) | Left single (NOT scaled): `ceph-rbd-mirror`=1 (**D-108** rules single-unit for VR1), -`glance-simplestreams-sync`=1 (utility), `memcached`=1 (cache; optional 3), `designate-bind`=1 -(DNS backend; optional 2). +`glance-simplestreams-sync`=1 (utility), `designate-bind`=1 +(DNS backend; optional 2). **memcached UPDATED 2026-08-06 (DOCFIX-212): now =3, NOT +single** -- operator directed the scale-up 2026-07-31; it was carried in the (now-retired) +dc-ha-scaleup.yaml overlay but BUNDLEFIX-053 did not fold it, so bundle.yaml + live sat at +1 until BUNDLEFIX-055 (2026-08-06) folded `num_units: 3` + `to:[lxd:0,1,2]` into the base +bundle (3 independent caches, no hacluster/VIP). LIVE is still 1 -- a redeploy from +bundle.yaml (or a live `add-unit -n 2`) brings it to 3. ### Two realities the "mechanical 1->3" framing hides (fold into the execution estimate)