diff --git a/docs/archive/dc-ha-scaleup-RETIRED-20260806.yaml b/docs/archive/dc-ha-scaleup-RETIRED-20260806.yaml new file mode 100644 index 0000000..02b6258 --- /dev/null +++ b/docs/archive/dc-ha-scaleup-RETIRED-20260806.yaml @@ -0,0 +1,254 @@ +# ============================================================================ +# RETIRED 2026-08-06 -- operator ruling "Retire the redundancy and archive". +# This overlay is NO LONGER a deploy input and is NO LONGER a live test fixture. +# BUNDLEFIX-053 (2026-08-05) folded its ENTIRE content into bundle.yaml (num_units +# 1->3 + one-per-control-node placement on all 14 HA apps; cluster_count 1->3 on the +# 12 hacluster subs; vault-hacluster declared; rabbitmq min-cluster-size:3; memcached +# 1->3; the [vault:ha, vault-hacluster:ha] relation), so applying this on top was an +# all-keys no-op deep-merge. It supersedes R6's "apply the overlay whole, sequenced +# after the VIPs" ruling (see docs/design-decisions.md D-121 R6 supersession note). +# KEPT here for history only. DO NOT pass it to `juju deploy --overlay`. +# ============================================================================ +# overlays/dc-ha-scaleup.yaml (original header follows) +# +# D-121 (ADOPTED IN PART, 2026-07-15) VR1 HA scale-up overlay -- makes the +# decorative single-unit control plane REAL by scaling 14 services from +# num_units: 1 to 3. VR1-SPECIFIC: VR0/testcloud stays single-unit BY DESIGN +# (D-009), so this is an overlay, NOT a bundle.yaml edit -- same pattern as +# overlays/dc-dc-ipv6-family-matrix.yaml. +# +# APPLIED SEPARATELY, AFTER the per-DC set -- NOT in the same command as the VIP +# overlay. R6 (RULED 2026-07-27, docs/design-decisions.md): operator utterance "Close +# the two VIP gaps first, then apply the overlay whole", and the ruling's own sequencing +# line, "R11 is therefore a hard Stage-5 precondition, sequenced BEFORE the overlay, not +# a parallel item". The two gaps are vault and designate; R11 landed their VIPs (.61 / +# .62) in the per-DC vips overlays. Merged into ONE command the scale-up and the VIPs +# arrive together and the ruled ordering is not observable, which is exactly the shape +# R6 refused (option (b): a 3-node pacemaker cluster for the CA with no VIP to manage). +# +# Step 1 -- deploy the per-DC set, let it settle: +# juju deploy ./bundle.yaml \ +# --overlay "overlays/${DC}-vips.yaml" \ +# --overlay "overlays/${DC}-machines.yaml" \ +# --overlay "overlays/${DC}-octavia-pki.yaml" \ +# -m "${DC_MODEL}" +# Step 2 -- only then, this overlay, whole: +# juju deploy ./bundle.yaml \ +# --overlay "overlays/${DC}-vips.yaml" \ +# --overlay "overlays/${DC}-machines.yaml" \ +# --overlay "overlays/${DC}-octavia-pki.yaml" \ +# --overlay overlays/dc-ha-scaleup.yaml \ +# -m "${DC_MODEL}" +# +# FILENAMES CORRECTED 2026-07-29 (NEW-4 / NEW-6). The command above previously named +# two files that cannot be passed as written: +# - `overlays/octavia-pki.yaml` -- RENAMED PER-DC under the D-109 amendment to +# `overlays/${DC}-octavia-pki.yaml` (.gitignore:41-46 documents the rename and +# covers it by GLOB precisely so each per-DC file stays ignored). +# - `overlays/${DC}-hostnames.yaml` -- HAS NEVER BEEN AUTHORED. It is proposed D-106 +# hostname work (bundle.yaml:13-18 records it as proposed, not built) and +# `os-public-hostname` remains unset everywhere; bundle.yaml's B5 IP-ONLY posture +# stands. juju errors out on a missing overlay file, so a phantom name inside a +# runnable command is a live failure, not a documentation nit. Removed rather than +# renamed -- there is nothing to point it at. +# +# NOT YET APPLIED to any live model -- authored prep-only (STAGE 5 prep; the +# per-DC bundle deploy is two stages out). No live cloud reached this session. +# +# ===================================================================== +# ##### RENDERED 2026-07-24 -- tokens bound to logical control ids 0/1/2 ##### +# The three {{VR1_DC0_CONTROL_0N}} tokens are RENDERED to the bundle.yaml machines +# block's LOGICAL control ids (0,1,2 -- the 3 control nodes, D-121 Option C). These +# ids are DC-neutral (both dc0 and dc1 use the same logical machines block; the DC +# tag is applied per-DC via overlays/vr1-dc1-machines.yaml), so this overlay is now +# DC-agnostic. The positional 3-entry `to:` lists give one-per-control-node +# anti-affinity (unit0->0, unit1->1, unit2->2); provider-bundle-check.py enforces the +# 3-distinct-machine spread on the MERGED bundle, so a token/id collision is caught. +# ===================================================================== +# +# PLACEMENT / ANTI-AFFINITY (D-121 balance rule). Juju bundles have NO dedicated +# anti-affinity primitive; one-per-node is expressed POSITIONALLY -- a 3-entry +# `to:` list binds unit0->entry0, unit1->entry1, unit2->entry2. Every scaled app +# below uses the SAME three-control-node spread, so: +# - each app lands exactly one unit per control node (14 apps x 1 = ~14 API +# LXD containers per control node -- the even ~14/node density the 64 GiB +# control-node sizing was validated for, D-121); and +# - the RAM-heavy stateful trio (mysql-innodb-cluster already 3 in the base +# bundle, rabbitmq-server + vault scaled here) auto-balances to one replica +# per control node -- never two stacked on one node. +# +# THREE ITEMS BELOW GO BEYOND A LITERAL `num_units: 3` BUMP. They are NOT +# scope-creep: each is the same class of "mechanical 1->3 hides a gotcha" that +# D-121 itself flags (D-121 caught the placement rewrite; these are the two it +# under-states plus the one its own wording got wrong), and an overlay titled +# "make HA real" that shipped decorative HA would not meet its stated purpose. +# All three are called out in the delivery changelog, not slipped in: +# (A) rabbitmq-server min-cluster-size: 3. Per D-009 AMENDMENT (2026-07-02): +# "rabbitmq scale-up is NOT mechanical without min-cluster-size" -- without +# it the charm accepts client relations BEFORE the cluster forms (same race +# as the D-062 mysql formation failure). Roosevelt/VR1 delta = set it +# alongside num_units: 3. rabbitmq clusters natively (quorum), NO hacluster. +# (B) hacluster cluster_count: 3 (all 12 active hacluster subs + the re-added +# vault-hacluster). The base bundle pins cluster_count: 1 -- explicitly +# "decorative on single-unit testcloud" (D-009 / BUNDLEFIX-003). At 3 units +# with cluster_count still 1, pacemaker can bootstrap the VIP off a single +# node = the SAME decorative behavior D-121 exists to eliminate. 3 makes +# the charm wait for a real 3-node quorum before configuring resources. +# VERIFY at deploy (not fetched this session): confirm the hacluster +# 2.4/stable charm treats cluster_count as "peers required before the +# cluster forms / VIP binds" -- the base bundle explicitly pinned it to 1 +# as decorative, and real 3-node quorum needs 3 regardless of the charm +# default. Flag if `juju config hacluster` disagrees. +# (C) vault-hacluster is DEFINED here, not just related. D-121's prose says it +# "exists among the hacluster subs" -- it does NOT: bundle.yaml line ~923 +# has it COMMENTED OUT (removed by BUNDLEFIX-002 when vault was de-HA'd, +# C1). So the re-added `[vault:ha, vault-hacluster:ha]` relation would +# reference a non-existent app (provider-bundle-check rule 4 would FAIL). +# This overlay re-declares it, matching the other haclusters' shape. +# +# VAULT BACKEND (D-121 sub-ruling = v-a, RESOLVED 2026-07-15): vault stays +# MYSQL-backed (on the already-3-unit mysql-innodb-cluster via vault-mysql-router, +# unchanged) and scales to 3 -- NO etcd, NO Raft here. GATE (Stage 5, before +# commit): verify the deployed vault 1.8/stable charm actually does HA +# leader-election on the MySQL storage backend at 3 units. If it does NOT: do +# NOT fall back to etcd -- evaluate Raft integrated storage and route to D-068. +# 3-unit vault ALSO triples the manual-unseal burden (intersects OPEN SEC-003). +# +# NOT changed here (deliberately, per D-121): mysql-innodb-cluster / ovn-central +# / ceph-mon (already 3 in the base bundle); ceph-osd / nova-compute (scale-out, +# sized to node count -- Option C = ceph-osd 4 (D-121 R-3 amendment), nova-compute 2, on the storage / +# compute nodes, NOT control-node containers); designate-bind / +# glance-simplestreams-sync unchanged. +# +# memcached WAS in that "unchanged" list and is now SCALED (see its block below, +# added 2026-07-31 by operator direction). This comment is RE-POINTED rather than +# left stale -- it is the only place the exclusion set is written down. +# +# >>> ceph-rbd-mirror STAYS 1, AND THAT IS PINNED, NOT SETTLED. <<< +# D-108 rules it: "one rbd-mirror daemon per DC ... daemon HA is single-unit for VR1 +# (matches the D-104 controller economy), NOTED FOR ROOSEVELT SCALE-UP". So the +# single unit is a ruled VR1 scope decision, not an oversight -- but D-108 itself +# flags that it must be revisited, and operator direction 2026-07-31 is that it be +# brought into line with the rest of the HA posture rather than quietly left behind. +# It is NOT a num_units edit: see docs/dc-dc-deployment-workflow.md gap register +# item 22 for the options, the recommendation, and why this one needs a D-108 +# amendment before anything is built. DO NOT scale it here without that ruling. +# ===================================================================== + +applications: + + # ---- Clustered API charms (hacluster + dual VIP already wired in base) ---- + # Each: 1 -> 3 units, one per control node (positional anti-affinity). + keystone: + num_units: 3 + to: ["lxd:0", "lxd:1", "lxd:2"] + + glance: + num_units: 3 + to: ["lxd:0", "lxd:1", "lxd:2"] + + neutron-api: + num_units: 3 + to: ["lxd:0", "lxd:1", "lxd:2"] + + nova-cloud-controller: + num_units: 3 + to: ["lxd:0", "lxd:1", "lxd:2"] + + placement: + num_units: 3 + to: ["lxd:0", "lxd:1", "lxd:2"] + + cinder: + num_units: 3 + to: ["lxd:0", "lxd:1", "lxd:2"] + + openstack-dashboard: + num_units: 3 + to: ["lxd:0", "lxd:1", "lxd:2"] + + octavia: + num_units: 3 + to: ["lxd:0", "lxd:1", "lxd:2"] + + barbican: + num_units: 3 + to: ["lxd:0", "lxd:1", "lxd:2"] + + magnum: + num_units: 3 + to: ["lxd:0", "lxd:1", "lxd:2"] + + designate: + num_units: 3 + to: ["lxd:0", "lxd:1", "lxd:2"] + + ceph-radosgw: + num_units: 3 + to: ["lxd:0", "lxd:1", "lxd:2"] + + # ---- RabbitMQ: native quorum clustering, NO hacluster. min-cluster-size is + # MANDATORY at multi-unit (D-009 AMENDMENT 2026-07-02, item A above). ---- + rabbitmq-server: + num_units: 3 + options: + min-cluster-size: 3 + to: ["lxd:0", "lxd:1", "lxd:2"] + + # ---- memcached: 1 -> 3, ADDED 2026-07-31 by operator direction. It was left at 1 + # by D-121's sizing table, which listed it as "memcached=1 (cache; optional 3)" + # -- CONSIDERED and marked optional, but with no recorded reason for choosing 1. + # MEASURED consumers in bundle.yaml: nova-cloud-controller:memcache and + # designate:coordinator-memcached. (An earlier claim in-session that keystone + # consumes it was WRONG -- keystone holds no memcache relation in this bundle.) + # NO hacluster and NO vip, deliberately: memcached is not clustered. Three units + # are three INDEPENDENT caches; clients receive the full server list over the + # relation and hash keys across it, so losing one costs that shard's hit rate, + # not availability. That is the standard model, not a compromise. + # VERIFY AT DEPLOY: the consumers pick up all three servers (the relation should + # yield three memcached_servers entries, not one). + memcached: + num_units: 3 + to: ["lxd:0", "lxd:1", "lxd:2"] + + # ---- Vault: MySQL-backed (v-a), scale to 3, re-HA'd via vault-hacluster + # (defined below) + the relation re-added at the foot of this file. ---- + vault: + num_units: 3 + to: ["lxd:0", "lxd:1", "lxd:2"] + + # ---- vault-hacluster: RE-DECLARED (item C above). Removed by BUNDLEFIX-002 + # when vault was de-HA'd; D-121 (v-a) brings it back. Shape matches the + # other hacluster subs in bundle.yaml, with cluster_count: 3 (real HA). + # It carries an explicit `base` for the same reason all 56 applications in + # bundle.yaml do (RULED 2026-07-31, GA-R5 option D; provider-bundle-check + # invariant 12): this is the ONLY application any overlay INTRODUCES, so + # without it this later deploy step re-opens the once-only-deployable trap + # that invariant 12 exists to close. Found by that invariant on its first + # run, in a file the fix was not looking at. ---- + vault-hacluster: {base: ubuntu@22.04/stable, charm: hacluster, channel: 2.4/stable, options: {cluster_count: 3}, bindings: {'': metal-admin, ha: metal-internal, hanode: metal-internal, pacemaker-remote: metal-internal, peer-availability: metal-internal}} + + # ---- cluster_count: 1 -> 3 for the 12 active hacluster subs (item B above). + # Overlay apply deep-merges options per-key -- the SAME mechanism + # dc-dc-ipv6-family-matrix.yaml already relies on (it supplies only + # `options:` for keystone et al. and depends on charm/channel/bindings + # surviving from base). So ONLY cluster_count is overridden here; each + # sub's base bindings/charm/channel are preserved. ---- + keystone-hacluster: {options: {cluster_count: 3}} + glance-hacluster: {options: {cluster_count: 3}} + neutron-api-hacluster: {options: {cluster_count: 3}} + nova-cloud-controller-hacluster: {options: {cluster_count: 3}} + placement-hacluster: {options: {cluster_count: 3}} + openstack-dashboard-hacluster: {options: {cluster_count: 3}} + cinder-hacluster: {options: {cluster_count: 3}} + octavia-hacluster: {options: {cluster_count: 3}} + barbican-hacluster: {options: {cluster_count: 3}} + magnum-hacluster: {options: {cluster_count: 3}} + ceph-radosgw-hacluster: {options: {cluster_count: 3}} + designate-hacluster: {options: {cluster_count: 3}} + +relations: + # ---- Re-add vault HA (removed by BUNDLEFIX-002; D-121 v-a re-adds it). + # vault already carries the `ha: metal-internal` binding in the base + # bundle, and vault-hacluster is declared above. ---- + - [vault:ha, vault-hacluster:ha] diff --git a/docs/changelog-20260806-stage5-dc0-f4-postwave.md b/docs/changelog-20260806-stage5-dc0-f4-postwave.md index 741285b..ff6a24f 100644 --- a/docs/changelog-20260806-stage5-dc0-f4-postwave.md +++ b/docs/changelog-20260806-stage5-dc0-f4-postwave.md @@ -36,11 +36,11 @@ NO `~/repo-stage` (dc1 HELD/undeployed). Re-stage matters only at the NEXT dc0 deploy (running cloud already converged). Prepared + gated -- rack write not executed unattended. - NOTE: F-A (Item 4) changes bundle.yaml, so the re-stage MUST carry the post-F-A bundle. -- ATTEMPTED 2026-08-06 after committing F-A (scp post-F-A bundle.yaml + vr1-dc0-vips.yaml to - the dc0 rack + sha256-verify); the Claude Code auto-mode classifier DENIED the rack write. - That is the hard rule 3 gate working as designed (read-only audits passed all session; the - rack WRITE is blocked). F9 stays PENDING explicit operator approval / a Bash permission rule. - Revert: n/a (no write occurred). +- First attempt DENIED by the auto-mode classifier (hard rule 3 gate). OPERATOR APPROVED + 2026-08-06; re-run SUCCEEDED. Both files sha256-verified on the dc0 rack == repo HEAD: + bundle.yaml=754f1d9e (post-F-A), vr1-dc0-vips.yaml=3f403408. dc0 rack ~/repo-stage now current. + dc1 rack has no ~/repo-stage (HELD) -- future standup work, not owed now. + Revert: re-stage any prior committed bundle/overlay from git (staging is a copy of tracked files). ## Item 4 -- F-A / BUNDLEFIX-054: bundle.yaml HA-chain header corrected [OPS] @@ -67,10 +67,43 @@ relation (1058). Applying dc-ha-scaleup.yaml on top would be an all-keys no-op deep-merge. - The overlay header itself records "NOT YET APPLIED to any live model"; the live 14/14 scale-up was done via per-app `juju add-unit`, never via this overlay. -- **RECOMMENDATION (operator decision -- NOT executed):** retire `overlays/dc-ha-scaleup.yaml` - and its references (runbook Step 2 whole-overlay apply; provider-bundle-check T17 ordering - assertion; the R6 2026-07-27 "apply the overlay whole, sequenced after VIPs" ruling). Because - R6 is a RULING and D-121 references the overlay, retiring it needs an operator ruling / D-121 - amendment, not an unattended edit (hard rule 1). Logged here + already flagged in - provider-bundle-check T16 comment. -- Revert: n/a (finding only). +- RECOMMENDATION was put to the operator; RULED 2026-08-06 -- see Item 6. + +## Item 6 -- RETIRE + ARCHIVE dc-ha-scaleup.yaml (operator-ruled 2026-08-06; R6 SUPERSEDED) [DOCFIX-211] + +- **Operator ruling, exact utterance: "Retire the redundancy and archive."** Recorded as a + GA-R5 SUPERSESSION note on D-121's R6 ruling (`docs/design-decisions.md`), which retains the + original 2026-07-27 record below it as history. The overlay was WHOLLY REDUNDANT: BUNDLEFIX-053 + folded its entire content into `bundle.yaml`, so applying it was an all-keys no-op deep-merge. +- **Archived:** `git mv overlays/dc-ha-scaleup.yaml docs/archive/dc-ha-scaleup-RETIRED-20260806.yaml` + + a RETIRED banner (do-not-deploy). NO prior precedent for archiving an overlay (the only past + retirement, `vr0-dc0-testcloud.yaml`, went to a one-time `remove/` dir); `docs/archive/` chosen + explicitly as the repo's archive location. +- **Harness (tests/provider-bundle-check/run-tests.sh) re-pointed off the retired fixture:** + T17 REMOVED (once vault-hacluster moved to base it became identical to T18); T17b/T32/T33/T34 + RE-POINTED onto `good.yaml` (the base HA chain) -- T33/T34 now use the deterministic `mutate()` + helper instead of sed-on-overlay (a wrong key path KeyErrors loudly; no silent no-op fixture). + Each re-pointed FAIL case (`run 1`) proven to fire against the real checker message (captured + standalone before editing). Count 58 -> 57 (T17 dropped). provider-bundle-check 57/0. +- **Docs reconciled (present-structure only; dated history left as history):** + design-decisions.md (R6 supersession note + `:503` + the `:4563` "overlay encodes v-a" ref, now + bundle.yaml/BUNDLEFIX-053); runbook `dc-dc-phase4` Step-4 note + Step-12.3(a) gate (the two-phase + "base at cluster_count:1, then apply overlay to reach 3" model RETIRED -- base now deploys 3-unit + HA directly, so `cluster_count:3` is the expected post-Step-4 value); `dc-dc-deployment-workflow.md` + item 22; `scripts/provider-bundle-check.py` + `scripts/cloud-assert.sh` comments. +- **Rendered vips overlays:** the "land BEFORE dc-ha-scaleup.yaml" comment lives in the RENDER + SOURCE (`render/values/vr1-dc{0,1}-vips.yaml`, D-136). Edited the SOURCE and RE-RENDERED (never + hand-edited the overlay -- the W-1 trap); overlay diff vs HEAD is exactly the two comment lines; + render-drift 4/0. Hand-maintained `vr1-dc{0,1}-machines.yaml` deploy-command comments edited direct. +- **CURRENT-STATE:** NO edit needed -- its active Stage-5 status already lists the deploy input as + "bundle + -vips/-machines/-octavia-pki overlays" (no HA overlay), consistent with BUNDLEFIX-053. + The remaining `dc-ha-scaleup` mentions are dated historical narrative (2026-07-27 captures, NEW-7 + findings) and are retained as history (not present status; GA-R1). +- **Staging note:** the dc0 rack `~/repo-stage/overlays/vr1-dc0-vips.yaml` re-staged in Item 3 is + now stale-by-ONE-COMMENT vs the re-render (functionally identical -- no VIP/data change). Not + re-staged (a comment does not warrant another gated rack write); it re-syncs + sha-verifies at + the next real deploy per D-138. OWED-at-next-deploy, logged. +- Gates: gauntlet run (see close); repo-lint 0 fail (1 pre-existing L1 warn); provider-bundle-check + 57/0; render-drift 4/0. +- Revert: `git mv` the overlay back to overlays/ (strip the banner); restore the 5 harness cases and + the doc/comment blocks (git revert this commit); re-render the vips overlays from reverted values. diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md index abd6c32..c9e1891 100644 --- a/docs/dc-dc-deployment-workflow.md +++ b/docs/dc-dc-deployment-workflow.md @@ -1279,8 +1279,9 @@ where the DR posture is real, and D-108 already anticipates exactly that. (c) is over-engineering for a rehearsal. - **DO NOT scale it in `overlays/dc-ha-scaleup.yaml` without the D-108 amendment** -- - that overlay now carries a pointer to this item saying so. + **DO NOT scale it (`ceph-rbd-mirror`) into HA without the D-108 amendment** -- if it + is ever scaled, that edit goes in `bundle.yaml` (`dc-ha-scaleup.yaml` was retired + 2026-08-06, its HA chain folded into the base by BUNDLEFIX-053), never silently. **STATUS: PINNED, awaiting a GA-R5 ruling. NOT to be closed by silently scaling it.** diff --git a/docs/design-decisions.md b/docs/design-decisions.md index b1e6e01..a5ee5f6 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -499,8 +499,10 @@ `VIP_COUNT_EXPECT` also moves 11 -> 13. **Execution is a SEPARATE gated step.** Standard delivery discipline applies to the checker -change (harness green, gauntlet, repo-lint, changelog with revert). Per the R6 ruling these -VIPs land BEFORE `overlays/dc-ha-scaleup.yaml` is applied. +change (harness green, gauntlet, repo-lint, changelog with revert). These VIPs land in the +per-DC `*-vips.yaml` overlay, merged with the base bundle at deploy. (R6's "before the HA +overlay" sequencing is SUPERSEDED 2026-08-06 -- `dc-ha-scaleup.yaml` was retired once +BUNDLEFIX-053 folded the HA chain into the base bundle; see the R6 supersession note.) ## D-021: Octavia amphora image pipeline on the no-DNS dual-endpoint deploy @@ -4560,7 +4562,8 @@ delivery "Proceed", never vault -- R3-F01). The operator confirmed v-a on 2026-07-16 after (v-b) etcd was rejected (deprecated) and Raft was found INFEASIBLE on the deployable 1.8/stable charm (Raft is a 1.16-charm feature; 1.16 is ruled incompatible, D-068/BUNDLEFIX-010). Raft is parked as - the D-068/Roosevelt target. No code delta -- `overlays/dc-ha-scaleup.yaml` already encodes v-a. + the D-068/Roosevelt target. No code delta at ruling time -- v-a was encoded in the HA overlay, + then folded into `bundle.yaml` by BUNDLEFIX-053 (2026-08-05); the overlay was retired 2026-08-06. 3. **Re-validation OWED (R3-F06):** the whole-host capacity model `scratchpad/optc-calc.py` is UNCOMMITTED and absent, so the changelog's "reproducible" claim is currently false. It must be committed as a harnessed calculator and RE-RUN for the 3+2+4 shape AND Model B nesting overhead @@ -4638,6 +4641,24 @@ **Status:** RULED 2026-07-27 (operator, GA-R5). Records a ruling; amends nothing in D-121's node layout or its 3-unit intent. +**>>> SUPERSEDED 2026-08-06 (operator, GA-R5). R6's sequencing is MOOT -- `dc-ha-scaleup.yaml` +was RETIRED. <<<** BUNDLEFIX-053 (2026-08-05) folded the overlay's ENTIRE content into +`bundle.yaml` (14 apps `num_units` 1->3 + one-per-control-node placement; 12 hacluster subs +`cluster_count` 1->3; `vault-hacluster` declared; rabbitmq `min-cluster-size: 3`; memcached +1->3; the `[vault:ha, vault-hacluster:ha]` relation) -- so applying the overlay became an +all-keys no-op deep-merge (Task-#1 post-wave review, MEASURED against the merged dc0 input). +There is no longer a separate "apply the overlay after the VIPs" step: the base bundle carries +the full HA chain, the per-DC VIP overlay supplies vault's + designate's VIPs, and a single +`juju deploy` of base+vips lands 3-unit HA directly. **Question as presented:** "dc-ha-scaleup.yaml +is now wholly redundant with bundle.yaml (BUNDLEFIX-053) -- retire it?" **Operator utterance, +exact: "Retire the redundancy and archive."** The overlay is archived at +`docs/archive/dc-ha-scaleup-RETIRED-20260806.yaml`; its five harness fixtures (T17 removed as +redundant with T18; T17b/T32/T33/T34 re-pointed onto the base HA chain, gauntlet green). The +ordering CONCERN R6 addressed -- vault/designate VIPs must exist before HA is "real" -- STILL +HOLDS and is still enforced by `provider-bundle-check` (T16b/T17b: an hacluster principal with +no VIP FAILS); it is simply no longer sequenced via a separate overlay. The record below is +retained as the history of why the two-phase sequence once existed. + Question as presented, verbatim: "R6 -- HA scale-up overlay at Stage 5. D-121's premise is 'VR1 makes HA real', but applying the overlay as it stands gives vault a 3-node pacemaker cluster with no VIP to manage -- decorative HA for the CA that 23 relations depend on. diff --git a/overlays/dc-ha-scaleup.yaml b/overlays/dc-ha-scaleup.yaml deleted file mode 100644 index 0513165..0000000 --- a/overlays/dc-ha-scaleup.yaml +++ /dev/null @@ -1,243 +0,0 @@ -# overlays/dc-ha-scaleup.yaml -# -# D-121 (ADOPTED IN PART, 2026-07-15) VR1 HA scale-up overlay -- makes the -# decorative single-unit control plane REAL by scaling 14 services from -# num_units: 1 to 3. VR1-SPECIFIC: VR0/testcloud stays single-unit BY DESIGN -# (D-009), so this is an overlay, NOT a bundle.yaml edit -- same pattern as -# overlays/dc-dc-ipv6-family-matrix.yaml. -# -# APPLIED SEPARATELY, AFTER the per-DC set -- NOT in the same command as the VIP -# overlay. R6 (RULED 2026-07-27, docs/design-decisions.md): operator utterance "Close -# the two VIP gaps first, then apply the overlay whole", and the ruling's own sequencing -# line, "R11 is therefore a hard Stage-5 precondition, sequenced BEFORE the overlay, not -# a parallel item". The two gaps are vault and designate; R11 landed their VIPs (.61 / -# .62) in the per-DC vips overlays. Merged into ONE command the scale-up and the VIPs -# arrive together and the ruled ordering is not observable, which is exactly the shape -# R6 refused (option (b): a 3-node pacemaker cluster for the CA with no VIP to manage). -# -# Step 1 -- deploy the per-DC set, let it settle: -# juju deploy ./bundle.yaml \ -# --overlay "overlays/${DC}-vips.yaml" \ -# --overlay "overlays/${DC}-machines.yaml" \ -# --overlay "overlays/${DC}-octavia-pki.yaml" \ -# -m "${DC_MODEL}" -# Step 2 -- only then, this overlay, whole: -# juju deploy ./bundle.yaml \ -# --overlay "overlays/${DC}-vips.yaml" \ -# --overlay "overlays/${DC}-machines.yaml" \ -# --overlay "overlays/${DC}-octavia-pki.yaml" \ -# --overlay overlays/dc-ha-scaleup.yaml \ -# -m "${DC_MODEL}" -# -# FILENAMES CORRECTED 2026-07-29 (NEW-4 / NEW-6). The command above previously named -# two files that cannot be passed as written: -# - `overlays/octavia-pki.yaml` -- RENAMED PER-DC under the D-109 amendment to -# `overlays/${DC}-octavia-pki.yaml` (.gitignore:41-46 documents the rename and -# covers it by GLOB precisely so each per-DC file stays ignored). -# - `overlays/${DC}-hostnames.yaml` -- HAS NEVER BEEN AUTHORED. It is proposed D-106 -# hostname work (bundle.yaml:13-18 records it as proposed, not built) and -# `os-public-hostname` remains unset everywhere; bundle.yaml's B5 IP-ONLY posture -# stands. juju errors out on a missing overlay file, so a phantom name inside a -# runnable command is a live failure, not a documentation nit. Removed rather than -# renamed -- there is nothing to point it at. -# -# NOT YET APPLIED to any live model -- authored prep-only (STAGE 5 prep; the -# per-DC bundle deploy is two stages out). No live cloud reached this session. -# -# ===================================================================== -# ##### RENDERED 2026-07-24 -- tokens bound to logical control ids 0/1/2 ##### -# The three {{VR1_DC0_CONTROL_0N}} tokens are RENDERED to the bundle.yaml machines -# block's LOGICAL control ids (0,1,2 -- the 3 control nodes, D-121 Option C). These -# ids are DC-neutral (both dc0 and dc1 use the same logical machines block; the DC -# tag is applied per-DC via overlays/vr1-dc1-machines.yaml), so this overlay is now -# DC-agnostic. The positional 3-entry `to:` lists give one-per-control-node -# anti-affinity (unit0->0, unit1->1, unit2->2); provider-bundle-check.py enforces the -# 3-distinct-machine spread on the MERGED bundle, so a token/id collision is caught. -# ===================================================================== -# -# PLACEMENT / ANTI-AFFINITY (D-121 balance rule). Juju bundles have NO dedicated -# anti-affinity primitive; one-per-node is expressed POSITIONALLY -- a 3-entry -# `to:` list binds unit0->entry0, unit1->entry1, unit2->entry2. Every scaled app -# below uses the SAME three-control-node spread, so: -# - each app lands exactly one unit per control node (14 apps x 1 = ~14 API -# LXD containers per control node -- the even ~14/node density the 64 GiB -# control-node sizing was validated for, D-121); and -# - the RAM-heavy stateful trio (mysql-innodb-cluster already 3 in the base -# bundle, rabbitmq-server + vault scaled here) auto-balances to one replica -# per control node -- never two stacked on one node. -# -# THREE ITEMS BELOW GO BEYOND A LITERAL `num_units: 3` BUMP. They are NOT -# scope-creep: each is the same class of "mechanical 1->3 hides a gotcha" that -# D-121 itself flags (D-121 caught the placement rewrite; these are the two it -# under-states plus the one its own wording got wrong), and an overlay titled -# "make HA real" that shipped decorative HA would not meet its stated purpose. -# All three are called out in the delivery changelog, not slipped in: -# (A) rabbitmq-server min-cluster-size: 3. Per D-009 AMENDMENT (2026-07-02): -# "rabbitmq scale-up is NOT mechanical without min-cluster-size" -- without -# it the charm accepts client relations BEFORE the cluster forms (same race -# as the D-062 mysql formation failure). Roosevelt/VR1 delta = set it -# alongside num_units: 3. rabbitmq clusters natively (quorum), NO hacluster. -# (B) hacluster cluster_count: 3 (all 12 active hacluster subs + the re-added -# vault-hacluster). The base bundle pins cluster_count: 1 -- explicitly -# "decorative on single-unit testcloud" (D-009 / BUNDLEFIX-003). At 3 units -# with cluster_count still 1, pacemaker can bootstrap the VIP off a single -# node = the SAME decorative behavior D-121 exists to eliminate. 3 makes -# the charm wait for a real 3-node quorum before configuring resources. -# VERIFY at deploy (not fetched this session): confirm the hacluster -# 2.4/stable charm treats cluster_count as "peers required before the -# cluster forms / VIP binds" -- the base bundle explicitly pinned it to 1 -# as decorative, and real 3-node quorum needs 3 regardless of the charm -# default. Flag if `juju config hacluster` disagrees. -# (C) vault-hacluster is DEFINED here, not just related. D-121's prose says it -# "exists among the hacluster subs" -- it does NOT: bundle.yaml line ~923 -# has it COMMENTED OUT (removed by BUNDLEFIX-002 when vault was de-HA'd, -# C1). So the re-added `[vault:ha, vault-hacluster:ha]` relation would -# reference a non-existent app (provider-bundle-check rule 4 would FAIL). -# This overlay re-declares it, matching the other haclusters' shape. -# -# VAULT BACKEND (D-121 sub-ruling = v-a, RESOLVED 2026-07-15): vault stays -# MYSQL-backed (on the already-3-unit mysql-innodb-cluster via vault-mysql-router, -# unchanged) and scales to 3 -- NO etcd, NO Raft here. GATE (Stage 5, before -# commit): verify the deployed vault 1.8/stable charm actually does HA -# leader-election on the MySQL storage backend at 3 units. If it does NOT: do -# NOT fall back to etcd -- evaluate Raft integrated storage and route to D-068. -# 3-unit vault ALSO triples the manual-unseal burden (intersects OPEN SEC-003). -# -# NOT changed here (deliberately, per D-121): mysql-innodb-cluster / ovn-central -# / ceph-mon (already 3 in the base bundle); ceph-osd / nova-compute (scale-out, -# sized to node count -- Option C = ceph-osd 4 (D-121 R-3 amendment), nova-compute 2, on the storage / -# compute nodes, NOT control-node containers); designate-bind / -# glance-simplestreams-sync unchanged. -# -# memcached WAS in that "unchanged" list and is now SCALED (see its block below, -# added 2026-07-31 by operator direction). This comment is RE-POINTED rather than -# left stale -- it is the only place the exclusion set is written down. -# -# >>> ceph-rbd-mirror STAYS 1, AND THAT IS PINNED, NOT SETTLED. <<< -# D-108 rules it: "one rbd-mirror daemon per DC ... daemon HA is single-unit for VR1 -# (matches the D-104 controller economy), NOTED FOR ROOSEVELT SCALE-UP". So the -# single unit is a ruled VR1 scope decision, not an oversight -- but D-108 itself -# flags that it must be revisited, and operator direction 2026-07-31 is that it be -# brought into line with the rest of the HA posture rather than quietly left behind. -# It is NOT a num_units edit: see docs/dc-dc-deployment-workflow.md gap register -# item 22 for the options, the recommendation, and why this one needs a D-108 -# amendment before anything is built. DO NOT scale it here without that ruling. -# ===================================================================== - -applications: - - # ---- Clustered API charms (hacluster + dual VIP already wired in base) ---- - # Each: 1 -> 3 units, one per control node (positional anti-affinity). - keystone: - num_units: 3 - to: ["lxd:0", "lxd:1", "lxd:2"] - - glance: - num_units: 3 - to: ["lxd:0", "lxd:1", "lxd:2"] - - neutron-api: - num_units: 3 - to: ["lxd:0", "lxd:1", "lxd:2"] - - nova-cloud-controller: - num_units: 3 - to: ["lxd:0", "lxd:1", "lxd:2"] - - placement: - num_units: 3 - to: ["lxd:0", "lxd:1", "lxd:2"] - - cinder: - num_units: 3 - to: ["lxd:0", "lxd:1", "lxd:2"] - - openstack-dashboard: - num_units: 3 - to: ["lxd:0", "lxd:1", "lxd:2"] - - octavia: - num_units: 3 - to: ["lxd:0", "lxd:1", "lxd:2"] - - barbican: - num_units: 3 - to: ["lxd:0", "lxd:1", "lxd:2"] - - magnum: - num_units: 3 - to: ["lxd:0", "lxd:1", "lxd:2"] - - designate: - num_units: 3 - to: ["lxd:0", "lxd:1", "lxd:2"] - - ceph-radosgw: - num_units: 3 - to: ["lxd:0", "lxd:1", "lxd:2"] - - # ---- RabbitMQ: native quorum clustering, NO hacluster. min-cluster-size is - # MANDATORY at multi-unit (D-009 AMENDMENT 2026-07-02, item A above). ---- - rabbitmq-server: - num_units: 3 - options: - min-cluster-size: 3 - to: ["lxd:0", "lxd:1", "lxd:2"] - - # ---- memcached: 1 -> 3, ADDED 2026-07-31 by operator direction. It was left at 1 - # by D-121's sizing table, which listed it as "memcached=1 (cache; optional 3)" - # -- CONSIDERED and marked optional, but with no recorded reason for choosing 1. - # MEASURED consumers in bundle.yaml: nova-cloud-controller:memcache and - # designate:coordinator-memcached. (An earlier claim in-session that keystone - # consumes it was WRONG -- keystone holds no memcache relation in this bundle.) - # NO hacluster and NO vip, deliberately: memcached is not clustered. Three units - # are three INDEPENDENT caches; clients receive the full server list over the - # relation and hash keys across it, so losing one costs that shard's hit rate, - # not availability. That is the standard model, not a compromise. - # VERIFY AT DEPLOY: the consumers pick up all three servers (the relation should - # yield three memcached_servers entries, not one). - memcached: - num_units: 3 - to: ["lxd:0", "lxd:1", "lxd:2"] - - # ---- Vault: MySQL-backed (v-a), scale to 3, re-HA'd via vault-hacluster - # (defined below) + the relation re-added at the foot of this file. ---- - vault: - num_units: 3 - to: ["lxd:0", "lxd:1", "lxd:2"] - - # ---- vault-hacluster: RE-DECLARED (item C above). Removed by BUNDLEFIX-002 - # when vault was de-HA'd; D-121 (v-a) brings it back. Shape matches the - # other hacluster subs in bundle.yaml, with cluster_count: 3 (real HA). - # It carries an explicit `base` for the same reason all 56 applications in - # bundle.yaml do (RULED 2026-07-31, GA-R5 option D; provider-bundle-check - # invariant 12): this is the ONLY application any overlay INTRODUCES, so - # without it this later deploy step re-opens the once-only-deployable trap - # that invariant 12 exists to close. Found by that invariant on its first - # run, in a file the fix was not looking at. ---- - vault-hacluster: {base: ubuntu@22.04/stable, charm: hacluster, channel: 2.4/stable, options: {cluster_count: 3}, bindings: {'': metal-admin, ha: metal-internal, hanode: metal-internal, pacemaker-remote: metal-internal, peer-availability: metal-internal}} - - # ---- cluster_count: 1 -> 3 for the 12 active hacluster subs (item B above). - # Overlay apply deep-merges options per-key -- the SAME mechanism - # dc-dc-ipv6-family-matrix.yaml already relies on (it supplies only - # `options:` for keystone et al. and depends on charm/channel/bindings - # surviving from base). So ONLY cluster_count is overridden here; each - # sub's base bindings/charm/channel are preserved. ---- - keystone-hacluster: {options: {cluster_count: 3}} - glance-hacluster: {options: {cluster_count: 3}} - neutron-api-hacluster: {options: {cluster_count: 3}} - nova-cloud-controller-hacluster: {options: {cluster_count: 3}} - placement-hacluster: {options: {cluster_count: 3}} - openstack-dashboard-hacluster: {options: {cluster_count: 3}} - cinder-hacluster: {options: {cluster_count: 3}} - octavia-hacluster: {options: {cluster_count: 3}} - barbican-hacluster: {options: {cluster_count: 3}} - magnum-hacluster: {options: {cluster_count: 3}} - ceph-radosgw-hacluster: {options: {cluster_count: 3}} - designate-hacluster: {options: {cluster_count: 3}} - -relations: - # ---- Re-add vault HA (removed by BUNDLEFIX-002; D-121 v-a re-adds it). - # vault already carries the `ha: metal-internal` binding in the base - # bundle, and vault-hacluster is declared above. ---- - - [vault:ha, vault-hacluster:ha] diff --git a/overlays/vr1-dc0-machines.yaml b/overlays/vr1-dc0-machines.yaml index 4ef4dfd..9c153bd 100644 --- a/overlays/vr1-dc0-machines.yaml +++ b/overlays/vr1-dc0-machines.yaml @@ -99,8 +99,9 @@ # runbooks/dc-dc-phase4-juju-bundle-per-dc.md Step 4.4, run the moment `juju deploy` # returns and before the model settles. Assert it explicitly; do not assume. <<< # -# Applied as part of the vr1-dc0 per-DC set (R6: this set lands BEFORE -# overlays/dc-ha-scaleup.yaml, never in the same command as it): +# Applied as the vr1-dc0 per-DC set. The HA chain is in the base bundle now +# (BUNDLEFIX-053; dc-ha-scaleup.yaml retired 2026-08-06), so this IS the whole deploy +# -- there is no separate HA overlay to sequence after it (R6 superseded): # juju deploy ./bundle.yaml \ # --overlay overlays/vr1-dc0-vips.yaml \ # --overlay overlays/vr1-dc0-machines.yaml \ diff --git a/overlays/vr1-dc0-vips.yaml b/overlays/vr1-dc0-vips.yaml index 1fb058c..28acaf1 100644 --- a/overlays/vr1-dc0-vips.yaml +++ b/overlays/vr1-dc0-vips.yaml @@ -27,8 +27,8 @@ # the DEDICATED GUA VIP /64, admin+internal their own plane /64s # v6 host part MIRRORS the v4 octet textually (RULED 2026-07-27) -- .50 -> ::50 # -# vault .61 and designate .62 are R11 (a D-020 AMENDMENT, 2026-07-27). Per R6 these -# land BEFORE overlays/dc-ha-scaleup.yaml, which is what gives vault a real HA target. +# vault .61 and designate .62 are R11 (a D-020 AMENDMENT, 2026-07-27). They give +# vault and designate real HA VIP targets. The HA chain itself is in bundle.yaml (BUNDLEFIX-053; the dc-ha-scaleup overlay was retired 2026-08-06). # The B1 / B5 tokens below are DEFINED in bundle.yaml's own header block. applications: keystone: diff --git a/overlays/vr1-dc1-machines.yaml b/overlays/vr1-dc1-machines.yaml index 4fb19bc..385cd8d 100644 --- a/overlays/vr1-dc1-machines.yaml +++ b/overlays/vr1-dc1-machines.yaml @@ -22,9 +22,10 @@ # overlay whole". The two VIP gaps are vault and designate; R11 landed their VIPs # in the per-DC vips overlays (.61 / .62). In ONE command the two overlays merge # before anything deploys, so the scale-up and the VIPs arrive together and the -# "VIPs first" sequence R6 ruled cannot be observed at all. dc-ha-scaleup.yaml is -# applied SEPARATELY, AFTER this set is deployed and settled -- see that file's -# own header. Do not re-merge them. +# "VIPs first" sequence R6 ruled cannot be observed at all. UPDATE 2026-08-06: +# dc-ha-scaleup.yaml has since been RETIRED -- its content was folded into +# bundle.yaml (BUNDLEFIX-053), so there is no separate HA overlay to sequence and +# R6 is superseded. The command above (vips + machines + octavia-pki) is complete. # 2. It used a THIRD model-variable spelling, `${DC1_MODEL}`. The runbooks use # `${DC_MODEL}` (the per-DC model of whichever DC is being deployed); this overlay # is already dc1-scoped by its filename, so the variable must not be too. diff --git a/overlays/vr1-dc1-vips.yaml b/overlays/vr1-dc1-vips.yaml index 6c78b98..a308563 100644 --- a/overlays/vr1-dc1-vips.yaml +++ b/overlays/vr1-dc1-vips.yaml @@ -27,8 +27,8 @@ # the DEDICATED GUA VIP /64, admin+internal their own plane /64s # v6 host part MIRRORS the v4 octet textually (RULED 2026-07-27) -- .50 -> ::50 # -# vault .61 and designate .62 are R11 (a D-020 AMENDMENT, 2026-07-27). Per R6 these -# land BEFORE overlays/dc-ha-scaleup.yaml, which is what gives vault a real HA target. +# vault .61 and designate .62 are R11 (a D-020 AMENDMENT, 2026-07-27). They give +# vault and designate real HA VIP targets. The HA chain itself is in bundle.yaml (BUNDLEFIX-053; the dc-ha-scaleup overlay was retired 2026-08-06). # The B1 / B5 tokens below are DEFINED in bundle.yaml's own header block. applications: keystone: diff --git a/render/values/vr1-dc0-vips.yaml b/render/values/vr1-dc0-vips.yaml index a26f6e3..1e037f3 100644 --- a/render/values/vr1-dc0-vips.yaml +++ b/render/values/vr1-dc0-vips.yaml @@ -123,9 +123,9 @@ # - # vault .61 and designate .62 are R11 (a D-020 AMENDMENT, 2026-07-27). Per R6 these + # vault .61 and designate .62 are R11 (a D-020 AMENDMENT, 2026-07-27). They give - # land BEFORE overlays/dc-ha-scaleup.yaml, which is what gives vault a real HA target. + # vault and designate real HA VIP targets. The HA chain itself is in bundle.yaml (BUNDLEFIX-053; the dc-ha-scaleup overlay was retired 2026-08-06). # The B1 / B5 tokens below are DEFINED in bundle.yaml''s own header block. diff --git a/render/values/vr1-dc1-vips.yaml b/render/values/vr1-dc1-vips.yaml index a91a7e9..5e2ba25 100644 --- a/render/values/vr1-dc1-vips.yaml +++ b/render/values/vr1-dc1-vips.yaml @@ -101,9 +101,9 @@ # - # vault .61 and designate .62 are R11 (a D-020 AMENDMENT, 2026-07-27). Per R6 these + # vault .61 and designate .62 are R11 (a D-020 AMENDMENT, 2026-07-27). They give - # land BEFORE overlays/dc-ha-scaleup.yaml, which is what gives vault a real HA target. + # vault and designate real HA VIP targets. The HA chain itself is in bundle.yaml (BUNDLEFIX-053; the dc-ha-scaleup overlay was retired 2026-08-06). # The B1 / B5 tokens below are DEFINED in bundle.yaml''s own header block. diff --git a/runbooks/dc-dc-phase4-juju-bundle-per-dc.md b/runbooks/dc-dc-phase4-juju-bundle-per-dc.md index c25fac3..6c0fce9 100644 --- a/runbooks/dc-dc-phase4-juju-bundle-per-dc.md +++ b/runbooks/dc-dc-phase4-juju-bundle-per-dc.md @@ -567,11 +567,12 @@ 2026-07-25; both `*-vips.yaml` are RENDERED -- edit `render/values/${DC}-vips.yaml` and re-render, and `tests/render-drift` fails the gauntlet if you do not). -**`overlays/dc-ha-scaleup.yaml` is DELIBERATELY NOT in this command.** Per R6 the -VIPs land BEFORE the HA overlay; applying it is a separate, later step once the -base deploy has settled. That is also why the base deploy legitimately shows -`cluster_count: 1` -- see Step 12's gate, which states when `1` is correct and -when it is the failure. +**There is no separate HA overlay in this command, by design.** The HA chain now +lives IN the base bundle (BUNDLEFIX-053, 2026-08-05); `overlays/dc-ha-scaleup.yaml` +was RETIRED 2026-08-06 (its content folded into `bundle.yaml`; D-121 R6 supersession +note). So this single deploy of base + the per-DC VIP overlay lands 3-unit HA +directly -- `num_units: 3` and `cluster_count: 3` are the EXPECTED post-deploy values +(see Step 12's gate). There is no "apply the overlay later" step to sequence. **Octavia PKI is PER-DC.** Under the R7 amendment (D-109, RULED 2026-07-27) the generator is no longer dc0-frozen and each DC gets its own CA material, so this @@ -1116,14 +1117,18 @@ ```bash juju config -m "$DC_MODEL" -hacluster cluster_count ``` -**GATE, and read the precondition before judging it:** after the Step 4 base -deploy, `cluster_count: 1` is EXPECTED AND CORRECT -- the base bundle pins it -deliberately and `overlays/dc-ha-scaleup.yaml` is excluded from Step 4 per R6 -(`overlays/dc-ha-scaleup.yaml:51-58`). The `3` assertion applies ONLY after the -HA scale-up overlay has been applied; at THAT point a `cluster_count` still -reading `1` behind a 3-unit principal is the failure -- pacemaker will bootstrap -a VIP off a single node and every offline checker stays green. State which side -of the overlay you are on when you record this result. +**GATE (RECONCILED 2026-08-06 to BUNDLEFIX-053; the old two-phase model is RETIRED):** +the base bundle now carries `num_units: 3` + `cluster_count: 3` for the HA apps +DIRECTLY. The former "base deploys at `cluster_count: 1`, then apply +`dc-ha-scaleup.yaml` to reach `3`" sequence is gone -- that overlay was retired +2026-08-06 (its content folded into `bundle.yaml`; see the D-121 R6 supersession +note). So after the Step 4 deploy of base + the per-DC VIP overlay, +`cluster_count: 3` is EXPECTED AND CORRECT for each of the 12 hacluster subs. A +`cluster_count` reading `1` behind a 3-unit principal is the FAILURE -- pacemaker +bootstraps a VIP off a single node (decorative HA) and every offline checker stays +green, which is exactly why `provider-bundle-check.py`'s arity gate (cases +T32-T34) asserts `cluster_count == num_units` on the merged deploy input and +catches the mismatch pre-deploy. **(b) Ceph pool redundancy.** ```bash diff --git a/scripts/cloud-assert.sh b/scripts/cloud-assert.sh index 0ef2fda..63fe2e1 100644 --- a/scripts/cloud-assert.sh +++ b/scripts/cloud-assert.sh @@ -219,7 +219,7 @@ # DISCOVERY is from status JSON alone -- MEASURED shape, asbuilt/20260706-224851/ # juju-status.json: an hacluster subordinate application carries charm-name "hacluster" # and subordinate-to [""]. No app list is hardcoded, so the 14 apps -# overlays/dc-ha-scaleup.yaml scales are covered without naming any of them. +# (scaled 1->3 in bundle.yaml via BUNDLEFIX-053) are covered without naming any of them. # The DECLARED quorum comes from the charm's OWN config via `juju config -m # ` -- the form already in scripts/phase-05-octavia-verify.sh:54-57 and # runbooks/dc-dc-phase4-juju-bundle-per-dc.md:855 (`juju config -m "$DC_MODEL" @@ -254,7 +254,7 @@ done <<<"$HAP" fi # rabbitmq-server clusters NATIVELY (quorum, no hacluster), so the loop above cannot -# see it -- yet it is one of the 14 apps dc-ha-scaleup scales. Its declared quorum is +# see it -- yet it is one of the 14 HA apps (BUNDLEFIX-053). Its declared quorum is # min-cluster-size, MANDATORY at multi-unit per the D-009 AMENDMENT (2026-07-02): # without it the charm accepts client relations BEFORE the cluster forms. if jq -e '.applications["rabbitmq-server"]' >/dev/null 2>&1 <<<"$ST"; then diff --git a/scripts/provider-bundle-check.py b/scripts/provider-bundle-check.py index 67ae506..449d2b3 100644 --- a/scripts/provider-bundle-check.py +++ b/scripts/provider-bundle-check.py @@ -62,9 +62,8 @@ PREFER_IPV6_CHARMS below is the measured authority for 9a/9b. 10. HA ARITY (2026-07-29 gate hardening, the second half of R11's shape): an hacluster subordinate's `cluster_count` MUST equal its principal's `num_units`. Until this - landed, `cluster_count` was asserted NOWHERE in scripts/ or tests/ -- measured - against 20 occurrences in overlays/dc-ha-scaleup.yaml -- so rewriting every one of - those values 3->1 produced a BYTE-IDENTICAL PASS while pacemaker was free to + landed, `cluster_count` was asserted NOWHERE in scripts/ or tests/ -- so rewriting + every declared value 3->1 produced a BYTE-IDENTICAL PASS while pacemaker was free to bootstrap the VIP off a single node: exactly the decorative HA D-121 exists to retire. A MISSING cluster_count also FAILS (an unrecognised state refuses; the charm default is not a substitute for the declared quorum). diff --git a/tests/provider-bundle-check/run-tests.sh b/tests/provider-bundle-check/run-tests.sh index 7bf6ebb..1c93930 100644 --- a/tests/provider-bundle-check/run-tests.sh +++ b/tests/provider-bundle-check/run-tests.sh @@ -234,8 +234,10 @@ # -- was 12 before vault-hacluster moved from the HA overlay into the base. This # DISSOLVES the old T16(pre-overlay=12)/T17(post-overlay=13) ordering distinction: both # read 13 now. T16 on its own asserts the real dc0 deploy input is CLEAN, every one of -# the 13 base-bundle hacluster principals carrying a VIP. (Whether dc-ha-scaleup.yaml is -# now wholly redundant is a Task-#1 post-wave-review finding -- logged, not fixed here.) +# the 13 base-bundle hacluster principals carrying a VIP. dc-ha-scaleup.yaml was found +# WHOLLY REDUNDANT (Task-#1 post-wave review) and RETIRED 2026-08-06 (operator ruling +# "Retire the redundancy and archive"; archived to docs/archive/). Every case that used +# it as a fixture (T17/T17b/T32/T33/T34) is re-pointed onto good.yaml (the base HA chain). run 0 '13 hacluster principal\(s\) all carry a VIP' \ "T16 the real dc0 deploy input is CLEAN (full HA chain in base)" "$TMP/pristine.yaml" @@ -249,19 +251,16 @@ run 1 'hacluster relation but no vip: barbican' \ "T31 the VIP-free base ALONE fails for all clustered principals" "$REPO/bundle.yaml" -# T17 vault is the second R11 gap and it only appears under the HA overlay -- R6 ruled -# the VIPs land BEFORE that overlay, so this is the ordering made executable. -# RE-POINTED 2026-07-29: R6 ruled the VIPs land BEFORE the HA overlay. They have, so -# the ordering is now SATISFIED and this asserts that -- 13 principals once the overlay -# adds vault-hacluster. T17b proves the check still fires if vault's VIP goes missing. -runargs 0 '13 hacluster principal\(s\) all carry a VIP' \ - "T17 R6 ordering satisfied: VIPs precede dc-ha-scaleup" \ - "$TMP/good.yaml" --overlay "$REPO/overlays/dc-ha-scaleup.yaml" - +# T17 (R6 ordering) RETIRED 2026-08-06 with dc-ha-scaleup.yaml. It asserted "13 principals +# carry a VIP" UNDER the overlay; once vault-hacluster moved into the base (BUNDLEFIX-053) +# the overlay added nothing, making T17 IDENTICAL to T18 (same assertion on good.yaml, +# no overlay). T18 below keeps that coverage. T17b is KEPT -- it is the vault-SPECIFIC +# failing direction (T16b covers designate; T31 covers "all principals") -- and RE-POINTED +# off the retired overlay onto good.yaml, whose base HA chain includes vault-hacluster. mutate t17b.yaml 'b["applications"]["vault"]["options"].pop("vip"); b["applications"]["vault"]["options"].pop("prefer-ipv6",None)' -runargs 1 'hacluster relation but no vip: .*vault' \ - "T17b removing vault's VIP under the HA overlay still FAILS" \ - "$TMP/t17b.yaml" --overlay "$REPO/overlays/dc-ha-scaleup.yaml" +run 1 'hacluster relation but no vip: vault' \ + "T17b removing vault's VIP FAILS (vault-specific, on the base HA chain)" \ + "$TMP/t17b.yaml" # T18 the check can also PASS, not only fail: give designate its ruled VIP -> green. run 0 'hacluster principal\(s\) all carry a VIP' \ @@ -382,29 +381,38 @@ --overlay "$TMP/dc1-dual-wrong.yaml" --dc vr1-dc1 # ---- Invariant 10: HA ARITY -- cluster_count == the principal's num_units ---- -# Until 2026-07-29 `cluster_count` was asserted NOWHERE in scripts/ or tests/, against -# 20 occurrences in overlays/dc-ha-scaleup.yaml, so a 3->1 rewrite of every value was a -# BYTE-IDENTICAL PASS. These cases are the proof that is over. +# Until 2026-07-29 `cluster_count` was asserted NOWHERE in scripts/ or tests/, so a 3->1 +# rewrite of every value was a BYTE-IDENTICAL PASS. These cases are the proof that is over. +# The arity now lives in the base bundle (BUNDLEFIX-053), so the fixtures below mutate +# good.yaml directly (2026-08-06 re-point off the retired dc-ha-scaleup.yaml overlay). +# The checker ENUMERATES (one DECORATIVE line per offending sub) with no aggregate/threshold +# path -- MEASURED 2026-08-06: an all-13 rewrite exits 1 with 13 lines, a single-sub rewrite +# exits 1 with 1 line. So the single-sub mutate() fixtures below are equivalent to the old +# whole-overlay rewrite for proving the gate FIRES; they are not a narrowing. -# T32 the GREEN twin first: the real HA overlay declares matching arity for all 13. -runargs 0 '13 hacluster subordinate\(s\) declare cluster_count == principal num_units' \ - "T32 real dc-ha-scaleup overlay: arity matches for all 13 subordinates" \ - "$TMP/good.yaml" --overlay "$REPO/overlays/dc-ha-scaleup.yaml" +# T32 the GREEN twin first: the base HA chain (good.yaml, no overlay) declares matching +# arity for all 13 subordinates. RE-POINTED 2026-08-06 off the retired dc-ha-scaleup.yaml +# -- BUNDLEFIX-053 folded the arity into the base, so good.yaml is now the fixture. +run 0 '13 hacluster subordinate\(s\) declare cluster_count == principal num_units' \ + "T32 base HA chain: arity matches for all 13 subordinates" \ + "$TMP/good.yaml" -# T33 THE decorative-HA fixture: every cluster_count in the shipped overlay rewritten -# 3 -> 1, num_units left at 3. This exact rewrite used to exit 0. -sed 's/cluster_count: 3/cluster_count: 1/g' "$REPO/overlays/dc-ha-scaleup.yaml" > "$TMP/ha-3to1.yaml" -runargs 1 'cluster_count=1 but principal .* num_units=3 -- DECORATIVE HA' \ - "T33 cluster_count 3->1 across the HA overlay FAILS (was byte-identical PASS)" \ - "$TMP/good.yaml" --overlay "$TMP/ha-3to1.yaml" +# T33 THE decorative-HA fixture: a sub's cluster_count driven 3 -> 1 while its principal +# stays num_units=3. RE-POINTED 2026-08-06 onto good.yaml (the base HA chain) via the +# deterministic mutate() helper -- no overlay, and no sed pattern that could silently +# miss and yield a byte-identical fixture (a wrong key path KeyErrors loudly instead). +mutate ha-3to1.yaml 'b["applications"]["keystone-hacluster"]["options"]["cluster_count"]=1' +run 1 'cluster_count=1 but principal .* num_units=3 -- DECORATIVE HA' \ + "T33 a sub cluster_count 3->1 FAILS (decorative HA caught)" \ + "$TMP/ha-3to1.yaml" -# T34 the OTHER direction: a cluster_count of 3 outliving a scale-down to 1 unit. -# Equality, not a floor, is what catches this. -sed 's/^ num_units: 3/ num_units: 1/; s/^ \(.*\):$/ \1:/' "$REPO/overlays/dc-ha-scaleup.yaml" \ - | sed 's/num_units: 3/num_units: 1/g' > "$TMP/ha-scaledown.yaml" -runargs 1 'cluster_count=3 but principal .* num_units=1 -- DECORATIVE HA' \ - "T34 cluster_count 3 outliving a scale-down to 1 FAILS" \ - "$TMP/good.yaml" --overlay "$TMP/ha-scaledown.yaml" +# T34 the OTHER direction: a cluster_count of 3 outliving a principal scale-down to 1. +# Equality, not a floor, is what catches this. RE-POINTED 2026-08-06 onto good.yaml +# via mutate() (deterministic; a wrong key path KeyErrors loudly rather than no-op'ing). +mutate ha-scaledown.yaml 'b["applications"]["keystone"]["num_units"]=1' +run 1 'cluster_count=3 but principal .* num_units=1 -- DECORATIVE HA' \ + "T34 cluster_count 3 outliving a principal scale-down to 1 FAILS" \ + "$TMP/ha-scaledown.yaml" # T35 an ABSENT cluster_count refuses rather than accepting the charm default -- # "could not look" is never "nothing there". Mutated in the BASE, not an overlay: