Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260805-d121-ha-scaleup.md
@JANeumatrix JANeumatrix 1 day ago 5 KB BUNDLEFIX-053: bundle.yaml to full 3-unit HA (D-121); reverse BUNDLEFIX-002 vault de-HA
Raw Blame History

Changelog 2026-08-05 (D-121 HA scale-up: bundle + live Wave 1) -- IN PROGRESS

Session changelog. Branch dc-dc-stage5-preconditions. DISCLOSURE (GA-R2/D1): third same-day changelog in this continuous session, kept separate as a distinct work-stream from changelog-20260805-skill-close-convention-docfix.md and changelog-20260805-stage5-dc0-ceph-remediation.md (both committed). Flagged for the close bookend to cite all three. ledger-scan at this work's open: 4 decisions, 29 SEC, next-free BUNDLEFIX-053. Status claims live in docs/CURRENT-STATE.md ONLY.

Scope: execute D-121 ("VR1 makes HA real -- scale the decorative single-unit control plane to 3") on vr1-dc0, per operator ruling 2026-08-05 ("we have superseded the HA removal from all items ... stand up all remaining HA apps ... HA is a blocker for deployment sign off" + "testing as if this is Roosevelt"). Two halves: the running cloud (live add-unit) and the bundle-of-record (so dc1/Roosevelt reproduce it).

Item 1 -- BUNDLEFIX-053: bundle.yaml + comments updated to the full 3-unit HA posture

WHAT. bundle.yaml reversed from the D-009 "decorative single-unit" posture to real 3-unit HA. Applied via an operator-approved one-off script (repetitive lines) + hand-Edits (unique lines); scripted in-place edit of the tracked bundle is otherwise classifier-walled (CLAUDE.md caveman guardrail).

  • num_units: 1 -> 3 on the 14 HA apps: keystone, glance, nova-cloud-controller, placement, neutron-api, cinder, ceph-radosgw, openstack-dashboard, octavia, barbican, magnum, designate, rabbitmq-server, vault. to: [lxd:0] -> [lxd:0, lxd:1, lxd:2] (one-per-control-node).
  • options: {cluster_count: 1 -> 3} on the 12 hacluster subordinates.
  • vault-hacluster uncommented/added (cluster_count:3, bindings mirror the others) + relation [vault:ha, vault-hacluster:ha] uncommented -- the BUNDLEFIX-002 reversal, governed by D-121 (v-a) (MySQL-backed vault HA, re-add vault:ha, scale to 3).
  • rabbitmq-server gains options: {min-cluster-size: 3} (D-009 amendment 2026-07-02: rabbitmq scale-up is not mechanical without it).
  • Governing comments reconciled (D-009/BUNDLEFIX-002/003 single-unit notes -> D-121 built).
  • Overlays UNCHANGED (VIPs, incl. vault .61, already set).

VERIFICATION (this is the part the operator gated -- "verified against online docs?"). Each change checked against the charm/juju docs or the deployed charm, not assumed:

  • hacluster cluster_count: charmhub -- int, default 3, "best practice to set to the expected number of units ... with 3+ units quorum is enforced and services stop on quorum loss."
  • rabbitmq min-cluster-size: charmhub -- "Minimum number of units expected to exist before charm will attempt to form a rabbitmq cluster."
  • vault ha: the DEPLOYED vault charm exposes an ha endpoint bound metal-internal, identical to keystone (a known-good hacluster user).
  • vault MySQL-backend HA (D-121 (v-a) gate): HashiCorp docs -- "The MySQL storage backend supports high availability" (ha_enabled + lock_table). Caveat: MySQL lock lost on connection death -> possible leader flap; mitigate via interactive_timeout/wait_timeout.
  • num_units:3 / to:[0,1,2]: standard juju, proven live this session.

VALIDATED. provider-bundle-check.py --dc vr1-dc0 PASS (exit 0) -- notably "13 hacluster subordinate(s) declare cluster_count == principal num_units", "13 hacluster principal(s) all carry a VIP", "109 relations well-formed". repo-lint 0 fail.

Revert. git revert this commit (bundle change is 47+/46-). Restores the decorative single-unit posture -- do NOT without re-opening D-121, or the bundle diverges from the built HA cloud.

Item 2 -- D-121 live execution on vr1-dc0 (Wave 1) -- IN PROGRESS

Procedure (pilot-proven on placement): per app, juju add-unit <app> -n 2 --to lxd:1,lxd:2 -> juju config <app>-hacluster cluster_count=3 -> converge (3-node crm, VIPs Started, units active). Each an individual gated command (the classifier enforces "never batch destructive steps").

Findings (live):

  • apt-cacher-ng concurrency limit: 6 containers at once (3-app batch) hung 2 in apt-get update CLOSE-WAIT (today's outage class). Remediated per appendix-A (remove-unit -> remove-machine --force -> re-add) and switched to one app / 2 containers at a time -- clean thereafter. A transient cw>0 during active apt is normal; only a persistent stall is the hang.
  • transient VIP outage on hacluster scale-up: adding the 2 new hacluster units while cluster_count is still 1 briefly STOPS the leader's VIP resource; it recovers once cluster_count=3 reforms the cluster (~1-2 min). Harmless for stateless API apps; sequence keystone/vault deliberately.

Wave 1 (8 apps) status at close of this entry: neutron-api, glance, cinder, magnum, designate, openstack-dashboard, octavia, ceph-radosgw scaled to 3 with 3-node hacluster (designate/octavia remain workload-blocked on their DEFERRED-by-design config -- Stage-7 DNS / D-021 octavia enablement -- NOT a scale problem). Live status authority: CURRENT-STATE.

Revert. Live-cloud; not a git revert. To undo, remove-unit each app back to 1.

Owed (not done this entry)

  • Waves 2-4: keystone + nova-cloud-controller (auth-path, VIP-blip); rabbitmq (min-cluster-size handling); vault LAST -- operator unseals each new unit (SEC-003, guard-hook-blocked for the agent); cert-SAN check before the VIP flip.
  • D-121 execution runbook (hard-rule-4 gap -- needed verbatim for dc1).
  • CURRENT-STATE "D-121 (v-a) now BUILT 2026-08-05" note (this commit).