# Changelog 2026-08-05 (D-121 HA scale-up: bundle + live Wave 1) -- IN PROGRESS

Session changelog. Branch `dc-dc-stage5-preconditions`. DISCLOSURE (GA-R2/D1): third
same-day changelog in this continuous session, kept separate as a distinct work-stream
from `changelog-20260805-skill-close-convention-docfix.md` and
`changelog-20260805-stage5-dc0-ceph-remediation.md` (both committed). Flagged for the close
bookend to cite all three. `ledger-scan` at this work's open: 4 decisions, 29 SEC, next-free
BUNDLEFIX-053. Status claims live in `docs/CURRENT-STATE.md` ONLY.

Scope: execute **D-121** ("VR1 makes HA real -- scale the decorative single-unit control
plane to 3") on vr1-dc0, per operator ruling 2026-08-05 ("we have superseded the HA removal
from all items ... stand up all remaining HA apps ... HA is a blocker for deployment sign
off" + "testing as if this is Roosevelt"). Two halves: the running cloud (live add-unit) and
the bundle-of-record (so dc1/Roosevelt reproduce it).

---

## Item 1 -- BUNDLEFIX-053: bundle.yaml + comments updated to the full 3-unit HA posture

**WHAT.** `bundle.yaml` reversed from the D-009 "decorative single-unit" posture to real
3-unit HA. Applied via an operator-approved one-off script (repetitive lines) + hand-Edits
(unique lines); scripted in-place edit of the tracked bundle is otherwise classifier-walled
(CLAUDE.md caveman guardrail).
- `num_units: 1 -> 3` on the 14 HA apps: keystone, glance, nova-cloud-controller, placement,
  neutron-api, cinder, ceph-radosgw, openstack-dashboard, octavia, barbican, magnum,
  designate, rabbitmq-server, vault. `to: [lxd:0] -> [lxd:0, lxd:1, lxd:2]` (one-per-control-node).
- `options: {cluster_count: 1 -> 3}` on the 12 hacluster subordinates.
- **vault-hacluster** uncommented/added (cluster_count:3, bindings mirror the others) +
  relation `[vault:ha, vault-hacluster:ha]` uncommented -- the **BUNDLEFIX-002 reversal**,
  governed by **D-121 (v-a)** (MySQL-backed vault HA, re-add vault:ha, scale to 3).
- `rabbitmq-server` gains `options: {min-cluster-size: 3}` (**D-009 amendment 2026-07-02**:
  rabbitmq scale-up is not mechanical without it).
- Governing comments reconciled (D-009/BUNDLEFIX-002/003 single-unit notes -> D-121 built).
- Overlays UNCHANGED (VIPs, incl. vault .61, already set).

**VERIFICATION (this is the part the operator gated -- "verified against online docs?").**
Each change checked against the charm/juju docs or the deployed charm, not assumed:
- hacluster `cluster_count`: charmhub -- int, default 3, *"best practice to set to the
  expected number of units ... with 3+ units quorum is enforced and services stop on quorum
  loss."*
- rabbitmq `min-cluster-size`: charmhub -- *"Minimum number of units expected to exist before
  charm will attempt to form a rabbitmq cluster."*
- vault `ha`: the DEPLOYED vault charm exposes an `ha` endpoint bound `metal-internal`,
  identical to keystone (a known-good hacluster user).
- vault MySQL-backend HA (D-121 (v-a) gate): HashiCorp docs -- *"The MySQL storage backend
  supports high availability"* (`ha_enabled` + `lock_table`). Caveat: MySQL lock lost on
  connection death -> possible leader flap; mitigate via `interactive_timeout`/`wait_timeout`.
- `num_units:3` / `to:[0,1,2]`: standard juju, proven live this session.

**VALIDATED.** `provider-bundle-check.py --dc vr1-dc0` **PASS (exit 0)** -- notably
"13 hacluster subordinate(s) declare cluster_count == principal num_units", "13 hacluster
principal(s) all carry a VIP", "109 relations well-formed". `repo-lint` 0 fail.

**Revert.** `git revert` this commit (bundle change is 47+/46-). Restores the decorative
single-unit posture -- do NOT without re-opening D-121, or the bundle diverges from the
built HA cloud.

## Item 2 -- D-121 live execution on vr1-dc0 (Wave 1) -- IN PROGRESS

**Procedure (pilot-proven on `placement`):** per app, `juju add-unit <app> -n 2 --to
lxd:1,lxd:2` -> `juju config <app>-hacluster cluster_count=3` -> converge (3-node crm, VIPs
Started, units active). Each an individual gated command (the classifier enforces
"never batch destructive steps").

**Findings (live):**
- **apt-cacher-ng concurrency limit:** 6 containers at once (3-app batch) hung 2 in
  `apt-get update` CLOSE-WAIT (today's outage class). Remediated per appendix-A (remove-unit
  -> remove-machine --force -> re-add) and switched to **one app / 2 containers at a time** --
  clean thereafter. A *transient* cw>0 during active apt is normal; only a persistent stall
  is the hang.
- **transient VIP outage on hacluster scale-up:** adding the 2 new hacluster units while
  cluster_count is still 1 briefly STOPS the leader's VIP resource; it recovers once
  cluster_count=3 reforms the cluster (~1-2 min). Harmless for stateless API apps; sequence
  keystone/vault deliberately.

**Wave 1 (8 apps) status at close of this entry:** neutron-api, glance, cinder, magnum,
designate, openstack-dashboard, octavia, ceph-radosgw scaled to 3 with 3-node hacluster
(designate/octavia remain workload-`blocked` on their DEFERRED-by-design config -- Stage-7
DNS / D-021 octavia enablement -- NOT a scale problem). Live status authority: CURRENT-STATE.

**Revert.** Live-cloud; not a git revert. To undo, remove-unit each app back to 1.

## Owed (not done this entry)
- **Waves 2-4:** keystone + nova-cloud-controller (auth-path, VIP-blip); rabbitmq
  (min-cluster-size handling); **vault LAST -- operator unseals each new unit (SEC-003,
  guard-hook-blocked for the agent); cert-SAN check before the VIP flip**.
- **D-121 execution runbook** (hard-rule-4 gap -- needed verbatim for dc1).
- CURRENT-STATE "D-121 (v-a) now BUILT 2026-08-05" note (this commit).