SWEEP -- 2026-08-05 session: D-121 HA scale-up (14/14) + vault metal-only (D-020 amendment) ========================================================================================== Method (ruled 2026-07-31): enumerate every finding from the session, then grep each against the repo. FIRST SURFACE = exists only in this transcript and would be lost on a clear. This was a very large session; the committed changelogs capture the bulk -- this file leads with what they do NOT. == FIRST SURFACE (transcript-only until this file) ========================================= F1. OWN-MISTAKE: BUNDLEFIX-053 (bundle.yaml num_units 1->3, committed ae06c72) was SHIPPED WITHOUT RUNNING ITS HARNESS. I ran `provider-bundle-check.py` directly (PASS) but not `tests/provider-bundle-check/run-tests.sh`. The num_units:3 change broke the harness's rolesep fixture builder (lines ~157-163 set to:[lxd:0] single-placement while base now has num_units:3 -> placement/anti-affinity FAIL: T9). Latent until 2026-08-05's vault work re-ran the harness. Remediation = Task #2. Lesson: a BUNDLE change breaks the CHECK's harness even when the check script is untouched -- run the harness on any bundle edit. F2. provider-bundle-check HARNESS IS RED (6 cases: T9,T14,T16,T19,T21,T45) as of session close. Two causes: (a) F1 above; (b) the D-020 vault-metal-only amendment -- dc1 overlay still has vault as a triple (T14), and the VIP-count assertions expect 13 provider/admin/internal triples, now 12 (T16/T19/T21/T45). Reconciliation + commit = Task #2. repo-lint also FAILS (L10) while the amendment sits uncommitted in the working tree. F3. D-020 AMENDMENT (2026-08-05, vault METAL-ONLY) is RATIFIED but UNCOMMITTED. Operator utterances (GA-R5 fidelity): "(a) draft the amendment" then "Ratify as written". The amendment TEXT is written in docs/design-decisions.md (working tree, on disk, uncommitted); overlays/vr1-dc0-vips.yaml (vault -> "10.12.8.61 10.12.12.61") and scripts/provider-bundle-check.py (vault metal-only exception) are the applied edits, also uncommitted. LIVE change DONE: `juju config vault vip="10.12.8.61 10.12.12.61"` -> the dead provider VIP dropped, vault-hacluster UNBLOCKED (active, 2 metal VIPs Started). This is NOT durable until Task #2 commits + pushes it. F4. vault `ha_enabled` / the D-121 (v-a) NATIVE leader-election question is UNVERIFIED at close. Measured earlier: all 3 vault units `HA Enabled=false`, vault.hcl had NO ha_enabled. The hacluster was blocked then (on the provider VIP); it is now unblocked. Whether the charm now sets ha_enabled (true MySQL leader-election) vs. only VIP-level redundancy is NOT re-checked. The (v-a) gate says: if no MySQL leader-election, evaluate Raft/route to D-068. OWED: re-measure `vault status | grep 'HA Enabled'` on the 3 units now that hacluster is up. F5. keystone SCALE-UP CAUSED NO AUTH OUTAGE -- and the reason is a runbook-worthy sequencing fact: setting hacluster `cluster_count=3` PROACTIVELY (as the new subs join), rather than reactively after they cluster, avoided the transient VIP-stop that hit Wave 1. Measured: keystone VIP 10.12.12.50:5000 held HTTP 200 across the reform. Belongs in the D-121 execution runbook (owed). F6. vault API is PLAINTEXT (`tls_disable = 1` on the [::]:8200 and 127.0.0.1:8220 listeners). There is NO vault API TLS cert -- so the VIP flip had no SAN-mismatch risk (the cert-SAN prerequisite advisor flagged was moot). vault's consumers reach it over HTTP on the internal plane; the certificates relation passes data via the juju databag. F7. Multi-unit vault-on-MySQL bring-up shape (for the runbook): new units come up `Initialized=false` UNTIL their vault-mysql-router finishes "Waiting for proxied DB creation from cluster" -- then flip to `Initialized=true, Sealed=true` (shared `database="vault"`). They need UNSEAL (same 3-of-5 shares as vault/0), NEVER init. Operator unsealed vault/1,2 (Sealed=false confirmed). Deploying vault-hacluster is a FRESH `juju deploy hacluster vault-hacluster` + `juju integrate` (juju 3.x -- NOT `add-relation`). F8. ceph-radosgw reports workload "Services not running: radosgw" after its 3-unit scale-up (HA structure IS formed: crm 3 nodes, VIPs Started). Likely the new-unit stale-status class (single-unit radosgw showed the same and self-cleared earlier). OWED a verify. F9. dc0 rack ~/repo-stage/bundle.yaml MAY BE STALE relative to BUNDLEFIX-053 (num_units:3 + vault-hacluster). Not re-staged this session (no rack redeploy needed -- the scale-up was live `juju add-unit`). Re-stage + sha256-verify before any rack-side redeploy. == ALREADY ON SURFACE (grep-verified) ===================================================== - Ceph apt CLOSE-WAIT remediation + capture -> committed 71c5b97, docs/changelog-20260805-stage5-dc0-ceph-remediation.md, appendix-A entry, CURRENT-STATE. - Bundle 3-unit HA (BUNDLEFIX-053) + charm-doc verification -> committed ae06c72, docs/changelog-20260805-d121-ha-scaleup.md, CURRENT-STATE. - Skill close-convention DOCFIX-209 -> committed 33f9158, docs/changelog-20260805-skill-close-convention-docfix.md. - D-121 14/14 live status + apt-concurrency + transient-VIP findings -> CURRENT-STATE (4392118 says 12/14; the 14/14 update rides with Task #2's amendment commit -- status LAG noted). - vault-is-internal-only / magnum:certificates-is-service-cert / tenant-path-is-Barbican -> recorded in the D-020 amendment text (F3, uncommitted). == ALWAYS-SWEEP FIVE ======================================================================= 1. Gitignored state: none created this session. .claude/settings.local.json unchanged. 2. Dangling refs: the two committed changelogs + appendix-A cite paths that resolve. The UNCOMMITTED amendment cites the overlay/check edits (present, uncommitted). 3. Ruling fidelity: D-020 amendment 2026-08-05 utterances quoted verbatim (F3). D-121 execution authorised by the 2026-08-05 "stand up all remaining HA apps" + "testing as if Roosevelt" directives (recorded in CURRENT-STATE's D-121 block). 4. As-executed log: NO run-logged.sh window opened this session (live shell). Every mutation is in this transcript + the committed changelogs/CURRENT-STATE; the vault unseals were operator-run (secret custody). This file + the changelogs are the record; there is no script(1) capture. 5. Contradiction detector: CURRENT-STATE (committed) says 12/14; live is 14/14 -- corrected by Task #2. bundle.yaml (committed) says vault vip triple via overlay; the overlay is now metal-only (uncommitted). provider-bundle-check "13 clustered VIPs" is now 12. == TASKS OPEN AT CLOSE ===================================================================== #1 Post-wave bundle/overlay review vs live HA installs (should also fold the vault metal-only reconciliation). #2 Reconcile provider-bundle-check harness + commit the D-020 amendment (F1/F2/F3) + CURRENT-STATE.