diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 514bde9..5756446 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2353,12 +2353,30 @@ is the metal PAIR `10.12.8.61 10.12.12.61` (dc1: `10.12.68.61 10.12.72.61`), not the provider/admin/ internal triple; vault-hacluster unblocked. vault is the single non-triple clustered VIP and `provider-bundle-check.py` learned the per-DC exception (band + octet-uniqueness still enforced). - **EVIDENCE GRADE / GAP (GA-R1 rule 2): this 14/14 count is OPERATOR-ATTESTED (2026-08-05 standing - directive) + the part-2 session-ledger close and `changelog-20260805-d121-ha-scaleup.md` -- NOT yet - measurement-backed. A committed `juju status -m vr1-dc0` capture proving all 14 apps at 3 active/idle - units is OWED; it rides with the F4 vault `ha_enabled` live-verify sweep - (`docs/audit/queued-findings-20260805-d121-ha-vault.txt`), the next live-cloud step. Correct this - line from that capture when it lands (C2: measurement wins).** + **EVIDENCE GRADE: 14/14 now MEASUREMENT-BACKED (C2: measurement wins), 2026-08-06. Capture + `docs/audit/stage5-dc0-juju-status-14of14-20260806.txt` (`juju status -m vr1-dc0` from the dc0 rack + per D-138). All 14 D-121-enumerated apps (keystone, glance, nova-cloud-controller, placement, + neutron-api, cinder, ceph-radosgw, openstack-dashboard, octavia, barbican, magnum, designate, + rabbitmq-server, vault; `changelog-20260805-d121-ha-scaleup.md:24`) are at scale=3. HONEST SPLIT + (GA-R6 E3 -- NOT rounded up): 12 at `active/idle`; 2 at scale=3 but workload-`blocked` -- octavia + ("Awaiting end-user execution of `configure-resources` action") and designate ("nameservers must be + set"), both KNOWN deferred-by-design non-HA items (Stage-7 DNS / end-user action), not scale + failures. Model-wide unit tally from the same capture: 152 `active/idle`, 7 `blocked` (the 6 + octavia+designate units + `ceph-rbd-mirror/0` awaiting the dc1 peer), 1 `unknown` + (`glance-simplestreams-sync/0`, its normal state). `mysql-innodb-cluster` ONLINE R/O, can tolerate + one failure.** + **F4 vault `ha_enabled` -- MEASURED 2026-08-06, CONFIRMS D-121 (v-a) as-built:** all 3 vault units + report `vault status` HA Enabled=FALSE, Storage=mysql, unsealed, one shared Cluster ID (same capture + file). CAUSE (measured -- corrects an in-session draft that mis-blamed the mysql backend as + HA-incapable): the vault charm renders `storage "mysql"` with NO `ha_enabled` directive and exposes + NO such config option (its only HA-relevant options are `vip` + `dns-ha-access-record`). Vault's HA + here is the charm's VIP/hacluster model (one unit holds the metal VIP), NOT vault-native + active/standby -- so all 3 units show workload `active: true` and the VIP constrains live traffic to + one. This IS what (v-a) MySQL-backed specified; vault-NATIVE HA (Raft integrated storage) remains + owned by D-068 (already named as the (v-b)/Roosevelt alternative in D-121). No new decision -- + measurement VALIDATES the standing ruling. OPERATOR NOTE (not a gate): three unsealed actives on one + backend is safe only while the hacluster VIP is the sole ingress; revisiting that assumption reopens + under D-068.** OWED: D-121 execution runbook (dc1 reuse); post-wave bundle/overlay review vs the live HA installs (pinned task). Body: `docs/changelog-20260805-d121-ha-scaleup.md`. **NAMED-GATE DEFECT found by measurement -- `phase-03-core-verify.md` Step 3.1 asserts expected diff --git a/docs/audit/stage5-dc0-juju-status-14of14-20260806.txt b/docs/audit/stage5-dc0-juju-status-14of14-20260806.txt new file mode 100644 index 0000000..24fa10c --- /dev/null +++ b/docs/audit/stage5-dc0-juju-status-14of14-20260806.txt @@ -0,0 +1,362 @@ +=== capture host vvr1-dc0 at 2026-08-06T03:11:11Z === +=== juju status -m vr1-dc0 (tabular) === +Model Controller Cloud/Region Version SLA Timestamp +vr1-dc0 vr1-dc0-controller vr1-maas 3.6.27 unsupported 03:11:12Z + +App Version Status Scale Charm Channel Rev Exposed Message +barbican 18.0.0 active 3 barbican 2024.1/stable 265 no Unit is ready +barbican-hacluster 2.1.2 active 3 hacluster 2.4/stable 166 no Unit is ready and clustered +barbican-mysql-router 8.0.46 active 3 mysql-router 8.0/stable 1154 no Unit is ready +barbican-vault 5.0.0 active 3 barbican-vault 2024.1/stable 99 no Unit is ready +ceph-mon 19.2.3 active 3 ceph-mon squid/stable 491 no Unit is ready and clustered +ceph-osd 19.2.3 active 4 ceph-osd squid/stable 953 no Unit is ready (1 OSD) +ceph-radosgw 19.2.3 active 3 ceph-radosgw squid/stable 600 no Unit is ready +ceph-radosgw-hacluster 2.1.2 active 3 hacluster 2.4/stable 166 no Unit is ready and clustered +ceph-rbd-mirror 17.2.9 blocked 1 ceph-rbd-mirror squid/stable 62 no 'ceph-local' incomplete, 'ceph-remote' missing +cinder 24.2.0 active 3 cinder 2024.1/stable 832 no Unit is ready +cinder-backup 24.2.0 active 3 cinder-backup 2024.1/stable 94 no Unit is ready +cinder-ceph 24.2.0 active 3 cinder-ceph 2024.1/stable 568 no Unit is ready +cinder-hacluster 2.1.2 active 3 hacluster 2.4/stable 166 no Unit is ready and clustered +cinder-mysql-router 8.0.46 active 3 mysql-router 8.0/stable 1154 no Unit is ready +dashboard-mysql-router 8.0.46 active 3 mysql-router 8.0/stable 1154 no Unit is ready +designate 18.0.0 blocked 3 designate 2024.1/stable 418 no nameservers must be set +designate-bind 9.18.39 active 1 designate-bind 2024.1/stable 266 no Unit is ready +designate-hacluster 2.1.2 active 3 hacluster 2.4/stable 166 no Unit is ready and clustered +designate-mysql-router 8.0.46 active 3 mysql-router 8.0/stable 1154 no Unit is ready +glance 28.1.0 active 3 glance 2024.1/stable 681 no Unit is ready +glance-hacluster 2.1.2 active 3 hacluster 2.4/stable 166 no Unit is ready and clustered +glance-mysql-router 8.0.46 active 3 mysql-router 8.0/stable 1154 no Unit is ready +glance-simplestreams-sync unknown 1 glance-simplestreams-sync 2024.1/stable 152 no +keystone 25.0.0 active 3 keystone 2024.1/stable 857 no Application Ready +keystone-hacluster 2.1.2 active 3 hacluster 2.4/stable 166 no Unit is ready and clustered +keystone-mysql-router 8.0.46 active 3 mysql-router 8.0/stable 1154 no Unit is ready +magnum 18.0.1 active 3 magnum 2024.1/stable 96 no Unit is ready +magnum-dashboard 14.0.0 active 3 magnum-dashboard 2024.1/stable 122 no Unit is ready +magnum-hacluster 2.1.2 active 3 hacluster 2.4/stable 166 no Unit is ready and clustered +magnum-mysql-router 8.0.46 active 3 mysql-router 8.0/stable 1154 no Unit is ready +memcached active 1 memcached latest/stable 39 no Unit is ready +mysql-innodb-cluster 8.0.46 active 3 mysql-innodb-cluster 8.0/stable 164 no Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure. +ncc-mysql-router 8.0.46 active 3 mysql-router 8.0/stable 1154 no Unit is ready +neutron-api 24.1.0 active 3 neutron-api 2024.1/stable 710 no Unit is ready +neutron-api-hacluster 2.1.2 active 3 hacluster 2.4/stable 166 no Unit is ready and clustered +neutron-api-mysql-router 8.0.46 active 3 mysql-router 8.0/stable 1154 no Unit is ready +neutron-api-plugin-ovn 24.1.0 active 3 neutron-api-plugin-ovn 2024.1/stable 215 no Unit is ready +nova-cloud-controller 29.2.0 active 3 nova-cloud-controller 2024.1/stable 871 no Unit is ready +nova-cloud-controller-hacluster 2.1.2 active 3 hacluster 2.4/stable 166 no Unit is ready and clustered +nova-compute 29.2.0 active 2 nova-compute 2024.1/stable 894 no Unit is ready +octavia 14.0.0 blocked 3 octavia 2024.1/stable 571 no Awaiting end-user execution of `configure-resources` action to create required resources +octavia-dashboard 13.0.0 active 3 octavia-dashboard 2024.1/stable 168 no Unit is ready +octavia-diskimage-retrofit 1.0.1+git8.g... active 1 octavia-diskimage-retrofit 2024.1/stable 257 no Unit is ready +octavia-hacluster 2.1.2 active 3 hacluster 2.4/stable 166 no Unit is ready and clustered +octavia-mysql-router 8.0.46 active 3 mysql-router 8.0/stable 1154 no Unit is ready +openstack-dashboard 24.0.1 active 3 openstack-dashboard 2024.1/stable 750 no Unit is ready +openstack-dashboard-hacluster 2.1.2 active 3 hacluster 2.4/stable 166 no Unit is ready and clustered +ovn-central 22.09.1 active 3 ovn-central 24.03/stable 311 no Unit is ready (northd: active) +ovn-chassis 24.03.2 active 2 ovn-chassis 24.03/stable 396 no Unit is ready +ovn-chassis-octavia 24.03.2 active 3 ovn-chassis 24.03/stable 396 no Unit is ready +placement 11.0.0 active 3 placement 2024.1/stable 154 no Unit is ready +placement-hacluster 2.1.2 active 3 hacluster 2.4/stable 166 no Unit is ready and clustered +placement-mysql-router 8.0.46 active 3 mysql-router 8.0/stable 1154 no Unit is ready +rabbitmq-server 3.9.27 active 3 rabbitmq-server 3.9/stable 295 no Unit is ready and clustered +vault 1.8.8 active 3 vault 1.8/stable 724 no Unit is ready (active: true, mlock: disabled) +vault-hacluster 2.1.2 active 3 hacluster 2.4/stable 166 no Unit is ready and clustered +vault-mysql-router 8.0.46 active 3 mysql-router 8.0/stable 1154 no Unit is ready + +Unit Workload Agent Machine Public address Ports Message +barbican/0* active idle 0/lxd/0 10.12.12.125 9311-9312/tcp Unit is ready + barbican-hacluster/0* active idle 10.12.12.125 Unit is ready and clustered + barbican-mysql-router/0* active idle 10.12.12.125 Unit is ready + barbican-vault/0* active idle 10.12.12.125 Unit is ready +barbican/1 active idle 1/lxd/19 10.12.12.160 9311-9312/tcp Unit is ready + barbican-hacluster/1 active idle 10.12.12.160 Unit is ready and clustered + barbican-mysql-router/1 active idle 10.12.12.160 Unit is ready + barbican-vault/1 active idle 10.12.12.160 Unit is ready +barbican/2 active idle 2/lxd/17 10.12.12.159 9311-9312/tcp Unit is ready + barbican-hacluster/2 active idle 10.12.12.159 Unit is ready and clustered + barbican-mysql-router/2 active idle 10.12.12.159 Unit is ready + barbican-vault/2 active idle 10.12.12.159 Unit is ready +ceph-mon/0 active idle 0/lxd/1 10.12.32.105 Unit is ready and clustered +ceph-mon/1* active idle 1/lxd/0 10.12.32.2 Unit is ready and clustered +ceph-mon/3 active idle 2/lxd/5 10.12.32.106 Unit is ready and clustered +ceph-osd/0 active idle 5 2602:f3e2:f02:10::151 Unit is ready (1 OSD) +ceph-osd/1* active idle 6 2602:f3e2:f02:10::152 Unit is ready (1 OSD) +ceph-osd/2 active idle 7 2602:f3e2:f02:10::150 Unit is ready (1 OSD) +ceph-osd/3 active idle 8 2602:f3e2:f02:10::153 Unit is ready (1 OSD) +ceph-radosgw/1* active idle 0/lxd/20 10.12.12.130 80/tcp Unit is ready + ceph-radosgw-hacluster/0* active idle 10.12.12.130 Unit is ready and clustered +ceph-radosgw/2 active idle 1/lxd/16 10.12.12.149 80/tcp Unit is ready + ceph-radosgw-hacluster/2 active idle 10.12.12.149 Unit is ready and clustered +ceph-radosgw/3 active idle 2/lxd/14 10.12.12.154 80/tcp Unit is ready + ceph-radosgw-hacluster/1 active idle 10.12.12.154 Unit is ready and clustered +ceph-rbd-mirror/0* blocked idle 0/lxd/3 10.12.12.119 'ceph-local' incomplete, 'ceph-remote' missing +cinder/0* active idle 0/lxd/4 10.12.12.124 8776/tcp Unit is ready + cinder-backup/0* active idle 10.12.12.124 Unit is ready + cinder-ceph/0* active idle 10.12.12.124 Unit is ready + cinder-hacluster/0* active idle 10.12.12.124 Unit is ready and clustered + cinder-mysql-router/0* active idle 10.12.12.124 Unit is ready +cinder/2 active idle 2/lxd/8 10.12.12.136 8776/tcp Unit is ready + cinder-backup/1 active idle 10.12.12.136 Unit is ready + cinder-ceph/1 active idle 10.12.12.136 Unit is ready + cinder-hacluster/1 active idle 10.12.12.136 Unit is ready and clustered + cinder-mysql-router/1 active idle 10.12.12.136 Unit is ready +cinder/3 active idle 1/lxd/11 10.12.12.140 8776/tcp Unit is ready + cinder-backup/2 active idle 10.12.12.140 Unit is ready + cinder-ceph/2 active idle 10.12.12.140 Unit is ready + cinder-hacluster/2 active idle 10.12.12.140 Unit is ready and clustered + cinder-mysql-router/2 active idle 10.12.12.140 Unit is ready +designate-bind/0* active idle 2/lxd/1 10.12.12.2 Unit is ready +designate/0* blocked idle 0/lxd/5 10.12.12.110 9001/tcp nameservers must be set + designate-hacluster/0* active idle 10.12.12.110 Unit is ready and clustered + designate-mysql-router/0* active idle 10.12.12.110 Unit is ready +designate/1 blocked idle 1/lxd/13 10.12.12.144 9001/tcp nameservers must be set + designate-hacluster/1 active idle 10.12.12.144 Unit is ready and clustered + designate-mysql-router/1 active idle 10.12.12.144 Unit is ready +designate/2 blocked idle 2/lxd/11 10.12.12.143 9001/tcp nameservers must be set + designate-hacluster/2 active idle 10.12.12.143 Unit is ready and clustered + designate-mysql-router/2 active idle 10.12.12.143 Unit is ready +glance-simplestreams-sync/0* unknown idle 1/lxd/1 10.12.12.105 + octavia-diskimage-retrofit/0* active idle 10.12.12.105 Unit is ready +glance/0* active idle 0/lxd/6 10.12.12.114 9292/tcp Unit is ready + glance-hacluster/0* active idle 10.12.12.114 Unit is ready and clustered + glance-mysql-router/0* active idle 10.12.12.114 Unit is ready +glance/2 active idle 2/lxd/7 10.12.12.133 9292/tcp Unit is ready + glance-hacluster/1 active idle 10.12.12.133 Unit is ready and clustered + glance-mysql-router/1 active idle 10.12.12.133 Unit is ready +glance/3 active idle 1/lxd/10 10.12.12.139 9292/tcp Unit is ready + glance-hacluster/2 active idle 10.12.12.139 Unit is ready and clustered + glance-mysql-router/2 active idle 10.12.12.139 Unit is ready +keystone/0* active idle 0/lxd/7 10.12.12.116 5000/tcp PO: Unit is ready + keystone-hacluster/0* active idle 10.12.12.116 Unit is ready and clustered + keystone-mysql-router/0* active idle 10.12.12.116 Unit is ready +keystone/1 active idle 1/lxd/20 10.12.12.161 5000/tcp PO: Unit is ready + keystone-hacluster/1 active idle 10.12.12.161 Unit is ready and clustered + keystone-mysql-router/1 active idle 10.12.12.161 Unit is ready +keystone/2 active idle 2/lxd/18 10.12.12.162 5000/tcp PO: Unit is ready + keystone-hacluster/2 active idle 10.12.12.162 Unit is ready and clustered + keystone-mysql-router/2 active idle 10.12.12.162 Unit is ready +magnum/0* active idle 0/lxd/8 10.12.12.123 9511/tcp Unit is ready + magnum-hacluster/0* active idle 10.12.12.123 Unit is ready and clustered + magnum-mysql-router/0* active idle 10.12.12.123 Unit is ready +magnum/1 active idle 1/lxd/12 10.12.12.142 9511/tcp Unit is ready + magnum-hacluster/1 active idle 10.12.12.142 Unit is ready and clustered + magnum-mysql-router/1 active idle 10.12.12.142 Unit is ready +magnum/2 active idle 2/lxd/10 10.12.12.141 9511/tcp Unit is ready + magnum-hacluster/2 active idle 10.12.12.141 Unit is ready and clustered + magnum-mysql-router/2 active idle 10.12.12.141 Unit is ready +memcached/0* active idle 0/lxd/9 10.12.12.115 11211/tcp Unit is ready +mysql-innodb-cluster/0 active idle 0/lxd/10 10.12.12.107 Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure. +mysql-innodb-cluster/2* active idle 2/lxd/2 10.12.12.103 Unit is ready: Mode: R/W, Cluster is ONLINE and can tolerate up to ONE failure. +mysql-innodb-cluster/3 active idle 1/lxd/4 10.12.12.126 Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure. +neutron-api/0* active idle 0/lxd/11 10.12.12.111 9696/tcp Unit is ready + neutron-api-hacluster/0* active idle 10.12.12.111 Unit is ready and clustered + neutron-api-mysql-router/0* active idle 10.12.12.111 Unit is ready + neutron-api-plugin-ovn/0* active idle 10.12.12.111 Unit is ready +neutron-api/1 active idle 1/lxd/9 10.12.12.137 9696/tcp Unit is ready + neutron-api-hacluster/1 active idle 10.12.12.137 Unit is ready and clustered + neutron-api-mysql-router/1 active idle 10.12.12.137 Unit is ready + neutron-api-plugin-ovn/1 active idle 10.12.12.137 Unit is ready +neutron-api/2 active idle 2/lxd/9 10.12.12.138 9696/tcp Unit is ready + neutron-api-hacluster/2 active idle 10.12.12.138 Unit is ready and clustered + neutron-api-mysql-router/2 active idle 10.12.12.138 Unit is ready + neutron-api-plugin-ovn/2 active idle 10.12.12.138 Unit is ready +nova-cloud-controller/0* active idle 0/lxd/12 10.12.12.112 8774-8775/tcp Unit is ready + ncc-mysql-router/0* active idle 10.12.12.112 Unit is ready + nova-cloud-controller-hacluster/0* active idle 10.12.12.112 Unit is ready and clustered +nova-cloud-controller/1 active idle 1/lxd/17 10.12.12.155 8774-8775/tcp Unit is ready + ncc-mysql-router/1 active idle 10.12.12.155 Unit is ready + nova-cloud-controller-hacluster/1 active idle 10.12.12.155 Unit is ready and clustered +nova-cloud-controller/2 active idle 2/lxd/15 10.12.12.156 8774-8775/tcp Unit is ready + ncc-mysql-router/2 active idle 10.12.12.156 Unit is ready + nova-cloud-controller-hacluster/2 active idle 10.12.12.156 Unit is ready and clustered +nova-compute/0* active idle 3 2602:f3e2:f02:10::121 Unit is ready + ovn-chassis/0* active idle 2602:f3e2:f02:10::121 Unit is ready +nova-compute/1 active idle 4 2602:f3e2:f02:10::120 Unit is ready + ovn-chassis/1 active idle 2602:f3e2:f02:10::120 Unit is ready +octavia/0* blocked idle 0/lxd/13 10.12.12.113 9876/tcp Awaiting end-user execution of `configure-resources` action to create required resources + octavia-hacluster/0* active idle 10.12.12.113 Unit is ready and clustered + octavia-mysql-router/0* active idle 10.12.12.113 Unit is ready + ovn-chassis-octavia/0* active idle 10.12.12.113 Unit is ready +octavia/1 blocked idle 1/lxd/15 10.12.12.147 9876/tcp Awaiting leader to create required resources + octavia-hacluster/1 active idle 10.12.12.147 Unit is ready and clustered + octavia-mysql-router/1 active idle 10.12.12.147 Unit is ready + ovn-chassis-octavia/1 active idle 10.12.12.147 Unit is ready +octavia/2 blocked idle 2/lxd/13 10.12.12.148 9876/tcp Awaiting leader to create required resources + octavia-hacluster/2 active idle 10.12.12.148 Unit is ready and clustered + octavia-mysql-router/2 active idle 10.12.12.148 Unit is ready + ovn-chassis-octavia/2 active idle 10.12.12.148 Unit is ready +openstack-dashboard/0* active idle 0/lxd/14 10.12.12.108 80,443/tcp Unit is ready + dashboard-mysql-router/0* active idle 10.12.12.108 Unit is ready + magnum-dashboard/0* active idle 10.12.12.108 Unit is ready + octavia-dashboard/0* active idle 10.12.12.108 Unit is ready + openstack-dashboard-hacluster/0* active idle 10.12.12.108 Unit is ready and clustered +openstack-dashboard/1 active idle 1/lxd/14 10.12.12.145 80,443/tcp Unit is ready + dashboard-mysql-router/1 active idle 10.12.12.145 Unit is ready + magnum-dashboard/1 active idle 10.12.12.145 Unit is ready + octavia-dashboard/1 active idle 10.12.12.145 Unit is ready + openstack-dashboard-hacluster/1 active idle 10.12.12.145 Unit is ready and clustered +openstack-dashboard/2 active idle 2/lxd/12 10.12.12.146 80,443/tcp Unit is ready + dashboard-mysql-router/2 active idle 10.12.12.146 Unit is ready + magnum-dashboard/2 active idle 10.12.12.146 Unit is ready + octavia-dashboard/2 active idle 10.12.12.146 Unit is ready + openstack-dashboard-hacluster/2 active idle 10.12.12.146 Unit is ready and clustered +ovn-central/3 active idle 0/lxd/19 10.12.12.129 6641-6642/tcp Unit is ready (northd: active) +ovn-central/4* active idle 1/lxd/5 10.12.12.127 6641-6642/tcp Unit is ready (leader: ovnnb_db, ovnsb_db) +ovn-central/5 active idle 2/lxd/4 10.12.12.128 6641-6642/tcp Unit is ready +placement/0* active idle 0/lxd/16 10.12.12.122 8778/tcp Unit is ready + placement-hacluster/0* active idle 10.12.12.122 Unit is ready and clustered + placement-mysql-router/0* active idle 10.12.12.122 Unit is ready +placement/1 active idle 1/lxd/6 10.12.12.131 8778/tcp Unit is ready + placement-hacluster/2 active idle 10.12.12.131 Unit is ready and clustered + placement-mysql-router/2 active idle 10.12.12.131 Unit is ready +placement/2 active idle 2/lxd/6 10.12.12.132 8778/tcp Unit is ready + placement-hacluster/1 active idle 10.12.12.132 Unit is ready and clustered + placement-mysql-router/1 active idle 10.12.12.132 Unit is ready +rabbitmq-server/0* active idle 0/lxd/17 10.12.12.109 5672,15672/tcp Unit is ready and clustered +rabbitmq-server/1 active idle 1/lxd/18 10.12.12.157 5672,15672/tcp Unit is ready and clustered +rabbitmq-server/2 active idle 2/lxd/16 10.12.12.158 5672,15672/tcp Unit is ready and clustered +vault/0* active idle 0/lxd/18 10.12.12.117 8200/tcp Unit is ready (active: true, mlock: disabled) + vault-hacluster/3 active idle 10.12.12.117 Unit is ready and clustered + vault-mysql-router/0* active idle 10.12.12.117 Unit is ready +vault/1 active idle 1/lxd/21 10.12.12.164 8200/tcp Unit is ready (active: true, mlock: disabled) + vault-hacluster/0* active idle 10.12.12.164 Unit is ready and clustered + vault-mysql-router/1 active idle 10.12.12.164 Unit is ready +vault/2 active idle 2/lxd/19 10.12.12.163 8200/tcp Unit is ready (active: true, mlock: disabled) + vault-hacluster/2 active idle 10.12.12.163 Unit is ready and clustered + vault-mysql-router/2 active idle 10.12.12.163 Unit is ready + +Machine State Address Inst id Base AZ Message +0 started 2602:f3e2:f02:10::100 vr1-dc0-control-01 ubuntu@22.04 default Deployed +0/lxd/0 started 10.12.12.125 juju-d4f45a-0-lxd-0 ubuntu@22.04 default Container started +0/lxd/1 started 10.12.32.105 juju-d4f45a-0-lxd-1 ubuntu@22.04 default Container started +0/lxd/3 started 10.12.12.119 juju-d4f45a-0-lxd-3 ubuntu@22.04 default Container started +0/lxd/4 started 10.12.12.124 juju-d4f45a-0-lxd-4 ubuntu@22.04 default Container started +0/lxd/5 started 10.12.12.110 juju-d4f45a-0-lxd-5 ubuntu@22.04 default Container started +0/lxd/6 started 10.12.12.114 juju-d4f45a-0-lxd-6 ubuntu@22.04 default Container started +0/lxd/7 started 10.12.12.116 juju-d4f45a-0-lxd-7 ubuntu@22.04 default Container started +0/lxd/8 started 10.12.12.123 juju-d4f45a-0-lxd-8 ubuntu@22.04 default Container started +0/lxd/9 started 10.12.12.115 juju-d4f45a-0-lxd-9 ubuntu@22.04 default Container started +0/lxd/10 started 10.12.12.107 juju-d4f45a-0-lxd-10 ubuntu@22.04 default Container started +0/lxd/11 started 10.12.12.111 juju-d4f45a-0-lxd-11 ubuntu@22.04 default Container started +0/lxd/12 started 10.12.12.112 juju-d4f45a-0-lxd-12 ubuntu@22.04 default Container started +0/lxd/13 started 10.12.12.113 juju-d4f45a-0-lxd-13 ubuntu@22.04 default Container started +0/lxd/14 started 10.12.12.108 juju-d4f45a-0-lxd-14 ubuntu@22.04 default Container started +0/lxd/16 started 10.12.12.122 juju-d4f45a-0-lxd-16 ubuntu@22.04 default Container started +0/lxd/17 started 10.12.12.109 juju-d4f45a-0-lxd-17 ubuntu@22.04 default Container started +0/lxd/18 started 10.12.12.117 juju-d4f45a-0-lxd-18 ubuntu@22.04 default Container started +0/lxd/19 started 10.12.12.129 juju-d4f45a-0-lxd-19 ubuntu@22.04 default Container started +0/lxd/20 started 10.12.12.130 juju-d4f45a-0-lxd-20 ubuntu@22.04 default Container started +1 started 2602:f3e2:f02:10::101 vr1-dc0-control-02 ubuntu@22.04 default Deployed +1/lxd/0 started 10.12.32.2 juju-d4f45a-1-lxd-0 ubuntu@22.04 default Container started +1/lxd/1 started 10.12.12.105 juju-d4f45a-1-lxd-1 ubuntu@22.04 default Container started +1/lxd/4 started 10.12.12.126 juju-d4f45a-1-lxd-4 ubuntu@22.04 default Container started +1/lxd/5 started 10.12.12.127 juju-d4f45a-1-lxd-5 ubuntu@22.04 default Container started +1/lxd/6 started 10.12.12.131 juju-d4f45a-1-lxd-6 ubuntu@22.04 default Container started +1/lxd/9 started 10.12.12.137 juju-d4f45a-1-lxd-9 ubuntu@22.04 default Container started +1/lxd/10 started 10.12.12.139 juju-d4f45a-1-lxd-10 ubuntu@22.04 default Container started +1/lxd/11 started 10.12.12.140 juju-d4f45a-1-lxd-11 ubuntu@22.04 default Container started +1/lxd/12 started 10.12.12.142 juju-d4f45a-1-lxd-12 ubuntu@22.04 default Container started +1/lxd/13 started 10.12.12.144 juju-d4f45a-1-lxd-13 ubuntu@22.04 default Container started +1/lxd/14 started 10.12.12.145 juju-d4f45a-1-lxd-14 ubuntu@22.04 default Container started +1/lxd/15 started 10.12.12.147 juju-d4f45a-1-lxd-15 ubuntu@22.04 default Container started +1/lxd/16 started 10.12.12.149 juju-d4f45a-1-lxd-16 ubuntu@22.04 default Container started +1/lxd/17 started 10.12.12.155 juju-d4f45a-1-lxd-17 ubuntu@22.04 default Container started +1/lxd/18 started 10.12.12.157 juju-d4f45a-1-lxd-18 ubuntu@22.04 default Container started +1/lxd/19 started 10.12.12.160 juju-d4f45a-1-lxd-19 ubuntu@22.04 default Container started +1/lxd/20 started 10.12.12.161 juju-d4f45a-1-lxd-20 ubuntu@22.04 default Container started +1/lxd/21 started 10.12.12.164 juju-d4f45a-1-lxd-21 ubuntu@22.04 default Container started +2 started 2602:f3e2:f02:10::102 vr1-dc0-control-03 ubuntu@22.04 default Deployed +2/lxd/1 started 10.12.12.2 juju-d4f45a-2-lxd-1 ubuntu@22.04 default Container started +2/lxd/2 started 10.12.12.103 juju-d4f45a-2-lxd-2 ubuntu@22.04 default Container started +2/lxd/4 started 10.12.12.128 juju-d4f45a-2-lxd-4 ubuntu@22.04 default Container started +2/lxd/5 started 10.12.32.106 juju-d4f45a-2-lxd-5 ubuntu@22.04 default Container started +2/lxd/6 started 10.12.12.132 juju-d4f45a-2-lxd-6 ubuntu@22.04 default Container started +2/lxd/7 started 10.12.12.133 juju-d4f45a-2-lxd-7 ubuntu@22.04 default Container started +2/lxd/8 started 10.12.12.136 juju-d4f45a-2-lxd-8 ubuntu@22.04 default Container started +2/lxd/9 started 10.12.12.138 juju-d4f45a-2-lxd-9 ubuntu@22.04 default Container started +2/lxd/10 started 10.12.12.141 juju-d4f45a-2-lxd-10 ubuntu@22.04 default Container started +2/lxd/11 started 10.12.12.143 juju-d4f45a-2-lxd-11 ubuntu@22.04 default Container started +2/lxd/12 started 10.12.12.146 juju-d4f45a-2-lxd-12 ubuntu@22.04 default Container started +2/lxd/13 started 10.12.12.148 juju-d4f45a-2-lxd-13 ubuntu@22.04 default Container started +2/lxd/14 started 10.12.12.154 juju-d4f45a-2-lxd-14 ubuntu@22.04 default Container started +2/lxd/15 started 10.12.12.156 juju-d4f45a-2-lxd-15 ubuntu@22.04 default Container started +2/lxd/16 started 10.12.12.158 juju-d4f45a-2-lxd-16 ubuntu@22.04 default Container started +2/lxd/17 started 10.12.12.159 juju-d4f45a-2-lxd-17 ubuntu@22.04 default Container started +2/lxd/18 started 10.12.12.162 juju-d4f45a-2-lxd-18 ubuntu@22.04 default Container started +2/lxd/19 started 10.12.12.163 juju-d4f45a-2-lxd-19 ubuntu@22.04 default Container started +3 started 2602:f3e2:f02:10::121 vr1-dc0-compute-02 ubuntu@22.04 default Deployed +4 started 2602:f3e2:f02:10::120 vr1-dc0-compute-01 ubuntu@22.04 default Deployed +5 started 2602:f3e2:f02:10::151 vr1-dc0-storage-02 ubuntu@22.04 default Deployed +6 started 2602:f3e2:f02:10::152 vr1-dc0-storage-03 ubuntu@22.04 default Deployed +7 started 2602:f3e2:f02:10::150 vr1-dc0-storage-01 ubuntu@22.04 default Deployed +8 started 2602:f3e2:f02:10::153 vr1-dc0-storage-04 ubuntu@22.04 default Deployed + +=== F4 vault ha_enabled live-verify (per-unit vault status), captured 2026-08-06T03:13:58Z === +Source: ssh vr1-dc0-rack -> juju ssh -m vr1-dc0 vault/{0,1,2} -> vault status (unauthenticated, non-secret) +vault/0: Sealed=false Storage=mysql HA_Enabled=FALSE Cluster=vault-cluster-12969e82 ID=3a37fd77-...80d4 +vault/1: Sealed=false Storage=mysql HA_Enabled=FALSE Cluster=vault-cluster-12969e82 ID=3a37fd77-...80d4 +vault/2: Sealed=false Storage=mysql HA_Enabled=FALSE Cluster=vault-cluster-12969e82 ID=3a37fd77-...80d4 + +FINDING (F4 v-a): MySQL storage backend => Vault-native HA Enabled=false on all 3 units. + All three report workload 'active: true' because none does Vault leader-election; + redundancy is charm/VIP-level (vault-hacluster picks ONE unit for the metal VIP). + Routes the Vault-native-HA question to D-068 (Raft integrated storage) per the sweep. + +=== 14/14 D-121 HA apps -- app status + scale (from tabular above) === +ALL 14 at scale=3: barbican(a) ceph-radosgw(a) cinder(a) designate(BLOCKED:nameservers) glance(a) + keystone(a) magnum(a) neutron-api(a) nova-cloud-controller(a) octavia(BLOCKED:configure-resources) + openstack-dashboard(a) placement(a) vault(a) rabbitmq-server(a-clustered). + 13 hacluster subordinates all active/clustered + rabbitmq native cluster = 14 HA apps. + designate/octavia BLOCKED reasons are known non-HA items (Stage-7 DNS / end-user action). + +=== RAW: vault status per unit, verbatim stdout, captured 2026-08-06T04:04:50Z === +----- vault/0 ----- +Key Value +--- ----- +Seal Type shamir +Initialized true +Sealed false +Total Shares 5 +Threshold 3 +Version 1.8.8 +Storage Type mysql +Cluster Name vault-cluster-12969e82 +Cluster ID 3a37fd77-be3a-c78d-f2f0-1a658e2680d4 +HA Enabled false +----- vault/1 ----- +Key Value +--- ----- +Seal Type shamir +Initialized true +Sealed false +Total Shares 5 +Threshold 3 +Version 1.8.8 +Storage Type mysql +Cluster Name vault-cluster-12969e82 +Cluster ID 3a37fd77-be3a-c78d-f2f0-1a658e2680d4 +HA Enabled false +----- vault/2 ----- +Key Value +--- ----- +Seal Type shamir +Initialized true +Sealed false +Total Shares 5 +Threshold 3 +Version 1.8.8 +Storage Type mysql +Cluster Name vault-cluster-12969e82 +Cluster ID 3a37fd77-be3a-c78d-f2f0-1a658e2680d4 +HA Enabled false + +=== RAW: rendered vault.hcl HA-relevant lines (grep, no credential lines) === +3:storage "mysql" { + +=== F8 ceph-radosgw verify, 2026-08-06T04:08:21Z -- RESOLVED === +App active scale=3 'Unit is ready'; units ceph-radosgw/1*,2,3 all active/idle 'Unit is ready' 80/tcp. +The prior 'Services not running: radosgw' was the new-unit stale-status class; converged. No action. + +=== F9 staging drift, 2026-08-06T04:08:21Z -- MEASURED (re-stage OWED, gated) === +repo HEAD 63aa638: bundle.yaml=20ca73a6 vr1-dc0-vips=3f403408 vr1-dc1-vips=28cd31bb +dc0 rack ~/repo-stage: bundle.yaml=213b51ba STALE; overlays/vr1-dc0-vips.yaml=cf42b937 STALE; vr1-dc1-vips.yaml ABSENT (correct, dc0 rack). +dc1 rack: NO ~/repo-stage (dc1 HELD/undeployed) -- dc1 overlay staging is future standup work.