diff --git a/bundle.yaml b/bundle.yaml index da1817c..da4028a 100644 --- a/bundle.yaml +++ b/bundle.yaml @@ -683,13 +683,16 @@ '': metal-admin application-dashboard: metal-internal certificates: metal-internal - # BUNDLEFIX-011 / D-072 (2026-07-05): cluster on metal-admin, NOT metal-internal. - # The charm renders haproxy's 443 backend on the cluster-binding address but only - # creates apache SSL vhosts for the default-binding + public-binding addresses -- - # cluster on metal-internal = TLS backend on a vhost-less address = VIP https dead - # (L4 check masks it). metal-admin matches the default binding so the backend - # lands on a served vhost. Do NOT use provider-public (tenant-facing plane). - cluster: metal-admin + # BUNDLEFIX-057 / D-072 AMENDMENT (VR1) 2026-08-06: cluster RETURNS to metal-internal. + # The charm renders haproxy's 443 backend on the cluster-binding address; apache serves + # its SSL vhost on the plane get_network_addresses() resolves. In VR0 (single metal + # plane) that was metal-admin, so D-072/BUNDLEFIX-011 set cluster=metal-admin. In VR1 + # (metal split into admin+internal) this charm declares no admin/internal extra-binding + # and no os-*-network, so apache serves metal-INTERNAL (measured; charm-logged) -- so + # cluster=metal-admin became the vhost-less-backend trap D-072 warned of, INVERTED. + # metal-internal is now both the served plane AND the generic HA/cluster-peer rule (14 + # apps). Proven live 2026-08-06. Do NOT use provider-public (tenant-facing plane). + cluster: metal-internal dashboard: metal-internal dashboard-plugin: metal-internal ha: metal-internal diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 1330f3b..aa2e9d6 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2398,8 +2398,9 @@ IP-only; keystone VIP two-sourced (overlay 10.12.4.50 == live public endpoint); vault root CA TLS to keystone OK. **F-CV2 RESOLVED:** openstack CLI was ABSENT on the dc0 rack (07-30 queued-F1, HIT at Step 7) -> installed `python3-openstackclient 6.6.0-0ubuntu2` on the rack (see section 7 row). - **EXIT GATE OPEN on (a) F-CV3: dashboard VIP 10.12.4.58:443 serves PLAINTEXT (apache-SSL-inactive - despite certs present) -- Horizon exit-gate FAILS; and (b) Step 3.4 domain-manager policy NOT RUN. + **EXIT GATE: (a) F-CV3 dashboard VIP TLS -- RESOLVED 2026-08-06 (D-072 VR1 amendment / + BUNDLEFIX-057, see below); (b) Step 3.4 domain-manager policy STILL NOT RUN -- the one remaining + phase-03 exit-gate item. F-CV1: designate _admin backend DOWN -- **RESOLVED 2026-08-06 (BUNDLEFIX-056, operator-approved fix EXECUTED + VERIFIED).** Root cause (governing = D-052 + generic binding rule + the D-020 amendment's ruled .62 triple, NOT D-141): designate's bundle bindings OMITTED public + internal, @@ -2412,12 +2413,19 @@ span all 3 planes; cert reissued to cover provider-public; catalog shows the correct triple (public .4.62 / internal .12.62 / admin .8.62). provider-bundle-check EXPECT_PUBLIC_VIP 11->12 (+T16c/T16d failing-direction), binding-matrix reference row updated. gauntlet ALL GREEN 99.** - F-CV1 and F-CV3 are TWO SEPARATE findings; F-CV3 (dashboard = charm https frontend not effective, - bindings ARE correct) is STILL OPEN and needs its own triage session. Sweep: - docs/audit/queued-findings-20260806-phase03-coreverify.txt. ACCESS-MODEL CONTEXT added 2026-08-06 - (dc-dc-deployment-workflow.md gap-21): operators reach the dashboard via the metal-admin VIP over - the tailnet -> argues the F-CV3 fix should serve TLS on metal-admin (option A), not metal-internal - (option B, which reverses ruled D-072). A-vs-B resolution is the next F-CV3 step. + F-CV1 and F-CV3 are TWO SEPARATE findings. **F-CV3 (dashboard VIP TLS) -- RESOLVED 2026-08-06 + (D-072 AMENDMENT (VR1) ratified GA-R5 "Ratified, land the config-of-record" + BUNDLEFIX-057).** + Root cause: VR1 split-metal INVERTS the D-072 cluster placement -- the openstack-dashboard charm + declares no admin/internal extra-binding + no os-*-network (metadata + charmhub docs), so apache + serves its SSL vhost on metal-INTERNAL, while haproxy dialed cluster=metal-admin -> vhost-less -> + plaintext (the D-072 trap, inverted). Option A (serve metal-admin) unavailable in-deployment (no + charm lever). FIX (proven LIVE before ratifying, per operator process directive): retire the VR0 + exception, `openstack-dashboard cluster -> metal-internal` (generic rule + the served plane). + VERIFIED live: haproxy :433 backends moved to metal-internal; provider VIP 10.12.4.58 AND operator + metal-admin VIP 10.12.8.58 both TLS 200 CA-verified (were plaintext); reissued cert covers all 3 + VIP IPs incl 10.12.8.58 (resolves AH01909); 3 units active/idle. Config-of-record: bundle cluster + metal-internal, binding-reference exception RETIRED. dc1 inherits it via the shared bundle (no + rebind). Sweep: docs/audit/queued-findings-20260806-phase03-coreverify.txt. **NAMED-GATE DEFECT found by measurement -- `phase-03-core-verify.md` Step 3.1 asserts expected non-active/idle = 1 (octavia only); the VR1 roster yields 4 deferred-by-design + gss.** That gate is STALE for VR1 and a DOCFIX is owed (also owed on that runbook: `-m openstack` -> `-m vr1-dc0` diff --git a/docs/audit/queued-findings-20260806-phase03-coreverify.txt b/docs/audit/queued-findings-20260806-phase03-coreverify.txt index a83df4b..d09211d 100644 --- a/docs/audit/queued-findings-20260806-phase03-coreverify.txt +++ b/docs/audit/queued-findings-20260806-phase03-coreverify.txt @@ -30,7 +30,14 @@ Section-4 table (network-space-binding-reference.md:88) is "Generated from bundle.yaml" (descriptive of the defect), not intent. designate is Stage-7-blocked -> no urgency. -F-CV3 [root cause NOT nailed] dashboard VIP 10.12.4.58:443 serves PLAINTEXT (Horizon exit-gate +F-CV3 [RESOLVED 2026-08-06 -- D-072 AMENDMENT (VR1) ratified GA-R5 + BUNDLEFIX-057. Root cause: + VR1 split-metal inverts D-072 -- dashboard charm declares no admin/internal binding + no + os-*-network (metadata + charmhub docs), so apache serves metal-INTERNAL while haproxy dialed + cluster=metal-admin -> plaintext. Option A (serve metal-admin) unavailable (no charm lever). + Fix (proven LIVE before ratifying): cluster -> metal-internal. Verified: provider + metal-admin + VIPs TLS 200 CA-verified; cert covers all 3 VIPs (resolves AH01909); units active/idle. Config- + of-record landed; dc1 inherits via shared bundle (no rebind). Detail below retained as record.] + dashboard VIP 10.12.4.58:443 serves PLAINTEXT (Horizon exit-gate FAILS). Certs present under /etc/apache2/ssl/horizon/; apache :433 served by Ubuntu default-ssl.conf, NOT the charm's openstack_https_frontend.conf (glance, working, uses the charm frontend w/ SSLEngine on). Charm https frontend not effective for dashboard. SEPARATE diff --git a/docs/audit/stage5-dc0-phase03-coreverify-20260806.txt b/docs/audit/stage5-dc0-phase03-coreverify-20260806.txt index b89b192..43948af 100644 --- a/docs/audit/stage5-dc0-phase03-coreverify-20260806.txt +++ b/docs/audit/stage5-dc0-phase03-coreverify-20260806.txt @@ -356,3 +356,43 @@ (measured 07-29, dc-dc-phase4 Steps 7-9 caveat). Harmless on dc0 (resolves dc0's literals); on dc1 it is WRONG and SILENT. Fix before dc1's Step 7, or export every value explicitly. This session used explicit KEYSTONE_VIP=10.12.4.50, so unaffected. + +## F-CV3 REMEDIATION -- LIVE-TEST-FIRST (operator-directed sequencing 2026-08-06) +Operator process ruling: prove the fix live BEFORE ratifying the D-072 amendment, so a +failed fix does not churn a ruling. Matches D-072's own live-first pattern (BUNDLEFIX-011). + +ROOT CAUSE (established, (a) research complete): openstack-dashboard charm declares NO + admin/internal extra-bindings (only `public`; deployed metadata + charmhub docs) and no + os-*-network config, so ApacheSSLContext.get_network_addresses() serves apache TLS on + metal-internal (10.12.12.108, charm-logged + curl 200) + provider (10.12.4.140), NOT + metal-admin. haproxy dials cluster=metal-admin (10.12.8.196:433) -> no vhost -> plaintext. + Sweep confirms the dashboard is the D-072 exception; 14 apps use the generic cluster= + metal-internal; option A (serve metal-admin) is UNAVAILABLE (no charm lever). + +DRAFT D-072 AMENDMENT (VR1) -- PROPOSED, ratify only after live confirmation: + Retire the VR0 cluster=metal-admin exception for VR1's dual-metal topology. Set + openstack-dashboard cluster -> metal-internal (the generic-rule placement AND the plane + the dashboard's apache serves in VR1). VR0's metal-admin value stands for single-metal + topology as historical record. Cert-SAN for the metal-admin VIP (AH01909) rides the + os-public-hostname/FQDN-cert workstream (Stage 7), separate from this amendment. + +LIVE TEST (reversible; revert = juju bind openstack-dashboard cluster=metal-admin): + APPLIED 2026-08-06 (operator-approved): juju bind openstack-dashboard cluster=metal-internal + -> rc=0 (only cluster moved metal-admin->metal-internal; all else unchanged). 3 units + re-rendered + settled to idle. RESULT = CONFIRMED WORKING: + - haproxy :433 backends moved to metal-internal: openstack-dashboard-0/1/2 -> + 10.12.12.108 / .145 / .146 :433 (were 10.12.8.196 / 10.12.9.31 / .32). + - VIP https NOW 200 (was 000/plaintext): provider 10.12.4.58 CA-verified 200; metal-admin + 10.12.8.58 (OPERATOR PATH) CA-verified 200. + - CERT REFINEMENT: the reissued cert carries IP SANs for ALL THREE VIPs incl 10.12.8.58 + -> the AH01909 metal-admin-VIP name-mismatch is RESOLVED BY THE REBIND, not deferred. + Operator metal-admin-VIP access validates against the vault CA. (FQDN via os-public-hostname + stays a Stage-7 nicety, not a correctness gap.) + - 3 dashboard units active/idle, no collateral. + VERDICT: fix works. Draft D-072 amendment ready for operator ratification (GA-R5), then + config-of-record (bundle cluster->metal-internal + binding-reference + harness) + commit. + RATIFIED 2026-08-06 (operator GA-R5: "Ratified, land the config-of-record"). D-072 AMENDMENT + (VR1) recorded; config-of-record landed (BUNDLEFIX-057: bundle cluster=metal-internal, + binding-reference exception RETIRED). Live and bundle now RECONCILED (both metal-internal). + F-CV3 RESOLVED. dc1 inherits via the shared bundle (no rebind). + diff --git a/docs/changelog-20260806-phase03-coreverify.md b/docs/changelog-20260806-phase03-coreverify.md index 1bb4a7b..bc38817 100644 --- a/docs/changelog-20260806-phase03-coreverify.md +++ b/docs/changelog-20260806-phase03-coreverify.md @@ -83,13 +83,36 @@ **Revert.** `juju bind designate public=metal-admin internal=metal-admin` (live) + `git revert` this commit (bundle/checker/harness/reference). Both halves needed. +## Item 7 -- BUNDLEFIX-057 / D-072 AMENDMENT (VR1): dashboard cluster -> metal-internal (F-CV3 RESOLVED) +**What.** Retired the VR0 D-072 `openstack-dashboard cluster: metal-admin` exception for VR1. + - `bundle.yaml`: dashboard `cluster: metal-admin -> metal-internal` + rewrote the BUNDLEFIX-011 + comment to explain the VR1 inversion. + - `docs/design-decisions.md`: new **D-072 AMENDMENT (VR1, 2026-08-06)**, RULED GA-R5, operator + utterance "Ratified, land the config-of-record"; refined invariant (cluster = the plane apache + SERVES, = default only when the charm serves its internal vhost there). + - `docs/network-space-binding-reference.md`: matrix row (m-internal 10->11, m-admin-nondefault + 1->0), section-5 exception RETIRED, section-6 cross-ref updated. + - LIVE (proven BEFORE ratifying, per operator process directive): `juju bind openstack-dashboard + cluster=metal-internal` (rc=0). No harness asserts this binding (checked) -> none owed. +**Why.** VR1 split metal into admin+internal; the dashboard charm declares no admin/internal +extra-binding + no os-*-network (metadata + charmhub docs confirmed), so apache serves its SSL +vhost on metal-INTERNAL, while haproxy dialed cluster=metal-admin -> vhost-less -> plaintext +(the D-072 trap, inverted). Option A (serve metal-admin) unavailable (no charm lever). cluster -> +metal-internal is both the generic HA/cluster-peer rule (14 apps) and the served plane. +**Verify.** haproxy :433 backends moved to metal-internal (.12.108/.145/.146); provider VIP +10.12.4.58 AND operator metal-admin VIP 10.12.8.58 both TLS 200 CA-verified (were 000/plaintext); +reissued cert covers all 3 VIP IPs incl 10.12.8.58 (resolves AH01909); 3 units active/idle. +Evidence: docs/audit/stage5-dc0-phase03-coreverify-20260806.txt (F-CV3 REMEDIATION section). +dc1 inherits via the shared bundle -> correct-by-deploy, no rebind. +**Revert.** `juju bind openstack-dashboard cluster=metal-admin` (live) + `git revert` this commit. + ## Findings logged (NOT executed -- hard rule 1) - **F-CV1** designate-api plaintext on :8991 vs haproxy `check-ssl` -> backend DOWN -- **RESOLVED this session, see Item 6 (BUNDLEFIX-056).** designate-api is UP; the "collateral of the Stage-7 block" reading was RETRACTED. -- **F-CV3** dashboard VIP 10.12.4.58:443 serves plaintext (apache-SSL-inactive - despite certs under /etc/apache2/ssl/horizon/). Horizon exit-gate FAILS. NOT - D-072 by pattern-match; own triage owed. Operator authorized triage 2026-08-06. +- **F-CV3** dashboard VIP TLS plaintext -- **RESOLVED this session, see Item 7 (BUNDLEFIX-057 / + D-072 VR1 amendment).** Root cause was the VR1 split-metal inversion of D-072, not the initial + "charm frontend not effective" framing; proven live then ratified. - Shared shape: 2 services plaintext-vs-TLS-expecting while the other 12 serve TLS. - **DOCFIX candidate** phase-03-admin-openrc.sh / phase-04-* / vault-kv-health.sh read DC-dependent lib-net values without `lib_net_select_dc` (harmless on dc0, diff --git a/docs/design-decisions.md b/docs/design-decisions.md index 5f1c3c3..f4afac3 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -2020,6 +2020,53 @@ invariant from D-072 ("cluster binding space == default binding space for TLS-fronted charms") is UNCHANGED and remains the carry-forward. +## D-072 -- AMENDMENT (VR1, 2026-08-06): dashboard `cluster` RETURNS to metal-internal for the dual-metal topology + +**Status:** RULED 2026-08-06 (operator, GA-R5). Operator ratification, exact utterance: +**"Ratified, land the config-of-record."** Proven LIVE before ratification per the operator's +own process directive (*prove the fix, then rule it*, to avoid ruling-churn on an unproven +change) -- which matches D-072's own live-first-then-durable pattern. + +**Why the original value INVERTS in VR1.** D-072 set `openstack-dashboard cluster: metal-admin` +because VR0 had ONE metal plane: the charm's `ApacheSSLContext.get_network_addresses()` served +apache's SSL vhost on the metal-admin address, so `cluster: metal-admin` put haproxy's 443 +backend on a served vhost. VR1 SPLIT metal into metal-admin + metal-internal (D-052/D-141 +topology + the D-121 HA scale-up this deployment hardens). MEASURED 2026-08-06: the +openstack-dashboard charm declares NO `admin`/`internal` extra-binding (only `public`; deployed +metadata + charmhub docs) and exposes no `os-*-network` config, so `get_network_addresses()` +now serves apache TLS on **metal-internal** (`10.12.12.108`, the charm's own log + +`curl https://10.12.12.108:433` -> 200), NOT metal-admin. haproxy still dialed +`cluster`=metal-admin (`10.12.8.196:433`) -> no vhost -> PLAINTEXT (F-CV3, the exact D-072 trap, +inverted). Option A (steer apache back onto metal-admin) is UNAVAILABLE in-deployment -- the +charm gives no lever -- so the fix is on the cluster side. + +**Amendment.** `openstack-dashboard cluster` -> **metal-internal**. This is simultaneously +(i) the generic HA/cluster-peer rule (14 apps already use metal-internal) and (ii) the plane +the dashboard's apache actually serves in VR1. VR0's `metal-admin` value stands as historical +record for single-metal topology (unchanged for VR0). + +**Live proof (before ratification).** `juju bind openstack-dashboard cluster=metal-internal` +-> haproxy `:433` backends moved to metal-internal (`.12.108/.145/.146`); provider VIP +`10.12.4.58` AND operator metal-admin VIP `10.12.8.58` both serve TLS **200 CA-verified** (were +`000` plaintext); the reissued cert carries IP SANs for all three VIPs incl `10.12.8.58` +(resolves the `AH01909` name-mismatch); 3 units active/idle. Capture: +`docs/audit/stage5-dc0-phase03-coreverify-20260806.txt`. Config-of-record: **BUNDLEFIX-057** +(`bundle.yaml` `cluster: metal-internal`), binding-reference exception retired. + +**REFINED INVARIANT (supersedes the 07-06 carry-forward for the split-metal case).** The +D-072 rule is more precisely: *a TLS-fronted charm's `cluster` binding must be the plane its +apache SSL vhost is served on* -- which equals the default binding ONLY when the charm serves +its internal vhost there. For a charm that declares no admin/internal binding under a +split-metal topology, that plane is metal-internal, NOT the (metal-admin) default. Roosevelt +delta: a Roosevelt-faithful fix is upstream -- have the dashboard charm declare `admin`/ +`internal` extra-bindings (like the API charms) so apache can serve the operator (metal-admin) +plane directly; until then, `cluster: metal-internal` is correct for VR1+. + +**Operator-access note (informs the tailnet model, gap-register item 21).** Operators reach +the dashboard via the metal-admin VIP over the tailnet; that VIP is unaffected by this change +(haproxy still fronts `10.12.8.58`; only the unit-local backend hop moved). The FQDN-SAN cert +(`os-public-hostname`, D-106/D-008) remains a Stage-7 nicety, not a correctness gap. + ## D-073 -- AMENDMENT (2026-07-06): APPLIED live (d011-batch3 window) The staged 39-rule overrides.zip was attached to keystone in the logged diff --git a/docs/network-space-binding-reference.md b/docs/network-space-binding-reference.md index 9e171f7..da3fb84 100644 --- a/docs/network-space-binding-reference.md +++ b/docs/network-space-binding-reference.md @@ -115,7 +115,7 @@ | octavia-diskimage-retrofit | **metal-admin** | | 2 | | | | | | octavia-hacluster | **metal-admin** | | 4 | | | | | | octavia-mysql-router | **metal-admin** | | 3 | | | | | -| openstack-dashboard | **metal-admin** | 1 | 10 | | | | 1 | +| openstack-dashboard | **metal-admin** | 1 | 11 | | | | | | openstack-dashboard-hacluster | **metal-admin** | | 4 | | | | | | ovn-central | **metal-admin** | | 6 | | | | | | ovn-chassis | **metal-admin** | | 4 | | | 1 | | @@ -152,7 +152,7 @@ | Binding | Looks like | Actually | Governing | |---|---|---|---| -| `openstack-dashboard cluster: metal-admin` | should be metal-internal | horizon renders haproxy's 443 backend on the cluster address but only makes apache SSL vhosts for default+public -> cluster on metal-internal = **dashboard HTTPS VIP dead** | **D-072 / BUNDLEFIX-011** | +| ~~`openstack-dashboard cluster: metal-admin`~~ **RETIRED 2026-08-06** | -- | The VR0 exception is RETIRED for VR1: `cluster` is back to **metal-internal** (the generic rule). VR1's split metal makes the dashboard's apache serve metal-internal, so `cluster: metal-admin` became the vhost-less trap D-072 warned of (inverted). Proven live. | **D-072 AMENDMENT (VR1) / BUNDLEFIX-057** | | `designate dnsaas: provider-public` | svc-to-svc? | designate's **public DNS-native VIP** (dual-VIP pattern), a public-facing endpoint | **D-106** | | `octavia ovsdb-cms: data-tenant` | mis-placed | D-052 lists it under data-tenant; endpoint is **inert** (octavia reaches OVN via the chassis subordinate) | **D-052** | | `ceph-radosgw public/object-store/cluster` | Ceph nets? | radosgw is a **gateway**: `public`=S3/Swift API, `object-store`=svc API, `cluster`=radosgw HA -- none are Ceph OSD/public networks | **D-052** | @@ -173,7 +173,9 @@ ## 6. Cross-references - **D-052** (+ 2026-08-03 amendment, 2026-08-05 re-amendment) -- the governing decision. -- **D-072 / BUNDLEFIX-011** -- dashboard cluster:metal-admin exception. +- **D-072 / BUNDLEFIX-011** -- dashboard cluster:metal-admin exception (VR0). **RETIRED for VR1 + by D-072 AMENDMENT (VR1, 2026-08-06) / BUNDLEFIX-057: cluster -> metal-internal** (split-metal + inverts it; the dashboard's apache serves metal-internal in VR1). Proven live. - **D-106** -- designate reactivation (dnsaas dual-VIP). - **BUNDLEFIX-056 (2026-08-06)** -- designate's `public`->provider-public + `internal`->metal-internal bindings ADDED (matrix row updated 1->2 prov-pub, 6->7 m-internal). They were previously