diff --git a/bundle.yaml b/bundle.yaml index a6e2388..da1817c 100644 --- a/bundle.yaml +++ b/bundle.yaml @@ -935,10 +935,19 @@ shared-db: metal-internal certificates: metal-internal coordinator-memcached: metal-internal - dnsaas: provider-public # designate's REAL provides-side endpoint name - # (interface "designate") -- confirmed via its - # own metadata.yaml; NOT named "public" like the - # other API charms in this bundle. + # BUNDLEFIX-056 / conformance to the D-020 AMENDMENT (2026-07-27) + measured + # 2026-08-06: designate DOES declare public/admin/internal extra-bindings + # (deployed metadata.yaml on designate/0 verified in full -- identical to glance). + # Omitting public+internal left them on the '' metal-admin FALLBACK, orphaning + # the provider + metal-internal legs of designate's ruled .62 triple and leaving + # the _admin haproxy backend SSL-DOWN on the unserved metal-internal address + # (F-CV1). Every sibling API charm binds these; designate must too. + public: provider-public # tenant-facing multi-tenant REST API (charm desc) + internal: metal-internal # internal API plane + dnsaas: provider-public # ADDITIONAL provides-side DNS endpoint (interface + # "designate", D-106 dual-VIP) -- DISTINCT from the + # public REST API above, NOT a replacement for it + # (the prior comment here wrongly treated it as one). ha: metal-internal constraints: arch=amd64 diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index ecc7bbc..5501659 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2400,19 +2400,21 @@ at Step 7) -> installed `python3-openstackclient 6.6.0-0ubuntu2` on the rack (see section 7 row). **EXIT GATE OPEN on (a) F-CV3: dashboard VIP 10.12.4.58:443 serves PLAINTEXT (apache-SSL-inactive despite certs present) -- Horizon exit-gate FAILS; and (b) Step 3.4 domain-manager policy NOT RUN. - F-CV1: designate _admin backend DOWN (apache https frontend binds metal-admin 10.12.8.198:8991 - only; haproxy's admin backend dials metal-internal 10.12.12.110:8991 where no SSL vhost exists). - ROOT CAUSE researched 2026-08-06 (governing = D-052 + the generic binding rule, NOT D-141 -- - attribution corrected): designate's endpoint bindings deviate from every sibling API app. - `:internal`->metal-internal is a CONFIRMED DEFECT (on the `''` metal-admin fallback; no ruled - exception; cert already covers the metal-internal SANs; all 3 units have metal-internal addrs). - `:public`->provider-public is OPEN pending an operator RULING (designate uniquely carries - `:dnsaas` on provider-public per the D-106 dual-VIP -- its REST API may be intentionally - metal-admin-only). The earlier "collateral of the block" reading is RETRACTED.** F-CV1 and - F-CV3 are TWO SEPARATE findings (dashboard = charm https frontend not effective, NOT a binding - issue -- its bindings are correct; needs its own triage). Fix method = D-072 precedent - (bundle + live juju bind + haproxy-readback verify). LOGGED, not fixed (hard rule 1); triage - authorized 2026-08-06. Sweep: docs/audit/queued-findings-20260806-phase03-coreverify.txt. + F-CV1: designate _admin backend DOWN -- **RESOLVED 2026-08-06 (BUNDLEFIX-056, operator-approved + fix EXECUTED + VERIFIED).** Root cause (governing = D-052 + generic binding rule + the D-020 + amendment's ruled .62 triple, NOT D-141): designate's bundle bindings OMITTED public + internal, + defaulting them to the `''` metal-admin fallback -> orphaned the provider + metal-internal legs + of its VIP triple -> apache served SSL on metal-admin only while haproxy's _admin backend dialed + the unserved metal-internal address. The charm DECLARES public/admin/internal extra-bindings + (deployed metadata verified) so it supports the split. FIX: bundle +public:provider-public + +internal:metal-internal (config-of-record) AND live `juju bind designate public=provider-public + internal=metal-internal`. VERIFIED: full haproxy sweep 0 DOWN cloud-wide; apache https vhosts now + span all 3 planes; cert reissued to cover provider-public; catalog shows the correct triple + (public .4.62 / internal .12.62 / admin .8.62). provider-bundle-check EXPECT_PUBLIC_VIP 11->12 + (+T16c/T16d failing-direction), binding-matrix reference row updated. gauntlet ALL GREEN 99.** + F-CV1 and F-CV3 are TWO SEPARATE findings; F-CV3 (dashboard = charm https frontend not effective, + bindings ARE correct) is STILL OPEN and needs its own triage session. Sweep: + docs/audit/queued-findings-20260806-phase03-coreverify.txt. **NAMED-GATE DEFECT found by measurement -- `phase-03-core-verify.md` Step 3.1 asserts expected non-active/idle = 1 (octavia only); the VR1 roster yields 4 deferred-by-design + gss.** That gate is STALE for VR1 and a DOCFIX is owed (also owed on that runbook: `-m openstack` -> `-m vr1-dc0` diff --git a/docs/audit/queued-findings-20260806-phase03-coreverify.txt b/docs/audit/queued-findings-20260806-phase03-coreverify.txt index db4de8b..03a020f 100644 --- a/docs/audit/queued-findings-20260806-phase03-coreverify.txt +++ b/docs/audit/queued-findings-20260806-phase03-coreverify.txt @@ -7,7 +7,13 @@ --- FINDINGS (logged, NOT fixed -- hard rule 1; operator authorized triage 2026-08-06) --- -F-CV1 designate _admin haproxy backend DOWN. designate/0 apache https frontend binds ONLY +F-CV1 [RESOLVED 2026-08-06 -- BUNDLEFIX-056, operator-approved fix executed + verified: + bundle +public:provider-public +internal:metal-internal + live `juju bind designate + public=provider-public internal=metal-internal`. Full haproxy sweep 0 DOWN cloud-wide; + apache vhosts span all 3 planes; cert reissued for provider-public; catalog triple + correct (public .4.62 / internal .12.62 / admin .8.62). Checker EXPECT 11->12 (+T16c/d); + binding-matrix row updated; gauntlet ALL GREEN 99. Details below retained as the record.] + designate _admin haproxy backend DOWN. designate/0 apache https frontend binds ONLY 10.12.8.198:8991 (metal-admin); haproxy's `designate-api_admin_10.12.12.110` backend dials 10.12.12.110:8991 (METAL-INTERNAL) where apache has no SSL vhost -> check-ssl -> DOWN. ROOT CAUSE (researched 2026-08-06, governing = D-052 + the generic binding rule; NOT D-141): @@ -48,5 +54,11 @@ (-m openstack -> -m vr1-dc0; run-from-rack per D-138; the settle count). O7 rack kernel 6.8.0-136 running vs 6.8.0-137 available -- reboot NOT taken (would bounce the rack + libvirt + all nodes); a maintenance-window item, logged. +O8 OBSERVATION (NOT verified as a defect -- do the D-NNN check first): designate's `amqp` + endpoint is bound to metal-admin, while the generic rule (network-space-binding-reference + line 57) + glance put amqp on metal-internal. Surfaced during the BUNDLEFIX-056 bind output. + NOT causing a measured failure (absent from every sweep); NOT touched (out of F-CV1 scope, + hard rule 1). A future binding-conformance pass should check whether this is ruled/intentional + (like it should have for public/internal) before treating it as a defect. Candidate only. --- FIRST SURFACE in this file: F-CV1 (confirmed), F-CV3, O5, O7. --- diff --git a/docs/audit/stage5-dc0-phase03-coreverify-20260806.txt b/docs/audit/stage5-dc0-phase03-coreverify-20260806.txt index 562892f..b89b192 100644 --- a/docs/audit/stage5-dc0-phase03-coreverify-20260806.txt +++ b/docs/audit/stage5-dc0-phase03-coreverify-20260806.txt @@ -295,6 +295,33 @@ SEQUENCING: designate is Stage-7-blocked -> no urgency; do bundle+live together so a redeploy is correct and live is not left divergent. +## F-CV1 FIX EXECUTED + VERIFIED 2026-08-06 (operator-approved; BUNDLEFIX-056) -- RESOLVED +STATIC (config-of-record, gauntlet ALL GREEN 99 + repo-lint 0-fail after): + - bundle.yaml designate bindings: +public:provider-public +internal:metal-internal; the + misleading dnsaas comment corrected (it wrongly said designate has no `public` binding). + - provider-bundle-check.py EXPECT_PUBLIC_VIP 11->12 + header + rationale (designate joins; + vault stays out, metal-only -> NOT 13). Harness +T16c (count 11 FAILS) +T16d (count 13 + FAILS) -> both failing directions proven; harness 57->59, ALL PASS. + - network-space-binding-reference.md row 88 (1->2 prov-pub, 6->7 m-internal) + a ยง6 note. +LIVE (on the dc0 rack, model vr1-dc0, operator "Yes, run the live binding"): + juju bind designate public=provider-public internal=metal-internal -> rc=0 + ("moving internal metal-admin->metal-internal; public metal-admin->provider-public"; + admin/amqp/cluster/nrpe, the metal-internal set, and dnsaas all unchanged.) + Charm re-rendered ~3-4 min (cert reissue + apache/haproxy rebuild x3); settled to idle. +VERIFIED (assert the artifact, D-072 method): + - haproxy DOWN sweep: designate 0 DOWN; then FULL 39-unit sweep 0 DOWN CLOUD-WIDE. + - designate apache https vhosts now span 3 planes: 10.12.12.110 (metal-internal, was ABSENT), + 10.12.4.141 (provider-public, was ABSENT), 10.12.8.198 (metal-admin). Before = metal-admin only. + - cert SANs now include provider-public (10.12.4.141 + VIP 10.12.4.62) alongside the metal + planes -> reissued to cover the new legs. + - keystone catalog (openstack endpoint list --service dns): public https://10.12.4.62:9001 + (provider), internal https://10.12.12.62:9001 (metal-internal), admin https://10.12.8.62:9001 + (metal-admin) -- the correct 3-plane triple, matching every sibling. Before: all on .8.62. +RESULT: F-CV1 RESOLVED. Revert if ever needed: juju bind designate public=metal-admin + internal=metal-admin (+ revert the bundle/checker/reference commit). +NOTE: the phase-03 EXIT GATE remains OPEN on F-CV3 (dashboard, separate root cause -- its own + triage session) and Step 3.4 (not run). F-CV1 was a haproxy-gate item, now cleared. + ## Step 3.4 keystone domain-manager policy -- NOT RUN this session Deferred with the openstack-client-dependent verification depth; the PO: stage-1 check + the C.4 G3 behavioral probe (which mutates -- creates user/project) are owed. Recorded as diff --git a/docs/changelog-20260806-phase03-coreverify.md b/docs/changelog-20260806-phase03-coreverify.md index d80b3aa..1bb4a7b 100644 --- a/docs/changelog-20260806-phase03-coreverify.md +++ b/docs/changelog-20260806-phase03-coreverify.md @@ -57,9 +57,36 @@ --- +## Item 6 -- BUNDLEFIX-056: designate binding fix (F-CV1 RESOLVED) +**What.** designate's REST API `public` + `internal` endpoints were on the `''` +metal-admin fallback (omitted from the bundle), orphaning the provider + metal-internal +legs of its ruled `.62` VIP triple and leaving the `_admin` haproxy backend SSL-DOWN +on the unserved metal-internal address (F-CV1). + - `bundle.yaml`: designate bindings +`public: provider-public` +`internal: metal-internal`; + corrected the misleading `dnsaas` comment (it wrongly claimed designate has no `public` + binding -- the deployed charm metadata declares public/admin/internal extra-bindings). + - `scripts/provider-bundle-check.py`: `EXPECT_PUBLIC_VIP` 11->12 (designate joins; vault + stays out, metal-only) + header + rationale rewrite (preserves the "not 13" warning). + - `tests/provider-bundle-check/run-tests.sh`: +T16c (count 11 FAILS) +T16d (count 13 FAILS) + -- both failing directions proven; harness 57->59, ALL PASS. + - `docs/network-space-binding-reference.md`: row 88 (1->2 prov-pub, 6->7 m-internal) + a section-6 note. + - LIVE (operator-approved): `juju bind designate public=provider-public internal=metal-internal` + on the dc0 rack (rc=0). Charm re-rendered + reissued cert; settled to idle. +**Why.** Conformance to the D-020 amendment (2026-07-27, designate's established +provider/admin/internal triple) + the generic binding rule + every sibling API charm. +Governing: D-052 / D-020 amendment. Root of the defect: the prior reading that designate +had "no public binding, only dnsaas" -- dnsaas is ADDITIONAL, not a replacement. +**Verify.** Full haproxy sweep 0 DOWN cloud-wide; designate apache https vhosts span all +3 planes; cert SANs now include provider-public; catalog triple correct (public 10.12.4.62 / +internal 10.12.12.62 / admin 10.12.8.62). Gauntlet ALL GREEN (99); repo-lint 0-fail. +Evidence: docs/audit/stage5-dc0-phase03-coreverify-20260806.txt (F-CV1 FIX section). +**Revert.** `juju bind designate public=metal-admin internal=metal-admin` (live) + +`git revert` this commit (bundle/checker/harness/reference). Both halves needed. + ## Findings logged (NOT executed -- hard rule 1) -- **F-CV1** designate-api plaintext on :8991 vs haproxy `check-ssl` -> backend DOWN. - designate-api is UP; the "collateral of the Stage-7 block" reading is RETRACTED. +- **F-CV1** designate-api plaintext on :8991 vs haproxy `check-ssl` -> backend DOWN -- + **RESOLVED this session, see Item 6 (BUNDLEFIX-056).** designate-api is UP; the + "collateral of the Stage-7 block" reading was RETRACTED. - **F-CV3** dashboard VIP 10.12.4.58:443 serves plaintext (apache-SSL-inactive despite certs under /etc/apache2/ssl/horizon/). Horizon exit-gate FAILS. NOT D-072 by pattern-match; own triage owed. Operator authorized triage 2026-08-06. diff --git a/docs/network-space-binding-reference.md b/docs/network-space-binding-reference.md index 38ee24a..9e171f7 100644 --- a/docs/network-space-binding-reference.md +++ b/docs/network-space-binding-reference.md @@ -85,7 +85,7 @@ | cinder-hacluster | **metal-admin** | | 4 | | | | | | cinder-mysql-router | **metal-admin** | | 3 | | | | | | dashboard-mysql-router | **metal-admin** | | 3 | | | | | -| designate | **metal-admin** | 1 | 6 | | | | | +| designate | **metal-admin** | 2 | 7 | | | | | | designate-bind | **metal-admin** | | 1 | | | | | | designate-hacluster | **metal-admin** | | 4 | | | | | | designate-mysql-router | **metal-admin** | | 3 | | | | | @@ -175,6 +175,14 @@ - **D-052** (+ 2026-08-03 amendment, 2026-08-05 re-amendment) -- the governing decision. - **D-072 / BUNDLEFIX-011** -- dashboard cluster:metal-admin exception. - **D-106** -- designate reactivation (dnsaas dual-VIP). +- **BUNDLEFIX-056 (2026-08-06)** -- designate's `public`->provider-public + `internal`->metal-internal + bindings ADDED (matrix row updated 1->2 prov-pub, 6->7 m-internal). They were previously + omitted, defaulting to the `''` metal-admin fallback, which orphaned the provider + metal-internal + legs of designate's ruled `.62` triple (D-020 amendment 2026-07-27) and left its `_admin` TLS + backend SSL-DOWN (F-CV1). Conformance repair: the deployed charm DECLARES `public`/`admin`/`internal` + extra-bindings (metadata.yaml verified) and every sibling API charm binds them the same way. + **`dnsaas` (D-106) is ADDITIONAL to the public API endpoint, NOT a replacement** -- the prior + reading that designate had "no public binding, only dnsaas" was the defect's root. - **D-125** -- metal-internal egress isolation. - `scripts/lib-net.sh` -- plane CIDRs / names (per-DC). - `docs/audit/binding-plane-purpose-sweep-20260803.txt` -- the full 446-endpoint classification. diff --git a/scripts/provider-bundle-check.py b/scripts/provider-bundle-check.py index 449d2b3..21a07f1 100644 --- a/scripts/provider-bundle-check.py +++ b/scripts/provider-bundle-check.py @@ -4,7 +4,8 @@ Asserts ONLY the post-revert (D-052/D-053 + Pattern A) provider invariants on a Charmed-OpenStack bundle: - 1. exactly 11 API charms bind public -> provider-public; none remain on provider-vip + 1. exactly 12 API charms bind public -> provider-public; none remain on provider-vip + (12 since BUNDLEFIX-056 2026-08-06 -- designate joined; was 11; vault stays OUT, metal-only) 2. every clustered VIP is EITHER a v4 triple -- provider-public(10.12.4/22) admin(10.12.8/22) internal(10.12.12/22) -- OR a dual-family sextet appending the three v6 legs (R2, RULED 2026-07-27: dual-stack for both DCs). All legs @@ -119,11 +120,18 @@ "vr1-dc1": "openstack-vr1-dc1", } # D-020 AMENDMENT / R11 (RULED 2026-07-27): the band widens 50-60 -> 50-99 to admit -# vault .61 and designate .62. EXPECT_PUBLIC_VIP deliberately STAYS 11 -- measured, -# NEITHER vault nor designate carries a `public` binding, so they do not join that -# count and bumping both constants together breaks the gate. +# vault .61 and designate .62. +# EXPECT_PUBLIC_VIP = 12 (BUNDLEFIX-056, 2026-08-06): designate JOINS the public-> +# provider-public count. The prior "deliberately STAYS 11" reasoning was DESCRIPTIVE OF A +# DEFECT -- it read designate's then-missing `public:` binding as intent. designate was ruled +# the FULL provider/admin/internal triple (.62, provider leg included) and its DEPLOYED charm +# DECLARES a `public` extra-binding (metadata.yaml on designate/0 verified); leaving it off +# provider-public orphaned the .4.62 leg and broke its TLS (F-CV1). Invariant 2 already +# REQUIRES designate's provider VIP triple, so invariant 1 asserting no public binding was +# internally inconsistent. vault STAYS OUT (metal-only, D-020 amendment 2026-08-05): the count +# is 12, NOT 13 -- bumping to 13 to admit vault remains WRONG and must still fail. OCTET_LO, OCTET_HI = 50, 99 -EXPECT_PUBLIC_VIP = 11 +EXPECT_PUBLIC_VIP = 12 # Invariant 9's measured authority: the charms that DECLARE a `prefer-ipv6` config # option. Keyed by CHARM NAME, never by application name -- the application name is diff --git a/tests/provider-bundle-check/run-tests.sh b/tests/provider-bundle-check/run-tests.sh index 1c93930..906a8c8 100644 --- a/tests/provider-bundle-check/run-tests.sh +++ b/tests/provider-bundle-check/run-tests.sh @@ -245,6 +245,18 @@ run 1 'hacluster relation but no vip: designate' \ "T16b removing designate's VIP still FAILS" "$TMP/t16b.yaml" +# T16c/T16d BUNDLEFIX-056 (2026-08-06): designate JOINED public->provider-public (count 11->12). +# These prove EXPECT_PUBLIC_VIP=12 can still FAIL in BOTH directions -- an app dropping off +# provider-public (11) and vault wrongly joining (13). Without them the exact-count constant is +# decoration (the repo's mutation-pass lesson). See scripts/provider-bundle-check.py:121. +mutate t16c.yaml 'b["applications"]["designate"]["bindings"].pop("public")' +run 1 'public->provider-public count=11 \(expect 12\)' \ + "T16c dropping designate's public binding FAILS (count 11 != 12)" "$TMP/t16c.yaml" + +mutate t16d.yaml 'b["applications"]["vault"].setdefault("bindings",{})["public"]="provider-public"' +run 1 'public->provider-public count=13 \(expect 12\)' \ + "T16d vault joining public->provider-public FAILS (count 13 != 12; vault is metal-only)" "$TMP/t16d.yaml" + # T31 ruling 3's new invariant: the BARE base is not a deploy input. Validating it # alone must name every clustered principal, not silently pass -- a VIP-free # bundle deployed without its overlay is 11 charms of decorative HA.