diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index f4faa7a..b6a0e5b 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2400,10 +2400,14 @@ at Step 7) -> installed `python3-openstackclient 6.6.0-0ubuntu2` on the rack (see section 7 row). **EXIT GATE OPEN on (a) F-CV3: dashboard VIP 10.12.4.58:443 serves PLAINTEXT (apache-SSL-inactive despite certs present) -- Horizon exit-gate FAILS; and (b) Step 3.4 domain-manager policy NOT RUN. - F-CV1: designate-api is UP but plaintext on :8991 vs haproxy check-ssl (same SHAPE as F-CV3); - the earlier "collateral of the block" reading is RETRACTED.** F-CV1/F-CV3 = a shared-shape - plaintext-vs-TLS-expecting defect on 2 services while the other 12 serve TLS correctly; LOGGED, - not fixed (hard rule 1), triage authorized by operator 2026-08-06. + F-CV1: designate _admin backend DOWN, root cause CONFIRMED = a BIND-PLANE MISMATCH (apache + https frontend binds metal-admin 10.12.8.198:8991 only; haproxy's admin backend dials + metal-internal 10.12.12.110:8991 where no SSL vhost exists) -- VR1 dual-metal-plane specific + (D-141); the earlier "collateral of the block" reading is RETRACTED.** F-CV1 and F-CV3 are TWO + SEPARATE findings (designate = wrong-plane bind; dashboard = charm https frontend not effective, + root cause NOT nailed), both charm-TLS-frontend layer (certs ARE present -- not the ovn + CN-issuance class). LOGGED, not fixed (hard rule 1); triage authorized by operator 2026-08-06; + remediation is a focused GATED session. Sweep: docs/audit/queued-findings-20260806-phase03-coreverify.txt. **NAMED-GATE DEFECT found by measurement -- `phase-03-core-verify.md` Step 3.1 asserts expected non-active/idle = 1 (octavia only); the VR1 roster yields 4 deferred-by-design + gss.** That gate is STALE for VR1 and a DOCFIX is owed (also owed on that runbook: `-m openstack` -> `-m vr1-dc0` diff --git a/docs/audit/queued-findings-20260806-phase03-coreverify.txt b/docs/audit/queued-findings-20260806-phase03-coreverify.txt new file mode 100644 index 0000000..0ab44b7 --- /dev/null +++ b/docs/audit/queued-findings-20260806-phase03-coreverify.txt @@ -0,0 +1,44 @@ +# Queued findings -- Stage 5 dc0 Step 7 (phase-03 core verify), 2026-08-06 +# Survives-a-clear sweep. Full evidence: docs/audit/stage5-dc0-phase03-coreverify-20260806.txt +# Status authority is CURRENT-STATE.md (section 1 Stage-5 block + section 7 client row). + +STEP 7 EXIT GATE: NOT MET (GA-R6/E3, no conditional close). Core-API layer VERIFIED; +two named items keep it OPEN: F-CV3 (Horizon TLS) + Step 3.4 (not run). + +--- FINDINGS (logged, NOT fixed -- hard rule 1; operator authorized triage 2026-08-06) --- + +F-CV1 [CONFIRMED root cause] designate _admin haproxy backend DOWN. + designate/0 apache https frontend binds ONLY 10.12.8.198:8991 (metal-admin); haproxy's + `designate-api_admin_10.12.12.110` backend dials designate-0 at 10.12.12.110:8991 + (METAL-INTERNAL), where apache has no SSL vhost -> check-ssl hits plaintext -> DOWN. + BIND-PLANE MISMATCH, VR1 dual-metal-plane specific (D-141 .8 metal-admin vs .12 metal-internal). + NOT Stage-7 collateral (structural). Fix (focused session, GATED): align the admin-interface + plane -- charm binds metal-internal too, OR haproxy admin backend dials metal-admin; correct + plane is a D-141/B1 question. Re-run haproxy sweep after. + +F-CV3 [root cause NOT nailed] dashboard VIP 10.12.4.58:443 serves PLAINTEXT (Horizon exit-gate + FAILS). Certs present under /etc/apache2/ssl/horizon/; apache :433 served by Ubuntu + default-ssl.conf, NOT the charm's openstack_https_frontend.conf (glance, working, uses the + charm frontend w/ SSLEngine on). Charm https frontend not effective for dashboard. SEPARATE + finding from F-CV1 -- do NOT chase a common fix. Needs focused triage + gated remediation. + + NOTE both: certs ARE present -> NOT the ovn CN-issuance class; NOT a missing certificates + relation. Charm apache-TLS-frontend layer. Also owed with the Horizon-access remediation: + D-044 cookie override + D-075 root redirect (per-rebuild, not applied this rebuild). + +--- OWED (not findings, just must-not-evaporate) --- + +O2 dc1 RACK needs `apt-get install -y python3-openstackclient` (6.6.0-0ubuntu2) BEFORE dc1's + Step 7 -- F-CV2 is per-DC; only the dc0 rack was done this session (== 07-30 queued-F1). +O4 Step 3.4 keystone domain-manager policy gate (PO: stage-1 + C.4 G3 behavioral, G3 mutates) + still owed for phase-03 close. +O5 DOCFIX candidate: phase-03-admin-openrc.sh / phase-04-network-{create,verify}.sh / + phase-04-internal-cert-san-verify.sh / vault-kv-health.sh read DC-dependent lib-net values + WITHOUT lib_net_select_dc (harmless on dc0, WRONG+SILENT on dc1). Fix before dc1's Step 7. +O6 NAMED-GATE DEFECT (already in CURRENT-STATE): phase-03-core-verify.md Step 3.1 asserts + non-active/idle == 1; VR1 roster yields the 4 deferred-by-design + gss. Runbook DOCFIX owed + (-m openstack -> -m vr1-dc0; run-from-rack per D-138; the settle count). +O7 rack kernel 6.8.0-136 running vs 6.8.0-137 available -- reboot NOT taken (would bounce the + rack + libvirt + all nodes); a maintenance-window item, logged. + +--- FIRST SURFACE in this file: F-CV1 (confirmed), F-CV3, O5, O7. --- diff --git a/docs/audit/stage5-dc0-phase03-coreverify-20260806.txt b/docs/audit/stage5-dc0-phase03-coreverify-20260806.txt index bca6428..48ee27a 100644 --- a/docs/audit/stage5-dc0-phase03-coreverify-20260806.txt +++ b/docs/audit/stage5-dc0-phase03-coreverify-20260806.txt @@ -163,6 +163,62 @@ designate = api-plaintext-vs-haproxy-check-ssl), but the shape is shared and worth a single triage thread. LOGGED, not fixed (hard rule 1). +## TRIAGE (operator-authorized 2026-08-06) F-CV1 + F-CV3 -- read-only; NOT fixed (hard rule 1) +CONFIRMED: + - Both apps HAVE the certificates->vault relation (tls-certificates, global), same as the + 12 working apps. NOT a missing relation. + - Both have vault certs RENDERED: /etc/apache2/ssl/horizon/cert_juju-d4f45a-0-lxd-14.maas + and /etc/apache2/ssl/designate/cert_juju-d4f45a-0-lxd-5.maas (+ per-IP symlinks incl the + VIPs), dated Aug 5 (cascade). NOT a cert-issuance/empty-CN gap (unlike the ovn saga). + - Both serve their SSL backend via apache/mod_wsgi; ssl_module loaded. + - WORKING REFERENCE glance/leader: TLS via the charm vhost + /etc/apache2/sites-available/openstack_https_frontend.conf (SSLEngine on). + - DASHBOARD: apache :433 (SSL backend) is served by /etc/apache2/sites-enabled/ + default-ssl.conf (the UBUNTU DEFAULT vhost), binding the VIP addrs -- NOT the charm's + openstack_https_frontend.conf. Measured PLAINTEXT on :70/:433/:443. + - DESIGNATE: apache :8991 served by openstack_https_frontend.conf bound 10.12.8.198:8991 + (metal-admin), but haproxy backend targets 10.12.12.110:8991 (metal-internal) -> + possible BIND-PLANE MISMATCH (apache may not present SSL on the addr haproxy dials). + designate.conf `listen = 0.0.0.0:8981` is the inner api; apache fronts :8991. +INSTRUMENT CAVEAT (own it): an earlier `grep -r ... /etc/apache2/sites-enabled/` returned + "no SSLEngine" -- FALSE NEGATIVE. `grep -r` does NOT follow the SYMLINKS that populate + sites-enabled/, so it read nothing. `apache2ctl -S` is authoritative and corrected it. +NOT NAILED (do NOT assert one root cause -- the ovn root cause was called wrong 3x): + exact reason apache serves plaintext despite ssl vhosts present. Candidates: charm https + frontend not effective/enabled (dashboard uses default-ssl instead); bind-plane mismatch + (designate); apache not reloaded after cert render; an ssl config option off. Two DISTINCT + manifestations, one CLASS: the charm apache TLS-frontend is not effective on the address + the client/haproxy actually uses. +ROUTING / PROPOSED REMEDIATION (to be individually gated, recommend a FOCUSED session): + charm-config-layer, not infra. Likely path: investigate why the openstack-dashboard charm + did not render/enable its https frontend (vs glance), and whether designate's frontend + binds all required planes; then re-trigger the charm to re-render + reload apache (e.g. + `juju exec ... hooks/config-changed` or a relation refresh), verify apache serves SSL on + the haproxy-dialed address, then re-run the haproxy sweep + the dashboard fail-closed probe. + NOT attempted live this session (verification scope; remediation needs its own gate). + Re-check F-CV1 also at Stage-7 designate activation. + +CONFIRMED 2026-08-06 (advisor-directed, read the vhost file directly -- apache2ctl -S +had been head-truncated): DESIGNATE is a BIND-PLANE MISMATCH, now a NAMED defect: + - designate/0 apache https frontend = EXACTLY ONE `` + (metal-admin; the unit's own .8 address). + - haproxy has TWO designate backends: `designate-api_admin_10.12.12.110` dials + designate-0 at 10.12.12.110:8991 (METAL-INTERNAL); a second backend dials 10.12.8.198 + (metal-admin). The sweep DOWN was the _admin_10.12.12.110 one -- apache has NO SSL vhost + on the metal-internal address, so haproxy's check-ssl there hits plaintext -> DOWN. + - ROOT CAUSE (designate): apache presents SSL only on metal-admin (.8.198); haproxy's + admin backend dials metal-internal (.12.110). VR1 dual-metal-plane specific (D-141 split + of metal-admin .8 vs metal-internal .12). Fix shape (focused session, gated): align the + plane -- either the charm binds the metal-internal address too, or the admin backend + dials metal-admin; the correct plane for the ADMIN interface is a D-141/B1 question. + - This is STRUCTURAL, will NOT self-resolve at Stage-7 nameserver config -> my original + "collateral of the block" reading is definitively wrong (retraction stands). + +>>> F-CV1 and F-CV3 ARE TWO SEPARATE FINDINGS, not one (advisor): F-CV1 = designate + bind-plane mismatch (charm frontend present, wrong/one plane); F-CV3 = dashboard SSL + backend served by Ubuntu default-ssl.conf, charm https frontend not effective (different + question, no one-command discriminator, left as recorded). Do NOT chase a common fix. + ## Step 3.4 keystone domain-manager policy -- NOT RUN this session Deferred with the openstack-client-dependent verification depth; the PO: stage-1 check + the C.4 G3 behavioral probe (which mutates -- creates user/project) are owed. Recorded as