# Stage 5 dc0 -- Step 7: phase-03 core verify (ADAPTED for vr1-dc0), read-only pass
# Runbook: runbooks/phase-03-core-verify.md, adapted per dc-dc-phase4 Steps 7-9.
# Model: vr1-dc0. Run-location: dc0 rack (172.31.0.2 via ProxyJump voffice1) -- D-138.
# Started 2026-08-06. Author: Claude Code (background session), operator "continue".
#
# DISCIPLINE: hard rule 1 -- this is VERIFICATION. Findings are LOGGED here, not
# fixed mid-step (the sole sanctioned in-step remediation is the 3.1 haproxy
# `reload`, individually gated). Prediction is written BEFORE reading live status
# so the settle gate is falsifiable (advisor 2026-08-06; memory #17 / GA-R6
# "a checker that cannot fail is not a gate").
#
# =====================================================================
# PREDICTION (written 2026-08-06 BEFORE any `juju status` this session)
# =====================================================================
# Source of the prediction:
#   - 08-06 part-2 close: "14/14 HA ... 12 active/idle + 2 known-blocked
#     (octavia/designate)".
#   - 08-05 close: designate/octavia/rbd-mirror correctly-waiting on later stages.
#   - D-021: octavia BLOCKED "Awaiting configure-resources" until phase-05 (Stage 6).
#   - Stage-7 (dc-dc-phase6): designate activation.
#   - Stage-6 (dc-dc-phase5): ceph-rbd-mirror / radosgw multisite DR wiring.
#
# EXPECTED non-active/idle at this pass (the prediction to diff against):
#   E1. octavia/*            -- BLOCKED, D-021 (awaiting configure-resources; Stage 6)
#   E2. designate/*          -- waiting/blocked, Stage-7 activation
#   E3. ceph-rbd-mirror/*    -- waiting/blocked, Stage-6 DR wiring
#   E4. (tolerated-transient) glance-simplestreams-sync -- pre-run, if present
#
# DIFF RULE (the gate):
#   - Any live non-active/idle unit NOT in {E1,E2,E3,E4}  => FINDING (log below).
#   - Any unit in {E1,E2,E3} that came back ACTIVE/IDLE   => FINDING (record stale).
#   - octavia downstream-of-ovn note (08-04): ovn-central is now RESOLVED
#     (08-05), so octavia's block should be its OWN D-021 block, not an ovn
#     cascade. If octavia is waiting on ovn/cert, that is a FINDING.
#
# =====================================================================
# MEASUREMENT (filled in below, this session)
# =====================================================================

## Step 3.1 acceptance walk -- MEASURED 2026-08-06 (read-only)
[vr1-dc0-rack] juju status -m vr1-dc0 --format=yaml  (rc=0, 223979 bytes, 57 apps, 162 units)
Processed on jumphost. Non-active/idle units: 8 of 162 (154 active/idle):
  ceph-rbd-mirror/0            blocked  idle  'ceph-local' incomplete, 'ceph-remote' missing   [E3 Stage-6 DR]
  designate/0,1,2              blocked  idle  nameservers must be set                          [E2 Stage-7]
  glance-simplestreams-sync/0  unknown  idle  (empty)                                          [E4 transient]
  octavia/0                    blocked  idle  Awaiting end-user execution of `configure-resources` [E1 D-021]
  octavia/1,2                  blocked  idle  Awaiting leader to create required resources        [E1 D-021]

DIFF vs PREDICTION: MATCH. All 8 in {E1,E2,E3,E4}; none outside; none expected-blocked
came back active. GATE PASS (falsifiable -- prediction pre-dated the read).
NOTE: octavia/0 message is its OWN D-021 configure-resources block, NOT an ovn/cert
cascade -- confirms ovn-central RESOLVED (08-05) end-to-end; the 08-04 "octavia
downstream-of-ovn" concern is CLEARED.

## Step 3.1 haproxy backend sweep (DOCFIX-031/D-045) -- MEASURED 2026-08-06 (read-only)
Scoped to the 13 VIP-fronted apps (those carrying an *-hacluster subordinate),
39 units. Loop run ON the rack (juju ssh controller-local).
RESULT: 3 DOWN lines, ALL on designate (the known Stage-7 block, E2):
  [designate/0] designate-api_admin_10.12.12.110:8991  DOWN  L6RSP "SSL handshake failure / Layer6 invalid response"
  [designate/1] designate-api_admin_10.12.12.144:8991  DOWN  L6RSP  (same)
  [designate/2] designate-api_admin_10.12.12.143:8991  DOWN  L6RSP  (same)
CLEAN for all 12 ACTIVATED VIP apps: barbican ceph-radosgw cinder glance keystone
  magnum neutron-api nova-cloud-controller octavia openstack-dashboard placement vault
  -> zero DOWN.

FINDING F-CV1 (LOGGED, not fixed -- hard rule 1): designate-api _admin backend DOWN
  with an SSL-handshake/L6 signature on all 3 units. designate is BLOCKED pre-Stage-7
  ("nameservers must be set"), so this is plausibly collateral of the un-activated API,
  NOT necessarily the D-045 plaintext-vs-SSL defect. The sanctioned in-step haproxy
  `reload` is WITHHELD: designate is not a healthy app, and reload would not clear a
  Stage-7 activation gap. ACTION: re-run this sweep against designate AFTER Stage-7
  activation (dc-dc-phase6); if the L6/SSL DOWN persists on an ACTIVE designate, THAT
  is a D-045-class defect to remediate then. Note the signature here is L6RSP (SSL
  handshake), which differs from D-045's classic L7STS/400 plaintext-vs-SSL.

GATE VERDICT: PASS for the 12 activated VIP apps (the phase-03 core-verify scope at
  this pre-Stage-6/7 point). designate deferred to its Stage-7 activation, consistent
  with the settle-walk E2 classification.

## Step 3.2 admin-openrc -- MEASURED 2026-08-06
Config source (dc0 vips overlay line 36): keystone provider VIP = 10.12.4.50
  (v4-only D-141 triple ".50 / .8.50 / .12.50").
Rack state: admin-openrc + vault root CA both ABSENT (fresh build, nothing to clobber;
  API reachability never yet verified via openrc on this deploy). phase-03-admin-openrc.sh
  + extract_admin_password.py staged to rack, sha256 MATCH (2aeb20b720d83944 /
  7b9b25a8fafd75cb); lib-net.sh dep already staged, MATCH 3e465ae7de3bf92d.

BLOCKER F-CV2 (== the KNOWN 07-30 queued-finding F1, HIT at Step 7) -- RESOLVED this session:
  First run -> "FAIL: openstack not found". The openstack CLI was ABSENT on the dc0 rack
  (no snap/apt/venv). D-138 puts phase-03..06's openstack CLI ON THE RACK, but the 07-27
  install landed it only on voffice1 (which cannot reach keystone's provider VIP L3).
  Phase-01/02 were juju-only, so it first surfaced here. Documented remediation (07-30 F1):
  "Install on the DC client host before Step 7." EXECUTED (gated): apt-get install -y
  python3-openstackclient on the rack -> 6.6.0-0ubuntu2 (+ python3-openstacksdk
  3.0.0-0ubuntu2), matching the measured 07-27 pin exactly; snap correctly refuted, no UCA.
  REVERT: sudo apt-get purge -y python3-openstackclient on the rack.
  OWED DOC: add the dc0 rack to CURRENT-STATE section 7's OpenStack-client row (per F1's
  own instruction). Same install is OWED on the dc1 rack before dc1's Step 7.
  NOTE (pending kernel): rack reports 6.8.0-136 running vs 6.8.0-137 available -- NOT acted
  on (a reboot would bounce the rack + libvirt + all nodes); logged for a maintenance window.

BUILD (after unblock): MODEL=vr1-dc0 KEYSTONE_VIP=10.12.4.50 phase-03-admin-openrc.sh ->
  vault root CA subject "Vault Root Certificate Authority (charm-pki-local)", notAfter
  Aug 2 2036 (matches 08-05 init); admin project = admin (password len 16); wrote openrc
  0600; **[OK] scoped token issued** against https://10.12.4.50:5000/v3.

TWO-SOURCE VIP (advisor): CONFIRMED. Config (overlay) keystone provider VIP = 10.12.4.50;
  live artifact `openstack endpoint list --service keystone`:
    public   https://10.12.4.50:5000/v3     (provider)
    admin    https://10.12.8.50:35357/v3    (metal-admin, :35357)
    internal https://10.12.12.50:5000/v3    (metal-internal)
  Config == artifact. IP-only endpoint list PASS across ALL services (public .4.x /
  admin .8.x / internal .12.x -- the VR1 dual-metal-plane split, D-141; not VR0's single
  metal plane, and NOT a defect). s3/swift on radosgw VIP .60:443; image-stream HTTP on
  metal .8.192 (gss, expected HTTP). GATE PASS.

## Step 3.3 dashboard-VIP TLS probe (read-only; advisor: NOT deferred) -- MEASURED 2026-08-06
FINDING F-CV3 (LOGGED, not fixed -- hard rule 1): the dc0 dashboard VIP does NOT serve
  TLS on 443. Instrument checked first (advisor rule): OS_CACERT valid (the openstack TLS
  calls above succeeded on it), 443 TCP OPEN on BOTH 10.12.4.58 and 10.12.8.58, so not a
  connectivity/CA artifact. curl (even -k) -> http_code 000, errormsg "OpenSSL...wrong
  version number" == the server on 443 answers PLAINTEXT, not TLS. Plain http :80 root
  -> 200. This is the known D-072 / DOCFIX-089 "Horizon VIP https handshake death" pattern
  (the exact defect the fail-closed rewrite was built to catch after it passed the old
  fail-open gate for weeks). SCOPE: Horizon operator/tenant ACCESS, NOT a core-API blocker
  -- every core API endpoint proved working TLS + a scoped token. NEEDS TRIAGE (likely a
  dashboard-ssl/haproxy-TLS-not-terminating item; appendix-A "Horizon VIP https handshake
  death" + D-072) before Horizon is usable. Deferred with it: the Step 3.3 nginx external
  proxy repoint (genuinely VR0-external-proxy topology / different VR1 edge workstream).
NOTE: D-044 Secure-cookie override and D-075 root redirect (both PER-REBUILD apache
  mutations) are NOT applied this rebuild -- root http :80 -> 200 (not the D-075 302 to
  /horizon). Expected (per-rebuild steps not yet run), recorded so it is not mistaken for
  a defect; they belong with the Horizon-access remediation once F-CV3 clears.

## TLS-LAYER DISCRIMINATION (read-only, 2026-08-06) -- routes F-CV1 + F-CV3
Advisor-directed: characterize the LAYER before routing (a citation is an existence
claim; do not pattern-match the root cause -- the ovn saga was called wrong 3x).
DASHBOARD (openstack-dashboard/leader):
  - /etc/apache2/ssl/horizon/ EXISTS, dir mtime Aug 5 02:06 (== cert-cascade time). So
    certs were delivered -- NOT a bare cascade gap.
  - haproxy `bind *:443` with NO `ssl crt` clause; listeners 443 (haproxy), 433 + 70 (apache).
  - curl -k to 127.0.0.1 :70, :433 AND :443 ALL return "wrong version number" == every
    port answers PLAINTEXT. Apache's own :433 (the intended SSL backend) is plaintext too.
  - ROUTING: NOT the classic D-045 (haproxy-not-reloaded) -- the apache backend itself is
    plaintext. NOT the ovn cascade gap -- certs are present. It is an apache-SSL-NOT-ACTIVE
    condition (ssl vhost/module or charm ssl wiring). Needs its own triage; do NOT file it
    under D-072 by pattern-match (my earlier F-CV3 note did -- corrected here).
DESIGNATE (designate/leader):
  - designate-api process ACTIVE/up (systemctl active, pid running). So the earlier
    sweep DOWN is NOT "api down".
  - designate-api LISTENS PLAINTEXT on :8991; haproxy backend uses `server ... :8991 check
    check-ssl verify none` -- haproxy attempts an SSL handshake to a plaintext backend ->
    L6RSP "SSL handshake failure" -> backend DOWN. This is a TLS-PRESENTATION MISMATCH,
    same SHAPE as the dashboard.
CORRECTION to F-CV1: my "plausibly collateral of the Stage-7 nameservers block" guess is
  WRONG -- designate-api IS up; the DOWN is a plaintext-vs-check-ssl mismatch, not a
  not-yet-activated API. Whether the mismatch clears at Stage-7 activation is now an OPEN
  question, not an assumption. Re-check at Stage-7 STILL applies, but the classification
  "collateral" is RETRACTED.
COMMON SHAPE (F-CV1 + F-CV3): two services (designate-api, horizon-apache) serve plaintext
  while their TLS layer expects SSL, on a cloud where the OTHER 12 VIP apps serve TLS
  correctly. NOT asserted to be one identical root cause (dashboard = apache-ssl-inactive;
  designate = api-plaintext-vs-haproxy-check-ssl), but the shape is shared and worth a
  single triage thread. LOGGED, not fixed (hard rule 1).

## TRIAGE (operator-authorized 2026-08-06) F-CV1 + F-CV3 -- read-only; NOT fixed (hard rule 1)
CONFIRMED:
  - Both apps HAVE the certificates->vault relation (tls-certificates, global), same as the
    12 working apps. NOT a missing relation.
  - Both have vault certs RENDERED: /etc/apache2/ssl/horizon/cert_juju-d4f45a-0-lxd-14.maas
    and /etc/apache2/ssl/designate/cert_juju-d4f45a-0-lxd-5.maas (+ per-IP symlinks incl the
    VIPs), dated Aug 5 (cascade). NOT a cert-issuance/empty-CN gap (unlike the ovn saga).
  - Both serve their SSL backend via apache/mod_wsgi; ssl_module loaded.
  - WORKING REFERENCE glance/leader: TLS via the charm vhost
    /etc/apache2/sites-available/openstack_https_frontend.conf (SSLEngine on).
  - DASHBOARD: apache :433 (SSL backend) is served by /etc/apache2/sites-enabled/
    default-ssl.conf (the UBUNTU DEFAULT vhost), binding the VIP addrs -- NOT the charm's
    openstack_https_frontend.conf. Measured PLAINTEXT on :70/:433/:443.
  - DESIGNATE: apache :8991 served by openstack_https_frontend.conf bound 10.12.8.198:8991
    (metal-admin), but haproxy backend targets 10.12.12.110:8991 (metal-internal) ->
    possible BIND-PLANE MISMATCH (apache may not present SSL on the addr haproxy dials).
    designate.conf `listen = 0.0.0.0:8981` is the inner api; apache fronts :8991.
INSTRUMENT CAVEAT (own it): an earlier `grep -r ... /etc/apache2/sites-enabled/` returned
  "no SSLEngine" -- FALSE NEGATIVE. `grep -r` does NOT follow the SYMLINKS that populate
  sites-enabled/, so it read nothing. `apache2ctl -S` is authoritative and corrected it.
NOT NAILED (do NOT assert one root cause -- the ovn root cause was called wrong 3x):
  exact reason apache serves plaintext despite ssl vhosts present. Candidates: charm https
  frontend not effective/enabled (dashboard uses default-ssl instead); bind-plane mismatch
  (designate); apache not reloaded after cert render; an ssl config option off. Two DISTINCT
  manifestations, one CLASS: the charm apache TLS-frontend is not effective on the address
  the client/haproxy actually uses.
ROUTING / PROPOSED REMEDIATION (to be individually gated, recommend a FOCUSED session):
  charm-config-layer, not infra. Likely path: investigate why the openstack-dashboard charm
  did not render/enable its https frontend (vs glance), and whether designate's frontend
  binds all required planes; then re-trigger the charm to re-render + reload apache (e.g.
  `juju exec ... hooks/config-changed` or a relation refresh), verify apache serves SSL on
  the haproxy-dialed address, then re-run the haproxy sweep + the dashboard fail-closed probe.
  NOT attempted live this session (verification scope; remediation needs its own gate).
  Re-check F-CV1 also at Stage-7 designate activation.

CONFIRMED 2026-08-06 (advisor-directed, read the vhost file directly -- apache2ctl -S
had been head-truncated): DESIGNATE is a BIND-PLANE MISMATCH, now a NAMED defect:
  - designate/0 apache https frontend = EXACTLY ONE `<VirtualHost 10.12.8.198:8991>`
    (metal-admin; the unit's own .8 address).
  - haproxy has TWO designate backends: `designate-api_admin_10.12.12.110` dials
    designate-0 at 10.12.12.110:8991 (METAL-INTERNAL); a second backend dials 10.12.8.198
    (metal-admin). The sweep DOWN was the _admin_10.12.12.110 one -- apache has NO SSL vhost
    on the metal-internal address, so haproxy's check-ssl there hits plaintext -> DOWN.
  - ROOT CAUSE (designate): apache presents SSL only on metal-admin (.8.198); haproxy's
    admin backend dials metal-internal (.12.110). VR1 dual-metal-plane specific (D-141 split
    of metal-admin .8 vs metal-internal .12). Fix shape (focused session, gated): align the
    plane -- either the charm binds the metal-internal address too, or the admin backend
    dials metal-admin; the correct plane for the ADMIN interface is a D-141/B1 question.
  - This is STRUCTURAL, will NOT self-resolve at Stage-7 nameserver config -> my original
    "collateral of the block" reading is definitively wrong (retraction stands).

>>> F-CV1 and F-CV3 ARE TWO SEPARATE FINDINGS, not one (advisor): F-CV1 = designate
    bind-plane mismatch (charm frontend present, wrong/one plane); F-CV3 = dashboard SSL
    backend served by Ubuntu default-ssl.conf, charm https frontend not effective (different
    question, no one-command discriminator, left as recorded). Do NOT chase a common fix.

## RECONCILIATION 2026-08-06 (after prior-art + governing-decision research; advisor-directed)
Two CORRECTIONS to the F-CV1 framing above -- the earlier commits (aaeee93, CURRENT-STATE)
called it "CONFIRMED bind-plane mismatch (D-141)" BEFORE checking the governing decisions;
that ordering was backwards (memory #17). The check does not overturn the finding but
RESCOPES it and RE-ATTRIBUTES it:
 (1) ATTRIBUTION: NOT D-141 (that is dual-stack IPAM). The governing surface is D-052
     (binding topology) + the generic rule in docs/network-space-binding-reference.md
     Section 2/3: `''` default = metal-admin (OAM/mgmt FALLBACK, line 61); `:public` ->
     provider-public; internal API -> metal-internal.
 (2) SCOPE: split into a CONFIRMED half and an OPEN half:
     - `:internal` -> metal-internal is a CONFIRMED DEFECT. Every sibling API app binds
       `:internal` to metal-internal (glance m-internal=11, keystone=13, ...); designate's
       sits on the metal-admin FALLBACK (an omission, not a choice). designate HAS a
       metal-internal VIP (10.12.12.62) and its cert ALREADY carries the metal-internal SANs
       (10.12.12.110 + .62), but apache builds an SSL vhost only on metal-admin -> haproxy's
       _admin backend at 10.12.12.110 hits no vhost -> DOWN. No ruled exception covers it.
     - `:public` -> provider-public is OPEN (operator ruling needed). designate uniquely has
       prov-pub=1 = `:dnsaas` (the D-106 dual-VIP exception, ruled), NOT `:public`. The
       exceptions table (line 156) frames dnsaas as designate's public-facing surface, which
       is consistent with an intent that designate's REST API is metal-admin-only and tenants
       consume DNS not the API. The cert does NOT cover provider-public. So moving `:public`
       may be right or may be against design -- ASK, do not assume glance's pattern.
 (3) The Section-4 per-app table (line 88) is "Generated from bundle.yaml" (line 68) --
     DESCRIPTIVE of the current (defective) bundle, NOT prescriptive intent. It is not
     evidence that designate's bindings are as-designed.
BLOCKING CHECKS FOR THE FIX (advisor): (a) all 3 designate units HAVE metal-internal addrs
 (10.12.12.110/.144/.143) -- `juju bind internal=metal-internal` will NOT be refused, no
 --force; (b) cert already covers the metal-internal SANs -> low reissue risk, but RE-READ
 the issued cert SANs AND do haproxy-backend-readback + https-200 after (D-072 precedent
 method; assert the artifact not the config).
PROPOSED FIX (gated, together -- bundle + live, per D-072): add `internal: metal-internal`
 to designate's bundle.yaml bindings + `juju bind designate internal=metal-internal`; verify
 haproxy _admin backend readback moves onto a served address + comes UP, re-read cert SANs.
 `:public` deferred to the operator ruling. designate is Stage-7-blocked (not serving) ->
 no urgency; do it correctly, not fast.

## Step 3.4 keystone domain-manager policy -- NOT RUN this session
  Deferred with the openstack-client-dependent verification depth; the PO: stage-1 check +
  the C.4 G3 behavioral probe (which mutates -- creates user/project) are owed. Recorded as
  NOT RUN (not as passed), per the runbook's own "PO: proves parse, not that policy works".

## ============ STEP 7 / phase-03 EXIT GATE: NOT MET ============
GA-R6/E3: no conditional close -- the remainder keeps the step OPEN (or splits to its
own gate row). Verified vs the runbook EXIT GATE:
  [PASS] Cloud settled: only expected Stage-6/7 blocks (falsifiable, matched prediction).
  [PASS] haproxy backends UP -- for the 12 ACTIVATED VIP apps (designate = F-CV1).
  [PASS] admin-openrc scoped token; endpoint list IP-only; two-source keystone VIP.
  [PASS] vault root CA validates TLS to the keystone VIP (.4.50).
  [FAIL] Horizon reachable AND login works -- F-CV3, dashboard VIP serves plaintext on 443.
  [NOT RUN] Step 3.4 domain-manager policy (PO: stage-1 + G3 behavioral).
CORE-API layer: VERIFIED. Step 7 as a whole: OPEN (2 named items: F-CV3, Step 3.4).

## OWED (survive-a-clear):
  O1. CURRENT-STATE section 7 OpenStack-client row: now STALE. python3-openstackclient
      6.6.0-0ubuntu2 is INSTALLED ON THE dc0 RACK (this session) -- the row says client is
      on voffice1 only. GA-R1/C1: update in the same commit that records this. (Not churn --
      a real measured status change.)
  O2. dc1 rack needs the SAME python3-openstackclient install before dc1's Step 7 (F-CV2
      is per-DC; only dc0 done).
  O3. F-CV3 (dashboard apache-SSL-inactive) + F-CV1 (designate api-plaintext-vs-check-ssl):
      operator decision -- triage now, or defer with the Horizon-access workstream.
  O4. Step 3.4 domain-manager policy gate (PO: + G3) still owed for phase-03 close.
  O5. lib_net_select_dc DOCFIX candidate (below).

## DOCFIX candidate (advisor -- log now while looking): phase-03-admin-openrc.sh,
  phase-04-network-{create,verify}.sh, phase-04-internal-cert-san-verify.sh and
  vault-kv-health.sh read DC-dependent lib-net values WITHOUT calling lib_net_select_dc
  (measured 07-29, dc-dc-phase4 Steps 7-9 caveat). Harmless on dc0 (resolves dc0's
  literals); on dc1 it is WRONG and SILENT. Fix before dc1's Step 7, or export every value
  explicitly. This session used explicit KEYSTONE_VIP=10.12.4.50, so unaffected.
