# Stage 5 dc0 -- Step 7: phase-03 core verify (ADAPTED for vr1-dc0), read-only pass
# Runbook: runbooks/phase-03-core-verify.md, adapted per dc-dc-phase4 Steps 7-9.
# Model: vr1-dc0. Run-location: dc0 rack (172.31.0.2 via ProxyJump voffice1) -- D-138.
# Started 2026-08-06. Author: Claude Code (background session), operator "continue".
#
# DISCIPLINE: hard rule 1 -- this is VERIFICATION. Findings are LOGGED here, not
# fixed mid-step (the sole sanctioned in-step remediation is the 3.1 haproxy
# `reload`, individually gated). Prediction is written BEFORE reading live status
# so the settle gate is falsifiable (advisor 2026-08-06; memory #17 / GA-R6
# "a checker that cannot fail is not a gate").
#
# =====================================================================
# PREDICTION (written 2026-08-06 BEFORE any `juju status` this session)
# =====================================================================
# Source of the prediction:
#   - 08-06 part-2 close: "14/14 HA ... 12 active/idle + 2 known-blocked
#     (octavia/designate)".
#   - 08-05 close: designate/octavia/rbd-mirror correctly-waiting on later stages.
#   - D-021: octavia BLOCKED "Awaiting configure-resources" until phase-05 (Stage 6).
#   - Stage-7 (dc-dc-phase6): designate activation.
#   - Stage-6 (dc-dc-phase5): ceph-rbd-mirror / radosgw multisite DR wiring.
#
# EXPECTED non-active/idle at this pass (the prediction to diff against):
#   E1. octavia/*            -- BLOCKED, D-021 (awaiting configure-resources; Stage 6)
#   E2. designate/*          -- waiting/blocked, Stage-7 activation
#   E3. ceph-rbd-mirror/*    -- waiting/blocked, Stage-6 DR wiring
#   E4. (tolerated-transient) glance-simplestreams-sync -- pre-run, if present
#
# DIFF RULE (the gate):
#   - Any live non-active/idle unit NOT in {E1,E2,E3,E4}  => FINDING (log below).
#   - Any unit in {E1,E2,E3} that came back ACTIVE/IDLE   => FINDING (record stale).
#   - octavia downstream-of-ovn note (08-04): ovn-central is now RESOLVED
#     (08-05), so octavia's block should be its OWN D-021 block, not an ovn
#     cascade. If octavia is waiting on ovn/cert, that is a FINDING.
#
# =====================================================================
# MEASUREMENT (filled in below, this session)
# =====================================================================

## Step 3.1 acceptance walk -- MEASURED 2026-08-06 (read-only)
[vr1-dc0-rack] juju status -m vr1-dc0 --format=yaml  (rc=0, 223979 bytes, 57 apps, 162 units)
Processed on jumphost. Non-active/idle units: 8 of 162 (154 active/idle):
  ceph-rbd-mirror/0            blocked  idle  'ceph-local' incomplete, 'ceph-remote' missing   [E3 Stage-6 DR]
  designate/0,1,2              blocked  idle  nameservers must be set                          [E2 Stage-7]
  glance-simplestreams-sync/0  unknown  idle  (empty)                                          [E4 transient]
  octavia/0                    blocked  idle  Awaiting end-user execution of `configure-resources` [E1 D-021]
  octavia/1,2                  blocked  idle  Awaiting leader to create required resources        [E1 D-021]

DIFF vs PREDICTION: MATCH. All 8 in {E1,E2,E3,E4}; none outside; none expected-blocked
came back active. GATE PASS (falsifiable -- prediction pre-dated the read).
NOTE: octavia/0 message is its OWN D-021 configure-resources block, NOT an ovn/cert
cascade -- confirms ovn-central RESOLVED (08-05) end-to-end; the 08-04 "octavia
downstream-of-ovn" concern is CLEARED.

## Step 3.1 haproxy backend sweep (DOCFIX-031/D-045) -- MEASURED 2026-08-06 (read-only)
Scoped to the 13 VIP-fronted apps (those carrying an *-hacluster subordinate),
39 units. Loop run ON the rack (juju ssh controller-local).
RESULT: 3 DOWN lines, ALL on designate (the known Stage-7 block, E2):
  [designate/0] designate-api_admin_10.12.12.110:8991  DOWN  L6RSP "SSL handshake failure / Layer6 invalid response"
  [designate/1] designate-api_admin_10.12.12.144:8991  DOWN  L6RSP  (same)
  [designate/2] designate-api_admin_10.12.12.143:8991  DOWN  L6RSP  (same)
CLEAN for all 12 ACTIVATED VIP apps: barbican ceph-radosgw cinder glance keystone
  magnum neutron-api nova-cloud-controller octavia openstack-dashboard placement vault
  -> zero DOWN.

FINDING F-CV1 (LOGGED, not fixed -- hard rule 1): designate-api _admin backend DOWN
  with an SSL-handshake/L6 signature on all 3 units. designate is BLOCKED pre-Stage-7
  ("nameservers must be set"), so this is plausibly collateral of the un-activated API,
  NOT necessarily the D-045 plaintext-vs-SSL defect. The sanctioned in-step haproxy
  `reload` is WITHHELD: designate is not a healthy app, and reload would not clear a
  Stage-7 activation gap. ACTION: re-run this sweep against designate AFTER Stage-7
  activation (dc-dc-phase6); if the L6/SSL DOWN persists on an ACTIVE designate, THAT
  is a D-045-class defect to remediate then. Note the signature here is L6RSP (SSL
  handshake), which differs from D-045's classic L7STS/400 plaintext-vs-SSL.

GATE VERDICT: PASS for the 12 activated VIP apps (the phase-03 core-verify scope at
  this pre-Stage-6/7 point). designate deferred to its Stage-7 activation, consistent
  with the settle-walk E2 classification.

## Step 3.2 admin-openrc -- MEASURED 2026-08-06
Config source (dc0 vips overlay line 36): keystone provider VIP = 10.12.4.50
  (v4-only D-141 triple ".50 / .8.50 / .12.50").
Rack state: admin-openrc + vault root CA both ABSENT (fresh build, nothing to clobber;
  API reachability never yet verified via openrc on this deploy). phase-03-admin-openrc.sh
  + extract_admin_password.py staged to rack, sha256 MATCH (2aeb20b720d83944 /
  7b9b25a8fafd75cb); lib-net.sh dep already staged, MATCH 3e465ae7de3bf92d.

BLOCKER F-CV2 (== the KNOWN 07-30 queued-finding F1, HIT at Step 7) -- RESOLVED this session:
  First run -> "FAIL: openstack not found". The openstack CLI was ABSENT on the dc0 rack
  (no snap/apt/venv). D-138 puts phase-03..06's openstack CLI ON THE RACK, but the 07-27
  install landed it only on voffice1 (which cannot reach keystone's provider VIP L3).
  Phase-01/02 were juju-only, so it first surfaced here. Documented remediation (07-30 F1):
  "Install on the DC client host before Step 7." EXECUTED (gated): apt-get install -y
  python3-openstackclient on the rack -> 6.6.0-0ubuntu2 (+ python3-openstacksdk
  3.0.0-0ubuntu2), matching the measured 07-27 pin exactly; snap correctly refuted, no UCA.
  REVERT: sudo apt-get purge -y python3-openstackclient on the rack.
  OWED DOC: add the dc0 rack to CURRENT-STATE section 7's OpenStack-client row (per F1's
  own instruction). Same install is OWED on the dc1 rack before dc1's Step 7.
  NOTE (pending kernel): rack reports 6.8.0-136 running vs 6.8.0-137 available -- NOT acted
  on (a reboot would bounce the rack + libvirt + all nodes); logged for a maintenance window.

BUILD (after unblock): MODEL=vr1-dc0 KEYSTONE_VIP=10.12.4.50 phase-03-admin-openrc.sh ->
  vault root CA subject "Vault Root Certificate Authority (charm-pki-local)", notAfter
  Aug 2 2036 (matches 08-05 init); admin project = admin (password len 16); wrote openrc
  0600; **[OK] scoped token issued** against https://10.12.4.50:5000/v3.

TWO-SOURCE VIP (advisor): CONFIRMED. Config (overlay) keystone provider VIP = 10.12.4.50;
  live artifact `openstack endpoint list --service keystone`:
    public   https://10.12.4.50:5000/v3     (provider)
    admin    https://10.12.8.50:35357/v3    (metal-admin, :35357)
    internal https://10.12.12.50:5000/v3    (metal-internal)
  Config == artifact. IP-only endpoint list PASS across ALL services (public .4.x /
  admin .8.x / internal .12.x -- the VR1 dual-metal-plane split, D-141; not VR0's single
  metal plane, and NOT a defect). s3/swift on radosgw VIP .60:443; image-stream HTTP on
  metal .8.192 (gss, expected HTTP). GATE PASS.

## Step 3.3 dashboard-VIP TLS probe (read-only; advisor: NOT deferred) -- MEASURED 2026-08-06
FINDING F-CV3 (LOGGED, not fixed -- hard rule 1): the dc0 dashboard VIP does NOT serve
  TLS on 443. Instrument checked first (advisor rule): OS_CACERT valid (the openstack TLS
  calls above succeeded on it), 443 TCP OPEN on BOTH 10.12.4.58 and 10.12.8.58, so not a
  connectivity/CA artifact. curl (even -k) -> http_code 000, errormsg "OpenSSL...wrong
  version number" == the server on 443 answers PLAINTEXT, not TLS. Plain http :80 root
  -> 200. This is the known D-072 / DOCFIX-089 "Horizon VIP https handshake death" pattern
  (the exact defect the fail-closed rewrite was built to catch after it passed the old
  fail-open gate for weeks). SCOPE: Horizon operator/tenant ACCESS, NOT a core-API blocker
  -- every core API endpoint proved working TLS + a scoped token. NEEDS TRIAGE (likely a
  dashboard-ssl/haproxy-TLS-not-terminating item; appendix-A "Horizon VIP https handshake
  death" + D-072) before Horizon is usable. Deferred with it: the Step 3.3 nginx external
  proxy repoint (genuinely VR0-external-proxy topology / different VR1 edge workstream).
NOTE: D-044 Secure-cookie override and D-075 root redirect (both PER-REBUILD apache
  mutations) are NOT applied this rebuild -- root http :80 -> 200 (not the D-075 302 to
  /horizon). Expected (per-rebuild steps not yet run), recorded so it is not mistaken for
  a defect; they belong with the Horizon-access remediation once F-CV3 clears.

## TLS-LAYER DISCRIMINATION (read-only, 2026-08-06) -- routes F-CV1 + F-CV3
Advisor-directed: characterize the LAYER before routing (a citation is an existence
claim; do not pattern-match the root cause -- the ovn saga was called wrong 3x).
DASHBOARD (openstack-dashboard/leader):
  - /etc/apache2/ssl/horizon/ EXISTS, dir mtime Aug 5 02:06 (== cert-cascade time). So
    certs were delivered -- NOT a bare cascade gap.
  - haproxy `bind *:443` with NO `ssl crt` clause; listeners 443 (haproxy), 433 + 70 (apache).
  - curl -k to 127.0.0.1 :70, :433 AND :443 ALL return "wrong version number" == every
    port answers PLAINTEXT. Apache's own :433 (the intended SSL backend) is plaintext too.
  - ROUTING: NOT the classic D-045 (haproxy-not-reloaded) -- the apache backend itself is
    plaintext. NOT the ovn cascade gap -- certs are present. It is an apache-SSL-NOT-ACTIVE
    condition (ssl vhost/module or charm ssl wiring). Needs its own triage; do NOT file it
    under D-072 by pattern-match (my earlier F-CV3 note did -- corrected here).
DESIGNATE (designate/leader):
  - designate-api process ACTIVE/up (systemctl active, pid running). So the earlier
    sweep DOWN is NOT "api down".
  - designate-api LISTENS PLAINTEXT on :8991; haproxy backend uses `server ... :8991 check
    check-ssl verify none` -- haproxy attempts an SSL handshake to a plaintext backend ->
    L6RSP "SSL handshake failure" -> backend DOWN. This is a TLS-PRESENTATION MISMATCH,
    same SHAPE as the dashboard.
CORRECTION to F-CV1: my "plausibly collateral of the Stage-7 nameservers block" guess is
  WRONG -- designate-api IS up; the DOWN is a plaintext-vs-check-ssl mismatch, not a
  not-yet-activated API. Whether the mismatch clears at Stage-7 activation is now an OPEN
  question, not an assumption. Re-check at Stage-7 STILL applies, but the classification
  "collateral" is RETRACTED.
COMMON SHAPE (F-CV1 + F-CV3): two services (designate-api, horizon-apache) serve plaintext
  while their TLS layer expects SSL, on a cloud where the OTHER 12 VIP apps serve TLS
  correctly. NOT asserted to be one identical root cause (dashboard = apache-ssl-inactive;
  designate = api-plaintext-vs-haproxy-check-ssl), but the shape is shared and worth a
  single triage thread. LOGGED, not fixed (hard rule 1).

## Step 3.4 keystone domain-manager policy -- NOT RUN this session
  Deferred with the openstack-client-dependent verification depth; the PO: stage-1 check +
  the C.4 G3 behavioral probe (which mutates -- creates user/project) are owed. Recorded as
  NOT RUN (not as passed), per the runbook's own "PO: proves parse, not that policy works".

## ============ STEP 7 / phase-03 EXIT GATE: NOT MET ============
GA-R6/E3: no conditional close -- the remainder keeps the step OPEN (or splits to its
own gate row). Verified vs the runbook EXIT GATE:
  [PASS] Cloud settled: only expected Stage-6/7 blocks (falsifiable, matched prediction).
  [PASS] haproxy backends UP -- for the 12 ACTIVATED VIP apps (designate = F-CV1).
  [PASS] admin-openrc scoped token; endpoint list IP-only; two-source keystone VIP.
  [PASS] vault root CA validates TLS to the keystone VIP (.4.50).
  [FAIL] Horizon reachable AND login works -- F-CV3, dashboard VIP serves plaintext on 443.
  [NOT RUN] Step 3.4 domain-manager policy (PO: stage-1 + G3 behavioral).
CORE-API layer: VERIFIED. Step 7 as a whole: OPEN (2 named items: F-CV3, Step 3.4).

## OWED (survive-a-clear):
  O1. CURRENT-STATE section 7 OpenStack-client row: now STALE. python3-openstackclient
      6.6.0-0ubuntu2 is INSTALLED ON THE dc0 RACK (this session) -- the row says client is
      on voffice1 only. GA-R1/C1: update in the same commit that records this. (Not churn --
      a real measured status change.)
  O2. dc1 rack needs the SAME python3-openstackclient install before dc1's Step 7 (F-CV2
      is per-DC; only dc0 done).
  O3. F-CV3 (dashboard apache-SSL-inactive) + F-CV1 (designate api-plaintext-vs-check-ssl):
      operator decision -- triage now, or defer with the Horizon-access workstream.
  O4. Step 3.4 domain-manager policy gate (PO: + G3) still owed for phase-03 close.
  O5. lib_net_select_dc DOCFIX candidate (below).

## DOCFIX candidate (advisor -- log now while looking): phase-03-admin-openrc.sh,
  phase-04-network-{create,verify}.sh, phase-04-internal-cert-san-verify.sh and
  vault-kv-health.sh read DC-dependent lib-net values WITHOUT calling lib_net_select_dc
  (measured 07-29, dc-dc-phase4 Steps 7-9 caveat). Harmless on dc0 (resolves dc0's
  literals); on dc1 it is WRONG and SILENT. Fix before dc1's Step 7, or export every value
  explicitly. This session used explicit KEYSTONE_VIP=10.12.4.50, so unaffected.
