# Phase 03 -- Core Verify (settle, admin-openrc, Horizon)

After vault's cert cascade (phase-02), confirm the cloud settled to active/idle
(except the expected post-deploy blocks), build the IP-only `admin-openrc`, verify
API reachability, and repoint the external Horizon reverse proxy.

Decisions: B5 (IP-only endpoints; no FQDN), D-021 (octavia stays BLOCKED awaiting
configure-resources -- expected, cleared in phase-05). Troubleshooting: appendix-A --
DOCFIX-021 (action human-output corrupts captured artifacts), DOCFIX-018 (IP-only
OS_AUTH_URL), DOCFIX-022 (admin project discovered, not hardcoded).

---

## Prerequisites (must be true entering phase-03)
- phase-02 done: vault unsealed + authorized; root CA generated; the cert cascade is
  running/settling.
- The Vault root CA is available via the vault charm action (pulled below).

## Constants and env-literals (TAG: confirm per site on rebuild)
- `ENV(keystone-vip)` 10.12.4.50      (keystone PUBLIC endpoint = provider VIP; verify vs bundle)
- `ENV(admin-domain)` admin_domain    (charmed-keystone admin user + project domain)
- `ENV(dashboard-vip)` 10.12.4.58     (Horizon provider VIP; was .234 pre-R14)
- admin project: DISCOVERED at runtime (do not hardcode -- DOCFIX-022).

## Run-location legend
- `# RUN: jumphost` -- vopenstack-jesse as jessea123; `juju` + `openstack` + `openssl`.

---

## Step 3.1 -- Settle the cert cascade + acceptance walk
`# RUN: jumphost`  The cascade here is NARROW (mysql bootstrapped before vault init,
so only the Vault consumers clear: ovn-central x3, ovn-chassis x3,
ovn-chassis-octavia, neutron-api-plugin-ovn, barbican-vault). Watch, then walk units
AND subordinates.
```bash
juju status --color --watch 30s -m openstack     # Ctrl-C once settled
```
Acceptance walk (counts non-active/idle across units + subordinates):
```bash
juju status -m openstack --format=yaml | python3 -c "
import yaml,sys
d=yaml.safe_load(sys.stdin); apps=d.get('applications',{}); bad=[]
def chk(n,u):
    ws=(u.get('workload-status') or {}).get('current',''); js=(u.get('juju-status') or {}).get('current','')
    msg=(u.get('workload-status') or {}).get('message','')
    if ws!='active' or js!='idle': bad.append('%s: workload=%s juju=%s msg=%s'%(n,ws,js,msg))
for app,info in apps.items():
    for un,ud in (info.get('units') or {}).items():
        chk(un,ud)
        for sn,sd in (ud.get('subordinates') or {}).items(): chk(sn,sd)
print('Non-active/idle units: %d'%len(bad))
for b in bad: print('  '+b)
"
```
GATE: expected non-active/idle = **1** (octavia/0 BLOCKED "Awaiting configure-resources",
the D-021 next step) or briefly **2** (+ glance-simplestreams-sync, normal pre-run).
Any TLS consumer (the five above) persisting waiting/error past ~15 min is the concern
-- STOP and read its log + relations (do NOT assume TLS; a prior stall was a MySQL 1045
desync):
```bash
juju status --relations -m openstack ovn-central ovn-chassis ovn-chassis-octavia neutron-api-plugin-ovn barbican-vault
# juju ssh -m openstack <unit> -- 'sudo tail -120 /var/log/juju/unit-<unit-dashed>.log' </dev/null
```

## Step 3.2 -- Build admin-openrc (IP-only; canonical block)
`# RUN: jumphost`  Keystone PUBLIC = the provider VIP IP over HTTPS with the vault
CA (no FQDN, no /etc/hosts -- B5). This canonical block folds in three fixes:
the CA is pulled via `--format json` + jq because the action's human output wraps the
PEM in an INDENTED YAML block that is not valid PEM (appendix-A: DOCFIX-021); the
OS_AUTH_URL is the VIP IP (DOCFIX-018); and the admin project is DISCOVERED by a
scope-test loop rather than hardcoded, because the scoping project name varies by
charm rev (DOCFIX-022 -- the cause of a prior HTTP 401). `( set -e )` keeps
OS_PASSWORD inside the subshell and aborts cleanly on any failure.

```bash
KEYSTONE_VIP=10.12.4.50              # keystone PUBLIC endpoint = provider VIP (verify vs bundle on rebuild)
ADMIN_DOMAIN=admin_domain            # charmed-keystone admin user + project domain
PROJECT_CANDIDATES="admin admin_domain"   # tried in order; first that SCOPES wins (DOCFIX-022 variance)
CA="$HOME/vault-init/vault-ca-root.pem"
RC="$HOME/admin-openrc"

( set -e
  mkdir -p "$HOME/vault-init"
  # 1. Vault root CA -> file (JSON extract; DOCFIX-021 -- human output indents the PEM)
  juju run vault/leader get-root-ca -m openstack --format json \
    | jq -r '[.. | strings | select(test("-----BEGIN CERTIFICATE-----"))][0]' > "$CA"
  openssl x509 -in "$CA" -noout -subject -dates
  # 2. Admin password -> var (JSON extract, not human output)
  ADMIN_PASS=$(juju run keystone/leader get-admin-password -m openstack --format json | python3 -c "
import json,sys
d=json.load(sys.stdin)
def f(o):
    if isinstance(o,dict):
        for k in ('admin-password','password','Stdout'):
            if k in o and o[k]: return str(o[k]).strip()
        for v in o.values():
            r=f(v)
            if r: return r
    elif isinstance(o,list):
        for v in o:
            r=f(v)
            if r: return r
    return ''
print(f(d))
")
  [ -n "$ADMIN_PASS" ] || { echo "FATAL: password extract failed"; exit 1; }
  # 3. PROJECT LOOKUP: first candidate that issues a SCOPED token wins (DOCFIX-022)
  export OS_AUTH_URL="https://${KEYSTONE_VIP}:5000/v3" OS_USERNAME=admin OS_PASSWORD="$ADMIN_PASS"
  export OS_USER_DOMAIN_NAME="$ADMIN_DOMAIN" OS_PROJECT_DOMAIN_NAME="$ADMIN_DOMAIN"
  export OS_IDENTITY_API_VERSION=3 OS_REGION_NAME=RegionOne OS_CACERT="$CA"
  ADMIN_PROJECT=""
  for P in $PROJECT_CANDIDATES; do
    if OS_PROJECT_NAME="$P" openstack token issue >/dev/null 2>&1; then ADMIN_PROJECT="$P"; break; fi
  done
  [ -n "$ADMIN_PROJECT" ] || { echo "FATAL: no candidate project scoped (tried: $PROJECT_CANDIDATES)"; exit 1; }
  echo "[OK] admin project = $ADMIN_PROJECT ; password len ${#ADMIN_PASS}"
  # 4. Write ~/admin-openrc (backs up any existing one first)
  [ -f "$RC" ] && mv "$RC" "$RC.pre-$(date -u +%Y%m%dT%H%M%SZ)"
  cat > "$RC" <<EOF
export OS_AUTH_URL=https://${KEYSTONE_VIP}:5000/v3
export OS_USERNAME=admin
export OS_PASSWORD='$ADMIN_PASS'
export OS_PROJECT_NAME=$ADMIN_PROJECT
export OS_USER_DOMAIN_NAME=$ADMIN_DOMAIN
export OS_PROJECT_DOMAIN_NAME=$ADMIN_DOMAIN
export OS_IDENTITY_API_VERSION=3
export OS_REGION_NAME=RegionOne
export OS_CACERT=$CA
EOF
  chmod 600 "$RC"
)
# 5. Verify from the written file (password stayed inside the subshell above)
( source "$RC"; echo "auth -> $OS_AUTH_URL  project=$OS_PROJECT_NAME"; openstack token issue 2>&1 | head -6 )
( source "$RC"; openstack endpoint list -f value -c "Service Name" -c Interface -c URL 2>&1 | sort )
```
GATE: `token issue` returns a SCOPED token; `endpoint list` is IP-only across all
services (public on the provider VIP `.5x`, internal+admin on the metal VIP `.8.5x`,
keystone admin on `:35357`). Two non-blocking notes for later: s3/swift is registered
on the radosgw VIP `.60:443` (re-check vs the radosgw `:80` listener during any
Swift/S3 smoke); the gss image-stream is HTTP on metal `10.12.8.172`.

## Step 3.3 -- Horizon access via the external nginx reverse proxy
`# RUN: operator (outside the Juju model)`  Horizon is fronted by an
operator-managed nginx reverse proxy. On each rebuild / VIP relocation, repoint its
upstream to the CURRENT dashboard provider VIP (now `https://10.12.4.58`, was `.234`
pre-R14). Verify two interplays:
- ALLOWED_HOSTS: Horizon (bundle B5 setting) must permit whatever Host header reaches
  it, else HTTP 400 DisallowedHost. Either set the proxy `proxy_set_header Host` to the
  dashboard VIP, or add the proxy hostname to Horizon ALLOWED_HOSTS.
- Upstream TLS: the dashboard cert is vault-signed for the VIP IP (IP-SAN). The proxy
  must trust the vault root CA (`~/vault-init/vault-ca-root.pem`) for `proxy_ssl_verify`,
  or terminate/re-encrypt per policy.
LIVE-REVIEW: the proxy host + config path + reload command are operator-managed and
not captured here -- record them verbatim when wired, and confirm an external GET
reaches the Horizon login. (Roosevelt: this repoint folds into the access/DNS workstream.)

---

## EXIT GATE (phase-03 complete)
- Cloud settled: acceptance walk shows only the expected block(s) (octavia; maybe gss).
- `~/admin-openrc` (0600) authenticates and returns a SCOPED token; endpoint list IP-only.
- Vault root CA at `~/vault-init/vault-ca-root.pem` validates TLS to the keystone VIP.
- Horizon reachable through the repointed reverse proxy.

## As-built reference (2026-06-03 run -- audit trail)
- Cascade settled ~04:15Z: all five Vault consumers active/idle; only expected
  non-active/idle = octavia (blocked, D-021) + gss (pre-run). mysql primary on
  mysql-innodb-cluster/1 (R/W), /0+/2 R/O (normal innodb-cluster).
- admin-openrc IP-only: OS_AUTH_URL=https://10.12.4.50:5000/v3, OS_USERNAME=admin,
  OS_PROJECT_NAME=admin (scoped; project_id 65ce73e6798e4d1e8dd066609b7033ef),
  domains admin_domain, OS_CACERT=~/vault-init/vault-ca-root.pem.
- Vault root CA: subject "Vault Root Certificate Authority (charm-pki-local)",
  notBefore 2026-06-03, notAfter 2036-05-31; TLS to 10.12.4.50:5000 OK (B5 IP-SAN holds).
- Dashboard VIP 10.12.4.58 (nginx upstream repoint pending operator capture).

## Next
phase-04 -- network carve (external provider network).