Fork: 0
OpenStack / openstack-caracal-ipv4
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-ipv4 / runbooks / phase-03-core-verify.md
@JANeumatrix JANeumatrix 1 day ago 9 KB repo: sanitation sweep -- canonical phase-NN runbook + appendices; design-decisions D-001..D-042; canonical bundle; superseded files moved to remove/ for review
Raw Blame History

Phase 03 -- Core Verify (settle, admin-openrc, Horizon)

After vault's cert cascade (phase-02), confirm the cloud settled to active/idle (except the expected post-deploy blocks), build the IP-only admin-openrc, verify API reachability, and repoint the external Horizon reverse proxy.

Decisions: B5 (IP-only endpoints; no FQDN), D-021 (octavia stays BLOCKED awaiting configure-resources -- expected, cleared in phase-05). Troubleshooting: appendix-A -- DOCFIX-021 (action human-output corrupts captured artifacts), DOCFIX-018 (IP-only OS_AUTH_URL), DOCFIX-022 (admin project discovered, not hardcoded).

Prerequisites (must be true entering phase-03)

  • phase-02 done: vault unsealed + authorized; root CA generated; the cert cascade is running/settling.
  • The Vault root CA is available via the vault charm action (pulled below).

Constants and env-literals (TAG: confirm per site on rebuild)

  • ENV(keystone-vip) 10.12.4.50 (keystone PUBLIC endpoint = provider VIP; verify vs bundle)
  • ENV(admin-domain) admin_domain (charmed-keystone admin user + project domain)
  • ENV(dashboard-vip) 10.12.4.58 (Horizon provider VIP; was .234 pre-R14)
  • admin project: DISCOVERED at runtime (do not hardcode -- DOCFIX-022).

Run-location legend

  • # RUN: jumphost -- vopenstack-jesse as jessea123; juju + openstack + openssl.

Step 3.1 -- Settle the cert cascade + acceptance walk

# RUN: jumphost The cascade here is NARROW (mysql bootstrapped before vault init, so only the Vault consumers clear: ovn-central x3, ovn-chassis x3, ovn-chassis-octavia, neutron-api-plugin-ovn, barbican-vault). Watch, then walk units AND subordinates.

juju status --color --watch 30s -m openstack     # Ctrl-C once settled

Acceptance walk (counts non-active/idle across units + subordinates):

juju status -m openstack --format=yaml | python3 -c "
import yaml,sys
d=yaml.safe_load(sys.stdin); apps=d.get('applications',{}); bad=[]
def chk(n,u):
    ws=(u.get('workload-status') or {}).get('current',''); js=(u.get('juju-status') or {}).get('current','')
    msg=(u.get('workload-status') or {}).get('message','')
    if ws!='active' or js!='idle': bad.append('%s: workload=%s juju=%s msg=%s'%(n,ws,js,msg))
for app,info in apps.items():
    for un,ud in (info.get('units') or {}).items():
        chk(un,ud)
        for sn,sd in (ud.get('subordinates') or {}).items(): chk(sn,sd)
print('Non-active/idle units: %d'%len(bad))
for b in bad: print('  '+b)
"

GATE: expected non-active/idle = 1 (octavia/0 BLOCKED "Awaiting configure-resources", the D-021 next step) or briefly 2 (+ glance-simplestreams-sync, normal pre-run). Any TLS consumer (the five above) persisting waiting/error past ~15 min is the concern -- STOP and read its log + relations (do NOT assume TLS; a prior stall was a MySQL 1045 desync):

juju status --relations -m openstack ovn-central ovn-chassis ovn-chassis-octavia neutron-api-plugin-ovn barbican-vault
# juju ssh -m openstack <unit> -- 'sudo tail -120 /var/log/juju/unit-<unit-dashed>.log' </dev/null

Step 3.2 -- Build admin-openrc (IP-only; canonical block)

# RUN: jumphost Keystone PUBLIC = the provider VIP IP over HTTPS with the vault CA (no FQDN, no /etc/hosts -- B5). This canonical block folds in three fixes: the CA is pulled via --format json + jq because the action's human output wraps the PEM in an INDENTED YAML block that is not valid PEM (appendix-A: DOCFIX-021); the OS_AUTH_URL is the VIP IP (DOCFIX-018); and the admin project is DISCOVERED by a scope-test loop rather than hardcoded, because the scoping project name varies by charm rev (DOCFIX-022 -- the cause of a prior HTTP 401). ( set -e ) keeps OS_PASSWORD inside the subshell and aborts cleanly on any failure.

KEYSTONE_VIP=10.12.4.50              # keystone PUBLIC endpoint = provider VIP (verify vs bundle on rebuild)
ADMIN_DOMAIN=admin_domain            # charmed-keystone admin user + project domain
PROJECT_CANDIDATES="admin admin_domain"   # tried in order; first that SCOPES wins (DOCFIX-022 variance)
CA="$HOME/vault-init/vault-ca-root.pem"
RC="$HOME/admin-openrc"

( set -e
  mkdir -p "$HOME/vault-init"
  # 1. Vault root CA -> file (JSON extract; DOCFIX-021 -- human output indents the PEM)
  juju run vault/leader get-root-ca -m openstack --format json \
    | jq -r '[.. | strings | select(test("-----BEGIN CERTIFICATE-----"))][0]' > "$CA"
  openssl x509 -in "$CA" -noout -subject -dates
  # 2. Admin password -> var (JSON extract, not human output)
  ADMIN_PASS=$(juju run keystone/leader get-admin-password -m openstack --format json | python3 -c "
import json,sys
d=json.load(sys.stdin)
def f(o):
    if isinstance(o,dict):
        for k in ('admin-password','password','Stdout'):
            if k in o and o[k]: return str(o[k]).strip()
        for v in o.values():
            r=f(v)
            if r: return r
    elif isinstance(o,list):
        for v in o:
            r=f(v)
            if r: return r
    return ''
print(f(d))
")
  [ -n "$ADMIN_PASS" ] || { echo "FATAL: password extract failed"; exit 1; }
  # 3. PROJECT LOOKUP: first candidate that issues a SCOPED token wins (DOCFIX-022)
  export OS_AUTH_URL="https://${KEYSTONE_VIP}:5000/v3" OS_USERNAME=admin OS_PASSWORD="$ADMIN_PASS"
  export OS_USER_DOMAIN_NAME="$ADMIN_DOMAIN" OS_PROJECT_DOMAIN_NAME="$ADMIN_DOMAIN"
  export OS_IDENTITY_API_VERSION=3 OS_REGION_NAME=RegionOne OS_CACERT="$CA"
  ADMIN_PROJECT=""
  for P in $PROJECT_CANDIDATES; do
    if OS_PROJECT_NAME="$P" openstack token issue >/dev/null 2>&1; then ADMIN_PROJECT="$P"; break; fi
  done
  [ -n "$ADMIN_PROJECT" ] || { echo "FATAL: no candidate project scoped (tried: $PROJECT_CANDIDATES)"; exit 1; }
  echo "[OK] admin project = $ADMIN_PROJECT ; password len ${#ADMIN_PASS}"
  # 4. Write ~/admin-openrc (backs up any existing one first)
  [ -f "$RC" ] && mv "$RC" "$RC.pre-$(date -u +%Y%m%dT%H%M%SZ)"
  cat > "$RC" <<EOF
export OS_AUTH_URL=https://${KEYSTONE_VIP}:5000/v3
export OS_USERNAME=admin
export OS_PASSWORD='$ADMIN_PASS'
export OS_PROJECT_NAME=$ADMIN_PROJECT
export OS_USER_DOMAIN_NAME=$ADMIN_DOMAIN
export OS_PROJECT_DOMAIN_NAME=$ADMIN_DOMAIN
export OS_IDENTITY_API_VERSION=3
export OS_REGION_NAME=RegionOne
export OS_CACERT=$CA
EOF
  chmod 600 "$RC"
)
# 5. Verify from the written file (password stayed inside the subshell above)
( source "$RC"; echo "auth -> $OS_AUTH_URL  project=$OS_PROJECT_NAME"; openstack token issue 2>&1 | head -6 )
( source "$RC"; openstack endpoint list -f value -c "Service Name" -c Interface -c URL 2>&1 | sort )

GATE: token issue returns a SCOPED token; endpoint list is IP-only across all services (public on the provider VIP .5x, internal+admin on the metal VIP .8.5x, keystone admin on :35357). Two non-blocking notes for later: s3/swift is registered on the radosgw VIP .60:443 (re-check vs the radosgw :80 listener during any Swift/S3 smoke); the gss image-stream is HTTP on metal 10.12.8.172.

Step 3.3 -- Horizon access via the external nginx reverse proxy

# RUN: operator (outside the Juju model) Horizon is fronted by an operator-managed nginx reverse proxy. On each rebuild / VIP relocation, repoint its upstream to the CURRENT dashboard provider VIP (now https://10.12.4.58, was .234 pre-R14). Verify two interplays:

  • ALLOWED_HOSTS: Horizon (bundle B5 setting) must permit whatever Host header reaches it, else HTTP 400 DisallowedHost. Either set the proxy proxy_set_header Host to the dashboard VIP, or add the proxy hostname to Horizon ALLOWED_HOSTS.
  • Upstream TLS: the dashboard cert is vault-signed for the VIP IP (IP-SAN). The proxy must trust the vault root CA (~/vault-init/vault-ca-root.pem) for proxy_ssl_verify, or terminate/re-encrypt per policy. LIVE-REVIEW: the proxy host + config path + reload command are operator-managed and not captured here -- record them verbatim when wired, and confirm an external GET reaches the Horizon login. (Roosevelt: this repoint folds into the access/DNS workstream.)

EXIT GATE (phase-03 complete)

  • Cloud settled: acceptance walk shows only the expected block(s) (octavia; maybe gss).
  • ~/admin-openrc (0600) authenticates and returns a SCOPED token; endpoint list IP-only.
  • Vault root CA at ~/vault-init/vault-ca-root.pem validates TLS to the keystone VIP.
  • Horizon reachable through the repointed reverse proxy.

As-built reference (2026-06-03 run -- audit trail)

  • Cascade settled ~04:15Z: all five Vault consumers active/idle; only expected non-active/idle = octavia (blocked, D-021) + gss (pre-run). mysql primary on mysql-innodb-cluster/1 (R/W), /0+/2 R/O (normal innodb-cluster).
  • admin-openrc IP-only: OS_AUTH_URL=https://10.12.4.50:5000/v3, OS_USERNAME=admin, OS_PROJECT_NAME=admin (scoped; project_id 65ce73e6798e4d1e8dd066609b7033ef), domains admin_domain, OS_CACERT=~/vault-init/vault-ca-root.pem.
  • Vault root CA: subject "Vault Root Certificate Authority (charm-pki-local)", notBefore 2026-06-03, notAfter 2036-05-31; TLS to 10.12.4.50:5000 OK (B5 IP-SAN holds).
  • Dashboard VIP 10.12.4.58 (nginx upstream repoint pending operator capture).

Next

phase-04 -- network carve (external provider network).