diff --git a/docs/v1-redeploy-changelog.md b/docs/v1-redeploy-changelog.md index e76b485..f6a346c 100644 --- a/docs/v1-redeploy-changelog.md +++ b/docs/v1-redeploy-changelog.md @@ -1400,3 +1400,57 @@ template create (stage5) drops --keypair (keypair_id optional, default=None -- verified in source); stage6 already supplies --keypair as -cluster. Rejected alternative: creating the template as -cluster (blurs the Option-3 svc/cluster division for no gain). + +### 2026-07-02 (session 2, addendum 2) -- beta ACCEPTANCE PASS; offboard + vault-kv-health delivered + +TENANT ACCEPTANCE (gates t2-02 + t3-01, tenant beta, clean-room Option-3, zero admin fallback): +- Cluster beta-cluster (a360df32) CREATE_COMPLETE in ~13.5 min as beta-cluster (PASSWORD); + health_status=HEALTHY (D-042 1.4.0 pin confirmed on a tenant-created cluster). +- D-066 evidence: per-cluster child app cred magnum-a360df32-... OWNED BY beta-cluster (trust + impersonates the trustor); one trust, impersonation=True. TRUSTEE CORRECTION: the trustee is + the PER-CLUSTER user _ in the magnum domain -- magnum_domain_admin + is the identity that CREATES these trustees (the D-064 create_user op), not the trustee itself. + tenant-acceptance.sh P0 now asserts the real invariant (name == _, + domain == magnum). +- P1 kubeconfig fetched BY THE TENANT; 2 Ready nodes; 31/31 pods Running. P2 OCCM -> + Octavia LB (FIP 10.12.6.97) serving in ~2 min; kubeapi + service LBs ACTIVE/ONLINE. + P3 isolation: acme blind to every beta resource (cluster 404, net not-found, empty lists). +- host_href=None observation: did NOT affect cert-gen, cluster config fetch, or cert refs. + +NEW DELIVERABLES (mock-tested; harnesses committed under tests/): +- scripts/tenant-offboard.sh: full tenant offboarding, audit-by-default, --apply gated by typed + client name at /dev/tty, hard domain blocklist (Default/admin_domain/service_domain/magnum/capi), + dependency-ordered teardown (clusters-as-tenant first -> trusts -> LB/FIP/server -> L3 -> + users/projects/domain -> local creds retired). 6-branch harness ALL PASS incl. zero-mutation + audit proof and pty-driven confirmation tests. Immediate consumer: acme cleanup. +- scripts/vault-kv-health.sh + vault-kv-inner-probe.sh: D-068 item 3 proactive probe. Dynamic + consumer discovery from vault:secrets relations; per consumer: relation vault_url plane vs + METAL_INTERNAL_CIDR (lib-net.sh), conf render == relation data, AppRole login 200 from the + authentic source; plus vault-unsealed and the BUNDLEFIX-007 live invariant (external==access; + reports "D-067 clobber condition LIVE" on drift). Unknown consumer = loud coverage gap, exit 2. + Probe encoded at RUNTIME from the sibling file (no embed drift). 7-branch harness ALL PASS. + +PROBE/PORTABILITY LESSONS (added to conventions): +- mawk (Ubuntu default awk) does not support backslash-s -- use POSIX [[:space:]] classes in awk + and grep; never rely on gawk/GNU extensions in repo scripts. +- Never append an inline # comment inside a $( ... ) command substitution or pipeline -- it + comments out the closing paren (caught by bash -n in harness). +- Edit-plan discipline for scripted file surgery: overlapping replace targets must be ordered by + dependency or made disjoint (two fail-loud aborts this session, zero corrupted writes). + +SCRIPT BACKLOG (maintained; build at the appropriate step): +1. DONE scripts/tenant-offboard.sh 2. DONE scripts/vault-kv-health.sh +3. LARGELY DONE by cloud-assert.sh (DOCFIX-075, committed 2026-07-03: D-045/D-046/D-051/D-062/ + vault-sealed behavioral checks + harness). REMAINING: fold vault-kv-health.sh in as a + cloud-assert section (D-067/BUNDLEFIX-007/AppRole coverage is additive today), and use the + combined gate as the backbone for regenerating the stale restart-procedure runbook. +4. scripts/validate.sh -- draft the real D-011 runner (still PLACEHOLDER at f7bb36f); compose + from tenant-acceptance.sh + phase-03/04/05 verifies; D-019 dropped the DNS criterion. +5. scripts/tenant-cluster-create.sh -- generalize the t2-02 stage6+watch+D-066-evidence wrapper. +6. scripts/keystone-policy-drift.sh -- D-051 base_* alignment (LIVE-READ PENDING) via + oslopolicy-policy-generator diff; guards the 2024.2 upgrade-removal caution. +7. scripts/cloud-snapshot.sh -- pre/post-maintenance state capture. +8. HALF DONE: tests/tenant-onboard/ landed 2026-07-03; tests/tenant-acceptance/ still pending + (mock suite exists from the 2026-07-02 session). +NOTE (tooling gate): scripts/run-tests-all.sh (committed 2026-07-03) now runs every +tests/*/run-tests.sh; both harnesses in this package were verified under it pre-commit. diff --git a/scripts/tenant-acceptance.sh b/scripts/tenant-acceptance.sh index 235125d..5f26b0c 100644 --- a/scripts/tenant-acceptance.sh +++ b/scripts/tenant-acceptance.sh @@ -47,8 +47,18 @@ print(ts[0]['Trustee User ID'] if ts else '') except Exception: print('')" ) if [ -n "$TRUSTEE" ]; then + # Magnum mints a PER-CLUSTER trustee user in the magnum domain, named _ + # (magnum_domain_admin is the identity that CREATES these trustees -- the D-064 create_user op). TNAME=$( (admin_env; openstack user show "$TRUSTEE" -f value -c name &1) || true ) - echo " trustee: $TRUSTEE name=$TNAME (expect magnum_domain_admin)" + TDOM=$( (admin_env; openstack user show "$TRUSTEE" -f value -c domain_id &1) || true ) + MDOM=$( (admin_env; openstack domain show magnum -f value -c id &1) || true ) + TPID=$(awk -F= '/^project_id=/{print $2}' "$CF") + EXPECT="${BUUID}_${TPID}" + echo " trustee: $TRUSTEE name=$TNAME domain=$TDOM" + if [ "$TNAME" = "$EXPECT" ]; then echo " trustee name: PASS (per-cluster _)" + else echo " trustee name: WARN got '$TNAME' expected '$EXPECT'"; fi + if [ -n "$MDOM" ] && [ "$TDOM" = "$MDOM" ]; then echo " trustee domain: PASS (magnum domain)" + else echo " trustee domain: WARN got '$TDOM' expected magnum domain '$MDOM'"; fi else echo " trustee: WARN could not resolve"; fi echo diff --git a/scripts/tenant-offboard.sh b/scripts/tenant-offboard.sh new file mode 100644 index 0000000..1674151 --- /dev/null +++ b/scripts/tenant-offboard.sh @@ -0,0 +1,159 @@ +#!/usr/bin/env bash +# tenant-offboard.sh -- full tenant offboarding (Option-3 or legacy single-svc layout). +# Usage: tenant-offboard.sh [--audit|--apply] +# --audit (DEFAULT): read-only inventory of everything the offboard would remove. +# --apply : destructive. Requires typing the client name at the terminal. +# Order (dependency-driven): magnum clusters (as TENANT; cleans per-cluster trustee/trust/ +# child app-cred) -> residual trust sweep (admin) -> LB/FIP/server sweep -> L3 unwind -> +# users (app creds die with user) -> projects -> domain disable+delete -> local cred dir retired. +# Exit: 0 ok | 20 precondition/blocklist | 21 cluster delete failed | 22 resource sweep failed +# | 23 identity teardown failed | 24 confirmation refused/unavailable +set -u +CLIENT="${1:-}"; MODE="${2:---audit}" +BLOCKLIST="default admin_domain service_domain magnum capi heat" +[ -n "$CLIENT" ] || { echo "usage: tenant-offboard.sh [--audit|--apply]"; exit 20; } +case "$MODE" in --audit|--apply) : ;; *) echo "bad mode: $MODE"; exit 20;; esac +CL=$(printf '%s' "$CLIENT" | tr 'A-Z' 'a-z') +for b in $BLOCKLIST; do [ "$CL" = "$b" ] && { echo "REFUSED: '$CLIENT' is a protected domain"; exit 20; }; done + +admin_env(){ for v in $(env|awk -F= '/^OS_/{print $1}'); do unset "$v"; done; source "$HOME/admin-openrc"; } +is_id(){ case "$1" in *[!0-9a-f]*|'') return 1;; *) return 0;; esac; } + +admin_env +DOM=$(openstack domain show "$CLIENT" -f value -c id &1) || true +is_id "$DOM" || { echo "PRECOND: domain '$CLIENT' not found (raw: $DOM)"; exit 20; } +echo "=== OFFBOARD $MODE: client=$CLIENT domain=$DOM ===" + +# ---- tenant password cred discovery (magnum is tenant-scoped) ---- +TCF="" +for c in "$HOME/tenant-${CLIENT}/${CLIENT}-cluster-cred.txt" \ + "$HOME/tenant-${CLIENT}/${CLIENT}-svc-cred.txt" \ + "$HOME/${CLIENT}-svc-cred.txt"; do + [ -s "$c" ] && grep -qE '^password=' "$c" && { TCF="$c"; break; } +done +tenant_env(){ for v in $(env|awk -F= '/^OS_/{print $1}'); do unset "$v"; done + export OS_AUTH_URL="https://10.12.4.50:5000/v3" OS_IDENTITY_API_VERSION=3 + export OS_CACERT="${OS_CACERT:-$HOME/vault-init/vault-ca-root.pem}" + export OS_USERNAME="$(awk -F= '/^username=/{print $2}' "$TCF")" OS_USER_DOMAIN_ID="$DOM" + export OS_PROJECT_ID="$(awk -F= '/^project_id=/{print $2}' "$TCF")" + export OS_PASSWORD="$(awk -F= '/^password=/{print $2}' "$TCF")"; } + +# ---- inventory ---- +USERS=$(openstack user list --domain "$DOM" -f value -c ID -c Name &1) || true +PROJS=$(openstack project list --domain "$DOM" -f value -c ID -c Name &1) || true +echo "--- users in domain ---"; printf '%s\n' "${USERS:-}" | sed 's/^/ /' +echo "--- projects in domain ---"; printf '%s\n' "${PROJS:-}" | sed 's/^/ /' + +CLUSTERS="" +if [ -n "$TCF" ]; then + echo "--- clusters (tenant scope via $(basename "$TCF")) ---" + CLUSTERS=$( (tenant_env; openstack coe cluster list -f value -c uuid -c name -c status &1) || true ) + printf '%s\n' "${CLUSTERS:-}" | sed 's/^/ /' +else + echo "--- clusters: NO tenant password cred file found -- cannot enumerate (magnum is tenant-scoped) ---" +fi + +UIDLIST=$(printf '%s\n' "$USERS" | awk '{print $1}') +TRUSTS=$( openstack trust list -f json &1 | python3 -c " +import sys,json +uids=set('''$UIDLIST'''.split()) +try: + for t in json.load(sys.stdin): + if t.get('Trustor User ID') in uids: + print(t.get('ID'), t.get('Trustor User ID'), t.get('Trustee User ID')) +except Exception as e: print('trust-parse-error', e)" ) || true +echo "--- trusts with trustor in domain ---"; printf '%s\n' "${TRUSTS:-}" | sed 's/^/ /' + +for P in $(printf '%s\n' "$PROJS" | awk '{print $1}'); do + echo "--- project $P resources ---" + echo " LBs:"; openstack loadbalancer list --project "$P" -f value -c id -c name -c provisioning_status &1 | sed 's/^/ /' + echo " FIPs:"; openstack floating ip list --project "$P" -f value -c ID -c "Floating IP Address" &1 | sed 's/^/ /' + echo " servers:"; openstack server list --project "$P" -f value -c ID -c Name -c Status &1 | sed 's/^/ /' + echo " routers:"; openstack router list --project "$P" -f value -c ID -c Name &1 | sed 's/^/ /' + echo " subnets:"; openstack subnet list --project "$P" -f value -c ID -c Name &1 | sed 's/^/ /' + echo " networks:"; openstack network list --project "$P" -f value -c ID -c Name &1 | sed 's/^/ /' +done +echo "--- local cred artifacts ---" +ls -d "$HOME/tenant-${CLIENT}" 2>/dev/null | sed 's/^/ /' +ls "$HOME/${CLIENT}"-*.txt "$HOME/${CLIENT}"-*.pem 2>/dev/null | sed 's/^/ /' + +if [ "$MODE" = "--audit" ]; then echo "AUDIT COMPLETE (no changes). Re-run with --apply to execute."; exit 0; fi + +# ---- confirmation gate (typed at terminal; refuses if no tty) ---- +if [ ! -r /dev/tty ]; then echo "REFUSED: --apply needs an interactive terminal"; exit 24; fi +printf 'Type the client name (%s) to confirm DESTRUCTIVE offboard: ' "$CLIENT" > /dev/tty +IFS= read -r CONF < /dev/tty || CONF="" +[ "$CONF" = "$CLIENT" ] || { echo "REFUSED: confirmation mismatch"; exit 24; } + +# ---- Phase A: clusters (as tenant) ---- +if [ -n "$CLUSTERS" ] && [ "$CLUSTERS" != "" ]; then + [ -n "$TCF" ] || { echo "FATAL: clusters exist but no tenant cred"; exit 21; } + printf '%s\n' "$CLUSTERS" | while read -r CU CN CS; do + [ -n "$CU" ] || continue + echo "== deleting cluster $CN ($CU, was $CS) ==" + (tenant_env; openstack coe cluster delete "$CU" &1) | sed 's/^/ /' + done + for i in $(seq 1 40); do + LEFT=$( (tenant_env; openstack coe cluster list -f value -c name -c status &1) || true ) + printf '%s [%02d] remaining: %s\n' "$(date +%T)" "$i" "${LEFT:-}" + [ -z "$LEFT" ] && break + printf '%s\n' "$LEFT" | grep -q 'DELETE_FAILED' && { echo "FATAL: DELETE_FAILED -- see appendix-A stuck-delete"; exit 21; } + sleep 20 + done + LEFT=$( (tenant_env; openstack coe cluster list -f value -c name &1) || true ) + [ -z "$LEFT" ] || { echo "FATAL: clusters not gone in budget"; exit 21; } +fi + +admin_env +# ---- Phase B: residual trusts ---- +printf '%s\n' "${TRUSTS:-}" | while read -r TID REST; do + [ -n "$TID" ] && [ "$TID" != "" ] || continue + T2=$(openstack trust show "$TID" -f value -c id &1) || true + [ "$T2" = "$TID" ] && { echo " deleting residual trust $TID"; openstack trust delete "$TID" &1 | sed 's/^/ /'; } +done + +# ---- Phase C+D: per-project resource sweep + L3 unwind ---- +for P in $(printf '%s\n' "$PROJS" | awk '{print $1}'); do + for LB in $(openstack loadbalancer list --project "$P" -f value -c id &1); do + is_id "${LB//-/}" 2>/dev/null || true + echo " LB delete --cascade $LB"; openstack loadbalancer delete --cascade "$LB" &1 | sed 's/^/ /' + done + for i in $(seq 1 20); do + N=$(openstack loadbalancer list --project "$P" -f value -c id &1 | grep -c . || true) + [ "$N" = 0 ] && break; echo " waiting LBs gone ($N left)"; sleep 15 + done + for F in $(openstack floating ip list --project "$P" -f value -c ID &1); do + echo " FIP delete $F"; openstack floating ip delete "$F" &1 | sed 's/^/ /'; done + for S in $(openstack server list --project "$P" -f value -c ID &1); do + echo " server delete $S"; openstack server delete "$S" &1 | sed 's/^/ /'; done + for R in $(openstack router list --project "$P" -f value -c ID &1); do + for SN in $(openstack subnet list --project "$P" -f value -c ID &1); do + openstack router remove subnet "$R" "$SN" &1 || true; done + openstack router unset --external-gateway "$R" &1 || true + echo " router delete $R"; openstack router delete "$R" &1 | sed 's/^/ /'; done + for SN in $(openstack subnet list --project "$P" -f value -c ID &1); do + echo " subnet delete $SN"; openstack subnet delete "$SN" &1 | sed 's/^/ /'; done + for NW in $(openstack network list --project "$P" -f value -c ID &1); do + echo " network delete $NW"; openstack network delete "$NW" &1 | sed 's/^/ /'; done +done + +# ---- Phase E: identities ---- +FAILED=0 +for U in $(printf '%s\n' "$USERS" | awk '{print $1}'); do + echo " user delete $U"; openstack user delete "$U" &1 | sed 's/^/ /' || FAILED=1; done +for P in $(printf '%s\n' "$PROJS" | awk '{print $1}'); do + echo " project delete $P"; openstack project delete "$P" &1 | sed 's/^/ /' || FAILED=1; done +openstack domain set --disable "$DOM" &1 | sed 's/^/ /' +echo " domain delete $DOM"; openstack domain delete "$DOM" &1 | sed 's/^/ /' || FAILED=1 +D2=$(openstack domain show "$CLIENT" -f value -c id &1) || true +is_id "$D2" && { echo "FATAL: domain still present"; exit 23; } +[ "$FAILED" = 0 ] || { echo "FATAL: identity teardown had failures (domain gone; review output)"; exit 23; } + +# ---- Phase F: retire local creds ---- +if [ -d "$HOME/tenant-${CLIENT}" ]; then + DST="$HOME/tenant-${CLIENT}.offboarded-$(date -u +%Y%m%d-%H%M%S)" + mv "$HOME/tenant-${CLIENT}" "$DST" && chmod 700 "$DST" && echo " local creds retired -> $DST (shred manually when record no longer needed)" +fi +ls "$HOME/${CLIENT}"-*.txt "$HOME/${CLIENT}"-*.pem 2>/dev/null | sed 's/^/ LEGACY loose file (handle manually): /' +echo "OFFBOARD COMPLETE: $CLIENT" +exit 0 diff --git a/scripts/vault-kv-health.sh b/scripts/vault-kv-health.sh new file mode 100644 index 0000000..139a0e0 --- /dev/null +++ b/scripts/vault-kv-health.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# vault-kv-health.sh -- proactive Vault AppRole health for every vault-kv consumer (D-068 item 3). +# Read-only except one real AppRole login per consumer (60s-TTL token; charm token_ttl=60s). +# Checks: C0 vault unsealed | C1 BUNDLEFIX-007 live (external binding == access binding) +# C2 relation-data vault_url host inside metal-internal | C3 conf render == relation data +# C4 AppRole login HTTP 200 from the consumer principal's authentic source +# Consumer discovery is DYNAMIC (juju relations of vault:secrets). Consumers without a conf-path +# mapping are a COVERAGE GAP: reported loudly, exit 2. Update CONSUMER_MAP when adding consumers. +# Exit: 0 all pass | 1 any check failed | 2 warnings only (coverage gap / unparsed) +set -u +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=scripts/lib-net.sh +. "$SCRIPT_DIR/lib-net.sh" # provides METAL_INTERNAL_CIDR (single source of truth) +M="${MODEL:-openstack}" +FAIL=0; WARN=0 +fail(){ echo "FAIL: $*"; FAIL=$((FAIL+1)); } +warn(){ echo "WARN: $*"; WARN=$((WARN+1)); } +pass(){ echo "PASS: $*"; } +PROBE="$SCRIPT_DIR/vault-kv-inner-probe.sh" +[ -s "$PROBE" ] || { echo "FATAL: missing $PROBE (sibling probe source)"; exit 1; } +INNER_B64=$(base64 -w0 "$PROBE") +# consumer app -> "principal_unit:conf_path" +declare -A CONSUMER_MAP=( ["barbican-vault"]="barbican/0:/etc/barbican/barbican.conf" ) + +echo "=== C0: vault unsealed ===" +VS=$(juju ssh -m "$M" vault/0 -- 'VAULT_ADDR=http://127.0.0.1:8200 vault status 2>&1' &1 || true) +ACC=$(printf '%s\n' "$B" | awk '/^[[:space:]]+access:/{print $2; exit}') +EXT=$(printf '%s\n' "$B" | awk '/^[[:space:]]+external:/{print $2; exit}') +echo " access=$ACC external=$EXT" +if [ -n "$ACC" ] && [ "$ACC" = "$EXT" ]; then pass "external == access ($ACC)" +elif [ -z "$ACC" ] || [ -z "$EXT" ]; then warn "bindings unparsed" +else fail "external ($EXT) != access ($ACC) -- D-067 clobber condition LIVE"; fi + +echo "=== consumer discovery (relations of vault:secrets) ===" +CONS=$(juju status vault -m "$M" --format json 2>/dev/null | python3 -c " +import sys,json +try: + d=json.load(sys.stdin); r=d['applications']['vault'].get('relations',{}).get('secrets',[]) + for x in r: print(x['related-application'] if isinstance(x,dict) else x) +except Exception as e: print('PARSE-ERROR', e)") +printf '%s\n' "${CONS:-}" | sed 's/^/ /' +case "$CONS" in PARSE-ERROR*|'') warn "consumer discovery failed"; CONS="";; esac + +for APP in $CONS; do + echo "=== consumer: $APP ===" + MAPV="${CONSUMER_MAP[$APP]:-}" + if [ -z "$MAPV" ]; then warn "$APP: no conf mapping -- COVERAGE GAP, update CONSUMER_MAP"; continue; fi + PUNIT="${MAPV%%:*}"; CPATH="${MAPV#*:}" + # URL class is IPv4-only (v1); extend for bracketed IPv6 at v2 + RURL=$(juju show-unit "$APP/0" -m "$M" --format yaml 2>&1 | grep -E 'vault_url' | head -1 | grep -oE 'https?://[0-9A-Za-z.:]+' || true) + echo " relation vault_url: ${RURL:-}" + RHOST=${RURL#*//}; RHOST=${RHOST%%:*} + if [ -z "$RURL" ]; then fail "$APP: no vault_url in relation data" + elif python3 -c "import ipaddress,sys; sys.exit(0 if ipaddress.ip_address('$RHOST') in ipaddress.ip_network('$METAL_INTERNAL_CIDR') else 1)" 2>/dev/null; then + pass "$APP: C2 relation vault_url on metal-internal ($METAL_INTERNAL_CIDR)" + else fail "$APP: C2 relation vault_url $RHOST NOT in $METAL_INTERNAL_CIDR"; fi + P=$(printf '%s' "$INNER_B64" | juju ssh -m "$M" "$PUNIT" -- "base64 -d | sudo bash -s $CPATH" 2>&1 | tr -d '\r' || true) + printf '%s\n' "$P" | sed 's/^/ /' + CURL2=$(printf '%s\n' "$P" | awk -F= '/^conf_vault_url=/{print $2; exit}') + if [ -n "$RURL" ] && [ "$CURL2" = "$RURL" ]; then pass "$APP: C3 conf render matches relation data" + else fail "$APP: C3 conf ($CURL2) != relation ($RURL)"; fi + if printf '%s\n' "$P" | grep -q '^PROBE-PASS$'; then pass "$APP: C4 AppRole login 200 from $PUNIT" + else fail "$APP: C4 AppRole login FAILED from $PUNIT"; fi +done + +echo +echo "Summary: FAIL=$FAIL WARN=$WARN" +[ "$FAIL" -gt 0 ] && exit 1 +[ "$WARN" -gt 0 ] && exit 2 +exit 0 diff --git a/scripts/vault-kv-inner-probe.sh b/scripts/vault-kv-inner-probe.sh new file mode 100644 index 0000000..b71b1c4 --- /dev/null +++ b/scripts/vault-kv-inner-probe.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +# vault-kv-inner-probe.sh -- runs ON a vault-kv consumer principal as root, via +# base64 -d | sudo bash -s +# AppRole login from the unit's authentic source. Secrets never printed: values read into +# vars; login response body deleted unread on 200 (it carries a client token); .errors only +# on non-200. Side effect: one real 60s-TTL Vault token per run (charm sets token_ttl=60s). +set -u +umask 077 +CONF="${1:-}" +[ -n "$CONF" ] && [ -r "$CONF" ] || { echo "PROBE-FAIL: conf unreadable: ${CONF:-}"; exit 1; } +URL=$(grep -E '^[[:space:]]*vault_url[[:space:]]*=' "$CONF" | tail -1 | cut -d= -f2- | tr -d ' \t\r') +RID=$(grep -E '^[[:space:]]*approle_role_id[[:space:]]*=' "$CONF" | tail -1 | cut -d= -f2- | tr -d ' \t\r') +SID=$(grep -E '^[[:space:]]*approle_secret_id[[:space:]]*=' "$CONF" | tail -1 | cut -d= -f2- | tr -d ' \t\r') +echo "conf_vault_url=$URL" +[ -n "$URL" ] && [ -n "$RID" ] && [ -n "$SID" ] || { echo "PROBE-FAIL: required conf values missing"; exit 1; } +HOST=${URL#*//}; HOST=${HOST%%:*} +echo "route: $(ip -o route get "$HOST" 2>&1 | head -1)" +command -v curl >/dev/null || { echo "PROBE-FAIL: curl absent"; exit 1; } +BODY=$(printf '{"role_id":"%s","secret_id":"%s"}' "$RID" "$SID") +RESP=/root/.vaultkv-login-resp.json +HTTP=$(curl -s -o "$RESP" -w '%{http_code}' --max-time 10 -X POST -d "$BODY" "$URL/v1/auth/approle/login" 2>&1 || true) +echo "login_http=$HTTP" +RC=1 +if [ "$HTTP" = "200" ]; then echo "PROBE-PASS"; RC=0 +else echo "errors: $(python3 -c "import json;print(json.load(open('$RESP')).get('errors'))" 2>/dev/null || head -c 200 "$RESP")"; fi +rm -f "$RESP"; exit $RC diff --git a/tests/tenant-offboard/run-tests.sh b/tests/tenant-offboard/run-tests.sh new file mode 100644 index 0000000..cff2102 --- /dev/null +++ b/tests/tenant-offboard/run-tests.sh @@ -0,0 +1,84 @@ +#!/usr/bin/env bash +# tests/tenant-offboard/run-tests.sh -- mock harness for scripts/tenant-offboard.sh +# Scenarios: blocklist=20, missing-domain=20, audit=0 (ZERO mutations), tty-mismatch=24, +# apply-happy=0, cluster-DELETE_FAILED=21. Requires no cloud access. +set -u +SD="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"; SCRIPT="$SD/../../scripts/tenant-offboard.sh" +W=$(mktemp -d); trap 'rm -rf "$W"' EXIT +mkdir -p "$W/bin" "$W/vault-init"; touch "$W/admin-openrc" "$W/vault-init/vault-ca-root.pem" +printf 'username=acme-svc\nproject_id=%s\npassword=pw\n' "$(python3 -c "print('a'*32)")" > "$W/acme-svc-cred.txt" +printf '#!/usr/bin/env bash\nexit 0\n' > "$W/bin/sleep" +cat > "$W/bin/openstack" <<'OSMOCK' +#!/usr/bin/env bash +S="${MOCK_SCEN:-happy}"; ST="${MOCK_STATE:-/home/claude/mock6/state}"; mkdir -p "$ST" +mut(){ echo "$*" >> "$ST/mutations"; } +DOM=$(python3 -c "print('d'*32)"); U1=$(python3 -c "print('1'*32)"); U2=$(python3 -c "print('2'*32)") +P1=$(python3 -c "print('a'*32)") +case "$*" in + "domain show acme"*) + [ -f "$ST/domgone" ] && { echo "No domain with a name or ID of 'acme' exists."; exit 1; } + echo "$DOM";; + "domain show "*) echo "No domain found"; exit 1;; + "user list --domain"*) printf '%s acme-svc\n%s acme-domain-admin\n' "$U1" "$U2";; + "project list --domain"*) printf '%s acme-prod\n' "$P1";; + "coe cluster list"*) + if [ -f "$ST/cdel" ]; then + N=$(cat "$ST/cn" 2>/dev/null || echo 0); N=$((N+1)); echo "$N" > "$ST/cn" + if [ "$S" = delfail ]; then echo "cu1 acme-cluster DELETE_FAILED" + elif [ "$N" -le 2 ]; then echo "cu1 acme-cluster DELETE_IN_PROGRESS"; fi + else echo "cu1 acme-cluster CREATE_FAILED"; fi;; + "coe cluster delete"*) mut "$*"; touch "$ST/cdel";; + "trust list"*) + if [ -f "$ST/tdel" ]; then echo "[]"; else printf '[{"ID":"tr1","Trustor User ID":"%s","Trustee User ID":"tee"}]\n' "$U1"; fi;; + "trust show tr1"*) [ -f "$ST/tdel" ] && exit 1; echo "tr1";; + "trust delete"*) mut "$*"; touch "$ST/tdel";; + "loadbalancer list"*) [ -f "$ST/lbdel" ] || echo "lb1 old-lb ERROR";; + "loadbalancer delete"*) mut "$*"; touch "$ST/lbdel";; + "floating ip list"*) [ -f "$ST/fipdel" ] || echo "f1 10.12.5.99";; + "floating ip delete"*) mut "$*"; touch "$ST/fipdel";; + "server list"*) : ;; + "router list"*) [ -f "$ST/rdel" ] || echo "r1 acme-router";; + "router remove subnet"*|"router unset"*) mut "$*";; + "router delete"*) mut "$*"; touch "$ST/rdel";; + "subnet list"*) [ -f "$ST/sndel" ] || echo "sn1 acme-subnet";; + "subnet delete"*) mut "$*"; touch "$ST/sndel";; + "network list"*) [ -f "$ST/nwdel" ] || echo "nw1 acme-net";; + "network delete"*) mut "$*"; touch "$ST/nwdel";; + "user delete"*|"project delete"*) mut "$*";; + "domain set --disable"*) mut "$*";; + "domain delete"*) mut "$*"; touch "$ST/domgone";; +esac +OSMOCK +cat > "$W/ptyrun.py" <<'PTY' +import pty, os, sys +cmd = sys.argv[1:-1]; feed = sys.argv[-1] +pid, fd = pty.fork() +if pid == 0: + os.execvp(cmd[0], cmd) +os.write(fd, (feed + "\n").encode()) +out = b"" +while True: + try: + d = os.read(fd, 4096) + except OSError: + break + if not d: + break + out += d +_, st = os.waitpid(pid, 0) +sys.stdout.write(out.decode(errors="replace")) +sys.exit(os.waitstatus_to_exitcode(st)) +PTY +chmod +x "$W/bin/"* +P=0; F=0 +chk(){ if [ "$2" = "$3" ]; then echo "PASS: $1 (exit $2)"; P=$((P+1)); else echo "FAIL: $1 (exit $2, want $3)"; F=$((F+1)); fi; } +export MOCK_STATE="$W/state" +HOME="$W" PATH="$W/bin:$PATH" bash "$SCRIPT" magnum --apply >/dev/null 2>&1; chk blocklist $? 20 +rm -rf "$W/state"; HOME="$W" PATH="$W/bin:$PATH" bash "$SCRIPT" ghost >/dev/null 2>&1; chk missing-domain $? 20 +rm -rf "$W/state"; HOME="$W" PATH="$W/bin:$PATH" bash "$SCRIPT" acme --audit >/dev/null 2>&1; RC=$? +MUT=$(cat "$W/state/mutations" 2>/dev/null | wc -l) +if [ "$RC" = 0 ] && [ "$MUT" = 0 ]; then echo "PASS: audit (exit 0, 0 mutations)"; P=$((P+1)); else echo "FAIL: audit (exit $RC, mutations $MUT)"; F=$((F+1)); fi +rm -rf "$W/state"; HOME="$W" PATH="$W/bin:$PATH" python3 "$W/ptyrun.py" bash "$SCRIPT" acme --apply WRONG >/dev/null 2>&1; chk tty-mismatch $? 24 +rm -rf "$W/state"; HOME="$W" PATH="$W/bin:$PATH" python3 "$W/ptyrun.py" bash "$SCRIPT" acme --apply acme >/dev/null 2>&1; chk apply-happy $? 0 +rm -rf "$W/state"; HOME="$W" MOCK_SCEN=delfail PATH="$W/bin:$PATH" python3 "$W/ptyrun.py" bash "$SCRIPT" acme --apply acme >/dev/null 2>&1; chk delete-failed $? 21 +echo; [ "$F" = 0 ] && { echo "ALL PASS ($P/6)"; exit 0; } || { echo "FAILURES: $F"; exit 1; } diff --git a/tests/vault-kv-health/run-tests.sh b/tests/vault-kv-health/run-tests.sh new file mode 100644 index 0000000..25e2580 --- /dev/null +++ b/tests/vault-kv-health/run-tests.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# tests/vault-kv-health/run-tests.sh -- mock harness for scripts/vault-kv-health.sh +# Scenarios: happy=0, sealed=1, binding-drift=1 (D-067 clobber), plane-wrong=1, +# conf-mismatch=1, login-403=1, unknown-consumer=2 (coverage gap). +set -u +SD="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +W=$(mktemp -d); trap 'rm -rf "$W"' EXIT +mkdir -p "$W/bin" "$W/scripts" +cp "$SD/../../scripts/vault-kv-health.sh" "$SD/../../scripts/vault-kv-inner-probe.sh" "$W/scripts/" +printf 'METAL_INTERNAL_CIDR="10.12.12.0/22"\n' > "$W/scripts/lib-net.sh" +cat > "$W/bin/juju" <<'JMOCK' +#!/usr/bin/env bash +S="${MOCK_SCEN:-happy}" +case "$*" in + *"ssh"*vault/0*) echo "Initialized true"; if [ "$S" = sealed ]; then echo "Sealed true"; else echo "Sealed false"; fi;; + *"show-application vault"*) + echo "vault:"; echo " endpoint-bindings:"; echo " access: metal-internal" + if [ "$S" = binddrift ]; then echo " external: metal-admin"; else echo " external: metal-internal"; fi;; + *"status vault"*) + if [ "$S" = unknowncons ]; then R='[{"related-application":"barbican-vault"},{"related-application":"mystery-vault"}]'; else R='[{"related-application":"barbican-vault"}]'; fi + printf '{"applications":{"vault":{"relations":{"secrets":%s}}}}\n' "$R";; + *"show-unit barbican-vault/0"*) + if [ "$S" = planewrong ]; then echo " vault_url: '\"http://10.12.8.190:8200\"'"; else echo " vault_url: '\"http://10.12.12.117:8200\"'"; fi;; + *"ssh"*barbican/0*) + cat >/dev/null + if [ "$S" = confmismatch ]; then echo "conf_vault_url=http://10.12.8.190:8200"; else echo "conf_vault_url=http://10.12.12.117:8200"; fi + echo "route: 10.12.12.117 dev eth1 src 10.12.12.110" + if [ "$S" = login403 ]; then echo "login_http=403"; echo "errors: ['cidr']"; else echo "login_http=200"; echo "PROBE-PASS"; fi;; +esac +JMOCK +chmod +x "$W/bin/juju" +P=0; F=0 +for t in happy:0 sealed:1 binddrift:1 planewrong:1 confmismatch:1 login403:1 unknowncons:2; do + s="${t%%:*}"; e="${t##*:}" + MOCK_SCEN="$s" PATH="$W/bin:$PATH" bash "$W/scripts/vault-kv-health.sh" >/dev/null 2>&1; RC=$? + if [ "$RC" = "$e" ]; then echo "PASS: $s (exit $RC)"; P=$((P+1)); else echo "FAIL: $s (exit $RC, want $e)"; F=$((F+1)); fi +done +echo; [ "$F" = 0 ] && { echo "ALL PASS ($P/7)"; exit 0; } || { echo "FAILURES: $F"; exit 1; }