Newer
Older
openstack-caracal-ipv4 / clientdocs / tenant-skill / references / troubleshooting.md

Troubleshooting: triage in this order

Work the list top to bottom; most failures are caught by step 1 or 2. Never guess past a step -- verify it, then move on.

1. Token first

openstack token issue

Authentication failures explain most sudden breakage. If this fails, check in order:

  • Revoked or expired application credential (per-pipeline credentials are often minted with --expiration -- check the date with openstack application credential show <name> from the -svc account).
  • Missing CA bundle in the environment (OS_CACERT / cacert not set, or the file absent from the job environment). The fix is supplying the bundle, never disabling TLS verification.
  • Wrong account for the job (see SKILL.md Step 0). Check OS_USERNAME / OS_CLOUD before anything else.
  • A password was changed or the credential file is stale: a custodian requests a reset via {{ACCOUNT_CONTACT}} -- there is deliberately no self-service password recovery.

2. Quota second

openstack limits show --absolute

The second most common cause. Quota-exceeded errors are expected, actionable failures, NOT retryable platform faults:

  • First: look for leaked resources and clean them up (servers, volumes, load balancers, and especially detached floating IPs -- they count against quota too).
  • If consumption is legitimate: an authorized requester asks {{ACCOUNT_CONTACT}} for a raise. Your accounts cannot raise quotas; do not try.

3. Is this a boundary? (expected refusals)

Before treating a permission error as a fault, check this table. A match means the platform is working AS DESIGNED: do not retry, do not switch accounts to bypass it, and never attempt to "fix" the denial.

Operation attempted As Result Why it is correct
openstack user create / any role grant application credential DENIED identity work belongs to -domain-admin only
openstack coe cluster create / delete application credential DENIED cluster machinery requires the -cluster password login
grant admin to any user any account DENIED the platform refuses admin on client accounts by design
any quota change any of your accounts DENIED quotas are operator-set; request via an authorized requester
modify provider-ext, the floating IP pool, shared images, machine sizes any of your accounts DENIED shared substrate is operator-managed
anything in another domain any of your accounts DENIED / not visible that is your isolation working

Two inversions of this table are INCIDENTS -- report to {{ACCOUNT_CONTACT}} urgently, investigate second:

  • An operation in this table SUCCEEDS (e.g. an admin grant lands).
  • One of your accounts appears to hold rights beyond those listed in your Handover Pack.

4. Known failure signatures (check before escalating)

  • "Keypair not found" at cluster create: you are signed in as the wrong account -- the keypair belongs to -cluster. See references/kubernetes.md.
  • Cluster create fails instantly with the -svc credential: boundary (table above), not an outage.
  • Subnet create rejected for overlap: pick a range that does not collide with {{TENANT_SHORT_NAME}}-subnet or your other networks.
  • Network delete refused: it still has ports -- delete the servers and load balancers on it first, detach the router interface last.
  • Load balancer sitting in PENDING_CREATE for minutes: normal build time. Poll >=10s or use --wait; only escalate after it goes to ERROR.
  • Cluster build running for tens of minutes: normal. Only escalate on a FAILED status.
  • Dashboard refuses an identity action that -domain-admin should be able to do: use the CLI for that step (the API has the full capability set) and mention it to {{ACCOUNT_CONTACT}}.
  • SSH to a new VM times out: check (in order) the security group allows TCP/22 from your source range, the floating IP is attached, and you are using the right private key for {{TENANT_SHORT_NAME}}-key.

5. Escalate with the verbatim error

Anything left looks like a platform fault. Capture:

  1. The exact command you ran (with the account/credential type used, never the secret itself).
  2. The exact, unedited error text.
  3. What step 1 and step 2 showed (token OK? quota headroom?).

Send that to {{ACCOUNT_CONTACT}} -- marked urgent for outages or anything security-shaped. Verbatim errors get fast answers; paraphrased ones do not. While waiting, do not repeat mutating commands; repeated half-failed creates leak quota.