Fork: 0
OpenStack / openstack-caracal-ipv4
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-ipv4 / docs / session-findings-2026-07-02.md
@JANeumatrix JANeumatrix 2 hours ago 5 KB Tenant file updates
Raw Blame History

Session findings -- 2026-07-02 (multi-tenant tenant->cluster buildout)

Executive summary

The tenant IDENTITY/TRUST path is DONE and PROVEN. A tenant password identity now creates a Magnum cluster through create_user (D-064), create_trust (D-065), and into certificate generation. Cluster COMPLETION is blocked one step later by an OPERATOR-side Barbican/Vault substrate defect (D-067), independent of the tenant model. The Option-3 tenant account model (D-066) is adopted and to be used from the first tenant. Next session: fix D-067 (live), then full tenant buildout + tenant-facing tests.

The trust-blocker chain (how we got from "cluster 403s" to "done + one substrate bug")

  1. D-064 (prior): create_user template fix unblocked trustee-user creation.
  2. create_trust then 403'd for EVERY caller (admin included), even trustor==self via direct openstack trust create. Root cause: base policy shipped identity:create_trust with the non-resolving user_id:%(trust.trustor_user_id)s (Caracal populates target.trust.trustor_user_id). -> D-065: override with the target-prefixed form keystone itself ships. PROVEN by toggling the override off (still 403 -> base policy owns it) then on.
  3. After D-065, create_trust via APP CRED still failed: keystone _check_application_credential blocks trust creation from app-cred tokens "regardless of the unrestricted flag" (this build's docstring). -> D-066: cluster-create MUST be PASSWORD auth; adopt Option-3 account split. allow_insecure_application_credential_trust_escalation REJECTED (isolation).
  4. Password create_trust PASSED. Cluster then failed at cert-gen -> Barbican 500 -> castellan vault_key_manager -> Vault AppRole login rejected: "source address 10.12.8.176 unauthorized through CIDR restrictions". -> D-067.

D-067 root cause (and a corrected mis-diagnosis)

barbican reaches Vault on the METAL-ADMIN plane (vault_url=10.12.8.190, egress 10.12.8.176). Vault's barbican-vault AppRole binds the secret_id to the METAL-INTERNAL CIDR (where east-west service traffic belongs, D-052/D-053). Off-plane source -> rejected. The bundle is CORRECT (vault/barbican/barbican-vault all bind secrets endpoints to metal-internal, lines 130/667/700); the LIVE env drifted. Fix = live rebind to metal-internal (gated, next session), NOT CIDR-widen. CORRECTED: mid-session I hypothesized "secret_id TTL expiry". REFUTED -- juju run vault/leader refresh-secrets rotated the secret_id (barbican.conf re-rendered, service restarted) and the login STILL failed with the CIDR error. It was never expiry; it is plane/CIDR.

What is validated live (tenant acme)

  • Manager persona self-service via CLI (create_project/user/grant) -- D-064 G3. PASS.
  • Tenant isolation: anti-escalation (admin grant DENIED); cross-domain resource reads DENIED/hidden; domain enumeration OWN-DOMAIN-ONLY (tighter than appendix-C's SCS worst-case -- appendix-C corrected).
  • App-cred + keypair self-mint; tenant L3 (net/subnet/router/ext-gw, SNAT proven) by a non-admin app-cred identity.
  • Cluster template create (image by UUID -- name form has a quoting/derivation hazard).
  • Cluster create through create_user + create_trust (password) into cert-gen.

Decisions logged

  • D-066: Option-3 tenant accounts (domain-admin/cluster/svc); cluster-create requires password auth.
  • D-067: barbican-vault -> Vault must use metal-internal (live drift; the cert-gen blocker). ADOPTED, fix pending.
  • D-068: PROPOSED -- Vault substrate hardening (1.16 pin [bundle done], TLS, AppRole lifecycle).

Probe-discipline lessons (now runbook conventions -- these recurred and cost time)

  1. Validate raw output WHOLE, never extract-then-check. A tr -dc 0-9 MARK guard turned an error string ("...10.12.8.30:17070...") into MARK=123101283017070 and passed. Use case "$raw" in ''|*[!0-9]*) fail;; *) ok;; esac.
  2. Whitelist-print secrets, never blacklist-redact. approle_secret_id leaked past a secret-keyed redact (the key is _secret_id). Print only an allowlist of safe fields; never pipe secrets.
  3. No exit/bare-return in interactive PASTE blocks (they escape to the login shell and logged the operator out). Subshell-wrap ( ... ). NOTE: executed .sh scripts may use exit normally.
  4. Privileged reads over juju ssh use sudo cat file | ..., never sudo cmd < file (the redirect runs UNPRIVILEGED -> Permission denied).
  5. Use the deployment's DECLARED endpoint/scheme, not the conventional one (assumed Vault https; it serves http -- every probe errored on scheme until corrected).
  6. A parser that can print NOTHING has a silent third state -- read raw + self-report inputs (field lengths, raw body) so a malformed-request 400 can't masquerade as an auth failure.

Roosevelt hardening backlog (from this session)

  • D-067/D-068: metal-internal binding discipline for ALL vault-kv consumers; Vault 1.16 + TLS; AppRole secret_id lifecycle (TTL/renewal + proactive auth health probe).
  • Endpoint/credential "follow the topology" is now a recurring class (with D-057): consider stable VIP/DNS endpoints for substrate services so leader/re-IP changes don't silently break consumers.

Next session plan

  1. Repair live env: read-only binding diagnosis (juju show-application vault barbican barbican-vault, spaces<->subnets), then GATED juju bind of the barbican<->Vault secrets path to metal-internal; re-run refresh-secrets if needed; confirm barbican AppRole login HTTP 200 from metal-internal.
  2. Re-run tenant cluster-create (acme, ${CLIENT}-cluster password) -> cert-gen clears -> watch to CREATE_COMPLETE; capture the CAPO child-cred mint identity (confirms D-066).
  3. Full tenant buildout via scripts/tenant-onboard.sh; then clean-room beta (zero admin fallback).
  4. Tenant-facing tests: kubeconfig, nodes/CNI/CCM, a tenant LB, tenant isolation from a second tenant.