Kubernetes clusters on Omega Cloud

The one non-negotiable: the `-cluster` account, with its PASSWORD

Cluster create and delete run ONLY as {{TENANT_SHORT_NAME}}-cluster, signed in with its password. The platform's cluster machinery cannot operate through an application credential -- a cluster create attempted with the -svc credential fails every time, no matter how it is phrased. This is a platform constraint, not a configuration you (or the operator) can change. Do not "simplify" cluster automation to the application credential; it will never work.

export OS_AUTH_TYPE=password
export OS_AUTH_URL={{AUTH_URL}}
export OS_IDENTITY_API_VERSION=3
export OS_CACERT=<path to the delivered CA bundle>
export OS_USERNAME={{TENANT_SHORT_NAME}}-cluster
export OS_USER_DOMAIN_NAME={{TENANT_SHORT_NAME}}
export OS_PROJECT_NAME={{TENANT_SHORT_NAME}}-prod
export OS_PROJECT_DOMAIN_NAME={{TENANT_SHORT_NAME}}
read -rs -p "password for {{TENANT_SHORT_NAME}}-cluster: " OS_PASSWORD && export OS_PASSWORD
openstack token issue        # smoke test

The keypair-ownership trap

The SSH keypair {{TENANT_SHORT_NAME}}-key is OWNED by the -cluster account, and the platform validates keypair ownership in the cluster creator's context. Consequences:

Never delete {{TENANT_SHORT_NAME}}-key or recreate it under another account. A key with the same name owned by -svc or a team user will make every cluster create fail.
"Keypair not found" at cluster create almost always means you are signed in as the wrong account -- check OS_USERNAME before anything else.

Cluster lifecycle

The standard onboarding delivers a cluster template at handover:

openstack coe cluster template show {{TENANT_SHORT_NAME}}-k8s

If that template is missing, or you need a variant, create your own -- see "Creating your own cluster template" below.

Create (builds take TENS OF MINUTES -- poll slowly or just wait). Size <N> against TWO limits, not one: your quota, AND the physical capacity free on the platform right now. A request inside your quota can still fail to place a node -- the symptom is a node stuck in ERROR with "No valid host was found. There are not enough hosts available", which is NOT a quota problem. Start small (1 control + 1-2 workers), confirm it comes up, then scale the worker count:

openstack coe cluster create <cluster-name> \
  --cluster-template {{TENANT_SHORT_NAME}}-k8s \
  --keypair {{TENANT_SHORT_NAME}}-key \
  --master-count 1 --node-count <N>

openstack coe cluster show <cluster-name>   # poll >=10s apart
# done when status = CREATE_COMPLETE

Fetch the kubeconfig (treat it as a credential -- never print its contents; store it like any other secret):

openstack coe cluster config <cluster-name> --dir <secure-dir>
export KUBECONFIG=<secure-dir>/config
kubectl get nodes

The kubeconfig's client certificate is TIME-LIMITED. If kubectl or a pipeline that worked before starts failing to authenticate, the cert has expired -- re-fetch with the same command and update the stored credential. Read the expiry with:

kubectl config view --raw \
  -o jsonpath='{.users[0].user.client-certificate-data}' \
  | base64 -d | openssl x509 -noout -enddate

Cluster create, resize, and delete can be done in the dashboard OR on the command line; everything from the kubeconfig onward (kubectl, CI) is command-line ONLY -- the dashboard cannot drive kubectl.

Resize (add or remove workers after the cluster is up -- same account, same free-capacity limit as create):

openstack coe cluster resize <cluster-name> <new-node-count>

Delete (confirm with the human first, by name):

openstack coe cluster delete <cluster-name>

Creating your own cluster template

Templates are PROJECT-SCOPED: everyone in your project can already use them. The one hard rule: NEVER set the Public or Hidden flag (the two checkboxes in the dashboard's Create Cluster Template dialog, or --public / --hidden on the CLI). Publishing a template cloud-wide is an operator-only action, and the platform refuses the whole create with:

Not authorized to set public or hidden flag for cluster template (HTTP 403)

In the dashboard this surfaces only as the generic "Error: Unable to create cluster template." -- if you see that toast, the Public/Hidden checkboxes are the FIRST thing to check. You lose nothing by leaving them unchecked.

The second hard rule: use --network-driver calico. This platform supports calico. Do NOT choose flannel -- a flannel cluster never finishes coming up: the create sits IN_PROGRESS and eventually fails, with the nodes never going Ready. A cluster stuck in CREATE_IN_PROGRESS far past the normal build time is almost always this. Rebuild the template with calico.

A known-good CLI starting point, mirroring the platform-delivered template (works signed in as -svc or -cluster):

openstack coe cluster template create <template-name> \
  --coe kubernetes \
  --image <public kube image name or UUID> \
  --external-network provider-ext \
  --master-flavor <flavor> --flavor <flavor> \
  --network-driver calico --docker-storage-driver overlay2 \
  --master-lb-enabled --floating-ip-enabled \
  --fixed-network {{TENANT_SHORT_NAME}}-net \
  --fixed-subnet {{TENANT_SHORT_NAME}}-subnet

Pick the two machine sizes from openstack flavor list; if you have the delivered template, openstack coe cluster template show {{TENANT_SHORT_NAME}}-k8s shows the platform's known-good choices.

Do NOT bake --keypair into the template: the keypair is owned by the -cluster account (see the trap above), so it is passed at cluster create time instead, exactly as the lifecycle section shows.

Images: bring your own registry

The platform does NOT host a container registry. Push your application images to a registry you control -- a public one (Docker Hub, GitHub/ GitLab, Quay) or your own private one. The cluster reaches registries over the internet (it pulls its own system images the same way), so a PUBLIC image needs nothing extra.

For a PRIVATE registry, create an image-pull secret in the cluster and reference it from the Deployment:

kubectl create secret docker-registry regcred \
  --docker-server=<registry-host> --docker-username=<user> \
  --docker-password=<token> --docker-email=<you@example.com>

# then in the Deployment pod spec:
#   imagePullSecrets:
#     - name: regcred

Inject the registry password at pipeline run time from the CI secret store -- never in a manifest in a repository. Full detail: the Jenkins + Kubernetes Implementation Guide, section 5.0.

Deploying workloads: Services of type LoadBalancer

The cluster gets its own load balancer for the Kubernetes API, and it can publish Service objects of type: LoadBalancer through the same platform mechanism -- no extra setup needed:

kubectl expose deployment <app> --type=LoadBalancer --port=80

The Service's external address is allocated from the platform. Each such Service consumes a load balancer (and typically a floating IP) from YOUR quota -- prefer a single ingress controller of type LoadBalancer fronting many Services over one LoadBalancer per app.

A LoadBalancer Service (or an ingress) gets an IP, not a hostname: the platform provides no DNS. Map your own hostnames to that IP in whatever DNS you run. Ingress how-to: the Jenkins + Kubernetes Guide, section 5.4.

Persistent storage

Stateless workloads need nothing extra. A stateful one (a database, a cache with persistence) needs a PersistentVolume, which comes from a StorageClass -- check with kubectl get storageclass. If a class is marked (default), a PersistentVolumeClaim binds to it automatically; if NONE is listed, the cluster has no persistent-volume support and a PVC stays Pending -- raise it with {{ACCOUNT_CONTACT}}. Never work around it by writing to node-local disk (lost when a pod moves nodes).

Clusters and CI (the recommended split)

For the full worked pipeline (kubeconfig as a Jenkins credential, a deploy stage, and using the cluster as a build-agent pool), the delivered Jenkins + Kubernetes Implementation Guide is the single end-to-end reference. The split below is the principle it follows.

Create long-lived clusters manually (or on a controlled schedule) as -cluster, OUTSIDE CI.
Pipelines then deploy INTO the cluster using its kubeconfig, stored in the CI secret store like any other credential.
Do NOT put the -cluster password into CI so pipelines can create clusters. If your workflow genuinely needs cluster-per-run, raise it with {{ACCOUNT_CONTACT}} first -- there are capacity and quota implications.
For day-to-day build agents, a scoped Kubernetes ServiceAccount token (namespace-limited) is better practice than handing CI the full-admin kubeconfig. The Jenkins + Kubernetes Guide (section 6.1) has the steps.

Day-2 cloud work around the cluster (networks, floating IPs, volumes, extra load balancers) still runs as -svc per references/day2-operations.md; only cluster create/delete needs the -cluster password login.