# Kubernetes clusters on Omega Cloud ## The one non-negotiable: the `-cluster` account, with its PASSWORD Cluster create and delete run ONLY as `{{TENANT_SHORT_NAME}}-cluster`, signed in with its password. The platform's cluster machinery cannot operate through an application credential -- a cluster create attempted with the `-svc` credential fails every time, no matter how it is phrased. This is a platform constraint, not a configuration you (or the operator) can change. Do not "simplify" cluster automation to the application credential; it will never work. Sign in (prompt the human for the password -- never read it from a file into your context, never store it in CI): export OS_AUTH_TYPE=password export OS_AUTH_URL={{AUTH_URL}} export OS_IDENTITY_API_VERSION=3 export OS_CACERT= export OS_USERNAME={{TENANT_SHORT_NAME}}-cluster export OS_USER_DOMAIN_NAME={{TENANT_SHORT_NAME}} export OS_PROJECT_NAME={{TENANT_SHORT_NAME}}-prod export OS_PROJECT_DOMAIN_NAME={{TENANT_SHORT_NAME}} read -rs -p "password for {{TENANT_SHORT_NAME}}-cluster: " OS_PASSWORD && export OS_PASSWORD openstack token issue # smoke test ## The keypair-ownership trap The SSH keypair `{{TENANT_SHORT_NAME}}-key` is OWNED by the `-cluster` account, and the platform validates keypair ownership in the cluster creator's context. Consequences: - Never delete `{{TENANT_SHORT_NAME}}-key` or recreate it under another account. A key with the same name owned by `-svc` or a team user will make every cluster create fail. - "Keypair not found" at cluster create almost always means you are signed in as the wrong account -- check `OS_USERNAME` before anything else. ## Cluster lifecycle The standard onboarding delivers a cluster template at handover: openstack coe cluster template show {{TENANT_SHORT_NAME}}-k8s If that template is missing, or you need a variant, create your own -- see "Creating your own cluster template" below. Create (builds take TENS OF MINUTES -- poll slowly or just wait). Size `` against TWO limits, not one: your quota, AND the physical capacity free on the platform right now. A request inside your quota can still fail to place a node -- the symptom is a node stuck in ERROR with "No valid host was found. There are not enough hosts available", which is NOT a quota problem. Start small (1 control + 1-2 workers), confirm it comes up, then scale the worker count: openstack coe cluster create \ --cluster-template {{TENANT_SHORT_NAME}}-k8s \ --keypair {{TENANT_SHORT_NAME}}-key \ --master-count 1 --node-count openstack coe cluster show # poll >=10s apart # done when status = CREATE_COMPLETE Fetch the kubeconfig (treat it as a credential -- never print its contents; store it like any other secret): openstack coe cluster config --dir export KUBECONFIG=/config kubectl get nodes The kubeconfig's client certificate is TIME-LIMITED. If kubectl or a pipeline that worked before starts failing to authenticate, the cert has expired -- re-fetch with the same command and update the stored credential. Read the expiry with: kubectl config view --raw \ -o jsonpath='{.users[0].user.client-certificate-data}' \ | base64 -d | openssl x509 -noout -enddate Cluster create, resize, and delete can be done in the dashboard OR on the command line; everything from the kubeconfig onward (kubectl, CI) is command-line ONLY -- the dashboard cannot drive kubectl. Resize (add or remove workers after the cluster is up -- same account, same free-capacity limit as create): openstack coe cluster resize Delete (confirm with the human first, by name): openstack coe cluster delete ## Creating your own cluster template Templates are PROJECT-SCOPED: everyone in your project can already use them. The one hard rule: NEVER set the Public or Hidden flag (the two checkboxes in the dashboard's Create Cluster Template dialog, or `--public` / `--hidden` on the CLI). Publishing a template cloud-wide is an operator-only action, and the platform refuses the whole create with: Not authorized to set public or hidden flag for cluster template (HTTP 403) In the dashboard this surfaces only as the generic "Error: Unable to create cluster template." -- if you see that toast, the Public/Hidden checkboxes are the FIRST thing to check. You lose nothing by leaving them unchecked. The second hard rule: use `--network-driver calico`. This platform supports calico. Do NOT choose flannel -- a flannel cluster never finishes coming up: the create sits IN_PROGRESS and eventually fails, with the nodes never going Ready. A cluster stuck in CREATE_IN_PROGRESS far past the normal build time is almost always this. Rebuild the template with calico. A known-good CLI starting point, mirroring the platform-delivered template (works signed in as `-svc` or `-cluster`): openstack coe cluster template create \ --coe kubernetes \ --image \ --external-network provider-ext \ --master-flavor --flavor \ --network-driver calico --docker-storage-driver overlay2 \ --master-lb-enabled --floating-ip-enabled \ --fixed-network {{TENANT_SHORT_NAME}}-net \ --fixed-subnet {{TENANT_SHORT_NAME}}-subnet Pick the two machine sizes from `openstack flavor list`; if you have the delivered template, `openstack coe cluster template show {{TENANT_SHORT_NAME}}-k8s` shows the platform's known-good choices. Do NOT bake `--keypair` into the template: the keypair is owned by the `-cluster` account (see the trap above), so it is passed at cluster create time instead, exactly as the lifecycle section shows. ## Images: bring your own registry The platform does NOT host a container registry. Push your application images to a registry you control -- a public one (Docker Hub, GitHub/ GitLab, Quay) or your own private one. The cluster reaches registries over the internet (it pulls its own system images the same way), so a PUBLIC image needs nothing extra. For a PRIVATE registry, create an image-pull secret in the cluster and reference it from the Deployment: kubectl create secret docker-registry regcred \ --docker-server= --docker-username= \ --docker-password= --docker-email= # then in the Deployment pod spec: # imagePullSecrets: # - name: regcred Inject the registry password at pipeline run time from the CI secret store -- never in a manifest in a repository. Full detail: the Jenkins + Kubernetes Implementation Guide, section 5.0. ## Deploying workloads: Services of type LoadBalancer The cluster gets its own load balancer for the Kubernetes API, and it can publish `Service` objects of `type: LoadBalancer` through the same platform mechanism -- no extra setup needed: kubectl expose deployment --type=LoadBalancer --port=80 The Service's external address is allocated from the platform. Each such Service consumes a load balancer (and typically a floating IP) from YOUR quota -- prefer a single ingress controller of type LoadBalancer fronting many Services over one LoadBalancer per app. A LoadBalancer Service (or an ingress) gets an IP, not a hostname: the platform provides no DNS. Map your own hostnames to that IP in whatever DNS you run. Ingress how-to: the Jenkins + Kubernetes Guide, section 5.4. ## Persistent storage Stateless workloads need nothing extra. A stateful one (a database, a cache with persistence) needs a PersistentVolume, which comes from a StorageClass -- check with `kubectl get storageclass`. If a class is marked `(default)`, a PersistentVolumeClaim binds to it automatically; if NONE is listed, the cluster has no persistent-volume support and a PVC stays Pending -- raise it with {{ACCOUNT_CONTACT}}. Never work around it by writing to node-local disk (lost when a pod moves nodes). ## Clusters and CI (the recommended split) For the full worked pipeline (kubeconfig as a Jenkins credential, a deploy stage, and using the cluster as a build-agent pool), the delivered Jenkins + Kubernetes Implementation Guide is the single end-to-end reference. The split below is the principle it follows. - Create long-lived clusters manually (or on a controlled schedule) as `-cluster`, OUTSIDE CI. - Pipelines then deploy INTO the cluster using its kubeconfig, stored in the CI secret store like any other credential. - Do NOT put the `-cluster` password into CI so pipelines can create clusters. If your workflow genuinely needs cluster-per-run, raise it with {{ACCOUNT_CONTACT}} first -- there are capacity and quota implications. - For day-to-day build agents, a scoped Kubernetes ServiceAccount token (namespace-limited) is better practice than handing CI the full-admin kubeconfig. The Jenkins + Kubernetes Guide (section 6.1) has the steps. Day-2 cloud work around the cluster (networks, floating IPs, volumes, extra load balancers) still runs as `-svc` per `references/day2-operations.md`; only cluster create/delete needs the `-cluster` password login.