# Kubernetes clusters on Omega Cloud

## The one non-negotiable: the `-cluster` account, with its PASSWORD

Cluster create and delete run ONLY as `{{TENANT_SHORT_NAME}}-cluster`,
signed in with its password. The platform's cluster machinery cannot
operate through an application credential -- a cluster create
attempted with the `-svc` credential fails every time, no matter how
it is phrased. This is a platform constraint, not a configuration you
(or the operator) can change. Do not "simplify" cluster automation to
the application credential; it will never work.

Sign in (prompt the human for the password -- never read it from a
file into your context, never store it in CI):

    export OS_AUTH_TYPE=password
    export OS_AUTH_URL={{AUTH_URL}}
    export OS_IDENTITY_API_VERSION=3
    export OS_CACERT=<path to the delivered CA bundle>
    export OS_USERNAME={{TENANT_SHORT_NAME}}-cluster
    export OS_USER_DOMAIN_NAME={{TENANT_SHORT_NAME}}
    export OS_PROJECT_NAME={{TENANT_SHORT_NAME}}-prod
    export OS_PROJECT_DOMAIN_NAME={{TENANT_SHORT_NAME}}
    read -rs -p "password for {{TENANT_SHORT_NAME}}-cluster: " OS_PASSWORD && export OS_PASSWORD
    openstack token issue        # smoke test

## The keypair-ownership trap

The SSH keypair `{{TENANT_SHORT_NAME}}-key` is OWNED by the
`-cluster` account, and the platform validates keypair ownership in
the cluster creator's context. Consequences:

- Never delete `{{TENANT_SHORT_NAME}}-key` or recreate it under
  another account. A key with the same name owned by `-svc` or a team
  user will make every cluster create fail.
- "Keypair not found" at cluster create almost always means you are
  signed in as the wrong account -- check `OS_USERNAME` before
  anything else.

## Cluster lifecycle

The standard onboarding delivers a cluster template at handover:

    openstack coe cluster template show {{TENANT_SHORT_NAME}}-k8s

If that template is missing, or you need a variant, create your own --
see "Creating your own cluster template" below.

Create (builds take TENS OF MINUTES -- poll slowly or just wait). Size
`<N>` against TWO limits, not one: your quota, AND the physical capacity
free on the platform right now. A request inside your quota can still fail
to place a node -- the symptom is a node stuck in ERROR with "No valid
host was found. There are not enough hosts available", which is NOT a
quota problem. Start small (1 control + 1-2 workers), confirm it comes up,
then scale the worker count:

    openstack coe cluster create <cluster-name> \
      --cluster-template {{TENANT_SHORT_NAME}}-k8s \
      --keypair {{TENANT_SHORT_NAME}}-key \
      --master-count 1 --node-count <N>

    openstack coe cluster show <cluster-name>   # poll >=10s apart
    # done when status = CREATE_COMPLETE

Fetch the kubeconfig (treat it as a credential -- never print its
contents; store it like any other secret):

    openstack coe cluster config <cluster-name> --dir <secure-dir>
    export KUBECONFIG=<secure-dir>/config
    kubectl get nodes

The kubeconfig's client certificate is TIME-LIMITED. If kubectl or a
pipeline that worked before starts failing to authenticate, the cert has
expired -- re-fetch with the same command and update the stored
credential. Read the expiry with:

    kubectl config view --raw \
      -o jsonpath='{.users[0].user.client-certificate-data}' \
      | base64 -d | openssl x509 -noout -enddate

Cluster create, resize, and delete can be done in the dashboard OR on the
command line; everything from the kubeconfig onward (kubectl, CI) is
command-line ONLY -- the dashboard cannot drive kubectl.

Resize (add or remove workers after the cluster is up -- same account,
same free-capacity limit as create):

    openstack coe cluster resize <cluster-name> <new-node-count>

Delete (confirm with the human first, by name):

    openstack coe cluster delete <cluster-name>

## Creating your own cluster template

Templates are PROJECT-SCOPED: everyone in your project can already use
them. The one hard rule: NEVER set the Public or Hidden flag (the two
checkboxes in the dashboard's Create Cluster Template dialog, or
`--public` / `--hidden` on the CLI). Publishing a template cloud-wide
is an operator-only action, and the platform refuses the whole create
with:

    Not authorized to set public or hidden flag for cluster template (HTTP 403)

In the dashboard this surfaces only as the generic "Error: Unable to
create cluster template." -- if you see that toast, the Public/Hidden
checkboxes are the FIRST thing to check. You lose nothing by leaving
them unchecked.

The second hard rule: use `--network-driver calico`. This platform
supports calico. Do NOT choose flannel -- a flannel cluster never
finishes coming up: the create sits IN_PROGRESS and eventually fails,
with the nodes never going Ready. A cluster stuck in CREATE_IN_PROGRESS
far past the normal build time is almost always this. Rebuild the
template with calico.

A known-good CLI starting point, mirroring the platform-delivered
template (works signed in as `-svc` or `-cluster`):

    openstack coe cluster template create <template-name> \
      --coe kubernetes \
      --image <public kube image name or UUID> \
      --external-network provider-ext \
      --master-flavor <flavor> --flavor <flavor> \
      --network-driver calico --docker-storage-driver overlay2 \
      --master-lb-enabled --floating-ip-enabled \
      --fixed-network {{TENANT_SHORT_NAME}}-net \
      --fixed-subnet {{TENANT_SHORT_NAME}}-subnet

Pick the two machine sizes from `openstack flavor list`; if you have
the delivered template, `openstack coe cluster template show
{{TENANT_SHORT_NAME}}-k8s` shows the platform's known-good choices.

Do NOT bake `--keypair` into the template: the keypair is owned by the
`-cluster` account (see the trap above), so it is passed at cluster
create time instead, exactly as the lifecycle section shows.

## Images: bring your own registry

The platform does NOT host a container registry. Push your application
images to a registry you control -- a public one (Docker Hub, GitHub/
GitLab, Quay) or your own private one. The cluster reaches registries
over the internet (it pulls its own system images the same way), so a
PUBLIC image needs nothing extra.

For a PRIVATE registry, create an image-pull secret in the cluster and
reference it from the Deployment:

    kubectl create secret docker-registry regcred \
      --docker-server=<registry-host> --docker-username=<user> \
      --docker-password=<token> --docker-email=<you@example.com>

    # then in the Deployment pod spec:
    #   imagePullSecrets:
    #     - name: regcred

Inject the registry password at pipeline run time from the CI secret
store -- never in a manifest in a repository. Full detail: the Jenkins +
Kubernetes Implementation Guide, section 5.0.

## Deploying workloads: Services of type LoadBalancer

The cluster gets its own load balancer for the Kubernetes API, and it
can publish `Service` objects of `type: LoadBalancer` through the
same platform mechanism -- no extra setup needed:

    kubectl expose deployment <app> --type=LoadBalancer --port=80

The Service's external address is allocated from the platform. Each
such Service consumes a load balancer (and typically a floating IP)
from YOUR quota -- prefer a single ingress controller of type
LoadBalancer fronting many Services over one LoadBalancer per app.

A LoadBalancer Service (or an ingress) gets an IP, not a hostname: the
platform provides no DNS. Map your own hostnames to that IP in whatever
DNS you run. Ingress how-to: the Jenkins + Kubernetes Guide, section 5.4.

## Persistent storage

Stateless workloads need nothing extra. A stateful one (a database, a
cache with persistence) needs a PersistentVolume, which comes from a
StorageClass -- check with `kubectl get storageclass`. If a class is
marked `(default)`, a PersistentVolumeClaim binds to it automatically;
if NONE is listed, the cluster has no persistent-volume support and a
PVC stays Pending -- raise it with {{ACCOUNT_CONTACT}}. Never work around
it by writing to node-local disk (lost when a pod moves nodes).

## Clusters and CI (the recommended split)

For the full worked pipeline (kubeconfig as a Jenkins credential, a deploy
stage, and using the cluster as a build-agent pool), the delivered Jenkins +
Kubernetes Implementation Guide is the single end-to-end reference. The split
below is the principle it follows.

- Create long-lived clusters manually (or on a controlled schedule)
  as `-cluster`, OUTSIDE CI.
- Pipelines then deploy INTO the cluster using its kubeconfig, stored
  in the CI secret store like any other credential.
- Do NOT put the `-cluster` password into CI so pipelines can create
  clusters. If your workflow genuinely needs cluster-per-run, raise
  it with {{ACCOUNT_CONTACT}} first -- there are capacity and quota
  implications.
- For day-to-day build agents, a scoped Kubernetes ServiceAccount token
  (namespace-limited) is better practice than handing CI the full-admin
  kubeconfig. The Jenkins + Kubernetes Guide (section 6.1) has the steps.

Day-2 cloud work around the cluster (networks, floating IPs, volumes,
extra load balancers) still runs as `-svc` per
`references/day2-operations.md`; only cluster create/delete needs the
`-cluster` password login.
