# Omega Cloud -- Jenkins + Kubernetes Implementation Guide (TEMPLATE)

TEMPLATE NOTE (removed before delivery): fields written as {{THIS}} are filled
in by us per client; fields written as <this> are yours to fill on your side
(from the credential files delivered to your custodians, or from your own
Jenkins). Secrets are never written into this document, a repository, or a
pipeline file -- they are injected at run time from your Jenkins credential
store.

This is the one document you need to take a Jenkins server that already reaches
Omega Cloud and get it building and deploying to a Kubernetes cluster running in
your tenancy. It is written as a single top-to-bottom workflow so you do not
have to jump between documents to complete a task. Where a topic has a deeper
reference, it is named at the end (section 11); you can finish the whole flow
without opening any of them.

Two integration patterns are covered, and most teams use both:

- Pattern A (the main flow, sections 5 and 7): your Jenkins pipeline DEPLOYS an
  application into the cluster, using the cluster's kubeconfig as a stored
  credential. Jenkins stays where it is; only your app runs in the cluster.
- Pattern B (section 6): your Jenkins uses the cluster as an elastic pool of
  BUILD AGENTS -- each job runs in a short-lived pod inside the cluster.

--------------------------------------------------------------------------------

## 0. The picture: what you are connecting

    +-------------------+        identity + API         +---------------------+
    |  Jenkins server   |  ---------------------------> |    Omega Cloud      |
    |  (on your network)|   TLS, application credential  |  (your tenancy)     |
    +-------------------+                                +---------------------+
            |                                                      |
            |  kubeconfig (a stored Jenkins credential)            | creates
            |  talks to the cluster's API over its floating IP     v
            |                                            +---------------------+
            +------------------------------------------> | Kubernetes cluster  |
                     kubectl / helm / build-agent pods   |  (control + workers)|
                                                         +---------------------+
                                                                   |
                                                     LoadBalancer Service gets a
                                                     floating IP -> your app is
                                                     reachable from outside.

What each piece does:

- Jenkins server: yours, on your network. It already reaches the cloud (you have
  tested this). It holds two kinds of secret in its credential store: an
  application credential (for talking to the cloud API) and a kubeconfig (for
  talking to the cluster). Neither is ever written into a repository.
- The cloud API + identity service: authenticates your automation and lets you
  create the cluster, networks, load balancers, and floating IPs -- all within
  your quota.
- The Kubernetes cluster: a managed set of virtual machines (one or more control
  nodes plus worker nodes) running in your project. You create it once; your
  pipelines then deploy into it many times.
- A LoadBalancer Service: when your app asks Kubernetes for a load balancer, the
  cloud gives the cluster a real load balancer and a floating IP, so the app is
  reachable from outside the cluster. Each one draws from your quota.

Three accounts, and which one to use here (full detail in the Handover Pack):

| Account                          | Use it for                                  |
|----------------------------------|---------------------------------------------|
| `{{TENANT_SHORT_NAME}}-svc`      | All day-to-day automation via its           |
|                                  | application credential: networks, servers,  |
|                                  | load balancers, deploying into the cluster. |
| `{{TENANT_SHORT_NAME}}-cluster`  | Creating and deleting Kubernetes clusters,  |
|                                  | with its PASSWORD. Nothing else.            |
| `{{TENANT_SHORT_NAME}}-domain-admin` | Managing your own users. Identity only. |

The one rule that prevents the most common failure: cluster CREATE and DELETE
run only as `{{TENANT_SHORT_NAME}}-cluster` with its password. They cannot be
done with an application credential, so they do not belong in a pipeline. Create
the cluster by hand (section 2); let pipelines DEPLOY into it (section 5).

### Two ways to drive the cloud: dashboard or command line

Know up front which tool does what, because the two are not interchangeable
across this whole workflow:

- CLUSTER LIFECYCLE -- creating, resizing, and deleting the cluster, and managing
  networks, floating IPs, and load balancers -- can be done EITHER in the web
  dashboard OR on the command line. Use whichever you prefer; both are shown.
- EVERYTHING KUBERNETES -- fetching the kubeconfig, and every `kubectl` / `helm`
  step, Jenkins wiring, deploys, ingress, storage, and Kubernetes teardown -- is
  COMMAND LINE ONLY. The dashboard cannot drive `kubectl`.

So even if you create your cluster in the dashboard, you cross over to the command
line at the kubeconfig step (section 3) and stay there. There is no dashboard-only
path for the Jenkins workflow: install the OpenStack client, `kubectl`, and `helm`
on your Jenkins agents (section 1) no matter how you create the cluster.

Throughout the guide, steps that differ by path are marked **On the command line**
and **In the dashboard**. A step with neither marker is command line and applies
to both paths.

--------------------------------------------------------------------------------

## 1. Before you start

A checklist. If any item is missing, stop here and get it before continuing.

- [ ] The credential files for `{{TENANT_SHORT_NAME}}-svc`,
      `{{TENANT_SHORT_NAME}}-cluster`, and `{{TENANT_SHORT_NAME}}-domain-admin`,
      delivered to your custodians.
- [ ] The delivered CA bundle file ({{CA_BUNDLE_FILE}}). You point every client
      at it; you never disable certificate verification instead.
- [ ] The OpenStack command-line client on your Jenkins agents (or in the agent
      container image), plus `kubectl` and, if you use it, `helm`.
- [ ] Jenkins plugins: "Credentials Binding" and "Kubernetes CLI" for Pattern A;
      add "Kubernetes" (the cloud/agent plugin) for Pattern B.
- [ ] Confirmed network reach from Jenkins to the cloud API endpoint (you have
      tested this).

Installing the three clients: install the OpenStack command-line client (with its
cluster `coe` commands), `kubectl`, and `helm`, each per its own project's install
docs. Keep `kubectl` within one minor version of your cluster's Kubernetes (the
delivered template builds v1.34, so a `kubectl` in the 1.33-1.35 range is safe) --
a `kubectl` far newer or older than the cluster can misbehave in confusing ways.

Cloud connection settings. Put these in a `clouds.yaml` on the agent (the client
reads the current directory, then ~/.config/openstack/, then /etc/openstack/),
and select it with `export OS_CLOUD={{TENANT_SHORT_NAME}}`:

    clouds:
      {{TENANT_SHORT_NAME}}:
        auth_type: v3applicationcredential
        auth:
          auth_url: {{AUTH_URL}}
          application_credential_id: <id from the delivered credential file>
          application_credential_secret: <injected from your Jenkins secret store>
        region_name: {{REGION}}
        identity_api_version: 3
        cacert: <path to the delivered CA bundle {{CA_BUNDLE_FILE}}>

The application credential id and secret come from the delivered credential
file. The secret is injected at run time from your Jenkins credential store, not
written to disk in a repository. (Full CI credential handling: the CI/Automation
Integration Guide.)

--------------------------------------------------------------------------------

## 2. Step 1 -- create your Kubernetes cluster

You do this once, by hand, signed in as `{{TENANT_SHORT_NAME}}-cluster`. It is
not a pipeline step (see the rule in section 0).

### 2.1 Sign in as the cluster account

Prompt for the password; never read it from a file into a script, never store
it in Jenkins:

    export OS_AUTH_TYPE=password
    export OS_AUTH_URL={{AUTH_URL}}
    export OS_IDENTITY_API_VERSION=3
    export OS_CACERT=<path to the delivered CA bundle>
    export OS_USERNAME={{TENANT_SHORT_NAME}}-cluster
    export OS_USER_DOMAIN_NAME={{TENANT_SHORT_NAME}}
    export OS_PROJECT_NAME={{TENANT_SHORT_NAME}}-prod
    export OS_PROJECT_DOMAIN_NAME={{TENANT_SHORT_NAME}}
    read -rs -p "password for {{TENANT_SHORT_NAME}}-cluster: " OS_PASSWORD && export OS_PASSWORD
    openstack token issue        # smoke test: should print a token

### 2.2 Choose or build a cluster template

If a template named `{{TENANT_SHORT_NAME}}-k8s` was delivered with your tenancy,
use it and skip to 2.3:

    openstack coe cluster template show {{TENANT_SHORT_NAME}}-k8s

If you need to build your own, three rules keep the create from failing:

- Use `--network-driver calico`. This platform's cluster service supports
  calico. Do NOT use flannel: a flannel cluster never finishes coming up -- the
  create sits in progress and eventually fails, with the nodes never going
  ready. This is the single most common self-inflicted cluster failure.
- Do NOT set the Public or Hidden flag (the two checkboxes in the dashboard's
  Create Cluster Template dialog, or `--public` / `--hidden` on the command
  line). Publishing a template to the whole cloud is an operator-only action;
  setting either flag makes the entire create fail with, on the command line,
  "Not authorized to set public or hidden flag for cluster template", and in the
  dashboard, only the generic "Error: Unable to create cluster template."
  Templates are already usable by everyone in your project; you lose nothing by
  leaving these unchecked.
- Do NOT bake a keypair into the template. The SSH keypair is owned by the
  `{{TENANT_SHORT_NAME}}-cluster` account and is supplied at cluster-create time,
  not in the template.

A known-good template, mirroring the platform default:

    openstack coe cluster template create <template-name> \
      --coe kubernetes \
      --image ubuntu-jammy-kube-v1.34.8 \
      --external-network provider-ext \
      --master-flavor <master-flavor> --flavor <worker-flavor> \
      --network-driver calico --docker-storage-driver overlay2 \
      --master-lb-enabled --floating-ip-enabled \
      --fixed-network {{TENANT_SHORT_NAME}}-net \
      --fixed-subnet {{TENANT_SHORT_NAME}}-subnet

Pick flavors from `openstack flavor list`. The delivered template's choices are
a safe default; read them with the `template show` command above.

### 2.3 Create the cluster, sized to fit

    openstack coe cluster create <cluster-name> \
      --cluster-template {{TENANT_SHORT_NAME}}-k8s \
      --keypair {{TENANT_SHORT_NAME}}-key \
      --master-count 1 --node-count <N>

Size `<N>` against TWO limits, not one:

- Your quota (instances, cores, memory). The dashboard and CLI show your usage.
- The physical capacity free on the cloud right now. A request that is inside
  your quota can still fail if no host has room for another node of that size.
  The symptom is a node stuck in ERROR with "No valid host was found. There are
  not enough hosts available." It is not a quota problem and raising quota will
  not fix it.

Practical guidance: start SMALL -- 1 control node and 1 or 2 workers -- confirm
the cluster comes up, then scale the worker count up if you need more and
capacity allows. A large first request is the second most common cluster
failure, after the flannel mistake.

**In the dashboard:** Container Infra > Clusters > Create Cluster. Choose the
`{{TENANT_SHORT_NAME}}-k8s` template, set the keypair to
`{{TENANT_SHORT_NAME}}-key`, keep the node count small (1 master, 1-2 workers),
and leave Public and Hidden UNCHECKED. (Your dashboard URL is in the Handover
Pack.) The same two sizing limits -- quota and free capacity -- apply exactly as
above; the "No valid host" failure looks the same from the dashboard.

### 2.4 Wait for it, and verify

    openstack coe cluster show <cluster-name> -f value -c status -c status_reason
    # poll no faster than every 10 seconds; done when status = CREATE_COMPLETE

Cluster builds take tens of minutes -- this is normal. But if a cluster sits in
CREATE_IN_PROGRESS far longer than the build should take and never completes,
the usual cause is the network driver: rebuild with calico (see 2.2). Delete the
stuck cluster first: `openstack coe cluster delete <cluster-name>`.

### 2.5 Scaling the cluster later

Add or remove workers after the cluster is up. This is cluster lifecycle, so it
runs as `{{TENANT_SHORT_NAME}}-cluster`, the same account and the same free-capacity
limit as the first create (2.3):

    openstack coe cluster resize <cluster-name> <new-node-count>

**In the dashboard:** Container Infra > Clusters > (your cluster) > Resize, and set
the new node count.

Growing the cluster is subject to the same "No valid host" capacity limit as the
first create -- a resize larger than the free capacity fails the same way. When you
shrink, Kubernetes reschedules the removed nodes' pods onto the survivors, so make
sure the remaining workers have room for them.

--------------------------------------------------------------------------------

## 3. Step 2 -- get and understand the kubeconfig

The kubeconfig is the file `kubectl` and Jenkins use to talk to your cluster's
Kubernetes API. Fetch it, still signed in as `{{TENANT_SHORT_NAME}}-cluster`:

    umask 077
    openstack coe cluster config <cluster-name> --dir <secure-dir>
    export KUBECONFIG=<secure-dir>/config
    kubectl get nodes        # every node should report Ready

What is inside it, and why it matters:

- It points at your cluster's API server at its floating IP, on port 6443.
- It authenticates with a CLIENT CERTIFICATE embedded in the file (plus the
  cluster's CA). There is no username or password in it. Anyone holding this file
  has administrative access to your cluster -- treat it exactly like a password.
- The certificate is time-limited. When it expires, `kubectl` and your pipelines
  start failing to authenticate; re-fetch the kubeconfig with the same command
  and update the stored credential (section 4). Confirm the expiry date for your
  cluster -- you can read it from the certificate embedded in the kubeconfig, or
  ask {{ACCOUNT_CONTACT}} -- and set a renewal reminder ahead of it.

Read the expiry date straight from the kubeconfig you just fetched:

    kubectl config view --raw \
      -o jsonpath='{.users[0].user.client-certificate-data}' \
      | base64 -d | openssl x509 -noout -enddate

Never commit the kubeconfig to a repository. It goes into the Jenkins credential
store next, and nowhere else.

If you created the cluster in the dashboard: that is fine, but this step and
everything after it -- fetching the kubeconfig, storing it in Jenkins
(section 4), and the pipelines -- run through the OpenStack command-line client
and your application credential, not the dashboard. Fetch the kubeconfig with the
`openstack coe cluster config` command above; if you have not installed the
client yet (section 1), do that first.

--------------------------------------------------------------------------------

## 4. Step 3 -- store the kubeconfig in Jenkins

In Jenkins: Manage Jenkins > Credentials > (your domain) > Add Credentials.

- Kind: "Secret file".
- File: upload the `config` file you fetched in section 3.
- ID: `omega-k8s-kubeconfig` (used by the pipeline examples below).
- Scope: the narrowest that your jobs need.

That is the only place the kubeconfig lives inside Jenkins. Pipelines reference
it by ID; the "Kubernetes CLI" plugin loads it for the duration of a block and
masks it in the log. When you re-fetch the kubeconfig (expiry, or a rebuilt
cluster), update this one credential and every pipeline picks up the change.

--------------------------------------------------------------------------------

## 5. Step 4, Pattern A -- deploy your app into the cluster

This is the main flow: a pipeline that rolls an application out to the cluster
using the stored kubeconfig. Nothing here needs the cluster password; deploying
into an existing cluster is ordinary automation.

### 5.0 Build and publish your image first

Deploying pulls a container image; it does not build one. Building your app into
an image and pushing it to a registry the cluster can pull from is an upstream
step, and in a real pipeline it is a Build stage (and a push stage) that runs
before the Deploy stage below. The examples here use the public `nginx:stable`
image so you can prove the deploy path without a registry; swap in your own
image once it is built and pushed.

Omega Cloud does not provide a container registry. Push your images to a registry
you control -- a public one (Docker Hub, GitHub/GitLab, Quay) or your own private
one -- that the cluster can reach over the internet. It can: that is how the
cluster pulled its own system images at build time. For a PUBLIC image nothing
more is needed. For a PRIVATE registry, create an image-pull secret in the cluster
and reference it from your Deployment:

    kubectl create secret docker-registry regcred \
      --docker-server=<registry-host> --docker-username=<user> \
      --docker-password=<token-or-password> --docker-email=<you@example.com>

Then add to the Deployment's pod `spec`:

    spec:
      imagePullSecrets:
        - name: regcred
      containers:
        - name: web
          image: <registry-host>/<you>/<image>:<tag>

Inject the registry password from your Jenkins credential store at run time; never
write it into a manifest in a repository.

### 5.1 A minimal application manifest

Keep your Kubernetes manifests in your application repository, for example in a
`k8s/` directory. A minimal Deployment plus a LoadBalancer Service:

    # k8s/app.yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: sample-app
    spec:
      replicas: 2
      selector:
        matchLabels: { app: sample-app }
      template:
        metadata:
          labels: { app: sample-app }
        spec:
          containers:
            - name: web
              image: nginx:stable
              ports:
                - containerPort: 80
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: sample-app
    spec:
      type: LoadBalancer
      selector: { app: sample-app }
      ports:
        - port: 80
          targetPort: 80

When Kubernetes sees `type: LoadBalancer`, the cloud gives the cluster a load
balancer and a floating IP for this Service. That address is how the outside
world reaches your app.

### 5.2 The pipeline

    // Jenkinsfile -- Pattern A: deploy into the cluster
    pipeline {
      agent any
      options { timestamps(); timeout(time: 30, unit: 'MINUTES') }
      stages {
        stage('Deploy') {
          steps {
            // The Kubernetes CLI plugin loads the stored kubeconfig for this
            // block and masks it in the log.
            withKubeConfig([credentialsId: 'omega-k8s-kubeconfig']) {
              // Applying the whole k8s/ directory keeps deploy and teardown
              // symmetric: `kubectl delete -f k8s/` later removes everything
              // this created, including the Service (and its load balancer).
              sh 'kubectl apply -f k8s/'
              sh 'kubectl rollout status deployment/sample-app --timeout=5m'
            }
          }
        }
        stage('Show address') {
          steps {
            withKubeConfig([credentialsId: 'omega-k8s-kubeconfig']) {
              // The external IP appears once the load balancer finishes
              // provisioning (a few minutes on the first apply).
              sh 'kubectl get service sample-app -o wide'
            }
          }
        }
      }
    }

### 5.3 Load balancers and your quota

Every Service of `type: LoadBalancer` consumes one load balancer AND one
floating IP from your quota, and keeps them for as long as the Service exists.
Two consequences:

- Do not create a LoadBalancer Service per microservice. Prefer a single ingress
  controller (itself one LoadBalancer Service) that routes to many internal
  Services by hostname or path. That turns N load balancers into one.
- When you tear an app down, delete its Service (section 8). Deleting only the
  Deployment leaves the load balancer and floating IP allocated -- and billed
  against your quota.

### 5.4 One ingress instead of many load balancers

Section 5.3 said to prefer a single ingress controller over one LoadBalancer
Service per app. Here is how. Install an ingress controller once; it takes ONE
LoadBalancer Service -- and so one floating IP -- for your whole tenancy:

    helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
    helm install ingress-nginx ingress-nginx/ingress-nginx \
      --namespace ingress-nginx --create-namespace \
      --set controller.service.type=LoadBalancer

Wait for its external IP (the same load-balancer provisioning wait as any Service):

    kubectl get service -n ingress-nginx ingress-nginx-controller -o wide

Then give each app an ordinary ClusterIP Service (NOT `type: LoadBalancer`) and
route to it with an Ingress. Every additional app is one more Ingress rule behind
the same single load balancer:

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: sample-app
    spec:
      ingressClassName: nginx
      rules:
        - host: app.example.com          # a name you point at the ingress IP
          http:
            paths:
              - path: /
                pathType: Prefix
                backend:
                  service:
                    name: sample-app
                    port: { number: 80 }

The platform does not provide DNS. The ingress (like any LoadBalancer Service) gets
an IP, not a hostname; map your own hostnames to that IP in whatever DNS you run.

### 5.5 Persistent storage for stateful apps

The sample app is stateless. A workload that must keep data (a database, a cache
with persistence) needs a PersistentVolume, which comes from a StorageClass. Check
what your cluster offers first:

    kubectl get storageclass

- If a class is listed and marked `(default)`, a PersistentVolumeClaim like the one
  below binds to it automatically.
- If NO StorageClass is listed, your cluster was built without persistent-volume
  support (the delivered template does not enable it by default). Stateless
  workloads are unaffected; if you need persistent storage, ask {{ACCOUNT_CONTACT}}
  to enable it. Do not work around it by writing to node-local disk -- that data is
  lost whenever a pod moves to another node.

A claim, and how a pod uses it:

    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: data
    spec:
      accessModes: [ ReadWriteOnce ]
      resources: { requests: { storage: 5Gi } }
    # then in your pod spec:
    #   volumes:
    #     - name: data
    #       persistentVolumeClaim: { claimName: data }

Persistent volumes are backed by cloud block storage and count against your volume
quota, the same as any disk.

--------------------------------------------------------------------------------

## 6. Pattern B -- the cluster as a Jenkins build-agent pool

Instead of (or in addition to) deploying INTO the cluster, you can have Jenkins
run its BUILD jobs as short-lived pods inside the cluster. Each job gets a fresh,
isolated agent that disappears when the build ends.

Setup, using the "Kubernetes" plugin:

1. Manage Jenkins > Clouds > Add a new cloud > Kubernetes.
2. Kubernetes URL: your cluster's API endpoint, `https://<cluster-floating-ip>:6443`
   (the `server:` line in the kubeconfig from section 3).
3. Credentials: add the kubeconfig (or a Kubernetes service-account token scoped
   to a build namespace) as the cloud's credential. A dedicated service account
   with a limited role is better practice than the full-admin kubeconfig for
   day-to-day agents.
4. Jenkins tunnel / URL: pods need to reach your Jenkins to register as agents,
   so set the Jenkins URL to an address the cluster can reach.
5. Define a Pod Template: a `jnlp` agent container plus any tool containers your
   builds need (for example a container with `kubectl` and `helm`). Give it a
   label; jobs that request that label run in these pods.

Two things to keep in mind:

- These agent pods run on your cluster's worker nodes and consume their CPU and
  memory. Size the cluster (section 2.3) for your build concurrency, or set pod
  resource requests so builds queue rather than overcommit.
- Network reachability goes both ways: Jenkins must reach the cluster API
  (endpoint above), and the pods must reach Jenkins (step 4). You have already
  confirmed the first direction; confirm the second when you first enable this.

### 6.1 A scoped ServiceAccount instead of the admin kubeconfig

The kubeconfig from section 3 carries full cluster-admin. For agents that run day
in, day out, better practice is a Kubernetes ServiceAccount holding only the
rights those jobs need, and to hand Jenkins that account's token instead of the
admin kubeconfig. Run these once, authenticated with the section-3 kubeconfig:

    # a namespace for build workloads, and a service account inside it
    kubectl create namespace ci
    kubectl create serviceaccount jenkins -n ci

    # grant only what the agents need. 'edit' confined to the 'ci' namespace is a
    # reasonable starting point; scope it down with a custom Role if jobs need less.
    kubectl create rolebinding jenkins-edit -n ci \
      --clusterrole=edit --serviceaccount=ci:jenkins

Then get a token for that account -- two options, pick by whether your Jenkins
can refresh it:

- Short-lived (preferred): `kubectl -n ci create token jenkins --duration=24h`
  mints a time-boxed token. A pipeline step or small refresh job requests a new
  one before the old expires; nothing long-lived is stored.
- Long-lived (if Jenkins cannot refresh): create a Secret of type
  `kubernetes.io/service-account-token` annotated with
  `kubernetes.io/service-account.name: jenkins`, then read its `token` field.
  This token does not expire on its own -- treat it exactly like the kubeconfig,
  and revoke it by deleting the Secret if it ever leaks.

Store the chosen token in Jenkins as a "Secret text" credential and select it as
the Kubernetes cloud's credential (step 3 above) in place of the kubeconfig. A
namespace-scoped account cannot damage the rest of the cluster if a build is
compromised, which is the whole point of using one.

Pattern B does not replace Pattern A: a common setup is build agents in the
cluster (B) that then deploy the built artifact to the same or another cluster
(A).

--------------------------------------------------------------------------------

## 7. Worked example, end to end

This runs the whole flow once with a stock image, to prove the path before you
wire in your own app. Sections in parentheses point back to the detail. It is
written on the command line end to end; if you prefer the dashboard, create the
cluster (steps 1-2) there and pick up at step 3, which is command line for
everyone.

1. Sign in as `{{TENANT_SHORT_NAME}}-cluster` (2.1) and create a small cluster
   from the delivered template (2.3):

       openstack coe cluster create demo-k8s \
         --cluster-template {{TENANT_SHORT_NAME}}-k8s \
         --keypair {{TENANT_SHORT_NAME}}-key \
         --master-count 1 --node-count 1

2. Wait for CREATE_COMPLETE (2.4):

       openstack coe cluster show demo-k8s -f value -c status

3. Fetch and check the kubeconfig (3):

       umask 077
       openstack coe cluster config demo-k8s --dir ./demo-kube
       export KUBECONFIG=$PWD/demo-kube/config
       kubectl get nodes

4. Store `./demo-kube/config` in Jenkins as the "Secret file" credential
   `omega-k8s-kubeconfig` (4). (For a quick local proof you can skip Jenkins and
   use the `KUBECONFIG` export above directly.)

5. Deploy the sample app (5.1):

       kubectl apply -f k8s/
       kubectl rollout status deployment/sample-app --timeout=5m

6. Read the external address (wait for it to move from <pending> to an IP; the
   load balancer takes a few minutes on first creation):

       kubectl get service sample-app -o wide
       # then, from a host that can reach it:
       curl http://<external-ip>/

7. Tear the example down (8):

       kubectl delete -f k8s/         # deletes the Service too -> releases
                                      # the load balancer and floating IP
       openstack coe cluster delete demo-k8s

--------------------------------------------------------------------------------

## 8. Teardown and quota hygiene

Delete in reverse order of creation, and delete the Service before anything
else, so the load balancer and floating IP are released:

1. `kubectl delete -f k8s/` (or `kubectl delete service <name>`). This is what
   releases the load balancer and its floating IP. Deleting the same directory
   you applied catches every manifest, so no Service is left stranded.
2. Delete other cloud resources your build created (servers, volumes, extra
   floating IPs), newest first.
3. If you are finished with the cluster itself, delete it as
   `{{TENANT_SHORT_NAME}}-cluster`: `openstack coe cluster delete <cluster-name>`.

**In the dashboard:** you can release floating IPs (Network > Floating IPs), delete
load balancers (Network > Load Balancers), and delete the cluster (Container Infra >
Clusters) the same way. But deleting Kubernetes objects -- Services, Deployments --
is `kubectl` only, and deleting the Service is what releases its load balancer. So
delete the Service first (step 1) whichever way you remove the rest.

Two hygiene rules that keep quota from silently filling:

- A floating IP counts against your quota even when it is not attached to
  anything. Release IPs you are not using.
- Make cleanup automatic. In a Jenkins pipeline, put teardown of per-build
  resources in a `post { always { ... } }` block so a failed build cannot leak
  quota. The delivered starter-kit sweep script (`ci-cleanup-sweep.sh`) removes
  leftovers by name prefix and age; run it as a scheduled job as a backstop.

--------------------------------------------------------------------------------

## 9. Troubleshooting

Symptoms you are most likely to hit, and the fix:

- "Missing value auth-url required for auth plugin password" -- the shell has no
  cloud environment loaded. You started a fresh shell or a fresh agent. Re-source
  your `OS_*` environment or select your `clouds.yaml` entry
  (`export OS_CLOUD={{TENANT_SHORT_NAME}}`) before the command.

- Cluster TEMPLATE create fails; dashboard shows only "Error: Unable to create
  cluster template." -- the Public or Hidden flag is set. Uncheck both (2.2).

- Cluster create fails, a node in ERROR with "No valid host was found. There are
  not enough hosts available." -- the request is too big for the free capacity
  right now, even if it is within quota. Use a smaller node count or a smaller
  flavor and retry (2.3). Raising quota will not help.

- Cluster stays in CREATE_IN_PROGRESS far too long and never completes -- most
  often the network driver. Delete it and rebuild the template with calico;
  flannel is not supported (2.2).

- "Keypair not found" at cluster create -- you are signed in as the wrong
  account. Cluster create runs as `{{TENANT_SHORT_NAME}}-cluster`, which owns the
  keypair. Check `echo $OS_USERNAME` first.

- A pipeline tries to create a cluster and is refused -- cluster create and
  delete cannot use an application credential; they need the
  `{{TENANT_SHORT_NAME}}-cluster` password, so they are not pipeline steps.
  Create the cluster by hand (section 2); let the pipeline deploy into it.

- `kubectl` reports "Unable to connect" or a certificate error -- the kubeconfig
  is wrong or its certificate has expired. Re-fetch it (3) and update the stored
  Jenkins credential (4). Make sure you kept the delivered CA and did not disable
  verification.

- A LoadBalancer Service shows EXTERNAL-IP `<pending>` for a long time -- the
  load balancer is still provisioning (a few minutes is normal), or your load
  balancer / floating IP quota is exhausted. Check your quota usage.

- A pod is stuck `ImagePullBackOff` or `ErrImagePull` -- the cluster cannot pull
  the image. Check the image name and tag first; if it is in a PRIVATE registry,
  the cluster needs an image-pull secret referenced from the Deployment (5.0).

- A PersistentVolumeClaim stays `Pending` -- nothing bound it. Run `kubectl get
  storageclass`: if none is listed, the cluster has no persistent-volume support
  and the claim will never bind (5.5); contact {{ACCOUNT_CONTACT}} to enable it.

--------------------------------------------------------------------------------

## 10. Glossary

- Application credential: a scoped, revocable secret for the
  `{{TENANT_SHORT_NAME}}-svc` account. All automation authenticates with it. It
  cannot create clusters.
- The cluster account (`{{TENANT_SHORT_NAME}}-cluster`): the only account that
  creates and deletes clusters, using its password. It owns the cluster keypair.
- Cluster template: the recipe (image, flavors, network driver, options) a
  cluster is built from. Use calico; never set Public/Hidden.
- kubeconfig: the certificate-bearing file that grants admin access to a
  cluster's Kubernetes API. Treat it as a password; store it in Jenkins as a
  "Secret file" credential.
- LoadBalancer Service: a Kubernetes Service that asks the cloud for a real load
  balancer and floating IP so an app is reachable from outside. Each one draws
  from your quota.
- Ingress controller: one LoadBalancer Service that routes to many internal
  Services, so you spend one load balancer instead of many.
- Floating IP: a routable address from the external pool (`provider-ext`). Counts
  against quota even when unattached.

--------------------------------------------------------------------------------

## 11. Where to go deeper

You can complete this guide without these; open them when you want more than the
workflow above.

- Handover Pack: your delivered identifiers, endpoints, CA bundle, and the full
  three-account model with exact rights.
- Self-Service Guide: running your tenancy day to day (networks, servers, load
  balancers, secrets) through the dashboard or command line.
- CI/Automation Integration Guide: application-credential pipelines in depth --
  credential handling, least-privilege per-pipeline credentials, quota pacing,
  and cleanup discipline.
- Acceptance Checklist: the short set of checks that prove your tenancy works,
  including the cluster lifecycle rows.
- Starter-kit scripts (delivered under scripts/): `smoke-test.sh` (read-only
  token/catalog/quota check for the top of a pipeline), `tenancy-audit.sh`
  (read-only snapshot of what you have), `acceptance-run.sh` (a full
  build-and-teardown you can trim into a recurring smoke test), and
  `ci-cleanup-sweep.sh` (the backstop quota sweep).
- Your tenant AI assistant kit: the Kubernetes, day-2 operations, and
  troubleshooting references, for step-by-step help in context.
- The Glossary (at the end of the Handover Pack): plain-language definitions of
  every term used across your pack, if a component in this guide is unfamiliar.

For anything not covered here, or a change to your tenancy, contact
{{ACCOUNT_CONTACT}}.
