# Omega Cloud -- Jenkins + Kubernetes Implementation Guide (TEMPLATE) TEMPLATE NOTE (removed before delivery): fields written as {{THIS}} are filled in by us per client; fields written as are yours to fill on your side (from the credential files delivered to your custodians, or from your own Jenkins). Secrets are never written into this document, a repository, or a pipeline file -- they are injected at run time from your Jenkins credential store. This is the one document you need to take a Jenkins server that already reaches Omega Cloud and get it building and deploying to a Kubernetes cluster running in your tenancy. It is written as a single top-to-bottom workflow so you do not have to jump between documents to complete a task. Where a topic has a deeper reference, it is named at the end (section 11); you can finish the whole flow without opening any of them. Two integration patterns are covered, and most teams use both: - Pattern A (the main flow, sections 5 and 7): your Jenkins pipeline DEPLOYS an application into the cluster, using the cluster's kubeconfig as a stored credential. Jenkins stays where it is; only your app runs in the cluster. - Pattern B (section 6): your Jenkins uses the cluster as an elastic pool of BUILD AGENTS -- each job runs in a short-lived pod inside the cluster. -------------------------------------------------------------------------------- ## 0. The picture: what you are connecting +-------------------+ identity + API +---------------------+ | Jenkins server | ---------------------------> | Omega Cloud | | (on your network)| TLS, application credential | (your tenancy) | +-------------------+ +---------------------+ | | | kubeconfig (a stored Jenkins credential) | creates | talks to the cluster's API over its floating IP v | +---------------------+ +------------------------------------------> | Kubernetes cluster | kubectl / helm / build-agent pods | (control + workers)| +---------------------+ | LoadBalancer Service gets a floating IP -> your app is reachable from outside. What each piece does: - Jenkins server: yours, on your network. It already reaches the cloud (you have tested this). It holds two kinds of secret in its credential store: an application credential (for talking to the cloud API) and a kubeconfig (for talking to the cluster). Neither is ever written into a repository. - The cloud API + identity service: authenticates your automation and lets you create the cluster, networks, load balancers, and floating IPs -- all within your quota. - The Kubernetes cluster: a managed set of virtual machines (one or more control nodes plus worker nodes) running in your project. You create it once; your pipelines then deploy into it many times. - A LoadBalancer Service: when your app asks Kubernetes for a load balancer, the cloud gives the cluster a real load balancer and a floating IP, so the app is reachable from outside the cluster. Each one draws from your quota. Three accounts, and which one to use here (full detail in the Handover Pack): | Account | Use it for | |----------------------------------|---------------------------------------------| | `{{TENANT_SHORT_NAME}}-svc` | All day-to-day automation via its | | | application credential: networks, servers, | | | load balancers, deploying into the cluster. | | `{{TENANT_SHORT_NAME}}-cluster` | Creating and deleting Kubernetes clusters, | | | with its PASSWORD. Nothing else. | | `{{TENANT_SHORT_NAME}}-domain-admin` | Managing your own users. Identity only. | The one rule that prevents the most common failure: cluster CREATE and DELETE run only as `{{TENANT_SHORT_NAME}}-cluster` with its password. They cannot be done with an application credential, so they do not belong in a pipeline. Create the cluster by hand (section 2); let pipelines DEPLOY into it (section 5). -------------------------------------------------------------------------------- ## 1. Before you start A checklist. If any item is missing, stop here and get it before continuing. - [ ] The credential files for `{{TENANT_SHORT_NAME}}-svc`, `{{TENANT_SHORT_NAME}}-cluster`, and `{{TENANT_SHORT_NAME}}-domain-admin`, delivered to your custodians. - [ ] The delivered CA bundle file ({{CA_BUNDLE_FILE}}). You point every client at it; you never disable certificate verification instead. - [ ] The OpenStack command-line client on your Jenkins agents (or in the agent container image), plus `kubectl` and, if you use it, `helm`. - [ ] Jenkins plugins: "Credentials Binding" and "Kubernetes CLI" for Pattern A; add "Kubernetes" (the cloud/agent plugin) for Pattern B. - [ ] Confirmed network reach from Jenkins to the cloud API endpoint (you have tested this). Cloud connection settings. Put these in a `clouds.yaml` on the agent (the client reads the current directory, then ~/.config/openstack/, then /etc/openstack/), and select it with `export OS_CLOUD={{TENANT_SHORT_NAME}}`: clouds: {{TENANT_SHORT_NAME}}: auth_type: v3applicationcredential auth: auth_url: {{AUTH_URL}} application_credential_id: application_credential_secret: region_name: {{REGION}} identity_api_version: 3 cacert: The application credential id and secret come from the delivered credential file. The secret is injected at run time from your Jenkins credential store, not written to disk in a repository. (Full CI credential handling: the CI/Automation Integration Guide.) -------------------------------------------------------------------------------- ## 2. Step 1 -- create your Kubernetes cluster You do this once, by hand, signed in as `{{TENANT_SHORT_NAME}}-cluster`. It is not a pipeline step (see the rule in section 0). ### 2.1 Sign in as the cluster account Prompt for the password; never read it from a file into a script, never store it in Jenkins: export OS_AUTH_TYPE=password export OS_AUTH_URL={{AUTH_URL}} export OS_IDENTITY_API_VERSION=3 export OS_CACERT= export OS_USERNAME={{TENANT_SHORT_NAME}}-cluster export OS_USER_DOMAIN_NAME={{TENANT_SHORT_NAME}} export OS_PROJECT_NAME={{TENANT_SHORT_NAME}}-prod export OS_PROJECT_DOMAIN_NAME={{TENANT_SHORT_NAME}} read -rs -p "password for {{TENANT_SHORT_NAME}}-cluster: " OS_PASSWORD && export OS_PASSWORD openstack token issue # smoke test: should print a token ### 2.2 Choose or build a cluster template If a template named `{{TENANT_SHORT_NAME}}-k8s` was delivered with your tenancy, use it and skip to 2.3: openstack coe cluster template show {{TENANT_SHORT_NAME}}-k8s If you need to build your own, three rules keep the create from failing: - Use `--network-driver calico`. This platform's cluster service supports calico. Do NOT use flannel: a flannel cluster never finishes coming up -- the create sits in progress and eventually fails, with the nodes never going ready. This is the single most common self-inflicted cluster failure. - Do NOT set the Public or Hidden flag (the two checkboxes in the dashboard's Create Cluster Template dialog, or `--public` / `--hidden` on the command line). Publishing a template to the whole cloud is an operator-only action; setting either flag makes the entire create fail with, on the command line, "Not authorized to set public or hidden flag for cluster template", and in the dashboard, only the generic "Error: Unable to create cluster template." Templates are already usable by everyone in your project; you lose nothing by leaving these unchecked. - Do NOT bake a keypair into the template. The SSH keypair is owned by the `{{TENANT_SHORT_NAME}}-cluster` account and is supplied at cluster-create time, not in the template. A known-good template, mirroring the platform default: openstack coe cluster template create \ --coe kubernetes \ --image ubuntu-jammy-kube-v1.34.8 \ --external-network provider-ext \ --master-flavor --flavor \ --network-driver calico --docker-storage-driver overlay2 \ --master-lb-enabled --floating-ip-enabled \ --fixed-network {{TENANT_SHORT_NAME}}-net \ --fixed-subnet {{TENANT_SHORT_NAME}}-subnet Pick flavors from `openstack flavor list`. The delivered template's choices are a safe default; read them with the `template show` command above. ### 2.3 Create the cluster, sized to fit openstack coe cluster create \ --cluster-template {{TENANT_SHORT_NAME}}-k8s \ --keypair {{TENANT_SHORT_NAME}}-key \ --master-count 1 --node-count Size `` against TWO limits, not one: - Your quota (instances, cores, memory). The dashboard and CLI show your usage. - The physical capacity free on the cloud right now. A request that is inside your quota can still fail if no host has room for another node of that size. The symptom is a node stuck in ERROR with "No valid host was found. There are not enough hosts available." It is not a quota problem and raising quota will not fix it. Practical guidance: start SMALL -- 1 control node and 1 or 2 workers -- confirm the cluster comes up, then scale the worker count up if you need more and capacity allows. A large first request is the second most common cluster failure, after the flannel mistake. ### 2.4 Wait for it, and verify openstack coe cluster show -f value -c status -c status_reason # poll no faster than every 10 seconds; done when status = CREATE_COMPLETE Cluster builds take tens of minutes -- this is normal. But if a cluster sits in CREATE_IN_PROGRESS far longer than the build should take and never completes, the usual cause is the network driver: rebuild with calico (see 2.2). Delete the stuck cluster first: `openstack coe cluster delete `. -------------------------------------------------------------------------------- ## 3. Step 2 -- get and understand the kubeconfig The kubeconfig is the file `kubectl` and Jenkins use to talk to your cluster's Kubernetes API. Fetch it, still signed in as `{{TENANT_SHORT_NAME}}-cluster`: umask 077 openstack coe cluster config --dir export KUBECONFIG=/config kubectl get nodes # every node should report Ready What is inside it, and why it matters: - It points at your cluster's API server at its floating IP, on port 6443. - It authenticates with a CLIENT CERTIFICATE embedded in the file (plus the cluster's CA). There is no username or password in it. Anyone holding this file has administrative access to your cluster -- treat it exactly like a password. - The certificate is time-limited. When it expires, `kubectl` and your pipelines start failing to authenticate; re-fetch the kubeconfig with the same command and update the stored credential (section 4). Confirm the expiry date for your cluster -- you can read it from the certificate embedded in the kubeconfig, or ask {{ACCOUNT_CONTACT}} -- and set a renewal reminder ahead of it. Never commit the kubeconfig to a repository. It goes into the Jenkins credential store next, and nowhere else. -------------------------------------------------------------------------------- ## 4. Step 3 -- store the kubeconfig in Jenkins In Jenkins: Manage Jenkins > Credentials > (your domain) > Add Credentials. - Kind: "Secret file". - File: upload the `config` file you fetched in section 3. - ID: `omega-k8s-kubeconfig` (used by the pipeline examples below). - Scope: the narrowest that your jobs need. That is the only place the kubeconfig lives inside Jenkins. Pipelines reference it by ID; the "Kubernetes CLI" plugin loads it for the duration of a block and masks it in the log. When you re-fetch the kubeconfig (expiry, or a rebuilt cluster), update this one credential and every pipeline picks up the change. -------------------------------------------------------------------------------- ## 5. Step 4, Pattern A -- deploy your app into the cluster This is the main flow: a pipeline that rolls an application out to the cluster using the stored kubeconfig. Nothing here needs the cluster password; deploying into an existing cluster is ordinary automation. ### 5.0 Build and publish your image first Deploying pulls a container image; it does not build one. Building your app into an image and pushing it to a registry the cluster can pull from is an upstream step, and in a real pipeline it is a Build stage (and a push stage) that runs before the Deploy stage below. The examples here use the public `nginx:stable` image so you can prove the deploy path without a registry; swap in your own image once it is built and pushed. If your registry is private, create a Kubernetes image-pull secret in the cluster and reference it from your Deployment's `imagePullSecrets`, so the cluster can authenticate to pull it. ### 5.1 A minimal application manifest Keep your Kubernetes manifests in your application repository, for example in a `k8s/` directory. A minimal Deployment plus a LoadBalancer Service: # k8s/app.yaml apiVersion: apps/v1 kind: Deployment metadata: name: sample-app spec: replicas: 2 selector: matchLabels: { app: sample-app } template: metadata: labels: { app: sample-app } spec: containers: - name: web image: nginx:stable ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: sample-app spec: type: LoadBalancer selector: { app: sample-app } ports: - port: 80 targetPort: 80 When Kubernetes sees `type: LoadBalancer`, the cloud gives the cluster a load balancer and a floating IP for this Service. That address is how the outside world reaches your app. ### 5.2 The pipeline // Jenkinsfile -- Pattern A: deploy into the cluster pipeline { agent any options { timestamps(); timeout(time: 30, unit: 'MINUTES') } stages { stage('Deploy') { steps { // The Kubernetes CLI plugin loads the stored kubeconfig for this // block and masks it in the log. withKubeConfig([credentialsId: 'omega-k8s-kubeconfig']) { // Applying the whole k8s/ directory keeps deploy and teardown // symmetric: `kubectl delete -f k8s/` later removes everything // this created, including the Service (and its load balancer). sh 'kubectl apply -f k8s/' sh 'kubectl rollout status deployment/sample-app --timeout=5m' } } } stage('Show address') { steps { withKubeConfig([credentialsId: 'omega-k8s-kubeconfig']) { // The external IP appears once the load balancer finishes // provisioning (a few minutes on the first apply). sh 'kubectl get service sample-app -o wide' } } } } } ### 5.3 Load balancers and your quota Every Service of `type: LoadBalancer` consumes one load balancer AND one floating IP from your quota, and keeps them for as long as the Service exists. Two consequences: - Do not create a LoadBalancer Service per microservice. Prefer a single ingress controller (itself one LoadBalancer Service) that routes to many internal Services by hostname or path. That turns N load balancers into one. - When you tear an app down, delete its Service (section 8). Deleting only the Deployment leaves the load balancer and floating IP allocated -- and billed against your quota. -------------------------------------------------------------------------------- ## 6. Pattern B -- the cluster as a Jenkins build-agent pool Instead of (or in addition to) deploying INTO the cluster, you can have Jenkins run its BUILD jobs as short-lived pods inside the cluster. Each job gets a fresh, isolated agent that disappears when the build ends. Setup, using the "Kubernetes" plugin: 1. Manage Jenkins > Clouds > Add a new cloud > Kubernetes. 2. Kubernetes URL: your cluster's API endpoint, `https://:6443` (the `server:` line in the kubeconfig from section 3). 3. Credentials: add the kubeconfig (or a Kubernetes service-account token scoped to a build namespace) as the cloud's credential. A dedicated service account with a limited role is better practice than the full-admin kubeconfig for day-to-day agents. 4. Jenkins tunnel / URL: pods need to reach your Jenkins to register as agents, so set the Jenkins URL to an address the cluster can reach. 5. Define a Pod Template: a `jnlp` agent container plus any tool containers your builds need (for example a container with `kubectl` and `helm`). Give it a label; jobs that request that label run in these pods. Two things to keep in mind: - These agent pods run on your cluster's worker nodes and consume their CPU and memory. Size the cluster (section 2.3) for your build concurrency, or set pod resource requests so builds queue rather than overcommit. - Network reachability goes both ways: Jenkins must reach the cluster API (endpoint above), and the pods must reach Jenkins (step 4). You have already confirmed the first direction; confirm the second when you first enable this. Pattern B does not replace Pattern A: a common setup is build agents in the cluster (B) that then deploy the built artifact to the same or another cluster (A). -------------------------------------------------------------------------------- ## 7. Worked example, end to end This runs the whole flow once with a stock image, to prove the path before you wire in your own app. Sections in parentheses point back to the detail. 1. Sign in as `{{TENANT_SHORT_NAME}}-cluster` (2.1) and create a small cluster from the delivered template (2.3): openstack coe cluster create demo-k8s \ --cluster-template {{TENANT_SHORT_NAME}}-k8s \ --keypair {{TENANT_SHORT_NAME}}-key \ --master-count 1 --node-count 1 2. Wait for CREATE_COMPLETE (2.4): openstack coe cluster show demo-k8s -f value -c status 3. Fetch and check the kubeconfig (3): umask 077 openstack coe cluster config demo-k8s --dir ./demo-kube export KUBECONFIG=$PWD/demo-kube/config kubectl get nodes 4. Store `./demo-kube/config` in Jenkins as the "Secret file" credential `omega-k8s-kubeconfig` (4). (For a quick local proof you can skip Jenkins and use the `KUBECONFIG` export above directly.) 5. Deploy the sample app (5.1): kubectl apply -f k8s/ kubectl rollout status deployment/sample-app --timeout=5m 6. Read the external address (wait for it to move from to an IP; the load balancer takes a few minutes on first creation): kubectl get service sample-app -o wide # then, from a host that can reach it: curl http:/// 7. Tear the example down (8): kubectl delete -f k8s/ # deletes the Service too -> releases # the load balancer and floating IP openstack coe cluster delete demo-k8s -------------------------------------------------------------------------------- ## 8. Teardown and quota hygiene Delete in reverse order of creation, and delete the Service before anything else, so the load balancer and floating IP are released: 1. `kubectl delete -f k8s/` (or `kubectl delete service `). This is what releases the load balancer and its floating IP. Deleting the same directory you applied catches every manifest, so no Service is left stranded. 2. Delete other cloud resources your build created (servers, volumes, extra floating IPs), newest first. 3. If you are finished with the cluster itself, delete it as `{{TENANT_SHORT_NAME}}-cluster`: `openstack coe cluster delete `. Two hygiene rules that keep quota from silently filling: - A floating IP counts against your quota even when it is not attached to anything. Release IPs you are not using. - Make cleanup automatic. In a Jenkins pipeline, put teardown of per-build resources in a `post { always { ... } }` block so a failed build cannot leak quota. The delivered starter-kit sweep script (`ci-cleanup-sweep.sh`) removes leftovers by name prefix and age; run it as a scheduled job as a backstop. -------------------------------------------------------------------------------- ## 9. Troubleshooting Symptoms you are most likely to hit, and the fix: - "Missing value auth-url required for auth plugin password" -- the shell has no cloud environment loaded. You started a fresh shell or a fresh agent. Re-source your `OS_*` environment or select your `clouds.yaml` entry (`export OS_CLOUD={{TENANT_SHORT_NAME}}`) before the command. - Cluster TEMPLATE create fails; dashboard shows only "Error: Unable to create cluster template." -- the Public or Hidden flag is set. Uncheck both (2.2). - Cluster create fails, a node in ERROR with "No valid host was found. There are not enough hosts available." -- the request is too big for the free capacity right now, even if it is within quota. Use a smaller node count or a smaller flavor and retry (2.3). Raising quota will not help. - Cluster stays in CREATE_IN_PROGRESS far too long and never completes -- most often the network driver. Delete it and rebuild the template with calico; flannel is not supported (2.2). - "Keypair not found" at cluster create -- you are signed in as the wrong account. Cluster create runs as `{{TENANT_SHORT_NAME}}-cluster`, which owns the keypair. Check `echo $OS_USERNAME` first. - A pipeline tries to create a cluster and is refused -- cluster create and delete cannot use an application credential; they need the `{{TENANT_SHORT_NAME}}-cluster` password, so they are not pipeline steps. Create the cluster by hand (section 2); let the pipeline deploy into it. - `kubectl` reports "Unable to connect" or a certificate error -- the kubeconfig is wrong or its certificate has expired. Re-fetch it (3) and update the stored Jenkins credential (4). Make sure you kept the delivered CA and did not disable verification. - A LoadBalancer Service shows EXTERNAL-IP `` for a long time -- the load balancer is still provisioning (a few minutes is normal), or your load balancer / floating IP quota is exhausted. Check your quota usage. -------------------------------------------------------------------------------- ## 10. Glossary - Application credential: a scoped, revocable secret for the `{{TENANT_SHORT_NAME}}-svc` account. All automation authenticates with it. It cannot create clusters. - The cluster account (`{{TENANT_SHORT_NAME}}-cluster`): the only account that creates and deletes clusters, using its password. It owns the cluster keypair. - Cluster template: the recipe (image, flavors, network driver, options) a cluster is built from. Use calico; never set Public/Hidden. - kubeconfig: the certificate-bearing file that grants admin access to a cluster's Kubernetes API. Treat it as a password; store it in Jenkins as a "Secret file" credential. - LoadBalancer Service: a Kubernetes Service that asks the cloud for a real load balancer and floating IP so an app is reachable from outside. Each one draws from your quota. - Ingress controller: one LoadBalancer Service that routes to many internal Services, so you spend one load balancer instead of many. - Floating IP: a routable address from the external pool (`provider-ext`). Counts against quota even when unattached. -------------------------------------------------------------------------------- ## 11. Where to go deeper You can complete this guide without these; open them when you want more than the workflow above. - Handover Pack: your delivered identifiers, endpoints, CA bundle, and the full three-account model with exact rights. - Self-Service Guide: running your tenancy day to day (networks, servers, load balancers, secrets) through the dashboard or command line. - CI/Automation Integration Guide: application-credential pipelines in depth -- credential handling, least-privilege per-pipeline credentials, quota pacing, and cleanup discipline. - Acceptance Checklist: the short set of checks that prove your tenancy works, including the cluster lifecycle rows. - Starter-kit scripts (delivered under scripts/): `smoke-test.sh` (read-only token/catalog/quota check for the top of a pipeline), `tenancy-audit.sh` (read-only snapshot of what you have), `acceptance-run.sh` (a full build-and-teardown you can trim into a recurring smoke test), and `ci-cleanup-sweep.sh` (the backstop quota sweep). - Your tenant AI assistant kit: the Kubernetes, day-2 operations, and troubleshooting references, for step-by-step help in context. For anything not covered here, or a change to your tenancy, contact {{ACCOUNT_CONTACT}}.