diff --git a/clientdocs/jenkins-kubernetes-guide.md b/clientdocs/jenkins-kubernetes-guide.md index cc0d963..d148100 100644 --- a/clientdocs/jenkins-kubernetes-guide.md +++ b/clientdocs/jenkins-kubernetes-guide.md @@ -74,6 +74,27 @@ done with an application credential, so they do not belong in a pipeline. Create the cluster by hand (section 2); let pipelines DEPLOY into it (section 5). +### Two ways to drive the cloud: dashboard or command line + +Know up front which tool does what, because the two are not interchangeable +across this whole workflow: + +- CLUSTER LIFECYCLE -- creating, resizing, and deleting the cluster, and managing + networks, floating IPs, and load balancers -- can be done EITHER in the web + dashboard OR on the command line. Use whichever you prefer; both are shown. +- EVERYTHING KUBERNETES -- fetching the kubeconfig, and every `kubectl` / `helm` + step, Jenkins wiring, deploys, ingress, storage, and Kubernetes teardown -- is + COMMAND LINE ONLY. The dashboard cannot drive `kubectl`. + +So even if you create your cluster in the dashboard, you cross over to the command +line at the kubeconfig step (section 3) and stay there. There is no dashboard-only +path for the Jenkins workflow: install the OpenStack client, `kubectl`, and `helm` +on your Jenkins agents (section 1) no matter how you create the cluster. + +Throughout the guide, steps that differ by path are marked **On the command line** +and **In the dashboard**. A step with neither marker is command line and applies +to both paths. + -------------------------------------------------------------------------------- ## 1. Before you start @@ -92,6 +113,12 @@ - [ ] Confirmed network reach from Jenkins to the cloud API endpoint (you have tested this). +Installing the three clients: install the OpenStack command-line client (with its +cluster `coe` commands), `kubectl`, and `helm`, each per its own project's install +docs. Keep `kubectl` within one minor version of your cluster's Kubernetes (the +delivered template builds v1.34, so a `kubectl` in the 1.33-1.35 range is safe) -- +a `kubectl` far newer or older than the cluster can misbehave in confusing ways. + Cloud connection settings. Put these in a `clouds.yaml` on the agent (the client reads the current directory, then ~/.config/openstack/, then /etc/openstack/), and select it with `export OS_CLOUD={{TENANT_SHORT_NAME}}`: @@ -196,6 +223,13 @@ capacity allows. A large first request is the second most common cluster failure, after the flannel mistake. +**In the dashboard:** Container Infra > Clusters > Create Cluster. Choose the +`{{TENANT_SHORT_NAME}}-k8s` template, set the keypair to +`{{TENANT_SHORT_NAME}}-key`, keep the node count small (1 master, 1-2 workers), +and leave Public and Hidden UNCHECKED. (Your dashboard URL is in the Handover +Pack.) The same two sizing limits -- quota and free capacity -- apply exactly as +above; the "No valid host" failure looks the same from the dashboard. + ### 2.4 Wait for it, and verify openstack coe cluster show -f value -c status -c status_reason @@ -206,6 +240,22 @@ the usual cause is the network driver: rebuild with calico (see 2.2). Delete the stuck cluster first: `openstack coe cluster delete `. +### 2.5 Scaling the cluster later + +Add or remove workers after the cluster is up. This is cluster lifecycle, so it +runs as `{{TENANT_SHORT_NAME}}-cluster`, the same account and the same free-capacity +limit as the first create (2.3): + + openstack coe cluster resize + +**In the dashboard:** Container Infra > Clusters > (your cluster) > Resize, and set +the new node count. + +Growing the cluster is subject to the same "No valid host" capacity limit as the +first create -- a resize larger than the free capacity fails the same way. When you +shrink, Kubernetes reschedules the removed nodes' pods onto the survivors, so make +sure the remaining workers have room for them. + -------------------------------------------------------------------------------- ## 3. Step 2 -- get and understand the kubeconfig @@ -230,9 +280,22 @@ cluster -- you can read it from the certificate embedded in the kubeconfig, or ask {{ACCOUNT_CONTACT}} -- and set a renewal reminder ahead of it. +Read the expiry date straight from the kubeconfig you just fetched: + + kubectl config view --raw \ + -o jsonpath='{.users[0].user.client-certificate-data}' \ + | base64 -d | openssl x509 -noout -enddate + Never commit the kubeconfig to a repository. It goes into the Jenkins credential store next, and nowhere else. +If you created the cluster in the dashboard: that is fine, but this step and +everything after it -- fetching the kubeconfig, storing it in Jenkins +(section 4), and the pipelines -- run through the OpenStack command-line client +and your application credential, not the dashboard. Fetch the kubeconfig with the +`openstack coe cluster config` command above; if you have not installed the +client yet (section 1), do that first. + -------------------------------------------------------------------------------- ## 4. Step 3 -- store the kubeconfig in Jenkins @@ -264,9 +327,30 @@ step, and in a real pipeline it is a Build stage (and a push stage) that runs before the Deploy stage below. The examples here use the public `nginx:stable` image so you can prove the deploy path without a registry; swap in your own -image once it is built and pushed. If your registry is private, create a -Kubernetes image-pull secret in the cluster and reference it from your -Deployment's `imagePullSecrets`, so the cluster can authenticate to pull it. +image once it is built and pushed. + +Omega Cloud does not provide a container registry. Push your images to a registry +you control -- a public one (Docker Hub, GitHub/GitLab, Quay) or your own private +one -- that the cluster can reach over the internet. It can: that is how the +cluster pulled its own system images at build time. For a PUBLIC image nothing +more is needed. For a PRIVATE registry, create an image-pull secret in the cluster +and reference it from your Deployment: + + kubectl create secret docker-registry regcred \ + --docker-server= --docker-username= \ + --docker-password= --docker-email= + +Then add to the Deployment's pod `spec`: + + spec: + imagePullSecrets: + - name: regcred + containers: + - name: web + image: //: + +Inject the registry password from your Jenkins credential store at run time; never +write it into a manifest in a repository. ### 5.1 A minimal application manifest @@ -352,6 +436,78 @@ Deployment leaves the load balancer and floating IP allocated -- and billed against your quota. +### 5.4 One ingress instead of many load balancers + +Section 5.3 said to prefer a single ingress controller over one LoadBalancer +Service per app. Here is how. Install an ingress controller once; it takes ONE +LoadBalancer Service -- and so one floating IP -- for your whole tenancy: + + helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx + helm install ingress-nginx ingress-nginx/ingress-nginx \ + --namespace ingress-nginx --create-namespace \ + --set controller.service.type=LoadBalancer + +Wait for its external IP (the same load-balancer provisioning wait as any Service): + + kubectl get service -n ingress-nginx ingress-nginx-controller -o wide + +Then give each app an ordinary ClusterIP Service (NOT `type: LoadBalancer`) and +route to it with an Ingress. Every additional app is one more Ingress rule behind +the same single load balancer: + + apiVersion: networking.k8s.io/v1 + kind: Ingress + metadata: + name: sample-app + spec: + ingressClassName: nginx + rules: + - host: app.example.com # a name you point at the ingress IP + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: sample-app + port: { number: 80 } + +The platform does not provide DNS. The ingress (like any LoadBalancer Service) gets +an IP, not a hostname; map your own hostnames to that IP in whatever DNS you run. + +### 5.5 Persistent storage for stateful apps + +The sample app is stateless. A workload that must keep data (a database, a cache +with persistence) needs a PersistentVolume, which comes from a StorageClass. Check +what your cluster offers first: + + kubectl get storageclass + +- If a class is listed and marked `(default)`, a PersistentVolumeClaim like the one + below binds to it automatically. +- If NO StorageClass is listed, your cluster was built without persistent-volume + support (the delivered template does not enable it by default). Stateless + workloads are unaffected; if you need persistent storage, ask {{ACCOUNT_CONTACT}} + to enable it. Do not work around it by writing to node-local disk -- that data is + lost whenever a pod moves to another node. + +A claim, and how a pod uses it: + + apiVersion: v1 + kind: PersistentVolumeClaim + metadata: + name: data + spec: + accessModes: [ ReadWriteOnce ] + resources: { requests: { storage: 5Gi } } + # then in your pod spec: + # volumes: + # - name: data + # persistentVolumeClaim: { claimName: data } + +Persistent volumes are backed by cloud block storage and count against your volume +quota, the same as any disk. + -------------------------------------------------------------------------------- ## 6. Pattern B -- the cluster as a Jenkins build-agent pool @@ -384,6 +540,39 @@ (endpoint above), and the pods must reach Jenkins (step 4). You have already confirmed the first direction; confirm the second when you first enable this. +### 6.1 A scoped ServiceAccount instead of the admin kubeconfig + +The kubeconfig from section 3 carries full cluster-admin. For agents that run day +in, day out, better practice is a Kubernetes ServiceAccount holding only the +rights those jobs need, and to hand Jenkins that account's token instead of the +admin kubeconfig. Run these once, authenticated with the section-3 kubeconfig: + + # a namespace for build workloads, and a service account inside it + kubectl create namespace ci + kubectl create serviceaccount jenkins -n ci + + # grant only what the agents need. 'edit' confined to the 'ci' namespace is a + # reasonable starting point; scope it down with a custom Role if jobs need less. + kubectl create rolebinding jenkins-edit -n ci \ + --clusterrole=edit --serviceaccount=ci:jenkins + +Then get a token for that account -- two options, pick by whether your Jenkins +can refresh it: + +- Short-lived (preferred): `kubectl -n ci create token jenkins --duration=24h` + mints a time-boxed token. A pipeline step or small refresh job requests a new + one before the old expires; nothing long-lived is stored. +- Long-lived (if Jenkins cannot refresh): create a Secret of type + `kubernetes.io/service-account-token` annotated with + `kubernetes.io/service-account.name: jenkins`, then read its `token` field. + This token does not expire on its own -- treat it exactly like the kubeconfig, + and revoke it by deleting the Secret if it ever leaks. + +Store the chosen token in Jenkins as a "Secret text" credential and select it as +the Kubernetes cloud's credential (step 3 above) in place of the kubeconfig. A +namespace-scoped account cannot damage the rest of the cluster if a build is +compromised, which is the whole point of using one. + Pattern B does not replace Pattern A: a common setup is build agents in the cluster (B) that then deploy the built artifact to the same or another cluster (A). @@ -393,7 +582,10 @@ ## 7. Worked example, end to end This runs the whole flow once with a stock image, to prove the path before you -wire in your own app. Sections in parentheses point back to the detail. +wire in your own app. Sections in parentheses point back to the detail. It is +written on the command line end to end; if you prefer the dashboard, create the +cluster (steps 1-2) there and pick up at step 3, which is command line for +everyone. 1. Sign in as `{{TENANT_SHORT_NAME}}-cluster` (2.1) and create a small cluster from the delivered template (2.3): @@ -451,6 +643,12 @@ 3. If you are finished with the cluster itself, delete it as `{{TENANT_SHORT_NAME}}-cluster`: `openstack coe cluster delete `. +**In the dashboard:** you can release floating IPs (Network > Floating IPs), delete +load balancers (Network > Load Balancers), and delete the cluster (Container Infra > +Clusters) the same way. But deleting Kubernetes objects -- Services, Deployments -- +is `kubectl` only, and deleting the Service is what releases its load balancer. So +delete the Service first (step 1) whichever way you remove the rest. + Two hygiene rules that keep quota from silently filling: - A floating IP counts against your quota even when it is not attached to @@ -501,6 +699,14 @@ load balancer is still provisioning (a few minutes is normal), or your load balancer / floating IP quota is exhausted. Check your quota usage. +- A pod is stuck `ImagePullBackOff` or `ErrImagePull` -- the cluster cannot pull + the image. Check the image name and tag first; if it is in a PRIVATE registry, + the cluster needs an image-pull secret referenced from the Deployment (5.0). + +- A PersistentVolumeClaim stays `Pending` -- nothing bound it. Run `kubectl get + storageclass`: if none is listed, the cluster has no persistent-volume support + and the claim will never bind (5.5); contact {{ACCOUNT_CONTACT}} to enable it. + -------------------------------------------------------------------------------- ## 10. Glossary diff --git a/clientdocs/sweep-receipt.txt b/clientdocs/sweep-receipt.txt index 08b49ab..0f82357 100644 --- a/clientdocs/sweep-receipt.txt +++ b/clientdocs/sweep-receipt.txt @@ -15,7 +15,7 @@ 2e394bd459aa59237c4d68afde64e6cd185c17a900dad42c40a974b02b225bc0 clientdocs/ci-integration-guide.md e5df7e25b3c073a1169799990ba480d7d4eb81abe398f323c46a0649adac6991 clientdocs/handover-pack.md 3f08eb458bcbb36172a5ca2785a6ec72a69a4f8cd440fec53ed604f074b1b8c6 clientdocs/intake-form.md -2b6bf741c6415aaf42197b8af44769b0cb86cb015201234f77ca1d50a3445a8a clientdocs/jenkins-kubernetes-guide.md +b0dea4d001d14f9565aae607c21e38745336b66b2bafe8ba99f08fe7f35a5185 clientdocs/jenkins-kubernetes-guide.md 2acd231062e10b33864e3e350ace07a86eac831021825b739ed641f50149e3e3 clientdocs/scripts/Jenkinsfile.example 1955db5d789a2e48d9054e307fbec2678efc1fbafda18103caa389cb896701fd clientdocs/scripts/acceptance-run.sh c2a2ff7ad719cb5866b1b58f5ca2cac930ccc99b96efaf8a3f12301d645727d3 clientdocs/scripts/ci-cleanup-sweep.sh diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 77c5345..55856ac 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -382,10 +382,34 @@ cluster-create time and their live cluster already carries it. 3. Doc-gap: jenkins-kubernetes-guide.md is CLI-centric. Their next guide step (Section 3, kubeconfig fetch via `openstack coe cluster config`) has NO - Horizon equivalent documented -- magnum-ui offers a cluster-config download, - but the guide does not mention it, and the Jenkins integration requires the - CLI/app-cred path regardless. Candidate: a short "if you drive the dashboard" - note in Sections 2-3 pointing GUI users to the CLI for kubeconfig + Jenkins. + Horizon path -- the supported retrieval is the CLI. (Earlier draft of this + bullet claimed magnum-ui "offers a cluster-config download" -- UNVERIFIED, + retracted; do not assert it.) The Jenkins integration requires the + CLI/app-cred path regardless. +- **ADDRESSED: DOCFIX-130 (addendum 48, DRAFTED -- lint/harness green, NOT yet + committed pending operator review).** Grew from 2 gaps to a full guide pass on + operator instruction ("fold and add 1-7... make GUI/CLI paths better defined"). + clientdocs/jenkins-kubernetes-guide.md, additive only, no token touched: + - STRUCTURE: new Section 0 dashboard-vs-CLI selector + **On the command line** / + **In the dashboard** markers; dashboard equivalents for create (2.3), resize + (2.5 new), teardown (8). Rule: lifecycle = either path; everything Kubernetes + (kubeconfig onward) = CLI only. + - GAPS 1-7 closed: registry/imagePullSecret (5.0; VERIFIED no registry service + in catalog), ingress-nginx how-to + no-DNS (5.4 new), cluster resize + (2.5 new; VERIFIED syntax), persistent storage self-verify (5.5 new), cert + expiry read one-liner (3), tooling install + kubectl-skew (1); + ImagePullBackOff + and PVC-Pending troubleshooting rows (9). + - Leak harness CAUGHT two self-inflicted client-doc leaks ("Horizon", "magnum") + on first pass -- reworded, now PASS 8/0; gauntlet 38/38. + - ~/devteam-package NOT re-instantiated (refresh at next package sweep). + See the changelog DOCFIX-130 entry. Numbers: DOCFIX next-free now 131. +- **STORAGE OPERATOR FOLLOW-UP (logged, not actioned):** delivered cluster + template has volume_driver unset + no docker volume (MEASURED). Whether the + built cluster gets a default StorageClass from the cinder-CSI default is + UNVERIFIED (verifying = kubectl into devteam's live cluster; deferred for tenant + isolation). If none, persistent volumes are unavailable out of the box -- + candidate template improvement (would need a D-NNN). 5.5 is written self-verifying + so the doc is correct either way. ## REMAINING OPEN WORK (single list for session resume; 2026-07-07, updated post-devteam-window) diff --git a/docs/v1-redeploy-changelog.md b/docs/v1-redeploy-changelog.md index eb301e9..5089629 100644 --- a/docs/v1-redeploy-changelog.md +++ b/docs/v1-redeploy-changelog.md @@ -3193,4 +3193,67 @@ - Numbers consumed: DOCFIX-129. REVERT: git checkout HEAD~ -- clientdocs/tenant-skill clientdocs/sweep-receipt.txt docs/v1-redeploy-changelog.md docs/session-ledger.md; then bash scripts/repo-lint.sh --record-clientdocs-sweep; re-run the instantiation helper to rebuild the package. -Next-free after this push (per scan, keep this line unwrapped): D-076, DOCFIX-130, BUNDLEFIX-012. +### 2026-07-08 (addendum 48, jumphost stream) -- DOCFIX-130: jenkins-kubernetes-guide.md post-handover workflow gaps (k8s ServiceAccount how-to + dashboard-user CLI pointer) + +Reviewing the guide end to end while supporting devteam's first self-service +cluster surfaced a cluster of gaps a real Jenkins-to-Kubernetes team hits, plus a +structural weakness: the guide was CLI-first with ad-hoc dashboard asides, but +devteam drives the dashboard. All changes are ADDITIVE to +clientdocs/jenkins-kubernetes-guide.md -- no existing text removed, no client value +or {{token}} touched, no new token introduced. + +STRUCTURE -- dashboard vs command line made explicit: +- New Section 0 selector "Two ways to drive the cloud": cluster lifecycle + (create/resize/delete, networks, FIPs, LBs) can be dashboard OR CLI; everything + Kubernetes (kubeconfig, kubectl/helm, Jenkins, deploy, ingress, storage, k8s + teardown) is CLI-ONLY -- so a GUI user crosses to the CLI at Section 3 and stays. + Convention set: differing steps are marked **On the command line** / + **In the dashboard**; unmarked steps are CLI and apply to both. +- Dashboard equivalents added where lifecycle is GUI-able: Section 2.3 create + (template + keypair + small size + Public/Hidden unchecked -- the path devteam + actually used), Section 2.5 resize, Section 8 teardown (release FIP/LB/cluster in + the dashboard; k8s-object delete stays kubectl-only, Service first). + +GAP CLOSURES (the 7 reviewed with the operator): +1. Registry (5.0): stated plainly the platform provides NO container registry + (VERIFIED -- catalog has only glance + product-streams, no registry service); + bring your own; public pulls work (cluster pulled its own system images); added + the private-registry `kubectl create secret docker-registry` + `imagePullSecrets` + procedure the guide previously only named. +2. Ingress (5.4 new): the guide urged "one ingress instead of N LoadBalancers" with + no how-to; added the ingress-nginx helm install (one LB) + an Ingress manifest, + and folded in the no-platform-DNS point (map your own hostnames to the ingress IP). +3. Scaling (2.5 new): `openstack coe cluster resize ` (VERIFIED + syntax via --help) + dashboard Resize; same capacity limit as first create noted. +4. Persistent storage (5.5 new): SELF-VERIFYING -- `kubectl get storageclass`; if a + (default) class exists a PVC binds, if none the cluster has no PV support and to + contact us. Deliberately does NOT assert a StorageClass (delivered template has + volume_driver unset -- MEASURED; whether the cinder-CSI default yields a class is + UNVERIFIED, see operator follow-up). +5. Cert expiry (3): the guide flagged expiry but not how to see it; added a + `kubectl config view --raw | base64 -d | openssl x509 -noout -enddate` one-liner. +6. No DNS (5.4): folded into ingress -- LoadBalancer/ingress give an IP, not a name. +7. Tooling (1): install pointer for the OpenStack client (coe commands) + kubectl + + helm, and a kubectl-vs-1.34 version-skew caveat. +- Troubleshooting (9): added ImagePullBackOff/ErrImagePull (registry) and PVC + Pending (no StorageClass) symptom rows. + +- Leak harness caught two self-inflicted leaks on the first pass (the literal terms + "Horizon" and "magnum"/python-magnumclient in the selector + tooling text) -- both + reworded to "the dashboard" / "the OpenStack client (coe commands)"; the harness + did its job. FINAL: leak harness PASS 8/0, gauntlet ALL GREEN (38), repo-lint + 0 fail (1 legacy WARN), L7 receipt re-recorded (30 files). +- No D-NNN governs client-guide content (DOCFIX-class, cf. DOCFIX-123). Read-only + cloud audit only (catalog, template fields, resize --help); no mutation. +- OPERATOR FOLLOW-UP (logged, not actioned): the delivered cluster template has + volume_driver unset and no docker volume; confirm whether the built cluster gets a + default StorageClass from the cinder-CSI default. If NOT, persistent volumes are + unavailable out of the box -- candidate template improvement (needs a D-NNN / + future DOCFIX). Verifying live means kubectl into devteam's running cluster + (tenant isolation -- deferred to operator). +- DELIVERY NOTE: template edit only -- ~/devteam-package NOT re-instantiated; refresh + it at the next package sweep so devteam's copy carries the new sections. +- Numbers consumed: DOCFIX-130. +REVERT: git checkout HEAD~ -- clientdocs/jenkins-kubernetes-guide.md clientdocs/sweep-receipt.txt docs/v1-redeploy-changelog.md docs/session-ledger.md; then bash scripts/repo-lint.sh --record-clientdocs-sweep. + +Next-free after this push (per scan, keep this line unwrapped): D-076, DOCFIX-131, BUNDLEFIX-012.