diff --git a/clientdocs/README.md b/clientdocs/README.md index 6a30bb0..e24da15 100644 --- a/clientdocs/README.md +++ b/clientdocs/README.md @@ -24,6 +24,10 @@ TEMPLATE NOTE banner). - `ci-integration-guide.md` -- with the handover pack for any client that connects CI/automation (Jenkins etc.); same instantiation rule. + - `jenkins-kubernetes-guide.md` -- with the handover pack for any client + running a Jenkins-to-Kubernetes workflow; same instantiation rule. It is + the single end-to-end flow (cluster create -> kubeconfig -> deploy) and + cross-links the CI guide and self-service guide for detail. - `scripts/` (the tenant starter kit) -- WITH the handover pack: `tenancy-audit.sh`, `smoke-test.sh`, `acceptance-run.sh`, and `ci-cleanup-sweep.sh` go to every client; `clouds.yaml.template` and @@ -68,7 +72,8 @@ cloud access). Contents: `intake-form.md`, `welcome.md`, `self-service-guide.md`, -`handover-pack.md`, `ci-integration-guide.md`, `acceptance-checklist.md`, +`handover-pack.md`, `ci-integration-guide.md`, `jenkins-kubernetes-guide.md`, +`acceptance-checklist.md`, `ai-assistant-guide.md`, `tenant-skill/` (SKILL.md + references/), `scripts/` (starter kit: `tenancy-audit.sh`, `smoke-test.sh`, `acceptance-run.sh`, `ci-cleanup-sweep.sh`, `clouds.yaml.template`, diff --git a/clientdocs/jenkins-kubernetes-guide.md b/clientdocs/jenkins-kubernetes-guide.md new file mode 100644 index 0000000..6127c09 --- /dev/null +++ b/clientdocs/jenkins-kubernetes-guide.md @@ -0,0 +1,536 @@ +# Omega Cloud -- Jenkins + Kubernetes Implementation Guide (TEMPLATE) + +TEMPLATE NOTE (removed before delivery): fields written as {{THIS}} are filled +in by us per client; fields written as are yours to fill on your side +(from the credential files delivered to your custodians, or from your own +Jenkins). Secrets are never written into this document, a repository, or a +pipeline file -- they are injected at run time from your Jenkins credential +store. + +This is the one document you need to take a Jenkins server that already reaches +Omega Cloud and get it building and deploying to a Kubernetes cluster running in +your tenancy. It is written as a single top-to-bottom workflow so you do not +have to jump between documents to complete a task. Where a topic has a deeper +reference, it is named at the end (section 11); you can finish the whole flow +without opening any of them. + +Two integration patterns are covered, and most teams use both: + +- Pattern A (the main flow, sections 5 and 7): your Jenkins pipeline DEPLOYS an + application into the cluster, using the cluster's kubeconfig as a stored + credential. Jenkins stays where it is; only your app runs in the cluster. +- Pattern B (section 6): your Jenkins uses the cluster as an elastic pool of + BUILD AGENTS -- each job runs in a short-lived pod inside the cluster. + +-------------------------------------------------------------------------------- + +## 0. The picture: what you are connecting + + +-------------------+ identity + API +---------------------+ + | Jenkins server | ---------------------------> | Omega Cloud | + | (on your network)| TLS, application credential | (your tenancy) | + +-------------------+ +---------------------+ + | | + | kubeconfig (a stored Jenkins credential) | creates + | talks to the cluster's API over its floating IP v + | +---------------------+ + +------------------------------------------> | Kubernetes cluster | + kubectl / helm / build-agent pods | (control + workers)| + +---------------------+ + | + LoadBalancer Service gets a + floating IP -> your app is + reachable from outside. + +What each piece does: + +- Jenkins server: yours, on your network. It already reaches the cloud (you have + tested this). It holds two kinds of secret in its credential store: an + application credential (for talking to the cloud API) and a kubeconfig (for + talking to the cluster). Neither is ever written into a repository. +- The cloud API + identity service: authenticates your automation and lets you + create the cluster, networks, load balancers, and floating IPs -- all within + your quota. +- The Kubernetes cluster: a managed set of virtual machines (one or more control + nodes plus worker nodes) running in your project. You create it once; your + pipelines then deploy into it many times. +- A LoadBalancer Service: when your app asks Kubernetes for a load balancer, the + cloud gives the cluster a real load balancer and a floating IP, so the app is + reachable from outside the cluster. Each one draws from your quota. + +Three accounts, and which one to use here (full detail in the Handover Pack): + +| Account | Use it for | +|----------------------------------|---------------------------------------------| +| `{{TENANT_SHORT_NAME}}-svc` | All day-to-day automation via its | +| | application credential: networks, servers, | +| | load balancers, deploying into the cluster. | +| `{{TENANT_SHORT_NAME}}-cluster` | Creating and deleting Kubernetes clusters, | +| | with its PASSWORD. Nothing else. | +| `{{TENANT_SHORT_NAME}}-domain-admin` | Managing your own users. Identity only. | + +The one rule that prevents the most common failure: cluster CREATE and DELETE +run only as `{{TENANT_SHORT_NAME}}-cluster` with its password. They cannot be +done with an application credential, so they do not belong in a pipeline. Create +the cluster by hand (section 2); let pipelines DEPLOY into it (section 5). + +-------------------------------------------------------------------------------- + +## 1. Before you start + +A checklist. If any item is missing, stop here and get it before continuing. + +- [ ] The credential files for `{{TENANT_SHORT_NAME}}-svc`, + `{{TENANT_SHORT_NAME}}-cluster`, and `{{TENANT_SHORT_NAME}}-domain-admin`, + delivered to your custodians. +- [ ] The delivered CA bundle file ({{CA_BUNDLE_FILE}}). You point every client + at it; you never disable certificate verification instead. +- [ ] The OpenStack command-line client on your Jenkins agents (or in the agent + container image), plus `kubectl` and, if you use it, `helm`. +- [ ] Jenkins plugins: "Credentials Binding" and "Kubernetes CLI" for Pattern A; + add "Kubernetes" (the cloud/agent plugin) for Pattern B. +- [ ] Confirmed network reach from Jenkins to the cloud API endpoint (you have + tested this). + +Cloud connection settings. Put these in a `clouds.yaml` on the agent (the client +reads the current directory, then ~/.config/openstack/, then /etc/openstack/), +and select it with `export OS_CLOUD={{TENANT_SHORT_NAME}}`: + + clouds: + {{TENANT_SHORT_NAME}}: + auth_type: v3applicationcredential + auth: + auth_url: {{AUTH_URL}} + application_credential_id: + application_credential_secret: + region_name: {{REGION}} + identity_api_version: 3 + cacert: + +The application credential id and secret come from the delivered credential +file. The secret is injected at run time from your Jenkins credential store, not +written to disk in a repository. (Full CI credential handling: the CI/Automation +Integration Guide.) + +-------------------------------------------------------------------------------- + +## 2. Step 1 -- create your Kubernetes cluster + +You do this once, by hand, signed in as `{{TENANT_SHORT_NAME}}-cluster`. It is +not a pipeline step (see the rule in section 0). + +### 2.1 Sign in as the cluster account + +Prompt for the password; never read it from a file into a script, never store +it in Jenkins: + + export OS_AUTH_TYPE=password + export OS_AUTH_URL={{AUTH_URL}} + export OS_IDENTITY_API_VERSION=3 + export OS_CACERT= + export OS_USERNAME={{TENANT_SHORT_NAME}}-cluster + export OS_USER_DOMAIN_NAME={{TENANT_SHORT_NAME}} + export OS_PROJECT_NAME={{TENANT_SHORT_NAME}}-prod + export OS_PROJECT_DOMAIN_NAME={{TENANT_SHORT_NAME}} + read -rs -p "password for {{TENANT_SHORT_NAME}}-cluster: " OS_PASSWORD && export OS_PASSWORD + openstack token issue # smoke test: should print a token + +### 2.2 Choose or build a cluster template + +If a template named `{{TENANT_SHORT_NAME}}-k8s` was delivered with your tenancy, +use it and skip to 2.3: + + openstack coe cluster template show {{TENANT_SHORT_NAME}}-k8s + +If you need to build your own, three rules keep the create from failing: + +- Use `--network-driver calico`. This platform's cluster service supports + calico. Do NOT use flannel: a flannel cluster never finishes coming up -- the + create sits in progress and eventually fails, with the nodes never going + ready. This is the single most common self-inflicted cluster failure. +- Do NOT set the Public or Hidden flag (the two checkboxes in the dashboard's + Create Cluster Template dialog, or `--public` / `--hidden` on the command + line). Publishing a template to the whole cloud is an operator-only action; + setting either flag makes the entire create fail with, on the command line, + "Not authorized to set public or hidden flag for cluster template", and in the + dashboard, only the generic "Error: Unable to create cluster template." + Templates are already usable by everyone in your project; you lose nothing by + leaving these unchecked. +- Do NOT bake a keypair into the template. The SSH keypair is owned by the + `{{TENANT_SHORT_NAME}}-cluster` account and is supplied at cluster-create time, + not in the template. + +A known-good template, mirroring the platform default: + + openstack coe cluster template create \ + --coe kubernetes \ + --image ubuntu-jammy-kube-v1.34.8 \ + --external-network provider-ext \ + --master-flavor gp.mid --flavor \ + --network-driver calico --docker-storage-driver overlay2 \ + --master-lb-enabled --floating-ip-enabled \ + --fixed-network {{TENANT_SHORT_NAME}}-net \ + --fixed-subnet {{TENANT_SHORT_NAME}}-subnet + +Pick flavors from `openstack flavor list`. The delivered template's choices are +a safe default; read them with the `template show` command above. + +### 2.3 Create the cluster, sized to fit + + openstack coe cluster create \ + --cluster-template {{TENANT_SHORT_NAME}}-k8s \ + --keypair {{TENANT_SHORT_NAME}}-key \ + --master-count 1 --node-count + +Size `` against TWO limits, not one: + +- Your quota (instances, cores, memory). The dashboard and CLI show your usage. +- The physical capacity free on the cloud right now. A request that is inside + your quota can still fail if no host has room for another node of that size. + The symptom is a node stuck in ERROR with "No valid host was found. There are + not enough hosts available." It is not a quota problem and raising quota will + not fix it. + +Practical guidance: start SMALL -- 1 control node and 1 or 2 workers -- confirm +the cluster comes up, then scale the worker count up if you need more and +capacity allows. A large first request is the second most common cluster +failure, after the flannel mistake. + +### 2.4 Wait for it, and verify + + openstack coe cluster show -f value -c status -c status_reason + # poll no faster than every 10 seconds; done when status = CREATE_COMPLETE + +Cluster builds take tens of minutes -- this is normal. But if a cluster sits in +CREATE_IN_PROGRESS far longer than the build should take and never completes, +the usual cause is the network driver: rebuild with calico (see 2.2). Delete the +stuck cluster first: `openstack coe cluster delete `. + +-------------------------------------------------------------------------------- + +## 3. Step 2 -- get and understand the kubeconfig + +The kubeconfig is the file `kubectl` and Jenkins use to talk to your cluster's +Kubernetes API. Fetch it, still signed in as `{{TENANT_SHORT_NAME}}-cluster`: + + umask 077 + openstack coe cluster config --dir + export KUBECONFIG=/config + kubectl get nodes # every node should report Ready + +What is inside it, and why it matters: + +- It points at your cluster's API server at its floating IP, on port 6443. +- It authenticates with a CLIENT CERTIFICATE embedded in the file (plus the + cluster's CA). There is no username or password in it. Anyone holding this file + has administrative access to your cluster -- treat it exactly like a password. +- The certificate is time-limited. When it expires, `kubectl` and your pipelines + start failing to authenticate; re-fetch the kubeconfig with the same command + and update the stored credential (section 4). [VERIFY-LIVE: confirm the exact + certificate validity period against your delivered cluster and set a renewal + reminder ahead of it.] + +Never commit the kubeconfig to a repository. It goes into the Jenkins credential +store next, and nowhere else. + +-------------------------------------------------------------------------------- + +## 4. Step 3 -- store the kubeconfig in Jenkins + +In Jenkins: Manage Jenkins > Credentials > (your domain) > Add Credentials. + +- Kind: "Secret file". +- File: upload the `config` file you fetched in section 3. +- ID: `omega-k8s-kubeconfig` (used by the pipeline examples below). +- Scope: the narrowest that your jobs need. + +That is the only place the kubeconfig lives inside Jenkins. Pipelines reference +it by ID; the "Kubernetes CLI" plugin loads it for the duration of a block and +masks it in the log. When you re-fetch the kubeconfig (expiry, or a rebuilt +cluster), update this one credential and every pipeline picks up the change. + +-------------------------------------------------------------------------------- + +## 5. Step 4, Pattern A -- deploy your app into the cluster + +This is the main flow: a pipeline that rolls an application out to the cluster +using the stored kubeconfig. Nothing here needs the cluster password; deploying +into an existing cluster is ordinary automation. + +### 5.1 A minimal application manifest + +Keep your Kubernetes manifests in your application repository, for example in a +`k8s/` directory. A minimal Deployment plus a LoadBalancer Service: + + # k8s/app.yaml + apiVersion: apps/v1 + kind: Deployment + metadata: + name: sample-app + spec: + replicas: 2 + selector: + matchLabels: { app: sample-app } + template: + metadata: + labels: { app: sample-app } + spec: + containers: + - name: web + image: nginx:stable + ports: + - containerPort: 80 + --- + apiVersion: v1 + kind: Service + metadata: + name: sample-app + spec: + type: LoadBalancer + selector: { app: sample-app } + ports: + - port: 80 + targetPort: 80 + +When Kubernetes sees `type: LoadBalancer`, the cloud gives the cluster a load +balancer and a floating IP for this Service. That address is how the outside +world reaches your app. + +### 5.2 The pipeline + + // Jenkinsfile -- Pattern A: deploy into the cluster + pipeline { + agent any + options { timestamps(); timeout(time: 30, unit: 'MINUTES') } + stages { + stage('Deploy') { + steps { + // The Kubernetes CLI plugin loads the stored kubeconfig for this + // block and masks it in the log. + withKubeConfig([credentialsId: 'omega-k8s-kubeconfig']) { + sh 'kubectl apply -f k8s/' + sh 'kubectl rollout status deployment/sample-app --timeout=5m' + } + } + } + stage('Show address') { + steps { + withKubeConfig([credentialsId: 'omega-k8s-kubeconfig']) { + // The external IP appears once the load balancer finishes + // provisioning (a few minutes on the first apply). + sh 'kubectl get service sample-app -o wide' + } + } + } + } + } + +### 5.3 Load balancers and your quota + +Every Service of `type: LoadBalancer` consumes one load balancer AND one +floating IP from your quota, and keeps them for as long as the Service exists. +Two consequences: + +- Do not create a LoadBalancer Service per microservice. Prefer a single ingress + controller (itself one LoadBalancer Service) that routes to many internal + Services by hostname or path. That turns N load balancers into one. +- When you tear an app down, delete its Service (section 8). Deleting only the + Deployment leaves the load balancer and floating IP allocated -- and billed + against your quota. + +-------------------------------------------------------------------------------- + +## 6. Pattern B -- the cluster as a Jenkins build-agent pool + +Instead of (or in addition to) deploying INTO the cluster, you can have Jenkins +run its BUILD jobs as short-lived pods inside the cluster. Each job gets a fresh, +isolated agent that disappears when the build ends. + +Setup, using the "Kubernetes" plugin: + +1. Manage Jenkins > Clouds > Add a new cloud > Kubernetes. +2. Kubernetes URL: your cluster's API endpoint, `https://:6443` + (the `server:` line in the kubeconfig from section 3). +3. Credentials: add the kubeconfig (or a Kubernetes service-account token scoped + to a build namespace) as the cloud's credential. A dedicated service account + with a limited role is better practice than the full-admin kubeconfig for + day-to-day agents. +4. Jenkins tunnel / URL: pods need to reach your Jenkins to register as agents, + so set the Jenkins URL to an address the cluster can reach. +5. Define a Pod Template: a `jnlp` agent container plus any tool containers your + builds need (for example a container with `kubectl` and `helm`). Give it a + label; jobs that request that label run in these pods. + +Two things to keep in mind: + +- These agent pods run on your cluster's worker nodes and consume their CPU and + memory. Size the cluster (section 2.3) for your build concurrency, or set pod + resource requests so builds queue rather than overcommit. +- Network reachability goes both ways: Jenkins must reach the cluster API + (endpoint above), and the pods must reach Jenkins (step 4). You have already + confirmed the first direction; confirm the second when you first enable this. + +Pattern B does not replace Pattern A: a common setup is build agents in the +cluster (B) that then deploy the built artifact to the same or another cluster +(A). + +-------------------------------------------------------------------------------- + +## 7. Worked example, end to end + +This runs the whole flow once with a stock image, to prove the path before you +wire in your own app. Sections in parentheses point back to the detail. + +1. Sign in as `{{TENANT_SHORT_NAME}}-cluster` (2.1) and create a small cluster + from the delivered template (2.3): + + openstack coe cluster create demo-k8s \ + --cluster-template {{TENANT_SHORT_NAME}}-k8s \ + --keypair {{TENANT_SHORT_NAME}}-key \ + --master-count 1 --node-count 1 + +2. Wait for CREATE_COMPLETE (2.4): + + openstack coe cluster show demo-k8s -f value -c status + +3. Fetch and check the kubeconfig (3): + + umask 077 + openstack coe cluster config demo-k8s --dir ./demo-kube + export KUBECONFIG=$PWD/demo-kube/config + kubectl get nodes + +4. Store `./demo-kube/config` in Jenkins as the "Secret file" credential + `omega-k8s-kubeconfig` (4). (For a quick local proof you can skip Jenkins and + use the `KUBECONFIG` export above directly.) + +5. Deploy the sample app (5.1): + + kubectl apply -f k8s/app.yaml + kubectl rollout status deployment/sample-app --timeout=5m + +6. Read the external address (wait for it to move from to an IP; the + load balancer takes a few minutes on first creation): + + kubectl get service sample-app -o wide + # then, from a host that can reach it: + curl http:/// + +7. Tear the example down (8): + + kubectl delete -f k8s/app.yaml # deletes the Service too -> releases + # the load balancer and floating IP + openstack coe cluster delete demo-k8s + +-------------------------------------------------------------------------------- + +## 8. Teardown and quota hygiene + +Delete in reverse order of creation, and delete the Service before anything +else, so the load balancer and floating IP are released: + +1. `kubectl delete -f k8s/app.yaml` (or `kubectl delete service `). This + is what releases the load balancer and its floating IP. +2. Delete other cloud resources your build created (servers, volumes, extra + floating IPs), newest first. +3. If you are finished with the cluster itself, delete it as + `{{TENANT_SHORT_NAME}}-cluster`: `openstack coe cluster delete `. + +Two hygiene rules that keep quota from silently filling: + +- A floating IP counts against your quota even when it is not attached to + anything. Release IPs you are not using. +- Make cleanup automatic. In a Jenkins pipeline, put teardown of per-build + resources in a `post { always { ... } }` block so a failed build cannot leak + quota. The delivered starter-kit sweep script (`ci-cleanup-sweep.sh`) removes + leftovers by name prefix and age; run it as a scheduled job as a backstop. + +-------------------------------------------------------------------------------- + +## 9. Troubleshooting + +Symptoms you are most likely to hit, and the fix: + +- "Missing value auth-url required for auth plugin password" -- the shell has no + cloud environment loaded. You started a fresh shell or a fresh agent. Re-source + your `OS_*` environment or select your `clouds.yaml` entry + (`export OS_CLOUD={{TENANT_SHORT_NAME}}`) before the command. + +- Cluster TEMPLATE create fails; dashboard shows only "Error: Unable to create + cluster template." -- the Public or Hidden flag is set. Uncheck both (2.2). + +- Cluster create fails, a node in ERROR with "No valid host was found. There are + not enough hosts available." -- the request is too big for the free capacity + right now, even if it is within quota. Use a smaller node count or a smaller + flavor and retry (2.3). Raising quota will not help. + +- Cluster stays in CREATE_IN_PROGRESS far too long and never completes -- most + often the network driver. Delete it and rebuild the template with calico; + flannel is not supported (2.2). + +- "Keypair not found" at cluster create -- you are signed in as the wrong + account. Cluster create runs as `{{TENANT_SHORT_NAME}}-cluster`, which owns the + keypair. Check `echo $OS_USERNAME` first. + +- A pipeline tries to create a cluster and is refused -- cluster create and + delete cannot use an application credential; they need the + `{{TENANT_SHORT_NAME}}-cluster` password, so they are not pipeline steps. + Create the cluster by hand (section 2); let the pipeline deploy into it. + +- `kubectl` reports "Unable to connect" or a certificate error -- the kubeconfig + is wrong or its certificate has expired. Re-fetch it (3) and update the stored + Jenkins credential (4). Make sure you kept the delivered CA and did not disable + verification. + +- A LoadBalancer Service shows EXTERNAL-IP `` for a long time -- the + load balancer is still provisioning (a few minutes is normal), or your load + balancer / floating IP quota is exhausted. Check your quota usage. + +-------------------------------------------------------------------------------- + +## 10. Glossary + +- Application credential: a scoped, revocable secret for the + `{{TENANT_SHORT_NAME}}-svc` account. All automation authenticates with it. It + cannot create clusters. +- The cluster account (`{{TENANT_SHORT_NAME}}-cluster`): the only account that + creates and deletes clusters, using its password. It owns the cluster keypair. +- Cluster template: the recipe (image, flavors, network driver, options) a + cluster is built from. Use calico; never set Public/Hidden. +- kubeconfig: the certificate-bearing file that grants admin access to a + cluster's Kubernetes API. Treat it as a password; store it in Jenkins as a + "Secret file" credential. +- LoadBalancer Service: a Kubernetes Service that asks the cloud for a real load + balancer and floating IP so an app is reachable from outside. Each one draws + from your quota. +- Ingress controller: one LoadBalancer Service that routes to many internal + Services, so you spend one load balancer instead of many. +- Floating IP: a routable address from the external pool (`provider-ext`). Counts + against quota even when unattached. + +-------------------------------------------------------------------------------- + +## 11. Where to go deeper + +You can complete this guide without these; open them when you want more than the +workflow above. + +- Handover Pack: your delivered identifiers, endpoints, CA bundle, and the full + three-account model with exact rights. +- Self-Service Guide: running your tenancy day to day (networks, servers, load + balancers, secrets) through the dashboard or command line. +- CI/Automation Integration Guide: application-credential pipelines in depth -- + credential handling, least-privilege per-pipeline credentials, quota pacing, + and cleanup discipline. +- Acceptance Checklist: the short set of checks that prove your tenancy works, + including the cluster lifecycle rows. +- Starter-kit scripts (delivered under scripts/): `smoke-test.sh` (read-only + token/catalog/quota check for the top of a pipeline), `tenancy-audit.sh` + (read-only snapshot of what you have), `acceptance-run.sh` (a full + build-and-teardown you can trim into a recurring smoke test), and + `ci-cleanup-sweep.sh` (the backstop quota sweep). +- Your tenant AI assistant kit: the Kubernetes, day-2 operations, and + troubleshooting references, for step-by-step help in context. + +For anything not covered here, or a change to your tenancy, contact +{{ACCOUNT_CONTACT}}. diff --git a/clientdocs/sweep-receipt.txt b/clientdocs/sweep-receipt.txt index 0425931..6db6f4b 100644 --- a/clientdocs/sweep-receipt.txt +++ b/clientdocs/sweep-receipt.txt @@ -9,12 +9,13 @@ # Repo lint fails when any pinned file drifts, until clientdocs/ has been # reviewed for staleness and this receipt is re-recorded with: # bash scripts/repo-lint.sh --record-clientdocs-sweep -510222d55997d1729c5621dbf409d46b8d1a1bbd37c8bd6a1b04ed323bc01ac2 clientdocs/README.md +d405a782f1b783533580f3fc3ac7060bf1751b8617c1cdae70488ccef19699e9 clientdocs/README.md b402dda98d3fe63f9b8a6b2798b86dc0c41a8e9348cdc197f32265c1e7267289 clientdocs/acceptance-checklist.md 30c46fd7644637c0a929bfdd195c451dff8cd0df6785cccf3a304a8422b82711 clientdocs/ai-assistant-guide.md ae51da963d4d3704e6a1bd8613a243616ac22c0b973833b31f1bd22a40f3c66e clientdocs/ci-integration-guide.md 60c890b551e0a16d8d249fc809e2b561d44c6ec6121bf8a20f21e5fe6e2e9c60 clientdocs/handover-pack.md 3f08eb458bcbb36172a5ca2785a6ec72a69a4f8cd440fec53ed604f074b1b8c6 clientdocs/intake-form.md +3f8eed5359c7067d778e5ad63f40c971b759f379ca2fde2245c38a1f27c2fbd5 clientdocs/jenkins-kubernetes-guide.md 96a077bb2e5732a0443683755bcefa044a603fe323024ca887165177e05c390b clientdocs/scripts/Jenkinsfile.example 1955db5d789a2e48d9054e307fbec2678efc1fbafda18103caa389cb896701fd clientdocs/scripts/acceptance-run.sh c2a2ff7ad719cb5866b1b58f5ca2cac930ccc99b96efaf8a3f12301d645727d3 clientdocs/scripts/ci-cleanup-sweep.sh @@ -25,7 +26,7 @@ 7644e527b6ce66804fa9e08315e51e06159a8feb525a8b7d1b8c9edf23e4a86a clientdocs/tenant-skill/SKILL.md c7708c35d1d5fceab763268f6a10ed214e23e9458018b115cb944b6a3168f182 clientdocs/tenant-skill/references/ci-automation.md 22e7ec9c16bc66fcd64264ee2fb9bd2f12076594d7702ffd94bff0f00ddc7713 clientdocs/tenant-skill/references/day2-operations.md -0acbc175a3b7a0c0a5af49a0a731f25ce8f0114619066c1b32473045a05483b2 clientdocs/tenant-skill/references/kubernetes.md +8dfbbb8c053c32c7a63ed58910366f4262fb1d0ac2935defc9c507233b1d96b0 clientdocs/tenant-skill/references/kubernetes.md b844e00610df813275779320a13ecc131525a89b052bd5c8ab522d7666eb4fad clientdocs/tenant-skill/references/troubleshooting.md 2135f05ccdd411ce707bc6ec54c61040284259142b4b388c0c94cae5f6e7efcb clientdocs/welcome.md e00900e6d08d19c195edee6cab5d8198295a35371cd6333a50ca5ad28940b129 docs/tenant-onboarding-contract.md diff --git a/clientdocs/tenant-skill/references/kubernetes.md b/clientdocs/tenant-skill/references/kubernetes.md index 7dc7d4f..1de8b34 100644 --- a/clientdocs/tenant-skill/references/kubernetes.md +++ b/clientdocs/tenant-skill/references/kubernetes.md @@ -84,6 +84,13 @@ checkboxes are the FIRST thing to check. You lose nothing by leaving them unchecked. +The second hard rule: use `--network-driver calico`. This platform +supports calico. Do NOT choose flannel -- a flannel cluster never +finishes coming up: the create sits IN_PROGRESS and eventually fails, +with the nodes never going Ready. A cluster stuck in CREATE_IN_PROGRESS +far past the normal build time is almost always this. Rebuild the +template with calico. + A known-good CLI starting point, mirroring the platform-delivered template (works signed in as `-svc` or `-cluster`): diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 0acfbdb..2ce4bf5 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -31,8 +31,8 @@ - **OPEN security rows:** SEC-001 (rotate at v1 close, re-ruled 2026-07-06), SEC-003 (custodians + second-person unseal -- DEFERRED; keeps d011-06 MANUAL), SEC-004 (flip repo to private at v1 close -- deferral reaffirmed). -- **Next-free numbers:** D = 076, DOCFIX = 123, BUNDLEFIX = 012. - (Seeded from the 2026-07-08 scan, post-addendum-39.) +- **Next-free numbers:** D = 076, DOCFIX = 124, BUNDLEFIX = 012. + (Seeded from the 2026-07-08 scan, post-addendum-41.) --- @@ -306,6 +306,33 @@ (RECOMMENDED -- dormant shell); (d) quota-vs-capacity expectation (trim testcloud quotas or document in clientdocs) -- unruled. +## Jumphost stream -- session 2026-07-08 (cont.): Jenkins+K8s guide overnight (addendum 41) + +- **DELIVERED (DOCFIX-123):** new clientdocs/jenkins-kubernetes-guide.md -- + the single end-to-end Jenkins-to-Kubernetes workflow HUB (Pattern A deploy- + into-cluster + Pattern B build-agents; self-create cluster path; kubeconfig- + as-Jenkins-credential; NoValidHost sizing; flannel/Public-Hidden/keypair + traps; troubleshooting; glossary; cross-links). Flannel hole also closed in + tenant-skill kubernetes.md. Guide added to LEAK_FILES (harness change), + README updated, L7 receipt 30 files, gauntlet 38/38. Instantiated to + ~/devteam-package (devteam values). Operator pushes tonight. +- **HELD for operator (morning), in priority order:** + 1. Broad ease-of-use review across ALL handover docs (single-flow, minimize + doc-switching, enough component detail). The judgment call the operator + wants to drive. The Jenkins guide is the model/first instance. + 2. Running DOCFIX list (operator still appending): handover-pack.md data + review + it already carries {{DASHBOARD_URL}}; cred-.txt-file dashboard_url + field = change to cloud-mutating tenant-onboard.sh -> GATED (needs harness + + gated live check); driver guidance (DONE this addendum). + 3. Offboard Phase-E0 defect: admin-side identity:list_application_credentials + is always-403 (base admin_or_owner; measured), so offboard v2 --apply + counts 3 false failures and FATALs at exit 23 BEFORE Phase F (cred-dir + retirement). foil1 offboard hit this live 2026-07-08; cred dir retired by + hand. Fix: owner-context-or-skip E0 inventory, do not count known-403 as + failure; + harness case. First live --apply of offboard v2. + 4. devteam TestCluster2 (flannel, 5ef6e10f): stuck CREATE_IN_PROGRESS 6h+ -- + needs gated delete + calico rebuild. Real proof of the flannel finding. + ## REMAINING OPEN WORK (single list for session resume; 2026-07-07, updated post-devteam-window) 1. **Devteam delivery -- last two steps are OPERATOR-MANUAL (everything diff --git a/docs/v1-redeploy-changelog.md b/docs/v1-redeploy-changelog.md index a34e2d3..bb7b2d7 100644 --- a/docs/v1-redeploy-changelog.md +++ b/docs/v1-redeploy-changelog.md @@ -2953,4 +2953,59 @@ - Numbers consumed: none. REVERT: documentation-only entry (this file + docs/session-ledger.md); git checkout HEAD~ -- docs/v1-redeploy-changelog.md docs/session-ledger.md. -Next-free after this push (per scan, keep this line unwrapped): D-076, DOCFIX-121, BUNDLEFIX-012. +### 2026-07-08 (addendum 41, jumphost stream) -- DOCFIX-123: new end-to-end Jenkins+Kubernetes guide (single-flow HUB) + flannel-driver hole closed; devteam package instantiated + +Overnight deliverable (operator away, questions pre-answered before step-away). +Root problem the operator named: the information a client needs to stand up a +Jenkins-to-Kubernetes workflow is spread across 6+ handover files with heavy +duplication and no single flow, so a client must jump between documents and can +fall into workflow holes that cause errors (the flannel driver being the live +example -- devteam's self-made flannel cluster has sat un-converged 6h+ this +session, real confirmation). + +- **NEW clientdocs/jenkins-kubernetes-guide.md (DOCFIX-123):** one + top-to-bottom workflow a client can complete WITHOUT switching documents. + Covers both patterns the operator ruled in scope: Pattern A (pipeline deploys + INTO the cluster via a stored kubeconfig credential -- the spine) and Pattern + B (cluster as an elastic Jenkins build-agent pool). Sections: component map + + the 3-account rule; prerequisites + clouds.yaml; cluster self-create (the + ruled path) with the calico/no-flannel + no-Public/Hidden + no-baked-keypair + rules and the right-size-to-quota-AND-host-capacity (NoValidHost) caveat; + kubeconfig fetch/understand (client-cert auth, API on the cluster floating IP + :6443, treat as secret, cert expiry marked VERIFY-LIVE); store as a Jenkins + Secret-file credential; Pattern A manifest + Jenkinsfile + LB/quota guidance; + Pattern B cloud/pod-template setup; a stock end-to-end worked example; + teardown + quota hygiene; troubleshooting (7 verbatim symptoms incl. the + Missing-auth-url, Public/Hidden 403, NoValidHost, stuck-IN_PROGRESS=flannel, + keypair-wrong-account, app-cred-cannot-create, kubeconfig-expiry); glossary; + and a cross-link map so the guide is the HUB and the deep-dive docs are owned + once. Built from a two-agent read-only inventory (dup/gap map) + conventions + audit; the Jenkins-to-cluster bridge (kubeconfig-as-credential, dynamic + agents, a real deploy stage, the NoValidHost sizing rule) was the true hole + and is now owned here. +- **Flannel hole closed in the other doc devteam uses:** tenant-skill + references/kubernetes.md "Creating your own cluster template" gains the + explicit second hard rule (use calico; flannel never converges; a long + CREATE_IN_PROGRESS is almost always this). Was implicit (calico in the CLI + block only); now stated. +- **Gates:** pure ASCII+LF; zero internal-term leakage (the guide is now + scanned -- added to LEAK_FILES in tests/clientdocs-skill/run-tests.sh, a + harness change); placeholders are standard tokens only + (TENANT_SHORT_NAME/AUTH_URL/REGION/CA_BUNDLE_FILE/ACCOUNT_CONTACT + THIS + banner). README Contents + Packaging lists updated. L7 sweep receipt + re-recorded (30 files). repo-lint 0 fail; clientdocs-skill 8/8; + clientdocs-scripts 47/47; full gauntlet 38/38. +- **Package:** guide instantiated to ~/devteam-package/jenkins-kubernetes-guide.md + (devteam values, TEMPLATE banner + title suffix removed, zero tokens + verified); operator pushes with the rest tonight. +- **HELD for operator (morning):** the broad ease-of-use review across ALL + handover docs (single-workflow, minimize doc-switching, component-detail) -- + a judgment call the operator wants to drive; the running DOCFIX list + (handover-pack.md data review + dashboard URL, cred-.txt-file dashboard_url + field, which is a change to the cloud-mutating onboard script -> gated); the + offboard Phase-E0 always-403 defect (below). devteam's stuck flannel cluster + (TestCluster2) awaits a gated delete + calico rebuild. +- Numbers consumed: DOCFIX-123. +REVERT: git checkout HEAD~ -- clientdocs tests/clientdocs-skill/run-tests.sh docs/session-ledger.md docs/v1-redeploy-changelog.md; rm clientdocs/jenkins-kubernetes-guide.md ~/devteam-package/jenkins-kubernetes-guide.md; then bash scripts/repo-lint.sh --record-clientdocs-sweep (re-pin receipt to 29 files); no cloud-side change to revert. + +Next-free after this push (per scan, keep this line unwrapped): D-076, DOCFIX-124, BUNDLEFIX-012. diff --git a/tests/clientdocs-skill/run-tests.sh b/tests/clientdocs-skill/run-tests.sh index dc16bc1..a1d2b9c 100644 --- a/tests/clientdocs-skill/run-tests.sh +++ b/tests/clientdocs-skill/run-tests.sh @@ -66,6 +66,7 @@ LEAK_RX='D-0[0-9]{2}|DOCFIX|BUNDLEFIX|juju|charm|maas|vault|ceph|ovn[^a-z]|magnum|octavia|barbican|keystone|horizon|glance|cinder|neutron|nova[^a-z]|amphora|capi|caracal|runbook|10\.12\.|baldurkeep|vopenstack|neumatrix|jumphost' LEAK_FILES=("${FILES[@]}") [[ -f "$REPO/clientdocs/ai-assistant-guide.md" ]] && LEAK_FILES+=("$REPO/clientdocs/ai-assistant-guide.md") +[[ -f "$REPO/clientdocs/jenkins-kubernetes-guide.md" ]] && LEAK_FILES+=("$REPO/clientdocs/jenkins-kubernetes-guide.md") for s in "$REPO"/clientdocs/scripts/*.sh; do [[ -f "$s" ]] && LEAK_FILES+=("$s"); done LEAKS="$(grep -rniE "$LEAK_RX" "${LEAK_FILES[@]}" 2>/dev/null)" if [[ -z "$LEAKS" ]]; then