diff --git a/runbooks/00-pre-deploy.md b/runbooks/00-pre-deploy.md deleted file mode 100644 index aa07d86..0000000 --- a/runbooks/00-pre-deploy.md +++ /dev/null @@ -1,142 +0,0 @@ -# Runbook 00 — Pre-Deploy - -## Purpose - -Prepare for a clean Caracal rebuild of the VR0 DC0 Omega Cloud. Capture all -state needed for rollback, gracefully tear down dependent workloads, and verify -the destination environment is ready before destroying the existing OpenStack -model. - -## Prerequisites - -- SSH access to jumphost `vopenstack-jesse` as `jessea123` -- `admin-openrc` and `user1-openrc` available in `$HOME` -- Access to the Juju controller hosting the `openstack` model -- Access to the capi-mgmt.maas k3s cluster (kubeconfig present) -- NetBox IPv4 imports completed (per `netbox/ipv4-prefixes-import.py`) -- NetBox VLAN imports completed (per `netbox/vlans-import.py`) - -## Phase 1 — Verify NetBox readiness (gating) - -Run the verification path of the NetBox import scripts. Confirm all entries -appear correctly scoped to VR0 DC0. - -```bash -cd ~/vr0-dc0-caracal -NETBOX_URL=https://netbox.baldurkeep.com NETBOX_TOKEN= \ - python3 netbox/ipv4-prefixes-import.py --verify-only -NETBOX_URL=https://netbox.baldurkeep.com NETBOX_TOKEN= \ - python3 netbox/vlans-import.py --verify-only -``` - -Expected: all prefixes and VLANs report scope-OK, no MISSING entries. - -## Phase 2 — Capture current state - -Backups needed for potential rollback: - -```bash -# Vault unseal keys and root CA cert -juju ssh vault/0 -- sudo cat /var/snap/vault/common/vault.crt > ~/backups/$(date +%F)/vault-root-ca.crt -# (Unseal keys MUST be on file from initial Vault setup; verify presence) -ls -la ~/.vault-keys - -# Export current bundle -juju export-bundle --model openstack > ~/backups/$(date +%F)/bundle-pre-rebuild.yaml - -# Snapshot of current 'juju status' -juju status --model openstack --format=yaml > ~/backups/$(date +%F)/juju-status-pre-rebuild.yaml - -# Inventory of FIPs and tenant resources we might want to recreate -source ~/admin-openrc -openstack floating ip list -c "Floating IP Address" -c "Fixed IP Address" \ - -c "Project" -f csv > ~/backups/$(date +%F)/floating-ips.csv -openstack server list --all-projects -c ID -c Name -c Project -c Status -f csv \ - > ~/backups/$(date +%F)/servers.csv -openstack network list --all-projects -c ID -c Name -c Project -f csv \ - > ~/backups/$(date +%F)/networks.csv -openstack loadbalancer list -c id -c name -c project_id -c vip_address -f csv \ - > ~/backups/$(date +%F)/loadbalancers.csv -``` - -## Phase 3 — KVM snapshots of openstack0-3 - -From the jumphost (which is the hypervisor): - -```bash -for vm in openstack0 openstack1 openstack2 openstack3; do - sudo virsh snapshot-create-as --domain "$vm" \ - --name "pre-caracal-rebuild-$(date +%F)" \ - --description "Pre-Caracal rebuild baseline" \ - --atomic -done -sudo virsh snapshot-list openstack0 -``` - -These snapshots are the disaster-recovery point. - -## Phase 4 — Graceful CAPI workload teardown (D-013) - -Delete the CAPI workload cluster cleanly so its OpenStack resources (LBs, FIPs, -volumes, Octavia members) are released by CAPI controllers before model destroy. - -```bash -export KUBECONFIG=~/magnum-capi/phase3/capi-mgmt-cluster.kubeconfig -# (Adjust path if kubeconfig has moved) - -# Delete the workload cluster — CAPI handles tenant OpenStack cleanup -kubectl delete cluster capi-mgmt-cluster -n default -# Wait for finalizers; this may take ~10 minutes -kubectl wait --for=delete cluster/capi-mgmt-cluster -n default --timeout=15m -``` - -Verify on the OpenStack side that resources were released: - -```bash -source ~/admin-openrc -openstack server list --all-projects | grep -i capi || echo "No CAPI servers remaining" -openstack loadbalancer list | grep -i capi || echo "No CAPI LBs remaining" -openstack floating ip list -c "Floating IP Address" -c "Fixed IP Address" -f csv -``` - -## Phase 5 — Preserve capi-mgmt.maas itself - -The bootstrap k3s + CAPI controllers on `capi-mgmt.maas` are NOT destroyed — -they will be re-used post-rebuild as the Magnum CAPI mgmt plane. Verify the -controllers are still healthy: - -```bash -ssh capi-mgmt.maas -- sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml \ - get pods -A -``` - -Confirm: -- `capi-system` namespace pods Running -- `capo-system` (CAPI OpenStack provider) pods Running -- `cert-manager` pods Running -- `orc-system` (OpenStack Resource Controller) pods Running - -## Phase 6 — Final go/no-go checklist - -Do not proceed to `runbooks/01-destroy-model.md` until all of the following pass: - -- [ ] NetBox verification clean -- [ ] Vault unseal keys backed up and verified readable -- [ ] `bundle-pre-rebuild.yaml` exists and is non-empty -- [ ] `juju-status-pre-rebuild.yaml` shows desired-pre-destroy state captured -- [ ] All four KVM snapshots created (`virsh snapshot-list` confirms) -- [ ] CAPI workload cluster deletion completed (`kubectl get cluster` returns - "no resources found") -- [ ] OpenStack-side resources from CAPI workload are released (no orphaned LBs, - FIPs, volumes) -- [ ] capi-mgmt.maas k3s cluster controllers all Running - -## Notes - -- Snapshot disk space consumption can grow significantly during the rebuild - window. Verify free space on `/var/lib/libvirt/images` prior to running - the rebuild deploy. -- If Vault unseal keys cannot be located, STOP. A failed Vault re-init without - the original keys means lost issued certificates and is destructive to any - data sealed under the existing root key. This MUST be confirmed before model - destroy. diff --git a/runbooks/01a-octavia-pki-generation.md b/runbooks/01a-octavia-pki-generation.md deleted file mode 100644 index 65bd707..0000000 --- a/runbooks/01a-octavia-pki-generation.md +++ /dev/null @@ -1,650 +0,0 @@ -# Runbook 01a — Octavia LBaaS PKI generation - -**Status:** Pre-deploy execution. Runs between `01-destroy-model.md` and `02-deploy.md`. -**Numbering rationale:** Octavia PKI artifacts must exist on the deploy host before -`juju deploy` is invoked (the values are referenced by the overlay file). Placing -this between destroy and deploy aligns generation with the "fresh rebuild" framing. - -**Cross-references:** -- D-007 (Octavia in bundle from day one) -- Bundle `octavia.options` PKI material section -- `overlays/octavia-pki.yaml` (gitignored — output of this runbook) -- Workstream 3a decision (2026-05-22): generate fresh, EC P-384 CAs, overlay-file approach - ---- - -## 1. Purpose & scope - -This runbook generates a complete two-tier PKI for Charmed Octavia's -amphora load-balancer trust domain: - -- **Issuing CA** — Octavia uses this to sign each amphora's server certificate - at LB-creation time. Octavia receives the **private key** and **passphrase**. -- **Controller CA** — amphorae's trust anchor for connections **from** the - Octavia controller. Octavia only receives the **cert** (no key needed at - runtime); the controller's identity is proved by: -- **Controller certificate** — signed by Controller CA, presented by the - Octavia controller to each amphora. Bundled as cert + key into a single - PEM blob. - -Five charm options consume the artifacts (`octavia` application): - -| Charm option | Content | Format | -|---|---|---| -| `lb-mgmt-issuing-cacert` | Issuing CA certificate | base64-encoded PEM | -| `lb-mgmt-issuing-ca-private-key` | Issuing CA encrypted private key | base64-encoded PEM (already encrypted with passphrase) | -| `lb-mgmt-issuing-ca-key-passphrase` | Issuing CA key passphrase | plain string (NOT base64) | -| `lb-mgmt-controller-cacert` | Controller CA certificate | base64-encoded PEM | -| `lb-mgmt-controller-cert` | Controller cert + key, concatenated | base64-encoded PEM bundle | - -**Scope:** v1 testcloud (VR0 DC0 Omega Cloud). Roosevelt deltas documented in -section 14. - -**Out of scope:** Octavia API TLS (issued by Vault via `octavia:certificates` -relation); rotation procedure (deferred to Roosevelt runbook). - ---- - -## 2. Decisions captured - -Per workstream 3a sign-off (2026-05-22): - -| Decision | Choice | Roosevelt parallel | -|---|---|---| -| Cert provenance | Generate fresh (no Bobcat-backup copy) | Vault PKI engine | -| CA key algorithm | EC P-384 | EC P-384 (Vault root) | -| Controller cert algorithm | EC P-256 | EC P-256 | -| CA validity | 10 years | 5-year intermediate, Vault-rotated | -| Controller cert validity | 2 years | 90 days, auto-rotated | -| Distribution method | Juju overlay file (gitignored) | Vault-injected at deploy | -| Storage path on jumphost | `$HOME/octavia-pki/` | Vault PKI mounts | -| Passphrase strength | 32 random bytes, base64-encoded (44 chars) | Vault-generated | - -**Naming convention:** - -- Issuing CA CN: `VR0 DC0 Omega Cloud Octavia Issuing CA` -- Controller CA CN: `VR0 DC0 Omega Cloud Octavia Controller CA` -- Controller cert CN: `octavia-controller.omega.dc0.vr0.cloud.neumatrix.local` -- Controller cert SANs: above CN, plus `octavia.omega.dc0.vr0.cloud.neumatrix.local`, plus `10.12.4.233` (the Octavia API VIP per workstream 2) -- Organization (O): `Neumatrix` - ---- - -## 3. Prerequisites - -- Executor is on jumphost `vopenstack-jesse` as `jessea123`. -- `openssl` version 3.x or later installed (`openssl version` to confirm). -- `$HOME` is writable (snap-confined `openstackclients` cannot read `/tmp`; - all paths must resolve under `$HOME`). -- Git repository `openstack-caracal-ipv4` cloned on jumphost at a known path - (referred to as `$REPO` throughout). Set this in the executor's shell: - ```bash - export REPO=$HOME/repos/openstack-caracal-ipv4 # adjust to actual clone path - ``` -- Repository is on `main` branch and clean (`cd $REPO && git status` shows clean tree). -- Previous workstream 2 commit has been pushed (bundle has the VIP assignments and - active hacluster stack — verify with `grep -c "^ vip: 10.12.4." "$REPO/bundle.yaml"`, - expect 12). - ---- - -## 4. Pre-flight: gitignore patch (DO THIS FIRST) - -**Critical:** the `.gitignore` patch goes in BEFORE any private key material -exists on disk. This minimizes the race window for an accidental commit. - -```bash -cd "$REPO" - -# Append to .gitignore (idempotent — check if already present first) -grep -q "octavia-pki.yaml" .gitignore || cat >> .gitignore <<'EOF' - -# Octavia PKI artifacts — NEVER commit -overlays/octavia-pki.yaml -octavia-pki/ -*.key -*.key.enc -passphrase.txt -EOF - -# Review the diff -git diff .gitignore - -# Commit and push BEFORE generating any keys -git add .gitignore -git commit -m "gitignore: octavia PKI artifacts and overlay (runbook 01a)" -git push origin main -``` - -**Verify the gitignore is effective:** - -```bash -# This should NOT show overlays/octavia-pki.yaml even as untracked -touch overlays/octavia-pki.yaml -git status --short overlays/ # expect: empty output for octavia-pki.yaml -rm overlays/octavia-pki.yaml -``` - -If the test file does show as untracked, **STOP** and fix the gitignore syntax before -generating any secrets. - ---- - -## 5. Workspace setup - -```bash -WORKDIR=$HOME/octavia-pki -mkdir -p "$WORKDIR"/{issuing-ca,controller-ca,controller,overlay-build} -chmod 700 "$WORKDIR" -cd "$WORKDIR" -echo "Working in: $WORKDIR" -``` - -Resulting layout: - -``` -$HOME/octavia-pki/ -├── issuing-ca/ # passphrase.txt, .key.enc, .cert.pem -├── controller-ca/ # passphrase.txt, .key.enc, .cert.pem -├── controller/ # .key, .csr, .cert.pem, .bundle.pem, .cnf -└── overlay-build/ # base64 intermediates → consumed by step 10 -``` - ---- - -## 6. Generate Issuing CA - -EC P-384 key encrypted with random 32-byte passphrase. Self-signed cert, 10y validity. - -```bash -cd "$WORKDIR/issuing-ca" - -# Generate passphrase (no trailing newline — required for clean YAML embedding) -openssl rand -base64 32 | tr -d '\n' > passphrase.txt -chmod 600 passphrase.txt - -# Sanity-check -test $(wc -c < passphrase.txt) -eq 44 || { echo "ERROR: passphrase length wrong"; exit 1; } - -# Generate EC P-384 private key, encrypted with passphrase -openssl genpkey -algorithm EC \ - -pkeyopt ec_paramgen_curve:P-384 \ - -aes-256-cbc \ - -pass file:passphrase.txt \ - -out issuing-ca.key.enc -chmod 600 issuing-ca.key.enc - -# Self-sign cert (10 years, SHA-384) -openssl req -new -x509 -sha384 \ - -key issuing-ca.key.enc \ - -passin file:passphrase.txt \ - -days 3650 \ - -subj "/CN=VR0 DC0 Omega Cloud Octavia Issuing CA/O=Neumatrix" \ - -out issuing-ca.cert.pem - -# Verify -openssl x509 -in issuing-ca.cert.pem -noout -dates -subject -openssl verify -CAfile issuing-ca.cert.pem issuing-ca.cert.pem -# Expect: issuing-ca.cert.pem: OK - -ls -la -``` - ---- - -## 7. Generate Controller CA - -Identical pattern; different CN. - -```bash -cd "$WORKDIR/controller-ca" - -openssl rand -base64 32 | tr -d '\n' > passphrase.txt -chmod 600 passphrase.txt -test $(wc -c < passphrase.txt) -eq 44 || { echo "ERROR: passphrase length wrong"; exit 1; } - -openssl genpkey -algorithm EC \ - -pkeyopt ec_paramgen_curve:P-384 \ - -aes-256-cbc \ - -pass file:passphrase.txt \ - -out controller-ca.key.enc -chmod 600 controller-ca.key.enc - -openssl req -new -x509 -sha384 \ - -key controller-ca.key.enc \ - -passin file:passphrase.txt \ - -days 3650 \ - -subj "/CN=VR0 DC0 Omega Cloud Octavia Controller CA/O=Neumatrix" \ - -out controller-ca.cert.pem - -openssl x509 -in controller-ca.cert.pem -noout -dates -subject -openssl verify -CAfile controller-ca.cert.pem controller-ca.cert.pem -# Expect: controller-ca.cert.pem: OK -``` - -**Why Controller CA's key is encrypted even though Octavia never uses it:** -The Controller CA key is needed for future rotations of the controller cert. -Encrypting it (with its own passphrase, separate from Issuing CA's) is defense -in depth — if the jumphost is compromised, the key still requires the -passphrase to be useful for forging controller certs. - ---- - -## 8. Generate Controller certificate - -EC P-256 key (no encryption — Octavia must read it at startup), CSR with SAN -extensions, signed by Controller CA, 2y validity. - -```bash -cd "$WORKDIR/controller" - -# Generate unencrypted EC P-256 key -openssl genpkey -algorithm EC \ - -pkeyopt ec_paramgen_curve:P-256 \ - -out controller.key -chmod 600 controller.key - -# CSR config with SAN extensions -cat > controller.cnf <<'EOF' -[req] -distinguished_name = req_distinguished_name -req_extensions = v3_req -prompt = no - -[req_distinguished_name] -CN = octavia-controller.omega.dc0.vr0.cloud.neumatrix.local -O = Neumatrix - -[v3_req] -keyUsage = critical, digitalSignature, keyEncipherment -extendedKeyUsage = clientAuth, serverAuth -subjectAltName = @alt_names - -[alt_names] -DNS.1 = octavia-controller.omega.dc0.vr0.cloud.neumatrix.local -DNS.2 = octavia.omega.dc0.vr0.cloud.neumatrix.local -IP.1 = 10.12.4.233 -EOF - -# Generate CSR -openssl req -new -sha256 \ - -key controller.key \ - -config controller.cnf \ - -out controller.csr - -# Sign with Controller CA (2 years) -openssl x509 -req -sha256 \ - -in controller.csr \ - -CA "$WORKDIR/controller-ca/controller-ca.cert.pem" \ - -CAkey "$WORKDIR/controller-ca/controller-ca.key.enc" \ - -passin file:"$WORKDIR/controller-ca/passphrase.txt" \ - -CAcreateserial \ - -days 730 \ - -extfile controller.cnf \ - -extensions v3_req \ - -out controller.cert.pem - -# Bundle cert + key (the lb-mgmt-controller-cert option expects both in one PEM) -cat controller.cert.pem controller.key > controller.bundle.pem -chmod 600 controller.bundle.pem -``` - -**Verify the chain and SAN:** - -```bash -# Chain verifies -openssl verify -CAfile "$WORKDIR/controller-ca/controller-ca.cert.pem" controller.cert.pem -# Expect: controller.cert.pem: OK - -# SAN extensions present -openssl x509 -in controller.cert.pem -noout -ext subjectAltName -# Expect: -# DNS:octavia-controller.omega.dc0.vr0.cloud.neumatrix.local, -# DNS:octavia.omega.dc0.vr0.cloud.neumatrix.local, -# IP Address:10.12.4.233 - -# Validity -openssl x509 -in controller.cert.pem -noout -dates -# Expect: notAfter ~2 years from today - -# Bundle integrity (cert + key match) -openssl x509 -in controller.bundle.pem -noout -pubkey > /tmp/cert.pub -openssl pkey -in controller.bundle.pem -pubout > /tmp/key.pub -diff /tmp/cert.pub /tmp/key.pub && echo "Bundle cert/key match" -rm /tmp/cert.pub /tmp/key.pub -``` - ---- - -## 9. Final chain verification - -A standalone block to confirm the full chain is sound before consuming for Octavia: - -```bash -cd "$WORKDIR" - -echo "=== Issuing CA ===" -openssl x509 -in issuing-ca/issuing-ca.cert.pem -noout -subject -dates -openssl verify -CAfile issuing-ca/issuing-ca.cert.pem issuing-ca/issuing-ca.cert.pem - -echo "" -echo "=== Controller CA ===" -openssl x509 -in controller-ca/controller-ca.cert.pem -noout -subject -dates -openssl verify -CAfile controller-ca/controller-ca.cert.pem controller-ca/controller-ca.cert.pem - -echo "" -echo "=== Controller cert ===" -openssl x509 -in controller/controller.cert.pem -noout -subject -dates -openssl verify -CAfile controller-ca/controller-ca.cert.pem controller/controller.cert.pem -``` - -All three "verify" lines must show `: OK`. If any do not, **STOP** and investigate -before proceeding. - ---- - -## 10. Base64-encode artifacts - -Each base64 file is a single line (no wrapping); each becomes one YAML value. - -```bash -cd "$WORKDIR/overlay-build" - -# Issuing CA cert (base64) -base64 -w0 "$WORKDIR/issuing-ca/issuing-ca.cert.pem" > issuing-cacert.b64 - -# Issuing CA private key (already encrypted PEM → base64) -base64 -w0 "$WORKDIR/issuing-ca/issuing-ca.key.enc" > issuing-ca-private-key.b64 - -# Controller CA cert -base64 -w0 "$WORKDIR/controller-ca/controller-ca.cert.pem" > controller-cacert.b64 - -# Controller cert + key bundle -base64 -w0 "$WORKDIR/controller/controller.bundle.pem" > controller-cert.b64 - -# Sanity-check sizes (expect 500-2000 chars each) -wc -c *.b64 -``` - ---- - -## 11. Assemble the overlay file - -```bash -# Read each artifact into shell variables -ISSUING_CACERT=$(cat "$WORKDIR/overlay-build/issuing-cacert.b64") -ISSUING_CA_KEY=$(cat "$WORKDIR/overlay-build/issuing-ca-private-key.b64") -ISSUING_CA_PASS=$(cat "$WORKDIR/issuing-ca/passphrase.txt") -CONTROLLER_CACERT=$(cat "$WORKDIR/overlay-build/controller-cacert.b64") -CONTROLLER_CERT=$(cat "$WORKDIR/overlay-build/controller-cert.b64") - -# Assemble overlay (note: passphrase is YAML-quoted; cert blobs are not — they're -# guaranteed-safe base64 without special chars) -mkdir -p "$REPO/overlays" -cat > "$REPO/overlays/octavia-pki.yaml" < - # lb-mgmt-controller-cert: - # lb-mgmt-issuing-ca-key-passphrase: - # lb-mgmt-issuing-ca-private-key: - # lb-mgmt-issuing-cacert: -``` - -**With this block:** - -```yaml - # ----- PKI material ------------------------------------------------- - # 5 lb-mgmt-* options are supplied via overlays/octavia-pki.yaml - # (gitignored). Generated per runbooks/01a-octavia-pki-generation.md. - # Deploy with: - # juju deploy ./bundle.yaml \ - # --overlay overlays/vr0-dc0-testcloud.yaml \ - # --overlay overlays/octavia-pki.yaml -``` - -Commit this bundle change separately from the overlay generation work: - -```bash -cd "$REPO" -git diff bundle.yaml -git add bundle.yaml -git commit -m "bundle: octavia PKI moves to overlay (runbook 01a) - -Remove inline placeholders + TODO(octavia-cert) block. PKI values now -supplied via overlays/octavia-pki.yaml (gitignored), generated per -runbooks/01a-octavia-pki-generation.md. Decision per workstream 3a -(2026-05-22): industry-best-practice secret handling on testcloud -to rehearse Roosevelt's Vault-PKI-backed posture." -git push origin main -``` - ---- - -## 13. Sensitive-file backup - -The Issuing CA private key + its passphrase are the crown jewels of the LB trust -domain. Loss → cannot sign new amphora certs (LBs gradually break). Exposure → -attacker can forge amphora identities and intercept tenant LB traffic. - -**Minimum backup for testcloud:** - -```bash -cd $HOME -BACKUP_NAME="octavia-pki-backup-$(date +%Y%m%d-%H%M%S).tar.gz" - -tar -czf "$BACKUP_NAME" -C $HOME octavia-pki/ - -# Encrypt with strong symmetric cipher -gpg --symmetric --cipher-algo AES256 --output "${BACKUP_NAME}.gpg" "$BACKUP_NAME" - -# Shred the unencrypted tar -shred -uvz "$BACKUP_NAME" - -ls -la "${BACKUP_NAME}.gpg" -``` - -**Move `${BACKUP_NAME}.gpg` off-host** (your decision — admin workstation -encrypted drive, password-manager attachment, dedicated secrets vault, etc.). -Do NOT leave it sitting in $HOME on the jumphost long-term — that's a single -point of compromise. - -**Roosevelt note:** Vault PKI engine stores all of this — no manual backup -required; Vault's own backup mechanism covers it. The procedure above is -testcloud-only. - ---- - -## 14. Cleanup of intermediates - -After successful deploy + verification (section 14), shred files that are not -needed for future rotation: - -```bash -# Optional: shred the base64 intermediates (regeneratable from PEM sources) -shred -uvz "$WORKDIR/overlay-build/"*.b64 -rmdir "$WORKDIR/overlay-build" - -# Optional: shred the CSR (regeneratable if needed) -shred -uvz "$WORKDIR/controller/controller.csr" - -# DO NOT shred any of the following — they are needed for future operations: -# - issuing-ca/{issuing-ca.cert.pem, issuing-ca.key.enc, passphrase.txt} -# - controller-ca/{controller-ca.cert.pem, controller-ca.key.enc, passphrase.txt} -# - controller/{controller.key, controller.cert.pem, controller.bundle.pem, controller.cnf} -# -# Specifically: -# - Issuing CA artifacts: required for signing new amphoras (Octavia uses them runtime) -# - Controller CA artifacts: required for signing new controller certs (rotation) -# - Controller cert/key: required to repopulate the overlay if jumphost is rebuilt -``` - ---- - -## 15. Post-deploy verification - -After `runbooks/02-deploy.md` completes (`juju deploy` with the overlay), -verify Octavia is healthy and the PKI plumbing works. - -```bash -# Octavia charm active/idle -juju status octavia -# Expect: octavia/0 active idle - -# Octavia services running -juju ssh octavia/0 -- sudo systemctl is-active octavia-api octavia-worker octavia-housekeeping -# Expect: 3x "active" - -# Confirm PKI files landed on the unit -juju ssh octavia/0 -- sudo ls -la /etc/octavia/certs/ -# Expect: server_ca.cert.pem, server_ca.key.pem, client_ca.cert.pem, client.cert-and-key.pem -# (filenames are charm-controlled; presence is what matters) - -# Confirm Octavia can use them — verbose health-check from the API -juju ssh octavia/0 -- sudo journalctl -u octavia-api --since "5 minutes ago" \ - | grep -iE "(cert|ssl|tls|amphora)" | head -20 -# Expect: no errors related to cert loading -``` - -**Smoketest — create a test LB once amphora image is available:** - -```bash -# After `octavia-diskimage-retrofit` has populated Glance with the amphora image, -# and the LBaaS Mgmt network is wired (these are downstream runbook steps), -# a test LB creation exercises the full PKI chain: - -source ~/admin-openrc -openstack loadbalancer create --name pki-smoketest --vip-subnet-id - -# Watch for amphora spawn (3-5 minutes typical) -watch -n5 'openstack loadbalancer show pki-smoketest' -# Wait for: provisioning_status=ACTIVE, operating_status=ONLINE - -# Octavia-worker log should show successful amphora handshake (signed by Issuing CA, -# trusted via Controller CA): -juju ssh octavia/0 -- sudo journalctl -u octavia-worker --since "10 minutes ago" \ - | grep -iE "(amphora|cert)" | tail -20 -# Expect: "amphora connection established" or similar -# Expect: no TLS handshake errors, no cert validation errors - -# Cleanup the smoketest LB -openstack loadbalancer delete pki-smoketest --cascade -``` - -If amphora handshake fails with cert errors, the most likely causes are: - -1. SAN mismatch — the controller's connection to amphora uses the cert's CN/SAN; - verify the controller cert SAN covers all addresses Octavia uses to reach amphorae. -2. Bundle/key mismatch — `lb-mgmt-controller-cert` bundle should contain BOTH the - cert and the matching private key; if they're for different keys, handshake fails. -3. Encrypted Issuing CA key + wrong passphrase — verify the passphrase string in - the overlay matches what was used at generation. - ---- - -## 16. Roosevelt deltas (forward-look) - -When this runbook is adapted for Roosevelt bare-metal deploy: - -| Aspect | Testcloud (v1) | Roosevelt | -|---|---|---| -| Issuing CA root | Self-signed | Intermediate signed by Vault root CA | -| CA storage | Filesystem on jumphost | Vault PKI engine, encrypted at rest | -| Controller cert validity | 2 years | 90 days | -| Rotation | Manual (this runbook re-run) | Automated via Vault + cron + bundle redeploy | -| Backup | gpg tarball, off-host | Vault's own backup mechanism | -| Amphora image signing | Out of scope for v1 | Image signed by Vault PKI as well | -| Procedure file | `runbooks/01a-octavia-pki-generation.md` | New runbook in Roosevelt repo | - -The procedure structure (generate Issuing CA → Controller CA → Controller cert → -encode → overlay → backup → deploy) remains identical. Roosevelt just sources -the CA root from Vault instead of self-signing. - ---- - -## 17. Rotation/renewal pointer - -For testcloud, the 2-year controller cert and 10-year CAs are intentionally -"set and forget" — they will outlive the cloud at this scale. - -If rotation IS needed before testcloud teardown (e.g., a key leak event), the -re-run procedure is: - -1. Generate new Controller cert signed by **existing** Controller CA (re-run - sections 8-9 only). -2. Regenerate the overlay (section 11) with the new Controller cert; leave all - other values unchanged. -3. `juju config octavia lb-mgmt-controller-cert=` (single-option - update; does not require full bundle redeploy). -4. Octavia services may need a restart: `juju ssh octavia/0 -- sudo systemctl restart octavia-api octavia-worker octavia-housekeeping`. -5. Existing amphorae will need to reconnect using the new cert; in-flight LBs - may briefly drop. This is acceptable for a security-event rotation. - -For Roosevelt, this whole procedure is replaced by Vault automated rotation — -see Roosevelt runbook (TBD). - ---- - -## 18. Change log - -| Date | Change | Reference | -|---|---|---| -| 2026-05-22 | Document created. Fresh-generate, EC P-384 CAs, EC P-256 controller cert, overlay-file distribution. | Workstream 3a | diff --git a/runbooks/02-deploy.md b/runbooks/02-deploy.md deleted file mode 100644 index 4a52845..0000000 --- a/runbooks/02-deploy.md +++ /dev/null @@ -1,23 +0,0 @@ -# Runbook 02 — Deploy New Caracal Bundle - -**STATUS: PLACEHOLDER** — drafted alongside bundle.yaml. - -## Purpose - -Deploy the new Charmed OpenStack Caracal bundle and wait for the cloud to -settle in `active/idle`. - -## Prerequisites - -- Runbook 01 complete (model destroyed, MAAS state clean) -- `bundle.yaml` and `overlays/vr0-dc0-testcloud.yaml` drafted and reviewed -- `scripts/pre-flight-checks.sh` passes - -## TODO - -- [ ] `juju add-model openstack` -- [ ] `juju deploy ./bundle.yaml --overlay overlays/vr0-dc0-testcloud.yaml --trust` -- [ ] Wait for settle (`juju-wait` or `juju status --watch 30s`) -- [ ] Pause-points for Vault init (per Runbook 03) -- [ ] Acceptance: all charms `active/idle` modulo Vault (sealed) and any - charms waiting on Vault certificates diff --git a/runbooks/03-vault-init.md b/runbooks/03-vault-init.md deleted file mode 100644 index 40db379..0000000 --- a/runbooks/03-vault-init.md +++ /dev/null @@ -1,24 +0,0 @@ -# Runbook 03 — Vault Initialization - -**STATUS: PLACEHOLDER** — drafted during deploy phase. - -## Purpose - -Initialize the Vault instance(s), unseal, authorize, and let certificate -relations resolve so dependent charms reach `active/idle`. - -## Prerequisites - -- Bundle deployed; Vault charm in `blocked` waiting for init -- etcd cluster in `active/idle` (Vault HA backend per D-006) -- easyrsa active (TLS bootstrap) - -## TODO - -- [ ] `juju run vault/leader generate-root-ca` — capture root CA cert -- [ ] `vault operator init -key-shares=5 -key-threshold=3` — capture keys -- [ ] Unseal with 3 of 5 keys -- [ ] `juju run vault/leader authorize-charm token=` -- [ ] Verify all `:certificates` relations complete (no charms stuck - waiting on certs) -- [ ] Store unseal keys in `~/.vault-keys/` (chmod 600); back up diff --git a/runbooks/04-magnum-domain.md b/runbooks/04-magnum-domain.md deleted file mode 100644 index b400a18..0000000 --- a/runbooks/04-magnum-domain.md +++ /dev/null @@ -1,21 +0,0 @@ -# Runbook 04 — Magnum Keystone Domain Setup - -**STATUS: PLACEHOLDER** — drafted post-deploy. - -## Purpose - -Run the magnum charm's `domain-setup` action to create the Keystone domain, -trust role, and service user that Magnum requires for cluster operations. - -## Prerequisites - -- Magnum charm reached `active/idle` post Vault init -- Keystone reachable from jumphost via FQDN - -## TODO - -- [ ] `juju run magnum/leader domain-setup --wait=10m` -- [ ] Verify creation in Keystone: - `openstack domain show magnum` - `openstack user show magnum_domain_admin --domain magnum` -- [ ] Acceptance: domain present, trust role assigned, charm in active/idle diff --git a/runbooks/04a-capi-bootstrap-cluster.md b/runbooks/04a-capi-bootstrap-cluster.md deleted file mode 100644 index d98f20c..0000000 --- a/runbooks/04a-capi-bootstrap-cluster.md +++ /dev/null @@ -1,1056 +0,0 @@ -# Runbook 04a — CAPI bootstrap cluster - -**Status:** Executes after `02-deploy.md` (cloud up + all charms active/idle) -and `03-vault-init.md` (Vault initialized + root CA available). Precedes -`05-magnum-capi-driver.md` (driver graft consumes the workload kubeconfig -produced here). - -**D-017 posture:** L3 full teardown and rebuild every deployment cycle. -Nothing is preserved across cycles. capi-mgmt is wiped to MAAS Ready on -teardown; rebuilt from scratch by this runbook. - -**Cross-references:** -- D-017 (CAPI bootstrap cluster lifecycle) -- D-007 (Magnum two-layer install) -- D-002 (channel matrix — informs Vault CA chain) -- Workstream 3b decision (2026-05-22): ship Vault CA (no tls-insecure); pivot mandatory - ---- - -## 1. Purpose & scope - -This runbook stands up the CAPI bootstrap cluster on `capi-mgmt.maas` and -pivots cluster state into a self-managing workload cluster. Output: - -1. **Workload K8s cluster** (`capi-mgmt-cluster`) running in tenant VMs on - the cloud, self-managing post-pivot. -2. **Workload kubeconfig** copied to jumphost at a known path. Consumed by - `runbooks/05-magnum-capi-driver.md` for the Magnum CAPI Helm driver - graft. -3. **No remaining state** on the bootstrap k3s VM after pivot. capi-mgmt - becomes a disposable jump host. - -**Scope:** v1 testcloud. Roosevelt deltas in section 20. - -**Out of scope:** - -- Magnum-side configuration (runbook 05). -- Workload cluster's tenant lifecycle (Magnum's job, not this runbook's). -- Backup / DR for the workload cluster (Roosevelt concern). - ---- - -## 2. Decisions captured - -Per workstream 3b sign-off (2026-05-22): - -| Decision | Choice | Roosevelt parallel | -|---|---|---| -| Version pinning | Pin-at-execution with discovery in §4 | Same pattern; pins captured in deploy record | -| Cloud TLS trust | Ship Vault CA to capi-mgmt + workload nodes (no `tls-insecure`) | Image-baked CA; CK8sConfig redundancy | -| `clusterctl move` pivot | Mandatory; workload cluster becomes self-managing | Same | -| K8s flavor | Canonical Kubernetes (CK8s) | Same | -| OpenStack auth | v3applicationcredential | Same | -| Pod CIDR | `10.244.0.0/16` | Same (does not conflict with cloud `10.12.0.0/16` or tenant pool `10.20.0.0/16`) | -| Service CIDR | `10.96.0.0/12` | Same | -| Workload cluster name | `capi-mgmt-cluster` | Same | -| Workload node SSH user | `ubuntu` (MAAS/cloud-init convention) | Same | - -**Naming convention:** - -- Keystone project for CAPI: `capi-mgmt` (in `admin_domain`) -- Keystone user for CAPI: `capo` (CAPO operator) -- App credential: `capo-app-cred` -- Workload image (Glance): `noble-amd64` (existing; do NOT duplicate as `ubuntu-24.04-capi` — Bobcat lesson) -- Workload flavor: `capi-mgmt-node` (4 vCPU / 4 GiB / 30 GB) — control plane node sizing - ---- - -## 3. Prerequisites - -| Prereq | Verification | -|---|---| -| Cloud deployed; all charms `active/idle` per D-011 | `juju status --color\| grep -v "active.*idle"` returns only the header | -| Vault initialized + unsealed | `juju ssh vault/leader -- sudo vault status` shows `Sealed=false` | -| Vault root CA available on jumphost | `test -f $HOME/vault-pki/root-ca.pem && openssl x509 -in $HOME/vault-pki/root-ca.pem -noout -subject` | -| Keystone reachable via FQDN | `curl -sf --cacert $HOME/vault-pki/root-ca.pem https://keystone.omega.dc0.vr0.cloud.neumatrix.local:5000/v3 \| jq .version.id` returns `"v3.14"` or current | -| capi-mgmt VM exists in MAAS as Ready | `maas $MAAS_PROFILE machines read \| jq '.[] \| select(.hostname=="capi-mgmt") \| .status_name'` returns `"Ready"` | -| Admin openrc available | `test -f $HOME/admin-openrc && source $HOME/admin-openrc && openstack token issue \| head -3` | -| Workspace path under $HOME (snap confinement) | `WORK=$HOME/capi-bootstrap; mkdir -p "$WORK"; cd "$WORK"; pwd` shows under home | - -**Set shell context for the runbook:** - -```bash -export REPO=$HOME/repos/openstack-caracal-ipv4 # adjust if your clone is elsewhere -export WORK=$HOME/capi-bootstrap # runbook scratch dir -export VAULT_CA=$HOME/vault-pki/root-ca.pem # Vault root CA (from runbook 03) -export CAPI_MGMT_METAL_IP=10.12.8.21 # capi-mgmt metal interface -export CAPI_MGMT_PROVIDER_IP=10.12.4.21 # capi-mgmt provider interface -export CLUSTER_NAME=capi-mgmt-cluster -mkdir -p "$WORK" -cd "$WORK" -``` - ---- - -## 4. Version discovery (set pins) - -Bobcat ran "dynamic latest." This runbook pins explicit versions captured at -execution time, with the discovery procedure documented inline so each -rebuild's pins are reproducible AND traceable. - -**GitHub API: authenticated vs unauthenticated.** Unauth has 60 req/hr; -authenticated has 5000. For multiple rebuilds in a day, set a token: - -```bash -# Optional but recommended — avoids rate-limit headaches during rebuild -export GITHUB_TOKEN= -# Or skip if you can tolerate ~10 API calls slowly -``` - -**Discover current stable releases:** - -```bash -cd "$WORK" - -# Helper: fetch latest stable release tag from a GitHub repo -gh_latest() { - local repo=$1 - local auth="" - [ -n "$GITHUB_TOKEN" ] && auth="-H Authorization: Bearer $GITHUB_TOKEN" - curl -sfL $auth "https://api.github.com/repos/$repo/releases/latest" \ - | jq -r '.tag_name' -} - -# Pin captures (one file per pin, for the deploy-record convention) -mkdir -p pins -gh_latest "kubernetes-sigs/cluster-api" | tee pins/CAPI_VERSION -gh_latest "kubernetes-sigs/cluster-api-provider-openstack" | tee pins/CAPO_VERSION -gh_latest "canonical/cluster-api-k8s" | tee pins/CK8S_VERSION -gh_latest "cert-manager/cert-manager" | tee pins/CERT_MANAGER_VERSION -gh_latest "k-orc/openstack-resource-controller" | tee pins/ORC_VERSION -gh_latest "k3s-io/k3s" | tee pins/K3S_VERSION -gh_latest "helm/helm" | tee pins/HELM_VERSION - -# Load into shell -export CAPI_VERSION=$(cat pins/CAPI_VERSION) -export CAPO_VERSION=$(cat pins/CAPO_VERSION) -export CK8S_VERSION=$(cat pins/CK8S_VERSION) -export CERT_MANAGER_VERSION=$(cat pins/CERT_MANAGER_VERSION) -export ORC_VERSION=$(cat pins/ORC_VERSION) -export K3S_VERSION=$(cat pins/K3S_VERSION) -export HELM_VERSION=$(cat pins/HELM_VERSION) - -# Display for the deploy log -cat pins/*_VERSION | paste -d= <(ls pins/) - -``` - -**Sanity check:** all values should look like `v1.X.Y` or `v0.X.Y`. If any -returned `null` or empty, the GitHub API call failed — most likely -rate-limited. Wait an hour or set `$GITHUB_TOKEN` and retry. - -**Capture pins to repo as deploy record:** - -The pin files in `$WORK/pins/` should be appended to a deploy-log artifact -(NOT committed to the repo — these are deploy-time captures). Suggested -location: `$HOME/deploy-records/$(date +%Y%m%d-%H%M)/capi-pins/`. - -```bash -DEPLOY_RECORD=$HOME/deploy-records/$(date +%Y%m%d-%H%M%S)/capi-pins -mkdir -p "$DEPLOY_RECORD" -cp pins/*_VERSION "$DEPLOY_RECORD/" -ls -la "$DEPLOY_RECORD/" -``` - ---- - -## 5. MAAS-deploy capi-mgmt - -Prerequisite: capi-mgmt MAAS machine is in `Ready` state (see §3). -Network config in MAAS: - -- **eth0** on metal fabric, DHCP → `10.12.8.21` (MAAS-pinned static lease) -- **eth1** on provider fabric, static → `10.12.4.21` - -Deploy Ubuntu 24.04 (Noble): - -```bash -# Get the capi-mgmt system_id from MAAS -CAPI_MGMT_SYSTEM_ID=$(maas $MAAS_PROFILE machines read \ - | jq -r '.[] | select(.hostname=="capi-mgmt") | .system_id') -echo "capi-mgmt system_id: $CAPI_MGMT_SYSTEM_ID" - -# Deploy -maas $MAAS_PROFILE machine deploy "$CAPI_MGMT_SYSTEM_ID" \ - distro_series=noble \ - hwe_kernel=ga-24.04 -``` - -Poll for `Deployed`: - -```bash -while true; do - STATUS=$(maas $MAAS_PROFILE machine read "$CAPI_MGMT_SYSTEM_ID" \ - | jq -r '.status_name') - echo "$(date -Is) capi-mgmt status: $STATUS" - [ "$STATUS" = "Deployed" ] && break - [ "$STATUS" = "Failed deployment" ] && { echo "FAILED"; exit 1; } - sleep 30 -done -``` - -Typical deploy time: 5-8 minutes on this hardware. - -**SSH reachability:** - -```bash -# MAAS .maas zone may not resolve from jumphost — use IP directly per handoff lessons -ssh -o StrictHostKeyChecking=accept-new ubuntu@$CAPI_MGMT_METAL_IP -- hostname -# Expect: capi-mgmt -``` - -> **Gotcha:** MAAS-deployed Ubuntu uses the `ubuntu` user, not `jessea123`. -> See handoff "recurring technical pitfalls." - ---- - -## 6. SSH bootstrap + Vault CA install - -On the jumphost, prepare a transport bundle of essentials: - -```bash -mkdir -p "$WORK/bootstrap-bundle" -cp "$VAULT_CA" "$WORK/bootstrap-bundle/vault-ca.crt" -chmod 644 "$WORK/bootstrap-bundle/vault-ca.crt" - -# Bundle pin files so capi-mgmt can read versions -cp -r "$WORK/pins" "$WORK/bootstrap-bundle/" -``` - -SCP and install Vault CA on capi-mgmt: - -```bash -scp -r "$WORK/bootstrap-bundle" ubuntu@$CAPI_MGMT_METAL_IP:/home/ubuntu/ - -ssh ubuntu@$CAPI_MGMT_METAL_IP <<'EOF' -set -euo pipefail - -# Install Vault CA as a system-trusted root -sudo cp /home/ubuntu/bootstrap-bundle/vault-ca.crt /usr/local/share/ca-certificates/ -sudo update-ca-certificates 2>&1 | tail -3 - -# Verify -openssl s_client -connect keystone.omega.dc0.vr0.cloud.neumatrix.local:5000 \ - -CApath /etc/ssl/certs -verify_return_error &1 \ - | grep -E "(Verify return code|subject=)" || \ - { echo "TLS chain verify failed against Keystone — investigate before proceeding"; exit 1; } - -# Update apt + base utilities -sudo apt-get update -qq -sudo apt-get install -y -qq jq curl yq - -# Confirm -which jq curl yq -EOF -``` - -**Expected:** - -- `update-ca-certificates` reports "1 added" -- `openssl s_client` shows `Verify return code: 0 (ok)` and a Keystone cert - whose chain terminates at the Vault CA - -> **Why this matters:** Bobcat used `tls-insecure=true` in cloud.conf which -> skipped this entire trust path. Our workstream 3b decision (ship Vault CA) -> means OCCM and CAPO will validate certs against this trust store. If TLS -> verify fails here, OCCM will crashloop later. - ---- - -## 7. k3s install - -On capi-mgmt: - -```bash -ssh ubuntu@$CAPI_MGMT_METAL_IP "K3S_VERSION=$K3S_VERSION CAPI_MGMT_METAL_IP=$CAPI_MGMT_METAL_IP bash -s" <<'REMOTE_EOF' -set -euo pipefail - -# Install k3s with explicit bind/advertise/SAN flags -curl -sfL https://get.k3s.io | \ - INSTALL_K3S_VERSION="$K3S_VERSION" \ - sh -s - server \ - --bind-address="$CAPI_MGMT_METAL_IP" \ - --advertise-address="$CAPI_MGMT_METAL_IP" \ - --node-ip="$CAPI_MGMT_METAL_IP" \ - --tls-san="$CAPI_MGMT_METAL_IP" \ - --tls-san=capi-mgmt.maas \ - --write-kubeconfig-mode=0644 \ - --disable=traefik - -# Wait for k3s API to respond -for i in $(seq 1 30); do - if sudo kubectl get nodes 2>/dev/null | grep -q "Ready"; then - echo "k3s ready"; break - fi - echo "Waiting for k3s API... ($i/30)" - sleep 5 -done - -sudo kubectl get nodes -sudo kubectl get pods -A -REMOTE_EOF -``` - -> **Gotcha:** `--bind-address=$IP` makes k3s listen ONLY on that IP — not -> also on 127.0.0.1. The default kubeconfig at -> `/etc/rancher/k3s/k3s.yaml` has `server: https://127.0.0.1:6443` and will -> NOT work as-is. Sed-rewrite below. - ---- - -## 8. Kubeconfig server-URL rewrite - -```bash -ssh ubuntu@$CAPI_MGMT_METAL_IP "CAPI_MGMT_METAL_IP=$CAPI_MGMT_METAL_IP bash -s" <<'REMOTE_EOF' -set -euo pipefail - -# Copy k3s kubeconfig to ubuntu user; rewrite server URL -mkdir -p /home/ubuntu/.kube -sudo cp /etc/rancher/k3s/k3s.yaml /home/ubuntu/.kube/config -sudo chown ubuntu:ubuntu /home/ubuntu/.kube/config -chmod 600 /home/ubuntu/.kube/config - -# Rewrite 127.0.0.1 → metal IP -sed -i "s|server: https://127.0.0.1:6443|server: https://$CAPI_MGMT_METAL_IP:6443|" \ - /home/ubuntu/.kube/config - -# Verify rewrite -grep "server:" /home/ubuntu/.kube/config -# Expect: server: https://10.12.8.21:6443 - -# Confirm kubectl works as ubuntu user (no sudo) -kubectl get nodes -REMOTE_EOF -``` - ---- - -## 9. helm + clusterctl install - -```bash -ssh ubuntu@$CAPI_MGMT_METAL_IP "HELM_VERSION=$HELM_VERSION CAPI_VERSION=$CAPI_VERSION bash -s" <<'REMOTE_EOF' -set -euo pipefail - -# helm install (get-helm-3 fetches the version we specify) -cd /tmp -curl -sfL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 \ - | DESIRED_VERSION="$HELM_VERSION" bash -helm version --short - -# clusterctl install -CLUSTERCTL_URL="https://github.com/kubernetes-sigs/cluster-api/releases/download/${CAPI_VERSION}/clusterctl-linux-amd64" -sudo curl -sfL "$CLUSTERCTL_URL" -o /usr/local/bin/clusterctl -sudo chmod +x /usr/local/bin/clusterctl -clusterctl version -REMOTE_EOF -``` - ---- - -## 10. clusterctl init (CAPI controllers + cert-manager + ORC + CAPO + CK8s) - -```bash -ssh ubuntu@$CAPI_MGMT_METAL_IP "CK8S_VERSION=$CK8S_VERSION CERT_MANAGER_VERSION=$CERT_MANAGER_VERSION ORC_VERSION=$ORC_VERSION CAPO_VERSION=$CAPO_VERSION bash -s" <<'REMOTE_EOF' -set -euo pipefail - -# Configure clusterctl with provider URLs -mkdir -p ~/.cluster-api -cat > ~/.cluster-api/clusterctl.yaml </dev/null || true -kubectl wait --for=condition=Available --timeout=5m \ - deployment --all -n capo-system -kubectl wait --for=condition=Available --timeout=5m \ - deployment --all -n cert-manager - -# Install ORC -kubectl apply -f "https://github.com/k-orc/openstack-resource-controller/releases/${ORC_VERSION}/orc.yaml" -kubectl wait --for=condition=Available --timeout=5m \ - deployment --all -n orc-system - -# Confirm all controllers -kubectl get pods -A | grep -v "Running\|Completed" | grep -v NAME -# Expected: empty output (all pods Running or no abnormal state) -REMOTE_EOF -``` - -> **Gotcha:** the actual namespace names (`capi-system`, `capo-system`, etc.) -> are conventions. If a controller fails to land in the expected namespace, -> `kubectl get deployment -A` lists all deployments — diagnose from there. - ---- - -## 11. Cloud-side prep (Keystone, Nova, Glance) - -Back on the jumphost: - -```bash -source $HOME/admin-openrc - -# Inventory existing resources FIRST (Bobcat lesson: don't create duplicates) -echo "=== Existing images ===" -openstack image list -c ID -c Name -f json | jq -r '.[] | "\(.Name)\t\(.ID)"' -echo "" -echo "=== Existing flavors ===" -openstack flavor list -c Name -c ID -c RAM -c VCPUs -c Disk -f json \ - | jq -r '.[] | "\(.Name)\tRAM=\(.RAM)\tCPU=\(.VCPUs)\tDisk=\(.Disk)\tID=\(.ID)"' -echo "" -echo "=== Existing keypairs ===" -openstack keypair list -echo "" -echo "=== Existing projects in admin_domain ===" -openstack project list --domain admin_domain -``` - -**Create / verify resources:** - -```bash -# Keystone project + user -openstack project show capi-mgmt --domain admin_domain 2>/dev/null \ - || openstack project create capi-mgmt --domain admin_domain --description "CAPI management plane" - -openstack user show capo --domain admin_domain 2>/dev/null \ - || openstack user create capo --domain admin_domain --password-prompt --description "CAPO operator" - -# Role assignments (CAPO needs member + load-balancer_member at minimum; -# admin works for testcloud — Roosevelt should use least-privilege) -openstack role add --user capo --user-domain admin_domain \ - --project capi-mgmt --project-domain admin_domain \ - member - -openstack role add --user capo --user-domain admin_domain \ - --project capi-mgmt --project-domain admin_domain \ - load-balancer_member 2>/dev/null || \ - echo "(load-balancer_member role may not exist if Octavia not deployed yet)" - -# Application credential — captured to file under $HOME (snap confinement) -APP_CRED_FILE=$WORK/capo-app-cred.json -openstack --os-username capo --os-user-domain-name admin_domain \ - --os-project-name capi-mgmt --os-project-domain-name admin_domain \ - application credential create capo-app-cred \ - --description "CAPO operator app credential" \ - -f json > "$APP_CRED_FILE" -chmod 600 "$APP_CRED_FILE" - -# Extract credential ID + secret -export APP_CRED_ID=$(jq -r '.id' "$APP_CRED_FILE") -export APP_CRED_SECRET=$(jq -r '.secret' "$APP_CRED_FILE") -echo "App cred ID: $APP_CRED_ID" -``` - -**Nova keypair (workload node SSH key):** - -```bash -# Generate fresh keypair locally (do NOT reuse jumphost personal key) -ssh-keygen -t ed25519 -N '' -f "$WORK/capi-workload-key" \ - -C "capi-workload-$(date +%Y%m%d)" -chmod 600 "$WORK/capi-workload-key" - -# Upload public key to Keystone as a Nova keypair -openstack keypair create --public-key "$WORK/capi-workload-key.pub" capi-workload-key -openstack keypair show capi-workload-key -``` - -**Workload image:** - -```bash -# Inventory check — use noble-amd64 if it exists (Bobcat lesson: do NOT create ubuntu-24.04-capi as a dup) -NOBLE_IMAGE_ID=$(openstack image show noble-amd64 -c id -f value 2>/dev/null || echo "") - -if [ -z "$NOBLE_IMAGE_ID" ]; then - echo "noble-amd64 image not found — upload required." - echo "(Pull from https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img" - echo " then: openstack image create --disk-format qcow2 --container-format bare \\" - echo " --public --file noble-server-cloudimg-amd64.img noble-amd64)" - exit 1 -fi -echo "Using image: noble-amd64 ($NOBLE_IMAGE_ID)" -export WORKLOAD_IMAGE_ID=$NOBLE_IMAGE_ID -``` - -**Workload flavor:** - -```bash -openstack flavor show capi-mgmt-node 2>/dev/null \ - || openstack flavor create capi-mgmt-node \ - --vcpus 4 --ram 4096 --disk 30 \ - --description "CAPI workload node (control plane sizing)" - -export WORKLOAD_FLAVOR=capi-mgmt-node -``` - ---- - -## 12. clouds.yaml + cloud.conf composition (with Vault CA, no tls-insecure) - -The workload cluster's OCCM (OpenStack Cloud Controller Manager) and CAPO both -need to call OpenStack APIs. Two files: - -- `clouds.yaml` — CAPO's view of how to reach OpenStack (used at cluster - creation time on capi-mgmt) -- `cloud.conf` — OCCM's view, injected into the workload cluster's k8s - Secret (used continuously by OCCM running in the workload cluster) - -**Compose clouds.yaml:** - -```bash -cat > "$WORK/clouds.yaml" < "$WORK/clouds.yaml.b64" -``` - -**Compose cloud.conf** (INI format, NOT YAML): - -```bash -cat > "$WORK/cloud.conf" < "$WORK/cloud.conf.b64" -``` - -> **Critical delta from Bobcat:** the `ca-file` line replaces `tls-insecure=true`. -> The path `/usr/local/share/ca-certificates/vault-ca.crt` exists on capi-mgmt -> (from §6) AND will be injected into workload nodes via CK8sConfig in §13. - -**base64-encode Vault CA for CK8sConfig injection:** - -```bash -base64 -w0 "$VAULT_CA" > "$WORK/vault-ca.crt.b64" -wc -c "$WORK/vault-ca.crt.b64" -``` - ---- - -## 13. Cluster template rendering (with Vault CA injection) - -The cluster template defines: - -- Cluster object -- OpenStackCluster (CAPO infrastructure) -- CK8sControlPlane -- CK8sConfigTemplate (control plane bootstrap — includes Vault CA injection) -- MachineDeployment + CK8sConfigTemplate (workers — includes Vault CA injection) -- Secrets for clouds.yaml and cloud.conf - -Variables (18 total): - -```bash -export CLUSTER_NAME=capi-mgmt-cluster -export CLUSTER_NAMESPACE=default -export KUBERNETES_VERSION=v1.31.4 # adjust to CK8s-supported -export CONTROL_PLANE_MACHINE_COUNT=1 # 3 for HA on Roosevelt -export WORKER_MACHINE_COUNT=2 # 3 on Roosevelt -export OPENSTACK_DNS_NAMESERVERS=10.12.4.227 # designate VIP -export OPENSTACK_FAILURE_DOMAIN=nova -export OPENSTACK_EXTERNAL_NETWORK_ID=$(openstack network show ext_net -c id -f value) -export OPENSTACK_IMAGE_NAME=noble-amd64 -export OPENSTACK_FLAVOR=capi-mgmt-node -export OPENSTACK_SSH_KEY_NAME=capi-workload-key -export POD_CIDR=10.244.0.0/16 -export SERVICE_CIDR=10.96.0.0/12 -export CLOUDS_YAML_B64=$(cat "$WORK/clouds.yaml.b64") -export CLOUD_CONF_B64=$(cat "$WORK/cloud.conf.b64") -export VAULT_CA_B64=$(cat "$WORK/vault-ca.crt.b64") -export CLUSTER_DOMAIN=cluster.local -export OPENSTACK_CLOUD=capi-mgmt - -# Sanity print -env | grep -E "^(CLUSTER|KUBERNETES|CONTROL_PLANE|WORKER|OPENSTACK|POD|SERVICE|VAULT|CLOUD)" \ - | grep -v "B64\|SECRET\|PASS" | sort -``` - -**Render the cluster template:** - -```bash -cat > "$WORK/cluster-template.yaml" <<'TEMPLATE_EOF' -apiVersion: v1 -kind: Secret -metadata: - name: ${CLUSTER_NAME}-cloud-config - namespace: ${CLUSTER_NAMESPACE} -type: Opaque -data: - clouds.yaml: ${CLOUDS_YAML_B64} - cloud.conf: ${CLOUD_CONF_B64} - cacert: ${VAULT_CA_B64} ---- -apiVersion: cluster.x-k8s.io/v1beta1 -kind: Cluster -metadata: - name: ${CLUSTER_NAME} - namespace: ${CLUSTER_NAMESPACE} -spec: - clusterNetwork: - pods: - cidrBlocks: - - ${POD_CIDR} - services: - cidrBlocks: - - ${SERVICE_CIDR} - serviceDomain: ${CLUSTER_DOMAIN} - infrastructureRef: - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 - kind: OpenStackCluster - name: ${CLUSTER_NAME} - controlPlaneRef: - apiVersion: controlplane.cluster.x-k8s.io/v1beta2 - kind: CK8sControlPlane - name: ${CLUSTER_NAME}-control-plane ---- -apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 -kind: OpenStackCluster -metadata: - name: ${CLUSTER_NAME} - namespace: ${CLUSTER_NAMESPACE} -spec: - identityRef: - name: ${CLUSTER_NAME}-cloud-config - cloudName: ${OPENSTACK_CLOUD} - externalNetwork: - id: ${OPENSTACK_EXTERNAL_NETWORK_ID} - managedSecurityGroups: - allowAllInClusterTraffic: true - apiServerLoadBalancer: - enabled: true ---- -apiVersion: controlplane.cluster.x-k8s.io/v1beta2 -kind: CK8sControlPlane -metadata: - name: ${CLUSTER_NAME}-control-plane - namespace: ${CLUSTER_NAMESPACE} -spec: - replicas: ${CONTROL_PLANE_MACHINE_COUNT} - version: ${KUBERNETES_VERSION} - machineTemplate: - infrastructureTemplate: - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 - kind: OpenStackMachineTemplate - name: ${CLUSTER_NAME}-control-plane - spec: - files: - - path: /usr/local/share/ca-certificates/vault-ca.crt - owner: root:root - permissions: "0644" - contentFrom: - secret: - name: ${CLUSTER_NAME}-cloud-config - key: cacert - preRunCommands: - - update-ca-certificates - extraKubeAPIServerArgs: - "--cloud-provider": external ---- -apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 -kind: OpenStackMachineTemplate -metadata: - name: ${CLUSTER_NAME}-control-plane - namespace: ${CLUSTER_NAMESPACE} -spec: - template: - spec: - flavor: ${OPENSTACK_FLAVOR} - image: - filter: - name: ${OPENSTACK_IMAGE_NAME} - sshKeyName: ${OPENSTACK_SSH_KEY_NAME} - identityRef: - name: ${CLUSTER_NAME}-cloud-config - cloudName: ${OPENSTACK_CLOUD} ---- -apiVersion: cluster.x-k8s.io/v1beta1 -kind: MachineDeployment -metadata: - name: ${CLUSTER_NAME}-md-0 - namespace: ${CLUSTER_NAMESPACE} -spec: - clusterName: ${CLUSTER_NAME} - replicas: ${WORKER_MACHINE_COUNT} - selector: - matchLabels: {} - template: - spec: - clusterName: ${CLUSTER_NAME} - version: ${KUBERNETES_VERSION} - bootstrap: - configRef: - apiVersion: bootstrap.cluster.x-k8s.io/v1beta2 - kind: CK8sConfigTemplate - name: ${CLUSTER_NAME}-md-0 - infrastructureRef: - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 - kind: OpenStackMachineTemplate - name: ${CLUSTER_NAME}-md-0 ---- -apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 -kind: OpenStackMachineTemplate -metadata: - name: ${CLUSTER_NAME}-md-0 - namespace: ${CLUSTER_NAMESPACE} -spec: - template: - spec: - flavor: ${OPENSTACK_FLAVOR} - image: - filter: - name: ${OPENSTACK_IMAGE_NAME} - sshKeyName: ${OPENSTACK_SSH_KEY_NAME} - identityRef: - name: ${CLUSTER_NAME}-cloud-config - cloudName: ${OPENSTACK_CLOUD} ---- -apiVersion: bootstrap.cluster.x-k8s.io/v1beta2 -kind: CK8sConfigTemplate -metadata: - name: ${CLUSTER_NAME}-md-0 - namespace: ${CLUSTER_NAMESPACE} -spec: - template: - spec: - files: - - path: /usr/local/share/ca-certificates/vault-ca.crt - owner: root:root - permissions: "0644" - contentFrom: - secret: - name: ${CLUSTER_NAME}-cloud-config - key: cacert - preRunCommands: - - update-ca-certificates -TEMPLATE_EOF - -# envsubst to render -envsubst < "$WORK/cluster-template.yaml" > "$WORK/cluster-rendered.yaml" - -# Validate as YAML -python3 -c "import yaml; list(yaml.safe_load_all(open('$WORK/cluster-rendered.yaml'))); print('YAML OK')" - -# Quick visual check — no leftover ${...} markers -grep -n '\${' "$WORK/cluster-rendered.yaml" || echo "No unsubstituted variables — good" -``` - -> **CK8sConfig field name caveat:** the exact field names (`files`, -> `preRunCommands`) and their `contentFrom.secret` schema are CK8s-version- -> dependent. If `clusterctl init` failed earlier with schema warnings, -> consult the CK8s release notes for the pinned `$CK8S_VERSION`. - ---- - -## 14. Apply + poll-to-Ready - -Transfer rendered template to capi-mgmt and apply: - -```bash -scp "$WORK/cluster-rendered.yaml" ubuntu@$CAPI_MGMT_METAL_IP:/home/ubuntu/cluster.yaml - -ssh ubuntu@$CAPI_MGMT_METAL_IP <<'EOF' -set -euo pipefail -kubectl apply -f /home/ubuntu/cluster.yaml -echo "Applied. Waiting for cluster Available status (15-min timeout)..." - -for i in $(seq 1 90); do - STATUS=$(kubectl get cluster capi-mgmt-cluster -o json 2>/dev/null \ - | jq -r '.status.phase // "Unknown"') - READY=$(kubectl get cluster capi-mgmt-cluster -o json 2>/dev/null \ - | jq -r '.status.conditions[]? | select(.type=="Ready") | .status' \ - | head -1) - echo "$(date -Is) phase=$STATUS ready=$READY" - [ "$READY" = "True" ] && { echo "Cluster Ready"; break; } - sleep 10 -done - -kubectl get cluster,machines,kubeadmcontrolplane,machinedeployment -A -EOF -``` - -**If the poll times out before Ready,** typical diagnosis: - -```bash -ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl describe cluster capi-mgmt-cluster -ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl get machines -A -ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl logs -n capo-system deployment/capo-controller-manager --tail=100 -``` - -Common causes: - -- OpenStack API unreachable from capi-mgmt → check Vault CA install on capi-mgmt (§6) -- Image / flavor / network ID wrong in cluster template → re-check §11 variables -- Security group rules block kube-api LB → CAPO usually handles this; check OpenStackCluster status -- Application credential expired / wrong → re-check `$APP_CRED_ID` - ---- - -## 15. Extract workload kubeconfig - -```bash -ssh ubuntu@$CAPI_MGMT_METAL_IP -- clusterctl get kubeconfig capi-mgmt-cluster \ - > "$WORK/capi-mgmt-cluster.kubeconfig" -chmod 600 "$WORK/capi-mgmt-cluster.kubeconfig" - -# Sanity-check the workload cluster is reachable -kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get nodes -# Expect: 1 control plane + 2 workers, all Ready -``` - -If `get nodes` times out, the cluster's API LB may not have allocated its -external IP yet, or the firewall rules don't permit jumphost → workload API: - -```bash -# What IP is the cluster's API LB on? -ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl get openstackcluster capi-mgmt-cluster \ - -o json | jq '.status.externalNetwork, .status.controlPlaneEndpoint' - -# Test reachability -curl -sk --max-time 10 "https://:6443/version" && echo " ← reachable" || echo "API LB unreachable" -``` - ---- - -## 16. `clusterctl init` on target (workload cluster) - -The workload cluster must have the same CAPI providers installed before `move`. - -```bash -# Run from jumphost using the workload kubeconfig -KUBECONFIG="$WORK/capi-mgmt-cluster.kubeconfig" clusterctl init \ - --core "cluster-api:${CAPI_VERSION}" \ - --infrastructure "openstack:${CAPO_VERSION}" \ - --bootstrap "canonical-kubernetes:${CK8S_VERSION}" \ - --control-plane "canonical-kubernetes:${CK8S_VERSION}" \ - --cert-manager-version "${CERT_MANAGER_VERSION}" - -# ORC into workload cluster too -kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" apply \ - -f "https://github.com/k-orc/openstack-resource-controller/releases/${ORC_VERSION}/orc.yaml" - -# Wait for everything Available -kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" wait \ - --for=condition=Available --timeout=5m \ - deployment --all -n capi-system -kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" wait \ - --for=condition=Available --timeout=5m \ - deployment --all -n capo-system -kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" wait \ - --for=condition=Available --timeout=5m \ - deployment --all -n cert-manager -kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" wait \ - --for=condition=Available --timeout=5m \ - deployment --all -n orc-system -``` - -> **cert-manager double-install caveat:** if CK8s already installed -> cert-manager during workload bootstrap, the second `clusterctl init` may -> warn or skip. Check existing cert-manager version against `$CERT_MANAGER_VERSION` -> — if they differ, version-skew issues may surface post-pivot. Adjust the -> pin in §4 or accept the existing version. Roosevelt's standard practice -> is to install cert-manager via `clusterctl init` only (don't pre-install -> via CK8s) — same approach valid here if you want clean version control. - ---- - -## 17. `clusterctl move` pivot - -Move all CAPI CRs from bootstrap k3s → workload cluster: - -```bash -# Stage the target kubeconfig on capi-mgmt (where clusterctl move runs) -scp "$WORK/capi-mgmt-cluster.kubeconfig" ubuntu@$CAPI_MGMT_METAL_IP:/home/ubuntu/target.kubeconfig - -# Dry-run first to catch issues before commit -ssh ubuntu@$CAPI_MGMT_METAL_IP -- clusterctl move \ - --to-kubeconfig=/home/ubuntu/target.kubeconfig \ - --dry-run - -# Inspect dry-run output: list of objects to be moved. Should include: -# - Cluster, OpenStackCluster, OpenStackClusterTemplate -# - Secrets (cloud-config) -# - Machine objects, OpenStackMachineTemplate -# - CK8sControlPlane, CK8sConfigTemplate -# - MachineDeployment -# Should NOT include cert-manager state (cert-manager manages its own state -# on each cluster independently) -``` - -**If dry-run looks correct, execute the move:** - -```bash -ssh ubuntu@$CAPI_MGMT_METAL_IP -- clusterctl move \ - --to-kubeconfig=/home/ubuntu/target.kubeconfig - -# Move can take several minutes. Output ends with: "moved successfully" -``` - ---- - -## 18. Post-pivot verification - -```bash -echo "=== Bootstrap k3s (should now be empty of cluster CRs) ===" -ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl get cluster -A -# Expect: No resources found (or only a header) - -ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl get machines -A -# Expect: No resources found - -ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl get openstackcluster -A -# Expect: No resources found - -echo "" -echo "=== Workload cluster (should now own its own cluster CRs) ===" -kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get cluster -A -# Expect: capi-mgmt-cluster shown, phase=Provisioned - -kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get machines -A -# Expect: 3 machines (1 control-plane + 2 workers), all Running - -kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get openstackcluster -A - -echo "" -echo "=== CAPI controllers in workload ===" -kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get pods -A \ - | grep -E "(capi|capo|orc|cert-manager)" | grep -v "Running\|Completed" -# Expect: empty (all controller pods Running) - -echo "" -echo "=== OCCM not crash-looping (CRITICAL — main goal of TLS-verify work) ===" -kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get pods -n kube-system \ - -l k8s-app=openstack-cloud-controller-manager -# Expect: 1 pod Running, NOT CrashLoopBackOff - -kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" logs -n kube-system \ - -l k8s-app=openstack-cloud-controller-manager --tail=50 \ - | grep -iE "(tls|cert|error)" | head -20 -# Expect: no TLS/cert errors; OCCM should be healthy -``` - -> **If OCCM crash-loops with "x509: certificate signed by unknown authority":** -> Vault CA distribution failed. Check (a) `/usr/local/share/ca-certificates/vault-ca.crt` -> exists on workload nodes; (b) `update-ca-certificates` ran (check `/etc/ssl/certs/ca-certificates.crt` -> for the Vault CA's subject); (c) the secret reference in CK8sConfigTemplate -> matched the secret name. SSH into a worker via the jumphost key (`ssh -i -> $WORK/capi-workload-key ubuntu@`) to diagnose. - ---- - -## 19. Handoff to runbook 05 - -The workload kubeconfig at `$WORK/capi-mgmt-cluster.kubeconfig` is the input to -`runbooks/05-magnum-capi-driver.md`. Copy it to a stable path: - -```bash -mkdir -p $HOME/magnum-capi -cp "$WORK/capi-mgmt-cluster.kubeconfig" $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig -chmod 600 $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig -echo "Workload kubeconfig staged at: $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig" -``` - -> **Important — post-pivot semantic shift from Bobcat:** Magnum's -> `kubeconfig_file` setting (under `[capi_helm]` in -> `/etc/magnum/magnum.conf.d/99-capi.conf`, per D-007) now points to the -> workload cluster, not the bootstrap k3s. Bobcat had Magnum pointing at -> bootstrap k3s because the pivot was never executed. With pivot mandatory, -> Magnum's CAPI calls flow: -> -> ``` -> Magnum/leader → workload cluster API → CAPI controllers (running in workload) -> → create new Cluster CRs (tenant Magnum clusters) -> ``` -> -> The bootstrap k3s on capi-mgmt is now disposable. If you wanted, you could -> destroy capi-mgmt entirely at this point — the workload cluster manages -> itself. (Roosevelt may actually do this for cost savings.) For v1 testcloud, -> leave capi-mgmt running so its k3s can be inspected for diagnostics. - ---- - -## 20. Roosevelt deltas (forward-look) - -| Aspect | Testcloud (v1) | Roosevelt | -|---|---|---| -| Workload image | Default `noble-amd64` from cloud-images.ubuntu.com | Custom image baked with Vault CA pre-installed (no runtime install step) | -| Vault CA distribution | CK8sConfig `files:` + `preRunCommands:` (this runbook) | Image-baked + CK8sConfig (defense in depth) | -| App credential lifetime | No expiry set (testcloud) | Short-lived rotating credentials via Vault auth method | -| Workload cluster control plane | 1 node | 3 nodes (HA) | -| Workload cluster workers | 2 nodes | Per-tenant sizing; HPA-driven | -| `clusterctl init --cert-manager-version` | Pin from §4 | Pin to Vault PKI cert-manager profile (separate Roosevelt prep) | -| capi-mgmt VM lifecycle post-pivot | Kept running for diagnostics | Destroyed (cost savings; pivot makes it disposable) | -| Version pinning record | `$HOME/deploy-records//capi-pins/` | Same pattern, captured in Vault as audit artifact | -| Authentication to GitHub API | Optional PAT | Mandatory PAT (avoid rate-limit during automated rebuilds) | - ---- - -## 21. Rotation/refresh of pins - -The pins captured in §4 will age. Recommended cadence: - -- **Per rebuild:** re-discover all pins (Step 1 of next execution will catch - natural drift). -- **Out-of-band patch:** if a CVE drops for any pinned component, run §4 - discovery alone and capture the new pin into `$DEPLOY_RECORD/`. Then for - the affected component only, follow the upgrade procedure from its - upstream docs (does NOT necessarily require this whole runbook re-run). - -For Roosevelt, this becomes a tracked maintenance window task. - ---- - -## 22. Change log - -| Date | Change | Reference | -|---|---|---| -| 2026-05-22 | Document created. Vault CA distribution (no tls-insecure), mandatory `clusterctl move` pivot, pin-at-execution version model. | Workstream 3b | diff --git a/runbooks/05-magnum-capi-driver.md b/runbooks/05-magnum-capi-driver.md deleted file mode 100644 index e1414e1..0000000 --- a/runbooks/05-magnum-capi-driver.md +++ /dev/null @@ -1,529 +0,0 @@ -# Runbook 05 — Magnum CAPI Helm driver install - -**Status:** Executes after `04-magnum-domain.md` (Keystone wiring) and -`04a-capi-bootstrap-cluster.md` (workload cluster + kubeconfig staged). -Final post-deploy step to make Magnum capable of creating CAPI-managed -tenant K8s clusters. - -**Cross-references:** -- D-007 Layer B (Magnum two-layer install) -- D-017 (CAPI bootstrap cluster lifecycle) -- Runbook 04a §19 (workload kubeconfig handoff) -- Workstream 3c decision (2026-05-22): magnum-capi-helm 1.1.0 from PyPI; workload-cluster kubeconfig (NOT bootstrap k3s) - -**Known doc inconsistency (tracked for cleanup):** -D-007's Layer B currently states the kubeconfig points at "capi-mgmt.maas -bootstrap k3s". That language is correct for Bobcat (no pivot) but obsolete -post-workstream-3b (pivot mandatory). This runbook uses the workload cluster -kubeconfig as the canonical target. D-007 patch to follow in a workstream-3 -cleanup commit. - ---- - -## 1. Purpose & scope - -Graft the CAPI Helm driver onto the Charmed Magnum deployment so that -`openstack coe cluster create` provisions tenant K8s clusters via CAPI (in -the workload cluster) instead of via the deprecated Heat driver. - -**Output of this runbook:** - -- `magnum-capi-helm==1.1.0` installed on the magnum unit's system Python. -- `/etc/magnum/kubeconfig` populated with the workload cluster's - kubeconfig (post-pivot CAPI controller plane). -- `/etc/magnum/magnum.conf.d/99-capi.conf` configured with - `enabled_drivers = k8s_capi_helm_v1` and `[capi_helm] kubeconfig_file=`. -- Systemd overrides on `magnum-api` and `magnum-conductor` that replace - the init.d wrapper's ExecStart with explicit `--config-dir` invocation. -- Both services running cleanly with the CAPI driver loaded. - -**Scope:** v1 testcloud. Roosevelt deltas in §12. - -**Out of scope:** -- Magnum domain setup (runbook 04) -- Workload cluster lifecycle (runbook 04a) -- Smoketest tenant cluster creation is OPTIONAL (§11) — full validation - framework belongs in runbook 08. - ---- - -## 2. Decisions captured - -| Decision | Choice | Reason | -|---|---|---| -| Driver pin | `magnum-capi-helm==1.1.0` from PyPI | D-007 correction (stackhpc fork archived Dec 2024; canonical project on opendev/PyPI; 1.1.0 is last Caracal-cycle release) | -| Install method | `pip3 install --break-system-packages` | PEP 668 — Ubuntu 22.04+ requires explicit override for system-site-packages install | -| Install scope | System Python on magnum unit (not venv) | Magnum charm uses system-packaged python at `/usr/lib/python3/dist-packages/magnum/`; driver must import from same site | -| Kubeconfig target | Workload cluster (post-pivot) | Workstream 3b — bootstrap k3s is empty post-pivot; CAPI controllers live in workload | -| Kubeconfig source | `$HOME/magnum-capi/capi-mgmt-cluster.kubeconfig` (staged by 04a §19) | Documented handoff | -| Driver entry-point name | `k8s_capi_helm_v1` | Per upstream magnum-capi-helm 1.1.0; verify in §10 | -| Conf.d filename | `99-capi.conf` | Numeric prefix ensures it loads AFTER any charm-managed conf, so `enabled_drivers` override wins | -| File encoding | ASCII-only | Non-ASCII in conf.d causes silent magnum daemon failures (handoff lesson; cf. Horizon `local_settings.d` issue) | -| Trustee credential | Existing magnum-shared user (charm-managed) | Roosevelt will use app-credential pattern | - ---- - -## 3. Prerequisites - -| Prereq | Verification | -|---|---| -| Magnum charm active/idle | `juju status magnum \| grep magnum/0` shows `active idle` | -| Magnum domain setup completed (runbook 04) | `openstack domain show magnum \| grep enabled` returns `True` | -| Workload cluster reachable from jumphost | `kubectl --kubeconfig $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig get nodes` returns Ready nodes | -| CAPI controllers running in workload cluster | `kubectl --kubeconfig $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig get pods -n capi-system \| grep -v Running \| grep -v NAME` empty | -| Workload kubeconfig staged at expected path | `test -r $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig && stat -c %a $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig` shows `600` | -| `juju exec` works to magnum/leader (use exec, NOT ssh, for non-interactive — handoff lesson) | `juju exec --unit magnum/leader -- hostname` returns the unit hostname | - -**Set shell context:** - -```bash -export WORK=$HOME/magnum-capi -export WORKLOAD_KUBECONFIG=$WORK/capi-mgmt-cluster.kubeconfig -export DRIVER_VERSION=magnum-capi-helm==1.1.0 # per D-007 correction -cd "$WORK" -``` - -> **`juju ssh` vs `juju exec` choice:** the handoff lessons explicitly call -> out that `juju ssh` hangs when stdout is redirected (PTY allocation issue). -> This runbook uses `juju exec` for all non-interactive command execution and -> reserves `juju ssh` only for cases where you actually want an interactive -> shell. - ---- - -## 4. Pre-flight: capture current state - -Capture the magnum unit's state BEFORE making changes. Useful for diagnosis -if anything goes wrong, and as a record of what was changed. - -```bash -mkdir -p "$WORK/pre-state" - -# Service unit files (as managed by charm) -juju exec --unit magnum/leader -- \ - 'sudo systemctl cat magnum-api magnum-conductor 2>&1' \ - > "$WORK/pre-state/systemd-units.txt" - -# Currently-enabled drivers -juju exec --unit magnum/leader -- \ - 'sudo grep -r enabled_drivers /etc/magnum/ 2>/dev/null || echo "(no enabled_drivers found — charm default applies)"' \ - > "$WORK/pre-state/drivers-pre.txt" - -# Python site-packages — see what's already installed -juju exec --unit magnum/leader -- \ - 'sudo pip3 list 2>/dev/null | grep -iE "magnum|cluster|helm|kubernetes" || true' \ - > "$WORK/pre-state/pip-pre.txt" - -# conf.d state -juju exec --unit magnum/leader -- \ - 'sudo ls -la /etc/magnum/magnum.conf.d/ 2>/dev/null || echo "(no conf.d directory)"' \ - > "$WORK/pre-state/confd-pre.txt" - -# Service running state -juju exec --unit magnum/leader -- \ - 'sudo systemctl is-active magnum-api magnum-conductor' \ - > "$WORK/pre-state/service-state-pre.txt" - -# Display the captured state -cat "$WORK/pre-state/"*.txt -``` - -> **What to look for in pre-state:** the charm-managed `enabled_drivers` value -> probably includes Heat-based drivers (`heat_kubernetes`, etc.). The 99-capi.conf -> override in §7 replaces this with the single CAPI driver. The pre-state -> capture documents what was active before the override took effect. - ---- - -## 5. Install magnum-capi-helm 1.1.0 - -```bash -juju exec --unit magnum/leader -- \ - "sudo pip3 install $DRIVER_VERSION --break-system-packages" -``` - -**Verify install:** - -```bash -juju exec --unit magnum/leader -- \ - 'sudo pip3 show magnum-capi-helm | head -10' -# Expect: Name: magnum-capi-helm -# Version: 1.1.0 -# Location: /usr/lib/python3/dist-packages - -juju exec --unit magnum/leader -- \ - 'sudo python3 -c "import magnum_capi_helm; print(magnum_capi_helm.__file__)"' -# Expect: /usr/lib/python3/dist-packages/magnum_capi_helm/__init__.py -``` - -**Check that the driver entry point is registered:** - -```bash -juju exec --unit magnum/leader -- \ - 'sudo python3 -c " -from stevedore import driver -mgr = driver.DriverManager( - namespace=\"magnum.drivers\", - name=\"k8s_capi_helm_v1\", - invoke_on_load=False -) -print(\"Driver class:\", mgr.driver) -"' -# Expect: Driver class: -# (or similar — the actual class path is package-version-dependent) -``` - -> If the entry point check fails with "No 'k8s_capi_helm_v1' driver found", -> the driver name in 1.1.0 may differ from what D-007 documented. Inspect the -> installed package's `entry_points.txt`: -> -> ```bash -> juju exec --unit magnum/leader -- \ -> 'sudo cat /usr/lib/python3/dist-packages/magnum_capi_helm*.dist-info/entry_points.txt 2>/dev/null' -> ``` -> -> Find the entry under `[magnum.drivers]` — use that exact name in §7. - ---- - -## 6. Stage workload kubeconfig on magnum unit - -```bash -# Transfer kubeconfig from jumphost to magnum unit -juju scp "$WORKLOAD_KUBECONFIG" magnum/leader:/tmp/kubeconfig - -# Install with correct ownership/mode in one atomic step -juju exec --unit magnum/leader -- \ - 'sudo install -m 0640 -o root -g magnum /tmp/kubeconfig /etc/magnum/kubeconfig && sudo rm /tmp/kubeconfig' -``` - -**Verify:** - -```bash -juju exec --unit magnum/leader -- \ - 'sudo ls -la /etc/magnum/kubeconfig' -# Expect: -rw-r----- 1 root magnum ... /etc/magnum/kubeconfig - -# Confirm magnum user can read it -juju exec --unit magnum/leader -- \ - 'sudo -u magnum cat /etc/magnum/kubeconfig | head -3' -# Expect: apiVersion: v1 / clusters: / - cluster: - -# Confirm kubectl can use it from the magnum unit (sanity check on API reachability) -juju exec --unit magnum/leader -- \ - 'sudo -u magnum kubectl --kubeconfig /etc/magnum/kubeconfig get nodes 2>&1 | head -10' -# Expect: NAME ... STATUS=Ready for control plane + workers -# OR: kubectl not installed (acceptable — magnum-capi-helm uses Python client, not kubectl) -``` - -> **Why mode 0640 and group magnum:** kubeconfig contains auth tokens. Mode -> 0600 (owner-only) wouldn't let the `magnum` system user (which runs -> magnum-api/conductor) read it. Mode 0640 with `group: magnum` is the -> minimum-permission setup that works. NOT 0644 — keeps it off other users -> on the unit. - ---- - -## 7. Configure `/etc/magnum/magnum.conf.d/99-capi.conf` - -Generate the conf locally first (snap confinement does not apply to plain -bash on jumphost, but we keep paths under `$HOME` for consistency), then -transfer. - -**ASCII-only verification is critical** — the handoff documents non-ASCII -characters in `conf.d` files causing silent daemon failures (cf. Horizon -`local_settings.d`). Use plain straight quotes, ASCII dashes, no smart -typography. - -```bash -# Write locally -cat > "$WORK/99-capi.conf" <<'EOF' -[DEFAULT] -enabled_drivers = k8s_capi_helm_v1 - -[capi_helm] -kubeconfig_file = /etc/magnum/kubeconfig -EOF - -# Verify it is pure ASCII (no UTF-8 sneakers) -file "$WORK/99-capi.conf" -# Expect: ASCII text -# If it says "UTF-8 Unicode text", STOP and rewrite by hand — even one stray -# em-dash or smart quote will silently break magnum - -# Hex dump check (paranoid mode) -xxd "$WORK/99-capi.conf" | grep -v "^[0-9a-f]*: [0-9a-f ]* [a-zA-Z0-9 \[\]=._/]*$" | head -5 -# Expect: empty output (all bytes are printable ASCII) -``` - -**Stage and install:** - -```bash -juju scp "$WORK/99-capi.conf" magnum/leader:/tmp/99-capi.conf - -juju exec --unit magnum/leader -- \ - 'sudo mkdir -p /etc/magnum/magnum.conf.d && sudo install -m 0644 -o root -g root /tmp/99-capi.conf /etc/magnum/magnum.conf.d/99-capi.conf && sudo rm /tmp/99-capi.conf' - -# Verify -juju exec --unit magnum/leader -- \ - 'sudo ls -la /etc/magnum/magnum.conf.d/ && sudo cat /etc/magnum/magnum.conf.d/99-capi.conf' -# Expect: file listed; content matches what was written -``` - ---- - -## 8. Systemd override on magnum-api + magnum-conductor - -The Charmed Magnum unit files use a wrapper pattern: - -``` -ExecStart=/etc/init.d/magnum-api systemd-start -``` - -The wrapper does NOT pass `--config-dir` to magnum-api, so `/etc/magnum/magnum.conf.d/` -is never loaded. The 99-capi.conf would have no effect. - -Override with explicit `--config-file` + `--config-dir` invocation. - -**Generate override files locally:** - -```bash -cat > "$WORK/magnum-api-override.conf" <<'EOF' -[Service] -ExecStart= -ExecStart=/usr/bin/magnum-api --config-file=/etc/magnum/magnum.conf --config-dir=/etc/magnum/magnum.conf.d -EOF - -cat > "$WORK/magnum-conductor-override.conf" <<'EOF' -[Service] -ExecStart= -ExecStart=/usr/bin/magnum-conductor --config-file=/etc/magnum/magnum.conf --config-dir=/etc/magnum/magnum.conf.d -EOF - -# ASCII check -file "$WORK/magnum-api-override.conf" "$WORK/magnum-conductor-override.conf" -# Expect: ASCII text x2 -``` - -> **The empty `ExecStart=` line is critical.** Systemd accumulates ExecStart -> directives by default; an empty assignment is required to CLEAR the inherited -> directive before setting the replacement. Without the empty line, the unit -> would have BOTH the init.d wrapper AND the new direct invocation, and would -> likely fail to start. - -**Install on the unit:** - -```bash -juju scp "$WORK/magnum-api-override.conf" magnum/leader:/tmp/magnum-api-override.conf -juju scp "$WORK/magnum-conductor-override.conf" magnum/leader:/tmp/magnum-conductor-override.conf - -juju exec --unit magnum/leader -- \ - 'sudo mkdir -p /etc/systemd/system/magnum-api.service.d /etc/systemd/system/magnum-conductor.service.d && \ - sudo install -m 0644 -o root -g root /tmp/magnum-api-override.conf /etc/systemd/system/magnum-api.service.d/override.conf && \ - sudo install -m 0644 -o root -g root /tmp/magnum-conductor-override.conf /etc/systemd/system/magnum-conductor.service.d/override.conf && \ - sudo rm /tmp/magnum-api-override.conf /tmp/magnum-conductor-override.conf' - -# Reload systemd to pick up the overrides -juju exec --unit magnum/leader -- 'sudo systemctl daemon-reload' - -# Verify the overrides are effective (systemctl cat shows combined unit + overrides) -juju exec --unit magnum/leader -- 'sudo systemctl cat magnum-api | grep -A1 ExecStart' -# Expect: TWO ExecStart= lines — the empty clear-line and the new /usr/bin/magnum-api invocation -juju exec --unit magnum/leader -- 'sudo systemctl cat magnum-conductor | grep -A1 ExecStart' -# Expect: TWO ExecStart= lines as above for magnum-conductor -``` - -> **Charm reconciliation note:** the Magnum charm may rewrite its own systemd -> units on config changes or upgrades. The drop-in override at -> `/etc/systemd/system/magnum-api.service.d/override.conf` is OUTSIDE the -> charm's writable zone and should survive. Verify after any `juju refresh` or -> `juju config magnum` command by re-running the `systemctl cat` check above. - ---- - -## 9. Restart services + verify health - -```bash -juju exec --unit magnum/leader -- \ - 'sudo systemctl restart magnum-api magnum-conductor' - -# Wait briefly for services to initialize -sleep 5 - -# Check active state -juju exec --unit magnum/leader -- \ - 'sudo systemctl is-active magnum-api magnum-conductor' -# Expect: active (x2) - -# Examine recent journal for errors (the critical step — magnum's silent failure -# mode means we must read logs, not just trust is-active) -juju exec --unit magnum/leader -- \ - 'sudo journalctl -u magnum-api --since "2 minutes ago" --no-pager | tail -50' -juju exec --unit magnum/leader -- \ - 'sudo journalctl -u magnum-conductor --since "2 minutes ago" --no-pager | tail -50' -``` - -**Look for these red flags in the logs:** - -| Symptom | Likely cause | Remediation | -|---|---|---| -| `ImportError: No module named magnum_capi_helm` | §5 pip install failed | Re-run §5; check pip3 output | -| `EntryPointError: No 'k8s_capi_helm_v1' driver` | Driver entry-point name mismatch | Verify name per §5 footnote; update §7 | -| Service repeatedly restarts (look for "Started" appearing twice in 10s) | Likely a config error in 99-capi.conf | Re-check ASCII-only; check magnum.conf.d permissions | -| `kubeconfig_file` not honored | --config-dir not being passed | §8 override not active; re-run `systemctl daemon-reload` | -| Silent: no error but driver also not loading | Non-ASCII char snuck into a conf | `file /etc/magnum/magnum.conf.d/99-capi.conf` — if it says UTF-8, regenerate | - ---- - -## 10. CAPI driver enablement check - -Verify the driver is actually loaded by Magnum and reachable via the API. - -```bash -source $HOME/admin-openrc - -# List supported COE drivers via the Magnum API -openstack coe cluster template list -f json -# (empty templates list is fine — we are checking the endpoint responds) - -# Direct check on the unit: scan the service's loaded drivers -juju exec --unit magnum/leader -- \ - 'sudo journalctl -u magnum-conductor --since "5 minutes ago" --no-pager | grep -iE "driver|enabled" | head -20' -# Expect: a line mentioning k8s_capi_helm_v1 having been loaded -# (Magnum logs the loaded drivers at startup) - -# Definitive check: try creating a cluster template that requires the CAPI driver -openstack coe cluster template create magnum-capi-driver-check \ - --image noble-amd64 \ - --keypair capi-workload-key \ - --external-network ext_net \ - --master-flavor capi-mgmt-node \ - --flavor capi-mgmt-node \ - --coe kubernetes \ - --network-driver calico \ - --labels kube_tag=v1.31.4 - -openstack coe cluster template show magnum-capi-driver-check -c name -c coe -c labels -``` - -> **If template create fails with "driver not enabled" or similar:** the -> Magnum API process is not loading the conf.d. Verify the systemd override -> took effect — `sudo systemctl show magnum-api -p ExecStart` on the unit -> should show the explicit `--config-dir` invocation. If it still shows the -> init.d wrapper, the daemon-reload + restart did not pick up the override. - -**Cleanup the driver-check template:** - -```bash -openstack coe cluster template delete magnum-capi-driver-check -``` - ---- - -## 11. Optional smoketest — create a tenant CAPI cluster - -This step is **optional**. Full validation belongs in runbook 08. Use this -smoketest only if you want immediate confirmation that the entire chain -(Magnum API -> conductor -> magnum-capi-helm -> CAPI controllers in workload -cluster -> tenant K8s cluster on tenant VMs) works end-to-end. - -```bash -# Create a cluster template tuned for testcloud smoketest -openstack coe cluster template create magnum-smoketest-template \ - --image noble-amd64 \ - --keypair capi-workload-key \ - --external-network ext_net \ - --master-flavor capi-mgmt-node \ - --flavor capi-mgmt-node \ - --coe kubernetes \ - --network-driver calico \ - --labels boot_volume_size=20,kube_tag=v1.31.4,octavia_provider=ovn - -# Create a 1+1 cluster (minimum for smoketest) -openstack coe cluster create magnum-smoketest \ - --cluster-template magnum-smoketest-template \ - --master-count 1 \ - --node-count 1 - -# Poll for status (15-20 min typical; CAPI provisions tenant VMs end-to-end) -for i in $(seq 1 60); do - STATUS=$(openstack coe cluster show magnum-smoketest -c status -f value) - echo "$(date -Is) status=$STATUS" - case "$STATUS" in - CREATE_COMPLETE) echo "Smoketest passed"; break ;; - CREATE_FAILED) echo "Smoketest FAILED"; openstack coe cluster show magnum-smoketest; exit 1 ;; - esac - sleep 30 -done - -# Retrieve the smoketest cluster's kubeconfig -openstack coe cluster config magnum-smoketest --dir "$WORK/smoketest-kubeconfig" - -# Sanity-check the smoketest cluster -KUBECONFIG="$WORK/smoketest-kubeconfig/config" kubectl get nodes -KUBECONFIG="$WORK/smoketest-kubeconfig/config" kubectl get pods -A | head -20 - -# Cleanup the smoketest cluster -openstack coe cluster delete magnum-smoketest -openstack coe cluster template delete magnum-smoketest-template -``` - -> **What success looks like:** the CAPI controllers in the workload cluster -> receive the new Cluster CR (created by magnum-capi-helm in response to the -> Magnum API call), CAPO talks to OpenStack to provision tenant VMs, the -> tenant VMs join the new K8s cluster, and the new cluster has 1 control -> plane + 1 worker Ready. Octavia provides the API server LB (visible as a -> Floating IP in the tenant project). - ---- - -## 12. Roosevelt deltas (forward-look) - -| Aspect | Testcloud (v1) | Roosevelt | -|---|---|---| -| Driver pin source | PyPI `magnum-capi-helm==1.1.0` | Internal mirror with checksum verification | -| Driver pin record | Implicit in this runbook | Captured in Vault as audit artifact alongside CAPI pins | -| Kubeconfig source | Workload cluster (post-pivot per 04a §17) | Same | -| Kubeconfig rotation | Manual on capi-mgmt rebuild | Automated when workload cluster cert rotates | -| Trustee credential | Charm-default magnum-shared user | Per-tenant app credentials via Vault auth method | -| Magnum HA | num_units=1 (per D-009 testcloud) | num_units=3 with hacluster + provider VIP | -| Driver upgrade discipline | Manual re-run of §5 | Tracked maintenance window; Vault audit log | -| Systemd override | Drop-in at `/etc/systemd/system/magnum-*.service.d/override.conf` | Same — but provided via a charm overlay package, not manual file install | -| ASCII-only enforcement | Manual check (§7, §8) | Pre-flight lint in `scripts/pre-flight-checks.sh` | - ---- - -## 13. Documented runtime gotchas (carry-forward from handoff) - -These gotchas burned cycles during the Bobcat Magnum CAPI work. Each is -explicitly handled in this runbook; collecting them here for visibility: - -1. **PEP 668 `--break-system-packages`** (§5). Ubuntu 22.04+ refuses - `pip install` against system Python by default. The flag is required for - the magnum-capi-helm install path used by Charmed Magnum. -2. **`juju ssh` hangs on stdout redirect.** PTY allocation issue. - This runbook uses `juju exec` for all non-interactive command execution. -3. **Heredoc nesting in `juju ssh` is fragile.** This runbook writes - conf files locally first and uses `juju scp` + `juju exec install` to - transfer — single-level only. -4. **Non-ASCII characters in `conf.d` files cause silent daemon failures.** - §7 and §8 both include `file ` ASCII verification before transfer. -5. **`openstack -f value -c X -c Y` outputs in alphabetical field order, - not flag order.** This runbook uses single-column queries or `-f json | - jq` throughout. -6. **Charm-managed `enabled_drivers` is overridden, not appended.** The - `enabled_drivers = k8s_capi_helm_v1` line in 99-capi.conf REPLACES the - charm-default value (which would include the deprecated Heat drivers). -7. **The systemd override empty `ExecStart=` line is required** to clear - the inherited ExecStart before setting the replacement (§8). -8. **Snap-confined `openstack` CLI cannot read `/tmp`.** This runbook stages - files under `$WORK=$HOME/magnum-capi`. The smoketest in §11 also writes - to `$WORK/smoketest-kubeconfig`. - ---- - -## 14. Change log - -| Date | Change | Reference | -|---|---|---| -| 2026-05-22 | Document created. magnum-capi-helm 1.1.0 from PyPI; workload-cluster kubeconfig (post-pivot per workstream 3b); systemd override pattern; ASCII-only conf.d. | Workstream 3c | diff --git a/runbooks/06-tenant-setup.md b/runbooks/06-tenant-setup.md deleted file mode 100644 index 3915229..0000000 --- a/runbooks/06-tenant-setup.md +++ /dev/null @@ -1,41 +0,0 @@ -# Runbook 06 — Tenant Resource Recreation - -**STATUS: PLACEHOLDER** — drafted post-deploy. - -## Purpose - -Recreate the standard testcloud tenant resources (domain, project, user, -networks, images, keypairs, flavors) using a proper IPAM-aligned design -per D-010 + D-016 (not the ad-hoc `user1` pattern from the original test -cloud). - -## Prerequisites - -- Cloud fully deployed and validated -- DNS zones populated (Runbook 07 may precede this if Designate-via-tenant - DNS is in scope at tenant create time) -- NetBox IPv4 tenant pool prefix present (per D-016; default `10.20.0.0/16`) - -## TODO - -- [ ] Create domain `domain1` -- [ ] Create project `project1` in domain `domain1` -- [ ] Create user `user1` in project1 (member role + load-balancer_member - role for Octavia) -- [ ] Tenant network with CIDR carved from NetBox IPv4 tenant pool - - Suggested convention: `10.20..0/24` per D-016 - - project1 → `10.20.1.0/24` - - Per D-016 hybrid model, the per-project /24 is Neutron-managed and - NOT added back to NetBox -- [ ] Tenant router connected to ext_net (Provider 10.12.4.0/22) -- [ ] Glance image: noble-amd64 (cloud-init enabled) -- [ ] Flavor m1.small (1 vCPU, 2 GiB RAM, 20 GiB root) -- [ ] Keypair for user1 -- [ ] openrc files: `~/admin-openrc`, `~/user1-openrc` -- [ ] Application credentials for user1 (audit trail) -- [ ] Take second KVM snapshot (per D-012 Snapshot 2) - -## v1 vs. v2 note - -In v1, tenant networks are IPv4-only. v2 adds IPv6 tenant subnets carved -from the v2 IPv6 tenant pool (currently reservation status in NetBox). diff --git a/runbooks/07-dns-zones.md b/runbooks/07-dns-zones.md deleted file mode 100644 index 3b780de..0000000 --- a/runbooks/07-dns-zones.md +++ /dev/null @@ -1,36 +0,0 @@ -# Runbook 07 — Designate Zones and Records (v1: A records only) - -**STATUS: PLACEHOLDER** — drafted post-deploy. - -## Purpose - -Create the cloud's DNS zones in Designate, populate API VIP A records -(v1: IPv4 only), and configure Neutron defaults to push Designate as -tenant DNS resolver. - -## Prerequisites - -- Designate charm in `active/idle` -- Keystone, Neutron API reachable -- API VIP hostnames already in `/etc/hosts` on all OpenStack nodes - (per D-008 Layer 0 bootstrap) - -## TODO - -- [ ] Create primary zone: - `openstack zone create --email admin@neumatrix.local \ - omega.dc0.vr0.cloud.neumatrix.local.` -- [ ] Populate API VIP **A** records for each public service: - - keystone, glance, nova, neutron, cinder, placement, octavia, - barbican, magnum, horizon, designate - - **v1: A records only** (IPv4 VIPs from the Provider API VIP range - 10.12.4.224-.254) - - **v2 will add AAAA records when IPv6 Provider VIPs become active** -- [ ] Configure Neutron defaults: - `juju config neutron-api default-dns-domain=omega.dc0.vr0.cloud.neumatrix.local.` - `juju config neutron-api dns-domain=omega.dc0.vr0.cloud.neumatrix.local.` -- [ ] Configure Neutron DHCP to push Designate as resolver: - `juju config neutron-api dns-servers=` -- [ ] Verify from a test tenant VM: - `nslookup keystone.omega.dc0.vr0.cloud.neumatrix.local` - resolves to Provider API VIP diff --git a/runbooks/08-validate.md b/runbooks/08-validate.md deleted file mode 100644 index 7f7acfc..0000000 --- a/runbooks/08-validate.md +++ /dev/null @@ -1,33 +0,0 @@ -# Runbook 08 — Validation (Roosevelt-Rehearsal Bar) - -**STATUS: PLACEHOLDER** — drafted with scripts/validate.sh. - -## Purpose - -Execute the validation criteria from D-011 and confirm the cloud is ready to -be considered a successful rebuild. - -## Prerequisites - -- All prior runbooks complete - -## Validation criteria (per D-011) - -- [ ] All charms `active/idle` in `juju status` -- [ ] All public API VIPs respond on FQDN from jumphost -- [ ] All public API VIPs respond on FQDN from a tenant VM (Option B path) -- [ ] Octavia LB pattern passes: create LB, two members, round-robin verified, - failover verified, recovery verified -- [ ] Magnum CAPI cluster create end-to-end: cluster template + cluster create, - OCCM does not crash-loop, cluster reaches CREATE_COMPLETE -- [ ] Vault unseal + auto-unseal-after-reboot pattern: reboot vault unit, - confirm auto-unseal via etcd (or manual unseal per HA pattern) -- [ ] Designate resolves API hostnames from tenant subnet -- [ ] Snapshot 1 (post-deploy, pre-tenant) taken (per D-012) -- [ ] Snapshot 2 (post-tenant) taken (per D-012) - -## TODO - -- [ ] Run `scripts/validate.sh` and capture output -- [ ] Document any divergences from validation criteria in - `docs/design-decisions.md` change log diff --git a/runbooks/deprecated/00-pre-deploy.md b/runbooks/deprecated/00-pre-deploy.md new file mode 100644 index 0000000..aa07d86 --- /dev/null +++ b/runbooks/deprecated/00-pre-deploy.md @@ -0,0 +1,142 @@ +# Runbook 00 — Pre-Deploy + +## Purpose + +Prepare for a clean Caracal rebuild of the VR0 DC0 Omega Cloud. Capture all +state needed for rollback, gracefully tear down dependent workloads, and verify +the destination environment is ready before destroying the existing OpenStack +model. + +## Prerequisites + +- SSH access to jumphost `vopenstack-jesse` as `jessea123` +- `admin-openrc` and `user1-openrc` available in `$HOME` +- Access to the Juju controller hosting the `openstack` model +- Access to the capi-mgmt.maas k3s cluster (kubeconfig present) +- NetBox IPv4 imports completed (per `netbox/ipv4-prefixes-import.py`) +- NetBox VLAN imports completed (per `netbox/vlans-import.py`) + +## Phase 1 — Verify NetBox readiness (gating) + +Run the verification path of the NetBox import scripts. Confirm all entries +appear correctly scoped to VR0 DC0. + +```bash +cd ~/vr0-dc0-caracal +NETBOX_URL=https://netbox.baldurkeep.com NETBOX_TOKEN= \ + python3 netbox/ipv4-prefixes-import.py --verify-only +NETBOX_URL=https://netbox.baldurkeep.com NETBOX_TOKEN= \ + python3 netbox/vlans-import.py --verify-only +``` + +Expected: all prefixes and VLANs report scope-OK, no MISSING entries. + +## Phase 2 — Capture current state + +Backups needed for potential rollback: + +```bash +# Vault unseal keys and root CA cert +juju ssh vault/0 -- sudo cat /var/snap/vault/common/vault.crt > ~/backups/$(date +%F)/vault-root-ca.crt +# (Unseal keys MUST be on file from initial Vault setup; verify presence) +ls -la ~/.vault-keys + +# Export current bundle +juju export-bundle --model openstack > ~/backups/$(date +%F)/bundle-pre-rebuild.yaml + +# Snapshot of current 'juju status' +juju status --model openstack --format=yaml > ~/backups/$(date +%F)/juju-status-pre-rebuild.yaml + +# Inventory of FIPs and tenant resources we might want to recreate +source ~/admin-openrc +openstack floating ip list -c "Floating IP Address" -c "Fixed IP Address" \ + -c "Project" -f csv > ~/backups/$(date +%F)/floating-ips.csv +openstack server list --all-projects -c ID -c Name -c Project -c Status -f csv \ + > ~/backups/$(date +%F)/servers.csv +openstack network list --all-projects -c ID -c Name -c Project -f csv \ + > ~/backups/$(date +%F)/networks.csv +openstack loadbalancer list -c id -c name -c project_id -c vip_address -f csv \ + > ~/backups/$(date +%F)/loadbalancers.csv +``` + +## Phase 3 — KVM snapshots of openstack0-3 + +From the jumphost (which is the hypervisor): + +```bash +for vm in openstack0 openstack1 openstack2 openstack3; do + sudo virsh snapshot-create-as --domain "$vm" \ + --name "pre-caracal-rebuild-$(date +%F)" \ + --description "Pre-Caracal rebuild baseline" \ + --atomic +done +sudo virsh snapshot-list openstack0 +``` + +These snapshots are the disaster-recovery point. + +## Phase 4 — Graceful CAPI workload teardown (D-013) + +Delete the CAPI workload cluster cleanly so its OpenStack resources (LBs, FIPs, +volumes, Octavia members) are released by CAPI controllers before model destroy. + +```bash +export KUBECONFIG=~/magnum-capi/phase3/capi-mgmt-cluster.kubeconfig +# (Adjust path if kubeconfig has moved) + +# Delete the workload cluster — CAPI handles tenant OpenStack cleanup +kubectl delete cluster capi-mgmt-cluster -n default +# Wait for finalizers; this may take ~10 minutes +kubectl wait --for=delete cluster/capi-mgmt-cluster -n default --timeout=15m +``` + +Verify on the OpenStack side that resources were released: + +```bash +source ~/admin-openrc +openstack server list --all-projects | grep -i capi || echo "No CAPI servers remaining" +openstack loadbalancer list | grep -i capi || echo "No CAPI LBs remaining" +openstack floating ip list -c "Floating IP Address" -c "Fixed IP Address" -f csv +``` + +## Phase 5 — Preserve capi-mgmt.maas itself + +The bootstrap k3s + CAPI controllers on `capi-mgmt.maas` are NOT destroyed — +they will be re-used post-rebuild as the Magnum CAPI mgmt plane. Verify the +controllers are still healthy: + +```bash +ssh capi-mgmt.maas -- sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml \ + get pods -A +``` + +Confirm: +- `capi-system` namespace pods Running +- `capo-system` (CAPI OpenStack provider) pods Running +- `cert-manager` pods Running +- `orc-system` (OpenStack Resource Controller) pods Running + +## Phase 6 — Final go/no-go checklist + +Do not proceed to `runbooks/01-destroy-model.md` until all of the following pass: + +- [ ] NetBox verification clean +- [ ] Vault unseal keys backed up and verified readable +- [ ] `bundle-pre-rebuild.yaml` exists and is non-empty +- [ ] `juju-status-pre-rebuild.yaml` shows desired-pre-destroy state captured +- [ ] All four KVM snapshots created (`virsh snapshot-list` confirms) +- [ ] CAPI workload cluster deletion completed (`kubectl get cluster` returns + "no resources found") +- [ ] OpenStack-side resources from CAPI workload are released (no orphaned LBs, + FIPs, volumes) +- [ ] capi-mgmt.maas k3s cluster controllers all Running + +## Notes + +- Snapshot disk space consumption can grow significantly during the rebuild + window. Verify free space on `/var/lib/libvirt/images` prior to running + the rebuild deploy. +- If Vault unseal keys cannot be located, STOP. A failed Vault re-init without + the original keys means lost issued certificates and is destructive to any + data sealed under the existing root key. This MUST be confirmed before model + destroy. diff --git a/runbooks/deprecated/01a-octavia-pki-generation.md b/runbooks/deprecated/01a-octavia-pki-generation.md new file mode 100644 index 0000000..65bd707 --- /dev/null +++ b/runbooks/deprecated/01a-octavia-pki-generation.md @@ -0,0 +1,650 @@ +# Runbook 01a — Octavia LBaaS PKI generation + +**Status:** Pre-deploy execution. Runs between `01-destroy-model.md` and `02-deploy.md`. +**Numbering rationale:** Octavia PKI artifacts must exist on the deploy host before +`juju deploy` is invoked (the values are referenced by the overlay file). Placing +this between destroy and deploy aligns generation with the "fresh rebuild" framing. + +**Cross-references:** +- D-007 (Octavia in bundle from day one) +- Bundle `octavia.options` PKI material section +- `overlays/octavia-pki.yaml` (gitignored — output of this runbook) +- Workstream 3a decision (2026-05-22): generate fresh, EC P-384 CAs, overlay-file approach + +--- + +## 1. Purpose & scope + +This runbook generates a complete two-tier PKI for Charmed Octavia's +amphora load-balancer trust domain: + +- **Issuing CA** — Octavia uses this to sign each amphora's server certificate + at LB-creation time. Octavia receives the **private key** and **passphrase**. +- **Controller CA** — amphorae's trust anchor for connections **from** the + Octavia controller. Octavia only receives the **cert** (no key needed at + runtime); the controller's identity is proved by: +- **Controller certificate** — signed by Controller CA, presented by the + Octavia controller to each amphora. Bundled as cert + key into a single + PEM blob. + +Five charm options consume the artifacts (`octavia` application): + +| Charm option | Content | Format | +|---|---|---| +| `lb-mgmt-issuing-cacert` | Issuing CA certificate | base64-encoded PEM | +| `lb-mgmt-issuing-ca-private-key` | Issuing CA encrypted private key | base64-encoded PEM (already encrypted with passphrase) | +| `lb-mgmt-issuing-ca-key-passphrase` | Issuing CA key passphrase | plain string (NOT base64) | +| `lb-mgmt-controller-cacert` | Controller CA certificate | base64-encoded PEM | +| `lb-mgmt-controller-cert` | Controller cert + key, concatenated | base64-encoded PEM bundle | + +**Scope:** v1 testcloud (VR0 DC0 Omega Cloud). Roosevelt deltas documented in +section 14. + +**Out of scope:** Octavia API TLS (issued by Vault via `octavia:certificates` +relation); rotation procedure (deferred to Roosevelt runbook). + +--- + +## 2. Decisions captured + +Per workstream 3a sign-off (2026-05-22): + +| Decision | Choice | Roosevelt parallel | +|---|---|---| +| Cert provenance | Generate fresh (no Bobcat-backup copy) | Vault PKI engine | +| CA key algorithm | EC P-384 | EC P-384 (Vault root) | +| Controller cert algorithm | EC P-256 | EC P-256 | +| CA validity | 10 years | 5-year intermediate, Vault-rotated | +| Controller cert validity | 2 years | 90 days, auto-rotated | +| Distribution method | Juju overlay file (gitignored) | Vault-injected at deploy | +| Storage path on jumphost | `$HOME/octavia-pki/` | Vault PKI mounts | +| Passphrase strength | 32 random bytes, base64-encoded (44 chars) | Vault-generated | + +**Naming convention:** + +- Issuing CA CN: `VR0 DC0 Omega Cloud Octavia Issuing CA` +- Controller CA CN: `VR0 DC0 Omega Cloud Octavia Controller CA` +- Controller cert CN: `octavia-controller.omega.dc0.vr0.cloud.neumatrix.local` +- Controller cert SANs: above CN, plus `octavia.omega.dc0.vr0.cloud.neumatrix.local`, plus `10.12.4.233` (the Octavia API VIP per workstream 2) +- Organization (O): `Neumatrix` + +--- + +## 3. Prerequisites + +- Executor is on jumphost `vopenstack-jesse` as `jessea123`. +- `openssl` version 3.x or later installed (`openssl version` to confirm). +- `$HOME` is writable (snap-confined `openstackclients` cannot read `/tmp`; + all paths must resolve under `$HOME`). +- Git repository `openstack-caracal-ipv4` cloned on jumphost at a known path + (referred to as `$REPO` throughout). Set this in the executor's shell: + ```bash + export REPO=$HOME/repos/openstack-caracal-ipv4 # adjust to actual clone path + ``` +- Repository is on `main` branch and clean (`cd $REPO && git status` shows clean tree). +- Previous workstream 2 commit has been pushed (bundle has the VIP assignments and + active hacluster stack — verify with `grep -c "^ vip: 10.12.4." "$REPO/bundle.yaml"`, + expect 12). + +--- + +## 4. Pre-flight: gitignore patch (DO THIS FIRST) + +**Critical:** the `.gitignore` patch goes in BEFORE any private key material +exists on disk. This minimizes the race window for an accidental commit. + +```bash +cd "$REPO" + +# Append to .gitignore (idempotent — check if already present first) +grep -q "octavia-pki.yaml" .gitignore || cat >> .gitignore <<'EOF' + +# Octavia PKI artifacts — NEVER commit +overlays/octavia-pki.yaml +octavia-pki/ +*.key +*.key.enc +passphrase.txt +EOF + +# Review the diff +git diff .gitignore + +# Commit and push BEFORE generating any keys +git add .gitignore +git commit -m "gitignore: octavia PKI artifacts and overlay (runbook 01a)" +git push origin main +``` + +**Verify the gitignore is effective:** + +```bash +# This should NOT show overlays/octavia-pki.yaml even as untracked +touch overlays/octavia-pki.yaml +git status --short overlays/ # expect: empty output for octavia-pki.yaml +rm overlays/octavia-pki.yaml +``` + +If the test file does show as untracked, **STOP** and fix the gitignore syntax before +generating any secrets. + +--- + +## 5. Workspace setup + +```bash +WORKDIR=$HOME/octavia-pki +mkdir -p "$WORKDIR"/{issuing-ca,controller-ca,controller,overlay-build} +chmod 700 "$WORKDIR" +cd "$WORKDIR" +echo "Working in: $WORKDIR" +``` + +Resulting layout: + +``` +$HOME/octavia-pki/ +├── issuing-ca/ # passphrase.txt, .key.enc, .cert.pem +├── controller-ca/ # passphrase.txt, .key.enc, .cert.pem +├── controller/ # .key, .csr, .cert.pem, .bundle.pem, .cnf +└── overlay-build/ # base64 intermediates → consumed by step 10 +``` + +--- + +## 6. Generate Issuing CA + +EC P-384 key encrypted with random 32-byte passphrase. Self-signed cert, 10y validity. + +```bash +cd "$WORKDIR/issuing-ca" + +# Generate passphrase (no trailing newline — required for clean YAML embedding) +openssl rand -base64 32 | tr -d '\n' > passphrase.txt +chmod 600 passphrase.txt + +# Sanity-check +test $(wc -c < passphrase.txt) -eq 44 || { echo "ERROR: passphrase length wrong"; exit 1; } + +# Generate EC P-384 private key, encrypted with passphrase +openssl genpkey -algorithm EC \ + -pkeyopt ec_paramgen_curve:P-384 \ + -aes-256-cbc \ + -pass file:passphrase.txt \ + -out issuing-ca.key.enc +chmod 600 issuing-ca.key.enc + +# Self-sign cert (10 years, SHA-384) +openssl req -new -x509 -sha384 \ + -key issuing-ca.key.enc \ + -passin file:passphrase.txt \ + -days 3650 \ + -subj "/CN=VR0 DC0 Omega Cloud Octavia Issuing CA/O=Neumatrix" \ + -out issuing-ca.cert.pem + +# Verify +openssl x509 -in issuing-ca.cert.pem -noout -dates -subject +openssl verify -CAfile issuing-ca.cert.pem issuing-ca.cert.pem +# Expect: issuing-ca.cert.pem: OK + +ls -la +``` + +--- + +## 7. Generate Controller CA + +Identical pattern; different CN. + +```bash +cd "$WORKDIR/controller-ca" + +openssl rand -base64 32 | tr -d '\n' > passphrase.txt +chmod 600 passphrase.txt +test $(wc -c < passphrase.txt) -eq 44 || { echo "ERROR: passphrase length wrong"; exit 1; } + +openssl genpkey -algorithm EC \ + -pkeyopt ec_paramgen_curve:P-384 \ + -aes-256-cbc \ + -pass file:passphrase.txt \ + -out controller-ca.key.enc +chmod 600 controller-ca.key.enc + +openssl req -new -x509 -sha384 \ + -key controller-ca.key.enc \ + -passin file:passphrase.txt \ + -days 3650 \ + -subj "/CN=VR0 DC0 Omega Cloud Octavia Controller CA/O=Neumatrix" \ + -out controller-ca.cert.pem + +openssl x509 -in controller-ca.cert.pem -noout -dates -subject +openssl verify -CAfile controller-ca.cert.pem controller-ca.cert.pem +# Expect: controller-ca.cert.pem: OK +``` + +**Why Controller CA's key is encrypted even though Octavia never uses it:** +The Controller CA key is needed for future rotations of the controller cert. +Encrypting it (with its own passphrase, separate from Issuing CA's) is defense +in depth — if the jumphost is compromised, the key still requires the +passphrase to be useful for forging controller certs. + +--- + +## 8. Generate Controller certificate + +EC P-256 key (no encryption — Octavia must read it at startup), CSR with SAN +extensions, signed by Controller CA, 2y validity. + +```bash +cd "$WORKDIR/controller" + +# Generate unencrypted EC P-256 key +openssl genpkey -algorithm EC \ + -pkeyopt ec_paramgen_curve:P-256 \ + -out controller.key +chmod 600 controller.key + +# CSR config with SAN extensions +cat > controller.cnf <<'EOF' +[req] +distinguished_name = req_distinguished_name +req_extensions = v3_req +prompt = no + +[req_distinguished_name] +CN = octavia-controller.omega.dc0.vr0.cloud.neumatrix.local +O = Neumatrix + +[v3_req] +keyUsage = critical, digitalSignature, keyEncipherment +extendedKeyUsage = clientAuth, serverAuth +subjectAltName = @alt_names + +[alt_names] +DNS.1 = octavia-controller.omega.dc0.vr0.cloud.neumatrix.local +DNS.2 = octavia.omega.dc0.vr0.cloud.neumatrix.local +IP.1 = 10.12.4.233 +EOF + +# Generate CSR +openssl req -new -sha256 \ + -key controller.key \ + -config controller.cnf \ + -out controller.csr + +# Sign with Controller CA (2 years) +openssl x509 -req -sha256 \ + -in controller.csr \ + -CA "$WORKDIR/controller-ca/controller-ca.cert.pem" \ + -CAkey "$WORKDIR/controller-ca/controller-ca.key.enc" \ + -passin file:"$WORKDIR/controller-ca/passphrase.txt" \ + -CAcreateserial \ + -days 730 \ + -extfile controller.cnf \ + -extensions v3_req \ + -out controller.cert.pem + +# Bundle cert + key (the lb-mgmt-controller-cert option expects both in one PEM) +cat controller.cert.pem controller.key > controller.bundle.pem +chmod 600 controller.bundle.pem +``` + +**Verify the chain and SAN:** + +```bash +# Chain verifies +openssl verify -CAfile "$WORKDIR/controller-ca/controller-ca.cert.pem" controller.cert.pem +# Expect: controller.cert.pem: OK + +# SAN extensions present +openssl x509 -in controller.cert.pem -noout -ext subjectAltName +# Expect: +# DNS:octavia-controller.omega.dc0.vr0.cloud.neumatrix.local, +# DNS:octavia.omega.dc0.vr0.cloud.neumatrix.local, +# IP Address:10.12.4.233 + +# Validity +openssl x509 -in controller.cert.pem -noout -dates +# Expect: notAfter ~2 years from today + +# Bundle integrity (cert + key match) +openssl x509 -in controller.bundle.pem -noout -pubkey > /tmp/cert.pub +openssl pkey -in controller.bundle.pem -pubout > /tmp/key.pub +diff /tmp/cert.pub /tmp/key.pub && echo "Bundle cert/key match" +rm /tmp/cert.pub /tmp/key.pub +``` + +--- + +## 9. Final chain verification + +A standalone block to confirm the full chain is sound before consuming for Octavia: + +```bash +cd "$WORKDIR" + +echo "=== Issuing CA ===" +openssl x509 -in issuing-ca/issuing-ca.cert.pem -noout -subject -dates +openssl verify -CAfile issuing-ca/issuing-ca.cert.pem issuing-ca/issuing-ca.cert.pem + +echo "" +echo "=== Controller CA ===" +openssl x509 -in controller-ca/controller-ca.cert.pem -noout -subject -dates +openssl verify -CAfile controller-ca/controller-ca.cert.pem controller-ca/controller-ca.cert.pem + +echo "" +echo "=== Controller cert ===" +openssl x509 -in controller/controller.cert.pem -noout -subject -dates +openssl verify -CAfile controller-ca/controller-ca.cert.pem controller/controller.cert.pem +``` + +All three "verify" lines must show `: OK`. If any do not, **STOP** and investigate +before proceeding. + +--- + +## 10. Base64-encode artifacts + +Each base64 file is a single line (no wrapping); each becomes one YAML value. + +```bash +cd "$WORKDIR/overlay-build" + +# Issuing CA cert (base64) +base64 -w0 "$WORKDIR/issuing-ca/issuing-ca.cert.pem" > issuing-cacert.b64 + +# Issuing CA private key (already encrypted PEM → base64) +base64 -w0 "$WORKDIR/issuing-ca/issuing-ca.key.enc" > issuing-ca-private-key.b64 + +# Controller CA cert +base64 -w0 "$WORKDIR/controller-ca/controller-ca.cert.pem" > controller-cacert.b64 + +# Controller cert + key bundle +base64 -w0 "$WORKDIR/controller/controller.bundle.pem" > controller-cert.b64 + +# Sanity-check sizes (expect 500-2000 chars each) +wc -c *.b64 +``` + +--- + +## 11. Assemble the overlay file + +```bash +# Read each artifact into shell variables +ISSUING_CACERT=$(cat "$WORKDIR/overlay-build/issuing-cacert.b64") +ISSUING_CA_KEY=$(cat "$WORKDIR/overlay-build/issuing-ca-private-key.b64") +ISSUING_CA_PASS=$(cat "$WORKDIR/issuing-ca/passphrase.txt") +CONTROLLER_CACERT=$(cat "$WORKDIR/overlay-build/controller-cacert.b64") +CONTROLLER_CERT=$(cat "$WORKDIR/overlay-build/controller-cert.b64") + +# Assemble overlay (note: passphrase is YAML-quoted; cert blobs are not — they're +# guaranteed-safe base64 without special chars) +mkdir -p "$REPO/overlays" +cat > "$REPO/overlays/octavia-pki.yaml" < + # lb-mgmt-controller-cert: + # lb-mgmt-issuing-ca-key-passphrase: + # lb-mgmt-issuing-ca-private-key: + # lb-mgmt-issuing-cacert: +``` + +**With this block:** + +```yaml + # ----- PKI material ------------------------------------------------- + # 5 lb-mgmt-* options are supplied via overlays/octavia-pki.yaml + # (gitignored). Generated per runbooks/01a-octavia-pki-generation.md. + # Deploy with: + # juju deploy ./bundle.yaml \ + # --overlay overlays/vr0-dc0-testcloud.yaml \ + # --overlay overlays/octavia-pki.yaml +``` + +Commit this bundle change separately from the overlay generation work: + +```bash +cd "$REPO" +git diff bundle.yaml +git add bundle.yaml +git commit -m "bundle: octavia PKI moves to overlay (runbook 01a) + +Remove inline placeholders + TODO(octavia-cert) block. PKI values now +supplied via overlays/octavia-pki.yaml (gitignored), generated per +runbooks/01a-octavia-pki-generation.md. Decision per workstream 3a +(2026-05-22): industry-best-practice secret handling on testcloud +to rehearse Roosevelt's Vault-PKI-backed posture." +git push origin main +``` + +--- + +## 13. Sensitive-file backup + +The Issuing CA private key + its passphrase are the crown jewels of the LB trust +domain. Loss → cannot sign new amphora certs (LBs gradually break). Exposure → +attacker can forge amphora identities and intercept tenant LB traffic. + +**Minimum backup for testcloud:** + +```bash +cd $HOME +BACKUP_NAME="octavia-pki-backup-$(date +%Y%m%d-%H%M%S).tar.gz" + +tar -czf "$BACKUP_NAME" -C $HOME octavia-pki/ + +# Encrypt with strong symmetric cipher +gpg --symmetric --cipher-algo AES256 --output "${BACKUP_NAME}.gpg" "$BACKUP_NAME" + +# Shred the unencrypted tar +shred -uvz "$BACKUP_NAME" + +ls -la "${BACKUP_NAME}.gpg" +``` + +**Move `${BACKUP_NAME}.gpg` off-host** (your decision — admin workstation +encrypted drive, password-manager attachment, dedicated secrets vault, etc.). +Do NOT leave it sitting in $HOME on the jumphost long-term — that's a single +point of compromise. + +**Roosevelt note:** Vault PKI engine stores all of this — no manual backup +required; Vault's own backup mechanism covers it. The procedure above is +testcloud-only. + +--- + +## 14. Cleanup of intermediates + +After successful deploy + verification (section 14), shred files that are not +needed for future rotation: + +```bash +# Optional: shred the base64 intermediates (regeneratable from PEM sources) +shred -uvz "$WORKDIR/overlay-build/"*.b64 +rmdir "$WORKDIR/overlay-build" + +# Optional: shred the CSR (regeneratable if needed) +shred -uvz "$WORKDIR/controller/controller.csr" + +# DO NOT shred any of the following — they are needed for future operations: +# - issuing-ca/{issuing-ca.cert.pem, issuing-ca.key.enc, passphrase.txt} +# - controller-ca/{controller-ca.cert.pem, controller-ca.key.enc, passphrase.txt} +# - controller/{controller.key, controller.cert.pem, controller.bundle.pem, controller.cnf} +# +# Specifically: +# - Issuing CA artifacts: required for signing new amphoras (Octavia uses them runtime) +# - Controller CA artifacts: required for signing new controller certs (rotation) +# - Controller cert/key: required to repopulate the overlay if jumphost is rebuilt +``` + +--- + +## 15. Post-deploy verification + +After `runbooks/02-deploy.md` completes (`juju deploy` with the overlay), +verify Octavia is healthy and the PKI plumbing works. + +```bash +# Octavia charm active/idle +juju status octavia +# Expect: octavia/0 active idle + +# Octavia services running +juju ssh octavia/0 -- sudo systemctl is-active octavia-api octavia-worker octavia-housekeeping +# Expect: 3x "active" + +# Confirm PKI files landed on the unit +juju ssh octavia/0 -- sudo ls -la /etc/octavia/certs/ +# Expect: server_ca.cert.pem, server_ca.key.pem, client_ca.cert.pem, client.cert-and-key.pem +# (filenames are charm-controlled; presence is what matters) + +# Confirm Octavia can use them — verbose health-check from the API +juju ssh octavia/0 -- sudo journalctl -u octavia-api --since "5 minutes ago" \ + | grep -iE "(cert|ssl|tls|amphora)" | head -20 +# Expect: no errors related to cert loading +``` + +**Smoketest — create a test LB once amphora image is available:** + +```bash +# After `octavia-diskimage-retrofit` has populated Glance with the amphora image, +# and the LBaaS Mgmt network is wired (these are downstream runbook steps), +# a test LB creation exercises the full PKI chain: + +source ~/admin-openrc +openstack loadbalancer create --name pki-smoketest --vip-subnet-id + +# Watch for amphora spawn (3-5 minutes typical) +watch -n5 'openstack loadbalancer show pki-smoketest' +# Wait for: provisioning_status=ACTIVE, operating_status=ONLINE + +# Octavia-worker log should show successful amphora handshake (signed by Issuing CA, +# trusted via Controller CA): +juju ssh octavia/0 -- sudo journalctl -u octavia-worker --since "10 minutes ago" \ + | grep -iE "(amphora|cert)" | tail -20 +# Expect: "amphora connection established" or similar +# Expect: no TLS handshake errors, no cert validation errors + +# Cleanup the smoketest LB +openstack loadbalancer delete pki-smoketest --cascade +``` + +If amphora handshake fails with cert errors, the most likely causes are: + +1. SAN mismatch — the controller's connection to amphora uses the cert's CN/SAN; + verify the controller cert SAN covers all addresses Octavia uses to reach amphorae. +2. Bundle/key mismatch — `lb-mgmt-controller-cert` bundle should contain BOTH the + cert and the matching private key; if they're for different keys, handshake fails. +3. Encrypted Issuing CA key + wrong passphrase — verify the passphrase string in + the overlay matches what was used at generation. + +--- + +## 16. Roosevelt deltas (forward-look) + +When this runbook is adapted for Roosevelt bare-metal deploy: + +| Aspect | Testcloud (v1) | Roosevelt | +|---|---|---| +| Issuing CA root | Self-signed | Intermediate signed by Vault root CA | +| CA storage | Filesystem on jumphost | Vault PKI engine, encrypted at rest | +| Controller cert validity | 2 years | 90 days | +| Rotation | Manual (this runbook re-run) | Automated via Vault + cron + bundle redeploy | +| Backup | gpg tarball, off-host | Vault's own backup mechanism | +| Amphora image signing | Out of scope for v1 | Image signed by Vault PKI as well | +| Procedure file | `runbooks/01a-octavia-pki-generation.md` | New runbook in Roosevelt repo | + +The procedure structure (generate Issuing CA → Controller CA → Controller cert → +encode → overlay → backup → deploy) remains identical. Roosevelt just sources +the CA root from Vault instead of self-signing. + +--- + +## 17. Rotation/renewal pointer + +For testcloud, the 2-year controller cert and 10-year CAs are intentionally +"set and forget" — they will outlive the cloud at this scale. + +If rotation IS needed before testcloud teardown (e.g., a key leak event), the +re-run procedure is: + +1. Generate new Controller cert signed by **existing** Controller CA (re-run + sections 8-9 only). +2. Regenerate the overlay (section 11) with the new Controller cert; leave all + other values unchanged. +3. `juju config octavia lb-mgmt-controller-cert=` (single-option + update; does not require full bundle redeploy). +4. Octavia services may need a restart: `juju ssh octavia/0 -- sudo systemctl restart octavia-api octavia-worker octavia-housekeeping`. +5. Existing amphorae will need to reconnect using the new cert; in-flight LBs + may briefly drop. This is acceptable for a security-event rotation. + +For Roosevelt, this whole procedure is replaced by Vault automated rotation — +see Roosevelt runbook (TBD). + +--- + +## 18. Change log + +| Date | Change | Reference | +|---|---|---| +| 2026-05-22 | Document created. Fresh-generate, EC P-384 CAs, EC P-256 controller cert, overlay-file distribution. | Workstream 3a | diff --git a/runbooks/deprecated/02-deploy.md b/runbooks/deprecated/02-deploy.md new file mode 100644 index 0000000..4a52845 --- /dev/null +++ b/runbooks/deprecated/02-deploy.md @@ -0,0 +1,23 @@ +# Runbook 02 — Deploy New Caracal Bundle + +**STATUS: PLACEHOLDER** — drafted alongside bundle.yaml. + +## Purpose + +Deploy the new Charmed OpenStack Caracal bundle and wait for the cloud to +settle in `active/idle`. + +## Prerequisites + +- Runbook 01 complete (model destroyed, MAAS state clean) +- `bundle.yaml` and `overlays/vr0-dc0-testcloud.yaml` drafted and reviewed +- `scripts/pre-flight-checks.sh` passes + +## TODO + +- [ ] `juju add-model openstack` +- [ ] `juju deploy ./bundle.yaml --overlay overlays/vr0-dc0-testcloud.yaml --trust` +- [ ] Wait for settle (`juju-wait` or `juju status --watch 30s`) +- [ ] Pause-points for Vault init (per Runbook 03) +- [ ] Acceptance: all charms `active/idle` modulo Vault (sealed) and any + charms waiting on Vault certificates diff --git a/runbooks/deprecated/03-vault-init.md b/runbooks/deprecated/03-vault-init.md new file mode 100644 index 0000000..40db379 --- /dev/null +++ b/runbooks/deprecated/03-vault-init.md @@ -0,0 +1,24 @@ +# Runbook 03 — Vault Initialization + +**STATUS: PLACEHOLDER** — drafted during deploy phase. + +## Purpose + +Initialize the Vault instance(s), unseal, authorize, and let certificate +relations resolve so dependent charms reach `active/idle`. + +## Prerequisites + +- Bundle deployed; Vault charm in `blocked` waiting for init +- etcd cluster in `active/idle` (Vault HA backend per D-006) +- easyrsa active (TLS bootstrap) + +## TODO + +- [ ] `juju run vault/leader generate-root-ca` — capture root CA cert +- [ ] `vault operator init -key-shares=5 -key-threshold=3` — capture keys +- [ ] Unseal with 3 of 5 keys +- [ ] `juju run vault/leader authorize-charm token=` +- [ ] Verify all `:certificates` relations complete (no charms stuck + waiting on certs) +- [ ] Store unseal keys in `~/.vault-keys/` (chmod 600); back up diff --git a/runbooks/deprecated/04-magnum-domain.md b/runbooks/deprecated/04-magnum-domain.md new file mode 100644 index 0000000..b400a18 --- /dev/null +++ b/runbooks/deprecated/04-magnum-domain.md @@ -0,0 +1,21 @@ +# Runbook 04 — Magnum Keystone Domain Setup + +**STATUS: PLACEHOLDER** — drafted post-deploy. + +## Purpose + +Run the magnum charm's `domain-setup` action to create the Keystone domain, +trust role, and service user that Magnum requires for cluster operations. + +## Prerequisites + +- Magnum charm reached `active/idle` post Vault init +- Keystone reachable from jumphost via FQDN + +## TODO + +- [ ] `juju run magnum/leader domain-setup --wait=10m` +- [ ] Verify creation in Keystone: + `openstack domain show magnum` + `openstack user show magnum_domain_admin --domain magnum` +- [ ] Acceptance: domain present, trust role assigned, charm in active/idle diff --git a/runbooks/deprecated/04a-capi-bootstrap-cluster.md b/runbooks/deprecated/04a-capi-bootstrap-cluster.md new file mode 100644 index 0000000..d98f20c --- /dev/null +++ b/runbooks/deprecated/04a-capi-bootstrap-cluster.md @@ -0,0 +1,1056 @@ +# Runbook 04a — CAPI bootstrap cluster + +**Status:** Executes after `02-deploy.md` (cloud up + all charms active/idle) +and `03-vault-init.md` (Vault initialized + root CA available). Precedes +`05-magnum-capi-driver.md` (driver graft consumes the workload kubeconfig +produced here). + +**D-017 posture:** L3 full teardown and rebuild every deployment cycle. +Nothing is preserved across cycles. capi-mgmt is wiped to MAAS Ready on +teardown; rebuilt from scratch by this runbook. + +**Cross-references:** +- D-017 (CAPI bootstrap cluster lifecycle) +- D-007 (Magnum two-layer install) +- D-002 (channel matrix — informs Vault CA chain) +- Workstream 3b decision (2026-05-22): ship Vault CA (no tls-insecure); pivot mandatory + +--- + +## 1. Purpose & scope + +This runbook stands up the CAPI bootstrap cluster on `capi-mgmt.maas` and +pivots cluster state into a self-managing workload cluster. Output: + +1. **Workload K8s cluster** (`capi-mgmt-cluster`) running in tenant VMs on + the cloud, self-managing post-pivot. +2. **Workload kubeconfig** copied to jumphost at a known path. Consumed by + `runbooks/05-magnum-capi-driver.md` for the Magnum CAPI Helm driver + graft. +3. **No remaining state** on the bootstrap k3s VM after pivot. capi-mgmt + becomes a disposable jump host. + +**Scope:** v1 testcloud. Roosevelt deltas in section 20. + +**Out of scope:** + +- Magnum-side configuration (runbook 05). +- Workload cluster's tenant lifecycle (Magnum's job, not this runbook's). +- Backup / DR for the workload cluster (Roosevelt concern). + +--- + +## 2. Decisions captured + +Per workstream 3b sign-off (2026-05-22): + +| Decision | Choice | Roosevelt parallel | +|---|---|---| +| Version pinning | Pin-at-execution with discovery in §4 | Same pattern; pins captured in deploy record | +| Cloud TLS trust | Ship Vault CA to capi-mgmt + workload nodes (no `tls-insecure`) | Image-baked CA; CK8sConfig redundancy | +| `clusterctl move` pivot | Mandatory; workload cluster becomes self-managing | Same | +| K8s flavor | Canonical Kubernetes (CK8s) | Same | +| OpenStack auth | v3applicationcredential | Same | +| Pod CIDR | `10.244.0.0/16` | Same (does not conflict with cloud `10.12.0.0/16` or tenant pool `10.20.0.0/16`) | +| Service CIDR | `10.96.0.0/12` | Same | +| Workload cluster name | `capi-mgmt-cluster` | Same | +| Workload node SSH user | `ubuntu` (MAAS/cloud-init convention) | Same | + +**Naming convention:** + +- Keystone project for CAPI: `capi-mgmt` (in `admin_domain`) +- Keystone user for CAPI: `capo` (CAPO operator) +- App credential: `capo-app-cred` +- Workload image (Glance): `noble-amd64` (existing; do NOT duplicate as `ubuntu-24.04-capi` — Bobcat lesson) +- Workload flavor: `capi-mgmt-node` (4 vCPU / 4 GiB / 30 GB) — control plane node sizing + +--- + +## 3. Prerequisites + +| Prereq | Verification | +|---|---| +| Cloud deployed; all charms `active/idle` per D-011 | `juju status --color\| grep -v "active.*idle"` returns only the header | +| Vault initialized + unsealed | `juju ssh vault/leader -- sudo vault status` shows `Sealed=false` | +| Vault root CA available on jumphost | `test -f $HOME/vault-pki/root-ca.pem && openssl x509 -in $HOME/vault-pki/root-ca.pem -noout -subject` | +| Keystone reachable via FQDN | `curl -sf --cacert $HOME/vault-pki/root-ca.pem https://keystone.omega.dc0.vr0.cloud.neumatrix.local:5000/v3 \| jq .version.id` returns `"v3.14"` or current | +| capi-mgmt VM exists in MAAS as Ready | `maas $MAAS_PROFILE machines read \| jq '.[] \| select(.hostname=="capi-mgmt") \| .status_name'` returns `"Ready"` | +| Admin openrc available | `test -f $HOME/admin-openrc && source $HOME/admin-openrc && openstack token issue \| head -3` | +| Workspace path under $HOME (snap confinement) | `WORK=$HOME/capi-bootstrap; mkdir -p "$WORK"; cd "$WORK"; pwd` shows under home | + +**Set shell context for the runbook:** + +```bash +export REPO=$HOME/repos/openstack-caracal-ipv4 # adjust if your clone is elsewhere +export WORK=$HOME/capi-bootstrap # runbook scratch dir +export VAULT_CA=$HOME/vault-pki/root-ca.pem # Vault root CA (from runbook 03) +export CAPI_MGMT_METAL_IP=10.12.8.21 # capi-mgmt metal interface +export CAPI_MGMT_PROVIDER_IP=10.12.4.21 # capi-mgmt provider interface +export CLUSTER_NAME=capi-mgmt-cluster +mkdir -p "$WORK" +cd "$WORK" +``` + +--- + +## 4. Version discovery (set pins) + +Bobcat ran "dynamic latest." This runbook pins explicit versions captured at +execution time, with the discovery procedure documented inline so each +rebuild's pins are reproducible AND traceable. + +**GitHub API: authenticated vs unauthenticated.** Unauth has 60 req/hr; +authenticated has 5000. For multiple rebuilds in a day, set a token: + +```bash +# Optional but recommended — avoids rate-limit headaches during rebuild +export GITHUB_TOKEN= +# Or skip if you can tolerate ~10 API calls slowly +``` + +**Discover current stable releases:** + +```bash +cd "$WORK" + +# Helper: fetch latest stable release tag from a GitHub repo +gh_latest() { + local repo=$1 + local auth="" + [ -n "$GITHUB_TOKEN" ] && auth="-H Authorization: Bearer $GITHUB_TOKEN" + curl -sfL $auth "https://api.github.com/repos/$repo/releases/latest" \ + | jq -r '.tag_name' +} + +# Pin captures (one file per pin, for the deploy-record convention) +mkdir -p pins +gh_latest "kubernetes-sigs/cluster-api" | tee pins/CAPI_VERSION +gh_latest "kubernetes-sigs/cluster-api-provider-openstack" | tee pins/CAPO_VERSION +gh_latest "canonical/cluster-api-k8s" | tee pins/CK8S_VERSION +gh_latest "cert-manager/cert-manager" | tee pins/CERT_MANAGER_VERSION +gh_latest "k-orc/openstack-resource-controller" | tee pins/ORC_VERSION +gh_latest "k3s-io/k3s" | tee pins/K3S_VERSION +gh_latest "helm/helm" | tee pins/HELM_VERSION + +# Load into shell +export CAPI_VERSION=$(cat pins/CAPI_VERSION) +export CAPO_VERSION=$(cat pins/CAPO_VERSION) +export CK8S_VERSION=$(cat pins/CK8S_VERSION) +export CERT_MANAGER_VERSION=$(cat pins/CERT_MANAGER_VERSION) +export ORC_VERSION=$(cat pins/ORC_VERSION) +export K3S_VERSION=$(cat pins/K3S_VERSION) +export HELM_VERSION=$(cat pins/HELM_VERSION) + +# Display for the deploy log +cat pins/*_VERSION | paste -d= <(ls pins/) - +``` + +**Sanity check:** all values should look like `v1.X.Y` or `v0.X.Y`. If any +returned `null` or empty, the GitHub API call failed — most likely +rate-limited. Wait an hour or set `$GITHUB_TOKEN` and retry. + +**Capture pins to repo as deploy record:** + +The pin files in `$WORK/pins/` should be appended to a deploy-log artifact +(NOT committed to the repo — these are deploy-time captures). Suggested +location: `$HOME/deploy-records/$(date +%Y%m%d-%H%M)/capi-pins/`. + +```bash +DEPLOY_RECORD=$HOME/deploy-records/$(date +%Y%m%d-%H%M%S)/capi-pins +mkdir -p "$DEPLOY_RECORD" +cp pins/*_VERSION "$DEPLOY_RECORD/" +ls -la "$DEPLOY_RECORD/" +``` + +--- + +## 5. MAAS-deploy capi-mgmt + +Prerequisite: capi-mgmt MAAS machine is in `Ready` state (see §3). +Network config in MAAS: + +- **eth0** on metal fabric, DHCP → `10.12.8.21` (MAAS-pinned static lease) +- **eth1** on provider fabric, static → `10.12.4.21` + +Deploy Ubuntu 24.04 (Noble): + +```bash +# Get the capi-mgmt system_id from MAAS +CAPI_MGMT_SYSTEM_ID=$(maas $MAAS_PROFILE machines read \ + | jq -r '.[] | select(.hostname=="capi-mgmt") | .system_id') +echo "capi-mgmt system_id: $CAPI_MGMT_SYSTEM_ID" + +# Deploy +maas $MAAS_PROFILE machine deploy "$CAPI_MGMT_SYSTEM_ID" \ + distro_series=noble \ + hwe_kernel=ga-24.04 +``` + +Poll for `Deployed`: + +```bash +while true; do + STATUS=$(maas $MAAS_PROFILE machine read "$CAPI_MGMT_SYSTEM_ID" \ + | jq -r '.status_name') + echo "$(date -Is) capi-mgmt status: $STATUS" + [ "$STATUS" = "Deployed" ] && break + [ "$STATUS" = "Failed deployment" ] && { echo "FAILED"; exit 1; } + sleep 30 +done +``` + +Typical deploy time: 5-8 minutes on this hardware. + +**SSH reachability:** + +```bash +# MAAS .maas zone may not resolve from jumphost — use IP directly per handoff lessons +ssh -o StrictHostKeyChecking=accept-new ubuntu@$CAPI_MGMT_METAL_IP -- hostname +# Expect: capi-mgmt +``` + +> **Gotcha:** MAAS-deployed Ubuntu uses the `ubuntu` user, not `jessea123`. +> See handoff "recurring technical pitfalls." + +--- + +## 6. SSH bootstrap + Vault CA install + +On the jumphost, prepare a transport bundle of essentials: + +```bash +mkdir -p "$WORK/bootstrap-bundle" +cp "$VAULT_CA" "$WORK/bootstrap-bundle/vault-ca.crt" +chmod 644 "$WORK/bootstrap-bundle/vault-ca.crt" + +# Bundle pin files so capi-mgmt can read versions +cp -r "$WORK/pins" "$WORK/bootstrap-bundle/" +``` + +SCP and install Vault CA on capi-mgmt: + +```bash +scp -r "$WORK/bootstrap-bundle" ubuntu@$CAPI_MGMT_METAL_IP:/home/ubuntu/ + +ssh ubuntu@$CAPI_MGMT_METAL_IP <<'EOF' +set -euo pipefail + +# Install Vault CA as a system-trusted root +sudo cp /home/ubuntu/bootstrap-bundle/vault-ca.crt /usr/local/share/ca-certificates/ +sudo update-ca-certificates 2>&1 | tail -3 + +# Verify +openssl s_client -connect keystone.omega.dc0.vr0.cloud.neumatrix.local:5000 \ + -CApath /etc/ssl/certs -verify_return_error &1 \ + | grep -E "(Verify return code|subject=)" || \ + { echo "TLS chain verify failed against Keystone — investigate before proceeding"; exit 1; } + +# Update apt + base utilities +sudo apt-get update -qq +sudo apt-get install -y -qq jq curl yq + +# Confirm +which jq curl yq +EOF +``` + +**Expected:** + +- `update-ca-certificates` reports "1 added" +- `openssl s_client` shows `Verify return code: 0 (ok)` and a Keystone cert + whose chain terminates at the Vault CA + +> **Why this matters:** Bobcat used `tls-insecure=true` in cloud.conf which +> skipped this entire trust path. Our workstream 3b decision (ship Vault CA) +> means OCCM and CAPO will validate certs against this trust store. If TLS +> verify fails here, OCCM will crashloop later. + +--- + +## 7. k3s install + +On capi-mgmt: + +```bash +ssh ubuntu@$CAPI_MGMT_METAL_IP "K3S_VERSION=$K3S_VERSION CAPI_MGMT_METAL_IP=$CAPI_MGMT_METAL_IP bash -s" <<'REMOTE_EOF' +set -euo pipefail + +# Install k3s with explicit bind/advertise/SAN flags +curl -sfL https://get.k3s.io | \ + INSTALL_K3S_VERSION="$K3S_VERSION" \ + sh -s - server \ + --bind-address="$CAPI_MGMT_METAL_IP" \ + --advertise-address="$CAPI_MGMT_METAL_IP" \ + --node-ip="$CAPI_MGMT_METAL_IP" \ + --tls-san="$CAPI_MGMT_METAL_IP" \ + --tls-san=capi-mgmt.maas \ + --write-kubeconfig-mode=0644 \ + --disable=traefik + +# Wait for k3s API to respond +for i in $(seq 1 30); do + if sudo kubectl get nodes 2>/dev/null | grep -q "Ready"; then + echo "k3s ready"; break + fi + echo "Waiting for k3s API... ($i/30)" + sleep 5 +done + +sudo kubectl get nodes +sudo kubectl get pods -A +REMOTE_EOF +``` + +> **Gotcha:** `--bind-address=$IP` makes k3s listen ONLY on that IP — not +> also on 127.0.0.1. The default kubeconfig at +> `/etc/rancher/k3s/k3s.yaml` has `server: https://127.0.0.1:6443` and will +> NOT work as-is. Sed-rewrite below. + +--- + +## 8. Kubeconfig server-URL rewrite + +```bash +ssh ubuntu@$CAPI_MGMT_METAL_IP "CAPI_MGMT_METAL_IP=$CAPI_MGMT_METAL_IP bash -s" <<'REMOTE_EOF' +set -euo pipefail + +# Copy k3s kubeconfig to ubuntu user; rewrite server URL +mkdir -p /home/ubuntu/.kube +sudo cp /etc/rancher/k3s/k3s.yaml /home/ubuntu/.kube/config +sudo chown ubuntu:ubuntu /home/ubuntu/.kube/config +chmod 600 /home/ubuntu/.kube/config + +# Rewrite 127.0.0.1 → metal IP +sed -i "s|server: https://127.0.0.1:6443|server: https://$CAPI_MGMT_METAL_IP:6443|" \ + /home/ubuntu/.kube/config + +# Verify rewrite +grep "server:" /home/ubuntu/.kube/config +# Expect: server: https://10.12.8.21:6443 + +# Confirm kubectl works as ubuntu user (no sudo) +kubectl get nodes +REMOTE_EOF +``` + +--- + +## 9. helm + clusterctl install + +```bash +ssh ubuntu@$CAPI_MGMT_METAL_IP "HELM_VERSION=$HELM_VERSION CAPI_VERSION=$CAPI_VERSION bash -s" <<'REMOTE_EOF' +set -euo pipefail + +# helm install (get-helm-3 fetches the version we specify) +cd /tmp +curl -sfL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 \ + | DESIRED_VERSION="$HELM_VERSION" bash +helm version --short + +# clusterctl install +CLUSTERCTL_URL="https://github.com/kubernetes-sigs/cluster-api/releases/download/${CAPI_VERSION}/clusterctl-linux-amd64" +sudo curl -sfL "$CLUSTERCTL_URL" -o /usr/local/bin/clusterctl +sudo chmod +x /usr/local/bin/clusterctl +clusterctl version +REMOTE_EOF +``` + +--- + +## 10. clusterctl init (CAPI controllers + cert-manager + ORC + CAPO + CK8s) + +```bash +ssh ubuntu@$CAPI_MGMT_METAL_IP "CK8S_VERSION=$CK8S_VERSION CERT_MANAGER_VERSION=$CERT_MANAGER_VERSION ORC_VERSION=$ORC_VERSION CAPO_VERSION=$CAPO_VERSION bash -s" <<'REMOTE_EOF' +set -euo pipefail + +# Configure clusterctl with provider URLs +mkdir -p ~/.cluster-api +cat > ~/.cluster-api/clusterctl.yaml </dev/null || true +kubectl wait --for=condition=Available --timeout=5m \ + deployment --all -n capo-system +kubectl wait --for=condition=Available --timeout=5m \ + deployment --all -n cert-manager + +# Install ORC +kubectl apply -f "https://github.com/k-orc/openstack-resource-controller/releases/${ORC_VERSION}/orc.yaml" +kubectl wait --for=condition=Available --timeout=5m \ + deployment --all -n orc-system + +# Confirm all controllers +kubectl get pods -A | grep -v "Running\|Completed" | grep -v NAME +# Expected: empty output (all pods Running or no abnormal state) +REMOTE_EOF +``` + +> **Gotcha:** the actual namespace names (`capi-system`, `capo-system`, etc.) +> are conventions. If a controller fails to land in the expected namespace, +> `kubectl get deployment -A` lists all deployments — diagnose from there. + +--- + +## 11. Cloud-side prep (Keystone, Nova, Glance) + +Back on the jumphost: + +```bash +source $HOME/admin-openrc + +# Inventory existing resources FIRST (Bobcat lesson: don't create duplicates) +echo "=== Existing images ===" +openstack image list -c ID -c Name -f json | jq -r '.[] | "\(.Name)\t\(.ID)"' +echo "" +echo "=== Existing flavors ===" +openstack flavor list -c Name -c ID -c RAM -c VCPUs -c Disk -f json \ + | jq -r '.[] | "\(.Name)\tRAM=\(.RAM)\tCPU=\(.VCPUs)\tDisk=\(.Disk)\tID=\(.ID)"' +echo "" +echo "=== Existing keypairs ===" +openstack keypair list +echo "" +echo "=== Existing projects in admin_domain ===" +openstack project list --domain admin_domain +``` + +**Create / verify resources:** + +```bash +# Keystone project + user +openstack project show capi-mgmt --domain admin_domain 2>/dev/null \ + || openstack project create capi-mgmt --domain admin_domain --description "CAPI management plane" + +openstack user show capo --domain admin_domain 2>/dev/null \ + || openstack user create capo --domain admin_domain --password-prompt --description "CAPO operator" + +# Role assignments (CAPO needs member + load-balancer_member at minimum; +# admin works for testcloud — Roosevelt should use least-privilege) +openstack role add --user capo --user-domain admin_domain \ + --project capi-mgmt --project-domain admin_domain \ + member + +openstack role add --user capo --user-domain admin_domain \ + --project capi-mgmt --project-domain admin_domain \ + load-balancer_member 2>/dev/null || \ + echo "(load-balancer_member role may not exist if Octavia not deployed yet)" + +# Application credential — captured to file under $HOME (snap confinement) +APP_CRED_FILE=$WORK/capo-app-cred.json +openstack --os-username capo --os-user-domain-name admin_domain \ + --os-project-name capi-mgmt --os-project-domain-name admin_domain \ + application credential create capo-app-cred \ + --description "CAPO operator app credential" \ + -f json > "$APP_CRED_FILE" +chmod 600 "$APP_CRED_FILE" + +# Extract credential ID + secret +export APP_CRED_ID=$(jq -r '.id' "$APP_CRED_FILE") +export APP_CRED_SECRET=$(jq -r '.secret' "$APP_CRED_FILE") +echo "App cred ID: $APP_CRED_ID" +``` + +**Nova keypair (workload node SSH key):** + +```bash +# Generate fresh keypair locally (do NOT reuse jumphost personal key) +ssh-keygen -t ed25519 -N '' -f "$WORK/capi-workload-key" \ + -C "capi-workload-$(date +%Y%m%d)" +chmod 600 "$WORK/capi-workload-key" + +# Upload public key to Keystone as a Nova keypair +openstack keypair create --public-key "$WORK/capi-workload-key.pub" capi-workload-key +openstack keypair show capi-workload-key +``` + +**Workload image:** + +```bash +# Inventory check — use noble-amd64 if it exists (Bobcat lesson: do NOT create ubuntu-24.04-capi as a dup) +NOBLE_IMAGE_ID=$(openstack image show noble-amd64 -c id -f value 2>/dev/null || echo "") + +if [ -z "$NOBLE_IMAGE_ID" ]; then + echo "noble-amd64 image not found — upload required." + echo "(Pull from https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img" + echo " then: openstack image create --disk-format qcow2 --container-format bare \\" + echo " --public --file noble-server-cloudimg-amd64.img noble-amd64)" + exit 1 +fi +echo "Using image: noble-amd64 ($NOBLE_IMAGE_ID)" +export WORKLOAD_IMAGE_ID=$NOBLE_IMAGE_ID +``` + +**Workload flavor:** + +```bash +openstack flavor show capi-mgmt-node 2>/dev/null \ + || openstack flavor create capi-mgmt-node \ + --vcpus 4 --ram 4096 --disk 30 \ + --description "CAPI workload node (control plane sizing)" + +export WORKLOAD_FLAVOR=capi-mgmt-node +``` + +--- + +## 12. clouds.yaml + cloud.conf composition (with Vault CA, no tls-insecure) + +The workload cluster's OCCM (OpenStack Cloud Controller Manager) and CAPO both +need to call OpenStack APIs. Two files: + +- `clouds.yaml` — CAPO's view of how to reach OpenStack (used at cluster + creation time on capi-mgmt) +- `cloud.conf` — OCCM's view, injected into the workload cluster's k8s + Secret (used continuously by OCCM running in the workload cluster) + +**Compose clouds.yaml:** + +```bash +cat > "$WORK/clouds.yaml" < "$WORK/clouds.yaml.b64" +``` + +**Compose cloud.conf** (INI format, NOT YAML): + +```bash +cat > "$WORK/cloud.conf" < "$WORK/cloud.conf.b64" +``` + +> **Critical delta from Bobcat:** the `ca-file` line replaces `tls-insecure=true`. +> The path `/usr/local/share/ca-certificates/vault-ca.crt` exists on capi-mgmt +> (from §6) AND will be injected into workload nodes via CK8sConfig in §13. + +**base64-encode Vault CA for CK8sConfig injection:** + +```bash +base64 -w0 "$VAULT_CA" > "$WORK/vault-ca.crt.b64" +wc -c "$WORK/vault-ca.crt.b64" +``` + +--- + +## 13. Cluster template rendering (with Vault CA injection) + +The cluster template defines: + +- Cluster object +- OpenStackCluster (CAPO infrastructure) +- CK8sControlPlane +- CK8sConfigTemplate (control plane bootstrap — includes Vault CA injection) +- MachineDeployment + CK8sConfigTemplate (workers — includes Vault CA injection) +- Secrets for clouds.yaml and cloud.conf + +Variables (18 total): + +```bash +export CLUSTER_NAME=capi-mgmt-cluster +export CLUSTER_NAMESPACE=default +export KUBERNETES_VERSION=v1.31.4 # adjust to CK8s-supported +export CONTROL_PLANE_MACHINE_COUNT=1 # 3 for HA on Roosevelt +export WORKER_MACHINE_COUNT=2 # 3 on Roosevelt +export OPENSTACK_DNS_NAMESERVERS=10.12.4.227 # designate VIP +export OPENSTACK_FAILURE_DOMAIN=nova +export OPENSTACK_EXTERNAL_NETWORK_ID=$(openstack network show ext_net -c id -f value) +export OPENSTACK_IMAGE_NAME=noble-amd64 +export OPENSTACK_FLAVOR=capi-mgmt-node +export OPENSTACK_SSH_KEY_NAME=capi-workload-key +export POD_CIDR=10.244.0.0/16 +export SERVICE_CIDR=10.96.0.0/12 +export CLOUDS_YAML_B64=$(cat "$WORK/clouds.yaml.b64") +export CLOUD_CONF_B64=$(cat "$WORK/cloud.conf.b64") +export VAULT_CA_B64=$(cat "$WORK/vault-ca.crt.b64") +export CLUSTER_DOMAIN=cluster.local +export OPENSTACK_CLOUD=capi-mgmt + +# Sanity print +env | grep -E "^(CLUSTER|KUBERNETES|CONTROL_PLANE|WORKER|OPENSTACK|POD|SERVICE|VAULT|CLOUD)" \ + | grep -v "B64\|SECRET\|PASS" | sort +``` + +**Render the cluster template:** + +```bash +cat > "$WORK/cluster-template.yaml" <<'TEMPLATE_EOF' +apiVersion: v1 +kind: Secret +metadata: + name: ${CLUSTER_NAME}-cloud-config + namespace: ${CLUSTER_NAMESPACE} +type: Opaque +data: + clouds.yaml: ${CLOUDS_YAML_B64} + cloud.conf: ${CLOUD_CONF_B64} + cacert: ${VAULT_CA_B64} +--- +apiVersion: cluster.x-k8s.io/v1beta1 +kind: Cluster +metadata: + name: ${CLUSTER_NAME} + namespace: ${CLUSTER_NAMESPACE} +spec: + clusterNetwork: + pods: + cidrBlocks: + - ${POD_CIDR} + services: + cidrBlocks: + - ${SERVICE_CIDR} + serviceDomain: ${CLUSTER_DOMAIN} + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: OpenStackCluster + name: ${CLUSTER_NAME} + controlPlaneRef: + apiVersion: controlplane.cluster.x-k8s.io/v1beta2 + kind: CK8sControlPlane + name: ${CLUSTER_NAME}-control-plane +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: OpenStackCluster +metadata: + name: ${CLUSTER_NAME} + namespace: ${CLUSTER_NAMESPACE} +spec: + identityRef: + name: ${CLUSTER_NAME}-cloud-config + cloudName: ${OPENSTACK_CLOUD} + externalNetwork: + id: ${OPENSTACK_EXTERNAL_NETWORK_ID} + managedSecurityGroups: + allowAllInClusterTraffic: true + apiServerLoadBalancer: + enabled: true +--- +apiVersion: controlplane.cluster.x-k8s.io/v1beta2 +kind: CK8sControlPlane +metadata: + name: ${CLUSTER_NAME}-control-plane + namespace: ${CLUSTER_NAMESPACE} +spec: + replicas: ${CONTROL_PLANE_MACHINE_COUNT} + version: ${KUBERNETES_VERSION} + machineTemplate: + infrastructureTemplate: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: OpenStackMachineTemplate + name: ${CLUSTER_NAME}-control-plane + spec: + files: + - path: /usr/local/share/ca-certificates/vault-ca.crt + owner: root:root + permissions: "0644" + contentFrom: + secret: + name: ${CLUSTER_NAME}-cloud-config + key: cacert + preRunCommands: + - update-ca-certificates + extraKubeAPIServerArgs: + "--cloud-provider": external +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: OpenStackMachineTemplate +metadata: + name: ${CLUSTER_NAME}-control-plane + namespace: ${CLUSTER_NAMESPACE} +spec: + template: + spec: + flavor: ${OPENSTACK_FLAVOR} + image: + filter: + name: ${OPENSTACK_IMAGE_NAME} + sshKeyName: ${OPENSTACK_SSH_KEY_NAME} + identityRef: + name: ${CLUSTER_NAME}-cloud-config + cloudName: ${OPENSTACK_CLOUD} +--- +apiVersion: cluster.x-k8s.io/v1beta1 +kind: MachineDeployment +metadata: + name: ${CLUSTER_NAME}-md-0 + namespace: ${CLUSTER_NAMESPACE} +spec: + clusterName: ${CLUSTER_NAME} + replicas: ${WORKER_MACHINE_COUNT} + selector: + matchLabels: {} + template: + spec: + clusterName: ${CLUSTER_NAME} + version: ${KUBERNETES_VERSION} + bootstrap: + configRef: + apiVersion: bootstrap.cluster.x-k8s.io/v1beta2 + kind: CK8sConfigTemplate + name: ${CLUSTER_NAME}-md-0 + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: OpenStackMachineTemplate + name: ${CLUSTER_NAME}-md-0 +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: OpenStackMachineTemplate +metadata: + name: ${CLUSTER_NAME}-md-0 + namespace: ${CLUSTER_NAMESPACE} +spec: + template: + spec: + flavor: ${OPENSTACK_FLAVOR} + image: + filter: + name: ${OPENSTACK_IMAGE_NAME} + sshKeyName: ${OPENSTACK_SSH_KEY_NAME} + identityRef: + name: ${CLUSTER_NAME}-cloud-config + cloudName: ${OPENSTACK_CLOUD} +--- +apiVersion: bootstrap.cluster.x-k8s.io/v1beta2 +kind: CK8sConfigTemplate +metadata: + name: ${CLUSTER_NAME}-md-0 + namespace: ${CLUSTER_NAMESPACE} +spec: + template: + spec: + files: + - path: /usr/local/share/ca-certificates/vault-ca.crt + owner: root:root + permissions: "0644" + contentFrom: + secret: + name: ${CLUSTER_NAME}-cloud-config + key: cacert + preRunCommands: + - update-ca-certificates +TEMPLATE_EOF + +# envsubst to render +envsubst < "$WORK/cluster-template.yaml" > "$WORK/cluster-rendered.yaml" + +# Validate as YAML +python3 -c "import yaml; list(yaml.safe_load_all(open('$WORK/cluster-rendered.yaml'))); print('YAML OK')" + +# Quick visual check — no leftover ${...} markers +grep -n '\${' "$WORK/cluster-rendered.yaml" || echo "No unsubstituted variables — good" +``` + +> **CK8sConfig field name caveat:** the exact field names (`files`, +> `preRunCommands`) and their `contentFrom.secret` schema are CK8s-version- +> dependent. If `clusterctl init` failed earlier with schema warnings, +> consult the CK8s release notes for the pinned `$CK8S_VERSION`. + +--- + +## 14. Apply + poll-to-Ready + +Transfer rendered template to capi-mgmt and apply: + +```bash +scp "$WORK/cluster-rendered.yaml" ubuntu@$CAPI_MGMT_METAL_IP:/home/ubuntu/cluster.yaml + +ssh ubuntu@$CAPI_MGMT_METAL_IP <<'EOF' +set -euo pipefail +kubectl apply -f /home/ubuntu/cluster.yaml +echo "Applied. Waiting for cluster Available status (15-min timeout)..." + +for i in $(seq 1 90); do + STATUS=$(kubectl get cluster capi-mgmt-cluster -o json 2>/dev/null \ + | jq -r '.status.phase // "Unknown"') + READY=$(kubectl get cluster capi-mgmt-cluster -o json 2>/dev/null \ + | jq -r '.status.conditions[]? | select(.type=="Ready") | .status' \ + | head -1) + echo "$(date -Is) phase=$STATUS ready=$READY" + [ "$READY" = "True" ] && { echo "Cluster Ready"; break; } + sleep 10 +done + +kubectl get cluster,machines,kubeadmcontrolplane,machinedeployment -A +EOF +``` + +**If the poll times out before Ready,** typical diagnosis: + +```bash +ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl describe cluster capi-mgmt-cluster +ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl get machines -A +ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl logs -n capo-system deployment/capo-controller-manager --tail=100 +``` + +Common causes: + +- OpenStack API unreachable from capi-mgmt → check Vault CA install on capi-mgmt (§6) +- Image / flavor / network ID wrong in cluster template → re-check §11 variables +- Security group rules block kube-api LB → CAPO usually handles this; check OpenStackCluster status +- Application credential expired / wrong → re-check `$APP_CRED_ID` + +--- + +## 15. Extract workload kubeconfig + +```bash +ssh ubuntu@$CAPI_MGMT_METAL_IP -- clusterctl get kubeconfig capi-mgmt-cluster \ + > "$WORK/capi-mgmt-cluster.kubeconfig" +chmod 600 "$WORK/capi-mgmt-cluster.kubeconfig" + +# Sanity-check the workload cluster is reachable +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get nodes +# Expect: 1 control plane + 2 workers, all Ready +``` + +If `get nodes` times out, the cluster's API LB may not have allocated its +external IP yet, or the firewall rules don't permit jumphost → workload API: + +```bash +# What IP is the cluster's API LB on? +ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl get openstackcluster capi-mgmt-cluster \ + -o json | jq '.status.externalNetwork, .status.controlPlaneEndpoint' + +# Test reachability +curl -sk --max-time 10 "https://:6443/version" && echo " ← reachable" || echo "API LB unreachable" +``` + +--- + +## 16. `clusterctl init` on target (workload cluster) + +The workload cluster must have the same CAPI providers installed before `move`. + +```bash +# Run from jumphost using the workload kubeconfig +KUBECONFIG="$WORK/capi-mgmt-cluster.kubeconfig" clusterctl init \ + --core "cluster-api:${CAPI_VERSION}" \ + --infrastructure "openstack:${CAPO_VERSION}" \ + --bootstrap "canonical-kubernetes:${CK8S_VERSION}" \ + --control-plane "canonical-kubernetes:${CK8S_VERSION}" \ + --cert-manager-version "${CERT_MANAGER_VERSION}" + +# ORC into workload cluster too +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" apply \ + -f "https://github.com/k-orc/openstack-resource-controller/releases/${ORC_VERSION}/orc.yaml" + +# Wait for everything Available +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" wait \ + --for=condition=Available --timeout=5m \ + deployment --all -n capi-system +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" wait \ + --for=condition=Available --timeout=5m \ + deployment --all -n capo-system +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" wait \ + --for=condition=Available --timeout=5m \ + deployment --all -n cert-manager +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" wait \ + --for=condition=Available --timeout=5m \ + deployment --all -n orc-system +``` + +> **cert-manager double-install caveat:** if CK8s already installed +> cert-manager during workload bootstrap, the second `clusterctl init` may +> warn or skip. Check existing cert-manager version against `$CERT_MANAGER_VERSION` +> — if they differ, version-skew issues may surface post-pivot. Adjust the +> pin in §4 or accept the existing version. Roosevelt's standard practice +> is to install cert-manager via `clusterctl init` only (don't pre-install +> via CK8s) — same approach valid here if you want clean version control. + +--- + +## 17. `clusterctl move` pivot + +Move all CAPI CRs from bootstrap k3s → workload cluster: + +```bash +# Stage the target kubeconfig on capi-mgmt (where clusterctl move runs) +scp "$WORK/capi-mgmt-cluster.kubeconfig" ubuntu@$CAPI_MGMT_METAL_IP:/home/ubuntu/target.kubeconfig + +# Dry-run first to catch issues before commit +ssh ubuntu@$CAPI_MGMT_METAL_IP -- clusterctl move \ + --to-kubeconfig=/home/ubuntu/target.kubeconfig \ + --dry-run + +# Inspect dry-run output: list of objects to be moved. Should include: +# - Cluster, OpenStackCluster, OpenStackClusterTemplate +# - Secrets (cloud-config) +# - Machine objects, OpenStackMachineTemplate +# - CK8sControlPlane, CK8sConfigTemplate +# - MachineDeployment +# Should NOT include cert-manager state (cert-manager manages its own state +# on each cluster independently) +``` + +**If dry-run looks correct, execute the move:** + +```bash +ssh ubuntu@$CAPI_MGMT_METAL_IP -- clusterctl move \ + --to-kubeconfig=/home/ubuntu/target.kubeconfig + +# Move can take several minutes. Output ends with: "moved successfully" +``` + +--- + +## 18. Post-pivot verification + +```bash +echo "=== Bootstrap k3s (should now be empty of cluster CRs) ===" +ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl get cluster -A +# Expect: No resources found (or only a header) + +ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl get machines -A +# Expect: No resources found + +ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl get openstackcluster -A +# Expect: No resources found + +echo "" +echo "=== Workload cluster (should now own its own cluster CRs) ===" +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get cluster -A +# Expect: capi-mgmt-cluster shown, phase=Provisioned + +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get machines -A +# Expect: 3 machines (1 control-plane + 2 workers), all Running + +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get openstackcluster -A + +echo "" +echo "=== CAPI controllers in workload ===" +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get pods -A \ + | grep -E "(capi|capo|orc|cert-manager)" | grep -v "Running\|Completed" +# Expect: empty (all controller pods Running) + +echo "" +echo "=== OCCM not crash-looping (CRITICAL — main goal of TLS-verify work) ===" +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get pods -n kube-system \ + -l k8s-app=openstack-cloud-controller-manager +# Expect: 1 pod Running, NOT CrashLoopBackOff + +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" logs -n kube-system \ + -l k8s-app=openstack-cloud-controller-manager --tail=50 \ + | grep -iE "(tls|cert|error)" | head -20 +# Expect: no TLS/cert errors; OCCM should be healthy +``` + +> **If OCCM crash-loops with "x509: certificate signed by unknown authority":** +> Vault CA distribution failed. Check (a) `/usr/local/share/ca-certificates/vault-ca.crt` +> exists on workload nodes; (b) `update-ca-certificates` ran (check `/etc/ssl/certs/ca-certificates.crt` +> for the Vault CA's subject); (c) the secret reference in CK8sConfigTemplate +> matched the secret name. SSH into a worker via the jumphost key (`ssh -i +> $WORK/capi-workload-key ubuntu@`) to diagnose. + +--- + +## 19. Handoff to runbook 05 + +The workload kubeconfig at `$WORK/capi-mgmt-cluster.kubeconfig` is the input to +`runbooks/05-magnum-capi-driver.md`. Copy it to a stable path: + +```bash +mkdir -p $HOME/magnum-capi +cp "$WORK/capi-mgmt-cluster.kubeconfig" $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig +chmod 600 $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig +echo "Workload kubeconfig staged at: $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig" +``` + +> **Important — post-pivot semantic shift from Bobcat:** Magnum's +> `kubeconfig_file` setting (under `[capi_helm]` in +> `/etc/magnum/magnum.conf.d/99-capi.conf`, per D-007) now points to the +> workload cluster, not the bootstrap k3s. Bobcat had Magnum pointing at +> bootstrap k3s because the pivot was never executed. With pivot mandatory, +> Magnum's CAPI calls flow: +> +> ``` +> Magnum/leader → workload cluster API → CAPI controllers (running in workload) +> → create new Cluster CRs (tenant Magnum clusters) +> ``` +> +> The bootstrap k3s on capi-mgmt is now disposable. If you wanted, you could +> destroy capi-mgmt entirely at this point — the workload cluster manages +> itself. (Roosevelt may actually do this for cost savings.) For v1 testcloud, +> leave capi-mgmt running so its k3s can be inspected for diagnostics. + +--- + +## 20. Roosevelt deltas (forward-look) + +| Aspect | Testcloud (v1) | Roosevelt | +|---|---|---| +| Workload image | Default `noble-amd64` from cloud-images.ubuntu.com | Custom image baked with Vault CA pre-installed (no runtime install step) | +| Vault CA distribution | CK8sConfig `files:` + `preRunCommands:` (this runbook) | Image-baked + CK8sConfig (defense in depth) | +| App credential lifetime | No expiry set (testcloud) | Short-lived rotating credentials via Vault auth method | +| Workload cluster control plane | 1 node | 3 nodes (HA) | +| Workload cluster workers | 2 nodes | Per-tenant sizing; HPA-driven | +| `clusterctl init --cert-manager-version` | Pin from §4 | Pin to Vault PKI cert-manager profile (separate Roosevelt prep) | +| capi-mgmt VM lifecycle post-pivot | Kept running for diagnostics | Destroyed (cost savings; pivot makes it disposable) | +| Version pinning record | `$HOME/deploy-records//capi-pins/` | Same pattern, captured in Vault as audit artifact | +| Authentication to GitHub API | Optional PAT | Mandatory PAT (avoid rate-limit during automated rebuilds) | + +--- + +## 21. Rotation/refresh of pins + +The pins captured in §4 will age. Recommended cadence: + +- **Per rebuild:** re-discover all pins (Step 1 of next execution will catch + natural drift). +- **Out-of-band patch:** if a CVE drops for any pinned component, run §4 + discovery alone and capture the new pin into `$DEPLOY_RECORD/`. Then for + the affected component only, follow the upgrade procedure from its + upstream docs (does NOT necessarily require this whole runbook re-run). + +For Roosevelt, this becomes a tracked maintenance window task. + +--- + +## 22. Change log + +| Date | Change | Reference | +|---|---|---| +| 2026-05-22 | Document created. Vault CA distribution (no tls-insecure), mandatory `clusterctl move` pivot, pin-at-execution version model. | Workstream 3b | diff --git a/runbooks/deprecated/05-magnum-capi-driver.md b/runbooks/deprecated/05-magnum-capi-driver.md new file mode 100644 index 0000000..e1414e1 --- /dev/null +++ b/runbooks/deprecated/05-magnum-capi-driver.md @@ -0,0 +1,529 @@ +# Runbook 05 — Magnum CAPI Helm driver install + +**Status:** Executes after `04-magnum-domain.md` (Keystone wiring) and +`04a-capi-bootstrap-cluster.md` (workload cluster + kubeconfig staged). +Final post-deploy step to make Magnum capable of creating CAPI-managed +tenant K8s clusters. + +**Cross-references:** +- D-007 Layer B (Magnum two-layer install) +- D-017 (CAPI bootstrap cluster lifecycle) +- Runbook 04a §19 (workload kubeconfig handoff) +- Workstream 3c decision (2026-05-22): magnum-capi-helm 1.1.0 from PyPI; workload-cluster kubeconfig (NOT bootstrap k3s) + +**Known doc inconsistency (tracked for cleanup):** +D-007's Layer B currently states the kubeconfig points at "capi-mgmt.maas +bootstrap k3s". That language is correct for Bobcat (no pivot) but obsolete +post-workstream-3b (pivot mandatory). This runbook uses the workload cluster +kubeconfig as the canonical target. D-007 patch to follow in a workstream-3 +cleanup commit. + +--- + +## 1. Purpose & scope + +Graft the CAPI Helm driver onto the Charmed Magnum deployment so that +`openstack coe cluster create` provisions tenant K8s clusters via CAPI (in +the workload cluster) instead of via the deprecated Heat driver. + +**Output of this runbook:** + +- `magnum-capi-helm==1.1.0` installed on the magnum unit's system Python. +- `/etc/magnum/kubeconfig` populated with the workload cluster's + kubeconfig (post-pivot CAPI controller plane). +- `/etc/magnum/magnum.conf.d/99-capi.conf` configured with + `enabled_drivers = k8s_capi_helm_v1` and `[capi_helm] kubeconfig_file=`. +- Systemd overrides on `magnum-api` and `magnum-conductor` that replace + the init.d wrapper's ExecStart with explicit `--config-dir` invocation. +- Both services running cleanly with the CAPI driver loaded. + +**Scope:** v1 testcloud. Roosevelt deltas in §12. + +**Out of scope:** +- Magnum domain setup (runbook 04) +- Workload cluster lifecycle (runbook 04a) +- Smoketest tenant cluster creation is OPTIONAL (§11) — full validation + framework belongs in runbook 08. + +--- + +## 2. Decisions captured + +| Decision | Choice | Reason | +|---|---|---| +| Driver pin | `magnum-capi-helm==1.1.0` from PyPI | D-007 correction (stackhpc fork archived Dec 2024; canonical project on opendev/PyPI; 1.1.0 is last Caracal-cycle release) | +| Install method | `pip3 install --break-system-packages` | PEP 668 — Ubuntu 22.04+ requires explicit override for system-site-packages install | +| Install scope | System Python on magnum unit (not venv) | Magnum charm uses system-packaged python at `/usr/lib/python3/dist-packages/magnum/`; driver must import from same site | +| Kubeconfig target | Workload cluster (post-pivot) | Workstream 3b — bootstrap k3s is empty post-pivot; CAPI controllers live in workload | +| Kubeconfig source | `$HOME/magnum-capi/capi-mgmt-cluster.kubeconfig` (staged by 04a §19) | Documented handoff | +| Driver entry-point name | `k8s_capi_helm_v1` | Per upstream magnum-capi-helm 1.1.0; verify in §10 | +| Conf.d filename | `99-capi.conf` | Numeric prefix ensures it loads AFTER any charm-managed conf, so `enabled_drivers` override wins | +| File encoding | ASCII-only | Non-ASCII in conf.d causes silent magnum daemon failures (handoff lesson; cf. Horizon `local_settings.d` issue) | +| Trustee credential | Existing magnum-shared user (charm-managed) | Roosevelt will use app-credential pattern | + +--- + +## 3. Prerequisites + +| Prereq | Verification | +|---|---| +| Magnum charm active/idle | `juju status magnum \| grep magnum/0` shows `active idle` | +| Magnum domain setup completed (runbook 04) | `openstack domain show magnum \| grep enabled` returns `True` | +| Workload cluster reachable from jumphost | `kubectl --kubeconfig $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig get nodes` returns Ready nodes | +| CAPI controllers running in workload cluster | `kubectl --kubeconfig $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig get pods -n capi-system \| grep -v Running \| grep -v NAME` empty | +| Workload kubeconfig staged at expected path | `test -r $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig && stat -c %a $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig` shows `600` | +| `juju exec` works to magnum/leader (use exec, NOT ssh, for non-interactive — handoff lesson) | `juju exec --unit magnum/leader -- hostname` returns the unit hostname | + +**Set shell context:** + +```bash +export WORK=$HOME/magnum-capi +export WORKLOAD_KUBECONFIG=$WORK/capi-mgmt-cluster.kubeconfig +export DRIVER_VERSION=magnum-capi-helm==1.1.0 # per D-007 correction +cd "$WORK" +``` + +> **`juju ssh` vs `juju exec` choice:** the handoff lessons explicitly call +> out that `juju ssh` hangs when stdout is redirected (PTY allocation issue). +> This runbook uses `juju exec` for all non-interactive command execution and +> reserves `juju ssh` only for cases where you actually want an interactive +> shell. + +--- + +## 4. Pre-flight: capture current state + +Capture the magnum unit's state BEFORE making changes. Useful for diagnosis +if anything goes wrong, and as a record of what was changed. + +```bash +mkdir -p "$WORK/pre-state" + +# Service unit files (as managed by charm) +juju exec --unit magnum/leader -- \ + 'sudo systemctl cat magnum-api magnum-conductor 2>&1' \ + > "$WORK/pre-state/systemd-units.txt" + +# Currently-enabled drivers +juju exec --unit magnum/leader -- \ + 'sudo grep -r enabled_drivers /etc/magnum/ 2>/dev/null || echo "(no enabled_drivers found — charm default applies)"' \ + > "$WORK/pre-state/drivers-pre.txt" + +# Python site-packages — see what's already installed +juju exec --unit magnum/leader -- \ + 'sudo pip3 list 2>/dev/null | grep -iE "magnum|cluster|helm|kubernetes" || true' \ + > "$WORK/pre-state/pip-pre.txt" + +# conf.d state +juju exec --unit magnum/leader -- \ + 'sudo ls -la /etc/magnum/magnum.conf.d/ 2>/dev/null || echo "(no conf.d directory)"' \ + > "$WORK/pre-state/confd-pre.txt" + +# Service running state +juju exec --unit magnum/leader -- \ + 'sudo systemctl is-active magnum-api magnum-conductor' \ + > "$WORK/pre-state/service-state-pre.txt" + +# Display the captured state +cat "$WORK/pre-state/"*.txt +``` + +> **What to look for in pre-state:** the charm-managed `enabled_drivers` value +> probably includes Heat-based drivers (`heat_kubernetes`, etc.). The 99-capi.conf +> override in §7 replaces this with the single CAPI driver. The pre-state +> capture documents what was active before the override took effect. + +--- + +## 5. Install magnum-capi-helm 1.1.0 + +```bash +juju exec --unit magnum/leader -- \ + "sudo pip3 install $DRIVER_VERSION --break-system-packages" +``` + +**Verify install:** + +```bash +juju exec --unit magnum/leader -- \ + 'sudo pip3 show magnum-capi-helm | head -10' +# Expect: Name: magnum-capi-helm +# Version: 1.1.0 +# Location: /usr/lib/python3/dist-packages + +juju exec --unit magnum/leader -- \ + 'sudo python3 -c "import magnum_capi_helm; print(magnum_capi_helm.__file__)"' +# Expect: /usr/lib/python3/dist-packages/magnum_capi_helm/__init__.py +``` + +**Check that the driver entry point is registered:** + +```bash +juju exec --unit magnum/leader -- \ + 'sudo python3 -c " +from stevedore import driver +mgr = driver.DriverManager( + namespace=\"magnum.drivers\", + name=\"k8s_capi_helm_v1\", + invoke_on_load=False +) +print(\"Driver class:\", mgr.driver) +"' +# Expect: Driver class: +# (or similar — the actual class path is package-version-dependent) +``` + +> If the entry point check fails with "No 'k8s_capi_helm_v1' driver found", +> the driver name in 1.1.0 may differ from what D-007 documented. Inspect the +> installed package's `entry_points.txt`: +> +> ```bash +> juju exec --unit magnum/leader -- \ +> 'sudo cat /usr/lib/python3/dist-packages/magnum_capi_helm*.dist-info/entry_points.txt 2>/dev/null' +> ``` +> +> Find the entry under `[magnum.drivers]` — use that exact name in §7. + +--- + +## 6. Stage workload kubeconfig on magnum unit + +```bash +# Transfer kubeconfig from jumphost to magnum unit +juju scp "$WORKLOAD_KUBECONFIG" magnum/leader:/tmp/kubeconfig + +# Install with correct ownership/mode in one atomic step +juju exec --unit magnum/leader -- \ + 'sudo install -m 0640 -o root -g magnum /tmp/kubeconfig /etc/magnum/kubeconfig && sudo rm /tmp/kubeconfig' +``` + +**Verify:** + +```bash +juju exec --unit magnum/leader -- \ + 'sudo ls -la /etc/magnum/kubeconfig' +# Expect: -rw-r----- 1 root magnum ... /etc/magnum/kubeconfig + +# Confirm magnum user can read it +juju exec --unit magnum/leader -- \ + 'sudo -u magnum cat /etc/magnum/kubeconfig | head -3' +# Expect: apiVersion: v1 / clusters: / - cluster: + +# Confirm kubectl can use it from the magnum unit (sanity check on API reachability) +juju exec --unit magnum/leader -- \ + 'sudo -u magnum kubectl --kubeconfig /etc/magnum/kubeconfig get nodes 2>&1 | head -10' +# Expect: NAME ... STATUS=Ready for control plane + workers +# OR: kubectl not installed (acceptable — magnum-capi-helm uses Python client, not kubectl) +``` + +> **Why mode 0640 and group magnum:** kubeconfig contains auth tokens. Mode +> 0600 (owner-only) wouldn't let the `magnum` system user (which runs +> magnum-api/conductor) read it. Mode 0640 with `group: magnum` is the +> minimum-permission setup that works. NOT 0644 — keeps it off other users +> on the unit. + +--- + +## 7. Configure `/etc/magnum/magnum.conf.d/99-capi.conf` + +Generate the conf locally first (snap confinement does not apply to plain +bash on jumphost, but we keep paths under `$HOME` for consistency), then +transfer. + +**ASCII-only verification is critical** — the handoff documents non-ASCII +characters in `conf.d` files causing silent daemon failures (cf. Horizon +`local_settings.d`). Use plain straight quotes, ASCII dashes, no smart +typography. + +```bash +# Write locally +cat > "$WORK/99-capi.conf" <<'EOF' +[DEFAULT] +enabled_drivers = k8s_capi_helm_v1 + +[capi_helm] +kubeconfig_file = /etc/magnum/kubeconfig +EOF + +# Verify it is pure ASCII (no UTF-8 sneakers) +file "$WORK/99-capi.conf" +# Expect: ASCII text +# If it says "UTF-8 Unicode text", STOP and rewrite by hand — even one stray +# em-dash or smart quote will silently break magnum + +# Hex dump check (paranoid mode) +xxd "$WORK/99-capi.conf" | grep -v "^[0-9a-f]*: [0-9a-f ]* [a-zA-Z0-9 \[\]=._/]*$" | head -5 +# Expect: empty output (all bytes are printable ASCII) +``` + +**Stage and install:** + +```bash +juju scp "$WORK/99-capi.conf" magnum/leader:/tmp/99-capi.conf + +juju exec --unit magnum/leader -- \ + 'sudo mkdir -p /etc/magnum/magnum.conf.d && sudo install -m 0644 -o root -g root /tmp/99-capi.conf /etc/magnum/magnum.conf.d/99-capi.conf && sudo rm /tmp/99-capi.conf' + +# Verify +juju exec --unit magnum/leader -- \ + 'sudo ls -la /etc/magnum/magnum.conf.d/ && sudo cat /etc/magnum/magnum.conf.d/99-capi.conf' +# Expect: file listed; content matches what was written +``` + +--- + +## 8. Systemd override on magnum-api + magnum-conductor + +The Charmed Magnum unit files use a wrapper pattern: + +``` +ExecStart=/etc/init.d/magnum-api systemd-start +``` + +The wrapper does NOT pass `--config-dir` to magnum-api, so `/etc/magnum/magnum.conf.d/` +is never loaded. The 99-capi.conf would have no effect. + +Override with explicit `--config-file` + `--config-dir` invocation. + +**Generate override files locally:** + +```bash +cat > "$WORK/magnum-api-override.conf" <<'EOF' +[Service] +ExecStart= +ExecStart=/usr/bin/magnum-api --config-file=/etc/magnum/magnum.conf --config-dir=/etc/magnum/magnum.conf.d +EOF + +cat > "$WORK/magnum-conductor-override.conf" <<'EOF' +[Service] +ExecStart= +ExecStart=/usr/bin/magnum-conductor --config-file=/etc/magnum/magnum.conf --config-dir=/etc/magnum/magnum.conf.d +EOF + +# ASCII check +file "$WORK/magnum-api-override.conf" "$WORK/magnum-conductor-override.conf" +# Expect: ASCII text x2 +``` + +> **The empty `ExecStart=` line is critical.** Systemd accumulates ExecStart +> directives by default; an empty assignment is required to CLEAR the inherited +> directive before setting the replacement. Without the empty line, the unit +> would have BOTH the init.d wrapper AND the new direct invocation, and would +> likely fail to start. + +**Install on the unit:** + +```bash +juju scp "$WORK/magnum-api-override.conf" magnum/leader:/tmp/magnum-api-override.conf +juju scp "$WORK/magnum-conductor-override.conf" magnum/leader:/tmp/magnum-conductor-override.conf + +juju exec --unit magnum/leader -- \ + 'sudo mkdir -p /etc/systemd/system/magnum-api.service.d /etc/systemd/system/magnum-conductor.service.d && \ + sudo install -m 0644 -o root -g root /tmp/magnum-api-override.conf /etc/systemd/system/magnum-api.service.d/override.conf && \ + sudo install -m 0644 -o root -g root /tmp/magnum-conductor-override.conf /etc/systemd/system/magnum-conductor.service.d/override.conf && \ + sudo rm /tmp/magnum-api-override.conf /tmp/magnum-conductor-override.conf' + +# Reload systemd to pick up the overrides +juju exec --unit magnum/leader -- 'sudo systemctl daemon-reload' + +# Verify the overrides are effective (systemctl cat shows combined unit + overrides) +juju exec --unit magnum/leader -- 'sudo systemctl cat magnum-api | grep -A1 ExecStart' +# Expect: TWO ExecStart= lines — the empty clear-line and the new /usr/bin/magnum-api invocation +juju exec --unit magnum/leader -- 'sudo systemctl cat magnum-conductor | grep -A1 ExecStart' +# Expect: TWO ExecStart= lines as above for magnum-conductor +``` + +> **Charm reconciliation note:** the Magnum charm may rewrite its own systemd +> units on config changes or upgrades. The drop-in override at +> `/etc/systemd/system/magnum-api.service.d/override.conf` is OUTSIDE the +> charm's writable zone and should survive. Verify after any `juju refresh` or +> `juju config magnum` command by re-running the `systemctl cat` check above. + +--- + +## 9. Restart services + verify health + +```bash +juju exec --unit magnum/leader -- \ + 'sudo systemctl restart magnum-api magnum-conductor' + +# Wait briefly for services to initialize +sleep 5 + +# Check active state +juju exec --unit magnum/leader -- \ + 'sudo systemctl is-active magnum-api magnum-conductor' +# Expect: active (x2) + +# Examine recent journal for errors (the critical step — magnum's silent failure +# mode means we must read logs, not just trust is-active) +juju exec --unit magnum/leader -- \ + 'sudo journalctl -u magnum-api --since "2 minutes ago" --no-pager | tail -50' +juju exec --unit magnum/leader -- \ + 'sudo journalctl -u magnum-conductor --since "2 minutes ago" --no-pager | tail -50' +``` + +**Look for these red flags in the logs:** + +| Symptom | Likely cause | Remediation | +|---|---|---| +| `ImportError: No module named magnum_capi_helm` | §5 pip install failed | Re-run §5; check pip3 output | +| `EntryPointError: No 'k8s_capi_helm_v1' driver` | Driver entry-point name mismatch | Verify name per §5 footnote; update §7 | +| Service repeatedly restarts (look for "Started" appearing twice in 10s) | Likely a config error in 99-capi.conf | Re-check ASCII-only; check magnum.conf.d permissions | +| `kubeconfig_file` not honored | --config-dir not being passed | §8 override not active; re-run `systemctl daemon-reload` | +| Silent: no error but driver also not loading | Non-ASCII char snuck into a conf | `file /etc/magnum/magnum.conf.d/99-capi.conf` — if it says UTF-8, regenerate | + +--- + +## 10. CAPI driver enablement check + +Verify the driver is actually loaded by Magnum and reachable via the API. + +```bash +source $HOME/admin-openrc + +# List supported COE drivers via the Magnum API +openstack coe cluster template list -f json +# (empty templates list is fine — we are checking the endpoint responds) + +# Direct check on the unit: scan the service's loaded drivers +juju exec --unit magnum/leader -- \ + 'sudo journalctl -u magnum-conductor --since "5 minutes ago" --no-pager | grep -iE "driver|enabled" | head -20' +# Expect: a line mentioning k8s_capi_helm_v1 having been loaded +# (Magnum logs the loaded drivers at startup) + +# Definitive check: try creating a cluster template that requires the CAPI driver +openstack coe cluster template create magnum-capi-driver-check \ + --image noble-amd64 \ + --keypair capi-workload-key \ + --external-network ext_net \ + --master-flavor capi-mgmt-node \ + --flavor capi-mgmt-node \ + --coe kubernetes \ + --network-driver calico \ + --labels kube_tag=v1.31.4 + +openstack coe cluster template show magnum-capi-driver-check -c name -c coe -c labels +``` + +> **If template create fails with "driver not enabled" or similar:** the +> Magnum API process is not loading the conf.d. Verify the systemd override +> took effect — `sudo systemctl show magnum-api -p ExecStart` on the unit +> should show the explicit `--config-dir` invocation. If it still shows the +> init.d wrapper, the daemon-reload + restart did not pick up the override. + +**Cleanup the driver-check template:** + +```bash +openstack coe cluster template delete magnum-capi-driver-check +``` + +--- + +## 11. Optional smoketest — create a tenant CAPI cluster + +This step is **optional**. Full validation belongs in runbook 08. Use this +smoketest only if you want immediate confirmation that the entire chain +(Magnum API -> conductor -> magnum-capi-helm -> CAPI controllers in workload +cluster -> tenant K8s cluster on tenant VMs) works end-to-end. + +```bash +# Create a cluster template tuned for testcloud smoketest +openstack coe cluster template create magnum-smoketest-template \ + --image noble-amd64 \ + --keypair capi-workload-key \ + --external-network ext_net \ + --master-flavor capi-mgmt-node \ + --flavor capi-mgmt-node \ + --coe kubernetes \ + --network-driver calico \ + --labels boot_volume_size=20,kube_tag=v1.31.4,octavia_provider=ovn + +# Create a 1+1 cluster (minimum for smoketest) +openstack coe cluster create magnum-smoketest \ + --cluster-template magnum-smoketest-template \ + --master-count 1 \ + --node-count 1 + +# Poll for status (15-20 min typical; CAPI provisions tenant VMs end-to-end) +for i in $(seq 1 60); do + STATUS=$(openstack coe cluster show magnum-smoketest -c status -f value) + echo "$(date -Is) status=$STATUS" + case "$STATUS" in + CREATE_COMPLETE) echo "Smoketest passed"; break ;; + CREATE_FAILED) echo "Smoketest FAILED"; openstack coe cluster show magnum-smoketest; exit 1 ;; + esac + sleep 30 +done + +# Retrieve the smoketest cluster's kubeconfig +openstack coe cluster config magnum-smoketest --dir "$WORK/smoketest-kubeconfig" + +# Sanity-check the smoketest cluster +KUBECONFIG="$WORK/smoketest-kubeconfig/config" kubectl get nodes +KUBECONFIG="$WORK/smoketest-kubeconfig/config" kubectl get pods -A | head -20 + +# Cleanup the smoketest cluster +openstack coe cluster delete magnum-smoketest +openstack coe cluster template delete magnum-smoketest-template +``` + +> **What success looks like:** the CAPI controllers in the workload cluster +> receive the new Cluster CR (created by magnum-capi-helm in response to the +> Magnum API call), CAPO talks to OpenStack to provision tenant VMs, the +> tenant VMs join the new K8s cluster, and the new cluster has 1 control +> plane + 1 worker Ready. Octavia provides the API server LB (visible as a +> Floating IP in the tenant project). + +--- + +## 12. Roosevelt deltas (forward-look) + +| Aspect | Testcloud (v1) | Roosevelt | +|---|---|---| +| Driver pin source | PyPI `magnum-capi-helm==1.1.0` | Internal mirror with checksum verification | +| Driver pin record | Implicit in this runbook | Captured in Vault as audit artifact alongside CAPI pins | +| Kubeconfig source | Workload cluster (post-pivot per 04a §17) | Same | +| Kubeconfig rotation | Manual on capi-mgmt rebuild | Automated when workload cluster cert rotates | +| Trustee credential | Charm-default magnum-shared user | Per-tenant app credentials via Vault auth method | +| Magnum HA | num_units=1 (per D-009 testcloud) | num_units=3 with hacluster + provider VIP | +| Driver upgrade discipline | Manual re-run of §5 | Tracked maintenance window; Vault audit log | +| Systemd override | Drop-in at `/etc/systemd/system/magnum-*.service.d/override.conf` | Same — but provided via a charm overlay package, not manual file install | +| ASCII-only enforcement | Manual check (§7, §8) | Pre-flight lint in `scripts/pre-flight-checks.sh` | + +--- + +## 13. Documented runtime gotchas (carry-forward from handoff) + +These gotchas burned cycles during the Bobcat Magnum CAPI work. Each is +explicitly handled in this runbook; collecting them here for visibility: + +1. **PEP 668 `--break-system-packages`** (§5). Ubuntu 22.04+ refuses + `pip install` against system Python by default. The flag is required for + the magnum-capi-helm install path used by Charmed Magnum. +2. **`juju ssh` hangs on stdout redirect.** PTY allocation issue. + This runbook uses `juju exec` for all non-interactive command execution. +3. **Heredoc nesting in `juju ssh` is fragile.** This runbook writes + conf files locally first and uses `juju scp` + `juju exec install` to + transfer — single-level only. +4. **Non-ASCII characters in `conf.d` files cause silent daemon failures.** + §7 and §8 both include `file ` ASCII verification before transfer. +5. **`openstack -f value -c X -c Y` outputs in alphabetical field order, + not flag order.** This runbook uses single-column queries or `-f json | + jq` throughout. +6. **Charm-managed `enabled_drivers` is overridden, not appended.** The + `enabled_drivers = k8s_capi_helm_v1` line in 99-capi.conf REPLACES the + charm-default value (which would include the deprecated Heat drivers). +7. **The systemd override empty `ExecStart=` line is required** to clear + the inherited ExecStart before setting the replacement (§8). +8. **Snap-confined `openstack` CLI cannot read `/tmp`.** This runbook stages + files under `$WORK=$HOME/magnum-capi`. The smoketest in §11 also writes + to `$WORK/smoketest-kubeconfig`. + +--- + +## 14. Change log + +| Date | Change | Reference | +|---|---|---| +| 2026-05-22 | Document created. magnum-capi-helm 1.1.0 from PyPI; workload-cluster kubeconfig (post-pivot per workstream 3b); systemd override pattern; ASCII-only conf.d. | Workstream 3c | diff --git a/runbooks/deprecated/06-tenant-setup.md b/runbooks/deprecated/06-tenant-setup.md new file mode 100644 index 0000000..3915229 --- /dev/null +++ b/runbooks/deprecated/06-tenant-setup.md @@ -0,0 +1,41 @@ +# Runbook 06 — Tenant Resource Recreation + +**STATUS: PLACEHOLDER** — drafted post-deploy. + +## Purpose + +Recreate the standard testcloud tenant resources (domain, project, user, +networks, images, keypairs, flavors) using a proper IPAM-aligned design +per D-010 + D-016 (not the ad-hoc `user1` pattern from the original test +cloud). + +## Prerequisites + +- Cloud fully deployed and validated +- DNS zones populated (Runbook 07 may precede this if Designate-via-tenant + DNS is in scope at tenant create time) +- NetBox IPv4 tenant pool prefix present (per D-016; default `10.20.0.0/16`) + +## TODO + +- [ ] Create domain `domain1` +- [ ] Create project `project1` in domain `domain1` +- [ ] Create user `user1` in project1 (member role + load-balancer_member + role for Octavia) +- [ ] Tenant network with CIDR carved from NetBox IPv4 tenant pool + - Suggested convention: `10.20..0/24` per D-016 + - project1 → `10.20.1.0/24` + - Per D-016 hybrid model, the per-project /24 is Neutron-managed and + NOT added back to NetBox +- [ ] Tenant router connected to ext_net (Provider 10.12.4.0/22) +- [ ] Glance image: noble-amd64 (cloud-init enabled) +- [ ] Flavor m1.small (1 vCPU, 2 GiB RAM, 20 GiB root) +- [ ] Keypair for user1 +- [ ] openrc files: `~/admin-openrc`, `~/user1-openrc` +- [ ] Application credentials for user1 (audit trail) +- [ ] Take second KVM snapshot (per D-012 Snapshot 2) + +## v1 vs. v2 note + +In v1, tenant networks are IPv4-only. v2 adds IPv6 tenant subnets carved +from the v2 IPv6 tenant pool (currently reservation status in NetBox). diff --git a/runbooks/deprecated/07-dns-zones.md b/runbooks/deprecated/07-dns-zones.md new file mode 100644 index 0000000..3b780de --- /dev/null +++ b/runbooks/deprecated/07-dns-zones.md @@ -0,0 +1,36 @@ +# Runbook 07 — Designate Zones and Records (v1: A records only) + +**STATUS: PLACEHOLDER** — drafted post-deploy. + +## Purpose + +Create the cloud's DNS zones in Designate, populate API VIP A records +(v1: IPv4 only), and configure Neutron defaults to push Designate as +tenant DNS resolver. + +## Prerequisites + +- Designate charm in `active/idle` +- Keystone, Neutron API reachable +- API VIP hostnames already in `/etc/hosts` on all OpenStack nodes + (per D-008 Layer 0 bootstrap) + +## TODO + +- [ ] Create primary zone: + `openstack zone create --email admin@neumatrix.local \ + omega.dc0.vr0.cloud.neumatrix.local.` +- [ ] Populate API VIP **A** records for each public service: + - keystone, glance, nova, neutron, cinder, placement, octavia, + barbican, magnum, horizon, designate + - **v1: A records only** (IPv4 VIPs from the Provider API VIP range + 10.12.4.224-.254) + - **v2 will add AAAA records when IPv6 Provider VIPs become active** +- [ ] Configure Neutron defaults: + `juju config neutron-api default-dns-domain=omega.dc0.vr0.cloud.neumatrix.local.` + `juju config neutron-api dns-domain=omega.dc0.vr0.cloud.neumatrix.local.` +- [ ] Configure Neutron DHCP to push Designate as resolver: + `juju config neutron-api dns-servers=` +- [ ] Verify from a test tenant VM: + `nslookup keystone.omega.dc0.vr0.cloud.neumatrix.local` + resolves to Provider API VIP diff --git a/runbooks/deprecated/08-validate.md b/runbooks/deprecated/08-validate.md new file mode 100644 index 0000000..7f7acfc --- /dev/null +++ b/runbooks/deprecated/08-validate.md @@ -0,0 +1,33 @@ +# Runbook 08 — Validation (Roosevelt-Rehearsal Bar) + +**STATUS: PLACEHOLDER** — drafted with scripts/validate.sh. + +## Purpose + +Execute the validation criteria from D-011 and confirm the cloud is ready to +be considered a successful rebuild. + +## Prerequisites + +- All prior runbooks complete + +## Validation criteria (per D-011) + +- [ ] All charms `active/idle` in `juju status` +- [ ] All public API VIPs respond on FQDN from jumphost +- [ ] All public API VIPs respond on FQDN from a tenant VM (Option B path) +- [ ] Octavia LB pattern passes: create LB, two members, round-robin verified, + failover verified, recovery verified +- [ ] Magnum CAPI cluster create end-to-end: cluster template + cluster create, + OCCM does not crash-loop, cluster reaches CREATE_COMPLETE +- [ ] Vault unseal + auto-unseal-after-reboot pattern: reboot vault unit, + confirm auto-unseal via etcd (or manual unseal per HA pattern) +- [ ] Designate resolves API hostnames from tenant subnet +- [ ] Snapshot 1 (post-deploy, pre-tenant) taken (per D-012) +- [ ] Snapshot 2 (post-tenant) taken (per D-012) + +## TODO + +- [ ] Run `scripts/validate.sh` and capture output +- [ ] Document any divergences from validation criteria in + `docs/design-decisions.md` change log