# Runbook 04a — CAPI bootstrap cluster

**Status:** Executes after `02-deploy.md` (cloud up + all charms active/idle)
and `03-vault-init.md` (Vault initialized + root CA available). Precedes
`05-magnum-capi-driver.md` (driver graft consumes the workload kubeconfig
produced here).

**D-017 posture:** L3 full teardown and rebuild every deployment cycle.
Nothing is preserved across cycles. capi-mgmt is wiped to MAAS Ready on
teardown; rebuilt from scratch by this runbook.

**Cross-references:**
- D-017 (CAPI bootstrap cluster lifecycle)
- D-007 (Magnum two-layer install)
- D-002 (channel matrix — informs Vault CA chain)
- Workstream 3b decision (2026-05-22): ship Vault CA (no tls-insecure); pivot mandatory

---

## 1. Purpose & scope

This runbook stands up the CAPI bootstrap cluster on `capi-mgmt.maas` and
pivots cluster state into a self-managing workload cluster. Output:

1. **Workload K8s cluster** (`capi-mgmt-cluster`) running in tenant VMs on
   the cloud, self-managing post-pivot.
2. **Workload kubeconfig** copied to jumphost at a known path. Consumed by
   `runbooks/05-magnum-capi-driver.md` for the Magnum CAPI Helm driver
   graft.
3. **No remaining state** on the bootstrap k3s VM after pivot. capi-mgmt
   becomes a disposable jump host.

**Scope:** v1 testcloud. Roosevelt deltas in section 20.

**Out of scope:**

- Magnum-side configuration (runbook 05).
- Workload cluster's tenant lifecycle (Magnum's job, not this runbook's).
- Backup / DR for the workload cluster (Roosevelt concern).

---

## 2. Decisions captured

Per workstream 3b sign-off (2026-05-22):

| Decision | Choice | Roosevelt parallel |
|---|---|---|
| Version pinning | Pin-at-execution with discovery in §4 | Same pattern; pins captured in deploy record |
| Cloud TLS trust | Ship Vault CA to capi-mgmt + workload nodes (no `tls-insecure`) | Image-baked CA; CK8sConfig redundancy |
| `clusterctl move` pivot | Mandatory; workload cluster becomes self-managing | Same |
| K8s flavor | Canonical Kubernetes (CK8s) | Same |
| OpenStack auth | v3applicationcredential | Same |
| Pod CIDR | `10.244.0.0/16` | Same (does not conflict with cloud `10.12.0.0/16` or tenant pool `10.20.0.0/16`) |
| Service CIDR | `10.96.0.0/12` | Same |
| Workload cluster name | `capi-mgmt-cluster` | Same |
| Workload node SSH user | `ubuntu` (MAAS/cloud-init convention) | Same |

**Naming convention:**

- Keystone project for CAPI: `capi-mgmt` (in `admin_domain`)
- Keystone user for CAPI: `capo` (CAPO operator)
- App credential: `capo-app-cred`
- Workload image (Glance): `noble-amd64` (existing; do NOT duplicate as `ubuntu-24.04-capi` — Bobcat lesson)
- Workload flavor: `capi-mgmt-node` (4 vCPU / 4 GiB / 30 GB) — control plane node sizing

---

## 3. Prerequisites

| Prereq | Verification |
|---|---|
| Cloud deployed; all charms `active/idle` per D-011 | `juju status --color\| grep -v "active.*idle"` returns only the header |
| Vault initialized + unsealed | `juju ssh vault/leader -- sudo vault status` shows `Sealed=false` |
| Vault root CA available on jumphost | `test -f $HOME/vault-pki/root-ca.pem && openssl x509 -in $HOME/vault-pki/root-ca.pem -noout -subject` |
| Keystone reachable via FQDN | `curl -sf --cacert $HOME/vault-pki/root-ca.pem https://keystone.omega.dc0.vr0.cloud.neumatrix.local:5000/v3 \| jq .version.id` returns `"v3.14"` or current |
| capi-mgmt VM exists in MAAS as Ready | `maas $MAAS_PROFILE machines read \| jq '.[] \| select(.hostname=="capi-mgmt") \| .status_name'` returns `"Ready"` |
| Admin openrc available | `test -f $HOME/admin-openrc && source $HOME/admin-openrc && openstack token issue \| head -3` |
| Workspace path under $HOME (snap confinement) | `WORK=$HOME/capi-bootstrap; mkdir -p "$WORK"; cd "$WORK"; pwd` shows under home |

**Set shell context for the runbook:**

```bash
export REPO=$HOME/repos/openstack-caracal-ipv4   # adjust if your clone is elsewhere
export WORK=$HOME/capi-bootstrap                  # runbook scratch dir
export VAULT_CA=$HOME/vault-pki/root-ca.pem       # Vault root CA (from runbook 03)
export CAPI_MGMT_METAL_IP=10.12.8.21              # capi-mgmt metal interface
export CAPI_MGMT_PROVIDER_IP=10.12.4.21           # capi-mgmt provider interface
export CLUSTER_NAME=capi-mgmt-cluster
mkdir -p "$WORK"
cd "$WORK"
```

---

## 4. Version discovery (set pins)

Bobcat ran "dynamic latest." This runbook pins explicit versions captured at
execution time, with the discovery procedure documented inline so each
rebuild's pins are reproducible AND traceable.

**GitHub API: authenticated vs unauthenticated.** Unauth has 60 req/hr;
authenticated has 5000. For multiple rebuilds in a day, set a token:

```bash
# Optional but recommended — avoids rate-limit headaches during rebuild
export GITHUB_TOKEN=<your-PAT-with-public_repo-read>
# Or skip if you can tolerate ~10 API calls slowly
```

**Discover current stable releases:**

```bash
cd "$WORK"

# Helper: fetch latest stable release tag from a GitHub repo
gh_latest() {
  local repo=$1
  local auth=""
  [ -n "$GITHUB_TOKEN" ] && auth="-H Authorization: Bearer $GITHUB_TOKEN"
  curl -sfL $auth "https://api.github.com/repos/$repo/releases/latest" \
    | jq -r '.tag_name'
}

# Pin captures (one file per pin, for the deploy-record convention)
mkdir -p pins
gh_latest "kubernetes-sigs/cluster-api"                | tee pins/CAPI_VERSION
gh_latest "kubernetes-sigs/cluster-api-provider-openstack" | tee pins/CAPO_VERSION
gh_latest "canonical/cluster-api-k8s"                  | tee pins/CK8S_VERSION
gh_latest "cert-manager/cert-manager"                  | tee pins/CERT_MANAGER_VERSION
gh_latest "k-orc/openstack-resource-controller"        | tee pins/ORC_VERSION
gh_latest "k3s-io/k3s"                                 | tee pins/K3S_VERSION
gh_latest "helm/helm"                                  | tee pins/HELM_VERSION

# Load into shell
export CAPI_VERSION=$(cat pins/CAPI_VERSION)
export CAPO_VERSION=$(cat pins/CAPO_VERSION)
export CK8S_VERSION=$(cat pins/CK8S_VERSION)
export CERT_MANAGER_VERSION=$(cat pins/CERT_MANAGER_VERSION)
export ORC_VERSION=$(cat pins/ORC_VERSION)
export K3S_VERSION=$(cat pins/K3S_VERSION)
export HELM_VERSION=$(cat pins/HELM_VERSION)

# Display for the deploy log
cat pins/*_VERSION | paste -d= <(ls pins/) -
```

**Sanity check:** all values should look like `v1.X.Y` or `v0.X.Y`. If any
returned `null` or empty, the GitHub API call failed — most likely
rate-limited. Wait an hour or set `$GITHUB_TOKEN` and retry.

**Capture pins to repo as deploy record:**

The pin files in `$WORK/pins/` should be appended to a deploy-log artifact
(NOT committed to the repo — these are deploy-time captures). Suggested
location: `$HOME/deploy-records/$(date +%Y%m%d-%H%M)/capi-pins/`.

```bash
DEPLOY_RECORD=$HOME/deploy-records/$(date +%Y%m%d-%H%M%S)/capi-pins
mkdir -p "$DEPLOY_RECORD"
cp pins/*_VERSION "$DEPLOY_RECORD/"
ls -la "$DEPLOY_RECORD/"
```

---

## 5. MAAS-deploy capi-mgmt

Prerequisite: capi-mgmt MAAS machine is in `Ready` state (see §3).
Network config in MAAS:

- **eth0** on metal fabric, DHCP → `10.12.8.21` (MAAS-pinned static lease)
- **eth1** on provider fabric, static → `10.12.4.21`

Deploy Ubuntu 24.04 (Noble):

```bash
# Get the capi-mgmt system_id from MAAS
CAPI_MGMT_SYSTEM_ID=$(maas $MAAS_PROFILE machines read \
  | jq -r '.[] | select(.hostname=="capi-mgmt") | .system_id')
echo "capi-mgmt system_id: $CAPI_MGMT_SYSTEM_ID"

# Deploy
maas $MAAS_PROFILE machine deploy "$CAPI_MGMT_SYSTEM_ID" \
  distro_series=noble \
  hwe_kernel=ga-24.04
```

Poll for `Deployed`:

```bash
while true; do
  STATUS=$(maas $MAAS_PROFILE machine read "$CAPI_MGMT_SYSTEM_ID" \
    | jq -r '.status_name')
  echo "$(date -Is) capi-mgmt status: $STATUS"
  [ "$STATUS" = "Deployed" ] && break
  [ "$STATUS" = "Failed deployment" ] && { echo "FAILED"; exit 1; }
  sleep 30
done
```

Typical deploy time: 5-8 minutes on this hardware.

**SSH reachability:**

```bash
# MAAS .maas zone may not resolve from jumphost — use IP directly per handoff lessons
ssh -o StrictHostKeyChecking=accept-new ubuntu@$CAPI_MGMT_METAL_IP -- hostname
# Expect: capi-mgmt
```

> **Gotcha:** MAAS-deployed Ubuntu uses the `ubuntu` user, not `jessea123`.
> See handoff "recurring technical pitfalls."

---

## 6. SSH bootstrap + Vault CA install

On the jumphost, prepare a transport bundle of essentials:

```bash
mkdir -p "$WORK/bootstrap-bundle"
cp "$VAULT_CA" "$WORK/bootstrap-bundle/vault-ca.crt"
chmod 644 "$WORK/bootstrap-bundle/vault-ca.crt"

# Bundle pin files so capi-mgmt can read versions
cp -r "$WORK/pins" "$WORK/bootstrap-bundle/"
```

SCP and install Vault CA on capi-mgmt:

```bash
scp -r "$WORK/bootstrap-bundle" ubuntu@$CAPI_MGMT_METAL_IP:/home/ubuntu/

ssh ubuntu@$CAPI_MGMT_METAL_IP <<'EOF'
set -euo pipefail

# Install Vault CA as a system-trusted root
sudo cp /home/ubuntu/bootstrap-bundle/vault-ca.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates 2>&1 | tail -3

# Verify
openssl s_client -connect keystone.omega.dc0.vr0.cloud.neumatrix.local:5000 \
  -CApath /etc/ssl/certs -verify_return_error </dev/null 2>&1 \
  | grep -E "(Verify return code|subject=)" || \
  { echo "TLS chain verify failed against Keystone — investigate before proceeding"; exit 1; }

# Update apt + base utilities
sudo apt-get update -qq
sudo apt-get install -y -qq jq curl yq

# Confirm
which jq curl yq
EOF
```

**Expected:**

- `update-ca-certificates` reports "1 added"
- `openssl s_client` shows `Verify return code: 0 (ok)` and a Keystone cert
  whose chain terminates at the Vault CA

> **Why this matters:** Bobcat used `tls-insecure=true` in cloud.conf which
> skipped this entire trust path. Our workstream 3b decision (ship Vault CA)
> means OCCM and CAPO will validate certs against this trust store. If TLS
> verify fails here, OCCM will crashloop later.

---

## 7. k3s install

On capi-mgmt:

```bash
ssh ubuntu@$CAPI_MGMT_METAL_IP "K3S_VERSION=$K3S_VERSION CAPI_MGMT_METAL_IP=$CAPI_MGMT_METAL_IP bash -s" <<'REMOTE_EOF'
set -euo pipefail

# Install k3s with explicit bind/advertise/SAN flags
curl -sfL https://get.k3s.io | \
  INSTALL_K3S_VERSION="$K3S_VERSION" \
  sh -s - server \
    --bind-address="$CAPI_MGMT_METAL_IP" \
    --advertise-address="$CAPI_MGMT_METAL_IP" \
    --node-ip="$CAPI_MGMT_METAL_IP" \
    --tls-san="$CAPI_MGMT_METAL_IP" \
    --tls-san=capi-mgmt.maas \
    --write-kubeconfig-mode=0644 \
    --disable=traefik

# Wait for k3s API to respond
for i in $(seq 1 30); do
  if sudo kubectl get nodes 2>/dev/null | grep -q "Ready"; then
    echo "k3s ready"; break
  fi
  echo "Waiting for k3s API... ($i/30)"
  sleep 5
done

sudo kubectl get nodes
sudo kubectl get pods -A
REMOTE_EOF
```

> **Gotcha:** `--bind-address=$IP` makes k3s listen ONLY on that IP — not
> also on 127.0.0.1. The default kubeconfig at
> `/etc/rancher/k3s/k3s.yaml` has `server: https://127.0.0.1:6443` and will
> NOT work as-is. Sed-rewrite below.

---

## 8. Kubeconfig server-URL rewrite

```bash
ssh ubuntu@$CAPI_MGMT_METAL_IP "CAPI_MGMT_METAL_IP=$CAPI_MGMT_METAL_IP bash -s" <<'REMOTE_EOF'
set -euo pipefail

# Copy k3s kubeconfig to ubuntu user; rewrite server URL
mkdir -p /home/ubuntu/.kube
sudo cp /etc/rancher/k3s/k3s.yaml /home/ubuntu/.kube/config
sudo chown ubuntu:ubuntu /home/ubuntu/.kube/config
chmod 600 /home/ubuntu/.kube/config

# Rewrite 127.0.0.1 → metal IP
sed -i "s|server: https://127.0.0.1:6443|server: https://$CAPI_MGMT_METAL_IP:6443|" \
  /home/ubuntu/.kube/config

# Verify rewrite
grep "server:" /home/ubuntu/.kube/config
# Expect: server: https://10.12.8.21:6443

# Confirm kubectl works as ubuntu user (no sudo)
kubectl get nodes
REMOTE_EOF
```

---

## 9. helm + clusterctl install

```bash
ssh ubuntu@$CAPI_MGMT_METAL_IP "HELM_VERSION=$HELM_VERSION CAPI_VERSION=$CAPI_VERSION bash -s" <<'REMOTE_EOF'
set -euo pipefail

# helm install (get-helm-3 fetches the version we specify)
cd /tmp
curl -sfL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 \
  | DESIRED_VERSION="$HELM_VERSION" bash
helm version --short

# clusterctl install
CLUSTERCTL_URL="https://github.com/kubernetes-sigs/cluster-api/releases/download/${CAPI_VERSION}/clusterctl-linux-amd64"
sudo curl -sfL "$CLUSTERCTL_URL" -o /usr/local/bin/clusterctl
sudo chmod +x /usr/local/bin/clusterctl
clusterctl version
REMOTE_EOF
```

---

## 10. clusterctl init (CAPI controllers + cert-manager + ORC + CAPO + CK8s)

```bash
ssh ubuntu@$CAPI_MGMT_METAL_IP "CK8S_VERSION=$CK8S_VERSION CERT_MANAGER_VERSION=$CERT_MANAGER_VERSION ORC_VERSION=$ORC_VERSION CAPO_VERSION=$CAPO_VERSION bash -s" <<'REMOTE_EOF'
set -euo pipefail

# Configure clusterctl with provider URLs
mkdir -p ~/.cluster-api
cat > ~/.cluster-api/clusterctl.yaml <<EOF
providers:
  - name: "canonical-kubernetes"
    url: "https://github.com/canonical/cluster-api-k8s/releases/${CK8S_VERSION}/bootstrap-components.yaml"
    type: "BootstrapProvider"
  - name: "canonical-kubernetes"
    url: "https://github.com/canonical/cluster-api-k8s/releases/${CK8S_VERSION}/control-plane-components.yaml"
    type: "ControlPlaneProvider"
EOF

# Initialize CAPI with explicit versions
clusterctl init \
  --core "cluster-api:${CAPI_VERSION}" \
  --infrastructure "openstack:${CAPO_VERSION}" \
  --bootstrap "canonical-kubernetes:${CK8S_VERSION}" \
  --control-plane "canonical-kubernetes:${CK8S_VERSION}" \
  --cert-manager-version "${CERT_MANAGER_VERSION}"

# Wait for controllers to be Ready
kubectl wait --for=condition=Available --timeout=5m \
  deployment --all -n capi-system
kubectl wait --for=condition=Available --timeout=5m \
  deployment --all -n capi-kubeadm-bootstrap-system 2>/dev/null || true
kubectl wait --for=condition=Available --timeout=5m \
  deployment --all -n capo-system
kubectl wait --for=condition=Available --timeout=5m \
  deployment --all -n cert-manager

# Install ORC
kubectl apply -f "https://github.com/k-orc/openstack-resource-controller/releases/${ORC_VERSION}/orc.yaml"
kubectl wait --for=condition=Available --timeout=5m \
  deployment --all -n orc-system

# Confirm all controllers
kubectl get pods -A | grep -v "Running\|Completed" | grep -v NAME
# Expected: empty output (all pods Running or no abnormal state)
REMOTE_EOF
```

> **Gotcha:** the actual namespace names (`capi-system`, `capo-system`, etc.)
> are conventions. If a controller fails to land in the expected namespace,
> `kubectl get deployment -A` lists all deployments — diagnose from there.

---

## 11. Cloud-side prep (Keystone, Nova, Glance)

Back on the jumphost:

```bash
source $HOME/admin-openrc

# Inventory existing resources FIRST (Bobcat lesson: don't create duplicates)
echo "=== Existing images ==="
openstack image list -c ID -c Name -f json | jq -r '.[] | "\(.Name)\t\(.ID)"'
echo ""
echo "=== Existing flavors ==="
openstack flavor list -c Name -c ID -c RAM -c VCPUs -c Disk -f json \
  | jq -r '.[] | "\(.Name)\tRAM=\(.RAM)\tCPU=\(.VCPUs)\tDisk=\(.Disk)\tID=\(.ID)"'
echo ""
echo "=== Existing keypairs ==="
openstack keypair list
echo ""
echo "=== Existing projects in admin_domain ==="
openstack project list --domain admin_domain
```

**Create / verify resources:**

```bash
# Keystone project + user
openstack project show capi-mgmt --domain admin_domain 2>/dev/null \
  || openstack project create capi-mgmt --domain admin_domain --description "CAPI management plane"

openstack user show capo --domain admin_domain 2>/dev/null \
  || openstack user create capo --domain admin_domain --password-prompt --description "CAPO operator"

# Role assignments (CAPO needs member + load-balancer_member at minimum;
# admin works for testcloud — Roosevelt should use least-privilege)
openstack role add --user capo --user-domain admin_domain \
  --project capi-mgmt --project-domain admin_domain \
  member

openstack role add --user capo --user-domain admin_domain \
  --project capi-mgmt --project-domain admin_domain \
  load-balancer_member 2>/dev/null || \
  echo "(load-balancer_member role may not exist if Octavia not deployed yet)"

# Application credential — captured to file under $HOME (snap confinement)
APP_CRED_FILE=$WORK/capo-app-cred.json
openstack --os-username capo --os-user-domain-name admin_domain \
          --os-project-name capi-mgmt --os-project-domain-name admin_domain \
  application credential create capo-app-cred \
  --description "CAPO operator app credential" \
  -f json > "$APP_CRED_FILE"
chmod 600 "$APP_CRED_FILE"

# Extract credential ID + secret
export APP_CRED_ID=$(jq -r '.id' "$APP_CRED_FILE")
export APP_CRED_SECRET=$(jq -r '.secret' "$APP_CRED_FILE")
echo "App cred ID: $APP_CRED_ID"
```

**Nova keypair (workload node SSH key):**

```bash
# Generate fresh keypair locally (do NOT reuse jumphost personal key)
ssh-keygen -t ed25519 -N '' -f "$WORK/capi-workload-key" \
  -C "capi-workload-$(date +%Y%m%d)"
chmod 600 "$WORK/capi-workload-key"

# Upload public key to Keystone as a Nova keypair
openstack keypair create --public-key "$WORK/capi-workload-key.pub" capi-workload-key
openstack keypair show capi-workload-key
```

**Workload image:**

```bash
# Inventory check — use noble-amd64 if it exists (Bobcat lesson: do NOT create ubuntu-24.04-capi as a dup)
NOBLE_IMAGE_ID=$(openstack image show noble-amd64 -c id -f value 2>/dev/null || echo "")

if [ -z "$NOBLE_IMAGE_ID" ]; then
  echo "noble-amd64 image not found — upload required."
  echo "(Pull from https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img"
  echo " then: openstack image create --disk-format qcow2 --container-format bare \\"
  echo "        --public --file noble-server-cloudimg-amd64.img noble-amd64)"
  exit 1
fi
echo "Using image: noble-amd64 ($NOBLE_IMAGE_ID)"
export WORKLOAD_IMAGE_ID=$NOBLE_IMAGE_ID
```

**Workload flavor:**

```bash
openstack flavor show capi-mgmt-node 2>/dev/null \
  || openstack flavor create capi-mgmt-node \
       --vcpus 4 --ram 4096 --disk 30 \
       --description "CAPI workload node (control plane sizing)"

export WORKLOAD_FLAVOR=capi-mgmt-node
```

---

## 12. clouds.yaml + cloud.conf composition (with Vault CA, no tls-insecure)

The workload cluster's OCCM (OpenStack Cloud Controller Manager) and CAPO both
need to call OpenStack APIs. Two files:

- `clouds.yaml` — CAPO's view of how to reach OpenStack (used at cluster
  creation time on capi-mgmt)
- `cloud.conf` — OCCM's view, injected into the workload cluster's k8s
  Secret (used continuously by OCCM running in the workload cluster)

**Compose clouds.yaml:**

```bash
cat > "$WORK/clouds.yaml" <<EOF
clouds:
  capi-mgmt:
    region_name: RegionOne
    interface: public
    identity_api_version: 3
    auth_type: v3applicationcredential
    auth:
      auth_url: https://keystone.omega.dc0.vr0.cloud.neumatrix.local:5000/v3
      application_credential_id: $APP_CRED_ID
      application_credential_secret: $APP_CRED_SECRET
    cacert: /usr/local/share/ca-certificates/vault-ca.crt
    verify: true
EOF
chmod 600 "$WORK/clouds.yaml"

# base64-encode for cluster template embedding (no newline wrapping)
base64 -w0 "$WORK/clouds.yaml" > "$WORK/clouds.yaml.b64"
```

**Compose cloud.conf** (INI format, NOT YAML):

```bash
cat > "$WORK/cloud.conf" <<EOF
[Global]
auth-url=https://keystone.omega.dc0.vr0.cloud.neumatrix.local:5000/v3
application-credential-id=$APP_CRED_ID
application-credential-secret=$APP_CRED_SECRET
region=RegionOne
domain-name=admin_domain
ca-file=/usr/local/share/ca-certificates/vault-ca.crt

[LoadBalancer]
use-octavia=true
EOF
chmod 600 "$WORK/cloud.conf"

base64 -w0 "$WORK/cloud.conf" > "$WORK/cloud.conf.b64"
```

> **Critical delta from Bobcat:** the `ca-file` line replaces `tls-insecure=true`.
> The path `/usr/local/share/ca-certificates/vault-ca.crt` exists on capi-mgmt
> (from §6) AND will be injected into workload nodes via CK8sConfig in §13.

**base64-encode Vault CA for CK8sConfig injection:**

```bash
base64 -w0 "$VAULT_CA" > "$WORK/vault-ca.crt.b64"
wc -c "$WORK/vault-ca.crt.b64"
```

---

## 13. Cluster template rendering (with Vault CA injection)

The cluster template defines:

- Cluster object
- OpenStackCluster (CAPO infrastructure)
- CK8sControlPlane
- CK8sConfigTemplate (control plane bootstrap — includes Vault CA injection)
- MachineDeployment + CK8sConfigTemplate (workers — includes Vault CA injection)
- Secrets for clouds.yaml and cloud.conf

Variables (18 total):

```bash
export CLUSTER_NAME=capi-mgmt-cluster
export CLUSTER_NAMESPACE=default
export KUBERNETES_VERSION=v1.31.4              # adjust to CK8s-supported
export CONTROL_PLANE_MACHINE_COUNT=1           # 3 for HA on Roosevelt
export WORKER_MACHINE_COUNT=2                  # 3 on Roosevelt
export OPENSTACK_DNS_NAMESERVERS=10.12.4.227   # designate VIP
export OPENSTACK_FAILURE_DOMAIN=nova
export OPENSTACK_EXTERNAL_NETWORK_ID=$(openstack network show ext_net -c id -f value)
export OPENSTACK_IMAGE_NAME=noble-amd64
export OPENSTACK_FLAVOR=capi-mgmt-node
export OPENSTACK_SSH_KEY_NAME=capi-workload-key
export POD_CIDR=10.244.0.0/16
export SERVICE_CIDR=10.96.0.0/12
export CLOUDS_YAML_B64=$(cat "$WORK/clouds.yaml.b64")
export CLOUD_CONF_B64=$(cat "$WORK/cloud.conf.b64")
export VAULT_CA_B64=$(cat "$WORK/vault-ca.crt.b64")
export CLUSTER_DOMAIN=cluster.local
export OPENSTACK_CLOUD=capi-mgmt

# Sanity print
env | grep -E "^(CLUSTER|KUBERNETES|CONTROL_PLANE|WORKER|OPENSTACK|POD|SERVICE|VAULT|CLOUD)" \
  | grep -v "B64\|SECRET\|PASS" | sort
```

**Render the cluster template:**

```bash
cat > "$WORK/cluster-template.yaml" <<'TEMPLATE_EOF'
apiVersion: v1
kind: Secret
metadata:
  name: ${CLUSTER_NAME}-cloud-config
  namespace: ${CLUSTER_NAMESPACE}
type: Opaque
data:
  clouds.yaml: ${CLOUDS_YAML_B64}
  cloud.conf: ${CLOUD_CONF_B64}
  cacert: ${VAULT_CA_B64}
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: ${CLUSTER_NAME}
  namespace: ${CLUSTER_NAMESPACE}
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
        - ${POD_CIDR}
    services:
      cidrBlocks:
        - ${SERVICE_CIDR}
    serviceDomain: ${CLUSTER_DOMAIN}
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: OpenStackCluster
    name: ${CLUSTER_NAME}
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta2
    kind: CK8sControlPlane
    name: ${CLUSTER_NAME}-control-plane
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: OpenStackCluster
metadata:
  name: ${CLUSTER_NAME}
  namespace: ${CLUSTER_NAMESPACE}
spec:
  identityRef:
    name: ${CLUSTER_NAME}-cloud-config
    cloudName: ${OPENSTACK_CLOUD}
  externalNetwork:
    id: ${OPENSTACK_EXTERNAL_NETWORK_ID}
  managedSecurityGroups:
    allowAllInClusterTraffic: true
  apiServerLoadBalancer:
    enabled: true
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: CK8sControlPlane
metadata:
  name: ${CLUSTER_NAME}-control-plane
  namespace: ${CLUSTER_NAMESPACE}
spec:
  replicas: ${CONTROL_PLANE_MACHINE_COUNT}
  version: ${KUBERNETES_VERSION}
  machineTemplate:
    infrastructureTemplate:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: OpenStackMachineTemplate
      name: ${CLUSTER_NAME}-control-plane
  spec:
    files:
      - path: /usr/local/share/ca-certificates/vault-ca.crt
        owner: root:root
        permissions: "0644"
        contentFrom:
          secret:
            name: ${CLUSTER_NAME}-cloud-config
            key: cacert
    preRunCommands:
      - update-ca-certificates
    extraKubeAPIServerArgs:
      "--cloud-provider": external
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: OpenStackMachineTemplate
metadata:
  name: ${CLUSTER_NAME}-control-plane
  namespace: ${CLUSTER_NAMESPACE}
spec:
  template:
    spec:
      flavor: ${OPENSTACK_FLAVOR}
      image:
        filter:
          name: ${OPENSTACK_IMAGE_NAME}
      sshKeyName: ${OPENSTACK_SSH_KEY_NAME}
      identityRef:
        name: ${CLUSTER_NAME}-cloud-config
        cloudName: ${OPENSTACK_CLOUD}
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: ${CLUSTER_NAME}-md-0
  namespace: ${CLUSTER_NAMESPACE}
spec:
  clusterName: ${CLUSTER_NAME}
  replicas: ${WORKER_MACHINE_COUNT}
  selector:
    matchLabels: {}
  template:
    spec:
      clusterName: ${CLUSTER_NAME}
      version: ${KUBERNETES_VERSION}
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
          kind: CK8sConfigTemplate
          name: ${CLUSTER_NAME}-md-0
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: OpenStackMachineTemplate
        name: ${CLUSTER_NAME}-md-0
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: OpenStackMachineTemplate
metadata:
  name: ${CLUSTER_NAME}-md-0
  namespace: ${CLUSTER_NAMESPACE}
spec:
  template:
    spec:
      flavor: ${OPENSTACK_FLAVOR}
      image:
        filter:
          name: ${OPENSTACK_IMAGE_NAME}
      sshKeyName: ${OPENSTACK_SSH_KEY_NAME}
      identityRef:
        name: ${CLUSTER_NAME}-cloud-config
        cloudName: ${OPENSTACK_CLOUD}
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
kind: CK8sConfigTemplate
metadata:
  name: ${CLUSTER_NAME}-md-0
  namespace: ${CLUSTER_NAMESPACE}
spec:
  template:
    spec:
      files:
        - path: /usr/local/share/ca-certificates/vault-ca.crt
          owner: root:root
          permissions: "0644"
          contentFrom:
            secret:
              name: ${CLUSTER_NAME}-cloud-config
              key: cacert
      preRunCommands:
        - update-ca-certificates
TEMPLATE_EOF

# envsubst to render
envsubst < "$WORK/cluster-template.yaml" > "$WORK/cluster-rendered.yaml"

# Validate as YAML
python3 -c "import yaml; list(yaml.safe_load_all(open('$WORK/cluster-rendered.yaml'))); print('YAML OK')"

# Quick visual check — no leftover ${...} markers
grep -n '\${' "$WORK/cluster-rendered.yaml" || echo "No unsubstituted variables — good"
```

> **CK8sConfig field name caveat:** the exact field names (`files`,
> `preRunCommands`) and their `contentFrom.secret` schema are CK8s-version-
> dependent. If `clusterctl init` failed earlier with schema warnings,
> consult the CK8s release notes for the pinned `$CK8S_VERSION`.

---

## 14. Apply + poll-to-Ready

Transfer rendered template to capi-mgmt and apply:

```bash
scp "$WORK/cluster-rendered.yaml" ubuntu@$CAPI_MGMT_METAL_IP:/home/ubuntu/cluster.yaml

ssh ubuntu@$CAPI_MGMT_METAL_IP <<'EOF'
set -euo pipefail
kubectl apply -f /home/ubuntu/cluster.yaml
echo "Applied. Waiting for cluster Available status (15-min timeout)..."

for i in $(seq 1 90); do
  STATUS=$(kubectl get cluster capi-mgmt-cluster -o json 2>/dev/null \
    | jq -r '.status.phase // "Unknown"')
  READY=$(kubectl get cluster capi-mgmt-cluster -o json 2>/dev/null \
    | jq -r '.status.conditions[]? | select(.type=="Ready") | .status' \
    | head -1)
  echo "$(date -Is) phase=$STATUS ready=$READY"
  [ "$READY" = "True" ] && { echo "Cluster Ready"; break; }
  sleep 10
done

kubectl get cluster,machines,kubeadmcontrolplane,machinedeployment -A
EOF
```

**If the poll times out before Ready,** typical diagnosis:

```bash
ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl describe cluster capi-mgmt-cluster
ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl get machines -A
ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl logs -n capo-system deployment/capo-controller-manager --tail=100
```

Common causes:

- OpenStack API unreachable from capi-mgmt → check Vault CA install on capi-mgmt (§6)
- Image / flavor / network ID wrong in cluster template → re-check §11 variables
- Security group rules block kube-api LB → CAPO usually handles this; check OpenStackCluster status
- Application credential expired / wrong → re-check `$APP_CRED_ID`

---

## 15. Extract workload kubeconfig

```bash
ssh ubuntu@$CAPI_MGMT_METAL_IP -- clusterctl get kubeconfig capi-mgmt-cluster \
  > "$WORK/capi-mgmt-cluster.kubeconfig"
chmod 600 "$WORK/capi-mgmt-cluster.kubeconfig"

# Sanity-check the workload cluster is reachable
kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get nodes
# Expect: 1 control plane + 2 workers, all Ready
```

If `get nodes` times out, the cluster's API LB may not have allocated its
external IP yet, or the firewall rules don't permit jumphost → workload API:

```bash
# What IP is the cluster's API LB on?
ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl get openstackcluster capi-mgmt-cluster \
  -o json | jq '.status.externalNetwork, .status.controlPlaneEndpoint'

# Test reachability
curl -sk --max-time 10 "https://<API-IP>:6443/version" && echo " ← reachable" || echo "API LB unreachable"
```

---

## 16. `clusterctl init` on target (workload cluster)

The workload cluster must have the same CAPI providers installed before `move`.

```bash
# Run from jumphost using the workload kubeconfig
KUBECONFIG="$WORK/capi-mgmt-cluster.kubeconfig" clusterctl init \
  --core "cluster-api:${CAPI_VERSION}" \
  --infrastructure "openstack:${CAPO_VERSION}" \
  --bootstrap "canonical-kubernetes:${CK8S_VERSION}" \
  --control-plane "canonical-kubernetes:${CK8S_VERSION}" \
  --cert-manager-version "${CERT_MANAGER_VERSION}"

# ORC into workload cluster too
kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" apply \
  -f "https://github.com/k-orc/openstack-resource-controller/releases/${ORC_VERSION}/orc.yaml"

# Wait for everything Available
kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" wait \
  --for=condition=Available --timeout=5m \
  deployment --all -n capi-system
kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" wait \
  --for=condition=Available --timeout=5m \
  deployment --all -n capo-system
kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" wait \
  --for=condition=Available --timeout=5m \
  deployment --all -n cert-manager
kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" wait \
  --for=condition=Available --timeout=5m \
  deployment --all -n orc-system
```

> **cert-manager double-install caveat:** if CK8s already installed
> cert-manager during workload bootstrap, the second `clusterctl init` may
> warn or skip. Check existing cert-manager version against `$CERT_MANAGER_VERSION`
> — if they differ, version-skew issues may surface post-pivot. Adjust the
> pin in §4 or accept the existing version. Roosevelt's standard practice
> is to install cert-manager via `clusterctl init` only (don't pre-install
> via CK8s) — same approach valid here if you want clean version control.

---

## 17. `clusterctl move` pivot

Move all CAPI CRs from bootstrap k3s → workload cluster:

```bash
# Stage the target kubeconfig on capi-mgmt (where clusterctl move runs)
scp "$WORK/capi-mgmt-cluster.kubeconfig" ubuntu@$CAPI_MGMT_METAL_IP:/home/ubuntu/target.kubeconfig

# Dry-run first to catch issues before commit
ssh ubuntu@$CAPI_MGMT_METAL_IP -- clusterctl move \
  --to-kubeconfig=/home/ubuntu/target.kubeconfig \
  --dry-run

# Inspect dry-run output: list of objects to be moved. Should include:
#   - Cluster, OpenStackCluster, OpenStackClusterTemplate
#   - Secrets (cloud-config)
#   - Machine objects, OpenStackMachineTemplate
#   - CK8sControlPlane, CK8sConfigTemplate
#   - MachineDeployment
# Should NOT include cert-manager state (cert-manager manages its own state
# on each cluster independently)
```

**If dry-run looks correct, execute the move:**

```bash
ssh ubuntu@$CAPI_MGMT_METAL_IP -- clusterctl move \
  --to-kubeconfig=/home/ubuntu/target.kubeconfig

# Move can take several minutes. Output ends with: "moved successfully"
```

---

## 18. Post-pivot verification

```bash
echo "=== Bootstrap k3s (should now be empty of cluster CRs) ==="
ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl get cluster -A
# Expect: No resources found (or only a header)

ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl get machines -A
# Expect: No resources found

ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl get openstackcluster -A
# Expect: No resources found

echo ""
echo "=== Workload cluster (should now own its own cluster CRs) ==="
kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get cluster -A
# Expect: capi-mgmt-cluster shown, phase=Provisioned

kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get machines -A
# Expect: 3 machines (1 control-plane + 2 workers), all Running

kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get openstackcluster -A

echo ""
echo "=== CAPI controllers in workload ==="
kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get pods -A \
  | grep -E "(capi|capo|orc|cert-manager)" | grep -v "Running\|Completed"
# Expect: empty (all controller pods Running)

echo ""
echo "=== OCCM not crash-looping (CRITICAL — main goal of TLS-verify work) ==="
kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get pods -n kube-system \
  -l k8s-app=openstack-cloud-controller-manager
# Expect: 1 pod Running, NOT CrashLoopBackOff

kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" logs -n kube-system \
  -l k8s-app=openstack-cloud-controller-manager --tail=50 \
  | grep -iE "(tls|cert|error)" | head -20
# Expect: no TLS/cert errors; OCCM should be healthy
```

> **If OCCM crash-loops with "x509: certificate signed by unknown authority":**
> Vault CA distribution failed. Check (a) `/usr/local/share/ca-certificates/vault-ca.crt`
> exists on workload nodes; (b) `update-ca-certificates` ran (check `/etc/ssl/certs/ca-certificates.crt`
> for the Vault CA's subject); (c) the secret reference in CK8sConfigTemplate
> matched the secret name. SSH into a worker via the jumphost key (`ssh -i
> $WORK/capi-workload-key ubuntu@<worker-IP-via-FIP>`) to diagnose.

---

## 19. Handoff to runbook 05

The workload kubeconfig at `$WORK/capi-mgmt-cluster.kubeconfig` is the input to
`runbooks/05-magnum-capi-driver.md`. Copy it to a stable path:

```bash
mkdir -p $HOME/magnum-capi
cp "$WORK/capi-mgmt-cluster.kubeconfig" $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig
chmod 600 $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig
echo "Workload kubeconfig staged at: $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig"
```

> **Important — post-pivot semantic shift from Bobcat:** Magnum's
> `kubeconfig_file` setting (under `[capi_helm]` in
> `/etc/magnum/magnum.conf.d/99-capi.conf`, per D-007) now points to the
> workload cluster, not the bootstrap k3s. Bobcat had Magnum pointing at
> bootstrap k3s because the pivot was never executed. With pivot mandatory,
> Magnum's CAPI calls flow:
>
> ```
> Magnum/leader → workload cluster API → CAPI controllers (running in workload)
>                                       → create new Cluster CRs (tenant Magnum clusters)
> ```
>
> The bootstrap k3s on capi-mgmt is now disposable. If you wanted, you could
> destroy capi-mgmt entirely at this point — the workload cluster manages
> itself. (Roosevelt may actually do this for cost savings.) For v1 testcloud,
> leave capi-mgmt running so its k3s can be inspected for diagnostics.

---

## 20. Roosevelt deltas (forward-look)

| Aspect | Testcloud (v1) | Roosevelt |
|---|---|---|
| Workload image | Default `noble-amd64` from cloud-images.ubuntu.com | Custom image baked with Vault CA pre-installed (no runtime install step) |
| Vault CA distribution | CK8sConfig `files:` + `preRunCommands:` (this runbook) | Image-baked + CK8sConfig (defense in depth) |
| App credential lifetime | No expiry set (testcloud) | Short-lived rotating credentials via Vault auth method |
| Workload cluster control plane | 1 node | 3 nodes (HA) |
| Workload cluster workers | 2 nodes | Per-tenant sizing; HPA-driven |
| `clusterctl init --cert-manager-version` | Pin from §4 | Pin to Vault PKI cert-manager profile (separate Roosevelt prep) |
| capi-mgmt VM lifecycle post-pivot | Kept running for diagnostics | Destroyed (cost savings; pivot makes it disposable) |
| Version pinning record | `$HOME/deploy-records/<timestamp>/capi-pins/` | Same pattern, captured in Vault as audit artifact |
| Authentication to GitHub API | Optional PAT | Mandatory PAT (avoid rate-limit during automated rebuilds) |

---

## 21. Rotation/refresh of pins

The pins captured in §4 will age. Recommended cadence:

- **Per rebuild:** re-discover all pins (Step 1 of next execution will catch
  natural drift).
- **Out-of-band patch:** if a CVE drops for any pinned component, run §4
  discovery alone and capture the new pin into `$DEPLOY_RECORD/`. Then for
  the affected component only, follow the upgrade procedure from its
  upstream docs (does NOT necessarily require this whole runbook re-run).

For Roosevelt, this becomes a tracked maintenance window task.

---

## 22. Change log

| Date | Change | Reference |
|---|---|---|
| 2026-05-22 | Document created. Vault CA distribution (no tls-insecure), mandatory `clusterctl move` pivot, pin-at-execution version model. | Workstream 3b |