diff --git a/docs/v1-pre-deploy-fixes.md b/docs/v1-pre-deploy-fixes.md new file mode 100644 index 0000000..b870716 --- /dev/null +++ b/docs/v1-pre-deploy-fixes.md @@ -0,0 +1,832 @@ +# v1 Pre-Deploy Fixes (v2 — includes Designate deferral) + +**Purpose:** Single-pass repo hygiene before v1 deployment execution begins. Apply these fixes as one logical commit per group (nine commits total) before any execution document runs. + +**Status:** Authoritative for the v1 deploy track. Supersedes the v1 draft of this document. + +**Scope:** Repo-only changes. No cloud state is touched by this document. All changes are reviewed locally, committed to `main`, and pushed before the next do-document runs. + +**What changed in v2 of this change list (2026-05-27):** + +- Added commits 7-9 implementing the Designate-deferral decision (D-019). +- Amended commit 5 (deprecated runbook moves) — `07-dns-zones.md` is now permanently deprecated per D-019, not "replaced by v1-do-doc-10-dns." +- Amended commit 6 (deprecated README content) — same. +- Updated commit 4 (README.md refresh) — adds language reflecting Designate deferral to the v1 scope section. +- Updated §10 verification commands to expect 11 VIPs, not 12. + +## Cross-references + +- D-002 (channel matrix) — Vault row cleanup +- D-005 (Ceph Squid release) +- D-008 (DNS architecture) — superseded by D-019; v2-scope +- D-011 (validation bar) — amended by D-019 (Designate criterion dropped) +- D-014 (repo path) — stale path correction +- D-017 (CAPI bootstrap cluster lifecycle) — supersedes runbook 00 Phase 5 +- D-018 (teardown strategy) — supersedes runbook 00 Phase 4 +- **D-019 (NEW) — Designate deferral to v2; tenant resolvers use public DNS** +- Charmed Ceph `charm-ceph-osd` `config.yaml` and `metadata.yaml` (osd-devices semantics) +- Ceph BlueStore configuration reference (single-device co-located OSD pattern) + +--- + + +## 1. Bundle: remove ceph-osd `storage:` block + +### What + +In `bundle.yaml`, under the `ceph-osd` application, delete the entire `storage:` block. The `options.osd-devices: /dev/vdb` line stays. + +### Why + +The `osd-devices` declared under `storage:` is **additive** to the `options.osd-devices` value per the `charm-ceph-osd` `config.yaml`: "These devices are the range of devices that will be checked for and used across all service units, in addition to any volumes attached via the `--storage` flag during deployment." + +Concretely, with the current bundle: + +- `options.osd-devices: /dev/vdb` → one OSD per unit using the 512 GB libvirt-attached disk +- `storage.osd-devices: loop,1024M` → an *additional* 1 GB loopback OSD per unit + +Total: 2 OSDs per unit × 4 units = 8 OSDs, against `expected-osd-count: 4` on `ceph-mon`. The 1 GB OSDs are below practical minimums, asymmetric with the 512 GB primaries (CRUSH-weighting anti-pattern), and provide no operational value. + +The remaining `bluestore-db`, `bluestore-wal`, `cache-devices`, and `osd-journals` loopback entries are also being removed — not because they break anything, but because: + +1. BlueStore co-locates DB and WAL on the data device when no separate volume is supplied (Ceph Reef BlueStore reference). +2. Loopback files on the same backing storage as `/dev/vdb` are not "faster than the primary device," so the standard rationale for separate DB/WAL devices doesn't apply. +3. `osd-journals` is unused under BlueStore (default since Luminous). + +### Diff + +```yaml +# BEFORE (in bundle.yaml, applications.ceph-osd) + ceph-osd: + charm: ceph-osd + channel: squid/stable + num_units: 4 + to: ["8", "9", "10", "11"] + options: + source: *ceph-source + osd-devices: /dev/vdb # libvirt-attached, MAAS-untracked, wiped 2026-05-22 + bindings: *internal-bindings + constraints: arch=amd64 tags=openstack + storage: # Loop-backed auxiliaries (testcloud has no real SSDs) + bluestore-db: loop,1024M + bluestore-wal: loop,1024M + cache-devices: loop,10240M + osd-devices: loop,1024M + osd-journals: loop,1024M # Legacy storage name still in squid metadata; benign + +# AFTER + ceph-osd: + charm: ceph-osd + channel: squid/stable + num_units: 4 + to: ["8", "9", "10", "11"] + options: + source: *ceph-source + osd-devices: /dev/vdb # libvirt-attached, MAAS-untracked, wiped 2026-05-22 + bindings: *internal-bindings + constraints: arch=amd64 tags=openstack +``` + +### Commit message + +``` +bundle: remove ceph-osd storage block to match expected-osd-count + +The storage: block declared a second osd-devices entry (loop,1024M) which +is additive to options.osd-devices per the charm config. That produced +8 OSDs against expected-osd-count: 4 on ceph-mon, with 1 GB loopback +OSDs as the asymmetric secondaries — a CRUSH-weighting anti-pattern. + +Real production storage (DB/WAL on actual SSDs) will be declared on +Roosevelt. For the testcloud, BlueStore co-locates DB/WAL on the data +device which is the documented default for single-device setups. + +osd-journals is unused under BlueStore. +``` + +--- + + +## 2. design-decisions.md: D-002 — remove Vault from OpenStack-core row + +### What + +In `docs/design-decisions.md` under D-002 (channel matrix), the OpenStack-core row currently lists `, vault` as one of the components on `2024.1/stable`. Remove that token. + +### Why + +Vault has its own track per Canonical's charm delivery table — it runs on `1.8/stable`, not `2024.1/stable`. The D-002 table elsewhere (and the actual bundle.yaml) already reflects this; the OpenStack-core row description was a leftover from earlier drafting. + +### Diff + +```diff +-| OpenStack core API charms (keystone, glance, nova-cloud-controller, neutron-api, cinder, placement, octavia, barbican, magnum, designate, openstack-dashboard, vault) | `2024.1/stable` | ++| OpenStack core API charms (keystone, glance, nova-cloud-controller, neutron-api, cinder, placement, octavia, barbican, magnum, designate, openstack-dashboard) | `2024.1/stable` | +``` + +### Commit message + +``` +docs/design-decisions: D-002 — drop vault from OpenStack-core channel row + +Vault uses 1.8/stable per the Canonical charm delivery table, not the +2024.1/stable OpenStack-core track. The bundle.yaml already reflects +this; the design-decisions D-002 description had a stale token. +``` + +> **Note:** the `designate` token in the row above is correct as-of this commit (Designate is still on `2024.1/stable` channel). The Designate row is removed entirely by commit 8 (D-019). + +--- + + +## 3. design-decisions.md: D-014 — update repo path + +### What + +In `docs/design-decisions.md` under D-014 (repo location), the path currently shows the per-user namespace from before the 2026-05-27 move. Update to the OpenStack-group path. + +### Why + +Per the user-memory pinned note: "Caracal rebuild repo (moved to OpenStack group 2026-05-27): https://git.baldurkeep.com/OpenStack/openstack-caracal-ipv4 (web), https://git.baldurkeep.com/git/OpenStack/openstack-caracal-ipv4.git (clone). Old jesse.austin/openstack-caracal-ipv4 path no longer exists; GitBucket does not redirect." + +### Diff + +```diff + ## D-014: Repository location + +-**Decision:** `git.baldurkeep.com/jesse.austin/openstack-caracal-ipv4` for v1. ++**Decision:** `git.baldurkeep.com/OpenStack/openstack-caracal-ipv4` for v1. ++ ++- Web: `https://git.baldurkeep.com/OpenStack/openstack-caracal-ipv4` ++- Clone: `https://git.baldurkeep.com/git/OpenStack/openstack-caracal-ipv4.git` ++- Moved from `jesse.austin/openstack-caracal-ipv4` to the `OpenStack` group on 2026-05-27. GitBucket does not redirect from the old path. + + **Rationale:** Establishes a single repo per cloud lifecycle. v2 path TBD. +``` + +### Commit message + +``` +docs/design-decisions: D-014 — update repo path after move to OpenStack group + +Repository moved from jesse.austin/openstack-caracal-ipv4 to the +OpenStack group on 2026-05-27. GitBucket does not redirect, so the +prior path is dead. +``` + +--- + + +## 4. README.md: refresh stale references + +### What + +Three hunks in `README.md`: + +1. Update the design-decisions reference range from "D-001 through D-016" to "D-001 through D-019" (post-D-017/D-018/D-019 additions). +2. Replace the inline runbook 00 description (which still mentions backups + capi-mgmt graceful teardown — both invalidated by D-017/D-018). +3. Replace the 12-step deploy order with a pointer to the do-document set. +4. Add a v1 scope reduction note for Designate (per D-019). + +### Why + +The README's v1 deployment order block reflects the original 12-step runbook plan; the actual deploy path now flows through the v1-do-doc-NN execution documents. Keeping the README in sync prevents new operators from following the stale path. + +The D-019 note in the v1 scope section makes the Designate deferral explicit for anyone reading the README to understand v1 scope. + +### Diff (four hunks) + +**Hunk A — D-range:** + +```diff +-└── docs/ +- └── design-decisions.md # architectural record (D-001 through D-016) ++└── docs/ ++ ├── design-decisions.md # architectural record (D-001 through D-019) ++ └── netbox-vip-queue.md # post-deploy NetBox imports (workstream 2) +``` + +**Hunk B — runbook 00 description (in the layout block):** + +```diff +-│ ├── 00-pre-deploy.md # backups, capi-mgmt graceful teardown ++│ # (deprecated; see runbooks/deprecated/ — superseded by D-017 + D-018 + v1-do-doc-NN set) +``` + +**Hunk C — replace the deploy-order block:** + +```diff +-## v1 deployment order +- +-1. Verify NetBox state — run NetBox imports if not already applied +- - `netbox/ipv4-prefixes-import.py` — required +- - `netbox/ipv6-mark-reserved.py` — required (Q3: tag existing IPv6 entries) +-2. Run pre-flight checks (`scripts/pre-flight-checks.sh`) +-3. Backup current cloud state (`runbooks/00-pre-deploy.md`) +-4. Destroy existing OpenStack model (`runbooks/01-destroy-model.md`) +-5. Deploy new bundle (`runbooks/02-deploy.md`) +-6. Initialize Vault (`runbooks/03-vault-init.md`) +-7. Set up Magnum domain (`runbooks/04-magnum-domain.md`) +-8. Stand up CAPI bootstrap cluster on `capi-mgmt.maas` (`runbooks/04a-capi-bootstrap-cluster.md`) +-9. Install Magnum CAPI Helm driver (`runbooks/05-magnum-capi-driver.md`) +-10. Recreate tenant resources (`runbooks/06-tenant-setup.md`) +-11. Populate DNS zones (`runbooks/07-dns-zones.md`) +-12. Run validation (`runbooks/08-validate.md` + `scripts/validate.sh`) ++## v1 deployment order ++ ++The deploy is executed via the `runbooks/v1-do-doc-NN-*.md` execution documents in numeric order: ++ ++| Doc | Purpose | ++|---|---| ++| `v1-do-doc-01-prep.md` | Pre-flight state check (repo, openrc, MAAS state of 5 VMs) | ++| `v1-do-doc-02-pki.md` | Octavia PKI overlay generation | ++| `v1-do-doc-03-destroy.md` | Conditional model + MAAS teardown (clean state for rebuild) | ++| `v1-do-doc-04-deploy.md` | `juju deploy` + settle wait + on-disk PKI verification | ++| `v1-do-doc-05-vault-init.md` | Vault initialization + cert cascade + admin-openrc regeneration | ++| `v1-do-doc-06-magnum-domain.md` | Magnum Keystone domain setup | ++| `v1-do-doc-07-capi-bootstrap.md` | CAPI bootstrap cluster + workload pivot | ++| `v1-do-doc-08-magnum-driver.md` | Magnum CAPI Helm driver graft | ++| `v1-do-doc-09-tenant.md` | Tenant project/user/openrc + Snapshot 2 | ++| `v1-do-doc-10-validate.md` | D-011 acceptance criteria + Snapshot 3 | ++ ++NetBox imports are run separately (gated on external NetBox engineer review; see `netbox/README.md`). +``` + +**Hunk D — v1 scope note about Designate deferral:** + +```diff + ## v1-specific design decisions (summary; see docs/design-decisions.md for full record) + + - **D-015 v1/v2 fork** — IPv4-only v1; IPv6/dual-stack v2 deferred + - **D-016 IPv4 tenant pool hybrid model** — NetBox owns upstream `/16` pool; Neutron owns per-project subnets within it + - **D-003 Option B network architecture** — Provider `/22` carries both ext_net FIPs (`10.12.4.10–.223`) and OpenStack public API VIPs (`10.12.4.224–.254`) on the same L2 segment; fixes the tenant→API unreachability that caused Magnum OCCM crashloop on Bobcat testcloud + - **D-005 Ceph Squid** — matches Caracal default; rehearses Roosevelt + - **D-006 Vault HA backend = etcd + easyrsa** + - **D-007 Magnum from day one** — charm in bundle + CAPI Helm driver graft +-- **D-008 DNS via Designate from day one** — static /etc/hosts for bootstrap; Designate handles tenant-level resolution (A records only for v1) ++- **D-019 (supersedes D-008) DNS scope reduction for v1** — Designate deferred to v2 alongside corporate DNS / NS-delegation work. Tenant subnets use public DNS (`1.1.1.1` / `1.0.0.1`) directly via `--dns-nameserver`. `*.cloud.neumatrix.local` FQDN tree remains internal-only, resolved via static `/etc/hosts` on bootstrap-relevant hosts. + - **D-009 Hacluster relations included at num_units=1** — decorative on testcloud; documents the relation pattern for Roosevelt scale-up + - **No OVN pinning on testcloud** — Roosevelt bare-metal will pin via `ovn-source` +``` + +### Commit message + +``` +README: refresh stale runbook references, reflect D-019 scope reduction + +The v1 deployment order block referenced runbooks 00-08 which are +being moved to deprecated/ in favor of v1-do-doc-NN execution documents. +Replaces the order block with a pointer to the do-document set. + +Also reflects D-019: Designate deferred to v2; v1 tenant resolvers use +public DNS. Adds the netbox-vip-queue.md reference. Updates the design- +decisions D-range from D-001-D-016 to D-001-D-019. +``` + +--- + + +## 5. Move superseded runbooks to `runbooks/deprecated/` + +### What + +`git mv` each superseded runbook into a new `runbooks/deprecated/` directory. Add `runbooks/deprecated/README.md` explaining the deprecation in commit 6. + +### Why + +The v1-do-doc-NN set replaces the prior runbook 00-08 work. Keeping the originals in `runbooks/deprecated/` preserves the audit trail without misleading new operators into following the old paths. + +### Files to move + +| From | To | Replacement | +|---|---|---| +| `runbooks/00-pre-deploy.md` | `runbooks/deprecated/00-pre-deploy.md` | superseded by D-017 + D-018 (no per-cycle backups; teardown direct-to-MAAS); v1-do-doc-01 covers prep | +| `runbooks/01a-octavia-pki-generation.md` | `runbooks/deprecated/01a-octavia-pki-generation.md` | `v1-do-doc-02-pki.md` | +| `runbooks/02-deploy.md` | `runbooks/deprecated/02-deploy.md` | `v1-do-doc-04-deploy.md` | +| `runbooks/03-vault-init.md` | `runbooks/deprecated/03-vault-init.md` | `v1-do-doc-05-vault-init.md` | +| `runbooks/04-magnum-domain.md` | `runbooks/deprecated/04-magnum-domain.md` | `v1-do-doc-06-magnum-domain.md` | +| `runbooks/04a-capi-bootstrap-cluster.md` | `runbooks/deprecated/04a-capi-bootstrap-cluster.md` | `v1-do-doc-07-capi-bootstrap.md` | +| `runbooks/05-magnum-capi-driver.md` | `runbooks/deprecated/05-magnum-capi-driver.md` | `v1-do-doc-08-magnum-driver.md` | +| `runbooks/06-tenant-setup.md` | `runbooks/deprecated/06-tenant-setup.md` | `v1-do-doc-09-tenant.md` | +| **`runbooks/07-dns-zones.md`** | **`runbooks/deprecated/07-dns-zones.md`** | **deferred to v2 per D-019** (no v1 replacement) | +| `runbooks/08-validate.md` | `runbooks/deprecated/08-validate.md` | `v1-do-doc-10-validate.md` | + +### Files NOT moved + +| File | Why kept in `runbooks/` | +|---|---| +| `runbooks/01-destroy-model.md` | Referenced by v1-do-doc-03 as a conditional sub-procedure; still active | + +### Git commands + +```bash +cd "$HOME/openstack-caracal-ipv4" + +mkdir -p runbooks/deprecated + +git mv runbooks/00-pre-deploy.md runbooks/deprecated/ +git mv runbooks/01a-octavia-pki-generation.md runbooks/deprecated/ +git mv runbooks/02-deploy.md runbooks/deprecated/ +git mv runbooks/03-vault-init.md runbooks/deprecated/ +git mv runbooks/04-magnum-domain.md runbooks/deprecated/ +git mv runbooks/04a-capi-bootstrap-cluster.md runbooks/deprecated/ +git mv runbooks/05-magnum-capi-driver.md runbooks/deprecated/ +git mv runbooks/06-tenant-setup.md runbooks/deprecated/ +git mv runbooks/07-dns-zones.md runbooks/deprecated/ +git mv runbooks/08-validate.md runbooks/deprecated/ + +git status +# Expect: 10 renames staged, runbooks/01-destroy-model.md untouched +``` + +### Commit message + +``` +runbooks: move superseded files to runbooks/deprecated/ + +These are replaced by v1-do-doc-NN-*.md execution documents (added in +follow-up commits). The 01-destroy-model.md runbook stays in place — it's +referenced by v1-do-doc-03 as a conditional sub-procedure. + +07-dns-zones.md is deferred to v2 per D-019 with no v1 replacement +(Designate is no longer in v1 scope). + +History preserved via git mv. +``` + +--- + + +## 6. Add a deprecation banner to `runbooks/deprecated/README.md` + +### What + +Create a new file `runbooks/deprecated/README.md` with a banner explaining the deprecation scope and a replacement map. + +### Content + +```markdown +# Deprecated v1 Runbooks + +The runbooks in this directory have been superseded by the +`runbooks/v1-do-doc-NN-*.md` execution documents (or, in the case of +`07-dns-zones.md`, deferred to v2 entirely per D-019). + +They are preserved here so the audit trail from the early v1 drafting +phase remains accessible. **Do not execute them.** The v1 deploy is +gated through the do-document set. + +## Replacement map + +| Deprecated runbook | Replacement | +|---|---| +| `00-pre-deploy.md` | superseded by D-017 + D-018 (no per-cycle backups; direct MAAS teardown); `v1-do-doc-01-prep.md` covers prep | +| `01a-octavia-pki-generation.md` | `v1-do-doc-02-pki.md` | +| `02-deploy.md` | `v1-do-doc-04-deploy.md` | +| `03-vault-init.md` | `v1-do-doc-05-vault-init.md` | +| `04-magnum-domain.md` | `v1-do-doc-06-magnum-domain.md` | +| `04a-capi-bootstrap-cluster.md` | `v1-do-doc-07-capi-bootstrap.md` | +| `05-magnum-capi-driver.md` | `v1-do-doc-08-magnum-driver.md` | +| `06-tenant-setup.md` | `v1-do-doc-09-tenant.md` | +| `07-dns-zones.md` | **deferred to v2 per D-019** (no v1 replacement) | +| `08-validate.md` | `v1-do-doc-10-validate.md` | + +`01-destroy-model.md` is **not** in this directory — it remains active in +`runbooks/` and is referenced as a conditional sub-procedure by +`v1-do-doc-03-destroy.md`. +``` + +### Commit message + +``` +runbooks/deprecated: add README explaining the deprecation scope + +Includes the deprecated → replacement mapping for operators who arrive +via git log searches or stale internal references. +``` + +--- + + +## 7. Bundle: remove Designate (per D-019) + +### What + +In `bundle.yaml`, remove four applications, seven relations, and update a header comment to reflect Designate deferral to v2. VIP `10.12.4.227` becomes unused space in the `10.12.4.224-.254` range (same status as `10.12.4.225` reserved for v2 ceph-radosgw HA). + +### Why + +Per D-019 (added by commit 8 of this change list): Designate is deferred to v2 alongside corporate-DNS / NS-delegation work. v1 testcloud topology investigation (2026-05-27 session) confirmed: + +1. **Outside-in DNS** isn't needed for v1 — corporate clients reach the cloud through the existing `openstack.baldurkeep.com → 10.17.4.20 → 10.12.x` HTTPS proxy chain, not via the `*.cloud.neumatrix.local` FQDN tree. The edge nginx (`neumatrix-nginx` at `10.17.8.7`) cannot route to `10.12.x` directly anyway. +2. **Inside-out DNS** doesn't require Designate — tenant subnets can specify public DNS (`1.1.1.1`, `1.0.0.1`) directly via `--dns-nameserver` at subnet-create time. +3. **FIP DNS auto-registration** (the remaining v1 use case for Designate) is nice-to-have, not load-bearing for any v1 acceptance criterion. + +Removing Designate now removes one charm, four applications (with subordinate routers), seven relations, and one VIP from the v1 deploy surface, reducing first-deploy troubleshooting scope. + +### Diff + +**Remove four applications** (search-and-delete each block in `bundle.yaml`): + +```yaml + # ===================================================================== + # DNS: Designate (NEW for Caracal v1 per D-008) + # ===================================================================== + # Naming convention: .omega.dc0.vr0.cloud.neumatrix.local + + designate: + charm: designate + channel: 2024.1/stable + num_units: 1 + to: [lxd:8] + options: + openstack-origin: *openstack-origin + nameservers: "ns1.omega.dc0.vr0.cloud.neumatrix.local. ns2.omega.dc0.vr0.cloud.neumatrix.local." + vip: 10.12.4.227 + os-public-hostname: designate.omega.dc0.vr0.cloud.neumatrix.local + bindings: *api-bindings + constraints: arch=amd64 + + designate-mysql-router: + charm: mysql-router + channel: 8.0/stable + + designate-bind: + charm: designate-bind + channel: 2024.1/stable + num_units: 1 + to: [lxd:8] + bindings: + "": provider # unit on provider so bind9:53 reachable from tenants (D-003) + cluster: metal # peer traffic stays internal (decorative with num_units=1) + constraints: arch=amd64 +``` + +**Remove the `designate-hacluster:` line** from the hacluster subordinate block: + +```diff + keystone-hacluster: { charm: hacluster, channel: 2.4/stable } + glance-hacluster: { charm: hacluster, channel: 2.4/stable } + neutron-api-hacluster: { charm: hacluster, channel: 2.4/stable } + nova-cloud-controller-hacluster: { charm: hacluster, channel: 2.4/stable } + placement-hacluster: { charm: hacluster, channel: 2.4/stable } + openstack-dashboard-hacluster: { charm: hacluster, channel: 2.4/stable } + cinder-hacluster: { charm: hacluster, channel: 2.4/stable } + octavia-hacluster: { charm: hacluster, channel: 2.4/stable } + barbican-hacluster: { charm: hacluster, channel: 2.4/stable } + magnum-hacluster: { charm: hacluster, channel: 2.4/stable } + vault-hacluster: { charm: hacluster, channel: 2.4/stable } + # v2-deferred: ceph-radosgw-hacluster: { charm: hacluster, channel: 2.4/stable } +- designate-hacluster: { charm: hacluster, channel: 2.4/stable } ++ # v2-deferred (D-019): designate-hacluster: { charm: hacluster, channel: 2.4/stable } +``` + +**Remove seven relations** from the `relations:` block: + +```yaml + # ---- Designate (DNS) — NEW for Caracal v1 per D-008 + - [designate-mysql-router:db-router, mysql-innodb-cluster:db-router] + - [designate-mysql-router:shared-db, designate:shared-db] + - [designate:identity-service, keystone:identity-service] + - [designate:amqp, rabbitmq-server:amqp] + - [designate:certificates, vault:certificates] + - [designate:dns-backend, designate-bind:dns-backend] + - [designate:ha, designate-hacluster:ha] +``` + +**Update the header comment block** (decision references in the bundle's top-of-file block): + +```diff + D-006 Vault HA via etcd + easyrsa + D-007 Magnum Layer A + Layer B graft +- D-008 Designate day-one ++ D-019 (supersedes D-008) Designate deferred to v2 + D-009 hacluster subordinates (decorative on testcloud) +``` + +**Update the HA subordinate header comment:** + +```diff + # ===================================================================== +- # HA Cluster Subordinates (12 active for v1; ceph-radosgw deferred to v2) ++ # HA Cluster Subordinates (11 active for v1; ceph-radosgw + designate deferred to v2) + # ===================================================================== +``` + +### Verification post-edit + +```bash +cd "$HOME/openstack-caracal-ipv4" + +# 1. No designate application remains +grep -E "^ designate" bundle.yaml \ + && echo "[FAIL] designate-related application still present" \ + || echo "[OK] no designate applications" + +# 2. No designate relations remain +grep -E "designate" bundle.yaml | grep -vE "^[[:space:]]*#" +# Expect: no output (the only remaining 'designate' tokens should be commented) + +# 3. VIP count is now 11 +VIP_COUNT=$(grep -cE "^[[:space:]]+vip: 10\.12\.4\." bundle.yaml) +echo "VIPs: $VIP_COUNT (expect 11)" + +# 4. YAML still parses +python3 -c "import yaml; yaml.safe_load(open('bundle.yaml')); print('[OK] YAML parses')" +``` + +### Commit message + +``` +bundle: remove Designate per D-019 (deferred to v2) + +Removes the designate, designate-bind, designate-mysql-router, and +designate-hacluster applications, plus all seven designate-related +relations. Updates header comments to reflect the deferral. + +Rationale per D-019: v1 testcloud topology investigation confirmed +outside-in DNS is not needed (corporate clients reach the cloud via +the openstack.baldurkeep.com HTTPS proxy chain, not via *.cloud. +neumatrix.local FQDNs). Tenant subnets use public DNS directly via +--dns-nameserver. FIP DNS auto-registration is not load-bearing for +any v1 acceptance criterion. + +VIP 10.12.4.227 becomes unused space in 10.12.4.224-.254 (same +status as 10.12.4.225 reserved for v2 ceph-radosgw HA). + +Reduces v1 deploy surface by one charm + three subordinates + +seven relations + one VIP. +``` + +--- + + +## 8. design-decisions.md: add D-019 + amend D-008 and D-011 + +### What + +Three coordinated edits in `docs/design-decisions.md`: + +1. Add a new D-019 entry with the full deferral rationale and v2 deltas +2. Mark D-008 as "superseded by D-019 — v2-scope" +3. Amend D-011 (validation bar) to remove the "Designate resolves" criterion + +### Why + +Captures the decision in the authoritative design-decisions record. Preserves the audit trail (D-008 stays with its original content but with the superseded status flag). + +### Diff — Hunk A: amend D-008 status + +```diff + ## D-008: DNS architecture + +-**Decision:** Layered — static /etc/hosts for bootstrap + Designate (in bundle from day one) for tenant-level resolution. ++**Status:** Superseded by D-019 (2026-05-27). v2-scope. Original decision text preserved below for audit. ++ ++**Decision (original; superseded):** Layered — static /etc/hosts for bootstrap + Designate (in bundle from day one) for tenant-level resolution. +``` + +### Diff — Hunk B: amend D-011 validation bar + +In the D-011 section, the testcloud validation criteria currently list (among others) a Designate resolution check. Remove the relevant bullet. + +```diff + ## D-011: Roosevelt-rehearsal validation bar + + **Decision:** v1 testcloud must pass these criteria before being declared "deploy-equivalent" to Roosevelt: + + - All charms `active/idle` (per `juju status`) + - Tenant subnet → OpenStack API reachability (Bobcat Magnum OCCM crashloop regression test) + - Octavia LBaaS end-to-end (LB create + member health + failover + recovery) + - Magnum CAPI end-to-end (cluster template + cluster create + CREATE_COMPLETE) + - Vault unseal-after-reboot survives a power cycle +-- Designate resolves API FQDNs via the Designate VIP + - Snapshots 1, 2, 3 captured at the appropriate stages (per D-012) + ++**Amendment (2026-05-27):** Per D-019, the "Designate resolves" criterion is removed for v1. Designate is deferred to v2; tenant subnets resolve via public DNS. v2 will reinstate a DNS-resolution validation criterion calibrated to whatever DNS mechanism is in place (NS delegation from corporate DNS, or otherwise). +``` + +### Diff — Hunk C: add D-019 (new entry, append to end of design-decisions.md) + +```markdown +--- + + +## D-019: DNS scope reduction for v1 — Designate deferred to v2 + +**Decision (2026-05-27):** Designate is removed from the v1 testcloud bundle and deferred to v2 alongside corporate DNS / NS delegation work. v1 tenant subnets resolve via public DNS (`1.1.1.1`, `1.0.0.1`) directly via the `--dns-nameserver` option at subnet-create time. + +**Supersedes:** D-008 (DNS architecture). + +**Amends:** D-011 (validation bar — removes "Designate resolves" criterion). + +### Rationale + +Three findings from the 2026-05-27 testcloud topology investigation: + +1. **Outside-in DNS** (corporate clients resolving `*.cloud.neumatrix.local`) is not needed for v1. Corporate access to the cloud already flows through the existing `openstack.baldurkeep.com → 10.17.4.20 → 10.12.x` HTTPS proxy chain (handled by the edge nginx at `10.17.8.7`), which does not depend on corporate-side resolution of cloud-internal FQDNs. + +2. **The edge nginx cannot route to `10.12.x` directly.** Inspection confirmed the edge has only `10.17.8.7/22` plus a tailscale interface; reaching `10.12.4.x` requires the libvirt-host NAT path. Adding DNS to the testcloud would require parallel UDP/53 NAT/proxy plumbing across three hosts (edge nginx, libvirt host, internal nginx) for a feature that has no v1 consumer. + +3. **Inside-out DNS** (tenant VMs resolving external names) is satisfied by tenant subnets pointing `--dns-nameserver` at public DNS (`1.1.1.1`, `1.0.0.1`). Designate is not needed in the inside-out path either, since: + - Tenant VMs don't need to resolve cloud-internal FQDNs (their API access goes through documented IPs / `--cloud` configs in cloud.conf) + - Cross-tenant DNS visibility is not a v1 requirement + +The remaining v1 use case for Designate (FIP DNS auto-registration via the `neutron-api ↔ designate` integration) is informational only — nothing in v1 consumes those records. + +### v1 implementation + +- Tenant subnets created with `--dns-nameserver 1.1.1.1 --dns-nameserver 1.0.0.1` (or via the openrc `OS_DNS_NAMESERVERS` env) +- CAPI workload cluster template variable `OPENSTACK_DNS_NAMESERVERS` set to `1.1.1.1,1.0.0.1` (per `v1-do-doc-07-capi-bootstrap.md` §13) +- Cloud-internal `*.cloud.neumatrix.local` FQDN tree resolved via static `/etc/hosts` on bootstrap-relevant hosts (jumphost, openstack0-3, LXD containers per charm bootstrap, capi-mgmt — staged in `v1-do-doc-05-vault-init.md` §11 and `v1-do-doc-07-capi-bootstrap.md` §6) +- Charms continue to use FQDN-based `os-public-hostname` (cert SANs depend on it) — internal resolution via `/etc/hosts` is sufficient + +### v2 plan + +- Re-introduce Designate (charm + designate-bind + relations + hacluster sub) +- NS delegation from corporate DNS to designate-bind on a real (non-NAT) network VIP +- Tenant subnets transitioning to use Designate VIP as their resolver (after corporate DNS delegation lands) +- Designate v2 deploy on a real-network Roosevelt or v2-testcloud topology where the bridging-host complexity from v1 testcloud doesn't apply +- D-011 validation re-introduces a calibrated DNS-resolution criterion (mechanism TBD: NS delegation working end-to-end vs static A records at corporate DNS) + +### v2-residency note + +The IPv6 prefixes already imported into NetBox (and marked Reservation status) include allocations that would be appropriate for Designate's VIPs in a v2 design — these stay in NetBox as Reservation until v2 work begins. +``` + +### Commit message + +``` +docs/design-decisions: add D-019, amend D-008/D-011 (DNS scope reduction) + +D-019 captures the Designate deferral to v2 with rationale grounded in +the 2026-05-27 testcloud topology investigation: outside-in DNS not +needed (corporate clients use openstack.baldurkeep.com HTTPS chain); +edge nginx can't route to cloud-internal anyway; inside-out is +satisfied by tenant --dns-nameserver pointing at public DNS. + +D-008 (DNS architecture) marked superseded; original text preserved +for audit trail. + +D-011 validation bar amended to remove "Designate resolves" criterion; +v2 will reinstate a calibrated DNS criterion. +``` + +--- + + +## 9. netbox-vip-queue.md: drop the Designate VIP row + +### What + +In `docs/netbox-vip-queue.md`, remove the row for Designate's VIP `10.12.4.227`. The queue goes from 12 entries to 11. + +### Why + +Per D-019, Designate has no VIP in v1. `10.12.4.227` becomes unused space within the `10.12.4.224-.254` range — same status as `10.12.4.225` (reserved for v2 ceph-radosgw HA). + +### Diff + +```diff + # NetBox VIP Queue (post-deploy) + + The following 12 IPAddress entries should be added to NetBox after the + v1 deploy completes and the engineer-review of `netbox/ipv4-prefixes- + import.py` has landed. + +-| VIP | Service | Notes | +-|---|---|---| +-| 10.12.4.224 | barbican | per D-003 | +-| 10.12.4.226 | cinder | per D-003 | +-| 10.12.4.227 | designate | per D-003 + D-008 | +-| 10.12.4.228 | glance | per D-003 | +-| 10.12.4.229 | keystone | per D-003 | +-| 10.12.4.230 | magnum | per D-003 | +-| 10.12.4.231 | neutron-api | per D-003 | +-| 10.12.4.232 | nova-cloud-controller | per D-003 | +-| 10.12.4.233 | octavia | per D-003 | +-| 10.12.4.234 | openstack-dashboard | per D-003 | +-| 10.12.4.235 | placement | per D-003 | +-| 10.12.4.236 | vault | per D-003 | ++The following 11 IPAddress entries should be added to NetBox after the ++v1 deploy completes and the engineer-review of `netbox/ipv4-prefixes- ++import.py` has landed. ++ ++| VIP | Service | Notes | ++|---|---|---| ++| 10.12.4.224 | barbican | per D-003 | ++| 10.12.4.226 | cinder | per D-003 | ++| 10.12.4.228 | glance | per D-003 | ++| 10.12.4.229 | keystone | per D-003 | ++| 10.12.4.230 | magnum | per D-003 | ++| 10.12.4.231 | neutron-api | per D-003 | ++| 10.12.4.232 | nova-cloud-controller | per D-003 | ++| 10.12.4.233 | octavia | per D-003 | ++| 10.12.4.234 | openstack-dashboard | per D-003 | ++| 10.12.4.235 | placement | per D-003 | ++| 10.12.4.236 | vault | per D-003 | ++ ++**Reserved (unused in v1):** ++ ++- `10.12.4.225` — reserved for v2 ceph-radosgw HA (workstream-2 decision) ++- `10.12.4.227` — reserved for v2 designate (D-019 deferral) +``` + +(The exact line counts may differ if the existing file has more preamble — adjust to the actual structure when editing.) + +### Commit message + +``` +docs/netbox-vip-queue: drop designate row per D-019 + +VIP 10.12.4.227 is no longer in v1; designate deferred to v2. Adds a +"reserved (unused in v1)" section to make the unused slots in the +.224-.254 range explicit for future reference. +``` + +--- + + +## 10. Verification (read-only) after the nine commits land + +```bash +cd "$HOME/openstack-caracal-ipv4" +git pull +git log --oneline -10 + +echo "=== 1. ceph-osd no longer has a storage block ===" +grep -A 12 "^ ceph-osd:" bundle.yaml | grep "storage:" \ + && echo "[FAIL] storage: block still present in ceph-osd" \ + || echo "[OK] no storage: in ceph-osd" + +echo "=== 2. D-002 vault token removed ===" +grep -E "OpenStack core API charms" docs/design-decisions.md | grep -v "vault)" \ + && echo "[OK]" \ + || echo "[FAIL] D-002 still contains vault token" + +echo "=== 3. D-014 reflects new repo path ===" +grep "OpenStack/openstack-caracal-ipv4" docs/design-decisions.md \ + && echo "[OK]" \ + || echo "[FAIL]" + +echo "=== 4. README reflects v1-do-doc set ===" +grep "v1-do-doc-NN" README.md \ + && echo "[OK]" \ + || echo "[FAIL]" + +echo "=== 5. Designate gone from bundle ===" +grep -E "^ designate" bundle.yaml \ + && echo "[FAIL] designate-related app still present" \ + || echo "[OK] no designate apps" + +grep -E "designate" bundle.yaml | grep -vE "^[[:space:]]*#" +# Expect: empty (only commented references remain) + +echo "=== 6. VIP count is 11 ===" +VIP_COUNT=$(grep -cE "^[[:space:]]+vip: 10\.12\.4\." bundle.yaml) +echo "VIPs: $VIP_COUNT (expect 11)" + +echo "=== 7. D-019 added ===" +grep "^## D-019" docs/design-decisions.md \ + && echo "[OK]" \ + || echo "[FAIL]" + +echo "=== 8. D-008 marked superseded ===" +grep -A 2 "^## D-008" docs/design-decisions.md | grep "Superseded by D-019" \ + && echo "[OK]" \ + || echo "[FAIL]" + +echo "=== 9. netbox-vip-queue.md has 11 entries ===" +QUEUE_COUNT=$(grep -cE "^\| 10\.12\.4\." docs/netbox-vip-queue.md) +echo "VIP queue entries: $QUEUE_COUNT (expect 11)" + +echo "=== 10. runbooks/deprecated/ has 10 files + README ===" +ls runbooks/deprecated/ | wc -l +# Expect: 11 (10 deprecated runbooks + README.md) + +echo "=== 11. YAML still parses ===" +python3 -c "import yaml; yaml.safe_load(open('bundle.yaml')); print('[OK] YAML parses')" +``` + +--- + + +## 11. Acceptance criteria + +- [ ] All 9 commits landed and pushed to `main` +- [ ] Verification section 10 returns `[OK]` for all 11 checks +- [ ] `git log --oneline` shows the 9 commits in order +- [ ] `bundle.yaml` parses cleanly via `python3 -c "import yaml; yaml.safe_load(open('bundle.yaml'))"` +- [ ] No untracked changes (`git status` clean except for any local scratch) + +Once all checked, proceed to `v1-do-doc-01-prep.md` execution. + +--- + + +## 12. Change log + +| Date | Change | Reference | +|---|---|---| +| 2026-05-27 | v1 (six commits): ceph-osd storage block; D-002/D-014 cleanups; README refresh; deprecated runbook moves; deprecation README | Initial drafting | +| 2026-05-27 | v2 (nine commits): + Designate deferral commits 7/8/9; amended commit 5/6/10 to reflect 07-dns-zones permanent deprecation; updated commit 4 README hunks; VIP count expectations updated from 12 to 11 | D-019 decision; testcloud topology investigation | diff --git a/runbooks/v1-do-doc-01-prep.md b/runbooks/v1-do-doc-01-prep.md new file mode 100644 index 0000000..4ec3901 --- /dev/null +++ b/runbooks/v1-do-doc-01-prep.md @@ -0,0 +1,349 @@ +# v1 Do-Document 01 — Pre-Deploy State Check + +**Status:** First execution document of Batch A. Pure read-only verification. No cloud state is mutated. + +**Position in sequence:** Runs after `docs/v1-pre-deploy-fixes.md` (the repo-hygiene fixes) have been committed and pushed. Runs before `v1-do-doc-02-pki.md`. + +**Replaces:** `runbooks/deprecated/00-pre-deploy.md` (which contained references to D-013 graceful CAPI teardown, capi-mgmt preservation, and `netbox/vlans-import.py` — all superseded or removed). + +**Cross-references:** + +- D-014 (repo path) — verifies clone matches +- D-015 (v1/v2 fork) — v1 scope confirmed +- D-017 (CAPI bootstrap cluster lifecycle) — every cycle is a full rebuild; no preservation +- D-018 (teardown strategy) — MAAS-release-direct; pre-existing teardown verified + +--- + + +## 1. Purpose & scope + +This document confirms the jumphost, repo, openrc files, Juju controller, and MAAS state are all in the expected pre-deploy posture before any cloud-touching execution begins. + +**What this document does:** + +- Verifies jumphost identity and shell context +- Verifies repo cloned at `$HOME/openstack-caracal-ipv4` with credential helper configured +- Verifies the repo is on `main` and pulled up to date (so the pre-deploy fixes from `docs/v1-pre-deploy-fixes.md` are local) +- Verifies admin and user1 openrc files exist and source cleanly +- Verifies the Juju controller for the `openstack` model is reachable +- Verifies all 5 cloud-target VMs are MAAS-Ready +- Explicitly acknowledges what is NOT in scope for this prep phase + +**What this document does NOT do:** + +- Run NetBox imports (pinned for external NetBox-engineer review; not part of v1 deploy flow per `netbox/README.md`) +- Take KVM snapshots (per D-017, every cycle is a full rebuild; no rollback target needed; pre-existing snapshots remain as a safety net but are not refreshed here) +- Back up Vault unseal keys (the prior-cloud Vault keys are accepted lost per the Caracal_Rebuild handoff; the Caracal deploy will reinit Vault from scratch) +- Graceful CAPI workload teardown (superseded by D-018) +- Backup of `juju export-bundle` output (the canonical bundle is `bundle.yaml` in the repo; export is for diagnostics only and runs only if cloud is currently up) + +**Out of scope:** + +- Bundle review (done; tracked in `docs/v1-pre-deploy-fixes.md`) +- PKI generation (next document, `v1-do-doc-02-pki.md`) +- Cloud destroy (`runbooks/01-destroy-model.md`; pre-existing teardown likely already done; verify-only path used by v1-do-doc-03) + +--- + + +## 2. Decisions captured + +| Decision | Choice | Roosevelt parallel | +|---|---|---| +| Snapshot strategy for this rebuild | None taken here; existing KVM snapshots remain as safety net but per D-017 every cycle is a full rebuild | N/A — Roosevelt is bare-metal; equivalent is MAAS re-deploy of all hosts | +| Vault key backup | Skipped; prior keys accepted lost | Vault will be re-initialized from scratch each Roosevelt rebuild too | +| NetBox import sequence | Out of scope; pinned external | Roosevelt's NetBox state will be set up before the bundle pulls values | +| State capture | Skipped for this Caracal cycle (cloud is already down) | On Roosevelt, capture `juju export-bundle` and `juju status` BEFORE the next teardown cycle | + +--- + + +## 3. Prerequisites + +| Prereq | Verification command | +|---|---| +| You are SSH'd into jumphost `vopenstack-jesse` as `jessea123` | `hostname && id -un` | +| `$HOME` is writable | `test -w "$HOME" && echo OK` | +| `git`, `juju`, `maas`, `openssl`, and `python3` available on PATH | `for t in git juju maas openssl python3; do command -v $t || echo "MISSING: $t"; done` | +| The openstack snap is installed (used in later runbooks; not strictly needed here) | `snap list openstackclients 2>/dev/null || echo "(not installed; v1-do-doc-04 will check)"` | + +If any prereq fails, stop and resolve before continuing. + +--- + + +## 4. Step-by-step state check + +Each section is a copy-paste block. The blocks are read-only. No `exit` is used in interactive context; failures are reported as text and operator decides whether to proceed. + +### 4.1 Jumphost identity + +```bash +echo "=== Jumphost identity ===" +hostname +id -un +date -Iseconds +echo "=== Disk space ===" +df -h "$HOME" /var/lib/libvirt/images 2>/dev/null +``` + +**Expected:** + +- `hostname` → `vopenstack-jesse` (or your configured equivalent) +- `id -un` → `jessea123` +- `$HOME` free space: at least 5 GB (octavia PKI, deploy logs, kubeconfigs all live here) +- `/var/lib/libvirt/images` free space: enough for the existing OSD qcow2 files (already trimmed to ~200 KiB each per teardown) + +### 4.2 Repository clone present at agreed path + +```bash +export REPO="$HOME/openstack-caracal-ipv4" + +if [ ! -d "$REPO/.git" ]; then + echo "MISSING: $REPO does not contain a git checkout." + echo "Run the agreed clone procedure (HTTPS clone with 72h credential cache) first:" + echo " cd \$HOME && git clone https://git.baldurkeep.com/git/OpenStack/openstack-caracal-ipv4.git" + echo " cd $REPO && git config credential.helper \"cache --timeout=259200\"" + echo "" + echo "Stop here. Re-run this section after the clone." +else + echo "[OK] Repo present at: $REPO" + cd "$REPO" + echo "" + echo "=== Current branch ===" + git branch --show-current + echo "" + echo "=== Remote URL ===" + git remote get-url origin + echo "" + echo "=== Credential helper ===" + git config --get credential.helper || echo "(no credential helper configured)" + echo "" + echo "=== Working tree status ===" + git status --short + echo "" + echo "=== Latest commit ===" + git log --oneline -1 +fi +``` + +**Expected:** + +- Branch: `main` +- Remote URL: `https://git.baldurkeep.com/git/OpenStack/openstack-caracal-ipv4.git` +- Credential helper: `cache --timeout=259200` (or however configured) +- Working tree status: empty (clean) +- Latest commit: should reflect the most recent pre-deploy-fixes commit (the one that moved 8 runbooks to deprecated/) + +If branch is not `main`, the working tree is dirty, or commits are behind the remote, stop and reconcile before continuing. + +### 4.3 Pre-deploy fixes are present locally + +```bash +cd "$REPO" + +echo "=== ceph-osd block (after pre-deploy fix): should NOT contain a 'storage:' line ===" +grep -A 12 "^ ceph-osd:" bundle.yaml +echo "" + +echo "=== Verify ceph-osd has no storage block ===" +grep -A 12 "^ ceph-osd:" bundle.yaml | grep "^ storage:" \ + && echo "[FAIL] storage block still present under ceph-osd" \ + || echo "[OK] no storage block in ceph-osd" +echo "" + +echo "=== expected-osd-count on ceph-mon: should be 4 ===" +grep -A 8 "^ ceph-mon:" bundle.yaml | grep "expected-osd-count" +echo "" + +echo "=== VIP grep: should return 12 ===" +grep -cE "^[[:space:]]+vip: 10\.12\.4\." bundle.yaml +echo "" + +echo "=== D-002 Vault row dedup: dedicated row remains, OS-core row no longer mentions vault ===" +grep -c "Vault.*1.8/stable" docs/design-decisions.md +grep "magnum, vault)" docs/design-decisions.md && echo "[FAIL] vault still in OS-core row" || echo "[OK] vault not in OS-core row" +echo "" + +echo "=== D-014 repo path: should be OpenStack/openstack-caracal-ipv4 ===" +grep -A 3 "^## D-014" docs/design-decisions.md | grep "Repo path:" +echo "" + +echo "=== Deprecated runbooks moved: count should be 8 (or 9 with README) ===" +ls runbooks/deprecated/ 2>/dev/null | wc -l +``` + +**Expected:** + +- ceph-osd block shows `options.osd-devices: /dev/vdb` and NO `storage:` block +- `storage:` count under ceph-osd: 0 +- `expected-osd-count: 4` +- VIP grep returns 12 +- D-002: one `Vault.*1.8/stable` match; OS-core row does NOT mention vault +- D-014: shows `OpenStack/openstack-caracal-ipv4` +- `runbooks/deprecated/` contains 8 runbooks + 1 README + +If any of these don't match, the pre-deploy fixes did not land correctly. Stop and reconcile (re-pull, re-apply missing commits). + +### 4.4 Openrc files present and source cleanly + +```bash +echo "=== admin openrc ===" +if [ -f "$HOME/admin-openrc" ]; then + echo "[OK] $HOME/admin-openrc exists" + ( source "$HOME/admin-openrc"; \ + env | grep -E "^OS_" | grep -v PASSWORD | sort ) +else + echo "[MISSING] $HOME/admin-openrc" +fi + +echo "" +echo "=== user1 openrc ===" +if [ -f "$HOME/user1-openrc" ]; then + echo "[OK] $HOME/user1-openrc exists" + ( source "$HOME/user1-openrc"; \ + env | grep -E "^OS_" | grep -v PASSWORD | sort ) +else + echo "[MISSING] $HOME/user1-openrc" +fi +``` + +**Expected:** + +- Both files exist and contain OS_AUTH_URL, OS_USERNAME, OS_PROJECT_NAME, OS_USER_DOMAIN_NAME, OS_PROJECT_DOMAIN_NAME, OS_IDENTITY_API_VERSION, OS_REGION_NAME (and OS_PASSWORD, which is filtered out of the display). +- The subshell-source pattern (`( source ...; env ... )`) prevents the openrc from polluting your interactive shell environment. + +**Note:** these openrc files target the **prior** Bobcat cloud (or whatever was last running). They will NOT work against the Caracal cloud until that cloud is up. They're verified here as a sanity check that the files exist for the operator. New openrc files will be generated as part of `v1-do-doc-09-tenant.md` (Batch D) using the new Caracal endpoints. + +### 4.5 Juju controller reachable + +```bash +echo "=== Juju controllers ===" +juju controllers +echo "" +echo "=== Current controller ===" +juju show-controller 2>/dev/null | head -20 +echo "" +echo "=== Models on current controller ===" +juju models +``` + +**Expected:** + +- At least one controller is listed. +- The controller should be in `Available` state. +- The `openstack` model may or may not exist depending on whether the prior teardown completed. Per the Caracal_Rebuild handoff (2026-05-27 verification), it does not exist — that's expected post-teardown. + +If `juju controllers` errors, the controller is unreachable. Stop and resolve juju access before continuing. + +### 4.6 MAAS state of cloud-target VMs + +```bash +echo "=== MAAS profile ===" +maas list | head -5 +export MAAS_PROFILE=$(maas list | awk 'NR==1 {print $1}') +echo "Using MAAS_PROFILE=$MAAS_PROFILE" +echo "" + +echo "=== Cloud-target VMs (openstack0-3 + capi-mgmt): expect 5 in Ready ===" +maas "$MAAS_PROFILE" machines read 2>/dev/null \ + | python3 -c " +import json, sys +machines = json.load(sys.stdin) +targets = ['openstack0', 'openstack1', 'openstack2', 'openstack3', 'capi-mgmt'] +print(f'{\"hostname\":<15} {\"status\":<15} {\"owner\":<15} {\"system_id\":<25}') +print('-' * 70) +seen = [] +for m in machines: + h = m.get('hostname', '') + if h in targets: + seen.append(h) + status = m.get('status_name', '') + owner = m.get('owner', '') or '(none)' + sid = m.get('system_id', '') + print(f'{h:<15} {status:<15} {owner:<15} {sid:<25}') +missing = [t for t in targets if t not in seen] +if missing: + print(f'\nMISSING from MAAS: {missing}') +" +``` + +**Expected:** + +- All 5 hostnames appear: `openstack0`, `openstack1`, `openstack2`, `openstack3`, `capi-mgmt` +- Status for all 5: `Ready` +- Owner for all 5: `(none)` (unowned, ready for deploy) + +If any VM is not `Ready` or has an owner, stop. The teardown either did not complete or something has acquired the VMs since. + +### 4.7 Acknowledged "not done here" items + +This step prints a checklist for the operator to mentally acknowledge before proceeding: + +```bash +cat <<'EOF' +=== Items NOT done in this prep phase (acknowledged) === + +[ ] NetBox imports — pinned for external NetBox-engineer review; not blocking + v1 deploy. NetBox state is whatever the engineer has set up. + +[ ] KVM snapshots — per D-017, every cycle is a full rebuild; no rollback + target needed for this deploy. Pre-existing KVM-level snapshots (from + prior cycles) remain on disk as a safety net but are not refreshed here. + +[ ] Vault unseal key backup — prior Vault keys are accepted lost per the + Caracal_Rebuild handoff. Caracal deploy will run `vault operator init` + fresh in v1-do-doc-05. + +[ ] Graceful CAPI workload teardown — D-018 supersedes D-013; teardown is + MAAS-release-direct, not graceful. The teardown that happened pre-2026-05-27 + used the D-018 path. + +[ ] juju export-bundle / juju status capture — the cloud is currently down, + so there is nothing to capture. On future cycles where the cloud is up + pre-teardown, those captures happen in runbook 01 Phase A. + +Proceed only if every item above is acknowledged. +EOF +``` + +--- + + +## 5. Acceptance criteria — go/no-go for v1-do-doc-02 + +The following must all be true before proceeding: + +- [ ] §4.1: Identity correct, disk space sufficient +- [ ] §4.2: Repo cloned at `$HOME/openstack-caracal-ipv4`, on `main`, clean, with credential helper configured +- [ ] §4.3: All six pre-deploy fixes verified present in the local checkout (storage block removed, expected-osd-count=4, VIPs=12, D-002 dedup, D-014 path, 8 deprecated files) +- [ ] §4.4: Both openrc files exist and source cleanly (content of openrc not yet validated against new cloud — that's v1-do-doc-04+) +- [ ] §4.5: Juju controller reachable +- [ ] §4.6: All 5 cloud-target VMs MAAS-Ready, unowned +- [ ] §4.7: "Not done here" items all acknowledged + +If all checked, proceed to `v1-do-doc-02-pki.md`. + +--- + + +## 6. Roosevelt deltas (forward-look) + +| Aspect | Testcloud (v1) | Roosevelt | +|---|---|---| +| Jumphost | `vopenstack-jesse` KVM VM | TBD — likely a bastion VM or operator workstation | +| Cloud target VMs | 5 KVM VMs in libvirt on this jumphost | bare-metal MAAS-managed servers at the Roosevelt site | +| MAAS profile name | Single profile on the local MAAS | May be a Roosevelt-specific MAAS instance | +| openrc files | Manually-maintained `$HOME/admin-openrc`, `$HOME/user1-openrc` | Vault-issued or app-credential-based with rotation | +| KVM snapshots | Optional safety net | N/A — equivalent is MAAS re-deploy | +| Vault unseal keys | Generated fresh each Caracal cycle | Managed via Vault's own backup mechanism | + +--- + + +## 7. Change log + +| Date | Change | Reference | +|---|---|---| +| 2026-05-27 | Document created. Replaces stale runbook 00 (D-013 graceful teardown + capi-mgmt preservation, both superseded). | Batch A drafting | diff --git a/runbooks/v1-do-doc-02-pki.md b/runbooks/v1-do-doc-02-pki.md new file mode 100644 index 0000000..15f17c1 --- /dev/null +++ b/runbooks/v1-do-doc-02-pki.md @@ -0,0 +1,687 @@ +# v1 Do-Document 02 — Octavia LBaaS PKI Overlay Generation + +**Status:** Second execution document of Batch A. Generates local-only crypto material (no cloud touched). Runs after `v1-do-doc-01-prep.md` (state check). Runs before `runbooks/01-destroy-model.md` re-verification (Batch B doc 03 will point to that runbook). + +**Replaces:** `runbooks/deprecated/01a-octavia-pki-generation.md` (which had a buggy VIP-count grep pattern, an obsolete §12 bundle-housekeeping block, a self-reference off-by-one, and `exit 1` blocks in operator-facing context that violate the no-exit-in-pasted-shell rule). + +**Cross-references:** + +- D-007 (Magnum / Octavia inclusion) — Octavia bundle integration +- Bundle `octavia.options` PKI material section +- `overlays/octavia-pki.yaml` (gitignored — output of this document) +- Workstream 3a decision (2026-05-22): generate fresh, EC P-384 CAs, overlay-file approach + +--- + + +## 1. Purpose & scope + +Generate a complete two-tier PKI for Charmed Octavia's amphora load-balancer trust domain. The output is a single overlay file (`overlays/octavia-pki.yaml`) that the v1-do-doc-04 deploy step will pass to `juju deploy --overlay`. + +Octavia uses two CAs: + +- **Issuing CA** — signs each amphora's server certificate at LB-creation time. Octavia receives the private key and passphrase (so it can sign at runtime). +- **Controller CA** — trust anchor for connections **from** the Octavia controller to the amphorae. Octavia receives only the cert (no key needed at runtime; signing of controller certs is a humans-only rotation event). + +Plus one controller certificate (cert + key bundled) signed by the Controller CA. + +Five charm options on the `octavia` application consume the artifacts: + +| Charm option | Content | Format | +|---|---|---| +| `lb-mgmt-issuing-cacert` | Issuing CA certificate | base64-encoded PEM | +| `lb-mgmt-issuing-ca-private-key` | Issuing CA encrypted private key | base64-encoded PEM (already encrypted with passphrase) | +| `lb-mgmt-issuing-ca-key-passphrase` | Issuing CA key passphrase | plain string (NOT base64) | +| `lb-mgmt-controller-cacert` | Controller CA certificate | base64-encoded PEM | +| `lb-mgmt-controller-cert` | Controller cert + key, concatenated | base64-encoded PEM bundle | + +**Scope:** v1 testcloud (VR0 DC0 Omega Cloud). Roosevelt deltas in §14. + +**Out of scope:** + +- Octavia API TLS (issued by Vault via `octavia:certificates` relation in the bundle; separate concern) +- Rotation procedure (deferred to Roosevelt runbook; testcloud rotation pointer in §15) + +--- + + +## 2. Decisions captured + +Per workstream 3a sign-off (2026-05-22): + +| Decision | Choice | Roosevelt parallel | +|---|---|---| +| Cert provenance | Generate fresh (no Bobcat-backup copy) | Vault PKI engine | +| CA key algorithm | EC P-384 | EC P-384 (Vault root) | +| Controller cert algorithm | EC P-256 | EC P-256 | +| CA validity | 10 years | 5-year intermediate, Vault-rotated | +| Controller cert validity | 2 years | 90 days, auto-rotated | +| Distribution method | Juju overlay file (gitignored) | Vault-injected at deploy | +| Storage path on jumphost | `$HOME/octavia-pki/` | Vault PKI mounts | +| Passphrase strength | 32 random bytes, base64-encoded (44 chars) | Vault-generated | + +**Naming convention:** + +- Issuing CA CN: `VR0 DC0 Omega Cloud Octavia Issuing CA` +- Controller CA CN: `VR0 DC0 Omega Cloud Octavia Controller CA` +- Controller cert CN: `octavia-controller.omega.dc0.vr0.cloud.neumatrix.local` +- Controller cert SANs: above CN, plus `octavia.omega.dc0.vr0.cloud.neumatrix.local`, plus `10.12.4.233` (the Octavia API VIP) +- Organization (O): `Neumatrix` + +--- + + +## 3. Prerequisites + +| Prereq | Verification | +|---|---| +| `v1-do-doc-01-prep.md` completed cleanly | Manual confirmation; all §5 acceptance items checked | +| Executor on jumphost `vopenstack-jesse` as `jessea123` | `hostname && id -un` | +| `openssl` version 3.x or later | `openssl version` | +| `$HOME` writable | `test -w "$HOME" && echo OK` | +| Repository cloned at `$HOME/openstack-caracal-ipv4` | Verified in v1-do-doc-01 §4.2 | +| Repository on `main`, clean, up to date | Verified in v1-do-doc-01 §4.2 | +| Pre-deploy fixes commits all landed | Verified in v1-do-doc-01 §4.3 | + +**Shell context — paste once at start:** + +```bash +export REPO="$HOME/openstack-caracal-ipv4" +echo "REPO=$REPO" +test -d "$REPO/.git" && echo "[OK] repo present" || echo "[FAIL] repo missing" +cd "$REPO" +``` + +**Verify pre-deploy fixes are present (smoketest against the corrected grep pattern):** + +```bash +echo "VIP grep (corrected pattern, expect 12):" +grep -cE "^[[:space:]]+vip: 10\.12\.4\." "$REPO/bundle.yaml" +``` + +If this returns anything other than `12`, the bundle pre-deploy fix did not land — stop and reconcile before proceeding. + +> **Note on the grep pattern:** the deprecated runbook 01a used `^ vip: 10.12.4.` with a single literal space, which matches zero lines because the bundle's `vip:` entries live inside `options:` blocks indented six spaces deep. The corrected pattern `^[[:space:]]+vip: 10\.12\.4\.` matches any leading whitespace and escapes the dot literals. + +--- + + +## 4. Pre-flight: gitignore patch (DO THIS FIRST) + +**Critical:** the `.gitignore` patch must be in `main` BEFORE any private key material exists on disk in the workspace. This minimizes the race window for an accidental commit. + +The current `.gitignore` already catches `*.key`, `*.crt`, `*.pem` via wildcards, but does NOT catch `overlays/octavia-pki.yaml` (a `.yaml` file) or `passphrase.txt` (a `.txt` file). This step adds the missing patterns. + +```bash +cd "$REPO" + +# Idempotent patch — only add the block if the overlay path is not already protected +if ! grep -q "^overlays/octavia-pki.yaml" .gitignore; then + cat >> .gitignore <<'EOF' + +# Octavia PKI artifacts — never commit +overlays/octavia-pki.yaml +octavia-pki/ +passphrase.txt +EOF + echo "[OK] .gitignore patched" +else + echo "[OK] .gitignore already has overlay protection (no change)" +fi + +# Review the diff (will be empty if already patched) +git diff .gitignore +``` + +If the diff shows the new block, commit and push it before generating any keys: + +```bash +git add .gitignore +git commit -m "gitignore: octavia PKI artifacts and overlay (v1-do-doc-02 §4)" +git push origin main +``` + +**Verify the gitignore is effective** (this is a safety smoketest — touch a fake overlay file and ensure git ignores it): + +```bash +touch overlays/octavia-pki.yaml +STATUS=$(git status --short overlays/octavia-pki.yaml) +rm overlays/octavia-pki.yaml + +if [ -z "$STATUS" ]; then + echo "[OK] gitignore working — overlay file does not show as untracked" +else + echo "[FAIL] gitignore not effective — git sees:" + echo " $STATUS" + echo "Stop here. Fix .gitignore syntax before generating any secrets." +fi +``` + +If `[FAIL]`, do not proceed. The error means a generated PKI overlay could be accidentally committed. + +--- + + +## 5. Workspace setup + +```bash +WORKDIR="$HOME/octavia-pki" +mkdir -p "$WORKDIR"/{issuing-ca,controller-ca,controller,overlay-build} +chmod 700 "$WORKDIR" +cd "$WORKDIR" +echo "Working in: $WORKDIR" +ls -la "$WORKDIR" +``` + +Resulting layout: + +``` +$HOME/octavia-pki/ +├── issuing-ca/ # passphrase.txt, .key.enc, .cert.pem +├── controller-ca/ # passphrase.txt, .key.enc, .cert.pem +├── controller/ # .key, .csr, .cert.pem, .bundle.pem, .cnf +└── overlay-build/ # base64 intermediates → consumed by §10 +``` + +--- + + +## 6. Generate Issuing CA + +EC P-384 key encrypted with random 32-byte passphrase. Self-signed cert, 10-year validity. + +```bash +cd "$WORKDIR/issuing-ca" + +# Generate passphrase (no trailing newline — required for clean YAML embedding) +openssl rand -base64 32 | tr -d '\n' > passphrase.txt +chmod 600 passphrase.txt + +# Sanity-check length (no exit; operator-decided) +PASS_LEN=$(wc -c < passphrase.txt) +if [ "$PASS_LEN" -ne 44 ]; then + echo "[FAIL] passphrase length is $PASS_LEN bytes, expected 44 — investigate before continuing" +else + echo "[OK] passphrase length: $PASS_LEN bytes" +fi +``` + +> **On the length check:** `openssl rand -base64 32` produces 32 random bytes encoded as base64. Base64 encoding of 32 bytes is 44 characters (including two `=` padding chars). `tr -d '\n'` strips the trailing newline that openssl adds. The resulting file is exactly 44 bytes — verified by `wc -c`. + +If the length is wrong, stop and re-run the `openssl rand` line before proceeding. Do NOT continue with a wrong-length passphrase — it will silently break the overlay parsing later. + +```bash +# Generate EC P-384 private key, encrypted with passphrase +openssl genpkey -algorithm EC \ + -pkeyopt ec_paramgen_curve:P-384 \ + -aes-256-cbc \ + -pass file:passphrase.txt \ + -out issuing-ca.key.enc +chmod 600 issuing-ca.key.enc + +# Self-sign cert (10 years, SHA-384) +openssl req -new -x509 -sha384 \ + -key issuing-ca.key.enc \ + -passin file:passphrase.txt \ + -days 3650 \ + -subj "/CN=VR0 DC0 Omega Cloud Octavia Issuing CA/O=Neumatrix" \ + -out issuing-ca.cert.pem + +# Verify (no exit; visible output for operator) +echo "=== Issuing CA verification ===" +openssl x509 -in issuing-ca.cert.pem -noout -dates -subject +openssl verify -CAfile issuing-ca.cert.pem issuing-ca.cert.pem +# Expect: issuing-ca.cert.pem: OK + +ls -la +``` + +--- + + +## 7. Generate Controller CA + +Identical pattern; different CN. + +```bash +cd "$WORKDIR/controller-ca" + +openssl rand -base64 32 | tr -d '\n' > passphrase.txt +chmod 600 passphrase.txt + +PASS_LEN=$(wc -c < passphrase.txt) +if [ "$PASS_LEN" -ne 44 ]; then + echo "[FAIL] passphrase length is $PASS_LEN bytes, expected 44 — investigate before continuing" +else + echo "[OK] passphrase length: $PASS_LEN bytes" +fi + +openssl genpkey -algorithm EC \ + -pkeyopt ec_paramgen_curve:P-384 \ + -aes-256-cbc \ + -pass file:passphrase.txt \ + -out controller-ca.key.enc +chmod 600 controller-ca.key.enc + +openssl req -new -x509 -sha384 \ + -key controller-ca.key.enc \ + -passin file:passphrase.txt \ + -days 3650 \ + -subj "/CN=VR0 DC0 Omega Cloud Octavia Controller CA/O=Neumatrix" \ + -out controller-ca.cert.pem + +echo "=== Controller CA verification ===" +openssl x509 -in controller-ca.cert.pem -noout -dates -subject +openssl verify -CAfile controller-ca.cert.pem controller-ca.cert.pem +# Expect: controller-ca.cert.pem: OK + +ls -la +``` + +**Why Controller CA's key is encrypted even though Octavia never uses it:** the Controller CA key is needed for future rotations of the controller cert. Encrypting it (with its own passphrase, separate from Issuing CA's) is defense in depth — if the jumphost is compromised, the key still requires the passphrase to be useful for forging controller certs. + +--- + + +## 8. Generate Controller certificate + +EC P-256 key (no encryption — Octavia must read it at startup), CSR with SAN extensions, signed by Controller CA, 2-year validity. + +```bash +cd "$WORKDIR/controller" + +# Generate unencrypted EC P-256 key +openssl genpkey -algorithm EC \ + -pkeyopt ec_paramgen_curve:P-256 \ + -out controller.key +chmod 600 controller.key + +# CSR config with SAN extensions +cat > controller.cnf <<'EOF' +[req] +distinguished_name = req_distinguished_name +req_extensions = v3_req +prompt = no + +[req_distinguished_name] +CN = octavia-controller.omega.dc0.vr0.cloud.neumatrix.local +O = Neumatrix + +[v3_req] +keyUsage = critical, digitalSignature, keyEncipherment +extendedKeyUsage = clientAuth, serverAuth +subjectAltName = @alt_names + +[alt_names] +DNS.1 = octavia-controller.omega.dc0.vr0.cloud.neumatrix.local +DNS.2 = octavia.omega.dc0.vr0.cloud.neumatrix.local +IP.1 = 10.12.4.233 +EOF + +# Generate CSR +openssl req -new -sha256 \ + -key controller.key \ + -config controller.cnf \ + -out controller.csr + +# Sign with Controller CA (2 years) +openssl x509 -req -sha256 \ + -in controller.csr \ + -CA "$WORKDIR/controller-ca/controller-ca.cert.pem" \ + -CAkey "$WORKDIR/controller-ca/controller-ca.key.enc" \ + -passin file:"$WORKDIR/controller-ca/passphrase.txt" \ + -CAcreateserial \ + -days 730 \ + -extfile controller.cnf \ + -extensions v3_req \ + -out controller.cert.pem + +# Bundle cert + key (the lb-mgmt-controller-cert option expects both in one PEM) +cat controller.cert.pem controller.key > controller.bundle.pem +chmod 600 controller.bundle.pem +``` + +**Verify the chain and SAN:** + +```bash +echo "=== Chain verification ===" +openssl verify -CAfile "$WORKDIR/controller-ca/controller-ca.cert.pem" controller.cert.pem +# Expect: controller.cert.pem: OK + +echo "" +echo "=== SAN extensions ===" +openssl x509 -in controller.cert.pem -noout -ext subjectAltName +# Expect: +# DNS:octavia-controller.omega.dc0.vr0.cloud.neumatrix.local, +# DNS:octavia.omega.dc0.vr0.cloud.neumatrix.local, +# IP Address:10.12.4.233 + +echo "" +echo "=== Validity ===" +openssl x509 -in controller.cert.pem -noout -dates +# Expect: notAfter ~2 years from today + +echo "" +echo "=== Bundle integrity (cert and key match) ===" +# Use $HOME paths (not /tmp) per snap-confinement convention used elsewhere in repo +openssl x509 -in controller.bundle.pem -noout -pubkey > "$WORKDIR/controller/.cert.pub" +openssl pkey -in controller.bundle.pem -pubout > "$WORKDIR/controller/.key.pub" +if diff -q "$WORKDIR/controller/.cert.pub" "$WORKDIR/controller/.key.pub" >/dev/null; then + echo "[OK] bundle cert/key match" +else + echo "[FAIL] bundle cert/key DO NOT match — investigate before continuing" +fi +rm -f "$WORKDIR/controller/.cert.pub" "$WORKDIR/controller/.key.pub" +``` + +--- + + +## 9. Final chain verification + +A standalone block to confirm the full chain is sound before consuming for Octavia. All three "verify" lines must show `: OK`. If any do not, stop and investigate before proceeding. + +```bash +cd "$WORKDIR" + +echo "=== Issuing CA ===" +openssl x509 -in issuing-ca/issuing-ca.cert.pem -noout -subject -dates +openssl verify -CAfile issuing-ca/issuing-ca.cert.pem issuing-ca/issuing-ca.cert.pem + +echo "" +echo "=== Controller CA ===" +openssl x509 -in controller-ca/controller-ca.cert.pem -noout -subject -dates +openssl verify -CAfile controller-ca/controller-ca.cert.pem controller-ca/controller-ca.cert.pem + +echo "" +echo "=== Controller cert ===" +openssl x509 -in controller/controller.cert.pem -noout -subject -dates +openssl verify -CAfile controller-ca/controller-ca.cert.pem controller/controller.cert.pem +``` + +Operator-visible check: the three `verify` lines must all end with `: OK`. + +--- + + +## 10. Base64-encode artifacts + +Each base64 file is a single line (no wrapping); each becomes one YAML value. + +```bash +cd "$WORKDIR/overlay-build" + +# Issuing CA cert (base64) +base64 -w0 "$WORKDIR/issuing-ca/issuing-ca.cert.pem" > issuing-cacert.b64 + +# Issuing CA private key (already encrypted PEM → base64) +base64 -w0 "$WORKDIR/issuing-ca/issuing-ca.key.enc" > issuing-ca-private-key.b64 + +# Controller CA cert +base64 -w0 "$WORKDIR/controller-ca/controller-ca.cert.pem" > controller-cacert.b64 + +# Controller cert + key bundle +base64 -w0 "$WORKDIR/controller/controller.bundle.pem" > controller-cert.b64 + +# Sanity-check sizes (expect 500-2000 chars each) +wc -c *.b64 +``` + +--- + + +## 11. Assemble the overlay file + +```bash +# Read each artifact into shell variables +ISSUING_CACERT=$(cat "$WORKDIR/overlay-build/issuing-cacert.b64") +ISSUING_CA_KEY=$(cat "$WORKDIR/overlay-build/issuing-ca-private-key.b64") +ISSUING_CA_PASS=$(cat "$WORKDIR/issuing-ca/passphrase.txt") +CONTROLLER_CACERT=$(cat "$WORKDIR/overlay-build/controller-cacert.b64") +CONTROLLER_CERT=$(cat "$WORKDIR/overlay-build/controller-cert.b64") + +# Assemble overlay (passphrase is YAML-quoted; cert blobs are not — they're +# guaranteed-safe base64 without special chars) +mkdir -p "$REPO/overlays" +cat > "$REPO/overlays/octavia-pki.yaml" < + +# Watch for amphora spawn (3-5 minutes typical) +watch -n5 'openstack loadbalancer show pki-smoketest' +# Wait for: provisioning_status=ACTIVE, operating_status=ONLINE + +# Octavia-worker log should show successful amphora handshake (signed by Issuing CA, +# trusted via Controller CA): +juju ssh octavia/0 -- sudo journalctl -u octavia-worker --since "10 minutes ago" \ + | grep -iE "(amphora|cert)" | tail -20 +# Expect: "amphora connection established" or similar +# Expect: no TLS handshake errors, no cert validation errors + +# Cleanup the smoketest LB +openstack loadbalancer delete pki-smoketest --cascade +``` + +If amphora handshake fails with cert errors, the most likely causes are: + +1. **SAN mismatch** — the controller's connection to amphora uses the cert's CN/SAN; verify the controller cert SAN (§8) covers all addresses Octavia uses to reach amphorae. +2. **Bundle/key mismatch** — `lb-mgmt-controller-cert` bundle should contain BOTH the cert and the matching private key; if they're for different keys, handshake fails. (Verified in §8 with the pubkey diff.) +3. **Encrypted Issuing CA key + wrong passphrase** — verify the passphrase string in the overlay (§11) matches what was used at generation (§6). + +--- + + +## 15. Roosevelt deltas (forward-look) + +When this procedure is adapted for Roosevelt bare-metal deploy: + +| Aspect | Testcloud (v1) | Roosevelt | +|---|---|---| +| Issuing CA root | Self-signed | Intermediate signed by Vault root CA | +| CA storage | Filesystem on jumphost | Vault PKI engine, encrypted at rest | +| Controller cert validity | 2 years | 90 days | +| Rotation | Manual (this document re-run) | Automated via Vault + cron + bundle redeploy | +| Backup | gpg tarball, off-host | Vault's own backup mechanism | +| Amphora image signing | Out of scope for v1 | Image signed by Vault PKI as well | +| Procedure file | `runbooks/v1-do-doc-02-pki.md` | New runbook in Roosevelt repo | + +The procedure structure (generate Issuing CA → Controller CA → Controller cert → encode → overlay → backup → deploy) remains identical. Roosevelt just sources the CA root from Vault instead of self-signing. + +--- + + +## 16. Rotation/renewal pointer + +For testcloud, the 2-year controller cert and 10-year CAs are intentionally "set and forget" — they will outlive the cloud at this scale. + +If rotation IS needed before testcloud teardown (e.g., a key leak event), the re-run procedure is: + +1. Generate new Controller cert signed by **existing** Controller CA (re-run §8-9 only). +2. Regenerate the overlay (§11) with the new Controller cert; leave all other values unchanged. +3. `juju config octavia lb-mgmt-controller-cert=` (single-option update; does not require full bundle redeploy). +4. Octavia services may need a restart: `juju ssh octavia/0 -- sudo systemctl restart octavia-api octavia-worker octavia-housekeeping`. +5. Existing amphorae will need to reconnect using the new cert; in-flight LBs may briefly drop. This is acceptable for a security-event rotation. + +For Roosevelt, this whole procedure is replaced by Vault automated rotation. + +--- + + +## 17. Acceptance criteria — go/no-go for next step + +Before proceeding to Batch B (`v1-do-doc-03-destroy.md`): + +- [ ] §4 .gitignore patch applied and effective (overlay file is ignored) +- [ ] §6 Issuing CA generated; cert verifies OK; passphrase is 44 bytes +- [ ] §7 Controller CA generated; cert verifies OK; passphrase is 44 bytes +- [ ] §8 Controller cert generated and signed; chain verifies OK; SAN extensions present; bundle cert/key match +- [ ] §9 Final chain verification: all three `verify` lines show `: OK` +- [ ] §10 Four base64 artifacts produced +- [ ] §11 Overlay file written to `$REPO/overlays/octavia-pki.yaml`; parses as YAML; 5 non-empty option values +- [ ] §11 `git status --short` does NOT show the overlay file +- [ ] §12 Encrypted backup created and unencrypted tar shredded; backup moved off-host +- [ ] §13 deferred until after the deploy step and §14 verification + +If all checked, the overlay is ready for `v1-do-doc-04-deploy.md` (Batch B). The overlay is consumed by the deploy command: + +``` +juju deploy ./bundle.yaml --overlay overlays/octavia-pki.yaml --trust +``` + +(Note: only one overlay reference. The deprecated `overlays/vr0-dc0-testcloud.yaml` placeholder is not used; the bundle has its testcloud values inline.) + +--- + + +## 18. Change log + +| Date | Change | Reference | +|---|---|---| +| 2026-05-27 | Document created from `runbooks/deprecated/01a-octavia-pki-generation.md` with the following fixes: $REPO path corrected to `$HOME/openstack-caracal-ipv4`; §3 VIP-count grep corrected to `^[[:space:]]+vip:` pattern; old §12 (bundle housekeeping — already done) removed; §13/§14 self-reference fixed; operator-facing `exit 1` blocks replaced with non-exiting `[FAIL]` reports; intermediate diff files moved out of `/tmp` and into `$WORKDIR`. | Batch A drafting | diff --git a/runbooks/v1-do-doc-03-destroy.md b/runbooks/v1-do-doc-03-destroy.md new file mode 100644 index 0000000..3f69d77 --- /dev/null +++ b/runbooks/v1-do-doc-03-destroy.md @@ -0,0 +1,219 @@ +# v1 Do-Document 03 — Teardown / Pre-Deploy Cloud State + +**Status:** First execution document of Batch B. Conditional execution — current cloud state determines whether to skip or run the existing teardown runbook. + +**Position in sequence:** Runs after `v1-do-doc-02-pki.md` (overlay generated). Runs before `v1-do-doc-04-deploy.md` (juju deploy). + +**Type:** Thin pointer to the still-authoritative `runbooks/01-destroy-model.md`. This doc adds the state-detection and routing logic; the destructive procedure itself lives in the existing runbook. + +**Cross-references:** + +- D-017 (CAPI bootstrap cluster lifecycle) — every cycle is a full rebuild +- D-018 (teardown strategy) — skip graceful, release MAAS directly +- `runbooks/01-destroy-model.md` — the authoritative procedure (Phases A→D) + +--- + + +## 1. Purpose & scope + +This document determines whether the existing cloud needs to be torn down before deploy, and routes to `runbooks/01-destroy-model.md` if so. + +There are exactly three possible states going into this step: + +| State | Description | Routing | +|---|---|---| +| **Clean** | Cloud already down. All 5 VMs MAAS-Ready, no `openstack` Juju model. | Skip to §3 verification → §4 acceptance | +| **Dirty** | Cloud up (any unit deployed, any Juju model present). | Execute runbook 01-destroy-model.md in full → return to §3 | +| **Partial** | Some intermediate state (model destroying, machines not all Ready, etc.). | Execute runbook 01 Phase D remediation; resolve before continuing | + +**For the current first Caracal cycle (post-2026-05-27):** state is Clean (verified in `v1-do-doc-01-prep.md` §4.5 + §4.6). This document is effectively a one-step verification. + +**For future rebuild cycles:** state will be Dirty after each Caracal deploy. This document then routes through the full runbook 01 procedure. + +**Out of scope:** + +- The procedure itself — owned by `runbooks/01-destroy-model.md` +- KVM snapshot capture (per D-017, full rebuild every cycle; pre-existing KVM snapshots remain on disk but are not refreshed) + +--- + + +## 2. State detection + +```bash +export REPO="$HOME/openstack-caracal-ipv4" +cd "$REPO" + +echo "=== Juju model state ===" +if juju models 2>/dev/null | grep -qE "^openstack(\*| )"; then + JUJU_STATE="present" + juju models | grep "^openstack" || true +else + JUJU_STATE="absent" + echo "[OK] no 'openstack' model on current controller" +fi +echo "JUJU_STATE=$JUJU_STATE" +echo "" + +echo "=== MAAS machine state ===" +export MAAS_PROFILE=$(maas list 2>/dev/null | awk 'NR==1 {print $1}') +if [ -z "$MAAS_PROFILE" ]; then + echo "[FAIL] no MAAS profile logged in. Run 'maas login ' first." +fi + +READY_COUNT=$(maas "$MAAS_PROFILE" machines read 2>/dev/null \ + | python3 -c " +import json, sys +machines = json.load(sys.stdin) +targets = ['openstack0', 'openstack1', 'openstack2', 'openstack3', 'capi-mgmt'] +ready = [m for m in machines if m.get('hostname') in targets and m.get('status_name') == 'Ready'] +print(len(ready)) +") + +echo "Cloud-target VMs in Ready state: $READY_COUNT / 5" +if [ "$READY_COUNT" -eq 5 ]; then + MAAS_STATE="all_ready" +elif [ "$READY_COUNT" -eq 0 ]; then + MAAS_STATE="all_deployed_or_other" +else + MAAS_STATE="partial" +fi +echo "MAAS_STATE=$MAAS_STATE" +echo "" + +echo "=== Routing decision ===" +case "$JUJU_STATE:$MAAS_STATE" in + "absent:all_ready") + echo "[CLEAN] Cloud is already torn down. Skip to §3 verification." + ;; + "present:all_ready") + echo "[ANOMALY] Juju has the openstack model but MAAS shows machines Ready." + echo " Likely: model is in 'destroying' state. Run runbook 01 Phase D remediation." + ;; + "absent:all_deployed_or_other") + echo "[ANOMALY] No Juju model, but machines are not Ready." + echo " Likely: someone else owns the machines, OR commissioning is mid-flight." + echo " Investigate before continuing." + ;; + "present:all_deployed_or_other") + echo "[DIRTY] Cloud is up. Execute runbook 01-destroy-model.md in full." + ;; + *":partial") + echo "[PARTIAL] Mixed MAAS state. Use runbook 01 Phase C (release loop) and Phase D verification." + ;; + *) + echo "[UNKNOWN] state=$JUJU_STATE:$MAAS_STATE — investigate before continuing" + ;; +esac +``` + +--- + + +## 3. Execute (or verify) the teardown procedure + +### 3.1 If state is CLEAN (current Caracal cycle) + +No execution needed. Skip to §4 acceptance. + +Optional sanity check (read-only): + +```bash +echo "=== Confirm no 'openstack' model ===" +juju models | grep "^openstack" && echo "[FAIL] model still present" || echo "[OK] no openstack model" + +echo "" +echo "=== Confirm all 5 VMs Ready ===" +maas "$MAAS_PROFILE" machines read 2>/dev/null \ + | python3 -c " +import json, sys +machines = json.load(sys.stdin) +targets = ['openstack0', 'openstack1', 'openstack2', 'openstack3', 'capi-mgmt'] +for m in machines: + h = m.get('hostname', '') + if h in targets: + print(f'{h}: {m.get(\"status_name\", \"?\")}, owner={m.get(\"owner\") or \"(none)\"}') +" + +echo "" +echo "=== Confirm OSD qcow2 files still exist (should be ~200 KiB each after wipe) ===" +ls -la /var/lib/libvirt/images/openstack[0-3]-1.qcow2 2>/dev/null \ + || echo "[NOTE] OSD qcow2 files not visible to current user; check from jumphost user if needed" +``` + +### 3.2 If state is DIRTY (future rebuild cycles) + +Execute `runbooks/01-destroy-model.md` in full, in order: + +1. **Phase A** — Pre-destroy capture (~30 sec). Captures `juju export-bundle`, `juju status`, `juju models`, `juju controllers` to `$HOME/backups/pre-caracal-destroy-/`. Updates `$HOME/.last-pre-caracal-destroy-backup` pointer. + +2. **Phase B** — Force-destroy the Juju model. Returns in ~1-2 min; reaping continues for ~5-10 min. Command: + + ``` + juju destroy-model openstack --force --no-wait --destroy-storage --no-prompt + ``` + +3. **Phase C** — Release MAAS machines (parallel with Phase B; ~5 min). Either Path 1 (MAAS UI) or Path 2 (CLI loop). The CLI loop is filtered by owner — only releases machines you own. + +4. **Phase D** — Verification (~1 min). Confirms model is gone and all 5 VMs are Ready. + +**Critical pre-flight before Phase B:** verify you are about to destroy the right model. The destruction is not undoable short of restoring KVM snapshots: + +```bash +juju models +juju status --model openstack 2>/dev/null | head -20 +``` + +Confirm the model name and the unit counts match what you expect to lose. + +### 3.3 If state is PARTIAL or ANOMALY + +Use runbook 01 Phase D's remediation block (lines 161-173 of `runbooks/01-destroy-model.md`): + +```bash +juju machines -m openstack --format=yaml 2>/dev/null + +# For each lingering machine ID: +juju remove-machine -m openstack --force + +# Then re-attempt model removal: +juju destroy-model openstack --force --no-wait --no-prompt +``` + +If the model is still listed after the above, escalate — controller-side state may be corrupted (rare). + +--- + + +## 4. Acceptance criteria — go/no-go for v1-do-doc-04-deploy + +Before proceeding to deploy: + +- [ ] `juju models` does NOT list `openstack` +- [ ] All 5 VMs (`openstack0`, `openstack1`, `openstack2`, `openstack3`, `capi-mgmt`) report MAAS status `Ready`, owner `(none)` +- [ ] If state was DIRTY: `$HOME/.last-pre-caracal-destroy-backup` exists and points to a populated backup directory +- [ ] If OSD wipe is needed (rebuild after first Caracal cycle): verify the OSD qcow2 wipe procedure was executed per the 2026-05-22 protocol + +If all checked, proceed to `v1-do-doc-04-deploy.md`. + +--- + + +## 5. Roosevelt deltas (forward-look) + +| Aspect | Testcloud (v1) | Roosevelt | +|---|---|---| +| Teardown target | 5 KVM VMs on jumphost | Bare-metal MAAS-managed servers | +| Phase C release | libvirt VM owned by current MAAS user | Bare-metal owned by current MAAS user | +| OSD qcow2 wipe | Yes (libvirt secondary disks survive MAAS release) | No (real disks; MAAS commissioning wipes them) | +| Backup directory | `$HOME/backups/pre-caracal-destroy-*/` | Same convention, on bastion | + +--- + + +## 6. Change log + +| Date | Change | Reference | +|---|---|---| +| 2026-05-27 | Document created. Thin pointer to `runbooks/01-destroy-model.md` with state-detection routing for clean/dirty/partial cases. | Batch B drafting | diff --git a/runbooks/v1-do-doc-04-deploy.md b/runbooks/v1-do-doc-04-deploy.md new file mode 100644 index 0000000..1b62e89 --- /dev/null +++ b/runbooks/v1-do-doc-04-deploy.md @@ -0,0 +1,513 @@ +# v1 Do-Document 04 — Caracal Bundle Deploy + +**Status:** Second execution document of Batch B. First cloud-mutating step in the v1 deploy sequence. Triggers MAAS provisioning of 4 hosts, LXD container creation, charm installation, and the initial relation cascade. + +**Position in sequence:** Runs after `v1-do-doc-03-destroy.md` (state confirmed Clean: 5 VMs Ready, no openstack model). Runs before `v1-do-doc-05-vault-init.md` (manual Vault init). + +**Cross-references:** + +- D-001 (Path 2A: Juju-bundle paradigm) +- D-002 (channel matrix) +- D-006 (Vault HA backend; etcd + easyrsa bootstrap) +- D-007 (Magnum Layer A only at this step; Layer B is Batch C) +- D-017 (CAPI bootstrap cluster lifecycle; not touched in this doc — that's Batch C) +- `bundle.yaml` (canonical deploy artifact) +- `overlays/octavia-pki.yaml` (gitignored; from v1-do-doc-02) + +--- + + +## 1. Purpose & scope + +Execute the Caracal-bundle deploy and watch the model settle to a known-incomplete state: every charm reaches `active/idle` EXCEPT those waiting on `vault:certificates`, which sit in `blocked: 'certs not present yet'` until Vault is initialized in v1-do-doc-05. + +**What this document does:** + +- `juju add-model openstack` +- `juju deploy ./bundle.yaml --overlay overlays/octavia-pki.yaml` +- Watch the model settle (~60-90 minutes typical for this testcloud size) +- Verify Octavia received the PKI material on disk (the explicit on-disk verification — §8) +- Verify expected pre-Vault state: Vault charm blocked awaiting init; cert-dependent charms blocked awaiting Vault; non-cert-dependent charms active/idle + +**What this document does NOT do:** + +- Initialize Vault (next doc, v1-do-doc-05) +- Run any post-Vault functional verification of Octavia LB (deferred to v1-do-doc-02 §14, executed after v1-do-doc-05 completes) +- Magnum Keystone domain setup (Batch C, v1-do-doc-06) +- CAPI bootstrap or Magnum driver graft (Batch C) +- Tenant resources (Batch D) + +**Out of scope:** + +- KVM snapshot capture (per D-017, snapshots are not refreshed each cycle; pre-existing snapshots remain as last-resort safety net) +- NetBox VIP IPAddress writes (pinned post-deploy for external NetBox-engineer review; D-010 relaxed for v1) + +--- + + +## 2. Decisions captured + +| Decision | Choice | Notes | +|---|---|---| +| Model name | `openstack` | Matches `runbooks/01-destroy-model.md` Phase B target | +| Deploy command | `juju deploy ./bundle.yaml --overlay overlays/octavia-pki.yaml` | One overlay; `vr0-dc0-testcloud.yaml` was a placeholder and is empty per pre-deploy review | +| `--trust` flag | Not used | Standard OpenStack charms on MAAS do not require bundle-level trust. If a specific charm needs it post-deploy (none expected for Caracal v1), apply targeted `juju trust ` then. | +| Settle wait | Manual watch via `juju status --watch 30s`; ~60-90 min typical | Charms cycle blocked → maintenance → active/idle | +| Expected pre-Vault end state | Vault blocked; cert-relation consumers blocked; everything else active/idle | See §7.3 for the explicit blocked-charm list | +| PKI on-disk verification | Files-on-disk + fingerprint compare after Octavia config-changed hook completes | §8 — the explicit operator-asked confirmation | + +--- + + +## 3. Prerequisites + +| Prereq | Verification | +|---|---| +| `v1-do-doc-01-prep.md` ✓ (state-check passed) | Manual confirmation | +| `v1-do-doc-02-pki.md` ✓ (overlay generated) | `test -f "$REPO/overlays/octavia-pki.yaml"` | +| `v1-do-doc-03-destroy.md` ✓ (no `openstack` model, 5 VMs Ready) | Run §2 state-detection block from doc-03 again — should still return CLEAN | +| Pre-deploy fixes all committed and pulled locally | Verified in v1-do-doc-01 §4.3 | + +**Shell context — paste once:** + +```bash +export REPO="$HOME/openstack-caracal-ipv4" +cd "$REPO" +echo "REPO=$REPO" +test -f overlays/octavia-pki.yaml && echo "[OK] PKI overlay present" || echo "[FAIL] missing overlay" +git status --short +# Expect: clean working tree +``` + +--- + + +## 4. Pre-flight checks (all must pass) + +These are READ-ONLY safety checks. Stop if any FAIL. + +```bash +cd "$REPO" + +echo "=== 4.1 bundle.yaml YAML parses ===" +python3 -c "import yaml; yaml.safe_load(open('bundle.yaml'))" \ + && echo "[OK] bundle.yaml parses" || echo "[FAIL] bundle.yaml YAML error" +echo "" + +echo "=== 4.2 octavia-pki overlay YAML parses ===" +python3 -c " +import yaml +d = yaml.safe_load(open('overlays/octavia-pki.yaml')) +o = d['applications']['octavia']['options'] +keys = sorted(o.keys()) +expected = ['lb-mgmt-controller-cacert','lb-mgmt-controller-cert','lb-mgmt-issuing-ca-key-passphrase','lb-mgmt-issuing-ca-private-key','lb-mgmt-issuing-cacert'] +print('Keys in overlay:', keys) +print('All 5 keys present:', keys == expected) +print('All values non-empty:', all(v for v in o.values())) +" +echo "" + +echo "=== 4.3 ceph-osd has no storage block (pre-deploy fix #1) ===" +grep -A 12 "^ ceph-osd:" bundle.yaml | grep "^ storage:" \ + && echo "[FAIL] storage block present — pre-deploy fix not applied" \ + || echo "[OK] no storage block under ceph-osd" +echo "" + +echo "=== 4.4 expected-osd-count is 4 (matches one OSD per host) ===" +grep -A 8 "^ ceph-mon:" bundle.yaml | grep "expected-osd-count: 4" \ + && echo "[OK] expected-osd-count: 4" \ + || echo "[FAIL] expected-osd-count is not 4" +echo "" + +echo "=== 4.5 11 VIPs declared (Designate deferred to v2 per D-019) ===" +VIP_COUNT=$(grep -cE "^[[:space:]]+vip: 10\.12\.4\." bundle.yaml) +echo "VIP count: $VIP_COUNT (expect 11)" +echo "" + +echo "=== 4.6 No model named 'openstack' exists ===" +juju models | grep "^openstack" \ + && echo "[FAIL] model 'openstack' already exists — re-run doc-03 destroy" \ + || echo "[OK] no openstack model" +echo "" + +echo "=== 4.7 All 5 cloud-target VMs MAAS-Ready ===" +export MAAS_PROFILE=$(maas list 2>/dev/null | awk 'NR==1 {print $1}') +maas "$MAAS_PROFILE" machines read 2>/dev/null \ + | python3 -c " +import json, sys +machines = json.load(sys.stdin) +targets = ['openstack0', 'openstack1', 'openstack2', 'openstack3', 'capi-mgmt'] +ready = sum(1 for m in machines if m.get('hostname') in targets and m.get('status_name') == 'Ready' and not m.get('owner')) +print(f'Ready + unowned: {ready} / 5') +print('[OK]' if ready == 5 else '[FAIL]') +" +echo "" + +echo "=== 4.8 Juju controller available ===" +juju controllers +juju show-controller 2>/dev/null | head -10 +echo "" + +echo "=== 4.9 Disk space on /var/lib/libvirt/images ===" +df -h /var/lib/libvirt/images 2>/dev/null +echo " Need ≥ 4 × 8 GiB for openstack0-3 root + LXD container space; ≥ 4 × 512 GiB OSD qcow2 already allocated" +``` + +If any check above does not show `[OK]` (or the expected value), stop and investigate before continuing. + +--- + + +## 5. Add the Juju model + +```bash +juju add-model openstack +juju model-config -m openstack | head -20 +``` + +Expected: model `openstack` created on the current controller. `juju models` should now show it. + +> **Optional model-config tweaks (only if you have a reason):** +> +> - `default-base: ubuntu@22.04/stable` — already in the bundle's top-level config; model-level override not needed. +> - `transmit-vendor-metrics=false` — privacy posture; testcloud doesn't need to phone home. Optional. +> +> For this v1 cycle, leave model-config at defaults. Tweaks are easier to debug when only one is changed at a time. + +--- + + +## 6. Deploy command + +```bash +cd "$REPO" + +# Confirm working dir and overlay +pwd +ls -la bundle.yaml overlays/octavia-pki.yaml +echo "" + +# Deploy — this returns in a few seconds; actual provisioning runs in background +juju deploy ./bundle.yaml --overlay overlays/octavia-pki.yaml -m openstack +``` + +Expected output: a long list of deploy actions ("Deploying ...", "Resolving ...", "Located bundle ..."). Then control returns to the prompt. + +If the deploy command itself errors (YAML syntax, charm-not-found, etc.), stop here. The bundle has not started provisioning yet — fix and rerun. + +--- + + +## 7. Settle watch + +Provisioning takes **60-90 minutes typical** for this testcloud size on this jumphost. The bundle requests MAAS-deploy of 4 hosts + creation of ~25 LXD containers + 30+ charm installs + relation establishment. + +### 7.1 Recommended watch command + +In a dedicated terminal (don't share with the destroy / deploy terminal — interaction can interrupt screen redraws): + +```bash +juju status --color --watch 30s -m openstack +``` + +Refreshes every 30 seconds. Ctrl+C to exit. + +### 7.2 Expected progression + +Rough timeline (your mileage may vary): + +| Elapsed | What to expect | +|---|---| +| 0-3 min | MAAS commissioning starts on openstack0-3 (boot, fingerprint, partition) | +| 3-10 min | Ubuntu install on openstack0-3 via MAAS preseed | +| 10-15 min | Hosts in Juju show `pending` → `started`. LXD service comes up on each | +| 15-30 min | LXD containers being created; subordinate charms (mysql-router, hacluster) attaching | +| 30-50 min | Charm config-changed hooks running; relations forming; databases bootstrapping | +| 50-90 min | Most charms reach `blocked` (waiting on Vault) or `active/idle` (no Vault dep) | +| 90+ min | Settle stabilizes at the pre-Vault end state | + +If progress visibly stalls for >15 minutes in the middle of the timeline, see §7.4. + +### 7.3 Pre-Vault expected end state + +When the model has settled (stops progressing for >5 minutes), this is what `juju status` should show: + +**`blocked` (waiting on Vault):** + +The following charms have `:certificates` relations to `vault:certificates` and CANNOT reach active/idle until Vault is initialized in v1-do-doc-05: + +- `vault` itself — status: `Vault needs to be initialized` (this is the trigger to run doc-05) +- `mysql-innodb-cluster` (needs vault cert for inter-instance TLS) +- `keystone` (Keystone API TLS) +- `glance` (Glance API TLS) +- `nova-cloud-controller`, `placement`, `neutron-api`, `neutron-api-plugin-ovn`, `ovn-central`, `ovn-chassis`, `ovn-chassis-octavia` +- `cinder`, `octavia`, `octavia-dashboard` +- `barbican`, `barbican-vault` +- `magnum`, `magnum-dashboard` +- `glance-simplestreams-sync`, `openstack-dashboard`, `ceph-radosgw` +- `octavia-diskimage-retrofit` (subordinate of glance-simplestreams-sync) +- All 11 `*-hacluster` subordinates indirectly (because their principal is blocked; Designate's hacluster removed per D-019) + +**`active/idle` (no Vault dependency):** + +- `rabbitmq-server` +- `etcd` (uses easyrsa for its OWN TLS, not Vault) +- `easyrsa` +- `nova-compute` (no `:certificates` relation directly; it gets ceph keys via ceph-mon) +- `ceph-mon`, `ceph-osd` (Ceph cluster bootstraps independently) + +**Note on the chicken-and-egg:** + +Per D-006 (Vault HA backend), etcd's TLS is bootstrapped by easyrsa via the `easyrsa:client ↔ etcd:certificates` relation. This is what lets etcd come up active/idle BEFORE Vault is initialized. Then Vault uses etcd as its HA backend. Watch that `easyrsa/0` and `etcd/{0,1,2}` reach active/idle within the first 30 minutes; if etcd stays blocked beyond that, easyrsa-related certs likely didn't flow. + +### 7.4 Stalls and remediation + +If progress stalls for >15 min in the middle of the timeline: + +```bash +# Find which units are blocked or in error state +juju status -m openstack | grep -E "(blocked|error|maintenance)" + +# For any unit in error state, get its log +juju show-status-log -m openstack +# E.g.: juju show-status-log keystone/0 + +# For deeper inspection +juju ssh -m openstack -- sudo tail -200 /var/log/juju/unit-.log +``` + +Common stalls: + +1. **MAAS commissioning failure on a host** → check MAAS UI; may need to re-commission the host +2. **LXD container creation failure** → check `lxc list` on the host; container quotas, image availability +3. **Charm hook error** → check unit log; often a transient cloud-init issue; `juju resolved ` may help +4. **Relation never forms** → both ends must declare correct endpoint names; cross-check bundle + +Per D-018, do not pursue graceful recovery from major errors at this stage — full teardown via v1-do-doc-03 and redeploy is the canonical "reset" path. + +--- + + +## 8. Post-deploy PKI verification — explicit on-disk confirmation + +**This section addresses the operator-asked confirmation that the PKI overlay made it onto the Octavia unit's filesystem after the charm's config-changed hook completes.** + +Run this section **after** `octavia/0` has progressed past `pending`/`maintenance` and reached at least a `blocked` state (i.e., the charm has run its install + config-changed hooks but is waiting on Vault for the API TLS cert). The `lb-mgmt-*` options are consumed by the config-changed hook regardless of Vault status — so on-disk material should be present even with octavia/0 in `blocked`. + +### 8.1 Confirm the unit is past config-changed + +```bash +echo "=== Octavia unit status ===" +juju status octavia -m openstack +# Expect: octavia/0 in 'blocked' (cert relation pending) or 'maintenance' (still configuring). +# If status is still 'pending' or 'allocating', wait and re-run this section. +``` + +### 8.2 Inspect on-disk PKI directory + +```bash +echo "=== /etc/octavia/certs/ contents ===" +juju ssh octavia/0 -m openstack -- sudo ls -la /etc/octavia/certs/ +``` + +Expected: 4-5 PEM files. The exact filenames depend on the charm revision; commonly: + +- `server_ca.cert.pem` — Issuing CA cert (consumed from `lb-mgmt-issuing-cacert`) +- `server_ca.key.pem` — Issuing CA encrypted private key (consumed from `lb-mgmt-issuing-ca-private-key`) +- `client_ca.cert.pem` — Controller CA cert (consumed from `lb-mgmt-controller-cacert`) +- `client.cert-and-key.pem` — Controller cert + key bundle (consumed from `lb-mgmt-controller-cert`) + +If the directory is empty or missing, the config-changed hook hasn't run yet or failed. Re-check unit status; see §7.4 remediation. + +> **[unverified, flagging]:** the exact filenames above are typical for recent charm-octavia revisions but may vary. The verification below uses fingerprint comparison (content-based), which is filename-agnostic — adapt the filenames in the loop if `ls` shows different ones. + +### 8.3 Stage cert content from unit for comparison + +```bash +mkdir -p "$HOME/pki-verify" +chmod 700 "$HOME/pki-verify" +cd "$HOME/pki-verify" + +# Pull whatever PEM files are present in /etc/octavia/certs/ +juju ssh octavia/0 -m openstack -- sudo ls /etc/octavia/certs/ 2>/dev/null | \ + while read -r f; do + case "$f" in + *.pem|*.crt) + echo "Pulling $f ..." + juju ssh octavia/0 -m openstack -- sudo cat "/etc/octavia/certs/$f" \ + > "$HOME/pki-verify/unit-$f" + ;; + esac + done + +ls -la "$HOME/pki-verify/" +``` + +### 8.4 Fingerprint comparison — CA certs + +```bash +echo "=== Issuing CA fingerprint comparison ===" +# Find the issuing CA on the unit (charm naming: typically server_ca.cert.pem) +UNIT_FILE=$(ls "$HOME/pki-verify/" | grep -E "^unit-server.*ca.*\.cert\.pem$" | head -1) +if [ -z "$UNIT_FILE" ]; then + echo "[WARN] no unit-server*ca*.cert.pem file found — list and adapt:" + ls "$HOME/pki-verify/" +else + UNIT_FP=$(openssl x509 -in "$HOME/pki-verify/$UNIT_FILE" -noout -fingerprint -sha256 2>/dev/null | cut -d= -f2) + SRC_FP=$(openssl x509 -in "$HOME/octavia-pki/issuing-ca/issuing-ca.cert.pem" -noout -fingerprint -sha256 | cut -d= -f2) + echo "Unit ($UNIT_FILE): $UNIT_FP" + echo "Jumphost (issuing-ca): $SRC_FP" + if [ "$UNIT_FP" = "$SRC_FP" ]; then + echo "[OK] Issuing CA cert on unit matches jumphost source" + else + echo "[FAIL] fingerprints DIFFER — investigate before continuing" + fi +fi +echo "" + +echo "=== Controller CA fingerprint comparison ===" +UNIT_FILE=$(ls "$HOME/pki-verify/" | grep -E "^unit-client.*ca.*\.cert\.pem$" | head -1) +if [ -z "$UNIT_FILE" ]; then + echo "[WARN] no unit-client*ca*.cert.pem file found — list and adapt:" + ls "$HOME/pki-verify/" +else + UNIT_FP=$(openssl x509 -in "$HOME/pki-verify/$UNIT_FILE" -noout -fingerprint -sha256 2>/dev/null | cut -d= -f2) + SRC_FP=$(openssl x509 -in "$HOME/octavia-pki/controller-ca/controller-ca.cert.pem" -noout -fingerprint -sha256 | cut -d= -f2) + echo "Unit ($UNIT_FILE): $UNIT_FP" + echo "Jumphost (controller-ca): $SRC_FP" + if [ "$UNIT_FP" = "$SRC_FP" ]; then + echo "[OK] Controller CA cert on unit matches jumphost source" + else + echo "[FAIL] fingerprints DIFFER — investigate before continuing" + fi +fi +``` + +### 8.5 Controller cert bundle verification + +The `lb-mgmt-controller-cert` value contains BOTH the controller cert AND its key, concatenated. On the unit it lands as a single PEM bundle. Confirm: + +1. The cert in the bundle matches the cert we generated. +2. The key in the bundle matches the cert (private key proves possession). + +```bash +echo "=== Controller cert+key bundle verification ===" +UNIT_FILE=$(ls "$HOME/pki-verify/" | grep -E "^unit-client.*\.pem$" | grep -v "ca" | head -1) +# Common name: client.cert-and-key.pem +if [ -z "$UNIT_FILE" ]; then + echo "[WARN] no controller cert bundle found on unit — list and adapt:" + ls "$HOME/pki-verify/" +else + echo "Found bundle: $UNIT_FILE" + + # Compare cert fingerprint + UNIT_FP=$(openssl x509 -in "$HOME/pki-verify/$UNIT_FILE" -noout -fingerprint -sha256 2>/dev/null | cut -d= -f2) + SRC_FP=$(openssl x509 -in "$HOME/octavia-pki/controller/controller.cert.pem" -noout -fingerprint -sha256 | cut -d= -f2) + echo "Unit cert FP: $UNIT_FP" + echo "Source cert FP: $SRC_FP" + [ "$UNIT_FP" = "$SRC_FP" ] && echo "[OK] cert match" || echo "[FAIL] cert mismatch" + + # Confirm cert+key in bundle match each other (proof of possession) + CERT_PUB=$(openssl x509 -in "$HOME/pki-verify/$UNIT_FILE" -noout -pubkey 2>/dev/null | openssl md5) + KEY_PUB=$(openssl pkey -in "$HOME/pki-verify/$UNIT_FILE" -pubout 2>/dev/null | openssl md5) + echo "Cert pubkey md5: $CERT_PUB" + echo "Key pubkey md5: $KEY_PUB" + [ "$CERT_PUB" = "$KEY_PUB" ] && echo "[OK] cert and key in bundle are paired" || echo "[FAIL] cert and key DO NOT match" +fi +``` + +### 8.6 Issuing CA encrypted key + passphrase verification + +The Issuing CA's encrypted key sits on the unit. Confirm the passphrase from `octavia.conf` can decrypt it. This is the test that proves the runtime amphora-signing path will work once Octavia comes up post-Vault. + +```bash +echo "=== Passphrase round-trip test ===" + +# Pull the passphrase from octavia.conf +UNIT_PASS=$(juju ssh octavia/0 -m openstack -- \ + sudo grep "^ca_private_key_passphrase" /etc/octavia/octavia.conf 2>/dev/null | head -1 | cut -d= -f2- | sed 's/^[[:space:]]*//' | sed 's/[[:space:]]*$//') + +# Pull the encrypted key +UNIT_KEY=$(ls "$HOME/pki-verify/" | grep -E "^unit-server.*key.*\.pem$" | head -1) + +if [ -z "$UNIT_PASS" ]; then + echo "[WARN] passphrase line not found in octavia.conf — may not be present until vault-init" +elif [ -z "$UNIT_KEY" ]; then + echo "[WARN] no unit-server*key*.pem found" +else + # Try to decrypt the key using the passphrase from the unit + if openssl pkey -in "$HOME/pki-verify/$UNIT_KEY" -passin "pass:$UNIT_PASS" -noout 2>/dev/null; then + echo "[OK] passphrase in octavia.conf decrypts the on-disk Issuing CA key" + else + echo "[FAIL] passphrase did NOT decrypt the key — overlay value mismatch" + fi + + # Also confirm passphrase matches what we generated on jumphost + SRC_PASS=$(cat "$HOME/octavia-pki/issuing-ca/passphrase.txt") + if [ "$UNIT_PASS" = "$SRC_PASS" ]; then + echo "[OK] passphrase on unit matches jumphost source" + else + echo "[FAIL] passphrase on unit does NOT match jumphost source" + fi +fi + +# Clear the passphrase from shell +unset UNIT_PASS SRC_PASS +``` + +> **Note:** the `ca_private_key_passphrase` line in `octavia.conf` may not appear until the cert relation completes. If §8.6 reports `[WARN] passphrase line not found`, that is **expected at pre-Vault state** — the charm may defer writing the full `[certificates]` section until the cert relation has flowed. Re-run §8.6 after v1-do-doc-05 completes. + +### 8.7 Cleanup + +```bash +# Optional: shred the temp copies of the cert material +shred -uvz "$HOME/pki-verify/"*.pem 2>/dev/null +rmdir "$HOME/pki-verify" 2>/dev/null +``` + +The unit retains the originals; the jumphost-side originals are at `$HOME/octavia-pki/` (left in place per v1-do-doc-02 §13). + +--- + + +## 9. Acceptance criteria — go/no-go for v1-do-doc-05-vault-init + +Before proceeding to Vault init: + +- [ ] §4 all pre-flight checks `[OK]` +- [ ] §5 model `openstack` created +- [ ] §6 deploy command completed without bundle-level errors +- [ ] §7 model has settled (no progress for >5 min); pre-Vault end state matches §7.3 +- [ ] §7.3 critical infra `active/idle`: `rabbitmq-server`, `etcd/{0,1,2}`, `easyrsa`, `ceph-mon/{0,1,2}`, `ceph-osd/{0,1,2,3}`, `nova-compute/{0,1,2,3}` +- [ ] §7.3 Vault is `blocked: Vault needs to be initialized` (the trigger for doc-05) +- [ ] §8.2 PKI files present in `/etc/octavia/certs/` on octavia/0 +- [ ] §8.4 Issuing CA fingerprint match `[OK]` +- [ ] §8.4 Controller CA fingerprint match `[OK]` +- [ ] §8.5 Controller cert bundle cert+key paired `[OK]` +- [ ] §8.6 passphrase round-trip `[OK]` (or `[WARN] not yet in conf` — acceptable if pre-Vault) + +If all checked, proceed to `v1-do-doc-05-vault-init.md`. + +--- + + +## 10. Roosevelt deltas (forward-look) + +| Aspect | Testcloud (v1) | Roosevelt | +|---|---|---| +| Bundle | `bundle.yaml` (single file) | Multi-environment overlay structure | +| Hosts | 4 KVM VMs | Bare-metal MAAS-managed servers | +| LXD container layout | Dense (10+ on machine 8) | More spread; possibly real units instead of LXD for some apps | +| Overlay set | `overlays/octavia-pki.yaml` only | Site overlay (machine assignments, NIC MACs) + Vault overlay + PKI overlay | +| Settle time | 60-90 minutes | Likely 2-4 hours (more hosts, real provisioning) | +| Octavia PKI source | Operator-generated, overlay-distributed | Vault PKI engine | +| Octavia PKI verification | This §8 procedure | Vault-side audit trail; no manual comparison needed | + +--- + + +## 11. Change log + +| Date | Change | Reference | +|---|---|---| +| 2026-05-27 | Document created. Replaces `runbooks/deprecated/02-deploy.md` (placeholder). Adds explicit §8 on-disk PKI verification per operator request. | Batch B drafting | diff --git a/runbooks/v1-do-doc-05-vault-init.md b/runbooks/v1-do-doc-05-vault-init.md new file mode 100644 index 0000000..e2e7d7a --- /dev/null +++ b/runbooks/v1-do-doc-05-vault-init.md @@ -0,0 +1,596 @@ +# v1 Do-Document 05 — Vault Initialization & Cert-Relation Cascade + +**Status:** Third execution document of Batch B. Manual three-step Vault bring-up plus regeneration of admin-openrc. Last document in Batch B. + +**Position in sequence:** Runs after `v1-do-doc-04-deploy.md` (model settled at pre-Vault end state). Runs before `v1-do-doc-06-magnum-domain.md` (Batch C). + +**Cross-references:** + +- D-006 (Vault HA backend — etcd + easyrsa) +- D-009 (Hacluster modeling at testcloud scale) +- D-011 §6 (validation: Vault unseal + auto-unseal-after-reboot pattern) +- `bundle.yaml` Vault block (channel 1.8/stable, vip 10.12.4.236, vault-mysql-router subordinate) +- OpenStack charm-deployment-guide Appendix C — Vault (initialise / unseal / authorise) + +--- + + +## 1. Purpose & scope + +Initialize Vault, unseal it, authorize the charm, then watch the `vault:certificates` relation cascade flow certs to every API charm. The cascade unblocks roughly 20 charms that were `blocked` after doc-04. + +**The three manual Vault steps:** + +1. **Initialise** — generate the master encryption key + unseal keys + root token. ONE-SHOT per Vault lifetime. The unseal keys are the disaster-recovery material. +2. **Unseal** — provide 3-of-5 unseal keys to decrypt Vault's master key. Required after every Vault restart (including post-bundle-deploy and post-host-reboot). +3. **Authorize** — give the Vault charm a Vault token so it can manage policies, app roles, secrets storage for OpenStack consumers. + +**What this document does:** + +- Discovers Vault's address (HTTP at this point — pre-TLS) +- Runs `vault operator init` and captures the 5 unseal keys + 1 root token +- Runs `vault operator unseal` 3 times (with 3 different keys) +- Runs `juju run vault/leader authorize-charm token=...` to graft the charm +- Watches the cert-relation cascade settle +- Regenerates `$HOME/admin-openrc` against the new Keystone +- Smokes the post-Vault state: every charm `active/idle` + +**What this document does NOT do:** + +- Set up auto-unseal (Vault's transit-engine-based auto-unseal pattern) — out of scope for v1; manual unseal after host reboot is acceptable. Roosevelt may revisit. +- Set up Vault PKI engine for tenant-side use — out of scope for v1. +- Provision tenant resources or DNS zones (Batch D) +- Magnum domain or CAPI work (Batch C) + +**Out of scope security note:** the unseal keys captured in §3 are the disaster-recovery material. Per the Caracal_Rebuild handoff, the prior cycle's keys are accepted lost. The keys generated HERE need a secure off-host home — operator decision (admin workstation encrypted vault, password manager attachment, dedicated secrets store). For Roosevelt this becomes a real key-management procedure. + +--- + + +## 2. Decisions captured + +| Decision | Choice | Notes | +|---|---|---| +| Key shares / threshold | 5 keys, threshold 3 | Standard Shamir's-secret-sharing posture; allows quorum-of-3 unseal | +| Vault address scheme | HTTP via unit IP for init/unseal/authorize; HTTPS via VIP thereafter | Vault has no TLS until authorize-charm flips it on | +| Authorize-charm pattern | Direct `token=` parameter (channel 1.8/stable convention) | Newer revisions may require `token-secret-id=` via Juju secret; verify with `juju show-action vault authorize-charm` first | +| Admin-openrc location | `$HOME/admin-openrc` | Same path as prior cloud; overwritten | +| Admin domain/project | Charmed-Keystone defaults: user=admin, user-domain=admin_domain, project=admin_domain | `[unverified, flagging]` for the project — older charm versions used `admin` for project; verify by `openstack token issue` | +| Unseal key storage | Operator decision — secure off-host | This document warns; doesn't dictate the where | + +--- + + +## 3. Prerequisites + +| Prereq | Verification | +|---|---| +| `v1-do-doc-04-deploy.md` ✓ (model settled, pre-Vault end state confirmed) | Manual; re-check via §4.1 below | +| Octavia PKI on-disk verification `[OK]` (doc-04 §8) | Manual | +| `vault` CLI installed on jumphost | `command -v vault && vault --version` (any 1.7+ works for client) | +| Juju controller still reachable | `juju controllers` | + +**Shell context — paste once:** + +```bash +export REPO="$HOME/openstack-caracal-ipv4" +cd "$REPO" +echo "REPO=$REPO" +``` + +**Install vault CLI if missing** (using the Hashicorp APT repo; one-time per jumphost): + +```bash +if ! command -v vault >/dev/null 2>&1; then + echo "vault CLI not present. Install via Hashicorp APT repo:" + echo " wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg" + echo " echo \"deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com \$(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/hashicorp.list" + echo " sudo apt update && sudo apt install vault" + echo " (After install, you may need 'sudo setcap cap_ipc_lock= /usr/bin/vault' on hosts without IPC_LOCK capability.)" +else + vault --version +fi +``` + +--- + + +## 4. Pre-flight: confirm pre-Vault state + +```bash +echo "=== 4.1 Vault status (expect: blocked, 'Vault needs to be initialized') ===" +juju status vault -m openstack + +echo "" +echo "=== 4.2 Other charms expected blocked on vault (sample) ===" +juju status -m openstack keystone glance neutron-api octavia magnum 2>/dev/null | grep -E "(blocked|active|maintenance)" + +echo "" +echo "=== 4.3 Critical infra expected active/idle ===" +juju status -m openstack rabbitmq-server etcd easyrsa ceph-mon ceph-osd nova-compute 2>/dev/null | grep -E "(active|blocked|error)" +``` + +If `vault/0` is not in `blocked: Vault needs to be initialized`, stop. Either Vault hasn't reached config-changed yet (re-run §4 after a few minutes), or it's in a different blocked state that needs investigation. + +If `etcd` is NOT active/idle (e.g., still maintenance or blocked), Vault cannot use it as a backend. Investigate easyrsa→etcd cert flow before continuing. + +--- + + +## 5. Discover Vault's address & set environment + +Vault has no TLS yet. We connect via HTTP on the unit's port 8200. + +```bash +# Get the unit's IP (NOT the VIP — VIP is hacluster-managed and only active after Vault is up) +VAULT_UNIT_IP=$(juju show-unit vault/0 -m openstack 2>/dev/null | grep "public-address:" | head -1 | awk '{print $2}') + +if [ -z "$VAULT_UNIT_IP" ]; then + echo "[FAIL] could not resolve vault/0 public-address. Check 'juju show-unit vault/0'" +else + echo "vault/0 public-address: $VAULT_UNIT_IP" +fi + +# Set VAULT_ADDR for the vault CLI +export VAULT_ADDR="http://${VAULT_UNIT_IP}:8200" +echo "VAULT_ADDR=$VAULT_ADDR" + +# Confirm reachable +vault status 2>&1 | head -20 +# Expected output: +# Sealed: true +# Initialized: false +# ... (or similar — exit code 2 is expected when uninitialized) +``` + +> **Note on VAULT_ADDR scheme:** HTTP at this stage. After authorize-charm, Vault enables HTTPS using its own internal CA cert. From that point onward, `VAULT_ADDR=https://vault.omega.dc0.vr0.cloud.neumatrix.local:8200` (or `https://10.12.4.236:8200`) is the right address, but `vault` CLI will need the Vault CA root cert via `VAULT_CACERT` or `-tls-skip-verify`. For this document, we only use the HTTP address — once authorize-charm completes, the charm handles all subsequent Vault interactions internally. + +--- + + +## 6. Initialise Vault (one-shot per Vault lifetime) + +**WARNING:** the output of this command contains the unseal keys and the root token. If lost, Vault is unrecoverable — there is no "forgot password" path. If exposed, an attacker with the unseal keys can decrypt everything Vault holds. + +Capture the output to a file in `$HOME` (filesystem-encrypted assumed; if not, work on a tmpfs): + +```bash +mkdir -p "$HOME/vault-init" +chmod 700 "$HOME/vault-init" + +# Init with 5 key shares, threshold 3 +vault operator init -key-shares=5 -key-threshold=3 \ + > "$HOME/vault-init/init-output-$(date +%Y%m%d-%H%M%S).txt" + +# Permissions: tighten immediately +chmod 600 "$HOME/vault-init/"*.txt + +# Display the output +INIT_FILE=$(ls -t "$HOME/vault-init/"*.txt | head -1) +echo "Init output captured to: $INIT_FILE" +cat "$INIT_FILE" +``` + +Expected output format: + +``` +Unseal Key 1: <44-char base64> +Unseal Key 2: <44-char base64> +Unseal Key 3: <44-char base64> +Unseal Key 4: <44-char base64> +Unseal Key 5: <44-char base64> + +Initial Root Token: hvs. + +Vault initialized with 5 key shares and a key threshold of 3. ... +``` + +**Immediately:** + +1. Copy this output (the entire file) to your secure off-host store (admin workstation encrypted drive, password manager, secrets vault). +2. Verify you have it stored AND retrievable before proceeding to §7. + +The unseal keys are needed every time Vault restarts (including the deploy unit reboot). The root token is needed for `authorize-charm` in §8 and (potentially) for future Vault admin operations. + +**Re-running init is destructive.** If something goes wrong here and you decide to wipe Vault, the procedure is: `juju run vault/leader reissue-certificates` (does NOT re-init); or worst case, destroy + redeploy Vault (which discards encrypted state — anything stored in Vault is lost). + +--- + + +## 7. Unseal Vault (3 of 5) + +Provide three different unseal keys. Vault decrypts its master key progressively; after the third key, `Sealed: false`. + +```bash +# Extract keys to shell variables (do NOT print them all together) +INIT_FILE=$(ls -t "$HOME/vault-init/"*.txt | head -1) + +# Unseal step 1 — paste Key 1 when prompted (interactive; -prompts safer than passing on CLI) +vault operator unseal + +# Unseal step 2 — paste Key 2 +vault operator unseal + +# Unseal step 3 — paste Key 3 +vault operator unseal +``` + +After the third unseal, output should show: + +``` +Key Value +--- ----- +Seal Type shamir +Initialized true +Sealed false <-- this is the win condition +Total Shares 5 +Threshold 3 +Version 1.8.x +Cluster Name vault-cluster-XXXX +... +``` + +Verify: + +```bash +vault status +# Expect: Sealed: false, Initialized: true, HA Enabled: true (since etcd backend) +``` + +--- + + +## 8. Authorize the charm + +Vault is now unsealed. The charm needs a token to create its own policies and app roles for managing OpenStack-consumer secrets and certs. + +### 8.1 Verify the action signature + +The `authorize-charm` action signature has shifted across vault charm revisions. Check first: + +```bash +juju show-action authorize-charm --application vault -m openstack 2>/dev/null || \ + juju actions vault -m openstack | grep authorize +``` + +Look for one of these patterns: + +- **Direct-token (older revisions, expected on `1.8/stable`):** parameter is `token=` +- **Juju-secret (newer revisions):** parameter is `token-secret-id=`; the token must be in a Juju secret first + +For channel `1.8/stable` (what the bundle pins), the direct-token pattern is expected. If `juju show-action` indicates the secret-based pattern instead, use §8.3. + +### 8.2 Direct-token authorize (expected path) + +```bash +# Extract the root token +INIT_FILE=$(ls -t "$HOME/vault-init/"*.txt | head -1) +ROOT_TOKEN=$(grep "^Initial Root Token:" "$INIT_FILE" | awk '{print $NF}') + +if [ -z "$ROOT_TOKEN" ]; then + echo "[FAIL] could not extract root token from $INIT_FILE" +else + echo "Root token captured (length: ${#ROOT_TOKEN})" +fi + +# Run the action +juju run vault/leader authorize-charm token="$ROOT_TOKEN" -m openstack + +# Clear from shell +unset ROOT_TOKEN +``` + +### 8.3 Juju-secret authorize (fallback if §8.1 shows the secret-based signature) + +```bash +INIT_FILE=$(ls -t "$HOME/vault-init/"*.txt | head -1) +ROOT_TOKEN=$(grep "^Initial Root Token:" "$INIT_FILE" | awk '{print $NF}') + +# Create a Juju secret containing the token +SECRET_ID=$(juju add-secret vault-root-token token="$ROOT_TOKEN" -m openstack | grep -oE "secret:[a-z0-9]+") +echo "Secret created: $SECRET_ID" +unset ROOT_TOKEN + +# Grant the secret to the vault application +juju grant-secret "$SECRET_ID" vault -m openstack + +# Run the action with the secret-id parameter (parameter name may vary; check §8.1) +juju run vault/leader authorize-charm token-secret-id="$SECRET_ID" -m openstack + +# After authorize completes successfully, the secret can be removed +juju remove-secret "$SECRET_ID" -m openstack +``` + +### 8.4 Verify authorize succeeded + +```bash +echo "=== Vault status after authorize-charm ===" +juju status vault -m openstack +# Expect: vault/0 transitions out of 'blocked' to maintenance, then active/idle within 1-2 min +``` + +If vault/0 stays blocked after authorize-charm, check the unit log: + +```bash +juju ssh vault/0 -m openstack -- sudo tail -100 /var/log/juju/unit-vault-0.log +``` + +Common failures: invalid token format; token already revoked; charm trying to write to a path the token can't access. + +--- + + +## 9. Watch the cert-relation cascade + +After Vault is active/idle, the `vault:certificates` relation flows certs to ~20 charms. They progress from `blocked` → `maintenance` (writing certs, restarting services) → `active/idle`. + +Expected duration: **15-30 minutes** for the full cascade to settle. + +```bash +juju status --color --watch 30s -m openstack +``` + +### 9.1 Expected progression + +| Tier | Charms unblocked | Approximate time after authorize-charm | +|---|---|---| +| Tier 1 (direct certs) | `mysql-innodb-cluster`, `ovn-central`, `keystone`, `glance`, `neutron-api`, `cinder` | 2-8 min | +| Tier 2 (waited on Tier 1) | `nova-cloud-controller`, `placement`, `octavia`, `barbican`, `designate`, `magnum`, `openstack-dashboard` | 8-15 min | +| Tier 3 (subordinates + plugins) | `*-mysql-router`, `*-hacluster`, `neutron-api-plugin-ovn`, `ovn-chassis`, `ovn-chassis-octavia`, `barbican-vault`, `octavia-dashboard`, `magnum-dashboard` | 15-25 min | +| Tier 4 (downstream) | `glance-simplestreams-sync`, `octavia-diskimage-retrofit`, `designate-bind`, `ceph-radosgw` | 20-30 min | + +### 9.2 Final post-Vault end state + +When settled, every unit should be `active/idle`. Verify: + +```bash +echo "=== Any unit not in active/idle? ===" +juju status -m openstack --format=yaml \ + | python3 -c " +import yaml, sys +d = yaml.safe_load(sys.stdin) +apps = d.get('applications', {}) +issues = [] + +def check_unit(uname, udata): + ws = udata.get('workload-status', {}).get('current', '') + js = udata.get('juju-status', {}).get('current', '') + msg = udata.get('workload-status', {}).get('message', '') + if ws != 'active' or js != 'idle': + issues.append(f'{uname}: workload={ws}, juju={js}, msg={msg}') + +for app, info in apps.items(): + units = info.get('units', {}) or {} + for uname, udata in units.items(): + check_unit(uname, udata) + # Walk subordinates too (hacluster, mysql-router, etc.) + subs = udata.get('subordinates', {}) or {} + for sname, sdata in subs.items(): + check_unit(sname, sdata) + +print(f'Non-active/idle units: {len(issues)}') +for i in issues: + print(f' {i}') +" +``` + +Expected output: `Non-active/idle units: 0`. Anything else needs investigation before the openrc-regeneration step. + +--- + + +## 10. Regenerate admin-openrc + +Once Keystone is active/idle and Vault has issued its TLS cert, the new admin-openrc points at the new Caracal cloud. + +### 10.1 Pull the admin password from the keystone charm + +```bash +juju run keystone/leader get-admin-password -m openstack +# Output is YAML — operator extracts the 'admin-password' value manually OR via jq below +``` + +For scripted extraction: + +```bash +ADMIN_PASS=$(juju run keystone/leader get-admin-password -m openstack --format json 2>/dev/null \ + | python3 -c " +import json, sys +d = json.load(sys.stdin) +# Action result format varies; try common shapes +for k, v in d.items(): + if isinstance(v, dict): + r = v.get('results', {}) + for key in ('admin-password', 'Stdout', 'password'): + if key in r: + print(r[key].strip()) + exit(0) +print('', end='') +") + +if [ -z "$ADMIN_PASS" ]; then + echo "[FAIL] could not extract admin password from action output. Run 'juju run keystone/leader get-admin-password' manually." +else + echo "[OK] admin password captured (length: ${#ADMIN_PASS})" +fi +``` + +### 10.2 Pull the Vault CA root for openrc trust + +Keystone's TLS cert is signed by Vault's internal CA. To validate that cert from the openstack CLI, we need the Vault CA root. + +```bash +juju run vault/leader get-root-ca -m openstack > "$HOME/vault-init/vault-ca-root.pem" +# Strip any YAML wrapping if present (the action returns the cert inline in YAML) +# Inspect: +head -5 "$HOME/vault-init/vault-ca-root.pem" +``` + +If the output is wrapped (e.g., starts with `Running operation ...` or `unit-vault-0:`), extract just the PEM block. Common pattern: + +```bash +# If the action output wraps the cert, extract just the BEGIN/END CERTIFICATE block +python3 -c " +import re +with open('$HOME/vault-init/vault-ca-root.pem') as f: + content = f.read() +m = re.search(r'-----BEGIN CERTIFICATE-----.*?-----END CERTIFICATE-----', content, re.DOTALL) +if m: + print(m.group(0)) +else: + print('NO_CERT_FOUND') +" > "$HOME/vault-init/vault-ca-root-clean.pem" + +mv "$HOME/vault-init/vault-ca-root-clean.pem" "$HOME/vault-init/vault-ca-root.pem" +openssl x509 -in "$HOME/vault-init/vault-ca-root.pem" -noout -subject -dates +# Expect: a valid cert with the Vault-charm-generated subject +``` + +### 10.3 Write the new admin-openrc + +```bash +# Move any existing admin-openrc out of the way (the prior cycle's pointed at the destroyed cloud) +if [ -f "$HOME/admin-openrc" ]; then + mv "$HOME/admin-openrc" "$HOME/admin-openrc.pre-caracal-$(date +%Y%m%d-%H%M%S)" +fi + +cat > "$HOME/admin-openrc" < **[unverified, flagging] OS_PROJECT_NAME default:** Charmed-Keystone's admin user lives in `admin_domain`. The default admin project name has varied across charm revisions — common values are `admin_domain` (matching the domain) or `admin`. If the first `openstack token issue` (§10.4) fails with a project-not-found error, try `OS_PROJECT_NAME=admin` instead. + +### 10.4 Verify + +```bash +( source "$HOME/admin-openrc"; \ + echo "Testing auth against $OS_AUTH_URL ..."; \ + openstack token issue 2>&1 | head -20 ) +``` + +Expected: a token dump (id, expires, project_id, user_id). If you get `ProjectNotFoundException`, see the flagging note above and try `OS_PROJECT_NAME=admin` in the openrc. + +If you get a TLS error (`certificate verify failed`), the OS_CACERT path is wrong or the cert extraction in §10.2 didn't produce a clean cert. + +--- + + +## 11. /etc/hosts sanity (jumphost-side) + +The openrc uses FQDN `keystone.omega.dc0.vr0.cloud.neumatrix.local`. That hostname must resolve from the jumphost to the Keystone VIP (`10.12.4.229`) for openrc to work pre-Designate. + +```bash +echo "=== Jumphost /etc/hosts has the API VIPs ===" +grep -E "10\.12\.4\.(22[4-9]|23[0-6])" /etc/hosts || echo "[WARN] no API VIP hosts found in /etc/hosts" +``` + +If absent, add a block: + +```bash +sudo tee -a /etc/hosts > /dev/null </dev/null | tail -5 | awk '{print $1}' | head -1) 2>/dev/null || \ + echo "(Use 'juju run --wait' synchronous mode — action result was visible in §4.1 output)" +``` + +> **Note on Juju 3.x action output:** in Juju 3.x, `juju run` returns the action result synchronously to stdout. You don't generally need a follow-up query unless investigating a failure. If §4.1 output ended in `Running operation ... with 1 task` followed by `done`, the action succeeded. + +### 4.3 Verify Keystone state + +```bash +( source "$HOME/admin-openrc" + + echo "=== magnum domain exists and is enabled ===" + openstack domain show magnum -f json | python3 -c " +import json, sys +d = json.load(sys.stdin) +print(f' name: {d.get(\"name\")}') +print(f' enabled: {d.get(\"enabled\")}') +print(f' id: {d.get(\"id\")}') +print(f' desc: {d.get(\"description\")}') +" + + echo "" + echo "=== Users in magnum domain ===" + openstack user list --domain magnum + + echo "" + echo "=== Projects in magnum domain (likely empty — trust projects are per-cluster) ===" + openstack project list --domain magnum +) +``` + +**Expected:** + +- `magnum` domain present, `enabled: True`. +- At least one user in the magnum domain. Charm default name is `magnum_domain_admin` (verify exact name in output). +- Projects list may be empty at this stage — Magnum creates per-cluster trust projects on demand at cluster-create time. + +### 4.4 Verify charm state + +```bash +echo "=== Magnum unit status after domain-setup ===" +juju status magnum -m openstack +# Expect: magnum/0 still active/idle. The domain-setup action should not have moved it out of active. + +echo "" +echo "=== Action history for magnum (last 5) ===" +juju list-actions magnum -m openstack 2>/dev/null | tail -10 +``` + +--- + + +## 5. Acceptance criteria — go/no-go for v1-do-doc-07-capi-bootstrap + +- [ ] `juju run magnum/leader domain-setup --wait=10m` completed without error +- [ ] `openstack domain show magnum` returns `enabled: True` +- [ ] `openstack user list --domain magnum` returns at least one user (the trustee admin) +- [ ] `juju status magnum` still shows magnum/0 in active/idle + +If all checked, proceed to `v1-do-doc-07-capi-bootstrap.md`. + +--- + + +## 6. Re-run posture + +The `domain-setup` action is idempotent at the charm level — re-running it does not duplicate the domain or create extra users. The most common reason to re-run is if the charm was reconfigured or upgraded; the action re-ensures the domain state matches what the charm expects. + +If §4.3 shows the magnum domain already exists from a prior run, the action will report `completed` with no work performed — that's the expected behavior. + +--- + + +## 7. Roosevelt deltas (forward-look) + +| Aspect | Testcloud (v1) | Roosevelt | +|---|---|---| +| Domain name | `magnum` (charm default) | Same | +| Trustee user creation | Charm action `domain-setup` | Same — but Roosevelt may layer Vault PKI for trustee credentials | +| Action invocation | Manual `juju run` | Can be wrapped in a deploy script with idempotency check | +| Verification depth | Domain + user check (this doc §4.3) | Plus Vault audit trail confirming the trustee was created with correct permissions | + +--- + + +## 8. Change log + +| Date | Change | Reference | +|---|---|---| +| 2026-05-27 | Document created. Replaces placeholder runbook 04 (TODO-only). | Batch C drafting | diff --git a/runbooks/v1-do-doc-07-capi-bootstrap.md b/runbooks/v1-do-doc-07-capi-bootstrap.md new file mode 100644 index 0000000..3c3e893 --- /dev/null +++ b/runbooks/v1-do-doc-07-capi-bootstrap.md @@ -0,0 +1,1085 @@ +# v1 Do-Document 07 — CAPI Bootstrap Cluster + Workload Pivot + +**Status:** Second execution document of Batch C. Stands up the CAPI bootstrap cluster on `capi-mgmt.maas`, creates the workload cluster on the cloud, pivots cluster state into the workload via `clusterctl move`, and stages the workload kubeconfig for v1-do-doc-08. + +**Position in sequence:** Runs after `v1-do-doc-06-magnum-domain.md` (Magnum domain setup). Runs before `v1-do-doc-08-magnum-driver.md` (driver graft consumes the workload kubeconfig produced here). + +**Replaces:** `runbooks/04a-capi-bootstrap-cluster.md` — same substantive procedure with fixes applied. The old runbook moves to `runbooks/deprecated/` as part of this batch's commits. + +**Fixes applied vs the prior runbook (`runbooks/04a-capi-bootstrap-cluster.md`):** + +- `$REPO` corrected from `$HOME/repos/openstack-caracal-ipv4` to `$HOME/openstack-caracal-ipv4` +- `$VAULT_CA` corrected from `$HOME/vault-pki/root-ca.pem` to `$HOME/vault-init/vault-ca-root.pem` (matches v1-do-doc-05 §10.2 output) +- `$MAAS_PROFILE` now explicitly set in §3 shell context (prior version referenced it without setting it) +- §4 adds `KUBERNETES_VERSION` to dynamic pin discovery (was hardcoded `v1.31.4` in §13) +- §5 MAAS deploy poll's `exit 1` on Failed-deployment converted to non-exiting `[FAIL]` report +- §11 noble-amd64-missing branch's `exit 1` converted to non-exiting `[FAIL]` report +- Cross-references updated: "runbook 02" → v1-do-doc-04; "runbook 03" → v1-do-doc-05; "runbook 04" → v1-do-doc-06; "runbook 05" → v1-do-doc-08 + +**Cross-references:** + +- D-017 (CAPI bootstrap cluster lifecycle — full rebuild every cycle) +- D-007 (Magnum two-layer install — this is Layer B preparation) +- D-002 (channel matrix — informs Vault CA chain) +- Workstream 3b decision (2026-05-22): ship Vault CA (no tls-insecure); pivot mandatory + +--- + + +## 1. Purpose & scope + +Stand up the CAPI bootstrap cluster on `capi-mgmt.maas` and pivot cluster state into a self-managing workload cluster. Output: + +1. **Workload K8s cluster** (`capi-mgmt-cluster`) running in tenant VMs on the cloud, self-managing post-pivot. +2. **Workload kubeconfig** copied to jumphost at a known path. Consumed by `v1-do-doc-08-magnum-driver.md` for the Magnum CAPI Helm driver graft. +3. **No remaining state** on the bootstrap k3s VM after pivot. capi-mgmt becomes a disposable jump host. + +**D-017 posture:** L3 full teardown and rebuild every deployment cycle. Nothing is preserved across cycles. capi-mgmt is wiped to MAAS Ready on teardown; rebuilt from scratch by this runbook. + +**Scope:** v1 testcloud. Roosevelt deltas in §20. + +**Out of scope:** + +- Magnum-side configuration (v1-do-doc-08) +- Workload cluster's tenant lifecycle (Magnum's job, not this runbook's) +- Backup / DR for the workload cluster (Roosevelt concern) + +--- + + +## 2. Decisions captured + +Per workstream 3b sign-off (2026-05-22): + +| Decision | Choice | Roosevelt parallel | +|---|---|---| +| Version pinning | Pin-at-execution with discovery in §4 | Same pattern; pins captured in deploy record | +| Cloud TLS trust | Ship Vault CA to capi-mgmt + workload nodes (no `tls-insecure`) | Image-baked CA; CK8sConfig redundancy | +| `clusterctl move` pivot | Mandatory; workload cluster becomes self-managing | Same | +| K8s flavor | Canonical Kubernetes (CK8s) | Same | +| OpenStack auth | v3applicationcredential | Same | +| Pod CIDR | `10.244.0.0/16` | Same (no conflict with cloud `10.12.0.0/16` or tenant pool `10.20.0.0/16`) | +| Service CIDR | `10.96.0.0/12` | Same | +| Workload cluster name | `capi-mgmt-cluster` | Same | +| Workload node SSH user | `ubuntu` (MAAS/cloud-init convention) | Same | + +**Naming convention:** + +- Keystone project for CAPI: `capi-mgmt` (in `admin_domain`) +- Keystone user for CAPI: `capo` (CAPO operator) +- App credential: `capo-app-cred` +- Workload image (Glance): `noble-amd64` (do NOT duplicate as `ubuntu-24.04-capi` — Bobcat lesson) +- Workload flavor: `capi-mgmt-node` (4 vCPU / 4 GiB / 30 GB) — control plane node sizing + +--- + + +## 3. Prerequisites + +| Prereq | Verification | +|---|---| +| Cloud deployed; all charms `active/idle` per D-011 | `juju status --color | grep -v "active.*idle"` returns only the header | +| Vault initialized + unsealed (v1-do-doc-05) | `juju ssh vault/leader -- sudo vault status` shows `Sealed=false` | +| Vault root CA available on jumphost | `test -f $HOME/vault-init/vault-ca-root.pem && openssl x509 -in $HOME/vault-init/vault-ca-root.pem -noout -subject` | +| Keystone reachable via FQDN | `curl -sf --cacert $HOME/vault-init/vault-ca-root.pem https://keystone.omega.dc0.vr0.cloud.neumatrix.local:5000/v3 | jq .version.id` returns `"v3.14"` or current | +| Magnum domain set up (v1-do-doc-06) | `( source $HOME/admin-openrc; openstack domain show magnum -f value -c enabled )` returns `True` | +| capi-mgmt VM exists in MAAS as Ready | `maas $MAAS_PROFILE machines read | jq '.[] | select(.hostname=="capi-mgmt") | .status_name'` returns `"Ready"` | +| Admin openrc available | `test -f $HOME/admin-openrc && ( source $HOME/admin-openrc && openstack token issue | head -3 )` | +| Workspace path under $HOME (snap confinement) | `WORK=$HOME/capi-bootstrap; mkdir -p "$WORK"; cd "$WORK"; pwd` shows under home | + +**Set shell context for the runbook:** + +```bash +export REPO="$HOME/openstack-caracal-ipv4" +export WORK="$HOME/capi-bootstrap" +export VAULT_CA="$HOME/vault-init/vault-ca-root.pem" +export CAPI_MGMT_METAL_IP=10.12.8.21 +export CAPI_MGMT_PROVIDER_IP=10.12.4.21 +export CLUSTER_NAME=capi-mgmt-cluster +export MAAS_PROFILE=$(maas list 2>/dev/null | awk 'NR==1 {print $1}') + +mkdir -p "$WORK" +cd "$WORK" + +# Sanity-check setup +echo "REPO=$REPO" +echo "WORK=$WORK" +echo "VAULT_CA=$VAULT_CA" +echo "MAAS_PROFILE=$MAAS_PROFILE" +test -f "$VAULT_CA" && echo "[OK] Vault CA present" || echo "[FAIL] Vault CA missing" +test -n "$MAAS_PROFILE" && echo "[OK] MAAS_PROFILE set" || echo "[FAIL] MAAS_PROFILE empty — run 'maas login' first" +``` + +--- + + +## 4. Version discovery (set pins) + +Pin-at-execution with discovery procedure documented inline so each rebuild's pins are reproducible AND traceable. + +**GitHub API: authenticated vs unauthenticated.** Unauth has 60 req/hr; authenticated has 5000. For multiple rebuilds in a day, set a token: + +```bash +# Optional but recommended — avoids rate-limit headaches during rebuild +export GITHUB_TOKEN= +# Or skip if you can tolerate ~10 API calls slowly +``` + +**Discover current stable releases:** + +```bash +cd "$WORK" + +# Helper: fetch latest stable release tag from a GitHub repo +gh_latest() { + local repo=$1 + local auth="" + [ -n "$GITHUB_TOKEN" ] && auth="-H Authorization: Bearer $GITHUB_TOKEN" + curl -sfL $auth "https://api.github.com/repos/$repo/releases/latest" \ + | jq -r '.tag_name' +} + +# Pin captures (one file per pin) +mkdir -p pins +gh_latest "kubernetes-sigs/cluster-api" | tee pins/CAPI_VERSION +gh_latest "kubernetes-sigs/cluster-api-provider-openstack" | tee pins/CAPO_VERSION +gh_latest "canonical/cluster-api-k8s" | tee pins/CK8S_VERSION +gh_latest "cert-manager/cert-manager" | tee pins/CERT_MANAGER_VERSION +gh_latest "k-orc/openstack-resource-controller" | tee pins/ORC_VERSION +gh_latest "k3s-io/k3s" | tee pins/K3S_VERSION +gh_latest "helm/helm" | tee pins/HELM_VERSION + +# Load into shell +export CAPI_VERSION=$(cat pins/CAPI_VERSION) +export CAPO_VERSION=$(cat pins/CAPO_VERSION) +export CK8S_VERSION=$(cat pins/CK8S_VERSION) +export CERT_MANAGER_VERSION=$(cat pins/CERT_MANAGER_VERSION) +export ORC_VERSION=$(cat pins/ORC_VERSION) +export K3S_VERSION=$(cat pins/K3S_VERSION) +export HELM_VERSION=$(cat pins/HELM_VERSION) +``` + +**Discover Kubernetes version supported by the pinned CK8s release:** + +The CK8s release publishes a `metadata.yaml` alongside its components that names the Kubernetes versions it supports. Discover the latest supported patch: + +```bash +gh_supported_k8s() { + local ck8s_ver=$1 + # CK8s release metadata.yaml is typically published as a release asset + curl -sfL "https://github.com/canonical/cluster-api-k8s/releases/download/${ck8s_ver}/metadata.yaml" 2>/dev/null \ + | grep -oE "v1\.[0-9]+\.[0-9]+" | sort -uV | tail -1 +} + +KUBERNETES_VERSION=$(gh_supported_k8s "$CK8S_VERSION") + +if [ -z "$KUBERNETES_VERSION" ]; then + echo "[WARN] could not auto-discover k8s version for CK8s $CK8S_VERSION via metadata.yaml" + echo " Consult release notes at: https://github.com/canonical/cluster-api-k8s/releases/tag/$CK8S_VERSION" + echo " Then set manually: export KUBERNETES_VERSION=v1.X.Y" + echo " (Re-run rest of §4 after setting.)" +else + echo "[OK] Discovered KUBERNETES_VERSION=$KUBERNETES_VERSION for CK8s=$CK8S_VERSION" + echo "$KUBERNETES_VERSION" > pins/KUBERNETES_VERSION + export KUBERNETES_VERSION +fi + +# Display for the deploy log +echo "" +echo "=== Pinned versions ===" +for f in pins/*_VERSION; do + printf "%-30s %s\n" "$(basename "$f")" "$(cat "$f")" +done +``` + +**Sanity check:** all values should look like `v1.X.Y` or `v0.X.Y`. If any returned `null` or empty, the GitHub API call failed — most likely rate-limited. Wait an hour or set `$GITHUB_TOKEN` and retry. + +**Capture pins to deploy record:** + +```bash +DEPLOY_RECORD=$HOME/deploy-records/$(date +%Y%m%d-%H%M%S)/capi-pins +mkdir -p "$DEPLOY_RECORD" +cp pins/*_VERSION "$DEPLOY_RECORD/" +ls -la "$DEPLOY_RECORD/" +``` + +--- + + +## 5. MAAS-deploy capi-mgmt + +Prerequisite: capi-mgmt MAAS machine is in `Ready` state (see §3). Network config in MAAS: + +- **eth0** on metal fabric, DHCP → `10.12.8.21` (MAAS-pinned static lease) +- **eth1** on provider fabric, static → `10.12.4.21` + +Deploy Ubuntu 24.04 (Noble): + +```bash +# Get the capi-mgmt system_id from MAAS +CAPI_MGMT_SYSTEM_ID=$(maas $MAAS_PROFILE machines read \ + | jq -r '.[] | select(.hostname=="capi-mgmt") | .system_id') +echo "capi-mgmt system_id: $CAPI_MGMT_SYSTEM_ID" + +# Deploy +maas $MAAS_PROFILE machine deploy "$CAPI_MGMT_SYSTEM_ID" \ + distro_series=noble \ + hwe_kernel=ga-24.04 +``` + +**Poll for `Deployed`:** + +```bash +DEPLOY_OK=1 +for i in $(seq 1 60); do + STATUS=$(maas $MAAS_PROFILE machine read "$CAPI_MGMT_SYSTEM_ID" | jq -r '.status_name') + echo "$(date -Is) capi-mgmt status: $STATUS" + if [ "$STATUS" = "Deployed" ]; then + echo "[OK] capi-mgmt Deployed" + DEPLOY_OK=0 + break + fi + if [ "$STATUS" = "Failed deployment" ]; then + echo "[FAIL] capi-mgmt deployment failed — STOP here, investigate via MAAS UI before continuing" + DEPLOY_OK=2 + break + fi + sleep 30 +done + +if [ "$DEPLOY_OK" -ne 0 ]; then + echo "[FAIL] poll exited without a clean Deployed state. STATUS=$STATUS. Stop and investigate." +fi +``` + +Typical deploy time: 5-8 minutes on this hardware. + +**SSH reachability:** + +```bash +# MAAS .maas zone may not resolve from jumphost — use IP directly per handoff lessons +ssh -o StrictHostKeyChecking=accept-new ubuntu@$CAPI_MGMT_METAL_IP -- hostname +# Expect: capi-mgmt +``` + +> **Gotcha:** MAAS-deployed Ubuntu uses the `ubuntu` user, not `jessea123`. See handoff "recurring technical pitfalls." + +--- + + +## 6. SSH bootstrap + Vault CA install + +On the jumphost, prepare a transport bundle of essentials: + +```bash +mkdir -p "$WORK/bootstrap-bundle" +cp "$VAULT_CA" "$WORK/bootstrap-bundle/vault-ca.crt" +chmod 644 "$WORK/bootstrap-bundle/vault-ca.crt" + +# Bundle pin files so capi-mgmt can read versions +cp -r "$WORK/pins" "$WORK/bootstrap-bundle/" +``` + +SCP and install Vault CA on capi-mgmt: + +```bash +scp -r "$WORK/bootstrap-bundle" ubuntu@$CAPI_MGMT_METAL_IP:/home/ubuntu/ + +ssh ubuntu@$CAPI_MGMT_METAL_IP <<'EOF' +set -euo pipefail + +# Install Vault CA as a system-trusted root +sudo cp /home/ubuntu/bootstrap-bundle/vault-ca.crt /usr/local/share/ca-certificates/ +sudo update-ca-certificates 2>&1 | tail -3 + +# Verify +openssl s_client -connect keystone.omega.dc0.vr0.cloud.neumatrix.local:5000 \ + -CApath /etc/ssl/certs -verify_return_error &1 \ + | grep -E "(Verify return code|subject=)" || \ + { echo "TLS chain verify failed against Keystone — investigate before proceeding"; exit 1; } + +# Update apt + base utilities +sudo apt-get update -qq +sudo apt-get install -y -qq jq curl yq + +# Confirm +which jq curl yq +EOF +``` + +**Expected:** + +- `update-ca-certificates` reports "1 added" +- `openssl s_client` shows `Verify return code: 0 (ok)` and a Keystone cert whose chain terminates at the Vault CA + +> **Why this matters:** Bobcat used `tls-insecure=true` in cloud.conf which skipped this entire trust path. Our workstream 3b decision (ship Vault CA) means OCCM and CAPO will validate certs against this trust store. If TLS verify fails here, OCCM will crashloop later. + +> **`exit 1` inside ssh heredoc:** the heredoc body runs on the remote host inside its own bash session. `exit 1` there exits the REMOTE session, propagating a non-zero exit back to the local ssh — it does NOT kill the operator's shell. + +--- + + +## 7. k3s install + +On capi-mgmt: + +```bash +ssh ubuntu@$CAPI_MGMT_METAL_IP "K3S_VERSION=$K3S_VERSION CAPI_MGMT_METAL_IP=$CAPI_MGMT_METAL_IP bash -s" <<'REMOTE_EOF' +set -euo pipefail + +# Install k3s with explicit bind/advertise/SAN flags +curl -sfL https://get.k3s.io | \ + INSTALL_K3S_VERSION="$K3S_VERSION" \ + sh -s - server \ + --bind-address="$CAPI_MGMT_METAL_IP" \ + --advertise-address="$CAPI_MGMT_METAL_IP" \ + --node-ip="$CAPI_MGMT_METAL_IP" \ + --tls-san="$CAPI_MGMT_METAL_IP" \ + --tls-san=capi-mgmt.maas \ + --write-kubeconfig-mode=0644 \ + --disable=traefik + +# Wait for k3s API to respond +for i in $(seq 1 30); do + if sudo kubectl get nodes 2>/dev/null | grep -q "Ready"; then + echo "k3s ready"; break + fi + echo "Waiting for k3s API... ($i/30)" + sleep 5 +done + +sudo kubectl get nodes +sudo kubectl get pods -A +REMOTE_EOF +``` + +> **Gotcha:** `--bind-address=$IP` makes k3s listen ONLY on that IP — not also on 127.0.0.1. The default kubeconfig at `/etc/rancher/k3s/k3s.yaml` has `server: https://127.0.0.1:6443` and will NOT work as-is. Sed-rewrite below. + +--- + + +## 8. Kubeconfig server-URL rewrite + +```bash +ssh ubuntu@$CAPI_MGMT_METAL_IP "CAPI_MGMT_METAL_IP=$CAPI_MGMT_METAL_IP bash -s" <<'REMOTE_EOF' +set -euo pipefail + +# Copy k3s kubeconfig to ubuntu user; rewrite server URL +mkdir -p /home/ubuntu/.kube +sudo cp /etc/rancher/k3s/k3s.yaml /home/ubuntu/.kube/config +sudo chown ubuntu:ubuntu /home/ubuntu/.kube/config +chmod 600 /home/ubuntu/.kube/config + +# Rewrite 127.0.0.1 → metal IP +sed -i "s|server: https://127.0.0.1:6443|server: https://$CAPI_MGMT_METAL_IP:6443|" \ + /home/ubuntu/.kube/config + +# Verify rewrite +grep "server:" /home/ubuntu/.kube/config +# Expect: server: https://10.12.8.21:6443 + +# Confirm kubectl works as ubuntu user (no sudo) +kubectl get nodes +REMOTE_EOF +``` + +--- + + +## 9. helm + clusterctl install + +```bash +ssh ubuntu@$CAPI_MGMT_METAL_IP "HELM_VERSION=$HELM_VERSION CAPI_VERSION=$CAPI_VERSION bash -s" <<'REMOTE_EOF' +set -euo pipefail + +# helm install (get-helm-3 fetches the version we specify) +cd /tmp +curl -sfL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 \ + | DESIRED_VERSION="$HELM_VERSION" bash +helm version --short + +# clusterctl install +CLUSTERCTL_URL="https://github.com/kubernetes-sigs/cluster-api/releases/download/${CAPI_VERSION}/clusterctl-linux-amd64" +sudo curl -sfL "$CLUSTERCTL_URL" -o /usr/local/bin/clusterctl +sudo chmod +x /usr/local/bin/clusterctl +clusterctl version +REMOTE_EOF +``` + +--- + + +## 10. clusterctl init (CAPI controllers + cert-manager + ORC + CAPO + CK8s) + +```bash +ssh ubuntu@$CAPI_MGMT_METAL_IP "CK8S_VERSION=$CK8S_VERSION CERT_MANAGER_VERSION=$CERT_MANAGER_VERSION ORC_VERSION=$ORC_VERSION CAPO_VERSION=$CAPO_VERSION CAPI_VERSION=$CAPI_VERSION bash -s" <<'REMOTE_EOF' +set -euo pipefail + +# Configure clusterctl with provider URLs +mkdir -p ~/.cluster-api +cat > ~/.cluster-api/clusterctl.yaml </dev/null || true +kubectl wait --for=condition=Available --timeout=5m deployment --all -n capo-system +kubectl wait --for=condition=Available --timeout=5m deployment --all -n cert-manager + +# Install ORC +kubectl apply -f "https://github.com/k-orc/openstack-resource-controller/releases/${ORC_VERSION}/orc.yaml" +kubectl wait --for=condition=Available --timeout=5m deployment --all -n orc-system + +# Confirm all controllers +kubectl get pods -A | grep -v "Running\|Completed" | grep -v NAME +# Expected: empty output (all pods Running or no abnormal state) +REMOTE_EOF +``` + +> **Gotcha:** the actual namespace names (`capi-system`, `capo-system`, etc.) are conventions. If a controller fails to land in the expected namespace, `kubectl get deployment -A` lists all deployments — diagnose from there. + +--- + + +## 11. Cloud-side prep (Keystone, Nova, Glance) + +Back on the jumphost: + +```bash +source $HOME/admin-openrc + +# Inventory existing resources FIRST (Bobcat lesson: don't create duplicates) +echo "=== Existing images ===" +openstack image list -c ID -c Name -f json | jq -r '.[] | "\(.Name)\t\(.ID)"' +echo "" +echo "=== Existing flavors ===" +openstack flavor list -c Name -c ID -c RAM -c VCPUs -c Disk -f json \ + | jq -r '.[] | "\(.Name)\tRAM=\(.RAM)\tCPU=\(.VCPUs)\tDisk=\(.Disk)\tID=\(.ID)"' +echo "" +echo "=== Existing keypairs ===" +openstack keypair list +echo "" +echo "=== Existing projects in admin_domain ===" +openstack project list --domain admin_domain +``` + +**Create / verify resources:** + +```bash +# Keystone project + user +openstack project show capi-mgmt --domain admin_domain 2>/dev/null \ + || openstack project create capi-mgmt --domain admin_domain --description "CAPI management plane" + +openstack user show capo --domain admin_domain 2>/dev/null \ + || openstack user create capo --domain admin_domain --password-prompt --description "CAPO operator" + +# Role assignments (CAPO needs member + load-balancer_member at minimum; +# admin works for testcloud — Roosevelt should use least-privilege) +openstack role add --user capo --user-domain admin_domain \ + --project capi-mgmt --project-domain admin_domain \ + member + +openstack role add --user capo --user-domain admin_domain \ + --project capi-mgmt --project-domain admin_domain \ + load-balancer_member 2>/dev/null || \ + echo "(load-balancer_member role may not exist if Octavia not deployed yet)" + +# Application credential — captured to file under $HOME (snap confinement) +APP_CRED_FILE=$WORK/capo-app-cred.json +openstack --os-username capo --os-user-domain-name admin_domain \ + --os-project-name capi-mgmt --os-project-domain-name admin_domain \ + application credential create capo-app-cred \ + --description "CAPO operator app credential" \ + -f json > "$APP_CRED_FILE" +chmod 600 "$APP_CRED_FILE" + +# Extract credential ID + secret +export APP_CRED_ID=$(jq -r '.id' "$APP_CRED_FILE") +export APP_CRED_SECRET=$(jq -r '.secret' "$APP_CRED_FILE") +echo "App cred ID: $APP_CRED_ID" +``` + +**Nova keypair (workload node SSH key):** + +```bash +# Generate fresh keypair locally (do NOT reuse jumphost personal key) +ssh-keygen -t ed25519 -N '' -f "$WORK/capi-workload-key" \ + -C "capi-workload-$(date +%Y%m%d)" +chmod 600 "$WORK/capi-workload-key" + +# Upload public key to Keystone as a Nova keypair +openstack keypair create --public-key "$WORK/capi-workload-key.pub" capi-workload-key +openstack keypair show capi-workload-key +``` + +**Workload image:** + +```bash +# Inventory check — use noble-amd64 if it exists (Bobcat lesson: do NOT create ubuntu-24.04-capi as a dup) +NOBLE_IMAGE_ID=$(openstack image show noble-amd64 -c id -f value 2>/dev/null || echo "") + +if [ -z "$NOBLE_IMAGE_ID" ]; then + echo "[FAIL] noble-amd64 image not found in Glance. Upload required before proceeding:" + echo "" + echo " wget https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img -O $WORK/noble-server-cloudimg-amd64.img" + echo " openstack image create --disk-format qcow2 --container-format bare \\" + echo " --public --file $WORK/noble-server-cloudimg-amd64.img noble-amd64" + echo "" + echo " Then re-run this section." +else + echo "[OK] Using image: noble-amd64 ($NOBLE_IMAGE_ID)" + export WORKLOAD_IMAGE_ID=$NOBLE_IMAGE_ID +fi +``` + +If the image was missing, the rest of §11 cannot complete. Stop here, upload the image, and rerun §11 from the top. + +**Workload flavor:** + +```bash +openstack flavor show capi-mgmt-node 2>/dev/null \ + || openstack flavor create capi-mgmt-node \ + --vcpus 4 --ram 4096 --disk 30 \ + --description "CAPI workload node (control plane sizing)" + +export WORKLOAD_FLAVOR=capi-mgmt-node +``` + +--- + + +## 12. clouds.yaml + cloud.conf composition (with Vault CA, no tls-insecure) + +The workload cluster's OCCM (OpenStack Cloud Controller Manager) and CAPO both need to call OpenStack APIs. Two files: + +- `clouds.yaml` — CAPO's view of how to reach OpenStack (used at cluster creation time on capi-mgmt) +- `cloud.conf` — OCCM's view, injected into the workload cluster's k8s Secret (used continuously by OCCM running in the workload cluster) + +**Compose clouds.yaml:** + +```bash +cat > "$WORK/clouds.yaml" < "$WORK/clouds.yaml.b64" +``` + +**Compose cloud.conf** (INI format, NOT YAML): + +```bash +cat > "$WORK/cloud.conf" < "$WORK/cloud.conf.b64" +``` + +> **Critical delta from Bobcat:** the `ca-file` line replaces `tls-insecure=true`. The path `/usr/local/share/ca-certificates/vault-ca.crt` exists on capi-mgmt (from §6) AND will be injected into workload nodes via CK8sConfig in §13. + +**base64-encode Vault CA for CK8sConfig injection:** + +```bash +base64 -w0 "$VAULT_CA" > "$WORK/vault-ca.crt.b64" +wc -c "$WORK/vault-ca.crt.b64" +``` + +--- + + +## 13. Cluster template rendering (with Vault CA injection) + +The cluster template defines: Cluster, OpenStackCluster, CK8sControlPlane, CK8sConfigTemplate (control plane + workers), MachineDeployment, Secrets for clouds.yaml and cloud.conf. + +Variables (18 total): + +```bash +export CLUSTER_NAME=capi-mgmt-cluster +export CLUSTER_NAMESPACE=default +# KUBERNETES_VERSION was discovered in §4; verify it's set +test -n "$KUBERNETES_VERSION" || { echo "[FAIL] KUBERNETES_VERSION not set; rerun §4 discovery"; } +echo "Using KUBERNETES_VERSION=$KUBERNETES_VERSION" +export CONTROL_PLANE_MACHINE_COUNT=1 # 3 for HA on Roosevelt +export WORKER_MACHINE_COUNT=2 # 3 on Roosevelt +export OPENSTACK_DNS_NAMESERVERS=1.1.1.1,1.0.0.1 # public DNS, per D-019 (Designate deferred to v2) +export OPENSTACK_FAILURE_DOMAIN=nova +export OPENSTACK_EXTERNAL_NETWORK_ID=$(openstack network show ext_net -c id -f value) +export OPENSTACK_IMAGE_NAME=noble-amd64 +export OPENSTACK_FLAVOR=capi-mgmt-node +export OPENSTACK_SSH_KEY_NAME=capi-workload-key +export POD_CIDR=10.244.0.0/16 +export SERVICE_CIDR=10.96.0.0/12 +export CLOUDS_YAML_B64=$(cat "$WORK/clouds.yaml.b64") +export CLOUD_CONF_B64=$(cat "$WORK/cloud.conf.b64") +export VAULT_CA_B64=$(cat "$WORK/vault-ca.crt.b64") +export CLUSTER_DOMAIN=cluster.local +export OPENSTACK_CLOUD=capi-mgmt + +# Sanity print +env | grep -E "^(CLUSTER|KUBERNETES|CONTROL_PLANE|WORKER|OPENSTACK|POD|SERVICE|VAULT|CLOUD)" \ + | grep -v "B64\|SECRET\|PASS" | sort +``` + +**Render the cluster template:** + +```bash +cat > "$WORK/cluster-template.yaml" <<'TEMPLATE_EOF' +apiVersion: v1 +kind: Secret +metadata: + name: ${CLUSTER_NAME}-cloud-config + namespace: ${CLUSTER_NAMESPACE} +type: Opaque +data: + clouds.yaml: ${CLOUDS_YAML_B64} + cloud.conf: ${CLOUD_CONF_B64} + cacert: ${VAULT_CA_B64} +--- +apiVersion: cluster.x-k8s.io/v1beta1 +kind: Cluster +metadata: + name: ${CLUSTER_NAME} + namespace: ${CLUSTER_NAMESPACE} +spec: + clusterNetwork: + pods: + cidrBlocks: + - ${POD_CIDR} + services: + cidrBlocks: + - ${SERVICE_CIDR} + serviceDomain: ${CLUSTER_DOMAIN} + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: OpenStackCluster + name: ${CLUSTER_NAME} + controlPlaneRef: + apiVersion: controlplane.cluster.x-k8s.io/v1beta2 + kind: CK8sControlPlane + name: ${CLUSTER_NAME}-control-plane +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: OpenStackCluster +metadata: + name: ${CLUSTER_NAME} + namespace: ${CLUSTER_NAMESPACE} +spec: + identityRef: + name: ${CLUSTER_NAME}-cloud-config + cloudName: ${OPENSTACK_CLOUD} + externalNetwork: + id: ${OPENSTACK_EXTERNAL_NETWORK_ID} + managedSecurityGroups: + allowAllInClusterTraffic: true + apiServerLoadBalancer: + enabled: true +--- +apiVersion: controlplane.cluster.x-k8s.io/v1beta2 +kind: CK8sControlPlane +metadata: + name: ${CLUSTER_NAME}-control-plane + namespace: ${CLUSTER_NAMESPACE} +spec: + replicas: ${CONTROL_PLANE_MACHINE_COUNT} + version: ${KUBERNETES_VERSION} + machineTemplate: + infrastructureTemplate: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: OpenStackMachineTemplate + name: ${CLUSTER_NAME}-control-plane + spec: + files: + - path: /usr/local/share/ca-certificates/vault-ca.crt + owner: root:root + permissions: "0644" + contentFrom: + secret: + name: ${CLUSTER_NAME}-cloud-config + key: cacert + preRunCommands: + - update-ca-certificates + extraKubeAPIServerArgs: + "--cloud-provider": external +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: OpenStackMachineTemplate +metadata: + name: ${CLUSTER_NAME}-control-plane + namespace: ${CLUSTER_NAMESPACE} +spec: + template: + spec: + flavor: ${OPENSTACK_FLAVOR} + image: + filter: + name: ${OPENSTACK_IMAGE_NAME} + sshKeyName: ${OPENSTACK_SSH_KEY_NAME} + identityRef: + name: ${CLUSTER_NAME}-cloud-config + cloudName: ${OPENSTACK_CLOUD} +--- +apiVersion: cluster.x-k8s.io/v1beta1 +kind: MachineDeployment +metadata: + name: ${CLUSTER_NAME}-md-0 + namespace: ${CLUSTER_NAMESPACE} +spec: + clusterName: ${CLUSTER_NAME} + replicas: ${WORKER_MACHINE_COUNT} + selector: + matchLabels: {} + template: + spec: + clusterName: ${CLUSTER_NAME} + version: ${KUBERNETES_VERSION} + bootstrap: + configRef: + apiVersion: bootstrap.cluster.x-k8s.io/v1beta2 + kind: CK8sConfigTemplate + name: ${CLUSTER_NAME}-md-0 + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: OpenStackMachineTemplate + name: ${CLUSTER_NAME}-md-0 +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: OpenStackMachineTemplate +metadata: + name: ${CLUSTER_NAME}-md-0 + namespace: ${CLUSTER_NAMESPACE} +spec: + template: + spec: + flavor: ${OPENSTACK_FLAVOR} + image: + filter: + name: ${OPENSTACK_IMAGE_NAME} + sshKeyName: ${OPENSTACK_SSH_KEY_NAME} + identityRef: + name: ${CLUSTER_NAME}-cloud-config + cloudName: ${OPENSTACK_CLOUD} +--- +apiVersion: bootstrap.cluster.x-k8s.io/v1beta2 +kind: CK8sConfigTemplate +metadata: + name: ${CLUSTER_NAME}-md-0 + namespace: ${CLUSTER_NAMESPACE} +spec: + template: + spec: + files: + - path: /usr/local/share/ca-certificates/vault-ca.crt + owner: root:root + permissions: "0644" + contentFrom: + secret: + name: ${CLUSTER_NAME}-cloud-config + key: cacert + preRunCommands: + - update-ca-certificates +TEMPLATE_EOF + +# envsubst to render +envsubst < "$WORK/cluster-template.yaml" > "$WORK/cluster-rendered.yaml" + +# Validate as YAML +python3 -c "import yaml; list(yaml.safe_load_all(open('$WORK/cluster-rendered.yaml'))); print('YAML OK')" + +# Quick visual check — no leftover ${...} markers +grep -n '\${' "$WORK/cluster-rendered.yaml" || echo "No unsubstituted variables — good" +``` + +> **CK8sConfig field name caveat:** the exact field names (`files`, `preRunCommands`) and their `contentFrom.secret` schema are CK8s-version-dependent. If `clusterctl init` failed earlier with schema warnings, consult the CK8s release notes for the pinned `$CK8S_VERSION`. + +--- + + +## 14. Apply + poll-to-Ready + +Transfer rendered template to capi-mgmt and apply: + +```bash +scp "$WORK/cluster-rendered.yaml" ubuntu@$CAPI_MGMT_METAL_IP:/home/ubuntu/cluster.yaml + +ssh ubuntu@$CAPI_MGMT_METAL_IP <<'EOF' +set -euo pipefail +kubectl apply -f /home/ubuntu/cluster.yaml +echo "Applied. Waiting for cluster Available status (15-min timeout)..." + +for i in $(seq 1 90); do + STATUS=$(kubectl get cluster capi-mgmt-cluster -o json 2>/dev/null \ + | jq -r '.status.phase // "Unknown"') + READY=$(kubectl get cluster capi-mgmt-cluster -o json 2>/dev/null \ + | jq -r '.status.conditions[]? | select(.type=="Ready") | .status' \ + | head -1) + echo "$(date -Is) phase=$STATUS ready=$READY" + [ "$READY" = "True" ] && { echo "Cluster Ready"; break; } + sleep 10 +done + +kubectl get cluster,machines,kubeadmcontrolplane,machinedeployment -A +EOF +``` + +**If the poll times out before Ready,** typical diagnosis: + +```bash +ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl describe cluster capi-mgmt-cluster +ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl get machines -A +ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl logs -n capo-system deployment/capo-controller-manager --tail=100 +``` + +Common causes: + +- OpenStack API unreachable from capi-mgmt → check Vault CA install on capi-mgmt (§6) +- Image / flavor / network ID wrong in cluster template → re-check §11 variables +- Security group rules block kube-api LB → CAPO usually handles this; check OpenStackCluster status +- Application credential expired / wrong → re-check `$APP_CRED_ID` + +--- + + +## 15. Extract workload kubeconfig + +```bash +ssh ubuntu@$CAPI_MGMT_METAL_IP -- clusterctl get kubeconfig capi-mgmt-cluster \ + > "$WORK/capi-mgmt-cluster.kubeconfig" +chmod 600 "$WORK/capi-mgmt-cluster.kubeconfig" + +# Sanity-check the workload cluster is reachable +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get nodes +# Expect: 1 control plane + 2 workers, all Ready +``` + +If `get nodes` times out, the cluster's API LB may not have allocated its external IP yet, or the firewall rules don't permit jumphost → workload API: + +```bash +# What IP is the cluster's API LB on? +ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl get openstackcluster capi-mgmt-cluster \ + -o json | jq '.status.externalNetwork, .status.controlPlaneEndpoint' + +# Test reachability +curl -sk --max-time 10 "https://:6443/version" && echo " ← reachable" || echo "API LB unreachable" +``` + +--- + + +## 16. `clusterctl init` on target (workload cluster) + +The workload cluster must have the same CAPI providers installed before `move`. + +```bash +# Run from jumphost using the workload kubeconfig +KUBECONFIG="$WORK/capi-mgmt-cluster.kubeconfig" clusterctl init \ + --core "cluster-api:${CAPI_VERSION}" \ + --infrastructure "openstack:${CAPO_VERSION}" \ + --bootstrap "canonical-kubernetes:${CK8S_VERSION}" \ + --control-plane "canonical-kubernetes:${CK8S_VERSION}" \ + --cert-manager-version "${CERT_MANAGER_VERSION}" + +# ORC into workload cluster too +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" apply \ + -f "https://github.com/k-orc/openstack-resource-controller/releases/${ORC_VERSION}/orc.yaml" + +# Wait for everything Available +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" wait \ + --for=condition=Available --timeout=5m deployment --all -n capi-system +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" wait \ + --for=condition=Available --timeout=5m deployment --all -n capo-system +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" wait \ + --for=condition=Available --timeout=5m deployment --all -n cert-manager +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" wait \ + --for=condition=Available --timeout=5m deployment --all -n orc-system +``` + +> **cert-manager double-install caveat:** if CK8s already installed cert-manager during workload bootstrap, the second `clusterctl init` may warn or skip. Check existing cert-manager version against `$CERT_MANAGER_VERSION` — if they differ, version-skew issues may surface post-pivot. Adjust the pin in §4 or accept the existing version. Roosevelt's standard practice is to install cert-manager via `clusterctl init` only (don't pre-install via CK8s) — same approach valid here if you want clean version control. + +--- + + +## 17. `clusterctl move` pivot + +Move all CAPI CRs from bootstrap k3s → workload cluster: + +```bash +# Stage the target kubeconfig on capi-mgmt (where clusterctl move runs) +scp "$WORK/capi-mgmt-cluster.kubeconfig" ubuntu@$CAPI_MGMT_METAL_IP:/home/ubuntu/target.kubeconfig + +# Dry-run first to catch issues before commit +ssh ubuntu@$CAPI_MGMT_METAL_IP -- clusterctl move \ + --to-kubeconfig=/home/ubuntu/target.kubeconfig \ + --dry-run + +# Inspect dry-run output: list of objects to be moved. Should include: +# - Cluster, OpenStackCluster, OpenStackClusterTemplate +# - Secrets (cloud-config) +# - Machine objects, OpenStackMachineTemplate +# - CK8sControlPlane, CK8sConfigTemplate +# - MachineDeployment +# Should NOT include cert-manager state (cert-manager manages its own state +# on each cluster independently) +``` + +**If dry-run looks correct, execute the move:** + +```bash +ssh ubuntu@$CAPI_MGMT_METAL_IP -- clusterctl move \ + --to-kubeconfig=/home/ubuntu/target.kubeconfig + +# Move can take several minutes. Output ends with: "moved successfully" +``` + +--- + + +## 18. Post-pivot verification + +```bash +echo "=== Bootstrap k3s (should now be empty of cluster CRs) ===" +ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl get cluster -A +# Expect: No resources found (or only a header) + +ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl get machines -A +# Expect: No resources found + +ssh ubuntu@$CAPI_MGMT_METAL_IP -- kubectl get openstackcluster -A +# Expect: No resources found + +echo "" +echo "=== Workload cluster (should now own its own cluster CRs) ===" +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get cluster -A +# Expect: capi-mgmt-cluster shown, phase=Provisioned + +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get machines -A +# Expect: 3 machines (1 control-plane + 2 workers), all Running + +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get openstackcluster -A + +echo "" +echo "=== CAPI controllers in workload ===" +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get pods -A \ + | grep -E "(capi|capo|orc|cert-manager)" | grep -v "Running\|Completed" +# Expect: empty (all controller pods Running) + +echo "" +echo "=== OCCM not crash-looping (CRITICAL — main goal of TLS-verify work) ===" +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" get pods -n kube-system \ + -l k8s-app=openstack-cloud-controller-manager +# Expect: 1 pod Running, NOT CrashLoopBackOff + +kubectl --kubeconfig "$WORK/capi-mgmt-cluster.kubeconfig" logs -n kube-system \ + -l k8s-app=openstack-cloud-controller-manager --tail=50 \ + | grep -iE "(tls|cert|error)" | head -20 +# Expect: no TLS/cert errors; OCCM should be healthy +``` + +> **If OCCM crash-loops with "x509: certificate signed by unknown authority":** Vault CA distribution failed. Check (a) `/usr/local/share/ca-certificates/vault-ca.crt` exists on workload nodes; (b) `update-ca-certificates` ran (check `/etc/ssl/certs/ca-certificates.crt` for the Vault CA's subject); (c) the secret reference in CK8sConfigTemplate matched the secret name. SSH into a worker via the jumphost key (`ssh -i $WORK/capi-workload-key ubuntu@`) to diagnose. + +--- + + +## 19. Handoff to v1-do-doc-08 + +The workload kubeconfig at `$WORK/capi-mgmt-cluster.kubeconfig` is the input to `v1-do-doc-08-magnum-driver.md`. Copy it to a stable path: + +```bash +mkdir -p $HOME/magnum-capi +cp "$WORK/capi-mgmt-cluster.kubeconfig" $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig +chmod 600 $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig +echo "Workload kubeconfig staged at: $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig" +``` + +> **Important — post-pivot semantic:** Magnum's `kubeconfig_file` setting (under `[capi_helm]` in `/etc/magnum/magnum.conf.d/99-capi.conf`, per D-007 corrected language) points to the workload cluster, not the bootstrap k3s. With pivot mandatory, Magnum's CAPI calls flow: +> +> ``` +> Magnum/leader → workload cluster API → CAPI controllers (running in workload) +> → create new Cluster CRs (tenant Magnum clusters) +> ``` +> +> The bootstrap k3s on capi-mgmt is now disposable. For v1 testcloud, leave capi-mgmt running so its k3s can be inspected for diagnostics. Roosevelt may destroy capi-mgmt entirely at this point for cost savings. + +--- + + +## 20. Acceptance criteria — go/no-go for v1-do-doc-08 + +- [ ] §4 pins captured to `$DEPLOY_RECORD` and `KUBERNETES_VERSION` set +- [ ] §5 capi-mgmt MAAS-deployed (status `Deployed`) +- [ ] §6 Vault CA installed on capi-mgmt; `openssl s_client` against Keystone returns `Verify return code: 0 (ok)` +- [ ] §7-10 k3s + CAPI controllers + ORC all Running +- [ ] §11 cloud-side resources present (project, user, role assignments, app cred, keypair, image, flavor) +- [ ] §13 cluster template renders with no unsubstituted `${...}` markers; YAML parses +- [ ] §14 workload cluster Ready +- [ ] §15 workload kubeconfig extracted; `kubectl get nodes` shows 3 nodes Ready +- [ ] §16 workload cluster has CAPI providers installed +- [ ] §17 `clusterctl move` reported "moved successfully" +- [ ] §18 bootstrap k3s now empty; workload cluster owns Cluster/Machines/etc.; OCCM Running not CrashLoopBackOff +- [ ] §19 workload kubeconfig staged at `$HOME/magnum-capi/capi-mgmt-cluster.kubeconfig` + +If all checked, proceed to `v1-do-doc-08-magnum-driver.md`. + +--- + + +## 21. Roosevelt deltas (forward-look) + +| Aspect | Testcloud (v1) | Roosevelt | +|---|---|---| +| Workload image | Default `noble-amd64` from cloud-images.ubuntu.com | Custom image baked with Vault CA pre-installed (no runtime install step) | +| Vault CA distribution | CK8sConfig `files:` + `preRunCommands:` (this runbook) | Image-baked + CK8sConfig (defense in depth) | +| App credential lifetime | No expiry set (testcloud) | Short-lived rotating credentials via Vault auth method | +| Workload cluster control plane | 1 node | 3 nodes (HA) | +| Workload cluster workers | 2 nodes | Per-tenant sizing; HPA-driven | +| `clusterctl init --cert-manager-version` | Pin from §4 | Pin to Vault PKI cert-manager profile (separate Roosevelt prep) | +| capi-mgmt VM lifecycle post-pivot | Kept running for diagnostics | Destroyed (cost savings; pivot makes it disposable) | +| Version pinning record | `$HOME/deploy-records//capi-pins/` | Same pattern, captured in Vault as audit artifact | +| Authentication to GitHub API | Optional PAT | Mandatory PAT (avoid rate-limit during automated rebuilds) | + +--- + + +## 22. Change log + +| Date | Change | Reference | +|---|---|---| +| 2026-05-22 | Original runbook 04a created. Vault CA distribution (no tls-insecure), mandatory `clusterctl move` pivot, pin-at-execution version model. | Workstream 3b | +| 2026-05-27 | Adapted into v1-do-doc-07. Fixes: `$REPO` path; `$VAULT_CA` path; `$MAAS_PROFILE` set; §4 dynamic KUBERNETES_VERSION discovery; §5 MAAS poll exit converted to non-exiting [FAIL]; §11 noble-amd64 missing branch converted to [FAIL]; cross-references updated to v1-do-doc set. | Batch C drafting | diff --git a/runbooks/v1-do-doc-08-magnum-driver.md b/runbooks/v1-do-doc-08-magnum-driver.md new file mode 100644 index 0000000..6e8fb90 --- /dev/null +++ b/runbooks/v1-do-doc-08-magnum-driver.md @@ -0,0 +1,534 @@ +# v1 Do-Document 08 — Magnum CAPI Helm Driver Install + +**Status:** Third execution document of Batch C. Last document in the Magnum/CAPI stack. Grafts the CAPI Helm driver onto the deployed Magnum so `openstack coe cluster create` provisions tenant K8s clusters via the workload cluster's CAPI controllers (not via the deprecated Heat driver). + +**Position in sequence:** Runs after `v1-do-doc-07-capi-bootstrap.md` (workload cluster + kubeconfig staged at `$HOME/magnum-capi/capi-mgmt-cluster.kubeconfig`). Final document of Batch C. Followed by Batch D (tenant + DNS + validate). + +**Replaces:** `runbooks/05-magnum-capi-driver.md` — same substantive procedure with fixes applied. The old runbook moves to `runbooks/deprecated/` as part of this batch's commits. + +**Fixes applied vs the prior runbook (`runbooks/05-magnum-capi-driver.md`):** + +- §1 — removed the "Known doc inconsistency (tracked for cleanup)" notice about D-007. D-007's Layer B language was corrected in the 2026-05-22 change-log entry of `docs/design-decisions.md`; the notice is now obsolete. +- Cross-references updated: "runbook 04" → v1-do-doc-06; kubeconfig source documented as staged by v1-do-doc-07 §19. +- §11 optional smoketest `exit 1` on `CREATE_FAILED` converted to non-exiting `[FAIL]` report — the smoketest is optional and a failure should not kill the operator's shell. + +**Cross-references:** + +- D-007 Layer B (Magnum two-layer install) +- D-017 (CAPI bootstrap cluster lifecycle) +- v1-do-doc-07 §19 (workload kubeconfig handoff) +- Workstream 3c decision (2026-05-22): magnum-capi-helm 1.1.0 from PyPI; workload-cluster kubeconfig (NOT bootstrap k3s) + +--- + + +## 1. Purpose & scope + +Graft the CAPI Helm driver onto the Charmed Magnum deployment so that `openstack coe cluster create` provisions tenant K8s clusters via CAPI (in the workload cluster) instead of via the deprecated Heat driver. + +**Output of this runbook:** + +- `magnum-capi-helm==1.1.0` installed on the magnum unit's system Python. +- `/etc/magnum/kubeconfig` populated with the workload cluster's kubeconfig (post-pivot CAPI controller plane). +- `/etc/magnum/magnum.conf.d/99-capi.conf` configured with `enabled_drivers = k8s_capi_helm_v1` and `[capi_helm] kubeconfig_file=`. +- Systemd overrides on `magnum-api` and `magnum-conductor` that replace the init.d wrapper's ExecStart with explicit `--config-dir` invocation. +- Both services running cleanly with the CAPI driver loaded. + +**Scope:** v1 testcloud. Roosevelt deltas in §12. + +**Out of scope:** + +- Magnum domain setup (v1-do-doc-06) +- Workload cluster lifecycle (v1-do-doc-07) +- Smoketest tenant cluster creation is OPTIONAL (§11) — full validation framework belongs in v1-do-doc-11. + +--- + + +## 2. Decisions captured + +| Decision | Choice | Reason | +|---|---|---| +| Driver pin | `magnum-capi-helm==1.1.0` from PyPI | D-007 correction (stackhpc fork archived Dec 2024; canonical project on opendev/PyPI; 1.1.0 is last Caracal-cycle release) | +| Install method | `pip3 install --break-system-packages` | PEP 668 — Ubuntu 22.04+ requires explicit override for system-site-packages install | +| Install scope | System Python on magnum unit (not venv) | Magnum charm uses system-packaged python at `/usr/lib/python3/dist-packages/magnum/`; driver must import from same site | +| Kubeconfig target | Workload cluster (post-pivot) | Workstream 3b — bootstrap k3s is empty post-pivot; CAPI controllers live in workload | +| Kubeconfig source | `$HOME/magnum-capi/capi-mgmt-cluster.kubeconfig` (staged by v1-do-doc-07 §19) | Documented handoff | +| Driver entry-point name | `k8s_capi_helm_v1` | Per upstream magnum-capi-helm 1.1.0; verify in §5 | +| Conf.d filename | `99-capi.conf` | Numeric prefix ensures it loads AFTER any charm-managed conf, so `enabled_drivers` override wins | +| File encoding | ASCII-only | Non-ASCII in conf.d causes silent magnum daemon failures (handoff lesson; cf. Horizon `local_settings.d` issue) | +| Trustee credential | Existing magnum-shared user (charm-managed) | Roosevelt will use app-credential pattern | + +--- + + +## 3. Prerequisites + +| Prereq | Verification | +|---|---| +| Magnum charm active/idle | `juju status magnum | grep magnum/0` shows `active idle` | +| Magnum domain setup completed (v1-do-doc-06) | `( source $HOME/admin-openrc; openstack domain show magnum -f value -c enabled )` returns `True` | +| Workload cluster reachable from jumphost | `kubectl --kubeconfig $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig get nodes` returns Ready nodes | +| CAPI controllers running in workload cluster | `kubectl --kubeconfig $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig get pods -n capi-system | grep -v Running | grep -v NAME` empty | +| Workload kubeconfig staged at expected path | `test -r $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig && stat -c %a $HOME/magnum-capi/capi-mgmt-cluster.kubeconfig` shows `600` | +| `juju exec` works to magnum/leader (use exec, NOT ssh, for non-interactive — handoff lesson) | `juju exec --unit magnum/leader -- hostname` returns the unit hostname | + +**Set shell context:** + +```bash +export WORK=$HOME/magnum-capi +export WORKLOAD_KUBECONFIG=$WORK/capi-mgmt-cluster.kubeconfig +export DRIVER_VERSION=magnum-capi-helm==1.1.0 # per D-007 correction +cd "$WORK" +``` + +> **`juju ssh` vs `juju exec` choice:** the handoff lessons explicitly call out that `juju ssh` hangs when stdout is redirected (PTY allocation issue). This runbook uses `juju exec` for all non-interactive command execution and reserves `juju ssh` only for cases where you actually want an interactive shell. + +--- + + +## 4. Pre-flight: capture current state + +Capture the magnum unit's state BEFORE making changes. Useful for diagnosis if anything goes wrong, and as a record of what was changed. + +```bash +mkdir -p "$WORK/pre-state" + +# Service unit files (as managed by charm) +juju exec --unit magnum/leader -- \ + 'sudo systemctl cat magnum-api magnum-conductor 2>&1' \ + > "$WORK/pre-state/systemd-units.txt" + +# Currently-enabled drivers +juju exec --unit magnum/leader -- \ + 'sudo grep -r enabled_drivers /etc/magnum/ 2>/dev/null || echo "(no enabled_drivers found — charm default applies)"' \ + > "$WORK/pre-state/drivers-pre.txt" + +# Python site-packages — see what's already installed +juju exec --unit magnum/leader -- \ + 'sudo pip3 list 2>/dev/null | grep -iE "magnum|cluster|helm|kubernetes" || true' \ + > "$WORK/pre-state/pip-pre.txt" + +# conf.d state +juju exec --unit magnum/leader -- \ + 'sudo ls -la /etc/magnum/magnum.conf.d/ 2>/dev/null || echo "(no conf.d directory)"' \ + > "$WORK/pre-state/confd-pre.txt" + +# Service running state +juju exec --unit magnum/leader -- \ + 'sudo systemctl is-active magnum-api magnum-conductor' \ + > "$WORK/pre-state/service-state-pre.txt" + +# Display the captured state +cat "$WORK/pre-state/"*.txt +``` + +> **What to look for in pre-state:** the charm-managed `enabled_drivers` value probably includes Heat-based drivers (`heat_kubernetes`, etc.). The 99-capi.conf override in §7 replaces this with the single CAPI driver. The pre-state capture documents what was active before the override took effect. + +--- + + +## 5. Install magnum-capi-helm 1.1.0 + +```bash +juju exec --unit magnum/leader -- \ + "sudo pip3 install $DRIVER_VERSION --break-system-packages" +``` + +**Verify install:** + +```bash +juju exec --unit magnum/leader -- \ + 'sudo pip3 show magnum-capi-helm | head -10' +# Expect: Name: magnum-capi-helm +# Version: 1.1.0 +# Location: /usr/lib/python3/dist-packages + +juju exec --unit magnum/leader -- \ + 'sudo python3 -c "import magnum_capi_helm; print(magnum_capi_helm.__file__)"' +# Expect: /usr/lib/python3/dist-packages/magnum_capi_helm/__init__.py +``` + +**Check that the driver entry point is registered:** + +```bash +juju exec --unit magnum/leader -- \ + 'sudo python3 -c " +from stevedore import driver +mgr = driver.DriverManager( + namespace=\"magnum.drivers\", + name=\"k8s_capi_helm_v1\", + invoke_on_load=False +) +print(\"Driver class:\", mgr.driver) +"' +# Expect: Driver class: +# (or similar — the actual class path is package-version-dependent) +``` + +> If the entry point check fails with "No 'k8s_capi_helm_v1' driver found", the driver name in 1.1.0 may differ from what D-007 documented. Inspect the installed package's `entry_points.txt`: +> +> ```bash +> juju exec --unit magnum/leader -- \ +> 'sudo cat /usr/lib/python3/dist-packages/magnum_capi_helm*.dist-info/entry_points.txt 2>/dev/null' +> ``` +> +> Find the entry under `[magnum.drivers]` — use that exact name in §7. + +--- + + +## 6. Stage workload kubeconfig on magnum unit + +```bash +# Transfer kubeconfig from jumphost to magnum unit +juju scp "$WORKLOAD_KUBECONFIG" magnum/leader:/tmp/kubeconfig + +# Install with correct ownership/mode in one atomic step +juju exec --unit magnum/leader -- \ + 'sudo install -m 0640 -o root -g magnum /tmp/kubeconfig /etc/magnum/kubeconfig && sudo rm /tmp/kubeconfig' +``` + +**Verify:** + +```bash +juju exec --unit magnum/leader -- \ + 'sudo ls -la /etc/magnum/kubeconfig' +# Expect: -rw-r----- 1 root magnum ... /etc/magnum/kubeconfig + +# Confirm magnum user can read it +juju exec --unit magnum/leader -- \ + 'sudo -u magnum cat /etc/magnum/kubeconfig | head -3' +# Expect: apiVersion: v1 / clusters: / - cluster: + +# Confirm kubectl can use it from the magnum unit (sanity check on API reachability) +juju exec --unit magnum/leader -- \ + 'sudo -u magnum kubectl --kubeconfig /etc/magnum/kubeconfig get nodes 2>&1 | head -10' +# Expect: NAME ... STATUS=Ready for control plane + workers +# OR: kubectl not installed (acceptable — magnum-capi-helm uses Python client, not kubectl) +``` + +> **Why mode 0640 and group magnum:** kubeconfig contains auth tokens. Mode 0600 (owner-only) wouldn't let the `magnum` system user (which runs magnum-api/conductor) read it. Mode 0640 with `group: magnum` is the minimum-permission setup that works. NOT 0644 — keeps it off other users on the unit. + +--- + + +## 7. Configure `/etc/magnum/magnum.conf.d/99-capi.conf` + +Generate the conf locally first (keep paths under `$HOME` for consistency with snap confinement on other steps), then transfer. + +**ASCII-only verification is critical** — the handoff documents non-ASCII characters in `conf.d` files causing silent daemon failures (cf. Horizon `local_settings.d`). Use plain straight quotes, ASCII dashes, no smart typography. + +```bash +# Write locally +cat > "$WORK/99-capi.conf" <<'EOF' +[DEFAULT] +enabled_drivers = k8s_capi_helm_v1 + +[capi_helm] +kubeconfig_file = /etc/magnum/kubeconfig +EOF + +# Verify it is pure ASCII (no UTF-8 sneakers) +file "$WORK/99-capi.conf" +# Expect: ASCII text +# If it says "UTF-8 Unicode text", STOP and rewrite by hand — even one stray +# em-dash or smart quote will silently break magnum + +# Hex dump check (paranoid mode) +xxd "$WORK/99-capi.conf" | grep -v "^[0-9a-f]*: [0-9a-f ]* [a-zA-Z0-9 \[\]=._/]*$" | head -5 +# Expect: empty output (all bytes are printable ASCII) +``` + +**Stage and install:** + +```bash +juju scp "$WORK/99-capi.conf" magnum/leader:/tmp/99-capi.conf + +juju exec --unit magnum/leader -- \ + 'sudo mkdir -p /etc/magnum/magnum.conf.d && sudo install -m 0644 -o root -g root /tmp/99-capi.conf /etc/magnum/magnum.conf.d/99-capi.conf && sudo rm /tmp/99-capi.conf' + +# Verify +juju exec --unit magnum/leader -- \ + 'sudo ls -la /etc/magnum/magnum.conf.d/ && sudo cat /etc/magnum/magnum.conf.d/99-capi.conf' +# Expect: file listed; content matches what was written +``` + +--- + + +## 8. Systemd override on magnum-api + magnum-conductor + +The Charmed Magnum unit files use a wrapper pattern: + +``` +ExecStart=/etc/init.d/magnum-api systemd-start +``` + +The wrapper does NOT pass `--config-dir` to magnum-api, so `/etc/magnum/magnum.conf.d/` is never loaded. The 99-capi.conf would have no effect. + +Override with explicit `--config-file` + `--config-dir` invocation. + +**Generate override files locally:** + +```bash +cat > "$WORK/magnum-api-override.conf" <<'EOF' +[Service] +ExecStart= +ExecStart=/usr/bin/magnum-api --config-file=/etc/magnum/magnum.conf --config-dir=/etc/magnum/magnum.conf.d +EOF + +cat > "$WORK/magnum-conductor-override.conf" <<'EOF' +[Service] +ExecStart= +ExecStart=/usr/bin/magnum-conductor --config-file=/etc/magnum/magnum.conf --config-dir=/etc/magnum/magnum.conf.d +EOF + +# ASCII check +file "$WORK/magnum-api-override.conf" "$WORK/magnum-conductor-override.conf" +# Expect: ASCII text x2 +``` + +> **The empty `ExecStart=` line is critical.** Systemd accumulates ExecStart directives by default; an empty assignment is required to CLEAR the inherited directive before setting the replacement. Without the empty line, the unit would have BOTH the init.d wrapper AND the new direct invocation, and would likely fail to start. + +**Install on the unit:** + +```bash +juju scp "$WORK/magnum-api-override.conf" magnum/leader:/tmp/magnum-api-override.conf +juju scp "$WORK/magnum-conductor-override.conf" magnum/leader:/tmp/magnum-conductor-override.conf + +juju exec --unit magnum/leader -- \ + 'sudo mkdir -p /etc/systemd/system/magnum-api.service.d /etc/systemd/system/magnum-conductor.service.d && \ + sudo install -m 0644 -o root -g root /tmp/magnum-api-override.conf /etc/systemd/system/magnum-api.service.d/override.conf && \ + sudo install -m 0644 -o root -g root /tmp/magnum-conductor-override.conf /etc/systemd/system/magnum-conductor.service.d/override.conf && \ + sudo rm /tmp/magnum-api-override.conf /tmp/magnum-conductor-override.conf' + +# Reload systemd to pick up the overrides +juju exec --unit magnum/leader -- 'sudo systemctl daemon-reload' + +# Verify the overrides are effective (systemctl cat shows combined unit + overrides) +juju exec --unit magnum/leader -- 'sudo systemctl cat magnum-api | grep -A1 ExecStart' +# Expect: TWO ExecStart= lines — the empty clear-line and the new /usr/bin/magnum-api invocation +juju exec --unit magnum/leader -- 'sudo systemctl cat magnum-conductor | grep -A1 ExecStart' +# Expect: TWO ExecStart= lines as above for magnum-conductor +``` + +> **Charm reconciliation note:** the Magnum charm may rewrite its own systemd units on config changes or upgrades. The drop-in override at `/etc/systemd/system/magnum-api.service.d/override.conf` is OUTSIDE the charm's writable zone and should survive. Verify after any `juju refresh` or `juju config magnum` command by re-running the `systemctl cat` check above. + +--- + + +## 9. Restart services + verify health + +```bash +juju exec --unit magnum/leader -- \ + 'sudo systemctl restart magnum-api magnum-conductor' + +# Wait briefly for services to initialize +sleep 5 + +# Check active state +juju exec --unit magnum/leader -- \ + 'sudo systemctl is-active magnum-api magnum-conductor' +# Expect: active (x2) + +# Examine recent journal for errors (the critical step — magnum's silent failure +# mode means we must read logs, not just trust is-active) +juju exec --unit magnum/leader -- \ + 'sudo journalctl -u magnum-api --since "2 minutes ago" --no-pager | tail -50' +juju exec --unit magnum/leader -- \ + 'sudo journalctl -u magnum-conductor --since "2 minutes ago" --no-pager | tail -50' +``` + +**Look for these red flags in the logs:** + +| Symptom | Likely cause | Remediation | +|---|---|---| +| `ImportError: No module named magnum_capi_helm` | §5 pip install failed | Re-run §5; check pip3 output | +| `EntryPointError: No 'k8s_capi_helm_v1' driver` | Driver entry-point name mismatch | Verify name per §5 footnote; update §7 | +| Service repeatedly restarts (look for "Started" appearing twice in 10s) | Likely a config error in 99-capi.conf | Re-check ASCII-only; check magnum.conf.d permissions | +| `kubeconfig_file` not honored | --config-dir not being passed | §8 override not active; re-run `systemctl daemon-reload` | +| Silent: no error but driver also not loading | Non-ASCII char snuck into a conf | `file /etc/magnum/magnum.conf.d/99-capi.conf` — if it says UTF-8, regenerate | + +--- + + +## 10. CAPI driver enablement check + +Verify the driver is actually loaded by Magnum and reachable via the API. + +```bash +source $HOME/admin-openrc + +# List supported COE drivers via the Magnum API +openstack coe cluster template list -f json +# (empty templates list is fine — we are checking the endpoint responds) + +# Direct check on the unit: scan the service's loaded drivers +juju exec --unit magnum/leader -- \ + 'sudo journalctl -u magnum-conductor --since "5 minutes ago" --no-pager | grep -iE "driver|enabled" | head -20' +# Expect: a line mentioning k8s_capi_helm_v1 having been loaded +# (Magnum logs the loaded drivers at startup) + +# Definitive check: try creating a cluster template that requires the CAPI driver +openstack coe cluster template create magnum-capi-driver-check \ + --image noble-amd64 \ + --keypair capi-workload-key \ + --external-network ext_net \ + --master-flavor capi-mgmt-node \ + --flavor capi-mgmt-node \ + --coe kubernetes \ + --network-driver calico \ + --labels kube_tag=$KUBERNETES_VERSION + +openstack coe cluster template show magnum-capi-driver-check -c name -c coe -c labels +``` + +> **If template create fails with "driver not enabled" or similar:** the Magnum API process is not loading the conf.d. Verify the systemd override took effect — `sudo systemctl show magnum-api -p ExecStart` on the unit should show the explicit `--config-dir` invocation. If it still shows the init.d wrapper, the daemon-reload + restart did not pick up the override. + +> **`$KUBERNETES_VERSION` carry-over:** if your shell session no longer has `$KUBERNETES_VERSION` set from v1-do-doc-07 §4, re-read it from `$HOME/capi-bootstrap/pins/KUBERNETES_VERSION` or substitute the actual version in the `--labels kube_tag=` flag. + +**Cleanup the driver-check template:** + +```bash +openstack coe cluster template delete magnum-capi-driver-check +``` + +--- + + +## 11. Optional smoketest — create a tenant CAPI cluster + +This step is **optional**. Full validation belongs in v1-do-doc-11. Use this smoketest only if you want immediate confirmation that the entire chain (Magnum API → conductor → magnum-capi-helm → CAPI controllers in workload cluster → tenant K8s cluster on tenant VMs) works end-to-end. + +```bash +# Create a cluster template tuned for testcloud smoketest +openstack coe cluster template create magnum-smoketest-template \ + --image noble-amd64 \ + --keypair capi-workload-key \ + --external-network ext_net \ + --master-flavor capi-mgmt-node \ + --flavor capi-mgmt-node \ + --coe kubernetes \ + --network-driver calico \ + --labels boot_volume_size=20,kube_tag=$KUBERNETES_VERSION,octavia_provider=ovn + +# Create a 1+1 cluster (minimum for smoketest) +openstack coe cluster create magnum-smoketest \ + --cluster-template magnum-smoketest-template \ + --master-count 1 \ + --node-count 1 + +# Poll for status (15-20 min typical; CAPI provisions tenant VMs end-to-end) +SMOKETEST_RESULT="" +for i in $(seq 1 60); do + STATUS=$(openstack coe cluster show magnum-smoketest -c status -f value 2>/dev/null) + echo "$(date -Is) status=$STATUS" + case "$STATUS" in + CREATE_COMPLETE) + echo "[OK] Smoketest passed" + SMOKETEST_RESULT="pass" + break + ;; + CREATE_FAILED) + echo "[FAIL] Smoketest cluster creation failed. Investigate via:" + echo " openstack coe cluster show magnum-smoketest" + echo " openstack stack list # if any Heat stack remained" + echo " kubectl --kubeconfig \$HOME/magnum-capi/capi-mgmt-cluster.kubeconfig get cluster,machines -A" + echo " juju exec --unit magnum/leader -- sudo journalctl -u magnum-conductor --since '30 minutes ago'" + SMOKETEST_RESULT="fail" + break + ;; + esac + sleep 30 +done + +if [ "$SMOKETEST_RESULT" = "fail" ]; then + echo "" + echo "[FAIL] Smoketest did not complete cleanly. Stop here, investigate, and decide whether to proceed." + echo " The smoketest cluster may need manual cleanup — see cleanup block below if you want to remove it." +elif [ -z "$SMOKETEST_RESULT" ]; then + echo "[WARN] Smoketest poll timed out without reaching a terminal state. Cluster may still be provisioning." + echo " Manually check with: openstack coe cluster show magnum-smoketest" +fi +``` + +**If the smoketest reached `CREATE_COMPLETE`:** + +```bash +# Retrieve the smoketest cluster's kubeconfig +openstack coe cluster config magnum-smoketest --dir "$WORK/smoketest-kubeconfig" + +# Sanity-check the smoketest cluster +KUBECONFIG="$WORK/smoketest-kubeconfig/config" kubectl get nodes +KUBECONFIG="$WORK/smoketest-kubeconfig/config" kubectl get pods -A | head -20 +``` + +**Cleanup the smoketest cluster (regardless of pass/fail):** + +```bash +openstack coe cluster delete magnum-smoketest 2>/dev/null || echo "(cluster may already be deleting)" +openstack coe cluster template delete magnum-smoketest-template 2>/dev/null || echo "(template may already be deleted)" +``` + +> **What success looks like:** the CAPI controllers in the workload cluster receive the new Cluster CR (created by magnum-capi-helm in response to the Magnum API call), CAPO talks to OpenStack to provision tenant VMs, the tenant VMs join the new K8s cluster, and the new cluster has 1 control plane + 1 worker Ready. Octavia provides the API server LB (visible as a Floating IP in the tenant project). + +--- + + +## 12. Roosevelt deltas (forward-look) + +| Aspect | Testcloud (v1) | Roosevelt | +|---|---|---| +| Driver pin source | PyPI `magnum-capi-helm==1.1.0` | Internal mirror with checksum verification | +| Driver pin record | Implicit in this runbook | Captured in Vault as audit artifact alongside CAPI pins | +| Kubeconfig source | Workload cluster (post-pivot per v1-do-doc-07 §17) | Same | +| Kubeconfig rotation | Manual on capi-mgmt rebuild | Automated when workload cluster cert rotates | +| Trustee credential | Charm-default magnum-shared user | Per-tenant app credentials via Vault auth method | +| Magnum HA | num_units=1 (per D-009 testcloud) | num_units=3 with hacluster + provider VIP | +| Driver upgrade discipline | Manual re-run of §5 | Tracked maintenance window; Vault audit log | +| Systemd override | Drop-in at `/etc/systemd/system/magnum-*.service.d/override.conf` | Same — but provided via a charm overlay package, not manual file install | +| ASCII-only enforcement | Manual check (§7, §8) | Pre-flight lint in `scripts/pre-flight-checks.sh` | + +--- + + +## 13. Documented runtime gotchas (carry-forward from handoff) + +These gotchas burned cycles during the Bobcat Magnum CAPI work. Each is explicitly handled in this runbook; collecting them here for visibility: + +1. **PEP 668 `--break-system-packages`** (§5). Ubuntu 22.04+ refuses `pip install` against system Python by default. The flag is required for the magnum-capi-helm install path used by Charmed Magnum. +2. **`juju ssh` hangs on stdout redirect.** PTY allocation issue. This runbook uses `juju exec` for all non-interactive command execution. +3. **Heredoc nesting in `juju ssh` is fragile.** This runbook writes conf files locally first and uses `juju scp` + `juju exec install` to transfer — single-level only. +4. **Non-ASCII characters in `conf.d` files cause silent daemon failures.** §7 and §8 both include `file ` ASCII verification before transfer. +5. **`openstack -f value -c X -c Y` outputs in alphabetical field order, not flag order.** This runbook uses single-column queries or `-f json | jq` throughout. +6. **Charm-managed `enabled_drivers` is overridden, not appended.** The `enabled_drivers = k8s_capi_helm_v1` line in 99-capi.conf REPLACES the charm-default value (which would include the deprecated Heat drivers). +7. **The systemd override empty `ExecStart=` line is required** to clear the inherited ExecStart before setting the replacement (§8). +8. **Snap-confined `openstack` CLI cannot read `/tmp`.** This runbook stages files under `$WORK=$HOME/magnum-capi`. The smoketest in §11 also writes to `$WORK/smoketest-kubeconfig`. + +--- + + +## 14. Acceptance criteria — go/no-go for Batch D + +- [ ] §4 pre-state captured to `$WORK/pre-state/` +- [ ] §5 `magnum-capi-helm` installed; `pip3 show magnum-capi-helm` shows version 1.1.0 from `/usr/lib/python3/dist-packages` +- [ ] §5 driver entry point check returns the driver class (or operator confirmed the actual entry-point name in 1.1.0) +- [ ] §6 `/etc/magnum/kubeconfig` present with mode 0640 root:magnum; magnum user can read it +- [ ] §7 `/etc/magnum/magnum.conf.d/99-capi.conf` present; `file` reports ASCII text +- [ ] §8 systemd overrides at `/etc/systemd/system/magnum-*.service.d/override.conf`; `systemctl cat` shows TWO `ExecStart=` lines for both units +- [ ] §9 magnum-api and magnum-conductor active; no errors in recent journal +- [ ] §10 `magnum-capi-driver-check` template creation succeeded; deleted after +- [ ] §11 OPTIONAL — smoketest cluster reached `CREATE_COMPLETE` and was cleaned up (or skipped intentionally) + +If all required (non-optional) items checked, Batch C is complete. Proceed to Batch D (`v1-do-doc-09-tenant.md` first). + +--- + + +## 15. Change log + +| Date | Change | Reference | +|---|---|---| +| 2026-05-22 | Original runbook 05 created. magnum-capi-helm 1.1.0 from PyPI; workload-cluster kubeconfig (post-pivot per workstream 3b); systemd override pattern; ASCII-only conf.d. | Workstream 3c | +| 2026-05-27 | Adapted into v1-do-doc-08. Fixes: §1 obsolete D-007 inconsistency notice removed; cross-references updated to v1-do-doc set; §11 smoketest `exit 1` converted to non-exiting `[FAIL]` report (smoketest is optional and should not kill the operator shell). | Batch C drafting |