# Design Decisions — VR0 DC0 Omega Cloud This document is the architectural record for the VR0 DC0 testcloud rebuild. Every decision listed here has been deliberately made and discussed; it is not a wishlist or a brainstorm. If a decision is changed, this document is updated and the change is committed with a referencing message. **Scope split:** This repository implements **v1 (IPv4-only)**. Several decisions below are tagged with **[v2-scope]** — they remain valid design intent but are deferred to a future v2 deployment when upstream router infrastructure supports IPv6. See **D-015** for the v1/v2 fork record. --- ## D-001: Deployment target paradigm **Decision:** Path 2A — Charmed OpenStack Caracal (2024.1) via Juju bundle. **Alternatives considered:** - Path 2B — Canonical Sunbeam (microk8s-based). Rejected: discards most of the test-cloud experience accumulated to date; different operator paradigm. - Path 1 — Stay on Bobcat 2023.2. Rejected: defeats the purpose of a Caracal rehearsal ahead of Roosevelt bare-metal. **Consequences:** - Bundle-based deployment, suitable for both KVM testcloud and bare-metal scale. - Caracal-stable channel matrix applies (see D-002). - EOL date is April 2027 (Caracal upstream support window). --- ## D-002: Channel pinning matrix **Decision:** Pin every charm to a Caracal-stable channel. No OVN pinning on testcloud (Roosevelt will pin via `ovn-source`). | Charm group | Channel | | ----------------------------------------------------------------------------------------------------------------------- | -------------------------- | | OpenStack core (keystone, glance, nova-\*, neutron-api, cinder, placement, octavia, barbican, designate, magnum, vault) | `2024.1/stable` | | OVN (ovn-central, ovn-chassis, ovn-dedicated-chassis-octavia) | `24.03/stable` | | Ceph (ceph-mon, ceph-osd, ceph-radosgw if used) | `squid/stable` (see D-005) | | MySQL (mysql-innodb-cluster, mysql-router subordinates) | `8.0/stable` | | RabbitMQ | `3.9/stable` | | Vault | `1.8/stable` | | hacluster | `2.4/stable` | | etcd, easyrsa | `latest/stable` | **Verification source:** Caracal channel matrix per Canonical Charmed OpenStack Charm Delivery table, verified against Charmhub 2026-05-22 (keystone rev 778, ovn-central rev 332, hacluster `2.4/stable`). Canonical advises against `latest/stable` for OS-stack charms due to non-determinism on charm upgrades; hacluster is therefore pinned to its stable channel. Verify against Charmhub before each deploy via `scripts/pre-flight-checks.sh`. --- ## D-003: Network architecture — Option B **Decision:** Provider network carries BOTH ext_net (tenant FIPs + SNAT egress) AND OpenStack public API VIPs on the same L2 segment. **Rationale:** During Magnum CAPI Phase 3 on the Bobcat testcloud, OCCM crashloop was traced to tenant networks being unable to reach OpenStack API endpoints — the libvirt FORWARD chain rejected cross-bridge packets between provider (virbr1) and metal (virbr2) bridges. With API VIPs on metal, tenant workloads cannot reach them. Putting API VIPs on the same network as the FIPs makes the API path tenant-reachable by construction. **Address space layout for v1 (IPv4-only):** | Range | Purpose | | --------------------------- | -------------------------------------------------------- | | `10.12.4.10 – 10.12.4.223` | Neutron FIP pool | | `10.12.4.224 – 10.12.4.254` | Charm API VIPs (excluded from Neutron allocation\_pools) | The Provider `/22` (`10.12.4.0/22`) carries both ranges within a single Neutron subnet. Neutron `allocation_pools` MUST exclude the API VIP range. **v2-scope extension:** IPv6 Provider subnet adds parallel FIP and API VIP IPv6 IP Ranges within a single `/64`. See D-004. --- ## D-004 [v2-scope]: Dual-stack vs IPv6-only matrix **Decision (v2-scope, NOT for v1):** Network role determines address family. IPv6 preferred; IPv6-only where the network has no external clients. **v1 reality:** All networks are IPv4-only on the existing MAAS-provisioned layout. This matrix becomes active in v2. | Role | IPv4 (v1) | IPv4 (v2) | IPv6 (v2) | Reasoning | | -------------------------- | ------------- | --------- | --------- | --------------------------------------------------- | | Metal | ✓ | ✓ | ✓ | Charm-to-charm; MAAS PXE IPv4-first | | Provider | ✓ | ✓ | ✓ | Tenant FIPs need IPv4; API VIPs reachable from both | | Data (Geneve underlay) | ✓ | — | ✓ | v2: no external clients; underlay agnostic | | Storage (Ceph public) | ✓ | — | ✓ | v2: `ms-bind-ipv6: true`; no external clients | | Replication (Ceph cluster) | ✓ | — | ✓ | v2: internal OSD↔OSD only | | LBaaS Management | ✓ | ✓ | ✓ | Amphora image compatibility | | OOB | n/a | n/a | n/a | Bare-metal-only concern | | OpenStack Tenant pool | ✓ (v1: D-016) | — | ✓ | v1 IPv4 hybrid; v2 IPv6 modeled | --- ## D-004a [v2-scope]: Host management → Metal **Decision (v2-scope, NOT for v1):** Under v2, openstack0-3 host management IPs move from storage (`10.12.16.40-.43`) to Metal (`10.12.8.0/22`) when Storage becomes IPv6-only. v1 keeps host management on storage. --- ## D-005: Ceph release **Decision:** Squid (Ceph 19, released October 2024). **Rationale:** Matches Caracal default; one fewer source override in bundle; rehearses what Roosevelt will run. If Squid has rough edges, the testcloud is the place to find them, not production. **Alternatives considered:** - Reef (Ceph 18) — current on Bobcat testcloud; lower risk; would require `source: cloud:jammy-caracal` override on ceph-mon/ceph-osd while keeping `reef/stable` channel. Rejected: defeats the rehearsal purpose. --- ## D-006: Vault HA backend **Decision:** etcd + easyrsa, per Canonical Charmed Vault HA docs. **Rationale:** This is the documented charm path. The chicken-and-egg TLS dependency (Vault needs certs to start, but Vault issues certs) is resolved by easyrsa bootstrapping the etcd cluster's TLS, after which Vault relations to etcd come up cleanly. **Topology on testcloud (v1):** Vault num_units=1 + hacluster relation (decorative; documents the relation pattern). Vault HA quorum is not actually exercised at testcloud scale. **Topology on Roosevelt:** Vault num_units=3 + hacluster on metal space; etcd num_units=3; easyrsa num_units=1. --- ## D-007: Magnum inclusion **Decision:** Magnum in bundle from day one. Two-layer install. **Layer A — Bundle:** - `magnum` charm - `magnum-mysql-router` subordinate - `magnum-dashboard` subordinate - Standard relations: keystone, mysql-innodb-cluster (via router), rabbitmq-server, vault (certificates), openstack-dashboard - Binding: `public: provider` with VIP on provider API VIP range - Hacluster relation included (decorative on testcloud) **Layer B — Post-deploy runbook (`runbooks/05-magnum-capi-driver.md`):** - `juju run magnum/leader domain-setup --wait=10m` - pip install `magnum-capi-helm==1.1.0` from PyPI into the magnum charm venv with `--break-system-packages` (stackhpc/magnum-capi-helm fork archived Dec 2024; canonical project moved to `openstack/magnum-capi-helm` on opendev/PyPI; 1.1.0 is the last Caracal-cycle release. Upstream tests against Magnum 2023.1+, so backward-compatible through Caracal 2024.1.) - Deploy `/etc/magnum/kubeconfig` pointing at `capi-mgmt.maas` bootstrap k3s - Systemd override replacing init.d ExecStart to load `--config-dir` - `/etc/magnum/magnum.conf.d/99-capi.conf` setting `enabled_drivers=k8s_capi_helm_v1` and `[capi_helm] kubeconfig_file=/etc/magnum/kubeconfig` (ASCII-only; non-ASCII characters in conf.d cause silent daemon failures) **CAPI mgmt plane:** `capi-mgmt.maas` bootstrap k3s. Per **D-017**, this cluster is rebuilt from scratch every deployment cycle — there is no preserved-across-rebuild artifact. The install procedure for the bootstrap cluster lives in `runbooks/04a-capi-bootstrap-cluster.md` and runs **before** this runbook. This pattern transfers to Roosevelt unchanged. **Superseded portions:** The "preserved across rebuild" stance in earlier drafts of this decision is **superseded by D-017**. See D-017 for rationale. The earlier `stackhpc/magnum-capi-helm` v0.13.0 driver pin is superseded by the `openstack/magnum-capi-helm` 1.1.0 pin above (driver source repo moved + archived). --- ## D-008: DNS architecture **Decision:** Layered — static /etc/hosts for bootstrap + Designate (in bundle from day one) for tenant-level resolution. **Naming convention:** ``` ....cloud.neumatrix.local ``` Examples: - `keystone.omega.dc0.vr0.cloud.neumatrix.local` - `nova.omega.dc0.vr0.cloud.neumatrix.local` **Bootstrap order:** 1. Static `/etc/hosts` on jumphost + all openstack0-3 hosts + all LXD containers 2. Bundle deploys with `os-public-hostname: ` per API charm 3. Vault issues certs with FQDN in SAN 4. Post-deploy: Designate zone created, A records populated (v1: A records only; v2 adds AAAA records) 5. Neutron `default_dns_domain` and `dns_servers` configured to point at Designate 6. Tenant subnets created with `--dns-nameserver ` --- ## D-009: Hacluster modeling at testcloud scale **Decision:** Include hacluster + VIP relations at num_units=1 across all HA-eligible API charms. **Rationale:** Decorative at testcloud scale (a single unit can't form a real HA quorum). Documents the relation pattern so Roosevelt scale-up is mechanical: change `num_units: 1` → `num_units: 3` and rerun. **Charms with hacluster relation:** keystone, glance, neutron-api, nova-cloud-controller, placement, openstack-dashboard, cinder, octavia, barbican, magnum, vault, designate. --- ## D-010: NetBox-upstream policy **Decision:** NetBox is the single source of truth for IPAM at the **role and cloud-level pool** layer. Per-project tenant subnets are exempt under the hybrid model (D-016). **Workflow:** Update NetBox → update bundle/overlay → commit both with cross-reference. **Standing imports for v1 (gating the bundle):** - VR0 DC0 site exists in NetBox ✓ - IPv4 prefixes for v1: Metal /22, Provider /22, LBaaS Mgmt /22 (via `netbox/ipv4-prefixes-import.py`) — **pending** - Provider IP Ranges for FIPs and API VIPs (same script) — **pending** - IPv4 tenant pool /16 (same script, per D-016) — **pending** - IPv6 entries marked as Reservation status (via `netbox/ipv6-mark-reserved.py`) — **pending** **Deferred to v2 (per Q2):** VR0 DC0-VLANs group additions beyond VID 240 (already imported during prior session work). MAAS currently uses untagged-per-fabric; modeling additional VLANs in NetBox without corresponding network-side tagging would be misleading documentation. --- ## D-011: Validation bar — Roosevelt-rehearsal level **Decision:** Deployment is not considered successful until all of the following pass: 1. All charms `active/idle` in `juju status` 2. API reachability from jumphost (all public VIPs respond on hostname) 3. API reachability from a tenant VM (Option B verification) 4. Octavia LB pattern re-passes (round-robin, failover, recovery — per Bobcat v3 work) 5. End-to-end Magnum CAPI cluster creation succeeds, including OCCM not crash-looping 6. Vault unseal + auto-unseal-after-reboot pattern verified 7. KVM snapshot baseline taken (Phase 5) 8. Designate zones populated and tenant VMs resolve API hostnames Validation script: `scripts/validate.sh` (TBD). --- ## D-012: Snapshot strategy **Decision:** Two baseline snapshots. - **Snapshot 1:** Post-deploy, post-validation, pre-tenant-resources. Clean cloud state — what a fresh install looks like. - **Snapshot 2:** Post-tenant-setup. Includes domain1, project1, user1, openrc, flavors, base images (noble-amd64), keypair. Restore point for tenant work. Snapshots are KVM/qcow2-level on the jumphost hypervisor. Per-VM. --- ## D-013: ~~Clean teardown of existing capi-mgmt~~ (SUPERSEDED by D-018) **Original decision:** Before destroying the OpenStack model, gracefully delete the CAPI workload cluster on capi-mgmt.maas to allow OpenStack resources (LBs, FIPs, volumes) to be cleaned up properly by CAPI controllers. **Original steps:** `kubectl delete cluster capi-mgmt-cluster` → wait for CAPI to clean up tenant-side OpenStack resources → `juju destroy-model openstack --destroy-storage --no-prompt`. **Original "preserved across rebuild" claim:** capi-mgmt.maas bootstrap k3s + CAPI controllers re-used as the Magnum CAPI mgmt plane post-deploy. **Status:** Superseded. See **D-018** for the replacement teardown strategy (MAAS-release-direct, skip graceful) and **D-017** for the replacement bootstrap cluster lifecycle (full rebuild every cycle, nothing preserved). --- ## D-014: Repository storage location and naming **Decision:** Self-hosted GitBucket at `git.baldurkeep.com`. **Repo path:** `jesse.austin/openstack-caracal-ipv4` (v1; IPv4-only). **v2 repository:** TBD when v2 work begins. Two viable paths: sibling repo `openstack-caracal-ipv6` or `openstack-caracal-dualstack`, OR `v2` branch in this repo with an `overlays/v2-dualstack.yaml`. The single-repo-with-branch approach preserves history of what changed v1→v2 together; the sibling-repo approach keeps v1 frozen as a reference once v2 is in motion. **Branching strategy:** `main` is canonical. Per-phase work in feature branches when a deploy is in progress; merge back to `main` at successful validation. --- ## D-015: v1 / v2 Fork **Decision:** Caracal testcloud ships in two iterations. **v1 (this repository, `openstack-caracal-ipv4`):** IPv4-only Caracal on existing MAAS-provisioned network layout. Proves the bundle, Option B binding fix, Magnum CAPI graft, Designate-from-day-one, hacluster relation pattern, and validation framework. Ships first. **v2 (deferred):** Adds IPv6 / dual-stack per D-004. Requires upstream router infrastructure to be IPv6-capable, which is not currently the case in this environment. v2 work begins after v1 validation passes AND router-side IPv6 is in place. **Rationale:** Decoupling the OpenStack-side rebuild from the network-side IPv6 readiness lets us prove the more-important architectural fix (Option B) without waiting on infrastructure work outside the OpenStack deployment's control. The IPv6 design intent is preserved as NetBox Reservation-status entries (per D-010 and `netbox/ipv6-mark-reserved.py`). **v1→v2 migration scope (forward-look):** - Re-IP roles per D-004 (add IPv6 sibling to Metal/Provider/LBaaS; move Data/Storage/Replication to IPv6-only) - Move host management IPs from storage to Metal (D-004a) - Re-bind charms to listen on both families where dual-stack - Add AAAA records to Designate zones - Add tenant IPv6 pool carve-outs --- ## D-016: IPv4 tenant pool — hybrid model (v1) **Decision:** NetBox owns one upstream IPv4 tenant pool prefix for VR0 DC0. Per-project tenant subnets are Neutron-managed within that pool and are NOT modeled in NetBox. **Pool allocation:** `10.20.0.0/16` (default; configurable in `netbox/ipv4-prefixes-import.py`). 65,536 addresses; 256 `/24`s available for per-project tenant subnets. Modeled under VR0 DC0 with role `openstack-tenant`. **Per-project allocation pattern (operationally):** When a project is created, allocate a /24 from the pool. Operator records the allocation in tenant-setup runbook output but does NOT create a NetBox prefix entry for it. Suggested convention: `10.20..0/24`, starting with `10.20.1.0/24` for project1, etc. **Rationale (Option C from the discussion):** - Option A (NetBox-modeled per-project) — full IPAM rigor; high friction for tenant lifecycle; round-trips to NetBox for ephemeral tenants. - Option B (Neutron-only, no NetBox standing) — minimum friction; loses upstream visibility of total tenant footprint; violates D-010 in spirit. - Option C (hybrid, chosen) — NetBox documents what space is reserved for tenants and prevents accidental collision with infra ranges; Neutron owns the lifecycle of individual tenant subnets without NetBox round-trips. **Constraint:** Tenant CIDRs MUST be within the pool. The pre-flight checklist (`scripts/pre-flight-checks.sh`) should assert that proposed tenant subnets fall within the modeled pool. **v2-scope counterpart:** IPv6 tenant pool `2602:f3e2:ff:0::/56` (NetBox-modeled, Reservation status in v1) becomes active in v2 with the same hybrid model — pool has NetBox standing, per-project IPv6 subnets Neutron-managed. --- ## D-017: CAPI bootstrap cluster lifecycle **Decision:** L3 full teardown and rebuild every deployment cycle. The `capi-mgmt.maas` MAAS VM is released back to Ready state on teardown; on rebuild, it is re-deployed from scratch with Ubuntu 24.04, k3s, CAPI controllers, and ORC. **Nothing is preserved across cycles.** **Rationale:** - Rehearsal-first principle. If the bootstrap-cluster install procedure isn't documented and rehearsed, the runbook doesn't exist; if the runbook doesn't exist, surprises surface on Roosevelt. - Self-imposed forcing function. Every rebuild exercises the full path: MAAS deploy → Ubuntu cloud-init → Vault CA install → k3s install with correct bind-address/SAN flags → kubeconfig server-URL rewrite → helm + clusterctl install → clusterctl init with canonical-kubernetes provider URLs → ORC install → cloud-side prep → cluster manifest render → apply → poll-to-Ready → kubeconfig copy. - Disposability test. The Bobcat experience proved no critical state lives on capi-mgmt that isn't reproducible from the runbook and the OpenStack cloud. Wiping is safe. **Runbook:** `runbooks/04a-capi-bootstrap-cluster.md` documents the install sequence in full. It runs **after** `02-deploy.md` (OpenStack cloud up) and **before** `05-magnum-capi-driver.md` (driver graft, which needs the bootstrap k3s kubeconfig). **Supersedes:** the "preserved across rebuild" stance in earlier drafts of D-007 and D-013. **Alternatives considered:** - L1: Wipe just the cluster CRs, keep k3s + controllers. Rejected: skips the install rehearsal that's the whole point. - L2: Wipe just the controllers, keep k3s. Rejected: same reason; the `clusterctl init` step is exactly the surface that needs rehearsing. - L3 (chosen): Full wipe including the VM. --- ## D-018: Teardown strategy — skip graceful, release MAAS directly **Decision:** On teardown, do not pursue graceful CAPI workload deletion or graceful OpenStack model destroy. Instead: 1. (Optional) Capture pre-destroy state for reference 2. `juju destroy-model openstack --force --no-wait --destroy-storage --no-prompt` (background) 3. MAAS release all 5 VMs (openstack0, openstack1, openstack2, openstack3, capi-mgmt) → Ready (parallel) 4. Verify both sides **Rationale:** - The rebuild's goal is rehearsing the Roosevelt deploy path. Roosevelt starts from MAAS-Ready bare-metal machines. The most faithful rehearsal is teardown-to-MAAS-Ready. - Graceful CAPI workload teardown rehearses a different procedure (production cluster decommissioning) that doesn't transfer to Roosevelt's initial deploy. - `juju destroy-model --destroy-storage` can hang on stuck hooks and leave partial state. `--force --no-wait` plus MAAS release is more reliable. - Cloud-side OpenStack data (Keystone projects, Neutron networks, Glance images, app credentials) lives in MySQL on the openstack0-3 hosts. MAAS release wipes those hosts, so no separate cloud-side cleanup is needed. **What is lost vs. graceful path:** verified-clean release path for CAPI workload resources (Octavia LBs, FIPs, CAPO-managed networks). All of these are destined for obliteration anyway; the loss is theoretical. **What is gained:** ~30+ minutes saved; cleaner end-state guarantee; better Roosevelt rehearsal fidelity. **Supersedes:** D-013. **Runbook:** `runbooks/01-destroy-model.md` documents the four phases. --- From prior bundle review work — these are anti-patterns: - `magnum-shared-db` missing colon — causes a relation endpoint syntax error, deploy-blocking. Bundle must use `- - magnum:shared-db` (with the colon). - Empty `osd-devices` YAML anchor referenced by multiple ceph-osd applications. - `ovn-chassis` binding `overlay-suffix` — invalid binding name. Correct value is `data`. - GUI annotation collision between NUMA-split ceph-osd apps (not applicable to testcloud since we don't NUMA-split, but flagged for Roosevelt). - Hardcoded NIC name in `bridge-interface-mappings`. Use MAC where possible. - `openstack -f value` column ordering — column order is not guaranteed; use `-c -f value` for single-column output. - Snap confinement: `openstackclients` snap has home-only interface; commands cannot read paths under `/tmp`. File paths must resolve under `$HOME`. - Non-ASCII characters in `local_settings.d` overrides cause silent daemon failures in Horizon. --- ## Change log | Date | Change | Reference | | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- | | 2026-05-22 | Initial document captured | Caracal rebuild planning session | | 2026-05-22 | D-015 v1/v2 fork added; D-004 and D-004a marked v2-scope; D-016 IPv4 tenant pool hybrid model added; D-014 updated with new repo name | v1/v2 fork session | | 2026-05-22 | D-017 CAPI bootstrap full-rebuild lifecycle added; D-018 MAAS-release-direct teardown added. D-013 marked superseded by D-018. D-007 Layer B updated to reference D-017 and `runbooks/04a-capi-bootstrap-cluster.md`. | Teardown planning + handoff session | | 2026-05-22 | D-002 hacluster row added (channel `2.4/stable`) per Canonical Charm Delivery table, verified against Charmhub. D-007 Layer B driver pin updated: `stackhpc/magnum-capi-helm` v0.13.0 → `openstack/magnum-capi-helm` 1.1.0 (PyPI; stackhpc fork archived Dec 2024). | Caracal channel verification + driver pin correction |