# Design Decisions — VR0 DC0 Omega Cloud

This document is the architectural record for the VR0 DC0 testcloud rebuild. Every decision listed here has been deliberately made and discussed; it is not a wishlist or a brainstorm. If a decision is changed, this document is updated and the change is committed with a referencing message.

**Scope split:** This repository implements **v1 (IPv4-only)**. Several decisions below are tagged with **[v2-scope]** — they remain valid design intent but are deferred to a future v2 deployment when upstream router infrastructure supports IPv6. See **D-015** for the v1/v2 fork record.

---


## D-001: Deployment target paradigm

**Decision:** Path 2A — Charmed OpenStack Caracal (2024.1) via Juju bundle.

**Alternatives considered:**

- Path 2B — Canonical Sunbeam (microk8s-based). Rejected: discards most of the test-cloud experience accumulated to date; different operator paradigm.
- Path 1 — Stay on Bobcat 2023.2. Rejected: defeats the purpose of a Caracal rehearsal ahead of Roosevelt bare-metal.


**Consequences:**

- Bundle-based deployment, suitable for both KVM testcloud and bare-metal scale.
- Caracal-stable channel matrix applies (see D-002).
- EOL date is April 2027 (Caracal upstream support window).

---


## D-002: Channel pinning matrix

**Decision:** Pin every charm to a Caracal-stable channel. No OVN pinning on testcloud (Roosevelt will pin via `ovn-source`).

| Charm group                                                                                                             | Channel                    |
| ----------------------------------------------------------------------------------------------------------------------- | -------------------------- |
| OpenStack core (keystone, glance, nova-\*, neutron-api, cinder, placement, octavia, barbican, designate, magnum, vault) | `2024.1/stable`            |
| OVN (ovn-central, ovn-chassis, ovn-dedicated-chassis-octavia)                                                           | `24.03/stable`             |
| Ceph (ceph-mon, ceph-osd, ceph-radosgw if used)                                                                         | `squid/stable` (see D-005) |
| MySQL (mysql-innodb-cluster, mysql-router subordinates)                                                                 | `8.0/stable`               |
| RabbitMQ                                                                                                                | `3.9/stable`               |
| Vault                                                                                                                   | `1.8/stable`               |
| hacluster                                                                                                               | `2.4/stable`               |
| etcd, easyrsa                                                                                                           | `latest/stable`            |


**Verification source:** Caracal channel matrix per Canonical Charmed OpenStack Charm Delivery table, verified against Charmhub 2026-05-22 (keystone rev 778, ovn-central rev 332, hacluster `2.4/stable`). Canonical advises against `latest/stable` for OS-stack charms due to non-determinism on charm upgrades; hacluster is therefore pinned to its stable channel. Verify against Charmhub before each deploy via `scripts/pre-flight-checks.sh`.

---


## D-003: Network architecture — Option B

**Decision:** Provider network carries BOTH ext_net (tenant FIPs + SNAT egress) AND OpenStack public API VIPs on the same L2 segment.

**Rationale:** During Magnum CAPI Phase 3 on the Bobcat testcloud, OCCM crashloop was traced to tenant networks being unable to reach OpenStack API endpoints — the libvirt FORWARD chain rejected cross-bridge packets between provider (virbr1) and metal (virbr2) bridges. With API VIPs on metal, tenant workloads cannot reach them. Putting API VIPs on the same network as the FIPs makes the API path tenant-reachable by construction.

**Address space layout for v1 (IPv4-only):**

| Range                       | Purpose                                                  |
| --------------------------- | -------------------------------------------------------- |
| `10.12.4.10 – 10.12.4.223`  | Neutron FIP pool                                         |
| `10.12.4.224 – 10.12.4.254` | Charm API VIPs (excluded from Neutron allocation\_pools) |


The Provider `/22` (`10.12.4.0/22`) carries both ranges within a single Neutron subnet. Neutron `allocation_pools` MUST exclude the API VIP range.

**v2-scope extension:** IPv6 Provider subnet adds parallel FIP and API VIP IPv6 IP Ranges within a single `/64`. See D-004.

---


## D-004 [v2-scope]: Dual-stack vs IPv6-only matrix

**Decision (v2-scope, NOT for v1):** Network role determines address family. IPv6 preferred; IPv6-only where the network has no external clients.

**v1 reality:** All networks are IPv4-only on the existing MAAS-provisioned layout. This matrix becomes active in v2.

| Role                       | IPv4 (v1)     | IPv4 (v2) | IPv6 (v2) | Reasoning                                           |
| -------------------------- | ------------- | --------- | --------- | --------------------------------------------------- |
| Metal                      | ✓             | ✓         | ✓         | Charm-to-charm; MAAS PXE IPv4-first                 |
| Provider                   | ✓             | ✓         | ✓         | Tenant FIPs need IPv4; API VIPs reachable from both |
| Data (Geneve underlay)     | ✓             | —         | ✓         | v2: no external clients; underlay agnostic          |
| Storage (Ceph public)      | ✓             | —         | ✓         | v2: `ms-bind-ipv6: true`; no external clients       |
| Replication (Ceph cluster) | ✓             | —         | ✓         | v2: internal OSD↔OSD only                           |
| LBaaS Management           | ✓             | ✓         | ✓         | Amphora image compatibility                         |
| OOB                        | n/a           | n/a       | n/a       | Bare-metal-only concern                             |
| OpenStack Tenant pool      | ✓ (v1: D-016) | —         | ✓         | v1 IPv4 hybrid; v2 IPv6 modeled                     |

---


## D-004a [v2-scope]: Host management → Metal

**Decision (v2-scope, NOT for v1):** Under v2, openstack0-3 host management IPs move from storage (`10.12.16.40-.43`) to Metal (`10.12.8.0/22`) when Storage becomes IPv6-only. v1 keeps host management on storage.

---


## D-005: Ceph release

**Decision:** Squid (Ceph 19, released October 2024).

**Rationale:** Matches Caracal default; one fewer source override in bundle; rehearses what Roosevelt will run. If Squid has rough edges, the testcloud is the place to find them, not production.

**Alternatives considered:**

- Reef (Ceph 18) — current on Bobcat testcloud; lower risk; would require `source: cloud:jammy-caracal` override on ceph-mon/ceph-osd while keeping `reef/stable` channel. Rejected: defeats the rehearsal purpose.

---


## D-006: Vault HA backend

**Decision:** etcd + easyrsa, per Canonical Charmed Vault HA docs.

**Rationale:** This is the documented charm path. The chicken-and-egg TLS dependency (Vault needs certs to start, but Vault issues certs) is resolved by easyrsa bootstrapping the etcd cluster's TLS, after which Vault relations to etcd come up cleanly.

**Topology on testcloud (v1):** Vault num_units=1 + hacluster relation (decorative; documents the relation pattern). Vault HA quorum is not actually exercised at testcloud scale.

**Topology on Roosevelt:** Vault num_units=3 + hacluster on metal space; etcd num_units=3; easyrsa num_units=1.

---


## D-007: Magnum inclusion (Layer A current; Layer B mechanism/topology superseded -- see D-035 / D-037 / D-042)

**Decision:** Magnum in bundle from day one. Two-layer install.

**Layer A — Bundle:**

- `magnum` charm
- `magnum-mysql-router` subordinate
- `magnum-dashboard` subordinate
- Standard relations: keystone, mysql-innodb-cluster (via router), rabbitmq-server, vault (certificates), openstack-dashboard
- Binding: `public: provider` with VIP on provider API VIP range
- Hacluster relation included (decorative on testcloud)


**Layer B — Post-deploy runbook (`runbooks/05-magnum-capi-driver.md`):**

- `juju run magnum/leader domain-setup --wait=10m`
- pip install `magnum-capi-helm==1.1.0` from PyPI into the magnum charm venv with `--break-system-packages` (stackhpc/magnum-capi-helm fork archived Dec 2024; canonical project moved to `openstack/magnum-capi-helm` on opendev/PyPI; 1.1.0 is the last Caracal-cycle release. Upstream tests against Magnum 2023.1+, so backward-compatible through Caracal 2024.1.)
- Deploy `/etc/magnum/kubeconfig` pointing at the **workload cluster** (the post-pivot home of CAPI controllers per **runbook 04a §17** `clusterctl move`). Staged on jumphost at `$HOME/magnum-capi/capi-mgmt-cluster.kubeconfig` by runbook 04a §19, transferred to the magnum unit by runbook 05 §6. Bobcat had this pointing at bootstrap k3s because the pivot was never executed; workstream 3b (2026-05-22) made the pivot mandatory.
- Systemd override replacing init.d ExecStart to load `--config-dir`
- `/etc/magnum/magnum.conf.d/99-capi.conf` setting `enabled_drivers=k8s_capi_helm_v1` and `[capi_helm] kubeconfig_file=/etc/magnum/kubeconfig` (ASCII-only; non-ASCII characters in conf.d cause silent daemon failures)


**CAPI mgmt plane:** Post-pivot, the workload cluster IS the CAPI management plane (per **runbook 04a §17**, `clusterctl move` pivots cluster state from the `capi-mgmt.maas` bootstrap k3s into the workload cluster, which becomes self-managing). Per **D-017**, both the bootstrap k3s and the workload cluster are rebuilt from scratch every deployment cycle — there is no preserved-across-rebuild artifact. The bootstrap install + pivot procedure lives in `runbooks/04a-capi-bootstrap-cluster.md` and runs **before** this runbook. This pattern transfers to Roosevelt unchanged.

**Superseded portions:** The "preserved across rebuild" stance in earlier drafts of this decision is **superseded by D-017**. See D-017 for rationale. The earlier `stackhpc/magnum-capi-helm` v0.13.0 driver pin is superseded by the `openstack/magnum-capi-helm` 1.1.0 pin above (driver source repo moved + archived). The Layer B *mechanism and topology* are now further superseded: the CAPI management plane is an in-cloud single-homed VM with NO `clusterctl move` -- the kubeconfig points at that mgmt cluster, not a post-pivot workload cluster (**D-035**); the conductor graft is `/etc/default/magnum-conductor` `DAEMON_ARGS --config-dir` + `00-capi-helm.conf`, NOT a systemd-ExecStart override + `99-capi.conf` (**D-037**); the driver pin is **1.4.0** for CAPI-core contract coherence (**D-042**); and the per-cluster app-cred replaces any static `capo` credential (**D-039**). Layer A (the bundle) is current. Live deploy steps are runbooks/phase-06 (mgmt cluster) and phase-07 (conductor graft + `domain-setup` D-046 + keystone-v3 drop-in D-047); `runbooks/05-magnum-capi-driver.md` is historical.

---


## D-008: DNS architecture

**Decision:** Layered — static /etc/hosts for bootstrap + Designate (in bundle from day one) for tenant-level resolution.

**Naming convention:**

```
<service>.<cloud>.<dc>.<region>.cloud.neumatrix.local
```
Examples:

- `keystone.omega.dc0.vr0.cloud.neumatrix.local`
- `nova.omega.dc0.vr0.cloud.neumatrix.local`


**Bootstrap order:**

1. Static `/etc/hosts` on jumphost + all openstack0-3 hosts + all LXD containers
2. Bundle deploys with `os-public-hostname: <fqdn>` per API charm
3. Vault issues certs with FQDN in SAN
4. Post-deploy: Designate zone created, A records populated (v1: A records only; v2 adds AAAA records)
5. Neutron `default_dns_domain` and `dns_servers` configured to point at Designate
6. Tenant subnets created with `--dns-nameserver <designate-vip>`

---


## D-009: Hacluster modeling at testcloud scale

**Decision:** Include hacluster + VIP relations at num_units=1 across all HA-eligible API charms.

**Rationale:** Decorative at testcloud scale (a single unit can't form a real HA quorum). Documents the relation pattern so Roosevelt scale-up is mechanical: change `num_units: 1` → `num_units: 3` and rerun.

**Charms with hacluster relation:** keystone, glance, neutron-api, nova-cloud-controller, placement, openstack-dashboard, cinder, octavia, barbican, magnum, vault, designate.

---


## D-010: NetBox-upstream policy

**Decision:** NetBox is the single source of truth for IPAM at the **role and cloud-level pool** layer. Per-project tenant subnets are exempt under the hybrid model (D-016).

**Workflow:** Update NetBox → update bundle/overlay → commit both with cross-reference.

**Standing imports for v1 (gating the bundle):**

- VR0 DC0 site exists in NetBox ✓
- IPv4 prefixes for v1: Metal /22, Provider /22, LBaaS Mgmt /22 (via `netbox/ipv4-prefixes-import.py`) — **pending**
- Provider IP Ranges for FIPs and API VIPs (same script) — **pending**
- IPv4 tenant pool /16 (same script, per D-016) — **pending**
- IPv6 entries marked as Reservation status (via `netbox/ipv6-mark-reserved.py`) — **pending**


**Deferred to v2 (per Q2):** VR0 DC0-VLANs group additions beyond VID 240 (already imported during prior session work). MAAS currently uses untagged-per-fabric; modeling additional VLANs in NetBox without corresponding network-side tagging would be misleading documentation.

---


## D-011: Validation bar — Roosevelt-rehearsal level

**Decision:** Deployment is not considered successful until all of the following pass:

1. All charms `active/idle` in `juju status`
2. API reachability from jumphost (all public VIPs respond on hostname)
3. API reachability from a tenant VM (Option B verification)
4. Octavia LB pattern re-passes (round-robin, failover, recovery — per Bobcat v3 work)
5. End-to-end Magnum CAPI cluster creation succeeds, including OCCM not crash-looping
6. Vault unseal + auto-unseal-after-reboot pattern verified
7. KVM snapshot baseline taken (Phase 5)
8. Designate zones populated and tenant VMs resolve API hostnames


Validation script: `scripts/validate.sh` (TBD).

---


## D-012: Snapshot strategy

**Decision:** Two baseline snapshots.

- **Snapshot 1:** Post-deploy, post-validation, pre-tenant-resources. Clean cloud state — what a fresh install looks like.
- **Snapshot 2:** Post-tenant-setup. Includes domain1, project1, user1, openrc, flavors, base images (noble-amd64), keypair. Restore point for tenant work.


Snapshots are KVM/qcow2-level on the jumphost hypervisor. Per-VM.

---


## D-013: ~~Clean teardown of existing capi-mgmt~~ (SUPERSEDED by D-018)

**Original decision:** Before destroying the OpenStack model, gracefully delete the CAPI workload cluster on capi-mgmt.maas to allow OpenStack resources (LBs, FIPs, volumes) to be cleaned up properly by CAPI controllers.

**Original steps:** `kubectl delete cluster capi-mgmt-cluster` → wait for CAPI to clean up tenant-side OpenStack resources → `juju destroy-model openstack --destroy-storage --no-prompt`.

**Original "preserved across rebuild" claim:** capi-mgmt.maas bootstrap k3s + CAPI controllers re-used as the Magnum CAPI mgmt plane post-deploy.

**Status:** Superseded. See **D-018** for the replacement teardown strategy (MAAS-release-direct, skip graceful) and **D-017** for the replacement bootstrap cluster lifecycle (full rebuild every cycle, nothing preserved).

---


## D-014: Repository storage location and naming

**Decision:** Self-hosted GitBucket at `git.baldurkeep.com`.

**Repo path:** `jesse.austin/openstack-caracal-ipv4` (v1; IPv4-only).

**v2 repository:** TBD when v2 work begins. Two viable paths: sibling repo `openstack-caracal-ipv6` or `openstack-caracal-dualstack`, OR `v2` branch in this repo with an `overlays/v2-dualstack.yaml`. The single-repo-with-branch approach preserves history of what changed v1→v2 together; the sibling-repo approach keeps v1 frozen as a reference once v2 is in motion.

**Branching strategy:** `main` is canonical. Per-phase work in feature branches when a deploy is in progress; merge back to `main` at successful validation.

---


## D-015: v1 / v2 Fork

**Decision:** Caracal testcloud ships in two iterations.

**v1 (this repository, `openstack-caracal-ipv4`):** IPv4-only Caracal on existing MAAS-provisioned network layout. Proves the bundle, Option B binding fix, Magnum CAPI graft, Designate-from-day-one, hacluster relation pattern, and validation framework. Ships first.

**v2 (deferred):** Adds IPv6 / dual-stack per D-004. Requires upstream router infrastructure to be IPv6-capable, which is not currently the case in this environment. v2 work begins after v1 validation passes AND router-side IPv6 is in place.

**Rationale:** Decoupling the OpenStack-side rebuild from the network-side IPv6 readiness lets us prove the more-important architectural fix (Option B) without waiting on infrastructure work outside the OpenStack deployment's control. The IPv6 design intent is preserved as NetBox Reservation-status entries (per D-010 and `netbox/ipv6-mark-reserved.py`).

**v1→v2 migration scope (forward-look):**

- Re-IP roles per D-004 (add IPv6 sibling to Metal/Provider/LBaaS; move Data/Storage/Replication to IPv6-only)
- Move host management IPs from storage to Metal (D-004a)
- Re-bind charms to listen on both families where dual-stack
- Add AAAA records to Designate zones
- Add tenant IPv6 pool carve-outs

---


## D-016: IPv4 tenant pool — hybrid model (v1)

**Decision:** NetBox owns one upstream IPv4 tenant pool prefix for VR0 DC0. Per-project tenant subnets are Neutron-managed within that pool and are NOT modeled in NetBox.

**Pool allocation:** `10.20.0.0/16` (default; configurable in `netbox/ipv4-prefixes-import.py`). 65,536 addresses; 256 `/24`s available for per-project tenant subnets. Modeled under VR0 DC0 with role `openstack-tenant`.

**Per-project allocation pattern (operationally):**

When a project is created, allocate a /24 from the pool. Operator records the allocation in tenant-setup runbook output but does NOT create a NetBox prefix entry for it. Suggested convention: `10.20.<project-index>.0/24`, starting with `10.20.1.0/24` for project1, etc.

**Rationale (Option C from the discussion):**

- Option A (NetBox-modeled per-project) — full IPAM rigor; high friction for tenant lifecycle; round-trips to NetBox for ephemeral tenants.
- Option B (Neutron-only, no NetBox standing) — minimum friction; loses upstream visibility of total tenant footprint; violates D-010 in spirit.
- Option C (hybrid, chosen) — NetBox documents what space is reserved for tenants and prevents accidental collision with infra ranges; Neutron owns the lifecycle of individual tenant subnets without NetBox round-trips.


**Constraint:** Tenant CIDRs MUST be within the pool. The pre-flight checklist (`scripts/pre-flight-checks.sh`) should assert that proposed tenant subnets fall within the modeled pool.

**v2-scope counterpart:** IPv6 tenant pool `2602:f3e2:ff:0::/56` (NetBox-modeled, Reservation status in v1) becomes active in v2 with the same hybrid model — pool has NetBox standing, per-project IPv6 subnets Neutron-managed.

---


## D-017: CAPI bootstrap cluster lifecycle (bootstrap mechanism superseded by D-035; full-rebuild principle retained)

**Decision:** L3 full teardown and rebuild every deployment cycle. The `capi-mgmt.maas` MAAS VM is released back to Ready state on teardown; on rebuild, it is re-deployed from scratch with Ubuntu 24.04, k3s, CAPI controllers, and ORC. **Nothing is preserved across cycles.**

**Rationale:**

- Rehearsal-first principle. If the bootstrap-cluster install procedure isn't documented and rehearsed, the runbook doesn't exist; if the runbook doesn't exist, surprises surface on Roosevelt.
- Self-imposed forcing function. Every rebuild exercises the full path: MAAS deploy → Ubuntu cloud-init → Vault CA install → k3s install with correct bind-address/SAN flags → kubeconfig server-URL rewrite → helm + clusterctl install → clusterctl init with canonical-kubernetes provider URLs → ORC install → cloud-side prep → cluster manifest render → apply → poll-to-Ready → kubeconfig copy.
- Disposability test. The Bobcat experience proved no critical state lives on capi-mgmt that isn't reproducible from the runbook and the OpenStack cloud. Wiping is safe.


**Runbook:** `runbooks/04a-capi-bootstrap-cluster.md` documents the install sequence in full. It runs **after** `02-deploy.md` (OpenStack cloud up) and **before** `05-magnum-capi-driver.md` (driver graft, which needs the bootstrap k3s kubeconfig).

**Supersedes:** the "preserved across rebuild" stance in earlier drafts of D-007 and D-013.

**Superseded by:** **D-035** retired the bootstrap MECHANISM -- there is no `capi-mgmt.maas` MAAS VM, no k3s, and no `clusterctl move`/pivot; the management plane is an in-cloud single-homed tenant VM (`capi-mgmt-v2`, k8s-snap) built in runbooks/phase-06. The full-teardown-and-rebuild-every-cycle PRINCIPLE stated here is RETAINED (now realized by phase-00 teardown + the phase-06 rebuild). `runbooks/04a-capi-bootstrap-cluster.md` and `05-magnum-capi-driver.md` are historical (folded into phase-06/07).

**Alternatives considered:**

- L1: Wipe just the cluster CRs, keep k3s + controllers. Rejected: skips the install rehearsal that's the whole point.
- L2: Wipe just the controllers, keep k3s. Rejected: same reason; the `clusterctl init` step is exactly the surface that needs rehearsing.
- L3 (chosen): Full wipe including the VM.

---


## D-018: Teardown strategy — skip graceful, release MAAS directly

**Decision:** On teardown, do not pursue graceful CAPI workload deletion or graceful OpenStack model destroy. Instead:

1. (Optional) Capture pre-destroy state for reference
2. `juju destroy-model openstack --force --no-wait --destroy-storage --no-prompt` (background)
3. MAAS release all 5 VMs (openstack0, openstack1, openstack2, openstack3, capi-mgmt) → Ready (parallel)
4. Verify both sides


**Rationale:**

- The rebuild's goal is rehearsing the Roosevelt deploy path. Roosevelt starts from MAAS-Ready bare-metal machines. The most faithful rehearsal is teardown-to-MAAS-Ready.
- Graceful CAPI workload teardown rehearses a different procedure (production cluster decommissioning) that doesn't transfer to Roosevelt's initial deploy.
- `juju destroy-model --destroy-storage` can hang on stuck hooks and leave partial state. `--force --no-wait` plus MAAS release is more reliable.
- Cloud-side OpenStack data (Keystone projects, Neutron networks, Glance images, app credentials) lives in MySQL on the openstack0-3 hosts. MAAS release wipes those hosts, so no separate cloud-side cleanup is needed.


**What is lost vs. graceful path:** verified-clean release path for CAPI workload resources (Octavia LBs, FIPs, CAPO-managed networks). All of these are destined for obliteration anyway; the loss is theoretical.

**What is gained:** ~30+ minutes saved; cleaner end-state guarantee; better Roosevelt rehearsal fidelity.

**Supersedes:** D-013.

**Runbook:** `runbooks/01-destroy-model.md` documents the four phases.

---


## D-019: Cloud DNS (Designate) deferred to v2 / Roosevelt

**Decision:** v1 ships with NO cloud-internal DNS; Designate is not deployed. Public service endpoints use FQDNs (`os-public-hostname`) that resolve to the provider VIPs via external/corporate DNS; internal and admin endpoints stay IP-based on the metal VIPs. Tenant instances use upstream resolvers (1.1.1.1 / 1.0.0.1). The D-011 acceptance bar is amended to drop the cloud-DNS criterion, and the planned `v1-do-doc-10-dns` runbook is dropped.

**Consequence (documented, not a blocker):** metal-only charm units that make catalog-based client calls pull the PUBLIC (FQDN) endpoint and cannot resolve or route it (the internal-endpoint certs carry no FQDN SAN). This is the root of the gss/retrofit amphora-pipeline constraint recorded in D-021. The proper fix (cloud-internal DNS + FQDN-valid certs, or charms consuming internal endpoints) is a Roosevelt item.

**Status:** Decided (v1). Reconstructed into this doc from the deploy record (no standalone D-019 file).

**Related:** D-008 (DNS architecture), D-021 (amphora-pipeline consequence), D-011 (acceptance bar amended).

---

## D-020: Dual provider + metal API VIPs on clustered charms

**Decision:** Every clustered OpenStack API application (keystone, glance, nova-cloud-controller, neutron-api, cinder, placement, barbican, octavia, openstack-dashboard, magnum, vault) is configured with BOTH a provider VIP and a metal VIP, as a space-separated pair: `vip: "10.12.4.X 10.12.8.X"` (Option B).

**Rationale:** with a provider-only VIP, `charms_openstack/ip.py:resolve_address(INTERNAL)` returns `None` and raises `ValueError`, breaking `identity-service-relation-joined` (and the analogous internal-endpoint registration on every clustered API charm). Supplying a metal-network VIP alongside the provider VIP gives `resolve_address` an internal address to return, and keeps east-west service traffic on the metal network rather than the provider network.

**Status:** Decided (v1). Reconstructed into this doc from the deploy record (no standalone D-020 file).

**Related:** D-003 (network architecture), D-002 (channels).

---

## D-021: Octavia amphora image pipeline on the no-DNS dual-endpoint deploy

**Decision:** build the amphora image with the charm-native `octavia-diskimage-retrofit` set `use-internal-endpoints: true`, seeded by a manually uploaded stock Ubuntu base image carrying the five Glance properties the retrofit reads (architecture, os_distro, os_version, version_name, product_name). Park `glance-simplestreams-sync` for the amphora pipeline. The amphora image is `image-format: raw`, tagged `octavia-amphora` to match octavia's `amp-image-tag`.

**Root cause:** on the dual-endpoint, no-DNS topology (D-019), metal-net catalog-callers (gss + its retrofit subordinate) cannot reach Glance: the public Glance FQDN does not resolve/route from the metal net, and the internal-endpoint cert carries no FQDN SAN (so an `/etc/hosts` FQDN->metal-VIP mapping fails TLS). gss `use-internal-endpoints` steers only its Keystone auth to internal; its glance/swift clients still use the public FQDN and there is no further charm-native lever -- a charm gap on the no-DNS topology. The retrofit's `use-internal-endpoints` lever DOES cover its build path, so it is the charm-native amphora builder here.

**Status:** Decided + validated end-to-end (v1): the retrofit, over internal endpoints, reads the seeded base and writes the amphora; gss parked; octavia + subordinates active/idle.

**Roosevelt:** cloud-internal DNS + FQDN-valid certs removes the manual seed and fixes gss end to end.

**Related:** D-007 (Octavia inclusion), D-019 (no-DNS root cause).

---

## D-028: Defer the CAPI v1beta2-contract cutover (deploy the single-contract v1beta1 stack)

**Decision:** defer adopting the CAPI v1beta2-CONTRACT generation until upstream ships it correctly for this path; deploy the clean single-contract v1beta1 stack now.

**Context:** while grounding the (then-current) Canonical CK8s workload chart, the chart referenced control-plane/bootstrap kinds at apiVersion v1beta1 while the pinned provider served them only at v1beta2 (DOCFIX-022). The broader question -- is the v1beta2-contract generation available and correct for long-term support on this path -- resolved to "not yet."

**Status:** Decided (v1). The CK8s-chart-specific particulars were subsequently retired when D-031 replaced the direct-CAPI CK8s path with Magnum + the azimuth kubeadm charts; the single-contract principle carries forward, and D-042 later made the driver-side contract axis concrete.

**Builds on:** D-022 / D-023 (do-07-era CAPI/CRD work). **Related:** D-031, D-042.

---

## D-029: Defer Keystone SSO (k8s-keystone-auth) to Roosevelt

**Decision:** Keystone SSO for the workload clusters (the chart's `k8s-keystone-auth` addon) is deferred to the next deployment and folded into the Roosevelt cloud-internal-DNS + trusted-cert foundation. v1 workload clusters run the Kubernetes Dashboard with standard token auth; the `k8sKeystoneAuth` addon stays OFF; SSO is not validated on v1.

**Rationale:** enabling it on v1 would produce a non-functional SSO path (TLS failure to the private-CA Keystone endpoint) plus apiserver webhook error noise -- a checked box that does not work -- and forcing it would require forking the addon or fighting CAAPH, neither of which carries forward to Roosevelt.

**Finding (verified 2026-06-05):** k8s-keystone-auth 1.5.1 exposes no keystone-CA option, so it cannot trust a private-CA Keystone endpoint.

**Status:** Decided (v1). **Related:** D-028 (same "land it on the proper foundation later" principle).

---

## D-030: Management-cluster placement -- in-cloud (superseded twice; see D-033, D-035)

**Decision (as taken 2026-06-06):** run the CAPI management plane IN-CLOUD for the v1 rehearsal (CAPI core + CAPO + cluster-api-addon-provider as VMs on the OpenStack cloud, following an Azimuth seed + HA pattern with a `clusterctl move` pivot to a self-hosted in-cloud management cluster). Out-of-cloud was recorded as a deferred alternative for Roosevelt.

**Status:** SUPERSEDED. First by D-033 (out-of-cloud Canonical `k8s`-charm on MAAS); then -- after D-033's dual-homed node hit an unfixable pod-egress fault -- placement returned in-cloud in a simpler single-homed form under D-035. Retained here for lineage.

**Related:** D-031, D-033, D-035.

---

## D-031: Cluster-creation surface + engine -- Magnum + magnum-capi-helm + azimuth kubeadm charts

**Decision:** the tenant Kubernetes service is built from three layers:
- Surface: OpenStack Magnum (`openstack coe cluster ...`), so tenants and operators manage clusters through the OpenStack API.
- Driver: the in-tree Cluster API Helm driver `magnum-capi-helm` (opendev.org/openstack/magnum-capi-helm), pip-installed into the Magnum conductor and pointed at a CAPI management cluster via `[capi_helm] kubeconfig_file`.
- Engine: the azimuth-cloud `capi-helm-charts` `openstack-cluster` chart (kubeadm-based: KubeadmControlPlane / KubeadmConfigTemplate + CAPO OpenStackCluster / OpenStackMachineTemplate + MachineDeployment), with addons (Cilium CNI, OpenStack CCM, Cinder CSI, and so on) installed by the cluster-api-addon-provider.
- Management-cluster placement: in-cloud for v1 (D-030, later refined by D-035).

**Status:** Decided. Supersedes the do-07 direct-CAPI Canonical CK8s chart path; the CK8s-chart-specific findings (DOCFIX-022 ref patch, etc.) are retired for this path.

**Related:** D-030 / D-035 (placement), D-034 (version constellation), D-036 / D-042 (driver/chart/core coherence).

---

## D-033: Management cluster -- out-of-cloud Canonical k8s-charm on MAAS (superseded by D-035)

**Decision (as taken 2026-06-07):** management cluster = a Canonical Kubernetes cluster deployed with the `k8s` / `k8s-worker` machine charms on MAAS, OUTSIDE OpenStack, made HA by the charms; CAPI layer via `clusterctl init --infrastructure openstack` + cluster-api-addon-provider, version-pinned to the capi-helm-charts release (NOT the D-022 do-07 pins); the management cluster does not run the OpenStack CCM for itself (CAPO reaches OpenStack through a `clouds.yaml` pointed at the public API endpoints); lifecycle via Juju.

**Status:** SUPERSEDED by D-035. The chosen node (capi-mgmt MAAS VM) is necessarily dual-homed (MAAS PXE on metal, API VIPs on provider), and pod egress from that multi-NIC node to the API VIPs failed (the Cilium reverse-NAT reply was mis-forwarded out the wrong NIC instead of redirected into the pod). Retained here for lineage.

**Supersedes:** D-030 (placement) + D-032 (azimuth-config tooling). **Builds on:** D-031.

---

## D-034: CAPI version constellation pinned to capi-helm-charts dependencies.json

**Decision:** pin the management-cluster CAPI constellation to the `dependencies.json` published with a chosen `capi-helm-charts` RELEASE TAG, read at deploy time on the jumphost with `jq` (dynamic lookup, no hand-picked versions). Retire D-022 "Option A" (driver 1.3.0 / CAPO v0.10.x / v1alpha6) as obsolete.

**Rationale:** the magnum-capi-helm driver does not hand-pick component versions; its own CI installs the management CAPI stack by reading the per-release `dependencies.json` and running a fixed install sequence -- that file is the single matched-and-tested set. Hand-picking fights the upstream model, and v1alpha6 has been removed from current cluster-api-provider-openstack. (At tag 0.25.1 the set is CAPI v1.13.2, CAPO v0.14.4, cert-manager v1.20.2, ORC v2.5.0, addon-provider 0.12.0, janitor 0.11.0, helm v3.17.3; appendix-B carries the as-built snapshot.)

**Status:** Adopted 2026-06-08. **Supersedes:** D-022. **Amended by:** D-042 (adds the driver<->core contract-coherence rule). **Related:** D-031, D-028 (CRD-contract note, now subsumed).

---

## D-035: Management-cluster placement -- in-cloud single-homed tenant VM

**Decision:** run the CAPI management cluster as a single-homed in-cloud tenant VM (`capi-mgmt-v2`): one NIC on the management tenant subnet (10.20.0.0/24), reached via a floating IP (per-rebuild -- this rebuild 10.12.5.103; the original 10.12.7.40 is dead -- DOCFIX-038); k8s-snap (channel `1.32-classic/stable`), Cilium CNI; not CAPI-self-managed (no `clusterctl move`).

**Rationale:** D-033's out-of-cloud node was necessarily dual-homed and its pod egress to the OpenStack API VIPs failed -- the Cilium reverse-NAT reply was emitted back out the second NIC instead of being redirected into the pod via `cilium_host` (a multi-NIC reverse-path fault; the `k8s` charm exposes too few Cilium annotations to repair it). A single-homed VM removes the second NIC and the fault entirely. The single-NIC pod-egress premise was then proven by the Phase 4 hard gate (an agnhost pod TCP probe to the Keystone VIP 10.12.4.50:5000 returning exitCode 0).

**Status:** Adopted 2026-06-08; pod-egress premise validated. **Supersedes:** D-033 (revisits D-030 in simpler form); also retires the Layer B topology of **D-007** and the bootstrap mechanism of **D-017** (k3s-on-MAAS + `clusterctl move` -> in-cloud single-homed VM, no pivot). **Unaffected:** D-031, D-034.

**Trade-off:** a single-node management cluster is a SPOF with no self-heal -- see D-041 (manual-start policy) and D-040 (the OOM that surfaced it).

---

## D-036: magnum-capi-helm driver / chart / CAPO coherence (resolved)

**Decision / correction:** a mid-session "rebuild Phase 5 on chart 0.10.1" framing -- premised on the GA driver (1.3.0) emitting the v1alpha6 OpenStackCluster CRD and clashing with the modern v1beta1 stack -- is WRONG and is retired. Chart 0.10.1 is the retired v1alpha6 path that D-034 superseded; rebuilding on it would have reversed D-034.

**Verification:** the 1.3.0 driver is api_version-AGNOSTIC (driver.py has zero v1alpha6/v1beta1/apiVersion references; it helm-installs the chart and watches the CAPI `Cluster`, never writing OpenStackCluster directly). The OpenStackCluster apiVersion is set by the CHART: chart 0.25.1 emits `infrastructure.cluster.x-k8s.io/v1beta1`, matching the installed CAPO v0.14.4. The driver's built-in default chart is 0.10.1 (the v1alpha6-era chart); overriding `default_helm_chart_version` to 0.25.1 yields v1beta1. The "1.3.0 emits v1alpha6" claim was true only of the driver's DEFAULT chart, not of the driver pinned to chart 0.25.1.

**Status:** Resolved 2026-06-08. Implements D-031 Phase 3 under the D-034 constellation. NOTE: a SEPARATE axis -- the driver-vs-core CONTRACT, not the chart's CRD string -- is what later required the 1.4.0 driver pin; see D-042. **Related:** D-031, D-034.

---

## D-037: [capi_helm] config persistence on the charm-managed conductor

**Decision:** keep the `[capi_helm]` section in an oslo.config drop-in directory and point the conductor at it: `/etc/magnum/magnum.conf.d/00-capi-helm.conf` (0644, no secrets; it references the 0600 kubeconfig by path), with magnum-conductor launched with `--config-dir /etc/magnum/magnum.conf.d` so oslo.config merges the drop-in over the charm-rendered `magnum.conf`. The charm manages neither the .conf.d directory nor the launch extension, so this survives charm hooks and reproduces on Roosevelt.

**Problem:** the magnum charm (2024.1/stable rev 70) re-renders `magnum.conf` wholesale on hooks and exposes no conf-override option, so a `[capi_helm]` section written into `magnum.conf` would be clobbered.

**Mechanism (load-bearing correction):** the conductor's ExecStart is NOT a direct binary -- it is `/etc/init.d/magnum-conductor systemd-start` (an LSB init script wrapped by systemd), so a systemd ExecStart drop-in appending `--config-dir` is inert (the flag reaches the init script as an ignored positional). The adopted method instead creates `/etc/default/magnum-conductor` (0644; the charm does not manage it) containing `DAEMON_ARGS="$DAEMON_ARGS --config-dir /etc/magnum/magnum.conf.d"`; the init script sources `/etc/default/$NAME` after setting the base `DAEMON_ARGS`, then runs `exec $DAEMON $DAEMON_ARGS`. Verify behaviorally with `/etc/init.d/magnum-conductor show-args` and `ps -ww -C magnum-conductor -o args` (not string-presence).

**Status:** Adopted 2026-06-08 (mechanism revised mid-implementation). **Residual:** breaks silently if a future charm hook writes `/etc/default/magnum-conductor` -- detect via the same show-args/ps check. **Related:** D-031 Phase 3, D-036.

---

## D-039: Magnum mints per-cluster app-creds carrying the trustor's roles (grant load-balancer_member)

**Status:** ACCEPTED 2026-06-09 (applied in phase-06; asserted in phase-08 prereqs). Cited by phase-06 (DOCFIX-036 grant), phase-08, and appendix-A; recorded here to close the dangling reference.

**Context:** the magnum-capi-helm service path uses NO static, pre-provisioned application credential. At cluster-create magnum mints a per-cluster Keystone application credential from a trust, and that app-cred carries the TRUSTOR's roles FROZEN at mint time, delegated unfiltered. The trustor is the identity that creates the cluster in the capi-mgmt project (admin@admin_domain in v1).

**Decision:** the trustor must hold `load-balancer_member` (plus `member` and `reader`) on the capi-mgmt project BEFORE any cluster is created, so every minted app-cred carries Octavia authority. A trustor holding only `member` mints a frozen app-cred that 403s when CAPO queries Octavia to confirm LB state -- the workload cluster then wedges at API-LB provisioning, and a stuck-delete 403s the same way (appendix-A). The grant is idempotent (member + load-balancer_member + reader) and applied in phase-06.

**Roosevelt implication:** whichever identity creates Magnum/CAPI clusters must carry `load-balancer_member` + `reader` on the cluster project; `member`-only is a latent 403. There is no static `capo` user/app-cred to provision -- that pattern was retired with the per-cluster mint.

**Related:** D-031 / D-036 (driver/engine surface), D-046 (the trustee domain the trust resolves against), appendix-A (D-039 + stuck-delete recovery).

---

## D-040: Raise nova-compute reserved-host-memory on the hyperconverged hosts

**Decision:** set `nova-compute reserved-host-memory` to 8192 MB (from the default 512) so Nova placement accounts for the non-Nova memory co-located on each hyperconverged host. Charm config -> survives redeploy.

**Trigger / root cause:** during the first end-to-end Magnum workload-cluster create, openstack1 hit the kernel OOM-killer (no reboot; single boot since 2026-06-03) and killed a tenant qemu worker VM. The host co-locates nova-compute AND roughly 6 GiB of services invisible to Nova placement (mysqld [innodb-cluster member] ~2.9G, ceph-osd + ceph-mon ~1.2G, neutron workers ~0.7G, nova/apache/cinder/ovs ~1.4G) while Nova reserved only the default 512 MB; under the resulting memory pressure the host swap-thrashed (an ovsdb inactivity-probe storm made the workload API and Juju agent look "down" when the host was in fact thrashing, not down).

**Status:** Adopted + APPLIED 2026-06-09. **Related:** D-035 (the mgmt-VM SPOF the OOM hit), D-041.

---

## D-041: Non-HA deployments default to manual start

**Decision:** non-HA deployments default to MANUAL START -- no automatic VM power-on / auto-recovery is configured by default. Any non-HA deployment must be documented as non-HA, with the rationale that manual-down surfaces incidents (auto-restart masks capacity/health defects). Auto-recovery is an explicit, out-of-band exception, never the silent default.

**Trigger:** after the openstack1 OOM (D-040), CAPI's MachineHealthCheck self-healed the workload worker VMs automatically, but the single-node management VM (capi-mgmt-v2, D-035) was OOM-killed and stayed SHUTOFF -- it does not self-heal or auto-restart, which silently broke magnum reconcile/health and left workload nodes with the CAPI uninitialized taint until it was started by hand. The cost (downtime) was real, but the manual-down is also what forced the investigation that found the OOM root cause headed for Roosevelt.

**Status:** Adopted 2026-06-09 (policy/governance). **Related:** D-035 (the SPOF), D-040 (the OOM).

---

## D-042: magnum-capi-helm driver must be contract-coherent with the CAPI core

**Decision (amends D-034):** the magnum-capi-helm driver pin (Layer B) MUST be contract-coherent with the CAPI core that `dependencies.json` installs (Layer A). When the Layer-A lockfile is a v1beta2-contract core (CAPI v1.13), the driver pin must be a build that understands v1beta2 references; verify this intersection at deploy.

**Symptom / root cause:** capi-test-1 reached CREATE_COMPLETE with every real component healthy (3 Ready nodes, Calico, CCM/CSI/CoreDNS, API LB ACTIVE), yet magnum reported `health_status = UNHEALTHY` deterministically -- only the `infrastructure` sub-check failed ("Infrastructure resource not found"). The 1.3.0 driver reads `apiVersion` off the Cluster's `spec.infrastructureRef`, but under the v1beta2 contract that ref is version-less, so the health GET resolves nothing. The create path is unaffected (the chart templates the resource versions) -- a cosmetic health false-negative. The governing axis is the CAPI CONTRACT a provider implements toward core, not the CRD apiVersion string (per D-028); rolling back to a v1beta1 core would mean pinning an EOL CAPI for a Roosevelt rehearsal -- the wrong direction.

**Fix:** pin a driver build carrying the per-kind `[capi_helm] api_resources` override and set it so the health lookups use the served versions. As of 2026-06-09, D-042 recorded this capability as UNRELEASED (development series only; released line then 1.1.0/1.2.0/1.3.0), with the interim = a current-series commit for the testcloud and a released-tag pin deferred to Roosevelt.

**Subsequent update (driver-fix work):** the released `magnum-capi-helm==1.4.0` was then confirmed to ship the `api_resources` feature, so the released-tag pin is now available -- v1 pins 1.4.0 with an explicit `api_resources` and targets `health_status = HEALTHY` (installed in phase-07; as-built in appendix-B). This replaces D-042's interim dev-commit path.

**Operational caveat (while any health false-negative persists):** do NOT wire magnum auto-healing to `health_status` -- a persistent false UNHEALTHY could misfire; CAPI MachineHealthCheck handles node healing independently.

**Status:** Adopted 2026-06-09; fix landed via the 1.4.0 pin. **Amends:** D-034. **Related:** D-028 (the contract axis made concrete), D-031, D-035.

---


From prior bundle review work — these are anti-patterns:

- `magnum-shared-db` missing colon — causes a relation endpoint syntax error, deploy-blocking. Bundle must use `- - magnum:shared-db` (with the colon).
- Empty `osd-devices` YAML anchor referenced by multiple ceph-osd applications.
- `ovn-chassis` binding `overlay-suffix` — invalid binding name. Correct value is `data`.
- GUI annotation collision between NUMA-split ceph-osd apps (not applicable to testcloud since we don't NUMA-split, but flagged for Roosevelt).
- Hardcoded NIC name in `bridge-interface-mappings`. Use MAC where possible.
- `openstack -f value` column ordering — column order is not guaranteed; use `-c <column> -f value` for single-column output.
- Snap confinement: `openstackclients` snap has home-only interface; commands cannot read paths under `/tmp`. File paths must resolve under `$HOME`.
- Non-ASCII characters in `local_settings.d` overrides cause silent daemon failures in Horizon.

---


## Change log

| Date       | Change                                                                                                                                                                                                                            | Reference                                            |
| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
| 2026-05-22 | Initial document captured                                                                                                                                                                                                         | Caracal rebuild planning session                     |
| 2026-05-22 | D-015 v1/v2 fork added; D-004 and D-004a marked v2-scope; D-016 IPv4 tenant pool hybrid model added; D-014 updated with new repo name                                                                                             | v1/v2 fork session                                   |
| 2026-05-22 | D-017 CAPI bootstrap full-rebuild lifecycle added; D-018 MAAS-release-direct teardown added. D-013 marked superseded by D-018. D-007 Layer B updated to reference D-017 and `runbooks/04a-capi-bootstrap-cluster.md`.             | Teardown planning + handoff session                  |
| 2026-05-22 | D-002 hacluster row added (channel `2.4/stable`) per Canonical Charm Delivery table, verified against Charmhub. D-007 Layer B driver pin updated: `stackhpc/magnum-capi-helm` v0.13.0 → `openstack/magnum-capi-helm` 1.1.0 (PyPI; stackhpc fork archived Dec 2024). | Caracal channel verification + driver pin correction |
| 2026-05-22 | D-007 Layer B kubeconfig target corrected: bootstrap k3s → workload cluster (post-pivot per workstream 3b mandatory `clusterctl move`). CAPI mgmt plane paragraph updated accordingly. | Workstream 3 cleanup (post-pivot semantics) |
| 2026-05-29 | D-019 (Designate deferral) and D-020 (dual provider+metal API VIPs) recorded as already-taken; folded into this doc in the 2026-06-09 consolidation. | Deploy execution / handoff |
| 2026-05-30 | D-021 Octavia amphora pipeline (charm-native retrofit over internal endpoints; gss parked) added. | Octavia enablement |
| 2026-06-05 | D-028 (defer v1beta2-contract cutover) and D-029 (defer Keystone SSO) added. | CAPI path research |
| 2026-06-06 | D-030 (mgmt-cluster placement: in-cloud) and D-031 (Magnum + magnum-capi-helm + azimuth kubeadm engine) added. | Magnum/CAPI surface decisions |
| 2026-06-07 | D-033 (mgmt cluster: out-of-cloud k8s-charm on MAAS) added; supersedes D-030 and D-032. | Mgmt-cluster shape |
| 2026-06-08 | D-034 (CAPI constellation pinned to dependencies.json; supersedes D-022), D-035 (in-cloud single-homed mgmt VM; supersedes D-033), D-036 (driver/chart/CAPO coherence resolved), D-037 ([capi_helm] via /etc/default DAEMON_ARGS) added. | In-cloud mgmt pivot |
| 2026-06-09 | D-040 (reserved-host-memory 8192), D-041 (non-HA manual-start policy), D-042 (driver<->core contract coherence; 1.4.0 pin) added. | OOM incident + driver fix |
| 2026-06-09 | D-019..D-042 consolidated into this document (15 decisions). Existing D-001..D-018 left intact (em-dash style preserved); the new entries are ASCII. | Repo sanitation / doc refresh |
| 2026-06-17 | D-044 (Horizon secure-cookie override) + D-045 (haproxy confirmed-LOADED) folded from the changes-doc; D-046 (magnum trustee-domain setup) + D-047 (keystone v2.0 render bug / v3 drop-in) merged and renumbered from the 06-17 addendum (its "D-044/D-045"); D-048 (stage-and-verify canonical image seed, supersedes web-download) + D-049 (D1: kube v1.34.8 / capi-k8s-v1-34) added; D-042 amended (FINDING-5: UNHEALTHY is a <=1.3.0 false-negative, HEALTHY on 1.4.0); D-050 (PROPOSED: keystone policyd-override) recorded. | End-of-deploy runbook sweep |
| 2026-06-18 | D-039 (Magnum per-cluster app-cred roles; grant load-balancer_member) recorded to close a dangling reference (previously cited only in phase-06/08, bundle, appendix-A). D-007 + D-017 annotated as superseded by D-035 / D-037 / D-042 (in-cloud mgmt VM; /etc/default config-dir graft; 1.4.0 driver) -- historical bodies retained. | Pre-commit audit (runbook sweep) |

<!-- patchset-20260610-decisions-addendum -->

---

## D-042 -- AMENDMENT (2026-06-10): mechanism evidence + signature taxonomy

Evidence gathered during the 2026-06-10 recovery session strengthens and
sharpens D-042:

1. Steady-state cosmetic signature, verbatim: health_status=UNHEALTHY with
   reason {'cluster': 'Ready', 'infrastructure': 'Infrastructure resource not
   found.', 'controlplane': 'Ready', 'nodegroup': 'Ready'}. "Resource NOT
   FOUND" (a lookup/contract miss) rather than "resource unhealthy"
   corroborates the driver-vs-CAPO API-contract coherence diagnosis.
2. The driver health maps to the cluster-level v1beta2 Available aggregation.
   This session also observed a REAL Available=False (InfrastructureReady
   failed: kube-api LB reconcile timeout during a mgmt-VM cold start, LB left
   provisioning=ERROR). Two distinct mechanisms therefore share the UNHEALTHY
   presentation -- the operator cannot distinguish real from cosmetic without
   reading the reason field -- which strengthens the case for the staged
   forward-fix (magnum-capi-helm-driver-fix runbook).
3. Signature taxonomy for operators: reason EMPTY = conductor cannot reach the
   mgmt API (VM parked/down; NOT D-042); reason populated all-'Ready' with
   infrastructure 'Infrastructure resource not found.' = D-042 cosmetic;
   reason citing an LB/infrastructure failure = real, check Octavia first
   (ops-capi-recovery Step 4/5).

## D-043 -- PROPOSED: tenant-VM auto-resume policy (resume-guests-state-on-host-boot)

STATUS: PROPOSED (decision pending). Recorded 2026-06-10.

QUESTION: should nova-compute `resume-guests-state-on-host-boot=true` be set,
so tenant VMs (including the in-cloud CAPI management VM, D-035) return
automatically after host reboots?

TENSION: D-041 ("a non-HA component staying down is a signal to investigate,
not a nuisance to auto-mask") was written for control-plane charm services.
Tenant VMs are a different class: on Roosevelt, customers will expect their
VMs back after host maintenance, and auto-resume is the industry norm for
that class. Cost of the manual policy observed 2026-06-10: a deliberately
parked mgmt VM was mistaken for an outage and consumed roughly two hours of
diagnosis before the stop was traced to an API action.

OPTIONS:
  (a) Enable auto-resume cloud-wide + monitoring/alerting on VM state, so
      "down" remains a signal without being an outage. RECOMMENDED for
      Roosevelt; candidate for v1 redeploy as well.
  (b) Keep manual start for v1 (preserves D-041 discipline on the testcloud),
      explicitly record auto-resume as the Roosevelt setting.
Note: the restart procedure's failure-mode table already references the config
key for SHUTOFF guests; whichever option is chosen, align that table, this
decision, and the bundle/runbook with each other.

<!-- patchset-20260617-sweep-decisions : end-of-deploy runbook sweep -->

---

## D-044: Horizon Secure-cookie override for internal-HTTP dashboard access (DOCFIX-030)

**Status:** Adopted 2026-06-17 (PER-REBUILD; phase-03 Step 3.3). Resolves the mislabeled "D-043" tag used for this item in earlier phase-03/changes-doc drafts -- D-043 is the tenant-VM auto-resume decision.

**Decision:** the jumphost reaches Horizon over a plain-HTTP reverse-proxy leg, but the dashboard sets `SESSION_COOKIE_SECURE`/`CSRF_COOKIE_SECURE=True`, so the browser drops the session/CSRF cookies over HTTP and login fails. Apply a Django settings override on the openstack-dashboard leader (`_99_internal_http_cookies.py`, setting `SESSION_COOKIE_SECURE=False` + `CSRF_COOKIE_SECURE=False`) to allow cookie flow over the internal HTTP leg. A TESTCLOUD accommodation of the no-DNS / no-FQDN-cert posture.

**Trade-off / Roosevelt:** disabling Secure cookies is acceptable only because the proxy leg is internal and the cloud has no public DNS / FQDN-valid cert. The Roosevelt root-fix is cloud DNS + FQDN-valid certs end-to-end (which also fixes gss and the nginx proxy_ssl_name handling); then this override is removed. Self-signed-client-TLS approaches are NOT part of v1.

**Related:** DOCFIX-030 (phase-03 Step 3.3), D-043 (distinct -- auto-resume).

## D-045: Charm-rendered haproxy config must be confirmed LOADED, not just rendered (DOCFIX-031)

**Status:** Adopted + APPLIED 2026-06-11; re-verified 2026-06-16 post phase-05.

**Decision:** after the vault/TLS cert cascade settles, confirm every unit's haproxy is actually checking its backends over the freshly-rendered SSL config -- by a functional probe of haproxy's OWN backend state (admin-socket `show stat`, grep `,DOWN,`), NOT by `juju status`. Reload (not restart) any unit whose running haproxy predates the `check-ssl` re-render.

**Root cause (confirmed, not refined to a check-config defect):** nova-cloud-controller haproxy was not reloaded after the cert cascade, so its health checks ran plaintext against the now-SSL backend port and marked the nova-api backends DOWN -- while juju stayed active/idle (juju is BLIND to per-backend haproxy state). An 8s wire capture showed the checks switch to TLS after reload; both backends returned UP/L7OK/200. The reload is a real fix, not a band-aid.

**Status note:** the surfaced symptom was nova-api EOF / 503 behind a green juju. phase-03 Step 3.1 now gates a zero-DOWN sweep cloud-wide.

**Related:** DOCFIX-031 (phase-03 Step 3.1; appendix-A), D-046 / D-047 (the separate magnum-keystone incident).

## D-046: Magnum trustee-domain setup is a REQUIRED, asserted post-deploy step

**Status:** ACCEPTED 2026-06-17. Recorded in design-decisions-addendum-20260617.md as "D-044"; renumbered to D-046 here per the 06-17 reconciliation (D-044/D-045 were taken by the Horizon/haproxy decisions). Matches the rootcause doc + phase-08 handoff prompt.

**Context:** all `openstack coe ...` ops returned 403 ("Keystone client authentication failed"); magnum-api.log showed keystoneauth1 401 on every request since 2026-06-16. Root cause: the keystone domain `magnum` and user `magnum_domain_admin` that magnum.conf `[trust]` references did not exist. `magnum/common/policy.py:130` resolves `trustee_domain_id` on EVERY policy-enforced request (driver-agnostic), authenticating as the trustee domain admin; with the domain/user absent that is a hard 401 -> every coe op 403.

**Cause:** the magnum charm action `domain-setup` is MANUAL, not automatic; magnum reports active / "Unit is ready" regardless of whether it has run. The 2026-06-11 teardown/redeploy rebuilt keystone with fresh domains but the runbook did not re-run `domain-setup`.

**Decision:** `domain-setup` is a REQUIRED, ASSERTED post-deploy step on every (re)deploy, after the magnum + identity-service relation is up and BEFORE magnum is declared functional / before phase-08: (1) `juju run magnum/leader domain-setup`; (2) assert `openstack domain show magnum` and `openstack user show magnum_domain_admin --domain magnum` both succeed; (3) gate `openstack coe service list` (must return the conductor row, no 403). magnum's active/ready status MUST NOT be treated as evidence the trustee domain exists.

**Roosevelt:** carry as an explicit runbook step + assertion; consider upstreaming a charm change so domain-setup runs automatically, or so the charm surfaces "trustee domain not set up" instead of reporting ready.

**Related:** D-047 (the v2.0 render bug found in the same incident, but NOT the cause), D-031 / D-037 (magnum surface).

## D-047: keystone auth_version v2.0 charm-render bug -- keep the v3 drop-in

**Status:** ACCEPTED (keep) 2026-06-17. addendum "D-045" -> D-047 per the 06-17 reconciliation.

**Context:** the magnum charm template renders `auth_version = v2.0` due to a type bug (the keystone interface delivers `api_version` JSON-decoded as int 3; the template does a strict string compare `3 == "3"` -> False -> v2.0). Full analysis in incident-magnum-keystone-v2-rootcause-20260617.md.

**Finding:** the v2.0 render was NOT the cause of the coe 403 (that was D-046). On this deployment v2.0 is cosmetic -- magnum's `domain_admin_auth` rewrites v2.0 -> v3, v3 is discovered from the unversioned `auth_url`, and incoming token validation worked throughout.

**Decision (Jesse, 2026-06-17):** KEEP the magnum.conf.d v3 drop-in. v2.0 is the provably wrong value for Caracal (which does not serve v2.0); the drop-in forces v3 via the same config-dir mechanism as the D-037 conductor graft (no charm-file drift, survives re-render). Architectural correctness over minimize-delta, even though the drop-in did not unblock coe.

**As-built:** `/etc/magnum/magnum.conf.d/50-keystone-v3-override.conf` (auth_version=v3 + www_authenticate_uri/auth_url v3 in `[keystone_authtoken]` and `[keystone_auth]`); `/etc/default/magnum-api` DAEMON_ARGS adds `--config-dir /etc/magnum/magnum.conf.d` (mirrors D-037 for the standalone magnum-api).

**Roosevelt:** carry the v3 fix as a drop-in/overlay; upstream the template type bug (and the separate identity-service departed-hook IndexError crash documented in the rootcause doc).

**Related:** D-046, D-037.

## D-048: Stage-and-verify is the canonical image-seed method (supersedes web-download)

**Status:** Adopted 2026-06-17 (operator-approved). Supersedes the 2026-06-16 "web-download canonical" ruling.

**Decision:** seed ALL glance images (octavia amphora base, the noble mgmt image, the workload kube image) by STAGE-AND-VERIFY: download to `$HOME` (snap-readable; NOT /tmp), verify the file against the published checksum (azimuth-images manifest sha512 for kube images; ubuntu cloud-images SHA256SUMS for noble), then `openstack image create --file [--import]`. Web-download is retained as a TESTED ALTERNATIVE only (appendix-A).

**Rationale:** (1) FINDING-3 -- glance's web-download plugin fetches with urllib (UA `Python-urllib/3.x`) and the azimuth CDN 403s that UA, so web-download is UNUSABLE for kube images (202-accept, then stuck `queued`); (2) web-download cannot checksum-verify the fetched file (the CDN redirect strips the digest) -- weaker provenance; (3) stage-and-verify is one provenance-verified path cloud-wide -- less Roosevelt delta. CORRECTION-1: a plain `--file` PUT stores qcow2 (boots fine); `--import` runs glance image-conversion -> raw (Ceph fast-clone alignment).

**Roosevelt:** unify on stage-and-verify; the longer-term target remains gss-from-a-controlled-mirror once cloud DNS + FQDN certs land.

**Related:** D-021 (amphora pipeline), FINDING-3 (appendix-A image-seeding), phase-05 / 06 / 08.

## D-049: Workload kube image bumped v1.32.13 (EOL) -> v1.34.8 (D1)

**Status:** Adopted 2026-06-17. Procedure target; re-validation on v1.34.8 follows the stage-and-verify seed.

**Decision:** the workload-cluster kube image moves from the EOL ubuntu-jammy-kube-v1.32.13 to ubuntu-jammy-kube-v1.34.8 (azimuth-images 0.28.0, build 260518-1604; sha512 7efde485...760bdb3), and the cluster template is renamed `capi-k8s-v1-32` -> `capi-k8s-v1-34`. v1.34.8 is mature with good runway and within CAPI v1.13.2 support. The management cluster's OWN k8s stays at v1.32.13 (k8s-snap 1.32-classic) -- this bump is the workload image only.

**Note:** the 2026-06-09 D-011 acceptance ran on v1.32.13; the v1.34.8 image is seeded via D-048 stage-and-verify, and D-011 re-validation on v1.34.8 is the pending acceptance item. The template now pins `--network-driver calico` (DOCFIX-032).

**Related:** D-031 / D-034 (CAPI surface), D-048 (seed), DOCFIX-032 (CNI pin), phase-08.

## D-042 -- AMENDMENT (2026-06-17): FINDING-5 -- rescope to "<= 1.3.0" (HEALTHY on 1.4.0)

The D-042 cosmetic `health_status = UNHEALTHY` false-negative is a property of driver builds <= 1.3.0 (the v1beta2 contract-ref mismatch: those builds read `apiVersion` off the infrastructureRef, which CAPI v1.13's v1beta2 contract no longer carries). The RELEASED 1.4.0 driver carries the `api_resources` override and reports `health_status = HEALTHY` against the CAPI v1.13.2 / CAPO v0.14.4 stack (confirmed this rebuild). FINDING-5: D-042 is therefore CLOSED for v1 -- the UNHEALTHY caveat applies only to the historical <=1.3.0 holding state, NOT to the as-built 1.4.0 pin. Auto-heal is still NOT wired to health_status (CAPI MachineHealthCheck heals independently).

## D-050: RESOLVED / CLOSED (2026-07-06) -- keystone `use-policyd-override=true` with no policy zip (FINDING-1)

**Status:** RESOLVED / CLOSED 2026-07-06 (recorded 2026-06-17 as an unruled finding; operator ruling: resolved via D-051 option (b); see the RESOLVED amendment near the end of this file). Scan note: this line deliberately avoids the unruled-status keywords so ledger-scan closes the block; the history arrow lives in the amendment.

**Question:** keystone is configured with `use-policyd-override=true` but no policy override zip is supplied. This is currently a no-op (no custom policy applied), but the flag advertises an override capability that does not exist -- a latent footgun (a future operator may assume policy is being enforced, or a stray zip could silently change authz).

**Options (unresolved):** (a) set `use-policyd-override=false` for v1 (the override is unused) and revisit when a real policy is needed; (b) keep true and supply an explicit, reviewed policy zip; (c) leave as-is and document the no-op. No decision made -- recorded as an open point to rule on (cf. D-043, also pending).

**Related:** D-029 (Keystone SSO deferral), FINDING-1.

## D-051: Tenant identity self-service via the SCS Domain Manager policy (resolves D-050)

**Status:** Adopted 2026-06-23 (design; pre-redeploy). Implementation staged (`domain-manager-policy.yaml`), behavioral acceptance pending the bindings+policy redeploy (gate G3). RESOLVES D-050 via its option (b).

**Context / why:** the commercial multi-DC end goal requires Model-2 tenant self-service: a tenant domain-admin must create/manage users, projects, and role assignments WITHIN THEIR OWN DOMAIN, without operator involvement and without escaping the domain. The plain `admin` role cannot do this: on Keystone it is architecturally NOT domain-confinable (a domain/project-scoped `admin` can escalate beyond its boundary -- upstream hard-coded limitation). This is the root cause behind the unreliable Horizon domain-admin experiments (admin-on-domain + OPENSTACK_KEYSTONE_PREFER_DOMAIN_TOKEN scope-switch breakage).

**Decision:** implement the **SCS Domain Manager persona** (SCS scs-0302). A tenant domain-admin is a user holding the **`manager` role on their domain** (the `manager` role already exists on this cloud). Domain-confined identity self-service is granted by a **Keystone API policy override** shipped via the keystone charm's `policyd-override` resource (the mechanism `use-policyd-override=true`, already set, expects). The policy is `domain-manager-policy.yaml`, adapted VERBATIM from the canonical SCS reference (SovereignCloudStack/standards, scs-0302-w1-domain-manager-implementation-notes.md), Section A (base_* defaults) + Section B (Domain Manager extensions).

Key policy properties (anti-escalation + fit):
- `is_domain_manager` = `role:manager`; every domain-manager grant is gated on a `token.domain.id == target.*.domain_id` match -> strictly domain-confined.
- `is_domain_managed_role` (the roles a manager may assign/revoke) = **member + load-balancer_member ONLY**. `admin` is NEVER in this list (SCS hard constraint; the anti-escalation guarantee). `load-balancer_member` is included because the Magnum apiserver-LB tenant flow requires it (cf. D-039). `reader` and `manager` deliberately NOT added (least privilege; designation authority stays operator-side -- a domain manager cannot propagate domain-manager rights).
- Every rule falls through to `or rule:base_* or admin_required`, so cloud-admin retains all authority and service-user auth is unchanged.

**Why this approach (and NOT enforce_scope):** Charmed OpenStack deliberately does NOT implement Secure RBAC -- the keystone 2024.1/stable charm exposes NO `enforce-scope`/`enforce-new-defaults` option (verified live: both keys "not found"), and Canonical sets them False across the charms for upgrade safety. SCS scs-0302 states scope enforcement is NOT necessary for the Domain Manager persona on 2024.1 or below. So the persona is achieved purely by the policy override + `manager` role. Blast radius is LOW and bounded: the charm guarantees policy overrides alter ONLY tenant/user permissions, NEVER service users (keystone/glance/nova stay charm-managed). This is materially safer than a cloud-wide scope flip -- which is why the policy change is acceptable to combine with the explicit-bindings change in one redeploy.

**Mechanism / rollback:** `zip overrides.zip domain-manager-policy.yaml` -> `juju attach-resource keystone policyd-override=overrides.zip` (use-policyd-override already true). Lands at `/etc/keystone/policy.d/`; keystone restarts. `juju status` shows `PO:` on success, `PO (broken):` if YAML is unparseable (atomic -- whole override discarded; this is the current empty-resource state D-050 flagged). Rollback = `juju config keystone use-policyd-override=false` (reverts to charm defaults; the Juju resource cannot be deleted but disabling is sufficient).

**CRITICAL -- validation is YAML-only, not semantic:** the charm validates only that the override is parseable YAML; a misspelled rule key PASSES and silently no-ops. Acceptance MUST therefore be BEHAVIORAL (same "reports OK while broken" trap as D-046), never "PO: shows active." Gate G3: as a domain-scoped `manager`, create user/project + assign member/load-balancer_member within own domain PASS; assign `admin` or touch a different domain DENY; cloud-admin ops all still PASS.

**base_* alignment (open verification, non-blocking):** the base_* rules in `domain-manager-policy.yaml` are the SCS template's upstream-recent defaults. They MUST be diffed against THIS deploy's live defaults (`oslopolicy-policy-generator --namespace keystone` on keystone/leader) and any divergent line replaced before the policy is treated as final. The file is shippable for behavioral testing as-is; the diff hardens base_* fidelity. [LIVE-READ PENDING]

**Roosevelt / multi-DC:** the override is a single portable file replicated to every DC's keystone with zero per-DC bespoke work -- good "minimize delta to Roosevelt" properties. **Version caution (HARD):** this is the 2024.1 transitional (policy-based) implementation. The native Domain Manager persona ships in 2024.2 (Dalmatian); on any future upgrade to 2024.2+ this policy override MUST BE REMOVED (it conflicts with the native impl), except a customized `is_domain_managed_role` may be preserved. Track as an upgrade-time removal item.

**Resolves:** D-050 (use-policyd-override=true with no zip -> now supplies the reviewed zip, option (b)). **Related:** D-046 (magnum trustee; the admin_required fallthrough preserves it -- risk drops from the enforce_scope-era HIGH to LOW), D-039 (per-cluster app-creds carrying load-balancer_member), D-029 (Keystone SSO deferral), D-044 (Horizon secure-cookie; same charm, do not disturb). **Supersedes the in-session enforce_scope "Strategy 1" framing, which was found impossible on this charm and is abandoned.**

## D-052: Network-space inversion -- metal-internal carries all OpenStack service-to-service control

**Status:** ADOPTED 2026-06-25 (operator-approved). AUTHORITATIVE; SUPERSEDES the three conflicting D-052 drafts (D-052-decision-text.md, D-052-CORRECTED-architecture.md, D-051-D-052-decisions.md) that attached incompatible designs to this number. Implementation pending (bundle regen + MAAS cutover); ceph network config verified (unset / binding-derived, see ceph sub-decision).

**Context:** the as-deployed cloud collapses nearly all traffic onto a single `metal` space -- only `public`->provider, ceph `public`->storage, osd `cluster`->replication, and the geneve overlay->`data` are separated today. A prior agent's binding artifacts were corrupted (conflicting hallucinated definitions), so placement was re-derived from scratch against authoritative references, over a fresh pre-teardown endpoint inventory (`juju show-application` for all 50 apps; 446 endpoints).

**Decision:** adopt full three-plane isolation as the per-DC template:
- `provider-public` (10.12.4.0/22): public API endpoints + floating IPs.
- `metal-admin` (10.12.8.0/22, UNTAGGED): operator / MAAS / monitoring / the OpenStack `admin` API endpoint + the `""` default binding. DC-LOCAL (never crosses DC fiber).
- `metal-internal` (10.12.12.0/22, TAGGED VLAN): ALL OpenStack service-to-service control -- internal API, shared-db (Galera), amqp (RabbitMQ/RPC), certificates/secrets (Vault), cache, cluster peers, OVN NB/SB DB (ovsdb*), identity service-to-service, and the inter-service relations. The MAY-CROSS-FIBER plane.
- `fabric-data` (10.12.16.0/22): tenant geneve overlay (nova-compute:neutron-plugin, ovn-chassis:data, ovn-chassis-octavia:data, octavia:ovsdb-cms).
- `storage` (10.12.32.0/22): Ceph public -- RBD client + mon + ceph control relations.
- `replication` (10.12.36.0/22): Ceph OSD cluster replication.

Renames: provider->provider-public, data->fabric-data. storage/replication keep their names and RE-IP (.16->.32, .20->.36). `metal` SPLITS into metal-admin + metal-internal; metal-internal is a NEW tagged VLAN trunked to the LXD container hosts. The MAAS rename / re-IP / new-VLAN is a POST-TEARDOWN, PRE-DEPLOY step (the deploy gate, R1 risk).

**Rationale:** RHOSP network isolation puts API + RPC + database on the Internal API network, PXE/provisioning on its own network, public API + FIPs on External, and Ceph on Storage / Storage-Management (never on Internal API). The Charmed OpenStack charm model implements this with separate public/internal/admin spaces and binds `shared-db` and `amqp` to the INTERNAL space. So the inversion (DB/MQ/internal-API on the service plane, not the management plane) is the DOCUMENTED pattern, not a deviation; moving only the `internal` endpoint while leaving shared-db/amqp/certificates/peers on the management plane would invert the intended posture. Three planes = three trust levels / three failure domains, and it maps cleanly onto multi-DC: service-to-service may extend across DC fiber, MAAS/admin stays DC-local.

**Method:** a deterministic classifier (`classify_bindings.py`) over the live inventory (`live_bindings.json`): current provider/data/storage/replication -> rename or re-IP; current `metal` -> split by endpoint semantics; FAILS LOUD (NEEDS-REVIEW) on any unmatched endpoint rather than silently defaulting. Result: 446 endpoints classified, 0 unclassified. Distribution: metal-internal 252, metal-admin 160, provider-public 11, storage 17, replication 2, fabric-data 4. Artifacts: `target_bindings.json` (machine map) + `target_bindings_readable.txt` (audit). The bundle is regenerated MECHANICALLY from target_bindings.json (ruamel, comment-preserving), with a diff-identical verification gate vs HEAD.

**Judgment calls (resolved):**
- HA heartbeat (corosync/pacemaker; ha/hanode/peer-availability/pacemaker-remote, 57 eps) -> metal-internal for v1 (Canonical default; rides the NICs the service units already have; ~11 DC-local rings). DEFERRED to Roosevelt: a DEDICATED, redundant (knet dual-link) heartbeat ring -- corosync's Totem protocol is latency/jitter-sensitive and production guidance keeps it off contended/bursty planes (Proxmox: storage must never share corosync's network; RHEL HA enforces interconnect latency limits).
- Ceph control relations (ceph-mon client/mon/osd/mds/radosgw/rbd-mirror/bootstrap-source, ceph-osd:mon; 8 eps) -> storage, co-located with the ceph data path (RHOSP keeps Ceph off Internal API; keeps the storage subsystem self-contained per-DC). VERIFIED 2026-06-25: `ceph-public-network` and `ceph-cluster-network` are UNSET on both ceph-mon and ceph-osd, so the charms DERIVE the ceph networks from the `public`/`cluster` extra-bindings -- the bindings ARE the source of truth (public->storage = ceph public network; cluster->replication = ceph cluster network), not just relation-NIC steering. CONSEQUENCE: the re-IP needs NO bundle CIDR changes -- it is handled entirely at the MAAS/space level (the repo bundle carries no CIDR-valued `*-network` options; subnet literals appear only in comments). CORRECTION: ceph-mon:cluster -> replication (the ceph cluster-network extra-binding, matching ceph-osd:cluster); it had been mis-grouped onto storage. ceph-osd:secrets-storage stays metal-internal (OSD->Vault dm-crypt keys, a secrets relation).
- Live migration (nova-compute:migration) -> metal-internal for v1. DEFERRED to Roosevelt: a dedicated migration plane + QEMU-native TLS for tenant-memory transit.
- nova<->OVN-chassis integration: nova-compute:neutron-plugin KEPT on fabric-data (rename of `data`, not moved); ovn-chassis:nova-compute + ovn-chassis-octavia:nova-compute -> metal-admin (subordinate glue). SUBORDINATE SUBSET RULE: a subordinate's bound spaces must be a subset of its principal's. ovn-chassis (subordinate on nova-compute) binds `data` -> fabric-data for geneve, which is only legal if nova-compute is itself bound to fabric-data -- and nova-compute's ONLY fabric-data binding is neutron-plugin. An earlier draft of this decision moved neutron-plugin to metal-internal "for isolation"; that would strip nova-compute's fabric-data NIC and make the chassis geneve binding illegal, so it was REVERTED (caught by the bundle regenerator's fail-loud + subset verification). ovn-chassis-octavia:data -> fabric-data is likewise backed by octavia:ovsdb-cms -> fabric-data. Same pattern holds for cinder-ceph: it binds ceph -> storage, and its principal cinder already carries storage via cinder:ceph -> storage, so the subset holds.
- Vault: `access` -> metal-internal (services fetch certs/secrets here); `external` -> metal-admin (operator / unseal path); unused db/etcd/lb-provider -> metal-admin.
- Subordinate glue (ovsdb-subordinate; magnum/octavia Horizon dashboard plugins) -> metal-internal (keep related control together; low-stakes).
- Unused optional endpoints (legacy neutron plugins external-dns/infoblox/midonet/vsd/neutron-lb, rgw multisite master/primary/secondary/slave, nova-vmware/quantum/ironic/vgpu/ceilometer/ephemeral, vault db/etcd/lb-provider, hsm) -> metal-admin via the `""` default, and OMITTED from the bundle rather than enumerated. Rebind if/when activated.

**Roosevelt:** (1) dedicated, redundant corosync ring (heartbeat off the service plane); (2) dedicated live-migration plane + QEMU-native TLS; (3) revisit whether cross-DC service-to-service coordination on metal-internal warrants its own inter-DC plane.

**Related:** D-051 (combined in the SAME redeploy -- RBAC is keystone-only / low blast radius); supersedes the three conflicting D-052 drafts; depends on the MAAS rename/re-IP/new-VLAN gate (R1); the regenerated bundle differs substantially from BOTH repo HEAD (old space names) and the WIP bundle.yaml (which was Option-1 scope: only `internal` on metal-internal).

## D-053 -- Rename network space `fabric-data` -> `data-tenant` (2026-06-25)

**Status:** ADOPTED. Naming-only change to the D-052 plane set; no architectural change.

The geneve / tenant-overlay plane introduced as `fabric-data` in D-052 is renamed to
`data-tenant` to better fit the cloud's space-identity naming. This is a SPACE-NAME change
only: the subnet (`10.12.16.0/22`), its VLAN/fabric, the role (tenant geneve overlay), and
the *semantics* of every binding are unchanged. The four bindings that read `fabric-data`
(`nova-compute:neutron-plugin`, `ovn-chassis:data`, `ovn-chassis-octavia:data`,
`octavia:ovsdb-cms`) now read `data-tenant`; the D-052 subordinate-subset reasoning holds
verbatim with the new name substituted.

**Order of record:** applied in MAAS first (the live space is `data-tenant`), then
propagated into `bundle.yaml` (6 occurrences: 4 bindings + 2 binding-model header-comment
lines) so Juju's space lookup matches the live MAAS space at deploy. The bundle QA harness
was updated in lockstep (`verify_bundle.py` ALLOWED set + the two `fabric-data`-in-`planes()`
assertions; `target_bindings.json` 4 values) and re-run clean (8/8).

**Note on this log:** the D-052 entry above is left intact and still reads `fabric-data` --
that is the historical record of the original decision; this entry records the later rename
(append-only discipline).

## D-057: provider-vip plane -- separate tagged, routed plane for public API VIPs (2026-06-29)

**Status:** SUPERSEDED by D-060 (Pattern A revert, 2026-06-29) -- the provider-vip plane is abandoned; public API VIPs return to provider-public. [Originally DECIDED.] Full record + D-003B amendment: `docs/D-057-DECIDED-append.md`.

Root cause of the phase-06 FIP-unreachability blocker: API LXD containers bind `public`
to provider-public (untagged enp1s0); Juju bridges enp1s0 into a Linux bridge, starving
OVS `br-ex` so OVN's gateway ARP responder is dead. Fix: move the public API VIPs onto a
new tagged, routed plane `provider-vip` (VID 104 on the provider fabric), freeing untagged
enp1s0 for `br-ex`. FIPs stay on provider-public. This AMENDS D-003B (which co-located API
VIPs and FIPs on one L2): tenant->API is preserved via L3 routing, and it improves the
commercial hard-isolation posture (API VIPs out of the FIP broadcast domain). The durable
decision is the framework (a separate tagged/routed VIP plane); the specific subnet is set
by D-058 below.

## D-058: full plane renumber -- clean fabric-grouped /22 scheme (2026-06-29)

**Status:** SUPERSEDED by D-060 (Pattern A revert, 2026-06-29) -- the plane renumber is
abandoned; the cloud stays on the D-052/D-053 scheme. [Originally DECIDED (operator).] Full
map, jumphost ordering trap, NetBox-apex note, and the committed-foundation cascade:
`docs/D-058-renumber.md`. Supersedes the D-057 minimal-delta placement of provider-vip at
10.12.24.0/22; resolves R4 (oob).

Cloud-wide re-IP for Roosevelt addressing fidelity (contiguous /22 blocks grouped by
fabric), executed as a teardown/redeploy (no in-place re-CIDR, no transient overlap).
Rotate: 8->12, 12->16, 16->20, 24->8, 64->60; 4/32/36 fixed. Result:

| plane | CIDR | VLAN / fabric | gateway |
|---|---|---|---|
| provider-public | 10.12.4.0/22 | untagged / provider | 10.12.4.1 |
| provider-vip | 10.12.8.0/22 | VID 104 / provider | 10.12.8.1 |
| metal-admin | 10.12.12.0/22 | untagged / metal (PXE) | 10.12.12.1 |
| metal-internal | 10.12.16.0/22 | VID 103 / metal | none |
| data-tenant | 10.12.20.0/22 | untagged / data | none |
| storage | 10.12.32.0/22 | untagged / storage | none |
| replication | 10.12.36.0/22 | untagged / replication | none |
| oob | 10.12.60.0/22 | f_oob | 10.12.60.1 |

VIP triple = provider-vip .8.5x / metal-admin .12.5x / metal-internal .16.5x (octets 50-60).
lib-net.sh + bundle.yaml carry D-058; the foundation scripts/runbooks are swept as each
phase is executed (living-draft discipline). NetBox apex (netbox/ipv4-prefixes-import.py) is
stale pre-D-052 and must be de-staled to D-052/053 before it can carry D-058.

## D-059: host NIC budget -- build on five physical NICs; four-NIC collapse parked (2026-06-29)

**Decision:** Build and validate v1 on FIVE active physical NICs per host
(openstack0-3) -- the live as-built reality. Defer any reduction to a four-NIC
budget until the Roosevelt inbound-server hardware sheets are confirmed.

**Live inventory (measured 2026-06-29; all four hosts identical):**

| NIC     | plane(s)                                        | sharing      |
| ------- | ----------------------------------------------- | ------------ |
| enp1s0  | provider-public + provider-vip (VID 104)        | VLAN-shared  |
| enp7s0  | metal-admin + metal-internal (VID 103)          | VLAN-shared  |
| enp8s0  | data-tenant                                     | dedicated    |
| enp9s0  | storage (Ceph public)                           | dedicated    |
| enp10s0 | replication (Ceph cluster)                      | dedicated    |
| enp11s0 | ex-lbaas -- idle, REMOVED during this MAAS work | -            |

Provider and metal already carry two planes each via VLAN tagging; data, storage,
and replication each hold a dedicated NIC -- that trio is the five-vs-four overage.
NOTE: enp7s0 reads "unconfigured" in a physical-only interface query because its L3
lives on the br-metal / br-internal bridges built atop it, not on the raw NIC -- it
is load-bearing, not spare. The genuinely idle NIC is enp11s0 (ex-lbaas), and
removing it leaves five active, not four.

**Rationale:** Five is the measured truth; building on it avoids guessing a NIC count
before the hardware is known. The four-NIC fold is a contained, well-understood change
that can be made later without reworking the whole model.

**Four-NIC collapse path (PARKED; execute only if the Roosevelt sheets confirm a
four-NIC budget):** fold storage + replication onto ONE physical NIC -- Ceph public
untagged + Ceph cluster on a tagged VLAN -- mirroring the provider/metal pattern.
Result: enp1s0 (provider+vip) / enp7s0 (metal+internal) / enp8s0 (data) / enp9s0
(storage+replication) = four. Touches three places: carve-host-interfaces.sh
(storage/replication NIC wiring + a new tagged VLAN), the bundle's Ceph public/cluster
network bindings, and the MAAS VLAN layout (a replication VLAN on the storage fabric).

**Trade-off (explicit):** separating Ceph public from Ceph cluster onto distinct
physical NICs is a performance best-practice (replication traffic does not contend with
client traffic). The collapse keeps them logically separate via VLAN but shares one
physical NIC's bandwidth -- acceptable for a test rig; a Roosevelt judgement call. Hence
parked pending the sheets, with NIC bonding as an alternative if the hardware offers it.

**Status:** Adopted 2026-06-29. Pending input: Roosevelt inbound-server NIC count.
**Related:** D-057 (provider-vip plane), D-058 (plane renumber), D-052 (space inversion).
## D-060: revert to Pattern A on the hyperconverged test cloud -- supersedes D-057 + D-058 (2026-06-29)

**Decision:** Revert the provider-NIC architecture to **Pattern A** (MAAS builds the OVS
bridge `br-ex` on provider NIC `enp1s0`; the host static lands on `br-ex`; the API LXD
containers' `public` endpoint AND ovn-chassis share that one OVS `br-ex`) and KEEP the
existing D-052/D-053 plane scheme. **D-057 (provider-vip plane) and D-058 (full plane
renumber) are SUPERSEDED** -- both were responses to a self-inflicted regression, not to a
genuine architectural requirement.

**Root cause (the pivot).** The phase-06 floating-IP-unreachability blocker was introduced by
this Claude lineage, not by Pattern A. The working pre-teardown cloud ran Pattern A. A prior
instance advised switching to **Pattern B** (raw `enp1s0`, charm-built `br-ex`) on the
rebuild. On a hyperconverged node -- where the API LXD containers and ovn-chassis share one
provider NIC -- Pattern B makes Juju build a competing Linux bridge `br-enp1s0` that captures
`enp1s0` before ovn-chassis can enslave it into OVS `br-ex`, starving `br-ex` of carrier and
darkening the floating-IP plane (OVN's gateway ARP responder goes dead). D-057 (move the
public API VIPs to a tagged `provider-vip` plane, VID 104, to free untagged `enp1s0` for
`br-ex`) and D-058 (cloud-wide renumber) were built to work around that breakage. Reverting
Pattern B removes the cause, so the workarounds are no longer needed.

**No re-CIDR.** The live MAAS is already on D-052/D-053 (verified 2026-06-29: juju / lxd /
capi-mgmt on metal-admin 10.12.8.x; openstack0 data-tenant 10.12.16.40; openstack2 storage
10.12.32.42). D-058's renumber was never applied to the running cloud. So the revert is a
teardown + Pattern A re-carve + redeploy on the EXISTING subnet scheme -- no subnet migration,
no DHCP change, no `provider-vip` plane.

**Node-role learning (Roosevelt is NOT affected).** The collision is specific to
hyperconverged topology. Roosevelt is dedicated-role: Gateway Nodes run `br-ex` / Neutron and
Controller Nodes run the API on SEPARATE machines, so no single NIC is shared between the API
containers and ovn-chassis -- the Linux-bridge-vs-OVS race cannot occur there. On Roosevelt,
Pattern B (charm-built `br-ex` on a dedicated gateway uplink) is the clean fit; the test cloud
rehearses the STACK, not the node-role split. The metal-internal VID-103 trunk, which on the
test cloud rides one shared metal NIC, becomes a real trunk to the gateway / controller /
compute access ports at Roosevelt.

**Files (this revert):**
- `bundle.yaml` -- reverted to the `bf7de5a` provider config: `public: provider-public` x11,
  VIP triple `10.12.4.5x / 10.12.8.5x / 10.12.12.5x`, all four `br-ex` MACs in ovn-chassis
  `bridge-interface-mappings` (the openstack0 MAC `52:54:00:3d:fd:54` that D-057 trimmed is
  restored). The bundle diff `bf7de5a..HEAD` was verified to be PURELY D-057/D-058, so the
  revert is a clean wholesale checkout of that file.
- `scripts/lib-net.sh` -- CIDR / VIP constants back to D-052/D-053; `PROVIDER_VIP_CIDR` /
  `PROVIDER_VIP_VID` dropped; `provider-vip` added to `STALE_SPACES` (must-be-absent guard).
- `scripts/carve-host-interfaces.sh` -- provider section is now Pattern A (`build_ovs_brex`:
  unlink commissioning link -> create OVS `br-ex` on enp1s0 -> host static on `br-ex`); the
  `enp1s0.104` / `br-prov-api` provider-vip stack and the VID-104 assertion are removed;
  metal / data statics back to D-052/D-053 (br-metal .8, br-internal .12, data .16).
- `scripts/provider-bundle-check.py` -- NEW fail-closed deploy gate (provider-public x11,
  `.4/.8/.12` triple octet 50-60, all 4 chassis MACs present); REPLACES the retired
  `scripts/d057-bundle-check.py`. Verified PASS against the reverted bundle.
- `scripts/phase-00-maas-standup.sh` -- target table + narrative retargeted to D-052/D-053;
  a dry-run now reports "no drift" against the live cloud (refuse-to-clobber guard retained
  for a fresh cloud / Roosevelt).
- `scripts/phase-00-teardown.sh` -- header + next-pointer updated (Pattern A
  re-carve / standup-verify, not re-CIDR). Logic unchanged (dynamic system_id resolution,
  capi-mgmt delete, management-substrate exclusion).
- `runbooks/phase-00-teardown-maas-reset.md` -- rewritten script-driven and current
  (deletes capi-mgmt, dynamic system_ids, D-052/D-053 CIDRs, Pattern A re-carve, OSD wipe,
  8_lbaas net removal). The prior file was pre-D-052 stale (five-space scheme, hardcoded
  system_ids, "leave capi-mgmt Ready").

**Retired (git rm):** `scripts/phase-00-maas-recidr.sh` (D-058 re-CIDR; the live cloud is
already D-052/D-053), `scripts/provider-vip-standup.sh` (D-057), `scripts/d057-bundle-check.py`
(replaced), `runbooks/jumphost-provider-vip-gateway.md` (D-057),
`runbooks/provider-vip-maas-standup.md` (D-057), `runbooks/phase-00-maas-reconfigure.md`
(D-058 reconfigure spine).

**Status:** Adopted 2026-06-29 (operator "Go"). Supersedes D-057, D-058.
**Related:** D-052 / D-053 (plane scheme, retained), D-003B (API + FIP L2 co-location,
restored), D-059 (NIC budget retained; its inventory's "enp1s0 = provider-public + provider-vip"
row is now just provider-public -- the physical five-NIC count is unchanged).

## D-061: Machine-preserving teardown via `remove-machine --keep-instance` (supersedes D-055)

**Status:** ADOPTED 2026-06-30. Supersedes D-055. Documented behavior; the `--keep-instance`
path is UNVALIDATED on this virsh-pod MAAS at adoption (validate via the canary, below).

**Problem (observed 3x in one session):** `juju destroy-model` DECOMPOSES the MAAS pod-composed
openstack0-3 machines -- they are deleted from MAAS and the carve is lost, forcing a full
reenroll + re-carve. This happened with `--destroy-storage --force`, and AGAIN with
`--force --no-wait` and NO storage flag at all. So D-055's diagnosis ("omit --destroy-storage
and the machines release to Ready") is WRONG/insufficient: the storage flag is not the trigger;
`destroy-model` itself releases the instances, and a MAAS pod auto-decomposes a released
pod-composed VM.

**Root cause / mechanism:** machine retention is NOT a `destroy-model` option -- it is a
`remove-machine` option. Juju 3.6 `remove-machine` reference: a machine can be removed from the
Juju model without affecting the corresponding cloud instance via `--keep-instance`.
`destroy-model` has no machine-preserving flag. The openstack0-3 VMs are MAAS pod-composed
(VM host `lxd`), so any *release* auto-decomposes them; `--keep-instance` is the only documented
way to detach-from-model without release.

**Decision:** two teardown scripts, by intent:
- `scripts/phase-00-teardown-release.sh` -- KEEP machines. Per host:
  `juju remove-machine <id> -m openstack --keep-instance --force --no-wait` FIRST (detach,
  preserve the MAAS instance + carve), HARD-VERIFY each host survived in MAAS (FAIL LOUD if any
  decomposed), THEN `juju destroy-model openstack --release-storage --force --no-wait --no-prompt`.
  Result: hosts stay Deployed + carved, reusable with no reenroll/recarve.
- `scripts/phase-00-teardown-destroy.sh` -- DECOMPOSE machines (the from-scratch path):
  `juju destroy-model openstack --destroy-storage --force --no-wait --no-prompt`; reenroll+carve after.
Both default to dry-run and require typing the model name at a /dev/tty gate before mutating.

**First-run safety (UNVALIDATED caveat):** `--keep-instance` is documented but had not been
exercised on this virsh-pod MAAS when adopted. `phase-00-teardown-release.sh --apply --canary`
removes ONLY openstack0, verifies it survived in MAAS, and STOPS -- run that before trusting the
all-four path. The script also hard-verifies survival after every remove and exits 1 if any host
decomposed (so a wrong assumption fails loud, not silently).

**Roosevelt:** bare-metal nodes are enlisted, not pod-composed, so release does not auto-decompose
-- the keep path is simpler there. If a MAAS pod is ever used at Roosevelt, configure it to not
decompose-on-release (or keep erase-on-release OFF) and carry the --keep-instance discipline.

**Related:** D-055 (superseded), D-018 (MAAS-release-direct teardown intent), DOCFIX-057
(old phase-00-teardown.sh deprecated -- its "releases to Ready" premise was wrong).

## D-062: mysql-innodb-cluster deploys at target unit count; no single-unit seed-then-scale

**Status:** ADOPTED 2026-06-30. Refines the D-009 unit-count guidance for this charm.

**Context:** to avoid the 3-way Group-Replication formation race that twice parked a unit `blocked`
with a metadata half-join, a single-mysql seed (`num_units: 1`, ramp to 3 later) was tried. It
FAILED: the unit parked `blocked` `'cluster' missing, Instance not yet configured for clustering`.

**Root cause:** the OpenStack reactive `mysql-innodb-cluster` charm bootstraps the InnoDB cluster
(`create_cluster`) from inside its `cluster` PEER-relation handler. With `num_units: 1` the peer
relation never receives a `-relation-changed` carrying a peer, so the bootstrap handler never runs.
Live flags confirmed: `certificates.available`, `local.cluster.user-created`, leader passwords set,
but NO `cluster.available`/bootstrapped flag; the log showed only update-status/db-router hooks
cycling -- no `create_cluster` attempt. This is NOT the Canonical "Charmed MySQL" charm (different
interface) which DOES support single-then-scale; the OpenStack charm's README only documents
`-n 3` deploy.

**Decision:** deploy `mysql-innodb-cluster` at its target unit count (3) from cold. Peer-driven
bootstrap is the charm's happy path: with 3 units brought up together, the cluster forms and (this
deploy) reported clean -- `status: OK`, 1 R/W + 2 R/O, `instanceErrors=[]` on all members. The
single-unit seed-then-ramp idea is RETIRED for this charm.

**Half-join distinction (carried):** the metadata half-join (`instanceErrors: ["...not managed by
InnoDB cluster. Use cluster.rescan()..."]`) is an artifact of MID-LIFE instance addition, not cold
bootstrap. If it recurs during a scale operation, the documented fix is
`cluster.rescan({addInstances:"auto"})` via mysqlsh as `clusteruser` (NOT the charm's bare
`cluster-rescan` action, which reports-but-does-not-adopt). Cold 3-unit deploy did not hit it.

**As-executed this deploy:** the running model was (mistakenly) deployed single-mysql, then
corrected in place: `juju remove-unit mysql-innodb-cluster/1` (a stuck mis-placed unit), then
`juju add-unit mysql-innodb-cluster -n 2 --to lxd:1,lxd:2` (real live machine ids 0-3 this cycle).
Cluster formed clean. The committed bundle is reverted to `num_units: 3` / `to: [lxd:8,9,10]`
(matching its own `machines:` 8-11 declaration; the live 0-3 numbering was a deploy-time artifact).

**Related:** D-009 (3 units on Roosevelt), BUNDLEFIX (bundle reverted to 3-unit).

---

## D-063: RESOLVED / CLOSED (2026-07-03) -- tighten the capi-mgmt-cluster security group ingress for Roosevelt

**Status:** PROPOSED / OPEN (recorded 2026-07-01) -> RESOLVED / CLOSED 2026-07-03 (applied + functionally verified; see the RESOLVED/CLOSED amendment below).

**Context:** the phase-06 in-cloud CAPI management VM (`capi-mgmt-v2`, D-035) sits behind the
`capi-mgmt-sg` security group. As built by phase-06 (a CAPI/cluster-api default posture), that SG
opens BOTH `tcp/6443` (apiserver) AND `tcp/22` (ssh) to `0.0.0.0/0`. Phase-07 Step 7.1 relies on
this: the conductor reaches the apiserver via the FIP with no per-conductor rule (which is why the
old hardcoded `10.12.4.76/32` rule-add was dropped in DOCFIX-063).

**Question:** on v1 (single-DC virtual rehearsal) the FIP is the sole access point and `0.0.0.0/0`
on 6443/22 is acceptable. For ROOSEVELT (commercial, multi-DC, HARD tenant isolation) an apiserver
and ssh open to any source that can route to the FIP is too broad.

**Options (unresolved):**
- (a) Tighten `tcp/6443` ingress to the magnum-conductor's measured source (and any operator
  tooling), and `tcp/22` to the ops/jumphost source only. Requires knowing the conductor's
  post-NAT source as the mgmt VM sees it (measured, not the tenant/provider literal).
- (b) Front the mgmt apiserver with a scoped ingress (e.g. a dedicated LB/SG chain) rather than a
  raw FIP with a wide SG.
- (c) Leave wide on the rehearsal, and make the tightened SG a phase-06 build step on Roosevelt.

No decision made -- recorded as an open hardening item (cf. D-043, D-050, also pending). Whichever
option, phase-07 Step 7.1 stays verify-first (it does not depend on the SG being wide; if 6443 is
NOT already permitted it measures the source and adds exactly that rule).

**Related:** D-035 (the mgmt VM), D-039 (per-cluster app-creds), phase-06 (SG creation),
phase-07 Step 7.1 (DOCFIX-063 verify-first).

### D-063 -- AMENDMENT (2026-07-03): sources MEASURED; option (a) adopted as (a-amended)

Measurement (jumphost session 2026-07-03; committing this amendment constitutes adoption):
1. Route: magnum/0 -> FIP 10.12.7.222 is ON-LINK via its provider leg (src 10.12.4.154) -- the
   FIP lives inside 10.12.4.0/22, so no router/SNAT exists on that path.
2. 170s SYN capture on the mgmt VM (dst 6443, pod-net 10.1.0.0/16 and loopback excluded),
   with a deliberate operator kubectl trigger mid-window. Observed sources:
     3x 10.12.4.154  -- magnum conductor health poll, ~60s cycle (route prediction CONFIRMED)
     2x 10.12.4.1    -- the OPERATOR kubectl trigger, arriving MASQUERADED to the provider
                        gateway by the upstream router (jumphost 10.17.11.246 is erased)
     2x 10.20.0.207  -- the VM hairpinning its own FIP (DNAT returns with the fixed IP as
                        source); SG rules DO apply to hairpinned traffic
   Falsified along the way: "the conductor holds a persistent apiserver connection" -- it is
   transient per-cycle; instantaneous `ss` snapshots cannot capture the client set. Use the
   window-capture method (tcpdump SYN filter over >=2 poll cycles) for any future SG sourcing.

ADOPTED RULE SET (a-amended), 6443 ingress on capi-mgmt-sg:
  - magnum unit provider addresses, /32 each, derived DYNAMICALLY from Juju at apply time
    (PATTERN-1; per-UNIT -- Roosevelt's 3-unit magnum contributes 3 rules; enumerate, never
    assume one). Value: blocks tenant-workload sources (tenant egress arrives as each tenant
    router's own 10.12.4.x SNAT address, which is NOT in the set).
  - 10.12.4.1/32 -- the operator path, EXPLICITLY DOCUMENTED AS COARSE: upstream masquerade
    makes this equivalent to "any source routed in via the gateway". The SG cannot
    distinguish the jumphost; do not pretend otherwise in docs or reviews.
  - 10.20.0.0/24 (capi-mgmt-subnet) -- self/hairpin traffic; omitting it breaks on-VM use of
    the external endpoint silently.
SSH (22) ingress: 10.12.4.1/32 only (same coarseness caveat).
NOT adopted: 0.0.0.0/0 stays out; option (b) (LB in front) remains the Roosevelt-scale
candidate if magnum unit churn makes /32 maintenance noisy.
Roosevelt note: the 10.12.4.1 masquerade is a VR0 substrate artifact. The rule set MUST be
re-derived from a fresh window capture in every environment -- never copied forward.
Apply is a SEPARATE gated mutation with its own verify-before/after block; this amendment
records the decision, not the change. **Status: ADOPTED (a-amended) on commit; apply pending.**

### D-063 -- RESOLVED / CLOSED (2026-07-03): applied and functionally verified

APPLIED via scripts/d063-apply.sh --apply (DOCFIX-084/085): four ingress rules created and
read-back-gated BEFORE the two wide 0.0.0.0/0 rules were removed by measured ID (no window
where the conductor source was unauthorized). Final SG state (measured): 6443 from
10.12.4.154/32 (conductor), 6443 from 10.12.4.1/32 (operator, coarse), 6443 from 10.20.0.0/24
(hairpin), 22 from 10.12.4.1/32; wide 22/6443 ingress gone (only default egress remains).

FLOW MODEL (corrected + doc-backed). Earlier in-session I twice mischaracterized the
conductor->mgmt traffic as a "continuous health poll"; both were WRONG. As-built (phase-07:
/etc/magnum/kubeconfig server = mgmt FIP 10.12.7.222:6443, client-cert auth) plus upstream
docs (StackHPC magnum-capi-helm deep-dive; openstack magnum-capi-helm install guide) establish
the true model: magnum-conductor is a Kubernetes CLIENT of the mgmt apiserver, acting ONLY on
cluster lifecycle events (create/update/delete -> helm install/upgrade + watch). Connections
are per-operation and short-lived (client-cert kubeconfig, no held session), so the path is
IDLE at rest and busy only during ops. This fully explains why idle established-state and SYN
snapshots repeatedly showed nothing -- expected, not a fault. Required routes per StackHPC:
conductor -> mgmt cluster; mgmt cluster -> OpenStack external net; mgmt cluster -> workload API
LBs. Only the first traverses capi-mgmt-sg ingress; the rule set covers it.

VERIFICATION (proof by exercise, through the tightened SG):
- POSITIVE, load-bearing: R1 -- a live TCP handshake from magnum/0 to 10.12.7.222:6443
  (phase-07 Step 7.1 primitive: exec 3<>/dev/tcp) returned TCP-OK. A handshake cannot complete
  if the SG drops the source; conductor source measured = 10.12.4.154 (= the applied /32).
- POSITIVE, functional (gold standard): C3 -- forced a real op (openstack coe cluster resize
  beta-cluster 1->2 as the tenant); it reached UPDATE_COMPLETE / HEALTHY / node_count=2. A
  magnum-capi-helm resize CANNOT complete unless the conductor helm-upgraded the manifest on
  the mgmt apiserver (10.12.7.222:6443) through the SG and CAPI built the worker. The
  successful state transition is dispositive proof the path carries real helm ops post-tighten.
- NEGATIVE (isolation value): a beta tenant POD -> 10.12.7.222:6443 TIMED OUT (blocked). Tenant
  workloads egress as their router's own 10.12.4.x SNAT, which is NOT in the allow set. This is
  the isolation the ruling exists to provide.
- OPERATOR: kubectl get --raw /readyz via the mgmt kubeconfig returned ok (10.12.4.1 coarse
  rule, as designed).
- NOT RELIED ON: SYN packet captures (V4/R5) came back empty -- the transient per-op flow is
  the wrong target for snapshot/window SYN sniffing (learned 3x this thread; established-state
  and functional-outcome are the correct instruments). A conductor-log corroboration probe (C2)
  was empty because it queried journald; charmed magnum logs to
  /var/log/magnum/magnum-conductor.log. Neither is needed: C3's functional success supersedes
  wire-level corroboration (industry practice closes on functional verification of the flow).

BETA left at node_count=2 (UPDATE_COMPLETE/HEALTHY) -- the resize doubles as previously-untested
resize acceptance coverage; a change from the count=1 acceptance baseline, retained deliberately.

Roosevelt carry-forward: the rule set is per-UNIT and per-ENVIRONMENT -- 3-unit magnum yields 3
conductor /32s; the 10.12.4.1 operator collapse is a VR0 masquerade artifact; re-derive with
d063-apply.sh (live per-unit derivation) in every environment, never copy rules forward.

**Status:** RESOLVED / CLOSED (applied + functionally verified 2026-07-03). Supersedes the
"apply pending" state above. **Related:** DOCFIX-084/085 (the applier + its stderr-capture
fix), phase-07 Step 7.1 (the TCP primitive reused for R1), D-035/D-036 (mgmt-cluster model).


---

## D-064: Reconcile D-051 to the SCS Domain Manager standard (scs-0302); fix create-op policy templating

**Status:** ADOPTED 2026-07-01 (phase-08 keystone-policy blocker). Reconciles D-051 to the
authoritative SCS reference and discharges D-051's [LIVE-READ PENDING] base_* alignment gate.
Behavioral acceptance = Magnum trustee create + tenant G3 (below).

**Context / trigger:** phase-08 workload-cluster create failed at the FIRST step --
create_trustee_and_trust -> identity:create_user HTTP 403. magnum_domain_admin holds Admin on
the magnum domain (correct per Magnum docs + D-046) and was authenticated (401 would be auth;
403 = authorization). Root cause: this cloud's charm-rendered /etc/keystone/policy.json
(old-style defaults, enforce_scope=False) defines the create-op helper
admin_and_matching_user_domain_id with legacy templating domain_id:%(user.domain_id)s, which
does NOT resolve for create_user on Caracal (keystone populates target.user.domain_id, not the
bare user.domain_id). The admin-on-domain create branch silently evaluates false and only
cloud_admin passes -- the same defect that blocked tenant domain-admin self-service in the
2026-06-22 northwind rehearsal. One bug, two symptoms (Magnum trustee + tenant self-service).

**Finding (why D-051 alone did not fix it):** D-051 deliberately deviated from scs-0302 -- it
dropped Section A (base_*) and pointed fallthroughs at the LIVE helpers by name, including the
broken admin_and_matching_user_domain_id, and dropped the `or rule:admin_required` tail.
Consequently D-051's create_user fallthrough inherited the broken helper; only its manager
branch (correct token.domain.id:%(target.user.domain_id)s templating) worked. Attaching D-051
as-staged would have fixed tenant self-service (manager) but NOT Magnum (admin-on-domain trustee).

**Decision:** reconcile to scs-0302 by correcting the create-op fallthroughs to the cloud's
CORRECT %(target.*.domain_id)s helpers (mirroring the standard's base_create_*) and complete the
base_* alignment against the now fully-read live policy. Exactly 7 rules change:
- create_user / create_project / create_group: fallthrough admin_and_matching_<x>_domain_id
  (broken %(<x>.domain_id)s) -> admin_and_matching_target_<x>_domain_id (correct
  %(target.<x>.domain_id)s). BUG-FIX; restores the documented admin-on-domain create path and
  unblocks Magnum. Confined (domain match); does NOT touch tenants (they hold manager, not
  domain-admin).
- add_user_to_group: admin_and_matching_group_domain_id -> admin_and_matching_target_group_domain_id.
  ALIGNMENT (restore live default; D-051 had drifted to the broken helper).
- list_users / list_projects / list_groups: fallthrough admin_required (D-051 conservative
  [PENDING-LIVE-READ] guess, BROADER than live) -> cloud_admin or admin_and_matching_domain_id
  (the live default). ALIGNMENT; behavior-preserving; removes an unintended widening.
list_roles keeps admin_required (its live default). The manager persona (Section B),
is_domain_manager = role:manager, and is_domain_managed_role = member + load-balancer_member
(no admin, no manager -- anti-escalation) are UNCHANGED from D-051 / scs-0302.

**Consequence:** with create_user's admin fallthrough fixed, magnum_domain_admin creates
per-cluster trustees via the standard admin-on-domain path -- NO manager grant on the magnum
service account is required (an in-session Option A now moot). Magnum keeps its documented
trustee setup; the policy fix alone unblocks it.

**Validation:** oslo.policy parses all 37 rules (the charm validates YAML only -- a malformed
rule passes YAML and silently no-ops, cf. D-046 "reports OK while broken"); YAML + ASCII clean;
malformed-connector lint clean; every fallthrough tail diffed against the live effective default.
Behavioral gate (the real acceptance): (a) Magnum -- re-run phase-08 8.1, trustee create_user
passes, cluster converges; (b) tenant G3 -- a manager-on-domain user self-services user/project
+ member/load-balancer_member within its domain (PASS), is denied admin/cross-domain (DENY),
cloud-admin unaffected.

**Known SCS limitation (carried, per standard):** the manager persona grants list_domains and
list_roles cloud-wide (needed to resolve domain/role names). A tenant manager can ENUMERATE
domain names/ids + role names (not access resources). Inherent to the scs-0302 transitional
policy (upstream RBAC-scoping of domain list is a pending fix); the native 2024.2 persona closes
it. On upgrade to 2024.2+ this override MUST be removed (scs-0302 + D-051 caution).

**Mechanism:** zip overrides.zip domain-manager-policy.yaml -> juju attach-resource keystone
policyd-override=overrides.zip (use-policyd-override already true). keystone PO (broken) -> PO:.
Rollback: juju config keystone use-policyd-override=false. Full attach + G3 procedure in
runbooks/appendix-C-identity-rbac.md.

**Roosevelt:** single portable policy file replicated per-DC keystone; the attach + G3 gate
becomes a deploy step. Remove on any 2024.2+ upgrade.

**Related:** D-051 (reconciled here), D-046 (magnum trustee domain), D-039 (per-cluster app-cred
roles), D-050 (resolved by supplying the zip), scs-0302-w1 (the authoritative standard),
appendix-C (identity/RBAC reference). **Supersedes:** D-051's [LIVE-READ PENDING] gate (discharged).


---

## D-065: Fix identity:create_trust non-resolving policy template (Magnum multi-tenant cluster blocker)

**Status:** ADOPTED 2026-07-02. Extends the D-064 defect-class fix to the keystone trust family.
Behavioral acceptance = a tenant service identity clears create_trust and the cluster converges
(pending re-attach + retest at adoption).

**Context / trigger:** multi-tenant validation (tenant acme). A tenant service identity (acme-svc,
app cred, holding member + load-balancer_member on acme-prod) creating a Magnum cluster failed at
create_trust with identity:create_trust 403 -- AFTER D-064 unblocked create_user (the trustee user
was created). The SAME 403 reproduced (a) for admin via password auth, and (b) via a DIRECT
`openstack trust create` with trustor == caller == self, no magnum in the path. That proves it is
identity-independent and not a magnum bug.

**Root cause:** the charm-rendered base policy.json defines
identity:create_trust = "user_id:%(trust.trustor_user_id)s" -- the legacy NON-target-prefixed
template that does NOT resolve on Caracal (keystone populates target.trust.trustor_user_id). The
rule evaluates user_id == "" -> always false -> every trust create 403s regardless of caller. Same
defect class as D-064 (create_user's %(user.domain_id)s vs the resolving %(target.user.domain_id)s).
Keystone's OWN shipped default is the target-prefixed form:
"user_id:%(target.trust.trustor_user_id)s".

**Masking:** create_trust was never reached before D-064 because create_user 403'd first. The
2026-06-09 CREATE_COMPLETE was on the pre-2026-06-11-teardown cloud; this redeploy's create_trust
had never been exercised until D-064 opened the path.

**Causation proof (verify-before-fix):** with use-policyd-override=false (base policy only), the
direct trust create STILL 403s -> the D-064 override is NOT the cause; the base charm policy owns
the bug. Override re-enabled immediately (PO: restored) -- the cloud was not left without the
manager persona.

**Decision:** add identity:create_trust = "user_id:%(target.trust.trustor_user_id)s" to the D-064
override (policies/domain-manager-policy.yaml). Exactly the keystone-shipped default, and
consistent with this cloud's own already-correct sibling rules (list_trusts_for_trustor/trustee).
No admin fallback (matches the shipped default; trusts are self-delegation only). No manager branch
(create_trust is universal self-delegation, not part of the domain-manager persona).

**Scope (evidence-gated; NOT a blanket family sweep):** create_trust is the ONLY demonstrably
broken trust rule. list_trusts_for_trustor / list_trusts_for_trustee already carry the
target-prefixed form (match shipped). get_trust / delete_trust / list_trusts / list_roles_for_trust
/ get_role_for_trust are live "" (empty) but code-guarded in keystone (_trustor_trustee_only;
delete checks user==trustor or is_admin) -> functional, not broken, not blockers; left unchanged.

**Second-check clearance:** keystone's create_trust also enforces _require_trustor_has_role_in_project
(trustor must hold each delegated role on the project). Magnum delegates context.roles (the caller's
token roles), which by construction are a subset of what the caller holds on the scoped project ->
passes. acme-svc holds member + load-balancer_member on acme-prod; confirmed clear on retest.

**Validation:** oslo.policy parses all 38 rules; YAML + ASCII + connector lint clean.

**PROPOSED / OPEN (separate hardening, NOT actioned here):** live identity:list_trusts = "" lets any
authenticated user enumerate ALL trusts (trustor/trustee/project relationships) cloud-wide -- an
info-disclosure the newer keystone default tightens to
"rule:admin_required or (role:reader and system_scope:all)". This is a security-posture change, not
a functional blocker (create/get/delete are code-guarded), so it is recorded as an open item to rule
on rather than bundled into this blocker fix. If adopted, add list_trusts (and optionally the read
rules) at the shipped defaults in the same override.

**Roosevelt:** carry the create_trust override with the rest of the D-064 policy. The base-charm
non-resolving-template defect is upstream-worthy -- report against the keystone charm's rendered
policy.json for Caracal (2024.1). Remove on 2024.2+ (native secure-RBAC ships the resolving defaults).

**Related:** D-064 (same defect class; same override file), D-039 (the trustor roles that satisfy the
trustor-has-role check), D-046 (magnum trustee domain), D-035 (mgmt cluster), appendix-D (trust
model). **Revises:** appendix-D section D.3 -- the role-delegation hypothesis is REFUTED (a
clean-role tenant identity still 403'd; the cause was the non-resolving policy template, not role
delegation). appendix-D to be corrected on finalize.


---

## D-066: Tenant account model (Option-3 split); cluster-create requires PASSWORD auth

**Status:** ADOPTED 2026-07-02. Standard per-tenant identity set, used from the first tenant so
v1 and Roosevelt onboard identically (no interim model to retrofit). Trust path validated live
through create_user + create_trust + cert-gen entry (see D-064/D-065); full cluster completion is
gated on D-067 (Barbican/Vault substrate).

**The per-tenant accounts (operator creates the domain + manager; manager creates the rest):**
- `<client>-domain-admin` -- `manager` on the domain (SCS Domain Manager, D-051/D-064). Tenant
  IAM self-service. Operator-provisioned; password handed over.
- `<client>-cluster` -- `member` + `load-balancer_member` on `<client>-prod`. PASSWORD auth only.
  Sole purpose: `coe cluster` lifecycle (this is the identity that mints the Keystone trust). Also
  owns whatever magnum-internal creds the driver mints under it (the per-cluster CAPO child cred,
  D-039). Manager-created.
- `<client>-svc` -- `member` + `load-balancer_member` on `<client>-prod`. UNRESTRICTED app cred,
  for tenant-authored non-trust automation (CI/pipelines). NOT used for cluster-create. Manager-created.

**Why password (not app cred) for cluster-create -- the hard constraint:** keystone on this build
blocks trust creation from application-credential tokens unconditionally. Source
(keystone/api/trusts.py `_check_application_credential`, read live 2026-07-02): if the token method
is `application_credential` it raises "Using method 'application_credential' is not allowed for
managing trusts" -- and the docstring states this applies "regardless of the 'unrestricted' flag"
(this build is STRICTER than upstream, which exempts unrestricted). Confirmed live: an unrestricted
app cred (which minted a child cred, so genuinely unrestricted) still hit this block; the same
identity via PASSWORD passed it. So the cluster-creator must be password-authenticated.

**Security rationale (this is the correct side of the control, not a workaround):** the block exists
to stop a single-project app-cred token from extending the trust delegation chain beyond its scope.
A password token is full-user scope, so a trust minted from it delegates only what the user already
holds (member + load-balancer_member on its own project) -- no escalation, confined to the tenant.
Splitting `-cluster` (trust-capable password, cluster ops only) from `-svc` (app cred, everything
else) isolates the trust-capable credential to one identity doing one job.

**REJECTED:** `CONF.security_compliance.allow_insecure_application_credential_trust_escalation`
(the escape hatch in `_check_application_credential`). Enabling a setting named "insecure" +
"trust_escalation" -- which defeats a control designed to contain a compromised tenant credential --
undercuts the hard-isolation thesis of the whole build. Not adopted.

**Roosevelt:** onboard every tenant with this three-account set. Script it (scripts/tenant-onboard.sh,
DRAFT this session). CAPO-cred ownership under `-cluster` confirmed acceptable (magnum-internal,
created/destroyed with the cluster) -- the empirical "whose identity mints the child cred" check
is pending the D-067 substrate fix (cluster-create dies at cert-gen before the driver's mint step).

**Related:** D-051/D-064 (manager persona), D-065 (create_trust template fix), D-039 (trustor roles).
**Revises:** appendix-D D.3 (the app-cred-creator assumption is wrong -- corrected in appendix-D this session).

---

## D-067: barbican-vault -> Vault (vault-kv) must use the metal-internal plane (live drift; the cert-gen blocker)

**Status:** ADOPTED 2026-07-02 (fix pending next session -- a live rebind, gated). This is the defect
that blocks multi-tenant cluster COMPLETION (cluster-create clears trust, then dies generating certs).

**Symptom chain:** tenant cluster-create (password identity) cleared create_user + create_trust,
then `CREATE_FAILED: Failed to create certificates`. Root cause traced through magnum -> Barbican
(`POST /v1/secrets` 500) -> castellan vault_key_manager `_build_auth_headers` -> Vault AppRole login
rejected. Vault's RAW error (not an HTTP code -- the authoritative read):
`source address "10.12.8.176" unauthorized through CIDR restrictions on the secret ID`.

**Root cause:** barbican reaches Vault at Vault's METAL-ADMIN address (vault_url=http://10.12.8.190:8200,
egress src 10.12.8.176 via eth2). Vault's barbican-vault AppRole binds secret_id to the metal-internal
CIDR (where service-to-service traffic belongs, D-052/D-053). Off-plane source -> Vault rejects. This
is NOT secret_id expiry: `juju run vault/leader refresh-secrets` DID rotate the secret_id (barbican.conf
re-rendered, barbican restarted) and the login STILL failed with the CIDR error. The earlier
"expired TTL" reading was WRONG -- corrected here.

**The bundle is CORRECT; the LIVE env drifted.** bundle.yaml already binds every secrets-path endpoint
to metal-internal: vault `secrets: metal-internal` (L130), barbican `secrets: metal-internal` (L667),
barbican-vault `secrets-storage: metal-internal` (L700). So a redeploy from the bundle would NOT have
this defect. The live deployment's effective binding for the barbican<->vault secrets path resolves to
metal-admin -- a drift from the bundle (mechanism: TBD by read-only `juju show-application` binding +
relation-data diagnosis next session; candidate causes: bindings added to bundle post-deploy and never
applied live, or the subordinate address-advertisement not following the bound space).

**Fix (rebind, do NOT widen the CIDR):** reconcile the live binding to the bundle so barbican egresses
from its metal-internal address (10.12.12.110), which the AppRole CIDR already trusts. Widening the
AppRole CIDR to metal-admin is REJECTED: it loosens a security control, legitimizes east-west traffic
on the wrong plane, gets reverted by the charm, and leaves the next Vault consumer exposed. Next-session
sequence: (1) read-only binding diagnosis (`juju show-application vault barbican barbican-vault`,
spaces<->subnets map); (2) gated `juju bind` / relation refresh to metal-internal; (3) re-run
refresh-secrets if needed; (4) confirm barbican AppRole login = HTTP 200 from a metal-internal source;
(5) re-run tenant cluster-create -> cert-gen clears.

**Related:** D-052/D-053 (six-plane network; metal-internal carries east-west service traffic),
D-057 (prior topology-didn't-follow-binding defect -- same family). **Corrects:** the in-session
"secret_id TTL expiry" hypothesis (refuted by the refresh-secrets test).

### D-067 -- AMENDMENT (2026-07-02, post-fix): corrected mechanism; FIXED live; CLOSED for v1

The "bundle correct / live drifted" framing above is WRONG and is corrected here (append-only
discipline; the body above stands as the historical record). Root cause, read from charm source at
the exact vendored versions:

1. Live bindings were NEVER drifted -- `juju show-application` matched the bundle exactly (all
   secrets-path endpoints metal-internal). The drift was one relation-data key: vault/0 advertised
   binding-correct ingress (10.12.12.117) but an explicit `vault_url` of http://10.12.8.190:8200.
2. Mechanism: charm-vault `send_vault_url_and_ca()` (re-fires on every non-update-status hook)
   computes vault_url from the `access` binding AND vault_url_external from the `external` binding,
   publishing BOTH when they differ. The vault-kv interface (vendored commit 6f7848c, per
   src/build.lock) IGNORES the `remote_binding` selector (deprecated, LP#1895185) and writes the
   single `vault_url` key on ALL relations -- so the external publish CLOBBERS the access URL for
   every kv consumer. Simultaneously the AppRole secret_id is CIDR-bound to the CONSUMER's
   secrets-relation ingress /32 (10.12.12.110/32) -- self-inconsistent whenever access != external.
3. Why external resolved to metal-admin: the bundle OMITS vault's `external` binding, so it falls
   to the `''` default (metal-admin). D-052's judgment call ("vault external -> metal-admin,
   operator/unseal path") rested on a mistaken premise: in this charm the `external` endpoint's only
   functional use is this kv-URL advertisement (plus an inert VIP check); operator/unseal access
   does not traverse it, and the listener binds [::]:8200 regardless. So a fresh deploy from the
   bundle REPRODUCES the failure -- this was a bundle+charm defect, not live drift.

Fix as executed (2026-07-02, gated): `juju bind -m openstack vault external=metal-internal`. The two
computed URLs became equal, the charm's own equality guard suppressed the second publish, relation
data flipped to http://10.12.12.117:8200 within one poll cycle, barbican-vault re-rendered
barbican.conf, and no refresh-secrets was needed. Validated end-to-end: AppRole login HTTP 200 from
the authentic source (kernel route src 10.12.12.110), then a full admin `openstack secret`
store/get/payload-compare/delete round-trip through the exact POST /v1/secrets path that 500'd.
Bundle hardened with BUNDLEFIX-007 (`external: metal-internal` + comment); the double-publish is
present on charm-vault master as of 2026-07-02, so the guard is permanent, including the D-068
1.16 pin.

Observation logged during validation (not actioned): barbican's `Secret href` renders as
https://None:9312/... (host_href unset/None). Cosmetic for barbicanclient/castellan consumers
(clients extract the UUID and dial their own catalog endpoint) -- stage-6 cluster cert refs will
exercise it for real; investigate only if cert-ref retrieval misbehaves.

**Status:** CLOSED for v1 (fixed + validated live 2026-07-02). **Adds:** BUNDLEFIX-007.
**Corrects:** the "bundle is CORRECT; the LIVE env drifted" mechanism above. **Related:** D-052
(premise correction noted), D-068 (carry BUNDLEFIX-007 through the 1.16 pin).

---

## D-068: PROPOSED -- Vault substrate hardening (Roosevelt)

**Status:** PROPOSED / OPEN 2026-07-02. Grouped substrate items surfaced while diagnosing D-067.
None block v1 tenant work; all are Roosevelt-durability items.

1. **Vault version.** Live is 1.8.8 (charm 1.8/stable) -- EOL. bundle.yaml pinned to 1.16/stable
   this session (D-068). Live upgrade is a MAJOR operation (multi-minor jump; unseal keys ready,
   storage-format compat, re-unseal after restart, ideally rehearsed) -- NOT a casual `juju refresh`.
   Verify the channel exists and rehearse before applying live.
2. **Vault over cleartext HTTP.** vault_url is `http://...:8200`; barbican<->Vault secret writes
   (including cluster CA private keys) cross metal-internal unencrypted. Enable Vault listener TLS
   for Roosevelt.
3. **AppRole credential lifecycle.** `refresh-secrets` exists as a manual action -> implies no
   automatic secret_id renewal. Audit secret_id TTLs across all vault-kv consumers; provision
   long/renewable TTLs or charm auto-renewal; add a proactive health probe that validates each
   consumer's Vault auth (the T4-style login check) BEFORE expiry, rather than discovering failure
   via a tenant's failed cluster-create.

**Related:** D-067 (surfaced these), D-052/D-053.

<!-- patchset-20260702-redeploy-readiness : DOCFIX-066..072 sweep -->

---

## D-002 -- AMENDMENT (2026-07-02): channel matrix reconciled to as-built

The D-002 table is historical (append-only); as-built corrections: (1) the `etcd, easyrsa`
row is DEAD -- etcd/easyrsa were dropped when vault moved to the mysql storage backend
(phase-02 R3: "HA Enabled false" is correct for vault-on-mysql); (2) `memcached` (absent
from the table) is pinned `latest/stable` -- the only track that charm publishes; the
project-wide "never latest/stable" guidance targets OpenStack-project charms, which
memcached is not (verify at deploy: `juju info memcached`); (3) the vault row (`1.8/stable`)
is superseded by the **1.16/stable** pin (D-068 / BUNDLEFIX-007). Channel-staleness review
2026-07-02: every other pin confirmed current for Caracal/jammy; rabbitmq-server `3.9` is
that charm's TERMINAL track (no newer exists), not a stale pin. The upstream charm-guide
delivery page is frozen (2023-12) -- `juju info <charm>` is the only live authority; the
pre-deploy channel assert belongs in `scripts/pre-flight-checks.sh`.

## D-009 -- AMENDMENT (2026-07-02): rabbitmq scale-up is NOT mechanical without min-cluster-size

D-009 promises Roosevelt scale-up is "change num_units 1 -> 3 and rerun". For
`rabbitmq-server` that is insufficient: without `min-cluster-size: 3` the charm accepts
client relations BEFORE the cluster forms (same failure class as the D-062 mysql formation
race; upstream charm docs call min-cluster-size best practice for multi-unit deploys).
Roosevelt bundle delta: set `min-cluster-size: 3` alongside `num_units: 3`. No testcloud
change (single unit is correct here).

## D-011 -- AMENDMENT (2026-07-02): acceptance item 6 is MANUAL unseal (matches phase-08)

D-011 item 6 read "Vault unseal + auto-unseal-after-reboot pattern verified". The v1
standard, as ruled in phase-08 D-011.6, is MANUAL unseal (3-of-5 key shares at a hidden
prompt) after any vault restart; auto-unseal is an available option adopted case-by-case
and NOT configured in v1. Item 6 is therefore read as: "Vault manual unseal after reboot
verified (sealed-on-restart is BY DESIGN, not a fault)". The acceptance bar and the
acceptance runbook now agree.

## D-043 -- RESOLVED (2026-07-02): ADOPTED, option (a) -- tenant-VM auto-resume ON

**Decision:** adopt option (a): `nova-compute resume-guests-state-on-host-boot=true`,
cloud-wide, as charm config (already present in bundle HEAD -- this resolution brings the
decision record level with the implementation, closing the bundle-ahead-of-record drift
flagged as DOCFIX-072). Rationale: tenant VMs are a different class from control-plane
services -- auto-resume is the industry norm and the Roosevelt customer expectation; the
restart procedure already builds on it (guests resume per pre-shutdown state, so a
deliberately stopped guest STAYS stopped). D-041's "down is a signal" discipline is
UNCHANGED for control-plane services, which this option does not touch; the D-035 mgmt VM
remains manual-start by policy (it is a control-plane component that happens to be a VM --
if auto-resume masks a mgmt-VM incident class in practice, revisit with an explicit
exclusion). Roosevelt item: VM-state monitoring/alerting so "down" stays visible.
**Resolves:** D-043 (was PROPOSED). **Related:** D-040, D-041, restart procedure.

## D-051 -- AMENDMENT (2026-07-02): policy delivery is bundle-native (DOCFIX-071)

The D-051 mechanism ("zip -> juju attach-resource") was never folded into the deploy
schedule: the bundle set `use-policyd-override=true` with no resource, phase-01 knowingly
deployed into `PO (broken)`, and no phase attached the zip -- the live policy existed only
as a session action (D-064). A clean redeploy would have shipped WITHOUT the tenant-
isolation RBAC. **Amendment:** the policy ships IN the bundle -- keystone
`resources: {policyd-override: ./policies/overrides.zip}`, with the zip committed beside
its source. `attach-resource` (appendix-C C.3, now $HOME-pathed -- the /tmp form was
snap-blocked) remains the LIVE-UPDATE path only. Drift between zip and yaml is a
FAIL-CLOSED deploy-gate check in `provider-bundle-check.py` (content compare). Acceptance
remains BEHAVIORAL (phase-03 Step 3.4: `PO:` is stage 1; the C.4 G3 probe is the gate) --
unchanged from D-051's own warning that the charm validates YAML only.

## D-061 -- AMENDMENT (2026-07-02): validation scope + the runbook fork (DOCFIX-066)

Two clarifications for an error-free redeploy: (1) the canary in
`phase-00-teardown-release.sh` validates machine SURVIVAL only; machine RE-ACQUISITION by
a next model (juju's MAAS provider allocates Ready machines; a kept host stays Deployed)
is the still-unvalidated half. Until BOTH halves pass, the release path is NOT a route
into phase-01 -- it serves model-only teardown/surgery. (2) The VALIDATED full-redeploy
spine is therefore the DESTROY path: `phase-00-teardown-destroy.sh` -> 8_lbaas/OSD ops ->
`reenroll-hosts.sh` -> carve -> standup -> phase-01. The phase-00 runbook (DOCFIX-066)
now encodes this fork explicitly; it previously drove the deprecated
`phase-00-teardown.sh`, whose "releases to Ready" premise D-061 disproved.


## D-069: Vault unseal-key custody -- split custody, rehearsed second-person unseal

**Status:** ADOPTED 2026-07-03 (policy; custodian ASSIGNMENT pending operator input --
tracked as SEC-003 in docs/security-ledger.md).

**Problem:** the 5 Shamir shares (threshold 3) are held by a single operator. The restart
procedure names missing keys a hard recovery BLOCKER, and vault re-init wipes the TLS
plane -- so single-person custody is a bus-factor risk no commercial cloud should carry.

**Decision (standard Vault operational guidance):**
1. Distribute shares so NO individual holds threshold (3+): e.g. 2/2/1 across three
   custodians, or 1 per person plus sealed escrow. Custodian names + storage media are
   recorded OFF-REPO (custody detail is itself sensitive); the ledger row tracks status.
2. A SECOND-PERSON unseal rehearsal is an acceptance item: someone other than the
   initializing operator must successfully unseal (restart-procedure Stage 3) before
   the deployment is declared operational. "The keys work" and "a second human can use
   them" are different facts; only the second one survives the bus.
3. Key custody is reviewed at every re-init (re-init re-cuts the shares).

**Roosevelt:** evaluate auto-unseal (transit/cloud KMS) as a separate decision; split
custody remains the floor either way (recovery keys still exist under auto-unseal).

**Related:** D-011 (amended: manual unseal is the v1 standard), DOCFIX-006 (the
lost-init cautionary tale), docs/security-ledger.md SEC-003.

## D-070: KVM baseline snapshots -- SUPERSEDES D-012 (no snapshot restore path)

**Status:** ADOPTED 2026-07-03. Supersedes D-012.

**Finding:** D-012 (two KVM baseline snapshots per cycle) was never exercised -- no
`virsh snapshot` step exists in any phase runbook or script; the only "snapshot"
mentions in the repo are dated as-built value captures. An adopted-but-unexercised
decision is documentation debt pretending to be a safety net.

**Decision:** retire the KVM-snapshot restore path. D-017/D-018 already made
rebuild-from-runbooks THE restore path, and snapshots quietly compete with that
discipline: a team that restores a snapshot skips exactly the rehearsal this testcloud
exists to force, and snapshot restore does not transfer to Roosevelt bare metal.
The pre/post-maintenance BASELINE CAPTURES (juju export + inventories) are retained and
strengthened -- that role moves to `bash scripts/cloud-assert.sh --capture`, whose
committed asbuilt/<date>/ BOM is also the Roosevelt multi-DC drift baseline.

**Recorded counter-argument:** cheap mid-rehearsal rollback (e.g. tight phase-08
iteration loops) is a real convenience. If that pain materializes, revisit as an
explicitly-labeled DEV CONVENIENCE with a rule that a snapshot restore invalidates the
run for acceptance purposes -- never as a restore path.

**Related:** D-012 (superseded), D-017/D-018 (the actual restore path), DOCFIX-075
(cloud-assert --capture).

## D-071: Routine update cadence and Juju controller patch policy

**Status:** PROPOSED 2026-07-04 (filed by the jumphost controller-update workstream;
number relinquished by the main stream per the 2026-07-03 addendum-10 contention note).
Mechanism: `runbooks/ops-update-procedure.md` (DOCFIX-086). Operator to rule on the
policy points below; the runbook is usable per-window under individual gating meanwhile.

**Problem:** the cloud accumulates available updates on two layers with no standing
policy for either: (a) the Juju controller/agents (patch releases on the 3.6 track;
live measured 3.6.24 with client already 3.6.25), and (b) charm revisions floating
within their D-002-pinned channels (~20 apps showed `can-upgrade-to` on 2026-07-04).
Appendix-B says revision float is EXPECTED and re-baselines after validated state, but
nothing says WHEN updates are applied, in what order, or what controller jumps are
acceptable on a single non-HA controller.

**Proposed policy (rule each point):**
1. **Cadence trigger.** Updates are applied in deliberate maintenance windows (not on
   discovery). Proposed trigger: a monthly review of `can-upgrade-to` + controller
   patch delta, executed via ops-update-procedure when non-empty; security-driven
   charm revisions may pull a window forward.
2. **Patch-only controller rule.** A routine window may move the controller by PATCH
   level only within the pinned 3.6 track (e.g. 3.6.24 -> 3.6.25). Minor/major jumps
   (3.6 -> 3.7+) are each their own D-NNN with rehearsal, never a routine-window action.
3. **Standing order.** Controller -> model agents -> charm refreshes, keystone first,
   nova-compute last (the ops-update-procedure grouping). Never interleave layers.
4. **In-channel-only refreshes.** A routine window never changes a charm channel
   (D-002); channel moves are per-decision (see D-068 for the live example).

**Risk (explicit, for future reviewers): no-backup single-controller patch acceptance.**
The testcloud controller is SINGLE (non-HA) and Juju 3.6 ships no supported controller
backup (`create-backup` removed in 3.0). There is NO in-band downgrade of a controller.
Accepting routine controller patches therefore rests entirely on three compensating
controls: (a) patch-level-only jumps within 3.6/stable, (b) a proven-healthy pre-state
(`cloud-assert.sh --capture` PASS committed as the pre-change BOM), and (c) the D-070
restore posture -- a failed controller means re-bootstrap + rebuild-from-runbooks, i.e.
hours of rebuild, not minutes of restore. This risk is accepted for the virtual
rehearsal phase ONLY; Roosevelt must revisit it with HA controllers and a real
controller-state backup story before production operations.

**Roosevelt:** HA controllers change the upgrade blind-window and the risk calculus;
cadence and window sizing need an SLA-aware ruling; controller backup tooling is a
design item, not an ops improvisation.

**Related:** D-002 (channel pins), D-068 (Vault channel move, the non-routine
counter-example), D-070 (restore posture), DOCFIX-086 (ops-update-procedure runbook),
appendix-B (revision re-baseline policy).

## D-068 -- AMENDMENT (2026-07-05): vault 1.16 forward-pin is NOT viable; revert bundle to 1.8/stable

Item 1 ("Vault version") pinned the bundle to `1.16/stable` as a forward-pin off EOL Vault 1.8.8,
treating the move as "a MAJOR operation to rehearse." Deep-dive research (see
`docs/D-068-vault-1.8-vs-1.16-analysis.md`) shows it is not a hard *upgrade* but a largely-
**incompatible** different charm for this reactive Charmed OpenStack cloud:

- certificates interface changed **V0 -> V1** -- the 18 `vault:certificates` client charms are V0
  requirers and cannot bind a V1-only provider (the entire cloud TLS/CA layer fails to connect);
- storage went **Raft-only**, dropping the MySQL backend this deployment uses (`vault-mysql-router`);
- **no upgrade path** (backup-on-1.8 / restore-on-new-1.15, a CA-replacement migration);
- Canonical community notes report **new-vault + Charmed Ceph broken** (this cloud uses Ceph);
- Vault 1.15+ payload is **BUSL 1.1** (was MPL) -- commercial-licensing review item.

**Amendment:** (a) a deployable bundle MUST pin vault `1.8/stable` (the reactive, integration-
compatible charm) -- revert the `1.16/stable` bundle pin; (b) `1.16` is **ruled out** as the vault-
modernization path for this reactive cloud unless/until the OpenStack service charms gain
tls-certificates V1 support; (c) "get off EOL 1.8" stays **OPEN** as a Roosevelt-durability problem
(candidates: wait for OpenStack V1 support; OpenBao/MPL fork; or explicit EOL risk-acceptance for VR0
with a production remediation deadline -- operator/security call, not resolved here).

This also corrects the D-002 amendment note (item 3): the vault row reverts to `1.8/stable`; the
`1.16/stable` supersession is withdrawn. Bundle change coordinated with Code / applied at reconciliation.

**Related:** D-068 (this amends item 1), D-002 amendment (vault row), D-067/BUNDLEFIX-007 (vault-kv),
D-069 (unseal). **Evidence:** `docs/D-068-vault-1.8-vs-1.16-analysis.md`.

## D-071 -- AMENDMENT (2026-07-05): controller backup EXISTS on juju 3.6; risk section corrected

First execution of ops-update-procedure (ops-update-20260705) disproved the risk section's
premise "no supported backup in Juju 3.6": `juju create-backup` / `download-backup` exist on
the 3.6 client and were exercised live (902MB checksummed archive of the single controller,
~35s, downloaded to the jumphost pre-upgrade). The compensating controls for routine
controller patches are therefore FOUR, not three: patch-only jumps, proven-healthy pre-state
(committed pre-change BOM), a fresh `juju create-backup` archive (now the STANDARD 2.1 step,
DOCFIX-088), and the D-070 rebuild posture as last resort. Policy points 1-4 unchanged and
still PROPOSED for operator ruling. Restore-from-archive remains UNREHEARSED -- rehearsing a
controller restore is added to the Roosevelt open questions.

**Related:** D-071 (amends the Risk section), DOCFIX-088 (runbook 2.1 correction).

## D-072: Dashboard VIP TLS repair -- cluster binding moved to the default-binding space

**Status:** ADOPTED 2026-07-05 (operator-ruled: in-place fix on VR0); AS-EXECUTED same day.

**Problem (RCA in changelog addendum 15; symptom entry in appendix-A):** dashboard VIP
https was dead SINCE DEPLOY. The openstack-dashboard charm renders haproxy's 443 backend
on the CLUSTER-binding address but creates apache SSL vhosts only for the default-binding
and public-binding addresses. With `cluster` bound to metal-internal (a third space), the
TLS backend targeted a vhost-less socket that apache answers in PLAINTEXT, while haproxy's
Layer4-only health check kept the backend green. The fail-open phase-03 3.3 check (fixed
under DOCFIX-089) let it pass the deploy gate.

**Decision:** bind `cluster` to the SAME space as the application's default binding
(metal-admin), so the charm-rendered backend lands on an address apache serves.
Live: `juju bind openstack-dashboard cluster=metal-admin` (single unit; peer relation
carries no live traffic; hacluster rides the separate `ha` endpoint). Durable:
bundle.yaml bindings updated (BUNDLEFIX-011). NEVER provider-public (tenant plane).

**As-executed verification:** haproxy backends re-rendered to 10.12.8.182 (:70/:433);
https 200 on 10.12.4.58 and 10.12.8.58 (vault-CA-verified, csrftoken cookie present);
http leg unaffected; D-044 override intact; cloud-assert PASS.
Revert: `juju bind openstack-dashboard cluster=metal-internal` + git revert of the
bundle/docs commit.

**Upstream:** bug draft at docs/upstream-bug-draft-dashboard-tls.md (charm renders an
L4-masked dead TLS backend by default in multi-space deployments) -- operator to file.
**Roosevelt:** edge TLS/DNS (D-044 trajectory) supersedes this plumbing; carry the
invariant "cluster binding space == default binding space for TLS-fronted charms" into
the Roosevelt bundle review regardless.

**Related:** D-044, D-072 mechanism in appendix-A, DOCFIX-089 (fail-closed 3.3 check),
BUNDLEFIX-011, changelog addenda 15 + 18.

## D-050 -- RESOLVED (2026-07-06): closed via D-051 (operator ruling)

Operator ruled 2026-07-06: D-050 is CLOSED as resolved by D-051's option (b) -- the
`use-policyd-override=true` flag is no longer a no-op footgun because the reviewed
domain-manager policy zip ships bundle-native (D-051 amendment / DOCFIX-071), with a
FAIL-CLOSED zip-vs-yaml drift check in the deploy gate and behavioral acceptance at
G3 (appendix-C C.4 probe). Closure condition attached by the operator: the exact
settings + policy set for error-free tenant standup must remain clearly defined --
that set is: bundle keystone `use-policyd-override=true` + `resources.policyd-override`
zip (committed beside source), provider-bundle-check content compare, and the C.4
behavioral probe. Any future policy delta rides the same mechanism (see D-073 for the
first such delta). **Resolves:** D-050 (was PROPOSED/OPEN since 2026-06-17).
**Related:** D-051 + amendment, D-064, D-073, FINDING-1.

## D-073: ADOPTED -- tighten identity:list_trusts to the modern keystone default

**Status:** ADOPTED 2026-07-06 (operator-ruled); implementation + testing REQUIRED
before live apply -- nothing applied yet.

**Problem (carried from the D-064 create_trust fix, "PROPOSED/OPEN separate
hardening"):** live `identity:list_trusts = ""` lets ANY authenticated user enumerate
ALL trusts cloud-wide (trustor/trustee/project relationships) -- cross-tenant info
disclosure on a commercial multi-tenant cloud. Newer keystone defaults tighten this to
`rule:admin_required or (role:reader and system_scope:all)`.

**Decision:** adopt the modern default for `identity:list_trusts` (and evaluate the
read-rule siblings `identity:get_trust` / `identity:list_roles_for_trust` at the same
shipped defaults while in there) via the SAME policyd-override zip mechanism as
D-051/D-064 -- policy source edited, zip rebuilt, bundle resource updated, deploy-gate
drift check green, live update via the gated attach-resource path (appendix-C C.3),
behavioral verify: tenant user gets 403 on `openstack trust list`, operator
admin still succeeds, magnum cluster create/delete (trust machinery) unaffected.

**Sequencing (proposed, operator to confirm):** land this BEFORE batch-3 d011-05 so
the P3 isolation acceptance validates the final shipping policy, not a superseded one.

**Related:** D-064 (origin), D-051 (mechanism), D-050 (closed), scs-0302 posture.

## D-073 -- AMENDMENT (2026-07-06): implementation STAGED (zip rebuilt); live apply PENDING

- policies/domain-manager-policy.yaml: `identity:list_trusts` added at the modern
  keystone default VERBATIM (39 rules now). One documented exception to the overlay's
  "no new-style rules" header invariant: the (role:reader and system_scope:all) branch
  is provably inert under this cloud's old-style enforcement -- narrows only, kept
  verbatim for 2024.2+ reconciliation provenance. Trust-rule siblings left UNCHANGED
  (evidence-gated per D-064: code-guarded, and no verified shipped-default strings in
  repo evidence to copy).
- policies/overrides.zip rebuilt via python3 zipfile (no zip(1) on the jumphost);
  provider-bundle-check item 7 content-compare PASS.
- Validated here: yaml.safe_load (39 string rules), ASCII+LF, tooling compatibility
  (tenant-offboard/tenant-acceptance call `trust list` in the admin context only).
  NOT validated here: oslo.policy parse (module absent on the jumphost; D-064's parse
  ran in the main-chat sandbox -- rerun there if desired) and BEHAVIORAL acceptance.
- LIVE APPLY (gated, operator-present; in the batch-3 window do-doc): stage the zip
  under $HOME (snap /tmp limit), `juju attach-resource keystone
  policyd-override=<staged>/overrides.zip`, wait for `PO:`, then behavioral verify:
  tenant-token `openstack trust list` -> 403; admin context -> succeeds; magnum
  cluster create/delete (trust machinery) unaffected. Rollback: re-attach the prior
  zip from git history (`git show HEAD~:policies/overrides.zip`).

## D-072 -- AMENDMENT (2026-07-06): upstream bug NOT filed (operator ruling)

Operator ruled 2026-07-06: the drafted upstream bug (docs/upstream-bug-draft-dashboard-tls.md)
will NOT be submitted -- the root condition is judged a site misconfiguration (cluster
binding pointed at a third space) rather than a charm defect to report. The draft stays
in-repo for history; the addendum-18 OPERATOR ACTION item is withdrawn. The Roosevelt
invariant from D-072 ("cluster binding space == default binding space for TLS-fronted
charms") is UNCHANGED and remains the carry-forward.

## D-073 -- AMENDMENT (2026-07-06): APPLIED live (d011-batch3 window)

The staged 39-rule overrides.zip was attached to keystone in the logged
d011-batch3 window (addendum 27). Settled "PO: Unit is ready". Behavioral
verify per the amendment's block: tenant app-cred context 403 on
identity:list_trusts (verbatim deny); admin trust list OK; magnum trust
machinery re-proven by d011-05 PASS. Independent proof: first live run of the
H4 drift detector (staged) reported live overlay == repo, 39 rules. Status:
ADOPTED -> APPLIED. Rollback stays as recorded (re-attach the HEAD~ zip).

## D-011 -- AMENDMENT (2026-07-06): batch-3 CLOSE -- all acceptance items stand

With the d011-batch3 window (addendum 27): item 4 octavia PASS live (RR +
member failover + amphora failover with N+1 headroom evidence, A2 conditional
pre-approval honored); item 5 magnum e2e PASS live (health + OCCM + P3
isolation via the foil1 tenant). Items 1-3 stood from batches 1-2; item 6
remains MANUAL by design (SEC-003 second-person unseal rehearsal pending --
gates D-011 FULL close); items 7-8 DROPPED per the amended bar. D-011 is
therefore CLOSED-except-item-6. The v1-close checklist (handoff section 6)
is now QUEUED per the do-doc's close-out clause. Checker corrections made
in-window: DOCFIX-095 (tenant-assert owner-context probe), DOCFIX-096
(amphora_headroom live CLI shapes).

## D-074: Tenant CIDRs are tenant-chosen and may overlap (amends D-016)

**Status: ADOPTED 2026-07-06** (operator rulings recorded same day; proposed by
the main-chat stream in docs/tenant-cidr-overlap-correction-PLAN.md).

**Decision.** Tenant private (overlay) CIDRs are tenant-chosen and MAY overlap
across tenants. Non-collision is enforced ONLY against the reserved-ranges
registry, with real subnet-overlap math (containment in either direction),
plus RFC1918 membership (contract section 3 item 4's original intent).
Reserved ranges = the six PLANE_CIDRS /22s (scripts/lib-net.sh, keyed by
plane), the capi-mgmt tenant network (D-035; operator ruled: stays reserved;
resolved live by stable name capi-mgmt-subnet, never hardcoded), and the
metadata range. Default tenant CIDR policy (operator ruled): client-choice
with fallback default 10.100.0.0/24.

**Why.** The stage-4 guard over-enforced relative to the onboarding contract
(rejected tenant-vs-tenant overlap the contract never required) while
under-enforcing what matters (exact-string match missed partial overlaps
against reserved ranges). Overlap-allowed tenant CIDRs is the standard
isolated-multi-tenant model and matches the SCS Domain Manager hard-isolation
persona; per D-035 the management path never routes into tenant overlay space
(an earlier main-chat "CAPI/Magnum reachability" claim is retracted in the
PLAN doc). Gains: contract/implementation alignment, no find-a-free-/24 step,
clients can align to their own on-prem ranges, NetBox uniqueness duty shrinks
to the routed space, the one-/16 (256-tenant) ceiling goes away. Costs: an IP
no longer identifies a tenant (triage needs net+tenant context); no
tenant-private-IP routing between tenants (an anti-pattern here anyway).

**Phase-0 evidence (gates honored).** 0.1: allow_overlapping_ips=True read
live from the neutron-api leader's rendered neutron.conf (2026-07-06). 0.2:
exact-overlap proof executed inside foil1 (operator-ruled vehicle -- the
PLAN's "onboard an overlapping foil" was stale: foil1 already existed and the
old guard blocks exact-overlap onboarding): foil1-overlap-net/-subnet created
at 10.20.28.0/24 == beta-subnet EXACTLY, attached to foil1-router, and a
Magnum cluster (foil1-overlap, 1 master / 1 node, master-lb-enabled) built on
that overlapping network -- see the D-074 amendment recording the terminal
state and evidence path. Proof resources are torn down after evidence.

**Forward-only.** Existing tenants keep their CIDRs (beta 10.20.28.0/24,
foil1 10.20.29.0/24 -- measured 2026-07-06; the PLAN's "northwind
10.20.20.0/24" example was fictional). No re-CIDR; design-decisions forbids
transient-overlap re-CIDR.

**Implements:** guard rewrite in scripts/tenant-onboard.sh stage 4 +
scripts/net_overlap.py (DOCFIX-106); doc alignment (contract section 3,
onboarding runbook, clientdocs intake) rides the same delivery.
**Amends:** D-016 (its /16-pool uniqueness becomes vestigial, not reclaimed).
**Related:** D-035, D-051, D-066.

## D-016 -- AMENDMENT (2026-07-06): uniqueness carve-out superseded by D-074

D-016's single-pool model (carve non-overlapping /24s from 10.20.0.0/16)
stops being enforced forward: D-074 (ADOPTED) makes tenant CIDRs
tenant-chosen and overlap-allowed, guarded only against reserved ranges.
The /16 pool remains as history for existing allocations; nothing is
re-numbered. See D-074.

## D-074 -- AMENDMENT (2026-07-06): Phase 0.2 proof EXECUTED -- overlap+Magnum confirmed

Executed same-day in the logged d074-phase0 session (Claude Code lane).
foil1-overlap-net/-subnet created at 10.20.28.0/24 (EXACT overlap with
beta-subnet, different tenant); Magnum cluster foil1-overlap reached
CREATE_COMPLETE in ~14.5 min (matches beta's ~13.5 min profile); kubeapi
octavia LB ACTIVE/ONLINE; tenant-fetched kubeconfig returned 2 Ready nodes
THROUGH the API-LB with node addresses (10.20.28.229, .54) inside the
overlapping range -- while beta's cluster on the same CIDR served undisturbed.
Both tenants ran Kubernetes on the same /24 simultaneously: the strongest
refutation of the retracted reachability concern. Evidence:
~/openstack-baseline/d074-phase02-evidence-20260706-234352.txt (jumphost).
Proof resources torn down (cluster, template, router-if, subnet, net --
residual check clean); post-teardown cloud-assert PASS. Phase 0 gates are
fully discharged; the D-074 implementation (DOCFIX-106/107) ships.

## D-075: Dashboard VIP root -- redirect to /horizon + no autoindex (ADOPTED+EXECUTED 2026-07-07)

**Status: ADOPTED (operator ruling 2026-07-07, option c) + EXECUTED live same day.**

**Context.** During the devteam handover browser check the operator hit Apache
"Index of /" at https://10.12.4.58/ (provider-public dashboard VIP). Measured:
Horizon mounts at webroot /horizon (WSGIScriptAlias in conf-enabled/
openstack-dashboard.conf); the charm-rendered vhosts define no root handler;
Ubuntu stock apache2.conf grants `Options Indexes` on /var/www/. A directory
listing on a tenant-facing commercial surface is an information-disclosure /
posture defect, and a bare 200 at the root previously masqueraded as a healthy
dashboard (the D-072 as-executed "https 200" at the VIP root was THIS autoindex
page, not Horizon).

**Options presented:** (a) redirect / -> /horizon; (b) disable autoindex at the
root; (c) both. **Operator ruled (c).**

**Decision / mechanism.** Local override /etc/apache2/conf-available/
99-omega-root.conf on the dashboard unit (the charm exposes no config option
for either behavior; charm manages sites-enabled/ but never conf-enabled/, so
the snippet survives config-changed re-renders and in-channel charm refreshes):

    <Directory /var/www/>
        Options -Indexes
        RewriteEngine On
        RewriteRule ^$ %{REQUEST_SCHEME}://%{HTTP_HOST}/horizon [R=302,L]
    </Directory>

**Why per-directory mod_rewrite and NOT RedirectMatch:** first attempt used
`RedirectMatch 302 ^/$ /horizon`; mod_alias absolutizes the Location from the
BACKEND connection port when the client Host header carries none, emitting
http://<VIP>:70/horizon (and :433 on the TLS leg) -- ports haproxy does not
expose. Caught in this window's verify step. The rewrite echoes the client's
own Host header and request scheme, so both haproxy legs redirect correctly.
302 (not 301) so browsers do not cache permanently against a future webroot
change. `Options -Indexes` subtracts from the inherited option list and covers
/var/www/html by inheritance.

**Verification (as-executed):** on-unit root -> 302 + zero "Index of" match;
via VIP http leg: / -> 302 http://10.12.4.58/horizon -> follows to 200 at
/horizon/auth/login/. TLS-leg browser confirmation by operator.

**Durability:** PER-REBUILD -- teardown wipes the unit; reapply each rebuild
via the phase-03 step 3.3 block (DOCFIX-118), same lifecycle as the D-044
cookie override. **Roosevelt:** edge TLS/DNS (D-044 trajectory) supersedes
this plumbing; carry the invariant "tenant-facing roots never serve directory
listings" into the Roosevelt bundle review.

**Revert:** `sudo a2disconf 99-omega-root && sudo systemctl reload apache2`
on the dashboard unit; git-revert the DOCFIX-118 runbook/appendix additions.

**Related:** D-072 (VIP TLS repair; its root-200 evidence reinterpreted),
D-044 (cookie override precedent + Roosevelt edge trajectory), DOCFIX-089
(fail-closed dashboard checks), DOCFIX-118 (runbook + appendix-A delivery).

---

## D-100: Inter-DC network topology and fabric (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling). Instantiates the D-052 per-DC template across two DCs.

**Context:** VR1 is a virtual regional two-DC test environment (DC1, DC2, plus an Office1 headend), all nested libvirt VMs on one vcloud host, rehearsing the future bare-metal multi-DC site (Roosevelt). D-052 already defines the six-plane per-DC template and tags metal-admin DC-local / metal-internal "may-cross-fiber" -- but that label assumed a single stretched cloud.

**Decision:**
- Fabric: a full dark-fiber triangle -- DC1<->DC2, DC1<->Office1, DC2<->Office1 (mesh, not star) -- each modeled as a dedicated virtual L2 segment on vcloud. Each site additionally has its own OPNsense edge with a simulated ISP uplink ("external"), so every site has independent edge access.
- WAN simulation: apply `tc netem` (latency / jitter / loss / rate) on the inter-DC and site-uplink segments, tuned to same-metro dark fiber (low single-digit ms) unless the Roosevelt target says otherwise. The exact parameters are a Phase-0 knob recorded in the buildout doc.
- Plane span (independent-cloud refinement of D-052): because DC1 and DC2 are INDEPENDENT clouds (separate Keystones, no control-plane federation), no control-plane plane crosses the fiber. provider-public, metal-admin, metal-internal, data-tenant, and storage instantiate DC-LOCAL per DC. ONLY the replication plane crosses DC1<->DC2. This refines D-052's "metal-internal may-cross-fiber" (a stretched-cloud notion) to DC-local for the independent-cloud topology.
- Provider-NIC discipline: each DC's provider NIC stays RAW so ovn-chassis builds br-ex and enslaves it by MAC; MAAS must not bridge it. Public API VIPs live on provider-public (D-060 Pattern A). The D-057 provider-vip plane is NOT used (reverted by D-060).

**Rationale / tradeoffs:** mesh (not star) is required because replication is a direct DC1<->DC2 data path -- routing it through Office1 would be a bandwidth SPOF -- while management is Office1<->DC. Per-site ISP edges make each DC independently reachable and feed the per-DC airgap mirror upstream (D-107). Delta to Roosevelt: the dark-fiber-triangle-plus-per-site-edge shape IS the intended Roosevelt shape; the only virtual-only element is that the "fiber" is a vcloud virtual segment with netem standing in for real fiber and optics.

**References:** instantiates D-052 (six-plane per-DC template; independent-cloud cross-fiber refinement); carries the D-057 root-cause br-ex discipline as revised by D-060 (Pattern A); pairs with D-101 (families), D-107 (edge egress), D-108 (replication).

**Sub-items RULED 2026-07-09 (operator):** netem link profile -- same-metro dark-fiber lean CONFIRMED as the default (low single-digit ms, jumbo-capable); re-tune when a specific Roosevelt inter-DC target is known, not before. Office1<->DC fiber carries management traffic only (MAAS/Juju/operator) -- CONFIRMED per the stated lean (not one of the six items the buildout doc listed for explicit redline; adopted with the rest of D-100, flag if this needs separate reconsideration).

---

## D-101: DC-DC address plan and IPv6 family matrix (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling), AMENDED same day: (a) metal-admin gains a ULA leg (reverses this decision's own stated lean, see the family matrix below); (b) D-102 (IPv6 tenant addressing and MTU sub-policy) is MERGED into this decision per operator ruling -- D-102's entry is retained for numbering/history only, see its own Status line. Activates the developed family-follows-reachability IPv6 plan for VR1. NetBox (vcloud-local) is the IPAM authority for all literals.

**Context:** v1 / DC0 is IPv4-only on 10.12.0.0/16 (six planes in `scripts/lib-net.sh`). The developed IPv6 plan (in-project, not yet committed) classifies planes by reachability: GUA for external clients, ULA for internal-only, IPv4 where forced. DC-DC is the phase that activates it.

**Decision:**
- Per-DC distinct v4: DC1 INHERITS the DC0 six-plane v4 layout (provider-public 10.12.4.0/22 ... replication 10.12.36.0/22) unchanged, so DC1 equals the validated template. DC2 gets a DISTINCT, non-overlapping parallel supernet (same six-plane structure; exact prefixes assigned in NetBox, non-colliding with DC1, operator infrastructure, and client VPN ranges per D-074). Distinct ranges keep DR / backup endpoints unambiguous (no NAT hairpin).
- Family matrix (per DC), family follows reachability:
  - IPv6-only ULA: data-tenant (geneve underlay), storage (Ceph public), replication (Ceph cluster, including the cross-DC leg), Octavia lb-mgmt, and P2P links. Internal, no external clients.
  - Dual-stack (IPv4 + ULA v6): the metal-internal API-endpoint VIPs gain a v6 (ULA) address; each DC's Vault issues IPv6-SAN certs (D-109). **AMENDED 2026-07-09 (operator ruling):** metal-admin ALSO joins this dual-stack group (a ULA leg alongside its existing v4) -- reverses this decision's original lean of keeping metal-admin v4-only; PXE itself stays v4-first regardless (see below), the ULA leg is for the admin-API traffic on that plane, not PXE.
  - IPv4 retained: MAAS PXE / provisioning (PXE is v4-first, unaffected by the metal-admin amendment above); and datastore east-west (MySQL / RabbitMQ stay v4-bound -- the plane is dual-stack-capable but v6 there is not chased: marginal benefit, real cert / complexity cost).
  - GUA (ARIN 2602:f3e2::/32, region-0 /36): provider public API VIPs, native v6 ext_net, tenant pool delegation. Enabled here because the simulated OPNsense edge is ours -- the D-015 "upstream-router IPv6 readiness" gate on external GUA does NOT apply in VR1, so VR1 can rehearse the full internal-ULA plus external-GUA end state.
- Replication carrier correction: the DR seed names the cross-DC carrier as v4 10.12.36.0/22. Per this matrix and the operator ruling that replication is not IPv4-bound, the cross-DC replication carrier is IPv6-only ULA (a /64 from the org ULA /48). This SUPERSEDES the DR-seed carrier line. v1 / DC0 `lib-net.sh` v4 values are unchanged (DC-DC assigns families on its own layer).
- Literals: the org ULA /48, the per-DC GUA carve out of the /36, and DC2's v4 supernet are NetBox-authoritative, populated via the extended import pipeline (`netbox/*-import.py` extended for multi-DC and dual-stack). They are NOT hardcoded in this decision.
- **Tenant/MTU sub-policy (folded in from D-102, 2026-07-09 operator ruling):**
  - Tenant v6 routing: native GUA routing, no NAT66 -- the simulated OPNsense edge routes tenant GUA prefixes (prefix delegation or static routes). NAT66 is discouraged and we own the edge, so native is both correct and feasible.
  - Tenant v6 block size: /60 per tenant default (~4080 per cloud); /56 available on request.
  - Tenant address mode: SLAAC default (RA); DHCPv6-stateful available for tenants needing deterministic addressing.
  - MTU / geneve-over-v6: compute the exact byte budget in Phase 0 against the MEASURED host L2 MTU. An IPv6 outer header adds 20 bytes over IPv4 (40 vs 20); geneve-over-v6 overhead is about 40 (IPv6) + 8 (UDP) + 8 (Geneve base) = 56 bytes, before nested-virt and options. Recommendation: raise the vcloud underlay to jumbo (9000) end-to-end so tenant MTU stays 1500 and amphora/geneve fit; if the underlay is pinned at 1500, tenant MTU drops to about 1444 (v6-geneve) and MUST be set consistently across ovn geneve, tenant-network MTU, and amphora. The measured underlay MTU is a Phase-0 gate -- do not assume jumbo.
  - Phase-A rollback posture: go straight to IPv6-only on the internal three (data / storage / replication) rather than dual-stack-then-flip, given plane isolation and the Octavia ULA proof. Dual-then-flip is the conservative alternative if Phase-0 verification of Ceph-over-v6 / geneve-over-v6 is not clean.

**Rationale / tradeoffs:** matches the operator directive (v6 wherever possible, v4 only where forced) and maximizes Roosevelt rehearsal value (proves Ceph-over-v6, geneve-over-v6, native v6 tenant routing). The two forced-v4 points (PXE, datastore east-west) are explicitly justified rather than incidental. Native routing plus SLAAC for tenants is the IPv6-correct default; the MTU budget is the classic nested-OpenStack failure mode, handled by measurement plus jumbo. Ceph-over-v6 (`ms_bind_ipv6`) and OVN geneve-over-v6 remain Phase-0 verification gates in the buildout doc.

**References:** activates the developed IPv6 plan (its Phase A/B/C); carries D-016 / D-074 (tenant CIDR policy); supersedes the DR-seed v4 replication carrier; feeds D-106 (AAAA), D-108 (replication), D-109 (Vault v6-SAN, kept standalone per 2026-07-09 operator ruling -- not merged here). NetBox is IPAM apex.

**Remaining open item:** the exact NetBox literals (org ULA /48, per-DC GUA carve, DC2's v4 supernet) -- pending NetBox assignment (gap #3 in `docs/dc-dc-deployment-workflow.md`'s tooling gap register), not a ratification question.

---

## D-102: IPv6 tenant addressing and MTU sub-policy (VR1)

**Status:** MERGED INTO D-101 (2026-07-09, operator ruling). Content below is retained for numbering/history only, per this repo's append-only decision-record discipline -- D-101 is now the current, authoritative text for this sub-policy. Do not amend the body below; amend D-101 instead.

**Decision (leans; operator to ratify):**
- Tenant v6 routing: native GUA routing, no NAT66 -- the simulated OPNsense edge routes tenant GUA prefixes (prefix delegation or static routes). NAT66 is discouraged and we own the edge, so native is both correct and feasible.
- Tenant v6 block size: /60 per tenant default (~4080 per cloud); /56 available on request.
- Tenant address mode: SLAAC default (RA); DHCPv6-stateful available for tenants needing deterministic addressing.
- MTU / geneve-over-v6: compute the exact byte budget in Phase 0 against the MEASURED host L2 MTU. An IPv6 outer header adds 20 bytes over IPv4 (40 vs 20); geneve-over-v6 overhead is about 40 (IPv6) + 8 (UDP) + 8 (Geneve base) = 56 bytes, before nested-virt and options. Recommendation: raise the vcloud underlay to jumbo (9000) end-to-end so tenant MTU stays 1500 and amphora/geneve fit; if the underlay is pinned at 1500, tenant MTU drops to about 1444 (v6-geneve) and MUST be set consistently across ovn geneve, tenant-network MTU, and amphora. The measured underlay MTU is a Phase-0 gate -- do not assume jumbo.
- Phase-A rollback posture: go straight to IPv6-only on the internal three (data / storage / replication) rather than dual-stack-then-flip, given plane isolation and the Octavia ULA proof. Dual-then-flip is the conservative alternative if Phase-0 verification of Ceph-over-v6 / geneve-over-v6 is not clean.

**Rationale:** matches the developed plan's recommendations; native routing plus SLAAC is the IPv6-correct default; the MTU budget is the classic nested-OpenStack failure mode and is handled by measurement plus jumbo.

**References:** extends D-101. Ceph-over-v6 (`ms_bind_ipv6`) and OVN geneve-over-v6 are Phase-0 verification gates in the buildout doc.

---

## D-103: OpenTofu / MAAS / Juju lifecycle seam (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling). Resolves the double-specified node-VM ownership question.

**Decision:**
- OpenTofu owns create / destroy of: the libvirt domains that stand in for DC "hardware"; all virtual networks (the dark-fiber, plane, and ISP-edge segments); the OPNsense edge VMs; and the Office1 service VMs (MAAS-region, NetBox, GitBucket). OpenTofu runs from the Office1 operator VM against the vcloud host libvirt.
- OpenTofu registers each DC's libvirt host to that DC's MAAS rack controller as a virsh VM-host, so MAAS DISCOVERS the OpenTofu-created node VMs.
- MAAS owns commission / deploy / power / release of those node VMs; it does NOT compose new ones. Juju then deploys the OpenStack bundle onto the MAAS-deployed machines.
- NetBox remains the IPAM apex; MAAS and overlays consume NetBox values, never the reverse.

**Rationale / tradeoffs:** this keeps the entire physical substrate in versioned OpenTofu (auditable, reproducible, and exactly the "operator scripts the deployment from Office1" model), while MAAS owns the provisioning lifecycle -- mirroring Roosevelt, where MAAS enlists hardware that already exists rather than creating it. "MAAS owns lifecycle" here means the provisioning lifecycle, not domain creation; that reconciles the two operator inputs.

**Delta callout (test-env shim):** the OpenTofu "create the node VMs" module has NO Roosevelt analog. In Roosevelt that step is physical racking plus BMC enlistment, and MAAS enlists real hardware. This module is tagged in the buildout doc as a virtual-only shim and must NOT be mistaken for reusable Roosevelt IaC. Everything downstream of enlistment (MAAS commission / deploy, Juju bundle) IS Roosevelt-transferable.

**References:** ratifies MAAS-as-VM-host in the enlist sense; NetBox apex per the operating skill and environment reference; feeds Phases 2-4 of the buildout doc.

---

## D-104: Juju controller topology and HA posture (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling). Adds the controller-topology / HA posture the DC-DC phase was gated on. References D-071 (patch cadence); does NOT amend it.

**Decision:**
- One Juju controller PER DC, deployed in-DC (on a MAAS-deployed machine in that DC), each managing only its own cloud. DR-independent: a controller loss affects only its DC.
- Single-unit (not HA) for VR1, with the confirmed backup posture (juju 3.6 `create-backup` / `download-backup`). Controller backups scheduled and a restore drill run in Phase 5.
- HA (3-unit) DEFERRED to Roosevelt, when bare-metal resources allow. Recorded as a known VR1 test-only posture, not a Roosevelt recommendation.
- Office1 hosts NO OpenStack Juju controller (Office1 = MAAS-region + OpenTofu + NetBox + GitBucket only).

**Rationale / tradeoffs:** per-DC single controller is DR-honest (independence) and resource-frugal on the 256 vCPU / 1 TiB / 10 TiB host. A shared or Office1-hosted controller would couple both DCs' manageability -- a controller outage would take out both -- the opposite of what a DR drill should prove. Single-vs-HA is a deliberate test-only economy; backups are the mitigation and are drilled. Roosevelt scales to 3-unit HA per DC.

**References:** relates to D-071 (routine update cadence / Juju controller patch policy) without amending it; interacts with D-108 (per-DC control-plane backup / restore in the DR drill).

---

## D-105: CAPI management-cluster placement and COS / Magnum scope (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling). Carries D-035 to the two-DC topology.

**Decision:**
- Each DC runs its OWN in-cloud, single-homed capi-mgmt cluster (per D-035), independent per DC. No shared or Office1 management cluster.
- COS (observability) and Magnum / CAPI are in scope in BOTH DCs.
- The Magnum `domain-setup` action is re-run after every teardown / redeploy per DC (D-046 trap: magnum reports "Unit is ready" regardless).

**Rationale:** D-035 already placed capi-mgmt in-cloud single-homed; per-DC keeps each cloud self-contained and DR-independent. COS per DC (not central at Office1) keeps observability inside each independent failure domain.

**References:** carries D-035; carries D-046 (domain-setup re-run); COS / Magnum scope confirmed by operator.

**Sub-item RULED 2026-07-09 (operator):** COS is per-DC-only for VR1, CONFIRMED (no Office1 roll-up) -- revisit for Roosevelt.

---

## D-106: Designate reactivation (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling). SUPERSEDES D-019 (Designate deferral); REACTIVATES D-008 (DNS architecture).

**Decision:**
- Designate is deployed in-bundle in each DC, backend `designate-bind` (bind9, the Charmed-OpenStack-native backend, lowest delta).
- Naming per D-008 instantiated with the VR1 labels: `<service>.omega.dc1.vr1.cloud.neumatrix.local` and `...dc2.vr1...`. Per-DC INDEPENDENT zones (independent Keystones), served with A and AAAA records (AAAA for the v6-enabled planes per D-101).
- The D-008 bootstrap order applies per DC: static /etc/hosts bootstrap -> `os-public-hostname` per API charm -> Vault issues FQDN-SAN certs -> Designate zone plus A/AAAA -> Neutron `default_dns_domain` and `dns_servers` -> tenant subnets with `--dns-nameserver`.

**Rationale / consequence:** reactivating Designate plus FQDN-SAN certs removes the D-019 documented root cause (metal-only charms pulling the public FQDN endpoint they cannot resolve -- the D-021 amphora-pipeline constraint). The D-008 convention was designed multi-DC from the start (`dc0.vr0`), so this is reuse, not new design -- minimal delta.

**References:** supersedes D-019; reactivates D-008; resolves the D-019 / D-021 root cause; AAAA depends on D-101. Office1 headend services use a region-level subdomain outside the per-cloud convention (e.g. `maas.office1.vr1...`) -- exact scheme is a buildout-doc detail.

**Sub-item adopted per its stated lean (2026-07-09):** cross-DC name resolution (a DC1 tenant resolving a DC2 name) is NOT needed for VR1 (independent clouds) -- this was not one of the six items the buildout doc listed for explicit operator redline, so it is adopted along with the rest of D-106 rather than separately re-asked; revisit only if a federated-service requirement emerges.

---

## D-107: Airgap posture, per-DC artifact mirror, and NTP (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling).

**Decision:**
- Airgap is enforced at the NODE level: OpenStack nodes (control / compute / ceph) reach NO internet directly. They pull all artifacts (apt, snap, charmhub, container registry, images including amphora) ONLY from an in-DC mirror.
- PER-DC mirror (not a single Office1 mirror): each DC hosts its own mirror so a DC can redeploy independently even if Office1 or the peer DC is down -- a DR requirement the drill exercises.
- Controlled egress lives on each DC's OPNsense edge, narrowly scoped: NTP-only, plus the mirror's upstream sync. Nodes get time (chrony) from their DC edge, which syncs upstream to internet NTP pools. Office1 is NOT in the core-service path (no NTP, no node artifacts served from Office1).

**Rationale / tradeoffs:** this reconciles "airgap-honest" with the operator's "each DC syncs internet NTP" -- the airgap is at the node boundary, not the site boundary, and the edge holds a minimal controlled egress. Per-DC mirrors cost more disk (a factor in the D-101 / Ceph disk budget) but buy true DC-redeploy-independence, which a single Office1 mirror cannot. The Roosevelt airgap posture is the same shape (per-DC mirror, edge-controlled egress).

**References:** pairs with D-100 (per-site ISP edge), D-101 (disk budget), and the Phase-0 disk math.

**Sub-item RULED 2026-07-09 (operator):** mirror sync topology is independent upstream pull per DC, CONFIRMED (not one DC pulling and seeding the peer) -- for full DC-redeploy-independence; measure the disk cost in Phase 0.

---

## D-108: Cross-DC replication mechanism (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling). Engineers the mechanism for the SETTLED Tier-A DR decision (`docs/dc-dc-replication-DR-seed.md`); does not reopen the posture.

**Decision (mechanism):**
- Cinder volumes: `cinder-backup` to a cross-DC-replicated object store via `ceph-radosgw` MULTISITE (a zonegroup with the two DCs as zones; bucket-level bidirectional sync) as the replicated backup target. A restore reconstitutes the volume with its Cinder metadata in the peer -- the honest cross-independent-cloud path. Multisite is chosen over a single mirrored bucket for bidirectional, per-object async replication with conflict handling.
- Glance images: `ceph-rbd-mirror` of the Glance pool, snapshot-based, two-way, with a re-register step on failover. Daemon topology: one rbd-mirror daemon per DC, peering over the replication plane; daemon HA is single-unit for VR1 (matches the D-104 controller economy), noted for Roosevelt scale-up.
- Nova ephemeral: NOT replicated (rebuild from image plus volume).
- Carrier: the cross-DC leg (radosgw multisite sync plus rbd-mirror peer traffic) rides the IPv6-only ULA replication plane (per D-101; corrects the DR-seed v4 carrier). Explicit inter-DC route plus a bandwidth budget (MTU-aware; see D-102), distinct from intra-DC OSD cluster traffic.
- Consistency groups for multi-volume apps where a crash-consistent set matters.
- Staging: prove one-way (single mechanism, single direction) plus a clean failover / failback drill FIRST, then enable two-way. RPO ~15 min tunable; RTO is the measured drill output, not an SLA.
- Neutron state (ports / FIPs / security groups) is NOT replicated; it is recreated on failover per the runbook.

**Rationale:** matches the settled seed; multisite plus rbd-mirror are the delta-minimal Ceph-native primitives. This answers the seed's open questions (multisite vs mirrored bucket -> multisite; daemon HA -> single-unit VR1; peer bootstrap -> on the replication plane).

**References:** engineers `docs/dc-dc-replication-DR-seed.md`; carrier per D-101; interacts with D-104 (per-DC control-plane backup / restore in the drill); the failover / failback skeleton is fleshed out in the buildout doc.

---

## D-109: Vault posture for VR1

**Status:** ADOPTED 2026-07-09 (operator ruling). Confirmed to STAND ALONE, not merged into D-101 despite D-102 folding in the same day -- Vault/security posture is a distinct-enough surface to keep independently amendable.

**Decision:**
- Per-DC INDEPENDENT Vault roots (each DC's Vault is its own root CA; no regional root-of-trust this phase).
- Vault pinned 1.8/stable (the only viable charm lineage for this stack; the 1.16 operator lineage is ruled out per D-068). The D-068 EOL-substrate change is deliberately HELD OUT of VR1 to keep variables constant.
- Each DC's Vault issues certs with both IPv4 and IPv6 (ULA) SANs for the dual-stacked metal-internal API endpoints (per D-101).

**Rationale / tradeoffs:** independent roots keep the two clouds cryptographically independent (DR-honest) and match the confirmed operator ruling. Holding 1.8/stable isolates the DC-DC variables from the unresolved substrate modernization; the deliberate non-change is recorded against D-068 so it is on the record, not an oversight.

**References:** references D-068 (substrate hardening deferred to Roosevelt; getting off EOL 1.8.8 stays open there); v6-SAN per D-101.

---

## D-110: DC-DC numbering and repo-seed policy

**Status:** ADOPTED 2026-07-09 (operator ruling; meta / process).

**Decision:**
- The DC-DC decision series starts at D-100 in the repo `openstack-caracal-dc-dc`, seeded from `openstack-caracal-ipv4` at commit 8813efc (verified: new-repo HEAD equals 8813efc; `design-decisions.md` ceils at D-075).
- D-076 through D-099 are RESERVED as a source-project band and are NEVER assigned in this repo. (D-076 is claimed by the source-project dashboard workstream; 077-099 are runway.)
- A genuinely-shared future source-project decision is referenced by its original id (e.g. "see openstack-caracal-ipv4 D-0NN"), never copied and renumbered into this repo.
- `ledger-scan` is confirmed 3-digit-safe for the D-series (`grep -oE '^## D-[0-9]+' | sort -n`), so D-100+ next-free is correct. The deprecated `repo-lint` next-free path is band-limited above 099 and must not be used for next-free (standing rule).

**Rationale:** a round D-100 floor gives a clean project base plus about 24 numbers of cross-repo collision runway (the source is a maturing project unlikely to reach 100); reference-not-copy preserves provenance. A distinct prefix (DD-) was considered for permanent cross-repo immunity and declined for VR1 -- it would fork `ledger-scan`, the fence validator, and every D-NNN cross-reference, against minimize-delta.

**References:** process entry; governs all D-1NN entries in this repo.

**Log-for-sweep (not actioned):** `ledger-scan` DOCFIX / BUNDLEFIX patterns use `[0-9]{3}` -- safe through 999, latent at the 1000 boundary (the same class as the 100-boundary bug already fixed for those series).

---
