# Design Decisions — VR0 DC0 Omega Cloud

This document is the architectural record for the VR0 DC0 testcloud rebuild. Every decision listed here has been deliberately made and discussed; it is not a wishlist or a brainstorm. If a decision is changed, this document is updated and the change is committed with a referencing message.

**Scope split:** This repository implements **v1 (IPv4-only)**. Several decisions below are tagged with **[v2-scope]** — they remain valid design intent but are deferred to a future v2 deployment when upstream router infrastructure supports IPv6. See **D-015** for the v1/v2 fork record.

**Class tags (GA-R3, ratified 2026-07-18; one-time review ruled 2026-07-19).** Every register
entry carries a one-word class tag in its header: **[ARCH]** (architectural consequence beyond
the current stage, a Roosevelt-delta implication per the A1 test, or supersession of an existing
D-number) or **[OPS]** (operational choice recorded here historically; new OPS items get runbook/
as-built entries, NEVER a new D-number -- GA-R3 rule 2). New entries are ARCH by construction;
doubt resolves DOWN to OPS. Entries D-001..D-120 predate the rule and are tagged opportunistically,
not retroactively swept.

---


## D-001: Deployment target paradigm

**Decision:** Path 2A — Charmed OpenStack Caracal (2024.1) via Juju bundle.

**Alternatives considered:**

- Path 2B — Canonical Sunbeam (microk8s-based). Rejected: discards most of the test-cloud experience accumulated to date; different operator paradigm.
- Path 1 — Stay on Bobcat 2023.2. Rejected: defeats the purpose of a Caracal rehearsal ahead of Roosevelt bare-metal.


**Consequences:**

- Bundle-based deployment, suitable for both KVM testcloud and bare-metal scale.
- Caracal-stable channel matrix applies (see D-002).
- EOL date is April 2027 (Caracal upstream support window).

---


## D-002: Channel pinning matrix

**Decision:** Pin every charm to a Caracal-stable channel. No OVN pinning on testcloud (Roosevelt will pin via `ovn-source`).

| Charm group                                                                                                             | Channel                    |
| ----------------------------------------------------------------------------------------------------------------------- | -------------------------- |
| OpenStack core (keystone, glance, nova-\*, neutron-api, cinder, placement, octavia, barbican, designate, magnum, vault) | `2024.1/stable`            |
| OVN (ovn-central, ovn-chassis, ovn-dedicated-chassis-octavia)                                                           | `24.03/stable`             |
| Ceph (ceph-mon, ceph-osd, ceph-radosgw if used)                                                                         | `squid/stable` (see D-005) |
| MySQL (mysql-innodb-cluster, mysql-router subordinates)                                                                 | `8.0/stable`               |
| RabbitMQ                                                                                                                | `3.9/stable`               |
| Vault                                                                                                                   | `1.8/stable`               |
| hacluster                                                                                                               | `2.4/stable`               |
| etcd, easyrsa                                                                                                           | `latest/stable`            |


**Verification source:** Caracal channel matrix per Canonical Charmed OpenStack Charm Delivery table, verified against Charmhub 2026-05-22 (keystone rev 778, ovn-central rev 332, hacluster `2.4/stable`). Canonical advises against `latest/stable` for OS-stack charms due to non-determinism on charm upgrades; hacluster is therefore pinned to its stable channel. Verify against Charmhub before each deploy via `scripts/pre-flight-checks.sh`.

---


## D-003: Network architecture — Option B

**Decision:** Provider network carries BOTH ext_net (tenant FIPs + SNAT egress) AND OpenStack public API VIPs on the same L2 segment.

**Rationale:** During Magnum CAPI Phase 3 on the Bobcat testcloud, OCCM crashloop was traced to tenant networks being unable to reach OpenStack API endpoints — the libvirt FORWARD chain rejected cross-bridge packets between provider (virbr1) and metal (virbr2) bridges. With API VIPs on metal, tenant workloads cannot reach them. Putting API VIPs on the same network as the FIPs makes the API path tenant-reachable by construction.

**Address space layout for v1 (IPv4-only):**

| Range                       | Purpose                                                  |
| --------------------------- | -------------------------------------------------------- |
| `10.12.4.10 – 10.12.4.223`  | Neutron FIP pool                                         |
| `10.12.4.224 – 10.12.4.254` | Charm API VIPs (excluded from Neutron allocation\_pools) |


The Provider `/22` (`10.12.4.0/22`) carries both ranges within a single Neutron subnet. Neutron `allocation_pools` MUST exclude the API VIP range.

**v2-scope extension:** IPv6 Provider subnet adds parallel FIP and API VIP IPv6 IP Ranges within a single `/64`. See D-004.

---


## D-004 [v2-scope]: Dual-stack vs IPv6-only matrix

**Decision (v2-scope, NOT for v1):** Network role determines address family. IPv6 preferred; IPv6-only where the network has no external clients.

**v1 reality:** All networks are IPv4-only on the existing MAAS-provisioned layout. This matrix becomes active in v2.

| Role                       | IPv4 (v1)     | IPv4 (v2) | IPv6 (v2) | Reasoning                                           |
| -------------------------- | ------------- | --------- | --------- | --------------------------------------------------- |
| Metal                      | ✓             | ✓         | ✓         | Charm-to-charm; MAAS PXE IPv4-first                 |
| Provider                   | ✓             | ✓         | ✓         | Tenant FIPs need IPv4; API VIPs reachable from both |
| Data (Geneve underlay)     | ✓             | —         | ✓         | v2: no external clients; underlay agnostic          |
| Storage (Ceph public)      | ✓             | —         | ✓         | v2: `ms-bind-ipv6: true`; no external clients       |
| Replication (Ceph cluster) | ✓             | —         | ✓         | v2: internal OSD↔OSD only                           |
| LBaaS Management           | ✓             | ✓         | ✓         | Amphora image compatibility                         |
| OOB                        | n/a           | n/a       | n/a       | Bare-metal-only concern                             |
| OpenStack Tenant pool      | ✓ (v1: D-016) | —         | ✓         | v1 IPv4 hybrid; v2 IPv6 modeled                     |

---


## D-004a [v2-scope]: Host management → Metal

**Decision (v2-scope, NOT for v1):** Under v2, openstack0-3 host management IPs move from storage (`10.12.16.40-.43`) to Metal (`10.12.8.0/22`) when Storage becomes IPv6-only. v1 keeps host management on storage.

---


## D-005: Ceph release

**Decision:** Squid (Ceph 19, released October 2024).

**Rationale:** Matches Caracal default; one fewer source override in bundle; rehearses what Roosevelt will run. If Squid has rough edges, the testcloud is the place to find them, not production.

**Alternatives considered:**

- Reef (Ceph 18) — current on Bobcat testcloud; lower risk; would require `source: cloud:jammy-caracal` override on ceph-mon/ceph-osd while keeping `reef/stable` channel. Rejected: defeats the rehearsal purpose.

---


## D-006: Vault HA backend

**Decision:** etcd + easyrsa, per Canonical Charmed Vault HA docs.

**Rationale:** This is the documented charm path. The chicken-and-egg TLS dependency (Vault needs certs to start, but Vault issues certs) is resolved by easyrsa bootstrapping the etcd cluster's TLS, after which Vault relations to etcd come up cleanly.

**Topology on testcloud (v1):** Vault num_units=1 + hacluster relation (decorative; documents the relation pattern). Vault HA quorum is not actually exercised at testcloud scale.

**Topology on Roosevelt:** Vault num_units=3 + hacluster on metal space; etcd num_units=3; easyrsa num_units=1.

---


## D-007: Magnum inclusion (Layer A current; Layer B mechanism/topology superseded -- see D-035 / D-037 / D-042)

**Decision:** Magnum in bundle from day one. Two-layer install.

**Layer A — Bundle:**

- `magnum` charm
- `magnum-mysql-router` subordinate
- `magnum-dashboard` subordinate
- Standard relations: keystone, mysql-innodb-cluster (via router), rabbitmq-server, vault (certificates), openstack-dashboard
- Binding: `public: provider` with VIP on provider API VIP range
- Hacluster relation included (decorative on testcloud)


**Layer B — Post-deploy runbook (`runbooks/05-magnum-capi-driver.md`):**

- `juju run magnum/leader domain-setup --wait=10m`
- pip install `magnum-capi-helm==1.1.0` from PyPI into the magnum charm venv with `--break-system-packages` (stackhpc/magnum-capi-helm fork archived Dec 2024; canonical project moved to `openstack/magnum-capi-helm` on opendev/PyPI; 1.1.0 is the last Caracal-cycle release. Upstream tests against Magnum 2023.1+, so backward-compatible through Caracal 2024.1.)
- Deploy `/etc/magnum/kubeconfig` pointing at the **workload cluster** (the post-pivot home of CAPI controllers per **runbook 04a §17** `clusterctl move`). Staged on jumphost at `$HOME/magnum-capi/capi-mgmt-cluster.kubeconfig` by runbook 04a §19, transferred to the magnum unit by runbook 05 §6. Bobcat had this pointing at bootstrap k3s because the pivot was never executed; workstream 3b (2026-05-22) made the pivot mandatory.
- Systemd override replacing init.d ExecStart to load `--config-dir`
- `/etc/magnum/magnum.conf.d/99-capi.conf` setting `enabled_drivers=k8s_capi_helm_v1` and `[capi_helm] kubeconfig_file=/etc/magnum/kubeconfig` (ASCII-only; non-ASCII characters in conf.d cause silent daemon failures)


**CAPI mgmt plane:** Post-pivot, the workload cluster IS the CAPI management plane (per **runbook 04a §17**, `clusterctl move` pivots cluster state from the `capi-mgmt.maas` bootstrap k3s into the workload cluster, which becomes self-managing). Per **D-017**, both the bootstrap k3s and the workload cluster are rebuilt from scratch every deployment cycle — there is no preserved-across-rebuild artifact. The bootstrap install + pivot procedure lives in `runbooks/04a-capi-bootstrap-cluster.md` and runs **before** this runbook. This pattern transfers to Roosevelt unchanged.

**Superseded portions:** The "preserved across rebuild" stance in earlier drafts of this decision is **superseded by D-017**. See D-017 for rationale. The earlier `stackhpc/magnum-capi-helm` v0.13.0 driver pin is superseded by the `openstack/magnum-capi-helm` 1.1.0 pin above (driver source repo moved + archived). The Layer B *mechanism and topology* are now further superseded: the CAPI management plane is an in-cloud single-homed VM with NO `clusterctl move` -- the kubeconfig points at that mgmt cluster, not a post-pivot workload cluster (**D-035**); the conductor graft is `/etc/default/magnum-conductor` `DAEMON_ARGS --config-dir` + `00-capi-helm.conf`, NOT a systemd-ExecStart override + `99-capi.conf` (**D-037**); the driver pin is **1.4.0** for CAPI-core contract coherence (**D-042**); and the per-cluster app-cred replaces any static `capo` credential (**D-039**). Layer A (the bundle) is current. Live deploy steps are runbooks/phase-06 (mgmt cluster) and phase-07 (conductor graft + `domain-setup` D-046 + keystone-v3 drop-in D-047); `runbooks/05-magnum-capi-driver.md` is historical.

---


## D-008: DNS architecture

**Decision:** Layered — static /etc/hosts for bootstrap + Designate (in bundle from day one) for tenant-level resolution.

**Naming convention:**

```
<service>.<cloud>.<dc>.<region>.cloud.neumatrix.local
```
Examples:

- `keystone.omega.dc0.vr0.cloud.neumatrix.local`
- `nova.omega.dc0.vr0.cloud.neumatrix.local`


**Bootstrap order:**

1. Static `/etc/hosts` on jumphost + all openstack0-3 hosts + all LXD containers
2. Bundle deploys with `os-public-hostname: <fqdn>` per API charm
3. Vault issues certs with FQDN in SAN
4. Post-deploy: Designate zone created, A records populated (v1: A records only; v2 adds AAAA records)
5. Neutron `default_dns_domain` and `dns_servers` configured to point at Designate
6. Tenant subnets created with `--dns-nameserver <designate-vip>`

---


## D-009: Hacluster modeling at testcloud scale

**Decision:** Include hacluster + VIP relations at num_units=1 across all HA-eligible API charms.

**Rationale:** Decorative at testcloud scale (a single unit can't form a real HA quorum). Documents the relation pattern so Roosevelt scale-up is mechanical: change `num_units: 1` → `num_units: 3` and rerun.

**Charms with hacluster relation:** keystone, glance, neutron-api, nova-cloud-controller, placement, openstack-dashboard, cinder, octavia, barbican, magnum, vault, designate.

---


## D-010: NetBox-upstream policy

**Decision:** NetBox is the single source of truth for IPAM at the **role and cloud-level pool** layer. Per-project tenant subnets are exempt under the hybrid model (D-016).

**Note (DOCFIX-195, operator ruling 2026-07-15 -- VR1 phase):** During VR1 `office1-netbox` (10.10.1.10) HOLDS THE IPAM APEX ROLE: it is the authoritative NetBox every consuming system (OpenTofu, MAAS, overlays, the Juju bundle) derives its values from. It is a *working draft* (edited as the buildout proceeds), but it is the apex. `netbox.baldurkeep.com` is the eventual PRODUCTION apex, held during VR1 as a READ-ONLY v1 reference draft that takes NO write and is NOT consulted by VR1 systems. Merging the validated `office1-netbox` dataset back into `netbox.baldurkeep.com` is DEFERRED to end-of-deployment (a maybe). This DEFERS the apex write; it does NOT revoke D-010 -- `netbox.baldurkeep.com` remains the single eventual source of truth. See `docs/dc-dc-netbox-buildout-scope.md` section 8 and close-out C2 in `docs/dc-dc-deployment-workflow.md`.

**Workflow:** Update NetBox → update bundle/overlay → commit both with cross-reference.

**Standing imports for v1 (gating the bundle):**

- VR0 DC0 site exists in NetBox ✓
- IPv4 prefixes for v1: Metal /22, Provider /22, LBaaS Mgmt /22 (via `netbox/ipv4-prefixes-import.py`) — **pending**
- Provider IP Ranges for FIPs and API VIPs (same script) — **pending**
- IPv4 tenant pool /16 (same script, per D-016) — **pending**
- IPv6 entries marked as Reservation status (via `netbox/ipv6-mark-reserved.py`) — **pending**


**Deferred to v2 (per Q2):** VR0 DC0-VLANs group additions beyond VID 240 (already imported during prior session work). MAAS currently uses untagged-per-fabric; modeling additional VLANs in NetBox without corresponding network-side tagging would be misleading documentation.

---


## D-011: Validation bar — Roosevelt-rehearsal level

**Decision:** Deployment is not considered successful until all of the following pass:

1. All charms `active/idle` in `juju status`
2. API reachability from jumphost (all public VIPs respond on hostname)
3. API reachability from a tenant VM (Option B verification)
4. Octavia LB pattern re-passes (round-robin, failover, recovery — per Bobcat v3 work)
5. End-to-end Magnum CAPI cluster creation succeeds, including OCCM not crash-looping
6. Vault unseal + auto-unseal-after-reboot pattern verified
7. KVM snapshot baseline taken (Phase 5)
8. Designate zones populated and tenant VMs resolve API hostnames


Validation script: `scripts/validate.sh` (TBD).

---


## D-012: Snapshot strategy

**Decision:** Two baseline snapshots.

- **Snapshot 1:** Post-deploy, post-validation, pre-tenant-resources. Clean cloud state — what a fresh install looks like.
- **Snapshot 2:** Post-tenant-setup. Includes domain1, project1, user1, openrc, flavors, base images (noble-amd64), keypair. Restore point for tenant work.


Snapshots are KVM/qcow2-level on the jumphost hypervisor. Per-VM.

---


## D-013: ~~Clean teardown of existing capi-mgmt~~ (SUPERSEDED by D-018)

**Original decision:** Before destroying the OpenStack model, gracefully delete the CAPI workload cluster on capi-mgmt.maas to allow OpenStack resources (LBs, FIPs, volumes) to be cleaned up properly by CAPI controllers.

**Original steps:** `kubectl delete cluster capi-mgmt-cluster` → wait for CAPI to clean up tenant-side OpenStack resources → `juju destroy-model openstack --destroy-storage --no-prompt`.

**Original "preserved across rebuild" claim:** capi-mgmt.maas bootstrap k3s + CAPI controllers re-used as the Magnum CAPI mgmt plane post-deploy.

**Status:** Superseded. See **D-018** for the replacement teardown strategy (MAAS-release-direct, skip graceful) and **D-017** for the replacement bootstrap cluster lifecycle (full rebuild every cycle, nothing preserved).

---


## D-014: Repository storage location and naming

**Decision:** Self-hosted GitBucket at `git.baldurkeep.com`.

**Repo path:** `jesse.austin/openstack-caracal-ipv4` (v1; IPv4-only).

**v2 repository:** TBD when v2 work begins. Two viable paths: sibling repo `openstack-caracal-ipv6` or `openstack-caracal-dualstack`, OR `v2` branch in this repo with an `overlays/v2-dualstack.yaml`. The single-repo-with-branch approach preserves history of what changed v1→v2 together; the sibling-repo approach keeps v1 frozen as a reference once v2 is in motion.

**Branching strategy:** `main` is canonical. Per-phase work in feature branches when a deploy is in progress; merge back to `main` at successful validation.

---


## D-015: v1 / v2 Fork

**Decision:** Caracal testcloud ships in two iterations.

**v1 (this repository, `openstack-caracal-ipv4`):** IPv4-only Caracal on existing MAAS-provisioned network layout. Proves the bundle, Option B binding fix, Magnum CAPI graft, Designate-from-day-one, hacluster relation pattern, and validation framework. Ships first.

**v2 (deferred):** Adds IPv6 / dual-stack per D-004. Requires upstream router infrastructure to be IPv6-capable, which is not currently the case in this environment. v2 work begins after v1 validation passes AND router-side IPv6 is in place.

**Rationale:** Decoupling the OpenStack-side rebuild from the network-side IPv6 readiness lets us prove the more-important architectural fix (Option B) without waiting on infrastructure work outside the OpenStack deployment's control. The IPv6 design intent is preserved as NetBox Reservation-status entries (per D-010 and `netbox/ipv6-mark-reserved.py`).

**v1→v2 migration scope (forward-look):**

- Re-IP roles per D-004 (add IPv6 sibling to Metal/Provider/LBaaS; move Data/Storage/Replication to IPv6-only)
- Move host management IPs from storage to Metal (D-004a)
- Re-bind charms to listen on both families where dual-stack
- Add AAAA records to Designate zones
- Add tenant IPv6 pool carve-outs

---


## D-016: IPv4 tenant pool — hybrid model (v1)

**Decision:** NetBox owns one upstream IPv4 tenant pool prefix for VR0 DC0. Per-project tenant subnets are Neutron-managed within that pool and are NOT modeled in NetBox.

**Pool allocation:** `10.20.0.0/16` (default; configurable in `netbox/ipv4-prefixes-import.py`). 65,536 addresses; 256 `/24`s available for per-project tenant subnets. Modeled under VR0 DC0 with role `openstack-tenant`.

**Per-project allocation pattern (operationally):**

When a project is created, allocate a /24 from the pool. Operator records the allocation in tenant-setup runbook output but does NOT create a NetBox prefix entry for it. Suggested convention: `10.20.<project-index>.0/24`, starting with `10.20.1.0/24` for project1, etc.

**Rationale (Option C from the discussion):**

- Option A (NetBox-modeled per-project) — full IPAM rigor; high friction for tenant lifecycle; round-trips to NetBox for ephemeral tenants.
- Option B (Neutron-only, no NetBox standing) — minimum friction; loses upstream visibility of total tenant footprint; violates D-010 in spirit.
- Option C (hybrid, chosen) — NetBox documents what space is reserved for tenants and prevents accidental collision with infra ranges; Neutron owns the lifecycle of individual tenant subnets without NetBox round-trips.


**Constraint:** Tenant CIDRs MUST be within the pool. The pre-flight checklist (`scripts/pre-flight-checks.sh`) should assert that proposed tenant subnets fall within the modeled pool.

**v2-scope counterpart:** IPv6 tenant pool `2602:f3e2:ff:0::/56` (NetBox-modeled, Reservation status in v1) becomes active in v2 with the same hybrid model — pool has NetBox standing, per-project IPv6 subnets Neutron-managed.

---


## D-017: CAPI bootstrap cluster lifecycle (bootstrap mechanism superseded by D-035; full-rebuild principle retained)

**Decision:** L3 full teardown and rebuild every deployment cycle. The `capi-mgmt.maas` MAAS VM is released back to Ready state on teardown; on rebuild, it is re-deployed from scratch with Ubuntu 24.04, k3s, CAPI controllers, and ORC. **Nothing is preserved across cycles.**

**Rationale:**

- Rehearsal-first principle. If the bootstrap-cluster install procedure isn't documented and rehearsed, the runbook doesn't exist; if the runbook doesn't exist, surprises surface on Roosevelt.
- Self-imposed forcing function. Every rebuild exercises the full path: MAAS deploy → Ubuntu cloud-init → Vault CA install → k3s install with correct bind-address/SAN flags → kubeconfig server-URL rewrite → helm + clusterctl install → clusterctl init with canonical-kubernetes provider URLs → ORC install → cloud-side prep → cluster manifest render → apply → poll-to-Ready → kubeconfig copy.
- Disposability test. The Bobcat experience proved no critical state lives on capi-mgmt that isn't reproducible from the runbook and the OpenStack cloud. Wiping is safe.


**Runbook:** `runbooks/04a-capi-bootstrap-cluster.md` documents the install sequence in full. It runs **after** `02-deploy.md` (OpenStack cloud up) and **before** `05-magnum-capi-driver.md` (driver graft, which needs the bootstrap k3s kubeconfig).

**Supersedes:** the "preserved across rebuild" stance in earlier drafts of D-007 and D-013.

**Superseded by:** **D-035** retired the bootstrap MECHANISM -- there is no `capi-mgmt.maas` MAAS VM, no k3s, and no `clusterctl move`/pivot; the management plane is an in-cloud single-homed tenant VM (`capi-mgmt-v2`, k8s-snap) built in runbooks/phase-06. The full-teardown-and-rebuild-every-cycle PRINCIPLE stated here is RETAINED (now realized by phase-00 teardown + the phase-06 rebuild). `runbooks/04a-capi-bootstrap-cluster.md` and `05-magnum-capi-driver.md` are historical (folded into phase-06/07).

**Alternatives considered:**

- L1: Wipe just the cluster CRs, keep k3s + controllers. Rejected: skips the install rehearsal that's the whole point.
- L2: Wipe just the controllers, keep k3s. Rejected: same reason; the `clusterctl init` step is exactly the surface that needs rehearsing.
- L3 (chosen): Full wipe including the VM.

---


## D-018: Teardown strategy — skip graceful, release MAAS directly

**Decision:** On teardown, do not pursue graceful CAPI workload deletion or graceful OpenStack model destroy. Instead:

1. (Optional) Capture pre-destroy state for reference
2. `juju destroy-model openstack --force --no-wait --destroy-storage --no-prompt` (background)
3. MAAS release all 5 VMs (openstack0, openstack1, openstack2, openstack3, capi-mgmt) → Ready (parallel)
4. Verify both sides


**Rationale:**

- The rebuild's goal is rehearsing the Roosevelt deploy path. Roosevelt starts from MAAS-Ready bare-metal machines. The most faithful rehearsal is teardown-to-MAAS-Ready.
- Graceful CAPI workload teardown rehearses a different procedure (production cluster decommissioning) that doesn't transfer to Roosevelt's initial deploy.
- `juju destroy-model --destroy-storage` can hang on stuck hooks and leave partial state. `--force --no-wait` plus MAAS release is more reliable.
- Cloud-side OpenStack data (Keystone projects, Neutron networks, Glance images, app credentials) lives in MySQL on the openstack0-3 hosts. MAAS release wipes those hosts, so no separate cloud-side cleanup is needed.


**What is lost vs. graceful path:** verified-clean release path for CAPI workload resources (Octavia LBs, FIPs, CAPO-managed networks). All of these are destined for obliteration anyway; the loss is theoretical.

**What is gained:** ~30+ minutes saved; cleaner end-state guarantee; better Roosevelt rehearsal fidelity.

**Supersedes:** D-013.

**Runbook:** `runbooks/01-destroy-model.md` documents the four phases.

---


## D-019: Cloud DNS (Designate) deferred to v2 / Roosevelt

**Decision:** v1 ships with NO cloud-internal DNS; Designate is not deployed. Public service endpoints use FQDNs (`os-public-hostname`) that resolve to the provider VIPs via external/corporate DNS; internal and admin endpoints stay IP-based on the metal VIPs. Tenant instances use upstream resolvers (1.1.1.1 / 1.0.0.1). The D-011 acceptance bar is amended to drop the cloud-DNS criterion, and the planned `v1-do-doc-10-dns` runbook is dropped.

**Consequence (documented, not a blocker):** metal-only charm units that make catalog-based client calls pull the PUBLIC (FQDN) endpoint and cannot resolve or route it (the internal-endpoint certs carry no FQDN SAN). This is the root of the gss/retrofit amphora-pipeline constraint recorded in D-021. The proper fix (cloud-internal DNS + FQDN-valid certs, or charms consuming internal endpoints) is a Roosevelt item.

**Status:** Decided (v1). Reconstructed into this doc from the deploy record (no standalone D-019 file).

**Related:** D-008 (DNS architecture), D-021 (amphora-pipeline consequence), D-011 (acceptance bar amended).

---

## D-020: Dual provider + metal API VIPs on clustered charms

**Decision:** Every clustered OpenStack API application (keystone, glance, nova-cloud-controller, neutron-api, cinder, placement, barbican, octavia, openstack-dashboard, magnum, vault) is configured with BOTH a provider VIP and a metal VIP, as a space-separated pair: `vip: "10.12.4.X 10.12.8.X"` (Option B).

**Rationale:** with a provider-only VIP, `charms_openstack/ip.py:resolve_address(INTERNAL)` returns `None` and raises `ValueError`, breaking `identity-service-relation-joined` (and the analogous internal-endpoint registration on every clustered API charm). Supplying a metal-network VIP alongside the provider VIP gives `resolve_address` an internal address to return, and keeps east-west service traffic on the metal network rather than the provider network.

**Status:** Decided (v1). Reconstructed into this doc from the deploy record (no standalone D-020 file).

**Related:** D-003 (network architecture), D-002 (channels).

---

## D-021: Octavia amphora image pipeline on the no-DNS dual-endpoint deploy

**Decision:** build the amphora image with the charm-native `octavia-diskimage-retrofit` set `use-internal-endpoints: true`, seeded by a manually uploaded stock Ubuntu base image carrying the five Glance properties the retrofit reads (architecture, os_distro, os_version, version_name, product_name). Park `glance-simplestreams-sync` for the amphora pipeline. The amphora image is `image-format: raw`, tagged `octavia-amphora` to match octavia's `amp-image-tag`.

**Root cause:** on the dual-endpoint, no-DNS topology (D-019), metal-net catalog-callers (gss + its retrofit subordinate) cannot reach Glance: the public Glance FQDN does not resolve/route from the metal net, and the internal-endpoint cert carries no FQDN SAN (so an `/etc/hosts` FQDN->metal-VIP mapping fails TLS). gss `use-internal-endpoints` steers only its Keystone auth to internal; its glance/swift clients still use the public FQDN and there is no further charm-native lever -- a charm gap on the no-DNS topology. The retrofit's `use-internal-endpoints` lever DOES cover its build path, so it is the charm-native amphora builder here.

**Status:** Decided + validated end-to-end (v1): the retrofit, over internal endpoints, reads the seeded base and writes the amphora; gss parked; octavia + subordinates active/idle.

**Roosevelt:** cloud-internal DNS + FQDN-valid certs removes the manual seed and fixes gss end to end.

**Related:** D-007 (Octavia inclusion), D-019 (no-DNS root cause).

---

## D-028: Defer the CAPI v1beta2-contract cutover (deploy the single-contract v1beta1 stack)

**Decision:** defer adopting the CAPI v1beta2-CONTRACT generation until upstream ships it correctly for this path; deploy the clean single-contract v1beta1 stack now.

**Context:** while grounding the (then-current) Canonical CK8s workload chart, the chart referenced control-plane/bootstrap kinds at apiVersion v1beta1 while the pinned provider served them only at v1beta2 (DOCFIX-022). The broader question -- is the v1beta2-contract generation available and correct for long-term support on this path -- resolved to "not yet."

**Status:** Decided (v1). The CK8s-chart-specific particulars were subsequently retired when D-031 replaced the direct-CAPI CK8s path with Magnum + the azimuth kubeadm charts; the single-contract principle carries forward, and D-042 later made the driver-side contract axis concrete.

**Builds on:** D-022 / D-023 (do-07-era CAPI/CRD work). **Related:** D-031, D-042.

---

## D-029: Defer Keystone SSO (k8s-keystone-auth) to Roosevelt

**Decision:** Keystone SSO for the workload clusters (the chart's `k8s-keystone-auth` addon) is deferred to the next deployment and folded into the Roosevelt cloud-internal-DNS + trusted-cert foundation. v1 workload clusters run the Kubernetes Dashboard with standard token auth; the `k8sKeystoneAuth` addon stays OFF; SSO is not validated on v1.

**Rationale:** enabling it on v1 would produce a non-functional SSO path (TLS failure to the private-CA Keystone endpoint) plus apiserver webhook error noise -- a checked box that does not work -- and forcing it would require forking the addon or fighting CAAPH, neither of which carries forward to Roosevelt.

**Finding (verified 2026-06-05):** k8s-keystone-auth 1.5.1 exposes no keystone-CA option, so it cannot trust a private-CA Keystone endpoint.

**Status:** Decided (v1). **Related:** D-028 (same "land it on the proper foundation later" principle).

---

## D-030: Management-cluster placement -- in-cloud (superseded twice; see D-033, D-035)

**Decision (as taken 2026-06-06):** run the CAPI management plane IN-CLOUD for the v1 rehearsal (CAPI core + CAPO + cluster-api-addon-provider as VMs on the OpenStack cloud, following an Azimuth seed + HA pattern with a `clusterctl move` pivot to a self-hosted in-cloud management cluster). Out-of-cloud was recorded as a deferred alternative for Roosevelt.

**Status:** SUPERSEDED. First by D-033 (out-of-cloud Canonical `k8s`-charm on MAAS); then -- after D-033's dual-homed node hit an unfixable pod-egress fault -- placement returned in-cloud in a simpler single-homed form under D-035. Retained here for lineage.

**Related:** D-031, D-033, D-035.

---

## D-031: Cluster-creation surface + engine -- Magnum + magnum-capi-helm + azimuth kubeadm charts

**Decision:** the tenant Kubernetes service is built from three layers:
- Surface: OpenStack Magnum (`openstack coe cluster ...`), so tenants and operators manage clusters through the OpenStack API.
- Driver: the in-tree Cluster API Helm driver `magnum-capi-helm` (opendev.org/openstack/magnum-capi-helm), pip-installed into the Magnum conductor and pointed at a CAPI management cluster via `[capi_helm] kubeconfig_file`.
- Engine: the azimuth-cloud `capi-helm-charts` `openstack-cluster` chart (kubeadm-based: KubeadmControlPlane / KubeadmConfigTemplate + CAPO OpenStackCluster / OpenStackMachineTemplate + MachineDeployment), with addons (Cilium CNI, OpenStack CCM, Cinder CSI, and so on) installed by the cluster-api-addon-provider.
- Management-cluster placement: in-cloud for v1 (D-030, later refined by D-035).

**Status:** Decided. Supersedes the do-07 direct-CAPI Canonical CK8s chart path; the CK8s-chart-specific findings (DOCFIX-022 ref patch, etc.) are retired for this path.

**Related:** D-030 / D-035 (placement), D-034 (version constellation), D-036 / D-042 (driver/chart/core coherence).

---

## D-033: Management cluster -- out-of-cloud Canonical k8s-charm on MAAS (superseded by D-035)

**Decision (as taken 2026-06-07):** management cluster = a Canonical Kubernetes cluster deployed with the `k8s` / `k8s-worker` machine charms on MAAS, OUTSIDE OpenStack, made HA by the charms; CAPI layer via `clusterctl init --infrastructure openstack` + cluster-api-addon-provider, version-pinned to the capi-helm-charts release (NOT the D-022 do-07 pins); the management cluster does not run the OpenStack CCM for itself (CAPO reaches OpenStack through a `clouds.yaml` pointed at the public API endpoints); lifecycle via Juju.

**Status:** SUPERSEDED by D-035. The chosen node (capi-mgmt MAAS VM) is necessarily dual-homed (MAAS PXE on metal, API VIPs on provider), and pod egress from that multi-NIC node to the API VIPs failed (the Cilium reverse-NAT reply was mis-forwarded out the wrong NIC instead of redirected into the pod). Retained here for lineage.

**Supersedes:** D-030 (placement) + D-032 (azimuth-config tooling). **Builds on:** D-031.

---

## D-034: CAPI version constellation pinned to capi-helm-charts dependencies.json

**Decision:** pin the management-cluster CAPI constellation to the `dependencies.json` published with a chosen `capi-helm-charts` RELEASE TAG, read at deploy time on the jumphost with `jq` (dynamic lookup, no hand-picked versions). Retire D-022 "Option A" (driver 1.3.0 / CAPO v0.10.x / v1alpha6) as obsolete.

**Rationale:** the magnum-capi-helm driver does not hand-pick component versions; its own CI installs the management CAPI stack by reading the per-release `dependencies.json` and running a fixed install sequence -- that file is the single matched-and-tested set. Hand-picking fights the upstream model, and v1alpha6 has been removed from current cluster-api-provider-openstack. (At tag 0.25.1 the set is CAPI v1.13.2, CAPO v0.14.4, cert-manager v1.20.2, ORC v2.5.0, addon-provider 0.12.0, janitor 0.11.0, helm v3.17.3; appendix-B carries the as-built snapshot.)

**Status:** Adopted 2026-06-08. **Supersedes:** D-022. **Amended by:** D-042 (adds the driver<->core contract-coherence rule). **Related:** D-031, D-028 (CRD-contract note, now subsumed).

---

## D-035: Management-cluster placement -- in-cloud single-homed tenant VM

**Decision:** run the CAPI management cluster as a single-homed in-cloud tenant VM (`capi-mgmt-v2`): one NIC on the management tenant subnet (10.20.0.0/24), reached via a floating IP (per-rebuild -- this rebuild 10.12.5.103; the original 10.12.7.40 is dead -- DOCFIX-038); k8s-snap (channel `1.32-classic/stable`), Cilium CNI; not CAPI-self-managed (no `clusterctl move`).

**Rationale:** D-033's out-of-cloud node was necessarily dual-homed and its pod egress to the OpenStack API VIPs failed -- the Cilium reverse-NAT reply was emitted back out the second NIC instead of being redirected into the pod via `cilium_host` (a multi-NIC reverse-path fault; the `k8s` charm exposes too few Cilium annotations to repair it). A single-homed VM removes the second NIC and the fault entirely. The single-NIC pod-egress premise was then proven by the Phase 4 hard gate (an agnhost pod TCP probe to the Keystone VIP 10.12.4.50:5000 returning exitCode 0).

**Status:** Adopted 2026-06-08; pod-egress premise validated. **Supersedes:** D-033 (revisits D-030 in simpler form); also retires the Layer B topology of **D-007** and the bootstrap mechanism of **D-017** (k3s-on-MAAS + `clusterctl move` -> in-cloud single-homed VM, no pivot). **Unaffected:** D-031, D-034.

**Trade-off:** a single-node management cluster is a SPOF with no self-heal -- see D-041 (manual-start policy) and D-040 (the OOM that surfaced it).

---

## D-036: magnum-capi-helm driver / chart / CAPO coherence (resolved)

**Decision / correction:** a mid-session "rebuild Phase 5 on chart 0.10.1" framing -- premised on the GA driver (1.3.0) emitting the v1alpha6 OpenStackCluster CRD and clashing with the modern v1beta1 stack -- is WRONG and is retired. Chart 0.10.1 is the retired v1alpha6 path that D-034 superseded; rebuilding on it would have reversed D-034.

**Verification:** the 1.3.0 driver is api_version-AGNOSTIC (driver.py has zero v1alpha6/v1beta1/apiVersion references; it helm-installs the chart and watches the CAPI `Cluster`, never writing OpenStackCluster directly). The OpenStackCluster apiVersion is set by the CHART: chart 0.25.1 emits `infrastructure.cluster.x-k8s.io/v1beta1`, matching the installed CAPO v0.14.4. The driver's built-in default chart is 0.10.1 (the v1alpha6-era chart); overriding `default_helm_chart_version` to 0.25.1 yields v1beta1. The "1.3.0 emits v1alpha6" claim was true only of the driver's DEFAULT chart, not of the driver pinned to chart 0.25.1.

**Status:** Resolved 2026-06-08. Implements D-031 Phase 3 under the D-034 constellation. NOTE: a SEPARATE axis -- the driver-vs-core CONTRACT, not the chart's CRD string -- is what later required the 1.4.0 driver pin; see D-042. **Related:** D-031, D-034.

---

## D-037: [capi_helm] config persistence on the charm-managed conductor

**Decision:** keep the `[capi_helm]` section in an oslo.config drop-in directory and point the conductor at it: `/etc/magnum/magnum.conf.d/00-capi-helm.conf` (0644, no secrets; it references the 0600 kubeconfig by path), with magnum-conductor launched with `--config-dir /etc/magnum/magnum.conf.d` so oslo.config merges the drop-in over the charm-rendered `magnum.conf`. The charm manages neither the .conf.d directory nor the launch extension, so this survives charm hooks and reproduces on Roosevelt.

**Problem:** the magnum charm (2024.1/stable rev 70) re-renders `magnum.conf` wholesale on hooks and exposes no conf-override option, so a `[capi_helm]` section written into `magnum.conf` would be clobbered.

**Mechanism (load-bearing correction):** the conductor's ExecStart is NOT a direct binary -- it is `/etc/init.d/magnum-conductor systemd-start` (an LSB init script wrapped by systemd), so a systemd ExecStart drop-in appending `--config-dir` is inert (the flag reaches the init script as an ignored positional). The adopted method instead creates `/etc/default/magnum-conductor` (0644; the charm does not manage it) containing `DAEMON_ARGS="$DAEMON_ARGS --config-dir /etc/magnum/magnum.conf.d"`; the init script sources `/etc/default/$NAME` after setting the base `DAEMON_ARGS`, then runs `exec $DAEMON $DAEMON_ARGS`. Verify behaviorally with `/etc/init.d/magnum-conductor show-args` and `ps -ww -C magnum-conductor -o args` (not string-presence).

**Status:** Adopted 2026-06-08 (mechanism revised mid-implementation). **Residual:** breaks silently if a future charm hook writes `/etc/default/magnum-conductor` -- detect via the same show-args/ps check. **Related:** D-031 Phase 3, D-036.

---

## D-039: Magnum mints per-cluster app-creds carrying the trustor's roles (grant load-balancer_member)

**Status:** ACCEPTED 2026-06-09 (applied in phase-06; asserted in phase-08 prereqs). Cited by phase-06 (DOCFIX-036 grant), phase-08, and appendix-A; recorded here to close the dangling reference.

**Context:** the magnum-capi-helm service path uses NO static, pre-provisioned application credential. At cluster-create magnum mints a per-cluster Keystone application credential from a trust, and that app-cred carries the TRUSTOR's roles FROZEN at mint time, delegated unfiltered. The trustor is the identity that creates the cluster in the capi-mgmt project (admin@admin_domain in v1).

**Decision:** the trustor must hold `load-balancer_member` (plus `member` and `reader`) on the capi-mgmt project BEFORE any cluster is created, so every minted app-cred carries Octavia authority. A trustor holding only `member` mints a frozen app-cred that 403s when CAPO queries Octavia to confirm LB state -- the workload cluster then wedges at API-LB provisioning, and a stuck-delete 403s the same way (appendix-A). The grant is idempotent (member + load-balancer_member + reader) and applied in phase-06.

**Roosevelt implication:** whichever identity creates Magnum/CAPI clusters must carry `load-balancer_member` + `reader` on the cluster project; `member`-only is a latent 403. There is no static `capo` user/app-cred to provision -- that pattern was retired with the per-cluster mint.

**Related:** D-031 / D-036 (driver/engine surface), D-046 (the trustee domain the trust resolves against), appendix-A (D-039 + stuck-delete recovery).

---

## D-040: Raise nova-compute reserved-host-memory on the hyperconverged hosts

**Decision:** set `nova-compute reserved-host-memory` to 8192 MB (from the default 512) so Nova placement accounts for the non-Nova memory co-located on each hyperconverged host. Charm config -> survives redeploy.

**Trigger / root cause:** during the first end-to-end Magnum workload-cluster create, openstack1 hit the kernel OOM-killer (no reboot; single boot since 2026-06-03) and killed a tenant qemu worker VM. The host co-locates nova-compute AND roughly 6 GiB of services invisible to Nova placement (mysqld [innodb-cluster member] ~2.9G, ceph-osd + ceph-mon ~1.2G, neutron workers ~0.7G, nova/apache/cinder/ovs ~1.4G) while Nova reserved only the default 512 MB; under the resulting memory pressure the host swap-thrashed (an ovsdb inactivity-probe storm made the workload API and Juju agent look "down" when the host was in fact thrashing, not down).

**Status:** Adopted + APPLIED 2026-06-09. **Related:** D-035 (the mgmt-VM SPOF the OOM hit), D-041.

---

## D-041: Non-HA deployments default to manual start

**Decision:** non-HA deployments default to MANUAL START -- no automatic VM power-on / auto-recovery is configured by default. Any non-HA deployment must be documented as non-HA, with the rationale that manual-down surfaces incidents (auto-restart masks capacity/health defects). Auto-recovery is an explicit, out-of-band exception, never the silent default.

**Trigger:** after the openstack1 OOM (D-040), CAPI's MachineHealthCheck self-healed the workload worker VMs automatically, but the single-node management VM (capi-mgmt-v2, D-035) was OOM-killed and stayed SHUTOFF -- it does not self-heal or auto-restart, which silently broke magnum reconcile/health and left workload nodes with the CAPI uninitialized taint until it was started by hand. The cost (downtime) was real, but the manual-down is also what forced the investigation that found the OOM root cause headed for Roosevelt.

**Status:** Adopted 2026-06-09 (policy/governance). **Related:** D-035 (the SPOF), D-040 (the OOM).

---

## D-042: magnum-capi-helm driver must be contract-coherent with the CAPI core

**Decision (amends D-034):** the magnum-capi-helm driver pin (Layer B) MUST be contract-coherent with the CAPI core that `dependencies.json` installs (Layer A). When the Layer-A lockfile is a v1beta2-contract core (CAPI v1.13), the driver pin must be a build that understands v1beta2 references; verify this intersection at deploy.

**Symptom / root cause:** capi-test-1 reached CREATE_COMPLETE with every real component healthy (3 Ready nodes, Calico, CCM/CSI/CoreDNS, API LB ACTIVE), yet magnum reported `health_status = UNHEALTHY` deterministically -- only the `infrastructure` sub-check failed ("Infrastructure resource not found"). The 1.3.0 driver reads `apiVersion` off the Cluster's `spec.infrastructureRef`, but under the v1beta2 contract that ref is version-less, so the health GET resolves nothing. The create path is unaffected (the chart templates the resource versions) -- a cosmetic health false-negative. The governing axis is the CAPI CONTRACT a provider implements toward core, not the CRD apiVersion string (per D-028); rolling back to a v1beta1 core would mean pinning an EOL CAPI for a Roosevelt rehearsal -- the wrong direction.

**Fix:** pin a driver build carrying the per-kind `[capi_helm] api_resources` override and set it so the health lookups use the served versions. As of 2026-06-09, D-042 recorded this capability as UNRELEASED (development series only; released line then 1.1.0/1.2.0/1.3.0), with the interim = a current-series commit for the testcloud and a released-tag pin deferred to Roosevelt.

**Subsequent update (driver-fix work):** the released `magnum-capi-helm==1.4.0` was then confirmed to ship the `api_resources` feature, so the released-tag pin is now available -- v1 pins 1.4.0 with an explicit `api_resources` and targets `health_status = HEALTHY` (installed in phase-07; as-built in appendix-B). This replaces D-042's interim dev-commit path.

**Operational caveat (while any health false-negative persists):** do NOT wire magnum auto-healing to `health_status` -- a persistent false UNHEALTHY could misfire; CAPI MachineHealthCheck handles node healing independently.

**Status:** Adopted 2026-06-09; fix landed via the 1.4.0 pin. **Amends:** D-034. **Related:** D-028 (the contract axis made concrete), D-031, D-035.

---


From prior bundle review work — these are anti-patterns:

- `magnum-shared-db` missing colon — causes a relation endpoint syntax error, deploy-blocking. Bundle must use `- - magnum:shared-db` (with the colon).
- Empty `osd-devices` YAML anchor referenced by multiple ceph-osd applications.
- `ovn-chassis` binding `overlay-suffix` — invalid binding name. Correct value is `data`.
- GUI annotation collision between NUMA-split ceph-osd apps (not applicable to testcloud since we don't NUMA-split, but flagged for Roosevelt).
- Hardcoded NIC name in `bridge-interface-mappings`. Use MAC where possible.
- `openstack -f value` column ordering — column order is not guaranteed; use `-c <column> -f value` for single-column output.
- Snap confinement: `openstackclients` snap has home-only interface; commands cannot read paths under `/tmp`. File paths must resolve under `$HOME`.
- Non-ASCII characters in `local_settings.d` overrides cause silent daemon failures in Horizon.

---


## Change log

| Date       | Change                                                                                                                                                                                                                            | Reference                                            |
| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
| 2026-05-22 | Initial document captured                                                                                                                                                                                                         | Caracal rebuild planning session                     |
| 2026-05-22 | D-015 v1/v2 fork added; D-004 and D-004a marked v2-scope; D-016 IPv4 tenant pool hybrid model added; D-014 updated with new repo name                                                                                             | v1/v2 fork session                                   |
| 2026-05-22 | D-017 CAPI bootstrap full-rebuild lifecycle added; D-018 MAAS-release-direct teardown added. D-013 marked superseded by D-018. D-007 Layer B updated to reference D-017 and `runbooks/04a-capi-bootstrap-cluster.md`.             | Teardown planning + handoff session                  |
| 2026-05-22 | D-002 hacluster row added (channel `2.4/stable`) per Canonical Charm Delivery table, verified against Charmhub. D-007 Layer B driver pin updated: `stackhpc/magnum-capi-helm` v0.13.0 → `openstack/magnum-capi-helm` 1.1.0 (PyPI; stackhpc fork archived Dec 2024). | Caracal channel verification + driver pin correction |
| 2026-05-22 | D-007 Layer B kubeconfig target corrected: bootstrap k3s → workload cluster (post-pivot per workstream 3b mandatory `clusterctl move`). CAPI mgmt plane paragraph updated accordingly. | Workstream 3 cleanup (post-pivot semantics) |
| 2026-05-29 | D-019 (Designate deferral) and D-020 (dual provider+metal API VIPs) recorded as already-taken; folded into this doc in the 2026-06-09 consolidation. | Deploy execution / handoff |
| 2026-05-30 | D-021 Octavia amphora pipeline (charm-native retrofit over internal endpoints; gss parked) added. | Octavia enablement |
| 2026-06-05 | D-028 (defer v1beta2-contract cutover) and D-029 (defer Keystone SSO) added. | CAPI path research |
| 2026-06-06 | D-030 (mgmt-cluster placement: in-cloud) and D-031 (Magnum + magnum-capi-helm + azimuth kubeadm engine) added. | Magnum/CAPI surface decisions |
| 2026-06-07 | D-033 (mgmt cluster: out-of-cloud k8s-charm on MAAS) added; supersedes D-030 and D-032. | Mgmt-cluster shape |
| 2026-06-08 | D-034 (CAPI constellation pinned to dependencies.json; supersedes D-022), D-035 (in-cloud single-homed mgmt VM; supersedes D-033), D-036 (driver/chart/CAPO coherence resolved), D-037 ([capi_helm] via /etc/default DAEMON_ARGS) added. | In-cloud mgmt pivot |
| 2026-06-09 | D-040 (reserved-host-memory 8192), D-041 (non-HA manual-start policy), D-042 (driver<->core contract coherence; 1.4.0 pin) added. | OOM incident + driver fix |
| 2026-06-09 | D-019..D-042 consolidated into this document (15 decisions). Existing D-001..D-018 left intact (em-dash style preserved); the new entries are ASCII. | Repo sanitation / doc refresh |
| 2026-06-17 | D-044 (Horizon secure-cookie override) + D-045 (haproxy confirmed-LOADED) folded from the changes-doc; D-046 (magnum trustee-domain setup) + D-047 (keystone v2.0 render bug / v3 drop-in) merged and renumbered from the 06-17 addendum (its "D-044/D-045"); D-048 (stage-and-verify canonical image seed, supersedes web-download) + D-049 (D1: kube v1.34.8 / capi-k8s-v1-34) added; D-042 amended (FINDING-5: UNHEALTHY is a <=1.3.0 false-negative, HEALTHY on 1.4.0); D-050 (PROPOSED: keystone policyd-override) recorded. | End-of-deploy runbook sweep |
| 2026-06-18 | D-039 (Magnum per-cluster app-cred roles; grant load-balancer_member) recorded to close a dangling reference (previously cited only in phase-06/08, bundle, appendix-A). D-007 + D-017 annotated as superseded by D-035 / D-037 / D-042 (in-cloud mgmt VM; /etc/default config-dir graft; 1.4.0 driver) -- historical bodies retained. | Pre-commit audit (runbook sweep) |

<!-- patchset-20260610-decisions-addendum -->

---

## D-042 -- AMENDMENT (2026-06-10): mechanism evidence + signature taxonomy

Evidence gathered during the 2026-06-10 recovery session strengthens and
sharpens D-042:

1. Steady-state cosmetic signature, verbatim: health_status=UNHEALTHY with
   reason {'cluster': 'Ready', 'infrastructure': 'Infrastructure resource not
   found.', 'controlplane': 'Ready', 'nodegroup': 'Ready'}. "Resource NOT
   FOUND" (a lookup/contract miss) rather than "resource unhealthy"
   corroborates the driver-vs-CAPO API-contract coherence diagnosis.
2. The driver health maps to the cluster-level v1beta2 Available aggregation.
   This session also observed a REAL Available=False (InfrastructureReady
   failed: kube-api LB reconcile timeout during a mgmt-VM cold start, LB left
   provisioning=ERROR). Two distinct mechanisms therefore share the UNHEALTHY
   presentation -- the operator cannot distinguish real from cosmetic without
   reading the reason field -- which strengthens the case for the staged
   forward-fix (magnum-capi-helm-driver-fix runbook).
3. Signature taxonomy for operators: reason EMPTY = conductor cannot reach the
   mgmt API (VM parked/down; NOT D-042); reason populated all-'Ready' with
   infrastructure 'Infrastructure resource not found.' = D-042 cosmetic;
   reason citing an LB/infrastructure failure = real, check Octavia first
   (ops-capi-recovery Step 4/5).

## D-043 -- PROPOSED: tenant-VM auto-resume policy (resume-guests-state-on-host-boot)

STATUS: PROPOSED (decision pending). Recorded 2026-06-10.

QUESTION: should nova-compute `resume-guests-state-on-host-boot=true` be set,
so tenant VMs (including the in-cloud CAPI management VM, D-035) return
automatically after host reboots?

TENSION: D-041 ("a non-HA component staying down is a signal to investigate,
not a nuisance to auto-mask") was written for control-plane charm services.
Tenant VMs are a different class: on Roosevelt, customers will expect their
VMs back after host maintenance, and auto-resume is the industry norm for
that class. Cost of the manual policy observed 2026-06-10: a deliberately
parked mgmt VM was mistaken for an outage and consumed roughly two hours of
diagnosis before the stop was traced to an API action.

OPTIONS:
  (a) Enable auto-resume cloud-wide + monitoring/alerting on VM state, so
      "down" remains a signal without being an outage. RECOMMENDED for
      Roosevelt; candidate for v1 redeploy as well.
  (b) Keep manual start for v1 (preserves D-041 discipline on the testcloud),
      explicitly record auto-resume as the Roosevelt setting.
Note: the restart procedure's failure-mode table already references the config
key for SHUTOFF guests; whichever option is chosen, align that table, this
decision, and the bundle/runbook with each other.

<!-- patchset-20260617-sweep-decisions : end-of-deploy runbook sweep -->

---

## D-044: Horizon Secure-cookie override for internal-HTTP dashboard access (DOCFIX-030)

**Status:** Adopted 2026-06-17 (PER-REBUILD; phase-03 Step 3.3). Resolves the mislabeled "D-043" tag used for this item in earlier phase-03/changes-doc drafts -- D-043 is the tenant-VM auto-resume decision.

**Decision:** the jumphost reaches Horizon over a plain-HTTP reverse-proxy leg, but the dashboard sets `SESSION_COOKIE_SECURE`/`CSRF_COOKIE_SECURE=True`, so the browser drops the session/CSRF cookies over HTTP and login fails. Apply a Django settings override on the openstack-dashboard leader (`_99_internal_http_cookies.py`, setting `SESSION_COOKIE_SECURE=False` + `CSRF_COOKIE_SECURE=False`) to allow cookie flow over the internal HTTP leg. A TESTCLOUD accommodation of the no-DNS / no-FQDN-cert posture.

**Trade-off / Roosevelt:** disabling Secure cookies is acceptable only because the proxy leg is internal and the cloud has no public DNS / FQDN-valid cert. The Roosevelt root-fix is cloud DNS + FQDN-valid certs end-to-end (which also fixes gss and the nginx proxy_ssl_name handling); then this override is removed. Self-signed-client-TLS approaches are NOT part of v1.

**Related:** DOCFIX-030 (phase-03 Step 3.3), D-043 (distinct -- auto-resume).

## D-045: Charm-rendered haproxy config must be confirmed LOADED, not just rendered (DOCFIX-031)

**Status:** Adopted + APPLIED 2026-06-11; re-verified 2026-06-16 post phase-05.

**Decision:** after the vault/TLS cert cascade settles, confirm every unit's haproxy is actually checking its backends over the freshly-rendered SSL config -- by a functional probe of haproxy's OWN backend state (admin-socket `show stat`, grep `,DOWN,`), NOT by `juju status`. Reload (not restart) any unit whose running haproxy predates the `check-ssl` re-render.

**Root cause (confirmed, not refined to a check-config defect):** nova-cloud-controller haproxy was not reloaded after the cert cascade, so its health checks ran plaintext against the now-SSL backend port and marked the nova-api backends DOWN -- while juju stayed active/idle (juju is BLIND to per-backend haproxy state). An 8s wire capture showed the checks switch to TLS after reload; both backends returned UP/L7OK/200. The reload is a real fix, not a band-aid.

**Status note:** the surfaced symptom was nova-api EOF / 503 behind a green juju. phase-03 Step 3.1 now gates a zero-DOWN sweep cloud-wide.

**Related:** DOCFIX-031 (phase-03 Step 3.1; appendix-A), D-046 / D-047 (the separate magnum-keystone incident).

## D-046: Magnum trustee-domain setup is a REQUIRED, asserted post-deploy step

**Status:** ACCEPTED 2026-06-17. Recorded in design-decisions-addendum-20260617.md as "D-044"; renumbered to D-046 here per the 06-17 reconciliation (D-044/D-045 were taken by the Horizon/haproxy decisions). Matches the rootcause doc + phase-08 handoff prompt.

**Context:** all `openstack coe ...` ops returned 403 ("Keystone client authentication failed"); magnum-api.log showed keystoneauth1 401 on every request since 2026-06-16. Root cause: the keystone domain `magnum` and user `magnum_domain_admin` that magnum.conf `[trust]` references did not exist. `magnum/common/policy.py:130` resolves `trustee_domain_id` on EVERY policy-enforced request (driver-agnostic), authenticating as the trustee domain admin; with the domain/user absent that is a hard 401 -> every coe op 403.

**Cause:** the magnum charm action `domain-setup` is MANUAL, not automatic; magnum reports active / "Unit is ready" regardless of whether it has run. The 2026-06-11 teardown/redeploy rebuilt keystone with fresh domains but the runbook did not re-run `domain-setup`.

**Decision:** `domain-setup` is a REQUIRED, ASSERTED post-deploy step on every (re)deploy, after the magnum + identity-service relation is up and BEFORE magnum is declared functional / before phase-08: (1) `juju run magnum/leader domain-setup`; (2) assert `openstack domain show magnum` and `openstack user show magnum_domain_admin --domain magnum` both succeed; (3) gate `openstack coe service list` (must return the conductor row, no 403). magnum's active/ready status MUST NOT be treated as evidence the trustee domain exists.

**Roosevelt:** carry as an explicit runbook step + assertion; consider upstreaming a charm change so domain-setup runs automatically, or so the charm surfaces "trustee domain not set up" instead of reporting ready.

**Related:** D-047 (the v2.0 render bug found in the same incident, but NOT the cause), D-031 / D-037 (magnum surface).

## D-047: keystone auth_version v2.0 charm-render bug -- keep the v3 drop-in

**Status:** ACCEPTED (keep) 2026-06-17. addendum "D-045" -> D-047 per the 06-17 reconciliation.

**Context:** the magnum charm template renders `auth_version = v2.0` due to a type bug (the keystone interface delivers `api_version` JSON-decoded as int 3; the template does a strict string compare `3 == "3"` -> False -> v2.0). Full analysis in incident-magnum-keystone-v2-rootcause-20260617.md.

**Finding:** the v2.0 render was NOT the cause of the coe 403 (that was D-046). On this deployment v2.0 is cosmetic -- magnum's `domain_admin_auth` rewrites v2.0 -> v3, v3 is discovered from the unversioned `auth_url`, and incoming token validation worked throughout.

**Decision (Jesse, 2026-06-17):** KEEP the magnum.conf.d v3 drop-in. v2.0 is the provably wrong value for Caracal (which does not serve v2.0); the drop-in forces v3 via the same config-dir mechanism as the D-037 conductor graft (no charm-file drift, survives re-render). Architectural correctness over minimize-delta, even though the drop-in did not unblock coe.

**As-built:** `/etc/magnum/magnum.conf.d/50-keystone-v3-override.conf` (auth_version=v3 + www_authenticate_uri/auth_url v3 in `[keystone_authtoken]` and `[keystone_auth]`); `/etc/default/magnum-api` DAEMON_ARGS adds `--config-dir /etc/magnum/magnum.conf.d` (mirrors D-037 for the standalone magnum-api).

**Roosevelt:** carry the v3 fix as a drop-in/overlay; upstream the template type bug (and the separate identity-service departed-hook IndexError crash documented in the rootcause doc).

**Related:** D-046, D-037.

## D-048: Stage-and-verify is the canonical image-seed method (supersedes web-download)

**Status:** Adopted 2026-06-17 (operator-approved). Supersedes the 2026-06-16 "web-download canonical" ruling.

**Decision:** seed ALL glance images (octavia amphora base, the noble mgmt image, the workload kube image) by STAGE-AND-VERIFY: download to `$HOME` (snap-readable; NOT /tmp), verify the file against the published checksum (azimuth-images manifest sha512 for kube images; ubuntu cloud-images SHA256SUMS for noble), then `openstack image create --file [--import]`. Web-download is retained as a TESTED ALTERNATIVE only (appendix-A).

**Rationale:** (1) FINDING-3 -- glance's web-download plugin fetches with urllib (UA `Python-urllib/3.x`) and the azimuth CDN 403s that UA, so web-download is UNUSABLE for kube images (202-accept, then stuck `queued`); (2) web-download cannot checksum-verify the fetched file (the CDN redirect strips the digest) -- weaker provenance; (3) stage-and-verify is one provenance-verified path cloud-wide -- less Roosevelt delta. CORRECTION-1: a plain `--file` PUT stores qcow2 (boots fine); `--import` runs glance image-conversion -> raw (Ceph fast-clone alignment).

**Roosevelt:** unify on stage-and-verify; the longer-term target remains gss-from-a-controlled-mirror once cloud DNS + FQDN certs land.

**Related:** D-021 (amphora pipeline), FINDING-3 (appendix-A image-seeding), phase-05 / 06 / 08.

## D-049: Workload kube image bumped v1.32.13 (EOL) -> v1.34.8 (D1)

**Status:** Adopted 2026-06-17. Procedure target; re-validation on v1.34.8 follows the stage-and-verify seed.

**Decision:** the workload-cluster kube image moves from the EOL ubuntu-jammy-kube-v1.32.13 to ubuntu-jammy-kube-v1.34.8 (azimuth-images 0.28.0, build 260518-1604; sha512 7efde485...760bdb3), and the cluster template is renamed `capi-k8s-v1-32` -> `capi-k8s-v1-34`. v1.34.8 is mature with good runway and within CAPI v1.13.2 support. The management cluster's OWN k8s stays at v1.32.13 (k8s-snap 1.32-classic) -- this bump is the workload image only.

**Note:** the 2026-06-09 D-011 acceptance ran on v1.32.13; the v1.34.8 image is seeded via D-048 stage-and-verify, and D-011 re-validation on v1.34.8 is the pending acceptance item. The template now pins `--network-driver calico` (DOCFIX-032).

**Related:** D-031 / D-034 (CAPI surface), D-048 (seed), DOCFIX-032 (CNI pin), phase-08.

## D-042 -- AMENDMENT (2026-06-17): FINDING-5 -- rescope to "<= 1.3.0" (HEALTHY on 1.4.0)

The D-042 cosmetic `health_status = UNHEALTHY` false-negative is a property of driver builds <= 1.3.0 (the v1beta2 contract-ref mismatch: those builds read `apiVersion` off the infrastructureRef, which CAPI v1.13's v1beta2 contract no longer carries). The RELEASED 1.4.0 driver carries the `api_resources` override and reports `health_status = HEALTHY` against the CAPI v1.13.2 / CAPO v0.14.4 stack (confirmed this rebuild). FINDING-5: D-042 is therefore CLOSED for v1 -- the UNHEALTHY caveat applies only to the historical <=1.3.0 holding state, NOT to the as-built 1.4.0 pin. Auto-heal is still NOT wired to health_status (CAPI MachineHealthCheck heals independently).

## D-050: RESOLVED / CLOSED (2026-07-06) -- keystone `use-policyd-override=true` with no policy zip (FINDING-1)

**Status:** RESOLVED / CLOSED 2026-07-06 (recorded 2026-06-17 as an unruled finding; operator ruling: resolved via D-051 option (b); see the RESOLVED amendment near the end of this file). Scan note: this line deliberately avoids the unruled-status keywords so ledger-scan closes the block; the history arrow lives in the amendment.

**Question:** keystone is configured with `use-policyd-override=true` but no policy override zip is supplied. This is currently a no-op (no custom policy applied), but the flag advertises an override capability that does not exist -- a latent footgun (a future operator may assume policy is being enforced, or a stray zip could silently change authz).

**Options (unresolved):** (a) set `use-policyd-override=false` for v1 (the override is unused) and revisit when a real policy is needed; (b) keep true and supply an explicit, reviewed policy zip; (c) leave as-is and document the no-op. No decision made -- recorded as an open point to rule on (cf. D-043, also pending).

**Related:** D-029 (Keystone SSO deferral), FINDING-1.

## D-051: Tenant identity self-service via the SCS Domain Manager policy (resolves D-050)

**Status:** Adopted 2026-06-23 (design; pre-redeploy). Implementation staged (`domain-manager-policy.yaml`), behavioral acceptance pending the bindings+policy redeploy (gate G3). RESOLVES D-050 via its option (b).

**Context / why:** the commercial multi-DC end goal requires Model-2 tenant self-service: a tenant domain-admin must create/manage users, projects, and role assignments WITHIN THEIR OWN DOMAIN, without operator involvement and without escaping the domain. The plain `admin` role cannot do this: on Keystone it is architecturally NOT domain-confinable (a domain/project-scoped `admin` can escalate beyond its boundary -- upstream hard-coded limitation). This is the root cause behind the unreliable Horizon domain-admin experiments (admin-on-domain + OPENSTACK_KEYSTONE_PREFER_DOMAIN_TOKEN scope-switch breakage).

**Decision:** implement the **SCS Domain Manager persona** (SCS scs-0302). A tenant domain-admin is a user holding the **`manager` role on their domain** (the `manager` role already exists on this cloud). Domain-confined identity self-service is granted by a **Keystone API policy override** shipped via the keystone charm's `policyd-override` resource (the mechanism `use-policyd-override=true`, already set, expects). The policy is `domain-manager-policy.yaml`, adapted VERBATIM from the canonical SCS reference (SovereignCloudStack/standards, scs-0302-w1-domain-manager-implementation-notes.md), Section A (base_* defaults) + Section B (Domain Manager extensions).

Key policy properties (anti-escalation + fit):
- `is_domain_manager` = `role:manager`; every domain-manager grant is gated on a `token.domain.id == target.*.domain_id` match -> strictly domain-confined.
- `is_domain_managed_role` (the roles a manager may assign/revoke) = **member + load-balancer_member ONLY**. `admin` is NEVER in this list (SCS hard constraint; the anti-escalation guarantee). `load-balancer_member` is included because the Magnum apiserver-LB tenant flow requires it (cf. D-039). `reader` and `manager` deliberately NOT added (least privilege; designation authority stays operator-side -- a domain manager cannot propagate domain-manager rights).
- Every rule falls through to `or rule:base_* or admin_required`, so cloud-admin retains all authority and service-user auth is unchanged.

**Why this approach (and NOT enforce_scope):** Charmed OpenStack deliberately does NOT implement Secure RBAC -- the keystone 2024.1/stable charm exposes NO `enforce-scope`/`enforce-new-defaults` option (verified live: both keys "not found"), and Canonical sets them False across the charms for upgrade safety. SCS scs-0302 states scope enforcement is NOT necessary for the Domain Manager persona on 2024.1 or below. So the persona is achieved purely by the policy override + `manager` role. Blast radius is LOW and bounded: the charm guarantees policy overrides alter ONLY tenant/user permissions, NEVER service users (keystone/glance/nova stay charm-managed). This is materially safer than a cloud-wide scope flip -- which is why the policy change is acceptable to combine with the explicit-bindings change in one redeploy.

**Mechanism / rollback:** `zip overrides.zip domain-manager-policy.yaml` -> `juju attach-resource keystone policyd-override=overrides.zip` (use-policyd-override already true). Lands at `/etc/keystone/policy.d/`; keystone restarts. `juju status` shows `PO:` on success, `PO (broken):` if YAML is unparseable (atomic -- whole override discarded; this is the current empty-resource state D-050 flagged). Rollback = `juju config keystone use-policyd-override=false` (reverts to charm defaults; the Juju resource cannot be deleted but disabling is sufficient).

**CRITICAL -- validation is YAML-only, not semantic:** the charm validates only that the override is parseable YAML; a misspelled rule key PASSES and silently no-ops. Acceptance MUST therefore be BEHAVIORAL (same "reports OK while broken" trap as D-046), never "PO: shows active." Gate G3: as a domain-scoped `manager`, create user/project + assign member/load-balancer_member within own domain PASS; assign `admin` or touch a different domain DENY; cloud-admin ops all still PASS.

**base_* alignment (open verification, non-blocking):** the base_* rules in `domain-manager-policy.yaml` are the SCS template's upstream-recent defaults. They MUST be diffed against THIS deploy's live defaults (`oslopolicy-policy-generator --namespace keystone` on keystone/leader) and any divergent line replaced before the policy is treated as final. The file is shippable for behavioral testing as-is; the diff hardens base_* fidelity. [LIVE-READ PENDING]

**Roosevelt / multi-DC:** the override is a single portable file replicated to every DC's keystone with zero per-DC bespoke work -- good "minimize delta to Roosevelt" properties. **Version caution (HARD):** this is the 2024.1 transitional (policy-based) implementation. The native Domain Manager persona ships in 2024.2 (Dalmatian); on any future upgrade to 2024.2+ this policy override MUST BE REMOVED (it conflicts with the native impl), except a customized `is_domain_managed_role` may be preserved. Track as an upgrade-time removal item.

**Resolves:** D-050 (use-policyd-override=true with no zip -> now supplies the reviewed zip, option (b)). **Related:** D-046 (magnum trustee; the admin_required fallthrough preserves it -- risk drops from the enforce_scope-era HIGH to LOW), D-039 (per-cluster app-creds carrying load-balancer_member), D-029 (Keystone SSO deferral), D-044 (Horizon secure-cookie; same charm, do not disturb). **Supersedes the in-session enforce_scope "Strategy 1" framing, which was found impossible on this charm and is abandoned.**

## D-052: Network-space inversion -- metal-internal carries all OpenStack service-to-service control

**Status:** ADOPTED 2026-06-25 (operator-approved). AUTHORITATIVE; SUPERSEDES the three conflicting D-052 drafts (D-052-decision-text.md, D-052-CORRECTED-architecture.md, D-051-D-052-decisions.md) that attached incompatible designs to this number. Implementation pending (bundle regen + MAAS cutover); ceph network config verified (unset / binding-derived, see ceph sub-decision).

**Context:** the as-deployed cloud collapses nearly all traffic onto a single `metal` space -- only `public`->provider, ceph `public`->storage, osd `cluster`->replication, and the geneve overlay->`data` are separated today. A prior agent's binding artifacts were corrupted (conflicting hallucinated definitions), so placement was re-derived from scratch against authoritative references, over a fresh pre-teardown endpoint inventory (`juju show-application` for all 50 apps; 446 endpoints).

**Decision:** adopt full three-plane isolation as the per-DC template:
- `provider-public` (10.12.4.0/22): public API endpoints + floating IPs.
- `metal-admin` (10.12.8.0/22, UNTAGGED): operator / MAAS / monitoring / the OpenStack `admin` API endpoint + the `""` default binding. DC-LOCAL (never crosses DC fiber).
- `metal-internal` (10.12.12.0/22, TAGGED VLAN): ALL OpenStack service-to-service control -- internal API, shared-db (Galera), amqp (RabbitMQ/RPC), certificates/secrets (Vault), cache, cluster peers, OVN NB/SB DB (ovsdb*), identity service-to-service, and the inter-service relations. The MAY-CROSS-FIBER plane.
- `fabric-data` (10.12.16.0/22): tenant geneve overlay (nova-compute:neutron-plugin, ovn-chassis:data, ovn-chassis-octavia:data, octavia:ovsdb-cms).
- `storage` (10.12.32.0/22): Ceph public -- RBD client + mon + ceph control relations.
- `replication` (10.12.36.0/22): Ceph OSD cluster replication.

Renames: provider->provider-public, data->fabric-data. storage/replication keep their names and RE-IP (.16->.32, .20->.36). `metal` SPLITS into metal-admin + metal-internal; metal-internal is a NEW tagged VLAN trunked to the LXD container hosts. The MAAS rename / re-IP / new-VLAN is a POST-TEARDOWN, PRE-DEPLOY step (the deploy gate, R1 risk).

**Rationale:** RHOSP network isolation puts API + RPC + database on the Internal API network, PXE/provisioning on its own network, public API + FIPs on External, and Ceph on Storage / Storage-Management (never on Internal API). The Charmed OpenStack charm model implements this with separate public/internal/admin spaces and binds `shared-db` and `amqp` to the INTERNAL space. So the inversion (DB/MQ/internal-API on the service plane, not the management plane) is the DOCUMENTED pattern, not a deviation; moving only the `internal` endpoint while leaving shared-db/amqp/certificates/peers on the management plane would invert the intended posture. Three planes = three trust levels / three failure domains, and it maps cleanly onto multi-DC: service-to-service may extend across DC fiber, MAAS/admin stays DC-local.

**Method:** a deterministic classifier (`classify_bindings.py`) over the live inventory (`live_bindings.json`): current provider/data/storage/replication -> rename or re-IP; current `metal` -> split by endpoint semantics; FAILS LOUD (NEEDS-REVIEW) on any unmatched endpoint rather than silently defaulting. Result: 446 endpoints classified, 0 unclassified. Distribution: metal-internal 252, metal-admin 160, provider-public 11, storage 17, replication 2, fabric-data 4. Artifacts: `target_bindings.json` (machine map) + `target_bindings_readable.txt` (audit). The bundle is regenerated MECHANICALLY from target_bindings.json (ruamel, comment-preserving), with a diff-identical verification gate vs HEAD.

**Judgment calls (resolved):**
- HA heartbeat (corosync/pacemaker; ha/hanode/peer-availability/pacemaker-remote, 57 eps) -> metal-internal for v1 (Canonical default; rides the NICs the service units already have; ~11 DC-local rings). DEFERRED to Roosevelt: a DEDICATED, redundant (knet dual-link) heartbeat ring -- corosync's Totem protocol is latency/jitter-sensitive and production guidance keeps it off contended/bursty planes (Proxmox: storage must never share corosync's network; RHEL HA enforces interconnect latency limits).
- Ceph control relations (ceph-mon client/mon/osd/mds/radosgw/rbd-mirror/bootstrap-source, ceph-osd:mon; 8 eps) -> storage, co-located with the ceph data path (RHOSP keeps Ceph off Internal API; keeps the storage subsystem self-contained per-DC). VERIFIED 2026-06-25: `ceph-public-network` and `ceph-cluster-network` are UNSET on both ceph-mon and ceph-osd, so the charms DERIVE the ceph networks from the `public`/`cluster` extra-bindings -- the bindings ARE the source of truth (public->storage = ceph public network; cluster->replication = ceph cluster network), not just relation-NIC steering. CONSEQUENCE: the re-IP needs NO bundle CIDR changes -- it is handled entirely at the MAAS/space level (the repo bundle carries no CIDR-valued `*-network` options; subnet literals appear only in comments). CORRECTION: ceph-mon:cluster -> replication (the ceph cluster-network extra-binding, matching ceph-osd:cluster); it had been mis-grouped onto storage. ceph-osd:secrets-storage stays metal-internal (OSD->Vault dm-crypt keys, a secrets relation).
- Live migration (nova-compute:migration) -> metal-internal for v1. DEFERRED to Roosevelt: a dedicated migration plane + QEMU-native TLS for tenant-memory transit.
- nova<->OVN-chassis integration: nova-compute:neutron-plugin KEPT on fabric-data (rename of `data`, not moved); ovn-chassis:nova-compute + ovn-chassis-octavia:nova-compute -> metal-admin (subordinate glue). SUBORDINATE SUBSET RULE: a subordinate's bound spaces must be a subset of its principal's. ovn-chassis (subordinate on nova-compute) binds `data` -> fabric-data for geneve, which is only legal if nova-compute is itself bound to fabric-data -- and nova-compute's ONLY fabric-data binding is neutron-plugin. An earlier draft of this decision moved neutron-plugin to metal-internal "for isolation"; that would strip nova-compute's fabric-data NIC and make the chassis geneve binding illegal, so it was REVERTED (caught by the bundle regenerator's fail-loud + subset verification). ovn-chassis-octavia:data -> fabric-data is likewise backed by octavia:ovsdb-cms -> fabric-data. Same pattern holds for cinder-ceph: it binds ceph -> storage, and its principal cinder already carries storage via cinder:ceph -> storage, so the subset holds.
- Vault: `access` -> metal-internal (services fetch certs/secrets here); `external` -> metal-admin (operator / unseal path); unused db/etcd/lb-provider -> metal-admin.
- Subordinate glue (ovsdb-subordinate; magnum/octavia Horizon dashboard plugins) -> metal-internal (keep related control together; low-stakes).
- Unused optional endpoints (legacy neutron plugins external-dns/infoblox/midonet/vsd/neutron-lb, rgw multisite master/primary/secondary/slave, nova-vmware/quantum/ironic/vgpu/ceilometer/ephemeral, vault db/etcd/lb-provider, hsm) -> metal-admin via the `""` default, and OMITTED from the bundle rather than enumerated. Rebind if/when activated.

**Roosevelt:** (1) dedicated, redundant corosync ring (heartbeat off the service plane); (2) dedicated live-migration plane + QEMU-native TLS; (3) revisit whether cross-DC service-to-service coordination on metal-internal warrants its own inter-DC plane.

**Related:** D-051 (combined in the SAME redeploy -- RBAC is keystone-only / low blast radius); supersedes the three conflicting D-052 drafts; depends on the MAAS rename/re-IP/new-VLAN gate (R1); the regenerated bundle differs substantially from BOTH repo HEAD (old space names) and the WIP bundle.yaml (which was Option-1 scope: only `internal` on metal-internal).

## D-053 -- Rename network space `fabric-data` -> `data-tenant` (2026-06-25)

**Status:** ADOPTED. Naming-only change to the D-052 plane set; no architectural change.

The geneve / tenant-overlay plane introduced as `fabric-data` in D-052 is renamed to
`data-tenant` to better fit the cloud's space-identity naming. This is a SPACE-NAME change
only: the subnet (`10.12.16.0/22`), its VLAN/fabric, the role (tenant geneve overlay), and
the *semantics* of every binding are unchanged. The four bindings that read `fabric-data`
(`nova-compute:neutron-plugin`, `ovn-chassis:data`, `ovn-chassis-octavia:data`,
`octavia:ovsdb-cms`) now read `data-tenant`; the D-052 subordinate-subset reasoning holds
verbatim with the new name substituted.

**Order of record:** applied in MAAS first (the live space is `data-tenant`), then
propagated into `bundle.yaml` (6 occurrences: 4 bindings + 2 binding-model header-comment
lines) so Juju's space lookup matches the live MAAS space at deploy. The bundle QA harness
was updated in lockstep (`verify_bundle.py` ALLOWED set + the two `fabric-data`-in-`planes()`
assertions; `target_bindings.json` 4 values) and re-run clean (8/8).

**Note on this log:** the D-052 entry above is left intact and still reads `fabric-data` --
that is the historical record of the original decision; this entry records the later rename
(append-only discipline).

## D-057: provider-vip plane -- separate tagged, routed plane for public API VIPs (2026-06-29)

**Status:** SUPERSEDED by D-060 (Pattern A revert, 2026-06-29) -- the provider-vip plane is abandoned; public API VIPs return to provider-public. [Originally DECIDED.] Full record + D-003B amendment: `docs/archive/D-057-DECIDED-append.md`.

Root cause of the phase-06 FIP-unreachability blocker: API LXD containers bind `public`
to provider-public (untagged enp1s0); Juju bridges enp1s0 into a Linux bridge, starving
OVS `br-ex` so OVN's gateway ARP responder is dead. Fix: move the public API VIPs onto a
new tagged, routed plane `provider-vip` (VID 104 on the provider fabric), freeing untagged
enp1s0 for `br-ex`. FIPs stay on provider-public. This AMENDS D-003B (which co-located API
VIPs and FIPs on one L2): tenant->API is preserved via L3 routing, and it improves the
commercial hard-isolation posture (API VIPs out of the FIP broadcast domain). The durable
decision is the framework (a separate tagged/routed VIP plane); the specific subnet is set
by D-058 below.

## D-058: full plane renumber -- clean fabric-grouped /22 scheme (2026-06-29)

**Status:** SUPERSEDED by D-060 (Pattern A revert, 2026-06-29) -- the plane renumber is
abandoned; the cloud stays on the D-052/D-053 scheme. [Originally DECIDED (operator).] Full
map, jumphost ordering trap, NetBox-apex note, and the committed-foundation cascade:
`docs/archive/D-058-renumber.md`. Supersedes the D-057 minimal-delta placement of provider-vip at
10.12.24.0/22; resolves R4 (oob).

Cloud-wide re-IP for Roosevelt addressing fidelity (contiguous /22 blocks grouped by
fabric), executed as a teardown/redeploy (no in-place re-CIDR, no transient overlap).
Rotate: 8->12, 12->16, 16->20, 24->8, 64->60; 4/32/36 fixed. Result:

| plane | CIDR | VLAN / fabric | gateway |
|---|---|---|---|
| provider-public | 10.12.4.0/22 | untagged / provider | 10.12.4.1 |
| provider-vip | 10.12.8.0/22 | VID 104 / provider | 10.12.8.1 |
| metal-admin | 10.12.12.0/22 | untagged / metal (PXE) | 10.12.12.1 |
| metal-internal | 10.12.16.0/22 | VID 103 / metal | none |
| data-tenant | 10.12.20.0/22 | untagged / data | none |
| storage | 10.12.32.0/22 | untagged / storage | none |
| replication | 10.12.36.0/22 | untagged / replication | none |
| oob | 10.12.60.0/22 | f_oob | 10.12.60.1 |

VIP triple = provider-vip .8.5x / metal-admin .12.5x / metal-internal .16.5x (octets 50-60).
lib-net.sh + bundle.yaml carry D-058; the foundation scripts/runbooks are swept as each
phase is executed (living-draft discipline). NetBox apex (netbox/ipv4-prefixes-import.py) is
stale pre-D-052 and must be de-staled to D-052/053 before it can carry D-058.

## D-059: host NIC budget -- build on five physical NICs; four-NIC collapse parked (2026-06-29)

**Decision:** Build and validate v1 on FIVE active physical NICs per host
(openstack0-3) -- the live as-built reality. Defer any reduction to a four-NIC
budget until the Roosevelt inbound-server hardware sheets are confirmed.

**Live inventory (measured 2026-06-29; all four hosts identical):**

| NIC     | plane(s)                                        | sharing      |
| ------- | ----------------------------------------------- | ------------ |
| enp1s0  | provider-public + provider-vip (VID 104)        | VLAN-shared  |
| enp7s0  | metal-admin + metal-internal (VID 103)          | VLAN-shared  |
| enp8s0  | data-tenant                                     | dedicated    |
| enp9s0  | storage (Ceph public)                           | dedicated    |
| enp10s0 | replication (Ceph cluster)                      | dedicated    |
| enp11s0 | ex-lbaas -- idle, REMOVED during this MAAS work | -            |

Provider and metal already carry two planes each via VLAN tagging; data, storage,
and replication each hold a dedicated NIC -- that trio is the five-vs-four overage.
NOTE: enp7s0 reads "unconfigured" in a physical-only interface query because its L3
lives on the br-metal / br-internal bridges built atop it, not on the raw NIC -- it
is load-bearing, not spare. The genuinely idle NIC is enp11s0 (ex-lbaas), and
removing it leaves five active, not four.

**Rationale:** Five is the measured truth; building on it avoids guessing a NIC count
before the hardware is known. The four-NIC fold is a contained, well-understood change
that can be made later without reworking the whole model.

**Four-NIC collapse path (PARKED; execute only if the Roosevelt sheets confirm a
four-NIC budget):** fold storage + replication onto ONE physical NIC -- Ceph public
untagged + Ceph cluster on a tagged VLAN -- mirroring the provider/metal pattern.
Result: enp1s0 (provider+vip) / enp7s0 (metal+internal) / enp8s0 (data) / enp9s0
(storage+replication) = four. Touches three places: carve-host-interfaces.sh
(storage/replication NIC wiring + a new tagged VLAN), the bundle's Ceph public/cluster
network bindings, and the MAAS VLAN layout (a replication VLAN on the storage fabric).

**Trade-off (explicit):** separating Ceph public from Ceph cluster onto distinct
physical NICs is a performance best-practice (replication traffic does not contend with
client traffic). The collapse keeps them logically separate via VLAN but shares one
physical NIC's bandwidth -- acceptable for a test rig; a Roosevelt judgement call. Hence
parked pending the sheets, with NIC bonding as an alternative if the hardware offers it.

**Status:** Adopted 2026-06-29. Pending input: Roosevelt inbound-server NIC count.
**Related:** D-057 (provider-vip plane), D-058 (plane renumber), D-052 (space inversion).
## D-060: revert to Pattern A on the hyperconverged test cloud -- supersedes D-057 + D-058 (2026-06-29)

**Decision:** Revert the provider-NIC architecture to **Pattern A** (MAAS builds the OVS
bridge `br-ex` on provider NIC `enp1s0`; the host static lands on `br-ex`; the API LXD
containers' `public` endpoint AND ovn-chassis share that one OVS `br-ex`) and KEEP the
existing D-052/D-053 plane scheme. **D-057 (provider-vip plane) and D-058 (full plane
renumber) are SUPERSEDED** -- both were responses to a self-inflicted regression, not to a
genuine architectural requirement.

**Root cause (the pivot).** The phase-06 floating-IP-unreachability blocker was introduced by
this Claude lineage, not by Pattern A. The working pre-teardown cloud ran Pattern A. A prior
instance advised switching to **Pattern B** (raw `enp1s0`, charm-built `br-ex`) on the
rebuild. On a hyperconverged node -- where the API LXD containers and ovn-chassis share one
provider NIC -- Pattern B makes Juju build a competing Linux bridge `br-enp1s0` that captures
`enp1s0` before ovn-chassis can enslave it into OVS `br-ex`, starving `br-ex` of carrier and
darkening the floating-IP plane (OVN's gateway ARP responder goes dead). D-057 (move the
public API VIPs to a tagged `provider-vip` plane, VID 104, to free untagged `enp1s0` for
`br-ex`) and D-058 (cloud-wide renumber) were built to work around that breakage. Reverting
Pattern B removes the cause, so the workarounds are no longer needed.

**No re-CIDR.** The live MAAS is already on D-052/D-053 (verified 2026-06-29: juju / lxd /
capi-mgmt on metal-admin 10.12.8.x; openstack0 data-tenant 10.12.16.40; openstack2 storage
10.12.32.42). D-058's renumber was never applied to the running cloud. So the revert is a
teardown + Pattern A re-carve + redeploy on the EXISTING subnet scheme -- no subnet migration,
no DHCP change, no `provider-vip` plane.

**Node-role learning (Roosevelt is NOT affected).** The collision is specific to
hyperconverged topology. Roosevelt is dedicated-role: Gateway Nodes run `br-ex` / Neutron and
Controller Nodes run the API on SEPARATE machines, so no single NIC is shared between the API
containers and ovn-chassis -- the Linux-bridge-vs-OVS race cannot occur there. On Roosevelt,
Pattern B (charm-built `br-ex` on a dedicated gateway uplink) is the clean fit; the test cloud
rehearses the STACK, not the node-role split. The metal-internal VID-103 trunk, which on the
test cloud rides one shared metal NIC, becomes a real trunk to the gateway / controller /
compute access ports at Roosevelt.

**Files (this revert):**
- `bundle.yaml` -- reverted to the `bf7de5a` provider config: `public: provider-public` x11,
  VIP triple `10.12.4.5x / 10.12.8.5x / 10.12.12.5x`, all four `br-ex` MACs in ovn-chassis
  `bridge-interface-mappings` (the openstack0 MAC `52:54:00:3d:fd:54` that D-057 trimmed is
  restored). The bundle diff `bf7de5a..HEAD` was verified to be PURELY D-057/D-058, so the
  revert is a clean wholesale checkout of that file.
- `scripts/lib-net.sh` -- CIDR / VIP constants back to D-052/D-053; `PROVIDER_VIP_CIDR` /
  `PROVIDER_VIP_VID` dropped; `provider-vip` added to `STALE_SPACES` (must-be-absent guard).
- `scripts/carve-host-interfaces.sh` -- provider section is now Pattern A (`build_ovs_brex`:
  unlink commissioning link -> create OVS `br-ex` on enp1s0 -> host static on `br-ex`); the
  `enp1s0.104` / `br-prov-api` provider-vip stack and the VID-104 assertion are removed;
  metal / data statics back to D-052/D-053 (br-metal .8, br-internal .12, data .16).
- `scripts/provider-bundle-check.py` -- NEW fail-closed deploy gate (provider-public x11,
  `.4/.8/.12` triple octet 50-60, all 4 chassis MACs present); REPLACES the retired
  `scripts/d057-bundle-check.py`. Verified PASS against the reverted bundle.
- `scripts/phase-00-maas-standup.sh` -- target table + narrative retargeted to D-052/D-053;
  a dry-run now reports "no drift" against the live cloud (refuse-to-clobber guard retained
  for a fresh cloud / Roosevelt).
- `scripts/phase-00-teardown.sh` -- header + next-pointer updated (Pattern A
  re-carve / standup-verify, not re-CIDR). Logic unchanged (dynamic system_id resolution,
  capi-mgmt delete, management-substrate exclusion).
- `runbooks/phase-00-teardown-maas-reset.md` -- rewritten script-driven and current
  (deletes capi-mgmt, dynamic system_ids, D-052/D-053 CIDRs, Pattern A re-carve, OSD wipe,
  8_lbaas net removal). The prior file was pre-D-052 stale (five-space scheme, hardcoded
  system_ids, "leave capi-mgmt Ready").

**Retired (git rm):** `scripts/phase-00-maas-recidr.sh` (D-058 re-CIDR; the live cloud is
already D-052/D-053), `scripts/provider-vip-standup.sh` (D-057), `scripts/d057-bundle-check.py`
(replaced), `runbooks/jumphost-provider-vip-gateway.md` (D-057),
`runbooks/provider-vip-maas-standup.md` (D-057), `runbooks/phase-00-maas-reconfigure.md`
(D-058 reconfigure spine).

**Status:** Adopted 2026-06-29 (operator "Go"). Supersedes D-057, D-058.
**Related:** D-052 / D-053 (plane scheme, retained), D-003B (API + FIP L2 co-location,
restored), D-059 (NIC budget retained; its inventory's "enp1s0 = provider-public + provider-vip"
row is now just provider-public -- the physical five-NIC count is unchanged).

## D-061: Machine-preserving teardown via `remove-machine --keep-instance` (supersedes D-055)

**Status:** ADOPTED 2026-06-30. Supersedes D-055. Documented behavior; the `--keep-instance`
path is UNVALIDATED on this virsh-pod MAAS at adoption (validate via the canary, below).

**Problem (observed 3x in one session):** `juju destroy-model` DECOMPOSES the MAAS pod-composed
openstack0-3 machines -- they are deleted from MAAS and the carve is lost, forcing a full
reenroll + re-carve. This happened with `--destroy-storage --force`, and AGAIN with
`--force --no-wait` and NO storage flag at all. So D-055's diagnosis ("omit --destroy-storage
and the machines release to Ready") is WRONG/insufficient: the storage flag is not the trigger;
`destroy-model` itself releases the instances, and a MAAS pod auto-decomposes a released
pod-composed VM.

**Root cause / mechanism:** machine retention is NOT a `destroy-model` option -- it is a
`remove-machine` option. Juju 3.6 `remove-machine` reference: a machine can be removed from the
Juju model without affecting the corresponding cloud instance via `--keep-instance`.
`destroy-model` has no machine-preserving flag. The openstack0-3 VMs are MAAS pod-composed
(VM host `lxd`), so any *release* auto-decomposes them; `--keep-instance` is the only documented
way to detach-from-model without release.

**Decision:** two teardown scripts, by intent:
- `scripts/phase-00-teardown-release.sh` -- KEEP machines. Per host:
  `juju remove-machine <id> -m openstack --keep-instance --force --no-wait` FIRST (detach,
  preserve the MAAS instance + carve), HARD-VERIFY each host survived in MAAS (FAIL LOUD if any
  decomposed), THEN `juju destroy-model openstack --release-storage --force --no-wait --no-prompt`.
  Result: hosts stay Deployed + carved, reusable with no reenroll/recarve.
- `scripts/phase-00-teardown-destroy.sh` -- DECOMPOSE machines (the from-scratch path):
  `juju destroy-model openstack --destroy-storage --force --no-wait --no-prompt`; reenroll+carve after.
Both default to dry-run and require typing the model name at a /dev/tty gate before mutating.

**First-run safety (UNVALIDATED caveat):** `--keep-instance` is documented but had not been
exercised on this virsh-pod MAAS when adopted. `phase-00-teardown-release.sh --apply --canary`
removes ONLY openstack0, verifies it survived in MAAS, and STOPS -- run that before trusting the
all-four path. The script also hard-verifies survival after every remove and exits 1 if any host
decomposed (so a wrong assumption fails loud, not silently).

**Roosevelt:** bare-metal nodes are enlisted, not pod-composed, so release does not auto-decompose
-- the keep path is simpler there. If a MAAS pod is ever used at Roosevelt, configure it to not
decompose-on-release (or keep erase-on-release OFF) and carry the --keep-instance discipline.

**Related:** D-055 (superseded), D-018 (MAAS-release-direct teardown intent), DOCFIX-057
(old phase-00-teardown.sh deprecated -- its "releases to Ready" premise was wrong).

## D-062: mysql-innodb-cluster deploys at target unit count; no single-unit seed-then-scale

**Status:** ADOPTED 2026-06-30. Refines the D-009 unit-count guidance for this charm.

**Context:** to avoid the 3-way Group-Replication formation race that twice parked a unit `blocked`
with a metadata half-join, a single-mysql seed (`num_units: 1`, ramp to 3 later) was tried. It
FAILED: the unit parked `blocked` `'cluster' missing, Instance not yet configured for clustering`.

**Root cause:** the OpenStack reactive `mysql-innodb-cluster` charm bootstraps the InnoDB cluster
(`create_cluster`) from inside its `cluster` PEER-relation handler. With `num_units: 1` the peer
relation never receives a `-relation-changed` carrying a peer, so the bootstrap handler never runs.
Live flags confirmed: `certificates.available`, `local.cluster.user-created`, leader passwords set,
but NO `cluster.available`/bootstrapped flag; the log showed only update-status/db-router hooks
cycling -- no `create_cluster` attempt. This is NOT the Canonical "Charmed MySQL" charm (different
interface) which DOES support single-then-scale; the OpenStack charm's README only documents
`-n 3` deploy.

**Decision:** deploy `mysql-innodb-cluster` at its target unit count (3) from cold. Peer-driven
bootstrap is the charm's happy path: with 3 units brought up together, the cluster forms and (this
deploy) reported clean -- `status: OK`, 1 R/W + 2 R/O, `instanceErrors=[]` on all members. The
single-unit seed-then-ramp idea is RETIRED for this charm.

**Half-join distinction (carried):** the metadata half-join (`instanceErrors: ["...not managed by
InnoDB cluster. Use cluster.rescan()..."]`) is an artifact of MID-LIFE instance addition, not cold
bootstrap. If it recurs during a scale operation, the documented fix is
`cluster.rescan({addInstances:"auto"})` via mysqlsh as `clusteruser` (NOT the charm's bare
`cluster-rescan` action, which reports-but-does-not-adopt). Cold 3-unit deploy did not hit it.

**As-executed this deploy:** the running model was (mistakenly) deployed single-mysql, then
corrected in place: `juju remove-unit mysql-innodb-cluster/1` (a stuck mis-placed unit), then
`juju add-unit mysql-innodb-cluster -n 2 --to lxd:1,lxd:2` (real live machine ids 0-3 this cycle).
Cluster formed clean. The committed bundle is reverted to `num_units: 3` / `to: [lxd:8,9,10]`
(matching its own `machines:` 8-11 declaration; the live 0-3 numbering was a deploy-time artifact).

**Related:** D-009 (3 units on Roosevelt), BUNDLEFIX (bundle reverted to 3-unit).

---

## D-063: RESOLVED / CLOSED (2026-07-03) -- tighten the capi-mgmt-cluster security group ingress for Roosevelt

**Status:** PROPOSED / OPEN (recorded 2026-07-01) -> RESOLVED / CLOSED 2026-07-03 (applied + functionally verified; see the RESOLVED/CLOSED amendment below).

**Context:** the phase-06 in-cloud CAPI management VM (`capi-mgmt-v2`, D-035) sits behind the
`capi-mgmt-sg` security group. As built by phase-06 (a CAPI/cluster-api default posture), that SG
opens BOTH `tcp/6443` (apiserver) AND `tcp/22` (ssh) to `0.0.0.0/0`. Phase-07 Step 7.1 relies on
this: the conductor reaches the apiserver via the FIP with no per-conductor rule (which is why the
old hardcoded `10.12.4.76/32` rule-add was dropped in DOCFIX-063).

**Question:** on v1 (single-DC virtual rehearsal) the FIP is the sole access point and `0.0.0.0/0`
on 6443/22 is acceptable. For ROOSEVELT (commercial, multi-DC, HARD tenant isolation) an apiserver
and ssh open to any source that can route to the FIP is too broad.

**Options (unresolved):**
- (a) Tighten `tcp/6443` ingress to the magnum-conductor's measured source (and any operator
  tooling), and `tcp/22` to the ops/jumphost source only. Requires knowing the conductor's
  post-NAT source as the mgmt VM sees it (measured, not the tenant/provider literal).
- (b) Front the mgmt apiserver with a scoped ingress (e.g. a dedicated LB/SG chain) rather than a
  raw FIP with a wide SG.
- (c) Leave wide on the rehearsal, and make the tightened SG a phase-06 build step on Roosevelt.

No decision made -- recorded as an open hardening item (cf. D-043, D-050, also pending). Whichever
option, phase-07 Step 7.1 stays verify-first (it does not depend on the SG being wide; if 6443 is
NOT already permitted it measures the source and adds exactly that rule).

**Related:** D-035 (the mgmt VM), D-039 (per-cluster app-creds), phase-06 (SG creation),
phase-07 Step 7.1 (DOCFIX-063 verify-first).

### D-063 -- AMENDMENT (2026-07-03): sources MEASURED; option (a) adopted as (a-amended)

Measurement (jumphost session 2026-07-03; committing this amendment constitutes adoption):
1. Route: magnum/0 -> FIP 10.12.7.222 is ON-LINK via its provider leg (src 10.12.4.154) -- the
   FIP lives inside 10.12.4.0/22, so no router/SNAT exists on that path.
2. 170s SYN capture on the mgmt VM (dst 6443, pod-net 10.1.0.0/16 and loopback excluded),
   with a deliberate operator kubectl trigger mid-window. Observed sources:
     3x 10.12.4.154  -- magnum conductor health poll, ~60s cycle (route prediction CONFIRMED)
     2x 10.12.4.1    -- the OPERATOR kubectl trigger, arriving MASQUERADED to the provider
                        gateway by the upstream router (jumphost 10.17.11.246 is erased)
     2x 10.20.0.207  -- the VM hairpinning its own FIP (DNAT returns with the fixed IP as
                        source); SG rules DO apply to hairpinned traffic
   Falsified along the way: "the conductor holds a persistent apiserver connection" -- it is
   transient per-cycle; instantaneous `ss` snapshots cannot capture the client set. Use the
   window-capture method (tcpdump SYN filter over >=2 poll cycles) for any future SG sourcing.

ADOPTED RULE SET (a-amended), 6443 ingress on capi-mgmt-sg:
  - magnum unit provider addresses, /32 each, derived DYNAMICALLY from Juju at apply time
    (PATTERN-1; per-UNIT -- Roosevelt's 3-unit magnum contributes 3 rules; enumerate, never
    assume one). Value: blocks tenant-workload sources (tenant egress arrives as each tenant
    router's own 10.12.4.x SNAT address, which is NOT in the set).
  - 10.12.4.1/32 -- the operator path, EXPLICITLY DOCUMENTED AS COARSE: upstream masquerade
    makes this equivalent to "any source routed in via the gateway". The SG cannot
    distinguish the jumphost; do not pretend otherwise in docs or reviews.
  - 10.20.0.0/24 (capi-mgmt-subnet) -- self/hairpin traffic; omitting it breaks on-VM use of
    the external endpoint silently.
SSH (22) ingress: 10.12.4.1/32 only (same coarseness caveat).
NOT adopted: 0.0.0.0/0 stays out; option (b) (LB in front) remains the Roosevelt-scale
candidate if magnum unit churn makes /32 maintenance noisy.
Roosevelt note: the 10.12.4.1 masquerade is a VR0 substrate artifact. The rule set MUST be
re-derived from a fresh window capture in every environment -- never copied forward.
Apply is a SEPARATE gated mutation with its own verify-before/after block; this amendment
records the decision, not the change. **Status: ADOPTED (a-amended) on commit; apply pending.**

### D-063 -- RESOLVED / CLOSED (2026-07-03): applied and functionally verified

APPLIED via scripts/d063-apply.sh --apply (DOCFIX-084/085): four ingress rules created and
read-back-gated BEFORE the two wide 0.0.0.0/0 rules were removed by measured ID (no window
where the conductor source was unauthorized). Final SG state (measured): 6443 from
10.12.4.154/32 (conductor), 6443 from 10.12.4.1/32 (operator, coarse), 6443 from 10.20.0.0/24
(hairpin), 22 from 10.12.4.1/32; wide 22/6443 ingress gone (only default egress remains).

FLOW MODEL (corrected + doc-backed). Earlier in-session I twice mischaracterized the
conductor->mgmt traffic as a "continuous health poll"; both were WRONG. As-built (phase-07:
/etc/magnum/kubeconfig server = mgmt FIP 10.12.7.222:6443, client-cert auth) plus upstream
docs (StackHPC magnum-capi-helm deep-dive; openstack magnum-capi-helm install guide) establish
the true model: magnum-conductor is a Kubernetes CLIENT of the mgmt apiserver, acting ONLY on
cluster lifecycle events (create/update/delete -> helm install/upgrade + watch). Connections
are per-operation and short-lived (client-cert kubeconfig, no held session), so the path is
IDLE at rest and busy only during ops. This fully explains why idle established-state and SYN
snapshots repeatedly showed nothing -- expected, not a fault. Required routes per StackHPC:
conductor -> mgmt cluster; mgmt cluster -> OpenStack external net; mgmt cluster -> workload API
LBs. Only the first traverses capi-mgmt-sg ingress; the rule set covers it.

VERIFICATION (proof by exercise, through the tightened SG):
- POSITIVE, load-bearing: R1 -- a live TCP handshake from magnum/0 to 10.12.7.222:6443
  (phase-07 Step 7.1 primitive: exec 3<>/dev/tcp) returned TCP-OK. A handshake cannot complete
  if the SG drops the source; conductor source measured = 10.12.4.154 (= the applied /32).
- POSITIVE, functional (gold standard): C3 -- forced a real op (openstack coe cluster resize
  beta-cluster 1->2 as the tenant); it reached UPDATE_COMPLETE / HEALTHY / node_count=2. A
  magnum-capi-helm resize CANNOT complete unless the conductor helm-upgraded the manifest on
  the mgmt apiserver (10.12.7.222:6443) through the SG and CAPI built the worker. The
  successful state transition is dispositive proof the path carries real helm ops post-tighten.
- NEGATIVE (isolation value): a beta tenant POD -> 10.12.7.222:6443 TIMED OUT (blocked). Tenant
  workloads egress as their router's own 10.12.4.x SNAT, which is NOT in the allow set. This is
  the isolation the ruling exists to provide.
- OPERATOR: kubectl get --raw /readyz via the mgmt kubeconfig returned ok (10.12.4.1 coarse
  rule, as designed).
- NOT RELIED ON: SYN packet captures (V4/R5) came back empty -- the transient per-op flow is
  the wrong target for snapshot/window SYN sniffing (learned 3x this thread; established-state
  and functional-outcome are the correct instruments). A conductor-log corroboration probe (C2)
  was empty because it queried journald; charmed magnum logs to
  /var/log/magnum/magnum-conductor.log. Neither is needed: C3's functional success supersedes
  wire-level corroboration (industry practice closes on functional verification of the flow).

BETA left at node_count=2 (UPDATE_COMPLETE/HEALTHY) -- the resize doubles as previously-untested
resize acceptance coverage; a change from the count=1 acceptance baseline, retained deliberately.

Roosevelt carry-forward: the rule set is per-UNIT and per-ENVIRONMENT -- 3-unit magnum yields 3
conductor /32s; the 10.12.4.1 operator collapse is a VR0 masquerade artifact; re-derive with
d063-apply.sh (live per-unit derivation) in every environment, never copy rules forward.

**Status:** RESOLVED / CLOSED (applied + functionally verified 2026-07-03). Supersedes the
"apply pending" state above. **Related:** DOCFIX-084/085 (the applier + its stderr-capture
fix), phase-07 Step 7.1 (the TCP primitive reused for R1), D-035/D-036 (mgmt-cluster model).


---

## D-064: Reconcile D-051 to the SCS Domain Manager standard (scs-0302); fix create-op policy templating

**Status:** ADOPTED 2026-07-01 (phase-08 keystone-policy blocker). Reconciles D-051 to the
authoritative SCS reference and discharges D-051's [LIVE-READ PENDING] base_* alignment gate.
Behavioral acceptance = Magnum trustee create + tenant G3 (below).

**Context / trigger:** phase-08 workload-cluster create failed at the FIRST step --
create_trustee_and_trust -> identity:create_user HTTP 403. magnum_domain_admin holds Admin on
the magnum domain (correct per Magnum docs + D-046) and was authenticated (401 would be auth;
403 = authorization). Root cause: this cloud's charm-rendered /etc/keystone/policy.json
(old-style defaults, enforce_scope=False) defines the create-op helper
admin_and_matching_user_domain_id with legacy templating domain_id:%(user.domain_id)s, which
does NOT resolve for create_user on Caracal (keystone populates target.user.domain_id, not the
bare user.domain_id). The admin-on-domain create branch silently evaluates false and only
cloud_admin passes -- the same defect that blocked tenant domain-admin self-service in the
2026-06-22 northwind rehearsal. One bug, two symptoms (Magnum trustee + tenant self-service).

**Finding (why D-051 alone did not fix it):** D-051 deliberately deviated from scs-0302 -- it
dropped Section A (base_*) and pointed fallthroughs at the LIVE helpers by name, including the
broken admin_and_matching_user_domain_id, and dropped the `or rule:admin_required` tail.
Consequently D-051's create_user fallthrough inherited the broken helper; only its manager
branch (correct token.domain.id:%(target.user.domain_id)s templating) worked. Attaching D-051
as-staged would have fixed tenant self-service (manager) but NOT Magnum (admin-on-domain trustee).

**Decision:** reconcile to scs-0302 by correcting the create-op fallthroughs to the cloud's
CORRECT %(target.*.domain_id)s helpers (mirroring the standard's base_create_*) and complete the
base_* alignment against the now fully-read live policy. Exactly 7 rules change:
- create_user / create_project / create_group: fallthrough admin_and_matching_<x>_domain_id
  (broken %(<x>.domain_id)s) -> admin_and_matching_target_<x>_domain_id (correct
  %(target.<x>.domain_id)s). BUG-FIX; restores the documented admin-on-domain create path and
  unblocks Magnum. Confined (domain match); does NOT touch tenants (they hold manager, not
  domain-admin).
- add_user_to_group: admin_and_matching_group_domain_id -> admin_and_matching_target_group_domain_id.
  ALIGNMENT (restore live default; D-051 had drifted to the broken helper).
- list_users / list_projects / list_groups: fallthrough admin_required (D-051 conservative
  [PENDING-LIVE-READ] guess, BROADER than live) -> cloud_admin or admin_and_matching_domain_id
  (the live default). ALIGNMENT; behavior-preserving; removes an unintended widening.
list_roles keeps admin_required (its live default). The manager persona (Section B),
is_domain_manager = role:manager, and is_domain_managed_role = member + load-balancer_member
(no admin, no manager -- anti-escalation) are UNCHANGED from D-051 / scs-0302.

**Consequence:** with create_user's admin fallthrough fixed, magnum_domain_admin creates
per-cluster trustees via the standard admin-on-domain path -- NO manager grant on the magnum
service account is required (an in-session Option A now moot). Magnum keeps its documented
trustee setup; the policy fix alone unblocks it.

**Validation:** oslo.policy parses all 37 rules (the charm validates YAML only -- a malformed
rule passes YAML and silently no-ops, cf. D-046 "reports OK while broken"); YAML + ASCII clean;
malformed-connector lint clean; every fallthrough tail diffed against the live effective default.
Behavioral gate (the real acceptance): (a) Magnum -- re-run phase-08 8.1, trustee create_user
passes, cluster converges; (b) tenant G3 -- a manager-on-domain user self-services user/project
+ member/load-balancer_member within its domain (PASS), is denied admin/cross-domain (DENY),
cloud-admin unaffected.

**Known SCS limitation (carried, per standard):** the manager persona grants list_domains and
list_roles cloud-wide (needed to resolve domain/role names). A tenant manager can ENUMERATE
domain names/ids + role names (not access resources). Inherent to the scs-0302 transitional
policy (upstream RBAC-scoping of domain list is a pending fix); the native 2024.2 persona closes
it. On upgrade to 2024.2+ this override MUST be removed (scs-0302 + D-051 caution).

**Mechanism:** zip overrides.zip domain-manager-policy.yaml -> juju attach-resource keystone
policyd-override=overrides.zip (use-policyd-override already true). keystone PO (broken) -> PO:.
Rollback: juju config keystone use-policyd-override=false. Full attach + G3 procedure in
runbooks/appendix-C-identity-rbac.md.

**Roosevelt:** single portable policy file replicated per-DC keystone; the attach + G3 gate
becomes a deploy step. Remove on any 2024.2+ upgrade.

**Related:** D-051 (reconciled here), D-046 (magnum trustee domain), D-039 (per-cluster app-cred
roles), D-050 (resolved by supplying the zip), scs-0302-w1 (the authoritative standard),
appendix-C (identity/RBAC reference). **Supersedes:** D-051's [LIVE-READ PENDING] gate (discharged).


---

## D-065: Fix identity:create_trust non-resolving policy template (Magnum multi-tenant cluster blocker)

**Status:** ADOPTED 2026-07-02. Extends the D-064 defect-class fix to the keystone trust family.
Behavioral acceptance = a tenant service identity clears create_trust and the cluster converges
(pending re-attach + retest at adoption).

**Context / trigger:** multi-tenant validation (tenant acme). A tenant service identity (acme-svc,
app cred, holding member + load-balancer_member on acme-prod) creating a Magnum cluster failed at
create_trust with identity:create_trust 403 -- AFTER D-064 unblocked create_user (the trustee user
was created). The SAME 403 reproduced (a) for admin via password auth, and (b) via a DIRECT
`openstack trust create` with trustor == caller == self, no magnum in the path. That proves it is
identity-independent and not a magnum bug.

**Root cause:** the charm-rendered base policy.json defines
identity:create_trust = "user_id:%(trust.trustor_user_id)s" -- the legacy NON-target-prefixed
template that does NOT resolve on Caracal (keystone populates target.trust.trustor_user_id). The
rule evaluates user_id == "" -> always false -> every trust create 403s regardless of caller. Same
defect class as D-064 (create_user's %(user.domain_id)s vs the resolving %(target.user.domain_id)s).
Keystone's OWN shipped default is the target-prefixed form:
"user_id:%(target.trust.trustor_user_id)s".

**Masking:** create_trust was never reached before D-064 because create_user 403'd first. The
2026-06-09 CREATE_COMPLETE was on the pre-2026-06-11-teardown cloud; this redeploy's create_trust
had never been exercised until D-064 opened the path.

**Causation proof (verify-before-fix):** with use-policyd-override=false (base policy only), the
direct trust create STILL 403s -> the D-064 override is NOT the cause; the base charm policy owns
the bug. Override re-enabled immediately (PO: restored) -- the cloud was not left without the
manager persona.

**Decision:** add identity:create_trust = "user_id:%(target.trust.trustor_user_id)s" to the D-064
override (policies/domain-manager-policy.yaml). Exactly the keystone-shipped default, and
consistent with this cloud's own already-correct sibling rules (list_trusts_for_trustor/trustee).
No admin fallback (matches the shipped default; trusts are self-delegation only). No manager branch
(create_trust is universal self-delegation, not part of the domain-manager persona).

**Scope (evidence-gated; NOT a blanket family sweep):** create_trust is the ONLY demonstrably
broken trust rule. list_trusts_for_trustor / list_trusts_for_trustee already carry the
target-prefixed form (match shipped). get_trust / delete_trust / list_trusts / list_roles_for_trust
/ get_role_for_trust are live "" (empty) but code-guarded in keystone (_trustor_trustee_only;
delete checks user==trustor or is_admin) -> functional, not broken, not blockers; left unchanged.

**Second-check clearance:** keystone's create_trust also enforces _require_trustor_has_role_in_project
(trustor must hold each delegated role on the project). Magnum delegates context.roles (the caller's
token roles), which by construction are a subset of what the caller holds on the scoped project ->
passes. acme-svc holds member + load-balancer_member on acme-prod; confirmed clear on retest.

**Validation:** oslo.policy parses all 38 rules; YAML + ASCII + connector lint clean.

**PROPOSED / OPEN (separate hardening, NOT actioned here):** live identity:list_trusts = "" lets any
authenticated user enumerate ALL trusts (trustor/trustee/project relationships) cloud-wide -- an
info-disclosure the newer keystone default tightens to
"rule:admin_required or (role:reader and system_scope:all)". This is a security-posture change, not
a functional blocker (create/get/delete are code-guarded), so it is recorded as an open item to rule
on rather than bundled into this blocker fix. If adopted, add list_trusts (and optionally the read
rules) at the shipped defaults in the same override.

**Roosevelt:** carry the create_trust override with the rest of the D-064 policy. The base-charm
non-resolving-template defect is upstream-worthy -- report against the keystone charm's rendered
policy.json for Caracal (2024.1). Remove on 2024.2+ (native secure-RBAC ships the resolving defaults).

**Related:** D-064 (same defect class; same override file), D-039 (the trustor roles that satisfy the
trustor-has-role check), D-046 (magnum trustee domain), D-035 (mgmt cluster), appendix-D (trust
model). **Revises:** appendix-D section D.3 -- the role-delegation hypothesis is REFUTED (a
clean-role tenant identity still 403'd; the cause was the non-resolving policy template, not role
delegation). appendix-D to be corrected on finalize.


---

## D-066: Tenant account model (Option-3 split); cluster-create requires PASSWORD auth

**Status:** ADOPTED 2026-07-02. Standard per-tenant identity set, used from the first tenant so
v1 and Roosevelt onboard identically (no interim model to retrofit). Trust path validated live
through create_user + create_trust + cert-gen entry (see D-064/D-065); full cluster completion is
gated on D-067 (Barbican/Vault substrate).

**The per-tenant accounts (operator creates the domain + manager; manager creates the rest):**
- `<client>-domain-admin` -- `manager` on the domain (SCS Domain Manager, D-051/D-064). Tenant
  IAM self-service. Operator-provisioned; password handed over.
- `<client>-cluster` -- `member` + `load-balancer_member` on `<client>-prod`. PASSWORD auth only.
  Sole purpose: `coe cluster` lifecycle (this is the identity that mints the Keystone trust). Also
  owns whatever magnum-internal creds the driver mints under it (the per-cluster CAPO child cred,
  D-039). Manager-created.
- `<client>-svc` -- `member` + `load-balancer_member` on `<client>-prod`. UNRESTRICTED app cred,
  for tenant-authored non-trust automation (CI/pipelines). NOT used for cluster-create. Manager-created.

**Why password (not app cred) for cluster-create -- the hard constraint:** keystone on this build
blocks trust creation from application-credential tokens unconditionally. Source
(keystone/api/trusts.py `_check_application_credential`, read live 2026-07-02): if the token method
is `application_credential` it raises "Using method 'application_credential' is not allowed for
managing trusts" -- and the docstring states this applies "regardless of the 'unrestricted' flag"
(this build is STRICTER than upstream, which exempts unrestricted). Confirmed live: an unrestricted
app cred (which minted a child cred, so genuinely unrestricted) still hit this block; the same
identity via PASSWORD passed it. So the cluster-creator must be password-authenticated.

**Security rationale (this is the correct side of the control, not a workaround):** the block exists
to stop a single-project app-cred token from extending the trust delegation chain beyond its scope.
A password token is full-user scope, so a trust minted from it delegates only what the user already
holds (member + load-balancer_member on its own project) -- no escalation, confined to the tenant.
Splitting `-cluster` (trust-capable password, cluster ops only) from `-svc` (app cred, everything
else) isolates the trust-capable credential to one identity doing one job.

**REJECTED:** `CONF.security_compliance.allow_insecure_application_credential_trust_escalation`
(the escape hatch in `_check_application_credential`). Enabling a setting named "insecure" +
"trust_escalation" -- which defeats a control designed to contain a compromised tenant credential --
undercuts the hard-isolation thesis of the whole build. Not adopted.

**Roosevelt:** onboard every tenant with this three-account set. Script it (scripts/tenant-onboard.sh,
DRAFT this session). CAPO-cred ownership under `-cluster` confirmed acceptable (magnum-internal,
created/destroyed with the cluster) -- the empirical "whose identity mints the child cred" check
is pending the D-067 substrate fix (cluster-create dies at cert-gen before the driver's mint step).

**Related:** D-051/D-064 (manager persona), D-065 (create_trust template fix), D-039 (trustor roles).
**Revises:** appendix-D D.3 (the app-cred-creator assumption is wrong -- corrected in appendix-D this session).

---

## D-067: barbican-vault -> Vault (vault-kv) must use the metal-internal plane (live drift; the cert-gen blocker)

**Status:** ADOPTED 2026-07-02 (fix pending next session -- a live rebind, gated). This is the defect
that blocks multi-tenant cluster COMPLETION (cluster-create clears trust, then dies generating certs).

**Symptom chain:** tenant cluster-create (password identity) cleared create_user + create_trust,
then `CREATE_FAILED: Failed to create certificates`. Root cause traced through magnum -> Barbican
(`POST /v1/secrets` 500) -> castellan vault_key_manager `_build_auth_headers` -> Vault AppRole login
rejected. Vault's RAW error (not an HTTP code -- the authoritative read):
`source address "10.12.8.176" unauthorized through CIDR restrictions on the secret ID`.

**Root cause:** barbican reaches Vault at Vault's METAL-ADMIN address (vault_url=http://10.12.8.190:8200,
egress src 10.12.8.176 via eth2). Vault's barbican-vault AppRole binds secret_id to the metal-internal
CIDR (where service-to-service traffic belongs, D-052/D-053). Off-plane source -> Vault rejects. This
is NOT secret_id expiry: `juju run vault/leader refresh-secrets` DID rotate the secret_id (barbican.conf
re-rendered, barbican restarted) and the login STILL failed with the CIDR error. The earlier
"expired TTL" reading was WRONG -- corrected here.

**The bundle is CORRECT; the LIVE env drifted.** bundle.yaml already binds every secrets-path endpoint
to metal-internal: vault `secrets: metal-internal` (L130), barbican `secrets: metal-internal` (L667),
barbican-vault `secrets-storage: metal-internal` (L700). So a redeploy from the bundle would NOT have
this defect. The live deployment's effective binding for the barbican<->vault secrets path resolves to
metal-admin -- a drift from the bundle (mechanism: TBD by read-only `juju show-application` binding +
relation-data diagnosis next session; candidate causes: bindings added to bundle post-deploy and never
applied live, or the subordinate address-advertisement not following the bound space).

**Fix (rebind, do NOT widen the CIDR):** reconcile the live binding to the bundle so barbican egresses
from its metal-internal address (10.12.12.110), which the AppRole CIDR already trusts. Widening the
AppRole CIDR to metal-admin is REJECTED: it loosens a security control, legitimizes east-west traffic
on the wrong plane, gets reverted by the charm, and leaves the next Vault consumer exposed. Next-session
sequence: (1) read-only binding diagnosis (`juju show-application vault barbican barbican-vault`,
spaces<->subnets map); (2) gated `juju bind` / relation refresh to metal-internal; (3) re-run
refresh-secrets if needed; (4) confirm barbican AppRole login = HTTP 200 from a metal-internal source;
(5) re-run tenant cluster-create -> cert-gen clears.

**Related:** D-052/D-053 (six-plane network; metal-internal carries east-west service traffic),
D-057 (prior topology-didn't-follow-binding defect -- same family). **Corrects:** the in-session
"secret_id TTL expiry" hypothesis (refuted by the refresh-secrets test).

### D-067 -- AMENDMENT (2026-07-02, post-fix): corrected mechanism; FIXED live; CLOSED for v1

The "bundle correct / live drifted" framing above is WRONG and is corrected here (append-only
discipline; the body above stands as the historical record). Root cause, read from charm source at
the exact vendored versions:

1. Live bindings were NEVER drifted -- `juju show-application` matched the bundle exactly (all
   secrets-path endpoints metal-internal). The drift was one relation-data key: vault/0 advertised
   binding-correct ingress (10.12.12.117) but an explicit `vault_url` of http://10.12.8.190:8200.
2. Mechanism: charm-vault `send_vault_url_and_ca()` (re-fires on every non-update-status hook)
   computes vault_url from the `access` binding AND vault_url_external from the `external` binding,
   publishing BOTH when they differ. The vault-kv interface (vendored commit 6f7848c, per
   src/build.lock) IGNORES the `remote_binding` selector (deprecated, LP#1895185) and writes the
   single `vault_url` key on ALL relations -- so the external publish CLOBBERS the access URL for
   every kv consumer. Simultaneously the AppRole secret_id is CIDR-bound to the CONSUMER's
   secrets-relation ingress /32 (10.12.12.110/32) -- self-inconsistent whenever access != external.
3. Why external resolved to metal-admin: the bundle OMITS vault's `external` binding, so it falls
   to the `''` default (metal-admin). D-052's judgment call ("vault external -> metal-admin,
   operator/unseal path") rested on a mistaken premise: in this charm the `external` endpoint's only
   functional use is this kv-URL advertisement (plus an inert VIP check); operator/unseal access
   does not traverse it, and the listener binds [::]:8200 regardless. So a fresh deploy from the
   bundle REPRODUCES the failure -- this was a bundle+charm defect, not live drift.

Fix as executed (2026-07-02, gated): `juju bind -m openstack vault external=metal-internal`. The two
computed URLs became equal, the charm's own equality guard suppressed the second publish, relation
data flipped to http://10.12.12.117:8200 within one poll cycle, barbican-vault re-rendered
barbican.conf, and no refresh-secrets was needed. Validated end-to-end: AppRole login HTTP 200 from
the authentic source (kernel route src 10.12.12.110), then a full admin `openstack secret`
store/get/payload-compare/delete round-trip through the exact POST /v1/secrets path that 500'd.
Bundle hardened with BUNDLEFIX-007 (`external: metal-internal` + comment); the double-publish is
present on charm-vault master as of 2026-07-02, so the guard is permanent, including the D-068
1.16 pin.

Observation logged during validation (not actioned): barbican's `Secret href` renders as
https://None:9312/... (host_href unset/None). Cosmetic for barbicanclient/castellan consumers
(clients extract the UUID and dial their own catalog endpoint) -- stage-6 cluster cert refs will
exercise it for real; investigate only if cert-ref retrieval misbehaves.

**Status:** CLOSED for v1 (fixed + validated live 2026-07-02). **Adds:** BUNDLEFIX-007.
**Corrects:** the "bundle is CORRECT; the LIVE env drifted" mechanism above. **Related:** D-052
(premise correction noted), D-068 (carry BUNDLEFIX-007 through the 1.16 pin).

---

## D-068: PROPOSED -- Vault substrate hardening (Roosevelt)

**Status:** PROPOSED / OPEN 2026-07-02. Grouped substrate items surfaced while diagnosing D-067.
None block v1 tenant work; all are Roosevelt-durability items.

1. **Vault version.** Live is 1.8.8 (charm 1.8/stable) -- EOL. bundle.yaml pinned to 1.16/stable
   this session (D-068). Live upgrade is a MAJOR operation (multi-minor jump; unseal keys ready,
   storage-format compat, re-unseal after restart, ideally rehearsed) -- NOT a casual `juju refresh`.
   Verify the channel exists and rehearse before applying live.
2. **Vault over cleartext HTTP.** vault_url is `http://...:8200`; barbican<->Vault secret writes
   (including cluster CA private keys) cross metal-internal unencrypted. Enable Vault listener TLS
   for Roosevelt.
3. **AppRole credential lifecycle.** `refresh-secrets` exists as a manual action -> implies no
   automatic secret_id renewal. Audit secret_id TTLs across all vault-kv consumers; provision
   long/renewable TTLs or charm auto-renewal; add a proactive health probe that validates each
   consumer's Vault auth (the T4-style login check) BEFORE expiry, rather than discovering failure
   via a tenant's failed cluster-create.

**Related:** D-067 (surfaced these), D-052/D-053.

<!-- patchset-20260702-redeploy-readiness : DOCFIX-066..072 sweep -->

---

## D-002 -- AMENDMENT (2026-07-02): channel matrix reconciled to as-built

The D-002 table is historical (append-only); as-built corrections: (1) the `etcd, easyrsa`
row is DEAD -- etcd/easyrsa were dropped when vault moved to the mysql storage backend
(phase-02 R3: "HA Enabled false" is correct for vault-on-mysql); (2) `memcached` (absent
from the table) is pinned `latest/stable` -- the only track that charm publishes; the
project-wide "never latest/stable" guidance targets OpenStack-project charms, which
memcached is not (verify at deploy: `juju info memcached`); (3) the vault row (`1.8/stable`)
is superseded by the **1.16/stable** pin (D-068 / BUNDLEFIX-007). Channel-staleness review
2026-07-02: every other pin confirmed current for Caracal/jammy; rabbitmq-server `3.9` is
that charm's TERMINAL track (no newer exists), not a stale pin. The upstream charm-guide
delivery page is frozen (2023-12) -- `juju info <charm>` is the only live authority; the
pre-deploy channel assert belongs in `scripts/pre-flight-checks.sh`.

## D-009 -- AMENDMENT (2026-07-02): rabbitmq scale-up is NOT mechanical without min-cluster-size

D-009 promises Roosevelt scale-up is "change num_units 1 -> 3 and rerun". For
`rabbitmq-server` that is insufficient: without `min-cluster-size: 3` the charm accepts
client relations BEFORE the cluster forms (same failure class as the D-062 mysql formation
race; upstream charm docs call min-cluster-size best practice for multi-unit deploys).
Roosevelt bundle delta: set `min-cluster-size: 3` alongside `num_units: 3`. No testcloud
change (single unit is correct here).

## D-011 -- AMENDMENT (2026-07-02): acceptance item 6 is MANUAL unseal (matches phase-08)

D-011 item 6 read "Vault unseal + auto-unseal-after-reboot pattern verified". The v1
standard, as ruled in phase-08 D-011.6, is MANUAL unseal (3-of-5 key shares at a hidden
prompt) after any vault restart; auto-unseal is an available option adopted case-by-case
and NOT configured in v1. Item 6 is therefore read as: "Vault manual unseal after reboot
verified (sealed-on-restart is BY DESIGN, not a fault)". The acceptance bar and the
acceptance runbook now agree.

## D-043 -- RESOLVED (2026-07-02): ADOPTED, option (a) -- tenant-VM auto-resume ON

**Decision:** adopt option (a): `nova-compute resume-guests-state-on-host-boot=true`,
cloud-wide, as charm config (already present in bundle HEAD -- this resolution brings the
decision record level with the implementation, closing the bundle-ahead-of-record drift
flagged as DOCFIX-072). Rationale: tenant VMs are a different class from control-plane
services -- auto-resume is the industry norm and the Roosevelt customer expectation; the
restart procedure already builds on it (guests resume per pre-shutdown state, so a
deliberately stopped guest STAYS stopped). D-041's "down is a signal" discipline is
UNCHANGED for control-plane services, which this option does not touch; the D-035 mgmt VM
remains manual-start by policy (it is a control-plane component that happens to be a VM --
if auto-resume masks a mgmt-VM incident class in practice, revisit with an explicit
exclusion). Roosevelt item: VM-state monitoring/alerting so "down" stays visible.
**Resolves:** D-043 (was PROPOSED). **Related:** D-040, D-041, restart procedure.

## D-051 -- AMENDMENT (2026-07-02): policy delivery is bundle-native (DOCFIX-071)

The D-051 mechanism ("zip -> juju attach-resource") was never folded into the deploy
schedule: the bundle set `use-policyd-override=true` with no resource, phase-01 knowingly
deployed into `PO (broken)`, and no phase attached the zip -- the live policy existed only
as a session action (D-064). A clean redeploy would have shipped WITHOUT the tenant-
isolation RBAC. **Amendment:** the policy ships IN the bundle -- keystone
`resources: {policyd-override: ./policies/overrides.zip}`, with the zip committed beside
its source. `attach-resource` (appendix-C C.3, now $HOME-pathed -- the /tmp form was
snap-blocked) remains the LIVE-UPDATE path only. Drift between zip and yaml is a
FAIL-CLOSED deploy-gate check in `provider-bundle-check.py` (content compare). Acceptance
remains BEHAVIORAL (phase-03 Step 3.4: `PO:` is stage 1; the C.4 G3 probe is the gate) --
unchanged from D-051's own warning that the charm validates YAML only.

## D-061 -- AMENDMENT (2026-07-02): validation scope + the runbook fork (DOCFIX-066)

Two clarifications for an error-free redeploy: (1) the canary in
`phase-00-teardown-release.sh` validates machine SURVIVAL only; machine RE-ACQUISITION by
a next model (juju's MAAS provider allocates Ready machines; a kept host stays Deployed)
is the still-unvalidated half. Until BOTH halves pass, the release path is NOT a route
into phase-01 -- it serves model-only teardown/surgery. (2) The VALIDATED full-redeploy
spine is therefore the DESTROY path: `phase-00-teardown-destroy.sh` -> 8_lbaas/OSD ops ->
`reenroll-hosts.sh` -> carve -> standup -> phase-01. The phase-00 runbook (DOCFIX-066)
now encodes this fork explicitly; it previously drove the deprecated
`phase-00-teardown.sh`, whose "releases to Ready" premise D-061 disproved.


## D-069: Vault unseal-key custody -- split custody, rehearsed second-person unseal

**Status:** ADOPTED 2026-07-03 (policy; custodian ASSIGNMENT pending operator input --
tracked as SEC-003 in docs/security-ledger.md).

**Problem:** the 5 Shamir shares (threshold 3) are held by a single operator. The restart
procedure names missing keys a hard recovery BLOCKER, and vault re-init wipes the TLS
plane -- so single-person custody is a bus-factor risk no commercial cloud should carry.

**Decision (standard Vault operational guidance):**
1. Distribute shares so NO individual holds threshold (3+): e.g. 2/2/1 across three
   custodians, or 1 per person plus sealed escrow. Custodian names + storage media are
   recorded OFF-REPO (custody detail is itself sensitive); the ledger row tracks status.
2. A SECOND-PERSON unseal rehearsal is an acceptance item: someone other than the
   initializing operator must successfully unseal (restart-procedure Stage 3) before
   the deployment is declared operational. "The keys work" and "a second human can use
   them" are different facts; only the second one survives the bus.
3. Key custody is reviewed at every re-init (re-init re-cuts the shares).

**Roosevelt:** evaluate auto-unseal (transit/cloud KMS) as a separate decision; split
custody remains the floor either way (recovery keys still exist under auto-unseal).

**Related:** D-011 (amended: manual unseal is the v1 standard), DOCFIX-006 (the
lost-init cautionary tale), docs/security-ledger.md SEC-003.

## D-070: KVM baseline snapshots -- SUPERSEDES D-012 (no snapshot restore path)

**Status:** ADOPTED 2026-07-03. Supersedes D-012.

**Finding:** D-012 (two KVM baseline snapshots per cycle) was never exercised -- no
`virsh snapshot` step exists in any phase runbook or script; the only "snapshot"
mentions in the repo are dated as-built value captures. An adopted-but-unexercised
decision is documentation debt pretending to be a safety net.

**Decision:** retire the KVM-snapshot restore path. D-017/D-018 already made
rebuild-from-runbooks THE restore path, and snapshots quietly compete with that
discipline: a team that restores a snapshot skips exactly the rehearsal this testcloud
exists to force, and snapshot restore does not transfer to Roosevelt bare metal.
The pre/post-maintenance BASELINE CAPTURES (juju export + inventories) are retained and
strengthened -- that role moves to `bash scripts/cloud-assert.sh --capture`, whose
committed asbuilt/<date>/ BOM is also the Roosevelt multi-DC drift baseline.

**Recorded counter-argument:** cheap mid-rehearsal rollback (e.g. tight phase-08
iteration loops) is a real convenience. If that pain materializes, revisit as an
explicitly-labeled DEV CONVENIENCE with a rule that a snapshot restore invalidates the
run for acceptance purposes -- never as a restore path.

**Related:** D-012 (superseded), D-017/D-018 (the actual restore path), DOCFIX-075
(cloud-assert --capture).

## D-071: Routine update cadence and Juju controller patch policy

**Status:** PROPOSED 2026-07-04 (filed by the jumphost controller-update workstream;
number relinquished by the main stream per the 2026-07-03 addendum-10 contention note).
Mechanism: `runbooks/ops-update-procedure.md` (DOCFIX-086). Operator to rule on the
policy points below; the runbook is usable per-window under individual gating meanwhile.

**Problem:** the cloud accumulates available updates on two layers with no standing
policy for either: (a) the Juju controller/agents (patch releases on the 3.6 track;
live measured 3.6.24 with client already 3.6.25), and (b) charm revisions floating
within their D-002-pinned channels (~20 apps showed `can-upgrade-to` on 2026-07-04).
Appendix-B says revision float is EXPECTED and re-baselines after validated state, but
nothing says WHEN updates are applied, in what order, or what controller jumps are
acceptable on a single non-HA controller.

**Proposed policy (rule each point):**
1. **Cadence trigger.** Updates are applied in deliberate maintenance windows (not on
   discovery). Proposed trigger: a monthly review of `can-upgrade-to` + controller
   patch delta, executed via ops-update-procedure when non-empty; security-driven
   charm revisions may pull a window forward.
2. **Patch-only controller rule.** A routine window may move the controller by PATCH
   level only within the pinned 3.6 track (e.g. 3.6.24 -> 3.6.25). Minor/major jumps
   (3.6 -> 3.7+) are each their own D-NNN with rehearsal, never a routine-window action.
3. **Standing order.** Controller -> model agents -> charm refreshes, keystone first,
   nova-compute last (the ops-update-procedure grouping). Never interleave layers.
4. **In-channel-only refreshes.** A routine window never changes a charm channel
   (D-002); channel moves are per-decision (see D-068 for the live example).

**Risk (explicit, for future reviewers): no-backup single-controller patch acceptance.**
The testcloud controller is SINGLE (non-HA) and Juju 3.6 ships no supported controller
backup (`create-backup` removed in 3.0). There is NO in-band downgrade of a controller.
Accepting routine controller patches therefore rests entirely on three compensating
controls: (a) patch-level-only jumps within 3.6/stable, (b) a proven-healthy pre-state
(`cloud-assert.sh --capture` PASS committed as the pre-change BOM), and (c) the D-070
restore posture -- a failed controller means re-bootstrap + rebuild-from-runbooks, i.e.
hours of rebuild, not minutes of restore. This risk is accepted for the virtual
rehearsal phase ONLY; Roosevelt must revisit it with HA controllers and a real
controller-state backup story before production operations.

**Roosevelt:** HA controllers change the upgrade blind-window and the risk calculus;
cadence and window sizing need an SLA-aware ruling; controller backup tooling is a
design item, not an ops improvisation.

**Related:** D-002 (channel pins), D-068 (Vault channel move, the non-routine
counter-example), D-070 (restore posture), DOCFIX-086 (ops-update-procedure runbook),
appendix-B (revision re-baseline policy).

## D-068 -- AMENDMENT (2026-07-05): vault 1.16 forward-pin is NOT viable; revert bundle to 1.8/stable

Item 1 ("Vault version") pinned the bundle to `1.16/stable` as a forward-pin off EOL Vault 1.8.8,
treating the move as "a MAJOR operation to rehearse." Deep-dive research (see
`docs/D-068-vault-1.8-vs-1.16-analysis.md`) shows it is not a hard *upgrade* but a largely-
**incompatible** different charm for this reactive Charmed OpenStack cloud:

- certificates interface changed **V0 -> V1** -- the 18 `vault:certificates` client charms are V0
  requirers and cannot bind a V1-only provider (the entire cloud TLS/CA layer fails to connect);
- storage went **Raft-only**, dropping the MySQL backend this deployment uses (`vault-mysql-router`);
- **no upgrade path** (backup-on-1.8 / restore-on-new-1.15, a CA-replacement migration);
- Canonical community notes report **new-vault + Charmed Ceph broken** (this cloud uses Ceph);
- Vault 1.15+ payload is **BUSL 1.1** (was MPL) -- commercial-licensing review item.

**Amendment:** (a) a deployable bundle MUST pin vault `1.8/stable` (the reactive, integration-
compatible charm) -- revert the `1.16/stable` bundle pin; (b) `1.16` is **ruled out** as the vault-
modernization path for this reactive cloud unless/until the OpenStack service charms gain
tls-certificates V1 support; (c) "get off EOL 1.8" stays **OPEN** as a Roosevelt-durability problem
(candidates: wait for OpenStack V1 support; OpenBao/MPL fork; or explicit EOL risk-acceptance for VR0
with a production remediation deadline -- operator/security call, not resolved here).

This also corrects the D-002 amendment note (item 3): the vault row reverts to `1.8/stable`; the
`1.16/stable` supersession is withdrawn. Bundle change coordinated with Code / applied at reconciliation.

**Related:** D-068 (this amends item 1), D-002 amendment (vault row), D-067/BUNDLEFIX-007 (vault-kv),
D-069 (unseal). **Evidence:** `docs/D-068-vault-1.8-vs-1.16-analysis.md`.

## D-071 -- AMENDMENT (2026-07-05): controller backup EXISTS on juju 3.6; risk section corrected

First execution of ops-update-procedure (ops-update-20260705) disproved the risk section's
premise "no supported backup in Juju 3.6": `juju create-backup` / `download-backup` exist on
the 3.6 client and were exercised live (902MB checksummed archive of the single controller,
~35s, downloaded to the jumphost pre-upgrade). The compensating controls for routine
controller patches are therefore FOUR, not three: patch-only jumps, proven-healthy pre-state
(committed pre-change BOM), a fresh `juju create-backup` archive (now the STANDARD 2.1 step,
DOCFIX-088), and the D-070 rebuild posture as last resort. Policy points 1-4 unchanged and
still PROPOSED for operator ruling. Restore-from-archive remains UNREHEARSED -- rehearsing a
controller restore is added to the Roosevelt open questions.

**Related:** D-071 (amends the Risk section), DOCFIX-088 (runbook 2.1 correction).

## D-072: Dashboard VIP TLS repair -- cluster binding moved to the default-binding space

**Status:** ADOPTED 2026-07-05 (operator-ruled: in-place fix on VR0); AS-EXECUTED same day.

**Problem (RCA in changelog addendum 15; symptom entry in appendix-A):** dashboard VIP
https was dead SINCE DEPLOY. The openstack-dashboard charm renders haproxy's 443 backend
on the CLUSTER-binding address but creates apache SSL vhosts only for the default-binding
and public-binding addresses. With `cluster` bound to metal-internal (a third space), the
TLS backend targeted a vhost-less socket that apache answers in PLAINTEXT, while haproxy's
Layer4-only health check kept the backend green. The fail-open phase-03 3.3 check (fixed
under DOCFIX-089) let it pass the deploy gate.

**Decision:** bind `cluster` to the SAME space as the application's default binding
(metal-admin), so the charm-rendered backend lands on an address apache serves.
Live: `juju bind openstack-dashboard cluster=metal-admin` (single unit; peer relation
carries no live traffic; hacluster rides the separate `ha` endpoint). Durable:
bundle.yaml bindings updated (BUNDLEFIX-011). NEVER provider-public (tenant plane).

**As-executed verification:** haproxy backends re-rendered to 10.12.8.182 (:70/:433);
https 200 on 10.12.4.58 and 10.12.8.58 (vault-CA-verified, csrftoken cookie present);
http leg unaffected; D-044 override intact; cloud-assert PASS.
Revert: `juju bind openstack-dashboard cluster=metal-internal` + git revert of the
bundle/docs commit.

**Upstream:** bug draft at docs/archive/upstream-bug-draft-dashboard-tls.md (charm renders an
L4-masked dead TLS backend by default in multi-space deployments) -- operator to file.
**Roosevelt:** edge TLS/DNS (D-044 trajectory) supersedes this plumbing; carry the
invariant "cluster binding space == default binding space for TLS-fronted charms" into
the Roosevelt bundle review regardless.

**Related:** D-044, D-072 mechanism in appendix-A, DOCFIX-089 (fail-closed 3.3 check),
BUNDLEFIX-011, changelog addenda 15 + 18.

## D-050 -- RESOLVED (2026-07-06): closed via D-051 (operator ruling)

Operator ruled 2026-07-06: D-050 is CLOSED as resolved by D-051's option (b) -- the
`use-policyd-override=true` flag is no longer a no-op footgun because the reviewed
domain-manager policy zip ships bundle-native (D-051 amendment / DOCFIX-071), with a
FAIL-CLOSED zip-vs-yaml drift check in the deploy gate and behavioral acceptance at
G3 (appendix-C C.4 probe). Closure condition attached by the operator: the exact
settings + policy set for error-free tenant standup must remain clearly defined --
that set is: bundle keystone `use-policyd-override=true` + `resources.policyd-override`
zip (committed beside source), provider-bundle-check content compare, and the C.4
behavioral probe. Any future policy delta rides the same mechanism (see D-073 for the
first such delta). **Resolves:** D-050 (was PROPOSED/OPEN since 2026-06-17).
**Related:** D-051 + amendment, D-064, D-073, FINDING-1.

## D-073: ADOPTED -- tighten identity:list_trusts to the modern keystone default

**Status:** ADOPTED 2026-07-06 (operator-ruled); implementation + testing REQUIRED
before live apply -- nothing applied yet.

**Problem (carried from the D-064 create_trust fix, "PROPOSED/OPEN separate
hardening"):** live `identity:list_trusts = ""` lets ANY authenticated user enumerate
ALL trusts cloud-wide (trustor/trustee/project relationships) -- cross-tenant info
disclosure on a commercial multi-tenant cloud. Newer keystone defaults tighten this to
`rule:admin_required or (role:reader and system_scope:all)`.

**Decision:** adopt the modern default for `identity:list_trusts` (and evaluate the
read-rule siblings `identity:get_trust` / `identity:list_roles_for_trust` at the same
shipped defaults while in there) via the SAME policyd-override zip mechanism as
D-051/D-064 -- policy source edited, zip rebuilt, bundle resource updated, deploy-gate
drift check green, live update via the gated attach-resource path (appendix-C C.3),
behavioral verify: tenant user gets 403 on `openstack trust list`, operator
admin still succeeds, magnum cluster create/delete (trust machinery) unaffected.

**Sequencing (proposed, operator to confirm):** land this BEFORE batch-3 d011-05 so
the P3 isolation acceptance validates the final shipping policy, not a superseded one.

**Related:** D-064 (origin), D-051 (mechanism), D-050 (closed), scs-0302 posture.

## D-073 -- AMENDMENT (2026-07-06): implementation STAGED (zip rebuilt); live apply PENDING

- policies/domain-manager-policy.yaml: `identity:list_trusts` added at the modern
  keystone default VERBATIM (39 rules now). One documented exception to the overlay's
  "no new-style rules" header invariant: the (role:reader and system_scope:all) branch
  is provably inert under this cloud's old-style enforcement -- narrows only, kept
  verbatim for 2024.2+ reconciliation provenance. Trust-rule siblings left UNCHANGED
  (evidence-gated per D-064: code-guarded, and no verified shipped-default strings in
  repo evidence to copy).
- policies/overrides.zip rebuilt via python3 zipfile (no zip(1) on the jumphost);
  provider-bundle-check item 7 content-compare PASS.
- Validated here: yaml.safe_load (39 string rules), ASCII+LF, tooling compatibility
  (tenant-offboard/tenant-acceptance call `trust list` in the admin context only).
  NOT validated here: oslo.policy parse (module absent on the jumphost; D-064's parse
  ran in the main-chat sandbox -- rerun there if desired) and BEHAVIORAL acceptance.
- LIVE APPLY (gated, operator-present; in the batch-3 window do-doc): stage the zip
  under $HOME (snap /tmp limit), `juju attach-resource keystone
  policyd-override=<staged>/overrides.zip`, wait for `PO:`, then behavioral verify:
  tenant-token `openstack trust list` -> 403; admin context -> succeeds; magnum
  cluster create/delete (trust machinery) unaffected. Rollback: re-attach the prior
  zip from git history (`git show HEAD~:policies/overrides.zip`).

## D-072 -- AMENDMENT (2026-07-06): upstream bug NOT filed (operator ruling)

Operator ruled 2026-07-06: the drafted upstream bug (docs/archive/upstream-bug-draft-dashboard-tls.md)
will NOT be submitted -- the root condition is judged a site misconfiguration (cluster
binding pointed at a third space) rather than a charm defect to report. The draft stays
in-repo for history; the addendum-18 OPERATOR ACTION item is withdrawn. The Roosevelt
invariant from D-072 ("cluster binding space == default binding space for TLS-fronted
charms") is UNCHANGED and remains the carry-forward.

## D-073 -- AMENDMENT (2026-07-06): APPLIED live (d011-batch3 window)

The staged 39-rule overrides.zip was attached to keystone in the logged
d011-batch3 window (addendum 27). Settled "PO: Unit is ready". Behavioral
verify per the amendment's block: tenant app-cred context 403 on
identity:list_trusts (verbatim deny); admin trust list OK; magnum trust
machinery re-proven by d011-05 PASS. Independent proof: first live run of the
H4 drift detector (staged) reported live overlay == repo, 39 rules. Status:
ADOPTED -> APPLIED. Rollback stays as recorded (re-attach the HEAD~ zip).

## D-011 -- AMENDMENT (2026-07-06): batch-3 CLOSE -- all acceptance items stand

With the d011-batch3 window (addendum 27): item 4 octavia PASS live (RR +
member failover + amphora failover with N+1 headroom evidence, A2 conditional
pre-approval honored); item 5 magnum e2e PASS live (health + OCCM + P3
isolation via the foil1 tenant). Items 1-3 stood from batches 1-2; item 6
remains MANUAL by design (SEC-003 second-person unseal rehearsal pending --
gates D-011 FULL close); items 7-8 DROPPED per the amended bar. D-011 is
therefore CLOSED-except-item-6. The v1-close checklist (handoff section 6)
is now QUEUED per the do-doc's close-out clause. Checker corrections made
in-window: DOCFIX-095 (tenant-assert owner-context probe), DOCFIX-096
(amphora_headroom live CLI shapes).

## D-074: Tenant CIDRs are tenant-chosen and may overlap (amends D-016)

**Status: ADOPTED 2026-07-06** (operator rulings recorded same day; proposed by
the main-chat stream in docs/archive/tenant-cidr-overlap-correction-PLAN.md).

**Decision.** Tenant private (overlay) CIDRs are tenant-chosen and MAY overlap
across tenants. Non-collision is enforced ONLY against the reserved-ranges
registry, with real subnet-overlap math (containment in either direction),
plus RFC1918 membership (contract section 3 item 4's original intent).
Reserved ranges = the six PLANE_CIDRS /22s (scripts/lib-net.sh, keyed by
plane), the capi-mgmt tenant network (D-035; operator ruled: stays reserved;
resolved live by stable name capi-mgmt-subnet, never hardcoded), and the
metadata range. Default tenant CIDR policy (operator ruled): client-choice
with fallback default 10.100.0.0/24.

**Why.** The stage-4 guard over-enforced relative to the onboarding contract
(rejected tenant-vs-tenant overlap the contract never required) while
under-enforcing what matters (exact-string match missed partial overlaps
against reserved ranges). Overlap-allowed tenant CIDRs is the standard
isolated-multi-tenant model and matches the SCS Domain Manager hard-isolation
persona; per D-035 the management path never routes into tenant overlay space
(an earlier main-chat "CAPI/Magnum reachability" claim is retracted in the
PLAN doc). Gains: contract/implementation alignment, no find-a-free-/24 step,
clients can align to their own on-prem ranges, NetBox uniqueness duty shrinks
to the routed space, the one-/16 (256-tenant) ceiling goes away. Costs: an IP
no longer identifies a tenant (triage needs net+tenant context); no
tenant-private-IP routing between tenants (an anti-pattern here anyway).

**Phase-0 evidence (gates honored).** 0.1: allow_overlapping_ips=True read
live from the neutron-api leader's rendered neutron.conf (2026-07-06). 0.2:
exact-overlap proof executed inside foil1 (operator-ruled vehicle -- the
PLAN's "onboard an overlapping foil" was stale: foil1 already existed and the
old guard blocks exact-overlap onboarding): foil1-overlap-net/-subnet created
at 10.20.28.0/24 == beta-subnet EXACTLY, attached to foil1-router, and a
Magnum cluster (foil1-overlap, 1 master / 1 node, master-lb-enabled) built on
that overlapping network -- see the D-074 amendment recording the terminal
state and evidence path. Proof resources are torn down after evidence.

**Forward-only.** Existing tenants keep their CIDRs (beta 10.20.28.0/24,
foil1 10.20.29.0/24 -- measured 2026-07-06; the PLAN's "northwind
10.20.20.0/24" example was fictional). No re-CIDR; design-decisions forbids
transient-overlap re-CIDR.

**Implements:** guard rewrite in scripts/tenant-onboard.sh stage 4 +
scripts/net_overlap.py (DOCFIX-106); doc alignment (contract section 3,
onboarding runbook, clientdocs intake) rides the same delivery.
**Amends:** D-016 (its /16-pool uniqueness becomes vestigial, not reclaimed).
**Related:** D-035, D-051, D-066.

## D-016 -- AMENDMENT (2026-07-06): uniqueness carve-out superseded by D-074

D-016's single-pool model (carve non-overlapping /24s from 10.20.0.0/16)
stops being enforced forward: D-074 (ADOPTED) makes tenant CIDRs
tenant-chosen and overlap-allowed, guarded only against reserved ranges.
The /16 pool remains as history for existing allocations; nothing is
re-numbered. See D-074.

## D-074 -- AMENDMENT (2026-07-06): Phase 0.2 proof EXECUTED -- overlap+Magnum confirmed

Executed same-day in the logged d074-phase0 session (Claude Code lane).
foil1-overlap-net/-subnet created at 10.20.28.0/24 (EXACT overlap with
beta-subnet, different tenant); Magnum cluster foil1-overlap reached
CREATE_COMPLETE in ~14.5 min (matches beta's ~13.5 min profile); kubeapi
octavia LB ACTIVE/ONLINE; tenant-fetched kubeconfig returned 2 Ready nodes
THROUGH the API-LB with node addresses (10.20.28.229, .54) inside the
overlapping range -- while beta's cluster on the same CIDR served undisturbed.
Both tenants ran Kubernetes on the same /24 simultaneously: the strongest
refutation of the retracted reachability concern. Evidence:
~/openstack-baseline/d074-phase02-evidence-20260706-234352.txt (jumphost).
Proof resources torn down (cluster, template, router-if, subnet, net --
residual check clean); post-teardown cloud-assert PASS. Phase 0 gates are
fully discharged; the D-074 implementation (DOCFIX-106/107) ships.

## D-075: Dashboard VIP root -- redirect to /horizon + no autoindex (ADOPTED+EXECUTED 2026-07-07)

**Status: ADOPTED (operator ruling 2026-07-07, option c) + EXECUTED live same day.**

**Context.** During the devteam handover browser check the operator hit Apache
"Index of /" at https://10.12.4.58/ (provider-public dashboard VIP). Measured:
Horizon mounts at webroot /horizon (WSGIScriptAlias in conf-enabled/
openstack-dashboard.conf); the charm-rendered vhosts define no root handler;
Ubuntu stock apache2.conf grants `Options Indexes` on /var/www/. A directory
listing on a tenant-facing commercial surface is an information-disclosure /
posture defect, and a bare 200 at the root previously masqueraded as a healthy
dashboard (the D-072 as-executed "https 200" at the VIP root was THIS autoindex
page, not Horizon).

**Options presented:** (a) redirect / -> /horizon; (b) disable autoindex at the
root; (c) both. **Operator ruled (c).**

**Decision / mechanism.** Local override /etc/apache2/conf-available/
99-omega-root.conf on the dashboard unit (the charm exposes no config option
for either behavior; charm manages sites-enabled/ but never conf-enabled/, so
the snippet survives config-changed re-renders and in-channel charm refreshes):

    <Directory /var/www/>
        Options -Indexes
        RewriteEngine On
        RewriteRule ^$ %{REQUEST_SCHEME}://%{HTTP_HOST}/horizon [R=302,L]
    </Directory>

**Why per-directory mod_rewrite and NOT RedirectMatch:** first attempt used
`RedirectMatch 302 ^/$ /horizon`; mod_alias absolutizes the Location from the
BACKEND connection port when the client Host header carries none, emitting
http://<VIP>:70/horizon (and :433 on the TLS leg) -- ports haproxy does not
expose. Caught in this window's verify step. The rewrite echoes the client's
own Host header and request scheme, so both haproxy legs redirect correctly.
302 (not 301) so browsers do not cache permanently against a future webroot
change. `Options -Indexes` subtracts from the inherited option list and covers
/var/www/html by inheritance.

**Verification (as-executed):** on-unit root -> 302 + zero "Index of" match;
via VIP http leg: / -> 302 http://10.12.4.58/horizon -> follows to 200 at
/horizon/auth/login/. TLS-leg browser confirmation by operator.

**Durability:** PER-REBUILD -- teardown wipes the unit; reapply each rebuild
via the phase-03 step 3.3 block (DOCFIX-118), same lifecycle as the D-044
cookie override. **Roosevelt:** edge TLS/DNS (D-044 trajectory) supersedes
this plumbing; carry the invariant "tenant-facing roots never serve directory
listings" into the Roosevelt bundle review.

**Revert:** `sudo a2disconf 99-omega-root && sudo systemctl reload apache2`
on the dashboard unit; git-revert the DOCFIX-118 runbook/appendix additions.

**Related:** D-072 (VIP TLS repair; its root-200 evidence reinterpreted),
D-044 (cookie override precedent + Roosevelt edge trajectory), DOCFIX-089
(fail-closed dashboard checks), DOCFIX-118 (runbook + appendix-A delivery).

---

## D-100: Inter-DC network topology and fabric (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling). Instantiates the D-052 per-DC template across two DCs.

**Context:** VR1 is a virtual regional two-DC test environment (DC1, DC2, plus an Office1 headend), all nested libvirt VMs on one vcloud host, rehearsing the future bare-metal multi-DC site (Roosevelt). D-052 already defines the six-plane per-DC template and tags metal-admin DC-local / metal-internal "may-cross-fiber" -- but that label assumed a single stretched cloud.

**Decision:**
- Fabric: a full dark-fiber triangle -- DC1<->DC2, DC1<->Office1, DC2<->Office1 (mesh, not star) -- each modeled as a dedicated virtual L2 segment on vcloud. Each site additionally has its own OPNsense edge with a simulated ISP uplink ("external"), so every site has independent edge access.
- WAN simulation: apply `tc netem` (latency / jitter / loss / rate) on the inter-DC and site-uplink segments, tuned to same-metro dark fiber (low single-digit ms) unless the Roosevelt target says otherwise. The exact parameters are a Phase-0 knob recorded in the buildout doc.
- Plane span (independent-cloud refinement of D-052): because DC1 and DC2 are INDEPENDENT clouds (separate Keystones, no control-plane federation), no control-plane plane crosses the fiber. provider-public, metal-admin, metal-internal, data-tenant, and storage instantiate DC-LOCAL per DC. ONLY the replication plane crosses DC1<->DC2. This refines D-052's "metal-internal may-cross-fiber" (a stretched-cloud notion) to DC-local for the independent-cloud topology.
- Provider-NIC discipline: each DC's provider NIC stays RAW so ovn-chassis builds br-ex and enslaves it by MAC; MAAS must not bridge it. Public API VIPs live on provider-public (D-060 Pattern A). The D-057 provider-vip plane is NOT used (reverted by D-060).

**Rationale / tradeoffs:** mesh (not star) is required because replication is a direct DC1<->DC2 data path -- routing it through Office1 would be a bandwidth SPOF -- while management is Office1<->DC. Per-site ISP edges make each DC independently reachable and feed the per-DC airgap mirror upstream (D-107). Delta to Roosevelt: the dark-fiber-triangle-plus-per-site-edge shape IS the intended Roosevelt shape; the only virtual-only element is that the "fiber" is a vcloud virtual segment with netem standing in for real fiber and optics.

**References:** instantiates D-052 (six-plane per-DC template; independent-cloud cross-fiber refinement); carries the D-057 root-cause br-ex discipline as revised by D-060 (Pattern A); pairs with D-101 (families), D-107 (edge egress), D-108 (replication).

**Sub-items RULED 2026-07-09 (operator):** netem link profile -- same-metro dark-fiber lean CONFIRMED as the default (low single-digit ms, jumbo-capable); re-tune when a specific Roosevelt inter-DC target is known, not before. Office1<->DC fiber carries management traffic only (MAAS/Juju/operator) -- CONFIRMED per the stated lean (not one of the six items the buildout doc listed for explicit redline; adopted with the rest of D-100, flag if this needs separate reconsideration).

---

## D-101: DC-DC address plan and IPv6 family matrix (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling), AMENDED same day: (a) metal-admin gains a ULA leg (reverses this decision's own stated lean, see the family matrix below); (b) D-102 (IPv6 tenant addressing and MTU sub-policy) is MERGED into this decision per operator ruling -- D-102's entry is retained for numbering/history only, see its own Status line. Activates the developed family-follows-reachability IPv6 plan for VR1. NetBox (vcloud-local) is the IPAM authority for all literals.

**Context:** v1 / DC0 is IPv4-only on 10.12.0.0/16 (six planes in `scripts/lib-net.sh`). The developed IPv6 plan (in-project, not yet committed) classifies planes by reachability: GUA for external clients, ULA for internal-only, IPv4 where forced. DC-DC is the phase that activates it.

**Decision:**
- Per-DC distinct v4: DC1 INHERITS the DC0 six-plane v4 layout (provider-public 10.12.4.0/22 ... replication 10.12.36.0/22) unchanged, so DC1 equals the validated template. DC2 gets a DISTINCT, non-overlapping parallel supernet (same six-plane structure; exact prefixes assigned in NetBox, non-colliding with DC1, operator infrastructure, and client VPN ranges per D-074). Distinct ranges keep DR / backup endpoints unambiguous (no NAT hairpin).
- Family matrix (per DC), family follows reachability:
  - IPv6-only ULA: data-tenant (geneve underlay), storage (Ceph public), replication (Ceph cluster, including the cross-DC leg), Octavia lb-mgmt, and P2P links. Internal, no external clients.
  - Dual-stack (IPv4 + ULA v6): the metal-internal API-endpoint VIPs gain a v6 (ULA) address; each DC's Vault issues IPv6-SAN certs (D-109). **AMENDED 2026-07-09 (operator ruling):** metal-admin ALSO joins this dual-stack group (a ULA leg alongside its existing v4) -- reverses this decision's original lean of keeping metal-admin v4-only; PXE itself stays v4-first regardless (see below), the ULA leg is for the admin-API traffic on that plane, not PXE.
  - IPv4 retained: MAAS PXE / provisioning (PXE is v4-first, unaffected by the metal-admin amendment above); and datastore east-west (MySQL / RabbitMQ stay v4-bound -- the plane is dual-stack-capable but v6 there is not chased: marginal benefit, real cert / complexity cost).
  - GUA (ARIN 2602:f3e2::/32, region-0 /36): provider public API VIPs, native v6 ext_net, tenant pool delegation. Enabled here because the simulated OPNsense edge is ours -- the D-015 "upstream-router IPv6 readiness" gate on external GUA does NOT apply in VR1, so VR1 can rehearse the full internal-ULA plus external-GUA end state.
- Replication carrier correction: the DR seed names the cross-DC carrier as v4 10.12.36.0/22. Per this matrix and the operator ruling that replication is not IPv4-bound, the cross-DC replication carrier is IPv6-only ULA (a /64 from the org ULA /48). This SUPERSEDES the DR-seed carrier line. v1 / DC0 `lib-net.sh` v4 values are unchanged (DC-DC assigns families on its own layer).
- Literals: the org ULA /48, the per-DC GUA carve out of the /36, and DC2's v4 supernet are NetBox-authoritative, populated via the extended import pipeline (`netbox/*-import.py` extended for multi-DC and dual-stack). They are NOT hardcoded in this decision.
- **Tenant/MTU sub-policy (folded in from D-102, 2026-07-09 operator ruling):**
  - Tenant v6 routing: native GUA routing, no NAT66 -- the simulated OPNsense edge routes tenant GUA prefixes (prefix delegation or static routes). NAT66 is discouraged and we own the edge, so native is both correct and feasible.
  - Tenant v6 block size: /60 per tenant default (~4080 per cloud); /56 available on request.
  - Tenant address mode: SLAAC default (RA); DHCPv6-stateful available for tenants needing deterministic addressing.
  - MTU / geneve-over-v6: compute the exact byte budget in Phase 0 against the MEASURED host L2 MTU. An IPv6 outer header adds 20 bytes over IPv4 (40 vs 20); geneve-over-v6 overhead is about 40 (IPv6) + 8 (UDP) + 8 (Geneve base) = 56 bytes, before nested-virt and options. Recommendation: raise the vcloud underlay to jumbo (9000) end-to-end so tenant MTU stays 1500 and amphora/geneve fit; if the underlay is pinned at 1500, tenant MTU drops to about 1444 (v6-geneve) and MUST be set consistently across ovn geneve, tenant-network MTU, and amphora. The measured underlay MTU is a Phase-0 gate -- do not assume jumbo.
  - Phase-A rollback posture: go straight to IPv6-only on the internal three (data / storage / replication) rather than dual-stack-then-flip, given plane isolation and the Octavia ULA proof. Dual-then-flip is the conservative alternative if Phase-0 verification of Ceph-over-v6 / geneve-over-v6 is not clean.

**Rationale / tradeoffs:** matches the operator directive (v6 wherever possible, v4 only where forced) and maximizes Roosevelt rehearsal value (proves Ceph-over-v6, geneve-over-v6, native v6 tenant routing). The two forced-v4 points (PXE, datastore east-west) are explicitly justified rather than incidental. Native routing plus SLAAC for tenants is the IPv6-correct default; the MTU budget is the classic nested-OpenStack failure mode, handled by measurement plus jumbo. Ceph-over-v6 (`ms_bind_ipv6`) and OVN geneve-over-v6 remain Phase-0 verification gates in the buildout doc.

**References:** activates the developed IPv6 plan (its Phase A/B/C); carries D-016 / D-074 (tenant CIDR policy); supersedes the DR-seed v4 replication carrier; feeds D-106 (AAAA), D-108 (replication), D-109 (Vault v6-SAN, kept standalone per 2026-07-09 operator ruling -- not merged here). NetBox is IPAM apex.

**Remaining open item:** the exact NetBox literals (org ULA /48, per-DC GUA carve, DC2's v4 supernet) -- pending NetBox assignment (gap #3 in `docs/dc-dc-deployment-workflow.md`'s tooling gap register), not a ratification question.

---

## D-102: IPv6 tenant addressing and MTU sub-policy (VR1)

**Status:** MERGED INTO D-101 (2026-07-09, operator ruling). Content below is retained for numbering/history only, per this repo's append-only decision-record discipline -- D-101 is now the current, authoritative text for this sub-policy. Do not amend the body below; amend D-101 instead.

**Decision (leans; operator to ratify):**
- Tenant v6 routing: native GUA routing, no NAT66 -- the simulated OPNsense edge routes tenant GUA prefixes (prefix delegation or static routes). NAT66 is discouraged and we own the edge, so native is both correct and feasible.
- Tenant v6 block size: /60 per tenant default (~4080 per cloud); /56 available on request.
- Tenant address mode: SLAAC default (RA); DHCPv6-stateful available for tenants needing deterministic addressing.
- MTU / geneve-over-v6: compute the exact byte budget in Phase 0 against the MEASURED host L2 MTU. An IPv6 outer header adds 20 bytes over IPv4 (40 vs 20); geneve-over-v6 overhead is about 40 (IPv6) + 8 (UDP) + 8 (Geneve base) = 56 bytes, before nested-virt and options. Recommendation: raise the vcloud underlay to jumbo (9000) end-to-end so tenant MTU stays 1500 and amphora/geneve fit; if the underlay is pinned at 1500, tenant MTU drops to about 1444 (v6-geneve) and MUST be set consistently across ovn geneve, tenant-network MTU, and amphora. The measured underlay MTU is a Phase-0 gate -- do not assume jumbo.
- Phase-A rollback posture: go straight to IPv6-only on the internal three (data / storage / replication) rather than dual-stack-then-flip, given plane isolation and the Octavia ULA proof. Dual-then-flip is the conservative alternative if Phase-0 verification of Ceph-over-v6 / geneve-over-v6 is not clean.

**Rationale:** matches the developed plan's recommendations; native routing plus SLAAC is the IPv6-correct default; the MTU budget is the classic nested-OpenStack failure mode and is handled by measurement plus jumbo.

**References:** extends D-101. Ceph-over-v6 (`ms_bind_ipv6`) and OVN geneve-over-v6 are Phase-0 verification gates in the buildout doc.

---

## D-103: OpenTofu / MAAS / Juju lifecycle seam (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling). Resolves the double-specified node-VM ownership question.

**AMENDED 2026-07-16 (D-123 Model B):** the DC node-VM PLACEMENT half of this seam is SUPERSEDED --
the vr1-dc0/dc1 OpenStack node VMs are now created INSIDE the site containment VM (`vvr1-dc0`), not at
vcloud level, and the in-VM MAAS rack discovers them there. The OpenTofu-creates + MAAS-discovers
lifecycle is otherwise unchanged; only the libvirt HOST the nodes land on moves from vcloud to
`vvr1-dc0`. Office1's placement is unaffected.

**Decision:**
- OpenTofu owns create / destroy of: the libvirt domains that stand in for DC "hardware"; all virtual networks (the dark-fiber, plane, and ISP-edge segments); the OPNsense edge VMs; and the Office1 service VMs (MAAS-region, NetBox, GitBucket). OpenTofu runs from the Office1 operator VM against the vcloud host libvirt.
- OpenTofu registers each DC's libvirt host to that DC's MAAS rack controller as a virsh VM-host, so MAAS DISCOVERS the OpenTofu-created node VMs.
- MAAS owns commission / deploy / power / release of those node VMs; it does NOT compose new ones. Juju then deploys the OpenStack bundle onto the MAAS-deployed machines.
- NetBox remains the IPAM apex; MAAS and overlays consume NetBox values, never the reverse. **(DOCFIX-195, 2026-07-15: during VR1 that apex is `office1-netbox` (10.10.1.10) -- the working draft every consuming system reads; `netbox.baldurkeep.com` is the eventual production apex, frozen as the v1 reference draft with merge-back deferred. Defers the write, does not change this consume-not-reverse rule.)**

**Rationale / tradeoffs:** this keeps the entire physical substrate in versioned OpenTofu (auditable, reproducible, and exactly the "operator scripts the deployment from Office1" model), while MAAS owns the provisioning lifecycle -- mirroring Roosevelt, where MAAS enlists hardware that already exists rather than creating it. "MAAS owns lifecycle" here means the provisioning lifecycle, not domain creation; that reconciles the two operator inputs.

**Delta callout (test-env shim):** the OpenTofu "create the node VMs" module has NO Roosevelt analog. In Roosevelt that step is physical racking plus BMC enlistment, and MAAS enlists real hardware. This module is tagged in the buildout doc as a virtual-only shim and must NOT be mistaken for reusable Roosevelt IaC. Everything downstream of enlistment (MAAS commission / deploy, Juju bundle) IS Roosevelt-transferable.

**References:** ratifies MAAS-as-VM-host in the enlist sense; NetBox apex per the operating skill and environment reference; feeds Phases 2-4 of the buildout doc.

---

## D-104: Juju controller topology and HA posture (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling). Adds the controller-topology / HA posture the DC-DC phase was gated on. References D-071 (patch cadence); does NOT amend it.

**Decision:**
- One Juju controller PER DC, deployed in-DC (on a MAAS-deployed machine in that DC), each managing only its own cloud. DR-independent: a controller loss affects only its DC.
- Single-unit (not HA) for VR1, with the confirmed backup posture (juju 3.6 `create-backup` / `download-backup`). Controller backups scheduled and a restore drill run in Phase 5.
- HA (3-unit) DEFERRED to Roosevelt, when bare-metal resources allow. Recorded as a known VR1 test-only posture, not a Roosevelt recommendation.
- Office1 hosts NO OpenStack Juju controller (Office1 = MAAS-region + OpenTofu + NetBox + GitBucket only).
  **[AMENDED by D-116, 2026-07-13: the GitBucket half is SUPERSEDED. No Office1-local GitBucket is built; `git.baldurkeep.com` remains the git service of record. Office1 = MAAS-region + OpenTofu + NetBox + Tailscale.]**

**Rationale / tradeoffs:** per-DC single controller is DR-honest (independence) and resource-frugal on the 256 vCPU / 1 TiB / 10 TiB host. A shared or Office1-hosted controller would couple both DCs' manageability -- a controller outage would take out both -- the opposite of what a DR drill should prove. Single-vs-HA is a deliberate test-only economy; backups are the mitigation and are drilled. Roosevelt scales to 3-unit HA per DC.

**References:** relates to D-071 (routine update cadence / Juju controller patch policy) without amending it; interacts with D-108 (per-DC control-plane backup / restore in the DR drill).

---

## D-105: CAPI management-cluster placement and COS / Magnum scope (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling). Carries D-035 to the two-DC topology.

**Decision:**
- Each DC runs its OWN in-cloud, single-homed capi-mgmt cluster (per D-035), independent per DC. No shared or Office1 management cluster.
- COS (observability) and Magnum / CAPI are in scope in BOTH DCs.
- The Magnum `domain-setup` action is re-run after every teardown / redeploy per DC (D-046 trap: magnum reports "Unit is ready" regardless).

**Rationale:** D-035 already placed capi-mgmt in-cloud single-homed; per-DC keeps each cloud self-contained and DR-independent. COS per DC (not central at Office1) keeps observability inside each independent failure domain.

**References:** carries D-035; carries D-046 (domain-setup re-run); COS / Magnum scope confirmed by operator.

**Sub-item RULED 2026-07-09 (operator):** COS is per-DC-only for VR1, CONFIRMED (no Office1 roll-up) -- revisit for Roosevelt.

---

## D-106: Designate reactivation (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling). SUPERSEDES D-019 (Designate deferral); REACTIVATES D-008 (DNS architecture).

**Decision:**
- Designate is deployed in-bundle in each DC, backend `designate-bind` (bind9, the Charmed-OpenStack-native backend, lowest delta).
- Naming per D-008 instantiated with the VR1 labels: `<service>.omega.dc1.vr1.cloud.neumatrix.local` and `...dc2.vr1...`. Per-DC INDEPENDENT zones (independent Keystones), served with A and AAAA records (AAAA for the v6-enabled planes per D-101).
- The D-008 bootstrap order applies per DC: static /etc/hosts bootstrap -> `os-public-hostname` per API charm -> Vault issues FQDN-SAN certs -> Designate zone plus A/AAAA -> Neutron `default_dns_domain` and `dns_servers` -> tenant subnets with `--dns-nameserver`.

**Rationale / consequence:** reactivating Designate plus FQDN-SAN certs removes the D-019 documented root cause (metal-only charms pulling the public FQDN endpoint they cannot resolve -- the D-021 amphora-pipeline constraint). The D-008 convention was designed multi-DC from the start (`dc0.vr0`), so this is reuse, not new design -- minimal delta.

**References:** supersedes D-019; reactivates D-008; resolves the D-019 / D-021 root cause; AAAA depends on D-101. Office1 headend services use a region-level subdomain outside the per-cloud convention (e.g. `maas.office1.vr1...`) -- exact scheme is a buildout-doc detail.

**Sub-item adopted per its stated lean (2026-07-09):** cross-DC name resolution (a DC1 tenant resolving a DC2 name) is NOT needed for VR1 (independent clouds) -- this was not one of the six items the buildout doc listed for explicit operator redline, so it is adopted along with the rest of D-106 rather than separately re-asked; revisit only if a federated-service requirement emerges.

---

## D-107: Airgap posture, per-DC artifact mirror, and NTP (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling).

**Decision:**
- Airgap is enforced at the NODE level: OpenStack nodes (control / compute / ceph) reach NO internet directly. They pull all artifacts (apt, snap, charmhub, container registry, images including amphora) ONLY from an in-DC mirror.
- PER-DC mirror (not a single Office1 mirror): each DC hosts its own mirror so a DC can redeploy independently even if Office1 or the peer DC is down -- a DR requirement the drill exercises.
- Controlled egress lives on each DC's OPNsense edge, narrowly scoped: NTP-only, plus the mirror's upstream sync. Nodes get time (chrony) from their DC edge, which syncs upstream to internet NTP pools. Office1 is NOT in the core-service path (no NTP, no node artifacts served from Office1).

**Rationale / tradeoffs:** this reconciles "airgap-honest" with the operator's "each DC syncs internet NTP" -- the airgap is at the node boundary, not the site boundary, and the edge holds a minimal controlled egress. Per-DC mirrors cost more disk (a factor in the D-101 / Ceph disk budget) but buy true DC-redeploy-independence, which a single Office1 mirror cannot. The Roosevelt airgap posture is the same shape (per-DC mirror, edge-controlled egress).

**References:** pairs with D-100 (per-site ISP edge), D-101 (disk budget), and the Phase-0 disk math.

**Sub-item RULED 2026-07-09 (operator):** mirror sync topology is independent upstream pull per DC, CONFIRMED (not one DC pulling and seeding the peer) -- for full DC-redeploy-independence; measure the disk cost in Phase 0.

### D-107 -- AMENDMENT (2026-07-16): scope "node artifacts" (C4); make the DR-independence claim honest (C5)

Two review corrections (R3-F03/F05); the per-DC-mirror + edge-egress + NTP posture is otherwise unchanged:
- **(C4) "no node artifacts served from Office1" is SCOPED to the supply-chain mirror** (apt / snap /
  charmhub / container registry / amphora images) -- the class served to nodes at and after Juju deploy.
  It EXCLUDES the intra-MAAS provisioning-image channel: under D-123 the DC rack proxies MAAS ephemeral
  commission/deploy (PXE) images FROM the Office1 region, a different control-plane channel. Both D-107
  and D-123 stand once "node artifacts" is read this way.
- **(C5) The DR claim "a DC can redeploy independently even if Office1 ... is down -- a DR requirement
  the drill exercises" is CORRECTED.** (i) Redeploy is a MAAS operation, and with the only region on
  Office1 (D-123, matching buildout-design's PERMANENT single-region), an Office1 outage removes
  commission/deploy/power from BOTH DCs -- a permanent architectural property, not stale VR1 wording.
  (ii) The Phase-5 drill invokes NO MAAS -- it exercises Ceph/Glance/Cinder/Neutron WORKLOAD failover,
  not node redeploy. **Honest scope:** the per-DC mirror gives ARTIFACT + WORKLOAD-FAILOVER independence
  (running DC clouds survive an Office1 outage -- MAAS is a day-1/2 dependency, not runtime); node
  PROVISIONING is unavailable during an Office1-region outage, accepted by design (a standby region per
  DC would contradict the single-region decision). The peer-DC-down half of the original claim stands.

---

## D-108: Cross-DC replication mechanism (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling). Engineers the mechanism for the SETTLED Tier-A DR decision (`docs/archive/dc-dc-replication-DR-seed.md`); does not reopen the posture.

**Decision (mechanism):**
- Cinder volumes: `cinder-backup` to a cross-DC-replicated object store via `ceph-radosgw` MULTISITE (a zonegroup with the two DCs as zones; bucket-level bidirectional sync) as the replicated backup target. A restore reconstitutes the volume with its Cinder metadata in the peer -- the honest cross-independent-cloud path. Multisite is chosen over a single mirrored bucket for bidirectional, per-object async replication with conflict handling.
- Glance images: `ceph-rbd-mirror` of the Glance pool, snapshot-based, two-way, with a re-register step on failover. Daemon topology: one rbd-mirror daemon per DC, peering over the replication plane; daemon HA is single-unit for VR1 (matches the D-104 controller economy), noted for Roosevelt scale-up.
- Nova ephemeral: NOT replicated (rebuild from image plus volume).
- Carrier: the cross-DC leg (radosgw multisite sync plus rbd-mirror peer traffic) rides the IPv6-only ULA replication plane (per D-101; corrects the DR-seed v4 carrier). Explicit inter-DC route plus a bandwidth budget (MTU-aware; see D-102), distinct from intra-DC OSD cluster traffic.
- Consistency groups for multi-volume apps where a crash-consistent set matters.
- Staging: prove one-way (single mechanism, single direction) plus a clean failover / failback drill FIRST, then enable two-way. RPO ~15 min tunable; RTO is the measured drill output, not an SLA.
- Neutron state (ports / FIPs / security groups) is NOT replicated; it is recreated on failover per the runbook.

**Rationale:** matches the settled seed; multisite plus rbd-mirror are the delta-minimal Ceph-native primitives. This answers the seed's open questions (multisite vs mirrored bucket -> multisite; daemon HA -> single-unit VR1; peer bootstrap -> on the replication plane).

**References:** engineers `docs/archive/dc-dc-replication-DR-seed.md`; carrier per D-101; interacts with D-104 (per-DC control-plane backup / restore in the drill); the failover / failback skeleton is fleshed out in the buildout doc.

---

## D-109: Vault posture for VR1

**Status:** ADOPTED 2026-07-09 (operator ruling). Confirmed to STAND ALONE, not merged into D-101 despite D-102 folding in the same day -- Vault/security posture is a distinct-enough surface to keep independently amendable.

**Decision:**
- Per-DC INDEPENDENT Vault roots (each DC's Vault is its own root CA; no regional root-of-trust this phase).
- Vault pinned 1.8/stable (the only viable charm lineage for this stack; the 1.16 operator lineage is ruled out per D-068). The D-068 EOL-substrate change is deliberately HELD OUT of VR1 to keep variables constant.
- Each DC's Vault issues certs with both IPv4 and IPv6 (ULA) SANs for the dual-stacked metal-internal API endpoints (per D-101).

**Rationale / tradeoffs:** independent roots keep the two clouds cryptographically independent (DR-honest) and match the confirmed operator ruling. Holding 1.8/stable isolates the DC-DC variables from the unresolved substrate modernization; the deliberate non-change is recorded against D-068 so it is on the record, not an oversight.

**References:** references D-068 (substrate hardening deferred to Roosevelt; getting off EOL 1.8.8 stays open there); v6-SAN per D-101.

---

## D-110: DC-DC numbering and repo-seed policy

**Status:** ADOPTED 2026-07-09 (operator ruling; meta / process).

**Decision:**
- The DC-DC decision series starts at D-100 in the repo `openstack-caracal-dc-dc`, seeded from `openstack-caracal-ipv4` at commit 8813efc (verified: new-repo HEAD equals 8813efc; `design-decisions.md` ceils at D-075).
- D-076 through D-099 are RESERVED as a source-project band and are NEVER assigned in this repo. (D-076 is claimed by the source-project dashboard workstream; 077-099 are runway.)
- A genuinely-shared future source-project decision is referenced by its original id (e.g. "see openstack-caracal-ipv4 D-0NN"), never copied and renumbered into this repo.
- `ledger-scan` is confirmed 3-digit-safe for the D-series (`grep -oE '^## D-[0-9]+' | sort -n`), so D-100+ next-free is correct. The deprecated `repo-lint` next-free path is band-limited above 099 and must not be used for next-free (standing rule).

**Rationale:** a round D-100 floor gives a clean project base plus about 24 numbers of cross-repo collision runway (the source is a maturing project unlikely to reach 100); reference-not-copy preserves provenance. A distinct prefix (DD-) was considered for permanent cross-repo immunity and declined for VR1 -- it would fork `ledger-scan`, the fence validator, and every D-NNN cross-reference, against minimize-delta.

**References:** process entry; governs all D-1NN entries in this repo.

**Log-for-sweep (not actioned):** `ledger-scan` DOCFIX / BUNDLEFIX patterns use `[0-9]{3}` -- safe through 999, latent at the 1000 boundary (the same class as the 100-boundary bug already fixed for those series).

---

## D-111: VR1 per-DC v6 subcarve aligned to the deployed NN mnemonic

**Status:** ADOPTED 2026-07-11 (operator deferred to Code stream's recommendation; ratifies `docs/dc-dc-netbox-buildout-scope.md` sub-decision #1).

**Decision:**
- `netbox/dc-dc-prefixes-import.py`'s per-plane v6 subcarve is aligned to the DEPLOYED VR0-DC0 / Willamette net-byte (NN) mnemonic, replacing the original contiguous-plane-index proposal (which was explicitly un-ratified). Each plane gets a `/60` container + `/64` active, mirroring the live template:
  - provider-public (GUA, out of `DC_GUA_PREFIX`): `/60` + `/64` active at `:10`, API-VIP `/64` at `:11`.
  - metal-admin + metal-internal (ULA) SHARE the metal `/60` at `:20` -- admin `/64` at `:20`, internal `/64` at `:21` -- mirroring how provider `:10` / VIP `:11` share the provider `/60` (the deployed model had a single `metal` plane; the D-052/D-053 six-plane split nests both legs as `/64`s in it).
  - data-tenant `:30`, storage `:40`, replication `:50` (ULA): each its own `/60` + `/64`.
- ULA legs are carved from a per-DC `/56` of the org ULA `/48`, indexed to the GUA site nibble (dc1 -> `...:02xx`, dc2 -> `...:03xx`) so the 4th hextet reads `DC.NN` in BOTH families.
- Addresses are computed by direct index arithmetic, NEVER `list(...subnets())` enumeration (preserves the DOCFIX-181 no-hang property).

**Rationale:** the operator's standing directive is to keep VR1 as close to the real deployment as possible. The live NetBox export (2026-07-11) shows VR0-DC0 / Willamette use exactly this NN mnemonic + `/60`-per-plane structure; aligning the tool means VR1's GUA and ULA both read `:20`=metal, `:30`=data, etc., consistent with the deployed clouds, instead of arbitrary contiguous indices.

**Out of scope (mirrored separately, not via this six-plane tool):** the `lbaas-mgmt` (`:80`), `vpn` (`:e0`), `oob` (`:f0`) networks from the deployed template. The G5 DC-site rename (`vr1-dc0`/`vr1-dc1` -> `vr1-dc1`/`vr1-dc2`) is a distinct ruling, recorded under its own D-number when executed.

**References:** `netbox/dc-dc-prefixes-import.py` (`carve_v6`), `docs/dc-dc-netbox-buildout-scope.md` section 5.1, D-101 (family matrix), D-052/D-053 (six-plane), the 2026-07-11 live NetBox export. Harness `tests/dc-dc-prefixes-import/` 53/53. Applies to the Office1 simulation NetBox first; flows back to `netbox.baldurkeep.com` as refined draft.

---

## D-112: OPNsense edge config delivery -- the Configuration Importer cannot work on a nano image

**Status:** ADOPTED 2026-07-12 -- **option (c), post-boot network provisioning.** Operator ruling:
"C since it is the way people automate opnsense" -- prefer the mainstream, industry-standard
automation path (the OPNsense REST API, as used by `opn-cli` and the Ansible OPNsense collection)
over a bespoke image-bake or media-import trick. Consistent with the standing "transferable answers
beat quick fixes" constraint: a real Roosevelt edge is provisioned over the network too.

IMPLEMENTATION NOT STARTED. The BOOTSTRAP sub-problem below is unsolved and is itself unruled --
"push the config over the API" is not reachable from a factory-default OPNsense without first
solving how we authenticate and how we reach it. Do not start building until B1/B2 are ruled.

**Bootstrap problem (measured/sourced, must be solved BEFORE any config push):**
1. **No route to it.** Factory-default OPNsense serves LAN on `192.168.1.1/24` on `office1-local`.
   `office1-local` is an ISOLATED libvirt network -- the vcloud host has NO leg on `virbr2`
   (measured: no IPv4 on the bridge). So nothing can currently talk to the edge's LAN at all.
   The WAN side is reachable at the DHCP lease `172.30.1.126`, but OPNsense **blocks all inbound on
   WAN by default** -- so the WAN leg is NOT a way in.
2. **No credentialed API.** The OPNsense REST API authenticates with an **API key + secret**, not the
   root password. Keys are minted in the GUI -- a chicken-and-egg for a fresh box. SSH is
   **DISABLED** in the factory default, so `ssh` is not an entry point either.
3. **Our serial console is write-only.** `modules/opnsense-edge` attaches `<serial type='file'>` --
   a one-way capture. It CANNOT be typed into, so the console is not currently an entry point
   either. Making it interactive means `type='pty'` + `virsh console` (a module change + recreate).

**Unruled sub-choices (operator, next session):**
- **B1 -- how do we REACH it?** (i) temporarily add a host leg on `virbr2`
  (`ip addr add 192.168.1.10/24`) -- reversible, non-persistent, cheapest; or (ii) give
  `office1-local` a permanent host address in the OpenTofu network definition (touches D-100's
  network design); or (iii) drive it entirely over an interactive serial console and never need IP
  reachability for bootstrap.
- **B2 -- how do we AUTHENTICATE the first time?** (i) interactive serial console (`type='pty'`) ->
  root login -> enable SSH + install the ed25519 pubkey we already generated in
  `~/vr1-office1-creds/`, after which everything is plain SSH/SCP and fully scriptable; or
  (ii) authenticated GUI session with the default root credentials (curl + CSRF token) to mint an
  API key, then use the REST API proper.
  NOTE: (i) is console-driven but is NOT the rejected option (d) -- (d) was rejected for racing the
  importer's 7-second timeout. A console *login shell* has no timing race and can be driven at
  leisure. These are materially different risks; do not conflate them.

**Option (a) RE-EXAMINED 2026-07-12 at operator request and MEASURED INFEASIBLE on this host.**
Recorded so it is not re-litigated. Baking `/conf/config.xml` into the nano requires writing
FreeBSD UFS2 from Linux, and every route is closed here:

| Requirement | Measured result |
|---|---|
| Kernel UFS write | `# CONFIG_UFS_FS_WRITE is not set` -- driver is READ-ONLY (`CONFIG_UFS_FS=m`) |
| A FAT partition to write instead | NONE -- MBR (`dos`), a single FreeBSD `a5` partition; UFS only |
| FUSE UFS driver (`fuse-ufs`, `ufs2-tools`) | not present, not packaged |
| libguestfs (`guestfish` / `virt-copy-in`) | not installed |

Two ways (a) could still be FORCED, both REJECTED:
- **FreeBSD helper VM** (boot a FreeBSD live system with the nano as a second disk; FreeBSD mounts
  UFS rw natively; drop the file; power off). It genuinely works and `xorriso` IS present, so a live
  ISO could be remastered with an auto-run script -- but it means fetching + remastering a FreeBSD
  ISO and maintaining a builder VM forever to deliver one 4.5 KB file. Cost exceeds (c) entirely.
- **Raw byte-patch of the image** (pad the new XML to the default `config.xml`'s exact byte length
  and overwrite its data blocks in place, leaving the inode untouched). Probably works -- and
  REJECTED anyway: it silently corrupts if the file is not block-contiguous, and it is
  NON-TRANSFERABLE (you cannot byte-patch a real appliance's disk on Roosevelt hardware), which is
  precisely what this repo's governing constraint forbids.

**Net: examining (a) properly STRENGTHENED (c).** (a) is not merely less mainstream here -- it is
unavailable without a FreeBSD-shaped detour costing more than (c)'s whole bootstrap.

**BOOTSTRAP RULED 2026-07-12 (operator: "proceed with the C bootstrap"):**
- **B1 (reach) = (i)** temporary host leg on `virbr2` (`ip addr add 192.168.1.10/24`) -- reversible,
  non-persistent, no change to the D-100 network definition. Removed/re-addressed once the edge's
  LAN moves to `10.10.0.1`.
- **B2 (authenticate) = (i)** one interactive console login -> enable SSH + install the ed25519
  pubkey ALREADY generated in `~/vr1-office1-creds/`. Thereafter everything is plain SSH/SCP and
  fully scriptable. Chosen over GUI/CSRF-scraping to mint an API key (brittle, no gain).

**Once bootstrapped, the steady-state mechanism is:** push `config.xml` (or the equivalent API
calls) to the edge, reload, and verify. The config we already render (`opnsense-config.xml.tmpl`,
LAN `10.10.0.1/24` + DHCP `.100-.199`, WAN static `172.30.1.2`) is REUSED unchanged -- only the
DELIVERY changes.

**Consequently RETIRED (do not delete until (c) actually works):** the config-ISO delivery path --
`scripts/opnsense-build-config-iso.sh` + its harness, and the module's `config_seed` volume + cdrom
disk. They are inert (the Importer never reads them), not harmful. Remove them in the same change
that lands (c), not before.

**Finding (MEASURED, not inferred).** The Office1 edge boots (DOCFIX-188 memory + DOCFIX-190 ACPI)
but comes up on FACTORY DEFAULTS. The config ISO is well-formed and correctly attached -- verified:
ISO9660 labelled `OPNSENSE_CFG`, containing `CONF/CONFIG.XML` with our `<opnsense>` payload and
`10.10.0.1`; the guest sees it as `cd0` (`QEMU QEMU DVD-ROM`, 186x2048 = exactly our 380928-byte
image). The boot log shows the importer being invoked and then doing NOTHING:

    >>> Invoking import script 'importer'
    (no further output)

**Root cause -- architectural, not a bug.** `rc.syshook.d/import/20-importer` runs
`/usr/local/sbin/opnsense-importer -b`. Upstream source (opnsense/core, `src/sbin/opnsense-importer`):

    INSTALL="/.probe.for.readonly"
    ...
    if [ -n "${DO_BOOT}" ]; then
            touch ${INSTALL} 2> /dev/null
            if [ -f ${INSTALL} -a -f /conf/config.xml ]; then
                    bootstrap_and_exit 0
            fi

The `touch` is a PROBE FOR A READ-ONLY ROOT. On INSTALLER media the root is read-only, the touch
fails, the marker is absent -- so the importer proceeds to scan attached media (and yes, its scan
path does support cd9660). On a PRE-INSTALLED image (our nano) the root is WRITABLE and a factory
`/conf/config.xml` ALREADY EXISTS, so BOTH conditions hold and it `bootstrap_and_exit 0`s
immediately -- it never enumerates a single device.

**Therefore the Configuration Importer can NEVER fire on a pre-installed nano image, by design.**
Nothing was ever going to read our ISO.

**What the prior research got right and wrong** (`opentofu/README.md` "OPNsense deployment
research", items 2-3): it correctly established that the Importer exists and that its scan path
supports ISO9660 for VM automation. It never established that the Importer would RUN AT ALL on a
nano image -- and it does not. The module header has always flagged this exact mechanism UNVERIFIED;
that flag was correct and is now discharged NEGATIVE.

**Options (operator to rule; none implemented):**

- **(a) Bake `config.xml` into the image before first boot.** `opnsense-prep-image.sh` writes
  `/conf/config.xml` directly into the nano's UFS root. Cleanest and most immutable-infra; no
  importer, no ISO, no boot-time interactivity. RISK/UNVERIFIED: writing FreeBSD UFS2 from Linux --
  most distro kernels ship `CONFIG_UFS_FS_WRITE` off, so this may be infeasible without libguestfs
  or a FreeBSD helper. FEASIBILITY MUST BE PROVEN BEFORE ADOPTING.
- **(b) Switch from the nano image to the installer image.** On installer media the root IS
  read-only, so the Importer runs and the existing ISO mechanism works AS DESIGNED and unchanged.
  Cost: an actual install step per edge; needs checking whether it is non-interactive.
- **(c) Boot factory-default, then push config over the network.** Provision post-boot via
  OPNsense's API/SSH from Office1 (factory default LAN is `192.168.1.1/24` on `office1-local`, which
  we own). Most Roosevelt-transferable -- real edges get provisioned over the network too -- but it
  is two-phase and needs a bootstrap path onto the default LAN.
- **(d) Serial-console expect-scripting.** Drive the interactive importer prompt over the serial
  console. NOT RECOMMENDED: fragile, timing-dependent (the prompt has a 7-second timeout).

**Impacts if not (b):** `scripts/opnsense-build-config-iso.sh`, its harness, the module's
`config_seed` volume + cdrom disk, and the README research section all assume the ISO mechanism.

**References:** `docs/archive/incident-20260712-opnsense-edge-boot-triplefault.md`,
`docs/archive/changelogs/changelog-20260712-libvirt-acpi-kernel-panic.md` (DOCFIX-190), upstream
`opnsense/core:src/sbin/opnsense-importer` + `src/etc/rc.syshook.d/import/20-importer`.
Also gates Stage 3 (the per-DC OPNsense edges use this same module).

---

## D-113: is OPNsense the right edge platform, or should the simulated edges be VyOS/Linux?

**Status:** **ADOPTED 2026-07-13 -- option (a2): STAY ON OPNSENSE, MOVE CONFIG TO THE REST API.**
Operator ruling ("go ahead and continue with your rest API recommendation"), after two
constraints were surfaced during the discussion that narrowed the field decisively:

1. **GUI access is a REQUIREMENT** (operator, 2026-07-13). This ELIMINATES option (b) VyOS and
   option (c) plain-Linux outright -- neither has a GUI. Both had been the leading candidates
   until this was stated; recording it so the reasoning is not re-run from the wrong premise.
2. **VyOS LTS binaries are subscription-gated** (verified 2026-07-13, vyos.net/get + the
   Software Access Subscription page). Free VyOS means the ROLLING release -- a moving target
   that fights this repo's appendix-B version-pinning discipline -- or building from source.
   Independent of the GUI point, this was real friction.

Remaining open-source field was OPNsense / pfSense CE / OpenWrt / IPFire. pfSense REJECTED
(same GUI-owned XML lineage -- a migration that buys nothing). IPFire: weakest automation
story. **OpenWrt was the credible alternative** -- its UCI config is text-first BY DESIGN
(LuCI and the CLI read/write the same `/etc/config/*` files, so editing config as text is a
supported path, structurally unlike OPNsense's GUI-owned `config.xml`) -- but it costs a new
image pipeline, a fresh bootstrap problem (it also defaults to LAN `192.168.1.1`), and dnsmasq
instead of Kea, all to escape a problem fixable in place.

**The ruling rests on this:** the bug class was never "OPNsense". It was hand-authoring
`config.xml`, a GUI-owned format, when OPNsense ships a documented REST API with mature Ansible
collections modelling exactly what we got wrong (`dhcp_subnet`, firewall rules, system settings
as typed first-class resources). **None of DOCFIX-191/192/193 is expressible through the API** --
you cannot forget to enable sshd in a format where sshd is a typed field with a default. (a2)
keeps the GUI, keeps a WORKING routing+DHCP edge, keeps the two libvirt fixes, and removes the
defect source -- and it is the smallest change of anything considered.

**AMENDMENT 2026-07-13 -- the template is DELETED, not reduced (operator ruling: "I'll defer
to your lean").** The original (a2) plan was to REDUCE `opnsense-config.xml.tmpl` to a minimal
bootstrap (sshd + root key + console + a seeded API key). That turned out to be unnecessary:
every one of those is covered without any `config.xml` at all.

- sshd + root key: the **D-112(c) console bootstrap** installs them (proven on Office1).
- the API key: **`scripts/opnsense-bootstrap-apikey.sh`** mints one by calling OPNsense's OWN
  model (`Auth\User -> apikeys->add()`) over SSH -- no GUI click, and no re-implementation of the
  vendor's `$6$` crypt format (which would fail SILENTLY on any format drift).
- everything else -- DHCP, firewall, interfaces: **the REST API**.

So the provisioning chain has NO `config.xml` in it, and a renderer that nobody should ever run is
not a safety net -- it is a loaded gun pointed at a live router. The 2026-07-13 safety sweep is the
evidence: the repo still contained runbook steps instructing an operator to render a config and push
it to the edge, which by then would have CLOBBERED live API-managed DHCP.

**DELETED:** `opentofu/templates/opnsense-config.xml.tmpl`, `scripts/opnsense-render-config.sh`,
`scripts/opnsense-build-config-iso.sh`, and both their harnesses. The `opnsense-edge` module's
`config_seed` volume + cdrom disk are removed (the ISO was INERT -- D-112). The
`xorriso`/`genisoimage` prereq is dropped. All remain in git history.

**If you think the API cannot express something:** that is a finding to raise against D-113 -- not
a reason to hand-write XML.

**Superseded by this ruling:** the "stay on (a1)" lean in the original recommendation below.
(a1) is now explicitly REJECTED: its cost compounds per feature and it depends on an
undocumented self-heal of an internal format (see the 667-element finding).

**Scope.** The VR1 edges are SIMULATED customer-site / ISP routers (Office1 today; Stage 3 adds
one per DC). They are lab infrastructure, not a product surface. Nothing a tenant touches
depends on this choice; what depends on it is our own automation cost, for three edges.

### First, an honest cost accounting of 2026-07-12 (five bugs)

| # | bug | OPNsense's fault? |
|---|---|---|
| DOCFIX-188 | guest got 2 MiB RAM (`memory_unit`) -- triple fault | **NO** -- libvirt module bug; would hit ANY guest |
| DOCFIX-190 | `acpi=off` -> FreeBSD kernel panic | **NO** -- libvirt module bug; generic |
| DOCFIX-191 | rendered config had no sshd + no key -> would have locked management out | yes (hand-authored `config.xml`) |
| DOCFIX-192 | rendered config silenced the only console | yes (hand-authored `config.xml`) |
| DOCFIX-193 | LAN DHCP was never implemented (and ISC `<dhcpd>` would have been inert -- 26.1 is Kea) | yes (hand-authored `config.xml`) |

Two of five were platform-agnostic libvirt bugs whose fixes now protect every VR1 VM
(MAAS/NetBox/GitBucket, the DC nodes). Finding them via OPNsense was lucky, not costly.

**The other three share ONE root cause: we hand-author the appliance's internal, GUI-owned
`config.xml`.** That is a known-bad idea in any product of this class -- and pfSense would be
identical, since it is the same format lineage.

**2026-07-13 adds a fourth data point of the same kind.** A full-config push REPLACES
`/conf/config.xml`; the rendered template is 128 elements against a live 796, so it DROPS ~667
migration-populated elements -- including the only two firewall pass rules on the box
("Default allow LAN to any"), against a pf base policy of block-drop-all. It works only because
OPNsense regenerates them on boot (proven across 4 push+boot cycles, `/conf/backup/` trail).
**We are depending on an undocumented self-heal of an internal format.** That is the ongoing
cost, and it is the thing to weigh -- not the boot bugs.

### THE CONFOUND (must be resolved before ruling)

**D-112 ADOPTED option (c) with the explicit rationale "C since it is the way people automate
opnsense" -- i.e. the OPNsense REST API (`opn-cli`, the Ansible OPNsense collection).**

**What we actually built is NOT that.** We render a full `config.xml` from a Jinja-ish template
and scp it over the wire. That is a config-file bake delivered post-boot -- it satisfies the
letter of (c) (provisioning happens over the network, after boot) but NOT the rationale the
ruling was made on.

**Every OPNsense-specific bug we are now charging against the platform (DOCFIX-191/192/193, and
the 667-element drop) is an artifact of hand-authoring XML -- exactly what the REST API exists
to avoid.** The API models DHCP, sshd, console, and firewall rules as first-class resources;
none of those three bugs is expressible through it.

So the fair comparison is not "OPNsense vs VyOS". It is:

- **(a1) OPNsense + `config.xml` templating** -- the status quo. Cheapest right now; every new
  feature (VPN, HA, VLANs) means reverse-engineering another slice of an internal format, and
  the self-heal dependency above is permanent.
- **(a2) OPNsense + the REST API** -- what D-112's rationale actually called for. Config becomes
  declarative API calls; the internal format stops being our problem. Cost: retire the template,
  build/adopt an API client (`opn-cli` is packaged and mainstream), and keep only a MINIMAL
  bootstrap config (sshd + key + console) as a file. The boot bugs stay fixed either way.
- **(b) VyOS** -- purpose-built for this. Plain declarative text config, first-class cloud-init,
  a real config API; it exists to be a lab/edge router. Friction: LTS images are
  subscription-gated (rolling is free), it is less "appliance-like" than a real site router, and
  switching discards a WORKING, routing, DHCP-serving edge to re-run discovery on a new platform.
- **(c) Plain Linux router** (Debian/Ubuntu + nftables + Kea + FRR, provisioned by cloud-init).
  Maximum fidelity to the rest of our tooling -- same image pipeline, same cloud-init, same
  Ansible/OpenTofu idioms as every other VR1 VM, and zero appliance-internal formats. Friction:
  we hand-build router behaviour that OPNsense/VyOS ship; least faithful to "a real site runs an
  appliance".
- **(d) pfSense** -- **REJECT.** Same config-format lineage as OPNsense; buys nothing, costs a
  migration.

### Recommendation (mine; the operator rules)

**Stay on OPNsense -- but the real question is (a1) vs (a2), and I lean (a2).**

Rationale for staying: the hard part is DONE and MEASURED WORKING (boots, routes, NATs, reaches
the internet, serves DHCP, managed over SSH with a verified push pipeline). Edges 2 and 3 reuse
the module and template -- different tokens, same everything -- so the cost paid was one-time,
not per-edge. Switching platforms discards that and re-runs the same class of discovery
elsewhere. VyOS is the defensible blank-sheet pick, but we are not on a blank sheet.

Rationale for (a2) over (a1): (a1)'s cost COMPOUNDS. Each new feature is another slice of
reverse-engineered XML plus a self-heal we do not control. (a2) pays a bounded, one-time cost to
get onto the path D-112's ruling actually named, and it is the honest reading of that ruling.

**What would change my mind entirely (i.e. rule (b)/(c)):** if, having moved to (a2), the API
still forces us to reverse-engineer the config format for each new feature -- then OPNsense is
not buying us the appliance abstraction we are paying for, and VyOS's plain-text config wins.

### Cost of deferring

Low but not zero: Stage 3 builds two more edges from this same module. Ruling BEFORE Stage 3
means the choice is made once; ruling after means migrating three edges instead of one.

**Evidence:** `docs/archive/changelogs/changelog-20260712-office1-opnsense-edge-build.md`,
`docs/archive/changelogs/changelog-20260712-opnsense-edge-boot-fixes.md`,
`docs/archive/changelogs/changelog-20260713-office1-dhcp-apply.md` (the 667-element self-heal, measured),
D-112 (the (c) ruling and its stated rationale).

### D-113 -- AMENDMENT (2026-07-14): "interfaces: the REST API" IS FALSE. Base-interface addressing has NO API.

A factual correction to this decision's own text, not a re-ruling. The (a2) rationale above ends
with: *"everything else -- DHCP, firewall, **interfaces**: the REST API."* **The `interfaces` half
of that claim is wrong**, and it was never measured before being written -- it was inferred from
DHCP and firewall being API-native.

**MEASURED on the live Office1 edge (2026-07-14), OPNsense 26.1:**

| Surface | Served by | API? |
|---|---|---|
| Interfaces > [LAN] / [WAN] -- **address, incl. `ipaddrv6`** | `/interfaces.php?if=lan` | **NO -- legacy PHP page, config.xml-backed** |
| Interfaces > Devices (bridge/gif/gre/lagg/loopback/vlan/vxlan) | `/ui/interfaces/*` | yes |
| Interfaces > Virtual IPs | `/api/interfaces/vip_settings/*` | yes (`ipalias`/`carp`/`proxyarp`; iface `wan`/`lan`/`lo0`) |
| Services > Router Advertisements (radvd) | `/api/radvd/settings/*` | yes (`getEntry`/`addEntry`/`searchEntry`) |
| Services > Kea DHCPv4 / DHCPv6 | `/api/kea/*` | yes |

Only the **Devices** and **Neighbors** sub-trees were migrated to MVC. A base interface's IP
addressing was not. So there is no API call that sets LAN's `ipaddrv6`.

**What this does and does NOT change:**

- **It does NOT reopen (a2).** The prohibition D-113 actually adopted is on **hand-authored
  `config.xml` and full rendered pushes** -- the clobber path that would wipe the API-managed DHCP.
  D-113 also records **"GUI access is a REQUIREMENT"** as the constraint that eliminated VyOS and
  plain-Linux. A one-time operator GUI edit is therefore the vendor interface this decision
  deliberately PRESERVED. It is not a violation and must not be described as one.
- **It DOES mean the provisioning chain is not 100% API-scriptable** for any surface that needs a
  base-interface address -- which is every edge that gets IPv6, i.e. Office1 now and the DC0/DC1
  edges at Stage 3. That is a per-edge manual step unless the VIP path below works.

**The open question (an EXPERIMENT, not a plan):** the `radvd` entry model has **no prefix field**
-- it derives the advertised prefix from the interface's configured IPv6. Whether an **`ipalias`
VIP** counts as "the interface's configured IPv6" for OPNsense's radvd config generator, or whether
the generator only reads the primary `ipaddrv6`, is **UNKNOWN and must not be assumed**. If the VIP
path works, the whole IPv6 edge build is API-native and reproducible per DC. If it does not, the
LAN GUA is an operator GUI step at every edge. This is cheap and reversible to test on the lab edge
and is gated on an operator ruling.

---

## D-114: VR1 site containment VMs + MAAS-composed LXD VMs for the non-stack machines

**Status:** **ADOPTED 2026-07-13 (operator ruling).** AMENDS D-103 (which assigned Office1 three
sibling service VMs and put every VR1 domain directly on the vcloud host). Supersedes nothing else.

**AMENDED 2026-07-16 (D-123 Model B):** the containment-VM scope is EXTENDED for the DCs --
`vvr1-dc0`/`vvr1-dc1` now ALSO hold the OpenStack node VMs (this entry previously scoped them to the
MAAS/LXD headend + the non-stack LXD-composed machines ONLY, NOT the OpenStack nodes). This makes a
single-object `virsh destroy` a true site-down primitive for DCs, matching Office1. Consequence: the DC
containment VM is resized to hold the node fleet (~416 GiB, derived) with `expose_nested_virt = true` (D-124
rack sizing void). Office1's `voffice1` scope is unchanged.

**Context.** Stage 2 was about to build Office1 per D-103: three peer VMs (MAAS-region, NetBox,
GitBucket) created directly on vcloud libvirt, with Tailscale installed on the headend. The
operator described a different shape -- each site (Office1, DC1, DC2) living inside its OWN VM
environment, with MAAS running LXD to hold "the rest of the servers/services." An initial reading
of `bundle.yaml` found only Juju's `lxd:8`-style CONTAINER placements and concluded there was no
precedent for MAAS composing LXD VMs. **That conclusion was WRONG**, and is recorded here because
the error is instructive: the bundle's Juju LXD containers and MAAS's LXD VM host are TWO DIFFERENT
LXD usages in the same cloud, and reading only the bundle collapses them into one.

**Measured as-built evidence (VR0 / `openstack-caracal-ipv4` MAAS 3.7.2, operator screenshots
2026-07-13).** Seven machines, all Deployed, ALL powered via Virsh:

- `openstack0-3` -- 8 cores / 32 GiB / 1.1 TB, 6 spaces, tags `openstack, virtual`. Juju machines
  8-11; the OpenStack API services run in JUJU-created LXD CONTAINERS on these (21 `lxd:N`
  placements in `bundle.yaml`). MAAS never sees those containers, and never needed to.
- `juju` -- 2 cores / 4 GiB, tags `juju, virtual`. The controller.
- `lxd` -- 8 cores / 16 GiB / 687 GB, tags `lxd, virtual`. A MAAS-DEPLOYED, virsh-powered libvirt
  VM that is then REGISTERED BACK INTO MAAS as an LXD KVM host (project `default`, LXD **5.21.4**;
  host NICs enp1s0 / enp7s0 / lxdbr0).
- `tailscale` -- 2 cores / 2 GiB / 25 GB, Ubuntu 24.04, tags `pod-console-logging, virtual`.
  **It is a VM COMPOSED BY MAAS INSIDE the `lxd` KVM host.** This is the proof of the pattern.

So the pattern the operator asked for is not new -- it is VR0's own, already proven in production
on this cloud, and it costs 2 cores / 2 GiB per service machine.

### Ruling

1. **Non-stack machines become MAAS-composed LXD VMs.** Per site, one MAAS-deployed machine runs
   LXD and is registered back into MAAS as an LXD KVM host; the machines that live OUTSIDE the
   OpenStack stack (NetBox, GitBucket, Tailscale, and any future service box) are COMPOSED into it
   by MAAS. They are therefore MAAS-visible: enlisted, commissioned, deployed, powered, released.
   This is what the operator meant by visibility, and it is `lxd` + `tailscale` reproduced per site.
   **Explicitly NOT in scope:** the Juju-deployed OpenStack services on the DC nodes. Those stay in
   Juju-created LXD CONTAINERS (bundle `lxd:N` placements), invisible to MAAS, exactly as in VR0.
   Operator: "I don't care about visibility into the juju deployments into the openstack0-3
   machines as much as the other machines running outside of the stack."

2. **Each site gets a containment VM** -- `voffice1`, `vdc1`, `vdc2` -- simulating the facility that
   the site's servers and services operate within. Rationale (operator): **full-facility snapshots**
   and **the ability to simulate total geographical failure of a facility.** The second is the
   decisive one: D-108's DR failover drill currently has NO honest "hard-down an entire DC"
   primitive, and `virsh destroy vdc1` IS that primitive. This is a capability the flat model
   cannot offer at any price.

3. **Sequencing (operator ruling):** Office1 FIRST, fully up and running, on this model. DC1 is
   **GATED** behind it and is not started until Office1 is complete. DC2 follows DC1.

4. **LXD is version-pinned to the 5.21 LTS track.** MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7:
   LXD 6.7 consolidated API endpoints and MAAS's pinned pylxd 2.3.5 cannot speak to them (fixed in
   pylxd >= 2.3.9, not yet in a MAAS release). Canonical's own guidance is "use LXD <= 6.6 or the
   5.21 LTS release until further notice." VR0 is already on 5.21.4, i.e. on the safe side by
   accident of timing rather than by decision -- this makes it a decision. Goes in appendix-B.
   Source: https://discourse.maas.io/t/maas-incompatibility-with-lxd-6-7/15749

### What this changes in D-103

D-103 said OpenTofu owns creation of "the Office1 service VMs (MAAS-region, NetBox, GitBucket)" as
domains on the vcloud host. Under D-114, OpenTofu owns creation of the SITE CONTAINMENT VMs and the
site's networks; INSIDE a site, the service machines are composed by MAAS into LXD. D-103's core
principle is UNCHANGED and is in fact extended: MAAS still owns the provisioning lifecycle, and
still does not compose the OpenStack NODE VMs (`modules/node-vm` pre-creates those; `maas_vm_host`
registers the virsh host so MAAS DISCOVERS them -- `maas_vm_host_machine` remains ruled out for
nodes). The composition right granted here is scoped to the LXD VM host and the non-stack machines.

### Known risk, and its entry gate

**Nesting depth.** `vcloud` is ITSELF a KVM guest (measured: `systemd-detect-virt` -> `kvm`). A site
containment VM therefore puts the site's servers one level deeper than VR0 runs them: DC node VMs
land at L3 and OpenStack TENANT INSTANCES at L4. VR0 proves nesting works to the depth IT uses
(`tailscale` inside `lxd`), but VR0's hosts sit beside vcloud on the same 10.17.11.0/22 lab network
and are thus one level SHALLOWER than a VR1 pod. L4 tenant instances are UNPROVEN, and nova-compute
needs working KVM at the bottom of that stack. The 2026-07-12 OPNsense BTX triple-fault is prior
evidence that this environment is already fragile at depth.

**Office1 IS the nesting probe -- and that is a feature of the sequencing, not a coincidence.**
(An earlier draft of this entry claimed "Office1 needs no nested KVM." That was WRONG and is
corrected here: LXD **virtual machines** are qemu/KVM guests, so composing NetBox/GitBucket/
Tailscale as LXD VMs inside `voffice1` requires nested KVM at exactly the same depth the DC nodes
would need it -- L3. Only LXD *containers* would be free, and MAAS does not enlist containers.)
Office1 is therefore the CHEAP place to prove the model: no OpenStack, no Ceph, small VMs. If an
LXD VM boots inside `voffice1`, the pod model is proven to L3 and DC1 inherits that proof.

**Entry gate for DC1:** (a) Office1 fully up on this model (operator ruling: DC1 is gated behind
it); (b) inside `vdc1`, confirm nested KVM is exposed and a guest actually BOOTS -- which also
proves L4 is reachable for tenant instances, the one depth VR0 has never exercised. If L3 fails at
Office1, the containment-VM model is abandoned early having cost one VM. If L3 holds but L4 fails
at DC1, DC compute stays flat and Office1 keeps its containment VM. Both outcomes are scoped
retreats, not redesigns -- the ruling is deliberately written to make them cheap.

**DHCP.** OPNsense/Kea is authoritative on `office1-local` (10.10.0.0/24, pool .100-.199). LXD's
`lxdbr0` runs its OWN dnsmasq -- that is FINE as long as it stays on LXD's internal NAT bridge and
is never bridged onto `office1-local`. Two DHCP servers on one L2 is an intermittent failure that
is genuinely unpleasant to diagnose. Any LXD networking change must not put `lxdbr0` on the site LAN.
Corollary bonus: `voffice1` booting on `office1-local` will take the FIRST REAL DHCP LEASE from Kea
-- a path proven only at the daemon level so far, never end to end.

### Delta to Roosevelt

The containment VM is simulation-only machinery: in Roosevelt a "facility" is a building full of
real servers, not a VM, and it is torn down by pulling power, not `virsh destroy`. That is an
ACCEPTED delta, bought deliberately: it is what makes a geographic-failure drill executable at all
in a virtual rehearsal. The LXD half carries NO delta -- it is exactly what VR0 runs today and what
Roosevelt would run on real hardware (a service host running LXD, registered to MAAS).

> **D-119 NAMING AMENDMENT (2026-07-14).** The DC containment VMs named `vdc1`/`vdc2` above are
> **RENAMED `vvr1-dc0` / `vvr1-dc1`** to match the region-qualified DC namespace. Neither is built
> yet (only `voffice1` exists), so this costs nothing -- and it removes a genuinely dangerous
> ambiguity: this decision's own DR primitive is **`virsh destroy vdc1`**, which under the old
> naming would read as "destroy VR1 DC1" while actually meaning "destroy VR1 DC0". A mislabelled
> destroy command in a DR drill is not cosmetic. `voffice1` KEEPS its name (the office keeps its
> number, per D-119). The `vdc1`/`vdc2` spellings in the body above are HISTORICAL.

## D-115: VR1 Office1 addressing -- the office carve, and the v4 role gaps it exposed

**Status:** **ADOPTED 2026-07-13 (operator ruling -- see the amendment at the end of this entry).** (Raised and ruled the same day; the earlier draft state is described in the amendment, kept off this Status line so `ledger-scan` reads the ADOPTED token cleanly -- review R3-F09.) Found during Office1 deployment
testing: the headend was built on address space that NetBox never assigned. Per the operator, the
IPAM carve matters MORE than the deployment already standing on it -- "we can always change the
current IP addressing on the deployment; getting the IPAM carve proper and correct is more
important than going back and correcting deployment work." Nothing here is blocked; Office1 runs.
This is about ratifying or replacing what it runs on, BEFORE NetBox/GitBucket land on it.

### What the IPAM apex actually encodes (MEASURED 2026-07-13 against netbox.baldurkeep.com)

**IPv6 is a complete, consistent, region-hierarchical model.** `2602:f3e2::/36` (ARIN, Abysius Inc).
Every REGION gets a `/40`; inside it, SITES are `/48`s on a FIXED slot pattern:

| slot | meaning | VR0 (`e00::/40`) | VR1 (`f00::/40`) | Eugene (`100::/40`) |
|---|---|---|---|---|
| `X00::/48` | Infrastructure / site-to-site links | `e00::` | `f00::` | `100::` |
| `X01::/48` | **Offices** | `e01::` | **`f01::`** | `101::` |
| `X02::/48` | Datacenter (first) | `e02::` (VR0 DC0) | `f02::` | `102::` Willamette |
| `X03::/48` | Datacenter (second) | `e03::` (VR0 DC1) | `f03::` | `103::` Roosevelt |

Inside an office `/48`: a reserved `/56`, then ONE `/56` PER OFFICE, then a `/64` office subnet.
VR0 Off0 = `e01:100::/56` -> `e01:100::/64`. Eugene's Charnelton = `101:100::/56`. Inside a DC `/48`:
the D-111 NN mnemonic (`:10` provider, `:11` provider API VIP, `:20` metal, `:30` data, `:40`
storage, `:50` replication, `:80` lbaas-mgmt, `:e0` VPN, `:f0` OOB), each `/60` + `/64`.
Also global: `2602:f3e2::/40` inter-regional infrastructure, `2602:f3e2:fe::/48` **Edge Networks**,
`2602:f3e2:ff::/48` OpenStack tenant reservation.

**IPv6 for Office1 therefore has NO design freedom** -- the pattern determines it:
`2602:f3e2:f01:100::/56`, office subnet `2602:f3e2:f01:100::/64`. VR1's office `/48` already exists.

**IPv4 is a much thinner, ROLE-based model** -- and this is where the gaps are:
`10.0.0.0/8` "Corp Private" has exactly three children: `10.12.0.0/16` **Cloud**, `10.16.0.0/16`
Datacenter (Willamette dev), `10.17.0.0/16` Datacenter (Roosevelt dev -- the physical lab net that
vcloud and the VR0 jumphost actually sit on). Plus `172.16.0.0/12` "Corp OOB Private".

### The gaps (all four are real; none was known before today)

1. **There is NO Office role in v4.** Office1's LAN `10.10.0.0/24` (chosen during the OPNsense edge
   build, 2026-07-12) and its LXD compose network `10.10.1.0/24` (chosen 2026-07-13) are SQUAT --
   unallocated, unregistered, colliding with nothing but blessed by nothing.
2. **DC2's planned `10.13.0.0/19` is OUTSIDE the Cloud `/16`.** `docs/dc-dc-netbox-buildout-scope.md`
   recommends it, but NetBox's Cloud role is `10.12.0.0/16`; `10.13.x` is unallocated. DC2 was
   already set to squat, and nobody had noticed.
3. **`office1-wan` (`172.30.1.0/24`) squats INSIDE `172.16.0.0/12` "Corp OOB Private"** -- and it is
   not OOB, it is a SIMULATED ISP UPLINK. That is a wrong-role problem, not merely an unregistered one.
4. **"Edge Networks" has a v6 `/48` (`2602:f3e2:fe::/48`) but NO v4 counterpart**, so the simulated
   ISP/WAN segments have nowhere legitimate to live in v4 at all.

(Also, still open and pre-existing: NetBox holds ZERO aggregates -- gap G2 -- so nothing declares
these allocations at the RIR level. And `2602:f3e2:f01:100::/56` needs a `VR1 Off1` SITE, which
does not exist in NetBox either.)

### The design tension the operator must rule on

**v6 here is REGION-hierarchical; v4 is ROLE-based.** That asymmetry is defensible and, I think,
correct: v6 is abundant, so it can afford geography; v4 is scarce, so it is pooled by function.
The alternative -- forcing v4 to mirror the v6 region tree -- burns scarce v4 AND requires
RENUMBERING DC1 off `10.12.x`, which is exactly the inheritance D-101 bought in order to reuse
VR0 DC0's bundle unchanged (minimize delta to Roosevelt). That is a large cost for conceptual tidiness.

### Options

**(a) RECOMMENDED -- keep v4 role-based; fill in the MISSING roles. Ratify the deployed addresses.**
- New v4 role **Office**: `10.10.0.0/16`, carved **`/22` per office**.
  VR1 Office1 = **`10.10.0.0/22`** -> LAN `10.10.0.0/24` (as deployed) + LXD compose `10.10.1.0/24`
  (as deployed) + `.2`/`.3` spare. **This BLESSES exactly what is already running -- zero renumber.**
- New v4 role **Edge** (mirroring v6's `2602:f3e2:fe::/48`): `172.30.0.0/16` for simulated ISP/WAN
  segments, carved out of the `172.16.0.0/12` container but with its OWN role, not OOB.
  Office1's WAN `172.30.1.0/24` (as deployed) becomes legitimate; DC1/DC2 edges take `.2`/`.3`.
- **DC2 moves INSIDE the Cloud `/16`**: `10.12.64.0/19` (free -- DC1's six `/22`s occupy
  `10.12.4/8/12/16/32/36`, so `10.12.64.0/19` collides with nothing and DC1<->DC2 stay routable
  over the dark fiber). SUPERSEDES the scope doc's `10.13.0.0/19` recommendation.
- IPv6 per the pattern: Office1 = `2602:f3e2:f01:100::/56` + `/64` subnet; create the `VR1 Off1` site.
- COST: two NetBox roles + a handful of prefixes. **No deployment renumber at all.**

**(b) Mirror the v6 region model in v4.** Give each virtual region a v4 `/16` (e.g. VR1 = `10.15.0.0/16`)
and carve sites inside it (infra / office / DC1 / DC2). Conceptually cleanest; the two families would
read identically. COST: RENUMBERS DC1's six planes off `10.12.x`, which BREAKS D-101's
inherit-DC0-unchanged economy and forces edits to `lib-net.sh`, MAAS subnets, and the bundle's VIPs.
It also spends a whole `/16` per virtual region on a test environment. Recommended ONLY if the
operator wants v4 and v6 structurally identical and accepts the DC renumber.

**(c) Ratify the squat as-is and register it verbatim.** Record `10.10.0.0/24`, `10.10.1.0/24`,
`172.30.1.0/24`, `10.13.0.0/19` in NetBox exactly as deployed, with no new roles and no structure.
Cheapest today, and WRONG: it enshrines an ISP-sim network inside the corp OOB block, leaves offices
with no allocatable block for the next office, and leaves DC2 outside the Cloud role. Rejected
unless the operator wants pure speed.

### What is NOT at stake

Nothing is blocked and nothing is broken. Office1 is live and routing on the squat addresses. The
question is only what to write into the IPAM apex -- and the renumber cost is LOWEST RIGHT NOW,
before NetBox and GitBucket are deployed onto that network. Under option (a) the renumber cost is
ZERO, because (a) is shaped to bless what is already there.

**References:** measured 2026-07-13 against `netbox.baldurkeep.com` (read-only; SEC-006 token).
D-101 (family matrix + DC1-inherits-DC0), D-111 (NN mnemonic subcarve), D-016 (tenant pool
`10.20.0.0/16` -- the reason the obvious `10.20.x` was NOT used for the compose net),
`docs/dc-dc-netbox-buildout-scope.md` (whose `DC2_V4_SUPERNET = 10.13.0.0/19` recommendation this
decision would supersede under option (a)), D-114 (which built the headend that exposed all of this).

### D-115 -- AMENDMENT (2026-07-13): ADOPTED by operator ruling, with a measured constraint attached

**Operator ruled all three questions:**
1. **v4 stays ROLE-BASED; fill in the missing roles.** Option (a). v6 keeps its region hierarchy;
   v4 is pooled by function. No DC renumber; D-101's inherit-DC0-unchanged economy is preserved.
2. **`/22` per office.** VR1 Office1 = **`10.10.0.0/22`** -- which BLESSES the deployed LAN
   (`10.10.0.0/24`) and LXD compose net (`10.10.1.0/24`) exactly. Zero deployment renumber.
3. **Register AND deploy dual-stack on Office1 NOW** -- Office1 becomes the first real dual-stack
   proof and de-risks D-101's family matrix early.

**THE ADOPTED CARVE**

v4 (new roles marked):
```
10.0.0.0/8   Corp Private
 |- 10.10.0.0/16   Office        <-- NEW ROLE
 |    |- 10.10.0.0/22    VR1 Office1
 |         |- 10.10.0.0/24   office1-local (site LAN; Kea on the edge)   [as deployed]
 |         |- 10.10.1.0/24   lxdbr0 compose net (MAAS DHCP/PXE)          [as deployed]
 |         |- 10.10.2.0/24, 10.10.3.0/24  spare
 |- 10.12.0.0/16   Cloud
 |    |- 10.12.{4,8,12,16,32,36}.0/22   DC1 planes    [unchanged -- D-101 inheritance]
 |    |- 10.12.64.0/19                  DC2           <-- MOVED INSIDE the Cloud /16
 |- 10.16.0.0/16   Datacenter (Willamette dev)
 |- 10.17.0.0/16   Datacenter (Roosevelt dev -- the physical lab net)
172.16.0.0/12  Corp
 |- 172.30.0.0/16  Edge (simulated ISP uplinks)       <-- NEW ROLE, mirrors v6's fe::/48
      |- 172.30.1.0/24   office1-wan                   [as deployed]
```
**This SUPERSEDES `docs/dc-dc-netbox-buildout-scope.md`'s `DC2_V4_SUPERNET = 10.13.0.0/19`**, which
sat outside every allocated block. DC2 becomes `10.12.64.0/19` -- free (DC1 occupies
`10.12.4/8/12/16/32/36`), inside the Cloud role, and still routable to DC1 over the dark fiber.

v6 (determined by the existing pattern -- no design freedom):
```
2602:f3e2:f01::/48        VR1 Offices                        [exists]
 |- 2602:f3e2:f01:100::/56   VR1 Office1                     <-- CREATE (+ the "VR1 Off1" site)
      |- 2602:f3e2:f01:100::/64   Office1 office subnet      <-- CREATE
```

### MEASURED CONSTRAINT (2026-07-13) -- IPv6 DOES NOT EGRESS THE LAB

Discovered while planning ruling 3. On vcloud:
- Global v6 addresses present (`2602:f3e2:e00:100::/64` SLAAC + a `/128`), and a v6 default route
  via RA.
- The v6 default gateway (link-local `fe80::e68d:8cff:fe14:7cd5`) **IS reachable** -- RA and L2 work.
- **But v6 INTERNET is UNREACHABLE**: both `2606:4700:4700::1111` and `2001:4860:4860::8888` fail,
  100% loss. v4 egress from the same host is fine.

**Consequence, stated plainly so nobody builds on a false premise:** deploying dual-stack on Office1
proves v6 ADDRESSING, RA, and services binding v6 -- which is most of D-101's family matrix and is
worth doing. It does **NOT** prove v6 INTERNET reachability, and it cannot: the GUA is not routed
past the lab boundary. **Public-facing GUA reachability (public API endpoints on v6, tenant GUA
egress) is UNPROVABLE in VR1 until the lab's upstream actually carries v6.** That is a network
issue OUTSIDE this cloud and outside this repo -- it needs the lab/upstream operator, not a change
here. Do not schedule a VR1 test that depends on it, and do not let a green internal-v6 result be
read as "v6 works end to end".

`office1-wan` currently has **no v6 at all** (v4 NAT only), and `office1-local` is an addressless
isolated bridge (correct -- the OPNsense edge owns addressing on it).

### NetBox write path -- NOT done unilaterally

The carve above is recorded HERE (the repo) as the decision of record. Writing it into
`netbox.baldurkeep.com` is a MUTATION ON A PRODUCTION SYSTEM and is gated on explicit operator
approval; the standing architecture is read=source / write=feedback-only, with the Office1 sandbox
NetBox as the place changes are simulated first. The Office1 NetBox VM is composed and `Ready` but
not yet deployed -- so the natural sequence is: deploy it, import this carve there, verify, then
feed back to the real NetBox.

## D-116: no Office1-local GitBucket -- keep using the existing git.baldurkeep.com

**Status:** **ADOPTED 2026-07-13 (operator ruling).** SUPERSEDES the GitBucket half of D-103 and
removes GitBucket from Stage 2's build list.

**Operator ruling, verbatim in substance:** "For the gitbucket, we've decided not to build a
separate one on the test cloud. We will continue to use the current setup and create new repos for
deployment tasks as needed."

**What changes.** D-103 assigned Office1 three service VMs -- MAAS-region, NetBox, GitBucket -- and
D-114 recast them as MAAS-composed LXD VMs. **GitBucket is now OUT OF SCOPE entirely**: no third
service machine is composed, and the existing `git.baldurkeep.com` remains the git service of record
for VR1. New repos are created there as deployment tasks require.

**What this closes.** The Stage 2 runbook carried an explicit OPEN QUESTION about the Office1-local
GitBucket -- "what it mirrors, and under what repo path, is not decided" -- because a *second*,
Office1-local GitBucket alongside the pre-existing `git.baldurkeep.com` raised a real
mirror/authority question nobody had answered. **That question is now MOOT.** There is one git
service, it is the existing one, and it is authoritative. This is a genuine simplification, not a
deferral.

**Rationale.** The Office1-local GitBucket only ever existed to rehearse a per-site git mirror. The
operator judges the rehearsal not worth a machine: the real deployment work is served by the
existing instance, and a second one buys a mirroring problem (which repo is authoritative, what
syncs, in which direction) in exchange for no capability the deployment actually needs.

**Delta to Roosevelt.** Roosevelt may still want a per-site git mirror for node-airgap reasons
(cf. D-107's per-DC artifact mirror, which is a SEPARATE concern and is NOT affected by this
ruling -- D-107 is about OS/package artifacts for node provisioning, not about hosting git repos).
If Roosevelt later needs a site-local git service, it is a fresh decision made against real
requirements, not a rehearsal artifact carried forward. Recording that explicitly so nobody reads
this ruling as "Roosevelt will never have a git mirror".

**Consequences (all applied in the same change):** GitBucket is removed from Stage 2's Build/Gate in
`docs/dc-dc-deployment-workflow.md`, from the Office1 standup runbook's sequence and its open
questions, and from `docs/vr1-office1-as-built.md`. Office1's composed service machines are
therefore exactly TWO: `office1-netbox` and `office1-tailscale`, both LIVE.

---

## D-117: VR1 site naming -- the apex says DC0/DC1, the repo says DC1/DC2 (G5)

**Status:** **ADOPTED 2026-07-13 (operator ruling: Option B, DC-only). FULLY EXECUTED BY D-119
(2026-07-14) -- the "PARTIALLY EXECUTED" caveat below is DISCHARGED; read D-119 for the end state.**
The REPO moves to the apex's 0-indexed convention **for the DATACENTERS**. NetBox is the IPAM apex
and is NOT bent to fit the repo. Supersedes the `docs/dc-dc-netbox-buildout-scope.md` G5
recommendation (which proposed the opposite, Option A) and the D-106 `dc1`/`dc2` zone labels.
Closes G5.

> **D-119 completed this decision, it did not reverse it.** D-117 renamed only the NetBox IMPORT
> PATH, which left `dc0` meaning VR1's first DC in the importer and **VR0's live cloud** in
> `lib-net.sh` -- one string, two clouds. D-119 executes this decision's own amendment
> (region-qualify the selectors: `vr0-dc0` / `vr1-dc0` / `vr1-dc1`) across `lib-net.sh`,
> `lib-hosts.sh` and `opentofu/`, making the importer's DC->site mapping an IDENTITY. **The
> alternative -- renaming the apex to match the repo's `dc1`/`dc2` -- was considered and REJECTED;
> see D-119.** The apex is NOT renamed and no production IPAM write is involved.

**Operator refinement (verbatim):** *"go ahead and rename the DC numbers. The office number is
fine."* -- so the **OFFICE KEEPS ITS NUMBER**: it stays **Office1**, and its NetBox site is
**`vr1-off1` / "VR1 Off1"**. That site does NOT exist upstream yet, so it is CREATED, not renamed --
no collision (VR0's `vr0-off0` is untouched), and **nothing deployed changes**.

**This is the G5 gap** (`docs/dc-dc-netbox-buildout-scope.md`), promoted to a decision because it
blocks the Office1 NetBox import and because the tool that trips over it WRITES BY DEFAULT.

**The collision, measured live against `netbox.baldurkeep.com` (4.5.8) on 2026-07-13:**

| v6 site `/48` | Prod NetBox site (the apex) | `dc-dc-prefixes-import.py` |
|---|---|---|
| `2602:f3e2:f02::/48` | **VR1 DC0** (`vr1-dc0`, id 43) | `--dc dc1` (its docstring: "`f02::/48` for dc1") |
| `2602:f3e2:f03::/48` | **VR1 DC1** (`vr1-dc1`, id 44) | `--dc dc2` |

The **addressing agrees**; only the **labels are off by one**. The repo's "DC1" and the apex's "DC0"
are the SAME datacenter. Same off-by-one on the office: the repo's "Office1" carve
(`2602:f3e2:f01:100::/56`, D-115) exactly mirrors VR0's **Off0** (`e01:100::/56`).

**Why this is dangerous, not merely cosmetic.** `dc-dc-prefixes-import.py` maps `dc1 -> slug
vr1-dc1` and **has no `--commit`** -- `--verify-only` is its read-only mode, so it WRITES BY
DEFAULT. Run it as documented against a NetBox seeded from prod and it binds DC1's `f02::/48`
prefixes to the site `vr1-dc1`, which upstream holds `f03::/48` -- i.e. **the other DC**, silently.
(`roles-aggregates-import.py` is the opposite: dry by default. Do not assume the tools are uniform.)

**How it happened, recorded because the mistake is reusable.** The script's own comment: *"Slugs are
new -- no prior convention existed in this repo (grepped `opentofu/*.tf` and the buildout design;
none found)."* It grepped the **repo**. The convention lived in the **apex**. NetBox is declared the
IPAM apex (D-010/D-103) precisely so that naming is NOT a repo-local invention -- but the check for
prior art was run against the repo. Same class of error as concluding "no VR0 precedent for
MAAS-composed LXD VMs" after grepping only `bundle.yaml` (2026-07-13, same session).

**What the site byte does and does NOT mean.** Within a region `/40`: `00`=Infrastructure,
`01`=Offices, `02`/`03`=the DCs. That is **sequential allocation, not a DC index** -- there is no
`02 -> DC0` mnemonic to preserve. An earlier draft of this entry claimed there was; that was wrong
and is corrected here, because it materially weakens the case against Option A.

**Blast radius, measured.** `vr1-dc0` and `vr1-dc1` are **skeletal**: 1 prefix each (their own `/48`
container), and **0** devices / racks / VLANs / clusters. (Contrast `vr0-dc0`: 18 prefixes.) Renaming
them upstream touches 2 site records + their 2 prefix descriptions. Renaming the REPO instead touches
the live surfaces (`lib-net.sh`, the `--dc` flag, `opentofu/`, runbooks, SKILL.md, and the text of
D-101/D-106/D-111/D-115) and contradicts **already-ratified D-106** (Designate `dc1`/`dc2` zones).

### Options

**A. Rename the two skeletal apex sites to `vr1-dc1`/`vr1-dc2`; add `vr1-off1`. (RECOMMENDED)**
The scope doc's standing recommendation. Cheapest by a wide margin, and it preserves the `dc1`/`dc2`
language already ADOPTED in D-106 and wired through every live repo surface. Cost: VR1 reads DC1/DC2
while VR0 reads DC0/DC1 -- a cosmetic cross-region inconsistency, and VR1's office becomes `Off1`
against VR0's `Off0`.
**TRAP -- rename ORDER matters:** `vr1-dc0 -> vr1-dc1` while `vr1-dc1` still exists is a **slug
collision**. Rename **`vr1-dc1` -> `vr1-dc2` FIRST**, then `vr1-dc0` -> `vr1-dc1`. Update the two
`/48` descriptions ("Datacenter 0 (DC0)") in the same change or they contradict their own site.

**B. Rename the repo to the apex's 0-indexed `DC0`/`DC1` + `Off0`.**
Uniform 0-indexing across VR0 and VR1, and the apex is never bent to fit the repo. Cost: reopens
ratified D-106, and rewrites every live repo surface. Historical changelogs are NOT rewritten.

**C. Re-point the tool only (`SITES = {dc1: vr1-dc0, dc2: vr1-dc1}`).**
Two lines, nothing else moves. **Argued against:** it makes `--dc dc1` write to a site *displayed* as
"VR1 DC0" forever -- a permanent hidden indirection, which is how this class of bug got here.

**Whichever is chosen, one fix is unconditional:** give `dc-dc-prefixes-import.py` a `--commit` flag
so it is DRY BY DEFAULT, matching `roles-aggregates-import.py`. A write-by-default IPAM tool is a
standing hazard regardless of how the naming lands.

### D-117 -- AMENDMENT (2026-07-13): ADOPTED (Option B), scope narrowed by operator ruling

**Ruled:** VR1's datacenters are **DC0** (`vr1-dc0`, `2602:f3e2:f02::/48`) and **DC1** (`vr1-dc1`,
`2602:f3e2:f03::/48`) -- matching the apex, which already holds both sites. The repo's `dc1`/`dc2`
labels are retired in favour of **`dc0`/`dc1`**.

VR1's office **remains Office1** -> NetBox site **`vr1-off1` ("VR1 Off1")**, carrying the D-115 carve
(`2602:f3e2:f01:100::/56` + `/64`, `10.10.0.0/22`). Note VR1 therefore reads **DC0/DC1/Off1** while
VR0 reads DC0/DC1/Off0 -- an accepted cosmetic asymmetry. The office site is NEW upstream, so it is
created cleanly and **no deployed object is renamed**.

**SCOPE -- what B does and does NOT touch.** B is a ruling about **IPAM naming authority**: the
NetBox site record and the selector that resolves to it. It does NOT extend to deployment substrate.

- **IN scope (renamed, DC ONLY):** the DC half of the NetBox site identity; the importer's `SITES` +
  GUA map; the `--dc` selector (`dc1|dc2` -> `dc0|dc1`); `lib-net.sh`/`lib-hosts.sh` DC keys;
  OpenTofu DC module labels; forward-looking DC prose.
- **OUT of scope -- NOTHING office-side is renamed.** Per the operator refinement above, `Office1`
  keeps its number, so `voffice1`, `office1-opnsense`, the `office1-local`/`office1-wan` networks,
  `office1-pool`, the MAAS VM host `office1-lxd`, the machines `office1-netbox` /
  `office1-tailscale`, the OPNsense `OFFICE1_LXD_GW` object and the Tailscale node all stand as-is.
  **No teardown, no rebuild.**
- **Also out of scope, independently:** deployment substrate is not a carve. Even where a name DID
  change (the DC libvirt objects), the rename is a label on empty, unused infrastructure -- see the
  table below.

**TRAP 1 -- this is a SEMANTIC REMAP, not a string replace.** `dc1 -> dc0` AND `dc2 -> dc1`, so
`dc1` denotes different things before and after. A sequential `sed` double-applies and corrupts the
tree. Any mechanical pass must go through a placeholder (`dc1 -> __DCA__`, `dc2 -> __DCB__`, then
`__DCA__ -> dc0`, `__DCB__ -> dc1`).

**TRAP 2 -- and this one NO mechanical pass can catch.** Existing prose uses "DC0" to mean **VR0's
rehearsal DC** while using "DC1"/"DC2" to mean **VR1's** DCs. The importer's docstring: *"DC1
INHERITS the DC0 six-plane CIDRs unchanged (D-101 -- 'DC1 equals the DC0 template') ... not the
existing `vr0-dc0` site."* Remap VR1's `dc1 -> dc0` and that reads *"DC0 equals the DC0 template"* --
two DIFFERENT DC0s (`vr0-dc0` and `vr1-dc0`) that were previously disambiguated by NUMBER ALONE.
**Every "DCn" in prose must be region-qualified by hand** (`VR0 DC0` vs `VR1 DC0`). This is why the
decision text below is ANNOTATED, not rewritten.

**Treatment by surface kind (adopted with the ruling):**

| Surface | Treatment |
|---|---|
| Operational/structured (importer, `lib-net.sh`, `--dc`, tofu labels) | Mechanical rename. **This tier ALONE unblocks the NetBox import.** |
| **ADOPTED decision text** (D-101, D-106, D-111, D-115) | **ANNOTATE in place, do NOT rewrite** -- the D-103->D-116 supersession precedent. Ratified prose is a historical record. |
| Forward-looking prose (runbooks, design docs) | Hand-edit, region-qualifying every "DCn" (Trap 2). |
| DC libvirt objects (`dc1-*`/`dc2-*` nets, `dc1-pool`/`dc2-pool`) | **Empty and unused** (0 volumes, no DC VMs). Rename the module labels; let the next **GATED** DC apply reconcile them. **No surprise destroy/recreate.** |
| Historical changelogs | **Never rewritten.** |

**Unconditional companion fix (also ruled):** `netbox/dc-dc-prefixes-import.py` becomes **DRY BY
DEFAULT** with an explicit `--commit`, matching `roles-aggregates-import.py`. A write-by-default tool
pointed at the IPAM apex is a standing hazard, and it is the mechanism by which this naming bug would
have landed silently.

### D-117 -- AMENDMENT (2026-07-13): the `$DC` shell selector is a SECOND, COLLIDING namespace

Found while executing the rename. **`scripts/lib-net.sh` / `lib-hosts.sh` use a THREE-value `$DC`
selector in which `dc0` ALREADY MEANS VR0's DC0** (the deployed rehearsal), with `dc1`/`dc2` meaning
VR1's two DCs:

```
dc0|dc1)   # lib-net.sh: VR0 DC0 and VR1's first DC share v4 literals (D-101 inheritance)
dc2)       # VR1's second DC -- FAILS LOUD, no literals assigned yet
```

So the repo has been carrying **two different `dcN` namespaces at once**: the shell libs' (where
`dc0` = VR0) and the NetBox importer's (where `dc0` did not exist and `dc1` = VR1's first DC). That
is Trap 2 in its purest form, and it is why a mechanical `sed` across this rename would have been
**actively dangerous**: renaming VR1's `dc1 -> dc0` in `lib-net.sh` collides HEAD-ON with VR0's
existing `dc0` and would silently hand VR1's selector VR0's deployed literals.

**Proposed fix (NOT yet executed -- does not block the NetBox import):** region-qualify the shell
selector to **the apex's own slugs** -- `vr0-dc0`, `vr1-dc0`, `vr1-dc1` -- so there is ONE namespace
across NetBox, the shell libs, and the importer, and every identifier is unambiguous by construction.
This is the same principle D-117 already adopts: the apex names things; the repo follows.

**Blast radius:** `lib-net.sh`, `lib-hosts.sh`, and the scripts that take `$DC`
(`carve-host-interfaces.sh`, `dc-dc-ceph-disk-budget.sh`, `dc-dc-dr-drill.sh`,
`dc-dc-radosgw-multisite.sh`, `dc-dc-rbd-mirror.sh`, `phase-00-maas-standup.sh`,
`reenroll-hosts.sh`) plus their harnesses and `tests/dc-selector/`.

**Until it is executed, the shell `$DC` selector retains its OLD meaning** (`dc0`=VR0 DC0,
`dc1`/`dc2`=VR1) while `netbox/dc-dc-prefixes-import.py --dc` uses the NEW, apex-aligned meaning
(`dc0`/`dc1` = VR1's two DCs). **These two are NOT interchangeable.** Do not copy a `$DC` value from
a shell script into `--dc`, or vice versa, without re-reading this entry.

---

## D-118: the org ULA /48 -- ADOPT `fd50:840e:74e2::/48`

**Status:** **ADOPTED 2026-07-13 (operator ruling).** Closes gap **G3**
(`docs/dc-dc-netbox-buildout-scope.md`) and the D-101 open literal `ORG_ULA_48`. **UNBLOCKS the VR1
DC six-plane NetBox import**, which could not run without it.

**Ruled:** `ORG_ULA_48 = **fd50:840e:74e2::/48**`.

D-101 rules the org ULA `/48` a NetBox-assigned literal, "NOT hardcoded in this decision", and every
tool takes it as a REQUIRED env var with no default. Until now no decision assigned it, so
`netbox/dc-dc-prefixes-import.py` was unrunnable -- supplying a value would have been an inferred
value (hard rule 2).

**Validation (measured 2026-07-13, not asserted):**

| Property | Result |
|---|---|
| Inside `fc00::/7` | yes |
| First byte `0xfd` -> **L=1**, locally assigned (RFC 4193) | yes |
| 40-bit Global ID | `0x50840e74e2` -- no discernible structure |
| Prefix length | `/48` -> **256 x `/56`**, and D-101 allocates one `/56` per DC |
| Collides with Tailscale's ULA `fd7a:115c:a1e0::/48` | **NO** |
| Collides with anything else in the apex | no -- the apex holds only the `fc00::/8` / `fd00::/8` containers and Tailscale's `/48` |

**The Tailscale non-collision is the load-bearing check, not a formality.** `office1-tailscale` runs
with `--accept-routes` on the self-hosted control plane, so an org ULA overlapping
`fd7a:115c:a1e0::/48` would be a LIVE routing conflict on the tailnet, not a paper one. `fd50` vs
`fd7a` -- distinct.

**Provenance, stated honestly.** `docs/dc-dc-netbox-buildout-scope.md` 4c describes this value as
CSPRNG-generated. That claim originates in an earlier session and **cannot be re-verified now**. It
does not change the ruling: RFC 4193's "SHA1 of time + MAC" construction is a **SHOULD**, not a MUST
-- the actual requirement is a pseudo-random 40-bit Global ID, which this satisfies. Regenerating
would not make the value MORE correct; it would only substitute a different random number, at the
cost of re-editing the 7 places that already carry this one and risking a stale literal surviving in
one of them. Not worth it.

**Corrects a FALSE claim in the repo.** `docs/archive/changelogs/changelog-20260711-ula-gen-command-fix.md` called this
"the **ratified** VR1 value" -- it was NOT ratified; no D-number ever assigned it. That phantom
ratification is very likely why the value has been treated as settled for two days while
`grep ORG_ULA docs/design-decisions.md` returned nothing. This entry makes the claim true, and the
changelog is annotated. **Same failure mode as D-117:** a value treated as authoritative because a
document asserted it, with no decision behind it.

**Consequences.** The DC import can now run:
```
ORG_ULA_48=fd50:840e:74e2::/48  DC_GUA_PREFIX=2602:f3e2:f02::/48  --dc dc0   # VR1 DC0
ORG_ULA_48=fd50:840e:74e2::/48  DC_GUA_PREFIX=2602:f3e2:f03::/48 \
    DC1_V4_SUPERNET=10.12.64.0/19  --dc dc1                                   # VR1 DC1
```
(`DC_GUA_PREFIX` values are MEASURED off the apex per D-117; `DC1_V4_SUPERNET` is assigned by D-115.)
Per D-101/D-111 each DC takes a `/56` of this `/48`, indexed to its GUA site nibble, so the 4th
hextet reads `DC.NN` in BOTH families.

## D-119: the VR1 DC namespace -- region-qualify the selectors; the repo adopts the apex

**Status:** **ADOPTED 2026-07-14 (operator ruling).** COMPLETES D-117 rather than reversing it:
it executes D-117's own amendment (region-qualify the shell selectors) across the three surfaces
D-117 never touched. D-117 stays ADOPTED; its "partially executed" caveat is discharged here.
Closes tooling-gap #19 and Stage 2 close-out item C3.

### The problem, stated exactly

After D-117, the token `dc0` meant TWO DIFFERENT CLOUDS depending on which file you were reading:

| Surface | `dc0` meant | `dc1` meant |
|---|---|---|
| NetBox apex (the IPAM authority) | VR1's FIRST DC | VR1's SECOND DC |
| `scripts/lib-net.sh` / `lib-hosts.sh` | **VR0's DC0 -- a DIFFERENT, LIVE cloud** | VR1's first DC |
| `opentofu/main.tf` | (no `dc0`) | VR1's first DC |

One string, two clouds, one of them in production. A mechanical renumber makes it WORSE, not
better (D-117 TRAP 1): `sed s/dc1/dc0/` over `opentofu/` would hand VR1's first DC the string
`dc0`, which `lib-net.sh` already resolves to VR0's deployed literals.

### The ruling

**Region-qualify the DC selector everywhere. The repo adopts the apex's names verbatim.**

    vr0-dc0   VR0's DC0        the LIVE testcloud (a different region)
    vr1-dc0   VR1's FIRST  DC  GUA 2602:f3e2:f02::/48
    vr1-dc1   VR1's SECOND DC  GUA 2602:f3e2:f03::/48

Bare `dc0` / `dc1` / `dc2` are REJECTED loudly by the selectors. They are the ambiguity being
deleted; accepting them "for compatibility" would preserve the bug.

**The apex is NOT renamed. There is NO production IPAM write in this decision.** The apex was
already self-consistent and 0-indexed; MEASURED 2026-07-14 against `netbox.baldurkeep.com`:

    2602:f3e2:f02::/48  -> site vr1-dc0   "Virtual Region 1 (VR1) Datacenter 0 (DC0)"
    2602:f3e2:f03::/48  -> site vr1-dc1   "Virtual Region 1 (VR1) Datacenter 1 (DC1)"
    (VR0, for contrast: vr0-dc0, vr0-dc1, vr0-off0 -- the same 0-indexed convention)

The repo was the only surface out of step, because -- operator, verbatim in substance -- *"we
didn't catch the dc0 missing and went straight to one."*

### Why this beats the alternative (renaming NetBox to match the repo)

Considered and REJECTED: rename the apex to `vr1-dc1`/`vr1-dc2` so the ordinals match the repo's
`dc1`/`dc2`. It was rejected because it would:
- require a PRODUCTION IPAM write on the authority, ordered highest-slug-first or it violates
  NetBox's unique-slug constraint;
- make VR1 the ONLY 1-indexed region, diverging from VR0 -- the very region VR1 rehearses; and
- create a worse ambiguity: `vr1-dc1` would mean VR1's FIRST DC while `vr0-dc1` still means VR0's
  SECOND DC. Same suffix, different ordinal, one NetBox.

(For the record, the apex is NOT uniformly `dcN`: the Eugene/Roosevelt sites are place-named --
Charnelton, Cleveland, Roosevelt, Stevenson, Willamette. The `dcN` convention belongs to the
REHEARSAL regions VR0/VR1, and within those it is 0-indexed. The consistency claim is scoped to
them, not to the whole apex.)

### The prize: the off-by-one becomes structurally impossible

The importer's DC->site mapping becomes an IDENTITY (`vr1-dc0` -> slug `vr1-dc0`). **There is no
offset table left to get wrong.** This matters because the original defect was precisely a WRONG
LOOKUP TABLE -- `--dc dc1` bound to slug `vr1-dc1` while treating `f02::/48` as dc1's, though the
apex binds `f02` to VR1 DC0. It would have written the FIRST DC's prefixes onto the SECOND DC's
site, silently. Deleting the mapping deletes the bug class.

### Scope -- what this decision does NOT change

- **The office KEEPS its number** (operator ruling): NetBox site `vr1-off1`, and `Office1` /
  `voffice1` / `office1-local` / `office1-netbox` / `~/vr1-office1-creds` throughout. The site is
  BUILT AND RUNNING; renaming it churns hostnames, creds paths and libvirt networks for zero
  functional gain. This leaves a known, ACCEPTED asymmetry with VR0's `vr0-off0`. Documented, not
  fixed.
- **VR0 is not renamed.** Its DC stays DC0; only the SELECTOR STRING is region-qualified
  (`dc0` -> `vr0-dc0`). No VR0 literal, host, or deployed object changes.
- **The ULA hextet still mirrors the GUA site nibble, NOT the DC number** (D-111): `vr1-dc0` ->
  `fd50:840e:74e2:2NN::`, `vr1-dc1` -> `:3NN::`, tracking `f02`/`f03`. This is DELIBERATE so the
  4th hextet reads the same in both families. Do not "fix" it to match the DC index -- that would
  break the already-imported sandbox prefixes.
- **NetBox data is untouched.** No re-import, no re-dump of `netbox/draft/vr1-draft.json`.

### Blast radius (MEASURED 2026-07-14, before any edit)

`opentofu/` renames change the libvirt objects' `name` attribute, so `tofu apply` will
DESTROY AND RECREATE them. Measured: the 6 `dc1_planes` networks, both DC storage pools, and the
3 mesh links are live in state but **EMPTY** -- no guests attached (Stage 3 has not run) and no
volumes in the DC pools (the only volumes belong to `office1_opnsense`, `ubuntu_noble_base`, and
`voffice1`, none of which are in a DC pool). The recreate is therefore safe, but it IS a
destroying apply and is gated: the plan is presented before it runs.

### Standing rule this leaves behind

**Never write a bare `dcN` for a VR1 datacenter again, in code or in prose.** Region-qualify it.
The one durable lesson of D-117 -- which survives its completion -- is that an unqualified `dcN`
in a repo that operates more than one cloud is a latent off-by-one waiting for a reader who does
not know which cloud the sentence is about.

## D-120: static-services addressing convention for the MAAS-managed compose /24s (ADOPTED)

**Status:** **ADOPTED 2026-07-15 -- option (a), EXECUTED.** Operator endorsed the direction (static
services belong in a low band, not mixed with deployed nodes) and ruled "it doesn't matter where the
IPs land in the sub-50 scope; go ahead with changing the IPs." The band layout below and the Office1
service addresses (`.10`/`.11`) are ratified, and the Office1 re-IP was executed the same day (see
"Execution" and `runbooks/dc-dc-office1-service-reip.md`). Refines D-115 (which ratified that
`10.10.1.0/24` exists as the Office1 compose net but left its INTERNAL layout as just "MAAS
DHCP/PXE"). Does NOT reopen D-115's /22 carve.

### The problem

On `10.10.1.0/24` (Office1 `lxdbr0`, MAAS-managed), the persistent site services sit at
`office1-netbox = 10.10.1.201` and `office1-tailscale = 10.10.1.202` -- addressed IDENTICALLY to how
a deployed OpenStack node would be. MAAS auto-assigns deployed machines from `.201+` (just past its
`.100-.200` dynamic range), so the services landed there by default, not by design. They are NOT in
the DHCP dynamic range (no collision), but they are not distinguishable from ephemeral compute nodes
either. There is no ratified sub-layout for the /24.

### Decision (PROPOSED) -- a functional band layout for EVERY MAAS-managed compose /24

Applies to Office1's `10.10.1.0/24` AND to each DC's compose /24 when the DCs are built (they inherit
the template -- minimize-delta):

    .1          gateway (the site containment VM: voffice1, and each DC's equivalent)
    .2 - .49    STATIC site infrastructure/services (MAAS static-assign; OUTSIDE the dynamic range)
    .50 - .99   reserved / spare (headroom for more static services)
    .100 - .200 MAAS DYNAMIC (enlistment / commissioning / PXE / DHCP)   [as already deployed]
    .201 - .254 deployed compute / OpenStack nodes (MAAS auto-assign)    [as already deployed]

The dynamic and node bands are UNCHANGED from what MAAS already does -- only the static-services band
`.2-.49` is newly reserved, and it already sits outside the dynamic range, so no MAAS range edit is
needed, only per-service static IP assignment.

### Office1 service addresses (PROPOSED; operator to ratify)

    office1-netbox     10.10.1.201 -> 10.10.1.10
    office1-tailscale  10.10.1.202 -> 10.10.1.11

`.10`/`.11` are arbitrary within `.2-.49`; chosen low and adjacent. Operator may pick otherwise.

### Recording it in NetBox (the IPAM authority deciding where NetBox itself lives)

The band layout is a carve that belongs IN NetBox: register `10.10.1.0/24`'s child ranges (static /
dynamic / node) and the two service IP assignments. Do this through the sandbox loop (D-115 / the
C4-documented loop), not by hand -- NetBox is the authority for exactly this.

### Consequence / cost (why this is not a free change)

Re-IPing Office1's two RUNNING services cascaded to the repo references, INCLUDING the hardcoded
`SANDBOX_HOSTS` allowlist (`{localhost, 127.0.0.1, 10.10.1.201}` -> `.10`) in
`roles-aggregates-import.py`, `dc-dc-prefixes-import.py`, and `d115-office-carve.py` (added
2026-07-14) plus its test. The sweep + gauntlet ran with the re-IP (see "Execution" above and
`runbooks/dc-dc-office1-service-reip.md`). Dated changelogs keep `.201`/`.202` as historical
snapshots.

### Options

- **(a) ADOPT the convention AND re-IP Office1 now** (recommended) -- Office1 is the proven template;
  making it match means the DCs inherit a clean, already-exercised layout. Cost: a gated re-IP of two
  running services + the 20-reference sweep.
- **(b) ADOPT the convention for the DCs ONLY; leave Office1 at `.201/.202`** -- zero disruption now,
  but Office1 (the template everyone copies) permanently diverges from the convention, and the
  `SANDBOX_HOSTS` allowlist keeps a "deployed-node-band" address for a static service.
- **(c) Reject** -- keep services in the node band. Not recommended: it is the ambiguity this entry
  exists to remove.

**Recommendation: (a).** Get the template right once.

### Execution (2026-07-15) -- option (a), done

Re-IP executed on the two running services, no wipe, verified on the wire:

    office1-netbox     10.10.1.201 -> 10.10.1.10   NetBox HTTP 200 at .10, reachable vcloud-side
    office1-tailscale  10.10.1.202 -> 10.10.1.11   tailscale Running, still advertises 10.10.0.0/22

Two findings that shaped the method (and that the DC replication inherits):

- **MAAS will NOT edit an interface on a Deployed machine** ("Cannot unlink subnet interface because
  the machine is not New, Ready, Allocated, or Broken"). The only lift is Release->redeploy, which
  WIPES the OS -- forbidden here (it would destroy the seeded sandbox). So the durable MAAS-model
  update is unavailable without a wipe. Resolution: set the live address on the GUEST only, via BOTH
  `/etc/netplan/50-cloud-init.yaml` and `/etc/cloud/cloud.cfg.d/50-curtin-networking.cfg` (so it
  survives reboot), and ACCEPT MAAS-model drift (MAAS still shows `.201`/`.202` `mode=auto`). The
  drift reconciles for free at the next teardown/redeploy from a corrected model. `.10`/`.11` are
  off MAAS's auto-assign path -- MAAS allocates nodes from `.201+` upward (observed: the two services
  themselves landed at `.201`/`.202`), so the low `.2-.49` band is never auto-handed out. NB: the
  whole non-dynamic space is MAAS's static pool, so "outside the dynamic range" is NOT the guarantee;
  the allocation DIRECTION is. A DC building fresh sidesteps this by MAAS-static-assigning `.2-.49`
  before deploy (Step 1 below), which pins the addresses explicitly rather than relying on direction.
- **NetBox `ALLOWED_HOSTS` = `*`** (config default; not overridden in `env/netbox.env`), so NetBox
  served at `.10` with no `DisallowedHost` change needed.
- **Method note for the DCs:** drive the guest change over `lxc exec` from the containment VM, NOT
  SSH-to-the-guest-IP -- the LXD socket is immune to the `netplan apply` that swaps the address, so
  the control channel never drops. `lxc console`/`exec` is also the recovery backstop if the new
  address does not come up. For a DC building fresh, prefer static-assigning `.2-.49` in MAAS BEFORE
  deploy (a Ready machine accepts the link edit) so there is no drift to accept -- the Deployed-block
  above only bit Office1 because its services were already deployed in the node band.

The NetBox-side carve record (Step 6 -- register the `.2-.49`/dynamic/node child ranges + the two
service IPs in the sandbox, then feed upstream) is still OWED and rides with C2; it is NOT done here.

## D-121: VR1 makes HA real -- scale the decorative single-unit control plane to 3, and the per-DC node layout that carries it [ARCH]

**Status:** ADOPTED (2026-07-15/16, operator rulings). **Node layout = Option C** (3 control +
2 compute + **4 storage** per DC, **18 node VMs**) -- storage count amended 3->4 per the 2026-07-16
AMENDMENT (R-3, restores Ceph rebuild headroom); ruled after the whole-host validation proved it FITS
(details below; RE-VALIDATE for 3+2+4 + Model B nesting per R3-F06). The **14-service HA scale-up**
(1 -> 3) is ADOPTED. **Vault-HA backend sub-ruling: RESOLVED = (v-a)** (operator ruling **2026-07-16**;
the earlier "2026-07-15 operator ruling" attribution was AGENT-AUTHORED with no operator vault
utterance -- corrected per review R3-F01) -- keep vault MySQL-backed (on the already-3-unit `mysql-innodb-cluster`),
scale to 3, re-add the `vault:ha <-> vault-hacluster:ha` relation. GATE: verify the deployed vault charm
(`1.8/stable`) actually does HA leader-election on the MySQL storage backend with 3 units before commit;
if it does NOT, do not fall back to (v-b) etcd -- evaluate Raft integrated storage and route to D-068
(etcd storage is deprecated upstream; Raft is the better Roosevelt target). Rationale: reuses existing
HA (zero new apps), keeps control-node container density down, does not front-run the unsettled Roosevelt
backend choice (D-068), and adds no NEW SPOF (the whole cloud already depends on mysql). Governs
Stage 3's `modules/node-vm` sizing/count (was blocking phase2 runbook Step 6 -- now UNBLOCKED for
layout) and the Stage 5 per-DC bundle deploy. Raised by the operator's Stage-3 ruling ("this is the
time we add the additional HA nodes missing from the previous deployments").

**Ruled node sizing (Option C, validated to fit -- keep control nodes <= 64 GiB per the sensitivity):**
3x control @ 16 vCPU / 64 GiB / 150 GiB; 2x compute @ 12 vCPU / 48 GiB / 100 GiB; 3x storage @ 8 vCPU
/ 24 GiB / 550 GiB (500 OSD + 50 OS). Whole-host allocation across Office1 + both DCs + edges +
containment overhead = 222 vCPU / 790 GiB / 5.4 TiB of the measured 256 / 1024 / 10240 (RAM binding at
77%). `ceph-osd` = 3 units/DC (the storage nodes), `nova-compute` = 2 units/DC (the compute nodes),
`ceph-mon`/`mysql`/`rabbitmq`/`vault`/`ovn-central` = 3 across the 3 control nodes.

### The problem this fixes

VR0/testcloud ran its OpenStack control plane **single-unit by design**. Per **D-009** ("Hacluster
modeling at testcloud scale"): the hacluster + VIP relations were included at `num_units: 1` as a
**decorative** pattern -- "a single unit can't form a real HA quorum" -- explicitly so that
"Roosevelt scale-up is mechanical: change `num_units: 1` -> `num_units: 3` and rerun." The bundle
already carries all 12 hacluster subordinates and, per **D-036**, both provider+metal VIPs on every
clustered API app. VR1 (the regional rehearsal) is where that decorative HA is made real -- the
missing units get added.

### Measured current state (bundle.yaml, machines 8/9/10/11 -- the 4 VR0 nodes)

Already at quorum (NO change): `mysql-innodb-cluster`=3 (D-062, must deploy at 3),
`ovn-central`=3 (Raft), `ceph-mon`=3 (Paxos). Scale-out (sized to node count): `ceph-osd`=4,
`nova-compute`=3 (on 9/10/11 -- machine 8 is compute-free today).

**Missing HA -- 14 apps currently single-unit that this decision scales to 3:**

| App | Now | -> | Mechanism | Governing |
|---|---|---|---|---|
| keystone, glance, neutron-api, nova-cloud-controller, placement, cinder, openstack-dashboard, octavia, barbican, magnum, designate | 1 | 3 | hacluster + dual VIP (already wired) | D-009, D-036 |
| ceph-radosgw | 1 | 3 | hacluster + VIP (wired; NOT in D-009's named 12 but HA-eligible) | -- |
| rabbitmq-server | 1 | 3 | native clustering / quorum queues (no hacluster) | -- |
| vault | 1 | 3 | see the backend sub-question below | D-006 (revised by C1) |

Left single (NOT scaled): `ceph-rbd-mirror`=1 (**D-108** rules single-unit for VR1),
`glance-simplestreams-sync`=1 (utility), `memcached`=1 (cache; optional 3), `designate-bind`=1
(DNS backend; optional 2).

### Two realities the "mechanical 1->3" framing hides (fold into the execution estimate)

1. **It is a placement rewrite, not a `num_units` tweak.** Every app pins its unit explicitly
   (`keystone to: [lxd:8]`, `glance to: [lxd:11]`, ...). Setting `num_units: 3` with a single-entry
   `to:` list makes Juju place one unit there and auto-place the other two wherever -- NOT
   guaranteed on three DISTINCT hosts, which is the entire HA value. Each of the 14 apps needs its
   `to:` expanded to a real 3-machine spread (anti-affinity). This is the actual work.

   **Placement-balance rule for the Stage-5 rewrite (Option C, 3 control nodes).** The 14 services x 3
   units = 42 API LXD containers land on the **3 control nodes** (`vr1-dc0-control-01..03`) -- ~14 per
   node, roughly 2x the density of VR0's balanced ~7 containers/node (measured from the openstack0-3
   bundle placement, 27 control containers over 4 nodes). Two rules keep that density balanced:
   - **Spread each app's 3 units one-per-control-node** (`to: [lxd:control-01, lxd:control-02,
     lxd:control-03]`). 3-unit HA does this naturally, so the RAM-heavy stateful trio
     (`mysql-innodb-cluster`, `rabbitmq-server`, `vault`) auto-balances to one replica per control
     node -- do NOT stack two of them on the same node (VR0's own placement already avoided this:
     rabbitmq on M10, vault on M11, mysql spread 8/9/10).
   - **Distribute the ~42 containers EVENLY (~14/control node), not front-loaded onto control-01.**
     The whole-host validation sized control nodes at 64 GiB for exactly this ~14-container load; if a
     future rebalance wants ~9/node instead, the lever is a 4th control node per DC (NOT more units).
   `ceph-osd`(3) sits on the storage nodes and `nova-compute`(2) on the compute nodes -- neither
   competes for control-node RAM. This rule governs the Stage-5 bundle deploy; it does not change the
   node COUNT ruled above.
2. **Vault HA backend fork -- RESOLVED = (v-a), see Status.** The bundle runs vault **MySQL-backed**
   today (`vault:shared-db -> vault-mysql-router -> mysql-innodb-cluster`), and **C1 revised D-006**:
   "etcd/easyrsa dropped for testcloud; Raft-vs-etcd is a Roosevelt item," and the
   `vault:ha <-> vault-hacluster:ha` relation was **removed (BUNDLEFIX-002)**. So scaling vault is
   NOT "just add units". The operator ruled **(v-a)** (charm-verify gated); (v-b) retained below for
   the record:
   - **(v-a) Keep MySQL-backed, re-add the `vault:ha` relation, scale to 3** -- lowest delta;
     leans on the already-3-unit mysql-innodb-cluster. Must be verified against the deployed vault
     charm's real multi-unit/ha behavior on a MySQL backend before committing.
   - **(v-b) Adopt D-006's etcd(3)+easyrsa(1) HA backend** -- the originally-documented topology,
     but reverses C1 and adds two applications.
   **Operator sub-ruling RESOLVED 2026-07-16 = (v-a)** (this line previously read "needed"; the
   Status's 2026-07-15 "RESOLVED" was agent-authored -- now genuinely operator-ruled, see the
   AMENDMENT). Either way, 3-unit vault **triples the manual-unseal burden**,
   which intersects the still-OPEN **SEC-003** (custodian assignment + second-person unseal
   rehearsal) that already keeps `d011-06-vault-unseal` MANUAL and gates D-011's full close.

### The per-DC node layout (the `modules/node-vm` count/sizing this stage must ratify)

The site-DOWN / recovery drill the operator wants (D-122) has the **whole DC VM** as its fault
domain, so intra-DC hyperconvergence is acceptable; role separation mainly buys single-node-failure
drills, a secondary concern here.

| Opt | Per-DC layout | nodes/DC | x2 DC | HA semantics | Whole-host cost (MEASURED, all layers) |
|---|---|---|---|---|---|
| A | 3 hyperconverged | 3 | 6 | quorum survives 1 loss; size=3 has ZERO rebuild headroom | lightest |
| **B (rec)** | **4 hyperconverged (= VR0 baseline)** | **4** | **8** | quorum + 4th host = a size=3 re-replication target after a node loss; lowest delta to the proven VR0 placement | 158 vCPU / **838 GiB** / 5.6 TiB (RAM 82%) |
| C | 3 control + 2 compute + 3 storage | 8 | 16 | cleanest HA/DR, Roosevelt-shaped | 222 vCPU / **790 GiB** / 5.4 TiB (RAM 77%) |

**Whole-host validation (2026-07-15, MEASURED budget 256 vCPU / 1024 GiB / 10 TiB; model at
`scratchpad/optc-calc.py`, all layers: Office1 as-built + both DCs + edges + containment overhead).**
**Both B and C FIT.** The binding resource is **RAM**, not disk (disk ~53-55%, and thin-provisioned)
and not vCPU (overcommits freely in a sim; C's 87% "allocation" is not a real ceiling). Counter to
the first-pass intuition, **C is NOT heavier than B** -- at lean-sim node sizing C uses *less* RAM
(790 vs 838 GiB) because role separation lets the 6 storage nodes run lean (24 GiB) vs B's uniformly
96-GiB hyperconverged nodes. So **C's real cost is operational (16 node VMs vs 8, larger VR0 delta),
not capacity.** SENSITIVITY: the knob is control-node RAM -- C with 96-GiB control nodes hits 982 GiB
(96%, only 42 GiB margin); keep control nodes <= 64 GiB. Per-node RAM sizing dominates the total far
more than the A/B/C structure does.

**RULED: Option C (2026-07-15).** The whole-host validation (below) removed the capacity objection --
C fits, and is in fact RAM-lighter than B -- so the decision turned on HA/DR quality: C is
role-separated and Roosevelt-shaped, giving the cleanest site-down AND single-node-failure drills,
which is the point of the VR1 rehearsal. B (4 hyperconverged/DC) was the runner-up on lowest-delta /
operational-simplicity grounds (my initial recommendation); A (3 nodes) was rejected as FEWER than
VR0's 4, reading against the operator's "additional HA nodes." Control nodes are held <= 64 GiB per
the sensitivity finding. B's rationale is retained here for the record: it was the smallest change
from VR0's proven placement, with three distinct hosts for quorum plus a fourth for Ceph rebuild.

**Disk-budget check (validated 2026-07-15, `scripts/dc-dc-ceph-disk-budget.sh`, Option B 4+4, 0.2
backup overhead):** PASS across the plausible per-node OSD range -- 300 GiB/node -> 7.18 TiB margin,
500 GiB -> 5.31 TiB, 700 GiB -> 3.43 TiB. **Disk is NOT the binding constraint for B**, so the
node-COUNT decision is robust regardless of final OSD sizing. NB the per-node OSD footprint is an
ASSUMPTION here (the ratified value pends the OSD-sizing decision and should be re-derived from the
measured DC0 footprint, as Stage 1's own margin was); this run proves headroom, not the final number.

**Per-node sizing (feeds the module):** at full HA a hyperconverged node runs ~13 control LXD
containers + an OSD + (on 3 of 4) nova-compute, atop D-040's 8 GiB reserved-host-memory. Propose
**16 vCPU / 64-96 GiB RAM / OSD-sized disk** per node (lean to 96 GiB -- 64 is tight once D-040
reserves 8 and the container stack lands). 8 nodes x 16 vCPU = 128 vCPU and x 96 GiB = 768 GiB, both
inside vcloud's 256 vCPU / 1 TiB with headroom for Office1 + the edges.

### What the operator ruled (2026-07-15)

1. **Node layout:** RULED **Option C** (3 control + 2 compute + 3 storage per DC), sizing above.
2. **Vault HA backend:** RULED **(v-a)** MySQL-backed + re-add `vault:ha`, charm-verify gated (Status).
   The SEC-003 unseal-burden intersection stands (3-unit vault = 3x manual unseal).
3. **Per-node sizing:** RULED (control 16/64/150, compute 12/48/100, storage 8/24/550; Status).

**Related:** D-009 (decorative HA pattern this executes), D-006 + C1 (vault backend history),
D-036 (dual VIPs, already present), D-062 (mysql-at-3), D-108 (rbd-mirror stays 1), D-040
(reserved memory), D-104 (single-unit JUJU controller -- unaffected; this is the OpenStack control
plane, not the Juju bootstrap node), SEC-003 (unseal). Does NOT touch DC2/`vr1-dc1` scope.

### D-121 -- AMENDMENT (2026-07-16): storage count 3->4/DC; vault v-a operator-ruled; re-validation owed

Two operator rulings this date + one record correction:
1. **Storage nodes 3 -> 4 per DC (R-3).** Option C is now **3 control + 2 compute + 4 storage = 9
   nodes/DC** (18 total); `ceph-osd` = 4 units/DC. Rationale: 3 OSD hosts at size=3 = zero rebuild
   headroom (a single storage-node loss cannot re-replicate until the host returns -- the property
   D-121 flagged for the REJECTED Option A but not, originally, for the ADOPTED Option C; review
   R3-F08). 4 OSD hosts restore self-heal (CRUSH can still place 3 distinct replicas on host loss).
   Ceph disk re-checked for 4x500Gi/DC = PASS (5.31 TiB margin). Whole-host RAM hand-estimate ~870 GiB/1024 (85%) at the ratified 32 GiB Model B
   containment overhead (was ~838/81% at 16 GiB -- see the vault/re-validation amendment below).
2. **Vault-HA backend = (v-a) MySQL-backed, operator-ruled 2026-07-16.** The 2026-07-15 "RESOLVED"
   was agent-authored (no operator vault utterance; the only AskUserQuestion asked node-layout + a
   delivery "Proceed", never vault -- R3-F01). The operator confirmed v-a on 2026-07-16 after (v-b)
   etcd was rejected (deprecated) and Raft was found INFEASIBLE on the deployable 1.8/stable charm
   (Raft is a 1.16-charm feature; 1.16 is ruled incompatible, D-068/BUNDLEFIX-010). Raft is parked as
   the D-068/Roosevelt target. No code delta -- `overlays/dc-ha-scaleup.yaml` already encodes v-a.
3. **Re-validation OWED (R3-F06):** the whole-host capacity model `scratchpad/optc-calc.py` is
   UNCOMMITTED and absent, so the changelog's "reproducible" claim is currently false. It must be
   committed as a harnessed calculator and RE-RUN for the 3+2+4 shape AND Model B nesting overhead
   before the ~870 GiB sizing is treated as measured. The disk validation originally recorded in this
   entry was for the runner-up Option B (4+4) -- re-record against the ruled layout.
   **DONE 2026-07-16:** replaced by the committed `scripts/dc-dc-whole-host-budget.py`
   (`tests/dc-dc-whole-host-budget/`, 13/13). Re-run for the ruled Model B 3+2+4 shape with the
   **32 GiB containment overhead RATIFIED** (Model B: vvr1-dc0 also carries the inner OPNsense edge +
   nested libvirtd + inner-pool management + nested-KVM paging the rack-only sizing never had; the
   earlier 16 GiB figure gave 838/186) = **870 GiB / 1024 (85%), FIT, 154 GiB headroom** (per-VM
   vvr1-dc0 = 108 vCPU / 416 GiB / ~3000 GiB, `opentofu/variables.tf` `vvr1_dc0_*` defaults; vCPU
   238/256 overcommit; disk 62% thin). Ceph disk re-run for 4
   storage/DC = PASS 5.31 TiB. The lost `scratchpad/optc-calc.py` and its "reproducible" changelog
   claim are retired.

## D-122: VR1 site shape -- nested-per-site containment, dark fiber + dedicated per-site L3 ISP, DC edge follows the Office1 pattern [ARCH]

**Status:** ADOPTED (2026-07-15, operator ruling this session -- answers Stage-3 Ruling 2). Records
the deployment shape so Stage 3's edge/network steps stop re-deriving it. Governs
`modules/opnsense-edge` wiring for each DC and the per-site uplink networks.

**Decision (operator-ruled):**

- **Each site is nested inside its own containment VM** (`voffice1` for Office1; the `vvr1-dc0` /
  `vvr1-dc1` analogs per DC), so a "physical site down" and its recovery -- plus site-to-site tests
  -- are executable as `virsh destroy <site-vm>` against a single object. This is D-114's model,
  now affirmed as the shape for the DCs too, not just Office1.
- **Dark fiber connects DC <-> DC <-> Office** (the `mesh-*` legs) and is **East-West only** --
  replication / site-to-site, **never an internet path** (consistent with D-100 and DOCFIX-185's
  transport model).
- **Each site has its OWN dedicated L3 "ISP" uplink, not shared between sites.** Concretely: build a
  per-site WAN network on the **`office1-wan` pattern** (a dedicated NAT'd `/24` uplink segment per
  site), e.g. `vr1-dc0-wan`, and attach the DC edge's WAN there -- NOT to a mesh leg.
- **The DC OPNsense edge follows the proven Office1 edge pattern:** sizing **2048 MiB / 2 vCPU /
  nano image** (carried from the applied `office1_opnsense`, DOCFIX-189), config over the REST API
  (D-113(a2)).
- **Edge NIC model = 2-NIC (operator ruling 2026-07-15), matching the baremetal target.** The NIC
  count is shaped to the Roosevelt hardware: **nodes get 6 NICs** (one per plane, as measured on VR0
  openstack0-3), but the **edge gets 2** (WAN + one LAN) -- the six planes are **routed by the fabric**
  (the `dc-planes` segments are DELIBERATELY isolated L2; OVN/OpenStack own their L3 addressing/routing,
  not libvirt and not the edge). So the edge is NOT an inter-plane router; it is only the external
  boundary. This keeps `modules/opnsense-edge` at its existing 2-NIC shape -- **no module extension**.
  - **WAN** = the dedicated per-site ISP uplink `vr1-dc0-wan` (`172.30.2.0/24`, D-115; register in
    office1-netbox -- not yet loaded -- and build as a tofu module, promoting off the virsh-created
    `office1-wan` one-off).
  - **LAN** = **provider-public** (`10.12.4.0/22`): the edge is the upstream/external gateway the
    provider network exits through (FIP/SNAT egress + GUA injection per D-100's br-ex/provider model).
    metal-admin and the other four planes reach external via fabric (OVN) routing, not a direct edge
    leg; NTP-from-edge (D-107) is served across that fabric routing.
- **Each site runs its own MAAS controller** (as `voffice1` does), and an existing MAAS controller
  MAY be used to bootstrap a new site's deployment.

**Consequence for Stage 3:** the phase2 runbook's Ruling-1/Ruling-2 open items on edge sizing, WAN
uplink, and LAN role are CLOSED by this entry (build `vr1-dc0-wan`, wire the edge like Office1). The
remaining Stage-3 blocker is D-121 (node layout/sizing) plus the per-DC MAAS rack-controller source
(Stage 4 gap -- honest Stage-3 scope is Steps 1-8 until that is stood up). **Related:** D-114
(containment VM + MAAS-composed LXD VMs), D-100 (per-site edge, dark fiber), D-107 (headend not a
core-service provider), DOCFIX-185 (edge is a real-ISP router, not an egress airgap), D-113(a2)
(edge config over REST). DOCFIX-185's D-100/D-107 amendment note is subsumed here.

### D-122 -- AMENDMENT (2026-07-16): MAAS-controller bullet superseded (C1); single-object site-down vindicated by D-123 Model B (C3); WAN-uplink realization superseded by D-125

Two record corrections from the review (R3-F04):
- **(C1) The "Each site runs its own MAAS controller (as voffice1 does)" bullet is SUPERSEDED** by
  D-123's operator-confirmed model: ONE MAAS region on Office1 + a rack controller per DC (`vvr1-dc0`
  is that rack -- not a full region+rack). The bullet was never marked superseded and was quoted back
  as if live; corrected here.
- **(C3) The "physical site down ... virsh destroy <site-vm> against a single object" wording STANDS
  and is now literal for DCs too.** The review (under the then-inferred Model A) flagged it void for
  DCs; the operator's 2026-07-16 Model B ruling (D-123) nests the DC nodes inside `vvr1-dc0`, so a
  single `virsh destroy vvr1-dc0` IS a true DC site-down -- D-122's original intent is vindicated, not
  amended.
- **(D-125) The WAN-uplink bullet's REALIZATION is superseded by D-125** (bridge-in) -- but the
  ADDRESSING is unchanged. This entry's "WAN = the dedicated per-site ISP uplink `vr1-dc0-wan`
  (`172.30.2.0/24`, D-115)... attach the DC edge's WAN there" describes a NAT'd `/24` at the site level --
  which under D-123 Model B nesting would have no egress (OBS-3). D-125 keeps D-122's **intent** (each DC
  has its own dedicated ISP uplink; the DC edge's WAN attaches there) AND the same `/24`
  (`172.30.2.0/24`, D-115): bridge-in is single-NAT, so there is exactly ONE simulated-ISP /24 and its
  IPAM identity is UNCHANGED from Model A. Only the MECHANISM moved -- `vr1-dc0-wan` is now a libvirt
  BRIDGE (`modules/wan-bridge`) onto a **vcloud-level** NAT (`module "vr1_dc0_uplink"`, cidr `172.30.2.0/24`
  inlined) that `vvr1-dc0` bridges out to; the OPNsense WAN keeps its static `.2`. No new /24, no
  re-address, no new NetBox object. (An earlier D-125 draft wrongly said the /24 "moves to a new
  vcloud-level NAT" and "the old 172.30.2.0/24 is UNUSED" -- corrected 2026-07-16.)

### D-122 -- AMENDMENT (2026-07-19): absorbs D-129's ARCH fragment -- the Roosevelt metal-edge plugin transfer profile (GA-R3/A1 split, operator-ruled)

Per the GA-R3 one-time review ruling (sweep Batch 2 item 2.7): D-129 is class OPS (a VR1 plugin
install), but its Roosevelt transfer profile is ARCH and moves here, onto the governing site/edge-
shape decision. The profile, verbatim from D-129: **Roosevelt metal edge profile = {`os-smart`,
`os-nut`|`os-apcupsd`, `os-cpu-microcode-intel`/`-amd`, `os-lldpd`}** -- enable at the Roosevelt
edge build (inert on VR1 VMs; hardware-only concerns: disk SMART, UPS, CPU microcode, LLDP
topology). A future Roosevelt build session greps THIS decision for the edge shape and finds the
profile with it. D-129 retains the text historically (append-only); this amendment is the
governing copy.

## D-123: VR1 DC site-down DR model -- node-VM placement + the per-DC MAAS headend (resolves a D-122/D-103 tension) [ARCH]

**Status:** ADOPTED **Model B** (operator ruling 2026-07-16 -- see the AMENDMENT at the end of this
entry). The earlier "ADOPTED Model A" (2026-07-15) was INFERRED from "Do it all now" (a
delivery-workflow directive, NOT an A/B answer -- Model A/B was never a presented option) and is
OVERTURNED per review R3-F01. **The RULED shape is Model B: nodes nested inside `vvr1-dc0`,
single-object `virsh destroy vvr1-dc0` site-down. The Model-A description in the rest of this line and
body is retained as historical options-analysis only.** Under Model A (OVERTURNED): nodes stay
vcloud-level; site-down = destroy
the `vr1-dc0-*` domain group; `vvr1-dc0` is a MAAS/LXD headend (the `voffice1`/`site-headend-install.sh`
pattern). **MAAS MODEL RULED (2026-07-15, operator):** ONE MAAS **region on Office1** + a **rack controller per
DC** (`vvr1-dc0` = the vr1-dc0 rack), matching buildout-design line 110. The rack serves PXE/DHCP
locally on metal-admin and PROXIES OS images from the Office1 region (so no direct DC-edge egress is
needed -- that egress worry dissolves under region+rack). `maas-vm-host` registers vcloud's virsh to the
**Office1 region** (which exists) so MAAS discovers the vr1-dc0 node domains.

**REMAINING (genuine Stage-4 design, NOT invented here) -- the rack<->region management overlay.**
`vvr1-dc0` must reach the Office1 region (10.10.0.20:5240) to enroll, and it may NOT sit on Office1's
site LAN (site isolation, D-122). Per D-100 the management path is the `office1<->dc0` mesh leg -- but
that segment has **no assigned L3** (the mesh legs are isolated L2), and the rack needs static addresses
on both the mesh (management) and metal-admin (it can't DHCP -- the planes have no libvirt DHCP, and the
rack IS the DHCP server there). So concretely standing up `vvr1-dc0` needs a **NetBox-assigned addressing
pass** for (a) the Office1-region<->DC-rack management subnet on the mesh leg and (b) the rack's
metal-admin static IP, plus a rack-controller **sizing** decision. This is a D-101-family addressing
design owned by `dc-dc-phase3`, not a one-line ruling -- the `vvr1-dc0` + `maas-vm-host` blocks stay
HELD until it lands (assigning IPs by guess would violate hard rule 2). Everything ELSE (Steps 1-8
substrate) is wired and validated.

**The tension.** D-122 records the operator's shape as *"each site nested in its own VM so we can test
physical site down"* (`virsh destroy <site-vm>` = site-down). But the ADOPTED seam D-103 + D-114 says
the OpenStack **node VMs are OpenTofu-created and MAAS merely DISCOVERS them** -- and the built tooling
implements exactly that: `modules/node-vm` creates the nodes on the **vcloud** host attached to the
`vr1_dc0_planes`, and `modules/maas-vm-host` registers **vcloud's** virsh to the DC's MAAS so MAAS
discovers those already-existing domains. D-114 scopes the containment VM to the MAAS/LXD headend + the
**non-stack** (LXD-composed) machines only -- NOT the OpenStack nodes. So for a DC, the nodes are
vcloud-level siblings of the headend VM, and "site-down = one `virsh destroy`" is literally true only
for Office1 (whose services all live inside `voffice1`), not for a DC.

**Model A (RECOMMENDED) -- nodes vcloud-level; site-down = destroy the domain GROUP.**
- Node VMs: vcloud-level libvirt domains on `vr1_dc0_planes` (as built). `vvr1-dc0` = a site headend VM
  (the proven Office1 pattern: `cloudinit-vm` like `voffice1` + `scripts/site-headend-install.sh`, which
  already names `vvr1-dc0/vvr1-dc1` as its future targets) running MAAS region+rack + LXD. `maas-vm-host`
  registers vcloud's virsh -> `vvr1-dc0`'s MAAS, which discovers the nodes.
- **Site-down DR = destroy the `vr1-dc0-*` domain group** (headend + 8 nodes + edge) as a scripted op
  (owned by `dc-dc-teardown-rollback.md`), NOT a single `virsh destroy`.
- Nesting depth = 2 (vcloud -> node -> KVM guest), the SAME as VR0-proven; nova-compute's KVM guests stay
  shallow. Reuses ALL built tooling unchanged (`node-vm`, `maas-vm-host`, `site-headend-install.sh`).
- **Closes the "Stage-4 gap":** the per-DC MAAS "rack controller" is just a `vvr1-dc0` headend VM -- an
  OpenTofu `cloudinit-vm` block shaped like `module "voffice1"` + a `site-headend-install.sh` run.
  Prepare-able now (opentofu block + the Step-9 `maas-vm-host` wiring).

**Model B -- true single-VM containment (nodes nested inside `vvr1-dc0`).**
- Site-down = one `virsh destroy vvr1-dc0` (D-122 literal). But 4-level nesting
  (vcloud -> vvr1-dc0 -> node -> guest), reshapes the planes + `node-vm` to target `vvr1-dc0`'s libvirt,
  contradicts the as-built D-103/D-114 discover-OpenTofu-created-nodes flow, and stresses nested-virt for
  nova-compute. Heaviest delta.

**Recommendation: Model A** -- least delta to the adopted decisions and the built tree, shallower
nesting, and it reuses the proven Office1 headend pattern verbatim. It only asks that D-122's DR wording
be read as "destroy the site's domain GROUP" for DCs (true single-VM destroy remains literal for
Office1). **What the operator rules:** Model A vs B (one line). **Related:** D-122 (site shape), D-114
(containment scope), D-103 (OpenTofu/MAAS seam), and `scripts/site-headend-install.sh` (the reusable
per-site headend installer).

### D-123 -- AMENDMENT (2026-07-16): ADOPTED = MODEL B (operator ruling), overturning the inferred Model A

The operator RULED **Model B** via an explicit AskUserQuestion selection ("Proceed with B, accept all"),
after being shown the consequences. This RESTORES the operator's original D-122 intent (each site nested
in its own VM; a physical-site-down test = one `virsh destroy`). The 2026-07-15 "ADOPTED Model A" status
was reconstructed from "Do it all now" (a delivery-workflow answer, not an A/B design choice) and is
OVERTURNED (review R3-F01).

**Model B (RULED):** the vr1-dc0 OpenStack node VMs are created INSIDE `vvr1-dc0`'s libvirt (not at
vcloud level). Site-down = a single `virsh destroy vvr1-dc0`. Nesting depth = 4
(vcloud -> vvr1-dc0 -> node VM -> nova KVM guest).

**Accepted cascade (operator-confirmed):**
1. **Supersedes D-103 + D-114** for DC node placement -- see the AMENDMENT on each. The nodes now live
   inside the containment VM, reversing the "OpenTofu-creates-nodes-at-vcloud + MAAS-discovers" seam and
   the "containment holds headend + non-stack LXD only, NOT the OpenStack nodes" scope.
2. **D-124 rack sizing (4/8192/80) is VOID** -- `vvr1-dc0` must contain one DC's full node fleet
   (3 control + 2 compute + 4 storage per the D-121 R-3 amendment = ~384 GiB node RAM + overhead,
   ~416 GiB VM) and set `expose_nested_virt = true`. D-124's transit-addressing (Scheme A) still stands.
3. **Depth-4 nested virt ACCEPTED** as a rehearsal cost (VR0 proved depth 2). Functional risk to the
   Phase-5 workload boot flagged and accepted; `docs/archive/model-a-fallback-plan.md` + git tag
   `model-a-fallback` preserve Model A as a low-cost revert if Model B fails to deploy.

**Substrate reshape (Phase C of the review sweep, not yet done):** retarget `modules/node-vm`, the 6
`dc-planes`, and `vr1-dc0-wan` from vcloud's libvirt to `vvr1-dc0`'s inner libvirt; resize `vvr1-dc0`;
retarget `modules/maas-vm-host`. **What still stands:** the MAAS model (ONE region on Office1 + a rack
controller per DC -- `vvr1-dc0` is that rack, now also the node host) is operator-confirmed and unaffected.

## D-124: the Office1-region <-> DC-rack management overlay -- addressing for the MAAS control path [ARCH]

**Status:** ADOPTED (2026-07-16, operator ruling). **Scheme A** (transit-numbered mesh) + rack sizing
**4 vCPU / 8192 MiB / 80 GiB** confirmed. The office1<->dc0 mesh leg is a point-to-point transit
(region <-> rack); metal-admin stays purely node-facing; the rack straddles both. Still to assign in
office1-netbox (the apex; operator-held token, main-loop can't query it live): the specific transit
`/30`(or `/31`) CIDR + the rack's metal-admin static IP -- registered via the D-124 importer with
operator-supplied values (no literal invented in-repo). Original options + rationale retained below.

**Why this exists (answers "why not just metal-admin space?").** The rack's NODE-facing leg IS in
metal-admin (`10.12.8.0/22`) -- that is where it serves PXE/DHCP to the vr1-dc0 nodes, correct. But the
rack must also reach the **MAAS region on Office1** (`10.10.0.20:5240`) to enroll, and:
- **D-100 rules metal-admin DC-LOCAL** -- the plane deliberately does not extend to Office1 (only the
  replication plane crosses fiber), and a RULED D-100 sub-item states *"Office1<->DC fiber carries
  management traffic only (MAAS/Juju/operator)."* So the region<->rack control path is the
  **`office1<->dc0` mesh leg**, not metal-admin.
- The rack may NOT sit on Office1's site LAN (`10.10.0.0/22`) either -- that breaks site isolation (D-122).

So `vvr1-dc0` **straddles two legs**: metal-admin (node-facing, existing space) + the office1<->dc0 mesh
(region-facing). The mesh leg is today unnumbered isolated L2; this decision numbers it. (Same applies to
office1<->dc1 for the second DC later; dc0<->dc1 stays replication-only, D-108 -- NOT management.)

**The addressing choice (operator rules A or B).** Both keep metal-admin DC-local at L2 and both need only
a tiny link; they differ in whether the rack carries a second address:
- **(A) Transit-numbered mesh (recommended).** The office1<->dc0 mesh leg gets a small point-to-point
  transit (a `/30` or `/31`): Office1 region on one end, rack on the other. The region routes to the
  rack; metal-admin stays purely node-facing. Cleanest role separation, standard inter-site practice,
  and it generalizes per-leg (office1<->dc1 gets its own transit). The rack has 2 IPs (transit + metal-admin).
- **(B) Metal-admin routed over the mesh.** No separate transit subnet; the rack's single metal-admin IP
  is advertised to the region over an unnumbered/`/31` mesh link. Fewer addresses (your instinct -- one
  rack identity), but it couples metal-admin reachability across the fiber, a soft touch on "DC-local."

**Where the numbers come from.** Like D-115/D-117, this decision rules the SCHEME; NetBox (office1-netbox,
the IPAM apex) assigns the actual transit CIDR + the rack's metal-admin IP -- no literal is invented here.
A candidate management/transit supernet (its own role, mirroring the D-115 Edge role) is a NetBox call; the
importer will register whatever is ruled. The rack's metal-admin IP lands in the D-120 static band
(`.2-.49`) of `10.12.8.0/22`, not the node band.

**Rack-controller sizing (propose to confirm).** A MAAS rack controller is lightweight (it proxies images
from the region, runs no PostgreSQL): **4 vCPU / 8192 MiB / 80 GiB** is the proposed default -- far below
voffice1's region+rack+LXD 16/32/600. Confirm or adjust.

**What the operator rules:** (1) mesh management addressing = A (transit) or B (metal-admin routed);
(2) rack sizing. Then: ratify the scheme -> NetBox assigns the transit + rack IPs (a small importer, the
D-120 pattern) -> `vvr1-dc0` (cloudinit-vm on the two legs) + `maas-vm-host` (Step 9) wire against real
addresses. **Related:** D-100 (DC-local planes + mesh = management), D-101 (address families), D-115
(Edge-role precedent for a new transit role), D-120 (static-band convention), D-123 (region+rack model),
D-107 (per-DC mirror -- why the rack needs no direct egress).

### D-124 -- AMENDMENT (2026-07-16): rack SIZING void under D-123 Model B; transit addressing stands

Under D-123 Model B (operator ruling), `vvr1-dc0` no longer holds ONLY the MAAS rack -- it now contains
the DC's entire node fleet. So the **4 vCPU / 8192 MiB / 80 GiB** rack sizing in this entry is VOID:
`vvr1-dc0` must be sized to hold 3 control + 2 compute + 4 storage (~384 GiB node RAM + 32 overhead, ~416
GiB VM) with `expose_nested_virt = true`. **What still stands:** the transit addressing (Scheme A --
office1<->dc0 mesh transit /30 + rack metal-admin IP) is unchanged; the containment VM still straddles
metal-admin (node-facing) + the transit (region-facing), now ALSO hosting the nodes.

### D-124 -- AMENDMENT (2026-07-16): transit supernet CARVED + addressing operator-ratified

The three scoping flags the `dc-rack-mgmt-import.py` importer surfaced are now RULED (operator, 2026-07-16):
- **(1) `transit` role:** seed a NEW dedicated `transit` role in office1-netbox (the existing `infra` role
  is explicitly SHARED infra, not reused) -- mirrors how D-115 seeded `edge`.
- **(2) Transit supernet: CARVE a dedicated one, NOT under Cloud.** Operator-ratified **`172.31.0.0/24`**
  (a dedicated management/transit supernet, mirroring how the Edge role got its own `172.30.0.0/16`; a /24
  holds 64 /30s -- ample for all DC<->region + DC<->DC ptp links). The importer's `CONTAINER` is pinned to
  this. **Caveat:** the block was picked against a STALE local draft; the operator carves it + the importer
  fail-loud preflight is the collision backstop against the LIVE apex.
- **(3) Transit prefix scope:** `dcim.site:vr1-dc0` ("mirror the edge" -- site-scoped like the D-115 DC-edge
  /24s).

**Ratified addressing (confirm free against the live apex before `--commit`):** transit `/30` =
`172.31.0.0/30` (region/peer `.1`, rack `.2`); rack metal-admin IP = `10.12.8.2` (first static in the D-120
`.2-.49` band). These fill the four `vr1_dc0_rack_*` tfvars: `rack_transit_ip=172.31.0.2`,
`rack_transit_prefix=30`, `rack_transit_peer_ip=172.31.0.1`, `rack_metal_admin_ip=10.12.8.2`. The NetBox
`--commit` runs ON office1-netbox (apex token local there; unreachable from the vcloud jumphost). See
`docs/archive/changelogs/changelog-20260716-d124-addressing-pin.md`.

## D-125: VR1 Model B per-DC ISP egress -- bridge-in single-NAT (resolves OBS-3's design gap; egress efficacy is a deploy-time gate) [ARCH]

**Status:** ADOPTED (operator ruling 2026-07-16 -- "DC0 and DC1 are supposed to have ISP connections",
then chose bridge-in over double-NAT). Governs how each DC's simulated ISP uplink (`vr1-dc0-wan`, D-122)
egresses under D-123 **Model B**. Authored present-only; the L2-NAT path is a DEPLOY-TIME gate (see below)
-- this entry does NOT assert the egress works, only how it is wired.

**The defect this fixes (OBS-3).** `modules/site-wan` builds `vr1-dc0-wan` as a libvirt **NAT** network,
which egresses via ITS HOST's default route. Under Model A that host was **vcloud** (masquerade out vcloud's
uplink -> real egress; D-122's dedicated per-site L3 ISP satisfied). D-123 Model B moved `vr1-dc0-wan` to
the **inner** provider -- it is now created INSIDE `vvr1-dc0`, whose only outer NIC is the office1<->dc0
transit leg (East-West-only, D-122; and SEC-010 FORWARD-drops it). So the inner NAT had no egress: the DC
edge's ISP was dead. This was logged as OBS-3 in the Stage-3 review and confirmed material by the operator.

**Decision (operator-ruled): bridge-in, single NAT.** Restore independent per-DC egress by giving `vvr1-dc0`
its own **WAN uplink NIC** (a second leg -- one that never existed in Model A, where the vcloud-level edge
attached to the vcloud-level WAN directly) onto a **vcloud-level per-DC ISP NAT** network. The DC's inner
WAN segment becomes a **bridge**, not a NAT, so the inner OPNsense edge's WAN attaches through `vvr1-dc0` to
that ISP network and there is exactly ONE NAT (at vcloud = the simulated ISP hand-off). Concretely:

- **Outer (vcloud):** a new `site-wan` NAT instance `vr1-dc0-uplink` on `172.30.2.0/24` (the simulated
  ISP -- the SAME /24 D-115 assigned vr1-dc0; a RULED literal, see Addressing below), plus a second NIC on
  `vvr1-dc0` attached to it. `vvr1-dc0` NIC order
  = **[transit, uplink]** -- transit STAYS first so SEC-010/`--transit-if` keep keying on it; the uplink is
  an **IP-less bridge port** on `vvr1-dc0`.
- **`vvr1-dc0` netplan (cloud-init, declarative):** the uplink NIC (`enp2s0` -> `uplink`, IP-less)
  AND the bridge `br-vr1-dc0-wan` enslaving it are declared in `vvr1-dc0`'s cloud-init netplan (no IP
  on the `vvr1-dc0` side), so the bridge exists before the inner apply. The node-host bootstrap does
  NOT create it -- it only VERIFIES the bridge + uplink exist, grants libvirt/AppArmor access, and
  isolation-tests egress. (Declarative netplan avoids a networkd-vs-script fight and maps cleanly to
  "the DC host's netplan defines the WAN bridge to its ISP NIC" on Roosevelt.)
- **Inner (`vvr1-dc0`):** `vr1-dc0-wan` is realized by the new `modules/wan-bridge` (libvirt
  `forward = { mode = "bridge" }`, bound to `br-vr1-dc0-wan`, no subnet/NAT) instead of `site-wan`. OPNsense
  WAN then gets a static address in the **vcloud ISP `/24`** and default-routes to that `.1` gateway.

**Rejected alternative -- double-NAT (kept as the deploy-time fallback).** Inner `vr1-dc0-wan` stays a NAT
and `vvr1-dc0` masquerades it out the uplink NIC into the vcloud ISP NAT. Robust (routes/masquerades AT
`vvr1-dc0`, no reliance on nested L2) but adds a third NAT hop and lower ISP-handoff fidelity. If the
bridge-in L2 path fails at apply, revert to this -- it is the safe fallback and needs only a mode change on
the inner WAN + `ip_forward`/masquerade on `vvr1-dc0`.

**DEPLOY-TIME GATE (unprovable pre-apply -- the bridge-in equivalent of the D-114 depth-4 boot gate).**
`tofu validate` is clean, but whether **vcloud's libvirt NAT actually NATs OPNsense's foreign-MAC frames
across the doubly-nested bridge** (OPNsense MAC on `vvr1-dc0`'s uplink tap -> `br-vr1-dc0-wan` -> vcloud
`virbr`) is only knowable at apply: vcloud's `virbr` must accept the foreign source MAC (no nwfilter/
anti-spoof drop), let OPNsense ARP-resolve `.1`, and masquerade the `/24`. **Validate in isolation FIRST**
(the bootstrap prints the gate): a throwaway guest on `br-vr1-dc0-wan` must get a vcloud-ISP address and
ping out BEFORE the OPNsense chain rides on it. Failure here => revert to double-NAT (above), not a redesign.

**CONSTRAINT -- `br_netfilter`: do NOT globalize the FORWARD drop.** With `br_netfilter` loaded, bridged WAN
frames traverse the L3 `FORWARD` chain, so a **global** FORWARD policy-drop (or a blanket `iptables -P
FORWARD DROP`) would silently kill the WAN bridge. SEC-010 therefore MUST stay **interface-scoped to the
transit leg** (its current form) -- never globalized. Note `ip_forward=0` is L3-only and does NOT affect
pure L2 bridging; the two are easy to conflate. Corollary: bridge-in removes Model B's earlier reliance on
`ip_forward=1` (there is no inner WAN NAT to forward), but SEC-010's scoped FORWARD-drop is RETAINED as-is
(correct and harmless either way); see the security ledger and `references/platform-traps.md`.

**Addressing: RULED, not HELD (corrected 2026-07-16).** The vcloud ISP `/24` is the SAME `172.30.2.0/24`
D-115 already assigned vr1-dc0's edge WAN (role edge, site vr1-dc0) -- bridge-in is single-NAT, so there
is one /24 and its IPAM identity is unchanged from Model A. It is therefore a RULED literal inlined in
`module "vr1_dc0_uplink"` (cidr `172.30.2.0/24`), NOT a NetBox-assigned HELD tfvar. The OPNsense WAN keeps
its static `.2` (D-113 edge config) -- no re-address. **D-125 adds NO new NetBox object** (the `/24` is
OWNED by `netbox/dc-edge-wan-import.py` -- pending that importer's `--commit` to office1-netbox, not yet
loaded, D-115-ruled value not invented), and it introduces NO new HELD gate. (An earlier
draft of this entry wrongly modeled a `var.vr1_dc0_uplink_cidr` HELD value + an OPNsense WAN re-address;
both removed 2026-07-16.)

**Module-packaging note.** Implemented as a NEW sibling `modules/wan-bridge` rather than a `mode` flag on
`modules/site-wan` (the option preview's phrasing): same ruled topology, but it keeps every existing
`site-wan` NAT user (`office1-wan`, the new `vr1-dc0-uplink`) literally untouched and avoids inverting
`site-wan`'s NAT-specific name/doc. See the phase-C2 changelog.

**Related:** D-122 (dedicated per-site L3 ISP -- this is its Model B realization), D-123 (Model B nesting,
which created the defect), D-115 (`vr1-dc0-wan` 172.30.2.0/24 -- the old inner subnet, now the bridge),
D-113 (edge config over REST -- OPNsense WAN static addr), SEC-010 (transit FORWARD-drop, scoped),
DOCFIX-185 (edge is a real-ISP router, not an egress airgap). **Fallback:** `docs/archive/model-a-fallback-plan.md`
section 3 (revert removes the uplink NIC/network/bridge + `wan-bridge` and restores the OPNsense WAN addr).

## D-126: durable, rootless vcloud->site-service-VM access -- SSH local-forward via systemd --user (RULED: Option A) [OPS]

**Status:** RULED 2026-07-16 (operator chose Option A: "We have been using SSH proxyjump already so lets
continue that"). Tooling delivered (`scripts/site-forward.sh` + harness, reviewed-but-UNEXERCISED: this
session has no SSH key to voffice1, so the operator supplies `--key` and runs the first connect).

**Problem.** The Office1 service VMs (office1-netbox `10.10.1.10`, and later the DC service VMs) live on
`lxdbr0` (`10.10.1.0/24`), reachable from vcloud only THROUGH voffice1's LXD NAT. vcloud's default route
is the lab, so each session had to re-add `sudo ip route replace 10.10.1.0/24 via 10.10.0.20 dev virbr2`
by hand -- lost on reboot, sudo/TTY friction every time, and it will recur identically for the DCs. The
operator directed a durable, generalizable fix.

**Options weighed.**
- **A (RULED) -- SSH local-forward held by a `systemd --user` unit.** Reuses the transport we ALREADY use
  by hand (ssh through voffice1 / a DC containment VM). Rootless; survives reboot with `enable-linger`;
  `http://127.0.0.1:<port>` on the jumphost reaches the service VM. For office1-netbox the primitive is a
  plain `-L` through voffice1 (the lxdbr0 gateway); the DC form adds a `-J` ProxyJump through the
  containment VM to an inner VM -- so it generalizes by construction.
- **B (rejected) -- persist the L3 route.** Needs root (sudoers), and virbr2 is LIBVIRT-managed, so a
  netplan route races libvirt at boot (would need an `After=libvirtd` unit / net-hook, not a plain
  stanza). Divergent per-DC mechanism, and it keeps the very sudo friction we set out to remove.

**Why A wins.** Zero new mechanisms (same ssh path as the inner tofu root's `qemu+ssh`), rootless (kills
the sudo/TTY friction), and the `-J`-through-containment-VM shape is exactly how the DCs will be reached.
MEASURED substrate: user-systemd is running (`XDG_RUNTIME_DIR=/run/user/1001`), OpenSSH 9.6; `Linger=no`
today so boot-start needs a one-time `loginctl enable-linger jessea123` (printed by `install`, not run).

**Secrets boundary (unchanged by this).** The SSH key is operator-supplied via `--key` (a PATH handed to
`ssh -i`; never read into context, never baked into the tool). The forward carries ONLY read/UI TCP -- the
NetBox API token stays a host-local secret on office1-netbox (`/root/netbox-secrets/api.token`), so the
importers' dry-run AND `--commit` remain operator-run there. `127.0.0.1` bind only (never `0.0.0.0`).
SANDBOX_HOSTS already allows `localhost`/`127.0.0.1`, so an importer pointed at the forwarded port passes
its own upstream-write gate with no code change.

**Deliverables.** `scripts/site-forward.sh` ({render|install|status|remove|list}); site table seeded with
`office1-netbox` only (MEASURED: voffice1 `10.10.0.20` forwards to `10.10.1.10:8000`, HTTP 302) -- DC rows
are DEFERRED until their containment-VM + inner IPs are measured (hard rule 2; the harness fails if an
un-commented DC IP appears). `tests/site-forward/run-tests.sh` 38/38 (pins loopback-bind, no-baked-secret,
rootless, self-locate). `docs/vr1-office1-as-built.md` access table updated (D-126 preferred; sudo route
demoted to ad-hoc fallback).

**AMENDMENT (2026-07-16) -- SHELL layer (operator: "You're going to need access to the shell in Office and
the DCs").** The port-forward buys durable *reachability* to one service port, not a *shell*; the operator
needs interactive shell into the Office AND DC VMs. Same SSH transport, standard form: a generated
`~/.ssh/config.d/vr1-sites` fragment of Host aliases (`voffice1`, `office1-netbox`, `office1-tailscale`,
`office1-opnsense`) with the as-built User + `ProxyJump` baked, so `ssh office1-netbox` just works and the
`-J`-through-the-containment-VM shape generalizes to the DC VMs by construction. This is now the FOUNDATION;
the systemd forward is a convenience that can ride the aliases. Deliverable: `scripts/site-ssh-config.sh`
({render|install|remove|list}) + `tests/site-ssh-config/run-tests.sh` 30/30 (pins the as-built
host/user/ProxyJump mapping, Include idempotency + 0600 perms, no-baked-secret, no un-measured DC row). Same
secrets boundary: the key is operator-supplied via `--key` -> `IdentityFile` (a PATH, never read/baked); the
generated fragment names that path and is written LOCALLY (NOT committed). DC aliases DEFERRED until their
inner IPs are measured post-apply (only voffice1 exists today). Key confirmed one-key closed env:
`~/vr1-office1-creds/office1_svc_ed25519` (named in `creds-manifests/vr1-office1.manifest`), auths all three
users. **EXERCISED 2026-07-16** (operator ran `install`; the agent then proved access from this session):
`ssh voffice1` -> jessea123, `ssh office1-netbox` + `ssh office1-tailscale` -> ubuntu (via ProxyJump voffice1),
`ssh office1-opnsense` -> root (tcsh) all return live. This PROVES the load-bearing assumption -- the agent
can `ssh <alias>` with the key read from the config `IdentityFile` (no creds path in the command), so it
passes the guard hook + classifier. The advisor's acceptance-test steer (exercise -> record -> commit) is
met. The systemd FORWARD layer (`site-forward.sh`) remains reviewed-but-unexercised (not installed).

**PER-ENV KEY for the DCs (operator ruling 2026-07-16: "individual key... for each env access").** Office1
is one env with one key across its three VMs (already conforms: `office1_svc_ed25519` in
`~/vr1-office1-creds/`). Each DC (`vr1-dc0`, `vr1-dc1`) gets a DEDICATED access key in its OWN creds folder
(`~/vr1-dc0-creds/`, `~/vr1-dc1-creds/`) -- NOT office1's key -- for per-env blast-radius isolation (a leaked
key compromises one env, not all). Realization: the DC keys are MINTED into those creds folders at DC deploy
(pubkey wired into the DC's cloud-init), tracked in `creds-manifests/<site>.manifest`, not generated
speculatively before the DC exists. Tool mechanism: `site-ssh-config.sh` installs each DC env as its own
`~/.ssh/config.d/vr1-<dc>` fragment via a per-env `--key` (the `--env` selector is added when the first DC
row lands with MEASURED IPs).

**Applies to the CONTAINMENT VM too (operator ruling 2026-07-16, option a).** Asked whether per-env-key
covers `vvr1-dc0` itself (its cloud-init had installed the OFFICE1 pubkey, and the inner root's `qemu+ssh`
authenticated with it), the operator ruled **(a) dedicated dc0 key** over (b) office1-key-for-the-container
(zero-code). IMPLEMENTED: minted `~/vr1-dc0-creds/vr1-dc0_svc_ed25519` (600/644; `creds-audit vr1-dc0`
CLEAN); added `variable "vr1_dc0_ssh_pubkey_path"`; `vvr1-dc0`'s cloud-init `ssh_authorized_keys` now reads
`file(var.vr1_dc0_ssh_pubkey_path)` (main.tf, the vvr1_dc0 block only -- voffice1 keeps office1's key);
set the tfvar in `d124-rack.auto.tfvars`. The inner root's `qemu+ssh` to `vvr1-dc0` authenticates with the
dc0 PRIVATE half (jumphost-local, never read). Re-validated + re-planned: outer plan unchanged at 5 add /
0 change / 6 destroy (the cloud-init edit is create-only content, no new destroys/replaces).

**Related:** D-107 (Tailscale front door -- the workstation path; D-126 is the vcloud-local path), D-123
(Model B containment VMs -- what the DC `-J` form jumps through), D-124 (transit addressing the DC forwards
will ride), DOCFIX-195 (office1-netbox is the working IPAM apex this reaches). **Revert:** `git checkout`
the as-built rows + delete this entry; `git rm` `scripts/site-forward.sh`, `scripts/site-ssh-config.sh` and
their `tests/` dirs; on any host where `install` ran, `bash scripts/site-forward.sh remove office1-netbox`
(rootless) + optionally `loginctl disable-linger`, and `bash scripts/site-ssh-config.sh remove` (leaves the
Include line + `~/.ssh/config`).

**AMENDMENT (2026-07-17) -- durable BASE L3 leg (the layer this decision assumed).** D-126 made the
transport ABOVE base reachability durable (the ProxyJump aliases + the systemd `--user` forward), but it
ASSUMED the vcloud host could already reach voffice1 at `10.10.0.20`. The vcloud OS-patch+reboot (2026-07-17,
-134 -> -136) exposed that assumption: `office1-local` is an ISOLATED libvirt net (no `<ip>`/`<forward>`),
so the host's own L3 presence -- `10.10.0.10/24` on the bridge (as-built `:41`) plus the compose route
`10.10.1.0/24 via 10.10.0.20` (`:88`) -- lives NOWHERE in the libvirt XML and vanished on reboot, stranding
`ssh voffice1` (measured: `ip route get 10.10.0.20` fell through to the default lab gw; `ping .20` 100% loss).
So the base leg is the un-persisted foundation D-126 never owned. **This NARROWLY reopens the rejected Option
B -- for the BASE LEG ONLY.** The original rejection stood on a netplan-vs-libvirt race; a `Type=oneshot`
unit ordered `After=libvirtd.service` runs AFTER libvirt has created the autostart bridge, so the race does
not exist, and it is root purely because L3-on-a-bridge has no rootless equivalent (the D-126 forward cannot
create it -- it depends on it). **Deliverable:** `scripts/site-baseleg.sh` ({render|apply|check|install|
remove|list}) + `tests/site-baseleg/run-tests.sh` (24/24, offline). It DISCOVERS the bridge from the stable
net NAME (`virsh net-info office1-local`), never a baked `virbrN` (lib-net.sh PATTERN-1); `apply` is
idempotent + root-guarded and is what the boot unit invokes; `check` is the read-only session-verify a new
session runs to confirm the leg without re-deriving it. **Ordering was MEASURED, not inferred:** this host
runs monolithic `libvirtd.service` (active; `virtnetworkd`/`virtqemud` inactive), which owns network
creation, so `After=libvirtd.service` is correct here; the unit ALSO orders after `virtnetworkd.service` so
it stays correct on a modular libvirt host (Ubuntu 24.04 / libvirt 10.x may split the daemons) -- ordering
against an inactive/absent unit is a harmless no-op, and the bridge poll in `apply` is the real guarantee.
**Known limitation:** `Type=oneshot` has no `Restart=`, so the `BRIDGE_WAIT` poll (60s) is the ONLY retry --
if the bridge misses that window at a contended cold boot the leg silently does not come up; widen the wait
if observed. The `ExecStart` is the script's ABSOLUTE path, so a repo move/rename (cf. D-110) breaks the boot
unit -- re-run `install` after any relocation. **State: INSTALLED 2026-07-17; boot-race UNPROVEN.** The
`install` was run (operator sudo): the unit is written + `enabled` (symlinked into `multi-user.target.wants`),
and its `apply` ran once at install time -> `status=0/SUCCESS`, `check office1` -> ok (leg + route present).
BUT that run was POST-boot (bridge already up), so it exercised the `apply` logic, NOT the `After=libvirtd` +
poll race that is the whole point. Boot-time durability stays UNPROVEN until an ACTUAL reboot shows `ssh
voffice1` self-heals with no manual step. The ad-hoc leg from earlier this session and the unit's `apply`
converge on the same live state, so `check` alone cannot distinguish them -- only a reboot does. See D-128
(this is the vcloud-local half of the operating model). **Revert:** `bash scripts/site-baseleg.sh remove
office1` (root) on any host where installed; `git rm scripts/site-baseleg.sh` + its `tests/` dir; delete this
amendment.

## D-127: VR1 host-level VM autostart policy -- foundational service VMs autostart, DC containment VMs manual [OPS]

**Status:** RULED 2026-07-16 (operator: "set the autostart for the VMs so we don't run into problems if
the hosts restart"). Recorded before an OS-update/reboot of the vcloud host; the live `virsh autostart`
settings are applied AFTER a verified clean cold boot (operator sequencing).

**Problem.** All vcloud host-level libvirt domains were `autostart=disable`, so a host reboot leaves the
Office1 VMs down until manually started -- a footgun as the buildout adds restart events. Need a standing
policy for which VMs auto-boot with the host vs which require a deliberate manual bring-up.

**Policy (the rule of thumb).** Autostart = YES for FOUNDATIONAL service VMs with no fragile or gated boot.
MANUAL = the DC CONTAINMENT VMs and anything whose bring-up is a deliberate, gated, resource-heavy, or
ordering-sensitive act.

**Classification (host-level libvirt domains on vcloud):**
- **voffice1 -> AUTOSTART.** Office1 host (MAAS region, LXD host for netbox/tailscale, libvirt). Foundational;
  no boot fragility; must return after any host restart.
- **office1-opnsense -> AUTOSTART, but enable only AFTER one verified post-fix cold boot.** The triple-fault
  (`incident-20260712-opnsense-edge-boot-triplefault.md`) was a FIXED config bug (DOCFIX-188: `memory_unit`
  KiB->MiB starving the bootloader), not live fragility -- the running edge proves the fix; only the cold
  boot post-fix is unverified ("live boot verification is the remaining gated step"). Watch it boot once,
  then `virsh autostart`.
- **vvr1-dc0 (and future vr1-dc1 containment) -> MANUAL.** The 416 GiB/108-vCPU nested-KVM containment VM:
  bring-up is a gated act (depth-4 nested-virt boot gate, D-123 site-down semantics, resource-heavy). Do NOT
  auto-boot on host restart. Already default-off (the `cloudinit-vm` module sets `running=true`, no
  autostart) -- keep it.

**Non-host layers (so "Office1 fully returns" holds):** LXD service VMs INSIDE voffice1 (office1-netbox,
office1-tailscale) set autostart WITHIN voffice1 (`lxc config set <inst> boot.autostart true`); inner node
VMs inside vvr1-dc0 stay MAAS-power-controlled (inner libvirt autostart off); inner OPNsense edge is
manual/gated like the host edge.

**Implementation (two layers):** (1) LIVE, immediate, no guest bounce -- `virsh autostart voffice1` (and
`office1-opnsense` after its verified cold boot); runs as jessea123 via the libvirt group (no sudo). (2)
REPRODUCIBLE (IaC) -- **DONE 2026-07-17** (operator: bake the matrix into the deploy tooling so DC0/DC1 are
configured at installation, not missed). Added an `autostart` bool var (default false) to `modules/cloudinit-vm`,
`modules/opnsense-edge`, `modules/node-vm` (wired to `libvirt_domain.autostart`); set EXPLICITLY per instance
in the roots -- voffice1=true, office1-opnsense=true, vvr1-dc0=FALSE (outer); vr1-dc0-opnsense=true,
node-vm=FALSE (inner). Shared modules mean autostart is managed for EVERY instance, so IaC is now the source
of truth (omitting it would fight the live layer). `tests/opentofu-validate/run-tests.sh` T11-T15 pin the
matrix against drift (auto-booting the 416 GiB container / MAAS nodes is the exact defect guarded). VERIFIED
by `tofu plan`: the two EXISTING Office1 domains (voffice1, office1-opnsense) show `autostart false->true`
**update-in-place** (NOT replacement -- confirmed; replacement would destroy the live host/edge); vvr1-dc0
is a new create with autostart=false baked in. Outer plan is now `5 add / 2 in-place / 6 destroy`, which
collapses back to `0 change` on those two once the bookend has set them autostart=true live before the
fresh-session apply. LXD half: `boot.autostart=true` set live on office1-netbox/tailscale + captured in the
Office1 standup runbook. DC0/DC1 create NO LXD instances (inner root = node-vm + opnsense-edge + planes/pool/
wan-bridge), so the DC side is purely this VM matrix -- no DC-LXD gap.

**Sequencing for a host reboot:** `virsh autostart voffice1` (returns on its own) -> reboot -> manually
`virsh start office1-opnsense` and WATCH it cold-boot (the live-boot verification) -> then `virsh autostart
office1-opnsense`. Bring DC containment VMs up manually per their gated procedure, never on host boot.

**Related:** D-123 (Model B site-down = the reason vvr1-dc0 is manual), DOCFIX-188 (the opnsense mem-unit
fix that makes the edge safe to autostart), `runbooks/ops-restart-procedure.md` (the cloud-side analogue:
`resume-guests-state-on-host-boot`). **Revert:** `virsh autostart --disable <domain>`; drop the IaC
`autostart` args; delete this entry.

## D-128: VR1 operating model -- Claude on the vcloud jumphost; two-plane origin; workstation tailnet is the human path [OPS]

**Status:** RULED 2026-07-17 (operator: "I will go with your recommendation"). Recorded + folded into the
operating skill so no future session re-derives it. Governs WHERE work originates, not any built surface.

**Problem.** The vcloud patch+reboot stranded Office1 access (see D-126 amendment) and raised the operating
question: the operator SSHes into the vcloud host to run Claude, but the operator's WORKSTATION has tailnet
access (D-107) that reaches all of `10.10.0.0/22` directly. Should the session relocate -- run Claude from a
repo on the workstation and tunnel into both vcloud and Office1? We are simulating a deployment FROM Office1
to the DCs, so "sit in Office1" has real pull.

**Decision -- two planes, fixed origins; Claude stays on the jumphost.**
- **Plane 1 (substrate / stagehand):** creating the DC *host* VMs (`vvr1-dc0`), libvirt nets/pools, the
  OUTER `tofu` root (`libvirt_uri = qemu:///system`, `dc-dc-phase0.auto.tfvars:6`) + the ~13 local `virsh`
  calls. Runs ON the vcloud host. **No Roosevelt analog** -- in Roosevelt the DC hardware exists; nobody
  creates it. Throwaway simulation scaffolding.
- **Plane 2 (deployment / management / testing):** MAAS enlist/commission/deploy, the NetBox apex, the INNER
  `tofu` root (`opentofu/vr1-dc0-substrate/`, `qemu+ssh` FROM Office1 into `vvr1-dc0`, R-5), tenant tests.
  This is what we REHEARSE for Roosevelt, and it EXECUTES on `voffice1` (the headend) -- reached by `ssh
  voffice1 '...'` over the D-126 leg, so a runbook step reads "from the Office1 headend", which transfers.
- **Claude's process stays on the vcloud jumphost.** The execution hosts above are fixed BY DESIGN; Claude's
  location only picks the TRANSPORT it uses to reach them, not where work runs.
- **The operator's workstation tailnet is the HUMAN path** (MAAS UI, NetBox UI, browser, an SSH console) --
  orthogonal to where Claude runs. Keep using it; it needs no decision and touches nothing scoped.

**Why not relocate to the workstation (Option B, rejected).** (1) It trades away the standing "ONE jumphost
session reached via Remote Control" design that already solves multi-workstation hopping -- putting the
session ON a workstation ties it to that machine. (2) It relocates the entire gating/secrets contract (the
PreToolUse guard, permission `ask` rules, creds folders, CLAUDE.md) onto a less-controlled host and puts
creds there. (3) It puts the riskiest Plane-1 ops (create/destroy host VMs) over a droppable `qemu+ssh`
transport instead of local libvirt. The Office1-reach cleanliness it would buy is already delivered by the
D-126 base-leg oneshot with none of those costs. Roosevelt fidelity lives in the runbook's DECLARED ORIGIN +
transport (achieved by executing Plane-2 ON voffice1), not in which host Claude's process runs on.

**Durable Office1 reach = the D-126 base-leg oneshot (`scripts/site-baseleg.sh`), NOT tailscale-on-vcloud.**
This is precisely why **D-107 stays untouched**: tailscale is the WORKSTATION path; the jumphost's path is
its own L3 leg. DCs are outside the `/22` carve and keep the D-126 `-J`-through-`vvr1-dc0` shape regardless.

**Related:** D-126 + its 2026-07-17 amendment (the mechanism), D-107 (workstation tailnet), D-123/R-5
(inner-root-from-Office1), `docs/dc-dc-deployment-workflow.md` (Cross-cutting: the plane split as doctrine),
the operating skill (invariant). **Revert:** delete this entry + the skill invariant; the operating model has
no built-surface footprint to unwind.

## D-129: OPNsense edge plugin / add-on base profile (VR1 -> Roosevelt) [OPS]

**Status:** OPEN (PARTIALLY RULED 2026-07-18) -- profile adopted, FOUR sub-decisions still OPEN.
Operator directive: "install the OPNsense recommendations and pin the full engineering for the rest for
the appropriate steps in the upcoming deployment(s)". The
plugin PROFILE is adopted: `os-qemu-guest-agent` + `os-iperf` directed for the VR1 edge via the D-113
firmware API (`core/firmware/install/<pkg>`); the live install on `office1-opnsense` is operator-gated
(auto-mode classifier blocked the agent's POST -- cloud mutations stay gated), verification pending. The
rest is PINNED to its deployment step (mapping below). **FOUR sub-decisions remain OPEN** (a one-line
directive does not rule these -- they were flagged as decisions, not adds): (i) COS-scrapes-the-edge
scope, (ii) `os-frr` dynamic routing, (iii) `os-tailscale` subnet-router consolidation, (iv)
edge-as-site-NTP.

**Pinned at deployment steps (the "engineering for the rest").**
- **VR1 edge base profile = {`os-qemu-guest-agent`, `os-iperf`}** -> applied at DC-edge bring-up.
  PINNED in `runbooks/dc-dc-phase2-tofu-dc-substrate.md` (Model B inner-apply / edge section). The
  reusable per-profile installer (thin wrapper over `scripts/opnsense-api.sh` firmware install,
  idempotent, `--check`) is itself engineering to BUILD at that step -- not built now (avoids
  premature tooling; `opnsense-api.sh` already carries the install this session).
- **Roosevelt metal edge profile = {`os-smart`, `os-nut`|`os-apcupsd`, `os-cpu-microcode-intel`/`-amd`,
  `os-lldpd`}** -> recorded here + in the review doc; enable at the Roosevelt edge build (inert on VR1 VMs).
- **Observability (`os-node_exporter`)** -> PINNED in `runbooks/dc-dc-phase6-designate-cos-magnum.md`
  Step 10, gated on the COS-scrapes-edge scope sub-decision (i).
- **Security hardening** (Suricata content, crowdsec, Zenarmor, acme, SIEM) -> DEFERRED per closed-test
  posture to post-teardown / Roosevelt; recorded in the review doc, not pinned to a VR1 step.

**Verification 2026-07-18 (install attempt on `office1-opnsense`, operator-run via firmware API).**
- `os-qemu-guest-agent`: **package INSTALLED but NON-FUNCTIONAL.** Daemon not running/enabled AND the
  libvirt domain has **no `org.qemu.guest_agent.0` virtio-serial channel** (`opnsense-edge` module defines
  none -- grep-confirmed), so `virsh ... guest-ping` returns "QEMU guest agent is not configured".
  **Making qga work is therefore NOT a plugin install -- it is: (1) inject the guest-agent channel into
  the domain XML in `modules/opnsense-edge` (provider XSLT-or-equivalent -- verify the exact dmacvicar
  mechanism when building; do NOT assume a `qemu_agent` arg exists); (2) enable the qga service on the
  edge; (3) restart the edge.** NEW DC edges get (1) at creation (born with the channel, no restart);
  `office1-opnsense` retrofits at its NEXT SCHEDULED restart. Functional enablement DEFERRED accordingly.
- `os-iperf`: **first install FAILED** -- two back-to-back `core/firmware/install` POSTs collided on the
  OPNsense firmware LOCK (the 2nd ran while the 1st held it). **Lesson for the profile-installer: install
  ONE plugin at a time and POLL `upgradestatus` to completion before the next.** Re-run of `os-iperf`
  pending (still classifier-gated for the agent -- operator-run or a Bash permission rule).

**Implementation prep (autonomous 2026-07-18; NOTHING applied to the cloud).**
- **Profile-installer BUILT: `scripts/opnsense-plugins.sh` + `tests/opnsense-plugins/` (17/17, gauntlet
  68 GREEN).** `list` + `apply <vr1-edge|metal-edge>`; installs ONE plugin at a time + polls
  `upgradestatus` (encodes the firmware-lock lesson), delegates auth to `opnsense-api.sh`, warns on the qga
  channel dependency, offline `--dry-run`. This CLOSES the "build the reusable installer" pin; live `apply`
  is operator-gated (firmware POST). Runbook pin updated to USE it.
- **qga libvirt channel (the functional gap) -- design, NOT applied.** Add to `modules/opnsense-edge` as an
  **OPT-IN variable defaulting to OFF** (e.g. `enable_qemu_agent = false`) so EXISTING edges get NO plan
  change -- CRITICAL: a default-on edit to the shared module would force a change/bounce on the LIVE
  `office1-opnsense` and dirty the imminent DC0 outer plan (5/0/6). Set true only for NEW DC-edge
  instantiations (born with the channel, no retrofit); `office1-opnsense` opts in + restarts on its own
  schedule. **MECHANISM TO VERIFY before writing (not in `references/opentofu-provider-docs.md` -- do NOT
  assume):** whether dmacvicar/libvirt **0.9.8** exposes a native `qemu_agent = true` domain arg or needs an
  `xml { xslt = ... }` channel injection. Then enable the service + restart.
- **`os-git-backup` wiring (Finding #1) NEEDS OPERATOR INPUT -- cannot be done autonomously:** (a) choose a
  target git repo (a repo on `git.baldurkeep.com` for edge-config backups); (b) the plugin generates an SSH
  keypair -- add its PUBLIC key as a DEPLOY KEY on that repo; (c) set branch + schedule + enable. Steps
  (a)/(b) are operator/GitBucket actions; the write itself is also a gated edge mutation.

**Problem.** `office1-opnsense` was updated to OPNsense 26.7 and is a near-stock install (only
`os-git-backup`, itself unconfigured). The edge is the reusable per-site boundary profile that will
land on Roosevelt baremetal, so "what should be on the base edge?" is a real, transferable question.
Full catalog review: `docs/opnsense-edge-addon-review-20260718.md` (104 plugins, all categorized).

**Recommendation (for the operator to rule, bucket by bucket).**
- **Roosevelt metal edge profile (record now, inert in VR1 VMs):** `os-smart`, `os-nut` OR
  `os-apcupsd`, `os-cpu-microcode-intel`/`-amd`, `os-lldpd`. These are the genuine base-deployment
  items; they only run on baremetal.
- **VR1 sim scaffolding (add on VR1 VM edges now; uninstall at metal, like netem):**
  `os-qemu-guest-agent` -- clean libvirt-guest coordination; aids the D-126/D-127 reboot work.
- **Buildout diagnostics:** `os-iperf` on edges to validate DC-DC transit throughput (D-122/D-124).
- **Observability -- CONDITIONAL:** `os-node_exporter` (Prometheus-native, matches the COS stack) IF
  the operator rules the boundary edge in-scope for COS scraping. D-105 scoped COS per-DC INSIDE the
  cloud; the edge is outside that failure domain -- so this is a scope decision, not an automatic add.
- **Defer per closed-test posture (Roosevelt / post-teardown):** Suricata (core) + IDS content
  plugins, `os-crowdsec`, `os-sunnyvalley` (Zenarmor), `os-maltrail`, `os-acme-client`, SIEM agents.

**Open decision points.** (1) which buckets to adopt; (2) COS-scrapes-the-edge scope (D-105); (3)
`os-frr` -- dynamic routing is unneeded for VR1's static transit, a Roosevelt-scale question; (4)
`os-tailscale` -- consolidate the separate subnet-router VM onto the edge vs keep D-107 as-is; (5)
whether the edge anchors site NTP. Items 3-5 are their own sub-rulings.

**Adjacent finding (not part of the ruling):** `os-git-backup` is installed but UNCONFIGURED -- no
config backup is running. Recommend wiring it to `git.baldurkeep.com` (config-as-code) regardless of
D-129's outcome; tracked in the review doc's Findings.

**Implementation path (once ruled).** Bake the chosen plugin set into the edge bootstrap over the
REST-API path (D-113) via the OPNsense firmware API (`/api/core/firmware/install`), with the touched
script's harness + a changelog entry with a revert. VR1 vs Roosevelt sets differ (sim scaffolding is
VR1-only; the hardware profile is metal-only).

**Related:** D-100/D-122 (edge = boundary role), D-113 (REST-API config), D-105 (COS scope), D-106
(Designate owns DNS), D-107 (Tailscale subnet-router VM), D-124/D-125 (static transit), D-127
(autostart), D-126 (base-leg). **Revert:** delete this entry + the review doc; nothing was installed.

### D-129 -- AMENDMENT (2026-07-19): class OPS; ARCH fragment split out (GA-R3/A1)

One-time-review ruling (operator, 2026-07-19): D-129 is tagged **[OPS]** (dominant purpose = the
VR1 plugin install + pins). Its ARCH fragment -- the Roosevelt metal-edge transfer profile -- is
SPLIT to a dated AMENDMENT on D-122 (the governing site/edge-shape ARCH decision), which is now
the governing copy; the text above stays as history (append-only register). The four OPEN
sub-decisions (i)-(iv) are unaffected and remain OPEN on this entry.

## D-130: cloudinit seed-volume durable fix -- ignore_changes on the staging-derived create [ARCH]

**Status:** ADOPTED 2026-07-19 (operator; GA-R5 -- question + utterance below; sweep Batch 1 item 3).

**Question as presented (2026-07-19):** "D-130 -- durable mechanism for the cloudinit seed-volume
forced replacement (recurs after every host reboot; affects voffice1 + vvr1-dc0 + vvr1-dc1). v8
CONFIRMED your contingent lean (a): ignore_changes fully suppresses the replacement (7/2/7 -> 6/2/6,
zero forces-replacement). v7 additionally showed the replace itself would not bounce a running guest.
Which mechanism do you adopt? (a) ignore_changes / (b) persistent TMPDIR staging / neither-defer."
**Operator answer (2026-07-19), exact utterance:** "(a) ignore_changes (Recommended)".

**Ruling:** `modules/cloudinit-vm`'s `libvirt_volume.seed` carries
`lifecycle { ignore_changes = [create] }`. The provider's staging ISO path (Go os.TempDir-derived,
no provider knob -- verified via `tofu providers schema -json`) is re-minted after every host
reboot; without the lifecycle guard that config-derived `create.content.url` change forces
replacement of a correct, in-use volume on every post-reboot plan (GA-F01; mechanism verified
against provider v0.9.8 source in docs/finding-20260718-voffice1-cloudinit-seed-replace.md).

**Evidence (captured, quoted per charter 1a):**
- G4-v8 `docs/audit/outer-plan-20260719-v8-ignorechanges.txt`: with the edit,
  "Plan: 6 to add, 2 to change, 6 to destroy." and ZERO forces-replacement lines
  (baseline same day, `outer-plan-20260719-baseline.txt`: "Plan: 7 to add, 2 to change,
  7 to destroy." -- identical to the 2026-07-18 capture, no drift). DC0's own seed volume
  still plans as a clean create -- the build path is unaffected.
- G4-v7 `docs/audit/throwaway-v7-20260719.txt`: defect reproduced on an isolated
  `throwaway-v7-*` domain (own scratch pool, zero live contact); the replacement applied
  UNDER the running domain completed rc=0 with NO bounce ("domid before: 3" / "domid after: 3",
  domstate running); teardown verified zero residue. Corollary recorded: the source-verified
  bounce path is the autostart/domain Update path, NOT the volume replace -- the churn's real
  hazard is plan noise plus an unscoped apply carrying the autostart diffs (which DO bounce).

**Rejected:** (b) persistent `$TMPDIR` staging + env guard -- environment-dependent, fragile
across operators/hosts; not needed on the v8 evidence. Defer/no-fix -- leaves every post-reboot
plan carrying a replace pair and keeps G7's plan-matches-evidence gate noisy.

**Scope:** all consumers of `modules/cloudinit-vm` (voffice1, vvr1-dc0, later vvr1-dc1).
**Delivery:** module edit + `tests/cloudinit-vm/run-tests.sh` harness (static lifecycle-guard
assertions; `tofu validate` when the binary is present). **Revert:** remove the lifecycle block
and the harness; the replace pair returns on the next post-reboot plan.
