# Design Decisions — VR0 DC0 Omega Cloud

This document is the architectural record for the VR0 DC0 testcloud rebuild. Every decision listed here has been deliberately made and discussed; it is not a wishlist or a brainstorm. If a decision is changed, this document is updated and the change is committed with a referencing message.

**Scope split:** This repository implements **v1 (IPv4-only)**. Several decisions below are tagged with **[v2-scope]** — they remain valid design intent but are deferred to a future v2 deployment when upstream router infrastructure supports IPv6. See **D-015** for the v1/v2 fork record.

**Class tags (GA-R3, ratified 2026-07-18; one-time review ruled 2026-07-19).** Every register
entry carries a one-word class tag in its header: **[ARCH]** (architectural consequence beyond
the current stage, a Roosevelt-delta implication per the A1 test, or supersession of an existing
D-number) or **[OPS]** (operational choice recorded here historically; new OPS items get runbook/
as-built entries, NEVER a new D-number -- GA-R3 rule 2). New entries are ARCH by construction;
doubt resolves DOWN to OPS. Entries D-001..D-120 predate the rule and are tagged opportunistically,
not retroactively swept.

---


## D-001: Deployment target paradigm

**Decision:** Path 2A — Charmed OpenStack Caracal (2024.1) via Juju bundle.

**Alternatives considered:**

- Path 2B — Canonical Sunbeam (microk8s-based). Rejected: discards most of the test-cloud experience accumulated to date; different operator paradigm.
- Path 1 — Stay on Bobcat 2023.2. Rejected: defeats the purpose of a Caracal rehearsal ahead of Roosevelt bare-metal.


**Consequences:**

- Bundle-based deployment, suitable for both KVM testcloud and bare-metal scale.
- Caracal-stable channel matrix applies (see D-002).
- EOL date is April 2027 (Caracal upstream support window).

---


## D-002: Channel pinning matrix

**Decision:** Pin every charm to a Caracal-stable channel. No OVN pinning on testcloud (Roosevelt will pin via `ovn-source`).

| Charm group                                                                                                             | Channel                    |
| ----------------------------------------------------------------------------------------------------------------------- | -------------------------- |
| OpenStack core (keystone, glance, nova-\*, neutron-api, cinder, placement, octavia, barbican, designate, magnum, vault) | `2024.1/stable`            |
| OVN (ovn-central, ovn-chassis, ovn-dedicated-chassis-octavia)                                                           | `24.03/stable`             |
| Ceph (ceph-mon, ceph-osd, ceph-radosgw if used)                                                                         | `squid/stable` (see D-005) |
| MySQL (mysql-innodb-cluster, mysql-router subordinates)                                                                 | `8.0/stable`               |
| RabbitMQ                                                                                                                | `3.9/stable`               |
| Vault                                                                                                                   | `1.8/stable`               |
| hacluster                                                                                                               | `2.4/stable`               |
| etcd, easyrsa                                                                                                           | `latest/stable`            |


**Verification source:** Caracal channel matrix per Canonical Charmed OpenStack Charm Delivery table, verified against Charmhub 2026-05-22 (keystone rev 778, ovn-central rev 332, hacluster `2.4/stable`). Canonical advises against `latest/stable` for OS-stack charms due to non-determinism on charm upgrades; hacluster is therefore pinned to its stable channel. Verify against Charmhub before each deploy via `scripts/pre-flight-checks.sh`.

---


## D-003: Network architecture — Option B

**Decision:** Provider network carries BOTH ext_net (tenant FIPs + SNAT egress) AND OpenStack public API VIPs on the same L2 segment.

**Rationale:** During Magnum CAPI Phase 3 on the Bobcat testcloud, OCCM crashloop was traced to tenant networks being unable to reach OpenStack API endpoints — the libvirt FORWARD chain rejected cross-bridge packets between provider (virbr1) and metal (virbr2) bridges. With API VIPs on metal, tenant workloads cannot reach them. Putting API VIPs on the same network as the FIPs makes the API path tenant-reachable by construction.

**Address space layout for v1 (IPv4-only):**

| Range                       | Purpose                                                  |
| --------------------------- | -------------------------------------------------------- |
| `10.12.4.10 – 10.12.4.223`  | Neutron FIP pool                                         |
| `10.12.4.224 – 10.12.4.254` | Charm API VIPs (excluded from Neutron allocation\_pools) |


The Provider `/22` (`10.12.4.0/22`) carries both ranges within a single Neutron subnet. Neutron `allocation_pools` MUST exclude the API VIP range.

**v2-scope extension:** IPv6 Provider subnet adds parallel FIP and API VIP IPv6 IP Ranges within a single `/64`. See D-004.

---


## D-004 [v2-scope]: Dual-stack vs IPv6-only matrix

**Decision (v2-scope, NOT for v1):** Network role determines address family. IPv6 preferred; IPv6-only where the network has no external clients.

**v1 reality:** All networks are IPv4-only on the existing MAAS-provisioned layout. This matrix becomes active in v2.

| Role                       | IPv4 (v1)     | IPv4 (v2) | IPv6 (v2) | Reasoning                                           |
| -------------------------- | ------------- | --------- | --------- | --------------------------------------------------- |
| Metal                      | ✓             | ✓         | ✓         | Charm-to-charm; MAAS PXE IPv4-first                 |
| Provider                   | ✓             | ✓         | ✓         | Tenant FIPs need IPv4; API VIPs reachable from both |
| Data (Geneve underlay)     | ✓             | —         | ✓         | v2: no external clients; underlay agnostic          |
| Storage (Ceph public)      | ✓             | —         | ✓         | v2: `ms-bind-ipv6: true`; no external clients       |
| Replication (Ceph cluster) | ✓             | —         | ✓         | v2: internal OSD↔OSD only                           |
| LBaaS Management           | ✓             | ✓         | ✓         | Amphora image compatibility                         |
| OOB                        | n/a           | n/a       | n/a       | Bare-metal-only concern                             |
| OpenStack Tenant pool      | ✓ (v1: D-016) | —         | ✓         | v1 IPv4 hybrid; v2 IPv6 modeled                     |

---


## D-004a [v2-scope]: Host management → Metal

**Decision (v2-scope, NOT for v1):** Under v2, openstack0-3 host management IPs move from storage (`10.12.16.40-.43`) to Metal (`10.12.8.0/22`) when Storage becomes IPv6-only. v1 keeps host management on storage.

---


## D-005: Ceph release

**Decision:** Squid (Ceph 19, released October 2024).

**Rationale:** Matches Caracal default; one fewer source override in bundle; rehearses what Roosevelt will run. If Squid has rough edges, the testcloud is the place to find them, not production.

**Alternatives considered:**

- Reef (Ceph 18) — current on Bobcat testcloud; lower risk; would require `source: cloud:jammy-caracal` override on ceph-mon/ceph-osd while keeping `reef/stable` channel. Rejected: defeats the rehearsal purpose.

---


## D-006: Vault HA backend

**Decision:** etcd + easyrsa, per Canonical Charmed Vault HA docs.

**Rationale:** This is the documented charm path. The chicken-and-egg TLS dependency (Vault needs certs to start, but Vault issues certs) is resolved by easyrsa bootstrapping the etcd cluster's TLS, after which Vault relations to etcd come up cleanly.

**Topology on testcloud (v1):** Vault num_units=1 + hacluster relation (decorative; documents the relation pattern). Vault HA quorum is not actually exercised at testcloud scale.

**Topology on Roosevelt:** Vault num_units=3 + hacluster on metal space; etcd num_units=3; easyrsa num_units=1.

---


## D-007: Magnum inclusion (Layer A current; Layer B mechanism/topology superseded -- see D-035 / D-037 / D-042)

**Decision:** Magnum in bundle from day one. Two-layer install.

**Layer A — Bundle:**

- `magnum` charm
- `magnum-mysql-router` subordinate
- `magnum-dashboard` subordinate
- Standard relations: keystone, mysql-innodb-cluster (via router), rabbitmq-server, vault (certificates), openstack-dashboard
- Binding: `public: provider` with VIP on provider API VIP range
- Hacluster relation included (decorative on testcloud)


**Layer B — Post-deploy runbook (`runbooks/05-magnum-capi-driver.md`):**

- `juju run magnum/leader domain-setup --wait=10m`
- pip install `magnum-capi-helm==1.1.0` from PyPI into the magnum charm venv with `--break-system-packages` (stackhpc/magnum-capi-helm fork archived Dec 2024; canonical project moved to `openstack/magnum-capi-helm` on opendev/PyPI; 1.1.0 is the last Caracal-cycle release. Upstream tests against Magnum 2023.1+, so backward-compatible through Caracal 2024.1.)
- Deploy `/etc/magnum/kubeconfig` pointing at the **workload cluster** (the post-pivot home of CAPI controllers per **runbook 04a §17** `clusterctl move`). Staged on jumphost at `$HOME/magnum-capi/capi-mgmt-cluster.kubeconfig` by runbook 04a §19, transferred to the magnum unit by runbook 05 §6. Bobcat had this pointing at bootstrap k3s because the pivot was never executed; workstream 3b (2026-05-22) made the pivot mandatory.
- Systemd override replacing init.d ExecStart to load `--config-dir`
- `/etc/magnum/magnum.conf.d/99-capi.conf` setting `enabled_drivers=k8s_capi_helm_v1` and `[capi_helm] kubeconfig_file=/etc/magnum/kubeconfig` (ASCII-only; non-ASCII characters in conf.d cause silent daemon failures)


**CAPI mgmt plane:** Post-pivot, the workload cluster IS the CAPI management plane (per **runbook 04a §17**, `clusterctl move` pivots cluster state from the `capi-mgmt.maas` bootstrap k3s into the workload cluster, which becomes self-managing). Per **D-017**, both the bootstrap k3s and the workload cluster are rebuilt from scratch every deployment cycle — there is no preserved-across-rebuild artifact. The bootstrap install + pivot procedure lives in `runbooks/04a-capi-bootstrap-cluster.md` and runs **before** this runbook. This pattern transfers to Roosevelt unchanged.

**Superseded portions:** The "preserved across rebuild" stance in earlier drafts of this decision is **superseded by D-017**. See D-017 for rationale. The earlier `stackhpc/magnum-capi-helm` v0.13.0 driver pin is superseded by the `openstack/magnum-capi-helm` 1.1.0 pin above (driver source repo moved + archived). The Layer B *mechanism and topology* are now further superseded: the CAPI management plane is an in-cloud single-homed VM with NO `clusterctl move` -- the kubeconfig points at that mgmt cluster, not a post-pivot workload cluster (**D-035**); the conductor graft is `/etc/default/magnum-conductor` `DAEMON_ARGS --config-dir` + `00-capi-helm.conf`, NOT a systemd-ExecStart override + `99-capi.conf` (**D-037**); the driver pin is **1.4.0** for CAPI-core contract coherence (**D-042**); and the per-cluster app-cred replaces any static `capo` credential (**D-039**). Layer A (the bundle) is current. Live deploy steps are runbooks/phase-06 (mgmt cluster) and phase-07 (conductor graft + `domain-setup` D-046 + keystone-v3 drop-in D-047); `runbooks/05-magnum-capi-driver.md` is historical.

---


## D-008: DNS architecture

**Decision:** Layered — static /etc/hosts for bootstrap + Designate (in bundle from day one) for tenant-level resolution.

**Naming convention:**

```
<service>.<cloud>.<dc>.<region>.cloud.neumatrix.local
```
Examples:

- `keystone.omega.dc0.vr0.cloud.neumatrix.local`
- `nova.omega.dc0.vr0.cloud.neumatrix.local`


**Bootstrap order:**

1. Static `/etc/hosts` on jumphost + all openstack0-3 hosts + all LXD containers
2. Bundle deploys with `os-public-hostname: <fqdn>` per API charm
3. Vault issues certs with FQDN in SAN
4. Post-deploy: Designate zone created, A records populated (v1: A records only; v2 adds AAAA records)
5. Neutron `default_dns_domain` and `dns_servers` configured to point at Designate
6. Tenant subnets created with `--dns-nameserver <designate-vip>`

---


## D-009: Hacluster modeling at testcloud scale

**Decision:** Include hacluster + VIP relations at num_units=1 across all HA-eligible API charms.

**Rationale:** Decorative at testcloud scale (a single unit can't form a real HA quorum). Documents the relation pattern so Roosevelt scale-up is mechanical: change `num_units: 1` → `num_units: 3` and rerun.

**Charms with hacluster relation:** keystone, glance, neutron-api, nova-cloud-controller, placement, openstack-dashboard, cinder, octavia, barbican, magnum, vault, designate.

---


## D-010: NetBox-upstream policy

**Decision:** NetBox is the single source of truth for IPAM at the **role and cloud-level pool** layer. Per-project tenant subnets are exempt under the hybrid model (D-016).

**Note (DOCFIX-195, operator ruling 2026-07-15 -- VR1 phase):** During VR1 `office1-netbox` (10.10.1.10) HOLDS THE IPAM APEX ROLE: it is the authoritative NetBox every consuming system (OpenTofu, MAAS, overlays, the Juju bundle) derives its values from. It is a *working draft* (edited as the buildout proceeds), but it is the apex. `netbox.baldurkeep.com` is the eventual PRODUCTION apex, held during VR1 as a READ-ONLY v1 reference draft that takes NO write and is NOT consulted by VR1 systems. Merging the validated `office1-netbox` dataset back into `netbox.baldurkeep.com` is DEFERRED to end-of-deployment (a maybe). This DEFERS the apex write; it does NOT revoke D-010 -- `netbox.baldurkeep.com` remains the single eventual source of truth. See `docs/dc-dc-netbox-buildout-scope.md` section 8 and close-out C2 in `docs/dc-dc-deployment-workflow.md`.

**Workflow:** Update NetBox → update bundle/overlay → commit both with cross-reference.

**Standing imports for v1 (gating the bundle):**

- VR0 DC0 site exists in NetBox ✓
- IPv4 prefixes for v1: Metal /22, Provider /22, LBaaS Mgmt /22 (via `netbox/ipv4-prefixes-import.py`) — **pending**
- Provider IP Ranges for FIPs and API VIPs (same script) — **pending**
- IPv4 tenant pool /16 (same script, per D-016) — **pending**
- IPv6 entries marked as Reservation status (via `netbox/ipv6-mark-reserved.py`) — **pending**


**Deferred to v2 (per Q2):** VR0 DC0-VLANs group additions beyond VID 240 (already imported during prior session work). MAAS currently uses untagged-per-fabric; modeling additional VLANs in NetBox without corresponding network-side tagging would be misleading documentation.

---


## D-011: Validation bar — Roosevelt-rehearsal level

**Decision:** Deployment is not considered successful until all of the following pass:

1. All charms `active/idle` in `juju status`
2. API reachability from jumphost (all public VIPs respond on hostname)
3. API reachability from a tenant VM (Option B verification)
4. Octavia LB pattern re-passes (round-robin, failover, recovery — per Bobcat v3 work)
5. End-to-end Magnum CAPI cluster creation succeeds, including OCCM not crash-looping
6. Vault unseal + auto-unseal-after-reboot pattern verified
7. KVM snapshot baseline taken (Phase 5)
8. Designate zones populated and tenant VMs resolve API hostnames


Validation script: `scripts/validate.sh` (TBD).

---


## D-012: Snapshot strategy

**Decision:** Two baseline snapshots.

- **Snapshot 1:** Post-deploy, post-validation, pre-tenant-resources. Clean cloud state — what a fresh install looks like.
- **Snapshot 2:** Post-tenant-setup. Includes domain1, project1, user1, openrc, flavors, base images (noble-amd64), keypair. Restore point for tenant work.


Snapshots are KVM/qcow2-level on the jumphost hypervisor. Per-VM.

---


## D-013: ~~Clean teardown of existing capi-mgmt~~ (SUPERSEDED by D-018)

**Original decision:** Before destroying the OpenStack model, gracefully delete the CAPI workload cluster on capi-mgmt.maas to allow OpenStack resources (LBs, FIPs, volumes) to be cleaned up properly by CAPI controllers.

**Original steps:** `kubectl delete cluster capi-mgmt-cluster` → wait for CAPI to clean up tenant-side OpenStack resources → `juju destroy-model openstack --destroy-storage --no-prompt`.

**Original "preserved across rebuild" claim:** capi-mgmt.maas bootstrap k3s + CAPI controllers re-used as the Magnum CAPI mgmt plane post-deploy.

**Status:** Superseded. See **D-018** for the replacement teardown strategy (MAAS-release-direct, skip graceful) and **D-017** for the replacement bootstrap cluster lifecycle (full rebuild every cycle, nothing preserved).

---


## D-014: Repository storage location and naming

**Decision:** Self-hosted GitBucket at `git.baldurkeep.com`.

**Repo path:** `jesse.austin/openstack-caracal-ipv4` (v1; IPv4-only).

**v2 repository:** TBD when v2 work begins. Two viable paths: sibling repo `openstack-caracal-ipv6` or `openstack-caracal-dualstack`, OR `v2` branch in this repo with an `overlays/v2-dualstack.yaml`. The single-repo-with-branch approach preserves history of what changed v1→v2 together; the sibling-repo approach keeps v1 frozen as a reference once v2 is in motion.

**Branching strategy:** `main` is canonical. Per-phase work in feature branches when a deploy is in progress; merge back to `main` at successful validation.

---


## D-015: v1 / v2 Fork

**Decision:** Caracal testcloud ships in two iterations.

**v1 (this repository, `openstack-caracal-ipv4`):** IPv4-only Caracal on existing MAAS-provisioned network layout. Proves the bundle, Option B binding fix, Magnum CAPI graft, Designate-from-day-one, hacluster relation pattern, and validation framework. Ships first.

**v2 (deferred):** Adds IPv6 / dual-stack per D-004. Requires upstream router infrastructure to be IPv6-capable, which is not currently the case in this environment. v2 work begins after v1 validation passes AND router-side IPv6 is in place.

**Rationale:** Decoupling the OpenStack-side rebuild from the network-side IPv6 readiness lets us prove the more-important architectural fix (Option B) without waiting on infrastructure work outside the OpenStack deployment's control. The IPv6 design intent is preserved as NetBox Reservation-status entries (per D-010 and `netbox/ipv6-mark-reserved.py`).

**v1→v2 migration scope (forward-look):**

- Re-IP roles per D-004 (add IPv6 sibling to Metal/Provider/LBaaS; move Data/Storage/Replication to IPv6-only)
- Move host management IPs from storage to Metal (D-004a)
- Re-bind charms to listen on both families where dual-stack
- Add AAAA records to Designate zones
- Add tenant IPv6 pool carve-outs

---


## D-016: IPv4 tenant pool — hybrid model (v1)

**Decision:** NetBox owns one upstream IPv4 tenant pool prefix for VR0 DC0. Per-project tenant subnets are Neutron-managed within that pool and are NOT modeled in NetBox.

**Pool allocation:** `10.20.0.0/16` (default; configurable in `netbox/ipv4-prefixes-import.py`). 65,536 addresses; 256 `/24`s available for per-project tenant subnets. Modeled under VR0 DC0 with role `openstack-tenant`.

**Per-project allocation pattern (operationally):**

When a project is created, allocate a /24 from the pool. Operator records the allocation in tenant-setup runbook output but does NOT create a NetBox prefix entry for it. Suggested convention: `10.20.<project-index>.0/24`, starting with `10.20.1.0/24` for project1, etc.

**Rationale (Option C from the discussion):**

- Option A (NetBox-modeled per-project) — full IPAM rigor; high friction for tenant lifecycle; round-trips to NetBox for ephemeral tenants.
- Option B (Neutron-only, no NetBox standing) — minimum friction; loses upstream visibility of total tenant footprint; violates D-010 in spirit.
- Option C (hybrid, chosen) — NetBox documents what space is reserved for tenants and prevents accidental collision with infra ranges; Neutron owns the lifecycle of individual tenant subnets without NetBox round-trips.


**Constraint:** Tenant CIDRs MUST be within the pool. The pre-flight checklist (`scripts/pre-flight-checks.sh`) should assert that proposed tenant subnets fall within the modeled pool.

**v2-scope counterpart:** IPv6 tenant pool `2602:f3e2:ff:0::/56` (NetBox-modeled, Reservation status in v1) becomes active in v2 with the same hybrid model — pool has NetBox standing, per-project IPv6 subnets Neutron-managed.

---


## D-017: CAPI bootstrap cluster lifecycle (bootstrap mechanism superseded by D-035; full-rebuild principle retained)

**Decision:** L3 full teardown and rebuild every deployment cycle. The `capi-mgmt.maas` MAAS VM is released back to Ready state on teardown; on rebuild, it is re-deployed from scratch with Ubuntu 24.04, k3s, CAPI controllers, and ORC. **Nothing is preserved across cycles.**

**Rationale:**

- Rehearsal-first principle. If the bootstrap-cluster install procedure isn't documented and rehearsed, the runbook doesn't exist; if the runbook doesn't exist, surprises surface on Roosevelt.
- Self-imposed forcing function. Every rebuild exercises the full path: MAAS deploy → Ubuntu cloud-init → Vault CA install → k3s install with correct bind-address/SAN flags → kubeconfig server-URL rewrite → helm + clusterctl install → clusterctl init with canonical-kubernetes provider URLs → ORC install → cloud-side prep → cluster manifest render → apply → poll-to-Ready → kubeconfig copy.
- Disposability test. The Bobcat experience proved no critical state lives on capi-mgmt that isn't reproducible from the runbook and the OpenStack cloud. Wiping is safe.


**Runbook:** `runbooks/04a-capi-bootstrap-cluster.md` documents the install sequence in full. It runs **after** `02-deploy.md` (OpenStack cloud up) and **before** `05-magnum-capi-driver.md` (driver graft, which needs the bootstrap k3s kubeconfig).

**Supersedes:** the "preserved across rebuild" stance in earlier drafts of D-007 and D-013.

**Superseded by:** **D-035** retired the bootstrap MECHANISM -- there is no `capi-mgmt.maas` MAAS VM, no k3s, and no `clusterctl move`/pivot; the management plane is an in-cloud single-homed tenant VM (`capi-mgmt-v2`, k8s-snap) built in runbooks/phase-06. The full-teardown-and-rebuild-every-cycle PRINCIPLE stated here is RETAINED (now realized by phase-00 teardown + the phase-06 rebuild). `runbooks/04a-capi-bootstrap-cluster.md` and `05-magnum-capi-driver.md` are historical (folded into phase-06/07).

**Alternatives considered:**

- L1: Wipe just the cluster CRs, keep k3s + controllers. Rejected: skips the install rehearsal that's the whole point.
- L2: Wipe just the controllers, keep k3s. Rejected: same reason; the `clusterctl init` step is exactly the surface that needs rehearsing.
- L3 (chosen): Full wipe including the VM.

---


## D-018: Teardown strategy — skip graceful, release MAAS directly

**Decision:** On teardown, do not pursue graceful CAPI workload deletion or graceful OpenStack model destroy. Instead:

1. (Optional) Capture pre-destroy state for reference
2. `juju destroy-model openstack --force --no-wait --destroy-storage --no-prompt` (background)
3. MAAS release all 5 VMs (openstack0, openstack1, openstack2, openstack3, capi-mgmt) → Ready (parallel)
4. Verify both sides


**Rationale:**

- The rebuild's goal is rehearsing the Roosevelt deploy path. Roosevelt starts from MAAS-Ready bare-metal machines. The most faithful rehearsal is teardown-to-MAAS-Ready.
- Graceful CAPI workload teardown rehearses a different procedure (production cluster decommissioning) that doesn't transfer to Roosevelt's initial deploy.
- `juju destroy-model --destroy-storage` can hang on stuck hooks and leave partial state. `--force --no-wait` plus MAAS release is more reliable.
- Cloud-side OpenStack data (Keystone projects, Neutron networks, Glance images, app credentials) lives in MySQL on the openstack0-3 hosts. MAAS release wipes those hosts, so no separate cloud-side cleanup is needed.


**What is lost vs. graceful path:** verified-clean release path for CAPI workload resources (Octavia LBs, FIPs, CAPO-managed networks). All of these are destined for obliteration anyway; the loss is theoretical.

**What is gained:** ~30+ minutes saved; cleaner end-state guarantee; better Roosevelt rehearsal fidelity.

**Supersedes:** D-013.

**Runbook:** `runbooks/01-destroy-model.md` documents the four phases.

---


## D-019: Cloud DNS (Designate) deferred to v2 / Roosevelt

**Decision:** v1 ships with NO cloud-internal DNS; Designate is not deployed. Public service endpoints use FQDNs (`os-public-hostname`) that resolve to the provider VIPs via external/corporate DNS; internal and admin endpoints stay IP-based on the metal VIPs. Tenant instances use upstream resolvers (1.1.1.1 / 1.0.0.1). The D-011 acceptance bar is amended to drop the cloud-DNS criterion, and the planned `v1-do-doc-10-dns` runbook is dropped.

**Consequence (documented, not a blocker):** metal-only charm units that make catalog-based client calls pull the PUBLIC (FQDN) endpoint and cannot resolve or route it (the internal-endpoint certs carry no FQDN SAN). This is the root of the gss/retrofit amphora-pipeline constraint recorded in D-021. The proper fix (cloud-internal DNS + FQDN-valid certs, or charms consuming internal endpoints) is a Roosevelt item.

**Status:** Decided (v1). Reconstructed into this doc from the deploy record (no standalone D-019 file).

**Related:** D-008 (DNS architecture), D-021 (amphora-pipeline consequence), D-011 (acceptance bar amended).

---

## D-020: Dual provider + metal API VIPs on clustered charms

**Decision:** Every clustered OpenStack API application (keystone, glance, nova-cloud-controller, neutron-api, cinder, placement, barbican, octavia, openstack-dashboard, magnum, vault) is configured with BOTH a provider VIP and a metal VIP, as a space-separated pair: `vip: "10.12.4.X 10.12.8.X"` (Option B).

**Rationale:** with a provider-only VIP, `charms_openstack/ip.py:resolve_address(INTERNAL)` returns `None` and raises `ValueError`, breaking `identity-service-relation-joined` (and the analogous internal-endpoint registration on every clustered API charm). Supplying a metal-network VIP alongside the provider VIP gives `resolve_address` an internal address to return, and keeps east-west service traffic on the metal network rather than the provider network.

**Status:** Decided (v1). Reconstructed into this doc from the deploy record (no standalone D-020 file).

**Related:** D-003 (network architecture), D-002 (channels).

---

## D-020 -- AMENDMENT (2026-07-27): vault's VIP is BUILT, designate JOINS the enumeration, the band widens to .99, and the gate learns to enforce it

**Status:** RULED 2026-07-27 (operator, GA-R5). Question as presented, verbatim: "R11 --
VIPs for vault and designate. D-020 already names vault as requiring both a provider and a
metal VIP, so vault is a conformance repair; designate is not in D-020's list and is a new
decision. Octets .61 and .62 are free, and the gate's band cap (.60) must be widened either
way. What shape?" Options presented: (a) both full triples, dual-family, plus the gate fix;
(b) vault metal-only pair per the 2026-07-25 expansion review, designate triple; (c) both
triples with the gate changes deferred to R15. Operator selection, exact utterance: **"Both
full triples (.61 vault, .62 designate), dual-family, and fix the gate (Recommended)"**.

**VAULT WAS ALREADY RULED, AND NEVER BUILT.** D-020's decision text above enumerates vault
by name among the clustered applications that carry BOTH a provider and a metal VIP. It has
none. Measured: base `vault` is `num_units: 1` with an EMPTY options block. So this half is
a CONFORMANCE REPAIR of a 2026-era decision, not a new choice -- and it is the same class as
the D-134 bands existing only in prose (R4). Two ruled decisions in this audit turned out
never to have been implemented.

**DESIGNATE IS GENUINELY NEW.** It is absent from D-020's enumeration, which is why it
needed a ruling rather than a repair. This amendment ADDS it. Coherence check that supports
the provider leg: designate's `dnsaas` endpoint is already bound to `provider-public` in the
base bundle, so DNS-as-a-service is tenant-facing by design.

**RULED SHAPE -- both get the ESTABLISHED triple, not a new form.** Measured octet map, all
eleven existing VIPs are consecutive provider/admin/internal triples: keystone `.50`,
barbican `.51`, cinder `.52`, glance `.53`, magnum `.54`, neutron-api `.55`,
nova-cloud-controller `.56`, octavia `.57`, openstack-dashboard `.58`, placement `.59`,
ceph-radosgw `.60`. The next free octets are `.61` and `.62`:
  - **vault -> .61**, **designate -> .62**, each a provider/admin/internal triple in the
    per-DC bands, and **DUAL-FAMILY** per the R2 ruling (v4 + v6 from the outset -- adding
    v4-only now and re-doing them later would mean re-issuing certificate SANs on a live
    cloud, and for vault that is 23 certificate relations).

**WHY OPTION (b) WAS REFUSED.** The 2026-07-25 expansion review proposed vault as metal-only
with no provider leg, and it has a real technical argument (all 23 vault consumers are
internal). But it CONTRADICTS D-020's own enumeration, so it would have required a D-020
amendment departing from the decision rather than conforming to it, and it would have made
vault the single non-triple in the bundle -- a permanent special case for
`provider-bundle-check.py`. The conflict is recorded here so the expansion review's proposal
is not later mistaken for the ruled position.

**D-020 -- AMENDMENT (2026-08-05): vault is METAL-ONLY; the provider VIP is DROPPED**
(ADOPTS the 2026-07-25 expansion-review proposal recorded as "refused" immediately above --
build-time proved it is the only buildable and the security-correct shape).

**Status:** RULED 2026-08-05 (operator, GA-R5). **Question as presented:** at the D-121
vault-HA build (2026-08-05), vault's provider-public VIP `10.12.4.61` FAILED to bind --
hacluster `res_vault_..._vip start ... '[findif] failed'` on all 3 nodes -- because vault's
containers have NO provider-public interface (MEASURED: metal-admin `10.12.8.x` + metal-internal
`10.12.12.x` only; the metal-admin `.8.61` and metal-internal `.12.61` VIPs bound fine).
Verified LIVE that EVERY vault consumer is an internal OpenStack charm (`vault:certificates`
-> ~17 control-plane charms including magnum's OWN service cert; `vault:secrets` ->
barbican-vault only); nothing tenant-facing reaches vault -- the tenant-secrets path is
tenant -> Barbican provider VIP `.51` -> barbican-vault -> vault-internal, and tenant k8s PKI
is CAPI/kubeadm-generated, not from the cloud vault. Options: (a) vault METAL-ONLY -- drop the
provider VIP, keep metal-admin `.8.61` + metal-internal `.12.61`; (b) give vault a
provider-public leg so the VIP binds -- which would expose the PKI/CA ROOT on the tenant-facing
plane (security regression). Operator selection, exact utterance: **"(a) draft the amendment"**,
then on the drafted text: **"Ratify as written"**.

**RULED SHAPE.** vault's VIP becomes the metal PAIR `"10.12.8.61 10.12.12.61"` (admin +
internal), NOT the provider/admin/internal triple. vault is the internal PKI/CA root and MUST
NOT be tenant-reachable, and has no provider leg to host `10.12.4.61` regardless. vault becomes
the single non-triple clustered VIP in the bundle; `provider-bundle-check.py`'s
"clustered VIPs are provider/admin/internal" invariant learns the vault metal-only exception.

**CONSEQUENCE / BUILD.** (1) `overlays/vr1-dc0-vips.yaml` vault vip
`"10.12.4.61 10.12.8.61 10.12.12.61"` -> `"10.12.8.61 10.12.12.61"`; `overlays/vr1-dc1-vips.yaml`
vault vip the dc1 dual-family sextet -> the dc1 metal pair `"10.12.68.61 10.12.72.61"` (the
provider leg AND the D-141-deprecated `fd50:` ULA metal v6 legs dropped). Both DCs keep vault a
**v4 metal pair, and it STAYS a v4 metal pair through a future D-141 rule-3 dual-stack promotion**
until a new amendment rules on vault's v6 metal legs -- so a later dual-stack session does not
read the missing v6 legs as an oversight and "restore" them. (2) LIVE: `juju config vault
vip="10.12.8.61 10.12.12.61"` drops the dead provider VIP resource -> vault-hacluster unblocks.
(3) `provider-bundle-check.py` gains a vault metal-only exception (DOCFIX): the metal pair is
asserted **per-DC and family-aware** against the `--dc` PROVIDER/ADMIN/INTERNAL bands (NOT a
hardcoded `10.12.4.` prefix, which was blind to dc1's `10.12.64.x` provider), and is STILL
band-checked and octet-uniqueness-checked -- vault leaves the triple count but does not leave
those gates. (4) `.61` freed on provider-public. **Roosevelt delta:** vault is metal-only at
every DC standup; the PKI/CA root is never on a tenant-facing plane. Supersedes the "full triple
for vault" half of the 2026-07-27 amendment; the designate `.62` triple and the band-widening
below are UNCHANGED.

**BAND WIDENING (mechanical, two files, no choice in it).** `.61` and `.62` are legal under
the D-134 AMENDMENT's `.50-.99` VIP band but are REJECTED by the gate as it stands:
`scripts/provider-bundle-check.py:41` holds `OCTET_LO, OCTET_HI = 50, 60` and
`scripts/lib-net.sh:54-59` holds `VIP_OCTET_MAX=60`. Both widen to `.99` to match D-134.
Note these are SEPARATELY NAMED constants in two files -- raising the band is a two-file
change, not one (audit finding L3-7).

**GATE HARDENING, ruled IN SCOPE rather than deferred.** `provider-bundle-check.py` is
extended to FAIL on an application that has an `hacluster` relation but no `vip`. Measured
justification: `grep -rn cluster_count scripts/ tests/` returns NOTHING, and a constructed
overlay rewriting all 20 `cluster_count` values from 3 to 1 produces a byte-identical PASS
(audit finding L4-3). That is why decorative HA was found by an audit instead of by a gate.
`VIP_COUNT_EXPECT` also moves 11 -> 13.

**Execution is a SEPARATE gated step.** Standard delivery discipline applies to the checker
change (harness green, gauntlet, repo-lint, changelog with revert). These VIPs land in the
per-DC `*-vips.yaml` overlay, merged with the base bundle at deploy. (R6's "before the HA
overlay" sequencing is SUPERSEDED 2026-08-06 -- `dc-ha-scaleup.yaml` was retired once
BUNDLEFIX-053 folded the HA chain into the base bundle; see the R6 supersession note.)

## D-021: Octavia amphora image pipeline on the no-DNS dual-endpoint deploy

**Decision:** build the amphora image with the charm-native `octavia-diskimage-retrofit` set `use-internal-endpoints: true`, seeded by a manually uploaded stock Ubuntu base image carrying the five Glance properties the retrofit reads (architecture, os_distro, os_version, version_name, product_name). Park `glance-simplestreams-sync` for the amphora pipeline. The amphora image is `image-format: raw`, tagged `octavia-amphora` to match octavia's `amp-image-tag`.

**Root cause:** on the dual-endpoint, no-DNS topology (D-019), metal-net catalog-callers (gss + its retrofit subordinate) cannot reach Glance: the public Glance FQDN does not resolve/route from the metal net, and the internal-endpoint cert carries no FQDN SAN (so an `/etc/hosts` FQDN->metal-VIP mapping fails TLS). gss `use-internal-endpoints` steers only its Keystone auth to internal; its glance/swift clients still use the public FQDN and there is no further charm-native lever -- a charm gap on the no-DNS topology. The retrofit's `use-internal-endpoints` lever DOES cover its build path, so it is the charm-native amphora builder here.

**Status:** Decided + validated end-to-end (v1): the retrofit, over internal endpoints, reads the seeded base and writes the amphora; gss parked; octavia + subordinates active/idle.

**Roosevelt:** cloud-internal DNS + FQDN-valid certs removes the manual seed and fixes gss end to end.

**Related:** D-007 (Octavia inclusion), D-019 (no-DNS root cause).

---

## D-028: Defer the CAPI v1beta2-contract cutover (deploy the single-contract v1beta1 stack)

**Decision:** defer adopting the CAPI v1beta2-CONTRACT generation until upstream ships it correctly for this path; deploy the clean single-contract v1beta1 stack now.

**Context:** while grounding the (then-current) Canonical CK8s workload chart, the chart referenced control-plane/bootstrap kinds at apiVersion v1beta1 while the pinned provider served them only at v1beta2 (DOCFIX-022). The broader question -- is the v1beta2-contract generation available and correct for long-term support on this path -- resolved to "not yet."

**Status:** Decided (v1). The CK8s-chart-specific particulars were subsequently retired when D-031 replaced the direct-CAPI CK8s path with Magnum + the azimuth kubeadm charts; the single-contract principle carries forward, and D-042 later made the driver-side contract axis concrete.

**Builds on:** D-022 / D-023 (do-07-era CAPI/CRD work). **Related:** D-031, D-042.

---

## D-029: Defer Keystone SSO (k8s-keystone-auth) to Roosevelt

**Decision:** Keystone SSO for the workload clusters (the chart's `k8s-keystone-auth` addon) is deferred to the next deployment and folded into the Roosevelt cloud-internal-DNS + trusted-cert foundation. v1 workload clusters run the Kubernetes Dashboard with standard token auth; the `k8sKeystoneAuth` addon stays OFF; SSO is not validated on v1.

**Rationale:** enabling it on v1 would produce a non-functional SSO path (TLS failure to the private-CA Keystone endpoint) plus apiserver webhook error noise -- a checked box that does not work -- and forcing it would require forking the addon or fighting CAAPH, neither of which carries forward to Roosevelt.

**Finding (verified 2026-06-05):** k8s-keystone-auth 1.5.1 exposes no keystone-CA option, so it cannot trust a private-CA Keystone endpoint.

**Status:** Decided (v1). **Related:** D-028 (same "land it on the proper foundation later" principle).

---

## D-030: Management-cluster placement -- in-cloud (superseded twice; see D-033, D-035)

**Decision (as taken 2026-06-06):** run the CAPI management plane IN-CLOUD for the v1 rehearsal (CAPI core + CAPO + cluster-api-addon-provider as VMs on the OpenStack cloud, following an Azimuth seed + HA pattern with a `clusterctl move` pivot to a self-hosted in-cloud management cluster). Out-of-cloud was recorded as a deferred alternative for Roosevelt.

**Status:** SUPERSEDED. First by D-033 (out-of-cloud Canonical `k8s`-charm on MAAS); then -- after D-033's dual-homed node hit an unfixable pod-egress fault -- placement returned in-cloud in a simpler single-homed form under D-035. Retained here for lineage.

**Related:** D-031, D-033, D-035.

---

## D-031: Cluster-creation surface + engine -- Magnum + magnum-capi-helm + azimuth kubeadm charts

**Decision:** the tenant Kubernetes service is built from three layers:
- Surface: OpenStack Magnum (`openstack coe cluster ...`), so tenants and operators manage clusters through the OpenStack API.
- Driver: the in-tree Cluster API Helm driver `magnum-capi-helm` (opendev.org/openstack/magnum-capi-helm), pip-installed into the Magnum conductor and pointed at a CAPI management cluster via `[capi_helm] kubeconfig_file`.
- Engine: the azimuth-cloud `capi-helm-charts` `openstack-cluster` chart (kubeadm-based: KubeadmControlPlane / KubeadmConfigTemplate + CAPO OpenStackCluster / OpenStackMachineTemplate + MachineDeployment), with addons (Cilium CNI, OpenStack CCM, Cinder CSI, and so on) installed by the cluster-api-addon-provider.
- Management-cluster placement: in-cloud for v1 (D-030, later refined by D-035).

**Status:** Decided. Supersedes the do-07 direct-CAPI Canonical CK8s chart path; the CK8s-chart-specific findings (DOCFIX-022 ref patch, etc.) are retired for this path.

**Related:** D-030 / D-035 (placement), D-034 (version constellation), D-036 / D-042 (driver/chart/core coherence).

---

## D-033: Management cluster -- out-of-cloud Canonical k8s-charm on MAAS (superseded by D-035)

**Decision (as taken 2026-06-07):** management cluster = a Canonical Kubernetes cluster deployed with the `k8s` / `k8s-worker` machine charms on MAAS, OUTSIDE OpenStack, made HA by the charms; CAPI layer via `clusterctl init --infrastructure openstack` + cluster-api-addon-provider, version-pinned to the capi-helm-charts release (NOT the D-022 do-07 pins); the management cluster does not run the OpenStack CCM for itself (CAPO reaches OpenStack through a `clouds.yaml` pointed at the public API endpoints); lifecycle via Juju.

**Status:** SUPERSEDED by D-035. The chosen node (capi-mgmt MAAS VM) is necessarily dual-homed (MAAS PXE on metal, API VIPs on provider), and pod egress from that multi-NIC node to the API VIPs failed (the Cilium reverse-NAT reply was mis-forwarded out the wrong NIC instead of redirected into the pod). Retained here for lineage.

**Supersedes:** D-030 (placement) + D-032 (azimuth-config tooling). **Builds on:** D-031.

---

## D-034: CAPI version constellation pinned to capi-helm-charts dependencies.json

**Decision:** pin the management-cluster CAPI constellation to the `dependencies.json` published with a chosen `capi-helm-charts` RELEASE TAG, read at deploy time on the jumphost with `jq` (dynamic lookup, no hand-picked versions). Retire D-022 "Option A" (driver 1.3.0 / CAPO v0.10.x / v1alpha6) as obsolete.

**Rationale:** the magnum-capi-helm driver does not hand-pick component versions; its own CI installs the management CAPI stack by reading the per-release `dependencies.json` and running a fixed install sequence -- that file is the single matched-and-tested set. Hand-picking fights the upstream model, and v1alpha6 has been removed from current cluster-api-provider-openstack. (At tag 0.25.1 the set is CAPI v1.13.2, CAPO v0.14.4, cert-manager v1.20.2, ORC v2.5.0, addon-provider 0.12.0, janitor 0.11.0, helm v3.17.3; appendix-B carries the as-built snapshot.)

**Status:** Adopted 2026-06-08. **Supersedes:** D-022. **Amended by:** D-042 (adds the driver<->core contract-coherence rule). **Related:** D-031, D-028 (CRD-contract note, now subsumed).

---

## D-035: Management-cluster placement -- in-cloud single-homed tenant VM

**Decision:** run the CAPI management cluster as a single-homed in-cloud tenant VM (`capi-mgmt-v2`): one NIC on the management tenant subnet (10.20.0.0/24), reached via a floating IP (per-rebuild -- this rebuild 10.12.5.103; the original 10.12.7.40 is dead -- DOCFIX-038); k8s-snap (channel `1.32-classic/stable`), Cilium CNI; not CAPI-self-managed (no `clusterctl move`).

**Rationale:** D-033's out-of-cloud node was necessarily dual-homed and its pod egress to the OpenStack API VIPs failed -- the Cilium reverse-NAT reply was emitted back out the second NIC instead of being redirected into the pod via `cilium_host` (a multi-NIC reverse-path fault; the `k8s` charm exposes too few Cilium annotations to repair it). A single-homed VM removes the second NIC and the fault entirely. The single-NIC pod-egress premise was then proven by the Phase 4 hard gate (an agnhost pod TCP probe to the Keystone VIP 10.12.4.50:5000 returning exitCode 0).

**Status:** Adopted 2026-06-08; pod-egress premise validated. **Supersedes:** D-033 (revisits D-030 in simpler form); also retires the Layer B topology of **D-007** and the bootstrap mechanism of **D-017** (k3s-on-MAAS + `clusterctl move` -> in-cloud single-homed VM, no pivot). **Unaffected:** D-031, D-034.

**Trade-off:** a single-node management cluster is a SPOF with no self-heal -- see D-041 (manual-start policy) and D-040 (the OOM that surfaced it).

---

## D-036: magnum-capi-helm driver / chart / CAPO coherence (resolved)

**Decision / correction:** a mid-session "rebuild Phase 5 on chart 0.10.1" framing -- premised on the GA driver (1.3.0) emitting the v1alpha6 OpenStackCluster CRD and clashing with the modern v1beta1 stack -- is WRONG and is retired. Chart 0.10.1 is the retired v1alpha6 path that D-034 superseded; rebuilding on it would have reversed D-034.

**Verification:** the 1.3.0 driver is api_version-AGNOSTIC (driver.py has zero v1alpha6/v1beta1/apiVersion references; it helm-installs the chart and watches the CAPI `Cluster`, never writing OpenStackCluster directly). The OpenStackCluster apiVersion is set by the CHART: chart 0.25.1 emits `infrastructure.cluster.x-k8s.io/v1beta1`, matching the installed CAPO v0.14.4. The driver's built-in default chart is 0.10.1 (the v1alpha6-era chart); overriding `default_helm_chart_version` to 0.25.1 yields v1beta1. The "1.3.0 emits v1alpha6" claim was true only of the driver's DEFAULT chart, not of the driver pinned to chart 0.25.1.

**Status:** Resolved 2026-06-08. Implements D-031 Phase 3 under the D-034 constellation. NOTE: a SEPARATE axis -- the driver-vs-core CONTRACT, not the chart's CRD string -- is what later required the 1.4.0 driver pin; see D-042. **Related:** D-031, D-034.

---

## D-037: [capi_helm] config persistence on the charm-managed conductor

**Decision:** keep the `[capi_helm]` section in an oslo.config drop-in directory and point the conductor at it: `/etc/magnum/magnum.conf.d/00-capi-helm.conf` (0644, no secrets; it references the 0600 kubeconfig by path), with magnum-conductor launched with `--config-dir /etc/magnum/magnum.conf.d` so oslo.config merges the drop-in over the charm-rendered `magnum.conf`. The charm manages neither the .conf.d directory nor the launch extension, so this survives charm hooks and reproduces on Roosevelt.

**Problem:** the magnum charm (2024.1/stable rev 70) re-renders `magnum.conf` wholesale on hooks and exposes no conf-override option, so a `[capi_helm]` section written into `magnum.conf` would be clobbered.

**Mechanism (load-bearing correction):** the conductor's ExecStart is NOT a direct binary -- it is `/etc/init.d/magnum-conductor systemd-start` (an LSB init script wrapped by systemd), so a systemd ExecStart drop-in appending `--config-dir` is inert (the flag reaches the init script as an ignored positional). The adopted method instead creates `/etc/default/magnum-conductor` (0644; the charm does not manage it) containing `DAEMON_ARGS="$DAEMON_ARGS --config-dir /etc/magnum/magnum.conf.d"`; the init script sources `/etc/default/$NAME` after setting the base `DAEMON_ARGS`, then runs `exec $DAEMON $DAEMON_ARGS`. Verify behaviorally with `/etc/init.d/magnum-conductor show-args` and `ps -ww -C magnum-conductor -o args` (not string-presence).

**Status:** Adopted 2026-06-08 (mechanism revised mid-implementation). **Residual:** breaks silently if a future charm hook writes `/etc/default/magnum-conductor` -- detect via the same show-args/ps check. **Related:** D-031 Phase 3, D-036.

---

## D-039: Magnum mints per-cluster app-creds carrying the trustor's roles (grant load-balancer_member)

**Status:** ACCEPTED 2026-06-09 (applied in phase-06; asserted in phase-08 prereqs). Cited by phase-06 (DOCFIX-036 grant), phase-08, and appendix-A; recorded here to close the dangling reference.

**Context:** the magnum-capi-helm service path uses NO static, pre-provisioned application credential. At cluster-create magnum mints a per-cluster Keystone application credential from a trust, and that app-cred carries the TRUSTOR's roles FROZEN at mint time, delegated unfiltered. The trustor is the identity that creates the cluster in the capi-mgmt project (admin@admin_domain in v1).

**Decision:** the trustor must hold `load-balancer_member` (plus `member` and `reader`) on the capi-mgmt project BEFORE any cluster is created, so every minted app-cred carries Octavia authority. A trustor holding only `member` mints a frozen app-cred that 403s when CAPO queries Octavia to confirm LB state -- the workload cluster then wedges at API-LB provisioning, and a stuck-delete 403s the same way (appendix-A). The grant is idempotent (member + load-balancer_member + reader) and applied in phase-06.

**Roosevelt implication:** whichever identity creates Magnum/CAPI clusters must carry `load-balancer_member` + `reader` on the cluster project; `member`-only is a latent 403. There is no static `capo` user/app-cred to provision -- that pattern was retired with the per-cluster mint.

**Related:** D-031 / D-036 (driver/engine surface), D-046 (the trustee domain the trust resolves against), appendix-A (D-039 + stuck-delete recovery).

---

## D-040: Raise nova-compute reserved-host-memory on the hyperconverged hosts

**Decision:** set `nova-compute reserved-host-memory` to 8192 MB (from the default 512) so Nova placement accounts for the non-Nova memory co-located on each hyperconverged host. Charm config -> survives redeploy.

**Trigger / root cause:** during the first end-to-end Magnum workload-cluster create, openstack1 hit the kernel OOM-killer (no reboot; single boot since 2026-06-03) and killed a tenant qemu worker VM. The host co-locates nova-compute AND roughly 6 GiB of services invisible to Nova placement (mysqld [innodb-cluster member] ~2.9G, ceph-osd + ceph-mon ~1.2G, neutron workers ~0.7G, nova/apache/cinder/ovs ~1.4G) while Nova reserved only the default 512 MB; under the resulting memory pressure the host swap-thrashed (an ovsdb inactivity-probe storm made the workload API and Juju agent look "down" when the host was in fact thrashing, not down).

**Status:** Adopted + APPLIED 2026-06-09. **Related:** D-035 (the mgmt-VM SPOF the OOM hit), D-041.

---

## D-041: Non-HA deployments default to manual start

**Decision:** non-HA deployments default to MANUAL START -- no automatic VM power-on / auto-recovery is configured by default. Any non-HA deployment must be documented as non-HA, with the rationale that manual-down surfaces incidents (auto-restart masks capacity/health defects). Auto-recovery is an explicit, out-of-band exception, never the silent default.

**Trigger:** after the openstack1 OOM (D-040), CAPI's MachineHealthCheck self-healed the workload worker VMs automatically, but the single-node management VM (capi-mgmt-v2, D-035) was OOM-killed and stayed SHUTOFF -- it does not self-heal or auto-restart, which silently broke magnum reconcile/health and left workload nodes with the CAPI uninitialized taint until it was started by hand. The cost (downtime) was real, but the manual-down is also what forced the investigation that found the OOM root cause headed for Roosevelt.

**Status:** Adopted 2026-06-09 (policy/governance). **Related:** D-035 (the SPOF), D-040 (the OOM).

---

## D-042: magnum-capi-helm driver must be contract-coherent with the CAPI core

**Decision (amends D-034):** the magnum-capi-helm driver pin (Layer B) MUST be contract-coherent with the CAPI core that `dependencies.json` installs (Layer A). When the Layer-A lockfile is a v1beta2-contract core (CAPI v1.13), the driver pin must be a build that understands v1beta2 references; verify this intersection at deploy.

**Symptom / root cause:** capi-test-1 reached CREATE_COMPLETE with every real component healthy (3 Ready nodes, Calico, CCM/CSI/CoreDNS, API LB ACTIVE), yet magnum reported `health_status = UNHEALTHY` deterministically -- only the `infrastructure` sub-check failed ("Infrastructure resource not found"). The 1.3.0 driver reads `apiVersion` off the Cluster's `spec.infrastructureRef`, but under the v1beta2 contract that ref is version-less, so the health GET resolves nothing. The create path is unaffected (the chart templates the resource versions) -- a cosmetic health false-negative. The governing axis is the CAPI CONTRACT a provider implements toward core, not the CRD apiVersion string (per D-028); rolling back to a v1beta1 core would mean pinning an EOL CAPI for a Roosevelt rehearsal -- the wrong direction.

**Fix:** pin a driver build carrying the per-kind `[capi_helm] api_resources` override and set it so the health lookups use the served versions. As of 2026-06-09, D-042 recorded this capability as UNRELEASED (development series only; released line then 1.1.0/1.2.0/1.3.0), with the interim = a current-series commit for the testcloud and a released-tag pin deferred to Roosevelt.

**Subsequent update (driver-fix work):** the released `magnum-capi-helm==1.4.0` was then confirmed to ship the `api_resources` feature, so the released-tag pin is now available -- v1 pins 1.4.0 with an explicit `api_resources` and targets `health_status = HEALTHY` (installed in phase-07; as-built in appendix-B). This replaces D-042's interim dev-commit path.

**Operational caveat (while any health false-negative persists):** do NOT wire magnum auto-healing to `health_status` -- a persistent false UNHEALTHY could misfire; CAPI MachineHealthCheck handles node healing independently.

**Status:** Adopted 2026-06-09; fix landed via the 1.4.0 pin. **Amends:** D-034. **Related:** D-028 (the contract axis made concrete), D-031, D-035.

---


From prior bundle review work — these are anti-patterns:

- `magnum-shared-db` missing colon — causes a relation endpoint syntax error, deploy-blocking. Bundle must use `- - magnum:shared-db` (with the colon).
- Empty `osd-devices` YAML anchor referenced by multiple ceph-osd applications.
- `ovn-chassis` binding `overlay-suffix` — invalid binding name. Correct value is `data`.
- GUI annotation collision between NUMA-split ceph-osd apps (not applicable to testcloud since we don't NUMA-split, but flagged for Roosevelt).
- Hardcoded NIC name in `bridge-interface-mappings`. Use MAC where possible.
- `openstack -f value` column ordering — column order is not guaranteed; use `-c <column> -f value` for single-column output.
- Snap confinement: `openstackclients` snap has home-only interface; commands cannot read paths under `/tmp`. File paths must resolve under `$HOME`.
- Non-ASCII characters in `local_settings.d` overrides cause silent daemon failures in Horizon.

---


## Change log

| Date       | Change                                                                                                                                                                                                                            | Reference                                            |
| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
| 2026-05-22 | Initial document captured                                                                                                                                                                                                         | Caracal rebuild planning session                     |
| 2026-05-22 | D-015 v1/v2 fork added; D-004 and D-004a marked v2-scope; D-016 IPv4 tenant pool hybrid model added; D-014 updated with new repo name                                                                                             | v1/v2 fork session                                   |
| 2026-05-22 | D-017 CAPI bootstrap full-rebuild lifecycle added; D-018 MAAS-release-direct teardown added. D-013 marked superseded by D-018. D-007 Layer B updated to reference D-017 and `runbooks/04a-capi-bootstrap-cluster.md`.             | Teardown planning + handoff session                  |
| 2026-05-22 | D-002 hacluster row added (channel `2.4/stable`) per Canonical Charm Delivery table, verified against Charmhub. D-007 Layer B driver pin updated: `stackhpc/magnum-capi-helm` v0.13.0 → `openstack/magnum-capi-helm` 1.1.0 (PyPI; stackhpc fork archived Dec 2024). | Caracal channel verification + driver pin correction |
| 2026-05-22 | D-007 Layer B kubeconfig target corrected: bootstrap k3s → workload cluster (post-pivot per workstream 3b mandatory `clusterctl move`). CAPI mgmt plane paragraph updated accordingly. | Workstream 3 cleanup (post-pivot semantics) |
| 2026-05-29 | D-019 (Designate deferral) and D-020 (dual provider+metal API VIPs) recorded as already-taken; folded into this doc in the 2026-06-09 consolidation. | Deploy execution / handoff |
| 2026-05-30 | D-021 Octavia amphora pipeline (charm-native retrofit over internal endpoints; gss parked) added. | Octavia enablement |
| 2026-06-05 | D-028 (defer v1beta2-contract cutover) and D-029 (defer Keystone SSO) added. | CAPI path research |
| 2026-06-06 | D-030 (mgmt-cluster placement: in-cloud) and D-031 (Magnum + magnum-capi-helm + azimuth kubeadm engine) added. | Magnum/CAPI surface decisions |
| 2026-06-07 | D-033 (mgmt cluster: out-of-cloud k8s-charm on MAAS) added; supersedes D-030 and D-032. | Mgmt-cluster shape |
| 2026-06-08 | D-034 (CAPI constellation pinned to dependencies.json; supersedes D-022), D-035 (in-cloud single-homed mgmt VM; supersedes D-033), D-036 (driver/chart/CAPO coherence resolved), D-037 ([capi_helm] via /etc/default DAEMON_ARGS) added. | In-cloud mgmt pivot |
| 2026-06-09 | D-040 (reserved-host-memory 8192), D-041 (non-HA manual-start policy), D-042 (driver<->core contract coherence; 1.4.0 pin) added. | OOM incident + driver fix |
| 2026-06-09 | D-019..D-042 consolidated into this document (15 decisions). Existing D-001..D-018 left intact (em-dash style preserved); the new entries are ASCII. | Repo sanitation / doc refresh |
| 2026-06-17 | D-044 (Horizon secure-cookie override) + D-045 (haproxy confirmed-LOADED) folded from the changes-doc; D-046 (magnum trustee-domain setup) + D-047 (keystone v2.0 render bug / v3 drop-in) merged and renumbered from the 06-17 addendum (its "D-044/D-045"); D-048 (stage-and-verify canonical image seed, supersedes web-download) + D-049 (D1: kube v1.34.8 / capi-k8s-v1-34) added; D-042 amended (FINDING-5: UNHEALTHY is a <=1.3.0 false-negative, HEALTHY on 1.4.0); D-050 (PROPOSED: keystone policyd-override) recorded. | End-of-deploy runbook sweep |
| 2026-06-18 | D-039 (Magnum per-cluster app-cred roles; grant load-balancer_member) recorded to close a dangling reference (previously cited only in phase-06/08, bundle, appendix-A). D-007 + D-017 annotated as superseded by D-035 / D-037 / D-042 (in-cloud mgmt VM; /etc/default config-dir graft; 1.4.0 driver) -- historical bodies retained. | Pre-commit audit (runbook sweep) |

<!-- patchset-20260610-decisions-addendum -->

---

## D-042 -- AMENDMENT (2026-06-10): mechanism evidence + signature taxonomy

Evidence gathered during the 2026-06-10 recovery session strengthens and
sharpens D-042:

1. Steady-state cosmetic signature, verbatim: health_status=UNHEALTHY with
   reason {'cluster': 'Ready', 'infrastructure': 'Infrastructure resource not
   found.', 'controlplane': 'Ready', 'nodegroup': 'Ready'}. "Resource NOT
   FOUND" (a lookup/contract miss) rather than "resource unhealthy"
   corroborates the driver-vs-CAPO API-contract coherence diagnosis.
2. The driver health maps to the cluster-level v1beta2 Available aggregation.
   This session also observed a REAL Available=False (InfrastructureReady
   failed: kube-api LB reconcile timeout during a mgmt-VM cold start, LB left
   provisioning=ERROR). Two distinct mechanisms therefore share the UNHEALTHY
   presentation -- the operator cannot distinguish real from cosmetic without
   reading the reason field -- which strengthens the case for the staged
   forward-fix (magnum-capi-helm-driver-fix runbook).
3. Signature taxonomy for operators: reason EMPTY = conductor cannot reach the
   mgmt API (VM parked/down; NOT D-042); reason populated all-'Ready' with
   infrastructure 'Infrastructure resource not found.' = D-042 cosmetic;
   reason citing an LB/infrastructure failure = real, check Octavia first
   (ops-capi-recovery Step 4/5).

## D-043 -- PROPOSED: tenant-VM auto-resume policy (resume-guests-state-on-host-boot)

STATUS: PROPOSED (decision pending). Recorded 2026-06-10.

QUESTION: should nova-compute `resume-guests-state-on-host-boot=true` be set,
so tenant VMs (including the in-cloud CAPI management VM, D-035) return
automatically after host reboots?

TENSION: D-041 ("a non-HA component staying down is a signal to investigate,
not a nuisance to auto-mask") was written for control-plane charm services.
Tenant VMs are a different class: on Roosevelt, customers will expect their
VMs back after host maintenance, and auto-resume is the industry norm for
that class. Cost of the manual policy observed 2026-06-10: a deliberately
parked mgmt VM was mistaken for an outage and consumed roughly two hours of
diagnosis before the stop was traced to an API action.

OPTIONS:
  (a) Enable auto-resume cloud-wide + monitoring/alerting on VM state, so
      "down" remains a signal without being an outage. RECOMMENDED for
      Roosevelt; candidate for v1 redeploy as well.
  (b) Keep manual start for v1 (preserves D-041 discipline on the testcloud),
      explicitly record auto-resume as the Roosevelt setting.
Note: the restart procedure's failure-mode table already references the config
key for SHUTOFF guests; whichever option is chosen, align that table, this
decision, and the bundle/runbook with each other.

<!-- patchset-20260617-sweep-decisions : end-of-deploy runbook sweep -->

---

## D-044: Horizon Secure-cookie override for internal-HTTP dashboard access (DOCFIX-030)

**Status:** Adopted 2026-06-17 (PER-REBUILD; phase-03 Step 3.3). Resolves the mislabeled "D-043" tag used for this item in earlier phase-03/changes-doc drafts -- D-043 is the tenant-VM auto-resume decision.

**Decision:** the jumphost reaches Horizon over a plain-HTTP reverse-proxy leg, but the dashboard sets `SESSION_COOKIE_SECURE`/`CSRF_COOKIE_SECURE=True`, so the browser drops the session/CSRF cookies over HTTP and login fails. Apply a Django settings override on the openstack-dashboard leader (`_99_internal_http_cookies.py`, setting `SESSION_COOKIE_SECURE=False` + `CSRF_COOKIE_SECURE=False`) to allow cookie flow over the internal HTTP leg. A TESTCLOUD accommodation of the no-DNS / no-FQDN-cert posture.

**Trade-off / Roosevelt:** disabling Secure cookies is acceptable only because the proxy leg is internal and the cloud has no public DNS / FQDN-valid cert. The Roosevelt root-fix is cloud DNS + FQDN-valid certs end-to-end (which also fixes gss and the nginx proxy_ssl_name handling); then this override is removed. Self-signed-client-TLS approaches are NOT part of v1.

**Related:** DOCFIX-030 (phase-03 Step 3.3), D-043 (distinct -- auto-resume).

## D-045: Charm-rendered haproxy config must be confirmed LOADED, not just rendered (DOCFIX-031)

**Status:** Adopted + APPLIED 2026-06-11; re-verified 2026-06-16 post phase-05.

**Decision:** after the vault/TLS cert cascade settles, confirm every unit's haproxy is actually checking its backends over the freshly-rendered SSL config -- by a functional probe of haproxy's OWN backend state (admin-socket `show stat`, grep `,DOWN,`), NOT by `juju status`. Reload (not restart) any unit whose running haproxy predates the `check-ssl` re-render.

**Root cause (confirmed, not refined to a check-config defect):** nova-cloud-controller haproxy was not reloaded after the cert cascade, so its health checks ran plaintext against the now-SSL backend port and marked the nova-api backends DOWN -- while juju stayed active/idle (juju is BLIND to per-backend haproxy state). An 8s wire capture showed the checks switch to TLS after reload; both backends returned UP/L7OK/200. The reload is a real fix, not a band-aid.

**Status note:** the surfaced symptom was nova-api EOF / 503 behind a green juju. phase-03 Step 3.1 now gates a zero-DOWN sweep cloud-wide.

**Related:** DOCFIX-031 (phase-03 Step 3.1; appendix-A), D-046 / D-047 (the separate magnum-keystone incident).

## D-046: Magnum trustee-domain setup is a REQUIRED, asserted post-deploy step

**Status:** ACCEPTED 2026-06-17. Recorded in design-decisions-addendum-20260617.md as "D-044"; renumbered to D-046 here per the 06-17 reconciliation (D-044/D-045 were taken by the Horizon/haproxy decisions). Matches the rootcause doc + phase-08 handoff prompt.

**Context:** all `openstack coe ...` ops returned 403 ("Keystone client authentication failed"); magnum-api.log showed keystoneauth1 401 on every request since 2026-06-16. Root cause: the keystone domain `magnum` and user `magnum_domain_admin` that magnum.conf `[trust]` references did not exist. `magnum/common/policy.py:130` resolves `trustee_domain_id` on EVERY policy-enforced request (driver-agnostic), authenticating as the trustee domain admin; with the domain/user absent that is a hard 401 -> every coe op 403.

**Cause:** the magnum charm action `domain-setup` is MANUAL, not automatic; magnum reports active / "Unit is ready" regardless of whether it has run. The 2026-06-11 teardown/redeploy rebuilt keystone with fresh domains but the runbook did not re-run `domain-setup`.

**Decision:** `domain-setup` is a REQUIRED, ASSERTED post-deploy step on every (re)deploy, after the magnum + identity-service relation is up and BEFORE magnum is declared functional / before phase-08: (1) `juju run magnum/leader domain-setup`; (2) assert `openstack domain show magnum` and `openstack user show magnum_domain_admin --domain magnum` both succeed; (3) gate `openstack coe service list` (must return the conductor row, no 403). magnum's active/ready status MUST NOT be treated as evidence the trustee domain exists.

**Roosevelt:** carry as an explicit runbook step + assertion; consider upstreaming a charm change so domain-setup runs automatically, or so the charm surfaces "trustee domain not set up" instead of reporting ready.

**Related:** D-047 (the v2.0 render bug found in the same incident, but NOT the cause), D-031 / D-037 (magnum surface).

## D-047: keystone auth_version v2.0 charm-render bug -- keep the v3 drop-in

**Status:** ACCEPTED (keep) 2026-06-17. addendum "D-045" -> D-047 per the 06-17 reconciliation.

**Context:** the magnum charm template renders `auth_version = v2.0` due to a type bug (the keystone interface delivers `api_version` JSON-decoded as int 3; the template does a strict string compare `3 == "3"` -> False -> v2.0). Full analysis in incident-magnum-keystone-v2-rootcause-20260617.md.

**Finding:** the v2.0 render was NOT the cause of the coe 403 (that was D-046). On this deployment v2.0 is cosmetic -- magnum's `domain_admin_auth` rewrites v2.0 -> v3, v3 is discovered from the unversioned `auth_url`, and incoming token validation worked throughout.

**Decision (Jesse, 2026-06-17):** KEEP the magnum.conf.d v3 drop-in. v2.0 is the provably wrong value for Caracal (which does not serve v2.0); the drop-in forces v3 via the same config-dir mechanism as the D-037 conductor graft (no charm-file drift, survives re-render). Architectural correctness over minimize-delta, even though the drop-in did not unblock coe.

**As-built:** `/etc/magnum/magnum.conf.d/50-keystone-v3-override.conf` (auth_version=v3 + www_authenticate_uri/auth_url v3 in `[keystone_authtoken]` and `[keystone_auth]`); `/etc/default/magnum-api` DAEMON_ARGS adds `--config-dir /etc/magnum/magnum.conf.d` (mirrors D-037 for the standalone magnum-api).

**Roosevelt:** carry the v3 fix as a drop-in/overlay; upstream the template type bug (and the separate identity-service departed-hook IndexError crash documented in the rootcause doc).

**Related:** D-046, D-037.

## D-048: Stage-and-verify is the canonical image-seed method (supersedes web-download)

**Status:** Adopted 2026-06-17 (operator-approved). Supersedes the 2026-06-16 "web-download canonical" ruling.

**Decision:** seed ALL glance images (octavia amphora base, the noble mgmt image, the workload kube image) by STAGE-AND-VERIFY: download to `$HOME` (snap-readable; NOT /tmp), verify the file against the published checksum (azimuth-images manifest sha512 for kube images; ubuntu cloud-images SHA256SUMS for noble), then `openstack image create --file [--import]`. Web-download is retained as a TESTED ALTERNATIVE only (appendix-A).

**Rationale:** (1) FINDING-3 -- glance's web-download plugin fetches with urllib (UA `Python-urllib/3.x`) and the azimuth CDN 403s that UA, so web-download is UNUSABLE for kube images (202-accept, then stuck `queued`); (2) web-download cannot checksum-verify the fetched file (the CDN redirect strips the digest) -- weaker provenance; (3) stage-and-verify is one provenance-verified path cloud-wide -- less Roosevelt delta. CORRECTION-1: a plain `--file` PUT stores qcow2 (boots fine); `--import` runs glance image-conversion -> raw (Ceph fast-clone alignment).

**Roosevelt:** unify on stage-and-verify; the longer-term target remains gss-from-a-controlled-mirror once cloud DNS + FQDN certs land.

**Related:** D-021 (amphora pipeline), FINDING-3 (appendix-A image-seeding), phase-05 / 06 / 08.

## D-049: Workload kube image bumped v1.32.13 (EOL) -> v1.34.8 (D1)

**Status:** Adopted 2026-06-17. Procedure target; re-validation on v1.34.8 follows the stage-and-verify seed.

**Decision:** the workload-cluster kube image moves from the EOL ubuntu-jammy-kube-v1.32.13 to ubuntu-jammy-kube-v1.34.8 (azimuth-images 0.28.0, build 260518-1604; sha512 7efde485...760bdb3), and the cluster template is renamed `capi-k8s-v1-32` -> `capi-k8s-v1-34`. v1.34.8 is mature with good runway and within CAPI v1.13.2 support. The management cluster's OWN k8s stays at v1.32.13 (k8s-snap 1.32-classic) -- this bump is the workload image only.

**Note:** the 2026-06-09 D-011 acceptance ran on v1.32.13; the v1.34.8 image is seeded via D-048 stage-and-verify, and D-011 re-validation on v1.34.8 is the pending acceptance item. The template now pins `--network-driver calico` (DOCFIX-032).

**Related:** D-031 / D-034 (CAPI surface), D-048 (seed), DOCFIX-032 (CNI pin), phase-08.

## D-042 -- AMENDMENT (2026-06-17): FINDING-5 -- rescope to "<= 1.3.0" (HEALTHY on 1.4.0)

The D-042 cosmetic `health_status = UNHEALTHY` false-negative is a property of driver builds <= 1.3.0 (the v1beta2 contract-ref mismatch: those builds read `apiVersion` off the infrastructureRef, which CAPI v1.13's v1beta2 contract no longer carries). The RELEASED 1.4.0 driver carries the `api_resources` override and reports `health_status = HEALTHY` against the CAPI v1.13.2 / CAPO v0.14.4 stack (confirmed this rebuild). FINDING-5: D-042 is therefore CLOSED for v1 -- the UNHEALTHY caveat applies only to the historical <=1.3.0 holding state, NOT to the as-built 1.4.0 pin. Auto-heal is still NOT wired to health_status (CAPI MachineHealthCheck heals independently).

## D-050: RESOLVED / CLOSED (2026-07-06) -- keystone `use-policyd-override=true` with no policy zip (FINDING-1)

**Status:** RESOLVED / CLOSED 2026-07-06 (recorded 2026-06-17 as an unruled finding; operator ruling: resolved via D-051 option (b); see the RESOLVED amendment near the end of this file). Scan note: this line deliberately avoids the unruled-status keywords so ledger-scan closes the block; the history arrow lives in the amendment.

**Question:** keystone is configured with `use-policyd-override=true` but no policy override zip is supplied. This is currently a no-op (no custom policy applied), but the flag advertises an override capability that does not exist -- a latent footgun (a future operator may assume policy is being enforced, or a stray zip could silently change authz).

**Options (unresolved):** (a) set `use-policyd-override=false` for v1 (the override is unused) and revisit when a real policy is needed; (b) keep true and supply an explicit, reviewed policy zip; (c) leave as-is and document the no-op. No decision made -- recorded as an open point to rule on (cf. D-043, also pending).

**Related:** D-029 (Keystone SSO deferral), FINDING-1.

## D-051: Tenant identity self-service via the SCS Domain Manager policy (resolves D-050)

**Status:** Adopted 2026-06-23 (design; pre-redeploy). Implementation staged (`domain-manager-policy.yaml`), behavioral acceptance pending the bindings+policy redeploy (gate G3). RESOLVES D-050 via its option (b).

**Context / why:** the commercial multi-DC end goal requires Model-2 tenant self-service: a tenant domain-admin must create/manage users, projects, and role assignments WITHIN THEIR OWN DOMAIN, without operator involvement and without escaping the domain. The plain `admin` role cannot do this: on Keystone it is architecturally NOT domain-confinable (a domain/project-scoped `admin` can escalate beyond its boundary -- upstream hard-coded limitation). This is the root cause behind the unreliable Horizon domain-admin experiments (admin-on-domain + OPENSTACK_KEYSTONE_PREFER_DOMAIN_TOKEN scope-switch breakage).

**Decision:** implement the **SCS Domain Manager persona** (SCS scs-0302). A tenant domain-admin is a user holding the **`manager` role on their domain** (the `manager` role already exists on this cloud). Domain-confined identity self-service is granted by a **Keystone API policy override** shipped via the keystone charm's `policyd-override` resource (the mechanism `use-policyd-override=true`, already set, expects). The policy is `domain-manager-policy.yaml`, adapted VERBATIM from the canonical SCS reference (SovereignCloudStack/standards, scs-0302-w1-domain-manager-implementation-notes.md), Section A (base_* defaults) + Section B (Domain Manager extensions).

Key policy properties (anti-escalation + fit):
- `is_domain_manager` = `role:manager`; every domain-manager grant is gated on a `token.domain.id == target.*.domain_id` match -> strictly domain-confined.
- `is_domain_managed_role` (the roles a manager may assign/revoke) = **member + load-balancer_member ONLY**. `admin` is NEVER in this list (SCS hard constraint; the anti-escalation guarantee). `load-balancer_member` is included because the Magnum apiserver-LB tenant flow requires it (cf. D-039). `reader` and `manager` deliberately NOT added (least privilege; designation authority stays operator-side -- a domain manager cannot propagate domain-manager rights).
- Every rule falls through to `or rule:base_* or admin_required`, so cloud-admin retains all authority and service-user auth is unchanged.

**Why this approach (and NOT enforce_scope):** Charmed OpenStack deliberately does NOT implement Secure RBAC -- the keystone 2024.1/stable charm exposes NO `enforce-scope`/`enforce-new-defaults` option (verified live: both keys "not found"), and Canonical sets them False across the charms for upgrade safety. SCS scs-0302 states scope enforcement is NOT necessary for the Domain Manager persona on 2024.1 or below. So the persona is achieved purely by the policy override + `manager` role. Blast radius is LOW and bounded: the charm guarantees policy overrides alter ONLY tenant/user permissions, NEVER service users (keystone/glance/nova stay charm-managed). This is materially safer than a cloud-wide scope flip -- which is why the policy change is acceptable to combine with the explicit-bindings change in one redeploy.

**Mechanism / rollback:** `zip overrides.zip domain-manager-policy.yaml` -> `juju attach-resource keystone policyd-override=overrides.zip` (use-policyd-override already true). Lands at `/etc/keystone/policy.d/`; keystone restarts. `juju status` shows `PO:` on success, `PO (broken):` if YAML is unparseable (atomic -- whole override discarded; this is the current empty-resource state D-050 flagged). Rollback = `juju config keystone use-policyd-override=false` (reverts to charm defaults; the Juju resource cannot be deleted but disabling is sufficient).

**CRITICAL -- validation is YAML-only, not semantic:** the charm validates only that the override is parseable YAML; a misspelled rule key PASSES and silently no-ops. Acceptance MUST therefore be BEHAVIORAL (same "reports OK while broken" trap as D-046), never "PO: shows active." Gate G3: as a domain-scoped `manager`, create user/project + assign member/load-balancer_member within own domain PASS; assign `admin` or touch a different domain DENY; cloud-admin ops all still PASS.

**base_* alignment (open verification, non-blocking):** the base_* rules in `domain-manager-policy.yaml` are the SCS template's upstream-recent defaults. They MUST be diffed against THIS deploy's live defaults (`oslopolicy-policy-generator --namespace keystone` on keystone/leader) and any divergent line replaced before the policy is treated as final. The file is shippable for behavioral testing as-is; the diff hardens base_* fidelity. [LIVE-READ PENDING]

**Roosevelt / multi-DC:** the override is a single portable file replicated to every DC's keystone with zero per-DC bespoke work -- good "minimize delta to Roosevelt" properties. **Version caution (HARD):** this is the 2024.1 transitional (policy-based) implementation. The native Domain Manager persona ships in 2024.2 (Dalmatian); on any future upgrade to 2024.2+ this policy override MUST BE REMOVED (it conflicts with the native impl), except a customized `is_domain_managed_role` may be preserved. Track as an upgrade-time removal item.

**Resolves:** D-050 (use-policyd-override=true with no zip -> now supplies the reviewed zip, option (b)). **Related:** D-046 (magnum trustee; the admin_required fallthrough preserves it -- risk drops from the enforce_scope-era HIGH to LOW), D-039 (per-cluster app-creds carrying load-balancer_member), D-029 (Keystone SSO deferral), D-044 (Horizon secure-cookie; same charm, do not disturb). **Supersedes the in-session enforce_scope "Strategy 1" framing, which was found impossible on this charm and is abandoned.**

## D-052: Network-space inversion -- metal-internal carries all OpenStack service-to-service control

**Status:** ADOPTED 2026-06-25 (operator-approved). AUTHORITATIVE; SUPERSEDES the three conflicting D-052 drafts (D-052-decision-text.md, D-052-CORRECTED-architecture.md, D-051-D-052-decisions.md) that attached incompatible designs to this number. Implementation pending (bundle regen + MAAS cutover); ceph network config verified (unset / binding-derived, see ceph sub-decision).

**Context:** the as-deployed cloud collapses nearly all traffic onto a single `metal` space -- only `public`->provider, ceph `public`->storage, osd `cluster`->replication, and the geneve overlay->`data` are separated today. A prior agent's binding artifacts were corrupted (conflicting hallucinated definitions), so placement was re-derived from scratch against authoritative references, over a fresh pre-teardown endpoint inventory (`juju show-application` for all 50 apps; 446 endpoints).

**Decision:** adopt full three-plane isolation as the per-DC template:
- `provider-public` (10.12.4.0/22): public API endpoints + floating IPs.
- `metal-admin` (10.12.8.0/22, UNTAGGED): operator / MAAS / monitoring / the OpenStack `admin` API endpoint + the `""` default binding. DC-LOCAL (never crosses DC fiber).
- `metal-internal` (10.12.12.0/22, TAGGED VLAN): ALL OpenStack service-to-service control -- internal API, shared-db (Galera), amqp (RabbitMQ/RPC), certificates/secrets (Vault), cache, cluster peers, OVN NB/SB DB (ovsdb*), identity service-to-service, and the inter-service relations. The MAY-CROSS-FIBER plane.
- `fabric-data` (10.12.16.0/22): tenant geneve overlay (nova-compute:neutron-plugin, ovn-chassis:data, ovn-chassis-octavia:data, octavia:ovsdb-cms).
- `storage` (10.12.32.0/22): Ceph public -- RBD client + mon + ceph control relations.
- `replication` (10.12.36.0/22): Ceph OSD cluster replication.

Renames: provider->provider-public, data->fabric-data. storage/replication keep their names and RE-IP (.16->.32, .20->.36). `metal` SPLITS into metal-admin + metal-internal; metal-internal is a NEW tagged VLAN trunked to the LXD container hosts. The MAAS rename / re-IP / new-VLAN is a POST-TEARDOWN, PRE-DEPLOY step (the deploy gate, R1 risk).

**Rationale:** RHOSP network isolation puts API + RPC + database on the Internal API network, PXE/provisioning on its own network, public API + FIPs on External, and Ceph on Storage / Storage-Management (never on Internal API). The Charmed OpenStack charm model implements this with separate public/internal/admin spaces and binds `shared-db` and `amqp` to the INTERNAL space. So the inversion (DB/MQ/internal-API on the service plane, not the management plane) is the DOCUMENTED pattern, not a deviation; moving only the `internal` endpoint while leaving shared-db/amqp/certificates/peers on the management plane would invert the intended posture. Three planes = three trust levels / three failure domains, and it maps cleanly onto multi-DC: service-to-service may extend across DC fiber, MAAS/admin stays DC-local.

**Method:** a deterministic classifier (`classify_bindings.py`) over the live inventory (`live_bindings.json`): current provider/data/storage/replication -> rename or re-IP; current `metal` -> split by endpoint semantics; FAILS LOUD (NEEDS-REVIEW) on any unmatched endpoint rather than silently defaulting. Result: 446 endpoints classified, 0 unclassified. Distribution: metal-internal 252, metal-admin 160, provider-public 11, storage 17, replication 2, fabric-data 4. Artifacts: `target_bindings.json` (machine map) + `target_bindings_readable.txt` (audit). The bundle is regenerated MECHANICALLY from target_bindings.json (ruamel, comment-preserving), with a diff-identical verification gate vs HEAD.

**Judgment calls (resolved):**
- HA heartbeat (corosync/pacemaker; ha/hanode/peer-availability/pacemaker-remote, 57 eps) -> metal-internal for v1 (Canonical default; rides the NICs the service units already have; ~11 DC-local rings). DEFERRED to Roosevelt: a DEDICATED, redundant (knet dual-link) heartbeat ring -- corosync's Totem protocol is latency/jitter-sensitive and production guidance keeps it off contended/bursty planes (Proxmox: storage must never share corosync's network; RHEL HA enforces interconnect latency limits).
- Ceph control relations (ceph-mon client/mon/osd/mds/radosgw/rbd-mirror/bootstrap-source, ceph-osd:mon; 8 eps) -> storage, co-located with the ceph data path (RHOSP keeps Ceph off Internal API; keeps the storage subsystem self-contained per-DC). VERIFIED 2026-06-25: `ceph-public-network` and `ceph-cluster-network` are UNSET on both ceph-mon and ceph-osd, so the charms DERIVE the ceph networks from the `public`/`cluster` extra-bindings -- the bindings ARE the source of truth (public->storage = ceph public network; cluster->replication = ceph cluster network), not just relation-NIC steering. CONSEQUENCE: the re-IP needs NO bundle CIDR changes -- it is handled entirely at the MAAS/space level (the repo bundle carries no CIDR-valued `*-network` options; subnet literals appear only in comments). CORRECTION: ceph-mon:cluster -> replication (the ceph cluster-network extra-binding, matching ceph-osd:cluster); it had been mis-grouped onto storage. ceph-osd:secrets-storage stays metal-internal (OSD->Vault dm-crypt keys, a secrets relation).
- Live migration (nova-compute:migration) -> metal-internal for v1. DEFERRED to Roosevelt: a dedicated migration plane + QEMU-native TLS for tenant-memory transit.
- nova<->OVN-chassis integration: nova-compute:neutron-plugin KEPT on fabric-data (rename of `data`, not moved); ovn-chassis:nova-compute + ovn-chassis-octavia:nova-compute -> metal-admin (subordinate glue). SUBORDINATE SUBSET RULE: a subordinate's bound spaces must be a subset of its principal's. ovn-chassis (subordinate on nova-compute) binds `data` -> fabric-data for geneve, which is only legal if nova-compute is itself bound to fabric-data -- and nova-compute's ONLY fabric-data binding is neutron-plugin. An earlier draft of this decision moved neutron-plugin to metal-internal "for isolation"; that would strip nova-compute's fabric-data NIC and make the chassis geneve binding illegal, so it was REVERTED (caught by the bundle regenerator's fail-loud + subset verification). ovn-chassis-octavia:data -> fabric-data is likewise backed by octavia:ovsdb-cms -> fabric-data. Same pattern holds for cinder-ceph: it binds ceph -> storage, and its principal cinder already carries storage via cinder:ceph -> storage, so the subset holds.
- Vault: `access` -> metal-internal (services fetch certs/secrets here); `external` -> metal-admin (operator / unseal path); unused db/etcd/lb-provider -> metal-admin.
- Subordinate glue (ovsdb-subordinate; magnum/octavia Horizon dashboard plugins) -> metal-internal (keep related control together; low-stakes).
- Unused optional endpoints (legacy neutron plugins external-dns/infoblox/midonet/vsd/neutron-lb, rgw multisite master/primary/secondary/slave, nova-vmware/quantum/ironic/vgpu/ceilometer/ephemeral, vault db/etcd/lb-provider, hsm) -> metal-admin via the `""` default, and OMITTED from the bundle rather than enumerated. Rebind if/when activated.

**Roosevelt:** (1) dedicated, redundant corosync ring (heartbeat off the service plane); (2) dedicated live-migration plane + QEMU-native TLS; (3) revisit whether cross-DC service-to-service coordination on metal-internal warrants its own inter-DC plane.

**Related:** D-051 (combined in the SAME redeploy -- RBAC is keystone-only / low blast radius); supersedes the three conflicting D-052 drafts; depends on the MAAS rename/re-IP/new-VLAN gate (R1); the regenerated bundle differs substantially from BOTH repo HEAD (old space names) and the WIP bundle.yaml (which was Option-1 scope: only `internal` on metal-internal).

### AMENDMENT 2026-08-03 -- ovn-central's `""` default binding moves metal-admin -> metal-internal; and a full-bundle plane-purpose sweep

**Operator ruling, exact utterance: "ovn-central should be in metal-internal."** (Follow-on same message: "Complete a bundle sweep for any other bindings that are bound to a plane that does not match the planes intended purpose.")

**WHY.** The FIRST real deploy of D-052's multi-space bindings (2026-08-03) exposed
`charm-ovn-central` bug **LP #2044324** (multi-space-specific): ovn-central resolves its
internal/admin/public cert addresses via the `""` DEFAULT binding, and on `metal-admin` in a
multi-space model it finds no address, skips the server-cert request, and never gets a server
cert -- so its TLS OVSDB listeners never start and the OVN NB/SB Raft cluster never forms
(measured: "joining cluster", nothing on 6641/6642). Its `certificates` and `ovsdb*` endpoints
already bind `metal-internal`; moving the `""` default there makes ovn-central effectively
single-space for cert resolution (the shape of every SUCCESSFUL pre-D-052 single-`metal`
deploy) while leaving D-052 isolation intact for every other app. Pre-D-052 ovn-central used
`bindings: *internal-bindings` = `"": metal` (one space) -- so this is not restoring a dropped
binding, it is adapting ONE app to the multi-space model the charm mishandles.

**SCOPE: this amends the `""` default for `ovn-central` ONLY.** Every other app keeps its
D-052 bindings. The API charms (keystone et al.) are multi-space too and resolve certs
correctly; the defect is specific to the ovn-central charm.

**THE PLANE-PURPOSE SWEEP (all 56 apps, capture `docs/audit/binding-plane-purpose-sweep-20260803.txt`).**
Every binding was classified against the plane purposes above and verified against the relation
topology. Result: **NO other real deviation.** The
first pass flagged `openstack-dashboard:cluster` (metal-admin) as a 2nd deviation; **CORRECTED
-- it is D-072 / BUNDLEFIX-011**, a RULED, as-executed fix documented in a comment at
`bundle.yaml:667`: horizon renders haproxy's 443 backend on the cluster-binding address but only
creates apache SSL vhosts for the default+public addresses, so cluster on metal-internal =
dashboard VIP HTTPS dead. Reverting it reintroduces the bug D-072 fixed. The sweep flagged it
because it compared binding VALUES against generic plane purposes WITHOUT grepping the governing
D-NNN (which CLAUDE.md requires before touching a built surface); the lesson recorded. Five
classifier flags verified NON-issues (D-072 dashboard; octavia:ovsdb-cms dangling/unused;
ceph-radosgw public/object-store/cluster are gateway endpoints correctly placed). **Only the
ovn-central default binding changes.**

**Roosevelt delta:** the per-DC bundle template binds ovn-central's `""` default to
metal-internal from the start; a charm that resolves endpoint addresses via the default binding
needs that default on the plane its certs live on, or it hits LP #2044324 on any multi-space DC.

**CLARIFICATION 2026-08-03 (after the live rebind did NOT fix the cert issue): this amendment
is CORRECT ON ARCHITECTURAL GROUNDS and STANDS regardless of the cert bug.** ovn-central IS a
metal-internal service by D-052's own categorization ("OVN NB/SB DB (ovsdb*)" is listed under
metal-internal) -- it has no public/admin API endpoint and every functional endpoint already
binds metal-internal, so the `""` default belongs there too; metal-admin was the API-charm
convention mis-inherited by a non-API service. **What did NOT hold was the expectation that this
rebind would resolve LP #2044324.** It did not: post-bind the charm still skips its
internal/admin/public cert requests ("no local address found") and vault still issues no server
cert. Those endpoint address-types are not driven by the default binding; the cert failure is a
separate charm/interface defect (LP #2044324, server-cert request/response) and is tracked as its
own item. The binding is retained because it is the right categorization, not as a cert fix.

### RE-AMENDMENT 2026-08-05 -- ovn-central `""` default REVERTED to metal-admin (SUPERSEDES the 2026-08-03 `""` change)

**Operator ruling, exact utterance: "Once that is done then move on to the proposed move to
metal-admin."** (Ruled after a full-board binding/dataflow audit of all 56 apps + all 6 planes,
which the operator directed and reviewed; the reference is now `docs/network-space-binding-reference.md`.)
The "proposed move" is: ovn-central `"": metal-internal` -> `"": metal-admin`; its functional
endpoints (`certificates`, `ovsdb*`, `coordinator`) STAY metal-internal.

**Why the 2026-08-03 amendment's `""` change is REVERSED (measured refutation -- the D-139
ruling-note shape):** the 2026-08-03 amendment moved `""` to metal-internal on TWO stated
grounds, both now refuted:
1. **The cert-fix ground** was already withdrawn in the CLARIFICATION above (the rebind never
   issued a server cert). The real cert cause is `get_hostname()` reverse-DNS failing on the
   ISOLATED metal-internal plane, fixed INDEPENDENTLY OF BINDING by the `/etc/hosts` postruncmd
   (`scripts/dc-node-etchosts.sh`, phase-01 Step 1.2b, PROVEN 2026-08-04).
2. **The "right categorization" ground is WRONG about what the `""` default IS.** The `""`
   default is not a service-category tag; it is the **management binding** -- a unit's primary
   NIC and its juju agent <-> controller path. MEASURED 2026-08-04 (the amendment's FIRST real
   deploy): with `"": metal-internal`, all three ovn-central containers came up single-legged on
   10.12.12.x (the isolated plane), with NO route to the controller (`10.12.8.5:17070`
   UNREACHABLE), so the agent-binary download failed (`Attempt 547 ...`) and ovn-central could
   NEVER start. **ovn-central is the ONLY app of 56 without a metal-admin leg**; every other
   service -- including pure-storage (ceph-osd) and pure-DB (mysql-innodb-cluster) services --
   keeps `"": metal-admin` and rides its data plane via EXPLICIT endpoint bindings. ovn-central's
   data IS a metal-internal service (its 6 functional endpoints stay there); its MANAGEMENT
   belongs on metal-admin like everyone else's.

**SCOPE.** Reverts ONLY ovn-central's `""` default. Every other D-052 binding is unchanged and
was re-verified correct against dataflow in the 2026-08-05 audit (the only ruled exceptions are
D-072 dashboard:cluster and D-106 designate:dnsaas). **Roosevelt delta corrected:** the per-DC
template binds ovn-central `"": metal-admin` (universal) with functional endpoints on
metal-internal; the cert-CN fix is the `/etc/hosts` mechanism, not a management-binding move.

**Delivery:** bundle.yaml ovn-central binding block updated this ruling; re-staged to the DC
rack; applied to the live vr1-dc0 model (ovn-central re-homed). Reference + rationale:
`docs/network-space-binding-reference.md`.

## D-053 -- Rename network space `fabric-data` -> `data-tenant` (2026-06-25)

**Status:** ADOPTED. Naming-only change to the D-052 plane set; no architectural change.

The geneve / tenant-overlay plane introduced as `fabric-data` in D-052 is renamed to
`data-tenant` to better fit the cloud's space-identity naming. This is a SPACE-NAME change
only: the subnet (`10.12.16.0/22`), its VLAN/fabric, the role (tenant geneve overlay), and
the *semantics* of every binding are unchanged. The four bindings that read `fabric-data`
(`nova-compute:neutron-plugin`, `ovn-chassis:data`, `ovn-chassis-octavia:data`,
`octavia:ovsdb-cms`) now read `data-tenant`; the D-052 subordinate-subset reasoning holds
verbatim with the new name substituted.

**Order of record:** applied in MAAS first (the live space is `data-tenant`), then
propagated into `bundle.yaml` (6 occurrences: 4 bindings + 2 binding-model header-comment
lines) so Juju's space lookup matches the live MAAS space at deploy. The bundle QA harness
was updated in lockstep (`verify_bundle.py` ALLOWED set + the two `fabric-data`-in-`planes()`
assertions; `target_bindings.json` 4 values) and re-run clean (8/8).

**Note on this log:** the D-052 entry above is left intact and still reads `fabric-data` --
that is the historical record of the original decision; this entry records the later rename
(append-only discipline).

## D-057: provider-vip plane -- separate tagged, routed plane for public API VIPs (2026-06-29)

**Status:** SUPERSEDED by D-060 (Pattern A revert, 2026-06-29) -- the provider-vip plane is abandoned; public API VIPs return to provider-public. [Originally DECIDED.] Full record + D-003B amendment: `docs/archive/D-057-DECIDED-append.md`.

Root cause of the phase-06 FIP-unreachability blocker: API LXD containers bind `public`
to provider-public (untagged enp1s0); Juju bridges enp1s0 into a Linux bridge, starving
OVS `br-ex` so OVN's gateway ARP responder is dead. Fix: move the public API VIPs onto a
new tagged, routed plane `provider-vip` (VID 104 on the provider fabric), freeing untagged
enp1s0 for `br-ex`. FIPs stay on provider-public. This AMENDS D-003B (which co-located API
VIPs and FIPs on one L2): tenant->API is preserved via L3 routing, and it improves the
commercial hard-isolation posture (API VIPs out of the FIP broadcast domain). The durable
decision is the framework (a separate tagged/routed VIP plane); the specific subnet is set
by D-058 below.

## D-058: full plane renumber -- clean fabric-grouped /22 scheme (2026-06-29)

**Status:** SUPERSEDED by D-060 (Pattern A revert, 2026-06-29) -- the plane renumber is
abandoned; the cloud stays on the D-052/D-053 scheme. [Originally DECIDED (operator).] Full
map, jumphost ordering trap, NetBox-apex note, and the committed-foundation cascade:
`docs/archive/D-058-renumber.md`. Supersedes the D-057 minimal-delta placement of provider-vip at
10.12.24.0/22; resolves R4 (oob).

Cloud-wide re-IP for Roosevelt addressing fidelity (contiguous /22 blocks grouped by
fabric), executed as a teardown/redeploy (no in-place re-CIDR, no transient overlap).
Rotate: 8->12, 12->16, 16->20, 24->8, 64->60; 4/32/36 fixed. Result:

| plane | CIDR | VLAN / fabric | gateway |
|---|---|---|---|
| provider-public | 10.12.4.0/22 | untagged / provider | 10.12.4.1 |
| provider-vip | 10.12.8.0/22 | VID 104 / provider | 10.12.8.1 |
| metal-admin | 10.12.12.0/22 | untagged / metal (PXE) | 10.12.12.1 |
| metal-internal | 10.12.16.0/22 | VID 103 / metal | none |
| data-tenant | 10.12.20.0/22 | untagged / data | none |
| storage | 10.12.32.0/22 | untagged / storage | none |
| replication | 10.12.36.0/22 | untagged / replication | none |
| oob | 10.12.60.0/22 | f_oob | 10.12.60.1 |

VIP triple = provider-vip .8.5x / metal-admin .12.5x / metal-internal .16.5x (octets 50-60).
lib-net.sh + bundle.yaml carry D-058; the foundation scripts/runbooks are swept as each
phase is executed (living-draft discipline). NetBox apex (netbox/ipv4-prefixes-import.py) is
stale pre-D-052 and must be de-staled to D-052/053 before it can carry D-058.

## D-059: host NIC budget -- build on five physical NICs; four-NIC collapse parked (2026-06-29)

**Decision:** Build and validate v1 on FIVE active physical NICs per host
(openstack0-3) -- the live as-built reality. Defer any reduction to a four-NIC
budget until the Roosevelt inbound-server hardware sheets are confirmed.

**Live inventory (measured 2026-06-29; all four hosts identical):**

| NIC     | plane(s)                                        | sharing      |
| ------- | ----------------------------------------------- | ------------ |
| enp1s0  | provider-public + provider-vip (VID 104)        | VLAN-shared  |
| enp7s0  | metal-admin + metal-internal (VID 103)          | VLAN-shared  |
| enp8s0  | data-tenant                                     | dedicated    |
| enp9s0  | storage (Ceph public)                           | dedicated    |
| enp10s0 | replication (Ceph cluster)                      | dedicated    |
| enp11s0 | ex-lbaas -- idle, REMOVED during this MAAS work | -            |

Provider and metal already carry two planes each via VLAN tagging; data, storage,
and replication each hold a dedicated NIC -- that trio is the five-vs-four overage.
NOTE: enp7s0 reads "unconfigured" in a physical-only interface query because its L3
lives on the br-metal / br-internal bridges built atop it, not on the raw NIC -- it
is load-bearing, not spare. The genuinely idle NIC is enp11s0 (ex-lbaas), and
removing it leaves five active, not four.

**Rationale:** Five is the measured truth; building on it avoids guessing a NIC count
before the hardware is known. The four-NIC fold is a contained, well-understood change
that can be made later without reworking the whole model.

**Four-NIC collapse path (PARKED; execute only if the Roosevelt sheets confirm a
four-NIC budget):** fold storage + replication onto ONE physical NIC -- Ceph public
untagged + Ceph cluster on a tagged VLAN -- mirroring the provider/metal pattern.
Result: enp1s0 (provider+vip) / enp7s0 (metal+internal) / enp8s0 (data) / enp9s0
(storage+replication) = four. Touches three places: carve-host-interfaces.sh
(storage/replication NIC wiring + a new tagged VLAN), the bundle's Ceph public/cluster
network bindings, and the MAAS VLAN layout (a replication VLAN on the storage fabric).

**Trade-off (explicit):** separating Ceph public from Ceph cluster onto distinct
physical NICs is a performance best-practice (replication traffic does not contend with
client traffic). The collapse keeps them logically separate via VLAN but shares one
physical NIC's bandwidth -- acceptable for a test rig; a Roosevelt judgement call. Hence
parked pending the sheets, with NIC bonding as an alternative if the hardware offers it.

**Status:** Adopted 2026-06-29. Pending input: Roosevelt inbound-server NIC count.
**Related:** D-057 (provider-vip plane), D-058 (plane renumber), D-052 (space inversion).
## D-060: revert to Pattern A on the hyperconverged test cloud -- supersedes D-057 + D-058 (2026-06-29)

**Decision:** Revert the provider-NIC architecture to **Pattern A** (MAAS builds the OVS
bridge `br-ex` on provider NIC `enp1s0`; the host static lands on `br-ex`; the API LXD
containers' `public` endpoint AND ovn-chassis share that one OVS `br-ex`) and KEEP the
existing D-052/D-053 plane scheme. **D-057 (provider-vip plane) and D-058 (full plane
renumber) are SUPERSEDED** -- both were responses to a self-inflicted regression, not to a
genuine architectural requirement.

**Root cause (the pivot).** The phase-06 floating-IP-unreachability blocker was introduced by
this Claude lineage, not by Pattern A. The working pre-teardown cloud ran Pattern A. A prior
instance advised switching to **Pattern B** (raw `enp1s0`, charm-built `br-ex`) on the
rebuild. On a hyperconverged node -- where the API LXD containers and ovn-chassis share one
provider NIC -- Pattern B makes Juju build a competing Linux bridge `br-enp1s0` that captures
`enp1s0` before ovn-chassis can enslave it into OVS `br-ex`, starving `br-ex` of carrier and
darkening the floating-IP plane (OVN's gateway ARP responder goes dead). D-057 (move the
public API VIPs to a tagged `provider-vip` plane, VID 104, to free untagged `enp1s0` for
`br-ex`) and D-058 (cloud-wide renumber) were built to work around that breakage. Reverting
Pattern B removes the cause, so the workarounds are no longer needed.

**No re-CIDR.** The live MAAS is already on D-052/D-053 (verified 2026-06-29: juju / lxd /
capi-mgmt on metal-admin 10.12.8.x; openstack0 data-tenant 10.12.16.40; openstack2 storage
10.12.32.42). D-058's renumber was never applied to the running cloud. So the revert is a
teardown + Pattern A re-carve + redeploy on the EXISTING subnet scheme -- no subnet migration,
no DHCP change, no `provider-vip` plane.

**Node-role learning (Roosevelt is NOT affected).** The collision is specific to
hyperconverged topology. Roosevelt is dedicated-role: Gateway Nodes run `br-ex` / Neutron and
Controller Nodes run the API on SEPARATE machines, so no single NIC is shared between the API
containers and ovn-chassis -- the Linux-bridge-vs-OVS race cannot occur there. On Roosevelt,
Pattern B (charm-built `br-ex` on a dedicated gateway uplink) is the clean fit; the test cloud
rehearses the STACK, not the node-role split. The metal-internal VID-103 trunk, which on the
test cloud rides one shared metal NIC, becomes a real trunk to the gateway / controller /
compute access ports at Roosevelt.

**Files (this revert):**
- `bundle.yaml` -- reverted to the `bf7de5a` provider config: `public: provider-public` x11,
  VIP triple `10.12.4.5x / 10.12.8.5x / 10.12.12.5x`, all four `br-ex` MACs in ovn-chassis
  `bridge-interface-mappings` (the openstack0 MAC `52:54:00:3d:fd:54` that D-057 trimmed is
  restored). The bundle diff `bf7de5a..HEAD` was verified to be PURELY D-057/D-058, so the
  revert is a clean wholesale checkout of that file.
- `scripts/lib-net.sh` -- CIDR / VIP constants back to D-052/D-053; `PROVIDER_VIP_CIDR` /
  `PROVIDER_VIP_VID` dropped; `provider-vip` added to `STALE_SPACES` (must-be-absent guard).
- `scripts/carve-host-interfaces.sh` -- provider section is now Pattern A (`build_ovs_brex`:
  unlink commissioning link -> create OVS `br-ex` on enp1s0 -> host static on `br-ex`); the
  `enp1s0.104` / `br-prov-api` provider-vip stack and the VID-104 assertion are removed;
  metal / data statics back to D-052/D-053 (br-metal .8, br-internal .12, data .16).
- `scripts/provider-bundle-check.py` -- NEW fail-closed deploy gate (provider-public x11,
  `.4/.8/.12` triple octet 50-60, all 4 chassis MACs present); REPLACES the retired
  `scripts/d057-bundle-check.py`. Verified PASS against the reverted bundle.
- `scripts/phase-00-maas-standup.sh` -- target table + narrative retargeted to D-052/D-053;
  a dry-run now reports "no drift" against the live cloud (refuse-to-clobber guard retained
  for a fresh cloud / Roosevelt).
- `scripts/phase-00-teardown.sh` -- header + next-pointer updated (Pattern A
  re-carve / standup-verify, not re-CIDR). Logic unchanged (dynamic system_id resolution,
  capi-mgmt delete, management-substrate exclusion).
- `runbooks/phase-00-teardown-maas-reset.md` -- rewritten script-driven and current
  (deletes capi-mgmt, dynamic system_ids, D-052/D-053 CIDRs, Pattern A re-carve, OSD wipe,
  8_lbaas net removal). The prior file was pre-D-052 stale (five-space scheme, hardcoded
  system_ids, "leave capi-mgmt Ready").

**Retired (git rm):** `scripts/phase-00-maas-recidr.sh` (D-058 re-CIDR; the live cloud is
already D-052/D-053), `scripts/provider-vip-standup.sh` (D-057), `scripts/d057-bundle-check.py`
(replaced), `runbooks/jumphost-provider-vip-gateway.md` (D-057),
`runbooks/provider-vip-maas-standup.md` (D-057), `runbooks/phase-00-maas-reconfigure.md`
(D-058 reconfigure spine).

**Status:** Adopted 2026-06-29 (operator "Go"). Supersedes D-057, D-058.
**Related:** D-052 / D-053 (plane scheme, retained), D-003B (API + FIP L2 co-location,
restored), D-059 (NIC budget retained; its inventory's "enp1s0 = provider-public + provider-vip"
row is now just provider-public -- the physical five-NIC count is unchanged).

## D-061: Machine-preserving teardown via `remove-machine --keep-instance` (supersedes D-055)

**Status:** ADOPTED 2026-06-30. Supersedes D-055. Documented behavior; the `--keep-instance`
path is UNVALIDATED on this virsh-pod MAAS at adoption (validate via the canary, below).

**Problem (observed 3x in one session):** `juju destroy-model` DECOMPOSES the MAAS pod-composed
openstack0-3 machines -- they are deleted from MAAS and the carve is lost, forcing a full
reenroll + re-carve. This happened with `--destroy-storage --force`, and AGAIN with
`--force --no-wait` and NO storage flag at all. So D-055's diagnosis ("omit --destroy-storage
and the machines release to Ready") is WRONG/insufficient: the storage flag is not the trigger;
`destroy-model` itself releases the instances, and a MAAS pod auto-decomposes a released
pod-composed VM.

**Root cause / mechanism:** machine retention is NOT a `destroy-model` option -- it is a
`remove-machine` option. Juju 3.6 `remove-machine` reference: a machine can be removed from the
Juju model without affecting the corresponding cloud instance via `--keep-instance`.
`destroy-model` has no machine-preserving flag. The openstack0-3 VMs are MAAS pod-composed
(VM host `lxd`), so any *release* auto-decomposes them; `--keep-instance` is the only documented
way to detach-from-model without release.

**Decision:** two teardown scripts, by intent:
- `scripts/phase-00-teardown-release.sh` -- KEEP machines. Per host:
  `juju remove-machine <id> -m openstack --keep-instance --force --no-wait` FIRST (detach,
  preserve the MAAS instance + carve), HARD-VERIFY each host survived in MAAS (FAIL LOUD if any
  decomposed), THEN `juju destroy-model openstack --release-storage --force --no-wait --no-prompt`.
  Result: hosts stay Deployed + carved, reusable with no reenroll/recarve.
- `scripts/phase-00-teardown-destroy.sh` -- DECOMPOSE machines (the from-scratch path):
  `juju destroy-model openstack --destroy-storage --force --no-wait --no-prompt`; reenroll+carve after.
Both default to dry-run and require typing the model name at a /dev/tty gate before mutating.

**First-run safety (UNVALIDATED caveat):** `--keep-instance` is documented but had not been
exercised on this virsh-pod MAAS when adopted. `phase-00-teardown-release.sh --apply --canary`
removes ONLY openstack0, verifies it survived in MAAS, and STOPS -- run that before trusting the
all-four path. The script also hard-verifies survival after every remove and exits 1 if any host
decomposed (so a wrong assumption fails loud, not silently).

**Roosevelt:** bare-metal nodes are enlisted, not pod-composed, so release does not auto-decompose
-- the keep path is simpler there. If a MAAS pod is ever used at Roosevelt, configure it to not
decompose-on-release (or keep erase-on-release OFF) and carry the --keep-instance discipline.

**Related:** D-055 (superseded), D-018 (MAAS-release-direct teardown intent), DOCFIX-057
(old phase-00-teardown.sh deprecated -- its "releases to Ready" premise was wrong).

## D-062: mysql-innodb-cluster deploys at target unit count; no single-unit seed-then-scale

**Status:** ADOPTED 2026-06-30. Refines the D-009 unit-count guidance for this charm.

**Context:** to avoid the 3-way Group-Replication formation race that twice parked a unit `blocked`
with a metadata half-join, a single-mysql seed (`num_units: 1`, ramp to 3 later) was tried. It
FAILED: the unit parked `blocked` `'cluster' missing, Instance not yet configured for clustering`.

**Root cause:** the OpenStack reactive `mysql-innodb-cluster` charm bootstraps the InnoDB cluster
(`create_cluster`) from inside its `cluster` PEER-relation handler. With `num_units: 1` the peer
relation never receives a `-relation-changed` carrying a peer, so the bootstrap handler never runs.
Live flags confirmed: `certificates.available`, `local.cluster.user-created`, leader passwords set,
but NO `cluster.available`/bootstrapped flag; the log showed only update-status/db-router hooks
cycling -- no `create_cluster` attempt. This is NOT the Canonical "Charmed MySQL" charm (different
interface) which DOES support single-then-scale; the OpenStack charm's README only documents
`-n 3` deploy.

**Decision:** deploy `mysql-innodb-cluster` at its target unit count (3) from cold. Peer-driven
bootstrap is the charm's happy path: with 3 units brought up together, the cluster forms and (this
deploy) reported clean -- `status: OK`, 1 R/W + 2 R/O, `instanceErrors=[]` on all members. The
single-unit seed-then-ramp idea is RETIRED for this charm.

**Half-join distinction (carried):** the metadata half-join (`instanceErrors: ["...not managed by
InnoDB cluster. Use cluster.rescan()..."]`) is an artifact of MID-LIFE instance addition, not cold
bootstrap. If it recurs during a scale operation, the documented fix is
`cluster.rescan({addInstances:"auto"})` via mysqlsh as `clusteruser` (NOT the charm's bare
`cluster-rescan` action, which reports-but-does-not-adopt). Cold 3-unit deploy did not hit it.

**As-executed this deploy:** the running model was (mistakenly) deployed single-mysql, then
corrected in place: `juju remove-unit mysql-innodb-cluster/1` (a stuck mis-placed unit), then
`juju add-unit mysql-innodb-cluster -n 2 --to lxd:1,lxd:2` (real live machine ids 0-3 this cycle).
Cluster formed clean. The committed bundle is reverted to `num_units: 3` / `to: [lxd:8,9,10]`
(matching its own `machines:` 8-11 declaration; the live 0-3 numbering was a deploy-time artifact).

**Related:** D-009 (3 units on Roosevelt), BUNDLEFIX (bundle reverted to 3-unit).

---

## D-063: RESOLVED / CLOSED (2026-07-03) -- tighten the capi-mgmt-cluster security group ingress for Roosevelt

**Status:** PROPOSED / OPEN (recorded 2026-07-01) -> RESOLVED / CLOSED 2026-07-03 (applied + functionally verified; see the RESOLVED/CLOSED amendment below).

**Context:** the phase-06 in-cloud CAPI management VM (`capi-mgmt-v2`, D-035) sits behind the
`capi-mgmt-sg` security group. As built by phase-06 (a CAPI/cluster-api default posture), that SG
opens BOTH `tcp/6443` (apiserver) AND `tcp/22` (ssh) to `0.0.0.0/0`. Phase-07 Step 7.1 relies on
this: the conductor reaches the apiserver via the FIP with no per-conductor rule (which is why the
old hardcoded `10.12.4.76/32` rule-add was dropped in DOCFIX-063).

**Question:** on v1 (single-DC virtual rehearsal) the FIP is the sole access point and `0.0.0.0/0`
on 6443/22 is acceptable. For ROOSEVELT (commercial, multi-DC, HARD tenant isolation) an apiserver
and ssh open to any source that can route to the FIP is too broad.

**Options (unresolved):**
- (a) Tighten `tcp/6443` ingress to the magnum-conductor's measured source (and any operator
  tooling), and `tcp/22` to the ops/jumphost source only. Requires knowing the conductor's
  post-NAT source as the mgmt VM sees it (measured, not the tenant/provider literal).
- (b) Front the mgmt apiserver with a scoped ingress (e.g. a dedicated LB/SG chain) rather than a
  raw FIP with a wide SG.
- (c) Leave wide on the rehearsal, and make the tightened SG a phase-06 build step on Roosevelt.

No decision made -- recorded as an open hardening item (cf. D-043, D-050, also pending). Whichever
option, phase-07 Step 7.1 stays verify-first (it does not depend on the SG being wide; if 6443 is
NOT already permitted it measures the source and adds exactly that rule).

**Related:** D-035 (the mgmt VM), D-039 (per-cluster app-creds), phase-06 (SG creation),
phase-07 Step 7.1 (DOCFIX-063 verify-first).

### D-063 -- AMENDMENT (2026-07-03): sources MEASURED; option (a) adopted as (a-amended)

Measurement (jumphost session 2026-07-03; committing this amendment constitutes adoption):
1. Route: magnum/0 -> FIP 10.12.7.222 is ON-LINK via its provider leg (src 10.12.4.154) -- the
   FIP lives inside 10.12.4.0/22, so no router/SNAT exists on that path.
2. 170s SYN capture on the mgmt VM (dst 6443, pod-net 10.1.0.0/16 and loopback excluded),
   with a deliberate operator kubectl trigger mid-window. Observed sources:
     3x 10.12.4.154  -- magnum conductor health poll, ~60s cycle (route prediction CONFIRMED)
     2x 10.12.4.1    -- the OPERATOR kubectl trigger, arriving MASQUERADED to the provider
                        gateway by the upstream router (jumphost 10.17.11.246 is erased)
     2x 10.20.0.207  -- the VM hairpinning its own FIP (DNAT returns with the fixed IP as
                        source); SG rules DO apply to hairpinned traffic
   Falsified along the way: "the conductor holds a persistent apiserver connection" -- it is
   transient per-cycle; instantaneous `ss` snapshots cannot capture the client set. Use the
   window-capture method (tcpdump SYN filter over >=2 poll cycles) for any future SG sourcing.

ADOPTED RULE SET (a-amended), 6443 ingress on capi-mgmt-sg:
  - magnum unit provider addresses, /32 each, derived DYNAMICALLY from Juju at apply time
    (PATTERN-1; per-UNIT -- Roosevelt's 3-unit magnum contributes 3 rules; enumerate, never
    assume one). Value: blocks tenant-workload sources (tenant egress arrives as each tenant
    router's own 10.12.4.x SNAT address, which is NOT in the set).
  - 10.12.4.1/32 -- the operator path, EXPLICITLY DOCUMENTED AS COARSE: upstream masquerade
    makes this equivalent to "any source routed in via the gateway". The SG cannot
    distinguish the jumphost; do not pretend otherwise in docs or reviews.
  - 10.20.0.0/24 (capi-mgmt-subnet) -- self/hairpin traffic; omitting it breaks on-VM use of
    the external endpoint silently.
SSH (22) ingress: 10.12.4.1/32 only (same coarseness caveat).
NOT adopted: 0.0.0.0/0 stays out; option (b) (LB in front) remains the Roosevelt-scale
candidate if magnum unit churn makes /32 maintenance noisy.
Roosevelt note: the 10.12.4.1 masquerade is a VR0 substrate artifact. The rule set MUST be
re-derived from a fresh window capture in every environment -- never copied forward.
Apply is a SEPARATE gated mutation with its own verify-before/after block; this amendment
records the decision, not the change. **Status: ADOPTED (a-amended) on commit; apply pending.**

### D-063 -- RESOLVED / CLOSED (2026-07-03): applied and functionally verified

APPLIED via scripts/d063-apply.sh --apply (DOCFIX-084/085): four ingress rules created and
read-back-gated BEFORE the two wide 0.0.0.0/0 rules were removed by measured ID (no window
where the conductor source was unauthorized). Final SG state (measured): 6443 from
10.12.4.154/32 (conductor), 6443 from 10.12.4.1/32 (operator, coarse), 6443 from 10.20.0.0/24
(hairpin), 22 from 10.12.4.1/32; wide 22/6443 ingress gone (only default egress remains).

FLOW MODEL (corrected + doc-backed). Earlier in-session I twice mischaracterized the
conductor->mgmt traffic as a "continuous health poll"; both were WRONG. As-built (phase-07:
/etc/magnum/kubeconfig server = mgmt FIP 10.12.7.222:6443, client-cert auth) plus upstream
docs (StackHPC magnum-capi-helm deep-dive; openstack magnum-capi-helm install guide) establish
the true model: magnum-conductor is a Kubernetes CLIENT of the mgmt apiserver, acting ONLY on
cluster lifecycle events (create/update/delete -> helm install/upgrade + watch). Connections
are per-operation and short-lived (client-cert kubeconfig, no held session), so the path is
IDLE at rest and busy only during ops. This fully explains why idle established-state and SYN
snapshots repeatedly showed nothing -- expected, not a fault. Required routes per StackHPC:
conductor -> mgmt cluster; mgmt cluster -> OpenStack external net; mgmt cluster -> workload API
LBs. Only the first traverses capi-mgmt-sg ingress; the rule set covers it.

VERIFICATION (proof by exercise, through the tightened SG):
- POSITIVE, load-bearing: R1 -- a live TCP handshake from magnum/0 to 10.12.7.222:6443
  (phase-07 Step 7.1 primitive: exec 3<>/dev/tcp) returned TCP-OK. A handshake cannot complete
  if the SG drops the source; conductor source measured = 10.12.4.154 (= the applied /32).
- POSITIVE, functional (gold standard): C3 -- forced a real op (openstack coe cluster resize
  beta-cluster 1->2 as the tenant); it reached UPDATE_COMPLETE / HEALTHY / node_count=2. A
  magnum-capi-helm resize CANNOT complete unless the conductor helm-upgraded the manifest on
  the mgmt apiserver (10.12.7.222:6443) through the SG and CAPI built the worker. The
  successful state transition is dispositive proof the path carries real helm ops post-tighten.
- NEGATIVE (isolation value): a beta tenant POD -> 10.12.7.222:6443 TIMED OUT (blocked). Tenant
  workloads egress as their router's own 10.12.4.x SNAT, which is NOT in the allow set. This is
  the isolation the ruling exists to provide.
- OPERATOR: kubectl get --raw /readyz via the mgmt kubeconfig returned ok (10.12.4.1 coarse
  rule, as designed).
- NOT RELIED ON: SYN packet captures (V4/R5) came back empty -- the transient per-op flow is
  the wrong target for snapshot/window SYN sniffing (learned 3x this thread; established-state
  and functional-outcome are the correct instruments). A conductor-log corroboration probe (C2)
  was empty because it queried journald; charmed magnum logs to
  /var/log/magnum/magnum-conductor.log. Neither is needed: C3's functional success supersedes
  wire-level corroboration (industry practice closes on functional verification of the flow).

BETA left at node_count=2 (UPDATE_COMPLETE/HEALTHY) -- the resize doubles as previously-untested
resize acceptance coverage; a change from the count=1 acceptance baseline, retained deliberately.

Roosevelt carry-forward: the rule set is per-UNIT and per-ENVIRONMENT -- 3-unit magnum yields 3
conductor /32s; the 10.12.4.1 operator collapse is a VR0 masquerade artifact; re-derive with
d063-apply.sh (live per-unit derivation) in every environment, never copy rules forward.

**Status:** RESOLVED / CLOSED (applied + functionally verified 2026-07-03). Supersedes the
"apply pending" state above. **Related:** DOCFIX-084/085 (the applier + its stderr-capture
fix), phase-07 Step 7.1 (the TCP primitive reused for R1), D-035/D-036 (mgmt-cluster model).


---

## D-064: Reconcile D-051 to the SCS Domain Manager standard (scs-0302); fix create-op policy templating

**Status:** ADOPTED 2026-07-01 (phase-08 keystone-policy blocker). Reconciles D-051 to the
authoritative SCS reference and discharges D-051's [LIVE-READ PENDING] base_* alignment gate.
Behavioral acceptance = Magnum trustee create + tenant G3 (below).

**Context / trigger:** phase-08 workload-cluster create failed at the FIRST step --
create_trustee_and_trust -> identity:create_user HTTP 403. magnum_domain_admin holds Admin on
the magnum domain (correct per Magnum docs + D-046) and was authenticated (401 would be auth;
403 = authorization). Root cause: this cloud's charm-rendered /etc/keystone/policy.json
(old-style defaults, enforce_scope=False) defines the create-op helper
admin_and_matching_user_domain_id with legacy templating domain_id:%(user.domain_id)s, which
does NOT resolve for create_user on Caracal (keystone populates target.user.domain_id, not the
bare user.domain_id). The admin-on-domain create branch silently evaluates false and only
cloud_admin passes -- the same defect that blocked tenant domain-admin self-service in the
2026-06-22 northwind rehearsal. One bug, two symptoms (Magnum trustee + tenant self-service).

**Finding (why D-051 alone did not fix it):** D-051 deliberately deviated from scs-0302 -- it
dropped Section A (base_*) and pointed fallthroughs at the LIVE helpers by name, including the
broken admin_and_matching_user_domain_id, and dropped the `or rule:admin_required` tail.
Consequently D-051's create_user fallthrough inherited the broken helper; only its manager
branch (correct token.domain.id:%(target.user.domain_id)s templating) worked. Attaching D-051
as-staged would have fixed tenant self-service (manager) but NOT Magnum (admin-on-domain trustee).

**Decision:** reconcile to scs-0302 by correcting the create-op fallthroughs to the cloud's
CORRECT %(target.*.domain_id)s helpers (mirroring the standard's base_create_*) and complete the
base_* alignment against the now fully-read live policy. Exactly 7 rules change:
- create_user / create_project / create_group: fallthrough admin_and_matching_<x>_domain_id
  (broken %(<x>.domain_id)s) -> admin_and_matching_target_<x>_domain_id (correct
  %(target.<x>.domain_id)s). BUG-FIX; restores the documented admin-on-domain create path and
  unblocks Magnum. Confined (domain match); does NOT touch tenants (they hold manager, not
  domain-admin).
- add_user_to_group: admin_and_matching_group_domain_id -> admin_and_matching_target_group_domain_id.
  ALIGNMENT (restore live default; D-051 had drifted to the broken helper).
- list_users / list_projects / list_groups: fallthrough admin_required (D-051 conservative
  [PENDING-LIVE-READ] guess, BROADER than live) -> cloud_admin or admin_and_matching_domain_id
  (the live default). ALIGNMENT; behavior-preserving; removes an unintended widening.
list_roles keeps admin_required (its live default). The manager persona (Section B),
is_domain_manager = role:manager, and is_domain_managed_role = member + load-balancer_member
(no admin, no manager -- anti-escalation) are UNCHANGED from D-051 / scs-0302.

**Consequence:** with create_user's admin fallthrough fixed, magnum_domain_admin creates
per-cluster trustees via the standard admin-on-domain path -- NO manager grant on the magnum
service account is required (an in-session Option A now moot). Magnum keeps its documented
trustee setup; the policy fix alone unblocks it.

**Validation:** oslo.policy parses all 37 rules (the charm validates YAML only -- a malformed
rule passes YAML and silently no-ops, cf. D-046 "reports OK while broken"); YAML + ASCII clean;
malformed-connector lint clean; every fallthrough tail diffed against the live effective default.
Behavioral gate (the real acceptance): (a) Magnum -- re-run phase-08 8.1, trustee create_user
passes, cluster converges; (b) tenant G3 -- a manager-on-domain user self-services user/project
+ member/load-balancer_member within its domain (PASS), is denied admin/cross-domain (DENY),
cloud-admin unaffected.

**Known SCS limitation (carried, per standard):** the manager persona grants list_domains and
list_roles cloud-wide (needed to resolve domain/role names). A tenant manager can ENUMERATE
domain names/ids + role names (not access resources). Inherent to the scs-0302 transitional
policy (upstream RBAC-scoping of domain list is a pending fix); the native 2024.2 persona closes
it. On upgrade to 2024.2+ this override MUST be removed (scs-0302 + D-051 caution).

**Mechanism:** zip overrides.zip domain-manager-policy.yaml -> juju attach-resource keystone
policyd-override=overrides.zip (use-policyd-override already true). keystone PO (broken) -> PO:.
Rollback: juju config keystone use-policyd-override=false. Full attach + G3 procedure in
runbooks/appendix-C-identity-rbac.md.

**Roosevelt:** single portable policy file replicated per-DC keystone; the attach + G3 gate
becomes a deploy step. Remove on any 2024.2+ upgrade.

**Related:** D-051 (reconciled here), D-046 (magnum trustee domain), D-039 (per-cluster app-cred
roles), D-050 (resolved by supplying the zip), scs-0302-w1 (the authoritative standard),
appendix-C (identity/RBAC reference). **Supersedes:** D-051's [LIVE-READ PENDING] gate (discharged).


---

## D-065: Fix identity:create_trust non-resolving policy template (Magnum multi-tenant cluster blocker)

**Status:** ADOPTED 2026-07-02. Extends the D-064 defect-class fix to the keystone trust family.
Behavioral acceptance = a tenant service identity clears create_trust and the cluster converges
(pending re-attach + retest at adoption).

**Context / trigger:** multi-tenant validation (tenant acme). A tenant service identity (acme-svc,
app cred, holding member + load-balancer_member on acme-prod) creating a Magnum cluster failed at
create_trust with identity:create_trust 403 -- AFTER D-064 unblocked create_user (the trustee user
was created). The SAME 403 reproduced (a) for admin via password auth, and (b) via a DIRECT
`openstack trust create` with trustor == caller == self, no magnum in the path. That proves it is
identity-independent and not a magnum bug.

**Root cause:** the charm-rendered base policy.json defines
identity:create_trust = "user_id:%(trust.trustor_user_id)s" -- the legacy NON-target-prefixed
template that does NOT resolve on Caracal (keystone populates target.trust.trustor_user_id). The
rule evaluates user_id == "" -> always false -> every trust create 403s regardless of caller. Same
defect class as D-064 (create_user's %(user.domain_id)s vs the resolving %(target.user.domain_id)s).
Keystone's OWN shipped default is the target-prefixed form:
"user_id:%(target.trust.trustor_user_id)s".

**Masking:** create_trust was never reached before D-064 because create_user 403'd first. The
2026-06-09 CREATE_COMPLETE was on the pre-2026-06-11-teardown cloud; this redeploy's create_trust
had never been exercised until D-064 opened the path.

**Causation proof (verify-before-fix):** with use-policyd-override=false (base policy only), the
direct trust create STILL 403s -> the D-064 override is NOT the cause; the base charm policy owns
the bug. Override re-enabled immediately (PO: restored) -- the cloud was not left without the
manager persona.

**Decision:** add identity:create_trust = "user_id:%(target.trust.trustor_user_id)s" to the D-064
override (policies/domain-manager-policy.yaml). Exactly the keystone-shipped default, and
consistent with this cloud's own already-correct sibling rules (list_trusts_for_trustor/trustee).
No admin fallback (matches the shipped default; trusts are self-delegation only). No manager branch
(create_trust is universal self-delegation, not part of the domain-manager persona).

**Scope (evidence-gated; NOT a blanket family sweep):** create_trust is the ONLY demonstrably
broken trust rule. list_trusts_for_trustor / list_trusts_for_trustee already carry the
target-prefixed form (match shipped). get_trust / delete_trust / list_trusts / list_roles_for_trust
/ get_role_for_trust are live "" (empty) but code-guarded in keystone (_trustor_trustee_only;
delete checks user==trustor or is_admin) -> functional, not broken, not blockers; left unchanged.

**Second-check clearance:** keystone's create_trust also enforces _require_trustor_has_role_in_project
(trustor must hold each delegated role on the project). Magnum delegates context.roles (the caller's
token roles), which by construction are a subset of what the caller holds on the scoped project ->
passes. acme-svc holds member + load-balancer_member on acme-prod; confirmed clear on retest.

**Validation:** oslo.policy parses all 38 rules; YAML + ASCII + connector lint clean.

**PROPOSED / OPEN (separate hardening, NOT actioned here):** live identity:list_trusts = "" lets any
authenticated user enumerate ALL trusts (trustor/trustee/project relationships) cloud-wide -- an
info-disclosure the newer keystone default tightens to
"rule:admin_required or (role:reader and system_scope:all)". This is a security-posture change, not
a functional blocker (create/get/delete are code-guarded), so it is recorded as an open item to rule
on rather than bundled into this blocker fix. If adopted, add list_trusts (and optionally the read
rules) at the shipped defaults in the same override.

**Roosevelt:** carry the create_trust override with the rest of the D-064 policy. The base-charm
non-resolving-template defect is upstream-worthy -- report against the keystone charm's rendered
policy.json for Caracal (2024.1). Remove on 2024.2+ (native secure-RBAC ships the resolving defaults).

**Related:** D-064 (same defect class; same override file), D-039 (the trustor roles that satisfy the
trustor-has-role check), D-046 (magnum trustee domain), D-035 (mgmt cluster), appendix-D (trust
model). **Revises:** appendix-D section D.3 -- the role-delegation hypothesis is REFUTED (a
clean-role tenant identity still 403'd; the cause was the non-resolving policy template, not role
delegation). appendix-D to be corrected on finalize.


---

## D-066: Tenant account model (Option-3 split); cluster-create requires PASSWORD auth

**Status:** ADOPTED 2026-07-02. Standard per-tenant identity set, used from the first tenant so
v1 and Roosevelt onboard identically (no interim model to retrofit). Trust path validated live
through create_user + create_trust + cert-gen entry (see D-064/D-065); full cluster completion is
gated on D-067 (Barbican/Vault substrate).

**The per-tenant accounts (operator creates the domain + manager; manager creates the rest):**
- `<client>-domain-admin` -- `manager` on the domain (SCS Domain Manager, D-051/D-064). Tenant
  IAM self-service. Operator-provisioned; password handed over.
- `<client>-cluster` -- `member` + `load-balancer_member` on `<client>-prod`. PASSWORD auth only.
  Sole purpose: `coe cluster` lifecycle (this is the identity that mints the Keystone trust). Also
  owns whatever magnum-internal creds the driver mints under it (the per-cluster CAPO child cred,
  D-039). Manager-created.
- `<client>-svc` -- `member` + `load-balancer_member` on `<client>-prod`. UNRESTRICTED app cred,
  for tenant-authored non-trust automation (CI/pipelines). NOT used for cluster-create. Manager-created.

**Why password (not app cred) for cluster-create -- the hard constraint:** keystone on this build
blocks trust creation from application-credential tokens unconditionally. Source
(keystone/api/trusts.py `_check_application_credential`, read live 2026-07-02): if the token method
is `application_credential` it raises "Using method 'application_credential' is not allowed for
managing trusts" -- and the docstring states this applies "regardless of the 'unrestricted' flag"
(this build is STRICTER than upstream, which exempts unrestricted). Confirmed live: an unrestricted
app cred (which minted a child cred, so genuinely unrestricted) still hit this block; the same
identity via PASSWORD passed it. So the cluster-creator must be password-authenticated.

**Security rationale (this is the correct side of the control, not a workaround):** the block exists
to stop a single-project app-cred token from extending the trust delegation chain beyond its scope.
A password token is full-user scope, so a trust minted from it delegates only what the user already
holds (member + load-balancer_member on its own project) -- no escalation, confined to the tenant.
Splitting `-cluster` (trust-capable password, cluster ops only) from `-svc` (app cred, everything
else) isolates the trust-capable credential to one identity doing one job.

**REJECTED:** `CONF.security_compliance.allow_insecure_application_credential_trust_escalation`
(the escape hatch in `_check_application_credential`). Enabling a setting named "insecure" +
"trust_escalation" -- which defeats a control designed to contain a compromised tenant credential --
undercuts the hard-isolation thesis of the whole build. Not adopted.

**Roosevelt:** onboard every tenant with this three-account set. Script it (scripts/tenant-onboard.sh,
DRAFT this session). CAPO-cred ownership under `-cluster` confirmed acceptable (magnum-internal,
created/destroyed with the cluster) -- the empirical "whose identity mints the child cred" check
is pending the D-067 substrate fix (cluster-create dies at cert-gen before the driver's mint step).

**Related:** D-051/D-064 (manager persona), D-065 (create_trust template fix), D-039 (trustor roles).
**Revises:** appendix-D D.3 (the app-cred-creator assumption is wrong -- corrected in appendix-D this session).

---

## D-067: barbican-vault -> Vault (vault-kv) must use the metal-internal plane (live drift; the cert-gen blocker)

**Status:** ADOPTED 2026-07-02 (fix pending next session -- a live rebind, gated). This is the defect
that blocks multi-tenant cluster COMPLETION (cluster-create clears trust, then dies generating certs).

**Symptom chain:** tenant cluster-create (password identity) cleared create_user + create_trust,
then `CREATE_FAILED: Failed to create certificates`. Root cause traced through magnum -> Barbican
(`POST /v1/secrets` 500) -> castellan vault_key_manager `_build_auth_headers` -> Vault AppRole login
rejected. Vault's RAW error (not an HTTP code -- the authoritative read):
`source address "10.12.8.176" unauthorized through CIDR restrictions on the secret ID`.

**Root cause:** barbican reaches Vault at Vault's METAL-ADMIN address (vault_url=http://10.12.8.190:8200,
egress src 10.12.8.176 via eth2). Vault's barbican-vault AppRole binds secret_id to the metal-internal
CIDR (where service-to-service traffic belongs, D-052/D-053). Off-plane source -> Vault rejects. This
is NOT secret_id expiry: `juju run vault/leader refresh-secrets` DID rotate the secret_id (barbican.conf
re-rendered, barbican restarted) and the login STILL failed with the CIDR error. The earlier
"expired TTL" reading was WRONG -- corrected here.

**The bundle is CORRECT; the LIVE env drifted.** bundle.yaml already binds every secrets-path endpoint
to metal-internal: vault `secrets: metal-internal` (L130), barbican `secrets: metal-internal` (L667),
barbican-vault `secrets-storage: metal-internal` (L700). So a redeploy from the bundle would NOT have
this defect. The live deployment's effective binding for the barbican<->vault secrets path resolves to
metal-admin -- a drift from the bundle (mechanism: TBD by read-only `juju show-application` binding +
relation-data diagnosis next session; candidate causes: bindings added to bundle post-deploy and never
applied live, or the subordinate address-advertisement not following the bound space).

**Fix (rebind, do NOT widen the CIDR):** reconcile the live binding to the bundle so barbican egresses
from its metal-internal address (10.12.12.110), which the AppRole CIDR already trusts. Widening the
AppRole CIDR to metal-admin is REJECTED: it loosens a security control, legitimizes east-west traffic
on the wrong plane, gets reverted by the charm, and leaves the next Vault consumer exposed. Next-session
sequence: (1) read-only binding diagnosis (`juju show-application vault barbican barbican-vault`,
spaces<->subnets map); (2) gated `juju bind` / relation refresh to metal-internal; (3) re-run
refresh-secrets if needed; (4) confirm barbican AppRole login = HTTP 200 from a metal-internal source;
(5) re-run tenant cluster-create -> cert-gen clears.

**Related:** D-052/D-053 (six-plane network; metal-internal carries east-west service traffic),
D-057 (prior topology-didn't-follow-binding defect -- same family). **Corrects:** the in-session
"secret_id TTL expiry" hypothesis (refuted by the refresh-secrets test).

### D-067 -- AMENDMENT (2026-07-02, post-fix): corrected mechanism; FIXED live; CLOSED for v1

The "bundle correct / live drifted" framing above is WRONG and is corrected here (append-only
discipline; the body above stands as the historical record). Root cause, read from charm source at
the exact vendored versions:

1. Live bindings were NEVER drifted -- `juju show-application` matched the bundle exactly (all
   secrets-path endpoints metal-internal). The drift was one relation-data key: vault/0 advertised
   binding-correct ingress (10.12.12.117) but an explicit `vault_url` of http://10.12.8.190:8200.
2. Mechanism: charm-vault `send_vault_url_and_ca()` (re-fires on every non-update-status hook)
   computes vault_url from the `access` binding AND vault_url_external from the `external` binding,
   publishing BOTH when they differ. The vault-kv interface (vendored commit 6f7848c, per
   src/build.lock) IGNORES the `remote_binding` selector (deprecated, LP#1895185) and writes the
   single `vault_url` key on ALL relations -- so the external publish CLOBBERS the access URL for
   every kv consumer. Simultaneously the AppRole secret_id is CIDR-bound to the CONSUMER's
   secrets-relation ingress /32 (10.12.12.110/32) -- self-inconsistent whenever access != external.
3. Why external resolved to metal-admin: the bundle OMITS vault's `external` binding, so it falls
   to the `''` default (metal-admin). D-052's judgment call ("vault external -> metal-admin,
   operator/unseal path") rested on a mistaken premise: in this charm the `external` endpoint's only
   functional use is this kv-URL advertisement (plus an inert VIP check); operator/unseal access
   does not traverse it, and the listener binds [::]:8200 regardless. So a fresh deploy from the
   bundle REPRODUCES the failure -- this was a bundle+charm defect, not live drift.

Fix as executed (2026-07-02, gated): `juju bind -m openstack vault external=metal-internal`. The two
computed URLs became equal, the charm's own equality guard suppressed the second publish, relation
data flipped to http://10.12.12.117:8200 within one poll cycle, barbican-vault re-rendered
barbican.conf, and no refresh-secrets was needed. Validated end-to-end: AppRole login HTTP 200 from
the authentic source (kernel route src 10.12.12.110), then a full admin `openstack secret`
store/get/payload-compare/delete round-trip through the exact POST /v1/secrets path that 500'd.
Bundle hardened with BUNDLEFIX-007 (`external: metal-internal` + comment); the double-publish is
present on charm-vault master as of 2026-07-02, so the guard is permanent, including the D-068
1.16 pin.

Observation logged during validation (not actioned): barbican's `Secret href` renders as
https://None:9312/... (host_href unset/None). Cosmetic for barbicanclient/castellan consumers
(clients extract the UUID and dial their own catalog endpoint) -- stage-6 cluster cert refs will
exercise it for real; investigate only if cert-ref retrieval misbehaves.

**Status:** CLOSED for v1 (fixed + validated live 2026-07-02). **Adds:** BUNDLEFIX-007.
**Corrects:** the "bundle is CORRECT; the LIVE env drifted" mechanism above. **Related:** D-052
(premise correction noted), D-068 (carry BUNDLEFIX-007 through the 1.16 pin).

---

## D-068: PROPOSED -- Vault substrate hardening (Roosevelt)

**Status:** PARTIALLY RULED (item 2 RULED 2026-07-21; items 1 and 3 open). Filed PROPOSED /
OPEN 2026-07-02 -- grouped substrate items surfaced while diagnosing D-067. None block v1
tenant work; all are Roosevelt-durability items. Item 2 RULED 2026-07-21 (GA-R5): question
as presented = "rule Vault listener TLS (barbican<->Vault secret traffic currently crosses
metal-internal as cleartext HTTP)?"; operator selection, exact utterance: "Require for
Roosevelt (Recommended)". Vault listener TLS is a ROOSEVELT BUILD REQUIREMENT (lands at
Roosevelt Vault standup with its own runbook step + verification); VR1/testcloud stays
as-is under rehearsal posture. Item 3 RULED 2026-07-21 (GA-R5): question as presented =
"rule the AppRole secret_id lifecycle posture (TTL audit + renewal + proactive auth
probe)?"; operator selection, exact utterance: "Adopt + probe on VR1 now" (NOT the
recommended lean -- operator chose the wider scope). All three legs are Roosevelt build
requirements (TTL audit at standup; long/renewable TTLs or charm auto-renewal; proactive
per-consumer Vault auth probe in standing health checks), AND the auth probe additionally
ships into `cloud-assert.sh` for the CURRENT deployment (T4-style login check; must no-op
gracefully while the VR1 substrate carries no Juju/Vault yet -- Stage 5 arrival). Delivery
queued this session: script change + harness + changelog per the change-delivery loop.
Item 1 (version migration) additionally needs the re-scoped plan per the 2026-07-05
amendment before it can be presented. Only item 1 now remains open on this entry.

1. **Vault version.** Live is 1.8.8 (charm 1.8/stable) -- EOL. bundle.yaml pinned to 1.16/stable
   this session (D-068). Live upgrade is a MAJOR operation (multi-minor jump; unseal keys ready,
   storage-format compat, re-unseal after restart, ideally rehearsed) -- NOT a casual `juju refresh`.
   Verify the channel exists and rehearse before applying live.
2. **Vault over cleartext HTTP.** vault_url is `http://...:8200`; barbican<->Vault secret writes
   (including cluster CA private keys) cross metal-internal unencrypted. Enable Vault listener TLS
   for Roosevelt.
3. **AppRole credential lifecycle.** `refresh-secrets` exists as a manual action -> implies no
   automatic secret_id renewal. Audit secret_id TTLs across all vault-kv consumers; provision
   long/renewable TTLs or charm auto-renewal; add a proactive health probe that validates each
   consumer's Vault auth (the T4-style login check) BEFORE expiry, rather than discovering failure
   via a tenant's failed cluster-create.

**Related:** D-067 (surfaced these), D-052/D-053.

<!-- patchset-20260702-redeploy-readiness : DOCFIX-066..072 sweep -->

---

## D-002 -- AMENDMENT (2026-07-02): channel matrix reconciled to as-built

The D-002 table is historical (append-only); as-built corrections: (1) the `etcd, easyrsa`
row is DEAD -- etcd/easyrsa were dropped when vault moved to the mysql storage backend
(phase-02 R3: "HA Enabled false" is correct for vault-on-mysql); (2) `memcached` (absent
from the table) is pinned `latest/stable` -- the only track that charm publishes; the
project-wide "never latest/stable" guidance targets OpenStack-project charms, which
memcached is not (verify at deploy: `juju info memcached`); (3) the vault row (`1.8/stable`)
is superseded by the **1.16/stable** pin (D-068 / BUNDLEFIX-007). Channel-staleness review
2026-07-02: every other pin confirmed current for Caracal/jammy; rabbitmq-server `3.9` is
that charm's TERMINAL track (no newer exists), not a stale pin. The upstream charm-guide
delivery page is frozen (2023-12) -- `juju info <charm>` is the only live authority; the
pre-deploy channel assert belongs in `scripts/pre-flight-checks.sh`.

## D-009 -- AMENDMENT (2026-07-02): rabbitmq scale-up is NOT mechanical without min-cluster-size

D-009 promises Roosevelt scale-up is "change num_units 1 -> 3 and rerun". For
`rabbitmq-server` that is insufficient: without `min-cluster-size: 3` the charm accepts
client relations BEFORE the cluster forms (same failure class as the D-062 mysql formation
race; upstream charm docs call min-cluster-size best practice for multi-unit deploys).
Roosevelt bundle delta: set `min-cluster-size: 3` alongside `num_units: 3`. No testcloud
change (single unit is correct here).

## D-011 -- AMENDMENT (2026-07-02): acceptance item 6 is MANUAL unseal (matches phase-08)

D-011 item 6 read "Vault unseal + auto-unseal-after-reboot pattern verified". The v1
standard, as ruled in phase-08 D-011.6, is MANUAL unseal (3-of-5 key shares at a hidden
prompt) after any vault restart; auto-unseal is an available option adopted case-by-case
and NOT configured in v1. Item 6 is therefore read as: "Vault manual unseal after reboot
verified (sealed-on-restart is BY DESIGN, not a fault)". The acceptance bar and the
acceptance runbook now agree.

## D-043 -- RESOLVED (2026-07-02): ADOPTED, option (a) -- tenant-VM auto-resume ON

**Decision:** adopt option (a): `nova-compute resume-guests-state-on-host-boot=true`,
cloud-wide, as charm config (already present in bundle HEAD -- this resolution brings the
decision record level with the implementation, closing the bundle-ahead-of-record drift
flagged as DOCFIX-072). Rationale: tenant VMs are a different class from control-plane
services -- auto-resume is the industry norm and the Roosevelt customer expectation; the
restart procedure already builds on it (guests resume per pre-shutdown state, so a
deliberately stopped guest STAYS stopped). D-041's "down is a signal" discipline is
UNCHANGED for control-plane services, which this option does not touch; the D-035 mgmt VM
remains manual-start by policy (it is a control-plane component that happens to be a VM --
if auto-resume masks a mgmt-VM incident class in practice, revisit with an explicit
exclusion). Roosevelt item: VM-state monitoring/alerting so "down" stays visible.
**Resolves:** D-043 (was PROPOSED). **Related:** D-040, D-041, restart procedure.

## D-051 -- AMENDMENT (2026-07-02): policy delivery is bundle-native (DOCFIX-071)

The D-051 mechanism ("zip -> juju attach-resource") was never folded into the deploy
schedule: the bundle set `use-policyd-override=true` with no resource, phase-01 knowingly
deployed into `PO (broken)`, and no phase attached the zip -- the live policy existed only
as a session action (D-064). A clean redeploy would have shipped WITHOUT the tenant-
isolation RBAC. **Amendment:** the policy ships IN the bundle -- keystone
`resources: {policyd-override: ./policies/overrides.zip}`, with the zip committed beside
its source. `attach-resource` (appendix-C C.3, now $HOME-pathed -- the /tmp form was
snap-blocked) remains the LIVE-UPDATE path only. Drift between zip and yaml is a
FAIL-CLOSED deploy-gate check in `provider-bundle-check.py` (content compare). Acceptance
remains BEHAVIORAL (phase-03 Step 3.4: `PO:` is stage 1; the C.4 G3 probe is the gate) --
unchanged from D-051's own warning that the charm validates YAML only.

## D-061 -- AMENDMENT (2026-07-02): validation scope + the runbook fork (DOCFIX-066)

Two clarifications for an error-free redeploy: (1) the canary in
`phase-00-teardown-release.sh` validates machine SURVIVAL only; machine RE-ACQUISITION by
a next model (juju's MAAS provider allocates Ready machines; a kept host stays Deployed)
is the still-unvalidated half. Until BOTH halves pass, the release path is NOT a route
into phase-01 -- it serves model-only teardown/surgery. (2) The VALIDATED full-redeploy
spine is therefore the DESTROY path: `phase-00-teardown-destroy.sh` -> 8_lbaas/OSD ops ->
`reenroll-hosts.sh` -> carve -> standup -> phase-01. The phase-00 runbook (DOCFIX-066)
now encodes this fork explicitly; it previously drove the deprecated
`phase-00-teardown.sh`, whose "releases to Ready" premise D-061 disproved.


## D-069: Vault unseal-key custody -- split custody, rehearsed second-person unseal

**Status:** ADOPTED 2026-07-03 (policy; custodian ASSIGNMENT pending operator input --
tracked as SEC-003 in docs/security-ledger.md).

**Problem:** the 5 Shamir shares (threshold 3) are held by a single operator. The restart
procedure names missing keys a hard recovery BLOCKER, and vault re-init wipes the TLS
plane -- so single-person custody is a bus-factor risk no commercial cloud should carry.

**Decision (standard Vault operational guidance):**
1. Distribute shares so NO individual holds threshold (3+): e.g. 2/2/1 across three
   custodians, or 1 per person plus sealed escrow. Custodian names + storage media are
   recorded OFF-REPO (custody detail is itself sensitive); the ledger row tracks status.
2. A SECOND-PERSON unseal rehearsal is an acceptance item: someone other than the
   initializing operator must successfully unseal (restart-procedure Stage 3) before
   the deployment is declared operational. "The keys work" and "a second human can use
   them" are different facts; only the second one survives the bus.
3. Key custody is reviewed at every re-init (re-init re-cuts the shares).

**Roosevelt:** evaluate auto-unseal (transit/cloud KMS) as a separate decision; split
custody remains the floor either way (recovery keys still exist under auto-unseal).

**Related:** D-011 (amended: manual unseal is the v1 standard), DOCFIX-006 (the
lost-init cautionary tale), docs/security-ledger.md SEC-003.

## D-070: KVM baseline snapshots -- SUPERSEDES D-012 (no snapshot restore path)

**Status:** ADOPTED 2026-07-03. Supersedes D-012.

**Finding:** D-012 (two KVM baseline snapshots per cycle) was never exercised -- no
`virsh snapshot` step exists in any phase runbook or script; the only "snapshot"
mentions in the repo are dated as-built value captures. An adopted-but-unexercised
decision is documentation debt pretending to be a safety net.

**Decision:** retire the KVM-snapshot restore path. D-017/D-018 already made
rebuild-from-runbooks THE restore path, and snapshots quietly compete with that
discipline: a team that restores a snapshot skips exactly the rehearsal this testcloud
exists to force, and snapshot restore does not transfer to Roosevelt bare metal.
The pre/post-maintenance BASELINE CAPTURES (juju export + inventories) are retained and
strengthened -- that role moves to `bash scripts/cloud-assert.sh --capture`, whose
committed asbuilt/<date>/ BOM is also the Roosevelt multi-DC drift baseline.

**Recorded counter-argument:** cheap mid-rehearsal rollback (e.g. tight phase-08
iteration loops) is a real convenience. If that pain materializes, revisit as an
explicitly-labeled DEV CONVENIENCE with a rule that a snapshot restore invalidates the
run for acceptance purposes -- never as a restore path.

**Related:** D-012 (superseded), D-017/D-018 (the actual restore path), DOCFIX-075
(cloud-assert --capture).

## D-071: Routine update cadence and Juju controller patch policy

**Status:** ADOPTED -- all four policy points RULED 2026-07-21, each its own GA-R5 exchange
(utterances quoted below). Filed PROPOSED 2026-07-04 (by the jumphost controller-update
workstream; number relinquished by the main stream per the 2026-07-03 addendum-10 contention
note). Mechanism: `runbooks/ops-update-procedure.md` (DOCFIX-086), now the POLICY vehicle.
Point 4 RULED 2026-07-21 (GA-R5): question as presented = "adopt in-channel-only refreshes
for routine windows (channel moves always per-decision)?"; operator selection, exact
utterance: "Adopt as proposed (Recommended)". Routine windows refresh revisions only inside
D-002-pinned channels; every channel move is its own decision. Point 1 RULED 2026-07-21 (GA-R5): question as presented = "adopt the proposed
cadence trigger (monthly review window; security pulls forward)?"; operator selection, exact
utterance: "Adopt as proposed (Recommended)". Cadence = monthly review of `can-upgrade-to` +
controller patch delta; non-empty review opens a maintenance window run via
ops-update-procedure; security-driven charm revisions may pull a window forward; no
on-discovery updates. Point 2 RULED 2026-07-21 (GA-R5): question as presented = "adopt the
patch-only controller rule for routine windows?"; operator selection, exact utterance:
"Adopt as proposed (Recommended)". Routine windows move the controller by PATCH level only
within the pinned 3.6 track; minor/major jumps are each their own D-NNN with rehearsal,
never a routine-window action. Point 3 RULED 2026-07-21 (GA-R5): question as presented =
"adopt the standing update order (controller -> agents -> charms; keystone first,
nova-compute last; never interleave)?"; operator selection, exact utterance: "Adopt as
proposed (Recommended)". The ops-update-procedure grouping is now policy, not habit.

**Problem:** the cloud accumulates available updates on two layers with no standing
policy for either: (a) the Juju controller/agents (patch releases on the 3.6 track;
live measured 3.6.24 with client already 3.6.25), and (b) charm revisions floating
within their D-002-pinned channels (~20 apps showed `can-upgrade-to` on 2026-07-04).
Appendix-B says revision float is EXPECTED and re-baselines after validated state, but
nothing says WHEN updates are applied, in what order, or what controller jumps are
acceptable on a single non-HA controller.

**Proposed policy (rule each point):**
1. **Cadence trigger.** Updates are applied in deliberate maintenance windows (not on
   discovery). Proposed trigger: a monthly review of `can-upgrade-to` + controller
   patch delta, executed via ops-update-procedure when non-empty; security-driven
   charm revisions may pull a window forward.
2. **Patch-only controller rule.** A routine window may move the controller by PATCH
   level only within the pinned 3.6 track (e.g. 3.6.24 -> 3.6.25). Minor/major jumps
   (3.6 -> 3.7+) are each their own D-NNN with rehearsal, never a routine-window action.
3. **Standing order.** Controller -> model agents -> charm refreshes, keystone first,
   nova-compute last (the ops-update-procedure grouping). Never interleave layers.
4. **In-channel-only refreshes.** A routine window never changes a charm channel
   (D-002); channel moves are per-decision (see D-068 for the live example).

**Risk (explicit, for future reviewers): no-backup single-controller patch acceptance.**
The testcloud controller is SINGLE (non-HA) and Juju 3.6 ships no supported controller
backup (`create-backup` removed in 3.0). There is NO in-band downgrade of a controller.
Accepting routine controller patches therefore rests entirely on three compensating
controls: (a) patch-level-only jumps within 3.6/stable, (b) a proven-healthy pre-state
(`cloud-assert.sh --capture` PASS committed as the pre-change BOM), and (c) the D-070
restore posture -- a failed controller means re-bootstrap + rebuild-from-runbooks, i.e.
hours of rebuild, not minutes of restore. This risk is accepted for the virtual
rehearsal phase ONLY; Roosevelt must revisit it with HA controllers and a real
controller-state backup story before production operations.

**Roosevelt:** HA controllers change the upgrade blind-window and the risk calculus;
cadence and window sizing need an SLA-aware ruling; controller backup tooling is a
design item, not an ops improvisation.

**Related:** D-002 (channel pins), D-068 (Vault channel move, the non-routine
counter-example), D-070 (restore posture), DOCFIX-086 (ops-update-procedure runbook),
appendix-B (revision re-baseline policy).

## D-068 -- AMENDMENT (2026-07-05): vault 1.16 forward-pin is NOT viable; revert bundle to 1.8/stable

Item 1 ("Vault version") pinned the bundle to `1.16/stable` as a forward-pin off EOL Vault 1.8.8,
treating the move as "a MAJOR operation to rehearse." Deep-dive research (see
`docs/D-068-vault-1.8-vs-1.16-analysis.md`) shows it is not a hard *upgrade* but a largely-
**incompatible** different charm for this reactive Charmed OpenStack cloud:

- certificates interface changed **V0 -> V1** -- the 18 `vault:certificates` client charms are V0
  requirers and cannot bind a V1-only provider (the entire cloud TLS/CA layer fails to connect);
- storage went **Raft-only**, dropping the MySQL backend this deployment uses (`vault-mysql-router`);
- **no upgrade path** (backup-on-1.8 / restore-on-new-1.15, a CA-replacement migration);
- Canonical community notes report **new-vault + Charmed Ceph broken** (this cloud uses Ceph);
- Vault 1.15+ payload is **BUSL 1.1** (was MPL) -- commercial-licensing review item.

**Amendment:** (a) a deployable bundle MUST pin vault `1.8/stable` (the reactive, integration-
compatible charm) -- revert the `1.16/stable` bundle pin; (b) `1.16` is **ruled out** as the vault-
modernization path for this reactive cloud unless/until the OpenStack service charms gain
tls-certificates V1 support; (c) "get off EOL 1.8" stays **OPEN** as a Roosevelt-durability problem
(candidates: wait for OpenStack V1 support; OpenBao/MPL fork; or explicit EOL risk-acceptance for VR0
with a production remediation deadline -- operator/security call, not resolved here).

This also corrects the D-002 amendment note (item 3): the vault row reverts to `1.8/stable`; the
`1.16/stable` supersession is withdrawn. Bundle change coordinated with Code / applied at reconciliation.

**Related:** D-068 (this amends item 1), D-002 amendment (vault row), D-067/BUNDLEFIX-007 (vault-kv),
D-069 (unseal). **Evidence:** `docs/D-068-vault-1.8-vs-1.16-analysis.md`.

## D-071 -- AMENDMENT (2026-07-05): controller backup EXISTS on juju 3.6; risk section corrected

First execution of ops-update-procedure (ops-update-20260705) disproved the risk section's
premise "no supported backup in Juju 3.6": `juju create-backup` / `download-backup` exist on
the 3.6 client and were exercised live (902MB checksummed archive of the single controller,
~35s, downloaded to the jumphost pre-upgrade). The compensating controls for routine
controller patches are therefore FOUR, not three: patch-only jumps, proven-healthy pre-state
(committed pre-change BOM), a fresh `juju create-backup` archive (now the STANDARD 2.1 step,
DOCFIX-088), and the D-070 rebuild posture as last resort. Policy points 1-4 unchanged and
still PROPOSED for operator ruling. Restore-from-archive remains UNREHEARSED -- rehearsing a
controller restore is added to the Roosevelt open questions.

**Related:** D-071 (amends the Risk section), DOCFIX-088 (runbook 2.1 correction).

## D-072: Dashboard VIP TLS repair -- cluster binding moved to the default-binding space

**Status:** ADOPTED 2026-07-05 (operator-ruled: in-place fix on VR0); AS-EXECUTED same day.

**Problem (RCA in changelog addendum 15; symptom entry in appendix-A):** dashboard VIP
https was dead SINCE DEPLOY. The openstack-dashboard charm renders haproxy's 443 backend
on the CLUSTER-binding address but creates apache SSL vhosts only for the default-binding
and public-binding addresses. With `cluster` bound to metal-internal (a third space), the
TLS backend targeted a vhost-less socket that apache answers in PLAINTEXT, while haproxy's
Layer4-only health check kept the backend green. The fail-open phase-03 3.3 check (fixed
under DOCFIX-089) let it pass the deploy gate.

**Decision:** bind `cluster` to the SAME space as the application's default binding
(metal-admin), so the charm-rendered backend lands on an address apache serves.
Live: `juju bind openstack-dashboard cluster=metal-admin` (single unit; peer relation
carries no live traffic; hacluster rides the separate `ha` endpoint). Durable:
bundle.yaml bindings updated (BUNDLEFIX-011). NEVER provider-public (tenant plane).

**As-executed verification:** haproxy backends re-rendered to 10.12.8.182 (:70/:433);
https 200 on 10.12.4.58 and 10.12.8.58 (vault-CA-verified, csrftoken cookie present);
http leg unaffected; D-044 override intact; cloud-assert PASS.
Revert: `juju bind openstack-dashboard cluster=metal-internal` + git revert of the
bundle/docs commit.

**Upstream:** bug draft at docs/archive/upstream-bug-draft-dashboard-tls.md (charm renders an
L4-masked dead TLS backend by default in multi-space deployments) -- operator to file.
**Roosevelt:** edge TLS/DNS (D-044 trajectory) supersedes this plumbing; carry the
invariant "cluster binding space == default binding space for TLS-fronted charms" into
the Roosevelt bundle review regardless.

**Related:** D-044, D-072 mechanism in appendix-A, DOCFIX-089 (fail-closed 3.3 check),
BUNDLEFIX-011, changelog addenda 15 + 18.

## D-050 -- RESOLVED (2026-07-06): closed via D-051 (operator ruling)

Operator ruled 2026-07-06: D-050 is CLOSED as resolved by D-051's option (b) -- the
`use-policyd-override=true` flag is no longer a no-op footgun because the reviewed
domain-manager policy zip ships bundle-native (D-051 amendment / DOCFIX-071), with a
FAIL-CLOSED zip-vs-yaml drift check in the deploy gate and behavioral acceptance at
G3 (appendix-C C.4 probe). Closure condition attached by the operator: the exact
settings + policy set for error-free tenant standup must remain clearly defined --
that set is: bundle keystone `use-policyd-override=true` + `resources.policyd-override`
zip (committed beside source), provider-bundle-check content compare, and the C.4
behavioral probe. Any future policy delta rides the same mechanism (see D-073 for the
first such delta). **Resolves:** D-050 (was PROPOSED/OPEN since 2026-06-17).
**Related:** D-051 + amendment, D-064, D-073, FINDING-1.

## D-073: ADOPTED -- tighten identity:list_trusts to the modern keystone default

**Status:** ADOPTED 2026-07-06 (operator-ruled); implementation + testing REQUIRED
before live apply -- nothing applied yet.

**Problem (carried from the D-064 create_trust fix, "PROPOSED/OPEN separate
hardening"):** live `identity:list_trusts = ""` lets ANY authenticated user enumerate
ALL trusts cloud-wide (trustor/trustee/project relationships) -- cross-tenant info
disclosure on a commercial multi-tenant cloud. Newer keystone defaults tighten this to
`rule:admin_required or (role:reader and system_scope:all)`.

**Decision:** adopt the modern default for `identity:list_trusts` (and evaluate the
read-rule siblings `identity:get_trust` / `identity:list_roles_for_trust` at the same
shipped defaults while in there) via the SAME policyd-override zip mechanism as
D-051/D-064 -- policy source edited, zip rebuilt, bundle resource updated, deploy-gate
drift check green, live update via the gated attach-resource path (appendix-C C.3),
behavioral verify: tenant user gets 403 on `openstack trust list`, operator
admin still succeeds, magnum cluster create/delete (trust machinery) unaffected.

**Sequencing (proposed, operator to confirm):** land this BEFORE batch-3 d011-05 so
the P3 isolation acceptance validates the final shipping policy, not a superseded one.

**Related:** D-064 (origin), D-051 (mechanism), D-050 (closed), scs-0302 posture.

## D-073 -- AMENDMENT (2026-07-06): implementation STAGED (zip rebuilt); live apply PENDING

- policies/domain-manager-policy.yaml: `identity:list_trusts` added at the modern
  keystone default VERBATIM (39 rules now). One documented exception to the overlay's
  "no new-style rules" header invariant: the (role:reader and system_scope:all) branch
  is provably inert under this cloud's old-style enforcement -- narrows only, kept
  verbatim for 2024.2+ reconciliation provenance. Trust-rule siblings left UNCHANGED
  (evidence-gated per D-064: code-guarded, and no verified shipped-default strings in
  repo evidence to copy).
- policies/overrides.zip rebuilt via python3 zipfile (no zip(1) on the jumphost);
  provider-bundle-check item 7 content-compare PASS.
- Validated here: yaml.safe_load (39 string rules), ASCII+LF, tooling compatibility
  (tenant-offboard/tenant-acceptance call `trust list` in the admin context only).
  NOT validated here: oslo.policy parse (module absent on the jumphost; D-064's parse
  ran in the main-chat sandbox -- rerun there if desired) and BEHAVIORAL acceptance.
- LIVE APPLY (gated, operator-present; in the batch-3 window do-doc): stage the zip
  under $HOME (snap /tmp limit), `juju attach-resource keystone
  policyd-override=<staged>/overrides.zip`, wait for `PO:`, then behavioral verify:
  tenant-token `openstack trust list` -> 403; admin context -> succeeds; magnum
  cluster create/delete (trust machinery) unaffected. Rollback: re-attach the prior
  zip from git history (`git show HEAD~:policies/overrides.zip`).

## D-072 -- AMENDMENT (2026-07-06): upstream bug NOT filed (operator ruling)

Operator ruled 2026-07-06: the drafted upstream bug (docs/archive/upstream-bug-draft-dashboard-tls.md)
will NOT be submitted -- the root condition is judged a site misconfiguration (cluster
binding pointed at a third space) rather than a charm defect to report. The draft stays
in-repo for history; the addendum-18 OPERATOR ACTION item is withdrawn. The Roosevelt
invariant from D-072 ("cluster binding space == default binding space for TLS-fronted
charms") is UNCHANGED and remains the carry-forward.

## D-072 -- AMENDMENT (VR1, 2026-08-06): dashboard `cluster` RETURNS to metal-internal for the dual-metal topology

**Status:** RULED 2026-08-06 (operator, GA-R5). Operator ratification, exact utterance:
**"Ratified, land the config-of-record."** Proven LIVE before ratification per the operator's
own process directive (*prove the fix, then rule it*, to avoid ruling-churn on an unproven
change) -- which matches D-072's own live-first-then-durable pattern.

**Why the original value INVERTS in VR1.** D-072 set `openstack-dashboard cluster: metal-admin`
because VR0 had ONE metal plane: the charm's `ApacheSSLContext.get_network_addresses()` served
apache's SSL vhost on the metal-admin address, so `cluster: metal-admin` put haproxy's 443
backend on a served vhost. VR1 SPLIT metal into metal-admin + metal-internal (D-052/D-141
topology + the D-121 HA scale-up this deployment hardens). MEASURED 2026-08-06: the
openstack-dashboard charm declares NO `admin`/`internal` extra-binding (only `public`; deployed
metadata + charmhub docs) and exposes no `os-*-network` config, so `get_network_addresses()`
now serves apache TLS on **metal-internal** (`10.12.12.108`, the charm's own log +
`curl https://10.12.12.108:433` -> 200), NOT metal-admin. haproxy still dialed
`cluster`=metal-admin (`10.12.8.196:433`) -> no vhost -> PLAINTEXT (F-CV3, the exact D-072 trap,
inverted). Option A (steer apache back onto metal-admin) is UNAVAILABLE in-deployment -- the
charm gives no lever -- so the fix is on the cluster side.

**Amendment.** `openstack-dashboard cluster` -> **metal-internal**. This is simultaneously
(i) the generic HA/cluster-peer rule (14 apps already use metal-internal) and (ii) the plane
the dashboard's apache actually serves in VR1. VR0's `metal-admin` value stands as historical
record for single-metal topology (unchanged for VR0).

**Live proof (before ratification).** `juju bind openstack-dashboard cluster=metal-internal`
-> haproxy `:433` backends moved to metal-internal (`.12.108/.145/.146`); provider VIP
`10.12.4.58` AND operator metal-admin VIP `10.12.8.58` both serve TLS **200 CA-verified** (were
`000` plaintext); the reissued cert carries IP SANs for all three VIPs incl `10.12.8.58`
(resolves the `AH01909` name-mismatch); 3 units active/idle. Capture:
`docs/audit/stage5-dc0-phase03-coreverify-20260806.txt`. Config-of-record: **BUNDLEFIX-057**
(`bundle.yaml` `cluster: metal-internal`), binding-reference exception retired.

**REFINED INVARIANT (supersedes the 07-06 carry-forward for the split-metal case).** The
D-072 rule is more precisely: *a TLS-fronted charm's `cluster` binding must be the plane its
apache SSL vhost is served on* -- which equals the default binding ONLY when the charm serves
its internal vhost there. For a charm that declares no admin/internal binding under a
split-metal topology, that plane is metal-internal, NOT the (metal-admin) default. Roosevelt
delta: a Roosevelt-faithful fix is upstream -- have the dashboard charm declare `admin`/
`internal` extra-bindings (like the API charms) so apache can serve the operator (metal-admin)
plane directly; until then, `cluster: metal-internal` is correct for VR1+.

**Operator-access note (informs the tailnet model, gap-register item 21).** Operators reach
the dashboard via the metal-admin VIP over the tailnet; that VIP is unaffected by this change
(haproxy still fronts `10.12.8.58`; only the unit-local backend hop moved). The FQDN-SAN cert
(`os-public-hostname`, D-106/D-008) remains a Stage-7 nicety, not a correctness gap.

## D-073 -- AMENDMENT (2026-07-06): APPLIED live (d011-batch3 window)

The staged 39-rule overrides.zip was attached to keystone in the logged
d011-batch3 window (addendum 27). Settled "PO: Unit is ready". Behavioral
verify per the amendment's block: tenant app-cred context 403 on
identity:list_trusts (verbatim deny); admin trust list OK; magnum trust
machinery re-proven by d011-05 PASS. Independent proof: first live run of the
H4 drift detector (staged) reported live overlay == repo, 39 rules. Status:
ADOPTED -> APPLIED. Rollback stays as recorded (re-attach the HEAD~ zip).

## D-011 -- AMENDMENT (2026-07-06): batch-3 CLOSE -- all acceptance items stand

With the d011-batch3 window (addendum 27): item 4 octavia PASS live (RR +
member failover + amphora failover with N+1 headroom evidence, A2 conditional
pre-approval honored); item 5 magnum e2e PASS live (health + OCCM + P3
isolation via the foil1 tenant). Items 1-3 stood from batches 1-2; item 6
remains MANUAL by design (SEC-003 second-person unseal rehearsal pending --
gates D-011 FULL close); items 7-8 DROPPED per the amended bar. D-011 is
therefore CLOSED-except-item-6. The v1-close checklist (handoff section 6)
is now QUEUED per the do-doc's close-out clause. Checker corrections made
in-window: DOCFIX-095 (tenant-assert owner-context probe), DOCFIX-096
(amphora_headroom live CLI shapes).

## D-074: Tenant CIDRs are tenant-chosen and may overlap (amends D-016)

**Status: ADOPTED 2026-07-06** (operator rulings recorded same day; proposed by
the main-chat stream in docs/archive/tenant-cidr-overlap-correction-PLAN.md).

**Decision.** Tenant private (overlay) CIDRs are tenant-chosen and MAY overlap
across tenants. Non-collision is enforced ONLY against the reserved-ranges
registry, with real subnet-overlap math (containment in either direction),
plus RFC1918 membership (contract section 3 item 4's original intent).
Reserved ranges = the six PLANE_CIDRS /22s (scripts/lib-net.sh, keyed by
plane), the capi-mgmt tenant network (D-035; operator ruled: stays reserved;
resolved live by stable name capi-mgmt-subnet, never hardcoded), and the
metadata range. Default tenant CIDR policy (operator ruled): client-choice
with fallback default 10.100.0.0/24.

**Why.** The stage-4 guard over-enforced relative to the onboarding contract
(rejected tenant-vs-tenant overlap the contract never required) while
under-enforcing what matters (exact-string match missed partial overlaps
against reserved ranges). Overlap-allowed tenant CIDRs is the standard
isolated-multi-tenant model and matches the SCS Domain Manager hard-isolation
persona; per D-035 the management path never routes into tenant overlay space
(an earlier main-chat "CAPI/Magnum reachability" claim is retracted in the
PLAN doc). Gains: contract/implementation alignment, no find-a-free-/24 step,
clients can align to their own on-prem ranges, NetBox uniqueness duty shrinks
to the routed space, the one-/16 (256-tenant) ceiling goes away. Costs: an IP
no longer identifies a tenant (triage needs net+tenant context); no
tenant-private-IP routing between tenants (an anti-pattern here anyway).

**Phase-0 evidence (gates honored).** 0.1: allow_overlapping_ips=True read
live from the neutron-api leader's rendered neutron.conf (2026-07-06). 0.2:
exact-overlap proof executed inside foil1 (operator-ruled vehicle -- the
PLAN's "onboard an overlapping foil" was stale: foil1 already existed and the
old guard blocks exact-overlap onboarding): foil1-overlap-net/-subnet created
at 10.20.28.0/24 == beta-subnet EXACTLY, attached to foil1-router, and a
Magnum cluster (foil1-overlap, 1 master / 1 node, master-lb-enabled) built on
that overlapping network -- see the D-074 amendment recording the terminal
state and evidence path. Proof resources are torn down after evidence.

**Forward-only.** Existing tenants keep their CIDRs (beta 10.20.28.0/24,
foil1 10.20.29.0/24 -- measured 2026-07-06; the PLAN's "northwind
10.20.20.0/24" example was fictional). No re-CIDR; design-decisions forbids
transient-overlap re-CIDR.

**Implements:** guard rewrite in scripts/tenant-onboard.sh stage 4 +
scripts/net_overlap.py (DOCFIX-106); doc alignment (contract section 3,
onboarding runbook, clientdocs intake) rides the same delivery.
**Amends:** D-016 (its /16-pool uniqueness becomes vestigial, not reclaimed).
**Related:** D-035, D-051, D-066.

## D-016 -- AMENDMENT (2026-07-06): uniqueness carve-out superseded by D-074

D-016's single-pool model (carve non-overlapping /24s from 10.20.0.0/16)
stops being enforced forward: D-074 (ADOPTED) makes tenant CIDRs
tenant-chosen and overlap-allowed, guarded only against reserved ranges.
The /16 pool remains as history for existing allocations; nothing is
re-numbered. See D-074.

## D-074 -- AMENDMENT (2026-07-06): Phase 0.2 proof EXECUTED -- overlap+Magnum confirmed

Executed same-day in the logged d074-phase0 session (Claude Code lane).
foil1-overlap-net/-subnet created at 10.20.28.0/24 (EXACT overlap with
beta-subnet, different tenant); Magnum cluster foil1-overlap reached
CREATE_COMPLETE in ~14.5 min (matches beta's ~13.5 min profile); kubeapi
octavia LB ACTIVE/ONLINE; tenant-fetched kubeconfig returned 2 Ready nodes
THROUGH the API-LB with node addresses (10.20.28.229, .54) inside the
overlapping range -- while beta's cluster on the same CIDR served undisturbed.
Both tenants ran Kubernetes on the same /24 simultaneously: the strongest
refutation of the retracted reachability concern. Evidence:
~/openstack-baseline/d074-phase02-evidence-20260706-234352.txt (jumphost).
Proof resources torn down (cluster, template, router-if, subnet, net --
residual check clean); post-teardown cloud-assert PASS. Phase 0 gates are
fully discharged; the D-074 implementation (DOCFIX-106/107) ships.

## D-075: Dashboard VIP root -- redirect to /horizon + no autoindex (ADOPTED+EXECUTED 2026-07-07)

**Status: ADOPTED (operator ruling 2026-07-07, option c) + EXECUTED live same day.**

**Context.** During the devteam handover browser check the operator hit Apache
"Index of /" at https://10.12.4.58/ (provider-public dashboard VIP). Measured:
Horizon mounts at webroot /horizon (WSGIScriptAlias in conf-enabled/
openstack-dashboard.conf); the charm-rendered vhosts define no root handler;
Ubuntu stock apache2.conf grants `Options Indexes` on /var/www/. A directory
listing on a tenant-facing commercial surface is an information-disclosure /
posture defect, and a bare 200 at the root previously masqueraded as a healthy
dashboard (the D-072 as-executed "https 200" at the VIP root was THIS autoindex
page, not Horizon).

**Options presented:** (a) redirect / -> /horizon; (b) disable autoindex at the
root; (c) both. **Operator ruled (c).**

**Decision / mechanism.** Local override /etc/apache2/conf-available/
99-omega-root.conf on the dashboard unit (the charm exposes no config option
for either behavior; charm manages sites-enabled/ but never conf-enabled/, so
the snippet survives config-changed re-renders and in-channel charm refreshes):

    <Directory /var/www/>
        Options -Indexes
        RewriteEngine On
        RewriteRule ^$ %{REQUEST_SCHEME}://%{HTTP_HOST}/horizon [R=302,L]
    </Directory>

**Why per-directory mod_rewrite and NOT RedirectMatch:** first attempt used
`RedirectMatch 302 ^/$ /horizon`; mod_alias absolutizes the Location from the
BACKEND connection port when the client Host header carries none, emitting
http://<VIP>:70/horizon (and :433 on the TLS leg) -- ports haproxy does not
expose. Caught in this window's verify step. The rewrite echoes the client's
own Host header and request scheme, so both haproxy legs redirect correctly.
302 (not 301) so browsers do not cache permanently against a future webroot
change. `Options -Indexes` subtracts from the inherited option list and covers
/var/www/html by inheritance.

**Verification (as-executed):** on-unit root -> 302 + zero "Index of" match;
via VIP http leg: / -> 302 http://10.12.4.58/horizon -> follows to 200 at
/horizon/auth/login/. TLS-leg browser confirmation by operator.

**Durability:** PER-REBUILD -- teardown wipes the unit; reapply each rebuild
via the phase-03 step 3.3 block (DOCFIX-118), same lifecycle as the D-044
cookie override. **Roosevelt:** edge TLS/DNS (D-044 trajectory) supersedes
this plumbing; carry the invariant "tenant-facing roots never serve directory
listings" into the Roosevelt bundle review.

**Revert:** `sudo a2disconf 99-omega-root && sudo systemctl reload apache2`
on the dashboard unit; git-revert the DOCFIX-118 runbook/appendix additions.

**Related:** D-072 (VIP TLS repair; its root-200 evidence reinterpreted),
D-044 (cookie override precedent + Roosevelt edge trajectory), DOCFIX-089
(fail-closed dashboard checks), DOCFIX-118 (runbook + appendix-A delivery).

---

## D-100: Inter-DC network topology and fabric (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling). Instantiates the D-052 per-DC template across two DCs.

**Context:** VR1 is a virtual regional two-DC test environment (DC1, DC2, plus an Office1 headend), all nested libvirt VMs on one vcloud host, rehearsing the future bare-metal multi-DC site (Roosevelt). D-052 already defines the six-plane per-DC template and tags metal-admin DC-local / metal-internal "may-cross-fiber" -- but that label assumed a single stretched cloud.

**Decision:**
- Fabric: a full dark-fiber triangle -- DC1<->DC2, DC1<->Office1, DC2<->Office1 (mesh, not star) -- each modeled as a dedicated virtual L2 segment on vcloud. Each site additionally has its own OPNsense edge with a simulated ISP uplink ("external"), so every site has independent edge access.
- WAN simulation: apply `tc netem` (latency / jitter / loss / rate) on the inter-DC and site-uplink segments, tuned to same-metro dark fiber (low single-digit ms) unless the Roosevelt target says otherwise. The exact parameters are a Phase-0 knob recorded in the buildout doc.
- Plane span (independent-cloud refinement of D-052): because DC1 and DC2 are INDEPENDENT clouds (separate Keystones, no control-plane federation), no control-plane plane crosses the fiber. provider-public, metal-admin, metal-internal, data-tenant, and storage instantiate DC-LOCAL per DC. ONLY the replication plane crosses DC1<->DC2. This refines D-052's "metal-internal may-cross-fiber" (a stretched-cloud notion) to DC-local for the independent-cloud topology.
- Provider-NIC discipline: each DC's provider NIC stays RAW so ovn-chassis builds br-ex and enslaves it by MAC; MAAS must not bridge it. Public API VIPs live on provider-public (D-060 Pattern A). The D-057 provider-vip plane is NOT used (reverted by D-060).

**Rationale / tradeoffs:** mesh (not star) is required because replication is a direct DC1<->DC2 data path -- routing it through Office1 would be a bandwidth SPOF -- while management is Office1<->DC. Per-site ISP edges make each DC independently reachable and feed the per-DC airgap mirror upstream (D-107). Delta to Roosevelt: the dark-fiber-triangle-plus-per-site-edge shape IS the intended Roosevelt shape; the only virtual-only element is that the "fiber" is a vcloud virtual segment with netem standing in for real fiber and optics.

**References:** instantiates D-052 (six-plane per-DC template; independent-cloud cross-fiber refinement); carries the D-057 root-cause br-ex discipline as revised by D-060 (Pattern A); pairs with D-101 (families), D-107 (edge egress), D-108 (replication).

**Sub-items RULED 2026-07-09 (operator):** netem link profile -- same-metro dark-fiber lean CONFIRMED as the default (low single-digit ms, jumbo-capable); re-tune when a specific Roosevelt inter-DC target is known, not before. RE-CONFIRMED 2026-07-21 (GA-R5): question as presented = "does the placeholder profile stand until a Roosevelt link spec exists, or can you supply target numbers now?"; operator selection, exact utterance: "Placeholder stands (Recommended)". The live virbr5 profile (delay 3ms jitter 1ms loss 0.01%, stepE capture) IS the rehearsal profile; gap #11 is now AWAITING EXTERNAL INPUT (the Roosevelt link spec) with its trigger defined -- nothing to ask until the spec exists. Office1<->DC fiber carries management traffic only (MAAS/Juju/operator) -- CONFIRMED per the stated lean (not one of the six items the buildout doc listed for explicit redline; adopted with the rest of D-100, flag if this needs separate reconsideration).

---

## D-101: DC-DC address plan and IPv6 family matrix (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling), AMENDED same day: (a) metal-admin gains a ULA leg (reverses this decision's own stated lean, see the family matrix below); (b) D-102 (IPv6 tenant addressing and MTU sub-policy) is MERGED into this decision per operator ruling -- D-102's entry is retained for numbering/history only, see its own Status line. Activates the developed family-follows-reachability IPv6 plan for VR1. NetBox (vcloud-local) is the IPAM authority for all literals.

**ANNOTATION 2026-07-30 (DOCFIX-205) -- DC LABELS IN THE ORIGINAL BLOCK ARE PRE-D-117; THIS
ENTRY CARRIES BOTH NAMESPACES AND THEY ARE NOT INTERCHANGEABLE.** Recorded per D-117's ruled
treatment ("ADOPTED decision text: ANNOTATE in place, do NOT rewrite"). D-117 (2026-07-13)
retired the repo's `dc1`/`dc2` labels in favour of `dc0`/`dc1`; D-119 (2026-07-14) then
region-qualified them to `vr1-dc0`/`vr1-dc1`. Read this entry by DATE:

    Original decision + same-day amendments (2026-07-09), PRE-D-117:
      "DC1" = VR1's FIRST  DC = `vr1-dc0`      "DC2" = VR1's SECOND DC = `vr1-dc1`
      "DC0" in "DC1 INHERITS the DC0 six-plane v4 layout" = VR0's DC0 = `vr0-dc0`
    Amendments dated 2026-07-14 or later (e.g. the 2026-07-27 dual-stack and R9 blocks):
      `vr1-dc0` / `vr1-dc1` used with their D-119 meanings, already correct.

**This is D-117 TRAP 2 in the one entry where it bites hardest:** the token `dc1` appears in
BOTH halves of this entry meaning DIFFERENT datacenters -- VR1's first DC in the 2026-07-09
text, VR1's SECOND DC in the 2026-07-27 amendments. Never carry a bare `dcN` out of this entry
without checking the date of the block it came from. The v6 literals are unaffected either way:
per D-119 the ULA hextet tracks the GUA site nibble (`f02` -> `:2NN`, `f03` -> `:3NN`), NOT the
DC index, and must not be "fixed" to match it.

**Context:** v1 / DC0 is IPv4-only on 10.12.0.0/16 (six planes in `scripts/lib-net.sh`). The developed IPv6 plan (in-project, not yet committed) classifies planes by reachability: GUA for external clients, ULA for internal-only, IPv4 where forced. DC-DC is the phase that activates it.

**GOVERNING RATIONALE -- recorded 2026-07-27 from the operator, verbatim, because it
existed in NO repo surface and is decision-shaping beyond this entry.** D-101 rules WHICH
family each plane carries; it recorded nothing about WHY, so a reader could not tell a
deliberate posture from an accident, and a v4-heavy deployment reads as IPv6 neglect.
Operator, exact words:

> "The ULA only networks are fully internal, yes I do understand that. Roosevelt will have
> full ipv4 and ipv6 edge transport. There are real IPv4 sizing constraints that we are
> running up against in future expansion plans. We are building as close to full IPv6 as
> possible on every layer possible. We are using IPv4 where necessary only because it is
> necessary on same layers at this point in time. We are also using IPv4 because bringing
> the full stack live on IPv4 was easier in earlier stages then also troubleshooting IPv6
> configurations around modules that may not have supported it as default."

**THE STANDING PRINCIPLE this establishes: IPv6 unless IPv4 is NECESSARY.** It answers
future per-plane and per-service questions without a fresh ruling each time -- the default
is v6, and every v4 leg is a justified exception, not a neutral choice.

**THREE CONSEQUENCES a future session must not misread.**
1. **The v4-first sequencing was DELIBERATE RISK REDUCTION, not neglect.** Bringing the
   stack up on v4 first avoided debugging the cloud and IPv6-in-charms simultaneously.
   Do NOT "correct" the v4 surfaces as though they were oversights, and do NOT read the
   absence of v6 somewhere as evidence it was not wanted.
2. **The driver is IPv4 SIZING in future expansion**, i.e. commercial capacity, not
   protocol preference. That makes v6 coverage a business requirement, which is why it is
   worth cost here rather than deferring wholesale to Roosevelt.
3. **Roosevelt has FULL v4 AND v6 EDGE TRANSPORT.** Consequence measured 2026-07-27:
   **NAT64/DNS64 was considered as a way to simulate v6 egress in VR1 and REJECTED** --
   with native v6 transport at Roosevelt it is a shim with NO Roosevelt analog, testing a
   translation path that will never run in production, while adding a component to the
   OPNsense edges (the most fragile surface here) and re-coupling the rack v6 leg through
   DNS64's dependence on the D-131 forwarder. The ULA planes are internal BY DESIGN, so
   nothing in VR1 needs v6 egress. **External IPv6 routing is deliberately NOT added this
   deployment; it arrives with the next full deployment.**

**Decision:**
- Per-DC distinct v4: DC1 INHERITS the DC0 six-plane v4 layout (provider-public 10.12.4.0/22 ... replication 10.12.36.0/22) unchanged, so DC1 equals the validated template. DC2 gets a DISTINCT, non-overlapping parallel supernet (same six-plane structure; exact prefixes assigned in NetBox, non-colliding with DC1, operator infrastructure, and client VPN ranges per D-074). Distinct ranges keep DR / backup endpoints unambiguous (no NAT hairpin).
- Family matrix (per DC), family follows reachability:
  - IPv6-only ULA: data-tenant (geneve underlay), storage (Ceph public), replication (Ceph cluster, including the cross-DC leg), Octavia lb-mgmt, and P2P links. Internal, no external clients.
  - Dual-stack (IPv4 + ULA v6): the metal-internal API-endpoint VIPs gain a v6 (ULA) address; each DC's Vault issues IPv6-SAN certs (D-109). **AMENDED 2026-07-09 (operator ruling):** metal-admin ALSO joins this dual-stack group (a ULA leg alongside its existing v4) -- reverses this decision's original lean of keeping metal-admin v4-only; PXE itself stays v4-first regardless (see below), the ULA leg is for the admin-API traffic on that plane, not PXE.
  - IPv4 retained: MAAS PXE / provisioning (PXE is v4-first, unaffected by the metal-admin amendment above); and datastore east-west (MySQL / RabbitMQ stay v4-bound -- the plane is dual-stack-capable but v6 there is not chased: marginal benefit, real cert / complexity cost).
  - GUA (ARIN 2602:f3e2::/32, region-0 /36): provider public API VIPs, native v6 ext_net, tenant pool delegation. Enabled here because the simulated OPNsense edge is ours -- the D-015 "upstream-router IPv6 readiness" gate on external GUA does NOT apply in VR1, so VR1 can rehearse the full internal-ULA plus external-GUA end state.
- Replication carrier correction: the DR seed names the cross-DC carrier as v4 10.12.36.0/22. Per this matrix and the operator ruling that replication is not IPv4-bound, the cross-DC replication carrier is IPv6-only ULA (a /64 from the org ULA /48). This SUPERSEDES the DR-seed carrier line. v1 / DC0 `lib-net.sh` v4 values are unchanged (DC-DC assigns families on its own layer).
- Literals: the org ULA /48, the per-DC GUA carve out of the /36, and DC2's v4 supernet are NetBox-authoritative, populated via the extended import pipeline (`netbox/*-import.py` extended for multi-DC and dual-stack). They are NOT hardcoded in this decision.
- **Tenant/MTU sub-policy (folded in from D-102, 2026-07-09 operator ruling):**
  - Tenant v6 routing: native GUA routing, no NAT66 -- the simulated OPNsense edge routes tenant GUA prefixes (prefix delegation or static routes). NAT66 is discouraged and we own the edge, so native is both correct and feasible.
  - Tenant v6 block size: /60 per tenant default (~4080 per cloud); /56 available on request.
  - Tenant address mode: SLAAC default (RA); DHCPv6-stateful available for tenants needing deterministic addressing.
  - MTU / geneve-over-v6: compute the exact byte budget in Phase 0 against the MEASURED host L2 MTU. An IPv6 outer header adds 20 bytes over IPv4 (40 vs 20); geneve-over-v6 overhead is about 40 (IPv6) + 8 (UDP) + 8 (Geneve base) = 56 bytes, before nested-virt and options. Recommendation: raise the vcloud underlay to jumbo (9000) end-to-end so tenant MTU stays 1500 and amphora/geneve fit; if the underlay is pinned at 1500, tenant MTU drops to about 1444 (v6-geneve) and MUST be set consistently across ovn geneve, tenant-network MTU, and amphora. The measured underlay MTU is a Phase-0 gate -- do not assume jumbo.
  - Phase-A rollback posture: go straight to IPv6-only on the internal three (data / storage / replication) rather than dual-stack-then-flip, given plane isolation and the Octavia ULA proof. Dual-then-flip is the conservative alternative if Phase-0 verification of Ceph-over-v6 / geneve-over-v6 is not clean.

**Rationale / tradeoffs:** matches the operator directive (v6 wherever possible, v4 only where forced) and maximizes Roosevelt rehearsal value (proves Ceph-over-v6, geneve-over-v6, native v6 tenant routing). The two forced-v4 points (PXE, datastore east-west) are explicitly justified rather than incidental. Native routing plus SLAAC for tenants is the IPv6-correct default; the MTU budget is the classic nested-OpenStack failure mode, handled by measurement plus jumbo. Ceph-over-v6 (`ms_bind_ipv6`) and OVN geneve-over-v6 remain Phase-0 verification gates in the buildout doc.

**References:** activates the developed IPv6 plan (its Phase A/B/C); carries D-016 / D-074 (tenant CIDR policy); supersedes the DR-seed v4 replication carrier; feeds D-106 (AAAA), D-108 (replication), D-109 (Vault v6-SAN, kept standalone per 2026-07-09 operator ruling -- not merged here). NetBox is IPAM apex.

**Remaining open item:** the exact NetBox literals (org ULA /48, per-DC GUA carve, DC2's v4 supernet) -- pending NetBox assignment (gap #3 in `docs/dc-dc-deployment-workflow.md`'s tooling gap register), not a ratification question.

### RULING NOTE 2026-07-25 -- D-101 address-family policy CONFIRMED (no amendment)

**Status:** D-101's family matrix CONFIRMED by operator ruling 2026-07-25 (GA-R5). This
NOTE records the ruling; it changes NOTHING in D-101 (the matrix rationale already reads
"v6 wherever possible, v4 only where forced").

Question as presented: whether dc1 phase-4 deploys v4-only (deferring IPv6, per the
expansion review's option (b)) or dual-stack.

Operator utterance, verbatim (2026-07-25): "Dual stack to be used where IPv4 is required,
IPv6 where IPv6 only makes sense."

Clarifying exchange, verbatim: "Dual stack deployment for DC0 and DC1. This is the
deployment when the dual-stack is added"

**Effect:** the v4-only phasing option (expansion review option (b)) is CLOSED, for BOTH
DC0 and DC1 in this deployment. `overlays/vr1-dc1-vips.yaml` (v4-only) is not a
D-101-conformant deploy input on its own; the `vr1-dc1-vips` / `dc-dc-ipv6-family-matrix`
per-key REPLACE collision must be reconciled to ONE dual-family `vip` per app per DC
before deploy. STILL OPEN under this ruling: octavia's family -- D-101 places `lb-mgmt`
in the IPv6-only ULA group, but `overlays/dc-dc-ipv6-family-matrix.yaml` documents live
upstream IPv6 breakage there (LP #1911788, #1913409); operator sub-ruling pending (see the
D-136 open questions).

### RULING NOTE 2026-07-27 -- D-101 RE-CONFIRMED against measured substrate; the literals move ONTO the Stage-5 critical path

**Status:** RULED 2026-07-27 (operator, GA-R5). This NOTE, like its 2026-07-25 predecessor,
records a ruling and AMENDS NOTHING in D-101's matrix.

Question as presented, verbatim: "R2 -- Address family. D-101 RULED dual-stack for both DCs
*this deployment* and closed the v4-only option, but there is zero IPv6 anywhere in the DC
substrate: no v6 subnet on any of the 12 plane fabrics, no v6 link on any of the 18 nodes.
How should Stage 5 proceed?" Options presented: (a) carve v6 and deploy dual-stack as ruled;
(b) deploy v4-only now and add v6 after, via an explicit D-101 amendment; (c) dc0 dual-stack
/ dc1 v4-only as a per-DC experiment. Operator selection, exact utterance:
**"Carve v6 and deploy dual-stack as ruled (Recommended)"**.

**What made this a live question rather than a settled one.** The 2026-07-27 Stage-5
grounding audit MEASURED the substrate against the 07-25 ruling and found the ruling
unimplemented: `maas admin subnets read` returns exactly ONE IPv6 subnet cloud-wide
(`2602:f3e2:f01:100::/64`, on `fabric-0`, the Office1 base net, space `undefined`); none of
the ten named plane fabrics nor either metal-admin fabric carries an IPv6 subnet; zero IPv6
links exist across all 18 nodes; and every node reports `default_gateways.ipv6 = NONE`.
D-101's family matrix requires ULA on data-tenant, storage and replication plus a ULA leg on
metal-admin and metal-internal -- none of which has any MAAS v6 presence to bind against.

**Effect -- CORRECTED 2026-07-27, same session, before any dependent work.** This note
first stated that D-101's "Remaining open item" (the org ULA /48 and per-DC GUA carve)
moved onto the critical path because dual-stack cannot deploy against literals that do not
exist. **That was WRONG, and the error was mine.** The operator asked whether the NetBox
apex had actually been polled; it had not. It has now been read, and the literals are
**ASSIGNED, RATIFIED AND RECORDED** -- they have been since 2026-07-11.

MEASURED from the apex record `netbox/draft/vr1-office1-current-20260725.json` (139
prefixes, 103 IPv6), every entry tagged `D-101/D-111`:
- **ULA `fd50:840e:74e2::/48`.** VR1 DC0: metal-admin `:220::/64`, metal-internal
  `:221::/64` (inside the metal `/60`), data-tenant `:230::/64`, storage `:240::/64`,
  replication `:250::/64`. VR1 DC1: `:320::/64`, `:321::/64`, `:330::/64`, `:340::/64`,
  `:350::/64`. Each with its parent `/60`.
- **GUA provider-public.** DC0 `2602:f3e2:f02:10::/64` plus VIP `/64` at `f02:11::/64`;
  DC1 `2602:f3e2:f03:10::/64` plus VIP `/64` at `f03:11::/64`.

**D-111 is ADOPTED 2026-07-11** (`docs/design-decisions.md:2351`) and is the ratifying
decision. `docs/CURRENT-STATE.md:264` already recorded that the apex "holds the entire IP
PREFIX layer (all 12 planes + transit + uplink + v6 GUA/ULA), DRIFT-FREE".

**So the real effect is narrower and better: the Stage-5 precondition is PROPAGATION, not
ASSIGNMENT, and it requires no further ruling.** The ratified values exist in the apex and
in `netbox/`; they are absent from the two places Stage 5 actually reads:
1. **`scripts/lib-net.sh` carries NO v6 at all** (measured: essentially zero v6 tokens in
   the file). Its `PLANE_CIDRS` arms are v4-only for both DCs.
2. **MAAS carries no v6 on any of the 12 DC plane fabrics** (measured: one v6 subnet
   cloud-wide, on the Office1 base fabric).
Note that CURRENT-STATE's "DRIFT-FREE across netbox/lib-net/artifacts" is not contradicted
by this: drift-free means the values that appear in more than one place AGREE, not that
coverage is complete. `lib-net.sh` simply never gained a v6 arm.

**D-101's own "Remaining open item" paragraph above is therefore STALE** -- it still reads
"pending NetBox assignment (gap #3 ...)" for literals that D-111 adopted on 2026-07-11.
That is a DOCFIX-class contradiction of a later ruling, the same class as DOCFIX-200 and
DOCFIX-204, and it is what misled this session. Logged, not fixed here (hard rule 1);
queued in the Stage-5 audit's Phase-3 batch.

**PROCESS FAILURE, OWNED.** The audit's own lens 2 recorded the apex as UNMEASURED and
warned in terms: "D-101's literals may exist in NetBox and simply not be carved into MAAS
-- L2-3's claim is scoped to MAAS + node reality and does not assert the apex is empty."
That warning was correct, was available, and was not acted on before a question was put to
the operator. This is the exact failure the audit exists to catch -- trusting stale
decision prose over an available measurement -- committed by the audit itself.

**Sequenced consequences, all of which now inherit "dual-family" from this ruling:**
1. The `vr1-dc1-vips` / `dc-dc-ipv6-family-matrix` per-key REPLACE collision must be
   reconciled BEFORE either authority location is populated. The audit MEASURED it in both
   directions (finding L3-9): with the ipv6 overlay applied last the merged input FAILS with
   ten `vip not a triple` errors; with the vips overlay last the v6 legs are REPLACED
   wholesale and the result **silently PASSES as pure-v4**. The second is the dangerous one --
   it is a green gate over a lost ruling.
2. Any VIP added under R11 (vault, designate) is dual-family from the outset. Adding v4-only
   VIPs and re-doing them later means re-issuing certificate SANs on a live cloud.
3. R9 (where dc1's OpenStack-layer literals live) must be answered knowing both families
   are in scope.
4. Octavia's family remains the standing sub-ruling recorded in the 07-25 note and is
   presented separately as R8; this ruling does NOT resolve it.

**WITHDRAWN:** this note first queued a sub-question "R2a -- WHICH literals". It is
withdrawn as never having been open. D-111 assigned them on 2026-07-11 and the apex carries
them; see the corrected Effect above. Nothing further is owed from the operator on the
address values.

### RULING NOTE 2026-07-27 -- D-101: Octavia lb-mgmt CONFIRMED IPv6-ULA, charm-owned; the open sub-ruling is CLOSED

**Status:** RULED 2026-07-27 (operator, GA-R5). CLOSES the sub-ruling the 2026-07-25 note
left open ("STILL OPEN under this ruling: octavia's family"). Amends nothing in D-101's
matrix -- it CONFIRMS it.

Question as presented: whether Octavia's `lb-mgmt-net` should be IPv6 (per D-101's
IPv6-only-ULA classification and the charm's own default), v4-only as a recorded D-101
exception (the shape `overlays/dc-dc-ipv6-family-matrix.yaml` drafted), or dual-stack; and
separately whether the prefix should come from the D-111 carve via
`create-mgmt-network: false` or be left to the charm. Operator selection, exact utterance:
**"I want to take a Octavia creates and owns its own IPv6 network."** Supporting reasoning
given by the operator, recorded because it is load-bearing: "We have to research to
troubleshoot if we run into MTU bug issues down the road. We have already deployed using
this topology in the v1 DC test deployments that got us to this point."

**THIS RULING REQUIRES NO ARTIFACT CHANGE.** Measured: `create-mgmt-network` is set NOWHERE
in `bundle.yaml` or any overlay, so the charm default (`True`) has always applied, and
Octavia's whole options block is `debug`, `openstack-origin`, `amp-image-tag`, `vip`. VR0
deployed Octavia on precisely this shape. The ruling ratifies the existing topology rather
than changing it -- the operator's point that it is already proven in the v1 DC test
deployments is correct on the artifacts.

**THE EVIDENCE THAT SUPPORTED THE v4-ONLY DRAFT DOES NOT HOLD.** Full research capture:
`docs/audit/octavia-ipv6-research-20260727.md`. Both citations in
`overlays/dc-dc-ipv6-family-matrix.yaml` fail on inspection: **LP #1913409** is Fix Released
(2021) against **kolla-ansible**, a different installer with no bearing on a charm deploy;
**LP #1911788** is Incomplete and a DUPLICATE of **LP #1896630**, and is NOT an IPv6 defect
-- its diagnosed cause is an OVN port-binding shortname-vs-FQDN mismatch for LXD containers
on MAAS, fixed by `ovs-record-hostname.service` **in OVS 2.15**. This deployment's own dc0
mirror was queried: our jammy nodes install **openvswitch-switch 2.17.0 / 2.17.9**, so that
mechanism is fixed here. Against them, upstream Octavia documents that "IPv6 subnets can be
used for the LB Network", and **LP #1897418 records the charm's default verbatim: "By
default, Octavia charm uses ipv6 for its lb-mgmt-subnet."**

**FAMILY CONFIRMATION -- the charm's default IS a ULA.** The amphora address quoted in LP
#1911788 is `fc00:fa21:3d5c:9cfd:...`, i.e. `fc00::/7`. So charm default, D-101's
"IPv6-only ULA ... Octavia lb-mgmt ... Internal, no external clients" classification, and
the existing as-built all agree.

**AND THE VR0 GUA CONTRADICTION WAS A PAPER ALLOCATION.** The apex carries
`2602:f3e2:e02:80::/60,/64` "VR0 DC0 Omega Cloud LBaaS Management Prefix/Subnet" in GUA
space, which appeared to contradict the ULA classification. It was never deployed:
`lib-net.sh:22-29` gives VR0 six IPv4 planes and no lbaas plane; `lib-net.sh:34` lists
`lbaas` in `STALE_SPACES`; `maas-as-built-reference.md:75` records the lbaas NIC as "idle
(undefined; ex-lbaas), raw NIC, no link". D-101's own context line says VR0 is IPv4-only.

**CONSEQUENCE FOR THE D-111 CARVE:** because the charm owns the network and exposes NO CIDR
option, the absence of an lb-mgmt `:x80` prefix in the VR1 ULA carve is now CORRECT rather
than a gap -- the prefix is charm-generated and cannot come from the apex. That absence
should be recorded as DELIBERATE so a later reader does not "fix" it.

**APEX COMPLETENESS IS GATED, NOT SETTLED -- see gate G18.** Because the prefix is
charm-generated, NetBox is knowingly INCOMPLETE for this one plane. Whether it gets
back-filled post-deploy or is recorded as deliberately out of apex scope was DEFERRED BY
OPERATOR DIRECTION 2026-07-27 until the cloud is live and IPv6 behaviour has been observed,
and made a BLOCKING ruling-type gate: **the deployment may not be declared complete while
G18 is open.** Ruling it early would also pre-empt the unruled D-136 render-pipeline
decision, which covers the same apex-authority ground.

**STANDING OBLIGATION, carried by the operator's own reasoning:** LP #2018998 ("MTU mismatch
between o-hm0 and lb-mgmt-net", charm-octavia, High) is Fix Released across our lineage but
**recurred 2025-12-31 against octavia 14.0.0 / 2024.1 stable -- our exact pin**. A jumbo
`lb-mgmt-net` beside a 1500 `o-hm0` silently drops health messages >1500B and causes
SPURIOUS failovers. This is the direct interaction with the R3 jumbo-underlay ruling. Owed
at the Octavia step of Stage 5: **verify `o-hm0`'s MTU MATCHES `lb-mgmt-net`'s, measured,
not merely that the charm claims to set it.** Workaround if needed:
`ovs-vsctl set interface o-hm0 mtu_request=<match>`.

**ONE THING NOT ASSERTED:** whether the charm attaches an EXTERNAL GATEWAY to the Neutron
router it creates (tagged "Router for IPv6 RA or north/south mgmt traffic"). There is no
config surface telling it to, and ULA is not globally routable, but isolation was not
proven from the docs. Cheaply settled at deploy time by inspecting the router the charm
creates -- queued as a Stage-5 observation, not a blocker.

### RULING NOTE 2026-07-27 -- D-101 MTU sub-policy RESOLVED to the jumbo branch

**Status:** RULED 2026-07-27 (operator, GA-R5). D-102 (the original MTU sub-policy) is
MERGED INTO D-101 and its body says "amend D-101 instead", so this note lands here.

Question as presented, verbatim: "R3 -- MTU budget. The underlay is already jumbo on every
mesh and plane bridge; only two segments lag (the rack transit NIC inside both containment
VMs at 1500, and all 17 MAAS VLAN records at 1500). The budget script verdicts: underlay
9000 -> tenant MTU stays 1500; underlay 1500 -> tenant MTU 1444, which must then be set
consistently across ovn geneve, tenant-network and amphora. Which branch?" Options
presented: (a) raise the two lagging segments to 9000; (b) accept 1500 and pin tenant MTU
to 1444; (c) raise the MAAS VLANs only, leaving the transit at 1500. Operator selection,
exact utterance: **"Raise the two lagging segments to 9000 (Recommended)"**.

**MEASURED FIRST -- capture `docs/audit/mtu-budget-20260727.txt`.** D-101 requires that
"the measured underlay MTU is a Phase-0 gate -- do not assume jumbo", and
`scripts/dc-dc-mtu-geneve-budget.sh` had existed without ever being run to a recorded
verdict (audit finding L2-5). It has now been run both ways. The audit's original framing
("jumbo bridges, 1500 VLANs, neither branch") was correct but understated how nearly done
the jumbo branch already is:
- Every vcloud MESH leg is already 9000, including `mesh-vr1-dc0-vr1-dc1` (virbr5), the
  inter-DC path itself. `office1-local` is 9000. All six plane bridges on both racks are 9000.
- The four 1500 legs are the D-125 SIMULATED-ISP uplinks (`vr1-dc0-uplink`,
  `vr1-dc1-uplink`, `office1-wan`, `wan`). They MODEL THE INTERNET, D-125's egress
  isolation gate is defined against them, and **they must STAY 1500**. They are not a defect.
- Exactly TWO segments lag: the rack transit NIC `enp1s0` inside both containment VMs, and
  the MAAS VLAN MTU records (all 17 read 1500).

**Why the MAAS record is the one that bites silently:** MAAS renders the VLAN MTU into node
netplan, so a jumbo BRIDGE beneath a 1500 MAAS record still yields 1500 node interfaces.
Jumbo bridges alone do not deliver a jumbo underlay.

**Rationale for (a).** Tenant MTU stays 1500 -- the ordinary case -- so no per-charm MTU
coordination is needed at all. Option (b) required ovn geneve, tenant-network MTU and
amphora to agree exactly and permanently across both DCs, and NOTHING in this repo checks
MTU (neither `cloud-assert.sh` nor `provider-bundle-check.py` has any MTU assertion), so a
drift between those three would be silent -- D-101 itself names this "the classic
nested-OpenStack failure mode". Option (c) was refused as the worst of the three: it looks
jumbo at the plane layer while the transit remains capped, so D-108 `ceph-rbd-mirror`
replication and radosgw multisite are throttled by a chokepoint that is invisible where an
operator would look for it.

**COUPLING TO R2, stated because it changes the arithmetic:** the 56-byte overhead in the
budget is the IPv6 figure, and it applies BECAUSE R2 ruled dual-stack. Under a v4-only
deployment the overhead would have been 42 and the 1500-underlay tenant MTU 1458. R2 and
R3 are not independent.

**Execution is a SEPARATE gated step, NOT authorised by this ruling.** Four items, in
`docs/audit/mtu-budget-20260727.txt` section 5: raise `enp1s0` persistently in both
containment VMs; raise the MAAS VLAN MTU records on the plane fabrics; LEAVE the ISP uplink
legs at 1500; re-run the budget with `--underlay-mtu 9000` and capture the closing verdict.
**Verification owed is behavioural, not declarative:** an end-to-end large-frame test with
DF set across the inter-DC path (dc0 rack -> virbr5 -> dc1 rack). An interface claiming
9000 is not proof a 9000-byte frame survives the path -- that is the assert-on-content rule
applied to MTU.

**SUB-RULING 2026-07-27 (R8a) -- the OCTAVIA arm of this verification. OPS under GA-R3 (a
script change; doubt resolves DOWN, so no D-number is assigned).** Question as presented:
given the octavia charm exposes NO MTU option, the failure is silent, and LP #2018998
recurred on our exact pin, how should the `o-hm0` / `lb-mgmt-net` mismatch be caught?
Options: (a) extend the existing o-hm0 verifier to compare MTUs; (b) deploy and rely on the
charm's update-status fix; (c) apply the `ovs-vsctl mtu_request` workaround unconditionally.
Operator selection, exact utterance: **"Extend the existing o-hm0 verifier to compare MTUs
(Recommended)"**.

Why it is cheap: `scripts/phase-05-octavia-verify.sh:104` ALREADY inspects o-hm0 -- it
asserts `state != DOWN` and the presence of an `fc00::/` ULA -- so it holds the handle
already and an MTU comparison is a natural extension, not new machinery. Measured gap:
`grep -i mtu` across `cloud-assert.sh`, `phase-05-octavia-verify.sh` and
`phase-04-network-verify.sh` returns NOTHING -- MTU is asserted nowhere in any post-deploy
verifier. Upstream states the failure precisely: "The lb-mgmt-net is created by the charm
with mtu 8942 while on o-hm0 charm sets mtu 1500."

Option (b) refused: the 2025-12-31 recurrence on octavia 14.0.0 / 2024.1 stable is evidence
the charm-side self-heal is not reliable, and the symptom -- spurious failovers under load
-- reads as a Ceph, network or amphora fault long before anyone suspects MTU. Option (c)
refused: a blind unconditional `mtu_request` fights the charm's own fix where that fix IS
working, and would MASK a future regression rather than surface it.

Delivery: the assertion ships in `scripts/phase-05-octavia-verify.sh` with its harness
green, gauntlet ALL GREEN, repo-lint 0-fail and a changelog entry carrying a revert. Apply
`ovs-vsctl set interface o-hm0 mtu_request=<match>` ONLY if the check fires.

**DELIVERY CORRECTED 2026-07-29 -- the paragraph above was written in the present tense on
the day of the ruling and read as a statement of fact. It was not one.** Measured at that
date: `grep -in mtu scripts/phase-05-octavia-verify.sh` returned NOTHING, and the file's
last commit predated this ruling by a month. So this decision record ASSERTED a delivery
that had not happened -- a sharper form of the RULED-IS-NOT-BUILT class this project keeps
finding, because here the decision doc itself was the false witness rather than merely
silent. **BUILT 2026-07-29:** the comparison resolves `lb-mgmt-net` BY TAG (never by name),
reads o-hm0's MTU from `ip -o link`, FAILS on mismatch in either direction, and REFUSES
(rather than passing) when either value cannot be read. The script also gained its FIRST
harness, `tests/phase-05-octavia-verify/` (10 cases, stubbed `openstack`/`juju`, no live
cloud) -- which is what makes this delivery claim checkable instead of asserted.

**Incidental corroboration of R8, from this repo's OWN tooling:** that verifier expects an
`fc00::/` ULA on o-hm0 and is VR0-era, which independently confirms the octavia charm's
default lb-mgmt-subnet is an IPv6 ULA. The as-built already assumed what R8 ruled.

**G18 RULED 2026-08-08 (GA-R5) -- annotates this R8 note; see `docs/CURRENT-STATE.md` G18
row for the primary record.** The apex-recording question R8 left open (presented separately
as gate G18) is now ruled **option (b)**: the charm-created `lb-mgmt-net` is recorded as
deliberately charm-owned and OUT of apex scope (the apex carries a structural note, not a
concrete prefix row the deploy consumes) -- D-141-compliant, since D-141 governs CONSUMED
allocations and the charm-generated ULA (regenerates per deploy) is not one. R8's ownership /
family / charm-generated-prefix ruling is NOT reopened. The SEPARATE D-139 apex GUA `lb-mgmt`
/64 is kept `reserved` (a distinct MAAS-underlay object with no charm consumer -- see the
D-139 annotation). Operator's exact selection: "B: record out of apex scope + keep D-139 /64
reserved (Recommended)". No new D-number. Prep + governing-decision reconciliation:
`docs/audit/g18-lb-mgmt-ipam-ruling-prep-20260808.md`.

**What IS owed, all mechanical, no ruling required:** propagate the ratified v6 carve into
`scripts/lib-net.sh` (a v6 arm per DC, keyed the same way the v4 `PLANE_CIDRS` arms are),
and carve the twelve plane prefixes into MAAS as subnets on their existing fabrics. Both
have an authoritative source to copy from, so neither is a judgement call. Sequence them
BEFORE the L3-9 overlay-collision reconciliation, since that reconciliation needs the
per-app dual-family values these produce.

### RULING NOTE 2026-07-31 -- D-101 / R2: `prefer-ipv6` is DROPPED from the six charms that do not declare it; the v6 VIP legs STAY

**Status:** RULED 2026-07-31 (operator, GA-R5). Like its predecessors this NOTE records a
ruling and AMENDS NOTHING in D-101's matrix -- the dual-stack posture is UNCHANGED and every
v6 VIP leg is retained. **OPS under GA-R3 (a renderer + gate change; doubt resolves DOWN, so
no D-number is assigned).** It corrects the MECHANISM belief inside R2's implementation, not
the address-family policy R2 confirmed.

Why the question arose: bundle deploy attempt 1 failed 2026-07-31 with `ERROR cannot deploy
bundle: cannot deploy application "barbican": unknown option "prefer-ipv6"`. R2's
implementation applies `prefer-ipv6: true` to all thirteen VIP apps, but only SEVEN charms
declare the option.

**THE MEASUREMENT, from the charm ARTIFACTS at their pinned revisions** (capture
`docs/audit/stage5-prefer-ipv6-charm-research-20260731.txt`; barbican 265, designate 418,
magnum 96, octavia 571, placement 154, vault 724, with keystone 857 + cinder 820 as
declaring-charm controls). **`prefer-ipv6` is not what makes HAProxy bind `:::port`, and
never was, for any of the thirteen.** In BOTH template families the v6 frontend bind is
gated on `ipv6_enabled = not is_ipv6_disabled()` -- a read of the kernel
`net.ipv6.conf.all.disable_ipv6` sysctl, MEASURED `0` on a live MAAS-deployed jammy node in
dc0. Pacemaker selects `ocf:heartbeat:IPv6addr` by per-address family detection, also
independent of the option. Vault ships no haproxy template at all; its listener is the
hardcoded literal `[::]:8200`. What `prefer-ipv6` really is: a UNIT ADDRESS-FAMILY switch --
`get_relation_ip()` returns early with the unit's IPv6 address for every relation, ignoring
the network-space binding.

Question as presented: for those six charms at their pinned channels, does the generated
HAProxy config bind `:::port` without the option -- and given it does, (b) drop the option
only and keep the v6 legs, or (c) drop both and make the six v4-only?

Operator utterance, verbatim (2026-07-31): **"I want it to use IPv6, if there is spam
mechanisms being applied then that is bad. Even if it doesn't cause an issue now, it might
in the future. A clean IPv6 network is better than one with unusable and possibly future
breaking configurations."**

Confirming exchange, verbatim, taken because the first utterance stated a principle rather
than selecting an option and GA-R5 forbids adopting an inferred ruling: **"Yes -- keep the
v6 legs, remove only the option"**.

**Effect.** The six -- `barbican`, `designate`, `magnum`, `octavia`, `placement`, `vault` --
KEEP all three v6 VIP legs each, and `prefer-ipv6` stops being emitted for them. Both DCs,
symmetric.

**What this costs, measured BEFORE the ruling was put so it was taken on the real price:
it is NOT a values-file edit.** `prefer-ipv6: true` is injected UNCONDITIONALLY by
`scripts/render-dc-overlays.py:243-246` for every app whenever `fam == "dual"`, and
`family: dual` is a single TOP-LEVEL key in `render/values/vr1-dc{0,1}-vips.yaml:3`, not
per-app data. Implementing this ruling means (i) teaching the renderer which charms DECLARE
the option, (ii) re-pointing `tests/render-drift`, and (iii) REPLACING
`provider-bundle-check` invariant 9, whose mechanism claim ("prefer-ipv6 makes HAProxy bind
:::port in ADDITION to *:port, so the two must travel together") is refuted above. The
invariant's ARITY half was a real measured defect (L3-9) and is retained; only the coupling
half changes, and the replacement is proven able to fail in both directions. **Never deleted
to go green** -- the standing rule that a checker which cannot fail is not a gate.

**NOT RULED HERE, and deliberately not bundled (GA-R5: one decision per exchange): what the
SEVEN declaring charms should do.** R2 set the option true on all thirteen in order to
obtain v6 VIP binding, which is not what it does. On the seven, true means every relation
carries an IPv6 address chosen from the unit's primary interface, ignoring the network-space
binding -- in a bundle that binds relations across six named spaces. Never observed live.
That is its own GA-R5 exchange, and the operator's stated principle above ("unusable and
possibly future breaking configurations") bears on it but has not been applied to it.

---

### RULING NOTE 2026-07-31 (b) -- D-101: `prefer-ipv6` is set on NO application until IPv6 is operational

**Status:** RULED 2026-07-31 (operator, GA-R5). **OPS under GA-R3** -- it changes no
address-family policy, only which charms receive an option that does not do what R2 assumed.
D-101's matrix is UNAMENDED and every dual-family VIP is retained.

This SUPERSEDES the emission half of the 2026-07-31 (a) note (the six that never declared the
option). The direction is the same and now covers all thirteen.

Question as presented: DC0's deploy is blocked -- all SEVEN charms carrying `prefer-ipv6: true`
are in `error` with `Exception: Interface 'eth0' does not have a scope global non-temporary
ipv6 address`. MEASURED cause: `get_relation_ip()` returns early with `get_ipv6_addr()[0]`
when the option is true, and the LXD containers the API charms run in hold ONLY a link-local
v6 (`keystone/0 eth0 -> fe80::216:3eff:feb4:5518/64`) even though the HOSTS are fully
dual-stacked. Options: set it false on the seven and keep every v6 VIP leg; or stop and build
the missing v6 layers on DC0 first.

**Operator answer, exact utterance: "Set it false on the seven, keep every v6 VIP leg
(Recommended)".** Standing context from the same exchange, verbatim: **"The dual stack was
only a safety net instead of jumping straight into a ipv6 only deployment but it appears that
more items were not configured with IPv6 like they should have been"**, and **"We have DC1 to
stand up with the IPv6 configuration changes. Lets continue with the IPv4/6 stand up on DC0.
We will fold in all lessons learned from the DC0 stand up into the DC1 stand up."**

**Effect.** No application receives `prefer-ipv6`. Every `vip` stays dual-family; node v6
statics, the six per-plane host bridges, both-family spaces and the Octavia v6 SANs are all
untouched. IPv6 remains the TARGET -- D-101's own rationale is "v6 wherever possible, v4 only
where forced" -- and v4 is the safety net it was ruled to be.

**WHY THIS IS NOT ABANDONING v6, stated because it would be easy to read it that way.** The
option is a unit ADDRESS-FAMILY switch, not a listener switch (measured from the charm source
2026-07-31): HAProxy's `:::port` bind is gated on `ipv6_enabled = not is_ipv6_disabled()`, and
pacemaker selects `ocf:heartbeat:IPv6addr` by address-family detection. Neither consults it.

**HONEST RESIDUAL, recorded not glossed:** pacemaker will place a v6 VIP on a container whose
`eth0` has no global v6. Whether that leg is ROUTABLE is **UNVERIFIED**. It is not a
regression -- it is true today -- but it is precisely what the v6 completion work must close.

**THE REAL FINDING BEHIND THIS RULING is a "RULED IS NOT BUILT" instance, and it is bigger
than the charm layer.** A systematic dc0 sweep found the v6 half carved as ADDRESSES but never
made OPERATIONAL: all six v6 plane subnets carry ZERO allocatable ip ranges; the rack holds NO
global v6 at all, so the mirror, node-DNS forwarder and MAAS rack agent are v4-only by
construction; the mirror does not answer over v6; and nodes have no v6 default route. Those
are the layers DC1 is to be stood up with, and then folded back to DC0.

**GATE CONSEQUENCE.** `provider-bundle-check` invariant 9b coupled the option to the v6 legs
for declaring charms; with the option absent and the legs retained it would FAIL. It is
RE-POINTED to the new invariant -- the option must be ABSENT from every application -- and
re-proven able to fail. NOT deleted. When IPv6 becomes operational and the option is
reinstated, 9b returns to its coupling form and this note is the record of why it left.

---

## D-101 -- AMENDMENT (2026-08-10): metal-admin commissioning TARGETS IPv6 (gated on a live canary), reconsidering the "PXE is v4-first" clause

**Status: RULED 2026-08-10 (GA-R5).** Reconsiders D-101's `IPv4 retained: MAAS PXE / provisioning
(PXE is v4-first ...)` clause on FRESH capability evidence -- the object was NOT to follow the prior
ruling but to test whether IPv6 is actually feasible on this link (operator direction 2026-08-10).
The prior clause was an unmeasured JUDGEMENT; the fresh research (pass6, upstream docs/source/bug
trackers, deployed MAAS 3.7.2) found: DHCPv6 commissioning has NO MAAS-side blocker at 3.7.2 (the two
historical blockers LP#1718209/LP#1939034 closed 2017 / MAAS 3.1); MAAS rack<->region over IPv6 is
FEASIBLE-NOW (source prefers `AF_INET6`; LP#2020142 fixed in 3.5.0); the ONLY hard blocker is
legacy-BIOS PXE (Intel Boot Agent / PXE-2.1, IPv4-only firmware) -- which UEFI HTTP boot avoids, and
the VR1 node VMs' loader is ours to set. Record: `docs/audit/container-elim-pass/pass6-w1-maas-ipv6-
feasibility.md` + `pass6-ipv6-feasibility-note-20260810.md`.

**Question as presented.** "Given the fresh capability evidence, should the 10.13 redeploy adopt IPv6
metal-admin commissioning (UEFI + DHCPv6 + HTTP boot), or keep v4 commissioning? Adopt = target it,
gated on a live canary test with v4 fallback, not commit the fleet blind." **Operator selection, exact
utterance: "Adopt v6 (gated on live test)".**

**RULED.** For the 10.13 redeploy, metal-admin node commissioning **TARGETS IPv6** (UEFI-boot VMs +
DHCPv6 + HTTP boot). This AMENDS D-101's "PXE is v4-first" clause -- v4-first was a build choice and an
unmeasured judgement, not a capability ceiling. The metal-admin PLANE stays dual-stack (D-101 07-09
amendment / D-139 unchanged); this ruling concerns the COMMISSIONING/boot-path family only.

**GATE (hard -- no fleet commit on an inferred value, hard rule 2).** IPv6 commissioning is proven on
**ONE canary node** on this stack (DHCPv6 lease -> UEFI HTTP boot -> enlist -> commission -> READY,
captured) BEFORE the full fleet is committed; **v4 commissioning is the fallback** if the canary fails.
The ungated full-fleet option was explicitly NOT chosen.

**Build implications (owed at redeploy design time; logged, not built):** VR1 node-VM definitions set
a UEFI (OVMF) loader; the metal-admin commissioning range gains DHCPv6 + the MAAS HTTP-boot path; the
canary test + its captured evidence is a new gate. Legacy-BIOS nodes are OUT (would reintroduce the
one real firmware blocker). Interacts with D-144's owed metal-admin/power work (the flat rebuild
already redefines these node VMs).

**Reconciliation.** AMENDS D-101 (the PXE-v4-first clause only; the IPv6-primary principle and the
family matrix are UNCHANGED -- this ADVANCES "v6 wherever possible"). CONSISTENT-WITH D-139 (metal-admin
plane stays dual-stack; only the boot-path family shifts). Supersedes the pass5 "PXE stays v4 by ruling"
framing and moots the standalone "soften D-101 PXE wording" DOCFIX (this amendment IS that correction,
now with a ruling behind it). **Roosevelt-delta:** the rehearsal proves the v6-commissioning path a
bare-metal Roosevelt build would want (UEFI HTTP boot over v6); the legacy-BIOS caveat is moot on
modern server firmware. provider-public needed NO reconsideration -- D-101 already rules it dual-stack
with native v6 GUA ext_net; the v4 there is retained for external v4-internet reachability, unchanged.

## D-102: IPv6 tenant addressing and MTU sub-policy (VR1)

**Status:** MERGED INTO D-101 (2026-07-09, operator ruling). Content below is retained for numbering/history only, per this repo's append-only decision-record discipline -- D-101 is now the current, authoritative text for this sub-policy. Do not amend the body below; amend D-101 instead.

**Decision (leans; operator to ratify):**
- Tenant v6 routing: native GUA routing, no NAT66 -- the simulated OPNsense edge routes tenant GUA prefixes (prefix delegation or static routes). NAT66 is discouraged and we own the edge, so native is both correct and feasible.
- Tenant v6 block size: /60 per tenant default (~4080 per cloud); /56 available on request.
- Tenant address mode: SLAAC default (RA); DHCPv6-stateful available for tenants needing deterministic addressing.
- MTU / geneve-over-v6: compute the exact byte budget in Phase 0 against the MEASURED host L2 MTU. An IPv6 outer header adds 20 bytes over IPv4 (40 vs 20); geneve-over-v6 overhead is about 40 (IPv6) + 8 (UDP) + 8 (Geneve base) = 56 bytes, before nested-virt and options. Recommendation: raise the vcloud underlay to jumbo (9000) end-to-end so tenant MTU stays 1500 and amphora/geneve fit; if the underlay is pinned at 1500, tenant MTU drops to about 1444 (v6-geneve) and MUST be set consistently across ovn geneve, tenant-network MTU, and amphora. The measured underlay MTU is a Phase-0 gate -- do not assume jumbo.
- Phase-A rollback posture: go straight to IPv6-only on the internal three (data / storage / replication) rather than dual-stack-then-flip, given plane isolation and the Octavia ULA proof. Dual-then-flip is the conservative alternative if Phase-0 verification of Ceph-over-v6 / geneve-over-v6 is not clean.

**Rationale:** matches the developed plan's recommendations; native routing plus SLAAC is the IPv6-correct default; the MTU budget is the classic nested-OpenStack failure mode and is handled by measurement plus jumbo.

**References:** extends D-101. Ceph-over-v6 (`ms_bind_ipv6`) and OVN geneve-over-v6 are Phase-0 verification gates in the buildout doc.

---

## D-103: OpenTofu / MAAS / Juju lifecycle seam (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling). Resolves the double-specified node-VM ownership question.

**AMENDED 2026-07-16 (D-123 Model B):** the DC node-VM PLACEMENT half of this seam is SUPERSEDED --
the vr1-dc0/dc1 OpenStack node VMs are now created INSIDE the site containment VM (`vvr1-dc0`), not at
vcloud level, and the in-VM MAAS rack discovers them there. The OpenTofu-creates + MAAS-discovers
lifecycle is otherwise unchanged; only the libvirt HOST the nodes land on moves from vcloud to
`vvr1-dc0`. Office1's placement is unaffected.

**AMENDED 2026-07-20 (VM-host registration MECHANISM refuted by measurement; INTENT stands):** the
"registers ... as a virsh VM-host" bullet below is DEAD for the DCs, on two measured grounds from the
step-D deploy: (1) snap-confined MAAS gets `Permission denied` on `/var/run/libvirt/libvirt-sock` and
`snap connections maas` lists NO libvirt interface, so a LOCAL `qemu:///system` pod is impossible with
snap MAAS; (2) the `qemu+ssh` replacement pod then fails at refresh on `domblkinfo ... missing storage
backend for 'volume'` (reproduced locally on the rack with an active pool) -- MAAS virsh pods are
incompatible with `modules/node-vm`'s pool+volume disk refs. The pod was also UNNECESSARY: its job in
this seam was DISCOVERY, and PXE enlistment across the transit already delivers it (all 9 nodes
enlisted 2026-07-20, `docs/audit/stepD-enlistment-20260720.txt`). **RULED 2026-07-20: per-machine
`power_type=virsh`** (`scripts/maas-node-power.sh`, MAC-matched, write-verified by
`query-power-state`) -- which is ALSO the Roosevelt shape (per-node IPMI/BMC, no pod). Everything
else in this seam (OpenTofu creates, MAAS owns commission/deploy/power/release, never composes)
SURVIVES and is proven end to end. `modules/maas-vm-host` and `opentofu/vr1-dc0-maas/` are refuted
for DC use; their retire-or-keep is a stage-close question (SEC-013 tied).

**Decision:**
- OpenTofu owns create / destroy of: the libvirt domains that stand in for DC "hardware"; all virtual networks (the dark-fiber, plane, and ISP-edge segments); the OPNsense edge VMs; and the Office1 service VMs (MAAS-region, NetBox, GitBucket). OpenTofu runs from the Office1 operator VM against the vcloud host libvirt.
- OpenTofu registers each DC's libvirt host to that DC's MAAS rack controller as a virsh VM-host, so MAAS DISCOVERS the OpenTofu-created node VMs.
- MAAS owns commission / deploy / power / release of those node VMs; it does NOT compose new ones. Juju then deploys the OpenStack bundle onto the MAAS-deployed machines.
- NetBox remains the IPAM apex; MAAS and overlays consume NetBox values, never the reverse. **(DOCFIX-195, 2026-07-15: during VR1 that apex is `office1-netbox` (10.10.1.10) -- the working draft every consuming system reads; `netbox.baldurkeep.com` is the eventual production apex, frozen as the v1 reference draft with merge-back deferred. Defers the write, does not change this consume-not-reverse rule.)**

**Rationale / tradeoffs:** this keeps the entire physical substrate in versioned OpenTofu (auditable, reproducible, and exactly the "operator scripts the deployment from Office1" model), while MAAS owns the provisioning lifecycle -- mirroring Roosevelt, where MAAS enlists hardware that already exists rather than creating it. "MAAS owns lifecycle" here means the provisioning lifecycle, not domain creation; that reconciles the two operator inputs.

**Delta callout (test-env shim):** the OpenTofu "create the node VMs" module has NO Roosevelt analog. In Roosevelt that step is physical racking plus BMC enlistment, and MAAS enlists real hardware. This module is tagged in the buildout doc as a virtual-only shim and must NOT be mistaken for reusable Roosevelt IaC. Everything downstream of enlistment (MAAS commission / deploy, Juju bundle) IS Roosevelt-transferable.

**References:** ratifies MAAS-as-VM-host in the enlist sense; NetBox apex per the operating skill and environment reference; feeds Phases 2-4 of the buildout doc.

---

## D-104: Juju controller topology and HA posture (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling). Adds the controller-topology / HA posture the DC-DC phase was gated on. References D-071 (patch cadence); does NOT amend it.

**Decision:**
- One Juju controller PER DC, deployed in-DC (on a MAAS-deployed machine in that DC), each managing only its own cloud. DR-independent: a controller loss affects only its DC.
- Single-unit (not HA) for VR1, with the confirmed backup posture (juju 3.6 `create-backup` / `download-backup`). Controller backups scheduled and a restore drill run in Phase 5.
- HA (3-unit) DEFERRED to Roosevelt, when bare-metal resources allow. Recorded as a known VR1 test-only posture, not a Roosevelt recommendation.
- Office1 hosts NO OpenStack Juju controller (Office1 = MAAS-region + OpenTofu + NetBox + GitBucket only).
  **[AMENDED by D-116, 2026-07-13: the GitBucket half is SUPERSEDED. No Office1-local GitBucket is built; `git.baldurkeep.com` remains the git service of record. Office1 = MAAS-region + OpenTofu + NetBox + Tailscale.]**

**Rationale / tradeoffs:** per-DC single controller is DR-honest (independence) and resource-frugal on the 256 vCPU / 1 TiB / 10 TiB host. A shared or Office1-hosted controller would couple both DCs' manageability -- a controller outage would take out both -- the opposite of what a DR drill should prove. Single-vs-HA is a deliberate test-only economy; backups are the mitigation and are drilled. Roosevelt scales to 3-unit HA per DC.

**References:** relates to D-071 (routine update cadence / Juju controller patch policy) without amending it; interacts with D-108 (per-DC control-plane backup / restore in the DR drill).

### D-104 -- AMENDMENT (2026-07-24): the per-DC controller is a DEDICATED 10th node VM, not one of the 9

**Status:** ADOPTED 2026-07-24 (operator ruling, GA-R5). Question as presented
(Track-2 Fork 1, committee-reviewed
`docs/audit/committee-20260724-track2-bundle-render.md`): "The vr1-dc1 substrate
is exactly 9 nodes (3 control + 2 compute + 4 storage), all needed for D-121
Option C roles, with no controller slot. D-104 requires a per-DC Juju controller
on an in-DC MAAS machine (VR0 precedent = a dedicated `juju` VM). Where should the
dc1 Juju controller live?" Operator answer, exact selection: **"Dedicated 10th VM
(Recommended)"** (over "Bootstrap onto a node, shrink a role" and "Co-locate on a
control node"). The committee was unanimous for this option on all four lenses.

**Decision.** D-104's "in-DC MAAS-deployed machine" is instantiated as a
DEDICATED controller VM per DC, SEPARATE from the D-121 Option C role nodes. The
built substrate carried exactly 9 role nodes (a latent gap against D-104); this
amendment closes it by ADDING a 10th node VM to each DC substrate for the Juju
controller. It matches the VERIFIED VR0 precedent (a dedicated `juju` VM separate
from openstack0-3, `docs/maas-as-built-reference.md`) and the Roosevelt end-state
(dedicated controller hardware, scaling to 3-unit HA there).

- **The 9 OpenStack-role nodes are UNCHANGED** -- D-121 Option C (3 control + 2
  compute + 4 storage) stands; the controller does NOT consume a role node, so the
  Ceph rebuild headroom (R-3) and the quorum/anti-affinity spread are preserved.
  The rejected alternatives each broke exactly one of those (shrink-a-role reversed
  R-3; co-locate collapsed the 3-node control quorum).
- **Additive, not a substrate reopening** -- the inner caller is
  `for_each = local.vr1_dc1_nodes` (`opentofu/vr1-dc1-substrate/main.tf:140`),
  keyed; adding the 10th key plans `1 add / 0 change / 0 destroy`.
- **Distinct tag, no role tag** -- the controller VM carries its OWN MAAS tag
  (`juju-controller-vr1-dc1`) and NONE of the role tags, so `juju bootstrap
  --constraints tags=juju-controller-vr1-dc1` targets it deterministically and no
  OpenStack `tags=` role constraint can cross-grab it (pairs with the Fork-3
  per-role-tag scheme).
- **Capacity GATE: PASS (measured, not inferred).** `scripts/dc-dc-whole-host-budget.py`
  for the 10-node x 2-DC shape (controller modeled as +8 GiB / +4 vCPU per-DC
  containment overhead) -> RAM 854/1024 GiB = 83%, FIT, 170 GiB headroom
  (`docs/audit/stage5-controller-capacity-20260724.txt`; baseline 3+2+4 = 838 GiB
  / 82%). vCPU allocation 96% (overcommits in a sim; not a hard ceiling).
- **Controller sizing** is small/single-unit (VR1 test-only economy, unchanged);
  the exact vCPU/mem/disk is a substrate-authoring value, MEASURED into the tfvars
  at build time, not fixed here.

**Roosevelt analog:** a dedicated per-DC controller machine (bare metal), scaling
to 3-unit HA. **Cross-note to D-121:** the Option C node COUNT (9 role nodes/DC)
is not changed by this amendment; the controller is a separate machine, tracked
here under D-104.

---

## D-105: CAPI management-cluster placement and COS / Magnum scope (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling). Carries D-035 to the two-DC topology.

**Decision:**
- Each DC runs its OWN in-cloud, single-homed capi-mgmt cluster (per D-035), independent per DC. No shared or Office1 management cluster.
- COS (observability) and Magnum / CAPI are in scope in BOTH DCs.
- The Magnum `domain-setup` action is re-run after every teardown / redeploy per DC (D-046 trap: magnum reports "Unit is ready" regardless).

**Rationale:** D-035 already placed capi-mgmt in-cloud single-homed; per-DC keeps each cloud self-contained and DR-independent. COS per DC (not central at Office1) keeps observability inside each independent failure domain.

**References:** carries D-035; carries D-046 (domain-setup re-run); COS / Magnum scope confirmed by operator.

**Sub-item RULED 2026-07-09 (operator):** COS is per-DC-only for VR1, CONFIRMED (no Office1 roll-up) -- revisit for Roosevelt.

---

## D-106: Designate reactivation (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling). SUPERSEDES D-019 (Designate deferral); REACTIVATES D-008 (DNS architecture).

**ANNOTATION 2026-07-30 (DOCFIX-205) -- THE ZONE LABELS BELOW ARE SUPERSEDED BY D-117.**
Recorded per D-117's own treatment table ("ADOPTED decision text: ANNOTATE in place, do NOT
rewrite -- ratified prose is a historical record"), so the Decision block is left exactly as
ratified and this block carries the correction.

D-117 (ADOPTED 2026-07-13, four days after this entry) names this supersession in its own
Status line -- "Supersedes ... and the D-106 `dc1`/`dc2` zone labels" -- and its amendment
rules the replacement: "The repo's `dc1`/`dc2` labels are retired in favour of `dc0`/`dc1`."
READ the labels below as:

    `dc1.vr1` -> `dc0.vr1`   (substrate `vr1-dc0`, NetBox site `vr1-dc0`, GUA f02::/48)
    `dc2.vr1` -> `dc1.vr1`   (substrate `vr1-dc1`, NetBox site `vr1-dc1`, GUA f03::/48)

The SHAPE is unchanged: D-008's `<service>.<cloud>.<dc>.<region>.cloud.neumatrix.local` still
governs, so the VR1 FQDNs are `<service>.omega.dc0.vr1.cloud.neumatrix.local` and
`<service>.omega.dc1.vr1.cloud.neumatrix.local`.

**Do NOT collapse `<dc>.<region>` into D-119's single `vr1-dc0` selector token.** D-119's
standing rule ("never write a bare `dcN` for a VR1 datacenter again ... region-qualify it")
states its own rationale as "a reader who does not know which cloud the sentence is about" --
a hazard that does not exist here, because in this FQDN the very next label IS the region.
D-119 also says it COMPLETES D-117 rather than reversing it, and D-117's ruled replacement
labels are themselves bare `dc0`/`dc1`; a rule forbidding those in every context would
invalidate the ruling D-119 says it completes. Collapsing the two labels would additionally
amend D-008's ratified shape, which no ruling has done.

**The selector token and the D-008 `<dc>` label are DIFFERENT STRINGS.** Do not interpolate a
`$DC` selector value into an FQDN -- `vr1-dc0` in the `<dc>` position yields
`omega.vr1-dc0.vr1.cloud...`, which carries the region TWICE. Derive both halves from the site
token instead: `${SITE%%-*}` -> region (`vr1`), `${SITE#*-}` -> DC label (`dc0`). Verified for
`vr1-dc0`, `vr1-dc1` and `vr0-dc0`.

**Why this annotation exists.** It was owed from 2026-07-13 and never written. On 2026-07-30 a
Stage-5 session read this un-annotated text as live, re-derived the already-retired
`dc1`-vs-`dc2` question from it, and filed it as a blocking ruling -- see the D-117 Status
correction of the same date for the mechanism that hid it.

**Decision:**
- Designate is deployed in-bundle in each DC, backend `designate-bind` (bind9, the Charmed-OpenStack-native backend, lowest delta).
- Naming per D-008 instantiated with the VR1 labels: `<service>.omega.dc1.vr1.cloud.neumatrix.local` and `...dc2.vr1...`. Per-DC INDEPENDENT zones (independent Keystones), served with A and AAAA records (AAAA for the v6-enabled planes per D-101).
- The D-008 bootstrap order applies per DC: static /etc/hosts bootstrap -> `os-public-hostname` per API charm -> Vault issues FQDN-SAN certs -> Designate zone plus A/AAAA -> Neutron `default_dns_domain` and `dns_servers` -> tenant subnets with `--dns-nameserver`.

**Rationale / consequence:** reactivating Designate plus FQDN-SAN certs removes the D-019 documented root cause (metal-only charms pulling the public FQDN endpoint they cannot resolve -- the D-021 amphora-pipeline constraint). The D-008 convention was designed multi-DC from the start (`dc0.vr0`), so this is reuse, not new design -- minimal delta.

**References:** supersedes D-019; reactivates D-008; resolves the D-019 / D-021 root cause; AAAA depends on D-101. Office1 headend services use a region-level subdomain outside the per-cloud convention (e.g. `maas.office1.vr1...`) -- exact scheme is a buildout-doc detail.

**Sub-item adopted per its stated lean (2026-07-09):** cross-DC name resolution (a DC1 tenant resolving a DC2 name) is NOT needed for VR1 (independent clouds) -- this was not one of the six items the buildout doc listed for explicit operator redline, so it is adopted along with the rest of D-106 rather than separately re-asked; revisit only if a federated-service requirement emerges.

---

### RULING NOTE 2026-07-27 -- D-106: Designate DEPLOYS at Stage 5 and is CONFIGURED at Stage 7; Stage 7 Step 5 stops being a deploy step

**Status:** RULED 2026-07-27 (operator, GA-R5). This note records a ruling and AMENDS
NOTHING in D-106's bootstrap order, which stands exactly as written.

Question as presented, verbatim: "R5 -- Designate. The four apps and eight relations are
deploy-ready at Stage 5 and will deploy from the bundle, but sit inert because
os-public-hostname is set nowhere. Stage 7's Step 5 is written as a DEPLOY step with a gate
requiring the diff show only new designate apps -- and phase-6 contradicts itself internally
(line 172 says no designate block exists, line 178 says it is deployed in-bundle). How
should this resolve?" Options presented: (a) accept at Stage 5 and rewrite Stage 7 Step 5 to
configure-not-deploy; (b) suppress designate at Stage 5 to preserve the incremental-add
shape; (c) accept at Stage 5 AND set `os-public-hostname` there. Operator selection, exact
utterance: **"Accept at Stage 5; rewrite Stage 7 Step 5 to configure-not-deploy
(Recommended)"**.

**CORRECTION RECORDED IN THE DECISION ITSELF.** When this question was first put to the
operator, the audit stated that option (a) "inverts D-106's bootstrap order, which puts
`os-public-hostname` + FQDN-SAN certs BEFORE Designate". **That was wrong.** D-106's order
(`static /etc/hosts -> os-public-hostname per API charm -> Vault issues FQDN-SAN certs ->
Designate zone plus A/AAAA -> Neutron default_dns_domain and dns_servers -> tenant subnets
with --dns-nameserver`) is a CONFIGURATION sequence. It governs when the zones and DNS
wiring happen, NOT when the charm is installed. Deploying the designate application earlier
does not violate it: the zones still come after the certs. Installing a charm and doing
D-106's work are different things, and conflating them nearly cost the operator a decision
made on a false constraint.

**MEASURED STATE (2026-07-27).** The bundle carries four applications -- `designate`
(`2024.1/stable`, `num_units: 1`, `to: [lxd:0]`), `designate-bind` (`2024.1/stable`,
`to: [lxd:2]`), `designate-mysql-router` (`8.0/stable`) and `designate-hacluster`
(`2.4/stable`, `cluster_count: 1`) -- plus EIGHT relations, all of whose peers Stage 5
creates: `mysql-innodb-cluster`, `keystone`, `rabbitmq-server`, `vault`, `memcached` and
`designate-bind` itself. They are therefore DEPLOY-READY at Stage 5 and will deploy.
They will also be FUNCTIONALLY INERT: `os-public-hostname` is set in NO deploy artifact --
`bundle.yaml:11` records the current posture as "IP-ONLY -- os-public-hostname dropped on
all API charms; the dual VIPs ARE the catalog endpoint". So the charm runs and the feature
does not, which is precisely the state Stage 7 exists to resolve.

**THE SURFACE DEFECT: `runbooks/dc-dc-phase6-designate-cos-magnum.md` CONTRADICTS ITSELF.**
`:172` states "There is no `designate:` or `designate-bind:` application block anywhere in
..." while `:178` states "Designate is deployed in-bundle in each DC, backend
`designate-bind`". Both cannot be true; the bundle settles it, and DOCFIX-167 put designate
there on 2026-07-10. Step 5 (`:351`) is written as "Deploy designate + designate-bind into
the LIVE per-DC model" with a gate (`:366-367`) requiring "the diff shows ONLY the new
designate/... applications and their relations being added" -- unsatisfiable against a model
that already has them. The runbook already knows where the real work is: `:210` calls it a
"larger surface than 'add designate': every API charm's `os-public-hostname`".

**RULED SHAPE.** Designate lands as INERT INFRASTRUCTURE at Stage 5. Stage 7 retains the
substantive D-106 work it always owned -- `os-public-hostname` on every API charm, Vault
FQDN-SAN certs, the zone plus A/AAAA records, Neutron `default_dns_domain`/`dns_servers`,
and tenant subnets with `--dns-nameserver` -- and Step 5 is rewritten from "deploy" to
"verify present and active, then configure". Its diff-gate is replaced by a presence-and-
state assertion. Step 1's inventory prose (`:172`) is corrected to match `:178` and the
bundle. The rewrite is owed regardless of this ruling, because the runbook is internally
inconsistent today.

**Why (b) was refused:** suppressing designate at Stage 5 needs machinery that does not
exist (a subtractive overlay or a rendered per-stage bundle) and partially undoes
DOCFIX-167, a deliberate decision. It would build a mechanism to satisfy a stale gate
rather than fix the gate.

**Why (c) was refused, and this one matters:** setting `os-public-hostname` at Stage 5 is
the ONE option that genuinely collides with D-106. Publishing a public FQDN endpoint before
Vault has issued FQDN-SAN certs recreates the exact D-019 root cause D-106 was written to
remove -- metal-only charms pulling a public FQDN endpoint they cannot resolve, which is
also the D-021 amphora-pipeline constraint. It would need its own ruling on cert sequencing
and was not taken.

**Execution:** a DOCFIX against the phase-6 runbook (Step 1 inventory prose, Step 5 verb and
gate). Logged to the Stage-5 audit's Phase-3 batch; not executed under this ruling.

## D-107: Airgap posture, per-DC artifact mirror, and NTP (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling).

**Decision:**
- Airgap is enforced at the NODE level: OpenStack nodes (control / compute / ceph) reach NO internet directly. They pull all artifacts (apt, snap, charmhub, container registry, images including amphora) ONLY from an in-DC mirror.
- PER-DC mirror (not a single Office1 mirror): each DC hosts its own mirror so a DC can redeploy independently even if Office1 or the peer DC is down -- a DR requirement the drill exercises.
- Controlled egress lives on each DC's OPNsense edge, narrowly scoped: NTP-only, plus the mirror's upstream sync. Nodes get time (chrony) from their DC edge, which syncs upstream to internet NTP pools. Office1 is NOT in the core-service path (no NTP, no node artifacts served from Office1).

**Rationale / tradeoffs:** this reconciles "airgap-honest" with the operator's "each DC syncs internet NTP" -- the airgap is at the node boundary, not the site boundary, and the edge holds a minimal controlled egress. Per-DC mirrors cost more disk (a factor in the D-101 / Ceph disk budget) but buy true DC-redeploy-independence, which a single Office1 mirror cannot. The Roosevelt airgap posture is the same shape (per-DC mirror, edge-controlled egress).

**References:** pairs with D-100 (per-site ISP edge), D-101 (disk budget), and the Phase-0 disk math.

**Sub-item RULED 2026-07-09 (operator):** mirror sync topology is independent upstream pull per DC, CONFIRMED (not one DC pulling and seeding the peer) -- for full DC-redeploy-independence; measure the disk cost in Phase 0.

### D-107 -- AMENDMENT (2026-07-16): scope "node artifacts" (C4); make the DR-independence claim honest (C5)

Two review corrections (R3-F03/F05); the per-DC-mirror + edge-egress + NTP posture is otherwise unchanged:
- **(C4) "no node artifacts served from Office1" is SCOPED to the supply-chain mirror** (apt / snap /
  charmhub / container registry / amphora images) -- the class served to nodes at and after Juju deploy.
  It EXCLUDES the intra-MAAS provisioning-image channel: under D-123 the DC rack proxies MAAS ephemeral
  commission/deploy (PXE) images FROM the Office1 region, a different control-plane channel. Both D-107
  and D-123 stand once "node artifacts" is read this way.
- **(C5) The DR claim "a DC can redeploy independently even if Office1 ... is down -- a DR requirement
  the drill exercises" is CORRECTED.** (i) Redeploy is a MAAS operation, and with the only region on
  Office1 (D-123, matching buildout-design's PERMANENT single-region), an Office1 outage removes
  commission/deploy/power from BOTH DCs -- a permanent architectural property, not stale VR1 wording.
  (ii) The Phase-5 drill invokes NO MAAS -- it exercises Ceph/Glance/Cinder/Neutron WORKLOAD failover,
  not node redeploy. **Honest scope:** the per-DC mirror gives ARTIFACT + WORKLOAD-FAILOVER independence
  (running DC clouds survive an Office1 outage -- MAAS is a day-1/2 dependency, not runtime); node
  PROVISIONING is unavailable during an Office1-region outage, accepted by design (a standby region per
  DC would contradict the single-region decision). The peer-DC-down half of the original claim stands.

---

## D-108: Cross-DC replication mechanism (VR1)

**Status:** ADOPTED 2026-07-09 (operator ruling). Engineers the mechanism for the SETTLED Tier-A DR decision (`docs/archive/dc-dc-replication-DR-seed.md`); does not reopen the posture.

**Decision (mechanism):**
- Cinder volumes: `cinder-backup` to a cross-DC-replicated object store via `ceph-radosgw` MULTISITE (a zonegroup with the two DCs as zones; bucket-level bidirectional sync) as the replicated backup target. A restore reconstitutes the volume with its Cinder metadata in the peer -- the honest cross-independent-cloud path. Multisite is chosen over a single mirrored bucket for bidirectional, per-object async replication with conflict handling.
- Glance images: `ceph-rbd-mirror` of the Glance pool, snapshot-based, two-way, with a re-register step on failover. Daemon topology: one rbd-mirror daemon per DC, peering over the replication plane; daemon HA is single-unit for VR1 (matches the D-104 controller economy), noted for Roosevelt scale-up.
- Nova ephemeral: NOT replicated (rebuild from image plus volume).
- Carrier: the cross-DC leg (radosgw multisite sync plus rbd-mirror peer traffic) rides the IPv6-only ULA replication plane (per D-101; corrects the DR-seed v4 carrier). Explicit inter-DC route plus a bandwidth budget (MTU-aware; see D-102), distinct from intra-DC OSD cluster traffic.
- Consistency groups for multi-volume apps where a crash-consistent set matters.
- Staging: prove one-way (single mechanism, single direction) plus a clean failover / failback drill FIRST, then enable two-way. RPO ~15 min tunable; RTO is the measured drill output, not an SLA.
- Neutron state (ports / FIPs / security groups) is NOT replicated; it is recreated on failover per the runbook.

**Rationale:** matches the settled seed; multisite plus rbd-mirror are the delta-minimal Ceph-native primitives. This answers the seed's open questions (multisite vs mirrored bucket -> multisite; daemon HA -> single-unit VR1; peer bootstrap -> on the replication plane).

**References:** engineers `docs/archive/dc-dc-replication-DR-seed.md`; carrier per D-101; interacts with D-104 (per-DC control-plane backup / restore in the drill); the failover / failback skeleton is fleshed out in the buildout doc.

---

## D-109: Vault posture for VR1

**Status:** ADOPTED 2026-07-09 (operator ruling). Confirmed to STAND ALONE, not merged into D-101 despite D-102 folding in the same day -- Vault/security posture is a distinct-enough surface to keep independently amendable.

**Decision:**
- Per-DC INDEPENDENT Vault roots (each DC's Vault is its own root CA; no regional root-of-trust this phase).
- Vault pinned 1.8/stable (the only viable charm lineage for this stack; the 1.16 operator lineage is ruled out per D-068). The D-068 EOL-substrate change is deliberately HELD OUT of VR1 to keep variables constant.
- Each DC's Vault issues certs with both IPv4 and IPv6 (ULA) SANs for the dual-stacked metal-internal API endpoints (per D-101).

**Rationale / tradeoffs:** independent roots keep the two clouds cryptographically independent (DR-honest) and match the confirmed operator ruling. Holding 1.8/stable isolates the DC-DC variables from the unresolved substrate modernization; the deliberate non-change is recorded against D-068 so it is on the record, not an oversight.

**References:** references D-068 (substrate hardening deferred to Roosevelt; getting off EOL 1.8.8 stays open there); v6-SAN per D-101.

---

## RULING NOTE 2026-07-29 (b) -- D-109: each DC's Octavia PKI is GENERATED ON the Plane-2 headend

**Status: RULED 2026-07-29.** OPS under GA-R3 -- it fixes an execution LOCATION for an
already-ruled artifact, so doubt resolves DOWN and no new D-number is assigned. Recorded here
because D-109 is the Octavia-PKI authority after its 2026-07-27 amendment (R7), and because the
choice fixes where a CA PRIVATE KEY lives at rest, which is security posture rather than
mechanics.

**Question as presented.** Three surfaces contradicted each other, measured 2026-07-29:
`runbooks/phase-01-bundle-deploy.md` Step 1.0-GEN says **RUN -- jumphost**;
`creds-manifests/host-identity` binds `jumphost` to **vcloud** and the eighteen Octavia rows in
`creds-matrix.tsv` carry `host-role=jumphost`; but `runbooks/dc-dc-phase4-juju-bundle-per-dc.md`
sets RUN LOCATION to **`voffice1`, "NOT the vcloud jumphost"**, and states that the
"deliberately-absent gitignored octavia PKI overlay ... is another reason this runs on
`voffice1`". Measured client inventory settles which host can deploy at all: **`juju` is ABSENT
on vcloud and 3.6.27 on voffice1.** So generating per the runbook's own label would have put the
PKI on a host the deploy cannot read it from.

Options presented: (a) generate ON voffice1 -- one at-rest location per DC's CA; (b) generate on
vcloud as the register declares, then transfer the overlay to voffice1 as an explicit gated step
-- TWO at-rest locations per CA private key, each needing its own register rows; (c) hold and
rule separately, leaving Stage 5 blocked on the overlay.

**Operator selection, exact utterance: "Generate on voffice1 (Recommended)".**

**WHY THIS WAS PUT BEFORE GENERATION RATHER THAN AFTER.** F3 of the same session established
that `~/octavia-pki/` and the overlay are ABSENT on both hosts, so nothing exists to migrate and
the fix is free -- and that window closes permanently the moment the first CA is minted, because
from then on any change of location is a key MOVE rather than a path edit. Option (b) was
declined on the same posture ground R7 used to refuse a shared CA: the overlay carries CA
private keys plus a plaintext issuing-CA passphrase, SEC-004 records this repo as PUBLIC, and a
second at-rest copy widens that exposure instead of containing it.

**Work implied (each gated, NOT authorised by this ruling):** relabel Step 1.0-GEN's RUN
markers to `voffice1` and state that `$REPO` there means the headend clone; move the eighteen
Octavia rows from `host-role=jumphost` to `headend`; point the Octavia entries in
`vm-secret-locations` at the headend; and bind `headend` in `host-identity` so the F6 wrong-host
refusal covers them -- on vcloud they must read NOT PROBED rather than absent.

**Roosevelt analog:** artifacts are generated on the host that consumes them, so a per-DC secret
has exactly one at-rest location and no copy step to forget.

## RULING NOTE 2026-07-29 -- D-109: the Octavia controller cert gains IPv6 IP SANs

**Status: RULED 2026-07-29.** OPS under GA-R3 -- a generator change extending an
already-ruled principle to a surface D-109 already governs; doubt resolves DOWN, so no
new D-number is assigned. Recorded here because D-109 is the Octavia-PKI authority after
its 2026-07-27 amendment (R7).

**Question as presented.** The R7 work of 2026-07-29 left the controller cert's IP SAN
derived by `.split()[0]` -- the PROVIDER v4 leg only -- while ruling 3 commit 2 had made
every API VIP dual-family. So the amphora control-plane cert carries no IPv6 IP SAN even
though the VIP it is derived from now has three v6 legs. This was flagged as a gap rather
than a break (it is a separate trust domain from the API TLS that Vault issues with IPv6
SANs under D-109's original text), and was put to the operator as needing its own ruling.

**Operator answer, exact utterance: "Yes, add the v6 IP sans."**

**Scope, stated precisely so a later reader does not over-read it.** The generator now
emits `IP.1` = this DC's provider **v4** VIP leg and `IP.2` = its provider **v6** leg,
both DERIVED from the same per-DC overlay under the `$DC` selector R7 introduced. The
admin and internal legs stay EXCLUDED, exactly as they were before this ruling -- the
DOCFIX-067 design has always been provider-leg-only, and widening it was not what was
asked. A v4-only (three-address) VIP still yields `IP.1` alone and no v6 SAN, so the
generator remains correct on a tree where the dual-stack ADD has not landed.

**OBSERVATION RECORDED, NOT ACTED ON.** The address amphorae actually use to reach the
controller is `o-hm0`'s charm-generated `fc00::/64` ULA (`controller_ip_port_list`), NOT
any VIP leg -- so neither `IP.1` nor `IP.2` matches that path. Whether the amphora
verifies the controller cert against an IP at all is not asserted here: measured upstream,
the CONTROLLER verifies the amphora by UUID (`assert_hostname = self.uuid`), and the
reverse direction was not established from source. This ruling therefore makes the SAN set
consistent with the VIP it has always been derived from; whether that is the right
derivation is a separate question, and the place to settle it is by inspection at the
Octavia step of Stage 5.

## D-109 -- AMENDMENT (2026-07-27): per-DC cryptographic independence EXTENDS to the Octavia control-plane PKI

**Status:** RULED 2026-07-27 (operator, GA-R5). Question as presented, verbatim: "R7 --
Octavia CA scope. The generator bakes a 'VR0 DC0' CA subject and gates on dc0's VIP band, so
it cannot produce a dc1 artifact as written; a DOCFIX is owed either way. D-109 rules per-DC
independent Vault roots for DR-honesty but does not mention Octavia PKI. Do the two DCs
share one Octavia CA, or get independent trust domains?" Options presented: (a) per-DC
independent Octavia PKI, fixing the generator first; (b) reuse dc0's existing CA across both
DCs; (c) fix the generator now and defer the trust-domain decision. Operator selection,
exact utterance: **"Per-DC independent Octavia PKI; fix the generator first (Recommended)"**.

**WHAT THIS AMENDS.** D-109 established "Per-DC INDEPENDENT Vault roots (each DC's Vault is
its own root CA; no regional root-of-trust this phase)", rationale "independent roots keep
the two clouds cryptographically independent (DR-honest)". It did NOT mention the Octavia
amphora control-plane PKI, which is a SEPARATE trust domain with its own CA, generated
outside Vault by `runbooks/phase-01-bundle-deploy.md` step 1.0-GEN. This amendment extends
the same posture to it: **each DC gets its own Octavia CA; no cross-DC amphora
root-of-trust.**

**MEASURED STATE -- the generator is dc0-frozen in TWO ways of DIFFERENT severity**, and the
distinction matters because only one of them is a real design problem:
- The CA SUBJECT is a BAKED LITERAL: `-subj "/CN=VR0 DC0 Omega Cloud Octavia Controller
  CA/O=Neumatrix"`. Reusing it puts a VR0-DC0-named CA at the root of dc1's amphora chain.
- The controller cert's SAN is **DERIVED, not baked** -- the step states it carries the
  controller FQDN, the octavia API FQDN and the Octavia API VIP "derived from the bundle at
  generation time -- DOCFIX-067; never a baked literal". **That half is already
  per-DC-correct by design** and needs no change.
- The blocker is the VIP gate: `grep -qE '^10\.12\.4\.[0-9]{1,3}$'` accepts ONLY dc0's band,
  while dc1's octavia VIP is `10.12.64.57` and lives in `overlays/vr1-dc1-vips.yaml`, not in
  `bundle.yaml` where the generator reads it (audit finding L5-3).

**A DOCFIX IS OWED REGARDLESS OF THIS RULING.** The generator cannot produce a dc1 artifact
as written, and `runbooks/phase-01-bundle-deploy.md:144-145` hard-ABORTS the deploy when
`overlays/octavia-pki.yaml` is absent -- which it currently is (it is a gitignored secret,
and preflight P4 CHECK 0 fails on it by design). So option (c) would have deferred only the
part D-109 already answers by precedent, while leaving the required work untouched.

**SECURITY POSTURE, and why reuse was refused.** `overlays/octavia-pki.yaml` carries CA
private keys plus a plaintext issuing-CA passphrase inside the repo clone, and SEC-004
records the repo as still PUBLIC. Reuse would place ONE amphora control-plane CA private key
across two clouds that D-100 defines as independent -- widening an existing exposure rather
than containing it, in a commercial multi-tenant cloud with hard tenant isolation (the SCS
Domain Manager posture). Independent per-DC CAs keep the blast radius of that exposure to
one DC.

**Work implied (each gated, NOT authorised by this ruling):** parameterise the CA subject
per DC; widen or DC-parameterise the VIP gate so it reads the per-DC VIP from the merged
deploy input rather than `bundle.yaml`'s dc0 band; then generate dc1's own PKI. Note the
generator's VIP source interacts with the 2026-07-25 ruling-3 VIP extraction and with R11's
new `.61`/`.62` allocations -- the gate must read the MERGED input, not the base bundle, or
it will break again the moment the extraction lands.

**Roosevelt analog:** per-DC amphora CAs match per-DC IPMI/BMC credentials and per-DC MAAS
power keys (SEC-012/-016) -- the same "no cross-DC shared secret" principle already ruled
twice in this deployment.

## D-110: DC-DC numbering and repo-seed policy

**Status:** ADOPTED 2026-07-09 (operator ruling; meta / process).

**Decision:**
- The DC-DC decision series starts at D-100 in the repo `openstack-caracal-dc-dc`, seeded from `openstack-caracal-ipv4` at commit 8813efc (verified: new-repo HEAD equals 8813efc; `design-decisions.md` ceils at D-075).
- D-076 through D-099 are RESERVED as a source-project band and are NEVER assigned in this repo. (D-076 is claimed by the source-project dashboard workstream; 077-099 are runway.)
- A genuinely-shared future source-project decision is referenced by its original id (e.g. "see openstack-caracal-ipv4 D-0NN"), never copied and renumbered into this repo.
- `ledger-scan` is confirmed 3-digit-safe for the D-series (`grep -oE '^## D-[0-9]+' | sort -n`), so D-100+ next-free is correct. The deprecated `repo-lint` next-free path is band-limited above 099 and must not be used for next-free (standing rule).

**Rationale:** a round D-100 floor gives a clean project base plus about 24 numbers of cross-repo collision runway (the source is a maturing project unlikely to reach 100); reference-not-copy preserves provenance. A distinct prefix (DD-) was considered for permanent cross-repo immunity and declined for VR1 -- it would fork `ledger-scan`, the fence validator, and every D-NNN cross-reference, against minimize-delta.

**References:** process entry; governs all D-1NN entries in this repo.

**Log-for-sweep (not actioned):** `ledger-scan` DOCFIX / BUNDLEFIX patterns use `[0-9]{3}` -- safe through 999, latent at the 1000 boundary (the same class as the 100-boundary bug already fixed for those series).

---

## D-111: VR1 per-DC v6 subcarve aligned to the deployed NN mnemonic

**Status:** ADOPTED 2026-07-11 (operator deferred to Code stream's recommendation; ratifies `docs/dc-dc-netbox-buildout-scope.md` sub-decision #1).

**ANNOTATION 2026-07-30 (DOCFIX-205) -- DC LABELS BELOW ARE PRE-D-117.** Recorded per D-117's
ruled treatment ("ADOPTED decision text: ANNOTATE in place, do NOT rewrite"). D-117
(2026-07-13) retired the repo's `dc1`/`dc2` labels in favour of `dc0`/`dc1`; D-119 (2026-07-14)
region-qualified them. Read `dc1` -> `vr1-dc0` and `dc2` -> `vr1-dc1` throughout this entry --
including the ULA indexing line, which is UNCHANGED IN SUBSTANCE: the hextet tracks the GUA
site nibble (`f02::/48` -> `...:02xx`, `f03::/48` -> `...:03xx`), NOT the DC index. D-119 says
so explicitly and warns against "fixing" it to match the DC number, which would break the
already-imported sandbox prefixes.

**The "Out of scope" line below is now HISTORICAL and its recommendation was REJECTED.** It
says the G5 DC-site rename (`vr1-dc0`/`vr1-dc1` -> `vr1-dc1`/`vr1-dc2`) is "a distinct ruling,
recorded under its own D-number when executed." That ruling was taken -- it is D-117 -- and it
went the OTHER WAY: the apex is NOT renamed, the REPO adopted the apex's 0-indexed
`vr1-dc0`/`vr1-dc1`. No such rename is owed; a future session must not execute it.

**Decision:**
- `netbox/dc-dc-prefixes-import.py`'s per-plane v6 subcarve is aligned to the DEPLOYED VR0-DC0 / Willamette net-byte (NN) mnemonic, replacing the original contiguous-plane-index proposal (which was explicitly un-ratified). Each plane gets a `/60` container + `/64` active, mirroring the live template:
  - provider-public (GUA, out of `DC_GUA_PREFIX`): `/60` + `/64` active at `:10`, API-VIP `/64` at `:11`.
  - metal-admin + metal-internal (ULA) SHARE the metal `/60` at `:20` -- admin `/64` at `:20`, internal `/64` at `:21` -- mirroring how provider `:10` / VIP `:11` share the provider `/60` (the deployed model had a single `metal` plane; the D-052/D-053 six-plane split nests both legs as `/64`s in it).
  - data-tenant `:30`, storage `:40`, replication `:50` (ULA): each its own `/60` + `/64`.
- ULA legs are carved from a per-DC `/56` of the org ULA `/48`, indexed to the GUA site nibble (dc1 -> `...:02xx`, dc2 -> `...:03xx`) so the 4th hextet reads `DC.NN` in BOTH families.
- Addresses are computed by direct index arithmetic, NEVER `list(...subnets())` enumeration (preserves the DOCFIX-181 no-hang property).

**Rationale:** the operator's standing directive is to keep VR1 as close to the real deployment as possible. The live NetBox export (2026-07-11) shows VR0-DC0 / Willamette use exactly this NN mnemonic + `/60`-per-plane structure; aligning the tool means VR1's GUA and ULA both read `:20`=metal, `:30`=data, etc., consistent with the deployed clouds, instead of arbitrary contiguous indices.

**Out of scope (mirrored separately, not via this six-plane tool):** the `lbaas-mgmt` (`:80`), `vpn` (`:e0`), `oob` (`:f0`) networks from the deployed template. The G5 DC-site rename (`vr1-dc0`/`vr1-dc1` -> `vr1-dc1`/`vr1-dc2`) is a distinct ruling, recorded under its own D-number when executed.

**References:** `netbox/dc-dc-prefixes-import.py` (`carve_v6`), `docs/dc-dc-netbox-buildout-scope.md` section 5.1, D-101 (family matrix), D-052/D-053 (six-plane), the 2026-07-11 live NetBox export. Harness `tests/dc-dc-prefixes-import/` 53/53. Applies to the Office1 simulation NetBox first; flows back to `netbox.baldurkeep.com` as refined draft.

---

## D-112: OPNsense edge config delivery -- the Configuration Importer cannot work on a nano image

**Status:** ADOPTED 2026-07-12 -- **option (c), post-boot network provisioning.** Operator ruling:
"C since it is the way people automate opnsense" -- prefer the mainstream, industry-standard
automation path (the OPNsense REST API, as used by `opn-cli` and the Ansible OPNsense collection)
over a bespoke image-bake or media-import trick. Consistent with the standing "transferable answers
beat quick fixes" constraint: a real Roosevelt edge is provisioned over the network too.

IMPLEMENTATION NOT STARTED. The BOOTSTRAP sub-problem below is unsolved and is itself unruled --
"push the config over the API" is not reachable from a factory-default OPNsense without first
solving how we authenticate and how we reach it. Do not start building until B1/B2 are ruled.

**Bootstrap problem (measured/sourced, must be solved BEFORE any config push):**
1. **No route to it.** Factory-default OPNsense serves LAN on `192.168.1.1/24` on `office1-local`.
   `office1-local` is an ISOLATED libvirt network -- the vcloud host has NO leg on `virbr2`
   (measured: no IPv4 on the bridge). So nothing can currently talk to the edge's LAN at all.
   The WAN side is reachable at the DHCP lease `172.30.1.126`, but OPNsense **blocks all inbound on
   WAN by default** -- so the WAN leg is NOT a way in.
2. **No credentialed API.** The OPNsense REST API authenticates with an **API key + secret**, not the
   root password. Keys are minted in the GUI -- a chicken-and-egg for a fresh box. SSH is
   **DISABLED** in the factory default, so `ssh` is not an entry point either.
3. **Our serial console is write-only.** `modules/opnsense-edge` attaches `<serial type='file'>` --
   a one-way capture. It CANNOT be typed into, so the console is not currently an entry point
   either. Making it interactive means `type='pty'` + `virsh console` (a module change + recreate).

**Unruled sub-choices (operator, next session):**
- **B1 -- how do we REACH it?** (i) temporarily add a host leg on `virbr2`
  (`ip addr add 192.168.1.10/24`) -- reversible, non-persistent, cheapest; or (ii) give
  `office1-local` a permanent host address in the OpenTofu network definition (touches D-100's
  network design); or (iii) drive it entirely over an interactive serial console and never need IP
  reachability for bootstrap.
- **B2 -- how do we AUTHENTICATE the first time?** (i) interactive serial console (`type='pty'`) ->
  root login -> enable SSH + install the ed25519 pubkey we already generated in
  `~/vr1-office1-creds/`, after which everything is plain SSH/SCP and fully scriptable; or
  (ii) authenticated GUI session with the default root credentials (curl + CSRF token) to mint an
  API key, then use the REST API proper.
  NOTE: (i) is console-driven but is NOT the rejected option (d) -- (d) was rejected for racing the
  importer's 7-second timeout. A console *login shell* has no timing race and can be driven at
  leisure. These are materially different risks; do not conflate them.

**Option (a) RE-EXAMINED 2026-07-12 at operator request and MEASURED INFEASIBLE on this host.**
Recorded so it is not re-litigated. Baking `/conf/config.xml` into the nano requires writing
FreeBSD UFS2 from Linux, and every route is closed here:

| Requirement | Measured result |
|---|---|
| Kernel UFS write | `# CONFIG_UFS_FS_WRITE is not set` -- driver is READ-ONLY (`CONFIG_UFS_FS=m`) |
| A FAT partition to write instead | NONE -- MBR (`dos`), a single FreeBSD `a5` partition; UFS only |
| FUSE UFS driver (`fuse-ufs`, `ufs2-tools`) | not present, not packaged |
| libguestfs (`guestfish` / `virt-copy-in`) | not installed |

Two ways (a) could still be FORCED, both REJECTED:
- **FreeBSD helper VM** (boot a FreeBSD live system with the nano as a second disk; FreeBSD mounts
  UFS rw natively; drop the file; power off). It genuinely works and `xorriso` IS present, so a live
  ISO could be remastered with an auto-run script -- but it means fetching + remastering a FreeBSD
  ISO and maintaining a builder VM forever to deliver one 4.5 KB file. Cost exceeds (c) entirely.
- **Raw byte-patch of the image** (pad the new XML to the default `config.xml`'s exact byte length
  and overwrite its data blocks in place, leaving the inode untouched). Probably works -- and
  REJECTED anyway: it silently corrupts if the file is not block-contiguous, and it is
  NON-TRANSFERABLE (you cannot byte-patch a real appliance's disk on Roosevelt hardware), which is
  precisely what this repo's governing constraint forbids.

**Net: examining (a) properly STRENGTHENED (c).** (a) is not merely less mainstream here -- it is
unavailable without a FreeBSD-shaped detour costing more than (c)'s whole bootstrap.

**BOOTSTRAP RULED 2026-07-12 (operator: "proceed with the C bootstrap"):**
- **B1 (reach) = (i)** temporary host leg on `virbr2` (`ip addr add 192.168.1.10/24`) -- reversible,
  non-persistent, no change to the D-100 network definition. Removed/re-addressed once the edge's
  LAN moves to `10.10.0.1`.
- **B2 (authenticate) = (i)** one interactive console login -> enable SSH + install the ed25519
  pubkey ALREADY generated in `~/vr1-office1-creds/`. Thereafter everything is plain SSH/SCP and
  fully scriptable. Chosen over GUI/CSRF-scraping to mint an API key (brittle, no gain).

**Once bootstrapped, the steady-state mechanism is:** push `config.xml` (or the equivalent API
calls) to the edge, reload, and verify. The config we already render (`opnsense-config.xml.tmpl`,
LAN `10.10.0.1/24` + DHCP `.100-.199`, WAN static `172.30.1.2`) is REUSED unchanged -- only the
DELIVERY changes.

**Consequently RETIRED (do not delete until (c) actually works):** the config-ISO delivery path --
`scripts/opnsense-build-config-iso.sh` + its harness, and the module's `config_seed` volume + cdrom
disk. They are inert (the Importer never reads them), not harmful. Remove them in the same change
that lands (c), not before.

**Finding (MEASURED, not inferred).** The Office1 edge boots (DOCFIX-188 memory + DOCFIX-190 ACPI)
but comes up on FACTORY DEFAULTS. The config ISO is well-formed and correctly attached -- verified:
ISO9660 labelled `OPNSENSE_CFG`, containing `CONF/CONFIG.XML` with our `<opnsense>` payload and
`10.10.0.1`; the guest sees it as `cd0` (`QEMU QEMU DVD-ROM`, 186x2048 = exactly our 380928-byte
image). The boot log shows the importer being invoked and then doing NOTHING:

    >>> Invoking import script 'importer'
    (no further output)

**Root cause -- architectural, not a bug.** `rc.syshook.d/import/20-importer` runs
`/usr/local/sbin/opnsense-importer -b`. Upstream source (opnsense/core, `src/sbin/opnsense-importer`):

    INSTALL="/.probe.for.readonly"
    ...
    if [ -n "${DO_BOOT}" ]; then
            touch ${INSTALL} 2> /dev/null
            if [ -f ${INSTALL} -a -f /conf/config.xml ]; then
                    bootstrap_and_exit 0
            fi

The `touch` is a PROBE FOR A READ-ONLY ROOT. On INSTALLER media the root is read-only, the touch
fails, the marker is absent -- so the importer proceeds to scan attached media (and yes, its scan
path does support cd9660). On a PRE-INSTALLED image (our nano) the root is WRITABLE and a factory
`/conf/config.xml` ALREADY EXISTS, so BOTH conditions hold and it `bootstrap_and_exit 0`s
immediately -- it never enumerates a single device.

**Therefore the Configuration Importer can NEVER fire on a pre-installed nano image, by design.**
Nothing was ever going to read our ISO.

**What the prior research got right and wrong** (`opentofu/README.md` "OPNsense deployment
research", items 2-3): it correctly established that the Importer exists and that its scan path
supports ISO9660 for VM automation. It never established that the Importer would RUN AT ALL on a
nano image -- and it does not. The module header has always flagged this exact mechanism UNVERIFIED;
that flag was correct and is now discharged NEGATIVE.

**Options (operator to rule; none implemented):**

- **(a) Bake `config.xml` into the image before first boot.** `opnsense-prep-image.sh` writes
  `/conf/config.xml` directly into the nano's UFS root. Cleanest and most immutable-infra; no
  importer, no ISO, no boot-time interactivity. RISK/UNVERIFIED: writing FreeBSD UFS2 from Linux --
  most distro kernels ship `CONFIG_UFS_FS_WRITE` off, so this may be infeasible without libguestfs
  or a FreeBSD helper. FEASIBILITY MUST BE PROVEN BEFORE ADOPTING.
- **(b) Switch from the nano image to the installer image.** On installer media the root IS
  read-only, so the Importer runs and the existing ISO mechanism works AS DESIGNED and unchanged.
  Cost: an actual install step per edge; needs checking whether it is non-interactive.
- **(c) Boot factory-default, then push config over the network.** Provision post-boot via
  OPNsense's API/SSH from Office1 (factory default LAN is `192.168.1.1/24` on `office1-local`, which
  we own). Most Roosevelt-transferable -- real edges get provisioned over the network too -- but it
  is two-phase and needs a bootstrap path onto the default LAN.
- **(d) Serial-console expect-scripting.** Drive the interactive importer prompt over the serial
  console. NOT RECOMMENDED: fragile, timing-dependent (the prompt has a 7-second timeout).

**Impacts if not (b):** `scripts/opnsense-build-config-iso.sh`, its harness, the module's
`config_seed` volume + cdrom disk, and the README research section all assume the ISO mechanism.

**References:** `docs/archive/incident-20260712-opnsense-edge-boot-triplefault.md`,
`docs/archive/changelogs/changelog-20260712-libvirt-acpi-kernel-panic.md` (DOCFIX-190), upstream
`opnsense/core:src/sbin/opnsense-importer` + `src/etc/rc.syshook.d/import/20-importer`.
Also gates Stage 3 (the per-DC OPNsense edges use this same module).

---

## D-113: is OPNsense the right edge platform, or should the simulated edges be VyOS/Linux?

**Status:** **ADOPTED 2026-07-13 -- option (a2): STAY ON OPNSENSE, MOVE CONFIG TO THE REST API.**
Operator ruling ("go ahead and continue with your rest API recommendation"), after two
constraints were surfaced during the discussion that narrowed the field decisively:

1. **GUI access is a REQUIREMENT** (operator, 2026-07-13). This ELIMINATES option (b) VyOS and
   option (c) plain-Linux outright -- neither has a GUI. Both had been the leading candidates
   until this was stated; recording it so the reasoning is not re-run from the wrong premise.
2. **VyOS LTS binaries are subscription-gated** (verified 2026-07-13, vyos.net/get + the
   Software Access Subscription page). Free VyOS means the ROLLING release -- a moving target
   that fights this repo's appendix-B version-pinning discipline -- or building from source.
   Independent of the GUI point, this was real friction.

Remaining open-source field was OPNsense / pfSense CE / OpenWrt / IPFire. pfSense REJECTED
(same GUI-owned XML lineage -- a migration that buys nothing). IPFire: weakest automation
story. **OpenWrt was the credible alternative** -- its UCI config is text-first BY DESIGN
(LuCI and the CLI read/write the same `/etc/config/*` files, so editing config as text is a
supported path, structurally unlike OPNsense's GUI-owned `config.xml`) -- but it costs a new
image pipeline, a fresh bootstrap problem (it also defaults to LAN `192.168.1.1`), and dnsmasq
instead of Kea, all to escape a problem fixable in place.

**The ruling rests on this:** the bug class was never "OPNsense". It was hand-authoring
`config.xml`, a GUI-owned format, when OPNsense ships a documented REST API with mature Ansible
collections modelling exactly what we got wrong (`dhcp_subnet`, firewall rules, system settings
as typed first-class resources). **None of DOCFIX-191/192/193 is expressible through the API** --
you cannot forget to enable sshd in a format where sshd is a typed field with a default. (a2)
keeps the GUI, keeps a WORKING routing+DHCP edge, keeps the two libvirt fixes, and removes the
defect source -- and it is the smallest change of anything considered.

**AMENDMENT 2026-07-13 -- the template is DELETED, not reduced (operator ruling: "I'll defer
to your lean").** The original (a2) plan was to REDUCE `opnsense-config.xml.tmpl` to a minimal
bootstrap (sshd + root key + console + a seeded API key). That turned out to be unnecessary:
every one of those is covered without any `config.xml` at all.

- sshd + root key: the **D-112(c) console bootstrap** installs them (proven on Office1).
- the API key: **`scripts/opnsense-bootstrap-apikey.sh`** mints one by calling OPNsense's OWN
  model (`Auth\User -> apikeys->add()`) over SSH -- no GUI click, and no re-implementation of the
  vendor's `$6$` crypt format (which would fail SILENTLY on any format drift).
- everything else -- DHCP, firewall, interfaces: **the REST API**.

So the provisioning chain has NO `config.xml` in it, and a renderer that nobody should ever run is
not a safety net -- it is a loaded gun pointed at a live router. The 2026-07-13 safety sweep is the
evidence: the repo still contained runbook steps instructing an operator to render a config and push
it to the edge, which by then would have CLOBBERED live API-managed DHCP.

**DELETED:** `opentofu/templates/opnsense-config.xml.tmpl`, `scripts/opnsense-render-config.sh`,
`scripts/opnsense-build-config-iso.sh`, and both their harnesses. The `opnsense-edge` module's
`config_seed` volume + cdrom disk are removed (the ISO was INERT -- D-112). The
`xorriso`/`genisoimage` prereq is dropped. All remain in git history.

**If you think the API cannot express something:** that is a finding to raise against D-113 -- not
a reason to hand-write XML.

**Superseded by this ruling:** the "stay on (a1)" lean in the original recommendation below.
(a1) is now explicitly REJECTED: its cost compounds per feature and it depends on an
undocumented self-heal of an internal format (see the 667-element finding).

**Scope.** The VR1 edges are SIMULATED customer-site / ISP routers (Office1 today; Stage 3 adds
one per DC). They are lab infrastructure, not a product surface. Nothing a tenant touches
depends on this choice; what depends on it is our own automation cost, for three edges.

### First, an honest cost accounting of 2026-07-12 (five bugs)

| # | bug | OPNsense's fault? |
|---|---|---|
| DOCFIX-188 | guest got 2 MiB RAM (`memory_unit`) -- triple fault | **NO** -- libvirt module bug; would hit ANY guest |
| DOCFIX-190 | `acpi=off` -> FreeBSD kernel panic | **NO** -- libvirt module bug; generic |
| DOCFIX-191 | rendered config had no sshd + no key -> would have locked management out | yes (hand-authored `config.xml`) |
| DOCFIX-192 | rendered config silenced the only console | yes (hand-authored `config.xml`) |
| DOCFIX-193 | LAN DHCP was never implemented (and ISC `<dhcpd>` would have been inert -- 26.1 is Kea) | yes (hand-authored `config.xml`) |

Two of five were platform-agnostic libvirt bugs whose fixes now protect every VR1 VM
(MAAS/NetBox/GitBucket, the DC nodes). Finding them via OPNsense was lucky, not costly.

**The other three share ONE root cause: we hand-author the appliance's internal, GUI-owned
`config.xml`.** That is a known-bad idea in any product of this class -- and pfSense would be
identical, since it is the same format lineage.

**2026-07-13 adds a fourth data point of the same kind.** A full-config push REPLACES
`/conf/config.xml`; the rendered template is 128 elements against a live 796, so it DROPS ~667
migration-populated elements -- including the only two firewall pass rules on the box
("Default allow LAN to any"), against a pf base policy of block-drop-all. It works only because
OPNsense regenerates them on boot (proven across 4 push+boot cycles, `/conf/backup/` trail).
**We are depending on an undocumented self-heal of an internal format.** That is the ongoing
cost, and it is the thing to weigh -- not the boot bugs.

### THE CONFOUND (must be resolved before ruling)

**D-112 ADOPTED option (c) with the explicit rationale "C since it is the way people automate
opnsense" -- i.e. the OPNsense REST API (`opn-cli`, the Ansible OPNsense collection).**

**What we actually built is NOT that.** We render a full `config.xml` from a Jinja-ish template
and scp it over the wire. That is a config-file bake delivered post-boot -- it satisfies the
letter of (c) (provisioning happens over the network, after boot) but NOT the rationale the
ruling was made on.

**Every OPNsense-specific bug we are now charging against the platform (DOCFIX-191/192/193, and
the 667-element drop) is an artifact of hand-authoring XML -- exactly what the REST API exists
to avoid.** The API models DHCP, sshd, console, and firewall rules as first-class resources;
none of those three bugs is expressible through it.

So the fair comparison is not "OPNsense vs VyOS". It is:

- **(a1) OPNsense + `config.xml` templating** -- the status quo. Cheapest right now; every new
  feature (VPN, HA, VLANs) means reverse-engineering another slice of an internal format, and
  the self-heal dependency above is permanent.
- **(a2) OPNsense + the REST API** -- what D-112's rationale actually called for. Config becomes
  declarative API calls; the internal format stops being our problem. Cost: retire the template,
  build/adopt an API client (`opn-cli` is packaged and mainstream), and keep only a MINIMAL
  bootstrap config (sshd + key + console) as a file. The boot bugs stay fixed either way.
- **(b) VyOS** -- purpose-built for this. Plain declarative text config, first-class cloud-init,
  a real config API; it exists to be a lab/edge router. Friction: LTS images are
  subscription-gated (rolling is free), it is less "appliance-like" than a real site router, and
  switching discards a WORKING, routing, DHCP-serving edge to re-run discovery on a new platform.
- **(c) Plain Linux router** (Debian/Ubuntu + nftables + Kea + FRR, provisioned by cloud-init).
  Maximum fidelity to the rest of our tooling -- same image pipeline, same cloud-init, same
  Ansible/OpenTofu idioms as every other VR1 VM, and zero appliance-internal formats. Friction:
  we hand-build router behaviour that OPNsense/VyOS ship; least faithful to "a real site runs an
  appliance".
- **(d) pfSense** -- **REJECT.** Same config-format lineage as OPNsense; buys nothing, costs a
  migration.

### Recommendation (mine; the operator rules)

**Stay on OPNsense -- but the real question is (a1) vs (a2), and I lean (a2).**

Rationale for staying: the hard part is DONE and MEASURED WORKING (boots, routes, NATs, reaches
the internet, serves DHCP, managed over SSH with a verified push pipeline). Edges 2 and 3 reuse
the module and template -- different tokens, same everything -- so the cost paid was one-time,
not per-edge. Switching platforms discards that and re-runs the same class of discovery
elsewhere. VyOS is the defensible blank-sheet pick, but we are not on a blank sheet.

Rationale for (a2) over (a1): (a1)'s cost COMPOUNDS. Each new feature is another slice of
reverse-engineered XML plus a self-heal we do not control. (a2) pays a bounded, one-time cost to
get onto the path D-112's ruling actually named, and it is the honest reading of that ruling.

**What would change my mind entirely (i.e. rule (b)/(c)):** if, having moved to (a2), the API
still forces us to reverse-engineer the config format for each new feature -- then OPNsense is
not buying us the appliance abstraction we are paying for, and VyOS's plain-text config wins.

### Cost of deferring

Low but not zero: Stage 3 builds two more edges from this same module. Ruling BEFORE Stage 3
means the choice is made once; ruling after means migrating three edges instead of one.

**Evidence:** `docs/archive/changelogs/changelog-20260712-office1-opnsense-edge-build.md`,
`docs/archive/changelogs/changelog-20260712-opnsense-edge-boot-fixes.md`,
`docs/archive/changelogs/changelog-20260713-office1-dhcp-apply.md` (the 667-element self-heal, measured),
D-112 (the (c) ruling and its stated rationale).

### D-113 -- AMENDMENT (2026-07-14): "interfaces: the REST API" IS FALSE. Base-interface addressing has NO API.

A factual correction to this decision's own text, not a re-ruling. The (a2) rationale above ends
with: *"everything else -- DHCP, firewall, **interfaces**: the REST API."* **The `interfaces` half
of that claim is wrong**, and it was never measured before being written -- it was inferred from
DHCP and firewall being API-native.

**MEASURED on the live Office1 edge (2026-07-14), OPNsense 26.1:**

| Surface | Served by | API? |
|---|---|---|
| Interfaces > [LAN] / [WAN] -- **address, incl. `ipaddrv6`** | `/interfaces.php?if=lan` | **NO -- legacy PHP page, config.xml-backed** |
| Interfaces > Devices (bridge/gif/gre/lagg/loopback/vlan/vxlan) | `/ui/interfaces/*` | yes |
| Interfaces > Virtual IPs | `/api/interfaces/vip_settings/*` | yes (`ipalias`/`carp`/`proxyarp`; iface `wan`/`lan`/`lo0`) |
| Services > Router Advertisements (radvd) | `/api/radvd/settings/*` | yes (`getEntry`/`addEntry`/`searchEntry`) |
| Services > Kea DHCPv4 / DHCPv6 | `/api/kea/*` | yes |

Only the **Devices** and **Neighbors** sub-trees were migrated to MVC. A base interface's IP
addressing was not. So there is no API call that sets LAN's `ipaddrv6`.

**What this does and does NOT change:**

- **It does NOT reopen (a2).** The prohibition D-113 actually adopted is on **hand-authored
  `config.xml` and full rendered pushes** -- the clobber path that would wipe the API-managed DHCP.
  D-113 also records **"GUI access is a REQUIREMENT"** as the constraint that eliminated VyOS and
  plain-Linux. A one-time operator GUI edit is therefore the vendor interface this decision
  deliberately PRESERVED. It is not a violation and must not be described as one.
- **It DOES mean the provisioning chain is not 100% API-scriptable** for any surface that needs a
  base-interface address -- which is every edge that gets IPv6, i.e. Office1 now and the DC0/DC1
  edges at Stage 3. That is a per-edge manual step unless the VIP path below works.

**The open question (an EXPERIMENT, not a plan):** the `radvd` entry model has **no prefix field**
-- it derives the advertised prefix from the interface's configured IPv6. Whether an **`ipalias`
VIP** counts as "the interface's configured IPv6" for OPNsense's radvd config generator, or whether
the generator only reads the primary `ipaddrv6`, is **UNKNOWN and must not be assumed**. If the VIP
path works, the whole IPv6 edge build is API-native and reproducible per DC. If it does not, the
LAN GUA is an operator GUI step at every edge. This is cheap and reversible to test on the lab edge
and is gated on an operator ruling.

---

## D-114: VR1 site containment VMs + MAAS-composed LXD VMs for the non-stack machines

**Status:** **ADOPTED 2026-07-13 (operator ruling).** AMENDS D-103 (which assigned Office1 three
sibling service VMs and put every VR1 domain directly on the vcloud host). Supersedes nothing else.

**AMENDED 2026-07-16 (D-123 Model B):** the containment-VM scope is EXTENDED for the DCs --
`vvr1-dc0`/`vvr1-dc1` now ALSO hold the OpenStack node VMs (this entry previously scoped them to the
MAAS/LXD headend + the non-stack LXD-composed machines ONLY, NOT the OpenStack nodes). This makes a
single-object `virsh destroy` a true site-down primitive for DCs, matching Office1. Consequence: the DC
containment VM is resized to hold the node fleet (~416 GiB, derived) with `expose_nested_virt = true` (D-124
rack sizing void). Office1's `voffice1` scope is unchanged.

**Context.** Stage 2 was about to build Office1 per D-103: three peer VMs (MAAS-region, NetBox,
GitBucket) created directly on vcloud libvirt, with Tailscale installed on the headend. The
operator described a different shape -- each site (Office1, DC1, DC2) living inside its OWN VM
environment, with MAAS running LXD to hold "the rest of the servers/services." An initial reading
of `bundle.yaml` found only Juju's `lxd:8`-style CONTAINER placements and concluded there was no
precedent for MAAS composing LXD VMs. **That conclusion was WRONG**, and is recorded here because
the error is instructive: the bundle's Juju LXD containers and MAAS's LXD VM host are TWO DIFFERENT
LXD usages in the same cloud, and reading only the bundle collapses them into one.

**Measured as-built evidence (VR0 / `openstack-caracal-ipv4` MAAS 3.7.2, operator screenshots
2026-07-13).** Seven machines, all Deployed, ALL powered via Virsh:

- `openstack0-3` -- 8 cores / 32 GiB / 1.1 TB, 6 spaces, tags `openstack, virtual`. Juju machines
  8-11; the OpenStack API services run in JUJU-created LXD CONTAINERS on these (21 `lxd:N`
  placements in `bundle.yaml`). MAAS never sees those containers, and never needed to.
- `juju` -- 2 cores / 4 GiB, tags `juju, virtual`. The controller.
- `lxd` -- 8 cores / 16 GiB / 687 GB, tags `lxd, virtual`. A MAAS-DEPLOYED, virsh-powered libvirt
  VM that is then REGISTERED BACK INTO MAAS as an LXD KVM host (project `default`, LXD **5.21.4**;
  host NICs enp1s0 / enp7s0 / lxdbr0).
- `tailscale` -- 2 cores / 2 GiB / 25 GB, Ubuntu 24.04, tags `pod-console-logging, virtual`.
  **It is a VM COMPOSED BY MAAS INSIDE the `lxd` KVM host.** This is the proof of the pattern.

So the pattern the operator asked for is not new -- it is VR0's own, already proven in production
on this cloud, and it costs 2 cores / 2 GiB per service machine.

### Ruling

1. **Non-stack machines become MAAS-composed LXD VMs.** Per site, one MAAS-deployed machine runs
   LXD and is registered back into MAAS as an LXD KVM host; the machines that live OUTSIDE the
   OpenStack stack (NetBox, GitBucket, Tailscale, and any future service box) are COMPOSED into it
   by MAAS. They are therefore MAAS-visible: enlisted, commissioned, deployed, powered, released.
   This is what the operator meant by visibility, and it is `lxd` + `tailscale` reproduced per site.
   **Explicitly NOT in scope:** the Juju-deployed OpenStack services on the DC nodes. Those stay in
   Juju-created LXD CONTAINERS (bundle `lxd:N` placements), invisible to MAAS, exactly as in VR0.
   Operator: "I don't care about visibility into the juju deployments into the openstack0-3
   machines as much as the other machines running outside of the stack."

2. **Each site gets a containment VM** -- `voffice1`, `vdc1`, `vdc2` -- simulating the facility that
   the site's servers and services operate within. Rationale (operator): **full-facility snapshots**
   and **the ability to simulate total geographical failure of a facility.** The second is the
   decisive one: D-108's DR failover drill currently has NO honest "hard-down an entire DC"
   primitive, and `virsh destroy vdc1` IS that primitive. This is a capability the flat model
   cannot offer at any price.

3. **Sequencing (operator ruling):** Office1 FIRST, fully up and running, on this model. DC1 is
   **GATED** behind it and is not started until Office1 is complete. DC2 follows DC1.

4. **LXD is version-pinned to the 5.21 LTS track.** MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7:
   LXD 6.7 consolidated API endpoints and MAAS's pinned pylxd 2.3.5 cannot speak to them (fixed in
   pylxd >= 2.3.9, not yet in a MAAS release). Canonical's own guidance is "use LXD <= 6.6 or the
   5.21 LTS release until further notice." VR0 is already on 5.21.4, i.e. on the safe side by
   accident of timing rather than by decision -- this makes it a decision. Goes in appendix-B.
   Source: https://discourse.maas.io/t/maas-incompatibility-with-lxd-6-7/15749

### What this changes in D-103

D-103 said OpenTofu owns creation of "the Office1 service VMs (MAAS-region, NetBox, GitBucket)" as
domains on the vcloud host. Under D-114, OpenTofu owns creation of the SITE CONTAINMENT VMs and the
site's networks; INSIDE a site, the service machines are composed by MAAS into LXD. D-103's core
principle is UNCHANGED and is in fact extended: MAAS still owns the provisioning lifecycle, and
still does not compose the OpenStack NODE VMs (`modules/node-vm` pre-creates those; `maas_vm_host`
registers the virsh host so MAAS DISCOVERS them -- `maas_vm_host_machine` remains ruled out for
nodes). The composition right granted here is scoped to the LXD VM host and the non-stack machines.

### Known risk, and its entry gate

**Nesting depth.** `vcloud` is ITSELF a KVM guest (measured: `systemd-detect-virt` -> `kvm`). A site
containment VM therefore puts the site's servers one level deeper than VR0 runs them: DC node VMs
land at L3 and OpenStack TENANT INSTANCES at L4. VR0 proves nesting works to the depth IT uses
(`tailscale` inside `lxd`), but VR0's hosts sit beside vcloud on the same 10.17.11.0/22 lab network
and are thus one level SHALLOWER than a VR1 pod. L4 tenant instances are UNPROVEN, and nova-compute
needs working KVM at the bottom of that stack. The 2026-07-12 OPNsense BTX triple-fault is prior
evidence that this environment is already fragile at depth.

**Office1 IS the nesting probe -- and that is a feature of the sequencing, not a coincidence.**
(An earlier draft of this entry claimed "Office1 needs no nested KVM." That was WRONG and is
corrected here: LXD **virtual machines** are qemu/KVM guests, so composing NetBox/GitBucket/
Tailscale as LXD VMs inside `voffice1` requires nested KVM at exactly the same depth the DC nodes
would need it -- L3. Only LXD *containers* would be free, and MAAS does not enlist containers.)
Office1 is therefore the CHEAP place to prove the model: no OpenStack, no Ceph, small VMs. If an
LXD VM boots inside `voffice1`, the pod model is proven to L3 and DC1 inherits that proof.

**Entry gate for DC1:** (a) Office1 fully up on this model (operator ruling: DC1 is gated behind
it); (b) inside `vdc1`, confirm nested KVM is exposed and a guest actually BOOTS -- which also
proves L4 is reachable for tenant instances, the one depth VR0 has never exercised. If L3 fails at
Office1, the containment-VM model is abandoned early having cost one VM. If L3 holds but L4 fails
at DC1, DC compute stays flat and Office1 keeps its containment VM. Both outcomes are scoped
retreats, not redesigns -- the ruling is deliberately written to make them cheap.

**DHCP.** OPNsense/Kea is authoritative on `office1-local` (10.10.0.0/24, pool .100-.199). LXD's
`lxdbr0` runs its OWN dnsmasq -- that is FINE as long as it stays on LXD's internal NAT bridge and
is never bridged onto `office1-local`. Two DHCP servers on one L2 is an intermittent failure that
is genuinely unpleasant to diagnose. Any LXD networking change must not put `lxdbr0` on the site LAN.
Corollary bonus: `voffice1` booting on `office1-local` will take the FIRST REAL DHCP LEASE from Kea
-- a path proven only at the daemon level so far, never end to end.

### Delta to Roosevelt

The containment VM is simulation-only machinery: in Roosevelt a "facility" is a building full of
real servers, not a VM, and it is torn down by pulling power, not `virsh destroy`. That is an
ACCEPTED delta, bought deliberately: it is what makes a geographic-failure drill executable at all
in a virtual rehearsal. The LXD half carries NO delta -- it is exactly what VR0 runs today and what
Roosevelt would run on real hardware (a service host running LXD, registered to MAAS).

> **D-119 NAMING AMENDMENT (2026-07-14).** The DC containment VMs named `vdc1`/`vdc2` above are
> **RENAMED `vvr1-dc0` / `vvr1-dc1`** to match the region-qualified DC namespace. Neither is built
> yet (only `voffice1` exists), so this costs nothing -- and it removes a genuinely dangerous
> ambiguity: this decision's own DR primitive is **`virsh destroy vdc1`**, which under the old
> naming would read as "destroy VR1 DC1" while actually meaning "destroy VR1 DC0". A mislabelled
> destroy command in a DR drill is not cosmetic. `voffice1` KEEPS its name (the office keeps its
> number, per D-119). The `vdc1`/`vdc2` spellings in the body above are HISTORICAL.

## D-115: VR1 Office1 addressing -- the office carve, and the v4 role gaps it exposed

**Status:** **ADOPTED 2026-07-13 (operator ruling -- see the amendment at the end of this entry).** (Raised and ruled the same day; the earlier draft state is described in the amendment, kept off this Status line so `ledger-scan` reads the ADOPTED token cleanly -- review R3-F09.) Found during Office1 deployment
testing: the headend was built on address space that NetBox never assigned. Per the operator, the
IPAM carve matters MORE than the deployment already standing on it -- "we can always change the
current IP addressing on the deployment; getting the IPAM carve proper and correct is more
important than going back and correcting deployment work." Nothing here is blocked; Office1 runs.

**ANNOTATION 2026-07-30 (DOCFIX-205) -- DC LABELS BELOW ARE PRE-D-117, BUT THE OFFICE IS NOT.**
Recorded per D-117's ruled treatment ("ADOPTED decision text: ANNOTATE in place, do NOT
rewrite"). This entry was ruled 2026-07-13, the SAME DAY as D-117 and before its rename landed.
Read the DC labels as `DC1` -> `vr1-dc0` and `DC2` -> `vr1-dc1` throughout (so "DC2 moves INSIDE
the Cloud /16: 10.12.64.0/19" is `vr1-dc1`'s supernet, which is what was built), and note that
in the `X03::/48` table row "VR0 DC1" means VR0's SECOND DC -- a different region, correctly
labelled, and NOT subject to this remap.

**The OFFICE is explicitly OUT of D-117's scope and keeps its number.** D-117's operator
refinement, verbatim: "go ahead and rename the DC numbers. The office number is fine." So
Office1 stays Office1 -- NetBox site `vr1-off1`, and `voffice1` / `office1-local` /
`office1-netbox` / `~/vr1-office1-creds` all stand. Nothing office-side in this entry is
renamed, and the accepted asymmetry with VR0's `vr0-off0` is documented, not a defect.
This is about ratifying or replacing what it runs on, BEFORE NetBox/GitBucket land on it.

### What the IPAM apex actually encodes (MEASURED 2026-07-13 against netbox.baldurkeep.com)

**IPv6 is a complete, consistent, region-hierarchical model.** `2602:f3e2::/36` (ARIN, Abysius Inc).
Every REGION gets a `/40`; inside it, SITES are `/48`s on a FIXED slot pattern:

| slot | meaning | VR0 (`e00::/40`) | VR1 (`f00::/40`) | Eugene (`100::/40`) |
|---|---|---|---|---|
| `X00::/48` | Infrastructure / site-to-site links | `e00::` | `f00::` | `100::` |
| `X01::/48` | **Offices** | `e01::` | **`f01::`** | `101::` |
| `X02::/48` | Datacenter (first) | `e02::` (VR0 DC0) | `f02::` | `102::` Willamette |
| `X03::/48` | Datacenter (second) | `e03::` (VR0 DC1) | `f03::` | `103::` Roosevelt |

Inside an office `/48`: a reserved `/56`, then ONE `/56` PER OFFICE, then a `/64` office subnet.
VR0 Off0 = `e01:100::/56` -> `e01:100::/64`. Eugene's Charnelton = `101:100::/56`. Inside a DC `/48`:
the D-111 NN mnemonic (`:10` provider, `:11` provider API VIP, `:20` metal, `:30` data, `:40`
storage, `:50` replication, `:80` lbaas-mgmt, `:e0` VPN, `:f0` OOB), each `/60` + `/64`.
Also global: `2602:f3e2::/40` inter-regional infrastructure, `2602:f3e2:fe::/48` **Edge Networks**,
`2602:f3e2:ff::/48` OpenStack tenant reservation.

**IPv6 for Office1 therefore has NO design freedom** -- the pattern determines it:
`2602:f3e2:f01:100::/56`, office subnet `2602:f3e2:f01:100::/64`. VR1's office `/48` already exists.

**IPv4 is a much thinner, ROLE-based model** -- and this is where the gaps are:
`10.0.0.0/8` "Corp Private" has exactly three children: `10.12.0.0/16` **Cloud**, `10.16.0.0/16`
Datacenter (Willamette dev), `10.17.0.0/16` Datacenter (Roosevelt dev -- the physical lab net that
vcloud and the VR0 jumphost actually sit on). Plus `172.16.0.0/12` "Corp OOB Private".

### The gaps (all four are real; none was known before today)

1. **There is NO Office role in v4.** Office1's LAN `10.10.0.0/24` (chosen during the OPNsense edge
   build, 2026-07-12) and its LXD compose network `10.10.1.0/24` (chosen 2026-07-13) are SQUAT --
   unallocated, unregistered, colliding with nothing but blessed by nothing.
2. **DC2's planned `10.13.0.0/19` is OUTSIDE the Cloud `/16`.** `docs/dc-dc-netbox-buildout-scope.md`
   recommends it, but NetBox's Cloud role is `10.12.0.0/16`; `10.13.x` is unallocated. DC2 was
   already set to squat, and nobody had noticed.
3. **`office1-wan` (`172.30.1.0/24`) squats INSIDE `172.16.0.0/12` "Corp OOB Private"** -- and it is
   not OOB, it is a SIMULATED ISP UPLINK. That is a wrong-role problem, not merely an unregistered one.
4. **"Edge Networks" has a v6 `/48` (`2602:f3e2:fe::/48`) but NO v4 counterpart**, so the simulated
   ISP/WAN segments have nowhere legitimate to live in v4 at all.

(Also, still open and pre-existing: NetBox holds ZERO aggregates -- gap G2 -- so nothing declares
these allocations at the RIR level. And `2602:f3e2:f01:100::/56` needs a `VR1 Off1` SITE, which
does not exist in NetBox either.)

### The design tension the operator must rule on

**v6 here is REGION-hierarchical; v4 is ROLE-based.** That asymmetry is defensible and, I think,
correct: v6 is abundant, so it can afford geography; v4 is scarce, so it is pooled by function.
The alternative -- forcing v4 to mirror the v6 region tree -- burns scarce v4 AND requires
RENUMBERING DC1 off `10.12.x`, which is exactly the inheritance D-101 bought in order to reuse
VR0 DC0's bundle unchanged (minimize delta to Roosevelt). That is a large cost for conceptual tidiness.

### Options

**(a) RECOMMENDED -- keep v4 role-based; fill in the MISSING roles. Ratify the deployed addresses.**
- New v4 role **Office**: `10.10.0.0/16`, carved **`/22` per office**.
  VR1 Office1 = **`10.10.0.0/22`** -> LAN `10.10.0.0/24` (as deployed) + LXD compose `10.10.1.0/24`
  (as deployed) + `.2`/`.3` spare. **This BLESSES exactly what is already running -- zero renumber.**
- New v4 role **Edge** (mirroring v6's `2602:f3e2:fe::/48`): `172.30.0.0/16` for simulated ISP/WAN
  segments, carved out of the `172.16.0.0/12` container but with its OWN role, not OOB.
  Office1's WAN `172.30.1.0/24` (as deployed) becomes legitimate; DC1/DC2 edges take `.2`/`.3`.
- **DC2 moves INSIDE the Cloud `/16`**: `10.12.64.0/19` (free -- DC1's six `/22`s occupy
  `10.12.4/8/12/16/32/36`, so `10.12.64.0/19` collides with nothing and DC1<->DC2 stay routable
  over the dark fiber). SUPERSEDES the scope doc's `10.13.0.0/19` recommendation.
- IPv6 per the pattern: Office1 = `2602:f3e2:f01:100::/56` + `/64` subnet; create the `VR1 Off1` site.
- COST: two NetBox roles + a handful of prefixes. **No deployment renumber at all.**

**(b) Mirror the v6 region model in v4.** Give each virtual region a v4 `/16` (e.g. VR1 = `10.15.0.0/16`)
and carve sites inside it (infra / office / DC1 / DC2). Conceptually cleanest; the two families would
read identically. COST: RENUMBERS DC1's six planes off `10.12.x`, which BREAKS D-101's
inherit-DC0-unchanged economy and forces edits to `lib-net.sh`, MAAS subnets, and the bundle's VIPs.
It also spends a whole `/16` per virtual region on a test environment. Recommended ONLY if the
operator wants v4 and v6 structurally identical and accepts the DC renumber.

**(c) Ratify the squat as-is and register it verbatim.** Record `10.10.0.0/24`, `10.10.1.0/24`,
`172.30.1.0/24`, `10.13.0.0/19` in NetBox exactly as deployed, with no new roles and no structure.
Cheapest today, and WRONG: it enshrines an ISP-sim network inside the corp OOB block, leaves offices
with no allocatable block for the next office, and leaves DC2 outside the Cloud role. Rejected
unless the operator wants pure speed.

### What is NOT at stake

Nothing is blocked and nothing is broken. Office1 is live and routing on the squat addresses. The
question is only what to write into the IPAM apex -- and the renumber cost is LOWEST RIGHT NOW,
before NetBox and GitBucket are deployed onto that network. Under option (a) the renumber cost is
ZERO, because (a) is shaped to bless what is already there.

**References:** measured 2026-07-13 against `netbox.baldurkeep.com` (read-only; SEC-006 token).
D-101 (family matrix + DC1-inherits-DC0), D-111 (NN mnemonic subcarve), D-016 (tenant pool
`10.20.0.0/16` -- the reason the obvious `10.20.x` was NOT used for the compose net),
`docs/dc-dc-netbox-buildout-scope.md` (whose `DC2_V4_SUPERNET = 10.13.0.0/19` recommendation this
decision would supersede under option (a)), D-114 (which built the headend that exposed all of this).

### D-115 -- AMENDMENT (2026-07-13): ADOPTED by operator ruling, with a measured constraint attached

**Operator ruled all three questions:**
1. **v4 stays ROLE-BASED; fill in the missing roles.** Option (a). v6 keeps its region hierarchy;
   v4 is pooled by function. No DC renumber; D-101's inherit-DC0-unchanged economy is preserved.
2. **`/22` per office.** VR1 Office1 = **`10.10.0.0/22`** -- which BLESSES the deployed LAN
   (`10.10.0.0/24`) and LXD compose net (`10.10.1.0/24`) exactly. Zero deployment renumber.
3. **Register AND deploy dual-stack on Office1 NOW** -- Office1 becomes the first real dual-stack
   proof and de-risks D-101's family matrix early.

**THE ADOPTED CARVE**

v4 (new roles marked):
```
10.0.0.0/8   Corp Private
 |- 10.10.0.0/16   Office        <-- NEW ROLE
 |    |- 10.10.0.0/22    VR1 Office1
 |         |- 10.10.0.0/24   office1-local (site LAN; Kea on the edge)   [as deployed]
 |         |- 10.10.1.0/24   lxdbr0 compose net (MAAS DHCP/PXE)          [as deployed]
 |         |- 10.10.2.0/24, 10.10.3.0/24  spare
 |- 10.12.0.0/16   Cloud
 |    |- 10.12.{4,8,12,16,32,36}.0/22   DC1 planes    [unchanged -- D-101 inheritance]
 |    |- 10.12.64.0/19                  DC2           <-- MOVED INSIDE the Cloud /16
 |- 10.16.0.0/16   Datacenter (Willamette dev)
 |- 10.17.0.0/16   Datacenter (Roosevelt dev -- the physical lab net)
172.16.0.0/12  Corp
 |- 172.30.0.0/16  Edge (simulated ISP uplinks)       <-- NEW ROLE, mirrors v6's fe::/48
      |- 172.30.1.0/24   office1-wan                   [as deployed]
```
**This SUPERSEDES `docs/dc-dc-netbox-buildout-scope.md`'s `DC2_V4_SUPERNET = 10.13.0.0/19`**, which
sat outside every allocated block. DC2 becomes `10.12.64.0/19` -- free (DC1 occupies
`10.12.4/8/12/16/32/36`), inside the Cloud role, and still routable to DC1 over the dark fiber.

v6 (determined by the existing pattern -- no design freedom):
```
2602:f3e2:f01::/48        VR1 Offices                        [exists]
 |- 2602:f3e2:f01:100::/56   VR1 Office1                     <-- CREATE (+ the "VR1 Off1" site)
      |- 2602:f3e2:f01:100::/64   Office1 office subnet      <-- CREATE
```

### MEASURED CONSTRAINT (2026-07-13) -- IPv6 DOES NOT EGRESS THE LAB

Discovered while planning ruling 3. On vcloud:
- Global v6 addresses present (`2602:f3e2:e00:100::/64` SLAAC + a `/128`), and a v6 default route
  via RA.
- The v6 default gateway (link-local `fe80::e68d:8cff:fe14:7cd5`) **IS reachable** -- RA and L2 work.
- **But v6 INTERNET is UNREACHABLE**: both `2606:4700:4700::1111` and `2001:4860:4860::8888` fail,
  100% loss. v4 egress from the same host is fine.

**Consequence, stated plainly so nobody builds on a false premise:** deploying dual-stack on Office1
proves v6 ADDRESSING, RA, and services binding v6 -- which is most of D-101's family matrix and is
worth doing. It does **NOT** prove v6 INTERNET reachability, and it cannot: the GUA is not routed
past the lab boundary. **Public-facing GUA reachability (public API endpoints on v6, tenant GUA
egress) is UNPROVABLE in VR1 until the lab's upstream actually carries v6.** That is a network
issue OUTSIDE this cloud and outside this repo -- it needs the lab/upstream operator, not a change
here. Do not schedule a VR1 test that depends on it, and do not let a green internal-v6 result be
read as "v6 works end to end".

`office1-wan` currently has **no v6 at all** (v4 NAT only), and `office1-local` is an addressless
isolated bridge (correct -- the OPNsense edge owns addressing on it).

### NetBox write path -- NOT done unilaterally

The carve above is recorded HERE (the repo) as the decision of record. Writing it into
`netbox.baldurkeep.com` is a MUTATION ON A PRODUCTION SYSTEM and is gated on explicit operator
approval; the standing architecture is read=source / write=feedback-only, with the Office1 sandbox
NetBox as the place changes are simulated first. The Office1 NetBox VM is composed and `Ready` but
not yet deployed -- so the natural sequence is: deploy it, import this carve there, verify, then
feed back to the real NetBox.

## D-116: no Office1-local GitBucket -- keep using the existing git.baldurkeep.com

**Status:** **ADOPTED 2026-07-13 (operator ruling).** SUPERSEDES the GitBucket half of D-103 and
removes GitBucket from Stage 2's build list.

**Operator ruling, verbatim in substance:** "For the gitbucket, we've decided not to build a
separate one on the test cloud. We will continue to use the current setup and create new repos for
deployment tasks as needed."

**What changes.** D-103 assigned Office1 three service VMs -- MAAS-region, NetBox, GitBucket -- and
D-114 recast them as MAAS-composed LXD VMs. **GitBucket is now OUT OF SCOPE entirely**: no third
service machine is composed, and the existing `git.baldurkeep.com` remains the git service of record
for VR1. New repos are created there as deployment tasks require.

**What this closes.** The Stage 2 runbook carried an explicit OPEN QUESTION about the Office1-local
GitBucket -- "what it mirrors, and under what repo path, is not decided" -- because a *second*,
Office1-local GitBucket alongside the pre-existing `git.baldurkeep.com` raised a real
mirror/authority question nobody had answered. **That question is now MOOT.** There is one git
service, it is the existing one, and it is authoritative. This is a genuine simplification, not a
deferral.

**Rationale.** The Office1-local GitBucket only ever existed to rehearse a per-site git mirror. The
operator judges the rehearsal not worth a machine: the real deployment work is served by the
existing instance, and a second one buys a mirroring problem (which repo is authoritative, what
syncs, in which direction) in exchange for no capability the deployment actually needs.

**Delta to Roosevelt.** Roosevelt may still want a per-site git mirror for node-airgap reasons
(cf. D-107's per-DC artifact mirror, which is a SEPARATE concern and is NOT affected by this
ruling -- D-107 is about OS/package artifacts for node provisioning, not about hosting git repos).
If Roosevelt later needs a site-local git service, it is a fresh decision made against real
requirements, not a rehearsal artifact carried forward. Recording that explicitly so nobody reads
this ruling as "Roosevelt will never have a git mirror".

**Consequences (all applied in the same change):** GitBucket is removed from Stage 2's Build/Gate in
`docs/dc-dc-deployment-workflow.md`, from the Office1 standup runbook's sequence and its open
questions, and from `docs/vr1-office1-as-built.md`. Office1's composed service machines are
therefore exactly TWO: `office1-netbox` and `office1-tailscale`, both LIVE.

---

## D-117: VR1 site naming -- the apex says DC0/DC1, the repo says DC1/DC2 (G5)

**Status:** **ADOPTED 2026-07-13 (operator ruling: Option B, DC-only). EXECUTED IN TWO HALVES:
the SELECTOR half by D-119 (2026-07-14) -- the "PARTIALLY EXECUTED" caveat below is DISCHARGED
FOR THAT HALF; read D-119 for the end state -- and the DECISION-TEXT ANNOTATION half not until
DOCFIX-205 (2026-07-30). See the execution correction immediately below.**

**EXECUTION CORRECTION 2026-07-30 (DOCFIX-205; GA-R1/C2 -- measurement corrects the document).**
This Status line previously read "FULLY EXECUTED BY D-119 (2026-07-14) -- the 'PARTIALLY
EXECUTED' caveat below is DISCHARGED". **That overstated what D-119 did, and the overstatement
is the mechanism by which the un-done half survived for 17 days.** D-119 scopes its own
discharge precisely: it "executes D-117's own amendment (region-qualify the shell SELECTORS)
across the three surfaces D-117 never touched" -- `lib-net.sh`, `lib-hosts.sh`, `opentofu/`.
It never claimed, and never performed, this decision's OTHER ruled treatment: the
"ADOPTED decision text (D-101, D-106, D-111, D-115) -- ANNOTATE in place" row of the treatment
table below.

MEASURED 2026-07-30: **zero of those four entries carried any D-117 annotation.** Because this
Status line said "fully executed", a reader checking whether the annotation was owed was told
it was already done and stopped looking. The consequence was concrete -- a Stage-5 session read
un-annotated D-106 as live, re-derived the retired `dc1`-vs-`dc2` question, and filed it as a
blocking ruling on a premise this decision had retired on 2026-07-13. All four entries are
annotated as of DOCFIX-205; this correction records why they were not.

**Standing lesson (the reusable half): a ruling's Status line is a CLAIM ABOUT EXECUTION, not
evidence of it.** Where a decision rules several DIFFERENT treatments across surface kinds, a
later decision discharging ONE of them does not discharge the rest -- and no gate reads prose,
so nothing in this repo can catch a Status line that says otherwise. Check the artifact, not
the decision.
The REPO moves to the apex's 0-indexed convention **for the DATACENTERS**. NetBox is the IPAM apex
and is NOT bent to fit the repo. Supersedes the `docs/dc-dc-netbox-buildout-scope.md` G5
recommendation (which proposed the opposite, Option A) and the D-106 `dc1`/`dc2` zone labels.
Closes G5.

> **D-119 completed this decision, it did not reverse it.** D-117 renamed only the NetBox IMPORT
> PATH, which left `dc0` meaning VR1's first DC in the importer and **VR0's live cloud** in
> `lib-net.sh` -- one string, two clouds. D-119 executes this decision's own amendment
> (region-qualify the selectors: `vr0-dc0` / `vr1-dc0` / `vr1-dc1`) across `lib-net.sh`,
> `lib-hosts.sh` and `opentofu/`, making the importer's DC->site mapping an IDENTITY. **The
> alternative -- renaming the apex to match the repo's `dc1`/`dc2` -- was considered and REJECTED;
> see D-119.** The apex is NOT renamed and no production IPAM write is involved.

**Operator refinement (verbatim):** *"go ahead and rename the DC numbers. The office number is
fine."* -- so the **OFFICE KEEPS ITS NUMBER**: it stays **Office1**, and its NetBox site is
**`vr1-off1` / "VR1 Off1"**. That site does NOT exist upstream yet, so it is CREATED, not renamed --
no collision (VR0's `vr0-off0` is untouched), and **nothing deployed changes**.

**This is the G5 gap** (`docs/dc-dc-netbox-buildout-scope.md`), promoted to a decision because it
blocks the Office1 NetBox import and because the tool that trips over it WRITES BY DEFAULT.

**The collision, measured live against `netbox.baldurkeep.com` (4.5.8) on 2026-07-13:**

| v6 site `/48` | Prod NetBox site (the apex) | `dc-dc-prefixes-import.py` |
|---|---|---|
| `2602:f3e2:f02::/48` | **VR1 DC0** (`vr1-dc0`, id 43) | `--dc dc1` (its docstring: "`f02::/48` for dc1") |
| `2602:f3e2:f03::/48` | **VR1 DC1** (`vr1-dc1`, id 44) | `--dc dc2` |

The **addressing agrees**; only the **labels are off by one**. The repo's "DC1" and the apex's "DC0"
are the SAME datacenter. Same off-by-one on the office: the repo's "Office1" carve
(`2602:f3e2:f01:100::/56`, D-115) exactly mirrors VR0's **Off0** (`e01:100::/56`).

**Why this is dangerous, not merely cosmetic.** `dc-dc-prefixes-import.py` maps `dc1 -> slug
vr1-dc1` and **has no `--commit`** -- `--verify-only` is its read-only mode, so it WRITES BY
DEFAULT. Run it as documented against a NetBox seeded from prod and it binds DC1's `f02::/48`
prefixes to the site `vr1-dc1`, which upstream holds `f03::/48` -- i.e. **the other DC**, silently.
(`roles-aggregates-import.py` is the opposite: dry by default. Do not assume the tools are uniform.)

**How it happened, recorded because the mistake is reusable.** The script's own comment: *"Slugs are
new -- no prior convention existed in this repo (grepped `opentofu/*.tf` and the buildout design;
none found)."* It grepped the **repo**. The convention lived in the **apex**. NetBox is declared the
IPAM apex (D-010/D-103) precisely so that naming is NOT a repo-local invention -- but the check for
prior art was run against the repo. Same class of error as concluding "no VR0 precedent for
MAAS-composed LXD VMs" after grepping only `bundle.yaml` (2026-07-13, same session).

**What the site byte does and does NOT mean.** Within a region `/40`: `00`=Infrastructure,
`01`=Offices, `02`/`03`=the DCs. That is **sequential allocation, not a DC index** -- there is no
`02 -> DC0` mnemonic to preserve. An earlier draft of this entry claimed there was; that was wrong
and is corrected here, because it materially weakens the case against Option A.

**Blast radius, measured.** `vr1-dc0` and `vr1-dc1` are **skeletal**: 1 prefix each (their own `/48`
container), and **0** devices / racks / VLANs / clusters. (Contrast `vr0-dc0`: 18 prefixes.) Renaming
them upstream touches 2 site records + their 2 prefix descriptions. Renaming the REPO instead touches
the live surfaces (`lib-net.sh`, the `--dc` flag, `opentofu/`, runbooks, SKILL.md, and the text of
D-101/D-106/D-111/D-115) and contradicts **already-ratified D-106** (Designate `dc1`/`dc2` zones).

### Options

**A. Rename the two skeletal apex sites to `vr1-dc1`/`vr1-dc2`; add `vr1-off1`. (RECOMMENDED)**
The scope doc's standing recommendation. Cheapest by a wide margin, and it preserves the `dc1`/`dc2`
language already ADOPTED in D-106 and wired through every live repo surface. Cost: VR1 reads DC1/DC2
while VR0 reads DC0/DC1 -- a cosmetic cross-region inconsistency, and VR1's office becomes `Off1`
against VR0's `Off0`.
**TRAP -- rename ORDER matters:** `vr1-dc0 -> vr1-dc1` while `vr1-dc1` still exists is a **slug
collision**. Rename **`vr1-dc1` -> `vr1-dc2` FIRST**, then `vr1-dc0` -> `vr1-dc1`. Update the two
`/48` descriptions ("Datacenter 0 (DC0)") in the same change or they contradict their own site.

**B. Rename the repo to the apex's 0-indexed `DC0`/`DC1` + `Off0`.**
Uniform 0-indexing across VR0 and VR1, and the apex is never bent to fit the repo. Cost: reopens
ratified D-106, and rewrites every live repo surface. Historical changelogs are NOT rewritten.

**C. Re-point the tool only (`SITES = {dc1: vr1-dc0, dc2: vr1-dc1}`).**
Two lines, nothing else moves. **Argued against:** it makes `--dc dc1` write to a site *displayed* as
"VR1 DC0" forever -- a permanent hidden indirection, which is how this class of bug got here.

**Whichever is chosen, one fix is unconditional:** give `dc-dc-prefixes-import.py` a `--commit` flag
so it is DRY BY DEFAULT, matching `roles-aggregates-import.py`. A write-by-default IPAM tool is a
standing hazard regardless of how the naming lands.

### D-117 -- AMENDMENT (2026-07-13): ADOPTED (Option B), scope narrowed by operator ruling

**Ruled:** VR1's datacenters are **DC0** (`vr1-dc0`, `2602:f3e2:f02::/48`) and **DC1** (`vr1-dc1`,
`2602:f3e2:f03::/48`) -- matching the apex, which already holds both sites. The repo's `dc1`/`dc2`
labels are retired in favour of **`dc0`/`dc1`**.

VR1's office **remains Office1** -> NetBox site **`vr1-off1` ("VR1 Off1")**, carrying the D-115 carve
(`2602:f3e2:f01:100::/56` + `/64`, `10.10.0.0/22`). Note VR1 therefore reads **DC0/DC1/Off1** while
VR0 reads DC0/DC1/Off0 -- an accepted cosmetic asymmetry. The office site is NEW upstream, so it is
created cleanly and **no deployed object is renamed**.

**SCOPE -- what B does and does NOT touch.** B is a ruling about **IPAM naming authority**: the
NetBox site record and the selector that resolves to it. It does NOT extend to deployment substrate.

- **IN scope (renamed, DC ONLY):** the DC half of the NetBox site identity; the importer's `SITES` +
  GUA map; the `--dc` selector (`dc1|dc2` -> `dc0|dc1`); `lib-net.sh`/`lib-hosts.sh` DC keys;
  OpenTofu DC module labels; forward-looking DC prose.
- **OUT of scope -- NOTHING office-side is renamed.** Per the operator refinement above, `Office1`
  keeps its number, so `voffice1`, `office1-opnsense`, the `office1-local`/`office1-wan` networks,
  `office1-pool`, the MAAS VM host `office1-lxd`, the machines `office1-netbox` /
  `office1-tailscale`, the OPNsense `OFFICE1_LXD_GW` object and the Tailscale node all stand as-is.
  **No teardown, no rebuild.**
- **Also out of scope, independently:** deployment substrate is not a carve. Even where a name DID
  change (the DC libvirt objects), the rename is a label on empty, unused infrastructure -- see the
  table below.

**TRAP 1 -- this is a SEMANTIC REMAP, not a string replace.** `dc1 -> dc0` AND `dc2 -> dc1`, so
`dc1` denotes different things before and after. A sequential `sed` double-applies and corrupts the
tree. Any mechanical pass must go through a placeholder (`dc1 -> __DCA__`, `dc2 -> __DCB__`, then
`__DCA__ -> dc0`, `__DCB__ -> dc1`).

**TRAP 2 -- and this one NO mechanical pass can catch.** Existing prose uses "DC0" to mean **VR0's
rehearsal DC** while using "DC1"/"DC2" to mean **VR1's** DCs. The importer's docstring: *"DC1
INHERITS the DC0 six-plane CIDRs unchanged (D-101 -- 'DC1 equals the DC0 template') ... not the
existing `vr0-dc0` site."* Remap VR1's `dc1 -> dc0` and that reads *"DC0 equals the DC0 template"* --
two DIFFERENT DC0s (`vr0-dc0` and `vr1-dc0`) that were previously disambiguated by NUMBER ALONE.
**Every "DCn" in prose must be region-qualified by hand** (`VR0 DC0` vs `VR1 DC0`). This is why the
decision text below is ANNOTATED, not rewritten.

**Treatment by surface kind (adopted with the ruling):**

| Surface | Treatment |
|---|---|
| Operational/structured (importer, `lib-net.sh`, `--dc`, tofu labels) | Mechanical rename. **This tier ALONE unblocks the NetBox import.** |
| **ADOPTED decision text** (D-101, D-106, D-111, D-115) | **ANNOTATE in place, do NOT rewrite** -- the D-103->D-116 supersession precedent. Ratified prose is a historical record. |
| Forward-looking prose (runbooks, design docs) | Hand-edit, region-qualifying every "DCn" (Trap 2). |
| DC libvirt objects (`dc1-*`/`dc2-*` nets, `dc1-pool`/`dc2-pool`) | **Empty and unused** (0 volumes, no DC VMs). Rename the module labels; let the next **GATED** DC apply reconcile them. **No surprise destroy/recreate.** |
| Historical changelogs | **Never rewritten.** |

**Unconditional companion fix (also ruled):** `netbox/dc-dc-prefixes-import.py` becomes **DRY BY
DEFAULT** with an explicit `--commit`, matching `roles-aggregates-import.py`. A write-by-default tool
pointed at the IPAM apex is a standing hazard, and it is the mechanism by which this naming bug would
have landed silently.

### D-117 -- AMENDMENT (2026-07-13): the `$DC` shell selector is a SECOND, COLLIDING namespace

Found while executing the rename. **`scripts/lib-net.sh` / `lib-hosts.sh` use a THREE-value `$DC`
selector in which `dc0` ALREADY MEANS VR0's DC0** (the deployed rehearsal), with `dc1`/`dc2` meaning
VR1's two DCs:

```
dc0|dc1)   # lib-net.sh: VR0 DC0 and VR1's first DC share v4 literals (D-101 inheritance)
dc2)       # VR1's second DC -- FAILS LOUD, no literals assigned yet
```

So the repo has been carrying **two different `dcN` namespaces at once**: the shell libs' (where
`dc0` = VR0) and the NetBox importer's (where `dc0` did not exist and `dc1` = VR1's first DC). That
is Trap 2 in its purest form, and it is why a mechanical `sed` across this rename would have been
**actively dangerous**: renaming VR1's `dc1 -> dc0` in `lib-net.sh` collides HEAD-ON with VR0's
existing `dc0` and would silently hand VR1's selector VR0's deployed literals.

**Proposed fix (NOT yet executed -- does not block the NetBox import):** region-qualify the shell
selector to **the apex's own slugs** -- `vr0-dc0`, `vr1-dc0`, `vr1-dc1` -- so there is ONE namespace
across NetBox, the shell libs, and the importer, and every identifier is unambiguous by construction.
This is the same principle D-117 already adopts: the apex names things; the repo follows.

**Blast radius:** `lib-net.sh`, `lib-hosts.sh`, and the scripts that take `$DC`
(`carve-host-interfaces.sh`, `dc-dc-ceph-disk-budget.sh`, `dc-dc-dr-drill.sh`,
`dc-dc-radosgw-multisite.sh`, `dc-dc-rbd-mirror.sh`, `phase-00-maas-standup.sh`,
`reenroll-hosts.sh`) plus their harnesses and `tests/dc-selector/`.

**Until it is executed, the shell `$DC` selector retains its OLD meaning** (`dc0`=VR0 DC0,
`dc1`/`dc2`=VR1) while `netbox/dc-dc-prefixes-import.py --dc` uses the NEW, apex-aligned meaning
(`dc0`/`dc1` = VR1's two DCs). **These two are NOT interchangeable.** Do not copy a `$DC` value from
a shell script into `--dc`, or vice versa, without re-reading this entry.

---

## D-118: the org ULA /48 -- ADOPT `fd50:840e:74e2::/48`

**Status:** **ADOPTED 2026-07-13 (operator ruling).** Closes gap **G3**
(`docs/dc-dc-netbox-buildout-scope.md`) and the D-101 open literal `ORG_ULA_48`. **UNBLOCKS the VR1
DC six-plane NetBox import**, which could not run without it.

**Ruled:** `ORG_ULA_48 = **fd50:840e:74e2::/48**`.

D-101 rules the org ULA `/48` a NetBox-assigned literal, "NOT hardcoded in this decision", and every
tool takes it as a REQUIRED env var with no default. Until now no decision assigned it, so
`netbox/dc-dc-prefixes-import.py` was unrunnable -- supplying a value would have been an inferred
value (hard rule 2).

**Validation (measured 2026-07-13, not asserted):**

| Property | Result |
|---|---|
| Inside `fc00::/7` | yes |
| First byte `0xfd` -> **L=1**, locally assigned (RFC 4193) | yes |
| 40-bit Global ID | `0x50840e74e2` -- no discernible structure |
| Prefix length | `/48` -> **256 x `/56`**, and D-101 allocates one `/56` per DC |
| Collides with Tailscale's ULA `fd7a:115c:a1e0::/48` | **NO** |
| Collides with anything else in the apex | no -- the apex holds only the `fc00::/8` / `fd00::/8` containers and Tailscale's `/48` |

**The Tailscale non-collision is the load-bearing check, not a formality.** `office1-tailscale` runs
with `--accept-routes` on the self-hosted control plane, so an org ULA overlapping
`fd7a:115c:a1e0::/48` would be a LIVE routing conflict on the tailnet, not a paper one. `fd50` vs
`fd7a` -- distinct.

**Provenance, stated honestly.** `docs/dc-dc-netbox-buildout-scope.md` 4c describes this value as
CSPRNG-generated. That claim originates in an earlier session and **cannot be re-verified now**. It
does not change the ruling: RFC 4193's "SHA1 of time + MAC" construction is a **SHOULD**, not a MUST
-- the actual requirement is a pseudo-random 40-bit Global ID, which this satisfies. Regenerating
would not make the value MORE correct; it would only substitute a different random number, at the
cost of re-editing the 7 places that already carry this one and risking a stale literal surviving in
one of them. Not worth it.

**Corrects a FALSE claim in the repo.** `docs/archive/changelogs/changelog-20260711-ula-gen-command-fix.md` called this
"the **ratified** VR1 value" -- it was NOT ratified; no D-number ever assigned it. That phantom
ratification is very likely why the value has been treated as settled for two days while
`grep ORG_ULA docs/design-decisions.md` returned nothing. This entry makes the claim true, and the
changelog is annotated. **Same failure mode as D-117:** a value treated as authoritative because a
document asserted it, with no decision behind it.

**Consequences.** The DC import can now run:
```
ORG_ULA_48=fd50:840e:74e2::/48  DC_GUA_PREFIX=2602:f3e2:f02::/48  --dc dc0   # VR1 DC0
ORG_ULA_48=fd50:840e:74e2::/48  DC_GUA_PREFIX=2602:f3e2:f03::/48 \
    DC1_V4_SUPERNET=10.12.64.0/19  --dc dc1                                   # VR1 DC1
```
(`DC_GUA_PREFIX` values are MEASURED off the apex per D-117; `DC1_V4_SUPERNET` is assigned by D-115.)
Per D-101/D-111 each DC takes a `/56` of this `/48`, indexed to its GUA site nibble, so the 4th
hextet reads `DC.NN` in BOTH families.

## D-119: the VR1 DC namespace -- region-qualify the selectors; the repo adopts the apex

**Status:** **ADOPTED 2026-07-14 (operator ruling).** COMPLETES D-117 rather than reversing it:
it executes D-117's own amendment (region-qualify the shell selectors) across the three surfaces
D-117 never touched. D-117 stays ADOPTED; its "partially executed" caveat is discharged here.
Closes tooling-gap #19 and Stage 2 close-out item C3.

### The problem, stated exactly

After D-117, the token `dc0` meant TWO DIFFERENT CLOUDS depending on which file you were reading:

| Surface | `dc0` meant | `dc1` meant |
|---|---|---|
| NetBox apex (the IPAM authority) | VR1's FIRST DC | VR1's SECOND DC |
| `scripts/lib-net.sh` / `lib-hosts.sh` | **VR0's DC0 -- a DIFFERENT, LIVE cloud** | VR1's first DC |
| `opentofu/main.tf` | (no `dc0`) | VR1's first DC |

One string, two clouds, one of them in production. A mechanical renumber makes it WORSE, not
better (D-117 TRAP 1): `sed s/dc1/dc0/` over `opentofu/` would hand VR1's first DC the string
`dc0`, which `lib-net.sh` already resolves to VR0's deployed literals.

### The ruling

**Region-qualify the DC selector everywhere. The repo adopts the apex's names verbatim.**

    vr0-dc0   VR0's DC0        the LIVE testcloud (a different region)
    vr1-dc0   VR1's FIRST  DC  GUA 2602:f3e2:f02::/48
    vr1-dc1   VR1's SECOND DC  GUA 2602:f3e2:f03::/48

Bare `dc0` / `dc1` / `dc2` are REJECTED loudly by the selectors. They are the ambiguity being
deleted; accepting them "for compatibility" would preserve the bug.

**The apex is NOT renamed. There is NO production IPAM write in this decision.** The apex was
already self-consistent and 0-indexed; MEASURED 2026-07-14 against `netbox.baldurkeep.com`:

    2602:f3e2:f02::/48  -> site vr1-dc0   "Virtual Region 1 (VR1) Datacenter 0 (DC0)"
    2602:f3e2:f03::/48  -> site vr1-dc1   "Virtual Region 1 (VR1) Datacenter 1 (DC1)"
    (VR0, for contrast: vr0-dc0, vr0-dc1, vr0-off0 -- the same 0-indexed convention)

The repo was the only surface out of step, because -- operator, verbatim in substance -- *"we
didn't catch the dc0 missing and went straight to one."*

### Why this beats the alternative (renaming NetBox to match the repo)

Considered and REJECTED: rename the apex to `vr1-dc1`/`vr1-dc2` so the ordinals match the repo's
`dc1`/`dc2`. It was rejected because it would:
- require a PRODUCTION IPAM write on the authority, ordered highest-slug-first or it violates
  NetBox's unique-slug constraint;
- make VR1 the ONLY 1-indexed region, diverging from VR0 -- the very region VR1 rehearses; and
- create a worse ambiguity: `vr1-dc1` would mean VR1's FIRST DC while `vr0-dc1` still means VR0's
  SECOND DC. Same suffix, different ordinal, one NetBox.

(For the record, the apex is NOT uniformly `dcN`: the Eugene/Roosevelt sites are place-named --
Charnelton, Cleveland, Roosevelt, Stevenson, Willamette. The `dcN` convention belongs to the
REHEARSAL regions VR0/VR1, and within those it is 0-indexed. The consistency claim is scoped to
them, not to the whole apex.)

### The prize: the off-by-one becomes structurally impossible

The importer's DC->site mapping becomes an IDENTITY (`vr1-dc0` -> slug `vr1-dc0`). **There is no
offset table left to get wrong.** This matters because the original defect was precisely a WRONG
LOOKUP TABLE -- `--dc dc1` bound to slug `vr1-dc1` while treating `f02::/48` as dc1's, though the
apex binds `f02` to VR1 DC0. It would have written the FIRST DC's prefixes onto the SECOND DC's
site, silently. Deleting the mapping deletes the bug class.

### Scope -- what this decision does NOT change

- **The office KEEPS its number** (operator ruling): NetBox site `vr1-off1`, and `Office1` /
  `voffice1` / `office1-local` / `office1-netbox` / `~/vr1-office1-creds` throughout. The site is
  BUILT AND RUNNING; renaming it churns hostnames, creds paths and libvirt networks for zero
  functional gain. This leaves a known, ACCEPTED asymmetry with VR0's `vr0-off0`. Documented, not
  fixed.
- **VR0 is not renamed.** Its DC stays DC0; only the SELECTOR STRING is region-qualified
  (`dc0` -> `vr0-dc0`). No VR0 literal, host, or deployed object changes.
- **The ULA hextet still mirrors the GUA site nibble, NOT the DC number** (D-111): `vr1-dc0` ->
  `fd50:840e:74e2:2NN::`, `vr1-dc1` -> `:3NN::`, tracking `f02`/`f03`. This is DELIBERATE so the
  4th hextet reads the same in both families. Do not "fix" it to match the DC index -- that would
  break the already-imported sandbox prefixes.
- **NetBox data is untouched.** No re-import, no re-dump of `netbox/draft/vr1-draft.json`.

### Blast radius (MEASURED 2026-07-14, before any edit)

`opentofu/` renames change the libvirt objects' `name` attribute, so `tofu apply` will
DESTROY AND RECREATE them. Measured: the 6 `dc1_planes` networks, both DC storage pools, and the
3 mesh links are live in state but **EMPTY** -- no guests attached (Stage 3 has not run) and no
volumes in the DC pools (the only volumes belong to `office1_opnsense`, `ubuntu_noble_base`, and
`voffice1`, none of which are in a DC pool). The recreate is therefore safe, but it IS a
destroying apply and is gated: the plan is presented before it runs.

### Standing rule this leaves behind

**Never write a bare `dcN` for a VR1 datacenter again, in code or in prose.** Region-qualify it.
The one durable lesson of D-117 -- which survives its completion -- is that an unqualified `dcN`
in a repo that operates more than one cloud is a latent off-by-one waiting for a reader who does
not know which cloud the sentence is about.

## D-119 -- AMENDMENT (2026-07-27): the dc1 arm's OpenStack-layer literals are DERIVED from the overlay, not authored twice

**Status:** RULED 2026-07-27 (operator, GA-R5). Question as presented, verbatim: "R9 -- dc1's
OpenStack-layer literals (VIP bands, FIP pool, keystone VIP). They are ratified in
`overlays/vr1-dc1-vips.yaml` but `lib-net.sh`'s dc1 arm unsets them, and 20 of 28 consumers
never call the selector so they silently get dc0 values. R2 made these dual-family. Where
should the authority live?" Options presented: (a) derive lib-net's dc1 arm from the overlay
with a drift check; (b) hand-populate the dc1 arm from the ratified values; (c) re-point the
`phase-0*` scripts at the overlay as authority. Operator selection, exact utterance:
**"Derive lib-net's dc1 arm from the overlay, with a drift check (Recommended)"**.

**RULED SHAPE.** `scripts/lib-net.sh`'s `vr1-dc1` arm is GENERATED from
`overlays/vr1-dc1-vips.yaml` (the ratified, deploy-input copy), with a render-drift check
that FAILS the gauntlet when the checked-in arm and a fresh render diverge. Consumers keep
sourcing `lib-net.sh` unchanged -- no script learns to parse YAML -- and there is exactly ONE
AUTHORED copy of the values.

**PRECEDENT, and why this is not new machinery.** D-137 sub-ruling 2 already ruled and built
exactly this shape for credentials: `creds-manifests/*` are DERIVED from `creds-matrix.tsv`,
`--render` regenerates them, and the gauntlet fails on rendered-vs-checked-in drift. This
amendment applies the proven pattern to network literals.

**IT DOES NOT PRE-EMPT D-136.** D-136 (NetBox-coupled per-DC render pipeline) is PROPOSED and
UNRULED. This ruling is deliberately the smaller, compatible step: it makes the overlay the
single authored source for the SCRIPT-facing values. If D-136 is later adopted, the apex
becomes the source and BOTH the overlay and this derived arm become generated -- this
amendment is a sub-case of that, not a competitor to it.

**MEASURED BASIS.** `lib-net.sh:76-79` states the backward-compatibility design in terms:
sourcing without the selector "continues to populate PLANE_CIDRS ... exactly as above
(VR0/DC0's real, measured values) -- every existing script that sources lib-net.sh keeps
working completely unchanged." Consequence, and it inverts the intuitive reading: the `unset`
block fires ONLY for selector-CALLERS, so the 8 correctly-updated scripts break LOUDLY under
`set -u` while the 20 that were never updated fail SILENTLY on dc0's `10.12.4/8/12` literals.
The silent half is the dangerous half.

**GUARD, carried from R11 -- do NOT let the derivation touch these.** The dc1 arm unsets NINE
variables for TWO different reasons. `METAL_INTERNAL_VID` and `METAL_INTERNAL_IFACE` are
**correctly** unset: D-133 abolished the VLAN-103 / `br-internal` stack for VR1 and those
facts genuinely do not exist. ONLY the VIP/FIP/keystone group
(`VIP_PREFIX_PROVIDER`/`ADMIN`/`INTERNAL`, `VIP_COUNT_EXPECT`, `FIP_POOL_START`/`END`,
`KEYSTONE_VIP_DEFAULT`) is superseded and derivable. A generator that populates all nine would
silently reintroduce a stack D-133 retired.

**COUPLED RULINGS.** R2 made these DUAL-FAMILY, so the derivation must carry both families,
not v4 only. R11 moved `VIP_COUNT_EXPECT` 11 -> 13 (vault `.61`, designate `.62`) and widened
the octet band to `.99`, so the generator and the checker constants must agree on the same
band -- note `OCTET_LO/HI` in `provider-bundle-check.py` and `VIP_OCTET_MAX` in `lib-net.sh`
are SEPARATELY NAMED, a two-file change (L3-7).

**STILL OWED AND NOT COVERED BY THIS RULING:** the CONSUMER SWEEP. This decides where the
values live; it does not make the 20 non-selector scripts call `lib_net_select_dc`. Measured
Stage-5 exposure is TWO of them (`phase-03-core-verify.sh`, `deploy-watch.sh`); the rest bite
at Stages 6-7. `scripts/carve-host-interfaces.sh:48-67` (DOCFIX-166) is the in-repo precedent
for how a consumer is correctly wired.

**Execution is a SEPARATE gated step.** Standard delivery discipline: generator + drift check
ship with a harness green, gauntlet ALL GREEN, repo-lint 0-fail, and a changelog entry
carrying a revert.

## D-120: static-services addressing convention for the MAAS-managed compose /24s (ADOPTED)

**Status:** **ADOPTED 2026-07-15 -- option (a), EXECUTED.** Operator endorsed the direction (static
services belong in a low band, not mixed with deployed nodes) and ruled "it doesn't matter where the
IPs land in the sub-50 scope; go ahead with changing the IPs." The band layout below and the Office1
service addresses (`.10`/`.11`) are ratified, and the Office1 re-IP was executed the same day (see
"Execution" and `runbooks/dc-dc-office1-service-reip.md`). Refines D-115 (which ratified that
`10.10.1.0/24` exists as the Office1 compose net but left its INTERNAL layout as just "MAAS
DHCP/PXE"). Does NOT reopen D-115's /22 carve.

### The problem

On `10.10.1.0/24` (Office1 `lxdbr0`, MAAS-managed), the persistent site services sit at
`office1-netbox = 10.10.1.201` and `office1-tailscale = 10.10.1.202` -- addressed IDENTICALLY to how
a deployed OpenStack node would be. MAAS auto-assigns deployed machines from `.201+` (just past its
`.100-.200` dynamic range), so the services landed there by default, not by design. They are NOT in
the DHCP dynamic range (no collision), but they are not distinguishable from ephemeral compute nodes
either. There is no ratified sub-layout for the /24.

### Decision (PROPOSED) -- a functional band layout for EVERY MAAS-managed compose /24

Applies to Office1's `10.10.1.0/24` AND to each DC's compose /24 when the DCs are built (they inherit
the template -- minimize-delta):

    .1          gateway (the site containment VM: voffice1, and each DC's equivalent)
    .2 - .49    STATIC site infrastructure/services (MAAS static-assign; OUTSIDE the dynamic range)
    .50 - .99   reserved / spare (headroom for more static services)
    .100 - .200 MAAS DYNAMIC (enlistment / commissioning / PXE / DHCP)   [as already deployed]
    .201 - .254 deployed compute / OpenStack nodes (MAAS auto-assign)    [as already deployed]

The dynamic and node bands are UNCHANGED from what MAAS already does -- only the static-services band
`.2-.49` is newly reserved, and it already sits outside the dynamic range, so no MAAS range edit is
needed, only per-service static IP assignment.

### Office1 service addresses (PROPOSED; operator to ratify)

    office1-netbox     10.10.1.201 -> 10.10.1.10
    office1-tailscale  10.10.1.202 -> 10.10.1.11

`.10`/`.11` are arbitrary within `.2-.49`; chosen low and adjacent. Operator may pick otherwise.

### Recording it in NetBox (the IPAM authority deciding where NetBox itself lives)

The band layout is a carve that belongs IN NetBox: register `10.10.1.0/24`'s child ranges (static /
dynamic / node) and the two service IP assignments. Do this through the sandbox loop (D-115 / the
C4-documented loop), not by hand -- NetBox is the authority for exactly this.

### Consequence / cost (why this is not a free change)

Re-IPing Office1's two RUNNING services cascaded to the repo references, INCLUDING the hardcoded
`SANDBOX_HOSTS` allowlist (`{localhost, 127.0.0.1, 10.10.1.201}` -> `.10`) in
`roles-aggregates-import.py`, `dc-dc-prefixes-import.py`, and `d115-office-carve.py` (added
2026-07-14) plus its test. The sweep + gauntlet ran with the re-IP (see "Execution" above and
`runbooks/dc-dc-office1-service-reip.md`). Dated changelogs keep `.201`/`.202` as historical
snapshots.

### Options

- **(a) ADOPT the convention AND re-IP Office1 now** (recommended) -- Office1 is the proven template;
  making it match means the DCs inherit a clean, already-exercised layout. Cost: a gated re-IP of two
  running services + the 20-reference sweep.
- **(b) ADOPT the convention for the DCs ONLY; leave Office1 at `.201/.202`** -- zero disruption now,
  but Office1 (the template everyone copies) permanently diverges from the convention, and the
  `SANDBOX_HOSTS` allowlist keeps a "deployed-node-band" address for a static service.
- **(c) Reject** -- keep services in the node band. Not recommended: it is the ambiguity this entry
  exists to remove.

**Recommendation: (a).** Get the template right once.

### Execution (2026-07-15) -- option (a), done

Re-IP executed on the two running services, no wipe, verified on the wire:

    office1-netbox     10.10.1.201 -> 10.10.1.10   NetBox HTTP 200 at .10, reachable vcloud-side
    office1-tailscale  10.10.1.202 -> 10.10.1.11   tailscale Running, still advertises 10.10.0.0/22

Two findings that shaped the method (and that the DC replication inherits):

- **MAAS will NOT edit an interface on a Deployed machine** ("Cannot unlink subnet interface because
  the machine is not New, Ready, Allocated, or Broken"). The only lift is Release->redeploy, which
  WIPES the OS -- forbidden here (it would destroy the seeded sandbox). So the durable MAAS-model
  update is unavailable without a wipe. Resolution: set the live address on the GUEST only, via BOTH
  `/etc/netplan/50-cloud-init.yaml` and `/etc/cloud/cloud.cfg.d/50-curtin-networking.cfg` (so it
  survives reboot), and ACCEPT MAAS-model drift (MAAS still shows `.201`/`.202` `mode=auto`). The
  drift reconciles for free at the next teardown/redeploy from a corrected model. `.10`/`.11` are
  off MAAS's auto-assign path -- MAAS allocates nodes from `.201+` upward (observed: the two services
  themselves landed at `.201`/`.202`), so the low `.2-.49` band is never auto-handed out. NB: the
  whole non-dynamic space is MAAS's static pool, so "outside the dynamic range" is NOT the guarantee;
  the allocation DIRECTION is. A DC building fresh sidesteps this by MAAS-static-assigning `.2-.49`
  before deploy (Step 1 below), which pins the addresses explicitly rather than relying on direction.
- **NetBox `ALLOWED_HOSTS` = `*`** (config default; not overridden in `env/netbox.env`), so NetBox
  served at `.10` with no `DisallowedHost` change needed.
- **Method note for the DCs:** drive the guest change over `lxc exec` from the containment VM, NOT
  SSH-to-the-guest-IP -- the LXD socket is immune to the `netplan apply` that swaps the address, so
  the control channel never drops. `lxc console`/`exec` is also the recovery backstop if the new
  address does not come up. For a DC building fresh, prefer static-assigning `.2-.49` in MAAS BEFORE
  deploy (a Ready machine accepts the link edit) so there is no drift to accept -- the Deployed-block
  above only bit Office1 because its services were already deployed in the node band.

The NetBox-side carve record (Step 6 -- register the `.2-.49`/dynamic/node child ranges + the two
service IPs in the sandbox, then feed upstream) is still OWED and rides with C2; it is NOT done here.

## D-121: VR1 makes HA real -- scale the decorative single-unit control plane to 3, and the per-DC node layout that carries it [ARCH]

**Status:** ADOPTED (2026-07-15/16, operator rulings). **Node layout = Option C** (3 control +
2 compute + **4 storage** per DC, **18 node VMs**) -- storage count amended 3->4 per the 2026-07-16
AMENDMENT (R-3, restores Ceph rebuild headroom); ruled after the whole-host validation proved it FITS
(details below; RE-VALIDATE for 3+2+4 + Model B nesting per R3-F06). The **14-service HA scale-up**
(1 -> 3) is ADOPTED. **Vault-HA backend sub-ruling: RESOLVED = (v-a)** (operator ruling **2026-07-16**;
the earlier "2026-07-15 operator ruling" attribution was AGENT-AUTHORED with no operator vault
utterance -- corrected per review R3-F01) -- keep vault MySQL-backed (on the already-3-unit `mysql-innodb-cluster`),
scale to 3, re-add the `vault:ha <-> vault-hacluster:ha` relation. GATE: verify the deployed vault charm
(`1.8/stable`) actually does HA leader-election on the MySQL storage backend with 3 units before commit;
if it does NOT, do not fall back to (v-b) etcd -- evaluate Raft integrated storage and route to D-068
(etcd storage is deprecated upstream; Raft is the better Roosevelt target). Rationale: reuses existing
HA (zero new apps), keeps control-node container density down, does not front-run the unsettled Roosevelt
backend choice (D-068), and adds no NEW SPOF (the whole cloud already depends on mysql). Governs
Stage 3's `modules/node-vm` sizing/count (was blocking phase2 runbook Step 6 -- now UNBLOCKED for
layout) and the Stage 5 per-DC bundle deploy. Raised by the operator's Stage-3 ruling ("this is the
time we add the additional HA nodes missing from the previous deployments").

**Ruled node sizing (Option C, validated to fit -- keep control nodes <= 64 GiB per the sensitivity):**
3x control @ 16 vCPU / 64 GiB / 150 GiB; 2x compute @ 12 vCPU / 48 GiB / 100 GiB; 3x storage @ 8 vCPU
/ 24 GiB / 550 GiB (500 OSD + 50 OS). Whole-host allocation across Office1 + both DCs + edges +
containment overhead = 222 vCPU / 790 GiB / 5.4 TiB of the measured 256 / 1024 / 10240 (RAM binding at
77%). `ceph-osd` = 3 units/DC (the storage nodes), `nova-compute` = 2 units/DC (the compute nodes),
`ceph-mon`/`mysql`/`rabbitmq`/`vault`/`ovn-central` = 3 across the 3 control nodes.

### The problem this fixes

VR0/testcloud ran its OpenStack control plane **single-unit by design**. Per **D-009** ("Hacluster
modeling at testcloud scale"): the hacluster + VIP relations were included at `num_units: 1` as a
**decorative** pattern -- "a single unit can't form a real HA quorum" -- explicitly so that
"Roosevelt scale-up is mechanical: change `num_units: 1` -> `num_units: 3` and rerun." The bundle
already carries all 12 hacluster subordinates and, per **D-020**, both provider+metal VIPs on every
clustered API app. VR1 (the regional rehearsal) is where that decorative HA is made real -- the
missing units get added.

### Measured current state (bundle.yaml, machines 8/9/10/11 -- the 4 VR0 nodes)

Already at quorum (NO change): `mysql-innodb-cluster`=3 (D-062, must deploy at 3),
`ovn-central`=3 (Raft), `ceph-mon`=3 (Paxos). Scale-out (sized to node count): `ceph-osd`=4,
`nova-compute`=3 (on 9/10/11 -- machine 8 is compute-free today).

**Missing HA -- 14 apps currently single-unit that this decision scales to 3:**

| App | Now | -> | Mechanism | Governing |
|---|---|---|---|---|
| keystone, glance, neutron-api, nova-cloud-controller, placement, cinder, openstack-dashboard, octavia, barbican, magnum, designate | 1 | 3 | hacluster + dual VIP (already wired) | D-009, D-020 |
| ceph-radosgw | 1 | 3 | hacluster + VIP (wired; NOT in D-009's named 12 but HA-eligible) | -- |
| rabbitmq-server | 1 | 3 | native clustering / quorum queues (no hacluster) | -- |
| vault | 1 | 3 | see the backend sub-question below | D-006 (revised by C1) |

Left single (NOT scaled): `ceph-rbd-mirror`=1 (**D-108** rules single-unit for VR1),
`glance-simplestreams-sync`=1 (utility), `designate-bind`=1
(DNS backend; optional 2). **memcached UPDATED 2026-08-06 (DOCFIX-212): now =3, NOT
single** -- operator directed the scale-up 2026-07-31; it was carried in the (now-retired)
dc-ha-scaleup.yaml overlay but BUNDLEFIX-053 did not fold it, so bundle.yaml + live sat at
1 until BUNDLEFIX-055 (2026-08-06) folded `num_units: 3` + `to:[lxd:0,1,2]` into the base
bundle (3 independent caches, no hacluster/VIP). LIVE is still 1 -- a redeploy from
bundle.yaml (or a live `add-unit -n 2`) brings it to 3.

### Two realities the "mechanical 1->3" framing hides (fold into the execution estimate)

1. **It is a placement rewrite, not a `num_units` tweak.** Every app pins its unit explicitly
   (`keystone to: [lxd:8]`, `glance to: [lxd:11]`, ...). Setting `num_units: 3` with a single-entry
   `to:` list makes Juju place one unit there and auto-place the other two wherever -- NOT
   guaranteed on three DISTINCT hosts, which is the entire HA value. Each of the 14 apps needs its
   `to:` expanded to a real 3-machine spread (anti-affinity). This is the actual work.

   **Placement-balance rule for the Stage-5 rewrite (Option C, 3 control nodes).** The 14 services x 3
   units = 42 API LXD containers land on the **3 control nodes** (`vr1-dc0-control-01..03`) -- ~14 per
   node, roughly 2x the density of VR0's balanced ~7 containers/node (measured from the openstack0-3
   bundle placement, 27 control containers over 4 nodes). Two rules keep that density balanced:
   - **Spread each app's 3 units one-per-control-node** (`to: [lxd:control-01, lxd:control-02,
     lxd:control-03]`). 3-unit HA does this naturally, so the RAM-heavy stateful trio
     (`mysql-innodb-cluster`, `rabbitmq-server`, `vault`) auto-balances to one replica per control
     node -- do NOT stack two of them on the same node (VR0's own placement already avoided this:
     rabbitmq on M10, vault on M11, mysql spread 8/9/10).
   - **Distribute the ~42 containers EVENLY (~14/control node), not front-loaded onto control-01.**
     The whole-host validation sized control nodes at 64 GiB for exactly this ~14-container load; if a
     future rebalance wants ~9/node instead, the lever is a 4th control node per DC (NOT more units).
   `ceph-osd`(3) sits on the storage nodes and `nova-compute`(2) on the compute nodes -- neither
   competes for control-node RAM. This rule governs the Stage-5 bundle deploy; it does not change the
   node COUNT ruled above.
2. **Vault HA backend fork -- RESOLVED = (v-a), see Status.** The bundle runs vault **MySQL-backed**
   today (`vault:shared-db -> vault-mysql-router -> mysql-innodb-cluster`), and **C1 revised D-006**:
   "etcd/easyrsa dropped for testcloud; Raft-vs-etcd is a Roosevelt item," and the
   `vault:ha <-> vault-hacluster:ha` relation was **removed (BUNDLEFIX-002)**. So scaling vault is
   NOT "just add units". The operator ruled **(v-a)** (charm-verify gated); (v-b) retained below for
   the record:
   - **(v-a) Keep MySQL-backed, re-add the `vault:ha` relation, scale to 3** -- lowest delta;
     leans on the already-3-unit mysql-innodb-cluster. Must be verified against the deployed vault
     charm's real multi-unit/ha behavior on a MySQL backend before committing.
   - **(v-b) Adopt D-006's etcd(3)+easyrsa(1) HA backend** -- the originally-documented topology,
     but reverses C1 and adds two applications.
   **Operator sub-ruling RESOLVED 2026-07-16 = (v-a)** (this line previously read "needed"; the
   Status's 2026-07-15 "RESOLVED" was agent-authored -- now genuinely operator-ruled, see the
   AMENDMENT). Either way, 3-unit vault **triples the manual-unseal burden**,
   which intersects the still-OPEN **SEC-003** (custodian assignment + second-person unseal
   rehearsal) that already keeps `d011-06-vault-unseal` MANUAL and gates D-011's full close.

### The per-DC node layout (the `modules/node-vm` count/sizing this stage must ratify)

The site-DOWN / recovery drill the operator wants (D-122) has the **whole DC VM** as its fault
domain, so intra-DC hyperconvergence is acceptable; role separation mainly buys single-node-failure
drills, a secondary concern here.

| Opt | Per-DC layout | nodes/DC | x2 DC | HA semantics | Whole-host cost (MEASURED, all layers) |
|---|---|---|---|---|---|
| A | 3 hyperconverged | 3 | 6 | quorum survives 1 loss; size=3 has ZERO rebuild headroom | lightest |
| **B (rec)** | **4 hyperconverged (= VR0 baseline)** | **4** | **8** | quorum + 4th host = a size=3 re-replication target after a node loss; lowest delta to the proven VR0 placement | 158 vCPU / **838 GiB** / 5.6 TiB (RAM 82%) |
| C | 3 control + 2 compute + 3 storage | 8 | 16 | cleanest HA/DR, Roosevelt-shaped | 222 vCPU / **790 GiB** / 5.4 TiB (RAM 77%) |

**Whole-host validation (2026-07-15, MEASURED budget 256 vCPU / 1024 GiB / 10 TiB; model at
`scratchpad/optc-calc.py`, all layers: Office1 as-built + both DCs + edges + containment overhead).**
**Both B and C FIT.** The binding resource is **RAM**, not disk (disk ~53-55%, and thin-provisioned)
and not vCPU (overcommits freely in a sim; C's 87% "allocation" is not a real ceiling). Counter to
the first-pass intuition, **C is NOT heavier than B** -- at lean-sim node sizing C uses *less* RAM
(790 vs 838 GiB) because role separation lets the 6 storage nodes run lean (24 GiB) vs B's uniformly
96-GiB hyperconverged nodes. So **C's real cost is operational (16 node VMs vs 8, larger VR0 delta),
not capacity.** SENSITIVITY: the knob is control-node RAM -- C with 96-GiB control nodes hits 982 GiB
(96%, only 42 GiB margin); keep control nodes <= 64 GiB. Per-node RAM sizing dominates the total far
more than the A/B/C structure does.

**RULED: Option C (2026-07-15).** The whole-host validation (below) removed the capacity objection --
C fits, and is in fact RAM-lighter than B -- so the decision turned on HA/DR quality: C is
role-separated and Roosevelt-shaped, giving the cleanest site-down AND single-node-failure drills,
which is the point of the VR1 rehearsal. B (4 hyperconverged/DC) was the runner-up on lowest-delta /
operational-simplicity grounds (my initial recommendation); A (3 nodes) was rejected as FEWER than
VR0's 4, reading against the operator's "additional HA nodes." Control nodes are held <= 64 GiB per
the sensitivity finding. B's rationale is retained here for the record: it was the smallest change
from VR0's proven placement, with three distinct hosts for quorum plus a fourth for Ceph rebuild.

**Disk-budget check (validated 2026-07-15, `scripts/dc-dc-ceph-disk-budget.sh`, Option B 4+4, 0.2
backup overhead):** PASS across the plausible per-node OSD range -- 300 GiB/node -> 7.18 TiB margin,
500 GiB -> 5.31 TiB, 700 GiB -> 3.43 TiB. **Disk is NOT the binding constraint for B**, so the
node-COUNT decision is robust regardless of final OSD sizing. NB the per-node OSD footprint is an
ASSUMPTION here (the ratified value pends the OSD-sizing decision and should be re-derived from the
measured DC0 footprint, as Stage 1's own margin was); this run proves headroom, not the final number.

**Per-node sizing (feeds the module):** at full HA a hyperconverged node runs ~13 control LXD
containers + an OSD + (on 3 of 4) nova-compute, atop D-040's 8 GiB reserved-host-memory. Propose
**16 vCPU / 64-96 GiB RAM / OSD-sized disk** per node (lean to 96 GiB -- 64 is tight once D-040
reserves 8 and the container stack lands). 8 nodes x 16 vCPU = 128 vCPU and x 96 GiB = 768 GiB, both
inside vcloud's 256 vCPU / 1 TiB with headroom for Office1 + the edges.

### What the operator ruled (2026-07-15)

1. **Node layout:** RULED **Option C** (3 control + 2 compute + 3 storage per DC), sizing above.
2. **Vault HA backend:** RULED **(v-a)** MySQL-backed + re-add `vault:ha`, charm-verify gated (Status).
   The SEC-003 unseal-burden intersection stands (3-unit vault = 3x manual unseal).
3. **Per-node sizing:** RULED (control 16/64/150, compute 12/48/100, storage 8/24/550; Status).

**Related:** D-009 (decorative HA pattern this executes), D-006 + C1 (vault backend history),
D-020 (dual VIPs, already present), D-062 (mysql-at-3), D-108 (rbd-mirror stays 1), D-040
(reserved memory), D-104 (single-unit JUJU controller -- unaffected; this is the OpenStack control
plane, not the Juju bootstrap node), SEC-003 (unseal). Does NOT touch DC2/`vr1-dc1` scope.

### D-121 -- AMENDMENT (2026-07-16): storage count 3->4/DC; vault v-a operator-ruled; re-validation owed

Two operator rulings this date + one record correction:
1. **Storage nodes 3 -> 4 per DC (R-3).** Option C is now **3 control + 2 compute + 4 storage = 9
   nodes/DC** (18 total); `ceph-osd` = 4 units/DC. Rationale: 3 OSD hosts at size=3 = zero rebuild
   headroom (a single storage-node loss cannot re-replicate until the host returns -- the property
   D-121 flagged for the REJECTED Option A but not, originally, for the ADOPTED Option C; review
   R3-F08). 4 OSD hosts restore self-heal (CRUSH can still place 3 distinct replicas on host loss).
   Ceph disk re-checked for 4x500Gi/DC = PASS (5.31 TiB margin). Whole-host RAM hand-estimate ~870 GiB/1024 (85%) at the ratified 32 GiB Model B
   containment overhead (was ~838/81% at 16 GiB -- see the vault/re-validation amendment below).
2. **Vault-HA backend = (v-a) MySQL-backed, operator-ruled 2026-07-16.** The 2026-07-15 "RESOLVED"
   was agent-authored (no operator vault utterance; the only AskUserQuestion asked node-layout + a
   delivery "Proceed", never vault -- R3-F01). The operator confirmed v-a on 2026-07-16 after (v-b)
   etcd was rejected (deprecated) and Raft was found INFEASIBLE on the deployable 1.8/stable charm
   (Raft is a 1.16-charm feature; 1.16 is ruled incompatible, D-068/BUNDLEFIX-010). Raft is parked as
   the D-068/Roosevelt target. No code delta at ruling time -- v-a was encoded in the HA overlay,
   then folded into `bundle.yaml` by BUNDLEFIX-053 (2026-08-05); the overlay was retired 2026-08-06.
3. **Re-validation OWED (R3-F06):** the whole-host capacity model `scratchpad/optc-calc.py` is
   UNCOMMITTED and absent, so the changelog's "reproducible" claim is currently false. It must be
   committed as a harnessed calculator and RE-RUN for the 3+2+4 shape AND Model B nesting overhead
   before the ~870 GiB sizing is treated as measured. The disk validation originally recorded in this
   entry was for the runner-up Option B (4+4) -- re-record against the ruled layout.
   **DONE 2026-07-16:** replaced by the committed `scripts/dc-dc-whole-host-budget.py`
   (`tests/dc-dc-whole-host-budget/`, 13/13). Re-run for the ruled Model B 3+2+4 shape with the
   **32 GiB containment overhead RATIFIED** (Model B: vvr1-dc0 also carries the inner OPNsense edge +
   nested libvirtd + inner-pool management + nested-KVM paging the rack-only sizing never had; the
   earlier 16 GiB figure gave 838/186) = **870 GiB / 1024 (85%), FIT, 154 GiB headroom** (per-VM
   vvr1-dc0 = 108 vCPU / 416 GiB / ~3000 GiB, `opentofu/variables.tf` `vvr1_dc0_*` defaults; vCPU
   238/256 overcommit; disk 62% thin). Ceph disk re-run for 4
   storage/DC = PASS 5.31 TiB. The lost `scratchpad/optc-calc.py` and its "reproducible" changelog
   claim are retired.

### D-121 AMENDMENT 2026-07-27 -- the ceph-osd data device is a REAL second disk on the storage nodes

**Status:** RULED 2026-07-27 (operator, GA-R5). Question as presented, verbatim:
"R1 -- Ceph OSD device. `bundle.yaml:560` sets `osd-devices: /dev/vdb`, but all 18 VR1
nodes have exactly one block device (`vda`), and `modules/node-vm` is hard-coded to a
single disk. How should this be resolved?" Options presented: (a) add an OSD volume to
`modules/node-vm` and re-apply both substrate roots; (b) re-point `osd-devices` at a
directory/partition path on `vda`; (c) re-shape the storage role. Operator selection,
exact utterance: **"Add an OSD volume to node-vm (Recommended)"**.

**The defect.** Found by the 2026-07-27 Stage-5 grounding audit, INDEPENDENTLY by two
lenses using different instruments: `virsh domblklist` on both racks, and MAAS
`physicalblockdevice_set` on all 18 nodes. Both return exactly ONE device, `vda`, on
every node. `bundle.yaml:560` reads `osd-devices: /dev/vdb   # libvirt-attached,
MAAS-untracked, wiped pre-deploy` -- a VR0 as-built comment. VR0's hosts genuinely had
an attached second disk; **the VR1 node-VM module never created one**
(`opentofu/modules/node-vm/main.tf:26` declares a single `libvirt_volume "disk"`,
`:104-125` attaches it at `target.dev = "vda"`, and neither substrate root carries an
OSD-disk variable). Left unfixed the deploy yields ZERO OSDs, `expected-osd-count: 4`
(`bundle.yaml:533`) is never met, and every storage-backed service blocks behind it.

**This was BUDGETED and never BUILT.** D-121's own capacity re-validation above already
records "Ceph disk re-run for 4 storage/DC = PASS 5.31 TiB". The second disk was
anticipated by the sizing model; only the implementation is missing. That is why this
lands as a D-121 amendment rather than a new D-number (GA-R3: architectural consequence,
but it amends a ruled node layout rather than establishing new ground).

**SCOPE IS 8 VOLUMES, NOT 18.** `ceph-osd` is placed on the four storage nodes only
(`bundle.yaml:556-557`, `num_units: 4`, `to: ["5","6","7","8"]`), so the change is 4
volumes per DC. The other five nodes per DC neither run `ceph-osd` nor need a second
device.

**Rationale for (a) over (b).** MINIMIZE DELTA TO ROOSEVELT: Roosevelt's storage nodes
have real dedicated disks, so a real second block device is what the rehearsal should
exercise. Option (b) was additionally NOT SAFE TO PICK AS PRESENTED -- whether the
`ceph-osd` charm at the pinned `squid/stable` supports a directory- or partition-backed
OSD was explicitly UNRESEARCHED (recorded as UNMEASURED by the audit's lens 5), and the
audit refused to assert it either way. Option (c) would have reopened D-121 Option C's
ruled 3/2/4 shape and its capacity budget.

**Execution preconditions (the apply is a SEPARATE operator-gated step, not authorised
by this ruling):**
1. It re-opens the substrate. All three OpenTofu roots currently plan ZERO DIFF
   (measured 2026-07-27); this deliberately spends that property on the two INNER roots.
   Re-converge both to zero diff after the apply and capture it.
2. Rack disk headroom MEASURED 2026-07-27 and sufficient: dc0 `/var/lib/libvirt` 2.0T
   available (2.9T total, 34% used -- the 949G full mirror is the bulk), dc1 2.9T
   available (1% used). Thin-provisioned qcow2, so creation cost is small.
3. The nodes are powered off in `Ready` under the READY-handoff ruling (DOCFIX-200), so
   attaching a device does not disturb a running guest -- but MAAS must RE-COMMISSION the
   four storage nodes per DC afterwards for the new device to appear in its inventory,
   and the audit did NOT verify whether re-commissioning preserves the D-134 statics and
   the pinned MACs. **Verify that before the apply, not after** -- the 2026-07-20
   MAC-regeneration incident is the precedent for exactly this class.
4. `modules/node-vm` is SHARED by both DCs. Gate the change so it is additive and
   opt-in per node role, or it will plan a change against all 18 domains.

### RULING NOTE 2026-07-27 -- D-121: the HA scale-up overlay applies only AFTER the two VIP gaps close

**Status:** RULED 2026-07-27 (operator, GA-R5). Records a ruling; amends nothing in D-121's
node layout or its 3-unit intent.

**>>> SUPERSEDED 2026-08-06 (operator, GA-R5). R6's sequencing is MOOT -- `dc-ha-scaleup.yaml`
was RETIRED. <<<** BUNDLEFIX-053 (2026-08-05) folded the overlay's ENTIRE content into
`bundle.yaml` (14 apps `num_units` 1->3 + one-per-control-node placement; 12 hacluster subs
`cluster_count` 1->3; `vault-hacluster` declared; rabbitmq `min-cluster-size: 3`; memcached
1->3; the `[vault:ha, vault-hacluster:ha]` relation) -- so applying the overlay became an
all-keys no-op deep-merge (Task-#1 post-wave review, MEASURED against the merged dc0 input).
There is no longer a separate "apply the overlay after the VIPs" step: the base bundle carries
the full HA chain, the per-DC VIP overlay supplies vault's + designate's VIPs, and a single
`juju deploy` of base+vips lands 3-unit HA directly. **Question as presented:** "dc-ha-scaleup.yaml
is now wholly redundant with bundle.yaml (BUNDLEFIX-053) -- retire it?" **Operator utterance,
exact: "Retire the redundancy and archive."** The overlay is archived at
`docs/archive/dc-ha-scaleup-RETIRED-20260806.yaml`; its five harness fixtures (T17 removed as
redundant with T18; T17b/T32/T33/T34 re-pointed onto the base HA chain, gauntlet green). The
ordering CONCERN R6 addressed -- vault/designate VIPs must exist before HA is "real" -- STILL
HOLDS and is still enforced by `provider-bundle-check` (T16b/T17b: an hacluster principal with
no VIP FAILS); it is simply no longer sequenced via a separate overlay. The record below is
retained as the history of why the two-phase sequence once existed.

Question as presented, verbatim: "R6 -- HA scale-up overlay at Stage 5. D-121's premise is
'VR1 makes HA real', but applying the overlay as it stands gives vault a 3-node pacemaker
cluster with no VIP to manage -- decorative HA for the CA that 23 relations depend on.
Eleven of twelve principals are fine; only designate and vault lack VIPs. How should this
sequence?" Options presented: (a) close the two VIP gaps first, then apply the overlay
whole; (b) apply as-is and remediate vault HA afterwards; (c) defer the overlay and scale
after Stage 6. Operator selection, exact utterance: **"Close the two VIP gaps first, then
apply the overlay whole (Recommended)"**.

**THE TENSION THIS RESOLVES.** D-121 is titled "VR1 makes HA real -- scale the decorative
single-unit control plane to 3". Option (c) would have deployed VR1 in precisely the shape
D-121 was written to retire. But option (b) would have shipped, for vault, exactly the
defect D-121 exists to remove -- a pacemaker cluster with nothing to manage. The ruled
sequence is the only one that satisfies the decision rather than half of it.

**MEASURED STATE (capture `docs/audit/r6-r15-measurements-20260727.txt`).**
`overlays/dc-ha-scaleup.yaml` scales FOURTEEN applications 1 -> 3 and moves all TWELVE base
hacluster subordinates from `cluster_count: 1` to `3`. Eleven of those twelve principals
carry proper VIP triples. The gaps are exactly two, and they are NOT the same shape:
- **designate** is in the base bundle WITH an hacluster subordinate (`cluster_count: 1`) and
  an `ha: metal-internal` binding, but no `vip`. Consistent at base scale; decorative at 3.
- **vault** has NO hacluster in the base bundle AT ALL and an EMPTY options block. Its
  entire HA apparatus -- the `vault-hacluster` APPLICATION (which the base does not
  contain), its `cluster_count: 3`, and the `vault:ha <-> vault-hacluster:ha` relation --
  exists ONLY in the overlay. And still no `vip`.

**WHY VAULT IS THE DECIDING CASE.** Twenty-three relations consume `vault:certificates`
(mysql-innodb-cluster, keystone, glance, glance-simplestreams-sync, nova-cloud-controller,
placement, neutron-api, neutron-api-plugin-ovn, ovn-central, ovn-chassis, cinder,
ceph-radosgw, openstack-dashboard, octavia, ovn-chassis-octavia, barbican, barbican-vault,
magnum, designate), plus `barbican-vault:secrets-storage <-> vault:secrets`. Vault is the
certificate authority for the entire cloud. With no VIP every consumer binds a UNIT
address, so there is nothing for a failover to move. Remediating that on a LIVE 3-unit
vault means re-pointing 23 certificate relations and re-issuing SANs on a running cloud --
which is why (b) was refused despite being faster.

**SEQUENCE RULED:** answer R11 (the VIP shape for vault and designate), land those VIPs,
THEN apply `dc-ha-scaleup.yaml` whole at Stage 5. **R11 is therefore a hard Stage-5
precondition, sequenced BEFORE the overlay**, not a parallel item.

**SCOPE NOTE that makes this tractable:** it is a TWO-APPLICATION change, not a rework.
`octavia` already carries a correct VIP triple (`10.12.4.57 10.12.8.57 10.12.12.57` in the
dc0 band), so the pattern is established in-repo and there is a shape to copy rather than
design.

**NOT RESOLVED BY THIS RULING, and tracked separately:**
- The Stage-6 radosgw multisite path is single-unit-shaped (`--unit U`, one restart, per
  `scripts/dc-dc-radosgw-multisite.sh --help`) and this ruling scales `ceph-radosgw` to 3.
  That coupling applies to any option that scales radosgw and needs its own DOCFIX.
- Whether `provider-bundle-check.py` should FAIL on an hacluster relation without a VIP, or
  on a `cluster_count` that does not match its principal's `num_units`. Today it checks
  `cluster_count` NOWHERE, which is why this survived to be found by audit rather than by
  gate. Queued under R11/R15.

**Address demand is NOT an argument against this** -- R4 already ruled the reserved bands
get created, which covers the overlay's growth from 27 to ~55 LXD units.

**Execution is a SEPARATE gated step, not authorised here.**

## D-122: VR1 site shape -- nested-per-site containment, dark fiber + dedicated per-site L3 ISP, DC edge follows the Office1 pattern [ARCH]

**Status:** ADOPTED (2026-07-15, operator ruling this session -- answers Stage-3 Ruling 2). Records
the deployment shape so Stage 3's edge/network steps stop re-deriving it. Governs
`modules/opnsense-edge` wiring for each DC and the per-site uplink networks.

**Decision (operator-ruled):**

- **Each site is nested inside its own containment VM** (`voffice1` for Office1; the `vvr1-dc0` /
  `vvr1-dc1` analogs per DC), so a "physical site down" and its recovery -- plus site-to-site tests
  -- are executable as `virsh destroy <site-vm>` against a single object. This is D-114's model,
  now affirmed as the shape for the DCs too, not just Office1.
- **Dark fiber connects DC <-> DC <-> Office** (the `mesh-*` legs) and is **East-West only** --
  replication / site-to-site, **never an internet path** (consistent with D-100 and DOCFIX-185's
  transport model).
- **Each site has its OWN dedicated L3 "ISP" uplink, not shared between sites.** Concretely: build a
  per-site WAN network on the **`office1-wan` pattern** (a dedicated NAT'd `/24` uplink segment per
  site), e.g. `vr1-dc0-wan`, and attach the DC edge's WAN there -- NOT to a mesh leg.
- **The DC OPNsense edge follows the proven Office1 edge pattern:** sizing **2048 MiB / 2 vCPU /
  nano image** (carried from the applied `office1_opnsense`, DOCFIX-189), config over the REST API
  (D-113(a2)).
- **Edge NIC model = 2-NIC (operator ruling 2026-07-15), matching the baremetal target.** The NIC
  count is shaped to the Roosevelt hardware: **nodes get 6 NICs** (one per plane, as measured on VR0
  openstack0-3), but the **edge gets 2** (WAN + one LAN) -- the six planes are **routed by the fabric**
  (the `dc-planes` segments are DELIBERATELY isolated L2; OVN/OpenStack own their L3 addressing/routing,
  not libvirt and not the edge). So the edge is NOT an inter-plane router; it is only the external
  boundary. This keeps `modules/opnsense-edge` at its existing 2-NIC shape -- **no module extension**.
  - **WAN** = the dedicated per-site ISP uplink `vr1-dc0-wan` (`172.30.2.0/24`, D-115; register in
    office1-netbox -- not yet loaded -- and build as a tofu module, promoting off the virsh-created
    `office1-wan` one-off).
  - **LAN** = **provider-public** (`10.12.4.0/22`): the edge is the upstream/external gateway the
    provider network exits through (FIP/SNAT egress + GUA injection per D-100's br-ex/provider model).
    metal-admin and the other four planes reach external via fabric (OVN) routing, not a direct edge
    leg; NTP-from-edge (D-107) is served across that fabric routing.
- **Each site runs its own MAAS controller** (as `voffice1` does), and an existing MAAS controller
  MAY be used to bootstrap a new site's deployment.

**Consequence for Stage 3:** the phase2 runbook's Ruling-1/Ruling-2 open items on edge sizing, WAN
uplink, and LAN role are CLOSED by this entry (build `vr1-dc0-wan`, wire the edge like Office1). The
remaining Stage-3 blocker is D-121 (node layout/sizing) plus the per-DC MAAS rack-controller source
(Stage 4 gap -- honest Stage-3 scope is Steps 1-8 until that is stood up). **Related:** D-114
(containment VM + MAAS-composed LXD VMs), D-100 (per-site edge, dark fiber), D-107 (headend not a
core-service provider), DOCFIX-185 (edge is a real-ISP router, not an egress airgap), D-113(a2)
(edge config over REST). DOCFIX-185's D-100/D-107 amendment note is subsumed here.

### D-122 -- AMENDMENT (2026-07-16): MAAS-controller bullet superseded (C1); single-object site-down vindicated by D-123 Model B (C3); WAN-uplink realization superseded by D-125

Two record corrections from the review (R3-F04):
- **(C1) The "Each site runs its own MAAS controller (as voffice1 does)" bullet is SUPERSEDED** by
  D-123's operator-confirmed model: ONE MAAS region on Office1 + a rack controller per DC (`vvr1-dc0`
  is that rack -- not a full region+rack). The bullet was never marked superseded and was quoted back
  as if live; corrected here.
- **(C3) The "physical site down ... virsh destroy <site-vm> against a single object" wording STANDS
  and is now literal for DCs too.** The review (under the then-inferred Model A) flagged it void for
  DCs; the operator's 2026-07-16 Model B ruling (D-123) nests the DC nodes inside `vvr1-dc0`, so a
  single `virsh destroy vvr1-dc0` IS a true DC site-down -- D-122's original intent is vindicated, not
  amended.
- **(D-125) The WAN-uplink bullet's REALIZATION is superseded by D-125** (bridge-in) -- but the
  ADDRESSING is unchanged. This entry's "WAN = the dedicated per-site ISP uplink `vr1-dc0-wan`
  (`172.30.2.0/24`, D-115)... attach the DC edge's WAN there" describes a NAT'd `/24` at the site level --
  which under D-123 Model B nesting would have no egress (OBS-3). D-125 keeps D-122's **intent** (each DC
  has its own dedicated ISP uplink; the DC edge's WAN attaches there) AND the same `/24`
  (`172.30.2.0/24`, D-115): bridge-in is single-NAT, so there is exactly ONE simulated-ISP /24 and its
  IPAM identity is UNCHANGED from Model A. Only the MECHANISM moved -- `vr1-dc0-wan` is now a libvirt
  BRIDGE (`modules/wan-bridge`) onto a **vcloud-level** NAT (`module "vr1_dc0_uplink"`, cidr `172.30.2.0/24`
  inlined) that `vvr1-dc0` bridges out to; the OPNsense WAN keeps its static `.2`. No new /24, no
  re-address, no new NetBox object. (An earlier D-125 draft wrongly said the /24 "moves to a new
  vcloud-level NAT" and "the old 172.30.2.0/24 is UNUSED" -- corrected 2026-07-16.)

### D-122 -- AMENDMENT (2026-07-19): absorbs D-129's ARCH fragment -- the Roosevelt metal-edge plugin transfer profile (GA-R3/A1 split, operator-ruled)

Per the GA-R3 one-time review ruling (sweep Batch 2 item 2.7): D-129 is class OPS (a VR1 plugin
install), but its Roosevelt transfer profile is ARCH and moves here, onto the governing site/edge-
shape decision. The profile, verbatim from D-129: **Roosevelt metal edge profile = {`os-smart`,
`os-nut`|`os-apcupsd`, `os-cpu-microcode-intel`/`-amd`, `os-lldpd`}** -- enable at the Roosevelt
edge build (inert on VR1 VMs; hardware-only concerns: disk SMART, UPS, CPU microcode, LLDP
topology). A future Roosevelt build session greps THIS decision for the edge shape and finds the
profile with it. D-129 retains the text historically (append-only); this amendment is the
governing copy.

## D-123: VR1 DC site-down DR model -- node-VM placement + the per-DC MAAS headend (resolves a D-122/D-103 tension) [ARCH]

> **SUPERSEDED 2026-08-10 by D-144** (GA-R5, DEC-01: "New D-144, supersedes D-123"). The 10.13
> redeploy ELIMINATES the Model-B `vvr1-dcN` containment layer and flattens to Option 1 (node VMs
> directly on vcloud libvirt + a per-DC `vr1-dcN-client` VM). This entry is retained as history --
> **do not build from it**; Model B remains the LIVE shape only until the redeploy executes. See D-144.

**Status:** ADOPTED **Model B** (operator ruling 2026-07-16 -- see the AMENDMENT at the end of this
entry). The earlier "ADOPTED Model A" (2026-07-15) was INFERRED from "Do it all now" (a
delivery-workflow directive, NOT an A/B answer -- Model A/B was never a presented option) and is
OVERTURNED per review R3-F01. **The RULED shape is Model B: nodes nested inside `vvr1-dc0`,
single-object `virsh destroy vvr1-dc0` site-down. The Model-A description in the rest of this line and
body is retained as historical options-analysis only.** Under Model A (OVERTURNED): nodes stay
vcloud-level; site-down = destroy
the `vr1-dc0-*` domain group; `vvr1-dc0` is a MAAS/LXD headend (the `voffice1`/`site-headend-install.sh`
pattern). **MAAS MODEL RULED (2026-07-15, operator):** ONE MAAS **region on Office1** + a **rack controller per
DC** (`vvr1-dc0` = the vr1-dc0 rack), matching buildout-design line 110. The rack serves PXE/DHCP
locally on metal-admin and PROXIES OS images from the Office1 region (so no direct DC-edge egress is
needed -- that egress worry dissolves under region+rack). `maas-vm-host` registers vcloud's virsh to the
**Office1 region** (which exists) so MAAS discovers the vr1-dc0 node domains.

**REMAINING (genuine Stage-4 design, NOT invented here) -- the rack<->region management overlay.**
`vvr1-dc0` must reach the Office1 region (10.10.0.20:5240) to enroll, and it may NOT sit on Office1's
site LAN (site isolation, D-122). Per D-100 the management path is the `office1<->dc0` mesh leg -- but
that segment has **no assigned L3** (the mesh legs are isolated L2), and the rack needs static addresses
on both the mesh (management) and metal-admin (it can't DHCP -- the planes have no libvirt DHCP, and the
rack IS the DHCP server there). So concretely standing up `vvr1-dc0` needs a **NetBox-assigned addressing
pass** for (a) the Office1-region<->DC-rack management subnet on the mesh leg and (b) the rack's
metal-admin static IP, plus a rack-controller **sizing** decision. This is a D-101-family addressing
design owned by `dc-dc-phase3`, not a one-line ruling -- the `vvr1-dc0` + `maas-vm-host` blocks stay
HELD until it lands (assigning IPs by guess would violate hard rule 2). Everything ELSE (Steps 1-8
substrate) is wired and validated.

**The tension.** D-122 records the operator's shape as *"each site nested in its own VM so we can test
physical site down"* (`virsh destroy <site-vm>` = site-down). But the ADOPTED seam D-103 + D-114 says
the OpenStack **node VMs are OpenTofu-created and MAAS merely DISCOVERS them** -- and the built tooling
implements exactly that: `modules/node-vm` creates the nodes on the **vcloud** host attached to the
`vr1_dc0_planes`, and `modules/maas-vm-host` registers **vcloud's** virsh to the DC's MAAS so MAAS
discovers those already-existing domains. D-114 scopes the containment VM to the MAAS/LXD headend + the
**non-stack** (LXD-composed) machines only -- NOT the OpenStack nodes. So for a DC, the nodes are
vcloud-level siblings of the headend VM, and "site-down = one `virsh destroy`" is literally true only
for Office1 (whose services all live inside `voffice1`), not for a DC.

**Model A (RECOMMENDED) -- nodes vcloud-level; site-down = destroy the domain GROUP.**
- Node VMs: vcloud-level libvirt domains on `vr1_dc0_planes` (as built). `vvr1-dc0` = a site headend VM
  (the proven Office1 pattern: `cloudinit-vm` like `voffice1` + `scripts/site-headend-install.sh`, which
  already names `vvr1-dc0/vvr1-dc1` as its future targets) running MAAS region+rack + LXD. `maas-vm-host`
  registers vcloud's virsh -> `vvr1-dc0`'s MAAS, which discovers the nodes.
- **Site-down DR = destroy the `vr1-dc0-*` domain group** (headend + 8 nodes + edge) as a scripted op
  (owned by `dc-dc-teardown-rollback.md`), NOT a single `virsh destroy`.
- Nesting depth = 2 (vcloud -> node -> KVM guest), the SAME as VR0-proven; nova-compute's KVM guests stay
  shallow. Reuses ALL built tooling unchanged (`node-vm`, `maas-vm-host`, `site-headend-install.sh`).
- **Closes the "Stage-4 gap":** the per-DC MAAS "rack controller" is just a `vvr1-dc0` headend VM -- an
  OpenTofu `cloudinit-vm` block shaped like `module "voffice1"` + a `site-headend-install.sh` run.
  Prepare-able now (opentofu block + the Step-9 `maas-vm-host` wiring).

**Model B -- true single-VM containment (nodes nested inside `vvr1-dc0`).**
- Site-down = one `virsh destroy vvr1-dc0` (D-122 literal). But 4-level nesting
  (vcloud -> vvr1-dc0 -> node -> guest), reshapes the planes + `node-vm` to target `vvr1-dc0`'s libvirt,
  contradicts the as-built D-103/D-114 discover-OpenTofu-created-nodes flow, and stresses nested-virt for
  nova-compute. Heaviest delta.

**Recommendation: Model A** -- least delta to the adopted decisions and the built tree, shallower
nesting, and it reuses the proven Office1 headend pattern verbatim. It only asks that D-122's DR wording
be read as "destroy the site's domain GROUP" for DCs (true single-VM destroy remains literal for
Office1). **What the operator rules:** Model A vs B (one line). **Related:** D-122 (site shape), D-114
(containment scope), D-103 (OpenTofu/MAAS seam), and `scripts/site-headend-install.sh` (the reusable
per-site headend installer).

### D-123 -- AMENDMENT (2026-07-16): ADOPTED = MODEL B (operator ruling), overturning the inferred Model A

The operator RULED **Model B** via an explicit AskUserQuestion selection ("Proceed with B, accept all"),
after being shown the consequences. This RESTORES the operator's original D-122 intent (each site nested
in its own VM; a physical-site-down test = one `virsh destroy`). The 2026-07-15 "ADOPTED Model A" status
was reconstructed from "Do it all now" (a delivery-workflow answer, not an A/B design choice) and is
OVERTURNED (review R3-F01).

**Model B (RULED):** the vr1-dc0 OpenStack node VMs are created INSIDE `vvr1-dc0`'s libvirt (not at
vcloud level). Site-down = a single `virsh destroy vvr1-dc0`. Nesting depth = 4
(vcloud -> vvr1-dc0 -> node VM -> nova KVM guest).

**Accepted cascade (operator-confirmed):**
1. **Supersedes D-103 + D-114** for DC node placement -- see the AMENDMENT on each. The nodes now live
   inside the containment VM, reversing the "OpenTofu-creates-nodes-at-vcloud + MAAS-discovers" seam and
   the "containment holds headend + non-stack LXD only, NOT the OpenStack nodes" scope.
2. **D-124 rack sizing (4/8192/80) is VOID** -- `vvr1-dc0` must contain one DC's full node fleet
   (3 control + 2 compute + 4 storage per the D-121 R-3 amendment = ~384 GiB node RAM + overhead,
   ~416 GiB VM) and set `expose_nested_virt = true`. D-124's transit-addressing (Scheme A) still stands.
3. **Depth-4 nested virt ACCEPTED** as a rehearsal cost (VR0 proved depth 2). Functional risk to the
   Phase-5 workload boot flagged and accepted; `docs/archive/model-a-fallback-plan.md` + git tag
   `model-a-fallback` preserve Model A as a low-cost revert if Model B fails to deploy.

**Substrate reshape (Phase C of the review sweep, not yet done):** retarget `modules/node-vm`, the 6
`dc-planes`, and `vr1-dc0-wan` from vcloud's libvirt to `vvr1-dc0`'s inner libvirt; resize `vvr1-dc0`;
retarget `modules/maas-vm-host`. **What still stands:** the MAAS model (ONE region on Office1 + a rack
controller per DC -- `vvr1-dc0` is that rack, now also the node host) is operator-confirmed and unaffected.

### D-123 -- AMENDMENT (2026-07-20): the Model-B virsh-POD mechanism is REFUTED by measurement; per-machine virsh power replaces it (RULED)

Model B itself (nodes inside `vvr1-dc0`; site-down = one `virsh destroy`; depth-4) is BUILT and PROVEN
-- this amendment corrects only the "vm-host to register is vvr1-dc0's OWN (inner, local
`qemu:///system`) virsh" mechanism this entry and `modules/maas-vm-host`'s header stated. Measured
during step D (2026-07-20): (1) snap MAAS cannot open `/var/run/libvirt/libvirt-sock` (confinement;
no libvirt snap interface exists), so the LOCAL pod is impossible; (2) a `qemu+ssh` pod (dedicated
operator-ruled key, reachable from both snaps) still fails -- `domblkinfo ... missing storage backend
for 'volume'` -- because MAAS virsh pods cannot read `modules/node-vm`'s pool+volume disk refs. The
pod's only job here was DISCOVERY, which PXE enlistment already achieved (9/9 enlisted, shapes exact
to D-121 Option C). **RULED 2026-07-20: per-machine `power_type=virsh`** via
`scripts/maas-node-power.sh` (MAC-matched, dry-by-default, each write verified by a live
`query-power-state`) -- the same per-node shape Roosevelt uses (IPMI/BMC per machine, no pod).
D-103 carries the matching amendment; `opentofu/vr1-dc0-maas/` (the pod root) is retained UNUSED
pending the stage-close retire-or-keep ruling (SEC-013 tied).

## D-124: the Office1-region <-> DC-rack management overlay -- addressing for the MAAS control path [ARCH]

**Status:** ADOPTED (2026-07-16, operator ruling). **Scheme A** (transit-numbered mesh) + rack sizing
**4 vCPU / 8192 MiB / 80 GiB** confirmed. The office1<->dc0 mesh leg is a point-to-point transit
(region <-> rack); metal-admin stays purely node-facing; the rack straddles both. Still to assign in
office1-netbox (the apex; operator-held token, main-loop can't query it live): the specific transit
`/30`(or `/31`) CIDR + the rack's metal-admin static IP -- registered via the D-124 importer with
operator-supplied values (no literal invented in-repo). Original options + rationale retained below.

**Why this exists (answers "why not just metal-admin space?").** The rack's NODE-facing leg IS in
metal-admin (`10.12.8.0/22`) -- that is where it serves PXE/DHCP to the vr1-dc0 nodes, correct. But the
rack must also reach the **MAAS region on Office1** (`10.10.0.20:5240`) to enroll, and:
- **D-100 rules metal-admin DC-LOCAL** -- the plane deliberately does not extend to Office1 (only the
  replication plane crosses fiber), and a RULED D-100 sub-item states *"Office1<->DC fiber carries
  management traffic only (MAAS/Juju/operator)."* So the region<->rack control path is the
  **`office1<->dc0` mesh leg**, not metal-admin.
- The rack may NOT sit on Office1's site LAN (`10.10.0.0/22`) either -- that breaks site isolation (D-122).

So `vvr1-dc0` **straddles two legs**: metal-admin (node-facing, existing space) + the office1<->dc0 mesh
(region-facing). The mesh leg is today unnumbered isolated L2; this decision numbers it. (Same applies to
office1<->dc1 for the second DC later; dc0<->dc1 stays replication-only, D-108 -- NOT management.)

**The addressing choice (operator rules A or B).** Both keep metal-admin DC-local at L2 and both need only
a tiny link; they differ in whether the rack carries a second address:
- **(A) Transit-numbered mesh (recommended).** The office1<->dc0 mesh leg gets a small point-to-point
  transit (a `/30` or `/31`): Office1 region on one end, rack on the other. The region routes to the
  rack; metal-admin stays purely node-facing. Cleanest role separation, standard inter-site practice,
  and it generalizes per-leg (office1<->dc1 gets its own transit). The rack has 2 IPs (transit + metal-admin).
- **(B) Metal-admin routed over the mesh.** No separate transit subnet; the rack's single metal-admin IP
  is advertised to the region over an unnumbered/`/31` mesh link. Fewer addresses (your instinct -- one
  rack identity), but it couples metal-admin reachability across the fiber, a soft touch on "DC-local."

**Where the numbers come from.** Like D-115/D-117, this decision rules the SCHEME; NetBox (office1-netbox,
the IPAM apex) assigns the actual transit CIDR + the rack's metal-admin IP -- no literal is invented here.
A candidate management/transit supernet (its own role, mirroring the D-115 Edge role) is a NetBox call; the
importer will register whatever is ruled. The rack's metal-admin IP lands in the D-120 static band
(`.2-.49`) of `10.12.8.0/22`, not the node band.

**Rack-controller sizing (propose to confirm).** A MAAS rack controller is lightweight (it proxies images
from the region, runs no PostgreSQL): **4 vCPU / 8192 MiB / 80 GiB** is the proposed default -- far below
voffice1's region+rack+LXD 16/32/600. Confirm or adjust.

**What the operator rules:** (1) mesh management addressing = A (transit) or B (metal-admin routed);
(2) rack sizing. Then: ratify the scheme -> NetBox assigns the transit + rack IPs (a small importer, the
D-120 pattern) -> `vvr1-dc0` (cloudinit-vm on the two legs) + `maas-vm-host` (Step 9) wire against real
addresses. **Related:** D-100 (DC-local planes + mesh = management), D-101 (address families), D-115
(Edge-role precedent for a new transit role), D-120 (static-band convention), D-123 (region+rack model),
D-107 (per-DC mirror -- why the rack needs no direct egress).

### D-124 -- AMENDMENT (2026-07-16): rack SIZING void under D-123 Model B; transit addressing stands

Under D-123 Model B (operator ruling), `vvr1-dc0` no longer holds ONLY the MAAS rack -- it now contains
the DC's entire node fleet. So the **4 vCPU / 8192 MiB / 80 GiB** rack sizing in this entry is VOID:
`vvr1-dc0` must be sized to hold 3 control + 2 compute + 4 storage (~384 GiB node RAM + 32 overhead, ~416
GiB VM) with `expose_nested_virt = true`. **What still stands:** the transit addressing (Scheme A --
office1<->dc0 mesh transit /30 + rack metal-admin IP) is unchanged; the containment VM still straddles
metal-admin (node-facing) + the transit (region-facing), now ALSO hosting the nodes.

### D-124 -- AMENDMENT (2026-07-16): transit supernet CARVED + addressing operator-ratified

The three scoping flags the `dc-rack-mgmt-import.py` importer surfaced are now RULED (operator, 2026-07-16):
- **(1) `transit` role:** seed a NEW dedicated `transit` role in office1-netbox (the existing `infra` role
  is explicitly SHARED infra, not reused) -- mirrors how D-115 seeded `edge`.
- **(2) Transit supernet: CARVE a dedicated one, NOT under Cloud.** Operator-ratified **`172.31.0.0/24`**
  (a dedicated management/transit supernet, mirroring how the Edge role got its own `172.30.0.0/16`; a /24
  holds 64 /30s -- ample for all DC<->region + DC<->DC ptp links). The importer's `CONTAINER` is pinned to
  this. **Caveat:** the block was picked against a STALE local draft; the operator carves it + the importer
  fail-loud preflight is the collision backstop against the LIVE apex.
- **(3) Transit prefix scope:** `dcim.site:vr1-dc0` ("mirror the edge" -- site-scoped like the D-115 DC-edge
  /24s).

**Ratified addressing (confirm free against the live apex before `--commit`):** transit `/30` =
`172.31.0.0/30` (region/peer `.1`, rack `.2`); rack metal-admin IP = `10.12.8.2` (first static in the D-120
`.2-.49` band). These fill the four `vr1_dc0_rack_*` tfvars: `rack_transit_ip=172.31.0.2`,
`rack_transit_prefix=30`, `rack_transit_peer_ip=172.31.0.1`, `rack_metal_admin_ip=10.12.8.2`. The NetBox
`--commit` runs ON office1-netbox (apex token local there; unreachable from the vcloud jumphost). See
`docs/archive/changelogs/changelog-20260716-d124-addressing-pin.md`.

### D-124 -- AMENDMENT (2026-07-21): vr1-dc1 addressing RATIFIED (the G12 [R] leg)

RULED 2026-07-21 (GA-R5): question as presented = "G12 addressing ruling: adopt the
pattern-mirror dc1 addressing scheme (table above, apex-confirm at authoring)?"; operator
selection, exact utterance: "Adopt scheme as derived (Recommended)". Every value below is
pattern-derived from ruled decisions (D-115 supernet, D-101 role set, this decision's transit
scheme, D-120 bands, D-131 forwarder, D-125/D-122 edge shape) and is CONFIRMED FREE against
the live apex (office1-netbox) at var-authoring time BEFORE any commit -- same discipline as
the dc0 block above.

vr1-dc1 ratified addressing (supernet 10.12.64.0/19, D-115):
- planes, CONTIGUOUS first-six /22s keeping dc0's role ORDER (dc0's 4/8/12/16/32/36 offsets
  cannot fit inside a /19 -- deliberate, documented divergence; 10.12.88+92.0/22 spare):
  provider-public 10.12.64.0/22 (edge LAN gw 10.12.64.1), metal-admin 10.12.68.0/22,
  metal-internal 10.12.72.0/22, data-tenant 10.12.76.0/22, storage 10.12.80.0/22,
  replication 10.12.84.0/22.
- region<->rack transit: 172.31.0.4/30 (region .5, rack .6) -- next /30 in this decision's
  172.31.0.0/24 transit supernet.
- rack metal-admin IP 10.12.68.2 (first static, D-120 .2-.49 band); D-131 forwarder alias
  10.12.68.3; MAAS dynamic range 10.12.68.100-.200 (D-120 inheritance).
- simulated ISP uplink: 172.30.3.0/24 (WAN .2 on edge, gw .1) -- next per-DC /24 in the
  D-115 Edge role (172.30.1 = office1, 172.30.2 = dc0).
- creds: ~/vr1-dc1-creds/vr1-dc1_svc_ed25519 (D-126 option a per-env key convention).
- standup definition-of-done carries the ruled invariants: scripts/dc-rack-net.sh site row
  (legs MEASURED at standup) + install, D-131 forwarder, MAC pinning from the FIRST apply,
  per-machine virsh power via maas-node-power.sh (pods refuted, D-103/D-123 amendments).

These fill the four vr1_dc1_rack_* tfvars when authored: rack_transit_ip=172.31.0.6,
rack_transit_prefix=30, rack_transit_peer_ip=172.31.0.5, rack_metal_admin_ip=10.12.68.2.
G12's remaining [V] leg: apex confirm-free + vars + dc1 substrate authoring + build.
lib-net.sh's vr1-dc1 selector arm stays a FAIL until the apex assignment lands (the arm's
own rule: NetBox assigns, then the literals enter the lib in the same change).

## D-125: VR1 Model B per-DC ISP egress -- bridge-in single-NAT (resolves OBS-3's design gap; egress efficacy is a deploy-time gate) [ARCH]

> **TERMINATED 2026-08-10 by D-144** (container-layer elimination). The bridge-in single-NAT existed
> only to give the `vvr1-dcN` containment VM egress (OBS-3); with the containment layer gone the DC
> edge WAN attaches directly to the per-DC `site-wan` NAT and `modules/wan-bridge` retires -- no
> successor. Retained as history. See D-144.

**Status:** ADOPTED (operator ruling 2026-07-16 -- "DC0 and DC1 are supposed to have ISP connections",
then chose bridge-in over double-NAT). Governs how each DC's simulated ISP uplink (`vr1-dc0-wan`, D-122)
egresses under D-123 **Model B**. Authored present-only; the L2-NAT path is a DEPLOY-TIME gate (see below)
-- this entry does NOT assert the egress works, only how it is wired.

**The defect this fixes (OBS-3).** `modules/site-wan` builds `vr1-dc0-wan` as a libvirt **NAT** network,
which egresses via ITS HOST's default route. Under Model A that host was **vcloud** (masquerade out vcloud's
uplink -> real egress; D-122's dedicated per-site L3 ISP satisfied). D-123 Model B moved `vr1-dc0-wan` to
the **inner** provider -- it is now created INSIDE `vvr1-dc0`, whose only outer NIC is the office1<->dc0
transit leg (East-West-only, D-122; and SEC-010 FORWARD-drops it). So the inner NAT had no egress: the DC
edge's ISP was dead. This was logged as OBS-3 in the Stage-3 review and confirmed material by the operator.

**Decision (operator-ruled): bridge-in, single NAT.** Restore independent per-DC egress by giving `vvr1-dc0`
its own **WAN uplink NIC** (a second leg -- one that never existed in Model A, where the vcloud-level edge
attached to the vcloud-level WAN directly) onto a **vcloud-level per-DC ISP NAT** network. The DC's inner
WAN segment becomes a **bridge**, not a NAT, so the inner OPNsense edge's WAN attaches through `vvr1-dc0` to
that ISP network and there is exactly ONE NAT (at vcloud = the simulated ISP hand-off). Concretely:

- **Outer (vcloud):** a new `site-wan` NAT instance `vr1-dc0-uplink` on `172.30.2.0/24` (the simulated
  ISP -- the SAME /24 D-115 assigned vr1-dc0; a RULED literal, see Addressing below), plus a second NIC on
  `vvr1-dc0` attached to it. `vvr1-dc0` NIC order
  = **[transit, uplink]** -- transit STAYS first so SEC-010/`--transit-if` keep keying on it; the uplink is
  an **IP-less bridge port** on `vvr1-dc0`.
- **`vvr1-dc0` netplan (cloud-init, declarative):** the uplink NIC (`enp2s0` -> `uplink`, IP-less)
  AND the bridge `br-vr1-dc0-wan` enslaving it are declared in `vvr1-dc0`'s cloud-init netplan (no IP
  on the `vvr1-dc0` side), so the bridge exists before the inner apply. The node-host bootstrap does
  NOT create it -- it only VERIFIES the bridge + uplink exist, grants libvirt/AppArmor access, and
  isolation-tests egress. (Declarative netplan avoids a networkd-vs-script fight and maps cleanly to
  "the DC host's netplan defines the WAN bridge to its ISP NIC" on Roosevelt.)
- **Inner (`vvr1-dc0`):** `vr1-dc0-wan` is realized by the new `modules/wan-bridge` (libvirt
  `forward = { mode = "bridge" }`, bound to `br-vr1-dc0-wan`, no subnet/NAT) instead of `site-wan`. OPNsense
  WAN then gets a static address in the **vcloud ISP `/24`** and default-routes to that `.1` gateway.

**Rejected alternative -- double-NAT (kept as the deploy-time fallback).** Inner `vr1-dc0-wan` stays a NAT
and `vvr1-dc0` masquerades it out the uplink NIC into the vcloud ISP NAT. Robust (routes/masquerades AT
`vvr1-dc0`, no reliance on nested L2) but adds a third NAT hop and lower ISP-handoff fidelity. If the
bridge-in L2 path fails at apply, revert to this -- it is the safe fallback and needs only a mode change on
the inner WAN + `ip_forward`/masquerade on `vvr1-dc0`.

**DEPLOY-TIME GATE (unprovable pre-apply -- the bridge-in equivalent of the D-114 depth-4 boot gate).**
`tofu validate` is clean, but whether **vcloud's libvirt NAT actually NATs OPNsense's foreign-MAC frames
across the doubly-nested bridge** (OPNsense MAC on `vvr1-dc0`'s uplink tap -> `br-vr1-dc0-wan` -> vcloud
`virbr`) is only knowable at apply: vcloud's `virbr` must accept the foreign source MAC (no nwfilter/
anti-spoof drop), let OPNsense ARP-resolve `.1`, and masquerade the `/24`. **Validate in isolation FIRST**
(the bootstrap prints the gate): a throwaway guest on `br-vr1-dc0-wan` must get a vcloud-ISP address and
ping out BEFORE the OPNsense chain rides on it. Failure here => revert to double-NAT (above), not a redesign.

**CONSTRAINT -- `br_netfilter`: do NOT globalize the FORWARD drop.** With `br_netfilter` loaded, bridged WAN
frames traverse the L3 `FORWARD` chain, so a **global** FORWARD policy-drop (or a blanket `iptables -P
FORWARD DROP`) would silently kill the WAN bridge. SEC-010 therefore MUST stay **interface-scoped to the
transit leg** (its current form) -- never globalized. Note `ip_forward=0` is L3-only and does NOT affect
pure L2 bridging; the two are easy to conflate. Corollary: bridge-in removes Model B's earlier reliance on
`ip_forward=1` (there is no inner WAN NAT to forward), but SEC-010's scoped FORWARD-drop is RETAINED as-is
(correct and harmless either way); see the security ledger and `references/platform-traps.md`.

**Addressing: RULED, not HELD (corrected 2026-07-16).** The vcloud ISP `/24` is the SAME `172.30.2.0/24`
D-115 already assigned vr1-dc0's edge WAN (role edge, site vr1-dc0) -- bridge-in is single-NAT, so there
is one /24 and its IPAM identity is unchanged from Model A. It is therefore a RULED literal inlined in
`module "vr1_dc0_uplink"` (cidr `172.30.2.0/24`), NOT a NetBox-assigned HELD tfvar. The OPNsense WAN keeps
its static `.2` (D-113 edge config) -- no re-address. **D-125 adds NO new NetBox object** (the `/24` is
OWNED by `netbox/dc-edge-wan-import.py` -- pending that importer's `--commit` to office1-netbox, not yet
loaded, D-115-ruled value not invented), and it introduces NO new HELD gate. (An earlier
draft of this entry wrongly modeled a `var.vr1_dc0_uplink_cidr` HELD value + an OPNsense WAN re-address;
both removed 2026-07-16.)

**Module-packaging note.** Implemented as a NEW sibling `modules/wan-bridge` rather than a `mode` flag on
`modules/site-wan` (the option preview's phrasing): same ruled topology, but it keeps every existing
`site-wan` NAT user (`office1-wan`, the new `vr1-dc0-uplink`) literally untouched and avoids inverting
`site-wan`'s NAT-specific name/doc. See the phase-C2 changelog.

**Related:** D-122 (dedicated per-site L3 ISP -- this is its Model B realization), D-123 (Model B nesting,
which created the defect), D-115 (`vr1-dc0-wan` 172.30.2.0/24 -- the old inner subnet, now the bridge),
D-113 (edge config over REST -- OPNsense WAN static addr), SEC-010 (transit FORWARD-drop, scoped),
DOCFIX-185 (edge is a real-ISP router, not an egress airgap). **Fallback:** `docs/archive/model-a-fallback-plan.md`
section 3 (revert removes the uplink NIC/network/bridge + `wan-bridge` and restores the OPNsense WAN addr).

## D-126: durable, rootless vcloud->site-service-VM access -- SSH local-forward via systemd --user (RULED: Option A) [OPS]

**Status:** RULED 2026-07-16 (operator chose Option A: "We have been using SSH proxyjump already so lets
continue that"). Tooling delivered (`scripts/site-forward.sh` + harness, reviewed-but-UNEXERCISED: this
session has no SSH key to voffice1, so the operator supplies `--key` and runs the first connect).

**Problem.** The Office1 service VMs (office1-netbox `10.10.1.10`, and later the DC service VMs) live on
`lxdbr0` (`10.10.1.0/24`), reachable from vcloud only THROUGH voffice1's LXD NAT. vcloud's default route
is the lab, so each session had to re-add `sudo ip route replace 10.10.1.0/24 via 10.10.0.20 dev virbr2`
by hand -- lost on reboot, sudo/TTY friction every time, and it will recur identically for the DCs. The
operator directed a durable, generalizable fix.

**Options weighed.**
- **A (RULED) -- SSH local-forward held by a `systemd --user` unit.** Reuses the transport we ALREADY use
  by hand (ssh through voffice1 / a DC containment VM). Rootless; survives reboot with `enable-linger`;
  `http://127.0.0.1:<port>` on the jumphost reaches the service VM. For office1-netbox the primitive is a
  plain `-L` through voffice1 (the lxdbr0 gateway); the DC form adds a `-J` ProxyJump through the
  containment VM to an inner VM -- so it generalizes by construction.
- **B (rejected) -- persist the L3 route.** Needs root (sudoers), and virbr2 is LIBVIRT-managed, so a
  netplan route races libvirt at boot (would need an `After=libvirtd` unit / net-hook, not a plain
  stanza). Divergent per-DC mechanism, and it keeps the very sudo friction we set out to remove.

**Why A wins.** Zero new mechanisms (same ssh path as the inner tofu root's `qemu+ssh`), rootless (kills
the sudo/TTY friction), and the `-J`-through-containment-VM shape is exactly how the DCs will be reached.
MEASURED substrate: user-systemd is running (`XDG_RUNTIME_DIR=/run/user/1001`), OpenSSH 9.6; `Linger=no`
today so boot-start needs a one-time `loginctl enable-linger jessea123` (printed by `install`, not run).

**Secrets boundary (unchanged by this).** The SSH key is operator-supplied via `--key` (a PATH handed to
`ssh -i`; never read into context, never baked into the tool). The forward carries ONLY read/UI TCP -- the
NetBox API token stays a host-local secret on office1-netbox (`/root/netbox-secrets/api.token`), so the
importers' dry-run AND `--commit` remain operator-run there. `127.0.0.1` bind only (never `0.0.0.0`).
SANDBOX_HOSTS already allows `localhost`/`127.0.0.1`, so an importer pointed at the forwarded port passes
its own upstream-write gate with no code change.

**Deliverables.** `scripts/site-forward.sh` ({render|install|status|remove|list}); site table seeded with
`office1-netbox` only (MEASURED: voffice1 `10.10.0.20` forwards to `10.10.1.10:8000`, HTTP 302) -- DC rows
are DEFERRED until their containment-VM + inner IPs are measured (hard rule 2; the harness fails if an
un-commented DC IP appears). `tests/site-forward/run-tests.sh` 38/38 (pins loopback-bind, no-baked-secret,
rootless, self-locate). `docs/vr1-office1-as-built.md` access table updated (D-126 preferred; sudo route
demoted to ad-hoc fallback).

**AMENDMENT (2026-07-16) -- SHELL layer (operator: "You're going to need access to the shell in Office and
the DCs").** The port-forward buys durable *reachability* to one service port, not a *shell*; the operator
needs interactive shell into the Office AND DC VMs. Same SSH transport, standard form: a generated
`~/.ssh/config.d/vr1-sites` fragment of Host aliases (`voffice1`, `office1-netbox`, `office1-tailscale`,
`office1-opnsense`) with the as-built User + `ProxyJump` baked, so `ssh office1-netbox` just works and the
`-J`-through-the-containment-VM shape generalizes to the DC VMs by construction. This is now the FOUNDATION;
the systemd forward is a convenience that can ride the aliases. Deliverable: `scripts/site-ssh-config.sh`
({render|install|remove|list}) + `tests/site-ssh-config/run-tests.sh` 30/30 (pins the as-built
host/user/ProxyJump mapping, Include idempotency + 0600 perms, no-baked-secret, no un-measured DC row). Same
secrets boundary: the key is operator-supplied via `--key` -> `IdentityFile` (a PATH, never read/baked); the
generated fragment names that path and is written LOCALLY (NOT committed). DC aliases DEFERRED until their
inner IPs are measured post-apply (only voffice1 exists today). Key confirmed one-key closed env:
`~/vr1-office1-creds/office1_svc_ed25519` (named in `creds-manifests/vr1-office1.manifest`), auths all three
users. **EXERCISED 2026-07-16** (operator ran `install`; the agent then proved access from this session):
`ssh voffice1` -> jessea123, `ssh office1-netbox` + `ssh office1-tailscale` -> ubuntu (via ProxyJump voffice1),
`ssh office1-opnsense` -> root (tcsh) all return live. This PROVES the load-bearing assumption -- the agent
can `ssh <alias>` with the key read from the config `IdentityFile` (no creds path in the command), so it
passes the guard hook + classifier. The advisor's acceptance-test steer (exercise -> record -> commit) is
met. The systemd FORWARD layer (`site-forward.sh`) remains reviewed-but-unexercised (not installed).

**PER-ENV KEY for the DCs (operator ruling 2026-07-16: "individual key... for each env access").** Office1
is one env with one key across its three VMs (already conforms: `office1_svc_ed25519` in
`~/vr1-office1-creds/`). Each DC (`vr1-dc0`, `vr1-dc1`) gets a DEDICATED access key in its OWN creds folder
(`~/vr1-dc0-creds/`, `~/vr1-dc1-creds/`) -- NOT office1's key -- for per-env blast-radius isolation (a leaked
key compromises one env, not all). Realization: the DC keys are MINTED into those creds folders at DC deploy
(pubkey wired into the DC's cloud-init), tracked in `creds-manifests/<site>.manifest`, not generated
speculatively before the DC exists. Tool mechanism: `site-ssh-config.sh` installs each DC env as its own
`~/.ssh/config.d/vr1-<dc>` fragment via a per-env `--key` (the `--env` selector is added when the first DC
row lands with MEASURED IPs).

**Applies to the CONTAINMENT VM too (operator ruling 2026-07-16, option a).** Asked whether per-env-key
covers `vvr1-dc0` itself (its cloud-init had installed the OFFICE1 pubkey, and the inner root's `qemu+ssh`
authenticated with it), the operator ruled **(a) dedicated dc0 key** over (b) office1-key-for-the-container
(zero-code). IMPLEMENTED: minted `~/vr1-dc0-creds/vr1-dc0_svc_ed25519` (600/644; `creds-audit vr1-dc0`
CLEAN); added `variable "vr1_dc0_ssh_pubkey_path"`; `vvr1-dc0`'s cloud-init `ssh_authorized_keys` now reads
`file(var.vr1_dc0_ssh_pubkey_path)` (main.tf, the vvr1_dc0 block only -- voffice1 keeps office1's key);
set the tfvar in `d124-rack.auto.tfvars`. The inner root's `qemu+ssh` to `vvr1-dc0` authenticates with the
dc0 PRIVATE half (jumphost-local, never read). Re-validated + re-planned: outer plan unchanged at 5 add /
0 change / 6 destroy (the cloud-init edit is create-only content, no new destroys/replaces).

**Related:** D-107 (Tailscale front door -- the workstation path; D-126 is the vcloud-local path), D-123
(Model B containment VMs -- what the DC `-J` form jumps through), D-124 (transit addressing the DC forwards
will ride), DOCFIX-195 (office1-netbox is the working IPAM apex this reaches). **Revert:** `git checkout`
the as-built rows + delete this entry; `git rm` `scripts/site-forward.sh`, `scripts/site-ssh-config.sh` and
their `tests/` dirs; on any host where `install` ran, `bash scripts/site-forward.sh remove office1-netbox`
(rootless) + optionally `loginctl disable-linger`, and `bash scripts/site-ssh-config.sh remove` (leaves the
Include line + `~/.ssh/config`).

**AMENDMENT (2026-07-17) -- durable BASE L3 leg (the layer this decision assumed).** D-126 made the
transport ABOVE base reachability durable (the ProxyJump aliases + the systemd `--user` forward), but it
ASSUMED the vcloud host could already reach voffice1 at `10.10.0.20`. The vcloud OS-patch+reboot (2026-07-17,
-134 -> -136) exposed that assumption: `office1-local` is an ISOLATED libvirt net (no `<ip>`/`<forward>`),
so the host's own L3 presence -- `10.10.0.10/24` on the bridge (as-built `:41`) plus the compose route
`10.10.1.0/24 via 10.10.0.20` (`:88`) -- lives NOWHERE in the libvirt XML and vanished on reboot, stranding
`ssh voffice1` (measured: `ip route get 10.10.0.20` fell through to the default lab gw; `ping .20` 100% loss).
So the base leg is the un-persisted foundation D-126 never owned. **This NARROWLY reopens the rejected Option
B -- for the BASE LEG ONLY.** The original rejection stood on a netplan-vs-libvirt race; a `Type=oneshot`
unit ordered `After=libvirtd.service` runs AFTER libvirt has created the autostart bridge, so the race does
not exist, and it is root purely because L3-on-a-bridge has no rootless equivalent (the D-126 forward cannot
create it -- it depends on it). **Deliverable:** `scripts/site-baseleg.sh` ({render|apply|check|install|
remove|list}) + `tests/site-baseleg/run-tests.sh` (24/24, offline). It DISCOVERS the bridge from the stable
net NAME (`virsh net-info office1-local`), never a baked `virbrN` (lib-net.sh PATTERN-1); `apply` is
idempotent + root-guarded and is what the boot unit invokes; `check` is the read-only session-verify a new
session runs to confirm the leg without re-deriving it. **Ordering was MEASURED, not inferred:** this host
runs monolithic `libvirtd.service` (active; `virtnetworkd`/`virtqemud` inactive), which owns network
creation, so `After=libvirtd.service` is correct here; the unit ALSO orders after `virtnetworkd.service` so
it stays correct on a modular libvirt host (Ubuntu 24.04 / libvirt 10.x may split the daemons) -- ordering
against an inactive/absent unit is a harmless no-op, and the bridge poll in `apply` is the real guarantee.
**Known limitation:** `Type=oneshot` has no `Restart=`, so the `BRIDGE_WAIT` poll (60s) is the ONLY retry --
if the bridge misses that window at a contended cold boot the leg silently does not come up; widen the wait
if observed. The `ExecStart` is the script's ABSOLUTE path, so a repo move/rename (cf. D-110) breaks the boot
unit -- re-run `install` after any relocation. **State: INSTALLED 2026-07-17; boot-race UNPROVEN.** The
`install` was run (operator sudo): the unit is written + `enabled` (symlinked into `multi-user.target.wants`),
and its `apply` ran once at install time -> `status=0/SUCCESS`, `check office1` -> ok (leg + route present).
BUT that run was POST-boot (bridge already up), so it exercised the `apply` logic, NOT the `After=libvirtd` +
poll race that is the whole point. Boot-time durability stays UNPROVEN until an ACTUAL reboot shows `ssh
voffice1` self-heals with no manual step. The ad-hoc leg from earlier this session and the unit's `apply`
converge on the same live state, so `check` alone cannot distinguish them -- only a reboot does. See D-128
(this is the vcloud-local half of the operating model). **Revert:** `bash scripts/site-baseleg.sh remove
office1` (root) on any host where installed; `git rm scripts/site-baseleg.sh` + its `tests/` dir; delete this
amendment.

## D-127: VR1 host-level VM autostart policy -- foundational service VMs autostart, DC containment VMs manual [OPS]

**Status:** RULED 2026-07-16 (operator: "set the autostart for the VMs so we don't run into problems if
the hosts restart"). Recorded before an OS-update/reboot of the vcloud host; the live `virsh autostart`
settings are applied AFTER a verified clean cold boot (operator sequencing).

**Problem.** All vcloud host-level libvirt domains were `autostart=disable`, so a host reboot leaves the
Office1 VMs down until manually started -- a footgun as the buildout adds restart events. Need a standing
policy for which VMs auto-boot with the host vs which require a deliberate manual bring-up.

**Policy (the rule of thumb).** Autostart = YES for FOUNDATIONAL service VMs with no fragile or gated boot.
MANUAL = the DC CONTAINMENT VMs and anything whose bring-up is a deliberate, gated, resource-heavy, or
ordering-sensitive act.

**Classification (host-level libvirt domains on vcloud):**
- **voffice1 -> AUTOSTART.** Office1 host (MAAS region, LXD host for netbox/tailscale, libvirt). Foundational;
  no boot fragility; must return after any host restart.
- **office1-opnsense -> AUTOSTART, but enable only AFTER one verified post-fix cold boot.** The triple-fault
  (`incident-20260712-opnsense-edge-boot-triplefault.md`) was a FIXED config bug (DOCFIX-188: `memory_unit`
  KiB->MiB starving the bootloader), not live fragility -- the running edge proves the fix; only the cold
  boot post-fix is unverified ("live boot verification is the remaining gated step"). Watch it boot once,
  then `virsh autostart`.
- **vvr1-dc0 (and future vr1-dc1 containment) -> MANUAL.** The 416 GiB/108-vCPU nested-KVM containment VM:
  bring-up is a gated act (depth-4 nested-virt boot gate, D-123 site-down semantics, resource-heavy). Do NOT
  auto-boot on host restart. Already default-off (the `cloudinit-vm` module sets `running=true`, no
  autostart) -- keep it.

**Non-host layers (so "Office1 fully returns" holds):** LXD service VMs INSIDE voffice1 (office1-netbox,
office1-tailscale) set autostart WITHIN voffice1 (`lxc config set <inst> boot.autostart true`); inner node
VMs inside vvr1-dc0 stay MAAS-power-controlled (inner libvirt autostart off); inner OPNsense edge is
manual/gated like the host edge.

**Implementation (two layers):** (1) LIVE, immediate, no guest bounce -- `virsh autostart voffice1` (and
`office1-opnsense` after its verified cold boot); runs as jessea123 via the libvirt group (no sudo). (2)
REPRODUCIBLE (IaC) -- **DONE 2026-07-17** (operator: bake the matrix into the deploy tooling so DC0/DC1 are
configured at installation, not missed). Added an `autostart` bool var (default false) to `modules/cloudinit-vm`,
`modules/opnsense-edge`, `modules/node-vm` (wired to `libvirt_domain.autostart`); set EXPLICITLY per instance
in the roots -- voffice1=true, office1-opnsense=true, vvr1-dc0=FALSE (outer); vr1-dc0-opnsense=true,
node-vm=FALSE (inner). Shared modules mean autostart is managed for EVERY instance, so IaC is now the source
of truth (omitting it would fight the live layer). `tests/opentofu-validate/run-tests.sh` T11-T15 pin the
matrix against drift (auto-booting the 416 GiB container / MAAS nodes is the exact defect guarded). VERIFIED
by `tofu plan`: the two EXISTING Office1 domains (voffice1, office1-opnsense) show `autostart false->true`
**update-in-place** (NOT replacement -- confirmed; replacement would destroy the live host/edge); vvr1-dc0
is a new create with autostart=false baked in. Outer plan is now `5 add / 2 in-place / 6 destroy`, which
collapses back to `0 change` on those two once the bookend has set them autostart=true live before the
fresh-session apply. LXD half: `boot.autostart=true` set live on office1-netbox/tailscale + captured in the
Office1 standup runbook. DC0/DC1 create NO LXD instances (inner root = node-vm + opnsense-edge + planes/pool/
wan-bridge), so the DC side is purely this VM matrix -- no DC-LXD gap.

**Sequencing for a host reboot:** `virsh autostart voffice1` (returns on its own) -> reboot -> manually
`virsh start office1-opnsense` and WATCH it cold-boot (the live-boot verification) -> then `virsh autostart
office1-opnsense`. Bring DC containment VMs up manually per their gated procedure, never on host boot.

**Related:** D-123 (Model B site-down = the reason vvr1-dc0 is manual), DOCFIX-188 (the opnsense mem-unit
fix that makes the edge safe to autostart), `runbooks/ops-restart-procedure.md` (the cloud-side analogue:
`resume-guests-state-on-host-boot`). **Revert:** `virsh autostart --disable <domain>`; drop the IaC
`autostart` args; delete this entry.

## D-128: VR1 operating model -- Claude on the vcloud jumphost; two-plane origin; workstation tailnet is the human path [OPS]

> **AMENDED 2026-08-10 by D-144** (container-layer elimination). With the `vvr1-dcN` containment VM
> eliminated, Plane 2's definition SHRINKS to MAAS / NetBox / inner-tofu; the DC substrate build
> becomes wholly Plane 1 (one flat vcloud-local `tofu` apply, no qemu+ssh substrate dial). The
> two-plane model itself stands. See D-144. (The D-138 amendment to Plane 2 is preserved.)

**Status:** RULED 2026-07-17 (operator: "I will go with your recommendation"). Recorded + folded into the
operating skill so no future session re-derives it. Governs WHERE work originates, not any built surface.

**Problem.** The vcloud patch+reboot stranded Office1 access (see D-126 amendment) and raised the operating
question: the operator SSHes into the vcloud host to run Claude, but the operator's WORKSTATION has tailnet
access (D-107) that reaches all of `10.10.0.0/22` directly. Should the session relocate -- run Claude from a
repo on the workstation and tunnel into both vcloud and Office1? We are simulating a deployment FROM Office1
to the DCs, so "sit in Office1" has real pull.

**Decision -- two planes, fixed origins; Claude stays on the jumphost.**
- **Plane 1 (substrate / stagehand):** creating the DC *host* VMs (`vvr1-dc0`), libvirt nets/pools, the
  OUTER `tofu` root (`libvirt_uri = qemu:///system`, `dc-dc-phase0.auto.tfvars:6`) + the ~13 local `virsh`
  calls. Runs ON the vcloud host. **No Roosevelt analog** -- in Roosevelt the DC hardware exists; nobody
  creates it. Throwaway simulation scaffolding.
- **Plane 2 (deployment / management / testing):** MAAS enlist/commission/deploy, the NetBox apex, the INNER
  `tofu` root (`opentofu/vr1-dc0-substrate/`, `qemu+ssh` FROM Office1 into `vvr1-dc0`, R-5), tenant tests.
  This is what we REHEARSE for Roosevelt, and it EXECUTES on `voffice1` (the headend) -- reached by `ssh
  voffice1 '...'` over the D-126 leg, so a runbook step reads "from the Office1 headend", which transfers.
- **Claude's process stays on the vcloud jumphost.** The execution hosts above are fixed BY DESIGN; Claude's
  location only picks the TRANSPORT it uses to reach them, not where work runs.
- **The operator's workstation tailnet is the HUMAN path** (MAAS UI, NetBox UI, browser, an SSH console) --
  orthogonal to where Claude runs. Keep using it; it needs no decision and touches nothing scoped.

**Why not relocate to the workstation (Option B, rejected).** (1) It trades away the standing "ONE jumphost
session reached via Remote Control" design that already solves multi-workstation hopping -- putting the
session ON a workstation ties it to that machine. (2) It relocates the entire gating/secrets contract (the
PreToolUse guard, permission `ask` rules, creds folders, CLAUDE.md) onto a less-controlled host and puts
creds there. (3) It puts the riskiest Plane-1 ops (create/destroy host VMs) over a droppable `qemu+ssh`
transport instead of local libvirt. The Office1-reach cleanliness it would buy is already delivered by the
D-126 base-leg oneshot with none of those costs. Roosevelt fidelity lives in the runbook's DECLARED ORIGIN +
transport (achieved by executing Plane-2 ON voffice1), not in which host Claude's process runs on.

**Durable Office1 reach = the D-126 base-leg oneshot (`scripts/site-baseleg.sh`), NOT tailscale-on-vcloud.**
This is precisely why **D-107 stays untouched**: tailscale is the WORKSTATION path; the jumphost's path is
its own L3 leg. DCs are outside the `/22` carve and keep the D-126 `-J`-through-`vvr1-dc0` shape regardless.

**Related:** D-126 + its 2026-07-17 amendment (the mechanism), D-107 (workstation tailnet), D-123/R-5
(inner-root-from-Office1), `docs/dc-dc-deployment-workflow.md` (Cross-cutting: the plane split as doctrine),
the operating skill (invariant). **Revert:** delete this entry + the skill invariant; the operating model has
no built-surface footprint to unwind.

## D-129: OPNsense edge plugin / add-on base profile (VR1 -> Roosevelt) [OPS]

**Status:** RULED -- all four sub-decisions ruled as of 2026-07-21 (profile adopted
2026-07-18). REMAINING EXECUTION, tracked at gate G13 (not a decision): operator-gated live
install of the VR1 profile on office1-opnsense + qga channel retrofit at that edge's next
scheduled restart. Sub-decision (iv) RULED 2026-07-21 (GA-R5): question as presented = "does
the OPNsense edge anchor site NTP, or does the MAAS NTP hierarchy remain the time authority?";
operator selection, exact utterance: "Keep MAAS hierarchy (Recommended)". No NTP role on the
edge; the vendor-default MAAS chain (nodes -> rack -> region -> upstream) stays the time
authority -- measured working (rendered dhcpd option ntp-servers = the rack,
docs/audit/d131-sub3-dhcpd-option6-20260721.txt; commissioning proved node sync). A dedicated
Roosevelt site time source, if ever wanted, rules with D-132's utility-node design. Sub-decision (iii) RULED 2026-07-21 (GA-R5, one
clarifying round): first presentation drew an operator clarification, quoted verbatim as it is
itself standing architectural context: "Roosevelt deisgn, and this deployment, will have a site
tailscale installation to access the metal-admin plane at each location. Does that change your
recommendation?" [sic]. Question as re-presented = "where does each site's Tailscale
installation live?"; operator selection, exact utterance: "Dedicated node per site
(Recommended)". The D-107 shape becomes the STANDING PER-SITE pattern: a dedicated
subnet-router (small VM or container; placement per-DC is deployment-time engineering, may
land on D-132's utility nodes) on each site's metal-admin plane. The edge gets NO
management-plane leg and NO os-tailscale; the D-131 sub-2 revisit clause therefore never
triggers. Per-site ACL scoping, key custody, and a SEC row land at implementation. D-107
remains the governing decision for the Office1 installation; this ruling extends its shape
per-site rather than amending its text. Sub-decision (ii) RULED 2026-07-21 (GA-R5):
question as presented = "how do we dispatch the os-frr dynamic-routing question for the edge?";
operator selection, exact utterance: "Pin to Roosevelt design (Recommended)". VR1 stays static
(D-124/D-125, already ruled); os-frr is PINNED to Roosevelt inter-DC network design time, to be
ruled alongside the D-100 link spec and the D-132 topology questions (the routing answer depends
on both). Trigger: Roosevelt network design session -- grep this entry + D-132 there. Sub-decision (i) RULED 2026-07-21 (GA-R5):
question as presented = "is the boundary edge (OPNsense) in-scope for COS scraping, within
D-105's per-DC observability model?"; operator selection, exact utterance: "In-scope, per-DC
(Recommended)". Each DC's COS scrapes its OWN edge; the Step-10 pin (os-node-exporter + API
backend) is therefore UNGATED and lands at its pinned deployment step; exporter listener bound
to the management plane only; D-105's per-DC failure-domain isolation preserved (no Office1
roll-up; Roosevelt revisit rides D-105's existing sub-item).
Operator directive: "install the OPNsense recommendations and pin the full engineering for the rest for
the appropriate steps in the upcoming deployment(s)". The
plugin PROFILE is adopted: `os-qemu-guest-agent` + `os-iperf` directed for the VR1 edge via the D-113
firmware API (`core/firmware/install/<pkg>`); the live install on `office1-opnsense` is operator-gated
(auto-mode classifier blocked the agent's POST -- cloud mutations stay gated), verification pending. The
rest is PINNED to its deployment step (mapping below). **FOUR sub-decisions remain OPEN** (a one-line
directive does not rule these -- they were flagged as decisions, not adds): (i) COS-scrapes-the-edge
scope (RULED 2026-07-21, see Status), (ii) `os-frr` dynamic routing (RULED 2026-07-21: pinned
to Roosevelt design, see Status), (iii) `os-tailscale` subnet-router consolidation (RULED
2026-07-21: dedicated node per site, edge excluded -- see Status), (iv) edge-as-site-NTP
(RULED 2026-07-21: MAAS hierarchy stays the time authority, no edge NTP role -- see Status).

**Pinned at deployment steps (the "engineering for the rest").**
- **VR1 edge base profile = {`os-qemu-guest-agent`, `os-iperf`}** -> applied at DC-edge bring-up.
  PINNED in `runbooks/dc-dc-phase2-tofu-dc-substrate.md` (Model B inner-apply / edge section). The
  reusable per-profile installer (thin wrapper over `scripts/opnsense-api.sh` firmware install,
  idempotent, `--check`) is itself engineering to BUILD at that step -- not built now (avoids
  premature tooling; `opnsense-api.sh` already carries the install this session).
- **Roosevelt metal edge profile = {`os-smart`, `os-nut`|`os-apcupsd`, `os-cpu-microcode-intel`/`-amd`,
  `os-lldpd`}** -> recorded here + in the review doc; enable at the Roosevelt edge build (inert on VR1 VMs).
- **Observability (`os-node_exporter`)** -> PINNED in `runbooks/dc-dc-phase6-designate-cos-magnum.md`
  Step 10, gated on the COS-scrapes-edge scope sub-decision (i).
- **Security hardening** (Suricata content, crowdsec, Zenarmor, acme, SIEM) -> DEFERRED per closed-test
  posture to post-teardown / Roosevelt; recorded in the review doc, not pinned to a VR1 step.

**Verification 2026-07-18 (install attempt on `office1-opnsense`, operator-run via firmware API).**
- `os-qemu-guest-agent`: **package INSTALLED but NON-FUNCTIONAL.** Daemon not running/enabled AND the
  libvirt domain has **no `org.qemu.guest_agent.0` virtio-serial channel** (`opnsense-edge` module defines
  none -- grep-confirmed), so `virsh ... guest-ping` returns "QEMU guest agent is not configured".
  **Making qga work is therefore NOT a plugin install -- it is: (1) inject the guest-agent channel into
  the domain XML in `modules/opnsense-edge` (provider XSLT-or-equivalent -- verify the exact dmacvicar
  mechanism when building; do NOT assume a `qemu_agent` arg exists); (2) enable the qga service on the
  edge; (3) restart the edge.** NEW DC edges get (1) at creation (born with the channel, no restart);
  `office1-opnsense` retrofits at its NEXT SCHEDULED restart. Functional enablement DEFERRED accordingly.
- `os-iperf`: **first install FAILED** -- two back-to-back `core/firmware/install` POSTs collided on the
  OPNsense firmware LOCK (the 2nd ran while the 1st held it). **Lesson for the profile-installer: install
  ONE plugin at a time and POLL `upgradestatus` to completion before the next.** Re-run of `os-iperf`
  pending (still classifier-gated for the agent -- operator-run or a Bash permission rule).

**Implementation prep (autonomous 2026-07-18; NOTHING applied to the cloud).**
- **Profile-installer BUILT: `scripts/opnsense-plugins.sh` + `tests/opnsense-plugins/` (17/17, gauntlet
  68 GREEN).** `list` + `apply <vr1-edge|metal-edge>`; installs ONE plugin at a time + polls
  `upgradestatus` (encodes the firmware-lock lesson), delegates auth to `opnsense-api.sh`, warns on the qga
  channel dependency, offline `--dry-run`. This CLOSES the "build the reusable installer" pin; live `apply`
  is operator-gated (firmware POST). Runbook pin updated to USE it.
- **qga libvirt channel (the functional gap) -- design, NOT applied.** Add to `modules/opnsense-edge` as an
  **OPT-IN variable defaulting to OFF** (e.g. `enable_qemu_agent = false`) so EXISTING edges get NO plan
  change -- CRITICAL: a default-on edit to the shared module would force a change/bounce on the LIVE
  `office1-opnsense` and dirty the imminent DC0 outer plan (5/0/6). Set true only for NEW DC-edge
  instantiations (born with the channel, no retrofit); `office1-opnsense` opts in + restarts on its own
  schedule. **MECHANISM TO VERIFY before writing (not in `references/opentofu-provider-docs.md` -- do NOT
  assume):** whether dmacvicar/libvirt **0.9.8** exposes a native `qemu_agent = true` domain arg or needs an
  `xml { xslt = ... }` channel injection. Then enable the service + restart.
- **`os-git-backup` wiring (Finding #1) NEEDS OPERATOR INPUT -- cannot be done autonomously:** (a) choose a
  target git repo (a repo on `git.baldurkeep.com` for edge-config backups); (b) the plugin generates an SSH
  keypair -- add its PUBLIC key as a DEPLOY KEY on that repo; (c) set branch + schedule + enable. Steps
  (a)/(b) are operator/GitBucket actions; the write itself is also a gated edge mutation.

**Problem.** `office1-opnsense` was updated to OPNsense 26.7 and is a near-stock install (only
`os-git-backup`, itself unconfigured). The edge is the reusable per-site boundary profile that will
land on Roosevelt baremetal, so "what should be on the base edge?" is a real, transferable question.
Full catalog review: `docs/opnsense-edge-addon-review-20260718.md` (104 plugins, all categorized).

**Recommendation (for the operator to rule, bucket by bucket).**
- **Roosevelt metal edge profile (record now, inert in VR1 VMs):** `os-smart`, `os-nut` OR
  `os-apcupsd`, `os-cpu-microcode-intel`/`-amd`, `os-lldpd`. These are the genuine base-deployment
  items; they only run on baremetal.
- **VR1 sim scaffolding (add on VR1 VM edges now; uninstall at metal, like netem):**
  `os-qemu-guest-agent` -- clean libvirt-guest coordination; aids the D-126/D-127 reboot work.
- **Buildout diagnostics:** `os-iperf` on edges to validate DC-DC transit throughput (D-122/D-124).
- **Observability -- CONDITIONAL:** `os-node_exporter` (Prometheus-native, matches the COS stack) IF
  the operator rules the boundary edge in-scope for COS scraping. D-105 scoped COS per-DC INSIDE the
  cloud; the edge is outside that failure domain -- so this is a scope decision, not an automatic add.
- **Defer per closed-test posture (Roosevelt / post-teardown):** Suricata (core) + IDS content
  plugins, `os-crowdsec`, `os-sunnyvalley` (Zenarmor), `os-maltrail`, `os-acme-client`, SIEM agents.

**Open decision points.** (1) which buckets to adopt; (2) COS-scrapes-the-edge scope (D-105); (3)
`os-frr` -- dynamic routing is unneeded for VR1's static transit, a Roosevelt-scale question; (4)
`os-tailscale` -- consolidate the separate subnet-router VM onto the edge vs keep D-107 as-is; (5)
whether the edge anchors site NTP. Items 3-5 are their own sub-rulings.

**Adjacent finding (not part of the ruling):** `os-git-backup` is installed but UNCONFIGURED -- no
config backup is running. Recommend wiring it to `git.baldurkeep.com` (config-as-code) regardless of
D-129's outcome; tracked in the review doc's Findings.

**Implementation path (once ruled).** Bake the chosen plugin set into the edge bootstrap over the
REST-API path (D-113) via the OPNsense firmware API (`/api/core/firmware/install`), with the touched
script's harness + a changelog entry with a revert. VR1 vs Roosevelt sets differ (sim scaffolding is
VR1-only; the hardware profile is metal-only).

**Related:** D-100/D-122 (edge = boundary role), D-113 (REST-API config), D-105 (COS scope), D-106
(Designate owns DNS), D-107 (Tailscale subnet-router VM), D-124/D-125 (static transit), D-127
(autostart), D-126 (base-leg). **Revert:** delete this entry + the review doc; nothing was installed.

### D-129 -- AMENDMENT (2026-07-19): class OPS; ARCH fragment split out (GA-R3/A1)

One-time-review ruling (operator, 2026-07-19): D-129 is tagged **[OPS]** (dominant purpose = the
VR1 plugin install + pins). Its ARCH fragment -- the Roosevelt metal-edge transfer profile -- is
SPLIT to a dated AMENDMENT on D-122 (the governing site/edge-shape ARCH decision), which is now
the governing copy; the text above stays as history (append-only register). The four OPEN
sub-decisions (i)-(iv) are unaffected and remain OPEN on this entry.

### D-129(iii) -- AMENDMENT (2026-08-07): per-DC Tailscale subnet-router IMPLEMENTATION rulings (a)-(d)

Sub-decision (iii) ruled the SHAPE (dedicated subnet-router node per site on metal-admin, edge
excluded). D-129(iii) itself states "per-site ACL scoping, key custody, and a SEC row land at
implementation" -- these are that implementation, ruled 2026-08-07 (GA-R5, one per exchange).
Operator directive framing them, verbatim: **"Lets plan and push to both DC0 and DC1 in this
step."** So the four rulings apply to BOTH DCs symmetrically and the build executes for both.
Standing per-DC-standup requirement (gap-register item 21; queued 2026-07-31 operator directive
"install tailscale in DC0, DC1, and any future DCs ... add it as a requirement").

- **(a) PLACEMENT + OCTET.** Question: dedicated VM vs co-locate; octet. Operator exact
  utterance: **"Dedicated VM at utility .7 (Recommended)"**. A DEDICATED subnet-router VM in the
  D-134 utility band at **`.7`** -- `10.12.8.7` (dc0) / `10.12.68.7` (dc1). Fate-separated from
  the artifact mirror (.4) and MAAS region (.6). **This EXTENDS D-134's standing octet map:
  `.4` artifact / `.5` Juju controller / `.6` MAAS region / `.7` Tailscale subnet router**;
  divergence between DCs at the same octet remains a DEFECT (see the D-134 amendment of this date).
- **(b) REACHABILITY MODEL = STAR.** Question: star (operator->DC only) vs mesh. Operator exact
  utterance: **"Star: operator->DC only (Recommended)"**. Operators reach each DC's metal-admin;
  NO DC-to-DC management paths. This IS the security boundary -- an unpoliced Headscale is
  allow-all, so the star is enforced by the Headscale ACL policy (tag:operators -> each DC
  router's metal-admin; router<->router DENY). Keeps a compromised DC's blast radius inside that
  DC, consistent with SEC-010/D-125. Cross-DC REPLICATION (rbd-mirror/radosgw) rides the
  replication plane, not metal-admin, so it is not an argument for a management mesh. **[ARCH-
  standing: a Roosevelt build session greps this before wiring operator access at region-region-
  with-N-DCs scale; mesh's O(N^2) ACL does not survive it.]**
- **(c) HA COUNT = SINGLE, HA PINNED.** Question: one router vs two (HA). Operator exact
  utterance: **"We will not be creating HA for this now. Pin HA scale up for Headscale/Tailscale."**
  ONE subnet router per DC now; two-router HA is PINNED (deferred), alongside the D-121-class HA
  scale-ups. This also discharges the Headscale-version dependency for the ruling: the >=0.18.0
  failover / >=0.29.0 HA-probing version gates only bind for two-router HA, which is not built.
- **(d) SNAT = ON now, source-IP preservation PINNED.** Question: `--snat-subnet-routes` on/off.
  Operator exact utterance: **"SNAT ON now; pin source-IP preservation (Recommended)"**. The
  router masquerades operator traffic (default on), so served metal-admin hosts need NO
  route changes and access works immediately. Per-operator source-IP attribution in DC-side audit
  logs (SNAT off + a `100.64.0.0/10` return route on the served hosts) is PINNED as a later
  hardening with the HA scale-up.

**CITATION CORRECTION (DOCFIX, gap-21 finding 2026-07-31, recorded here per append-only
discipline).** Earlier text in THIS entry cites "D-107 (Tailscale subnet-router VM)" and "D-107
remains the governing decision for the Office1 installation". **D-107 is titled "Airgap posture,
per-DC artifact mirror, and NTP (VR1)" and rules NOTHING about Tailscale** (read in full
2026-07-31). D-129(iii) is ITSELF the governing per-site Tailscale-shape decision; the Office1
installation and these per-DC installs are governed by D-129(iii), not D-107.

**IMPLEMENTATION NOTES (vendor-researched 2026-07-31, not new rulings -- verified at build).**
(1) TAGGED identity, not user identity (Headscale tag removes user-auth + key-expiry); the
EXISTING Office1 node is UNTAGGED (`AdvertiseTags: null`, measured 2026-08-07) -> a 180-day
key-expiry outage risk, a DEFECT to fix in the same pass. (2) `autoApprovers` MUST be written to
the Headscale policy BEFORE a router first advertises (does not apply retroactively). (3)
Non-overlapping metal-admin CIDRs per DC permanently (`10.12.8.0/22` dc0 / `10.12.68.0/22` dc1;
Office1 `10.10.0.0/22`) -- a HARD addressing constraint (Headscale has no 4via6). (4) Headscale
server version is DEFERRED/unmeasured (Cloudflare-fronted; operator has no access this session);
tag support (>=0.28.0) + autoApprovers (>=0.17.0) are VERIFIED EMPIRICALLY at apply time, not
assumed. **SEC row for key custody opens at key-mint time in the build.** Roosevelt analog:
per-DC operator-access VPN, dedicated router, star ACL, per-DC key custody.

### D-129(iii) -- AMENDMENT (2026-08-07 b): VR1 install is UNTAGGED (office1-mirrored); TAGGED identity + autoApprovers + star ACL DEFERRED to bare-metal

The IMPLEMENTATION NOTES above lean on a TAGGED-identity Headscale design (note 1) plus
autoApprovers (note 2) and the star ACL (ruling (b)). The operator DEFERRED all three for the
VR1 rehearsal and directed the per-DC `.7` routers be brought up exactly as the working Office1
router was. **Operator directive, verbatim (2026-08-07):** *"For now, I don't care about the
other autoapprovers or acl policy. We can pin those for the bare metal install. For now, just
follow the steps we used to bring office1 online."*

**CONSEQUENCE for the VR1 build (Roosevelt design in ruling (b)/notes 1-2 UNCHANGED -- only
deferred):**
- The VR1 `.7` routers join **UNTAGGED**, mirroring `office1-tailscale` (MEASURED 2026-08-07:
  `Tags: None`, `AdvertiseRoutes: ['10.10.0.0/22']`, `RouteAll: True`, SNAT on, control
  `https://tailscale.baldurkeep.com:443`). A plain (non-tag-scoped) pre-auth key suffices; no
  `tagOwners`/`autoApprovers`/star-ACL policy is a prerequisite for VR1.
- Route approval is **MANUAL on Headscale** (operator approves the advertised metal-admin `/22`
  after the router first advertises) -- there is no autoApprovers window to burn, which removes
  the plain-key first-advertise hazard recorded on 2026-08-07 (a).
- **ACCEPTED cost, stated not glossed:** an untagged node carries Headscale's key-expiry clock
  (the note-1 defect) and the star boundary is NOT yet enforced by ACL (an unpoliced Headscale is
  allow-all). Both are the deferred bare-metal work; they are a KNOWN VR1 exposure, not an
  oversight. The per-DC key-custody SEC row still opens at key-mint (notes above).
- **Tool encoding:** `scripts/site-tailscale.sh` defaults `TS_TAG` EMPTY (untagged) and its
  `install` omits `--advertise-tags` while adding `--accept-routes` (office1's `RouteAll`); its
  `check` skips the tag assertion when untagged. `TS_TAG=tag:subnet-router` restores the tagged
  design for the bare-metal/Roosevelt build, at which point autoApprovers + the star ACL become
  prerequisites again. A new `prep` verb installs the package (from a rack-staged `.deb`, since
  the `.7` has no external egress) and enables IP forwarding -- the load-bearing subnet-router
  property that `tailscale up --advertise-routes` warns-and-succeeds without, and that `check` now
  asserts.

**[OPS]** -- this is a phase-scoped deferral of an already-ruled design; no new D-number (GA-R3).
Roosevelt sessions grep D-129(iii): the tagged/ACL work lands THERE, and this note says so.

## D-130: cloudinit seed-volume durable fix -- ignore_changes on the staging-derived create [ARCH]

**Status:** ADOPTED 2026-07-19 (operator; GA-R5 -- question + utterance below; sweep Batch 1 item 3).

**Question as presented (2026-07-19):** "D-130 -- durable mechanism for the cloudinit seed-volume
forced replacement (recurs after every host reboot; affects voffice1 + vvr1-dc0 + vvr1-dc1). v8
CONFIRMED your contingent lean (a): ignore_changes fully suppresses the replacement (7/2/7 -> 6/2/6,
zero forces-replacement). v7 additionally showed the replace itself would not bounce a running guest.
Which mechanism do you adopt? (a) ignore_changes / (b) persistent TMPDIR staging / neither-defer."
**Operator answer (2026-07-19), exact utterance:** "(a) ignore_changes (Recommended)".

**Ruling:** `modules/cloudinit-vm`'s `libvirt_volume.seed` carries
`lifecycle { ignore_changes = [create] }`. The provider's staging ISO path (Go os.TempDir-derived,
no provider knob -- verified via `tofu providers schema -json`) is re-minted after every host
reboot; without the lifecycle guard that config-derived `create.content.url` change forces
replacement of a correct, in-use volume on every post-reboot plan (GA-F01; mechanism verified
against provider v0.9.8 source in docs/finding-20260718-voffice1-cloudinit-seed-replace.md).

**Evidence (captured, quoted per charter 1a):**
- G4-v8 `docs/audit/outer-plan-20260719-v8-ignorechanges.txt`: with the edit,
  "Plan: 6 to add, 2 to change, 6 to destroy." and ZERO forces-replacement lines
  (baseline same day, `outer-plan-20260719-baseline.txt`: "Plan: 7 to add, 2 to change,
  7 to destroy." -- identical to the 2026-07-18 capture, no drift). DC0's own seed volume
  still plans as a clean create -- the build path is unaffected.
- G4-v7 `docs/audit/throwaway-v7-20260719.txt`: defect reproduced on an isolated
  `throwaway-v7-*` domain (own scratch pool, zero live contact); the replacement applied
  UNDER the running domain completed rc=0 with NO bounce ("domid before: 3" / "domid after: 3",
  domstate running); teardown verified zero residue. Corollary recorded: the source-verified
  bounce path is the autostart/domain Update path, NOT the volume replace -- the churn's real
  hazard is plan noise plus an unscoped apply carrying the autostart diffs (which DO bounce).

**Rejected:** (b) persistent `$TMPDIR` staging + env guard -- environment-dependent, fragile
across operators/hosts; not needed on the v8 evidence. Defer/no-fix -- leaves every post-reboot
plan carrying a replace pair and keeps G7's plan-matches-evidence gate noisy.

**Scope:** all consumers of `modules/cloudinit-vm` (voffice1, vvr1-dc0, later vvr1-dc1).
**Delivery:** module edit + `tests/cloudinit-vm/run-tests.sh` harness (static lifecycle-guard
assertions; `tofu validate` when the binary is present). **Revert:** remove the lifecycle block
and the harness; the replace pair returns on the next post-reboot plan.

## D-131: node-facing DNS strategy for rack-only controllers [ARCH]

**Status:** PARTIALLY RULED. Sub-decision 1 RULED 2026-07-21 (GA-R5): question as presented =
"is the rack dnsmasq node-DNS forwarder the STANDING per-DC pattern, or a temporary measure
until the upstream MAAS agent-resolver defect is fixed?"; operator selection, exact utterance:
"Standing per-DC pattern (Recommended)". The forwarder is therefore the ruled node-DNS
mechanism for rack-only DCs: site-keyed repo-carried delivery (unit + config + install/check
script + harness, folding in rack-legs persistence per D-128 register item 20), applied to dc0
and part of every future DC standup's definition-of-done; Roosevelt default unless the LP
outcome (sub-decision 4, OPEN) rules otherwise. Sub-decision 2 RULED 2026-07-21 (GA-R5):
question as presented = "should the rack node-DNS forwarder serve metal-admin only (the
current, proven scope), or be extended to the edge / other planes?"; operator selection,
exact utterance: "Metal-admin only (Recommended)". Scope is therefore FIXED at metal-admin
only (dnsmasq bound to the single metal-admin alias; edge keeps its own WAN-side DNS;
SEC-010 untouched); revisit only if a D-129 (iii)/(iv) ruling later consolidates site
services onto the edge. Sub-decision 3 RESOLVED 2026-07-21 by measurement (dispatch ruling
GA-R5: question = "how should we dispatch the dhcpd option-6 prepend (rack IP still offered
first despite allow_dns=false)?"; operator selection, exact utterance: "Investigate now
(Recommended)"). Investigation result: the shipped 3.7.2 source (snap rev 41649,
maasserver/dhcp.py) makes the rack-IP prepend UNREACHABLE when allow_dns=false --
get_default_dns_servers() returns [] before any rack-IP logic -- and the rendered
dhcpd.conf now carries option 6 = 10.12.8.3 ALONE (operator-run capture,
docs/audit/d131-sub3-dhcpd-option6-20260721.txt). The morning prepend observation was a
STALE READ pre-Temporal-re-render. NO defect, NO second LP; nodes do not rely on SERVFAIL
failover. Sub-decision 4 remains OPEN, and now carries a PINNED ARCHITECTURAL REVIEW
(operator-directed 2026-07-21) to be executed at next-deployment design time: (a) does the
node-DNS setup conform to stack best practices; (b) security implications of the applied
metal-admin forwarder (exposure scope, cache behavior, logging, hardening); (c) whether a
vendor-documented configuration recommendation exists for the "utility nodes" DC-to-DC
traffic configuration we are utilizing. Review feeds the sub-4 Roosevelt ruling alongside
the LP outcome.

**Context (measured, 2026-07-21).** MAAS 3.7's rack-only controllers serve node DNS from the
`maas-agent` resolver, which on an internet-isolated rack SERVFAILs ALL queries -- including its
own authoritative `maas-internal` zone -- because it resolves by walking public root hints and
ignores `/etc/resolv.conf` (no on-disk config surface; Temporal-pushed). cloud-init's
commissioning `cloud-config-url` uses a `maas-internal` name, so commissioning fails silently
(node boots to login, never fetches scripts). Co-located region+rack controllers are immune
(region BIND owns node-facing DNS), which is why VR0 and Office1 never hit this. Full chain:
`docs/audit/commissioning-diag-20260721.txt`; LP report queued.

**Roosevelt relevance (the A1 test).** Roosevelt DCs are rack-only sites behind an edge, exactly
this topology. Any Roosevelt build session must decide node DNS BEFORE first commissioning.

**Interim ruling (2026-07-21, operator utterance "Rack dnsmasq forwarder (Recommended)"):**
`dc0-node-dns.service` on the rack -- dnsmasq bound to a metal-admin alias (10.12.8.3),
`no-resolv`, `server=<region BIND over transit>`; rack OUTPUT traffic only, SEC-010 untouched;
metal-admin subnet `dns_servers=10.12.8.3`, `allow_dns=false`. Live and proven (canary Ready in
~3 min). NOT yet repo-carried.

**Open sub-decisions:**
1. Adopt the forwarder as the STANDING per-DC pattern (repo-carried, site-keyed unit + module or
   script delivery, harness) vs treat as temporary until the MAAS agent defect is fixed upstream.
2. RULED 2026-07-21 (Status block above): metal-admin only.
3. RESOLVED 2026-07-21 (Status block above): stale read, no defect -- option 6 carries the
   forwarder alone post-re-render; prepend logic is source-unreachable under allow_dns=false.
4. Roosevelt: same forwarder shape, or region-DNS-over-WAN with a firewall exception, or wait
   for the upstream fix and require a minimum MAAS version. Evidence needed: LP outcome.

## D-132: PROPOSED -- Roosevelt per-DC MAAS topology: HA regions, rack-top racks, cross-site backup custody [ARCH]

**Status:** PROPOSED / OPEN 2026-07-21. Operator-pinned to the NEXT DEPLOYMENT -- no ruling
yet; present options at Roosevelt MAAS design time, BEFORE any DC's first commissioning.
Filed from the queue-pass session; the pin directive is quoted verbatim below (the pin is
an instruction to carry the questions, not a ruling on any of them -- GA-R5 applies to each
question individually when it is presented).

**Operator directive (2026-07-21, exact utterance):** "Pin the following MAAS questions and
changes to the next deployment: HA MAAS regional VMs per DC, rack top MAAS that defers back
to the site specific regional MAAS controller since there will be multiple racks at some
DCs. Regional MAAS controllers at other sites hold the backup data for other DCs in case of
a rebuild requirement."

**Questions to rule (each its own GA-R5 exchange when presented):**
1. **HA MAAS region per DC.** Move from the current single-region-per-site shape (VR1:
   one region on voffice1) to HA regional controller VMs per DC. Sub-questions: HA member
   count, postgres HA mechanism, VIP/load-balancing for the region API, and whether the
   region VMs live on utility/infra nodes per DC.
2. **Rack-top rack controllers deferring to the site region.** One rack controller per rack
   (some DCs will have multiple racks), each enrolled to the DC's OWN regional MAAS.
   Interactions to carry into the options: D-131 (the node-DNS forwarder is rack-scoped --
   composes naturally with per-rack controllers, but each rack needs its own forwarder +
   dc-rack-net delivery), the MAAS 3.7 rack-only agent-resolver defect (D-131 context; LP
   outcome may set a minimum MAAS version), and SEC-010-class transit posture per rack.
3. **Cross-site backup custody.** Each site's regional MAAS backup data held by OTHER
   sites' regional controllers for rebuild-from-loss. Sub-questions: what constitutes the
   backup set (postgres dump, images, secrets -- secrets handling is a SEC row when built),
   transport + encryption, custody rotation, and a restore REHEARSAL as definition-of-done.

**Context (measured, VR1).** dc0 today: single Office1 region + single dc0 rack over the
D-126 transit; per-machine virsh power (D-103/D-123 amendments); rack-side twin invariant
(D-131 sub-1) delivered via scripts/dc-rack-net.sh. The A1 test passes: any Roosevelt build
session must grep this before laying out MAAS -- topology decides commissioning order,
DNS/DHCP shape per rack, and backup custody across DCs.

---

## D-068 -- AMENDMENT (2026-07-23): item-1 re-scoped migration plan DRAFTED; item presentable

The 2026-07-05 amendment left item 1 ("get off EOL Vault 1.8") OPEN pending "a re-scoped
migration plan". That plan is now DRAFTED at operator direction ("Draft now", 2026-07-23
queue-pass session): `docs/D-068-vault-migration-plan-draft.md` -- an options paper that
splits the item into three separately-ruled questions (Q1 interim rehearsal posture; Q2
Roosevelt production path, four options incl. a NEW OpenBao-payload-under-reactive-charm
candidate and a NEW shrink-vault's-job candidate; Q3 triggers/review via the D-071 monthly
vehicle) with a verify-live list (V1-V5) that MUST be re-run before any Q2 ruling. Two
dated web probes (2026-07-23): no tls-certificates V1 adoption signal in the reactive
service charms; no OpenBao Juju charm exists. Nothing is adopted by the draft; item 1
remains OPEN, now PRESENTABLE -- rulings proceed one per exchange per GA-R5.

**Status:** item 1 OPEN / presentable (draft exists). Items 2-3 unchanged (RULED
2026-07-21). **Related:** D-069, D-071 (review vehicle), D-132 (same design window).

## D-068 -- AMENDMENT (2026-07-23): item 1 / Q1 RULED -- rehearsal-scoped EOL risk-acceptance, posture 1b

Question as presented (migration-plan draft section 2; presented with options 1a/1b/defer):
"D-068 Q1: what interim posture covers EOL Vault 1.8.8 for the rehearsal phases (VR0 + VR1)?"
Operator answer, exact utterances: "Q1: Yes, record it." then, on the exact recording text
(1a-vs-1b resolved to 1b consistent with the operator's Q3 answer), "Confirmed (Recommended)".

RULED: explicit risk-acceptance of EOL Vault 1.8.8, scoped to the REHEARSAL phases (VR0
testcloud + VR1 when Stage 5 brings Vault) ONLY -- rationale: no tenant data, no external
surface, metal-internal listener, operator-held unseal (D-069). Production/Roosevelt is
explicitly NOT covered -- Q2 owns it. Posture 1b: the acceptance carries a Vault 1.8.x
CVE-scan line in the D-071 monthly review (line lands via the Q3 mechanics ruling).

**Status:** Q1 RULED 2026-07-23. Item 1 remains open on Q2 (Roosevelt path) + Q3 (review
mechanics), each its own exchange. **Related:** D-069, D-071 (vehicle), the 2026-07-23
draft amendment above.

## D-068 -- AMENDMENT (2026-07-23): item 1 / Q2 RULED -- Roosevelt baselines on 1.8 + funded remediation track (structure only)

Question as presented (migration-plan draft section 3; the structural option named in the
2026-07-23 discussion): "Roosevelt baselines on 1.8 with a funded remediation track" as the
standing planning assumption. Operator answer, exact utterances: 'Q2: "Roosevelt baselines
on 1.8 with a funded remediation track" assumtion is correct.' then, on the exact recording
text, "Confirmed (Recommended)".

RULED (STRUCTURE ONLY): Roosevelt-initial deploys on Vault `1.8/stable` -- the proven
reactive stack, with the item-2 listener TLS build requirement (RULED 2026-07-21) as a
mitigating layer -- and carries a FUNDED remediation track: remediation is a committed,
resourced work item, not an open-ended hope. NOT ruled here: the remediation path (draft
options 2a wait-for-V1 / 2b OpenBao-payload fork / 2c shrink-vault's-job) and its deadline
-- both are ruled at Roosevelt Vault design time on freshly re-verified V1-V5 facts (the
2026-07-23 probes MUST NOT be cited as current then).

**Status:** Q2 structural assumption RULED 2026-07-23; path + deadline OPEN to Roosevelt
design time. **Related:** D-068 item 2 (listener TLS), D-132 (same design window), the
draft (`docs/D-068-vault-migration-plan-draft.md`).

## D-068 -- AMENDMENT (2026-07-23): item 1 / Q3 RULED -- monthly-review lines + design-time re-verify trigger

Question as presented (migration-plan draft section 4, confirmed on exact recording text):
adopt the review mechanics. Operator answer, exact utterances: "Q3 Yes, add to the monthly
review." then "Confirmed (Recommended)".

RULED: (i) TWO lines join the D-071 monthly review checklist (vehicle:
`runbooks/ops-update-procedure.md`, per D-071's adopted policy) -- a Vault 1.8.x CVE scan
(Q1 posture 1b) and a re-run of the migration-draft probes (reactive-charm
tls-certificates V1 adoption signal; OpenBao charm ecosystem + release cadence).
(ii) HARD TRIGGER: at Roosevelt Vault design time, verify items V1-V5 are RE-RUN and the
Q2 path ruling is made only on those fresh results (the 2026-07-23 probes must not be
cited as current then). (iii) If 2b (OpenBao payload fork) is later ruled the remediation
track, V3/V4 are its first funded work package, BEFORE any charm fork begins.

**Status:** Q3 RULED 2026-07-23; item 1 remains OPEN on its sole remainder -- the Q2 PATH
selection (2a/2b/2c) + deadline at Roosevelt Vault design time. Q1 ruled, Q2 structure
ruled, Q3 ruled. **Related:** D-071 (vehicle), the two amendments above.

---

## D-133: ADOPTED -- VR1 DC plane carve is FLAT per-NIC (no VLAN-103 stack); next deployment rehearses hardware-faithful NIC topology (bonds/trunks) [ARCH]

**Status:** ADOPTED 2026-07-23 (Stage 4 open, branch dc-dc-stage4-phase3-maas-deploy).
Question as presented: "How should metal-internal be carved on VR1 DC nodes? The design
text (VR0 Pattern A) says a VLAN-103 stack on the metal NIC, but the built substrate gives
each node a dedicated flat metal-internal NIC (enp3s0) on its own isolated plane network."
Options presented: flat on the dedicated NIC (recommended) / VLAN-103 stack on enp1s0 /
defer for deeper analysis. Operator asked for discussion first; the analysis below was
presented. Operator ruling, exact utterance (2026-07-23): "Fine, keep it flat but on the
next deployment we need to build the VMs with exact hardware duplicated networking so we
can work out any bonding/trunking kinks."

**Decision (two coupled parts, one exchange):**
1. **VR1 DCs carve every plane FLAT on its dedicated node NIC.** Metal-internal is an
   untagged subnet on the enp3s0 plane network (dc0: 10.12.12.0/22, dc1: 10.12.72.0/22);
   no br-metal.103 / br-internal stack exists in VR1. VID 103 is a VR0-only fact. This
   SUPERSEDES the D-052-era Pattern A VLAN-103 shape FOR VR1 DC NODES ONLY -- VR0's
   as-built carve is untouched and remains the validated tagged-VLAN precedent.
2. **PINNED TO THE NEXT DEPLOYMENT (standing requirement, not a VR1 change):** the next
   rehearsal's node VMs must duplicate the TARGET HARDWARE's exact NIC topology --
   real NIC count, bond members, VLAN trunks -- so bonding/trunking kinks are worked
   out in rehearsal, not on metal. The VR1 6-flat-NIC node-vm shape is a tagged shim
   and must NOT be carried forward once real hardware specs exist. This composes with
   D-132 (Roosevelt MAAS topology, also pinned to next-deployment design time).

**Context (measured 2026-07-23, docs/audit/stage4-discovery-20260723.txt).** VR0's
as-built table (this file, 2026-06-29 inventory) shows VLAN-sharing occurred exactly
where planes outnumbered NICs (enp1s0 provider+VIP VID 104; enp7s0 metal-admin+internal
VID 103) and every plane with its own NIC ran flat -- the VLAN was a NIC-scarcity
workaround, not a design principle. The VR1 substrate (D-121/D-123, applied both DCs)
gives each node 6 NICs, one per plane, MACs config-pinned; bundle.yaml binds by Juju
space name, which maps to MAAS subnets -- charms are agnostic to VLAN vs flat beneath
the space. Roosevelt's real carve (bonds + trunks on few NICs) is neither option here
and is re-derived from real switch/NIC inventory at Roosevelt design time; part 2 makes
that rehearsal explicit definition-of-scope for the next deployment's substrate.

**Affected surfaces (DOCFIX/delivery queued to this stage, not silently edited):**
dc-dc-buildout-design.md plane table "VID 103" + "VLAN-103 bridge stack" build lines;
dc-dc-deployment-workflow.md Stage 4 Build row; runbooks/dc-dc-phase3-maas-enlist-deploy.md
Step 5's VR0-literal carve tree; scripts/lib-net.sh vr1-dc0 arm (inherits VR0's
METAL_INTERNAL_VID=103/IFACE=br-internal via the D-101 no-op -- needs an explicit
VR1-carve comment or unset, WITH its harness, at the carve delivery). A1 test: a
Roosevelt build session greps this before designing the bond/trunk carve.

---

## D-134: ADOPTED -- VR1 DC node static addressing: per-role octet bands with headroom [ARCH]

**Status:** ADOPTED 2026-07-23 (Stage 4, branch dc-dc-stage4-phase3-maas-deploy).
Question as presented: "Adopt the per-role band plan for VR1 DC node statics (both DCs,
same octet per node across all six planes): control .10-.19, compute .20-.49, storage
.201-.254, with .4-.9 reserved for future utility nodes and .50-.99 formally reserved as
the VIP band?" Operator answer: "Adopt as recommended." (The .40-.48 flat band was
presented first and REJECTED -- operator directive, exact utterance: "Per role bands
with headroom. Give me a recommendation".)

**Decision.** On every VR1 DC plane (/22), the LAST-OCTET bands are:

**SUPERSEDED by the 2026-07-23 AMENDMENT below (DOCFIX-203) -- do NOT use these bands. The
CONTIGUOUS table under `## D-134 -- AMENDMENT (2026-07-23)` is authoritative (.4-.49 utility,
.50-.99 VIP, .100-.200 nodes [control .100-.119 / compute .120-.149 / storage .150-.200],
.201-.254 dynamic). This original is kept only as the record of what was rejected and why.**

| Band | Purpose |
|---|---|
| .1-.9 | infrastructure: .1 gateway (routed planes), .2 rack, .3 DNS forwarder (metal-admin); .4-.9 RESERVED for future per-DC utility/infra nodes (D-131 sub-4 / D-132 context) |
| .10-.19 | CONTROL nodes (10 slots) |
| .20-.49 | COMPUTE nodes (30 slots) |
| .50-.99 | VIP band, formally reserved (measured today: .50-.60, 11 VIPs, VIP_COUNT_EXPECT=11 -- headroom to ~3x for Roosevelt service growth) |
| .100-.200 | dynamic/enlistment range (D-120 inheritance; live on metal-admin) |
| .201-.254 | STORAGE nodes (54 slots -- the role most likely to sprawl; echoes D-120's high node band on the office1 compose net) |

One octet per node, IDENTICAL across all six planes (D-052 stable-octet discipline
kept). Same bands in BOTH DCs (each plane is its own L2; no cross-DC collision).
VR1 assignments: control-01..03 = .10-.12, compute-01..02 = .20-.21,
storage-01..04 = .201-.204, keyed by tofu node name (config-of-record identity via
pinned boot MAC -- NEVER by the random MAAS hostnames).

**Context (measured 2026-07-23).** Collision check against every live reservation:
bundle VIPs .50-.60 (grep of bundle.yaml), 10.12.12.4 also present in bundle (low
single digits not fully free -- bands start at .10), FIP pool 10.12.5.0-10.12.7.254
(provider third-octet space, not the .0/24 row), metal-admin dynamic .100-.200 (D-120),
rack .2 + forwarder .3 (D-131 sub-1). Supersedes nothing: VR0's .40-.43 (D-052) stays
VR0's as-built; this is the VR1+ convention. A1 test: Roosevelt greps this before
assigning any node address plan.

---

## D-134 -- AMENDMENT (2026-07-23): CONTIGUOUS band table (supersedes the original table)

**Status:** RULED 2026-07-23, same session as the original entry. The operator rejected
the original table's interleaving -- exact utterance: "I want to find carves were we are
not breaking up IP addresses with other blockes in between like what is currently layed
out. Check the reservations and carves to resuffle the IP addresses and blocks in a
better order." Movability analysis presented (pinned: .1/.2/.3 live infra, .50-.60
bundle VIP literals, FIP pool rows 1-3 on provider forcing node suffixes into row 0;
movable: the D-120-inherited dynamic range, a MAAS ip-range object, nodes powered off).
Two contiguous layouts presented; question: "Which contiguous band layout should replace
D-134's table?" Operator answer: "Option B: swap (Recommended)".

**Superseding band table (row 0 of every VR1 DC plane /22, both DCs):**

| Band | Purpose |
|---|---|
| .1-.3 | infrastructure: .1 gateway, .2 rack, .3 DNS forwarder (metal-admin) |
| .4-.49 | RESERVED: future per-DC utility/infra nodes (46 slots) |
| .50-.99 | VIP band (measured .50-.60 today; ~3x headroom) |
| .100-.200 | NODES, one contiguous run: control .100-.119, compute .120-.149, storage .150-.200 |
| .201-.254 | dynamic/enlistment range (MOVED from .100-.200; 54 leases) |

VR1 assignments: control-01..03 = .100-.102, compute-01..02 = .120-.121,
storage-01..04 = .150-.153. Stable octet per node across all six planes unchanged.

**Consequences.** The DC metal-admin dynamic ranges (dc0 10.12.8.100-200, dc1
10.12.68.100-200 -- the D-120-INHERITED copies, ruled at Step D / dc1 standup) MOVE to
.201-.254 as the carve's first gated mutation per DC, BEFORE static assignment; done
with nodes powered off (no active leases; future enlistments indifferent to range
position). D-120's own office1 compose-net ranges (10.10.1.0/24) are UNTOUCHED -- this
amendment diverges the DC planes from the compose-net band order deliberately (nodes
mid-row, dynamic high) and is the authority for that divergence.

---

## D-134 -- AMENDMENT (2026-07-29): the per-DC Juju controller takes utility-band .5, and the octet map is a STANDING cross-DC standard

**Status: RULED 2026-07-29** (operator, GA-R5).

**Question as presented.** The D-104-amendment controller VM (`<dc>-juju-01`) was built and
auto-enlisted into MAAS, but D-134's bands cover no such node: `.4-.49` is UTILITY,
`.50-.99` VIP, and `.100-.200` is NODES split into control `.100-.119` / compute
`.120-.149` / storage `.150-.200` -- all three role sub-bands being OpenStack roles a Juju
controller is not. A measured occupancy table was put to the operator (dc0 metal-admin: only
NINE addresses actually allocated; `.1` held by nothing, `.2` rack leg, `.3` D-131 DNS
forwarder, `.4` mirror/proxy, `.5-.49` entirely free and RESERVED in MAAS) with three
options: (a) `.5`, in the utility band beside the mirror/proxy -- groups by FUNCTION, but
crosses D-134's stated node/utility split by KIND, since the utility band today holds only
host-level services while the controller is a MAAS-deployed node; (b) `.103`, the first free
address in the control band -- keeps every MAAS machine inside `.100-.200` at the cost of
sitting in a band named for OpenStack control nodes; (c) a new named sub-band inside
`.100-.200`.

**Operator answer, exact utterance: "Rule .5 for the juju controller, with the other utility
nodes. These assignments will follow all DC deployments to make sure standardized
configuration is upheld through multiple datacenter stand ups."**

**RULED (1) -- the address.** `<dc>-juju-01` takes octet **.5** on every plane it is
attached to, immediately after the `.4` artifact service, inside the D-134 utility band. The
band is RESERVED in MAAS, so MAAS never auto-allocates there and a static assignment is
deliberate and collision-free. This RESOLVES the node/utility tension in favour of function:
the utility band is hereby the band for PER-DC INFRASTRUCTURE that the OpenStack nodes
consume, whether that infrastructure is host-level (the mirror) or a MAAS-managed machine
(the controller). `.100-.200` remains the band for OpenStack ROLE nodes specifically.

**RULED (2) -- and this is the load-bearing half: THE OCTET MAP IS A STANDING CROSS-DC
STANDARD.** The assignments are not per-DC choices to be re-made at each standup; they
FOLLOW ALL DC DEPLOYMENTS so that standardized configuration is upheld through multiple
datacenter stand ups. Consequences that bind future work:

  - Every DC's `juju-01` is `.5`. Every DC's artifact service is `.4`. Every DC's rack leg is
    `.2` and its node-DNS forwarder `.3`. A new DC does not get to choose.
  - A new per-DC infrastructure service takes the NEXT free utility octet and takes THE SAME
    one in every DC. Assigning it in one DC assigns it in all of them.
  - Divergence between DCs at the same octet is a DEFECT, not a local decision. This is what
    makes `dc-plane-ipam.sh check <site>` meaningful across sites rather than per-site.
  - Roosevelt analog: this is the transferable half. The rehearsal's value is that the map,
    not merely the method, carries forward -- a Roosevelt DC standup reads the same table.

**Scope note.** This amendment assigns an octet and establishes the standing rule; it does
NOT by itself create the MAAS record, the tag, or the tofu pin. Those remain separate gated
work, and the controller VM cannot be commissioned until it has a `power_type` (measured
EMPTY at enlistment -- the same state that blocked all nine role nodes on 2026-07-20).

## D-134 -- AMENDMENT (2026-08-07): the per-DC Tailscale subnet router takes utility-band .7

Extends the standing utility-band octet map by one slot, per the D-129(iii) implementation
ruling (a) of the same date (operator utterance "Dedicated VM at utility .7 (Recommended)").
**The STANDING cross-DC utility-band octet map is now: `.4` artifact service (mirror or proxy)
/ `.5` Juju controller / `.6` MAAS region / `.7` Tailscale subnet router.** Addresses:
`10.12.8.7` (dc0) / `10.12.68.7` (dc1). Divergence between DCs at the same octet remains a
DEFECT (the standing rule this map encodes). Same scope note as the `.5` amendment: this
assigns the octet and the standing rule only; the MAAS record, tag, tofu pin, VM build, and
Tailscale join are separate gated work (the per-DC Tailscale build, both DCs). A future DC
standup carves `.7` for its subnet router by this standard.

## D-134 -- AMENDMENT (2026-07-27): the bands become ENFORCED in MAAS, via a DC-aware tool; and v6 gains band discipline

**Status:** RULED 2026-07-27 (operator, GA-R5). Question as presented, verbatim: "R4 --
Reserved address bands. MAAS holds ZERO reserved ipranges; dc1 metal-admin's lowest free
span is .5-.99, exactly the D-134 utility+VIP bands, and the dc1 deploy needs ~55 LXD
addresses. No VR1 tool exists to create reservations (phase-00 refuses non-VR0), and
D-134's bands are v4-only so the newly-ruled v6 carve has no band discipline. What scope?"
Options presented: (a) build a DC-aware tool, full v4 scheme + FIP now, v6 bands after the
carve; (b) one-off API calls now, no tool; (c) minimum viable, VIP band only. Operator
selection, exact utterance: **"Build a DC-aware tool; full v4 scheme + FIP now, v6 bands
after the carve (Recommended)"**.

**THE DEFECT: D-134's bands have never existed anywhere but prose.** Measured 2026-07-27:
`maas admin ipranges read` returns exactly THREE ranges cloud-wide and ALL THREE are
`type=dynamic` -- there are **ZERO reserved ranges anywhere in this MAAS**. The band table
in the 2026-07-23 amendment above has been, since it was written, a document.

**THE COLLISION IS QUANTIFIED, not hypothetical.** MAAS's own
`subnet unreserved-ip-ranges` for dc1 metal-admin (subnet 11, `10.12.68.0/22`) reports its
free spans as:
```
10.12.68.5   - 10.12.68.99     (95 addrs)   <- the ENTIRE utility + VIP band
10.12.68.103 - 10.12.68.119    (17)         <- gaps between node statics
10.12.68.122 - 10.12.68.149    (28)
10.12.68.154 - 10.12.68.200    (47)
10.12.68.255 - 10.12.71.254    (768)
```
The LOWEST free span is `.5-.99`, i.e. precisely the `.4-.49` utility band and the whole
`.50-.99` VIP band. Against that, the dc1 bundle places **27 LXD units** in base form and
**~55** once `overlays/dc-ha-scaleup.yaml` scales 14 applications -- every one needing an
address MAAS must allocate. The VIP band is where the 33 per-DC VIPs live.
NOT ASSERTED (and deliberately so): MAAS's exact allocation ORDER. What is asserted is
that ~55 allocations will be drawn from a pool whose lowest 95 slots are the VIP band, with
nothing reserving it. Zero `10.12.*` addresses are allocated today, so nothing has collided
yet -- this is a pre-emption, not an incident.

**NO VR1 PATH EXISTS.** Only two scripts in the tree can create an iprange:
`scripts/site-headend-install.sh` (office1-only) and `scripts/phase-00-maas-standup.sh`,
which REFUSES for any non-VR0 DC -- measured at `:136`, "FAIL: PLANES table is
DC0-hardcoded ... refusing to plan another DC's scheme". That refusal is CORRECT behaviour
(it is the cross-DC mixing guard `lib-net`'s selector exists to prevent); the gap is that
nothing replaced it for VR1. D-058 remains the single authority for WHAT gets reserved;
this amendment is about the VR1 mechanism and its enforcement.

**RULED SCOPE:**
1. Ship a **site-keyed reservation tool**, following the established
   `dc-mirror.sh` / `dc-rack-net.sh` / `dc-cache-proxy.sh` pattern (`check` / `install`
   per site, harnessed, fails closed). It is needed regardless -- no VR1 path exists -- and
   it is Roosevelt-transferable, since per-DC address reservation is a real bare-metal need.
   It also gives D-134 an EXECUTABLE gate instead of prose, which is what a `check` arm
   buys: today nothing can detect that the bands are unenforced.
2. One gated pass for the **full v4 scheme**: utility `.4-.49` and VIP `.50-.99` on all 12
   DC plane subnets, plus the **FIP pool** `10.12.5.0-10.12.7.254`, whose absence
   `scripts/phase-04-network-verify.sh:100` already hard-fails on.

   **dc1's FIP POOL RULED 2026-07-27 (GA-R5) -- this clause named only dc0's.** Question as
   presented: the band pass executed for both DCs but `reserve vr1-dc1` REFUSED the FIP pool
   and exited 1, because `scripts/lib-net.sh:158` UNSETS `FIP_POOL_START`/`END` for the dc1
   arm BY DESIGN ("OpenStack-layer values ... are NOT yet ruled/measured for vr1-dc1 ... UNSET
   so any use fails loud"). Mirroring dc0's shape unasked would be an inferred value under
   hard rule 2, so R4 could not close for dc1 without a ruling. Operator answer, exact
   utterance: **"Rule dc1 FIP pool 10.12.65.0-10.12.67.254"**.
   VALIDATED BEFORE RECORDING, not assumed: the range is inside the dc1 provider-public plane
   `10.12.64.0/22` (which spans `10.12.64.0`-`10.12.67.255`); it is **767 addresses, byte-for-
   byte the same size as dc0's** `10.12.5.0-10.12.7.254`, so the two DCs are structurally
   symmetric; and it does NOT overlap the D-134 utility `.64.4-.49` or VIP `.64.50-.99` bands.
   EFFECT: `lib-net.sh`'s dc1 arm now sets `FIP_POOL_START`/`END` instead of unsetting them,
   and R4's ruled scope can close for dc1. **NOTE the neighbouring values stay UNSET** --
   `VIP_PREFIX_*`, `VIP_COUNT_EXPECT` and `KEYSTONE_VIP_DEFAULT` remain deliberately unset for
   dc1 (R9's D-119 amendment rules them GENERATED from the overlay), so this ruling narrows
   that unset list by exactly one pair and does not reopen it.
3. **v6 bands follow as a second pass with the same tool, after the R2 carve lands.** This
   is FORCED SEQUENCING, not a deferral: a reserved range cannot be created on a subnet
   that does not yet exist, and the v6 plane subnets are not in MAAS.

**NEW ARCHITECTURAL CONTENT -- v6 band discipline.** D-134's table is v4-only. R2's
dual-stack ruling makes the 33 VIPs dual-family, so the v6 subnets need equivalent
protection or the same collision reappears in the other family. This amendment establishes
that **the v6 planes inherit an equivalent band discipline**, mirroring the v4 scheme's
intent (a reserved low band for utility/VIP use, node statics above it). The exact v6 octet
mapping is NOT ruled here -- it lands with the carve, and mirroring the v4 host-part layout
is the obvious mechanical default rather than a fresh design question.

**IPv6 HOST-PART CONVENTION RULED 2026-07-27 (GA-R5) -- this closes the mapping the
paragraph above left open.** Question as presented: the v6 host part is unruled and the apex
carries every v6 PREFIX but ZERO per-address v6 objects, so nothing can be populated without
it. Mirroring the v4 octet is well-defined for the ULA metal legs, which have one `/64` per
plane shared with node statics -- but is genuinely ambiguous for the GUA provider leg, which
has a DEDICATED VIP `/64` (`2602:f3e2:f02:11::/64` dc0, `f03:11::/64` dc1) separate from its
node `/64` at `:10::/64`: does keystone take `f02:11::50`, mirroring the octet, or
`f02:11::1`, first-in-a-dedicated-block? Operator answer, exact utterance: **"Rule D-136
option D, mirror the v4 octet for both legs but make a note to review at the end of the
project to review for adjustment."** (The first clause of that utterance rules D-136; this
entry records the second.)

**EFFECT: the v4 octet is mirrored on BOTH legs -- ULA and GUA alike.** So an application at
v4 octet `X` takes host part `::X` in every v6 plane:

| app | octet | dc0 v6 triple (provider GUA / metal-admin ULA / metal-internal ULA) |
|---|---|---|
| keystone | 50 | `2602:f3e2:f02:11::50` `fd50:840e:74e2:220::50` `fd50:840e:74e2:221::50` |
| ceph-radosgw | 60 | `...f02:11::60` `...:220::60` `...:221::60` |
| vault | 61 | `...f02:11::61` `...:220::61` `...:221::61` |
| designate | 62 | `...f02:11::62` `...:220::62` `...:221::62` |

dc1 is the same host parts under its own prefixes (`2602:f3e2:f03:11::/64`,
`fd50:840e:74e2:320::/64`, `:321::/64`). **Rationale as ruled: ONE rule for every plane.**
The alternative -- octet-mirroring on ULA and first-in-block on GUA -- would have made the
renderer carry a per-leg special case and made every VIP's host part unpredictable from its
v4 octet, which is the property that makes the mapping checkable by eye and by gate.
**Accepted cost, and the reason for the review below:** on the GUA leg the dedicated VIP
`/64` is then used from `::50` upward rather than from `::1`, so `::1`-`::49` sit unused in a
block that has no node statics competing for them. That is deliberate consistency, not an
oversight.

**NODE IPv6 ACQUISITION RULED 2026-07-27 (GA-R5).** Question as presented: with the v6
plane subnets now carved in MAAS, how do NODES actually acquire a v6 address -- (a) MAAS
STATIC assignment at deploy, which needs only the subnet and is already satisfied;
(b) DHCPv6, which needs a v6 dynamic range and `dhcpd6`, both absent by design; or
(c) SLAAC/RA, which needs a router advertising on each plane and would make the rack a
router. The three differ materially and only (a) is already satisfied. Operator answer,
exact utterance: **"MAAS static assignment is correct"**.

**EFFECT, and it CLOSES rather than opens work.** (1) The v6 carve already applied IS
SUFFICIENT for node addressing -- nothing further is required in MAAS. (2) **The
rack-bridge v6 legs are NOT needed**, which resolves the U17 sweep item: its finding that
"the rack bridges need v6 too" was measured on the DATA PATH, but with static assignment
nothing consumes a rack v6 leg, so `dc-rack-net.sh`'s `LEGS` table stays v4-only and is
NOT extended. (3) `dhcpd6` staying **off** is CORRECT, not a gap -- MAAS starts it only
for a v6 dynamic range, and under this ruling none should exist. (4) No plane needs a
router advertising RAs, so the DC edges take no new role. This mirrors how v4 node
addressing already works here (D-134 statics, one octet per node), which is why it is the
low-delta answer.

**VERIFICATION OWED AT STAGE-5 FIRST BOOT, not assertable now.** Every node is powered off
in `Ready`, so that MAAS actually assigns the v6 statics cannot be observed until first
boot -- the same one-time window gate **G17** already exists for. PROPOSED (not adopted
here, since amending a gate row is the operator's): fold a fourth assertion into G17 --
that a booted node carries a global v6 address from its plane's `/64` in the ruled
`::100`-`::200` node band. Recorded so the window is not missed; G17's current three
assertions are unchanged by this entry.

**REVIEW OWED AT PROJECT CLOSE, by the same ruling** ("make a note to review at the end of
the project to review for adjustment"): revisit whether the GUA leg should instead pack from
`::1`, once the cloud is live and the v6 behaviour has been observed. Recorded as forward
item **F4** in `docs/dc-dc-deployment-workflow.md`, whose trigger is the END OF THIS
DEPLOYMENT -- not the next one, unlike F1-F3. Adjusting it later is a renderer input change
plus a re-render, not a redesign, precisely because the convention is uniform.

**Execution is a SEPARATE gated step, not authorised by this amendment.** Standard delivery
discipline applies: the tool ships with its `tests/<name>/run-tests.sh` green, gauntlet ALL
GREEN, repo-lint 0-fail, and a changelog entry carrying a revert. The reservation pass
itself is an operator-gated live MAAS mutation across 12 subnets and is not batched with
the tool's delivery.

### ANNOTATION 2026-08-06 (Roosevelt-delta OBSERVATION, operator-requested; NOT a ruling)

**What this band scheme pins, and what it deliberately does not: LXD CONTAINER addresses
are auto-picked, not carved.** D-134 pins two address classes to specific, plane-aligned
octets -- NODE statics (per-role bands, e.g. control .100-.102) and the VIP band (.50-.99,
reserved so MAAS never auto-allocates into it; the bundle/hacluster own it, `lib-net`'s
matching-octet rule keeps a VIP's host-part identical across planes). The OpenStack control
plane, however, runs in Juju-provisioned LXD containers, and those container addresses are
NOT in either pinned class.

MEASURED 2026-08-06 (dc0, via the region API): the container `juju-d4f45a-0-lxd-5` (a
designate unit) is a MAAS **device** whose three plane interfaces are each `mode=static`
(fixed, not DHCP) -- but the addresses sit in the **.100-.200 auto pool** (outside the
`.4-.49`+`.50-.99` reserved bands and the `.201-.254` metal-admin dynamic/DHCP range) and
were **auto-picked by MAAS independently per subnet**: `10.12.4.141 / 10.12.8.198 /
10.12.12.110` -- three unrelated host-parts. Consequence: **container octets float across
planes and are not guaranteed stable across a full teardown/redeploy** (MAAS re-allocates
next-free from the pool), whereas node and VIP octets survive a rebuild because they are
explicitly pinned. This is correct and harmless for VR1: clients use the VIPs (the catalog
endpoints), haproxy proxies to whatever the backend addresses are, and the vault cert is
reissued to cover whatever a unit holds (observed live when BUNDLEFIX-056 rebound designate
-- the new provider-public leg was picked up automatically).

**Roosevelt-delta:** a bare-metal build that wants PREDICTABLE / pinned control-plane
container addressing (the property nodes and VIPs already have here) would add a
per-container-per-plane MAAS static RESERVATION -- a "container carve" analogous to the node
carve this decision enforces. VR1 deliberately does not do this: it pins nodes (D-134) and
VIPs (`lib-net`) and lets MAAS auto-pick container statics. Whether Roosevelt should pin them
is the open question; nothing in VR1 depends on the answer. GA-R3: OBSERVATION, no new
D-number -- annotated onto D-134 (the decision a Roosevelt build session greps for the
addressing scheme). Evidence: this session's phase-03 core-verify thread; MAAS `ipranges` +
device interface `mode` captured live, no secrets.

## D-134 -- AMENDMENT (2026-08-07): MAAS HOSTNAME naming convention -- machines carry their vr1-<dc>-<role>-NN name, set after enlistment

**Status: RULED 2026-08-07** (operator, GA-R5). After a session left the dc0 Tailscale `.7`
VM at its random MAAS name (`known-marten`) and rationalised the skip off the juju
controller's own un-renamed state (a DEVIATION used as precedent), the operator corrected --
exact utterances: **"You are not following naming conventions."** and, asked to make it
standing, **"Record that as the preferred naming convention going forward."**

**The convention (per-DC node IDENTITY standard, alongside this decision's octet map):** every
VR1 DC machine's MAAS hostname is SET to its convention name `vr1-<dc>-<role>-NN`, which equals
its libvirt domain name and its `power_id` (`scripts/lib-hosts.sh:26`). MAAS assigns a RANDOM
name that is re-minted on every (re-)enlistment -- which is exactly why the carve/power tooling
resolves machines by pinned BOOT MAC and never by name (`lib-hosts.sh:110-112`) -- but "random
by default" is NOT "leave it random": the role is read off the name in `juju status`,
dashboards and MAAS views, so the standing practice is to rename to convention after
commission/deploy. **Definition-of-done for a DC standup: zero non-`vr1-<dc>-*` hostnames in
that DC's region** (`maas <region> machines read`).

**Applied 2026-08-07 (dc0):** `known-marten`->`vr1-dc0-tailscale-01`,
`subtle-grouse`->`vr1-dc0-juju-01`; the 9 role nodes already carried convention names (all 11
now `vr1-dc0-*`). dc1's nodes (still random-named in the Office1 region) get their `vr1-dc1-*`
names when they are REBUILT FRESH into `vr1-dc1-region` (operator ruling "Rebuild fresh into
dc1-region", 2026-08-07).

**Roosevelt-delta (the A1 test):** every DC standup names its machines to convention; a
Roosevelt build session greps this before treating a random MAAS name as acceptable. [ARCH],
onto D-134 (the per-DC identity standard) rather than a new number -- naming is the sibling of
the octet map this decision already governs.

## D-134 -- AMENDMENT (2026-08-10): utility-band .8 = the per-DC container-elim client VM (vr1-dcN-client)

**Status: RULED 2026-08-10 (GA-R5)** as DEC-13 of the D-144 container-layer-elimination ruling.
Question as presented: whether the new per-DC `vr1-dcN-client` VM (D-144 -- inherits the D-138
cloud-facing client role once the containment VM is eliminated) takes the next-free utility octet or
a different one. **Operator selection, exact utterance: ".8 / vr1-dcN-client".** The standing cross-DC
utility-band map GAINS `.8`: `.4` artifact / `.5` juju / `.6` MAAS region / `.7` tailscale / **`.8`
client**. Reserved at EVERY DC per this map's standing-cross-DC-standard clause (divergence at the
same octet is a defect). Under D-143 octet-preserving addressing it reads `10.13.8.8` (dc0) /
`10.13.68.8` (dc1) on metal-admin. Governing ruling: D-144. The client VM is a flat non-hypervisor
sibling (`expose_nested_virt=false`) -- the name deliberately avoids the eliminated `vvr1-dcN` class.

## D-135: ADOPTED (AMENDED 2026-07-24) -- VR1 per-DC artifact mirror realization (D-107 build-out, staged) [ARCH]

**Status:** ADOPTED 2026-07-23 (Stage 4, branch dc-dc-stage4-phase3-maas-deploy). Two
GA-R5 exchanges, both quoted. (1) The Stage-4 mirror gate: a rehearsal-scoped D-107
exception was presented as Recommended and REJECTED -- operator answer: "Build the
mirror now". (2) This staged plan: question "Adopt the D-135 staged mirror plan
(rack-host placement at the utility .4 address per DC) and build item 1 -- the apt +
Ubuntu Cloud Archive mirror, debmirror+nginx, both DCs -- now?"; operator answer:
"Adopt + build item 1 (Recommended)".

**AMENDED 2026-07-24 (GA-R5, third exchange -- Stage 4): the DC1 apt path.**
Question: "DC1 is blocked waiting on its full ~950 GiB debmirror (~35% done);
adopt a caching proxy (apt-cacher-ng) on DC1 as the INTERIM apt path to unblock
the DC1 build now, keeping DC0 on the full mirror?" Operator answer, exact
utterance: "We will use it on DC1 to keep the build moving forward. We will
replace DC1 cache with a full mirror afterwards and test the full mirror
deployment on DC0." CONSEQUENCE: DC1 = interim caching proxy
(`scripts/dc-cache-proxy.sh`, apt-cacher-ng proxy mode at the utility .4:3142,
consumed via `juju model-config apt-http-proxy`), to be REPLACED by a full
mirror afterward; DC0 stays the full-mirror deployment under test. Both
artifact-delivery models thus get exercised this rehearsal -- direct Roosevelt
input for the D-132/D-133 utility-node design. SCOPE: the proxy caches apt only
(archive/security/UCA); charmhub is controller-side (Office1), snaps ride the
still-open edge egress or await D-135 item 3 (egress narrowing remains the
deferred closing mutation). The paused DC1 debmirror (~330 GiB) is KEPT dormant,
not deleted, as the fallback for the later full-mirror completion.

**Decision.** D-107's per-DC mirror is BUILT (not excepted) for VR1, staged by what
Stage 5 consumes:
- **Placement (all classes):** ON each DC's rack host (vvr1-dc0 / vvr1-dc1; measured
  2026-07-23: 2.9T disk, ~9G used, ~400G free RAM each), served at the FIRST D-134
  utility-band address on metal-admin: dc0 10.12.8.4, dc1 10.12.68.4 (alias on the
  metal-admin bridge, forwarder-.3 pattern; bridge DISCOVERED from the libvirt net
  name, never a baked virbrN). Repo-carried, site-keyed `scripts/dc-mirror.sh` +
  harness; part of DC-standup definition-of-done from here on. Roosevelt analog: the
  same service on per-DC utility nodes (D-132/D-133 pinned set).
- **Item 1 (BUILT NOW): apt + UCA.** Full jammy amd64 BINARY mirror
  (main/restricted/universe/multiverse; jammy + jammy-updates + jammy-security; no
  sources) plus Ubuntu Cloud Archive jammy-updates/caracal, synced per-DC through
  that DC's OWN edge (D-107 sub-item: independent upstream pull, confirmed), served
  by nginx. Stage-5 wiring: `juju model-config apt-mirror=` per DC model -- MAAS
  global config and office1 untouched.
- **Item 2 (PINNED to Stage 5 open): images/simplestreams** (sstream-mirror; Juju
  container-image metadata + glance-simplestreams-sync mirror config; amphora stays
  locally built via octavia-diskimage-retrofit -- no mirror class).
- **Item 3 (measure-then-rule at Stage 5 start): charmhub + snaps.** NOT designed
  here; first deploy measures what the controller/nodes actually fetch, then options
  (local charm files via `juju download` / caching proxy / snap-store-proxy) get
  their own ruling.
- **Egress narrowing is the CLOSING mutation:** D-107's edge scope (NTP + mirror
  sync ONLY; general node egress killed) is applied AFTER the mirror proves out,
  not before.
- **NTP conflict resolved:** D-107's nodes-chrony-from-edge is SUPERSEDED by the
  later D-129(iv) ruling (MAAS hierarchy stays the time authority); the edge remains
  the SITE's upstream sync point.

**AMENDMENT 2026-07-27 -- the two DCs test two STRATEGIES; dc1's full mirror is REMOVED,
not paused.** The 2026-07-24 amendment recorded dc1's debmirror as "PAUSED ... kept dormant
as the fallback" and said "DC1's proxy is REPLACED by a full mirror afterward", i.e. a
temporary state converging on two full mirrors. That is SUPERSEDED. Operator statement,
exact utterance (2026-07-27): **"DC0 is the test of a full mirror. DC1 is the test of the
mirror proxy. DC1 needed the partially created full mirror and associated
script/files/settings removed."** So the per-DC split is the DELIBERATE EXPERIMENT, not an
interim: dc0 = full mirror, dc1 = caching proxy, each proven on its own DC. There is no
fallback and no later convergence. EXECUTED the same day: dc1's 330G partial mirror, its
units, timer, sync helper, nginx vhost, resolved drop-in and the `debmirror` package are all
removed (capture `docs/audit/dc1-mirror-teardown-20260727.txt`).

**Consequence that had to be fixed FIRST (a measured coupling trap).** `dc-cache-proxy.sh`
did not own its utility net layer -- it REUSED `dc-mirror.sh`'s `<site>-mirror-net.service`
and refused to install without it. On dc1, whose ruled path is the PROXY, three files named
`dc1-mirror-*` (net unit, apply helper, resolved DNS drop-in) were therefore load-bearing
FOR THE PROXY. Two consequences, both real: the recorded mirror-removal procedure would have
silently killed the proxy (and with it the Stage-5 dc1 apt path), and the proxy could not be
REBUILT after such a teardown, because nothing but `dc-mirror.sh install` created the net
layer -- so rebuilding a proxy meant reinstalling the whole mirror apparatus including an
ENABLED daily debmirror timer. Teardown-and-rebuild did not converge. The net layer now
lives in `dc-cache-proxy.sh` as `<site>-cache-proxy-net.service` (+ apply helper + its own
resolved drop-in), neutrally named so a proxy-only DC carries NO mirror-named dependency;
the two net units are safe to coexist where a site runs both (idempotent `ip addr replace` /
`ip route replace` on identical MEASURED values), so dc0 is unaffected. Proven by removing
the live address and route outright and applying the proxy-owned unit ALONE. **Roosevelt: a
DC's artifact-delivery strategy must be changeable without rebuilding the other strategy's
apparatus -- per-DC utility-node services own their own network prerequisites.**

**A1 test:** Roosevelt greps this for the mirror service shape, the utility-address
convention, the staged class order, the measure-then-rule posture on charmhub/snaps, and
the per-DC-strategy rule that a utility service owns its own net prerequisites.

### D-135 -- AMENDMENT (2026-07-31): the apt mirror carries `jammy-backports`

**Status:** RULED 2026-07-31 (operator, GA-R5). **OPS under GA-R3** -- a scope correction to
an already-adopted mirror, doubt resolves DOWN, no new D-number. D-135's per-DC artifact
strategy (dc0 = full mirror, dc1 = caching proxy) is UNCHANGED.

Question as presented: bundle deploy attempt 3 succeeded, then 22 of 33 units failed their
`install` hook on a single cause, MEASURED end to end --
`E: The repository 'http://10.12.8.4/ubuntu jammy-backports Release' does not have a Release
file.` The mirror serves `jammy` / `jammy-security` / `jammy-updates` **200** and
`jammy-backports` **404**, which is exactly the "jammy triple" scope
`scripts/dc-mirror.sh:179` was built with, while every deployed node's
`/etc/apt/sources.list` carries `deb http://10.12.8.4/ubuntu jammy-backports main restricted
universe multiverse` -- because the Step-3.5 `apt-mirror` model-config rewrites the stock
Ubuntu sources to point EVERY suite at the DC mirror, including one the mirror was never
built to hold. Options put: (a) add `jammy-backports` to the mirror and re-sync;
(b) remove backports from the nodes' sources.

**Operator answer, exact utterance: "Sync the backports into the mirror".**

**Effect.** `--dist=` in the generated `<site>-mirror-sync` gains `jammy-backports`, and the
mirror's declared scope becomes the jammy triple PLUS backports. **Fixed at SOURCE**
(`scripts/dc-mirror.sh`), not only on the live rack, so a reinstall or a future DC standup
cannot silently regenerate the old scope. `tests/dc-mirror` T10 asserted the old dist string
verbatim and is RE-POINTED to the new invariant rather than deleted -- the standing rule that
a test asserting a literal finding gets re-pointed, never removed to go green.

**Cost, measured before the ruling was put:** 1,070,687,402 bytes (~1.00 GiB) across 461
packages -- main 859 MB / 338 pkgs, universe 212 MB / 123 pkgs, restricted and multiverse
both EMPTY in backports. Against a 951 GB existing mirror with 1.8 T free that is ~0.1%.
Roughly 46% of it is LibreOffice, which no OpenStack control-plane node installs; recorded
because it matters if backports is ever mirrored for another release or across many DCs.

**The generalisable defect, and it outlives this fix: `dc-mirror.sh check` verifies sync
STATUS, not SUITE COVERAGE.** It read PASS throughout, because nothing compares the suites
the mirror serves against the suites the deployed image's `sources.list` actually requests.
That is the same checker-cannot-see-the-real-failure class the mirror gate was already fixed
for once (2026-07-27, the `last-sync.status` existence check). A coverage assertion is OWED.

**Roosevelt analog:** an artifact mirror's scope is a CONTRACT with the node image. Whenever
the image or its `sources.list` changes, the mirror's suite list is part of that change --
and the gate should compare the two rather than trusting either.

---

### RULING NOTE 2026-07-31 -- D-135 / D-107: node artifact sources point at the IN-DC copy, and snaps get an in-DC forward proxy

**Status:** TWO rulings, both RULED 2026-07-31 (operator, GA-R5). **Presented together at
operator direction ("Yes, both") and answered SEPARATELY -- each carries its own utterance, so
neither is a batch adoption.** Both are **OPS under GA-R3**: they implement D-107's existing
airgap posture rather than changing it, and no D-number is assigned.

**RULING 1 -- the UCA source points at the mirrored copy.**
Question as presented: the charms add `deb http://ubuntu-cloud.archive.canonical.com/ubuntu
jammy-updates/caracal main`, MEASURED unreachable from a node (000) while the mirrored copy at
`http://10.12.8.4/cloud-archive` answers 200 with real content. Six apps set it explicitly
across TWO option names -- `openstack-origin` on barbican/magnum/octavia, `source` on
ceph-mon/ceph-osd/ceph-radosgw -- and the rest inherit the charm default `caracal`, which
resolves to the same upstream pocket. Options: point them at the mirrored UCA via the per-DC
overlay, or give nodes narrow egress to the upstream instead.
**Operator answer, exact utterance: "Point origin/source at the mirrored UCA, per-DC overlay
(Recommended)".**
Effect: dc0 gets an explicit deb line in `overlays/vr1-dc0-machines.yaml` (its own header
declares it "the per-DC HAND-MAINTAINED overlay", and it already carries a per-DC value). The
mirror address is per-DC, so this CANNOT live in `bundle.yaml`. **dc1 is UNCHANGED** -- its
apt-cacher-ng forwards whatever URL it is handed, so `cloud:jammy-caracal` keeps working
there. **That asymmetry is D-135's experiment RESULT, not a defect: the full-mirror DC must
rewrite every non-Ubuntu source; the proxy DC rewrites none.** MEASURED precondition: the UCA
signing key is already present on the nodes
(`/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cloud-archive.gpg`), so a raw `deb` line verifies
with no `|key` suffix.

**RULING 2 -- snaps reach an IN-DC forward proxy, not the internet.**
Question as presented: `mysql-innodb-cluster` (`snap install mysql-shell`), `ovn-central`
(`prometheus-ovn-exporter`) and `vault` (`core`) all fail because nodes cannot reach
`api.snapcraft.io`. This is the D-135 items 2-3 gap, and it hits dc1 IDENTICALLY since
apt-cacher-ng proxies apt and not snaps. MEASURED: juju exposes `snap-http-proxy`,
`snap-https-proxy`, `snap-store-proxy`, `snap-store-proxy-url`, `snap-store-assertions`; and
the RACK already reaches the snap store. Options: an in-DC forward proxy plus juju's
`snap-https-proxy`; Canonical's Snap Store Proxy; or narrow edge egress for the snap
endpoints.
**Operator answer, exact utterance: "HTTP(S) forward proxy in the DC utility band + juju
snap-https-proxy (Recommended)".**
Effect: **D-107's core statement stays TRUE and UNAMENDED** -- "OpenStack nodes ... reach NO
internet directly" -- because nodes reach an in-DC proxy, not the store. Same
application-layer-vs-routing distinction SEC-010's own justification draws, and the same one
that makes the voffice1->rack MAAS tunnel legitimate. It closes the D-135 items 2-3 gap for
BOTH DCs with one mechanism rather than widening the mirror-vs-proxy asymmetry. Placement (the
rack, which already holds the egress, versus its own utility-band VM needing a ruled octet)
and the service choice are engineering at build time; if it takes its own VM the D-134 octet
map needs a ruled octet first, since that map is a standing cross-DC standard.

**Both rulings leave the artifact classes honest:** apt from the in-DC mirror (dc0) or proxy
(dc1); UCA from the in-DC mirror (dc0) or proxy (dc1); snaps from the in-DC forward proxy
(both). MAAS boot images still come from `images.maas.io` and are NOT node artifacts under
D-107's C4 amendment, so they are out of scope here.

---

## D-136: NetBox-coupled per-DC render pipeline for bundle overlays and tfvars [ARCH]

**Status:** **ADOPTED 2026-07-27 -- option (D)** (GA-R5). Question as presented: options (A)
build-the-generator-now-and-gate-the-deploy, (B) hand-render-now-generator-for-Roosevelt, or
(C) values-file back half now with the NetBox front half later -- noting the operator had
directed a shape matching NONE of the three: populate the apex FIRST, then build the renderer,
then dry-run it WITHOUT gating this deployment on it. That shape was written up as **option (D)
below** so the ruling would not adopt something the entry did not contain. Operator answer,
exact utterance: **"Rule D-136 option D, mirror the v4 octet for both legs but make a note to
review at the end of the project to review for adjustment."** (The second clause of that
utterance rules the IPv6 host-part convention, recorded under D-134's R4 amendment, not here.)
IMPLEMENTATION IS UNBLOCKED. The carve-outs agreed the same day are pinned as forward items
**F2** (the automation half -- CI runner, event delivery, status-back) and **F3** (the MAC/VLAN
scope boundary) in `docs/dc-dc-deployment-workflow.md`; both are OUT of this adoption's build
scope. Prior status, kept as history: PROPOSED 2026-07-25 (Chat design seat, repo HEAD bc9a2df).
Escalates the renderer deferral in
`docs/audit/committee-20260724-track2-bundle-render.md` (Fork 2, "hand-render dc1 now,
extract the tool when dc0 forces the second render") to a NetBox-coupled generator.
Supersedes nothing.

**Problem.** Addressing is authoritative in NetBox (office1-netbox, the VR1 working
apex, DOCFIX-195) but reaches the deploy as HAND-RENDERED static literals: NetBox ->
imported by `netbox/*.py` -> literals in `scripts/lib-net.sh` / `lib-hosts.sh` and
tfvars -> hand-rendered into `bundle.yaml` and `overlays/*.yaml`. There is no live pull
and no generator. Measured fragility:

- BUNDLEFIX-052 (ceph-rbd-mirror binding) and a YAML flow-comma slip, one session.
- The app-to-VIP set exists in FOUR live places and no two agree: D-020's ratified list
  (11 apps, includes vault, excludes ceph-radosgw); `bundle.yaml` and
  `overlays/vr1-dc1-vips.yaml` (11, vault dropped, ceph-radosgw added);
  `overlays/dc-dc-ipv6-family-matrix.yaml` (10, plus octavia deliberately commented at
  lines 140-152).
- Two overlays own `vip:` for the same 10 applications under per-key REPLACE and
  last-wins, so the deploy input depends on overlay ORDER, not on a reconciled value.
- vault carries no `vip:` anywhere in the tracked tree while `overlays/dc-ha-scaleup.yaml`
  scales it to 3 units and declares `vault-hacluster` at `cluster_count: 3`. HA intent
  the artifact does not express, and an unrecorded departure from D-020.

Under the 2026-07-25 family ruling (see the ruling record for this date) the per-app
VIP becomes a six-address dual-family string, for BOTH DCs. That roughly doubles the
hand-maintained literal count at the precise point where the hand-render has already
failed twice.

**What the pipeline PULLS** (verified present in
`netbox/draft/vr1-office1-current-20260725.json`, captured 2026-07-25):

- Six-plane v4 `/22`s and v6 `/60`+`/64` per DC, selectable by `scope_site`
  (`vr1-dc0`, `vr1-dc1`); 92 of 139 prefixes carry `scope_site`, and the `role` slugs
  match the plane names exactly.
- The D-111 API-VIP `/64` per DC (`2602:f3e2:f02:11::/64` dc0, `2602:f3e2:f03:11::/64`
  dc1) and the shared metal `/60` at `:20`, admin `/64` `:20`, internal `/64` `:21`.
- Aggregates (`2602:f3e2::/36` ARIN, `fd50:840e:74e2::/48` org ULA, RFC1918,
  `23.157.124.0/24`), the D-120 bands as ip-ranges, two rack-controller static IPs.

**What it CANNOT pull today** (scoping constraint, not a defect): VIPs -- the record
holds four ip-addresses and none is a VIP. MAC pins and node inventory -- there is no
`dcim/devices` and no `dcim/interfaces` in the record at all. VLANs/VIDs -- no
`ipam/vlans`; the VLAN-group buildout is the open G4 item at
`docs/dc-dc-netbox-buildout-scope.md` section 4e.

**Where the gap actually is (per `docs/audit/decision-recon-20260725.md`, 4-lens,
2026-07-25).** The PREFIX layer is COMPLETE and DRIFT-FREE -- all 12 plane `/22`s, both
transit `/30`s, both uplinks, both GUA `/48`s, the ARIN `/36`, the full v6 subcarves and
the org ULA are present and match `lib-net.sh` value-for-value. The gap is entirely
SUB-PREFIX, and it is what this decision exists to close:

- **The D-134 bands** (as amended 2026-07-23, the CONTIGUOUS table: `.1-.3` infra,
  `.4-.49` reserved utility, `.50-.99` VIP, `.100-.200` nodes with control `.100-.119` /
  compute `.120-.149` / storage `.150-.200`, `.201-.254` dynamic) exist ONLY as decision
  prose -- in neither NetBox nor `lib-net.sh`. `lib-net.sh` carries only the dc0 VIP
  octet band `.50-.60`, UNSET for dc1.
- **The 33 VIP addresses** (11 services x provider/admin/internal, `.50-.60`) exist ONLY
  as hand-maintained literals in `overlays/vr1-dc1-vips.yaml`. They are internally
  consistent with the bundle octet-for-octet, but they are not a projection of the apex.

So a generator has nothing to pull for either. **Division of labour, corrected:** the
coupling MODELS the bands (as ip-ranges) and the VIP addresses (as scoped ip-addresses)
IN the apex, then GENERATES the overlays from them, so the overlay becomes a projection
of NetBox rather than a parallel hand-maintained truth. The per-DC values file is
therefore TRANSITIONAL -- it carries bands and VIPs by hand only until they land in the
apex, after which it shrinks to genuinely non-IPAM convention.

**Options.**

- **(A) Build the generator now and gate the dc1 deploy on it.** The forced dual-stack
  reconciliation IS its first job, so hand-rendering then rebuilding pays twice.
  Against: an unproven tool on the critical path of an otherwise-ready deploy; its
  first output has nothing to diff against; MAC pins and VLANs are absent from the
  record, so the tool cannot be complete on this timeline.

- **(B) Hand-render both DCs dual-stack; build the generator for Roosevelt.** Fork 2's
  lean, and "render once to learn the parameter surface" has real force. Against:
  roughly 170 dual-family literals by hand, at the point the hand-render has already
  produced two defects on the simpler v4-only artifact.

- **(C) Split at the seam: build the BACK half now, add the FRONT half later.
  RECOMMENDED.** Reconcile once, by hand, INTO a per-DC values file carrying every
  plane CIDR, VIP band, app-to-VIP map (one reconciled dual-family string per app,
  vault included) and role tag. Render the overlays from it with a thin deterministic
  substitution step. Both DCs ship on it. The NetBox front half (record -> values file)
  is added later without touching the back half, because the interface is frozen.

- **(D) Populate the apex FIRST, then build the renderer, then dry-run it -- WITHOUT gating
  this deployment on it. ADOPTED 2026-07-27.** Added to this entry on the day it was ruled,
  because the operator's directed shape matched none of (A)/(B)/(C) and adopting an option the
  entry did not contain would leave a future reader unable to find what was decided.
  Order: (1) make the gates trustworthy, (2) reconcile the values, (3) import them into every
  authoritative source -- MAAS, `lib-net.sh`, and the apex -- *even where the value was
  hand-written before the source existed*, (4) build the renderer, (5) dry-run it to confirm it
  pulls correctly *even though the data is already in place*, (6) audit the
  `data source -> renderer -> deployment mechanism` chain.
  **How it differs from (C):** (C) defers the NetBox front half and keeps VIPs/bands in a
  hand-maintained values file indefinitely; (D) closes the apex gap NOW, so the values file is
  transitional in fact and not just in intent. **How it differs from (A):** (A) puts an unproven
  tool on the deploy's critical path; (D) explicitly does not -- the renderer is dry-run and its
  output reviewed, and this deployment can proceed on reviewed artifacts regardless.
  **Why the deploy is not delayed by it:** the deploy is already blocked on the controller VM,
  per-role tags, OSD devices, v6 carve and dc0's missing overlays -- all of which are the
  renderer's own INPUTS. The tool work does not compete with the deploy for the critical path.
  **Cost, stated honestly:** the front half is built against an apex schema that only exists
  once step (3) lands, so (3) must complete before (4)'s front half is written -- which is the
  ordering the ruling already specifies. **Validation advantage over both (A) and (C):** the
  renderer is first proven by REPRODUCTION against `tests/render-baseline/` -- the known-good
  v4 11-application artifacts frozen 2026-07-27 precisely because the R2/R11 reconciliation
  destroys them. (A) has no such baseline; the fixtures make (D) checkable.

**Why (C).** [Superseded as the recommendation by the (D) adoption above; retained because its
reasoning is what (D) builds on.] It captures the whole benefit -- one source of truth, no two lists to
drift -- without absorbing the front half's unknowns (VIP representability, MAC and
VLAN buildout, token handling, network reach), none of which are on this deploy's
critical path. It makes the front half testable BY REPRODUCTION: once the schema is
frozen, NetBox-generated values are validated by diffing against known-good
hand-authored values for a DC that already deployed. Option (A) has no such gate. And
the values file is not scaffolding -- it is the permanent seam between IPAM fact and
deployment convention in the target operating model (pinned separately).

**Honest cost of (C):** the schema and substitution layer are written before the NetBox
schema is fully known, so the seam may need one refactor when the front half lands.

**Eliminated by construction.**

1. The VIP-overlay collision -- one reconciled `vip` per app per DC, emitted once; no
   second overlay to race, so deploy input stops depending on overlay order.
2. Vault decorative HA -- an application scaled with an `hacluster` peer and no VIP
   becomes a GENERATOR-TIME error instead of a silent runtime condition.
3. Family drift -- D-101's matrix applied once in code instead of by a human ~170
   times. The validator asserts family-per-plane-per-SITE, not mere presence: the
   record currently carries GUA storage prefixes for `vr0-dc0` and `willamette` while
   VR1 storage is ULA, so a generator selecting on `role` alone pulls the wrong family.
4. Reserved-range skew -- per D-074 the tenant-CIDR reserved set is the six plane
   `/22`s (today sourced from `scripts/lib-net.sh` into `scripts/net_overlap.py` as
   argv) plus the live-resolved `capi-mgmt-subnet` and the metadata range. Generating
   the plane rows from the same values file removes the risk of onboarding a tenant
   validated against the WRONG DC's planes.

**DC-aware gates (part of this proposal).** `scripts/preflight.sh` validates the real
per-DC deploy input via `scripts/provider-bundle-check.py --overlay --dc`, not the base
bundle alone. `scripts/cloud-assert.sh` gains a per-hacluster VIP-owned check, so a
scaled application with no VIP fails rather than passes. Bootstrap carries the
controller-tag constraint.

**Mechanism (front half, when built).** Extending `netbox/*.py` is preferred over a
third-party NetBox OpenTofu provider. The existing tools already establish the correct
posture: one script per SOURCE (`office1-record-dump.py` vs `prod-draft-dump.py`),
read-only, token from the environment, never embedded or printed. A provider must
additionally be mirrorable into the per-DC airgap mirror (D-107) or it is disqualified
on Roosevelt, and it would put a write-capable credential in the tofu run while
write-back is deferred (DOCFIX-195). Reads only; no write-back under this decision.

**Scope boundary.**

- IN: the six operator planes per DC, VIP bands, plane CIDRs, per-DC role tags, and the
  generated per-DC overlays and tfvars.
- OUT -- TENANT SPACE, permanently. Per `docs/dc-dc-netbox-buildout-scope.md:128`
  per-tenant `/60`s are Neutron-managed (D-016/D-074) and not enumerated in NetBox. Per
  D-074 tenant private CIDRs are TENANT-CHOSEN and MAY OVERLAP, enforced only against
  the reserved-range registry. NetBox models pools, aggregates and reserved ranges; it
  never models a tenant's own networks. A maintainer extending the generator "one more
  table" into tenant space would either fight Neutron or gate tenant provisioning on an
  operator-side IPAM write, breaking the tenant self-service model this cloud is sold
  on.
- IN: WRITES TO office1-netbox. DOCFIX-195 (operator ruling 2026-07-15) rules that
  during VR1 `office1-netbox` takes ALL writes and `netbox.baldurkeep.com` stays a
  READ-ONLY v1 reference with merge-back DEFERRED to end-of-deployment. Precedent tool:
  `netbox/d120-compose-bands.py --commit` already loaded the D-120 bands into the apex.
  (An earlier draft of this entry scoped apex writes OUT on a misreading of DOCFIX-195;
  that would have scoped the coupling out of its own core job -- see the gap below.)
- OUT: writes to `netbox.baldurkeep.com` (deferred, DOCFIX-195; P2 merge-back).
- OUT: VLAN / space / fabric -- PERMANENTLY, not pending. office1-netbox holds zero VLAN
  objects and that is CORRECT BY DESIGN: VR1 is untagged-per-fabric on every plane in
  both DCs (D-133, superseding VR0's VID-103/104), so a VLAN object would document
  tagging that does not exist on the wire. The bundle binds by SPACE NAME -> MAAS subnet
  by CIDR, never by VLAN/fabric/subnet id. VLAN/space/fabric is MAAS-sourced; NetBox is
  IPAM only.
- OUT: MAC pins -- as a NETBOX SOURCE. They stay substrate/`lib-hosts`-sourced; this is not
  a NetBox gap, because the apex holds NO `dcim/devices` and NO `dcim/interfaces` at all
  (measured 2026-07-27 in `netbox/draft/vr1-office1-current-20260725.json`: `ipam/*` plus
  `dcim/regions` and `dcim/sites` only). There is nothing to pull.
  **CORRECTION 2026-07-27 -- the ORIGINAL REASONING FOR THIS SCOPE-OUT IS FALSIFIED, and the
  conclusion changes.** This bullet previously read that MAC pins "are drift-free across
  substrate `main.tf` <-> `lib-hosts.sh` `HOST_BOOT_MAC` <-> bundle ovn-chassis <->
  discovery." MEASURED: `bundle.yaml:462-464` sets `bridge-interface-mappings:
  br-ex:52:54:01:d1:04:02 br-ex:52:54:01:d1:05:02`, and per `scripts/lib-hosts.sh:132` the
  `52:54:01:d1:NN:01` scheme is **vr1-dc1's** pinned convention (`:04`/`:05` = compute-01/-02).
  So the file documented as vr1-dc0's source of truth carries **dc1's** compute provider MACs
  -- not drift-free at all. It is a deliberate residue of the 2026-07-24 dc1-first render, and
  vr1-dc0's own boot MACs are a DIFFERENT, NON-schematic set (`52:54:00:be:69:c5`, ...,
  lib-hosts:118-122) that cannot be derived from a pattern the way dc1's can.
  **CONSEQUENCE: `ovn-chassis` `bridge-interface-mappings` is a PER-DC value and must become
  renderer OUTPUT into the per-DC overlay, sourced from `lib-hosts.sh`.** It must also move OUT
  of the base bundle for the 2026-07-25 symmetric-overlay ruling to be truthful. Left in place
  it means a dc0 deploy matches no local MAC, `ovn-chassis` builds no br-ex mapping, provider
  egress is dead on dc0 compute, and NO GATE FAILS. This was latent while the plan was
  dc1-first; the 2026-07-27 ruling that both DCs deploy by the same procedure makes a dc0
  deploy certain. Forward item F3 in `docs/dc-dc-deployment-workflow.md` carries the boundary.

**Roosevelt-delta (A1).** Roosevelt is N bare-metal DCs. Hand-rendering is O(N) human
renders of a ~170-literal dual-family artifact; a generator plus one values file per DC
plus one validator is O(1) tooling and O(N) data. The transferable artifact is the
triple: renderer, values file, validator. This is why the back half is worth building
even while the front half waits.

**Artifact shape (RULED 2026-07-25 -- operator ruling, GA-R5).** Question as presented:
there is no `vr1-dc0-vips.yaml`; dc0's VIPs are inline in `bundle.yaml` while dc1 has an
overlay -- under the family ruling both DCs need v6 legs, so the asymmetry must break:
(i) symmetric (extract dc0 to its own overlay, base becomes VIP-free) or (ii) asymmetric
(dc0 v6 legs into base, two renderer output modes forever)? Operator utterance, verbatim:
"Your question about 1 or 2, go with 1. Base bundle and everything else is applied via
overlay." Effect (option (i), symmetric per-DC overlays): `bundle.yaml` becomes VIP-free
and holds topology only; every per-DC value, dc0's included, arrives via overlay. dc0's
VIPs are extracted from base into `overlays/vr1-dc0-vips.yaml`. The renderer therefore has
ONE output shape per DC and Roosevelt's Nth DC is the same shape as its first.

**PREREQUISITE, sequenced FIRST (per the recon's sequencing insight).** Extend
`netbox/sandbox-fidelity-check.py` BEFORE the coupling is built. Today it covers only the
D-115/D-117/D-120 delta and diffs the FROZEN 90-prefix `netbox.baldurkeep.com` draft, not
the current 139-prefix office1-netbox apex; it is blind to D-124 and D-134. It gates every
apex reconciliation this coupling depends on, and the coupling's whole value proposition
is eliminating the drift those reconciliations chase. An unextended checker cannot verify
the apex mutations in the gap list above, so building the generator first would mean
writing to the apex with no trustworthy fidelity gate.

**A hard tooling coupling nobody has flagged.** `scripts/lib-net.sh:59` sets
`VIP_COUNT_EXPECT=11`, consumed by `scripts/pre-flight-checks.sh` at :79 (warn on vip
line count) and :99-102, where a count mismatch is a **fail**, not a warn. Adding vault's
VIP takes the count to 12, so `VIP_COUNT_EXPECT` must move to 12 in the same change or
the gate fails. Note that CURRENT-STATE flags `pre-flight-checks.sh` as VR0-frozen (no
`$DC` selector), so whether it gates VR1 today needs confirming -- but the constant is in
the shared library and any DC-aware successor inherits it. **Sibling constant (flagged by
the 2026-07-25 pack review, missed by the pack):** `scripts/lib-net.sh:58` sets
`VIP_OCTET_MAX=60`, consumed by the octet range-check at `pre-flight-checks.sh:81`; vault's
`.61` violates it, so `VIP_OCTET_MAX` must bump alongside `VIP_COUNT_EXPECT` (`.61` is
band-legal per the D-134-amended `.50-.99` VIP band -- the fix is the constant, not the
octet). A second stale `VIP_COUNT_EXPECT=11` restatement lives in D-134's band-table prose,
and `lib-net.sh:157` UNSETS both for the vr1-dc1 arm. A generator emitting the VIP set
should DERIVE this count rather than carry it as a second literal.

**Gate relationship.** This decision does NOT gate the dc1 deploy. What the family
ruling gates, and what must be resolved before deploy, is three items: (1) the vault
VIP plus its band octet (a D-020 conformance repair, OPS not ARCH); (2) the octavia
family sub-ruling; (3) the reconciled dual-family VIP literals for both DCs, in the
symmetric overlay shape above. The subcarve is NOT among them -- D-111 (ADOPTED
2026-07-11) already ratifies the per-plane v6 subcarve and the values are imported for
both DCs.

**Open questions (operator).**

1. Metal VIP host addresses come straight out of the plane `/64` and share it with node
   addresses; D-111's VIP-dedicated `/64` at `:11` exists only on the provider/GUA leg.
   Confirm no MAAS deployed-node band claims the VIP octet on the metal planes. A
   collision is repaired by a D-111 amendment (a metal VIP sub-range), not by values.
2. Octavia family. D-101 places `lb-mgmt` in the IPv6-only ULA group, but
   `overlays/dc-dc-ipv6-family-matrix.yaml:63-82` documents live upstream IPv6
   breakage on that path (LP #1911788, #1913409). Recommendation: dual-stack octavia's
   API endpoint like its ten peers, keep `lb-mgmt` v4 as a logged D-101 exception, on
   the ruling's own standard -- a path with open upstream bugs is not where IPv6-only
   "makes sense". This is an exception to an ADOPTED decision and is the operator's.
3. Does the reserved-range registry become a first-class NetBox object, or continue to
   be derived (plane rows from prefixes; `capi-mgmt-subnet` and metadata live-resolved
   per D-074/D-035)? Deriving is recommended; D-074's stated gain is that NetBox's
   uniqueness duty shrinks to the routed space. Operator has asked for more information
   before ruling.
4. Values-file format and location; and whether generated artifacts are COMMITTED
   (reviewable diffs, recommended) or generated at deploy time (unreviewable).
5. Automation runner, event delivery, and status-back (CORRECTED 2026-07-25 -- the
   earlier "GitBucket has no runner" gap framing is VOID). **Jenkins IS the runner** --
   operator-side and already running (v1 K8s + app deploys); nothing is stood up and no
   git-host / Actions-style runner is required. Carry-forward: this is the revisit trigger
   for the deferred "Jenkins CI poll job" at `docs/handoff-20260703-open-items.md:159`. The
   real requirements are TWO, NEITHER yet verified against the running GitBucket instance
   (do NOT promote to a SpryGit must-have until checked): (a) **reliable event emission** --
   webhook delivery with retry, a delivery log, and manual replay (a silently dropped
   webhook renders nothing and nobody notices -- worse than no automation); (b) a
   **status-back API** -- Jenkins writes the `provider-bundle-check` and `repo-lint`
   verdicts onto the merge request as commit status/checks, so the RENDERED-DIFF approval
   gate (the operating model's gate) can actually see the validator verdict -- this is the
   requirement that protects the gate. PLACEMENT is OPEN (operator ruling required): the
   Jenkins job on `voffice1` for VR1 (D-128 already routes Plane 2 -- MAAS/NetBox/inner
   tofu -- there, and the render reads Office1-side NetBox + git), with per-DC utility-node
   agents as the Roosevelt shape once D-132 ratifies -- BUT D-132 is PROPOSED and per-DC
   utility nodes are NOT built in VR1 (D-134's .4-.49 band is reserved for FUTURE nodes;
   D-135's mirror at .4 is a RACK-HOST service address, not a node); and whether an Office1
   job emitting BOTH DCs' overlays brushes D-105's cross-DC roll-up prohibition. The HTTPS
   credential is a security-ledger row, not part of this decision.

**Related:** D-020 (dual provider+metal VIPs -- NOT D-036, mis-cited as such in SEVEN
live sites; see DOCFIX-201 in this session's changelog), D-100/D-101 (addressing,
families, mesh), D-111 (per-DC v6 subcarve, ADOPTED), D-052/D-053 (bindings, six-plane
split), D-104 + 2026-07-24 amendment (controller VM), D-107 (per-DC airgap mirror),
D-108 (cross-DC), D-119 (region-qualified tags), D-121 (Option C + R-3), D-016/D-074
(tenant CIDRs), D-035 (capi-mgmt reserved), D-067 (barbican-vault on metal-internal),
D-110 (repo has moved; never hardcode its path), D-133 (flat per-NIC carve, no VLAN
stack -- why VLANs are correctly absent from the apex), D-134 + its 2026-07-23
AMENDMENT (the contiguous band table; the amendment is authoritative and the original
table above it is superseded), D-124 (rack statics), D-131 (DNS forwarder .3), D-135
(mirror .4), D-105 (per-DC-only, cross-DC roll-up prohibition -- open Q5 placement),
D-128 (Plane-2 operating model -- the VR1 render/Jenkins host), D-132 (per-DC MAAS
topology, PROPOSED -- the Roosevelt per-DC-agent shape), DOCFIX-195 (apex takes all VR1
writes; baldurkeep merge-back deferred),
BUNDLEFIX-052. Companion audit records: `docs/audit/decision-recon-20260725.md`,
`docs/audit/stage5-expansion-review-20260724.md`,
`docs/audit/committee-20260724-track2-bundle-render.md`.


## D-137: ADOPTED -- credential mint-and-consolidate pipeline: make consolidation STRUCTURAL, not remembered [ARCH]

**Status:** ADOPTED 2026-07-26 -- all five sub-rulings RULED individually (GA-R5), each
committed and pushed before the next was asked; implementation UNBLOCKED. Sub-ruling
selections, verbatim: (1) enforcement = "Blocking in preflight"; (2) derivation = "Derive
manifests from matrix (Recommended)"; (3) remote scope = "Declared locations only
(Recommended)"; (4) policy home = "D-137 is the authority (Recommended)"; (5) identity =
"Fold in as a D-137 invariant (Recommended)". Full question texts and consequences are in
the sub-ruling blocks below. Originally PROPOSED 2026-07-25 (repo HEAD `1155729`; operator asked for
"a better best practices method for minting and saving these credentials going forward"
and "a durable rule to make sure when accounts are created there is a consolidation
that happens every time so credentials aren't misplaced or lost"). Supersedes nothing.
Would make the SEC-009 standing convention a D-governed policy with a Roosevelt-transferable home.

**ADOPTED 2026-07-26 -- all five sub-rulings RULED individually (GA-R5, one exchange
each), each committed and pushed before the next was asked. Implementation is UNBLOCKED.
The five sub-rulings and their verbatim operator selections are recorded below.**

**Sub-ruling 1 (ENFORCEMENT STRENGTH) -- RULED 2026-07-25.** Question as presented:
"D-137 ruling 1 of 5 -- enforcement strength. This decides whether the rule binds or
merely advises, and it gates rulings 2-5." Options presented: blocking in preflight /
blocking plus a PreToolUse guard / advisory only. Operator selection, exact:
**"Blocking in preflight"**. CONSEQUENCE: the credential check runs inside
`scripts/preflight.sh` as a new `Pn` and HARD-FAILS (exit 1) on any expected-but-absent,
undeclared, or per-DC-asymmetric credential -- no deploy proceeds past a credential gap.
NOT adopted: the PreToolUse guard (so an agent is not constrained at write time, only at
the deploy gate) and advisory-only (explicitly the posture that failed -- the phase-3
prose rule was advisory and never fired at either DC standup: dc0 carried 3 and dc1 1
undeclared secret). Preflight is the correct home per its own header intent: it exists to
collapse a fragmented gate surface into one artifact, so a creds gate operators must
remember to run separately is the exact failure mode preflight was built to delete.

**Sub-ruling 2 (DERIVATION) -- RULED 2026-07-25.** Question as presented: "D-137 ruling
2 of 5 -- derivation. The committee's main risk finding was GA-F06: a hand-maintained
matrix becomes a fourth surface that goes stale, fails falsely, and gets switched off.
How do the matrix and the manifests relate?" Options presented: derive manifests from the
matrix / cross-check only / thin overlay on manifests. Operator selection, exact:
**"Derive manifests from matrix (Recommended)"**. CONSEQUENCE: the matrix is the SINGLE
SOURCE. `creds-matrix` gains a `--render` mode that regenerates
`creds-manifests/*.manifest`, and the gauntlet FAILS if the checked-in manifests differ
from rendered output -- dual maintenance is eliminated, matching the D-136
generate-don't-duplicate instinct. ACCEPTED COST, stated at ruling time: the manifests
become GENERATED artifacts, so hand-edits will be clobbered and today's governance prose
(SEC-NNN linkage, mint dates/hosts, the "off-manifest" carve-outs for the Juju client
credential store, the deliberately-un-consolidated findings) must become MATRIX FIELDS or
be regenerated into the rendered header -- it must not be silently dropped. That prose is
load-bearing: `vr1-dc1.manifest:18-20` is what recorded the dc0 edge-keypair backfill
debt, and `vr1-office1.manifest:24-27` carries the un-consolidated residue.

**Sub-ruling 3 (REMOTE DISCOVERY SCOPE) -- RULED 2026-07-26** (this session crossed the
date boundary; sub-rulings 1-2 were given and pushed 2026-07-25). Question as presented:
"D-137 ruling 3 of 5 -- remote discovery scope. This is the half that finds
unknown-unknowns (it's what would catch the shadow `*-creds/` stores on voffice1,
SEC-022). The tension is tenant isolation on a commercial multi-tenant cloud." Options
presented: declared locations only / declared plus a bounded home sweep / broad
filesystem sweep. Operator selection, exact: **"Declared locations only (Recommended)"**.
CONSEQUENCE: a new declared list (`creds-manifests/vm-secret-locations`, rows
`<host-role> <path>`) bounds `--remote` absolutely -- it never walks outside the list. No
tenant surface is touched; the D-069 custody boundary and the hard-isolation posture are
preserved, and the check stays fast and auditable on a live region.

**FAITHFUL-IMPLEMENTATION NOTE (not an override).** "Declared locations" must be
populated with everything ALREADY KNOWN to hold credential material, or this ruling
leaves measured defects uncovered. The list must therefore include, at minimum: per-site
creds folders on the jumphost AND on the headend (SEC-022 -- the shadow stores), the
region's `/root/maas-secrets/` (SEC-020), and the three directories the creation-point
inventory found outside the SEC-009 convention (`docs/audit/creds-creation-points-20260725.md`
FINDING 2): the Vault init directory (5 unseal shares + root token), the Octavia PKI
directory (8 artifacts incl. two CA private keys), and the per-tenant directories. ALSO
`overlays/octavia-pki.yaml` inside the repo clone. Accepted residual, inherent to the
ruling: a secret minted at a location nobody ever declares stays invisible -- so ADDING a
row to this list is part of the definition-of-done for any new mint site.

**Sub-ruling 4 (POLICY HOME) -- RULED 2026-07-26.** Question as presented: "D-137 ruling
4 of 5 -- policy home. Where does the credential-lifecycle RULE live as authority? Today
it lives only in the security ledger, which is a register of exposures -- so no gate can
cite it, and it has no Roosevelt-transferable home." Options presented: D-137 is the
authority / policy stays in the ledger / split by audience. Operator selection, exact:
**"D-137 is the authority (Recommended)"**. CONSEQUENCE: D-137 becomes the
credential-lifecycle POLICY AUTHORITY. The SEC-009 standing-convention block in
`docs/security-ledger.md` DEMOTES to a pointer at D-137, and the ledger returns to being
purely a register of exposures and rotation obligations (its stated purpose, and the GA-R4
F3 amendment's intent). Gates may then cite a D-number as authority rather than citing an
exposure register. Satisfies the GA-R3 A1 test directly: a Roosevelt build session greps
`docs/design-decisions.md` before touching a built surface, so the rule must live there to
be found. IMPLEMENTATION CARE: the demotion must MOVE the convention text, not delete it
-- SEC-009's block carries the founding history (the NetBox-token miss that motivated the
control) and the enforcement description; that history stays in the ledger as history,
while the RULE moves to D-137.

**Sub-ruling 5 (IDENTITY FOLD-IN) -- RULED 2026-07-26.** Question as presented: "D-137
ruling 5 of 5 -- identity fold-in. SEC-020 found that `admin` is both the human GUI login
and the automation identity (its API key drives 19 call sites in 4 scripts). I split that
off to avoid batching rulings, but the matrix has a `principal` column, which would make
the conflation machine-detectable. Fold it in, or keep it separate?" Options presented:
fold in as a D-137 invariant / keep separate as its own D / fold in with detection
deferred. Operator selection, exact: **"Fold in as a D-137 invariant (Recommended)"**.
CONSEQUENCE: D-137 gains the invariant **one identity serves one principal type**, and the
matrix's `principal` column ENFORCES it -- a credential serving both `human` and `service`
is a FAIL, not an observation. The SEC-020 conflation is therefore machine-detectable, and
NO separate D-number is needed (the identity question that SEC-020 left "proposable but
unassigned" is hereby absorbed here). The `operator`/`admin` split performed 2026-07-25 is
the reference fix pattern. Note the invariant will FAIL on today's tree by design: `admin`
serves a human GUI row and a service API row simultaneously, which is the defect, and
`juju-vr1-dc0`/`-dc1` are superusers with unstored passwords (service-only by ruling) --
those rows encode the intended end state, so the first run is expected to be red.

**ALL FIVE SUB-RULINGS ARE NOW RULED. This entry moves from PROPOSED to ADOPTED and
implementation is unblocked** -- see the Status line above. Design record, committee review and the measured findings
that shaped them: plan `whimsical-wandering-spindle`, capture
`docs/audit/creds-creation-points-20260725.md`, rows SEC-021/-022/-023.

**Problem -- the existing control cannot detect the failure class it was built for.**
SEC-009 (2026-07-15) established the `~/<site>-creds/` convention AND its enforcement
(`creds-manifests/<site>.manifest` + `scripts/creds-audit.sh`), explicitly BECAUSE a
VM-minted secret had been missed (the NetBox token addendum names that exact class).
Measured evidence that it did not hold:

- `scripts/site-headend-install.sh:416-455` minted FOUR secrets on the region VM on
  2026-07-13 (`/root/maas-secrets/{db.pass,admin.pass,admin.apikey,lxd-trust.pass}`,
  via `newpass`). The office1 manifest was seeded with 11 entries on 2026-07-15 and
  `creds-audit vr1-office1` reported CLEAN -- while all four sat undeclared and
  invisible. `admin.pass` surfaced only on 2026-07-25, TWELVE days later, and only
  because a task went looking for it (SEC-020).
- **Root mechanism: the audit is DECLARATION-based.** The manifest IS the declaration,
  so an undeclared secret is structurally undetectable. The `<vm>:<path>` provenance
  field exists for precisely this class but only helps once a human remembers to write
  the row. A green CLEAN is therefore not evidence that nothing was missed.
- `creds-audit.sh:25` takes a single mandatory `<site>` argument -- no `--all`. Auditing
  the estate means three invocations and remembering three site names.
- It is wired into exactly ONE place, and as PROSE:
  `runbooks/dc-dc-phase3-maas-enlist-deploy.md:498` ("MUST report ... before declaring
  this DC done"). It is NOT in `preflight.sh` (THE single pre-deploy gate), NOT in
  `cloud-assert.sh`, NOT in `repo-lint`, NOT in the gauntlet. Nothing executable
  enforces it.
- Consequence measured at this session's open: dc0 carried THREE undeclared secrets and
  dc1 ONE -- i.e. that standup definition-of-done line did not fire at EITHER DC
  standup, and both racks were declared done anyway.
- No D governs credential mint/custody generally: D-069 covers Vault unseal-key custody
  only; D-126 covers per-env access-key ISOLATION, not lifecycle. The rule currently
  lives in `docs/security-ledger.md`, a register of EXPOSURES -- not a decision
  authority, so it has no Roosevelt-transferable home and no gate can cite it.

**Proposal.**

*1. The rule (the invariant).* Minting and consolidating are ONE operation, never two
steps with a human in between. A credential that exists without a manifest row is a
DEFECT, not a to-do. A secret minted on a site VM is PULLED to the jumphost by the
jumphost-side caller in the same runbook step that mints it -- never "consolidated
later", because later is when it is lost. Values are verified by format or behavior,
never printed, never in argv (the SEC-018 discipline; note `maas changepassword` has no
`--password` and `createadmin --password` violates it, so stdin is the sanctioned path
-- proven working 2026-07-25).

*2. The tooling that makes it structural.*
- (a) `scripts/creds-mint.sh` -- the ONLY sanctioned way a credential comes into
  existence. Modes: `--generate` (openssl, umask 077), `--pull <vm>:<path>` (VM-minted,
  sets provenance automatically), `--stdin` (externally minted). In ONE run it writes
  the file 0600, appends the manifest row, refuses to overwrite an existing entry, and
  exits non-zero unless `creds-audit` is CLEAN afterward. Consolidation becomes
  impossible to skip because it is not a separate action.
- (b) `creds-audit.sh --all` (iterate `creds-manifests/*.manifest`; no site name to
  remember) and `--remote` -- the DISCOVERY inversion. `--remote` walks a declared set
  of known secret locations (new `creds-manifests/vm-secret-locations`: `<vm> <path>`),
  enumerates what is actually THERE by metadata only, and FAILS on any file with no
  matching `<vm>:<path>` row. This is the only proposed piece that would have caught
  `admin.pass` on 2026-07-15; without it the audit stays blind by construction.
- (c) Gate wiring so it cannot be forgotten: `creds-audit --all` inside
  `scripts/preflight.sh`, plus an offline-safe harness in the gauntlet; the phase-3
  standup line stays as belt-and-braces rather than as the only control.
- (d) `site-headend-install.sh` amendment: on completion emit a machine-readable list of
  the secrets it minted (paths + modes, NO values) so the caller can `creds-mint --pull`
  each one deterministically. This script minted all four missed secrets; closing the
  class at its source is the highest-leverage single change.

*3. Roosevelt delta (the A1 test).* Roosevelt mints per-DC IPMI/BMC credentials, Vault
material, and per-DC MAAS service credentials on real hardware where a lost credential
can mean a lost node. A Roosevelt build session should grep THIS before creating any
credential. That is why the rule belongs in a D and not only in an exposure register.

**Open forks (operator rules; PROPOSED means present, not implement).**
1. ENFORCEMENT STRENGTH: (a) advisory -- audit reports, human acts; (b) blocking --
   `preflight.sh` hard-fails on an unconsolidated or undeclared secret, so no deploy
   proceeds; (c) blocking PLUS a PreToolUse guard that refuses secret-writing command
   shapes outside `creds-mint.sh`. Trade: (c) is the only option that constrains an
   agent mid-session, but adds a hook that can itself misfire on legitimate shapes.
2. `--remote` DISCOVERY SCOPE: declared-directories-only (predictable, but a secret
   minted somewhere undeclared stays invisible) vs a broader filesystem sweep for
   secret-shaped filenames (catches more, but is invasive on a live region and could
   surface tenant material -- a hard-isolation concern on a commercial multi-tenant
   cloud).
3. POLICY HOME: this D becomes the credential-lifecycle authority and SEC-009's
   convention block demotes to a pointer, vs the policy staying in the security ledger
   with this D covering only the tooling.

**Related, deliberately NOT folded in.** SEC-020 flagged a separate identity question --
`admin` doubles as the human GUI login and the automation identity (its API key drives
the `maas admin` profile across 19 call sites in 4 scripts). That is "who holds
accounts"; this D is "how secret material is stored". Same lifecycle family, different
decisions; folding them would batch two rulings and violate GA-R5.

**IMPLEMENTATION AMENDMENT 2026-07-26 (12-column matrix schema) [OPS].** Recorded here
because the amendment previously existed only in a session changelog and the build-spec
file; GA-R2 rule 1 makes a changelog non-citable and rule 5 consolidates it away, so the
only durable home for an operational choice that outlives the stage is this entry.
The matrix schema went from 10 to 12 columns and a row was re-keyed from one credential to
one (credential, location) PAIR. Added: `custody`
(`consolidated` | `source-of-record` | `off-manifest-known` | `not-consolidated-ruled`) and
`notes-ref` (an id-keyed pointer into `creds-matrix-notes.md`). Question as presented:
"reply with a go on the schema amendment"; operator answer, exact utterance:
"go with the 12-column amendment", 2026-07-26.
WHY: a read-first round-trip check of the three checked-in manifests proved the 10-column
form could not carry what sub-ruling 2 forbids dropping, and sub-ruling 3 places four
deliberate, reasoned credential copies INSIDE `--remote`'s declared locations, where
`creds-audit.sh`'s undeclared-file check would report every one as a defect. This
IMPLEMENTS sub-ruling 2 ("this prose must become MATRIX FIELDS") rather than changing what
D-137 decides; the five sub-rulings are untouched. OPS under GA-R3 (mechanism detail,
doubt resolves DOWN); no new D-number.
NOTE the `not-consolidated-ruled` token encodes standing SEC-020(i) operator rulings as
machine-readable data -- that is the reason this belongs in a decision surface and not
only in session scratch.

**COMMITTEE AUDIT 2026-07-26 -- implementation defects found post-adoption.** A six-lens
read-only review of the tier-1 + tier-2 build reproduced eight false-greens, the most
consequential being that tier 2 keys existence on host-role ALONE and never consults
`site-key`, so a credential present at one site satisfies the row for every site. Status,
the full defect list, and remediation progress live in `docs/CURRENT-STATE.md` (GA-R1);
this note exists so a reader of D-137 is not left believing the adopted design shipped
sound. The DESIGN is unchanged and was not faulted; the defects are implementation.

**SUB-RULING 1 ENFORCEMENT EXECUTED 2026-07-26 (tier 1).** The adopted ruling is
"blocking in preflight"; the open question was whether to wire it while the register is
red by design, or defer until after per-row remediation. Question as presented: wire the
blocking `Pn` as ruled, or defer. Operator answer, exact utterance: "wire tier 1 as the
blocking preflight Pn". Tier 1 (STATIC, offline) now runs as `scripts/preflight.sh` P5,
blocking, ahead of the stage-2 reminders block, and FAILS CLOSED if the checker is absent.
Tier 1 was the half that needed no further exchange because its findings ARE the three
ruled failure classes (expected-but-absent / undeclared / per-DC-asymmetric). Tier 2's
`Pn` is deliberately NOT wired: it requires `--remote`, `--privileged` and a
caller-supplied `--pending-stage`, which do not belong in an unattended gate -- that
remains an open operator decision. Status and the live consequence live in
`docs/CURRENT-STATE.md` (GA-R1).

**CREDENTIAL-LIFECYCLE POLICY (moved here 2026-07-26 by sub-ruling 4; supersedes the
SEC-009 convention block, which is now a pointer).** This is the standing rule; the
security ledger is an exposure/rotation register and no longer states it.

1. **Consolidation.** ALL sensitive files and environment/config files for a site live in
   a single `~/<site>-creds/` folder on the jumphost: folder 0700, files 0600, public key
   halves 0644. No loose credential files in `~`. Every site follows this from day one --
   never accumulate loose `<site>*.env` and consolidate later.
2. **At mint time, not later.** A credential is consolidated and registered AT THE MOMENT
   IT IS MINTED. "Later" is when it is lost. This is a data-loss/continuity control as
   much as a security one: scattered credentials disappear between sessions.
3. **The register is the expected state.** `creds-matrix.tsv` declares what MUST exist;
   `creds-manifests/*.manifest` are DERIVED from it (sub-ruling 2), so a manifest is no
   longer the place to declare anything by hand.
4. **Discovery is bounded absolutely** by `creds-manifests/vm-secret-locations`
   (sub-ruling 3). A location that is not declared is not audited, and an unaudited
   location is how SEC-022 happened. **Adding a location row is definition-of-done for
   any new mint site.**
5. **One identity serves one principal type** (sub-ruling 5), enforced by the matrix
   `principal` column.
6. **Enforcement is blocking, not advisory** (sub-ruling 1): tier 1 runs as
   `scripts/preflight.sh` P5 and fails the deploy. Advisory was rejected as the posture
   that demonstrably failed -- `creds-audit` read CLEAN while four region-VM secrets sat
   undeclared.

CARRIED FORWARD WITH CORRECTIONS, not verbatim: the moved text's specific counts
("11 entries", "7/7") and its description of manifests as hand-declared were stale by
2026-07-26 and are not reproduced. The rehearsal-scoped posture it recorded still holds --
real credential hardening (rotation, revocation, non-shared tokens) remains deferred to
post-teardown cycles per SEC-009's original rationale, while consolidation happens NOW.

### SUB-RULING 6 (D-137) -- RULED 2026-07-27: the register is staged to what a stage ACTUALLY mints, and provenance debt converges via `runbook:` refs, not a new tool

**Status:** RULED 2026-07-27 (operator, GA-R5). Question as presented, verbatim: "R13 --
credential reproducibility. Three distinct problems: 30 rows at `mint-ref=operator-terminal`
(7 of them unrecoverable-in-place keypairs); `creds-mint.sh` unbuilt; and the register
mis-stages what Stage 5 actually mints (vault init, per-DC Octavia PKI, admin-openrc) so P5
goes green through the biggest mint event. What is the scope before Stage 5?" Options
presented: (a) register-first with no new tool -- fix the staging AND flip the 7 keypairs;
(b) build `creds-mint.sh` first; (c) all three now. Operator selection, exact utterance:
**"Register-first, no new tool: fix the staging AND flip the 7 keypairs (Recommended)"**.

**PART 1 -- RE-STAGE WHAT STAGE 5 ACTUALLY MINTS.** Measured (audit finding L7-2): the
register carries ZERO rows for Stage 5's real mint output. The vault-init, Octavia-PKI and
`admin-openrc` rows are `cardinality=singleton` with `mint-stage` in
{`vr0-phase01`,`vr0-phase02`,`vr0-phase03`}, and `creds-manifests/stages-reached` marks all
three `pending` -- so flipping `stage5` to `reached` never makes them expected. The measured
consequence is a FALSE GREEN over the largest minting event of the deployment:
`[ok] E1 18 expected artifact(s) deferred as not-yet-minted per
creds-manifests/stages-reached`. **R7 made this MORE urgent, not less:** ruling per-DC
independent Octavia PKI means TWO CA mints where the register expects none. Re-stage those
rows so they are EXPECTED at Stage 5 and P5 can actually fail on their absence.

**PART 2 -- CONVERGE THE PROVENANCE DEBT WITHOUT NEW TOOLING.** 30 rows across 16 ids carry
`mint-ref=operator-terminal`, and `grep -rnI "ssh-keygen" .` returns ZERO hits repo-wide, so
those credentials are not reproducible from the repo. The checker currently reports this as
`[ok] S4 provenance debt: 30 row(s) ... Admitted by design`. **The S4 check ALREADY resolves
`runbook:<path>:<line>` references**, so recording each mint as a numbered runbook step and
flipping `mint-ref` from `operator-terminal` to `runbook:` makes the debt CONVERGENT using
machinery that already exists and is already tested.

**PRIORITY ORDER within Part 2 -- the SEVEN unrecoverable-in-place keypairs first.** Loss of
any of these means losing the only access path, so they are the ones where irreproducibility
is an operational risk rather than hygiene:
`dc0-edge-key`, `dc1-edge-key` (edge SSH -- SEC-007/-015 make this the ONLY management path
to the DC edges), `dc0-svc-key`, `dc1-svc-key` (rack service access), `dc0-maas-power-key`,
`dc1-maas-power-key` (MAAS -> libvirt power control), `office1-svc-key` (headend access).
The remaining rows (`office1-edge-root` password+hash, netbox, vault-ca-root,
rgw-multisite-system-key, `office1-env`, tailscale authkey, `maas-region-operator`) follow.

**`creds-mint.sh` STAYS QUEUED for its own ruling and is NOT built here.** It is orthogonal:
it would prevent the NEXT unregistered mint but does not make any existing key reproducible
and does not fix the staging. Bundling it would have made the urgent no-tool work wait on
unscoped tooling work -- it needs its own decision about what it mints, where it writes and
how it registers. D-137 proposal 2(a) remains the reference.

**Execution is a SEPARATE gated step.** Matrix edits ship with `creds-matrix.py` clean or
its findings explained, harness green, gauntlet ALL GREEN, repo-lint 0-fail, and a changelog
entry carrying a revert. NOTE the T24 caveat carried from the 2026-07-27 close: the
finding-class baseline covers TIER 1 ONLY, so a tier-2/3 regression introduced by these edits
would not turn the gauntlet red on its own.

---

## D-138: ADOPTED -- the cloud-facing client lives IN the DC; the Office1 transit stays management-terminating, not management-transiting [ARCH]

**Status: RULED 2026-07-30** (operator, GA-R5). Raised by the FIRST live `juju bootstrap`
of Stage 5, which failed on this and could not have succeeded without it.

**Question as presented.** Stage 5 could not proceed: `voffice1` -- the D-128 Plane-2
execution host, and therefore the Juju client -- has NO L3 path to any DC node plane, and
two deliberate controls forbid creating one. This is a contradiction between surfaces that
are BOTH ruled:

- **SEC-010 + D-052** make `metal-admin` DC-LOCAL. The SEC-010 row states it in writing --
  "Nothing routes across the fiber THROUGH vvr1-dc0; the rack proxies MAAS at the app layer
  (originated/terminated, not forwarded)" and "Region route must target only the rack
  transit /30, never 10.12.8.0/22". CLOSED 2026-07-20 on the operator's exact utterance
  "Close SEC-010 (Recommended)", applied and verified both ends by a named gate (EXIT 0,
  `docs/audit/stepB-check-20260720-final.txt`).
- **D-100** says the Office1<->DC fiber "carries management traffic only (MAAS/Juju/
  operator)" -- naming Juju explicitly.
- **D-128** puts the Plane-2 execution host, and so the Juju CLIENT, on `voffice1`.

**Operator answer, exact utterance: "Move the cloud-facing client into the DC
(Recommended)".**

**RULED (1) -- the client moves, the boundary does not.** Tools that dial the CLOUD at L3
run from INSIDE the DC: the Juju client (bootstrap, and every agent connection) and the
`openstack` CLI work in `phase-03`..`phase-06`. `voffice1` remains the Plane-2 host for
everything that is application-proxied or `qemu+ssh`-mediated -- MAAS, NetBox, the inner
`tofu` roots. SEC-010, D-052's DC-local invariant, and D-125's proven egress isolation are
UNCHANGED; nothing is punctured.

**RULED (2) -- why the contradiction was real and is now resolved.** SEC-010's cost was
priced with the sentence "a MAAS rack proxies at the application layer and needs no kernel
forwarding, so pinning is free." That is TRUE of every Plane-2 tool then in use and FALSE
of Juju, which dials the MACHINE at L3. **Juju was not in scope when the pin was priced.**
D-100's "carries Juju traffic" is preserved in the sense that matters: management traffic
crosses the fiber and TERMINATES on the rack; it does not TRANSIT the rack onto a node
plane. D-128's "Plane 2 runs on voffice1" is AMENDED to exclude cloud-facing tools.

**Scope was established by ENUMERATION before ruling, not discovered afterwards.** The gap
is not bootstrap-only: `scripts/phase-03-admin-openrc.sh` builds
`OS_AUTH_URL=https://${KEYSTONE_VIP}:5000/v3`, and keystone's VIP front-loads `10.12.4.50`
on PROVIDER-PUBLIC -- a second plane `voffice1` also cannot reach. Routing instead would
have meant opening two planes per DC across ports 22, 17070, 5000, 9292, 8774, 9696, 8776,
8778, 9876, 9311, 9511, 443 and more. That breadth is what made the client-side answer the
smaller change.

**CONSEQUENCE THAT MUST BE HANDLED AT EVERY DC STANDUP -- a per-DC cloud credential now
lands on a DC-local host.** The Juju client needs its DC's MAAS credential where it runs.
The `juju-vr1-dcN` API key is MAAS **admin-scoped** (SEC-018/-019: full machine/power/
deploy) and the MAAS region is SHARED across both DCs, so a DC-local host holding it has
region-wide MAAS admin. Two standing requirements follow: (a) each DC's client host
receives ONLY THAT DC's credential -- never a copy of the whole client credential store,
which would put dc1's credential on dc0's rack and destroy the SEC-018/-019 per-DC
isolation; (b) the residency is a NEW exposure and gets its own security-ledger row at the
standup that creates it. Roosevelt analog: the per-DC management bastion holds only its own
DC's cloud credential.

**Roosevelt analog.** Each DC has a management entry point inside it; the NOC reaches that
entry point, not every DC's admin plane directly. That is the more faithful model for bare
metal, where routing a headend onto every DC's admin network is exactly the flat-management
-plane design this project is rehearsing away from.

**Cross-notes.** AMENDS D-128 (Plane-2 host scope). PRESERVES SEC-010, D-052, D-125.
Resolves the `scripts/site-baseleg.sh:42,47` deferral -- the DC rows stay DEFERRED and
their `# MEASURE first` note is now ANSWERED: the measurement was taken 2026-07-30 and the
answer is that no host-side leg is wanted. Workflow-doc gap register item 20's DC half
CLOSES on this ruling rather than on a tool. Failure capture:
`docs/audit/stage5-bootstrap-reachability-20260730.txt`.

---

## D-132 -- AMENDMENT (2026-07-30): question 1 RULED FOR VR1 -- a MAAS REGION CONTROLLER PER DC, pulled forward from the next deployment [ARCH]

**Status: RULED 2026-07-30** (operator, GA-R5) for the VR1 deployment. D-132's questions 2
(rack-top controllers) and 3 (cross-site backup custody) remain PROPOSED and stay pinned to
the next deployment; only the per-DC region shape is ruled here.

**Question as presented.** The third Stage-5 bootstrap attempt got PAST the juju agent fetch
and then failed with `ERROR creating MAAS environ: Get
"http://10.10.0.20:5240/MAAS/api/2.0/version/": connect: connection refused`. `jujud` runs ON
the controller node INSIDE the DC and dials the MAAS region API continuously for its whole
life -- not only at bootstrap. MEASURED: the rack reaches that API (`code=200`) because
SEC-010 permits rack-ORIGINATED traffic; the NODE cannot, because that is FORWARDED traffic,
blocked by SEC-010's transit drop AND by libvirt's isolated-net rejects. The rack does NOT
proxy the region API -- `ss -lntp` shows nginx on **5248** (metadata) and nothing on 5240.

**The finding underneath the failure: SEC-010 as written is incompatible with the
deployment's own control-plane topology.** D-104 puts the Juju controller IN the DC; the MAAS
region lives at Office1; so the provider API dependency MUST cross the fiber at L3. D-100
says that fiber exists to carry exactly this traffic. All six flows the deployment needs were
enumerated and exactly ONE crosses the boundary (controller `10.12.8.5` -> `10.10.0.20:5240`);
the other five are same-segment or already-working rack-originated paths.

**Options put:** (a) a narrow SEC-010 exception for that single flow; (b) a MAAS region
controller per DC, removing the cross-fiber dependency entirely; (c) stop and re-plan the
control-plane topology.

**Operator answer, exact utterance: "Put a MAAS region controller in each DC".**

**RULED.** Each DC gets its OWN MAAS region controller, so the Juju controller's provider API
is DC-LOCAL and no MAAS traffic crosses the Office1 fiber. **SEC-010 is PRESERVED UNAMENDED**
-- this resolution removes the requirement rather than granting an exception to it, which is
why it was chosen over option (a). D-138 stands and is NECESSARY BUT WAS NOT SUFFICIENT: it
moved the cloud-facing CLIENT into the DC, but `jujud` is itself a provider client and its
MAAS dependency was not enumerated when D-138 was scoped. This amendment closes that gap.

**SINGLE, NOT HA, FOR VR1.** The operator's utterance is singular ("a MAAS region
controller"). D-132 question 1's HA sub-questions (member count, postgres HA, region API VIP)
stay PROPOSED for the next deployment. This mirrors the D-104 economy -- single-unit Juju
controller for VR1, HA deferred to Roosevelt.

**>>> ANNOTATION 2026-08-02 -- THE PARAGRAPH BELOW DESCRIBES THE ONE-TIME TRANSITION OF THE
TWO DCs THAT ALREADY EXISTED. IT IS NOT THE PER-DC STANDUP PROCEDURE. <<<**
Recorded per D-117's treatment rule (annotate ratified prose in place, do not rewrite).
**Operator correction 2026-08-02, verbatim: "MAAS region gets installed directly on the DC, it
is not migrated. If you have read this somewhere we need to make sure it is corrected so future
deployments don't follow a regional migration path."**
The words "re-enrol and re-commission 9 nodes" below are TRUE ONLY for dc0 and dc1, whose
machines were already enrolled in the OFFICE1 region when this amendment was ruled. **For any
DC stood up from here on -- and for Roosevelt -- the region controller is INSTALLED IN THE DC at
standup and its nodes ENLIST INTO IT FIRST TIME. There is no migration, no re-enrolment, and no
carve to recreate, because nothing was ever carved anywhere else.** A future session must NOT
read the list below as a standup checklist; the standup path is: install the region in the DC,
then enlist and commission normally. The `MAAS_PROFILE` / region-target parameters the last
sentence calls for are still required, because a per-DC region must be addressed explicitly --
that part generalises; the re-enrolment does not. Cross-reference:
`.claude/skills/openstack-cloud-ops/SKILL.md` carries the same correction at the head of its
(retained, historical) dc0-migration block.

**CONSEQUENCE, STATED PLAINLY: this REOPENS the MAAS layer of Stages 3 and 4, both of which
are CLOSED and MERGED to `main`.** A MAAS region owns its own database; machines, fabrics,
VLANs, subnets, spaces, reserved ranges, tags, images, DHCP and power configuration do not
migrate between regions. Per DC this means: install region + PostgreSQL; re-enrol and
re-commission 9 nodes; recreate the D-133 flat carve, the D-134 bands, the D-119 + role tags,
the six spaces and their VLANs; re-sync images; move DHCP; re-wire the SEC-012/-016 power
credentials; and re-point the Juju cloud endpoint. Much of this is already scripted
(`dc-rack-net.sh`, `maas-role-tags.sh`, `carve-host-interfaces.sh`, `maas-node-power.sh`, the
apex importer) but every one of those scripts currently assumes the single Office1 region and
will need a region-target parameter. **Office1's region is RETAINED for the Office1 site
itself** (`office1-netbox`, `office1-tailscale`).

**OPEN SUB-QUESTION, blocking the build and NOT ruled by this amendment: WHERE each DC's
region controller lives** -- on the DC rack host alongside its rack controller, or in its own
VM inside the DC. If a VM, the D-134 amendment's standing octet map requires a ruled octet in
the utility band (`.4` artifact service, `.5` Juju controller, so `.6` is the next free) and
that VM must be authored into the DC substrate root. Present as its own GA-R5 exchange before
any install.

**Roosevelt analog:** this IS the Roosevelt shape, arriving one deployment early -- per-DC
regions are what D-132 was filed to propose. The rehearsal value is real: the cross-fiber
provider-API dependency would have been discovered at Roosevelt scale otherwise.

## D-132 -- AMENDMENT ADDENDUM (2026-07-30): the per-DC region lives in its OWN VM at utility-band .6

**Status: RULED 2026-07-30** (operator, GA-R5), immediately following the per-DC region
ruling above. That ruling deliberately left placement open; this closes it.

**Question as presented.** Where does each DC's MAAS region controller live -- co-located on
the DC rack host, or in its own VM? Context put to the operator: the rack host already runs
the MAAS RACK controller, the D-131 node-DNS forwarder and the artifact service, AND is the
libvirt host for every node VM, so it is already the DC's busiest single failure point. A
dedicated VM follows the D-134 standing octet map (`.4` artifact service, `.5` Juju
controller, so `.6` next free) and needs authoring into that DC's substrate root via the same
additive `for_each` pattern that added the D-104 controller VM.

**Operator answer, exact utterance: "Dedicated region VM per DC at utility .6
(Recommended)".**

**RULED (1) -- placement.** Each DC's MAAS region controller (regiond + PostgreSQL) runs in
its OWN VM inside that DC, NOT on the rack host. Rationale carried from the question: the
region DATABASE must not share fate with the hypervisor running every node it manages. That
independence is a direct precondition for D-132 question 3 (cross-site backup custody), which
remains pinned -- a backup whose only copy dies with the rack host is not custody.

**RULED (2) -- the address, and it EXTENDS the D-134 standing octet map.** The region VM takes
utility-band octet **`.6`** on every plane it is attached to: `10.12.8.6` (dc0) and
`10.12.68.6` (dc1) on metal-admin. Per the D-134 amendment of 2026-07-29 the octet map is a
STANDING CROSS-DC STANDARD and divergence between DCs at the same octet is a DEFECT, so `.6`
is now reserved for the per-DC MAAS region at EVERY future DC standup. Running map:
`.4` artifact service (mirror or proxy), `.5` Juju controller, `.6` MAAS region.

**Build notes carried forward (not rulings).** The VM needs a provider-public leg for image
sync egress in addition to metal-admin -- the same under-carve that cost three bootstrap
attempts on the Juju controller VM; author BOTH legs at creation rather than discovering the
gap at first sync. Capacity must be re-measured with `scripts/dc-dc-whole-host-budget.py`
before applying: the D-104 amendment's gate passed at RAM 83% for the 10-node shape and this
adds a 12th domain per rack. The substrate caller is `for_each`-keyed, so the addition plans
as `1 add / 0 change / 0 destroy`.

## D-139: ADOPTED -- VR1 goes IPv6-ONLY on the east-west planes, and the whole plane carve moves to GUA [ARCH]

**Status: TWO RULINGS, BOTH 2026-07-31** (operator, GA-R5), taken in SEPARATE exchanges and
recorded separately -- neither is a batch adoption. Ruling A sets the FAMILY per plane;
ruling B sets the ADDRESSING MODEL. A is independent of B and would stand if B were
reversed.

**ANNOTATION 2026-08-08 (G18 ruling cross-ref -- see `docs/CURRENT-STATE.md` G18 row + the
D-101/R8 note):** ruling B carved an apex GUA `lb-mgmt` /64 (dc0 `2602:f3e2:f02:80::/64`) as
a NEW MAAS-underlay plane (execution step 5). MEASURED: octavia binds to NO `lb-mgmt` space
(bundle.yaml octavia bindings; network-space-binding-reference), so this apex GUA plane has
NO charm consumer and is a DISTINCT object from the charm's Neutron `lb-mgmt-net` (the ULA
`fc00::/64` the octavia charm creates and owns per R8). Per the G18 ruling (option b), this
D-139 apex GUA `lb-mgmt` /64 is kept **`reserved`** -- an architectural underlay allocation
with intent recorded and no live consumer (D-141 `reserved`-until-consumable discipline),
explicitly NOT the charm's management network. A future session must not conflate the two or
"fix" the empty plane by pointing octavia at it (that would reopen R8).

Raised by a live measurement session that overturned the recorded root cause of the
IPv4-only-container problem (see "What forced this" below). Admitted under GA-R3's A1 test:
a Roosevelt build session would grep this before carving a plane, and Roosevelt's
`2602:f3e2:103::/48` currently has NO plane carve at all -- whatever VR1 proves, Roosevelt
inherits.

### Ruling A -- the family matrix

**Question as presented.** Which planes keep IPv4, given the standing D-101 principle
"IPv6 unless IPv4 is NECESSARY"? A per-plane qualification was put to the operator naming,
for each plane, what actually forces v4: `metal-admin` (PXE/commissioning, the MAAS region
API `jujud` dials for life, the node-DNS forwarder, the entire v4-only D-134 utility band,
and a rack with NO global v6 at all); `provider-public` (tenant FIPs against a
still-substantially-v4 internet, external API VIPs, Octavia tenant LB VIPs, the edge default
route); `metal-internal` (D-101 keeps datastore east-west v4-bound, but as a JUDGEMENT --
"marginal benefit, real cert cost" -- not a protocol requirement); and `data-tenant`,
`storage`, `replication`, `lb-mgmt` (nothing found forcing v4 -- all east-west between hosts
that already hold v6, no external clients, no gateway needed on-link).

**Operator answer, exact utterance: "IPv6 on all planes except for metal-admin and
provider-public which will remain dual stack".** (Corrects a mistyped earlier form; the
operator issued the correction unprompted after the ambiguity was put to them rather than
guessed at.)

**RULED.** Per DC:

| Plane | IPv4 | IPv6 | Family |
|---|---|---|---|
| `provider-public` | RETAINED | yes | dual-stack |
| `metal-admin` | RETAINED | yes | dual-stack |
| `metal-internal` | REMOVED | yes | **IPv6-only** |
| `data-tenant` | REMOVED | yes | **IPv6-only** |
| `storage` | REMOVED | yes | **IPv6-only** |
| `replication` | REMOVED | yes | **IPv6-only** |
| `lb-mgmt` | n/a | yes | **IPv6-only** (NEW plane) |

**This AMENDS D-101's family matrix in two places** and D-101 is otherwise unchanged:
(i) `metal-internal`'s "IPv4 retained ... datastore east-west (MySQL / RabbitMQ stay
v4-bound)" is SUPERSEDED -- that plane is now v6-only; (ii) `lb-mgmt` was already ruled
IPv6-only by D-101 but had never been carved ANYWHERE, and is now a first-class plane.
D-101's governing principle is unchanged and this is its application, not its revision.

### Ruling B -- the addressing model

**Question as presented.** Three schemes were put with their full costs: (1) full GUA on
every plane; (2) ULA/GUA by plane allocation -- GUA on the two dual-stack planes, ULA on the
five v6-only ones, on the reasoning that RFC 6724 precedence only bites where both families
compete; (3) keep the split exactly as carved, GUA on `provider-public` only.

**Operator answer, exact utterance: "Full GUA on every plane (Recommended)".**

**RULED.** Every VR1 plane is carved from its DC's GUA `/48` -- `2602:f3e2:f02::/48` (dc0),
`2602:f3e2:f03::/48` (dc1) -- following the octet map VR0 DC0 and the Willamette site
ALREADY use, so this conforms to an existing org standard rather than inventing one:

| Plane | `/60` parent | dc0 `/64` | dc1 `/64` |
|---|---|---|---|
| `provider-public` | `f0X:10::/60` | `2602:f3e2:f02:10::/64` | `2602:f3e2:f03:10::/64` |
| `provider-public` VIP | (in `:10::/60`) | `2602:f3e2:f02:11::/64` | `2602:f3e2:f03:11::/64` |
| `metal-admin` | `f0X:20::/60` | `2602:f3e2:f02:20::/64` | `2602:f3e2:f03:20::/64` |
| `metal-internal` | (in `:20::/60`) | `2602:f3e2:f02:21::/64` | `2602:f3e2:f03:21::/64` |
| `data-tenant` | `f0X:30::/60` | `2602:f3e2:f02:30::/64` | `2602:f3e2:f03:30::/64` |
| `storage` | `f0X:40::/60` | `2602:f3e2:f02:40::/64` | `2602:f3e2:f03:40::/64` |
| `replication` | `f0X:50::/60` | `2602:f3e2:f02:50::/64` | `2602:f3e2:f03:50::/64` |
| `lb-mgmt` | `f0X:80::/60` | `2602:f3e2:f02:80::/64` | `2602:f3e2:f03:80::/64` |

**THE ULA `/48` `fd50:840e:74e2::/48` IS RETIRED FOR VR1.** It remains a valid org aggregate
in the apex; it simply carries no VR1 plane. **This AMENDS D-101**, whose text names that
ULA prefix as a literal, and D-111.

**>>> THE ORIGINAL DECIDING REASON IS STRUCK -- REFUTED BY MEASUREMENT 2026-08-01. DO NOT
CITE IT. <<<** It read: *"RFC 6724's default policy table ranks IPv4-mapped
(`::ffff:0:0/96`) at precedence 35 and ULA (`fc00::/7`) at 3, while GUA falls under `::/0` at
40. On a DUAL-STACK plane a ULA leg therefore LOSES address selection to IPv4 and is largely
decorative, whereas GUA WINS."* **That is true of the RFC and FALSE of the software this
cloud runs.** glibc 2.35 -- jammy's `libc-bin 2.35-0ubuntu3` -- implements the **RFC 3484**
table, as glibc's own in-source comment states, and it has **NO `fc00::/7` entry at all**:
ULA falls through to `::/0` = **40**, identical to GUA, while IPv4-mapped is **10**. So ULA
and GUA are EQUAL and BOTH outrank IPv4. The shipped `/etc/gai.conf` carries ZERO active
lines, so nothing overrides it. The `fc00::/7` entry that does exist is in `default_labels[]`
(label 6) and drives SOURCE-selection rules 5/6 -- a different mechanism; do not conflate the
two into a rescue of the struck argument. Method and full table:
`docs/audit/gai-conf-rfc6724-verification-20260801.txt`.

**THE RULING STANDS on the reasons below -- see RULING NOTE 2026-08-01 at the end of this
entry.** Those reasons: conformance with Willamette (a REAL site) and VR0 DC0, both full GUA;
and MINIMIZE DELTA TO ROOSEVELT, which holds a GUA `/48` with no ULA and no plane carve yet.

**Isolation is NOT weakened, and this was checked rather than assumed.** Containment lives
at the FORWARDING layer -- SEC-010's transit drop, D-125's proven egress isolation, D-107's
airgap posture -- and the DC edge carries no v6 transport at all. GUA addressing creates no
reachability without a route. SEC-010, D-052, D-125 and D-107 are all UNCHANGED.

### What forced this -- the measurement that overturned the recorded cause

`docs/CURRENT-STATE.md` recorded that containers came up v4-only because "MAAS ... has
nothing to give for v6", the v6 subnets carrying no IP ranges. **MEASURED and WRONG:**
`subnet statistics` on `fd50:840e:74e2:220::/64` returns `available_string "100%"`,
`num_available 18446744069414584320`. Zero explicit `ipranges` rows means zero
RESTRICTIONS, not zero availability -- and `scripts/dc-plane-ipam.sh:368-372` had recorded
the correct MAAS behaviour since 2026-07-27, in the tree, contradicting the authoritative
document.

**The real mechanism is juju-side and there is NO knob.** `state/linklayerdevices.go`
(`EthernetDeviceForBridge`, pinned at tag `v3.6.27`) takes `addrs[0]` -- ONE address, from an
UNSORTED mongo query -- and derives exactly one subnet CIDR, which the MAAS provider turns
into a single `LinkSubnetArgs`. There is zero family-awareness anywhere on that path and no
config, constraint or binding that changes it. gomaasapi's own doc states "Any number of
STATIC links can exist on an interface", so **MAAS would accept both families; the limit is
juju's.** LP #1723240 (`Triaged`/`Low`, open since 2017-10-12) is this exact symptom, and the
theory recorded in CURRENT-STATE is the one its reporter rebutted in-thread.

**CONSEQUENCE that makes ruling A the achievable configuration rather than merely the
desired one: on a plane consumed by LXD containers, DUAL-STACK IS NOT EXPRESSIBLE** -- juju
picks one family non-deterministically. A v6-ONLY plane IS expressible. **CARRIED RISK,
reaffirmed by the operator after being put to them:** `metal-admin` stays dual-stack AND
carries containers, so each container's leg lands on one family by mongo order (today v4).
If it ever flipped to v6 the container would lose its path to the v4-only mirror, snap proxy
and MAAS region. No knob exists, so the mitigation is DETECTION -- a Stage-5 gate asserting
every container's `metal-admin` leg is IPv4. Owed, not built.

### Build constraints measured before this was drafted

- **MAAS auto-reserves `::1`-`::ffff:ffff` on EVERY IPv6 `/64`** (MAAS 3.7
  `subnet_utilization.py`; recorded at `dc-plane-ipam.sh:368-372` since 2026-07-27). Explicit
  STATIC assignment inside that block works -- it is how the existing 54 node statics live at
  `::100` etc. AUTO allocation draws from OUTSIDE it. **So juju containers on v6-only planes
  will receive `<prefix>:0:1::`-shaped addresses, NOT the D-134 textual mirror.** No v6
  `ip-range` rows are needed or possible; MAAS refuses ranges inside its own reserved block
  and the band is already protected more broadly than the range would be.
- **All six existing v6 subnets carry `gateway_ip=None`** (measured). Correct for the
  on-link east-west planes. Two exceptions are OPEN, not ruled here: `provider-public` (the
  edge has no v6 transport; D-101 defers external v6 routing to the next deployment) and
  `replication`'s CROSS-DC leg, which has no v6 route.
- `ceph-mon`, `ceph-osd` and `mysql-innodb-cluster` are all in `PREFER_IPV6_CHARMS`, so
  `storage`, `replication` and `metal-internal` going v6-only REOPENS the 2026-07-31 D-101
  ruling note that set `prefer-ipv6` false on every application. That note was correct for a
  cloud with no operational v6; it must be revisited per-app as each plane converts.

### Execution -- NOT part of this ruling, and none of it is done

Ordered, each individually gated: apex push (prefixes + retire the VR1 ULA rows); MAAS carve
per DC; re-carve 54 node v6 statics (`dc-node-v6-carve.py`); reissue the Octavia v6 IP SANs
(`octavia-pki.sh reissue` -- the rotation tool this is exactly the case for); update
`lib-net.sh`'s v6 arm and its harness; carve `lb-mgmt` a VLAN/space/subnet, which it has
never had; remove the v4 subnets from the five v6-only planes LAST, after each is proven.

### RULING NOTE 2026-08-01 -- D-139: ruling B RECONSIDERED after its deciding reason was refuted, and CONFIRMED

**Why this note exists.** D-139 ruling B ("Full GUA on every plane") was recorded on
2026-07-31 WITH an explicitly-named unverified premise: the deployed jammy image's
`/etc/gai.conf`. Closing that owed item on 2026-08-01 **refuted the ruling's stated deciding
reason** -- glibc 2.35 implements RFC 3484, not RFC 6724, so ULA and GUA are EQUAL at
precedence 40 and both outrank IPv4 at 10 (capture
`docs/audit/gai-conf-rfc6724-verification-20260801.txt`; the struck text is marked above).

Under GA-R5 a ruling is the operator's, so the refutation was put BACK to them rather than
the ruling being silently kept because the answer was still liked, or silently reversed.

**Question as presented.** The RFC 6724 precedence argument -- the reason given as deciding
-- is refuted for this deployment; ULA and GUA are behaviourally equivalent for address
selection on jammy. Three reasons survive and are independent of it: conformance with
Willamette (a REAL site) and VR0 DC0, both full GUA; MINIMIZE DELTA TO ROOSEVELT, whose
`2602:f3e2:103::/48` is GUA with no ULA and no plane carve yet; and isolation being
unaffected either way, since containment lives at the forwarding layer. Does ruling B stand
on those alone, or is it revisited? Nothing of D-139 is executed, so reversal costs nothing
but the exchange.

**Operator answer, exact utterance: "Deciding to hold to gua does not cost anything
operationally and the case for ULA over gua is not very strong. Let's stay with gua".**

**RULED: D-139 ruling B STANDS, UNCHANGED IN SUBSTANCE.** The GUA carve in ruling B's table
is confirmed; the ULA `/48` `fd50:840e:74e2::/48` remains RETIRED for VR1; the D-101 and
D-111 amendments stand.

**What changed is the RATIONALE OF RECORD, and this matters for anyone citing it later.**
Ruling B now rests on exactly two arguments, both of which are project-constraint arguments
rather than protocol ones:
1. **Conformance** -- Willamette and VR0 DC0 already carve GUA on every plane; VR1 was the
   outlier.
2. **Minimize delta to Roosevelt** -- the governing design constraint. Roosevelt's carve does
   not exist yet, so whatever VR1 proves it inherits.

Plus the operator's own recorded reasoning: **holding to GUA costs nothing operationally, and
the case for ULA over GUA is not strong.** The refuted precedence argument must NOT be
re-cited in support of this or any successor decision.

**A CONSEQUENCE THAT IS NOW EXPLICIT AND WAS NOT BEFORE:** because ULA and GUA are equal at
precedence 40 on glibc 2.35, choosing GUA buys **no address-selection advantage** on the two
dual-stack planes. It buys conformance and Roosevelt fidelity. Anyone later reasoning that
"GUA was chosen so v6 would win selection over v4" is reasoning from the struck argument and
is wrong -- on this glibc, ULA would have won equally.

**GENERALISABLE, and it is a NEW variant of this repo's citation rule.** The standing rule is
"a citation is an EXISTENCE claim; only its CONTENT is evidence" -- open the source, check its
STATUS and DATES. Here the citation was real, current, correctly quoted AND correctly
understood, and still wrong, because **nobody checked whether the IMPLEMENTATION follows the
standard.** For a STANDARDS citation, add a third check: confirm the deployed software
implements it, at the deployed version. An RFC is not a description of your system.

### CORRECTION NOTE 2026-08-01 -- D-139: the EXECUTION list was defective; replaced

**Not a ruling and not a change to either ruling.** D-139's rulings A and B are untouched;
this replaces the ORDERED EXECUTION list at the end of the entry, which was authored with
three defects, one of them a silent-failure class. The original text is preserved here so a
later reader can see what was wrong rather than wondering why it changed.

**ORIGINAL, now SUPERSEDED:** *"apex push (prefixes + retire the VR1 ULA rows); MAAS carve per
DC; re-carve 54 node v6 statics (dc-node-v6-carve.py); reissue the Octavia v6 IP SANs; update
lib-net.sh's v6 arm and its harness; carve lb-mgmt a VLAN/space/subnet; remove the v4 subnets
from the five v6-only planes LAST, after each is proven."*

**DEFECT 1 -- "remove v4 LAST, after each is proven" is UNSATISFIABLE.** Per-plane conversion
is atomic: there is no state in which a plane is proven v6-only while it still holds v4. The
clause reads like caution and cannot be executed.

**DEFECT 2 -- SILENT FAILURE, and this is the consequential one.
`scripts/dc-node-v6-carve.py` is STRUCTURALLY DEPENDENT ON IPv4 EXISTING.** Verified against
its source: it assigns v6 "on every plane **where it already carries IPv4**", selects the v6
subnet "on the SAME MAAS vlan as that v4 link", derives the host part from "the last octet of
the node's own v4 address", and its loop body is a bare `if not v4: continue`. **Under D-139
five planes lose v4 and `lb-mgmt` never had a v4 twin, so run AFTER v4 removal it would carve
FOUR FEWER PLANES PER NODE AND EXIT CLEAN.** Ordering the v4 removal before the v6 carve --
which the original list's "LAST" wording invited -- would have produced a silently incomplete
carve that every gate reads green.

**DEFECT 3 -- the apex RETIRE half is not safe in the same step as CREATE.** The dry run
measured **52 dependent ip-addresses** inside the retiring ULA `/64`s (26 per DC, the v6 VIP
legs). Retiring the prefixes before those addresses are re-homed orphans them.

**REPLACEMENT LIST. Each step individually gated; each is a no-op for the next if skipped.**
1. **Apex push, CREATE ONLY.** `netbox/d139-gua-carve.py` without the retire half. The tool
   already reports retirement rather than executing it; keep it that way until step 6.
2. **MAAS carve per DC** -- the GUA `/64`s alongside the existing ULA, not replacing them.
   Both families coexist in MAAS at this point; nothing on a node has changed.
3. **Re-carve node v6 statics onto the GUA prefixes WHILE v4 IS STILL PRESENT.** This is the
   ordering defect 2 forces. `dc-node-v6-carve.py` works here because every plane still carries
   its v4 twin. **If it is ever to run after v4 removal it must first be rewritten to pivot on
   the PLANE (space/vlan) rather than on the presence of a v4 link.**
4. **Reissue the Octavia v6 IP SANs** (`octavia-pki.sh reissue` -- the rotation counterpart
   this is exactly the case for), then update `lib-net.sh`'s v6 arm and its harness.
5. **Carve `lb-mgmt` its VLAN/space/subnet.** It has never had one anywhere, in either family.
6. **Re-home the 52 VIP ip-addresses onto the GUA prefixes, THEN retire the ULA rows** in the
   apex and in MAAS. Only now is the retire half safe.
7. **v4 removal is a SEPARATE, EXPERIMENTAL step and is NOT part of this list.** Per the
   2026-08-01 "B plus C" ruling it applies to `storage` + `replication` ONLY -- and to BOTH
   TOGETHER, because `ceph-osd`'s `prefer-ipv6` sets `ms_bind_ipv4 = False` globally and so
   couples Ceph's public and cluster networks. `metal-internal`, `data-tenant` and `lb-mgmt`
   keep v4 pending upstream fixes.

**PREREQUISITE FOR STEP 7, STILL NOT TAKEN:** does `network-get` return a v6 address on a
v6-only bound space? It needs a deployed unit, so it rides the next deploy. Step 7 must not
start before it is answered.

### ORDERING RULING 2026-08-01 -- D-139 execution list vs the Stage-5 deploy

**Not a change to ruling A or B, and not a new D-number** (GA-R3: execution ordering is
OPS; it is recorded here because it sits against D-139's own execution list and against a
standing operator directive, and a future session would otherwise read the two as
conflicting).

**The conflict, stated as it was put.** The 2026-07-30 standing directive reads "we have to
continue to juju deployment next session no matter what". D-139's replaced execution list
(the CORRECTION NOTE above) puts the apex push, the MAAS carve and the node re-carve BEFORE
anything else. Deploying first is cheapest now and yields the `network-get`-on-a-v6-only-space
measurement that step 7 is blocked on, since that needs a deployed unit -- but it means steps
2-6 later re-address nodes UNDER a live cloud, and D-139 records that addresses bake into every
unit's `agent.conf` at deploy time.

**Operator answer, exact utterance: "Apex push AND the MAAS/node GUA carve, then deploy".**

**CONSEQUENCE.** D-139 execution steps **1, 2 and 3** run BEFORE the Stage-5 bundle deploy, so
the cloud is deployed ONCE, on its final GUA addresses, and nothing is re-addressed underneath
it. The standing directive is thereby satisfied in substance rather than literally: the deploy
remains the goal and nothing else is admitted ahead of it, but it is sequenced after the carve
that would otherwise have to be redone through it.

**INTERPRETATION STATED SO IT CAN BE CORRECTED RATHER THAN ASSUMED.** The options as presented
named steps 1-3. Steps **4-6** (Octavia v6 SAN reissue + `lib-net.sh`'s v6 arm; the `lb-mgmt`
VLAN/space/subnet; re-homing the 52 VIP ip-addresses and only then retiring the ULA rows) were
NOT put, and at least step 6 has a deploy coupling that this ruling does not settle: the
per-DC VIP overlays carry v6 VIPs in the ULA range, so deploying after steps 1-3 alone would
place a cloud whose NODES are GUA while its declared v6 VIPs are still ULA. That needs its own
GA-R5 exchange before Step 4, and is NOT inferred from this ruling.

**Step 7 (v4 removal) is unchanged and still out of this list**, still scoped to `storage` +
`replication` together per the "B plus C" ruling, and still gated on the unmeasured
`network-get` question -- which now arrives later, since the deploy that produces it is
sequenced after the carve.

### AMENDMENT 2026-08-01 -- D-139 ruling A gains an OOB plane, DUAL-STACK, and D-058's `10.12.60.0/22` is ruled back into force for it

**Operator direction, exact utterance: "make sure to include oob in the dual stack recordings"**,
then on the family/value question: **"Dual-stack; rule 10.12.60.0/22 back into force for OOB"**.

**WHY IT WAS MISSING, measured rather than assumed.** D-139's carve table (`netbox/d139-gua-carve.py`
`CARVE`) carried hextets `10,11,20,21,30,40,50,80` and **no `0xf0` (OOB) and no `0xe0` (VPN)**.
`docs/design-decisions.md:3066` is the origin: `lbaas-mgmt (:80)`, `vpn (:e0)` and `oob (:f0)` were all
declared OUT OF SCOPE of the six-plane mirroring tool. D-139 brought `:80` back as a first-class plane
and left `:e0`/`:f0` behind. The apex already carries `oob` and `vpn` IPAM roles, so nothing blocked
them. **Had this not been caught, the step-1 push would have written a carve that is incomplete against
the very org standard ruling B cites as its reason.**

**WHAT THE PRECEDENTS ACTUALLY HOLD, and the ruling KNOWINGLY DIVERGES FROM THEM.** VR0 DC0
(`2602:f3e2:e02:f0::/60` + `:f0::/64`) and Willamette (`2602:f3e2:102:f0::/60` + `:f0::/64`) both carry
OOB **IPv6-ONLY**; a query across every apex prefix returns **ZERO** v4 rows with role `oob`, at any
site. So dual-stack OOB is NOT conformance -- it is a deliberate correction, and the Roosevelt reason
is sound: BMC/IPMI is overwhelmingly IPv4 in practice, so a v6-only OOB plane would be unusable on real
hardware. VR1 proves it and Roosevelt inherits it. This also SUPERSEDES the standing
`| OOB | n/a | n/a | n/a | Bare-metal-only concern |` row at `docs/design-decisions.md:93`: OOB is no
longer n/a for the virtual rehearsals.

**ON THE v4 VALUE, stated precisely so no one reads this as reviving a dead decision.** `10.12.60.0/22`
appears at `docs/design-decisions.md:924` inside **D-058, which is SUPERSEDED by D-060** and whose plane
renumber is abandoned. **D-058 remains superseded in full.** This ruling reinstates that one ROW'S VALUE
under D-139's own authority; it does not revive D-058 or any other row of it. MEASURED FREE before the
ruling was applied: the apex holds `10.12.64.0/22` and `10.12.68.0/22` (both `vr1-dc1`) and nothing in
`10.12.60.0`-`10.12.63.255`; dc0's MAAS region holds six v4 subnets, none of them in that range.

**RULED, amending ruling A's family matrix:**

| Plane | `/60` parent | dc0 `/64` | dc1 `/64` | IPv4 | Family |
|---|---|---|---|---|---|
| `oob` | `f0X:f0::/60` | `2602:f3e2:f02:f0::/64` | `2602:f3e2:f03:f0::/64` | `10.12.60.0/22` | **dual-stack** |

**OPEN, NOT INFERRED -- the per-DC v4 split.** `10.12.60.0/22` is ONE block and this deployment carves
v4 PER DC (dc0 `10.12.{4,8,12,16,32,36}.0/22`; dc1 `10.12.{64,68,72,76,80,84}.0/22`, a mapping that is
NOT a uniform offset -- 4->64/8->68/12->72/16->76 are +60 but 32->80/36->84 are +48, so no rule can be
derived from it). Whether dc0 and dc1 split `10.12.60.0/22` into two `/23`s, or dc1 takes a separate
block, is NOT settled by this ruling and must not be guessed. **The v6 half is unaffected and complete**
-- each DC's OOB `/64` comes from its own `/48` by construction. **The apex push (step 1) is v6-only, so
it is NOT blocked by this open item.**

**VPN (`:e0`) IS THE SAME OMISSION AND IS NOT RULED HERE.** VR0 and Willamette both carry
`:e0::/60` with role `vpn`; VR1 has none. It was surfaced alongside OOB and the operator's direction
named OOB only, so it stays OPEN rather than being folded in silently.

### AMENDMENT 2026-08-01 (b) -- OOB's IPv4 per-DC allocation: dc0 `10.12.40.0/22`, dc1 `10.12.88.0/22`

**Operator answer, exact utterance: "dc0 10.12.40.0/22, dc1 10.12.88.0/22 (Recommended)"**, given
after asking for "an oob recommendation that has a similar pattern" to the existing per-DC carve.

**THIS SUPERSEDES `10.12.60.0/22`** from amendment 2026-08-01 (a) above. That value came from
D-058 (itself SUPERSEDED by D-060) and was ruled back into force before the allocation pattern
had been measured. D-058 remains superseded in full; `10.12.60.0/22` now carries NO VR1 role and
returns to the free pool.

**THE MEASURED PATTERN, because the obvious rule is not the real one.** Per-DC v4 today:

    dc0:  4,  8, 12, 16, [20 24 28 FREE], 32, 36
    dc1: 64, 68, 72, 76,                  80, 84     <- contiguous, no gap
    offset: +60 +60 +60 +60               +48 +48

**The dc0->dc1 offset is a CONSEQUENCE, not a rule:** dc0 skipped 20-28 (historical, D-052/D-058
era), dc1 never reproduced the skip, so the last two planes drifted from +60 to +48. The only
uniform property is that each DC appends in ascending order and dc1 has stayed contiguous.

**dc0's 20-28 gap is UNUSABLE for a symmetric pair, measured rather than assumed:** `20+60 = 80`
and `20+48 = 68` are both allocated, so no free counterpart exists for it in either offset.

**WHY THIS PAIR.** `40` and `88` are each the next free `/22` immediately after their own DC's
highest plane (36 -> 40; 84 -> 88), so both runs stay contiguous and dc0's historical gap is left
alone. The offset is **+48, exactly what storage and replication already use** -- it continues the
current relationship instead of reverting to the older +60. Against the superseded `10.12.60.0/22`:
`60` covers 60-63 and abuts dc1's block at 64, which would make it the only dc0 plane touching
dc1's range, and its +60 partner `120` sits far outside dc1's contiguous 64-88 run.

**RULED, completing the OOB row of D-139 ruling A's family matrix:**

| Plane | dc0 IPv4 | dc1 IPv4 | dc0 IPv6 | dc1 IPv6 | Family |
|---|---|---|---|---|---|
| `oob` | `10.12.40.0/22` | `10.12.88.0/22` | `2602:f3e2:f02:f0::/64` | `2602:f3e2:f03:f0::/64` | dual-stack |

**STATE: the v6 half is BUILT** (apex push 2026-08-01, `f02:f0::/60` + `f02:f0::/64` live for dc0;
dc1 not yet pushed). **The v4 half is RULED AND NOT BUILT** -- neither `/22` exists in the apex or
in MAAS. `netbox/d139-gua-carve.py` cannot create them: it is v6-prefixes-only by construction.
The v4 rows are OWED and gate nothing in D-139 or the Stage-5 deploy.

### DEFERRAL 2026-08-01 -- VPN (`:e0`) is knowingly ABSENT from VR1's carve, pinned to Roosevelt design time

**Operator answer, exact utterance: "Defer to Roosevelt design time".**

**RECORDED RATHER THAN LEFT SILENT, and that is the whole point of this note.** VPN was the
SECOND omission found alongside OOB in D-139's `CARVE` table (`design-decisions.md:3066` put
`lbaas-mgmt (:80)`, `vpn (:e0)` and `oob (:f0)` all out of scope of the six-plane tool; D-139
brought `:80` back, the 2026-08-01 amendment brought `:f0` back, and `:e0` is now deferred BY
DECISION). The OOB gap survived undetected precisely because nothing recorded it; this note
exists so a future session finds a ruled deferral instead of rediscovering a hole.

**MEASURED STATE at the time of the ruling, so it does not have to be re-derived:**

| Site | VPN prefix | Shape |
|---|---|---|
| VR0 DC0 | `2602:f3e2:e02:e0::/60` | **`/60` parent ONLY -- no `/64`** |
| Willamette | `2602:f3e2:102:e0::/60` | **`/60` parent ONLY -- no `/64`** |
| VR1 dc0 / dc1 | none | -- |
| (global, unscoped) | `fd7a:115c:a1e0::/48` role `vpn` | "Tailscale Addresses" |

**VPN IS A RESERVATION AT BOTH PRECEDENTS, NEVER A CARVED PLANE** -- neither holds a `/64`,
and neither holds any IPv4. That is a different shape from OOB, which holds `/60` + `/64` at
both and was therefore a real gap.

**WHY DEFERRAL IS COHERENT AND NOT AN OVERSIGHT.** VR1's VPN is Tailscale under D-129(iii)
(operator: "Dedicated node per site"), and Tailscale supplies its own addressing from
`fd7a:115c:a1e0::/48` -- a range this deployment does not carve. A VR1 `:e0::/60` would
therefore reserve space with no defined occupant. A real site-to-site VPN is a Roosevelt
concern, which is where the ruling pins it.

**CONSEQUENCE, stated plainly: VR1's octet map diverges from VR0 and Willamette on exactly one
hextet (`:e0`), by decision.** D-139 ruling B's "conforms to an existing org standard" claim
should be read with that one documented exception. Nothing in D-139's execution list, the
Stage-5 deploy, or any gate depends on `:e0`.

**TRIGGER:** Roosevelt VPN/site-to-site design. Present alongside D-132 (per-DC MAAS topology)
and D-131 sub-4 (pinned DNS review), which are pinned to the same design window.

### AMENDMENT 2026-08-02 -- D-135: dc0 CONVERGES ON THE PROXY AT ITS NEXT REBUILD

**Operator ruling, exact utterance: "if we have to rebuild in dc0 for any reason we will be
using a proxy rather than a full mirror rebuild."**

**RULED.** The dc0 full mirror is NOT reconstructed. Should dc0's artifact service be rebuilt
for any reason, it comes back as an **apt caching proxy** on the dc1 pattern
(`scripts/dc-cache-proxy.sh`, apt-cacher-ng on the D-134 utility `.4:3142`, consumed via
`juju apt-http-proxy`). Both DCs then run the same strategy.

**THIS CHANGES A STANDING TERM OF D-135 AND SAYS SO PLAINLY.** D-135's amended text describes
the dc0-mirror / dc1-proxy split as "the EXPERIMENT, not an interim state -- there is no
fallback and no later convergence". **That clause is now qualified: there is no convergence
while the mirror stands, but a rebuild converges rather than reinstates.** The experiment is
not abandoned -- it ran, and its arms are compared below.

**WHY THIS IS THE RIGHT CALL ON THE EVIDENCE ALREADY IN HAND.** The mirror arm cost 953 GB of
rack storage, a nightly `debmirror` window, an upstream dependency that fails independently of
the DC (2026-08-02: `500 Can't connect to ubuntu-cloud.archive.canonical.com`), and a checker
defect that read CLEAN over a FAIL status until it was fixed on 2026-07-27. The proxy arm
carries none of that: no bulk storage, no sync window, no attested-currency problem, and a
cache miss degrades to a fetch rather than to stale content.

**WHAT MUST BE CAPTURED BEFORE THE MIRROR GOES, or the experiment produced nothing.** D-135
existed to compare the two strategies. If the mirror is dismantled without recording its
outcome, the comparison is lost and Roosevelt inherits a preference with no evidence behind it.
Owed at the rebuild: the measured storage cost, the sync-failure modes seen, the currency-gate
defect class, and the observed behaviour of each arm under the 2026-08-01 outage (the mirror
kept SERVING stale-but-valid content with its upstream gone; a proxy would have failed closed
on a miss). That comparison is the experiment's deliverable.

**CONSEQUENCE FOR THE RUNBOOK FOLD:** the phase runbooks can now carry ONE artifact strategy
(the proxy) with the mirror recorded as the superseded experiment arm, instead of branching
per DC. `dc-mirror.sh` is retained for the historical arm and for the currency lesson its
`check` encodes; `dc-mirror.sh install dc0` becomes a deliberate strategy REVERSAL and never a
repair -- the same warning dc1's row already carries.

**UNCHANGED:** dc1 is unaffected (it was already the proxy arm). Nothing here touches the
currently-running dc0 mirror, which is verified intact (236/236 + 2/2 indices, 2026-08-02).

### AMENDMENT 2026-08-02 (b) -- D-135: dc0 CONVERGES NOW; THE EXPERIMENT IS COMPLETE AND ITS RESULT IS INFORMATIONAL ONLY

**Operator ruling, exact utterance: "Option A, converge. We have gathered the information we
need on the mirror buildout. It will be a future implementation but will not be used in
Roosevelt but much later so keep the information as informational only with no pin or
expectation on a future deployment date."**

**QUESTION AS PRESENTED.** The dc0 Stage-5 bundle deploy failed with nine units at
`hook failed: "install"`, root cause `NO_PUBKEY 5EDB1B62EC4926EA` -- the mirrored UCA is
reachable and its signatures intact, but no cloud-archive keyring reaches a freshly deployed
node, because the 2026-07-31 repoint replaced `cloud:jammy-caracal` with a raw `deb` line and
the `cloud:` path is what installs `ubuntu-cloud-keyring`. The as-built comparison
(`docs/audit/uca-origin-asbuilt-comparison-20260802.txt`) showed the only configuration this
project has ever deployed successfully is `cloud:jammy-caracal` on six apps, with ZERO raw
`deb` lines and ZERO `key:` options in any capture, and that the PROXY arm needs no override
at all. Options put: **(A)** converge dc0 on the proxy now and DELETE the origin block,
returning to the as-built configuration; **(B)** keep the mirror and restructure the armored
signing key behind a single YAML anchor.

**RULED (A).** dc0's artifact strategy becomes the apt caching proxy
(`scripts/dc-cache-proxy.sh`, apt-cacher-ng on the D-134 utility `.4:3142`, consumed via
`juju model-config apt-http-proxy`), on the dc1 pattern. **Both DCs now run ONE strategy.**
The per-DC origin override is DELETED rather than repaired: the proxy forwards whatever URL it
is handed, so `cloud:jammy-caracal` -- the as-built-proven value, defined once behind
`bundle.yaml`'s `&openstack-origin` / `&ceph-source` anchors -- works unmodified, and the
signing-key problem does not exist to be solved.

**THIS SUPERSEDES THE TRIGGER CLAUSE OF THE AMENDMENT ABOVE.** That amendment made convergence
conditional on a REBUILD of dc0's artifact service. This one converges on the ruling itself,
without waiting for a rebuild.

**THE EXPERIMENT IS COMPLETE -- "we have gathered the information we need on the mirror
buildout".** The comparison the amendment above lists as "owed at the rebuild" is therefore
owed NOW, and it is the experiment's deliverable. It is recorded as EVIDENCE, and its status
is set by this ruling:

**>>> INFORMATIONAL ONLY. NO PIN. NO EXPECTED DEPLOYMENT DATE. NOT A ROOSEVELT ITEM. <<<**
The operator's words are explicit: the mirror "will be a future implementation but will not be
used in Roosevelt but much later". So this material carries **no trigger, no owed work, no
review date, and no forward commitment.** It is retained because the comparison is worth
having, not because anything is scheduled on it. **A later session must NOT convert this into
a pinned item, a gap-register row, a deferred-forward item with a trigger, or a Roosevelt
delta** -- doing so would manufacture an expectation the operator explicitly declined to
create. `ledger-scan` and the forward-items register must stay clean of it.

**WHAT `dc-mirror.sh` IS NOW.** Retained for the historical arm and for the currency lesson its
`check` encodes (assert on CONTENT, never on existence; an unrecognised state REFUSES). It is
NOT deleted. `dc-mirror.sh install <site>` is a deliberate strategy REVERSAL, never a repair.

**NOT RULED HERE, and deliberately left alone:** whether the running dc0 mirror is torn down,
and when its 953 GB is reclaimed. Convergence is about which strategy the DEPLOY consumes;
dismantling the mirror is a separate destructive action needing its own approval. The mirror
may stand indefinitely without affecting this ruling.

## D-140: PINNED -- OpenTofu management of the Juju layer, after a hardened and tested dc0 deployment [ARCH]

**Status: PINNED 2026-08-02 to the end-of-deployment review.** Operator ruling, exact
utterance: **"I accept the opentofu plan. Hardened dc0 deployment then opentofu management
with a successful tested deployment method from dc0. Something to pin for the end of
deployment, review and add opentofu management to the remaining tasks."**

**RULED -- the SEQUENCING, which is the substance.** The Juju layer is NOT converted to
OpenTofu now. Order: (1) dc0 deploys and is HARDENED; (2) that deployment method is TESTED and
demonstrated to work; (3) the proven method becomes the INPUT to the OpenTofu translation;
(4) the conversion is reviewed at end-of-deployment and enters the remaining-task set.

**WHY NOT NOW, recorded so the sequencing is not re-litigated:** the bundle deploy has never
completed end to end at VR1. Changing the deploy MECHANISM before the deploy SUCCEEDS means
debugging two unknowns at once, and the failures to date (2026-07-31: `unknown option
prefer-ipv6`, a missing `policies/overrides.zip`, `default-base` deployable exactly once;
2026-08-01/02: model/spaces/`apt-mirror`/snap-proxy/VIP-overlay work) would have been harder to
localise, not easier, with a translation layer in between.

**WHERE THE VALUE ACTUALLY IS, measured rather than assumed.** Almost every failure in this
stage has been in the MODEL and CONFIG layer -- `add-model`, spaces, `apt-mirror`, snap proxy,
`prefer-ipv6`, the VIP overlay -- and NOT in the application topology. Preflight gate **P8**
(2026-08-01) demonstrated the value of declarative drift detection by finding substrate drift
that had been sitting unseen. So the strongest candidate is the model/spaces/model-config
layer, with `bundle.yaml` retained for topology, rather than a wholesale replacement.

**THE COST THAT MUST BE PRICED AT REVIEW TIME, not hand-waved.** The Juju provider is
RESOURCE-shaped, not bundle-shaped: it declares applications and integrations, not a
`bundle.yaml`. The measured 2026-07-31 plan is **56 applications / 108 relations**, so a
wholesale translation is a substantial deliverable in its own right. `docs/dc-dc-buildout-
design.md:12` also records that this repo was seeded from `openstack-caracal-ipv4` at commit
`8813efc` SPECIFICALLY so `bundle.yaml` and `runbooks/phase-01..08` could be reused; abandoning
bundles diverges from that lineage and from the Charmed OpenStack reference material.

**OWED BEFORE THE REVIEW CAN RULE:** provider capabilities must be READ, not recalled --
`references/opentofu-provider-docs.md` gives the sources and the fetch method, and this repo
treats an unverified option name or version as the cardinal sin. Options to present: (a)
model/config layer only, bundle retained; (b) full translation; (c) decline and keep the
scripted path with gates. No option is pre-selected here.

**A1 TEST (GA-R3):** admitted as a D-number because a Roosevelt build session would grep this
before choosing how to manage its Juju layer, and because it decides what the deliverable
runbooks look like.

### ORDERING RULING 2026-08-02 -- D-139 step 6 executes IN FULL before the Stage-5 deploy

**Not a change to ruling A or B, and not a new D-number** (GA-R3: execution ordering is OPS).
It is recorded here because it CLOSES the exchange that the `ORDERING RULING 2026-08-01`
above explicitly left owed: *"at least step 6 has a deploy coupling that this ruling does not
settle ... That needs its own GA-R5 exchange before Step 4, and is NOT inferred from this
ruling."* This is that exchange.

**THE PREMISE THAT THE 08-01 RULING STATED IS NO LONGER TRUE, and the question was put on
re-measurement rather than on that text.** The 08-01 note reasoned that "the per-DC VIP
overlays carry v6 VIPs in the ULA range, so deploying after steps 1-3 alone would place a
cloud whose NODES are GUA while its declared v6 VIPs are still ULA". Measured 2026-08-02:
`overlays/vr1-dc0-vips.yaml` carries **ZERO** `fd50:` legs and 39 GUA legs
(`2602:f3e2:f02:11` x13, `f02:20` x13, `f02:21` x13). The overlay was re-rendered onto GUA on
2026-08-02 and the dc0 rack's staged copy was brought into line the same day. **The deploy
input is therefore already correct; the residual coupling is in the APEX, not the overlay.**

**MEASURED STATE AT THE TIME OF THE RULING** (`netbox/d139-gua-carve.py --dc vr1-dc0`, dry
run, against the WORKING VR1 apex `office1-netbox` http://10.10.1.10:8000 per DOCFIX-195):

    CREATE 0 | EXISTS 16 | RETIRE-REPORT 9
    dependent objects inside the retiring prefixes: 26 ip-address(es), 0 ip-range(s)

Step 1 is fully applied and idempotent. The 26 dependents were ENUMERATED rather than
inferred from the count: all 26 are VIP records, 13 in `fd50:840e:74e2:220::/64`
(metal-admin) and 13 in `fd50:840e:74e2:221::/64` (metal-internal), octets `::50`-`::62`,
described e.g. `VIP keystone metal-admin v6 (vr1-dc0) -- octet mirror ::50`. The other three
retiring `/64`s (data-tenant, storage, replication) hold **zero** addresses.

**THE QUESTION AS PUT.** Nothing at deploy time reads the apex, so the divergence is
record-vs-reality and not a functional blocker -- but the apex would not know the GUA VIP
addresses are in use, so a later allocation from those `/64`s could hand out an address a VIP
already holds. No repo tool re-homes addresses: `d139-gua-carve.py`'s header is explicit --
"Prefixes only: no addresses, no MAAS, and NO deletion ever." Options presented: (a) re-home
the 26 now and retire the prefixes later (split step 6); (b) deploy now with the whole of
step 6 deferred as a NAMED gate rather than a loose obligation; (c) full step 6 first, then
deploy.

**Operator answer, exact utterance: "Full step 6 first, then deploy".**

**CONSEQUENCE.** D-139 step 6 runs IN FULL before Step 4: create the 26 GUA VIP
ip-addresses, delete the 26 ULA VIP ip-addresses, and retire the 9 ULA prefix rows -- after
which the deploy lands on a fully consistent apex. **The Stage-5 deploy is BLOCKED on this**
and on nothing else in the D-139 list; steps 4, 5 and 7 remain sequenced after the deploy
(step 7 necessarily so -- its `network-get` prerequisite needs a deployed unit).

**WHAT THIS COSTS, STATED PLAINLY BECAUSE IT IS THE ARGUMENT AGAINST THE OPTION CHOSEN.** It
requires a repo tool that this repo does not have and has deliberately never had: one with a
DELETE path against the apex. `d139-gua-carve.py` refuses to delete BY DESIGN, and that
refusal is why the step-1 push was safe to run. The new tool is therefore the largest new
surface introduced immediately before a deploy, and it must be built to the standard the
step-1 tool was: independently reviewed, dry-run first, its assertions proven able to FAIL,
and CREATE strictly before DELETE so that no window exists in which an address is recorded
nowhere.

**ORDERING WITHIN STEP 6, which the ruling does not itself fix and which is taken as the
minimal safe reading:** create all 26 GUA records FIRST, verify, then delete the 26 ULA
records, then retire the 9 prefixes. Reversed, there would be an interval in which the apex
records no address for a live VIP -- the same orphaning hazard that DEFECT 3 of the
2026-08-01 CORRECTION NOTE identified for the prefixes.

### AMENDMENT 2026-08-02 -- D-139 step 6: "retire" means DEPRECATE, and nothing is deleted

**Operator ruling, taken in a SEPARATE exchange from the ordering ruling above** (GA-R5: one
decision per exchange -- the ordering ruling settled WHEN step 6 runs, this settles WHAT it
does). Not a new D-number: it defines a term inside an existing execution list (GA-R3, OPS).

**WHY IT HAD TO BE ASKED RATHER THAN INFERRED.** D-139's execution list says "retire the ULA
rows in the apex and in MAAS", and this repo has NEVER defined what retire means for an IPAM
OBJECT. Measured: every `netbox/*.py` importer uses only `active`, `container` or `reserved`
-- `deprecated` appears nowhere -- and `d139-gua-carve.py:4` states "no addresses, no MAAS,
and NO deletion ever". So the word could equally have meant DELETE the rows or MARK them
unusable, and the two differ irreversibly in one direction. Verified against the LIVE API
before the options were put, rather than assumed from NetBox documentation:

    OPTIONS ipam/prefixes     status -> ['container', 'active', 'reserved', 'deprecated']
    OPTIONS ipam/ip-addresses status -> ['active', 'reserved', 'deprecated', 'dhcp', 'slaac']

**Operator answer, exact utterance: "Deprecate both, delete nothing".**

**CONSEQUENCE, and it materially shrinks the risk the ordering ruling flagged.** Step 6 is:

1. CREATE the 26 GUA VIP ip-addresses (`2602:f3e2:f02:20::50-::62` metal-admin,
   `2602:f3e2:f02:21::50-::62` metal-internal), **`status=reserved`**, descriptions mirroring
   the ULA originals. **CORRECTED 2026-08-02 BY MEASUREMENT (GA-R1 C2): this line first read
   `status=active`, which was an INFERENCE.** The live ULA VIP records carry
   `status="reserved"`, and `netbox/dc-plane-apex-import.py:186,200` also creates addresses
   `reserved`. The GUA records therefore mirror the originals rather than introducing a
   status the repo does not use for addresses.
2. Set `status=deprecated` on the 26 ULA VIP ip-addresses.
3. Set `status=deprecated` on the 9 ULA prefix rows.

**THERE IS NO DELETE PATH.** The ordering ruling recorded that step 6 "needs a repo tool with
a DELETE path against the apex, which this repo has deliberately never had" and called that
the largest new surface before a deploy. **That cost is now withdrawn:** the tool is
CREATE-plus-STATUS-UPDATE only, `d139-gua-carve.py`'s never-delete posture is preserved
repo-wide, and every step is reversible by flipping a status back. The ULA-to-GUA migration
also stays VISIBLE in the apex rather than vanishing from it.

**ORDERING WITHIN STEP 6 IS UNCHANGED and still matters:** CREATE all 26 GUA records FIRST
and verify, THEN deprecate. Reversed, there would be an interval in which the apex marks a
live VIP's only record unusable.

**NOTE FOR A LATER READER: `deprecated` is ADVISORY in NetBox, not enforcing.** It does not
prevent an allocation; it signals one should not be made. The functional protection against
handing out an in-use GUA address comes from step 1 above -- the 26 GUA records EXISTING --
not from the deprecation.

### AMENDMENT 2026-08-09 -- geneve-over-v6 CONFIRMED viable; containerized OVN chassis need CARVED v6 + an UNBRACKETED ovn-encap-ip

**GA-R5.** Question put to the operator: "How should the geneve-over-v6 finding
(containerized-chassis v6 requirement + the ovn-chassis bracket bug + v6-only data-tenant
confirmed viable) be recorded in design-decisions.md -- as a D-139 amendment or a new D-number?"
**Operator answer: "D-139 amendment".** Recorded 2026-08-09. Amends D-139 ruling A/B; mints no
new number (the finding extends D-139's own family matrix + v6-only execution, and this project's
mint-a-number trap argues against a fresh D for an extension).

**WHAT WAS MEASURED (live, vr1-dc0, D-138; full record + tested sequence:
`docs/audit/geneve-over-v6-rootcause-20260808.md`).** D-139 ruling A intends a v6-only data-tenant
geneve underlay but never proved geneve-over-v6 FORWARDS on this stack. Proved 2026-08-09: a real
VM-to-VM cross-compute ping over a v6 geneve tunnel = 8/8, 0% loss on OVS 3.3.0 / OVN 24.03.2 /
kernel 5.15.0-186. **geneve-over-IPv6 WORKS here; v4-forced is OFF the table.** (A first same-day
test wrongly read "v6 broken" from an OVN `localport` source that never tunnels by design --
retracted; instrument-currency #23 sibling.)

**TWO conditions, both necessary, neither guaranteed by "make the plane v6-only":**
1. **The containerized OVN chassis (octavia / ovn-chassis-octavia LXD) must take a CARVED v6
   data-tenant address.** They auto-pick v4 (D-134), producing the 2026-08-08 family split
   (control v4 encap, metal v6 encap -> no cross-family tunnel). Carve/assign, do not auto-pick.
2. **`ovn-encap-ip` must reach OVS UNBRACKETED.** ovn-chassis 24.03 sets it bracketed
   (`"[2602:...]"`); OVS geneve rejects that (`bad geneve 'remote_ip'`, tunnel `ofport -1`), so
   every v6 tunnel is dead while a family-only check still reads PASS. Fix: a fixed ovn-chassis
   revision, or a persistent post-deploy override
   (`ovs-vsctl set open_vswitch . external_ids:ovn-encap-ip=<unbracketed v6>`). Kernel/OVS support
   v6 geneve fine once unbracketed (measured: unbracketed port instantiates, bracketed does not).

Plus **neutron `overlay_ip_version=6`** for correct tenant MTU (~1422; v6 geneve is 20 bytes
heavier), else a later large-packet failure.

**GATE (built 2026-08-09; replaces the ruled-not-built "geneve-over-v6 verified" item).**
`scripts/geneve-encap-assert.sh` (harness `tests/geneve-encap-assert/`, 16/16) asserts encap family
consistency (`--expect-family v6`) AND every geneve tunnel `ofport>=0` -- the tunnel-health half
catches the bracket bug a family-only check misses. Wired into
`runbooks/dc-dc-phase4-juju-bundle-per-dc.md` Step 12.2.

**ROOSEVELT-DELTA.** A v6 build with LXD-hosted ovn-chassis (the norm on MAAS metal) must carve v6
for those containers AND ensure an unbracketed encap-ip, and must PROVE geneve-over-v6 forwards
before committing a plane's overlay to v6 -- the gate above is that proof, run at Stage-5 close
before any workload smoke. Fixed-ovn-chassis-revision confirmation (charm-ovn-chassis IPv6 bracket
bug family) remains OWED for the rebuild.

## D-141: IPAM allocations are DUAL-STACK, status-distinguished -- v4 active, v6 reserved-until-capable [ARCH]

**Status: ADOPTED 2026-08-03 (operator ruling, GA-R5).** Question as presented: with the
container-hosted API charms reverting to IPv4 VIPs (juju LP #1723240 gives containers no v6
address; hacluster/mysql/ceph-osd/OVN carry their own v6 defects -- see
`docs/charm-ip-family-compatibility.md`), should the v6 VIP allocation be KEPT in the NetBox
apex for future v4->v6 conversion when compatibility catches up? **Operator answer, exact
utterance: "Yes, but you should expand on the record. It should be noted that when creating
future ipam allocations the IPv4 and IPv6 blocks must follow this same structure."**

**RULED -- the immediate decision:** the GUA v6 VIP allocation STAYS in the apex. No NetBox
mutation is needed: D-139 step 6 already CREATED the 26 GUA VIP addresses and they carry
status **`reserved`** (measured), which is the truth-bearing state -- allocated and held
against collision, but NOT asserted as live. They are NOT promoted to `active` until the v6
VIPs are actually deployed and verified.

**RULED -- the STANDING RULE, which is the architectural half and the reason this is a
D-number (GA-R3 A1 test: a future/Roosevelt build session greps this before making any IPAM
allocation):** **every IPAM allocation is authored DUAL-STACK, and the STATUS field carries
the truth about which family is live.** The structure is:

| Family | NetBox status | Meaning |
|---|---|---|
| IPv4 | `active` | live, deployed, consumed by the running cloud |
| IPv6 (GUA) | `reserved` | carved and collision-protected; planned for conversion when the stack is capable |
| superseded (e.g. ULA) | `deprecated` | history; advisory-only, do not allocate |

**This makes the apex record BOTH current reality and future plan in one place, legible from
the status field alone.** It is the apex driving the deploy (intent recorded first, deploy
consumes when able), NOT the apex back-filled to match a deploy -- the authority direction
G18/lens-7 requires.

**RULES that follow, binding on all future allocations:**
1. **Author v4 and v6 together.** A new plane, VIP band, pool or reservation is carved in
   BOTH families at allocation time, not v4-now-v6-later. The v6 block mirrors the v4 block's
   structure (same host numbering within each family's prefix).
2. **Status bears the truth, never prose.** `active` means live. `reserved` means planned,
   not deployed. Never mark a family `active` before it is deployed and verified -- that is
   the "apex says built when it isn't" defect this repo has hit (RULED-not-BUILT class).
3. **Promotion is gated on capability, and the gate is named.** A `reserved` v6 block moves to
   `active` only when the consuming layer can use it. For the API-charm VIPs that gate is the
   two-layer set in `docs/charm-ip-family-compatibility.md` (juju addressing LP #1723240 +
   the per-charm fixes). A reserved block SHOULD carry a note pointing at what unblocks it.
4. **Never delete a reserved future-family allocation to "clean up".** It is the conversion
   input; deleting it forces a re-carve and frees the address to collide.

**Relationships:** consumes the D-111 per-DC v6 subcarve and the D-139 GUA family as the v6
side; the reserved-not-deleted discipline is the D-139 step-6 pattern generalised.
**Adjacent to gate G18** (IPAM apex completeness, deferred-until-live): D-141 governs the
STRUCTURE of apex allocations; G18 remains the separate ruling on the charm-created lb-mgmt
prefix and must not be treated as answered by this. Roosevelt analog: every bare-metal DC's
IPAM is authored dual-stack from day one, v6 reserved until the platform is v6-capable, so no
DC ever needs a v4->v6 re-carve -- only a status promotion.

---

## D-142: PROPOSED -- Vault-init workflow QoL sweep (interaction-shape, VR1-fit, save-gate)

**Status:** PROPOSED / OPEN -- APPROVED IN PRINCIPLE, IMPLEMENTATION DEFERRED. Filed
2026-08-05. Operator approved all of R1-R5 (exact utterance: *"Yes, I approve of all the
recommendations so far"*), then directed that the 2026-08-05 dc0 vault init run on the CURRENT (unchanged)
runbook and the QoL work be implemented + TESTED next opportunity (exact utterance: *"We
will complete a vault init using current commands, take the qol and updated script/steps
developed and we can test them on the next opportunity"*). **R2's off-host transport is
UNRESOLVED** (open question to operator). So: approved-in-principle, NOT yet built, one
sub-item still needs an operator input before it can be built.

**Scope.** A QoL / interaction-shape sweep of `runbooks/phase-02-vault-bringup.md` (the
vault init/unseal/authorize secret-handling runbook). The commands are correct; the target
is the *friction* -- three interactive `juju ssh` opens, a cross-terminal secret hop, VR0
`-m openstack` / "jumphost" substitution on every line (D-138), and the one unguarded
catastrophic step (off-host save of `init.txt`).

**Distinct from existing vault decisions (checked before numbering, per CLAUDE.md):**
- **D-068** = vault *substrate* hardening (version / listener-TLS / AppRole lifecycle) --
  Roosevelt-durability. Not the init workflow.
- **D-011.6** = manual-unseal acceptance *bar* -- a PRESERVE-item this sweep explicitly keeps.
Neither governs the init *workflow/interaction shape*; hence a new number, not an amendment.

**The five recommendations (full engineering + safety analysis):**
`docs/audit/vault-init-qol-proposal-20260805.md`. In brief: **R1** parameterize
`-m "$DC_MODEL"` + run-location = DC rack (a DEFECT fix -- `-m openstack` targets a
non-existent VR1 model; also filed as fold-register **F13**); **R2** enforce the off-host
save as a GATE (blocked on the transport question); **R3** a guard-safe
`scripts/vault-init-assist.sh` scaffold that does everything AROUND the one-shots and never
runs them; **R4** cut 3 interactive sessions to the minimum (init stays isolated); **R5**
small conveniences (status wrapper + at-a-glance card).

**Binding safety constraint (operator, verbatim):** *"when the keys are printed ... that at
no point is the data caught up with the hidden prompt options that could cause the keys to
generate and be lost ... changes around this part of the process have to be doublechecked."*
The proposal doc carries the full paste-buffer/hidden-prompt analysis, the advisor-caught
residual `tee`-write-failure risk + the pre-init writability probe, and the mandatory R3
harness constraints (mutation-prove the emitted init unit carries `2>&1 | tee` + `mkdir -p`
and that no emitted block puts a hidden-prompt line ahead of another).

**PRESERVE (do not let the QoL pass erode):** discrete un-batched gating; hidden prompts /
`read -s`; `2>&1 | tee` never `>`; short-lived child token; manual unseal (D-011.6).

**Relationships:** target runbook is the phase-02 vault bring-up; **D-138** (client on the
DC rack) is the source of R1's substitution; **DOCFIX-006/029** (init capture + paste-alone)
and **D-011.6** are the invariants preserved. Roosevelt analog: the per-DC Vault standup
gets a tested, low-friction, guard-safe init procedure rather than a head-substituted VR0
runbook.

---

## D-143: ADOPTED -- VR1 re-IP `10.12.0.0/16` -> `10.13.0.0/16` (AMENDS D-115 premise; TERMINATES D-101 v4-inherit clause) [ARCH]

**Status: RULED 2026-08-09 (GA-R5).** The VR1 build's `10.12.0.0/16` collides with the
still-online live IPv4 cloud (`vr0-dc0`, `vopenstack`), surfaced 2026-08-08 at the Headscale
route step. dc0 finishes as a checkpoint, then the DC substrate is torn down and REDEPLOYED
on `10.13.0.0/16` -- never editing live infra. Ruling-prep package (blast radius, decision
reconciliation, owed checks): `docs/audit/reip-1013-ga-r5-ruling-prep-20260808.md`. Next-free
verified at ruling time (`ledger-scan`: highest-header D-142, D-143 free).

**Ruled in TWO operator exchanges (GA-R5; option selections are the exact utterances):**

- **Exchange 1 (direction + check-#3 acceptance + octet-preserving mapping).** Question as
  presented: "The re-IP ruling (VR1 10.12.0.0/16 -> 10.13.0.0/16, would be D-143): 2 of 3
  owed live-free checks now PASS with fresh measurement; check #3 (VR0 internals) is a
  separate tailnet-only trust domain unreachable from the jumphost. How do you want to
  proceed?" -- the question text specified "adopt 10.13.0.0/16 ... octet-preserving
  10.12.a.b -> 10.13.a.b". **Operator selection, exact utterance: "Accept evidence + adopt
  10.13".** CONSEQUENCE: the tailnet + apex measurements are ACCEPTED as sufficient for owed
  check #3 (the VR0-internal MAAS/neutron sweep is NOT required before the re-carve); the
  re-IP is ADOPTED; the mapping is the octet-preserving `10.12.a.b -> 10.13.a.b` shift (C.2).
  **C.2 alternative SEEN-AND-REJECTED (confirmed 2026-08-09, advisor-flagged):** the
  ruling-prep's alternative -- regularize dc0's non-contiguous plane offsets (4/8/12/16 then
  32/36) into a contiguous /19 block symmetric with dc1 -- was considered and NOT taken. It
  breaks octet-preservation and diverges further from the VR0/Roosevelt template; the
  octet-preserving shift is confirmed (operator: "Octet-preserving confirmed"), and D-134's
  standing cross-DC octet map endorses it.

- **Exchange 2 (the two structural sub-forks; C.1 governs C.3).**
  - **C.1 apex fork.** Question: "the live cloud stays on 10.12 untouched. Where does the
    newly-allocated 10.13.0.0/16 land in the NetBox IPAM role model?" **Operator selection,
    exact utterance: "B2: new 'Cloud -- VR1 rebuild' role".** CONSEQUENCE: 10.13.0.0/16 gets
    its OWN VR1-scoped NetBox role; the `Cloud` role STAYS 10.12-only (the VR0/live record is
    NOT re-labelled -- B3 was rejected for exactly that reason). No live object is edited.
  - **C.3 lib-net.sh structure.** Question: "scripts/lib-net.sh's flat defaults ARE
    vr0-dc0's live as-built values (line 37), and the vr0-dc0 arm is a no-op over them. After
    the re-IP, which shape?" **Operator selection, exact utterance: "(i) Keep flat defaults
    at 10.12; VR1 arms get full 10.13 blocks".** CONSEQUENCE: `lib-net.sh`'s flat defaults
    STAY at 10.12 (= the live cloud's measured values, preserving the line-37 invariant and
    the honest vr0-dc0 no-op); the `vr1-dc0` and `vr1-dc1` arms each gain a COMPLETE explicit
    10.13 literal block.

**Decision reconciliation (verbs, from the ruling-prep Section A -- each governing block read
in full):**
- **D-101 -- TERMINATES one clause.** D-101's "VR1 DC0 INHERITS the DC0 six-plane v4 layout
  ... unchanged" (`:2304`) is TERMINATED: `vr1-dc0` no longer inherits `vr0-dc0`'s v4 layout,
  it takes a distinct non-overlapping one. The IPv6 family matrix, the "IPv6 wherever
  possible" principle (`:2282`) and the dual-stack ruling are UNTOUCHED -- this is a v4-only
  re-IP. The inheritance rationale D-101 gave (minimize delta to Roosevelt / reuse the
  validated bundle) is the cost the re-IP knowingly trades for collision-freedom.
- **D-115 -- AMENDS a factual premise (its ruling holds).** D-115's v4-role-based ruling
  (v4 role-based; /22 per office; DC2 inside the Cloud /16) is NOT reversed. What is
  superseded is its FINDING that "10.13.x is unallocated / outside every allocated block"
  (`:3766-3767`, `:3856-3858`) -- post-pivot, 10.13.0.0/16 is DELIBERATELY allocated under
  the existing `10.0.0.0/8` Corp Private parent. Verb: AMENDS the factual premise;
  CONSISTENT-WITH the role-based ruling.
- **D-124 -- CONSISTENT-WITH (routes re-point).** The `172.31.0.0/24` transit supernet is NOT
  in 10.12 and does not move. The rack metal-admin statics shift by the 12->13 rule
  (`10.12.8.2 -> 10.13.8.2`, `10.12.68.2 -> 10.13.68.2`) and the transit-`/30` routes to DC
  destinations re-point 10.12.* -> 10.13.*.
- **D-134 -- CONSISTENT-WITH (survives; endorses the map).** The per-plane octet bands and the
  standing cross-DC octet map (`.4` artifact / `.5` juju / `.6` MAAS region / `.7` tailscale)
  are offset-relative to the plane /22, not tied to the second octet, so `10.12.8.5 ->
  10.13.8.5` preserves "juju = .5" exactly. D-134's standing-cross-DC-standard clause
  (`:5965-5968`) actively ARGUES FOR the 1:1 octet-preserving shift.

**LIVE evidence at ruling time (measured 2026-08-09, operator-authorized read-only
discovery; captures + method in `docs/audit/open-items-review-20260809.md` Appendix and its
sweep `docs/audit/queued-findings-20260809-open-items-review.txt` FS1):**
- Check #2 (live NetBox apex, `office1-record-dump.py`): **10.13.0.0/16 = 0 objects; 10.12 =
  149 (positive control).** Nests cleanly under the existing `10.0.0.0/8` aggregate.
- Check #1 (Headscale/tailnet, `tailscale status --json` from `office1-tailscale`): **no
  approved subnet route overlaps 10.13.0.0/16**; the positive control reproduced the exact
  10.12 collision (`vopenstack-jesse-tailscale` advertising `10.12.4.0/22` + `10.12.8.0/22`).
  10.13 is also clear of Roosevelt (`10.17.x`), willamette (`10.1`/`10.16`) and Office1
  (`10.10`) on the shared tailnet.
- Check #3 (live VR0 internals): NOT run -- VR0 is a tailnet-only separate trust domain
  unreachable from the authorized vantage; the tailnet+apex evidence was ACCEPTED as
  sufficient (Exchange 1). Blind spot recorded: the tailnet view is approved-routes
  (AllowedIPs); an advertised-but-pending view would need the Headscale server itself.

**OWED EXECUTION (hard rule 1 -- this ruling DECIDES; it does NOT re-carve here). The re-IP
work is subsequent gated steps, sequenced by the checkpoint plan:**
1. NetBox apex: create the B2 role `Cloud -- VR1 rebuild` owning `10.13.0.0/16`; re-carve the
   VR1 prefixes/VIPs/ranges under it by the octet-preserving map. `Cloud` (10.12) untouched.
2. `scripts/lib-net.sh`: keep flat defaults at 10.12; give `vr1-dc0`/`vr1-dc1` full 10.13
   literal blocks; UPDATE the now-false `:124-134` "inherits VR0 unchanged" comment (F13).
3. `10.13` naming-collision DOCFIX (review R16): reconcile `netbox/README.md:49`
   (`VR1_DC1_V4_SUPERNET=10.13.0.0/19`, pre-D-115) and the `10.13.0.5` test literal at
   `tests/dc-rack-mgmt-import/test_logic.py:265` so the /16 adoption does not put a rack-IP
   fixture inside the new Cloud space.
4. D-124 transit routes re-point + rack statics shift (12->13).
5. Teardown must REVOKE the dc0-substrate credentials before redeploy (review R7).

**A1 TEST (GA-R3):** admitted as a D-number -- a future/Roosevelt build session greps this
before laying out addressing, it supersedes a prior ruling's premise (D-115) and terminates a
clause of another (D-101), and it decides the apex role model + the `lib-net.sh` shape that
every downstream consumer reads. Roosevelt relevance: the octet-preserving-shift discipline
and the "never re-label a live record; give the new build its own role" pattern (B2) transfer
directly to any future re-IP or parallel-build.

## D-144: ADOPTED -- VR1 container-layer ELIMINATION + layered module workflow (SUPERSEDES D-123 Model B; the 10.13 redeploy flattens to Option 1) [ARCH]

**Status: RULED 2026-08-10 (GA-R5).** The dc0/dc1 CONTAINMENT VM layer (`vvr1-dcN`, D-122/D-123
Model B: `vcloud outer libvirt -> vvr1-dcN inner libvirt -> node VMs`) is ELIMINATED as part of the
D-143 `10.13.0.0/16` redeploy. Node VMs become flat siblings directly on vcloud's libvirt
(`vcloud libvirt -> node VMs`, nesting depth 4->2, the VR0-proven shape). This ruling DECIDES the
shape; it does NOT build or execute -- **Model B remains the LIVE shape until the redeploy runs**
(hard rule 1). Planning pass of record: `docs/audit/container-elim-pass/` (multi-agent READ-ONLY,
phases 0-4 + a final advisor; agents recorded as "the administrator"/"the advisor", no model name
asserted). Operator report: `docs/audit/container-elim-pass/operator-report.md`; full change-set +
module design + sequencing: `FINAL-PLAN.md`; advisor review: `FINAL-advisor-review.md`. Next-free
verified at ruling time (`ledger-scan`: highest-header D-143 -> D-144 free; SEC-034 next-free).

**Ruled in THREE operator exchanges (GA-R5; option selections are the exact utterances):**

- **Exchange 1 (DEC-01 -- the decision FORM).** Question as presented: "How should the
  container-layer elimination (flatten Model B -> Option 1 flat topology + per-DC client VM,
  adopted for the 10.13 redeploy) be recorded -- a new D-144 that supersedes D-123 (Model B), or an
  amendment appended to D-123 in place?" **Operator selection, exact utterance: "New D-144,
  supersedes D-123".** CONSEQUENCE: minted as its own entry naming D-123 SUPERSEDED (following the
  D-143 precedent -- which amended D-115 and terminated a D-101 clause yet was minted as its own
  number); the reconciliation ledger below is adopted with it.

- **Exchange 2 (DEC-11 -- root topology).** Question: "Ratify the tofu root shape for the flat
  topology -- (B) shared-outer + per-DC-flat roots (3 roots: unchanged outer + two per-DC roots =
  today's inner roots minus the qemu+ssh indirection), or a single merged root for both DCs?"
  **Operator selection, exact utterance: "(B) shared-outer + per-DC-flat".** CONSEQUENCE: 3 roots;
  per-DC state-blast-radius isolation; destroy scoping re-earns D-122's one-command site-down almost
  literally; the client VM's apply groups with its DC's flat root. [OPS]-grade shape, ratified in the
  delivery change-set. NOTE: (B) is tofu STATE isolation only -- it mitigates NONE of the three
  isolation concerns below (both DCs still share vcloud's one kernel).

- **Exchange 3 (DEC-13 -- client-VM octet + name).** Question: "The new per-DC `vr1-dcN-client` VM
  needs a utility-band octet. D-134's standing map is `.4` artifact / `.5` juju / `.6` MAAS region /
  `.7` tailscale -- all occupied. Adopt `.8` (next free, reserved cross-DC) and the name
  `vr1-dcN-client`?" **Operator selection, exact utterance: ".8 / vr1-dcN-client".** CONSEQUENCE:
  the client VM takes utility-band `.8` (`10.13.8.8` dc0 / `10.13.68.8` dc1 under D-143
  octet-preserving); name avoids the eliminated `vvr1-dcN` class; filed as a dated D-134 AMENDMENT
  (this session) so the standing octet map stays self-contained.

**What is adopted (the shape -- Option 1, operator-confirmed at the Phase-0 gate 2026-08-09, ruled
here):** flat node VMs on vcloud libvirt + one small non-hypervisor `vr1-dcN-client` VM per DC
(~4 vCPU / 8192 MiB / 80 GiB; metal-admin + transit legs) carrying the D-138 cloud-facing client role
and that DC's credential residencies. ELIMINATED: the containment VMs + their inner libvirtd, the two
inner tofu roots AS roots, the bootstrap gate's `--host-nodes` duty (~134 lines), the qemu+ssh
provider dial + its D-126 per-env keys (no successor), `modules/wan-bridge` + the D-125 bridge-in
plumbing. RE-HOMED with zero body changes: the six planes, the 12 node VMs, the DC edge (WAN attaches
directly to the per-DC `site-wan` NAT). One new IaC module `modules/dc-site` composes the per-DC
stack. The MAAS region stays on `vr1-dcN-maas-01` (.6, D-132 addendum -- unchanged). Structural win:
the container layer was the ONE violation of the repo's IaC->procedure boundary (the inner root's
qemu+ssh provider is IaC reaching across a live-dial boundary); flattening removes it structurally.

**The accepted power-key blast-radius tradeoff (named in the body per GA-R3 A1, NOT buried in a SEC
row):** flattening puts both DCs' node fleets + `voffice1` on vcloud's ONE libvirtd (verified:
`voffice1` is a vcloud-libvirtd domain, `opentofu/main.tf:175`; no libvirt access-scoping exists
in-repo). Re-deriving node power to that libvirtd would give each DC's region VM virsh control over
EVERY vcloud domain, so **SEC-012/016's per-DC power-key separation becomes VACUOUS** absent a
mitigation. Adopted knowingly; mitigated by the owed power-key control (DEC-15, below); its gate is a
NEGATIVE test (the key CANNOT reach the other DC / voffice1), never existence-only.

**Decision reconciliation (verbs -- from the pass's Phase-4 ledger, each governing block read in full
during the pass):**
- **D-123 (Model B) -- SUPERSEDED.** Its core node-placement ruling is reversed; history stays
  intact (append-only). Do not build from D-123.
- **D-125 (bridge-in) -- TERMINATES.** OBS-3's precondition (nested containment egress) disappears
  with the nesting; `modules/wan-bridge` retires, no successor.
- **D-122 (site shape) -- AMENDS.** Intent preserved; the one-command site-down LITERAL
  (`virsh destroy vvr1-dcN`) regresses to a root-scoped `tofu destroy` + an emergency virsh-loop
  lever -- a real, honestly-stated capability loss re-earned by owed tooling.
- **D-124 (transit) -- AMENDS.** Scheme-A addressing survives on the client VM (the new transit
  bearer); the Model-B sizing-void clause is re-caused (exact figure owed).
- **D-127 (autostart) -- AMENDS (DEFERRED).** The containment row loses its object; the client-VM
  autostart value is owed (DEC-22) -- the D-127 amendment is filed once that value is ruled.
- **D-128 (operating model) -- AMENDS.** Plane 2's definition shrinks to MAAS/NetBox/inner-tofu; the
  substrate build becomes wholly Plane 1 (no qemu+ssh substrate dial). The two-plane model stands.
- **D-131 (node DNS) -- AMENDS (GATED).** Retire-with-evidence; independently triggered by D-132's
  per-DC regions; GATED on an owed current-day live re-measure; the dc0/dc1 asymmetry (dc1's
  forwarder currently load-bearing, dc0's not) is real and closes in the rebuild.
- **D-134 (octet map) -- AMENDS.** Adds `.8` = `vr1-dcN-client` (this session's dated amendment);
  bands/CIDRs otherwise unchanged.
- **D-126 / D-132-addendum / D-138 -- PRESERVE.** Principles intact; only the concrete realization
  changes (D-126: the qemu+ssh consumer's key retires with no successor; D-138: the client role's
  concrete host moves from `vvr1-dcN` to `vr1-dcN-client`, IN the DC as the principle requires).
- **D-114 / D-133 / D-139 / D-140 / D-143 -- UNTOUCHED.** Named to prevent scope creep -- D-114
  especially: Office1/voffice1's containment is a DIFFERENT, KEPT pattern, not eliminated here.

**OWED (this ruling decides the SHAPE; the following are ruled/built at redeploy design time, several
GATED on owed live measurements -- no inferred values, hard rule 2. Full spec: `FINAL-PLAN.md`
Section 7):**
- **DEC-15 -- the power-key mitigation MECHANISM** (restricted SSH key / per-DC virsh wrapper /
  polkit ACL) + its SEC row (SEC-034 next-free). Must decide REACH (how an in-plane `vr1-dcN-maas-01`
  reaches vcloud's own libvirtd WITHOUT a host address on a plane bridge -- which the (a) control
  forbids) AND authorization together. GATED on vcloud's live polkit/libvirt config. CRITICAL PATH:
  six test/tool edits are frozen until it rules; the interim RED on `dc-selector`/`maas-region-power-key`
  is the wanted fail-loud state.
- **DEC-14 / DEC-16 -- the (a) cross-DC host isolation control** (vcloud-level nftables asserting no
  inter-DC/inter-plane forwarding; a mechanical `--check` gate + SEC row) and the SEC-010 transit-drop
  successor (endpoints = client VM + voffice1; new-row-vs-SEC-010-amendment disposition). Two distinct
  controls; neither exists yet.
- **DEC-08 / DEC-09 -- rack-controller retirement + D-131 retire-with-evidence.** GATED on a
  current-day live re-measure of `primary_rack`/rackd state at BOTH DCs (the pass cites are 2-10 days
  old).
- **DEC-24 -- the dc0<->dc1 mesh-leg consumer + cross-DC Ceph replication path** (raised by the
  advisor review). `virbr5` (the netem target, `opentofu/main.tf:334-356`) is the D-100/D-108
  cross-DC replication carrier; the pass reassigned the Office1-transit legs to the client VM cleanly
  but left this leg's post-flatten attachment + netem routing undefined. Partly pre-existing (dc1 HELD
  -> never built), so the flat design must DEFINE it without violating the (a) control. Pairs with
  DEC-14.
- **13 owed artifacts + 9 owed live measurements** (teardown primitive, `dc-site` module, the three
  isolation controls, R7 revocation checklist, MAAS record-release, FIT-calc extension, geneve/jumbo
  live assert, MAC re-measure, ...). All LOGGED, none built.
- **Two-axis discipline:** every redeploy change is attributable to `[D-143]` (10.12->10.13 value
  substitution) or `[CE]` (this shape change); four dual-cause items carry `[both]`. `lib-net.sh`
  carries ZERO container-elim edits.
- **IPv6-unlock design input (added 2026-08-10, read-only pass5):** the collapse unlocks little NEW
  IPv6 -- of a 19-surface family matrix, only the MAAS power-dial reach path is a clean collapse-caused
  v6 unlock (and it is UNVERIFIED, DEC-15-gated). Actionable inputs folded into the owed DECs: (i)
  DEC-15 should prefer a v6 reach (vcloud sshd, off any plane bridge); (ii) DEC-14/16 should decide the
  UNRULED D-124 transit-overlay family (v6 candidate `f00::/40` reserved in the apex) on the same leg
  they govern, and keep the (a)/SEC-010-successor controls' nftables `inet` (v6-native) shape; (iii)
  DEC-24 must resolve the cross-DC replication carrier AND its still-open D-139 v6-route together (same
  leg). NOT unlocked by the collapse (different layers, do NOT credit D-144): D-139's plane family
  matrix, the geneve-over-v6 fix, and the LXD-container/juju LP#1723240 gap (D-141). Full note:
  `docs/audit/container-elim-pass/ipv6-unlock-design-note-20260810.md`.
- **IPv6 FEASIBILITY re-check (added 2026-08-10, pass6 -- CORRECTS the pass5 bullet above, which relied
  on prior rulings instead of researching feasibility).** Fresh ruling-independent research (upstream
  docs/source/bug-trackers, deployed versions) finds IPv6 substantially MORE feasible than D-101/D-139
  held: metal-admin **IPv6 commissioning is FEASIBLE-WITH-WORK on UEFI nodes** (DHCPv6 no MAAS blocker
  @3.7.2; only LEGACY-BIOS PXE is a firmware blocker, avoidable via UEFI), MAAS rack<->region v6
  **FEASIBLE-NOW** (source prefers AF_INET6) -- so D-101's absolute "PXE is v4-first" is an overstated
  judgement, not a capability limit. provider-public v6 **FEASIBLE-NOW via direct GUA routing** (v6 uses
  no floating IP; "dual-stack because the internet is v4" conflated the provider-network FAMILY -- v6-
  capable now -- with EXTERNAL v6 internet reachability, the only true v4 need). Octavia LB VIP v6
  FEASIBLE-NOW; API-VIP/hacluster FEASIBLE-WITH-WORK (one charm bug LP#2111852); uplink edge v6
  FEASIBLE-WITH-WORK (OPNsense ready; routed not NAT66). CONFIRMED still-hard: juju **LP#1723240**
  (container API-charm VIPs -- live-verified still open in 3.6.27; D-141 HOLDS) + external v6 internet
  (deferred). NEW DEC-24 constraint: Ceph binds one family per daemon -> v6 cross-DC replication needs
  the WHOLE cluster single-family. These CHALLENGE prior framings (D-101 PXE, D-139 provider-public) and
  are surfaced for operator RECONSIDERATION -- NOT ruled here (GA-R5). Two DOCFIX candidates flagged
  (stale OVN "No" row in `charm-ip-family-compatibility.md` vs the repo's own geneve proof; soften
  D-101's "PXE v4-first"). Full note: `docs/audit/container-elim-pass/pass6-ipv6-feasibility-note-20260810.md`.

**A1 TEST (GA-R3):** admitted -- SUPERSEDES D-123's core Model-B ruling; a Roosevelt / pre-Roosevelt
build session greps this before laying out substrate shape; and the layered module workflow (L0-L5,
IaC + procedure modules) is the pre-Roosevelt-bare-metal deliverable, with the L1 client-VM / L3 MAAS
/ L4 juju-bundle procedure modules transferring to bare metal UNCHANGED (L0/L2 virtualization IaC and
the two co-residency-specific controls do not transfer). Precedent for the mint-not-amend form: D-143.
