D-100: Inter-DC network topology and fabric (VR1)

Status: PROPOSED 2026-07-09 (DC-DC planning). Instantiates the D-052 per-DC template across two DCs.

Context: VR1 is a virtual regional two-DC test environment (DC1, DC2, plus an Office1 headend), all nested libvirt VMs on one vcloud host, rehearsing the future bare-metal multi-DC site (Roosevelt). D-052 already defines the six-plane per-DC template and tags metal-admin DC-local / metal-internal "may-cross-fiber" -- but that label assumed a single stretched cloud.

Decision:

• Fabric: a full dark-fiber triangle -- DC1<->DC2, DC1<->Office1, DC2<->Office1 (mesh, not star) -- each modeled as a dedicated virtual L2 segment on vcloud. Each site additionally has its own OPNsense edge with a simulated ISP uplink ("external"), so every site has independent edge access.
• WAN simulation: apply tc netem (latency / jitter / loss / rate) on the inter-DC and site-uplink segments, tuned to same-metro dark fiber (low single-digit ms) unless the Roosevelt target says otherwise. The exact parameters are a Phase-0 knob recorded in the buildout doc.
• Plane span (independent-cloud refinement of D-052): because DC1 and DC2 are INDEPENDENT clouds (separate Keystones, no control-plane federation), no control-plane plane crosses the fiber. provider-public, metal-admin, metal-internal, data-tenant, and storage instantiate DC-LOCAL per DC. ONLY the replication plane crosses DC1<->DC2. This refines D-052's "metal-internal may-cross-fiber" (a stretched-cloud notion) to DC-local for the independent-cloud topology.
• Provider-NIC discipline: each DC's provider NIC stays RAW so ovn-chassis builds br-ex and enslaves it by MAC; MAAS must not bridge it. Public API VIPs live on provider-public (D-060 Pattern A). The D-057 provider-vip plane is NOT used (reverted by D-060).

Rationale / tradeoffs: mesh (not star) is required because replication is a direct DC1<->DC2 data path -- routing it through Office1 would be a bandwidth SPOF -- while management is Office1<->DC. Per-site ISP edges make each DC independently reachable and feed the per-DC airgap mirror upstream (D-107). Delta to Roosevelt: the dark-fiber-triangle-plus-per-site-edge shape IS the intended Roosevelt shape; the only virtual-only element is that the "fiber" is a vcloud virtual segment with netem standing in for real fiber and optics.

References: instantiates D-052 (six-plane per-DC template; independent-cloud cross-fiber refinement); carries the D-057 root-cause br-ex discipline as revised by D-060 (Pattern A); pairs with D-101 (families), D-107 (edge egress), D-108 (replication).

Open sub-items: exact netem parameters (Phase-0 measurement + Roosevelt target); whether Office1<->DC fiber carries anything beyond MAAS / Juju / operator management (lean: management only).

D-101: DC-DC address plan and IPv6 family matrix (VR1)

Status: PROPOSED 2026-07-09. Activates the developed family-follows-reachability IPv6 plan for VR1. NetBox (vcloud-local) is the IPAM authority for all literals.

Context: v1 / DC0 is IPv4-only on 10.12.0.0/16 (six planes in scripts/lib-net.sh). The developed IPv6 plan (in-project, not yet committed) classifies planes by reachability: GUA for external clients, ULA for internal-only, IPv4 where forced. DC-DC is the phase that activates it.

Decision:

• Per-DC distinct v4: DC1 INHERITS the DC0 six-plane v4 layout (provider-public 10.12.4.0/22 ... replication 10.12.36.0/22) unchanged, so DC1 equals the validated template. DC2 gets a DISTINCT, non-overlapping parallel supernet (same six-plane structure; exact prefixes assigned in NetBox, non-colliding with DC1, operator infrastructure, and client VPN ranges per D-074). Distinct ranges keep DR / backup endpoints unambiguous (no NAT hairpin).
• Family matrix (per DC), family follows reachability:
  • IPv6-only ULA: data-tenant (geneve underlay), storage (Ceph public), replication (Ceph cluster, including the cross-DC leg), Octavia lb-mgmt, and P2P links. Internal, no external clients.
  • Dual-stack (IPv4 + ULA v6): the metal-internal API-endpoint VIPs gain a v6 (ULA) address; each DC's Vault issues IPv6-SAN certs (D-109).
  • IPv4 retained: MAAS PXE / provisioning (PXE is v4-first); metal-admin (operator / MAAS / admin API, DC-local); and datastore east-west (MySQL / RabbitMQ stay v4-bound -- the plane is dual-stack-capable but v6 there is not chased: marginal benefit, real cert / complexity cost).
  • GUA (ARIN 2602:f3e2::/32, region-0 /36): provider public API VIPs, native v6 ext_net, tenant pool delegation. Enabled here because the simulated OPNsense edge is ours -- the D-015 "upstream-router IPv6 readiness" gate on external GUA does NOT apply in VR1, so VR1 can rehearse the full internal-ULA plus external-GUA end state.
• Replication carrier correction: the DR seed names the cross-DC carrier as v4 10.12.36.0/22. Per this matrix and the operator ruling that replication is not IPv4-bound, the cross-DC replication carrier is IPv6-only ULA (a /64 from the org ULA /48). This SUPERSEDES the DR-seed carrier line. v1 / DC0 lib-net.sh v4 values are unchanged (DC-DC assigns families on its own layer).
• Literals: the org ULA /48, the per-DC GUA carve out of the /36, and DC2's v4 supernet are NetBox-authoritative, populated via the extended import pipeline (netbox/*-import.py extended for multi-DC and dual-stack). They are NOT hardcoded in this decision.

Rationale / tradeoffs: matches the operator directive (v6 wherever possible, v4 only where forced) and maximizes Roosevelt rehearsal value (proves Ceph-over-v6, geneve-over-v6, native v6 tenant routing). The two forced-v4 points (PXE, datastore east-west) are explicitly justified rather than incidental.

References: activates the developed IPv6 plan (its Phase A/B/C); carries D-016 / D-074 (tenant CIDR policy); supersedes the DR-seed v4 replication carrier; feeds D-102 (tenant/MTU), D-106 (AAAA), D-108 (replication), D-109 (Vault v6-SAN). NetBox is IPAM apex.

Open sub-items: the exact NetBox literals; whether the metal-admin admin-API endpoint also gets a ULA leg (lean: no -- keep metal-admin v4-only, it is DC-local operator/PXE).

D-102: IPv6 tenant addressing and MTU sub-policy (VR1)

Status: PROPOSED 2026-07-09. Kept separate from D-101 (tenant-facing sub-policy vs the plane matrix); merge on operator preference.

Decision (leans; operator to ratify):

• Tenant v6 routing: native GUA routing, no NAT66 -- the simulated OPNsense edge routes tenant GUA prefixes (prefix delegation or static routes). NAT66 is discouraged and we own the edge, so native is both correct and feasible.
• Tenant v6 block size: /60 per tenant default (~4080 per cloud); /56 available on request.
• Tenant address mode: SLAAC default (RA); DHCPv6-stateful available for tenants needing deterministic addressing.
• MTU / geneve-over-v6: compute the exact byte budget in Phase 0 against the MEASURED host L2 MTU. An IPv6 outer header adds 20 bytes over IPv4 (40 vs 20); geneve-over-v6 overhead is about 40 (IPv6) + 8 (UDP) + 8 (Geneve base) = 56 bytes, before nested-virt and options. Recommendation: raise the vcloud underlay to jumbo (9000) end-to-end so tenant MTU stays 1500 and amphora/geneve fit; if the underlay is pinned at 1500, tenant MTU drops to about 1444 (v6-geneve) and MUST be set consistently across ovn geneve, tenant-network MTU, and amphora. The measured underlay MTU is a Phase-0 gate -- do not assume jumbo.
• Phase-A rollback posture: go straight to IPv6-only on the internal three (data / storage / replication) rather than dual-stack-then-flip, given plane isolation and the Octavia ULA proof. Dual-then-flip is the conservative alternative if Phase-0 verification of Ceph-over-v6 / geneve-over-v6 is not clean.

Rationale: matches the developed plan's recommendations; native routing plus SLAAC is the IPv6-correct default; the MTU budget is the classic nested-OpenStack failure mode and is handled by measurement plus jumbo.

References: extends D-101. Ceph-over-v6 (ms_bind_ipv6) and OVN geneve-over-v6 are Phase-0 verification gates in the buildout doc.

D-103: OpenTofu / MAAS / Juju lifecycle seam (VR1)

Status: PROPOSED 2026-07-09. Resolves the double-specified node-VM ownership question.

Decision:

• OpenTofu owns create / destroy of: the libvirt domains that stand in for DC "hardware"; all virtual networks (the dark-fiber, plane, and ISP-edge segments); the OPNsense edge VMs; and the Office1 service VMs (MAAS-region, NetBox, GitBucket). OpenTofu runs from the Office1 operator VM against the vcloud host libvirt.
• OpenTofu registers each DC's libvirt host to that DC's MAAS rack controller as a virsh VM-host, so MAAS DISCOVERS the OpenTofu-created node VMs.
• MAAS owns commission / deploy / power / release of those node VMs; it does NOT compose new ones. Juju then deploys the OpenStack bundle onto the MAAS-deployed machines.
• NetBox remains the IPAM apex; MAAS and overlays consume NetBox values, never the reverse.

Rationale / tradeoffs: this keeps the entire physical substrate in versioned OpenTofu (auditable, reproducible, and exactly the "operator scripts the deployment from Office1" model), while MAAS owns the provisioning lifecycle -- mirroring Roosevelt, where MAAS enlists hardware that already exists rather than creating it. "MAAS owns lifecycle" here means the provisioning lifecycle, not domain creation; that reconciles the two operator inputs.

Delta callout (test-env shim): the OpenTofu "create the node VMs" module has NO Roosevelt analog. In Roosevelt that step is physical racking plus BMC enlistment, and MAAS enlists real hardware. This module is tagged in the buildout doc as a virtual-only shim and must NOT be mistaken for reusable Roosevelt IaC. Everything downstream of enlistment (MAAS commission / deploy, Juju bundle) IS Roosevelt-transferable.

References: ratifies MAAS-as-VM-host in the enlist sense; NetBox apex per the operating skill and environment reference; feeds Phases 2-4 of the buildout doc.

D-104: Juju controller topology and HA posture (VR1)

Status: PROPOSED 2026-07-09. Adds the controller-topology / HA posture the DC-DC phase was gated on. References D-071 (patch cadence); does NOT amend it.

Decision:

• One Juju controller PER DC, deployed in-DC (on a MAAS-deployed machine in that DC), each managing only its own cloud. DR-independent: a controller loss affects only its DC.
• Single-unit (not HA) for VR1, with the confirmed backup posture (juju 3.6 create-backup / download-backup). Controller backups scheduled and a restore drill run in Phase 5.
• HA (3-unit) DEFERRED to Roosevelt, when bare-metal resources allow. Recorded as a known VR1 test-only posture, not a Roosevelt recommendation.
• Office1 hosts NO OpenStack Juju controller (Office1 = MAAS-region + OpenTofu + NetBox + GitBucket only).

Rationale / tradeoffs: per-DC single controller is DR-honest (independence) and resource-frugal on the 256 vCPU / 1 TiB / 10 TiB host. A shared or Office1-hosted controller would couple both DCs' manageability -- a controller outage would take out both -- the opposite of what a DR drill should prove. Single-vs-HA is a deliberate test-only economy; backups are the mitigation and are drilled. Roosevelt scales to 3-unit HA per DC.

References: relates to D-071 (routine update cadence / Juju controller patch policy) without amending it; interacts with D-108 (per-DC control-plane backup / restore in the DR drill).

D-105: CAPI management-cluster placement and COS / Magnum scope (VR1)

Status: PROPOSED 2026-07-09. Carries D-035 to the two-DC topology.

Decision:

• Each DC runs its OWN in-cloud, single-homed capi-mgmt cluster (per D-035), independent per DC. No shared or Office1 management cluster.
• COS (observability) and Magnum / CAPI are in scope in BOTH DCs.
• The Magnum domain-setup action is re-run after every teardown / redeploy per DC (D-046 trap: magnum reports "Unit is ready" regardless).

Rationale: D-035 already placed capi-mgmt in-cloud single-homed; per-DC keeps each cloud self-contained and DR-independent. COS per DC (not central at Office1) keeps observability inside each independent failure domain.

References: carries D-035; carries D-046 (domain-setup re-run); COS / Magnum scope confirmed by operator.

Open sub-item: whether COS is per-DC-only or also aggregates a light roll-up at Office1 (lean: per-DC only for VR1; revisit for Roosevelt).

D-106: Designate reactivation (VR1)

Status: PROPOSED 2026-07-09. SUPERSEDES D-019 (Designate deferral); REACTIVATES D-008 (DNS architecture).

Decision:

• Designate is deployed in-bundle in each DC, backend designate-bind (bind9, the Charmed-OpenStack-native backend, lowest delta).
• Naming per D-008 instantiated with the VR1 labels: <service>.omega.dc1.vr1.cloud.neumatrix.local and ...dc2.vr1.... Per-DC INDEPENDENT zones (independent Keystones), served with A and AAAA records (AAAA for the v6-enabled planes per D-101).
• The D-008 bootstrap order applies per DC: static /etc/hosts bootstrap -> os-public-hostname per API charm -> Vault issues FQDN-SAN certs -> Designate zone plus A/AAAA -> Neutron default_dns_domain and dns_servers -> tenant subnets with --dns-nameserver.

Rationale / consequence: reactivating Designate plus FQDN-SAN certs removes the D-019 documented root cause (metal-only charms pulling the public FQDN endpoint they cannot resolve -- the D-021 amphora-pipeline constraint). The D-008 convention was designed multi-DC from the start (dc0.vr0), so this is reuse, not new design -- minimal delta.

References: supersedes D-019; reactivates D-008; resolves the D-019 / D-021 root cause; AAAA depends on D-101. Office1 headend services use a region-level subdomain outside the per-cloud convention (e.g. maas.office1.vr1...) -- exact scheme is a buildout-doc detail.

Open sub-item: cross-DC name resolution (a DC1 tenant resolving a DC2 name) -- lean NOT needed for VR1 (independent clouds); revisit only if a federated-service requirement emerges.

D-107: Airgap posture, per-DC artifact mirror, and NTP (VR1)

Status: PROPOSED 2026-07-09.

Decision:

• Airgap is enforced at the NODE level: OpenStack nodes (control / compute / ceph) reach NO internet directly. They pull all artifacts (apt, snap, charmhub, container registry, images including amphora) ONLY from an in-DC mirror.
• PER-DC mirror (not a single Office1 mirror): each DC hosts its own mirror so a DC can redeploy independently even if Office1 or the peer DC is down -- a DR requirement the drill exercises.
• Controlled egress lives on each DC's OPNsense edge, narrowly scoped: NTP-only, plus the mirror's upstream sync. Nodes get time (chrony) from their DC edge, which syncs upstream to internet NTP pools. Office1 is NOT in the core-service path (no NTP, no node artifacts served from Office1).

Rationale / tradeoffs: this reconciles "airgap-honest" with the operator's "each DC syncs internet NTP" -- the airgap is at the node boundary, not the site boundary, and the edge holds a minimal controlled egress. Per-DC mirrors cost more disk (a factor in the D-101 / Ceph disk budget) but buy true DC-redeploy-independence, which a single Office1 mirror cannot. The Roosevelt airgap posture is the same shape (per-DC mirror, edge-controlled egress).

References: pairs with D-100 (per-site ISP edge), D-101 (disk budget), and the Phase-0 disk math.

Open sub-item: mirror sync topology (each DC pulls upstream independently vs one DC pulls and peers seed) -- lean independent upstream pull per DC for full independence; measure disk cost in Phase 0.

D-108: Cross-DC replication mechanism (VR1)

Status: PROPOSED 2026-07-09. Engineers the mechanism for the SETTLED Tier-A DR decision (docs/dc-dc-replication-DR-seed.md); does not reopen the posture.

Decision (mechanism):

• Cinder volumes: cinder-backup to a cross-DC-replicated object store via ceph-radosgw MULTISITE (a zonegroup with the two DCs as zones; bucket-level bidirectional sync) as the replicated backup target. A restore reconstitutes the volume with its Cinder metadata in the peer -- the honest cross-independent-cloud path. Multisite is chosen over a single mirrored bucket for bidirectional, per-object async replication with conflict handling.
• Glance images: ceph-rbd-mirror of the Glance pool, snapshot-based, two-way, with a re-register step on failover. Daemon topology: one rbd-mirror daemon per DC, peering over the replication plane; daemon HA is single-unit for VR1 (matches the D-104 controller economy), noted for Roosevelt scale-up.
• Nova ephemeral: NOT replicated (rebuild from image plus volume).
• Carrier: the cross-DC leg (radosgw multisite sync plus rbd-mirror peer traffic) rides the IPv6-only ULA replication plane (per D-101; corrects the DR-seed v4 carrier). Explicit inter-DC route plus a bandwidth budget (MTU-aware; see D-102), distinct from intra-DC OSD cluster traffic.
• Consistency groups for multi-volume apps where a crash-consistent set matters.
• Staging: prove one-way (single mechanism, single direction) plus a clean failover / failback drill FIRST, then enable two-way. RPO ~15 min tunable; RTO is the measured drill output, not an SLA.
• Neutron state (ports / FIPs / security groups) is NOT replicated; it is recreated on failover per the runbook.

Rationale: matches the settled seed; multisite plus rbd-mirror are the delta-minimal Ceph-native primitives. This answers the seed's open questions (multisite vs mirrored bucket -> multisite; daemon HA -> single-unit VR1; peer bootstrap -> on the replication plane).

References: engineers docs/dc-dc-replication-DR-seed.md; carrier per D-101; interacts with D-104 (per-DC control-plane backup / restore in the drill); the failover / failback skeleton is fleshed out in the buildout doc.

D-109: Vault posture for VR1

Status: PROPOSED 2026-07-09.

Decision:

• Per-DC INDEPENDENT Vault roots (each DC's Vault is its own root CA; no regional root-of-trust this phase).
• Vault pinned 1.8/stable (the only viable charm lineage for this stack; the 1.16 operator lineage is ruled out per D-068). The D-068 EOL-substrate change is deliberately HELD OUT of VR1 to keep variables constant.
• Each DC's Vault issues certs with both IPv4 and IPv6 (ULA) SANs for the dual-stacked metal-internal API endpoints (per D-101).

Rationale / tradeoffs: independent roots keep the two clouds cryptographically independent (DR-honest) and match the confirmed operator ruling. Holding 1.8/stable isolates the DC-DC variables from the unresolved substrate modernization; the deliberate non-change is recorded against D-068 so it is on the record, not an oversight.

References: references D-068 (substrate hardening deferred to Roosevelt; getting off EOL 1.8.8 stays open there); v6-SAN per D-101.

D-110: DC-DC numbering and repo-seed policy

Status: PROPOSED 2026-07-09 (meta / process).

Decision:

• The DC-DC decision series starts at D-100 in the repo openstack-caracal-dc-dc, seeded from openstack-caracal-ipv4 at commit 8813efc (verified: new-repo HEAD equals 8813efc; design-decisions.md ceils at D-075).
• D-076 through D-099 are RESERVED as a source-project band and are NEVER assigned in this repo. (D-076 is claimed by the source-project dashboard workstream; 077-099 are runway.)
• A genuinely-shared future source-project decision is referenced by its original id (e.g. "see openstack-caracal-ipv4 D-0NN"), never copied and renumbered into this repo.
• ledger-scan is confirmed 3-digit-safe for the D-series (grep -oE '^## D-[0-9]+' | sort -n), so D-100+ next-free is correct. The deprecated repo-lint next-free path is band-limited above 099 and must not be used for next-free (standing rule).

Rationale: a round D-100 floor gives a clean project base plus about 24 numbers of cross-repo collision runway (the source is a maturing project unlikely to reach 100); reference-not-copy preserves provenance. A distinct prefix (DD-) was considered for permanent cross-repo immunity and declined for VR1 -- it would fork ledger-scan, the fence validator, and every D-NNN cross-reference, against minimize-delta.

References: process entry; governs all D-1NN entries in this repo.

Log-for-sweep (not actioned): ledger-scan DOCFIX / BUNDLEFIX patterns use [0-9]{3} -- safe through 999, latent at the 1000 boundary (the same class as the 100-boundary bug already fixed for those series).