diff --git a/docs/changelog-20260709-dc-dc-design.md b/docs/changelog-20260709-dc-dc-design.md new file mode 100644 index 0000000..bf46878 --- /dev/null +++ b/docs/changelog-20260709-dc-dc-design.md @@ -0,0 +1,45 @@ +# Changelog 2026-07-09 -- DC-DC (VR1) originating design + +Delivery from the main-chat DC-DC planning stream. Grounded on +`openstack-caracal-dc-dc` HEAD 8813efc (seed-verified). All decisions PROPOSED -- +this is a design/plan deliverable, not an implementation. Under blanket approval this +changelog is the review surface: what, why (evidence), and how to revert, per item. + +## Items + +1. ADD `docs/dc-dc-buildout-design.md` -- originating buildout design for the virtual + regional two-DC test env (VR1), Phase 0-6 spine, gates, D-refs, shim register, DR drill + skeleton, host/disk/MTU plan. + Why: the mission deliverable; no prior DC-DC design doc existed. + Revert: delete the file. + +2. ADD `docs/dc-dc-D100-D110.append.md` -- append fragment carrying PROPOSED decisions + D-100..D-110 for concatenation onto `docs/design-decisions.md`. + Why: encodes the settled DC-DC inputs as append-only D-entries; fragment (not direct + edit of design-decisions.md) keeps the overlay a clean documented append. + Revert: delete the fragment; if already appended to design-decisions.md, remove the + D-100..D-110 block (append-only, so a tail trim) and re-run ledger-scan. + +## Grounded corrections captured (not silent) + +- D-052 "metal-internal may-cross-fiber" refined to DC-local for the independent-cloud + topology (only replication crosses). Evidence: D-052 text (stretched-cloud assumption) + + independent-Keystones topology. +- D-057 provider-vip plane NOT carried -- reverted by D-060 (Pattern A). Only the + root-cause br-ex discipline is carried. Evidence: D-057 live status header. +- DR-seed v4 replication carrier (10.12.36.0/22) superseded -> IPv6-only ULA. Evidence: + operator ruling + D-101 family matrix. +- No IPv6 literals hardcoded -- ULA /48, per-DC GUA carve, DC2 v4 supernet are + NetBox-authoritative. Evidence: repo grep found no committed authoritative literals. + +## Landing (operator) + +Per `docs/dc-dc-buildout-design.md` Section 11: append the fragment to design-decisions.md, +run `ledger-scan.sh` (expect D next-free 111), add a main-chat ledger-fence line, +`repo-lint.sh` (ignore >099 next-free noise), commit. + +## Open for redline + +Ceph size=3 vs documented size=2 fallback (Phase-0 disk math); netem link profile; +D-102/D-101 merge; D-109 standalone; metal-admin ULA leg; COS roll-up; mirror sync topology. +See design doc Section 10. diff --git a/docs/dc-dc-D100-D110.append.md b/docs/dc-dc-D100-D110.append.md new file mode 100644 index 0000000..4a8314d --- /dev/null +++ b/docs/dc-dc-D100-D110.append.md @@ -0,0 +1,206 @@ + + +## D-100: Inter-DC network topology and fabric (VR1) + +**Status:** PROPOSED 2026-07-09 (DC-DC planning). Instantiates the D-052 per-DC template across two DCs. + +**Context:** VR1 is a virtual regional two-DC test environment (DC1, DC2, plus an Office1 headend), all nested libvirt VMs on one vcloud host, rehearsing the future bare-metal multi-DC site (Roosevelt). D-052 already defines the six-plane per-DC template and tags metal-admin DC-local / metal-internal "may-cross-fiber" -- but that label assumed a single stretched cloud. + +**Decision:** +- Fabric: a full dark-fiber triangle -- DC1<->DC2, DC1<->Office1, DC2<->Office1 (mesh, not star) -- each modeled as a dedicated virtual L2 segment on vcloud. Each site additionally has its own OPNsense edge with a simulated ISP uplink ("external"), so every site has independent edge access. +- WAN simulation: apply `tc netem` (latency / jitter / loss / rate) on the inter-DC and site-uplink segments, tuned to same-metro dark fiber (low single-digit ms) unless the Roosevelt target says otherwise. The exact parameters are a Phase-0 knob recorded in the buildout doc. +- Plane span (independent-cloud refinement of D-052): because DC1 and DC2 are INDEPENDENT clouds (separate Keystones, no control-plane federation), no control-plane plane crosses the fiber. provider-public, metal-admin, metal-internal, data-tenant, and storage instantiate DC-LOCAL per DC. ONLY the replication plane crosses DC1<->DC2. This refines D-052's "metal-internal may-cross-fiber" (a stretched-cloud notion) to DC-local for the independent-cloud topology. +- Provider-NIC discipline: each DC's provider NIC stays RAW so ovn-chassis builds br-ex and enslaves it by MAC; MAAS must not bridge it. Public API VIPs live on provider-public (D-060 Pattern A). The D-057 provider-vip plane is NOT used (reverted by D-060). + +**Rationale / tradeoffs:** mesh (not star) is required because replication is a direct DC1<->DC2 data path -- routing it through Office1 would be a bandwidth SPOF -- while management is Office1<->DC. Per-site ISP edges make each DC independently reachable and feed the per-DC airgap mirror upstream (D-107). Delta to Roosevelt: the dark-fiber-triangle-plus-per-site-edge shape IS the intended Roosevelt shape; the only virtual-only element is that the "fiber" is a vcloud virtual segment with netem standing in for real fiber and optics. + +**References:** instantiates D-052 (six-plane per-DC template; independent-cloud cross-fiber refinement); carries the D-057 root-cause br-ex discipline as revised by D-060 (Pattern A); pairs with D-101 (families), D-107 (edge egress), D-108 (replication). + +**Open sub-items:** exact netem parameters (Phase-0 measurement + Roosevelt target); whether Office1<->DC fiber carries anything beyond MAAS / Juju / operator management (lean: management only). + +--- + +## D-101: DC-DC address plan and IPv6 family matrix (VR1) + +**Status:** PROPOSED 2026-07-09. Activates the developed family-follows-reachability IPv6 plan for VR1. NetBox (vcloud-local) is the IPAM authority for all literals. + +**Context:** v1 / DC0 is IPv4-only on 10.12.0.0/16 (six planes in `scripts/lib-net.sh`). The developed IPv6 plan (in-project, not yet committed) classifies planes by reachability: GUA for external clients, ULA for internal-only, IPv4 where forced. DC-DC is the phase that activates it. + +**Decision:** +- Per-DC distinct v4: DC1 INHERITS the DC0 six-plane v4 layout (provider-public 10.12.4.0/22 ... replication 10.12.36.0/22) unchanged, so DC1 equals the validated template. DC2 gets a DISTINCT, non-overlapping parallel supernet (same six-plane structure; exact prefixes assigned in NetBox, non-colliding with DC1, operator infrastructure, and client VPN ranges per D-074). Distinct ranges keep DR / backup endpoints unambiguous (no NAT hairpin). +- Family matrix (per DC), family follows reachability: + - IPv6-only ULA: data-tenant (geneve underlay), storage (Ceph public), replication (Ceph cluster, including the cross-DC leg), Octavia lb-mgmt, and P2P links. Internal, no external clients. + - Dual-stack (IPv4 + ULA v6): the metal-internal API-endpoint VIPs gain a v6 (ULA) address; each DC's Vault issues IPv6-SAN certs (D-109). + - IPv4 retained: MAAS PXE / provisioning (PXE is v4-first); metal-admin (operator / MAAS / admin API, DC-local); and datastore east-west (MySQL / RabbitMQ stay v4-bound -- the plane is dual-stack-capable but v6 there is not chased: marginal benefit, real cert / complexity cost). + - GUA (ARIN 2602:f3e2::/32, region-0 /36): provider public API VIPs, native v6 ext_net, tenant pool delegation. Enabled here because the simulated OPNsense edge is ours -- the D-015 "upstream-router IPv6 readiness" gate on external GUA does NOT apply in VR1, so VR1 can rehearse the full internal-ULA plus external-GUA end state. +- Replication carrier correction: the DR seed names the cross-DC carrier as v4 10.12.36.0/22. Per this matrix and the operator ruling that replication is not IPv4-bound, the cross-DC replication carrier is IPv6-only ULA (a /64 from the org ULA /48). This SUPERSEDES the DR-seed carrier line. v1 / DC0 `lib-net.sh` v4 values are unchanged (DC-DC assigns families on its own layer). +- Literals: the org ULA /48, the per-DC GUA carve out of the /36, and DC2's v4 supernet are NetBox-authoritative, populated via the extended import pipeline (`netbox/*-import.py` extended for multi-DC and dual-stack). They are NOT hardcoded in this decision. + +**Rationale / tradeoffs:** matches the operator directive (v6 wherever possible, v4 only where forced) and maximizes Roosevelt rehearsal value (proves Ceph-over-v6, geneve-over-v6, native v6 tenant routing). The two forced-v4 points (PXE, datastore east-west) are explicitly justified rather than incidental. + +**References:** activates the developed IPv6 plan (its Phase A/B/C); carries D-016 / D-074 (tenant CIDR policy); supersedes the DR-seed v4 replication carrier; feeds D-102 (tenant/MTU), D-106 (AAAA), D-108 (replication), D-109 (Vault v6-SAN). NetBox is IPAM apex. + +**Open sub-items:** the exact NetBox literals; whether the metal-admin admin-API endpoint also gets a ULA leg (lean: no -- keep metal-admin v4-only, it is DC-local operator/PXE). + +--- + +## D-102: IPv6 tenant addressing and MTU sub-policy (VR1) + +**Status:** PROPOSED 2026-07-09. Kept separate from D-101 (tenant-facing sub-policy vs the plane matrix); merge on operator preference. + +**Decision (leans; operator to ratify):** +- Tenant v6 routing: native GUA routing, no NAT66 -- the simulated OPNsense edge routes tenant GUA prefixes (prefix delegation or static routes). NAT66 is discouraged and we own the edge, so native is both correct and feasible. +- Tenant v6 block size: /60 per tenant default (~4080 per cloud); /56 available on request. +- Tenant address mode: SLAAC default (RA); DHCPv6-stateful available for tenants needing deterministic addressing. +- MTU / geneve-over-v6: compute the exact byte budget in Phase 0 against the MEASURED host L2 MTU. An IPv6 outer header adds 20 bytes over IPv4 (40 vs 20); geneve-over-v6 overhead is about 40 (IPv6) + 8 (UDP) + 8 (Geneve base) = 56 bytes, before nested-virt and options. Recommendation: raise the vcloud underlay to jumbo (9000) end-to-end so tenant MTU stays 1500 and amphora/geneve fit; if the underlay is pinned at 1500, tenant MTU drops to about 1444 (v6-geneve) and MUST be set consistently across ovn geneve, tenant-network MTU, and amphora. The measured underlay MTU is a Phase-0 gate -- do not assume jumbo. +- Phase-A rollback posture: go straight to IPv6-only on the internal three (data / storage / replication) rather than dual-stack-then-flip, given plane isolation and the Octavia ULA proof. Dual-then-flip is the conservative alternative if Phase-0 verification of Ceph-over-v6 / geneve-over-v6 is not clean. + +**Rationale:** matches the developed plan's recommendations; native routing plus SLAAC is the IPv6-correct default; the MTU budget is the classic nested-OpenStack failure mode and is handled by measurement plus jumbo. + +**References:** extends D-101. Ceph-over-v6 (`ms_bind_ipv6`) and OVN geneve-over-v6 are Phase-0 verification gates in the buildout doc. + +--- + +## D-103: OpenTofu / MAAS / Juju lifecycle seam (VR1) + +**Status:** PROPOSED 2026-07-09. Resolves the double-specified node-VM ownership question. + +**Decision:** +- OpenTofu owns create / destroy of: the libvirt domains that stand in for DC "hardware"; all virtual networks (the dark-fiber, plane, and ISP-edge segments); the OPNsense edge VMs; and the Office1 service VMs (MAAS-region, NetBox, GitBucket). OpenTofu runs from the Office1 operator VM against the vcloud host libvirt. +- OpenTofu registers each DC's libvirt host to that DC's MAAS rack controller as a virsh VM-host, so MAAS DISCOVERS the OpenTofu-created node VMs. +- MAAS owns commission / deploy / power / release of those node VMs; it does NOT compose new ones. Juju then deploys the OpenStack bundle onto the MAAS-deployed machines. +- NetBox remains the IPAM apex; MAAS and overlays consume NetBox values, never the reverse. + +**Rationale / tradeoffs:** this keeps the entire physical substrate in versioned OpenTofu (auditable, reproducible, and exactly the "operator scripts the deployment from Office1" model), while MAAS owns the provisioning lifecycle -- mirroring Roosevelt, where MAAS enlists hardware that already exists rather than creating it. "MAAS owns lifecycle" here means the provisioning lifecycle, not domain creation; that reconciles the two operator inputs. + +**Delta callout (test-env shim):** the OpenTofu "create the node VMs" module has NO Roosevelt analog. In Roosevelt that step is physical racking plus BMC enlistment, and MAAS enlists real hardware. This module is tagged in the buildout doc as a virtual-only shim and must NOT be mistaken for reusable Roosevelt IaC. Everything downstream of enlistment (MAAS commission / deploy, Juju bundle) IS Roosevelt-transferable. + +**References:** ratifies MAAS-as-VM-host in the enlist sense; NetBox apex per the operating skill and environment reference; feeds Phases 2-4 of the buildout doc. + +--- + +## D-104: Juju controller topology and HA posture (VR1) + +**Status:** PROPOSED 2026-07-09. Adds the controller-topology / HA posture the DC-DC phase was gated on. References D-071 (patch cadence); does NOT amend it. + +**Decision:** +- One Juju controller PER DC, deployed in-DC (on a MAAS-deployed machine in that DC), each managing only its own cloud. DR-independent: a controller loss affects only its DC. +- Single-unit (not HA) for VR1, with the confirmed backup posture (juju 3.6 `create-backup` / `download-backup`). Controller backups scheduled and a restore drill run in Phase 5. +- HA (3-unit) DEFERRED to Roosevelt, when bare-metal resources allow. Recorded as a known VR1 test-only posture, not a Roosevelt recommendation. +- Office1 hosts NO OpenStack Juju controller (Office1 = MAAS-region + OpenTofu + NetBox + GitBucket only). + +**Rationale / tradeoffs:** per-DC single controller is DR-honest (independence) and resource-frugal on the 256 vCPU / 1 TiB / 10 TiB host. A shared or Office1-hosted controller would couple both DCs' manageability -- a controller outage would take out both -- the opposite of what a DR drill should prove. Single-vs-HA is a deliberate test-only economy; backups are the mitigation and are drilled. Roosevelt scales to 3-unit HA per DC. + +**References:** relates to D-071 (routine update cadence / Juju controller patch policy) without amending it; interacts with D-108 (per-DC control-plane backup / restore in the DR drill). + +--- + +## D-105: CAPI management-cluster placement and COS / Magnum scope (VR1) + +**Status:** PROPOSED 2026-07-09. Carries D-035 to the two-DC topology. + +**Decision:** +- Each DC runs its OWN in-cloud, single-homed capi-mgmt cluster (per D-035), independent per DC. No shared or Office1 management cluster. +- COS (observability) and Magnum / CAPI are in scope in BOTH DCs. +- The Magnum `domain-setup` action is re-run after every teardown / redeploy per DC (D-046 trap: magnum reports "Unit is ready" regardless). + +**Rationale:** D-035 already placed capi-mgmt in-cloud single-homed; per-DC keeps each cloud self-contained and DR-independent. COS per DC (not central at Office1) keeps observability inside each independent failure domain. + +**References:** carries D-035; carries D-046 (domain-setup re-run); COS / Magnum scope confirmed by operator. + +**Open sub-item:** whether COS is per-DC-only or also aggregates a light roll-up at Office1 (lean: per-DC only for VR1; revisit for Roosevelt). + +--- + +## D-106: Designate reactivation (VR1) + +**Status:** PROPOSED 2026-07-09. SUPERSEDES D-019 (Designate deferral); REACTIVATES D-008 (DNS architecture). + +**Decision:** +- Designate is deployed in-bundle in each DC, backend `designate-bind` (bind9, the Charmed-OpenStack-native backend, lowest delta). +- Naming per D-008 instantiated with the VR1 labels: `.omega.dc1.vr1.cloud.neumatrix.local` and `...dc2.vr1...`. Per-DC INDEPENDENT zones (independent Keystones), served with A and AAAA records (AAAA for the v6-enabled planes per D-101). +- The D-008 bootstrap order applies per DC: static /etc/hosts bootstrap -> `os-public-hostname` per API charm -> Vault issues FQDN-SAN certs -> Designate zone plus A/AAAA -> Neutron `default_dns_domain` and `dns_servers` -> tenant subnets with `--dns-nameserver`. + +**Rationale / consequence:** reactivating Designate plus FQDN-SAN certs removes the D-019 documented root cause (metal-only charms pulling the public FQDN endpoint they cannot resolve -- the D-021 amphora-pipeline constraint). The D-008 convention was designed multi-DC from the start (`dc0.vr0`), so this is reuse, not new design -- minimal delta. + +**References:** supersedes D-019; reactivates D-008; resolves the D-019 / D-021 root cause; AAAA depends on D-101. Office1 headend services use a region-level subdomain outside the per-cloud convention (e.g. `maas.office1.vr1...`) -- exact scheme is a buildout-doc detail. + +**Open sub-item:** cross-DC name resolution (a DC1 tenant resolving a DC2 name) -- lean NOT needed for VR1 (independent clouds); revisit only if a federated-service requirement emerges. + +--- + +## D-107: Airgap posture, per-DC artifact mirror, and NTP (VR1) + +**Status:** PROPOSED 2026-07-09. + +**Decision:** +- Airgap is enforced at the NODE level: OpenStack nodes (control / compute / ceph) reach NO internet directly. They pull all artifacts (apt, snap, charmhub, container registry, images including amphora) ONLY from an in-DC mirror. +- PER-DC mirror (not a single Office1 mirror): each DC hosts its own mirror so a DC can redeploy independently even if Office1 or the peer DC is down -- a DR requirement the drill exercises. +- Controlled egress lives on each DC's OPNsense edge, narrowly scoped: NTP-only, plus the mirror's upstream sync. Nodes get time (chrony) from their DC edge, which syncs upstream to internet NTP pools. Office1 is NOT in the core-service path (no NTP, no node artifacts served from Office1). + +**Rationale / tradeoffs:** this reconciles "airgap-honest" with the operator's "each DC syncs internet NTP" -- the airgap is at the node boundary, not the site boundary, and the edge holds a minimal controlled egress. Per-DC mirrors cost more disk (a factor in the D-101 / Ceph disk budget) but buy true DC-redeploy-independence, which a single Office1 mirror cannot. The Roosevelt airgap posture is the same shape (per-DC mirror, edge-controlled egress). + +**References:** pairs with D-100 (per-site ISP edge), D-101 (disk budget), and the Phase-0 disk math. + +**Open sub-item:** mirror sync topology (each DC pulls upstream independently vs one DC pulls and peers seed) -- lean independent upstream pull per DC for full independence; measure disk cost in Phase 0. + +--- + +## D-108: Cross-DC replication mechanism (VR1) + +**Status:** PROPOSED 2026-07-09. Engineers the mechanism for the SETTLED Tier-A DR decision (`docs/dc-dc-replication-DR-seed.md`); does not reopen the posture. + +**Decision (mechanism):** +- Cinder volumes: `cinder-backup` to a cross-DC-replicated object store via `ceph-radosgw` MULTISITE (a zonegroup with the two DCs as zones; bucket-level bidirectional sync) as the replicated backup target. A restore reconstitutes the volume with its Cinder metadata in the peer -- the honest cross-independent-cloud path. Multisite is chosen over a single mirrored bucket for bidirectional, per-object async replication with conflict handling. +- Glance images: `ceph-rbd-mirror` of the Glance pool, snapshot-based, two-way, with a re-register step on failover. Daemon topology: one rbd-mirror daemon per DC, peering over the replication plane; daemon HA is single-unit for VR1 (matches the D-104 controller economy), noted for Roosevelt scale-up. +- Nova ephemeral: NOT replicated (rebuild from image plus volume). +- Carrier: the cross-DC leg (radosgw multisite sync plus rbd-mirror peer traffic) rides the IPv6-only ULA replication plane (per D-101; corrects the DR-seed v4 carrier). Explicit inter-DC route plus a bandwidth budget (MTU-aware; see D-102), distinct from intra-DC OSD cluster traffic. +- Consistency groups for multi-volume apps where a crash-consistent set matters. +- Staging: prove one-way (single mechanism, single direction) plus a clean failover / failback drill FIRST, then enable two-way. RPO ~15 min tunable; RTO is the measured drill output, not an SLA. +- Neutron state (ports / FIPs / security groups) is NOT replicated; it is recreated on failover per the runbook. + +**Rationale:** matches the settled seed; multisite plus rbd-mirror are the delta-minimal Ceph-native primitives. This answers the seed's open questions (multisite vs mirrored bucket -> multisite; daemon HA -> single-unit VR1; peer bootstrap -> on the replication plane). + +**References:** engineers `docs/dc-dc-replication-DR-seed.md`; carrier per D-101; interacts with D-104 (per-DC control-plane backup / restore in the drill); the failover / failback skeleton is fleshed out in the buildout doc. + +--- + +## D-109: Vault posture for VR1 + +**Status:** PROPOSED 2026-07-09. + +**Decision:** +- Per-DC INDEPENDENT Vault roots (each DC's Vault is its own root CA; no regional root-of-trust this phase). +- Vault pinned 1.8/stable (the only viable charm lineage for this stack; the 1.16 operator lineage is ruled out per D-068). The D-068 EOL-substrate change is deliberately HELD OUT of VR1 to keep variables constant. +- Each DC's Vault issues certs with both IPv4 and IPv6 (ULA) SANs for the dual-stacked metal-internal API endpoints (per D-101). + +**Rationale / tradeoffs:** independent roots keep the two clouds cryptographically independent (DR-honest) and match the confirmed operator ruling. Holding 1.8/stable isolates the DC-DC variables from the unresolved substrate modernization; the deliberate non-change is recorded against D-068 so it is on the record, not an oversight. + +**References:** references D-068 (substrate hardening deferred to Roosevelt; getting off EOL 1.8.8 stays open there); v6-SAN per D-101. + +--- + +## D-110: DC-DC numbering and repo-seed policy + +**Status:** PROPOSED 2026-07-09 (meta / process). + +**Decision:** +- The DC-DC decision series starts at D-100 in the repo `openstack-caracal-dc-dc`, seeded from `openstack-caracal-ipv4` at commit 8813efc (verified: new-repo HEAD equals 8813efc; `design-decisions.md` ceils at D-075). +- D-076 through D-099 are RESERVED as a source-project band and are NEVER assigned in this repo. (D-076 is claimed by the source-project dashboard workstream; 077-099 are runway.) +- A genuinely-shared future source-project decision is referenced by its original id (e.g. "see openstack-caracal-ipv4 D-0NN"), never copied and renumbered into this repo. +- `ledger-scan` is confirmed 3-digit-safe for the D-series (`grep -oE '^## D-[0-9]+' | sort -n`), so D-100+ next-free is correct. The deprecated `repo-lint` next-free path is band-limited above 099 and must not be used for next-free (standing rule). + +**Rationale:** a round D-100 floor gives a clean project base plus about 24 numbers of cross-repo collision runway (the source is a maturing project unlikely to reach 100); reference-not-copy preserves provenance. A distinct prefix (DD-) was considered for permanent cross-repo immunity and declined for VR1 -- it would fork `ledger-scan`, the fence validator, and every D-NNN cross-reference, against minimize-delta. + +**References:** process entry; governs all D-1NN entries in this repo. + +**Log-for-sweep (not actioned):** `ledger-scan` DOCFIX / BUNDLEFIX patterns use `[0-9]{3}` -- safe through 999, latent at the 1000 boundary (the same class as the 100-boundary bug already fixed for those series). + +--- diff --git a/docs/dc-dc-buildout-design.md b/docs/dc-dc-buildout-design.md new file mode 100644 index 0000000..23a1a12 --- /dev/null +++ b/docs/dc-dc-buildout-design.md @@ -0,0 +1,225 @@ +# DC-DC virtual regional two-DC buildout design (VR1) + +**Status:** DESIGN / PLAN. Originating design for the virtual regional two-DC test +environment (VR1: DC1, DC2, Office1 headend). PROPOSED decisions live in +`docs/design-decisions.md` as D-100..D-110. No live mutation is authorized from this +document; it is the plan the phased runbooks execute against. + +**Governing constraint:** minimize delta to Roosevelt (the future bare-metal multi-DC +production site). Every virtual choice maps to a defensible real one; the one deliberate +exception is tagged in the shim register (Section 9). + +**Ground truth / copy point:** seeded from `openstack-caracal-ipv4` at commit 8813efc into +`openstack-caracal-dc-dc` (verified). Six-plane values are `scripts/lib-net.sh` +(CIDR-keyed); fabrics / VLANs are `docs/maas-as-built-reference.md`; NetBox (vcloud-local) +is the IPAM apex for all address literals. + +--- + +## 1. Settled inputs (locked with the operator) + +- Topology: full dark-fiber triangle (DC1<->DC2, DC1<->Office1, DC2<->Office1) plus a + per-site OPNsense simulated ISP edge. Mesh, not star. (D-100) +- Independent clouds: DC1 and DC2 are separate Charmed OpenStack clouds, separate + Keystones. No control-plane federation. Only the replication plane crosses fiber. (D-100) +- IaC seam: OpenTofu creates the libvirt "physical" layer (domains, virtual networks, + OPNsense, Office1 service VMs) from Office1; MAAS enlists / commissions / deploys the + node VMs; Juju deploys the bundle. NetBox is IPAM apex. (D-103) +- Controllers: one single-unit Juju controller per DC, in-DC, with backups; HA deferred to + Roosevelt. Office1 hosts no OpenStack controller. (D-104) +- CAPI mgmt per DC, in-cloud single-homed; COS and Magnum / CAPI in both DCs. (D-105) +- DNS: Designate reactivated in-bundle per DC (backend designate-bind), per-DC zones, + A + AAAA. Supersedes D-019, reactivates D-008. (D-106) +- Airgap at the node level; per-DC artifact mirror; NTP from each DC's edge (edge syncs + internet); Office1 out of the core-service path. (D-107) +- Replication: Tier A, bidirectional -- cinder-backup via radosgw multisite; Glance via + rbd-mirror; Nova ephemeral not replicated; one-way then two-way; carrier is the + IPv6-only ULA replication plane. (D-108, corrects the DR-seed v4 carrier) +- Vault: per-DC independent roots, pinned 1.8/stable; v4 + IPv6-SAN certs. (D-109) +- IPv6: family follows reachability -- v6 wherever possible, v4 only where forced. (D-101) +- Numbering: DC-DC series starts D-100; 076-099 reserved source-band. (D-110) + +## 2. Address families (per DC) -- summary of D-101 + +| Plane | v1/DC0 v4 (DC1 inherits) | VR1 family | Note | +|---|---|---|---| +| provider-public | 10.12.4.0/22 | v4 + GUA | public API VIPs + FIP/ext_net; GUA via owned edge | +| metal-admin | 10.12.8.0/22 | v4 only | operator / MAAS / PXE / admin API; DC-local | +| metal-internal | 10.12.12.0/22 (VID 103) | v4 + ULA | API endpoints dual-stack; MySQL/RabbitMQ stay v4 | +| data-tenant | 10.12.16.0/22 | ULA only | geneve overlay; tenant GUA delegated separately | +| storage | 10.12.32.0/22 | ULA only | Ceph public; ms_bind_ipv6 | +| replication | 10.12.36.0/22 | ULA only | Ceph cluster; cross-DC leg rides this (ULA) | + +DC2 mirrors this structure on a distinct, non-overlapping supernet. All literals +(ULA /48, per-DC GUA carve from the /36, DC2 v4 supernet) are NetBox-assigned -- not +hardcoded. PXE and datastore east-west are the two deliberate v4-retained points. + +## 3. Host and resource plan (Phase-0 gates) + +Single vcloud host: ~256 vCPU / 1 TiB RAM / 10 TiB disk, dedicated to VR1. CPU and RAM +are abundant; disk is the binding constraint (two Ceph clusters + per-DC backup/mirror + +images). Phase-0 gates: + +- Nested KVM: confirm / enable (`nested` module param; Intel vs AMD differs). Verify guest + virtualization is exposed to the DC node VMs. +- Ceph replication factor: target size=3 / min_size=2 (production-honest, lowest delta). + This is CONDITIONAL on the disk budget. The Phase-0 disk math must account for: DC1 Ceph + (size=3) + DC2 Ceph (size=3) + per-DC radosgw backup pools (themselves replicated within + each cluster, and duplicated cross-DC by multisite) + Glance rbd-mirror targets + images + + per-DC mirrors. If size=3 across both clusters plus the backup/mirror footprint exceeds + 10 TiB, fall back to size=2 as an EXPLICIT, documented test-only deviation (its own + decision note), never a silent choice -- Roosevelt sizing must not inherit a hidden 2x. + Thin-provisioning is the lever; the reserve split (Ceph vs backup/mirror/images) is set in + Phase 0 from the measured DC0 per-node OSD footprint. +- MTU: measure the host L2 MTU. Compute the geneve-over-v6 budget (Section D-102). Prefer + jumbo (9000) end-to-end; if pinned at 1500, set the reduced tenant MTU consistently. Do + not assume jumbo. MTU stacking (nested virt + geneve + VLAN) is the classic failure mode. + +## 4. Phase spine + +Each phase has a single goal, a verify-before-mutate gate, and owning D-entries. Phases run +in order; a phase is not entered until the prior gate passes. Per operating discipline, the +per-phase runbooks (later deliverables) carry the command-level steps with harnesses; this +document is the plan they execute. + +### Phase 0 -- vcloud host preparation +Goal: turn the bare vcloud host into a substrate the Office1 headend can deploy from. +Build: nested KVM enabled; libvirt storage pools + thin-provisioning; the dark-fiber and +per-site ISP-edge virtual networks; the six per-DC plane segments (as virtual networks); +the MTU plan set on the underlay. +Gate: nested KVM verified; disk budget computed and the Ceph size decision recorded; +measured MTU recorded; virtual networks present and isolated as designed. +Owns: D-100 (fabric), D-101/D-102 (MTU), Section 3 gates. + +### Phase 1 -- Office1 headend standup +Goal: stand up the operator's deployment headend first, before any DC. +Build: MAAS region controller; OpenTofu; NetBox (vcloud-local, importer extended for +multi-DC + dual-stack); GitBucket (vcloud-local mirror); Tailscale front door (Office1 only). +Gate: MAAS region reachable; NetBox authoritative and populated (planes, per-DC v4, ULA/GUA +carve); GitBucket serving; OpenTofu can reach the vcloud host libvirt; Tailscale access +confirmed to Office1 only. +Owns: D-103 (Office1 service VMs are OpenTofu-created), D-107 (headend is not a core-service +provider). + +### Phase 2 -- OpenTofu builds each DC substrate (from Office1) +Goal: from Office1, script each DC's "physical" layer into existence. +Build: per DC, OpenTofu creates the node-VM libvirt domains (SHIM -- Section 9), the DC's +plane networks, and the DC's OPNsense edge; registers the DC libvirt host to that DC's MAAS +rack controller as a virsh VM-host. +Gate: MAAS rack controller per DC up; MAAS discovers the OpenTofu-created node VMs; edge +routing + simulated ISP uplink up; netem parameters applied and measured. +Owns: D-103 (seam), D-100 (edges). + +### Phase 3 -- MAAS enlist / commission / deploy (per DC) +Goal: bring the discovered node VMs to a deployed OS, ready for Juju. +Build: per DC, MAAS commissions and deploys the node VMs; provider NIC left RAW (br-ex +discipline, D-100); the VLAN-103 metal-internal bridge stack pre-built by MAAS. +Gate: nodes deployed; the six planes present per node with correct fabrics/VLANs; provider +NIC raw; PXE (v4) working; per-DC mirror reachable from nodes; NTP from edge working. +Owns: D-103, D-100 (br-ex), D-107 (mirror + NTP). + +### Phase 4 -- Juju controller + OpenStack bundle (per DC) +Goal: a running independent OpenStack cloud per DC. +Build: per DC, bootstrap the single-unit Juju controller in-DC; deploy the Caracal bundle; +Vault init with per-DC independent root, issuing v4 + IPv6-SAN certs; the family matrix +applied (ULA on data/storage/replication, dual-stack metal-internal API, GUA provider). +Gate: `preflight.sh` PASS before add-model; post-deploy `cloud-assert.sh --capture` per DC; +controller backup taken and stored; Ceph-over-v6 and geneve-over-v6 verified. +Owns: D-104 (controllers), D-109 (Vault), D-101 (families). + +### Phase 5 -- DR wiring and failover drill +Goal: prove Tier-A recoverability, one-way first, then two-way. +Build: replication-plane peering (ULA); radosgw multisite zonegroup (two DCs as zones); +rbd-mirror for the Glance pool; cinder-backup to the replicated object store; consistency +groups where needed. Run the failover / failback drill (Section 8). Stage one-way, then +enable two-way after a clean drill. +Gate: one-way drill clean (RTO/RPO measured); then two-way; controller restore drill +(D-104) exercised; RTO recorded as a measured output, not an SLA. +Owns: D-108 (replication), D-104 (control-plane backup/restore in the drill). + +### Phase 6 -- Designate, COS, Magnum / CAPI (per DC) +Goal: complete the service set in both DCs. +Build: per DC, Designate (designate-bind) with per-DC zones + A/AAAA per the D-008 +bootstrap order; COS observability; Magnum / CAPI with in-cloud single-homed capi-mgmt +(`domain-setup` re-run per D-046). +Gate: Designate resolving A + AAAA; FQDN-SAN certs valid (D-019/D-021 root cause closed); +COS scraping; a CAPI workload cluster comes up per DC. +Owns: D-106 (Designate), D-105 (CAPI/COS). + +## 5. Cross-cutting: OpenTofu / MAAS / NetBox / Juju boundary + +- NetBox: IPAM apex. All address values originate here; MAAS and overlays consume them. +- OpenTofu: create/destroy of libvirt domains, virtual networks, OPNsense, Office1 VMs; + runs from Office1. Owns the "physical" layer only. +- MAAS: enlist (discover) / commission / deploy / power / release of node VMs. Does not + compose. One region (Office1) + one rack per DC (virsh VM-host). +- Juju: deploys and manages the OpenStack bundle onto MAAS machines; one controller per DC. + +## 6. WAN simulation + +`tc netem` on the inter-DC and site-uplink virtual segments. Default lean: same-metro dark +fiber (low single-digit ms, jumbo-capable). Parameters (latency / jitter / loss / rate) are +a Phase-0 knob and are recorded per drill so RTO/RPO actuals are attributable to a known +link profile. Re-tune to the confirmed Roosevelt inter-DC profile when known. + +## 7. IPv6 rehearsal scope + +VR1 rehearses the full developed IPv6 plan because the simulated OPNsense edge is ours (the +D-015 upstream-router gate does not apply): internal IPv6-only ULA (data / storage / +replication / Octavia lb-mgmt / P2P), internal dual-stack ULA on metal-internal API +endpoints, and external GUA (provider public VIPs, native v6 ext_net, tenant delegation). +Ceph-over-v6 (`ms_bind_ipv6`), geneve-over-v6 MTU, and native v6 tenant routing are the +high-value Roosevelt proofs. Phase-A internal-three goes straight to v6-only (lean; +dual-then-flip is the conservative fallback if Phase-0 verification is not clean). + +## 8. Failover / failback drill skeleton (from the DR seed, carrier corrected) + +Failover (DC1 lost, recover at DC2): +1. Confirm DC1 truly down (avoid split-brain from a transient partition). +2. Glance: promote the mirrored Glance pool at DC2 (force-promote if DC1 unreachable); + re-register images into DC2 Glance. +3. Cinder: restore required volumes into DC2 Cinder from the replicated backups. +4. Rebuild instances from image + restored volume; re-create Neutron ports / FIPs / SGs. +5. Record RTO / RPO actuals. + +Failback (DC1 recovered) -- split-brain-safe for two-way: +1. Do NOT let recovered DC1 resume as primary automatically. +2. Demote DC1's Glance pool; resync from DC2 (now primary). +3. Reconcile Cinder: back up any DC2-side changes; restore into DC1. +4. In a controlled window, optionally flip primary back (demote DC2, promote DC1). +5. Verify both directions healthy; record the drill. + +## 9. Shim register -- the no-Roosevelt-analog steps + +The only steps with NO Roosevelt analog (tag them clearly; do not treat as reusable +production IaC): + +- OpenTofu "create the node VMs" module (D-103). In Roosevelt, node "hardware" already + exists; that step is physical racking + BMC enlistment. Everything from MAAS enlistment + onward IS Roosevelt-transferable. +- `tc netem` WAN simulation (D-100). In Roosevelt the inter-DC link is real fiber; netem + is replaced by the physical link. The route + bandwidth budget it stands in for IS real. +- Single-unit controllers (D-104) and single-unit rbd-mirror daemon (D-108) are VR1 test + economies; Roosevelt scales both to HA. + +## 10. Open items for operator redline + +- Ceph size=3 vs a documented size=2 fallback (gated on the Phase-0 disk math). +- netem link profile (same-metro lean vs a specific Roosevelt inter-DC target). +- Whether D-102 folds into D-101, and whether D-109 stands alone. +- metal-admin admin-API ULA leg (lean: no). +- COS per-DC-only vs an Office1 roll-up (lean: per-DC only). +- Mirror sync topology (independent upstream pull per DC vs seed-from-peer). + +## 11. Landing this deliverable + +1. Append the body of `docs/dc-dc-D100-D110.append.md` (from its first `## D-100` line) to + the end of `docs/design-decisions.md`, then remove the fragment file. +2. Run `bash scripts/ledger-scan.sh`; confirm D next-free = 111 and the machine-derived + ledger block regenerates cleanly. Do not hand-edit the machine-derived block. +3. Add a one-line entry in the main-chat fence of `docs/session-ledger.md` recording this + deliverable (design doc + D-100..D-110 PROPOSED). +4. `bash scripts/repo-lint.sh` for ASCII/LF/format (ignore any next-free noise above 099 -- + known band bug; ledger-scan is the next-free authority). +5. Commit. These are PROPOSED: present for redline, do not implement until the operator + ratifies each to ADOPTED.