diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md new file mode 100644 index 0000000..6e89cd2 --- /dev/null +++ b/docs/dc-dc-deployment-workflow.md @@ -0,0 +1,253 @@ +# DC-DC (VR1) deployment workflow overview + +**Purpose.** A single followable checklist across the whole VR1 (two-DC virtual +regional) buildout, spanning many sessions. `docs/dc-dc-buildout-design.md` is the +WHY/WHAT (goals, gates, D-refs per phase); this doc is the HOW-TO-FOLLOW-IT +sequence, cross-referenced to that plan, to D-100..D-110, and to what already +exists in this repo vs what still needs to be authored. Update the "state" column +as stages complete -- this doc is a tracker, not a one-time snapshot. + +**Governing constraint (unchanged throughout):** minimize delta to Roosevelt. Reuse +VR0 DC0 material wherever it transfers; the only deliberate no-Roosevelt-analog +steps are the Section 9 shim register in the buildout design (OpenTofu node-VM +creation, `tc netem` WAN simulation, single-unit controllers/rbd-mirror). + +--- + +## Stage 0 -- Decision ratification (BLOCKING) + +Nothing in Stage 1 onward starts until this clears. D-100..D-110 are ALL +**PROPOSED**, not ADOPTED (see `docs/design-decisions.md`). + +1. Operator reviews `docs/dc-dc-buildout-design.md` in full. +2. Rule on each of D-100..D-110 individually (ADOPTED / AMENDED / REJECTED) -- + not a blanket approval; each carries distinct, separable scope. +3. Rule on the 6 open-redline items (buildout design Section 10): + - Ceph `size=3` vs a documented `size=2` fallback (gated on Phase-0 disk math) + - `tc netem` link profile (same-metro lean vs a specific Roosevelt target) + - Whether D-102 folds into D-101, and whether D-109 stands alone + - metal-admin admin-API ULA leg (lean: no) + - COS per-DC-only vs an Office1 roll-up (lean: per-DC only) + - Mirror sync topology (independent upstream pull per DC vs seed-from-peer) +4. Record each ruling as an amendment in `docs/design-decisions.md` (same + convention as every prior D-NNN ruling in this repo). + +**Exit gate:** every D-100..D-110 is ADOPTED or explicitly deferred with a +documented fallback. `bash scripts/ledger-scan.sh` should show none of them under +"PROPOSED / OPEN decisions" (or only the explicitly-deferred ones). + +**State:** NOT STARTED. + +--- + +## Stage 1 -- Phase 0: vcloud host preparation + +| | | +|---|---| +| **Goal** | Turn the bare vcloud host into a substrate Office1 can deploy from. | +| **Build** | Nested KVM enabled; libvirt storage pools + thin-provisioning; dark-fiber + per-site ISP-edge virtual networks; the six per-DC plane segments as virtual networks; MTU plan set on the underlay. | +| **Gate** | Nested KVM verified; disk budget computed + Ceph size decision recorded; measured MTU recorded; virtual networks present and isolated as designed. | +| **Owns** | D-100 (fabric), D-101/D-102 (MTU), buildout-design Section 3 (host/disk/MTU gates). | +| **Reuse vs new** | NEW. No VR0-DC0 analog -- that testcloud IS the single vcloud host, already prepared; this is preparing the host to carry TWO independent DC substrates plus Office1. | +| **Authoring status** | No runbook yet. First `phase-0-dc-dc-*.md`-class runbook to write. | + +**State:** NOT STARTED. + +--- + +## Stage 2 -- Phase 1: Office1 headend standup + +| | | +|---|---| +| **Goal** | Stand up the operator's deployment headend first, before any DC. | +| **Build** | MAAS region controller; OpenTofu; NetBox (vcloud-local, importer extended for multi-DC + dual-stack); GitBucket (vcloud-local mirror); Tailscale front door (Office1 only). | +| **Gate** | MAAS region reachable; NetBox authoritative + populated (planes, per-DC v4, ULA/GUA carve); GitBucket serving; OpenTofu reaches vcloud host libvirt; Tailscale confirmed to Office1 only. | +| **Owns** | D-103 (Office1 VMs are OpenTofu-created), D-107 (headend is not a core-service provider). | +| **Reuse vs new** | NEW infra (Office1 didn't exist for VR0 DC0), but the *practices* transfer directly: NetBox-as-IPAM-apex discipline, the `netbox/*-import.py` pattern, and this repo's own git/GitBucket hosting model are all precedent, just extended. | +| **Authoring status** | No runbook yet. | + +**State:** NOT STARTED. + +--- + +## Stage 3 -- Phase 2: OpenTofu builds each DC substrate (from Office1) + +| | | +|---|---| +| **Goal** | From Office1, script each DC's "physical" layer into existence. | +| **Build** | Per DC: OpenTofu creates node-VM libvirt domains (SHIM, Section 9), the DC's plane networks, and its OPNsense edge; registers the DC libvirt host to that DC's MAAS rack controller as a virsh VM-host. | +| **Gate** | MAAS rack controller per DC up; MAAS discovers the OpenTofu-created node VMs; edge routing + simulated ISP uplink up; netem parameters applied and measured. | +| **Owns** | D-103 (seam), D-100 (edges). | +| **Reuse vs new** | NEW (no OpenTofu/multi-rack precedent in VR0 DC0). Explicitly tagged shim: node-VM creation has no Roosevelt analog (physical racking + BMC enlistment replaces it there) -- do not treat as reusable production IaC. | +| **Authoring status** | No runbook/OpenTofu module yet. | + +**State:** NOT STARTED. + +--- + +## Stage 4 -- Phase 3: MAAS enlist / commission / deploy (per DC) + +| | | +|---|---| +| **Goal** | Bring the discovered node VMs to a deployed OS, ready for Juju. | +| **Build** | Per DC: MAAS commissions + deploys the node VMs; provider NIC left RAW (br-ex discipline, D-100); VLAN-103 metal-internal bridge stack pre-built by MAAS. | +| **Gate** | Nodes deployed; six planes present per node with correct fabrics/VLANs; provider NIC raw; PXE (v4) working; per-DC mirror reachable; NTP from edge working. | +| **Owns** | D-103, D-100 (br-ex), D-107 (mirror + NTP). | +| **Reuse vs new** | PARTIAL reuse. `runbooks/phase-00-teardown-maas-reset.md` carries the MAAS commission/deploy + br-ex discipline this repo already validated for a single rack; the per-DC multi-rack registration is new, the deploy mechanics are precedent. | +| **Authoring status** | No dc-dc runbook yet; adapt phase-00's MAAS section per-DC. | + +**State:** NOT STARTED. + +--- + +## Stage 5 -- Phase 4: Juju controller + OpenStack bundle (per DC) -- HIGH REUSE + +| | | +|---|---| +| **Goal** | A running independent OpenStack cloud per DC. | +| **Build** | Per DC: bootstrap the single-unit Juju controller in-DC; deploy the Caracal bundle; Vault init with a per-DC independent root, issuing v4 + IPv6-SAN certs; apply the family matrix (ULA on data/storage/replication, dual-stack metal-internal API, GUA provider). | +| **Gate** | `preflight.sh` PASS before add-model; post-deploy `cloud-assert.sh --capture` per DC; controller backup taken + stored; Ceph-over-v6 and geneve-over-v6 verified. | +| **Owns** | D-104 (controllers), D-109 (Vault), D-101 (families). | +| **Reuse vs new** | **This is the ground-truth/copy point** (`docs/dc-dc-buildout-design.md` line 12: seeded from `openstack-caracal-ipv4` at 8813efc). `bundle.yaml` and `runbooks/phase-01..08-*.md` are the per-DC template -- run essentially the existing VR0 DC0 deploy loop TWICE (once per DC), adapted for: independent Keystones (no cross-DC federation), the IPv6 family matrix (D-101/D-102), and per-DC Vault roots (D-109) instead of the single shared root. | +| **Authoring status** | Existing `phase-01..08` runbooks are the starting point; need a per-DC parameterization pass (D-101 family matrix, D-109 Vault) before reuse, not a from-scratch write. | + +**State:** NOT STARTED. **This is where the existing repo tooling (`preflight.sh`, +`cloud-assert.sh`, `scripts/lib-net.sh`/`lib-hosts.sh`, `run-logged.sh`) does the +most direct work -- confirmed repo-agnostic as of the 2026-07-09 sweep, so they +apply to either DC's clone/session without modification.** + +--- + +## Stage 6 -- Phase 5: DR wiring and failover drill + +| | | +|---|---| +| **Goal** | Prove Tier-A recoverability, one-way first, then two-way. | +| **Build** | Replication-plane peering (ULA); radosgw multisite zonegroup (two DCs as zones); rbd-mirror for the Glance pool; cinder-backup to the replicated object store; consistency groups where needed. Run the failover/failback drill. | +| **Gate** | One-way drill clean (RTO/RPO measured); then two-way; controller restore drill (D-104) exercised; RTO recorded as a measured output, not an SLA. | +| **Owns** | D-108 (replication), D-104 (control-plane backup/restore in the drill). | +| **Reuse vs new** | Ground truth: `docs/dc-dc-replication-DR-seed.md` (the original Tier-A/bidirectional/cinder-backup+rbd-mirror decision) -- D-108 corrected its carrier from the seed's v4 replication plane to IPv6-only ULA. The failover/failback drill skeleton is already drafted (buildout design Section 8): confirm DC1 down -> promote Glance -> restore Cinder -> rebuild instances -> record RTO/RPO; failback is split-brain-safe (no auto-resume, controlled window to flip primary back). | +| **Authoring status** | Skeleton exists; needs a runbook with command-level steps + harness. | + +**State:** NOT STARTED. + +--- + +## Stage 7 -- Phase 6: Designate, COS, Magnum/CAPI (per DC) + +| | | +|---|---| +| **Goal** | Complete the service set in both DCs. | +| **Build** | Per DC: Designate (designate-bind) with per-DC zones + A/AAAA per the D-008 bootstrap order; COS observability; Magnum/CAPI with in-cloud single-homed capi-mgmt (`domain-setup` re-run per D-046). | +| **Gate** | Designate resolving A + AAAA; FQDN-SAN certs valid (D-019/D-021 root cause closed); COS scraping; a CAPI workload cluster comes up per DC. | +| **Owns** | D-106 (Designate), D-105 (CAPI/COS). | +| **Reuse vs new** | HIGH REUSE for CAPI/Magnum: `runbooks/phase-06-incloud-mgmt-cluster.md` + `phase-07-conductor-graft.md` + `phase-08-workload-cluster-acceptance.md` are direct precedent, per-DC. Designate is a REVERSAL, not new build: VR0 DC0 deferred it (D-019, "DNS scope reduction"); D-106 reactivates it (supersedes D-019, reactivates the original D-008 bootstrap order) -- read D-008 and D-019 together before writing this stage's runbook, the static-`/etc/hosts` workaround they document goes away. | +| **Authoring status** | No dc-dc runbook yet; adapt phase-06/07/08 per-DC, write Designate reactivation fresh. | + +**State:** NOT STARTED. + +--- + +## Cross-cutting discipline (applies at every stage, every session) + +- **Session bootstrap, every session:** `git pull` -> `bash scripts/repo-lint.sh` + (0 fail / 1 documented legacy WARN expected) -> `bash scripts/ledger-scan.sh` + to reconcile open decisions/next-free numbers against `docs/session-ledger.md`. +- **Any session that will mutate the cloud:** `bash scripts/run-logged.sh