Newer
Older
openstack-caracal-dc-dc / docs / dc0-deploy-readiness.md

DC0 (vr1-dc0) deployment readiness + prep checklist

Prep/handoff for the Stage-3 Phase-2 DC0 build (runbooks/dc-dc-phase2-tofu-dc-substrate.md, Model B two-root). Written 2026-07-16 at the outer-apply gate. A FRESH session starts the deployment from here. Nothing has been applied -- the outer tofu apply is the first mutation and is operator-gated. vr1-dc1 is HELD (no ruled addressing; see below), DC0 only.


A. Readiness ESTABLISHED (done, verified this session)

  • Durable agent access (D-126) -- ssh voffice1 / office1-netbox / office1-tailscale / office1-opnsense all EXERCISED live (ProxyJump via ~/.ssh/config.d/vr1-sites, one-key office1 env ~/vr1-office1-creds/office1_svc_ed25519). Committed+pushed (bc2be83).
  • DC creds targets -- ~/vr1-dc0-creds/ + ~/vr1-dc1-creds/ (0700, empty); creds-manifests/vr1-dc{0,1}.manifest declare each DC's dedicated per-env key.
  • dc0 rack tfvars -- opentofu/d124-rack.auto.tfvars (LOCAL/gitignored) holds the four D-124-ruled vr1_dc0_rack_* values (10.12.8.2, 172.31.0.2, /30, 172.31.0.1).
  • Stage-1 gate -- all six vr1-dc0 planes + 3 mesh links + vr1-dc0-pool/office1-pool active.
  • opentofu-validate -- root + 12 modules clean (both roots tofu validate).
  • Blocking design rulings LANDED -- D-121/122/123/124 ADOPTED; R-1 Model B, R-2 vault=v-a MySQL (Raft infeasible on 1.8 charm), R-3 4 storage/DC=9 nodes. Whole-host fit 870/1024 GiB.

B. The outer-root plan -- VERIFIED SAFE (this is the apply gate)

tofu -chdir=opentofu plan => 5 add / 0 change / 6 destroy, decomposed and verified:

  • 5 create: vvr1_dc0 containment VM (425984 MiB = 416 GiB / 108 vCPU -- full-fleet D-121/R-3 sizing, expose_nested_virt) + disk/seed/cloudinit; vr1_dc0_uplink (D-125 simulated-ISP NAT).
  • 6 destroy: the six vr1-dc0 plane networks -- INTENDED Model B relocation from vcloud (outer) to inside vvr1-dc0 (inner root recreates them, vr1-dc0-substrate/main.tf:28). All six are EMPTY (0 leases, 0 attached domains). No live/consumed infra touched.
  • RAM gate: host 980 GiB available >= 416 GiB needed (only voffice1 32G + opnsense 2G committed).

A fresh session MUST re-run the plan and re-confirm 5-add/0-change/6-destroy-empty before apply (live state can drift; verify, don't assume).


C. REMAINING PREP -- before / during the deploy

C1. Pre-apply (settle BEFORE the outer tofu apply)

  1. [RESOLVED 2026-07-16] Per-env key for vvr1-dc0 = option (a) dedicated dc0 key (operator ruling). DONE: minted ~/vr1-dc0-creds/vr1-dc0_svc_ed25519 (600/644, creds-audit vr1-dc0 CLEAN); added variable "vr1_dc0_ssh_pubkey_path"; vvr1-dc0 cloud-init now installs the dc0 pubkey (voffice1 keeps office1's key); tfvar set. Re-validated + re-planned -> outer plan UNCHANGED (5/0/6, create-only content). Remaining for the inner apply: the inner root's qemu+ssh to vvr1-dc0 must use the dc0 PRIVATE key -- wire an ssh alias/-i ~/vr1-dc0-creds/vr1-dc0_svc_ed25519 AFTER the outer apply boots vvr1-dc0 (its transit IP is knowable then; D-126 DC ssh-config rows are deferred to that point).
  2. [ENV] bunzip2 -- required by scripts/opnsense-prep-image.sh. Under Model B that prep runs ON vvr1-dc0 (bootstrap step B), so install it there (apt) post-boot; NOT needed for the outer apply. (Absent on vcloud today -- fine unless any prep is run vcloud-side.)

C2. Mid-deploy prerequisites (prep now, needed at a specific gate)

  1. [SECRET/REACH] MAAS reachability for Step 9 (maas-vm-host register, Model B step D): maas_api_url/maas_api_key are deliberately unset (empirically revealed at Step 8/9); maas_api_key is a SECRET via TF_VAR_maas_api_key. Confirm the Office1 MAAS region is reachable and export the key before Step 9. Steps A-C (outer apply -> bootstrap -> inner node/plane/edge) do NOT need MAAS.
  2. [RULED 2026-07-16] netem parameters = PLACEHOLDER for the rehearsal (operator ruling). Exercise the netem-link mechanism with a placeholder value at Step 11; the "final" latency/loss ruling (D-100 gap #11) is DEFERRED. Non-blocking for the outer apply.

C3. Deploy-time GATES (exercised in-sequence during the deploy, per the runbook)

  • Outer: vvr1-dc0 boots + sized + nested-KVM on (the rack IS vvr1-dc0).
  • Bootstrap (B): site-headend-install.sh --role rack --host-nodes --check PASS incl. SEC-010 (transit FORWARD-drop, DC-LOCAL); prep OPNsense base ON vvr1-dc0.
  • Inner (C): node VMs boot at depth-4 nested virt (D-114/D-123 boot gate -- flagged risk, accepted); boot-measure WAN_IF/LAN_IF on the real OPNsense; MAAS zone/pool + power_address.
  • D-125 egress: foreign-MAC-across-nested-bridge isolation test (Step 12) -- throwaway probe.
  • MAAS (D): discovers the inner node VMs (needs C2.3).

C4. IPAM record hygiene (parallel, NON-blocking -- now runnable via ssh office1-netbox)

  1. D-124 NetBox import into office1-netbox (transit role + 172.31.0.0/24 carve + rack IP): run ON the netbox host (token local at /root/netbox-secrets/api.token) -- python3 netbox/dc-rack-mgmt-import.py --transit-cidr 172.31.0.0/30 --rack-ip 10.12.8.2 (dry-run; add --commit). tofu plan does NOT need this (no netbox provider); it is IPAM correctness.
  2. Edge-WAN import (172.30.2.0/24, .3.0/24) via netbox/dc-edge-wan-import.py (after d115-office-carve.py seeds Office+Edge roles).

C5. Record-hygiene sweep (NON-blocking; queued from the Stage-3 adversarial review)

  1. Sweep the queued amendments into docs/design-decisions.md: D-122 MAAS-bullet + site-down wording, D-107 node-artifacts scoping + DR-claim honesty, D-121 Option-C disk record (the optc-calc.py "reproducible" reference is to an ABSENT script -- fix or commit the calc; whole-host math lives in scripts/dc-dc-whole-host-budget.py), D-115 status-line vs ledger-scan, SEC-011 reconsider (downgraded to OBSERVATION).

C6. vr1-dc1 -- HELD (do not attempt)

No ruled dc1 transit/rack addressing (the 172.31.0.0/24 transit supernet is dc0-scoped), and no vr1_dc1_rack_* variables or dc1 substrate exist. dc1 needs its own transit-prefix ruling + variable

  • substrate creation. Separate decision; NOT part of the DC0 build.

D. Fresh-session bootstrap (start the deployment cleanly)

  1. git pull ; bash scripts/repo-lint.sh (0 fail; 1 legacy WARN) ; bash scripts/run-tests-all.sh.
  2. Read docs/session-ledger.md + bash scripts/ledger-scan.sh -- reconcile in-flight state.
  3. Read THIS doc + runbooks/dc-dc-phase2-tofu-dc-substrate.md (Model B reshape section, ~line 144).
  4. Start logging (cloud-mutating session): bash scripts/run-logged.sh dc0-deploy.
  5. Settle C1.1 (the per-env-key ruling) first.
  6. Re-run the outer plan; confirm 5-add/0-change/6-destroy-empty; then the gated outer apply.
  7. Proceed A -> B -> C -> D -> E, each mutation individually approved; review after each step.

E. Gated deploy sequence (Model B, from the runbook)

Step Action Type
A outer tofu apply -- vvr1-dc0 (416G/108) + vr1_dc0_uplink; relocate 6 empty planes MUTATION (gated)
B bootstrap vvr1-dc0: site-headend-install.sh --role rack --host-nodes + SEC-010; prep OPNsense base there MUTATION (gated)
C inner tofu apply (vr1-dc0-substrate, qemu+ssh): 9 node VMs + 6 planes + OPNsense edge INSIDE vvr1-dc0 MUTATION (gated)
D maas-vm-host register vvr1-dc0 inner virsh -> Office1 region; MAAS discovers nodes MUTATION (gated, needs C2.3)
E netem on the vcloud mesh bridges (outer) MUTATION (gated, needs C2.4)
GATE D-125 foreign-MAC egress isolation test; depth-4 boot; SEC-010 packet-drop verify

EXIT -> Stage 4 (per-DC), once dc1 clears its own gate.