DC0 (vr1-dc0) deployment readiness + prep checklist

Prep/handoff for the Stage-3 Phase-2 DC0 build (runbooks/dc-dc-phase2-tofu-dc-substrate.md, Model B two-root). Written 2026-07-16 at the outer-apply gate. A FRESH session starts the deployment from here. Nothing has been applied -- the outer tofu apply is the first mutation and is operator-gated. vr1-dc1 is HELD (no ruled addressing; see below), DC0 only.

STATUS 2026-07-17: PREP COMPLETE -- DC0 IS DEPLOY-READY. Final gate re-checked: opentofu-validate clean (both roots + 12 modules), the four vr1_dc0_rack_* + vr1_dc0_ssh_pubkey_path tfvars in place, creds-audit vr1-dc0 CLEAN, gauntlet ALL GREEN (65), repo-lint 0-fail. Per-env-key (D-126 option a) DONE; netem = placeholder (deploy-time, Step 11). The ONLY remaining items are NOT prep: (1) the final bookend restart -- OS patch + reboot of vcloud + voffice1 + office1-opnsense, done together as the boundary into the new session (voffice1 is load-bearing like vcloud -- it hosts netbox/MAAS + the only netbox path, so it defers WITH vcloud, not mid-prep); (2) fresh-session start = re-verify the plan (state drifts after a reboot -- VMs stop) + open a logged shell + gated outer apply. Decoupled hygiene (NetBox D-124/edge-WAN import; the Stage-3 record sweep) is OPTIONAL and can run anytime -- it does NOT gate the deploy (no netbox provider in the tree).

A. Readiness ESTABLISHED (done, verified this session)

• Durable agent access (D-126) -- ssh voffice1 / office1-netbox / office1-tailscale / office1-opnsense all EXERCISED live (ProxyJump via ~/.ssh/config.d/vr1-sites, one-key office1 env ~/vr1-office1-creds/office1_svc_ed25519). Committed+pushed (bc2be83).
• DC creds targets -- ~/vr1-dc0-creds/ + ~/vr1-dc1-creds/ (0700, empty); creds-manifests/vr1-dc{0,1}.manifest declare each DC's dedicated per-env key.
• dc0 rack tfvars -- opentofu/d124-rack.auto.tfvars (LOCAL/gitignored) holds the four D-124-ruled vr1_dc0_rack_* values (10.12.8.2, 172.31.0.2, /30, 172.31.0.1).
• Stage-1 gate -- all six vr1-dc0 planes + 3 mesh links + vr1-dc0-pool/office1-pool active.
• opentofu-validate -- root + 12 modules clean (both roots tofu validate).
• Blocking design rulings LANDED -- D-121/122/123/124 ADOPTED; R-1 Model B, R-2 vault=v-a MySQL (Raft infeasible on 1.8 charm), R-3 4 storage/DC=9 nodes. Whole-host fit 870/1024 GiB.

B. The outer-root plan -- VERIFIED SAFE (this is the apply gate)

tofu -chdir=opentofu plan => 5 add / 2 in-place / 6 destroy, decomposed and verified:

• 2 in-place (D-127 autostart, 2026-07-17): voffice1 + office1-opnsense get autostart false->true -- an update-in-place (verified: NOT replacement; replacement would destroy the live host/edge). These COLLAPSE to 0-change once the bookend sets them autostart=true live before the fresh-session apply.
• 5 create: vvr1_dc0 containment VM (425984 MiB = 416 GiB / 108 vCPU -- full-fleet D-121/R-3 sizing, expose_nested_virt) + disk/seed/cloudinit; vr1_dc0_uplink (D-125 simulated-ISP NAT).
• 6 destroy: the six vr1-dc0 plane networks -- INTENDED Model B relocation from vcloud (outer) to inside vvr1-dc0 (inner root recreates them, vr1-dc0-substrate/main.tf:28). All six are EMPTY (0 leases, 0 attached domains). No live/consumed infra touched.
• RAM gate: host 980 GiB available >= 416 GiB needed (only voffice1 32G + opnsense 2G committed).

A fresh session MUST re-run the plan and re-confirm 5-add/(0-or-2)-change/6-destroy-empty (autostart in-place collapses to 0 post-bookend) before apply (live state can drift; verify, don't assume).

C. REMAINING PREP -- before / during the deploy

C1. Pre-apply (settle BEFORE the outer tofu apply)

1. [RESOLVED 2026-07-16] Per-env key for vvr1-dc0 = option (a) dedicated dc0 key (operator ruling). DONE: minted ~/vr1-dc0-creds/vr1-dc0_svc_ed25519 (600/644, creds-audit vr1-dc0 CLEAN); added variable "vr1_dc0_ssh_pubkey_path"; vvr1-dc0 cloud-init now installs the dc0 pubkey (voffice1 keeps office1's key); tfvar set. Re-validated + re-planned -> outer plan UNCHANGED (5/0/6, create-only content). Remaining for the inner apply: the inner root's qemu+ssh to vvr1-dc0 must use the dc0 PRIVATE key -- wire an ssh alias/-i ~/vr1-dc0-creds/vr1-dc0_svc_ed25519 AFTER the outer apply boots vvr1-dc0 (its transit IP is knowable then; D-126 DC ssh-config rows are deferred to that point).
2. [ENV] bunzip2 -- required by scripts/opnsense-prep-image.sh. Under Model B that prep runs ON vvr1-dc0 (bootstrap step B), so install it there (apt) post-boot; NOT needed for the outer apply. (Absent on vcloud today -- fine unless any prep is run vcloud-side.)

C2. Mid-deploy prerequisites (prep now, needed at a specific gate)

1. [SECRET/REACH] MAAS reachability for Step 9 (maas-vm-host register, Model B step D): maas_api_url/maas_api_key are deliberately unset (empirically revealed at Step 8/9); maas_api_key is a SECRET via TF_VAR_maas_api_key. Confirm the Office1 MAAS region is reachable and export the key before Step 9. Steps A-C (outer apply -> bootstrap -> inner node/plane/edge) do NOT need MAAS.
2. [RULED 2026-07-16] netem parameters = PLACEHOLDER for the rehearsal (operator ruling). Exercise the netem-link mechanism with a placeholder value at Step 11; the "final" latency/loss ruling (D-100 gap #11) is DEFERRED. Non-blocking for the outer apply.

C3. Deploy-time GATES (exercised in-sequence during the deploy, per the runbook)

• Outer: vvr1-dc0 boots + sized + nested-KVM on (the rack IS vvr1-dc0).
• Bootstrap (B): site-headend-install.sh --role rack --host-nodes --check PASS incl. SEC-010 (transit FORWARD-drop, DC-LOCAL); prep OPNsense base ON vvr1-dc0.
• Inner (C): node VMs boot at depth-4 nested virt (D-114/D-123 boot gate -- flagged risk, accepted); boot-measure WAN_IF/LAN_IF on the real OPNsense; MAAS zone/pool + power_address.
• D-125 egress: foreign-MAC-across-nested-bridge isolation test (Step 12) -- throwaway probe.
• MAAS (D): discovers the inner node VMs (needs C2.3).

C4. IPAM record hygiene (parallel, NON-blocking -- now runnable via ssh office1-netbox)

1. D-124 NetBox import -- DONE 2026-07-17. netbox/d124-transit-seed.py (NEW) seeded role transit
   • 172.31.0.0/24 container; dc-rack-mgmt-import placed 172.31.0.0/30 (vr1-dc0) + 10.12.8.2/22 (rack). Ran on the apex host, token server-side, sandbox-only.
2. Edge-WAN import -- DONE 2026-07-17. dc-edge-wan-import created 172.30.2.0/24 + 172.30.3.0/24 (edge). (Also recovered the apex: netbox-docker was down; brought up + restart: unless-stopped added.)

C5. Record-hygiene sweep (NON-blocking; queued from the Stage-3 adversarial review)

1. Sweep the queued amendments into docs/design-decisions.md: D-122 MAAS-bullet + site-down wording, D-107 node-artifacts scoping + DR-claim honesty, D-121 Option-C disk record (the optc-calc.py "reproducible" reference is to an ABSENT script -- fix or commit the calc; whole-host math lives in scripts/dc-dc-whole-host-budget.py), D-115 status-line vs ledger-scan, SEC-011 reconsider (downgraded to OBSERVATION).

C6. vr1-dc1 -- HELD (do not attempt)

No ruled dc1 transit/rack addressing (the 172.31.0.0/24 transit supernet is dc0-scoped), and no vr1_dc1_rack_* variables or dc1 substrate exist. dc1 needs its own transit-prefix ruling + variable

• substrate creation. Separate decision; NOT part of the DC0 build.

BOOKEND -- vcloud HOST patch + reboot (run BEFORE section D)

Closes the last bookend item: patch the vcloud host and reboot onto kernel 6.8.0-136 (the guests already run -136; the host lags at -134). Measured 2026-07-17: -136 is PENDING in apt but NOT installed, no reboot-required flag -- so a bare reboot gains nothing; the kernel must be apt-installed first. Pending set is linux-*-generic (-134->-136) + libntfs-3g89t64 ntfs-3g linux-libc-dev linux-tools-common tar ubuntu-pro-client{,-l10n} -- no qemu/libvirt, so no hypervisor upgrade to complicate guest bring-up.

Preconditions / rationale:

• Remote Control attached + all decisions done FIRST -- Step 4 (sudo reboot) ENDS the running session.
• voffice1 + office1-opnsense are BOTH autostart=enable (set 2026-07-17, edge after its verified cold boot) -- so the host reboot self-recovers them. LXD boot.autostart=true on netbox+tailscale inside voffice1.
• libvirt-guests is enabled+active (default ON_SHUTDOWN=suspend = managedsave). Step 3 explicitly virsh shutdowns the guests first so the reboot exercises the verified cold-boot autostart path (not managedsave-restore). Ordered to patch with VMs UP; quiesce only for the reboot.

# Run ON vcloud. Steps 2-4 are gated/sudo; STEP 4 ENDS THIS SESSION.

# STEP 0  preflight / baseline (read-only)
uptime; uname -r; virsh list --all; free -g

# STEP 1  refresh lists + confirm the plan (read-only)
sudo apt update
apt list --upgradable 2>/dev/null            # expect linux-*-generic -134 -> -136 + ~7 pkgs
sudo apt-get -s full-upgrade | tail -20      # dry-run: review before committing

# STEP 2  install upgrades (guests stay UP; no reboot yet)
sudo apt full-upgrade -y
dpkg -l 'linux-image-6.8.0-136-generic' | grep '^ii' && echo "kernel -136 INSTALLED (pending reboot)"
cat /var/run/reboot-required 2>/dev/null

# STEP 3  gracefully quiesce guests (cold-boot path)
virsh shutdown voffice1                        # cascades: LXD stops netbox + tailscale cleanly
virsh shutdown office1-opnsense
for d in voffice1 office1-opnsense; do virsh domstate $d; done   # wait until BOTH "shut off"

# STEP 4  reboot the host  (THIS ENDS THE SESSION)
sudo reboot

Post-reboot verify (this feeds section D step 1; autostart should have restored everything):

uname -r                                        # expect 6.8.0-136-generic
virsh list --all                                # voffice1 + office1-opnsense running (autostart)
ssh voffice1 'ping -c2 10.10.0.1; ping -c2 -I 10.10.0.20 1.1.1.1'          # edge LAN + egress
ssh office1-netbox 'curl -s -o /dev/null -w "netbox=%{http_code}\n" http://localhost:8000/'
ssh office1-tailscale 'tailscale status | head -1'

After this lands the host is on -136 (matching the guests) and the bookend is CLOSED -> proceed to section D.

D. Fresh-session bootstrap (start the deployment cleanly)

1. git pull ; bash scripts/repo-lint.sh (0 fail; 1 legacy WARN) ; bash scripts/run-tests-all.sh.
2. Read docs/session-ledger.md + bash scripts/ledger-scan.sh -- reconcile in-flight state.
3. Read THIS doc + runbooks/dc-dc-phase2-tofu-dc-substrate.md (Model B reshape section, ~line 144).
4. Start logging (cloud-mutating session): bash scripts/run-logged.sh dc0-deploy.
5. Settle C1.1 (the per-env-key ruling) first.
6. Re-run the outer plan; confirm 5-add/(0-or-2)-change/6-destroy-empty (autostart in-place collapses to 0 post-bookend); then the gated outer apply.
7. Proceed A -> B -> C -> D -> E, each mutation individually approved; review after each step.

E. Gated deploy sequence (Model B, from the runbook)

EXIT -> Stage 4 (per-DC), once dc1 clears its own gate.