# DC0 (vr1-dc0) deployment readiness + prep checklist

Prep/handoff for the Stage-3 Phase-2 DC0 build (`runbooks/dc-dc-phase2-tofu-dc-substrate.md`,
Model B two-root). Written 2026-07-16 at the outer-apply gate. A FRESH session starts the
deployment from here. **Nothing has been applied** -- the outer `tofu apply` is the first mutation
and is operator-gated. `vr1-dc1` is HELD (no ruled addressing; see below), DC0 only.

> **STATUS 2026-07-17: PREP COMPLETE -- DC0 IS DEPLOY-READY.** Final gate re-checked: opentofu-validate
> clean (both roots + 12 modules), the four `vr1_dc0_rack_*` + `vr1_dc0_ssh_pubkey_path` tfvars in place,
> `creds-audit vr1-dc0` CLEAN, gauntlet ALL GREEN (65), repo-lint 0-fail. Per-env-key (D-126 option a)
> DONE; netem = placeholder (deploy-time, Step 11). The ONLY remaining items are NOT prep: (1) the final
> **bookend restart** -- OS patch + reboot of vcloud + voffice1 + office1-opnsense, done together as the
> boundary into the new session (voffice1 is load-bearing like vcloud -- it hosts netbox/MAAS + the only
> netbox path, so it defers WITH vcloud, not mid-prep); (2) fresh-session start = re-verify the plan
> (state drifts after a reboot -- VMs stop) + open a logged shell + gated outer apply. Decoupled hygiene
> (NetBox D-124/edge-WAN import; the Stage-3 record sweep) is OPTIONAL and can run anytime -- it does NOT
> gate the deploy (no netbox provider in the tree).

---

## A. Readiness ESTABLISHED (done, verified this session)

- **Durable agent access (D-126)** -- `ssh voffice1` / `office1-netbox` / `office1-tailscale` /
  `office1-opnsense` all EXERCISED live (ProxyJump via `~/.ssh/config.d/vr1-sites`, one-key office1
  env `~/vr1-office1-creds/office1_svc_ed25519`). Committed+pushed (`bc2be83`).
- **DC creds targets** -- `~/vr1-dc0-creds/` + `~/vr1-dc1-creds/` (0700, empty);
  `creds-manifests/vr1-dc{0,1}.manifest` declare each DC's dedicated per-env key.
- **dc0 rack tfvars** -- `opentofu/d124-rack.auto.tfvars` (LOCAL/gitignored) holds the four
  D-124-ruled `vr1_dc0_rack_*` values (`10.12.8.2`, `172.31.0.2`, /30, `172.31.0.1`).
- **Stage-1 gate** -- all six `vr1-dc0` planes + 3 mesh links + `vr1-dc0-pool`/`office1-pool` active.
- **`opentofu-validate`** -- root + 12 modules clean (both roots `tofu validate`).
- **Blocking design rulings LANDED** -- D-121/122/123/124 ADOPTED; R-1 Model B, R-2 vault=v-a
  MySQL (Raft infeasible on 1.8 charm), R-3 4 storage/DC=9 nodes. Whole-host fit 870/1024 GiB.

## B. The outer-root plan -- VERIFIED SAFE (this is the apply gate)

`tofu -chdir=opentofu plan` => **5 add / 2 in-place / 6 destroy**, decomposed and verified:
- **2 in-place (D-127 autostart, 2026-07-17):** voffice1 + office1-opnsense get `autostart false->true`
  -- an **update-in-place** (verified: NOT replacement; replacement would destroy the live host/edge).
  These COLLAPSE to 0-change once the bookend sets them autostart=true live before the fresh-session apply.
- **5 create:** `vvr1_dc0` containment VM (**425984 MiB = 416 GiB / 108 vCPU** -- full-fleet D-121/R-3
  sizing, expose_nested_virt) + disk/seed/cloudinit; `vr1_dc0_uplink` (D-125 simulated-ISP NAT).
- **6 destroy:** the six `vr1-dc0` plane networks -- INTENDED Model B relocation from vcloud (outer)
  to inside `vvr1-dc0` (inner root recreates them, `vr1-dc0-substrate/main.tf:28`). All six are
  EMPTY (0 leases, 0 attached domains). No live/consumed infra touched.
- **RAM gate:** host 980 GiB available >= 416 GiB needed (only voffice1 32G + opnsense 2G committed).

> A fresh session MUST re-run the plan and re-confirm 5-add/(0-or-2)-change/6-destroy-empty (autostart in-place collapses to 0 post-bookend) before apply
> (live state can drift; verify, don't assume).

---

## C. REMAINING PREP -- before / during the deploy

### C1. Pre-apply (settle BEFORE the outer `tofu apply`)
1. **[RESOLVED 2026-07-16] Per-env key for `vvr1-dc0` = option (a) dedicated dc0 key (operator ruling).**
   DONE: minted `~/vr1-dc0-creds/vr1-dc0_svc_ed25519` (600/644, `creds-audit vr1-dc0` CLEAN); added
   `variable "vr1_dc0_ssh_pubkey_path"`; `vvr1-dc0` cloud-init now installs the dc0 pubkey (voffice1
   keeps office1's key); tfvar set. Re-validated + re-planned -> outer plan UNCHANGED (5/0/6, create-only
   content). **Remaining for the inner apply:** the inner root's `qemu+ssh` to `vvr1-dc0` must use the dc0
   PRIVATE key -- wire an ssh alias/`-i ~/vr1-dc0-creds/vr1-dc0_svc_ed25519` AFTER the outer apply boots
   `vvr1-dc0` (its transit IP is knowable then; D-126 DC ssh-config rows are deferred to that point).
2. **[ENV] `bunzip2`** -- required by `scripts/opnsense-prep-image.sh`. Under Model B that prep runs
   ON `vvr1-dc0` (bootstrap step B), so install it there (apt) post-boot; NOT needed for the outer
   apply. (Absent on vcloud today -- fine unless any prep is run vcloud-side.)

### C2. Mid-deploy prerequisites (prep now, needed at a specific gate)
3. **[SECRET/REACH] MAAS reachability** for Step 9 (`maas-vm-host` register, Model B step D):
   `maas_api_url`/`maas_api_key` are deliberately unset (empirically revealed at Step 8/9);
   `maas_api_key` is a SECRET via `TF_VAR_maas_api_key`. Confirm the Office1 MAAS region is reachable
   and export the key before Step 9. Steps A-C (outer apply -> bootstrap -> inner node/plane/edge) do
   NOT need MAAS.
4. **[RULED 2026-07-16] netem parameters = PLACEHOLDER for the rehearsal (operator ruling).** Exercise
   the `netem-link` mechanism with a placeholder value at Step 11; the "final" latency/loss ruling
   (D-100 gap #11) is DEFERRED. Non-blocking for the outer apply.

### C3. Deploy-time GATES (exercised in-sequence during the deploy, per the runbook)
- Outer: `vvr1-dc0` boots + sized + nested-KVM on (the rack IS `vvr1-dc0`).
- Bootstrap (B): `site-headend-install.sh --role rack --host-nodes --check` PASS incl. SEC-010
  (transit FORWARD-drop, DC-LOCAL); prep OPNsense base ON `vvr1-dc0`.
- Inner (C): node VMs boot at **depth-4** nested virt (D-114/D-123 boot gate -- flagged risk,
  accepted); boot-measure `WAN_IF`/`LAN_IF` on the real OPNsense; MAAS zone/pool + `power_address`.
- D-125 egress: foreign-MAC-across-nested-bridge isolation test (Step 12) -- throwaway probe.
- MAAS (D): discovers the inner node VMs (needs C2.3).

### C4. IPAM record hygiene (parallel, NON-blocking -- now runnable via `ssh office1-netbox`)
5. **D-124 NetBox import** -- DONE 2026-07-17. `netbox/d124-transit-seed.py` (NEW) seeded role `transit`
   + `172.31.0.0/24` container; `dc-rack-mgmt-import` placed `172.31.0.0/30` (vr1-dc0) + `10.12.8.2/22`
   (rack). Ran on the apex host, token server-side, sandbox-only.
6. **Edge-WAN import** -- DONE 2026-07-17. `dc-edge-wan-import` created `172.30.2.0/24` + `172.30.3.0/24`
   (edge). (Also recovered the apex: netbox-docker was down; brought up + `restart: unless-stopped` added.)

### C5. Record-hygiene sweep (NON-blocking; queued from the Stage-3 adversarial review)
7. Sweep the queued amendments into `docs/design-decisions.md`: D-122 MAAS-bullet + site-down
   wording, D-107 node-artifacts scoping + DR-claim honesty, D-121 Option-C disk record (the
   `optc-calc.py` "reproducible" reference is to an ABSENT script -- fix or commit the calc; whole-host
   math lives in `scripts/dc-dc-whole-host-budget.py`), D-115 status-line vs ledger-scan, SEC-011
   reconsider (downgraded to OBSERVATION).

### C6. `vr1-dc1` -- HELD (do not attempt)
No ruled dc1 transit/rack addressing (the `172.31.0.0/24` transit supernet is dc0-scoped), and no
`vr1_dc1_rack_*` variables or dc1 substrate exist. dc1 needs its own transit-prefix ruling + variable
+ substrate creation. Separate decision; NOT part of the DC0 build.

---

## BOOKEND -- vcloud HOST patch + reboot (run BEFORE section D)

Closes the last bookend item: patch the vcloud host and reboot onto kernel **6.8.0-136** (the guests
already run -136; the host lags at -134). Measured 2026-07-17: `-136` is PENDING in apt but NOT installed,
no `reboot-required` flag -- so a bare reboot gains nothing; the kernel must be `apt`-installed first.
Pending set is `linux-*-generic` (-134->-136) + `libntfs-3g89t64 ntfs-3g linux-libc-dev linux-tools-common
tar ubuntu-pro-client{,-l10n}` -- **no qemu/libvirt**, so no hypervisor upgrade to complicate guest bring-up.

Preconditions / rationale:
- **Remote Control attached + all decisions done FIRST** -- Step 4 (`sudo reboot`) ENDS the running session.
- `voffice1` + `office1-opnsense` are BOTH `autostart=enable` (set 2026-07-17, edge after its verified cold
  boot) -- so the host reboot self-recovers them. LXD `boot.autostart=true` on netbox+tailscale inside voffice1.
- `libvirt-guests` is enabled+active (default `ON_SHUTDOWN=suspend` = managedsave). Step 3 explicitly
  `virsh shutdown`s the guests first so the reboot exercises the verified **cold-boot autostart** path
  (not managedsave-restore). Ordered to patch with VMs UP; quiesce only for the reboot.

```bash
# Run ON vcloud. Steps 2-4 are gated/sudo; STEP 4 ENDS THIS SESSION.

# STEP 0  preflight / baseline (read-only)
uptime; uname -r; virsh list --all; free -g

# STEP 1  refresh lists + confirm the plan (read-only)
sudo apt update
apt list --upgradable 2>/dev/null            # expect linux-*-generic -134 -> -136 + ~7 pkgs
sudo apt-get -s full-upgrade | tail -20      # dry-run: review before committing

# STEP 2  install upgrades (guests stay UP; no reboot yet)
sudo apt full-upgrade -y
dpkg -l 'linux-image-6.8.0-136-generic' | grep '^ii' && echo "kernel -136 INSTALLED (pending reboot)"
cat /var/run/reboot-required 2>/dev/null

# STEP 3  gracefully quiesce guests (cold-boot path)
virsh shutdown voffice1                        # cascades: LXD stops netbox + tailscale cleanly
virsh shutdown office1-opnsense
for d in voffice1 office1-opnsense; do virsh domstate $d; done   # wait until BOTH "shut off"

# STEP 4  reboot the host  (THIS ENDS THE SESSION)
sudo reboot
```

Post-reboot verify (this feeds section D step 1; autostart should have restored everything):
```bash
uname -r                                        # expect 6.8.0-136-generic
virsh list --all                                # voffice1 + office1-opnsense running (autostart)
ssh voffice1 'ping -c2 10.10.0.1; ping -c2 -I 10.10.0.20 1.1.1.1'          # edge LAN + egress
ssh office1-netbox 'curl -s -o /dev/null -w "netbox=%{http_code}\n" http://localhost:8000/'
ssh office1-tailscale 'tailscale status | head -1'
```
After this lands the host is on -136 (matching the guests) and the bookend is CLOSED -> proceed to section D.

---

## D. Fresh-session bootstrap (start the deployment cleanly)

1. `git pull` ; `bash scripts/repo-lint.sh` (0 fail; 1 legacy WARN) ; `bash scripts/run-tests-all.sh`.
2. Read `docs/session-ledger.md` + `bash scripts/ledger-scan.sh` -- reconcile in-flight state.
3. Read THIS doc + `runbooks/dc-dc-phase2-tofu-dc-substrate.md` (Model B reshape section, ~line 144).
4. **Start logging (cloud-mutating session):** `bash scripts/run-logged.sh dc0-deploy`.
5. Settle **C1.1** (the per-env-key ruling) first.
6. Re-run the outer plan; confirm 5-add/(0-or-2)-change/6-destroy-empty (autostart in-place collapses to 0 post-bookend); then the **gated outer apply**.
7. Proceed A -> B -> C -> D -> E, each mutation individually approved; review after each step.

## E. Gated deploy sequence (Model B, from the runbook)

| Step | Action | Type |
|---|---|---|
| A | outer `tofu apply` -- `vvr1-dc0` (416G/108) + `vr1_dc0_uplink`; relocate 6 empty planes | MUTATION (gated) |
| B | bootstrap `vvr1-dc0`: `site-headend-install.sh --role rack --host-nodes` + SEC-010; prep OPNsense base there | MUTATION (gated) |
| C | inner `tofu apply` (`vr1-dc0-substrate`, qemu+ssh): 9 node VMs + 6 planes + OPNsense edge INSIDE `vvr1-dc0` | MUTATION (gated) |
| D | `maas-vm-host` register `vvr1-dc0` inner virsh -> Office1 region; MAAS discovers nodes | MUTATION (gated, needs C2.3) |
| E | netem on the vcloud mesh bridges (outer) | MUTATION (gated, needs C2.4) |
| GATE | D-125 foreign-MAC egress isolation test; depth-4 boot; SEC-010 packet-drop | verify |

EXIT -> Stage 4 (per-DC), once dc1 clears its own gate.