diff --git a/creds-manifests/vr1-dc0.manifest b/creds-manifests/vr1-dc0.manifest index 8161e60..f8f8563 100644 --- a/creds-manifests/vr1-dc0.manifest +++ b/creds-manifests/vr1-dc0.manifest @@ -7,9 +7,9 @@ # = ":" -- MINTED ON A SITE VM; a working copy MUST be pulled into # the folder at mint time. # -# STATUS: the ~/vr1-dc0-creds/ folder was pre-created (0700) as the landing target; the -# access keypair below is MINTED AT DC DEPLOY (generated on vcloud, pubkey -> vvr1-dc0 -# cloud-init) per D-126 per-env-key isolation (a DEDICATED dc0 key, NOT office1's). +# STATUS: the access keypair below is MINTED (2026-07-16, on vcloud) -- generated EARLY (not at +# deploy) because vvr1-dc0's cloud-init reads the pubkey with file() at plan time (D-126 per-env-key +# option a: a DEDICATED dc0 key, NOT office1's). `creds-audit vr1-dc0` -> CLEAN. # vr1-dc0*.env files (e.g. TF_VAR_* / MAAS) are ADDED to this manifest as the dc0 deploy # design determines them -- not invented here (hard rule 2). diff --git a/docs/dc0-deploy-readiness.md b/docs/dc0-deploy-readiness.md new file mode 100644 index 0000000..e6583b9 --- /dev/null +++ b/docs/dc0-deploy-readiness.md @@ -0,0 +1,115 @@ +# DC0 (vr1-dc0) deployment readiness + prep checklist + +Prep/handoff for the Stage-3 Phase-2 DC0 build (`runbooks/dc-dc-phase2-tofu-dc-substrate.md`, +Model B two-root). Written 2026-07-16 at the outer-apply gate. A FRESH session starts the +deployment from here. **Nothing has been applied** -- the outer `tofu apply` is the first mutation +and is operator-gated. `vr1-dc1` is HELD (no ruled addressing; see below), DC0 only. + +--- + +## A. Readiness ESTABLISHED (done, verified this session) + +- **Durable agent access (D-126)** -- `ssh voffice1` / `office1-netbox` / `office1-tailscale` / + `office1-opnsense` all EXERCISED live (ProxyJump via `~/.ssh/config.d/vr1-sites`, one-key office1 + env `~/vr1-office1-creds/office1_svc_ed25519`). Committed+pushed (`bc2be83`). +- **DC creds targets** -- `~/vr1-dc0-creds/` + `~/vr1-dc1-creds/` (0700, empty); + `creds-manifests/vr1-dc{0,1}.manifest` declare each DC's dedicated per-env key. +- **dc0 rack tfvars** -- `opentofu/d124-rack.auto.tfvars` (LOCAL/gitignored) holds the four + D-124-ruled `vr1_dc0_rack_*` values (`10.12.8.2`, `172.31.0.2`, /30, `172.31.0.1`). +- **Stage-1 gate** -- all six `vr1-dc0` planes + 3 mesh links + `vr1-dc0-pool`/`office1-pool` active. +- **`opentofu-validate`** -- root + 12 modules clean (both roots `tofu validate`). +- **Blocking design rulings LANDED** -- D-121/122/123/124 ADOPTED; R-1 Model B, R-2 vault=v-a + MySQL (Raft infeasible on 1.8 charm), R-3 4 storage/DC=9 nodes. Whole-host fit 870/1024 GiB. + +## B. The outer-root plan -- VERIFIED SAFE (this is the apply gate) + +`tofu -chdir=opentofu plan` => **5 add / 0 change / 6 destroy**, decomposed and verified: +- **5 create:** `vvr1_dc0` containment VM (**425984 MiB = 416 GiB / 108 vCPU** -- full-fleet D-121/R-3 + sizing, expose_nested_virt) + disk/seed/cloudinit; `vr1_dc0_uplink` (D-125 simulated-ISP NAT). +- **6 destroy:** the six `vr1-dc0` plane networks -- INTENDED Model B relocation from vcloud (outer) + to inside `vvr1-dc0` (inner root recreates them, `vr1-dc0-substrate/main.tf:28`). All six are + EMPTY (0 leases, 0 attached domains). No live/consumed infra touched. +- **RAM gate:** host 980 GiB available >= 416 GiB needed (only voffice1 32G + opnsense 2G committed). + +> A fresh session MUST re-run the plan and re-confirm 5-add/0-change/6-destroy-empty before apply +> (live state can drift; verify, don't assume). + +--- + +## C. REMAINING PREP -- before / during the deploy + +### C1. Pre-apply (settle BEFORE the outer `tofu apply`) +1. **[RESOLVED 2026-07-16] Per-env key for `vvr1-dc0` = option (a) dedicated dc0 key (operator ruling).** + DONE: minted `~/vr1-dc0-creds/vr1-dc0_svc_ed25519` (600/644, `creds-audit vr1-dc0` CLEAN); added + `variable "vr1_dc0_ssh_pubkey_path"`; `vvr1-dc0` cloud-init now installs the dc0 pubkey (voffice1 + keeps office1's key); tfvar set. Re-validated + re-planned -> outer plan UNCHANGED (5/0/6, create-only + content). **Remaining for the inner apply:** the inner root's `qemu+ssh` to `vvr1-dc0` must use the dc0 + PRIVATE key -- wire an ssh alias/`-i ~/vr1-dc0-creds/vr1-dc0_svc_ed25519` AFTER the outer apply boots + `vvr1-dc0` (its transit IP is knowable then; D-126 DC ssh-config rows are deferred to that point). +2. **[ENV] `bunzip2`** -- required by `scripts/opnsense-prep-image.sh`. Under Model B that prep runs + ON `vvr1-dc0` (bootstrap step B), so install it there (apt) post-boot; NOT needed for the outer + apply. (Absent on vcloud today -- fine unless any prep is run vcloud-side.) + +### C2. Mid-deploy prerequisites (prep now, needed at a specific gate) +3. **[SECRET/REACH] MAAS reachability** for Step 9 (`maas-vm-host` register, Model B step D): + `maas_api_url`/`maas_api_key` are deliberately unset (empirically revealed at Step 8/9); + `maas_api_key` is a SECRET via `TF_VAR_maas_api_key`. Confirm the Office1 MAAS region is reachable + and export the key before Step 9. Steps A-C (outer apply -> bootstrap -> inner node/plane/edge) do + NOT need MAAS. +4. **[RULED 2026-07-16] netem parameters = PLACEHOLDER for the rehearsal (operator ruling).** Exercise + the `netem-link` mechanism with a placeholder value at Step 11; the "final" latency/loss ruling + (D-100 gap #11) is DEFERRED. Non-blocking for the outer apply. + +### C3. Deploy-time GATES (exercised in-sequence during the deploy, per the runbook) +- Outer: `vvr1-dc0` boots + sized + nested-KVM on (the rack IS `vvr1-dc0`). +- Bootstrap (B): `site-headend-install.sh --role rack --host-nodes --check` PASS incl. SEC-010 + (transit FORWARD-drop, DC-LOCAL); prep OPNsense base ON `vvr1-dc0`. +- Inner (C): node VMs boot at **depth-4** nested virt (D-114/D-123 boot gate -- flagged risk, + accepted); boot-measure `WAN_IF`/`LAN_IF` on the real OPNsense; MAAS zone/pool + `power_address`. +- D-125 egress: foreign-MAC-across-nested-bridge isolation test (Step 12) -- throwaway probe. +- MAAS (D): discovers the inner node VMs (needs C2.3). + +### C4. IPAM record hygiene (parallel, NON-blocking -- now runnable via `ssh office1-netbox`) +5. **D-124 NetBox import** into office1-netbox (transit role + `172.31.0.0/24` carve + rack IP): + run ON the netbox host (token local at `/root/netbox-secrets/api.token`) -- + `python3 netbox/dc-rack-mgmt-import.py --transit-cidr 172.31.0.0/30 --rack-ip 10.12.8.2` (dry-run; + add `--commit`). tofu plan does NOT need this (no netbox provider); it is IPAM correctness. +6. **Edge-WAN import** (`172.30.2.0/24`, `.3.0/24`) via `netbox/dc-edge-wan-import.py` (after + `d115-office-carve.py` seeds Office+Edge roles). + +### C5. Record-hygiene sweep (NON-blocking; queued from the Stage-3 adversarial review) +7. Sweep the queued amendments into `docs/design-decisions.md`: D-122 MAAS-bullet + site-down + wording, D-107 node-artifacts scoping + DR-claim honesty, D-121 Option-C disk record (the + `optc-calc.py` "reproducible" reference is to an ABSENT script -- fix or commit the calc; whole-host + math lives in `scripts/dc-dc-whole-host-budget.py`), D-115 status-line vs ledger-scan, SEC-011 + reconsider (downgraded to OBSERVATION). + +### C6. `vr1-dc1` -- HELD (do not attempt) +No ruled dc1 transit/rack addressing (the `172.31.0.0/24` transit supernet is dc0-scoped), and no +`vr1_dc1_rack_*` variables or dc1 substrate exist. dc1 needs its own transit-prefix ruling + variable ++ substrate creation. Separate decision; NOT part of the DC0 build. + +--- + +## D. Fresh-session bootstrap (start the deployment cleanly) + +1. `git pull` ; `bash scripts/repo-lint.sh` (0 fail; 1 legacy WARN) ; `bash scripts/run-tests-all.sh`. +2. Read `docs/session-ledger.md` + `bash scripts/ledger-scan.sh` -- reconcile in-flight state. +3. Read THIS doc + `runbooks/dc-dc-phase2-tofu-dc-substrate.md` (Model B reshape section, ~line 144). +4. **Start logging (cloud-mutating session):** `bash scripts/run-logged.sh dc0-deploy`. +5. Settle **C1.1** (the per-env-key ruling) first. +6. Re-run the outer plan; confirm 5-add/0-change/6-destroy-empty; then the **gated outer apply**. +7. Proceed A -> B -> C -> D -> E, each mutation individually approved; review after each step. + +## E. Gated deploy sequence (Model B, from the runbook) + +| Step | Action | Type | +|---|---|---| +| A | outer `tofu apply` -- `vvr1-dc0` (416G/108) + `vr1_dc0_uplink`; relocate 6 empty planes | MUTATION (gated) | +| B | bootstrap `vvr1-dc0`: `site-headend-install.sh --role rack --host-nodes` + SEC-010; prep OPNsense base there | MUTATION (gated) | +| C | inner `tofu apply` (`vr1-dc0-substrate`, qemu+ssh): 9 node VMs + 6 planes + OPNsense edge INSIDE `vvr1-dc0` | MUTATION (gated) | +| D | `maas-vm-host` register `vvr1-dc0` inner virsh -> Office1 region; MAAS discovers nodes | MUTATION (gated, needs C2.3) | +| E | netem on the vcloud mesh bridges (outer) | MUTATION (gated, needs C2.4) | +| GATE | D-125 foreign-MAC egress isolation test; depth-4 boot; SEC-010 packet-drop | verify | + +EXIT -> Stage 4 (per-DC), once dc1 clears its own gate. diff --git a/docs/design-decisions.md b/docs/design-decisions.md index 3b0e553..18db469 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -3857,7 +3857,17 @@ (pubkey wired into the DC's cloud-init), tracked in `creds-manifests/.manifest`, not generated speculatively before the DC exists. Tool mechanism: `site-ssh-config.sh` installs each DC env as its own `~/.ssh/config.d/vr1-` fragment via a per-env `--key` (the `--env` selector is added when the first DC -row lands with MEASURED IPs). Recorded now; deferred to DC deploy. +row lands with MEASURED IPs). + +**Applies to the CONTAINMENT VM too (operator ruling 2026-07-16, option a).** Asked whether per-env-key +covers `vvr1-dc0` itself (its cloud-init had installed the OFFICE1 pubkey, and the inner root's `qemu+ssh` +authenticated with it), the operator ruled **(a) dedicated dc0 key** over (b) office1-key-for-the-container +(zero-code). IMPLEMENTED: minted `~/vr1-dc0-creds/vr1-dc0_svc_ed25519` (600/644; `creds-audit vr1-dc0` +CLEAN); added `variable "vr1_dc0_ssh_pubkey_path"`; `vvr1-dc0`'s cloud-init `ssh_authorized_keys` now reads +`file(var.vr1_dc0_ssh_pubkey_path)` (main.tf, the vvr1_dc0 block only -- voffice1 keeps office1's key); +set the tfvar in `d124-rack.auto.tfvars`. The inner root's `qemu+ssh` to `vvr1-dc0` authenticates with the +dc0 PRIVATE half (jumphost-local, never read). Re-validated + re-planned: outer plan unchanged at 5 add / +0 change / 6 destroy (the cloud-init edit is create-only content, no new destroys/replaces). **Related:** D-107 (Tailscale front door -- the workstation path; D-126 is the vcloud-local path), D-123 (Model B containment VMs -- what the DC `-J` form jumps through), D-124 (transit addressing the DC forwards diff --git a/docs/session-ledger.md b/docs/session-ledger.md index dc657e1..1ed3ca4 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -139,6 +139,18 @@ `office1-opnsense` all return live from this session -- so the agent now HAS durable shell into Office1 (load-bearing assumption confirmed: key read from config `IdentityFile`, no creds path in command, passes the guard/classifier). The systemd FORWARD layer stays reviewed-but-unexercised (not installed). +**DC0 DEPLOY READINESS (2026-07-16) -- at the outer-apply gate; see `docs/dc0-deploy-readiness.md`.** +Stage-3 Phase-2 DC0 is EXECUTABLE (D-121..125 ADOPTED, R-1/R-2/R-3 ruled, tfvars in, opentofu-validate +clean). Outer-root `tofu plan` RUN + VERIFIED SAFE: 5 add (`vvr1-dc0` 416GiB/108vcpu + D-125 uplink) / +0 change / 6 destroy (the six EMPTY vr1-dc0 planes relocating to the inner root -- verified 0 leases/0 +attached). RAM gate: 980 GiB avail >= 416. **Nothing applied.** Next = the gated outer `tofu apply`. +Pre-apply per-env-key DECISION now RULED + IMPLEMENTED (operator 2026-07-16, option a): `vvr1-dc0` +gets its OWN key -- minted `~/vr1-dc0-creds/vr1-dc0_svc_ed25519` (creds-audit CLEAN), added +`vr1_dc0_ssh_pubkey_path` var, `vvr1-dc0` cloud-init reads it (voffice1 keeps office1's key), tfvar set; +re-plan UNCHANGED (5/0/6). netem RULED = placeholder for the rehearsal. Full remaining-prep checklist + +fresh-session bootstrap + gated A-E sequence in `docs/dc0-deploy-readiness.md` (the entry point). Inner +apply must use the dc0 private key (wire post-outer-apply). Gauntlet ALL GREEN (65). Nothing applied. + Original build note: this session had NO ssh key to voffice1 (none in `~/.ssh`, no agent; key is operator-held in `~/vr1-office1-creds/`, correctly un-enumerable), so the operator supplies `--key` and runs `install` + first-connect + the one-time diff --git a/opentofu/main.tf b/opentofu/main.tf index 4170da7..dab5976 100644 --- a/opentofu/main.tf +++ b/opentofu/main.tf @@ -389,7 +389,9 @@ shell: /bin/bash sudo: "ALL=(ALL) NOPASSWD:ALL" ssh_authorized_keys: - - ${trimspace(file(var.office1_ssh_pubkey_path))} + # D-126 per-env-key (ruling 2026-07-16, option a): vvr1-dc0 authorizes the DEDICATED + # dc0 key, NOT office1's -- per-env blast-radius isolation. Inner root's qemu+ssh matches. + - ${trimspace(file(var.vr1_dc0_ssh_pubkey_path))} package_update: true packages: - qemu-guest-agent diff --git a/opentofu/variables.tf b/opentofu/variables.tf index e688393..de04094 100644 --- a/opentofu/variables.tf +++ b/opentofu/variables.tf @@ -82,6 +82,17 @@ type = string } +variable "vr1_dc0_ssh_pubkey_path" { + description = <<-EOT + Path to the PUBLIC half of the DEDICATED vr1-dc0 service SSH key (D-126 per-env-key + ruling 2026-07-16, option a: the containment VM gets its OWN key, NOT office1's). + Injected into vvr1-dc0's cloud-init; the inner root's qemu+ssh authenticates with the + matching private half (jumphost-local at ~/vr1-dc0-creds/, never read). Read with file() + at plan time -- the .pub must exist. Per-env blast-radius isolation from Office1. + EOT + type = string +} + variable "voffice1_vcpu" { description = "voffice1 vCPU count. Hosts MAAS-region + the LXD VM host and the service VMs MAAS composes into it (D-114)." type = number