# DC-DC Phase 2 -- OpenTofu builds each DC substrate (Stage 3)

From Office1, script each DC's "physical" layer into existence: the node-VM
libvirt domains MAAS will discover and deploy, the DC's plane networks, and
its OPNsense edge -- then register the DC's libvirt host to that DC's own
MAAS rack controller as a virsh VM-host. This runbook builds **`vr1-dc0`**
(VR1's FIRST DC -- the build target this pass). **`vr1-dc1`** (VR1's SECOND
DC) is explicitly gated (see the callout below) and is NOT part of this
session's run until that gate clears -- do not duplicate these steps for
`vr1-dc1` with invented values in the meantime.

!!! **NAMING -- this runbook was written pre-D-119 and has since been
    reconciled to the region-qualified namespace.** Its original prose spoke
    the bare `dcN` namespace (`dc1` = the build target, `dc2` = the second
    DC). **D-119 REJECTS bare `dcN` for VR1 datacenters** -- the token `dc0`
    already means VR0's live DC in `scripts/lib-net.sh`, so an unqualified
    `dcN` is a latent off-by-one. The mapping applied throughout this file is:
    runbook "DC1" -> **`vr1-dc0`** (already live in `opentofu/main.tf` as the
    `vr1_dc0_*` modules), runbook "DC2" -> **`vr1-dc1`** (still gated). All
    object names, module labels, plan files, and network/pool names below are
    region-qualified to match `main.tf`.
    **HAZARD:** following ANY residual bare-`dcN` literal in a copy of this
    runbook (or in muscle memory) literally reintroduces the exact `dc0`/`dc1`
    cross-cloud collision D-119 was written to delete -- region-qualify it or
    stop. (D-119.)

**Governing docs:** `docs/dc-dc-buildout-design.md` Section 4 "Phase 2" (the
goal/build/gate this runbook satisfies), Section 9 (shim register -- the
node-VM creation module has **NO Roosevelt analog**, see the callout below),
and Section 6 (WAN simulation / netem, same-metro lean).
`docs/dc-dc-deployment-workflow.md` Stage 3 row and the Tooling gap register
(items 1, 4, 11 govern this stage directly). `opentofu/README.md` and
`opentofu/templates/README.md` for exact module/token status -- read both
before touching any `.tf` file or the OPNsense template.

Decisions this runbook owns: **D-103** (the OpenTofu/MAAS/Juju lifecycle seam
-- OpenTofu creates, MAAS discovers/commissions, never composes), **D-100**
(the dark-fiber mesh + per-site OPNsense edge, and the netem WAN-simulation
mechanism riding on it).

---

!!! **Depends on Stage 2 (Office1 headend standup).** This stage's MAAS
    registration step needs a REACHABLE MAAS region controller (Stage 2's
    gate) and Stage 1's OpenTofu-reaches-vcloud-libvirt gate already closed.
    If Stage 2 has not completed when this session runs, STOP at Step 9 (MAAS
    registration) -- everything through Step 8 (planes/pool/edge/node-VM
    creation at the libvirt layer) does not itself require MAAS and may
    proceed, but do not fabricate a `maas_api_url`/`maas_api_key` pair to push
    past that dependency. Confirm Stage 2's actual state in
    `docs/dc-dc-deployment-workflow.md` before starting.

!!! **`vr1-dc1` is explicitly gated -- by SEQUENCING, not by a missing
    literal.** The earlier rationale ("NetBox has not assigned the supernet")
    is STALE: **D-115 ALREADY assigned it** -- `vr1-dc1` gets `10.12.64.0/19`
    (moved inside the Cloud `/16`, colliding with nothing `vr1-dc0` occupies), and
    that carve is imported to `office1-netbox`. So the blocker is no longer a
    missing address; `vr1-dc1` stays out of scope THIS pass purely because we
    build one DC at a time and `vr1-dc0` is the target. `opentofu/main.tf`'s
    `vr1_dc1_planes` block is still commented out (its `vr1_dc1_storage` and
    the three mesh legs ARE wired, being address-independent). Do NOT
    uncomment the planes block, do NOT add `vr1-dc1` build steps, and do NOT
    run this runbook's steps against `vr1-dc1` this pass. Re-run the whole
    runbook for `vr1-dc1` when it is sequenced. This is a hard gate, re-stated
    in the GATE section.
    NB: `main.tf`'s commented `vr1_dc1_planes` block still carries the OLD
    "wait for NetBox to assign D-101's supernet" comment -- that comment is
    now stale per D-115; flagged as a divergence to fix under the `main.tf`
    owner, out of scope for this doc-only pass.

!!! **Node-VM creation has NO Roosevelt analog (Section 9 shim register).**
    `modules/node-vm` (the blank-disk, PXE-boot libvirt domain this stage
    calls) is a virtual-only stand-in for what, in Roosevelt, is physical
    racking plus BMC enlistment -- MAAS enlists hardware that already exists
    there. Do not treat this module, or the act of Terraform-creating "node
    hardware," as reusable production IaC. Everything from MAAS enlistment
    onward (commission/deploy, Juju bundle) IS Roosevelt-transferable; this
    one step is not.

!!! **Node sizing is now RULED (D-121, Option C, 2026-07-15).**
    `modules/node-vm`'s `memory_mib`/`vcpu`/`disk_size_bytes` still have no
    module defaults by design, but the VALUES to pass are no longer open:
    D-121 ruled the per-DC layout as **3 control + 2 compute + 3 storage** (8
    node VMs) at the sizes wired into Step 6. This was the exact decision that
    used to block Step 6; it is now UNBLOCKED. Still open (one sub-ruling
    D-121 notes): the vault-HA backend fork -- but that is a Stage-5 bundle
    concern, NOT a Step-6 blocker (node layout is fully ruled). Do not invent
    any OTHER node spec; use D-121's numbers exactly.

!!! **netem parameters are still an unruled D-100 sub-item (gap #11).** This
    stage's own gate text says "netem parameters applied and measured," but
    the exact latency/jitter/loss/rate values do not exist as a ruled
    decision -- only the qualitative buildout-design Section 6 lean exists:
    "same-metro dark fiber... low single-digit ms, jitter-capable, jumbo-
    capable." Step 11 below applies ONLY that qualitative lean as an
    explicitly-labeled PLACEHOLDER profile (recorded as such, with a citation
    to Section 6), pending a firmer numeric ruling. Do not write a specific
    number (e.g. "delay 3ms") into `netem_args` and represent it as measured
    or ruled -- if you pick a number to instantiate the mechanism for real,
    log it as exactly what it is: a placeholder pending D-100's still-open
    sub-item, not a final value.

!!! **OPNsense prep tools.** `scripts/opnsense-prep-image.sh` requires
    `bunzip2` + `qemu-img` on PATH. **The ISO9660 builder requirement is GONE
    (2026-07-13):** the config-ISO builder (`opnsense-build-config-iso.sh`,
    formerly under scripts/) is DELETED (D-112 --
    the Importer can never fire on a nano image, so the ISO was never read;
    D-113(a2) -- config is done over the REST API). `genisoimage`/`xorriso` are
    no longer needed and are no longer installed by
    `scripts/prereqs/install-image-tools.sh`. Step 2 checks the remaining tools.

---

## Sequence (`vr1-dc0`; `vr1-dc1` remains gated -- see callout above)

Honest achievable scope for a first run is **Steps 1-8** (libvirt-layer
objects + the edge). Step 9 onward is gated on `vr1-dc0`'s own MAAS rack
controller existing -- a Stage-4 gap (per-DC rack-controller source is not yet
stood up); do not treat Steps 9-12 as runnable until it is.

```
1.  Pre-flight: confirm Stage 1 + Stage 2 gates closed       (read-only)
2.  Confirm OPNsense prep-tool prereqs on the vcloud host    (read-only)
3.  Confirm/record decisions this stage depends on (node
    sizing = D-121 RESOLVED; edge/WAN/LAN = D-122 RESOLVED;
    MAAS zone/pool; netem parameters) -- STOP on any open     (read-only)
4.  Prepare vr1-dc0's OPNsense base image (prep only; edge
    config is done over the REST API -- D-113(a2))            [MUTATION: host-local files, gated]
5.  Wire modules/opnsense-edge for vr1-dc0 into main.tf       [repo change, gated]
    (prereq: build the vr1-dc0-wan uplink, office1-wan shape)
6.  Wire the 8 modules/node-vm calls for vr1-dc0's node VMs
    into main.tf (D-121 Option C sizes)                       [repo change, gated]
7.  tofu init / validate / plan -- review before apply         (read-only)
8.  tofu apply -- vr1-dc0 OPNsense edge + node-VM domains      [MUTATION: creates libvirt objects, gated]
9.  Wire + apply modules/maas-vm-host -- register vr1-dc0's
    libvirt host with vr1-dc0's MAAS rack controller           [MUTATION: MAAS + repo change, gated]
10. Verify MAAS discovers the node VMs                          (read-only)
11. Wire + apply modules/netem-link on vr1-dc0's mesh legs,
    using the Section-6 placeholder lean, explicitly flagged     [MUTATION: gated]
12. Verify edge routing + simulated ISP uplink + netem applied   (read-only)
    -> EXIT GATE -> Stage 4 (per-DC, once vr1-dc1 clears its own gate)
```

### D-123 MODEL B RESHAPE (2026-07-16) -- the single-root sequence above is REGROUPED; read this

The operator ruled D-123 = **Model B**: the vr1-dc0 node fleet nests INSIDE the containment VM
`vvr1-dc0` (single-object `virsh destroy vvr1-dc0` = site-down). A libvirt provider cannot target a VM
created in the same apply, so the substrate is now TWO ROOTS with a bootstrap gate between. The Steps
above still describe the WORK; they regroup under the new roots + a new apply order:

- **A. OUTER apply** (`opentofu/`, vcloud) -- creates + sizes `vvr1-dc0` (~416 GiB, `expose_nested_virt=true`,
  single transit leg) + the transit. Steps 1-3 (preflight/decisions) apply here.  `[MUTATION: gated]`
- **B. BOOTSTRAP GATE** on vvr1-dc0 (after it boots): `scripts/site-headend-install.sh --role rack
  --host-nodes --region-url ... --enroll-secret-file ...` -- enroll the rack to the Office1 region AND
  make vvr1-dc0 a nested libvirt host (qemu/libvirt, `kvm nested=1`, inner pool, AppArmor, the SEC-010
  transit FORWARD-drop). Prep the OPNsense base ON vvr1-dc0 (Step 4 -- run `opnsense-prep-image.sh`
  there). GATE: `site-headend-install.sh --role rack --host-nodes --check` must pass (incl. SEC-010).
- **C. INNER apply** (`opentofu/vr1-dc0-substrate/`, provider = qemu+ssh to vvr1-dc0 over the transit,
  R-5) -- the 6 planes + `vr1-dc0-wan` + edge + inner pool + 9 node VMs. Steps 5-6 (wire edge/nodes) are
  ALREADY authored in this inner root; Step 7 (init/validate/plan) + Step 8 (apply) run HERE, against
  vvr1-dc0's libvirt, NOT vcloud.  `[MUTATION: gated]`
- **D. maas-vm-host** (Step 9) -- register vvr1-dc0's OWN (inner, local `qemu:///system`) virsh to the
  Office1 REGION; MAAS discovers the inner node VMs (Step 10). Still DEFERRED (DOCFIX-179).
- **E. netem** (Step 11) -- runs on the vcloud-level mesh bridges, outer. TARGET
  CORRECTED (DOCFIX-197, 2026-07-21; executed reality): the dc0<->dc1 leg
  (measured `virbr5` at apply time), NOT the dc0<->office1 leg this line used to
  name -- that leg carries the LIVE rack<->region transit (MAAS, node DNS, the
  inner root's qemu+ssh) and shaping it would perturb operations. See Step 11.

- **Edge plugin profile (D-129, pinned 2026-07-18).** After the DC edge boots and its REST API is up
  (inner apply C), apply the VR1 edge base profile **{`os-qemu-guest-agent`, `os-iperf`}** via the
  D-113 firmware API (`scripts/opnsense-api.sh POST core/firmware/install/<pkg>`) -- `qemu-guest-agent`
  gives the inner libvirt host clean guest coordination (graceful shutdown/fsfreeze/IP report; pairs
  with D-127 autostart), `iperf` validates the transit path. **Verify the `qemu-guest-agent` daemon is
  running and answers `virsh ... guest-ping` -- not just the package present.** USE the built installer:
  `OPNSENSE_API_HOST=<edge-lan> bash scripts/opnsense-plugins.sh apply vr1-edge` (BUILT 2026-07-18,
  `tests/opnsense-plugins/`) -- it installs **one plugin at a time and polls `core/firmware/upgradestatus`
  to completion** before the next (back-to-back installs collide on the OPNsense firmware lock and the later
  one fails silently -- measured 2026-07-18). **For qga to actually work, `modules/opnsense-edge` must expose
  the `org.qemu.guest_agent.0` virtio-serial channel** (verified ABSENT 2026-07-18; add it as an OPT-IN var
  defaulting OFF so existing edges are unchanged, set true for new DC edges -- see D-129 Implementation prep,
  mechanism-to-verify on dmacvicar 0.9.8). NEW DC edges then boot with the channel, no retrofit.
  NOT the Roosevelt-metal profile (baremetal-only) and NOT `os-node_exporter` (gated on the COS-edge
  scope, D-129).

Node count is R-3 = 3 control + 2 compute + **4 storage = 9/DC** (Step 6's "8" is superseded). The Model
A single-root layout is preserved as a revert (git tag `model-a-fallback` + `docs/archive/model-a-fallback-plan.md`).
Sizing derived by `scripts/dc-dc-whole-host-budget.py` (870/1024 GiB, FIT). SEC-010 DC-LOCAL is enforced
by the bootstrap's transit FORWARD-drop (NOT a global `ip_forward=0`), kept interface-scoped -- **never
globalize it** (D-125 br_netfilter constraint: bridged WAN frames traverse L3 FORWARD). See
`docs/archive/changelogs/changelog-20260716-review-sweep-phaseC.md`.

**D-125 (bridge-in per-DC ISP egress -- closes OBS-3).** Under Model B the inner `vr1-dc0-wan` was a NAT
with no egress (its host `vvr1-dc0` is transit-only). Fix, folded into the roots above:
- **A (OUTER)** ALSO creates the vcloud ISP NAT `vr1-dc0-uplink` on `172.30.2.0/24` (the SAME /24 D-115
  ruled for vr1-dc0's edge WAN -- a ruled literal, NOT a HELD tfvar; no new NetBox object) + a 2nd IP-less
  uplink NIC + the `br-vr1-dc0-wan` netplan bridge on `vvr1-dc0`.
- **B (BOOTSTRAP)** `--check` ALSO verifies `br-vr1-dc0-wan` exists with `uplink` enslaved (fail-open guard).
- **C (INNER)** `vr1-dc0-wan` is now a BRIDGE (`modules/wan-bridge`) onto `br-vr1-dc0-wan`; the OPNsense
  WAN keeps its static `.2` on that same /24 (edge config, D-113 -- unchanged, no re-address).
- **DEPLOY-TIME GATE (Step 12):** before trusting the OPNsense chain, isolation-test egress -- a throwaway
  guest on `br-vr1-dc0-wan` must get a vcloud-ISP address + ping out. FAIL => revert to the D-125 double-NAT
  fallback (not a redesign). See D-125 + `docs/archive/changelogs/changelog-20260716-review-sweep-phaseC2-D125.md`.

---

## Step 1 -- Pre-flight: confirm Stage 1 + Stage 2 gates closed (READ-ONLY)

**CHECK -- Stage 1 (vcloud host prep) closed?**
```bash
virsh net-list --all
virsh pool-list --all
```
Expect: `vr1-dc0`'s six plane networks, the three mesh-link networks
(`vr1-dc0-vr1-dc1`, `vr1-dc0-office1`, `vr1-dc1-office1` -- the region-
qualified names now in `main.tf`), and the `vr1-dc0`/`office1` storage pools,
all `active` -- Stage 1's Step 11 verification. If any are missing, Stage 1 is
not actually done regardless of what `docs/dc-dc-deployment-workflow.md`
says -- stop and reconcile there first.

**CHECK -- Stage 2 (Office1 headend) closed?**
Confirm against `docs/dc-dc-deployment-workflow.md`'s Stage 2 row and, if a
Stage 2 runbook exists by the time you run this, its own exit gate. At
minimum, confirm:
```bash
# from Office1 (or wherever this session's tofu apply will run from)
tofu -chdir=opentofu version
curl -sI "$MAAS_API_URL" 2>/dev/null | head -1   # MAAS_API_URL: this session's real value, not invented
```
Expect a real MAAS region controller reachable at a real URL. If Stage 2 is
not done, everything through Step 8 below (libvirt-layer objects only) may
still proceed since it needs no MAAS reach; Step 9 onward (MAAS registration)
is a hard stop until Stage 2 closes -- do not substitute a placeholder MAAS
endpoint to push past it.

**GATE:** Stage 1 objects present and active; Stage 2's MAAS reachability
either confirmed, or explicitly recorded as NOT YET (in which case this
session's scope for today is Steps 1-8 only).

---

## Step 2 -- Confirm OPNsense prep-tool prerequisites on the vcloud host (READ-ONLY)

**CHECK**
```bash
for bin in bunzip2 qemu-img genisoimage xorriso curl wget; do
  command -v "$bin" >/dev/null 2>&1 && echo "present: $bin" || echo "absent:  $bin"
done
```
Required: `bunzip2` + `qemu-img` (for `scripts/opnsense-prep-image.sh`), and
(The `genisoimage`/`xorriso` requirement is GONE -- the config-ISO path is deleted.)
plus `curl` or `wget` for the image fetch. If any required tool is absent,
install it on the vcloud host (an OS-specific package install, out of scope
to prescribe here) before Step 4 -- do not skip the tool and hand-build the
ISO/qcow2 by some improvised method not covered by this repo's tested
scripts.

**GATE:** all required tools present (or installed and re-checked).

---

## Step 3 -- Confirm/record still-open decisions this stage depends on (READ-ONLY)

Before wiring anything into `main.tf`, walk this list. Two of the items that
used to block this stage are now RULED (D-121, D-122) and are recorded as
RESOLVED below; the genuinely-open ones remain STOPs. Record each plainly
rather than inventing a value to keep moving:

- **Node sizing** (`memory_mib`/`vcpu`/`disk_size_bytes` and count for
  `modules/node-vm`) -- **RESOLVED (D-121, Option C, 2026-07-15).** Per DC: 3
  control @ 16 vCPU / 65536 MiB / 150 GiB; 2 compute @ 12 vCPU / 49152 MiB /
  100 GiB; 3 storage @ 8 vCPU / 24576 MiB / 550 GiB -- 8 node VMs total. Step
  6 wires exactly these. Open sub-ruling D-121 still carries (the vault-HA
  backend fork) does NOT block Step 6 -- it is a Stage-5 bundle concern.
- **Edge sizing / WAN uplink / LAN role** -- **RESOLVED (D-122, 2026-07-15).**
  The DC edge follows the proven Office1 pattern: `memory_mib = 2048`,
  `vcpu = 2`, nano base image (NO `disk_size_bytes`, like `office1_opnsense`).
  The edge is **2-NIC** (baremetal-matched): `wan_network_name` is a
  **dedicated per-site ISP uplink `vr1-dc0-wan`** (`172.30.2.0/24`, D-115) built
  on the `office1-wan` pattern -- NOT a mesh leg (dark fiber is East-West only).
  `lan_network_name` is **provider-public** -- the six planes are routed by the
  fabric (OVN/OpenStack on isolated-L2 segments), so the edge is only the
  external gateway on provider-public (D-100), not an inter-plane router. See
  Step 5. PREREQUISITES: build `vr1-dc0-wan` (as a tofu module) and register
  `172.30.2.0/24` in office1-netbox (not yet loaded).
- **MAAS zone/pool for `vr1-dc0`'s VM-host registration**
  (`modules/maas-vm-host`'s `zone`/`pool` variables) -- both are optional
  (MAAS computes them if left `null`, confirmed safe per `opentofu/README.md`'s
  audit pass), so this is NOT necessarily a blocker; record whichever real
  zone/pool name the operator wants, or confirm the `null` no-op is
  intentional. Affects Step 9.
- **`vr1-dc0`'s real `power_address`** (the virsh URI MAAS uses to reach the
  vcloud host) -- must be measured this session (or carried from Stage 1's
  `libvirt_uri` tfvar, if it is the SAME endpoint -- confirm, do not assume
  identical per `modules/maas-vm-host`'s own variable note). Blocks Step 9.
- **`vr1-dc0`'s MAAS rack controller** -- a Stage-4 gap: the per-DC
  rack-controller source is not yet stood up. This is the reason honest Step-9
  scope is BLOCKED and this pass's real reach is Steps 1-8 (see the GATE
  section). Blocks Steps 9-12.
- **Edge post-boot interface mapping** (formerly the `{{WAN_IF}}`/`{{LAN_IF}}`
  `config.xml` tokens) -- NO LONGER a config-render blocker. There is no
  `config.xml` render (D-112/D-113(a2)); the real `vtnetN` mapping is measured
  AFTER boot over SSH (`ifconfig`), then applied over the REST API. See the
  REPLACEMENT chain in Step 4. Nothing here blocks wiring Step 5.
- **netem parameters** -- still an unruled D-100 sub-item (gap #11); Step 11
  uses the Section-6 qualitative placeholder only, explicitly labeled as
  such. Not a hard blocker for Step 11 (the mechanism can be exercised with a
  labeled placeholder), but IS a blocker for calling this stage's netem gate
  "final" -- see the GATE section.

**GATE:** every item above is either resolved with a real, measured/ruled
value, or explicitly recorded as NOT YET with the corresponding downstream
step named as blocked. Do not proceed past a step whose input is still
"NOT YET" by inventing one.

---

## Step 4 -- Prepare `vr1-dc0`'s OPNsense base image [MUTATION: host-local files, gated]

> ## THE EDGE-CONFIG PATH (settled 2026-07-13 -- this IS the path)
>
> This step is now **prep the base image only.** Edge configuration is NOT baked into an image
> or seeded from an ISO -- it is done over the **REST API after boot**, the path proven end to
> end on the live Office1 edge (2026-07-12/13). The two deleted approaches, and why:
>
> 1. **The config ISO was INERT -- it could never be read.** Per **D-112** (root-caused from
>    upstream source): `opnsense-importer -b` probes for a read-only root. On INSTALLER media the
>    probe fails and the importer scans attached media. On our PRE-INSTALLED NANO image the root
>    is writable AND a factory `/conf/config.xml` already exists, so BOTH conditions hold and it
>    `bootstrap_and_exit 0`s **without enumerating a single device**. The Configuration Importer
>    can NEVER fire on a nano image, by design -- so the ISO builder is DELETED and the module no
>    longer takes a `config_iso_path`.
> 2. **Full-`config.xml` rendering is SUPERSEDED** by **D-113(a2)** (ruled 2026-07-13): edge
>    config is done over the **REST API** (`scripts/opnsense-api.sh`), proven end to end -- read
>    AND write -- against the live Office1 edge on 2026-07-13. Hand-authoring the appliance's
>    GUI-owned XML caused DOCFIX-191 (management lockout), DOCFIX-192 (dead console) and
>    DOCFIX-193 (no DHCP, and an ISC `<dhcpd>` block that would have been inert against a Kea
>    backend). None of those bugs is expressible through the API. The renderer is DELETED.
>
> **There is no `WAN_IF`/`LAN_IF` chicken-and-egg.** It only ever existed because we tried to seed
> a full config before first boot. The D-112(c) bootstrap (boot on factory defaults -> reach the
> console -> enable SSH + install the key -> configure over the network) measures the real
> `vtnetN` mapping *after* boot, where it is knowable -- nothing to guess before Step 8.
>
> **What `vr1-dc0`'s edge does (the proven Office1 path):** prep the image (this step) -> Step 5
> wires the domain -> Step 8 boots it -> D-112(c) console bootstrap -> configure over SSH + the
> REST API (the REPLACEMENT chain below). See
> `docs/archive/changelogs/changelog-20260712-office1-opnsense-edge-build.md`,
> `docs/archive/changelogs/changelog-20260713-opnsense-api-proven.md`, and
> `docs/archive/changelogs/changelog-20260713-opnsense-api-write-proven.md`.

This step produces one plain file on the vcloud host filesystem -- the prepped
base image -- no libvirt or MAAS object is created yet. Run from the vcloud
host (or wherever this script executes with reach to write there).

**MUTATION -- prep the base image (needs Step 2's tools)**
```bash
bash scripts/opnsense-prep-image.sh   # see the script's own header for its exact args/output path
```
**Expect:** a decompressed, qcow2-converted, resized OPNsense nano image at a
real output path -- record that path, it feeds Step 5's `base_volume_path`.
That single path is this step's ONLY output. There is no config.xml render and
no config-ISO build -- both are deleted (D-112 / D-113(a2), see the box above).

---

### The REPLACEMENT chain (proven on Office1, 2026-07-12/13)

No `config.xml` anywhere. After Step 5 wires the domain and Step 8 boots it on FACTORY DEFAULTS:

1. **Console bootstrap (D-112(c))** -- reach the serial console, enable SSH, install the service
   public key. This is the only step that needs the console; everything after it is scripted.

   **Payload requirements (DOCFIX-199 -- every one MEASURED, dc0 2026-07-20 lessons i-v +
   dc1 2026-07-23 confirmation; skip one and the bootstrap fails exactly the way it already
   failed once):**
   - Ship the PHP payload as **base64 chunks <= 160 chars** reassembled on the edge
     (`openssl base64 -d`) -- the serial console WRAPS long lines (lesson i).
   - The payload MUST `require_once("config.inc")` **AND `require_once("util.inc")`** --
     `write_config()` calls `shell_safe()`, which lives in util.inc (measured:
     `grep -l shell_safe /usr/local/etc/inc/*.inc`; lesson iv). A payload without it dies at
     `Call to undefined function shell_safe()` BEFORE writing. Any surviving dc0 `.b64`
     artifact on a rack predates this line and is missing the include -- do not replay it
     verbatim.
   - Writing config.xml does NOT materialize `/root/.ssh/authorized_keys` -- the payload must
     call the vendor's **`local_user_set()`** (auth.inc) after the config write (lesson v; the
     measured reason key auth "mysteriously" failed after a clean write).
   - The edge root shell is **tcsh**: never patch files via nested-quoted one-liners
     (`Unmatched '"'"'`; `2>&1` = "Ambiguous output redirect") -- re-ship the WHOLE payload
     instead (lesson iii; platform-traps carries the shell trap).
   - When driving the rack over a piped `bash -s`, every inner `ssh` takes **`-n`** or it EATS
     the remaining driver stdin (measured on the dc1 mint: retrieve/cleanup lines swallowed).
   - Immediately after the bootstrap's `configctl openssh restart`, first contact can get a
     transient `Connection closed` -- `opnsense-bootstrap-apikey.sh` now waits for sshd and
     retries its first scp (2026-07-23).
2. **Mint the API key** -- no GUI click:
   ```bash
   export OPNSENSE_SSH_KEY=...   # the service private key
   bash scripts/opnsense-bootstrap-apikey.sh <edge-lan-ip> <creds-out-file>
   ```
3. **Configure over REST** -- interfaces, DHCP, firewall:
   ```bash
   export OPNSENSE_API_HOST=<edge-lan-ip> OPNSENSE_API_CREDS=<creds-out-file>
   bash scripts/opnsense-api.sh GET  core/firmware/status        # auth smoke test
   bash scripts/opnsense-api.sh POST kea/dhcpv4/set_subnet/<uuid> '<json>'
   bash scripts/opnsense-api.sh POST kea/service/reconfigure '{}'
   ```

**The real `vtnetN` interface mapping is measured AFTER boot** (`ifconfig` over SSH), where it is
knowable -- there is no interface value to guess before the domain exists.

**GATE:** the prepped base image path recorded as a real host-local file path
(it feeds Step 5's `base_volume_path`). Nothing else is produced or pending
here -- no config-ISO path exists to record.

---

## Step 5 -- Wire `modules/opnsense-edge` for `vr1-dc0` into `main.tf` [repo change, gated]

Proceed once Step 4 produced a real `base_volume_path`. The edge SHAPE is
ruled (D-122): it mirrors the already-applied `office1_opnsense` block in
`main.tf` -- `memory_mib = 2048`, `vcpu = 2`, nano image, **no
`disk_size_bytes`** (the disk is a direct copy of the prepped nano; the module
does not accept a size input -- DOCFIX-189), and **no `config_iso_path`** (the
module dropped it -- D-112). There is nothing to invent for the edge specs
themselves; the two values that ARE still per-site (the base image path, and
the two network names) are called out below.

**PREREQUISITE -- build the `vr1-dc0-wan` uplink first.** D-122 gives the DC
its OWN dedicated L3 "ISP" uplink, NOT a mesh leg (dark fiber is East-West
replication only). Build a per-site NAT'd `/24` uplink network `vr1-dc0-wan`
on the **`office1-wan` pattern** (the same mechanical shape `office1-wan` was
created in -- a documented D-103 debt to formalize into a module later; today
it is a literal network name the edge's `wan_network_name` points at). This is
a prerequisite to wiring the block below with a real `wan_network_name`.

**LAN side -- RESOLVED (D-122, operator ruling 2026-07-15): `provider-public`.**
The edge is **2-NIC** (WAN + LAN), matching the baremetal target: the six
planes are **routed by the fabric** (the `dc-planes` segments are isolated L2;
OVN/OpenStack own their L3, not the edge), so the edge is NOT an inter-plane
router -- only the external boundary. Its LAN attaches to **provider-public**
(`10.12.4.0/22`), where the edge is the upstream/external gateway the provider
network exits through (FIP/SNAT egress + GUA injection, D-100 br-ex/provider
model). metal-admin and the other planes reach external via fabric routing, not
a direct edge leg. So `lan_network_name` = the `vr1_dc0_planes` provider-public
output (confirm the exact output name in `modules/dc-planes`). No new
site-internal network module is needed (this SUPERSEDES the earlier "build an
office1-network analog" flag).

**MUTATION -- edit `opentofu/main.tf`**, adding a block shaped like
`office1_opnsense` (main.tf, the applied reference):
```hcl
module "vr1_dc0_opnsense" {
  source           = "./modules/opnsense-edge"
  vm_name          = "vr1-dc0-opnsense"
  memory_mib       = 2048                              # D-122: Office1 pattern
  vcpu             = 2                                  # D-122: Office1 pattern
  pool_name        = module.vr1_dc0_storage.pool_name
  # No disk_size_bytes: nano direct-copy, size from opnsense-prep-image.sh's GROW (DOCFIX-189).
  base_volume_path = "<Step 4's real prepped-image path>"
  lan_network_name = <the vr1_dc0_planes provider-public output -- D-122: edge is the external gateway on provider-public; confirm the output name in modules/dc-planes>
  wan_network_name = "vr1-dc0-wan"                     # D-122: dedicated per-site ISP uplink, office1-wan pattern (build first, promote to a tofu module)
}
```
Before commit: the base image path (Step 4) must be real, `vr1-dc0-wan` must be
built (as a tofu module -- D-122) and `172.30.2.0/24` registered in office1-netbox
(D-115, not yet loaded), and `lan_network_name` bound to the real provider-public
plane output. `memory_mib`/`vcpu`/the nano-no-disk shape and the 2-NIC WAN/LAN
model are ruled (D-122), not invented.

**GATE:** `opentofu/main.tf` diff shows exactly one new `module "vr1_dc0_opnsense"`
block, no `disk_size_bytes` and no `config_iso_path` line, every argument a
real value (with `vr1-dc0-wan` and the site-internal LAN network genuinely
built), nothing else changed.

---

## Step 6 -- Wire `modules/node-vm` calls for `vr1-dc0`'s node VMs into `main.tf` [repo change, gated]

**Node sizing is RULED (D-121, Option C).** Wire **8** `module "vr1_dc0_node_*"`
blocks in three role-sizes:

| Role    | Count | vCPU | `memory_mib` | disk        | Names                              |
|---------|-------|------|--------------|-------------|------------------------------------|
| control | 3     | 16   | 65536        | 150 GiB     | `vr1-dc0-control-01..03`           |
| compute | 2     | 12   | 49152        | 100 GiB     | `vr1-dc0-compute-01..02`           |
| storage | 3     | 8    | 24576        | 550 GiB     | `vr1-dc0-storage-01..03`           |

`disk_size_bytes` is BYTES: express each as GiB x 1024^3 in HCL --
`150 * 1024 * 1024 * 1024`, `100 * 1024 * 1024 * 1024`,
`550 * 1024 * 1024 * 1024` (the module's `disk_size_bytes` expects bytes; the
GROW/qcow2 layer is thin-provisioned so this is a max, not a preallocation).

**Why 3 control (context):** this layout exists to carry the Stage-5 HA
scale-up (D-121). The bundle's **14 currently-single-unit services scale
1 -> 3**: the 11 hacluster+VIP API apps (keystone, glance, neutron-api,
nova-cloud-controller, placement, cinder, openstack-dashboard, octavia,
barbican, magnum, designate) plus `ceph-radosgw`, `rabbitmq-server`, and
`vault`. Those 3-unit control-plane services LIVE on the 3 control nodes --
alongside the services that are ALREADY at 3 and do not change
(`mysql-innodb-cluster`, `ovn-central`, `ceph-mon`). Meanwhile `ceph-osd` = 3
units on the 3 storage nodes and `nova-compute` = 2 units on the 2 compute
nodes. The node-COUNT/sizing here is fully ruled; the still-open D-121
sub-item (the vault-HA backend fork) is a Stage-5 bundle concern and does NOT
gate this step.

**MUTATION -- edit `opentofu/main.tf`**, one block per node VM (control shown;
compute/storage identical shape with their row's vCPU/memory/disk/name):
```hcl
module "vr1_dc0_control_01" {
  source          = "./modules/node-vm"
  vm_name         = "vr1-dc0-control-01"
  memory_mib      = 65536                              # D-121: control
  vcpu            = 16                                  # D-121: control
  pool_name       = module.vr1_dc0_storage.pool_name
  disk_size_bytes = 150 * 1024 * 1024 * 1024           # D-121: control 150 GiB
  network_names   = [ <ordered list of module.vr1_dc0_planes outputs -- confirm which plane goes first for PXE priority, per D-103> ]
}
```
Repeat for `vr1_dc0_control_02`/`03`, `vr1_dc0_compute_01`/`02` (12 vCPU /
49152 MiB / `100 * 1024 * 1024 * 1024`), and `vr1_dc0_storage_01`/`02`/`03`
(8 vCPU / 24576 MiB / `550 * 1024 * 1024 * 1024`). **`network_names` is STILL a
flagged STOP** in every block: D-121 ruled the SIZES, not the plane
attachment/PXE order -- that ordered plane list is an unresolved value (D-103
PXE priority) and must not be invented to make the block parse.

**GATE:** exactly 8 `module "vr1_dc0_node_*"`-shaped blocks (3 control / 2
compute / 3 storage) with D-121's sizes verbatim; each `network_names` either
a genuinely-resolved plane list or explicitly recorded as the remaining STOP,
never a guessed order.

---

## Step 7 -- `tofu init` / `validate` / `plan` (READ-ONLY against providers)

```bash
cd opentofu
tofu fmt -check -recursive -diff .
tofu init -backend=false -input=false
tofu validate
tofu plan -out=phase2-vr1-dc0.tfplan
```
Review the plan line by line: expect creates for `module.vr1_dc0_opnsense`
(the edge VM's boot volume and domain -- NO config-seed volume/cdrom anymore;
that path is deleted, D-112) and each `module.vr1_dc0_*` node (a blank boot
volume + domain per node: 3 control, 2 compute, 3 storage). Confirm nothing
else is planned -- in particular, confirm no `vr1-dc1` resource appears (its
`vr1_dc1_planes` block should still be commented out per the gate above).

**GATE:** plan matches this expectation exactly; no unexpected creates,
updates, or destroys. Do not apply a plan you have not read.

---

## Step 8 -- `tofu apply` -- `vr1-dc0` OPNsense edge + node-VM domains [MUTATION: gated]

```bash
cd opentofu
tofu apply phase2-vr1-dc0.tfplan
```
Confirm this is the exact reviewed plan file from Step 7 (not re-planned)
before running. This is the first live mutation THIS stage performs.

**VERIFY**
```bash
virsh list --all
virsh domblklist vr1-dc0-opnsense
virsh domblklist vr1-dc0-control-01   # per real node name
```
Expect the new domains present (likely `shut off` until first boot/PXE), the
OPNsense domain showing a SINGLE boot disk (a direct copy of the prepped nano
-- no config-ISO cdrom; that path is deleted, D-112), each node VM showing a
single blank boot disk.

**GATE:** all new domains defined as planned; disk attachments match Step
5/6's wiring.

**State file reminder:** re-check `chmod 600 opentofu/terraform.tfstate` /
`git status` after this apply too, same as every stage since Stage 1 Step 10
first created the file (`opentofu/README.md` "State file handling",
DOCFIX-175) -- Step 9 next is the first step in this stage that needs a REAL
(non-placeholder) `TF_VAR_maas_api_key` to actually succeed, so this is the
last checkpoint before a genuine credential is likely to enter state if it
hasn't already.

---

## Step 9 -- Wire + apply `modules/maas-vm-host` -- register `vr1-dc0`'s libvirt host with `vr1-dc0`'s MAAS rack controller [MUTATION: gated]

> **SUPERSEDED (DOCFIX-198, 2026-07-23). Do NOT execute Steps 9-10 as written.**
> The virsh-POD mechanism this step prescribes (`modules/maas-vm-host`) was
> REFUTED BY MEASUREMENT on the dc0 build and the module's root retired:
> snap-confined MAAS cannot reach a local libvirt socket, and a `qemu+ssh` pod
> fails on `modules/node-vm`'s pool+volume disk refs (D-123 AMENDMENT
> 2026-07-20 -- the ruling authority; D-103 amendment likewise). The pod's
> discovery job is done by PXE enlistment; the RULED replacement is
> **per-machine `power_type=virsh` via `scripts/maas-node-power.sh`**
> (MAC-matched, dry-by-default, verified by a real `query-power-state`) --
> the Roosevelt shape (per-node IPMI). Also note the appendix-A trap: a MAAS
> pod delete CASCADES to its linked machine records -- read the pod's machine
> list BEFORE any `vm-host delete`; non-empty = STOP (2026-07-21 incident).
> The as-executed sequence both DCs actually used is the "DC standup
> definition-of-done" section after Step 12. D-123's "PROPOSED, recommend
> Model A" framing below is history (Model B ADOPTED 2026-07-16).

**Hard dependency -- `vr1-dc0`'s MAAS controller.** This step requires
`vr1-dc0`'s own MAAS to exist and be reachable. **D-123 (PROPOSED, recommend
Model A) resolves what that is:** a `vvr1-dc0` **site headend VM** built the
proven Office1 way -- an OpenTofu `cloudinit-vm` block shaped like
`module "voffice1"` + a run of `scripts/site-headend-install.sh` (which already
names `vvr1-dc0/vvr1-dc1` as targets) to install MAAS region+rack + LXD. Under
Model A the node VMs stay vcloud-level and this MAAS DISCOVERS them via the
`maas-vm-host` registration below (power_address = vcloud's virsh). Until Model
A is ruled and `vvr1-dc0` is stood up, this is the honest scope boundary:
**Steps 1-8 are this pass's achievable scope; Steps 9-12 STOP here** --
`vr1-dc0`'s Phase 2 is then INCOMPLETE, not silently skippable. (Also inherits
Stage 2's reachable-MAAS-region dependency.)

**MUTATION -- edit `opentofu/main.tf`**
```hcl
module "vr1_dc0_maas_vm_host" {
  source         = "./modules/maas-vm-host"
  vm_host_name   = "vr1-dc0-vcloud"
  power_address  = "<Step 3's real, measured virsh URI -- confirm same as var.libvirt_uri or a distinct value>"
  zone           = <real zone name, or leave unset for MAAS's computed default -- per Step 3>
  pool           = <real pool name, or leave unset -- per Step 3>
}
```

```bash
cd opentofu
tofu plan -out=phase2-vr1-dc0-maashost.tfplan   # review: exactly one new maas_vm_host resource
tofu apply phase2-vr1-dc0-maashost.tfplan
```

**GATE:** `maas_vm_host` resource created; no other resource touched by this
plan.

---

## Step 10 -- Verify MAAS discovers the node VMs (READ-ONLY)

> **SUPERSEDED with Step 9 (DOCFIX-198):** discovery happens by PXE
> enlistment against the pinned node MACs, not via a VM host; the machines
> appear with MAAS-random hostnames, matched by MAC (see the DoD section
> after Step 12). The `vm-hosts read` check below has no object to read on
> the ruled path.

**CHECK**
```bash
maas admin vm-hosts read | jq -r '.[] | select(.name=="vr1-dc0-vcloud") | {id, name, resources}'
maas admin machines read | jq -r '.[] | select(.hostname|test("^vr1-dc0-")) | "\(.hostname)\t\(.status_name)"'
```
Expect `vr1-dc0`'s node VMs listed as MAAS machines (status likely `New` --
enlistment discovered them; commissioning is Stage 4's job, not this
stage's). If the node VMs do not appear, do not proceed to declare this
stage's MAAS-discovery gate met -- troubleshoot the VM-host registration
(power_address correctness, virsh reachability from the MAAS rack
controller) before moving on.

**GATE:** `vr1-dc0`'s OpenTofu-created node VMs visible in MAAS, associated with
the `vr1-dc0-vcloud` VM host.

---

## Step 11 -- Wire + apply `modules/netem-link` on `vr1-dc0`'s mesh legs, placeholder profile [MUTATION: gated]

Per the netem callout above: this step applies ONLY the buildout design's
qualitative Section 6 lean ("same-metro dark fiber... low single-digit ms,
jitter-capable, jumbo-capable"), explicitly recorded as a PLACEHOLDER, not a
ruled numeric value. If the operator has since ruled a specific numeric
profile, use that instead and cite the ruling; do not use this placeholder
language once a real ruling exists.

**LEG SELECTION CORRECTED (DOCFIX-197, 2026-07-21 -- this is how step E was
actually executed).** This step's original text targeted `vr1-dc0`<->`office1`
as "the only leg with substrate on both ends". Superseded by operations: that
leg now CARRIES the live rack<->region transit (MAAS enlistment/commissioning,
the D-131 node-DNS forwarder path, the inner root's `qemu+ssh`), so shaping it
perturbs a production path. The step-E target is the `vr1-dc0`<->`vr1-dc1`
leg -- zero traffic while `vr1-dc1` is gated, which exercises the netem
MECHANISM (this step's actual goal, per the 2026-07-16 placeholder ruling)
with no operational impact. The netem-tc sudoers fragment
(`scripts/sudoers.d/netem-tc`) grants all three mesh bridges, so extending to
other legs later is config-only -- but shaping a leg that carries live
transit gets its own gated decision first.

**MUTATION -- edit `opentofu/main.tf`** (as executed 2026-07-21; captures
`docs/audit/outer-{plan,apply}-20260721-netem*.txt` + `stepE-netem-20260721.txt`):
```hcl
module "netem_vr1_dc0_vr1_dc1" {
  source      = "./modules/netem-link"
  link_name   = "vr1-dc0-vr1-dc1"
  bridge_name = "<real OS bridge -- re-measure via `virsh net-info mesh-vr1-dc0-vr1-dc1` at apply time; virbrN is a drifting ID>"
  netem_args  = "<PLACEHOLDER pending D-100 sub-item ruling -- record explicitly as provisional, e.g. a low single-digit-ms delay with modest jitter and negligible loss, cited to buildout-design Section 6, NOT presented as a measured or ruled value>"
  # vcloud_host_ssh_target: leave at its empty default = LOCAL mode. The outer
  # root executes ON vcloud (D-128 Plane 1), where the bridges live; the SSH
  # hop exists only for a root that runs remotely (module amendment 2026-07-21).
}
```

```bash
cd opentofu
tofu plan -out=phase2-vr1-dc0-netem.tfplan
tofu apply phase2-vr1-dc0-netem.tfplan
```

**GATE:** the `terraform_data`/`local-exec` resource applies without error;
record the exact `netem_args` string used and its PLACEHOLDER status in the
as-executed log, alongside a note that D-100's exact parameters remain open
(gap #11).

---

## Step 12 -- Verify edge routing + simulated ISP uplink + netem applied (READ-ONLY)

**CHECK -- from the vcloud host**
```bash
sudo tc qdisc show dev <bridge_name>   # per Step 11's real bridge name
```
Expect a `netem` qdisc present with the applied (placeholder) parameters.

**CHECK -- OPNsense edge reachability** (only meaningful once the domain has
been booted and configured over the REST API per Step 4's REPLACEMENT chain --
the `vtnetN` mapping and WAN/LAN addressing are measured/applied AFTER boot)
```bash
virsh domstate vr1-dc0-opnsense
```
If the domain has not yet been through the console + REST-API bootstrap (the
D-112(c) chain), this verification is necessarily partial this session --
record that honestly rather than declaring edge routing verified when it is
not.

**GATE (this step):** netem qdisc present and matching Step 11's recorded
placeholder profile; OPNsense domain running IF it has been through the
post-boot REST-API bootstrap, otherwise recorded as blocked pending that
bootstrap (console -> SSH -> API, per Step 4).

---

## Step 13 -- BACK UP the inner tfstate to the jumphost [MUTATION: gated, RE-RUN AFTER EVERY APPLY]

> **WHY THIS EXISTS (added 2026-07-30).** The inner `terraform.tfstate` is the substrate's
> STATE-OF-RECORD, it lives ONLY on the Office1 headend, and it is gitignored AND untracked --
> proven both, deliberately, because it must never enter the repo. This document said it should
> be "add[ed] to the site backup set"; **measured 2026-07-30, no such backup set existed** --
> `scripts/cloud-snapshot.sh` is a juju-layer capture and mentions octavia, tfstate and
> terraform ZERO times. So the state describing 20 node VMs across two DCs sat in one place on
> one VM with no copy. Lose it and `tofu` no longer knows those resources exist: the next plan
> proposes to CREATE what is already running, and the substrate has to be reconciled by import
> or rebuilt.
>
> **IT IS NOT THE SAME AS THE PKI BACKUP, in one way that matters.** The Octavia CAs are static
> (10-year), so one backup holds indefinitely. **tfstate is DYNAMIC** -- it changes on every
> apply, so a backup is only as good as its `serial`, and restoring a STALE state is actively
> dangerous: `tofu` will believe resources created since that serial do not exist. **Record the
> serial every time, and re-run this step after every apply.**
>
> **CLASSIFICATION, measured rather than assumed.** The OUTER state on the jumphost IS a
> declared credential location -- DOCFIX-175, the Office1 MAAS API key reaches it in PLAINTEXT
> (`creds-matrix.tsv` id `office1-tofu-maas-apikey`). The INNER states are different: a
> name-based inspection found NO `*api_key*`, `*password*`, `*secret*`, `*private_key*`,
> `*token*`, `user_data`, `cloudinit` or `authorized*` keys in either, consistent with the inner
> root talking only to libvirt over `qemu+ssh` (whose key is not in state). **That is strong
> evidence, NOT proof** -- a name-based absence cannot exclude material embedded in a value. The
> backup is therefore registered and stored as if sensitive: uncertainty about content resolves
> toward MORE auditing, and an unaudited directory beside audited ones is how the SEC-022 shadow
> stores happened.

**RUN -- voffice1**
```bash
# RUN: voffice1 -- one gzip per DC, serial RECORDED (a tfstate backup without its serial is ambiguous)
( {
  DC="${DC:?export DC=vr1-dc0|vr1-dc1}"
  # $REPO, never a literal clone name: this repo has been renamed once already (D-110) and
  # repo-lint L9 fails on a hardcoded path. On voffice1 $REPO is the HEADEND clone.
  REPO="${REPO:?REPO not set -- export REPO=<path to the headend clone>}"
  cd "$REPO" || exit 1
  S="opentofu/${DC}-substrate/terraform.tfstate"
  [ -f "$S" ] || { echo "ABORT: no inner state at $S"; exit 1; }
  SER=$(python3 -c "import json,sys;print(json.load(open(sys.argv[1]))['serial'])" "$S")
  umask 077
  gzip -c "$S" > "$HOME/${DC}-substrate.tfstate.gz"
  chmod 600 "$HOME/${DC}-substrate.tfstate.gz"
  echo "serial=$SER  sha256=$(sha256sum < "$HOME/${DC}-substrate.tfstate.gz" | cut -d' ' -f1)"
} )
```

**RUN -- vcloud**
```bash
# RUN: vcloud -- pull, verify against source, then drop the staging copy
( {
  DC="${DC:?export DC=vr1-dc0|vr1-dc1}"
  install -d -m 700 "$HOME/${DC}-creds"
  scp -q "voffice1:~/${DC}-substrate.tfstate.gz" "$HOME/${DC}-creds/${DC}-substrate.tfstate.gz"
  chmod 600 "$HOME/${DC}-creds/${DC}-substrate.tfstate.gz"
  A=$(sha256sum < "$HOME/${DC}-creds/${DC}-substrate.tfstate.gz" | cut -d' ' -f1)
  B=$(ssh voffice1 "sha256sum < ~/${DC}-substrate.tfstate.gz" | cut -d' ' -f1)
  [ "$A" = "$B" ] || { echo "ABORT: sha256 MISMATCH -- do NOT delete the source"; exit 1; }
  # RESTORE PROOF, not just a transfer check: the archive must decompress to valid JSON and
  # report the serial the source reported. A truncated gzip passes a size check and fails here.
  SER=$(gzip -dc "$HOME/${DC}-creds/${DC}-substrate.tfstate.gz" | python3 -c "import json,sys;print(json.load(sys.stdin)['serial'])")
  echo "OK sha256 matches; restored serial=$SER"
  ssh voffice1 "rm -f ~/${DC}-substrate.tfstate.gz"
} )
```

Run both blocks once per DC, and **again after every inner apply**. The staging copy is removed
only after the checksum matches AND the archive decompresses to valid JSON reporting a serial --
a truncated `gzip` would otherwise leave a plausible-looking backup that cannot be restored.

**FORWARD ITEM:** this is the INTERIM home. When the secrets-storage solution is pinned it MUST
carry a certificate/credential backup process, and this step and the Octavia one
(`phase-01-bundle-deploy.md` 1.0-GEN.e) are both steps that must fold into it -- see
`creds-matrix-notes.md` `## n-pki-backup`.

**KNOWN AND NOT FIXED HERE:** both live inner states are mode **664**, i.e. group-writable. They
are the authority `tofu` trusts, so a group member who rewrites one can make `tofu` destroy or
orphan real resources. Tightening them is a separate gated change against a file `tofu` rewrites
on every apply, so it needs its own verification that the provider preserves the mode.

## DC standup definition-of-done (DOCFIX-198 -- as executed on dc0 + dc1; the dc2+/Roosevelt checklist)

Every item below was established the hard way on the dc0 build (2026-07-19..21)
and pre-empted the same failure on dc1 (2026-07-22/23). A DC standup is NOT
done until each row is verified; the citations are the ruling/lesson
authorities. Every mutation stays individually operator-gated as usual.

1. **Host->site reach measured, never inferred (D-126 / register item 20).**
   At standup, MEASURE how vcloud reaches the DC's containment/service net.
   An ISOLATED host-local net gets a measured leg row + `site-baseleg.sh`
   unit install; a routed/NAT path needs none. DCs keep the `-J`-through-
   containment ssh shape.
2. **Rack network twin (D-131 sub-1, RULED).** `scripts/dc-rack-net.sh
   install <site>` on the rack: reboot-persistent rack bridge legs + the
   node-DNS dnsmasq forwarder on metal-admin, forwarding to region BIND over
   the rack's own transit. WITHOUT it, MAAS 3.7's rack-only agent resolver
   SERVFAILs on an isolated rack and commissioning silently times out
   (appendix-A carries the symptom pair). Verify: `dc-rack-net.sh check
   <site>` PASS + forwarder answers an authoritative `maas-internal` SOA.
3. **Serial-log staging dir** `/var/lib/libvirt/vr1/staging/` on the rack --
   created by `site-headend-install.sh --host-nodes` (since 2026-07-23; its
   absence failed the dc1 inner apply at 18/28). `--check` verifies it.
4. **Region-side MAAS subnet config on the DC's metal-admin subnet:** dynamic
   range at **.201-.254** (D-134 AMENDMENT 2026-07-23 supersedes the original
   D-120-inherited .100-.200 position -- .100-.200 is now the contiguous NODE
   band), D-131 `dns_servers=<rack forwarder IP>` + `allow_dns=false`, then
   DHCP `dhcp_on=true primary_rack=<rack system_id>` -- verify dhcpd RUNNING
   by process on the rack, not by MAAS self-report (the 2026-07-20 Temporal
   incident class).
5. **Per-DC MAAS->libvirt power key (SEC-012 dc0 / SEC-016 dc1 -- standing
   invariant, RULED).** Each DC gets its OWN dedicated power keypair, never
   cross-DC reuse: private half in the region MAAS snap
   (`/var/snap/maas/current/root/.ssh/` + a per-host `ssh config` block so
   each rack transit IP uses its own key); public half authorizes the
   libvirt-group user on that rack. **FRAGILITY: the snap-side key + config
   live under per-revision `/var/snap/maas/current/` -- RE-ASSERT after any
   MAAS snap refresh** (queued rotation/scope questions live in
   `docs/security-ledger.md`). Roosevelt analog: per-DC IPMI/BMC creds.
6. **Per-machine node power, no pods (D-103/D-123 amendments, RULED).**
   `scripts/maas-node-power.sh` dry -> commit -> each write verified by a
   real `query-power-state`. Node identity rides the config-pinned MACs
   (modules/node-vm; 54 per DC), which is what makes PXE re-enlistment
   deterministic.
7. **Commissioning gate:** all 9 nodes READY, shapes exact to D-121 Option C
   (3x16cpu/64GiB + 2x12cpu/48GiB + 4x8cpu/24GiB), captured to a dated
   audit file (dc0: `commissioning-diag-20260721`; dc1:
   `dc1-commissioning-verify-20260723` -- READY in ~3.5 min when 1-6 are in
   place).
8. **Edge pf ruleset regenerated + LAN-client egress proven (2026-07-23
   incident).** After the edge's v4 addressing, run `configctl filter
   reload` on the edge, then from the RACK prove LAN-side egress: ping an
   internet IP 0% loss + `curl -sI http://1.1.1.1/` returning 301 (the
   D-125 signature). WITHOUT the reload a fresh edge passes NO LAN traffic
   (automatic outbound NAT rules are never generated) with a perfectly
   reachable gateway -- appendix-A carries the symptom. Do NOT rely on
   plugin installs to reload the filter as a side effect (that masking is
   exactly how dc0 hid this from dc1).
9. **Per-DC artifact mirror (D-135 item 1).** `scripts/dc-mirror.sh install
   <site>` on the rack (net layer -> printed apt line -> re-run install),
   then `sync <site>`; verify `check <site>` PASS and, post-sync,
   `last-sync.status` reads OK. The mirror serves the D-134 utility .4 on
   metal-admin; Stage 5 wires it via `juju model-config apt-mirror=`.

---

## GATE -- Stage 3 exit condition (honest state, not aspirational)

Buildout-design Section 4 / deployment-workflow Stage 3 states the gate as:
"MAAS rack controller per DC up; MAAS discovers the OpenTofu-created node
VMs; edge routing + simulated ISP uplink up; netem parameters applied and
measured." As of what this runbook can actually close this session:

- **MAAS rack controller per DC up:** achievable ONLY once `vr1-dc0`'s own
  MAAS rack controller is stood up -- a Stage-4 gap that is NOT yet done
  (D-122 confirms each site runs its own controller; the SOURCE for it is the
  open item). Steps 1/9 depend on it. This is why honest scope for a first run
  is Steps 1-8. Record actual state, do not assume it.
- **MAAS discovers the OpenTofu-created node VMs:** achievable through Step
  10, CONDITIONAL on the per-DC rack controller existing (Step 9's dependency)
  and Stage 2's MAAS region being reachable. Node sizing itself is no longer a
  condition -- it is RULED (D-121, Step 6).
- **Edge routing + simulated ISP uplink up:** PARTIALLY achievable. The
  OPNsense domain and its network attachments can be created (Step 8), and the
  edge shape/uplink are RULED (D-122: `vr1-dc0-wan` dedicated ISP uplink,
  Office1-pattern edge), but full edge routing verification depends on the
  post-boot REST-API bootstrap (Step 4's REPLACEMENT chain -- console -> SSH
  -> API, where the real `vtnetN` mapping and WAN/LAN addressing are measured
  and applied). Do not mark this sub-gate fully closed until that bootstrap
  has actually run against the booted domain.
- **netem parameters applied and measured:** applied, YES (Step 11), using
  an explicitly-labeled PLACEHOLDER same-metro-lean profile per
  buildout-design Section 6 -- MEASURED in the sense of "the qdisc is
  present and its effect can be observed" (e.g. via `ping`/`iperf` once the
  domains are reachable), but NOT measured against a ruled, final numeric
  target, because that target does not exist yet (D-100 gap #11). Do not
  represent this sub-gate as "done" in the same sense as a stage with no
  open decisions -- it is done AS FAR AS A PLACEHOLDER ALLOWS, with the
  remaining gap named.

**This stage's exit gate is therefore CONDITIONALLY MET at best** on any run
before (a) `vr1-dc0`'s per-DC MAAS rack controller is stood up (Stage-4 gap)
and Stage 2's MAAS region is reachable, (b) `vr1-dc0`'s OPNsense edge has been
through the post-boot REST-API bootstrap (`vtnetN` + WAN/LAN applied), and
(c) D-100's exact netem parameters are ruled. Node sizing (D-121) and the edge
shape/uplink (D-122) are NO LONGER open. Update
`docs/dc-dc-deployment-workflow.md`'s Stage 3 row to reflect the REAL state
after running this (e.g. "PARTIAL -- libvirt objects created, MAAS
registration blocked on the per-DC rack controller" or similar), never to a
blanket done unless every sub-condition above is genuinely true.

**`vr1-dc1`:** none of the above applies to `vr1-dc1` this pass. Its v4
supernet IS assigned (D-115, `10.12.64.0/19`) -- the blocker is sequencing,
not a missing literal -- but its `vr1_dc1_planes` block stays commented out
and it stays out of scope. Re-run this entire runbook for `vr1-dc1` when it is
sequenced; do not attempt a partial `vr1-dc1` pass now.

-> Proceed to Stage 4 (MAAS enlist/commission/deploy) for `vr1-dc0` once this
gate's applicable sub-conditions are genuinely met; `vr1-dc1` remains blocked
at Stage 3 by sequencing until it is taken up as its own pass.

---

## Delivery checklist (this repo's standard discipline)

- [ ] `bash scripts/repo-lint.sh` clean (0 fail) before committing any repo
      changes made while executing this runbook (`main.tf` edits, any new
      tfvars, minus secrets).
- [ ] `bash scripts/opentofu-validate.sh` green (the harness for `opentofu/`,
      per its own README) -- re-run after every `main.tf` change in Steps
      5/6/9/11, not just once at the end.
- [ ] Changelog entry for this runbook's first real execution (next DOCFIX
      number via `bash scripts/ledger-scan.sh`), noting the ACTUAL measured/
      decided values used (node sizing, MAAS zone/pool, power_address, the
      netem placeholder profile and its provisional status) -- redact
      nothing that isn't a secret.
- [ ] `docs/session-ledger.md` updated with the outcome, including which
      sub-steps were blocked and why (per-DC MAAS rack-controller Stage-4 gap,
      Stage 2 MAAS-region dependency, the edge post-boot bootstrap, D-100
      netem sub-item).
- [ ] `docs/CURRENT-STATE.md` updated to the REAL, honest Stage 3 state
      (GA-R1: the only status authority; per the GATE section above),
      never rounded up with open sub-conditions.
- [ ] `opentofu/README.md` updated to reflect `vr1-dc0`'s OPNsense edge /
      node-VM / MAAS-vm-host / netem-link modules moving from "not
      instantiated" to instantiated (or partially so) once this runbook
      actually applies them, including any real bug found while doing so
      (matching this repo's practice of logging schema surprises as they're
      found).
- [ ] `vr1-dc1` gate re-confirmed as still closed BY SEQUENCING (its supernet
      IS assigned -- D-115 `10.12.64.0/19`) if this session did not take it
      up -- do not let this runbook's `vr1-dc0` completion be misread as
      covering `vr1-dc1`.
