# 2026-07-15 -- Stage 3 kickoff: D-121 (HA scale-up + Option C node layout) + D-122 (site shape); phase2 runbook fix-forward

## Context

Opened Stage 3 (VR1 DC substrate) on branch `dc-dc-stage3-phase2-dc-substrate`. Walked
`runbooks/dc-dc-phase2-tofu-dc-substrate.md` to surface its blocking decisions before any
`tofu apply`. The walk (plus a bundle.yaml HA review and a whole-host resource validation)
produced two operator rulings and a runbook fix-forward. No cloud mutation this session --
design decisions, one resource calculation, and doc reconciliation only.

## What changed

**D-121 (ADOPTED IN PART) -- VR1 makes HA real + the node layout that carries it.**
`docs/design-decisions.md`. The bundle review found the root fact: VR0 ran its control plane
**single-unit by design** (D-009: hacluster + VIP relations at `num_units: 1`, "decorative...
scale-up is mechanical: 1 -> 3"). The 12 hacluster subordinates and D-036's dual VIPs are already
wired; the units were never added. D-121 adopts the **14-service 1->3 scale-up** (keystone, glance,
neutron-api, nova-cloud-controller, placement, cinder, openstack-dashboard, octavia, barbican,
magnum, designate, ceph-radosgw + rabbitmq + vault; `ceph-rbd-mirror` stays 1 per D-108). Two
realities recorded so the cost is not under-stated: (1) it is a **placement rewrite** (each app's
`to:` must spread across 3 distinct hosts -- co-located units are decorative again; a **Stage-5
placement-balance rule** was folded in from the openstack0-3 balance analysis: spread each app's 3
units one-per-control-node so the RAM-heavy trio auto-balances, and distribute the ~42 API containers
evenly ~14/control-node, sized for at 64 GiB); (2) **vault**
is MySQL-backed today (C1 revised D-006; `vault:ha` removed by BUNDLEFIX-002), so its HA path was a
sub-ruling -- **RULED (v-a)** 2026-07-15: keep MySQL-backed, scale to 3, re-add `vault:ha`, gated on
verifying the `1.8/stable` charm does MySQL-backend HA with 3 units (else evaluate Raft via D-068, NOT
etcd). 3-unit vault triples the unseal burden, intersecting OPEN SEC-003.
**Node layout = Option C (operator-ruled)**: 3 control + 2 compute + 3 storage per DC (16 node VMs).
Ruled sizing: control 16 vCPU/64 GiB/150 GiB, compute 12/48/100, storage 8/24/550. `ceph-osd`=3/DC,
`nova-compute`=2/DC, `ceph-mon`/`mysql`/`rabbitmq`/`vault`/`ovn-central`=3 across the control nodes.

**Whole-host resource validation (Option C).** Measured the actual VR1 host (not the design
estimate): **256 vCPU / 1024 GiB RAM / 10 TiB disk** (`nproc`/`free`/`df`; currently only Office1
runs -- `voffice1` 16/32/600 + `office1-opnsense` 2/2/11, netbox+tailscale nested inside voffice1).
Modeled Option C across all layers (Office1 as-built + both DCs + edges + containment overhead) at
`scratchpad/optc-calc.py`: **222 vCPU / 790 GiB / 5.4 TiB -> FITS** (RAM binding at 77%; vCPU
overcommits; disk thin-provisioned at 53%). Finding: C is NOT heavier than B -- it is RAM-lighter
(790 vs 838 GiB) because role separation lets storage nodes run lean. Sensitivity: control-node RAM
is the knob (96 GiB -> 982 GiB / 96%); held <= 64 GiB. This corrected D-121's first-pass "C is
heaviest" characterization. Also ran `scripts/dc-dc-ceph-disk-budget.sh` (Option B/C 4+4 and 3+2+3):
PASS with wide margin -- disk is not the binding constraint.

**D-122 (ADOPTED) -- VR1 site shape.** `docs/design-decisions.md`. Records the operator's Ruling 2:
each site nested in its own containment VM (site-down/recovery + site-to-site drills); dark fiber
DC-DC-Office is East-West ONLY (never an internet path); each site has its own dedicated L3 "ISP"
uplink (build `vr1-dc0-wan` on the office1-wan pattern -- NOT a mesh leg); the DC OPNsense edge
follows the proven Office1 pattern (2048 MiB / 2 vCPU / nano, REST-API config per D-113(a2)); each
site runs its own MAAS controller (may bootstrap from an existing one). Closes the phase2 runbook's
Ruling-1/Ruling-2 STOP-gates. Subsumes the DOCFIX-185 amendment note.
**Edge NIC model ruled (D-122 refinement, operator 2026-07-15): 2-NIC, baremetal-matched.** NIC count
mirrors the Roosevelt hardware -- nodes = 6 NICs (one per plane, measured on VR0 openstack0-3), edge = 2
(WAN + LAN). The six planes are routed by the FABRIC (OVN/OpenStack on the deliberately isolated-L2
`dc-planes` segments), so the edge is only the external boundary, not an inter-plane router -- no
`opnsense-edge` module extension. Edge WAN = `vr1-dc0-wan` (`172.30.2.0/24`, D-115); edge LAN =
**provider-public** (`10.12.4.0/22`; external gateway per D-100). This resolves the runbook's earlier
"LAN role / build a site-internal segment" flag: no new segment is needed. Two build prerequisites
remain: build `vr1-dc0-wan` as a tofu module (promote off the virsh `office1-wan` one-off), and register
`172.30.2.0/24` in office1-netbox (D-115-ruled, not yet loaded -- absent from the draft; no import
covered the DC Edge /24s).

**phase2 runbook fix-forward.** `runbooks/dc-dc-phase2-tofu-dc-substrate.md`:
- Region-qualified to the D-119 namespace -- every `dc1`/`dc2` object renamed to `vr1-dc0`/`vr1-dc1`
  (matching `main.tf`); added a naming note + a HAZARD line that any residual bare `dcN` reintroduces
  the exact cross-cloud collision D-119 deletes. Revert: `git checkout` the file.
- Steps 4-5 rewritten to the D-113(a2) REST-API edge bootstrap -- dropped the config.xml render + the
  config-ISO path (D-112: the Importer cannot fire on a nano image; the module already dropped
  `config_iso_path`); edge is now prep-image + post-boot console/SSH/API. WAN_IF/LAN_IF agonizing
  removed (measured post-boot).
- D-122 edge shape wired into Step 5: `memory_mib=2048`/`vcpu=2`/nano, no `disk_size_bytes`,
  `wan_network_name = vr1-dc0-wan` (dedicated per-site ISP uplink, office1-wan pattern),
  `lan_network_name` = site-internal network -- both flagged as genuine build prerequisites.
- D-121 Option C wired into Step 6: 8 `module "vr1_dc0_node_*"` blocks in three role-sizes (3 control
  16 vCPU/65536 MiB/150 GiB, 2 compute 12/49152/100 GiB, 3 storage 8/24576/550 GiB; disk in bytes via
  `N * 1024*1024*1024`), with the Stage-5 HA-scale-up context.
- Corrected the `vr1-dc1` gate rationale (top callout + exit GATE + checklist): its supernet IS
  assigned (D-115 `10.12.64.0/19`, imported to office1-netbox); it stays out of scope by SEQUENCING,
  not a missing literal. No `vr1-dc1` build steps added.
- Sequence block, exit GATE, Step-3 register, and delivery checklist updated to match; open items kept
  as flagged STOPs (per-DC MAAS rack controller = Stage-4 gap, honest first-run scope Steps 1-8; the
  vault-HA backend sub-ruling; netem params). Revert: `git checkout runbooks/dc-dc-phase2-tofu-dc-substrate.md`.

**Two gaps surfaced by the fix-forward (LOGGED, not fixed -- they need real values/build work):**
1. `opentofu/main.tf`'s commented `vr1_dc1_planes` block (~L38-50) still says "wait for NetBox to
   assign D-101's supernet" -- STALE per D-115. Left in place (this was a doc-only pass); flagged for
   the `main.tf` owner.
2. **Edge network prerequisites (LAN side RESOLVED after the operator's fabric-routing ruling).** The
   fix-forward first flagged "no site-internal LAN segment for `vr1-dc0`"; the operator then ruled the
   planes are **fabric-routed** and the edge is **2-NIC**, so the **LAN = provider-public** (a
   `vr1_dc0_planes` output) -- NO new site-internal segment needed. Two REAL prerequisites remain before
   Step 5's `tofu apply`: build **`vr1-dc0-wan`** as a tofu module (promote off the virsh `office1-wan`
   one-off), and register **`172.30.2.0/24`** in office1-netbox (D-115-ruled, not yet loaded). Recorded
   in the runbook, not invented.

**Ledger + tracker reconciliation.** `docs/session-ledger.md` (next-free D 121->123; a Stage-3
open-work block with the two rulings, the validation, the fix-forward-in-flight, the stale-gate
correction, and the logged-not-executed calc-promotion follow-up) and
`docs/dc-dc-deployment-workflow.md` (Stage 3 State row updated from "runbook written" to
"IN PROGRESS + two rulings landed").

## Tests / verification

- `bash scripts/repo-lint.sh` -> 0 fail, 1 legacy WARN (the documented D-001..018 non-ASCII
  carve-out; new D-121/D-122 text is ASCII).
- `bash scripts/ledger-scan.sh` -> next-free D = 123; D-121/D-122 not flagged PROPOSED/OPEN.
- Resource model reproducible: `python3 scratchpad/optc-calc.py`.

## Revert

- D-121 / D-122: delete the two `## D-121:` / `## D-122:` sections from `docs/design-decisions.md`
  (they are append-only additions; nothing else references them yet in code).
- Ledger / workflow: revert the `docs/session-ledger.md` next-free line + the Stage-3 block, and the
  `docs/dc-dc-deployment-workflow.md` Stage 3 State paragraph, to their prior text.
- Runbook: `git checkout runbooks/dc-dc-phase2-tofu-dc-substrate.md` (see the runbook section for the
  per-change reverts).
- No cloud state was touched, so there is nothing to unwind on the host.