Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260715-d121-d122-stage3-ha-nodelayout.md
@JANeumatrix JANeumatrix 21 hours ago 8 KB Stage 3 (vr1-dc0) prep batch: D-121..D-124 rulings + substrate/rack wiring, HA overlay, NetBox importers
Raw Blame History

2026-07-15 -- Stage 3 kickoff: D-121 (HA scale-up + Option C node layout) + D-122 (site shape); phase2 runbook fix-forward

Context

Opened Stage 3 (VR1 DC substrate) on branch dc-dc-stage3-phase2-dc-substrate. Walked runbooks/dc-dc-phase2-tofu-dc-substrate.md to surface its blocking decisions before any tofu apply. The walk (plus a bundle.yaml HA review and a whole-host resource validation) produced two operator rulings and a runbook fix-forward. No cloud mutation this session -- design decisions, one resource calculation, and doc reconciliation only.

What changed

D-121 (ADOPTED IN PART) -- VR1 makes HA real + the node layout that carries it. docs/design-decisions.md. The bundle review found the root fact: VR0 ran its control plane single-unit by design (D-009: hacluster + VIP relations at num_units: 1, "decorative... scale-up is mechanical: 1 -> 3"). The 12 hacluster subordinates and D-036's dual VIPs are already wired; the units were never added. D-121 adopts the 14-service 1->3 scale-up (keystone, glance, neutron-api, nova-cloud-controller, placement, cinder, openstack-dashboard, octavia, barbican, magnum, designate, ceph-radosgw + rabbitmq + vault; ceph-rbd-mirror stays 1 per D-108). Two realities recorded so the cost is not under-stated: (1) it is a placement rewrite (each app's to: must spread across 3 distinct hosts -- co-located units are decorative again; a Stage-5 placement-balance rule was folded in from the openstack0-3 balance analysis: spread each app's 3 units one-per-control-node so the RAM-heavy trio auto-balances, and distribute the ~42 API containers evenly ~14/control-node, sized for at 64 GiB); (2) vault is MySQL-backed today (C1 revised D-006; vault:ha removed by BUNDLEFIX-002), so its HA path was a sub-ruling -- RULED (v-a) 2026-07-15: keep MySQL-backed, scale to 3, re-add vault:ha, gated on verifying the 1.8/stable charm does MySQL-backend HA with 3 units (else evaluate Raft via D-068, NOT etcd). 3-unit vault triples the unseal burden, intersecting OPEN SEC-003. Node layout = Option C (operator-ruled): 3 control + 2 compute + 3 storage per DC (16 node VMs). Ruled sizing: control 16 vCPU/64 GiB/150 GiB, compute 12/48/100, storage 8/24/550. ceph-osd=3/DC, nova-compute=2/DC, ceph-mon/mysql/rabbitmq/vault/ovn-central=3 across the control nodes.

Whole-host resource validation (Option C). Measured the actual VR1 host (not the design estimate): 256 vCPU / 1024 GiB RAM / 10 TiB disk (nproc/free/df; currently only Office1 runs -- voffice1 16/32/600 + office1-opnsense 2/2/11, netbox+tailscale nested inside voffice1). Modeled Option C across all layers (Office1 as-built + both DCs + edges + containment overhead) at scratchpad/optc-calc.py: 222 vCPU / 790 GiB / 5.4 TiB -> FITS (RAM binding at 77%; vCPU overcommits; disk thin-provisioned at 53%). Finding: C is NOT heavier than B -- it is RAM-lighter (790 vs 838 GiB) because role separation lets storage nodes run lean. Sensitivity: control-node RAM is the knob (96 GiB -> 982 GiB / 96%); held <= 64 GiB. This corrected D-121's first-pass "C is heaviest" characterization. Also ran scripts/dc-dc-ceph-disk-budget.sh (Option B/C 4+4 and 3+2+3): PASS with wide margin -- disk is not the binding constraint.

D-122 (ADOPTED) -- VR1 site shape. docs/design-decisions.md. Records the operator's Ruling 2: each site nested in its own containment VM (site-down/recovery + site-to-site drills); dark fiber DC-DC-Office is East-West ONLY (never an internet path); each site has its own dedicated L3 "ISP" uplink (build vr1-dc0-wan on the office1-wan pattern -- NOT a mesh leg); the DC OPNsense edge follows the proven Office1 pattern (2048 MiB / 2 vCPU / nano, REST-API config per D-113(a2)); each site runs its own MAAS controller (may bootstrap from an existing one). Closes the phase2 runbook's Ruling-1/Ruling-2 STOP-gates. Subsumes the DOCFIX-185 amendment note. Edge NIC model ruled (D-122 refinement, operator 2026-07-15): 2-NIC, baremetal-matched. NIC count mirrors the Roosevelt hardware -- nodes = 6 NICs (one per plane, measured on VR0 openstack0-3), edge = 2 (WAN + LAN). The six planes are routed by the FABRIC (OVN/OpenStack on the deliberately isolated-L2 dc-planes segments), so the edge is only the external boundary, not an inter-plane router -- no opnsense-edge module extension. Edge WAN = vr1-dc0-wan (172.30.2.0/24, D-115); edge LAN = provider-public (10.12.4.0/22; external gateway per D-100). This resolves the runbook's earlier "LAN role / build a site-internal segment" flag: no new segment is needed. Two build prerequisites remain: build vr1-dc0-wan as a tofu module (promote off the virsh office1-wan one-off), and register 172.30.2.0/24 in office1-netbox (D-115-ruled, not yet loaded -- absent from the draft; no import covered the DC Edge /24s).

phase2 runbook fix-forward. runbooks/dc-dc-phase2-tofu-dc-substrate.md:

  • Region-qualified to the D-119 namespace -- every dc1/dc2 object renamed to vr1-dc0/vr1-dc1 (matching main.tf); added a naming note + a HAZARD line that any residual bare dcN reintroduces the exact cross-cloud collision D-119 deletes. Revert: git checkout the file.
  • Steps 4-5 rewritten to the D-113(a2) REST-API edge bootstrap -- dropped the config.xml render + the config-ISO path (D-112: the Importer cannot fire on a nano image; the module already dropped config_iso_path); edge is now prep-image + post-boot console/SSH/API. WAN_IF/LAN_IF agonizing removed (measured post-boot).
  • D-122 edge shape wired into Step 5: memory_mib=2048/vcpu=2/nano, no disk_size_bytes, wan_network_name = vr1-dc0-wan (dedicated per-site ISP uplink, office1-wan pattern), lan_network_name = site-internal network -- both flagged as genuine build prerequisites.
  • D-121 Option C wired into Step 6: 8 module "vr1_dc0_node_*" blocks in three role-sizes (3 control 16 vCPU/65536 MiB/150 GiB, 2 compute 12/49152/100 GiB, 3 storage 8/24576/550 GiB; disk in bytes via N * 1024*1024*1024), with the Stage-5 HA-scale-up context.
  • Corrected the vr1-dc1 gate rationale (top callout + exit GATE + checklist): its supernet IS assigned (D-115 10.12.64.0/19, imported to office1-netbox); it stays out of scope by SEQUENCING, not a missing literal. No vr1-dc1 build steps added.
  • Sequence block, exit GATE, Step-3 register, and delivery checklist updated to match; open items kept as flagged STOPs (per-DC MAAS rack controller = Stage-4 gap, honest first-run scope Steps 1-8; the vault-HA backend sub-ruling; netem params). Revert: git checkout runbooks/dc-dc-phase2-tofu-dc-substrate.md.

Two gaps surfaced by the fix-forward (LOGGED, not fixed -- they need real values/build work):

  1. opentofu/main.tf's commented vr1_dc1_planes block (~L38-50) still says "wait for NetBox to assign D-101's supernet" -- STALE per D-115. Left in place (this was a doc-only pass); flagged for the main.tf owner.
  2. Edge network prerequisites (LAN side RESOLVED after the operator's fabric-routing ruling). The fix-forward first flagged "no site-internal LAN segment for vr1-dc0"; the operator then ruled the planes are fabric-routed and the edge is 2-NIC, so the LAN = provider-public (a vr1_dc0_planes output) -- NO new site-internal segment needed. Two REAL prerequisites remain before Step 5's tofu apply: build vr1-dc0-wan as a tofu module (promote off the virsh office1-wan one-off), and register 172.30.2.0/24 in office1-netbox (D-115-ruled, not yet loaded). Recorded in the runbook, not invented.

Ledger + tracker reconciliation. docs/session-ledger.md (next-free D 121->123; a Stage-3 open-work block with the two rulings, the validation, the fix-forward-in-flight, the stale-gate correction, and the logged-not-executed calc-promotion follow-up) and docs/dc-dc-deployment-workflow.md (Stage 3 State row updated from "runbook written" to "IN PROGRESS + two rulings landed").

Tests / verification

  • bash scripts/repo-lint.sh -> 0 fail, 1 legacy WARN (the documented D-001..018 non-ASCII carve-out; new D-121/D-122 text is ASCII).
  • bash scripts/ledger-scan.sh -> next-free D = 123; D-121/D-122 not flagged PROPOSED/OPEN.
  • Resource model reproducible: python3 scratchpad/optc-calc.py.

Revert

  • D-121 / D-122: delete the two ## D-121: / ## D-122: sections from docs/design-decisions.md (they are append-only additions; nothing else references them yet in code).
  • Ledger / workflow: revert the docs/session-ledger.md next-free line + the Stage-3 block, and the docs/dc-dc-deployment-workflow.md Stage 3 State paragraph, to their prior text.
  • Runbook: git checkout runbooks/dc-dc-phase2-tofu-dc-substrate.md (see the runbook section for the per-change reverts).
  • No cloud state was touched, so there is nothing to unwind on the host.