# Committee record -- Track 2: vr1-dc1 deploy-artifact (bundle) render

**Date:** 2026-07-24. **Branch:** `dc-dc-stage4-phase3-maas-deploy` (Stage 4 open).
**Convened by:** operator ("Deploy a committee to review Track 2 and provide
recommendations based on overall end project goals and decisions already
researched").

**Status of this document.** ADVISORY / audit record. It ADOPTS nothing: the
Fork-1 recommendation below is PENDING an operator ruling (GA-R5); a decision
exists only when its Status line quotes the question + the operator's exact
utterance in `docs/design-decisions.md`. `docs/CURRENT-STATE.md` remains the
sole status authority (GA-R1). Precedent for the format:
`docs/audit/commissioning-committee-20260720.md`.

## Method

Four independent, read-only reviewers, each a distinct lens, each given a
self-contained brief grounded in the end-project goals (MINIMIZE DELTA TO
ROOSEVELT; VR1 exists to make HA real and prove DR drills) and the researched
decisions (D-121, D-104, D-119, D-052, D-062, D-009, D-106, D-108):

1. Roosevelt-transferability / architectural-delta
2. HA/DR correctness & multi-tenant isolation
3. Operational risk / minimal-delta-now
4. Juju/MAAS/charm mechanism-feasibility

Every load-bearing claim was re-verified against repo HEAD by the adjudicator
before adoption into this record; two verifications corrected the brief given to
the committee (see "Corrections", below).

## The blocker being reviewed

The committed `bundle.yaml` is VR0's 4-machine ("8"-"11") HYPERCONVERGED layout
(`tags=openstack`, single-unit control plane by D-009). VR1 dc1 is 9
ROLE-SEPARATED nodes (D-121 Option C: 3 control + 2 compute + 4 storage), built
and merged (G12). The deploy ARTIFACTS were never rendered to that architecture.
The substrate is exactly 9 node VMs (no controller slot), all Ready, all sharing
ONE uniform MAAS tag `openstack-vr1-dc1`, zone `default`
(`docs/audit/stage4-discovery-20260723.txt`).

## Facts verified against the repo (not from prose)

- **Designate IS in the Stage-5 bundle** -- `bundle.yaml:812` (`designate`),
  `:840` (`designate-bind`), `:922` (`designate-hacluster`, noted "D-106
  reactivated 2026-07-10, supersedes D-019"). The scale-up overlay's
  `designate:3` references a real deployed app. The phase-4 runbook line 192
  ("bundle ships NO designate") is STALE and is part of the de-stale work.
- **Fork-1 option (a) is additive, not a substrate reopening** -- the inner
  caller is `module "vr1_dc1_node" { for_each = local.vr1_dc1_nodes }`
  (`opentofu/vr1-dc1-substrate/main.tf:140`), keyed (not `count`). Adding a 10th
  node key plans `1 to add, 0 to change, 0 to destroy`; the 9 merged nodes are
  untouched.
- **Capacity for (a) is a MEASURED gate, not a quotable figure** -- the signed
  doc's `790 GiB / 77%` is the OLD 3+2+3 (8-node) layout
  (`docs/design-decisions.md:3451`); the built layout is 3+2+4 and no `~85%`
  figure is carried in CURRENT-STATE. `scripts/dc-dc-whole-host-budget.py` must
  be RUN for the 10-node x 2-DC shape before the apply (GA-R1: figures from
  captured output).
- **H1 defect CONFIRMED** (see below) -- `bundle.yaml` `ceph-rbd-mirror` carries
  a duplicate `bindings:` key; `yaml.safe_load` resolves it to an orphan.

## Fork verdicts

### FORK 1 -- controller arithmetic: UNANIMOUS (a), a dedicated 10th controller VM

All four lenses independently reached (a) and rejected (b)/(c):

- **(b)** bootstrap onto one of the 9 + shrink a role (e.g. ceph-osd 4->3)
  REVERSES the ratified D-121 R-3 amendment and deletes the Ceph rebuild
  headroom the DR drill exists to demonstrate ("does not degrade the drill, it
  deletes the thing being drilled").
- **(c)** co-locate on a control node: `juju bootstrap` on MAAS consumes a WHOLE
  machine (mechanism-confirmed), collapsing 3 control nodes -> 2, breaking the
  mysql/ovn/ceph-mon quorum and coupling controller-loss to the single-node
  drill. Rejected.
- **(a)** is the only option preserving 3+2+4, matches D-104's VERIFIED VR0
  precedent (a dedicated `juju` VM separate from the OpenStack nodes,
  `docs/maas-as-built-reference.md`), and is Roosevelt-shaped. It is additive
  (see Facts) and light on capacity (subject to the measured gate).
- **Least-bad co-location, if (a) ever cannot fit** (HA/DR lens): a COMPUTE node
  (hosts neither quorum nor OSD headroom), never a control node.

**Classification: a D-104 amendment** (controller placement is D-104's subject;
substrate-count changes went through amendment for storage 3->4). PENDING
operator ruling.

### FORK 2 -- render mechanism: adjudicated -- hand-render now, GATED by an extended validator

Split panel: the Roosevelt / HA-DR / mechanism lenses lean to SOME automation
(the render is a ~56-app cross-consistency rewrite); the operational lens calls a
renderer tool gold-plating and says hand-render now. The dividing line is not
hand-vs-tool -- ALL FOUR require an automated PLACEMENT check, which
`provider-bundle-check.py` does not do today (it validates D-052 bindings +
mysql=3 only). Adjudication:

- **Hand-render the concrete dc1 `bundle.yaml` now** (a concrete file is directly
  reviewable, which the "render AND review" constraint demands; you must render
  once to learn the parameter surface anyway).
- **Gate it behind an EXTENDED validator** -- add assertions to the EXISTING
  tested `provider-bundle-check.py` (no new tooling): 3-distinct-host
  anti-affinity for the 14 scaled apps; `ceph-osd=4` on storage-tagged nodes;
  `nova-compute=2` on compute-tagged nodes; controller-tag carries no role tag;
  and NO dangling `to:` (every referenced machine ID exists).
- **Defer a general renderer tool** until dc0 forces the second render and proves
  real churn. The `{{TOKEN}}` / overlay pattern already carries the per-DC parts.

### FORK 3 -- role->node binding: UNANIMOUS per-role MAAS tags

- **per-role tags** `constraints: tags=openstack-vr1-dc1,control` -- MAAS
  AND-filters at allocation; deterministic; survives re-enlist; Roosevelt-shaped.
  ADOPTED recommendation. Prerequisite: add `control`/`compute`/`storage` tags to
  the respective nodes (a small, additive, gated MAAS mutation) AND wire that
  application into the re-enlist runbook so a node rebuild re-applies it.
- **`mem=` -- REJECTED**: mechanism-confirmed a FLOOR, not exact-match; a 64 GiB
  control node satisfies a `mem=49152` compute request -> silent mis-placement
  race under parallel deploy.
- **system-id pinning -- REJECTED**: not cleanly bundle-expressible, and system
  IDs are re-minted on every re-enlist (DOCFIX-040); VR1 recovery IS PXE
  re-enlist.

## Confirmed defects (independent of the fork rulings)

- **H1 / BUNDLEFIX-052 -- `ceph-rbd-mirror` duplicate `bindings:`.**
  `bundle.yaml:891-895` is the correct block (`ceph-local`/`ceph-remote` ->
  replication, `nrpe-external-master` -> metal-internal; D-108/D-052); an ORPHAN
  block at `:907-910` (`certificates`/`dashboard`) is absorbed into
  `ceph-rbd-mirror` (YAML takes-last), clobbering the replication-plane bindings
  (defeating D-108) and injecting a phantom `dashboard` binding to a charm with
  no such endpoint -> `juju deploy` validation reject. Pre-existing; latent
  because the app was flagged prep-only. Fix = delete the orphan. FIXED this
  session (changelog).
- **Stale phase-4 runbook Step 4** -- "deploy the EXISTING bundle.yaml", "ships
  NO designate", and region-naming drift (its "DC1"=vr1-dc0, "DC2"=vr1-dc1).
  De-stale as part of the render.

## The render work, specified (the load-bearing part)

- **Re-home every base placement** (in `bundle.yaml`, NOT the overlay -- the
  highest silent risk): `mysql-innodb-cluster`/`ovn-central`/`ceph-mon`
  `to:[lxd:8,9,10]` -> 3 control IDs; `nova-compute` `["9","10","11"]` **3->2**
  -> the 2 compute IDs; `ceph-osd` `["8","9","10","11"]` -> the 4 storage IDs;
  `memcached`/`glance-simplestreams-sync`/`designate-bind`/`ceph-rbd-mirror`
  singletons -> control IDs. Any un-updated `to:` = a dangling reference =
  deploy-time validation failure.
- **Machines block**: 9 role-tagged IDs + the 10th controller VM with a DISTINCT
  tag (e.g. `juju-controller-vr1-dc1`) carrying NO role tag, so neither
  `juju bootstrap --constraints` nor a role constraint can cross-grab.
- **Re-scope the dc0-named tokens** `{{VR1_DC0_CONTROL_0N}}` -> dc1 control IDs;
  positional anti-affinity is a 3-entry `to:` list (unit0->entry0), mechanism
  confirmed -- token collision silently defeats it, so validate distinctness.
- **Wire overlays** `vr1-dc1-vips.yaml` + `dc-ha-scaleup.yaml` into the deploy;
  de-stale phase-4 Step 4.

## Deploy-time VERIFY-LIVE gates (into the runbook; not render blockers)

- **Vault 3-unit HA-on-MySQL** -- D-121's own gate: verify the `1.8/stable` charm
  truly does leader-election at 3 units BEFORE commit (mechanism lens: genuinely
  UNSURE); if not, evaluate Raft -> D-068, NEVER etcd. Plus 3x manual-unseal
  burden (OPEN SEC-003).
- **`hacluster cluster_count:3`** actually gates VIP-bind-on-quorum (else
  decorative HA).
- **dc1 VIPs** (`10.12.64/68/72`) are outside the checker's hardcoded dc0 bands
  -- validate the merged result.
- **CRUSH failure-domain = host** (the rebuild-headroom argument rests on it).

## Bootstrap ordering (reinforced)

Bootstrap the 10th VM (by its distinct controller tag) WHILE the DC1 edge egress
is OPEN -- the juju agent stream + juju/juju-db snaps are fetched pre-apt and are
NOT mirrored (D-135 items 2-3 unbuilt). No egress-narrowing (D-135/D-107) until
the controller is up.

## Open operator decisions (one ruling per exchange, GA-R5)

1. **FORK 1 (blocking, a D-104 amendment):** committee unanimous for **(a)
   dedicated 10th controller VM**, additive, gated on the whole-host budget
   script. -- PENDING.
2. **FORK 2:** hand-render + extended validator; defer a renderer tool. --
   PENDING (follows Fork 1).
3. **FORK 3:** per-role MAAS tags + re-enlist wiring. -- PENDING (follows Fork 1).

## Highest-priority concerns (one per lens)

- Roosevelt: extend the placement validator before bootstrap, or any render can
  silently ship one-unit-per-node-less-ness = the decorative HA D-121 exists to
  kill.
- HA/DR: no automated placement gate today; the un-re-homed base placements
  (`lxd:8,9,10`) are the most likely silent decorative-HA path.
- Operational: confirm the inner caller is `for_each`-keyed (DONE -- it is) and
  constrain bootstrap to a distinct controller tag, or the closed G12 substrate
  reopens.
- Mechanism: re-render COMPLETENESS -- only 14 of ~56 apps are placed by the
  overlay; a single stale `to:` fails `juju deploy` with no partial success. And
  fix H1 first (DONE).