Committee record -- Track 2: vr1-dc1 deploy-artifact (bundle) render

Date: 2026-07-24. Branch: dc-dc-stage4-phase3-maas-deploy (Stage 4 open). Convened by: operator ("Deploy a committee to review Track 2 and provide recommendations based on overall end project goals and decisions already researched").

Status of this document. ADVISORY / audit record. It ADOPTS nothing: the Fork-1 recommendation below is PENDING an operator ruling (GA-R5); a decision exists only when its Status line quotes the question + the operator's exact utterance in docs/design-decisions.md. docs/CURRENT-STATE.md remains the sole status authority (GA-R1). Precedent for the format: docs/audit/commissioning-committee-20260720.md.

Method

Four independent, read-only reviewers, each a distinct lens, each given a self-contained brief grounded in the end-project goals (MINIMIZE DELTA TO ROOSEVELT; VR1 exists to make HA real and prove DR drills) and the researched decisions (D-121, D-104, D-119, D-052, D-062, D-009, D-106, D-108):

Roosevelt-transferability / architectural-delta
HA/DR correctness & multi-tenant isolation
Operational risk / minimal-delta-now
Juju/MAAS/charm mechanism-feasibility

Every load-bearing claim was re-verified against repo HEAD by the adjudicator before adoption into this record; two verifications corrected the brief given to the committee (see "Corrections", below).

The blocker being reviewed

The committed bundle.yaml is VR0's 4-machine ("8"-"11") HYPERCONVERGED layout (tags=openstack, single-unit control plane by D-009). VR1 dc1 is 9 ROLE-SEPARATED nodes (D-121 Option C: 3 control + 2 compute + 4 storage), built and merged (G12). The deploy ARTIFACTS were never rendered to that architecture. The substrate is exactly 9 node VMs (no controller slot), all Ready, all sharing ONE uniform MAAS tag openstack-vr1-dc1, zone default (docs/audit/stage4-discovery-20260723.txt).

Facts verified against the repo (not from prose)

Designate IS in the Stage-5 bundle -- bundle.yaml:812 (designate), :840 (designate-bind), :922 (designate-hacluster, noted "D-106 reactivated 2026-07-10, supersedes D-019"). The scale-up overlay's designate:3 references a real deployed app. The phase-4 runbook line 192 ("bundle ships NO designate") is STALE and is part of the de-stale work.
Fork-1 option (a) is additive, not a substrate reopening -- the inner caller is module "vr1_dc1_node" { for_each = local.vr1_dc1_nodes } (opentofu/vr1-dc1-substrate/main.tf:140), keyed (not count). Adding a 10th node key plans 1 to add, 0 to change, 0 to destroy; the 9 merged nodes are untouched.
Capacity for (a) is a MEASURED gate, not a quotable figure -- the signed doc's 790 GiB / 77% is the OLD 3+2+3 (8-node) layout (docs/design-decisions.md:3451); the built layout is 3+2+4 and no ~85% figure is carried in CURRENT-STATE. scripts/dc-dc-whole-host-budget.py must be RUN for the 10-node x 2-DC shape before the apply (GA-R1: figures from captured output).
H1 defect CONFIRMED (see below) -- bundle.yaml ceph-rbd-mirror carries a duplicate bindings: key; yaml.safe_load resolves it to an orphan.

Fork verdicts

FORK 1 -- controller arithmetic: UNANIMOUS (a), a dedicated 10th controller VM

All four lenses independently reached (a) and rejected (b)/(c):

(b) bootstrap onto one of the 9 + shrink a role (e.g. ceph-osd 4->3) REVERSES the ratified D-121 R-3 amendment and deletes the Ceph rebuild headroom the DR drill exists to demonstrate ("does not degrade the drill, it deletes the thing being drilled").
(c) co-locate on a control node: juju bootstrap on MAAS consumes a WHOLE machine (mechanism-confirmed), collapsing 3 control nodes -> 2, breaking the mysql/ovn/ceph-mon quorum and coupling controller-loss to the single-node drill. Rejected.
(a) is the only option preserving 3+2+4, matches D-104's VERIFIED VR0 precedent (a dedicated juju VM separate from the OpenStack nodes, docs/maas-as-built-reference.md), and is Roosevelt-shaped. It is additive (see Facts) and light on capacity (subject to the measured gate).
Least-bad co-location, if (a) ever cannot fit (HA/DR lens): a COMPUTE node (hosts neither quorum nor OSD headroom), never a control node.

Classification: a D-104 amendment (controller placement is D-104's subject; substrate-count changes went through amendment for storage 3->4). PENDING operator ruling.

FORK 2 -- render mechanism: adjudicated -- hand-render now, GATED by an extended validator

Split panel: the Roosevelt / HA-DR / mechanism lenses lean to SOME automation (the render is a ~56-app cross-consistency rewrite); the operational lens calls a renderer tool gold-plating and says hand-render now. The dividing line is not hand-vs-tool -- ALL FOUR require an automated PLACEMENT check, which provider-bundle-check.py does not do today (it validates D-052 bindings + mysql=3 only). Adjudication:

Hand-render the concrete dc1 bundle.yaml now (a concrete file is directly reviewable, which the "render AND review" constraint demands; you must render once to learn the parameter surface anyway).
Gate it behind an EXTENDED validator -- add assertions to the EXISTING tested provider-bundle-check.py (no new tooling): 3-distinct-host anti-affinity for the 14 scaled apps; ceph-osd=4 on storage-tagged nodes; nova-compute=2 on compute-tagged nodes; controller-tag carries no role tag; and NO dangling to: (every referenced machine ID exists).
Defer a general renderer tool until dc0 forces the second render and proves real churn. The {{TOKEN}} / overlay pattern already carries the per-DC parts.

FORK 3 -- role->node binding: UNANIMOUS per-role MAAS tags

per-role tags constraints: tags=openstack-vr1-dc1,control -- MAAS AND-filters at allocation; deterministic; survives re-enlist; Roosevelt-shaped. ADOPTED recommendation. Prerequisite: add control/compute/storage tags to the respective nodes (a small, additive, gated MAAS mutation) AND wire that application into the re-enlist runbook so a node rebuild re-applies it.
mem= -- REJECTED: mechanism-confirmed a FLOOR, not exact-match; a 64 GiB control node satisfies a mem=49152 compute request -> silent mis-placement race under parallel deploy.
system-id pinning -- REJECTED: not cleanly bundle-expressible, and system IDs are re-minted on every re-enlist (DOCFIX-040); VR1 recovery IS PXE re-enlist.

Confirmed defects (independent of the fork rulings)

H1 / BUNDLEFIX-052 -- ceph-rbd-mirror duplicate bindings:. bundle.yaml:891-895 is the correct block (ceph-local/ceph-remote -> replication, nrpe-external-master -> metal-internal; D-108/D-052); an ORPHAN block at :907-910 (certificates/dashboard) is absorbed into ceph-rbd-mirror (YAML takes-last), clobbering the replication-plane bindings (defeating D-108) and injecting a phantom dashboard binding to a charm with no such endpoint -> juju deploy validation reject. Pre-existing; latent because the app was flagged prep-only. Fix = delete the orphan. FIXED this session (changelog).
Stale phase-4 runbook Step 4 -- "deploy the EXISTING bundle.yaml", "ships NO designate", and region-naming drift (its "DC1"=vr1-dc0, "DC2"=vr1-dc1). De-stale as part of the render.

The render work, specified (the load-bearing part)

Re-home every base placement (in bundle.yaml, NOT the overlay -- the highest silent risk): mysql-innodb-cluster/ovn-central/ceph-mon to:[lxd:8,9,10] -> 3 control IDs; nova-compute ["9","10","11"] 3->2 -> the 2 compute IDs; ceph-osd ["8","9","10","11"] -> the 4 storage IDs; memcached/glance-simplestreams-sync/designate-bind/ceph-rbd-mirror singletons -> control IDs. Any un-updated to: = a dangling reference = deploy-time validation failure.
Machines block: 9 role-tagged IDs + the 10th controller VM with a DISTINCT tag (e.g. juju-controller-vr1-dc1) carrying NO role tag, so neither juju bootstrap --constraints nor a role constraint can cross-grab.
Re-scope the dc0-named tokens {{VR1_DC0_CONTROL_0N}} -> dc1 control IDs; positional anti-affinity is a 3-entry to: list (unit0->entry0), mechanism confirmed -- token collision silently defeats it, so validate distinctness.
Wire overlays vr1-dc1-vips.yaml + dc-ha-scaleup.yaml into the deploy; de-stale phase-4 Step 4.

Deploy-time VERIFY-LIVE gates (into the runbook; not render blockers)

Vault 3-unit HA-on-MySQL -- D-121's own gate: verify the 1.8/stable charm truly does leader-election at 3 units BEFORE commit (mechanism lens: genuinely UNSURE); if not, evaluate Raft -> D-068, NEVER etcd. Plus 3x manual-unseal burden (OPEN SEC-003).
hacluster cluster_count:3 actually gates VIP-bind-on-quorum (else decorative HA).
dc1 VIPs (10.12.64/68/72) are outside the checker's hardcoded dc0 bands -- validate the merged result.
CRUSH failure-domain = host (the rebuild-headroom argument rests on it).

Bootstrap ordering (reinforced)

Bootstrap the 10th VM (by its distinct controller tag) WHILE the DC1 edge egress is OPEN -- the juju agent stream + juju/juju-db snaps are fetched pre-apt and are NOT mirrored (D-135 items 2-3 unbuilt). No egress-narrowing (D-135/D-107) until the controller is up.

Open operator decisions (one ruling per exchange, GA-R5)

FORK 1 (blocking, a D-104 amendment): committee unanimous for (a) dedicated 10th controller VM, additive, gated on the whole-host budget script. -- PENDING.
FORK 2: hand-render + extended validator; defer a renderer tool. -- PENDING (follows Fork 1).
FORK 3: per-role MAAS tags + re-enlist wiring. -- PENDING (follows Fork 1).

Highest-priority concerns (one per lens)

Roosevelt: extend the placement validator before bootstrap, or any render can silently ship one-unit-per-node-less-ness = the decorative HA D-121 exists to kill.
HA/DR: no automated placement gate today; the un-re-homed base placements (lxd:8,9,10) are the most likely silent decorative-HA path.
Operational: confirm the inner caller is for_each-keyed (DONE -- it is) and constrain bootstrap to a distinct controller tag, or the closed G12 substrate reopens.
Mechanism: re-render COMPLETENESS -- only 14 of ~56 apps are placed by the overlay; a single stale to: fails juju deploy with no partial success. And fix H1 first (DONE).