diff --git a/bundle.yaml b/bundle.yaml index 5f61cdb..ac94b92 100644 --- a/bundle.yaml +++ b/bundle.yaml @@ -903,11 +903,11 @@ # cluster_count: 1 (decorative on single-unit testcloud, D-009 / BUNDLEFIX-003). # VIPs front-loaded into the MAAS-reserved provider/metal /26 per B1 (.2-.63). # vault-hacluster stays commented (vault single-unit on mysql, C1 / BUNDLEFIX-002). - # - bindings: - '': metal-admin - certificates: metal-internal - dashboard: metal-internal + # BUNDLEFIX-052 (2026-07-24): removed an orphaned 4-space-indented `bindings:` block here + # that YAML take-last had absorbed into ceph-rbd-mirror above, clobbering its real + # ceph-local/ceph-remote (replication) + nrpe-external-master bindings (D-108/D-052) and + # injecting a phantom `dashboard` binding (a charm with no such endpoint) -> juju deploy + # validation reject. Confirmed via yaml.safe_load; committee-20260724-track2-bundle-render. keystone-hacluster: {charm: hacluster, channel: 2.4/stable, options: {cluster_count: 1}, bindings: {'': metal-admin, ha: metal-internal, hanode: metal-internal, pacemaker-remote: metal-internal, peer-availability: metal-internal}} glance-hacluster: {charm: hacluster, channel: 2.4/stable, options: {cluster_count: 1}, bindings: {'': metal-admin, ha: metal-internal, hanode: metal-internal, pacemaker-remote: metal-internal, peer-availability: metal-internal}} neutron-api-hacluster: {charm: hacluster, channel: 2.4/stable, options: {cluster_count: 1}, bindings: {'': metal-admin, ha: metal-internal, hanode: metal-internal, pacemaker-remote: metal-internal, peer-availability: metal-internal}} diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 379ea43..f832c69 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -127,26 +127,49 @@ SNAPS pre-apt (not covered by the apt proxy; D-135 items 2-3 agent/snap mirror NOT built) -- it works ONLY while the DC1 edge egress is OPEN. Do NOT apply the D-135/D-107 egress-narrowing until AFTER bootstrap (or mirror agents/snaps first).** - - **BLOCKER (committee Q1): the committed `bundle.yaml` is still VR0's 4-node - HYPERCONVERGED layout** (4 machines "8"-"11", `tags=openstack`, `num_units: 1`, - LXD-on-metal `to:`), but VR1 dc1 is 9 ROLE-SEPARATED nodes (3 control/2 compute/ - 4 storage, tag `openstack-vr1-dc1`) per D-121 Option C. The ARCHITECTURE is - fully DECIDED (D-121 role-sep + HA scale-up 1->3; D-104 keeps the JUJU - controller single-unit; D-119 tags) and the substrate is built to it -- but the - DEPLOY ARTIFACTS were never rendered to it: `overlays/dc-ha-scaleup.yaml` is - PARTIAL (control-plane only, not compute/storage/quorum), UNRENDERED - ({{VR1_DC0_CONTROL_0N}} tokens), dc0-NAMED, STALE (ceph-osd 3 vs amended 4), and - NOT wired into any runbook; the machines block (4->9), tag, role-based placement, - nova-compute 3->2, and num_units 1->3 are UNHANDLED; the phase-4 runbook Step 4 - is STALE ("deploy the EXISTING bundle.yaml", omits both overlays, region-naming - drift: its "DC1"=vr1-dc0, its "DC2"=vr1-dc1-marked-BLOCKED). Neither the gap - register nor this doc tracked it before now. - - **NEXT (new session -- context exhaustion 2026-07-24):** render the vr1-dc1 - 9-node role-separated HA `bundle.yaml` (machines block, `openstack-vr1-dc1` tag, - role-based `to:`, num_units 1->3) + re-scope/render/wire `dc-ha-scaleup.yaml` - (+ ceph-osd/nova-compute/quorum placement) + the VIP overlay + de-stale the - phase-4 runbook (+ region-naming). THEN controller placement (consumes 1 of 9; - resolve the arithmetic) -> gated bootstrap (egress OPEN) -> deploy. + - **RENDER (committee-adjudicated 2026-07-24; record + `docs/audit/committee-20260724-track2-bundle-render.md`).** The committed + `bundle.yaml` is still VR0's 4-node HYPERCONVERGED layout (machines "8"-"11", + `tags=openstack`); VR1 dc1 is 9 ROLE-SEPARATED nodes (3 control/2 compute/4 + storage, tag `openstack-vr1-dc1`, D-121 Option C). Architecture DECIDED; deploy + ARTIFACTS not yet rendered. A 4-lens committee reviewed it and every load-bearing + claim was re-verified against the repo -- TWO corrected the brief: designate IS + in-bundle (`bundle.yaml:812/840/922`, D-106 supersedes D-019), so the phase-4 + "ships NO designate" (line 192) is STALE and part of the de-stale work; and the + inner caller is `for_each`-keyed (`opentofu/vr1-dc1-substrate/main.tf:140`) so a + 10th node is ADDITIVE (`1 add/0/0`), not a substrate reopening. Correcting an + earlier characterization: `dc-ha-scaleup.yaml` DELIBERATELY excludes + ceph-osd/nova-compute (scale-out, not control containers) -- it is NOT "stale + ceph-osd 3"; its real gaps are the dc0-named tokens + that the machines block + belongs in `bundle.yaml`. + - **Fork 1 RULED 2026-07-24 (operator, GA-R5 -- D-104 AMENDMENT):** "Dedicated + 10th VM (Recommended)" -- a dedicated per-DC Juju-controller node VM, separate + from the 9 role nodes, distinct tag (`juju-controller-vr1-dc1`, no role tag). + Capacity GATE PASS (measured): `scripts/dc-dc-whole-host-budget.py` 10-node x + 2-DC = RAM 854/1024 GiB = 83%, FIT, 170 GiB headroom (baseline 3+2+4 = 838/82%); + capture `docs/audit/stage5-controller-capacity-20260724.txt`. Fork 2 (hand-render + NOW + EXTEND `provider-bundle-check.py` with placement/anti-affinity/count + assertions; DEFER a renderer tool) and Fork 3 (per-role MAAS tags + + re-enlist-runbook wiring) = OPS, committee-UNANIMOUS, doubt-resolves-DOWN + (GA-R3) -- executed under gating; the Fork-3 tag mutation and the 10th-VM apply + stay individually operator-gated. + - **BUNDLEFIX-052 FIXED 2026-07-24:** `ceph-rbd-mirror` carried a duplicate + `bindings:` (an orphan at the old line ~907) that YAML take-last absorbed, + clobbering its D-108 replication bindings (`ceph-local`/`ceph-remote`) + + injecting a phantom `dashboard` binding -> `juju deploy` reject. Orphan removed; + `yaml.safe_load` confirms the correct set restored; `provider-bundle-check.py` + PASS. Pre-existing latent defect (app was prep-only, never deployed). + - **NEXT:** (1) commit+push this ruling batch (GA-R5: rulings land before + dependent work); (2) gated: add per-role MAAS tags (control/compute/storage) + + author the 10th controller VM in `opentofu/vr1-dc1-substrate` (apply with edge + egress OPEN); (3) render the 9-node role-separated `bundle.yaml` -- machines + block; RE-HOME the base placements (mysql/ovn/ceph-mon `lxd:8,9,10` -> control + IDs; nova-compute 3->2 -> compute IDs; ceph-osd -> the 4 storage IDs; singleton + utilities -> control IDs); re-scope the dc0-named tokens -- + extend the + validator + wire the `vr1-dc1-vips`/`dc-ha-scaleup` overlays + de-stale phase-4 + Step 4; (4) gated `juju bootstrap` (by controller tag, egress OPEN) -> deploy. + Deploy-time VERIFY-LIVE gates for the runbook: vault-3-unit HA-on-MySQL (D-121), + hacluster cluster_count:3, dc1 VIP validation, CRUSH failure-domain=host. Separately: DC0 full-mirror completes -> reachability gate -> stage close-out. - Position inside Stage 3: deploy step A EXECUTED 2026-07-19 (6/0/6 exact; convergence zero -- diff --git a/docs/audit/committee-20260724-track2-bundle-render.md b/docs/audit/committee-20260724-track2-bundle-render.md new file mode 100644 index 0000000..603126e --- /dev/null +++ b/docs/audit/committee-20260724-track2-bundle-render.md @@ -0,0 +1,194 @@ +# Committee record -- Track 2: vr1-dc1 deploy-artifact (bundle) render + +**Date:** 2026-07-24. **Branch:** `dc-dc-stage4-phase3-maas-deploy` (Stage 4 open). +**Convened by:** operator ("Deploy a committee to review Track 2 and provide +recommendations based on overall end project goals and decisions already +researched"). + +**Status of this document.** ADVISORY / audit record. It ADOPTS nothing: the +Fork-1 recommendation below is PENDING an operator ruling (GA-R5); a decision +exists only when its Status line quotes the question + the operator's exact +utterance in `docs/design-decisions.md`. `docs/CURRENT-STATE.md` remains the +sole status authority (GA-R1). Precedent for the format: +`docs/audit/commissioning-committee-20260720.md`. + +## Method + +Four independent, read-only reviewers, each a distinct lens, each given a +self-contained brief grounded in the end-project goals (MINIMIZE DELTA TO +ROOSEVELT; VR1 exists to make HA real and prove DR drills) and the researched +decisions (D-121, D-104, D-119, D-052, D-062, D-009, D-106, D-108): + +1. Roosevelt-transferability / architectural-delta +2. HA/DR correctness & multi-tenant isolation +3. Operational risk / minimal-delta-now +4. Juju/MAAS/charm mechanism-feasibility + +Every load-bearing claim was re-verified against repo HEAD by the adjudicator +before adoption into this record; two verifications corrected the brief given to +the committee (see "Corrections", below). + +## The blocker being reviewed + +The committed `bundle.yaml` is VR0's 4-machine ("8"-"11") HYPERCONVERGED layout +(`tags=openstack`, single-unit control plane by D-009). VR1 dc1 is 9 +ROLE-SEPARATED nodes (D-121 Option C: 3 control + 2 compute + 4 storage), built +and merged (G12). The deploy ARTIFACTS were never rendered to that architecture. +The substrate is exactly 9 node VMs (no controller slot), all Ready, all sharing +ONE uniform MAAS tag `openstack-vr1-dc1`, zone `default` +(`docs/audit/stage4-discovery-20260723.txt`). + +## Facts verified against the repo (not from prose) + +- **Designate IS in the Stage-5 bundle** -- `bundle.yaml:812` (`designate`), + `:840` (`designate-bind`), `:922` (`designate-hacluster`, noted "D-106 + reactivated 2026-07-10, supersedes D-019"). The scale-up overlay's + `designate:3` references a real deployed app. The phase-4 runbook line 192 + ("bundle ships NO designate") is STALE and is part of the de-stale work. +- **Fork-1 option (a) is additive, not a substrate reopening** -- the inner + caller is `module "vr1_dc1_node" { for_each = local.vr1_dc1_nodes }` + (`opentofu/vr1-dc1-substrate/main.tf:140`), keyed (not `count`). Adding a 10th + node key plans `1 to add, 0 to change, 0 to destroy`; the 9 merged nodes are + untouched. +- **Capacity for (a) is a MEASURED gate, not a quotable figure** -- the signed + doc's `790 GiB / 77%` is the OLD 3+2+3 (8-node) layout + (`docs/design-decisions.md:3451`); the built layout is 3+2+4 and no `~85%` + figure is carried in CURRENT-STATE. `scripts/dc-dc-whole-host-budget.py` must + be RUN for the 10-node x 2-DC shape before the apply (GA-R1: figures from + captured output). +- **H1 defect CONFIRMED** (see below) -- `bundle.yaml` `ceph-rbd-mirror` carries + a duplicate `bindings:` key; `yaml.safe_load` resolves it to an orphan. + +## Fork verdicts + +### FORK 1 -- controller arithmetic: UNANIMOUS (a), a dedicated 10th controller VM + +All four lenses independently reached (a) and rejected (b)/(c): + +- **(b)** bootstrap onto one of the 9 + shrink a role (e.g. ceph-osd 4->3) + REVERSES the ratified D-121 R-3 amendment and deletes the Ceph rebuild + headroom the DR drill exists to demonstrate ("does not degrade the drill, it + deletes the thing being drilled"). +- **(c)** co-locate on a control node: `juju bootstrap` on MAAS consumes a WHOLE + machine (mechanism-confirmed), collapsing 3 control nodes -> 2, breaking the + mysql/ovn/ceph-mon quorum and coupling controller-loss to the single-node + drill. Rejected. +- **(a)** is the only option preserving 3+2+4, matches D-104's VERIFIED VR0 + precedent (a dedicated `juju` VM separate from the OpenStack nodes, + `docs/maas-as-built-reference.md`), and is Roosevelt-shaped. It is additive + (see Facts) and light on capacity (subject to the measured gate). +- **Least-bad co-location, if (a) ever cannot fit** (HA/DR lens): a COMPUTE node + (hosts neither quorum nor OSD headroom), never a control node. + +**Classification: a D-104 amendment** (controller placement is D-104's subject; +substrate-count changes went through amendment for storage 3->4). PENDING +operator ruling. + +### FORK 2 -- render mechanism: adjudicated -- hand-render now, GATED by an extended validator + +Split panel: the Roosevelt / HA-DR / mechanism lenses lean to SOME automation +(the render is a ~56-app cross-consistency rewrite); the operational lens calls a +renderer tool gold-plating and says hand-render now. The dividing line is not +hand-vs-tool -- ALL FOUR require an automated PLACEMENT check, which +`provider-bundle-check.py` does not do today (it validates D-052 bindings + +mysql=3 only). Adjudication: + +- **Hand-render the concrete dc1 `bundle.yaml` now** (a concrete file is directly + reviewable, which the "render AND review" constraint demands; you must render + once to learn the parameter surface anyway). +- **Gate it behind an EXTENDED validator** -- add assertions to the EXISTING + tested `provider-bundle-check.py` (no new tooling): 3-distinct-host + anti-affinity for the 14 scaled apps; `ceph-osd=4` on storage-tagged nodes; + `nova-compute=2` on compute-tagged nodes; controller-tag carries no role tag; + and NO dangling `to:` (every referenced machine ID exists). +- **Defer a general renderer tool** until dc0 forces the second render and proves + real churn. The `{{TOKEN}}` / overlay pattern already carries the per-DC parts. + +### FORK 3 -- role->node binding: UNANIMOUS per-role MAAS tags + +- **per-role tags** `constraints: tags=openstack-vr1-dc1,control` -- MAAS + AND-filters at allocation; deterministic; survives re-enlist; Roosevelt-shaped. + ADOPTED recommendation. Prerequisite: add `control`/`compute`/`storage` tags to + the respective nodes (a small, additive, gated MAAS mutation) AND wire that + application into the re-enlist runbook so a node rebuild re-applies it. +- **`mem=` -- REJECTED**: mechanism-confirmed a FLOOR, not exact-match; a 64 GiB + control node satisfies a `mem=49152` compute request -> silent mis-placement + race under parallel deploy. +- **system-id pinning -- REJECTED**: not cleanly bundle-expressible, and system + IDs are re-minted on every re-enlist (DOCFIX-040); VR1 recovery IS PXE + re-enlist. + +## Confirmed defects (independent of the fork rulings) + +- **H1 / BUNDLEFIX-052 -- `ceph-rbd-mirror` duplicate `bindings:`.** + `bundle.yaml:891-895` is the correct block (`ceph-local`/`ceph-remote` -> + replication, `nrpe-external-master` -> metal-internal; D-108/D-052); an ORPHAN + block at `:907-910` (`certificates`/`dashboard`) is absorbed into + `ceph-rbd-mirror` (YAML takes-last), clobbering the replication-plane bindings + (defeating D-108) and injecting a phantom `dashboard` binding to a charm with + no such endpoint -> `juju deploy` validation reject. Pre-existing; latent + because the app was flagged prep-only. Fix = delete the orphan. FIXED this + session (changelog). +- **Stale phase-4 runbook Step 4** -- "deploy the EXISTING bundle.yaml", "ships + NO designate", and region-naming drift (its "DC1"=vr1-dc0, "DC2"=vr1-dc1). + De-stale as part of the render. + +## The render work, specified (the load-bearing part) + +- **Re-home every base placement** (in `bundle.yaml`, NOT the overlay -- the + highest silent risk): `mysql-innodb-cluster`/`ovn-central`/`ceph-mon` + `to:[lxd:8,9,10]` -> 3 control IDs; `nova-compute` `["9","10","11"]` **3->2** + -> the 2 compute IDs; `ceph-osd` `["8","9","10","11"]` -> the 4 storage IDs; + `memcached`/`glance-simplestreams-sync`/`designate-bind`/`ceph-rbd-mirror` + singletons -> control IDs. Any un-updated `to:` = a dangling reference = + deploy-time validation failure. +- **Machines block**: 9 role-tagged IDs + the 10th controller VM with a DISTINCT + tag (e.g. `juju-controller-vr1-dc1`) carrying NO role tag, so neither + `juju bootstrap --constraints` nor a role constraint can cross-grab. +- **Re-scope the dc0-named tokens** `{{VR1_DC0_CONTROL_0N}}` -> dc1 control IDs; + positional anti-affinity is a 3-entry `to:` list (unit0->entry0), mechanism + confirmed -- token collision silently defeats it, so validate distinctness. +- **Wire overlays** `vr1-dc1-vips.yaml` + `dc-ha-scaleup.yaml` into the deploy; + de-stale phase-4 Step 4. + +## Deploy-time VERIFY-LIVE gates (into the runbook; not render blockers) + +- **Vault 3-unit HA-on-MySQL** -- D-121's own gate: verify the `1.8/stable` charm + truly does leader-election at 3 units BEFORE commit (mechanism lens: genuinely + UNSURE); if not, evaluate Raft -> D-068, NEVER etcd. Plus 3x manual-unseal + burden (OPEN SEC-003). +- **`hacluster cluster_count:3`** actually gates VIP-bind-on-quorum (else + decorative HA). +- **dc1 VIPs** (`10.12.64/68/72`) are outside the checker's hardcoded dc0 bands + -- validate the merged result. +- **CRUSH failure-domain = host** (the rebuild-headroom argument rests on it). + +## Bootstrap ordering (reinforced) + +Bootstrap the 10th VM (by its distinct controller tag) WHILE the DC1 edge egress +is OPEN -- the juju agent stream + juju/juju-db snaps are fetched pre-apt and are +NOT mirrored (D-135 items 2-3 unbuilt). No egress-narrowing (D-135/D-107) until +the controller is up. + +## Open operator decisions (one ruling per exchange, GA-R5) + +1. **FORK 1 (blocking, a D-104 amendment):** committee unanimous for **(a) + dedicated 10th controller VM**, additive, gated on the whole-host budget + script. -- PENDING. +2. **FORK 2:** hand-render + extended validator; defer a renderer tool. -- + PENDING (follows Fork 1). +3. **FORK 3:** per-role MAAS tags + re-enlist wiring. -- PENDING (follows Fork 1). + +## Highest-priority concerns (one per lens) + +- Roosevelt: extend the placement validator before bootstrap, or any render can + silently ship one-unit-per-node-less-ness = the decorative HA D-121 exists to + kill. +- HA/DR: no automated placement gate today; the un-re-homed base placements + (`lxd:8,9,10`) are the most likely silent decorative-HA path. +- Operational: confirm the inner caller is `for_each`-keyed (DONE -- it is) and + constrain bootstrap to a distinct controller tag, or the closed G12 substrate + reopens. +- Mechanism: re-render COMPLETENESS -- only 14 of ~56 apps are placed by the + overlay; a single stale `to:` fails `juju deploy` with no partial success. And + fix H1 first (DONE). diff --git a/docs/audit/stage5-controller-capacity-20260724.txt b/docs/audit/stage5-controller-capacity-20260724.txt new file mode 100644 index 0000000..2bc1940 --- /dev/null +++ b/docs/audit/stage5-controller-capacity-20260724.txt @@ -0,0 +1,37 @@ +### GATE CAPTURE: Fork-1 (a) 10th controller VM -- whole-host RAM budget +### Date 2026-07-24. Controller modeled as +8 GiB / +4 vCPU per-DC containment overhead +### (Model B: the juju controller VM is a peer of the node VMs inside each vvr1-dcN). + +===== BASELINE: built layout 3+2+4, 2 DCs (defaults) ===== +=== dc-dc-whole-host-budget: VR1 nested capacity (D-121 / R-3 / Model B) === + Per-DC node fleet: 104 vCPU / 384 GiB / 2850 GiB + Model B: each containment VM (vvr1-dcN) holds one DC's fleet -> ~400 GiB RAM each + + containment overhead 2 x (4 vCPU / 16 GiB) + Office1 voffice1: 16/32/600; edges: 3 x (2 vCPU / 2 GiB) + + TOTAL allocation : 238 vCPU / 838 GiB / 6300 GiB + MEASURED host : 256 vCPU / 1024 GiB / 10240 GiB + RAM : 838/1024 GiB = 82% <- BINDING resource + vCPU : 238/256 = 93% (overcommits in a sim; not a hard ceiling) + disk : 6300/10240 GiB = 62% (thin-provisioned) + +VERDICT: FIT -- RAM within budget (186 GiB headroom). +WHOLE-HOST: model=B ram=838/1024GiB verdict=FIT headroom=186GiB vcpu=238/256 disk=6300/10240GiB +(exit 0) + +===== +CONTROLLER: +8 GiB/+4 vCPU per-DC containment overhead (x2 DCs) ===== +=== dc-dc-whole-host-budget: VR1 nested capacity (D-121 / R-3 / Model B) === + Per-DC node fleet: 104 vCPU / 384 GiB / 2850 GiB + Model B: each containment VM (vvr1-dcN) holds one DC's fleet -> ~408 GiB RAM each + + containment overhead 2 x (8 vCPU / 24 GiB) + Office1 voffice1: 16/32/600; edges: 3 x (2 vCPU / 2 GiB) + + TOTAL allocation : 246 vCPU / 854 GiB / 6300 GiB + MEASURED host : 256 vCPU / 1024 GiB / 10240 GiB + RAM : 854/1024 GiB = 83% <- BINDING resource + vCPU : 246/256 = 96% (overcommits in a sim; not a hard ceiling) + disk : 6300/10240 GiB = 62% (thin-provisioned) + +VERDICT: FIT -- RAM within budget (170 GiB headroom). +WHOLE-HOST: model=B ram=854/1024GiB verdict=FIT headroom=170GiB vcpu=246/256 disk=6300/10240GiB +(exit 0) diff --git a/docs/changelog-20260724-stage5-committee-fork1.md b/docs/changelog-20260724-stage5-committee-fork1.md new file mode 100644 index 0000000..86cd498 --- /dev/null +++ b/docs/changelog-20260724-stage5-committee-fork1.md @@ -0,0 +1,78 @@ +# Session changelog 2026-07-24 -- Stage-5 bundle-render committee + Fork-1 ruling + +Branch `dc-dc-stage4-phase3-maas-deploy`. Session resumed the DC1 Stage-5 +bundle-rework blocker under a committee review (operator-requested). Status lives +ONLY in `docs/CURRENT-STATE.md` (GA-R1); this changelog is the review surface -- +each item states what / why / revert. No cloud state was mutated this session. + +## Item 1 -- Committee record (Track 2 bundle render) + +**What:** new `docs/audit/committee-20260724-track2-bundle-render.md` -- a 4-lens +(Roosevelt-delta / HA-DR / operational-risk / Juju-MAAS-mechanism) read-only +committee review of the vr1-dc1 deploy-artifact render, with the adjudication and +open operator decisions. Advisory/audit record; adopts nothing (GA-R5). +**Why:** operator: "Deploy a committee to review Track 2 and provide +recommendations based on overall end project goals and decisions already +researched." Durable, citable authority (commissioning-committee precedent). +**Revert:** `git rm docs/audit/committee-20260724-track2-bundle-render.md`. + +## Item 2 -- Capacity gate capture (Fork-1 option a) + +**What:** new `docs/audit/stage5-controller-capacity-20260724.txt` -- captured +`scripts/dc-dc-whole-host-budget.py` runs: baseline built layout 3+2+4 x2 DC = +838/1024 GiB (82%, FIT); +controller (+8 GiB/+4 vCPU per-DC containment overhead) += 854/1024 GiB (83%, FIT, 170 GiB headroom). Read-only calculator; nothing +queried live. +**Why:** GA-R1 -- the Fork-1 (a) capacity claim must be MEASURED, not inferred +(the signed doc's 790/77% is the OLD 8-node layout). Gate = PASS. +**Revert:** `git rm docs/audit/stage5-controller-capacity-20260724.txt`. + +## Item 3 -- D-104 amendment (Fork-1 ruling: dedicated 10th controller VM) + +**What:** appended a D-104 amendment (2026-07-24) in `docs/design-decisions.md` +recording the operator's GA-R5 ruling -- the per-DC Juju controller is a DEDICATED +10th node VM (distinct tag `juju-controller-vr1-dc1`, no role tag), separate from +the 9 Option C role nodes; additive (`for_each`-keyed); capacity gate PASS +(cites item 2); the 9-node D-121 layout is unchanged. +**Why:** Fork 1 was the blocking architectural decision; operator selected +"Dedicated 10th VM (Recommended)" over shrink-a-role and co-locate. Controller +placement is D-104's subject; recorded as its amendment (not a new D-number). +**Revert:** delete the "### D-104 -- AMENDMENT (2026-07-24)" block in +`docs/design-decisions.md` (git range-diff of this commit). + +## Item 4 -- CURRENT-STATE Stage-4 bullet update (GA-R1 / C1) + +**What:** rewrote the Stage-4 bullet's render sub-bullets: committee-adjudicated; +Fork 1 RULED (a) + capacity gate; Fork 2/3 as OPS committee-unanimous +recommendations (gated at execution); BUNDLEFIX-052 fixed; a corrected NEXT list. +Also corrected an earlier inaccurate characterization (dc-ha-scaleup is not "stale +ceph-osd 3" -- it deliberately excludes ceph-osd) and the "consumes 1 of 9" +arithmetic (now a dedicated 10th VM). +**Why:** GA-R1/C1 -- a status-changing commit updates CURRENT-STATE in the SAME +commit (repo-lint L10). Status authority stays single-sourced. +**Revert:** restore the prior Stage-4 bullet text (git range-diff of this commit). + +## Item 5 -- BUNDLEFIX-052: ceph-rbd-mirror duplicate `bindings:` + +**What:** removed an orphaned 4-space-indented `bindings:` block in `bundle.yaml` +(old line ~907) that `yaml.safe_load` take-last had absorbed into `ceph-rbd-mirror`, +clobbering its real `ceph-local`/`ceph-remote` (replication) + `nrpe-external-master` +bindings (D-108/D-052) and injecting a phantom `dashboard` binding (a charm with no +such endpoint) -> `juju deploy` validation reject. +**Why:** confirmed pre-existing latent defect (committee mechanism lens; verified +via `yaml.safe_load`). Fork-independent; fix now so the dc1 render starts clean. +**Validation:** `yaml.safe_load` confirms `ceph-rbd-mirror.bindings` restored to +`{'': metal-admin, ceph-local: replication, ceph-remote: replication, +nrpe-external-master: metal-internal}`; `provider-bundle-check.py bundle.yaml` PASS. +**Revert:** re-insert the orphan block after the vault-hacluster comment (not +advised -- it is the bug). + +## Next (not done this session; gated) + +Commit+push this batch (GA-R5: rulings land before dependent work). Then, each +operator-gated: per-role MAAS tags; author + apply the 10th controller VM +(egress OPEN); render the 9-node `bundle.yaml` (re-home base placements, re-scope +tokens) + extend `provider-bundle-check.py` with placement assertions + wire +overlays + de-stale phase-4 Step 4; gated `juju bootstrap` (by controller tag, +egress OPEN) -> deploy. Deploy-time VERIFY-LIVE gates: vault-3-unit HA-on-MySQL, +hacluster cluster_count:3, dc1 VIPs, CRUSH failure-domain=host. diff --git a/docs/design-decisions.md b/docs/design-decisions.md index 621e94a..bd26269 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -2104,6 +2104,53 @@ **References:** relates to D-071 (routine update cadence / Juju controller patch policy) without amending it; interacts with D-108 (per-DC control-plane backup / restore in the DR drill). +### D-104 -- AMENDMENT (2026-07-24): the per-DC controller is a DEDICATED 10th node VM, not one of the 9 + +**Status:** ADOPTED 2026-07-24 (operator ruling, GA-R5). Question as presented +(Track-2 Fork 1, committee-reviewed +`docs/audit/committee-20260724-track2-bundle-render.md`): "The vr1-dc1 substrate +is exactly 9 nodes (3 control + 2 compute + 4 storage), all needed for D-121 +Option C roles, with no controller slot. D-104 requires a per-DC Juju controller +on an in-DC MAAS machine (VR0 precedent = a dedicated `juju` VM). Where should the +dc1 Juju controller live?" Operator answer, exact selection: **"Dedicated 10th VM +(Recommended)"** (over "Bootstrap onto a node, shrink a role" and "Co-locate on a +control node"). The committee was unanimous for this option on all four lenses. + +**Decision.** D-104's "in-DC MAAS-deployed machine" is instantiated as a +DEDICATED controller VM per DC, SEPARATE from the D-121 Option C role nodes. The +built substrate carried exactly 9 role nodes (a latent gap against D-104); this +amendment closes it by ADDING a 10th node VM to each DC substrate for the Juju +controller. It matches the VERIFIED VR0 precedent (a dedicated `juju` VM separate +from openstack0-3, `docs/maas-as-built-reference.md`) and the Roosevelt end-state +(dedicated controller hardware, scaling to 3-unit HA there). + +- **The 9 OpenStack-role nodes are UNCHANGED** -- D-121 Option C (3 control + 2 + compute + 4 storage) stands; the controller does NOT consume a role node, so the + Ceph rebuild headroom (R-3) and the quorum/anti-affinity spread are preserved. + The rejected alternatives each broke exactly one of those (shrink-a-role reversed + R-3; co-locate collapsed the 3-node control quorum). +- **Additive, not a substrate reopening** -- the inner caller is + `for_each = local.vr1_dc1_nodes` (`opentofu/vr1-dc1-substrate/main.tf:140`), + keyed; adding the 10th key plans `1 add / 0 change / 0 destroy`. +- **Distinct tag, no role tag** -- the controller VM carries its OWN MAAS tag + (`juju-controller-vr1-dc1`) and NONE of the role tags, so `juju bootstrap + --constraints tags=juju-controller-vr1-dc1` targets it deterministically and no + OpenStack `tags=` role constraint can cross-grab it (pairs with the Fork-3 + per-role-tag scheme). +- **Capacity GATE: PASS (measured, not inferred).** `scripts/dc-dc-whole-host-budget.py` + for the 10-node x 2-DC shape (controller modeled as +8 GiB / +4 vCPU per-DC + containment overhead) -> RAM 854/1024 GiB = 83%, FIT, 170 GiB headroom + (`docs/audit/stage5-controller-capacity-20260724.txt`; baseline 3+2+4 = 838 GiB + / 82%). vCPU allocation 96% (overcommits in a sim; not a hard ceiling). +- **Controller sizing** is small/single-unit (VR1 test-only economy, unchanged); + the exact vCPU/mem/disk is a substrate-authoring value, MEASURED into the tfvars + at build time, not fixed here. + +**Roosevelt analog:** a dedicated per-DC controller machine (bare metal), scaling +to 3-unit HA. **Cross-note to D-121:** the Option C node COUNT (9 role nodes/DC) +is not changed by this amendment; the controller is a separate machine, tracked +here under D-104. + --- ## D-105: CAPI management-cluster placement and COS / Magnum scope (VR1)