diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index fd059e7..de799ee 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -442,6 +442,19 @@ (21 items, Phase 3 of the readiness doc) is LOGGED NOT EXECUTED -- nearly every runbook fix interlocks with an unanswered ruling, so landing them now would encode assumptions. The sole mechanical fix taken this session is the G3 row correction above. + **RULINGS IN PROGRESS (operator returned 2026-07-27; ONE exchange each per GA-R5).** + **R1 RULED 2026-07-27 -- exact utterance "Add an OSD volume to node-vm (Recommended)"**, + recorded as a **D-121 AMENDMENT** (`docs/design-decisions.md` is the ruling authority). + The `ceph-osd` data device becomes a REAL second block device on the four storage nodes + per DC -- **8 volumes, not 18**, since `ceph-osd` is placed on machines 5-8 only. D-121's + own capacity re-validation had already budgeted it ("Ceph disk re-run for 4 storage/DC = + PASS 5.31 TiB"): the disk was budgeted and never built. **The APPLY is a SEPARATE + operator-gated step and is NOT authorised by this ruling** -- four preconditions are + recorded in the amendment, including that it deliberately spends the inner roots' + ZERO-DIFF property, and that whether re-commissioning preserves the D-134 statics and + pinned MACs must be verified BEFORE the apply (the 2026-07-20 MAC-regeneration incident + is the precedent). Remaining: R2-R11 blocking, R12-R15 standing, in + `docs/audit/queued-rulings-20260727.md`. - Position inside Stage 3: deploy step A EXECUTED 2026-07-19 (6/0/6 exact; convergence zero -- `docs/audit/outer-plan-20260719-postA-converged.txt`). **Deploy step B diff --git a/docs/audit/queued-rulings-20260727.md b/docs/audit/queued-rulings-20260727.md index 1d47cdf..80fe36d 100644 --- a/docs/audit/queued-rulings-20260727.md +++ b/docs/audit/queued-rulings-20260727.md @@ -53,7 +53,16 @@ - **(c)** Re-shape the storage role (e.g. fewer, larger storage nodes with a dedicated disk each). -OPERATOR UTTERANCE: +OPERATOR UTTERANCE: **"Add an OSD volume to node-vm (Recommended)"** -- RULED +2026-07-27. **R1 IS CLOSED.** Recorded as a D-121 AMENDMENT (the governing decision +already ratifies `modules/node-vm` sizing, and its own capacity re-validation already +budgeted "Ceph disk re-run for 4 storage/DC = PASS 5.31 TiB" -- the disk was budgeted +and never built). Two scope corrections landed with the ruling: it is **8 volumes, not +18** (ceph-osd is placed on the four storage nodes per DC only), and the apply is a +SEPARATE operator-gated step with four preconditions, including verifying that +re-commissioning preserves the D-134 statics and pinned MACs BEFORE the apply -- the +2026-07-20 MAC-regeneration incident is the precedent. Full text: `docs/design-decisions.md`, +"D-121 AMENDMENT 2026-07-27". --- diff --git a/docs/design-decisions.md b/docs/design-decisions.md index f4bbf3f..593330d 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -3597,6 +3597,63 @@ storage/DC = PASS 5.31 TiB. The lost `scratchpad/optc-calc.py` and its "reproducible" changelog claim are retired. +### D-121 AMENDMENT 2026-07-27 -- the ceph-osd data device is a REAL second disk on the storage nodes + +**Status:** RULED 2026-07-27 (operator, GA-R5). Question as presented, verbatim: +"R1 -- Ceph OSD device. `bundle.yaml:560` sets `osd-devices: /dev/vdb`, but all 18 VR1 +nodes have exactly one block device (`vda`), and `modules/node-vm` is hard-coded to a +single disk. How should this be resolved?" Options presented: (a) add an OSD volume to +`modules/node-vm` and re-apply both substrate roots; (b) re-point `osd-devices` at a +directory/partition path on `vda`; (c) re-shape the storage role. Operator selection, +exact utterance: **"Add an OSD volume to node-vm (Recommended)"**. + +**The defect.** Found by the 2026-07-27 Stage-5 grounding audit, INDEPENDENTLY by two +lenses using different instruments: `virsh domblklist` on both racks, and MAAS +`physicalblockdevice_set` on all 18 nodes. Both return exactly ONE device, `vda`, on +every node. `bundle.yaml:560` reads `osd-devices: /dev/vdb # libvirt-attached, +MAAS-untracked, wiped pre-deploy` -- a VR0 as-built comment. VR0's hosts genuinely had +an attached second disk; **the VR1 node-VM module never created one** +(`opentofu/modules/node-vm/main.tf:26` declares a single `libvirt_volume "disk"`, +`:104-125` attaches it at `target.dev = "vda"`, and neither substrate root carries an +OSD-disk variable). Left unfixed the deploy yields ZERO OSDs, `expected-osd-count: 4` +(`bundle.yaml:533`) is never met, and every storage-backed service blocks behind it. + +**This was BUDGETED and never BUILT.** D-121's own capacity re-validation above already +records "Ceph disk re-run for 4 storage/DC = PASS 5.31 TiB". The second disk was +anticipated by the sizing model; only the implementation is missing. That is why this +lands as a D-121 amendment rather than a new D-number (GA-R3: architectural consequence, +but it amends a ruled node layout rather than establishing new ground). + +**SCOPE IS 8 VOLUMES, NOT 18.** `ceph-osd` is placed on the four storage nodes only +(`bundle.yaml:556-557`, `num_units: 4`, `to: ["5","6","7","8"]`), so the change is 4 +volumes per DC. The other five nodes per DC neither run `ceph-osd` nor need a second +device. + +**Rationale for (a) over (b).** MINIMIZE DELTA TO ROOSEVELT: Roosevelt's storage nodes +have real dedicated disks, so a real second block device is what the rehearsal should +exercise. Option (b) was additionally NOT SAFE TO PICK AS PRESENTED -- whether the +`ceph-osd` charm at the pinned `squid/stable` supports a directory- or partition-backed +OSD was explicitly UNRESEARCHED (recorded as UNMEASURED by the audit's lens 5), and the +audit refused to assert it either way. Option (c) would have reopened D-121 Option C's +ruled 3/2/4 shape and its capacity budget. + +**Execution preconditions (the apply is a SEPARATE operator-gated step, not authorised +by this ruling):** +1. It re-opens the substrate. All three OpenTofu roots currently plan ZERO DIFF + (measured 2026-07-27); this deliberately spends that property on the two INNER roots. + Re-converge both to zero diff after the apply and capture it. +2. Rack disk headroom MEASURED 2026-07-27 and sufficient: dc0 `/var/lib/libvirt` 2.0T + available (2.9T total, 34% used -- the 949G full mirror is the bulk), dc1 2.9T + available (1% used). Thin-provisioned qcow2, so creation cost is small. +3. The nodes are powered off in `Ready` under the READY-handoff ruling (DOCFIX-200), so + attaching a device does not disturb a running guest -- but MAAS must RE-COMMISSION the + four storage nodes per DC afterwards for the new device to appear in its inventory, + and the audit did NOT verify whether re-commissioning preserves the D-134 statics and + the pinned MACs. **Verify that before the apply, not after** -- the 2026-07-20 + MAC-regeneration incident is the precedent for exactly this class. +4. `modules/node-vm` is SHARED by both DCs. Gate the change so it is additive and + opt-in per node role, or it will plan a change against all 18 domains. + ## D-122: VR1 site shape -- nested-per-site containment, dark fiber + dedicated per-site L3 ISP, DC edge follows the Office1 pattern [ARCH] **Status:** ADOPTED (2026-07-15, operator ruling this session -- answers Stage-3 Ruling 2). Records