diff --git a/docs/changelog-20260728-vip-arity-gate.md b/docs/changelog-20260728-vip-arity-gate.md index f6adea6..ae56587 100644 --- a/docs/changelog-20260728-vip-arity-gate.md +++ b/docs/changelog-20260728-vip-arity-gate.md @@ -795,3 +795,80 @@ **Evidence.** Gauntlet **ALL GREEN (87) on vcloud** (was 86); repo-lint 0 fail / 614 files. `preflight` verdict unchanged in kind (2 fatal: the absent octavia-pki secret and MAAS unreachable from vcloud). + +--- + +# 2026-07-29 -- the two Stage-5 blockers AUTHORED (not applied), and the Octavia question RESOLVED + +## 31. Octavia overlay-merge: RESOLVED, and it is a clean negative + +The chain audit flagged that `overlays/octavia-pki.yaml` lands LAST and writes the same +`applications.octavia.options` map as the vips overlay -- so under REPLACE semantics +octavia would deploy with no `vip` and no `prefer-ipv6`, and every offline gate would stay +green because `provider-bundle-check` deep-merges and cannot discriminate. + +**Researched against source, not docs prose (the docs are silent on merge mechanics).** +juju 3.6 pins `github.com/juju/charm/v12 v12.1.1` (verified in juju's `go.mod`). +`mergeStructs` (`overlay.go:661-693`) does, for a map-kind field: +`for _, srcKey := range srcVal.MapKeys() { dstMap.SetMapIndex(srcKey, srcMapVal) }` -- it +KEEPS the destination map and writes into it per key. `ApplicationSpec.Options` is +`map[string]interface{}`, so it takes that branch. **DEEP-MERGE: both overlays' keys +survive; octavia ends up with all seven.** The 2.9 and 3.6 versions of that file are +byte-identical in the merge path, so there is no version risk. + +**The checker's own "MIRRORS juju's documented merge" comment is therefore CORRECT** -- +previously an unverified claim, now sourced. + +**ONE REAL TRAP FOUND AND LOGGED: LP #2002371** (Triaged/High, unfixed) -- an overlay whose +`options:` key is EMPTY/NULL does not merge, it WIPES the base map +(`isZero(srcVal) -> reflect.MakeMap`). Also a whole-app empty body deletes the app. Our +overlays do not have that shape, but nothing gates against acquiring it. + +## 32. R1 AUTHORED: the ceph-osd data device, opt-in per role + +`modules/node-vm` gains `osd_disk_size_bytes` (default **0** = creates nothing). The volume +is `count`-gated and the disk entry is appended via `concat`, so **a node that does not opt +in has no extra resource in state and no plan diff** -- that is precondition 4 of the +amendment verbatim ("gate the change so it is additive and opt-in per node role, or it will +plan a change against all 18 domains"). + +Both substrate roots: the FOUR storage nodes per DC set `osd_gib = 500`. Scope is **8 +volumes, not 18**, exactly as the amendment scoped it. + +**500 GiB is the budgeted figure, not a value I chose** -- D-121's Option-C capacity +re-check ran at "4x500Gi/DC = PASS (5.31 TiB margin)". **FLAGGED: D-121 also records that +the per-node OSD footprint is an ASSUMPTION whose ratified value "pends the OSD-sizing +decision and should be re-derived from the measured DC0 footprint."** So the size is +evidenced but not separately ruled. + +**HEADROOM CAVEAT worth surfacing before the apply:** precondition 2 measured dc0 +`/var/lib/libvirt` at **2.0T available**, and 4 x 500 GiB is 2.0 TiB nominal. qcow2 is thin +so creation costs almost nothing, but dc0 has no room for those OSDs to actually fill. dc1 +has 2.9T available. Not a blocker for the apply; a real one for sustained use. + +## 33. D-104 AUTHORED: the dedicated per-DC Juju controller VM + +A 10th key in each root's node map, `-juju-01`. `for_each` is keyed, so it plans +**1 add / 0 change / 0 destroy** -- additive, not a substrate reopening. The nine role +nodes are untouched, so Option C's 3/2/4 shape, the R-3 Ceph rebuild headroom and the +control-quorum spread all stand. + +**Sizing is the gate's own model, not an invention:** 8 GiB / 4 vCPU are what +`dc-dc-whole-host-budget.py` modelled for the PASS (854/1024 GiB = 83 pct, FIT, 170 GiB +headroom). Disk 100 GiB is the authoring value the amendment explicitly left open. + +**MACs deliberately EMPTY.** `modules/node-vm` documents that as correct ONLY before a node +is enlisted anywhere -- which is this VM's state. They must be PINNED FROM MEASUREMENT +after its first apply, exactly as the nine role nodes were; leaving them unpinned past +enlistment is the 2026-07-21 MAC-regeneration trap that stranded the fleet. + +## 34. NEITHER IS APPLIED, and precondition 3 is NOT satisfied + +`opentofu-validate` PASS on both roots. **No apply has been run.** Both are live substrate +mutations and are individually operator-gated. + +**R1 precondition 3 remains OPEN and I have not closed it:** the amendment requires +verifying, BEFORE the apply, that re-commissioning the storage nodes preserves the D-134 +statics and the pinned MACs -- and it names the 2026-07-20 MAC-regeneration incident as the +precedent for exactly this class. That verification needs live MAAS, which is absent on +this host. **I am not treating it as satisfied.** diff --git a/opentofu/modules/node-vm/main.tf b/opentofu/modules/node-vm/main.tf index 566538c..d61ddd4 100644 --- a/opentofu/modules/node-vm/main.tf +++ b/opentofu/modules/node-vm/main.tf @@ -23,6 +23,24 @@ # wrong, `tofu plan` will simply reject the attribute name, not silently # misconfigure anything. +# The OSD data device (R1 / D-121 amendment). count=0 by default, so a node that does +# not opt in has NO extra resource in state and NO plan diff -- which is what makes this +# safe to land on a module shared by all 18 domains. +resource "libvirt_volume" "osd" { + count = var.osd_disk_size_bytes > 0 ? 1 : 0 + name = "${var.vm_name}-osd.qcow2" + pool = var.pool_name + capacity = var.osd_disk_size_bytes + + # Same nested-target form as the boot disk below -- `format` is NOT a top-level + # attribute of libvirt_volume (DOCFIX-194). + target = { + format = { + type = "qcow2" + } + } +} + resource "libvirt_volume" "disk" { name = "${var.vm_name}-disk.qcow2" pool = var.pool_name @@ -102,7 +120,7 @@ } devices = { - disks = [ + disks = concat([ { source = { volume = { @@ -123,7 +141,30 @@ order = 2 } } - ] + ], + # R1: the OSD data device, present ONLY for a node that opted in. An empty list + # for every other node means `concat` yields exactly the single-disk list this + # module produced before the variable existed -- byte-identical, zero plan diff. + var.osd_disk_size_bytes > 0 ? [ + { + source = { + volume = { + pool = libvirt_volume.osd[0].pool + volume = libvirt_volume.osd[0].name + } + } + target = { + dev = "vdb" + bus = "virtio" + } + driver = { + type = "qcow2" + } + # No boot entry: this is a DATA device. ceph-osd consumes it as + # /dev/vdb per bundle.yaml; it must never be a boot candidate. + } + ] : [] + ) interfaces = [ for i, net_name in var.network_names : { diff --git a/opentofu/modules/node-vm/variables.tf b/opentofu/modules/node-vm/variables.tf index 6d7e2d6..1c5ba43 100644 --- a/opentofu/modules/node-vm/variables.tf +++ b/opentofu/modules/node-vm/variables.tf @@ -86,3 +86,28 @@ type = string default = "/var/lib/libvirt/vr1/staging" } + +variable "osd_disk_size_bytes" { + description = <<-EOT + OPT-IN second block device, attached as vdb. 0 (the default) creates NOTHING and + leaves the domain byte-identical to before this variable existed. + + R1 (D-121 AMENDMENT, RULED 2026-07-27, operator: "Add an OSD volume to node-vm + (Recommended)"): `bundle.yaml` sets `osd-devices: /dev/vdb`, but every VR1 node was + built with exactly one device. Measured independently by two instruments -- `virsh + domblklist` on both racks and MAAS `physicalblockdevice_set` on all 18 nodes. Left + unfixed the deploy yields ZERO OSDs and `expected-osd-count: 4` is never met. + + SCOPE IS 8 VOLUMES, NOT 18: ceph-osd is placed on the four STORAGE nodes per DC only, + so only those callers set this. That is precondition 4 of the amendment -- this + module is SHARED by both DCs, so the change had to be opt-in per role or it would + plan a change against all 18 domains. + EOT + type = number + default = 0 + + validation { + condition = var.osd_disk_size_bytes >= 0 + error_message = "osd_disk_size_bytes must be 0 (no OSD disk) or a positive byte count." + } +} diff --git a/opentofu/vr1-dc0-substrate/main.tf b/opentofu/vr1-dc0-substrate/main.tf index a63ffda..d4ce1cc 100644 --- a/opentofu/vr1-dc0-substrate/main.tf +++ b/opentofu/vr1-dc0-substrate/main.tf @@ -114,22 +114,43 @@ "52:54:00:18:ab:b4", "52:54:00:50:48:88", "52:54:00:78:fb:c5", "52:54:00:cc:84:61", "52:54:00:e4:ab:df", "52:54:00:5b:93:c4", ] } - "vr1-dc0-storage-01" = { vcpu = 8, mem = 24576, disk_gib = 550, macs = [ + "vr1-dc0-storage-01" = { vcpu = 8, mem = 24576, disk_gib = 550, osd_gib = 500, macs = [ "52:54:00:5f:8d:42", "52:54:00:4c:69:7b", "52:54:00:ac:2c:4d", "52:54:00:28:91:d2", "52:54:00:fa:7c:53", "52:54:00:0c:7a:ab", ] } - "vr1-dc0-storage-02" = { vcpu = 8, mem = 24576, disk_gib = 550, macs = [ + "vr1-dc0-storage-02" = { vcpu = 8, mem = 24576, disk_gib = 550, osd_gib = 500, macs = [ "52:54:00:48:86:2c", "52:54:00:2e:8b:55", "52:54:00:d7:4e:38", "52:54:00:05:60:af", "52:54:00:c6:02:bd", "52:54:00:2c:f8:42", ] } - "vr1-dc0-storage-03" = { vcpu = 8, mem = 24576, disk_gib = 550, macs = [ + "vr1-dc0-storage-03" = { vcpu = 8, mem = 24576, disk_gib = 550, osd_gib = 500, macs = [ "52:54:00:b1:94:d1", "52:54:00:bd:d1:24", "52:54:00:80:67:87", "52:54:00:0f:a6:35", "52:54:00:e0:73:ed", "52:54:00:62:65:77", ] } - "vr1-dc0-storage-04" = { vcpu = 8, mem = 24576, disk_gib = 550, macs = [ # R-3: 4th OSD host + "vr1-dc0-storage-04" = { vcpu = 8, mem = 24576, disk_gib = 550, osd_gib = 500, macs = [ # R-3: 4th OSD host "52:54:00:2b:ed:ab", "52:54:00:90:ca:d0", "52:54:00:2a:ac:14", "52:54:00:af:34:fd", "52:54:00:28:b1:6d", "52:54:00:ab:5c:50", ] } + # ---- D-104 AMENDMENT (RULED 2026-07-24): the DEDICATED per-DC Juju controller. + # NOT a D-121 role node -- the 9 OpenStack nodes above are unchanged, so Option C's + # 3/2/4 shape, the R-3 Ceph rebuild headroom and the control-quorum spread all stand. + # `for_each` is keyed, so this plans 1 add / 0 change / 0 destroy. + # + # SIZING: 8 GiB / 4 vCPU are the values the capacity GATE actually modelled + # (`scripts/dc-dc-whole-host-budget.py`, 10-node x 2-DC -> RAM 854/1024 = 83 pct, FIT, + # 170 GiB headroom; capture docs/audit/stage5-controller-capacity-20260724.txt). + # Disk is the authoring value the amendment left open; 100 GiB matches the smallest + # existing node class. + # + # MACs DELIBERATELY EMPTY. modules/node-vm documents that as correct ONLY before a + # node is enlisted anywhere -- which is this VM's state. PIN THEM FROM MEASUREMENT + # after its first apply, exactly as the nine role nodes were (2026-07-21); leaving + # them unpinned past enlistment is the MAC-regeneration trap that stranded the fleet. + # + # Its MAAS tag is `juju-controller-vr1-dc0` and it carries NO role tag, so + # `juju bootstrap --constraints tags=juju-controller-vr1-dc0` targets it + # deterministically and no OpenStack role constraint can cross-grab it. + "vr1-dc0-juju-01" = { vcpu = 4, mem = 8192, disk_gib = 100, macs = [] } + } } @@ -142,7 +163,10 @@ vcpu = each.value.vcpu memory_mib = each.value.mem disk_size_bytes = each.value.disk_gib * 1024 * 1024 * 1024 - pool_name = module.inner_storage.pool_name - network_names = local.vr1_dc0_node_nics - interface_macs = each.value.macs # pinned -- see the vr1_dc0_nodes comment + # R1 (D-121 amendment): opt-in OSD data device, storage nodes only. + # `try(..., 0)` keeps every other node's plan byte-identical. + osd_disk_size_bytes = try(each.value.osd_gib, 0) * 1024 * 1024 * 1024 + pool_name = module.inner_storage.pool_name + network_names = local.vr1_dc0_node_nics + interface_macs = each.value.macs # pinned -- see the vr1_dc0_nodes comment } diff --git a/opentofu/vr1-dc1-substrate/main.tf b/opentofu/vr1-dc1-substrate/main.tf index f3a8c68..897148c 100644 --- a/opentofu/vr1-dc1-substrate/main.tf +++ b/opentofu/vr1-dc1-substrate/main.tf @@ -117,22 +117,43 @@ "52:54:01:d1:05:01", "52:54:01:d1:05:02", "52:54:01:d1:05:03", "52:54:01:d1:05:04", "52:54:01:d1:05:05", "52:54:01:d1:05:06", ] } - "vr1-dc1-storage-01" = { vcpu = 8, mem = 24576, disk_gib = 550, macs = [ + "vr1-dc1-storage-01" = { vcpu = 8, mem = 24576, disk_gib = 550, osd_gib = 500, macs = [ "52:54:01:d1:06:01", "52:54:01:d1:06:02", "52:54:01:d1:06:03", "52:54:01:d1:06:04", "52:54:01:d1:06:05", "52:54:01:d1:06:06", ] } - "vr1-dc1-storage-02" = { vcpu = 8, mem = 24576, disk_gib = 550, macs = [ + "vr1-dc1-storage-02" = { vcpu = 8, mem = 24576, disk_gib = 550, osd_gib = 500, macs = [ "52:54:01:d1:07:01", "52:54:01:d1:07:02", "52:54:01:d1:07:03", "52:54:01:d1:07:04", "52:54:01:d1:07:05", "52:54:01:d1:07:06", ] } - "vr1-dc1-storage-03" = { vcpu = 8, mem = 24576, disk_gib = 550, macs = [ + "vr1-dc1-storage-03" = { vcpu = 8, mem = 24576, disk_gib = 550, osd_gib = 500, macs = [ "52:54:01:d1:08:01", "52:54:01:d1:08:02", "52:54:01:d1:08:03", "52:54:01:d1:08:04", "52:54:01:d1:08:05", "52:54:01:d1:08:06", ] } - "vr1-dc1-storage-04" = { vcpu = 8, mem = 24576, disk_gib = 550, macs = [ # Option C: 4th OSD host + "vr1-dc1-storage-04" = { vcpu = 8, mem = 24576, disk_gib = 550, osd_gib = 500, macs = [ # Option C: 4th OSD host "52:54:01:d1:09:01", "52:54:01:d1:09:02", "52:54:01:d1:09:03", "52:54:01:d1:09:04", "52:54:01:d1:09:05", "52:54:01:d1:09:06", ] } + # ---- D-104 AMENDMENT (RULED 2026-07-24): the DEDICATED per-DC Juju controller. + # NOT a D-121 role node -- the 9 OpenStack nodes above are unchanged, so Option C's + # 3/2/4 shape, the R-3 Ceph rebuild headroom and the control-quorum spread all stand. + # `for_each` is keyed, so this plans 1 add / 0 change / 0 destroy. + # + # SIZING: 8 GiB / 4 vCPU are the values the capacity GATE actually modelled + # (`scripts/dc-dc-whole-host-budget.py`, 10-node x 2-DC -> RAM 854/1024 = 83 pct, FIT, + # 170 GiB headroom; capture docs/audit/stage5-controller-capacity-20260724.txt). + # Disk is the authoring value the amendment left open; 100 GiB matches the smallest + # existing node class. + # + # MACs DELIBERATELY EMPTY. modules/node-vm documents that as correct ONLY before a + # node is enlisted anywhere -- which is this VM's state. PIN THEM FROM MEASUREMENT + # after its first apply, exactly as the nine role nodes were (2026-07-21); leaving + # them unpinned past enlistment is the MAC-regeneration trap that stranded the fleet. + # + # Its MAAS tag is `juju-controller-vr1-dc1` and it carries NO role tag, so + # `juju bootstrap --constraints tags=juju-controller-vr1-dc1` targets it + # deterministically and no OpenStack role constraint can cross-grab it. + "vr1-dc1-juju-01" = { vcpu = 4, mem = 8192, disk_gib = 100, macs = [] } + } } @@ -145,7 +166,10 @@ vcpu = each.value.vcpu memory_mib = each.value.mem disk_size_bytes = each.value.disk_gib * 1024 * 1024 * 1024 - pool_name = module.inner_storage.pool_name - network_names = local.vr1_dc1_node_nics - interface_macs = each.value.macs # pinned FROM THE FIRST APPLY -- see the scheme comment + # R1 (D-121 amendment): opt-in OSD data device, storage nodes only. + # `try(..., 0)` keeps every other node's plan byte-identical. + osd_disk_size_bytes = try(each.value.osd_gib, 0) * 1024 * 1024 * 1024 + pool_name = module.inner_storage.pool_name + network_names = local.vr1_dc1_node_nics + interface_macs = each.value.macs # pinned FROM THE FIRST APPLY -- see the scheme comment }