diff --git a/bundle.yaml b/bundle.yaml index e5e0668..91a5040 100644 --- a/bundle.yaml +++ b/bundle.yaml @@ -12,10 +12,14 @@ # catalog endpoints (public 10.12.4.N / internal+admin 10.12.8.N). Vault issues # per-VIP IP-SAN certs. No control-plane DNS dependency. (B5) # VR1 DC-DC NOTE (2026-07-10, DOCFIX-167): D-106 reactivates os-public-hostname -# PER-DC via a separate overlay (overlays/${DC}-hostnames.yaml, proposed in -# runbooks/dc-dc-phase6-designate-cos-magnum.md) once Designate exists -- this base -# bundle stays DC-agnostic; B5's IP-only posture is this file's own DEFAULT, only -# overridden per-DC via that overlay, not edited here. +# PER-DC via a separate overlay, PROPOSED (Step 1.4 of +# runbooks/dc-dc-phase6-designate-cos-magnum.md) and NOT AUTHORED -- no such +# file exists in overlays/ under any spelling, so never pass one to --overlay +# (juju errors out on a missing overlay file). Until it is authored and +# reviewed, B5's IP-only posture below is the LIVE posture, not a placeholder: +# os-public-hostname is set nowhere, which is why the designate applications +# this bundle carries deploy INERT (D-106 ruling note 2026-07-27). This base +# bundle stays DC-agnostic; the reactivation is overlay work, never edited here. # HA chain: hacluster subordinates + :ha relations ACTIVE for 12 charms (10 v1 API charms + # ceph-radosgw, un-deferred, + designate, reactivated 2026-07-10 per D-106) -- of # these, 11 carry the dual-VIP pattern (designate uses its own dnsaas/DNS-native @@ -37,12 +41,21 @@ # ceph-*-network config, so the LXD-contained mon actually gets a storage NIC. # Clients bind ceph->storage; container principals carry it too (subset rule). (C2) # Magnum: Layer A only -- CAPI driver graft is Layer B (runbooks/phase-06..08) -# Octavia: lb-mgmt PKI options supplied via overlays/octavia-pki.yaml (gitignored). +# Octavia: lb-mgmt PKI options supplied via overlays/${DC}-octavia-pki.yaml (gitignored). # Amphora-pipeline options baked (use-internal-endpoints etc.). (B4) -# OVN tunnels: geneve overlay on the DATA space (10.12.12.0/22) -- ovn-chassis + ovn-chassis-octavia +# OVN tunnels: geneve overlay on the data-tenant space (10.12.16.0/22) -- ovn-chassis + ovn-chassis-octavia # 'data' binding; their principals also carry data (nova-compute:neutron-plugin bare-metal, # octavia:ovsdb-cms provisions the container NIC) per the subset rule. Prereq: enp8s0 -# link-subnet to 10.12.12.4N (rebuild-prep, machines Ready). +# link-subnet to 10.12.16.4N (rebuild-prep, machines Ready). +# CIDR CORRECTED 2026-07-29 (NEW-1): this block read 10.12.12.0/22 and +# 10.12.12.4N, which is metal-internal, NOT the geneve plane. MEASURED: +# scripts/lib-net.sh:26-27 maps 10.12.12.0/22 -> metal-internal and +# 10.12.16.0/22 -> data-tenant, and docs/maas-as-built-reference.md:27/137 +# records data-tenant 10.12.16.0/22 on raw enp8s0 (10.12.16.N) as the +# geneve carrier. The two planes carry OPPOSITE ruled address families +# under D-101, so this was never cosmetic. The 'data' BINDINGS below +# always named the space (data-tenant) and were never wrong -- only this +# header's CIDR gloss was. # Resources: omitted -- let charms use latest available resource revisions # ============================================================ @@ -447,22 +460,34 @@ ovsdb-server: metal-internal constraints: arch=amd64 - # ovn-chassis: subordinate to nova-compute. MAC-based bridge-interface-mappings captured from - # MAAS 2026-05-22 (Bobcat used hardcoded 'enp1s0' -- anti-pattern fix). The charm picks whichever - # MAC is found locally per unit; non-matching MACs are ignored. + # ovn-chassis: subordinate to nova-compute. MAC-based bridge-interface-mappings (Bobcat used + # a hardcoded 'enp1s0' -- anti-pattern fix). The charm picks whichever MAC is found locally + # per unit; non-matching MACs are ignored. + # + # `bridge-interface-mappings` IS NOT IN THIS FILE, and must never be re-added here. + # MOVED OUT 2026-07-29 (item 3.8). It is a PER-DC value: this base bundle is documented as + # vr1-dc0's source of truth, but it carried vr1-DC1's compute provider MACs + # (br-ex:52:54:01:d1:04:02 / :05:02), a residue of the 2026-07-24 dc1-first render. Because + # the charm SILENTLY ignores every non-local MAC, a dc0 deploy matched nothing, ovn-chassis + # built NO br-ex mapping, provider egress was dead on dc0 compute, and NO GATE FAILED. The + # governing text is the D-124 coupling entry's 2026-07-27 correction in + # docs/design-decisions.md: "ovn-chassis bridge-interface-mappings is a PER-DC value and + # must become renderer OUTPUT into the per-DC overlay ... It must also move OUT of the base + # bundle for the 2026-07-25 symmetric-overlay ruling to be truthful." + # + # It now arrives via overlays/vr1-dc0-machines.yaml and overlays/vr1-dc1-machines.yaml -- + # the same `overlays/${DC}-machines.yaml` slot preflight.sh's P2 already assembles, and the + # hand-maintained half of the per-DC overlay pair (the *-vips.yaml half is GENERATED by + # scripts/render-dc-overlays.py and would clobber a hand-added key on the next render). + # CAUTION: provider-bundle-check only COUNTS well-formed MACs and skips the key entirely + # when absent, so it cannot tell you this is missing or that it is the wrong DC's set -- + # deploy the base WITHOUT its per-DC machines overlay and you get silent dead egress. ovn-chassis: charm: ovn-chassis channel: 24.03/stable options: ovn-bridge-mappings: physnet1:br-ex prefer-chassis-as-gw: true # B2 -- elects gateway chassis so tenant routers get external egress - # VR1 dc1: the 2 compute nodes' provider-public (enp2s0) MACs = 52:54:01:d1:NN:02, - # NN = node ordinal (compute-01=04, compute-02=05), pinned in opentofu/vr1-dc1-substrate. - # The charm picks whichever MAC is local per unit; ovn-chassis is subordinate to the - # 2-unit nova-compute, so only the compute hosts carry br-ex. - bridge-interface-mappings: >- - br-ex:52:54:01:d1:04:02 - br-ex:52:54:01:d1:05:02 bindings: '': metal-admin amqp: metal-internal @@ -646,12 +671,19 @@ openstack-origin: *openstack-origin amp-image-tag: octavia-amphora # B4 -- MUST match the tag octavia-diskimage-retrofit stamps # ----- PKI material ------------------------------------------------- - # 5 lb-mgmt-* options are supplied via overlays/octavia-pki.yaml - # (gitignored). Generated per runbooks/01a-octavia-pki-generation.md. - # Deploy with: + # 5 lb-mgmt-* options are supplied via overlays/${DC}-octavia-pki.yaml + # (gitignored, PER-DC under the D-109 amendment; .gitignore:46 covers it by + # glob). Generated per runbooks/01a-octavia-pki-generation.md. + # Deploy with (the per-DC set preflight.sh P2 also validates): # juju deploy ./bundle.yaml \ - # --overlay overlays/vr0-dc0-testcloud.yaml \ - # --overlay overlays/octavia-pki.yaml + # --overlay "overlays/${DC}-vips.yaml" \ + # --overlay "overlays/${DC}-machines.yaml" \ + # --overlay "overlays/${DC}-octavia-pki.yaml" + # CORRECTED 2026-07-29 (NEW-2): this block named + # `overlays/vr0-dc0-testcloud.yaml`, a file that has NEVER existed in this + # repo -- juju errors out on a missing overlay. Ruling 3 (2026-07-25) made + # this bundle VIP-FREE and symmetric per-DC, so the per-DC vips + machines + # overlays are what actually belong here. bindings: '': metal-admin amqp: metal-internal diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 79e4c48..55d57dc 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1006,6 +1006,28 @@ would silently produce a cloud whose compute nodes have no provider egress. It is also a per-DC value that is NOT a NetBox assignment, so it must move into the per-DC overlays for the ruled symmetric shape to be truthful. Logged, NOT fixed (hard rule 1). + **3.8 IS NOW FIXED (2026-07-29).** `bridge-interface-mappings` is OUT of `bundle.yaml` and + per-DC in `overlays/vr1-dc0-machines.yaml` (NEW) and `vr1-dc1-machines.yaml`. The MACHINES + overlay was chosen over the vips overlay for a measured reason: both `*-vips.yaml` files are + marked GENERATED / do-not-hand-edit, so a key added there is clobbered on the next render, + and `preflight.sh` already assembles `overlays/${DC}-machines.yaml`. Creating the dc0 file + also closes NEW-3 from the other side -- preflight's `[ -f ]` guard was silently skipping a + file the deploy needs. + **AND THE FIX EXPOSED A WORSE DEFECT THAN THE ONE IT REPAIRED: TWO REPO RECORDS NAME THE + WRONG NIC.** dc0's compute provider-public MACs are `52:54:00:8c:2a:8c` / + `52:54:00:50:48:88` -- position **[1]** in each node's `macs` list. Audit register row **U5** + and the D-124 correction both name the **[0]** values (`52:54:00:1b:19:e6` / + `52:54:00:18:ab:b4`), which are the **PXE/boot** MACs, and the D-124 correction additionally + instructs a reader to source them "from `lib-hosts.sh`" -- **verified here independently: + `lib-hosts.sh` declares only `HOST_BOOT_MAC` and therefore CANNOT supply a provider MAC at + all.** Following either record would have bridged `br-ex` onto the boot NIC: not a missing + mapping that fails loudly, but a WRONG mapping that comes up and misbehaves. Confirmed three + independent ways -- `opentofu/vr1-dc0-substrate/main.tf:109-116` list order, the macpin plan + capture, and live MAAS showing `br-ex` over `enp2s0`. The correct values and the [0]-vs-[1] + trap are now written into the overlay itself, next to the data. + **ALSO LOGGED, NOT FIXED:** `provider-bundle-check.py` treats an ABSENT + `bridge-interface-mappings` as a silent skip and never checks the MAC prefix against `--dc`, + so it could not have failed on either variant of this defect. **HOME FOR THE DEFERRED NOC POSTURE RULED 2026-07-27 (GA-R5). Question as presented:** does it get its own D-number, or fold into D-100 -- noting GA-R3's A1 amendment ("Mentioning Roosevelt, or recording a Roosevelt-era preference, does not qualify; changing what gets built diff --git a/overlays/dc-ha-scaleup.yaml b/overlays/dc-ha-scaleup.yaml index 18ad8c8..c99a0af 100644 --- a/overlays/dc-ha-scaleup.yaml +++ b/overlays/dc-ha-scaleup.yaml @@ -4,13 +4,42 @@ # decorative single-unit control plane REAL by scaling 14 services from # num_units: 1 to 3. VR1-SPECIFIC: VR0/testcloud stays single-unit BY DESIGN # (D-009), so this is an overlay, NOT a bundle.yaml edit -- same pattern as -# overlays/dc-dc-ipv6-family-matrix.yaml. Apply ALONGSIDE bundle.yaml's other -# overlays, e.g.: -# juju deploy ./bundle.yaml \ -# --overlay overlays/octavia-pki.yaml \ -# --overlay "overlays/${DC}-hostnames.yaml" \ -# --overlay overlays/dc-ha-scaleup.yaml \ -# -m "${DC_MODEL}" +# overlays/dc-dc-ipv6-family-matrix.yaml. +# +# APPLIED SEPARATELY, AFTER the per-DC set -- NOT in the same command as the VIP +# overlay. R6 (RULED 2026-07-27, docs/design-decisions.md): operator utterance "Close +# the two VIP gaps first, then apply the overlay whole", and the ruling's own sequencing +# line, "R11 is therefore a hard Stage-5 precondition, sequenced BEFORE the overlay, not +# a parallel item". The two gaps are vault and designate; R11 landed their VIPs (.61 / +# .62) in the per-DC vips overlays. Merged into ONE command the scale-up and the VIPs +# arrive together and the ruled ordering is not observable, which is exactly the shape +# R6 refused (option (b): a 3-node pacemaker cluster for the CA with no VIP to manage). +# +# Step 1 -- deploy the per-DC set, let it settle: +# juju deploy ./bundle.yaml \ +# --overlay "overlays/${DC}-vips.yaml" \ +# --overlay "overlays/${DC}-machines.yaml" \ +# --overlay "overlays/${DC}-octavia-pki.yaml" \ +# -m "${DC_MODEL}" +# Step 2 -- only then, this overlay, whole: +# juju deploy ./bundle.yaml \ +# --overlay "overlays/${DC}-vips.yaml" \ +# --overlay "overlays/${DC}-machines.yaml" \ +# --overlay "overlays/${DC}-octavia-pki.yaml" \ +# --overlay overlays/dc-ha-scaleup.yaml \ +# -m "${DC_MODEL}" +# +# FILENAMES CORRECTED 2026-07-29 (NEW-4 / NEW-6). The command above previously named +# two files that cannot be passed as written: +# - `overlays/octavia-pki.yaml` -- RENAMED PER-DC under the D-109 amendment to +# `overlays/${DC}-octavia-pki.yaml` (.gitignore:41-46 documents the rename and +# covers it by GLOB precisely so each per-DC file stays ignored). +# - `overlays/${DC}-hostnames.yaml` -- HAS NEVER BEEN AUTHORED. It is proposed D-106 +# hostname work (bundle.yaml:13-18 records it as proposed, not built) and +# `os-public-hostname` remains unset everywhere; bundle.yaml's B5 IP-ONLY posture +# stands. juju errors out on a missing overlay file, so a phantom name inside a +# runnable command is a live failure, not a documentation nit. Removed rather than +# renamed -- there is nothing to point it at. # # NOT YET APPLIED to any live model -- authored prep-only (STAGE 5 prep; the # per-DC bundle deploy is two stages out). No live cloud reached this session. diff --git a/overlays/vr1-dc0-machines.yaml b/overlays/vr1-dc0-machines.yaml new file mode 100644 index 0000000..a502712 --- /dev/null +++ b/overlays/vr1-dc0-machines.yaml @@ -0,0 +1,105 @@ +# overlays/vr1-dc0-machines.yaml +# +# The vr1-dc0 half of the hand-maintained per-DC overlay pair. Created 2026-07-29 +# (item 3.8). Its sibling is overlays/vr1-dc1-machines.yaml. +# +# WHY THIS FILE EXISTS AT ALL, given that bundle.yaml's `machines:` block already +# carries dc0's tags: `overlays/${DC}-machines.yaml` is a path that TWO consumers +# already name -- scripts/preflight.sh's P2 overlay assembly and +# runbooks/dc-dc-phase6-designate-cos-magnum.md. Both guard it with `[ -f ]` or the +# equivalent, so while the dc0 file did not exist they SILENTLY SKIPPED it: P2 went +# green having validated a different input than the deploy would use. Creating the +# file closes that hole from the artifact side rather than by adding another guard. +# +# THIS OVERLAY IS `applications:`-ONLY -- IT RETAGS NOTHING. +# bundle.yaml's machines block is the vr1-dc0 source of truth and already reads +# `tags=openstack-vr1-dc0,` for ids 0-8, so dc0 needs no machines-block delta; +# only dc1 does (that is what its sibling overlay is for). Do not add a `machines:` +# block here to make the pair look symmetric -- it would duplicate the base values in +# a second place and create a drift surface for no gain. The filename is therefore +# slightly broader than "machines": it is the per-DC HAND-MAINTAINED overlay. Renaming +# it would require editing scripts/preflight.sh, which this file does not own. +# +# ---- ovn-chassis bridge-interface-mappings: WHY IT LIVES HERE ---- +# MOVED OUT OF bundle.yaml 2026-07-29 (item 3.8). The base bundle carried vr1-DC1's +# MACs while being documented as dc0's source of truth, so a dc0 deploy matched no +# local MAC, ovn-chassis built no br-ex mapping, provider egress was dead on dc0 +# compute, and no gate failed (provider-bundle-check only COUNTS well-formed MACs; it +# never checks the prefix, and `if not bim: continue` makes ABSENCE a silent skip). +# Governing text: the D-124 coupling entry's 2026-07-27 correction in +# docs/design-decisions.md -- "ovn-chassis bridge-interface-mappings is a PER-DC value +# and must become renderer OUTPUT into the per-DC overlay ... It must also move OUT of +# the base bundle for the 2026-07-25 symmetric-overlay ruling to be truthful." +# It is placed in the MACHINES overlay, not the vips overlay, because +# overlays/vr1-dc0-vips.yaml is GENERATED by scripts/render-dc-overlays.py and says +# "do not hand-edit" -- a key added there is clobbered on the next render. When the +# renderer is extended to emit this value (the "renderer OUTPUT" half of the same +# correction), it should emit INTO this file, not into the vips file. +# +# ---- THE MAC VALUES: MEASURED, AND NOT WHERE TWO REPO RECORDS SAY THEY ARE ---- +# vr1-dc0's MACs are NON-schematic (measured 52:54:00:* values), unlike dc1's derivable +# 52:54:01:d1:NN:PP scheme, so every one of them has to be read from a capture. +# +# SOURCE OF TRUTH: opentofu/vr1-dc0-substrate/main.tf, the `vr1_dc0_nodes` map. Its +# own comment fixes the ordering: "Order = vr1_dc0_node_nics order (metal-admin, +# provider-public, metal-internal, data-tenant, storage, replication)" -- metal-admin +# FIRST (:77 "metal-admin FIRST = PXE/boot plane"). So the provider-public MAC is +# element [1], the SECOND entry, never the first: +# vr1-dc0-compute-01 (main.tf:109-112) -> [0] 52:54:00:1b:19:e6 metal-admin/PXE +# [1] 52:54:00:8c:2a:8c provider-public <-- used +# vr1-dc0-compute-02 (main.tf:113-116) -> [0] 52:54:00:18:ab:b4 metal-admin/PXE +# [1] 52:54:00:50:48:88 provider-public <-- used +# +# CROSS-CHECKED against two independent captures: +# docs/audit/inner-plan-20260721-macpin.txt:36-72 / :85-123 -- the tofu plan that +# pinned them, listing each domain's interfaces in vr1_dc0_node_nics order. +# docs/audit/stage5-committee-raw-20260727.md:323 -- live MAAS, 18/18 nodes: +# "`enp1s0..enp6s0` + `br-ex`. `enp2s0` correctly carries no link (D-100 +# provider-raw); `br-ex` is a bridge over `enp2s0`". NIC position 2 = provider- +# public = the br-ex member. This is the same positional slot dc1 encodes as `:02`. +# +# !! TWO REPO RECORDS ARE WRONG ABOUT THIS AND WILL MISLEAD THE NEXT READER !! +# - docs/audit/stage5-unmeasured-register-20260727.md, row U5, states dc0's +# "compute provider-public MACs" are 52:54:00:1b:19:e6 / 52:54:00:18:ab:b4 and +# notes they match scripts/lib-hosts.sh:121-122. They DO match lib-hosts -- but +# lib-hosts.sh:121-122 is HOST_BOOT_MAC, the metal-admin/PXE NIC ([0]), NOT +# provider-public. U5 labels the boot MACs as provider MACs. +# - the D-124 coupling correction says this value should be "sourced from +# lib-hosts.sh". lib-hosts.sh carries ONLY HOST_BOOT_MAC and therefore CANNOT +# source it; the substrate main.tf is the only in-repo source for NIC [1]. +# Taking either record at face value would have put the PXE MAC in br-ex -- br-ex +# bridged onto the boot/metal-admin NIC instead of the provider NIC. Both are logged +# as findings; neither file is edited here. +# +# The charm picks whichever MAC is local per unit and ignores non-matching entries; +# ovn-chassis is subordinate to the 2-unit nova-compute, so exactly these two compute +# hosts carry br-ex. Exactly 2 MACs belong here -- a third means a wrong node. +# +# VERIFY-LIVE (this file is a NEW merge surface -- assert it at `juju deploy --dry-run` +# BEFORE any deploy; no offline juju merger exists to prove it here). The dry-run must +# show base `ovn-chassis` keeping BOTH of its existing options alongside the new one: +# ovn-bridge-mappings: physnet1:br-ex +# prefer-chassis-as-gw: true (B2 -- gateway-chassis election) +# This overlay sets only `bridge-interface-mappings` under `options`. That is safe IF +# juju merges the `options` MAP key-by-key, and unsafe if it REPLACES the map -- in +# which case the physnet mapping and the B2 gateway election both vanish from the +# deploy input and provider egress dies, by a different route than the defect this +# file exists to fix. Key-by-key is near-certain (the whole *-vips.yaml architecture +# depends on it -- those overlays set `options.vip` on apps carrying other base +# options) but it is NOT PROVEN HERE: provider-bundle-check.py's `_deep_merge` merges +# key-by-key by construction, so validating through the checker cannot tell the two +# behaviours apart. Only the live dry-run can. Assert it explicitly; do not assume. +# +# Applied as part of the vr1-dc0 per-DC set (R6: this set lands BEFORE +# overlays/dc-ha-scaleup.yaml, never in the same command as it): +# juju deploy ./bundle.yaml \ +# --overlay overlays/vr1-dc0-vips.yaml \ +# --overlay overlays/vr1-dc0-machines.yaml \ +# --overlay overlays/vr1-dc0-octavia-pki.yaml \ +# -m "${DC_MODEL}" +applications: + ovn-chassis: + options: + bridge-interface-mappings: >- + br-ex:52:54:00:8c:2a:8c + br-ex:52:54:00:50:48:88 diff --git a/overlays/vr1-dc1-machines.yaml b/overlays/vr1-dc1-machines.yaml index ceac025..bb2a13a 100644 --- a/overlays/vr1-dc1-machines.yaml +++ b/overlays/vr1-dc1-machines.yaml @@ -8,16 +8,79 @@ # bundle.yaml is the vr1-dc0 source of truth (matches overlays/vr1-dc1-vips.yaml); # this overlay + the VIP overlay are the two per-DC deltas for dc1. # -# Applied (with the HA scale-up + VIP overlays): +# Applied as the vr1-dc1 per-DC set: # juju deploy ./bundle.yaml \ -# --overlay overlays/vr1-dc1-machines.yaml \ # --overlay overlays/vr1-dc1-vips.yaml \ -# --overlay overlays/dc-ha-scaleup.yaml -m "${DC1_MODEL}" +# --overlay overlays/vr1-dc1-machines.yaml \ +# --overlay overlays/vr1-dc1-octavia-pki.yaml \ +# -m "${DC_MODEL}" +# +# CORRECTED 2026-07-29 (NEW-7), two defects in the command above: +# 1. It listed `--overlay overlays/dc-ha-scaleup.yaml` in the SAME deploy command as +# the VIP overlay. R6 (RULED 2026-07-27, docs/design-decisions.md) forbids that +# ordering: operator utterance "Close the two VIP gaps first, then apply the +# overlay whole". The two VIP gaps are vault and designate; R11 landed their VIPs +# in the per-DC vips overlays (.61 / .62). In ONE command the two overlays merge +# before anything deploys, so the scale-up and the VIPs arrive together and the +# "VIPs first" sequence R6 ruled cannot be observed at all. dc-ha-scaleup.yaml is +# applied SEPARATELY, AFTER this set is deployed and settled -- see that file's +# own header. Do not re-merge them. +# 2. It used a THIRD model-variable spelling, `${DC1_MODEL}`. The runbooks use +# `${DC_MODEL}` (the per-DC model of whichever DC is being deployed); this overlay +# is already dc1-scoped by its filename, so the variable must not be too. # # VERIFY-LIVE: confirm the machines-block override merges as expected at # `juju deploy --dry-run` BEFORE deploy (no offline juju merger exists to prove it # here). If it does not take, the fallback is inlining these dc1 tags into a rendered # dc1 bundle (cheap) -- see committee-20260724-track2-bundle-render.md, Fork 2. +# +# VERIFY-LIVE, SECOND ASSERTION (added 2026-07-29 with the ovn-chassis block below -- +# this file is no longer machines-only, so the old note no longer covers everything it +# changes). The dry-run must ALSO show that base `ovn-chassis` keeps BOTH of its +# existing options alongside the new one: +# ovn-bridge-mappings: physnet1:br-ex +# prefer-chassis-as-gw: true (B2 -- gateway-chassis election) +# This overlay sets only `bridge-interface-mappings` under `options`. That is safe IF +# juju merges the `options` MAP key-by-key, and unsafe if it REPLACES the map -- in +# which case the physnet mapping and the B2 gateway election both vanish from the +# deploy input and provider egress dies, by a different route than the defect this +# move fixed. Key-by-key is near-certain (the whole *-vips.yaml architecture depends +# on it -- those overlays set `options.vip` on apps carrying other base options) but +# it is NOT PROVEN HERE: provider-bundle-check.py's `_deep_merge` merges key-by-key +# by construction, so validating through the checker cannot tell the two behaviours +# apart. Only the live dry-run can. Assert it explicitly; do not assume. +# +# ---- ovn-chassis bridge-interface-mappings: MOVED HERE FROM bundle.yaml 2026-07-29 ---- +# These exact two values previously sat in the BASE bundle (item 3.8), which is +# documented as vr1-DC0's source of truth -- a residue of the 2026-07-24 dc1-first +# render. They are unchanged in value, only relocated to the DC they actually describe. +# The defect they caused was on the OTHER side: a dc0 deploy matched no local MAC, +# ovn-chassis built no br-ex mapping, and provider egress was dead on dc0 compute with +# no gate failing. Governing text: the D-124 coupling entry's 2026-07-27 correction in +# docs/design-decisions.md -- bridge-interface-mappings "is a PER-DC value and must +# become renderer OUTPUT into the per-DC overlay ... must also move OUT of the base +# bundle for the 2026-07-25 symmetric-overlay ruling to be truthful". +# +# VALUES RE-MEASURED, not copied on trust: opentofu/vr1-dc1-substrate/main.tf:112-119. +# The scheme is 52:54:01:d1:NN:PP where "PP = NIC ordinal 01..06 in vr1_dc1_node_nics +# order (metal-admin first)" (:95-98), so provider-public is PP=02, the SECOND NIC -- +# NOT :01, which is the metal-admin/PXE boot MAC in lib-hosts.sh HOST_BOOT_MAC. +# vr1-dc1-compute-01 -> NN=04 -> 52:54:01:d1:04:02 +# vr1-dc1-compute-02 -> NN=05 -> 52:54:01:d1:05:02 +# Cross-checked live: docs/audit/stage5-committee-raw-20260727.md:323 records 18/18 +# nodes as `enp1s0..enp6s0` + `br-ex`, with `br-ex` a bridge over `enp2s0` (position 2 +# = provider-public, D-100 provider-raw). Same positional slot as dc0's measured pair. +# The charm ignores non-local MACs; ovn-chassis is subordinate to the 2-unit +# nova-compute, so exactly these two compute hosts carry br-ex. Exactly 2 MACs belong +# here -- a third means a wrong node. See overlays/vr1-dc0-machines.yaml for dc0's set +# and for the two repo records that mislabel the PXE MACs as the provider MACs. +applications: + ovn-chassis: + options: + bridge-interface-mappings: >- + br-ex:52:54:01:d1:04:02 + br-ex:52:54:01:d1:05:02 + machines: "0": constraints: arch=amd64 tags=openstack-vr1-dc1,control diff --git a/runbooks/dc-dc-phase6-designate-cos-magnum.md b/runbooks/dc-dc-phase6-designate-cos-magnum.md index df6f5fd..d4b10ae 100644 --- a/runbooks/dc-dc-phase6-designate-cos-magnum.md +++ b/runbooks/dc-dc-phase6-designate-cos-magnum.md @@ -59,9 +59,10 @@ !!! COS MECHANISM IS UNDETERMINED -- do not invent it. D-105 rules the SCOPE (COS in-scope, per-DC, no Office1 roll-up) but no D-NNN or design doc in this repo specifies the COS deployment MECHANISM (bundle vs. separate - Juju model, exact channel, cross-model-relation topology). Step 5 below + Juju model, exact channel, cross-model-relation topology). Step 10 below presents the standard options for the operator to pick from at execution - time -- it does not pre-select one. + time -- it does not pre-select one. (Corrected 2026-07-29: this read "Step 5", + which is the Designate step, not the COS step.) --- @@ -76,8 +77,10 @@ 2. Static /etc/hosts bootstrap (D-008 step 1, per DC) 3. os-public-hostname per API charm [MUTATION: juju config, per DC] 4. Vault issues FQDN-SAN certs [MUTATION: cert cascade, per DC] -5. Deploy designate + designate-bind [MUTATION: bundle-diff apply into the - into the LIVE per-DC model live model, per DC] +5. Verify designate present + active, (read-only assertion, then juju config + then CONFIGURE it only -- NOT a deploy; designate ships + in the bundle and Stage 5 installs it. + D-106 ruling note 2026-07-27) 6. Designate zone + A/AAAA records [MUTATION: per DC, from measured values] 7. Neutron default_dns_domain/ [MUTATION: per DC] dns_servers -> Designate @@ -147,30 +150,46 @@ ## Step 1 -- bundle.yaml edit: reactivate Designate (REPO CHANGE, gated behind D-106) -### 1.1 -- What is currently in the repo (quoted, not paraphrased) +### 1.1 -- What is currently in the repo (measured, not paraphrased) -`bundle.yaml`'s own description block states, at the line documenting decision -D-019: -``` - D-019 (supersedes D-008) Designate deferred to v2 -``` -`runbooks/phase-01-bundle-deploy.md` states the dry-run/deploy expectation -explicitly: -``` -relations include -`octavia:certificates - vault:certificates`, `vault:shared-db - vault-mysql-router`, -`mysql-innodb-cluster:certificates - vault:certificates`; NO `vault:ha`, NO designate -(D-019). -``` -And `bundle.yaml` itself carries the deferred application/subordinate as -comments, not live entries: -``` - # designate-hacluster stays deferred (D-019). - ... - # v2-deferred (D-019): designate-hacluster: { charm: hacluster, channel: 2.4/stable } -``` -There is no `designate:` or `designate-bind:` application block anywhere in -`bundle.yaml` today. This is the concrete state D-106 reverses. +**CORRECTED 2026-07-29 per the D-106 RULING NOTE of 2026-07-27.** This section +previously ended "There is no `designate:` or `designate-bind:` application block +anywhere in `bundle.yaml` today", which CONTRADICTED section 1.2 below and the +bundle itself. The ruling names this defect directly: ":172 states 'There is no +`designate:` or `designate-bind:` application block anywhere in ...' while :178 +states 'Designate is deployed in-bundle in each DC' ... Both cannot be true; the +bundle settles it, and DOCFIX-167 put designate there on 2026-07-10." + +**MEASURED STATE OF `bundle.yaml` TODAY -- designate IS in the bundle.** Four +application blocks plus eight relations, all of whose peers Stage 5 creates: + +| Application | Charm / channel | Placement | Where | +|---|---|---|---| +| `designate` | `designate` `2024.1/stable`, `num_units: 1` | `to: [lxd:0]` | `bundle.yaml:853` | +| `designate-mysql-router` | `mysql-router` `8.0/stable` | subordinate | `bundle.yaml:872` | +| `designate-bind` | `designate-bind` `2024.1/stable`, `num_units: 1` | `to: [lxd:2]` | `bundle.yaml:881` | +| `designate-hacluster` | `hacluster` `2.4/stable`, `cluster_count: 1` | subordinate | `bundle.yaml:963` | + +(Line numbers are as of 2026-07-29; grep the app names, not the numbers, if the +bundle has moved since.) + +Note `designate`'s provides-side binding is `dnsaas: provider-public`, not +`public` (see 1.5) -- which is why it does not join provider-bundle-check's +`EXPECT_PUBLIC_VIP` count of 11. + +**HISTORICAL -- the pre-D-106 state, retained because it is what the decision +reverses.** Before DOCFIX-167 (2026-07-10), `bundle.yaml` recorded the D-019 +deferral in its description block (`D-019 (supersedes D-008) Designate deferred to +v2`) and carried the subordinate as a comment rather than a live entry +(`# v2-deferred (D-019): designate-hacluster: ...`); +`runbooks/phase-01-bundle-deploy.md` still states the VR0 dry-run expectation +`NO vault:ha, NO designate (D-019)`. That phase-01 line is VR0/testcloud scope and +is NOT a claim about this bundle's current VR1 content -- do not read it as one. + +**CONSEQUENCE FOR THIS RUNBOOK, RULED:** designate lands as INERT INFRASTRUCTURE +at Stage 5 -- it deploys with the bundle and does nothing, because +`os-public-hostname` is set in no deploy artifact. Stage 7 (this runbook) does not +install it. Step 5 below is a CONFIGURE step, not a deploy step. ### 1.2 -- The governing decision, quoted @@ -231,8 +250,9 @@ tooling gap register). The new `os-public-hostname` values are NOT DC-agnostic -- they embed `dc1` or `dc2` per D-106's naming convention. Do not hardcode a DC-specific FQDN into the shared `bundle.yaml`. Follow the existing overlay -pattern already established for secrets (`overlays/octavia-pki.yaml`, -Step 1.0 of `phase-01-bundle-deploy.md`) and add one per-DC hostnames overlay: +pattern already established for secrets (`overlays/${DC}-octavia-pki.yaml` -- +itself made PER-DC by the D-109 amendment, gitignored by glob; Step 1.0 of +`phase-01-bundle-deploy.md`) and add one per-DC hostnames overlay: `overlays/dc1-hostnames.yaml`, `overlays/dc2-hostnames.yaml`, each setting only the `os-public-hostname` options for that DC, applied at deploy time with an additional `--overlay` alongside whichever overlays that DC's deploy @@ -269,8 +289,10 @@ bash scripts/repo-lint.sh # expect 0 fail bash scripts/preflight.sh # expect PREFLIGHT: PASS (incl. Charmhub channel assert) ``` -**GATE:** repo-lint clean; preflight PASS. Only after this passes does Step 5 -(the live per-DC apply) become eligible. Deliver this edit with a changelog +**GATE:** repo-lint clean; preflight PASS. Only after this passes do the live +per-DC steps become eligible. (Corrected 2026-07-29: this read "does Step 5 (the +live per-DC apply) become eligible" -- Step 5 is no longer an apply, see the +D-106 ruling note; the applies live in Steps 3/6/7/8.) Deliver this edit with a changelog entry (what changed, why -- D-106, how to revert) per this repo's standing delivery rule, same as any other script/runbook/bundle change. @@ -332,7 +354,9 @@ cert cascade (Vault's `certificates` interface) picks up the new SAN automatically once the relation re-fires -- no manual CSR step is expected on this charm family (confirm this is still true for the designate charm's own -certificate once Step 5 relates it). +certificate -- its `certificates` relation ships in the bundle and is already up +by the time you reach here, so there is nothing to relate at Step 5; corrected +2026-07-29, this read "once Step 5 relates it"). **CHECK (read-only) -- jumphost, per charm** ```bash @@ -348,48 +372,115 @@ --- -## Step 5 -- Deploy designate + designate-bind into the LIVE per-DC model +## Step 5 -- Verify designate is present and active, then configure it (NOT a deploy step) -This DC's OpenStack cloud is already running (Stage 4 delivered it). Adding -Designate here is a bundle-diff INCREMENTAL apply against the live model, not -a fresh `juju deploy` from empty -- Juju's bundle handling adds only the new -applications/relations the diff introduces; it does not touch already-running -apps whose config is unchanged in the bundle. +**REWRITTEN 2026-07-29 UNDER THE D-106 RULING NOTE OF 2026-07-27 (GA-R5).** +Question put to the operator, and its selected option, verbatim: **"Accept at +Stage 5; rewrite Stage 7 Step 5 to configure-not-deploy (Recommended)"**. The +ruled shape, quoted: "Designate lands as INERT INFRASTRUCTURE at Stage 5 ... +Step 5 is rewritten from 'deploy' to 'verify present and active, then +configure'. Its diff-gate is replaced by a presence-and-state assertion." -**CHECK (read-only) -- jumphost -- dry-run against the live model first** +**DO NOT RUN `juju deploy` IN THIS STEP.** The four designate applications and +their eight relations ship in `bundle.yaml` (see 1.1) and every relation peer is +created at Stage 5, so Stage 5's own bundle deploy already installed them. They +arrive FUNCTIONALLY INERT because `os-public-hostname` is set in no deploy +artifact -- which is the state this stage exists to resolve, and the work is +Steps 3/4/6/7/8, not a second deploy here. + +What was here before, and why it could not work: +- The step was written as a deploy with a gate requiring "the diff shows ONLY + the new designate/... applications and their relations being added". That is + UNSATISFIABLE against a model that already has them -- the ruling names it as + the surface defect. +- Both commands passed `--overlay "overlays/${DC}-machines.yaml"` UNGUARDED, and + the second was the real apply against a LIVE model. `overlays/vr1-dc0-machines.yaml` + did not exist, so a dc0 run would have errored out at `juju deploy` while + `preflight.sh`'s P2 -- which guards the same path with `[ -f ]` -- went green. + That gap is now closed from BOTH sides: the deploy commands are gone from this + step, and `overlays/vr1-dc0-machines.yaml` was created 2026-07-29 (item 3.8), so + the P2 guard no longer silently skips a file the deploy requires. +- It named `overlays/octavia-pki.yaml`, renamed PER-DC to + `overlays/${DC}-octavia-pki.yaml` by the D-109 amendment. + +The per-DC deploy command itself belongs to Stage 5 +(`runbooks/dc-dc-phase4-juju-bundle-per-dc.md`), not to this runbook. Do not +reintroduce a copy of it here. + +### 5.1 -- Presence-and-state assertion (replaces the old diff gate) + +!!! COMMANDS BELOW ARE UNVERIFIED FROM THE AUTHORING HOST. `juju` is ABSENT on + vcloud (measured 2026-07-27, register row U9: `juju` at `/snap/bin/juju` on + voffice1, absent on vcloud), so nothing in this step could be executed while + it was written. Run them from the Plane-2 host (voffice1) per D-128, and + treat any flag that errors as an authoring defect to fix here, not as a + reason to improvise a different command. + +**CHECK (read-only) -- from the Office1 headend (voffice1)** ```bash -# CORRECTED 2026-07-29 (chain audit findings 18/19). This command previously passed NO -# VIP overlay -- and since ruling 3 (2026-07-25) `bundle.yaml` is VIP-FREE, so the merged -# input carried ZERO VIPs including the designate this step exists to add; -# provider-bundle-check invariant 8 would reject it, but nothing ran the checker here. -# It also named `overlays/${DC}-hostnames.yaml`, which HAS NEVER BEEN AUTHORED -- juju -# errors out on a missing overlay file. That overlay is D-106 hostname work and is not -# built; it is omitted rather than referenced, and `os-public-hostname` stays unset -# (bundle.yaml records IP-ONLY endpoints as the current posture, B5). -juju deploy ./bundle.yaml \ - --overlay "overlays/${DC}-vips.yaml" \ - --overlay "overlays/${DC}-machines.yaml" \ - --overlay overlays/octavia-pki.yaml \ - -m "${DC_MODEL}" --dry-run -``` -**GATE:** the diff shows ONLY the new designate/designate-bind/designate- -mysql-router/designate-hacluster applications and their relations being -added -- zero unexpected changes to any already-running application. If the -diff shows anything else changing (e.g. an existing charm's binding), STOP -and reconcile before applying. +# All four applications must ALREADY exist in this DC's live model. +# One `juju status --format=yaml` carries applications AND their relations, so it +# answers all three questions below without relying on a separate flag. +juju status -m "${DC_MODEL}" --format=yaml > /tmp/designate-presence.yaml -**RUN -- jumphost** -```bash -juju deploy ./bundle.yaml \ - --overlay "overlays/${DC}-vips.yaml" \ - --overlay "overlays/${DC}-machines.yaml" \ - --overlay overlays/octavia-pki.yaml \ - -m "${DC_MODEL}" +# 1. presence: +grep -E '^\s+(designate|designate-bind|designate-mysql-router|designate-hacluster):' \ + /tmp/designate-presence.yaml + +# 2. workload/agent state, human-readable: +juju status -m "${DC_MODEL}" designate designate-bind + +# 3. the in-bundle relations, designate side: +grep -i -A3 designate /tmp/designate-presence.yaml | grep -i 'relation\|endpoint' ``` -**GATE:** `designate`, `designate-bind`, `designate-mysql-router`, -`designate-hacluster` all reach `active/idle`; `designate:dns-backend - -designate-bind:dns-backend` and `designate:identity-service - -keystone:identity-service` relations up. +**GATE (presence AND state -- both, not either):** +1. `designate`, `designate-bind`, `designate-mysql-router` and + `designate-hacluster` are all PRESENT in the model. If any is MISSING, that + is a Stage-5 deploy defect, not work for this step -- STOP and reconcile + against Stage 5's own gate; do not deploy it from here. +2. All four are `active/idle`. +3. `designate:dns-backend - designate-bind:dns-backend` and + `designate:identity-service - keystone:identity-service` are up. + +An `active/idle` designate at this point means the CHARM is healthy. It does NOT +mean DNS works -- the feature is inert until the configuration below and Steps +6/7/8 land. Do not read unit status as evidence of function (same class of trap +as the D-046 magnum `domain-setup` warning at the top of this runbook). + +### 5.2 -- Configure designate + +Steps 2/3/4 have by now set `os-public-hostname` per API charm and re-issued +Vault FQDN-SAN certs; Steps 6/7/8 own the zone + A/AAAA records, Neutron's +`default_dns_domain`/`dns_servers`, and tenant subnet `--dns-nameserver`. What +remains to THIS step is designate's own charm configuration -- the settings that +make the already-running charm serve this DC's zone. + +**MEASURE THE OPTION NAMES, DO NOT CITE THEM FROM HERE.** This runbook +deliberately names no designate charm config key: no such key is recorded in any +ratified decision or measured capture in this repo, and 1.5 already records that +several designate endpoints were confirmed only by reading the charm's own +source. Read the real option set at execution time and configure from that: + +```bash +# THE authoritative read: for a deployed application this returns the full option +# set -- real key names, current values, defaults and descriptions. It is +# sufficient on its own; nothing else is needed to drive this step. +juju config -m "${DC_MODEL}" designate +``` +If you want the charm's raw `config.yaml` as well, the unit-local path is +`/var/lib/juju/agents/unit-designate-*/charm/config.yaml` -- UNVERIFIED, recorded +as a fallback only. If it is not there, do not go hunting: `juju config` above is +the supported surface and already answers the question. +**GATE:** designate returns to `active/idle` after any `juju config` change, and +every key you set was read from the charm's own `config.yaml` in this session -- +not carried from memory or from another cloud. If a value you need is a per-DC +literal (a nameserver address, an FQDN), MEASURE it live per this runbook's +standing rule; the zone NAME is the one literal cited from decision text (D-106), +and it is cited in Step 6 where the zone is created. + +If nothing in the charm's config set needs to change for this DC, say so +explicitly in the as-executed log and proceed -- an empty configure is a valid +outcome of this step and is not a reason to invent one. --- @@ -649,15 +740,24 @@ ## Delivery checklist -- [ ] `bundle.yaml` + new `overlays/dc1-hostnames.yaml` / `overlays/dc2- - hostnames.yaml` edited per Step 1; `scripts/repo-lint.sh` 0 fail; - `scripts/preflight.sh` PASS (incl. Charmhub channel assert on the new - designate/designate-bind charms). -- [ ] Changelog entry for the bundle.yaml + overlay change: what changed - (designate reactivation + os-public-hostname re-add), why (D-106, - superseding D-019), how to revert (re-comment the designate blocks, - drop the hostnames overlays, re-run preflight to confirm the revert is - clean). +- [ ] NOTE: the designate half of Step 1 is ALREADY DONE in the repo (DOCFIX-167, + 2026-07-10) -- `bundle.yaml` carries all four designate applications and + their eight relations, and Stage 5 deploys them. Do NOT re-add them. The + only Step 1 repo work still owed is the per-DC `os-public-hostname` + overlay proposed in 1.4, which is NOT AUTHORED + (`overlays/dc1-hostnames.yaml` / `overlays/dc2-hostnames.yaml` do not + exist -- and neither does the `overlays/${DC}-hostnames.yaml` spelling + other surfaces used to reference; never pass a name from this bullet to + `--overlay` before authoring the file, juju errors out on a missing + overlay). It is a PROPOSED pattern per 1.4: present it at the review gate, + do not silently adopt it. +- [ ] Once that overlay is authored: `scripts/repo-lint.sh` 0 fail; + `scripts/preflight.sh` PASS (incl. Charmhub channel assert). +- [ ] Changelog entry for the overlay change: what changed + (os-public-hostname re-add per DC), why (D-106, superseding D-019), how to + revert (drop the hostnames overlay, re-run preflight to confirm the revert + is clean). Reverting designate itself is NOT part of this stage -- it is + Stage 5 bundle content under the D-106 ruling note. - [ ] `docs/session-ledger.md` updated with this runbook's authoring and, once executed, each DC's Stage 7 pass/fail state. - [ ] `docs/dc-dc-deployment-workflow.md` Stage 7 row's `**State:**` line