diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 8ca9d81..c3f939c 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1308,6 +1308,46 @@ **G17's dc0 half is NOW GENUINELY ARMABLE and this is its one-shot window** -- the existing capture was taken on the CONTROLLER VM and says so; the nine ROLE nodes are booting for the first time as of this entry. + **>>> THREE ARTIFACT/CONFIG DEFECTS SURFACED POST-DEPLOY, EACH MASKED BY THE ONE BEFORE + IT** (`apt-get update --error-on=any` fails on the FIRST bad source, so only one is ever + visible). Full detail appended to `docs/audit/stage5-dc0-deploy-attempt3-20260731.txt`. + **D1 `jammy-backports` 404 -- RULED AND FIXED** (see the D-135 amendment above); verified on + CONTENT and the mirror grew 951G -> 952G. Units then moved 22 error -> 17, with 15 apps past + install (was 1), so the fix is working and auto-retry IS running. + **D2 THE UPSTREAM UCA IS UNREACHABLE FROM NODES, LOGGED NOT FIXED.** MEASURED FROM A NODE + (not the rack, which holds a default route nodes lack -- the recorded wrong-host class): + `node -> ubuntu-cloud.archive.canonical.com` **000**, `node -> 10.12.8.4/cloud-archive` + **200 with real content**. juju's `apt-mirror` rewrites only the Ubuntu archive, not the UCA + source the charm adds. **SIX apps set it explicitly across TWO option names** -- + `openstack-origin` on barbican/magnum/octavia, `source` on ceph-mon/ceph-osd/ceph-radosgw -- + and every other OpenStack charm defaults to `openstack-origin: caracal`, which resolves to + the same upstream pocket. **The UCA signing key is ALREADY on the node** + (`ubuntu-keyring-2012-cloud-archive.gpg`), so a raw `deb` line will verify with no `|key` + suffix. **GENUINE D-135 EXPERIMENT RESULT: the full-mirror DC must rewrite every non-Ubuntu + source; the proxy DC needs none, because apt-cacher-ng forwards whatever URL it is handed. + dc1 will not hit this at all.** + **D3 >>> `prefer-ipv6: true` IS FATAL ON THE SEVEN CHARMS THAT DECLARE IT, AND IT IS NOW THE + BLOCKER. <<<** This is the EXACT risk this session flagged as "never observed live" when it + was deliberately left OUT of the D-101 ruling note. MEASURED 19:08: `unit-keystone-0 ... + Exception: Interface 'eth0' does not have a scope global non-temporary ipv6 address.` Cause + matches the charm source read earlier today exactly -- `get_relation_ip()` + (`charmhelpers/contrib/network/ip.py:617`) returns EARLY with `get_ipv6_addr(...)[0]` when + the option is true. **MEASURED on the container: `keystone/0 eth0 -> fe80::216:3eff:feb4:5518/64`, + LINK-LOCAL ONLY, no global v6. The NODES are dual-stacked; the LXD CONTAINERS the API charms + run in are NOT.** Live config confirms the split: keystone, cinder, glance, neutron-api, + nova-cloud-controller, openstack-dashboard and ceph-radosgw all read `prefer-ipv6=true` and + **ALL SEVEN are in `error`**; ceph-mon and mysql-innodb-cluster read `false` and fail for + other reasons; ovn-central and vault do not declare it. **NOT TAKEN -- R2 / D-101 are ruled + surfaces and this needs its own GA-R5 exchange.** The measurement chain says dropping it + should be safe (the `:::port` bind is gated on the kernel sysctl, and the container carries + a link-local v6 so v6 is not disabled; pacemaker assigns the v6 VIP by family detection), + but **one question is STATED AND NOT ANSWERED: whether a v6 VIP on a container whose eth0 + has no global v6 is ROUTABLE.** That is separate from whether the charm installs. + **memcached SCALED to 3 in `overlays/dc-ha-scaleup.yaml`** (operator-directed 2026-07-31); + its exclusion comment is RE-POINTED rather than left stale. **`ceph-rbd-mirror` is PINNED, + NOT SETTLED** -- gap register **item 22** carries four options and the recommendation + ((d) add the missing detection now, (b) scale to 2 active/standby at Roosevelt), and the + overlay carries a pointer forbidding a silent scale without a D-108 amendment. **>>> THE DEPLOY IS BLOCKED ON ONE PRECISE ARTIFACT DEFECT: THE dc0 MIRROR DOES NOT CARRY `jammy-backports`. <<<** MEASURED 2026-07-31, end to end: 22 of 33 units in `error`, every one `hook failed: "install"`, and the hook output is diff --git a/docs/audit/stage5-dc0-deploy-attempt3-20260731.txt b/docs/audit/stage5-dc0-deploy-attempt3-20260731.txt index 72acbb2..164c445 100644 --- a/docs/audit/stage5-dc0-deploy-attempt3-20260731.txt +++ b/docs/audit/stage5-dc0-deploy-attempt3-20260731.txt @@ -76,3 +76,61 @@ the cloud is up. 9 machines are allocating, 33 units are pending, and the settle to phase-01's documented pre-vault-init end state takes hours. Nothing here asserts unit health, relation settling, or any service verdict. + +================================================================================ +POST-DEPLOY: THREE ARTIFACT/CONFIG DEFECTS, IN THE ORDER THEY UNMASKED +================================================================================ +Each was hidden by the one before it. `apt-get update --error-on=any` fails on the +FIRST bad source, so only one is ever visible at a time. + +D1 -- `jammy-backports` 404 on the dc0 mirror. RULED + FIXED 2026-07-31 (D-135 + amendment, operator: "Sync the backports into the mirror"). Fixed at source in + scripts/dc-mirror.sh, T10 re-pointed, applied live, sync OK, verified on CONTENT + (Release fields + a real .deb fetch: elfutils_0.188-1~bpo22.04.1_amd64.deb, 200, + 502468 B). Mirror 951G -> 952G. + +D2 -- the UPSTREAM UCA is unreachable from nodes. MEASURED FROM A NODE (not the rack, + which holds a default route nodes lack): + node -> http://ubuntu-cloud.archive.canonical.com/... 000 + node -> http://10.12.8.4/cloud-archive/... 200, real content + The node's source line points UPSTREAM. juju's `apt-mirror` model-config rewrites + only the Ubuntu archive, not the UCA source the charm adds. SIX apps set it + explicitly across TWO option names -- `openstack-origin` on barbican/magnum/octavia + and `source` on ceph-mon/ceph-osd/ceph-radosgw -- and every other OpenStack charm + defaults to `openstack-origin: caracal`, which resolves to the same upstream pocket. + The UCA signing key IS already on the node + (/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cloud-archive.gpg), so a raw `deb` line + pointing at the mirrored UCA will verify with no `|key` suffix. + EXPERIMENT RESULT WORTH KEEPING: this asymmetry is a genuine D-135 finding -- the + FULL-MIRROR DC must rewrite every non-Ubuntu source; the PROXY DC needs none, + because apt-cacher-ng forwards whatever URL it is handed. dc1 will not hit this. + +D3 -- >>> `prefer-ipv6: true` IS FATAL ON THE SEVEN CHARMS THAT DECLARE IT. <<< + MEASURED 2026-07-31 19:08, and this is the exact risk this session flagged as + "never observed live" when it was deliberately left OUT of the D-101 ruling note: + unit-keystone-0 ... Exception: Interface 'eth0' does not have a scope global + non-temporary ipv6 address. + Cause, and it matches the charm source read earlier today EXACTLY: + `get_relation_ip()` (charmhelpers/contrib/network/ip.py:617) returns EARLY with + `get_ipv6_addr(dynamic_only=False)[0]` whenever `prefer-ipv6` is true -- so the unit + tries to advertise a v6 address for every relation. MEASURED on the container: + keystone/0 eth0 -> fe80::216:3eff:feb4:5518/64 (LINK-LOCAL ONLY, no global v6) + The NODES are dual-stacked; the LXD CONTAINERS the API charms run in are NOT. + Live config confirms the split: keystone, cinder, glance, neutron-api, + nova-cloud-controller, openstack-dashboard, ceph-radosgw all read + `prefer-ipv6=true` and ALL SEVEN are in `error`. ceph-mon and mysql-innodb-cluster + read `false` (charm default) and fail for other reasons; ovn-central and vault do + not declare the option at all. + CONSEQUENCE: the 2026-07-31 ruling removed the option from the SIX charms that + never declared it, which was correct and necessary. The SEVEN that DO declare it + were explicitly NOT covered by that ruling -- and they are now the blocker. + NOT TAKEN HERE: R2 / D-101 are ruled surfaces and this needs its own GA-R5 exchange. + + WHY DROPPING IT IS EXPECTED TO BE SAFE, from the same measurement chain: the option + is a UNIT ADDRESS-FAMILY switch, not a listener switch. HAProxy's `:::port` bind is + gated on `ipv6_enabled = not is_ipv6_disabled()` (the kernel sysctl), and the + container carries a link-local v6, so v6 is NOT disabled and the bind still emits. + Pacemaker assigns the v6 VIP by address-family detection, independent of the option. + OPEN QUESTION that belongs to the operator, stated not glossed: whether a v6 VIP + placed on a container whose eth0 has no global v6 is ROUTABLE. That is a different + question from whether the charm installs, and it is not answered here. diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md index 02bac93..abd6c32 100644 --- a/docs/dc-dc-deployment-workflow.md +++ b/docs/dc-dc-deployment-workflow.md @@ -1226,6 +1226,64 @@ CIDR. **Roosevelt analog: this is the operator's out-of-band access path to every site, so it is a day-1 standup item, not a convenience added later.** +22. **NEW 2026-07-31 -- PINNED, NOT SETTLED: `ceph-rbd-mirror` is the last single-unit + service in the control plane and must be brought into line with the HA posture.** + + Operator direction 2026-07-31: pin it with options and a recommendation "to make sure + it is not missed as we need to bring it in line with the rest of the HA posture". + + **It is RULED, so this is not an oversight to fix -- it is a ruling to revisit.** + D-108 (ADOPTED 2026-07-09): *"Glance images: `ceph-rbd-mirror` of the Glance pool, + snapshot-based, two-way, with a re-register step on failover. Daemon topology: one + rbd-mirror daemon per DC, peering over the replication plane; **daemon HA is + single-unit for VR1 (matches the D-104 controller economy), noted for Roosevelt + scale-up**."* D-121's sizing table repeats it: "Left single (NOT scaled): + `ceph-rbd-mirror`=1 (**D-108** rules single-unit for VR1)". + + **WHY IT MATTERS.** After the 2026-07-31 memcached change, `ceph-rbd-mirror` is the + only control-plane container left at one unit with no HA story (`designate-bind` and + `glance-simplestreams-sync` are also 1, but D-121 classes them as a DNS backend and a + periodic utility respectively -- neither is on a data-durability path). + rbd-mirror IS on one: it is the mechanism carrying Glance images between DCs for the + Tier-A DR posture. A single daemon means the cross-DC replication stream stops when + that container stops -- silently, since nothing currently alarms on mirror lag. + + **WHY IT IS NOT A `num_units` EDIT.** Two things a bump does not address, and this is + what makes it a D-108 amendment rather than an overlay line: + - **Daemon identity and peering.** Each rbd-mirror daemon authenticates to both the + local and remote Ceph clusters. Two daemons per DC is a topology choice + (active/active with the pool split between them, or active/standby with one idle) + and it changes the peer configuration on BOTH sides, not just this DC. + - **It is symmetric by nature.** rbd-mirror peers DC-to-DC, so scaling one DC's + daemon count without the other is not a half-step -- it is a different topology + than the peer expects. + + **OPTIONS, for the ruling when it is taken:** + - **(a) Leave at 1 for VR1, scale at Roosevelt -- what D-108 currently says.** No + work now; the exposure is that a single container failure halts cross-DC Glance + replication until noticed, and nothing watches mirror lag today. + - **(b) Scale to 2 per DC, active/standby.** The cheap half-step: a second daemon + that takes over if the first stops. Needs the D-108 amendment, a second client + identity per DC, and symmetric application at BOTH DCs. + - **(c) Scale to 2+ per DC, active/active with the pool split.** Real throughput and + redundancy; the most work, and the least justified at VR1's rehearsal scale. + - **(d) Leave at 1 but ADD the missing detection** -- a mirror-lag / daemon-liveness + assertion in `cloud-assert.sh`, so a stopped daemon is loud rather than silent. + + **RECOMMENDATION: (d) NOW, then (b) AT ROOSEVELT.** The measured gap today is not + redundancy, it is that a stopped mirror daemon is INVISIBLE -- `grep -i mtu`-style + checks aside, nothing in `cloud-assert.sh` reads rbd-mirror state at all, so option + (a)'s exposure is unbounded in time rather than merely present. (d) is small, needs no + D-amendment (it adds an assertion, it does not change the ruled topology), and it + converts a silent failure into a detected one. (b) is the right shape for Roosevelt + where the DR posture is real, and D-108 already anticipates exactly that. (c) is + over-engineering for a rehearsal. + + **DO NOT scale it in `overlays/dc-ha-scaleup.yaml` without the D-108 amendment** -- + that overlay now carries a pointer to this item saying so. + + **STATUS: PINNED, awaiting a GA-R5 ruling. NOT to be closed by silently scaling it.** + --- ## Where things stand today (2026-07-09) -- update this as stages close diff --git a/overlays/dc-ha-scaleup.yaml b/overlays/dc-ha-scaleup.yaml index 190591a..0513165 100644 --- a/overlays/dc-ha-scaleup.yaml +++ b/overlays/dc-ha-scaleup.yaml @@ -106,8 +106,22 @@ # NOT changed here (deliberately, per D-121): mysql-innodb-cluster / ovn-central # / ceph-mon (already 3 in the base bundle); ceph-osd / nova-compute (scale-out, # sized to node count -- Option C = ceph-osd 4 (D-121 R-3 amendment), nova-compute 2, on the storage / -# compute nodes, NOT control-node containers); ceph-rbd-mirror stays 1 (D-108); -# memcached / designate-bind / glance-simplestreams-sync unchanged. +# compute nodes, NOT control-node containers); designate-bind / +# glance-simplestreams-sync unchanged. +# +# memcached WAS in that "unchanged" list and is now SCALED (see its block below, +# added 2026-07-31 by operator direction). This comment is RE-POINTED rather than +# left stale -- it is the only place the exclusion set is written down. +# +# >>> ceph-rbd-mirror STAYS 1, AND THAT IS PINNED, NOT SETTLED. <<< +# D-108 rules it: "one rbd-mirror daemon per DC ... daemon HA is single-unit for VR1 +# (matches the D-104 controller economy), NOTED FOR ROOSEVELT SCALE-UP". So the +# single unit is a ruled VR1 scope decision, not an oversight -- but D-108 itself +# flags that it must be revisited, and operator direction 2026-07-31 is that it be +# brought into line with the rest of the HA posture rather than quietly left behind. +# It is NOT a num_units edit: see docs/dc-dc-deployment-workflow.md gap register +# item 22 for the options, the recommendation, and why this one needs a D-108 +# amendment before anything is built. DO NOT scale it here without that ruling. # ===================================================================== applications: @@ -170,6 +184,22 @@ min-cluster-size: 3 to: ["lxd:0", "lxd:1", "lxd:2"] + # ---- memcached: 1 -> 3, ADDED 2026-07-31 by operator direction. It was left at 1 + # by D-121's sizing table, which listed it as "memcached=1 (cache; optional 3)" + # -- CONSIDERED and marked optional, but with no recorded reason for choosing 1. + # MEASURED consumers in bundle.yaml: nova-cloud-controller:memcache and + # designate:coordinator-memcached. (An earlier claim in-session that keystone + # consumes it was WRONG -- keystone holds no memcache relation in this bundle.) + # NO hacluster and NO vip, deliberately: memcached is not clustered. Three units + # are three INDEPENDENT caches; clients receive the full server list over the + # relation and hash keys across it, so losing one costs that shard's hit rate, + # not availability. That is the standard model, not a compromise. + # VERIFY AT DEPLOY: the consumers pick up all three servers (the relation should + # yield three memcached_servers entries, not one). + memcached: + num_units: 3 + to: ["lxd:0", "lxd:1", "lxd:2"] + # ---- Vault: MySQL-backed (v-a), scale to 3, re-HA'd via vault-hacluster # (defined below) + the relation re-added at the foot of this file. ---- vault: