diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 9911666..da0fa5f 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2207,6 +2207,36 @@ safe). **GA-R2 HOUSEKEEPING:** the build agent wrote its own `docs/changelog-20260801-g19-node-v6-verify.md`; it is FOLDED VERBATIM into this session's single changelog and REMOVED. The session crossing midnight does not start a new session. + **NEXT STEPS EXECUTED 2026-08-01, three items.** **(1) THE TWO TEST NODES ARE RELEASED** -- + `t7ymp6` and `fg6gxm` read back **`Ready` / `owner=None`**, released individually, nothing + stranded and no manual cleanup needed. The boot window is closed; G19 was run live before it + closed, so the gate ships PROVEN rather than fixture-green. + **(2) D-139's EXECUTION LIST WAS DEFECTIVE AND IS REPLACED** (`### CORRECTION NOTE 2026-08-01 + -- D-139`). **Neither ruling is touched** -- this replaces only the ordered execution list, + which I authored with three defects. The original text is preserved in the note rather than + silently overwritten. **Defect 1: "remove v4 LAST, after each is proven" is UNSATISFIABLE** + (per-plane conversion is atomic). **Defect 2, the consequential one and a SILENT-FAILURE + class: `dc-node-v6-carve.py` is structurally dependent on IPv4 existing**, so run after v4 + removal it would carve FOUR FEWER PLANES PER NODE and exit clean -- and the original list's + "LAST" wording actively invited that ordering. **Defect 3: the apex RETIRE half is unsafe in + the same step as CREATE**, because 52 dependent ip-addresses (the v6 VIP legs) sit inside the + retiring ULA `/64`s. The replacement is a 7-step list that pushes CREATE-only first, carves + node statics WHILE v4 IS STILL PRESENT, re-homes the 52 VIPs before any retire, and treats v4 + removal as a SEPARATE experimental step scoped to `storage` + `replication` TOGETHER per the + "B plus C" ruling. **Its stated prerequisite is still unmet: `network-get` on a v6-only bound + space has NOT been measured, and step 7 must not start before it is.** + **(3) THE LP DRAFT FOR THE "C" HALF IS WRITTEN, NOT FILED** -- + `docs/audit/lp-draft-20260801-ceph-osd-ipv6-static.md`. **Operator files it; Claude does not + post to Launchpad** (precedent: `lp-draft-20260721-maas-agent-resolver.md`). It is a COMMENT + on the EXISTING **LP #2061836** asking for the `charm-ceph-osd` task to be closed as + `charm-ceph-mon`'s already was -- **not a new bug**. Evidence quoted from the published + rev-953 artifact (`hooks/utils.py:205`, `hooks/ceph_hooks.py:547`, both calling + `get_ipv6_addr()` without `dynamic_only=False`), with the `prefer-ipv6=false` path's + `inet_aton` / `IN A` failure noted as the likely subject of LP #2109798. The draft carries an + explicit operator note **NOT to cite LP #1590598** (`Fix Released`, a different defect -- + citing it would be the inverted-citation class), and states plainly that **this bug is not + what blocks a v6-only Ceph plane locally**, since setting `ceph-public-network` / + `ceph-cluster-network` to the v6 CIDRs bypasses the buggy call entirely. - Project: Omega Cloud, VR1 DC-DC rehearsal -- a two-DC + Office1-headend virtual rehearsal on KVM (vcloud host), rehearsing the future bare-metal Roosevelt deployment (D-100, `docs/design-decisions.md:1946`). diff --git a/docs/audit/lp-draft-20260801-ceph-osd-ipv6-static.md b/docs/audit/lp-draft-20260801-ceph-osd-ipv6-static.md new file mode 100644 index 0000000..c92977e --- /dev/null +++ b/docs/audit/lp-draft-20260801-ceph-osd-ipv6-static.md @@ -0,0 +1,87 @@ +# LP draft 2026-08-01 -- charm-ceph-osd, IPv6 static addresses (LP #2061836) + +**STATUS: DRAFT. The operator files this; Claude does not post to Launchpad.** +Precedent: `docs/audit/lp-draft-20260721-maas-agent-resolver.md`. + +**Target: add a comment to the EXISTING bug LP #2061836 and ask for the +`charm-ceph-osd` task to be closed. Do NOT file a new bug** -- the defect is +already reported there, the `charm-ceph-mon` task is `Fix Committed` (2024-06-14), +and only the `charm-ceph-osd` task remains `New` and unfixed. + +Ruled under D-139 "B plus C" (2026-08-01, operator: *"B plus C"*): pursue the +upstream fix, because it is one word and already merged on the sibling charm. + +--- + +## Suggested comment text + +> Confirming this is still present in `charm-ceph-osd` and asking whether the +> ceph-osd task can be closed the same way the ceph-mon task was. +> +> Measured against the **published `ceph-osd` revision 953** (the current +> `2024.1/stable` artifact), `get_ipv6_addr()` is still called **without** +> `dynamic_only=False` at two sites: +> +> * `hooks/utils.py:205` -- inside `get_host_ip()`: +> +> ```python +> def get_host_ip(hostname=None): +> if config('prefer-ipv6'): +> return get_ipv6_addr()[0] +> ``` +> +> * `hooks/ceph_hooks.py:547` -- inside the `prefer-ipv6` branch that builds the +> ceph context: +> +> ```python +> if config('prefer-ipv6'): +> cephcontext['ms_bind_ipv4'] = False +> cephcontext['ms_bind_ipv6'] = True +> dynamic_ipv6_address = get_ipv6_addr()[0] +> ``` +> +> The default `dynamic_only=True` restricts the result to dynamically-assigned +> (SLAAC/RA) addresses. On a deployment where the provisioning layer assigns +> **static** IPv6 addresses -- in our case MAAS assigning a static per node on each +> of six bound spaces -- there is no dynamic address to find, so the lookup does not +> return the address the node actually holds. +> +> This matches the original report on this bug, where the reporter noted: +> *"I replaced all the get_ipv6_addr() calls with get_ipv6_addr(dynamic_only=False) +> ... The machines have a single global static ipv6 address. Everything works +> correctly."* +> +> The equivalent change has already landed on `charm-ceph-mon`. Is there anything +> blocking the same one-word change on `charm-ceph-osd`, or can that task be moved +> to Fix Committed as well? +> +> **Context on why this matters beyond cosmetics.** With `prefer-ipv6=false` the +> alternative path is also IPv4-only: `get_host_ip()` falls through to +> `socket.inet_aton(hostname)` and then `dns.resolver.query(hostname, 'A')` +> (`hooks/utils.py:208-217`), which cannot resolve an IPv6 literal -- the code's own +> comment acknowledges this "may throw an NXDOMAIN exception ... just let it kill +> the hook". So on a genuinely IPv6-only storage/cluster network, **both** settings +> of `prefer-ipv6` currently fail, in different ways. That second path appears to be +> what LP #2109798 is reporting. + +--- + +## Notes for the operator, NOT part of the comment + +* **Every code claim above was read from the artifact**, not from a repo note: the + `ceph-osd_953.charm` was downloaded and `hooks/utils.py` / `hooks/ceph_hooks.py` + extracted and quoted. Line numbers are from that revision and should be + re-checked if a newer revision has published before filing. +* **Do not cite LP #1590598 in support.** It is `Fix Released` and about + IPv6-**only** host interfaces not being bridged into containers -- a different + defect. Citing it here would be an inverted citation, which this repo has been + bitten by before. +* **This bug is not, by itself, what blocks a v6-only Ceph plane for us.** Measured + separately: setting the charm's `ceph-public-network` / `ceph-cluster-network` + options to the v6 CIDRs means the buggy call is never used for address selection, + because both assignments are guarded by `if not public_network:` / + `if not cluster_network:`. The fix is still worth pursuing -- it removes a + config-dependent workaround and closes a real defect -- but the local path does + not depend on it landing. +* Filing this is the "C" half of the 2026-08-01 ruling. The "B" half (the + storage+replication v6-only experiment) is independent and does not wait on it. diff --git a/docs/changelog-20260731-snap-proxy-apply-ipv6.md b/docs/changelog-20260731-snap-proxy-apply-ipv6.md index eaecca6..339c255 100644 --- a/docs/changelog-20260731-snap-proxy-apply-ipv6.md +++ b/docs/changelog-20260731-snap-proxy-apply-ipv6.md @@ -931,3 +931,52 @@ then `bash scripts/run-tests-all.sh --record-manifest`. Nothing else consumes these files: no runbook, no `preflight.sh` sub-gate and no other script references them, so removal is self-contained. + +--- + +## Item 20 -- next steps executed: nodes released, D-139 execution list replaced, LP draft written + +**(1) Test nodes RELEASED.** `t7ymp6` / `fg6gxm` -> `Ready` / `owner=None`, released +individually and read back. Nothing stranded. G19 was run live before the window closed, which +was the whole reason for holding them. + +**(2) `docs/design-decisions.md` -- D-139 gains a `CORRECTION NOTE 2026-08-01` replacing its +EXECUTION list.** **Neither ruling is touched.** The original list is preserved verbatim inside +the note rather than silently overwritten -- the same reason the struck RFC 6724 rationale was +kept: a later reader must be able to see what was wrong, or they will re-derive it. + +Three defects, in consequence order: +- **SILENT FAILURE (the reason this could not wait):** `dc-node-v6-carve.py` assigns v6 "on + every plane **where it already carries IPv4**", keys the subnet off "the SAME MAAS vlan as + that v4 link", derives the host part from the v4 octet, and its loop body is + `if not v4: continue`. Run after v4 removal it carves **four fewer planes per node and exits + clean**. The original list's "remove v4 LAST" wording actively invited that ordering. +- **UNSATISFIABLE:** "remove v4 LAST, after each is proven" -- per-plane conversion is atomic, + so no such state exists. It read like caution and could not be executed. +- **UNSAFE STEP FUSION:** apex CREATE and RETIRE in one step, with 52 dependent ip-addresses + (the v6 VIP legs) inside the retiring ULA `/64`s. + +Replacement is 7 steps: CREATE-only apex push; MAAS GUA carve ALONGSIDE the ULA; node statics +re-carved **while v4 is still present**; Octavia SAN reissue + `lib-net.sh`; `lb-mgmt` carve; +re-home the 52 VIPs THEN retire; and **v4 removal as a SEPARATE experimental step**, scoped to +`storage` + `replication` TOGETHER per "B plus C" -- together because `ceph-osd`'s `prefer-ipv6` +sets `ms_bind_ipv4=False` globally. Step 7's prerequisite (`network-get` on a v6-only bound +space) is stated as unmet. + +**(3) `docs/audit/lp-draft-20260801-ceph-osd-ipv6-static.md` (NEW).** The "C" half of the +ruling. **A COMMENT on the existing LP #2061836, not a new bug** -- the ceph-mon task is +already `Fix Committed` and only ceph-osd remains `New`. Every code claim quoted from the +downloaded rev-953 artifact, not from a repo note. **Claude does not post to Launchpad; the +operator files it**, per the `lp-draft-20260721` precedent. + +Two things deliberately written INTO the draft's operator notes rather than left implicit: +**do not cite LP #1590598** (`Fix Released`, about IPv6-only host interfaces not being bridged +-- citing it would be the inverted-citation class this repo has twice been bitten by); and +**this bug does not block us locally**, because setting `ceph-public-network` / +`ceph-cluster-network` to the v6 CIDRs bypasses the buggy call. Overstating our dependence on +an upstream fix would be a bad-faith way to raise its priority. + +**Revert.** `git checkout -- docs/design-decisions.md docs/CURRENT-STATE.md` (restores the +defective execution list -- note that reverting reinstates a silent-failure ordering); +`git rm docs/audit/lp-draft-20260801-ceph-osd-ipv6-static.md`. The node release is reversed by +re-deploying, not by git. diff --git a/docs/design-decisions.md b/docs/design-decisions.md index c374152..31c25e9 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -7060,3 +7060,57 @@ understood, and still wrong, because **nobody checked whether the IMPLEMENTATION follows the standard.** For a STANDARDS citation, add a third check: confirm the deployed software implements it, at the deployed version. An RFC is not a description of your system. + +### CORRECTION NOTE 2026-08-01 -- D-139: the EXECUTION list was defective; replaced + +**Not a ruling and not a change to either ruling.** D-139's rulings A and B are untouched; +this replaces the ORDERED EXECUTION list at the end of the entry, which was authored with +three defects, one of them a silent-failure class. The original text is preserved here so a +later reader can see what was wrong rather than wondering why it changed. + +**ORIGINAL, now SUPERSEDED:** *"apex push (prefixes + retire the VR1 ULA rows); MAAS carve per +DC; re-carve 54 node v6 statics (dc-node-v6-carve.py); reissue the Octavia v6 IP SANs; update +lib-net.sh's v6 arm and its harness; carve lb-mgmt a VLAN/space/subnet; remove the v4 subnets +from the five v6-only planes LAST, after each is proven."* + +**DEFECT 1 -- "remove v4 LAST, after each is proven" is UNSATISFIABLE.** Per-plane conversion +is atomic: there is no state in which a plane is proven v6-only while it still holds v4. The +clause reads like caution and cannot be executed. + +**DEFECT 2 -- SILENT FAILURE, and this is the consequential one. +`scripts/dc-node-v6-carve.py` is STRUCTURALLY DEPENDENT ON IPv4 EXISTING.** Verified against +its source: it assigns v6 "on every plane **where it already carries IPv4**", selects the v6 +subnet "on the SAME MAAS vlan as that v4 link", derives the host part from "the last octet of +the node's own v4 address", and its loop body is a bare `if not v4: continue`. **Under D-139 +five planes lose v4 and `lb-mgmt` never had a v4 twin, so run AFTER v4 removal it would carve +FOUR FEWER PLANES PER NODE AND EXIT CLEAN.** Ordering the v4 removal before the v6 carve -- +which the original list's "LAST" wording invited -- would have produced a silently incomplete +carve that every gate reads green. + +**DEFECT 3 -- the apex RETIRE half is not safe in the same step as CREATE.** The dry run +measured **52 dependent ip-addresses** inside the retiring ULA `/64`s (26 per DC, the v6 VIP +legs). Retiring the prefixes before those addresses are re-homed orphans them. + +**REPLACEMENT LIST. Each step individually gated; each is a no-op for the next if skipped.** +1. **Apex push, CREATE ONLY.** `netbox/d139-gua-carve.py` without the retire half. The tool + already reports retirement rather than executing it; keep it that way until step 6. +2. **MAAS carve per DC** -- the GUA `/64`s alongside the existing ULA, not replacing them. + Both families coexist in MAAS at this point; nothing on a node has changed. +3. **Re-carve node v6 statics onto the GUA prefixes WHILE v4 IS STILL PRESENT.** This is the + ordering defect 2 forces. `dc-node-v6-carve.py` works here because every plane still carries + its v4 twin. **If it is ever to run after v4 removal it must first be rewritten to pivot on + the PLANE (space/vlan) rather than on the presence of a v4 link.** +4. **Reissue the Octavia v6 IP SANs** (`octavia-pki.sh reissue` -- the rotation counterpart + this is exactly the case for), then update `lib-net.sh`'s v6 arm and its harness. +5. **Carve `lb-mgmt` its VLAN/space/subnet.** It has never had one anywhere, in either family. +6. **Re-home the 52 VIP ip-addresses onto the GUA prefixes, THEN retire the ULA rows** in the + apex and in MAAS. Only now is the retire half safe. +7. **v4 removal is a SEPARATE, EXPERIMENTAL step and is NOT part of this list.** Per the + 2026-08-01 "B plus C" ruling it applies to `storage` + `replication` ONLY -- and to BOTH + TOGETHER, because `ceph-osd`'s `prefer-ipv6` sets `ms_bind_ipv4 = False` globally and so + couples Ceph's public and cluster networks. `metal-internal`, `data-tenant` and `lb-mgmt` + keep v4 pending upstream fixes. + +**PREREQUISITE FOR STEP 7, STILL NOT TAKEN:** does `network-get` return a v6 address on a +v6-only bound space? It needs a deployed unit, so it rides the next deploy. Step 7 must not +start before it is answered.