diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 6aa0309..7799043 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -992,6 +992,74 @@ `get_relation_ip`/`ipv6` helpers, and whether `hacluster` assigns a v6 VIP independently of the principal's binding. **STAGE 5 IS BLOCKED ON THIS RESEARCH** -- the deploy cannot complete until the six are resolved one way or the other. + **^ THE RESEARCH IS DONE 2026-07-31, AND IT ANSWERS THE QUESTION AGAINST ITS OWN PREMISE.** + Capture `docs/audit/stage5-prefer-ipv6-charm-research-20260731.txt`. Measured by + DOWNLOADING each charm at its pinned channel for `amd64/ubuntu@22.04` and reading the code + and templates inside it -- not Charmhub's `config-yaml`, which answers only whether the knob + is DECLARED and cannot answer what the charm DOES. Revisions read: barbican 265, designate + 418, magnum 96, octavia 571, placement 154, vault 724 (`1.8/stable`), with keystone 857 and + cinder 820 as declaring-charm controls. **`prefer-ipv6` IS NOT WHAT MAKES HAProxy BIND + `:::port`, AND IT NEVER WAS -- FOR ANY OF THE THIRTEEN.** In BOTH template families the v6 + frontend bind is gated on `ipv6_enabled`, which is `not is_ipv6_disabled()` -- a read of the + kernel `net.ipv6.conf.all.disable_ipv6` sysctl (`charmhelpers/contrib/openstack/context.py:1069` + and `charms_openstack/adapters.py:811`, the latter on `APIConfigurationAdapter`, the DEFAULT + adapter for OpenStack API charms). The five charms.openstack templates are BYTE-IDENTICAL to + each other (md5 `f248679f...`) and differ from the charmhelpers one only in variable + namespacing. What `prefer-ipv6` actually sets in that context is `local_host` (the STATS + listener) and `haproxy_host` -- and **`haproxy_host` is DEAD: it is consumed by no template + in any charm downloaded.** **PRECONDITION MEASURED, not assumed:** on a live MAAS-deployed + jammy node in this DC (the dc0 controller, via `juju ssh -m controller 0` from the rack; + hostname read back `subtle-grouse`, matching the recorded inst id) + `net.ipv6.conf.all.disable_ipv6 = 0`, so the `bind :::port` line IS emitted. The v6 VIP is + independently assigned by pacemaker on per-address family detection (`ha/utils.py:286-291` + and `interface_hacluster/common.py:920-927` both branch on `is_ipv6`/`IPv6Address`, never on + the option). **VAULT IS THE STRONGEST CASE AND THE QUESTION IS MALFORMED FOR IT:** vault@1.8 + ships NO haproxy template at all -- its listener is the hardcoded literal + `address = "[::]:8200"` in `templates/vault.hcl.j2`, and `vault_handlers.py:517-534` hands + EVERY vip in the string to hacluster. **`prefer-ipv6` is a UNIT ADDRESS-FAMILY switch, not a + listener switch:** its real effect is that `get_relation_ip()` returns EARLY with + `get_ipv6_addr(...)[0]` (`network/ip.py:617`), so the unit advertises a v6 address on EVERY + relation and the network-space branch below it is never reached; plus `bind_host` `::`, + ip6-localhost stats, and keystone's `sync_db_with_multi_ipv6_addresses`. Undeclared resolves + cleanly to FALSE in both families (`hookenv.config()` -> `.get()` -> None; `getattr(self, + 'prefer_ipv6', False)`). **THE FORK IS THEREFORE DECIDABLE ON EVIDENCE: option (b) -- drop + the option from the six and KEEP their v6 VIP legs -- is what the measurement supports; + option (c) would remove v6 that measurably works and would contradict D-101.** NOT TAKEN + HERE: the ruling is the operator's, the overlay is untouched, R2 is unamended. + **CONSEQUENCE THAT TRAVELS WITH THE RULING: `provider-bundle-check` INVARIANT 9's MECHANISM + CLAIM IS FACTUALLY WRONG** (`scripts/provider-bundle-check.py:30-33`, `:288-295`: "prefer-ipv6 + makes HAProxy bind :::port in ADDITION to *:port, so the two must travel together"). Its + ARITY half was a real measured defect (L3-9) and stays worth having; its COUPLING half + (`prefer6 != dual` FAILS) is what will reject any fix to the six. This is the repo's own + "RULED IS NOT BUILT -- CHECK THE ARTIFACT" class one turn further in: a gate built on a + mechanism nobody read the charm to confirm. **It must be REPLACED with a charm-schema-aware + invariant and proven able to fail in both directions -- never deleted to go green.** + **A NEW AND SEPARATE QUESTION IS RAISED, STATED NOT DECIDED, AND DELIBERATELY NOT FOLDED + INTO THE RULED FORK: what about the SEVEN?** R2 set `prefer-ipv6: true` on all 13 to obtain + v6 VIP binding, which is not what the option does. On the seven that declare it, true means + every relation carries an IPv6 address chosen from the unit's primary interface, IGNORING + the network-space binding -- in a bundle that binds relations across six named spaces. After + the six are fixed the deploy would carry seven charms advertising v6 relation addresses and + forty-nine advertising v4. Never observed live (attempt 1 aborted atomically; the model is + empty). R2 / D-101 ruling-surface material, its own GA-R5 exchange. + **FULL OPTION-NAME SWEEP OF THE dc0 DEPLOY INPUT, and it is CLEAN apart from this defect.** + Because `--dry-run` does not validate option NAMES and juju aborts the whole bundle on the + first bad one, every option assignment in attempt 1's exact input (`bundle.yaml` + + `vr1-dc0-vips` + `vr1-dc0-machines` + `vr1-dc0-octavia-pki`) was compared against each + charm's own `config.yaml` at its pinned channel: **56 applications, 81 option assignments, + 20 charm schemas, 0 unresolvable, and the ONLY findings are `prefer-ipv6` on the same six.** + So no further option-name failure is waiting for attempt 2. An unreadable or unrecognised + schema REFUSES rather than passing. **A REPO GATE FOR THIS IS OWED, NOT BUILT** (hard rule 1 + + the standing directive); the method is reproduced verbatim in the capture. + **RUNBOOK DEFECT FOUND IN PASSING, LOGGED NOT FIXED (DOCFIX material):** + `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:553-557` gives the dc0 deploy WITHOUT + `overlays/vr1-dc0-machines.yaml`, while the dc1 block three lines below includes its + machines overlay. Attempt 1 as executed correctly included it. **The overlay is NOT a + no-op** -- measured, the only delta it makes to the merged input is + `ovn-chassis.options.bridge-interface-mappings = 'br-ex:52:54:00:8c:2a:8c + br-ex:52:54:00:50:48:88'`, the two dc0 compute provider MACs. Deploying dc0 exactly as the + runbook reads would leave ovn-chassis with no provider bridge mapping, surfacing later as + tenant networks with no external path rather than as a deploy error. - Project: Omega Cloud, VR1 DC-DC rehearsal -- a two-DC + Office1-headend virtual rehearsal on KVM (vcloud host), rehearsing the future bare-metal diff --git a/docs/audit/stage5-prefer-ipv6-charm-research-20260731.txt b/docs/audit/stage5-prefer-ipv6-charm-research-20260731.txt new file mode 100644 index 0000000..9811b5e --- /dev/null +++ b/docs/audit/stage5-prefer-ipv6-charm-research-20260731.txt @@ -0,0 +1,372 @@ +STAGE 5 -- `prefer-ipv6` / v6 VIP RESEARCH (2026-07-31) +====================================================== + +THE RULED QUESTION (GA-R5, 2026-07-31; operator utterance: "Research what those 6 +charms do with v6 VIPs first (Recommended)"), restated verbatim from +docs/CURRENT-STATE.md section 1: + + for `barbican`, `designate`, `magnum`, `octavia`, `placement` and `vault` at their + pinned channels, does the charm's generated HAProxy configuration bind `:::port` + (or the v6 VIP explicitly) WITHOUT a `prefer-ipv6` option -- or does it bind v4 + only, making a v6 VIP leg an address pacemaker manages and nothing listens on? + +>>> ANSWER: THEY BIND v6. `prefer-ipv6` IS NOT WHAT MAKES HAProxy BIND `:::port`, +>>> AND IT NEVER WAS -- FOR ANY OF THE THIRTEEN. The `:::port` frontend bind is +>>> gated on a HOST-LEVEL kernel check (`net.ipv6.conf.all.disable_ipv6`), which is +>>> MEASURED `0` on a live node in this DC. The v6 VIP is assigned by pacemaker on +>>> per-address family detection, also independent of the option. Option (b) -- +>>> drop `prefer-ipv6` from the six and KEEP their v6 VIP legs -- is supported by +>>> measurement. Option (c) would remove working v6 for no reason. + +METHOD -- and why it is the charm ARTIFACT and not Charmhub's `config-yaml` +-------------------------------------------------------------------------- +The 2026-07-31 attempt-1 root cause was measured against Charmhub's `config-yaml`, +which answers ONLY "is the knob declared". It cannot answer what the charm DOES. +This research downloads the charm each app actually deploys, at its pinned channel, +for the base and arch these nodes run, and reads the code and templates inside it. + + Run FROM voffice1 (has juju 3.6.27 + egress). Staged under a plain $HOME dir + (~/charmwork) because the juju SNAP has a private /tmp and its `home` interface + grants NON-HIDDEN files only -- both traps are already recorded in the phase-4 + runbook. + + juju download --channel --base ubuntu@22.04 --arch amd64 \ + --no-progress --filepath ~/charmwork/.charm + + `unzip` is ABSENT on voffice1; extraction used python3 zipfile. + +PROVENANCE -- exact revisions read (2024.1/stable and 1.8/stable MOVE; these are the +revisions this research measured, quoted from juju's own "Fetching charm" line): + + THE SIX (do NOT declare prefer-ipv6) THE CONTROLS (DO declare it) + barbican 2024.1/stable rev 265 keystone 2024.1/stable rev 857 + designate 2024.1/stable rev 418 cinder 2024.1/stable rev 820 + magnum 2024.1/stable rev 96 + octavia 2024.1/stable rev 571 + placement 2024.1/stable rev 154 + vault 1.8/stable rev 724 + + INDEPENDENT CONFIRMATION OF THE 7/6 SPLIT, from the artifact rather than the API: + `grep -c prefer-ipv6 /config.yaml` = 0 for all six. Same answer as the + Charmhub query, reached by a different instrument. + +FINDING 1 -- THE FRONTEND v6 BIND IS GATED ON `ipv6_enabled`, NOT ON `prefer-ipv6` +--------------------------------------------------------------------------------- +Two template families exist, and BOTH emit the v6 bind under the same predicate. + + (a) charms.openstack family -- barbican, designate, magnum, octavia, placement + templates/haproxy.cfg (md5 f248679faf2ff083dc7d4f8b34186700, BYTE-IDENTICAL + across all five): + 62: bind *:{{ ports[0] }} + 63: {% if options.ipv6_enabled -%} + 64: bind :::{{ ports[0] }} + + (b) charmhelpers family -- keystone, cinder (and the other five declaring charms) + charmhelpers/contrib/openstack/templates/haproxy.cfg + (md5 92ea339fc054eeb54eecdf9571ba5e80, identical keystone vs cinder): + 67: bind *:{{ ports[0] }} + 68: {% if ipv6_enabled -%} + 69: bind :::{{ ports[0] }} + + The two templates differ ONLY in variable namespacing (`options.` prefix). The + bind logic is the same. + +WHERE `ipv6_enabled` COMES FROM -- a kernel sysctl, in both families: + + charmhelpers/contrib/openstack/context.py:1069 (HAProxyContext) + ctxt['ipv6_enabled'] = not is_ipv6_disabled() + + charms_openstack/adapters.py:811 (APIConfigurationAdapter -- the DEFAULT + configuration_class for OpenStack API charms, charms_openstack/charm/classes.py:593) + @property + def ipv6_enabled(self): + return not ch_ip.is_ipv6_disabled() + + charmhelpers/contrib/network/ip.py:242 + def is_ipv6_disabled(): + ... sysctl net.ipv6.conf.all.disable_ipv6 ... + return "net.ipv6.conf.all.disable_ipv6 = 1" in result + + `prefer-ipv6` appears NOWHERE in that path. + +WHAT `prefer-ipv6` SETS IN HAProxyContext, and why it does not reach the frontend: + + charmhelpers/contrib/openstack/context.py:1062-1067 + if config('prefer-ipv6'): + ctxt['local_host'] = 'ip6-localhost' + ctxt['haproxy_host'] = '::' + else: + ctxt['local_host'] = '127.0.0.1' + ctxt['haproxy_host'] = '0.0.0.0' + + `local_host` binds the STATS listener only. **`haproxy_host` is DEAD** -- a grep + for it across every downloaded charm's template directories returns NOTHING. It is + assigned in the context and consumed by no template. So the one value `prefer-ipv6` + sets that sounds like a bind address is never rendered. + +FINDING 2 -- THE PRECONDITION IS MEASURED, NOT ASSUMED +------------------------------------------------------ +`ipv6_enabled` is true only if IPv6 is enabled on the machine. Measured on a LIVE +MAAS-deployed jammy node in this DC -- the dc0 Juju controller machine, reached with +`juju ssh -m controller 0` from the dc0 rack: + + subtle-grouse + net.ipv6.conf.all.disable_ipv6 = 0 + enp1s0 fd50:840e:74e2:220::5/64 + + `subtle-grouse` is the inst id CURRENT-STATE records for the bootstrapped + controller, so the instrument is reading the intended host. The nodes the bundle + will land on come from the same MAAS image and the same v6 carve + (`dc-node-v6-carve.py check vr1-dc0` = 54 links correct). `is_ipv6_disabled()` + therefore returns False and the `bind :::port` line IS emitted. + +FINDING 3 -- THE v6 VIP IS ASSIGNED BY PACEMAKER REGARDLESS OF THE OPTION +------------------------------------------------------------------------ +The resource type is chosen per address, by family detection on the `vip` string: + + charmhelpers/contrib/openstack/ha/utils.py:286-291 (update_hacluster_vip) + for vip in cluster_config['vip'].split(): + if is_ipv6(vip): + res_vip = 'ocf:heartbeat:IPv6addr'; vip_params = 'ipv6addr' + else: + res_vip = 'ocf:heartbeat:IPaddr2'; vip_params = 'ip' + + interface_hacluster/common.py:920-927 (VirtualIP.configure_resource -- the path + vault uses, since vault talks to hacluster directly): + ipaddr = ipaddress.ip_address(self.vip) + if isinstance(ipaddr, ipaddress.IPv4Address): res_type = 'ocf:heartbeat:IPaddr2' + else: res_type = 'ocf:heartbeat:IPv6addr' + + `prefer-ipv6` is not consulted in either. A v6 address in the `vip` string gets an + IPv6addr resource whether or not the option exists. + +FINDING 4 -- VAULT IS NOT AN HAProxy CHARM, AND ITS LISTENER IS ALREADY DUAL-STACK +---------------------------------------------------------------------------------- +The ruled question is malformed for vault, and the answer is stronger than for the +other five. vault@1.8 rev 724 ships NO haproxy template at all (`find` returns +nothing). Its listener is a hardcoded literal in templates/vault.hcl.j2: + + listener "tcp" { + address = "[::]:8200" + + `[::]` is the v6 wildcard; with Linux's default `bindv6only=0` it accepts v4 as + well. Vault binds v6 unconditionally and always has. + + Its VIPs reach pacemaker through reactive/vault_handlers.py:517-534, which iterates + EVERY entry of `config('vip')` and calls `hacluster.add_vip(...)` -- so all six of + vault's legs (3 v4 + 3 v6) are managed. Separately, lib/charm/vault.py:161-180 + `get_vip(binding)` selects ONE vip matched to the binding's CIDR for vault's own + advertised api_addr; with a dual-family sextet that resolves to the v4 leg of the + bound plane. That is address SELECTION for self-advertisement, not binding, and it + is unaffected by `prefer-ipv6` (which vault does not have). + +FINDING 5 -- WHAT `prefer-ipv6` ACTUALLY DOES (this is the load-bearing part) +---------------------------------------------------------------------------- +It is a UNIT ADDRESS-FAMILY switch, not a listener switch. Every consumer found: + + 1. charmhelpers/contrib/network/ip.py:617 -- `get_relation_ip()`: + if config('prefer-ipv6'): + assert_charm_supports_ipv6() + return get_ipv6_addr(dynamic_only=False)[0] + elif cidr_network: ... + This returns EARLY. The unit advertises an IPv6 address on EVERY relation, and + the `cidr_network` / network-space branch below it is never reached. + 2. charmhelpers/contrib/openstack/ip.py:223 -- resolve_address() fallback becomes + `get_ipv6_addr(exc_list=vips)[0]`. + 3. charmhelpers/contrib/openstack/context.py:1712 -- BindHostContext: the API + service's own `bind_host` becomes `::` instead of `0.0.0.0`. + 4. context.py:1062 -- HAProxy STATS listener binds ip6-localhost. + 5. Per-charm: keystone_hooks.py:231/315 call `sync_db_with_multi_ipv6_addresses()` + (registers keystone's DB grants against its v6 addresses) and `setup_ipv6()`; + keystone_utils.py:957 makes the local bypass endpoint a v6 literal. + (`setup_ipv6()` at keystone_utils.py:2375 is effectively a NO-OP on jammy -- + its only action is a trusty/pre-liberty haproxy backport.) + 6. charms_openstack/adapters.py:808 -- `ipv6_mode` = `getattr(self,'prefer_ipv6', + False)`, driving `local_address` to `get_ipv6_addr()`. + + FOR THE SIX, undeclared resolves cleanly to FALSE in both families -- this is not + an exception path, it is the ordinary default: + - charmhelpers `hookenv.config(scope)` returns `_cache_config.get(scope)` + (hookenv.py:450) -> None -> falsy for a key the charm does not declare. + - charms.openstack `getattr(self, 'prefer_ipv6', False)` -> False, because the + adapter only grows attributes for keys the charm declares. + +CONSEQUENCE 1 -- `provider-bundle-check` INVARIANT 9's RATIONALE IS FACTUALLY WRONG +---------------------------------------------------------------------------------- +scripts/provider-bundle-check.py:30-33 and :288-295 state: + + "prefer-ipv6 makes HAProxy bind :::port in ADDITION to *:port, so the two must + travel together" + +That mechanism claim is REFUTED above: the `:::port` bind is gated on the host +sysctl. The invariant's OTHER half -- that a merge order dropping the v6 legs while +keeping the option exits 0 -- was a real measured defect (L3-9) about arity, and the +arity check itself is still worth having. But the coupling as coded (`prefer6 != +dual` is a FAIL) is what will reject any fix to the six. + + This is the repo's own "RULED IS NOT BUILT -- CHECK THE ARTIFACT" class, one turn + further in: a gate was built on a mechanism nobody read the charm to confirm. + + WHEN A RULING LANDS, INVARIANT 9 MUST BE REPLACED, NEVER DELETED TO GO GREEN (the + standing rule that a checker which cannot fail is not a gate; and the rule that a + test asserting a literal finding string gets RE-POINTED at the new invariant). + The replacement invariant has to be charm-schema-aware: `prefer-ipv6` may only + appear on a charm that DECLARES it, and dual-family arity is asserted on its own. + It must be proven able to fail in both directions. + +CONSEQUENCE 2 -- A NEW QUESTION ABOUT THE **SEVEN**, WHICH THIS RESEARCH RAISES AND +DOES NOT ANSWER +----------------------------------------------------------------------------------- +R2 set `prefer-ipv6: true` on all 13 in order to get v6 VIP binding. That is not what +the option does. On the seven that DO declare it, setting it true has the Finding-5 +effects -- most consequentially, `get_relation_ip()` returns early and hands every +relation an IPv6 address chosen from the unit's primary interface, IGNORING the +network-space binding. This bundle binds relations across six named spaces. + + So after the six are fixed, the deploy would carry SEVEN charms advertising v6 + relation addresses and forty-nine advertising v4. That asymmetry is a deploy risk + that has never been observed live (attempt 1 aborted atomically; the model is + empty), and it is a DIFFERENT question from the ruled one. + + STATED, NOT DECIDED, and NOT folded into the question below. It is R2 / D-101 + ruling-surface material and belongs in its own GA-R5 exchange. + +WHAT IS PUT TO THE OPERATOR (the ruled fork, now decidable on evidence) +---------------------------------------------------------------------- + (b) Drop `prefer-ipv6` from the six, KEEP their dual-family v6 VIP legs. + SUPPORTED BY THE MEASUREMENT ABOVE. Their HAProxy binds `:::port`, pacemaker + assigns the v6 VIP, and vault already listens on `[::]`. + (c) Drop both, making those six v4-only. + Would remove v6 that measurably works, and would contradict D-101 (dual-stack + where IPv4 is required, IPv6-only where sound, BOTH DCs this deployment). + + The overlays are GENERATED (D-136 (D)). Whatever is ruled is applied by editing + render/values/vr1-dc{0,1}-vips.yaml and re-rendering -- never by hand-editing + overlays/, which `tests/render-drift` exists to catch. BOTH DCs, symmetric. + +ADDITIONAL MEASUREMENT -- FULL OPTION-NAME SWEEP OF THE dc0 DEPLOY INPUT +----------------------------------------------------------------------- +Attempt 1 proved `juju deploy --dry-run` does not validate config option NAMES, and +juju aborts the WHOLE bundle atomically on the first bad one. So a second unknown +option elsewhere would cost another full attempt. Every option assignment in the +actual dc0 deploy input was therefore checked against each charm's own config.yaml, +downloaded at its pinned channel: + + deploy input: bundle.yaml + overlays/vr1-dc0-vips.yaml + + overlays/vr1-dc0-machines.yaml + + overlays/vr1-dc0-octavia-pki.yaml (= attempt 1's set) + 56 applications; 81 option assignments; 20 charm schemas; 0 unresolvable. + + [FAIL] barbican barbican@2024.1/stable does not declare option 'prefer-ipv6' + [FAIL] designate designate@2024.1/stable does not declare option 'prefer-ipv6' + [FAIL] magnum magnum@2024.1/stable does not declare option 'prefer-ipv6' + [FAIL] octavia octavia@2024.1/stable does not declare option 'prefer-ipv6' + [FAIL] placement placement@2024.1/stable does not declare option 'prefer-ipv6' + [FAIL] vault vault@1.8/stable does not declare option 'prefer-ipv6' + +>>> `prefer-ipv6` ON THOSE SIX IS THE ONLY UNKNOWN OPTION IN THE ENTIRE DEPLOY INPUT. +Once it is resolved, no further option-name failure is waiting for attempt 2. + + Schemas read (revision at 2024.1/stable unless noted): barbican 265, ceph-mon 491, + ceph-osd 953, ceph-radosgw 600 (squid/stable), cinder 820, designate 418, + glance 681, glance-simplestreams-sync 152, hacluster 166 (2.4/stable), + keystone 857, magnum 96, neutron-api 710, nova-cloud-controller 823, + nova-compute 894, octavia 571, octavia-diskimage-retrofit 257, + openstack-dashboard 750, ovn-chassis 396 (24.03/stable), placement 154, + vault 724 (1.8/stable). + + REFUSAL BEHAVIOUR: a charm whose config.yaml could not be read, or whose shape is + unrecognised, is REPORTED and exits 2 -- "could not look" is never "nothing there". + Zero occurred here. + + A REPO GATE FOR THIS IS **OWED, NOT BUILT** (hard rule 1 + the standing directive: + hardening is out of scope unless it blocks the deploy, and this one-shot + measurement already de-risks attempt 2). CURRENT-STATE already records that no gate + in this repo reads a charm config schema. The method is reproduced verbatim at the + end of this file so it is not lost. + +OBSERVATION -- A RUNBOOK DEFECT FOUND IN PASSING (DOCFIX material, LOGGED NOT FIXED) +------------------------------------------------------------------------------------ +runbooks/dc-dc-phase4-juju-bundle-per-dc.md:553-557 gives the dc0 deploy as + + juju deploy ./bundle.yaml \ + --overlay overlays/vr1-dc0-vips.yaml \ + --overlay overlays/vr1-dc0-octavia-pki.yaml + +-- WITHOUT `overlays/vr1-dc0-machines.yaml`, while the dc1 block three lines below +DOES include its machines overlay. Attempt 1 (executed) correctly included it. + +MEASURED delta between the two merged inputs -- the overlay is NOT a no-op: + + ONLY-WITH-OVERLAY .applications.ovn-chassis.options.bridge-interface-mappings + = 'br-ex:52:54:00:8c:2a:8c br-ex:52:54:00:50:48:88' + +That is the pair of dc0 compute provider MACs. Deploying dc0 exactly as the runbook +reads would give ovn-chassis NO provider bridge mapping, and the failure would appear +later as tenant networks with no external path -- not as a deploy error. + +REPRODUCTION -- the sweep, verbatim +----------------------------------- +Run on voffice1 with the repo clone at the branch HEAD and the charm schemas staged +under ~/charmwork/x___/config.yaml: + + # 1. enumerate the unique charm@channel pairs that carry options + python3 - . overlays/vr1-dc0-vips.yaml overlays/vr1-dc0-machines.yaml \ + overlays/vr1-dc0-octavia-pki.yaml <<'PY' + import sys, yaml, os + def dm(b,o): + for k,v in (o or {}).items(): + if isinstance(v,dict) and isinstance(b.get(k),dict): dm(b[k],v) + else: b[k]=v + return b + repo=sys.argv[1] + m=yaml.safe_load(open(os.path.join(repo,'bundle.yaml'))) + for ov in sys.argv[2:]: dm(m, yaml.safe_load(open(os.path.join(repo,ov))) or {}) + seen=set() + for a,s in sorted((m.get('applications') or {}).items()): + s=s or {} + if not (s.get('options') or {}): continue + seen.add((s.get('charm'), s.get('channel'))) + for c,ch in sorted(seen, key=lambda x:(str(x[0]),str(x[1]))): print("%s %s"%(c,ch)) + PY + + # 2. download each pair's charm and keep only its config.yaml + while read -r c ch; do + d="x_${c}__$(echo "$ch" | tr '/' '_')" + juju download "$c" --channel "$ch" --base ubuntu@22.04 --arch amd64 \ + --no-progress --filepath ~/charmwork/dl.charm + mkdir -p "$d" + python3 -c 'import zipfile,sys; z=zipfile.ZipFile("dl.charm"); [z.extract(n,sys.argv[1]) for n in z.namelist() if n=="config.yaml"]' "$d" + done < pairs.tsv + + # 3. merge the deploy input the way juju does (dicts merge, scalars replace) and + # compare every app's option KEYS against its charm's declared keys. An + # unreadable or unrecognised schema REFUSES (exit 2) rather than passing. + + The merge semantics matter: juju DEEP-MERGES overlay `options` (measured + 2026-07-29 against charm v12.1.1 source and recorded in the session ledger), so a + replace-style merge would under-report. + +HOSTS AND INSTRUMENT IDENTITY +----------------------------- + charm downloads + schema reads : voffice1 (juju 3.6.27, egress) + live IPv6 sysctl : dc0 Juju controller machine via + `juju ssh -m controller 0` FROM the dc0 rack + (D-138 client host); hostname read back as + `subtle-grouse`, matching the recorded inst id + deploy-input hashes cross-checked on all three hosts before anything was trusted: + bundle.yaml 5cc1542fc53ebf05caf6c51b502d84fcdebcacbfb3b78ae651f2db384eba91f7 + overlays/vr1-dc0-vips.yaml 3ae79e820a3fe6ed23edb04ca700e2b6b85479b683936a2da787121fe4611b7d + overlays/vr1-dc0-machines.yaml b70e4eedc2bdd3157e7bd78fcf582e44e98997eaca8a838085672ed1dcaabac9 + overlays/vr1-dc0-octavia-pki.yaml 5fc117f1883320422f1854b4a82c9c8025dd912326b25fa6ad95686dad40227f + (identical vcloud / voffice1 / dc0 rack ~/repo-stage; the PKI overlay is + gitignored and was compared voffice1 vs rack only, where it exists) + NOTE: voffice1's clone is at 8c16bbe, two commits behind vcloud's ca2d636. Those + two commits touch the skill and permission rules only; the four deploy-input files + were hash-compared rather than assumed equal. + +STATUS OF STAGE 5 AFTER THIS RESEARCH +------------------------------------- +Still BLOCKED, now on the RULING rather than on the research. Nothing was edited: the +overlay is untouched, R2 is unamended, invariant 9 is unchanged. The model remains +empty. diff --git a/docs/changelog-20260731-prefer-ipv6-research.md b/docs/changelog-20260731-prefer-ipv6-research.md new file mode 100644 index 0000000..372e244 --- /dev/null +++ b/docs/changelog-20260731-prefer-ipv6-research.md @@ -0,0 +1,112 @@ +# Changelog 2026-07-31 -- the `prefer-ipv6` / v6 VIP charm research + +Session changelog (GA-R2/D1: ONE per session). Branch `dc-dc-stage5-preconditions`. +Status claims live ONLY in `docs/CURRENT-STATE.md`. + +Context: bundle deploy attempt 1 failed 2026-07-31 with +`unknown option "prefer-ipv6"` on barbican, and the operator ruled *"Research what +those 6 charms do with v6 VIPs first (Recommended)"*. Stage 5 is blocked on that +research. This session executes it. **Read-only throughout: no cloud mutation, no +overlay edit, no ruling adopted.** + +--- + +## Item 1 -- `docs/audit/stage5-prefer-ipv6-charm-research-20260731.txt` (NEW) + +**What.** The capture answering the ruled question, plus two findings that came out +of the same measurement. + +**Method, and why it is not the Charmhub `config-yaml` API.** The attempt-1 root +cause was measured against `config-yaml`, which answers only *is the knob declared*. +It cannot answer *what the charm does*. This research downloads the charm each app +actually deploys -- `juju download --channel --base ubuntu@22.04 +--arch amd64` from voffice1 -- and reads the templates and code inside it. Revisions +are recorded in the capture because `2024.1/stable` moves. Two declaring charms +(keystone 857, cinder 820) were downloaded as CONTROLS: without one, "binds v4 only" +and "binds v6 by another mechanism" cannot be told apart. + +**The answer.** `prefer-ipv6` is not what makes HAProxy bind `:::port`, and never was, +for any of the thirteen. In both template families the v6 frontend bind is gated on +`ipv6_enabled = not is_ipv6_disabled()` -- a read of the kernel +`net.ipv6.conf.all.disable_ipv6` sysctl. Measured `0` on a live MAAS-deployed jammy +node in this DC. The v6 VIP is assigned by pacemaker on per-address family detection, +also independent of the option. Vault ships no haproxy template at all and its +listener is the hardcoded literal `[::]:8200`. + +**What the option really does:** it is a unit address-family switch -- +`get_relation_ip()` returns early with the unit's IPv6 address for EVERY relation, +ignoring the network-space binding; plus `bind_host ::`, ip6-localhost stats, and +keystone's `sync_db_with_multi_ipv6_addresses`. The value that sounds like a bind +address, `haproxy_host`, is consumed by NO template in any charm downloaded. + +**Consequence recorded, not acted on:** `provider-bundle-check` invariant 9's +mechanism claim is factually wrong; when a ruling lands the invariant must be +REPLACED with a charm-schema-aware one and proven able to fail in both directions, +never deleted to go green. + +**Revert.** `git rm docs/audit/stage5-prefer-ipv6-charm-research-20260731.txt`. The +file is a read-only capture; nothing depends on it but the CURRENT-STATE entry in +item 2. + +## Item 2 -- `docs/CURRENT-STATE.md` section 1: the research result + +**What.** The research outcome appended under the 2026-07-31 ruling block that +records the question -- the answer, its evidence, the invariant-9 consequence, the +new question about the seven declaring charms, the option-name sweep result, and the +runbook defect. Required in the same commit by GA-R1 rule 8 (C1); repo-lint L10 +enforces it. + +**Revert.** `git revert` this commit, or delete the appended block; the preceding +ruling text is unchanged and still reads correctly on its own. + +## Item 3 -- MEASUREMENT ONLY (no artifact): full option-name sweep of the dc0 deploy input + +**What.** Attempt 1 proved `juju deploy --dry-run` does not validate config option +NAMES, and juju aborts the whole bundle atomically on the first bad one -- so a second +unknown option elsewhere costs another full attempt. Every option assignment in +attempt 1's exact input (`bundle.yaml` + `vr1-dc0-vips` + `vr1-dc0-machines` + +`vr1-dc0-octavia-pki`) was compared against each charm's own `config.yaml` downloaded +at its pinned channel. + +**Result: 56 applications, 81 option assignments, 20 charm schemas, 0 unresolvable, +and the only findings are `prefer-ipv6` on the same six.** No further option-name +failure is waiting for attempt 2. An unreadable or unrecognised schema REFUSES rather +than passing. + +**Not built as a gate, deliberately.** Hard rule 1 plus the standing operator +directive: hardening is out of scope unless it blocks the deploy, and this one-shot +measurement already de-risks attempt 2. The method is reproduced verbatim in the +capture so it can be re-run or promoted later. CURRENT-STATE already records that no +gate in this repo reads a charm config schema. + +**Revert.** Nothing to revert -- no repo artifact was created by this item. + +## Item 4 -- LOGGED NOT FIXED: the dc0 deploy block omits a non-no-op overlay + +**What.** `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:553-557` gives the dc0 deploy +without `overlays/vr1-dc0-machines.yaml`, while the dc1 block three lines below +includes its machines overlay. Measured, the overlay's ONLY delta to the merged input +is `ovn-chassis.options.bridge-interface-mappings = 'br-ex:52:54:00:8c:2a:8c +br-ex:52:54:00:50:48:88'` -- the two dc0 compute provider MACs. Deploying dc0 exactly +as the runbook reads would leave ovn-chassis with no provider bridge mapping, which +surfaces later as tenant networks with no external path, not as a deploy error. +Attempt 1 as executed correctly included the overlay. + +**Why not fixed here.** Hard rule 1 -- this session's step is the research. It is +DOCFIX material and is recorded in CURRENT-STATE and in the capture so it cannot be +lost. + +**Revert.** Nothing to revert. + +--- + +## What was NOT done, stated so it is not assumed + +- **The overlay is untouched and R2 is unamended.** The operator's ruling was to + research, not to fix; the fork between (b) and (c) is theirs. +- **No `prefer-ipv6` was removed from anything**, including the seven charms that + declare it -- that is a separate, newly-raised question. +- **`provider-bundle-check` is unchanged.** Its invariant 9 is now known to rest on a + wrong mechanism claim, but changing a gate before the ruling it gates would be the + wrong order. +- **No cloud state changed.** The `vr1-dc0` model is still empty.