STAGE 5 / D-139 -- WHAT THE CHARMS DO ON A v6-ONLY PLANE Research 2026-08-01, read from Charmhub ARTIFACTS (base ubuntu@22.04, amd64) + vendor docs. Staged /tmp/v6research (NOT in the repo). No cloud mutation, no repo write. ARTIFACT PROVENANCE (charm, channel, revision, sha256 prefix of the .charm) ceph-mon squid/stable rev 491 4fec8dc10971ac7b ceph-osd squid/stable rev 953 60699796f134a2d0 ceph-radosgw squid/stable rev 600 cede6d47c0590213 mysql-innodb-cluster 8.0/stable rev 164 457225589673d835 mysql-router 8.0/stable rev 1154 eb3f21bed93a4490 ovn-central 24.03/stable rev 311 0946d7d6e6ed84d1 ovn-chassis 24.03/stable rev 396 ec5cad529e813466 rabbitmq-server 3.9/stable rev 295 25dba936d5647463 hacluster 2.4/stable rev 166 ce38ccaa2d4e56bf Acquired via api.charmhub.io (juju is NOT installed on vcloud), base/arch pinned in the query. TOPOLOGY FACTS USED (from the repo, not inferred) bundle.yaml default-base ubuntu@22.04/stable; ALL 56 apps default-bind ('') to metal-admin. ALL 56 apps bind at least one endpoint into one of the five v6-only planes. VR1 nodes have 6 flat per-plane NICs: enp1s0 metal-admin(PXE), enp2s0 provider-public, enp3s0 metal-internal, enp4s0 data-tenant, enp5s0 storage, enp6s0 replication (docs/CURRENT-STATE.md:1941-1943, :762-764). Node v6 is MAAS mode=STATIC (scripts/dc-node-v6-carve.py:180-181), no SLAAC, no RA (all six v6 subnets gateway_ip=None, D-139). ceph-mon / mysql-innodb-cluster / ovn-central / rabbitmq-server / ceph-radosgw run in LXD (bundle to: lxd:0..2). ceph-osd and nova-compute run on METAL. ========================================================================= 1. THE OPTION'S ADDRESS LOOKUP IS ANCHORED TO private-address, NOT THE SPACE ========================================================================= charmhelpers/contrib/network/ip.py:340-351 @sniff_iface injects kwargs['iface'] = get_iface_from_addr(unit_get('private-address')) into get_ipv6_addr(). private-address = the DEFAULT binding = metal-admin = enp1s0/eth0. So prefer-ipv6 NEVER reads the bound space's address. Confirms and extends the 2026-07-31 capture's Finding 5. TWO DISTINCT FAILURE MODES, both already evidenced in this repo: (a) LXD units: container eth0 (metal-admin, stays DUAL-STACK under D-139) gets ONE family from juju's addrs[0] -- today v4 (D-139's own measurement). No global v6 on eth0 => Exception: Interface 'eth0' does not have a scope global non-temporary ipv6 address. ALREADY OBSERVED LIVE 2026-07-31 (docs/CURRENT-STATE.md:1332; D-101 ruling note (b)). (b) METAL units with STATIC v6: ceph-osd and hacluster call get_ipv6_addr() WITHOUT dynamic_only=False. ip.py:397-411 then requires the line to match inet6 (.+)/[0-9]+ scope global.* dynamic.* AND the address to endswith the EUI-64 MAC suffix. A MAAS mode=STATIC address is neither => addrs empty => ip.py:416-419 raises the SAME exception. ceph-mon uses get_ipv6_addr(dynamic_only=False) and is NOT exposed to (b). ceph-osd hooks/ceph_hooks.py:547 dynamic_ipv6_address = get_ipv6_addr()[0] ceph-osd hooks/utils.py:205 return get_ipv6_addr()[0] hacluster hooks/utils.py:454 return utils.get_ipv6_addr(exc_list=excludes)[0] hacluster hooks/utils.py:214 ipv6_addr = utils.get_ipv6_addr(iface=iface)[0] ceph-mon src/ceph_hooks.py:210 local_addresses = get_ipv6_addr(dynamic_only=False) ceph-mon src/utils.py:132 return get_ipv6_addr(dynamic_only=False)[0] ========================================================================= 2. CEPH ========================================================================= VENDOR (docs.ceph.com/en/squid/rados/configuration/network-config-ref/): ms_bind_ipv4 type bool default: true ms_bind_ipv6 type bool default: false So a v6-only cluster needs BOTH flipped. Charm emits BOTH -- ONLY under prefer-ipv6. ceph-mon src/ceph_hooks.py:207-236 if config('prefer-ipv6'): cephcontext['ms_bind_ipv4'] = False cephcontext['ms_bind_ipv6'] = True local_addresses = get_ipv6_addr(dynamic_only=False) public_addr = get_public_addr(); cluster_addr = get_cluster_addr() if public_addr not in local_addresses: -> WARNING, public_addr = local_addresses[0] if cluster_addr not in local_addresses: -> WARNING, cluster_addr = local_addresses[0] else: cephcontext['public_addr'] = get_public_addr() cephcontext['cluster_addr'] = get_cluster_addr() templates/ceph.conf:26-31 (ceph-osd templates/ceph.conf:18-23) render exactly: {% if ms_bind_ipv6 %}ms_bind_ipv6 = true{%- endif %} {%- if ms_bind_ipv4 == false %}ms_bind_ipv4 = false{% endif %} ceph-osd hooks/ceph_hooks.py:544-551 -- CRUDER, and this is the decisive asymmetry: if config('prefer-ipv6'): cephcontext['ms_bind_ipv4'] = False cephcontext['ms_bind_ipv6'] = True dynamic_ipv6_address = get_ipv6_addr()[0] # dynamic_only=True <-- BUG if not public_network: cephcontext['public_addr'] = dynamic_ipv6_address if not cluster_network: cephcontext['cluster_addr'] = dynamic_ipv6_address -> ONE address used for BOTH public and cluster, taken from the metal-admin NIC. -> raises on static addressing before it even gets that far. Bundle sets NEITHER ceph-public-network NOR ceph-cluster-network (measured from bundle.yaml: ceph-mon options = expected-osd-count/monitor-count/source; ceph-osd = osd-devices/source). So both charms fall through to network_get_primary_address('public'|'cluster') -- utils.py get_public_addr()/get_cluster_addr() -- i.e. SPACE-AWARE, and correct on v6-only storage/replication. That path is only reached when prefer-ipv6 is FALSE. NET: prefer-ipv6 FALSE + v6-only storage/replication public addr / cluster addr = correct per-space v6 (space-aware, unbracketed) BUT ms_bind_ipv6 absent => Ceph defaults ms_bind_ipv6=false / ms_bind_ipv4=true => daemons will not bind the v6 address. BROKEN. prefer-ipv6 TRUE ms_bind_* correct, but addresses are taken from metal-admin, collapsing storage+replication onto the admin plane; and on ceph-osd it raises first. BROKEN DIFFERENTLY. => NEITHER SETTING OF prefer-ipv6 GIVES A CORRECT CEPH ON THIS TOPOLOGY. THE SURGICAL PATH THAT EXISTS (verified option name, verified render path): `config-flags` IS declared by BOTH ceph-mon and ceph-osd (config.yaml). charmhelpers/contrib/storage/linux/ceph.py:2303-2339 CephConfContext; permitted sections ceph-mon src/ceph_hooks.py:249 = ['global','mds','mon']; ceph-osd hooks/ceph_hooks.py:569 = ['global','osd']. templates/ceph.conf renders the user [global] block AFTER the ms_bind conditionals. So config-flags="{'global': {'ms_bind_ipv6': 'true', 'ms_bind_ipv4': 'false'}}" sets the two flags WITHOUT prefer-ipv6, leaving the space-aware address path intact. PARSER PATH VERIFIED, not assumed: charmhelpers/contrib/openstack/utils.py:1967-1974 config_flags_parser takes the YAML branch when a colon precedes any '=' (here colon=9, equals=-1) -> yaml.safe_load -> nested dict -> ordered() accepts it -> CephConfContext's isinstance(conf, dict) passes. RENDERED the actual template with that context; output ends with, in [global]: ms_bind_ipv6 = true / ms_bind_ipv4 = false. STILL NOT PROVEN LIVE, and config-flags is documented by the charm itself as a stop-gap. SETTLED IN PASSING: `{% if ceph_public_network is string %}` with ', '.join([]) = '' DOES render bare `public network = ` and `cluster network = ` lines. This happens with IPv4 too (the bundle has never set ceph-public-network/ceph-cluster-network), so it is PRE-EXISTING and family-independent -- it does not change the v6 verdict either way. mon host IS bracketed: ceph-mon src/ceph_hooks.py:413/423 format_ipv6_addr(addr) or addr (charmhelpers ip.py:232-241 returns "[%s]"). public addr / cluster addr are emitted UNBRACKETED on the charm's own IPv6 path -- so upstream treats a bare v6 as acceptable there. NOT independently verified against Ceph's parser. ========================================================================= 3. MYSQL-INNODB-CLUSTER (and mysql-router, 13 instances, unscoped by the tuple) ========================================================================= Verified myself, quoted: mysql_innodb_cluster.py:590-595 _script = ("dba.configure_instance('{user}:{pw}@{addr}')\n".format(...addr=address)) mysql_router.py:527-530 "--bootstrap", "{}:{}@{}".format(self.db_router_user, self.db_router_password, self.cluster_address) Both bare -- a v6 literal makes an unparseable MySQL Shell / mysqlrouter URI; the correct form is user:pw@[addr]. Seven URI sites total in mysql-innodb-cluster (:591, :685, :722, :824, :889, :1138, :1693). NEITHER charm ever calls format_ipv6_addr / is_ipv6 (zero hits). First failure is unconditional: initialize_cluster -> configure_instance -> logs ERROR and returns (:596-602); leader flag never set; unit parks at "Instance not yet configured for clustering" (:1515-1519). prefer-ipv6 does NOT help -- it is consumed only in bundled libs, never in charm code, and fixes none of the URI sites. NOT a Group Replication bracketing bug: the charm writes no group_replication_* variables (zero hits for group_replication_local_address / group_replication_group_seeds / 33061); mysqlsh derives them. group_replication_ip_allowlist IS charm-built (:627-637) as a bare comma list, which is the correct format. mysql-router --conf-bind-address 127.0.0.1 (mysql_router.py:59-64) is deliberate subordinate loopback, NOT a plane address -- not a break. NOT ESTABLISHED: whether juju's v6 rendering matches mysqld's for @'host' grant matching (skip_name_resolve=ON, mysqld.cnf:105); whether juju/charms.reactive accept ':' in the leader-settings key / flag name built by .replace(".","-") (:49-56). ========================================================================= 4. OVN -- data-tenant AND metal-internal, and NOT in the tuple ========================================================================= Neither ovn-central nor ovn-chassis declares prefer-ipv6 or any IPv6 option (grep -i ipv6 on config.yaml = 0 hits in both). ROOT DEFECT, verified in this environment: hooks/relations/ovsdb/lib/ovsdb.py:33-47 _format_addr brackets IPv6, and is NOT idempotent: ipaddress.ip_address('[fd00::1]') -> ValueError (idempotent for IPv4) :184-185 publish_cluster_local_addr publishes the ALREADY-BRACKETED string :82-87 _remote_addrs re-parses it and `except ValueError: continue` -- silently drops it. All four copies of the lib are byte-identical (md5 7ea671df11a1d964e793c0ab3e554167). CONSEQUENCE on a v6-only metal-internal / data-tenant, ALL SILENT: chassis lib/charms/ovn_charm.py:1124-1146 -- sb_conn is '' so `if sb_conn:` is False and ovn-encap-type / ovn-encap-ip / system-id / ovn-remote are NEVER WRITTEN AT ALL. central templates/ussuri/ovn-central -- --db-nb/sb-cluster-remote-addr= empty on non-leaders; ovsdb-tool join-cluster called with no remotes. No hook error, no blocked status: readiness gates test bound-address raw, and the Jinja Environment is built without StrictUndefined. Latent, only after a fix: get_data_ip() (ovn_charm.py:962-973) would write external-ids:ovn-encap-ip=[fd00::1] -- bracketed. Independent: the Geneve NOTRACK workaround is iptables-only (ovn_charm.py:1401-1402, no ip6tables anywhere) and is inert on a v6 tunnel plane. ========================================================================= 5. HACLUSTER (8 subordinates, hanode/peer-availability all on metal-internal) ========================================================================= VENDOR (corosync.conf(5), jammy): "If IPv6 networking is used, the nodeid field in nodelist must be specified." "corosync resolves ringX_addr names/IP addresses using getaddrinfo(3) with respect of totem.ip_version setting." ip_version default if unspecified = ipv6-4 for knet/udpu -- but the charm ALWAYS writes it. hooks/utils.py:245-250 if config('prefer-ipv6'): ip_version = 'ipv6' ... else: ip_version = 'ipv4' hooks/utils.py:271-272 if config('prefer-ipv6'): conf['nodeid'] = get_corosync_id(local_unit()) templates/corosync.conf:33-37 renders nodeid / ip_version. => prefer-ipv6 FALSE writes `ip_version: ipv4` AND omits nodeid: corosync cannot form over v6. => prefer-ipv6 TRUE supplies both, but routes through hooks/utils.py:454 utils.get_ipv6_addr(exc_list=excludes)[0] -- dynamic_only=True -- failure mode (b). hacluster imports the v6-safe get_host_ip (charmhelpers network/ip.py:478-494, is_ip-based), so :472/:484/:486 do NOT crash on a v6 literal. corosync_transport default 'unicast'. ========================================================================= 6. RABBITMQ-SERVER -- AND A DEFECT IN THE REPO'S "MEASURED AUTHORITY" ========================================================================= rabbitmq-server@3.9/stable rev 295 DOES declare prefer-ipv6 (read from the ARTIFACT's config.yaml), and it is ABSENT from scripts/provider-bundle-check.py PREFER_IPV6_CHARMS, whose comment at :57 calls that tuple "the measured authority for 9a/9b". Invariant 9a (:379-384) FAILS any app setting prefer-ipv6 whose charm is not in the tuple, with the message "its charm does NOT declare that option -- juju rejects the whole bundle". For rabbitmq-server that message is FACTUALLY WRONG. Full sweep of all 33 unique charm@channel pairs in bundle.yaml against Charmhub config-yaml: rabbitmq-server is the ONLY mismatch; the other 32 agree with the tuple; no tuple entry is unused. (Sweep used default-release, so its revisions are not base-pinned; but for the 9 charms I also read as jammy/amd64 ARTIFACTS the declaration answer is identical.) What the option does there: hooks/rabbitmq_context.py:267-270 if config('prefer-ipv6'): context['settings']['RABBITMQ_SERVER_START_ARGS'] = "'-proto_dist inet6_tcp'" i.e. the Erlang distribution family for CLUSTERING. Matters at HA scale-up (num_units 1 today). ========================================================================= 7. GENERIC METAL-INTERNAL SURFACE (all 56 apps) ========================================================================= With prefer-ipv6 FALSE, get_relation_ip() (charmhelpers ip.py:605-642) falls through to network_get_primary_address(interface) -- SPACE-AWARE. On a v6-only space juju has only a v6 address to return, so charms get v6 by construction. NOT MEASURED in this cloud; owed. Downstream bracketing is NOT uniformly handled. Example measured: charmhelpers/contrib/openstack/templates/haproxy.cfg:88/90 server {{ unit }} {{ address }}:{{ ports[1] }} check -- unbracketed. Reached only with >1 peer (HAProxyContext returns {} with no peers), i.e. at dc-ha-scaleup, not initial deploy. Whether HAProxy parses an unbracketed v6:port is NOT ESTABLISHED. ========================================================================= 8. UPSTREAM STATUS (Launchpad, opened via api.launchpad.net with all 11 statuses passed) -- CORROBORATES THE ARTIFACT READS ========================================================================= LP #2061836 "charm only accepts dynamic ipv6 addresses" (2024-04-16, Federico Bosi) charm-ceph-mon = FIX COMMITTED (2024-06-14); charm-ceph-osd = NEW, never fixed. Reporter: "I tried ... replacing all the get_ipv6_addr() calls with get_ipv6_addr(dynamic_only=False) ... The machines have a single global static ipv6 address. Everything works correctly." => the dynamic_only asymmetry I derived from the artifact is ALREADY REPORTED, and the one-word fix is merged on mon and NOT on osd. Master still shows the bare get_ipv6_addr() in charm-ceph-osd/hooks/ceph_hooks.py. DO NOT FILE A NEW BUG. LP #2056337 "No config flag available to disable OSDs binding to ipv4 addresses" (2024-03-06) FIX COMMITTED both charms; Gerrit 911592 MERGED 2024-04-12 "Don't bind to ipv4 if prefer-ipv6 is true." -- this is the ms_bind_ipv4=false emission I read in the template. Upstream root cause tracker.ceph.com/issues/52867 is still NEW after ~5 years (created 2021-10-07). LP #2109798 "If IPv6 address is set in the mon relation, unit fall in error state" charm-ceph-osd, NEW/unassigned, 2025-05-01. OSError: illegal IP address string passed to inet_aton, then dns.resolver.NoNameservers ... IN A -> mon-relation-changed hook error. *** VERIFIED IN THE rev 953 ARTIFACT, and it MATTERS: *** hooks/ceph_hooks.py:716-728 get_mon_hosts() addr = relation_get('ceph-public-address',...) or get_host_ip( relation_get('private-address',...)) hooks/utils.py:203-216 get_host_ip() (the LOCAL one, imported at ceph_hooks.py:89) socket.inet_aton(hostname) ... except socket.error: answers = dns.resolver.query(hostname, 'A') On a v6-only storage plane, private-address IS v6. Whenever ceph-public-address has not yet been published, the `or` fallback fires and the hook dies. => ceph-osd has an UNFIXED v6 crash path that prefer-ipv6=FALSE does NOT avoid. This WEAKENS the "storage/replication are salvageable" verdict: config-flags fixes ms_bind_*, it does NOT fix this. (format_ipv6_addr(addr) or addr at :728 does correctly bracket the mon host.) Reporter's only workaround, verbatim: "force ceph-mon and ceph-osd not to use IPv6 addresse using the prefer-ipv6=false and forcing the ipv4 cidr for the ceph-cluster-network and ceph-public-network options." LP #2111852 charm-hacluster (2025-05-27, FIX COMMITTED) -- exactly the corosync defect I read: "under the totem section of corosync.conf this setting is present: ip_version: ipv4 And in the section nodelist as ring0_addr an IPv6 addresse is used". Fires at the DEFAULT prefer-ipv6=false. *** BUT: FIX COMMITTED IS NOT FIX RELEASED, AND IT IS NOT IN THE ARTIFACT. *** hacluster 2.4/stable rev 166, released 2026-06-29, STILL has (grep for ip_version = the complete set of references at that revision): hooks/utils.py:246 ip_version = 'ipv6' hooks/utils.py:249 ip_version = 'ipv4' hooks/utils.py:257 / :289 'ip_version': ip_version templates/corosync.conf:36-37 i.e. the charm this cloud would deploy still hard-writes ip_version: ipv4. LP #1966135 ufw rejects bracketed IPv6 from cms-client-bound-address, charm-ovn-central TRIAGED/HIGH since 2022-03-23, charm-ovn-chassis NEW since 2024-03-27. Unfixed 4 years. ========================================================================= 9. IS THE MIXED SHAPE RUN BY ANYONE? -- SEARCH PERIMETER STATED ========================================================================= NOTHING FOUND, and here is the instrument with its positive control: openstack-discuss HyperKitty q=prefer-ipv6 -> 0 (control: q=ceph -> 2483) openstack-dev + openstack-operators: 168 monthly archives, 111 MB, 2013-2019, grepped prefer-ipv6 -> 0 in all 168; ipv6 AND (charm|juju|maas) -> 0 lines (control: 99 files contain ipv6 / 4982 lines; 41 contain juju) opendev codesearch ipv6 in tests/bundles/ -> NO MATCHES (control: ceph-osd -> 35 repos) opendev codesearch ipv6 in tests/tests.yaml -> NO MATCHES (control: bundle -> 88 repos) zuul: ipv6 in charm .zuul.yaml -> ZERO charm-* repos opendev codesearch: prefer-ipv6 in ANY .rst -> no matches (documented only in config.yaml) charm-guide + charm-deployment-guide: grep -ci ipv6 = 0 on every page; the network-SPACES page does not mention IPv6, dual-stack or address family at all discourse.ubuntu.com / charmhub: two unanswered operator threads only (charmhub 5265 2021-10-27, Canonical reply defers to the MAAS forum; charmhub 17777 2025-05-16 "juju pick ipv6 for lxd deployments", ZERO replies) VERDICT: zero IPv6 test bundles, zero IPv6 gate jobs, zero IPv6 documentation across the entire OpenStack-charms ecosystem. Two operators (Bosi 2024, Marchand 2025) are visibly the only people publicly exercising these code paths, and both did it by filing bugs. ========================================================================= 10. CEPH VENDOR DOCS -- SQUID SAYS LESS THAN PACIFIC, NOT MORE ========================================================================= squid doc/rados/configuration/msgr2.rst:86-91 * ms_bind_ipv4 [default: true] controls whether a daemon binds to an IPv4 address * ms_bind_ipv6 [default: false] controls whether a daemon binds to an IPv6 address PACIFIC network-config-ref.rst:217-237 carried the explicit instruction, DELETED in quincy: "Binding to IPv4 is enabled by default, so if you just add the option to bind to IPv6 you'll actually put yourself into dual stack mode. If you want just IPv6, then disable IPv4 and enable IPv6." PACIFIC msgr2.rst:93-95, also DELETED in quincy: ".. note:: ... dual-stack support is not yet supported as of Quincy v17.2.0." Byte counts confirm deletion rather than replacement: network-config-ref.rst 17102 (pacific) vs 13853 (quincy/reef/squid), "Dual Stack" 3/0/0/0. NO replacement statement was added. So: BOTH flags required for v6-only is still correct; squid simply stopped saying it. squid gives NO IPv6 CIDR example for public_network/cluster_network and no bracket guidance for public_addr. ========================================================================= 11. OVN / OVS -- VENDOR ========================================================================= ovn-sb.xml:565 (branch-24.03, identical on main): "The IPv4 address of the encapsulation tunnel endpoint." <- the SB schema documents the column as IPv4. OVS vswitchd/vswitch.xml: geneve = "An Ethernet over Geneve ... IPv4/IPv6 tunnel"; options:remote_ip = "An IPv4 or IPv6 address (not a DNS name)". No bracket form. OVN controller/encaps.c passes encap->ip straight to smap_add(&options,"remote_ip",...) with no parse/validation/bracket handling. OVN NEWS v2.6.0 (2016-09-27): "Added support for IPv6 tunnels". TEST COVERAGE: every ovn-encap-ip / encap-ip value in OVN 24.03's tests (ovn.at 1.6 MB / 137 encap matches, system-ovn.at 186, ovn-controller.at 29, multinode.at) is IPv4. NO test sets an IPv6 encap IP. OVSDB remote syntax IS confirmed: Documentation/ref/ovsdb.7.rst:715-721 "To specify an IPv6 address, wrap it in square brackets, e.g. ssl:[::1]:6640. Passive IP-based connection methods by default listen for IPv4 connections only; use [::] as the address to accept both IPv4 and IPv6 connections, e.g. pssl:6640:[::]" ^ NOTE the passive default: an IPv6-only SB/NB listener needs pssl:6641:[::]. ========================================================================= 12. MYSQL VENDOR ========================================================================= dev.mysql.com/doc/refman/8.0/en/group-replication-ipv6.html: "When specified with a port number, an IPv6 address must be specified in square brackets, for example: group_replication_local_address= "[2001:db8:...:7348]:33061"" group_replication_group_seeds uses the same bracketed form. group_replication_ip_allowlist takes BARE addresses, no brackets (8.0.22+; before that group_replication_ip_whitelist) -- which is what the charm builds, correctly. So the server side is fine and the charm's allowlist is fine; the break is the mysqlsh / mysqlrouter URI concatenation, which is upstream of Group Replication ever being reached.