STAGE 5 -- `prefer-ipv6` / v6 VIP RESEARCH (2026-07-31)
======================================================

THE RULED QUESTION (GA-R5, 2026-07-31; operator utterance: "Research what those 6
charms do with v6 VIPs first (Recommended)"), restated verbatim from
docs/CURRENT-STATE.md section 1:

  for `barbican`, `designate`, `magnum`, `octavia`, `placement` and `vault` at their
  pinned channels, does the charm's generated HAProxy configuration bind `:::port`
  (or the v6 VIP explicitly) WITHOUT a `prefer-ipv6` option -- or does it bind v4
  only, making a v6 VIP leg an address pacemaker manages and nothing listens on?

>>> ANSWER: THEY BIND v6. `prefer-ipv6` IS NOT WHAT MAKES HAProxy BIND `:::port`,
>>> AND IT NEVER WAS -- FOR ANY OF THE THIRTEEN. The `:::port` frontend bind is
>>> gated on a HOST-LEVEL kernel check (`net.ipv6.conf.all.disable_ipv6`), which is
>>> MEASURED `0` on a live node in this DC. The v6 VIP is assigned by pacemaker on
>>> per-address family detection, also independent of the option. Option (b) --
>>> drop `prefer-ipv6` from the six and KEEP their v6 VIP legs -- is supported by
>>> measurement. Option (c) would remove working v6 for no reason.

METHOD -- and why it is the charm ARTIFACT and not Charmhub's `config-yaml`
--------------------------------------------------------------------------
The 2026-07-31 attempt-1 root cause was measured against Charmhub's `config-yaml`,
which answers ONLY "is the knob declared". It cannot answer what the charm DOES.
This research downloads the charm each app actually deploys, at its pinned channel,
for the base and arch these nodes run, and reads the code and templates inside it.

  Run FROM voffice1 (has juju 3.6.27 + egress). Staged under a plain $HOME dir
  (~/charmwork) because the juju SNAP has a private /tmp and its `home` interface
  grants NON-HIDDEN files only -- both traps are already recorded in the phase-4
  runbook.

    juju download <charm> --channel <pinned> --base ubuntu@22.04 --arch amd64 \
      --no-progress --filepath ~/charmwork/<charm>.charm

  `unzip` is ABSENT on voffice1; extraction used python3 zipfile.

PROVENANCE -- exact revisions read (2024.1/stable and 1.8/stable MOVE; these are the
revisions this research measured, quoted from juju's own "Fetching charm" line):

  THE SIX (do NOT declare prefer-ipv6)      THE CONTROLS (DO declare it)
    barbican   2024.1/stable  rev 265         keystone  2024.1/stable  rev 857
    designate  2024.1/stable  rev 418         cinder    2024.1/stable  rev 820
    magnum     2024.1/stable  rev  96
    octavia    2024.1/stable  rev 571
    placement  2024.1/stable  rev 154
    vault      1.8/stable     rev 724

  INDEPENDENT CONFIRMATION OF THE 7/6 SPLIT, from the artifact rather than the API:
  `grep -c prefer-ipv6 <charm>/config.yaml` = 0 for all six. Same answer as the
  Charmhub query, reached by a different instrument.

FINDING 1 -- THE FRONTEND v6 BIND IS GATED ON `ipv6_enabled`, NOT ON `prefer-ipv6`
---------------------------------------------------------------------------------
Two template families exist, and BOTH emit the v6 bind under the same predicate.

  (a) charms.openstack family -- barbican, designate, magnum, octavia, placement
      templates/haproxy.cfg (md5 f248679faf2ff083dc7d4f8b34186700, BYTE-IDENTICAL
      across all five):
          62:    bind *:{{ ports[0] }}
          63:    {% if options.ipv6_enabled -%}
          64:    bind :::{{ ports[0] }}

  (b) charmhelpers family -- keystone, cinder (and the other five declaring charms)
      charmhelpers/contrib/openstack/templates/haproxy.cfg
      (md5 92ea339fc054eeb54eecdf9571ba5e80, identical keystone vs cinder):
          67:    bind *:{{ ports[0] }}
          68:    {% if ipv6_enabled -%}
          69:    bind :::{{ ports[0] }}

  The two templates differ ONLY in variable namespacing (`options.` prefix). The
  bind logic is the same.

WHERE `ipv6_enabled` COMES FROM -- a kernel sysctl, in both families:

  charmhelpers/contrib/openstack/context.py:1069 (HAProxyContext)
      ctxt['ipv6_enabled'] = not is_ipv6_disabled()

  charms_openstack/adapters.py:811 (APIConfigurationAdapter -- the DEFAULT
  configuration_class for OpenStack API charms, charms_openstack/charm/classes.py:593)
      @property
      def ipv6_enabled(self):
          return not ch_ip.is_ipv6_disabled()

  charmhelpers/contrib/network/ip.py:242
      def is_ipv6_disabled():
          ... sysctl net.ipv6.conf.all.disable_ipv6 ...
          return "net.ipv6.conf.all.disable_ipv6 = 1" in result

  `prefer-ipv6` appears NOWHERE in that path.

WHAT `prefer-ipv6` SETS IN HAProxyContext, and why it does not reach the frontend:

  charmhelpers/contrib/openstack/context.py:1062-1067
      if config('prefer-ipv6'):
          ctxt['local_host']   = 'ip6-localhost'
          ctxt['haproxy_host'] = '::'
      else:
          ctxt['local_host']   = '127.0.0.1'
          ctxt['haproxy_host'] = '0.0.0.0'

  `local_host` binds the STATS listener only. **`haproxy_host` is DEAD** -- a grep
  for it across every downloaded charm's template directories returns NOTHING. It is
  assigned in the context and consumed by no template. So the one value `prefer-ipv6`
  sets that sounds like a bind address is never rendered.

FINDING 2 -- THE PRECONDITION IS MEASURED, NOT ASSUMED
------------------------------------------------------
`ipv6_enabled` is true only if IPv6 is enabled on the machine. Measured on a LIVE
MAAS-deployed jammy node in this DC -- the dc0 Juju controller machine, reached with
`juju ssh -m controller 0` from the dc0 rack:

    subtle-grouse
    net.ipv6.conf.all.disable_ipv6 = 0
    enp1s0 fd50:840e:74e2:220::5/64

  `subtle-grouse` is the inst id CURRENT-STATE records for the bootstrapped
  controller, so the instrument is reading the intended host. The nodes the bundle
  will land on come from the same MAAS image and the same v6 carve
  (`dc-node-v6-carve.py check vr1-dc0` = 54 links correct). `is_ipv6_disabled()`
  therefore returns False and the `bind :::port` line IS emitted.

FINDING 3 -- THE v6 VIP IS ASSIGNED BY PACEMAKER REGARDLESS OF THE OPTION
------------------------------------------------------------------------
The resource type is chosen per address, by family detection on the `vip` string:

  charmhelpers/contrib/openstack/ha/utils.py:286-291 (update_hacluster_vip)
      for vip in cluster_config['vip'].split():
          if is_ipv6(vip):
              res_vip = 'ocf:heartbeat:IPv6addr';  vip_params = 'ipv6addr'
          else:
              res_vip = 'ocf:heartbeat:IPaddr2';   vip_params = 'ip'

  interface_hacluster/common.py:920-927 (VirtualIP.configure_resource -- the path
  vault uses, since vault talks to hacluster directly):
      ipaddr = ipaddress.ip_address(self.vip)
      if isinstance(ipaddr, ipaddress.IPv4Address):  res_type = 'ocf:heartbeat:IPaddr2'
      else:                                          res_type = 'ocf:heartbeat:IPv6addr'

  `prefer-ipv6` is not consulted in either. A v6 address in the `vip` string gets an
  IPv6addr resource whether or not the option exists.

FINDING 4 -- VAULT IS NOT AN HAProxy CHARM, AND ITS LISTENER IS ALREADY DUAL-STACK
----------------------------------------------------------------------------------
The ruled question is malformed for vault, and the answer is stronger than for the
other five. vault@1.8 rev 724 ships NO haproxy template at all (`find` returns
nothing). Its listener is a hardcoded literal in templates/vault.hcl.j2:

    listener "tcp" {
      address = "[::]:8200"

  `[::]` is the v6 wildcard; with Linux's default `bindv6only=0` it accepts v4 as
  well. Vault binds v6 unconditionally and always has.

  Its VIPs reach pacemaker through reactive/vault_handlers.py:517-534, which iterates
  EVERY entry of `config('vip')` and calls `hacluster.add_vip(...)` -- so all six of
  vault's legs (3 v4 + 3 v6) are managed. Separately, lib/charm/vault.py:161-180
  `get_vip(binding)` selects ONE vip matched to the binding's CIDR for vault's own
  advertised api_addr; with a dual-family sextet that resolves to the v4 leg of the
  bound plane. That is address SELECTION for self-advertisement, not binding, and it
  is unaffected by `prefer-ipv6` (which vault does not have).

FINDING 5 -- WHAT `prefer-ipv6` ACTUALLY DOES (this is the load-bearing part)
----------------------------------------------------------------------------
It is a UNIT ADDRESS-FAMILY switch, not a listener switch. Every consumer found:

  1. charmhelpers/contrib/network/ip.py:617 -- `get_relation_ip()`:
         if config('prefer-ipv6'):
             assert_charm_supports_ipv6()
             return get_ipv6_addr(dynamic_only=False)[0]
         elif cidr_network: ...
     This returns EARLY. The unit advertises an IPv6 address on EVERY relation, and
     the `cidr_network` / network-space branch below it is never reached.
  2. charmhelpers/contrib/openstack/ip.py:223 -- resolve_address() fallback becomes
     `get_ipv6_addr(exc_list=vips)[0]`.
  3. charmhelpers/contrib/openstack/context.py:1712 -- BindHostContext: the API
     service's own `bind_host` becomes `::` instead of `0.0.0.0`.
  4. context.py:1062 -- HAProxy STATS listener binds ip6-localhost.
  5. Per-charm: keystone_hooks.py:231/315 call `sync_db_with_multi_ipv6_addresses()`
     (registers keystone's DB grants against its v6 addresses) and `setup_ipv6()`;
     keystone_utils.py:957 makes the local bypass endpoint a v6 literal.
     (`setup_ipv6()` at keystone_utils.py:2375 is effectively a NO-OP on jammy --
     its only action is a trusty/pre-liberty haproxy backport.)
  6. charms_openstack/adapters.py:808 -- `ipv6_mode` = `getattr(self,'prefer_ipv6',
     False)`, driving `local_address` to `get_ipv6_addr()`.

  FOR THE SIX, undeclared resolves cleanly to FALSE in both families -- this is not
  an exception path, it is the ordinary default:
    - charmhelpers `hookenv.config(scope)` returns `_cache_config.get(scope)`
      (hookenv.py:450) -> None -> falsy for a key the charm does not declare.
    - charms.openstack `getattr(self, 'prefer_ipv6', False)` -> False, because the
      adapter only grows attributes for keys the charm declares.

CONSEQUENCE 1 -- `provider-bundle-check` INVARIANT 9's RATIONALE IS FACTUALLY WRONG
----------------------------------------------------------------------------------
scripts/provider-bundle-check.py:30-33 and :288-295 state:

    "prefer-ipv6 makes HAProxy bind :::port in ADDITION to *:port, so the two must
     travel together"

That mechanism claim is REFUTED above: the `:::port` bind is gated on the host
sysctl. The invariant's OTHER half -- that a merge order dropping the v6 legs while
keeping the option exits 0 -- was a real measured defect (L3-9) about arity, and the
arity check itself is still worth having. But the coupling as coded (`prefer6 !=
dual` is a FAIL) is what will reject any fix to the six.

  This is the repo's own "RULED IS NOT BUILT -- CHECK THE ARTIFACT" class, one turn
  further in: a gate was built on a mechanism nobody read the charm to confirm.

  WHEN A RULING LANDS, INVARIANT 9 MUST BE REPLACED, NEVER DELETED TO GO GREEN (the
  standing rule that a checker which cannot fail is not a gate; and the rule that a
  test asserting a literal finding string gets RE-POINTED at the new invariant).
  The replacement invariant has to be charm-schema-aware: `prefer-ipv6` may only
  appear on a charm that DECLARES it, and dual-family arity is asserted on its own.
  It must be proven able to fail in both directions.

CONSEQUENCE 2 -- A NEW QUESTION ABOUT THE **SEVEN**, WHICH THIS RESEARCH RAISES AND
DOES NOT ANSWER
-----------------------------------------------------------------------------------
R2 set `prefer-ipv6: true` on all 13 in order to get v6 VIP binding. That is not what
the option does. On the seven that DO declare it, setting it true has the Finding-5
effects -- most consequentially, `get_relation_ip()` returns early and hands every
relation an IPv6 address chosen from the unit's primary interface, IGNORING the
network-space binding. This bundle binds relations across six named spaces.

  So after the six are fixed, the deploy would carry SEVEN charms advertising v6
  relation addresses and forty-nine advertising v4. That asymmetry is a deploy risk
  that has never been observed live (attempt 1 aborted atomically; the model is
  empty), and it is a DIFFERENT question from the ruled one.

  STATED, NOT DECIDED, and NOT folded into the question below. It is R2 / D-101
  ruling-surface material and belongs in its own GA-R5 exchange.

WHAT IS PUT TO THE OPERATOR (the ruled fork, now decidable on evidence)
----------------------------------------------------------------------
  (b) Drop `prefer-ipv6` from the six, KEEP their dual-family v6 VIP legs.
      SUPPORTED BY THE MEASUREMENT ABOVE. Their HAProxy binds `:::port`, pacemaker
      assigns the v6 VIP, and vault already listens on `[::]`.
  (c) Drop both, making those six v4-only.
      Would remove v6 that measurably works, and would contradict D-101 (dual-stack
      where IPv4 is required, IPv6-only where sound, BOTH DCs this deployment).

  The overlays are GENERATED (D-136 (D)) -- never hand-edit overlays/, which
  `tests/render-drift` exists to catch. BOTH DCs, symmetric.

  WHAT (b) COSTS, MEASURED BEFORE PUTTING THE FORK -- IT IS NOT A VALUES-FILE EDIT.
  `prefer-ipv6: true` is INJECTED UNCONDITIONALLY BY THE RENDERER:

    scripts/render-dc-overlays.py:243-246
        if fam == "dual":
            # prefer-ipv6 and the v6 legs must land together, or HAProxy binds
            # :::port with no v6 VIP for pacemaker to manage.
            out.append("      prefer-ipv6: true\n")

  and `family: dual` is a SINGLE TOP-LEVEL key at render/values/vr1-dc0-vips.yaml:3
  (same at dc1:3) -- not per-app data. There is no value to edit. Implementing (b)
  means teaching the renderer which charms DECLARE the option, re-pointing
  tests/render-drift, and replacing invariant 9. A code change with a harness, not a
  one-line data fix. Note also that the injection's own comment repeats the mechanism
  claim refuted above.

ADDITIONAL MEASUREMENT -- FULL OPTION-NAME SWEEP OF THE dc0 DEPLOY INPUT
-----------------------------------------------------------------------
Attempt 1 proved `juju deploy --dry-run` does not validate config option NAMES, and
juju aborts the WHOLE bundle atomically on the first bad one. So a second unknown
option elsewhere would cost another full attempt. Every option assignment in the
actual dc0 deploy input was therefore checked against each charm's own config.yaml,
downloaded at its pinned channel:

  deploy input: bundle.yaml + overlays/vr1-dc0-vips.yaml
                            + overlays/vr1-dc0-machines.yaml
                            + overlays/vr1-dc0-octavia-pki.yaml   (= attempt 1's set)
  56 applications; 81 option assignments; 20 charm schemas; 0 unresolvable.

  [FAIL] barbican   barbican@2024.1/stable  does not declare option 'prefer-ipv6'
  [FAIL] designate  designate@2024.1/stable does not declare option 'prefer-ipv6'
  [FAIL] magnum     magnum@2024.1/stable    does not declare option 'prefer-ipv6'
  [FAIL] octavia    octavia@2024.1/stable   does not declare option 'prefer-ipv6'
  [FAIL] placement  placement@2024.1/stable does not declare option 'prefer-ipv6'
  [FAIL] vault      vault@1.8/stable        does not declare option 'prefer-ipv6'

>>> `prefer-ipv6` ON THOSE SIX IS THE ONLY UNKNOWN OPTION IN THE ENTIRE DEPLOY INPUT.

SCOPE OF THAT CLAIM, stated so it is not read as "attempt 2 is safe". This checked
option NAMES, for the dc0 input only. TWO EXTENSIONS ARE OWED BEFORE THE REDEPLOY,
each able to cost a whole attempt the same way:
  (i)  OPTION VALUE TYPES -- every downloaded config.yaml carries a `type:` field
       that this pass discarded. A boolean assigned a string fails the same way.
  (ii) THE dc1 INPUT -- unswept. There is no ruled DC ordering and the fix is
       symmetric, so a dc1-only defect would surface as a further failed deploy.
Sequenced AFTER the ruling: doing them now would be precondition work the standing
directive puts out of scope.

  Schemas read (revision at 2024.1/stable unless noted): barbican 265, ceph-mon 491,
  ceph-osd 953, ceph-radosgw 600 (squid/stable), cinder 820, designate 418,
  glance 681, glance-simplestreams-sync 152, hacluster 166 (2.4/stable),
  keystone 857, magnum 96, neutron-api 710, nova-cloud-controller 823,
  nova-compute 894, octavia 571, octavia-diskimage-retrofit 257,
  openstack-dashboard 750, ovn-chassis 396 (24.03/stable), placement 154,
  vault 724 (1.8/stable).

  REFUSAL BEHAVIOUR: a charm whose config.yaml could not be read, or whose shape is
  unrecognised, is REPORTED and exits 2 -- "could not look" is never "nothing there".
  Zero occurred here.

  A REPO GATE FOR THIS IS **OWED, NOT BUILT** (hard rule 1 + the standing directive:
  hardening is out of scope unless it blocks the deploy, and this one-shot
  measurement already de-risks attempt 2). CURRENT-STATE already records that no gate
  in this repo reads a charm config schema. The method is reproduced verbatim at the
  end of this file so it is not lost.

OBSERVATION -- A RUNBOOK DEFECT FOUND IN PASSING (DOCFIX material, LOGGED NOT FIXED)
------------------------------------------------------------------------------------
runbooks/dc-dc-phase4-juju-bundle-per-dc.md:553-557 gives the dc0 deploy as

    juju deploy ./bundle.yaml \
      --overlay overlays/vr1-dc0-vips.yaml \
      --overlay overlays/vr1-dc0-octavia-pki.yaml

-- WITHOUT `overlays/vr1-dc0-machines.yaml`, while the dc1 block three lines below
DOES include its machines overlay. Attempt 1 (executed) correctly included it.

MEASURED delta between the two merged inputs -- the overlay is NOT a no-op:

    ONLY-WITH-OVERLAY .applications.ovn-chassis.options.bridge-interface-mappings
        = 'br-ex:52:54:00:8c:2a:8c br-ex:52:54:00:50:48:88'

That is the pair of dc0 compute provider MACs. Deploying dc0 exactly as the runbook
reads would give ovn-chassis NO provider bridge mapping, and the failure would appear
later as tenant networks with no external path -- not as a deploy error.

REPRODUCTION -- the sweep, verbatim
-----------------------------------
Run on voffice1 with the repo clone at the branch HEAD and the charm schemas staged
under ~/charmwork/x_<charm>__<channel with / as _>/config.yaml:

  # 1. enumerate the unique charm@channel pairs that carry options
  python3 - . overlays/vr1-dc0-vips.yaml overlays/vr1-dc0-machines.yaml \
             overlays/vr1-dc0-octavia-pki.yaml <<'PY'
  import sys, yaml, os
  def dm(b,o):
      for k,v in (o or {}).items():
          if isinstance(v,dict) and isinstance(b.get(k),dict): dm(b[k],v)
          else: b[k]=v
      return b
  repo=sys.argv[1]
  m=yaml.safe_load(open(os.path.join(repo,'bundle.yaml')))
  for ov in sys.argv[2:]: dm(m, yaml.safe_load(open(os.path.join(repo,ov))) or {})
  seen=set()
  for a,s in sorted((m.get('applications') or {}).items()):
      s=s or {}
      if not (s.get('options') or {}): continue
      seen.add((s.get('charm'), s.get('channel')))
  for c,ch in sorted(seen, key=lambda x:(str(x[0]),str(x[1]))): print("%s %s"%(c,ch))
  PY

  # 2. download each pair's charm and keep only its config.yaml
  while read -r c ch; do
    d="x_${c}__$(echo "$ch" | tr '/' '_')"
    juju download "$c" --channel "$ch" --base ubuntu@22.04 --arch amd64 \
      --no-progress --filepath ~/charmwork/dl.charm
    mkdir -p "$d"
    python3 -c 'import zipfile,sys; z=zipfile.ZipFile("dl.charm"); [z.extract(n,sys.argv[1]) for n in z.namelist() if n=="config.yaml"]' "$d"
  done < pairs.tsv

  # 3. merge the deploy input the way juju does (dicts merge, scalars replace) and
  #    compare every app's option KEYS against its charm's declared keys. An
  #    unreadable or unrecognised schema REFUSES (exit 2) rather than passing.

  The merge semantics matter: juju DEEP-MERGES overlay `options` (measured
  2026-07-29 against charm v12.1.1 source and recorded in the session ledger), so a
  replace-style merge would under-report.

HOSTS AND INSTRUMENT IDENTITY
-----------------------------
  charm downloads + schema reads : voffice1  (juju 3.6.27, egress)
  live IPv6 sysctl               : dc0 Juju controller machine via
                                   `juju ssh -m controller 0` FROM the dc0 rack
                                   (D-138 client host); hostname read back as
                                   `subtle-grouse`, matching the recorded inst id
  deploy-input hashes cross-checked on all three hosts before anything was trusted:
    bundle.yaml                        5cc1542fc53ebf05caf6c51b502d84fcdebcacbfb3b78ae651f2db384eba91f7
    overlays/vr1-dc0-vips.yaml         3ae79e820a3fe6ed23edb04ca700e2b6b85479b683936a2da787121fe4611b7d
    overlays/vr1-dc0-machines.yaml     b70e4eedc2bdd3157e7bd78fcf582e44e98997eaca8a838085672ed1dcaabac9
    overlays/vr1-dc0-octavia-pki.yaml  5fc117f1883320422f1854b4a82c9c8025dd912326b25fa6ad95686dad40227f
      (identical vcloud / voffice1 / dc0 rack ~/repo-stage; the PKI overlay is
       gitignored and was compared voffice1 vs rack only, where it exists)
  NOTE: voffice1's clone is at 8c16bbe, two commits behind vcloud's ca2d636. Those
  two commits touch the skill and permission rules only; the four deploy-input files
  were hash-compared rather than assumed equal.

STATUS OF STAGE 5 AFTER THIS RESEARCH
-------------------------------------
Still BLOCKED, now on the RULING rather than on the research. Nothing was edited: the
overlay is untouched, R2 is unamended, invariant 9 is unchanged. The model remains
empty.