Changelog 2026-07-31 -- the prefer-ipv6 / v6 VIP charm research

Session changelog (GA-R2/D1: ONE per session). Branch dc-dc-stage5-preconditions. Status claims live ONLY in docs/CURRENT-STATE.md.

Context: bundle deploy attempt 1 failed 2026-07-31 with unknown option "prefer-ipv6" on barbican, and the operator ruled "Research what those 6 charms do with v6 VIPs first (Recommended)". Stage 5 is blocked on that research. This session executes it, the operator RULES on the result, and the ruling is then BUILT and verified.

Scope correction to this header, made when it stopped being true. It first read "Read-only throughout: no cloud mutation, no overlay edit, no ruling adopted", which was accurate for items 1-5. After the ruling landed the session edited the renderer, the gate, both overlays and both values files, and refreshed the deploy input staged on the dc0 rack. No cloud state changed at any point -- the vr1-dc0 model is still empty and no juju deploy was run.

Item 1 -- docs/audit/stage5-prefer-ipv6-charm-research-20260731.txt (NEW)

What. The capture answering the ruled question, plus two findings that came out of the same measurement.

Method, and why it is not the Charmhub config-yaml API. The attempt-1 root cause was measured against config-yaml, which answers only is the knob declared. It cannot answer what the charm does. This research downloads the charm each app actually deploys -- juju download <charm> --channel <pinned> --base ubuntu@22.04 --arch amd64 from voffice1 -- and reads the templates and code inside it. Revisions are recorded in the capture because 2024.1/stable moves. Two declaring charms (keystone 857, cinder 820) were downloaded as CONTROLS: without one, "binds v4 only" and "binds v6 by another mechanism" cannot be told apart.

The answer. prefer-ipv6 is not what makes HAProxy bind :::port, and never was, for any of the thirteen. In both template families the v6 frontend bind is gated on ipv6_enabled = not is_ipv6_disabled() -- a read of the kernel net.ipv6.conf.all.disable_ipv6 sysctl. Measured 0 on a live MAAS-deployed jammy node in this DC. The v6 VIP is assigned by pacemaker on per-address family detection, also independent of the option. Vault ships no haproxy template at all and its listener is the hardcoded literal [::]:8200.

What the option really does: it is a unit address-family switch -- get_relation_ip() returns early with the unit's IPv6 address for EVERY relation, ignoring the network-space binding; plus bind_host ::, ip6-localhost stats, and keystone's sync_db_with_multi_ipv6_addresses. The value that sounds like a bind address, haproxy_host, is consumed by NO template in any charm downloaded.

Consequence recorded, not acted on: provider-bundle-check invariant 9's mechanism claim is factually wrong; when a ruling lands the invariant must be REPLACED with a charm-schema-aware one and proven able to fail in both directions, never deleted to go green.

Revert. git rm docs/audit/stage5-prefer-ipv6-charm-research-20260731.txt. The file is a read-only capture; nothing depends on it but the CURRENT-STATE entry in item 2.

Item 2 -- docs/CURRENT-STATE.md section 1: the research result

What. The research outcome appended under the 2026-07-31 ruling block that records the question -- the answer, its evidence, the invariant-9 consequence, the new question about the seven declaring charms, the option-name sweep result, and the runbook defect. Required in the same commit by GA-R1 rule 8 (C1); repo-lint L10 enforces it.

Revert. git revert this commit, or delete the appended block; the preceding ruling text is unchanged and still reads correctly on its own.

Item 3 -- MEASUREMENT ONLY (no artifact): full option-name sweep of the dc0 deploy input

What. Attempt 1 proved juju deploy --dry-run does not validate config option NAMES, and juju aborts the whole bundle atomically on the first bad one -- so a second unknown option elsewhere costs another full attempt. Every option assignment in attempt 1's exact input (bundle.yaml + vr1-dc0-vips + vr1-dc0-machines + vr1-dc0-octavia-pki) was compared against each charm's own config.yaml downloaded at its pinned channel.

Result: 56 applications, 81 option assignments, 20 charm schemas, 0 unresolvable, and the only findings are prefer-ipv6 on the same six. No further option-name failure is waiting for attempt 2. An unreadable or unrecognised schema REFUSES rather than passing.

Not built as a gate, deliberately. Hard rule 1 plus the standing operator directive: hardening is out of scope unless it blocks the deploy, and this one-shot measurement already de-risks attempt 2. The method is reproduced verbatim in the capture so it can be re-run or promoted later. CURRENT-STATE already records that no gate in this repo reads a charm config schema.

Both owed extensions were then RUN, after the ruling landed -- option VALUE TYPES (every downloaded config.yaml carries a type: field the first pass discarded) and the dc1 input. dc0 and dc1 each: 75 assignments / 20 schemas / 0 unknown name / 0 type mismatch / 0 note / 0 unresolvable, EXIT 0. dc1 introduces no new charm@channel pair, checked rather than assumed.

PROOF OF TEETH. A clean sweep across two inputs is exactly the implausibly-uniform result this repo has been burned by, so three defects were planted in a THROWAWAY copy: a restored prefer-ipv6 on barbican ([FAIL name]), a quoted "true" on keystone's boolean ([FAIL type]), and an unquoted float where a string is declared ([note]). All three caught, EXIT 1. The EXIT-0 readings are a measurement, not a silent no-op.

Type judgement, stated because it decides what the sweep is worth. A mismatch is reported only where juju's own coercion cannot save it (a boolean given a non-boolean, an int/float given a non-number). A scalar where a STRING is declared is a NOTE -- juju coerces it -- but it is still surfaced, because an unquoted value is how a version string silently becomes a float. An UNRECOGNISED declared type REFUSES rather than passing.

Revert. Nothing to revert -- no repo artifact was created by this item.

Item 5 -- MEASUREMENT ONLY: what option (b) would actually cost

What. Before putting the fork to the operator, the implementation cost of (b) was measured rather than assumed. prefer-ipv6: true is INJECTED UNCONDITIONALLY by scripts/render-dc-overlays.py:243-246 for every app whenever fam == "dual", and family: dual is a SINGLE TOP-LEVEL key at render/values/vr1-dc{0,1}-vips.yaml:3 -- not per-app data.

Consequence. There is no value to edit. (b) means teaching the renderer which charms declare the option, re-pointing tests/render-drift, and replacing provider-bundle-check invariant 9 -- a code change with a harness, not a one-line data fix. The injection's own inline comment repeats the mechanism claim this session's research refutes.

Revert. Nothing to revert.

Item 4 -- LOGGED NOT FIXED: the dc0 deploy block omits a non-no-op overlay

What. runbooks/dc-dc-phase4-juju-bundle-per-dc.md:553-557 gives the dc0 deploy without overlays/vr1-dc0-machines.yaml, while the dc1 block three lines below includes its machines overlay. Measured, the overlay's ONLY delta to the merged input is ovn-chassis.options.bridge-interface-mappings = 'br-ex:52:54:00:8c:2a:8c br-ex:52:54:00:50:48:88' -- the two dc0 compute provider MACs. Deploying dc0 exactly as the runbook reads would leave ovn-chassis with no provider bridge mapping, which surfaces later as tenant networks with no external path, not as a deploy error. Attempt 1 as executed correctly included the overlay.

Why not fixed here. Hard rule 1 -- this session's step is the research. It is DOCFIX material and is recorded in CURRENT-STATE and in the capture so it cannot be lost.

Revert. Nothing to revert.

Item 6 -- THE RULING, and the three-part build that implements it

Ruling. Recorded under GA-R5 in docs/design-decisions.md as a D-101 RULING NOTE dated 2026-07-31 (OPS under GA-R3; D-101's matrix UNAMENDED), and in docs/CURRENT-STATE.md. Committed and pushed BEFORE any dependent work. Operator utterance, verbatim: "I want it to use IPv6, if there is spam mechanisms being applied then that is bad. Even if it doesn't cause an issue now, it might in the future. A clean IPv6 network is better than one with unusable and possibly future breaking configurations." -- and, because that stated a principle rather than selecting an option, a confirming exchange: "Yes -- keep the v6 legs, remove only the option".

(i) scripts/provider-bundle-check.py -- PREFER_IPV6_CHARMS (NEW) + invariant 9 REPLACED. The constant is the measured authority, keyed by CHARM name (an application name is the deployer's choice; only a charm has a schema) and carrying the revisions and a re-measure-if-a-pin-moves warning. Invariant 9's old text -- "prefer-ipv6 makes HAProxy bind :::port in ADDITION to *:port, so the two must travel together" -- is refuted by this session's measurement and is REPLACED, never deleted: 9a the option is only legal on a charm that declares it (asserted on PRESENCE, since prefer-ipv6: false is the same fatal unknown option to juju as true); 9b on a declaring charm it still travels with the v6 legs, keeping the L3-9 protection intact for the seven; 9c arity.

(ii) scripts/render-dc-overlays.py + both overlays re-rendered. The renderer emits the option only for a declaring charm, reading the set from provider-bundle-check with ast rather than restating it (the read-don't-restate rule APP_OCTET already uses). render() keeps its purity property -- the set is a PARAMETER, and the default REFUSES rather than defaulting to empty, because an empty set renders a plausible-looking artifact with the option nowhere. The values-file headers were updated too, since they carried the refuted mechanism claim. Measured delta: exactly six lines removed per DC and nothing else -- every vip string byte-identical, so all v6 legs are retained.

(iii) tests/provider-bundle-check/ 44 -> 48. T43 reproduces the attempt-1 defect; T44 proves presence-not-truthiness; T45 is the positive control (without it, 9a could be satisfied by rejecting everything); T46 asserts the DIAGNOSIS. T21's comment was re-pointed to its real rationale -- the assertion is unchanged and still correct, for a different reason than the one originally written.

Mutation pass -- six mutations, every one killed tests. 9a deleted (T43+T44 die); 9a keyed on value (T44 dies); 9b restored to all charms (T19+T45 die, 10 failures); the charm list widened with barbican+vault (T43-T45 die); renderer emitting for every dual app, and prefer6_charms() returning empty instead of refusing (both kill render-drift). Source restored byte-identical after each.

ONE NEW ASSERTION WAS DECORATION AND WAS REPLACED. T46's first form mutated only a vip; a non-declaring charm with no option has prefer6 == dual == False and reaches neither branch, so it could not fail under ANY mutation of the charm list. Re-written to assert that an input matching BOTH rules is diagnosed by 9a rather than 9b -- a misleading diagnosis would send the next session to add v6 legs when the fix is to remove the option -- and re-proven against the fold-9a-into-9b mutation. Third time in a week that "prove each new assertion can FAIL" has caught a real hole.

Verification. Gauntlet ALL GREEN (93); repo-lint 0 fail; the fixed dc0 deploy input now sweeps CLEAN on option names (75 assignments, 20 schemas, 0 unknown, 0 unresolvable).

Item 7 -- A DEFECT IN ITEM 6, found by a repo-wide grep and corrected the same session

What was wrong. Two faults with one cause: item 6 measured and gated only the THIRTEEN VIP charms.

1. PREFER_IPV6_CHARMS was WRONG. Re-measured across ALL 33 charms in bundle.yaml: twelve declare the option, not seven -- the seven VIP ones plus ceph-mon 491, ceph-osd 953 (squid/stable), nova-compute 894, hacluster 166 (2.4/stable) and mysql-innodb-cluster 164 (8.0/stable).
2. Invariant 9a was written INSIDE the VIP loop, so an application with no vip never reached it.

Why it mattered, and why no test would have caught it. The repo ALREADY has the missed case: overlays/dc-dc-ipv6-family-matrix.yaml sets prefer-ipv6 on ceph-mon, which carries no VIP. That overlay is a LATER deploy step (runbooks/dc-dc-phase4-juju-bundle-per-dc.md:728/737), not part of the swept attempt-2 input -- so the miss would have passed attempt 2 cleanly and surfaced at the step after, looking like a new fault. ceph-mon happens to declare the option, so nothing would have broken today; the gate was simply blind there.

Fixed. 9a now runs over EVERY application, before the VIP loop, and the VIP loop skips an app 9a already flagged so the diagnosis is not doubled. ceph-mon's in-file claim ("CONFIRMED real option, charm-ceph-mon config.yaml") is now INDEPENDENTLY VERIFIED from the artifact rather than taken from the comment. Harness 48 -> 50/50: T47 (a non-VIP app whose charm lacks the option FAILS) and T48 (ceph-mon PASSES), both mutation-proven -- re-scoping 9a back to the VIP loop kills T47; dropping ceph-mon from the list kills T48. Gauntlet ALL GREEN (93); repo-lint 0 fail.

The generalisable lesson, and it is one this repo keeps paying for: I measured the population the QUESTION named (the 13 VIP charms) rather than the population the INVARIANT covers (every application juju validates). The grep that found it took two minutes and was not prompted by any gate.

Revert. Folded into item 6's revert.

Revert (item 6). git revert this commit. It restores the previous renderer behaviour, the previous invariant 9 and the previous overlays together -- they must move as one, since a re-rendered overlay without the matching renderer fails tests/render-drift.

What was NOT done, stated so it is not assumed

• R2 is unamended and D-101's matrix is unchanged. The dual-stack posture does not move; every v6 VIP leg is retained.
• Nothing changed for the SEVEN charms that DO declare the option -- that is a separate, newly-raised question and was deliberately not bundled into the ruling (GA-R5: one decision per exchange).
• No cloud state changed. The vr1-dc0 model is still empty; no juju deploy was run this session.
• No charm-schema GATE was built. The one-shot sweep is a measurement; a gate is logged as owed.

Item 8 -- pre-deploy sequence: rack input refreshed, preflight run, dry-run green

Rack deploy input refreshed and hash-verified. ~/repo-stage on the D-138 client host is a COPY, not a git clone, so nothing updates it automatically and it was still carrying the pre-ruling vips overlay. Exactly one of four files differed; it was copied and all four then compared against repo HEAD. The gitignored vr1-dc0-octavia-pki.yaml was deliberately NOT touched -- a wholesale directory refresh would have risked clobbering or re-permissioning it (SEC-029) -- and it verifies byte-identical to voffice1's at 5fc117f1, still 0600.

Preflight captured: docs/audit/stage5-preflight-dc0-20260731.txt (242 lines, exit 1). Run with MAAS_PROFILE=vr1-dc0-region, because preflight is REGION-BLIND and without it emits 19 false negatives. The instrument was proven current first -- maas-profile-assert.sh vr1-dc0-region hot-kid exits 0. P1-P4 and P7 PASS; P5 FAIL with 11 findings.

The P5 delta is enumerated, not waved through. The 2026-07-30 acceptance covered SIX findings and says it covers "these six, enumerated, and nothing else". Five NEW ones -- all vr1-dc1 S2 rows from SEC-027/-028 -- are the D-137 forward register correctly reporting that dc1's region has not been built. The diff is appended to the capture and put to the operator as its own GA-R5 exchange.

Step 4.2 --dry-run against the fixed input: exit 0, 56 apps / 108 relations / 33 unit placements. Its green is not evidence on option names -- that is attempt 1's lesson -- but it confirms the re-rendered overlay resolves and plans. The ceph-osd tags=openstack exposure is now MEASURED rather than reasoned: the plan places ceph-osd/0..3 on machines 5-8 by explicit id, so the absent tag never has to match and the initial deploy is unaffected. The residual (a later UNPLACED juju add-unit) is unchanged and still logged.

Revert. Nothing to revert in the repo beyond the capture. The rack-side file copy is reversed by copying the previous overlay back; its pre-change hash was 3ae79e82.

Item 9 -- SECOND RULING: the five new P5 findings accepted

Question as presented. Preflight for dc0 is RED on P5 only. It reports 11 findings; the 2026-07-30 acceptance covered SIX, enumerated, and says it covers "these six, enumerated, and nothing else". All six are still present. Five are NEW, all vr1-dc1, all opened after that ruling -- maas-region-db-password, maas-region-admin-password, maas-region-api-key.txt (SEC-027) and maas-juju-api-key.txt, maas-juju-user-password (SEC-028) -- and every one is the D-137 forward register correctly reporting that dc1's MAAS region is authored but not applied. Accept and proceed, or stop and remediate?

Operator answer, exact utterance: "Accept the five and proceed to the dc0 deploy".

Consequence. The five are carried as known, by-design absences on their existing SEC-027 / SEC-028 rows. They describe dc1, which is not built; nothing about them affects the dc0 deploy. preflight.sh keeps exiting FAIL on P5 for the rest of this stage, that RED is ruled-accepted, and it must NOT be made green by deleting or weakening a matrix row. Like its predecessor this covers these FIVE, enumerated, and nothing else. Total accepted at P5 is now ELEVEN across the two rulings.

Revert. A ruling is a record, not a change; there is nothing to revert. Withdrawing it would require a new GA-R5 exchange.

Item 10 -- BUNDLE DEPLOY ATTEMPT 2: the prefer-ipv6 defect is gone; two new ones found

What happened. The deploy was run from the dc0 rack against the fixed, hash-verified input. The prefer-ipv6 failure did not recur. Two new defects did, and for the first time the model is PARTIALLY POPULATED: 23 applications, 0 machines, 0 units -- measured immediately, not assumed. Attempt 1 aborted during VALIDATION and left the model empty; this one got past validation into EXECUTION, which is why it left residue. Nothing is provisioned: no MAAS machine left Ready, no disk was written.

Defect 1 -- the deploy input on the client host is a PARTIAL COPY of the repo. ~/repo-stage has no policies/, and bundle.yaml:214 is the only local-file reference in the whole deploy input (policyd-override: ./policies/overrides.zip), resolved relative to the bundle and uploaded as a charm resource -- so it must exist ON THE HOST THAT DEPLOYS. Nothing caught it: --dry-run does not upload resources (the SECOND --dry-run blind spot in two days), attempt 1 aborted before resource upload, and preflight.sh runs on voffice1 where policies/ DOES exist. Same wrong-host instrument class as the dc-mirror.sh and egress-probe errors, and the second instance this session. Fixed forward under gating: both policy files copied and sha256-verified.

Defect 2 -- a bundle that relies on default-base is deployable exactly once. The re-run failed with barbican: downgrades are not currently supported: deployed revision 265 is newer than requested revision 261. Measured: barbican 2024.1/stable is rev 265 at --base ubuntu@22.04 and 261 at --base ubuntu@24.04 or with no base. The 23 existing apps are all on 22.04, so run 2a honoured bundle.yaml:85 -- but on the re-run, for an application that ALREADY EXISTS, juju resolves without that default.

Two fixes tested READ-ONLY (--dry-run reproduces the error against the populated model, making it a safe harness): a model-level default-base=ubuntu@22.04 has NO EFFECT (set, read back, re-run identical, then RESET -- model config is back to its documented Step-3.5 state); an explicit per-application base: WORKS (added to barbican alone, the error moved to barbican-vault).

OWNED -- I walked into a trap this repo has already recorded twice. The first two attempts at the second test staged a throwaway bundle under /tmp and returned ERROR no charm was found at "./bundle.yaml". That is the juju SNAP's PRIVATE /tmp, documented in CURRENT-STATE and in the phase-4 runbook. Under $HOME it worked immediately.

The rollback decision tree does not reach this case. runbooks/dc-dc-teardown-rollback.md:586 is written for tofu apply and defaults to fix-forward, which is measured not to work here. A partial juju deploy is a failure mode the repo does not cover -- logged as a finding, not fixed here.

Revert. The repo change is the capture and the CURRENT-STATE entry. The rack-side policy copy is reversed with rm -rf ~/repo-stage/policies. The model residue is the subject of the operator recovery decision and is deliberately untouched.