# Changelog 2026-07-31 -- the `prefer-ipv6` / v6 VIP charm research Session changelog (GA-R2/D1: ONE per session). Branch `dc-dc-stage5-preconditions`. Status claims live ONLY in `docs/CURRENT-STATE.md`. Context: bundle deploy attempt 1 failed 2026-07-31 with `unknown option "prefer-ipv6"` on barbican, and the operator ruled *"Research what those 6 charms do with v6 VIPs first (Recommended)"*. Stage 5 is blocked on that research. This session executes it, the operator RULES on the result, and the ruling is then BUILT and verified. **Scope correction to this header, made when it stopped being true.** It first read "Read-only throughout: no cloud mutation, no overlay edit, no ruling adopted", which was accurate for items 1-5. After the ruling landed the session edited the renderer, the gate, both overlays and both values files, and refreshed the deploy input staged on the dc0 rack. **No cloud state changed at any point** -- the `vr1-dc0` model is still empty and no `juju deploy` was run. --- ## Item 1 -- `docs/audit/stage5-prefer-ipv6-charm-research-20260731.txt` (NEW) **What.** The capture answering the ruled question, plus two findings that came out of the same measurement. **Method, and why it is not the Charmhub `config-yaml` API.** The attempt-1 root cause was measured against `config-yaml`, which answers only *is the knob declared*. It cannot answer *what the charm does*. This research downloads the charm each app actually deploys -- `juju download --channel --base ubuntu@22.04 --arch amd64` from voffice1 -- and reads the templates and code inside it. Revisions are recorded in the capture because `2024.1/stable` moves. Two declaring charms (keystone 857, cinder 820) were downloaded as CONTROLS: without one, "binds v4 only" and "binds v6 by another mechanism" cannot be told apart. **The answer.** `prefer-ipv6` is not what makes HAProxy bind `:::port`, and never was, for any of the thirteen. In both template families the v6 frontend bind is gated on `ipv6_enabled = not is_ipv6_disabled()` -- a read of the kernel `net.ipv6.conf.all.disable_ipv6` sysctl. Measured `0` on a live MAAS-deployed jammy node in this DC. The v6 VIP is assigned by pacemaker on per-address family detection, also independent of the option. Vault ships no haproxy template at all and its listener is the hardcoded literal `[::]:8200`. **What the option really does:** it is a unit address-family switch -- `get_relation_ip()` returns early with the unit's IPv6 address for EVERY relation, ignoring the network-space binding; plus `bind_host ::`, ip6-localhost stats, and keystone's `sync_db_with_multi_ipv6_addresses`. The value that sounds like a bind address, `haproxy_host`, is consumed by NO template in any charm downloaded. **Consequence recorded, not acted on:** `provider-bundle-check` invariant 9's mechanism claim is factually wrong; when a ruling lands the invariant must be REPLACED with a charm-schema-aware one and proven able to fail in both directions, never deleted to go green. **Revert.** `git rm docs/audit/stage5-prefer-ipv6-charm-research-20260731.txt`. The file is a read-only capture; nothing depends on it but the CURRENT-STATE entry in item 2. ## Item 2 -- `docs/CURRENT-STATE.md` section 1: the research result **What.** The research outcome appended under the 2026-07-31 ruling block that records the question -- the answer, its evidence, the invariant-9 consequence, the new question about the seven declaring charms, the option-name sweep result, and the runbook defect. Required in the same commit by GA-R1 rule 8 (C1); repo-lint L10 enforces it. **Revert.** `git revert` this commit, or delete the appended block; the preceding ruling text is unchanged and still reads correctly on its own. ## Item 3 -- MEASUREMENT ONLY (no artifact): full option-name sweep of the dc0 deploy input **What.** Attempt 1 proved `juju deploy --dry-run` does not validate config option NAMES, and juju aborts the whole bundle atomically on the first bad one -- so a second unknown option elsewhere costs another full attempt. Every option assignment in attempt 1's exact input (`bundle.yaml` + `vr1-dc0-vips` + `vr1-dc0-machines` + `vr1-dc0-octavia-pki`) was compared against each charm's own `config.yaml` downloaded at its pinned channel. **Result: 56 applications, 81 option assignments, 20 charm schemas, 0 unresolvable, and the only findings are `prefer-ipv6` on the same six.** No further option-name failure is waiting for attempt 2. An unreadable or unrecognised schema REFUSES rather than passing. **Not built as a gate, deliberately.** Hard rule 1 plus the standing operator directive: hardening is out of scope unless it blocks the deploy, and this one-shot measurement already de-risks attempt 2. The method is reproduced verbatim in the capture so it can be re-run or promoted later. CURRENT-STATE already records that no gate in this repo reads a charm config schema. **Both owed extensions were then RUN, after the ruling landed** -- option VALUE TYPES (every downloaded `config.yaml` carries a `type:` field the first pass discarded) and the dc1 input. **dc0 and dc1 each: 75 assignments / 20 schemas / 0 unknown name / 0 type mismatch / 0 note / 0 unresolvable, EXIT 0.** dc1 introduces no new charm@channel pair, checked rather than assumed. **PROOF OF TEETH.** A clean sweep across two inputs is exactly the implausibly-uniform result this repo has been burned by, so three defects were planted in a THROWAWAY copy: a restored `prefer-ipv6` on barbican (`[FAIL name]`), a quoted `"true"` on keystone's boolean (`[FAIL type]`), and an unquoted float where a string is declared (`[note]`). All three caught, EXIT 1. The EXIT-0 readings are a measurement, not a silent no-op. **Type judgement, stated because it decides what the sweep is worth.** A mismatch is reported only where juju's own coercion cannot save it (a boolean given a non-boolean, an int/float given a non-number). A scalar where a STRING is declared is a NOTE -- juju coerces it -- but it is still surfaced, because an unquoted value is how a version string silently becomes a float. An UNRECOGNISED declared type REFUSES rather than passing. **Revert.** Nothing to revert -- no repo artifact was created by this item. ## Item 5 -- MEASUREMENT ONLY: what option (b) would actually cost **What.** Before putting the fork to the operator, the implementation cost of (b) was measured rather than assumed. `prefer-ipv6: true` is INJECTED UNCONDITIONALLY by `scripts/render-dc-overlays.py:243-246` for every app whenever `fam == "dual"`, and `family: dual` is a SINGLE TOP-LEVEL key at `render/values/vr1-dc{0,1}-vips.yaml:3` -- not per-app data. **Consequence.** There is no value to edit. (b) means teaching the renderer which charms declare the option, re-pointing `tests/render-drift`, and replacing `provider-bundle-check` invariant 9 -- a code change with a harness, not a one-line data fix. The injection's own inline comment repeats the mechanism claim this session's research refutes. **Revert.** Nothing to revert. ## Item 4 -- LOGGED NOT FIXED: the dc0 deploy block omits a non-no-op overlay **What.** `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:553-557` gives the dc0 deploy without `overlays/vr1-dc0-machines.yaml`, while the dc1 block three lines below includes its machines overlay. Measured, the overlay's ONLY delta to the merged input is `ovn-chassis.options.bridge-interface-mappings = 'br-ex:52:54:00:8c:2a:8c br-ex:52:54:00:50:48:88'` -- the two dc0 compute provider MACs. Deploying dc0 exactly as the runbook reads would leave ovn-chassis with no provider bridge mapping, which surfaces later as tenant networks with no external path, not as a deploy error. Attempt 1 as executed correctly included the overlay. **Why not fixed here.** Hard rule 1 -- this session's step is the research. It is DOCFIX material and is recorded in CURRENT-STATE and in the capture so it cannot be lost. **Revert.** Nothing to revert. ## Item 6 -- THE RULING, and the three-part build that implements it **Ruling.** Recorded under GA-R5 in `docs/design-decisions.md` as a D-101 RULING NOTE dated 2026-07-31 (OPS under GA-R3; D-101's matrix UNAMENDED), and in `docs/CURRENT-STATE.md`. Committed and pushed BEFORE any dependent work. Operator utterance, verbatim: *"I want it to use IPv6, if there is spam mechanisms being applied then that is bad. Even if it doesn't cause an issue now, it might in the future. A clean IPv6 network is better than one with unusable and possibly future breaking configurations."* -- and, because that stated a principle rather than selecting an option, a confirming exchange: *"Yes -- keep the v6 legs, remove only the option"*. **(i) `scripts/provider-bundle-check.py` -- `PREFER_IPV6_CHARMS` (NEW) + invariant 9 REPLACED.** The constant is the measured authority, keyed by CHARM name (an application name is the deployer's choice; only a charm has a schema) and carrying the revisions and a re-measure-if-a-pin-moves warning. Invariant 9's old text -- "prefer-ipv6 makes HAProxy bind :::port in ADDITION to *:port, so the two must travel together" -- is refuted by this session's measurement and is REPLACED, never deleted: 9a the option is only legal on a charm that declares it (asserted on PRESENCE, since `prefer-ipv6: false` is the same fatal `unknown option` to juju as `true`); 9b on a declaring charm it still travels with the v6 legs, keeping the L3-9 protection intact for the seven; 9c arity. **(ii) `scripts/render-dc-overlays.py` + both overlays re-rendered.** The renderer emits the option only for a declaring charm, reading the set from provider-bundle-check with `ast` rather than restating it (the read-don't-restate rule `APP_OCTET` already uses). `render()` keeps its purity property -- the set is a PARAMETER, and the default REFUSES rather than defaulting to empty, because an empty set renders a plausible-looking artifact with the option nowhere. The values-file headers were updated too, since they carried the refuted mechanism claim. **Measured delta: exactly six lines removed per DC and nothing else -- every `vip` string byte-identical, so all v6 legs are retained.** **(iii) `tests/provider-bundle-check/` 44 -> 48.** T43 reproduces the attempt-1 defect; T44 proves presence-not-truthiness; T45 is the positive control (without it, 9a could be satisfied by rejecting everything); T46 asserts the DIAGNOSIS. T21's comment was re-pointed to its real rationale -- the assertion is unchanged and still correct, for a different reason than the one originally written. **Mutation pass -- six mutations, every one killed tests.** 9a deleted (T43+T44 die); 9a keyed on value (T44 dies); 9b restored to all charms (T19+T45 die, 10 failures); the charm list widened with barbican+vault (T43-T45 die); renderer emitting for every dual app, and `prefer6_charms()` returning empty instead of refusing (both kill `render-drift`). Source restored byte-identical after each. **ONE NEW ASSERTION WAS DECORATION AND WAS REPLACED.** T46's first form mutated only a vip; a non-declaring charm with no option has `prefer6 == dual == False` and reaches neither branch, so it could not fail under ANY mutation of the charm list. Re-written to assert that an input matching BOTH rules is diagnosed by 9a rather than 9b -- a misleading diagnosis would send the next session to add v6 legs when the fix is to remove the option -- and re-proven against the fold-9a-into-9b mutation. **Third time in a week that "prove each new assertion can FAIL" has caught a real hole.** **Verification.** Gauntlet ALL GREEN (93); repo-lint 0 fail; the fixed dc0 deploy input now sweeps CLEAN on option names (75 assignments, 20 schemas, 0 unknown, 0 unresolvable). ## Item 7 -- A DEFECT IN ITEM 6, found by a repo-wide grep and corrected the same session **What was wrong.** Two faults with one cause: item 6 measured and gated only the THIRTEEN VIP charms. 1. **`PREFER_IPV6_CHARMS` was WRONG.** Re-measured across ALL 33 charms in `bundle.yaml`: **twelve declare the option, not seven** -- the seven VIP ones plus `ceph-mon` 491, `ceph-osd` 953 (squid/stable), `nova-compute` 894, `hacluster` 166 (2.4/stable) and `mysql-innodb-cluster` 164 (8.0/stable). 2. **Invariant 9a was written INSIDE the VIP loop**, so an application with no `vip` never reached it. **Why it mattered, and why no test would have caught it.** The repo ALREADY has the missed case: `overlays/dc-dc-ipv6-family-matrix.yaml` sets `prefer-ipv6` on `ceph-mon`, which carries no VIP. That overlay is a LATER deploy step (`runbooks/dc-dc-phase4-juju-bundle-per-dc.md:728/737`), not part of the swept attempt-2 input -- **so the miss would have passed attempt 2 cleanly and surfaced at the step after, looking like a new fault.** ceph-mon happens to declare the option, so nothing would have broken today; the gate was simply blind there. **Fixed.** 9a now runs over EVERY application, before the VIP loop, and the VIP loop skips an app 9a already flagged so the diagnosis is not doubled. `ceph-mon`'s in-file claim ("CONFIRMED real option, charm-ceph-mon config.yaml") is now INDEPENDENTLY VERIFIED from the artifact rather than taken from the comment. Harness 48 -> **50/50**: T47 (a non-VIP app whose charm lacks the option FAILS) and T48 (ceph-mon PASSES), both mutation-proven -- re-scoping 9a back to the VIP loop kills T47; dropping ceph-mon from the list kills T48. Gauntlet ALL GREEN (93); repo-lint 0 fail. **The generalisable lesson, and it is one this repo keeps paying for:** I measured the population the QUESTION named (the 13 VIP charms) rather than the population the INVARIANT covers (every application juju validates). The grep that found it took two minutes and was not prompted by any gate. **Revert.** Folded into item 6's revert. **Revert (item 6).** `git revert` this commit. It restores the previous renderer behaviour, the previous invariant 9 and the previous overlays together -- they must move as one, since a re-rendered overlay without the matching renderer fails `tests/render-drift`. --- ## What was NOT done, stated so it is not assumed - **R2 is unamended and D-101's matrix is unchanged.** The dual-stack posture does not move; every v6 VIP leg is retained. - **Nothing changed for the SEVEN charms that DO declare the option** -- that is a separate, newly-raised question and was deliberately not bundled into the ruling (GA-R5: one decision per exchange). - **No cloud state changed.** The `vr1-dc0` model is still empty; no `juju deploy` was run this session. - **No charm-schema GATE was built.** The one-shot sweep is a measurement; a gate is logged as owed. ## Item 8 -- pre-deploy sequence: rack input refreshed, preflight run, dry-run green **Rack deploy input refreshed and hash-verified.** `~/repo-stage` on the D-138 client host is a COPY, not a git clone, so nothing updates it automatically and it was still carrying the pre-ruling vips overlay. Exactly one of four files differed; it was copied and all four then compared against repo HEAD. The gitignored `vr1-dc0-octavia-pki.yaml` was deliberately NOT touched -- a wholesale directory refresh would have risked clobbering or re-permissioning it (SEC-029) -- and it verifies byte-identical to voffice1's at `5fc117f1`, still `0600`. **Preflight captured**: `docs/audit/stage5-preflight-dc0-20260731.txt` (242 lines, exit 1). Run with `MAAS_PROFILE=vr1-dc0-region`, because preflight is REGION-BLIND and without it emits 19 false negatives. The instrument was proven current first -- `maas-profile-assert.sh vr1-dc0-region hot-kid` exits 0. P1-P4 and P7 PASS; P5 FAIL with 11 findings. **The P5 delta is enumerated, not waved through.** The 2026-07-30 acceptance covered SIX findings and says it covers "these six, enumerated, and nothing else". Five NEW ones -- all vr1-dc1 S2 rows from SEC-027/-028 -- are the D-137 forward register correctly reporting that dc1's region has not been built. The diff is appended to the capture and put to the operator as its own GA-R5 exchange. **Step 4.2 `--dry-run` against the fixed input: exit 0, 56 apps / 108 relations / 33 unit placements.** Its green is not evidence on option names -- that is attempt 1's lesson -- but it confirms the re-rendered overlay resolves and plans. The `ceph-osd tags=openstack` exposure is now MEASURED rather than reasoned: the plan places `ceph-osd/0..3` on machines 5-8 by explicit id, so the absent tag never has to match and the initial deploy is unaffected. The residual (a later UNPLACED `juju add-unit`) is unchanged and still logged. **Revert.** Nothing to revert in the repo beyond the capture. The rack-side file copy is reversed by copying the previous overlay back; its pre-change hash was `3ae79e82`. ## Item 9 -- SECOND RULING: the five new P5 findings accepted **Question as presented.** Preflight for dc0 is RED on P5 only. It reports 11 findings; the 2026-07-30 acceptance covered SIX, enumerated, and says it covers "these six, enumerated, and nothing else". All six are still present. Five are NEW, all vr1-dc1, all opened after that ruling -- `maas-region-db-password`, `maas-region-admin-password`, `maas-region-api-key.txt` (SEC-027) and `maas-juju-api-key.txt`, `maas-juju-user-password` (SEC-028) -- and every one is the D-137 forward register correctly reporting that dc1's MAAS region is authored but not applied. Accept and proceed, or stop and remediate? **Operator answer, exact utterance: "Accept the five and proceed to the dc0 deploy".** **Consequence.** The five are carried as known, by-design absences on their existing SEC-027 / SEC-028 rows. They describe dc1, which is not built; nothing about them affects the dc0 deploy. `preflight.sh` keeps exiting FAIL on P5 for the rest of this stage, that RED is ruled-accepted, and it must NOT be made green by deleting or weakening a matrix row. **Like its predecessor this covers these FIVE, enumerated, and nothing else.** Total accepted at P5 is now ELEVEN across the two rulings. **Revert.** A ruling is a record, not a change; there is nothing to revert. Withdrawing it would require a new GA-R5 exchange.