# Changelog 2026-07-31 -- the `prefer-ipv6` / v6 VIP charm research

Session changelog (GA-R2/D1: ONE per session). Branch `dc-dc-stage5-preconditions`.
Status claims live ONLY in `docs/CURRENT-STATE.md`.

Context: bundle deploy attempt 1 failed 2026-07-31 with
`unknown option "prefer-ipv6"` on barbican, and the operator ruled *"Research what
those 6 charms do with v6 VIPs first (Recommended)"*. Stage 5 is blocked on that
research. This session executes it, the operator RULES on the result, and the ruling is
then BUILT and verified.

**Scope correction to this header, made when it stopped being true.** It first read
"Read-only throughout: no cloud mutation, no overlay edit, no ruling adopted", which was
accurate for items 1-5. After the ruling landed the session edited the renderer, the
gate, both overlays and both values files, and refreshed the deploy input staged on the
dc0 rack. **No cloud state changed at any point** -- the `vr1-dc0` model is still empty
and no `juju deploy` was run.

---

## Item 1 -- `docs/audit/stage5-prefer-ipv6-charm-research-20260731.txt` (NEW)

**What.** The capture answering the ruled question, plus two findings that came out
of the same measurement.

**Method, and why it is not the Charmhub `config-yaml` API.** The attempt-1 root
cause was measured against `config-yaml`, which answers only *is the knob declared*.
It cannot answer *what the charm does*. This research downloads the charm each app
actually deploys -- `juju download <charm> --channel <pinned> --base ubuntu@22.04
--arch amd64` from voffice1 -- and reads the templates and code inside it. Revisions
are recorded in the capture because `2024.1/stable` moves. Two declaring charms
(keystone 857, cinder 820) were downloaded as CONTROLS: without one, "binds v4 only"
and "binds v6 by another mechanism" cannot be told apart.

**The answer.** `prefer-ipv6` is not what makes HAProxy bind `:::port`, and never was,
for any of the thirteen. In both template families the v6 frontend bind is gated on
`ipv6_enabled = not is_ipv6_disabled()` -- a read of the kernel
`net.ipv6.conf.all.disable_ipv6` sysctl. Measured `0` on a live MAAS-deployed jammy
node in this DC. The v6 VIP is assigned by pacemaker on per-address family detection,
also independent of the option. Vault ships no haproxy template at all and its
listener is the hardcoded literal `[::]:8200`.

**What the option really does:** it is a unit address-family switch --
`get_relation_ip()` returns early with the unit's IPv6 address for EVERY relation,
ignoring the network-space binding; plus `bind_host ::`, ip6-localhost stats, and
keystone's `sync_db_with_multi_ipv6_addresses`. The value that sounds like a bind
address, `haproxy_host`, is consumed by NO template in any charm downloaded.

**Consequence recorded, not acted on:** `provider-bundle-check` invariant 9's
mechanism claim is factually wrong; when a ruling lands the invariant must be
REPLACED with a charm-schema-aware one and proven able to fail in both directions,
never deleted to go green.

**Revert.** `git rm docs/audit/stage5-prefer-ipv6-charm-research-20260731.txt`. The
file is a read-only capture; nothing depends on it but the CURRENT-STATE entry in
item 2.

## Item 2 -- `docs/CURRENT-STATE.md` section 1: the research result

**What.** The research outcome appended under the 2026-07-31 ruling block that
records the question -- the answer, its evidence, the invariant-9 consequence, the
new question about the seven declaring charms, the option-name sweep result, and the
runbook defect. Required in the same commit by GA-R1 rule 8 (C1); repo-lint L10
enforces it.

**Revert.** `git revert` this commit, or delete the appended block; the preceding
ruling text is unchanged and still reads correctly on its own.

## Item 3 -- MEASUREMENT ONLY (no artifact): full option-name sweep of the dc0 deploy input

**What.** Attempt 1 proved `juju deploy --dry-run` does not validate config option
NAMES, and juju aborts the whole bundle atomically on the first bad one -- so a second
unknown option elsewhere costs another full attempt. Every option assignment in
attempt 1's exact input (`bundle.yaml` + `vr1-dc0-vips` + `vr1-dc0-machines` +
`vr1-dc0-octavia-pki`) was compared against each charm's own `config.yaml` downloaded
at its pinned channel.

**Result: 56 applications, 81 option assignments, 20 charm schemas, 0 unresolvable,
and the only findings are `prefer-ipv6` on the same six.** No further option-name
failure is waiting for attempt 2. An unreadable or unrecognised schema REFUSES rather
than passing.

**Not built as a gate, deliberately.** Hard rule 1 plus the standing operator
directive: hardening is out of scope unless it blocks the deploy, and this one-shot
measurement already de-risks attempt 2. The method is reproduced verbatim in the
capture so it can be re-run or promoted later. CURRENT-STATE already records that no
gate in this repo reads a charm config schema.

**Both owed extensions were then RUN, after the ruling landed** -- option VALUE TYPES
(every downloaded `config.yaml` carries a `type:` field the first pass discarded) and
the dc1 input. **dc0 and dc1 each: 75 assignments / 20 schemas / 0 unknown name / 0 type
mismatch / 0 note / 0 unresolvable, EXIT 0.** dc1 introduces no new charm@channel pair,
checked rather than assumed.

**PROOF OF TEETH.** A clean sweep across two inputs is exactly the implausibly-uniform
result this repo has been burned by, so three defects were planted in a THROWAWAY copy:
a restored `prefer-ipv6` on barbican (`[FAIL name]`), a quoted `"true"` on keystone's
boolean (`[FAIL type]`), and an unquoted float where a string is declared (`[note]`).
All three caught, EXIT 1. The EXIT-0 readings are a measurement, not a silent no-op.

**Type judgement, stated because it decides what the sweep is worth.** A mismatch is
reported only where juju's own coercion cannot save it (a boolean given a non-boolean,
an int/float given a non-number). A scalar where a STRING is declared is a NOTE -- juju
coerces it -- but it is still surfaced, because an unquoted value is how a version string
silently becomes a float. An UNRECOGNISED declared type REFUSES rather than passing.

**Revert.** Nothing to revert -- no repo artifact was created by this item.

## Item 5 -- MEASUREMENT ONLY: what option (b) would actually cost

**What.** Before putting the fork to the operator, the implementation cost of (b) was
measured rather than assumed. `prefer-ipv6: true` is INJECTED UNCONDITIONALLY by
`scripts/render-dc-overlays.py:243-246` for every app whenever `fam == "dual"`, and
`family: dual` is a SINGLE TOP-LEVEL key at `render/values/vr1-dc{0,1}-vips.yaml:3` --
not per-app data.

**Consequence.** There is no value to edit. (b) means teaching the renderer which
charms declare the option, re-pointing `tests/render-drift`, and replacing
`provider-bundle-check` invariant 9 -- a code change with a harness, not a one-line
data fix. The injection's own inline comment repeats the mechanism claim this
session's research refutes.

**Revert.** Nothing to revert.

## Item 4 -- LOGGED NOT FIXED: the dc0 deploy block omits a non-no-op overlay

**What.** `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:553-557` gives the dc0 deploy
without `overlays/vr1-dc0-machines.yaml`, while the dc1 block three lines below
includes its machines overlay. Measured, the overlay's ONLY delta to the merged input
is `ovn-chassis.options.bridge-interface-mappings = 'br-ex:52:54:00:8c:2a:8c
br-ex:52:54:00:50:48:88'` -- the two dc0 compute provider MACs. Deploying dc0 exactly
as the runbook reads would leave ovn-chassis with no provider bridge mapping, which
surfaces later as tenant networks with no external path, not as a deploy error.
Attempt 1 as executed correctly included the overlay.

**Why not fixed here.** Hard rule 1 -- this session's step is the research. It is
DOCFIX material and is recorded in CURRENT-STATE and in the capture so it cannot be
lost.

**Revert.** Nothing to revert.

## Item 6 -- THE RULING, and the three-part build that implements it

**Ruling.** Recorded under GA-R5 in `docs/design-decisions.md` as a D-101 RULING NOTE
dated 2026-07-31 (OPS under GA-R3; D-101's matrix UNAMENDED), and in
`docs/CURRENT-STATE.md`. Committed and pushed BEFORE any dependent work. Operator
utterance, verbatim: *"I want it to use IPv6, if there is spam mechanisms being applied
then that is bad. Even if it doesn't cause an issue now, it might in the future. A clean
IPv6 network is better than one with unusable and possibly future breaking
configurations."* -- and, because that stated a principle rather than selecting an
option, a confirming exchange: *"Yes -- keep the v6 legs, remove only the option"*.

**(i) `scripts/provider-bundle-check.py` -- `PREFER_IPV6_CHARMS` (NEW) + invariant 9
REPLACED.** The constant is the measured authority, keyed by CHARM name (an application
name is the deployer's choice; only a charm has a schema) and carrying the revisions and
a re-measure-if-a-pin-moves warning. Invariant 9's old text -- "prefer-ipv6 makes HAProxy
bind :::port in ADDITION to *:port, so the two must travel together" -- is refuted by
this session's measurement and is REPLACED, never deleted: 9a the option is only legal on
a charm that declares it (asserted on PRESENCE, since `prefer-ipv6: false` is the same
fatal `unknown option` to juju as `true`); 9b on a declaring charm it still travels with
the v6 legs, keeping the L3-9 protection intact for the seven; 9c arity.

**(ii) `scripts/render-dc-overlays.py` + both overlays re-rendered.** The renderer emits
the option only for a declaring charm, reading the set from provider-bundle-check with
`ast` rather than restating it (the read-don't-restate rule `APP_OCTET` already uses).
`render()` keeps its purity property -- the set is a PARAMETER, and the default REFUSES
rather than defaulting to empty, because an empty set renders a plausible-looking
artifact with the option nowhere. The values-file headers were updated too, since they
carried the refuted mechanism claim. **Measured delta: exactly six lines removed per DC
and nothing else -- every `vip` string byte-identical, so all v6 legs are retained.**

**(iii) `tests/provider-bundle-check/` 44 -> 48.** T43 reproduces the attempt-1 defect;
T44 proves presence-not-truthiness; T45 is the positive control (without it, 9a could be
satisfied by rejecting everything); T46 asserts the DIAGNOSIS. T21's comment was
re-pointed to its real rationale -- the assertion is unchanged and still correct, for a
different reason than the one originally written.

**Mutation pass -- six mutations, every one killed tests.** 9a deleted (T43+T44 die); 9a
keyed on value (T44 dies); 9b restored to all charms (T19+T45 die, 10 failures); the
charm list widened with barbican+vault (T43-T45 die); renderer emitting for every dual
app, and `prefer6_charms()` returning empty instead of refusing (both kill
`render-drift`). Source restored byte-identical after each.

**ONE NEW ASSERTION WAS DECORATION AND WAS REPLACED.** T46's first form mutated only a
vip; a non-declaring charm with no option has `prefer6 == dual == False` and reaches
neither branch, so it could not fail under ANY mutation of the charm list. Re-written to
assert that an input matching BOTH rules is diagnosed by 9a rather than 9b -- a
misleading diagnosis would send the next session to add v6 legs when the fix is to remove
the option -- and re-proven against the fold-9a-into-9b mutation. **Third time in a week
that "prove each new assertion can FAIL" has caught a real hole.**

**Verification.** Gauntlet ALL GREEN (93); repo-lint 0 fail; the fixed dc0 deploy input
now sweeps CLEAN on option names (75 assignments, 20 schemas, 0 unknown, 0 unresolvable).

## Item 7 -- A DEFECT IN ITEM 6, found by a repo-wide grep and corrected the same session

**What was wrong.** Two faults with one cause: item 6 measured and gated only the
THIRTEEN VIP charms.

1. **`PREFER_IPV6_CHARMS` was WRONG.** Re-measured across ALL 33 charms in
   `bundle.yaml`: **twelve declare the option, not seven** -- the seven VIP ones plus
   `ceph-mon` 491, `ceph-osd` 953 (squid/stable), `nova-compute` 894, `hacluster` 166
   (2.4/stable) and `mysql-innodb-cluster` 164 (8.0/stable).
2. **Invariant 9a was written INSIDE the VIP loop**, so an application with no `vip`
   never reached it.

**Why it mattered, and why no test would have caught it.** The repo ALREADY has the
missed case: `overlays/dc-dc-ipv6-family-matrix.yaml` sets `prefer-ipv6` on `ceph-mon`,
which carries no VIP. That overlay is a LATER deploy step
(`runbooks/dc-dc-phase4-juju-bundle-per-dc.md:728/737`), not part of the swept attempt-2
input -- **so the miss would have passed attempt 2 cleanly and surfaced at the step
after, looking like a new fault.** ceph-mon happens to declare the option, so nothing
would have broken today; the gate was simply blind there.

**Fixed.** 9a now runs over EVERY application, before the VIP loop, and the VIP loop
skips an app 9a already flagged so the diagnosis is not doubled. `ceph-mon`'s in-file
claim ("CONFIRMED real option, charm-ceph-mon config.yaml") is now INDEPENDENTLY
VERIFIED from the artifact rather than taken from the comment. Harness 48 -> **50/50**:
T47 (a non-VIP app whose charm lacks the option FAILS) and T48 (ceph-mon PASSES), both
mutation-proven -- re-scoping 9a back to the VIP loop kills T47; dropping ceph-mon from
the list kills T48. Gauntlet ALL GREEN (93); repo-lint 0 fail.

**The generalisable lesson, and it is one this repo keeps paying for:** I measured the
population the QUESTION named (the 13 VIP charms) rather than the population the
INVARIANT covers (every application juju validates). The grep that found it took two
minutes and was not prompted by any gate.

**Revert.** Folded into item 6's revert.

**Revert (item 6).** `git revert` this commit. It restores the previous renderer
behaviour, the previous invariant 9 and the previous overlays together -- they must move as one, since
a re-rendered overlay without the matching renderer fails `tests/render-drift`.

---

## What was NOT done, stated so it is not assumed

- **R2 is unamended and D-101's matrix is unchanged.** The dual-stack posture does not
  move; every v6 VIP leg is retained.
- **Nothing changed for the SEVEN charms that DO declare the option** -- that is a
  separate, newly-raised question and was deliberately not bundled into the ruling
  (GA-R5: one decision per exchange).
- **No cloud state changed.** The `vr1-dc0` model is still empty; no `juju deploy` was
  run this session.
- **No charm-schema GATE was built.** The one-shot sweep is a measurement; a gate is
  logged as owed.

## Item 8 -- pre-deploy sequence: rack input refreshed, preflight run, dry-run green

**Rack deploy input refreshed and hash-verified.** `~/repo-stage` on the D-138 client
host is a COPY, not a git clone, so nothing updates it automatically and it was still
carrying the pre-ruling vips overlay. Exactly one of four files differed; it was copied
and all four then compared against repo HEAD. The gitignored `vr1-dc0-octavia-pki.yaml`
was deliberately NOT touched -- a wholesale directory refresh would have risked
clobbering or re-permissioning it (SEC-029) -- and it verifies byte-identical to
voffice1's at `5fc117f1`, still `0600`.

**Preflight captured**: `docs/audit/stage5-preflight-dc0-20260731.txt` (242 lines, exit
1). Run with `MAAS_PROFILE=vr1-dc0-region`, because preflight is REGION-BLIND and
without it emits 19 false negatives. The instrument was proven current first --
`maas-profile-assert.sh vr1-dc0-region hot-kid` exits 0. P1-P4 and P7 PASS; P5 FAIL with
11 findings.

**The P5 delta is enumerated, not waved through.** The 2026-07-30 acceptance covered SIX
findings and says it covers "these six, enumerated, and nothing else". Five NEW ones --
all vr1-dc1 S2 rows from SEC-027/-028 -- are the D-137 forward register correctly
reporting that dc1's region has not been built. The diff is appended to the capture and
put to the operator as its own GA-R5 exchange.

**Step 4.2 `--dry-run` against the fixed input: exit 0, 56 apps / 108 relations / 33
unit placements.** Its green is not evidence on option names -- that is attempt 1's
lesson -- but it confirms the re-rendered overlay resolves and plans. The `ceph-osd
tags=openstack` exposure is now MEASURED rather than reasoned: the plan places
`ceph-osd/0..3` on machines 5-8 by explicit id, so the absent tag never has to match and
the initial deploy is unaffected. The residual (a later UNPLACED `juju add-unit`) is
unchanged and still logged.

**Revert.** Nothing to revert in the repo beyond the capture. The rack-side file copy is
reversed by copying the previous overlay back; its pre-change hash was `3ae79e82`.

## Item 9 -- SECOND RULING: the five new P5 findings accepted

**Question as presented.** Preflight for dc0 is RED on P5 only. It reports 11 findings;
the 2026-07-30 acceptance covered SIX, enumerated, and says it covers "these six,
enumerated, and nothing else". All six are still present. Five are NEW, all vr1-dc1, all
opened after that ruling -- `maas-region-db-password`, `maas-region-admin-password`,
`maas-region-api-key.txt` (SEC-027) and `maas-juju-api-key.txt`,
`maas-juju-user-password` (SEC-028) -- and every one is the D-137 forward register
correctly reporting that dc1's MAAS region is authored but not applied. Accept and
proceed, or stop and remediate?

**Operator answer, exact utterance: "Accept the five and proceed to the dc0 deploy".**

**Consequence.** The five are carried as known, by-design absences on their existing
SEC-027 / SEC-028 rows. They describe dc1, which is not built; nothing about them
affects the dc0 deploy. `preflight.sh` keeps exiting FAIL on P5 for the rest of this
stage, that RED is ruled-accepted, and it must NOT be made green by deleting or
weakening a matrix row. **Like its predecessor this covers these FIVE, enumerated, and
nothing else.** Total accepted at P5 is now ELEVEN across the two rulings.

**Revert.** A ruling is a record, not a change; there is nothing to revert. Withdrawing
it would require a new GA-R5 exchange.

## Item 10 -- BUNDLE DEPLOY ATTEMPT 2: the prefer-ipv6 defect is gone; two new ones found

**What happened.** The deploy was run from the dc0 rack against the fixed, hash-verified
input. The `prefer-ipv6` failure did not recur. Two new defects did, and for the first
time the model is PARTIALLY POPULATED: **23 applications, 0 machines, 0 units** --
measured immediately, not assumed. Attempt 1 aborted during VALIDATION and left the model
empty; this one got past validation into EXECUTION, which is why it left residue.
**Nothing is provisioned**: no MAAS machine left `Ready`, no disk was written.

**Defect 1 -- the deploy input on the client host is a PARTIAL COPY of the repo.**
`~/repo-stage` has no `policies/`, and `bundle.yaml:214` is the only local-file reference
in the whole deploy input (`policyd-override: ./policies/overrides.zip`), resolved
relative to the bundle and uploaded as a charm resource -- so it must exist ON THE HOST
THAT DEPLOYS. Nothing caught it: `--dry-run` does not upload resources (the SECOND
`--dry-run` blind spot in two days), attempt 1 aborted before resource upload, and
`preflight.sh` runs on voffice1 where `policies/` DOES exist. **Same wrong-host instrument
class as the `dc-mirror.sh` and egress-probe errors, and the second instance this
session.** Fixed forward under gating: both policy files copied and sha256-verified.

**Defect 2 -- a bundle that relies on `default-base` is deployable exactly once.** The
re-run failed with `barbican: downgrades are not currently supported: deployed revision
265 is newer than requested revision 261`. Measured: barbican `2024.1/stable` is rev 265
at `--base ubuntu@22.04` and 261 at `--base ubuntu@24.04` or with no base. The 23
existing apps are all on 22.04, so run 2a honoured `bundle.yaml:85` -- but on the re-run,
for an application that ALREADY EXISTS, juju resolves without that default.

**Two fixes tested READ-ONLY** (`--dry-run` reproduces the error against the populated
model, making it a safe harness): a model-level `default-base=ubuntu@22.04` has NO EFFECT
(set, read back, re-run identical, then RESET -- model config is back to its documented
Step-3.5 state); an explicit per-application `base:` WORKS (added to barbican alone, the
error moved to barbican-vault).

**OWNED -- I walked into a trap this repo has already recorded twice.** The first two
attempts at the second test staged a throwaway bundle under `/tmp` and returned `ERROR no
charm was found at "./bundle.yaml"`. That is the juju SNAP's PRIVATE `/tmp`, documented in
CURRENT-STATE and in the phase-4 runbook. Under `$HOME` it worked immediately.

**The rollback decision tree does not reach this case.**
`runbooks/dc-dc-teardown-rollback.md:586` is written for `tofu apply` and defaults to
fix-forward, which is measured not to work here. A partial `juju deploy` is a failure
mode the repo does not cover -- logged as a finding, not fixed here.

**Revert.** The repo change is the capture and the CURRENT-STATE entry. The rack-side
policy copy is reversed with `rm -rf ~/repo-stage/policies`. The model residue is the
subject of the operator recovery decision and is deliberately untouched.

## Item 11 -- THIRD RULING: option D -- fix the bundle AND clear the model

**Question as presented.** Recovery from the partial deploy. Options put: (A) clear the
23 definitions and deploy once cleanly from the unchanged bundle, logging the re-run
defect; (B) add an explicit per-app `base:` to all 56 and continue incrementally.

**Operator answer, exact utterance:** *"I want to do option D: Fix the bundle by defining
the base 22.04 on each app and then clearing the modeling and deploying from a clear
model. It is very cheap to clear and redeploy."*

**The operator composed a fourth option, and it is strictly stronger than any offered.**
(B)'s durable fix lands, AND the deploy runs the path that is MEASURED to work -- a clean
deploy onto an empty model, which is exactly what run 2a did before it hit the missing
resource file -- rather than juju's incremental re-run path, which is the one carrying
the defect. Their rationale settles the cost question I had left open: clearing is cheap
because nothing is provisioned.

**Classification.** OPS under GA-R3 -- a bundle defect fix; doubt resolves DOWN, no
D-number. No ruled surface changes: an explicit `base:` per application states what
`default-base: ubuntu@22.04/stable` at `bundle.yaml:85` already means, it does not alter
it. The invariant it establishes -- a bundle whose applications rely solely on
`default-base` is deployable exactly once -- is skill material at stage close.

**Revert.** A ruling is a record; withdrawing it needs a new GA-R5 exchange.

## Item 12 -- BUILD option D's first half: explicit base on all 56, gated by invariant 12

**`bundle.yaml`** -- all 56 applications now carry `base: ubuntu@22.04/stable`, plus a
note at the `default-base` line recording WHY the redundancy is deliberate and telling a
future reader not to simplify it away.

**Proven against the real model, not inferred.** The earlier one-app test showed only
that the error MOVED to the next app. The fully-based bundle was staged as a throwaway on
the rack and `--dry-run` against the LIVE 23-application model -- the same command that
errored on `barbican` 265-vs-261 -- **exits 0**.

**Subordinates are in scope, measured:** `mysql-router` is rev 1154 at 22.04 and 1178 at
24.04, so exempting subordinates would have left the trap open. `hacluster` shares rev
166 across both bases, which is not a property to depend on.

**Two silent under-matches, both caught by cross-checking the edit against the parsed
file rather than by review.**
1. A first pass keyed on the block form `^  <app>:$` inserted **44 of 56** -- the twelve
   `-hacluster` applications are single-line FLOW MAPPINGS and matched nothing. The
   inserter asserts its app list equals what `yaml.safe_load` sees and refuses otherwise,
   which is what caught it. The final edit also asserts the ARTIFACT: every app carries
   the base, and the parsed structures with `base` stripped are IDENTICAL to the
   original.
2. **`overlays/dc-ha-scaleup.yaml` DEFINES `vault-hacluster`, an application that exists
   nowhere in `bundle.yaml`, and it had no base.** That overlay is a LATER deploy step,
   so this would have re-opened the trap after the deploy succeeded. **Found by the new
   gate on its FIRST run, in a file the fix was not looking at** -- the same
   later-step-overlay class as the `ceph-mon` `prefer-ipv6` miss in item 7. Fixed.

**`provider-bundle-check` invariant 12 (NEW).** Every application carries an explicit
`base` equal to the bundle's OWN `default-base` -- keyed on that value, not a literal, so
a future base change moves one line. A bundle with no `default-base` REFUSES rather than
passing vacuously.

**Five cases, each mutation-proven individually** (`tests/provider-bundle-check` 50 ->
55/55): T49 stripping every base -- a byte-identical PASS before this landed; T50 a
single app; T51 a base disagreeing with `default-base`; T52 a SUBORDINATE, not exempted;
T53 the refusal. Neutering `nobase` kills T49/T50/T52; neutering `wrongbase` kills T51;
neutering the refuse branch kills T53. Source restored byte-identical after each.

Gauntlet ALL GREEN (93); repo-lint 0 fail.

**Revert.** `git revert` this commit -- bundle, overlay, gate and harness must move
together, since the gate fails on a bundle without the base lines.

## Item 13 -- option D half 2: model cleared, and BUNDLE DEPLOY ATTEMPT 3 SUCCEEDED

**`Deploy of bundle completed.` EXIT 0.** Capture
`docs/audit/stage5-dc0-deploy-attempt3-20260731.txt`.

**The clear.** All 23 application definitions removed INDIVIDUALLY, never batched (hard
rule 3), each read back. The precondition was re-verified immediately before the first
removal -- 0 units, 0 machines, nothing provisioned -- and the model read back as `Model
"vr1-dc0" is empty` afterwards. **Step-3.5 state survived, checked not assumed:**
`apt-mirror` still set, all six spaces still bound with both address families.

**Measured immediately after the deploy:** juju 56 applications / 9 machines / 33 units,
machines `pending`/`allocating`; MAAS 9 `Deploying` + 1 `Deployed` (the controller).

**Operational note, recorded because the first removal read as a failure:** `juju
remove-application` PROMPTS by default and aborts on non-interactive stdin (`ERROR
application removal: aborted`, exit 1). `--no-prompt` is required from a non-interactive
session.

**What is NOT claimed.** `Deploy of bundle completed.` means juju ACCEPTED and QUEUED the
bundle. It does not mean the cloud is up. The settle takes hours and nothing here asserts
unit health, relation settling or any service verdict.

**G17's dc0 half is now genuinely armable** -- the existing capture was taken on the
CONTROLLER VM and says so. The nine role nodes are booting for the first time.

**Revert.** Not revertible as a repo change; the deploy is live cloud state. Reversing it
means `juju remove-application` per app or destroying the model, which is an operator
decision.

## Item 14 -- DEPLOY BLOCKER FOUND, LOGGED NOT FIXED: the dc0 mirror lacks `jammy-backports`

**Measured end to end.** 22 of 33 units in `error`, every one `hook failed: "install"`.
Hook output: `E: The repository 'http://10.12.8.4/ubuntu jammy-backports Release' does not
have a Release file.` Per-suite probe against the mirror: `jammy` / `jammy-security` /
`jammy-updates` **200**, `jammy-backports` **404** -- exactly D-135's "jammy triple"
scope. The node's `/etc/apt/sources.list` carries `deb http://10.12.8.4/ubuntu
jammy-backports ...`, because the Step-3.5 `apt-mirror` model-config rewrites the stock
Ubuntu sources to point EVERY suite at the DC mirror, including one it was never built to
hold. All 9 machines are `started`/`running` -- purely the artifact layer, and units retry.

**Not fixed: D-135 scope is a ruled surface** (hard rule 1). Add the suite and re-sync, or
drop backports from the nodes' sources -- operator's call.

**`dc-mirror.sh check dc0` PASSES while this is true.** It verifies sync STATUS, not suite
COVERAGE against what the deployed image asks for. Logged as the gap, not fixed.

**Revert.** Nothing to revert.

## Item 15 -- QUEUED BY OPERATOR DIRECTION: per-DC Tailscale as a standing standup requirement

**Operator direction, verbatim (2026-07-31):** *"A closer real work analog would be to
install tailscale in DC0, DC1, and any future DCs as they would be running their own
tailscale on, or beside, the utility node for that particular DC. Queue up the tailscale
installation in DC0 and add it as a requirement for DC1 and any future installations."*

**QUEUED, NOT EXECUTED** -- the current step is the Stage-5 deploy. Home of record:
`docs/dc-dc-deployment-workflow.md` tooling gap register **item 21** (new), carrying the
requirement, a per-DC definition-of-done, the vendor-documented build constraints, and the
four sub-decisions needing GA-R5 rulings before any build: the utility-band OCTET (D-134's
map is a standing cross-DC standard, so it cannot be chosen per-DC), the
admin-reachability model (star vs mesh -- the real architecture question at region-region
scale), HA count per site, and SNAT on/off.

**This is EXECUTION of the already-ruled D-129(iii) shape**, not a new decision.

**Measured while scoping it, all read-only:**
- **SEC-010 does not need relaxing.** Tailscale reaches each router as ordinary WireGuard
  UDP to the VM's own address; the forwarding to permit is on that VM between `tailscale0`
  and metal-admin -- a different decision from the transit-leg DROP rule.
- **voffice1 holds no route to `10.12.x` at all** (`ip route get 10.12.8.1` falls to the
  default; only the two transit /30s), so the tailnet stops at Office1 by construction.
  A subnet router can only forward to nets its own host can reach.
- **The existing Office1 node is UNTAGGED** (`AdvertiseTags: None`) -- user-owned, with a
  180-day key-expiry clock on the operator's only tailnet path. A defect to fix alongside.
- **D-129(iii) cites D-107 as governing the Office1 installation; D-107 rules nothing
  about Tailscale** (read in full). Second instance of the F7 miscitation class, this time
  inside a ruling.

**Research basis.** Tailscale KB (subnet routers, HA, site-to-site, ACL tags, autoApprovers,
key expiry, 4via6, exit nodes) and Headscale's own docs, fetched and quoted rather than
recalled. Two findings the build must honour: `autoApprovers` does not apply retroactively,
so the policy must exist BEFORE a router first advertises; and Headscale does not document
4via6 support, which makes globally-unique per-DC management CIDRs a hard constraint at
region scale rather than a nicety.

**Revert.** `git revert` -- register item 21 and the CURRENT-STATE entry move together.

## Item 16 -- OPERATOR DIRECTION: all Tailscale decisions deferred to the end of the step

**Operator direction, verbatim (2026-07-31):** *"To make sure we don't muddle the in
process deployment, queue all the tailscale decisions to the end of the step. When you
bring them back up give me decisions choices with your recommendation for each."*

All four sub-decisions in gap register item 21 (utility-band octet; admin-reachability
model star-vs-mesh; HA count per site; SNAT on/off) are DEFERRED to the end of the
current step. When re-raised they are to be presented as decision CHOICES, each with an
explicit recommendation. Nothing about Tailscale is to be built or ruled before then.

**Revert.** Nothing to revert.

## Item 17 -- MEASUREMENT: the size of `jammy-backports` if it is added to the dc0 mirror

Measured 2026-07-31 against `archive.ubuntu.com` from the dc0 rack, summing the `Size:`
field of every entry in each component's `binary-amd64/Packages.gz`, at the EXACT scope
`dc-mirror.sh:177-181` uses (`--section=main,restricted,universe,multiverse --arch=amd64
--nosource --diff=none`):

| component | bytes | packages |
|---|---|---|
| main | 859,001,998 | 338 |
| universe | 211,685,404 | 123 |
| restricted | 0 | 0 (empty in backports) |
| multiverse | 0 | 0 (empty in backports) |
| **total** | **1,070,687,402 (~1.00 GiB)** | **461** |

**Context measured alongside:** the existing dc0 mirror is **951 GB**
(`/var/lib/dc-mirror/ubuntu`), with **1.8 T free** on the rack. Backports is therefore
**~0.1% of the mirror** and a non-issue for capacity.

**Sanity checks run, because a clean total is exactly the shape this repo has been burned
by:** the five largest entries were listed to confirm the `Size:` field was being read
(top entry `libreoffice-dev-doc` at 190 MB -- plausible); a separate `binary-all` index
was probed and returns **404**, so arch:all packages are already inside `binary-amd64`
(301 of main's 338 entries) and are not double-counted or missed.

**Worth knowing before choosing: ~462 MB of the 859 MB in main is LibreOffice** (two
`libreoffice-dev-doc` versions at 190 MB + 184 MB, plus `-core` and `-common`) -- i.e.
roughly 46% of the suite is a desktop package set no OpenStack control-plane node will
ever install.

**Transfer estimate, stated as the weak measurement it is:** a single-stream fetch of one
Packages.gz measured **932 KB/s** egress from the rack, which puts 1.07 GB at roughly
**19 minutes**. That is one file on one stream; debmirror parallelises, so treat it as an
upper bound rather than a figure.

**Revert.** Measurement only; nothing changed.

## Item 18 -- BUILD the D-135 amendment: `jammy-backports` added at SOURCE, T10 re-pointed

**`scripts/dc-mirror.sh`** -- the generated `<site>-mirror-sync` `--dist=` line gains
`jammy-backports`, plus a comment block at the generator recording WHY it is not optional:
the MAAS jammy image ships backports in its `sources.list` and juju's `apt-mirror`
model-config rewrites every suite to this mirror, so a mirror without it 404s and every
charm install hook fails. That cost 22 units on the 2026-07-31 dc0 deploy.

**Fixed at SOURCE, not only on the live rack.** A live-only edit would have left the
generator regenerating the old scope at the next `install` or at any future DC standup --
the "ruled is not built" class inverted.

**`tests/dc-mirror` T10 RE-POINTED, not deleted.** It asserted
`--dist=jammy,jammy-updates,jammy-security` verbatim, which was correct for the old scope
and had to MOVE when the scope changed -- the standing rule that a test asserting a
literal finding gets re-pointed to the new invariant, never removed to go green. The
reason is recorded in-file so a later reader does not "simplify" it back.

**Mutation-proven:** reverting the generator's dist line to the old scope turns T10 RED
(23/1); restored byte-identical, 24/24. Gauntlet ALL GREEN (93); repo-lint 0 fail.

**Revert.** `git revert` -- generator and harness move together.

## Item 19 -- BUILD D-135/D-107 RULING 2, REPO SIDE ONLY: the in-DC snap forward proxy

**What.** `scripts/dc-snap-proxy.sh` (NEW, `check|install|node`, site-keyed dc0/dc1) +
`tests/dc-snap-proxy/run-tests.sh` (NEW, 52 cases) + `tests/HARNESS-MANIFEST` (one name
added) + `docs/audit/stage5-snap-proxy-measurements-20260731.txt` (NEW) +
`docs/CURRENT-STATE.md` (the RULING-2 status paragraph). It implements the ruled
"HTTP(S) forward proxy in the DC utility band + juju snap-https-proxy" as a **dedicated
squid instance ON THE RACK at the D-134 utility `.4:3129`**, CONNECT-only and restricted
to the Canonical-documented snap-store/CDN hosts.

**NOTHING WAS APPLIED.** No package installed, no service started, no `juju config` or
`juju model-config` set, no file written on any rack or node. **The D-135 items 2-3 gap
is still OPEN** and the operator applies this separately. Two declared side effects of
the read-only probing: the `curl` probes traversed MAAS's existing squid and may have
added entries to ITS cache, and `apt-get download squid` wrote a .deb into `/tmp` ON THE
JUMPHOST so the shipped squid config could be READ rather than assumed.

**Why these design choices, each from a MEASUREMENT rather than a preference.**
1. **The client ACL is METAL-ADMIN, not metal-internal.** The three failing applications
   are LXD CONTAINERS. `juju status` prints `mysql-innodb-cluster/0` at `10.12.12.116`,
   but measured INSIDE it: `eth0 10.12.8.122/22`, `eth1 10.12.12.116/22`,
   `ip route get 10.12.8.4 -> src 10.12.8.122`, and NO default route. An ACL built on the
   displayed address would have denied every client the proxy exists for.
2. **Port 3129, and a DEDICATED instance.** MAAS's own squid holds `*:3128` AND `*:8000`
   WILDCARD-bound on BOTH racks. Ubuntu noble's `squid.conf` pins `http_port 3128` and
   `/etc/default/squid` is EMPTY, so the packaged unit cannot be repointed and can only
   FATAL on bind -- `install` disables it and `check` fails if it is active or enabled.
   (`conf.d` was checked too: its include sits before `http_access deny all`, but a
   drop-in cannot UNSET an `http_port`.)
3. **No new D-134 octet.** The utility `.4` is already aliased on both racks, so the
   ruling note's "if it takes its own VM it needs a ruled octet first" never triggers.
4. **This script OWNS its net layer** (`<site>-snap-proxy-net.service` + helper +
   resolved drop-in), never reusing the mirror's or the cache proxy's -- the 2026-07-27
   D-135 lesson, with `ip addr replace`/`ip route replace` so the units coexist.
5. **squid, not tinyproxy.** Every documented snap endpoint is `:443`, so a CONNECT proxy
   is required and **no caching is possible at all** (squid "does not have access to those
   encrypted messages"; the only escape is SslBump, a deliberate MITM). So the choice is
   support posture: squid is noble **main** and defaults to `deny CONNECT !SSL_ports` +
   `deny all`; tinyproxy is universe with six CVEs in "Needs evaluation" and defaults to
   "all ports allowed" for CONNECT. nginx cannot do CONNECT at all.

**LOGGED, NOT ADOPTED, and not buried: MAAS's squid already does this job.** Measured
working to `api.snapcraft.io` from the rack, from the controller VM AND from the failing
container, and machines already consume `http://10.12.8.6:8000` for apt via
`90curtin-aptproxy`. So `juju model-config snap-https-proxy=http://10.12.8.6:8000` would
plausibly work today with no rack build. It was not taken because that config is
MAAS-generated under a per-revision `/var/snap/maas/current/` path, carries NO destination
restriction, and is exactly the hidden-coupling anti-pattern D-135 already has a scar
from. It remains a real operator option at apply time.

**A BUILD-TIME CHOICE THE OPERATOR HAS NOT RULED ON -- flagged, not smuggled.** The ruling
names the mechanism, not the scope. This build restricts CONNECT to the documented store +
CDN hosts; the alternative is allowing any `:443`, which hands the node planes general
HTTPS egress. Widening is one edit to `STORE_DOMAINS`, and the script header says so.

**`check` is built as a GA-R6 gate, not a status printer.** Three outcomes (PASS 0 / FAIL 1
/ REFUSE 3). It diffs all five owned artifacts against their generators (CONTENT, not
existence -- the `dc-mirror.sh` false-green class); asserts the bind is on the LISTEN
address and not wildcard; and makes three behavioral probes from the same source so each
isolates one rule: the store API asserted on `"channel-map"` + `sha3-384` (a 200 is not
evidence), the CDN tunnel asserted as NOT refused (snapd downloads through the same proxy,
so an API-only allowlist would authenticate and then fail), and a **deny-direction probe
that proves the allowlist is in force** -- if it is served, the verdict is "OPEN forward
proxy on the node plane". Every "could not look" REFUSES.

**Mutation-proven: 53 mutations run; 52 KILLED on the first pass, ONE recorded MALFORMED
and redone, final 53/53 killed, 0 survived** (`docs/audit/...` section 11 lists every one).
The malformed one is stated in that order deliberately: the driver printed `[SURVIVED] M18`,
and the cause was my mutation, not a weak assertion -- `true # squid -k parse -f` commented
out the following `then`, broke `bash -n`, and turned 29 cases red without turning T19 red.
A mutation that breaks syntax tests nothing. Redone as `if false; then` + `:`, T19 went red
alone. Each mutation neuters a PREDICATE, not a message (`miss`->`say`, `refuse`->`say`,
`nno`->`nok`, `elif <test>`->`elif false`, or a value swapped for a plausible wrong one);
the script was restored and sha256-verified byte-identical after each; baseline and final
both `52 passed, 0 failed`.
**THREE OF MY OWN CASES WERE FOUND TOO WEAK while doing this and were STRENGTHENED, not
left green:** T16 grepped the whole file for a CDN host the header discusses in prose, so
narrowing the real allowlist left it green; T18 grepped the whole file for a command that
`check`'s own failure message contains, so `install` could stop doing it; T20 matched
`apt-get install` inside the guidance STRING. All three now assert on the scoped value,
the function body, and an invocation position respectively.

**HARNESS-MANIFEST re-recorded deliberately** (`bash scripts/run-tests-all.sh
--record-manifest`), because R15(2) requires the reason be stated: exactly one harness was
ADDED (`dc-snap-proxy`), 93 -> 94, and without re-recording the gauntlet fails
"harness set DRIFTED" rather than running it.

**What this does NOT prove, said plainly.** No proxy has ever been installed, so no snap
has been installed through one -- the harness green is a FIXTURE green. Whether a RUNNING
snapd re-reads the proxy without a restart is UNVERIFIED (snapd writes `/etc/environment`
and reads it via `EnvironmentFile=` at service start; LP#1737332, LP#1791587), so per
client verify BOTH `snap get system proxy` AND a real egress probe. dc1 has no deployed
model, so the container-source finding is measured on dc0 only; `node dc1` is what will
prove it there. And the generated squid config has never been parsed by a squid binary --
`install` runs `squid -k parse` before enabling anything, so that fails loud at apply time.

**>>> THIS ITEM'S CHANGE SET IS SPLIT ACROSS HEAD AND THE WORKING TREE, AND NOT BY ME. <<<**
A CONCURRENT session committed **`e57ad09` ("Teardown: stalled on stopped agents, forced
clear, no nodes stranded")** at 21:34:58 with `git add -A` semantics, and it SWEPT UP two of
this item's files -- `docs/audit/stage5-snap-proxy-measurements-20260731.txt` and the
`docs/CURRENT-STATE.md` paragraph -- into a commit whose message is entirely about the model
teardown and does not mention the snap proxy. **I ran no `git commit` and authored no commit.**
Consequences for review: the diff of the working tree does NOT show the whole item, and the
still-untracked `scripts/dc-snap-proxy.sh` + `tests/dc-snap-proxy/` will be swept into that
session's NEXT commit unless they are dealt with first.

**Revert, stated per half because a single command no longer covers it.**
- ALREADY IN HEAD (e57ad09), so `git checkout --` is a NO-OP on them and would silently leave
  the status claim standing: `docs/audit/stage5-snap-proxy-measurements-20260731.txt` (whole
  file, added by that commit) and the `docs/CURRENT-STATE.md` RULING-2 paragraph (an addition
  inside a file that commit also changed for unrelated teardown reasons). Reverting means a
  FOLLOW-UP commit that `git rm`s the capture and deletes the paragraph -- NOT `git revert
  e57ad09`, which would also undo that session's teardown record.
- STILL WORKING-TREE ONLY: `git checkout -- docs/changelog-20260731-prefer-ipv6-research.md
  tests/HARNESS-MANIFEST` (the manifest returns to 93 names on its own that way);
  `rm -f scripts/dc-snap-proxy.sh`; `rm -rf tests/dc-snap-proxy`.
Nothing outside the repo changes either way -- **there is no live state to roll back**, which
is the one thing this split does not put at risk.
