# Session changelog 2026-07-28 -- the VIP arity gap + R11's ruled gate changes

ONE changelog per session (GA-R2/D1). Under blanket approval this is the review
surface: every item states WHAT, WHY (evidence), and HOW TO REVERT.

**No live mutation in this session.** Everything below is a repo change; the only
live access was READ-ONLY (a NetBox apex poll and a `pre-flight-checks` run). No
logged window was opened because nothing was mutated.

Status authority for everything here is `docs/CURRENT-STATE.md` (GA-R1). This
file is session-scoped scratch and is NOT citable as status or decision
authority.

---

## 1. `provider-bundle-check.py`: VIP arity, family coupling, and the apex-derived v6 bands

**What.** The `vip` validator now accepts a v4 TRIPLE (3 addresses) or a
dual-family SEXTET (6), validates the three v6 legs against the per-DC v6 /64s,
requires the v6 host part to MIRROR the v4 octet textually, and couples
`prefer-ipv6` to the arity in both directions. The v6 bands are READ FROM THE
NETBOX APEX RECORD, not hardcoded. The standalone octet-uniqueness block (old
item 6) folded into the same loop.

**Why.** Measured at `:137`: `if len(parts) != 3` fails EVERY application once R2's
ruled dual-stack lands, and `:149`'s `p.split(".")[-1]` returns the whole address
on a v6 literal, so every v6 leg was trivially "unique" and unchecked. The apex now
holds the dual-family set a renderer will emit (verified live this session: 160
ip-addresses, 78 IPv6, 156 VIP objects), so the only gate that validates overlays
could not validate what is about to be produced.

The `prefer-ipv6` coupling is the substance, not a bonus. `prefer-ipv6` makes
HAProxy bind `:::port` in ADDITION to `*:port`; the measured L3-9 finding is that
the overlay merge order which keeps `prefer-ipv6` while dropping the v6 VIP legs
is the one that EXITS 0 -- the dangerous order is the green one. Both directions
now fail.

**Authority.** The arity gap was carried in `docs/CURRENT-STATE.md` as an open
question, explicitly NOT covered by R11. Operator direction 2026-07-28, exact
utterance: **"Fix the arity gap first, then start the renderer"**, with the scope
fork (arity alone vs arity + R11's three ruled changes) put separately and
answered **"Arity gap + R11's three ruled changes (Recommended)"**. OPS under
GA-R3 -- a script change making an already-RULED end state expressible; doubt
resolves DOWN, no D-number assigned.

**Prior art followed, not reinvented.** The `(role, kind)` apex keying and the
"provider leg lives in the DEDICATED GUA VIP /64, admin/internal in their plane
/64s" rule are copied from `scripts/dc-plane-ipam.sh:102-131` and
`netbox/dc-plane-apex-import.py`. Both live-API and repo-dump shapes are handled
(nested `scope.slug` vs flat `scope_site`) -- writing against one and matching
zero objects on the other is a trap this repo already hit once.

**Refuses rather than passes.** A dual-family vip with an unreadable or absent
apex record exits **2**, never 0: "could not look" is never "nothing there". A
v4-only bundle needs no apex at all, so this introduces no new dependency for
the current tree (harness T25/T26 lock both halves).

**Revert.** `git revert` this commit. The checker returns to triple-only; nothing
live depends on it.

## 2. R11's three ruled gate changes (D-020 AMENDMENT, RULED 2026-07-27)

**What.**
- `provider-bundle-check.py`: `OCTET_LO, OCTET_HI` 50-60 -> **50-99**.
- `scripts/lib-net.sh`: `VIP_OCTET_MAX` 60 -> **99**, `VIP_COUNT_EXPECT` 11 -> **13**.
- `provider-bundle-check.py`: NEW invariant 8 -- an hacluster principal with no
  `vip` is a FAIL.
- `EXPECT_PUBLIC_VIP` deliberately **STAYS 11**.

**Why.** R11 ruled vault `.61` and designate `.62` as full dual-family triples and
ruled the gate hardening **IN SCOPE, not deferred**. `cluster_count` is asserted
NOWHERE in `scripts/` or `tests/`, so a 3->1 rewrite of all 20 values produces a
byte-identical PASS -- decorative HA was found by audit rather than by gate.

`EXPECT_PUBLIC_VIP` stays 11 because -- measured -- NEITHER vault nor designate
carries a `public` binding, so they do not join that count. Bumping both constants
together breaks the gate. The two band constants are SEPARATELY NAMED in two files;
this is a two-file change by construction, and both sites carry a comment saying so.

**MEASURED CONSEQUENCE, stated precisely.** Two sub-checks flip PASS -> FAIL:
- `provider-bundle-check` on the base bundle now FAILS with
  `hacluster relation but no vip: designate`.
- `pre-flight-checks` CHECK 1 now reports `VIP validation OK=11 bad=0 (want OK=13)`.

**`preflight.sh` was ALREADY exit 1 before this change** (octavia-pki overlay
absent, MAAS unreachable from the jumphost, P5's 7 credential findings). Measured
after: still exit 1, now `3 fatal, 2 warning`. This did NOT flip preflight from
pass to fail and **no deploy path that was open is closed by it** -- the same
precedent this document records for P5. The reds are the ruled gate reporting real
work owed: R11's `.61`/`.62` VIPs are RULED-BUT-NOT-BUILT, and per R6 they land
BEFORE the HA overlay.

**Revert.** Revert this commit; the constants and the invariant-8 block are
contiguous and independent of item 1.

## 3. Harness: 15 -> 30 cases, with three cases RE-POINTED rather than deleted

**What.** `tests/provider-bundle-check/run-tests.sh` gains 15 cases. The fixture
base splits: `pristine.yaml` is the repo bundle untouched; `good.yaml` is that
bundle PLUS designate's ruled `.62` VIP.

**Why the split.** Invariant 8 legitimately fails the real repo bundle. Left alone,
every rc=0 case (T1/T9/T14) would have been asserting "fails for the reason we
already know" instead of its own invariant. Per the standing rule -- *when
remediating a finding turns the harness red, REPLACE the assertion with the new
invariant and say so, never delete it to go green* -- the real tree's state is
kept honest by a NEW case (T16) that asserts `pristine.yaml` DOES trip it.
**When `.62` lands in `bundle.yaml`, T16 must be re-pointed, not deleted.**

**Proven able to fail AND to pass** -- the complement this project recorded on
2026-07-27 after building three gates whose every live run failed:
- T16 the pristine bundle FAILS on designate; T18 the same check goes GREEN once
  the ruled VIP lands.
- T17 vault FAILS under `dc-ha-scaleup.yaml` -- R6's "VIPs before the overlay"
  ordering, made executable.
- T19 a valid dual-family VIP PASSES and is counted as dual; T20/T21 both
  directions of the `prefer-ipv6` coupling FAIL; T22 a v6 leg in the node `/64`
  instead of the VIP `/64` FAILS; T23 the `::32` hex trap FAILS; T24 a 2-address
  vip FAILS; T25 an unreadable apex REFUSES at rc=2; T26 a v4-only bundle needs
  no apex; T27/T28 the band is widened to 99 but still bounded.

T29/T30 exercise the **dc1** arm of the apex lookup, which T19-T23 never reached
(they all run at the default `--dc vr1-dc0`): T29 resolves dc1's `f03:11::/64` /
`:320::/64` / `:321::/64` and passes, T30 proves a dc0 v6 leg under `--dc vr1-dc1`
FAILS, so a cross-DC copy-paste of a rendered overlay cannot pass. The renderer's
first real output is per-DC overlays, and a gate observed in only one DC mode is
the same untrustworthiness T16/T18 exist to rule out.

T8's assertion string moved `outside 50-60` -> `outside 50-99` (the same
re-pointing rule; the case still guards the band, at its ruled bounds).

**Evidence.** Harness **30/30** (was 15). Gauntlet **ALL GREEN (84 harnesses) on
vcloud** -- the host is named because the gauntlet is measured HOST-DEPENDENT
(2/81 fail on voffice1 on an identical commit). `repo-lint` 0 fail / 1 standing
warn / 605 files scanned. No harness was added or renamed, so
`tests/HARNESS-MANIFEST` is unchanged and needed no re-record.

**Revert.** Revert this commit; the harness returns to 15 cases.

---

## Findings LOGGED, NOT ACTIONED (hard rule 1)

1. **`pre-flight-checks.sh` CHECK 1 is triple-shaped.** Its awk parse assumes three
   columns, so it will mis-read a dual-family sextet when those artifacts land. Same
   class as the arity gap just fixed, one gate over. Not touched -- it interlocks
   with BLOCKER-1 (the ruling-3 VIP extraction makes CHECK 1 see 0 VIPs).
2. **`cluster_count` coherence is still unasserted.** Invariant 8 catches an
   hacluster principal with NO vip, which is what R11 ruled. It does NOT check that
   `cluster_count` matches the principal's `num_units` -- a 3->1 rewrite of all 20
   values still produces a PASS. Out of ruled scope; worth its own exchange.
3. **NetBox instance/target drift (raised by the operator this session).** Four
   `netbox/*.py` tools carry in-file usage examples pointing at
   `https://netbox.baldurkeep.com` -- the FROZEN v1 reference. Two of them
   (`ipv6-mark-reserved.py`, `ipv4-prefixes-import.py`) have **NO `SANDBOX_HOSTS`
   write guard at all** while carrying write paths (`update_status`,
   `create_or_report_prefix`, `create_or_report_iprange`). The guarded tools also
   expose a deliberate `--yes-write-upstream` escape hatch, so "writes only reach
   10.10.1.10" is a convention with an override, not a hard invariant.
4. **Credential env-file naming is inverted.** `~/vr1-office1-creds/vr1-netbox.env`
   points at `netbox.baldurkeep.com` (the read-only v1 reference) while
   `vr1-netbox-sandbox.env` points at the LIVE VR1 apex `http://10.10.1.10:8000`.
   The filename says the opposite of what each is.
5. **The NetBox `?site=` API filter silently does not filter.** Measured this
   session: `/api/ipam/ip-addresses/?site=vr1-dc0` and `?site=vr1-dc1` each returned
   the full 160. A per-site apex count taken that way would be wrong and look right.

---

# 2026-07-29 -- render-pipeline STEP 4: the renderer

Same branch, continued. Operator direction: **"You can order the steps in whatever
order you want. Deploy agents to help if needed."**

## 4. Ordering decision, and why it INVERTED after measurement

I first proposed doing ruling-3's VIP extraction before the renderer, then reversed
it once three read-only agents reported. Recording the reversal rather than the
conclusion, because the reason is the reusable part:

- The byte-for-byte reproduction WINDOW IS STILL OPEN and closes on its own.
  `overlays/vr1-dc1-vips.yaml` is still byte-identical to
  `tests/render-baseline/fixtures/vr1-dc1-vips.v4-11app.yaml` (`3f93ecb3...`)
  -- verified this session. The reconciliation (11->13 apps, dual-family) destroys it.
- The extraction's blast radius is FAR larger than ruling 3's "2 commits". Measured:
  it breaks a DEPLOY GATE (`runbooks/phase-01-bundle-deploy.md:170-176`, where
  `juju deploy` runs only on `11/11/0`), the Octavia SAN derivation (`:371`),
  preflight in two places, and 8 of 30 harness cases.
- So the extraction is better done BY the renderer, once the renderer is proven --
  which makes dc0's overlay a validated output rather than a hand edit.

## 5. `scripts/render-dc-overlays.py` -- two stages, split where F2 froze the interface

**What.** `derive` (apex + `lib-net.sh` -> a per-DC VALUES file) and `render`
(VALUES -> overlay TEXT). Plus `check`, which renders and byte-compares.

**Why split.** Forward item F2 froze the interface: *"the renderer is a
deterministic command-line tool whose inputs are the per-DC values file and whose
outputs are committed, reviewable artifacts. Nothing about automating it should
require changing the renderer."* `render` is therefore PURE -- no apex, no network,
no clock -- so it is offline-testable and a future CI job only re-runs `derive`.

**Why a text emitter, not `yaml.dump`.** The target is byte-compared. A round-tripper
destroys the comment header, normalises `vip: "a b c"` to a plain scalar, and imposes
its own key order -- all three are load-bearing. PyYAML reads the values file and
never writes an overlay.

**Two things that are INPUT, not output, and would otherwise be invented:**
- The 12-line comment header cites one-off measurements and a design-doc fork; none
  of it is computable from the apex. `derive --from-overlay` lifts it VERBATIM from
  the artifact being reproduced, so a reproduction run is honest about which bytes
  the tool actually generates.
- Emission order is ASCENDING OCTET, not alphabetical (keystone .50 before barbican
  .51). Sorting by name reproduces nothing.

**The ruled app->octet map is READ, not restated** -- parsed with `ast` out of
`netbox/dc-plane-apex-import.py`'s `APP_OCTET`, the tool that wrote those values into
the apex. A second hand-maintained copy is a second thing to drift.

**Revert.** Delete `scripts/render-dc-overlays.py`, `tests/render-dc-overlays/`,
`render/values/`, and revert the `render-baseline` T6 hunk + the manifest line.
Nothing else consumes any of it; no committed artifact was regenerated by it yet.

## 6. RESULT: byte-for-byte reproduction achieved, and PROVEN ABLE TO FAIL

`render` output hashes to
`3f93ecb3205cea8b303f5b2cbbbde343a2762156575277ca84684369da8bf896` -- identical to
both the committed overlay (1634 bytes) and the frozen fixture.

A byte-compare only ever observed PASSING is as untrustworthy as a gate only ever
observed failing, so three faults were seeded and all three fail it: a wrong octet, a
dropped header, and dc0 prefixes against the dc1 artifact. `render-baseline`'s new T6
was separately proven by making the renderer sort alphabetically -- hash changed,
case went red.

## 7. THE ARITY WORK PAID FOR ITSELF WITHIN THE HOUR -- a real renderer bug, caught by the gate

The renderer's FIRST dual-family output was malformed. The values file stores each
v6 `/64` base with trailing colons stripped (`2602:f3e2:f03:11`), and the emitter
joined with a SINGLE colon -- yielding `2602:f3e2:f03:11:50`. That is not a valid
address, and it still looks like one.

`provider-bundle-check.py` -- fixed yesterday for exactly this class -- rejected all
13 applications with `bad vip ip`. Fixed to `::%d` and locked by harness cases
T13/T13b, which assert both that the correct form is present AND that no
single-colon leg is emitted. **This is the acceptance criterion the whole arity fix
existed to provide, working on its first real use.**

## 8. End-to-end chain now demonstrated

apex -> values -> renderer -> overlay -> gate. The dual-family dc1 render with R11's
ruled-but-unbuilt apps included produces **13 clustered VIPs, all dual-family**, and
`provider-bundle-check --dc vr1-dc1` reports **PASS** including the new hacluster
invariant (12 principals, all carrying a VIP).

**Evidence.** New harness `tests/render-dc-overlays` **17/17**; `render-baseline`
**10/10** (was 9, T6 added -- ADDED per its README, existing assertions NOT repointed).
Gauntlet **ALL GREEN (85 harnesses) on vcloud** (was 84); `tests/HARNESS-MANIFEST`
re-recorded deliberately after the drift gate caught the addition -- which is the gate
working. repo-lint 0 fail / 608 files scanned.

---

## Findings from the three read-only agents -- LOGGED, NOT ACTIONED (hard rule 1)

6. **F3's MAC sourcing claim is FALSIFIED.** `docs/dc-dc-deployment-workflow.md:361`
   says `ovn-chassis` MACs "stay sourced from `scripts/lib-hosts.sh`". Measured:
   lib-hosts carries only `HOST_BOOT_MAC` (the PXE NIC, `...:NN:01`); `ovn-chassis`
   needs the provider-public NIC (`...:NN:02`), which is **not in lib-hosts at all**.
   The real source is `opentofu/vr1-dcN-substrate/main.tf`, `macs[1]` of each compute
   node -- and it exists for BOTH DCs. This also narrows CURRENT-STATE's claim that
   dc0's MACs "cannot be derived": not from a PATTERN, but they are pinned literals
   in tofu and are perfectly sourceable. F3's conclusion (MACs are renderer output,
   not apex input) still stands; only its sourcing sentence is wrong.
7. **`scripts/dc-plane-ipam.sh:120` has a latent scope-matcher bug.** It matches
   `scope.name` or `scope_site` but NOT `scope.slug`, so it would match ZERO prefixes
   against a live-shaped record. Latent only because it defaults to a repo dump. The
   other two tools use the three-way matcher; this one is the odd one out.
8. **`pre-flight-checks.sh` CHECK 1 is triple-only AND dc0-only.** It hardcodes
   `if(n!=3)` (`:86`), reads `bundle.yaml` as RAW TEXT (never a merged input), and
   never calls `lib_net_select_dc` -- and `lib-net.sh` deliberately UNSETS
   `VIP_PREFIX_*` for dc1, so CHECK 1 cannot evaluate dc1 at all. The renderer's
   dual-family output will pass `provider-bundle-check` and FAIL preflight P4.
9. **CURRENT-STATE is stale on dc1's FIP pool** -- an early paragraph still says it
   "needs a ruling"; a later entry records it RULED and applied, and `lib-net.sh`
   carries the values. Same in-document supersession class this doc keeps finding.
10. **`render-baseline` T3 compares COUNTS, not SETS.** A deleted pinned fixture plus
    an added unpinned one keeps the count equal; a `#` comment line in `SHA256SUMS`
    inflates the pin count and masks one unpinned file; and `-maxdepth 1` cannot see
    a subdirectory. It catches the fault it was seeded against; a set-difference
    would close all three.
11. **The B1/B5 comment tokens are defined in `bundle.yaml`'s own header** (`:11-13`,
    `:19-24`). Ten of the eleven inline VIP comments are the bare token `# B1`.
    Migrating them into an overlay STRANDS them from their definitions --
    `appendix-A:652-661` records the repo's rule for exactly this identifier class.
    This must be handled when the extraction lands, not discovered then.

---

# 2026-07-29 -- RULING 3, COMMIT 1: the neutral VIP MOVE (+ blast-radius repair)

Operator: **"Approved, continue"**. This is commit 1 of the two ruling 3 (2026-07-25)
prescribes. **The dual-stack ADD is commit 2 and is deliberately NOT in here.**

## 9. `bundle.yaml` is now VIP-FREE; `overlays/vr1-dc0-vips.yaml` is GENERATED

**What.** All 11 inline `vip:` lines removed from `bundle.yaml`; `placement`'s
then-empty `options:` block dropped rather than left parsing as `null`. The values
moved to `overlays/vr1-dc0-vips.yaml`, **rendered by `scripts/render-dc-overlays.py`**
from `render/values/vr1-dc0-vips.yaml` -- the renderer's first real output.

**The 11 inline comments were carried, not dropped.** Ten are the bare token `# B1`
and three carry real operational content (keystone's catalog-endpoint note,
ceph-radosgw's un-defer note, openstack-dashboard's ALLOWED_HOSTS warning). They were
extracted VERBATIM from `bundle.yaml` by a parser, not retyped, and the renderer grew
optional per-app comment support to carry them. An app with no comment emits none, so
the dc1 artifact still reproduces byte-for-byte (re-verified: 1634 bytes, unchanged).

**The `B1`/`B5` tokens stay DEFINED in `bundle.yaml`'s header**, where they always
were; the overlay header names that explicitly so the references do not dangle
(`appendix-A:652-661` is the repo's rule for this identifier class).

**PROVEN NEUTRAL by the ruling's own named check.** `provider-bundle-check` on the
pre-extraction bundle and on `bundle.yaml --overlay overlays/vr1-dc0-vips.yaml --dc
vr1-dc0` produce **line-for-line identical output**, including the same known
`hacluster relation but no vip: designate` failure. Separately, a structural diff
confirmed only `vip` changed: applications 56 -> 56, relations 108 -> 108, every
non-VIP option and every non-option key byte-identical.

**Revert.** `git revert`; the values file and overlay are additive and the bundle edit
is a pure line removal.

## 10. Blast-radius repair -- the part that made this bigger than "2 commits"

Every item below would have been BROKEN by the move. All were found by the read-only
agent sweep BEFORE the edit, not after.

- **`scripts/preflight.sh` P2** validated the BARE base, which post-move reports 11
  phantom "no vip" failures. Now validates the MERGED dc0 input; the overlay is passed
  only when present so a tree without it still runs.
- **`scripts/pre-flight-checks.sh` CHECK 1** read `bundle.yaml` as raw text and would
  have seen ZERO VIPs -> hard-fail. Now reads the merged source. **This is BLOCKER-1,
  fixed the ruled way (point it at the merged bundle) and explicitly NOT by bumping the
  expected count.** A measured trap on the way: the file sets `IFS=$'\n\t'`, so a
  space-joined `"$VIP_SRC"` reached `grep` as ONE filename, `grep` exited 2, `pipefail`
  propagated and the gate **died silently mid-check**. Fixed with a bash ARRAY.
- **`runbooks/phase-01-bundle-deploy.md`** -- the real hazard. Its RUN block gates
  `juju deploy` on `TOT/HI/LO == 11/11/0`; post-move that reads `0/0/0` and **aborts
  the deploy**. Both guard blocks now span base + overlay. A second bug had to be fixed
  in the same edit: `grep -c` over TWO files prints one count PER FILE, so the bare
  `$( )` would have captured a two-line string and every numeric test would have failed
  anyway -- the counts are now awk-summed. Verified end to end: the gate reads
  **11/11/0 -> DEPLOY**. The deploy line itself now carries
  `--overlay overlays/vr1-dc0-vips.yaml`.
- **`phase-01`'s Octavia SAN derivation** (DOCFIX-067) read
  `bundle['applications']['octavia']['options']['vip']`, which post-move raises
  KeyError -- swallowed by `|| true` into an EMPTY VIP, then a "implausible VIP" abort.
  Now reads the overlay. Verified: derives `10.12.4.57`.
- **`runbooks/phase-00-teardown-maas-reset.md:233`** bare checker call -> merged input.
- **`runbooks/dc-dc-phase4-juju-bundle-per-dc.md`** said "**DC1 needs NO VIP/CIDR
  changes at all**" and "deploy the EXISTING `bundle.yaml` with **NO VIP edits**".
  Followed verbatim post-move that deploys a VIP-free DC1. Superseded in place (the
  D-101 plane reasoning is kept, because it is still correct); the DC2 instruction to
  "edit `bundle.yaml`'s VIP lines" now points at rendering a new overlay instead.
- **`runbooks/appendix-A-troubleshooting.md` L3** told an operator to grep the bundle
  for the metal-side VIP token -- which now returns zero, and the block's `|| true`
  makes zero look like a clean result. Note added.
- **`tests/render-baseline/README.md`** records a capture-time `bundle.yaml` sha256
  that no longer re-verifies. Marked HISTORY with what changed and why, rather than
  re-pinned -- the fixture hashes themselves are untouched.

## 11. Harness: 30 -> 31, with the fixture BASE re-pointed

`tests/provider-bundle-check` now builds its fixture base by MERGING
`overlays/vr1-dc0-vips.yaml` into `bundle.yaml`, because that pair is the deploy input.
T16's label changed from "pristine repo bundle" to "the real dc0 deploy input" -- same
assertion, honest name. **NEW T31 asserts ruling 3's new invariant directly: the BARE
base fails for every clustered principal**, so "deployed without its overlay" is a
tested state rather than an assumption. 31/31.

**Evidence.** Gauntlet **ALL GREEN (85) on vcloud**; repo-lint 0 fail / 610 files;
`render-dc-overlays` 17/17; `render-baseline` 10/10 (dc1 reproduction still byte-exact
after the comment-support change); `preflight` 16/16.

**Still red, unchanged and expected:** `pre-flight-checks` CHECK 1 reports `OK=11 want
13` and `provider-bundle-check` reports designate -- both are R11's `.61`/`.62` VIPs
being RULED-BUT-NOT-BUILT, which is commit 2's job, not this one's.

---

# 2026-07-29 -- RULING 3, COMMIT 2: the dual-stack ADD (R2 + R11 built)

Operator: "Continue as autonomously as possible. Deploy agents as needed." Two read-only
agents mapped the blast radius before any edit -- the same move that caught the deploy-gate
break in commit 1, and it caught another one here.

## 12. Both per-DC VIP overlays are now DUAL-FAMILY, 13 apps

**What.** `overlays/vr1-dc0-vips.yaml` and `overlays/vr1-dc1-vips.yaml` re-rendered:
each app carries `prefer-ipv6: true` and a six-address `vip` (3 v4 legs + 3 v6), and
**vault `.61` and designate `.62` are now BUILT** (R11, a D-020 amendment). v6 prefixes
come from the apex; the host part mirrors the v4 octet textually.

**RESULT, measured both DCs:** `13 clustered VIP(s) ... (13 dual-family)` and
`12 hacluster principal(s) all carry a VIP` -> **PASS**. With `dc-ha-scaleup.yaml`
stacked on top: **13 principals, PASS** -- so **R6's ruled ordering (VIPs before the HA
overlay) is now satisfied and executable**, not just stated.

**Two decisions that were RULED-BUT-NEVER-BUILT are now artifacts:** R11's vault/designate
VIPs, and R2's dual-stack for the API VIP layer.

**Revert.** Re-render both overlays from the values files without `--dual-family
--include-unbuilt`, and revert the gate/harness hunks below.

## 13. THE L3-9 COLLISION IS RESOLVED -- and the recorded description of it was half stale

`overlays/dc-dc-ipv6-family-matrix.yaml` carried a `vip` + `prefer-ipv6` pair for ten API
charms, every value an unrendered `{{TOKEN}}`. Those duplicated what the per-DC overlays
now own, and take-last merge between them is the collision. **Narrowed to `ceph-mon` only.**

The vip and prefer-ipv6 keys were removed **TOGETHER**. Removing the vips alone would have
recreated the exact defect on purpose -- which is why this is worth stating.

**MEASURED, and it corrects this repo's own record:** CURRENT-STATE described L3-9 as "the
dangerous merge order is the one that PASSES". **That has not been true since 2026-07-28.**
Invariant 9 (added by the arity commit, precisely because of L3-9) makes vips-last fail on
the coupling, while matrix-last already failed on the placeholders -- so NEITHER order was
green. The gate closed the SILENT half; this commit closes the collision itself. **Both
merge orders now PASS, i.e. the result is order-independent**, which is the real fix.

**What survives, because it exists in NO other file:** ceph-mon's `ceph-public-network` /
`ceph-cluster-network` ULA-only config and its privacy-extension caveat. ceph-mon is in
neither VIP overlay and never will be -- it has no VIP. Also removed: the two Launchpad
citations arguing octavia lb-mgmt IPv6 was an open risk. Both were read in full and BOTH
failed (one a different installer's bug, Fix Released 2021; one not an IPv6 defect at all),
and **R8 RULED the question closed on 2026-07-27** -- leaving them standing in a live
artifact was a refuted citation presented as current.

## 14. Gate repairs the agents caught BEFORE the edit

- **`runbooks/phase-01-bundle-deploy.md` -- the deploy guard would have ABORTED AGAIN.**
  Its `HI` regex is `10\.12\.4\.(5[0-9]|60)`, which **excludes .61 and .62**, so the counts
  would have read `13/11/0` against a required `11/11/0` and `juju deploy` would not have
  run. Widened to `(5[0-9]|6[0-2])` and the expected counts moved to `13/13/0`. Verified end
  to end: **13/13/0 -> DEPLOY**. This is the second time this one guard has nearly blocked
  the deploy across two commits; it is worth knowing it is that brittle.
- **`scripts/pre-flight-checks.sh` CHECK 1** hardcoded `if(n!=3)` and would have printed
  `MALFORMED(not 3 IPs)` for all 13 and failed `OK=0 bad=13`. Now accepts a v4 triple OR an
  R2 sextet, and for a sextet asserts the last three legs are actually v6 literals. Deep v6
  prefix/mirror checking is deliberately NOT duplicated here -- `provider-bundle-check` does
  it against the apex. **Proven able to fail:** a 4-address vip -> `MALFORMED(not 3 or 6)`,
  a v4 in a v6 slot -> `NOT-IPv6 in v6 leg 1`, and a v4 triple still passes.

## 15. Harness: 31 -> 33, and 18 for the renderer, with everything RE-POINTED not deleted

- **T16 and T17 both asserted R11's gaps were OPEN.** R11 is now built, so both were
  inverted to the surviving invariant (the deploy input is CLEAN; R6's ordering is
  satisfied at 13 principals) and **paired with NEW T16b/T17b that remove a VIP and demand
  the check still fires** -- otherwise inverting them would have removed the teeth.
- **T20/T21, the coupling cases, had become NO-OPS** (the overlay already sets what they
  were setting) -- they would have gone GREEN while testing nothing, which an agent caught.
  Inverted: T20 now truncates the vip while keeping `prefer-ipv6`, T21 drops `prefer-ipv6`
  while keeping six legs.
- A **v4-only twin fixture** was added so the pre-dual-stack cases (T5/T8/T26/T28) still
  exercise band, octet-uniqueness and apex-independence instead of tripping the coupling
  check first and never reaching their own assertion.
- The renderer harness now proves reproduction on **BOTH** shapes: T3 against the FROZEN v4
  fixture (immutable, a stable target) and **new T3b against the LIVE dual-family overlay**
  (whose own header mandates it stay renderer-reproducible).

**Evidence.** `provider-bundle-check` **33/33**, `render-dc-overlays` **18/18**,
`render-baseline` 10/10, `preflight` 16/16. Gauntlet **ALL GREEN (85) on vcloud**;
repo-lint 0 fail / 611 files.

**PREFLIGHT: BOTH STANDING REDS CLEARED.** It was `3 fatal` with P2 and CHECK 1 among them;
it is now **`2 fatal`** -- the deliberately-absent octavia-pki overlay and MAAS unreachable
from vcloud (a missing-binary artifact that R10 measured clears on voffice1). Nothing this
work introduced is red.

---

## Findings LOGGED, NOT ACTIONED

12. **`ceph-mon`'s two CIDRs are gated by NOTHING.** `provider-bundle-check` skips any app
    without a `vip`, so `{{DC_STORAGE_ULA_CIDR}}` / `{{DC_REPLICATION_ULA_CIDR}}` are
    validated by no gate at all -- and they are now the ONLY content of that overlay. A
    prominent header warning was added; rejecting `{{...}}` in any option value would be the
    real fix and is a gate change out of this commit's scope.
13. **octavia's dual-family API VIP is an INFERENCE, flagged not hidden.** R2 ruled
    dual-stack and R8 ruled octavia's lb-mgmt network, but nothing explicitly rules that
    octavia's own API VIP goes dual-family. It is in both VIP overlays, so a uniform
    re-render gave it `prefer-ipv6: true`. That follows R2 applied evenly and matches the
    old matrix file's own reasoning ("no different from the other charms"), but it is not a
    quoted ruling -- **worth one operator confirmation.**
14. **Nothing checks vip-WITHOUT-hacluster.** Invariant 8 catches a clustered principal with
    no VIP; the inverse is unguarded. Relevant because `vault-hacluster` exists only in
    `dc-ha-scaleup.yaml`, so a base+dc0-vips deploy gives vault a VIP and no HA relation.
15. `runbooks/dc-dc-phase4-juju-bundle-per-dc.md` Step 6 still instructs passing the matrix
    overlay alongside **two overlays that do not exist on disk** and no VIP overlay. Wrong
    before this change; now wrong differently.

---

# 2026-07-29 -- the three Octavia items: R8a BUILT, two stale surfaces, R7 generator

Operator: "continue on with the next three", after an upstream/vendor research pass on
whether Octavia can run full IPv6 (it can; nothing inside Octavia requires IPv4).

## 16. R8a BUILT -- and a decision record that claimed it already shipped, corrected

**What.** `scripts/phase-05-octavia-verify.sh` gains the LP #2018998 o-hm0 / lb-mgmt-net
MTU comparison R8a ruled on 2026-07-27. It resolves the network **BY TAG** (never by the
name `lb-mgmt-net`), reads o-hm0's MTU from `ip -o link`, FAILS on mismatch in **either**
direction, and **REFUSES** rather than passing when either value cannot be read.

**Why it matters.** A jumbo `lb-mgmt-net` beside a 1500 `o-hm0` silently drops health
heartbeats >1500B and surfaces as SPURIOUS load-balancer failovers -- a symptom that reads
as a Ceph, network or amphora fault long before anyone suspects MTU. Fix Released for our
lineage, but recurring 2025-12-31 on octavia 14.0.0 / 2024.1 stable, the exact channel
`bundle.yaml` pins. R3's jumbo ruling is this bug's precondition.

**THE RECORD WAS FALSE, and that is the more important half.** `design-decisions.md`
carried "Delivery: the assertion ships in `scripts/phase-05-octavia-verify.sh` with its
harness green..." -- written in the present tense on the day of the ruling and reading as
a statement of fact. Measured: the file contained ZERO MTU references and its last commit
predated the ruling by a month. This is a sharper form of RULED-IS-NOT-BUILT: the decision
doc was the false witness rather than merely silent. Corrected in place with what was
measured and what has now actually been built.

**Revert.** Revert this commit; the MTU block is contiguous and the harness cases are
additive.

## 17. TWO OWNED ERRORS in how I did it

**(a) I concluded "NO octavia harness exists" from a name grep** (`ls tests/ | grep -i
octavia`) and wrote a NEW harness. The script's harness has existed all along as
`tests/phase-05/` -- it simply is not named after the script. Two harnesses for one script
is itself a drift hazard, so the duplicate was DELETED and the R8a cases folded into the
real one. Manifest went 85 -> 86 -> 85 across the mistake and its repair.

**(b) My first draft of the MTU block ABORTED THE SCRIPT.** `LBNET_MTU="$([ -n "$LBNET" ]
&& openstack ... )"` under `set -euo pipefail` WITH `inherit_errexit`: when the network is
unresolvable the substitution exits non-zero and kills the run mid-check instead of
reaching the refusal branch. **The pre-existing `tests/phase-05` harness caught it** -- the
harness I had just declared nonexistent. Fixed with explicit `|| true` captures and an
`if` block; a case now locks it ("REFUSES, does not abort").

Same defect class as the `IFS=$'\n\t'` word-splitting trap in commit 1: this repo's gate
scripts run under strict bash, and a "safe-looking" command substitution is a silent exit.

## 18. Two live surfaces that still contradicted R8

`runbooks/dc-dc-phase4-juju-bundle-per-dc.md:304-311` and
`docs/dc-dc-deployment-workflow.md:775-781` both still described Octavia's lb-mgmt IPv6
support as "a real, open risk, not resolved" and recommended **keeping it v4-only as a
logged D-101 exception** -- citing the two Launchpad bugs that were read in full and BOTH
failed. R8 ruled the opposite on 2026-07-27. We purged those citations from the overlay
and never from these two, so a runbook reader would have re-litigated a closed ruling in
the wrong direction. Both superseded in place, naming R8, the bug statuses, and the
upstream finding that the charm has **no IPv4 code path** for that subnet at all.

## 19. R7: the Octavia PKI generator is no longer dc0-frozen

**What.** `runbooks/phase-01-bundle-deploy.md` Step 1.0-GEN gains a `$DC` selector, and
the two things R7 ruled are derived from it: the baked `/CN=VR0 DC0 ...` CA subjects
become `${DC_LABEL}`, and the VIP gate's `^10\.12\.4\.` regex now derives this DC's
provider prefix **from the same overlay it read the VIP from**.

**Why it was broken.** dc1's octavia VIP is `10.12.64.57`, so the gate hard-ABORTED, and
`phase-01:144-145` hard-ABORTs the deploy without the overlay -- i.e. the generator could
not produce a dc1 artifact at all. R7's own caveat was explicit: the gate "must read the
MERGED deploy input, not `bundle.yaml`, or it breaks again the moment ruling-3's VIP
extraction and R11's `.61`/`.62` land." Both have since landed.

**Verified both ways:** dc0 `10.12.4.57` vs prefix `10.12.4` -> OK; dc1 `10.12.64.57` vs
prefix `10.12.64` -> OK; and the negative control (dc1's VIP against the old dc0-frozen
regex) still ABORTs, which is exactly what used to happen on every dc1 run.

**Deliberately NOT changed, both logged in place:** the controller cert's CN and DNS SANs
still carry `dc0.vr0` -- R7 scoped only the CA subject and the VIP gate, and those DNS
names are inert while `os-public-hostname` is unset on every API charm (IP-only endpoints,
B5). And `.split()[0]` still takes only the PROVIDER v4 leg, so this cert gains no IPv6 IP
SAN even though the VIP is dual-family. That is a gap, not a break -- this is the amphora
control-plane trust domain, separate from API TLS, which Vault issues with IPv6 SANs under
D-109. **Adding v6 IP SANs needs its own ruling.**

**Evidence.** `tests/phase-05` **14 cases ALL PASS** (was 9); gauntlet **ALL GREEN (85) on
vcloud**; repo-lint 0 fail / 611 files.

## 20. IPv6 IP SANs on the Octavia controller cert -- RULED and BUILT

**Ruling (GA-R5).** Question as presented: R7's work left the SAN derived by `.split()[0]`
-- the provider v4 leg only -- while ruling 3 commit 2 had made every API VIP dual-family,
so the amphora control-plane cert carried no IPv6 IP SAN. Flagged as a gap needing its own
ruling. **Operator answer, exact utterance: "Yes, add the v6 IP sans."** Recorded as a
D-109 RULING NOTE (2026-07-29); OPS under GA-R3, no new D-number.

**What.** The generator emits `IP.1` = this DC's provider v4 leg and `IP.2` = its provider
v6 leg, both derived from the same per-DC overlay under R7's `$DC` selector.

**Verified:** dc0 -> `10.12.4.57` + `2602:f3e2:f02:11::57`; dc1 -> `10.12.64.57` +
`2602:f3e2:f03:11::57`; a v4-only control emits NO v6 SAN, so the generator stays correct
before the dual-stack ADD. Admin and internal legs stay excluded, as they always were.

**Two repo guards fired on the way and both were right.** `repo-lint` L5 rejected a
heading that LED with `D-109` (it reads as a second definition of the decision) -- retitled
to the repo's own note form, `RULING NOTE <date> -- D-109:`. That is the third time this
trap has bitten in this branch's history. And L10 rejected the commit for touching a
design-decisions Status line without CURRENT-STATE in the same change (GA-R1 C1).

**OBSERVATION LOGGED, NOT ACTED ON:** amphorae reach the controller on `o-hm0`'s
charm-generated `fc00::/64` ULA (`controller_ip_port_list`), not on any VIP leg, so neither
SAN matches THAT path. Measured upstream, the controller verifies the amphora by UUID; the
reverse direction was not established from source. This ruling makes the SAN set consistent
with the VIP it has always been derived from -- whether that is the right derivation is a
separate question, to settle by inspection at the Octavia step.

---

# 2026-07-29 -- D-136 STEP 6: the chain audit, and what it found

Three read-only lenses over `data source -> rendering tool -> deployment mechanism`
(D-136 option (D)'s ruled final step). Lens 3 was adversarial and was told to make the
chain produce a wrong result that still comes out GREEN. **It succeeded three times.**

## 21. All three lenses converged: dc1 was gated, dc0 was not

`tests/render-dc-overlays` byte-compared the **dc1** overlay only. dc0's -- the artifact
`preflight.sh` P2 actually feeds the deploy gate -- was reproducible by nobody, and
`render/values/` was referenced by **ZERO scripts and ZERO harnesses**. Executed attacks
that passed every gate:

- **Hand-transposing two apps' octets in the dc0 overlay** (glance .53 <-> magnum .54,
  all six legs). Internally consistent, in-band, right /22 and /64, unique -> `PASS`.
- **One word: `family: dual` -> `family: v4`** in dc0's values file. That drops
  `prefer-ipv6` and the v6 legs TOGETHER, so invariant 9's coupling check is satisfied,
  and `vip_dual` is printed but never asserted -> dc0 silently reverts to IPv4-only
  against R2's ruling, `PASS`.

**Root cause: R9's render-drift check was RULED (D-119 amendment, 2026-07-27) and never
built.** `lib-net.sh:171-177` still carried the future-tense "becomes GENERATED".

## 22. `tests/render-drift/` -- R9's gate, built

Iterates `render/values/*.yaml`, reads each file's own `output:` field (inert until now),
renders, and byte-compares against the committed overlay. **A new per-DC values file is
gated the moment it is committed -- no harness edit.** Carries a non-zero floor (a drift
gate over zero files is not a gate) and a PROOF-OF-TEETH case that mutates a copy and
requires the check to go red. 4/4; both DCs byte-match today.

## 23. Three refusals added where the chain took the last writer

- **Duplicate `(role, kind)` in the apex.** Both readers were last-writer-wins. Measured:
  appending ONE spare prefix (`2602:f3e2:f02:99::/64`, routine IPAM work) silently re-homed
  **every dc0 VIP into a different /64** and the whole chain went green -- because the gate
  only checks overlay-vs-apex AGREEMENT, and both sides read the same wrong value. Which
  prefix won depended on JSON array order, which is not a contract. Both
  `render-dc-overlays.py` and `provider-bundle-check.py` now REFUSE (rc=2) and name both
  candidates.
- **`APP_OCTET` duplicates / out-of-band octets.** A duplicate app name emitted two YAML
  blocks for one app; `yaml.safe_load` kept the last and the gate passed, so one app's
  ruled octet vanished with nothing red. Now refused at source, where the cause is named.

## 24. Two harness cases that could not fail, fixed

- **T3b's byte-compare was CIRCULAR.** It derived the header `--from-overlay "$LIVE"` and
  then compared `--against "$LIVE"` -- ~24 of 76 lines self-copied, so the header could not
  fail. Proven by rewriting a header line to a false claim and still getting
  "byte-for-byte match". Now reads the COMMITTED values file, the same artifact
  `render-drift` gates.
- **T29's own fixture was INERT** -- its assertion held from the dc1 overlay alone, so it
  could not detect a fault in the thing it constructed. Re-pointed to strip dc1's prefixes
  from the apex and require a REFUSAL, which only the dc1 path can show.

**OWNED:** my first re-point of T29 asserted a `--show-legs` flag **that does not exist** --
I invented an interface rather than reading one. Caught immediately by the harness and
replaced with a real behaviour. Fabricating a flag is the cardinal sin in this repo's
posture; recording it rather than quietly fixing it.

**Evidence.** `provider-bundle-check` 33/33, `render-dc-overlays` 18/18, `render-drift`
4/4 (new); gauntlet **ALL GREEN (86) on vcloud** (was 85); repo-lint 0 fail / 612 files.

## 25. Chain findings LOGGED, NOT ACTIONED

16. **`preflight.sh` P2 validates a DIFFERENT overlay set than any deploy uses** -- it
    passes one overlay where dc0's deploy passes two, and it hardcodes `--dc vr1-dc0` with
    no dc1 branch, while `dc-dc-phase4:197-204` uses it as the **dc1** gate.
17. **dc1's ruled 3-overlay input exists in no executable path.** `grep dc-ha-scaleup
    runbooks/ scripts/` and `grep vr1-dc1-machines runbooks/ scripts/` both return ZERO.
    phase-4 Step 4 names only the vips overlay -> a dc1 deploy as documented merges a
    machines block still saying `tags=openstack-vr1-dc0`, which is exactly what that
    overlay was written to prevent.
18. **phase-6's deploy commands pass no VIP overlay at all** -- since ruling 3 that means a
    merged input with zero VIPs, including the designate those commands exist to add.
19. **`overlays/${DC}-hostnames.yaml` is named in three deploy commands and was never
    authored.**
20. **Role sub-tags `control`/`compute`/`storage` are created NOWHERE** while every machine
    constrains on them -- confirmed by measurement, and it fails hard at allocation (the
    good failure mode). This is the per-role-tags Stage-5 blocker, now evidenced.
21. **`octavia-pki.yaml` lands LAST and writes the same `applications.octavia.options` map
    as the vips overlay.** Whether octavia keeps `vip` + `prefer-ipv6` depends on juju's
    merge semantics, which cannot be settled offline -- and `provider-bundle-check`
    deep-merges, so it returns PASS under EITHER behaviour and structurally cannot
    discriminate. Needs a `--dry-run` with both overlays.
22. **The in-runbook VIP guard reads only the first v4 leg**, so dropping all three v6 legs
    plus `prefer-ipv6` still scores 13/13/0.
23. **No harness pins its own case count** -- deleting the three seeded-fault cases that
    prove the byte-compare has teeth still yields `ALL PASS`.
24. Stale GATE figures in `phase-01` (4 machines vs 9, 50 apps vs 56, VIPs 11/11/0 vs the
    13/13/0 the guard 20 lines later demands) train an operator to wave the gate through.
25. The apex dump the chain reads (`vr1-office1-current-20260725.json`) predates the
    2026-07-27 apex population and nothing re-dumps or compares it to live.

---

# 2026-07-29 -- the deploy-path half of the chain audit

Operator: "approved continue". Findings 16-20 from the chain audit, actioned.

## 26. `preflight` P2 is DC-AWARE and validates the deploy's ACTUAL overlay set

**What.** `DC=` (default `vr1-dc0`) selects the site; P2 assembles the same overlays the
deploy uses, in the same order, each added only if present.

**Why.** P2 hardcoded `overlays/vr1-dc0-vips.yaml --dc vr1-dc0` with NO dc1 branch, while
`dc-dc-phase4-juju-bundle-per-dc.md:197-204` invokes this script BARE as the **dc1** gate
and tells the operator to expect PASS. A dc1 operator got a green gate that had validated
dc0's numbers against dc0's bands and never parsed a dc1 artifact. It also passed one
overlay where the dc0 deploy passes two -- a gate validating a different input than the
deploy is not validating the deploy.

**Verified:** `DC=vr1-dc1 bash scripts/preflight.sh` now reports
`validating the MERGED vr1-dc1 deploy input: bundle.yaml --overlay overlays/vr1-dc1-vips.yaml
--overlay overlays/vr1-dc1-machines.yaml --dc vr1-dc1`.

## 27. dc1's deploy command now names ALL its overlays

`phase-4` Step 4 named ONLY the vips overlay, so a dc1 deploy run exactly as written would
have merged a machines block still saying `tags=openstack-vr1-dc0` -- precisely what
`overlays/vr1-dc1-machines.yaml` exists to prevent, since one Office1 MAAS region sees BOTH
DCs' nodes. Measured: `grep -rn vr1-dc1-machines runbooks/ scripts/` returned **ZERO** --
the overlay existed in no executable path at all.

Step 4 now carries the full command, states that `dc-ha-scaleup.yaml` is DELIBERATELY
excluded (R6: VIPs land BEFORE the HA overlay), points at `DC=vr1-dc1 preflight`, and adds
the `--dry-run` machines-merge confirmation that `vr1-dc1-machines.yaml`'s own VERIFY-LIVE
note required and that no runbook step performed.

## 28. `phase-6` deployed with NO VIP overlay, and named a file that never existed

Its two `juju deploy` blocks (dry-run and the real apply against a LIVE model) passed
`octavia-pki.yaml` and `overlays/${DC}-hostnames.yaml` -- and no VIP overlay. Since ruling 3
that means a merged input with **zero VIPs, including the designate those commands exist to
add**; invariant 8 would reject it, but nothing runs the checker there. And
`${DC}-hostnames.yaml` **has never been authored** -- juju errors out on a missing overlay.
Both blocks now pass the per-DC vips + machines overlays, and the never-authored hostnames
overlay is dropped with a note that it is unbuilt D-106 work.

## 29. `scripts/maas-role-tags.sh` -- Fork 3's per-role MAAS tags, with a harness

**The blocker.** Every machine in `bundle.yaml` constrains on TWO tags
(`tags=openstack-vr1-dc0,control`). Measured: only the SITE tag is ever created
(`reenroll-hosts.sh`); the three ROLE tags are created **NOWHERE**. Juju passes `tags=` to
MAAS as an allocation constraint, so an unmatched tag yields no candidate machine. It fails
LOUDLY at allocation -- the good failure mode -- but it blocks BOTH DCs.

**Role is DERIVED from two independent signals that must AGREE** (hard rule 2): the ruled
D-134 octet band (control .100-.119 / compute .120-.149 / storage .150-.200) and the role
token in the host's own name. Disagreement REFUSES for that node rather than picking one;
an octet in no ruled band also refuses. Nodes match MAAS by **pinned boot MAC**, never by
hostname -- MAAS renames machines at enlistment and the current names are random.

Dry by default; every write READ BACK; an ABSENT `maas` CLI refuses with a message that
says so rather than "MAAS unreachable" (this deployment has misdiagnosed that three times).

**Harness 9/9**, including the two directions the audit cares about: clean when everything
is tagged, red in the exact state measured today, and a REFUSAL when the two role signals
are made to disagree.

**NOT APPLIED.** `apply --commit` is a live MAAS mutation and is operator-gated.

## 30. OWNED: the `IFS` word-splitting trap, third appearance

`ROLES="control compute storage"` with `IFS=$'\n\t'` does not word-split, so
`for r in $ROLES` saw ONE item and the script reported a single absurd tag named
`'control compute storage'`. Caught by its own harness before shipping. Same class as
commit 1's `pre-flight-checks` CHECK 1 death and phase-05's aborting captures. **Arrays are
immune; this repo's strict-bash gates should not use space-separated string lists.**

Also fixed on the way: the role-derivation refusal SWALLOWED its own diagnostic -- the
python printed which node and why, the command substitution captured it, and the failure
path discarded it, leaving the operator "disagreed with itself" and nothing else. Now the
per-node reason is printed before the refusal.

**Evidence.** Gauntlet **ALL GREEN (87) on vcloud** (was 86); repo-lint 0 fail / 614 files.
`preflight` verdict unchanged in kind (2 fatal: the absent octavia-pki secret and MAAS
unreachable from vcloud).

---

# 2026-07-29 -- the two Stage-5 blockers AUTHORED (not applied), and the Octavia question RESOLVED

## 31. Octavia overlay-merge: RESOLVED, and it is a clean negative

The chain audit flagged that `overlays/octavia-pki.yaml` lands LAST and writes the same
`applications.octavia.options` map as the vips overlay -- so under REPLACE semantics
octavia would deploy with no `vip` and no `prefer-ipv6`, and every offline gate would stay
green because `provider-bundle-check` deep-merges and cannot discriminate.

**Researched against source, not docs prose (the docs are silent on merge mechanics).**
juju 3.6 pins `github.com/juju/charm/v12 v12.1.1` (verified in juju's `go.mod`).
`mergeStructs` (`overlay.go:661-693`) does, for a map-kind field:
`for _, srcKey := range srcVal.MapKeys() { dstMap.SetMapIndex(srcKey, srcMapVal) }` -- it
KEEPS the destination map and writes into it per key. `ApplicationSpec.Options` is
`map[string]interface{}`, so it takes that branch. **DEEP-MERGE: both overlays' keys
survive; octavia ends up with all seven.** The 2.9 and 3.6 versions of that file are
byte-identical in the merge path, so there is no version risk.

**The checker's own "MIRRORS juju's documented merge" comment is therefore CORRECT** --
previously an unverified claim, now sourced.

**ONE REAL TRAP FOUND AND LOGGED: LP #2002371** (Triaged/High, unfixed) -- an overlay whose
`options:` key is EMPTY/NULL does not merge, it WIPES the base map
(`isZero(srcVal) -> reflect.MakeMap`). Also a whole-app empty body deletes the app. Our
overlays do not have that shape, but nothing gates against acquiring it.

## 32. R1 AUTHORED: the ceph-osd data device, opt-in per role

`modules/node-vm` gains `osd_disk_size_bytes` (default **0** = creates nothing). The volume
is `count`-gated and the disk entry is appended via `concat`, so **a node that does not opt
in has no extra resource in state and no plan diff** -- that is precondition 4 of the
amendment verbatim ("gate the change so it is additive and opt-in per node role, or it will
plan a change against all 18 domains").

Both substrate roots: the FOUR storage nodes per DC set `osd_gib = 500`. Scope is **8
volumes, not 18**, exactly as the amendment scoped it.

**500 GiB is the budgeted figure, not a value I chose** -- D-121's Option-C capacity
re-check ran at "4x500Gi/DC = PASS (5.31 TiB margin)". **FLAGGED: D-121 also records that
the per-node OSD footprint is an ASSUMPTION whose ratified value "pends the OSD-sizing
decision and should be re-derived from the measured DC0 footprint."** So the size is
evidenced but not separately ruled.

**HEADROOM CAVEAT worth surfacing before the apply:** precondition 2 measured dc0
`/var/lib/libvirt` at **2.0T available**, and 4 x 500 GiB is 2.0 TiB nominal. qcow2 is thin
so creation costs almost nothing, but dc0 has no room for those OSDs to actually fill. dc1
has 2.9T available. Not a blocker for the apply; a real one for sustained use.

## 33. D-104 AUTHORED: the dedicated per-DC Juju controller VM

A 10th key in each root's node map, `<dc>-juju-01`. `for_each` is keyed, so it plans
**1 add / 0 change / 0 destroy** -- additive, not a substrate reopening. The nine role
nodes are untouched, so Option C's 3/2/4 shape, the R-3 Ceph rebuild headroom and the
control-quorum spread all stand.

**Sizing is the gate's own model, not an invention:** 8 GiB / 4 vCPU are what
`dc-dc-whole-host-budget.py` modelled for the PASS (854/1024 GiB = 83 pct, FIT, 170 GiB
headroom). Disk 100 GiB is the authoring value the amendment explicitly left open.

**MACs deliberately EMPTY.** `modules/node-vm` documents that as correct ONLY before a node
is enlisted anywhere -- which is this VM's state. They must be PINNED FROM MEASUREMENT
after its first apply, exactly as the nine role nodes were; leaving them unpinned past
enlistment is the 2026-07-21 MAC-regeneration trap that stranded the fleet.

## 34. NEITHER IS APPLIED, and precondition 3 is NOT satisfied

`opentofu-validate` PASS on both roots. **No apply has been run.** Both are live substrate
mutations and are individually operator-gated.

**R1 precondition 3 remains OPEN and I have not closed it:** the amendment requires
verifying, BEFORE the apply, that re-commissioning the storage nodes preserves the D-134
statics and the pinned MACs -- and it names the 2026-07-20 MAC-regeneration incident as the
precedent for exactly this class. That verification needs live MAAS, which is absent on
this host. **I am not treating it as satisfied.**

---

# 2026-07-29 -- dc0 SUBSTRATE APPLIED (live mutation, operator-gated)

## 35. R1's OSD volumes and the D-104 controller VM are BUILT on dc0

Operator approval, exact utterance: **"Apply dc0 first"**. Executed from voffice1 (D-128
Plane 2 -- the inner tfstate lives there) via a SAVED PLAN, with a same-session pre-apply
re-verify per the G8 precedent, so what was applied is exactly what was reviewed.

**Apply complete: 6 added, 4 changed, 0 destroyed.** Four `libvirt_volume.osd` + the
`vr1-dc0-juju-01` domain and its disk; the four storage domains updated IN-PLACE to attach
`vdb`.

**THE CHECKPOINT HELD -- MAC DRIFT COUNT 0.** Verified by `virsh domiflist` against
`lib-hosts`' pinned MACs BEFORE anything touched MAAS: all 9 nodes pinned==live, 6 NICs
each. This is the whole point of sequencing the check there. 2026-07-20 became a
fleet-wide outage because MAAS was told to re-commission while its records were already
stale -- a 0/9/0 in-place apply had regenerated every MAC and **the plan did not show it**.
So the clean plan was necessary and explicitly not sufficient. **First live evidence that
the 2026-07-21 MAC pinning survives an in-place domain update.**

**A grep that looked alarming and was not:** the saved plan's MAC audit returned one hit,
`type_machine = "q35"` inside the juju-01 CREATE block -- "mac" is a substring of
"machine". Zero actual MAC lines. Recorded so nobody has to rediscover it under pressure.

**Convergence restored (precondition 1):** the dc0 inner root re-plans **ZERO DIFF**. The
property the amendment said this apply would deliberately spend is back and captured.

**MAAS undisturbed:** 20 machines (18 Ready + 2 Deployed), 216 links, every one still
`mode=static`. And **MAAS still sees ONE block device per dc0 node** -- the new `vdb` is in
libvirt but not yet in MAAS's inventory, which is the expected state and confirms the
re-commission is genuinely required rather than a formality.

**Revert.** `tofu destroy -target` on the five new resources would remove the volumes and
the controller domain; the four in-place disk attachments revert by setting `osd_gib` back
out of the storage entries and re-applying. Nothing was destroyed, so nothing is lost.

**NOT DONE, gated separately:** the re-commission
(`maas admin machine commission <sysid> skip_networking=1`), and dc1 entirely.

## 36. Canary re-commission -- R1 precondition 3 closed by measurement

Operator: **"Commission storage-01 as the canary"**. `maas admin machine commission kghggm
skip_networking=1`, sysid matched by pinned boot MAC. Ready in ~200s.

**The entire before/after diff is one hunk: `vdb` (500 GiB exactly) added to blockdevices.**
links 12 -> 12, MACs identical, links identical. Every static address, mode and subnet
survives -- including the `.150` octet mirrored across six planes in both families.

Precondition 3 was previously answered from the API's help text; it is now answered by
measurement. The concern was legitimate (the default path DOES reconfigure networking) and
`skip_networking=1` is sufficient.

**Revert.** None applicable -- commissioning is idempotent and re-runnable; the node
returned to Ready with its configuration intact.

**Remaining:** dc0 storage-02/03/04, then dc1 (apply + re-commission).

## 37. juju-01 MACs PINNED and both controllers added to lib-hosts at the ruled .5

**What.** The twelve measured MACs (six per DC, `virsh domiflist`) are pinned into both
substrate roots, and `<dc>-juju-01` is added to `lib-hosts.sh` with `HOST_OCTET=5` per the
D-134 amendment and its measured boot MAC.

**Why now.** Both VMs are ENLISTED in MAAS, which is the exact boundary `modules/node-vm`
warns about: past enlistment an in-place apply can regenerate unpinned MACs and strand the
node. The nine role nodes went through this on 2026-07-21; these two were carrying the same
exposure.

**No race with the running agents:** they operate on the **voffice1** clone, this edit is on
**vcloud**'s. Nothing reaches their working tree until someone pulls there, which is
deliberately not being done until they report.

**Recorded, not smoothed over: dc1's juju-01 MACs are libvirt-generated `52:54:00:` values,
NOT dc1's schematic `52:54:01:d1:NN:PP` scheme.** That is a direct consequence of the
deliberate `macs = []` at first apply. Written into the config comment so a later reader does
not diagnose it as a defect.

## 38. Three harness assertions RE-POINTED, one harness mechanism fixed

Adding a 10th node per DC broke three exact-count assertions. All were re-pointed to the NEW
invariant and kept EXACT rather than relaxed to `>=` -- an exact count is the entire value of
these checks:

- `tests/dc-selector` fleet count 9 -> **10**, with the label corrected to say why
  (`D-121 Option C 9 role + D-104 juju-01`) rather than leaving it attributed to Option C,
  which is a 9-node ruling.
- `tests/node-vm` T8 `macs` lists 9 -> **10**, T9 pinned MAC literals 54 -> **60**.

**And a real harness defect found on the way:** T8's `grep -c 'macs = \['` counts COMMENTS.
My own config comment explaining the deliberate `macs = []` inflated the count to 11 -- which
reads exactly like a spurious extra node. Fixed to strip comment lines before counting. Same
class as the `ledger-scan` next-free defect, where a doc merely QUOTING a token inflated the
counter.

## 39. `maas-role-tags.sh` taught the utility band

Adding `juju-01` to `HOSTS` broke the role-tag script: it iterates `HOSTS` and refused,
because octet `.5` is in no ruled ROLE band. That refusal was correct behaviour meeting a new
fact.

Per the D-134 amendment, `.4-.49` is per-DC INFRASTRUCTURE, not OpenStack roles. A host there
with no role token in its name is now **SKIPPED and REPORTED** (never silently), while the
refusal keeps its teeth for genuine disagreement. Both directions are locked by new cases:
**T10** a utility-band host is skipped and reported; **T11** a ROLE-named host placed in the
utility band still REFUSES. Harness 9 -> **11**.

**Evidence.** Gauntlet **ALL GREEN (87) on vcloud**; `opentofu-validate` PASS; repo-lint
0 fail / 614 files. `lib-hosts` resolves both DCs: 10 hosts, juju octet 5, boot MACs present.

## 40. LIVE: R1's OSD carve and the D-104 controllers complete across BOTH DCs (two agents)

Operator: **"Finish the remaining three dc0 nodes as an agent. Start a separate agent for all
of dc1. Monitor and orchestrate both."**

**Result: eight re-commissions, every one the same single-change diff** -- `vdb` at
536870912000 bytes (500 GiB) added; links 12 -> 12; per-interface MACs and link lists
identical. dc0 storage-02/03/04 at 204/196/202s; dc1 storage-01..04 all at ~182s. dc1's
substrate applied 6 add / 4 change / 0 destroy with **MAC drift 0**, re-converging to
**ZERO DIFF**.

**I verified every claim from MAAS directly rather than accepting the reports** -- fleet 22
(18 Ready + 2 Deployed + 2 New), 216 links all static, both DCs `blockdevs {1:5, 2:4}`,
`tagcount {2:9}`. All matched. I also diffed a RAW dc1 before/after pair myself: exactly the
`vdb` hunk plus one `power_state` line, nothing else.

**Captures promoted** from `voffice1:/tmp` to `docs/audit/osd-carve-20260729/` (18 files).
Both agents were barred from repo writes, so **dc1 would otherwise have had no audit record
at all** -- dc0 had one only because I wrote it by hand earlier.

**Two things the agents found that I had not asked for, both valuable:**
- **MAAS tags SURVIVED re-commissioning on all 18 nodes.** Agent A checked this unprompted,
  reasoning that its own snapshot code structurally could not see tags. That removes a
  would-be Stage-5 Juju placement blocker nobody had verified.
- **`lib_hosts_select_dc` refuses a second DC in one shell** (deliberate). Agent A had muted
  it with `2>&1` and silently got dc0 data labelled dc1; it caught itself and reported the
  trap. Reusable for anyone scripting cross-DC lookups.

**ONE DISCLOSED JUDGMENT CALL, and it was correct.** Three dc1 pairs also showed
`power_state` moving `off`/`error`. Agent B declared a STOP, re-measured, then excluded
`power_state` from its verdict -- still printing it -- because 8 of 9 dc1 nodes read `error`
BEFORE any mutation. **Vindicated: all 18 role nodes now read `off`, including five dc1 nodes
never commissioned.** A transient MAAS power-query flap. It surfaced the call explicitly and
invited reversal, which is exactly right for a live-ops agent.

**Revert.** The volumes and controller domains are removable via targeted destroy; the
in-place disk attachments revert by removing `osd_gib` and re-applying. Commissioning is
idempotent. Nothing was destroyed.

**STILL BLOCKING for both controllers: `power_type` is EMPTY** -- the state that stalled all
nine role nodes on 2026-07-20. Now fixable, since both are in `lib-hosts` at the ruled `.5`
with MACs pinned. Gated next steps: set `power_type=virsh`; commission them on the DEFAULT
path (no `skip_networking` -- nothing to preserve); create the `juju-controller-<dc>` tags;
run `maas-role-tags.sh apply --commit`.

## 41. voffice1 pulled current; power_type=virsh set on both controllers

**Pull.** `c9cc79f` -> `75e3657`. Both inner tfstates confirmed `git check-ignore`-IGNORED
BEFORE the pull and sha256 **BYTE-IDENTICAL** after -- the same safety proof Phase 0 used,
because those files are the substrate's state-of-record and live inside the working tree.

**A diff the pull revealed, explained rather than waved through.** Both inner roots now plan
`0 add / 1 change / 0 destroy`. That is **MAC ADOPTION, not drift**: the plan adds
`mac = { address = ... }` blocks carrying values IDENTICAL to what is already live, because
tofu's state has no `mac` block for a domain that was created unpinned. Exactly the
2026-07-21 shape (`0/9/0, 54 mac adoptions, ZERO replaces`). **NOT applied** -- it is a
separate gated in-place update on a running domain, and "set power_type" did not authorise it.

**Power set, scoped narrowly.** `maas-node-power.sh --commit <addr> vr1-dcN-juju` -- the
prefix glob matches ONLY `vr1-dcN-juju-01`, so the nine already-configured role nodes per DC
were not re-touched. Both verified by a real `query-power-state`:
  `[ok] moved-troll   -> vr1-dc0-juju-01 (power state: off)`
  `[ok] square-ferret -> vr1-dc1-juju-01 (power state: off)`
That read-back is the ground truth -- a stored parameter is not power control; querying the
driver proves MAAS can actually reach it.

## 42. WHY power_type was not set during commissioning -- answered, with evidence

Operator question. Two reasons, the first simpler than the second:

**(1) These machines were never commissioned.** They are in `New`, which is POST-enlistment
and PRE-commissioning. Enlistment (PXE -> MAAS registers the machine) and commissioning (the
hardware-discovery run) are distinct phases. There was no commissioning pass in which
anything could have been set.

**(2) Even commissioned, MAAS would not have set it, and structurally cannot.** MAAS
auto-configures power during commissioning only for IPMI-based machines, by probing the BMC
in-band from the ephemeral environment. Evidence from THIS MAAS's own CLI
(`maas admin machine commission --help`):

    :param skip_bmc_config: Optional.  Whether to skip
    re-configuration of the BMC for IPMI based machines.

A KVM guest has no BMC to probe. And the virsh power parameters -- the `qemu+ssh://` URI, the
SSH credential, and the libvirt DOMAIN NAME -- are facts about the HYPERVISOR, entirely
outside the guest. Nothing running inside the VM can discover them.

**That is the chicken-and-egg behind the 2026-07-20 incident:** commissioning needs power
control (MAAS power-cycles the machine to run the ephemeral env), but for a virsh VM the
power config cannot be discovered BY commissioning. It must be set out-of-band first. Which
is exactly why `maas-node-power.sh` exists and why D-103/D-123 ruled per-machine
`power_type=virsh` over a MAAS pod. So this was never a failure -- it is structural, the repo
already knew it, and the only gap was that nobody had run the step for two brand-new machines.
