Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260728-vip-arity-gate.md

Session changelog 2026-07-28 -- the VIP arity gap + R11's ruled gate changes

ONE changelog per session (GA-R2/D1). Under blanket approval this is the review surface: every item states WHAT, WHY (evidence), and HOW TO REVERT.

No live mutation in this session. Everything below is a repo change; the only live access was READ-ONLY (a NetBox apex poll and a pre-flight-checks run). No logged window was opened because nothing was mutated.

Status authority for everything here is docs/CURRENT-STATE.md (GA-R1). This file is session-scoped scratch and is NOT citable as status or decision authority.


1. provider-bundle-check.py: VIP arity, family coupling, and the apex-derived v6 bands

What. The vip validator now accepts a v4 TRIPLE (3 addresses) or a dual-family SEXTET (6), validates the three v6 legs against the per-DC v6 /64s, requires the v6 host part to MIRROR the v4 octet textually, and couples prefer-ipv6 to the arity in both directions. The v6 bands are READ FROM THE NETBOX APEX RECORD, not hardcoded. The standalone octet-uniqueness block (old item 6) folded into the same loop.

Why. Measured at :137: if len(parts) != 3 fails EVERY application once R2's ruled dual-stack lands, and :149's p.split(".")[-1] returns the whole address on a v6 literal, so every v6 leg was trivially "unique" and unchecked. The apex now holds the dual-family set a renderer will emit (verified live this session: 160 ip-addresses, 78 IPv6, 156 VIP objects), so the only gate that validates overlays could not validate what is about to be produced.

The prefer-ipv6 coupling is the substance, not a bonus. prefer-ipv6 makes HAProxy bind :::port in ADDITION to *:port; the measured L3-9 finding is that the overlay merge order which keeps prefer-ipv6 while dropping the v6 VIP legs is the one that EXITS 0 -- the dangerous order is the green one. Both directions now fail.

Authority. The arity gap was carried in docs/CURRENT-STATE.md as an open question, explicitly NOT covered by R11. Operator direction 2026-07-28, exact utterance: "Fix the arity gap first, then start the renderer", with the scope fork (arity alone vs arity + R11's three ruled changes) put separately and answered "Arity gap + R11's three ruled changes (Recommended)". OPS under GA-R3 -- a script change making an already-RULED end state expressible; doubt resolves DOWN, no D-number assigned.

Prior art followed, not reinvented. The (role, kind) apex keying and the "provider leg lives in the DEDICATED GUA VIP /64, admin/internal in their plane /64s" rule are copied from scripts/dc-plane-ipam.sh:102-131 and netbox/dc-plane-apex-import.py. Both live-API and repo-dump shapes are handled (nested scope.slug vs flat scope_site) -- writing against one and matching zero objects on the other is a trap this repo already hit once.

Refuses rather than passes. A dual-family vip with an unreadable or absent apex record exits 2, never 0: "could not look" is never "nothing there". A v4-only bundle needs no apex at all, so this introduces no new dependency for the current tree (harness T25/T26 lock both halves).

Revert. git revert this commit. The checker returns to triple-only; nothing live depends on it.

2. R11's three ruled gate changes (D-020 AMENDMENT, RULED 2026-07-27)

What.

  • provider-bundle-check.py: OCTET_LO, OCTET_HI 50-60 -> 50-99.
  • scripts/lib-net.sh: VIP_OCTET_MAX 60 -> 99, VIP_COUNT_EXPECT 11 -> 13.
  • provider-bundle-check.py: NEW invariant 8 -- an hacluster principal with no vip is a FAIL.
  • EXPECT_PUBLIC_VIP deliberately STAYS 11.

Why. R11 ruled vault .61 and designate .62 as full dual-family triples and ruled the gate hardening IN SCOPE, not deferred. cluster_count is asserted NOWHERE in scripts/ or tests/, so a 3->1 rewrite of all 20 values produces a byte-identical PASS -- decorative HA was found by audit rather than by gate.

EXPECT_PUBLIC_VIP stays 11 because -- measured -- NEITHER vault nor designate carries a public binding, so they do not join that count. Bumping both constants together breaks the gate. The two band constants are SEPARATELY NAMED in two files; this is a two-file change by construction, and both sites carry a comment saying so.

MEASURED CONSEQUENCE, stated precisely. Two sub-checks flip PASS -> FAIL:

  • provider-bundle-check on the base bundle now FAILS with hacluster relation but no vip: designate.
  • pre-flight-checks CHECK 1 now reports VIP validation OK=11 bad=0 (want OK=13).

preflight.sh was ALREADY exit 1 before this change (octavia-pki overlay absent, MAAS unreachable from the jumphost, P5's 7 credential findings). Measured after: still exit 1, now 3 fatal, 2 warning. This did NOT flip preflight from pass to fail and no deploy path that was open is closed by it -- the same precedent this document records for P5. The reds are the ruled gate reporting real work owed: R11's .61/.62 VIPs are RULED-BUT-NOT-BUILT, and per R6 they land BEFORE the HA overlay.

Revert. Revert this commit; the constants and the invariant-8 block are contiguous and independent of item 1.

3. Harness: 15 -> 30 cases, with three cases RE-POINTED rather than deleted

What. tests/provider-bundle-check/run-tests.sh gains 15 cases. The fixture base splits: pristine.yaml is the repo bundle untouched; good.yaml is that bundle PLUS designate's ruled .62 VIP.

Why the split. Invariant 8 legitimately fails the real repo bundle. Left alone, every rc=0 case (T1/T9/T14) would have been asserting "fails for the reason we already know" instead of its own invariant. Per the standing rule -- when remediating a finding turns the harness red, REPLACE the assertion with the new invariant and say so, never delete it to go green -- the real tree's state is kept honest by a NEW case (T16) that asserts pristine.yaml DOES trip it. When .62 lands in bundle.yaml, T16 must be re-pointed, not deleted.

Proven able to fail AND to pass -- the complement this project recorded on 2026-07-27 after building three gates whose every live run failed:

  • T16 the pristine bundle FAILS on designate; T18 the same check goes GREEN once the ruled VIP lands.
  • T17 vault FAILS under dc-ha-scaleup.yaml -- R6's "VIPs before the overlay" ordering, made executable.
  • T19 a valid dual-family VIP PASSES and is counted as dual; T20/T21 both directions of the prefer-ipv6 coupling FAIL; T22 a v6 leg in the node /64 instead of the VIP /64 FAILS; T23 the ::32 hex trap FAILS; T24 a 2-address vip FAILS; T25 an unreadable apex REFUSES at rc=2; T26 a v4-only bundle needs no apex; T27/T28 the band is widened to 99 but still bounded.

T29/T30 exercise the dc1 arm of the apex lookup, which T19-T23 never reached (they all run at the default --dc vr1-dc0): T29 resolves dc1's f03:11::/64 / :320::/64 / :321::/64 and passes, T30 proves a dc0 v6 leg under --dc vr1-dc1 FAILS, so a cross-DC copy-paste of a rendered overlay cannot pass. The renderer's first real output is per-DC overlays, and a gate observed in only one DC mode is the same untrustworthiness T16/T18 exist to rule out.

T8's assertion string moved outside 50-60 -> outside 50-99 (the same re-pointing rule; the case still guards the band, at its ruled bounds).

Evidence. Harness 30/30 (was 15). Gauntlet ALL GREEN (84 harnesses) on vcloud -- the host is named because the gauntlet is measured HOST-DEPENDENT (2/81 fail on voffice1 on an identical commit). repo-lint 0 fail / 1 standing warn / 605 files scanned. No harness was added or renamed, so tests/HARNESS-MANIFEST is unchanged and needed no re-record.

Revert. Revert this commit; the harness returns to 15 cases.


Findings LOGGED, NOT ACTIONED (hard rule 1)

  1. pre-flight-checks.sh CHECK 1 is triple-shaped. Its awk parse assumes three columns, so it will mis-read a dual-family sextet when those artifacts land. Same class as the arity gap just fixed, one gate over. Not touched -- it interlocks with BLOCKER-1 (the ruling-3 VIP extraction makes CHECK 1 see 0 VIPs).
  2. cluster_count coherence is still unasserted. Invariant 8 catches an hacluster principal with NO vip, which is what R11 ruled. It does NOT check that cluster_count matches the principal's num_units -- a 3->1 rewrite of all 20 values still produces a PASS. Out of ruled scope; worth its own exchange.
  3. NetBox instance/target drift (raised by the operator this session). Four netbox/*.py tools carry in-file usage examples pointing at https://netbox.baldurkeep.com -- the FROZEN v1 reference. Two of them (ipv6-mark-reserved.py, ipv4-prefixes-import.py) have NO SANDBOX_HOSTS write guard at all while carrying write paths (update_status, create_or_report_prefix, create_or_report_iprange). The guarded tools also expose a deliberate --yes-write-upstream escape hatch, so "writes only reach 10.10.1.10" is a convention with an override, not a hard invariant.
  4. Credential env-file naming is inverted. ~/vr1-office1-creds/vr1-netbox.env points at netbox.baldurkeep.com (the read-only v1 reference) while vr1-netbox-sandbox.env points at the LIVE VR1 apex http://10.10.1.10:8000. The filename says the opposite of what each is.
  5. The NetBox ?site= API filter silently does not filter. Measured this session: /api/ipam/ip-addresses/?site=vr1-dc0 and ?site=vr1-dc1 each returned the full 160. A per-site apex count taken that way would be wrong and look right.

2026-07-29 -- render-pipeline STEP 4: the renderer

Same branch, continued. Operator direction: "You can order the steps in whatever order you want. Deploy agents to help if needed."

4. Ordering decision, and why it INVERTED after measurement

I first proposed doing ruling-3's VIP extraction before the renderer, then reversed it once three read-only agents reported. Recording the reversal rather than the conclusion, because the reason is the reusable part:

  • The byte-for-byte reproduction WINDOW IS STILL OPEN and closes on its own. overlays/vr1-dc1-vips.yaml is still byte-identical to tests/render-baseline/fixtures/vr1-dc1-vips.v4-11app.yaml (3f93ecb3...) -- verified this session. The reconciliation (11->13 apps, dual-family) destroys it.
  • The extraction's blast radius is FAR larger than ruling 3's "2 commits". Measured: it breaks a DEPLOY GATE (runbooks/phase-01-bundle-deploy.md:170-176, where juju deploy runs only on 11/11/0), the Octavia SAN derivation (:371), preflight in two places, and 8 of 30 harness cases.
  • So the extraction is better done BY the renderer, once the renderer is proven -- which makes dc0's overlay a validated output rather than a hand edit.

5. scripts/render-dc-overlays.py -- two stages, split where F2 froze the interface

What. derive (apex + lib-net.sh -> a per-DC VALUES file) and render (VALUES -> overlay TEXT). Plus check, which renders and byte-compares.

Why split. Forward item F2 froze the interface: "the renderer is a deterministic command-line tool whose inputs are the per-DC values file and whose outputs are committed, reviewable artifacts. Nothing about automating it should require changing the renderer." render is therefore PURE -- no apex, no network, no clock -- so it is offline-testable and a future CI job only re-runs derive.

Why a text emitter, not yaml.dump. The target is byte-compared. A round-tripper destroys the comment header, normalises vip: "a b c" to a plain scalar, and imposes its own key order -- all three are load-bearing. PyYAML reads the values file and never writes an overlay.

Two things that are INPUT, not output, and would otherwise be invented:

  • The 12-line comment header cites one-off measurements and a design-doc fork; none of it is computable from the apex. derive --from-overlay lifts it VERBATIM from the artifact being reproduced, so a reproduction run is honest about which bytes the tool actually generates.
  • Emission order is ASCENDING OCTET, not alphabetical (keystone .50 before barbican .51). Sorting by name reproduces nothing.

The ruled app->octet map is READ, not restated -- parsed with ast out of netbox/dc-plane-apex-import.py's APP_OCTET, the tool that wrote those values into the apex. A second hand-maintained copy is a second thing to drift.

Revert. Delete scripts/render-dc-overlays.py, tests/render-dc-overlays/, render/values/, and revert the render-baseline T6 hunk + the manifest line. Nothing else consumes any of it; no committed artifact was regenerated by it yet.

6. RESULT: byte-for-byte reproduction achieved, and PROVEN ABLE TO FAIL

render output hashes to 3f93ecb3205cea8b303f5b2cbbbde343a2762156575277ca84684369da8bf896 -- identical to both the committed overlay (1634 bytes) and the frozen fixture.

A byte-compare only ever observed PASSING is as untrustworthy as a gate only ever observed failing, so three faults were seeded and all three fail it: a wrong octet, a dropped header, and dc0 prefixes against the dc1 artifact. render-baseline's new T6 was separately proven by making the renderer sort alphabetically -- hash changed, case went red.

7. THE ARITY WORK PAID FOR ITSELF WITHIN THE HOUR -- a real renderer bug, caught by the gate

The renderer's FIRST dual-family output was malformed. The values file stores each v6 /64 base with trailing colons stripped (2602:f3e2:f03:11), and the emitter joined with a SINGLE colon -- yielding 2602:f3e2:f03:11:50. That is not a valid address, and it still looks like one.

provider-bundle-check.py -- fixed yesterday for exactly this class -- rejected all 13 applications with bad vip ip. Fixed to ::%d and locked by harness cases T13/T13b, which assert both that the correct form is present AND that no single-colon leg is emitted. This is the acceptance criterion the whole arity fix existed to provide, working on its first real use.

8. End-to-end chain now demonstrated

apex -> values -> renderer -> overlay -> gate. The dual-family dc1 render with R11's ruled-but-unbuilt apps included produces 13 clustered VIPs, all dual-family, and provider-bundle-check --dc vr1-dc1 reports PASS including the new hacluster invariant (12 principals, all carrying a VIP).

Evidence. New harness tests/render-dc-overlays 17/17; render-baseline 10/10 (was 9, T6 added -- ADDED per its README, existing assertions NOT repointed). Gauntlet ALL GREEN (85 harnesses) on vcloud (was 84); tests/HARNESS-MANIFEST re-recorded deliberately after the drift gate caught the addition -- which is the gate working. repo-lint 0 fail / 608 files scanned.


Findings from the three read-only agents -- LOGGED, NOT ACTIONED (hard rule 1)

  1. F3's MAC sourcing claim is FALSIFIED. docs/dc-dc-deployment-workflow.md:361 says ovn-chassis MACs "stay sourced from scripts/lib-hosts.sh". Measured: lib-hosts carries only HOST_BOOT_MAC (the PXE NIC, ...:NN:01); ovn-chassis needs the provider-public NIC (...:NN:02), which is not in lib-hosts at all. The real source is opentofu/vr1-dcN-substrate/main.tf, macs[1] of each compute node -- and it exists for BOTH DCs. This also narrows CURRENT-STATE's claim that dc0's MACs "cannot be derived": not from a PATTERN, but they are pinned literals in tofu and are perfectly sourceable. F3's conclusion (MACs are renderer output, not apex input) still stands; only its sourcing sentence is wrong.
  2. scripts/dc-plane-ipam.sh:120 has a latent scope-matcher bug. It matches scope.name or scope_site but NOT scope.slug, so it would match ZERO prefixes against a live-shaped record. Latent only because it defaults to a repo dump. The other two tools use the three-way matcher; this one is the odd one out.
  3. pre-flight-checks.sh CHECK 1 is triple-only AND dc0-only. It hardcodes if(n!=3) (:86), reads bundle.yaml as RAW TEXT (never a merged input), and never calls lib_net_select_dc -- and lib-net.sh deliberately UNSETS VIP_PREFIX_* for dc1, so CHECK 1 cannot evaluate dc1 at all. The renderer's dual-family output will pass provider-bundle-check and FAIL preflight P4.
  4. CURRENT-STATE is stale on dc1's FIP pool -- an early paragraph still says it "needs a ruling"; a later entry records it RULED and applied, and lib-net.sh carries the values. Same in-document supersession class this doc keeps finding.
  5. render-baseline T3 compares COUNTS, not SETS. A deleted pinned fixture plus an added unpinned one keeps the count equal; a # comment line in SHA256SUMS inflates the pin count and masks one unpinned file; and -maxdepth 1 cannot see a subdirectory. It catches the fault it was seeded against; a set-difference would close all three.
  6. The B1/B5 comment tokens are defined in bundle.yaml's own header (:11-13, :19-24). Ten of the eleven inline VIP comments are the bare token # B1. Migrating them into an overlay STRANDS them from their definitions -- appendix-A:652-661 records the repo's rule for exactly this identifier class. This must be handled when the extraction lands, not discovered then.

2026-07-29 -- RULING 3, COMMIT 1: the neutral VIP MOVE (+ blast-radius repair)

Operator: "Approved, continue". This is commit 1 of the two ruling 3 (2026-07-25) prescribes. The dual-stack ADD is commit 2 and is deliberately NOT in here.

9. bundle.yaml is now VIP-FREE; overlays/vr1-dc0-vips.yaml is GENERATED

What. All 11 inline vip: lines removed from bundle.yaml; placement's then-empty options: block dropped rather than left parsing as null. The values moved to overlays/vr1-dc0-vips.yaml, rendered by scripts/render-dc-overlays.py from render/values/vr1-dc0-vips.yaml -- the renderer's first real output.

The 11 inline comments were carried, not dropped. Ten are the bare token # B1 and three carry real operational content (keystone's catalog-endpoint note, ceph-radosgw's un-defer note, openstack-dashboard's ALLOWED_HOSTS warning). They were extracted VERBATIM from bundle.yaml by a parser, not retyped, and the renderer grew optional per-app comment support to carry them. An app with no comment emits none, so the dc1 artifact still reproduces byte-for-byte (re-verified: 1634 bytes, unchanged).

The B1/B5 tokens stay DEFINED in bundle.yaml's header, where they always were; the overlay header names that explicitly so the references do not dangle (appendix-A:652-661 is the repo's rule for this identifier class).

PROVEN NEUTRAL by the ruling's own named check. provider-bundle-check on the pre-extraction bundle and on bundle.yaml --overlay overlays/vr1-dc0-vips.yaml --dc vr1-dc0 produce line-for-line identical output, including the same known hacluster relation but no vip: designate failure. Separately, a structural diff confirmed only vip changed: applications 56 -> 56, relations 108 -> 108, every non-VIP option and every non-option key byte-identical.

Revert. git revert; the values file and overlay are additive and the bundle edit is a pure line removal.

10. Blast-radius repair -- the part that made this bigger than "2 commits"

Every item below would have been BROKEN by the move. All were found by the read-only agent sweep BEFORE the edit, not after.

  • scripts/preflight.sh P2 validated the BARE base, which post-move reports 11 phantom "no vip" failures. Now validates the MERGED dc0 input; the overlay is passed only when present so a tree without it still runs.
  • scripts/pre-flight-checks.sh CHECK 1 read bundle.yaml as raw text and would have seen ZERO VIPs -> hard-fail. Now reads the merged source. This is BLOCKER-1, fixed the ruled way (point it at the merged bundle) and explicitly NOT by bumping the expected count. A measured trap on the way: the file sets IFS=$'\n\t', so a space-joined "$VIP_SRC" reached grep as ONE filename, grep exited 2, pipefail propagated and the gate died silently mid-check. Fixed with a bash ARRAY.
  • runbooks/phase-01-bundle-deploy.md -- the real hazard. Its RUN block gates juju deploy on TOT/HI/LO == 11/11/0; post-move that reads 0/0/0 and aborts the deploy. Both guard blocks now span base + overlay. A second bug had to be fixed in the same edit: grep -c over TWO files prints one count PER FILE, so the bare $( ) would have captured a two-line string and every numeric test would have failed anyway -- the counts are now awk-summed. Verified end to end: the gate reads 11/11/0 -> DEPLOY. The deploy line itself now carries --overlay overlays/vr1-dc0-vips.yaml.
  • phase-01's Octavia SAN derivation (DOCFIX-067) read bundle['applications']['octavia']['options']['vip'], which post-move raises KeyError -- swallowed by || true into an EMPTY VIP, then a "implausible VIP" abort. Now reads the overlay. Verified: derives 10.12.4.57.
  • runbooks/phase-00-teardown-maas-reset.md:233 bare checker call -> merged input.
  • runbooks/dc-dc-phase4-juju-bundle-per-dc.md said "DC1 needs NO VIP/CIDR changes at all" and "deploy the EXISTING bundle.yaml with NO VIP edits". Followed verbatim post-move that deploys a VIP-free DC1. Superseded in place (the D-101 plane reasoning is kept, because it is still correct); the DC2 instruction to "edit bundle.yaml's VIP lines" now points at rendering a new overlay instead.
  • runbooks/appendix-A-troubleshooting.md L3 told an operator to grep the bundle for the metal-side VIP token -- which now returns zero, and the block's || true makes zero look like a clean result. Note added.
  • tests/render-baseline/README.md records a capture-time bundle.yaml sha256 that no longer re-verifies. Marked HISTORY with what changed and why, rather than re-pinned -- the fixture hashes themselves are untouched.

11. Harness: 30 -> 31, with the fixture BASE re-pointed

tests/provider-bundle-check now builds its fixture base by MERGING overlays/vr1-dc0-vips.yaml into bundle.yaml, because that pair is the deploy input. T16's label changed from "pristine repo bundle" to "the real dc0 deploy input" -- same assertion, honest name. NEW T31 asserts ruling 3's new invariant directly: the BARE base fails for every clustered principal, so "deployed without its overlay" is a tested state rather than an assumption. 31/31.

Evidence. Gauntlet ALL GREEN (85) on vcloud; repo-lint 0 fail / 610 files; render-dc-overlays 17/17; render-baseline 10/10 (dc1 reproduction still byte-exact after the comment-support change); preflight 16/16.

Still red, unchanged and expected: pre-flight-checks CHECK 1 reports OK=11 want 13 and provider-bundle-check reports designate -- both are R11's .61/.62 VIPs being RULED-BUT-NOT-BUILT, which is commit 2's job, not this one's.


2026-07-29 -- RULING 3, COMMIT 2: the dual-stack ADD (R2 + R11 built)

Operator: "Continue as autonomously as possible. Deploy agents as needed." Two read-only agents mapped the blast radius before any edit -- the same move that caught the deploy-gate break in commit 1, and it caught another one here.

12. Both per-DC VIP overlays are now DUAL-FAMILY, 13 apps

What. overlays/vr1-dc0-vips.yaml and overlays/vr1-dc1-vips.yaml re-rendered: each app carries prefer-ipv6: true and a six-address vip (3 v4 legs + 3 v6), and vault .61 and designate .62 are now BUILT (R11, a D-020 amendment). v6 prefixes come from the apex; the host part mirrors the v4 octet textually.

RESULT, measured both DCs: 13 clustered VIP(s) ... (13 dual-family) and 12 hacluster principal(s) all carry a VIP -> PASS. With dc-ha-scaleup.yaml stacked on top: 13 principals, PASS -- so R6's ruled ordering (VIPs before the HA overlay) is now satisfied and executable, not just stated.

Two decisions that were RULED-BUT-NEVER-BUILT are now artifacts: R11's vault/designate VIPs, and R2's dual-stack for the API VIP layer.

Revert. Re-render both overlays from the values files without --dual-family --include-unbuilt, and revert the gate/harness hunks below.

13. THE L3-9 COLLISION IS RESOLVED -- and the recorded description of it was half stale

overlays/dc-dc-ipv6-family-matrix.yaml carried a vip + prefer-ipv6 pair for ten API charms, every value an unrendered {{TOKEN}}. Those duplicated what the per-DC overlays now own, and take-last merge between them is the collision. Narrowed to ceph-mon only.

The vip and prefer-ipv6 keys were removed TOGETHER. Removing the vips alone would have recreated the exact defect on purpose -- which is why this is worth stating.

MEASURED, and it corrects this repo's own record: CURRENT-STATE described L3-9 as "the dangerous merge order is the one that PASSES". That has not been true since 2026-07-28. Invariant 9 (added by the arity commit, precisely because of L3-9) makes vips-last fail on the coupling, while matrix-last already failed on the placeholders -- so NEITHER order was green. The gate closed the SILENT half; this commit closes the collision itself. Both merge orders now PASS, i.e. the result is order-independent, which is the real fix.

What survives, because it exists in NO other file: ceph-mon's ceph-public-network / ceph-cluster-network ULA-only config and its privacy-extension caveat. ceph-mon is in neither VIP overlay and never will be -- it has no VIP. Also removed: the two Launchpad citations arguing octavia lb-mgmt IPv6 was an open risk. Both were read in full and BOTH failed (one a different installer's bug, Fix Released 2021; one not an IPv6 defect at all), and R8 RULED the question closed on 2026-07-27 -- leaving them standing in a live artifact was a refuted citation presented as current.

14. Gate repairs the agents caught BEFORE the edit

  • runbooks/phase-01-bundle-deploy.md -- the deploy guard would have ABORTED AGAIN. Its HI regex is 10\.12\.4\.(5[0-9]|60), which excludes .61 and .62, so the counts would have read 13/11/0 against a required 11/11/0 and juju deploy would not have run. Widened to (5[0-9]|6[0-2]) and the expected counts moved to 13/13/0. Verified end to end: 13/13/0 -> DEPLOY. This is the second time this one guard has nearly blocked the deploy across two commits; it is worth knowing it is that brittle.
  • scripts/pre-flight-checks.sh CHECK 1 hardcoded if(n!=3) and would have printed MALFORMED(not 3 IPs) for all 13 and failed OK=0 bad=13. Now accepts a v4 triple OR an R2 sextet, and for a sextet asserts the last three legs are actually v6 literals. Deep v6 prefix/mirror checking is deliberately NOT duplicated here -- provider-bundle-check does it against the apex. Proven able to fail: a 4-address vip -> MALFORMED(not 3 or 6), a v4 in a v6 slot -> NOT-IPv6 in v6 leg 1, and a v4 triple still passes.

15. Harness: 31 -> 33, and 18 for the renderer, with everything RE-POINTED not deleted

  • T16 and T17 both asserted R11's gaps were OPEN. R11 is now built, so both were inverted to the surviving invariant (the deploy input is CLEAN; R6's ordering is satisfied at 13 principals) and paired with NEW T16b/T17b that remove a VIP and demand the check still fires -- otherwise inverting them would have removed the teeth.
  • T20/T21, the coupling cases, had become NO-OPS (the overlay already sets what they were setting) -- they would have gone GREEN while testing nothing, which an agent caught. Inverted: T20 now truncates the vip while keeping prefer-ipv6, T21 drops prefer-ipv6 while keeping six legs.
  • A v4-only twin fixture was added so the pre-dual-stack cases (T5/T8/T26/T28) still exercise band, octet-uniqueness and apex-independence instead of tripping the coupling check first and never reaching their own assertion.
  • The renderer harness now proves reproduction on BOTH shapes: T3 against the FROZEN v4 fixture (immutable, a stable target) and new T3b against the LIVE dual-family overlay (whose own header mandates it stay renderer-reproducible).

Evidence. provider-bundle-check 33/33, render-dc-overlays 18/18, render-baseline 10/10, preflight 16/16. Gauntlet ALL GREEN (85) on vcloud; repo-lint 0 fail / 611 files.

PREFLIGHT: BOTH STANDING REDS CLEARED. It was 3 fatal with P2 and CHECK 1 among them; it is now 2 fatal -- the deliberately-absent octavia-pki overlay and MAAS unreachable from vcloud (a missing-binary artifact that R10 measured clears on voffice1). Nothing this work introduced is red.


Findings LOGGED, NOT ACTIONED

  1. ceph-mon's two CIDRs are gated by NOTHING. provider-bundle-check skips any app without a vip, so {{DC_STORAGE_ULA_CIDR}} / {{DC_REPLICATION_ULA_CIDR}} are validated by no gate at all -- and they are now the ONLY content of that overlay. A prominent header warning was added; rejecting {{...}} in any option value would be the real fix and is a gate change out of this commit's scope.
  2. octavia's dual-family API VIP is an INFERENCE, flagged not hidden. R2 ruled dual-stack and R8 ruled octavia's lb-mgmt network, but nothing explicitly rules that octavia's own API VIP goes dual-family. It is in both VIP overlays, so a uniform re-render gave it prefer-ipv6: true. That follows R2 applied evenly and matches the old matrix file's own reasoning ("no different from the other charms"), but it is not a quoted ruling -- worth one operator confirmation.
  3. Nothing checks vip-WITHOUT-hacluster. Invariant 8 catches a clustered principal with no VIP; the inverse is unguarded. Relevant because vault-hacluster exists only in dc-ha-scaleup.yaml, so a base+dc0-vips deploy gives vault a VIP and no HA relation.
  4. runbooks/dc-dc-phase4-juju-bundle-per-dc.md Step 6 still instructs passing the matrix overlay alongside two overlays that do not exist on disk and no VIP overlay. Wrong before this change; now wrong differently.

2026-07-29 -- the three Octavia items: R8a BUILT, two stale surfaces, R7 generator

Operator: "continue on with the next three", after an upstream/vendor research pass on whether Octavia can run full IPv6 (it can; nothing inside Octavia requires IPv4).

16. R8a BUILT -- and a decision record that claimed it already shipped, corrected

What. scripts/phase-05-octavia-verify.sh gains the LP #2018998 o-hm0 / lb-mgmt-net MTU comparison R8a ruled on 2026-07-27. It resolves the network BY TAG (never by the name lb-mgmt-net), reads o-hm0's MTU from ip -o link, FAILS on mismatch in either direction, and REFUSES rather than passing when either value cannot be read.

Why it matters. A jumbo lb-mgmt-net beside a 1500 o-hm0 silently drops health heartbeats >1500B and surfaces as SPURIOUS load-balancer failovers -- a symptom that reads as a Ceph, network or amphora fault long before anyone suspects MTU. Fix Released for our lineage, but recurring 2025-12-31 on octavia 14.0.0 / 2024.1 stable, the exact channel bundle.yaml pins. R3's jumbo ruling is this bug's precondition.

THE RECORD WAS FALSE, and that is the more important half. design-decisions.md carried "Delivery: the assertion ships in scripts/phase-05-octavia-verify.sh with its harness green..." -- written in the present tense on the day of the ruling and reading as a statement of fact. Measured: the file contained ZERO MTU references and its last commit predated the ruling by a month. This is a sharper form of RULED-IS-NOT-BUILT: the decision doc was the false witness rather than merely silent. Corrected in place with what was measured and what has now actually been built.

Revert. Revert this commit; the MTU block is contiguous and the harness cases are additive.

17. TWO OWNED ERRORS in how I did it

(a) I concluded "NO octavia harness exists" from a name grep (ls tests/ | grep -i octavia) and wrote a NEW harness. The script's harness has existed all along as tests/phase-05/ -- it simply is not named after the script. Two harnesses for one script is itself a drift hazard, so the duplicate was DELETED and the R8a cases folded into the real one. Manifest went 85 -> 86 -> 85 across the mistake and its repair.

(b) My first draft of the MTU block ABORTED THE SCRIPT. LBNET_MTU="$([ -n "$LBNET" ] && openstack ... )" under set -euo pipefail WITH inherit_errexit: when the network is unresolvable the substitution exits non-zero and kills the run mid-check instead of reaching the refusal branch. The pre-existing tests/phase-05 harness caught it -- the harness I had just declared nonexistent. Fixed with explicit || true captures and an if block; a case now locks it ("REFUSES, does not abort").

Same defect class as the IFS=$'\n\t' word-splitting trap in commit 1: this repo's gate scripts run under strict bash, and a "safe-looking" command substitution is a silent exit.

18. Two live surfaces that still contradicted R8

runbooks/dc-dc-phase4-juju-bundle-per-dc.md:304-311 and docs/dc-dc-deployment-workflow.md:775-781 both still described Octavia's lb-mgmt IPv6 support as "a real, open risk, not resolved" and recommended keeping it v4-only as a logged D-101 exception -- citing the two Launchpad bugs that were read in full and BOTH failed. R8 ruled the opposite on 2026-07-27. We purged those citations from the overlay and never from these two, so a runbook reader would have re-litigated a closed ruling in the wrong direction. Both superseded in place, naming R8, the bug statuses, and the upstream finding that the charm has no IPv4 code path for that subnet at all.

19. R7: the Octavia PKI generator is no longer dc0-frozen

What. runbooks/phase-01-bundle-deploy.md Step 1.0-GEN gains a $DC selector, and the two things R7 ruled are derived from it: the baked /CN=VR0 DC0 ... CA subjects become ${DC_LABEL}, and the VIP gate's ^10\.12\.4\. regex now derives this DC's provider prefix from the same overlay it read the VIP from.

Why it was broken. dc1's octavia VIP is 10.12.64.57, so the gate hard-ABORTED, and phase-01:144-145 hard-ABORTs the deploy without the overlay -- i.e. the generator could not produce a dc1 artifact at all. R7's own caveat was explicit: the gate "must read the MERGED deploy input, not bundle.yaml, or it breaks again the moment ruling-3's VIP extraction and R11's .61/.62 land." Both have since landed.

Verified both ways: dc0 10.12.4.57 vs prefix 10.12.4 -> OK; dc1 10.12.64.57 vs prefix 10.12.64 -> OK; and the negative control (dc1's VIP against the old dc0-frozen regex) still ABORTs, which is exactly what used to happen on every dc1 run.

Deliberately NOT changed, both logged in place: the controller cert's CN and DNS SANs still carry dc0.vr0 -- R7 scoped only the CA subject and the VIP gate, and those DNS names are inert while os-public-hostname is unset on every API charm (IP-only endpoints, B5). And .split()[0] still takes only the PROVIDER v4 leg, so this cert gains no IPv6 IP SAN even though the VIP is dual-family. That is a gap, not a break -- this is the amphora control-plane trust domain, separate from API TLS, which Vault issues with IPv6 SANs under D-109. Adding v6 IP SANs needs its own ruling.

Evidence. tests/phase-05 14 cases ALL PASS (was 9); gauntlet ALL GREEN (85) on vcloud; repo-lint 0 fail / 611 files.

20. IPv6 IP SANs on the Octavia controller cert -- RULED and BUILT

Ruling (GA-R5). Question as presented: R7's work left the SAN derived by .split()[0] -- the provider v4 leg only -- while ruling 3 commit 2 had made every API VIP dual-family, so the amphora control-plane cert carried no IPv6 IP SAN. Flagged as a gap needing its own ruling. Operator answer, exact utterance: "Yes, add the v6 IP sans." Recorded as a D-109 RULING NOTE (2026-07-29); OPS under GA-R3, no new D-number.

What. The generator emits IP.1 = this DC's provider v4 leg and IP.2 = its provider v6 leg, both derived from the same per-DC overlay under R7's $DC selector.

Verified: dc0 -> 10.12.4.57 + 2602:f3e2:f02:11::57; dc1 -> 10.12.64.57 + 2602:f3e2:f03:11::57; a v4-only control emits NO v6 SAN, so the generator stays correct before the dual-stack ADD. Admin and internal legs stay excluded, as they always were.

Two repo guards fired on the way and both were right. repo-lint L5 rejected a heading that LED with D-109 (it reads as a second definition of the decision) -- retitled to the repo's own note form, RULING NOTE <date> -- D-109:. That is the third time this trap has bitten in this branch's history. And L10 rejected the commit for touching a design-decisions Status line without CURRENT-STATE in the same change (GA-R1 C1).

OBSERVATION LOGGED, NOT ACTED ON: amphorae reach the controller on o-hm0's charm-generated fc00::/64 ULA (controller_ip_port_list), not on any VIP leg, so neither SAN matches THAT path. Measured upstream, the controller verifies the amphora by UUID; the reverse direction was not established from source. This ruling makes the SAN set consistent with the VIP it has always been derived from -- whether that is the right derivation is a separate question, to settle by inspection at the Octavia step.


2026-07-29 -- D-136 STEP 6: the chain audit, and what it found

Three read-only lenses over data source -> rendering tool -> deployment mechanism (D-136 option (D)'s ruled final step). Lens 3 was adversarial and was told to make the chain produce a wrong result that still comes out GREEN. It succeeded three times.

21. All three lenses converged: dc1 was gated, dc0 was not

tests/render-dc-overlays byte-compared the dc1 overlay only. dc0's -- the artifact preflight.sh P2 actually feeds the deploy gate -- was reproducible by nobody, and render/values/ was referenced by ZERO scripts and ZERO harnesses. Executed attacks that passed every gate:

  • Hand-transposing two apps' octets in the dc0 overlay (glance .53 <-> magnum .54, all six legs). Internally consistent, in-band, right /22 and /64, unique -> PASS.
  • One word: family: dual -> family: v4 in dc0's values file. That drops prefer-ipv6 and the v6 legs TOGETHER, so invariant 9's coupling check is satisfied, and vip_dual is printed but never asserted -> dc0 silently reverts to IPv4-only against R2's ruling, PASS.

Root cause: R9's render-drift check was RULED (D-119 amendment, 2026-07-27) and never built. lib-net.sh:171-177 still carried the future-tense "becomes GENERATED".

22. tests/render-drift/ -- R9's gate, built

Iterates render/values/*.yaml, reads each file's own output: field (inert until now), renders, and byte-compares against the committed overlay. A new per-DC values file is gated the moment it is committed -- no harness edit. Carries a non-zero floor (a drift gate over zero files is not a gate) and a PROOF-OF-TEETH case that mutates a copy and requires the check to go red. 4/4; both DCs byte-match today.

23. Three refusals added where the chain took the last writer

  • Duplicate (role, kind) in the apex. Both readers were last-writer-wins. Measured: appending ONE spare prefix (2602:f3e2:f02:99::/64, routine IPAM work) silently re-homed every dc0 VIP into a different /64 and the whole chain went green -- because the gate only checks overlay-vs-apex AGREEMENT, and both sides read the same wrong value. Which prefix won depended on JSON array order, which is not a contract. Both render-dc-overlays.py and provider-bundle-check.py now REFUSE (rc=2) and name both candidates.
  • APP_OCTET duplicates / out-of-band octets. A duplicate app name emitted two YAML blocks for one app; yaml.safe_load kept the last and the gate passed, so one app's ruled octet vanished with nothing red. Now refused at source, where the cause is named.

24. Two harness cases that could not fail, fixed

  • T3b's byte-compare was CIRCULAR. It derived the header --from-overlay "$LIVE" and then compared --against "$LIVE" -- ~24 of 76 lines self-copied, so the header could not fail. Proven by rewriting a header line to a false claim and still getting "byte-for-byte match". Now reads the COMMITTED values file, the same artifact render-drift gates.
  • T29's own fixture was INERT -- its assertion held from the dc1 overlay alone, so it could not detect a fault in the thing it constructed. Re-pointed to strip dc1's prefixes from the apex and require a REFUSAL, which only the dc1 path can show.

OWNED: my first re-point of T29 asserted a --show-legs flag that does not exist -- I invented an interface rather than reading one. Caught immediately by the harness and replaced with a real behaviour. Fabricating a flag is the cardinal sin in this repo's posture; recording it rather than quietly fixing it.

Evidence. provider-bundle-check 33/33, render-dc-overlays 18/18, render-drift 4/4 (new); gauntlet ALL GREEN (86) on vcloud (was 85); repo-lint 0 fail / 612 files.

25. Chain findings LOGGED, NOT ACTIONED

  1. preflight.sh P2 validates a DIFFERENT overlay set than any deploy uses -- it passes one overlay where dc0's deploy passes two, and it hardcodes --dc vr1-dc0 with no dc1 branch, while dc-dc-phase4:197-204 uses it as the dc1 gate.
  2. dc1's ruled 3-overlay input exists in no executable path. grep dc-ha-scaleup runbooks/ scripts/ and grep vr1-dc1-machines runbooks/ scripts/ both return ZERO. phase-4 Step 4 names only the vips overlay -> a dc1 deploy as documented merges a machines block still saying tags=openstack-vr1-dc0, which is exactly what that overlay was written to prevent.
  3. phase-6's deploy commands pass no VIP overlay at all -- since ruling 3 that means a merged input with zero VIPs, including the designate those commands exist to add.
  4. overlays/${DC}-hostnames.yaml is named in three deploy commands and was never authored.
  5. Role sub-tags control/compute/storage are created NOWHERE while every machine constrains on them -- confirmed by measurement, and it fails hard at allocation (the good failure mode). This is the per-role-tags Stage-5 blocker, now evidenced.
  6. octavia-pki.yaml lands LAST and writes the same applications.octavia.options map as the vips overlay. Whether octavia keeps vip + prefer-ipv6 depends on juju's merge semantics, which cannot be settled offline -- and provider-bundle-check deep-merges, so it returns PASS under EITHER behaviour and structurally cannot discriminate. Needs a --dry-run with both overlays.
  7. The in-runbook VIP guard reads only the first v4 leg, so dropping all three v6 legs plus prefer-ipv6 still scores 13/13/0.
  8. No harness pins its own case count -- deleting the three seeded-fault cases that prove the byte-compare has teeth still yields ALL PASS.
  9. Stale GATE figures in phase-01 (4 machines vs 9, 50 apps vs 56, VIPs 11/11/0 vs the 13/13/0 the guard 20 lines later demands) train an operator to wave the gate through.
  10. The apex dump the chain reads (vr1-office1-current-20260725.json) predates the 2026-07-27 apex population and nothing re-dumps or compares it to live.

2026-07-29 -- the deploy-path half of the chain audit

Operator: "approved continue". Findings 16-20 from the chain audit, actioned.

26. preflight P2 is DC-AWARE and validates the deploy's ACTUAL overlay set

What. DC= (default vr1-dc0) selects the site; P2 assembles the same overlays the deploy uses, in the same order, each added only if present.

Why. P2 hardcoded overlays/vr1-dc0-vips.yaml --dc vr1-dc0 with NO dc1 branch, while dc-dc-phase4-juju-bundle-per-dc.md:197-204 invokes this script BARE as the dc1 gate and tells the operator to expect PASS. A dc1 operator got a green gate that had validated dc0's numbers against dc0's bands and never parsed a dc1 artifact. It also passed one overlay where the dc0 deploy passes two -- a gate validating a different input than the deploy is not validating the deploy.

Verified: DC=vr1-dc1 bash scripts/preflight.sh now reports validating the MERGED vr1-dc1 deploy input: bundle.yaml --overlay overlays/vr1-dc1-vips.yaml --overlay overlays/vr1-dc1-machines.yaml --dc vr1-dc1.

27. dc1's deploy command now names ALL its overlays

phase-4 Step 4 named ONLY the vips overlay, so a dc1 deploy run exactly as written would have merged a machines block still saying tags=openstack-vr1-dc0 -- precisely what overlays/vr1-dc1-machines.yaml exists to prevent, since one Office1 MAAS region sees BOTH DCs' nodes. Measured: grep -rn vr1-dc1-machines runbooks/ scripts/ returned ZERO -- the overlay existed in no executable path at all.

Step 4 now carries the full command, states that dc-ha-scaleup.yaml is DELIBERATELY excluded (R6: VIPs land BEFORE the HA overlay), points at DC=vr1-dc1 preflight, and adds the --dry-run machines-merge confirmation that vr1-dc1-machines.yaml's own VERIFY-LIVE note required and that no runbook step performed.

28. phase-6 deployed with NO VIP overlay, and named a file that never existed

Its two juju deploy blocks (dry-run and the real apply against a LIVE model) passed octavia-pki.yaml and overlays/${DC}-hostnames.yaml -- and no VIP overlay. Since ruling 3 that means a merged input with zero VIPs, including the designate those commands exist to add; invariant 8 would reject it, but nothing runs the checker there. And ${DC}-hostnames.yaml has never been authored -- juju errors out on a missing overlay. Both blocks now pass the per-DC vips + machines overlays, and the never-authored hostnames overlay is dropped with a note that it is unbuilt D-106 work.

29. scripts/maas-role-tags.sh -- Fork 3's per-role MAAS tags, with a harness

The blocker. Every machine in bundle.yaml constrains on TWO tags (tags=openstack-vr1-dc0,control). Measured: only the SITE tag is ever created (reenroll-hosts.sh); the three ROLE tags are created NOWHERE. Juju passes tags= to MAAS as an allocation constraint, so an unmatched tag yields no candidate machine. It fails LOUDLY at allocation -- the good failure mode -- but it blocks BOTH DCs.

Role is DERIVED from two independent signals that must AGREE (hard rule 2): the ruled D-134 octet band (control .100-.119 / compute .120-.149 / storage .150-.200) and the role token in the host's own name. Disagreement REFUSES for that node rather than picking one; an octet in no ruled band also refuses. Nodes match MAAS by pinned boot MAC, never by hostname -- MAAS renames machines at enlistment and the current names are random.

Dry by default; every write READ BACK; an ABSENT maas CLI refuses with a message that says so rather than "MAAS unreachable" (this deployment has misdiagnosed that three times).

Harness 9/9, including the two directions the audit cares about: clean when everything is tagged, red in the exact state measured today, and a REFUSAL when the two role signals are made to disagree.

NOT APPLIED. apply --commit is a live MAAS mutation and is operator-gated.

30. OWNED: the IFS word-splitting trap, third appearance

ROLES="control compute storage" with IFS=$'\n\t' does not word-split, so for r in $ROLES saw ONE item and the script reported a single absurd tag named 'control compute storage'. Caught by its own harness before shipping. Same class as commit 1's pre-flight-checks CHECK 1 death and phase-05's aborting captures. Arrays are immune; this repo's strict-bash gates should not use space-separated string lists.

Also fixed on the way: the role-derivation refusal SWALLOWED its own diagnostic -- the python printed which node and why, the command substitution captured it, and the failure path discarded it, leaving the operator "disagreed with itself" and nothing else. Now the per-node reason is printed before the refusal.

Evidence. Gauntlet ALL GREEN (87) on vcloud (was 86); repo-lint 0 fail / 614 files. preflight verdict unchanged in kind (2 fatal: the absent octavia-pki secret and MAAS unreachable from vcloud).


2026-07-29 -- the two Stage-5 blockers AUTHORED (not applied), and the Octavia question RESOLVED

31. Octavia overlay-merge: RESOLVED, and it is a clean negative

The chain audit flagged that overlays/octavia-pki.yaml lands LAST and writes the same applications.octavia.options map as the vips overlay -- so under REPLACE semantics octavia would deploy with no vip and no prefer-ipv6, and every offline gate would stay green because provider-bundle-check deep-merges and cannot discriminate.

Researched against source, not docs prose (the docs are silent on merge mechanics). juju 3.6 pins github.com/juju/charm/v12 v12.1.1 (verified in juju's go.mod). mergeStructs (overlay.go:661-693) does, for a map-kind field: for _, srcKey := range srcVal.MapKeys() { dstMap.SetMapIndex(srcKey, srcMapVal) } -- it KEEPS the destination map and writes into it per key. ApplicationSpec.Options is map[string]interface{}, so it takes that branch. DEEP-MERGE: both overlays' keys survive; octavia ends up with all seven. The 2.9 and 3.6 versions of that file are byte-identical in the merge path, so there is no version risk.

The checker's own "MIRRORS juju's documented merge" comment is therefore CORRECT -- previously an unverified claim, now sourced.

ONE REAL TRAP FOUND AND LOGGED: LP #2002371 (Triaged/High, unfixed) -- an overlay whose options: key is EMPTY/NULL does not merge, it WIPES the base map (isZero(srcVal) -> reflect.MakeMap). Also a whole-app empty body deletes the app. Our overlays do not have that shape, but nothing gates against acquiring it.

32. R1 AUTHORED: the ceph-osd data device, opt-in per role

modules/node-vm gains osd_disk_size_bytes (default 0 = creates nothing). The volume is count-gated and the disk entry is appended via concat, so a node that does not opt in has no extra resource in state and no plan diff -- that is precondition 4 of the amendment verbatim ("gate the change so it is additive and opt-in per node role, or it will plan a change against all 18 domains").

Both substrate roots: the FOUR storage nodes per DC set osd_gib = 500. Scope is 8 volumes, not 18, exactly as the amendment scoped it.

500 GiB is the budgeted figure, not a value I chose -- D-121's Option-C capacity re-check ran at "4x500Gi/DC = PASS (5.31 TiB margin)". FLAGGED: D-121 also records that the per-node OSD footprint is an ASSUMPTION whose ratified value "pends the OSD-sizing decision and should be re-derived from the measured DC0 footprint." So the size is evidenced but not separately ruled.

HEADROOM CAVEAT worth surfacing before the apply: precondition 2 measured dc0 /var/lib/libvirt at 2.0T available, and 4 x 500 GiB is 2.0 TiB nominal. qcow2 is thin so creation costs almost nothing, but dc0 has no room for those OSDs to actually fill. dc1 has 2.9T available. Not a blocker for the apply; a real one for sustained use.

33. D-104 AUTHORED: the dedicated per-DC Juju controller VM

A 10th key in each root's node map, <dc>-juju-01. for_each is keyed, so it plans 1 add / 0 change / 0 destroy -- additive, not a substrate reopening. The nine role nodes are untouched, so Option C's 3/2/4 shape, the R-3 Ceph rebuild headroom and the control-quorum spread all stand.

Sizing is the gate's own model, not an invention: 8 GiB / 4 vCPU are what dc-dc-whole-host-budget.py modelled for the PASS (854/1024 GiB = 83 pct, FIT, 170 GiB headroom). Disk 100 GiB is the authoring value the amendment explicitly left open.

MACs deliberately EMPTY. modules/node-vm documents that as correct ONLY before a node is enlisted anywhere -- which is this VM's state. They must be PINNED FROM MEASUREMENT after its first apply, exactly as the nine role nodes were; leaving them unpinned past enlistment is the 2026-07-21 MAC-regeneration trap that stranded the fleet.

34. NEITHER IS APPLIED, and precondition 3 is NOT satisfied

opentofu-validate PASS on both roots. No apply has been run. Both are live substrate mutations and are individually operator-gated.

R1 precondition 3 remains OPEN and I have not closed it: the amendment requires verifying, BEFORE the apply, that re-commissioning the storage nodes preserves the D-134 statics and the pinned MACs -- and it names the 2026-07-20 MAC-regeneration incident as the precedent for exactly this class. That verification needs live MAAS, which is absent on this host. I am not treating it as satisfied.


2026-07-29 -- dc0 SUBSTRATE APPLIED (live mutation, operator-gated)

35. R1's OSD volumes and the D-104 controller VM are BUILT on dc0

Operator approval, exact utterance: "Apply dc0 first". Executed from voffice1 (D-128 Plane 2 -- the inner tfstate lives there) via a SAVED PLAN, with a same-session pre-apply re-verify per the G8 precedent, so what was applied is exactly what was reviewed.

Apply complete: 6 added, 4 changed, 0 destroyed. Four libvirt_volume.osd + the vr1-dc0-juju-01 domain and its disk; the four storage domains updated IN-PLACE to attach vdb.

THE CHECKPOINT HELD -- MAC DRIFT COUNT 0. Verified by virsh domiflist against lib-hosts' pinned MACs BEFORE anything touched MAAS: all 9 nodes pinned==live, 6 NICs each. This is the whole point of sequencing the check there. 2026-07-20 became a fleet-wide outage because MAAS was told to re-commission while its records were already stale -- a 0/9/0 in-place apply had regenerated every MAC and the plan did not show it. So the clean plan was necessary and explicitly not sufficient. First live evidence that the 2026-07-21 MAC pinning survives an in-place domain update.

A grep that looked alarming and was not: the saved plan's MAC audit returned one hit, type_machine = "q35" inside the juju-01 CREATE block -- "mac" is a substring of "machine". Zero actual MAC lines. Recorded so nobody has to rediscover it under pressure.

Convergence restored (precondition 1): the dc0 inner root re-plans ZERO DIFF. The property the amendment said this apply would deliberately spend is back and captured.

MAAS undisturbed: 20 machines (18 Ready + 2 Deployed), 216 links, every one still mode=static. And MAAS still sees ONE block device per dc0 node -- the new vdb is in libvirt but not yet in MAAS's inventory, which is the expected state and confirms the re-commission is genuinely required rather than a formality.

Revert. tofu destroy -target on the five new resources would remove the volumes and the controller domain; the four in-place disk attachments revert by setting osd_gib back out of the storage entries and re-applying. Nothing was destroyed, so nothing is lost.

NOT DONE, gated separately: the re-commission (maas admin machine commission <sysid> skip_networking=1), and dc1 entirely.

36. Canary re-commission -- R1 precondition 3 closed by measurement

Operator: "Commission storage-01 as the canary". maas admin machine commission kghggm skip_networking=1, sysid matched by pinned boot MAC. Ready in ~200s.

The entire before/after diff is one hunk: vdb (500 GiB exactly) added to blockdevices. links 12 -> 12, MACs identical, links identical. Every static address, mode and subnet survives -- including the .150 octet mirrored across six planes in both families.

Precondition 3 was previously answered from the API's help text; it is now answered by measurement. The concern was legitimate (the default path DOES reconfigure networking) and skip_networking=1 is sufficient.

Revert. None applicable -- commissioning is idempotent and re-runnable; the node returned to Ready with its configuration intact.

Remaining: dc0 storage-02/03/04, then dc1 (apply + re-commission).