| 2026-07-29 |
voffice1 pulled current; power_type=virsh set on both controllers
...
Pull c9cc79f -> 75e3657. Both inner tfstates confirmed git check-ignore
IGNORED before the pull and sha256 byte-identical after -- the Phase-0 safety
proof, since those files are the substrate's state-of-record and live inside
the working tree.
A diff the pull revealed, explained rather than waved through: both inner
roots now plan 0 add / 1 change / 0 destroy. That is MAC ADOPTION, not drift
-- the plan adds mac = { address = ... } blocks carrying values IDENTICAL to
live, because tofu state has no mac block for a domain created unpinned.
Exactly the 2026-07-21 shape. NOT applied: a separate gated in-place update
on a running domain, which 'set power_type' did not authorise.
Power set scoped to the vr1-dcN-juju prefix so the nine already-configured
role nodes per DC were not re-touched. Both verified by a real
query-power-state returning off, which is ground truth -- a stored parameter
is not power control.
WHY power_type was not set during commissioning, answered with evidence:
(1) these machines were never commissioned; New is post-enlistment and
pre-commissioning, and those are distinct phases. (2) Even commissioned, MAAS
auto-configures power only for IPMI-based machines by probing the BMC in-band
-- per this MAAS's own CLI, 'skip_bmc_config ... for IPMI based machines'. A
KVM guest has no BMC, and the virsh parameters (qemu+ssh URI, credential,
libvirt domain name) are hypervisor facts nothing inside the guest can
discover. That is the chicken-and-egg behind 2026-07-20: commissioning needs
power control, but for a virsh VM power config cannot be discovered by
commissioning. Structural and already known; the only gap was that nobody had
run the step for two brand-new machines.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
LIVE: OSD carve + D-104 controllers complete across both DCs (two agents)
...
Operator directed two background agents: one to finish dc0 storage-02/03/04,
one for all of dc1. Both completed.
Eight re-commissions, every one the same single-change diff: vdb at
536870912000 bytes (500 GiB) added; links 12 -> 12; per-interface MACs and
link lists identical. dc0 at 204/196/202s, dc1 all ~182s. dc1's substrate
applied 6 add / 4 change / 0 destroy with MAC drift 0 and re-converged to
ZERO DIFF.
I verified every claim from MAAS directly rather than accepting the reports:
fleet 22 (18 Ready + 2 Deployed + 2 New), 216 links all static, both DCs
blockdevs {1:5, 2:4}, tagcount {2:9}. All matched. I also diffed a raw dc1
before/after pair myself -- exactly the vdb hunk plus one power_state line.
Captures promoted from voffice1:/tmp (which does not survive a reboot) to
docs/audit/osd-carve-20260729/. Both agents were barred from repo writes, so
dc1 would otherwise have had no audit record at all.
Two unprompted agent findings worth keeping: MAAS tags SURVIVED
re-commissioning on all 18 nodes (checked because the snapshot code
structurally could not see tags -- removes a would-be Stage-5 placement
blocker); and lib_hosts_select_dc's one-DC-per-shell guard fires correctly
but can be muted with 2>&1, silently yielding one DC's data labelled as the
other's.
One disclosed judgment call, and it was correct: three dc1 pairs also showed
power_state moving off/error. Agent B declared a STOP, re-measured, then
excluded power_state from its verdict while still printing it, because 8 of 9
dc1 nodes read error BEFORE any mutation. Vindicated -- all 18 role nodes now
read off, including five dc1 nodes never commissioned. A transient MAAS
power-query flap.
Still blocking for both controllers: power_type is EMPTY, the state that
stalled all nine role nodes on 2026-07-20. Now fixable since both are in
lib-hosts at the ruled .5 with MACs pinned.
Gauntlet ALL GREEN (87) on vcloud; repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Pin juju-01 MACs, add both controllers to lib-hosts at the ruled .5
...
Twelve measured MACs (six per DC, virsh domiflist) pinned into both
substrate roots; <dc>-juju-01 added to lib-hosts.sh with HOST_OCTET=5 per
the D-134 amendment and its measured boot MAC. Done now because both VMs are
ENLISTED -- the boundary modules/node-vm warns about, past which an in-place
apply can regenerate unpinned MACs and strand the node, as happened to the
nine role nodes on 2026-07-21.
No race with the running agents: they operate on the voffice1 clone; this
edit is on vcloud's, and nothing reaches their tree until someone pulls
there.
Recorded rather than smoothed over: dc1's juju-01 MACs are libvirt-generated
52:54:00: values, not dc1's schematic 52:54:01:d1: scheme -- a direct
consequence of the deliberate macs = [] at first apply.
Three exact-count assertions re-pointed to the new invariant and kept EXACT
rather than relaxed: dc-selector fleet 9 -> 10 (label corrected, since Option
C is a 9-node ruling and the 10th is D-104's), node-vm T8 9 -> 10 and T9
54 -> 60.
A real harness defect surfaced doing it: T8's grep -c 'macs = [' counts
COMMENTS, and my own comment explaining the deliberate macs = [] inflated the
count to 11 -- which reads exactly like a spurious extra node. Fixed to strip
comment lines. Same class as the ledger-scan next-free defect where a doc
merely quoting a token inflated the counter.
maas-role-tags.sh taught the utility band: adding juju-01 to HOSTS made it
refuse, because .5 is in no ROLE band -- correct behaviour meeting a new
fact. Per the amendment a utility-band host with no role token is now SKIPPED
and REPORTED, while the refusal keeps its teeth for genuine disagreement.
New cases T10 and T11; harness 9 -> 11.
Gauntlet ALL GREEN (87) on vcloud; opentofu-validate PASS; repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
LIVE: canary re-commission closes R1 precondition 3 by measurement
...
Operator: 'Commission storage-01 as the canary'. maas admin machine
commission kghggm skip_networking=1, sysid matched by pinned boot MAC never
by name. Commissioning -> Ready in ~200s, no timeout, no SERVFAIL.
The full before/after diff of status, power, block devices, interfaces, MACs
and links is EXACTLY ONE HUNK: vdb (536870912000 bytes = 500 GiB) added to
blockdevices. links 12 -> 12; MACs identical; links identical. All twelve
static links survive with addresses, modes and subnets intact, including the
D-134 octet .150 mirrored across all six planes in both families -- the v6
host part ::150 being the 2026-07-27 D-136 mirror ruling visible in live
data.
So skip_networking=1 preserves the network EXACTLY while storage is still
re-scanned, which is the entire point. Precondition 3 was previously answered
from the API help text; it is now answered by measurement. The amendment's
concern was legitimate -- the default path does reconfigure networking -- and
the vendor's own control is sufficient.
Remaining: dc0 storage-02/03/04, then all of dc1.
Capture: docs/audit/dc0-canary-commission-20260729.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
LIVE: dc0 substrate applied -- R1 OSD volumes + the D-104 controller VM
...
Operator approval: "Apply dc0 first". Executed from voffice1 via a SAVED
PLAN with a same-session pre-apply re-verify (G8 precedent), so what was
applied is exactly what was reviewed.
Apply complete: 6 added, 4 changed, 0 destroyed. Four libvirt_volume.osd +
the vr1-dc0-juju-01 domain and its disk; the four storage domains updated
in-place to attach vdb.
THE CHECKPOINT HELD -- MAC DRIFT COUNT 0. Verified by virsh domiflist
against lib-hosts' pinned MACs BEFORE anything touched MAAS: all 9 nodes
pinned==live, 6 NICs each. That sequencing is the point. 2026-07-20 became a
fleet-wide outage because MAAS was told to re-commission while its records
were already stale, after a 0/9/0 in-place apply regenerated every MAC
without the plan showing it. So the clean plan was necessary and explicitly
not sufficient. First live evidence that the 2026-07-21 MAC pinning survives
an in-place domain update.
Convergence restored (precondition 1): the dc0 inner root re-plans ZERO
DIFF. MAAS undisturbed -- 20 machines (18 Ready + 2 Deployed), 216 links,
every one still mode=static. MAAS still sees ONE block device per dc0 node,
which is expected and is why the re-commission is genuinely required.
Also recorded: the saved plan's MAC audit returned one hit that was a false
positive -- type_machine = "q35" inside the juju-01 create block, since
"mac" is a substring of "machine".
NOT DONE, gated separately: the re-commission with skip_networking=1, and
dc1 entirely. Capture: docs/audit/dc0-osd-juju-apply-20260729.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Author both Stage-5 blockers (NOT applied); Octavia merge question resolved
...
Octavia RESOLVED as a clean negative. The chain audit flagged that
octavia-pki.yaml lands last and writes the same applications.octavia.options
map as the vips overlay, so under replace semantics octavia would lose its
vip and prefer-ipv6 with every offline gate still green. Researched against
source: juju 3.6 pins charm v12.1.1, and mergeStructs does
dstMap.SetMapIndex(srcKey, srcMapVal) per key for map-kind fields.
ApplicationSpec.Options is a Go map, so it deep-merges -- both overlays' keys
survive. The merge path is byte-identical between 2.9 and 3.6. The checker's
"MIRRORS juju's documented merge" comment is now sourced rather than assumed.
Logged: LP #2002371 (Triaged/High, unfixed) -- an EMPTY/null options: key
WIPES the base map rather than merging. Our overlays lack that shape and
nothing gates against acquiring it.
R1 AUTHORED. modules/node-vm gains osd_disk_size_bytes, default 0 = creates
nothing; the volume is count-gated and the disk entry appended via concat, so
an opted-out node has no extra resource and no plan diff -- precondition 4
verbatim. Four storage nodes per DC set osd_gib = 500: 8 volumes, not 18.
500 GiB is D-121's own budgeted figure (4x500Gi/DC = PASS, 5.31 TiB margin),
and D-121 records that footprint as an assumption pending an OSD-sizing
decision, so it is evidenced but not separately ruled. Headroom caveat
surfaced: dc0 has 2.0T available against 2.0 TiB nominal -- thin-provisioned,
so creation is cheap, but there is no room for those OSDs to fill.
D-104 AUTHORED. A 10th keyed entry per root, <dc>-juju-01, planning 1 add /
0 change / 0 destroy. The nine role nodes are untouched. 8 GiB / 4 vCPU are
the values the capacity gate actually modelled; disk 100 GiB is the authoring
value the amendment left open. MACs deliberately empty -- correct only before
enlistment, and they must be pinned from measurement after the first apply or
this reproduces the 2026-07-21 MAC-regeneration trap.
NEITHER IS APPLIED. opentofu-validate PASS on both roots; no apply run. Both
are live substrate mutations and individually operator-gated.
R1 precondition 3 is NOT satisfied and I am not treating it as such: the
amendment requires verifying BEFORE the apply that re-commissioning preserves
the D-134 statics and pinned MACs, and that needs live MAAS, absent here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Chain-audit deploy path: DC-aware preflight, dc1's real overlay set, Fork-3 role tags
...
Findings 16-20 from the D-136 chain audit, actioned.
preflight P2 is DC-AWARE and validates the deploy's ACTUAL overlay set. It
hardcoded --dc vr1-dc0 with no dc1 branch while dc-dc-phase4 invokes it BARE
as the dc1 gate and tells the operator to expect PASS -- so a dc1 operator
got a green gate that validated dc0's numbers against dc0's bands and never
parsed a dc1 artifact. It also passed one overlay where the deploy passes
two. DC=vr1-dc1 now merges vips + machines for that site.
dc1's deploy command names ALL its overlays. phase-4 Step 4 named only the
vips overlay, so a dc1 deploy run as written merged a machines block still
saying tags=openstack-vr1-dc0 -- grep for vr1-dc1-machines across runbooks
and scripts returned ZERO, i.e. that overlay was in no executable path. The
step now carries the full command, states dc-ha-scaleup is deliberately
excluded per R6, and adds the --dry-run machines-merge confirmation the
overlay's own VERIFY-LIVE note required and no step performed.
phase-6 deployed with NO VIP overlay and named a file that never existed.
Both blocks -- including the real apply against a live model -- passed
overlays/${DC}-hostnames.yaml, never authored, and no VIP overlay;
post-ruling-3 that is a merged input with zero VIPs including the designate
those commands exist to add.
scripts/maas-role-tags.sh ships Fork 3 (harness 9/9). Every machine
constrains on two tags; only the site tag is ever created, so the three role
tags exist nowhere and juju's allocation constraint matches no machine.
Role is DERIVED from two signals that must agree -- the ruled D-134 octet
band and the host's own name token -- and disagreement REFUSES rather than
picking one. Nodes match by pinned boot MAC, never hostname. Dry by default,
every write read back, absent CLI distinguished from unreachable service.
apply --commit is a live MAAS mutation and is NOT run here.
OWNED, third appearance of the IFS trap: ROLES="control compute storage"
under IFS=$'\n\t' does not word-split, so the script reported one absurd tag
named 'control compute storage'. Caught by its own harness pre-ship. Same
class as commit 1's CHECK 1 death and phase-05's aborting captures -- these
strict-bash gates should use arrays, never space-separated string lists.
Also fixed: the role-derivation refusal swallowed its own diagnostic.
Gauntlet ALL GREEN (87) on vcloud; repo-lint 0 fail / 614 files; preflight
verdict unchanged in kind.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-136 step 6: the chain audit -- R9's drift gate built, three refusals, two dead cases
...
Three read-only lenses over data source -> renderer -> deployment mechanism,
the ruled final step of D-136 option (D). Lens 3 was adversarial and made
the chain produce wrong-but-green THREE times, all executed.
All three converged on one root cause: dc1's overlay was byte-gated and
dc0's was gated by nothing. render/values/ was referenced by zero scripts
and zero harnesses, while preflight P2 feeds the deploy gate dc0's overlay.
R9's render-drift check had been RULED and never built.
Executed wrong-but-green:
- transposing two apps' octets in the dc0 overlay (consistent, in-band,
correct prefixes) -> PASS
- changing ONE WORD, family: dual -> v4, silently reverting dc0 to IPv4-only
against R2's ruling -> PASS, because dropping prefer-ipv6 and the v6 legs
together satisfies invariant 9, and vip_dual is printed but never asserted
tests/render-drift/ is R9's gate, built. It iterates render/values/*.yaml,
reads each file's own output: field (inert until now), renders and
byte-compares. A new per-DC values file is gated the moment it is committed.
Non-zero floor plus a proof-of-teeth case that mutates a copy and requires
red.
Three refusals where the chain took the last writer silently:
- duplicate (role, kind) in the apex, in BOTH readers. Appending one spare
prefix -- routine IPAM work -- re-homed every dc0 VIP into a different /64
and went green, because the gate checks overlay-vs-apex agreement and both
sides read the same wrong value. The winner depended on JSON array order.
- duplicate app name / duplicate octet / out-of-band octet in APP_OCTET. A
duplicate name emitted two YAML blocks, safe_load kept the last, and one
app's ruled octet vanished with nothing red.
Two harness cases that could not fail:
- T3b's byte-compare was CIRCULAR -- header derived --from-overlay $LIVE
then compared --against $LIVE, so a third of the bytes were self-copied.
Proven by rewriting a header line to a false claim and still matching.
- T29's own fixture was inert; its assertion held from the overlay alone.
OWNED: my first T29 re-point asserted a --show-legs flag THAT DOES NOT
EXIST. I invented an interface instead of reading one. The harness caught it
immediately; replaced with real behaviour and recorded rather than quietly
fixed.
Gauntlet ALL GREEN (86) on vcloud; repo-lint 0 fail / 612 files. The larger
deploy-path half is logged not actioned -- see the changelog, items 16-25.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-109 ruling note: the Octavia controller cert gains IPv6 IP SANs
...
Operator ruling (GA-R5), exact utterance: "Yes, add the v6 IP sans."
Recorded as a D-109 RULING NOTE (2026-07-29); OPS under GA-R3 -- a
generator change extending an already-ruled principle to a surface D-109
governs -- so no new D-number and next-free stays 138.
The generator now emits IP.1 = this DC's provider v4 leg and IP.2 = its
provider v6 leg, both derived from the same per-DC overlay under R7's $DC
selector. Verified: dc0 -> 10.12.4.57 + 2602:f3e2:f02:11::57; dc1 ->
10.12.64.57 + 2602:f3e2:f03:11::57; and a v4-only control emits NO v6 SAN,
so the generator stays correct on a tree where the dual-stack ADD has not
landed. Admin and internal legs stay EXCLUDED exactly as before --
DOCFIX-067's design has always been provider-leg-only.
Two repo guards fired and both were right: repo-lint L5 rejected a heading
that LED with the D-number (it reads as a second definition -- the third
time this trap has bitten on this branch), and L10 rejected the change for
touching a design-decisions Status line without CURRENT-STATE in the same
commit.
Observation logged, not acted on: amphorae reach the controller on o-hm0's
charm-generated fc00::/64 ULA, not on any VIP leg, so neither SAN matches
that path. Measured upstream, the controller verifies the amphora by UUID;
the reverse direction was not established from source. Settle by inspection
at the Octavia step rather than assuming.
Gauntlet ALL GREEN (85) on vcloud; repo-lint 0 fail / 611 files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Octavia: R8a BUILT (record was false), two stale surfaces, R7 generator de-frozen
...
Follows an upstream/vendor research pass answering "can Octavia run full
IPv6" -- yes; nothing inside Octavia requires IPv4. charm-octavia creates
the lb-mgmt subnet with ip_version: 6 and has no IPv4 code path at all; the
health manager is v6-capable both directions; the amphora agent binds '::'
and verifies certs by amphora UUID, not IP; and Nova metadata is not a
dependency (config_drive=True). Surviving v4 pressure is soft or external.
R8a BUILT. phase-05-octavia-verify.sh now compares o-hm0's MTU against
lb-mgmt-net's (resolved BY TAG, never by name), fails in either direction,
and REFUSES when a value cannot be read. This closes the LP#2018998
obligation, live on our exact pin after its 2025-12-31 recurrence.
THE DECISION RECORD HAD CLAIMED THIS SHIPPED. design-decisions.md carried a
present-tense "Delivery: the assertion ships in ..." written on the day of
the ruling, while the file held zero MTU references and its last commit
predated the ruling by a month. Corrected in place. Sharper than the usual
ruled-but-not-built class: the decision doc was the false witness.
TWO OWNED ERRORS:
- I concluded "no octavia harness exists" from a NAME grep. The script's
harness has always been tests/phase-05. Duplicate deleted, R8a cases
folded into the real one.
- My first MTU draft ABORTED the script: a $( ) under set -euo pipefail with
inherit_errexit exits the run rather than reaching the refusal branch. The
pre-existing harness I had just declared nonexistent caught it. Same class
as commit 1's IFS word-splitting trap; both now locked by cases.
Two live surfaces still contradicted R8 -- dc-dc-phase4 and the workflow doc
both called lb-mgmt IPv6 "a real, open risk" and recommended keeping it
v4-only, citing the two refuted bugs. A reader would have re-litigated a
closed ruling in the wrong direction. Both superseded in place.
R7 executed: phase-01 Step 1.0-GEN gains a $DC selector; the baked
/CN=VR0 DC0 CA subjects derive from it, and the VIP gate derives this DC's
provider prefix from the same overlay it read the VIP from -- R7's explicit
"read the MERGED input" caveat. dc1's 10.12.64.57 used to hard-ABORT, so no
dc1 artifact could be produced. Verified both DCs, negative control still
aborts.
Logged not built: the controller cert's CN/DNS SANs still carry dc0.vr0
(outside R7's ruled scope, inert while os-public-hostname is unset), and it
gains no IPv6 IP SAN though the VIP is dual-family -- a gap, not a break,
and adding v6 SANs needs its own ruling.
tests/phase-05 14 cases (was 9); gauntlet ALL GREEN (85) on vcloud;
repo-lint 0 fail / 611 files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Ruling 3 commit 2: dual-stack ADD -- R2 and R11 are now BUILT, L3-9 resolved
...
Both per-DC VIP overlays re-rendered dual-family: every app carries
prefer-ipv6: true and a six-address vip, and vault .61 + designate .62 are
built. Measured both DCs: 13 clustered VIPs (13 dual-family) + 12 hacluster
principals all carrying a VIP -> PASS. With dc-ha-scaleup stacked: 13
principals, PASS -- R6's ruled ordering (VIPs before the HA overlay) is now
satisfied and executable. Two ruled-but-never-built decisions became
artifacts in one pass.
L3-9 RESOLVED. dc-dc-ipv6-family-matrix.yaml is narrowed to ceph-mon only;
its ten duplicate vip + prefer-ipv6 pairs (every value an unrendered token)
are gone, removed TOGETHER -- removing the vips alone would have recreated
the defect deliberately. Both merge orders now PASS, i.e. order-independent,
which is the actual fix. ceph-mon's ULA-only networks survive because they
exist in no other file. The two Launchpad citations arguing octavia lb-mgmt
IPv6 was an open risk are removed: both were read in full and both failed,
and R8 closed that question on 2026-07-27.
Also corrected at source: CURRENT-STATE described L3-9 as "the dangerous
merge order is the one that PASSES". Measured -- that stopped being true
when invariant 9 shipped on 2026-07-28. The merge semantics it describes
remain accurate; only the green-over-loss conclusion is retired.
Gate repairs the agents caught BEFORE the edit:
- phase-01's deploy guard would have ABORTED THE DEPLOY AGAIN -- its HI regex
(5[0-9]|60) excludes .61/.62, so it would have read 13/11/0 against a
required 11/11/0. Widened, counts moved to 13, verified 13/13/0 -> DEPLOY.
Second near-miss on this one guard across two commits.
- pre-flight-checks CHECK 1 hardcoded if(n!=3) and would have failed all 13.
Now takes a triple OR a sextet and asserts a sextet's last three legs are
really v6. Proven able to fail on both new paths.
Harnesses re-pointed, never deleted. T16/T17 asserted R11's gaps were OPEN;
inverted to the surviving invariant and PAIRED with new T16b/T17b that remove
a VIP and demand the check still fires. T20/T21 had become no-ops that would
have gone GREEN testing nothing -- inverted. A v4-only twin fixture keeps the
pre-dual-stack cases testing their own invariants. The renderer harness now
proves reproduction on both shapes, including new T3b against the live
dual-family overlay.
provider-bundle-check 33/33; render-dc-overlays 18/18; render-baseline 10/10;
preflight 16/16; gauntlet ALL GREEN (85) on vcloud; repo-lint 0 fail / 611.
PREFLIGHT: BOTH STANDING REDS CLEARED, 3 fatal -> 2 fatal. The remaining two
are the deliberately-absent octavia-pki overlay and MAAS unreachable from
vcloud, which R10 measured clears on voffice1.
FLAGGED, not hidden: octavia's dual-family API VIP is an inference from R2
applied evenly, not a quoted ruling. Worth one operator confirmation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Ruling 3 commit 1: bundle.yaml is VIP-free; dc0's VIPs are now a RENDERED overlay
...
The 11 inline vip: lines are gone from bundle.yaml and live in
overlays/vr1-dc0-vips.yaml, GENERATED by scripts/render-dc-overlays.py from
render/values/vr1-dc0-vips.yaml -- the renderer's first real output.
placement's then-empty options: block was dropped rather than left parsing
as null. Ruling 3's dual-stack ADD is commit 2 and is NOT in here.
PROVEN NEUTRAL by the ruling's own named check: provider-bundle-check on the
pre-extraction bundle and on the merged pair produce line-for-line identical
output, same known designate failure included. A structural diff separately
confirmed only vip moved -- applications 56 -> 56, relations 108 -> 108,
every other key byte-identical.
The 11 inline comments were carried, not dropped: extracted verbatim by
parser rather than retyped, via new optional per-app comment support in the
renderer. An app with no comment emits none, so dc1 still reproduces
byte-for-byte at 1634 bytes. The B1/B5 tokens stay DEFINED in bundle.yaml's
header and the overlay header names that, so the references do not dangle.
The blast-radius repair is the bulk of this, and every item was found BEFORE
the edit by the read-only agent sweep:
- preflight.sh P2 validated the bare base (11 phantom failures post-move) ->
merged input.
- pre-flight-checks.sh CHECK 1 read the base as raw text and would have seen
ZERO VIPs. This is BLOCKER-1, fixed the ruled way and not by bumping the
count. Measured trap: the file sets IFS=$'\n\t', so a space-joined string
reached grep as ONE filename, grep exited 2, pipefail propagated and the
gate died silently mid-check. Fixed with a bash array.
- phase-01's RUN block gates juju deploy on 11/11/0 and would have read
0/0/0 and ABORTED THE DEPLOY. Both guards now span base + overlay, and a
second latent bug was fixed in the same edit: grep -c over two files prints
one count PER FILE, so the bare $( ) captured a two-line string and every
numeric test would have failed anyway. Now awk-summed; verified 11/11/0.
- phase-01's Octavia SAN derivation raised KeyError, swallowed by || true
into an empty VIP. Now reads the overlay, deriving 10.12.4.57.
- phase-00 teardown, dc-dc-phase4's "NO VIP edits" guidance, appendix-A L3,
and render-baseline's now-unverifiable bundle hash all repaired.
Harness 30 -> 31. The fixture base is now bundle + dc0 overlay, since that
pair is the deploy input; T16 renamed to match. NEW T31 asserts the bare base
fails for every clustered principal, so "deployed without its overlay" is a
tested state rather than an assumption.
Gauntlet ALL GREEN (85) on vcloud; repo-lint 0 fail / 610 files scanned.
Still red and expected: CHECK 1 at OK=11 want 13, and designate -- R11's
.61/.62 remain ruled-but-not-built, which is commit 2's job.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Render-pipeline step 4: the renderer, reproducing a reviewed artifact byte-for-byte
...
scripts/render-dc-overlays.py. Two stages, split exactly where forward item
F2 froze the interface: `derive` (apex + lib-net.sh -> a per-DC values file)
and `render` (values -> overlay text). `render` is PURE -- no apex, no
network, no clock -- so it is offline-testable and a future CI job re-runs
only `derive`.
ORDER INVERTED BY MEASUREMENT. The plan was ruling-3's VIP extraction first.
Three read-only agents measured two facts that reversed it: the byte-for-byte
reproduction window is still open (the live dc1 overlay is still identical to
the frozen fixture) and closes on its own when the R2/R11 reconciliation
lands; and the extraction's blast radius is far larger than ruling 3's "2
commits" -- it breaks the phase-01 deploy gate (juju deploy runs only on
11/11/0), the Octavia SAN derivation, preflight in two places, and 8 of 30
harness cases. Better done BY the renderer, so dc0's overlay is a validated
output rather than a hand edit.
A TEXT emitter, not yaml.dump: a round-tripper destroys the comment header,
normalises the quoted vip scalar, and imposes its own key order -- all three
are load-bearing bytes. Two things are INPUT rather than invented: the 12-line
comment header (lifted verbatim via --from-overlay, so a reproduction run is
honest about which bytes the tool actually generates) and the ascending-octet
emission order (sorting by name reproduces nothing). The ruled app->octet map
is READ with ast out of dc-plane-apex-import.py rather than restated as a
second drifting copy.
RESULT: output hashes to 3f93ecb3..., identical to both the committed overlay
(1634 bytes) and the frozen fixture. PROVEN ABLE TO FAIL, not merely observed
passing -- three seeded faults all fail the compare (wrong octet, dropped
header, dc0 prefixes against the dc1 artifact).
THE ARITY WORK PAID FOR ITSELF WITHIN THE HOUR. The first dual-family output
was malformed: the values file stores each v6 /64 base with trailing colons
stripped, and the emitter joined with a single colon -- 2602:f3e2:f03:11:50,
not a valid address, and it still looks like one. provider-bundle-check,
fixed yesterday for exactly this class, rejected all 13 applications. Locked
by T13/T13b.
End to end: apex -> values -> renderer -> overlay -> gate. The dual-family
dc1 render with R11's ruled-but-unbuilt apps produces 13 VIPs, all
dual-family, PASSING provider-bundle-check --dc vr1-dc1 including the new
hacluster invariant. Nothing committed has been regenerated by it yet.
tests/render-baseline gains T6, the reproduction case its README invited --
ADDED, with T1-T5 deliberately not repointed at live, and comparing against
the FIXTURE because live is supposed to diverge. T6 proven able to fail by
making the renderer sort alphabetically.
New harness 17/17; render-baseline 10/10 (was 9); gauntlet ALL GREEN (85) on
vcloud, manifest re-recorded after the drift gate correctly caught the
addition; repo-lint 0 fail / 608 files scanned.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-28 |
Exercise the dc1 arm of the apex lookup; confirm P2's rc=2 classification
...
T19-T23 all ran at the default --dc vr1-dc0, so the dc1 v6 bands
(f03:11::/64, :320::/64, :321::/64) were resolved by no case at all. The
renderer's first real output is per-DC overlays, and a gate observed in
only one DC mode is the same only-ever-seen-one-way untrustworthiness
T16/T18 exist to rule out.
T29 resolves the dc1 bands and passes. T30 proves a dc0 v6 leg under
--dc vr1-dc1 FAILS, so a cross-DC copy-paste of a rendered overlay
cannot pass -- the v6 analogue of T15.
Also verified rather than assumed: the checker's NEW exit-2 refusal
(dual-family vip + unreadable apex) is classified correctly by preflight.
P2 rc=2 is already ruled could-not-evaluate -> FAIL (preflight.sh:36,81)
and harness case T14 locks it, so a refusal can never downgrade to a
warning. No change needed there.
Harness 28 -> 30. Gauntlet ALL GREEN (84) on vcloud; repo-lint 0 fail /
605 files scanned.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
VIP arity gap CLOSED + R11's three ruled gate changes executed
...
provider-bundle-check.py could not express what R2 and R11 already ruled:
:137 required exactly 3 addresses per vip (a dual-family VIP is 6, so it
would have failed EVERY application) and :149's octet extraction returned
the whole string on a v6 literal, leaving every v6 leg trivially unique
and unchecked. The apex now holds the dual-family set a renderer will
emit, so the only gate that validates overlays could not validate what is
about to be produced.
Arity: a vip is now a v4 triple OR a dual-family sextet. The v6 legs are
validated against the per-DC v6 /64s READ FROM THE APEX RECORD (D-136
option (D)), reusing dc-plane-ipam.sh's (role, kind) keying -- the
provider leg takes the dedicated GUA VIP /64, admin/internal their plane
/64s. The v6 host part must MIRROR the v4 octet textually. prefer-ipv6
and the v6 legs are COUPLED in both directions: the measured L3-9 finding
is that the merge order which keeps prefer-ipv6 while dropping the v6
legs is the one that exits 0. An unreadable apex REFUSES at exit 2; a
v4-only bundle needs no apex.
R11 (D-020 amendment, ruled 2026-07-27), same pass: band 50-60 -> 50-99
in both separately-named sites; VIP_COUNT_EXPECT 11 -> 13;
EXPECT_PUBLIC_VIP deliberately stays 11 (measured -- neither vault nor
designate has a public binding); and an hacluster principal with no vip
now FAILS, since cluster_count is asserted nowhere and a 3->1 rewrite of
all 20 values produces a byte-identical PASS.
Measured consequence, not glossed: two sub-checks flip PASS -> FAIL
(designate's missing VIP; CHECK 1 at OK=11 want 13). preflight was
ALREADY exit 1 before this change and is still exit 1 after -- no deploy
path that was open is closed. Both reds are the ruled gate reporting real
work owed; per R6 the .61/.62 VIPs land before the HA overlay.
Harness 15 -> 28. Three cases RE-POINTED rather than deleted: the fixture
base splits into a pristine repo bundle and one carrying designate's
ruled .62, with new case T16 asserting the pristine bundle DOES trip the
new invariant. When .62 lands, T16 gets re-pointed, not deleted. Proven
able to both fail and pass on the same check (T16 red / T18 green).
Gauntlet ALL GREEN (84) on vcloud; repo-lint 0 fail / 604 files scanned.
Scope was operator-gated: "Fix the arity gap first, then start the
renderer", with the arity-alone vs arity-plus-R11 fork put separately and
answered "Arity gap + R11's three ruled changes (Recommended)". OPS under
GA-R3, no D-number.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
