G19 -- IPv6 PLANE GATE: BUILD CAPTURE 2026-08-01
Tool: scripts/dc-node-v6-verify.sh Harness: tests/dc-node-v6-verify/run-tests.sh
Branch dc-dc-stage5-preconditions at 9853cb7. Nothing in this session mutated the
cloud: every live command below is read-only (virsh net-dumpxml, sysfs reads, one
refused MAAS read). No MAAS write, no power op, no deploy.
DOCS NOT TOUCHED BY DESIGN: docs/CURRENT-STATE.md and docs/design-decisions.md are
the operator's; the G19 gate row is theirs to write. See "repo-lint" below -- that
split is exactly what L10 fires on, and the finding is expected, not a defect.
1. THE MANDATE, AND WHAT THE TOOL ACTUALLY ASSERTS
--------------------------------------------------------------------------------
Two DIFFERENT assertions, and the split into subcommands follows the split in the
question:
(1) THE ADDRESSES ARE UP, asserted ON THE INTERFACE. `node`
Per plane: the interface exists and has LOWER_UP; the address MAAS holds is
present with scope global; DAD is complete (not tentative, not dadfailed);
and it is the SOLE global on that NIC.
(2) THE PLANE CARRIES v6 BETWEEN TWO NODES. `node`, same pass
ping -6 to a PEER node's address on the same plane, THEN read `ip -6 neigh`
-- in that order, because the ping is what populates the cache the second
reading inspects. All four quadrants of (reply x neighbour-resolved) are
distinguished and each names a different next step.
DIAGNOSTIC, no verdict. `bridges` (runs on the RACK, not the node)
Prints each plane bridge's multicast_snooping / multicast_querier.
EXPECTATION SOURCE. `plan` (runs where the maas CLI is)
Derives the per-node expectation from live MAAS and prints one SPEC line per
machine. Asserts MAAS-side consistency only.
WHAT A GREEN DOES NOT MEAN, and it is in the script header so nobody has to find
this file: G19 asserts the NIC AGAINST MAAS, not against D-139's ruled GUA table.
Green means "the carve MAAS holds is actually up, whatever prefix that is".
MAAS-vs-ruling is dc-node-v6-carve.py check and tests/d139-gua-carve. `plan`
PRINTS the derived prefixes so a reader of any capture sees at a glance whether
the ULA or the GUA carve is the one being asserted.
2. HOW THE PREFIX-SOURCE CONSTRAINT WAS RESOLVED, AND WHY
--------------------------------------------------------------------------------
CONSTRAINT: D-139 (ADOPTED, confirmed by the 2026-08-01 ruling note) retires
fd50:840e:74e2::/48 for VR1 and moves every plane to 2602:f3e2:f02::/48 (dc0) /
2602:f3e2:f03::/48 (dc1). A site table with ULA literals is correct today and
wrong within days.
CHOSEN: DERIVE FROM LIVE MAAS, and pass the result to the node as INPUT.
* `plan` reads `maas <profile> machines read`, selects machines by the tag
openstack-<site>, and for every link whose subnet CIDR is v6 emits
iface=subnet_cidr=addr=peer=peerhost.
* `node` takes that SPEC as argv[3]. It contains NO prefix knowledge of its own.
* ZERO v6 prefix literals exist anywhere in the executed code. Harness T3
enforces this by grepping non-comment lines for fd..:/2602:/fc00::; T4 does the
same for v4 plane literals. T14 drives the SAME green plan case with a GUA
fixture and a ULA fixture -- the argument is executed, not asserted in prose.
WHY MAAS AND NOT THE NETBOX APEX. The gate's job is precisely the recorded lesson
"MAAS holding a link is not the same as an interface carrying it": MAAS is the
authority for WHAT SHOULD BE ON THE NIC, and the NIC is the authority for what is.
Using the apex instead would fold two independent failures (apex->MAAS drift and
MAAS->NIC drift) into one red, which is the failure mode this gate is built to
avoid. Apex-vs-MAAS already has its own owner.
WHY NOT AN OPERATOR-SUPPLIED PREFIX TABLE. It would need re-editing on carve day,
which is the exact rot the mandate names.
CORRECTION TO THE MANDATE'S PREMISE, and it matters. The mandate states
dc-node-v6-carve.py "derives them from the NetBox apex per D-136 option (D)". It
does not. Read at scripts/dc-node-v6-carve.py:96-137: it derives the v6 subnet from
LIVE MAAS -- the v6 subnet sharing a MAAS VLAN with the interface's EXISTING v4
link -- and D-136 option (D) is the separate ruling to mirror the v4 octet TEXTUALLY.
CONSEQUENCE, and this is an owed fix that is NOT mine to make under hard rule 1:
that v4-pivot BREAKS under D-139. Five planes lose their v4 subnet entirely, so
"the interface that already carries a v4 static" selects nothing on
metal-internal / data-tenant / storage / replication, and lb-mgmt never had a v4
twin at all. dc-node-v6-carve.py will silently carve four fewer planes per node
after the v4 removal step. LOGGED, not executed.
G19 does NOT inherit that defect: `plan` pivots on "the link's subnet CIDR is
v6", which needs no v4 twin.
3. THE FAILURE MODE THIS GATE EXISTS TO CATCH -- MEASURED, BOTH RACKS
--------------------------------------------------------------------------------
Measured 2026-08-01 by running the tool itself, read-only, over the intended
`ssh <rack> 'bash -s -- bridges <site>' < scripts/dc-node-v6-verify.sh` path.
ssh vr1-dc0-rack: ssh vr1-dc1-rack:
vr1-dc0-data-tenant virbr4 1 / 0 SUSPECT vr1-dc1-data-tenant virbr1 1 / 0
vr1-dc0-metal-admin virbr2 1 / 0 SUSPECT vr1-dc1-metal-admin virbr6 1 / 0
vr1-dc0-metal-internal virbr6 1 / 0 SUSPECT vr1-dc1-metal-internal virbr5 1 / 0
vr1-dc0-provider-public virbr5 1 / 0 SUSPECT vr1-dc1-provider-public virbr4 1 / 0
vr1-dc0-replication virbr1 1 / 0 SUSPECT vr1-dc1-replication virbr2 1 / 0
vr1-dc0-storage virbr3 1 / 0 SUSPECT vr1-dc1-storage virbr3 1 / 0
vr1-dc0-wan br-vr1-dc0-wan 1 / 0 SUSPECT vr1-dc1-wan br-vr1-dc1-wan 1 / 0
(numbers are multicast_snooping / multicast_querier; exit 0 = reported, 7 bridges)
This INDEPENDENTLY CONFIRMS the dc0 reading the mandate reports, and EXTENDS it:
dc1 is identical, and so is the WAN bridge. The jumphost's own two transit nets
(vr1-dc0-uplink virbr4, vr1-dc1-uplink virbr1) read the same -- so this is the
libvirt default across every bridge in the estate, not a per-rack setting.
NO LINUX BEHAVIOUR IS ASSERTED, here or in the tool. The mandate said the exact
behaviour is unconfirmed and should not be asserted; the subcommand REPORTS the two
values, tags the 1/0 combination "SUSPECT", and prints a NOTE saying in terms that
it diagnoses nothing. What makes it worth having is that a red assertion (2) can be
triaged in one step instead of three -- and `node`'s no-reply + INCOMPLETE branch
names this subcommand by its exact command line.
WHY IT SITS NEXT TO ASSERTION (2): D-139 records all six v6 subnets at
gateway_ip=None, so peers are on-link and Neighbour Discovery -- which is
multicast -- IS the resolution mechanism on these planes.
CARRIED, NOT CLOSED: nothing here shows the plane FAILS. No node currently carries
these addresses up (Stage 5 deploy is in flight), so assertion (2) has never run
against real traffic. The instrument is built; the experiment is the operator's.
4. WHAT WAS RUN LIVE, AND WHAT THAT PROVED
--------------------------------------------------------------------------------
bridges vr1-dc0 / vr1-dc1 on both racks exit 0, 7 bridges each (section 3)
bridges vr1-dc0 / vr1-dc1 on the jumphost exit 0, 1 bridge each (uplink only --
the site-prefix filter correctly declines to invent plane bridges that are
not on this host)
plan vr1-dc0 on vr1-dc0-maas exit 3, REFUSE, over ssh:
"'maas vr1-dc0-region machines read' failed -- usage: maas [-h] COMMAND ..."
The ssh user on that VM (ubuntu) has NO ~/.maascli.db, so no profile is
logged in. This is the REFUSE contract working end-to-end on a real host:
"could not look" produced 3, not a green.
NOT LIVE-VALIDATED, AND IT IS THIS TOOL'S LARGEST UNTESTED ASSUMPTION: that
`machines read` populates interface_set[].links[].subnet.cidr. It is NOT a guess --
scripts/carve-host-interfaces.sh:120 and :293 both consume `.links[]?.subnet.cidr`
against the MAAS interface serialization, and dc-node-v6-carve.py:129-136 consumes
`links[].subnet.vlan.id` off `machines read` itself and successfully carved the 54
statics -- but the exact field on the exact endpoint has not been seen by this
session. CLOSE IT WITH ONE COMMAND, from a shell that HAS a logged-in profile:
bash scripts/dc-node-v6-verify.sh plan vr1-dc0
If the SPEC block comes back empty while `dc-node-v6-carve.py check vr1-dc0` is
green, that field is the reason -- and `plan` will say FAIL ("carries NO v6 link"),
which is loud, not silent.
Cross-check for the tag question, REASONED not measured (no maas CLI on this
jumphost, so it entered no command): the 54 statics = 9 role nodes x 6 planes, and
lib-hosts.sh lists 10 machines per DC including juju-01. That arithmetic says the
openstack-<site> tag selects role nodes only. `plan` does not DEPEND on it: it makes
no count assertion at all, only that every selected machine carries an IDENTICAL v6
plane set. If the controller is ever tagged, its 2-plane carve turns that red -- and
that red is informative, not false.
5. DESIGN CHOICES A REVIEWER WOULD OTHERWISE HAVE TO RECONSTRUCT
--------------------------------------------------------------------------------
* THREE SUBCOMMANDS, NOT ONE, and the split is the dc-cache-proxy/dc-snap-proxy
precedent: `check` asserts rack-side facts a client cannot satisfy, so pointing a
client at it fails for reasons that say nothing about the client. Same here --
`node` has no maas CLI, no root, no repo; `bridges` needs virsh and sysfs.
* NO PLANE-COUNT LITERAL. "Six planes" is not asserted anywhere, because D-139
makes lb-mgmt a seventh. `plan` asserts SET EQUALITY across machines instead,
which catches the real hazard (an under-carved node) and survives the change.
* SOLE-GLOBAL-ON-THE-NIC is deliberate and is the ULA->GUA rot catcher: after the
carve moves, a node that gained its GUA but never lost its ULA reads green on
"the expected address is present" and only this predicate turns red.
>>> THE ONE KNOWN WAY THIS PREDICATE CAN GO WRONG, AND IT IS NOT RED TODAY. <<<
D-139 ruling B carves a provider-public VIP /64 (2602:f3e2:f0X:11::/64). If
hacluster places a v6 VIP as a SECONDARY on the same netdev that carries the
node's own static (br-ex on provider-public, the metal-admin NIC on the admin
plane), then whichever unit currently HOLDS the VIP grows a second global there
and this predicate FAILs on that node alone, intermittently, as the VIP moves --
the "gate that cannot reliably PASS" hazard dc-snap-proxy.sh:110-120 names.
NOT RED NOW: the v6 VIP row is RULED, NOT CARVED, and the -6 filter excludes
every existing v4 VIP. NO EXEMPTION WAS BUILT, deliberately: there is nothing to
measure one against and inventing the shape would be an inferred value (hard
rule 2). THE DISCRIMINATOR when it matters is simply whether the v6 VIP lands on
the SAME netdev as the node static; measure that on the first node that holds
one, then either scope this predicate or leave it. The FAIL message already
names an HA VIP as a benign cause and points here.
* CARRIER (LOWER_UP) ON br-ex IS THE ONE OUTPUT SHAPE THIS BUILD NEVER SAW.
Every other `ip` shape parsed here came from a real reading or a stable
well-known format. Under Pattern A (lib-hosts.sh BREX_PARENT_NIC="enp2s0": the
provider-public static lands on the OVS bridge br-ex and the member NIC holds no
L3 link) `plan` WILL emit a br-ex entry, and `node` gates it on LOWER_UP in
`ip -o link show dev br-ex`. Whether ovs-vswitchd sets carrier on an OVS INTERNAL
port in this deployment is NOT confirmed and is not asserted here.
ATTEMPTED AND FAILED THIS SESSION: no reachable host carries a br-ex. The dc0
rack has only lo/enp1s0/enp2s0/br-vr1-dc0-wan/virbr0-6/vnet*; vr1-dc0-juju
refused the key, and by lib-hosts.sh the controller has no br-ex anyway.
CLOSE IT WITH ONE READ-ONLY COMMAND on the first node whose br-ex is up:
ip -o link show dev br-ex
If LOWER_UP is ABSENT on a healthy br-ex, this predicate is a guaranteed false
red on provider-public for every role node and must be scoped to physical NICs.
Record the result either way -- a false red here would be read as a carve defect.
* THE FOUR QUADRANTS. ping-OK + neighbour FAILED/INCOMPLETE/absent is reported as a
CONTRADICTION rather than a pass, because one of the two readings is lying and a
gate must not close on that. ping-FAIL + neighbour RESOLVED is explicitly NOT
diagnosed as ND -- it says so, and points at filtering/routing instead.
* PEER SELECTION is deterministic (lowest address among the other holders) and the
peer hostname is printed, so a capture names which two machines were tested. A
plane with ONE holder gets peer '-', `plan` REFUSEs it, and `node` REFUSEs that
entry -- a peerless plane is unevaluable, never a pass.
* `| grep -q` under pipefail is avoided throughout; every match is a here-string or
an awk field comparison (the repo has been bitten by that short-circuit twice).
* TEST SEAM: DC_NODE_V6_SYSROOT (sysfs) plus fake maas/ip/ping/virsh on PATH --
the DC_SNAP_PROXY_SYSROOT idiom.
6. HARNESS, MUTATION PASS, GATES
--------------------------------------------------------------------------------
tests/dc-node-v6-verify/run-tests.sh -- 55 cases, 55 passed, 0 failed.
T1-T10 static: no v6/v4 literal in code, exit contract, header <=25 lines,
site+mode validation, node arm free of sudo/systemctl/virsh/maas,
ping-before-neighbour ordering.
T11-T25 plan driven against a fake maas: green (ULA fixture AND GUA fixture),
under-carve, zero-v6 machine, two links on one plane, peerless plane,
empty tag selection, MAAS error, unparseable JSON, absent maas CLI,
profile provenance, bad option.
T26-T46 node driven against fake ip/ping: green, address missing, tentative,
dadfailed, extra global, iface absent, no carrier, all four
reply/neighbour quadrants, unrecognised neighbour state, ping could not
run, peerless entry, malformed/missing SPEC, absent ip, absent ping
(address half still asserted), multi-entry evaluation.
T47-T54 bridges: reads both values, SUSPECT tagging, no false SUSPECT when a
querier exists, the no-diagnosis NOTE, unreadable sysfs, no plane
network, net with no bridge, absent virsh.
MUTATION PASS -- three predicates neutered one at a time, harness re-run, restored,
sha256 re-checked. Baseline and post-restore sha256 of scripts/dc-node-v6-verify.sh
are IDENTICAL: 3ea87ced6352cf3aa0f4e8a354ae4831003b2dff7a156d880f5d8091e0c251ca
(the pass was RE-RUN in full after the extra-global FAIL message gained its HA-VIP
note, so the recorded sha is the delivered file's, not a pre-edit one)
M1 node, the CONTRADICTION predicate: [ "$prc" -eq 0 ] -> [ "$prc" -eq 99 ]
RED: T37 "replies YET neighbour INCOMPLETE -> FAIL"
T38 "replies with NO neighbour entry at all -> FAIL" (53/2)
M2 node, the stale-prefix catcher: [ -n "$extra" ] -> [ -n "" ]
RED: T31 "an EXTRA global on the plane -> FAIL" (54/1)
M3 plan, the plane-set-identity predicate: `if s != modal` -> `if False`
RED: T15 "an under-carved machine -> FAIL" (54/1)
All three are BEHAVIORAL, not textual: each mutation left the file syntactically
valid, and the tests that went red are the ones that DRIVE the branch, not greps
that match its text.
A HARNESS BUG THE MUTATION DISCIPLINE DID NOT CATCH, but the first run did, and it
is the reason the "absent tool" cases are worth re-reading anywhere they appear:
the absent-ip and absent-virsh REFUSE cases were first driven with
PATH="/usr/bin:/bin". MEASURED: this jumphost carries `ip` AND `virsh` on that
path, so both cases silently executed the HOST's real binaries. They reported red
for the right reason by luck. Fixed with MINBIN -- a mktemp PATH holding symlinks
to exactly the coreutils the script uses and nothing else.
repo-lint: see section 7.
gauntlet (bash scripts/run-tests-all.sh, > 2 min): see section 7.
tests/HARNESS-MANIFEST re-recorded per R15(2). REASON: a new harness directory,
tests/dc-node-v6-verify, was ADDED. Without the re-record the gauntlet's drift gate
FAILS with "present but UNPINNED", which is the gate working correctly.
7. GATE RESULTS
--------------------------------------------------------------------------------
Revert line: docs/changelog-20260801-g19-node-v6-verify.md
bash tests/dc-node-v6-verify/run-tests.sh 55 passed, 0 failed
bash scripts/run-tests-all.sh GAUNTLET: ALL GREEN (96 harnesses)
(took just over 2 minutes)
bash scripts/repo-lint.sh FAIL: 1 fail, 1 warn, 647 files
The WARN is pre-existing and untouched by this change set:
[WARN] L1 docs/design-decisions.md: 239 non-ASCII byte(s) (legacy D-001..018
carve-out; NEW entries must be ASCII)
The FAIL is L10 ONLY, and it is this session's instructions colliding by
construction -- verbatim:
[FAIL] L10 pending change set modifies state-bearing surface(s)
[docs/audit/: docs/audit/g19-ipv6-plane-gate-build-20260801.txt]
without touching docs/CURRENT-STATE.md -- GA-R1 rule 8 (C1): a
status-changing commit updates CURRENT-STATE.md in the SAME commit
The capture was MANDATORY and docs/CURRENT-STATE.md was OFF LIMITS (the operator
owns the G19 gate row). L10 clears the moment that row lands in the same commit.
NOTHING ELSE FAILED: L3 (ghost refs), L9 (self-ref) and the rest are clean, and
all three new files are ASCII + LF.
Files: scripts/dc-node-v6-verify.sh 270 lines (24-line header, charter budget
25); tests/dc-node-v6-verify/run-tests.sh 362 lines; this capture 232 lines.
The script is the length it is because it carries three distinct hosts' worth of
assertions and every FAIL/REFUSE branch has to say what to do next -- the prose
is compressed (no rationale, no worked examples, no alternatives-considered;
those live here and in the changelog), the discipline is not.
