G19 -- IPv6 PLANE GATE: BUILD CAPTURE 2026-08-01 Tool: scripts/dc-node-v6-verify.sh Harness: tests/dc-node-v6-verify/run-tests.sh Branch dc-dc-stage5-preconditions at 9853cb7. Nothing in this session mutated the cloud: every live command below is read-only (virsh net-dumpxml, sysfs reads, one refused MAAS read). No MAAS write, no power op, no deploy. DOCS NOT TOUCHED BY DESIGN: docs/CURRENT-STATE.md and docs/design-decisions.md are the operator's; the G19 gate row is theirs to write. See "repo-lint" below -- that split is exactly what L10 fires on, and the finding is expected, not a defect. 1. THE MANDATE, AND WHAT THE TOOL ACTUALLY ASSERTS -------------------------------------------------------------------------------- Two DIFFERENT assertions, and the split into subcommands follows the split in the question: (1) THE ADDRESSES ARE UP, asserted ON THE INTERFACE. `node` Per plane: the interface exists and has LOWER_UP; the address MAAS holds is present with scope global; DAD is complete (not tentative, not dadfailed); and it is the SOLE global on that NIC. (2) THE PLANE CARRIES v6 BETWEEN TWO NODES. `node`, same pass ping -6 to a PEER node's address on the same plane, THEN read `ip -6 neigh` -- in that order, because the ping is what populates the cache the second reading inspects. All four quadrants of (reply x neighbour-resolved) are distinguished and each names a different next step. DIAGNOSTIC, no verdict. `bridges` (runs on the RACK, not the node) Prints each plane bridge's multicast_snooping / multicast_querier. EXPECTATION SOURCE. `plan` (runs where the maas CLI is) Derives the per-node expectation from live MAAS and prints one SPEC line per machine. Asserts MAAS-side consistency only. WHAT A GREEN DOES NOT MEAN, and it is in the script header so nobody has to find this file: G19 asserts the NIC AGAINST MAAS, not against D-139's ruled GUA table. Green means "the carve MAAS holds is actually up, whatever prefix that is". MAAS-vs-ruling is dc-node-v6-carve.py check and tests/d139-gua-carve. `plan` PRINTS the derived prefixes so a reader of any capture sees at a glance whether the ULA or the GUA carve is the one being asserted. 2. HOW THE PREFIX-SOURCE CONSTRAINT WAS RESOLVED, AND WHY -------------------------------------------------------------------------------- CONSTRAINT: D-139 (ADOPTED, confirmed by the 2026-08-01 ruling note) retires fd50:840e:74e2::/48 for VR1 and moves every plane to 2602:f3e2:f02::/48 (dc0) / 2602:f3e2:f03::/48 (dc1). A site table with ULA literals is correct today and wrong within days. CHOSEN: DERIVE FROM LIVE MAAS, and pass the result to the node as INPUT. * `plan` reads `maas machines read`, selects machines by the tag openstack-, and for every link whose subnet CIDR is v6 emits iface=subnet_cidr=addr=peer=peerhost. * `node` takes that SPEC as argv[3]. It contains NO prefix knowledge of its own. * ZERO v6 prefix literals exist anywhere in the executed code. Harness T3 enforces this by grepping non-comment lines for fd..:/2602:/fc00::; T4 does the same for v4 plane literals. T14 drives the SAME green plan case with a GUA fixture and a ULA fixture -- the argument is executed, not asserted in prose. WHY MAAS AND NOT THE NETBOX APEX. The gate's job is precisely the recorded lesson "MAAS holding a link is not the same as an interface carrying it": MAAS is the authority for WHAT SHOULD BE ON THE NIC, and the NIC is the authority for what is. Using the apex instead would fold two independent failures (apex->MAAS drift and MAAS->NIC drift) into one red, which is the failure mode this gate is built to avoid. Apex-vs-MAAS already has its own owner. WHY NOT AN OPERATOR-SUPPLIED PREFIX TABLE. It would need re-editing on carve day, which is the exact rot the mandate names. CORRECTION TO THE MANDATE'S PREMISE, and it matters. The mandate states dc-node-v6-carve.py "derives them from the NetBox apex per D-136 option (D)". It does not. Read at scripts/dc-node-v6-carve.py:96-137: it derives the v6 subnet from LIVE MAAS -- the v6 subnet sharing a MAAS VLAN with the interface's EXISTING v4 link -- and D-136 option (D) is the separate ruling to mirror the v4 octet TEXTUALLY. CONSEQUENCE, and this is an owed fix that is NOT mine to make under hard rule 1: that v4-pivot BREAKS under D-139. Five planes lose their v4 subnet entirely, so "the interface that already carries a v4 static" selects nothing on metal-internal / data-tenant / storage / replication, and lb-mgmt never had a v4 twin at all. dc-node-v6-carve.py will silently carve four fewer planes per node after the v4 removal step. LOGGED, not executed. G19 does NOT inherit that defect: `plan` pivots on "the link's subnet CIDR is v6", which needs no v4 twin. 3. THE FAILURE MODE THIS GATE EXISTS TO CATCH -- MEASURED, BOTH RACKS -------------------------------------------------------------------------------- Measured 2026-08-01 by running the tool itself, read-only, over the intended `ssh 'bash -s -- bridges ' < scripts/dc-node-v6-verify.sh` path. ssh vr1-dc0-rack: ssh vr1-dc1-rack: vr1-dc0-data-tenant virbr4 1 / 0 SUSPECT vr1-dc1-data-tenant virbr1 1 / 0 vr1-dc0-metal-admin virbr2 1 / 0 SUSPECT vr1-dc1-metal-admin virbr6 1 / 0 vr1-dc0-metal-internal virbr6 1 / 0 SUSPECT vr1-dc1-metal-internal virbr5 1 / 0 vr1-dc0-provider-public virbr5 1 / 0 SUSPECT vr1-dc1-provider-public virbr4 1 / 0 vr1-dc0-replication virbr1 1 / 0 SUSPECT vr1-dc1-replication virbr2 1 / 0 vr1-dc0-storage virbr3 1 / 0 SUSPECT vr1-dc1-storage virbr3 1 / 0 vr1-dc0-wan br-vr1-dc0-wan 1 / 0 SUSPECT vr1-dc1-wan br-vr1-dc1-wan 1 / 0 (numbers are multicast_snooping / multicast_querier; exit 0 = reported, 7 bridges) This INDEPENDENTLY CONFIRMS the dc0 reading the mandate reports, and EXTENDS it: dc1 is identical, and so is the WAN bridge. The jumphost's own two transit nets (vr1-dc0-uplink virbr4, vr1-dc1-uplink virbr1) read the same -- so this is the libvirt default across every bridge in the estate, not a per-rack setting. NO LINUX BEHAVIOUR IS ASSERTED, here or in the tool. The mandate said the exact behaviour is unconfirmed and should not be asserted; the subcommand REPORTS the two values, tags the 1/0 combination "SUSPECT", and prints a NOTE saying in terms that it diagnoses nothing. What makes it worth having is that a red assertion (2) can be triaged in one step instead of three -- and `node`'s no-reply + INCOMPLETE branch names this subcommand by its exact command line. WHY IT SITS NEXT TO ASSERTION (2): D-139 records all six v6 subnets at gateway_ip=None, so peers are on-link and Neighbour Discovery -- which is multicast -- IS the resolution mechanism on these planes. CARRIED, NOT CLOSED: nothing here shows the plane FAILS. No node currently carries these addresses up (Stage 5 deploy is in flight), so assertion (2) has never run against real traffic. The instrument is built; the experiment is the operator's. 4. WHAT WAS RUN LIVE, AND WHAT THAT PROVED -------------------------------------------------------------------------------- bridges vr1-dc0 / vr1-dc1 on both racks exit 0, 7 bridges each (section 3) bridges vr1-dc0 / vr1-dc1 on the jumphost exit 0, 1 bridge each (uplink only -- the site-prefix filter correctly declines to invent plane bridges that are not on this host) plan vr1-dc0 on vr1-dc0-maas exit 3, REFUSE, over ssh: "'maas vr1-dc0-region machines read' failed -- usage: maas [-h] COMMAND ..." The ssh user on that VM (ubuntu) has NO ~/.maascli.db, so no profile is logged in. This is the REFUSE contract working end-to-end on a real host: "could not look" produced 3, not a green. NOT LIVE-VALIDATED, AND IT IS THIS TOOL'S LARGEST UNTESTED ASSUMPTION: that `machines read` populates interface_set[].links[].subnet.cidr. It is NOT a guess -- scripts/carve-host-interfaces.sh:120 and :293 both consume `.links[]?.subnet.cidr` against the MAAS interface serialization, and dc-node-v6-carve.py:129-136 consumes `links[].subnet.vlan.id` off `machines read` itself and successfully carved the 54 statics -- but the exact field on the exact endpoint has not been seen by this session. CLOSE IT WITH ONE COMMAND, from a shell that HAS a logged-in profile: bash scripts/dc-node-v6-verify.sh plan vr1-dc0 If the SPEC block comes back empty while `dc-node-v6-carve.py check vr1-dc0` is green, that field is the reason -- and `plan` will say FAIL ("carries NO v6 link"), which is loud, not silent. Cross-check for the tag question, REASONED not measured (no maas CLI on this jumphost, so it entered no command): the 54 statics = 9 role nodes x 6 planes, and lib-hosts.sh lists 10 machines per DC including juju-01. That arithmetic says the openstack- tag selects role nodes only. `plan` does not DEPEND on it: it makes no count assertion at all, only that every selected machine carries an IDENTICAL v6 plane set. If the controller is ever tagged, its 2-plane carve turns that red -- and that red is informative, not false. 5. DESIGN CHOICES A REVIEWER WOULD OTHERWISE HAVE TO RECONSTRUCT -------------------------------------------------------------------------------- * THREE SUBCOMMANDS, NOT ONE, and the split is the dc-cache-proxy/dc-snap-proxy precedent: `check` asserts rack-side facts a client cannot satisfy, so pointing a client at it fails for reasons that say nothing about the client. Same here -- `node` has no maas CLI, no root, no repo; `bridges` needs virsh and sysfs. * NO PLANE-COUNT LITERAL. "Six planes" is not asserted anywhere, because D-139 makes lb-mgmt a seventh. `plan` asserts SET EQUALITY across machines instead, which catches the real hazard (an under-carved node) and survives the change. * SOLE-GLOBAL-ON-THE-NIC is deliberate and is the ULA->GUA rot catcher: after the carve moves, a node that gained its GUA but never lost its ULA reads green on "the expected address is present" and only this predicate turns red. >>> THE ONE KNOWN WAY THIS PREDICATE CAN GO WRONG, AND IT IS NOT RED TODAY. <<< D-139 ruling B carves a provider-public VIP /64 (2602:f3e2:f0X:11::/64). If hacluster places a v6 VIP as a SECONDARY on the same netdev that carries the node's own static (br-ex on provider-public, the metal-admin NIC on the admin plane), then whichever unit currently HOLDS the VIP grows a second global there and this predicate FAILs on that node alone, intermittently, as the VIP moves -- the "gate that cannot reliably PASS" hazard dc-snap-proxy.sh:110-120 names. NOT RED NOW: the v6 VIP row is RULED, NOT CARVED, and the -6 filter excludes every existing v4 VIP. NO EXEMPTION WAS BUILT, deliberately: there is nothing to measure one against and inventing the shape would be an inferred value (hard rule 2). THE DISCRIMINATOR when it matters is simply whether the v6 VIP lands on the SAME netdev as the node static; measure that on the first node that holds one, then either scope this predicate or leave it. The FAIL message already names an HA VIP as a benign cause and points here. * CARRIER (LOWER_UP) ON br-ex IS THE ONE OUTPUT SHAPE THIS BUILD NEVER SAW. Every other `ip` shape parsed here came from a real reading or a stable well-known format. Under Pattern A (lib-hosts.sh BREX_PARENT_NIC="enp2s0": the provider-public static lands on the OVS bridge br-ex and the member NIC holds no L3 link) `plan` WILL emit a br-ex entry, and `node` gates it on LOWER_UP in `ip -o link show dev br-ex`. Whether ovs-vswitchd sets carrier on an OVS INTERNAL port in this deployment is NOT confirmed and is not asserted here. ATTEMPTED AND FAILED THIS SESSION: no reachable host carries a br-ex. The dc0 rack has only lo/enp1s0/enp2s0/br-vr1-dc0-wan/virbr0-6/vnet*; vr1-dc0-juju refused the key, and by lib-hosts.sh the controller has no br-ex anyway. CLOSE IT WITH ONE READ-ONLY COMMAND on the first node whose br-ex is up: ip -o link show dev br-ex If LOWER_UP is ABSENT on a healthy br-ex, this predicate is a guaranteed false red on provider-public for every role node and must be scoped to physical NICs. Record the result either way -- a false red here would be read as a carve defect. * THE FOUR QUADRANTS. ping-OK + neighbour FAILED/INCOMPLETE/absent is reported as a CONTRADICTION rather than a pass, because one of the two readings is lying and a gate must not close on that. ping-FAIL + neighbour RESOLVED is explicitly NOT diagnosed as ND -- it says so, and points at filtering/routing instead. * PEER SELECTION is deterministic (lowest address among the other holders) and the peer hostname is printed, so a capture names which two machines were tested. A plane with ONE holder gets peer '-', `plan` REFUSEs it, and `node` REFUSEs that entry -- a peerless plane is unevaluable, never a pass. * `| grep -q` under pipefail is avoided throughout; every match is a here-string or an awk field comparison (the repo has been bitten by that short-circuit twice). * TEST SEAM: DC_NODE_V6_SYSROOT (sysfs) plus fake maas/ip/ping/virsh on PATH -- the DC_SNAP_PROXY_SYSROOT idiom. 6. HARNESS, MUTATION PASS, GATES -------------------------------------------------------------------------------- tests/dc-node-v6-verify/run-tests.sh -- 55 cases, 55 passed, 0 failed. T1-T10 static: no v6/v4 literal in code, exit contract, header <=25 lines, site+mode validation, node arm free of sudo/systemctl/virsh/maas, ping-before-neighbour ordering. T11-T25 plan driven against a fake maas: green (ULA fixture AND GUA fixture), under-carve, zero-v6 machine, two links on one plane, peerless plane, empty tag selection, MAAS error, unparseable JSON, absent maas CLI, profile provenance, bad option. T26-T46 node driven against fake ip/ping: green, address missing, tentative, dadfailed, extra global, iface absent, no carrier, all four reply/neighbour quadrants, unrecognised neighbour state, ping could not run, peerless entry, malformed/missing SPEC, absent ip, absent ping (address half still asserted), multi-entry evaluation. T47-T54 bridges: reads both values, SUSPECT tagging, no false SUSPECT when a querier exists, the no-diagnosis NOTE, unreadable sysfs, no plane network, net with no bridge, absent virsh. MUTATION PASS -- three predicates neutered one at a time, harness re-run, restored, sha256 re-checked. Baseline and post-restore sha256 of scripts/dc-node-v6-verify.sh are IDENTICAL: 3ea87ced6352cf3aa0f4e8a354ae4831003b2dff7a156d880f5d8091e0c251ca (the pass was RE-RUN in full after the extra-global FAIL message gained its HA-VIP note, so the recorded sha is the delivered file's, not a pre-edit one) M1 node, the CONTRADICTION predicate: [ "$prc" -eq 0 ] -> [ "$prc" -eq 99 ] RED: T37 "replies YET neighbour INCOMPLETE -> FAIL" T38 "replies with NO neighbour entry at all -> FAIL" (53/2) M2 node, the stale-prefix catcher: [ -n "$extra" ] -> [ -n "" ] RED: T31 "an EXTRA global on the plane -> FAIL" (54/1) M3 plan, the plane-set-identity predicate: `if s != modal` -> `if False` RED: T15 "an under-carved machine -> FAIL" (54/1) All three are BEHAVIORAL, not textual: each mutation left the file syntactically valid, and the tests that went red are the ones that DRIVE the branch, not greps that match its text. A HARNESS BUG THE MUTATION DISCIPLINE DID NOT CATCH, but the first run did, and it is the reason the "absent tool" cases are worth re-reading anywhere they appear: the absent-ip and absent-virsh REFUSE cases were first driven with PATH="/usr/bin:/bin". MEASURED: this jumphost carries `ip` AND `virsh` on that path, so both cases silently executed the HOST's real binaries. They reported red for the right reason by luck. Fixed with MINBIN -- a mktemp PATH holding symlinks to exactly the coreutils the script uses and nothing else. repo-lint: see section 7. gauntlet (bash scripts/run-tests-all.sh, > 2 min): see section 7. tests/HARNESS-MANIFEST re-recorded per R15(2). REASON: a new harness directory, tests/dc-node-v6-verify, was ADDED. Without the re-record the gauntlet's drift gate FAILS with "present but UNPINNED", which is the gate working correctly. 7. GATE RESULTS -------------------------------------------------------------------------------- Revert line: docs/changelog-20260801-g19-node-v6-verify.md bash tests/dc-node-v6-verify/run-tests.sh 55 passed, 0 failed bash scripts/run-tests-all.sh GAUNTLET: ALL GREEN (96 harnesses) (took just over 2 minutes) bash scripts/repo-lint.sh FAIL: 1 fail, 1 warn, 647 files The WARN is pre-existing and untouched by this change set: [WARN] L1 docs/design-decisions.md: 239 non-ASCII byte(s) (legacy D-001..018 carve-out; NEW entries must be ASCII) The FAIL is L10 ONLY, and it is this session's instructions colliding by construction -- verbatim: [FAIL] L10 pending change set modifies state-bearing surface(s) [docs/audit/: docs/audit/g19-ipv6-plane-gate-build-20260801.txt] without touching docs/CURRENT-STATE.md -- GA-R1 rule 8 (C1): a status-changing commit updates CURRENT-STATE.md in the SAME commit The capture was MANDATORY and docs/CURRENT-STATE.md was OFF LIMITS (the operator owns the G19 gate row). L10 clears the moment that row lands in the same commit. NOTHING ELSE FAILED: L3 (ghost refs), L9 (self-ref) and the rest are clean, and all three new files are ASCII + LF. Files: scripts/dc-node-v6-verify.sh 270 lines (24-line header, charter budget 25); tests/dc-node-v6-verify/run-tests.sh 362 lines; this capture 232 lines. The script is the length it is because it carries three distinct hosts' worth of assertions and every FAIL/REFUSE branch has to say what to do next -- the prose is compressed (no rationale, no worked examples, no alternatives-considered; those live here and in the changelog), the discipline is not.