diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 3410c03..9911666 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2149,6 +2149,64 @@ being built can be run against a LIVE node rather than shipping fixture-green -- the lesson from the snap proxy, whose harness was green for a week before anything was proven end to end. Release is owed once the gate has run. + **>>> GATE G19 IS BUILT AND HAS PASSED LIVE -- NOT FIXTURE-GREEN. <<<** `scripts/dc-node-v6-verify.sh` + (270 lines, 24-line header) + `tests/dc-node-v6-verify/` (55 cases); manifest 95 -> 96. + Build capture `docs/audit/g19-ipv6-plane-gate-build-20260801.txt`; live run appended to + `docs/audit/g19-ipv6-node-plane-verify-20260801.txt` (147 lines). Three subcommands, all + exercised live this session: **`plan vr1-dc0` PASS exit 0** (tag selects 9 machines, v6 + plane set IDENTICAL across all 9, 6 planes, prefixes DERIVED); **`node vr1-dc0 ` on + `vr1-dc0-storage-01` against live peer `storage-02` PASS exit 0, 12 assertions -- six + address-presence and six peer-reachability, every plane "up, global, DAD complete, sole + global on the NIC" and "replies, neighbour DELAY"**; and `bridges vr1-dc0` reporting 7 + bridges. Harness 55/55, gauntlet ALL GREEN (96), three mutations each turning a NAMED test + red. **It was run against a REAL node while the boot window was open specifically so it + would not ship fixture-green -- the snap proxy's harness was green for a week before + anything was proven end to end.** + **NO v6 PREFIX LITERAL EXISTS IN THE FILE, and that was a build constraint, not a nicety:** + D-139 retires the VR1 ULA `/48` for GUA, so a baked table would be right today and wrong on + carve day. `plan` derives the expectation from LIVE MAAS and emits a SPEC; `node` consumes + the SPEC and knows no prefixes. A harness case drives the same green `plan` path with a ULA + fixture AND a GUA fixture, so the property is EXECUTED rather than asserted. + **WHAT A GREEN DOES NOT MEAN, and the tool's own header says so: it asserts the NIC against + MAAS, NOT against D-139's ruled GUA table.** MAAS-versus-ruling is a different gate. + **>>> A LATENT DEFECT IN D-139's OWN EXECUTION LIST, FOUND BY THE BUILD AND VERIFIED HERE + AGAINST THE SCRIPT'S SOURCE. <<<** D-139's execution list names `dc-node-v6-carve.py` to + "re-carve 54 node v6 statics". **That tool is STRUCTURALLY DEPENDENT ON IPv4 EXISTING** -- + its own header states it assigns v6 "on every plane **where it already carries IPv4**", + selects the v6 subnet "on the SAME MAAS vlan as that v4 link", and derives the host part from + "the last octet of the node's own v4 address"; the loop body is a bare `if not v4: continue`. + **Under D-139 five planes lose v4 entirely and `lb-mgmt` never had a v4 twin, so after the + v4-removal step the tool would SILENTLY CARVE FOUR FEWER PLANES PER NODE rather than + failing.** That is the silent-truncation class this repo already has scars from. **CONSEQUENCE: + the v6 carve MUST run BEFORE v4 removal, or the tool must be rewritten -- and this compounds + the already-recorded fact that D-139's "remove v4 LAST, after each is proven" clause is + unsatisfiable.** LOGGED, NOT FIXED. **My own agent mandate also stated this tool's input + source WRONGLY** (I said it derives from the NetBox apex per D-136 option (D); it derives + from live MAAS) -- the agent checked rather than inheriting the error, which is why the + defect surfaced at all. + **THREE FURTHER FINDINGS FROM THE BUILD, none red today.** (i) **The multicast reading is + ESTATE-WIDE, not a dc0 quirk:** `bridges` run on BOTH racks shows all six plane bridges + **plus the WAN bridge** at `multicast_snooping=1 multicast_querier=0` on dc0 AND dc1, and the + jumphost's own uplink bridges read the same -- it is the libvirt/kernel default here, not a + setting anyone chose. The subcommand tags it SUSPECT and **asserts no Linux behaviour**, + which is correct: A2 above already showed cold-start multicast ND working across it. + (ii) **`plan`'s PEER SELECTION IS DEPLOYMENT-BLIND** -- it picked `vr1-dc0-control-01` as + peer for every machine, and control-01 is NOT deployed, so the derived SPEC cannot pass in a + PARTIAL deployment. Correct for the full-deployment case it targets, wrong for exactly the + bounded experiment it was built to support. Worked around by substituting the live peer; + LOGGED, NOT FIXED. (iii) **A harness bug the instrument-currency rule caught, NINTH of its + kind:** the absent-`ip`/absent-`virsh` REFUSE cases were first driven with + `PATH="/usr/bin:/bin"`, which on this jumphost still contains both -- so they ran the real + binaries and were red for the WRONG REASON. Fixed with a mktemp `MINBIN` carrying only the + coreutils the script uses. + **TWO OPEN CHECKS, both settled here on the live nodes rather than left open:** the + sole-global-per-NIC predicate was untested against a node carrying a second global (measured: + every NIC carries exactly ONE, so it holds today, and a v6 HA VIP landing as a secondary + would trip it -- no exemption exists), and `LOWER_UP` on an OVS-internal `br-ex` had never + been seen by the build (measured: ``, so the predicate is + safe). **GA-R2 HOUSEKEEPING:** the build agent wrote its own + `docs/changelog-20260801-g19-node-v6-verify.md`; it is FOLDED VERBATIM into this session's + single changelog and REMOVED. The session crossing midnight does not start a new session. - Project: Omega Cloud, VR1 DC-DC rehearsal -- a two-DC + Office1-headend virtual rehearsal on KVM (vcloud host), rehearsing the future bare-metal Roosevelt deployment (D-100, `docs/design-decisions.md:1946`). diff --git a/docs/audit/g19-ipv6-node-plane-verify-20260801.txt b/docs/audit/g19-ipv6-node-plane-verify-20260801.txt index cd0be49..5ef2219 100644 --- a/docs/audit/g19-ipv6-node-plane-verify-20260801.txt +++ b/docs/audit/g19-ipv6-node-plane-verify-20260801.txt @@ -95,3 +95,53 @@ source is (a) MAAS-served and NOT the DC edge, satisfying G17 assertion 2, and (b) ALREADY RUNNING OVER IPv6 IN PRODUCTION, unprompted. This is not a test path: it is a live operational dependency on IPv6 that predates any of this verification work. + +### A5 -- THE G19 GATE ITSELF, RUN LIVE (added after the tool was built) +The tool exists so this stops being a hand-run measurement. Run against the SAME two +nodes while the boot window was still open, so it ships PROVEN, not fixture-green -- +the snap proxy's harness was green for a week before anything was proven end to end. + +$ ssh voffice1 "bash -s -- plan vr1-dc0 --profile vr1-dc0-region" < scripts/dc-node-v6-verify.sh + -> PASS, EXIT 0. tag openstack-vr1-dc0 selects 9 machines; no machine holds two v6 + links on one plane; v6 plane set IDENTICAL across all 9 (6 planes). + DERIVED prefixes (the file contains NO v6 literal): + 2602:f3e2:f02:10::/64 fd50:840e:74e2:{220,221,230,240,250}::/64 + + FINDING -- PEER SELECTION IS DEPLOYMENT-BLIND. plan picked vr1-dc0-control-01 + (::100) as peer for every machine. control-01 is NOT deployed (only storage-01/02 + are), so the derived SPEC cannot pass in a PARTIAL deployment: every plane would + fail against a powered-off peer. Correct for the full-deployment case the gate + targets, wrong for exactly the bounded experiment it was built to support. Worked + around here by substituting the live peer; LOGGED, NOT FIXED. + +$ ssh "bash -s -- node vr1-dc0 " < scripts/dc-node-v6-verify.sh + (spec = plan output for storage-01 with the peer substituted to the LIVE + storage-02 ::151 -- one sed, no other change) + dc-node-v6-verify node (vr1-dc0) -- on vr1-dc0-storage-01 + OK br-ex (2602:f3e2:f02:10::/64): 2602:f3e2:f02:10::150/64 up, global, DAD complete, sole global on the NIC + OK br-ex (2602:f3e2:f02:10::/64): 2602:f3e2:f02:10::151 (vr1-dc0-storage-02) replies, neighbour DELAY + OK enp1s0 (fd50:840e:74e2:220::/64): fd50:840e:74e2:220::150/64 up, global, DAD complete, sole global on the NIC + OK enp1s0 (fd50:840e:74e2:220::/64): fd50:840e:74e2:220::151 (vr1-dc0-storage-02) replies, neighbour DELAY + OK enp3s0 (fd50:840e:74e2:221::/64): fd50:840e:74e2:221::150/64 up, global, DAD complete, sole global on the NIC + OK enp3s0 (fd50:840e:74e2:221::/64): fd50:840e:74e2:221::151 (vr1-dc0-storage-02) replies, neighbour DELAY + OK enp4s0 (fd50:840e:74e2:230::/64): fd50:840e:74e2:230::150/64 up, global, DAD complete, sole global on the NIC + OK enp4s0 (fd50:840e:74e2:230::/64): fd50:840e:74e2:230::151 (vr1-dc0-storage-02) replies, neighbour DELAY + OK enp5s0 (fd50:840e:74e2:240::/64): fd50:840e:74e2:240::150/64 up, global, DAD complete, sole global on the NIC + OK enp5s0 (fd50:840e:74e2:240::/64): fd50:840e:74e2:240::151 (vr1-dc0-storage-02) replies, neighbour DELAY + OK enp6s0 (fd50:840e:74e2:250::/64): fd50:840e:74e2:250::150/64 up, global, DAD complete, sole global on the NIC + OK enp6s0 (fd50:840e:74e2:250::/64): fd50:840e:74e2:250::151 (vr1-dc0-storage-02) replies, neighbour DELAY + dc-node-v6-verify node (vr1-dc0): PASS + EXIT=0 + +$ ssh vr1-dc0-rack "bash -s -- bridges vr1-dc0" < scripts/dc-node-v6-verify.sh (DIAGNOSTIC) + dc-node-v6-verify bridges (vr1-dc0) -- DIAGNOSTIC ONLY, no verdict + READ vr1-dc0-data-tenant virbr4 multicast_snooping=1 multicast_querier=0 <-- SUSPECT + READ vr1-dc0-metal-admin virbr2 multicast_snooping=1 multicast_querier=0 <-- SUSPECT + READ vr1-dc0-metal-internal virbr6 multicast_snooping=1 multicast_querier=0 <-- SUSPECT + READ vr1-dc0-provider-public virbr5 multicast_snooping=1 multicast_querier=0 <-- SUSPECT + READ vr1-dc0-replication virbr1 multicast_snooping=1 multicast_querier=0 <-- SUSPECT + READ vr1-dc0-storage virbr3 multicast_snooping=1 multicast_querier=0 <-- SUSPECT + READ vr1-dc0-wan br-vr1-dc0-wan multicast_snooping=1 multicast_querier=0 <-- SUSPECT + NOTE snooping=1 with querier=0 is REPORTED as suspect, not diagnosed: IPv6 + Neighbour Discovery is multicast, and no Linux behaviour is asserted here. + dc-node-v6-verify bridges (vr1-dc0): reported 7 bridge(s) diff --git a/docs/audit/g19-ipv6-plane-gate-build-20260801.txt b/docs/audit/g19-ipv6-plane-gate-build-20260801.txt new file mode 100644 index 0000000..e4f861f --- /dev/null +++ b/docs/audit/g19-ipv6-plane-gate-build-20260801.txt @@ -0,0 +1,289 @@ +G19 -- IPv6 PLANE GATE: BUILD CAPTURE 2026-08-01 +Tool: scripts/dc-node-v6-verify.sh Harness: tests/dc-node-v6-verify/run-tests.sh +Branch dc-dc-stage5-preconditions at 9853cb7. Nothing in this session mutated the +cloud: every live command below is read-only (virsh net-dumpxml, sysfs reads, one +refused MAAS read). No MAAS write, no power op, no deploy. + +DOCS NOT TOUCHED BY DESIGN: docs/CURRENT-STATE.md and docs/design-decisions.md are +the operator's; the G19 gate row is theirs to write. See "repo-lint" below -- that +split is exactly what L10 fires on, and the finding is expected, not a defect. + + +1. THE MANDATE, AND WHAT THE TOOL ACTUALLY ASSERTS +-------------------------------------------------------------------------------- +Two DIFFERENT assertions, and the split into subcommands follows the split in the +question: + + (1) THE ADDRESSES ARE UP, asserted ON THE INTERFACE. `node` + Per plane: the interface exists and has LOWER_UP; the address MAAS holds is + present with scope global; DAD is complete (not tentative, not dadfailed); + and it is the SOLE global on that NIC. + (2) THE PLANE CARRIES v6 BETWEEN TWO NODES. `node`, same pass + ping -6 to a PEER node's address on the same plane, THEN read `ip -6 neigh` + -- in that order, because the ping is what populates the cache the second + reading inspects. All four quadrants of (reply x neighbour-resolved) are + distinguished and each names a different next step. + + DIAGNOSTIC, no verdict. `bridges` (runs on the RACK, not the node) + Prints each plane bridge's multicast_snooping / multicast_querier. + + EXPECTATION SOURCE. `plan` (runs where the maas CLI is) + Derives the per-node expectation from live MAAS and prints one SPEC line per + machine. Asserts MAAS-side consistency only. + +WHAT A GREEN DOES NOT MEAN, and it is in the script header so nobody has to find +this file: G19 asserts the NIC AGAINST MAAS, not against D-139's ruled GUA table. +Green means "the carve MAAS holds is actually up, whatever prefix that is". +MAAS-vs-ruling is dc-node-v6-carve.py check and tests/d139-gua-carve. `plan` +PRINTS the derived prefixes so a reader of any capture sees at a glance whether +the ULA or the GUA carve is the one being asserted. + + +2. HOW THE PREFIX-SOURCE CONSTRAINT WAS RESOLVED, AND WHY +-------------------------------------------------------------------------------- +CONSTRAINT: D-139 (ADOPTED, confirmed by the 2026-08-01 ruling note) retires +fd50:840e:74e2::/48 for VR1 and moves every plane to 2602:f3e2:f02::/48 (dc0) / +2602:f3e2:f03::/48 (dc1). A site table with ULA literals is correct today and +wrong within days. + +CHOSEN: DERIVE FROM LIVE MAAS, and pass the result to the node as INPUT. + * `plan` reads `maas machines read`, selects machines by the tag + openstack-, and for every link whose subnet CIDR is v6 emits + iface=subnet_cidr=addr=peer=peerhost. + * `node` takes that SPEC as argv[3]. It contains NO prefix knowledge of its own. + * ZERO v6 prefix literals exist anywhere in the executed code. Harness T3 + enforces this by grepping non-comment lines for fd..:/2602:/fc00::; T4 does the + same for v4 plane literals. T14 drives the SAME green plan case with a GUA + fixture and a ULA fixture -- the argument is executed, not asserted in prose. + +WHY MAAS AND NOT THE NETBOX APEX. The gate's job is precisely the recorded lesson +"MAAS holding a link is not the same as an interface carrying it": MAAS is the +authority for WHAT SHOULD BE ON THE NIC, and the NIC is the authority for what is. +Using the apex instead would fold two independent failures (apex->MAAS drift and +MAAS->NIC drift) into one red, which is the failure mode this gate is built to +avoid. Apex-vs-MAAS already has its own owner. + +WHY NOT AN OPERATOR-SUPPLIED PREFIX TABLE. It would need re-editing on carve day, +which is the exact rot the mandate names. + +CORRECTION TO THE MANDATE'S PREMISE, and it matters. The mandate states +dc-node-v6-carve.py "derives them from the NetBox apex per D-136 option (D)". It +does not. Read at scripts/dc-node-v6-carve.py:96-137: it derives the v6 subnet from +LIVE MAAS -- the v6 subnet sharing a MAAS VLAN with the interface's EXISTING v4 +link -- and D-136 option (D) is the separate ruling to mirror the v4 octet TEXTUALLY. +CONSEQUENCE, and this is an owed fix that is NOT mine to make under hard rule 1: +that v4-pivot BREAKS under D-139. Five planes lose their v4 subnet entirely, so +"the interface that already carries a v4 static" selects nothing on +metal-internal / data-tenant / storage / replication, and lb-mgmt never had a v4 +twin at all. dc-node-v6-carve.py will silently carve four fewer planes per node +after the v4 removal step. LOGGED, not executed. + G19 does NOT inherit that defect: `plan` pivots on "the link's subnet CIDR is + v6", which needs no v4 twin. + + +3. THE FAILURE MODE THIS GATE EXISTS TO CATCH -- MEASURED, BOTH RACKS +-------------------------------------------------------------------------------- +Measured 2026-08-01 by running the tool itself, read-only, over the intended +`ssh 'bash -s -- bridges ' < scripts/dc-node-v6-verify.sh` path. + + ssh vr1-dc0-rack: ssh vr1-dc1-rack: + vr1-dc0-data-tenant virbr4 1 / 0 SUSPECT vr1-dc1-data-tenant virbr1 1 / 0 + vr1-dc0-metal-admin virbr2 1 / 0 SUSPECT vr1-dc1-metal-admin virbr6 1 / 0 + vr1-dc0-metal-internal virbr6 1 / 0 SUSPECT vr1-dc1-metal-internal virbr5 1 / 0 + vr1-dc0-provider-public virbr5 1 / 0 SUSPECT vr1-dc1-provider-public virbr4 1 / 0 + vr1-dc0-replication virbr1 1 / 0 SUSPECT vr1-dc1-replication virbr2 1 / 0 + vr1-dc0-storage virbr3 1 / 0 SUSPECT vr1-dc1-storage virbr3 1 / 0 + vr1-dc0-wan br-vr1-dc0-wan 1 / 0 SUSPECT vr1-dc1-wan br-vr1-dc1-wan 1 / 0 + (numbers are multicast_snooping / multicast_querier; exit 0 = reported, 7 bridges) + +This INDEPENDENTLY CONFIRMS the dc0 reading the mandate reports, and EXTENDS it: +dc1 is identical, and so is the WAN bridge. The jumphost's own two transit nets +(vr1-dc0-uplink virbr4, vr1-dc1-uplink virbr1) read the same -- so this is the +libvirt default across every bridge in the estate, not a per-rack setting. + +NO LINUX BEHAVIOUR IS ASSERTED, here or in the tool. The mandate said the exact +behaviour is unconfirmed and should not be asserted; the subcommand REPORTS the two +values, tags the 1/0 combination "SUSPECT", and prints a NOTE saying in terms that +it diagnoses nothing. What makes it worth having is that a red assertion (2) can be +triaged in one step instead of three -- and `node`'s no-reply + INCOMPLETE branch +names this subcommand by its exact command line. + +WHY IT SITS NEXT TO ASSERTION (2): D-139 records all six v6 subnets at +gateway_ip=None, so peers are on-link and Neighbour Discovery -- which is +multicast -- IS the resolution mechanism on these planes. + +CARRIED, NOT CLOSED: nothing here shows the plane FAILS. No node currently carries +these addresses up (Stage 5 deploy is in flight), so assertion (2) has never run +against real traffic. The instrument is built; the experiment is the operator's. + + +4. WHAT WAS RUN LIVE, AND WHAT THAT PROVED +-------------------------------------------------------------------------------- + bridges vr1-dc0 / vr1-dc1 on both racks exit 0, 7 bridges each (section 3) + bridges vr1-dc0 / vr1-dc1 on the jumphost exit 0, 1 bridge each (uplink only -- + the site-prefix filter correctly declines to invent plane bridges that are + not on this host) + plan vr1-dc0 on vr1-dc0-maas exit 3, REFUSE, over ssh: + "'maas vr1-dc0-region machines read' failed -- usage: maas [-h] COMMAND ..." + The ssh user on that VM (ubuntu) has NO ~/.maascli.db, so no profile is + logged in. This is the REFUSE contract working end-to-end on a real host: + "could not look" produced 3, not a green. + +NOT LIVE-VALIDATED, AND IT IS THIS TOOL'S LARGEST UNTESTED ASSUMPTION: that +`machines read` populates interface_set[].links[].subnet.cidr. It is NOT a guess -- +scripts/carve-host-interfaces.sh:120 and :293 both consume `.links[]?.subnet.cidr` +against the MAAS interface serialization, and dc-node-v6-carve.py:129-136 consumes +`links[].subnet.vlan.id` off `machines read` itself and successfully carved the 54 +statics -- but the exact field on the exact endpoint has not been seen by this +session. CLOSE IT WITH ONE COMMAND, from a shell that HAS a logged-in profile: + bash scripts/dc-node-v6-verify.sh plan vr1-dc0 +If the SPEC block comes back empty while `dc-node-v6-carve.py check vr1-dc0` is +green, that field is the reason -- and `plan` will say FAIL ("carries NO v6 link"), +which is loud, not silent. + +Cross-check for the tag question, REASONED not measured (no maas CLI on this +jumphost, so it entered no command): the 54 statics = 9 role nodes x 6 planes, and +lib-hosts.sh lists 10 machines per DC including juju-01. That arithmetic says the +openstack- tag selects role nodes only. `plan` does not DEPEND on it: it makes +no count assertion at all, only that every selected machine carries an IDENTICAL v6 +plane set. If the controller is ever tagged, its 2-plane carve turns that red -- and +that red is informative, not false. + + +5. DESIGN CHOICES A REVIEWER WOULD OTHERWISE HAVE TO RECONSTRUCT +-------------------------------------------------------------------------------- +* THREE SUBCOMMANDS, NOT ONE, and the split is the dc-cache-proxy/dc-snap-proxy + precedent: `check` asserts rack-side facts a client cannot satisfy, so pointing a + client at it fails for reasons that say nothing about the client. Same here -- + `node` has no maas CLI, no root, no repo; `bridges` needs virsh and sysfs. +* NO PLANE-COUNT LITERAL. "Six planes" is not asserted anywhere, because D-139 + makes lb-mgmt a seventh. `plan` asserts SET EQUALITY across machines instead, + which catches the real hazard (an under-carved node) and survives the change. +* SOLE-GLOBAL-ON-THE-NIC is deliberate and is the ULA->GUA rot catcher: after the + carve moves, a node that gained its GUA but never lost its ULA reads green on + "the expected address is present" and only this predicate turns red. + >>> THE ONE KNOWN WAY THIS PREDICATE CAN GO WRONG, AND IT IS NOT RED TODAY. <<< + D-139 ruling B carves a provider-public VIP /64 (2602:f3e2:f0X:11::/64). If + hacluster places a v6 VIP as a SECONDARY on the same netdev that carries the + node's own static (br-ex on provider-public, the metal-admin NIC on the admin + plane), then whichever unit currently HOLDS the VIP grows a second global there + and this predicate FAILs on that node alone, intermittently, as the VIP moves -- + the "gate that cannot reliably PASS" hazard dc-snap-proxy.sh:110-120 names. + NOT RED NOW: the v6 VIP row is RULED, NOT CARVED, and the -6 filter excludes + every existing v4 VIP. NO EXEMPTION WAS BUILT, deliberately: there is nothing to + measure one against and inventing the shape would be an inferred value (hard + rule 2). THE DISCRIMINATOR when it matters is simply whether the v6 VIP lands on + the SAME netdev as the node static; measure that on the first node that holds + one, then either scope this predicate or leave it. The FAIL message already + names an HA VIP as a benign cause and points here. + +* CARRIER (LOWER_UP) ON br-ex IS THE ONE OUTPUT SHAPE THIS BUILD NEVER SAW. + Every other `ip` shape parsed here came from a real reading or a stable + well-known format. Under Pattern A (lib-hosts.sh BREX_PARENT_NIC="enp2s0": the + provider-public static lands on the OVS bridge br-ex and the member NIC holds no + L3 link) `plan` WILL emit a br-ex entry, and `node` gates it on LOWER_UP in + `ip -o link show dev br-ex`. Whether ovs-vswitchd sets carrier on an OVS INTERNAL + port in this deployment is NOT confirmed and is not asserted here. + ATTEMPTED AND FAILED THIS SESSION: no reachable host carries a br-ex. The dc0 + rack has only lo/enp1s0/enp2s0/br-vr1-dc0-wan/virbr0-6/vnet*; vr1-dc0-juju + refused the key, and by lib-hosts.sh the controller has no br-ex anyway. + CLOSE IT WITH ONE READ-ONLY COMMAND on the first node whose br-ex is up: + ip -o link show dev br-ex + If LOWER_UP is ABSENT on a healthy br-ex, this predicate is a guaranteed false + red on provider-public for every role node and must be scoped to physical NICs. + Record the result either way -- a false red here would be read as a carve defect. +* THE FOUR QUADRANTS. ping-OK + neighbour FAILED/INCOMPLETE/absent is reported as a + CONTRADICTION rather than a pass, because one of the two readings is lying and a + gate must not close on that. ping-FAIL + neighbour RESOLVED is explicitly NOT + diagnosed as ND -- it says so, and points at filtering/routing instead. +* PEER SELECTION is deterministic (lowest address among the other holders) and the + peer hostname is printed, so a capture names which two machines were tested. A + plane with ONE holder gets peer '-', `plan` REFUSEs it, and `node` REFUSEs that + entry -- a peerless plane is unevaluable, never a pass. +* `| grep -q` under pipefail is avoided throughout; every match is a here-string or + an awk field comparison (the repo has been bitten by that short-circuit twice). +* TEST SEAM: DC_NODE_V6_SYSROOT (sysfs) plus fake maas/ip/ping/virsh on PATH -- + the DC_SNAP_PROXY_SYSROOT idiom. + + +6. HARNESS, MUTATION PASS, GATES +-------------------------------------------------------------------------------- +tests/dc-node-v6-verify/run-tests.sh -- 55 cases, 55 passed, 0 failed. + T1-T10 static: no v6/v4 literal in code, exit contract, header <=25 lines, + site+mode validation, node arm free of sudo/systemctl/virsh/maas, + ping-before-neighbour ordering. + T11-T25 plan driven against a fake maas: green (ULA fixture AND GUA fixture), + under-carve, zero-v6 machine, two links on one plane, peerless plane, + empty tag selection, MAAS error, unparseable JSON, absent maas CLI, + profile provenance, bad option. + T26-T46 node driven against fake ip/ping: green, address missing, tentative, + dadfailed, extra global, iface absent, no carrier, all four + reply/neighbour quadrants, unrecognised neighbour state, ping could not + run, peerless entry, malformed/missing SPEC, absent ip, absent ping + (address half still asserted), multi-entry evaluation. + T47-T54 bridges: reads both values, SUSPECT tagging, no false SUSPECT when a + querier exists, the no-diagnosis NOTE, unreadable sysfs, no plane + network, net with no bridge, absent virsh. + +MUTATION PASS -- three predicates neutered one at a time, harness re-run, restored, +sha256 re-checked. Baseline and post-restore sha256 of scripts/dc-node-v6-verify.sh +are IDENTICAL: 3ea87ced6352cf3aa0f4e8a354ae4831003b2dff7a156d880f5d8091e0c251ca +(the pass was RE-RUN in full after the extra-global FAIL message gained its HA-VIP +note, so the recorded sha is the delivered file's, not a pre-edit one) + + M1 node, the CONTRADICTION predicate: [ "$prc" -eq 0 ] -> [ "$prc" -eq 99 ] + RED: T37 "replies YET neighbour INCOMPLETE -> FAIL" + T38 "replies with NO neighbour entry at all -> FAIL" (53/2) + M2 node, the stale-prefix catcher: [ -n "$extra" ] -> [ -n "" ] + RED: T31 "an EXTRA global on the plane -> FAIL" (54/1) + M3 plan, the plane-set-identity predicate: `if s != modal` -> `if False` + RED: T15 "an under-carved machine -> FAIL" (54/1) + + All three are BEHAVIORAL, not textual: each mutation left the file syntactically + valid, and the tests that went red are the ones that DRIVE the branch, not greps + that match its text. + +A HARNESS BUG THE MUTATION DISCIPLINE DID NOT CATCH, but the first run did, and it +is the reason the "absent tool" cases are worth re-reading anywhere they appear: +the absent-ip and absent-virsh REFUSE cases were first driven with +PATH="/usr/bin:/bin". MEASURED: this jumphost carries `ip` AND `virsh` on that +path, so both cases silently executed the HOST's real binaries. They reported red +for the right reason by luck. Fixed with MINBIN -- a mktemp PATH holding symlinks +to exactly the coreutils the script uses and nothing else. + +repo-lint: see section 7. +gauntlet (bash scripts/run-tests-all.sh, > 2 min): see section 7. +tests/HARNESS-MANIFEST re-recorded per R15(2). REASON: a new harness directory, +tests/dc-node-v6-verify, was ADDED. Without the re-record the gauntlet's drift gate +FAILS with "present but UNPINNED", which is the gate working correctly. + + +7. GATE RESULTS +-------------------------------------------------------------------------------- +Revert line: docs/changelog-20260801-g19-node-v6-verify.md + + bash tests/dc-node-v6-verify/run-tests.sh 55 passed, 0 failed + bash scripts/run-tests-all.sh GAUNTLET: ALL GREEN (96 harnesses) + (took just over 2 minutes) + bash scripts/repo-lint.sh FAIL: 1 fail, 1 warn, 647 files + + The WARN is pre-existing and untouched by this change set: + [WARN] L1 docs/design-decisions.md: 239 non-ASCII byte(s) (legacy D-001..018 + carve-out; NEW entries must be ASCII) + The FAIL is L10 ONLY, and it is this session's instructions colliding by + construction -- verbatim: + [FAIL] L10 pending change set modifies state-bearing surface(s) + [docs/audit/: docs/audit/g19-ipv6-plane-gate-build-20260801.txt] + without touching docs/CURRENT-STATE.md -- GA-R1 rule 8 (C1): a + status-changing commit updates CURRENT-STATE.md in the SAME commit + The capture was MANDATORY and docs/CURRENT-STATE.md was OFF LIMITS (the operator + owns the G19 gate row). L10 clears the moment that row lands in the same commit. + NOTHING ELSE FAILED: L3 (ghost refs), L9 (self-ref) and the rest are clean, and + all three new files are ASCII + LF. + + Files: scripts/dc-node-v6-verify.sh 270 lines (24-line header, charter budget + 25); tests/dc-node-v6-verify/run-tests.sh 362 lines; this capture 232 lines. + The script is the length it is because it carries three distinct hosts' worth of + assertions and every FAIL/REFUSE branch has to say what to do next -- the prose + is compressed (no rationale, no worked examples, no alternatives-considered; + those live here and in the changelog), the discipline is not. diff --git a/docs/changelog-20260731-snap-proxy-apply-ipv6.md b/docs/changelog-20260731-snap-proxy-apply-ipv6.md index e053ffb..eaecca6 100644 --- a/docs/changelog-20260731-snap-proxy-apply-ipv6.md +++ b/docs/changelog-20260731-snap-proxy-apply-ipv6.md @@ -832,3 +832,102 @@ **Revert.** `maas vr1-dc0-region machine release t7ymp6` and `... fg6gxm` return both to `Ready` (`enable_disk_erasing_on_release=false`, so it takes minutes); `git rm` the capture and drop the CURRENT-STATE paragraph. Nothing else was mutated. + +--- + +## Item 19 -- BUILD G19: `scripts/dc-node-v6-verify.sh`, the IPv6 plane gate + +**FOLDED IN FROM `docs/changelog-20260801-g19-node-v6-verify.md`, which the build +agent wrote as a separate file and which is REMOVED here.** GA-R2/D1 is ONE changelog +per SESSION, never per topic -- two files means two review surfaces. The session +crossed midnight, which is why the agent dated a new one; that does not start a new +session. Original text preserved verbatim below the rule. + +### (folded) G19 build -- agent-authored text, verbatim + +**What.** New named executable gate answering two DIFFERENT questions: do the DC +nodes actually bring their IPv6 statics UP on every plane (asserted on the +INTERFACE, never on MAAS), and does a plane CARRY IPv6 between two nodes +(reachability to a peer plus the `ip -6 neigh` state that ping alone cannot +distinguish from a lie). Plus a rack-side diagnostic that reports each plane +bridge's multicast snooping/querier state so a red data-path result can be triaged +in one step. + +**Files added** + +| Path | Role | +|---|---| +| `scripts/dc-node-v6-verify.sh` | the gate: `plan` \| `node` \| `bridges` | +| `tests/dc-node-v6-verify/run-tests.sh` | 55-case harness, both directions driven | +| `docs/audit/g19-ipv6-plane-gate-build-20260801.txt` | build capture, live readings, mutation results | + +**File changed:** `tests/HARNESS-MANIFEST` (+1 line, `dc-node-v6-verify`). +R15(2) requires a deliberate re-record with a stated reason: **a new harness +directory was added.** Without it the gauntlet's drift gate FAILS with "present but +UNPINNED" -- the gate working correctly. + +**Governing decisions.** D-139 (VR1 goes IPv6-only east-west; whole plane carve +moves to GUA), D-101 (IPv6 unless IPv4 is necessary), D-134 (node octets). +Rationale lives there and in the capture, not in the script. + +**The prefix constraint, and how it is resolved.** D-139 retires the VR1 ULA `/48` +and moves every plane to GUA, so a site table of prefix literals would be correct +today and wrong on carve day. **There is no v6 prefix literal in this tool.** +`plan` derives the expectation from live MAAS and emits a per-machine SPEC; +`node` takes that SPEC as input and knows no prefixes of its own. Harness T3 +enforces the absence on non-comment lines; T14 drives the same green `plan` case +with a ULA fixture and a GUA fixture. + +**Scope of a green, stated in the script header too:** this asserts the NIC against +MAAS, not against D-139's ruled GUA table. MAAS-vs-ruling belongs to +`dc-node-v6-carve.py check` and `tests/d139-gua-carve`. + +**Gates.** Harness 55/55. Gauntlet ALL GREEN (96 harnesses). `repo-lint`: **1 fail, +1 warn.** The warn is the pre-existing legacy `D-001..018` non-ASCII carve-out. The +fail is **L10 only** -- this change set adds a `docs/audit/` file without touching +`docs/CURRENT-STATE.md`, which this session was instructed not to edit because the +operator owns the G19 gate row. It clears when that row lands, in the same commit. + +**Mutation pass** (each predicate neutered alone, harness re-run, restored, +`sha256sum` verified identical -- `3ea87ced...c251ca` before and after; the +pass was re-run in full after the last message edit): + +| Mutation | Predicate | NAMED test that went red | +|---|---|---| +| M1 | `node` contradiction branch `[ "$prc" -eq 0 ]` | T37, T38 | +| M2 | `node` extra-global catcher `[ -n "$extra" ]` | T31 | +| M3 | `plan` plane-set identity `s != modal` | T15 | + +**Live, read-only, this session.** `bridges` was run over the intended +`ssh 'bash -s -- bridges '` path on BOTH racks: all six plane bridges +plus the WAN bridge read `multicast_snooping=1 multicast_querier=0` on dc0 AND +dc1, independently confirming and extending the dc0 measurement this gate was +commissioned against. `plan` was exercised on `vr1-dc0-maas` and correctly returned +**REFUSE / exit 3** (no MAAS profile for the ssh user) -- the three-outcome contract +proven on a real host. No cloud state was mutated. + +**Two open checks, both in the capture, neither red today.** (a) The +sole-global-per-NIC predicate would FAIL on whichever node holds a v6 HA VIP *if* +D-139's ruled provider-public VIP `/64` lands as a secondary on the same netdev as +the node static. No exemption was built -- there is nothing to measure one against. +(b) `LOWER_UP` on an OVS-internal `br-ex` is the one `ip` output shape this build +never saw; no reachable host carries a br-ex. One `ip -o link show dev br-ex` on the +first node that has one settles it. + +**Carried, not closed.** No node currently carries these addresses up, so +assertion (2) has never run against real traffic; this is a fixture-and-diagnostic +green, not a plane that has been proven to work. And `plan`'s use of +`machines read -> interface_set[].links[].subnet.cidr` is corroborated by +`carve-host-interfaces.sh:120,293` but not yet seen live -- one `plan` run from a +shell with a logged-in profile closes it. Both are in the capture. + +**Also logged, NOT executed** (hard rule 1): `dc-node-v6-carve.py` pivots on "the +interface already carries a v4 static", which selects nothing on the five planes +D-139 strips of IPv4. It will silently under-carve after the v4-removal step. G19 +does not inherit the defect -- `plan` pivots on the link's subnet being v6. + +**Revert.** `git rm scripts/dc-node-v6-verify.sh tests/dc-node-v6-verify/run-tests.sh +docs/audit/g19-ipv6-plane-gate-build-20260801.txt docs/changelog-20260801-g19-node-v6-verify.md` +then `bash scripts/run-tests-all.sh --record-manifest`. Nothing else consumes these +files: no runbook, no `preflight.sh` sub-gate and no other script references them, +so removal is self-contained. diff --git a/scripts/dc-node-v6-verify.sh b/scripts/dc-node-v6-verify.sh new file mode 100644 index 0000000..9a6ab90 --- /dev/null +++ b/scripts/dc-node-v6-verify.sh @@ -0,0 +1,270 @@ +#!/usr/bin/env bash +# scripts/dc-node-v6-verify.sh [spec] -- gate G19: +# do the DC nodes actually BRING UP their IPv6 statics, and does a plane carry +# IPv6 between two nodes? site = vr1-dc0 | vr1-dc1. Governing: D-139 (GUA carve), +# D-101 (v6 unless v4 is necessary), D-134 (node octets). +# +# plan [--profile P] on a host holding the maas CLI. DERIVES the +# expectation from live MAAS and prints one SPEC per machine. NO prefix +# literal exists in this file: D-139 retires the VR1 ULA /48 for GUA, so a +# baked table would be right today and wrong on carve day. +# node ON the node: bash -s -- node . +# Read-only, no root; touches only ip and ping. +# bridges on the RACK. DIAGNOSTIC, never a verdict: prints +# each plane bridge's multicast_snooping/multicast_querier so a red `node` +# can be triaged in one step. Exits 0 (reported) or 3 (could not read). +# +# WHAT A GREEN DOES NOT MEAN: this asserts the NIC against MAAS, not against +# D-139's ruled GUA table. MAAS-vs-ruling is dc-node-v6-carve.py check and +# tests/d139-gua-carve. `plan` prints the derived prefixes, so a reader can see +# which carve is being asserted. +# +# SPEC := entry[,entry...]; entry := iface=subnet_cidr=addr=peer=peerhost +# peer '-' means no second machine on that plane -- that plane REFUSES. +# +# EXIT: 0 PASS | 1 FAIL | 2 bad args | 3 REFUSE (could not evaluate -- NEVER a +# pass). ASCII + LF. +set -uo pipefail + +MODE="${1:-}"; SITE="${2:-}" +case "$MODE" in plan|node|bridges) ;; *) + echo "usage: dc-node-v6-verify.sh [spec]" >&2; exit 2 ;; esac +case "$SITE" in vr1-dc0|vr1-dc1) ;; *) + echo "FAIL: unknown site '$SITE' (expected vr1-dc0 | vr1-dc1)" >&2; exit 2 ;; esac + +# TEST SEAM. EMPTY in every production invocation, so the sysfs paths below are +# byte-identical to the absolute ones. Precedent: DC_SNAP_PROXY_SYSROOT. +SYSROOT="${DC_NODE_V6_SYSROOT:-}" + +FAILED=0; REFUSED=0 +ok(){ echo " OK $1"; } +no(){ echo " FAIL $1"; FAILED=1; } +refuse(){ echo " REFUSE $1"; REFUSED=1; } +verdict(){ # verdict