g19-ipv6-node-plane-verify-20260801.txt
=======================================
IPv6 node-plane verification, vr1-dc0. Read-only ON the nodes.
Date: 2026-08-01T01:53:00Z Operator-approved MAAS deploy of TWO storage
nodes (least-central role) purely to open a boot window; they are released afterwards.
Region ASSERTED before any MAAS call: profile vr1-dc0-region -> exactly ONE rack
controller, hot-kid/c3aqh8; profile admin -> voffice1,vvr1-dc0,vvr1-dc1. Distinguished.
Nodes: vr1-dc0-storage-01 t7ymp6 (::150) / vr1-dc0-storage-02 fg6gxm (::151), jammy.
Transport: ssh -J vr1-dc0-rack, office1_svc_ed25519 (deployed nodes carry the
MAAS-registered key). NOTE: these are ULA addresses -- D-139 moves the carve to GUA,
so the ADDRESSES here are pre-D-139; the PROPERTY tested is family-agnostic.
### A1 -- DO THE v6 ADDRESSES ACTUALLY COME UP ON THE INTERFACES?
Asserted on the INTERFACE, never on MAAS: 'MAAS holding a link is not the same as an
interface carrying it'. MAAS records 6 per node; below is what the NIC carries.
$ ip -6 -o addr show scope global (on vr1-dc0-storage-01)
enp1s0 fd50:840e:74e2:220::150/64
enp3s0 fd50:840e:74e2:221::150/64
enp4s0 fd50:840e:74e2:230::150/64
enp5s0 fd50:840e:74e2:240::150/64
enp6s0 fd50:840e:74e2:250::150/64
br-ex 2602:f3e2:f02:10::150/64
count=6
tentative-or-dadfailed=0
$ ip -6 route show (on-link routes, and the ABSENCE of a default)
::1 dev lo proto kernel metric 256 pref medium
2602:f3e2:f02:10::/64 dev br-ex proto kernel metric 256 pref medium
fd50:840e:74e2:220::/64 dev enp1s0 proto kernel metric 256 pref medium
fd50:840e:74e2:221::/64 dev enp3s0 proto kernel metric 256 pref medium
fd50:840e:74e2:230::/64 dev enp4s0 proto kernel metric 256 pref medium
fd50:840e:74e2:240::/64 dev enp5s0 proto kernel metric 256 pref medium
fd50:840e:74e2:250::/64 dev enp6s0 proto kernel metric 256 pref medium
(no v6 default route -- expected, gateway_ip is None on all six subnets)
### A2 -- DOES THE PLANE CARRY v6 BETWEEN TWO NODES?
One node proves addresses come up and proves NOTHING about the data path. The neighbour
table was COLD (nodes had just booted), so the first solicitation had to go out as
MULTICAST to the solicited-node address -- which is the specific risk this was built to
catch: every plane bridge on the rack measures multicast_snooping=1 / multicast_querier=0.
$ ping6 -c3 -W2 -I <iface> <peer> (storage-01 -> storage-02, all six planes)
enp1s0 -> fd50:840e:74e2:220::151 loss=0%
enp3s0 -> fd50:840e:74e2:221::151 loss=0%
enp4s0 -> fd50:840e:74e2:230::151 loss=0%
enp5s0 -> fd50:840e:74e2:240::151 loss=0%
enp6s0 -> fd50:840e:74e2:250::151 loss=0%
br-ex -> 2602:f3e2:f02:10::151 loss=0%
$ ip -6 neigh show (a 0% ping with a FAILED neighbour would be a contradiction)
fd50:840e:74e2:240::151 enp5s0 REACHABLE
fd50:840e:74e2:221::151 enp3s0 REACHABLE
fd50:840e:74e2:220::151 enp1s0 REACHABLE
2602:f3e2:f02:10::151 br-ex DELAY
fd50:840e:74e2:230::151 enp4s0 REACHABLE
fd50:840e:74e2:220::6 enp1s0 STALE
fd50:840e:74e2:250::151 enp6s0 DELAY
### A3 -- G17 ROLE-NODE HALF (never before captured on a role node; the dc0 G17
capture of 2026-07-30 was taken on the CONTROLLER VM and said so).
$ curl -fsS http://10.12.8.4/ubuntu/dists/jammy/Release + body fields
curl exit=0 http=200 bytes=269219
OK Origin: Ubuntu
OK Suite: jammy
OK Components:
OK Architectures:
$ per-suite probe -- is the D-135 jammy-backports amendment BUILT? (it was the
2026-07-31 deploy blocker: backports answered 404 and 22 units failed install)
jammy -> 200
jammy-updates -> 200
jammy-security -> 200
jammy-backports -> 200
### A4 -- NODE TIME SOURCE (G17 assertion 2), AND THE HEADLINE FINDING
G17's text names 'chronyc sources'. chrony is NOT on the MAAS jammy image, so the gate
as written can only ever REFUSE (recorded 2026-07-30). The stack that EXISTS is
systemd-timesyncd and it asserts the same property.
chrony present? NO
NTP=yes
NTPSynchronized=yes
ServerName = s "fd50:840e:74e2:220::6"
ServerAddress = (iay) 10 16 253 80 132 14 116 226 2 32 0 0 0 0 0 0 0 6
-- timedatectl timesync-status --
Server: fd50:840e:74e2:220::6 (fd50:840e:74e2:220::6)
Poll interval: 4min 16s (min: 32s; max 34min 8s)
Leap: normal
Version: 4
Stratum: 3
Reference: 5BBD5B9D
Precision: 1us (-23)
Root distance: 57.464ms (max: 5s)
INTERPRETATION OF A4: ServerAddress family 10 = AF_INET6, 16 bytes =
fd50:840e:74e2:220::6 -- the MAAS region VM's metal-admin v6 leg. So the node's time
source is (a) MAAS-served and NOT the DC edge, satisfying G17 assertion 2, and (b)
ALREADY RUNNING OVER IPv6 IN PRODUCTION, unprompted. This is not a test path: it is a
live operational dependency on IPv6 that predates any of this verification work.
### A5 -- THE G19 GATE ITSELF, RUN LIVE (added after the tool was built)
The tool exists so this stops being a hand-run measurement. Run against the SAME two
nodes while the boot window was still open, so it ships PROVEN, not fixture-green --
the snap proxy's harness was green for a week before anything was proven end to end.
$ ssh voffice1 "bash -s -- plan vr1-dc0 --profile vr1-dc0-region" < scripts/dc-node-v6-verify.sh
-> PASS, EXIT 0. tag openstack-vr1-dc0 selects 9 machines; no machine holds two v6
links on one plane; v6 plane set IDENTICAL across all 9 (6 planes).
DERIVED prefixes (the file contains NO v6 literal):
2602:f3e2:f02:10::/64 fd50:840e:74e2:{220,221,230,240,250}::/64
FINDING -- PEER SELECTION IS DEPLOYMENT-BLIND. plan picked vr1-dc0-control-01
(::100) as peer for every machine. control-01 is NOT deployed (only storage-01/02
are), so the derived SPEC cannot pass in a PARTIAL deployment: every plane would
fail against a powered-off peer. Correct for the full-deployment case the gate
targets, wrong for exactly the bounded experiment it was built to support. Worked
around here by substituting the live peer; LOGGED, NOT FIXED.
$ ssh <storage-01> "bash -s -- node vr1-dc0 <spec>" < scripts/dc-node-v6-verify.sh
(spec = plan output for storage-01 with the peer substituted to the LIVE
storage-02 ::151 -- one sed, no other change)
dc-node-v6-verify node (vr1-dc0) -- on vr1-dc0-storage-01
OK br-ex (2602:f3e2:f02:10::/64): 2602:f3e2:f02:10::150/64 up, global, DAD complete, sole global on the NIC
OK br-ex (2602:f3e2:f02:10::/64): 2602:f3e2:f02:10::151 (vr1-dc0-storage-02) replies, neighbour DELAY
OK enp1s0 (fd50:840e:74e2:220::/64): fd50:840e:74e2:220::150/64 up, global, DAD complete, sole global on the NIC
OK enp1s0 (fd50:840e:74e2:220::/64): fd50:840e:74e2:220::151 (vr1-dc0-storage-02) replies, neighbour DELAY
OK enp3s0 (fd50:840e:74e2:221::/64): fd50:840e:74e2:221::150/64 up, global, DAD complete, sole global on the NIC
OK enp3s0 (fd50:840e:74e2:221::/64): fd50:840e:74e2:221::151 (vr1-dc0-storage-02) replies, neighbour DELAY
OK enp4s0 (fd50:840e:74e2:230::/64): fd50:840e:74e2:230::150/64 up, global, DAD complete, sole global on the NIC
OK enp4s0 (fd50:840e:74e2:230::/64): fd50:840e:74e2:230::151 (vr1-dc0-storage-02) replies, neighbour DELAY
OK enp5s0 (fd50:840e:74e2:240::/64): fd50:840e:74e2:240::150/64 up, global, DAD complete, sole global on the NIC
OK enp5s0 (fd50:840e:74e2:240::/64): fd50:840e:74e2:240::151 (vr1-dc0-storage-02) replies, neighbour DELAY
OK enp6s0 (fd50:840e:74e2:250::/64): fd50:840e:74e2:250::150/64 up, global, DAD complete, sole global on the NIC
OK enp6s0 (fd50:840e:74e2:250::/64): fd50:840e:74e2:250::151 (vr1-dc0-storage-02) replies, neighbour DELAY
dc-node-v6-verify node (vr1-dc0): PASS
EXIT=0
$ ssh vr1-dc0-rack "bash -s -- bridges vr1-dc0" < scripts/dc-node-v6-verify.sh (DIAGNOSTIC)
dc-node-v6-verify bridges (vr1-dc0) -- DIAGNOSTIC ONLY, no verdict
READ vr1-dc0-data-tenant virbr4 multicast_snooping=1 multicast_querier=0 <-- SUSPECT
READ vr1-dc0-metal-admin virbr2 multicast_snooping=1 multicast_querier=0 <-- SUSPECT
READ vr1-dc0-metal-internal virbr6 multicast_snooping=1 multicast_querier=0 <-- SUSPECT
READ vr1-dc0-provider-public virbr5 multicast_snooping=1 multicast_querier=0 <-- SUSPECT
READ vr1-dc0-replication virbr1 multicast_snooping=1 multicast_querier=0 <-- SUSPECT
READ vr1-dc0-storage virbr3 multicast_snooping=1 multicast_querier=0 <-- SUSPECT
READ vr1-dc0-wan br-vr1-dc0-wan multicast_snooping=1 multicast_querier=0 <-- SUSPECT
NOTE snooping=1 with querier=0 is REPORTED as suspect, not diagnosed: IPv6
Neighbour Discovery is multicast, and no Linux behaviour is asserted here.
dc-node-v6-verify bridges (vr1-dc0): reported 7 bridge(s)
