g19-ipv6-node-plane-verify-20260801.txt
=======================================
IPv6 node-plane verification, vr1-dc0. Read-only ON the nodes.
Date: 2026-08-01T01:53:00Z  Operator-approved MAAS deploy of TWO storage
nodes (least-central role) purely to open a boot window; they are released afterwards.
Region ASSERTED before any MAAS call: profile vr1-dc0-region -> exactly ONE rack
controller, hot-kid/c3aqh8; profile admin -> voffice1,vvr1-dc0,vvr1-dc1. Distinguished.
Nodes: vr1-dc0-storage-01 t7ymp6 (::150) / vr1-dc0-storage-02 fg6gxm (::151), jammy.
Transport: ssh -J vr1-dc0-rack, office1_svc_ed25519 (deployed nodes carry the
MAAS-registered key). NOTE: these are ULA addresses -- D-139 moves the carve to GUA,
so the ADDRESSES here are pre-D-139; the PROPERTY tested is family-agnostic.

### A1 -- DO THE v6 ADDRESSES ACTUALLY COME UP ON THE INTERFACES?
Asserted on the INTERFACE, never on MAAS: 'MAAS holding a link is not the same as an
interface carrying it'. MAAS records 6 per node; below is what the NIC carries.
$ ip -6 -o addr show scope global   (on vr1-dc0-storage-01)
   enp1s0     fd50:840e:74e2:220::150/64
   enp3s0     fd50:840e:74e2:221::150/64
   enp4s0     fd50:840e:74e2:230::150/64
   enp5s0     fd50:840e:74e2:240::150/64
   enp6s0     fd50:840e:74e2:250::150/64
   br-ex      2602:f3e2:f02:10::150/64
   count=6
   tentative-or-dadfailed=0

$ ip -6 route show   (on-link routes, and the ABSENCE of a default)
   ::1 dev lo proto kernel metric 256 pref medium
   2602:f3e2:f02:10::/64 dev br-ex proto kernel metric 256 pref medium
   fd50:840e:74e2:220::/64 dev enp1s0 proto kernel metric 256 pref medium
   fd50:840e:74e2:221::/64 dev enp3s0 proto kernel metric 256 pref medium
   fd50:840e:74e2:230::/64 dev enp4s0 proto kernel metric 256 pref medium
   fd50:840e:74e2:240::/64 dev enp5s0 proto kernel metric 256 pref medium
   fd50:840e:74e2:250::/64 dev enp6s0 proto kernel metric 256 pref medium
   (no v6 default route -- expected, gateway_ip is None on all six subnets)

### A2 -- DOES THE PLANE CARRY v6 BETWEEN TWO NODES?
One node proves addresses come up and proves NOTHING about the data path. The neighbour
table was COLD (nodes had just booted), so the first solicitation had to go out as
MULTICAST to the solicited-node address -- which is the specific risk this was built to
catch: every plane bridge on the rack measures multicast_snooping=1 / multicast_querier=0.
$ ping6 -c3 -W2 -I <iface> <peer>   (storage-01 -> storage-02, all six planes)
   enp1s0   -> fd50:840e:74e2:220::151      loss=0%
   enp3s0   -> fd50:840e:74e2:221::151      loss=0%
   enp4s0   -> fd50:840e:74e2:230::151      loss=0%
   enp5s0   -> fd50:840e:74e2:240::151      loss=0%
   enp6s0   -> fd50:840e:74e2:250::151      loss=0%
   br-ex    -> 2602:f3e2:f02:10::151        loss=0%

$ ip -6 neigh show   (a 0% ping with a FAILED neighbour would be a contradiction)
   fd50:840e:74e2:240::151      enp5s0   REACHABLE
   fd50:840e:74e2:221::151      enp3s0   REACHABLE
   fd50:840e:74e2:220::151      enp1s0   REACHABLE
   2602:f3e2:f02:10::151        br-ex    DELAY
   fd50:840e:74e2:230::151      enp4s0   REACHABLE
   fd50:840e:74e2:220::6        enp1s0   STALE
   fd50:840e:74e2:250::151      enp6s0   DELAY

### A3 -- G17 ROLE-NODE HALF (never before captured on a role node; the dc0 G17
capture of 2026-07-30 was taken on the CONTROLLER VM and said so).
$ curl -fsS http://10.12.8.4/ubuntu/dists/jammy/Release  + body fields
   curl exit=0 http=200 bytes=269219
   OK   Origin: Ubuntu
   OK   Suite: jammy
   OK   Components:
   OK   Architectures:

$ per-suite probe -- is the D-135 jammy-backports amendment BUILT? (it was the
  2026-07-31 deploy blocker: backports answered 404 and 22 units failed install)
   jammy -> 200
   jammy-updates -> 200
   jammy-security -> 200
   jammy-backports -> 200

### A4 -- NODE TIME SOURCE (G17 assertion 2), AND THE HEADLINE FINDING
G17's text names 'chronyc sources'. chrony is NOT on the MAAS jammy image, so the gate
as written can only ever REFUSE (recorded 2026-07-30). The stack that EXISTS is
systemd-timesyncd and it asserts the same property.
   chrony present? NO
   NTP=yes
   NTPSynchronized=yes
   ServerName    = s "fd50:840e:74e2:220::6"
   ServerAddress = (iay) 10 16 253 80 132 14 116 226 2 32 0 0 0 0 0 0 0 6
   -- timedatectl timesync-status --
          Server: fd50:840e:74e2:220::6 (fd50:840e:74e2:220::6)
   Poll interval: 4min 16s (min: 32s; max 34min 8s)
            Leap: normal
         Version: 4
         Stratum: 3
       Reference: 5BBD5B9D
       Precision: 1us (-23)
   Root distance: 57.464ms (max: 5s)

INTERPRETATION OF A4: ServerAddress family 10 = AF_INET6, 16 bytes =
fd50:840e:74e2:220::6 -- the MAAS region VM's metal-admin v6 leg. So the node's time
source is (a) MAAS-served and NOT the DC edge, satisfying G17 assertion 2, and (b)
ALREADY RUNNING OVER IPv6 IN PRODUCTION, unprompted. This is not a test path: it is a
live operational dependency on IPv6 that predates any of this verification work.

### A5 -- THE G19 GATE ITSELF, RUN LIVE (added after the tool was built)
The tool exists so this stops being a hand-run measurement. Run against the SAME two
nodes while the boot window was still open, so it ships PROVEN, not fixture-green --
the snap proxy's harness was green for a week before anything was proven end to end.

$ ssh voffice1 "bash -s -- plan vr1-dc0 --profile vr1-dc0-region" < scripts/dc-node-v6-verify.sh
   -> PASS, EXIT 0. tag openstack-vr1-dc0 selects 9 machines; no machine holds two v6
      links on one plane; v6 plane set IDENTICAL across all 9 (6 planes).
   DERIVED prefixes (the file contains NO v6 literal):
      2602:f3e2:f02:10::/64  fd50:840e:74e2:{220,221,230,240,250}::/64

   FINDING -- PEER SELECTION IS DEPLOYMENT-BLIND. plan picked vr1-dc0-control-01
   (::100) as peer for every machine. control-01 is NOT deployed (only storage-01/02
   are), so the derived SPEC cannot pass in a PARTIAL deployment: every plane would
   fail against a powered-off peer. Correct for the full-deployment case the gate
   targets, wrong for exactly the bounded experiment it was built to support. Worked
   around here by substituting the live peer; LOGGED, NOT FIXED.

$ ssh <storage-01> "bash -s -- node vr1-dc0 <spec>" < scripts/dc-node-v6-verify.sh
   (spec = plan output for storage-01 with the peer substituted to the LIVE
    storage-02 ::151 -- one sed, no other change)
   dc-node-v6-verify node (vr1-dc0) -- on vr1-dc0-storage-01
     OK      br-ex (2602:f3e2:f02:10::/64): 2602:f3e2:f02:10::150/64 up, global, DAD complete, sole global on the NIC
     OK      br-ex (2602:f3e2:f02:10::/64): 2602:f3e2:f02:10::151 (vr1-dc0-storage-02) replies, neighbour DELAY
     OK      enp1s0 (fd50:840e:74e2:220::/64): fd50:840e:74e2:220::150/64 up, global, DAD complete, sole global on the NIC
     OK      enp1s0 (fd50:840e:74e2:220::/64): fd50:840e:74e2:220::151 (vr1-dc0-storage-02) replies, neighbour DELAY
     OK      enp3s0 (fd50:840e:74e2:221::/64): fd50:840e:74e2:221::150/64 up, global, DAD complete, sole global on the NIC
     OK      enp3s0 (fd50:840e:74e2:221::/64): fd50:840e:74e2:221::151 (vr1-dc0-storage-02) replies, neighbour DELAY
     OK      enp4s0 (fd50:840e:74e2:230::/64): fd50:840e:74e2:230::150/64 up, global, DAD complete, sole global on the NIC
     OK      enp4s0 (fd50:840e:74e2:230::/64): fd50:840e:74e2:230::151 (vr1-dc0-storage-02) replies, neighbour DELAY
     OK      enp5s0 (fd50:840e:74e2:240::/64): fd50:840e:74e2:240::150/64 up, global, DAD complete, sole global on the NIC
     OK      enp5s0 (fd50:840e:74e2:240::/64): fd50:840e:74e2:240::151 (vr1-dc0-storage-02) replies, neighbour DELAY
     OK      enp6s0 (fd50:840e:74e2:250::/64): fd50:840e:74e2:250::150/64 up, global, DAD complete, sole global on the NIC
     OK      enp6s0 (fd50:840e:74e2:250::/64): fd50:840e:74e2:250::151 (vr1-dc0-storage-02) replies, neighbour DELAY
   dc-node-v6-verify node (vr1-dc0): PASS
   EXIT=0

$ ssh vr1-dc0-rack "bash -s -- bridges vr1-dc0" < scripts/dc-node-v6-verify.sh   (DIAGNOSTIC)
   dc-node-v6-verify bridges (vr1-dc0) -- DIAGNOSTIC ONLY, no verdict
     READ    vr1-dc0-data-tenant          virbr4     multicast_snooping=1 multicast_querier=0  <-- SUSPECT
     READ    vr1-dc0-metal-admin          virbr2     multicast_snooping=1 multicast_querier=0  <-- SUSPECT
     READ    vr1-dc0-metal-internal       virbr6     multicast_snooping=1 multicast_querier=0  <-- SUSPECT
     READ    vr1-dc0-provider-public      virbr5     multicast_snooping=1 multicast_querier=0  <-- SUSPECT
     READ    vr1-dc0-replication          virbr1     multicast_snooping=1 multicast_querier=0  <-- SUSPECT
     READ    vr1-dc0-storage              virbr3     multicast_snooping=1 multicast_querier=0  <-- SUSPECT
     READ    vr1-dc0-wan                  br-vr1-dc0-wan multicast_snooping=1 multicast_querier=0  <-- SUSPECT
     NOTE    snooping=1 with querier=0 is REPORTED as suspect, not diagnosed: IPv6
             Neighbour Discovery is multicast, and no Linux behaviour is asserted here.
   dc-node-v6-verify bridges (vr1-dc0): reported 7 bridge(s)