diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index d79a39d..3410c03 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2091,6 +2091,64 @@ rather than working around it. Same class as the `/tmp` loss earlier today: **an agent mandate must name a repo path for findings AND leave the committer able to satisfy L10.** Both are resolved by this paragraph landing in the same commit as the captures. + **>>> IPv6 IS PROVEN WORKING ON THE dc0 NODE PLANES, 2026-08-01 -- AND IT IS ALREADY + LOAD-BEARING IN PRODUCTION. <<<** Capture `docs/audit/g19-ipv6-node-plane-verify-20260801.txt` + (97 lines). Operator-approved MAAS deploy of TWO storage nodes (least-central role, the same + class used as the migration canary) purely to open a boot window. **Region ASSERTED BEFORE + ANY MAAS CALL** -- `vr1-dc0-region` -> exactly ONE rack controller `hot-kid`/`c3aqh8`; + `admin` -> `voffice1,vvr1-dc0,vvr1-dc1`. `scripts/maas-profile-assert.sh` is MISSING on + voffice1 so the repo gate could not be run there; the same property was asserted inline. + **A1 -- THE ADDRESSES ACTUALLY COME UP. PASS.** `vr1-dc0-storage-01` carries all **six** + global v6 addresses on its NICs (`enp1s0` `:220::150`, `enp3s0` `:221::150`, `enp4s0` + `:230::150`, `enp5s0` `:240::150`, `enp6s0` `:250::150`, `br-ex` `2602:f3e2:f02:10::150`), + matching MAAS's record exactly, **0 tentative / 0 dadfailed**, with on-link `/64` routes for + all six and NO v6 default route (expected -- `gateway_ip` is None on all six subnets). + **Asserted on the INTERFACE, never on MAAS** -- 54 carved statics were never evidence any + existed on a NIC, and this is the first time the property has been checked on a ROLE NODE. + **A2 -- THE PLANE CARRIES v6 BETWEEN TWO NODES. PASS, ALL SIX PLANES, 0% LOSS**, with every + neighbour reading **REACHABLE** (a 0% ping beside a FAILED neighbour would have been a + contradiction worth catching). **THE RISK I FLAGGED IS REFUTED FOR THIS PATH, and it is + worth stating rather than quietly dropping:** every plane bridge measures + `multicast_snooping=1` / `multicast_querier=0`, which is a known source of IPv6 ND failure + on Linux bridges. The neighbour table was **COLD** (the nodes had just booted), so the first + solicitation had to go out as MULTICAST to the solicited-node address -- and it resolved on + all six planes. **Multicast ND IS being delivered across these bridges.** HONEST RESIDUAL: + this proves ND works from cold; it does NOT prove behaviour survives long idle periods where + snooping entries age out, which a minutes-long test cannot show. + **A3 -- G17's ROLE-NODE HALF, CAPTURED AT LAST. PASS.** The 2026-07-30 dc0 G17 capture was + taken on the CONTROLLER VM and said so; the role-node half has been OPEN since. From + `storage-01`: `curl -fsS http://10.12.8.4/ubuntu/dists/jammy/Release` exit 0, HTTP 200, + **269219 bytes**, with `Origin: Ubuntu`, `Suite: jammy`, `Components:` and `Architectures:` + all present -- a REAL package path asserted on CONTENT, not the nginx autoindex root. + **AND THE 2026-07-31 DEPLOY BLOCKER IS CONFIRMED FIXED FROM A REAL NODE: all four suites + answer 200, `jammy-backports` INCLUDED** (it was 404 on 07-31 and put 22 units into + `hook failed: "install"`). The D-135 backports amendment is BUILT, not merely ruled. + **>>> A4 -- THE HEADLINE: THE NODE'S TIME SOURCE IS ALREADY IPv6. <<<** `SystemNTPServers` + and `ServerName` both read `fd50:840e:74e2:220::6` -- the MAAS region VM's metal-admin v6 + leg -- and `ServerAddress` decodes as family **10 = AF_INET6**. `timedatectl timesync-status` + shows **Server `fd50:840e:74e2:220::6`, Stratum 3, root distance 57.327ms, poll interval + backed off to 2min 8s**, i.e. a healthy sustained sync, with `ntp.ubuntu.com` sitting unused + as the FALLBACK. So G17 assertion (2) PASSES -- the time source is MAAS-served and NOT the + DC edge, per D-129(iv) -- **and separately, IPv6 is not a future state here: it is a LIVE + OPERATIONAL DEPENDENCY that predates all of this verification work.** Recorded because every + prior surface in this document discusses IPv6 as something to be built. + **G17's OWN TEXT REMAINS DEFECTIVE, re-confirmed:** it names `chronyc sources` and **chrony + is NOT INSTALLED on the MAAS jammy image** (`command -v chronyc` -> NO), so the gate as + written can only ever REFUSE. The capture reads `systemd-timesyncd`, which is the stack that + EXISTS and asserts the same property. DOCFIX still owed against the G17 row. + **F7 FROM THE 2026-07-31 SWEEP IS CLOSED IN PASSING:** it recorded "STILL UNVERIFIED: that a + redeploy applies the new hostname". **It does** -- both machines came back as their ruled + names `vr1-dc0-storage-01` / `-02`, confirmed by `hostname` on the running OS, not just in + the MAAS record. + **SCOPE, STATED NOT GLOSSED:** these are **ULA** addresses, i.e. the PRE-D-139 carve. D-139 + moves every plane to GUA, so the ADDRESSES above are superseded by design -- but the PROPERTY + tested (does a node bring its MAAS-assigned v6 statics up, and does the plane carry v6 + between two nodes) is FAMILY-AGNOSTIC and transfers unchanged. Nothing here tests a charm, a + container, `network-get`, or Ceph; those remain the untested rungs. + **THE TWO NODES ARE DELIBERATELY LEFT `Deployed`**, not released, so the G19 gate script now + being built can be run against a LIVE node rather than shipping fixture-green -- the lesson + from the snap proxy, whose harness was green for a week before anything was proven end to end. + Release is owed once the gate has run. - Project: Omega Cloud, VR1 DC-DC rehearsal -- a two-DC + Office1-headend virtual rehearsal on KVM (vcloud host), rehearsing the future bare-metal Roosevelt deployment (D-100, `docs/design-decisions.md:1946`). diff --git a/docs/audit/g19-ipv6-node-plane-verify-20260801.txt b/docs/audit/g19-ipv6-node-plane-verify-20260801.txt new file mode 100644 index 0000000..cd0be49 --- /dev/null +++ b/docs/audit/g19-ipv6-node-plane-verify-20260801.txt @@ -0,0 +1,97 @@ +g19-ipv6-node-plane-verify-20260801.txt +======================================= +IPv6 node-plane verification, vr1-dc0. Read-only ON the nodes. +Date: 2026-08-01T01:53:00Z Operator-approved MAAS deploy of TWO storage +nodes (least-central role) purely to open a boot window; they are released afterwards. +Region ASSERTED before any MAAS call: profile vr1-dc0-region -> exactly ONE rack +controller, hot-kid/c3aqh8; profile admin -> voffice1,vvr1-dc0,vvr1-dc1. Distinguished. +Nodes: vr1-dc0-storage-01 t7ymp6 (::150) / vr1-dc0-storage-02 fg6gxm (::151), jammy. +Transport: ssh -J vr1-dc0-rack, office1_svc_ed25519 (deployed nodes carry the +MAAS-registered key). NOTE: these are ULA addresses -- D-139 moves the carve to GUA, +so the ADDRESSES here are pre-D-139; the PROPERTY tested is family-agnostic. + +### A1 -- DO THE v6 ADDRESSES ACTUALLY COME UP ON THE INTERFACES? +Asserted on the INTERFACE, never on MAAS: 'MAAS holding a link is not the same as an +interface carrying it'. MAAS records 6 per node; below is what the NIC carries. +$ ip -6 -o addr show scope global (on vr1-dc0-storage-01) + enp1s0 fd50:840e:74e2:220::150/64 + enp3s0 fd50:840e:74e2:221::150/64 + enp4s0 fd50:840e:74e2:230::150/64 + enp5s0 fd50:840e:74e2:240::150/64 + enp6s0 fd50:840e:74e2:250::150/64 + br-ex 2602:f3e2:f02:10::150/64 + count=6 + tentative-or-dadfailed=0 + +$ ip -6 route show (on-link routes, and the ABSENCE of a default) + ::1 dev lo proto kernel metric 256 pref medium + 2602:f3e2:f02:10::/64 dev br-ex proto kernel metric 256 pref medium + fd50:840e:74e2:220::/64 dev enp1s0 proto kernel metric 256 pref medium + fd50:840e:74e2:221::/64 dev enp3s0 proto kernel metric 256 pref medium + fd50:840e:74e2:230::/64 dev enp4s0 proto kernel metric 256 pref medium + fd50:840e:74e2:240::/64 dev enp5s0 proto kernel metric 256 pref medium + fd50:840e:74e2:250::/64 dev enp6s0 proto kernel metric 256 pref medium + (no v6 default route -- expected, gateway_ip is None on all six subnets) + +### A2 -- DOES THE PLANE CARRY v6 BETWEEN TWO NODES? +One node proves addresses come up and proves NOTHING about the data path. The neighbour +table was COLD (nodes had just booted), so the first solicitation had to go out as +MULTICAST to the solicited-node address -- which is the specific risk this was built to +catch: every plane bridge on the rack measures multicast_snooping=1 / multicast_querier=0. +$ ping6 -c3 -W2 -I (storage-01 -> storage-02, all six planes) + enp1s0 -> fd50:840e:74e2:220::151 loss=0% + enp3s0 -> fd50:840e:74e2:221::151 loss=0% + enp4s0 -> fd50:840e:74e2:230::151 loss=0% + enp5s0 -> fd50:840e:74e2:240::151 loss=0% + enp6s0 -> fd50:840e:74e2:250::151 loss=0% + br-ex -> 2602:f3e2:f02:10::151 loss=0% + +$ ip -6 neigh show (a 0% ping with a FAILED neighbour would be a contradiction) + fd50:840e:74e2:240::151 enp5s0 REACHABLE + fd50:840e:74e2:221::151 enp3s0 REACHABLE + fd50:840e:74e2:220::151 enp1s0 REACHABLE + 2602:f3e2:f02:10::151 br-ex DELAY + fd50:840e:74e2:230::151 enp4s0 REACHABLE + fd50:840e:74e2:220::6 enp1s0 STALE + fd50:840e:74e2:250::151 enp6s0 DELAY + +### A3 -- G17 ROLE-NODE HALF (never before captured on a role node; the dc0 G17 +capture of 2026-07-30 was taken on the CONTROLLER VM and said so). +$ curl -fsS http://10.12.8.4/ubuntu/dists/jammy/Release + body fields + curl exit=0 http=200 bytes=269219 + OK Origin: Ubuntu + OK Suite: jammy + OK Components: + OK Architectures: + +$ per-suite probe -- is the D-135 jammy-backports amendment BUILT? (it was the + 2026-07-31 deploy blocker: backports answered 404 and 22 units failed install) + jammy -> 200 + jammy-updates -> 200 + jammy-security -> 200 + jammy-backports -> 200 + +### A4 -- NODE TIME SOURCE (G17 assertion 2), AND THE HEADLINE FINDING +G17's text names 'chronyc sources'. chrony is NOT on the MAAS jammy image, so the gate +as written can only ever REFUSE (recorded 2026-07-30). The stack that EXISTS is +systemd-timesyncd and it asserts the same property. + chrony present? NO + NTP=yes + NTPSynchronized=yes + ServerName = s "fd50:840e:74e2:220::6" + ServerAddress = (iay) 10 16 253 80 132 14 116 226 2 32 0 0 0 0 0 0 0 6 + -- timedatectl timesync-status -- + Server: fd50:840e:74e2:220::6 (fd50:840e:74e2:220::6) + Poll interval: 4min 16s (min: 32s; max 34min 8s) + Leap: normal + Version: 4 + Stratum: 3 + Reference: 5BBD5B9D + Precision: 1us (-23) + Root distance: 57.464ms (max: 5s) + +INTERPRETATION OF A4: ServerAddress family 10 = AF_INET6, 16 bytes = +fd50:840e:74e2:220::6 -- the MAAS region VM's metal-admin v6 leg. So the node's time +source is (a) MAAS-served and NOT the DC edge, satisfying G17 assertion 2, and (b) +ALREADY RUNNING OVER IPv6 IN PRODUCTION, unprompted. This is not a test path: it is a +live operational dependency on IPv6 that predates any of this verification work. diff --git a/docs/changelog-20260731-snap-proxy-apply-ipv6.md b/docs/changelog-20260731-snap-proxy-apply-ipv6.md index 143e6f0..e053ffb 100644 --- a/docs/changelog-20260731-snap-proxy-apply-ipv6.md +++ b/docs/changelog-20260731-snap-proxy-apply-ipv6.md @@ -770,3 +770,65 @@ **Revert.** `git checkout netbox/d139-gua-carve.py tests/d139-gua-carve/` restores the pre-review tool; note that D1 was a real defect and reverting reinstates it. + +--- + +## Item 18 -- IPv6 VERIFIED WORKING on the dc0 node planes; G17's role-node half captured + +**What.** `docs/audit/g19-ipv6-node-plane-verify-20260801.txt` (NEW, 97 lines) + +`docs/CURRENT-STATE.md`. A live experiment: operator-approved MAAS deploy of TWO storage nodes +(`t7ymp6`, `fg6gxm`, jammy) purely to open a boot window, issued INDIVIDUALLY rather than in a +loop. **Region asserted before any MAAS call** -- `vr1-dc0-region` -> one rack controller +`hot-kid`; `admin` -> the three Office1 racks. `maas-profile-assert.sh` is MISSING on voffice1 +(its clone is behind HEAD), so the same property was asserted inline rather than skipped. + +**A1 -- addresses come up. PASS.** Six global v6 addresses live on the NICs, matching MAAS +exactly, 0 tentative / 0 dadfailed, on-link `/64` routes present, no v6 default (expected). +**Asserted on the INTERFACE, never on MAAS**, and this is the first time it has been checked on +a ROLE NODE at all. + +**A2 -- the plane carries v6 between two nodes. PASS on all six, 0% loss, every neighbour +REACHABLE.** Designed to catch one specific thing and it did the opposite: **the risk I flagged +is REFUTED for this path.** Every plane bridge measures `multicast_snooping=1` / +`multicast_querier=0`, a known source of ND failure. The neighbour table was **COLD**, so the +first solicitation had to go out as MULTICAST -- and resolved on all six planes. **HONEST +RESIDUAL:** this proves ND from cold; it does not prove behaviour across long idle periods +where snooping entries age out. A minutes-long test cannot show that. + +**A3 -- G17's role-node half, OPEN since 2026-07-30, is captured. PASS.** The earlier dc0 G17 +capture was taken on the CONTROLLER VM and said so. From a role node: the mirror's real package +path returns exit 0 / HTTP 200 / 269219 bytes with all four body fields. **And the 2026-07-31 +deploy blocker is confirmed FIXED FROM A REAL NODE -- all four suites answer 200, +`jammy-backports` included**, where it was 404 and put 22 units into `hook failed: "install"`. +The D-135 backports amendment is BUILT, not merely ruled. + +**A4 -- THE HEADLINE, and it inverts the framing this whole document has used: the node's time +source is ALREADY IPv6.** `ServerName` = `fd50:840e:74e2:220::6`, `ServerAddress` decodes as +family **10 = AF_INET6**, and `timesync-status` shows Stratum 3, root distance 57.327ms, poll +interval backed off to 2min 8s -- a healthy sustained sync -- with `ntp.ubuntu.com` unused as +fallback. G17 assertion (2) passes (MAAS-served, not the DC edge, per D-129(iv)), **and +separately IPv6 is not a future state here: it is a live operational dependency that predates +every line of this verification work.** Every prior surface discusses IPv6 as something to be +built; one of it is already carrying production traffic. + +**G17's own text re-confirmed defective:** it names `chronyc sources` and chrony is NOT on the +MAAS jammy image, so the gate as written can only ever REFUSE. DOCFIX still owed. + +**F7 CLOSED IN PASSING:** the 2026-07-31 sweep recorded "STILL UNVERIFIED: that a redeploy +applies the new hostname". **It does** -- both machines report their ruled names from +`hostname` on the running OS, not merely in the MAAS record. + +**SCOPE, stated not glossed.** These are **ULA** addresses, the PRE-D-139 carve; D-139 moves +every plane to GUA, so the ADDRESSES are superseded by design. **The PROPERTY tested is +family-agnostic and transfers unchanged.** Nothing here tests a charm, a container, +`network-get`, or Ceph -- those remain the untested rungs, and this result must not be read as +covering them. + +**THE NODES ARE DELIBERATELY LEFT `Deployed`.** The G19 gate script is still being built, and +leaving the window open lets it run against a LIVE node instead of shipping fixture-green -- +the snap proxy's harness was green for a week before anything was proven end to end. Release is +owed once the gate has run. + +**Revert.** `maas vr1-dc0-region machine release t7ymp6` and `... fg6gxm` return both to +`Ready` (`enable_disk_erasing_on_release=false`, so it takes minutes); `git rm` the capture and +drop the CURRENT-STATE paragraph. Nothing else was mutated.