diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 5776758..f39ef1e 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -232,6 +232,58 @@ reach it to run the check, and the fault was still being diagnosed. G17 stays OPEN and re-arms at the next successful boot; its checks must run FROM the node or from the rack, which is on-segment. + **^ SUPERSEDED SAME DAY -- G17 (vr1-dc0) WAS CAPTURED, and BOTH GATED ASSERTIONS PASS.** + Capture `docs/audit/g17-dc0-firstboot-20260730.txt` (42 lines, exit 0). The SECOND bootstrap + attempt left the node booted long enough to take it. (1) ARTIFACT REACHABILITY asserted on + CONTENT: `curl -fsS http://10.12.8.4/ubuntu/dists/jammy/Release` exit 0, 269218 bytes, and + four body fields asserted -- `Origin: Ubuntu`, `Suite: jammy`, `Components: main restricted + universe multiverse`, `Architectures: ... amd64`. A REAL package path, never the nginx + autoindex root that answers 200 empty. (2) NODE TIME SOURCE is MAAS-served and NOT the DC + edge: `SystemNTPServers=10.12.8.2` = the DC RACK's metal-admin leg, `NTPSynchronized=yes`, + `Reference=AC1F0001` = 172.31.0.1 = the region -- so the chain is node -> rack -> region. + (3) unrecognised/unreachable states REFUSE. **SCOPE, stated not glossed: the node captured is + the dc0 CONTROLLER VM (`vr1-dc0-juju-01`/`7n87bt`) on the ruled metal-admin artifact path, + NOT one of the nine role nodes**, which were still powered off. **G17's dc1 half remains + OPEN.** + **TWO DEFECTS IN G17'S OWN TEXT, both measured:** (a) it names `chronyc sources`, and + **chrony is NOT INSTALLED on the MAAS jammy image** -- the node runs `systemd-timesyncd`, so + the gate as written can only ever REFUSE. The capture reads the stack that EXISTS and asserts + the same property. (b) A first pass at assertion (1) false-FAILED on `printf | grep -q` under + `set -o pipefail` -- grep short-circuits, printf takes SIGPIPE, the pipeline reports failure. + **That is the identical trap this repo recorded earlier the SAME DAY** ("two silenced errors + (`| grep -q` under pipefail)"). The capture uses here-strings instead. Both are DOCFIX + material against the G17 row. + **ROOT CAUSE OF BOTH BOOTSTRAP FAILURES IS NOW MEASURED, AND IT IS NOT ARCHITECTURAL -- + THE CONTROLLER VMs ARE UNDER-CARVED.** The second attempt, run from the rack per D-138, + SSH-connected successfully (`Connected to 10.12.8.5` -- the D-138 path works) and then spent + ~85 minutes at `Attempt 339 to download agent binaries from + 'https://streams.canonical.com/juju/tools/...' -- curl: (7) Couldn't connect to server`. + Measured ON the node: its ONLY route is `10.12.8.0/22 dev enp1s0` -- **NO DEFAULT ROUTE**; + DNS resolves fine via 10.12.8.3; the on-segment mirror answers 200. Compared against role + node `nhg3nf`, which carries links on ALL SIX planes dual-stacked INCLUDING + `br-ex 10.12.4.100` on provider-public **with `gw=10.12.4.1`**. The controller has + metal-admin ONLY. **Same root cause as the `.5`/v6 gap: the controller VMs were added + 2026-07-29, after the D-134 carve AND the IPv6 carve, and the Stage-4 carve's "90 NIC + re-homes (18 nodes)" covered the 18 ROLE nodes only.** The controller's `enp2s0`..`enp6s0` + are still on auto-created `fabric-198..202`. Fix is ruled-but-not-built work under the + D-134 amendment's "`.5` on every plane it is attached to", not a new decision. + **CORRECTION TO THIS SESSION'S OWN EARLIER CLAIM:** the egress re-probe recorded above was + run FROM THE RACK HOSTS and reported the bootstrap window OPEN. **The rack is not + representative of a node** -- it holds a default route via the edge that nodes lack. The + probe was sound for what it measured and did NOT discharge the shelf-life caveat for the + bootstrap, which fetches FROM THE NODE. Re-probe from a node, or from a host proven to share + its routing, before treating a bootstrap window as open. Same wrong-host class as the + `dc-mirror.sh` instrument note recorded above, made after that note was written. + **BLOCKED ON A PERMISSION WALL, NOT ON A DECISION.** The four MAAS calls that complete the + carve (`interface update vlan=` + `interface link-subnet ... mode=STATIC` + for `7n87bt` iface 420 -> vlan 5189 / subnet 7 / `10.12.4.5`, and `p8tdwg` iface 427 -> + vlan 5194 / subnet 10 / `10.12.64.5`) were refused twice by the harness classifier. NOT + retried in altered shapes. Also owed before the next attempt: `juju unregister + vr1-dc0-controller` on the rack plus a MAAS release of `7n87bt` -- `kill-controller` CANNOT + do it because the controller API never came up. **The v6 GUA leg on provider-public + (`2602:f3e2:f02:10::5`) was DELIBERATELY NOT applied**: a default route needs only the v4 + leg, and handing the controller a public GUA is an operator call. Recorded so a later session + does not silently "fix" the asymmetry against role nodes. - Project: Omega Cloud, VR1 DC-DC rehearsal -- a two-DC + Office1-headend virtual rehearsal on KVM (vcloud host), rehearsing the future bare-metal diff --git a/docs/audit/g17-dc0-firstboot-20260730.txt b/docs/audit/g17-dc0-firstboot-20260730.txt new file mode 100644 index 0000000..5b328f1 --- /dev/null +++ b/docs/audit/g17-dc0-firstboot-20260730.txt @@ -0,0 +1,42 @@ +G17 CAPTURE -- vr1-dc0 -- FROM A BOOTED NODE +============================================ +host=moved-troll addr=10.12.8.5/22 utc=2026-07-30T13:05:51Z + +NODE IDENTITY (stated, not glossed): this is the DC0 Juju controller VM +vr1-dc0-juju-01 / MAAS 7n87bt, booted on its real NICs on metal-admin during the +2026-07-30 bootstrap attempt. It sits on the ruled dc0 artifact path +(metal-admin -> 10.12.8.4). It is NOT one of the nine OpenStack role nodes; those +were still powered off. Judge the scope of this capture accordingly. + +--- (1) ARTIFACT REACHABILITY -- asserted on CONTENT, with an exit-code predicate --- + URL is a REAL package path, never the nginx autoindex root (which answers 200 empty). + curl exit=0 bytes=269218 + OK mirror served a REAL jammy Release; asserted on four body fields: + Origin: Ubuntu + Label: Ubuntu + Suite: jammy + Version: 22.04 + Codename: jammy + Date: Thu, 21 Apr 2022 17:16:08 UTC + Architectures: amd64 arm64 armhf i386 ppc64el riscv64 s390x + Components: main restricted universe multiverse + +--- (2) NODE TIME SOURCE -- MAAS-served, NOT the DC edge (D-129(iv)) --- + G17's text names 'chronyc sources'. MEASURED: chrony is NOT INSTALLED on the + MAAS jammy image; the node runs systemd-timesyncd. Reading the stack that EXISTS. + SystemNTPServers = 10.12.8.2 + ServerAddress = 10.12.8.2 + NTPSynchronized = yes + NTPMessage={ Leap=0, Version=4, Mode=4, Stratum=4, Precision=-24, RootDelay=110.733ms, RootDispersion=2.517ms, Reference=AC1F0001, OriginateTimestamp=Thu 2026-07-30 12:42:56 UTC, ReceiveTimestamp=Thu 2026-07-30 12:42:56 UTC, TransmitTimestamp=Thu 2026-07-30 12:42:56 UTC, DestinationTimestamp=Thu 2026-07-30 12:42:56 UTC, Ignored=no PacketCount=8, Jitter=1.651ms } + OK time source is 10.12.8.2 = the DC RACK's metal-admin leg (the MAAS rack + controller), i.e. the MAAS-served hierarchy, and NOT the DC edge. + +--- (3) SUPPORTING CONTEXT (recorded, not gated) --- + resolver: Current DNS Server: 10.12.8.3 + default route: + ^ EMPTY BY MEASUREMENT. The controller VM was carved on metal-admin ONLY, so it + has no default route and no internet path -- which is why the bootstrap could + not fetch the juju agent stream. Role nodes DO carry a provider-public leg with + gw 10.12.4.1. This is the under-carve, not an artifact-path fault. + +G17 (vr1-dc0): BOTH GATED ASSERTIONS PASS