diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 32424c2..923a236 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -158,6 +158,64 @@ amended by this** -- the decision is unchanged; only the flag that implements it is clarified. A DOCFIX to correct the Step 2 command and annotate D-104's mechanism sentence is OWED and is recorded in this session's changelog. + **BOOTSTRAP ATTEMPTED 2026-07-30 AND FAILED -- STAGE 5 IS BLOCKED ON AN UNRULED + REACHABILITY QUESTION.** Full capture: `docs/audit/stage5-bootstrap-reachability-20260730.txt`. + **What WORKED, measured:** machine selection was correct -- juju took `7n87bt`, the + tagged controller VM, NOT one of the nine role nodes, so the D-104 amendment's + requirement held (this run cannot show WHICH of the two ruled flags did the selecting, + only that A constraint preferred the tagged machine over nine role-tagged candidates). + The MAAS deploy path is HEALTHY END TO END: `maas admin events query + hostname=moved-troll` shows Deploying -> PXE -> ephemeral -> storage -> Installing OS -> + Configuring OS -> Rebooting -> **`Image Deployed -- deployed ubuntu/jammy/amd64/generic` + -> Deployed** (10:46:16 to 10:51:03). That also CONFIRMS the `--bootstrap-base + ubuntu@22.04` pin against a live deploy. **What FAILED:** juju then spent 15 minutes + unable to SSH the machine and gave up -- `Attempting to connect to 10.12.8.1:22` -> + `ERROR failed to bootstrap model: cancelled` -> Releasing 11:06:14. + **ROOT CAUSE -- there is NO L3 path from the juju client to any DC node plane, and there + never was.** `juju bootstrap` needs direct client->machine SSH. voffice1 holds ONLY the + two transit /30s (`172.31.0.0/30`, `172.31.0.4/30`); `ip route get 10.12.8.1` falls + through to the DEFAULT via `10.10.0.1`, and voffice1 cannot ping the controller, a role + node, OR the mirror. MAAS commissioning was never affected because the RACK proxies + DHCP/PXE on-segment -- the region needs no L3 to the nodes. Juju does. A route alone + would NOT fix it: (a) `table inet sec010` on the rack drops ALL forwarding in and out of + the transit leg (`oifname "enp1s0" drop` / `iifname "enp1s0" drop`, priority filter-10, + terminal and ahead of libvirt's chains); (b) the plane nets are ISOLATED libvirt nets + (no ``, no `` -- which is what makes D-125's egress isolation TRUE), so + libvirt blanket-rejects forwarding into `virbr2`; (c) the DC edge is attached only to + `provider-public` + `wan`, so it is not a management router either. The rack is the ONLY + host with legs on both sides and SEC-010 forbids it forwarding between them. + **THIS IS A CONTRADICTION BETWEEN RULED SURFACES, NOT A BROKEN CONFIG.** SEC-010's row + states its purpose is the metal-admin DC-LOCAL invariant (D-052/D-100) and names the + forbidden route in writing: "Nothing routes across the fiber THROUGH vvr1-dc0" and + "Region route must target only the rack transit /30, never 10.12.8.0/22" (CLOSED + 2026-07-20, operator utterance "Close SEC-010 (Recommended)", gate-verified EXIT 0). + Its justification -- "a MAAS rack proxies at the application layer and needs no kernel + forwarding, so pinning is free" -- is TRUE for MAAS, the inner tofu root and NetBox, and + FALSE for Juju, which dials the machine at L3. **Juju was not in scope when the cost of + that pin was assessed.** Against this, D-100 says the Office1<->DC fiber "carries + management traffic only (MAAS/Juju/operator)" -- naming Juju -- and D-128 puts the Juju + CLIENT on voffice1. So the question is WHICH RULED SURFACE GOVERNS, with the other + amended; it is D-number material under the GA-R3 A1 test because it decides where the + Juju client lives at every future DC standup. **QUESTION IS OPEN AND BLOCKING; no + reachability change has been made.** + **UNBLOCKED AND NOW BUILT, both DCs (the D-134 amendment stops being ruled-but-not-built):** + operator ruled 2026-07-30, exact utterance **"Fix now: static .5 + v6, re-bootstrap + (Recommended)"**. Question as presented: the controller had bootstrapped onto an AUTO, + v4-only `10.12.8.1` while all nine role nodes hold STATIC dual-stacked addresses in + their D-134 bands, because the controller VM was added after both the D-134 carve and + the IPv6 carve; the address is baked into every unit's `agent.conf` at deploy time, so + it is cheap to fix while the controller is empty and expensive later. APPLIED and read + back: `moved-troll: static=10.12.8.5 static=fd50:840e:74e2:220::5` and + `square-ferret: static=10.12.68.5 static=fd50:840e:74e2:320::5`. The v6 prefixes were + confirmed BY VLAN PAIRING, not inferred from one node (dc0 v4 subnet 6 and v6 subnet 21 + are both vlan 5005/fabric-4; dc1 v4 subnet 11 and v6 subnet 27 are both vlan + 5143/fabric-142), with the 2026-07-27 ruling that the v6 host part mirrors the v4 octet. + Both DCs now match, so the amendment's cross-DC octet standard holds by construction. + **G17 NOT CAPTURED, window opened and closed.** The node genuinely booted between + 10:51:03 and 11:06:18, which was G17's first-boot window; nothing on the headend could + reach it to run the check, and the fault was still being diagnosed. G17 stays OPEN and + re-arms at the next successful boot; its checks must run FROM the node or from the rack, + which is on-segment. - Project: Omega Cloud, VR1 DC-DC rehearsal -- a two-DC + Office1-headend virtual rehearsal on KVM (vcloud host), rehearsing the future bare-metal diff --git a/docs/audit/stage5-bootstrap-reachability-20260730.txt b/docs/audit/stage5-bootstrap-reachability-20260730.txt new file mode 100644 index 0000000..77f0ca7 --- /dev/null +++ b/docs/audit/stage5-bootstrap-reachability-20260730.txt @@ -0,0 +1,237 @@ +STAGE-5 BOOTSTRAP FAILURE -- ROOT CAUSE CAPTURE (2026-07-30) +============================================================ +All commands read-only unless noted. Run from vcloud via `ssh voffice1`, and to the +rack via `ssh voffice1 "ssh 172.31.0.2 ..."` (transit IP from scripts/lib-hosts.sh). + +1. THE FAILURE +-------------- +Command (as-executed log 2026-07-30-stage5-dc0-deploy.log), operator-ruled flag form +("Use both flags", GA-R5 2026-07-30): + + juju bootstrap vr1-maas vr1-dc0-controller \ + --credential vr1-dc0-cred \ + --bootstrap-base ubuntu@22.04 \ + --bootstrap-constraints tags=juju-controller-vr1-dc0 \ + --constraints tags=juju-controller-vr1-dc0 + +Output: + Creating Juju controller "vr1-dc0-controller" on vr1-maas/default + Looking for packaged Juju agent version 3.6.27 for amd64 + Located Juju agent version 3.6.27-ubuntu-amd64 at https://streams.canonical.com/... + Launching controller instance(s) on vr1-maas/default... + - 7n87bt (arch=amd64 mem=8G cores=4) + Installing Juju agent on bootstrap instance + Waiting for address + Attempting to connect to 10.12.8.1:22 + ERROR failed to bootstrap model: cancelled + +NOTE WHAT WORKED: machine SELECTION was correct. 7n87bt is the tagged controller VM +(vr1-dc0-juju-01), NOT one of the nine role nodes -- the D-104 amendment's requirement +held. CLAIM DISCIPLINE: this run cannot show WHICH of the two flags did the selecting, +only that A constraint selected the tagged machine over nine role-tagged candidates. +Do not read it as evidence for either flag individually. + +THE MACHINE FULLY DEPLOYED AND BOOTED. MEASURED, not inferred -- `maas admin events +query hostname=moved-troll limit=50`: + + 10:46:16 Deploying + 10:46:33 Performing PXE boot + 10:47:27 Loading ephemeral + 10:48:33 Configuring storage + 10:48:39 Installing OS + 10:48:57 Configuring OS + 10:50:17 Rebooting + 10:51:03 Image Deployed -- deployed ubuntu/jammy/amd64/generic + 10:51:03 Deployed + 11:06:14 Releasing <- juju gave up ~15 min after the OS was up + 11:06:18 Released + +CONSEQUENCES OF THIS EVIDENCE: + (a) The MAAS deploy path is HEALTHY end to end. Nothing is wrong with power, + PXE, storage, or the image. + (b) `--bootstrap-base ubuntu@22.04` was HONOURED -- "deployed ubuntu/jammy/amd64/ + generic" is the ruled base, so the jammy pin (derived from the bundle's + default-base + cloud:jammy-caracal + the dc0 mirror's jammy-only content) is + confirmed correct against a live deploy. + (c) The ONLY failure is client->machine reachability. See section 2. + +CORRECTION TO AN EARLIER READING IN THIS SESSION: the post-failure state + {"status_name":"Ready","status_message":"Released","power_state":"off", + "osystem":"","distro_series":""} +was first read as "the machine never booted". That was WRONG -- those fields are +CLEARED BY THE RELEASE, so they describe the state after juju handed the machine +back, not what happened during the attempt. The event log is the authority for what +happened; a released machine's current fields are not evidence about its past. + +G17: its one-time first-boot window DID open, between 10:51:03 and 11:06:18, and was +NOT captured -- the reachability fault was still being diagnosed and, in any case, +nothing on the headend side could have reached the node to run the check. G17 remains +OPEN and will re-arm at the next successful boot; the checks must be run FROM the node +or from the rack, which is on-segment. + +2. ROOT CAUSE -- THERE IS NO L3 PATH FROM THE JUJU CLIENT TO ANY DC NODE PLANE +------------------------------------------------------------------------------ +`juju bootstrap` requires DIRECT client->machine SSH. The client is on voffice1 +(D-128 Plane-2 host). voffice1 has no route to 10.12.8.0/22: + + voffice1$ ip route get 10.12.8.1 + 10.12.8.1 via 10.10.0.1 dev enp1s0 src 10.10.0.20 <- falls to the DEFAULT + voffice1$ ip route | grep -E '10\.12\.|172\.31\.' + 172.31.0.0/30 dev enp2s0 proto kernel scope link src 172.31.0.1 + 172.31.0.4/30 dev enp3s0 proto kernel scope link src 172.31.0.5 + (ONLY the two transit /30s. No DC plane routes at all.) + + voffice1$ ping -c2 10.12.8.1 -> 100% loss + voffice1$ ping -c2 10.12.8.100 -> 100% loss (a role node) + voffice1$ ping -c2 10.12.8.4 -> 100% loss (the mirror) + TCP 22 to 10.12.8.1 -> UNREACHABLE + +MAAS commissioning was never affected because the RACK controller proxies DHCP/PXE +on-segment; the region needs no L3 to the nodes. Juju does. + +3. WHY A ROUTE ALONE WOULD NOT FIX IT -- TWO INDEPENDENT, DELIBERATE BLOCKS +--------------------------------------------------------------------------- +(a) SEC-010 transit FORWARD-drop, on the rack (gate G10 applied + verified it): + + rack$ sudo nft list table inet sec010 + table inet sec010 { + chain forward { + type filter hook forward priority filter - 10; policy accept; + oifname "enp1s0" drop + iifname "enp1s0" drop + } + } + + enp1s0 on the rack is the Office1 transit leg (172.31.0.2/30). ALL forwarding in or + out of the transit is dropped. `drop` at priority filter-10 is terminal and runs + BEFORE libvirt's chains. + +(b) The plane networks are ISOLATED libvirt nets. `virsh net-dumpxml + vr1-dc0-metal-admin` has NO and NO element (only name/bridge/mtu/ + domain), which is what makes D-125's egress-isolation result true. libvirt + therefore installs a blanket reject on forwarding INTO the bridge: + + rack$ sudo nft list ruleset | grep -A6 'chain LIBVIRT_FWI' + chain LIBVIRT_FWI { + oifname "virbr4" ... reject + oifname "virbr6" ... reject + oifname "virbr3" ... reject + oifname "virbr5" ... reject + oifname "virbr2" ... reject <- virbr2 = vr1-dc0-metal-admin + } + (counters 0 -- nothing has ever tried, consistent with this never having worked.) + +(c) The DC edge is NOT a management router. It is attached to two networks only: + rack$ sudo virsh dumpxml vr1-dc0-opnsense | grep 'source network' + + + No metal-admin leg, so it cannot route the headend onto the node plane either. + +CONCLUSION: nothing regressed. This deployment has never had a juju-client->controller +path. The rack host is the ONLY machine with legs on both sides +(enp1s0 172.31.0.2/30 transit; virbr2 10.12.8.2/3/4 metal-admin; virbr5 10.12.4.2 +provider-public), and it is forbidden by SEC-010 from forwarding between them. + +3b. SEC-010 IS NOT AN ACCIDENT -- IT FORBIDS EXACTLY THIS ROUTE, IN WRITING +--------------------------------------------------------------------------- +Read the SEC-010 row in docs/security-ledger.md before framing any fix as a mere +firewall tweak. Its stated PURPOSE is the metal-admin DC-LOCAL invariant (D-052/D-100), +and it names the forbidden route explicitly. Quoting the row: + + "metal-admin DC-LOCAL invariant (D-052/D-100) is PRESERVED but NOT ENFORCED in the + committed Stage-3 config." + "Cross-plane routing (metal-admin <-> the whole Office1 /22, bidirectional) is blocked + ONLY by Ubuntu's distro default; the deferred MAAS-rack install could silently flip it. + A MAAS rack proxies at the application layer and needs no kernel forwarding, so + pinning is free." + "Nothing routes across the fiber THROUGH vvr1-dc0; the rack proxies MAAS at the app + layer (originated/terminated, not forwarded)." + "Region route must target only the rack transit /30, never 10.12.8.0/22." + +Status: CLOSED 2026-07-20 on the operator's exact utterance "Close SEC-010 +(Recommended)", applied and verified BOTH ends in the logged dc0-deploy step-B window +(named gate `site-headend-install.sh --check` EXIT 0, +docs/audit/stepB-check-20260720-final.txt). + +THE LOAD-BEARING SENTENCE IS THE JUSTIFICATION, AND IT ONLY EVER CONSIDERED MAAS: +"a MAAS rack proxies at the application layer and needs no kernel forwarding, so pinning +is free." That is TRUE of MAAS, and of the inner tofu root (qemu+ssh to the rack), and of +NetBox. It is FALSE of Juju: `juju bootstrap` and every subsequent agent connection dial +the MACHINE directly, at L3. Juju was not in scope when the cost of the pin was assessed, +so the pin was recorded as free when for this one tool it is not. + +THE RESULTING CONTRADICTION, between surfaces that are BOTH ruled: + - D-100 says the Office1<->DC fiber "carries management traffic only (MAAS/Juju/ + operator)" -- Juju named explicitly as management traffic the fiber carries. + - D-128 puts the Plane-2 execution host, and therefore the Juju CLIENT, on voffice1. + - SEC-010 + D-052 make metal-admin DC-LOCAL and forbid the region routing to + 10.12.8.0/22 at all. +Under SEC-010's model the fiber carries management traffic that TERMINATES ON THE RACK. +Juju is the one Plane-2 tool that must instead reach THROUGH the rack to the node plane. +So the question is not "which firewall rule" but WHICH RULED SURFACE GOVERNS, with the +other amended. That is a D-number question (GA-R3 A1: a Roosevelt build session would +grep it before touching a built surface -- it decides where the Juju client lives in +every future DC). + +NOTE ON A NAIVE FIX THAT MUST NOT BE REACHED FOR: making the plane nets +`` with an `` element would start a libvirt dnsmasq on +virbr2 and COLLIDE with MAAS's DHCP on that segment. If routing is ruled, the mode to +evaluate is `` (libvirt installs no filtering rules; the admin owns +them), which must be tested for whether it removes the LIBVIRT_FWI/FWO rejects WITHOUT +requiring ``. It needs net-destroy/net-start, which drops virbr2 and its manually +added .2/.3/.4 legs -- so `dc-rack-net.sh install ` must be re-run and re-checked +after, and `mtu 9000` preserved. + +SCOPE THE FIX ONCE, BY ENUMERATION, IF ROUTING IS RULED: SSH/22 alone gets bootstrap +further and then fails at the Juju API on 17070. Stage 5+ also runs `openstack` against +VIPs from the headend (phase-03..06). Enumerate every plane and port the headend must +dial BEFORE writing a rule, or this gets fixed three times. + +dc1 CARRIES THE IDENTICAL GAP: same `table inet sec010` shape on its own transit leg, +same isolated plane nets, same absent voffice1 routes. Whatever is ruled applies to both +DCs, and site-baseleg.sh's deferred DC rows are the templated home for the voffice1-side +half. + +4. WHAT THE GAP REGISTER ALREADY SAID +-------------------------------------- +scripts/site-baseleg.sh:42,47 -- + # DC rows are DEFERRED: the DCs nest inside vvr1-dc0 and are reached by qemu+ssh + # [vr1-dc0]="|||" # MEASURE first (qemu+ssh may make this moot) +The deferral's stated assumption -- "reached by qemu+ssh" -- is TRUE for virsh/tofu +provisioning and FALSE for `juju bootstrap`, which needs real L3. Workflow-doc gap +register item 20 ("durable host->site reachability had no owning tool -- CLOSED for +Office1; DC path templated") is the item that has now reopened at the DC, which is +exactly the outcome the site-keyed tooling was built to prevent. + +Repo-wide grep for prior art on this question returned ZERO hits: + grep -rn -iE 'juju client.*reach|reach.*controller|route to 10\.12\.8|10\.12\.8\.0/22 via|forward.*transit' \ + docs/CURRENT-STATE.md docs/design-decisions.md runbooks/dc-dc-phase4-juju-bundle-per-dc.md + +D-100 states the Office1<->DC fiber "carries management traffic only (MAAS/Juju/ +operator)", so the INTENT exists; the implementation never did. + +5. WHAT IS VIABLE, MEASURED +--------------------------- +The rack CAN originate traffic to the MAAS region (SEC-010 drops FORWARD, not the +rack's own INPUT/OUTPUT): + juju cloud endpoint: {'vr1-maas': {'endpoint': 'http://10.10.0.20:5240/MAAS'}} + rack$ ip route get 10.10.0.20 -> via 172.31.0.1 dev enp1s0 src 172.31.0.2 + rack$ TCP 10.10.0.20:5240 -> REACHABLE + rack$ command -v juju -> ABSENT (no client installed) + +DC edge egress remains OPEN (re-probed this session from both racks, --noproxy '*'): +juju agent stream 200, api.snapcraft.io answering, archive 200, 1.1.1.1 0% loss. + +6. SEPARATE FINDING, ALSO UNRESOLVED (ruled, then overtaken by this failure) +--------------------------------------------------------------------------- +The controller VM holds an AUTO, v4-only address, unlike every role node: + 7n87bt enp1s0 mode=auto ip=10.12.8.1 (no v6) + nhg3nf enp1s0 mode=static ip=10.12.8.100 fd50:840e:74e2:220::100 + 10.12.8.5 (the D-134-amendment ruled controller octet) is FREE. +Operator ruled 2026-07-30: "Fix now: static .5 + v6, re-bootstrap (Recommended)". +The v6 prefix is CONFIRMED BY PAIRING, not inferred from one node: dc0 metal-admin v4 +subnet id 6 is vlan_id 5005 / fabric-4, and the v6 subnet on vlan 5005 / fabric-4 is +fd50:840e:74e2:220::/64. With the 2026-07-27 ruling that the v6 host part mirrors the +v4 octet, the controller's pair is 10.12.8.5 + fd50:840e:74e2:220::5. +This work is UNBLOCKED-BUT-NOT-DONE: it must be applied whichever reachability option +is ruled, and re-bootstrap now depends on that ruling. diff --git a/docs/changelog-20260730-stage5-open.md b/docs/changelog-20260730-stage5-open.md index a3e302c..48fd1e8 100644 --- a/docs/changelog-20260730-stage5-open.md +++ b/docs/changelog-20260730-stage5-open.md @@ -105,3 +105,74 @@ `juju-controller-vr1-dc0` and no `openstack-vr1-dc0`. **Revert.** N/A -- nothing delivered yet; this item is the record that it is owed. + +### 4. Bootstrap ATTEMPTED and FAILED; root cause captured; Stage 5 blocked on a ruling + +**What.** New capture `docs/audit/stage5-bootstrap-reachability-20260730.txt`. Status +recorded in `docs/CURRENT-STATE.md` section 1. No reachability change was made. + +**Why (evidence).** `juju bootstrap` selected the correct machine and MAAS deployed +jammy end to end (`Image Deployed -- deployed ubuntu/jammy/amd64/generic`), then juju +could not SSH it and released the machine. voffice1 has no route to any DC node plane, +and two independent, DELIBERATE controls forbid creating one: SEC-010's transit +FORWARD-drop on the rack, and libvirt's blanket reject into the isolated plane bridges. +The DC edge has no metal-admin leg. This is a contradiction between ruled surfaces -- +SEC-010/D-052 make metal-admin DC-local and forbid the region routing to `10.12.8.0/22`, +while D-100 says the fiber carries Juju traffic and D-128 puts the Juju client on +voffice1 -- so it needs a ruling, not a firewall edit. + +**The generalisable lesson, and it is the reason this was invisible for weeks.** +SEC-010's cost was assessed with the sentence "a MAAS rack proxies at the application +layer and needs no kernel forwarding, so pinning is free." That is true of every +Plane-2 tool THEN in use -- MAAS, the inner tofu root over `qemu+ssh`, NetBox -- and +false of the one tool that had not been run yet. **A security control priced against +the tools you have is not priced against the tools the next stage introduces.** +`scripts/site-baseleg.sh:42,47` recorded the same blind spot from the other side, +deferring the DC rows because "the DCs nest inside vvr1-dc0 and are reached by +qemu+ssh" with a `# MEASURE first` note that was never executed. + +**Two claim-discipline corrections made to the capture before it was committed**, both +worth keeping as examples: (i) the post-failure machine fields +(`osystem:""`, `status:Ready/Released`) were first read as "the machine never booted" -- +those fields are CLEARED BY THE RELEASE and describe the aftermath, not the attempt; +the event log is the authority and it shows a clean full deploy. (ii) The capture +initially said the both-flags form "targeted it deterministically"; one run cannot +attribute the selection to either flag, only show that A constraint preferred the +tagged machine over nine role-tagged candidates. + +**Revert.** The capture and the CURRENT-STATE entry are records of a live event; +reverting them would delete evidence, not undo a change. Nothing was mutated on the +cloud by this item. + +### 5. D-134 amendment BUILT on both controllers (was ruled-but-not-built) + +**What.** `vr1-dc0-juju-01` (`7n87bt`, iface 419) and `vr1-dc1-juju-01` (`p8tdwg`, +iface 426) converted from a single AUTO v4 link to STATIC dual-stack: +`10.12.8.5` + `fd50:840e:74e2:220::5`, and `10.12.68.5` + `fd50:840e:74e2:320::5`. +Read back and asserted on content. + +**Why (evidence).** Operator ruling 2026-07-30, exact utterance **"Fix now: static .5 + +v6, re-bootstrap (Recommended)"**. Both controllers held AUTO, v4-only addresses while +all nine role nodes per DC hold STATIC dual-stacked addresses in their D-134 bands -- +the controller VMs were added on 2026-07-29, after both the D-134 carve and the IPv6 +carve, so neither reached them. The D-134 amendment of 2026-07-29 ruled `.5` "on every +plane it is attached to" and made the octet map a STANDING cross-DC standard; it was +ruled-but-not-built until this item, the same class the 2026-07-27 audit found twice. + +**AUTO was the live hazard, not the octet.** An auto address is not pinned: a future +redeploy of the controller machine could return a different address, and the controller +address is written into every unit's `agent.conf`, so every agent in the cloud would +lose its controller at once. + +**Values were derived, not typed.** The v6 prefixes were confirmed by VLAN pairing -- +dc0 v4 subnet 6 and v6 subnet 21 are both `vlan 5005/fabric-4`; dc1 v4 subnet 11 and v6 +subnet 27 are both `vlan 5143/fabric-142` -- combined with the 2026-07-27 ruling that +the v6 host part mirrors the v4 octet. Inferring the prefix from a single node's address +would have been the hard-rule-2 violation. + +**Revert.** Per controller: `maas admin interface unlink-subnet id=` +for each of the two new links, then +`maas admin interface link-subnet mode=AUTO subnet=` to restore +the original single auto link. dc0 = `7n87bt` iface `419` subnet `6`; dc1 = `p8tdwg` +iface `426` subnet `11`. Both machines are `Ready` and powered off, so this is safe at +any time before they are allocated.