STAGE-5 BOOTSTRAP FAILURE -- ROOT CAUSE CAPTURE (2026-07-30) ============================================================ All commands read-only unless noted. Run from vcloud via `ssh voffice1`, and to the rack via `ssh voffice1 "ssh 172.31.0.2 ..."` (transit IP from scripts/lib-hosts.sh). 1. THE FAILURE -------------- Command (as-executed log 2026-07-30-stage5-dc0-deploy.log), operator-ruled flag form ("Use both flags", GA-R5 2026-07-30): juju bootstrap vr1-maas vr1-dc0-controller \ --credential vr1-dc0-cred \ --bootstrap-base ubuntu@22.04 \ --bootstrap-constraints tags=juju-controller-vr1-dc0 \ --constraints tags=juju-controller-vr1-dc0 Output: Creating Juju controller "vr1-dc0-controller" on vr1-maas/default Looking for packaged Juju agent version 3.6.27 for amd64 Located Juju agent version 3.6.27-ubuntu-amd64 at https://streams.canonical.com/... Launching controller instance(s) on vr1-maas/default... - 7n87bt (arch=amd64 mem=8G cores=4) Installing Juju agent on bootstrap instance Waiting for address Attempting to connect to 10.12.8.1:22 ERROR failed to bootstrap model: cancelled NOTE WHAT WORKED: machine SELECTION was correct. 7n87bt is the tagged controller VM (vr1-dc0-juju-01), NOT one of the nine role nodes -- the D-104 amendment's requirement held. CLAIM DISCIPLINE: this run cannot show WHICH of the two flags did the selecting, only that A constraint selected the tagged machine over nine role-tagged candidates. Do not read it as evidence for either flag individually. THE MACHINE FULLY DEPLOYED AND BOOTED. MEASURED, not inferred -- `maas admin events query hostname=moved-troll limit=50`: 10:46:16 Deploying 10:46:33 Performing PXE boot 10:47:27 Loading ephemeral 10:48:33 Configuring storage 10:48:39 Installing OS 10:48:57 Configuring OS 10:50:17 Rebooting 10:51:03 Image Deployed -- deployed ubuntu/jammy/amd64/generic 10:51:03 Deployed 11:06:14 Releasing <- juju gave up ~15 min after the OS was up 11:06:18 Released CONSEQUENCES OF THIS EVIDENCE: (a) The MAAS deploy path is HEALTHY end to end. Nothing is wrong with power, PXE, storage, or the image. (b) `--bootstrap-base ubuntu@22.04` was HONOURED -- "deployed ubuntu/jammy/amd64/ generic" is the ruled base, so the jammy pin (derived from the bundle's default-base + cloud:jammy-caracal + the dc0 mirror's jammy-only content) is confirmed correct against a live deploy. (c) The ONLY failure is client->machine reachability. See section 2. CORRECTION TO AN EARLIER READING IN THIS SESSION: the post-failure state {"status_name":"Ready","status_message":"Released","power_state":"off", "osystem":"","distro_series":""} was first read as "the machine never booted". That was WRONG -- those fields are CLEARED BY THE RELEASE, so they describe the state after juju handed the machine back, not what happened during the attempt. The event log is the authority for what happened; a released machine's current fields are not evidence about its past. G17: its one-time first-boot window DID open, between 10:51:03 and 11:06:18, and was NOT captured -- the reachability fault was still being diagnosed and, in any case, nothing on the headend side could have reached the node to run the check. G17 remains OPEN and will re-arm at the next successful boot; the checks must be run FROM the node or from the rack, which is on-segment. 2. ROOT CAUSE -- THERE IS NO L3 PATH FROM THE JUJU CLIENT TO ANY DC NODE PLANE ------------------------------------------------------------------------------ `juju bootstrap` requires DIRECT client->machine SSH. The client is on voffice1 (D-128 Plane-2 host). voffice1 has no route to 10.12.8.0/22: voffice1$ ip route get 10.12.8.1 10.12.8.1 via 10.10.0.1 dev enp1s0 src 10.10.0.20 <- falls to the DEFAULT voffice1$ ip route | grep -E '10\.12\.|172\.31\.' 172.31.0.0/30 dev enp2s0 proto kernel scope link src 172.31.0.1 172.31.0.4/30 dev enp3s0 proto kernel scope link src 172.31.0.5 (ONLY the two transit /30s. No DC plane routes at all.) voffice1$ ping -c2 10.12.8.1 -> 100% loss voffice1$ ping -c2 10.12.8.100 -> 100% loss (a role node) voffice1$ ping -c2 10.12.8.4 -> 100% loss (the mirror) TCP 22 to 10.12.8.1 -> UNREACHABLE MAAS commissioning was never affected because the RACK controller proxies DHCP/PXE on-segment; the region needs no L3 to the nodes. Juju does. 3. WHY A ROUTE ALONE WOULD NOT FIX IT -- TWO INDEPENDENT, DELIBERATE BLOCKS --------------------------------------------------------------------------- (a) SEC-010 transit FORWARD-drop, on the rack (gate G10 applied + verified it): rack$ sudo nft list table inet sec010 table inet sec010 { chain forward { type filter hook forward priority filter - 10; policy accept; oifname "enp1s0" drop iifname "enp1s0" drop } } enp1s0 on the rack is the Office1 transit leg (172.31.0.2/30). ALL forwarding in or out of the transit is dropped. `drop` at priority filter-10 is terminal and runs BEFORE libvirt's chains. (b) The plane networks are ISOLATED libvirt nets. `virsh net-dumpxml vr1-dc0-metal-admin` has NO and NO element (only name/bridge/mtu/ domain), which is what makes D-125's egress-isolation result true. libvirt therefore installs a blanket reject on forwarding INTO the bridge: rack$ sudo nft list ruleset | grep -A6 'chain LIBVIRT_FWI' chain LIBVIRT_FWI { oifname "virbr4" ... reject oifname "virbr6" ... reject oifname "virbr3" ... reject oifname "virbr5" ... reject oifname "virbr2" ... reject <- virbr2 = vr1-dc0-metal-admin } (counters 0 -- nothing has ever tried, consistent with this never having worked.) (c) The DC edge is NOT a management router. It is attached to two networks only: rack$ sudo virsh dumpxml vr1-dc0-opnsense | grep 'source network' No metal-admin leg, so it cannot route the headend onto the node plane either. CONCLUSION: nothing regressed. This deployment has never had a juju-client->controller path. The rack host is the ONLY machine with legs on both sides (enp1s0 172.31.0.2/30 transit; virbr2 10.12.8.2/3/4 metal-admin; virbr5 10.12.4.2 provider-public), and it is forbidden by SEC-010 from forwarding between them. 3b. SEC-010 IS NOT AN ACCIDENT -- IT FORBIDS EXACTLY THIS ROUTE, IN WRITING --------------------------------------------------------------------------- Read the SEC-010 row in docs/security-ledger.md before framing any fix as a mere firewall tweak. Its stated PURPOSE is the metal-admin DC-LOCAL invariant (D-052/D-100), and it names the forbidden route explicitly. Quoting the row: "metal-admin DC-LOCAL invariant (D-052/D-100) is PRESERVED but NOT ENFORCED in the committed Stage-3 config." "Cross-plane routing (metal-admin <-> the whole Office1 /22, bidirectional) is blocked ONLY by Ubuntu's distro default; the deferred MAAS-rack install could silently flip it. A MAAS rack proxies at the application layer and needs no kernel forwarding, so pinning is free." "Nothing routes across the fiber THROUGH vvr1-dc0; the rack proxies MAAS at the app layer (originated/terminated, not forwarded)." "Region route must target only the rack transit /30, never 10.12.8.0/22." Status: CLOSED 2026-07-20 on the operator's exact utterance "Close SEC-010 (Recommended)", applied and verified BOTH ends in the logged dc0-deploy step-B window (named gate `site-headend-install.sh --check` EXIT 0, docs/audit/stepB-check-20260720-final.txt). THE LOAD-BEARING SENTENCE IS THE JUSTIFICATION, AND IT ONLY EVER CONSIDERED MAAS: "a MAAS rack proxies at the application layer and needs no kernel forwarding, so pinning is free." That is TRUE of MAAS, and of the inner tofu root (qemu+ssh to the rack), and of NetBox. It is FALSE of Juju: `juju bootstrap` and every subsequent agent connection dial the MACHINE directly, at L3. Juju was not in scope when the cost of the pin was assessed, so the pin was recorded as free when for this one tool it is not. THE RESULTING CONTRADICTION, between surfaces that are BOTH ruled: - D-100 says the Office1<->DC fiber "carries management traffic only (MAAS/Juju/ operator)" -- Juju named explicitly as management traffic the fiber carries. - D-128 puts the Plane-2 execution host, and therefore the Juju CLIENT, on voffice1. - SEC-010 + D-052 make metal-admin DC-LOCAL and forbid the region routing to 10.12.8.0/22 at all. Under SEC-010's model the fiber carries management traffic that TERMINATES ON THE RACK. Juju is the one Plane-2 tool that must instead reach THROUGH the rack to the node plane. So the question is not "which firewall rule" but WHICH RULED SURFACE GOVERNS, with the other amended. That is a D-number question (GA-R3 A1: a Roosevelt build session would grep it before touching a built surface -- it decides where the Juju client lives in every future DC). NOTE ON A NAIVE FIX THAT MUST NOT BE REACHED FOR: making the plane nets `` with an `` element would start a libvirt dnsmasq on virbr2 and COLLIDE with MAAS's DHCP on that segment. If routing is ruled, the mode to evaluate is `` (libvirt installs no filtering rules; the admin owns them), which must be tested for whether it removes the LIBVIRT_FWI/FWO rejects WITHOUT requiring ``. It needs net-destroy/net-start, which drops virbr2 and its manually added .2/.3/.4 legs -- so `dc-rack-net.sh install ` must be re-run and re-checked after, and `mtu 9000` preserved. SCOPE THE FIX ONCE, BY ENUMERATION, IF ROUTING IS RULED: SSH/22 alone gets bootstrap further and then fails at the Juju API on 17070. Stage 5+ also runs `openstack` against VIPs from the headend (phase-03..06). Enumerate every plane and port the headend must dial BEFORE writing a rule, or this gets fixed three times. dc1 CARRIES THE IDENTICAL GAP: same `table inet sec010` shape on its own transit leg, same isolated plane nets, same absent voffice1 routes. Whatever is ruled applies to both DCs, and site-baseleg.sh's deferred DC rows are the templated home for the voffice1-side half. 4. WHAT THE GAP REGISTER ALREADY SAID -------------------------------------- scripts/site-baseleg.sh:42,47 -- # DC rows are DEFERRED: the DCs nest inside vvr1-dc0 and are reached by qemu+ssh # [vr1-dc0]="|||" # MEASURE first (qemu+ssh may make this moot) The deferral's stated assumption -- "reached by qemu+ssh" -- is TRUE for virsh/tofu provisioning and FALSE for `juju bootstrap`, which needs real L3. Workflow-doc gap register item 20 ("durable host->site reachability had no owning tool -- CLOSED for Office1; DC path templated") is the item that has now reopened at the DC, which is exactly the outcome the site-keyed tooling was built to prevent. Repo-wide grep for prior art on this question returned ZERO hits: grep -rn -iE 'juju client.*reach|reach.*controller|route to 10\.12\.8|10\.12\.8\.0/22 via|forward.*transit' \ docs/CURRENT-STATE.md docs/design-decisions.md runbooks/dc-dc-phase4-juju-bundle-per-dc.md D-100 states the Office1<->DC fiber "carries management traffic only (MAAS/Juju/ operator)", so the INTENT exists; the implementation never did. 5. WHAT IS VIABLE, MEASURED --------------------------- The rack CAN originate traffic to the MAAS region (SEC-010 drops FORWARD, not the rack's own INPUT/OUTPUT): juju cloud endpoint: {'vr1-maas': {'endpoint': 'http://10.10.0.20:5240/MAAS'}} rack$ ip route get 10.10.0.20 -> via 172.31.0.1 dev enp1s0 src 172.31.0.2 rack$ TCP 10.10.0.20:5240 -> REACHABLE rack$ command -v juju -> ABSENT (no client installed) DC edge egress remains OPEN (re-probed this session from both racks, --noproxy '*'): juju agent stream 200, api.snapcraft.io answering, archive 200, 1.1.1.1 0% loss. 6. SEPARATE FINDING, ALSO UNRESOLVED (ruled, then overtaken by this failure) --------------------------------------------------------------------------- The controller VM holds an AUTO, v4-only address, unlike every role node: 7n87bt enp1s0 mode=auto ip=10.12.8.1 (no v6) nhg3nf enp1s0 mode=static ip=10.12.8.100 fd50:840e:74e2:220::100 10.12.8.5 (the D-134-amendment ruled controller octet) is FREE. Operator ruled 2026-07-30: "Fix now: static .5 + v6, re-bootstrap (Recommended)". The v6 prefix is CONFIRMED BY PAIRING, not inferred from one node: dc0 metal-admin v4 subnet id 6 is vlan_id 5005 / fabric-4, and the v6 subnet on vlan 5005 / fabric-4 is fd50:840e:74e2:220::/64. With the 2026-07-27 ruling that the v6 host part mirrors the v4 octet, the controller's pair is 10.12.8.5 + fd50:840e:74e2:220::5. This work is UNBLOCKED-BUT-NOT-DONE: it must be applied whichever reachability option is ruled, and re-bootstrap now depends on that ruling.