STAGE-5 BOOTSTRAP FAILURE -- ROOT CAUSE CAPTURE (2026-07-30)
============================================================
All commands read-only unless noted. Run from vcloud via `ssh voffice1`, and to the
rack via `ssh voffice1 "ssh 172.31.0.2 ..."` (transit IP from scripts/lib-hosts.sh).
1. THE FAILURE
--------------
Command (as-executed log 2026-07-30-stage5-dc0-deploy.log), operator-ruled flag form
("Use both flags", GA-R5 2026-07-30):
juju bootstrap vr1-maas vr1-dc0-controller \
--credential vr1-dc0-cred \
--bootstrap-base ubuntu@22.04 \
--bootstrap-constraints tags=juju-controller-vr1-dc0 \
--constraints tags=juju-controller-vr1-dc0
Output:
Creating Juju controller "vr1-dc0-controller" on vr1-maas/default
Looking for packaged Juju agent version 3.6.27 for amd64
Located Juju agent version 3.6.27-ubuntu-amd64 at https://streams.canonical.com/...
Launching controller instance(s) on vr1-maas/default...
- 7n87bt (arch=amd64 mem=8G cores=4)
Installing Juju agent on bootstrap instance
Waiting for address
Attempting to connect to 10.12.8.1:22
ERROR failed to bootstrap model: cancelled
NOTE WHAT WORKED: machine SELECTION was correct. 7n87bt is the tagged controller VM
(vr1-dc0-juju-01), NOT one of the nine role nodes -- the D-104 amendment's requirement
held. CLAIM DISCIPLINE: this run cannot show WHICH of the two flags did the selecting,
only that A constraint selected the tagged machine over nine role-tagged candidates.
Do not read it as evidence for either flag individually.
THE MACHINE FULLY DEPLOYED AND BOOTED. MEASURED, not inferred -- `maas admin events
query hostname=moved-troll limit=50`:
10:46:16 Deploying
10:46:33 Performing PXE boot
10:47:27 Loading ephemeral
10:48:33 Configuring storage
10:48:39 Installing OS
10:48:57 Configuring OS
10:50:17 Rebooting
10:51:03 Image Deployed -- deployed ubuntu/jammy/amd64/generic
10:51:03 Deployed
11:06:14 Releasing <- juju gave up ~15 min after the OS was up
11:06:18 Released
CONSEQUENCES OF THIS EVIDENCE:
(a) The MAAS deploy path is HEALTHY end to end. Nothing is wrong with power,
PXE, storage, or the image.
(b) `--bootstrap-base ubuntu@22.04` was HONOURED -- "deployed ubuntu/jammy/amd64/
generic" is the ruled base, so the jammy pin (derived from the bundle's
default-base + cloud:jammy-caracal + the dc0 mirror's jammy-only content) is
confirmed correct against a live deploy.
(c) The ONLY failure is client->machine reachability. See section 2.
CORRECTION TO AN EARLIER READING IN THIS SESSION: the post-failure state
{"status_name":"Ready","status_message":"Released","power_state":"off",
"osystem":"","distro_series":""}
was first read as "the machine never booted". That was WRONG -- those fields are
CLEARED BY THE RELEASE, so they describe the state after juju handed the machine
back, not what happened during the attempt. The event log is the authority for what
happened; a released machine's current fields are not evidence about its past.
G17: its one-time first-boot window DID open, between 10:51:03 and 11:06:18, and was
NOT captured -- the reachability fault was still being diagnosed and, in any case,
nothing on the headend side could have reached the node to run the check. G17 remains
OPEN and will re-arm at the next successful boot; the checks must be run FROM the node
or from the rack, which is on-segment.
2. ROOT CAUSE -- THERE IS NO L3 PATH FROM THE JUJU CLIENT TO ANY DC NODE PLANE
------------------------------------------------------------------------------
`juju bootstrap` requires DIRECT client->machine SSH. The client is on voffice1
(D-128 Plane-2 host). voffice1 has no route to 10.12.8.0/22:
voffice1$ ip route get 10.12.8.1
10.12.8.1 via 10.10.0.1 dev enp1s0 src 10.10.0.20 <- falls to the DEFAULT
voffice1$ ip route | grep -E '10\.12\.|172\.31\.'
172.31.0.0/30 dev enp2s0 proto kernel scope link src 172.31.0.1
172.31.0.4/30 dev enp3s0 proto kernel scope link src 172.31.0.5
(ONLY the two transit /30s. No DC plane routes at all.)
voffice1$ ping -c2 10.12.8.1 -> 100% loss
voffice1$ ping -c2 10.12.8.100 -> 100% loss (a role node)
voffice1$ ping -c2 10.12.8.4 -> 100% loss (the mirror)
TCP 22 to 10.12.8.1 -> UNREACHABLE
MAAS commissioning was never affected because the RACK controller proxies DHCP/PXE
on-segment; the region needs no L3 to the nodes. Juju does.
3. WHY A ROUTE ALONE WOULD NOT FIX IT -- TWO INDEPENDENT, DELIBERATE BLOCKS
---------------------------------------------------------------------------
(a) SEC-010 transit FORWARD-drop, on the rack (gate G10 applied + verified it):
rack$ sudo nft list table inet sec010
table inet sec010 {
chain forward {
type filter hook forward priority filter - 10; policy accept;
oifname "enp1s0" drop
iifname "enp1s0" drop
}
}
enp1s0 on the rack is the Office1 transit leg (172.31.0.2/30). ALL forwarding in or
out of the transit is dropped. `drop` at priority filter-10 is terminal and runs
BEFORE libvirt's chains.
(b) The plane networks are ISOLATED libvirt nets. `virsh net-dumpxml
vr1-dc0-metal-admin` has NO <forward> and NO <ip> element (only name/bridge/mtu/
domain), which is what makes D-125's egress-isolation result true. libvirt
therefore installs a blanket reject on forwarding INTO the bridge:
rack$ sudo nft list ruleset | grep -A6 'chain LIBVIRT_FWI'
chain LIBVIRT_FWI {
oifname "virbr4" ... reject
oifname "virbr6" ... reject
oifname "virbr3" ... reject
oifname "virbr5" ... reject
oifname "virbr2" ... reject <- virbr2 = vr1-dc0-metal-admin
}
(counters 0 -- nothing has ever tried, consistent with this never having worked.)
(c) The DC edge is NOT a management router. It is attached to two networks only:
rack$ sudo virsh dumpxml vr1-dc0-opnsense | grep 'source network'
<source network='vr1-dc0-provider-public' ... bridge='virbr5'/>
<source network='vr1-dc0-wan' ... bridge='br-vr1-dc0-wan'/>
No metal-admin leg, so it cannot route the headend onto the node plane either.
CONCLUSION: nothing regressed. This deployment has never had a juju-client->controller
path. The rack host is the ONLY machine with legs on both sides
(enp1s0 172.31.0.2/30 transit; virbr2 10.12.8.2/3/4 metal-admin; virbr5 10.12.4.2
provider-public), and it is forbidden by SEC-010 from forwarding between them.
3b. SEC-010 IS NOT AN ACCIDENT -- IT FORBIDS EXACTLY THIS ROUTE, IN WRITING
---------------------------------------------------------------------------
Read the SEC-010 row in docs/security-ledger.md before framing any fix as a mere
firewall tweak. Its stated PURPOSE is the metal-admin DC-LOCAL invariant (D-052/D-100),
and it names the forbidden route explicitly. Quoting the row:
"metal-admin DC-LOCAL invariant (D-052/D-100) is PRESERVED but NOT ENFORCED in the
committed Stage-3 config."
"Cross-plane routing (metal-admin <-> the whole Office1 /22, bidirectional) is blocked
ONLY by Ubuntu's distro default; the deferred MAAS-rack install could silently flip it.
A MAAS rack proxies at the application layer and needs no kernel forwarding, so
pinning is free."
"Nothing routes across the fiber THROUGH vvr1-dc0; the rack proxies MAAS at the app
layer (originated/terminated, not forwarded)."
"Region route must target only the rack transit /30, never 10.12.8.0/22."
Status: CLOSED 2026-07-20 on the operator's exact utterance "Close SEC-010
(Recommended)", applied and verified BOTH ends in the logged dc0-deploy step-B window
(named gate `site-headend-install.sh --check` EXIT 0,
docs/audit/stepB-check-20260720-final.txt).
THE LOAD-BEARING SENTENCE IS THE JUSTIFICATION, AND IT ONLY EVER CONSIDERED MAAS:
"a MAAS rack proxies at the application layer and needs no kernel forwarding, so pinning
is free." That is TRUE of MAAS, and of the inner tofu root (qemu+ssh to the rack), and of
NetBox. It is FALSE of Juju: `juju bootstrap` and every subsequent agent connection dial
the MACHINE directly, at L3. Juju was not in scope when the cost of the pin was assessed,
so the pin was recorded as free when for this one tool it is not.
THE RESULTING CONTRADICTION, between surfaces that are BOTH ruled:
- D-100 says the Office1<->DC fiber "carries management traffic only (MAAS/Juju/
operator)" -- Juju named explicitly as management traffic the fiber carries.
- D-128 puts the Plane-2 execution host, and therefore the Juju CLIENT, on voffice1.
- SEC-010 + D-052 make metal-admin DC-LOCAL and forbid the region routing to
10.12.8.0/22 at all.
Under SEC-010's model the fiber carries management traffic that TERMINATES ON THE RACK.
Juju is the one Plane-2 tool that must instead reach THROUGH the rack to the node plane.
So the question is not "which firewall rule" but WHICH RULED SURFACE GOVERNS, with the
other amended. That is a D-number question (GA-R3 A1: a Roosevelt build session would
grep it before touching a built surface -- it decides where the Juju client lives in
every future DC).
NOTE ON A NAIVE FIX THAT MUST NOT BE REACHED FOR: making the plane nets
`<forward mode='route'/>` with an `<ip>` element would start a libvirt dnsmasq on
virbr2 and COLLIDE with MAAS's DHCP on that segment. If routing is ruled, the mode to
evaluate is `<forward mode='open'/>` (libvirt installs no filtering rules; the admin owns
them), which must be tested for whether it removes the LIBVIRT_FWI/FWO rejects WITHOUT
requiring `<ip>`. It needs net-destroy/net-start, which drops virbr2 and its manually
added .2/.3/.4 legs -- so `dc-rack-net.sh install <site>` must be re-run and re-checked
after, and `mtu 9000` preserved.
SCOPE THE FIX ONCE, BY ENUMERATION, IF ROUTING IS RULED: SSH/22 alone gets bootstrap
further and then fails at the Juju API on 17070. Stage 5+ also runs `openstack` against
VIPs from the headend (phase-03..06). Enumerate every plane and port the headend must
dial BEFORE writing a rule, or this gets fixed three times.
dc1 CARRIES THE IDENTICAL GAP: same `table inet sec010` shape on its own transit leg,
same isolated plane nets, same absent voffice1 routes. Whatever is ruled applies to both
DCs, and site-baseleg.sh's deferred DC rows are the templated home for the voffice1-side
half.
4. WHAT THE GAP REGISTER ALREADY SAID
--------------------------------------
scripts/site-baseleg.sh:42,47 --
# DC rows are DEFERRED: the DCs nest inside vvr1-dc0 and are reached by qemu+ssh
# [vr1-dc0]="<net>|<host-cidr>|<route-cidr>|<route-gw>" # MEASURE first (qemu+ssh may make this moot)
The deferral's stated assumption -- "reached by qemu+ssh" -- is TRUE for virsh/tofu
provisioning and FALSE for `juju bootstrap`, which needs real L3. Workflow-doc gap
register item 20 ("durable host->site reachability had no owning tool -- CLOSED for
Office1; DC path templated") is the item that has now reopened at the DC, which is
exactly the outcome the site-keyed tooling was built to prevent.
Repo-wide grep for prior art on this question returned ZERO hits:
grep -rn -iE 'juju client.*reach|reach.*controller|route to 10\.12\.8|10\.12\.8\.0/22 via|forward.*transit' \
docs/CURRENT-STATE.md docs/design-decisions.md runbooks/dc-dc-phase4-juju-bundle-per-dc.md
D-100 states the Office1<->DC fiber "carries management traffic only (MAAS/Juju/
operator)", so the INTENT exists; the implementation never did.
5. WHAT IS VIABLE, MEASURED
---------------------------
The rack CAN originate traffic to the MAAS region (SEC-010 drops FORWARD, not the
rack's own INPUT/OUTPUT):
juju cloud endpoint: {'vr1-maas': {'endpoint': 'http://10.10.0.20:5240/MAAS'}}
rack$ ip route get 10.10.0.20 -> via 172.31.0.1 dev enp1s0 src 172.31.0.2
rack$ TCP 10.10.0.20:5240 -> REACHABLE
rack$ command -v juju -> ABSENT (no client installed)
DC edge egress remains OPEN (re-probed this session from both racks, --noproxy '*'):
juju agent stream 200, api.snapcraft.io answering, archive 200, 1.1.1.1 0% loss.
6. SEPARATE FINDING, ALSO UNRESOLVED (ruled, then overtaken by this failure)
---------------------------------------------------------------------------
The controller VM holds an AUTO, v4-only address, unlike every role node:
7n87bt enp1s0 mode=auto ip=10.12.8.1 (no v6)
nhg3nf enp1s0 mode=static ip=10.12.8.100 fd50:840e:74e2:220::100
10.12.8.5 (the D-134-amendment ruled controller octet) is FREE.
Operator ruled 2026-07-30: "Fix now: static .5 + v6, re-bootstrap (Recommended)".
The v6 prefix is CONFIRMED BY PAIRING, not inferred from one node: dc0 metal-admin v4
subnet id 6 is vlan_id 5005 / fabric-4, and the v6 subnet on vlan 5005 / fabric-4 is
fd50:840e:74e2:220::/64. With the 2026-07-27 ruling that the v6 host part mirrors the
v4 octet, the controller's pair is 10.12.8.5 + fd50:840e:74e2:220::5.
This work is UNBLOCKED-BUT-NOT-DONE: it must be applied whichever reachability option
is ruled, and re-bootstrap now depends on that ruling.
