diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index f39ef1e..89e3a12 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -284,6 +284,41 @@ (`2602:f3e2:f02:10::5`) was DELIBERATELY NOT applied**: a default route needs only the v4 leg, and handing the controller a public GUA is an operator call. Recorded so a later session does not silently "fix" the asymmetry against role nodes. + **CARVE APPLIED 2026-07-30, BOTH CONTROLLERS; THIRD BOOTSTRAP RUN; NEW BLOCKER FOUND AND + RULED.** The carve landed after the machine was released to `Ready` (MAAS REFUSES interface + changes on a `Deployed` machine -- and note the trap: `interface update ... vlan=` returned a + full JSON object while changing NOTHING, and the refusal only surfaced later as + `{"subnet": ["None found with id=7."]}` because MAAS scopes the subnet lookup to the + interface's VLAN. **A 200-shaped MAAS response is not evidence the state changed; read the + interface back.**) Final state, symmetric: `moved-troll enp1s0 10.12.8.5 + + fd50:840e:74e2:220::5, enp2s0 10.12.4.5 gw=10.12.4.1`; `square-ferret enp1s0 10.12.68.5 + + fd50:840e:74e2:320::5, enp2s0 10.12.64.5 gw=10.12.64.1`. **THE UNDER-CARVE FIX WORKED, + measured on the redeployed node:** `default via 10.12.4.1 dev enp2s0` and + `agent-stream=200`, and the bootstrap then logged `Attempt 1 ... Agent binaries downloaded + successfully` -- the 339-retry failure class is CLOSED. + **THIRD BOOTSTRAP FAILED at the next layer:** `ERROR creating MAAS environ: Get + "http://10.10.0.20:5240/MAAS/api/2.0/version/": connect: connection refused`. `jujud` runs ON + the controller node and dials the MAAS region API for its whole life. MEASURED: the rack + reaches that API (`code=200`, rack-ORIGINATED, permitted by SEC-010); the node cannot + (FORWARDED, blocked by SEC-010 + libvirt's isolated-net rejects); and the rack does NOT proxy + it -- `ss -lntp` shows nginx on **5248** only, nothing on 5240. All six required flows were + enumerated and exactly ONE crosses the boundary. **This means SEC-010 as written is + incompatible with the deployment's own control-plane topology** (D-104 controller IN the DC + + MAAS region at Office1). **D-138 was necessary but NOT sufficient: it moved the cloud-facing + CLIENT into the DC, and `jujud` is itself a provider client whose MAAS dependency was not + enumerated when D-138 was scoped.** + **RULED 2026-07-30 -- D-132 question 1 ADOPTED FOR VR1 (GA-R5). Operator answer, exact + utterance: "Put a MAAS region controller in each DC".** Each DC gets its own MAAS region so + the provider API is DC-LOCAL and no MAAS traffic crosses the fiber. **SEC-010 is PRESERVED + UNAMENDED** -- the requirement is removed rather than excepted. SINGLE, not HA (the utterance + is singular; D-132's HA sub-questions stay pinned to the next deployment). D-132 questions 2 + and 3 remain PROPOSED. **This REOPENS the MAAS layer of Stages 3 and 4, both CLOSED and + MERGED** -- a MAAS region owns its own database, so machines, fabrics, VLANs, subnets, spaces, + reserved ranges, tags, images, DHCP and power config do not migrate between regions. + **STAGE 5 IS BLOCKED until the per-DC regions exist.** **OPEN SUB-QUESTION, blocking the + build: WHERE each DC's region lives** (rack host vs its own DC VM; if a VM, the D-134 octet + map needs a ruled octet -- `.6` is next free after `.4` artifact and `.5` Juju controller) -- + present as its own GA-R5 exchange before any install. - Project: Omega Cloud, VR1 DC-DC rehearsal -- a two-DC + Office1-headend virtual rehearsal on KVM (vcloud host), rehearsing the future bare-metal diff --git a/docs/design-decisions.md b/docs/design-decisions.md index c03b2ce..dc2aea8 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -6536,3 +6536,68 @@ answer is that no host-side leg is wanted. Workflow-doc gap register item 20's DC half CLOSES on this ruling rather than on a tool. Failure capture: `docs/audit/stage5-bootstrap-reachability-20260730.txt`. + +--- + +## D-132 -- AMENDMENT (2026-07-30): question 1 RULED FOR VR1 -- a MAAS REGION CONTROLLER PER DC, pulled forward from the next deployment [ARCH] + +**Status: RULED 2026-07-30** (operator, GA-R5) for the VR1 deployment. D-132's questions 2 +(rack-top controllers) and 3 (cross-site backup custody) remain PROPOSED and stay pinned to +the next deployment; only the per-DC region shape is ruled here. + +**Question as presented.** The third Stage-5 bootstrap attempt got PAST the juju agent fetch +and then failed with `ERROR creating MAAS environ: Get +"http://10.10.0.20:5240/MAAS/api/2.0/version/": connect: connection refused`. `jujud` runs ON +the controller node INSIDE the DC and dials the MAAS region API continuously for its whole +life -- not only at bootstrap. MEASURED: the rack reaches that API (`code=200`) because +SEC-010 permits rack-ORIGINATED traffic; the NODE cannot, because that is FORWARDED traffic, +blocked by SEC-010's transit drop AND by libvirt's isolated-net rejects. The rack does NOT +proxy the region API -- `ss -lntp` shows nginx on **5248** (metadata) and nothing on 5240. + +**The finding underneath the failure: SEC-010 as written is incompatible with the +deployment's own control-plane topology.** D-104 puts the Juju controller IN the DC; the MAAS +region lives at Office1; so the provider API dependency MUST cross the fiber at L3. D-100 +says that fiber exists to carry exactly this traffic. All six flows the deployment needs were +enumerated and exactly ONE crosses the boundary (controller `10.12.8.5` -> `10.10.0.20:5240`); +the other five are same-segment or already-working rack-originated paths. + +**Options put:** (a) a narrow SEC-010 exception for that single flow; (b) a MAAS region +controller per DC, removing the cross-fiber dependency entirely; (c) stop and re-plan the +control-plane topology. + +**Operator answer, exact utterance: "Put a MAAS region controller in each DC".** + +**RULED.** Each DC gets its OWN MAAS region controller, so the Juju controller's provider API +is DC-LOCAL and no MAAS traffic crosses the Office1 fiber. **SEC-010 is PRESERVED UNAMENDED** +-- this resolution removes the requirement rather than granting an exception to it, which is +why it was chosen over option (a). D-138 stands and is NECESSARY BUT WAS NOT SUFFICIENT: it +moved the cloud-facing CLIENT into the DC, but `jujud` is itself a provider client and its +MAAS dependency was not enumerated when D-138 was scoped. This amendment closes that gap. + +**SINGLE, NOT HA, FOR VR1.** The operator's utterance is singular ("a MAAS region +controller"). D-132 question 1's HA sub-questions (member count, postgres HA, region API VIP) +stay PROPOSED for the next deployment. This mirrors the D-104 economy -- single-unit Juju +controller for VR1, HA deferred to Roosevelt. + +**CONSEQUENCE, STATED PLAINLY: this REOPENS the MAAS layer of Stages 3 and 4, both of which +are CLOSED and MERGED to `main`.** A MAAS region owns its own database; machines, fabrics, +VLANs, subnets, spaces, reserved ranges, tags, images, DHCP and power configuration do not +migrate between regions. Per DC this means: install region + PostgreSQL; re-enrol and +re-commission 9 nodes; recreate the D-133 flat carve, the D-134 bands, the D-119 + role tags, +the six spaces and their VLANs; re-sync images; move DHCP; re-wire the SEC-012/-016 power +credentials; and re-point the Juju cloud endpoint. Much of this is already scripted +(`dc-rack-net.sh`, `maas-role-tags.sh`, `carve-host-interfaces.sh`, `maas-node-power.sh`, the +apex importer) but every one of those scripts currently assumes the single Office1 region and +will need a region-target parameter. **Office1's region is RETAINED for the Office1 site +itself** (`office1-netbox`, `office1-tailscale`). + +**OPEN SUB-QUESTION, blocking the build and NOT ruled by this amendment: WHERE each DC's +region controller lives** -- on the DC rack host alongside its rack controller, or in its own +VM inside the DC. If a VM, the D-134 amendment's standing octet map requires a ruled octet in +the utility band (`.4` artifact service, `.5` Juju controller, so `.6` is the next free) and +that VM must be authored into the DC substrate root. Present as its own GA-R5 exchange before +any install. + +**Roosevelt analog:** this IS the Roosevelt shape, arriving one deployment early -- per-DC +regions are what D-132 was filed to propose. The rehearsal value is real: the cross-fiber +provider-API dependency would have been discovered at Roosevelt scale otherwise.