# Pass 2 -- WORKER W2.3: carve/power/network scripts (container-layer elimination) **Author:** Phase-2 worker W2.3 (multi-agent pass, `SCOPE-AND-EXECUTION-PLAN.md` Section 4). **Date:** 2026-08-09. **Scope:** `scripts/maas-node-power.sh`, `scripts/dc-rack-net.sh`, `scripts/site-headend-install.sh`, the carve scripts (`dc-node-carve.sh`, `dc-node-v6-carve.py`, `carve-host-interfaces.sh`, `maas-role-tags.sh`), `scripts/ site-baseleg.sh`, `scripts/dc-mirror.sh`/`dc-cache-proxy.sh`. Owns the two highest-leverage Phase-1 open decisions (rack-remainder placement; SEC-010 successor endpoints). READ-ONLY; findings LOGGED only, nothing executed. Inputs read in full: `SCOPE-AND-EXECUTION-PLAN.md`, `pass0-admin-report.md`, `pass1-admin-report.md`, plus the scripts themselves and the design-decisions / security-ledger / changelog citations below. **Baseline consumed:** Option 1 CONFIRMED (flat node VMs on vcloud libvirt + one small non-hypervisor `vr1-dcN-client` VM per DC carrying the D-138 client role + SEC-028/SEC-029 credential residencies); cross-DC handling (a) CONFIRMED (new vcloud-level host isolation control, Phase-1 design item, not this worker's dimension); MAAS region stays on `vr1-dcN-maas-01` (no change); rack-controller-remainder placement OPEN (pass1 Section 7 open item 2, "THE highest-leverage open item"). --- ## 0. A load-bearing fact this worker surfaced, not present in pass0/pass1 **The "rack-controller remainder" is substantially ALREADY RETIRED IN PRACTICE for both DCs' DHCP/enrollment duty -- this is measured, live-executed history, not a proposal.** - **dc0** (`docs/changelog-20260730-dc0-region-migration.md` items 9, 10, 15): the D-132 region migration moved DHCP from Office1's rack (`primary_rack=7chphy`, i.e. `vvr1-dc0`) to the region VM `vr1-dc0-maas-01` ("hot-kid", `primary_rack=c3aqh8` **in its own region**). Executed 2026-07-30, read back by PROCESS (`pgrep`/`ps -ef`), not MAAS self-report: **"2 dhcpd on the region VM; rack still 0"** (item 15 step 4). Node DNS was ALSO measured and switched: item 9 -- `dns_servers=10.12.8.6` (the region VM's own BIND, not the D-131 forwarder `.3`), proven live (`dig` against `10.12.8.6` answers `archive.ubuntu.com` / `maas-internal` SOA correctly, `flags: qr rd ra`). Item 9's own words: **"Pointing node DNS at the DC-LOCAL region is the correct end state... removes the cross-fiber dependency that D-132 q1 exists to remove."** - **dc1** (`docs/changelog-20260807-dc1-region-sequence.md` Item 2): the same DHCP handover ran 2026-08-07 (`primary_rack=qtw8pm` in `vr1-dc1-region`; verified `ss :67` on `enp1s0` only, `vvr1-dc1` has NO `:67`). **But DNS was NOT re-derived** -- the cutover explicitly **"replicated verbatim"** the old config, so dc1's region still carries `dns_servers=10.12.68.3` (the D-131 forwarder alias), unlike dc0's corrected `10.12.8.6`. This is a real, present ASYMMETRY between the two DCs, not a documentation gap: dc1 still depends on the forwarder today; dc0 does not. **Implication for Decision 1:** the MAAS-rack-as-DHCP-server function has already left `vvr1-dcN` for both DCs. What remains genuinely resident on `vvr1-dcN` today is (a) its own idle rack-controller *registration* (a rackd process enrolled to Office1's region, doing no DHCP work), (b) the D-131 forwarder (load-bearing for dc1, provably NOT load-bearing for dc0), and (c) the artifact service. These three components do not need one placement answer -- they were never one thing operationally, even though they shared a host. --- ## 1. Per-script disposition table | Script:line | Containment binding today | Option-1 change | |---|---|---| | `scripts/maas-node-power.sh` (whole file, 123 lines) | **NONE in code.** `POWER_ADDRESS` is `$1` (`:41`), validated only as a `qemu*://` URI shape (`:51-54`); `PREFIX` is `$2` (`:42`). No hostname, IP, or containment literal is baked in -- confirmed by full read. | **NO CODE CHANGE.** Every *invocation site* / runbook example that passes `qemu+ssh://@172.31.0.2/system` (the containment VM's libvirtd) must instead pass vcloud's own libvirt endpoint. The new value is **UNKNOWN -- an OWED live measurement**, not inferred (hard rule 2). See Section 4 for why this value change is not "just a literal swap." | | `scripts/dc-rack-net.sh` (226 lines) | Whole-file: runs ON the rack host (`:8`, "RUNS ON THE DC RACK HOST (e.g. vvr1-dc0)"); `LEGS` table (`:62-64`, `:78-80`) is bare `ip addr` on the containment VM's **own inner libvirt bridges**, re-applied via `br_of()` -> `virsh -c qemu:///system net-dumpxml` (`:102-104`) -- a host-with-its-own-bridges pattern that only exists because `vvr1-dcN` IS the inner libvirtd. `DNS_UPSTREAM` (`:66`, `:81`) hardcodes `10.10.0.20` (Office1's BIND) for BOTH sites -- STALE for dc0 per Section 0 (measured 2026-07-30 that dc0 now points `dns_servers` at its own region, not this forwarder's upstream chain at all). | **RETIRE-WITH-EVIDENCE for the legs half; component-dependent for the DNS half (Section 4).** No per-DC VM under Option 1 is a libvirt host with its own bridges -- flat sibling VMs get their addresses from guest netplan / MAAS statics, and the (a) cross-DC control's whole posture is "vcloud keeps no L3 legs on DC planes" (pass1 Section 3). The `LEGS`/`br_of()` mechanism has no home to move to; it retires with the containment layer itself. The DNS-forwarder half is a SEPARATE decision, Section 4. | | `scripts/site-headend-install.sh` (565 lines) | `--role rack` (`:14-20`, `:371-409`): enrolls a DC rack to Office1's region -- STILL A LIVE CODE PATH even though DHCP has moved off it (Section 0); `--host-nodes` / `node_host_setup()` (`:248-342`, ~95 lines) + `node_host_check()` (`:206-244`, ~39 lines) = **~134 lines, the D-123 Model-B node-hosting bootstrap** (nested KVM, inner pool dir, AppArmor grant, OPNsense base staging, D-125 WAN-bridge verify). Embedded in `node_host_setup()`: the SEC-010 nftables writer (`:273-320`, ~48 lines) -- the one piece of this block that is NOT dead. | **`node_host_setup()`/`node_host_check()` (~134 lines): DEAD, delete wholesale** -- no inner root, no nested libvirt, no OPNsense-inner-edge staging under Option 1. **EXTRACT the SEC-010 writer (`:273-320`) OUT of `node_host_setup()` into its own role-agnostic subcommand** (e.g. `--transit-drop --transit-if `), so it can run on the client VM (and, per Section 5, on voffice1 too, replacing today's hand-mirrored install) WITHOUT dragging in the dead nested-KVM setup. `--role rack` itself (region-enrollment only, no `--host-nodes`): its disposition is **CONTINGENT on Decision 1** -- retired outright if the rack-controller identity fully consolidates onto `vr1-dcN-maas-01` (Section 4 recommendation); kept, retargeted to the client VM, only if the operator rejects that consolidation. D-125 WAN-bridge verify code (`:90-94`, `:228-243`, `:322-335`): dead, D-125 bridge-in itself retires per pass0 (edge WAN -> direct NAT). | | `scripts/dc-node-carve.sh` (485 lines), `scripts/dc-node-v6-carve.py` (301 lines), `scripts/carve-host-interfaces.sh` (301 lines), `scripts/maas-role-tags.sh` (210 lines) | **NONE found.** Grepped all four for `vvr1`, `containment`, `qemu+ssh`, `inner`, `172.31`, `POWER`, `voffice1` -- zero hits in every file. Confirmed by header read: these operate purely against **the MAAS API** (machine records by MAC/tag, `maas machines/interfaces/...`), run "where the maas CLI lives / the D-128 Plane-2 host" (`dc-node-v6-carve.py:109`, `maas-role-tags.sh:51`) -- a value/profile question, not a containment-binding one. | **NO CODE CHANGE.** These are already topology-agnostic; the only currency item is *where they are invoked from* (D-128's Plane-2 host shrinks per the D-128 amendment pass1 flagged, check 6) and *which `MAAS_PROFILE`* they target -- both are invocation-parameter concerns, already handled by the existing `MAAS_PROFILE`/`--profile` plumbing these scripts carry. | | `scripts/site-baseleg.sh:40-48` | DC rows are commented-out placeholders (`# [vr1-dc0]="|..."`, `:47`), explicitly deferred: "the DCs nest inside vvr1-dc0 and are reached by qemu+ssh... NOT necessarily an L3 leg" (`:41-42`). D-138 (2026-07-30) already answered this for the CURRENT shape: "no host-side leg is wanted" (design-decisions.md:7127-7129, cross-referenced from this file's own deferred-row comment). | **Premise moot, not merely re-answered.** Under Option 1 there is still no vcloud-side L3 leg wanted onto a DC plane (the (a) control's entire point is the opposite -- no cross-plane forwarding on vcloud's kernel). Stays a no-op; the comment block should be updated to cite D-138 + the (a) control rather than the retired qemu+ssh premise, but this is a doc-currency edit, not a behavior change. LOW. | | `scripts/dc-mirror.sh` (385 lines), `scripts/dc-cache-proxy.sh` (396 lines) | Whole-file: "RUNS ON THE DC RACK HOST (e.g. vvr1-dc0)" (`dc-mirror.sh:6`, `dc-cache-proxy.sh:13`), except `dc-cache-proxy.sh node` which runs on a DC node (`:18`, unaffected). `.4` listen alias added on a metal-admin libvirt bridge via the same rack-legs mechanism as `dc-rack-net.sh` (`dc-mirror.sh:16-23`). dc0's mirror pulls "several hundred GB" (jammy main/restricted/universe/multiverse + jammy-updates/security + UCA jammy-updates/caracal, `:29-34`); dc1's is a lighter apt-cacher-ng cache, not a full mirror (D-135 amendment, per-DC strategy split). | **New host + explicit disk sizing, NOT a doc-only relabel.** The rack-legs `.4`-alias mechanism retires with `dc-rack-net.sh`'s legs half (row above); the artifact SERVICE itself needs a fresh host binding. Neither Option-1 utility VM is sized for dc0's mirror footprint as currently authored: `vr1-dc0-maas-01` is `4 vCPU / 8 GiB / 150 GiB disk` and that 150 GiB is EXPLICITLY earmarked for "the region's PostgreSQL AND its boot-image set" (`opentofu/vr1-dc0-substrate/main.tf:179-180`), not spare; the client VM is sized `~4/8192/80` (pass0 Section 4). See Section 4 component 3. dc1's cache-proxy footprint is materially smaller and less likely to force a resizing decision, but should not be assumed to fit without the same sizing pass. | --- ## 2. OPEN DECISION 1 -- rack-controller-remainder placement (per component) Three components, three separate recommendations -- they were never one placement problem (Section 0). ### Component (i): the MAAS rack controller **Recommendation: RETIRE the standalone rack registration; let `vr1-dcN-maas-01`'s own `region+rack` install (already present) be the DC's sole rack.** - **This is smaller than "co-locate rack onto the region VM" -- it is already true today.** `vr1-dc0-maas-01` and `vr1-dc1-maas-01` were BOTH installed with `maas init region+rack` (`docs/changelog-20260807-dc0-tailscale-provisioning.md:200`, `:254`; the region+rack form, not region-only), so each already runs its own local rackd. DHCP authority for both DCs was measured and cut over to that local rackd 2026-07-30 (dc0) / 2026-08-07 (dc1), verified by live process, not self-report (Section 0). `vvr1-dcN`'s own rackd is a vestigial registration to Office1's region doing no work. - **Do NOT cite `site-headend-install.sh --role region+rack` as the delta artifact.** That role is the D-114 Office1/voffice1 build path and carries LXD install + LXD vm-host registration + compose-network DHCP (traps 1-4 of that script) -- none of which `vr1-dcN-maas-01` needs or has. The maas-01 VMs were already built correctly via a direct `maas init region+rack --database-uri ... --maas-url ...` sequence (`docs/changelog-20260807-dc0-tailscale-provisioning.md:196-203`), NOT via this script's region+rack role. The delta artifact this recommendation needs is therefore a GAP, not a reuse: a formal decommission step for `vvr1-dcN`'s Office1-registered rack object (`maas admin rack-controller delete` or equivalent -- not yet in any script) and a runbook note that new-build racks going forward install DIRECTLY as `region+rack` on the region VM the way maas-01 already was, never as a separate `--role rack` enrollment. - **`scripts/maas-node-power.sh --profile`/`MAAS_PROFILE` plumbing is unaffected** -- power config already targets whichever profile is passed; this is a rack-identity question, not a power-config one. - **OWED before this is treated as settled, not before it is recommended:** re-measure CURRENT live state (both DCs' `primary_rack` binding, `vvr1-dcN`'s rackd process state) as of THIS session (2026-08-09) -- the cited changelogs are 2026-07-30/2026-08-07, and this is a READ-ONLY planning pass, so the measurement is Phase-2/3 delivery work, not asserted here as current fact. The DIRECTION is well-evidenced; the CURRENT-DAY confirmation is not yet taken. - **Ride-along note for Phase 4 [ARCH] framing (not ruled here, GA-R5):** eliminating the separate rack does not touch the D-132 addendum's actual RULING ("region... NOT on the rack host" -- design-decisions.md:7232-7234) in the way "co-locate rack onto region" would have, because nothing is being co-located; the separate rack is being retired, and the addendum's own stated rationale (region must not share fate with "the hypervisor running every node it manages") is structurally moot under Option 1 regardless (no VM is a hypervisor for another VM). Still name this explicitly in the Phase-4 package alongside the D-128/D-125/D-138 ride-alongs pass1 already flagged (Section 7 item 10) -- it touches the same ruled decision's premises even though it does not reverse its letter. ### Component (ii): the D-131 node-DNS forwarder **Recommendation: RETIRE-WITH-EVIDENCE as the target end state for BOTH DCs under the 10.13 rebuild; carry the asymmetry honestly in the interim.** - D-131's own title scopes it to "rack-only controllers" (`design-decisions.md:5653`) -- its SERVFAIL bug is specifically the rack-only agent resolver walking public root hints with a remote region (`docs/audit/commissioning-diag-20260721.txt`, cited `dc-rack-net.sh:29`). D-132's per-DC region (regiond + BIND, on metal-admin) removes the precondition: a node can take `dns_servers=` directly, and the workaround has nothing left to work around. - **dc0 already proves this, measured, not hypothesized:** `dns_servers=10.12.8.6` (the region VM's own BIND) answers `archive.ubuntu.com`/`maas-internal` correctly with `dig` (`docs/changelog-20260730-dc0-region-migration.md:337-349`); the migration's own words: "Pointing node DNS at the DC-LOCAL region is the correct end state... removes the cross-fiber dependency that D-132 q1 exists to remove." `dc-rack-net.sh:66,81`'s `DNS_UPSTREAM="10.10.0.20"` for both sites is confirmed STALE by the same changelog entry (item 9's own follow-up note) -- an instrument-currency finding in this script's own site table, not a live-state guess. **dc1 is NOT yet at this end state** -- its 2026-08-07 cutover explicitly "replicated verbatim" the old forwarder-pointed config (`dns_servers=10.12.68.3`, `docs/changelog-20260807-dc1-region-sequence.md:85-87`) rather than re-deriving it, so dc1's forwarder is presently load-bearing. - **For the 10.13 rebuild specifically:** since BOTH DCs are being rebuilt from scratch under Option 1, the forwarder need not be stood up at all -- set `dns_servers` at each fresh region's own BIND address from the start (dc0's already-proven pattern), and RE-VERIFY with the same live-answer test dc0's migration used (`dig` for `archive.ubuntu.com` + the `maas-internal` SOA) against the new build before calling this closed. That collapses the asymmetry rather than carrying it into 10.13. - **If retirement is rejected** (e.g. a Roosevelt-transfer argument for keeping a forwarder pattern rehearsed): co-locate it with whichever host carries the rack-controller registration decision (component i) -- `dc-rack-net.sh`'s forwarder half is coupled to the same host's metal-admin leg by construction (`gen_dns_unit`'s `Requires=${SITE}-rack-legs. service`, `:155`), so it has no independent placement logic once the legs half is decided. ### Component (iii): the artifact service (`.4` mirror/proxy) **Recommendation: a right-sized, explicitly-provisioned home, decided at Phase 2 (W2.1 IaC module design) -- NOT silently inherited from whichever VM absorbs components (i)/(ii).** - This is a **storage** decision, not a MAAS-adjacency decision -- unlike the rack controller and forwarder, the mirror/proxy has no dependency on being co-resident with MAAS. Sizing rules it out of both existing Option-1 candidates as currently authored: `vr1-dcN-maas-01` is `150 GiB` disk, already earmarked for "the region's PostgreSQL AND its boot-image set" (`opentofu/vr1-dc0-substrate/main.tf:179-180`, explicit comment, no spare capacity claimed); the client VM is sized `~4/8192/80` (pass0 Section 4) -- neither has headroom for dc0's "several hundred GB" full debmirror (`dc-mirror.sh:29-34`: jammy main/restricted/universe/multiverse + updates/security + UCA jammy-updates/caracal). - dc1's `dc-cache-proxy.sh` footprint (an apt-cacher-ng cache, not a full mirror -- D-135 amendment's deliberate per-DC strategy split) is materially smaller and a plausible fit on either existing VM with a modest disk bump, but should not be assumed without the same sizing pass -- an inferred disk-size claim here would be exactly the hard-rule-2 trap. - **Concretely:** either (a) attach a dedicated volume to whichever VM ends up hosting it, sized explicitly for D-135's known per-DC footprint (dc0 full-mirror vs dc1 cache), or (b) keep it a distinct small utility VM if the operator wants mirror-storage growth isolated from either control-plane VM's disk. Both are legitimate; the FIT-calculator extension pass1 already flagged as owed (Section 6 item 7) is the right place to settle it with numbers rather than here with a guess. - `dc-rack-net.sh`'s legs mechanism that the mirror's `.4` alias rides today retires with the rest of that script's legs half (Section 1); the new host's `.4` alias becomes a guest-netplan/MAAS-static concern like every other flat-VM address, not a host-level `ip addr replace` unit. --- ## 3. New finding: the re-derived power address is a cross-DC credential blast-radius risk `scripts/maas-node-power.sh` needs no code change (Section 1), but the VALUE change is not innocent. Today, per DC, the qemu+ssh power key dials ONLY that DC's own nested libvirtd (the containment VM) -- `POWER_ADDRESS` per DC is scoped to that DC's own hardware. Under Option 1, node VMs move to being **flat siblings directly on vcloud's own libvirtd**, so the re-derived power address becomes vcloud's own `qemu:///system` (or a `qemu+ssh://` dial into vcloud) -- **one endpoint reachable from BOTH DCs' region VMs** (each DC's `vr1-dcN-maas-01` independently dials it for its own power control, per the "credential note" already in the script: `:28-30`, "MAAS dials the power address from... the REGION"). A per-DC region VM holding a virsh key that reaches vcloud's libvirtd has **virsh power over everything on vcloud** -- both DCs' node fleets, `voffice1`, the jumphost's own substrate -- not just its own DC's hardware. This recreates, through a different door, exactly the cross-DC blast radius SEC-026/D-132 worked to remove (D-132's whole point was DC-local MAAS so a DC-local host does not have region-wide reach; this reintroduces region-wide REACH via the power-control credential even though MAAS itself stays DC-local). **Neither the (a) cross-DC network-isolation control nor the SEC-010 successor (Section 4) covers this** -- both are network/forwarding controls; this is a credential-scope problem at the libvirt layer. Not in pass0 rows 4-5 or pass1's carried-forward items. **Flagged as an owed SEC-row + mitigation design** for Phase 2/4: a `command=`-restricted SSH key (virsh RPC allowlist) or a per-DC-scoped virsh wrapper/ACL on vcloud's libvirtd, so each region VM's power key can only touch its own DC's domain set, matching the per-DC isolation SEC-026 already establishes for the MAAS/cloud credential. --- ## 4. OPEN DECISION 2 -- the SEC-010 transit-leg FORWARD-drop successor endpoints **Recommendation: client VM (DC side) + voffice1 (Office1 side), unchanged shape, distinct from the (a) control.** - **The client VM is the structurally-forced DC-side successor**, not a choice among several: per the confirmed Option-1 shape, the client VM is the only DC-side VM that carries a transit leg at all (`vr1-dcN-client`, "legs = metal-admin + transit", pass0 Section 4). It reproduces SEC-010's ORIGINAL exposure shape essentially verbatim -- the ledger's own words for what SEC-010 was written against: the host "straddles metal-admin (DC-local) + the office1<->dc0 transit (crosses fiber)" (`docs/security-ledger.md:21`) -- so the protective claim carries without reinterpretation: nothing should route FROM the DC's node planes THROUGH the client VM ACROSS the transit leg; only the client VM's own originated/terminated traffic (operator `ssh -J`, any client-VM-terminated calls) should cross it. - **voffice1's end is unchanged** -- it was never containment-bound (it is not `vvr1-dcN`), and its peer role on the SAME physical mesh leg does not move under Option 1. - **Re-author, do not blind-copy, the rule content:** the qemu+ssh purpose that motivated the original drop is gone, but the drop's actual protective claim (no forwarding across the transit leg) is unchanged in spirit, so the SAME interface-scoped nftables idiom applies (`oifname "$TRANSIT_IF" drop` / `iifname "$TRANSIT_IF" drop` in the forward hook, `site-headend-install.sh:299-305`) -- only `$TRANSIT_IF` re-targets to the client VM's own transit NIC name. - **Naming-trap precedent, cite it explicitly at build time:** the LIVE dc0 interface was `enp1s0`, not the script's default `mgmt` -- "netplan set-name dropped" (`docs/security-ledger.md:21`, close note). Do not assume the client VM's transit NIC keeps any prior name; re-measure it live before writing the rule, same discipline hard rule 2 already requires. - **Consolidate the install, do not re-hand-mirror it:** today voffice1's end was applied as an "identical scoped artifact" by HAND, separately from the rack's scripted install (`docs/security-ledger.md:21`, "voffice1... identical scoped artifact + `sec010-fw.service` enabled"). The extraction recommended in Section 1 (pull the SEC-010 writer out of `node_host_setup()` into a standalone, role-agnostic subcommand) should make ONE tested artifact install BOTH ends, closing that hand-mirroring gap rather than carrying it forward. - **Distinct from control (a), the new vcloud-level cross-DC host-isolation control (pass0 Section 5, pass1 Section 3) -- two controls, two SEC rows, do not merge them.** Control (a) guards vcloud's OWN kernel against inter-plane/inter-DC forwarding now that both DCs' planes are co-resident on one libvirtd; the SEC-010 successor guards the Office1<->DC transit leg specifically. Different hosts, different attack surfaces, different SEC-NNN rows (pass1 check 3, "CONFIRMED distinct, BOTH OPEN, neither dropped"). --- ## 5. Top risks (bounded) 1. **Power-address blast radius (Section 3)** -- the highest-value new finding this worker surfaced; no existing control covers it. 2. **dc1's D-131 forwarder is presently load-bearing, dc0's is not** -- an asymmetry that must not be silently assumed equal when authoring the retirement step for 10.13. 3. **Artifact-service sizing** -- neither Option-1 candidate VM has spare disk for dc0's full mirror as currently authored; a placement chosen without the FIT-calculator pass (pass1 Section 6 item 7) risks an under-sized build discovered mid-sync (hours-long job). 4. **`--role rack`'s disposition in `site-headend-install.sh` is contingent on Decision 1's outcome** -- do not delete it prematurely; it is only fully dead if the rack-consolidation recommendation (Section 2, component i) is adopted. 5. All Decision-1/Decision-2 recommendations are proposals for the Phase-2 administrator -> Phase 4 -> operator (GA-R5); none is ruled here. **Durable doc:** `docs/audit/container-elim-pass/pass2-w3-scripts.md` (this file).