Newer
Older
openstack-caracal-dc-dc / docs / audit / container-elim-pass / pass2-w3-scripts.md

Pass 2 -- WORKER W2.3: carve/power/network scripts (container-layer elimination)

Author: Phase-2 worker W2.3 (multi-agent pass, SCOPE-AND-EXECUTION-PLAN.md Section 4). Date: 2026-08-09. Scope: scripts/maas-node-power.sh, scripts/dc-rack-net.sh, scripts/site-headend-install.sh, the carve scripts (dc-node-carve.sh, dc-node-v6-carve.py, carve-host-interfaces.sh, maas-role-tags.sh), scripts/ site-baseleg.sh, scripts/dc-mirror.sh/dc-cache-proxy.sh. Owns the two highest-leverage Phase-1 open decisions (rack-remainder placement; SEC-010 successor endpoints). READ-ONLY; findings LOGGED only, nothing executed. Inputs read in full: SCOPE-AND-EXECUTION-PLAN.md, pass0-admin-report.md, pass1-admin-report.md, plus the scripts themselves and the design-decisions / security-ledger / changelog citations below.

Baseline consumed: Option 1 CONFIRMED (flat node VMs on vcloud libvirt + one small non-hypervisor vr1-dcN-client VM per DC carrying the D-138 client role + SEC-028/SEC-029 credential residencies); cross-DC handling (a) CONFIRMED (new vcloud-level host isolation control, Phase-1 design item, not this worker's dimension); MAAS region stays on vr1-dcN-maas-01 (no change); rack-controller-remainder placement OPEN (pass1 Section 7 open item 2, "THE highest-leverage open item").


0. A load-bearing fact this worker surfaced, not present in pass0/pass1

The "rack-controller remainder" is substantially ALREADY RETIRED IN PRACTICE for both DCs' DHCP/enrollment duty -- this is measured, live-executed history, not a proposal.

  • dc0 (docs/changelog-20260730-dc0-region-migration.md items 9, 10, 15): the D-132 region migration moved DHCP from Office1's rack (primary_rack=7chphy, i.e. vvr1-dc0) to the region VM vr1-dc0-maas-01 ("hot-kid", primary_rack=c3aqh8 in its own region). Executed 2026-07-30, read back by PROCESS (pgrep/ps -ef), not MAAS self-report: "2 dhcpd on the region VM; rack still 0" (item 15 step 4). Node DNS was ALSO measured and switched: item 9 -- dns_servers=10.12.8.6 (the region VM's own BIND, not the D-131 forwarder .3), proven live (dig against 10.12.8.6 answers archive.ubuntu.com / maas-internal SOA correctly, flags: qr rd ra). Item 9's own words: "Pointing node DNS at the DC-LOCAL region is the correct end state... removes the cross-fiber dependency that D-132 q1 exists to remove."
  • dc1 (docs/changelog-20260807-dc1-region-sequence.md Item 2): the same DHCP handover ran 2026-08-07 (primary_rack=qtw8pm in vr1-dc1-region; verified ss :67 on enp1s0 only, vvr1-dc1 has NO :67). But DNS was NOT re-derived -- the cutover explicitly "replicated verbatim" the old config, so dc1's region still carries dns_servers=10.12.68.3 (the D-131 forwarder alias), unlike dc0's corrected 10.12.8.6. This is a real, present ASYMMETRY between the two DCs, not a documentation gap: dc1 still depends on the forwarder today; dc0 does not.

Implication for Decision 1: the MAAS-rack-as-DHCP-server function has already left vvr1-dcN for both DCs. What remains genuinely resident on vvr1-dcN today is (a) its own idle rack-controller registration (a rackd process enrolled to Office1's region, doing no DHCP work), (b) the D-131 forwarder (load-bearing for dc1, provably NOT load-bearing for dc0), and (c) the artifact service. These three components do not need one placement answer -- they were never one thing operationally, even though they shared a host.


1. Per-script disposition table

Script:line Containment binding today Option-1 change
scripts/maas-node-power.sh (whole file, 123 lines) NONE in code. POWER_ADDRESS is $1 (:41), validated only as a qemu*:// URI shape (:51-54); PREFIX is $2 (:42). No hostname, IP, or containment literal is baked in -- confirmed by full read. NO CODE CHANGE. Every invocation site / runbook example that passes qemu+ssh://<user>@172.31.0.2/system (the containment VM's libvirtd) must instead pass vcloud's own libvirt endpoint. The new value is UNKNOWN -- an OWED live measurement, not inferred (hard rule 2). See Section 4 for why this value change is not "just a literal swap."
scripts/dc-rack-net.sh (226 lines) Whole-file: runs ON the rack host (:8, "RUNS ON THE DC RACK HOST (e.g. vvr1-dc0)"); LEGS table (:62-64, :78-80) is bare ip addr on the containment VM's own inner libvirt bridges, re-applied via br_of() -> virsh -c qemu:///system net-dumpxml (:102-104) -- a host-with-its-own-bridges pattern that only exists because vvr1-dcN IS the inner libvirtd. DNS_UPSTREAM (:66, :81) hardcodes 10.10.0.20 (Office1's BIND) for BOTH sites -- STALE for dc0 per Section 0 (measured 2026-07-30 that dc0 now points dns_servers at its own region, not this forwarder's upstream chain at all). RETIRE-WITH-EVIDENCE for the legs half; component-dependent for the DNS half (Section 4). No per-DC VM under Option 1 is a libvirt host with its own bridges -- flat sibling VMs get their addresses from guest netplan / MAAS statics, and the (a) cross-DC control's whole posture is "vcloud keeps no L3 legs on DC planes" (pass1 Section 3). The LEGS/br_of() mechanism has no home to move to; it retires with the containment layer itself. The DNS-forwarder half is a SEPARATE decision, Section 4.
scripts/site-headend-install.sh (565 lines) --role rack (:14-20, :371-409): enrolls a DC rack to Office1's region -- STILL A LIVE CODE PATH even though DHCP has moved off it (Section 0); --host-nodes / node_host_setup() (:248-342, ~95 lines) + node_host_check() (:206-244, ~39 lines) = ~134 lines, the D-123 Model-B node-hosting bootstrap (nested KVM, inner pool dir, AppArmor grant, OPNsense base staging, D-125 WAN-bridge verify). Embedded in node_host_setup(): the SEC-010 nftables writer (:273-320, ~48 lines) -- the one piece of this block that is NOT dead. node_host_setup()/node_host_check() (~134 lines): DEAD, delete wholesale -- no inner root, no nested libvirt, no OPNsense-inner-edge staging under Option 1. EXTRACT the SEC-010 writer (:273-320) OUT of node_host_setup() into its own role-agnostic subcommand (e.g. --transit-drop --transit-if <name>), so it can run on the client VM (and, per Section 5, on voffice1 too, replacing today's hand-mirrored install) WITHOUT dragging in the dead nested-KVM setup. --role rack itself (region-enrollment only, no --host-nodes): its disposition is CONTINGENT on Decision 1 -- retired outright if the rack-controller identity fully consolidates onto vr1-dcN-maas-01 (Section 4 recommendation); kept, retargeted to the client VM, only if the operator rejects that consolidation. D-125 WAN-bridge verify code (:90-94, :228-243, :322-335): dead, D-125 bridge-in itself retires per pass0 (edge WAN -> direct NAT).
scripts/dc-node-carve.sh (485 lines), scripts/dc-node-v6-carve.py (301 lines), scripts/carve-host-interfaces.sh (301 lines), scripts/maas-role-tags.sh (210 lines) NONE found. Grepped all four for vvr1, containment, qemu+ssh, inner, 172.31, POWER, voffice1 -- zero hits in every file. Confirmed by header read: these operate purely against the MAAS API (machine records by MAC/tag, maas <profile> machines/interfaces/...), run "where the maas CLI lives / the D-128 Plane-2 host" (dc-node-v6-carve.py:109, maas-role-tags.sh:51) -- a value/profile question, not a containment-binding one. NO CODE CHANGE. These are already topology-agnostic; the only currency item is where they are invoked from (D-128's Plane-2 host shrinks per the D-128 amendment pass1 flagged, check 6) and which MAAS_PROFILE they target -- both are invocation-parameter concerns, already handled by the existing MAAS_PROFILE/--profile plumbing these scripts carry.
scripts/site-baseleg.sh:40-48 DC rows are commented-out placeholders (`# [vr1-dc0]=" ...",:47), explicitly deferred: "the DCs nest inside vvr1-dc0 and are reached by qemu+ssh... NOT necessarily an L3 leg" (:41-42`). D-138 (2026-07-30) already answered this for the CURRENT shape: "no host-side leg is wanted" (design-decisions.md:7127-7129, cross-referenced from this file's own deferred-row comment). Premise moot, not merely re-answered. Under Option 1 there is still no vcloud-side L3 leg wanted onto a DC plane (the (a) control's entire point is the opposite -- no cross-plane forwarding on vcloud's kernel). Stays a no-op; the comment block should be updated to cite D-138 + the (a) control rather than the retired qemu+ssh premise, but this is a doc-currency edit, not a behavior change. LOW.
scripts/dc-mirror.sh (385 lines), scripts/dc-cache-proxy.sh (396 lines) Whole-file: "RUNS ON THE DC RACK HOST (e.g. vvr1-dc0)" (dc-mirror.sh:6, dc-cache-proxy.sh:13), except dc-cache-proxy.sh node which runs on a DC node (:18, unaffected). .4 listen alias added on a metal-admin libvirt bridge via the same rack-legs mechanism as dc-rack-net.sh (dc-mirror.sh:16-23). dc0's mirror pulls "several hundred GB" (jammy main/restricted/universe/multiverse + jammy-updates/security + UCA jammy-updates/caracal, :29-34); dc1's is a lighter apt-cacher-ng cache, not a full mirror (D-135 amendment, per-DC strategy split). New host + explicit disk sizing, NOT a doc-only relabel. The rack-legs .4-alias mechanism retires with dc-rack-net.sh's legs half (row above); the artifact SERVICE itself needs a fresh host binding. Neither Option-1 utility VM is sized for dc0's mirror footprint as currently authored: vr1-dc0-maas-01 is 4 vCPU / 8 GiB / 150 GiB disk and that 150 GiB is EXPLICITLY earmarked for "the region's PostgreSQL AND its boot-image set" (opentofu/vr1-dc0-substrate/main.tf:179-180), not spare; the client VM is sized ~4/8192/80 (pass0 Section 4). See Section 4 component 3. dc1's cache-proxy footprint is materially smaller and less likely to force a resizing decision, but should not be assumed to fit without the same sizing pass.

2. OPEN DECISION 1 -- rack-controller-remainder placement (per component)

Three components, three separate recommendations -- they were never one placement problem (Section 0).

Component (i): the MAAS rack controller

Recommendation: RETIRE the standalone rack registration; let vr1-dcN-maas-01's own region+rack install (already present) be the DC's sole rack.

  • This is smaller than "co-locate rack onto the region VM" -- it is already true today. vr1-dc0-maas-01 and vr1-dc1-maas-01 were BOTH installed with maas init region+rack (docs/changelog-20260807-dc0-tailscale-provisioning.md:200, :254; the region+rack form, not region-only), so each already runs its own local rackd. DHCP authority for both DCs was measured and cut over to that local rackd 2026-07-30 (dc0) / 2026-08-07 (dc1), verified by live process, not self-report (Section 0). vvr1-dcN's own rackd is a vestigial registration to Office1's region doing no work.
  • Do NOT cite site-headend-install.sh --role region+rack as the delta artifact. That role is the D-114 Office1/voffice1 build path and carries LXD install + LXD vm-host registration + compose-network DHCP (traps 1-4 of that script) -- none of which vr1-dcN-maas-01 needs or has. The maas-01 VMs were already built correctly via a direct maas init region+rack --database-uri ... --maas-url ... sequence (docs/changelog-20260807-dc0-tailscale-provisioning.md:196-203), NOT via this script's region+rack role. The delta artifact this recommendation needs is therefore a GAP, not a reuse: a formal decommission step for vvr1-dcN's Office1-registered rack object (maas admin rack-controller delete or equivalent -- not yet in any script) and a runbook note that new-build racks going forward install DIRECTLY as region+rack on the region VM the way maas-01 already was, never as a separate --role rack enrollment.
  • scripts/maas-node-power.sh --profile/MAAS_PROFILE plumbing is unaffected -- power config already targets whichever profile is passed; this is a rack-identity question, not a power-config one.
  • OWED before this is treated as settled, not before it is recommended: re-measure CURRENT live state (both DCs' primary_rack binding, vvr1-dcN's rackd process state) as of THIS session (2026-08-09) -- the cited changelogs are 2026-07-30/2026-08-07, and this is a READ-ONLY planning pass, so the measurement is Phase-2/3 delivery work, not asserted here as current fact. The DIRECTION is well-evidenced; the CURRENT-DAY confirmation is not yet taken.
  • Ride-along note for Phase 4 [ARCH] framing (not ruled here, GA-R5): eliminating the separate rack does not touch the D-132 addendum's actual RULING ("region... NOT on the rack host" -- design-decisions.md:7232-7234) in the way "co-locate rack onto region" would have, because nothing is being co-located; the separate rack is being retired, and the addendum's own stated rationale (region must not share fate with "the hypervisor running every node it manages") is structurally moot under Option 1 regardless (no VM is a hypervisor for another VM). Still name this explicitly in the Phase-4 package alongside the D-128/D-125/D-138 ride-alongs pass1 already flagged (Section 7 item 10) -- it touches the same ruled decision's premises even though it does not reverse its letter.

Component (ii): the D-131 node-DNS forwarder

Recommendation: RETIRE-WITH-EVIDENCE as the target end state for BOTH DCs under the 10.13 rebuild; carry the asymmetry honestly in the interim.

  • D-131's own title scopes it to "rack-only controllers" (design-decisions.md:5653) -- its SERVFAIL bug is specifically the rack-only agent resolver walking public root hints with a remote region (docs/audit/commissioning-diag-20260721.txt, cited dc-rack-net.sh:29). D-132's per-DC region (regiond + BIND, on metal-admin) removes the precondition: a node can take dns_servers=<region's own address> directly, and the workaround has nothing left to work around.
  • dc0 already proves this, measured, not hypothesized: dns_servers=10.12.8.6 (the region VM's own BIND) answers archive.ubuntu.com/maas-internal correctly with dig (docs/changelog-20260730-dc0-region-migration.md:337-349); the migration's own words: "Pointing node DNS at the DC-LOCAL region is the correct end state... removes the cross-fiber dependency that D-132 q1 exists to remove." dc-rack-net.sh:66,81's DNS_UPSTREAM="10.10.0.20" for both sites is confirmed STALE by the same changelog entry (item 9's own follow-up note) -- an instrument-currency finding in this script's own site table, not a live-state guess. dc1 is NOT yet at this end state -- its 2026-08-07 cutover explicitly "replicated verbatim" the old forwarder-pointed config (dns_servers=10.12.68.3, docs/changelog-20260807-dc1-region-sequence.md:85-87) rather than re-deriving it, so dc1's forwarder is presently load-bearing.
  • For the 10.13 rebuild specifically: since BOTH DCs are being rebuilt from scratch under Option 1, the forwarder need not be stood up at all -- set dns_servers at each fresh region's own BIND address from the start (dc0's already-proven pattern), and RE-VERIFY with the same live-answer test dc0's migration used (dig for archive.ubuntu.com + the maas-internal SOA) against the new build before calling this closed. That collapses the asymmetry rather than carrying it into 10.13.
  • If retirement is rejected (e.g. a Roosevelt-transfer argument for keeping a forwarder pattern rehearsed): co-locate it with whichever host carries the rack-controller registration decision (component i) -- dc-rack-net.sh's forwarder half is coupled to the same host's metal-admin leg by construction (gen_dns_unit's Requires=${SITE}-rack-legs. service, :155), so it has no independent placement logic once the legs half is decided.

Component (iii): the artifact service (.4 mirror/proxy)

Recommendation: a right-sized, explicitly-provisioned home, decided at Phase 2 (W2.1 IaC module design) -- NOT silently inherited from whichever VM absorbs components (i)/(ii).

  • This is a storage decision, not a MAAS-adjacency decision -- unlike the rack controller and forwarder, the mirror/proxy has no dependency on being co-resident with MAAS. Sizing rules it out of both existing Option-1 candidates as currently authored: vr1-dcN-maas-01 is 150 GiB disk, already earmarked for "the region's PostgreSQL AND its boot-image set" (opentofu/vr1-dc0-substrate/main.tf:179-180, explicit comment, no spare capacity claimed); the client VM is sized ~4/8192/80 (pass0 Section 4) -- neither has headroom for dc0's "several hundred GB" full debmirror (dc-mirror.sh:29-34: jammy main/restricted/universe/multiverse + updates/security + UCA jammy-updates/caracal).
  • dc1's dc-cache-proxy.sh footprint (an apt-cacher-ng cache, not a full mirror -- D-135 amendment's deliberate per-DC strategy split) is materially smaller and a plausible fit on either existing VM with a modest disk bump, but should not be assumed without the same sizing pass -- an inferred disk-size claim here would be exactly the hard-rule-2 trap.
  • Concretely: either (a) attach a dedicated volume to whichever VM ends up hosting it, sized explicitly for D-135's known per-DC footprint (dc0 full-mirror vs dc1 cache), or (b) keep it a distinct small utility VM if the operator wants mirror-storage growth isolated from either control-plane VM's disk. Both are legitimate; the FIT-calculator extension pass1 already flagged as owed (Section 6 item 7) is the right place to settle it with numbers rather than here with a guess.
  • dc-rack-net.sh's legs mechanism that the mirror's .4 alias rides today retires with the rest of that script's legs half (Section 1); the new host's .4 alias becomes a guest-netplan/MAAS-static concern like every other flat-VM address, not a host-level ip addr replace unit.

3. New finding: the re-derived power address is a cross-DC credential blast-radius risk

scripts/maas-node-power.sh needs no code change (Section 1), but the VALUE change is not innocent. Today, per DC, the qemu+ssh power key dials ONLY that DC's own nested libvirtd (the containment VM) -- POWER_ADDRESS per DC is scoped to that DC's own hardware. Under Option 1, node VMs move to being flat siblings directly on vcloud's own libvirtd, so the re-derived power address becomes vcloud's own qemu:///system (or a qemu+ssh:// dial into vcloud) -- one endpoint reachable from BOTH DCs' region VMs (each DC's vr1-dcN-maas-01 independently dials it for its own power control, per the "credential note" already in the script: :28-30, "MAAS dials the power address from... the REGION").

A per-DC region VM holding a virsh key that reaches vcloud's libvirtd has virsh power over everything on vcloud -- both DCs' node fleets, voffice1, the jumphost's own substrate -- not just its own DC's hardware. This recreates, through a different door, exactly the cross-DC blast radius SEC-026/D-132 worked to remove (D-132's whole point was DC-local MAAS so a DC-local host does not have region-wide reach; this reintroduces region-wide REACH via the power-control credential even though MAAS itself stays DC-local). Neither the (a) cross-DC network-isolation control nor the SEC-010 successor (Section 4) covers this -- both are network/forwarding controls; this is a credential-scope problem at the libvirt layer.

Not in pass0 rows 4-5 or pass1's carried-forward items. Flagged as an owed SEC-row + mitigation design for Phase 2/4: a command=-restricted SSH key (virsh RPC allowlist) or a per-DC-scoped virsh wrapper/ACL on vcloud's libvirtd, so each region VM's power key can only touch its own DC's domain set, matching the per-DC isolation SEC-026 already establishes for the MAAS/cloud credential.


4. OPEN DECISION 2 -- the SEC-010 transit-leg FORWARD-drop successor endpoints

Recommendation: client VM (DC side) + voffice1 (Office1 side), unchanged shape, distinct from the (a) control.

  • The client VM is the structurally-forced DC-side successor, not a choice among several: per the confirmed Option-1 shape, the client VM is the only DC-side VM that carries a transit leg at all (vr1-dcN-client, "legs = metal-admin + transit", pass0 Section 4). It reproduces SEC-010's ORIGINAL exposure shape essentially verbatim -- the ledger's own words for what SEC-010 was written against: the host "straddles metal-admin (DC-local) + the office1<->dc0 transit (crosses fiber)" (docs/security-ledger.md:21) -- so the protective claim carries without reinterpretation: nothing should route FROM the DC's node planes THROUGH the client VM ACROSS the transit leg; only the client VM's own originated/terminated traffic (operator ssh -J, any client-VM-terminated calls) should cross it.
  • voffice1's end is unchanged -- it was never containment-bound (it is not vvr1-dcN), and its peer role on the SAME physical mesh leg does not move under Option 1.
  • Re-author, do not blind-copy, the rule content: the qemu+ssh purpose that motivated the original drop is gone, but the drop's actual protective claim (no forwarding across the transit leg) is unchanged in spirit, so the SAME interface-scoped nftables idiom applies (oifname "$TRANSIT_IF" drop / iifname "$TRANSIT_IF" drop in the forward hook, site-headend-install.sh:299-305) -- only $TRANSIT_IF re-targets to the client VM's own transit NIC name.
  • Naming-trap precedent, cite it explicitly at build time: the LIVE dc0 interface was enp1s0, not the script's default mgmt -- "netplan set-name dropped" (docs/security-ledger.md:21, close note). Do not assume the client VM's transit NIC keeps any prior name; re-measure it live before writing the rule, same discipline hard rule 2 already requires.
  • Consolidate the install, do not re-hand-mirror it: today voffice1's end was applied as an "identical scoped artifact" by HAND, separately from the rack's scripted install (docs/security-ledger.md:21, "voffice1... identical scoped artifact + sec010-fw.service enabled"). The extraction recommended in Section 1 (pull the SEC-010 writer out of node_host_setup() into a standalone, role-agnostic subcommand) should make ONE tested artifact install BOTH ends, closing that hand-mirroring gap rather than carrying it forward.
  • Distinct from control (a), the new vcloud-level cross-DC host-isolation control (pass0 Section 5, pass1 Section 3) -- two controls, two SEC rows, do not merge them. Control (a) guards vcloud's OWN kernel against inter-plane/inter-DC forwarding now that both DCs' planes are co-resident on one libvirtd; the SEC-010 successor guards the Office1<->DC transit leg specifically. Different hosts, different attack surfaces, different SEC-NNN rows (pass1 check 3, "CONFIRMED distinct, BOTH OPEN, neither dropped").

5. Top risks (bounded)

  1. Power-address blast radius (Section 3) -- the highest-value new finding this worker surfaced; no existing control covers it.
  2. dc1's D-131 forwarder is presently load-bearing, dc0's is not -- an asymmetry that must not be silently assumed equal when authoring the retirement step for 10.13.
  3. Artifact-service sizing -- neither Option-1 candidate VM has spare disk for dc0's full mirror as currently authored; a placement chosen without the FIT-calculator pass (pass1 Section 6 item 7) risks an under-sized build discovered mid-sync (hours-long job).
  4. --role rack's disposition in site-headend-install.sh is contingent on Decision 1's outcome -- do not delete it prematurely; it is only fully dead if the rack-consolidation recommendation (Section 2, component i) is adopted.
  5. All Decision-1/Decision-2 recommendations are proposals for the Phase-2 administrator -> Phase 4 -> operator (GA-R5); none is ruled here.

Durable doc: docs/audit/container-elim-pass/pass2-w3-scripts.md (this file).