Pass 2 -- ADMINISTRATOR REPORT: tools review (container-layer elimination)

Author: the Phase-2 administrator (multi-agent pass, SCOPE-AND-EXECUTION-PLAN.md Section 4). Date: 2026-08-09. Inputs: pass2-w1-tofu-modules.md, pass2-w2-lib-hosts-net.md, pass2-w3-scripts.md, pass2-w4-module-decomposition.md -- read in full, adversarially cross-checked against repo ground truth (checks logged in Section 1). Baseline consumed: pass0-admin-report.md incl. Section 7a (Option 1 CONFIRMED; cross-DC handling (a) CONFIRMED; MAAS region stays on vr1-dcN-maas-01) and pass1-admin-report.md (planning change-set; Section 7 open items 1-7 carried to this phase). READ-ONLY synthesis; no mutation; findings are LOGGED, not executed. All recommendations here are proposals for Phase 4 -> operator (GA-R5); "recommend," never "ruled."

1. Adversarial-check results (run against repo ground truth this session)

1. THREE isolation concerns confirmed distinct, each real, each open -- Section 2 is the canonical separation. (i) the cross-DC plane-bridge adjacency on vcloud's kernel (Phase-0 Section 5, handling (a) operator-confirmed); (ii) the SEC-010 transit-leg FORWARD-drop successor (pass1 check 3); (iii) NEW this phase (W2.3 Sec 3): the MAAS power-key blast radius -- verified REAL this session, evidence in Section 2.3. They are two network controls and one credential-scope control; no one artifact covers two of them; root-shape (B) mitigates NONE of them (check 3 below).

2. W2.3's "rack controller already retired in practice" -- CONFIRMED, with one citation correction. Spot-checked all three cites by direct read:

   • docs/changelog-20260807-dc1-region-sequence.md:80-89: dc1 DHCP cutover executed 2026-08-07 -- primary_rack=qtw8pm in vr1-dc1-region; verified ss :67 on the region VM's enp1s0 only; "vvr1-dc1 has NO :67 on the node segment." CONFIRMED.
   • docs/changelog-20260730-dc0-region-migration.md Item 15 (~:532-549): dc0 DHCP handover table -- pgrep -c dhcpd on the rack = 0, ps -ef on hot-kid = 2 dhcpd on the region VM; primary_rack=c3aqh8 (= hot-kid in its own region). CONFIRMED, process-measured.
   • Citation correction: W2.3 cites changelog-20260807-dc0-tailscale-provisioning.md :200,254 for "BOTH maas-01 VMs installed region+rack." Read directly: line 200 is the PLANNED init instruction and line 254 the EXECUTED maas init region+rack ... --maas-url http://10.12.68.6:5240/MAAS --force -- both are dc1's .6. No dc0 init transcript was found in the changelogs (grep region+rack across changelog-). dc0's region+rack status rests on FUNCTIONAL evidence instead: hot-kid is a registered rack in vr1-dc0-region (changelog-20260730:49 "racks=hot-kid"), is primary_rack, and measurably runs the segment's only dhcpd (rackd-managed in MAAS). Evidence grade: dc1 = transcript; dc0 = functional. The claim carries at both grades; the DIRECTION (retire the standalone rack) is unaffected. W2.3's own caveat stands: *a current-day live re-measure (both DCs' primary_rack, vvr1-dcN rackd state) is OWED before "settled" -- the cites are 2-10 days old (instrument-currency #20/#25).

3. Root-topology (B): rationale confirmed, and it is about tofu STATE isolation, NOT L2 network isolation. W2.1 Sec 2.2's three winning axes are state blast radius, destroy scoping, and apply-ordering simplicity -- all state/apply-plane facts. Under (B) both DCs' plane bridges and node VMs STILL share vcloud's one libvirtd/kernel; W2.1 Sec 2.3 itself keeps the (a) control's --check "before the first tofu apply of EITHER per-DC-flat root." Do not read per-DC roots as addressing the cross-DC network gap (concern i) or the power-key blast radius (concern iii) -- (B) changes neither.

4. dc0/dc1 D-131 asymmetry: CONFIRMED by direct read. dc0's forwarder is NOT load-bearing: changelog-20260730-dc0-region-migration.md Item 9 (~:336-352) -- dns_servers=10.12.8.6 (the region's own BIND), dig-proven (archive.ubuntu.com, flags: qr rd ra, ANSWER: 9), "Pointing node DNS at the DC-LOCAL region is the correct end state." dc1's forwarder IS load-bearing: changelog-20260807-dc1-region-sequence.md :80-89 -- cutover config "replicated verbatim," dns_servers=10.12.68.3 (the D-131 forwarder alias). A real rebuild cleanup item: the 10.13 build sets each fresh region's own BIND from the start and collapses the asymmetry (Section 4.2).

5. Client-VM octet .8 / name vr1-dcN-client (W2.2) vs W2.1's grouping: NO CONFLICT. W2.2 assigns identity (D-134 utility band, .8 = next free slot after the .7 amendment; cross-DC standard so it binds every DC); W2.1 assigns the apply site (the per-DC flat root, via dc-site's client_vm input taking metal_admin_ip/transit_ip as NetBox-assigned inputs). Identity vs grouping -- orthogonal, both hold. W2.2's hedge is proper: .8 reads on the metal-admin plane; the transit leg keeps its D-124 /30 addressing (not a plane octet); exact leg addressing confirmed at build, not inferred.

6. Worker contradictions reconciled + inferred-claim sweep:

   • W2.4 Sec 5 row 2 vs W2.3 rack retirement -- superseded conditional, not a contradiction. W2.4 has site-headend-install.sh --role rack landing on the client VM "pending the OPEN placement ruling"; W2.3's measured finding recommends RETIRING the standalone rack enrollment entirely. W2.4's row was explicitly conditional; with retirement adopted, nothing rack-shaped lands on the client VM, whose duty roster shrinks to: D-138 client role + SEC-028/SEC-029 credential residencies + concern-(ii) DC-side endpoint + the transit leg. This simplifies W2.1's open dc-site client_vm.user_data content question.
   • W2.4 "no tooling gap here" (Sec 5 row 4) vs W2.3's decommission GAP -- reconciled. W2.4's claim was about re-targeting existing script bodies (true); W2.3 correctly identifies one missing artifact on the retirement path: a maas rack-controller delete decommission step for vvr1-dcN's Office1-registered rack object exists in NO script. Resolution: it FOLDS INTO owed artifact #5 (the MAAS machine-record release/delete step, which pass1 already scoped to include "region-side residue -- the rack-controller's own enrollment record + primary_rack/DHCP reference") -- amended to name the command class explicitly, plus W2.3's runbook note (new builds install directly as region+rack on the region VM, never a separate --role rack enrollment). Not double-counted (Section 5).
   • W2.4's "net new procedure-module bodies: exactly TWO" -- CORRECTED, do not carry. With W2.3's findings adopted, net-new bodies are at least FOUR: the (a) control, the teardown primitive, the power-key mitigation (+ its SEC row), and the dc-site IaC module (W2.4's count was procedure-only and predates the power-key finding). The SEC-010-writer extraction is a refactor of an existing body; the rack decommission step folds into #5.
   • W2.2 Sec 4 item 1 and W2.3 Sec 3 are the SAME object seen from two sides -- MERGED (Section 2.3): the undetermined power-address target (W2.2) and the blast radius of re-deriving it (W2.3). Consequence wired into the change-set: the lib-hosts.sh power-address re-derivation AND every call-site/runbook literal update (pass0 row 5) are BLOCKED on the mitigation design, because the mitigation (restricted key / wrapper / ACL) determines the URI and key shape. Not a literal swap.
   • No inferred-value violations found. W2.2 and W2.3 both state unknowns as OWED (power-address target, client-VM plane legs, dc1 cache sizing) rather than asserting. Spot-checked load-bearing cites resolved: maas-region-power-key.sh:1-40 (key resident ON the region host; SEC-012/SEC-016 per-DC), maas-node-power.sh:28-30 ("MEASURED to be the REGION" dials power), opentofu/vr1-dc0-substrate/main.tf: 176-181 (maas-01's 150 GiB earmarked for PostgreSQL + boot images, "no spare"), opentofu/main.tf:175 (voffice1 on the outer/vcloud root), D-134 octet map + D-143 C.3 as quoted by W2.2.

2. THE THREE ISOLATION CONCERNS -- separated, not conflated

Three distinct controls/mitigations are owed. Different layers, different attack surfaces, different SEC rows. None substitutes for another; root-shape (B) resolves none of them.

2.1 Concern (i): cross-DC plane-bridge NETWORK adjacency -> the (a) vcloud host control

Both DCs' six plane bridges + node VMs become co-resident on vcloud's ONE libvirtd/kernel for the first time in any deployed shape. Status: operator-confirmed at the Phase-0 gate (handling (a)); design requirement consolidated at pass1 Section 3 -- a vcloud-level nftables artifact asserting no inter-plane/inter-DC forwarding, with a mechanical --check gate, its own harness, a new SEC-NNN row, a gap-register entry, Stage-1 home, installed and verified before the first flat apply of EITHER per-DC root (fork-robust invariant, unchanged by (B)). Artifact kind: procedure/L5 gate, NOT a tofu module (W2.1 Sec 3.4 and W2.4 Sec 5 row 1 both re-confirm; built ONCE). This is a network/forwarding control on vcloud's own kernel. Owed artifact #2.

2.2 Concern (ii): the SEC-010 transit-leg FORWARD-drop successor

The interface-scoped FORWARD-drop on the Office1<->DC transit legs. The qemu+ssh purpose dissolves, but operator ssh -J and Office1-originated flows still ride the transit, and the protective claim (nothing routes from DC planes across the transit) is unchanged in spirit. Endpoints resolved by recommendation (W2.3 Sec 4, this phase's owned decision): client VM (DC side) + voffice1 (Office1 side). The client VM is structurally forced -- it is the ONLY DC-side VM with a transit leg under Option 1, and it reproduces SEC-010's original exposure shape ("straddles metal-admin + the transit," security-ledger.md:21) verbatim. voffice1's end never was containment-bound and does not move. Implementation shape: extract the SEC-010 nftables writer out of site-headend-install.sh's node_host_setup() (:273-320) into a role-agnostic subcommand that installs BOTH ends -- closing today's hand-mirrored voffice1 install rather than carrying it forward. Carry the NIC-naming trap (live dc0 interface was enp1s0, not mgmt; re-measure the client VM's transit NIC before writing the rule). This is a network control on the transit endpoints. Owed artifact #3 (spec amended).

2.3 Concern (iii): the MAAS power-key BLAST RADIUS -- NEW, VERIFIED REAL

The most consequential risk Phase 2 surfaced (W2.3 Sec 3), verified this session on all three legs:

1. The region VM holds the power key. scripts/maas-region-power-key.sh installs "the per-DC MAAS->libvirt power key inside a MAAS region's snap. Runs ON the region host" (SEC-012 dc0 / SEC-016 dc1, per-DC keys, never cross-DC reuse). maas-node-power.sh: 28-30: "MAAS dials the power address from whichever controller it chooses -- MEASURED to be the REGION." So each vr1-dcN-maas-01 holds a live qemu+ssh virsh credential.
2. voffice1 is a domain on vcloud's libvirtd. opentofu/main.tf:175 -- voffice1 is created by the OUTER root (qemu:///system on vcloud), alongside (post-flatten) both DCs' entire node fleets, edges, and client VMs.
3. One libvirtd connection scopes to all its domains. libvirt's default qemu:///system access model is all-or-nothing per connection -- no per-domain scoping. No repo artifact configures any libvirt access scoping (grep of scripts/ + opentofu/ for polkit/ACL: only an unrelated file-ACL comment in modules/opnsense-edge). A read of vcloud's LIVE polkit/libvirt config is delivery work, not asserted here.

Consequence: re-deriving the power address to vcloud's own libvirtd hands EACH DC's region VM a virsh credential with power control (start/destroy/undefine/console) over every domain vcloud manages: both DCs' fleets + voffice1 + the vcloud substrate (mesh/pool objects) -- virsh control, not a host shell; do not overstate. This recreates, through the power-credential door, exactly the cross-DC blast radius SEC-026/D-132 removed for the MAAS/cloud credential. SEC-012/SEC-016's per-DC key separation becomes vacuous under Option 1: the keys stay distinct but both open the same libvirtd. Neither concern (i) nor (ii) covers this (both are network controls; this is credential scope at the libvirt layer), and root-shape (B) does not touch it (state isolation, not connection scoping).

Owed: a THIRD control -- its own mitigation artifact + its own SEC-NNN row (owed artifact #11). Candidate mechanisms (Phase-4 choice, not picked here): a command=-restricted SSH key on vcloud (virsh RPC / per-DC domain-set allowlist), a per-DC-scoped virsh wrapper as the forced command, or a libvirt polkit ACL scoping each key's connection. Dependency wired into the change-set: the lib-hosts.sh power-address re-derivation (rows 2-3 of Section 3.2) and every maas-node-power.sh call-site/runbook literal are BLOCKED on this mitigation's design -- the mechanism determines the URI and key shape.

3. The tools change-set

3.1 OpenTofu roots + modules (W2.1 -- full table in pass2-w1-tofu-modules.md Sec 1)

Zero module bodies need rewriting. Of 12 module types: 1 collapses (wan-bridge -- D-125 bridge-in dead; edge WAN -> direct site-wan NAT), 1 dead/orthogonal (maas-vm-host, never instantiated; SEC-013 retire-or-keep flagged to its owner, not acted on), 3 untouched (office1-network, mesh-link -- resolving pass1 open item 6: the mesh triangle survives, only the office1-leg consumer changes from vvr1-dcN NIC1 to the client VM's transit NIC -- and netem-link), 1 rewired input (site-wan output feeds the DC edge directly), 6 re-home with unchanged bodies (cloudinit-vm -- loses the 2 containment calls, gains the client-VM call; dc-planes; dc-storage-pool -- 2-per-DC collapses to 1; node-vm x12/DC; opnsense-edge -- one input re-pointed; base-image).

Root topology: recommend (B) -- shared-outer + per-DC-flat roots (3 total). Rationale (state blast radius incl. this repo's own CLAUDE.md-cited incident class; destroy scoping = cd <dc-root> && tofu destroy, the closest re-earn of D-122's one-command site-down; apply-ordering: DC1's planes structurally cannot be created from DC0's root, so the (a) ordering invariant reduces to "check before the first apply of either root"). (A)'s only edge (DRY) is absorbed by dc-site. STATE isolation only -- see check 3. Directional, Phase 4 ratifies. Root naming (vr1-dcN-flat vs reserving -substrate) is Phase 4's call.

New IaC: modules/dc-site (highest-leverage new artifact) -- composes pool + 6 planes

• edge + 12 node VMs + client VM; replaces the ~230-266-line copy-pasted per-DC inner-root bodies; interface sketched W2.1 Sec 3.1 (site-token parameterized; client_vm.user_data stays a pass-through input -- now simpler, per the shrunken client-VM duty roster of check 6). A site-client-vm wrapper is explicitly NOT built yet. The client VM apply-groups with its DC's flat root (teardown symmetry, SEC-026 state-level isolation, D-138 fidelity). D-140 stays a separate future L4 root consuming dc-site outputs -- noted, not folded in.

3.2 lib-hosts.sh / lib-net.sh (W2.2 -- full table in pass2-w2-lib-hosts-net.md Sec 1)

3.3 Scripts (W2.3 -- full table in pass2-w3-scripts.md Sec 1)

3.4 Module decomposition (W2.4 -- full tables in pass2-w4-module-decomposition.md)

51 deploy-path scripts classified: 4 libraries, 13 gates, 34 procedure modules (32/34 with harnesses; the 2 gaps routed in Section 6). The procedure-module contract (7 points, all grounded in existing repo patterns: single-purpose, $SITE-parameterized, own harness, declared I/O + exit contract, idempotent, layer-respecting, changelog+revert) is ADOPTED as the Phase-4 backbone. The IaC<->procedure boundary is identity, not orchestration: tofu ends at "booted domain with correct MACs/planes"; procedures re-derive everything from live identity (MAC/hostname/API) and never read tofu state -- and Option 1 structurally removes the one existing violation (the inner root's qemu+ssh provider dial). Net-new bodies: corrected to at least FOUR (check 6): the (a) control (L5 gate), the teardown primitive (L2/L5 wrapper), the concern-(iii) mitigation, and the dc-site IaC module. Everything else is re-instantiation or re-targeting via existing parameterization.

4. Resolutions of the Phase-2-owned open decisions (all: recommendation grade, Phase 4 ratifies)

4.1 Root topology (pass1 open item 1): (B) shared-outer + per-DC-flat roots

Per Section 3.1. Teardown primitive builds against (B): gated tofu destroy of one DC root

• emergency virsh destroy loop over that root's state-listed domains. NOT an L2-isolation answer (check 3).

4.2 Rack-controller remainder (pass1 open item 2, "THE highest-leverage open item") --

largely DISSOLVES, three components, three answers (W2.3, verified check 2):

• (i) MAAS rack controller: RETIRE the standalone registration. Both maas-01 VMs already run region+rack (dc1 transcript-grade, dc0 functional-grade -- check 2); DHCP authority measurably cut over on both DCs (2026-07-30 / 2026-08-07). vvr1-dcN's rackd is a vestigial Office1-region registration doing no work. Delta artifact = the decommission step folded into owed #5 + the runbook note (new builds init region+rack on the region VM directly; --role rack retires). OWED before settled: current-day live re-measure. Phase-4 ride-along: this touches D-132-addendum PREMISES (nothing is being co-located; the addendum's hypervisor-fate rationale is moot under Option 1) -- name it in the [ARCH] package.
• (ii) D-131 forwarder: RETIRE-WITH-EVIDENCE as the 10.13 end state for BOTH DCs. D-131's precondition (rack-only controller, remote region) is removed by D-132's per-DC regions; dc0 already proves the end state (dig-verified region BIND); dc1's forwarder is presently load-bearing (verbatim-replicated config) -- the asymmetry is real and must not be assumed equal (check 4). The fresh 10.13 regions set their own BIND from the start; the retirement EVIDENCE step (the dig test dc0's migration used) is owed artifact #13. If retirement is rejected, the forwarder co-locates with component (i)'s host.
• (iii) Artifact service (.4): a right-sized, explicitly-provisioned home -- a STORAGE decision, not a MAAS-adjacency one. Neither existing VM fits dc0's full mirror as authored (verified sizing cites, Section 3.3). Options: dedicated volume on a chosen VM, or a small dedicated utility VM. Decided WITH NUMBERS via the FIT-calculator extension (owed #7, spec amended to include mirror-disk sizing; dc0 full-mirror vs dc1 cache per D-135) -- not guessed here.

Net effect: the client VM's duty roster SHRINKS to D-138 client role + SEC-028/029 credential residencies + concern-(ii) DC-side endpoint + transit leg (check 6) -- no rackd, no forwarder, and the artifact service only if the sizing decision puts it there.

4.3 Client-VM octet + name (pass1 open item 5): octet .8, name vr1-dcN-client

(W2.2 Sec 2; D-134's ruled utility-band table enumerates through .7; cross-DC standard so .8 binds every DC; octet-preserving under D-143). Consistent with W2.1's flat-root grouping (check 5). The D-134 map addition is a RULING -- Phase 4 package.

4.4 SEC-010 successor endpoints (pass1 open item 4): client VM + voffice1, one

extracted installer for both ends (Section 2.2).

4.5 Also resolved at this phase

• Mesh-triangle survival (pass1 open item 6): CONFIRMED at the module level -- all 3 mesh-link legs + netem-link unchanged; only the office1-leg consumer re-points.
• site-headend-install.sh refactor scope (pass1 open item 7): delete --host-nodes (~134 lines) + extract the SEC-010 writer + retire --role rack contingent on 4.2(i).
• Client-VM lib-hosts identity: no row (Section 3.2).
• The (a) control's mechanism (pass1 open item 3): NOT resolved this phase -- concrete nftables rule set/check shape/SEC number remain Phase-3/4 design work; kind, home, and ordering are settled (Section 2.1).

5. OWED ARTIFACTS -- consolidated (Phase-1 list updated; count = 13)

Not double-counted: the rack decommission (folds into #5), artifact-service sizing (rides

#7), the SEC-010-writer extraction (IS #3's implementation shape).

6. OPEN -- carried forward

To Phase 3 (tests): harness requirements for #1/#2/#3/#11/#12; the maas-fabric-prune.sh/maas_fabric_classify.py harness gap (ROUTE as a named decision: build vs accept-as-named-exception -- pre-existing, surfaced W2.4); lib-identity.sh harness status (W2.4 flagged to W2.2, which did not cover it -- an unclosed worker handoff, re-routed to Phase 3); which existing harnesses assume the container layer (Phase 3's own charter).

To Phase 4 (decision framing; operator rules, GA-R5):

1. The container-elim [ARCH] ruling (D-123 amendment vs new D) + ride-alongs: D-125 retirement, D-138 concrete-host, D-128 amendment (Plane 2 shrinks), D-122 site-down re-earn, D-124 sizing-void, + NEW: the D-132-addendum premises note (4.2 i).
2. Ratify root topology (B) + root NAMING (-flat vs reserving -substrate).
3. Ratify rack-retirement (i) / D-131 retire-with-evidence (ii) -- after the owed live re-measure -- and rule the artifact-service home WITH the #7 numbers.
4. Rule the client-VM .8 octet into the D-134 standing map + the name.
5. Choose the concern-(iii) mitigation mechanism + mint its SEC row; then unblock the power-address re-derivation (URI + FROM_OFFICE1/FROM_DCREGION split question).
6. The (a) control's concrete mechanism + SEC number (with Phase 3's harness spec).
7. wan-bridge module directory: delete vs leave-unreferenced (repo append-only bias).
8. SEC-013 maas-vm-host retire-or-keep -- flagged to its owner, not this pass's ruling.
9. State-blast-radius weighing rides item 2 (per pass1).

Owed live measurements (delivery, not this pass): current-day rack/primary_rack state (4.2 i); vcloud's live polkit/libvirt access config (2.3); plus pass0 Sec 8's standing three (capacity, FIT, geneve/jumbo).

7. Settled vs open -- two grades, kept distinct

OPERATOR-CONFIRMED (do not re-open): Option-1 target; handling (a) as a required deliverable; MAAS region stays on vr1-dcN-maas-01; plus pass1's settled list (Stage-2/ D-114 out of scope; D-140 pins L4 as procedure; two-axis attribution; (a) ordering invariant + Stage-1 home).

RESOLVED BY RECOMMENDATION -- pending Phase-4 ratification (GA-R5): root topology (B); rack-controller retirement; D-131 retire-with-evidence; artifact-service = explicit sizing decision via #7; client-VM .8 + vr1-dcN-client; SEC-010 endpoints (client VM + voffice1) + single-installer shape; client VM out of lib-hosts.sh/CARVE_AUX_HOSTS; mesh triangle unchanged; dc-site as the IaC composition unit.

OPEN: everything in Section 6; the concern-(iii) mechanism; the (a) mechanism; the power-address value (blocked, by design, on #11).

8. Verification note

Author = "the administrator" (no model name asserted, operator instruction). Direct reads this session: the four worker docs in full; docs/changelog-20260807-dc1-region-sequence.md :70-95; docs/changelog-20260730-dc0-region-migration.md:330-355,525-555 + grep for region+rack/hot-kid; docs/changelog-20260807-dc0-tailscale-provisioning.md:180-260 (incl. the :200/:254 cite correction); scripts/maas-region-power-key.sh:1-60; scripts/maas-node-power.sh:25-35; opentofu/vr1-dc0-substrate/main.tf:172-185; opentofu/main.tf module greps; repo-wide polkit/ACL grep. Worker citations were spot-checked at their load-bearing points, not re-derived wholesale; every correction in Section 1 names its source lines. READ-ONLY; findings LOGGED only; nothing executed.