Newer
Older
openstack-caracal-dc-dc / docs / audit / container-elim-pass / pass0-admin-report.md

Pass 0 -- ADMINISTRATOR REPORT: baseline & target topology (container-layer elimination)

Author: the Phase-0 administrator (multi-agent pass, SCOPE-AND-EXECUTION-PLAN.md Section 4). Date: 2026-08-09. Inputs: pass0-w1-substrate-map.md, pass0-w2-network-map.md, pass0-w3-consumers.md, pass0-w4-targets.md -- read in full, adversarially cross-checked against repo ground truth (checks logged in Section 3). READ-ONLY synthesis; no mutation.

This report ends in the Phase-0 OPERATOR GATE (Section 7). Do not start Phase 1 before the operator rules on it.


1. As-is map (W0.1 + W0.2 reconciled into one picture)

1.1 The substrate (Model B, D-122/D-123)

vcloud (outer libvirt, qemu:///system)               <- OUTER ROOT: opentofu/main.tf
|-- Office1 arm: office1_network / office1-wan / office1_opnsense / voffice1  (NOT container-layer)
|-- ubuntu_noble_base (shared base image)                                     (persists)
|-- D-100 mesh TRIANGLE (mesh-link x3, MTU 9000) + netem on dc0<->dc1         (persists)
|     office1<->dc0 leg = D-124 transit: voffice1 NIC2 172.31.0.1/30 <-> vvr1-dc0 NIC1 172.31.0.2/30
|     office1<->dc1 leg = transit:       voffice1 NIC3 172.31.0.5/30 <-> vvr1-dc1 NIC1 172.31.0.6/30
|-- vr1_dc0_uplink / vr1_dc1_uplink (site-wan NAT 172.30.2.0/24 / 172.30.3.0/24, MTU 1500, D-125)
|-- vr1_dc0_storage / vr1_dc1_storage (outer pool -- holds ONLY the containment VM's own disk)
|
`-- vvr1-dc0 / vvr1-dc1  ==== CONTAINMENT VM (D-123 Model B) ====
      108 vCPU / 480 GiB / ~3000 GiB (opentofu/variables.tf:137-156; RAM raised 416->480 2026-08-01)
      NIC1 = transit (SEC-010 FORWARD-drop keys here); NIC2 = IP-less, enslaved into the
      br-vr1-dcN-wan netplan bridge (D-125 bridge-in); expose_nested_virt = true
      |
      |== BOOTSTRAP GATE (scripts/site-headend-install.sh --role rack --host-nodes):
      |   nested libvirtd + inner pool dir + kvm nested=1 + OPNsense image staging +
      |   SEC-010 nftables writer + D-125 bridge verify. Bash, NOT tofu -- sits BETWEEN the roots.
      |
      `== INNER ROOT: opentofu/vr1-dcN-substrate/ (qemu+ssh into the containment VM, run
          from voffice1 per D-128; D-126 per-env key auth)
           |-- inner_storage pool (inside the containment VM's own disk image)
           |-- the SIX planes (dc-planes, isolated L2, MTU 9000) -- RELOCATED here by D-123
           |-- vr1_dcN_wan (wan-bridge onto br-vr1-dcN-wan) + vr1_dcN_opnsense (DC edge)
           `-- vr1_dcN_node x 12: 9 D-121 role nodes (3 control / 2 compute / 4 storage)
               + vr1-dcN-juju-01 (.5, D-104) + vr1-dcN-maas-01 (.6, D-132 addendum)
               + vr1-dcN-tailscale-01 (.7, D-129(iii)); all MAC-pinned

1.2 The two-stage apply ordering (structurally forced, not preferred)

OUTER apply (vcloud, Plane 1) -> BOOTSTRAP GATE (bash, over SSH) -> INNER apply (voffice1, Plane 2, qemu+ssh). Forced because "a libvirt provider cannot be configured from a resource created in the same apply" (opentofu/main.tf:28-29). Eliminating the containment VM removes the forcing function entirely: one root, one state axis, no gate script, no cross-host provider dial. This is the single largest structural simplification the elimination buys (W0.1 Sec 3).

1.3 What the "container layer" is, precisely

The vvr1-dcN VMs + their inner libvirtd + the inner roots + the bootstrap gate's node-host mode + the D-125 bridge-in plumbing (wan-bridge module, IP-less uplink NIC, br-vr1-dcN-wan)

  • the qemu+ssh provider dial and its D-126 keys. It is NOT: the six planes, the node VMs, the DC edge, the mesh triangle, the uplink NATs, or the LXD API-charm containers on deployed nodes (all of which persist; the first three re-home, W0.1 Sec 5 / W0.2 Sec 4).

1.4 Network facts the elimination must preserve (W0.2)

  • MTU/geneve budget: 9000 jumbo underlay -> tenant MTU 1500 with ~56-byte v6-geneve overhead. The containment hop was a same-MTU bridge with no extra encapsulation -- removing it changes no byte budget (W0.2 Sec 3). Do not let later phases imply an MTU benefit.
  • The 2026-08-08/09 geneve-over-v6 defects (encap-family split, bracketed encap-ip) were OVN/OVS-layer, NOT containment-layer -- container-elim neither caused nor fixes them.
  • SEC-010 (CLOSED) is an interface-scoped FORWARD-drop on the transit leg of vvr1-dcN and voffice1. Its ledger rationale is nesting-specific: vvr1-dcN "bridges ALL 6 inner planes + the transit" -- SEC-010 is doing the isolation a separate host would do for free.

2. Consolidated blast-radius inventory (W0.3 + W0.1/W0.2, deduplicated)

Census baseline: 424 vvr1-dc occurrences across 78 canonical files; ~26 in scripts, 17 tofu files; the bulk is decision-record prose (append-only -- superseding notes only, never edits).

# Consumer (file:line) Keys to the containment layer via On flattening Severity
1 opentofu/main.tf:410-519,537-623 (module vvr1_dc0/_dc1) + sizing vars variables.tf:137-156,175-194 + D-124 rack-addressing vars :196-244 + D-126 pubkey vars IS the containment VM + its sizing/addressing/keys DELETED. Sizing/rack-addressing/pubkey vars go with it; dc-dc-whole-host-budget.py containment-overhead flags re-derived or dropped HIGH
2 opentofu/vr1-dc0-substrate/, vr1-dc1-substrate/ (whole inner roots + states) qemu+ssh provider into vvr1-dcN RETIRED as roots; their module CALLS (planes, edge, 12 node VMs, storage) re-home into the outer/flat root -- bodies unchanged, provider changes HIGH
3 modules/wan-bridge + vr1_dcN_wan calls + the IP-less uplink NIC + br-vr1-dcN-wan netplan D-125 bridge-in (exists only to fix OBS-3 nesting egress) DELETED; DC edge WAN attaches directly to the outer vr1_dcN_uplink NAT (Model A item 8 -- same /24, no re-address) MEDIUM
4 scripts/lib-hosts.sh:52,157-168,212-214,246-251 (VIRSH_POWER_ADDRESS*) qemu+ssh power URIs dial the containment VM's libvirtd Re-derived to vcloud's own virsh; the FROM_OFFICE1/FROM_DCREGION split may collapse. Wrong values = "MAAS unreachable" masquerading as a network fault HIGH
5 scripts/maas-node-power.sh power address as ARGUMENT (topology-agnostic) No code change; every invocation site/runbook example needs the new address MEDIUM
6 scripts/site-headend-install.sh --host-nodes (~140 lines) + SEC-010 writer THE containment-VM bootstrap + SEC-010 enforcement point node-host mode = dead code for VR1; SEC-010's implementation must be REBUILT for the new boundary (Section 5), not moved. Rack-role remainder needs a re-homing decision (Section 6) HIGH
7 scripts/dc-rack-net.sh (rack bridge-leg IPs .2/.3 + D-131 node DNS forwarder) runs ON vvr1-dcN; addresses are the containment VM's identity on its own inner bridges Needs a NEW HOME (the Option-1 client VM is the natural candidate) or a ruled retirement -- D-131 is a DNS-availability control; silently losing it reintroduces the SERVFAIL bug HIGH
8 SEC-028 juju service key + SEC-029 Octavia PKI overlay, resident on "the rack"; vm-secret-locations rack rows; SEC-026 isolation control vvr1-dcN is a credential-bearing host (first rack rows in the register) Residencies MIGRATE to the replacement client host; matrix/register rows re-point; SEC-026 "only that DC's credential" must be re-asserted; rotation triggers ("if the rack is rebuilt") FIRE on this change HIGH
9 runbooks/dc-dc-teardown-rollback.md (Paths A/B/C/M) vvr1-dcN as the teardown unit; D-122's "site-down = one virsh destroy" Re-authored around a scripted group-destroy of the vr1-dcN-* domain set. The one-command site-down is a REAL D-123 regression to carry honestly; module design should re-earn it (tofu-module-scoped destroy) MEDIUM-HIGH
10 D-128/D-138 execution-host surface (runbooks/dc-dc-phase3:424,430, phase4:165-167; ssh -J voffice1 to the rack transit IP) vvr1-dcN is the concrete D-138 client host THE load-bearing decision -- answered by proposal in Section 4 (Option 1's client VM), ruled at the gate CRITICAL (gate item)
11 netbox/dc-rack-mgmt-import.py + its harness imports vvr1-dcN as a NetBox DCIM device NetBox-side decommission/repoint (system-of-record data migration, not just code) MEDIUM
12 scripts/site-baseleg.sh:40-48 DC rows deferred BECAUSE DCs nest behind qemu+ssh (deferral answered by D-138) premise disappears; new leg only if the target needs one -- currently a no-op LOW
13 scripts/dc-mirror.sh:6, maas-profile-assert.sh:39, maas-role-tags.sh:37, dc-egress-check.sh:63,72 doc-comments / usage examples naming vvr1-dcN or "the rack host" doc-currency only; logic is host-agnostic LOW
14 lib-hosts.sh:69-72 NIC_PLANE_ORDER/BREX_PARENT_NIC; :29-34 CARVE_AUX_HOSTS conventions realized by the inner root's MAC pinning Conventions carry forward; the FILE that encodes the pinning changes (row 2). MAC re-capture owed post-rebuild (moot-ish: 10.13 rebuilds the fleet anyway) LOW-MEDIUM
15 Non-consumers, named to prevent double-counting: scripts/lib-net.sh (0 hits -- IPAM facts change under D-143 only, a separate ruled axis); plane CIDRs/families (D-139); the 66+17 design-decision/CURRENT-STATE hits (append-only record) -- -- NONE

3. Adversarial-check results (worker claims corrected or confirmed)

All checks run against repo ground truth this session; a plausible finding was not accepted as a verified one.

  1. W0.3's "three ruled roles on vvr1-dcN" CONFLATES region and rack -- corrected. The D-132 AMENDMENT ADDENDUM (docs/design-decisions.md:7216-7243, RULED 2026-07-30, exact utterance "Dedicated region VM per DC at utility .6 (Recommended)") puts each DC's MAAS REGION (regiond + PostgreSQL) in its OWN VM -- vr1-dcN-maas-01, already one of the 12 inner-root node VMs. The region survives flattening as a flat sibling with no redesign. What vvr1-dcN actually carries: the MAAS rack controller (--role rack), the D-131 node-DNS forwarder, and the D-138 client + SEC-028/SEC-029 credential residencies. This RE-GRADES W0.3's CRITICAL: the execution-host question is real but narrower than "three roles orphaned," and it is now ANSWERED-BY-PROPOSAL (Option 1's client VM), pending the gate.
  2. "Model A fallback is a validated artifact" -- TRUE but NOT TURNKEY; do not overstate. Verified: docs/archive/model-a-fallback-plan.md exists; tag model-a-fallback resolves to 114d392, dated 2026-07-16. That PREDATES the D-132 amendment + addendum and D-138 (both 2026-07-30), D-139's IPv6 family rulings (07-31), D-143 (10.13), the three utility node VMs, and the lb-mgmt plane. Its own MAAS line -- "region on Office1 + rack" -- is contradicted by D-132 as now ruled. Its Section-4 revert procedure (git checkout model-a-fallback -- opentofu/) must NOT be run: it would resurrect a 10.12-era, 9-node, Office1-region substrate. The real build path is re-homing the CURRENT inner-root module calls (W0.1's map) into a flat root, with Model A as shape precedent -- "Model A PLUS the rulings Model A never had," freshly validated. One thing the archive DOES verify in Option 1's favor: Model A itself kept a small non-hypervisor vvr1-dc0 headend (4/8192/80, expose_nested_virt=false, metal-admin + transit legs) -- Option 1's client VM is exactly that shape, repurposed.
  3. D-138 citation CONFIRMED. docs/design-decisions.md:7118-7124: "the per-DC management bastion holds only its own DC's cloud credential" / "Each DC has a management entry point inside it; the NOC reaches that entry point." The RULING is "the cloud-facing client lives IN the DC" -- a principle, with vvr1-dcN only as the then-concrete host. Option 1's client VM (inside the DC's planes, holding one DC's credential) is consistent with D-138's principle; the concrete-host change still needs recording (amendment vs. new D -- Phase 4 frames it, Section 6).
  4. Cross-DC adjacency gap: W0.4 does NOT address it -- OPEN. Grep-confirmed: the only "cross-DC" text in pass0-w4-targets.md is the credential row of the tradeoff table. Both options put both DCs' plane bridges and node VMs on vcloud's single libvirtd. See Section 5.
  5. Worker contradictions logged: (a) W0.2's diagram says "416 GiB" for the containment VM -- STALE; variables.tf:143-150 shows 416->480 GiB raised 2026-08-01 (W0.1/W0.4 are correct). (b) MTU: W0.2 rules the budget analytically unaffected; W0.4 lists it OWED -- reconciled as: analytically unchanged (same underlay_mtu=9000 var; vcloud-level mesh/office1 networks already run 9000), with a live jumbo/geneve assert still owed post-build (Section 8). (c) W0.3 row 4's "MAAS region is SHARED" echo (from SEC-026's pre-amendment text) is superseded by D-132's per-DC regions -- SEC-026's regional-blast -radius note is historical context, not current topology.
  6. Option-2 counter-evidence found during verification: runbooks/dc-dc-phase4-juju-bundle-per-dc.md:167 -- tool-placement table row: "anything at all | never the vcloud jumphost ... It picks the transport only" (measured 2026-07-29: no juju/maas binaries on vcloud). Runbook doctrine, not a D-ruling -- but Option 2 would reverse it, on top of inverting SEC-026's isolation control.
  7. No inferred-value violations found in the four worker docs beyond the items above -- claims spot-checked (ssh -J runbook lines, lib-hosts power URIs, SEC ledger rows, D-132 addendum, sizing vars) all resolved to the cited lines.

4. Target-topology options + recommendation

Both options share the flat core: 12 node VMs + DC edge as vcloud-libvirt siblings; the six planes re-homed to vcloud level (same CIDRs/families/MTU -- IPAM identity untouched); D-125 bridge-in deleted (edge WAN -> direct NAT); mesh triangle unchanged; nesting depth 4 -> 2 (VR0-proven); one tofu root/state axis, no bootstrap gate, no qemu+ssh dial.

Option 1 (RECOMMENDED) -- flat nodes + a small per-DC client VM (Model A shape, D-132/D-138-updated)

A dedicated ~4/8192/80 non-hypervisor VM per DC (vr1-dcN-client), legs = metal-admin + transit, carrying: the D-138 client role, the SEC-028/SEC-029 credential residencies, and (pending Section 6) the D-131 forwarder / rack-controller remainder. It is a flat utility sibling -- no nested libvirt, no expose_nested_virt -- so the containment PATTERN is gone even though a small VM remains.

  • For: lowest-delta path (Model A precedent + current inner-root module bodies re-homed); preserves SEC-026 per-DC credential isolation exactly as today; is LITERALLY the D-138 Roosevelt bastion analog, rehearsed early (transfers to the pre-Roosevelt bare-metal test); keeps the phase-4 "never the vcloud jumphost" doctrine intact.
  • Against: SEC-010 must be re-authored + re-measured for the new VM (new NIC-naming trap); D-123's one-command site-down is lost (re-earn via module-scoped destroy); the cross-DC gap (Section 5) still needs a ruling; ~2 small VMs of overhead retained (~8 GiB each).
  • Capacity: directionally frees ~176 GiB host RAM (2 x 96 GiB containment overhead removed, minus 2 x 8 GiB client VMs); vCPU a wash. QUALITATIVE -- exact FIT owed (Section 8).

Option 2 (NOT recommended) -- fully flat: client + credentials on vcloud itself

Same flat core; no client VM; juju/openstack + both DCs' MAAS-admin-scoped keys land on the shared jumphost via a new host-side leg into metal-admin.

  • For: ~8 GiB/DC less overhead; one fewer VM class.
  • Against: inverts SEC-026's load-bearing isolation control (both DCs' credentials on the widest-blast-radius host); contradicts the phase-4 runbook's "never the vcloud jumphost" row; no Roosevelt analog -- throwaway work undone at the bare-metal test; net-new unprecedented surface (host-side veth/bridge into a MAAS plane on the live jumphost).

Recommendation: Option 1. Every verified axis -- risk delta, credential isolation, Roosevelt fidelity, runbook doctrine -- favors it; Option 2's only advantage is a marginal ~8 GiB/DC. The cross-DC gap (Section 5) applies EQUALLY to both options, so it is a gate condition on the flattening itself, not a tiebreaker. Framing per W0.4: "eliminate the container layer" is fully satisfied by Option 1 -- no VM is a hypervisor for another VM; the retained client VM is the same class as juju-01/maas-01. If the operator's intent is literally zero additional VMs, that is Option 2 with the above accepted knowingly.


5. THE CROSS-DC ADJACENCY GAP -- unresolved by W0.4, stated plainly

W0.2's finding (Sec 4, "the single largest wiring risk"): today dc0's and dc1's six plane bridges live on SEPARATE kernels (each containment VM's own libvirtd). Flattening puts BOTH DCs' plane bridges + node VMs co-resident on vcloud's ONE libvirtd/kernel for the first time in any deployed shape. SEC-010 -- interface-scoped to transit legs that no longer carry this function -- does NOT cover host-level cross-DC leakage (an accidental host address on a plane bridge, a forward rule, br_netfilter interactions). Neither W0.4 option addresses this; verified absent from the doc. Nuances carried honestly: the adjacency existed in Model A's committed design, but Model A was never deployed and SEC-010 was priced AFTER it against Model-B's separate-kernel shape -- so no ruled control covers the flattened adjacency. This is DISTINCT from rebuilding SEC-010's transit drop on the client VM (row 6) -- two controls.

Handling options for the gate (presented, not picked -- GA-R5):

  • (a) Accept co-residency + a new host-level control: a vcloud-level nftables/isolation artifact asserting no inter-plane/inter-DC forwarding, with a mechanical --check gate and its own SEC-NNN row (SEC-010's proven pattern, one layer up). Designed in Phase 1/2.
  • (b) A per-DC isolation mechanism in the target design itself (e.g., per-DC network namespaces or equivalent separation on vcloud) -- stronger boundary, more engineering, partially re-introduces the complexity being eliminated.
  • (c) Reject single-host flattening for both DCs (not what the operator's directive implies, listed for completeness).

6. UNRESOLVED at Phase 0 / needs operator or later-phase decision

  1. Target topology itself -- the gate (Section 7).
  2. Cross-DC isolation mechanism -- Section 5's (a)/(b)/(c); at minimum a Phase-1 design item + a new SEC row if flattening proceeds.
  3. Rack-controller remainder + D-131 forwarder + mirror home: does MAAS rackd co-locate onto vr1-dcN-maas-01 (region VM), onto the Option-1 client VM, or elsewhere? Ditto the D-131 node-DNS forwarder and the .4 artifact-service placement (dc-mirror.sh says "runs on the rack host" today). W0.3 flagged; no worker resolved it.
  4. The client VM's octet + naming: inherit the rack's .2 metal-admin identity or take a new utility-band octet (D-134 standing map is a cross-DC STANDARD -- needs a ruled octet); name should NOT be vvr1-dcN (avoid conflation with the eliminated containment class).
  5. Transit leg's surviving purpose + SEC-010 re-pin points: qemu+ssh dial disappears; operator ssh -J access and any Office1-originated flows remain -- which ends get the re-authored FORWARD-drop (client VM + voffice1?), and does the mesh triangle survive unchanged (W0.4 implicitly kept it; W0.2 flagged it as a decision)?
  6. Credential-residency migration plan (SEC-026/-028/-029 + vm-secret-locations rack rows + rotation obligations that FIRE on rack rebuild) -- Phase 1/2 work, listed so it is not lost.
  7. Teardown-primitive replacement: the module-scoped group-destroy that re-earns D-122's one-command site-down (Phase 4 module design).
  8. Decision-record form: D-123 amendment vs. new D-number for the container-elim (+ the D-138 concrete-host change and the D-125 bridge-in retirement ride along). Phase 4 FRAMES it; the operator rules it. NOT ruled at this gate.
  9. NetBox DCIM migration for the vvr1-dcN device records (row 11).

7. >>> OPERATOR GATE (Phase-0 hard stop -- SCOPE plan Section 4) <<<

Decision put to the operator:

  1. Target topology -- confirm one:
    • Option 1 (recommended): flat node VMs on vcloud libvirt + one small non-hypervisor per-DC client VM (metal-admin + transit legs) carrying the D-138 client role and the per-DC credential residencies.
    • Option 2: fully flat; client + both DCs' credentials on vcloud itself (weakens SEC-026 isolation; contradicts the phase-4 "never the vcloud jumphost" doctrine row; no Roosevelt analog).
  2. Cross-DC isolation gap (applies to EITHER option): choose handling -- (a) accept co-residency + a new host-level isolation control with a mechanical gate + SEC row (designed Phase 1); (b) require a per-DC isolation mechanism in the target design; or (c) reject single-host flattening.
  3. Role re-homing: confirm the MAAS region stays on vr1-dcN-maas-01 (per D-132 addendum -- no change needed), and rule where the rack-controller remainder + D-131 forwarder land (client VM / maas-01 / retire-with-evidence).

Items 4-9 of Section 6 are flagged as downstream (Phase 1-4) work, not gate blockers.


7a. PHASE-0 GATE OUTCOME (operator, 2026-08-09)

Recorded as a DIRECTIONAL PLANNING CONFIRMATION at the Phase-0 gate -- NOT a GA-R5 [ARCH] ruling and NOT a D-number (the formal container-elim ruling, D-123 amendment vs. new D, is framed in Phase 4 and ruled then, per SCOPE Section 7). The operator was shown the layer+separation diagram (artifact, Option-1 target) before confirming.

  • Target topology: OPTION 1 CONFIRMED -- flat node VMs on vcloud libvirt + one small non-hypervisor per-DC client VM (metal-admin + transit legs) carrying the D-138 client role and that DC's credential residencies. Phases 1-4 plan against this.
  • Cross-DC adjacency gap (Section 5): handling (a) CONFIRMED -- accept co-residency and design a new vcloud-level host isolation control (SEC-010's nftables pattern one layer up) with a mechanical --check gate and its own SEC-NNN row. This is now a Phase-1 DESIGN ITEM, not an open gate fork.
  • Role re-homing: the MAAS region stays on vr1-dcN-maas-01 (.6, D-132 addendum -- no change). The rack-controller remainder + D-131 forwarder + artifact-service placement (client VM vs. maas-01 vs. retire-with-evidence) is carried into Phase 1 as an open placement decision (Section 6 item 3), not resolved at this gate.

Feeds forward to Phase 1: plan against Option 1; treat (a) as a required design deliverable; keep the container-elim change-set DISTINGUISHABLE from D-143's 10.12->10.13 address change (two rulable axes riding one redeploy).

8. OWED live measurements (read-only; do not infer -- W0.4 Sec 5 + reconciliation)

  1. vcloud host-capacity currency: dc-dc-whole-host-budget.py:66-67's committed "MEASURED host budget" (256 vCPU / 1024 GiB / 10240 GiB) needs a fresh read-only measurement before any FIT verdict.
  2. Exact FIT for the flat 12-VM/DC roster: the calculator lacks flags for the 3 utility node classes (juju/maas/tailscale, added after authoring) -- extend or hand-total before any freed-capacity number enters the Phase-4 change-set. Until then "~176 GiB freed" is directional only.
  3. MTU/jumbo/geneve assert on vcloud-level planes post-build: analytically unchanged (Section 3.5), but the live gate (geneve-encap-assert.sh + a jumbo-path check) is owed once the planes exist at vcloud level.