Newer
Older
openstack-caracal-dc-dc / docs / model-a-fallback-plan.md

Model A fallback + revert plan (D-123)

Purpose. The operator ruled Model B for D-123 (nodes nested inside vvr1-dc0, single-object virsh destroy site-down) -- the heavier, higher-risk path (depth-4 nested virt, supersedes D-103/D-114, ~416 GiB containment VM). This document preserves Model A as a fully-specified, already-implemented fallback so that if Model B fails to deploy, we revert WITHOUT re-engineering.

Revert anchor (git). Model A is not theoretical -- it is the CURRENTLY COMMITTED substrate. The last commit before any Model B reshape is tagged model-a-fallback (re-cut 2026-07-16 onto 114d392, the R-3-compliant Model A layout with 4 storage nodes/DC -- the prior anchor at 87a7a8a was R-3-stale, flagged by the Model B design cross-check). To restore Model A: git checkout model-a-fallback -- opentofu/ (or cherry-pick the substrate files), then re-run bash scripts/opentofu-validate.sh. No file needs to be re-authored -- Model A already validates (tofu validate Success; 11/11 modules).


1. Model A architecture (the as-built shape)

Nodes are vcloud-level libvirt siblings of the headend -- NOT nested inside it. This is the VR0-proven shape and the ADOPTED D-103/D-114 seam.

vcloud (host, L0)
|-- vvr1-dc0            MAAS rack headend  (cloudinit-vm; D-124: 4 vCPU / 8192 MiB / 80 GiB;
|                        expose_nested_virt = false; legs = metal-admin + office1<->dc0 transit)
|-- vr1-dc0-control-01..03   node VMs (16/65536/150)  \
|-- vr1-dc0-compute-01..02   node VMs (12/49152/100)   >  vcloud-level siblings, on vr1_dc0_planes
|-- vr1-dc0-storage-01..04   node VMs (8/24576/550)   /   (4 storage/DC per R-3)
|-- vr1-dc0-edge             opnsense (2/2048, 2-NIC: provider-public LAN + vr1-dc0-wan WAN)
|-- vr1-dc0-* planes         6 isolated-L2 libvirt networks (dc-planes) at vcloud level
|-- vr1-dc0-wan              NAT /24 simulated ISP uplink (site-wan)
`-- mesh-vr1-dc0-office1      transit leg (office1 <-> dc0)

nesting depth = 2  (vcloud -> node VM -> nova KVM guest)   <- VR0-PROVEN
site-down       = destroy the vr1-dc0-* domain GROUP (scripted, gated)
MAAS model      = region on Office1 + rack (vvr1-dc0); maas-vm-host registers VCLOUD's virsh so
                  MAAS discovers the OpenTofu-created node domains (D-103/D-114 as-built)

2. The committed artifacts that embody Model A (revert targets)

Artifact Model A content
opentofu/main.tf module "vr1_dc0_node" for_each = local.vr1_dc0_nodes; created on the vcloud libvirt provider; attached to module.vr1_dc0_planes outputs (6 NICs, metal-admin first = PXE).
opentofu/main.tf module "vvr1_dc0" cloudinit-vm, 4/8192/80 (D-124), expose_nested_virt = false, two legs (metal-admin + mesh transit). A small rack headend that holds NO nodes.
opentofu/main.tf module "vr1_dc0_planes" / mesh_* / vr1_dc0_wan all created at vcloud level.
Step-9 maas-vm-host (deferred, DOCFIX-179) registers vcloud's virsh to the DC's MAAS -> MAAS discovers the vcloud-level node domains.
scripts/site-headend-install.sh --role rack installs the rack controller on vvr1-dc0 (no LXD/compose in rack mode).
Site-down a scripted group-destroy of the vr1-dc0-* domains (owned by dc-dc-teardown-rollback.md); NOT a single virsh destroy.

3. What Model B changes vs Model A (the delta to undo on revert)

Reverting = undoing exactly these; nothing else moves.

  1. Node placement: B retargets module "vr1_dc0_node" (and the 6 planes + vr1-dc0-wan) from vcloud's libvirt to vvr1-dc0's inner libvirt. A restores them to vcloud level.
  2. Headend sizing: B resizes vvr1-dc0 from 4/8192/80 to ~416 GiB (must hold one DC's full node fleet) and sets expose_nested_virt = true. A restores D-124's 4/8192/80.
  3. maas-vm-host target: B registers vvr1-dc0's inner virsh; A registers vcloud's virsh.
  4. Governance: B supersedes D-103/D-114; A keeps them ADOPTED as-is. On revert, the D-103/D-114 supersession is withdrawn.
  5. Nesting depth: B = 4 (unproven); A = 2 (VR0-proven).
  6. Site-down primitive: B = one virsh destroy vvr1-dc0; A = scripted group-destroy.
  7. Inner root: B adds opentofu/vr1-dc0-substrate/ (a new root dir + its own state) and a site-headend-install.sh --host-nodes node-host bootstrap; A has neither. Revert removes them.
  8. Per-DC ISP egress (D-125 bridge-in): because B nests vr1-dc0-wan inside vvr1-dc0, B adds a vcloud-level ISP NAT (module "vr1_dc0_uplink", site-wan, cidr 172.30.2.0/24), a 2nd IP-less uplink NIC + the br-vr1-dc0-wan netplan bridge on vvr1-dc0, the new modules/wan-bridge, and the bootstrap's --uplink-if/--wan-bridge verify. The ADDRESSING is identical to Model A -- same 172.30.2.0/24 (D-115), OPNsense WAN still .2; only the libvirt realization differs (bridge through vvr1-dc0 vs a direct vcloud-level NAT). In Model A the extra plumbing does not exist -- vr1-dc0-wan is a vcloud-level NAT the vcloud-level edge attaches to directly, no uplink NIC, no bridge, no wan-bridge module. Revert removes that plumbing; NO re-address is needed (the address never changed). (No HELD gate here: the /24 is a ruled literal, not a tfvar.)

4. Revert procedure (if Model B deployment fails)

  1. STOP -- do not attempt to fix Model B in place if nested-virt (depth-4) is the failure mode; that is the known risk this fallback exists for.
  2. git checkout model-a-fallback -- opentofu/main.tf opentofu/variables.tf opentofu/modules/ (restores the Model A substrate verbatim -- this also drops module "vr1_dc0_uplink", since Model A has no vcloud uplink), then git rm -r opentofu/vr1-dc0-substrate (the inner root does not exist in Model A) and git rm -r opentofu/modules/wan-bridge (D-125, also absent in Model A), and revert the site-headend-install.sh node-host mode (incl. the D-125 --uplink-if/--wan-bridge WAN-bridge verify). The OPNsense WAN address is UNCHANGED (.2 on 172.30.2.0/24) -- nothing to restore.
  3. bash scripts/opentofu-validate.sh -> expect 11/11 PASS (Model A already validates).
  4. Re-instate D-103/D-114 as ADOPTED (they were only annotated superseded, not deleted -- see the sweep's supersession notes; revert removes those annotations).
  5. Restore D-124's rack sizing (4/8192/80) in tfvars/main.tf.
  6. Adopt the scripted group-destroy site-down (the vr1-dc0-* group op) in place of the single-object destroy.
  7. Re-run the Layer-1 gate (repo-lint, run-tests-all, tofu validate) before proceeding.

5. Failure signals that should trigger the revert

  • nova-compute guests fail to boot or are unusably slow at 3x-nested KVM (the depth-4 risk).
  • vvr1-dc0 cannot be allocated ~416 GiB on the host alongside the other layers.
  • MAAS enrolment/commissioning breaks because the node domains are no longer vcloud-visible.
  • expose_nested_virt = true on vvr1-dc0 destabilises the headend/rack.

Model A remains the recommended engineering choice on delta/risk grounds; Model B is the operator-ruled choice for its single-object site-down primitive. This plan makes the choice reversible at low cost. Kept in sync with the D-123 sweep; if Model B changes, update section 3.