diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md index 948e7ee..067e39e 100644 --- a/docs/dc-dc-deployment-workflow.md +++ b/docs/dc-dc-deployment-workflow.md @@ -150,28 +150,44 @@ | | | |---|---| | **Goal** | From Office1, script each DC's "physical" layer into existence. | -| **Build** | Per DC: OpenTofu creates node-VM libvirt domains (SHIM, Section 9), the DC's plane networks, and its OPNsense edge; registers the DC libvirt host to that DC's MAAS rack controller as a virsh VM-host. | -| **Gate** | MAAS rack controller per DC up; MAAS discovers the OpenTofu-created node VMs; edge routing + simulated ISP uplink up; netem parameters applied and measured. | -| **Owns** | D-103 (seam), D-100 (edges). | +| **Build** | Per DC, D-123 **Model B** (TWO OpenTofu roots + a bootstrap gate between): the OUTER root (vcloud) boots + sizes the containment VM `vvr1-dc0` (~416 GiB, `expose_nested_virt=true`) + the transit + the D-125 vcloud ISP uplink; a gated BOOTSTRAP (`site-headend-install.sh --host-nodes`) makes it a nested libvirt/KVM host AND the MAAS rack; the INNER root (qemu+ssh to `vvr1-dc0`, R-5) creates the 9 node-VM domains (3+2+4, R-3; SHIM, Section 9), the 6 plane networks, and the OPNsense edge INSIDE `vvr1-dc0`. `maas-vm-host` registers `vvr1-dc0`'s OWN (inner) virsh to the Office1 REGION. | +| **Gate** | `vvr1-dc0` up + sized + nested-KVM on (the rack IS `vvr1-dc0`, D-123); inner node VMs boot at depth-4 (D-114/D-123 boot gate); MAAS discovers them; edge routing + the D-125 per-DC ISP egress up (bridge-in isolation test passes); SEC-010 DC-LOCAL enforced; netem parameters applied and measured (D-100 gap #11, still unruled). | +| **Owns** | D-103 (seam, as AMENDED by D-123 Model B), D-100 (edges), D-121 (node layout, R-3 = 9/DC), D-122 (site/edge shape), D-123 (Model B site-down + two-root), D-124 (rack transit), D-125 (per-DC ISP egress). | | **Reuse vs new** | NEW (no OpenTofu/multi-rack precedent in VR0 DC0). Explicitly tagged shim: node-VM creation has no Roosevelt analog (physical racking + BMC enlistment replaces it there) -- do not treat as reusable production IaC. | -| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase2-tofu-dc-substrate.md`.** DC1-first; DC2 explicitly hard-gated (D-101 supernet unassigned) rather than run with placeholders. Walks every still-open decision (node sizing, MAAS zone/pool, `power_address`, config.xml tokens incl. boot-measured `WAN_IF`/`LAN_IF`, netem params) and stops at the corresponding step if unresolved rather than inventing values. States its own exit gate as "CONDITIONALLY MET AT BEST" pending Stage 2, node sizing, a real OPNsense boot measurement, and D-100's netem ruling. NOT YET EXECUTED. The plane/link/pool layer of `opentofu/` (see Stage 1) is reusable here too. `modules/node-vm` (blank disk, PXE-boot), `modules/opnsense-edge`, and `modules/maas-vm-host` (2026-07-09, registers the virsh host with MAAS via the official `canonical/maas` provider -- deliberately NOT `maas_vm_host_machine`, which composes new VMs and would fight `node-vm`) all now exist -- none instantiated (node sizing, OPNsense config.xml content, and a real MAAS zone/pool are all still pending decisions). `modules/netem-link` gives the netem mechanism for this stage's edge gate too. See `opentofu/README.md`. | +| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase2-tofu-dc-substrate.md`.** DC1-first; DC2 explicitly hard-gated (D-101 supernet unassigned) rather than run with placeholders. Originally walked a set of STOP-gate decisions; most are now RULED (see the rulings summary below): node sizing = D-121/R-3, edge/WAN shape = D-122, site-down/placement = D-123 Model B, rack transit = D-124, ISP egress = D-125, edge-config delivery = D-113(a2) REST (config.xml/ISO retired -- no more `config.xml` tokens). STILL open at execution: MAAS zone/pool + `power_address` (Stage-4 concerns), boot-measured `WAN_IF`/`LAN_IF` (real OPNsense boot), and netem params (D-100 gap #11). NOT YET EXECUTED. The plane/link/pool layer of `opentofu/` (see Stage 1) is reusable here too. `modules/node-vm` (blank disk, PXE-boot), `modules/opnsense-edge`, and `modules/maas-vm-host` (registers the virsh host with MAAS via the official `canonical/maas` provider -- deliberately NOT `maas_vm_host_machine`; under Model B it targets `vvr1-dc0`'s inner virsh -> Office1 region, DOCFIX-179-deferred) all now exist AND are instantiated across the two Model B roots (D-123): the outer root (`opentofu/`) + the inner root (`opentofu/vr1-dc0-substrate/`), plus the new `modules/wan-bridge` (D-125). Both roots `tofu validate`. Remaining un-instantiated is deploy-time (MAAS zone/pool, `power_address`) not authoring. `modules/netem-link` gives the netem mechanism for this stage's edge gate too. See `opentofu/README.md`. | -**State:** IN PROGRESS (branch `dc-dc-stage3-phase2-dc-substrate`). RUNBOOK WRITTEN, NOT YET EXECUTED. -**Two blocking rulings landed 2026-07-15** (from a walk of the runbook's decision register): -- **D-121 (ADOPTED IN PART)** -- node layout = **Option C** (3 control + 2 compute + 3 storage per DC, - 16 node VMs), ruled after a whole-host resource validation (256 vCPU / 1024 GiB / 10 TiB measured; - C fits at 222 / 790 GiB / 5.4 TiB, RAM binding). Ruled per-node sizing recorded in D-121. Also - adopts the 14-service HA scale-up (1->3) that this layout carries at Stage 5. OPEN sub-ruling: the - vault-HA backend fork. -- **D-122 (ADOPTED)** -- DC edge/site shape: Office1 pattern edge (2048/2/nano, REST-API config), a - dedicated per-site `vr1-dc0-wan` uplink (dark fiber is East-West only), site nested in its own - containment VM, per-site MAAS. This CLOSES the runbook's Ruling-1 (node sizing) and Ruling-2 (edge - LAN/WAN) STOP-gates. -Fix-forward IN FLIGHT: runbook reconciled to post-D-119 naming + Steps 4-5 rewritten to the D-113(a2) -REST-API bootstrap. Honest first-run scope remains **Steps 1-8**; Step 9 (MAAS registration) still -gates on the per-DC MAAS rack-controller source (a Stage-4 gap -- no OpenTofu module stands one up yet). -The stale `vr1-dc1` "supernet not assigned" gate is corrected (D-115 assigned `10.12.64.0/19`); vr1-dc1 -stays out of scope by sequencing. +**State:** IN PROGRESS (branch `dc-dc-stage3-phase2-dc-substrate`, pushed through `d34ec94`). RUNBOOK +WRITTEN + RESHAPED to Model B, NOT YET EXECUTED. Authoring for this stage is effectively COMPLETE; what +remains is operator-gated (HELD NetBox addressing) + deploy-time gates. + +**Rulings landed (through 2026-07-16; supersedes the earlier "two blocking rulings landed 2026-07-15" note):** +- **D-121 (ADOPTED) + R-3** -- node layout = **Option C** = 3 control + 2 compute + **4 storage = 9 + nodes/DC** (R-3 added the 4th storage node; the earlier "3 storage / 16 VMs" is superseded). Whole-host + fit re-validated under Model B: **FIT at 870/1024 GiB (154 GiB headroom)** via + `scripts/dc-dc-whole-host-budget.py`. Vault-HA fork RULED (**R-2**: v-a MySQL-backed; Raft infeasible on + the 1.8 charm, D-068/BUNDLEFIX-010). +- **D-122 (ADOPTED)** -- Office1-pattern edge (2048/2/nano, REST-API config), dedicated per-site ISP + uplink, site nested in its own containment VM, per-site MAAS. Its WAN-uplink REALIZATION is superseded by + **D-125** (bridge-in) and node PLACEMENT by **D-123**; D-122's intent stands. +- **D-123 (ADOPTED Model B)** -- DC nodes nest INSIDE `vvr1-dc0`; a single `virsh destroy vvr1-dc0` = + site-down. Forces the TWO-ROOT split (a libvirt provider can't target a same-apply VM). Overturns the + earlier inferred Model A (preserved as `docs/model-a-fallback-plan.md` + git tag `model-a-fallback`). +- **D-124 (ADOPTED)** -- rack transit addressing (office1<->dc0 mesh /30, Scheme A); the entry's rack + SIZING is VOID under Model B (`vvr1-dc0` now holds the fleet). +- **D-125 (ADOPTED)** -- per-DC ISP egress via bridge-in (resolves OBS-3's design gap; egress efficacy is a + deploy-time gate). + +Steps 4-5 are rewritten to the D-113(a2) REST-API bootstrap; naming is post-D-119. **First-run scope is +now the full outer -> bootstrap -> inner -> maas-register sequence** (the runbook's "D-123 MODEL B RESHAPE" +section). The per-DC MAAS rack source is NO LONGER a missing-module gap: `vvr1-dc0` IS the rack, stood up +by `site-headend-install.sh --host-nodes` (a bootstrap SCRIPT, not a tofu module) between the outer and +inner applies. The stale `vr1-dc1` "supernet not assigned" gate is corrected (D-115 assigned +`10.12.64.0/19`); vr1-dc1 stays out of scope by sequencing. + +**Blocking to EXECUTE (all operator-gated):** the HELD NetBox addressing -- rack transit/30 + rack IP +(D-124), `vr1_dc0_uplink_cidr` + OPNsense WAN (D-125) -- via office1-netbox (apex token); then the gated +applies; then the deploy-time gates (depth-4 nova boot D-114/D-123; D-125 foreign-MAC egress isolation +test; SEC-010 packet-drop). --- @@ -184,7 +200,7 @@ | **Gate** | Nodes deployed; six planes present per node with correct fabrics/VLANs; provider NIC raw; PXE (v4) working; per-DC mirror reachable; NTP from edge working. | | **Owns** | D-103, D-100 (br-ex), D-107 (mirror + NTP). | | **Reuse vs new** | PARTIAL reuse. `runbooks/phase-00-teardown-maas-reset.md` carries the MAAS commission/deploy + br-ex discipline this repo already validated for a single rack; the per-DC multi-rack registration is new, the deploy mechanics are precedent. | -| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase3-maas-enlist-deploy.md`.** Adapts `phase-00-teardown-maas-reset.md`'s validated single-rack commission/deploy/carve mechanics per DC, run once per `$DC` (dc1/dc2). Requires `lib_net_select_dc "$DC"` first (DOCFIX-151); explicitly documents that `lib_hosts_select_dc "$DC"` is EXPECTED to fail for both DCs until this stage's own real host enrollment populates `lib-hosts.sh` -- a deliberate chicken-and-egg, not a bug. Surfaces 7 real gaps in a named section (no OpenTofu module stands up a per-DC MAAS rack controller VM itself; `reenroll-hosts.sh`/`carve-host-interfaces.sh`/`phase-00-maas-standup.sh` are not `$DC`-parameterized; node count/sizing undecided; DC2 additionally blocked at the selector layer; no identified owner for the per-DC artifact mirror; MAAS status for VM-host-discovered domains unconfirmed; `HOST_TAG`/fabric-name literals are DC0-specific). NOT YET EXECUTED. | +| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase3-maas-enlist-deploy.md`.** Adapts `phase-00-teardown-maas-reset.md`'s validated single-rack commission/deploy/carve mechanics per DC, run once per `$DC` (dc1/dc2). Requires `lib_net_select_dc "$DC"` first (DOCFIX-151); explicitly documents that `lib_hosts_select_dc "$DC"` is EXPECTED to fail for both DCs until this stage's own real host enrollment populates `lib-hosts.sh` -- a deliberate chicken-and-egg, not a bug. Surfaces 7 real gaps in a named section (no OpenTofu module stands up a per-DC MAAS rack controller VM itself; `reenroll-hosts.sh`/`carve-host-interfaces.sh`/`phase-00-maas-standup.sh` are not `$DC`-parameterized; node count/sizing now RULED (D-121/R-3 = 9/DC), so that sub-gap is closed; DC2 additionally blocked at the selector layer; no identified owner for the per-DC artifact mirror; MAAS status for VM-host-discovered domains unconfirmed; `HOST_TAG`/fabric-name literals are DC0-specific). NOT YET EXECUTED. | **State:** RUNBOOK WRITTEN, NOT YET EXECUTED. @@ -301,10 +317,15 @@ > `docs/changelog-20260713-opnsense-api-proven.md` and > `docs/changelog-20260713-opnsense-api-write-proven.md`. > -> **OPEN (D-113(a2), not yet done):** reduce `opentofu/templates/opnsense-config.xml.tmpl` to a -> MINIMAL bootstrap (sshd + root key + console), retire the config-ISO path and the module's -> `config_seed`/cdrom wiring, and rewrite Stage 3's edge steps around the API. Until that lands, -> treat every OPNsense step in this doc and in `runbooks/dc-dc-phase1/phase2` as reference-only. +> **DONE (D-113(a2)) -- reconciled 2026-07-16; the repo won over this register's stale "OPEN".** The +> config-ISO path and the module's `config_seed`/cdrom wiring are RETIRED: `modules/opnsense-edge` no +> longer takes `config_iso_path` (verified -- no `config_seed`/`config_iso_path` anywhere in the module), +> `opentofu/templates/opnsense-config.xml.tmpl` is DELETED, and Stage 3's edge steps (phase-2 Steps 4-5) +> are REWRITTEN around the REST API (`scripts/opnsense-api.sh`, harnessed). So the OPNsense steps in the +> phase-1/phase-2 RUNBOOKS are the API path and are NOT reference-only. **Caveat:** THIS document's own +> prose below (and other gap items) still describe the old `config.xml`/ISO delivery in places -- treat +> **this workflow doc** as the stale surface and the runbooks + module as authoritative until a full +> gap-register reconciliation lands. Confirmed by grepping the actual repo (scripts/, tests/, netbox/, runbooks/), not inferred from the design doc alone. Ordered roughly by "you'll hit this first," @@ -814,7 +835,7 @@ | **D-114 site containment VM (`voffice1`)** | **BUILT + RUNNING 2026-07-13.** `module "voffice1"` (`modules/cloudinit-vm` off `modules/base-image`, Ubuntu 24.04 noble) is instantiated and applied; the domain is up on `office1-local`, DHCP from the edge's Kea, `expose_nested_virt = true`. **Remaining:** MAAS-region + LXD (5.21 LTS track) ON it, the LXD-KVM-host registration back into MAAS, and the MAAS-COMPOSED service machines (NetBox/GitBucket/Tailscale). The first LXD VM that boots inside it is the **L3 nesting proof that gates DC1**. | | `opentofu/` (networks/pools/node-VM PXE/cloud-init-VM patterns) | **VALIDATED 2026-07-13** (OpenTofu v1.12.3 now on the jumphost): root + **10/10 modules** pass `scripts/opentofu-validate.sh`. **APPLIED for real** -- state holds the DC1 planes, pools, 3 mesh links, `office1-network`, the OPNsense edge, and (2026-07-13, D-114) the `ubuntu_noble_base` image + the **`voffice1` containment VM**. **`modules/cloudinit-vm` is now instantiated for real** and gained `expose_nested_virt`. **DOCFIX-194 fixed 2 modules that had NEVER been parsed** (`node-vm`, `netem-link`) because root does not call them and root-only validation skips uncalled modules. `tofu plan`/`apply` for the Stage 3 DC substrate is still UNEXERCISED. | | OPNsense image+config mechanism (`modules/opnsense-edge`) | **BUILT + INSTANTIATED + RUNNING 2026-07-12/13.** `office1-opnsense` is UP: routing, NAT, egress 0.0% loss, serial console, SSH-key managed, **Kea DHCP serving on udp/67**. Four boot bugs closed (DOCFIX-188/190/191/192) + DHCP added (DOCFIX-193). The module's `config_seed`/cdrom wiring is now DEAD WEIGHT -- see the config.xml row. | -| OPNsense config.xml design (`templates/opnsense-config.xml.tmpl`) | **SUPERSEDED 2026-07-13 by D-113(a2): edge config is done over the REST API** (`scripts/opnsense-api.sh`), proven end-to-end (read AND write) against the live edge. **DANGER: a full rendered config.xml push would CLOBBER the now-API-managed DHCP** -- do not run the old scp/install/reboot path. The config-ISO path is INERT (D-112: the Importer can never fire on a nano image). **OPEN:** reduce the template to a MINIMAL bootstrap (sshd + root key + console + a pre-seeded API key) and retire the ISO path + the module's `config_seed`/cdrom wiring. | +| OPNsense config.xml design (`templates/opnsense-config.xml.tmpl`) | **SUPERSEDED 2026-07-13 by D-113(a2): edge config is done over the REST API** (`scripts/opnsense-api.sh`), proven end-to-end (read AND write) against the live edge. **DANGER: a full rendered config.xml push would CLOBBER the now-API-managed DHCP** -- do not run the old scp/install/reboot path. The config-ISO path is INERT (D-112: the Importer can never fire on a nano image). **DONE (reconciled 2026-07-16):** `templates/opnsense-config.xml.tmpl` is DELETED and the module's `config_seed`/cdrom + `config_iso_path` are RETIRED; edge config is the REST-API bootstrap. (Boot-measured `WAN_IF`/`LAN_IF` are still applied over the API at deploy time -- a deploy-time step, not an authoring gap.) | | MAAS VM-host registration (`modules/maas-vm-host`) | BUILT 2026-07-09, UNVALIDATED -- needs a real MAAS zone/pool + vcloud power_address, see `opentofu/README.md` | | `tc netem` mechanism (`modules/netem-link`) | **WAS BROKEN, FIXED 2026-07-13 (DOCFIX-194)** -- its destroy provisioner referenced `var.*`, which OpenTofu rejects at INIT, so the module could not even initialize. Now validates. Real latency/jitter/loss/rate parameters are STILL an unruled D-100 item. Never applied. | | `opentofu/` (DC2 planes) | NOT DONE -- deliberately deferred pending NetBox CIDR assignment, see `opentofu/README.md` |