diff --git a/docs/changelog-20260715-d121-d122-stage3-ha-nodelayout.md b/docs/changelog-20260715-d121-d122-stage3-ha-nodelayout.md new file mode 100644 index 0000000..9a08253 --- /dev/null +++ b/docs/changelog-20260715-d121-d122-stage3-ha-nodelayout.md @@ -0,0 +1,116 @@ +# 2026-07-15 -- Stage 3 kickoff: D-121 (HA scale-up + Option C node layout) + D-122 (site shape); phase2 runbook fix-forward + +## Context + +Opened Stage 3 (VR1 DC substrate) on branch `dc-dc-stage3-phase2-dc-substrate`. Walked +`runbooks/dc-dc-phase2-tofu-dc-substrate.md` to surface its blocking decisions before any +`tofu apply`. The walk (plus a bundle.yaml HA review and a whole-host resource validation) +produced two operator rulings and a runbook fix-forward. No cloud mutation this session -- +design decisions, one resource calculation, and doc reconciliation only. + +## What changed + +**D-121 (ADOPTED IN PART) -- VR1 makes HA real + the node layout that carries it.** +`docs/design-decisions.md`. The bundle review found the root fact: VR0 ran its control plane +**single-unit by design** (D-009: hacluster + VIP relations at `num_units: 1`, "decorative... +scale-up is mechanical: 1 -> 3"). The 12 hacluster subordinates and D-036's dual VIPs are already +wired; the units were never added. D-121 adopts the **14-service 1->3 scale-up** (keystone, glance, +neutron-api, nova-cloud-controller, placement, cinder, openstack-dashboard, octavia, barbican, +magnum, designate, ceph-radosgw + rabbitmq + vault; `ceph-rbd-mirror` stays 1 per D-108). Two +realities recorded so the cost is not under-stated: (1) it is a **placement rewrite** (each app's +`to:` must spread across 3 distinct hosts -- co-located units are decorative again; a **Stage-5 +placement-balance rule** was folded in from the openstack0-3 balance analysis: spread each app's 3 +units one-per-control-node so the RAM-heavy trio auto-balances, and distribute the ~42 API containers +evenly ~14/control-node, sized for at 64 GiB); (2) **vault** +is MySQL-backed today (C1 revised D-006; `vault:ha` removed by BUNDLEFIX-002), so its HA path was a +sub-ruling -- **RULED (v-a)** 2026-07-15: keep MySQL-backed, scale to 3, re-add `vault:ha`, gated on +verifying the `1.8/stable` charm does MySQL-backend HA with 3 units (else evaluate Raft via D-068, NOT +etcd). 3-unit vault triples the unseal burden, intersecting OPEN SEC-003. +**Node layout = Option C (operator-ruled)**: 3 control + 2 compute + 3 storage per DC (16 node VMs). +Ruled sizing: control 16 vCPU/64 GiB/150 GiB, compute 12/48/100, storage 8/24/550. `ceph-osd`=3/DC, +`nova-compute`=2/DC, `ceph-mon`/`mysql`/`rabbitmq`/`vault`/`ovn-central`=3 across the control nodes. + +**Whole-host resource validation (Option C).** Measured the actual VR1 host (not the design +estimate): **256 vCPU / 1024 GiB RAM / 10 TiB disk** (`nproc`/`free`/`df`; currently only Office1 +runs -- `voffice1` 16/32/600 + `office1-opnsense` 2/2/11, netbox+tailscale nested inside voffice1). +Modeled Option C across all layers (Office1 as-built + both DCs + edges + containment overhead) at +`scratchpad/optc-calc.py`: **222 vCPU / 790 GiB / 5.4 TiB -> FITS** (RAM binding at 77%; vCPU +overcommits; disk thin-provisioned at 53%). Finding: C is NOT heavier than B -- it is RAM-lighter +(790 vs 838 GiB) because role separation lets storage nodes run lean. Sensitivity: control-node RAM +is the knob (96 GiB -> 982 GiB / 96%); held <= 64 GiB. This corrected D-121's first-pass "C is +heaviest" characterization. Also ran `scripts/dc-dc-ceph-disk-budget.sh` (Option B/C 4+4 and 3+2+3): +PASS with wide margin -- disk is not the binding constraint. + +**D-122 (ADOPTED) -- VR1 site shape.** `docs/design-decisions.md`. Records the operator's Ruling 2: +each site nested in its own containment VM (site-down/recovery + site-to-site drills); dark fiber +DC-DC-Office is East-West ONLY (never an internet path); each site has its own dedicated L3 "ISP" +uplink (build `vr1-dc0-wan` on the office1-wan pattern -- NOT a mesh leg); the DC OPNsense edge +follows the proven Office1 pattern (2048 MiB / 2 vCPU / nano, REST-API config per D-113(a2)); each +site runs its own MAAS controller (may bootstrap from an existing one). Closes the phase2 runbook's +Ruling-1/Ruling-2 STOP-gates. Subsumes the DOCFIX-185 amendment note. +**Edge NIC model ruled (D-122 refinement, operator 2026-07-15): 2-NIC, baremetal-matched.** NIC count +mirrors the Roosevelt hardware -- nodes = 6 NICs (one per plane, measured on VR0 openstack0-3), edge = 2 +(WAN + LAN). The six planes are routed by the FABRIC (OVN/OpenStack on the deliberately isolated-L2 +`dc-planes` segments), so the edge is only the external boundary, not an inter-plane router -- no +`opnsense-edge` module extension. Edge WAN = `vr1-dc0-wan` (`172.30.2.0/24`, D-115); edge LAN = +**provider-public** (`10.12.4.0/22`; external gateway per D-100). This resolves the runbook's earlier +"LAN role / build a site-internal segment" flag: no new segment is needed. Two build prerequisites +remain: build `vr1-dc0-wan` as a tofu module (promote off the virsh `office1-wan` one-off), and register +`172.30.2.0/24` in office1-netbox (D-115-ruled, not yet loaded -- absent from the draft; no import +covered the DC Edge /24s). + +**phase2 runbook fix-forward.** `runbooks/dc-dc-phase2-tofu-dc-substrate.md`: +- Region-qualified to the D-119 namespace -- every `dc1`/`dc2` object renamed to `vr1-dc0`/`vr1-dc1` + (matching `main.tf`); added a naming note + a HAZARD line that any residual bare `dcN` reintroduces + the exact cross-cloud collision D-119 deletes. Revert: `git checkout` the file. +- Steps 4-5 rewritten to the D-113(a2) REST-API edge bootstrap -- dropped the config.xml render + the + config-ISO path (D-112: the Importer cannot fire on a nano image; the module already dropped + `config_iso_path`); edge is now prep-image + post-boot console/SSH/API. WAN_IF/LAN_IF agonizing + removed (measured post-boot). +- D-122 edge shape wired into Step 5: `memory_mib=2048`/`vcpu=2`/nano, no `disk_size_bytes`, + `wan_network_name = vr1-dc0-wan` (dedicated per-site ISP uplink, office1-wan pattern), + `lan_network_name` = site-internal network -- both flagged as genuine build prerequisites. +- D-121 Option C wired into Step 6: 8 `module "vr1_dc0_node_*"` blocks in three role-sizes (3 control + 16 vCPU/65536 MiB/150 GiB, 2 compute 12/49152/100 GiB, 3 storage 8/24576/550 GiB; disk in bytes via + `N * 1024*1024*1024`), with the Stage-5 HA-scale-up context. +- Corrected the `vr1-dc1` gate rationale (top callout + exit GATE + checklist): its supernet IS + assigned (D-115 `10.12.64.0/19`, imported to office1-netbox); it stays out of scope by SEQUENCING, + not a missing literal. No `vr1-dc1` build steps added. +- Sequence block, exit GATE, Step-3 register, and delivery checklist updated to match; open items kept + as flagged STOPs (per-DC MAAS rack controller = Stage-4 gap, honest first-run scope Steps 1-8; the + vault-HA backend sub-ruling; netem params). Revert: `git checkout runbooks/dc-dc-phase2-tofu-dc-substrate.md`. + +**Two gaps surfaced by the fix-forward (LOGGED, not fixed -- they need real values/build work):** +1. `opentofu/main.tf`'s commented `vr1_dc1_planes` block (~L38-50) still says "wait for NetBox to + assign D-101's supernet" -- STALE per D-115. Left in place (this was a doc-only pass); flagged for + the `main.tf` owner. +2. **Edge network prerequisites (LAN side RESOLVED after the operator's fabric-routing ruling).** The + fix-forward first flagged "no site-internal LAN segment for `vr1-dc0`"; the operator then ruled the + planes are **fabric-routed** and the edge is **2-NIC**, so the **LAN = provider-public** (a + `vr1_dc0_planes` output) -- NO new site-internal segment needed. Two REAL prerequisites remain before + Step 5's `tofu apply`: build **`vr1-dc0-wan`** as a tofu module (promote off the virsh `office1-wan` + one-off), and register **`172.30.2.0/24`** in office1-netbox (D-115-ruled, not yet loaded). Recorded + in the runbook, not invented. + +**Ledger + tracker reconciliation.** `docs/session-ledger.md` (next-free D 121->123; a Stage-3 +open-work block with the two rulings, the validation, the fix-forward-in-flight, the stale-gate +correction, and the logged-not-executed calc-promotion follow-up) and +`docs/dc-dc-deployment-workflow.md` (Stage 3 State row updated from "runbook written" to +"IN PROGRESS + two rulings landed"). + +## Tests / verification + +- `bash scripts/repo-lint.sh` -> 0 fail, 1 legacy WARN (the documented D-001..018 non-ASCII + carve-out; new D-121/D-122 text is ASCII). +- `bash scripts/ledger-scan.sh` -> next-free D = 123; D-121/D-122 not flagged PROPOSED/OPEN. +- Resource model reproducible: `python3 scratchpad/optc-calc.py`. + +## Revert + +- D-121 / D-122: delete the two `## D-121:` / `## D-122:` sections from `docs/design-decisions.md` + (they are append-only additions; nothing else references them yet in code). +- Ledger / workflow: revert the `docs/session-ledger.md` next-free line + the Stage-3 block, and the + `docs/dc-dc-deployment-workflow.md` Stage 3 State paragraph, to their prior text. +- Runbook: `git checkout runbooks/dc-dc-phase2-tofu-dc-substrate.md` (see the runbook section for the + per-change reverts). +- No cloud state was touched, so there is nothing to unwind on the host. diff --git a/docs/changelog-20260715-stage3-prereqs-prep.md b/docs/changelog-20260715-stage3-prereqs-prep.md new file mode 100644 index 0000000..df0c9a6 --- /dev/null +++ b/docs/changelog-20260715-stage3-prereqs-prep.md @@ -0,0 +1,88 @@ +# 2026-07-15 -- Stage 3 batch-deploy prep: OpenTofu wiring, DC-edge NetBox importer, HA overlay, D-123 + +## Context + +Operator directed "get as much ready as possible so we can batch deploy with the least +operator interaction." This batch prepares the Stage-3 substrate + Stage-5 HA artifacts so the +live steps reduce to review-plan-and-apply. NO cloud mutation this session -- all authoring + +offline validation; every `tofu apply`, netbox `--commit`, and juju deploy stays operator-gated. +Fanned out across three disjoint areas (opentofu / netbox / overlays); the opentofu agent hit a +transient API stall and its stream was completed in-session by the main loop. + +## What changed + +**D-123 (PROPOSED) -- DC site-down DR model / node-VM placement.** `docs/design-decisions.md`. +Surfaced a real tension: D-122 records the operator's "each site is one VM, `virsh destroy` = +site-down" intent, but the ADOPTED D-103/D-114 seam + the built tooling create the OpenStack node +VMs as **vcloud-level** domains that MAAS merely DISCOVERS (containment VM scoped to MAAS/LXD + +non-stack machines only). Model A (recommended): nodes stay vcloud-level; `vvr1-dc0` is a MAAS/LXD +headend VM (the proven `voffice1` pattern via `site-headend-install.sh`, which already names it a +target); site-down = destroy the `vr1-dc0-*` domain GROUP. Model B: true single-VM containment +(nodes nested), 4-level nesting, heaviest. **Needs a one-line operator ruling.** + +**OpenTofu Stage-3 wiring (`opentofu/`).** +- **New module `modules/site-wan/`** -- a NAT'd per-site simulated-ISP uplink /24 (generalizes the + virsh-created `office1-wan`; operator 2b ruling). MTU 1500 (ISP-uplink domain, NOT jumbo). Schema + CONFIRMED against the installed provider via `tofu providers schema -json` (the method dc-planes + calls for): `forward = { mode = "nat" }`, `ips = [{ address, prefix }]`, `mtu = { size }`, + attribute-style like dc-planes. DHCP omitted (the edge WAN is static, like office1-opnsense .2). + Per-module `versions.tf` (DOCFIX-179). +- **`main.tf` Stage-3 section** (NOT YET APPLIED, gated): `module "vr1_dc0_wan"` (172.30.2.0/24, + D-115); `module "vr1_dc0_opnsense"` (D-122: 2-NIC, `memory_mib=2048`/`vcpu=2`/nano, no + `disk_size_bytes`/`config_iso_path`, LAN = provider-public plane, WAN = the uplink); 8 node VMs via + a single `for_each` over a `locals` map in D-121 Option C sizes (3 control 16/65536/150GiB, 2 + compute 12/49152/100GiB, 3 storage 8/24576/550GiB), six NICs each with **metal-admin first (PXE)**; + netem HELD as a comment (Step 11, runtime SSH target + unruled params; the vr1-dc0<->office1 mesh + bridge is `virbr7`, measured). **HELD pending D-123:** the `vvr1-dc0` headend + `maas-vm-host` (Step + 9) blocks are not wired. +- Fixed the stale `vr1_dc1_planes` comment (supernet IS assigned, D-115; out of scope by sequencing). +- **`tofu validate` -> Success**; `scripts/opentofu-validate.sh` -> 11/11 modules PASS (incl. site-wan). + +**NetBox DC-edge WAN importer (`netbox/dc-edge-wan-import.py` + `tests/dc-edge-wan-import/`).** +Registers the D-115 DC-edge /24s in office1-netbox: `172.30.2.0/24` -> site `vr1-dc0`, +`172.30.3.0/24` -> site `vr1-dc1`, role `edge`, `scope_type=dcim.site` -- scoping CONFIRMED against +`d115-office-carve.py`/`sandbox-fidelity-check.py` (how office1-wan 172.30.1.0/24 is bound), not the +frozen draft (which predates D-115). Stdlib-only, mirrors `d120-compose-bands.py`: WHOLE-PLAN +PREFLIGHT (both sites resolve before any create, so a missing site cannot half-write), DRY-BY-DEFAULT, +`SANDBOX_HOSTS` guard + `--yes-write-upstream` refusal, WAF-safe UA. Creates ONLY the two /24s (role + +container + sites are read-only preconditions). Harness: **58/58 PASS** (19 static + 39 behavioral, +in-process against a fake NetBox). NOT run against a live NetBox (token operator-held). + +**Bundle HA scale-up overlay (`overlays/dc-ha-scaleup.yaml`).** VR1-specific overlay (NOT a bundle.yaml +edit -- VR0 stays single-unit per D-009) implementing D-121: 14 services -> `num_units: 3`, one-per- +control-node placement via a 3-entry `to:` with `{{VR1_DC0_CONTROL_0N}}` tokens (Stage-4-binding +banner). Three correctness items beyond the literal bump, each flagged in-file: (A) rabbitmq +`min-cluster-size: 3` (D-009 amendment -- avoids the D-062-class formation race); (B) `cluster_count: +3` on the 12 haclusters + vault-hacluster (base pins 1 = decorative); (C) **re-declares +`vault-hacluster`** (it is COMMENTED OUT by BUNDLEFIX-002) and re-adds `[vault:ha, vault-hacluster:ha]` +(D-121 v-a keeps vault MySQL-backed at 3). YAML valid; app names cross-checked vs bundle.yaml; +repo-lint clean; `provider-bundle-check.py` green on the base. + +## Tests / verification + +- `cd opentofu && tofu validate` -> Success; `bash scripts/opentofu-validate.sh` -> 11/11 PASS. +- `bash tests/dc-edge-wan-import/run-tests.sh` -> 58/58 PASS. +- `bash scripts/repo-lint.sh` -> 0 fail, 1 legacy WARN. +- `bash scripts/run-tests-all.sh` -> GAUNTLET: ALL GREEN (61 harnesses). + +## What remains operator-gated / to bind later (the residual for a smooth batch) + +1. **D-123 ruling** (Model A/B) -- unblocks wiring the `vvr1-dc0` headend + `maas-vm-host` (Step 9). +2. **Runtime prep before the edge apply:** `scripts/opnsense-prep-image.sh` to produce the nano at the + `base_volume_path`; the netbox `--commit` to register the two /24s (ready command in the importer's + own header; token operator-held). +3. **Stage-4 binding for the HA overlay:** render `{{VR1_DC0_CONTROL_0N}}` to the measured MAAS machine + IDs; the base `machines:` block + the already-3 services (mysql/ovn/ceph-mon) + memcached + + nova-compute(->2)/ceph-osd(->3) re-bind to the Option C nodes; confirm `--overlay ... --dry-run` + (the `to:` list-replace merge is the one unverified overlay mechanism). +4. **D-121 verifies:** vault charm MySQL-backend HA at 3 units (else Raft via D-068, not etcd); + hacluster `cluster_count` semantics. + +## Revert + +- `git rm -r opentofu/modules/site-wan`; revert the `opentofu/main.tf` Stage-3 section + the + `vr1_dc1_planes` comment; `rm .terraform.lock`-nothing (lock unchanged). +- `git rm netbox/dc-edge-wan-import.py && git rm -r tests/dc-edge-wan-import`. +- `git rm overlays/dc-ha-scaleup.yaml`. +- Delete the `## D-123:` section from `docs/design-decisions.md`. +- No cloud state was touched -- nothing to unwind on the host. diff --git a/docs/changelog-20260716-d124-rack-addressing.md b/docs/changelog-20260716-d124-rack-addressing.md new file mode 100644 index 0000000..d878c13 --- /dev/null +++ b/docs/changelog-20260716-d124-rack-addressing.md @@ -0,0 +1,92 @@ +# 2026-07-16 -- D-124 adopted; vr1-dc0 MAAS rack (vvr1-dc0) wired; rack-mgmt NetBox importer + +## Context + +Continuation of the Stage-3 batch prep. The operator ruled the MAAS model (region on Office1 + +rack controller per DC, D-123) and the rack<->region management addressing (D-124 Scheme A: a +transit link on the office1<->dc0 mesh leg) + rack sizing. This lands the `vvr1-dc0` rack wiring +and the NetBox importer for the addressing. NO cloud mutation -- authoring + offline validation +only; the NetBox `--commit`, the `tofu apply`, and the rack install stay operator-gated. + +## What changed + +**D-124 (ADOPTED 2026-07-16).** `docs/design-decisions.md`. The Office1-region<->DC-rack management +overlay. Answers "why not metal-admin?": the rack's node-facing leg IS metal-admin, but D-100 rules +metal-admin DC-LOCAL and a RULED sub-item makes the Office1<->DC mesh the management path -- so the +rack straddles metal-admin (nodes) + the mesh transit (region). **Scheme A** (transit-numbered mesh, +a /30 or /31 point-to-point) + rack sizing **4 vCPU / 8192 MiB / 80 GiB** ruled. The specific transit +CIDR + rack IP are assigned in office1-netbox (apex; operator-held token) via the D-124 importer -- no +literal invented in-repo. + +**`vvr1-dc0` rack wired (`opentofu/`).** +- `main.tf`: `module "vvr1_dc0"` (cloudinit-vm) -- 4/8192/80, base = the noble image, `expose_nested_virt + = false` (rack-only; the region + its LXD host compose, not the rack), two legs + (`network_names = [metal-admin plane, mesh-vr1-dc0-office1]`), static `network_config` on both legs + with IPs from NEW `vr1_dc0_rack_*` variables (NetBox-assigned via tfvars, not invented) + a route to + the Office1 region (`10.10.0.0/22`, as-built D-115) over the transit. The interface-naming trap + (enp1s0/enp2s0 predicted, confirm on first boot) is flagged in-block. +- `variables.tf`: `vr1_dc0_rack_metal_admin_ip`, `vr1_dc0_rack_transit_ip`, `vr1_dc0_rack_transit_prefix`, + `vr1_dc0_rack_transit_peer_ip` (no defaults). +- `modules/mesh-link/outputs.tf`: added a `network_name` output (for the rack's transit-NIC reference). +- **Step-9 `maas-vm-host` DELIBERATELY STILL DEFERRED (DOCFIX-179):** adding the `provider "maas"` block + it needs would force EVERY plan -- including the Steps 1-8 substrate apply -- to demand MAAS creds, + and vr1-dc0's MAAS does not exist until vvr1-dc0 is up. Wire it as its own step once the rack MAAS is + reachable. +- `tofu validate` -> Success; `scripts/opentofu-validate.sh` -> 11/11 modules PASS. + +**`site-headend-install.sh` rack-only mode.** Added `--role ` (default `region+rack`, +100% backward-compatible). `--role rack` enrolls a DC rack controller to Office1's EXISTING MAAS region +(D-123): disable timesyncd -> install the `maas` snap (3.7/stable) -> `maas init rack --maas-url +--secret `. It SKIPS all region+rack work (PostgreSQL, region init, admin/ +login, LXD, the LXD vm-host, compose-DHCP) -- a DC rack runs no DB, composes nothing (the DC nodes are +OpenTofu-created + discovered via the vcloud virsh vm-host on the region), and metal-admin DHCP is a +region-side runbook step. Traps 1-3 (LXD/dnsmasq) do not apply in rack mode; trap 4's spirit (rack +configures NO DHCP) holds. The enroll secret is read from the file at mutate-time and NEVER echoed +(dry-run prints ``; the harness asserts the sentinel never leaks). The +`maas init rack --maas-url --secret` form is CONFIRMED against the MAAS snap docs/discourse (3.x-stable, +so it holds for the 3.7 pin), not assumed. Harness `tests/site-headend-install/` 19 -> 32 checks +(rack arg-contract, secret-non-leak, region-step-absence, backward-compat). Gauntlet stays 62. + +**Rack-mgmt NetBox importer (`netbox/dc-rack-mgmt-import.py` + `tests/dc-rack-mgmt-import/`).** +Registers the D-124 addressing in office1-netbox: the transit prefix (role `transit`, site-scoped +vr1-dc0) + the rack's metal-admin IP. Stdlib-only, mirrors `dc-edge-wan-import.py`: whole-plan preflight +(transit is /30|/31 with host-bits clear + subnet_of the container; rack IP in `10.12.8.0/22` static +band `.2-.49`, not the `.1` gateway; the transit role + container + vr1-dc0 site exist), DRY-BY-DEFAULT, +`SANDBOX_HOSTS` guard. Both literals are REQUIRED inputs (`--transit-cidr`/`--rack-ip`) -- no invented +CIDR. Harness **96/96** (29 static + 67 behavioral). + +**Scoping flags surfaced by the importer (operator-facing, logged NOT resolved):** +1. **The `transit` role must be seeded in office1-netbox** (die-if-absent precondition, like `edge` + was). No transit/management role exists in the frozen draft; `infra` exists but is explicitly SHARED + infra ("ptp transit, device loopbacks, RR loopbacks, anycast"), so D-124's "own role, mirroring the + Edge role" is a NEW dedicated `transit` role to seed. +2. **The transit may need its OWN supernet, not `10.12.0.0/16` (Cloud).** D-124 mirrors the Edge role, + which got a dedicated `172.30.0.0/16`. If office1-netbox assigns the transit from a dedicated + management/transit supernet (NOT subnet_of Cloud), the importer correctly REFUSES at preflight and + the fix is a one-line `CONTAINER` constant update. Decide the transit supernet when assigning. +3. Transit prefix scope = `dcim.site:vr1-dc0` (mirrors the DC-edge /24s); a region scope may be + preferred -- operator's call at assignment. + +## Tests / verification + +- `cd opentofu && tofu validate` -> Success; `bash scripts/opentofu-validate.sh` -> 11/11 PASS. +- `bash tests/dc-rack-mgmt-import/run-tests.sh` -> 96/96 PASS. +- `bash scripts/repo-lint.sh` -> 0 fail, 1 legacy WARN. +- `bash scripts/run-tests-all.sh` -> GAUNTLET: ALL GREEN (62 harnesses). + +## Operator `--commit` (ready; apex token operator-held) + + NETBOX_URL=http://10.10.1.10:8000 NETBOX_TOKEN= \ + python3 netbox/dc-rack-mgmt-import.py \ + --transit-cidr dc0 transit /30|/31> --rack-ip <10.12.8.2-.49> # dry-run; add --commit to write + +(Seed the `transit` role + confirm the transit supernet first -- see scoping flags 1/2. Then fill the +`vr1_dc0_rack_*` tfvars with the assigned values.) + +## Revert + +- `git checkout opentofu/main.tf opentofu/variables.tf opentofu/modules/mesh-link/outputs.tf` + (removes the `vvr1_dc0` block, the rack vars, the mesh-link output). +- `git rm netbox/dc-rack-mgmt-import.py && git rm -r tests/dc-rack-mgmt-import`. +- Delete the `## D-124:` section from `docs/design-decisions.md`. +- No cloud state touched. diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md index 25e6971..948e7ee 100644 --- a/docs/dc-dc-deployment-workflow.md +++ b/docs/dc-dc-deployment-workflow.md @@ -156,7 +156,22 @@ commit, operator-gated) as part of its definition-of-done. Stage 3 branches off | **Reuse vs new** | NEW (no OpenTofu/multi-rack precedent in VR0 DC0). Explicitly tagged shim: node-VM creation has no Roosevelt analog (physical racking + BMC enlistment replaces it there) -- do not treat as reusable production IaC. | | **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase2-tofu-dc-substrate.md`.** DC1-first; DC2 explicitly hard-gated (D-101 supernet unassigned) rather than run with placeholders. Walks every still-open decision (node sizing, MAAS zone/pool, `power_address`, config.xml tokens incl. boot-measured `WAN_IF`/`LAN_IF`, netem params) and stops at the corresponding step if unresolved rather than inventing values. States its own exit gate as "CONDITIONALLY MET AT BEST" pending Stage 2, node sizing, a real OPNsense boot measurement, and D-100's netem ruling. NOT YET EXECUTED. The plane/link/pool layer of `opentofu/` (see Stage 1) is reusable here too. `modules/node-vm` (blank disk, PXE-boot), `modules/opnsense-edge`, and `modules/maas-vm-host` (2026-07-09, registers the virsh host with MAAS via the official `canonical/maas` provider -- deliberately NOT `maas_vm_host_machine`, which composes new VMs and would fight `node-vm`) all now exist -- none instantiated (node sizing, OPNsense config.xml content, and a real MAAS zone/pool are all still pending decisions). `modules/netem-link` gives the netem mechanism for this stage's edge gate too. See `opentofu/README.md`. | -**State:** RUNBOOK WRITTEN, NOT YET EXECUTED, hard-dependent on Stage 2 for its MAAS-registration half. +**State:** IN PROGRESS (branch `dc-dc-stage3-phase2-dc-substrate`). RUNBOOK WRITTEN, NOT YET EXECUTED. +**Two blocking rulings landed 2026-07-15** (from a walk of the runbook's decision register): +- **D-121 (ADOPTED IN PART)** -- node layout = **Option C** (3 control + 2 compute + 3 storage per DC, + 16 node VMs), ruled after a whole-host resource validation (256 vCPU / 1024 GiB / 10 TiB measured; + C fits at 222 / 790 GiB / 5.4 TiB, RAM binding). Ruled per-node sizing recorded in D-121. Also + adopts the 14-service HA scale-up (1->3) that this layout carries at Stage 5. OPEN sub-ruling: the + vault-HA backend fork. +- **D-122 (ADOPTED)** -- DC edge/site shape: Office1 pattern edge (2048/2/nano, REST-API config), a + dedicated per-site `vr1-dc0-wan` uplink (dark fiber is East-West only), site nested in its own + containment VM, per-site MAAS. This CLOSES the runbook's Ruling-1 (node sizing) and Ruling-2 (edge + LAN/WAN) STOP-gates. +Fix-forward IN FLIGHT: runbook reconciled to post-D-119 naming + Steps 4-5 rewritten to the D-113(a2) +REST-API bootstrap. Honest first-run scope remains **Steps 1-8**; Step 9 (MAAS registration) still +gates on the per-DC MAAS rack-controller source (a Stage-4 gap -- no OpenTofu module stands one up yet). +The stale `vr1-dc1` "supernet not assigned" gate is corrected (D-115 assigned `10.12.64.0/19`); vr1-dc1 +stays out of scope by sequencing. --- diff --git a/docs/design-decisions.md b/docs/design-decisions.md index a08a471..e4bb2db 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -3256,3 +3256,303 @@ Two findings that shaped the method (and that the DC replication inherits): The NetBox-side carve record (Step 6 -- register the `.2-.49`/dynamic/node child ranges + the two service IPs in the sandbox, then feed upstream) is still OWED and rides with C2; it is NOT done here. + +## D-121: VR1 makes HA real -- scale the decorative single-unit control plane to 3, and the per-DC node layout that carries it + +**Status:** ADOPTED IN PART (2026-07-15, operator ruling). **Node layout = Option C** (3 control + +2 compute + 3 storage per DC, 16 node VMs) -- ruled after the whole-host validation proved it FITS +(details below). The **14-service HA scale-up** (1 -> 3) is ADOPTED. **Vault-HA backend sub-ruling: RESOLVED = (v-a)** +(2026-07-15, operator ruling) -- keep vault MySQL-backed (on the already-3-unit `mysql-innodb-cluster`), +scale to 3, re-add the `vault:ha <-> vault-hacluster:ha` relation. GATE: verify the deployed vault charm +(`1.8/stable`) actually does HA leader-election on the MySQL storage backend with 3 units before commit; +if it does NOT, do not fall back to (v-b) etcd -- evaluate Raft integrated storage and route to D-068 +(etcd storage is deprecated upstream; Raft is the better Roosevelt target). Rationale: reuses existing +HA (zero new apps), keeps control-node container density down, does not front-run the unsettled Roosevelt +backend choice (D-068), and adds no NEW SPOF (the whole cloud already depends on mysql). Governs +Stage 3's `modules/node-vm` sizing/count (was blocking phase2 runbook Step 6 -- now UNBLOCKED for +layout) and the Stage 5 per-DC bundle deploy. Raised by the operator's Stage-3 ruling ("this is the +time we add the additional HA nodes missing from the previous deployments"). + +**Ruled node sizing (Option C, validated to fit -- keep control nodes <= 64 GiB per the sensitivity):** +3x control @ 16 vCPU / 64 GiB / 150 GiB; 2x compute @ 12 vCPU / 48 GiB / 100 GiB; 3x storage @ 8 vCPU +/ 24 GiB / 550 GiB (500 OSD + 50 OS). Whole-host allocation across Office1 + both DCs + edges + +containment overhead = 222 vCPU / 790 GiB / 5.4 TiB of the measured 256 / 1024 / 10240 (RAM binding at +77%). `ceph-osd` = 3 units/DC (the storage nodes), `nova-compute` = 2 units/DC (the compute nodes), +`ceph-mon`/`mysql`/`rabbitmq`/`vault`/`ovn-central` = 3 across the 3 control nodes. + +### The problem this fixes + +VR0/testcloud ran its OpenStack control plane **single-unit by design**. Per **D-009** ("Hacluster +modeling at testcloud scale"): the hacluster + VIP relations were included at `num_units: 1` as a +**decorative** pattern -- "a single unit can't form a real HA quorum" -- explicitly so that +"Roosevelt scale-up is mechanical: change `num_units: 1` -> `num_units: 3` and rerun." The bundle +already carries all 12 hacluster subordinates and, per **D-036**, both provider+metal VIPs on every +clustered API app. VR1 (the regional rehearsal) is where that decorative HA is made real -- the +missing units get added. + +### Measured current state (bundle.yaml, machines 8/9/10/11 -- the 4 VR0 nodes) + +Already at quorum (NO change): `mysql-innodb-cluster`=3 (D-062, must deploy at 3), +`ovn-central`=3 (Raft), `ceph-mon`=3 (Paxos). Scale-out (sized to node count): `ceph-osd`=4, +`nova-compute`=3 (on 9/10/11 -- machine 8 is compute-free today). + +**Missing HA -- 14 apps currently single-unit that this decision scales to 3:** + +| App | Now | -> | Mechanism | Governing | +|---|---|---|---|---| +| keystone, glance, neutron-api, nova-cloud-controller, placement, cinder, openstack-dashboard, octavia, barbican, magnum, designate | 1 | 3 | hacluster + dual VIP (already wired) | D-009, D-036 | +| ceph-radosgw | 1 | 3 | hacluster + VIP (wired; NOT in D-009's named 12 but HA-eligible) | -- | +| rabbitmq-server | 1 | 3 | native clustering / quorum queues (no hacluster) | -- | +| vault | 1 | 3 | see the backend sub-question below | D-006 (revised by C1) | + +Left single (NOT scaled): `ceph-rbd-mirror`=1 (**D-108** rules single-unit for VR1), +`glance-simplestreams-sync`=1 (utility), `memcached`=1 (cache; optional 3), `designate-bind`=1 +(DNS backend; optional 2). + +### Two realities the "mechanical 1->3" framing hides (fold into the execution estimate) + +1. **It is a placement rewrite, not a `num_units` tweak.** Every app pins its unit explicitly + (`keystone to: [lxd:8]`, `glance to: [lxd:11]`, ...). Setting `num_units: 3` with a single-entry + `to:` list makes Juju place one unit there and auto-place the other two wherever -- NOT + guaranteed on three DISTINCT hosts, which is the entire HA value. Each of the 14 apps needs its + `to:` expanded to a real 3-machine spread (anti-affinity). This is the actual work. + + **Placement-balance rule for the Stage-5 rewrite (Option C, 3 control nodes).** The 14 services x 3 + units = 42 API LXD containers land on the **3 control nodes** (`vr1-dc0-control-01..03`) -- ~14 per + node, roughly 2x the density of VR0's balanced ~7 containers/node (measured from the openstack0-3 + bundle placement, 27 control containers over 4 nodes). Two rules keep that density balanced: + - **Spread each app's 3 units one-per-control-node** (`to: [lxd:control-01, lxd:control-02, + lxd:control-03]`). 3-unit HA does this naturally, so the RAM-heavy stateful trio + (`mysql-innodb-cluster`, `rabbitmq-server`, `vault`) auto-balances to one replica per control + node -- do NOT stack two of them on the same node (VR0's own placement already avoided this: + rabbitmq on M10, vault on M11, mysql spread 8/9/10). + - **Distribute the ~42 containers EVENLY (~14/control node), not front-loaded onto control-01.** + The whole-host validation sized control nodes at 64 GiB for exactly this ~14-container load; if a + future rebalance wants ~9/node instead, the lever is a 4th control node per DC (NOT more units). + `ceph-osd`(3) sits on the storage nodes and `nova-compute`(2) on the compute nodes -- neither + competes for control-node RAM. This rule governs the Stage-5 bundle deploy; it does not change the + node COUNT ruled above. +2. **Vault HA backend fork -- RESOLVED = (v-a), see Status.** The bundle runs vault **MySQL-backed** + today (`vault:shared-db -> vault-mysql-router -> mysql-innodb-cluster`), and **C1 revised D-006**: + "etcd/easyrsa dropped for testcloud; Raft-vs-etcd is a Roosevelt item," and the + `vault:ha <-> vault-hacluster:ha` relation was **removed (BUNDLEFIX-002)**. So scaling vault is + NOT "just add units". The operator ruled **(v-a)** (charm-verify gated); (v-b) retained below for + the record: + - **(v-a) Keep MySQL-backed, re-add the `vault:ha` relation, scale to 3** -- lowest delta; + leans on the already-3-unit mysql-innodb-cluster. Must be verified against the deployed vault + charm's real multi-unit/ha behavior on a MySQL backend before committing. + - **(v-b) Adopt D-006's etcd(3)+easyrsa(1) HA backend** -- the originally-documented topology, + but reverses C1 and adds two applications. + **Operator sub-ruling needed.** Either way, 3-unit vault **triples the manual-unseal burden**, + which intersects the still-OPEN **SEC-003** (custodian assignment + second-person unseal + rehearsal) that already keeps `d011-06-vault-unseal` MANUAL and gates D-011's full close. + +### The per-DC node layout (the `modules/node-vm` count/sizing this stage must ratify) + +The site-DOWN / recovery drill the operator wants (D-122) has the **whole DC VM** as its fault +domain, so intra-DC hyperconvergence is acceptable; role separation mainly buys single-node-failure +drills, a secondary concern here. + +| Opt | Per-DC layout | nodes/DC | x2 DC | HA semantics | Whole-host cost (MEASURED, all layers) | +|---|---|---|---|---|---| +| A | 3 hyperconverged | 3 | 6 | quorum survives 1 loss; size=3 has ZERO rebuild headroom | lightest | +| **B (rec)** | **4 hyperconverged (= VR0 baseline)** | **4** | **8** | quorum + 4th host = a size=3 re-replication target after a node loss; lowest delta to the proven VR0 placement | 158 vCPU / **838 GiB** / 5.6 TiB (RAM 82%) | +| C | 3 control + 2 compute + 3 storage | 8 | 16 | cleanest HA/DR, Roosevelt-shaped | 222 vCPU / **790 GiB** / 5.4 TiB (RAM 77%) | + +**Whole-host validation (2026-07-15, MEASURED budget 256 vCPU / 1024 GiB / 10 TiB; model at +`scratchpad/optc-calc.py`, all layers: Office1 as-built + both DCs + edges + containment overhead).** +**Both B and C FIT.** The binding resource is **RAM**, not disk (disk ~53-55%, and thin-provisioned) +and not vCPU (overcommits freely in a sim; C's 87% "allocation" is not a real ceiling). Counter to +the first-pass intuition, **C is NOT heavier than B** -- at lean-sim node sizing C uses *less* RAM +(790 vs 838 GiB) because role separation lets the 6 storage nodes run lean (24 GiB) vs B's uniformly +96-GiB hyperconverged nodes. So **C's real cost is operational (16 node VMs vs 8, larger VR0 delta), +not capacity.** SENSITIVITY: the knob is control-node RAM -- C with 96-GiB control nodes hits 982 GiB +(96%, only 42 GiB margin); keep control nodes <= 64 GiB. Per-node RAM sizing dominates the total far +more than the A/B/C structure does. + +**RULED: Option C (2026-07-15).** The whole-host validation (below) removed the capacity objection -- +C fits, and is in fact RAM-lighter than B -- so the decision turned on HA/DR quality: C is +role-separated and Roosevelt-shaped, giving the cleanest site-down AND single-node-failure drills, +which is the point of the VR1 rehearsal. B (4 hyperconverged/DC) was the runner-up on lowest-delta / +operational-simplicity grounds (my initial recommendation); A (3 nodes) was rejected as FEWER than +VR0's 4, reading against the operator's "additional HA nodes." Control nodes are held <= 64 GiB per +the sensitivity finding. B's rationale is retained here for the record: it was the smallest change +from VR0's proven placement, with three distinct hosts for quorum plus a fourth for Ceph rebuild. + +**Disk-budget check (validated 2026-07-15, `scripts/dc-dc-ceph-disk-budget.sh`, Option B 4+4, 0.2 +backup overhead):** PASS across the plausible per-node OSD range -- 300 GiB/node -> 7.18 TiB margin, +500 GiB -> 5.31 TiB, 700 GiB -> 3.43 TiB. **Disk is NOT the binding constraint for B**, so the +node-COUNT decision is robust regardless of final OSD sizing. NB the per-node OSD footprint is an +ASSUMPTION here (the ratified value pends the OSD-sizing decision and should be re-derived from the +measured DC0 footprint, as Stage 1's own margin was); this run proves headroom, not the final number. + +**Per-node sizing (feeds the module):** at full HA a hyperconverged node runs ~13 control LXD +containers + an OSD + (on 3 of 4) nova-compute, atop D-040's 8 GiB reserved-host-memory. Propose +**16 vCPU / 64-96 GiB RAM / OSD-sized disk** per node (lean to 96 GiB -- 64 is tight once D-040 +reserves 8 and the container stack lands). 8 nodes x 16 vCPU = 128 vCPU and x 96 GiB = 768 GiB, both +inside vcloud's 256 vCPU / 1 TiB with headroom for Office1 + the edges. + +### What the operator ruled (2026-07-15) + +1. **Node layout:** RULED **Option C** (3 control + 2 compute + 3 storage per DC), sizing above. +2. **Vault HA backend:** RULED **(v-a)** MySQL-backed + re-add `vault:ha`, charm-verify gated (Status). + The SEC-003 unseal-burden intersection stands (3-unit vault = 3x manual unseal). +3. **Per-node sizing:** RULED (control 16/64/150, compute 12/48/100, storage 8/24/550; Status). + +**Related:** D-009 (decorative HA pattern this executes), D-006 + C1 (vault backend history), +D-036 (dual VIPs, already present), D-062 (mysql-at-3), D-108 (rbd-mirror stays 1), D-040 +(reserved memory), D-104 (single-unit JUJU controller -- unaffected; this is the OpenStack control +plane, not the Juju bootstrap node), SEC-003 (unseal). Does NOT touch DC2/`vr1-dc1` scope. + +## D-122: VR1 site shape -- nested-per-site containment, dark fiber + dedicated per-site L3 ISP, DC edge follows the Office1 pattern + +**Status:** ADOPTED (2026-07-15, operator ruling this session -- answers Stage-3 Ruling 2). Records +the deployment shape so Stage 3's edge/network steps stop re-deriving it. Governs +`modules/opnsense-edge` wiring for each DC and the per-site uplink networks. + +**Decision (operator-ruled):** + +- **Each site is nested inside its own containment VM** (`voffice1` for Office1; the `vvr1-dc0` / + `vvr1-dc1` analogs per DC), so a "physical site down" and its recovery -- plus site-to-site tests + -- are executable as `virsh destroy ` against a single object. This is D-114's model, + now affirmed as the shape for the DCs too, not just Office1. +- **Dark fiber connects DC <-> DC <-> Office** (the `mesh-*` legs) and is **East-West only** -- + replication / site-to-site, **never an internet path** (consistent with D-100 and DOCFIX-185's + transport model). +- **Each site has its OWN dedicated L3 "ISP" uplink, not shared between sites.** Concretely: build a + per-site WAN network on the **`office1-wan` pattern** (a dedicated NAT'd `/24` uplink segment per + site), e.g. `vr1-dc0-wan`, and attach the DC edge's WAN there -- NOT to a mesh leg. +- **The DC OPNsense edge follows the proven Office1 edge pattern:** sizing **2048 MiB / 2 vCPU / + nano image** (carried from the applied `office1_opnsense`, DOCFIX-189), config over the REST API + (D-113(a2)). +- **Edge NIC model = 2-NIC (operator ruling 2026-07-15), matching the baremetal target.** The NIC + count is shaped to the Roosevelt hardware: **nodes get 6 NICs** (one per plane, as measured on VR0 + openstack0-3), but the **edge gets 2** (WAN + one LAN) -- the six planes are **routed by the fabric** + (the `dc-planes` segments are DELIBERATELY isolated L2; OVN/OpenStack own their L3 addressing/routing, + not libvirt and not the edge). So the edge is NOT an inter-plane router; it is only the external + boundary. This keeps `modules/opnsense-edge` at its existing 2-NIC shape -- **no module extension**. + - **WAN** = the dedicated per-site ISP uplink `vr1-dc0-wan` (`172.30.2.0/24`, D-115; register in + office1-netbox -- not yet loaded -- and build as a tofu module, promoting off the virsh-created + `office1-wan` one-off). + - **LAN** = **provider-public** (`10.12.4.0/22`): the edge is the upstream/external gateway the + provider network exits through (FIP/SNAT egress + GUA injection per D-100's br-ex/provider model). + metal-admin and the other four planes reach external via fabric (OVN) routing, not a direct edge + leg; NTP-from-edge (D-107) is served across that fabric routing. +- **Each site runs its own MAAS controller** (as `voffice1` does), and an existing MAAS controller + MAY be used to bootstrap a new site's deployment. + +**Consequence for Stage 3:** the phase2 runbook's Ruling-1/Ruling-2 open items on edge sizing, WAN +uplink, and LAN role are CLOSED by this entry (build `vr1-dc0-wan`, wire the edge like Office1). The +remaining Stage-3 blocker is D-121 (node layout/sizing) plus the per-DC MAAS rack-controller source +(Stage 4 gap -- honest Stage-3 scope is Steps 1-8 until that is stood up). **Related:** D-114 +(containment VM + MAAS-composed LXD VMs), D-100 (per-site edge, dark fiber), D-107 (headend not a +core-service provider), DOCFIX-185 (edge is a real-ISP router, not an egress airgap), D-113(a2) +(edge config over REST). DOCFIX-185's D-100/D-107 amendment note is subsumed here. + +## D-123: VR1 DC site-down DR model -- node-VM placement + the per-DC MAAS headend (resolves a D-122/D-103 tension) + +**Status:** ADOPTED **Model A** (2026-07-15, operator ruling -- read from "Yes, fire off those tasks" +in response to the A/B question; flag if B was intended). Nodes stay vcloud-level; site-down = destroy +the `vr1-dc0-*` domain group; `vvr1-dc0` is a MAAS/LXD headend (the `voffice1`/`site-headend-install.sh` +pattern). **MAAS MODEL RULED (2026-07-15, operator):** ONE MAAS **region on Office1** + a **rack controller per +DC** (`vvr1-dc0` = the vr1-dc0 rack), matching buildout-design line 110. The rack serves PXE/DHCP +locally on metal-admin and PROXIES OS images from the Office1 region (so no direct DC-edge egress is +needed -- that egress worry dissolves under region+rack). `maas-vm-host` registers vcloud's virsh to the +**Office1 region** (which exists) so MAAS discovers the vr1-dc0 node domains. + +**REMAINING (genuine Stage-4 design, NOT invented here) -- the rack<->region management overlay.** +`vvr1-dc0` must reach the Office1 region (10.10.0.20:5240) to enroll, and it may NOT sit on Office1's +site LAN (site isolation, D-122). Per D-100 the management path is the `office1<->dc0` mesh leg -- but +that segment has **no assigned L3** (the mesh legs are isolated L2), and the rack needs static addresses +on both the mesh (management) and metal-admin (it can't DHCP -- the planes have no libvirt DHCP, and the +rack IS the DHCP server there). So concretely standing up `vvr1-dc0` needs a **NetBox-assigned addressing +pass** for (a) the Office1-region<->DC-rack management subnet on the mesh leg and (b) the rack's +metal-admin static IP, plus a rack-controller **sizing** decision. This is a D-101-family addressing +design owned by `dc-dc-phase3`, not a one-line ruling -- the `vvr1-dc0` + `maas-vm-host` blocks stay +HELD until it lands (assigning IPs by guess would violate hard rule 2). Everything ELSE (Steps 1-8 +substrate) is wired and validated. + +**The tension.** D-122 records the operator's shape as *"each site nested in its own VM so we can test +physical site down"* (`virsh destroy ` = site-down). But the ADOPTED seam D-103 + D-114 says +the OpenStack **node VMs are OpenTofu-created and MAAS merely DISCOVERS them** -- and the built tooling +implements exactly that: `modules/node-vm` creates the nodes on the **vcloud** host attached to the +`vr1_dc0_planes`, and `modules/maas-vm-host` registers **vcloud's** virsh to the DC's MAAS so MAAS +discovers those already-existing domains. D-114 scopes the containment VM to the MAAS/LXD headend + the +**non-stack** (LXD-composed) machines only -- NOT the OpenStack nodes. So for a DC, the nodes are +vcloud-level siblings of the headend VM, and "site-down = one `virsh destroy`" is literally true only +for Office1 (whose services all live inside `voffice1`), not for a DC. + +**Model A (RECOMMENDED) -- nodes vcloud-level; site-down = destroy the domain GROUP.** +- Node VMs: vcloud-level libvirt domains on `vr1_dc0_planes` (as built). `vvr1-dc0` = a site headend VM + (the proven Office1 pattern: `cloudinit-vm` like `voffice1` + `scripts/site-headend-install.sh`, which + already names `vvr1-dc0/vvr1-dc1` as its future targets) running MAAS region+rack + LXD. `maas-vm-host` + registers vcloud's virsh -> `vvr1-dc0`'s MAAS, which discovers the nodes. +- **Site-down DR = destroy the `vr1-dc0-*` domain group** (headend + 8 nodes + edge) as a scripted op + (owned by `dc-dc-teardown-rollback.md`), NOT a single `virsh destroy`. +- Nesting depth = 2 (vcloud -> node -> KVM guest), the SAME as VR0-proven; nova-compute's KVM guests stay + shallow. Reuses ALL built tooling unchanged (`node-vm`, `maas-vm-host`, `site-headend-install.sh`). +- **Closes the "Stage-4 gap":** the per-DC MAAS "rack controller" is just a `vvr1-dc0` headend VM -- an + OpenTofu `cloudinit-vm` block shaped like `module "voffice1"` + a `site-headend-install.sh` run. + Prepare-able now (opentofu block + the Step-9 `maas-vm-host` wiring). + +**Model B -- true single-VM containment (nodes nested inside `vvr1-dc0`).** +- Site-down = one `virsh destroy vvr1-dc0` (D-122 literal). But 4-level nesting + (vcloud -> vvr1-dc0 -> node -> guest), reshapes the planes + `node-vm` to target `vvr1-dc0`'s libvirt, + contradicts the as-built D-103/D-114 discover-OpenTofu-created-nodes flow, and stresses nested-virt for + nova-compute. Heaviest delta. + +**Recommendation: Model A** -- least delta to the adopted decisions and the built tree, shallower +nesting, and it reuses the proven Office1 headend pattern verbatim. It only asks that D-122's DR wording +be read as "destroy the site's domain GROUP" for DCs (true single-VM destroy remains literal for +Office1). **What the operator rules:** Model A vs B (one line). **Related:** D-122 (site shape), D-114 +(containment scope), D-103 (OpenTofu/MAAS seam), and `scripts/site-headend-install.sh` (the reusable +per-site headend installer). + +## D-124: the Office1-region <-> DC-rack management overlay -- addressing for the MAAS control path + +**Status:** ADOPTED (2026-07-16, operator ruling). **Scheme A** (transit-numbered mesh) + rack sizing +**4 vCPU / 8192 MiB / 80 GiB** confirmed. The office1<->dc0 mesh leg is a point-to-point transit +(region <-> rack); metal-admin stays purely node-facing; the rack straddles both. Still to assign in +office1-netbox (the apex; operator-held token, main-loop can't query it live): the specific transit +`/30`(or `/31`) CIDR + the rack's metal-admin static IP -- registered via the D-124 importer with +operator-supplied values (no literal invented in-repo). Original options + rationale retained below. + +**Why this exists (answers "why not just metal-admin space?").** The rack's NODE-facing leg IS in +metal-admin (`10.12.8.0/22`) -- that is where it serves PXE/DHCP to the vr1-dc0 nodes, correct. But the +rack must also reach the **MAAS region on Office1** (`10.10.0.20:5240`) to enroll, and: +- **D-100 rules metal-admin DC-LOCAL** -- the plane deliberately does not extend to Office1 (only the + replication plane crosses fiber), and a RULED D-100 sub-item states *"Office1<->DC fiber carries + management traffic only (MAAS/Juju/operator)."* So the region<->rack control path is the + **`office1<->dc0` mesh leg**, not metal-admin. +- The rack may NOT sit on Office1's site LAN (`10.10.0.0/22`) either -- that breaks site isolation (D-122). + +So `vvr1-dc0` **straddles two legs**: metal-admin (node-facing, existing space) + the office1<->dc0 mesh +(region-facing). The mesh leg is today unnumbered isolated L2; this decision numbers it. (Same applies to +office1<->dc1 for the second DC later; dc0<->dc1 stays replication-only, D-108 -- NOT management.) + +**The addressing choice (operator rules A or B).** Both keep metal-admin DC-local at L2 and both need only +a tiny link; they differ in whether the rack carries a second address: +- **(A) Transit-numbered mesh (recommended).** The office1<->dc0 mesh leg gets a small point-to-point + transit (a `/30` or `/31`): Office1 region on one end, rack on the other. The region routes to the + rack; metal-admin stays purely node-facing. Cleanest role separation, standard inter-site practice, + and it generalizes per-leg (office1<->dc1 gets its own transit). The rack has 2 IPs (transit + metal-admin). +- **(B) Metal-admin routed over the mesh.** No separate transit subnet; the rack's single metal-admin IP + is advertised to the region over an unnumbered/`/31` mesh link. Fewer addresses (your instinct -- one + rack identity), but it couples metal-admin reachability across the fiber, a soft touch on "DC-local." + +**Where the numbers come from.** Like D-115/D-117, this decision rules the SCHEME; NetBox (office1-netbox, +the IPAM apex) assigns the actual transit CIDR + the rack's metal-admin IP -- no literal is invented here. +A candidate management/transit supernet (its own role, mirroring the D-115 Edge role) is a NetBox call; the +importer will register whatever is ruled. The rack's metal-admin IP lands in the D-120 static band +(`.2-.49`) of `10.12.8.0/22`, not the node band. + +**Rack-controller sizing (propose to confirm).** A MAAS rack controller is lightweight (it proxies images +from the region, runs no PostgreSQL): **4 vCPU / 8192 MiB / 80 GiB** is the proposed default -- far below +voffice1's region+rack+LXD 16/32/600. Confirm or adjust. + +**What the operator rules:** (1) mesh management addressing = A (transit) or B (metal-admin routed); +(2) rack sizing. Then: ratify the scheme -> NetBox assigns the transit + rack IPs (a small importer, the +D-120 pattern) -> `vvr1-dc0` (cloudinit-vm on the two legs) + `maas-vm-host` (Step 9) wire against real +addresses. **Related:** D-100 (DC-local planes + mesh = management), D-101 (address families), D-115 +(Edge-role precedent for a new transit role), D-120 (static-band convention), D-123 (region+rack model), +D-107 (per-DC mirror -- why the rack needs no direct egress). diff --git a/docs/security-ledger.md b/docs/security-ledger.md index 8f7329f..581022d 100644 --- a/docs/security-ledger.md +++ b/docs/security-ledger.md @@ -18,6 +18,8 @@ per D-069. | SEC-007 | 2026-07-12 | `~/vr1-office1-creds/` on the jumphost holds the Office1 edge root password + its bcrypt hash + the `office1_svc` SSH PRIVATE key + (added 2026-07-13) the OPNsense **API key/secret** (`opnsense-api.txt`, 0600), all in plaintext (dir 0700, files 0600) | 2026-07-12 edge build; custody detail deliberately not recorded here per D-069 | operator | OPEN -- required for edge management (D-112(c) makes SSH the only management path), so this is a ROTATION obligation, not a delete-me. Rotate at v1 close / if the jumphost is rebuilt or shared | | SEC-008 | 2026-07-13 | `~/vr1-office1-creds/tailscale-authkey.txt` (vcloud, 0600) -- the operator's 48-char Tailscale auth key for the SELF-HOSTED control plane `tailscale.baldurkeep.com`. Used to enrol `office1-tailscale`. Consumed BY PATH and never printed into a session; the copy shipped to the node was SHREDDED after `tailscale up` (tailscaled holds its own node key now). | 2026-07-13 Office1 Tailscale enrolment; docs/vr1-office1-as-built.md | operator | OPEN -- ROTATION obligation, not a delete-me: the key is still on vcloud for re-enrolment. Revoke/reissue on the control server at v1 close, or immediately if vcloud is rebuilt or shared. | | SEC-009 | 2026-07-15 | Credential/env SPRAWL + world-readable exposure on vcloud: three env files sat loose in `~` outside the consolidated `~/vr1-office1-creds/` -- `.vr1-netbox.env` (upstream NetBox token), `vr1-office1.env` (edge root-password hash + SSH key path), and `vr1-stage1.env` which held `TF_VAR_maas_api_key` (a MAAS API secret) at mode **0664 (group/world-readable)**; `tailscale-authkey.txt` in the creds dir was also 0664. | docs/changelog-20260715-creds-consolidation.md | operator | **REMEDIATED 2026-07-15** -- all three moved into `~/vr1-office1-creds/`, every sensitive file `chmod 600`. STANDING CONVENTION established (below). Note: this is a CLOSED in-house test; the rotation obligations of SEC-005/006/007 still apply at v1 close, this row only closes the SPRAWL + PERMS exposure. | +| SEC-010 | 2026-07-16 | **metal-admin DC-LOCAL invariant (D-052/D-100) is PRESERVED but NOT ENFORCED in the committed Stage-3 config.** The `vvr1_dc0` rack straddles metal-admin (10.12.8.0/22, DC-local) + the office1<->dc0 transit (crosses fiber). Its committed cloud-init pins static IPs only -- **no `net.ipv4.ip_forward=0` sysctl, no host firewall on the transit leg.** Cross-plane routing (metal-admin <-> the whole Office1 /22, bidirectional) is blocked ONLY by Ubuntu's distro default; the deferred MAAS-rack install could silently flip it. A MAAS rack proxies at the application layer and needs no kernel forwarding, so pinning is free. | 2026-07-16 plane-segregation review (advisor + 2 read-only agents); `opentofu/main.tf` vvr1_dc0 :376-439 | operator | **OPEN -- PRE-APPLY hardening.** Add `net.ipv4.ip_forward=0` (+ v6) via cloud-init `write_files`/`bootcmd` sysctl on the rack, and an nft/ufw rule pinning the transit leg to MAAS region ports only. Same pin must follow onto `voffice1` when its transit leg is wired (currently single-homed, so F1 is latent). Region route must target only the rack transit /30, never 10.12.8.0/22 (NetBox/region config, off-repo). Close BEFORE `tofu apply`. | +| SEC-011 | 2026-07-16 | **Node least-connectivity gap (not an L2 breach).** Under D-121 Option C role separation, all 8 nodes get a uniform 6-plane NIC set, so a ceph-osd STORAGE node has a leg on provider-public (external/FIP) + data-tenant (tenant geneve) -- planes it never binds per D-052. Planes stay isolated L2 (no crosstalk), but it is unnecessary external attack surface that VR0's genuinely-hyperconverged all-6 node did not have. | 2026-07-16 plane-segregation review; `opentofu/main.tf` `local.vr1_dc0_node_nics` :320-352 | operator | **OPEN -- PRE-APPLY hardening.** Replace uniform 6-NIC with per-role NIC sets (storage/ceph-osd drops provider-public + data-tenant at minimum). Confirm the exact per-role plane set against the regenerated D-052 bundle bindings -- do NOT invent it. Attack-surface reduction; not blocking, but cheap to do before apply. | **STANDING CONVENTION (SEC-009, 2026-07-15): per-site credential/env consolidation.** ALL sensitive files AND environment/config files for a site live in a single `~/-creds/` folder on vcloud, diff --git a/docs/session-ledger.md b/docs/session-ledger.md index c53184b..b87023b 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -38,9 +38,19 @@ _Re-seeded from the 2026-07-15 scan. Re-run `bash scripts/ledger-scan.sh` to ref the pre-DC-DC controller HA/backup session). - **OPEN security rows:** SEC-001, SEC-003, SEC-004, SEC-005, SEC-006, SEC-007, **SEC-008**. (SEC-008 -- the vcloud re-enrolment key -- was MISSING from the prior block.) -- **Next-free numbers:** D = **121**, DOCFIX = 196, BUNDLEFIX = 013. - (Since the prior block: D-119 + D-120 assigned and ADOPTED, and DOCFIX-195 assigned by this change - -- so next-free advanced D 119->121 and DOCFIX 195->196.) +- **Next-free numbers:** D = **125**, DOCFIX = 196, BUNDLEFIX = 013. + (D-124 PROPOSED 2026-07-16: the Office1-region<->DC-rack management overlay addressing -- numbers the + office1<->dc0 mesh leg so the D-123 rack can reach the MAAS region. metal-admin stays DC-local (D-100); + the rack straddles metal-admin (nodes) + the mesh (region). Operator rules scheme A (transit /30-31) vs + B (metal-admin routed) + rack sizing; then NetBox assigns + the headend wires.) + (2026-07-15: D-121, D-122, D-123 assigned -- next-free advanced D 121->124. D-121 ADOPTED (node + layout = Option C; vault sub-ruling RESOLVED = v-a). D-122 ADOPTED (site shape + 2-NIC fabric-routed + edge). **D-123 ADOPTED Model A** (nodes vcloud-level; site-down = destroy the vr1-dc0-* group). MAAS + model RULED: region on Office1 + rack controller per DC (vvr1-dc0 = vr1-dc0 rack, proxies images from + the region). REMAINING (genuine Stage-4 design, NOT invented): the rack<->region management-overlay + ADDRESSING -- the office1<->dc0 mesh leg has no assigned L3 and the rack needs static IPs on mesh + + metal-admin. A NetBox D-101-family addressing pass owned by dc-dc-phase3; vvr1-dc0 + maas-vm-host stay + HELD until it lands.) - **Standing numbering rule:** never write an identifier-shaped token (D-/DOCFIX-/BUNDLEFIX-NNN) ABOVE the real high-water mark anywhere in `docs/` or `runbooks/` prose -- `ledger-scan` reads prose and a decoy token inflates the next-free counter. This has bitten twice. @@ -114,6 +124,117 @@ write-back stays DEFERRED to end-of-deployment. This closed the last piece of cl `docs/changelog-20260715-d120-load-and-iprange-tooling.md`. +**PLANE-SEGREGATION REVIEW of the Stage-3 commit (2026-07-16; advisor + 2 read-only agents).** +Reviewed the committed networking/plane changes (`a48a60f`, NOT applied) for crosstalk / segregation +breaches. **VERDICT: no crosstalk, no active violation -- committed state is SAFE.** All six `vr1-dc0-*` +planes + all three `mesh-*` legs are isolated L2 (measured: `net-dumpxml` shows no forward/ip/nat, +isolation ENFORCED BY CONSTRUCTION in `dc-planes`/`mesh-link`); the edge is 2-NIC so it structurally +cannot reach the 5 non-LAN planes; `vr1-dc0-wan` NAT touches only the edge WAN. metal-admin DC-LOCAL +(D-052/D-100) is **preserved but not ENFORCED** -- it rests on unpinned host defaults. **Two pre-apply +hardening items logged as SEC-010 (rack `ip_forward=0` + transit-leg firewall -- the one real +plane-bridge) and SEC-011 (node role-scoped NIC sets -- least-connectivity).** Say plainly at handoff: +"committed state is safe; two items to close before `tofu apply`." Plus three notes: +- **FUNCTIONAL (fix before apply, not security):** the rack's committed route `to 10.10.0.0/22 via + ` points at a next-hop with **no VM attached** -- `voffice1` is single-homed + (`network_names = [office1-local]` only, main.tf:184), NOT on `mesh-vr1-dc0-office1`. As committed, + apply yields a rack routing to a nonexistent peer, and the Office1<->DC region path is not closed. + Either wire `voffice1`'s transit leg or gate the rack route until it exists. +- **SEGREGATION CHECK (first boot):** the rack NIC->plane mapping (`enp1s0`->metal-admin, + `enp2s0`->transit) is assumed PCI-order==name; if swapped, the metal-admin /22 lands on the + Office1-facing wire. Verify via MAC (`virsh domiflist` x `network_names` order) before trusting + addressing. This is a segregation check, not cosmetic. +- **DEFERRED (track):** the edge WAN<->provider-public firewall is post-boot REST (opnsense-api), + not in the commit -- "enforced by absence" covers plane REACHABILITY only, not full edge hardening. +When `voffice1`'s transit leg is later wired it becomes an office1-local<->transit bridge and inherits +the SAME `ip_forward=0` pin requirement (a richer surface than the rack: MAAS region + LXD host). + + +**STAGE 3 IN PROGRESS (branch `dc-dc-stage3-phase2-dc-substrate`) -- two rulings landed 2026-07-15.** +Walked the phase2 runbook and surfaced its blocking decisions; the operator ruled both: +- **D-121 (ADOPTED IN PART).** VR1 makes the decorative single-unit control plane REAL: 14 services + scale 1->3 (the 12 hacluster-backed APIs + ceph-radosgw + rabbitmq + vault; `ceph-rbd-mirror` stays + 1 per D-108). This is D-009's "mechanical 1->3" pulled forward from Roosevelt -- but it is a PLACEMENT + rewrite (every app's `to:` must spread across 3 distinct hosts), not a num_units tweak. **Node layout + = Option C** (3 control + 2 compute + 3 storage per DC, 16 node VMs), ruled after a whole-host + validation PROVED it fits: measured budget 256 vCPU / 1024 GiB / 10 TiB, Option C allocates + 222 / 790 / 5.4Ti (RAM binding at 77%; model at `scratchpad/optc-calc.py`). Ruled sizing: control + 16/64GiB/150GiB, compute 12/48/100, storage 8/24/550; keep control <=64 GiB (96 GiB -> 96% RAM). + **Vault-HA sub-ruling RESOLVED = (v-a)** (2026-07-15): keep MySQL-backed (on the already-3-unit + mysql-innodb-cluster), scale to 3, re-add `vault:ha`. GATE: verify the `1.8/stable` vault charm does + HA on the MySQL backend with 3 units; if not, evaluate Raft (NOT etcd) via D-068. 3-unit vault + triples the unseal burden (intersects OPEN SEC-003). +- **D-122 (ADOPTED).** VR1 site shape: each site nested in its own containment VM (site-down/recovery + + site-to-site drills); dark fiber DC-DC-Office is East-West ONLY; each site has its own dedicated L3 + ISP uplink (build `vr1-dc0-wan` on the office1-wan pattern); DC edge follows the Office1 pattern + (2048MiB/2vCPU/nano, REST-API config, LAN faces the site network); each site its own MAAS controller. + Subsumes the DOCFIX-185 D-100/D-107 amendment note. +- **Fix-forward DONE (2026-07-15):** the phase2 runbook is reconciled to post-D-119 naming + (`vr1-dc0`/`vr1-dc1`, bare `dcN` rejected) and its obsolete Steps 4-5 rewritten to the D-113(a2) + REST-API bootstrap (the module already dropped `config_iso_path`); the D-121/D-122 rulings folded in + so its Ruling-1/Ruling-2 STOP-gates clear. Honest achievable first-run scope stays **Steps 1-8** -- + Step 9 (MAAS registration) gates on the per-DC MAAS "rack controller", now RESOLVED IN PRINCIPLE by + **D-123** (PROPOSED, recommend Model A): it is just a `vvr1-dc0` site headend VM (the proven Office1 + pattern -- `cloudinit-vm` like `voffice1` + `scripts/site-headend-install.sh`, which already names + `vvr1-dc0` as a target). Once Model A is ruled it is prepare-able (opentofu block + Step-9 wiring), + which effectively closes the Stage-4 gap for Step 9. D-123 also reconciles the D-122 "one virsh + destroy = site-down" wording (literal for Office1; for a DC, site-down = destroy the `vr1-dc0-*` + domain group). See `docs/changelog-20260715-d121-d122-stage3-ha-nodelayout.md`. +- **EDGE NIC MODEL RULED (D-122 refinement, 2026-07-15): 2-NIC, baremetal-matched.** NIC count mirrors + the Roosevelt hardware -- **nodes = 6 NICs** (one per plane, measured on VR0 openstack0-3), **edge = 2** + (WAN + LAN). The six planes are **routed by the fabric** (OVN/OpenStack on the deliberately isolated-L2 + `dc-planes` segments), so the edge is NOT an inter-plane router -- only the external boundary. No + `opnsense-edge` module extension needed. Edge **WAN = `vr1-dc0-wan`** (`172.30.2.0/24`), edge + **LAN = provider-public** (`10.12.4.0/22`; edge is the external gateway per D-100). This SUPERSEDES the + earlier "build an office1-network-analog site LAN" flag -- no new site-internal segment is needed. +- **STAGE-3 PREP BATCH DONE 2026-07-15 (authoring + offline validation; NOTHING applied).** See + `docs/changelog-20260715-stage3-prereqs-prep.md`. Gauntlet ALL GREEN (61), `tofu validate` Success, + repo-lint 0-fail. + - **OpenTofu:** new `modules/site-wan` (NAT /24 uplink; schema confirmed via `tofu providers schema`, + validated) + `main.tf` Stage-3 section wired (NOT APPLIED): `vr1_dc0_wan` (172.30.2.0/24), the 2-NIC + edge (LAN=provider-public, WAN=uplink), 8 Option C node VMs via `for_each` (metal-admin PXE-first), + netem HELD (Step 11; mesh bridge = virbr7 measured), stale `vr1_dc1_planes` comment fixed. **HELD + pending D-123:** the `vvr1-dc0` headend + `maas-vm-host` (Step 9) blocks. + - **NetBox:** `netbox/dc-edge-wan-import.py` + harness (58/58) -- registers the two DC-edge /24s + (172.30.2/3.0/24, role edge, site-scoped), DRY-BY-DEFAULT with whole-plan preflight. READY to + `--commit` (command in the tool header; token operator-held; guard blocked the live read). + - **Bundle HA overlay:** `overlays/dc-ha-scaleup.yaml` (Stage-5 prep) -- 14 services ->3, one-per- + control-node placement (`{{VR1_DC0_CONTROL_0N}}` tokens, Stage-4-bind), + 3 correctness items the + agent caught: rabbitmq `min-cluster-size:3`, hacluster `cluster_count:3`, re-declare `vault-hacluster` + (BUNDLEFIX-002 had commented it out) + re-add `vault:ha`. + - **The failed opentofu agent** (transient API stall, wrote nothing) was covered in-session by the main + loop; the netbox + overlay agents completed clean. +- **D-123/D-124 RULED + `vvr1-dc0` RACK WIRED (2026-07-16).** D-123 Model A + MAAS region-on-Office1 + + rack-per-DC; D-124 Scheme A (office1<->dc0 mesh transit /30) + rack sizing 4/8192/80. `main.tf` now + wires **`module "vvr1_dc0"`** (cloudinit-vm, 4/8192/80, two legs: metal-admin + mesh-office1 transit, + static IPs from NEW tfvars `vr1_dc0_rack_*` -- NetBox-assigned, not invented; interface-naming trap + flagged) + a `mesh-link` `network_name` output. `tofu validate` Success; opentofu-validate 11/11. + **Step-9 `maas-vm-host` STAYS DEFERRED (DOCFIX-179):** adding the `provider "maas"` block would force + every plan (incl. the Steps 1-8 apply) to demand MAAS creds, and vr1-dc0's MAAS doesn't exist until + vvr1-dc0 is up -- wire it as its own step once the rack MAAS is reachable. The rack MAAS INSTALL needs + a rack-only mode of `site-headend-install.sh` (follow-up). +- **D-124 rack-mgmt importer BUILT (2026-07-16):** `netbox/dc-rack-mgmt-import.py` + harness (96/96), + parameterized (`--transit-cidr`/`--rack-ip`, no invented literal), whole-plan preflight. Gauntlet 62. + **Two operator preconditions it flags before `--commit`:** (a) SEED a dedicated `transit` role in + office1-netbox (none exists; `infra` is shared, not reused) -- die-if-absent, like `edge`; (b) DECIDE + the transit supernet -- the importer assumes it nests under `10.12.0.0/16` (Cloud), but D-124 mirrors + the Edge role (which got its OWN `172.30.0.0/16`), so a dedicated mgmt/transit supernet may be wanted + (then a one-line `CONTAINER` const update). Transit prefix scope = site vr1-dc0 (region may be preferred). +- **RESIDUAL for a smooth batch deploy:** (1) NetBox: seed the `transit` role + decide the transit + supernet, then run the D-124 importer (+ the D-115 DC-edge importer) `--commit` with the assigned CIDR/ + IP, then fill the `vr1_dc0_rack_*` tfvars; (2) runtime: `opnsense-prep-image.sh` for the edge nano; + (3) Step-9 maas-vm-host wiring once the rack MAAS is up (**the rack-only `site-headend-install.sh --role + rack` mode is now DONE, 32/32** -- it enrolls a DC rack to Office1's region; `maas init rack` confirmed); + (4) Stage-4: render the HA overlay's control-node machine IDs + `--overlay --dry-run`; (5) D-121 + verifies (vault MySQL-HA at 3; hacluster cluster_count). All operator-gated / Stage-4. +- **`main.tf` stale comment (flagged):** the commented `vr1_dc1_planes` block (~L38-50) still says + "wait for NetBox to assign D-101's supernet" -- STALE per D-115 (assigned `10.12.64.0/19`). Doc-only + pass did not touch `main.tf`; the `main.tf` owner should correct the comment. +- **STALE gate corrected:** the runbook + `main.tf` say `vr1-dc1` is blocked on "NetBox assigning the + supernet" -- D-115 ALREADY assigned it (`10.12.64.0/19`, imported to office1-netbox). vr1-dc1 stays + out of scope this pass by SEQUENCING, not a missing literal. Do NOT uncomment `vr1_dc1_planes`. +- **Logged, NOT executed (adjacent improvement, hard rule 1):** promote `scratchpad/optc-calc.py` into + a committed whole-host capacity calculator with a harness (sibling to `dc-dc-ceph-disk-budget.sh`). + The authoritative per-stage tracker is `docs/dc-dc-deployment-workflow.md` (stage table + tooling gap register). Read it FIRST. This list is only what that doc does not already carry. diff --git a/netbox/dc-edge-wan-import.py b/netbox/dc-edge-wan-import.py new file mode 100644 index 0000000..d8c4033 --- /dev/null +++ b/netbox/dc-edge-wan-import.py @@ -0,0 +1,184 @@ +#!/usr/bin/env python3 +""" +Register the VR1 DC-edge simulated-ISP WAN /24s in office1-netbox (the VR1 apex). + +D-115 (ADOPTED) carved a v4 Edge role at 172.30.0.0/16 and blessed office1-wan as +172.30.1.0/24 (site vr1-off1). The two DC edges follow the same role, one /24 each: + + 172.30.2.0/24 role edge, active scope=dcim.site:vr1-dc0 (vr1-dc0 sim-ISP WAN) + 172.30.3.0/24 role edge, active scope=dcim.site:vr1-dc1 (vr1-dc1 sim-ISP WAN) + +These were NOT in the D-115 office carve (which only owned the office row) and no +prior import covered the DC-edge /24s -- so this tool registers them, and nothing +else. It creates NEITHER the Edge role NOR the 172.30.0.0/16 container NOR the DC +sites: all three are PRECONDITIONS (d115-office-carve.py owns the role + container; +the vr1-dc0/vr1-dc1 sites already exist in the apex). A missing precondition is a +hard STOP, never a silent create. + +DRY BY DEFAULT -- nothing is written without --commit. + +WRITING UPSTREAM IS GATED IN CODE. SANDBOX_HOSTS is the allowlist; a --commit at +any other host REFUSES without --yes-write-upstream. WHOLE-PLAN PREFLIGHT: the role, +the container, the in-container placement of every /24, and every target site are all +checked BEFORE any create, so a bad plan cannot half-write the apex (the failure that +bit roles-aggregates-import.py -- it created 4 objects then died on the 5th). + +Usage (on office1-netbox / through a tunnel, with the sandbox token): + NETBOX_URL=http://10.10.1.10:8000 NETBOX_TOKEN= python3 netbox/dc-edge-wan-import.py + ... same, add --commit, to write. +""" +import argparse +import ipaddress +import json +import os +import sys +import urllib.error +import urllib.parse +import urllib.request + +UA = "curl/8.5.0" # upstream 403s the default python UA -- see platform-traps.md + +# A sandbox is local, or the known Office1 sandbox address. Anything else is treated +# as the production apex and requires the explicit upstream flag. +SANDBOX_HOSTS = {"localhost", "127.0.0.1", "10.10.1.10"} + +ROLE_SLUG = "edge" # created by d115-office-carve.py; a precondition here +CONTAINER = "172.30.0.0/16" # Edge role container; a precondition here +STATUS = "active" # a real segment on the wire, like office1-wan .1/24 + +# (cidr, target-site slug, description). Site-scoped exactly like office1-wan +# 172.30.1.0/24 -> vr1-off1 (d115-office-carve.py). The DC sites already exist in the +# apex draft (vr1-dc0 / vr1-dc1); this tool resolves them, it never creates them. +TARGETS = [ + ("172.30.2.0/24", "vr1-dc0", "vr1-dc0 simulated-ISP edge WAN (D-115)"), + ("172.30.3.0/24", "vr1-dc1", "vr1-dc1 simulated-ISP edge WAN (D-115)"), +] + + +def die(msg: str): + print(f"FAIL: {msg}", file=sys.stderr) + sys.exit(2) + + +class NB: + """Stdlib NetBox client -- same shape as the other sandbox-loop tools; UA-aware so + it is not 403'd by the upstream User-Agent filter.""" + def __init__(self, base, token): + self.base = base.rstrip("/") + self.token = token + + def _req(self, method, path, body=None): + data = json.dumps(body).encode() if body is not None else None + req = urllib.request.Request(f"{self.base}/api/{path}", data=data, method=method, + headers={"Authorization": f"Token {self.token}", + "Accept": "application/json", + "Content-Type": "application/json", + "User-Agent": UA}) + try: + with urllib.request.urlopen(req, timeout=45) as r: + return json.load(r) if r.status != 204 else None + except urllib.error.HTTPError as exc: + detail = exc.read().decode(errors="replace")[:300] + if exc.code == 403 and "v1 token" in detail: + die("403 'Invalid v1 token' -- NetBox 4.6 wants the ASSEMBLED v2 token " + "nbt_., not the API's bare `token` field.") + if exc.code == 403: + die(f"403 on {path}. If curl works with this token, it is the upstream " + f"User-Agent filter, NOT the token (references/platform-traps.md).") + die(f"HTTP {exc.code} {method} {path}: {detail}") + + def one(self, path, **flt): + res = self._req("GET", f"{path}/?{urllib.parse.urlencode(flt)}&limit=1") + return res["results"][0] if res["results"] else None + + def create(self, path, payload): + return self._req("POST", f"{path}/", payload) + + +def get_nb(base, token): + """Client factory -- the injection seam the harness overrides to drive main() + against an in-memory fake without a live NetBox.""" + return NB(base, token) + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__.split("\n\n", 1)[0]) + ap.add_argument("--commit", action="store_true", + help="WRITE. Default is a DRY RUN that writes nothing.") + ap.add_argument("--yes-write-upstream", action="store_true", + help="Required (with --commit) to write to a NON-sandbox NetBox.") + args = ap.parse_args() + + url = os.environ.get("NETBOX_URL") + token = os.environ.get("NETBOX_TOKEN") + if not url or not token: + die("NETBOX_URL and NETBOX_TOKEN must be set.") + + host = (urllib.parse.urlparse(url).hostname or url).lower() + is_sandbox = host in SANDBOX_HOSTS + print(f"Target : {url} ({'SANDBOX' if is_sandbox else 'NOT a known sandbox'})") + if args.commit and not is_sandbox and not args.yes_write_upstream: + die(f"REFUSING to --commit to '{host}': not a known sandbox, so treated as the " + f"PRODUCTION apex. Re-run with --yes-write-upstream if that is intended.") + + print("\n*** DRY RUN -- nothing will be written. Re-run with --commit. ***" + if not args.commit else "\n*** COMMITTING. ***") + + nb = get_nb(url, token) + + # WHOLE-PLAN PREFLIGHT -- validate the ENTIRE plan (role, container, in-container + # placement, every target site) BEFORE any create, so a later-target failure (e.g. + # a missing vr1-dc1 site) cannot leave vr1-dc0 written and the apex half-populated. + container = ipaddress.ip_network(CONTAINER) + for cidr, _site_slug, _desc in TARGETS: + if not ipaddress.ip_network(cidr).subnet_of(container): + die(f"{cidr} is outside the Edge container {CONTAINER} -- refusing to place it.") + + role = nb.one("ipam/roles", slug=ROLE_SLUG) + if role is None: + die(f"role '{ROLE_SLUG}' absent -- seed the D-115 carve first " + f"(netbox/d115-office-carve.py). Refusing to place edge /24s with no role.") + + if nb.one("ipam/prefixes", prefix=CONTAINER) is None: + die(f"Edge container {CONTAINER} absent -- seed the D-115 carve first " + f"(netbox/d115-office-carve.py). Refusing to place /24s in an unallocated /16.") + + # Resolve every site and its already-present state up front -- THIS is the whole-plan + # gate: if any site is missing the run dies here, before the create loop below runs. + plan = [] + for cidr, site_slug, desc in TARGETS: + site = nb.one("dcim/sites", slug=site_slug) + if site is None: + die(f"site '{site_slug}' absent -- cannot scope {cidr}. The DC sites are a " + f"precondition (they already exist in the apex); this tool never creates them.") + present = nb.one("ipam/prefixes", prefix=cidr) is not None + plan.append((cidr, site, desc, present)) + + created = existing = 0 + + print("\nEdge DC /24s:") + for cidr, site, desc, present in plan: + if present: + print(f" EXISTS {cidr}") + existing += 1 + continue + if not args.commit: + print(f" [dry-run] would CREATE {cidr} role={ROLE_SLUG} " + f"scope=dcim.site:{site['slug']}") + created += 1 + continue + payload = {"prefix": cidr, "role": role["id"], "status": STATUS, + "description": desc, "scope_type": "dcim.site", "scope_id": site["id"]} + o = nb.create("ipam/prefixes", payload) + print(f" CREATED {cidr} (id={o['id']}) role={ROLE_SLUG} scope={site['slug']}") + created += 1 + + verb = "would create" if not args.commit else "created" + print(f"\n{'='*66}\n{verb}: {created} already present: {existing}") + if not args.commit: + print("DRY RUN -- nothing was written. Re-run with --commit.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/netbox/dc-rack-mgmt-import.py b/netbox/dc-rack-mgmt-import.py new file mode 100644 index 0000000..b0d41d1 --- /dev/null +++ b/netbox/dc-rack-mgmt-import.py @@ -0,0 +1,300 @@ +#!/usr/bin/env python3 +""" +Register the D-124 Office1-region <-> DC-rack MANAGEMENT TRANSIT addressing in +office1-netbox (the VR1 IPAM apex): the point-to-point transit on the office1<->dc0 +mesh leg, plus the rack's metal-admin static IP. Two objects, nothing else. + +D-124 (ADOPTED 2026-07-16, Scheme A -- transit-numbered mesh) rules that the +region<->rack MAAS control path rides a small point-to-point transit on the +office1<->dc0 mesh leg (NOT metal-admin, which D-100 keeps DC-local and node-facing). +The rack (vvr1-dc0) STRADDLES both legs: the transit (region-facing) and metal-admin +(node-facing, 10.12.8.0/22). So this tool registers exactly: + + 1. the transit prefix role=transit, scope=dcim.site:vr1-dc0 (the /30 or /31) + 2. the rack ip-address <rack-ip>/22 in metal-admin, D-120 static band .2-.49 + +Both LITERALS ARE OPERATOR INPUTS (--transit-cidr / --rack-ip): D-124, like D-115/ +D-117, rules the SCHEME and leaves the actual CIDR + IP to office1-netbox (the apex; +operator-held token, this loop cannot query it live). NO literal is invented in-repo. + +This tool creates NEITHER the transit ROLE nor any container/site: all are +PRECONDITIONS (a missing one is a hard STOP, never a silent create) -- exactly as +dc-edge-wan-import.py treats the `edge` role + the 172.30.0.0/16 container. + +DRY BY DEFAULT -- nothing is written without --commit. + +WRITING UPSTREAM IS GATED IN CODE. SANDBOX_HOSTS is the allowlist; a --commit at any +other host REFUSES without --yes-write-upstream. WHOLE-PLAN PREFLIGHT: the transit +role, the container the transit nests under, the transit CIDR shape+placement, the +rack IP's band (metal-admin, static .2-.49, NOT the .1 gateway), and the target site +are ALL checked BEFORE any create -- so a bad plan (e.g. a rack IP that is the .1 +gateway) cannot leave the transit prefix written and the apex half-populated (the +failure that bit roles-aggregates-import.py: 4 objects created, then it died on the 5th). + +SCOPING / ROLE choices flagged for the operator (see the D-124 return notes): + * ROLE_SLUG = "transit": D-124 calls for the transit's OWN role, "mirroring the + D-115 Edge role" (a DEDICATED role). An `infra` role exists in the apex whose + description reads "ptp transit, device loopbacks, RR loopbacks, anycast" -- it is + NOT reused here because it is explicitly SHARED infrastructure, not the dedicated + per-purpose role D-124 asks for. Operator to seed/confirm the `transit` slug in + office1-netbox (as d115-office-carve.py seeded `edge`). A missing role dies here. + * CONTAINER = "10.12.0.0/16" (Cloud): the transit CIDR must be subnet_of an EXISTING + container. The apex's only IPv4 container the transit plausibly nests under is Cloud + 10.12.0.0/16 (ipam/aggregates is empty; no dedicated management/transit supernet + exists yet). This gates ACCEPTANCE only -- NetBox auto-nests by CIDR, we set no + parent. If the operator numbers the transit outside Cloud, the tool DIES rather + than misplacing it. Update CONTAINER if a dedicated transit supernet is carved. + * transit prefix SCOPE = dcim.site:vr1-dc0. How an office1<->dc0 link "should" scope + (region vs. site) is unruled; site-scoping to vr1-dc0 mirrors the D-115 DC-edge + /24s (172.30.2.0/24 -> vr1-dc0). FLAGGED -- operator may prefer a region scope. + * The rack IP band (metal-admin 10.12.8.0/22, static .2-.49, gw .1) is enforced + ARITHMETICALLY (D-120/D-124), NOT via a prefix-object lookup: metal-admin is not + registered as a /22 prefix in the apex draft, and the band is a convention, not a + container. This asymmetry with the transit check is intentional. + +Usage (on office1-netbox / through a tunnel, with the sandbox token): + NETBOX_URL=http://10.10.1.10:8000 NETBOX_TOKEN=<tok> \ + python3 netbox/dc-rack-mgmt-import.py --transit-cidr <A.B.C.D/30> --rack-ip <10.12.8.X> + ... same, add --commit, to write. +""" +import argparse +import ipaddress +import json +import os +import sys +import urllib.error +import urllib.parse +import urllib.request + +UA = "curl/8.5.0" # upstream 403s the default python UA -- see platform-traps.md + +# A sandbox is local, or the known Office1 sandbox address. Anything else is treated +# as the production apex and requires the explicit upstream flag. +SANDBOX_HOSTS = {"localhost", "127.0.0.1", "10.10.1.10"} + +ROLE_SLUG = "transit" # the transit's OWN role (D-124); a precondition here +CONTAINER = "10.12.0.0/16" # Cloud; the transit CIDR must be subnet_of this (a precondition) +SITE_SLUG = "vr1-dc0" # the transit prefix is site-scoped here (mirrors the D-115 edge /24s) +STATUS = "active" # a real segment on the wire + +# metal-admin band arithmetic (D-052/D-124). The rack's metal-admin IP lands in the +# D-120 static band .2-.49 of this /22, and NOT the .1 gateway. +METAL_ADMIN = "10.12.8.0/22" +STATIC_BAND_LOW = 2 # .2 (first static site-services address, D-120) +STATIC_BAND_HIGH = 49 # .49 (last static site-services address, D-120) + +RACK_DNS = "vvr1-dc0" # the rack-controller VM (D-124 cloudinit-vm on the two legs) +TRANSIT_DESC = "office1<->dc0 management transit -- region<->rack MAAS control path (D-124 Scheme A)" +RACK_DESC = "vr1-dc0 MAAS rack controller (vvr1-dc0) metal-admin static IP (D-124; D-120 static band)" + + +def die(msg: str): + print(f"FAIL: {msg}", file=sys.stderr) + sys.exit(2) + + +class NB: + """Stdlib NetBox client -- same shape as the other sandbox-loop tools; UA-aware so + it is not 403'd by the upstream User-Agent filter.""" + def __init__(self, base, token): + self.base = base.rstrip("/") + self.token = token + + def _req(self, method, path, body=None): + data = json.dumps(body).encode() if body is not None else None + req = urllib.request.Request(f"{self.base}/api/{path}", data=data, method=method, + headers={"Authorization": f"Token {self.token}", + "Accept": "application/json", + "Content-Type": "application/json", + "User-Agent": UA}) + try: + with urllib.request.urlopen(req, timeout=45) as r: + return json.load(r) if r.status != 204 else None + except urllib.error.HTTPError as exc: + detail = exc.read().decode(errors="replace")[:300] + if exc.code == 403 and "v1 token" in detail: + die("403 'Invalid v1 token' -- NetBox 4.6 wants the ASSEMBLED v2 token " + "nbt_<key>.<plaintext>, not the API's bare `token` field.") + if exc.code == 403: + die(f"403 on {path}. If curl works with this token, it is the upstream " + f"User-Agent filter, NOT the token (references/platform-traps.md).") + die(f"HTTP {exc.code} {method} {path}: {detail}") + + def one(self, path, **flt): + res = self._req("GET", f"{path}/?{urllib.parse.urlencode(flt)}&limit=1") + return res["results"][0] if res["results"] else None + + def create(self, path, payload): + return self._req("POST", f"{path}/", payload) + + +def get_nb(base, token): + """Client factory -- the injection seam the harness overrides to drive main() + against an in-memory fake without a live NetBox.""" + return NB(base, token) + + +def parse_transit_cidr(raw: str) -> ipaddress.IPv4Network: + """The transit CIDR must be a valid /30 or /31 network (host bits clear), subnet_of + CONTAINER. Shape is validated here; containment against CONTAINER in preflight.""" + try: + net = ipaddress.ip_network(raw, strict=True) + except ValueError as exc: + die(f"--transit-cidr {raw!r} is not a valid network (host bits set?): {exc}") + if net.version != 4: + die(f"--transit-cidr {raw} must be IPv4 (the office1<->dc0 transit is v4).") + if net.prefixlen not in (30, 31): + die(f"--transit-cidr {raw} must be a /30 or /31 point-to-point (got /{net.prefixlen}).") + return net + + +def parse_rack_ip(raw: str) -> ipaddress.IPv4Address: + """Accept a bare host (10.12.8.X) or a masked form; if masked, the mask MUST be the + metal-admin /22. Returns the host address; band placement is checked in preflight.""" + try: + if "/" in raw: + iface = ipaddress.ip_interface(raw) + if iface.network.prefixlen != 22: + die(f"--rack-ip {raw} carries /{iface.network.prefixlen}; metal-admin is a /22. " + f"Pass a bare host (10.12.8.X) or the /22 form.") + host = iface.ip + else: + host = ipaddress.ip_address(raw) + except ValueError as exc: + die(f"--rack-ip {raw!r} is not a valid IPv4 address: {exc}") + if host.version != 4: + die(f"--rack-ip {raw} must be IPv4 (metal-admin is v4).") + return host + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__.split("\n\n", 1)[0]) + ap.add_argument("--transit-cidr", default=os.environ.get("TRANSIT_CIDR"), + help="REQUIRED. The /30 or /31 point-to-point for the office1<->dc0 " + "transit leg (operator-supplied, NetBox-assigned; env: TRANSIT_CIDR).") + ap.add_argument("--rack-ip", default=os.environ.get("RACK_IP"), + help="REQUIRED. The rack's metal-admin static IP within 10.12.8.0/22, " + "static band .2-.49 (operator-supplied; env: RACK_IP).") + ap.add_argument("--commit", action="store_true", + help="WRITE. Default is a DRY RUN that writes nothing.") + ap.add_argument("--yes-write-upstream", action="store_true", + help="Required (with --commit) to write to a NON-sandbox NetBox.") + args = ap.parse_args() + + url = os.environ.get("NETBOX_URL") + token = os.environ.get("NETBOX_TOKEN") + if not url or not token: + die("NETBOX_URL and NETBOX_TOKEN must be set.") + + # Inputs are args-or-env, so they cannot be argparse required=True (that would break + # the env fallback). Hand-roll the missing checks, each with its own die. + if not args.transit_cidr: + die("--transit-cidr (or TRANSIT_CIDR) is REQUIRED -- the NetBox-assigned transit " + "/30 or /31. No literal is invented in-repo (D-124).") + if not args.rack_ip: + die("--rack-ip (or RACK_IP) is REQUIRED -- the rack's NetBox-assigned metal-admin " + "static IP. No literal is invented in-repo (D-124).") + + host = (urllib.parse.urlparse(url).hostname or url).lower() + is_sandbox = host in SANDBOX_HOSTS + print(f"Target : {url} ({'SANDBOX' if is_sandbox else 'NOT a known sandbox'})") + if args.commit and not is_sandbox and not args.yes_write_upstream: + die(f"REFUSING to --commit to '{host}': not a known sandbox, so treated as the " + f"PRODUCTION apex. Re-run with --yes-write-upstream if that is intended.") + + print("\n*** DRY RUN -- nothing will be written. Re-run with --commit. ***" + if not args.commit else "\n*** COMMITTING. ***") + + # ---- WHOLE-PLAN PREFLIGHT: validate the ENTIRE plan (both objects) BEFORE any + # create, so a bad rack IP (e.g. the .1 gateway) cannot leave the transit written. ---- + + # (a) Transit CIDR shape + containment (local, no NetBox needed). + transit = parse_transit_cidr(args.transit_cidr) + container = ipaddress.ip_network(CONTAINER) + if not transit.subnet_of(container): + die(f"transit {transit} is outside the container {CONTAINER} (Cloud) -- refusing to " + f"place it. If the transit is carved elsewhere, that is an operator/D-124 call.") + + # (b) Rack IP band: within metal-admin /22, in the D-120 static band .2-.49, NOT the + # .1 gateway (a distinct die so the ".1 rejected" property is unambiguous). + rack = parse_rack_ip(args.rack_ip) + metal = ipaddress.ip_network(METAL_ADMIN) + if rack not in metal: + die(f"rack IP {rack} is outside metal-admin {METAL_ADMIN} -- refusing to place it.") + gateway = metal.network_address + 1 + band_low = metal.network_address + STATIC_BAND_LOW + band_high = metal.network_address + STATIC_BAND_HIGH + if rack == gateway: + die(f"rack IP {rack} is the .1 GATEWAY of {METAL_ADMIN} -- refusing (D-120: .1 is the " + f"site gateway, not a static-service address).") + if not (band_low <= rack <= band_high): + die(f"rack IP {rack} is outside the D-120 static band {band_low}-{band_high} of " + f"{METAL_ADMIN} -- the rack's metal-admin IP must land in .2-.49 (not the dynamic/" + f"node bands).") + rack_addr = f"{rack}/{metal.prefixlen}" # NetBox stores the host WITH the plane mask + + nb = get_nb(url, token) + + # (c) Preconditions in the apex: the transit role, the container prefix, the site. + role = nb.one("ipam/roles", slug=ROLE_SLUG) + if role is None: + die(f"role '{ROLE_SLUG}' absent -- D-124 calls for the transit's OWN dedicated role " + f"(mirroring the D-115 `edge` role); seed it in office1-netbox first. Refusing to " + f"place the transit prefix with no role.") + + if nb.one("ipam/prefixes", prefix=CONTAINER) is None: + die(f"container {CONTAINER} (Cloud) absent in the apex -- refusing to place the transit " + f"in an unallocated supernet.") + + site = nb.one("dcim/sites", slug=SITE_SLUG) + if site is None: + die(f"site '{SITE_SLUG}' absent -- cannot scope the transit prefix. The DC site is a " + f"precondition (it already exists in the apex); this tool never creates it.") + + # (d) Present-state (idempotency) -- resolved up front, part of the whole-plan gate. + transit_str = str(transit) + transit_present = nb.one("ipam/prefixes", prefix=transit_str) is not None + rack_present = nb.one("ipam/ip-addresses", address=rack_addr) is not None + + created = existing = 0 + + # ---- CREATE (transit prefix, then rack ip-address) -- only reached once the WHOLE + # plan above is viable, so this loop cannot half-write the apex. ---- + print("\nTransit prefix (office1<->dc0 mesh leg):") + if transit_present: + print(f" EXISTS {transit_str}") + existing += 1 + elif not args.commit: + print(f" [dry-run] would CREATE {transit_str} role={ROLE_SLUG} " + f"scope=dcim.site:{SITE_SLUG}") + created += 1 + else: + payload = {"prefix": transit_str, "role": role["id"], "status": STATUS, + "description": TRANSIT_DESC, "scope_type": "dcim.site", "scope_id": site["id"]} + o = nb.create("ipam/prefixes", payload) + print(f" CREATED {transit_str} (id={o['id']}) role={ROLE_SLUG} scope={SITE_SLUG}") + created += 1 + + print("\nRack metal-admin static IP:") + if rack_present: + print(f" EXISTS {rack_addr}") + existing += 1 + elif not args.commit: + print(f" [dry-run] would CREATE {rack_addr} dns={RACK_DNS} (metal-admin static band)") + created += 1 + else: + payload = {"address": rack_addr, "status": STATUS, + "dns_name": RACK_DNS, "description": RACK_DESC} + o = nb.create("ipam/ip-addresses", payload) + print(f" CREATED {rack_addr} (id={o['id']}) dns={RACK_DNS}") + created += 1 + + verb = "would create" if not args.commit else "created" + print(f"\n{'='*66}\n{verb}: {created} already present: {existing}") + if not args.commit: + print("DRY RUN -- nothing was written. Re-run with --commit.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/opentofu/main.tf b/opentofu/main.tf index f75981d..48eb444 100644 --- a/opentofu/main.tf +++ b/opentofu/main.tf @@ -35,11 +35,14 @@ module "vr1_dc0_storage" { target_path = var.vr1_dc0_pool_path } -# ---- DC2: PLANES deferred (operator ruling 2026-07-10, Option B) -- wait for -# NetBox to assign D-101's supernet/ULA/GUA before wiring vr1_dc1_planes. Do NOT -# fill in guessed CIDRs to make it "work" sooner. When NetBox assigns it, add a -# `vr1_dc1_planes` variable (same shape as `vr1_dc0_planes`) and uncomment this block. -# The storage pool + mesh legs are NOT address-dependent, so they ARE wired now. +# ---- vr1-dc1 (VR1's SECOND DC): PLANES deferred by SEQUENCING, not a missing +# literal. Its supernet IS assigned -- D-115 moved it to 10.12.64.0/19 (ADOPTED, +# imported to office1-netbox), superseding the old "wait for NetBox" premise. It +# stays out of scope until vr1-dc0 completes (operator ruling 2026-07-10, Option B: +# vr1-dc0-first). When vr1-dc1 is started, add a `vr1_dc1_planes` variable (same +# shape as `vr1_dc0_planes`, CIDRs derived per D-115) and uncomment this block. +# Do NOT uncomment it now. The storage pool + mesh legs are NOT address-dependent, +# so they ARE wired now. # # module "vr1_dc1_planes" { # source = "./modules/dc-planes" @@ -266,3 +269,171 @@ moved { from = module.mesh_dc2_office1 to = module.mesh_vr1_dc1_office1 } + +# ===================================================================== +# STAGE 3 -- vr1-dc0 substrate (runbooks/dc-dc-phase2-tofu-dc-substrate.md). +# Authored 2026-07-15. NOT YET APPLIED -- every block below is a gated Stage-3 +# mutation; `tofu apply` is the operator's step (Steps 8/11 of the runbook). +# Governing rulings: D-100 (edge/fabric), D-121 (Option C node layout + sizing), +# D-122 (2-NIC fabric-routed edge; dedicated per-site ISP uplink). +# +# D-123 Model A + MAAS model RULED (region on Office1 + rack per DC) + D-124 +# ADOPTED (Scheme A: office1<->dc0 mesh transit; rack sizing 4/8192/80). The +# `vvr1-dc0` rack controller IS wired below -- its two static IPs come from +# office1-netbox via variables (tfvars), NOT invented here. The Step-9 +# `maas-vm-host` registration is STILL DEFERRED (DOCFIX-179): adding the +# `provider "maas"` block it needs would force EVERY plan (incl. the Steps 1-8 +# substrate apply) to demand MAAS creds, and vr1-dc0's MAAS does not exist until +# vvr1-dc0 is up and its rack is installed. Wire maas-vm-host + the maas provider +# as its own step once the rack MAAS is reachable (runbook Step 9). The blocks +# below are the D-123-independent substrate (D-103/D-114 as-built) plus the rack. +# ===================================================================== + +# D-122: the dedicated per-site simulated-ISP uplink (172.30.2.0/24, D-115). +module "vr1_dc0_wan" { + source = "./modules/site-wan" + network_name = "vr1-dc0-wan" + cidr = "172.30.2.0/24" + # mtu defaults to 1500 (ISP-uplink domain; NOT the jumbo planes/mesh). +} + +# D-122: the DC edge -- 2-NIC (WAN + LAN), Office1 pattern (2048/2/nano). The six +# planes are fabric-routed (OVN/OpenStack), so the edge is only the external +# boundary: LAN = provider-public (edge is its external gateway, D-100), WAN = +# the dedicated uplink above. +module "vr1_dc0_opnsense" { + source = "./modules/opnsense-edge" + vm_name = "vr1-dc0-opnsense" + memory_mib = 2048 + vcpu = 2 + pool_name = module.vr1_dc0_storage.pool_name + # No disk_size_bytes: nano direct-copy, sized by opnsense-prep-image.sh (DOCFIX-189). + # PREREQUISITE (runbook Step 4): the prepped nano image must exist at this path. + base_volume_path = "/var/lib/libvirt/vr1/vr1-dc0/opnsense-26.1-nano.qcow2" + lan_network_name = module.vr1_dc0_planes.network_names["provider-public"] + wan_network_name = module.vr1_dc0_wan.network_name +} + +# D-121 Option C node layout: 3 control + 2 compute + 3 storage (16/12/8 vCPU, +# 64/48/24 GiB, 150/100/550 GiB). Six NICs each (one per plane, baremetal-matched +# D-122); metal-admin FIRST = the PXE/boot plane (D-052 default binding). +locals { + vr1_dc0_node_nics = [ + module.vr1_dc0_planes.network_names["metal-admin"], # PXE / boot fabric first + module.vr1_dc0_planes.network_names["provider-public"], + module.vr1_dc0_planes.network_names["metal-internal"], + module.vr1_dc0_planes.network_names["data-tenant"], + module.vr1_dc0_planes.network_names["storage"], + module.vr1_dc0_planes.network_names["replication"], + ] + + vr1_dc0_nodes = { + "vr1-dc0-control-01" = { vcpu = 16, mem = 65536, disk_gib = 150 } + "vr1-dc0-control-02" = { vcpu = 16, mem = 65536, disk_gib = 150 } + "vr1-dc0-control-03" = { vcpu = 16, mem = 65536, disk_gib = 150 } + "vr1-dc0-compute-01" = { vcpu = 12, mem = 49152, disk_gib = 100 } + "vr1-dc0-compute-02" = { vcpu = 12, mem = 49152, disk_gib = 100 } + "vr1-dc0-storage-01" = { vcpu = 8, mem = 24576, disk_gib = 550 } + "vr1-dc0-storage-02" = { vcpu = 8, mem = 24576, disk_gib = 550 } + "vr1-dc0-storage-03" = { vcpu = 8, mem = 24576, disk_gib = 550 } + } +} + +module "vr1_dc0_node" { + for_each = local.vr1_dc0_nodes + + source = "./modules/node-vm" + vm_name = each.key + vcpu = each.value.vcpu + memory_mib = each.value.mem + disk_size_bytes = each.value.disk_gib * 1024 * 1024 * 1024 + pool_name = module.vr1_dc0_storage.pool_name + network_names = local.vr1_dc0_node_nics +} + +# netem (runbook Step 11) -- HELD as a comment. It is a POST-apply step that runs +# SSH-wrapped provisioners from Office1 to the vcloud host, so it needs a real +# `vcloud_host_ssh_target` (a runtime value) and the netem params are an unruled +# D-100 sub-item (gap #11, placeholder only). Measured input ready: the +# vr1-dc0<->office1 mesh leg is bridge `virbr7`. Wire at Step 11, not before. +# module "netem_vr1_dc0_office1" { +# source = "./modules/netem-link" +# link_name = "vr1-dc0-office1" +# bridge_name = "virbr7" # measured 2026-07-15 +# vcloud_host_ssh_target = "<real ssh target -- runtime>" +# netem_args = "<PLACEHOLDER: buildout-design S6 same-metro lean; D-100 gap #11 unruled>" +# } + +# ===================================================================== +# D-123 / D-124: the vr1-dc0 MAAS RACK CONTROLLER (vvr1-dc0). An Ubuntu VM (reusing +# the noble base) that will run a MAAS rack controller enrolled to the Office1 MAAS +# REGION. Two legs: metal-admin (serves PXE/DHCP to the vr1-dc0 nodes) + the +# office1<->dc0 mesh transit (reaches the region, D-124 Scheme A). It PROXIES OS +# images from the region, so it needs no direct DC egress (D-107). NOT YET APPLIED. +# The MAAS rack INSTALL itself is a gated post-boot step (an adapted +# site-headend-install.sh rack-only mode -- follow-up), like voffice1's install. +# ===================================================================== +module "vvr1_dc0" { + source = "./modules/cloudinit-vm" + vm_name = "vvr1-dc0" + vcpu = 4 # D-124 (rack controller: lightweight) + memory_mib = 8192 # D-124 + disk_size_bytes = 80 * 1024 * 1024 * 1024 # D-124: 80 GiB + pool_name = module.vr1_dc0_storage.pool_name + base_volume_path = module.ubuntu_noble_base.path + expose_nested_virt = false # rack-only: the region + its LXD host compose VMs, not the rack + + network_names = [ + module.vr1_dc0_planes.network_names["metal-admin"], # leg 1: node-facing PXE/DHCP + module.mesh_vr1_dc0_office1.network_name, # leg 2: office1<->dc0 transit (region) + ] + + user_data = <<-EOT + #cloud-config + hostname: vvr1-dc0 + fqdn: vvr1-dc0.${var.domain_suffix} + manage_etc_hosts: true + users: + - name: jessea123 + groups: [adm, sudo] + shell: /bin/bash + sudo: "ALL=(ALL) NOPASSWD:ALL" + ssh_authorized_keys: + - ${trimspace(file(var.office1_ssh_pubkey_path))} + package_update: true + packages: + - qemu-guest-agent + runcmd: + - [systemctl, enable, --now, qemu-guest-agent] + EOT + + meta_data = <<-EOT + instance-id: vvr1-dc0-d123 + local-hostname: vvr1-dc0 + EOT + + # STATIC on both legs (planes + mesh have NO DHCP). IPs are NetBox-assigned + # (D-124), supplied via tfvars -- no literal here. INTERFACE-NAMING TRAP: the + # guest NIC names (enp1s0/enp2s0 vs ens3/...) depend on machine type and are NOT + # known until first boot -- CONFIRM on boot and adjust `match` before relying on + # this (documented trap; modules/opnsense-edge + the runbook). metal-admin gives + # the rack no default route; the region route rides the transit toward Office1 + # (10.10.0.0/22, as-built D-115). + network_config = <<-EOT + version: 2 + ethernets: + admin: + match: + name: "enp1s0" + set-name: admin + addresses: ["${var.vr1_dc0_rack_metal_admin_ip}/22"] + mgmt: + match: + name: "enp2s0" + set-name: mgmt + addresses: ["${var.vr1_dc0_rack_transit_ip}/${var.vr1_dc0_rack_transit_prefix}"] + routes: + - to: "10.10.0.0/22" + via: "${var.vr1_dc0_rack_transit_peer_ip}" + EOT +} diff --git a/opentofu/modules/mesh-link/outputs.tf b/opentofu/modules/mesh-link/outputs.tf index 91f8bcc..097e972 100644 --- a/opentofu/modules/mesh-link/outputs.tf +++ b/opentofu/modules/mesh-link/outputs.tf @@ -3,6 +3,11 @@ output "network_id" { value = libvirt_network.link.id } +output "network_name" { + description = "libvirt network name (mesh-<link_name>) -- for a consumer's network_names list (e.g. the vr1-dc0 rack's transit NIC, D-124)." + value = libvirt_network.link.name +} + # A "bridge_name" output (the OS-level bridge interface backing this network, # needed by whatever applies `tc netem` to it) is deliberately NOT included # here: `bridge` is documented as a configurable input block (delay, diff --git a/opentofu/modules/site-wan/main.tf b/opentofu/modules/site-wan/main.tf new file mode 100644 index 0000000..8b7c3c0 --- /dev/null +++ b/opentofu/modules/site-wan/main.tf @@ -0,0 +1,36 @@ +# site-wan: NAT'd per-site ISP uplink (see variables.tf for rationale). +# +# SCHEMA NOTE: this provider (registry.opentofu.org/dmacvicar/libvirt) uses +# attribute-style nested objects, confirmed 2026-07-15 via +# `tofu providers schema -json` (the method dc-planes/main.tf calls for): +# forward = { mode = "nat" } (nesting=single; no top-level `mode`) +# ips = [ { address, prefix } ] (nesting=list) +# mtu = { size = N } (nesting=single; matches dc-planes) +# domain = { name = ... } (nesting=single; matches dc-planes) +# This mirrors the live `office1-wan` (virsh): <forward mode='nat'>, .1/24. +# DHCP is intentionally OMITTED -- the only client is the edge, whose WAN is +# static (office1-opnsense's WAN is .2, outside office1-wan's DHCP range). + +resource "libvirt_network" "site_wan" { + name = var.network_name + autostart = true + + forward = { + mode = "nat" + } + + domain = { + name = var.network_name + } + + mtu = { + size = var.mtu + } + + ips = [ + { + address = cidrhost(var.cidr, 1) + prefix = tonumber(split("/", var.cidr)[1]) + } + ] +} diff --git a/opentofu/modules/site-wan/outputs.tf b/opentofu/modules/site-wan/outputs.tf new file mode 100644 index 0000000..1f260c0 --- /dev/null +++ b/opentofu/modules/site-wan/outputs.tf @@ -0,0 +1,4 @@ +output "network_name" { + description = "The libvirt network name, for a consumer's wan_network_name." + value = libvirt_network.site_wan.name +} diff --git a/opentofu/modules/site-wan/variables.tf b/opentofu/modules/site-wan/variables.tf new file mode 100644 index 0000000..8a11535 --- /dev/null +++ b/opentofu/modules/site-wan/variables.tf @@ -0,0 +1,31 @@ +# site-wan: a per-site simulated-ISP uplink network (D-100 / D-122). NAT'd /24, +# the site edge's WAN leg attaches here. Generalizes the virsh-created +# `office1-wan` (172.30.1.0/24) into a reusable module so every per-site ISP +# uplink is tofu-managed (D-122 / operator 2b, 2026-07-15). +# +# MTU is the ISP-uplink domain: 1500, NOT the jumbo 9000 the planes/mesh use. +# This is a hard distinction -- the simulated ISP link mirrors a real 1500-MTU +# internet uplink (see the two-MTU-domains note in the session ledger). Do not +# raise this to 9000. + +variable "network_name" { + description = "libvirt network name, e.g. \"vr1-dc0-wan\"." + type = string +} + +variable "cidr" { + description = <<-EOT + The uplink /24, e.g. "172.30.2.0/24" (Edge role 172.30.0.0/16, D-115). + The network gateway is the .1 host of this CIDR (cidrhost(cidr, 1)); the + site edge's WAN interface takes a STATIC address on this segment (e.g. .2), + configured on the edge itself via the REST API -- not by DHCP here (the + edge WAN is static, exactly as office1-opnsense's WAN 172.30.1.2 is). + EOT + type = string +} + +variable "mtu" { + description = "Uplink MTU. Default 1500 (the ISP-uplink domain; NEVER jumbo)." + type = number + default = 1500 +} diff --git a/opentofu/modules/site-wan/versions.tf b/opentofu/modules/site-wan/versions.tf new file mode 100644 index 0000000..f50888b --- /dev/null +++ b/opentofu/modules/site-wan/versions.tf @@ -0,0 +1,11 @@ +# Provider source mapping for this module (DOCFIX-179). Child modules MUST +# declare their own required_providers so OpenTofu maps the local name +# `libvirt` to dmacvicar/libvirt; version is pinned once at the ROOT +# (opentofu/versions.tf), child modules declare source only. +terraform { + required_providers { + libvirt = { + source = "dmacvicar/libvirt" + } + } +} diff --git a/opentofu/variables.tf b/opentofu/variables.tf index 25e0aa3..9718811 100644 --- a/opentofu/variables.tf +++ b/opentofu/variables.tf @@ -96,3 +96,28 @@ variable "voffice1_disk_bytes" { description = "voffice1 boot disk size in bytes." type = number } + +# ---- D-124: vr1-dc0 MAAS rack controller (vvr1-dc0) addressing. NO defaults -- +# these come from office1-netbox (the IPAM apex) via the D-124 importer, then +# into a tfvars entry. Do NOT invent them here (hard rule 2). The rack straddles +# two legs: metal-admin (node-facing PXE/DHCP) + the office1<->dc0 mesh transit +# (region-facing, Scheme A). ---- +variable "vr1_dc0_rack_metal_admin_ip" { + description = "vvr1-dc0 rack static IP in metal-admin (10.12.8.0/22), in the .2-.49 static band, not the .1 gateway. NetBox-assigned (D-124)." + type = string +} + +variable "vr1_dc0_rack_transit_ip" { + description = "vvr1-dc0 rack IP on the office1<->dc0 mesh transit (D-124 Scheme A). NetBox-assigned." + type = string +} + +variable "vr1_dc0_rack_transit_prefix" { + description = "Transit link prefix length (30 or 31, D-124 Scheme A)." + type = number +} + +variable "vr1_dc0_rack_transit_peer_ip" { + description = "Office1 region's IP on the transit link -- the rack's next-hop toward the MAAS region (10.10.0.0/22). NetBox-assigned (D-124)." + type = string +} diff --git a/overlays/dc-ha-scaleup.yaml b/overlays/dc-ha-scaleup.yaml new file mode 100644 index 0000000..4262fb9 --- /dev/null +++ b/overlays/dc-ha-scaleup.yaml @@ -0,0 +1,183 @@ +# overlays/dc-ha-scaleup.yaml +# +# D-121 (ADOPTED IN PART, 2026-07-15) VR1 HA scale-up overlay -- makes the +# decorative single-unit control plane REAL by scaling 14 services from +# num_units: 1 to 3. VR1-SPECIFIC: VR0/testcloud stays single-unit BY DESIGN +# (D-009), so this is an overlay, NOT a bundle.yaml edit -- same pattern as +# overlays/dc-dc-ipv6-family-matrix.yaml. Apply ALONGSIDE bundle.yaml's other +# overlays, e.g.: +# juju deploy ./bundle.yaml \ +# --overlay overlays/octavia-pki.yaml \ +# --overlay "overlays/${DC}-hostnames.yaml" \ +# --overlay overlays/dc-ha-scaleup.yaml \ +# -m "${DC_MODEL}" +# +# NOT YET APPLIED to any live model -- authored prep-only (STAGE 5 prep; the +# per-DC bundle deploy is two stages out). No live cloud reached this session. +# +# ===================================================================== +# ##### STAGE-4 BINDING REQUIRED -- READ BEFORE RENDER ##### +# Every `to:` target below is a {{TOKEN}} placeholder for one of the three +# Option C control nodes (vr1-dc0-control-01..03, D-121). The REAL Juju machine +# IDs do NOT exist yet -- they bind at Stage 4, AFTER MAAS enlists the Option C +# node VMs. Do NOT invent concrete `lxd:N` numbers here: a wrong number +# silently mis-places a unit and defeats the anti-affinity that is the entire +# HA value. Render this file (substitute the three tokens with the measured, +# enlisted control-node machine IDs) ONLY at Stage 4/5, exactly as +# dc-dc-ipv6-family-matrix.yaml's VIP tokens are rendered. The rendered +# bundle.yaml `machines:` block MUST define those same three IDs. +# {{VR1_DC0_CONTROL_01}} -> control node 1 machine id +# {{VR1_DC0_CONTROL_02}} -> control node 2 machine id +# {{VR1_DC0_CONTROL_03}} -> control node 3 machine id +# ===================================================================== +# +# PLACEMENT / ANTI-AFFINITY (D-121 balance rule). Juju bundles have NO dedicated +# anti-affinity primitive; one-per-node is expressed POSITIONALLY -- a 3-entry +# `to:` list binds unit0->entry0, unit1->entry1, unit2->entry2. Every scaled app +# below uses the SAME three-control-node spread, so: +# - each app lands exactly one unit per control node (14 apps x 1 = ~14 API +# LXD containers per control node -- the even ~14/node density the 64 GiB +# control-node sizing was validated for, D-121); and +# - the RAM-heavy stateful trio (mysql-innodb-cluster already 3 in the base +# bundle, rabbitmq-server + vault scaled here) auto-balances to one replica +# per control node -- never two stacked on one node. +# +# THREE ITEMS BELOW GO BEYOND A LITERAL `num_units: 3` BUMP. They are NOT +# scope-creep: each is the same class of "mechanical 1->3 hides a gotcha" that +# D-121 itself flags (D-121 caught the placement rewrite; these are the two it +# under-states plus the one its own wording got wrong), and an overlay titled +# "make HA real" that shipped decorative HA would not meet its stated purpose. +# All three are called out in the delivery changelog, not slipped in: +# (A) rabbitmq-server min-cluster-size: 3. Per D-009 AMENDMENT (2026-07-02): +# "rabbitmq scale-up is NOT mechanical without min-cluster-size" -- without +# it the charm accepts client relations BEFORE the cluster forms (same race +# as the D-062 mysql formation failure). Roosevelt/VR1 delta = set it +# alongside num_units: 3. rabbitmq clusters natively (quorum), NO hacluster. +# (B) hacluster cluster_count: 3 (all 12 active hacluster subs + the re-added +# vault-hacluster). The base bundle pins cluster_count: 1 -- explicitly +# "decorative on single-unit testcloud" (D-009 / BUNDLEFIX-003). At 3 units +# with cluster_count still 1, pacemaker can bootstrap the VIP off a single +# node = the SAME decorative behavior D-121 exists to eliminate. 3 makes +# the charm wait for a real 3-node quorum before configuring resources. +# VERIFY at deploy (not fetched this session): confirm the hacluster +# 2.4/stable charm treats cluster_count as "peers required before the +# cluster forms / VIP binds" -- the base bundle explicitly pinned it to 1 +# as decorative, and real 3-node quorum needs 3 regardless of the charm +# default. Flag if `juju config hacluster` disagrees. +# (C) vault-hacluster is DEFINED here, not just related. D-121's prose says it +# "exists among the hacluster subs" -- it does NOT: bundle.yaml line ~923 +# has it COMMENTED OUT (removed by BUNDLEFIX-002 when vault was de-HA'd, +# C1). So the re-added `[vault:ha, vault-hacluster:ha]` relation would +# reference a non-existent app (provider-bundle-check rule 4 would FAIL). +# This overlay re-declares it, matching the other haclusters' shape. +# +# VAULT BACKEND (D-121 sub-ruling = v-a, RESOLVED 2026-07-15): vault stays +# MYSQL-backed (on the already-3-unit mysql-innodb-cluster via vault-mysql-router, +# unchanged) and scales to 3 -- NO etcd, NO Raft here. GATE (Stage 5, before +# commit): verify the deployed vault 1.8/stable charm actually does HA +# leader-election on the MySQL storage backend at 3 units. If it does NOT: do +# NOT fall back to etcd -- evaluate Raft integrated storage and route to D-068. +# 3-unit vault ALSO triples the manual-unseal burden (intersects OPEN SEC-003). +# +# NOT changed here (deliberately, per D-121): mysql-innodb-cluster / ovn-central +# / ceph-mon (already 3 in the base bundle); ceph-osd / nova-compute (scale-out, +# sized to node count -- Option C = ceph-osd 3, nova-compute 2, on the storage / +# compute nodes, NOT control-node containers); ceph-rbd-mirror stays 1 (D-108); +# memcached / designate-bind / glance-simplestreams-sync unchanged. +# ===================================================================== + +applications: + + # ---- Clustered API charms (hacluster + dual VIP already wired in base) ---- + # Each: 1 -> 3 units, one per control node (positional anti-affinity). + keystone: + num_units: 3 + to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] + + glance: + num_units: 3 + to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] + + neutron-api: + num_units: 3 + to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] + + nova-cloud-controller: + num_units: 3 + to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] + + placement: + num_units: 3 + to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] + + cinder: + num_units: 3 + to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] + + openstack-dashboard: + num_units: 3 + to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] + + octavia: + num_units: 3 + to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] + + barbican: + num_units: 3 + to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] + + magnum: + num_units: 3 + to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] + + designate: + num_units: 3 + to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] + + ceph-radosgw: + num_units: 3 + to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] + + # ---- RabbitMQ: native quorum clustering, NO hacluster. min-cluster-size is + # MANDATORY at multi-unit (D-009 AMENDMENT 2026-07-02, item A above). ---- + rabbitmq-server: + num_units: 3 + options: + min-cluster-size: 3 + to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] + + # ---- Vault: MySQL-backed (v-a), scale to 3, re-HA'd via vault-hacluster + # (defined below) + the relation re-added at the foot of this file. ---- + vault: + num_units: 3 + to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] + + # ---- vault-hacluster: RE-DECLARED (item C above). Removed by BUNDLEFIX-002 + # when vault was de-HA'd; D-121 (v-a) brings it back. Shape matches the + # other hacluster subs in bundle.yaml, with cluster_count: 3 (real HA). ---- + vault-hacluster: {charm: hacluster, channel: 2.4/stable, options: {cluster_count: 3}, bindings: {'': metal-admin, ha: metal-internal, hanode: metal-internal, pacemaker-remote: metal-internal, peer-availability: metal-internal}} + + # ---- cluster_count: 1 -> 3 for the 12 active hacluster subs (item B above). + # Overlay apply deep-merges options per-key -- the SAME mechanism + # dc-dc-ipv6-family-matrix.yaml already relies on (it supplies only + # `options:` for keystone et al. and depends on charm/channel/bindings + # surviving from base). So ONLY cluster_count is overridden here; each + # sub's base bindings/charm/channel are preserved. ---- + keystone-hacluster: {options: {cluster_count: 3}} + glance-hacluster: {options: {cluster_count: 3}} + neutron-api-hacluster: {options: {cluster_count: 3}} + nova-cloud-controller-hacluster: {options: {cluster_count: 3}} + placement-hacluster: {options: {cluster_count: 3}} + openstack-dashboard-hacluster: {options: {cluster_count: 3}} + cinder-hacluster: {options: {cluster_count: 3}} + octavia-hacluster: {options: {cluster_count: 3}} + barbican-hacluster: {options: {cluster_count: 3}} + magnum-hacluster: {options: {cluster_count: 3}} + ceph-radosgw-hacluster: {options: {cluster_count: 3}} + designate-hacluster: {options: {cluster_count: 3}} + +relations: + # ---- Re-add vault HA (removed by BUNDLEFIX-002; D-121 v-a re-adds it). + # vault already carries the `ha: metal-internal` binding in the base + # bundle, and vault-hacluster is declared above. ---- + - [vault:ha, vault-hacluster:ha] diff --git a/runbooks/dc-dc-phase2-tofu-dc-substrate.md b/runbooks/dc-dc-phase2-tofu-dc-substrate.md index a57aa64..fa61feb 100644 --- a/runbooks/dc-dc-phase2-tofu-dc-substrate.md +++ b/runbooks/dc-dc-phase2-tofu-dc-substrate.md @@ -3,10 +3,26 @@ From Office1, script each DC's "physical" layer into existence: the node-VM libvirt domains MAAS will discover and deploy, the DC's plane networks, and its OPNsense edge -- then register the DC's libvirt host to that DC's own -MAAS rack controller as a virsh VM-host. This runbook is written **DC1-first**. -DC2 is explicitly gated (see the callout below) and is NOT part of this -session's run until that gate clears -- do not duplicate these steps for DC2 -with invented values in the meantime. +MAAS rack controller as a virsh VM-host. This runbook builds **`vr1-dc0`** +(VR1's FIRST DC -- the build target this pass). **`vr1-dc1`** (VR1's SECOND +DC) is explicitly gated (see the callout below) and is NOT part of this +session's run until that gate clears -- do not duplicate these steps for +`vr1-dc1` with invented values in the meantime. + +!!! **NAMING -- this runbook was written pre-D-119 and has since been + reconciled to the region-qualified namespace.** Its original prose spoke + the bare `dcN` namespace (`dc1` = the build target, `dc2` = the second + DC). **D-119 REJECTS bare `dcN` for VR1 datacenters** -- the token `dc0` + already means VR0's live DC in `scripts/lib-net.sh`, so an unqualified + `dcN` is a latent off-by-one. The mapping applied throughout this file is: + runbook "DC1" -> **`vr1-dc0`** (already live in `opentofu/main.tf` as the + `vr1_dc0_*` modules), runbook "DC2" -> **`vr1-dc1`** (still gated). All + object names, module labels, plan files, and network/pool names below are + region-qualified to match `main.tf`. + **HAZARD:** following ANY residual bare-`dcN` literal in a copy of this + runbook (or in muscle memory) literally reintroduces the exact `dc0`/`dc1` + cross-cloud collision D-119 was written to delete -- region-qualify it or + stop. (D-119.) **Governing docs:** `docs/dc-dc-buildout-design.md` Section 4 "Phase 2" (the goal/build/gate this runbook satisfies), Section 9 (shim register -- the @@ -34,15 +50,23 @@ mechanism riding on it). past that dependency. Confirm Stage 2's actual state in `docs/dc-dc-deployment-workflow.md` before starting. -!!! **DC2 is explicitly gated, not merely "later."** D-101 has not assigned - DC2's real, non-overlapping v4 supernet yet (open sub-item; Tooling gap - register item 3's DATA half -- the import pipeline mechanism - (`netbox/dc-dc-prefixes-import.py`) is built, but the literals themselves - are not assigned). `opentofu/main.tf`'s `dc2_planes`/`dc2_storage` blocks - are commented out for exactly this reason. Do NOT uncomment them, do NOT - invent a placeholder DC2 CIDR, and do NOT run this runbook's steps against - DC2 until NetBox has assigned the real supernet and `main.tf` reflects it - for real. This is a hard gate, re-stated in the GATE section. +!!! **`vr1-dc1` is explicitly gated -- by SEQUENCING, not by a missing + literal.** The earlier rationale ("NetBox has not assigned the supernet") + is STALE: **D-115 ALREADY assigned it** -- `vr1-dc1` gets `10.12.64.0/19` + (moved inside the Cloud `/16`, colliding with nothing `vr1-dc0` occupies), and + that carve is imported to `office1-netbox`. So the blocker is no longer a + missing address; `vr1-dc1` stays out of scope THIS pass purely because we + build one DC at a time and `vr1-dc0` is the target. `opentofu/main.tf`'s + `vr1_dc1_planes` block is still commented out (its `vr1_dc1_storage` and + the three mesh legs ARE wired, being address-independent). Do NOT + uncomment the planes block, do NOT add `vr1-dc1` build steps, and do NOT + run this runbook's steps against `vr1-dc1` this pass. Re-run the whole + runbook for `vr1-dc1` when it is sequenced. This is a hard gate, re-stated + in the GATE section. + NB: `main.tf`'s commented `vr1_dc1_planes` block still carries the OLD + "wait for NetBox to assign D-101's supernet" comment -- that comment is + now stale per D-115; flagged as a divergence to fix under the `main.tf` + owner, out of scope for this doc-only pass. !!! **Node-VM creation has NO Roosevelt analog (Section 9 shim register).** `modules/node-vm` (the blank-disk, PXE-boot libvirt domain this stage @@ -53,12 +77,15 @@ mechanism riding on it). onward (commission/deploy, Juju bundle) IS Roosevelt-transferable; this one step is not. -!!! **Node sizing is not decided -- do not invent vCPU/RAM/disk.** - `modules/node-vm`'s `memory_mib`, `vcpu`, and `disk_size_bytes` variables - have no defaults by design (buildout-design Section 3's Phase-0 host/ - disk-budget decision has not been made). If that decision is still open - when you reach Step 6, STOP there and flag it -- do not pick numbers that - "seem reasonable" to get `tofu plan` to run. +!!! **Node sizing is now RULED (D-121, Option C, 2026-07-15).** + `modules/node-vm`'s `memory_mib`/`vcpu`/`disk_size_bytes` still have no + module defaults by design, but the VALUES to pass are no longer open: + D-121 ruled the per-DC layout as **3 control + 2 compute + 3 storage** (8 + node VMs) at the sizes wired into Step 6. This was the exact decision that + used to block Step 6; it is now UNBLOCKED. Still open (one sub-ruling + D-121 notes): the vault-HA backend fork -- but that is a Stage-5 bundle + concern, NOT a Step-6 blocker (node layout is fully ruled). Do not invent + any OTHER node spec; use D-121's numbers exactly. !!! **netem parameters are still an unruled D-100 sub-item (gap #11).** This stage's own gate text says "netem parameters applied and measured," but @@ -84,28 +111,34 @@ mechanism riding on it). --- -## Sequence (DC1; DC2 remains gated -- see callout above) +## Sequence (`vr1-dc0`; `vr1-dc1` remains gated -- see callout above) + +Honest achievable scope for a first run is **Steps 1-8** (libvirt-layer +objects + the edge). Step 9 onward is gated on `vr1-dc0`'s own MAAS rack +controller existing -- a Stage-4 gap (per-DC rack-controller source is not yet +stood up); do not treat Steps 9-12 as runnable until it is. ``` 1. Pre-flight: confirm Stage 1 + Stage 2 gates closed (read-only) 2. Confirm OPNsense prep-tool prereqs on the vcloud host (read-only) -3. Confirm/record still-open decisions this stage depends - on (node sizing, MAAS zone/pool, netem parameters) - -- STOP on any that are unresolved (read-only) -4. Prepare DC1's OPNsense base image (render/config-ISO steps DELETED - 2026-07-13 -- config is done over the REST API) [MUTATION: host-local files, gated] -5. Wire modules/opnsense-edge for DC1 into main.tf [repo change, gated] -6. Wire modules/node-vm calls for DC1's node VMs into - main.tf (ONLY once sizing is real) [repo change, gated] +3. Confirm/record decisions this stage depends on (node + sizing = D-121 RESOLVED; edge/WAN/LAN = D-122 RESOLVED; + MAAS zone/pool; netem parameters) -- STOP on any open (read-only) +4. Prepare vr1-dc0's OPNsense base image (prep only; edge + config is done over the REST API -- D-113(a2)) [MUTATION: host-local files, gated] +5. Wire modules/opnsense-edge for vr1-dc0 into main.tf [repo change, gated] + (prereq: build the vr1-dc0-wan uplink, office1-wan shape) +6. Wire the 8 modules/node-vm calls for vr1-dc0's node VMs + into main.tf (D-121 Option C sizes) [repo change, gated] 7. tofu init / validate / plan -- review before apply (read-only) -8. tofu apply -- DC1 OPNsense edge + DC1 node-VM domains [MUTATION: creates libvirt objects, gated] -9. Wire + apply modules/maas-vm-host -- register DC1's - libvirt host with DC1's MAAS rack controller [MUTATION: MAAS + repo change, gated] +8. tofu apply -- vr1-dc0 OPNsense edge + node-VM domains [MUTATION: creates libvirt objects, gated] +9. Wire + apply modules/maas-vm-host -- register vr1-dc0's + libvirt host with vr1-dc0's MAAS rack controller [MUTATION: MAAS + repo change, gated] 10. Verify MAAS discovers the node VMs (read-only) -11. Wire + apply modules/netem-link on DC1's mesh legs, +11. Wire + apply modules/netem-link on vr1-dc0's mesh legs, using the Section-6 placeholder lean, explicitly flagged [MUTATION: gated] 12. Verify edge routing + simulated ISP uplink + netem applied (read-only) - -> EXIT GATE -> Stage 4 (per-DC, once DC2 clears its own gate) + -> EXIT GATE -> Stage 4 (per-DC, once vr1-dc1 clears its own gate) ``` --- @@ -117,9 +150,10 @@ mechanism riding on it). virsh net-list --all virsh pool-list --all ``` -Expect: DC1's six plane networks, the three mesh-link networks (`dc1-dc2`, -`dc1-office1`, `dc2-office1`), and the `dc1`/`office1` storage pools, all -`active` -- Stage 1's Step 11 verification. If any are missing, Stage 1 is +Expect: `vr1-dc0`'s six plane networks, the three mesh-link networks +(`vr1-dc0-vr1-dc1`, `vr1-dc0-office1`, `vr1-dc1-office1` -- the region- +qualified names now in `main.tf`), and the `vr1-dc0`/`office1` storage pools, +all `active` -- Stage 1's Step 11 verification. If any are missing, Stage 1 is not actually done regardless of what `docs/dc-dc-deployment-workflow.md` says -- stop and reconcile there first. @@ -166,35 +200,47 @@ scripts. ## Step 3 -- Confirm/record still-open decisions this stage depends on (READ-ONLY) -Before wiring anything into `main.tf`, walk this list. Any "NOT YET" item is a -STOP for the corresponding later step -- record it plainly rather than -inventing a value to keep moving: - -- **Node sizing** (`memory_mib`/`vcpu`/`disk_size_bytes` for `modules/node-vm`, - and how many node VMs DC1 needs) -- buildout-design Section 3's Phase-0 - host/disk-budget decision. Blocks Step 6. -- **MAAS zone/pool for DC1's VM-host registration** (`modules/maas-vm-host`'s - `zone`/`pool` variables) -- both are optional (MAAS computes them if left - `null`, confirmed safe per `opentofu/README.md`'s audit pass), so this is - NOT necessarily a blocker; record whichever real zone/pool name the - operator wants, or confirm the `null` no-op is intentional. Affects Step 9. -- **DC1's real `power_address`** (the virsh URI MAAS uses to reach the vcloud - host) -- must be measured this session (or carried from Stage 1's +Before wiring anything into `main.tf`, walk this list. Two of the items that +used to block this stage are now RULED (D-121, D-122) and are recorded as +RESOLVED below; the genuinely-open ones remain STOPs. Record each plainly +rather than inventing a value to keep moving: + +- **Node sizing** (`memory_mib`/`vcpu`/`disk_size_bytes` and count for + `modules/node-vm`) -- **RESOLVED (D-121, Option C, 2026-07-15).** Per DC: 3 + control @ 16 vCPU / 65536 MiB / 150 GiB; 2 compute @ 12 vCPU / 49152 MiB / + 100 GiB; 3 storage @ 8 vCPU / 24576 MiB / 550 GiB -- 8 node VMs total. Step + 6 wires exactly these. Open sub-ruling D-121 still carries (the vault-HA + backend fork) does NOT block Step 6 -- it is a Stage-5 bundle concern. +- **Edge sizing / WAN uplink / LAN role** -- **RESOLVED (D-122, 2026-07-15).** + The DC edge follows the proven Office1 pattern: `memory_mib = 2048`, + `vcpu = 2`, nano base image (NO `disk_size_bytes`, like `office1_opnsense`). + The edge is **2-NIC** (baremetal-matched): `wan_network_name` is a + **dedicated per-site ISP uplink `vr1-dc0-wan`** (`172.30.2.0/24`, D-115) built + on the `office1-wan` pattern -- NOT a mesh leg (dark fiber is East-West only). + `lan_network_name` is **provider-public** -- the six planes are routed by the + fabric (OVN/OpenStack on isolated-L2 segments), so the edge is only the + external gateway on provider-public (D-100), not an inter-plane router. See + Step 5. PREREQUISITES: build `vr1-dc0-wan` (as a tofu module) and register + `172.30.2.0/24` in office1-netbox (not yet loaded). +- **MAAS zone/pool for `vr1-dc0`'s VM-host registration** + (`modules/maas-vm-host`'s `zone`/`pool` variables) -- both are optional + (MAAS computes them if left `null`, confirmed safe per `opentofu/README.md`'s + audit pass), so this is NOT necessarily a blocker; record whichever real + zone/pool name the operator wants, or confirm the `null` no-op is + intentional. Affects Step 9. +- **`vr1-dc0`'s real `power_address`** (the virsh URI MAAS uses to reach the + vcloud host) -- must be measured this session (or carried from Stage 1's `libvirt_uri` tfvar, if it is the SAME endpoint -- confirm, do not assume identical per `modules/maas-vm-host`'s own variable note). Blocks Step 9. -- **OPNsense `config.xml` tokens** -- per `opentofu/templates/README.md`'s - token legend, several tokens are pending Stage-0-adjacent decisions or a - real boot measurement: `{{WAN_IF}}`/`{{LAN_IF}}` (only measurable by - booting the domain and running `ifconfig` -- genuinely cannot be filled in - before Step 8 creates the domain), `{{WAN_IPADDR}}`/`{{WAN_SUBNET_BITS}}`/ - `{{WAN_GATEWAY}}` (pending NetBox), `{{LAN_IPADDR}}`/`{{LAN_SUBNET_BITS}}` - (pending which plane(s) OPNsense's LAN interface actually serves -- - flagged in the template README as not fully specified in the buildout - design), `{{MIRROR_SYNC_PROTOCOL}}`/`{{MIRROR_UPSTREAM_NET}}`/ - `{{MIRROR_SYNC_PORT}}` (pending mirror-software choice, gap #3-adjacent), - and `{{ROOT_PASSWORD_HASH}}` (must be freshly generated per site, never - reused from the shipped sample). Blocks Step 4's render sub-step for - everything except `{{OPNSENSE_HOSTNAME}}`/`{{DOMAIN}}`/NTP defaults. +- **`vr1-dc0`'s MAAS rack controller** -- a Stage-4 gap: the per-DC + rack-controller source is not yet stood up. This is the reason honest Step-9 + scope is BLOCKED and this pass's real reach is Steps 1-8 (see the GATE + section). Blocks Steps 9-12. +- **Edge post-boot interface mapping** (formerly the `{{WAN_IF}}`/`{{LAN_IF}}` + `config.xml` tokens) -- NO LONGER a config-render blocker. There is no + `config.xml` render (D-112/D-113(a2)); the real `vtnetN` mapping is measured + AFTER boot over SSH (`ifconfig`), then applied over the REST API. See the + REPLACEMENT chain in Step 4. Nothing here blocks wiring Step 5. - **netem parameters** -- still an unruled D-100 sub-item (gap #11); Step 11 uses the Section-6 qualitative placeholder only, explicitly labeled as such. Not a hard blocker for Step 11 (the mechanism can be exercised with a @@ -208,45 +254,43 @@ step named as blocked. Do not proceed past a step whose input is still --- -## Step 4 -- Prepare DC1's OPNsense base image + config.xml + config ISO [MUTATION: host-local files, gated] +## Step 4 -- Prepare `vr1-dc0`'s OPNsense base image [MUTATION: host-local files, gated] -> ## !!! STOP -- THIS STEP IS OBSOLETE AS WRITTEN (2026-07-13) !!! +> ## THE EDGE-CONFIG PATH (settled 2026-07-13 -- this IS the path) > -> Nothing here clobbers a live system (DC1's edge does not exist yet), but **two of the three -> sub-steps below cannot achieve what they claim**, and following them will waste a session and -> produce a dead edge: +> This step is now **prep the base image only.** Edge configuration is NOT baked into an image +> or seeded from an ISO -- it is done over the **REST API after boot**, the path proven end to +> end on the live Office1 edge (2026-07-12/13). The two deleted approaches, and why: > -> 1. **The config ISO is INERT -- it can never be read.** Per **D-112** (root-caused from +> 1. **The config ISO was INERT -- it could never be read.** Per **D-112** (root-caused from > upstream source): `opnsense-importer -b` probes for a read-only root. On INSTALLER media the > probe fails and the importer scans attached media. On our PRE-INSTALLED NANO image the root > is writable AND a factory `/conf/config.xml` already exists, so BOTH conditions hold and it > `bootstrap_and_exit 0`s **without enumerating a single device**. The Configuration Importer -> can NEVER fire on a nano image, by design. Building the ISO is pure waste. +> can NEVER fire on a nano image, by design -- so the ISO builder is DELETED and the module no +> longer takes a `config_iso_path`. > 2. **Full-`config.xml` rendering is SUPERSEDED** by **D-113(a2)** (ruled 2026-07-13): edge -> config is done over the **REST API** (`scripts/opnsense-api.sh`), which was proven end to -> end -- read AND write -- against the live Office1 edge on 2026-07-13. Hand-authoring the -> appliance's GUI-owned XML caused DOCFIX-191 (management lockout), DOCFIX-192 (dead console) -> and DOCFIX-193 (no DHCP, and an ISC `<dhcpd>` block that would have been inert against a Kea -> backend). None of those bugs is expressible through the API. +> config is done over the **REST API** (`scripts/opnsense-api.sh`), proven end to end -- read +> AND write -- against the live Office1 edge on 2026-07-13. Hand-authoring the appliance's +> GUI-owned XML caused DOCFIX-191 (management lockout), DOCFIX-192 (dead console) and +> DOCFIX-193 (no DHCP, and an ISC `<dhcpd>` block that would have been inert against a Kea +> backend). None of those bugs is expressible through the API. The renderer is DELETED. > -> **The `WAN_IF`/`LAN_IF` chicken-and-egg problem discussed below is therefore MOOT** -- it was an -> artifact of trying to seed a full config before first boot. The D-112(c) bootstrap (boot on -> factory defaults -> reach the console -> enable SSH + install the key -> configure over the -> network) measures the real `vtnetN` mapping *after* boot, where it is knowable. +> **There is no `WAN_IF`/`LAN_IF` chicken-and-egg.** It only ever existed because we tried to seed +> a full config before first boot. The D-112(c) bootstrap (boot on factory defaults -> reach the +> console -> enable SSH + install the key -> configure over the network) measures the real +> `vtnetN` mapping *after* boot, where it is knowable -- nothing to guess before Step 8. > -> **What DC1's edge should actually do (the proven Office1 path):** prep the image -> boot it -> -> D-112(c) console bootstrap -> configure over SSH + the REST API. See +> **What `vr1-dc0`'s edge does (the proven Office1 path):** prep the image (this step) -> Step 5 +> wires the domain -> Step 8 boots it -> D-112(c) console bootstrap -> configure over SSH + the +> REST API (the REPLACEMENT chain below). See > `docs/changelog-20260712-office1-opnsense-edge-build.md`, > `docs/changelog-20260713-opnsense-api-proven.md`, and > `docs/changelog-20260713-opnsense-api-write-proven.md`. -> -> **This step is NOT yet rewritten.** Reducing the template to a minimal bootstrap and rewriting -> Stage 3's edge steps around the API is the open D-113(a2) work. Until that lands, STOP here and -> re-derive rather than following the sub-steps below. -This step produces plain files on the vcloud host filesystem -- no libvirt or -MAAS object is created yet. Run from the vcloud host (or wherever these -scripts execute with reach to write there). +This step produces one plain file on the vcloud host filesystem -- the prepped +base image -- no libvirt or MAAS object is created yet. Run from the vcloud +host (or wherever this script executes with reach to write there). **MUTATION -- prep the base image (needs Step 2's tools)** ```bash @@ -254,31 +298,14 @@ bash scripts/opnsense-prep-image.sh # see the script's own header for its exac ``` **Expect:** a decompressed, qcow2-converted, resized OPNsense nano image at a real output path -- record that path, it feeds Step 5's `base_volume_path`. - -**(DELETED 2026-07-13.)** The `config.xml` render step is GONE -- the renderer -(`opnsense-render-config.sh`, formerly under scripts/) no longer exists. Edge config is done over the REST API -(D-113(a2)). See the replacement chain at the end of this step. -`WAN_IF`/`LAN_IF` are the one pair the script's own `REQUIRED_VARS` list -demands that genuinely cannot be measured before the domain exists (Step 8 -creates it). If Step 3 flagged these as not yet measurable, this render -sub-step cannot complete for real this session -- STOP here for DC1's config -render, record it as blocked pending a real boot, and do not substitute a -guessed `vtnetN` name. (One documented path forward once the domain exists: -boot the domain with a MINIMAL/placeholder config good enough to reach a -console, measure `ifconfig`'s real `vtnetN` mapping, then re-render and -re-seed -- that re-seed mechanic is not designed in this delivery; treat it -as a real follow-up need if it comes to that, not an invented shortcut here.) - -**(DELETED 2026-07-13.)** The config-ISO step is GONE -- the ISO builder -(`opnsense-build-config-iso.sh`, formerly under scripts/) no longer exists, and the module no longer takes a -`config_iso_path`. The ISO was never read by anything (D-112: the Importer cannot fire on a -pre-installed nano image). +That single path is this step's ONLY output. There is no config.xml render and +no config-ISO build -- both are deleted (D-112 / D-113(a2), see the box above). --- ### The REPLACEMENT chain (proven on Office1, 2026-07-12/13) -No `config.xml` anywhere. After Step 5 creates the domain and it boots on FACTORY DEFAULTS: +No `config.xml` anywhere. After Step 5 wires the domain and Step 8 boots it on FACTORY DEFAULTS: 1. **Console bootstrap (D-112(c))** -- reach the serial console, enable SSH, install the service public key. This is the only step that needs the console; everything after it is scripted. @@ -295,74 +322,128 @@ No `config.xml` anywhere. After Step 5 creates the domain and it boots on FACTOR bash scripts/opnsense-api.sh POST kea/service/reconfigure '{}' ``` -**`WAN_IF`/`LAN_IF` are measured AFTER boot** (`ifconfig` over SSH), where they are knowable -- -which is why the chicken-and-egg problem this step used to agonize over simply does not exist -anymore. +**The real `vtnetN` interface mapping is measured AFTER boot** (`ifconfig` over SSH), where it is +knowable -- there is no interface value to guess before the domain exists. -**GATE:** base image path and (if unblocked) config ISO path recorded as real -host-local file paths. If the config.xml render step is blocked on -`WAN_IF`/`LAN_IF`, record that explicitly and treat Step 5's `config_iso_path` -input as still pending -- do not wire Step 5 with a placeholder ISO. +**GATE:** the prepped base image path recorded as a real host-local file path +(it feeds Step 5's `base_volume_path`). Nothing else is produced or pending +here -- no config-ISO path exists to record. --- -## Step 5 -- Wire `modules/opnsense-edge` for DC1 into `main.tf` [repo change, gated] - -Only proceed once Step 4 produced real `base_volume_path` and -`config_iso_path` values (or an explicit decision to proceed with a -genuinely-final, if minimal, config.xml -- not a placeholder). - -**MUTATION -- edit `opentofu/main.tf`**, adding a block shaped like: +## Step 5 -- Wire `modules/opnsense-edge` for `vr1-dc0` into `main.tf` [repo change, gated] + +Proceed once Step 4 produced a real `base_volume_path`. The edge SHAPE is +ruled (D-122): it mirrors the already-applied `office1_opnsense` block in +`main.tf` -- `memory_mib = 2048`, `vcpu = 2`, nano image, **no +`disk_size_bytes`** (the disk is a direct copy of the prepped nano; the module +does not accept a size input -- DOCFIX-189), and **no `config_iso_path`** (the +module dropped it -- D-112). There is nothing to invent for the edge specs +themselves; the two values that ARE still per-site (the base image path, and +the two network names) are called out below. + +**PREREQUISITE -- build the `vr1-dc0-wan` uplink first.** D-122 gives the DC +its OWN dedicated L3 "ISP" uplink, NOT a mesh leg (dark fiber is East-West +replication only). Build a per-site NAT'd `/24` uplink network `vr1-dc0-wan` +on the **`office1-wan` pattern** (the same mechanical shape `office1-wan` was +created in -- a documented D-103 debt to formalize into a module later; today +it is a literal network name the edge's `wan_network_name` points at). This is +a prerequisite to wiring the block below with a real `wan_network_name`. + +**LAN side -- RESOLVED (D-122, operator ruling 2026-07-15): `provider-public`.** +The edge is **2-NIC** (WAN + LAN), matching the baremetal target: the six +planes are **routed by the fabric** (the `dc-planes` segments are isolated L2; +OVN/OpenStack own their L3, not the edge), so the edge is NOT an inter-plane +router -- only the external boundary. Its LAN attaches to **provider-public** +(`10.12.4.0/22`), where the edge is the upstream/external gateway the provider +network exits through (FIP/SNAT egress + GUA injection, D-100 br-ex/provider +model). metal-admin and the other planes reach external via fabric routing, not +a direct edge leg. So `lan_network_name` = the `vr1_dc0_planes` provider-public +output (confirm the exact output name in `modules/dc-planes`). No new +site-internal network module is needed (this SUPERSEDES the earlier "build an +office1-network analog" flag). + +**MUTATION -- edit `opentofu/main.tf`**, adding a block shaped like +`office1_opnsense` (main.tf, the applied reference): ```hcl -module "dc1_opnsense" { +module "vr1_dc0_opnsense" { source = "./modules/opnsense-edge" - vm_name = "dc1-opnsense" - memory_mib = <REAL VALUE -- OPNsense's own sizing guidance, not invented> - vcpu = <REAL VALUE> - pool_name = module.dc1_storage.pool_name # confirm real output name in modules/dc-storage-pool - disk_size_bytes = <REAL VALUE> - base_volume_path = "<Step 4's real base image path>" - config_iso_path = "<Step 4's real config ISO path>" - lan_network_name = <the dc1_planes output this edge's LAN side attaches to -- confirm which plane per Step 3's open LAN-role item> - wan_network_name = module.mesh_dc1_office1... # or a dedicated per-site ISP-uplink segment -- confirm which, do not assume + vm_name = "vr1-dc0-opnsense" + memory_mib = 2048 # D-122: Office1 pattern + vcpu = 2 # D-122: Office1 pattern + pool_name = module.vr1_dc0_storage.pool_name + # No disk_size_bytes: nano direct-copy, size from opnsense-prep-image.sh's GROW (DOCFIX-189). + base_volume_path = "<Step 4's real prepped-image path>" + lan_network_name = <the vr1_dc0_planes provider-public output -- D-122: edge is the external gateway on provider-public; confirm the output name in modules/dc-planes> + wan_network_name = "vr1-dc0-wan" # D-122: dedicated per-site ISP uplink, office1-wan pattern (build first, promote to a tofu module) } ``` -Every placeholder above (`<...>`) must be a real, measured/decided value -before this is committed -- this template line exists to show shape, not to -be copy-pasted with invented numbers. `memory_mib`/`vcpu`/`disk_size_bytes` -here have no ratified default in the module either (same "no invented specs" -discipline as `modules/node-vm`). +Before commit: the base image path (Step 4) must be real, `vr1-dc0-wan` must be +built (as a tofu module -- D-122) and `172.30.2.0/24` registered in office1-netbox +(D-115, not yet loaded), and `lan_network_name` bound to the real provider-public +plane output. `memory_mib`/`vcpu`/the nano-no-disk shape and the 2-NIC WAN/LAN +model are ruled (D-122), not invented. -**GATE:** `opentofu/main.tf` diff shows exactly one new `module "dc1_opnsense"` -block, every argument a real value, nothing else changed. +**GATE:** `opentofu/main.tf` diff shows exactly one new `module "vr1_dc0_opnsense"` +block, no `disk_size_bytes` and no `config_iso_path` line, every argument a +real value (with `vr1-dc0-wan` and the site-internal LAN network genuinely +built), nothing else changed. --- -## Step 6 -- Wire `modules/node-vm` calls for DC1's node VMs into `main.tf` [repo change, gated] - -**ONLY if Step 3 confirmed node sizing is real.** If sizing is still "NOT -YET," STOP here -- this step does not proceed, and this session's DC1 pass -ends at whatever the last unblocked step was (Step 5 or wherever Step 4 left -off). Re-run from here once sizing exists; do not invent numbers to close -this out early. - -**MUTATION -- edit `opentofu/main.tf`**, one `module "dc1_node_XX"` block per -node VM, shaped like: +## Step 6 -- Wire `modules/node-vm` calls for `vr1-dc0`'s node VMs into `main.tf` [repo change, gated] + +**Node sizing is RULED (D-121, Option C).** Wire **8** `module "vr1_dc0_node_*"` +blocks in three role-sizes: + +| Role | Count | vCPU | `memory_mib` | disk | Names | +|---------|-------|------|--------------|-------------|------------------------------------| +| control | 3 | 16 | 65536 | 150 GiB | `vr1-dc0-control-01..03` | +| compute | 2 | 12 | 49152 | 100 GiB | `vr1-dc0-compute-01..02` | +| storage | 3 | 8 | 24576 | 550 GiB | `vr1-dc0-storage-01..03` | + +`disk_size_bytes` is BYTES: express each as GiB x 1024^3 in HCL -- +`150 * 1024 * 1024 * 1024`, `100 * 1024 * 1024 * 1024`, +`550 * 1024 * 1024 * 1024` (the module's `disk_size_bytes` expects bytes; the +GROW/qcow2 layer is thin-provisioned so this is a max, not a preallocation). + +**Why 3 control (context):** this layout exists to carry the Stage-5 HA +scale-up (D-121). The bundle's **14 currently-single-unit services scale +1 -> 3**: the 11 hacluster+VIP API apps (keystone, glance, neutron-api, +nova-cloud-controller, placement, cinder, openstack-dashboard, octavia, +barbican, magnum, designate) plus `ceph-radosgw`, `rabbitmq-server`, and +`vault`. Those 3-unit control-plane services LIVE on the 3 control nodes -- +alongside the services that are ALREADY at 3 and do not change +(`mysql-innodb-cluster`, `ovn-central`, `ceph-mon`). Meanwhile `ceph-osd` = 3 +units on the 3 storage nodes and `nova-compute` = 2 units on the 2 compute +nodes. The node-COUNT/sizing here is fully ruled; the still-open D-121 +sub-item (the vault-HA backend fork) is a Stage-5 bundle concern and does NOT +gate this step. + +**MUTATION -- edit `opentofu/main.tf`**, one block per node VM (control shown; +compute/storage identical shape with their row's vCPU/memory/disk/name): ```hcl -module "dc1_node_01" { +module "vr1_dc0_control_01" { source = "./modules/node-vm" - vm_name = "dc1-node-01" - memory_mib = <REAL VALUE from the Phase-0 sizing decision> - vcpu = <REAL VALUE> - pool_name = module.dc1_storage.pool_name - disk_size_bytes = <REAL VALUE> - network_names = [ <ordered list of module.dc1_planes outputs -- confirm which plane goes first for PXE priority, per D-103> ] + vm_name = "vr1-dc0-control-01" + memory_mib = 65536 # D-121: control + vcpu = 16 # D-121: control + pool_name = module.vr1_dc0_storage.pool_name + disk_size_bytes = 150 * 1024 * 1024 * 1024 # D-121: control 150 GiB + network_names = [ <ordered list of module.vr1_dc0_planes outputs -- confirm which plane goes first for PXE priority, per D-103> ] } ``` -Repeat per node, per the real node count from the sizing decision. - -**GATE:** every `module "dc1_node_*"` block uses only real, decided values; -node count matches the recorded sizing decision exactly. +Repeat for `vr1_dc0_control_02`/`03`, `vr1_dc0_compute_01`/`02` (12 vCPU / +49152 MiB / `100 * 1024 * 1024 * 1024`), and `vr1_dc0_storage_01`/`02`/`03` +(8 vCPU / 24576 MiB / `550 * 1024 * 1024 * 1024`). **`network_names` is STILL a +flagged STOP** in every block: D-121 ruled the SIZES, not the plane +attachment/PXE order -- that ordered plane list is an unresolved value (D-103 +PXE priority) and must not be invented to make the block parse. + +**GATE:** exactly 8 `module "vr1_dc0_node_*"`-shaped blocks (3 control / 2 +compute / 3 storage) with D-121's sizes verbatim; each `network_names` either +a genuinely-resolved plane list or explicitly recorded as the remaining STOP, +never a guessed order. --- @@ -373,24 +454,25 @@ cd opentofu tofu fmt -check -recursive -diff . tofu init -backend=false -input=false tofu validate -tofu plan -out=phase2-dc1.tfplan +tofu plan -out=phase2-vr1-dc0.tfplan ``` -Review the plan line by line: expect creates for `module.dc1_opnsense` -(the edge VM's boot volume, config-seed volume, and domain) and each -`module.dc1_node_*` (a blank boot volume + domain per node). Confirm nothing -else is planned -- in particular, confirm no DC2 resource appears (its -module blocks should still be commented out per the DC2 gate above). +Review the plan line by line: expect creates for `module.vr1_dc0_opnsense` +(the edge VM's boot volume and domain -- NO config-seed volume/cdrom anymore; +that path is deleted, D-112) and each `module.vr1_dc0_*` node (a blank boot +volume + domain per node: 3 control, 2 compute, 3 storage). Confirm nothing +else is planned -- in particular, confirm no `vr1-dc1` resource appears (its +`vr1_dc1_planes` block should still be commented out per the gate above). **GATE:** plan matches this expectation exactly; no unexpected creates, updates, or destroys. Do not apply a plan you have not read. --- -## Step 8 -- `tofu apply` -- DC1 OPNsense edge + node-VM domains [MUTATION: gated] +## Step 8 -- `tofu apply` -- `vr1-dc0` OPNsense edge + node-VM domains [MUTATION: gated] ```bash cd opentofu -tofu apply phase2-dc1.tfplan +tofu apply phase2-vr1-dc0.tfplan ``` Confirm this is the exact reviewed plan file from Step 7 (not re-planned) before running. This is the first live mutation THIS stage performs. @@ -398,12 +480,13 @@ before running. This is the first live mutation THIS stage performs. **VERIFY** ```bash virsh list --all -virsh domblklist dc1-opnsense -virsh domblklist dc1-node-01 # per real node name +virsh domblklist vr1-dc0-opnsense +virsh domblklist vr1-dc0-control-01 # per real node name ``` Expect the new domains present (likely `shut off` until first boot/PXE), the -OPNsense domain showing both the base overlay disk and the config ISO -attached as a secondary cdrom, each node VM showing a single blank boot disk. +OPNsense domain showing a SINGLE boot disk (a direct copy of the prepped nano +-- no config-ISO cdrom; that path is deleted, D-112), each node VM showing a +single blank boot disk. **GATE:** all new domains defined as planned; disk attachments match Step 5/6's wiring. @@ -418,20 +501,26 @@ hasn't already. --- -## Step 9 -- Wire + apply `modules/maas-vm-host` -- register DC1's libvirt host with DC1's MAAS rack controller [MUTATION: gated] - -**Hard dependency on Stage 2.** This step requires DC1's MAAS rack controller -(under the Stage-2-stood-up MAAS region) to actually exist and be reachable. -If Stage 2 (or the DC1 rack-controller enrollment specifically, however that -ends up sequenced relative to Stage 2's own runbook) is not done, STOP here --- this stage's remaining steps (9-12) do not proceed, and DC1's Phase 2 work -is INCOMPLETE, not silently skippable. +## Step 9 -- Wire + apply `modules/maas-vm-host` -- register `vr1-dc0`'s libvirt host with `vr1-dc0`'s MAAS rack controller [MUTATION: gated] + +**Hard dependency -- `vr1-dc0`'s MAAS controller.** This step requires +`vr1-dc0`'s own MAAS to exist and be reachable. **D-123 (PROPOSED, recommend +Model A) resolves what that is:** a `vvr1-dc0` **site headend VM** built the +proven Office1 way -- an OpenTofu `cloudinit-vm` block shaped like +`module "voffice1"` + a run of `scripts/site-headend-install.sh` (which already +names `vvr1-dc0/vvr1-dc1` as targets) to install MAAS region+rack + LXD. Under +Model A the node VMs stay vcloud-level and this MAAS DISCOVERS them via the +`maas-vm-host` registration below (power_address = vcloud's virsh). Until Model +A is ruled and `vvr1-dc0` is stood up, this is the honest scope boundary: +**Steps 1-8 are this pass's achievable scope; Steps 9-12 STOP here** -- +`vr1-dc0`'s Phase 2 is then INCOMPLETE, not silently skippable. (Also inherits +Stage 2's reachable-MAAS-region dependency.) **MUTATION -- edit `opentofu/main.tf`** ```hcl -module "dc1_maas_vm_host" { +module "vr1_dc0_maas_vm_host" { source = "./modules/maas-vm-host" - vm_host_name = "dc1-vcloud" + vm_host_name = "vr1-dc0-vcloud" power_address = "<Step 3's real, measured virsh URI -- confirm same as var.libvirt_uri or a distinct value>" zone = <real zone name, or leave unset for MAAS's computed default -- per Step 3> pool = <real pool name, or leave unset -- per Step 3> @@ -440,8 +529,8 @@ module "dc1_maas_vm_host" { ```bash cd opentofu -tofu plan -out=phase2-dc1-maashost.tfplan # review: exactly one new maas_vm_host resource -tofu apply phase2-dc1-maashost.tfplan +tofu plan -out=phase2-vr1-dc0-maashost.tfplan # review: exactly one new maas_vm_host resource +tofu apply phase2-vr1-dc0-maashost.tfplan ``` **GATE:** `maas_vm_host` resource created; no other resource touched by this @@ -453,22 +542,22 @@ plan. **CHECK** ```bash -maas admin vm-hosts read | jq -r '.[] | select(.name=="dc1-vcloud") | {id, name, resources}' -maas admin machines read | jq -r '.[] | select(.hostname|test("^dc1-node")) | "\(.hostname)\t\(.status_name)"' +maas admin vm-hosts read | jq -r '.[] | select(.name=="vr1-dc0-vcloud") | {id, name, resources}' +maas admin machines read | jq -r '.[] | select(.hostname|test("^vr1-dc0-")) | "\(.hostname)\t\(.status_name)"' ``` -Expect DC1's node VMs listed as MAAS machines (status likely `New` -- +Expect `vr1-dc0`'s node VMs listed as MAAS machines (status likely `New` -- enlistment discovered them; commissioning is Stage 4's job, not this stage's). If the node VMs do not appear, do not proceed to declare this stage's MAAS-discovery gate met -- troubleshoot the VM-host registration (power_address correctness, virsh reachability from the MAAS rack controller) before moving on. -**GATE:** DC1's OpenTofu-created node VMs visible in MAAS, associated with -the `dc1-vcloud` VM host. +**GATE:** `vr1-dc0`'s OpenTofu-created node VMs visible in MAAS, associated with +the `vr1-dc0-vcloud` VM host. --- -## Step 11 -- Wire + apply `modules/netem-link` on DC1's mesh legs, placeholder profile [MUTATION: gated] +## Step 11 -- Wire + apply `modules/netem-link` on `vr1-dc0`'s mesh legs, placeholder profile [MUTATION: gated] Per the netem callout above: this step applies ONLY the buildout design's qualitative Section 6 lean ("same-metro dark fiber... low single-digit ms, @@ -477,26 +566,29 @@ ruled numeric value. If the operator has since ruled a specific numeric profile, use that instead and cite the ruling; do not use this placeholder language once a real ruling exists. -**MUTATION -- edit `opentofu/main.tf`**, one block per relevant mesh leg -(at minimum DC1<->DC2 and DC1<->Office1; DC2<->Office1 is out of scope while -DC2 remains gated): +**MUTATION -- edit `opentofu/main.tf`**, one block per relevant mesh leg. The +only leg with substrate on both ends this pass is `vr1-dc0`<->`office1` +(`vr1-dc0`<->`vr1-dc1` and `vr1-dc1`<->`office1` have no `vr1-dc1`-side +substrate while `vr1-dc1` remains gated): ```hcl -module "netem_dc1_office1" { +module "netem_vr1_dc0_office1" { source = "./modules/netem-link" - link_name = "dc1-office1" + link_name = "vr1-dc0-office1" vcloud_host_ssh_target = "<real SSH target for the vcloud host -- measured, not invented>" - bridge_name = "<real OS bridge name for this mesh-link network -- confirm via `virsh net-info dc1-office1` or `tofu show`, not assumed>" + bridge_name = "<real OS bridge name for this mesh-link network -- confirm via `virsh net-info vr1-dc0-office1` or `tofu show`, not assumed>" netem_args = "<PLACEHOLDER pending D-100 sub-item ruling -- record explicitly as provisional, e.g. a low single-digit-ms delay with modest jitter and negligible loss, cited to buildout-design Section 6, NOT presented as a measured or ruled value>" } ``` -(A DC1<->DC2 block follows the same shape once DC2 itself is ungated; until -then, applying netem on that specific leg has no DC2-side substrate to -matter for, so it is naturally deferred alongside the rest of DC2's work.) +(A `vr1-dc0`<->`vr1-dc1` block -- `module "netem_vr1_dc0_vr1_dc1"`, +`link_name "vr1-dc0-vr1-dc1"` -- follows the same shape once `vr1-dc1` itself +is ungated; until then, applying netem on that leg has no `vr1-dc1`-side +substrate to matter for, so it is naturally deferred alongside the rest of +`vr1-dc1`'s work.) ```bash cd opentofu -tofu plan -out=phase2-dc1-netem.tfplan -tofu apply phase2-dc1-netem.tfplan +tofu plan -out=phase2-vr1-dc0-netem.tfplan +tofu apply phase2-vr1-dc0-netem.tfplan ``` **GATE:** the `terraform_data`/`local-exec` resource applies without error; @@ -514,20 +606,21 @@ sudo tc qdisc show dev <bridge_name> # per Step 11's real bridge name ``` Expect a `netem` qdisc present with the applied (placeholder) parameters. -**CHECK -- OPNsense edge reachability** (only meaningful once WAN_IF/LAN_IF -have been measured on a real boot per Step 4's note, and the domain has -actually been booted/configured) +**CHECK -- OPNsense edge reachability** (only meaningful once the domain has +been booted and configured over the REST API per Step 4's REPLACEMENT chain -- +the `vtnetN` mapping and WAN/LAN addressing are measured/applied AFTER boot) ```bash -virsh domstate dc1-opnsense +virsh domstate vr1-dc0-opnsense ``` -If the config.xml render was blocked in Step 4 (pending `WAN_IF`/`LAN_IF`), -this verification is necessarily partial this session -- record that -honestly rather than declaring edge routing verified when it is not. +If the domain has not yet been through the console + REST-API bootstrap (the +D-112(c) chain), this verification is necessarily partial this session -- +record that honestly rather than declaring edge routing verified when it is +not. **GATE (this step):** netem qdisc present and matching Step 11's recorded -placeholder profile; OPNsense domain running IF its config.xml was -successfully completed, otherwise recorded as blocked pending a real-boot -`vtnetN` measurement and re-seed. +placeholder profile; OPNsense domain running IF it has been through the +post-boot REST-API bootstrap, otherwise recorded as blocked pending that +bootstrap (console -> SSH -> API, per Step 4). --- @@ -538,18 +631,23 @@ Buildout-design Section 4 / deployment-workflow Stage 3 states the gate as: VMs; edge routing + simulated ISP uplink up; netem parameters applied and measured." As of what this runbook can actually close this session: -- **MAAS rack controller per DC up:** achievable ONLY once Stage 2 (or - whatever runbook stands up DC1's specific rack controller) is done -- - Step 1/9 depend on it. Record actual state, do not assume it. +- **MAAS rack controller per DC up:** achievable ONLY once `vr1-dc0`'s own + MAAS rack controller is stood up -- a Stage-4 gap that is NOT yet done + (D-122 confirms each site runs its own controller; the SOURCE for it is the + open item). Steps 1/9 depend on it. This is why honest scope for a first run + is Steps 1-8. Record actual state, do not assume it. - **MAAS discovers the OpenTofu-created node VMs:** achievable through Step - 10, CONDITIONAL on node sizing being real (Step 3/6) and Stage 2's MAAS - region being reachable (Step 9's dependency). + 10, CONDITIONAL on the per-DC rack controller existing (Step 9's dependency) + and Stage 2's MAAS region being reachable. Node sizing itself is no longer a + condition -- it is RULED (D-121, Step 6). - **Edge routing + simulated ISP uplink up:** PARTIALLY achievable. The - OPNsense domain and its network attachments can be created (Step 8), but - full edge routing verification depends on `WAN_IF`/`LAN_IF` being measured - on a real boot (Step 4's documented blocker) and on the WAN/LAN addressing - tokens (pending NetBox) -- do not mark this sub-gate fully closed unless - those are genuinely resolved. + OPNsense domain and its network attachments can be created (Step 8), and the + edge shape/uplink are RULED (D-122: `vr1-dc0-wan` dedicated ISP uplink, + Office1-pattern edge), but full edge routing verification depends on the + post-boot REST-API bootstrap (Step 4's REPLACEMENT chain -- console -> SSH + -> API, where the real `vtnetN` mapping and WAN/LAN addressing are measured + and applied). Do not mark this sub-gate fully closed until that bootstrap + has actually run against the booted domain. - **netem parameters applied and measured:** applied, YES (Step 11), using an explicitly-labeled PLACEHOLDER same-metro-lean profile per buildout-design Section 6 -- MEASURED in the sense of "the qdisc is @@ -561,21 +659,25 @@ measured." As of what this runbook can actually close this session: remaining gap named. **This stage's exit gate is therefore CONDITIONALLY MET at best** on any run -before (a) Stage 2 is complete, (b) node sizing is decided, (c) DC1's -OPNsense WAN/LAN addressing and `vtnetN` mapping are measured, and (d) D-100's -exact netem parameters are ruled. Update `docs/dc-dc-deployment-workflow.md`'s -Stage 3 row to reflect the REAL state after running this (e.g. "PARTIAL -- -libvirt objects created, MAAS registration blocked on Stage 2" or similar), -never to a blanket DONE unless every sub-condition above is genuinely true. - -**DC2:** none of the above applies to DC2 until NetBox assigns its real -supernet (D-101 open item) and `main.tf`'s DC2 blocks are uncommented with -real values -- re-run this entire runbook for DC2 at that point, do not -attempt a partial DC2 pass now. - --> Proceed to Stage 4 (MAAS enlist/commission/deploy) for DC1 once this -gate's applicable sub-conditions are genuinely met; DC2 remains blocked at -Stage 3 until its own gate (NetBox supernet) clears. +before (a) `vr1-dc0`'s per-DC MAAS rack controller is stood up (Stage-4 gap) +and Stage 2's MAAS region is reachable, (b) `vr1-dc0`'s OPNsense edge has been +through the post-boot REST-API bootstrap (`vtnetN` + WAN/LAN applied), and +(c) D-100's exact netem parameters are ruled. Node sizing (D-121) and the edge +shape/uplink (D-122) are NO LONGER open. Update +`docs/dc-dc-deployment-workflow.md`'s Stage 3 row to reflect the REAL state +after running this (e.g. "PARTIAL -- libvirt objects created, MAAS +registration blocked on the per-DC rack controller" or similar), never to a +blanket DONE unless every sub-condition above is genuinely true. + +**`vr1-dc1`:** none of the above applies to `vr1-dc1` this pass. Its v4 +supernet IS assigned (D-115, `10.12.64.0/19`) -- the blocker is sequencing, +not a missing literal -- but its `vr1_dc1_planes` block stays commented out +and it stays out of scope. Re-run this entire runbook for `vr1-dc1` when it is +sequenced; do not attempt a partial `vr1-dc1` pass now. + +-> Proceed to Stage 4 (MAAS enlist/commission/deploy) for `vr1-dc0` once this +gate's applicable sub-conditions are genuinely met; `vr1-dc1` remains blocked +at Stage 3 by sequencing until it is taken up as its own pass. --- @@ -593,16 +695,19 @@ Stage 3 until its own gate (NetBox supernet) clears. netem placeholder profile and its provisional status) -- redact nothing that isn't a secret. - [ ] `docs/session-ledger.md` updated with the outcome, including which - sub-steps were blocked and why (Stage 2 dependency, node sizing, - WAN/LAN token gaps, D-100 netem sub-item). + sub-steps were blocked and why (per-DC MAAS rack-controller Stage-4 gap, + Stage 2 MAAS-region dependency, the edge post-boot bootstrap, D-100 + netem sub-item). - [ ] `docs/dc-dc-deployment-workflow.md` Stage 3 row updated to the REAL, honest state (PARTIAL/DONE/NOT STARTED per the GATE section above), never rounded up to DONE with open sub-conditions. -- [ ] `opentofu/README.md` updated to reflect DC1's OPNsense edge / node-VM / - MAAS-vm-host / netem-link modules moving from "not instantiated" to - instantiated (or partially so) once this runbook actually applies - them, including any real bug found while doing so (matching this - repo's practice of logging schema surprises as they're found). -- [ ] DC2 gate re-confirmed as still closed (NetBox supernet not yet - assigned) if this session did not clear it -- do not let this - runbook's DC1 completion be misread as covering DC2. +- [ ] `opentofu/README.md` updated to reflect `vr1-dc0`'s OPNsense edge / + node-VM / MAAS-vm-host / netem-link modules moving from "not + instantiated" to instantiated (or partially so) once this runbook + actually applies them, including any real bug found while doing so + (matching this repo's practice of logging schema surprises as they're + found). +- [ ] `vr1-dc1` gate re-confirmed as still closed BY SEQUENCING (its supernet + IS assigned -- D-115 `10.12.64.0/19`) if this session did not take it + up -- do not let this runbook's `vr1-dc0` completion be misread as + covering `vr1-dc1`. diff --git a/scripts/site-headend-install.sh b/scripts/site-headend-install.sh index 46e7dd7..979503c 100644 --- a/scripts/site-headend-install.sh +++ b/scripts/site-headend-install.sh @@ -1,19 +1,39 @@ #!/usr/bin/env bash -# scripts/site-headend-install.sh [--check|--dry-run] --compose-cidr <CIDR> [--maas-url-ip <IP>] +# scripts/site-headend-install.sh [--check|--dry-run] [--role <region+rack|rack>] +# region+rack (default): --compose-cidr <CIDR> [--maas-url-ip <IP>] +# rack: --region-url <http://REGION-IP:5240/MAAS> --enroll-secret-file <path> # -# Installs a VR1 SITE HEADEND on the host it runs on: MAAS (region+rack) + LXD, with that -# LXD registered back into MAAS as an LXD VM host so MAAS can COMPOSE the site's non-stack -# service machines into it. This is D-114, and it is VR0's proven `lxd` + `tailscale` -# pattern applied per site. +# Installs a VR1 SITE HEADEND on the host it runs on. Two roles (D-123 rules ONE MAAS +# region on Office1 + a rack controller PER DC): # -# RUN IT ON THE SITE HOST (voffice1 today; vvr1-dc0/vvr1-dc1 later), as root: +# --role region+rack (DEFAULT -- unchanged legacy behavior): MAAS (region+rack) + LXD, with +# that LXD registered back into MAAS as an LXD VM host so MAAS can COMPOSE the site's +# non-stack service machines into it. This is D-114, VR0's proven `lxd` + `tailscale` +# pattern applied per site. This is the Office1 (voffice1) build. +# +# --role rack: a DC rack controller (vvr1-dc0) that ENROLLS to Office1's EXISTING region +# (D-123). A rack runs NO PostgreSQL, does its own `maas init rack`, and composes +# NOTHING -- the DC's OpenStack nodes are OpenTofu-created and DISCOVERED via the vcloud +# virsh vm-host registered to the REGION (not composed here). It therefore SKIPS all of +# the region DB / createadmin / LXD / vm-host / compose-DHCP work below. Enabling +# DHCP-on-metal-admin is a REGION-side op (Office1's MAAS, naming this rack as +# primary_rack) done as a RUNBOOK step -- NOT by this script (see rack note below). +# +# RUN IT ON THE SITE HOST, as root: +# region+rack (voffice1): # ssh <site-host> 'sudo bash -s' -- --compose-cidr 10.10.1.0/24 < scripts/site-headend-install.sh +# rack (vvr1-dc0): the region operator reads the enrollment secret from the REGION host's +# /var/snap/maas/common/maas/secret and drops it into a 0600 file on this host first: +# ssh <dc-rack> 'sudo bash -s' -- --role rack \ +# --region-url http://10.10.0.20:5240/MAAS --enroll-secret-file /root/region-enroll.secret \ +# < scripts/site-headend-install.sh # # --check report what is installed/configured; mutate nothing; exit 0 ok / 1 incomplete # --dry-run print what it would do; mutate nothing # (no flag) install + configure (idempotent; safe to re-run) # -# EXIT: 0 ok | 1 check-failed | 2 bad args | 3 unsupported OS | 4 install failed. ASCII + LF. +# EXIT: 0 ok | 1 check-failed | 2 bad args (incl. bad/missing --role args) | 3 unsupported OS +# | 4 install failed (incl. rack: enrollment-secret file absent/empty at mutate time). ASCII + LF. # # ===================================================================================== # THE FOUR TRAPS THIS SCRIPT EXISTS TO ENCODE (all hit for real on 2026-07-13; do not @@ -49,31 +69,60 @@ # servers on one L2 is an intermittent failure that is genuinely unpleasant to # diagnose. --compose-cidr is REQUIRED and has no default precisely so this can never # be picked by accident, and the script REFUSES to touch DHCP on any other subnet. +# +# TRAPS 1-3 (LXD/dnsmasq) DO NOT APPLY IN --role rack: a rack installs no LXD, creates no +# bridge, and runs no dnsmasq, so the :53/:67 conflict and the 5.21-track pin are moot. +# Trap 4's SPIRIT is kept absolutely: rack mode configures NO DHCP at all (metal-admin DHCP +# is a REGION-side op, not this script's), so it cannot fight the edge's Kea by construction. # ===================================================================================== set -uo pipefail CHECK=0; DRYRUN=0; COMPOSE_CIDR=""; MAAS_IP="" +ROLE="region+rack"; REGION_URL=""; ENROLL_SECRET_FILE="" while [ $# -gt 0 ]; do case "$1" in - --check) CHECK=1 ;; - --dry-run) DRYRUN=1 ;; - --compose-cidr) shift; COMPOSE_CIDR="${1:-}" ;; - --maas-url-ip) shift; MAAS_IP="${1:-}" ;; + --check) CHECK=1 ;; + --dry-run) DRYRUN=1 ;; + --role) shift; ROLE="${1:-}" ;; + --compose-cidr) shift; COMPOSE_CIDR="${1:-}" ;; + --maas-url-ip) shift; MAAS_IP="${1:-}" ;; + --region-url) shift; REGION_URL="${1:-}" ;; + --enroll-secret-file) shift; ENROLL_SECRET_FILE="${1:-}" ;; -h|--help) - echo "usage: site-headend-install.sh [--check|--dry-run] --compose-cidr <CIDR> [--maas-url-ip <IP>]" + echo "usage: site-headend-install.sh [--check|--dry-run] [--role <region+rack|rack>]" + echo " region+rack (default): --compose-cidr <CIDR> [--maas-url-ip <IP>]" + echo " rack: --region-url <http://REGION-IP:5240/MAAS> --enroll-secret-file <path>" exit 0 ;; *) echo "FAIL: unknown arg '$1'" >&2; exit 2 ;; esac shift done -[ -n "$COMPOSE_CIDR" ] || { echo "FAIL: --compose-cidr is REQUIRED (e.g. 10.10.1.0/24). It has no default on purpose: MAAS must serve DHCP on the compose network ONLY, never on the site LAN, which the edge's Kea already owns." >&2; exit 2; } -echo "$COMPOSE_CIDR" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+\.0/24$' || { echo "FAIL: --compose-cidr must be a /24 like 10.10.1.0/24 (got '$COMPOSE_CIDR')" >&2; exit 2; } +case "$ROLE" in + region+rack|rack) ;; + *) echo "FAIL: --role must be 'region+rack' (default) or 'rack' (got '$ROLE')" >&2; exit 2 ;; +esac -BASE="${COMPOSE_CIDR%.0/24}" # 10.10.1 -BRIDGE_IP="${BASE}.1" -RANGE_LO="${BASE}.100" -RANGE_HI="${BASE}.200" +# Per-role REQUIRED args are validated UNCONDITIONALLY here (mirrors the legacy --compose-cidr +# check, which already gated --check too), so a bad invocation is rejected before any mode runs. +if [ "$ROLE" = "rack" ]; then + # rack enrolls to Office1's EXISTING region -- no compose network (LXD-compose is a region + # concern); it needs the region URL + the region's enrollment secret (read from a file, never + # echoed -- same discipline as $SECRETS below; the region operator obtains it from the region + # host's /var/snap/maas/common/maas/secret). + [ -n "$REGION_URL" ] || { echo "FAIL: --role rack REQUIRES --region-url <http://REGION-IP:5240/MAAS>" >&2; exit 2; } + echo "$REGION_URL" | grep -qiE '^https?://.+/MAAS/?$' || { echo "FAIL: --region-url must look like http://REGION-IP:5240/MAAS (got '$REGION_URL')" >&2; exit 2; } + [ -n "$ENROLL_SECRET_FILE" ] || { echo "FAIL: --role rack REQUIRES --enroll-secret-file <path> (the region's enrollment secret; NEVER passed on the command line)" >&2; exit 2; } + # --compose-cidr is NOT used in rack mode; ignore it if present. +else + [ -n "$COMPOSE_CIDR" ] || { echo "FAIL: --compose-cidr is REQUIRED (e.g. 10.10.1.0/24). It has no default on purpose: MAAS must serve DHCP on the compose network ONLY, never on the site LAN, which the edge's Kea already owns." >&2; exit 2; } + echo "$COMPOSE_CIDR" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+\.0/24$' || { echo "FAIL: --compose-cidr must be a /24 like 10.10.1.0/24 (got '$COMPOSE_CIDR')" >&2; exit 2; } + + BASE="${COMPOSE_CIDR%.0/24}" # 10.10.1 + BRIDGE_IP="${BASE}.1" + RANGE_LO="${BASE}.100" + RANGE_HI="${BASE}.200" +fi SECRETS=/root/maas-secrets MAAS_CHANNEL="3.7/stable" PG_CHANNEL="16/stable" @@ -113,14 +162,81 @@ report() { return "$bad" } +# ---------------- check mode, RACK role (read-only) ---------------- +# A rack runs NO PostgreSQL and NO LXD, so we assert only: the maas snap is present and +# `maas status` shows `rackd` running. If the rack has been through `maas init rack`, the +# region URL it enrolled to is recorded (offline-readable) in the rackd config -- report it +# as the connected-to-region signal, best-effort (never fatal; the file is root-only). +report_rack() { + local bad=0 + snap list maas >/dev/null 2>&1 && echo "OK: maas snap $(snap list maas 2>/dev/null | awk 'NR>1{print $2" ("$4")"}')" || { echo "ABSENT: maas snap"; bad=1; } + if have maas && maas status 2>/dev/null | grep -q rackd; then echo "OK: MAAS rackd running"; else echo "ABSENT: MAAS rackd not running (not enrolled to a region)"; bad=1; fi + local ru; ru="$(grep -sh -m1 -E '^maas_url:' /var/snap/maas/current/rackd.conf 2>/dev/null | awk '{print $2}')" + [ -n "$ru" ] && echo "OK: enrolled to region $ru" + return "$bad" +} + +# Dispatch check/precheck to the role-appropriate reporter. +do_report() { if [ "$ROLE" = "rack" ]; then report_rack; else report; fi; } + report_st=0 -if [ "$CHECK" = "1" ]; then report; exit $?; fi -report >/dev/null 2>&1; report_st=$? +if [ "$CHECK" = "1" ]; then do_report; exit $?; fi +do_report >/dev/null 2>&1; report_st=$? [ "$report_st" = "0" ] && [ "$DRYRUN" = "0" ] && { echo "-- site headend already installed and configured --"; exit 0; } have apt-get || { echo "FAIL: Debian/Ubuntu (apt) only." >&2; exit 3; } # root is required to MUTATE, not to plan: --dry-run stays runnable by anyone (and by the harness). [ "$DRYRUN" = "1" ] || [ "$(id -u)" = "0" ] || { echo "FAIL: must run as root (use: ssh <host> 'sudo bash -s' -- ... < $0)" >&2; exit 4; } + +# ===================================================================================== +# RACK ROLE (D-123): enroll a DC rack controller to Office1's EXISTING region. ONLY these +# steps run -- and NONE of the region+rack work below (no PostgreSQL, no `maas init +# region+rack`, no createadmin/apikey/login, no LXD, no LXD vm-host registration, no +# compose-network DHCP). Rationale: a rack runs no DB, composes nothing (the DC's OpenStack +# nodes are OpenTofu-created + discovered via the vcloud virsh vm-host registered to the +# REGION, not composed here), and DHCP-on-metal-admin is a REGION-side op (enabled against +# Office1's MAAS naming this rack as primary_rack -- a runbook step, NOT this script). So +# traps 1-3 (LXD/dnsmasq) do not apply, and trap 4's spirit holds: rack mode touches NO DHCP. +# ===================================================================================== +if [ "$ROLE" = "rack" ]; then + echo "== 1. time: MAAS manages time via chrony; systemd-timesyncd conflicts ==" + run "disable systemd-timesyncd" -- systemctl disable --now systemd-timesyncd >/dev/null 2>&1 + + echo "== 2. MAAS $MAAS_CHANNEL (rack: NO PostgreSQL -- the rack runs no DB) ==" + snap list maas >/dev/null 2>&1 || run "install maas" -- snap install maas --channel="$MAAS_CHANNEL" || exit 4 + + echo "== 3. maas init rack -- enroll to the region at $REGION_URL ==" + # `maas init rack --maas-url <URL> --secret <SECRET>` is the MAAS snap rack-registration + # invocation -- CONFIRMED, not invented: this form is documented across the MAAS snap docs and + # discourse and is stable across 3.x (so it holds for our 3.7/stable pin), and the region's + # shared secret lives at /var/snap/maas/common/maas/secret on the region host (matches the + # runbook). The secret is READ from the file at + # execution time and NEVER echoed -- so this step is hand-rolled rather than routed through + # run() (which would print the expanded command, leaking the secret in --dry-run). + if [ "$DRYRUN" = "0" ]; then + [ -s "$ENROLL_SECRET_FILE" ] || { echo "FAIL: --enroll-secret-file '$ENROLL_SECRET_FILE' is absent or empty (need the region's enrollment secret)" >&2; exit 4; } + if maas status 2>/dev/null | grep -q rackd; then + echo " -> already enrolled (rackd running)" + else + maas init rack --maas-url "$REGION_URL" --secret "$(cat "$ENROLL_SECRET_FILE")" >/dev/null 2>&1 \ + || { echo "FAIL: maas init rack" >&2; exit 4; } + echo " -> rack registered to region $REGION_URL" + fi + else + echo " [dry-run] maas init rack --maas-url $REGION_URL --secret <read from $ENROLL_SECRET_FILE at run time; never printed>" + fi + + # SKIPPED IN RACK MODE (region concerns): PostgreSQL, maas init region+rack, + # createadmin/apikey/login, LXD (5.21 pin), LXD vm-host registration, compose-network DHCP. + # metal-admin DHCP (this rack as primary_rack) is enabled REGION-side as a runbook step. + if [ "$DRYRUN" = "1" ]; then echo " (dry-run: no changes made)"; exit 0; fi + echo + echo "RACK ENROLLED to $REGION_URL. Next (REGION-side runbook, NOT this script): enable" + echo "DHCP on metal-admin against Office1's MAAS, naming this host as primary_rack." + exit 0 +fi + +# ---- region+rack role (default; unchanged legacy behavior) ---- [ -n "$MAAS_IP" ] || MAAS_IP="$(ip -4 route get 1.1.1.1 2>/dev/null | awk '{print $7; exit}')" [ -n "$MAAS_IP" ] || { echo "FAIL: could not measure this host's outbound IP; pass --maas-url-ip" >&2; exit 4; } diff --git a/tests/dc-edge-wan-import/fake_netbox.py b/tests/dc-edge-wan-import/fake_netbox.py new file mode 100644 index 0000000..0dc65fb --- /dev/null +++ b/tests/dc-edge-wan-import/fake_netbox.py @@ -0,0 +1,43 @@ +""" +In-memory stand-in for the NB client in netbox/dc-edge-wan-import.py. + +No real NetBox / network access exists this session -- this fake implements just +the two methods the target script calls on its client (.one() and .create()) so the +REAL main()/preflight can be driven end-to-end without a live server. It is injected +at the get_nb() seam, exactly as tests/dc-dc-prefixes-import/ injects a fake pynetbox. + +FIDELITY NOTE. The target's real .one() issues GET ...?<filter>&limit=1 and takes +results[0] -- so it is single-match by construction and CANNOT raise on a duplicate +prefix (unlike pynetbox.get(), which the sibling fake models raising on multi-match). +This fake matches THAT contract: .one() returns the first record whose fields equal +the filter, or None. For an idempotency SKIP that is the safe direction -- a +pre-existing prefix reads as "present" and is not duplicated. .create() records every +write so a test can assert a rejected/dry run wrote NOTHING. +""" +from __future__ import annotations + + +class FakeNB: + def __init__(self, roles=(), prefixes=(), sites=()): + # store: endpoint path -> list of record dicts (as the API would return them) + self.store = { + "ipam/roles": [dict(r) for r in roles], + "ipam/prefixes": [dict(p) for p in prefixes], + "dcim/sites": [dict(s) for s in sites], + } + self.creates = [] # (path, payload) for every .create() -- the write ledger + self._next_id = 1000 + + def one(self, path, **flt): + for rec in self.store.get(path, []): + if all(str(rec.get(k)) == str(v) for k, v in flt.items()): + return rec + return None + + def create(self, path, payload): + self.creates.append((path, dict(payload))) + rec = dict(payload) + rec["id"] = self._next_id + self._next_id += 1 + self.store.setdefault(path, []).append(rec) + return rec diff --git a/tests/dc-edge-wan-import/run-tests.sh b/tests/dc-edge-wan-import/run-tests.sh new file mode 100644 index 0000000..a5aa1e1 --- /dev/null +++ b/tests/dc-edge-wan-import/run-tests.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# tests/dc-edge-wan-import/run-tests.sh +# +# Harness for netbox/dc-edge-wan-import.py. OFFLINE -- touches no NetBox. +# +# Two layers, both required (mirrors d120-compose-bands + dc-dc-prefixes-import): +# 1. STATIC greps that PIN the D-115 edge values + the guards every sandbox-loop +# writer must have (dry-by-default, upstream-write gated IN CODE, whole-plan +# preflight). A changed constant that a behavioral test would ALSO change is +# caught here. +# 2. test_logic.py drives the REAL main()/preflight against an in-memory fake NB +# (injected at the get_nb() seam) -- half-write guard, scope-not-swapped, +# idempotent-skip, dry-run-writes-nothing. In-process on purpose: no port / +# readiness / cleanup flakiness to redden the shared gauntlet. +# ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +S="$(cd "$HERE/../../netbox" && pwd)/dc-edge-wan-import.py" +pass=0; fail=0 +ok() { pass=$((pass+1)); } +bad() { fail=$((fail+1)); echo " FAIL: $1"; } + +command -v python3 >/dev/null 2>&1 || { echo "FAIL: python3 required"; exit 1; } + +python3 -c "import ast;ast.parse(open('$S').read())" 2>/dev/null && ok || bad "does not parse" +python3 "$S" --help >/dev/null 2>&1 && ok || bad "--help -> 0" + +# no env -> must fail loud, not guess a target +NETBOX_URL= NETBOX_TOKEN= python3 "$S" >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "missing NETBOX_URL/TOKEN must fail" + +# DRY BY DEFAULT +grep -q '"--commit", action="store_true"' "$S" && ok || bad "no --commit flag -- must be dry by default" +grep -q 'DRY RUN -- nothing will be written' "$S" && ok || bad "does not announce its dry run" + +# UPSTREAM WRITE GATED IN CODE (not by discipline) +grep -q 'yes-write-upstream' "$S" && ok || bad "lost the --yes-write-upstream gate" +grep -q 'SANDBOX_HOSTS' "$S" && ok || bad "lost the SANDBOX_HOSTS gate" +out="$(NETBOX_URL=https://netbox.baldurkeep.com NETBOX_TOKEN=x python3 "$S" --commit 2>&1)" +printf '%s' "$out" | grep -q "REFUSING to --commit" && ok \ + || bad "did NOT refuse a --commit to a non-sandbox host (it would have written to production)" + +# WAF: UA-aware or every upstream call 403s (looks like auth failure) +grep -q 'User-Agent' "$S" && grep -q 'curl/8.5.0' "$S" && ok || bad "lost the WAF-safe User-Agent" + +# WHOLE-PLAN PREFLIGHT -- validate role/container/placement/site BEFORE any create +grep -q "outside the Edge container" "$S" && ok || bad "lost the in-container preflight" +grep -q "container .* absent" "$S" && ok || bad "lost the container-exists preflight" +grep -q "role '.*' absent" "$S" && ok || bad "lost the role-exists preflight" +grep -q "site '.*' absent" "$S" && ok || bad "lost the site-resolves preflight" + +# THE D-115 EDGE VALUES -- the two DC /24s, their role, and their SITE scope. +grep -qF '("172.30.2.0/24", "vr1-dc0"' "$S" && ok || bad "172.30.2.0/24 -> vr1-dc0 binding is GONE / changed" +grep -qF '("172.30.3.0/24", "vr1-dc1"' "$S" && ok || bad "172.30.3.0/24 -> vr1-dc1 binding is GONE / changed" +grep -qF 'ROLE_SLUG = "edge"' "$S" && ok || bad "edge role slug changed" +grep -qF 'CONTAINER = "172.30.0.0/16"' "$S" && ok || bad "Edge container is not 172.30.0.0/16" +grep -q '"scope_type": "dcim.site"' "$S" && ok || bad "the /24s are not dcim.site-scoped (must match office1-wan)" + +# This tool must NOT create the role/container/sites -- they are preconditions. +grep -q 'ipam/roles".*payload\|create("ipam/roles"\|create("dcim/sites"' "$S" \ + && bad "this tool CREATES a role/site -- those are preconditions, it must only create the /24s" || ok + +# ---- behavioral layer: drive the real main() against the injected fake NB ---- +echo " -- behavioral (test_logic.py) --" +tl_out="$(python3 "$HERE/test_logic.py" 2>&1)"; tl_rc=$? +printf '%s\n' "$tl_out" | grep -E '^(FAIL|PASS: |test_logic:)' | sed 's/^/ /' +tl_n="$(printf '%s\n' "$tl_out" | sed -n 's/^test_logic: ALL PASS (\([0-9]*\) checks)$/\1/p')" +if [ "$tl_rc" -eq 0 ] && [ -n "$tl_n" ]; then + pass=$((pass + tl_n)) +else + bad "test_logic.py behavioral suite FAILED (rc=$tl_rc)" +fi + +echo +total=$((pass+fail)) +if [ "$fail" -eq 0 ]; then echo "dc-edge-wan-import: $pass/$total PASS"; exit 0; fi +echo "dc-edge-wan-import: $fail/$total FAIL"; exit 1 diff --git a/tests/dc-edge-wan-import/test_logic.py b/tests/dc-edge-wan-import/test_logic.py new file mode 100644 index 0000000..01ac01a --- /dev/null +++ b/tests/dc-edge-wan-import/test_logic.py @@ -0,0 +1,245 @@ +#!/usr/bin/env python3 +""" +Behavioral tests for netbox/dc-edge-wan-import.py -- drives the REAL main()/preflight +against an in-memory fake NB (tests/dc-edge-wan-import/fake_netbox.py) injected at the +get_nb() seam. No live NetBox. Invoked by run-tests.sh alongside the static greps. + +The properties pinned here map to this repo's documented IPAM bug classes: + * HALF-WRITE (roles-aggregates-import.py): a later target's missing precondition + must abort with ZERO creates, never "wrote dc0 then died". + * SWAPPED SCOPE (the D-117 wrong-target near-miss): the vr1-dc0 /24 must bind + vr1-dc0's id and the vr1-dc1 /24 must bind vr1-dc1's id -- not shared or swapped. + * ALREADY-PRESENT = SKIP (idempotent), distinct from missing-infra = die. + * DRY BY DEFAULT: no --commit writes NOTHING. + * UPSTREAM-WRITE guard: --commit to a non-sandbox host refuses without the flag. +""" +from __future__ import annotations + +import contextlib +import faulthandler +import importlib.util +import inspect +import io +import os +import sys + +faulthandler.dump_traceback_later(30, exit=True) + +HERE = os.path.dirname(os.path.abspath(__file__)) +REPO_ROOT = os.path.dirname(os.path.dirname(HERE)) +TARGET_PATH = os.path.join(REPO_ROOT, "netbox", "dc-edge-wan-import.py") +sys.path.insert(0, HERE) + +import fake_netbox # noqa: E402 + +P = 0 +F = 0 + + +def ok(label): + global P + P += 1 + print(f"PASS: {label}") + + +def no(label, detail=""): + global F + F += 1 + print(f"FAIL: {label}" + (f" ({detail})" if detail else "")) + + +def check(cond, label, detail=""): + ok(label) if cond else no(label, detail) + + +def load_target(): + spec = importlib.util.spec_from_file_location("dc_edge_wan_import", TARGET_PATH) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +T = load_target() + + +@contextlib.contextmanager +def captured_stdout(): + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + yield buf + + +def run_main(argv, fake): + T.get_nb = lambda base=None, token=None: fake + old = sys.argv + sys.argv = ["dc-edge-wan-import.py"] + argv + try: + return T.main() + finally: + sys.argv = old + + +def run_dies(label, argv, fake): + # swallow stdout AND stderr -- die() prints "FAIL: ..." to stderr by design, and + # for an EXPECTED death that is not a test failure, just noise on the gauntlet. + try: + with contextlib.redirect_stderr(io.StringIO()), captured_stdout(): + run_main(argv, fake) + no(label, "did not raise SystemExit") + except SystemExit: + ok(label) + + +# Full-precondition fixture builders (role + container + both DC sites present). +ROLE = {"slug": "edge", "name": "Edge", "id": 1} +CONTAINER = {"prefix": "172.30.0.0/16", "id": 10} +SITE_DC0 = {"slug": "vr1-dc0", "name": "VR1 DC0", "id": 20} +SITE_DC1 = {"slug": "vr1-dc1", "name": "VR1 DC1", "id": 21} + + +def full_fake(prefixes_extra=()): + return fake_netbox.FakeNB( + roles=[ROLE], + prefixes=[CONTAINER, *prefixes_extra], + sites=[SITE_DC0, SITE_DC1], + ) + + +os.environ["NETBOX_URL"] = "http://10.10.1.10:8000" # a known sandbox (guard passes) +os.environ["NETBOX_TOKEN"] = "fake-token" + +# ----------------------------------------------------------------------------- +# 1. DRY RUN is the default and writes NOTHING. +# ----------------------------------------------------------------------------- +fk = full_fake() +with captured_stdout() as out: + rc = run_main([], fk) +check(rc == 0, "dry run rc==0", str(rc)) +check(out.getvalue().count("would CREATE") == 2, "dry run PLANS both /24s", + str(out.getvalue().count("would CREATE"))) +check(out.getvalue().count("CREATED ") == 0, "dry run CREATES nothing (no CREATED line)") +check("DRY RUN" in out.getvalue(), "dry run says so, loudly") +check(len(fk.creates) == 0, "dry run left the NetBox untouched (0 writes)", str(len(fk.creates))) + +# ----------------------------------------------------------------------------- +# 2. --commit writes exactly the two /24s, correctly scoped (NO swap). +# ----------------------------------------------------------------------------- +fk = full_fake() +with captured_stdout() as out: + rc = run_main(["--commit"], fk) +check(rc == 0, "commit rc==0", str(rc)) +check(len(fk.creates) == 2, "commit created exactly 2 prefixes", str(len(fk.creates))) +check(all(path == "ipam/prefixes" for path, _ in fk.creates), + "commit wrote only ipam/prefixes (no stray sites/roles/container)") +by_prefix = {pl["prefix"]: pl for _, pl in fk.creates} +check(by_prefix.get("172.30.2.0/24", {}).get("scope_id") == 20, + "SCOPE: 172.30.2.0/24 binds vr1-dc0's id (20)", str(by_prefix.get("172.30.2.0/24"))) +check(by_prefix.get("172.30.3.0/24", {}).get("scope_id") == 21, + "SCOPE: 172.30.3.0/24 binds vr1-dc1's id (21) -- not swapped", + str(by_prefix.get("172.30.3.0/24"))) +check(all(pl.get("scope_type") == "dcim.site" for pl in by_prefix.values()), + "SCOPE: both /24s are dcim.site-scoped (matches office1-wan)") +check(all(pl.get("role") == 1 for pl in by_prefix.values()), "both /24s carry the edge role id") +check(all(pl.get("status") == "active" for pl in by_prefix.values()), "both /24s are status active") + +# ----------------------------------------------------------------------------- +# 3. Already-present = idempotent SKIP (both present -> 0 creates, exit clean). +# ----------------------------------------------------------------------------- +fk = full_fake(prefixes_extra=[{"prefix": "172.30.2.0/24", "id": 30}, + {"prefix": "172.30.3.0/24", "id": 31}]) +with captured_stdout() as out: + rc = run_main(["--commit"], fk) +check(rc == 0, "both-present idempotent run rc==0") +check(out.getvalue().count("EXISTS") == 2, "both-present run reports both as EXISTS") +check(len(fk.creates) == 0, "both-present run creates nothing new") + +# ----------------------------------------------------------------------------- +# 4. Partial-present: dc0 already there, dc1 absent -> creates ONLY dc1. +# ----------------------------------------------------------------------------- +fk = full_fake(prefixes_extra=[{"prefix": "172.30.2.0/24", "id": 30}]) +with captured_stdout(): + rc = run_main(["--commit"], fk) +check(rc == 0, "partial-present run rc==0") +check(len(fk.creates) == 1 and fk.creates[0][1]["prefix"] == "172.30.3.0/24", + "partial-present run creates ONLY the missing 172.30.3.0/24", + str([pl["prefix"] for _, pl in fk.creates])) + +# ----------------------------------------------------------------------------- +# 5. HALF-WRITE GUARD -- a LATER target's missing site aborts with ZERO creates. +# role + container + vr1-dc0 present, but vr1-dc1 SITE absent. +# ----------------------------------------------------------------------------- +fk = fake_netbox.FakeNB(roles=[ROLE], prefixes=[CONTAINER], sites=[SITE_DC0]) +run_dies("half-write: missing vr1-dc1 site is REJECTED before any write", ["--commit"], fk) +check(len(fk.creates) == 0, "half-write: the rejected run wrote NOTHING (vr1-dc0 not created)") + +# ----------------------------------------------------------------------------- +# 6-8. Missing infra = die (each precondition), each writing nothing. +# ----------------------------------------------------------------------------- +fk = fake_netbox.FakeNB(roles=[], prefixes=[CONTAINER], sites=[SITE_DC0, SITE_DC1]) +run_dies("missing edge role is REJECTED", ["--commit"], fk) +check(len(fk.creates) == 0, "missing-role run wrote nothing") + +fk = fake_netbox.FakeNB(roles=[ROLE], prefixes=[], sites=[SITE_DC0, SITE_DC1]) +run_dies("missing 172.30.0.0/16 container is REJECTED", ["--commit"], fk) +check(len(fk.creates) == 0, "missing-container run wrote nothing") + +fk = fake_netbox.FakeNB(roles=[ROLE], prefixes=[CONTAINER], sites=[SITE_DC1]) +run_dies("missing vr1-dc0 site is REJECTED", ["--commit"], fk) +check(len(fk.creates) == 0, "missing-dc0-site run wrote nothing") + +# ----------------------------------------------------------------------------- +# 9. /24 OUTSIDE the container is rejected by the local subnet preflight. +# ----------------------------------------------------------------------------- +_saved = T.TARGETS +T.TARGETS = [("10.0.0.0/24", "vr1-dc0", "bogus -- not inside 172.30.0.0/16")] +try: + run_dies("a /24 outside the Edge container is REJECTED", ["--commit"], full_fake()) +finally: + T.TARGETS = _saved + +# ----------------------------------------------------------------------------- +# 10. Missing NETBOX_URL/TOKEN fails loud (does not guess a target). +# ----------------------------------------------------------------------------- +_u = os.environ.pop("NETBOX_URL", None) +run_dies("missing NETBOX_URL fails loud", [], full_fake()) +os.environ["NETBOX_URL"] = _u + +# ----------------------------------------------------------------------------- +# 11. UPSTREAM-WRITE guard: --commit to a non-sandbox host refuses without the flag, +# proceeds with it. Sandbox needs no flag (covered above). +# ----------------------------------------------------------------------------- +os.environ["NETBOX_URL"] = "https://netbox.baldurkeep.com" # the production apex +fk = full_fake() +run_dies("upstream --commit WITHOUT --yes-write-upstream is REFUSED", ["--commit"], fk) +check(len(fk.creates) == 0, "the refused upstream --commit wrote NOTHING") + +fk = full_fake() +with captured_stdout(): + rc = run_main(["--commit", "--yes-write-upstream"], fk) +check(rc == 0 and len(fk.creates) == 2, + "upstream --commit --yes-write-upstream IS allowed and writes both /24s", + str((rc, len(fk.creates)))) +os.environ["NETBOX_URL"] = "http://10.10.1.10:8000" + +# ----------------------------------------------------------------------------- +# 12. Structural pins (a changed constant a behavioral test alone would miss). +# ----------------------------------------------------------------------------- +check(T.ROLE_SLUG == "edge", "ROLE_SLUG is 'edge'") +check(T.CONTAINER == "172.30.0.0/16", "CONTAINER is 172.30.0.0/16") +check(T.STATUS == "active", "STATUS is active (like office1-wan .1/24)") +check([c for c, _, _ in T.TARGETS] == ["172.30.2.0/24", "172.30.3.0/24"], + "TARGETS pins exactly the two DC /24s .2 and .3") +check([s for _, s, _ in T.TARGETS] == ["vr1-dc0", "vr1-dc1"], + "TARGETS scopes .2->vr1-dc0 and .3->vr1-dc1 (no swap in the source)") +check(T.SANDBOX_HOSTS == {"localhost", "127.0.0.1", "10.10.1.10"}, + "SANDBOX_HOSTS matches the sibling importers") +check("get_nb" in dir(T) and callable(T.get_nb), "get_nb() injection seam exists") +_src = inspect.getsource(T) +check('User-Agent"' in _src and "curl/8.5.0" in _src, "UA-aware (WAF-safe User-Agent set)") + +print() +if F == 0: + print(f"test_logic: ALL PASS ({P} checks)") + sys.exit(0) +print(f"test_logic: {F} FAIL of {P + F}") +sys.exit(1) diff --git a/tests/dc-rack-mgmt-import/fake_netbox.py b/tests/dc-rack-mgmt-import/fake_netbox.py new file mode 100644 index 0000000..09199e8 --- /dev/null +++ b/tests/dc-rack-mgmt-import/fake_netbox.py @@ -0,0 +1,45 @@ +""" +In-memory stand-in for the NB client in netbox/dc-rack-mgmt-import.py. + +No real NetBox / network access exists this session -- this fake implements just +the two methods the target script calls on its client (.one() and .create()) so the +REAL main()/preflight can be driven end-to-end without a live server. It is injected +at the get_nb() seam, exactly as tests/dc-edge-wan-import/ injects its fake. + +FIDELITY NOTE. The target's real .one() issues GET ...?<filter>&limit=1 and takes +results[0] -- so it is single-match by construction and CANNOT raise on a duplicate +(unlike pynetbox.get(), which the older sibling fake models raising on multi-match). +This fake matches THAT contract: .one() returns the first record whose fields equal +the filter, or None. For an idempotency SKIP that is the safe direction -- a +pre-existing object reads as "present" and is not duplicated. .create() records every +write so a test can assert a rejected/dry run wrote NOTHING. The store is keyed by +path, so ipam/prefixes, ipam/roles, dcim/sites AND ipam/ip-addresses all work. +""" +from __future__ import annotations + + +class FakeNB: + def __init__(self, roles=(), prefixes=(), sites=(), ip_addresses=()): + # store: endpoint path -> list of record dicts (as the API would return them) + self.store = { + "ipam/roles": [dict(r) for r in roles], + "ipam/prefixes": [dict(p) for p in prefixes], + "dcim/sites": [dict(s) for s in sites], + "ipam/ip-addresses": [dict(a) for a in ip_addresses], + } + self.creates = [] # (path, payload) for every .create() -- the write ledger + self._next_id = 1000 + + def one(self, path, **flt): + for rec in self.store.get(path, []): + if all(str(rec.get(k)) == str(v) for k, v in flt.items()): + return rec + return None + + def create(self, path, payload): + self.creates.append((path, dict(payload))) + rec = dict(payload) + rec["id"] = self._next_id + self._next_id += 1 + self.store.setdefault(path, []).append(rec) + return rec diff --git a/tests/dc-rack-mgmt-import/run-tests.sh b/tests/dc-rack-mgmt-import/run-tests.sh new file mode 100644 index 0000000..d8ca624 --- /dev/null +++ b/tests/dc-rack-mgmt-import/run-tests.sh @@ -0,0 +1,93 @@ +#!/usr/bin/env bash +# tests/dc-rack-mgmt-import/run-tests.sh +# +# Harness for netbox/dc-rack-mgmt-import.py. OFFLINE -- touches no NetBox. +# +# Two layers, both required (mirrors dc-edge-wan-import + d120-compose-bands): +# 1. STATIC greps that PIN the D-124 scheme values + the guards every sandbox-loop +# writer must have (dry-by-default, upstream-write gated IN CODE, whole-plan +# preflight, NO invented literal). A changed constant a behavioral test would ALSO +# change is caught here. +# 2. test_logic.py drives the REAL main()/preflight against an in-memory fake NB +# (injected at the get_nb() seam) -- half-write guard, role/scope binding, band +# arithmetic, idempotent-skip, dry-run-writes-nothing. In-process on purpose: no +# port / readiness / cleanup flakiness to redden the shared gauntlet. +# ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +S="$(cd "$HERE/../../netbox" && pwd)/dc-rack-mgmt-import.py" +pass=0; fail=0 +ok() { pass=$((pass+1)); } +bad() { fail=$((fail+1)); echo " FAIL: $1"; } + +command -v python3 >/dev/null 2>&1 || { echo "FAIL: python3 required"; exit 1; } + +python3 -c "import ast;ast.parse(open('$S').read())" 2>/dev/null && ok || bad "does not parse" +python3 "$S" --help >/dev/null 2>&1 && ok || bad "--help -> 0" + +# no env -> must fail loud, not guess a target +NETBOX_URL= NETBOX_TOKEN= python3 "$S" --transit-cidr 10.12.60.0/30 --rack-ip 10.12.8.5 \ + >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "missing NETBOX_URL/TOKEN must fail" + +# NO INVENTED LITERAL -- both inputs are REQUIRED (no default CIDR/IP baked in) +NETBOX_URL=http://10.10.1.10:8000 NETBOX_TOKEN=x python3 "$S" --rack-ip 10.12.8.5 \ + >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "missing --transit-cidr must fail (no invented CIDR)" +NETBOX_URL=http://10.10.1.10:8000 NETBOX_TOKEN=x python3 "$S" --transit-cidr 10.12.60.0/30 \ + >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "missing --rack-ip must fail (no invented IP)" +grep -q 'default=os.environ.get("TRANSIT_CIDR")' "$S" && ok || bad "lost the TRANSIT_CIDR env fallback" +grep -q 'default=os.environ.get("RACK_IP")' "$S" && ok || bad "lost the RACK_IP env fallback" +# guard against a baked-in transit /30 or /31 literal masquerading as a default +grep -qE '=\s*"10\.[0-9]+\.[0-9]+\.[0-9]+/3[01]"' "$S" && bad "a /30 or /31 literal is baked in -- must be an INPUT" || ok + +# DRY BY DEFAULT +grep -q '"--commit", action="store_true"' "$S" && ok || bad "no --commit flag -- must be dry by default" +grep -q 'DRY RUN -- nothing will be written' "$S" && ok || bad "does not announce its dry run" + +# UPSTREAM WRITE GATED IN CODE (not by discipline) +grep -q 'yes-write-upstream' "$S" && ok || bad "lost the --yes-write-upstream gate" +grep -q 'SANDBOX_HOSTS' "$S" && ok || bad "lost the SANDBOX_HOSTS gate" +out="$(NETBOX_URL=https://netbox.baldurkeep.com NETBOX_TOKEN=x python3 "$S" \ + --transit-cidr 10.12.60.0/30 --rack-ip 10.12.8.5 --commit 2>&1)" +printf '%s' "$out" | grep -q "REFUSING to --commit" && ok \ + || bad "did NOT refuse a --commit to a non-sandbox host (it would have written to production)" + +# WAF: UA-aware or every upstream call 403s (looks like auth failure) +grep -q 'User-Agent' "$S" && grep -q 'curl/8.5.0' "$S" && ok || bad "lost the WAF-safe User-Agent" + +# WHOLE-PLAN PREFLIGHT -- validate role/container/transit/band/site BEFORE any create +grep -q "outside the container" "$S" && ok || bad "lost the transit-in-container preflight" +grep -q "container .* absent" "$S" && ok || bad "lost the container-exists preflight" +grep -q "role '.*' absent" "$S" && ok || bad "lost the role-exists preflight" +grep -q "site '.*' absent" "$S" && ok || bad "lost the site-resolves preflight" +grep -q "GATEWAY" "$S" && ok || bad "lost the .1-gateway rejection" +grep -q "static band" "$S" && ok || bad "lost the D-120 static-band check" +grep -q "outside metal-admin" "$S" && ok || bad "lost the metal-admin containment check" +grep -q "must be a /30 or /31" "$S" && ok || bad "lost the /30-or-/31 shape check" + +# THE D-124 SCHEME VALUES -- role, container, site scope, metal-admin band, rack dns. +grep -qF 'ROLE_SLUG = "transit"' "$S" && ok || bad "transit role slug changed" +grep -qF 'CONTAINER = "10.12.0.0/16"' "$S" && ok || bad "container is not 10.12.0.0/16 (Cloud)" +grep -qF 'SITE_SLUG = "vr1-dc0"' "$S" && ok || bad "transit site scope is not vr1-dc0" +grep -qF 'METAL_ADMIN = "10.12.8.0/22"' "$S" && ok || bad "metal-admin is not 10.12.8.0/22" +grep -qF 'RACK_DNS = "vvr1-dc0"' "$S" && ok || bad "rack dns name is not vvr1-dc0" +grep -q '"scope_type": "dcim.site"' "$S" && ok || bad "the transit prefix is not dcim.site-scoped" + +# This tool must NOT create the role/container/site -- they are preconditions. +grep -q 'create("ipam/roles"\|create("dcim/sites"' "$S" \ + && bad "this tool CREATES a role/site -- those are preconditions" || ok + +# ---- behavioral layer: drive the real main() against the injected fake NB ---- +echo " -- behavioral (test_logic.py) --" +tl_out="$(python3 "$HERE/test_logic.py" 2>&1)"; tl_rc=$? +printf '%s\n' "$tl_out" | grep -E '^(FAIL|PASS: |test_logic:)' | sed 's/^/ /' +tl_n="$(printf '%s\n' "$tl_out" | sed -n 's/^test_logic: ALL PASS (\([0-9]*\) checks)$/\1/p')" +if [ "$tl_rc" -eq 0 ] && [ -n "$tl_n" ]; then + pass=$((pass + tl_n)) +else + bad "test_logic.py behavioral suite FAILED (rc=$tl_rc)" +fi + +echo +total=$((pass+fail)) +if [ "$fail" -eq 0 ]; then echo "dc-rack-mgmt-import: $pass/$total PASS"; exit 0; fi +echo "dc-rack-mgmt-import: $fail/$total FAIL"; exit 1 diff --git a/tests/dc-rack-mgmt-import/test_logic.py b/tests/dc-rack-mgmt-import/test_logic.py new file mode 100644 index 0000000..8ec679e --- /dev/null +++ b/tests/dc-rack-mgmt-import/test_logic.py @@ -0,0 +1,345 @@ +#!/usr/bin/env python3 +""" +Behavioral tests for netbox/dc-rack-mgmt-import.py -- drives the REAL main()/preflight +against an in-memory fake NB (tests/dc-rack-mgmt-import/fake_netbox.py) injected at the +get_nb() seam. No live NetBox. Invoked by run-tests.sh alongside the static greps. + +The properties pinned here map to this repo's documented IPAM bug classes + the D-124 +scheme (Scheme A -- transit-numbered mesh): + * HALF-WRITE (roles-aggregates-import.py): a bad rack IP (the .1 gateway) must abort + with ZERO creates, never "wrote the transit prefix then died on the rack IP". + * ROLE/SCOPE binding: the transit prefix carries the `transit` role id and is + dcim.site-scoped to vr1-dc0; the rack IP carries NO ipam role (NetBox ip-address + `role` is a choice field) -- identity is in dns_name/description. + * BAND arithmetic: rack IP within metal-admin 10.12.8.0/22, D-120 static band .2-.49, + NOT the .1 gateway, NOT .0. + * TRANSIT shape: /30 or /31, host-bits-clear, subnet_of the Cloud container. + * ALREADY-PRESENT = SKIP (idempotent), distinct from missing-infra = die. + * DRY BY DEFAULT: no --commit writes NOTHING. + * UPSTREAM-WRITE guard: --commit to a non-sandbox host refuses without the flag. + * NO INVENTED LITERAL: missing --transit-cidr / --rack-ip fails loud. +""" +from __future__ import annotations + +import contextlib +import faulthandler +import importlib.util +import inspect +import io +import os +import sys + +faulthandler.dump_traceback_later(30, exit=True) + +HERE = os.path.dirname(os.path.abspath(__file__)) +REPO_ROOT = os.path.dirname(os.path.dirname(HERE)) +TARGET_PATH = os.path.join(REPO_ROOT, "netbox", "dc-rack-mgmt-import.py") +sys.path.insert(0, HERE) + +import fake_netbox # noqa: E402 + +P = 0 +F = 0 + + +def ok(label): + global P + P += 1 + print(f"PASS: {label}") + + +def no(label, detail=""): + global F + F += 1 + print(f"FAIL: {label}" + (f" ({detail})" if detail else "")) + + +def check(cond, label, detail=""): + ok(label) if cond else no(label, detail) + + +def load_target(): + spec = importlib.util.spec_from_file_location("dc_rack_mgmt_import", TARGET_PATH) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +T = load_target() + + +@contextlib.contextmanager +def captured_stdout(): + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + yield buf + + +# The operator-supplied, NetBox-assigned example values under test. Chosen INSIDE the +# scheme: transit /30 subnet_of Cloud 10.12.0.0/16; rack IP in the .2-.49 static band. +TRANSIT = "10.12.60.0/30" +TRANSIT31 = "10.12.60.0/31" +RACK = "10.12.8.5" +RACK_ADDR = "10.12.8.5/22" # how the tool stores it (host + metal-admin /22 mask) + + +def run_main(argv, fake): + T.get_nb = lambda base=None, token=None: fake + old = sys.argv + sys.argv = ["dc-rack-mgmt-import.py"] + argv + try: + return T.main() + finally: + sys.argv = old + + +def run_dies(label, argv, fake): + # swallow stdout AND stderr -- die() prints "FAIL: ..." to stderr by design, and + # for an EXPECTED death that is not a test failure, just noise on the gauntlet. + try: + with contextlib.redirect_stderr(io.StringIO()), captured_stdout(): + run_main(argv, fake) + no(label, "did not raise SystemExit") + except SystemExit: + ok(label) + + +# Full-precondition fixture builders (transit role + Cloud container + vr1-dc0 site). +ROLE = {"slug": "transit", "name": "Transit", "id": 7} +CONTAINER = {"prefix": "10.12.0.0/16", "id": 10} +SITE_DC0 = {"slug": "vr1-dc0", "name": "VR1 DC0", "id": 20} + + +def full_fake(prefixes_extra=(), ip_extra=()): + return fake_netbox.FakeNB( + roles=[ROLE], + prefixes=[CONTAINER, *prefixes_extra], + sites=[SITE_DC0], + ip_addresses=list(ip_extra), + ) + + +def base_args(extra=()): + return ["--transit-cidr", TRANSIT, "--rack-ip", RACK, *extra] + + +os.environ["NETBOX_URL"] = "http://10.10.1.10:8000" # a known sandbox (guard passes) +os.environ["NETBOX_TOKEN"] = "fake-token" + +# ----------------------------------------------------------------------------- +# 1. DRY RUN is the default and writes NOTHING. +# ----------------------------------------------------------------------------- +fk = full_fake() +with captured_stdout() as out: + rc = run_main(base_args(), fk) +check(rc == 0, "dry run rc==0", str(rc)) +check(out.getvalue().count("would CREATE") == 2, "dry run PLANS both objects", + str(out.getvalue().count("would CREATE"))) +check(out.getvalue().count("CREATED ") == 0, "dry run CREATES nothing (no CREATED line)") +check("DRY RUN" in out.getvalue(), "dry run says so, loudly") +check(len(fk.creates) == 0, "dry run left the NetBox untouched (0 writes)", str(len(fk.creates))) + +# ----------------------------------------------------------------------------- +# 2. --commit writes exactly the transit prefix + the rack IP, correctly bound. +# ----------------------------------------------------------------------------- +fk = full_fake() +with captured_stdout(): + rc = run_main(base_args(["--commit"]), fk) +check(rc == 0, "commit rc==0", str(rc)) +check(len(fk.creates) == 2, "commit created exactly 2 objects", str(len(fk.creates))) +paths = [p for p, _ in fk.creates] +check(paths.count("ipam/prefixes") == 1, "commit wrote exactly one ipam/prefixes") +check(paths.count("ipam/ip-addresses") == 1, "commit wrote exactly one ipam/ip-addresses") +check("ipam/roles" not in paths and "dcim/sites" not in paths, + "commit created NO role / site (those are preconditions)") +pfx = next(pl for pp, pl in fk.creates if pp == "ipam/prefixes") +ipa = next(pl for pp, pl in fk.creates if pp == "ipam/ip-addresses") +check(pfx.get("prefix") == TRANSIT, "transit prefix is the supplied CIDR", str(pfx.get("prefix"))) +check(pfx.get("role") == 7, "transit prefix carries the transit role id (7)", str(pfx.get("role"))) +check(pfx.get("scope_type") == "dcim.site", "transit prefix is dcim.site-scoped", + str(pfx.get("scope_type"))) +check(pfx.get("scope_id") == 20, "transit prefix binds vr1-dc0's site id (20)", str(pfx.get("scope_id"))) +check(pfx.get("status") == "active", "transit prefix is status active") +check(ipa.get("address") == RACK_ADDR, "rack IP stored with the /22 metal-admin mask", + str(ipa.get("address"))) +check("role" not in ipa, "rack IP carries NO ipam role (NetBox ip role is a choice field)") +check(ipa.get("dns_name") == "vvr1-dc0", "rack IP dns_name identifies the rack controller vvr1-dc0", + str(ipa.get("dns_name"))) +check("rack controller" in ipa.get("description", "").lower(), + "rack IP description notes it is the vr1-dc0 MAAS rack controller") +check(ipa.get("status") == "active", "rack IP is status active") + +# ----------------------------------------------------------------------------- +# 3. A /31 transit is also accepted (point-to-point). +# ----------------------------------------------------------------------------- +fk = full_fake() +with captured_stdout(): + rc = run_main(["--transit-cidr", TRANSIT31, "--rack-ip", RACK, "--commit"], fk) +check(rc == 0 and len(fk.creates) == 2, "a /31 transit is accepted and writes both objects", + str((rc, len(fk.creates)))) + +# ----------------------------------------------------------------------------- +# 4. Already-present = idempotent SKIP (both present -> 0 creates, exit clean). +# ----------------------------------------------------------------------------- +fk = full_fake(prefixes_extra=[{"prefix": TRANSIT, "id": 30}], + ip_extra=[{"address": RACK_ADDR, "id": 31}]) +with captured_stdout() as out: + rc = run_main(base_args(["--commit"]), fk) +check(rc == 0, "both-present idempotent run rc==0") +check(out.getvalue().count("EXISTS") == 2, "both-present run reports both as EXISTS") +check(len(fk.creates) == 0, "both-present run creates nothing new") + +# ----------------------------------------------------------------------------- +# 5. Partial-present: transit already there, rack IP absent -> creates ONLY the rack IP. +# ----------------------------------------------------------------------------- +fk = full_fake(prefixes_extra=[{"prefix": TRANSIT, "id": 30}]) +with captured_stdout(): + rc = run_main(base_args(["--commit"]), fk) +check(rc == 0 and len(fk.creates) == 1 and fk.creates[0][0] == "ipam/ip-addresses", + "transit-present run creates ONLY the missing rack IP", + str([p for p, _ in fk.creates])) + +# 5b. Partial-present the other way: rack IP present, transit absent -> creates ONLY transit. +fk = full_fake(ip_extra=[{"address": RACK_ADDR, "id": 31}]) +with captured_stdout(): + rc = run_main(base_args(["--commit"]), fk) +check(rc == 0 and len(fk.creates) == 1 and fk.creates[0][0] == "ipam/prefixes", + "rack-IP-present run creates ONLY the missing transit prefix", + str([p for p, _ in fk.creates])) + +# ----------------------------------------------------------------------------- +# 6. HALF-WRITE GUARD -- a bad rack IP (the .1 gateway) aborts with ZERO creates, +# even though the transit prefix itself is perfectly valid. The transit must NOT +# be written before the rack-IP check fails. +# ----------------------------------------------------------------------------- +fk = full_fake() +run_dies("half-write: rack IP == .1 gateway is REJECTED before any write", + ["--transit-cidr", TRANSIT, "--rack-ip", "10.12.8.1", "--commit"], fk) +check(len(fk.creates) == 0, "half-write: the rejected run wrote NOTHING (transit not created)") + +# ----------------------------------------------------------------------------- +# 7. Missing infra = die (each precondition), each writing nothing. +# ----------------------------------------------------------------------------- +fk = fake_netbox.FakeNB(roles=[], prefixes=[CONTAINER], sites=[SITE_DC0]) +run_dies("missing transit role is REJECTED", base_args(["--commit"]), fk) +check(len(fk.creates) == 0, "missing-role run wrote nothing") + +fk = fake_netbox.FakeNB(roles=[ROLE], prefixes=[], sites=[SITE_DC0]) +run_dies("missing Cloud container 10.12.0.0/16 is REJECTED", base_args(["--commit"]), fk) +check(len(fk.creates) == 0, "missing-container run wrote nothing") + +fk = fake_netbox.FakeNB(roles=[ROLE], prefixes=[CONTAINER], sites=[]) +run_dies("missing vr1-dc0 site is REJECTED", base_args(["--commit"]), fk) +check(len(fk.creates) == 0, "missing-site run wrote nothing") + +# ----------------------------------------------------------------------------- +# 8. Transit CIDR shape / placement rejects. +# ----------------------------------------------------------------------------- +run_dies("a transit OUTSIDE the Cloud container is REJECTED", + ["--transit-cidr", "192.168.0.0/30", "--rack-ip", RACK, "--commit"], full_fake()) +run_dies("a /29 transit is REJECTED (not point-to-point)", + ["--transit-cidr", "10.12.60.0/29", "--rack-ip", RACK, "--commit"], full_fake()) +run_dies("a /32 transit is REJECTED", + ["--transit-cidr", "10.12.60.1/32", "--rack-ip", RACK, "--commit"], full_fake()) +run_dies("a host-bits-set transit (10.12.60.1/30) is REJECTED", + ["--transit-cidr", "10.12.60.1/30", "--rack-ip", RACK, "--commit"], full_fake()) +run_dies("a non-CIDR transit is REJECTED", + ["--transit-cidr", "not-a-cidr", "--rack-ip", RACK, "--commit"], full_fake()) +# each of the above must have written nothing -- prove it once with a fresh fake +_fk = full_fake() +run_dies("bad transit writes nothing (proof)", + ["--transit-cidr", "10.12.60.0/29", "--rack-ip", RACK, "--commit"], _fk) +check(len(_fk.creates) == 0, "the rejected bad-transit run wrote NOTHING") + +# ----------------------------------------------------------------------------- +# 9. Rack-IP band rejects. +# ----------------------------------------------------------------------------- +run_dies("rack IP OUTSIDE metal-admin 10.12.8.0/22 is REJECTED", + ["--transit-cidr", TRANSIT, "--rack-ip", "10.13.0.5", "--commit"], full_fake()) +run_dies("rack IP == .1 gateway is REJECTED", + ["--transit-cidr", TRANSIT, "--rack-ip", "10.12.8.1", "--commit"], full_fake()) +run_dies("rack IP == .0 network address is REJECTED", + ["--transit-cidr", TRANSIT, "--rack-ip", "10.12.8.0", "--commit"], full_fake()) +run_dies("rack IP in-/22-but-above-static-band (.50) is REJECTED", + ["--transit-cidr", TRANSIT, "--rack-ip", "10.12.8.50", "--commit"], full_fake()) +run_dies("rack IP in-/22-but-in-a-higher-/24 (10.12.9.5) is REJECTED", + ["--transit-cidr", TRANSIT, "--rack-ip", "10.12.9.5", "--commit"], full_fake()) +run_dies("rack IP with a non-/22 mask is REJECTED", + ["--transit-cidr", TRANSIT, "--rack-ip", "10.12.8.5/24", "--commit"], full_fake()) +# accepted: the boundaries of the static band (.2 low, .49 high) both write. +for edge_ip in ("10.12.8.2", "10.12.8.49"): + fk = full_fake() + with captured_stdout(): + rc = run_main(["--transit-cidr", TRANSIT, "--rack-ip", edge_ip, "--commit"], fk) + check(rc == 0 and len(fk.creates) == 2, f"rack IP band edge {edge_ip} is ACCEPTED", + str((rc, len(fk.creates)))) + +# ----------------------------------------------------------------------------- +# 10. NO INVENTED LITERAL -- missing input fails loud (does not guess a value). +# ----------------------------------------------------------------------------- +run_dies("missing --transit-cidr fails loud", ["--rack-ip", RACK], full_fake()) +run_dies("missing --rack-ip fails loud", ["--transit-cidr", TRANSIT], full_fake()) + +# env fallback works (args-or-env): both via env, no flags. +os.environ["TRANSIT_CIDR"] = TRANSIT +os.environ["RACK_IP"] = RACK +fk = full_fake() +with captured_stdout(): + rc = run_main(["--commit"], fk) +check(rc == 0 and len(fk.creates) == 2, "env TRANSIT_CIDR/RACK_IP are honored (args-or-env)", + str((rc, len(fk.creates)))) +del os.environ["TRANSIT_CIDR"] +del os.environ["RACK_IP"] + +# ----------------------------------------------------------------------------- +# 11. Missing NETBOX_URL/TOKEN fails loud (does not guess a target). +# ----------------------------------------------------------------------------- +_u = os.environ.pop("NETBOX_URL", None) +run_dies("missing NETBOX_URL fails loud", base_args(), full_fake()) +os.environ["NETBOX_URL"] = _u + +# ----------------------------------------------------------------------------- +# 12. UPSTREAM-WRITE guard: --commit to a non-sandbox host refuses without the flag, +# proceeds with it. Sandbox needs no flag (covered above). +# ----------------------------------------------------------------------------- +os.environ["NETBOX_URL"] = "https://netbox.baldurkeep.com" # the production apex +fk = full_fake() +run_dies("upstream --commit WITHOUT --yes-write-upstream is REFUSED", base_args(["--commit"]), fk) +check(len(fk.creates) == 0, "the refused upstream --commit wrote NOTHING") + +fk = full_fake() +with captured_stdout(): + rc = run_main(base_args(["--commit", "--yes-write-upstream"]), fk) +check(rc == 0 and len(fk.creates) == 2, + "upstream --commit --yes-write-upstream IS allowed and writes both objects", + str((rc, len(fk.creates)))) +# a DRY upstream run needs no flag and writes nothing (the guard is a --commit gate). +fk = full_fake() +with captured_stdout(): + rc = run_main(base_args(), fk) +check(rc == 0 and len(fk.creates) == 0, "a DRY upstream run is fine and writes nothing") +os.environ["NETBOX_URL"] = "http://10.10.1.10:8000" + +# ----------------------------------------------------------------------------- +# 13. Structural pins (a changed constant a behavioral test alone would miss). +# ----------------------------------------------------------------------------- +check(T.ROLE_SLUG == "transit", "ROLE_SLUG is 'transit'") +check(T.CONTAINER == "10.12.0.0/16", "CONTAINER is 10.12.0.0/16 (Cloud)") +check(T.SITE_SLUG == "vr1-dc0", "SITE_SLUG is vr1-dc0 (transit site scope)") +check(T.METAL_ADMIN == "10.12.8.0/22", "METAL_ADMIN is 10.12.8.0/22") +check(T.STATIC_BAND_LOW == 2 and T.STATIC_BAND_HIGH == 49, + "static band is .2-.49 (D-120)") +check(T.STATUS == "active", "STATUS is active") +check(T.RACK_DNS == "vvr1-dc0", "RACK_DNS is vvr1-dc0") +check(T.SANDBOX_HOSTS == {"localhost", "127.0.0.1", "10.10.1.10"}, + "SANDBOX_HOSTS matches the sibling importers") +check("get_nb" in dir(T) and callable(T.get_nb), "get_nb() injection seam exists") +_src = inspect.getsource(T) +check('User-Agent"' in _src and "curl/8.5.0" in _src, "UA-aware (WAF-safe User-Agent set)") + +print() +if F == 0: + print(f"test_logic: ALL PASS ({P} checks)") + sys.exit(0) +print(f"test_logic: {F} FAIL of {P + F}") +sys.exit(1) diff --git a/tests/site-headend-install/run-tests.sh b/tests/site-headend-install/run-tests.sh index 4aa5c18..c954a6a 100644 --- a/tests/site-headend-install/run-tests.sh +++ b/tests/site-headend-install/run-tests.sh @@ -68,6 +68,36 @@ grep -q 'is ON for a subnet that is NOT the compose network' "$S" \ # has no egress, and the first apt install on it hangs. Cost a redeploy on 2026-07-13. grep -q 'subnet update .*gateway_ip' "$S" && ok || bad "trap 5 LOST: gateway_ip is never set on the compose subnet -- deployed machines will have NO default route" +# 6. RACK ROLE (D-123: ONE region on Office1 + a rack controller per DC). --role rack enrolls +# to the EXISTING region and must SKIP every region+rack concern (DB/LXD/compose). These +# tests are the point of the rack mode: a regression that quietly ran a region step, or that +# leaked the enrollment secret, would not surface until a live DC build. +RURL="http://10.10.0.20:5240/MAAS" +t "bad --role -> 2" 2 bash "$S" --role bogus --dry-run +# rack REQUIRES --region-url + --enroll-secret-file (missing -> 2), mirroring --compose-cidr. +t "rack missing --region-url -> 2" 2 bash "$S" --role rack --dry-run --enroll-secret-file /nonexistent +t "rack missing --enroll-secret-file -> 2" 2 bash "$S" --role rack --dry-run --region-url "$RURL" +t "rack bad --region-url -> 2" 2 bash "$S" --role rack --dry-run --region-url ftp://x --enroll-secret-file /nonexistent +# --compose-cidr is NOT required in rack mode (LXD-compose is a region concern). +SECFILE="$(mktemp)"; SENTINEL="RACK_ENROLL_SENTINEL_9x7"; printf '%s\n' "$SENTINEL" > "$SECFILE" +trap 'rm -f "$SECFILE"' EXIT +t "rack --dry-run (no --compose-cidr) -> 0" 0 bash "$S" --role rack --dry-run --region-url "$RURL" --enroll-secret-file "$SECFILE" + +rackout="$(bash "$S" --role rack --dry-run --region-url "$RURL" --enroll-secret-file "$SECFILE" 2>&1)" +printf '%s' "$rackout" | grep -q 'maas init rack' && ok || bad "rack --dry-run must print the 'maas init rack' step" +# The enrollment secret must NEVER appear in output (read from the file at run time, never echoed). +printf '%s' "$rackout" | grep -q "$SENTINEL" && bad "rack --dry-run LEAKED the enrollment secret into output" || ok +# rack mode must NOT run any region+rack / LXD / compose step. +for m in 'lxdbr0' 'region+rack' 'compose' 'vm-host'; do + printf '%s' "$rackout" | grep -qi "$m" && bad "rack --dry-run printed a region+rack-only step: '$m'" || ok +done + +# 7. BACKWARD COMPAT: with no --role, the default is region+rack and --compose-cidr is still +# REQUIRED (a rack-mode regression must not relax the legacy contract). +t "default role still needs --compose-cidr -> 2" 2 bash "$S" --dry-run +defout="$(bash "$S" --dry-run --compose-cidr 10.10.1.0/24 2>&1)" +printf '%s' "$defout" | grep -q 'region+rack' && ok || bad "default (no --role) must still run the region+rack flow" + echo total=$((pass+fail)) if [ "$fail" -eq 0 ]; then echo "site-headend-install: $pass/$total PASS"; exit 0; fi