diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index d669de1..f3c3dd1 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -475,7 +475,7 @@ | G9 | DC0 outer apply (deploy step A) | [V] operator-gated, logged (`run-logged.sh`), after G1-G8; audit exit criteria met (charter Phase 6). SEC pre-apply dependency (S2): SEC-010's transit FORWARD-drop is applied+verified at deploy step B via `site-headend-install.sh --host-nodes --check` on vvr1-dc0 (gate G10) -- the ONLY SEC row gated on this apply (register of record: security-ledger). CANONICAL ENTRY DOC (probe hole H1): `runbooks/dc-dc-phase2-tofu-dc-substrate.md`, with `docs/dc0-deploy-readiness.md` section E as the step table | operator | CLOSED 2026-07-19: G8 same-session planes check passed (6x 0 leases, 0 attachments); saved plan == 6/0/6 applied in the logged dc0-deploy window; convergence re-plan = no differences; vvr1-dc0 running, prior guests untouched | | G10 | Deploy steps B-E in-sequence gates: SEC-010 `--host-nodes --check` on vvr1-dc0; depth-4 nested boot; D-125 foreign-MAC egress test; MAAS reachability + `TF_VAR_maas_api_key` before step D; netem placeholder step E | [V] exercised during the gated deploy | session (each mutation operator-approved) | Step B DONE 2026-07-20 (`--check` EXIT 0 incl. SEC-010, `docs/audit/stepB-check-20260720-final.txt`; interfaces enp1s0/enp2s0). Depth-4 nested boot DONE (10 domains running inside vvr1-dc0). D-125 egress isolation test PASS 2026-07-20 (`docs/audit/d125-egress-gate-20260720-matrix.txt`), and the edge itself now egresses 0% loss after the v4 addressing. Step D COMPLETE incl. commissioning: ALL 9 NODES READY 2026-07-21 (two stacked faults diagnosed + fixed -- `docs/audit/commissioning-diag-20260721.txt`; section 1). Step E (netem) DONE 2026-07-21: sudo fragment installed+verified, module local-mode amendment, targeted apply 1/0/0 exact (operator-ruled at the 1/1/0 STOP), placeholder profile live on virbr5, virbr7/virbr3 untouched (`docs/audit/stepE-netem-20260721.txt` + `outer-{plan,apply}-20260721-netem*.txt`). **G10 CLOSED 2026-07-21** | | G11 | Operator signs THIS document | [R] read top-to-bottom; discrepancies resolved in the document | operator | CLOSED: RE-SIGNED 2026-07-19 at audit exit, section 11 (replaces the 2026-07-18 signature) | -| G12 | `vr1-dc1` build | [R] operator rules dc1 transit/rack addressing; then vars + substrate authored | operator + session | OPEN -- [R] leg CLOSED 2026-07-21: addressing RATIFIED (D-124 amendment 2026-07-21, utterance quoted). [V] leg IN PROGRESS (branch `dc-dc-g12-dc1-substrate`): apex confirm-free DONE 2026-07-21 -- planes/uplink already assigned+consistent, transit 172.31.0.4/30 + rack 10.12.68.2 FREE (`docs/audit/dc1-apex-confirm-20260721.txt`); importer per-site dc1 support shipped (harness 117/117) with live dry-run preflight PASS (`docs/audit/dc1-rack-import-dryrun-20260721.txt`). vars + substrate root + lib-net dc1 arm AUTHORED 2026-07-21 but IN TREE UNCOMMITTED (session disconnected mid-item; drop-state record: changelog-20260721-g12-dc1-authoring.md item 3): dc-selector 40/40 but gauntlet **6/76 FAILED** -- six harnesses still assert the retired vr1-dc1 fail-loud selector arm (`docs/audit/gauntlet-20260722-g12-dropstate.txt`). Owed: harness reconcile -> gauntlet ALL GREEN -> commit+push. PENDING after that: operator-gated apex `--commit` (transit /30 + rack IP -- the only owed apex write), then tfvars + gated build | +| G12 | `vr1-dc1` build | [R] operator rules dc1 transit/rack addressing; then vars + substrate authored | operator + session | OPEN -- [R] leg CLOSED 2026-07-21: addressing RATIFIED (D-124 amendment 2026-07-21, utterance quoted). [V] leg IN PROGRESS (branch `dc-dc-g12-dc1-substrate`): apex confirm-free DONE 2026-07-21 -- planes/uplink already assigned+consistent, transit 172.31.0.4/30 + rack 10.12.68.2 FREE (`docs/audit/dc1-apex-confirm-20260721.txt`); importer per-site dc1 support shipped (harness 117/117) with live dry-run preflight PASS (`docs/audit/dc1-rack-import-dryrun-20260721.txt`). vars + substrate root + lib-net dc1 arm COMMITTED 2026-07-22 (successor session landed the disconnected item 3 + the harness reconcile as changelog item 4): six harnesses reconciled to the ratified dc1 arm, phase-00 PLANES parity guard added, rbd-mirror/radosgw cross-DC reminder fixed; gauntlet **ALL GREEN (76)** (`docs/audit/gauntlet-20260722-g12-reconcile.txt`), repo-lint 0-fail. PENDING: operator-gated apex `--commit` (transit /30 + rack IP -- the only owed apex write), then tfvars + gated build | | G13 | D-129 residuals | [R] operator-gated live plugin install on office1-opnsense; qga channel retrofit at that edge's next scheduled restart. All 4 sub-decisions RULED 2026-07-21 (D-129 Status line) -- only the two execution items remain | operator | OPEN (execution only; decision content complete) | | G14 | 9 OPEN SEC rows (SEC-001, -003..-008, plus SEC-012 + SEC-013 opened 2026-07-20 for credentials this deploy created; SEC-010 CLOSED 2026-07-20, operator-ruled, applied+verified both transit ends) | [R] per-row: rotations/flips at v1 close (external to VR1 track); SEC-012 also carries a SCOPE question (libvirt-group grant is broader than the power verbs MAAS needs), SEC-013 is tied to whether `opentofu/vr1-dc0-maas` is retired | operator / external | `docs/security-ledger.md` (register of record, GA-R4/F3); count re-verified vs `bash scripts/ledger-scan.sh` 2026-07-20 | | G15 | D-068 / D-071 rulings | [R] operator rules (section 8); neither blocks the VR1 substrate | operator | D-071 ADOPTED 2026-07-21 (all four points); D-068 remains PROPOSED/OPEN (items 2-3 + the item-1 re-scoped migration plan) | diff --git a/docs/audit/gauntlet-20260722-g12-reconcile.txt b/docs/audit/gauntlet-20260722-g12-reconcile.txt new file mode 100644 index 0000000..c289ec6 --- /dev/null +++ b/docs/audit/gauntlet-20260722-g12-reconcile.txt @@ -0,0 +1,78 @@ + carve-host-interfaces ALL PASS + checks ALL PASS (62 checks) + claude-guard ALL PASS + clientdocs-scripts ALL PASS + clientdocs-skill ALL PASS + cloud-assert ALL PASS + cloudinit-vm cloudinit-vm: 6 passed, 0 failed + cloud-snapshot ALL PASS + creds-audit creds-audit: 7/7 PASS + d063-apply ALL PASS (7/7) + d115-office-carve d115-office-carve: 20/20 PASS + d120-compose-bands d120-compose-bands: 16/16 PASS + d124-transit-seed d124-transit-seed: 21/21 PASS + dc-dc-ceph-disk-budget ALL PASS + dc-dc-dr-drill ALL PASS + dc-dc-mtu-geneve-budget ALL PASS + dc-dc-prefixes-import ALL PASS (91 checks) + dc-dc-radosgw-multisite ALL PASS + dc-dc-rbd-mirror ALL PASS + dc-dc-whole-host-budget dc-dc-whole-host-budget: 13/13 PASS + dc-edge-wan-import dc-edge-wan-import: 58/58 PASS + dc-rack-mgmt-import dc-rack-mgmt-import: 117/117 PASS + dc-rack-net dc-rack-net: 14 passed, 0 failed + dc-selector ALL PASS (40 checks) + juju-spaces-check ALL PASS (5/5) + keystone-policy-drift ALL PASS + ledger-scan ALL PASS (46 checks) + lib-validate ALL PASS (35 checks) + maas-node-power maas-node-power: 24/24 PASS + netem-link netem-link: 12 passed, 0 failed + netem-sudoers netem-sudoers: 9 passed, 0 failed + node-vm node-vm: 15 passed, 0 failed + opentofu-validate ALL PASS + opnsense-api ALL PASS + opnsense-bootstrap-apikey ALL PASS + opnsense-plugins opnsense-plugins: 20 passed, 0 failed + opnsense-prep-image ALL PASS + opnsense-set-interface-v4 opnsense-set-interface-v4: 53/53 PASS + opnsense-set-interface-v6 opnsense-set-interface-v6: 40/40 PASS + phase-00-maas-standup ALL PASS + phase-00-teardown-d061 ALL PASS + phase-02 ALL PASS + phase-03-adminrc ALL PASS + phase-03 ALL PASS + phase-04-create ALL PASS + phase-04-internal-cert-san ALL PASS + phase-04 ALL PASS + phase-05-amphora ALL PASS + phase-05 ALL PASS + phase-06-bootstrap ALL PASS + phase-06-capi-stack ALL PASS + phase-06-k8s-bootstrap ALL PASS + phase-06-kubeconfig-gate ALL PASS + phase-06-mgmt-vm ALL PASS + phase-06-net-setup ALL PASS + phase-07-conductor-graft ALL PASS + preflight ALL PASS + prereqs prereqs: 32/32 PASS + provider-bundle-check ALL PASS + record-audit record-audit: 35/35 PASS + reenroll-hosts ALL PASS + repo-lint ALL PASS + roles-aggregates-import roles-aggregates-import: 24/24 PASS + sandbox-fidelity-check sandbox-fidelity-check: 14/14 PASS + sandbox-seed sandbox-seed: 22/22 PASS + site-baseleg site-baseleg: 24 passed, 0 failed + site-forward site-forward: 38/38 PASS + site-headend-install site-headend-install: 54/54 PASS + site-ssh-config site-ssh-config: 30/30 PASS + tenant-acceptance ALL PASS + tenant-assert ALL PASS + tenant-offboard ALL PASS (22/22) + tenant-onboard ALL PASS + trust_filter ALL PASS + validate ALL PASS (29 checks) + vault-kv-health ALL PASS (10/10) + +GAUNTLET: ALL GREEN (76 harnesses) diff --git a/docs/changelog-20260721-g12-dc1-authoring.md b/docs/changelog-20260721-g12-dc1-authoring.md index b6237be..652a4ab 100644 --- a/docs/changelog-20260721-g12-dc1-authoring.md +++ b/docs/changelog-20260721-g12-dc1-authoring.md @@ -103,3 +103,47 @@ opentofu/main.tf opentofu/variables.tf scripts/lib-net.sh tests/dc-selector/run-tests.sh && rm -rf opentofu/vr1-dc1-substrate/`; delete the gauntlet capture file. + +## Item 4 -- harness reconcile: item 3 delivered (successor session, 2026-07-22) + +What: the six red harnesses reconciled to the ratified dc1 selector arm, plus +the two sighted surfaces dispositioned. Per script: + +- `phase-00-maas-standup.sh`: NEW PLANES/lib-net parity guard (exit 2, + precondition) -- the PLANES table is a DC0-hardcoded D-052/D-053 literal, and + with dc1 now selectable the script would otherwise have planned DC0 topology + under `DC=vr1-dc1` (the drop-state gauntlet showed exactly that: exit 0 + + WOULD: leak). Any selected DC whose lib-net plane set diverges from the table + is refused. Harness asserts the refusal (exit 2, no WOULD:/DO: leak). +- `dc-dc-rbd-mirror.sh` + `dc-dc-radosgw-multisite.sh`: the sighted cross-DC + value surfacing FIXED IN BOTH (radosgw carried the same defect as the sighted + rbd-mirror): the $DC gate ran only in a subshell, so the replication-plane + reminder always read the file's flat DC0 defaults -- under `--dc vr1-dc1` it + printed 10.12.36.0/22 (dc0's). On gate success the script now selects the DC + in its own shell; new harness cases pin the dc1 reminder to 10.12.84.0/22. +- `dc-dc-dr-drill` / `radosgw-multisite` / `rbd-mirror` harnesses: the dc1 + gate-refusal tests (want rc 3) became gate-OK dry-run + juju-guarded --apply + (rc 2) tests. The enforce path (exit 3) is retained in the scripts as + defense-in-depth but is no longer CLI-reachable (arg validation admits only + the two ratified tokens) -- noted in the test comments. +- `carve-host-interfaces` / `reenroll-hosts` harnesses: vr1-dc1 now fails at + the HOSTS layer like vr1-dc0 ("no enrolled hosts yet"); tests assert that + message + the no-mutation guard. Stale dc1/dc2-era header comments in both + scripts (and phase-00) updated. +- Sighting re-check: `tests/dc-dc-prefixes-import/` carries NO grep of the + retired selector message at HEAD -- the drop-state sighting does not + reproduce; harness passes unmodified (91 checks). No edit made there. + +Verification: the six harnesses individually green; full gauntlet **ALL GREEN +(76 harnesses)** -- `docs/audit/gauntlet-20260722-g12-reconcile.txt`; +repo-lint 0 fail. This item + the item-3 tree state land in one commit +(this delivery); G12's next gated step is unchanged: operator-gated apex +`--commit`, then tfvars + gated build. + +Revert: `git checkout main -- scripts/phase-00-maas-standup.sh +scripts/dc-dc-rbd-mirror.sh scripts/dc-dc-radosgw-multisite.sh +scripts/carve-host-interfaces.sh scripts/reenroll-hosts.sh +tests/phase-00-maas-standup/ tests/dc-dc-rbd-mirror/ +tests/dc-dc-radosgw-multisite/ tests/dc-dc-dr-drill/ +tests/carve-host-interfaces/ tests/reenroll-hosts/`; delete the reconcile +gauntlet capture (item-3 revert covers the authored substrate). diff --git a/opentofu/main.tf b/opentofu/main.tf index ec828c0..badcdbe 100644 --- a/opentofu/main.tf +++ b/opentofu/main.tf @@ -38,22 +38,13 @@ target_path = var.vr1_dc0_pool_path } -# ---- vr1-dc1 (VR1's SECOND DC): PLANES deferred by SEQUENCING, not a missing -# literal. Its supernet IS assigned -- D-115 moved it to 10.12.64.0/19 (ADOPTED, -# imported to office1-netbox), superseding the old "wait for NetBox" premise. It -# stays out of scope until vr1-dc0 completes (operator ruling 2026-07-10, Option B: -# vr1-dc0-first). When vr1-dc1 is started, add a `vr1_dc1_planes` variable (same -# shape as `vr1_dc0_planes`, CIDRs derived per D-115) and uncomment this block. -# Do NOT uncomment it now. The storage pool + mesh legs are NOT address-dependent, -# so they ARE wired now. -# -# module "vr1_dc1_planes" { -# source = "./modules/dc-planes" -# dc_name = "vr1-dc1" -# domain_suffix = var.domain_suffix -# mtu = var.underlay_mtu -# planes = var.vr1_dc1_planes -# } +# ---- vr1-dc1 (VR1's SECOND DC): STARTED 2026-07-21 (gate G12; the D-124 +# AMENDMENT ratified its addressing, apex confirm-free captured). Under D-123 +# Model B its six planes are NOT created here: like dc0's, they are created by +# the INNER root (opentofu/vr1-dc1-substrate/) on vvr1-dc1's own libvirt. The +# committed values-of-record live in var.vr1_dc1_planes (variables.tf); the old +# pre-Model-B `module "vr1_dc1_planes"` skeleton that stood here is retired -- +# do not re-add an outer planes module for any DC. ---- module "vr1_dc1_storage" { source = "./modules/dc-storage-pool" @@ -376,6 +367,18 @@ # mtu defaults to 1500 (the ISP-uplink domain; never jumbo). } +# vr1-dc1's uplink -- same D-125 bridge-in shape as dc0's above, same reasoning. +# The /24 is D-115's vr1-dc1 sim-ISP edge WAN, registered in the apex scoped +# vr1-dc1 (verified 2026-07-21, docs/audit/dc1-apex-confirm-20260721.txt) -- a +# RULED literal, not a HELD value. The D-125 egress isolation gate re-runs at +# dc1 standup (per-DC deploy-time gate), though bridge-in itself is PROVEN on dc0. +module "vr1_dc1_uplink" { + source = "./modules/site-wan" + network_name = "vr1-dc1-uplink" + cidr = "172.30.3.0/24" # D-115 vr1-dc1 sim-ISP edge WAN (D-124 amendment 2026-07-21) + # mtu defaults to 1500 (the ISP-uplink domain; never jumbo). +} + # ===================================================================== # D-123 MODEL B / D-124 AMENDMENT: vvr1-dc0 -- the vr1-dc0 SITE CONTAINMENT VM. Under # Model B it is BOTH the MAAS rack controller (enrolled to the Office1 region) AND the @@ -498,3 +501,100 @@ forward-delay: 0 EOT } + +# ===================================================================== +# G12 (2026-07-21): vvr1-dc1 -- the vr1-dc1 SITE CONTAINMENT VM. The dc0 block +# above is the PATTERN OF RECORD; this mirrors it exactly with the D-124-AMENDMENT +# addressing (transit 172.31.0.4/30 via tfvars after the apex --commit; uplink +# 172.30.3.0/24). Same Model B role: MAAS rack controller (enrolled to the Office1 +# region) AND nested libvirt host for dc1's 9-node fleet, created by the inner +# root opentofu/vr1-dc1-substrate/. Sizing identical (D-121 Option C is per-DC; +# the 2-DC whole-host budget was validated when dc0 was sized). D-126 per-env key: +# the DEDICATED dc1 key, not office1's or dc0's. NOT YET APPLIED -- the apply, the +# bootstrap gate (site-headend-install.sh node-host mode) and the inner-root apply +# are gated dc1-standup steps. Standup definition-of-done additionally carries +# (D-124 amendment): scripts/dc-rack-net.sh site row with legs MEASURED at standup +# + install; the D-131 forwarder (10.12.68.3); MAC pinning from the FIRST apply +# (the inner root pre-assigns; see vr1-dc1-substrate/main.tf); per-machine virsh +# power via maas-node-power.sh (pods refuted, D-103/D-123 amendments). +# ===================================================================== +module "vvr1_dc1" { + source = "./modules/cloudinit-vm" + vm_name = "vvr1-dc1" + autostart = false # D-127: DC containment VM -- MANUAL (gated bring-up), never on host boot + vcpu = var.vvr1_dc1_vcpu # same Option-C fleet derivation as dc0 + memory_mib = var.vvr1_dc1_memory_mib # derived 416 GiB (384 node + 32 overhead) + disk_size_bytes = var.vvr1_dc1_disk_bytes # derived ~3000 GiB (holds the inner libvirt pool) + pool_name = module.vr1_dc1_storage.pool_name + base_volume_path = module.ubuntu_noble_base.path + expose_nested_virt = true # D-123 Model B: LOAD-BEARING -- inner node VMs need svm passthrough + + # TWO outer legs, dc0 pattern: + # NIC1 (enp1s0 -> mgmt) = office1<->dc1 transit (D-124 Scheme A; its OWN mesh leg, + # NOT dc0's). SEC-010/--transit-if key on THIS interface. + # NIC2 (enp2s0 -> uplink) = D-125 bridge-in: IP-less port of br-vr1-dc1-wan on the + # vcloud ISP NAT (vr1-dc1-uplink). + network_names = [ + module.mesh_vr1_dc1_office1.network_name, # NIC1 enp1s0 -> mgmt (transit; SEC-010 keys here) + module.vr1_dc1_uplink.network_name, # NIC2 enp2s0 -> uplink (IP-less port of br-vr1-dc1-wan) + ] + + user_data = <<-EOT + #cloud-config + hostname: vvr1-dc1 + fqdn: vvr1-dc1.${var.domain_suffix} + manage_etc_hosts: true + users: + - name: jessea123 + groups: [adm, sudo] + shell: /bin/bash + sudo: "ALL=(ALL) NOPASSWD:ALL" + ssh_authorized_keys: + # D-126 per-env-key: vvr1-dc1 authorizes the DEDICATED dc1 key ONLY + # (~/vr1-dc1-creds/, D-124 amendment 2026-07-21). Inner root's qemu+ssh matches. + - ${trimspace(file(var.vr1_dc1_ssh_pubkey_path))} + package_update: true + packages: + - qemu-guest-agent + runcmd: + - [systemctl, enable, --now, qemu-guest-agent] + EOT + + meta_data = <<-EOT + instance-id: vvr1-dc1-d124a + local-hostname: vvr1-dc1 + EOT + + # STATIC on the transit leg; uplink is an IP-less bridge port (dc0 pattern, + # comments there are the authority). INTERFACE NAMING: enp1s0/enp2s0 are the + # names dc0 MEASURED on this exact q35 2-NIC cloudinit-vm shape (kernel rename + # log, 2026-07-19) -- expected to hold here, CONFIRM on vvr1-dc1's first boot + # before relying on it. set-name deliberately absent (dc0's measured netplan + # pitfall). The region route rides the transit toward Office1 (10.10.0.0/22). + # br_netfilter CONSTRAINT (D-125): SEC-010's FORWARD-drop stays scoped to + # enp1s0 -- never a global drop (would kill bridged WAN frames). + network_config = <<-EOT + version: 2 + ethernets: + transit: + match: + name: "enp1s0" + addresses: ["${var.vr1_dc1_rack_transit_ip}/${var.vr1_dc1_rack_transit_prefix}"] + routes: + - to: "10.10.0.0/22" + via: "${var.vr1_dc1_rack_transit_peer_ip}" + uplink: + match: + name: "enp2s0" + dhcp4: false + dhcp6: false + bridges: + br-vr1-dc1-wan: + interfaces: [uplink] + dhcp4: false + dhcp6: false + parameters: + stp: false + forward-delay: 0 + EOT +} diff --git a/opentofu/variables.tf b/opentofu/variables.tf index de04094..6c92524 100644 --- a/opentofu/variables.tf +++ b/opentofu/variables.tf @@ -58,13 +58,33 @@ } } -# vr1_dc1_planes has NO variable/default here yet, deliberately: VR1 DC1's v4 -# supernet is a D-101 OPEN SUB-ITEM ("exact prefixes assigned in NetBox, -# non-colliding with..."), not yet wired here. Do not add a guessed default -- -# once NetBox assigns it, add a vr1_dc1_planes variable shaped exactly like -# vr1_dc0_planes above and a matching module "vr1_dc1_planes" block in main.tf -# (see the commented skeleton there). -# +variable "vr1_dc1_planes" { + description = <<-EOT + vr1-dc1's six planes (D-124 AMENDMENT 2026-07-21, addressing RATIFIED; + supernet 10.12.64.0/19 per D-115). NOT a copy of dc0's layout: the six + /22s are CONTIGUOUS keeping dc0's role ORDER (dc0's 4/8/12/16/32/36 + offsets cannot fit inside a /19 -- deliberate, documented divergence; + 10.12.88+92.0/22 spare). These values are apex-VERIFIED, not invented: + all six exist in office1-netbox scoped dcim.site:vr1-dc1 with matching + roles (docs/audit/dc1-apex-confirm-20260721.txt). Under D-123 Model B + the planes are CREATED by the INNER root (opentofu/vr1-dc1-substrate/); + this default is the committed values-of-record the inner tfvars copies + (same relationship as vr1_dc0_planes above / lib-net.sh's vr1-dc1 arm -- + change one, change all in the SAME commit). + EOT + type = map(object({ + cidr = string + })) + default = { + provider-public = { cidr = "10.12.64.0/22" } + metal-admin = { cidr = "10.12.68.0/22" } + metal-internal = { cidr = "10.12.72.0/22" } + data-tenant = { cidr = "10.12.76.0/22" } + storage = { cidr = "10.12.80.0/22" } + replication = { cidr = "10.12.84.0/22" } + } +} + # D-119 NAMING: vr1-dc0 is VR1's FIRST DC, vr1-dc1 its SECOND -- matching the # NetBox apex slugs exactly. Never write a bare "dcN" for a VR1 DC here: "dc0" # means VR0's LIVE testcloud in scripts/lib-net.sh, and that collision is what @@ -132,6 +152,66 @@ default = 3221225472000 } +variable "vr1_dc1_ssh_pubkey_path" { + description = <<-EOT + Path to the PUBLIC half of the DEDICATED vr1-dc1 service SSH key (D-126 per-env-key + ruling, option a; dc1 row ratified in the D-124 amendment 2026-07-21: + ~/vr1-dc1-creds/vr1-dc1_svc_ed25519). Injected into vvr1-dc1's cloud-init; the inner + root's qemu+ssh authenticates with the matching private half (jumphost-local, never + read). Read with file() at plan time -- the .pub must exist. Per-env blast-radius + isolation from Office1 AND from dc0. + EOT + type = string +} + +# ---- vvr1-dc1 sizing: SAME derivation as vvr1-dc0 (D-121 Option C is per-DC -- +# identical 3 control + 2 compute + 4 storage fleet), via +# scripts/dc-dc-whole-host-budget.py: 104 node vCPU / 384 GiB / ~2850 GiB thin + +# containment overhead. The whole-host 2-DC total (870/1024 GiB, 85%) was budgeted +# FIT when dc0's sizing was derived -- dc1 IS the second half of that budget. ---- +variable "vvr1_dc1_vcpu" { + description = "vvr1-dc1 vCPU (Model B: hosts one DC's 9 node VMs). Derived: 104 node + 4 overhead." + type = number + default = 108 +} + +variable "vvr1_dc1_memory_mib" { + description = "vvr1-dc1 RAM in MiB (Model B). Derived: 384 GiB node fleet + 32 GiB overhead = 416 GiB." + type = number + default = 425984 +} + +variable "vvr1_dc1_disk_bytes" { + description = "vvr1-dc1 boot disk in bytes (Model B: holds the inner libvirt pool). Derived: ~2850 GiB node disks (thin) + ~150 GiB OS = ~3000 GiB." + type = number + default = 3221225472000 +} + +# ---- D-124 AMENDMENT (2026-07-21): vr1-dc1 MAAS rack controller (vvr1-dc1) +# addressing. NO defaults -- ratified values enter via tfvars ONLY AFTER the +# operator-gated apex --commit registers them in office1-netbox (dc0 precedent: +# rack_transit_ip=172.31.0.6, prefix=30, peer=172.31.0.5, metal_admin=10.12.68.2). +# Do NOT invent or pre-fill them here (hard rule 2). ---- +variable "vr1_dc1_rack_metal_admin_ip" { + description = "vvr1-dc1 rack static IP in metal-admin (10.12.68.0/22), in the .2-.49 static band, not the .1 gateway. NetBox-assigned (D-124 amendment)." + type = string +} + +variable "vr1_dc1_rack_transit_ip" { + description = "vvr1-dc1 rack IP on the office1<->dc1 mesh transit (D-124 Scheme A, next /30 in 172.31.0.0/24). NetBox-assigned." + type = string +} + +variable "vr1_dc1_rack_transit_prefix" { + description = "office1<->dc1 transit link prefix length (30 or 31, D-124 Scheme A)." + type = number +} + +variable "vr1_dc1_rack_transit_peer_ip" { + description = "Office1 region's IP on the office1<->dc1 transit link -- the rack's next-hop toward the MAAS region (10.10.0.0/22). NetBox-assigned (D-124 amendment)." + type = string +} + # ---- D-124: vr1-dc0 MAAS rack controller (vvr1-dc0) addressing. NO defaults -- # these come from office1-netbox (the IPAM apex) via the D-124 importer, then # into a tfvars entry. Do NOT invent them here (hard rule 2). The rack straddles diff --git a/opentofu/vr1-dc1-substrate/main.tf b/opentofu/vr1-dc1-substrate/main.tf new file mode 100644 index 0000000..f3a8c68 --- /dev/null +++ b/opentofu/vr1-dc1-substrate/main.tf @@ -0,0 +1,151 @@ +# ===================================================================== +# vr1-dc1 INNER SUBSTRATE (G12, authored 2026-07-21) -- D-123 MODEL B, created +# inside vvr1-dc1. MIRROR of opentofu/vr1-dc0-substrate/main.tf (the pattern of +# record: its comments carry the measured traps -- provider dial, known_hosts, +# upload-from-executing-host -- and apply here unchanged). Run this root AFTER: +# (1) the outer opentofu/ root has booted + sized vvr1-dc1, and (2) the bootstrap +# gate (site-headend-install.sh node-host mode) has installed libvirtd + the +# inner pool + kvm nested=1 + the OPNsense base image on vvr1-dc1. +# ===================================================================== + +provider "libvirt" { + # qemu+ssh to vvr1-dc1 over the office1<->dc1 transit. Key auth + known_hosts + # verified out of band (NO no_verify). The IP is MEASURED after the outer apply + # (hard rule 2). keyfile + sshauth are REQUIRED URI params (dc0, measured + # 2026-07-20: the provider's Go ssh transport ignores ~/.ssh/config and default + # identities). known_hosts: ALL key types must be present (dc0's measured + # "knownhosts: key mismatch" trap; seed via ssh-keyscan, cross-checked). + uri = "qemu+ssh://${var.vvr1_dc1_ssh_user}@${var.vvr1_dc1_transit_ip}/system?keyfile=${var.vvr1_dc1_ssh_keyfile}&sshauth=privkey" +} + +# Inner storage pool (backs the node + edge disks) -- created inside vvr1-dc1 at +# the path the bootstrap step provisioned. +module "inner_storage" { + source = "../modules/dc-storage-pool" + dc_name = "vr1-dc1-inner" + target_path = var.inner_pool_path +} + +# The six vr1-dc1 planes (D-124 amendment 2026-07-21 layout -- contiguous /22s in +# 10.12.64.0/19, dc0's role ORDER kept) -- isolated-L2 networks INSIDE vvr1-dc1. +# metal-admin is where the rack serves PXE/DHCP to the inner nodes. +module "vr1_dc1_planes" { + source = "../modules/dc-planes" + dc_name = "vr1-dc1" + domain_suffix = var.domain_suffix + mtu = var.underlay_mtu + planes = var.vr1_dc1_planes +} + +# D-125 bridge-in (dc0 pattern, PROVEN there end to end): the DC WAN segment is a +# BRIDGE onto vvr1-dc1's br-vr1-dc1-wan (the IP-less uplink NIC on the vcloud ISP +# NAT `vr1-dc1-uplink` = 172.30.3.0/24, declared in the outer root's vvr1-dc1 +# netplan). ONE NAT, at vcloud. host_bridge MUST match the netplan bridge name in +# opentofu/main.tf. The D-125 egress isolation gate re-runs at dc1 standup. +module "vr1_dc1_wan" { + source = "../modules/wan-bridge" + network_name = "vr1-dc1-wan" + host_bridge = "br-vr1-dc1-wan" + # mtu defaults to 1500 (ISP-uplink domain; NOT the jumbo planes/mesh). +} + +# D-122: the DC edge -- 2-NIC (WAN + LAN), Office1 pattern (2048/2/nano). LAN = +# provider-public (external gateway, D-100; ruled gw 10.12.64.1 per the D-124 +# amendment); WAN = the dedicated uplink above (static .2 on 172.30.3.0/24, set +# via the D-113 API path at the standup addressing step, not here). +module "vr1_dc1_opnsense" { + source = "../modules/opnsense-edge" + vm_name = "vr1-dc1-opnsense" + autostart = true # D-127: the DC edge comes up with its containment VM (routing for the site) + memory_mib = 2048 + vcpu = 2 + pool_name = module.inner_storage.pool_name + base_volume_path = var.opnsense_base_path + lan_network_name = module.vr1_dc1_planes.network_names["provider-public"] + wan_network_name = module.vr1_dc1_wan.network_name + # D-129: NEW DC edges boot WITH the qga channel so they never need the retrofit + # (operator-ruled 2026-07-20). + expose_qga_channel = true +} + +# D-121 Option C layout (per-DC): 3 control + 2 compute + 4 storage = 9 nodes. +# Six NICs each (one per plane); metal-admin FIRST = PXE/boot plane (D-052 default +# binding). +locals { + vr1_dc1_node_nics = [ + module.vr1_dc1_planes.network_names["metal-admin"], # PXE / boot fabric first + module.vr1_dc1_planes.network_names["provider-public"], + module.vr1_dc1_planes.network_names["metal-internal"], + module.vr1_dc1_planes.network_names["data-tenant"], + module.vr1_dc1_planes.network_names["storage"], + module.vr1_dc1_planes.network_names["replication"], + ] + + # macs: PRE-ASSIGNED at authoring (2026-07-21) so the fleet is pinned from the + # FIRST apply -- the D-124-amendment standup invariant, and the direct lesson of + # dc0's 2026-07-20 incident (unpinned MACs regenerated by an in-place apply + # stranded the whole fleet in MAAS; dc0 had to measure-and-pin after the fact). + # Scheme, deterministic and collision-free by construction: + # 52:54:01:d1:NN:PP + # ^^^^^^^^ locally-administered unicast (bit 0x02 of octet 1), one octet away + # from libvirt's auto-generation space 52:54:00:* -- an auto MAC can + # never collide with these. + # ^^ d1 tags vr1-DC1 (dc0's pins are measured 52:54:00 values, not this + # scheme; future DCs take their own tag octet). + # NN = node ordinal 01..09 in the map order below; + # PP = NIC ordinal 01..06 in vr1_dc1_node_nics order (metal-admin + # first). These are IDENTITY values, not tunables -- change one and + # MAAS no longer knows the node. + vr1_dc1_nodes = { + "vr1-dc1-control-01" = { vcpu = 16, mem = 65536, disk_gib = 150, macs = [ + "52:54:01:d1:01:01", "52:54:01:d1:01:02", "52:54:01:d1:01:03", + "52:54:01:d1:01:04", "52:54:01:d1:01:05", "52:54:01:d1:01:06", + ] } + "vr1-dc1-control-02" = { vcpu = 16, mem = 65536, disk_gib = 150, macs = [ + "52:54:01:d1:02:01", "52:54:01:d1:02:02", "52:54:01:d1:02:03", + "52:54:01:d1:02:04", "52:54:01:d1:02:05", "52:54:01:d1:02:06", + ] } + "vr1-dc1-control-03" = { vcpu = 16, mem = 65536, disk_gib = 150, macs = [ + "52:54:01:d1:03:01", "52:54:01:d1:03:02", "52:54:01:d1:03:03", + "52:54:01:d1:03:04", "52:54:01:d1:03:05", "52:54:01:d1:03:06", + ] } + "vr1-dc1-compute-01" = { vcpu = 12, mem = 49152, disk_gib = 100, macs = [ + "52:54:01:d1:04:01", "52:54:01:d1:04:02", "52:54:01:d1:04:03", + "52:54:01:d1:04:04", "52:54:01:d1:04:05", "52:54:01:d1:04:06", + ] } + "vr1-dc1-compute-02" = { vcpu = 12, mem = 49152, disk_gib = 100, macs = [ + "52:54:01:d1:05:01", "52:54:01:d1:05:02", "52:54:01:d1:05:03", + "52:54:01:d1:05:04", "52:54:01:d1:05:05", "52:54:01:d1:05:06", + ] } + "vr1-dc1-storage-01" = { vcpu = 8, mem = 24576, disk_gib = 550, macs = [ + "52:54:01:d1:06:01", "52:54:01:d1:06:02", "52:54:01:d1:06:03", + "52:54:01:d1:06:04", "52:54:01:d1:06:05", "52:54:01:d1:06:06", + ] } + "vr1-dc1-storage-02" = { vcpu = 8, mem = 24576, disk_gib = 550, macs = [ + "52:54:01:d1:07:01", "52:54:01:d1:07:02", "52:54:01:d1:07:03", + "52:54:01:d1:07:04", "52:54:01:d1:07:05", "52:54:01:d1:07:06", + ] } + "vr1-dc1-storage-03" = { vcpu = 8, mem = 24576, disk_gib = 550, macs = [ + "52:54:01:d1:08:01", "52:54:01:d1:08:02", "52:54:01:d1:08:03", + "52:54:01:d1:08:04", "52:54:01:d1:08:05", "52:54:01:d1:08:06", + ] } + "vr1-dc1-storage-04" = { vcpu = 8, mem = 24576, disk_gib = 550, macs = [ # Option C: 4th OSD host + "52:54:01:d1:09:01", "52:54:01:d1:09:02", "52:54:01:d1:09:03", + "52:54:01:d1:09:04", "52:54:01:d1:09:05", "52:54:01:d1:09:06", + ] } + } +} + +module "vr1_dc1_node" { + for_each = local.vr1_dc1_nodes + + source = "../modules/node-vm" + vm_name = each.key + autostart = false # D-127: node VMs are MAAS-power-controlled -- MANUAL, never on host boot + vcpu = each.value.vcpu + memory_mib = each.value.mem + disk_size_bytes = each.value.disk_gib * 1024 * 1024 * 1024 + pool_name = module.inner_storage.pool_name + network_names = local.vr1_dc1_node_nics + interface_macs = each.value.macs # pinned FROM THE FIRST APPLY -- see the scheme comment +} diff --git a/opentofu/vr1-dc1-substrate/variables.tf b/opentofu/vr1-dc1-substrate/variables.tf new file mode 100644 index 0000000..4f695d2 --- /dev/null +++ b/opentofu/vr1-dc1-substrate/variables.tf @@ -0,0 +1,48 @@ +# Inner-root inputs, mirroring vr1-dc0-substrate/variables.tf (the pattern of +# record -- its per-variable comments carry the measured traps and apply here +# unchanged). The two connection values (transit IP + ssh user) are MEASURED +# after vvr1-dc1 boots (hard rule 2 -- not invented); the rest mirror the outer +# root. + +variable "vvr1_dc1_transit_ip" { + description = "vvr1-dc1's static transit IP (D-124 Scheme A, NetBox-assigned via the amendment's apex commit) -- where this root's qemu+ssh inner provider connects. MEASURED after the outer apply boots vvr1-dc1." + type = string +} + +variable "vvr1_dc1_ssh_user" { + description = "SSH user on vvr1-dc1 for the qemu+ssh inner provider (the cloud-init user, e.g. jessea123). Key auth + known_hosts verified out of band (NO no_verify; ALL key types in known_hosts -- dc0's measured trap)." + type = string + default = "jessea123" +} + +variable "vvr1_dc1_ssh_keyfile" { + description = "PATH (on the EXECUTING host, i.e. voffice1 per D-128) to the DEDICATED dc1 private key for the qemu+ssh provider (D-126 option a; ~/vr1-dc1-creds/vr1-dc1_svc_ed25519). A path, never key material. REQUIRED: the provider's ssh dial reads NEITHER ~/.ssh/config nor default identities (dc0, measured 2026-07-20)." + type = string +} + +variable "inner_pool_path" { + description = "Directory path INSIDE vvr1-dc1 for the inner libvirt storage pool that backs the node/edge disks (created by the bootstrap step)." + type = string + default = "/var/lib/libvirt/vr1-dc1-inner" +} + +variable "opnsense_base_path" { + description = "Path ON THE EXECUTING HOST (voffice1, D-128 Plane 2) to the prepped OPNsense nano qcow2 -- the remote qemu+ssh provider UPLOADS volume content from ITS OWN filesystem (dc0, measured 2026-07-20). 26.7 nano: the DC-edge base PROVEN end to end on dc0 (boot, D-112(c) console bootstrap, D-113(a2) API, egress)." + type = string + default = "/var/lib/libvirt/vr1-dc1-inner/opnsense-26.7-nano.qcow2" +} + +variable "domain_suffix" { + description = "DNS domain suffix for the planes (mirrors the outer root)." + type = string +} + +variable "underlay_mtu" { + description = "Jumbo MTU for the internal fabric planes/mesh (9000). The wan bridge stays 1500." + type = number +} + +variable "vr1_dc1_planes" { + description = "The six vr1-dc1 planes (CIDR map), copied from the outer root's var of the same name (the committed values-of-record; D-124 amendment 2026-07-21, apex-verified). dc-planes creates them as isolated-L2 networks inside vvr1-dc1." + type = any +} diff --git a/opentofu/vr1-dc1-substrate/versions.tf b/opentofu/vr1-dc1-substrate/versions.tf new file mode 100644 index 0000000..ab929eb --- /dev/null +++ b/opentofu/vr1-dc1-substrate/versions.tf @@ -0,0 +1,18 @@ +# INNER root for vr1-dc1 (G12, 2026-07-21) -- D-123 MODEL B, mirroring +# opentofu/vr1-dc0-substrate/ (the pattern of record; its versions.tf comment is +# the authority for WHY this is a separate root + state). The vr1-dc1 substrate +# -- 6 dc-planes, the wan bridge, the OPNsense edge, an inner storage pool, and +# the 9 node VMs -- created INSIDE vvr1-dc1 via the qemu+ssh inner provider, +# NOT on vcloud. Apply order: outer (opentofu/) boots vvr1-dc1 -> bootstrap gate +# (site-headend-install.sh node-host mode) -> THIS root. Modules reused VERBATIM +# from ../modules/. Provider pin matches the outer root's versions.tf (0.9.8). +terraform { + required_version = ">= 1.6.0" + + required_providers { + libvirt = { + source = "dmacvicar/libvirt" + version = "0.9.8" + } + } +} diff --git a/scripts/carve-host-interfaces.sh b/scripts/carve-host-interfaces.sh index b3ef7b0..576141f 100644 --- a/scripts/carve-host-interfaces.sh +++ b/scripts/carve-host-interfaces.sh @@ -29,9 +29,10 @@ # # DC selector (opt-in, DOCFIX-166): set DC=vr0-dc0|vr1-dc0|vr1-dc1 to call # lib_net_select_dc/lib_hosts_select_dc explicitly. Unset (default) == DC0's -# real plane scheme + enrolled hosts, unchanged. dc1 no-ops at the network -# layer but FAILS at the host layer (no per-DC HOST_OCTET/HOST_BOOT_MAC yet); -# dc2 fails at both -- see scripts/lib-net.sh / scripts/lib-hosts.sh. +# real plane scheme + enrolled hosts, unchanged. vr1-dc0 no-ops at the +# network layer; vr1-dc1 selects its ratified planes (D-124 amendment); +# BOTH then FAIL at the host layer (no per-DC HOST_OCTET/HOST_BOOT_MAC +# yet) -- see scripts/lib-net.sh / scripts/lib-hosts.sh. # # Exit: 0 ok | 1 fatal | 2 warning diff --git a/scripts/dc-dc-radosgw-multisite.sh b/scripts/dc-dc-radosgw-multisite.sh index a66d18b..6894b92 100644 --- a/scripts/dc-dc-radosgw-multisite.sh +++ b/scripts/dc-dc-radosgw-multisite.sh @@ -144,6 +144,10 @@ DC_GATE_MSG="$(lib_net_select_dc "$DC" 2>&1 1>/dev/null)"; DC_GATE_RC=$? if [ "$DC_GATE_RC" -eq 0 ]; then DC_GATE_LINE="\$DC gate: OK ($DC)" + # The gate check ran in a subshell; select again HERE so the plane vars the + # reminder reads below belong to $DC, not the file's flat DC0 defaults + # (cross-DC value surfacing). rc 0 is already proven, safe under set -e. + lib_net_select_dc "$DC" else DC_GATE_LINE="\$DC gate: FAILED for '$DC' -- $DC_GATE_MSG" fi diff --git a/scripts/dc-dc-rbd-mirror.sh b/scripts/dc-dc-rbd-mirror.sh index 13433dd..8b7c65a 100644 --- a/scripts/dc-dc-rbd-mirror.sh +++ b/scripts/dc-dc-rbd-mirror.sh @@ -127,6 +127,10 @@ DC_GATE_MSG="$(lib_net_select_dc "$DC" 2>&1 1>/dev/null)"; DC_GATE_RC=$? if [ "$DC_GATE_RC" -eq 0 ]; then DC_GATE_LINE="\$DC gate: OK ($DC)" + # The gate check ran in a subshell; select again HERE so the plane vars the + # reminder reads below belong to $DC, not the file's flat DC0 defaults + # (cross-DC value surfacing). rc 0 is already proven, safe under set -e. + lib_net_select_dc "$DC" else DC_GATE_LINE="\$DC gate: FAILED for '$DC' -- $DC_GATE_MSG" fi diff --git a/scripts/lib-net.sh b/scripts/lib-net.sh index ec326a6..6195b74 100644 --- a/scripts/lib-net.sh +++ b/scripts/lib-net.sh @@ -94,6 +94,16 @@ # exists to delete. lib_net_select_dc() { local dc="${1:?usage: lib_net_select_dc }" + # ONE selection per shell (the convention above: call ONCE, right after + # sourcing). Mechanically enforced since the vr1-dc1 arm DIVERGES (2026-07-21): + # the vr0-dc0/vr1-dc0 arms are no-ops over the file's flat defaults, so + # CHANGING selection after a diverging arm ran would leave stale cross-DC + # values in scope -- refuse instead of silently mixing DCs. Re-selecting the + # SAME DC stays a no-op; re-source the lib to genuinely switch. + if [ -n "${_LIBNET_DC_SELECTED:-}" ] && [ "$_LIBNET_DC_SELECTED" != "$dc" ]; then + echo "FAIL: DC already selected as '$_LIBNET_DC_SELECTED' in this shell -- one selection per shell; re-source lib-net.sh to switch DC" >&2 + return 1 + fi case "$dc" in vr0-dc0) : # VR0's DC0 -- the LIVE testcloud. The literals sourced above ARE its @@ -108,8 +118,36 @@ # the thing that changes (VR1 DC0 gains its own v6 per the family matrix). ;; vr1-dc1) - echo "FAIL: vr1-dc1 (VR1's SECOND DC) has no assigned network literals yet (D-101 NetBox-literals open item, tooling gap register #3) -- do not select it until NetBox assigns real CIDRs for it" >&2 - return 1 + # VR1's SECOND DC -- literals RATIFIED by the D-124 AMENDMENT (2026-07-21) + # and apex-VERIFIED against office1-netbox the same day (all six planes + # exist scoped dcim.site:vr1-dc1 -- docs/audit/dc1-apex-confirm-20260721.txt), + # which is this arm's own landing rule (NetBox assigns, then the literals + # enter the lib). DIVERGES from dc0 BY DESIGN: contiguous /22s in the + # 10.12.64.0/19 supernet (D-115), dc0's role ORDER kept -- dc0's + # 4/8/12/16/32/36 offsets cannot fit inside a /19. Values-of-record twin: + # opentofu/variables.tf `vr1_dc1_planes` (change one, change BOTH in the + # same commit). + PLANE_CIDRS=( "10.12.64.0/22" "10.12.68.0/22" "10.12.72.0/22" "10.12.76.0/22" "10.12.80.0/22" "10.12.84.0/22" ) + PLANE_NAME=( + ["10.12.64.0/22"]="provider-public" + ["10.12.68.0/22"]="metal-admin" + ["10.12.72.0/22"]="metal-internal" + ["10.12.76.0/22"]="data-tenant" + ["10.12.80.0/22"]="storage" + ["10.12.84.0/22"]="replication" + ) + # .1 site gateways on the two routed planes (D-120 convention; provider-public + # gw 10.12.64.1 is the ruled edge LAN gateway, D-124 amendment). + PLANE_GW=( ["10.12.64.0/22"]="10.12.64.1" ["10.12.68.0/22"]="10.12.68.1" ) + DATA_PLANE_CIDRS=( "10.12.72.0/22" "10.12.76.0/22" "10.12.80.0/22" "10.12.84.0/22" ) + METAL_INTERNAL_CIDR="10.12.72.0/22" + # OpenStack-layer values (VIP bands, FIP pool, keystone VIP) and the VR0 + # metal fabric facts (VLAN id, bridge iface) are NOT yet ruled/measured for + # vr1-dc1 (they arrive with its Stage-5 analog). UNSET so any use fails + # loud under `set -u` instead of silently borrowing another DC's addresses. + unset METAL_INTERNAL_VID METAL_INTERNAL_IFACE + unset VIP_PREFIX_PROVIDER VIP_PREFIX_ADMIN VIP_PREFIX_INTERNAL VIP_COUNT_EXPECT + unset FIP_POOL_START FIP_POOL_END KEYSTONE_VIP_DEFAULT ;; dc0|dc1|dc2) echo "FAIL: bare '$dc' is RETIRED (D-119). It was AMBIGUOUS ACROSS REGIONS: 'dc0' meant VR0's live DC0 here, but VR1's FIRST DC in the NetBox importer. Use the region-qualified selector: vr0-dc0 | vr1-dc0 | vr1-dc1" >&2 @@ -120,6 +158,7 @@ return 1 ;; esac + _LIBNET_DC_SELECTED="$dc" # only successful arms reach here (failed arms return 1) } # --- tiny read-only helpers --- diff --git a/scripts/phase-00-maas-standup.sh b/scripts/phase-00-maas-standup.sh index 8044462..a72d4d6 100644 --- a/scripts/phase-00-maas-standup.sh +++ b/scripts/phase-00-maas-standup.sh @@ -52,16 +52,17 @@ # runbook's Step 1: an explicit $DC env var, never an inferred default. # Unset/empty $DC changes NOTHING -- this script runs exactly as it always # has, implicitly against the D-052/D-053 plane scheme (backward compatible -# by construction). Set DC=vr0-dc0|vr1-dc0|vr1-dc1 to select explicitly (D-119); vr1-dc1 FAILS LOUD -# today (no NetBox-assigned literals yet -- gap #3) via set -e, no second -# validation layer added. This script does not source lib-hosts.sh (it never -# touches host identity), so only the network selector is called here. +# by construction). Set DC=vr0-dc0|vr1-dc0|vr1-dc1 to select explicitly +# (D-119) via set -e, no second validation layer added. This script does not +# source lib-hosts.sh (it never touches host identity), so only the network +# selector is called here. # NOTE: the PLANES table below is still a hardcoded D-052/D-053 literal, not -# derived from lib-net.sh's flat vars -- selecting dc1 is a real no-op -# (D-101: dc1 inherits dc0's layout unchanged, so the hardcoded literals are -# still correct), but this selector call alone does NOT make this script -# dc2-aware; once dc2 gets real CIDRs (gap #3), the PLANES table itself will -# still need updating for a genuinely different dc2 scheme. +# derived from lib-net.sh's vars. vr0-dc0/vr1-dc0 match it (D-101 +# inheritance). vr1-dc1's ratified scheme DIVERGES (D-124 amendment +# 2026-07-21: contiguous /22s in 10.12.64.0/19), so a parity guard below the +# table refuses any selected DC whose lib-net plane set does not match the +# table -- this script must never plan one DC's topology under another DC's +# name. DC="${DC:-}" if [ -n "$DC" ]; then lib_net_select_dc "$DC" @@ -123,6 +124,21 @@ TBL )" +# --- PLANES/lib-net parity guard (only when a $DC was explicitly selected): +# every table row's name|cidr pair must match the selected DC's PLANE_NAME +# map, else this DC0-hardcoded table would silently plan DC0 topology under +# another DC's name (exactly the cross-DC mixing lib-net's selector exists +# to prevent). Exit 2 = precondition, per the header contract. +if [ -n "$DC" ]; then + while IFS='|' read -r pname pcidr _rest; do + [ -z "$pname" ] && continue + if [ "${PLANE_NAME[$pcidr]:-}" != "$pname" ]; then + echo "FAIL: PLANES table is DC0-hardcoded ($pname $pcidr) but DC='$DC' assigns different planes (lib-net.sh) -- this script's topology target is not $DC-aware; refusing to plan another DC's scheme" >&2 + exit 2 + fi + done <<<"$PLANES" +fi + dt() { [ "$1" = "-" ] && echo "" || echo "$1"; } # decode "-" sentinel to empty hdr "MAAS stand-up mode=$MODE (D-052/D-053 target scheme)" diff --git a/scripts/reenroll-hosts.sh b/scripts/reenroll-hosts.sh index 8a835dc..214601f 100644 --- a/scripts/reenroll-hosts.sh +++ b/scripts/reenroll-hosts.sh @@ -13,8 +13,8 @@ # # DC selector (opt-in, DOCFIX-166): set DC=vr0-dc0|vr1-dc0|vr1-dc1 to call # lib_net_select_dc/lib_hosts_select_dc explicitly. Unset (default) == DC0's -# real, enrolled hosts, unchanged. dc1/dc2 currently FAIL LOUD (no per-DC host -# inventory exists yet) -- see scripts/lib-hosts.sh. +# real, enrolled hosts, unchanged. vr1-dc0/vr1-dc1 currently FAIL LOUD at the +# host layer (no per-DC host inventory exists yet) -- see scripts/lib-hosts.sh. # # Discover-assert-pin: never creates a host that already exists. Idempotent -- # a re-run after a partial run only creates the still-missing hosts. @@ -44,8 +44,9 @@ # always has, implicitly against DC0/VR0's real, enrolled hosts (backward # compatible by construction). Set DC=vr0-dc0|vr1-dc0|vr1-dc1 to select explicitly (D-119); the # selectors' own fail-loud behavior becomes this script's own exit code via -# set -e (lib_hosts_select_dc fails for BOTH dc1 and dc2 today -- no per-DC -# host inventory exists yet for either). No second validation layer is added. +# set -e (lib_hosts_select_dc fails for BOTH vr1-dc0 and vr1-dc1 today -- no +# per-DC host inventory exists yet for either). No second validation layer is +# added. DC="${DC:-}" if [ -n "$DC" ]; then lib_net_select_dc "$DC" diff --git a/tests/carve-host-interfaces/run-tests.sh b/tests/carve-host-interfaces/run-tests.sh index 9574a53..142174b 100644 --- a/tests/carve-host-interfaces/run-tests.sh +++ b/tests/carve-host-interfaces/run-tests.sh @@ -87,8 +87,9 @@ has 'no enrolled hosts yet' absent 'DO:|WOULD:' # must exit before any MAAS interaction -run_dc 1 "DC=vr1-dc1: net selector fails loud first (NetBox gap)" vr1-dc1 -has 'no assigned network literals yet' +run_dc 1 "DC=vr1-dc1: hosts selector fails loud (net arm ratified, D-124)" vr1-dc1 +has 'no enrolled hosts yet' +absent 'DO:|WOULD:' # must exit before any MAAS interaction run_dc 1 "DC=bogus: unknown token fails loud" bogus has "unknown DC 'bogus'" diff --git a/tests/dc-dc-dr-drill/run-tests.sh b/tests/dc-dc-dr-drill/run-tests.sh index 1bd4664..77c83ee 100644 --- a/tests/dc-dc-dr-drill/run-tests.sh +++ b/tests/dc-dc-dr-drill/run-tests.sh @@ -76,11 +76,18 @@ grep -q '11.4 SKIPPED' <<<"$FB_SKIP" && ! grep -q '11.4a' <<<"$FB_SKIP" \ && { echo " PASS T20 --skip-11-4 stops the plan after 11.2/11.3"; PASS=$((PASS+1)); } || { echo " FAIL T20"; echo "$FB_SKIP" | sed 's/^/ /'; FAIL=$((FAIL+1)); } -# --- $DC gate: informational in dry-run (vr1-dc1 involved), blocking before --apply --- -grep -q 'gate: FAILED' <<<"$FB_OUT" && grep -q 'gate: OK (vr1-dc0)' <<<"$FB_OUT" \ - && { echo " PASS T21 failback dry-run reports vr1-dc0 OK / vr1-dc1 FAILED, still prints full plan"; PASS=$((PASS+1)); } || { echo " FAIL T21"; echo "$FB_OUT" | sed 's/^/ /'; FAIL=$((FAIL+1)); } +# --- $DC gate: both DCs now pass (vr1-dc1 literals RATIFIED, D-124 amendment +# 2026-07-21). The gate's refusal path (exit 3) is retained in the script as +# defense-in-depth for future unassigned DC tokens, but is no longer reachable +# through the CLI (arg validation admits only the two ratified tokens). +grep -q 'gate: OK (vr1-dc1)' <<<"$FB_OUT" && grep -q 'gate: OK (vr1-dc0)' <<<"$FB_OUT" \ + && { echo " PASS T21 failback dry-run reports BOTH DC gates OK (dc1 ratified), still prints full plan"; PASS=$((PASS+1)); } || { echo " FAIL T21"; echo "$FB_OUT" | sed 's/^/ /'; FAIL=$((FAIL+1)); } -run 3 '\$DC gate refused' "T22 failback --apply is BLOCKED by the dc2 gate (rc 3)" failback --pool glance --recovering-dc vr1-dc0 --recovering-unit rec/0 --primary-dc vr1-dc1 --primary-unit pri/0 --apply --no-prompt +if command -v juju >/dev/null 2>&1; then + echo " SKIP T22 juju-missing case (juju IS present in this environment -- can't exercise the missing-tool guard here)" +else + run 2 'juju required on PATH' "T22 failback --apply passes both DC gates, FAILS on missing juju (rc 2)" failback --pool glance --recovering-dc vr1-dc0 --recovering-unit rec/0 --primary-dc vr1-dc1 --primary-unit pri/0 --apply --no-prompt +fi # --- --apply never runs without --apply: no juju invocation attempted in dry-run --- if command -v juju >/dev/null 2>&1; then diff --git a/tests/dc-dc-radosgw-multisite/run-tests.sh b/tests/dc-dc-radosgw-multisite/run-tests.sh index e570c14..4a796f9 100644 --- a/tests/dc-dc-radosgw-multisite/run-tests.sh +++ b/tests/dc-dc-radosgw-multisite/run-tests.sh @@ -57,12 +57,21 @@ RESTART_OUT="$(bash "$SCRIPT" master-init --dc vr1-dc0 --unit ceph-radosgw/0 --realm R --zonegroup ZG --zone Z1 --endpoint http://x:80 --restart-action restart 2>&1)" grep -qE 'juju run ceph-radosgw/0 restart -m openstack' <<<"$RESTART_OUT" && { echo " PASS T15 --restart-action adds a juju run step"; PASS=$((PASS+1)); } || { echo " FAIL T15"; echo "$RESTART_OUT" | sed 's/^/ /'; FAIL=$((FAIL+1)); } -# --- $DC gate: informational in dry-run, blocking before --apply --- -DC2_DRY="$(bash "$SCRIPT" master-init --dc vr1-dc1 --unit ceph-radosgw/0 --realm R --zonegroup ZG --zone Z1 --endpoint http://x:80 2>&1)"; DC2_DRY_RC=$? -[[ "$DC2_DRY_RC" == 0 ]] && grep -q 'gate: FAILED' <<<"$DC2_DRY" && grep -q 'OK (dry-run)' <<<"$DC2_DRY" \ - && { echo " PASS T16 dc2 dry-run still prints the plan (gate is informational here)"; PASS=$((PASS+1)); } || { echo " FAIL T16"; echo "$DC2_DRY" | sed 's/^/ /'; FAIL=$((FAIL+1)); } +# --- $DC gate: vr1-dc1 now passes (literals RATIFIED, D-124 amendment +# 2026-07-21); the reminder must surface DC1's OWN replication plane, never +# dc0's (cross-DC value surfacing). The gate's refusal path (exit 3) is +# retained as defense-in-depth but no longer CLI-reachable. +DC1_DRY="$(bash "$SCRIPT" master-init --dc vr1-dc1 --unit ceph-radosgw/0 --realm R --zonegroup ZG --zone Z1 --endpoint http://x:80 2>&1)"; DC1_DRY_RC=$? +[[ "$DC1_DRY_RC" == 0 ]] && grep -q 'gate: OK (vr1-dc1)' <<<"$DC1_DRY" && grep -q 'OK (dry-run)' <<<"$DC1_DRY" \ + && { echo " PASS T16 dc1 dry-run prints the plan (gate OK, D-124 ratified)"; PASS=$((PASS+1)); } || { echo " FAIL T16"; echo "$DC1_DRY" | sed 's/^/ /'; FAIL=$((FAIL+1)); } +grep -q 'REPLICATION plane (10.12.84.0/22' <<<"$DC1_DRY" \ + && { echo " PASS T16b dc1 reminder shows dc1's OWN replication CIDR (not dc0's)"; PASS=$((PASS+1)); } || { echo " FAIL T16b (cross-DC value surfaced)"; echo "$DC1_DRY" | sed 's/^/ /'; FAIL=$((FAIL+1)); } -run 3 '\$DC gate refused' "T17 dc2 --apply is BLOCKED by the gate (rc 3)" master-init --dc vr1-dc1 --unit ceph-radosgw/0 --realm R --zonegroup ZG --zone Z1 --endpoint http://x:80 --apply +if command -v juju >/dev/null 2>&1; then + echo " SKIP T17 juju-missing case (juju IS present in this environment -- can't exercise the missing-tool guard here)" +else + run 2 'juju required on PATH' "T17 dc1 --apply passes the gate, FAILS on missing juju (rc 2)" master-init --dc vr1-dc1 --unit ceph-radosgw/0 --realm R --zonegroup ZG --zone Z1 --endpoint http://x:80 --apply +fi # --- secret redaction --- JR_OUT="$(bash "$SCRIPT" join-readonly --dc vr1-dc0 --unit ceph-radosgw/0 --zonegroup ZG --zone Z2 --endpoint http://x:80 --access-key AK123 --secret TOPSECRETVALUE 2>&1)" diff --git a/tests/dc-dc-rbd-mirror/run-tests.sh b/tests/dc-dc-rbd-mirror/run-tests.sh index 302c20f..c91991b 100644 --- a/tests/dc-dc-rbd-mirror/run-tests.sh +++ b/tests/dc-dc-rbd-mirror/run-tests.sh @@ -51,12 +51,21 @@ grep -q 'peer bootstrap import --site-name vr1-dc1 --direction rx-tx glance /tmp/tok2' <<<"$BS_RXTX" \ && { echo " PASS T16 bootstrap-secondary rx-tx (two-way) plan is correct"; PASS=$((PASS+1)); } || { echo " FAIL T16"; echo "$BS_RXTX" | sed 's/^/ /'; FAIL=$((FAIL+1)); } -# --- $DC gate: informational in dry-run, blocking before --apply --- -DC2_DRY="$(bash "$SCRIPT" bootstrap-primary --dc vr1-dc1 --unit ceph-mon/0 --pool glance --site-name vr1-dc1 2>&1)"; DC2_DRY_RC=$? -[[ "$DC2_DRY_RC" == 0 ]] && grep -q 'gate: FAILED' <<<"$DC2_DRY" && grep -q 'OK (dry-run)' <<<"$DC2_DRY" \ - && { echo " PASS T17 dc2 dry-run still prints the plan (gate is informational here)"; PASS=$((PASS+1)); } || { echo " FAIL T17"; echo "$DC2_DRY" | sed 's/^/ /'; FAIL=$((FAIL+1)); } +# --- $DC gate: vr1-dc1 now passes (literals RATIFIED, D-124 amendment +# 2026-07-21); the reminder must surface DC1's OWN replication plane, never +# dc0's (cross-DC value surfacing). The gate's refusal path (exit 3) is +# retained as defense-in-depth but no longer CLI-reachable. +DC1_DRY="$(bash "$SCRIPT" bootstrap-primary --dc vr1-dc1 --unit ceph-mon/0 --pool glance --site-name vr1-dc1 2>&1)"; DC1_DRY_RC=$? +[[ "$DC1_DRY_RC" == 0 ]] && grep -q 'gate: OK (vr1-dc1)' <<<"$DC1_DRY" && grep -q 'OK (dry-run)' <<<"$DC1_DRY" \ + && { echo " PASS T17 dc1 dry-run prints the plan (gate OK, D-124 ratified)"; PASS=$((PASS+1)); } || { echo " FAIL T17"; echo "$DC1_DRY" | sed 's/^/ /'; FAIL=$((FAIL+1)); } +grep -q 'REPLICATION plane (10.12.84.0/22' <<<"$DC1_DRY" \ + && { echo " PASS T17b dc1 reminder shows dc1's OWN replication CIDR (not dc0's)"; PASS=$((PASS+1)); } || { echo " FAIL T17b (cross-DC value surfaced)"; echo "$DC1_DRY" | sed 's/^/ /'; FAIL=$((FAIL+1)); } -run 3 '\$DC gate refused' "T18 dc2 --apply is BLOCKED by the gate (rc 3)" bootstrap-primary --dc vr1-dc1 --unit ceph-mon/0 --pool glance --site-name vr1-dc1 --apply +if command -v juju >/dev/null 2>&1; then + echo " SKIP T18 juju-missing case (juju IS present in this environment -- can't exercise the missing-tool guard here)" +else + run 2 'juju required on PATH' "T18 dc1 --apply passes the gate, FAILS on missing juju (rc 2)" bootstrap-primary --dc vr1-dc1 --unit ceph-mon/0 --pool glance --site-name vr1-dc1 --apply +fi if command -v juju >/dev/null 2>&1; then echo " SKIP T19 juju-missing case (juju IS present in this environment -- can't exercise the missing-tool guard here)" diff --git a/tests/dc-selector/run-tests.sh b/tests/dc-selector/run-tests.sh index b745bd2..29e66c0 100644 --- a/tests/dc-selector/run-tests.sh +++ b/tests/dc-selector/run-tests.sh @@ -6,11 +6,14 @@ # Asserts: # - sourcing either file with no further action is unaffected (backward # compatible by construction -- the flat vars populate exactly as before). -# - lib_net_select_dc: dc0/dc1 no-op (D-101: DC1 inherits DC0's v4 layout -# unchanged), dc2 fails loud, unknown token fails loud. -# - lib_hosts_select_dc: dc0 no-op, dc1 AND dc2 both fail loud (no real +# - lib_net_select_dc: vr0-dc0/vr1-dc0 no-op (D-101 inheritance); vr1-dc1 +# OVERRIDES to the D-124-amendment literals (2026-07-21: contiguous /22s in +# 10.12.64.0/19) and UNSETS the not-yet-ruled OpenStack-layer values; +# unknown/retired tokens fail loud; one selection per shell (the guard +# added with the diverging dc1 arm). +# - lib_hosts_select_dc: vr0-dc0 no-op, BOTH VR1 DCs fail loud (no real # per-DC host enrollment exists yet for either -- this is the documented -# asymmetry vs. lib-net.sh's dc0|dc1 no-op). +# asymmetry vs. lib-net.sh). # - Neither function ever silently invents/reuses a value across DCs. set -uo pipefail SD="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -37,15 +40,43 @@ chk "HOST_OCTET unaffected" "${HOST_OCTET[openstack0]}" "40" # --- lib_net_select_dc (D-119 region-qualified): vr0-dc0/vr1-dc0 no-op, -# vr1-dc1 fails loud, RETIRED bare dcN fails loud, unknown fails loud --- +# vr1-dc1 OVERRIDES (D-124 amendment 2026-07-21), RETIRED bare dcN fails +# loud, unknown fails loud --- ( lib_net_select_dc vr0-dc0 ); chk "net vr0-dc0 no-op rc" "$?" 0 ( lib_net_select_dc vr1-dc0 ); chk "net vr1-dc0 no-op rc" "$?" 0 -( lib_net_select_dc vr1-dc1 2>/dev/null ); chk "net vr1-dc1 fails-loud rc" "$?" 1 -NET_ERR="$(lib_net_select_dc vr1-dc1 2>&1 1>/dev/null || true)" -grep -q "NetBox" <<<"$NET_ERR" && ok "net vr1-dc1 error cites NetBox gap" || no "net vr1-dc1 error cites NetBox gap" +( lib_net_select_dc vr1-dc1 ); chk "net vr1-dc1 selects rc" "$?" 0 ( lib_net_select_dc bogus 2>/dev/null ); chk "net unknown-token fails-loud rc" "$?" 1 ( lib_net_select_dc 2>/dev/null ); chk "net missing-arg fails-loud rc" "$?" 1 +# vr1-dc1 arm: the D-124-amendment literals land (apex-verified 2026-07-21). +DC1_P0="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${PLANE_CIDRS[0]}")" +chk "net vr1-dc1 provider-public is 10.12.64.0/22" "$DC1_P0" "10.12.64.0/22" +DC1_P5="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${PLANE_CIDRS[5]}")" +chk "net vr1-dc1 replication is 10.12.84.0/22" "$DC1_P5" "10.12.84.0/22" +DC1_MA="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${PLANE_NAME[10.12.68.0/22]}")" +chk "net vr1-dc1 10.12.68.0/22 is metal-admin" "$DC1_MA" "metal-admin" +DC1_GW="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${PLANE_GW[10.12.64.0/22]}")" +chk "net vr1-dc1 provider-public gw is 10.12.64.1 (ruled edge LAN gw)" "$DC1_GW" "10.12.64.1" +DC1_MI="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "$METAL_INTERNAL_CIDR")" +chk "net vr1-dc1 metal-internal CIDR is 10.12.72.0/22" "$DC1_MI" "10.12.72.0/22" +# not-yet-ruled OpenStack-layer values are UNSET (fail loud under set -u, never +# silently another DC's addresses). +DC1_KV="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${KEYSTONE_VIP_DEFAULT+still-set}")" +chk "net vr1-dc1 unsets KEYSTONE_VIP_DEFAULT" "$DC1_KV" "" +DC1_VP="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${VIP_PREFIX_PROVIDER+still-set}")" +chk "net vr1-dc1 unsets VIP_PREFIX_PROVIDER" "$DC1_VP" "" +DC1_FP="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${FIP_POOL_START+still-set}")" +chk "net vr1-dc1 unsets FIP_POOL_START" "$DC1_FP" "" + +# ONE SELECTION PER SHELL (guard added with the diverging dc1 arm): changing +# selection after dc1 is REFUSED; re-selecting the SAME DC stays a no-op. +( lib_net_select_dc vr1-dc1 >/dev/null 2>&1; lib_net_select_dc vr0-dc0 2>/dev/null ); \ + chk "net dc1->vr0-dc0 re-select REFUSED (stale cross-DC values)" "$?" 1 +SWITCH_ERR="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; lib_net_select_dc vr0-dc0 2>&1 1>/dev/null || true)" +grep -q "already selected" <<<"$SWITCH_ERR" && ok "net re-select error says already selected" || no "net re-select error says already selected" +( lib_net_select_dc vr1-dc0 >/dev/null 2>&1; lib_net_select_dc vr1-dc0 ); \ + chk "net same-DC re-select stays a no-op rc" "$?" 0 + # D-119 REGRESSION GUARD: the bare dcN tokens are RETIRED and must be REJECTED. # Accepting them "for compatibility" would preserve the exact cross-region # ambiguity D-119 deletes -- 'dc0' meant VR0's LIVE cloud in lib-net.sh but VR1's diff --git a/tests/phase-00-maas-standup/run-tests.sh b/tests/phase-00-maas-standup/run-tests.sh index 91554a8..45412c2 100644 --- a/tests/phase-00-maas-standup/run-tests.sh +++ b/tests/phase-00-maas-standup/run-tests.sh @@ -94,8 +94,8 @@ has 'no drift' has 'OK \(dryrun\) -- topology consistent with D-052/D-053' -run_dc 1 "DC=vr1-dc1: net selector fails loud (NetBox gap)" vr1-dc1 -has 'no assigned network literals yet' +run_dc 2 "DC=vr1-dc1: PLANES parity guard refuses (diverging D-124 scheme)" vr1-dc1 +has 'PLANES table is DC0-hardcoded' absent 'DO:|WOULD:' # must exit before any MAAS interaction run_dc 1 "DC=bogus: unknown token fails loud" bogus diff --git a/tests/reenroll-hosts/run-tests.sh b/tests/reenroll-hosts/run-tests.sh index f37f9ee..d04bcfe 100644 --- a/tests/reenroll-hosts/run-tests.sh +++ b/tests/reenroll-hosts/run-tests.sh @@ -58,8 +58,8 @@ has 'no enrolled hosts yet' absent 'Current host status' # must exit before report() ever runs -run_dc 1 "DC=vr1-dc1: net selector fails loud first (NetBox gap)" vr1-dc1 -has 'no assigned network literals yet' +run_dc 1 "DC=vr1-dc1: hosts selector fails loud (net arm ratified, D-124)" vr1-dc1 +has 'no enrolled hosts yet' absent 'Current host status' # D-119 REGRESSION GUARD: the bare dcN tokens are RETIRED. 'dc0' used to mean