diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 060b556..ad3d328 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -774,6 +774,26 @@ where dc1's nine nodes still live; the rack DNS forwarder's upstream; and the two credential removals (the `vr1-dc0-region` profile on voffice1, SEC-026; Office1's copy of the dc0 power key). + **`scripts/dc-node-carve.sh` NOW EXISTS (2026-07-30) -- the "STILL OWED" item above is + discharged as far as the TOOL goes; the live apply is tracked separately below.** + Harness `tests/dc-node-carve` **40/40**, manifest re-recorded 92 -> 93, repo-lint 0 fail. + `check` is the named GA-R6 gate; `apply` is dry-run until `--commit`; `--host` restricts + to one machine so the fleet can be done canary-first then in batches. **The four-call + cycle was PROVEN ON ONE LIVE INTERFACE BEFORE THE TOOL WAS WRITTEN** -- nothing had + exercised `interface update vlan=` against an interface whose only link is `link_up` on + an auto fabric, which is the state all sixty NICs are in. On canary `6q4syf` `enp3s0`: + `vlan=5004` read back as `vr1-dc0-metal-internal`, then `static:10.12.12.153` read back. + MEASURED IN PASSING and it simplified the tool: **`link-subnet` REPLACES a `link_up` + link**, so plane NICs need no explicit unlink -- only the `br-ex` member does. + **A MUTATION PASS FOUND ONE ASSERTION THAT COULD NOT FAIL, plus a whole first pass that + proved less than it looked.** Neutering three assertions by deleting their MESSAGE turned + the suite red only via the tests that grep for that message -- which proves the assertion + EXISTS, not that its predicate works. Re-run neutering only the PREDICATE, three killed + properly and **the `br-ex` static-address compare SURVIVED**: the Pattern-B fixture was + being caught by the type/parent assertions, so the address compare was never exercised. + A `brexip` fixture (correctly-parented OVS bridge carrying the WRONG static) was added + and the kill re-proved. **This is the "prove each new assertion can FAIL" rule catching a + real hole for the second time in four days.** - Project: Omega Cloud, VR1 DC-DC rehearsal -- a two-DC + Office1-headend virtual rehearsal on KVM (vcloud host), rehearsing the future bare-metal diff --git a/docs/changelog-20260730-dc0-node-carve.md b/docs/changelog-20260730-dc0-node-carve.md new file mode 100644 index 0000000..2226c15 --- /dev/null +++ b/docs/changelog-20260730-dc0-node-carve.md @@ -0,0 +1,100 @@ +# Changelog 2026-07-30 -- dc0 node interface carve (`dc-node-carve.sh`) + +Session changelog (GA-R2/D1: ONE per session). Branch `dc-dc-stage5-preconditions`. +Status claims live ONLY in `docs/CURRENT-STATE.md`. + +Context: the dc0 machine migration to the per-DC MAAS region is COMPLETE -- all ten +machines `Ready`, four named gates green. The last thing between here and the Juju +deploy is the per-node v4 interface carve: 60 NIC re-homes, 9 OVS `br-ex` bridges and +54 statics, deliberately NOT written blind last session because it could not be +exercised until the nodes were re-enrolled. + +--- + +## Item 1 -- `scripts/dc-node-carve.sh` + `tests/dc-node-carve/` (NEW) + +**What.** The v4 node interface carve for a VR1 DC, run against that DC's own MAAS +region. `check` is the named GA-R6 gate; `apply` is dry-run by default and needs +`--commit` to mutate. `--host` restricts to one machine so the work can go canary-first +then in batches. + +**Why it did not already exist.** The equivalent work at Stage 4 was done ad-hoc through +the MAAS API and logged only to `~/as-executed/2026-07-23-stage4-carve.log`, which is +NOT in the repo. When D-132 q1 moved dc0 to its own region and every machine had to be +re-enrolled, there was nothing to re-run. A read-only survey last session identified +this as the largest of three no-tool gaps. + +**The four-call cycle was PROVEN ON ONE INTERFACE BEFORE THE TOOL WAS WRITTEN.** Nothing +in the repo had exercised `interface update vlan=` against an interface whose only link +is `link_up` on an auto-created fabric, which is the state all sixty NICs are in. Run on +the migration canary (`6q4syf`/civil-bug = `vr1-dc0-storage-04`, the least central node): + + interface update 6q4syf 10 vlan=5004 -> read back vlan=5004 / vr1-dc0-metal-internal + interface link-subnet ... mode=STATIC ... -> read back static:10.12.12.153 + +Both bit. MEASURED IN PASSING, and it simplified the tool: `link-subnet` REPLACES a +`link_up` link, so plane NICs need no explicit unlink. Only the `br-ex` member does, +because there the L3 has to move off the member onto the bridge. + +**Properties that are structural, not stylistic:** + +- **No default profile, and `maas-profile-assert.sh` runs before anything is read or + written.** The repo-wide default `MAAS_PROFILE=admin` resolves to the OFFICE1 region, + where dc1's nine nodes still live. A 60-NIC re-home against the wrong region is either + an idempotent no-op that prints PASS or damage to the wrong DC. +- **Site refusal is the mirror image of `carve-host-interfaces.sh`.** That script refuses + `vr1-*` because it builds the VR0 Pattern-A VID-103 stack; this one refuses everything + that is not `vr1-dc0|vr1-dc1` because it builds the D-133 flat per-NIC carve. +- **Every id is resolved live and keyed by a stable identity** -- machines by PINNED BOOT + MAC (system_id AND hostname are both re-minted on re-enlistment), subnets and VLANs by + CIDR, interfaces by name. The pre-migration capture is a rebuild reference for + ADDRESSES AND INTERFACE NAMES ONLY: its `vlan_id=5005` is dc0 metal-admin in Office1 + and `vr1-dc0-data-tenant` in the new region. +- **Every mutation is READ BACK and compared, and a mismatch is fatal.** MAAS has been + measured returning a full JSON object while changing nothing, surfacing only later as + `{"subnet": ["None found with id=N."]}`. +- **NIC order comes from `lib-hosts.sh NIC_PLANE_ORDER`, never `lib-net.sh PLANE_CIDRS`.** + The plane->CIDR map is INVERTED from lib-net's own `PLANE_NAME` rather than retyped, so + a plane re-address cannot silently disagree. +- **Role asymmetry is DERIVED from the host token** (`JUJU_HOST_SUFFIX`), never typed. Role + nodes take six planes plus an OVS `br-ex` parented on `enp2s0`; the Juju controller takes + metal-admin + provider-public RAW, no `br-ex`, and `enp3s0..enp6s0` deliberately left on + their auto VLANs. The gate asserts that deliberate ABSENCE, so a later session cannot + "helpfully" carve them and have everything stay green. +- **`build_ovs_brex`'s guard structure is preserved verbatim** from + `carve-host-interfaces.sh:211-236`. The VLAN move sits INSIDE the "br-ex does not exist + yet" branch on purpose: a MAAS bridge inherits its parent's VLAN at create, and once + br-ex exists the parent is enslaved and a VLAN update is invalid. Restructuring it for + symmetry with `carve_raw` would break re-runnability. +- **A link on the right subnet with the wrong address FAILS; it is never skipped.** + +**What this gate does NOT prove, stated in its own output.** The node's actual routing +table is a first-boot observation (G17). The carve proves the MAAS MODEL that produces a +default route -- a static on provider-public plus `gateway_ip` on that subnet -- and the +absence of that leg is exactly the under-carve that cost three bootstrap attempts on +2026-07-30. A green carve is not a proven default route, and the script says so rather +than letting a reader infer it. + +**Harness: 40/40, and every assertion is mutation-proven.** Eight fixtures each supply a +failing direction: `raw` (the measured uncarved state), `wrongip`, `patternb` (provider +static on the raw NIC with no bridge -- the D-057 shape that darkened the floating-IP +plane), `memberl3`, `brexip`, `jujucarved`, `jujubrex`, `nogw`. + +**A MUTATION PASS FOUND ONE ASSERTION THAT COULD NOT FAIL, and a first pass of mutations +that proved less than it appeared to.** Neutering three assertions by deleting their +MESSAGE turned the suite red only via the tests that grep for that message -- which +proves the assertion EXISTS, not that its predicate works. Re-run neutering only the +PREDICATE and keeping the message, three killed properly and **the `br-ex` static-address +compare SURVIVED**: `patternb` was being caught by the type/parent assertions, so the +address compare was never exercised. Added the `brexip` fixture (a correctly-parented OVS +bridge carrying the WRONG static) and re-proved the kill. + +**Revert.** `git rm scripts/dc-node-carve.sh tests/dc-node-carve/run-tests.sh` and +`bash scripts/run-tests-all.sh --record-manifest`. Nothing else depends on it yet. + +## Item 2 -- `tests/HARNESS-MANIFEST` re-recorded (92 -> 93) + +**What.** Re-recorded to pin the new harness. Required: the gauntlet's R15(2) drift gate +FAILS on a harness that is present but unpinned. + +**Revert.** `git checkout ^ -- tests/HARNESS-MANIFEST`. diff --git a/scripts/dc-node-carve.sh b/scripts/dc-node-carve.sh new file mode 100755 index 0000000..402fe02 --- /dev/null +++ b/scripts/dc-node-carve.sh @@ -0,0 +1,434 @@ +#!/usr/bin/env bash +# scripts/dc-node-carve.sh --profile

--expect-rack +# [--host ] [--commit] +# +# The v4 NODE interface carve for a VR1 DC, against that DC's OWN MAAS region +# (D-132 q1). Re-homes each node's six NICs onto their plane VLANs, builds the +# OVS `br-ex` on the provider uplink of every ROLE node, and lays the D-134 +# statics. IPv6 is NOT this script's job -- run `dc-node-v6-carve.py` afterwards +# (it needs br-ex to already exist, because the provider v6 static lands there). +# +# WHY THIS EXISTS (2026-07-30). The equivalent work at Stage 4 was done ad-hoc +# through the MAAS API and logged only to `~/as-executed/2026-07-23-stage4-carve.log`, +# which is NOT in the repo -- so when D-132 q1 moved dc0 to its own region and +# every machine had to be re-enrolled, there was nothing to re-run. This is that +# missing tool. dc1 rebuilds from it rather than from a transcript. +# +# TARGET TREE -- and it is NOT uniform across roles (lib-hosts.sh +# NODE_CARVE_ROLE_PLANES / NODE_CARVE_JUJU_PLANES; measured from the +# pre-migration capture, docs/audit/dc0-maas-carve-premigration-20260730.txt): +# +# ROLE NODE (the 9 `openstack-` machines), octet N: +# enp1s0 raw + STATIC .N (PXE leg; auto -> STATIC) +# enp2s0 ------> br-ex (OVS) + STATIC .N [Pattern A] +# enp2s0 itself holds NO L3 link. +# enp3s0 raw + STATIC .N +# enp4s0 raw + STATIC .N +# enp5s0 raw + STATIC .N +# enp6s0 raw + STATIC .N +# +# JUJU CONTROLLER (`-juju-01`), octet 5: +# enp1s0 raw + STATIC .5 +# enp2s0 raw + STATIC .5 -- RAW, no br-ex. +# enp3s0..enp6s0 DELIBERATELY LEFT on their auto-created VLANs. +# The controller runs no workloads, so it needs no tenant/storage/replication +# legs. It DOES need a default route, and the provider-public leg is what +# supplies it (the provider subnet carries gateway_ip). The absence of that +# leg is the under-carve that cost three bootstrap attempts on 2026-07-30. +# +# NIC ORDER IS `lib-hosts.sh NIC_PLANE_ORDER`, NEVER `lib-net.sh PLANE_CIDRS`. +# PLANE_CIDRS starts with provider-public; the NIC order starts with metal-admin. +# Walking PLANE_CIDRS positionally swaps enp1s0/enp2s0 and strands PXE on a plane +# with no DHCP. `tests/dc-selector` asserts the two orders differ at position 0 +# so that guard cannot go vacuous. +# +# EVERY id is resolved LIVE and keyed by a STABLE identity: machines by PINNED +# BOOT MAC (system_id AND hostname are both re-minted on re-enlistment -- only +# the MAC survives), subnets and VLANs by CIDR, interfaces by NAME. MAAS row ids +# do NOT cross regions: VLAN row 5005 is dc0's metal-admin in the Office1 region +# and `vr1-dc0-data-tenant` in the per-DC one. The pre-migration capture is a +# REBUILD REFERENCE for addresses and interface names ONLY -- never for an id. +# +# A 200-SHAPED MAAS RESPONSE IS NOT EVIDENCE THE STATE CHANGED. `interface update +# ... vlan=` has been measured returning a full JSON object while changing +# nothing, surfacing only later as `{"subnet": ["None found with id=N."]}` +# because MAAS scopes the subnet lookup to the interface's VLAN. Every mutation +# here is READ BACK and compared; a mismatch is fatal. +# +# Exit: 0 ok | 1 assertions failed | 2 refused (bad args / unproven region / no CLI) + +set -uo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO="$(cd "$HERE/.." && pwd)" + +ACTION=""; SITE=""; PROFILE=""; EXPECT_RACK=""; ONLY_HOST=""; COMMIT=0 +while [ $# -gt 0 ]; do + case "$1" in + check|apply) [ -z "$ACTION" ] && ACTION="$1" || SITE="$1"; shift ;; + --profile) PROFILE="${2:-}"; shift 2 ;; + --expect-rack) EXPECT_RACK="${2:-}"; shift 2 ;; + --host) ONLY_HOST="${2:-}"; shift 2 ;; + --commit) COMMIT=1; shift ;; + -h|--help) ACTION=""; break ;; + -*) echo "FAIL: unknown option '$1'" >&2; exit 2 ;; + *) if [ -z "$ACTION" ]; then ACTION="$1"; elif [ -z "$SITE" ]; then SITE="$1"; else + echo "FAIL: unexpected argument '$1'" >&2; exit 2; fi; shift ;; + esac +done + +usage() { + cat >&2 <<'U' +usage: dc-node-carve.sh --profile

--expect-rack [--host ] [--commit] + e.g. dc-node-carve.sh check vr1-dc0 --profile vr1-dc0-region --expect-rack hot-kid + dc-node-carve.sh apply vr1-dc0 --profile vr1-dc0-region --expect-rack hot-kid \ + --host vr1-dc0-storage-04 --commit + + check assert the target carve, read-only. This is the NAMED GATE (GA-R6). + apply without --commit: resolve every id live and print the plan. Changes nothing. + with --commit: execute, reading back after every mutation. + --host restrict to ONE host. Use it: enlistment precedent says canary first, + then batches of three. +U + exit 2 +} +[ -n "$ACTION" ] && [ -n "$SITE" ] || usage +case "$ACTION" in check|apply) ;; *) usage ;; esac + +# --- no default profile ---------------------------------------------------- +# The repo-wide default `MAAS_PROFILE=admin` resolves to the OFFICE1 region, +# where dc1's nine nodes still live. A 60-NIC re-home against the wrong region +# is either an idempotent no-op that prints PASS, or damage to the wrong DC. +[ -n "$PROFILE" ] || { echo "FAIL: --profile is REQUIRED. There is no default: 'admin' resolves to the OFFICE1 region." >&2; exit 2; } +[ -n "$EXPECT_RACK" ] || { echo "FAIL: --expect-rack is REQUIRED -- it is what proves the profile points at the region you mean." >&2; exit 2; } + +# --- site refusal, symmetric to carve-host-interfaces.sh ------------------- +# That script REFUSES vr1-* because it builds the VR0 Pattern-A VLAN-103 stack +# that D-133 supersedes. This one is its mirror image: it builds the D-133 FLAT +# per-NIC carve, which has no meaning on a VR0 fleet. +case "$SITE" in + vr1-dc0|vr1-dc1) ;; + *) echo "FAIL: site '$SITE' refused -- this script builds the D-133 FLAT per-NIC VR1 carve. VR0 hosts use scripts/carve-host-interfaces.sh (Pattern A + the VID-103 br-metal stack)." >&2; exit 2 ;; +esac + +command -v maas >/dev/null 2>&1 || { echo "REFUSE: no 'maas' CLI on this host" >&2; exit 2; } +command -v python3 >/dev/null 2>&1 || { echo "REFUSE: no python3 on this host" >&2; exit 2; } + +# --- region identity gate: BEFORE reading or writing anything -------------- +ASSERT="$HERE/maas-profile-assert.sh" +[ -f "$ASSERT" ] || { echo "REFUSE: $ASSERT missing -- refusing to touch a region whose identity cannot be proven" >&2; exit 2; } +if ! bash "$ASSERT" "$PROFILE" "$EXPECT_RACK"; then + echo "REFUSE: profile '$PROFILE' does not resolve to the expected region -- nothing was read or written" >&2 + exit 2 +fi + +# --- expected state: DERIVED from the libs, never typed -------------------- +# shellcheck source=/dev/null +. "$REPO/scripts/lib-net.sh" +# shellcheck source=/dev/null +. "$REPO/scripts/lib-hosts.sh" +lib_net_select_dc "$SITE" >/dev/null 2>&1 || { echo "FAIL: lib_net_select_dc rejected site '$SITE'" >&2; exit 2; } +lib_hosts_select_dc "$SITE" >/dev/null 2>&1 || { echo "FAIL: lib_hosts_select_dc rejected site '$SITE'" >&2; exit 2; } + +P=0; F=0 +ok(){ echo " [ok] $1"; P=$((P+1)); } +bad(){ echo " [FAIL] $1"; F=$((F+1)); } +ck(){ if [ "$1" = 0 ]; then ok "$2"; else bad "$2"; fi; } +note(){ echo " [note] $1"; } +hdr(){ echo; echo "=== $* ==="; } +refuse(){ echo "REFUSE: $1" >&2; exit 2; } + +m(){ maas "$PROFILE" "$@"; } + +# --- plane name -> CIDR, inverted from lib-net's own map ------------------- +# PLANE_NAME is CIDR->name; NIC_PLANE_ORDER is by name. Invert rather than +# retype, so a plane re-address in lib-net.sh cannot silently disagree here. +declare -A CIDR_OF=() +for _c in "${PLANE_CIDRS[@]}"; do CIDR_OF["${PLANE_NAME[$_c]}"]="$_c"; done +for _p in "${NIC_PLANE_ORDER[@]}"; do + [ -n "${CIDR_OF[$_p]:-}" ] || refuse "plane '$_p' from NIC_PLANE_ORDER has no CIDR in lib-net.sh PLANE_NAME for site $SITE" +done + +# plane_ip -- the D-134 host part is the SAME on every plane. +plane_ip(){ printf '%s.%s' "$(printf '%s' "${1%/*}" | cut -d. -f1-3)" "$2"; } +# nic_for -- macs[i] in the substrate root IS enps0 (measured on +# vr1-dc0-control-01 against the live carve). +nic_for(){ printf 'enp%ss0' "$(( $1 + 1 ))"; } + +# --- live id resolution, never persisted ----------------------------------- +SUBJSON=""; MACHJSON="" +reload_static() { + SUBJSON="$(m subnets read 2>/dev/null)" || refuse "could not read subnets" + MACHJSON="$(m machines read 2>/dev/null)" || refuse "could not read machines" + printf '%s' "$SUBJSON" | python3 -c 'import json,sys; json.load(sys.stdin)' 2>/dev/null || refuse "subnets payload is not JSON" + printf '%s' "$MACHJSON" | python3 -c 'import json,sys; json.load(sys.stdin)' 2>/dev/null || refuse "machines payload is not JSON" +} + +# sub_field -- subnets keyed by CIDR, the stable key. +sub_field(){ printf '%s' "$SUBJSON" | python3 -c ' +import json,sys +cidr,field=sys.argv[1],sys.argv[2] +for s in json.load(sys.stdin): + if s.get("cidr")==cidr: + if field=="vlan": print((s.get("vlan") or {}).get("id","")) + elif field=="gateway_ip": print(s.get("gateway_ip") or "none") + else: print(s.get(field,"")) + break +' "$1" "$2" 2>/dev/null; } + +# mach_by_mac -- the ONLY stable key. +mach_by_mac(){ printf '%s' "$MACHJSON" | python3 -c ' +import json,sys +mac,field=sys.argv[1].lower(),sys.argv[2] +for mm in json.load(sys.stdin): + bi=mm.get("boot_interface") or {} + if (bi.get("mac_address") or "").lower()==mac: + print(mm.get(field,"")); break +' "$1" "$2" 2>/dev/null; } + +IFJSON="" +reload_ifaces(){ IFJSON="$(m interfaces read "$1" 2>/dev/null)" || refuse "could not read interfaces for $1"; } + +# if_field from the loaded IFJSON +if_field(){ printf '%s' "$IFJSON" | python3 -c ' +import json,sys +n,field=sys.argv[1],sys.argv[2] +for i in json.load(sys.stdin): + if i.get("name")==n: + if field=="vlan": print((i.get("vlan") or {}).get("id","")) + elif field=="parents": print(",".join(i.get("parents") or [])) + else: print(i.get(field,"")) + break +' "$1" "$2" 2>/dev/null; } + +# if_link -- ":" for the link on that subnet, else "". +if_link(){ printf '%s' "$IFJSON" | python3 -c ' +import json,sys +n,cidr=sys.argv[1],sys.argv[2] +for i in json.load(sys.stdin): + if i.get("name")==n: + for l in (i.get("links") or []): + s=l.get("subnet") or {} + if s.get("cidr")==cidr: + print("%s:%s"%(l.get("mode",""),l.get("ip_address") or "-")); sys.exit() +' "$1" "$2" 2>/dev/null; } + +# if_l3_count -- how many links carry a subnet at all (br-ex member must be 0). +if_l3_count(){ printf '%s' "$IFJSON" | python3 -c ' +import json,sys +n=sys.argv[1] +for i in json.load(sys.stdin): + if i.get("name")==n: + print(sum(1 for l in (i.get("links") or []) if l.get("subnet"))); sys.exit() +print(0) +' "$1" 2>/dev/null; } + +# if_names -- every interface name on the machine +if_names(){ printf '%s' "$IFJSON" | python3 -c ' +import json,sys +for i in json.load(sys.stdin): print(i.get("name","")) +' 2>/dev/null; } + +# --- mutation with mandatory read-back ------------------------------------- +MUTATED=0 +# do_update_vlan -- move a NIC onto a plane VLAN. +do_update_vlan(){ + local sid="$1" ifid="$2" vlan="$3" nic="$4" got + if [ "$COMMIT" != 1 ]; then echo " [plan] $nic(id=$ifid) -> VLAN $vlan"; return 0; fi + m interface update "$sid" "$ifid" vlan="$vlan" >/dev/null 2>&1 || { bad "$nic: 'interface update vlan=$vlan' call failed"; return 1; } + reload_ifaces "$sid"; got="$(if_field "$nic" vlan)" + # READ BACK. MAAS has been measured returning 200 while changing nothing. + if [ "$got" = "$vlan" ]; then ok "$nic -> VLAN $vlan (read back '$got')"; MUTATED=$((MUTATED+1)); return 0 + else bad "$nic: VLAN read back as '$got', wanted '$vlan' -- the call reported success and changed nothing"; return 1; fi +} +# do_link_static +do_link_static(){ + local sid="$1" ifid="$2" sub="$3" ip="$4" nic="$5" cidr="$6" got + if [ "$COMMIT" != 1 ]; then echo " [plan] $nic(id=$ifid) -> STATIC $ip on subnet $sub ($cidr)"; return 0; fi + m interface link-subnet "$sid" "$ifid" mode=STATIC subnet="$sub" ip_address="$ip" >/dev/null 2>&1 \ + || { bad "$nic: 'link-subnet STATIC $ip' call failed"; return 1; } + reload_ifaces "$sid"; got="$(if_link "$nic" "$cidr")" + if [ "$got" = "static:$ip" ]; then ok "$nic -> STATIC $ip on $cidr (read back '$got')"; MUTATED=$((MUTATED+1)); return 0 + else bad "$nic: link read back as '$got', wanted 'static:$ip'"; return 1; fi +} +# do_unlink +do_unlink(){ + local sid="$1" ifid="$2" lid="$3" nic="$4" + if [ "$COMMIT" != 1 ]; then echo " [plan] unlink $nic(id=$ifid) link id=$lid (L3 moves to br-ex)"; return 0; fi + m interface unlink-subnet "$sid" "$ifid" id="$lid" >/dev/null 2>&1 || { bad "$nic: unlink link id=$lid failed"; return 1; } + reload_ifaces "$sid"; ok "$nic: unlinked id=$lid (now $(if_l3_count "$nic") L3 link(s))"; MUTATED=$((MUTATED+1)) +} + +# --- one raw plane leg: VLAN move then STATIC ------------------------------ +carve_raw(){ + local sid="$1" nic="$2" cidr="$3" ip="$4" ifid vlan sub cur link + ifid="$(if_field "$nic" id)" + [ -n "$ifid" ] || { bad "$nic not found on this machine"; return 1; } + vlan="$(sub_field "$cidr" vlan)"; sub="$(sub_field "$cidr" id)" + [ -n "$vlan" ] && [ -n "$sub" ] || { bad "$nic: no VLAN/subnet resolved for $cidr in this region -- REFUSING to guess"; return 1; } + + link="$(if_link "$nic" "$cidr")" + if [ "$link" = "static:$ip" ]; then note "$nic already STATIC $ip on $cidr -- SKIP"; ok "$nic STATIC $ip on $cidr"; return 0; fi + # A link on the RIGHT subnet with the WRONG address is a FAILURE, never a skip. + if [ -n "$link" ] && [ "$link" != "static:$ip" ]; then + bad "$nic: already linked to $cidr as '$link' but the ruled address is '$ip' -- refusing to silently accept it"; return 1 + fi + + cur="$(if_field "$nic" vlan)" + if [ "$cur" != "$vlan" ]; then do_update_vlan "$sid" "$ifid" "$vlan" "$nic" || return 1 + else note "$nic already on VLAN $vlan -- SKIP move"; fi + ifid="$(if_field "$nic" id)" + do_link_static "$sid" "$ifid" "$sub" "$ip" "$nic" "$cidr" +} + +# --- Pattern A: the OVS br-ex on the provider uplink ----------------------- +# GUARD STRUCTURE IS DELIBERATE AND MIRRORS carve-host-interfaces.sh:211-236. +# The VLAN move sits INSIDE the "br-ex does not exist yet" branch: a MAAS bridge +# inherits its parent's VLAN AT CREATE, so the parent must be moved first or the +# br-ex static lands on the wrong plane -- and once br-ex exists the parent is +# ENSLAVED, where a VLAN update is invalid. Restructuring this for symmetry with +# carve_raw breaks re-runnability. +build_ovs_brex(){ + local sid="$1" nic="$2" cidr="$3" ip="$4" ifid brid vlan sub lid link + ifid="$(if_field "$nic" id)" + [ -n "$ifid" ] || { bad "$nic not found on this machine"; return 1; } + vlan="$(sub_field "$cidr" vlan)"; sub="$(sub_field "$cidr" id)" + [ -n "$vlan" ] && [ -n "$sub" ] || { bad "br-ex: no VLAN/subnet resolved for $cidr -- REFUSING to guess"; return 1; } + + # 1) clear the member's commissioning link(s) so the host L3 lands on br-ex. + for lid in $(printf '%s' "$IFJSON" | python3 -c ' +import json,sys +n=sys.argv[1] +for i in json.load(sys.stdin): + if i.get("name")==n: + for l in (i.get("links") or []): + if l.get("subnet"): print(l.get("id")) +' "$nic" 2>/dev/null); do + do_unlink "$sid" "$ifid" "$lid" "$nic" || return 1 + done + + # 2) create the OVS bridge, moving the parent onto the provider VLAN FIRST. + brid="$(if_field br-ex id)" + if [ -z "$brid" ]; then + if [ "$(if_field "$nic" vlan)" != "$vlan" ]; then do_update_vlan "$sid" "$ifid" "$vlan" "$nic" || return 1; fi + ifid="$(if_field "$nic" id)" + if [ "$COMMIT" != 1 ]; then echo " [plan] create br-ex (OVS) parent=$nic(id=$ifid)" + else + m interfaces create-bridge "$sid" name=br-ex bridge_type=ovs parent="$ifid" >/dev/null 2>&1 \ + || { bad "br-ex: create-bridge failed"; return 1; } + reload_ifaces "$sid"; brid="$(if_field br-ex id)" + if [ -n "$brid" ] && [ "$(if_field br-ex type)" = "bridge" ]; then + ok "br-ex created (OVS) parent=$nic, read back type='bridge' parents='$(if_field br-ex parents)'"; MUTATED=$((MUTATED+1)) + else bad "br-ex: not present after create-bridge reported success"; return 1; fi + fi + else note "br-ex exists -- SKIP create"; fi + + # 3) the host static lands on br-ex, not on the member NIC. + [ "$COMMIT" = 1 ] && brid="$(if_field br-ex id)" || brid="${brid:-}" + link="$(if_link br-ex "$cidr")" + if [ "$link" = "static:$ip" ]; then note "br-ex already STATIC $ip on $cidr -- SKIP"; ok "br-ex STATIC $ip on $cidr"; return 0; fi + if [ -n "$link" ]; then bad "br-ex: linked to $cidr as '$link' but the ruled address is '$ip'"; return 1; fi + do_link_static "$sid" "$brid" "$sub" "$ip" "br-ex" "$cidr" +} + +# --- role resolution -- DERIVED from the host token, never typed ----------- +is_juju_host(){ case "$1" in *"$JUJU_HOST_SUFFIX") return 0 ;; *) return 1 ;; esac; } +planes_for_host(){ if is_juju_host "$1"; then echo "$NODE_CARVE_JUJU_PLANES"; else echo "$NODE_CARVE_ROLE_PLANES"; fi; } + +# =========================================================================== +reload_static + +PROV_CIDR="${CIDR_OF[provider-public]}" +MA_CIDR="${CIDR_OF[metal-admin]}" + +hdr "region + plane preconditions ($SITE via profile '$PROFILE')" +GW="$(sub_field "$PROV_CIDR" gateway_ip)" +# THE DEFAULT ROUTE IS A NODE-SIDE FACT AND IS NOT OBSERVABLE FROM HERE. What +# this gate can prove is the MODEL that produces it: a static link on +# provider-public plus a gateway on that subnet. The routing table itself is a +# FIRST-BOOT observation (gate G17), not something a green carve establishes. +ck "$([ "$GW" = "10.12.4.1" ] || [ "$GW" = "10.12.64.1" ] && echo 0 || echo 1)" \ + "provider-public $PROV_CIDR carries a gateway (got '$GW') -- the node default route depends on it" +ck "$([ "$(sub_field "$MA_CIDR" gateway_ip)" = "none" ] && echo 0 || echo 1)" \ + "metal-admin $MA_CIDR has NO gateway (got '$(sub_field "$MA_CIDR" gateway_ip)') -- D-134/D-125: it is not an egress plane" + +HOSTS_TODO=() +if [ -n "$ONLY_HOST" ]; then + _found=0; for h in "${HOSTS[@]}"; do [ "$h" = "$ONLY_HOST" ] && _found=1; done + [ "$_found" = 1 ] || refuse "--host '$ONLY_HOST' is not a $SITE host in lib-hosts.sh HOSTS" + HOSTS_TODO=( "$ONLY_HOST" ) +else + HOSTS_TODO=( "${HOSTS[@]}" ) +fi + +for HOST in "${HOSTS_TODO[@]}"; do + MAC="${HOST_BOOT_MAC[$HOST]:-}"; OCTET="${HOST_OCTET[$HOST]:-}" + [ -n "$MAC" ] && [ -n "$OCTET" ] || { bad "$HOST: no pinned boot MAC / octet in lib-hosts.sh"; continue; } + SID="$(mach_by_mac "$MAC" system_id)" + if [ -z "$SID" ]; then bad "$HOST: pinned boot MAC $MAC has NO machine in this region -- not enrolled here"; continue; fi + HN="$(mach_by_mac "$MAC" hostname)"; ST="$(mach_by_mac "$MAC" status_name)" + NPLANES="$(planes_for_host "$HOST")" + hdr "$HOST -> $SID/$HN (octet $OCTET, $NPLANES plane(s), status $ST)" + + # MAAS REFUSES interface changes on a Deployed machine. Assert before mutating + # rather than discovering it call-by-call. + if [ "$ACTION" = apply ] && [ "$ST" != "Ready" ]; then + bad "$HOST is '$ST', not 'Ready' -- MAAS rejects interface changes on a Deployed machine; skipping" + continue + fi + reload_ifaces "$SID" + + IDX=0 + for PLANE in "${NIC_PLANE_ORDER[@]}"; do + [ "$IDX" -lt "$NPLANES" ] || break + NIC="$(nic_for "$IDX")"; CIDR="${CIDR_OF[$PLANE]}"; IP="$(plane_ip "$CIDR" "$OCTET")" + if [ "$ACTION" = check ]; then + if [ "$PLANE" = provider-public ] && ! is_juju_host "$HOST"; then + ck "$([ "$(if_link br-ex "$CIDR")" = "static:$IP" ] && echo 0 || echo 1)" "$HOST br-ex STATIC $IP on $PLANE (got '$(if_link br-ex "$CIDR")')" + ck "$([ "$(if_field br-ex type)" = "bridge" ] && echo 0 || echo 1)" "$HOST br-ex is a bridge (got '$(if_field br-ex type)')" + ck "$([ "$(if_field br-ex parents)" = "$NIC" ] && echo 0 || echo 1)" "$HOST br-ex parent is $NIC (got '$(if_field br-ex parents)')" + # The member NIC must hold NO L3 link, or the host has two addresses on + # one plane and OVS competes with the raw NIC (the D-057 Pattern B fault + # that darkened the floating-IP plane). + ck "$([ "$(if_l3_count "$NIC")" = "0" ] && echo 0 || echo 1)" "$HOST $NIC holds no L3 link (got $(if_l3_count "$NIC")) -- it is an OVS member" + else + ck "$([ "$(if_link "$NIC" "$CIDR")" = "static:$IP" ] && echo 0 || echo 1)" "$HOST $NIC STATIC $IP on $PLANE (got '$(if_link "$NIC" "$CIDR")')" + ck "$([ "$(if_field "$NIC" vlan)" = "$(sub_field "$CIDR" vlan)" ] && echo 0 || echo 1)" "$HOST $NIC on the $PLANE VLAN (got '$(if_field "$NIC" vlan)' want '$(sub_field "$CIDR" vlan)')" + fi + else + if [ "$PLANE" = provider-public ] && ! is_juju_host "$HOST"; then + build_ovs_brex "$SID" "$NIC" "$CIDR" "$IP" || true + else + carve_raw "$SID" "$NIC" "$CIDR" "$IP" || true + fi + fi + IDX=$((IDX+1)) + done + + # The controller's enp3s0..enp6s0 are DELIBERATELY left on auto VLANs. Assert + # that deliberate absence, so a later session cannot "helpfully" carve them + # and have every gate stay green. + if [ "$ACTION" = check ] && is_juju_host "$HOST"; then + _stray=0 + for _i in $(seq "$NPLANES" 5); do + _n="$(nic_for "$_i")" + for _p in "${NIC_PLANE_ORDER[@]}"; do + [ -n "$(if_link "$_n" "${CIDR_OF[$_p]}")" ] && _stray=$((_stray+1)) + done + done + ck "$([ "$_stray" = 0 ] && echo 0 || echo 1)" "$HOST enp$((NPLANES+1))s0..enp6s0 carry NO plane link ($_stray found) -- deliberately left on auto VLANs" + ck "$([ -z "$(if_field br-ex id)" ] && echo 0 || echo 1)" "$HOST has NO br-ex (got id '$(if_field br-ex id)') -- the controller takes the provider leg RAW" + fi +done + +echo +if [ "$ACTION" = apply ] && [ "$COMMIT" != 1 ]; then + echo "PLAN ONLY -- nothing was changed. Re-run with --commit to execute." +fi +[ "$COMMIT" = 1 ] && echo "mutations applied (each read back): $MUTATED" +echo "NOTE: this gate proves the MAAS MODEL. The node's actual routing table is a" +echo " FIRST-BOOT observation (G17) -- a green carve is not a proven default route." +echo "RESULT: pass=$P fail=$F" +if [ "$F" -gt 0 ]; then echo "FAIL: dc-node-carve $ACTION $SITE"; exit 1; fi +echo "PASS: dc-node-carve $ACTION $SITE" +exit 0 diff --git a/tests/HARNESS-MANIFEST b/tests/HARNESS-MANIFEST index 3670a13..78aebbe 100644 --- a/tests/HARNESS-MANIFEST +++ b/tests/HARNESS-MANIFEST @@ -23,6 +23,7 @@ dc-dc-whole-host-budget dc-edge-wan-import dc-mirror +dc-node-carve dc-node-v6-carve dc-plane-ipam dc-rack-mgmt-import diff --git a/tests/dc-node-carve/run-tests.sh b/tests/dc-node-carve/run-tests.sh new file mode 100755 index 0000000..ca5c66b --- /dev/null +++ b/tests/dc-node-carve/run-tests.sh @@ -0,0 +1,280 @@ +#!/usr/bin/env bash +# tests/dc-node-carve/run-tests.sh -- unit tests for scripts/dc-node-carve.sh +# (2026-07-30). +# +# A stub `maas` on PATH serves JSON fixtures, so no live region is touched. +# STUB_STATE selects a fixture set: +# raw the MEASURED starting state of every migrated dc0 machine: +# enp1s0 with an AUTO link on metal-admin, enp2s0..enp6s0 each on a +# FRESH auto-created fabric with `link_up` and no subnet. +# carved the intended end state: 9 role nodes on all six planes with an +# OVS br-ex, the juju controller on two planes raw. +# wrongip carved, except one node's storage leg holds the WRONG address on +# the RIGHT subnet. This MUST fail, never skip. +# patternb carved, except a role node's provider static sits on the raw +# enp2s0 with no br-ex -- the D-057 Pattern B shape that darkened +# the floating-IP plane. +# memberl3 carved, except the br-ex MEMBER NIC still holds an L3 link, so +# the host has two addresses on the provider plane. +# brexip carved, and br-ex is a correctly-parented OVS bridge, but its +# static is the WRONG address. ADDED after a mutation pass found +# the br-ex static compare could be neutered with the suite still +# green: `patternb` was being caught by the type/parent assertions +# instead, so the address compare itself was never exercised. +# jujucarved carved, except the juju controller's enp3s0 has been "helpfully" +# carved onto a plane it is deliberately meant to leave alone. +# jujubrex carved, except the juju controller has grown a br-ex. +# nogw carved, except provider-public lost its gateway_ip -- the model +# fact the node default route depends on. +# +# THE POINT OF THIS HARNESS is that `check` must FAIL on every one of those +# partial or wrong states. The repo's receipts are three-for-three that freshly +# written assertions are decoration until each is proven able to fail, so every +# assertion added here has a failing-direction fixture above. +# +# It also pins the REGION GATE (no default profile; refuse when the profile does +# not resolve to the expected rack) and the SITE refusal, which is the mirror +# image of carve-host-interfaces.sh refusing vr1-*. +set -uo pipefail +SD="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "$SD/../.." && pwd)" +SCRIPT="$ROOT/scripts/dc-node-carve.sh" +P=0; F=0 +ok(){ echo "PASS: $1"; P=$((P+1)); } +no(){ echo "FAIL: $1"; F=$((F+1)); } +chk(){ [ "$2" = "$3" ] && ok "$1" || no "$1 (got '$2' want '$3')"; } +has(){ printf '%s' "$2" | grep -q -- "$3" && ok "$1" || no "$1 (output lacked '$3')"; } +hasnt(){ printf '%s' "$2" | grep -q -- "$3" && no "$1 (output unexpectedly had '$3')" || ok "$1"; } + +TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT +mkdir -p "$TMP/bin" + +cat > "$TMP/bin/maas" <<'STUB' +#!/usr/bin/env bash +prof="$1"; shift +sub="$1"; shift +S="${STUB_STATE:-carved}" + +# --- the ten dc0 machines, keyed by their PINNED boot MACs (lib-hosts.sh) ---- +# NOTE: this heredoc is DELIBERATELY unquoted -- STUB_S04_STATUS must expand. +# A quoted <<'M' leaves the literal ${...} in place and the payload stops being +# JSON, which is the same invisible-heredoc class as finding F8. +emit_machines() { +cat <` -> $1=read $2=sysid ; create-bridge -> $1=create-bridge $2=sysid + if [ "$act" = "create-bridge" ]; then echo '{"id":90}'; exit 0; fi + if [ "$S" = raw ]; then emit_raw; exit 0; fi + case "$sid" in + j01) emit_juju ;; + *) emit_role "$(octet_of "$sid")" ;; + esac ;; + interface) echo '{}' ;; + *) echo '[]' ;; +esac +STUB +chmod +x "$TMP/bin/maas" +export PATH="$TMP/bin:$PATH" + +R(){ STUB_STATE="$1" bash "$SCRIPT" "${@:2}" 2>&1; } +GATE=(--profile vr1-dc0-region --expect-rack hot-kid) + +echo "== T1..T6 argument + region + site gates ==" +out="$(bash "$SCRIPT" check vr1-dc0 --expect-rack hot-kid 2>&1)"; rc=$? +chk "T1 missing --profile refuses rc=2" "$rc" "2" +has "T1b names the OFFICE1 hazard" "$out" "OFFICE1" + +out="$(bash "$SCRIPT" check vr1-dc0 --profile vr1-dc0-region 2>&1)"; rc=$? +chk "T2 missing --expect-rack refuses rc=2" "$rc" "2" + +out="$(bash "$SCRIPT" check vr0-dc0 "${GATE[@]}" 2>&1)"; rc=$? +chk "T3 vr0 site refused rc=2" "$rc" "2" +has "T3b points at carve-host-interfaces.sh" "$out" "carve-host-interfaces.sh" + +out="$(STUB_RACK=wrong-rack bash "$SCRIPT" check vr1-dc0 "${GATE[@]}" 2>&1)"; rc=$? +chk "T4 wrong region refuses rc=2" "$rc" "2" +hasnt "T4b nothing was read after the refusal" "$out" "plane preconditions" + +out="$(STUB_RACK=none bash "$SCRIPT" check vr1-dc0 "${GATE[@]}" 2>&1)"; rc=$? +chk "T5 region with no racks refuses rc=2" "$rc" "2" + +out="$(R carved check vr1-dc0 "${GATE[@]}" --host not-a-host)"; rc=$? +chk "T6 unknown --host refuses rc=2" "$rc" "2" + +echo +echo "== T7..T9 the intended end state PASSES ==" +out="$(R carved check vr1-dc0 "${GATE[@]}")"; rc=$? +chk "T7 carved state passes rc=0" "$rc" "0" +has "T7b reports fail=0" "$out" "fail=0" +has "T8 asserts br-ex parent" "$out" "br-ex parent is enp2s0" +has "T8b asserts the member holds no L3" "$out" "holds no L3 link" +has "T9 asserts the controller has NO br-ex" "$out" "has NO br-ex" +has "T9b asserts the controller's spare NICs are left alone" "$out" "deliberately left on auto VLANs" +has "T9c states the routing table is a first-boot fact" "$out" "FIRST-BOOT observation" + +echo +echo "== T10..T15 every wrong state must FAIL (the mutation-proof set) ==" +out="$(R raw check vr1-dc0 "${GATE[@]}")"; rc=$? +chk "T10 the measured uncarved state FAILS rc=1" "$rc" "1" + +out="$(R wrongip check vr1-dc0 "${GATE[@]}")"; rc=$? +chk "T11 right subnet + WRONG address FAILS rc=1" "$rc" "1" +has "T11b names the wrong address" "$out" "10.12.32.199" + +out="$(R patternb check vr1-dc0 "${GATE[@]}")"; rc=$? +chk "T12 Pattern B (static on raw enp2s0, no br-ex) FAILS rc=1" "$rc" "1" + +out="$(R memberl3 check vr1-dc0 "${GATE[@]}")"; rc=$? +chk "T13 br-ex member still holding an L3 link FAILS rc=1" "$rc" "1" + +out="$(R jujucarved check vr1-dc0 "${GATE[@]}")"; rc=$? +chk "T14 controller's spare NIC carved FAILS rc=1" "$rc" "1" + +out="$(R jujubrex check vr1-dc0 "${GATE[@]}")"; rc=$? +chk "T15 controller grown a br-ex FAILS rc=1" "$rc" "1" + +out="$(R brexip check vr1-dc0 "${GATE[@]}")"; rc=$? +chk "T15c br-ex with the WRONG static FAILS rc=1" "$rc" "1" +has "T15d names the wrong br-ex address" "$out" "10.12.4.199" + +out="$(R nogw check vr1-dc0 "${GATE[@]}")"; rc=$? +chk "T16 provider-public with no gateway FAILS rc=1" "$rc" "1" +has "T16b explains the default route depends on it" "$out" "default route" + +echo +echo "== T17..T20 role asymmetry and NIC order are DERIVED, not typed ==" +out="$(R carved check vr1-dc0 "${GATE[@]}" --host vr1-dc0-juju-01)"; rc=$? +chk "T17 controller alone passes rc=0" "$rc" "0" +has "T17b controller carved on 2 planes" "$out" "2 plane(s)" +hasnt "T17c controller is NOT given a data-tenant leg" "$out" "enp4s0 STATIC 10.12.16" + +out="$(R carved check vr1-dc0 "${GATE[@]}" --host vr1-dc0-control-01)"; rc=$? +chk "T18 role node alone passes rc=0" "$rc" "0" +has "T18b role node carved on 6 planes" "$out" "6 plane(s)" +# enp1s0 is METAL-ADMIN, not provider-public. Walking lib-net.sh PLANE_CIDRS +# positionally would put PXE on provider-public and strand commissioning. +has "T19 enp1s0 is metal-admin (NIC_PLANE_ORDER, not PLANE_CIDRS)" "$out" "enp1s0 STATIC 10.12.8.100 on metal-admin" +has "T19b enp2s0/br-ex is provider-public" "$out" "br-ex STATIC 10.12.4.100 on provider-public" + +echo +echo "== T21..T24 apply is DRY-RUN without --commit ==" +out="$(R raw apply vr1-dc0 "${GATE[@]}" --host vr1-dc0-storage-04)"; rc=$? +has "T21 plan-only banner" "$out" "PLAN ONLY -- nothing was changed" +has "T22 plans the VLAN move" "$out" "\[plan\] enp3s0" +has "T23 plans the br-ex create" "$out" "\[plan\] create br-ex (OVS) parent=enp2s0" +hasnt "T24 dry-run reports no mutations" "$out" "mutations applied" + +out="$(STUB_S04_STATUS=Deployed R carved apply vr1-dc0 "${GATE[@]}" --host vr1-dc0-storage-04 --commit)"; rc=$? +chk "T25 apply on a Deployed machine FAILS rc=1" "$rc" "1" +has "T25b explains MAAS rejects it" "$out" "rejects interface changes" + +echo +echo "RESULT: $P passed, $F failed" +[ "$F" -eq 0 ] || exit 1