diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 3e04562..a72aec5 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -467,6 +467,50 @@ classifier refusing the wrapped form). Captures go to `docs/audit/*`, which is what GA-R6 requires of a gate; an index row is owed at close. A log that looks complete and is not is worse than one declaring its gap. + **MIGRATION PREREQ 4 DONE 2026-07-30 -- POWER PATH IS BUILT AND PROVEN AT dc0.** Capture + `docs/audit/dc0-region-power-key-20260730.txt`, `maas-region-power-key.sh check vr1-dc0` + **9 assertions / 0 failed, exit 0**, ending on the ARTIFACT: a real virsh connect to + `qemu+ssh://jessea123@10.12.8.2/system` that ENUMERATES 12 domains. MEASURED before the + work: `/var/snap/maas/current/root/.ssh/` did not exist on the new region at all, so + commissioning could not have powered a node on. The installed key is the SAME dc0-scoped + SEC-012 key already in the dc0 rack's `authorized_keys` (fingerprint + `SHA256:Dt/YXTXSF4nXGj9cz8f0owL+qreVpMYBVm/igW10FXY` matched at both ends), so per-DC + isolation holds and this is NOT a cross-DC reuse; **OWED: remove Office1's copy once the + dc0 migration completes**, as that region will no longer power dc0 machines. + **`scripts/lib-hosts.sh` NOW CARRIES THE POWER ADDRESS PER REGION** (harness + `tests/dc-selector` 81 checks, 4 mutations each killed tests): MAAS power ops originate + from the REGION, and from `vr1-dc0-maas-01` the rack's TRANSIT leg `172.31.0.2:22` is + **CLOSED** while its metal-admin leg `10.12.8.2:22` is **OPEN**. + `VIRSH_POWER_ADDRESS_FROM_DCREGION` is added for both DCs (dc1's `10.12.68.2` MEASURED on + the dc1 rack, not inferred from dc0's `.2`); `VIRSH_POWER_ADDRESS` still ALIASES the + Office1 form so `reenroll-hosts.sh` and the teardown runbook's virsh probes are unchanged. + Flipping the default is OWED once BOTH DCs have migrated. The default FAILS CLOSED. + **THE REBUILD IS BIGGER THAN "RE-RUN THE GENERATOR", MEASURED BY A READ-ONLY SURVEY.** + **(i) The pre-migration capture is NOT a diff target** -- its `vlan_id=5189..5193` and + `id=` values are MAAS DATABASE ROW IDS, not 802.1Q tags (every named fabric reads + `vid=0`), and its reserved-range section COALESCES state-dependently, so a perfect rebuild + would not textually match. Stable identities are fabric NAME, space NAME, subnet CIDR, tag + NAME, boot MAC, interface NAME -- which `lib-net.sh:8-10` already states repo-wide. The + gate is the `check` actions, not a file diff. **(ii) THREE components have NO repo tool:** + the named plane fabrics, the six v4 plane subnets, and the per-node v4 NIC carve (60 NIC + re-homes + 9 `br-ex` + 54 statics) -- all done ad-hoc in the Stage-4 window, logged only to + `~/as-executed/2026-07-23-stage4-carve.log`, which is NOT in the repo. The + `openstack-vr1-dc0` tag has no usable creator either. **(iii) `dc-node-v6-carve.py` IS + PROFILE-ENV-BLIND** (`PROFILE_DEFAULT = "admin"` at `:39`, `--profile` at `:70`, no env + read anywhere), so `export MAAS_PROFILE=...` silently targets OFFICE1 -- a live foot-gun on + a tool the migration needs. **(iv) `phase-00-maas-standup.sh` MUST NOT be pointed at the + new region:** unlike `carve-host-interfaces.sh:62-64` and `reenroll-hosts.sh:53-55`, which + REFUSE `vr1-*` outright, it does not refuse -- its parity guard PASSES, then it names + fabrics after the SPACE (`provider-public`, not `vr1-dc0-provider-public`) and builds + metal-internal as a tagged VID-103 VLAN that D-133 abolished for VR1. + **PRE-EXISTING GATE FAILURE FOUND AND FIXED: `tests/node-vm` T8/T9 were RED at HEAD + `c349ace`** (13 passed / 2 failed, PROVEN by stashing all of this session's work and + re-running). Commits `086c827` + `447315f` added the region VMs and pinned dc0's MACs, + moving the dc0 root to 11 macs lists / 66 MAC literals while the assertions still read + 10 / 60; the session that made those commits did not re-run the gauntlet. Assertions + RE-POINTED to the new invariant (11 nodes x 6 planes) with the reason recorded in-file and + re-proven able to fail. dc1 reads 11 lists / 60 literals -- EXPECTED, since + `vr1-dc1-maas-01` is authored with `macs = []` and not yet applied. - Project: Omega Cloud, VR1 DC-DC rehearsal -- a two-DC + Office1-headend virtual rehearsal on KVM (vcloud host), rehearsing the future bare-metal diff --git a/docs/audit/dc0-region-power-key-20260730.txt b/docs/audit/dc0-region-power-key-20260730.txt new file mode 100644 index 0000000..a118697 --- /dev/null +++ b/docs/audit/dc0-region-power-key-20260730.txt @@ -0,0 +1,26 @@ +PER-DC MAAS->libvirt POWER KEY -- new dc0 region (SEC-012) +Captured 2026-07-30 ON vr1-dc0-maas-01 (10.12.8.6), the region host. +========================================================== + +-- the key is the SAME dc0-scoped key already authorized on the dc0 rack -- +vcloud ~/vr1-dc0-creds/vr1-dc0-maas-power_ed25519.pub: +256 SHA256:Dt/YXTXSF4nXGj9cz8f0owL+qreVpMYBVm/igW10FXY maas-virsh-pod (D-123 amendment; MAAS snap -> local libvirt) (ED25519) +dc0 rack ~/.ssh/authorized_keys: +256 SHA256:DBkeStCyf2qGi3SYuvN6hQ+EsepmhpkN9oulLz09uAs vr1-dc0_svc (D-126 per-env key) (ED25519) +256 SHA256:Dt/YXTXSF4nXGj9cz8f0owL+qreVpMYBVm/igW10FXY maas-virsh-pod (D-123 amendment; MAAS snap -> local libvirt) (ED25519) + +-- named check, run on the region host -- +$ maas-region-power-key.sh check vr1-dc0 qemu+ssh://jessea123@10.12.8.2/system +maas-region-power-key check: site=vr1-dc0 dc=dc0 rack=10.12.8.2 user=jessea123 + key=/var/snap/maas/current/root/.ssh/id_dc0_power + [ok] power key present + [ok] key mode is 0600 (got 600) + [ok] key owned by root (got root) + [ok] key parses as a valid PRIVATE key (SHA256:Dt/YXTXSF4nXGj9cz8f0owL+qreVpMYBVm/igW10FXY) + [ok] ssh config present + [ok] config binds Host 10.12.8.2 -> /var/snap/maas/current/root/.ssh/id_dc0_power + [ok] config sets IdentitiesOnly yes for 10.12.8.2 + [ok] REAL virsh (snap run --shell maas -c virsh) connect to qemu+ssh://jessea123@10.12.8.2/system succeeds + [ok] virsh enumerates domains on the rack (got 12) +maas-region-power-key: 9 passed, 0 failed + EXIT = 0 diff --git a/docs/changelog-20260730-dc0-region-migration.md b/docs/changelog-20260730-dc0-region-migration.md index c7fc504..f0164da 100644 --- a/docs/changelog-20260730-dc0-region-migration.md +++ b/docs/changelog-20260730-dc0-region-migration.md @@ -130,7 +130,173 @@ --- -## Item 4 -- as-executed log NOT used for this window, and why +## Item 4 -- `lib-hosts.sh`: the power address is now per-REGION (both DCs) + +**What.** Each VR1 arm gains `VIRSH_POWER_ADDRESS_FROM_OFFICE1` (the rack's +transit leg) and `VIRSH_POWER_ADDRESS_FROM_DCREGION` (the rack's metal-admin leg). +`VIRSH_POWER_ADDRESS` is now an ALIAS of the Office1 form, byte-for-byte unchanged. +The flat/VR0 layer declares both as EMPTY. + +**Why (measured).** MAAS power ops originate from the REGION, so the reachable rack +address depends on which region dials. From `vr1-dc0-maas-01` (10.12.8.6): + + 172.31.0.2:22 CLOSED <- the transit address lib-hosts has always used + 10.12.8.2:22 OPEN <- the rack's metal-admin leg + routes: 10.12.4.0/22, 10.12.8.0/22, default via 10.12.4.1 + +dc1's `10.12.68.2` was MEASURED on the dc1 rack (`ip -4 -o addr` -> `virbr6 inet +10.12.68.2/22`) and corroborated against `dc-rack-net.sh`'s dc1 `LEGS` block. It was +NOT inferred from dc0's `.2` by symmetry. + +**Backward compatibility is the load-bearing half.** `reenroll-hosts.sh:132` and the +teardown runbook's `virsh -c "$VIRSH_POWER_ADDRESS"` probes must not silently change +target while Office1 still owns dc1's nodes. Flipping the default to the DC-region +form is OWED once BOTH DCs have migrated. + +**Harness `tests/dc-selector` 81 checks ALL PASS, 4 mutations each killed tests:** + +| mutation | tests killed | +|---|---| +| A: collapse dc0's DCREGION onto the transit form | 2 | +| B: dc1's DCREGION gets dc0's octet (cross-DC contamination) | 2 | +| C: `VIRSH_POWER_ADDRESS` silently repointed to the DC-region form | 3 | +| D: VR0 inherits a VR1 form | 1 | + +**Revert.** `git checkout ^ -- scripts/lib-hosts.sh tests/dc-selector/run-tests.sh`. + +--- + +## Item 5 -- `scripts/maas-region-power-key.sh` + harness (NEW), and INSTALLED at dc0 + +**What.** `check|install `. Installs the per-DC MAAS->libvirt +power key into a region's snap and verifies it with a REAL virsh call. Key is read +from STDIN, so it never lands on an intermediate host and never appears in a process +argument. + +**Why.** `runbooks/dc-dc-phase2-tofu-dc-substrate.md` prerequisite 5 carries this as +PROSE, and prose is not a gate. It must be re-run at every DC standup and -- per that +same prerequisite's own FRAGILITY note -- after EVERY MAAS snap refresh, because the +key lives under per-revision `/var/snap/maas/current/`. MEASURED before this work: +`/var/snap/maas/current/root/.ssh/` did not exist on the new region at all, so +commissioning could not have powered a node on. + +**LIVE at dc0** (capture `docs/audit/dc0-region-power-key-20260730.txt`), 9/9 exit 0, +ending on the artifact rather than the config: + + [ok] key mode 0600 / owned by root / parses as a valid PRIVATE key + (SHA256:Dt/YXTXSF4nXGj9cz8f0owL+qreVpMYBVm/igW10FXY) + [ok] config binds Host 10.12.8.2 -> id_dc0_power, IdentitiesOnly yes + [ok] REAL virsh connect to qemu+ssh://jessea123@10.12.8.2/system succeeds + [ok] virsh enumerates domains on the rack (got 12) + +The key is the SAME dc0-scoped key already authorized on the dc0 rack (fingerprint +matched against the rack's `authorized_keys`), so SEC-012's per-DC isolation holds -- +this is not a cross-DC reuse. **OWED: remove Office1's copy once dc0's migration +completes**, since that region will no longer power dc0 machines. + +**THREE DEFECTS THIS SCRIPT'S OWN TESTING FOUND, all measured, none by review:** + +1. **`ssh-keygen -lf ` reads an ADJACENT `.pub` when one exists** and reports + ITS fingerprint. A stale sibling made a freshly-installed key read back as a + different one, which would have made the idempotency and cross-DC-swap refusals + answer about the wrong key. Fixed with `ssh-keygen -y -f` (derive the public half + FROM the private key), which no neighbouring file can fool. **Found by the harness.** +2. **A snap has TWO roots and they are not interchangeable.** `/var/snap//current` + is writable data (where `root/.ssh` lives); `/snap//current` is the read-only + squashfs (where the binaries live). The first live run REFUSED with "no virsh binary + found" while `find /snap/maas/current -name virsh` had already located one. +3. **The snap's binary must be run INSIDE the snap's runtime.** Invoking + `/snap/maas/current/usr/bin/virsh` directly fails with `error while loading shared + libraries: libvirt-lxc.so.0`. `snap run --shell maas -c 'virsh ...'` executes it with + the snap's environment AND its ssh config -- which is exactly the context MAAS's own + power driver uses, so this is the correct artifact to grade, not a proxy. + +**Harness 44/44.** Assertions include every refusal direction (absent key, wrong-host +binding, missing `IdentitiesOnly`, unreachable virsh, virsh listing ZERO domains) and +the cross-DC-swap refusal, verified not to overwrite. Two harness assertions were +found passing FOR THE WRONG REASON during development and were made specific. + +**A NOTE ON MY OWN INSTRUMENT.** The first live install reported `EXIT=0` while the +check had actually returned 2 -- I had read `$?` from the end of a pipeline +(`... | tail`), so I measured `tail`'s exit, not the script's. **That is the identical +class this repo recorded twice on 2026-07-30** (`| grep -q` under `pipefail`). Fixed by +capturing into a variable and reading `$?` directly. + +**Revert.** `git rm scripts/maas-region-power-key.sh tests/maas-region-power-key/`. To +undo the live install: on the region VM, +`sudo rm /var/snap/maas/current/root/.ssh/id_dc0_power` and delete the `Host 10.12.8.2` +stanza from that dir's `config`. + +--- + +## Item 6 -- RECON: what actually builds the dc0 carve (read-only; changes nothing) + +A read-only survey mapped each of the seven carve components to its creating tool. +**Four findings change the migration plan; all four were verified against the files +before being recorded here.** + +**(a) THE PRE-MIGRATION CAPTURE IS NOT A DIFF TARGET, and I had planned to use it as +one.** Its `vlan_id=5189..5193` / `id=6,7,12..15,20..25` / `spaces id=1..6` values are +MAAS **database row ids, not 802.1Q tags** -- every named fabric line reads `vid=0`. +Row ids are region-local and will differ on a rebuild. Its RESERVED-RANGES section is +`subnet reserved-ip-ranges` output, which COALESCES and is state-dependent (metal-internal +et al. show a merged `.4-.99 reserved`, while provider-public splits `.4-.49`/`.50-.99` +because `.4-.49` there also carries `assigned-ip` from live statics). **A textual diff of +a perfectly rebuilt region against this file would NOT match.** The only stable identities +are fabric NAME, space NAME, subnet CIDR, tag NAME, node boot MAC, interface NAME -- which +is what `scripts/lib-net.sh:8-10` already says repo-wide ("CIDR is the stable key +throughout; MAAS subnet IDs drift across cutovers"). The real gate is the `check` actions +(`dc-plane-ipam.sh check`, `dc-node-v6-carve.py check`, `maas-role-tags.sh check`), not a +file diff. My task-7 plan is corrected accordingly. + +**(b) `scripts/dc-node-v6-carve.py` IS PROFILE-ENV-BLIND -- verified at the file.** +`PROFILE_DEFAULT = "admin"` (`:39`) and `--profile` (`:70`), with NO environment read +anywhere. So `export MAAS_PROFILE=vr1-dc0-region` -- the idiom every bash tool here +honours -- **silently targets Office1**. Combined with this changelog's own item 1 +reasoning (a carve run against the wrong profile is an idempotent no-op that prints +PASS), this is a live foot-gun on exactly the tool the migration needs. It must be +invoked with an explicit `--profile`, or fixed to read the env. + +**(c) THREE COMPONENTS HAVE NO REPO TOOL AT ALL.** The named plane fabrics, the six v4 +plane subnets, and the per-node v4 NIC carve (60 NIC re-homes + 9 `br-ex` bridges + 54 +statics) were all created ad-hoc during the Stage-4 window; the mutations went to +`~/as-executed/2026-07-23-stage4-carve.log`, which is NOT in the repo. The +`openstack-vr1-dc0` tag has no usable creator either. So the "re-run the generator +against the new endpoint" plan is only available for roughly half the work. + +**(d) TWO EXISTING SCRIPTS MUST NOT BE POINTED AT THE NEW REGION, both verified:** +- `scripts/carve-host-interfaces.sh:62-64` REFUSES `vr1-*` by design (it carves the VR0 + Pattern-A VLAN-103 stack, superseded for VR1 by D-133). `scripts/reenroll-hosts.sh:53-55` + refuses `vr1-*` likewise. +- `scripts/phase-00-maas-standup.sh` does NOT refuse, and that is the danger: its parity + guard PASSES for `vr1-dc0`, then it creates fabrics named after the SPACE + (`provider-public`, not `vr1-dc0-provider-public`, `:192-194`), creates a named + `metal-admin` fabric the as-built does not have, and builds metal-internal as a tagged + VID-103 VLAN riding metal-admin's fabric -- which D-133 abolished for VR1. It would + silently produce a divergent region. + +**Also recorded:** `runbooks/phase-01-bundle-deploy.md:128` iterates **hardcoded subnet +ids** (`for ID in 1 2 6 7 8 9`) and is profile-blind -- in a file that warns against +exactly this at `:54` and `:287-291`. `scripts/site-headend-install.sh:528` picks a rack +by list INDEX (`[0]['system_id']`), which is meaningless across regions. Both are +pre-existing and LOGGED, NOT EXECUTED (hard rule 1). + +**No revert** -- this item is a record, not a change. + +--- + +## Item 7 -- harness manifest re-recorded (91 harnesses) + +`scripts/run-tests-all.sh` FAILED with `harness set DRIFTED from tests/HARNESS-MANIFEST`, +listing `maas-profile-assert` and `maas-region-power-key` as present-but-unpinned. That is +the manifest gate working as designed on two harnesses added this session, not a +regression. Re-recorded with `--record-manifest`; 35 -> 37 entries in the diff, 91 total. + +**Revert.** `git checkout ^ -- tests/HARNESS-MANIFEST`. + +--- + +## Item 8 -- as-executed log NOT used for this window, and why `scripts/run-logged.sh` opens an INTERACTIVE `script(1)` subshell and is unusable from a non-interactive agent session. F6 (2026-07-30) already records the harness classifier diff --git a/scripts/lib-hosts.sh b/scripts/lib-hosts.sh index a8d4ec7..3cc0dd9 100644 --- a/scripts/lib-hosts.sh +++ b/scripts/lib-hosts.sh @@ -43,6 +43,12 @@ # Mirrors the surviving juju/lxd/tailscale virsh machines' live config (read # 2026-06-26). power_id is set per host to the hostname by the caller. VIRSH_POWER_ADDRESS="qemu+ssh://logxen@10.12.64.1/system" +# The per-region forms exist only for the VR1 DCs (D-132 q1). Declared EMPTY at +# the flat/VR0 layer so a shell that never selects a VR1 DC cannot pick up a +# neighbouring DC's address -- an empty value fails loud; a stale one powers off +# the wrong rack's machines. +VIRSH_POWER_ADDRESS_FROM_OFFICE1="" +VIRSH_POWER_ADDRESS_FROM_DCREGION="" HOST_ARCH="amd64" # MAAS tag the deploy bundle places units against (constraint tags=openstack). @@ -90,6 +96,20 @@ # (D-133 flat carve -- applied + verified live, capture above). # - VIRSH_POWER_ADDRESS is the per-DC rack transit URI (ruled step-D shape, # per-machine virsh power -- D-103/D-123 amendments; SEC-012/SEC-016 keys). +# - THE POWER ADDRESS DEPENDS ON WHICH MAAS REGION DIALS IT (measured +# 2026-07-30, D-132 q1 per-DC regions). MAAS's own power ops originate from +# the REGION, so the reachable rack address differs by region: +# VIRSH_POWER_ADDRESS_FROM_OFFICE1 -- the rack's TRANSIT /30 leg. What the +# Office1 region has always used. MEASURED CLOSED from a DC-local region. +# VIRSH_POWER_ADDRESS_FROM_DCREGION -- the rack's METAL-ADMIN leg. What a +# DC-local MAAS region (D-132 q1, at utility .6) must use. MEASURED from +# vr1-dc0-maas-01: 172.31.0.2:22 CLOSED, 10.12.8.2:22 OPEN; its only +# routes are 10.12.4.0/22, 10.12.8.0/22 and a default via 10.12.4.1. +# VIRSH_POWER_ADDRESS remains an ALIAS OF THE OFFICE1 FORM so existing +# callers (reenroll-hosts.sh, the teardown runbook's virsh probes) are +# bit-for-bit unchanged. Flipping the default to the DC-region form is OWED +# once BOTH DCs have migrated -- doing it now would silently change the +# teardown path while Office1 still owns dc1's nodes. # - HOST_TAG is the region-qualified per-DC placement tag (ruled 2026-07-23). # ONE SELECTION PER SHELL: cross-DC re-selection is REFUSED (stale-value # guard, lib-net precedent); same-DC re-selection is a no-op. @@ -127,7 +147,9 @@ [vr1-dc0-storage-03]=52:54:00:b1:94:d1 [vr1-dc0-storage-04]=52:54:00:2b:ed:ab [vr1-dc0-juju-01]=52:54:00:48:e7:1e ) - VIRSH_POWER_ADDRESS="qemu+ssh://jessea123@172.31.0.2/system" + VIRSH_POWER_ADDRESS_FROM_OFFICE1="qemu+ssh://jessea123@172.31.0.2/system" + VIRSH_POWER_ADDRESS_FROM_DCREGION="qemu+ssh://jessea123@10.12.8.2/system" + VIRSH_POWER_ADDRESS="$VIRSH_POWER_ADDRESS_FROM_OFFICE1" HOST_TAG="openstack-vr1-dc0" ;; vr1-dc1) @@ -153,7 +175,12 @@ [vr1-dc1-storage-03]=52:54:01:d1:08:01 [vr1-dc1-storage-04]=52:54:01:d1:09:01 [vr1-dc1-juju-01]=52:54:00:53:12:70 ) - VIRSH_POWER_ADDRESS="qemu+ssh://jessea123@172.31.0.6/system" + VIRSH_POWER_ADDRESS_FROM_OFFICE1="qemu+ssh://jessea123@172.31.0.6/system" + # 10.12.68.2 MEASURED on the dc1 rack 2026-07-30 (`ip -4 -o addr`: virbr6 + # inet 10.12.68.2/22), corroborated by dc-rack-net.sh's dc1 LEGS block. + # NOT inferred from dc0's .2 by symmetry. + VIRSH_POWER_ADDRESS_FROM_DCREGION="qemu+ssh://jessea123@10.12.68.2/system" + VIRSH_POWER_ADDRESS="$VIRSH_POWER_ADDRESS_FROM_OFFICE1" HOST_TAG="openstack-vr1-dc1" ;; dc0|dc1|dc2) diff --git a/scripts/maas-region-power-key.sh b/scripts/maas-region-power-key.sh new file mode 100644 index 0000000..676e763 --- /dev/null +++ b/scripts/maas-region-power-key.sh @@ -0,0 +1,270 @@ +#!/usr/bin/env bash +# scripts/maas-region-power-key.sh [--commit] +# +# Install and VERIFY the per-DC MAAS->libvirt power key inside a MAAS region's +# snap. Runs ON the region host (the snap's own machine). The private key is +# read from STDIN on `install`, so it is never written to an intermediate host +# and never appears in a process argument. +# +# check READ-ONLY. Exit 0 only if the key is present +# with 0600, the ssh config block targets the +# right host with IdentitiesOnly, AND a REAL +# `virsh -c version` succeeds as the snap +# runs it (root). +# install --commit Idempotent. Key on stdin. DRY without +# --commit. +# +# WHY THIS EXISTS (2026-07-30, D-132 q1 per-DC MAAS regions). +# `runbooks/dc-dc-phase2-tofu-dc-substrate.md` prerequisite 5 carries this as +# PROSE only -- and prose is not a gate. It has to be re-run at every DC standup +# and, per the SAME prerequisite's own FRAGILITY note, after EVERY MAAS snap +# refresh, because the key and config live under per-revision +# `/var/snap/maas/current/`. A per-DC secret with a documented re-assert +# requirement and no tool is the class this repo ruled against on 2026-07-30 +# ("every per-DC secret needs a rotation tool, not just a generation recipe"). +# +# SEC-012 (dc0) / SEC-016 (dc1): each DC gets its OWN power key, never a +# cross-DC reuse. This script does not mint -- it installs the key it is given +# and REFUSES to overwrite a DIFFERENT key already in place (that would be a +# silent cross-DC swap), so rotation is deliberate rather than accidental. +# +# ASSERT ON THE ARTIFACT, NEVER THE CONFIG: `check` ends on a real virsh call. +# A config file that names the right key proves nothing about whether libvirt +# accepts it -- the 2026-07-30 Octavia PKI work paid for this lesson twice. +# +# Exit: 0 all assertions pass | 1 assertion(s) failed | 2 could not evaluate. +# ASCII + LF. +set -uo pipefail + +ACTION="${1:-}"; SITE="${2:-}"; URI="${3:-}" +COMMIT=0 +if [ "$#" -gt 3 ]; then shift 3; else set --; fi +for a in "$@"; do + case "$a" in + --commit) COMMIT=1 ;; + *) echo "FAIL: unknown option '$a'" >&2; exit 2 ;; + esac +done + +usage() { + echo "usage: maas-region-power-key.sh [--commit]" >&2 + echo " e.g. maas-region-power-key.sh check vr1-dc0 qemu+ssh://jessea123@10.12.8.2/system" >&2 + echo " maas-region-power-key.sh install vr1-dc0 qemu+ssh://jessea123@10.12.8.2/system --commit < key" >&2 + exit 2 +} +[ -n "$ACTION" ] && [ -n "$SITE" ] && [ -n "$URI" ] || usage +case "$ACTION" in check|install) ;; *) usage ;; esac + +# Site token shape: -. DERIVE the label, never type it (the +# transposition class the octavia-pki work was hardened against). +case "$SITE" in + *-*) DCLABEL="${SITE#*-}" ;; + *) echo "FAIL: site '$SITE' is not - (e.g. vr1-dc0)" >&2; exit 2 ;; +esac +[ -n "$DCLABEL" ] || { echo "FAIL: could not derive a DC label from site '$SITE'" >&2; exit 2; } + +# Parse user@host out of qemu+ssh://user@host/system. Refuse anything else -- +# a bare host would silently dial as root. +case "$URI" in + qemu+ssh://*@*/*) ;; + *) echo "FAIL: uri '$URI' is not qemu+ssh://@/" >&2; exit 2 ;; +esac +_rest="${URI#qemu+ssh://}" +USERHOST="${_rest%%/*}" +SSHUSER="${USERHOST%@*}" +RACKHOST="${USERHOST#*@}" +[ -n "$SSHUSER" ] && [ -n "$RACKHOST" ] || { echo "FAIL: could not parse user/host from '$URI'" >&2; exit 2; } + +# MAAS_SNAP_ROOT exists so the harness can exercise the derivation and parsing +# logic without a snap. In production it is never set and the default is the +# per-revision path the FRAGILITY note warns about. +SNAPBASE="${MAAS_SNAP_ROOT:-/var/snap/maas/current}" +SNAPROOT="$SNAPBASE/root/.ssh" +KEY="$SNAPROOT/id_${DCLABEL}_power" +CFG="$SNAPROOT/config" + +# Echo the DERIVED values before any privileged call, so a transposition is +# visible in the capture even when a precondition later refuses. +echo "maas-region-power-key $ACTION: site=$SITE dc=$DCLABEL rack=$RACKHOST user=$SSHUSER" +echo " key=$KEY" + +command -v sudo >/dev/null 2>&1 || { echo "REFUSE: no sudo on this host" >&2; exit 2; } +sudo -n true 2>/dev/null || { echo "REFUSE: passwordless sudo unavailable -- cannot read the snap's root-owned ssh dir" >&2; exit 2; } +[ -d "$SNAPBASE" ] || { echo "REFUSE: $SNAPBASE absent -- is the MAAS snap installed on this host?" >&2; exit 2; } + +# ASSERT WITH THE BINARY MAAS ACTUALLY USES. The region host has no `virsh` of +# its own (measured on vr1-dc0-maas-01, 2026-07-30) -- the snap ships one, and +# that is what MAAS's virsh power driver drives, against the snap's own ssh +# config. Checking with a host-installed virsh would grade a different program +# reading a different config, which is precisely the "assert the artifact, never +# the config" failure this repo keeps paying for. +# +# A SNAP HAS TWO ROOTS AND THEY ARE NOT INTERCHANGEABLE -- measured the hard way +# on the first live run, which REFUSED with "no virsh binary found": +# /var/snap//current WRITABLE data (root/.ssh lives here) +# /snap//current READ-ONLY squashfs (the binaries live here) +# The first version of this script looked for virsh under the DATA root and +# found nothing, while `find /snap/maas/current -name virsh` had already located +# it. SNAPEXEC is derived from SNAPBASE rather than typed, so the harness +# override still works and the two cannot drift apart. +# AND THE SNAP'S BINARY MUST BE RUN INSIDE THE SNAP'S RUNTIME. Invoking +# $SNAPEXEC/usr/bin/virsh directly fails -- measured on the second live run: +# "error while loading shared libraries: libvirt-lxc.so.0" +# because the snap's libraries are not on the host's loader path. `snap run +# --shell -c 'virsh ...'` executes it with the snap's environment AND its +# ssh config, which is exactly the context MAAS's own power driver uses. +SNAPEXEC="${MAAS_SNAP_EXEC:-${SNAPBASE#/var}}" +_sb="${SNAPBASE%/current}"; SNAPNAME="${_sb##*/}" +VIRSH_MODE="" +if [ -n "${VIRSH_BIN:-}" ]; then + VIRSH_MODE="direct" # harness override +elif command -v snap >/dev/null 2>&1 && [ -x "$SNAPEXEC/usr/bin/virsh" ]; then + VIRSH_MODE="snap" +elif command -v virsh >/dev/null 2>&1; then + VIRSH_MODE="host"; VIRSH_BIN="$(command -v virsh)" +fi + +# virsh_run -- dispatch to whichever mode was resolved. +virsh_run() { + case "$VIRSH_MODE" in + direct) sudo -n "$VIRSH_BIN" "$@" ;; + host) sudo -n "$VIRSH_BIN" "$@" ;; + snap) sudo -n snap run --shell "$SNAPNAME" -c "virsh $*" ;; + *) return 127 ;; + esac +} +virsh_label() { + case "$VIRSH_MODE" in + snap) echo "snap run --shell $SNAPNAME -c virsh" ;; + *) echo "$VIRSH_BIN" ;; + esac +} + +# DERIVE the public half FROM the private key, then fingerprint that. +# `ssh-keygen -lf ` reads an ADJACENT .pub when one exists and +# reports ITS fingerprint -- so a stale sibling .pub makes the comparison answer +# about the wrong key entirely. Found by this script's own harness, 2026-07-30, +# where a leftover .pub let a freshly-installed key read back as a different one. +# `-y` reads the private key and cannot be fooled by a neighbouring file. +fp_of_privkey() { # $1 = path (root-owned); prints the SHA256 fingerprint only + sudo -n ssh-keygen -y -f "$1" 2>/dev/null | ssh-keygen -lf - 2>/dev/null | awk '{print $2}' +} +fp_of_local_privkey() { # $1 = path readable by this user + ssh-keygen -y -f "$1" 2>/dev/null | ssh-keygen -lf - 2>/dev/null | awk '{print $2}' +} + +# ---------------------------------------------------------------- check ---- +do_check() { + local pass=0 fail=0 + ck() { if [ "$1" = 0 ]; then echo " [ok] $2"; pass=$((pass+1)); else echo " [FAIL] $2"; fail=$((fail+1)); fi; } + + sudo -n test -f "$KEY"; ck $? "power key present" + if ! sudo -n test -f "$KEY"; then + echo "REFUSE: no key to evaluate -- 'could not look' is not 'nothing wrong'" >&2 + echo "maas-region-power-key: $pass passed, $fail failed"; return 2 + fi + + local mode; mode="$(sudo -n stat -c %a "$KEY" 2>/dev/null)" + [ "$mode" = "600" ]; ck $? "key mode is 0600 (got ${mode:-unknown})" + + local owner; owner="$(sudo -n stat -c %U "$KEY" 2>/dev/null)" + [ "$owner" = "root" ]; ck $? "key owned by root (got ${owner:-unknown})" + + local fp; fp="$(fp_of_privkey "$KEY")" + [ -n "$fp" ]; ck $? "key parses as a valid PRIVATE key${fp:+ ($fp)}" + + sudo -n test -f "$CFG"; ck $? "ssh config present" + # Assert the config actually BINDS this rack host to this key. A config that + # merely mentions the key proves nothing -- match the Host stanza. + local blk + blk="$(sudo -n awk -v h="$RACKHOST" ' + $1=="Host" { inblk = 0; for (i=2;i<=NF;i++) if ($i==h) inblk=1 } + inblk { print } + ' "$CFG" 2>/dev/null)" + printf '%s' "$blk" | grep -q "IdentityFile[[:space:]]\+$KEY"; ck $? "config binds Host $RACKHOST -> $KEY" + printf '%s' "$blk" | grep -qi "IdentitiesOnly[[:space:]]\+yes"; ck $? "config sets IdentitiesOnly yes for $RACKHOST" + + # THE ARTIFACT ASSERTION. Everything above is configuration; only this proves + # the region can actually drive power on that rack. + if [ -z "$VIRSH_MODE" ]; then + echo "REFUSE: no usable virsh found (looked for the $SNAPNAME snap's own, and PATH)" >&2 + echo " the config assertions above cannot substitute for the artifact test" >&2 + echo "maas-region-power-key: $pass passed, $fail failed"; return 2 + fi + local vout vrc + vout="$(virsh_run -c "$URI" version 2>&1)"; vrc=$? + [ "$vrc" -eq 0 ]; ck $? "REAL virsh ($(virsh_label)) connect to $URI succeeds" + if [ "$vrc" -ne 0 ]; then + echo " virsh said: $(printf '%s' "$vout" | tr '\n' ' ' | cut -c1-200)" + fi + + # And that it can enumerate domains -- a connect that lists nothing would let a + # power op no-op silently. + local n + n="$(virsh_run -c "$URI" list --all --name 2>/dev/null | sed '/^$/d' | wc -l)" + [ "${n:-0}" -gt 0 ]; ck $? "virsh enumerates domains on the rack (got ${n:-0})" + + echo "maas-region-power-key: $pass passed, $fail failed" + [ "$fail" -eq 0 ] || return 1 + return 0 +} + +# -------------------------------------------------------------- install ---- +do_install() { + if [ -t 0 ]; then + echo "FAIL: install reads the private key from STDIN -- redirect one in" >&2 + exit 2 + fi + local tmp; tmp="$(mktemp)"; chmod 600 "$tmp" + # shellcheck disable=SC2064 + trap "shred -u '$tmp' 2>/dev/null || rm -f '$tmp'" EXIT + cat > "$tmp" + [ -s "$tmp" ] || { echo "FAIL: empty key on stdin" >&2; exit 2; } + local newfp; newfp="$(fp_of_local_privkey "$tmp")" + [ -n "$newfp" ] || { echo "FAIL: stdin is not a readable private key" >&2; exit 2; } + + # REFUSE a silent cross-DC swap: if a DIFFERENT key already sits here, stop. + if sudo -n test -f "$KEY"; then + local curfp; curfp="$(fp_of_privkey "$KEY")" + if [ "$curfp" = "$newfp" ]; then + echo "[idempotent] $KEY already holds this key ($newfp)" + else + echo "REFUSE: $KEY already holds a DIFFERENT key" >&2 + echo " present: ${curfp:-unparseable}" >&2 + echo " offered: $newfp" >&2 + echo " Overwriting would silently re-point this region's power path." >&2 + echo " Move the existing key aside deliberately if rotation is intended." >&2 + exit 1 + fi + fi + + if [ "$COMMIT" -eq 0 ]; then + echo "DRY RUN (no --commit). Would:" + echo " install -d -m 0700 -o root -g root $SNAPROOT" + echo " install -m 0600 -o root -g root $KEY # fp $newfp" + echo " append Host $RACKHOST block to $CFG -> IdentityFile $KEY" + return 0 + fi + + sudo -n install -d -m 0700 -o root -g root "$SNAPROOT" || { echo "FAIL: could not create $SNAPROOT" >&2; exit 1; } + sudo -n install -m 0600 -o root -g root "$tmp" "$KEY" || { echo "FAIL: could not install $KEY" >&2; exit 1; } + + sudo -n touch "$CFG" && sudo -n chmod 0644 "$CFG" && sudo -n chown root:root "$CFG" + # Idempotent: only append if this Host has no stanza yet. + if sudo -n awk -v h="$RACKHOST" '$1=="Host"{for(i=2;i<=NF;i++) if($i==h) found=1} END{exit !found}' "$CFG" 2>/dev/null; then + echo "[idempotent] $CFG already has a Host $RACKHOST stanza -- left as-is" + else + printf '\nHost %s\n IdentityFile %s\n IdentitiesOnly yes\n StrictHostKeyChecking accept-new\n' \ + "$RACKHOST" "$KEY" | sudo -n tee -a "$CFG" >/dev/null + echo "appended Host $RACKHOST stanza to $CFG" + fi + + echo "installed. verifying:" + do_check + return $? +} + +case "$ACTION" in + check) do_check; exit $? ;; + install) do_install; exit $? ;; +esac diff --git a/tests/HARNESS-MANIFEST b/tests/HARNESS-MANIFEST index 63da221..2f82784 100644 --- a/tests/HARNESS-MANIFEST +++ b/tests/HARNESS-MANIFEST @@ -33,6 +33,8 @@ ledger-scan lib-validate maas-node-power +maas-profile-assert +maas-region-power-key maas-role-tags netem-link netem-sudoers diff --git a/tests/dc-selector/run-tests.sh b/tests/dc-selector/run-tests.sh index 27315ad..bf413e3 100644 --- a/tests/dc-selector/run-tests.sh +++ b/tests/dc-selector/run-tests.sh @@ -203,6 +203,49 @@ chk "hosts vr1-dc0 tag openstack-vr1-dc0 (ruled 2026-07-23)" "$H0_TAG" "openstack-vr1-dc0" H1_PWR="$(lib_hosts_select_dc vr1-dc1 >/dev/null 2>&1; echo "$VIRSH_POWER_ADDRESS")" chk "hosts vr1-dc1 power addr rack transit .6" "$H1_PWR" "qemu+ssh://jessea123@172.31.0.6/system" + +# --- per-REGION power address (D-132 q1 per-DC MAAS regions, 2026-07-30) --- +# MAAS power ops originate from the REGION, so the reachable rack address differs +# by which region dials. MEASURED from vr1-dc0-maas-01 (10.12.8.6): the rack's +# transit leg 172.31.0.2:22 is CLOSED, its metal-admin leg 10.12.8.2:22 is OPEN. +# dc1's 10.12.68.2 was measured on the dc1 rack, NOT inferred from dc0's .2. +H0_PO="$(lib_hosts_select_dc vr1-dc0 >/dev/null 2>&1; echo "$VIRSH_POWER_ADDRESS_FROM_OFFICE1")" +chk "hosts vr1-dc0 office1-region power addr = transit .2" "$H0_PO" "qemu+ssh://jessea123@172.31.0.2/system" +H0_PD="$(lib_hosts_select_dc vr1-dc0 >/dev/null 2>&1; echo "$VIRSH_POWER_ADDRESS_FROM_DCREGION")" +chk "hosts vr1-dc0 dc-region power addr = metal-admin 10.12.8.2" "$H0_PD" "qemu+ssh://jessea123@10.12.8.2/system" +H1_PO="$(lib_hosts_select_dc vr1-dc1 >/dev/null 2>&1; echo "$VIRSH_POWER_ADDRESS_FROM_OFFICE1")" +chk "hosts vr1-dc1 office1-region power addr = transit .6" "$H1_PO" "qemu+ssh://jessea123@172.31.0.6/system" +H1_PD="$(lib_hosts_select_dc vr1-dc1 >/dev/null 2>&1; echo "$VIRSH_POWER_ADDRESS_FROM_DCREGION")" +chk "hosts vr1-dc1 dc-region power addr = metal-admin 10.12.68.2" "$H1_PD" "qemu+ssh://jessea123@10.12.68.2/system" + +# BACKWARD COMPATIBILITY IS THE LOAD-BEARING HALF: VIRSH_POWER_ADDRESS must still +# be the Office1/transit form, byte-for-byte, or reenroll-hosts.sh and the +# teardown runbook's virsh probes silently change target. +H0_ALIAS="$(lib_hosts_select_dc vr1-dc0 >/dev/null 2>&1; [ "$VIRSH_POWER_ADDRESS" = "$VIRSH_POWER_ADDRESS_FROM_OFFICE1" ] && echo same || echo DIVERGED)" +chk "hosts vr1-dc0 VIRSH_POWER_ADDRESS still aliases the OFFICE1 form" "$H0_ALIAS" "same" +H1_ALIAS="$(lib_hosts_select_dc vr1-dc1 >/dev/null 2>&1; [ "$VIRSH_POWER_ADDRESS" = "$VIRSH_POWER_ADDRESS_FROM_OFFICE1" ] && echo same || echo DIVERGED)" +chk "hosts vr1-dc1 VIRSH_POWER_ADDRESS still aliases the OFFICE1 form" "$H1_ALIAS" "same" + +# THE TWO FORMS MUST DIFFER. If a future edit collapses them the migration would +# dial an unreachable address and every power op would fail -- this is the +# assertion that catches a copy-paste of the transit URI into the DCREGION slot. +H0_DIFF="$(lib_hosts_select_dc vr1-dc0 >/dev/null 2>&1; [ "$VIRSH_POWER_ADDRESS_FROM_OFFICE1" != "$VIRSH_POWER_ADDRESS_FROM_DCREGION" ] && echo differ || echo COLLAPSED)" +chk "hosts vr1-dc0 the two region forms are DISTINCT" "$H0_DIFF" "differ" +H1_DIFF="$(lib_hosts_select_dc vr1-dc1 >/dev/null 2>&1; [ "$VIRSH_POWER_ADDRESS_FROM_OFFICE1" != "$VIRSH_POWER_ADDRESS_FROM_DCREGION" ] && echo differ || echo COLLAPSED)" +chk "hosts vr1-dc1 the two region forms are DISTINCT" "$H1_DIFF" "differ" + +# CROSS-DC CONTAMINATION: dc0's DCREGION address must never carry dc1's octet and +# vice versa -- the D-117 off-by-one class, applied to the value that powers +# machines off. +case "$H0_PD" in *10.12.68.*) no "hosts vr1-dc0 dc-region addr leaked a dc1 plane octet" ;; *) ok "hosts vr1-dc0 dc-region addr carries no dc1 octet" ;; esac +case "$H1_PD" in *10.12.8.*) no "hosts vr1-dc1 dc-region addr leaked a dc0 plane octet" ;; *) ok "hosts vr1-dc1 dc-region addr carries no dc0 octet" ;; esac + +# VR0 must NOT inherit either VR1 form (empty at the flat layer -- an empty value +# fails loud, a stale one powers off the wrong rack). +V0_PO="$(lib_hosts_select_dc vr0-dc0 >/dev/null 2>&1; echo "${VIRSH_POWER_ADDRESS_FROM_OFFICE1:-EMPTY}")" +chk "hosts vr0-dc0 does not inherit a VR1 office1 power addr" "$V0_PO" "EMPTY" +V0_PD="$(lib_hosts_select_dc vr0-dc0 >/dev/null 2>&1; echo "${VIRSH_POWER_ADDRESS_FROM_DCREGION:-EMPTY}")" +chk "hosts vr0-dc0 does not inherit a VR1 dc-region power addr" "$V0_PD" "EMPTY" H0_N="$(lib_hosts_select_dc vr1-dc0 >/dev/null 2>&1; echo "${#HOSTS[@]}")" chk "hosts vr1-dc0 fleet count 10 (D-121 Option C 9 role + D-104 juju-01)" "$H0_N" "10" diff --git a/tests/maas-region-power-key/run-tests.sh b/tests/maas-region-power-key/run-tests.sh new file mode 100644 index 0000000..bfa54a4 --- /dev/null +++ b/tests/maas-region-power-key/run-tests.sh @@ -0,0 +1,223 @@ +#!/usr/bin/env bash +# tests/maas-region-power-key/run-tests.sh -- unit tests for +# scripts/maas-region-power-key.sh (2026-07-30). +# +# The script's real work happens as root inside a MAAS snap, so these tests +# exercise the parts that decide CORRECTNESS BEFORE any privileged call: +# argument validation, the - derivation, qemu+ssh URI parsing, and +# the refusal directions. A stubbed `sudo`/`virsh`/`ssh-keygen` on PATH lets the +# check/install paths run without a snap. +# +# THE DERIVATION AND THE URI PARSE ARE THE POINT. The key filename is +# id__power and the ssh stanza keys on the rack host -- get either wrong and +# the region silently dials the WRONG RACK, which powers off another DC's +# machines. Both are derived from arguments rather than typed, and a +# transposition is caught here rather than by review. +set -uo pipefail +SD="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SCRIPT="$SD/../../scripts/maas-region-power-key.sh" +P=0; F=0 +ok(){ echo "PASS: $1"; P=$((P+1)); } +no(){ echo "FAIL: $1"; F=$((F+1)); } +chk(){ [ "$2" = "$3" ] && ok "$1" || no "$1 (got '$2' want '$3')"; } + +TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT +mkdir -p "$TMP/bin" "$TMP/snap" + +# --- stubs ----------------------------------------------------------------- +# sudo: transparently runs the command (STUB_SUDO=deny makes `sudo -n true` fail). +# It EMULATES privilege rather than granting it: `-o root -g root` is stripped and +# `chown root:root` succeeds as a no-op, because an unprivileged test process +# cannot really chown to root. Everything else runs for real, so the install path +# is exercised end to end (dirs created, key written, stanza appended). +cat > "$TMP/bin/sudo" <<'STUB' +#!/usr/bin/env bash +[ "${STUB_SUDO:-allow}" = "deny" ] && exit 1 +while [ "${1:-}" = "-n" ]; do shift; done +if [ "${1:-}" = "chown" ]; then exit 0; fi +if [ "${1:-}" = "install" ]; then + args=(); shift + while [ "$#" -gt 0 ]; do + case "$1" in + -o|-g) shift 2; continue ;; + *) args+=("$1"); shift ;; + esac + done + exec install "${args[@]}" +fi +exec "$@" +STUB +# virsh: STUB_VIRSH controls connect success and domain count +cat > "$TMP/bin/virsh" <<'STUB' +#!/usr/bin/env bash +case "${STUB_VIRSH:-ok}" in + refuse) echo "error: failed to connect" >&2; exit 1 ;; + empty) case " $* " in *" list "*) exit 0 ;; *) echo "compiled 10.0.0"; exit 0 ;; esac ;; + *) case " $* " in *" list "*) printf 'dom-a\ndom-b\n'; exit 0 ;; *) echo "compiled 10.0.0"; exit 0 ;; esac ;; +esac +STUB +chmod +x "$TMP/bin/sudo" "$TMP/bin/virsh" +export PATH="$TMP/bin:$PATH" +# Point the script at the stub explicitly -- in production it resolves the SNAP's +# virsh, which no test host has. +export VIRSH_BIN="$TMP/bin/virsh" + +run(){ OUT="$(bash "$SCRIPT" "$@" 2>&1)"; RC=$?; } + +URI="qemu+ssh://jessea123@10.12.8.2/system" + +# --- usage / argument validation ------------------------------------------- +run; chk "no args -> 2" "$RC" 2 +run check; chk "action only -> 2" "$RC" 2 +run check vr1-dc0; chk "missing uri -> 2" "$RC" 2 +run bogus vr1-dc0 "$URI"; chk "unknown action -> 2" "$RC" 2 +run check vr1-dc0 "$URI" --nope; chk "unknown option -> 2" "$RC" 2 + +# --- site token derivation -------------------------------------------------- +run check dc0 "$URI" +chk "site without a region prefix -> 2" "$RC" 2 +case "$OUT" in *"not -"*) ok "bad site names the shape" ;; *) no "bad site names the shape (got '$OUT')" ;; esac + +# --- URI parsing ------------------------------------------------------------ +run check vr1-dc0 "qemu+ssh://10.12.8.2/system" +chk "uri without a user -> 2 (never dial as root by default)" "$RC" 2 +run check vr1-dc0 "ssh://jessea123@10.12.8.2/system" +chk "non-qemu+ssh scheme -> 2" "$RC" 2 +run check vr1-dc0 "10.12.8.2" +chk "bare host -> 2" "$RC" 2 + +# --- sudo unavailable ------------------------------------------------------- +OUT="$(STUB_SUDO=deny bash "$SCRIPT" check vr1-dc0 "$URI" 2>&1)"; RC=$? +chk "no passwordless sudo REFUSES -> 2" "$RC" 2 + +echo +echo "--- check path against a synthetic snap tree ---" +# MAAS_SNAP_ROOT points the script at a fake tree so derivation, URI parsing and +# every assertion can be exercised without a real snap. +export MAAS_SNAP_ROOT="$TMP/snap" +mkdir -p "$TMP/snap/root/.ssh" + +run check vr1-dc0 "$URI" +case "$OUT" in *"id_dc0_power"*) ok "derives key name id_dc0_power from site vr1-dc0" ;; *) no "derives key name id_dc0_power (got '$OUT')" ;; esac +run check vr1-dc1 "qemu+ssh://jessea123@10.12.68.2/system" +case "$OUT" in *"id_dc1_power"*) ok "derives key name id_dc1_power from site vr1-dc1" ;; *) no "derives key name id_dc1_power (got '$OUT')" ;; esac + +# TRANSPOSITION GUARD: a dc1 site must never produce dc0's key name, and the +# parsed rack must never be the other DC's. This is the assertion that catches a +# copy-paste between the two per-DC invocations. +run check vr1-dc1 "qemu+ssh://jessea123@10.12.68.2/system" +case "$OUT" in *"id_dc0_power"*) no "dc1 invocation leaked dc0's key name" ;; *) ok "dc1 invocation carries no dc0 key name" ;; esac +case "$OUT" in *"rack=10.12.68.2"*) ok "dc1 invocation parses rack 10.12.68.2" ;; *) no "dc1 invocation parses rack 10.12.68.2 (got '$OUT')" ;; esac +run check vr1-dc0 "$URI" +case "$OUT" in *"rack=10.12.8.2"*) ok "dc0 invocation parses rack 10.12.8.2" ;; *) no "dc0 invocation parses rack 10.12.8.2 (got '$OUT')" ;; esac +case "$OUT" in *"user=jessea123"*) ok "parses the ssh user out of the uri" ;; *) no "parses the ssh user out of the uri (got '$OUT')" ;; esac + +# --- check assertions against the synthetic tree ---------------------------- +echo +echo "--- check assertions ---" +KEYPATH="$TMP/snap/root/.ssh/id_dc0_power" +CFGPATH="$TMP/snap/root/.ssh/config" + +# no key at all must REFUSE (2), never report a clean tree +rm -f "$KEYPATH" "$CFGPATH" +run check vr1-dc0 "$URI" +chk "absent key REFUSES -> 2" "$RC" 2 +case "$OUT" in *"could not look"*) ok "absent key says 'could not look' is not 'nothing wrong'" ;; *) no "absent key explains the refusal (got '$OUT')" ;; esac + +ssh-keygen -q -t ed25519 -N '' -C 'dc0-power-test' -f "$KEYPATH" 2>/dev/null +if [ ! -f "$KEYPATH" ]; then + no "could not generate a test keypair (ssh-keygen missing?)" +else + # key present, config absent -> the config assertions must FAIL, not pass + rm -f "$CFGPATH" + run check vr1-dc0 "$URI" + case "$OUT" in *"[FAIL] ssh config present"*) ok "absent ssh config FAILS" ;; *) no "absent ssh config FAILS (got '$OUT')" ;; esac + [ "$RC" = 1 ]; chk "key-without-config exits 1" "$?" 0 + + # a config that mentions the key but for the WRONG Host must not satisfy the + # binding assertion -- this is the cross-DC swap the stanza check exists for. + printf 'Host 10.12.68.2\n IdentityFile %s\n IdentitiesOnly yes\n' "$KEYPATH" > "$CFGPATH" + run check vr1-dc0 "$URI" + case "$OUT" in *"[FAIL] config binds Host 10.12.8.2"*) ok "config bound to the WRONG rack FAILS" ;; *) no "config bound to the WRONG rack FAILS (got '$OUT')" ;; esac + + # correct binding, but IdentitiesOnly missing + printf 'Host 10.12.8.2\n IdentityFile %s\n' "$KEYPATH" > "$CFGPATH" + run check vr1-dc0 "$URI" + case "$OUT" in *"[FAIL] config sets IdentitiesOnly"*) ok "missing IdentitiesOnly FAILS" ;; *) no "missing IdentitiesOnly FAILS (got '$OUT')" ;; esac + + # fully correct config + reachable virsh -> pass + printf 'Host 10.12.8.2\n IdentityFile %s\n IdentitiesOnly yes\n' "$KEYPATH" > "$CFGPATH" + chmod 600 "$KEYPATH" + OUT="$(STUB_VIRSH=ok bash "$SCRIPT" check vr1-dc0 "$URI" 2>&1)"; RC=$? + case "$OUT" in *"[ok] config binds Host 10.12.8.2"*) ok "correct config binding passes" ;; *) no "correct config binding passes (got '$OUT')" ;; esac + case "$OUT" in *"[ok] REAL virsh ("*") connect"*) ok "reachable virsh passes" ;; *) no "reachable virsh passes (got '$OUT')" ;; esac + + # THE ARTIFACT ASSERTIONS. A perfect config with an unreachable or empty + # libvirt must FAIL -- otherwise the gate green-lights a region that cannot + # actually power anything, which is the whole point of checking. + OUT="$(STUB_VIRSH=refuse bash "$SCRIPT" check vr1-dc0 "$URI" 2>&1)"; RC=$? + case "$OUT" in *"[FAIL] REAL virsh ("*") connect"*) ok "unreachable virsh FAILS despite a perfect config" ;; *) no "unreachable virsh FAILS (got '$OUT')" ;; esac + [ "$RC" = 1 ]; chk "unreachable virsh exits 1" "$?" 0 + + OUT="$(STUB_VIRSH=empty bash "$SCRIPT" check vr1-dc0 "$URI" 2>&1)"; RC=$? + case "$OUT" in *"[FAIL] virsh enumerates domains"*) ok "virsh listing ZERO domains FAILS (a power op would no-op)" ;; *) no "virsh listing zero domains FAILS (got '$OUT')" ;; esac + [ "$RC" = 1 ]; chk "zero-domain virsh exits 1" "$?" 0 +fi + +# --- install refusals ------------------------------------------------------- +echo +echo "--- install path ---" +run install vr1-dc0 "$URI" < /dev/null +chk "empty stdin key -> 2" "$RC" 2 +printf 'this is not a key\n' > "$TMP/notakey" +run install vr1-dc0 "$URI" < "$TMP/notakey" +chk "unparseable stdin key -> 2" "$RC" 2 +case "$OUT" in *"not a readable private key"*) ok "bad key names the reason" ;; *) no "bad key names the reason (got '$OUT')" ;; esac + +ssh-keygen -q -t ed25519 -N '' -C 'test-k1' -f "$TMP/k1" 2>/dev/null +if [ ! -f "$TMP/k1" ]; then + no "could not generate a test keypair for install (ssh-keygen missing?)" +else + # DRY RUN must not write and must not claim to have installed. + # Remove the sibling .pub too: `ssh-keygen -lf ` prefers an adjacent + # .pub, which is exactly the trap fp_of_privkey was hardened against. + rm -f "$KEYPATH" "$KEYPATH.pub" "$CFGPATH" + run install vr1-dc0 "$URI" < "$TMP/k1" + case "$OUT" in *"DRY RUN"*) ok "install without --commit is a DRY RUN" ;; *) no "install without --commit is a DRY RUN (got '$OUT')" ;; esac + [ -f "$KEYPATH" ] && no "DRY RUN wrote the key anyway" || ok "DRY RUN wrote nothing" + + # --commit installs, then self-verifies. + # NOTE: overall exit 0 is UNREACHABLE in this synthetic tree -- an unprivileged + # test process cannot make the file root-owned, so the "key owned by root" + # assertion legitimately fails here. Asserting that specific line still passes + # in production is what the LIVE capture is for; here we assert the lines the + # harness can actually reach, and assert the root check FIRES rather than + # papering over it. + OUT="$(STUB_VIRSH=ok bash "$SCRIPT" install vr1-dc0 "$URI" --commit < "$TMP/k1" 2>&1)"; RC=$? + [ -f "$KEYPATH" ] && ok "--commit installed the key" || no "--commit installed the key" + case "$OUT" in *"appended Host 10.12.8.2"*) ok "--commit appended the Host stanza" ;; *) no "--commit appended the Host stanza (got '$OUT')" ;; esac + case "$OUT" in *"[ok] key mode is 0600"*) ok "--commit self-verify sees mode 0600" ;; *) no "--commit self-verify sees mode 0600 (got '$OUT')" ;; esac + case "$OUT" in *"[ok] config binds Host 10.12.8.2"*) ok "--commit self-verify sees the binding" ;; *) no "--commit self-verify sees the binding" ;; esac + case "$OUT" in *"[FAIL] key owned by root"*) ok "root-ownership assertion FIRES when not root-owned" ;; *) no "root-ownership assertion fires when not root-owned (got '$OUT')" ;; esac + + # re-running with the SAME key is idempotent, not a second stanza + OUT="$(STUB_VIRSH=ok bash "$SCRIPT" install vr1-dc0 "$URI" --commit < "$TMP/k1" 2>&1)"; RC=$? + case "$OUT" in *"[idempotent]"*) ok "re-install of the same key is idempotent" ;; *) no "re-install of the same key is idempotent (got '$OUT')" ;; esac + N_STANZA="$(grep -c '^Host 10.12.8.2$' "$CFGPATH")" + chk "re-install did not duplicate the Host stanza" "$N_STANZA" "1" + + # A DIFFERENT key over an existing one must REFUSE -- a silent overwrite here + # re-points a live region's power path at another DC's rack. + ssh-keygen -q -t ed25519 -N '' -C 'test-k2' -f "$TMP/k2" 2>/dev/null + OUT="$(STUB_VIRSH=ok bash "$SCRIPT" install vr1-dc0 "$URI" --commit < "$TMP/k2" 2>&1)"; RC=$? + chk "installing a DIFFERENT key over an existing one REFUSES -> 1" "$RC" 1 + case "$OUT" in *"already holds a DIFFERENT key"*) ok "the refusal names the cross-DC swap risk" ;; *) no "the refusal names the swap risk (got '$OUT')" ;; esac + # and it must NOT have overwritten + FP_NOW="$(ssh-keygen -lf "$KEYPATH" 2>/dev/null | awk '{print $2}')" + FP_K1="$(ssh-keygen -lf "$TMP/k1" 2>/dev/null | awk '{print $2}')" + chk "the refused install left the original key in place" "$FP_NOW" "$FP_K1" +fi + +echo +echo "maas-region-power-key: $P passed, $F failed" +[ "$F" -eq 0 ] || exit 1 diff --git a/tests/node-vm/run-tests.sh b/tests/node-vm/run-tests.sh index b6c7b3d..11559f8 100755 --- a/tests/node-vm/run-tests.sh +++ b/tests/node-vm/run-tests.sh @@ -58,12 +58,29 @@ # after its first apply (it is ENLISTED, which is the boundary past which unpinned MACs # can be regenerated and strand the node). So the ruled shape is 10 nodes x 6 planes. # Kept EXACT rather than relaxed to >=: an exact count is the whole value of these two. - [ "$MACS_LISTS" -eq 10 ] \ - && ok "T8 inner root: 10 per-node macs lists (9 role + juju controller)" \ - || no "T8 inner root: 10 per-node macs lists (found $MACS_LISTS)" - [ "$MAC_LITERALS" -eq 60 ] \ - && ok "T9 inner root: 60 pinned MAC literals (10 nodes x 6 planes)" \ - || no "T9 inner root: 60 pinned MAC literals (found $MAC_LITERALS)" + # + # RE-POINTED AGAIN 2026-07-30: now 11 / 66. D-132 q1 (RULED) puts a MAAS REGION in each + # DC, and its placement sub-ruling puts that region in its OWN VM at utility .6 -- + # -maas-01, authored into both substrate roots and APPLIED at dc0, with its six MACs + # pinned from measurement the same day. Same reasoning as the juju controller: once + # enlisted, unpinned MACs can be regenerated and strand the node. + # + # THIS HARNESS WENT RED ON 2026-07-30 AND THE SESSION THAT CAUSED IT DID NOT RE-RUN IT. + # Commits 086c827 (author both region VMs) and 447315f (pin dc0's MACs) moved the counts + # to 11/66 while the assertions still read 10/60; the failure was found by the NEXT + # session's gauntlet. Recorded here rather than silently corrected -- the lesson is that + # an exact-count assertion is only as good as the gauntlet run that follows the commit. + # + # ASYMMETRY IS EXPECTED RIGHT NOW, and is NOT a defect: dc1's root also carries 11 macs + # lists but only 60 literals, because `vr1-dc1-maas-01` is authored with `macs = []` and + # has NOT been applied -- there is nothing measured to pin yet. When dc1's region VM is + # applied and pinned, dc1 reaches 66 too. This harness reads the dc0 root only. + [ "$MACS_LISTS" -eq 11 ] \ + && ok "T8 inner root: 11 per-node macs lists (9 role + juju controller + MAAS region)" \ + || no "T8 inner root: 11 per-node macs lists (found $MACS_LISTS)" + [ "$MAC_LITERALS" -eq 66 ] \ + && ok "T9 inner root: 66 pinned MAC literals (11 nodes x 6 planes)" \ + || no "T9 inner root: 66 pinned MAC literals (found $MAC_LITERALS)" DUPES="$(grep -oE '"([0-9a-f]{2}:){5}[0-9a-f]{2}"' "$INNER" | sort | uniq -d | wc -l)" [ "$DUPES" -eq 0 ] \ && ok "T10 inner root: no duplicate MACs" \