diff --git a/docs/maas-as-built-reference.md b/docs/maas-as-built-reference.md new file mode 100644 index 0000000..663c484 --- /dev/null +++ b/docs/maas-as-built-reference.md @@ -0,0 +1,169 @@ +# MAAS As-Built Reference (VR0 / Baldurkeep) + +**Purpose:** Authoritative, captured snapshot of the MAAS substrate and the four +OpenStack KVM hosts, so a rebuild (or the next DC-DC region) can REPLAY the +host enrollment + interface carve instead of re-deriving it from live state. + +**Status:** Captured 2026-06-26 from live MAAS during the host re-enrollment +recovery. ASCII + LF. Append-only; correct in-place only for measured drift. + +**Trust order:** LIVE MAAS > this doc > committed bundle. Every per-host value +here was measured live this session. IDs (subnet/space/fabric/interface) DRIFT +across cutovers and re-enrollments -- **resolve by CIDR / hostname, never by a +stored ID.** (PATTERN-1, lib-net.sh.) + +--- + +## 1. Six network planes (post D-052 / D-053) + +Resolve by CIDR. Subnet/space IDs intentionally omitted -- they drift (the D-052 +cutover moved metal-internal off its old id; re-enrollment re-mints fabrics). + +| space | CIDR | VLAN / fabric | gateway | role | +|---|---|---|---|---| +| provider-public | 10.12.4.0/22 | untagged / 1_provider | 10.12.4.1 | public API VIPs + FIP/ext_net | +| metal-admin | 10.12.8.0/22 | untagged / 2_metal (PXE/DHCP) | 10.12.8.1 | operator/MAAS/PXE/admin API | +| metal-internal | 10.12.12.0/22 | **tagged VID 103 / 2_metal** | none | ALL OpenStack east-west (db/amqp/rpc/internal API/OVN-DB) | +| data-tenant | 10.12.16.0/22 | untagged / 4_data | none | OVN geneve overlay | +| storage | 10.12.32.0/22 | untagged / 8_storage | none | Ceph public | +| replication | 10.12.36.0/22 | untagged / 9_replication | none | Ceph cluster | + +Notes: +- metal-internal rides the **same L2 (2_metal)** as metal-admin, tagged VID 103, + carried on host bridge `br-internal` (over `br-metal.103`). It is the ONLY + tagged plane and the ONLY container-services plane that needs a bridge. +- Fabric cosmetic names (4_data / 8_storage / 9_replication) are reshuffled vs + the libvirt network names and are irrelevant: Juju binds by SPACE NAME + CIDR + + VLAN, all of which are correct. +- Spaceless / not bound: `f_oob` 10.12.60.0/22; LXD `fabric-4` + 10.37.195.0/24 + ULA fd42:8019:9206:57de::/64. +- Gateways: only provider-public and metal-admin route. A gateway on any other + plane is the D-052 "spurious-gw" defect class -- clear to none. +- DNS resolver for the internal planes: 10.12.8.1. +- VIP reserves: provider-public/metal-admin/metal-internal each 10.12.x.2-.100. + FIP pool 10.12.5.0-.7.254. PXE DHCP 10.12.9.0-.11.254. mgmt reserves + 10.12.4.101-.110 + 10.12.8.101-.110. + +--- + +## 2. The four KVM hosts -- identity (hostname-keyed; system_id DRIFTS) + +| hostname | libvirt domain / power_id | host octet | boot NIC (2_metal) MAC | +|---|---|---|---| +| openstack0 | openstack0 | .40 | 52:54:00:4f:1c:0b | +| openstack1 | openstack1 | .41 | 52:54:00:83:25:1f | +| openstack2 | openstack2 | .42 | 52:54:00:23:bd:72 | +| openstack3 | openstack3 | .43 | 52:54:00:b2:7b:30 | + +system_ids are minted fresh on every (re-)enrollment -- do NOT hardcode them. +Resolve at runtime by hostname (scripts/lib-hosts.sh `host_sysid`). The dead +pre-2026-06-26 ids were 4na83t/qdbqd6/h8frng/tmsafc. + +### Full per-host NIC inventory (libvirt domain XML; fixed MACs) + +libvirt source-network -> MAAS plane (the libvirt net NAMES are pre-cutover; +the CIDRs were re-IP'd onto the same L2 segments, so e.g. `3_data` now carries +data-tenant 10.12.16.0/22): + +| libvirt net | host NIC | plane | bridge/type | +|---|---|---|---| +| 1_provider | enp1s0 | provider-public | br-ex (OVS) | +| 2_metal (boot) | enp7s0 | metal-admin + metal-internal(VID103) | br-metal (OVS), br-metal.103 (vlan), br-internal (std bridge) | +| 3_data | enp8s0 | data-tenant | raw NIC | +| 4_storage | enp9s0 | storage | raw NIC | +| 5_replication | enp10s0 | replication | raw NIC | +| 8_lbaas | enp11s0 | idle (undefined; ex-lbaas) | raw NIC, no link | + +MAC inventory (host: provider / metal-boot / data / storage / replication / lbaas): +- openstack0: 3d:fd:54 / 4f:1c:0b / 07:41:0a / d0:ed:e0 / 8f:ba:61 / d9:af:46 +- openstack1: 9d:63:77 / 83:25:1f / 4e:71:6c / 42:50:8b / 86:78:ab / c6:56:12 +- openstack2: 89:7f:ce / 23:bd:72 / 24:70:08 / b8:5d:a3 / 28:bc:8c / b5:1e:61 +- openstack3: 99:fc:c2 / b2:7b:30 / c7:94:e9 / 41:cd:6b / bc:98:b0 / 6f:f5:ca + (all prefixed 52:54:00:) + +--- + +## 3. virsh power (non-secret) + enrollment + +- power_type: virsh +- power_address: `qemu+ssh://logxen@10.12.64.1/system` (MAAS -> libvirt over + SSH on the OOB host address; mirrors the surviving juju/lxd/tailscale machines) +- power_id: the libvirt domain name == the hostname +- power_pass: **read interactively; never stored.** The libvirt SSH password was + exposed in plaintext on 2026-06-26 (`maas machine power-parameters` echoes it) + -- **ROTATE that credential after the rebuild.** Never use `power-parameters` + for templating; read `power_type` and reconstruct the address pattern instead. + +Enrollment behaviour: `machines create` AUTO-COMMISSIONS (New -> Commissioning -> +Ready) by PXE off the 2_metal boot NIC. The libvirt domains must already exist +(this re-creates MAAS objects, not VMs). On commission, MAAS auto-discovers all +six raw NICs; the five non-boot NICs land on transient auto-fabrics +(fabric-NN, "Unconfigured") -- normal; the interface carve re-homes them by CIDR. + +Procedure: `scripts/reenroll-hosts.sh` (gated, idempotent, discover-assert-pin). +Constants: `scripts/lib-hosts.sh`. + +**Post-commission, before deploy:** re-apply the MAAS tag `openstack` to all four +(the bundle places units via constraint `tags=openstack`; the tag applies to no +machines after re-enrollment). + +--- + +## 4. Per-host interface carve target (Strategy-B reconstruction) + +After commission, MAAS has only the raw NICs. Rebuild this tree (octet N = +.40/.41/.42/.43 by host index). Build order is forced by parentage. + +| MAAS interface | type | parent | VLAN | space | static | +|---|---|---|---|---|---| +| enp1s0 | physical | - | untagged | provider-public | (carries br-ex) | +| br-ex | bridge **OVS** | enp1s0 | untagged | provider-public | 10.12.4.N | +| enp7s0 | physical | - | untagged | metal-admin | (carries br-metal) | +| br-metal | bridge **OVS** | enp7s0 | untagged | metal-admin | 10.12.8.N | +| br-metal.103 | vlan | br-metal | 103 | metal-internal | (carries br-internal) | +| br-internal | bridge **standard** | br-metal.103 | 103 | metal-internal | 10.12.12.N | +| enp8s0 | physical | - | untagged | data-tenant | 10.12.16.N | +| enp9s0 | physical | - | untagged | storage | 10.12.32.N | +| enp10s0 | physical | - | untagged | replication | 10.12.36.N | +| enp11s0 | physical | - | untagged | undefined (idle) | (none) | + +Build order: physicals (auto-discovered) -> br-ex / br-metal (OVS) -> +br-metal.103 (vlan) -> br-internal (std bridge) -> statics on +br-ex/br-metal/br-internal/enp8/enp9/enp10. Link by CIDR (re-homes the NIC onto +the correct fabric/space). + +CAVEAT: confirm each `bridge_type` (OVS vs standard) verbatim from a captured +machine-release JSON before carving -- it changes how MAAS renders netplan and +is the one value not to take from memory. + +Deployed-host bridge facts (for reference; created by Juju/LXD at deploy, NOT by +MAAS): storage/replication appear as Linux bridges br-enp9s0/br-enp10s0; +br-ex/br-metal/br-internal are OVS; data-tenant runs on raw enp8s0. + +--- + +## 5. MAAS substrate facts (stable) + +- MAAS 3.7.2; controller `maas.maas`, Region+Rack, Non-HA(3 VLANs). +- Pool `default`; zone `default`. Images synced (amd64): 24.04, 22.04, 20.04. + Hosts deploy on 22.04 (Jammy). +- Surviving (untouched) machines: capi-mgmt (lxd pod, Ready), juju/lxd/tailscale + (virsh, owner logxen). The LXD pod composes only capi-mgmt; the four OpenStack + hosts are virsh-individual (NOT pod-composed). +- Tags present: openstack (apply to the 4 hosts), capi-mgmt, virtual, + pod-console-logging, juju, lxd, tailscale. + +--- + +## 6. DC-DC reuse notes + +Region-specific (re-derive per region): the 10.12.x CIDRs, the boot/NIC MACs, +the virsh power_address, the host octets, the libvirt host OOB address. + +Structural (replays unchanged across regions): the six-plane model and its +roles; metal-internal as a tagged VID-103 bridged stack over the metal plane; +the per-host interface tree shape (OVS provider/metal + std-bridge internal + +raw data/storage/replication); hostname-keyed identity with runtime system_id +resolution; auto-commission-on-create; tag `openstack` re-apply before deploy; +`default-space` must resolve to a LIVE space (a stale default-space globally +poisons `network-get`). diff --git a/docs/v1-redeploy-changelog.md b/docs/v1-redeploy-changelog.md new file mode 100644 index 0000000..bbc1021 --- /dev/null +++ b/docs/v1-redeploy-changelog.md @@ -0,0 +1,193 @@ +# v1 Redeploy -- Running Change Log + +**Purpose:** Living log of design decisions, doc fixes, and runbook edits discovered +DURING the v1 redeploy rehearsal that must be folded into `docs/design-decisions.md` +and the phase runbooks UPON COMPLETION. This is the staging list for the completion +consolidation -- nothing here is applied to the runbooks or design-decisions yet. + +**Status:** OPEN -- accumulating. Append-only. ASCII + LF. + +**Session opened:** 2026-06-26 (redeploy from clean teardown; D-052/D-053 plane set). + +**Next free numbers at session open:** design decision D-054; doc fix DOCFIX-039. +(Verified by grep of design-decisions.md: max D-053, max DOCFIX-038.) + +--- + +## Verified-state checkpoint (measured this session -- authoritative as-built) + +`scripts/pre-flight-checks.sh` @ commit 40e3f9e -- ALL PASS, exit 0, 2026-06-26: + +Six MAAS planes resolved BY CIDR (subnet IDs are post-D-052-cutover, NOT the old map): + + provider-public 10.12.4.0/22 id=1 vid=0 gw=10.12.4.1 dns=[10.12.4.1] + metal-admin 10.12.8.0/22 id=2 vid=0 gw=10.12.8.1 dns=[10.12.8.1] + metal-internal 10.12.12.0/22 id=10 vid=103 gw=none dns=[10.12.8.1] (bridged br-internal) + data-tenant 10.12.16.0/22 id=6 vid=0 gw=none dns=[10.12.8.1] + storage 10.12.32.0/22 id=7 vid=0 gw=none dns=[10.12.8.1] + replication 10.12.36.0/22 id=8 vid=0 gw=none dns=[10.12.8.1] + +Per-host data/storage NIC links by CIDR, octets .40-.43, all four hosts: +br-internal -> .12, enp8s0 -> .16, enp9s0 -> .32, enp10s0 -> .36. + +Nodes openstack0-3 (4na83t / qdbqd6 / h8frng / tmsafc): all Ready, power off. +OSD secondary disks (`osd-blank-check.sh`): all four 512 GiB / 200 KiB blank, RC=0. +Bundle VIPs: 11 triple-column VIPs, aligned, .50-.60 band, OK=11 bad=0. +octavia-pki overlay: present, 5 lb-mgmt-* keys, ASCII clean. + +--- + +## Pending design-decisions.md appends + +### D-054 -- Reusable tested scripts in scripts/; runbooks reference them (ADOPTED in practice; formal append pending) + +**What:** Repeated discovery/verify logic lives in `scripts/`, authored and tested in a +sandbox against synthetic fixtures, committed to the repo, and referenced by the runbooks. +Runbooks document expected output and remain the gate authority; the scripts are the +executable truth. All pinned network values live once in `scripts/lib-net.sh` (single +source of truth), resolved BY CIDR (subnet IDs drift across cutovers). + +**Delivery workflow:** author + test in sandbox -> publish file + sha256 -> commit from +Windows -> jumphost `git pull` -> `sha256sum` match -> run via `bash scripts/X.sh`. + +**Convention:** ASCII + LF (`.gitattributes` `*.sh eol=lf`); `set -euo pipefail` + +`shopt -s inherit_errexit` + `IFS=$'\n\t'`; `fail`/`warn`/`pass`/`note` helpers with +exit 0 (pass) / 1 (fatal) / 2 (warning) for gate scripts; read-only discovery kept +separate from gated mutation; `lib-net.sh` is sourced, never executed (direct-run guard). + +**Why:** Eliminates the paste-corruption failure class (see Findings below) and turns +repeated discovery -- polled every redeploy cycle -- into a one-liner with a byte-identity +guarantee (sha256) instead of a fragile copy-paste block. + +**Scripts added this session:** `lib-net.sh` (new), `pre-flight-checks.sh` (implemented the +placeholder), `juju-spaces-check.sh` (new), `osd-blank-check.sh` (new). All tested +end-to-end against mock `maas`/`juju` + fixtures (positive + 7 negative fault injections +for pre-flight; 4 scenarios for spaces). Committed at 40e3f9e. + +--- + +## Pending DOCFIX entries + +### DOCFIX-039 -- phase-01-bundle-deploy.md gate reconciliation (PROPOSED) + +The phase-01 pre-deploy GATES encode the OLD plane layout (pre-D-052 CIDR->role map); the +deploy COMMANDS are fine. Superseded by `scripts/pre-flight-checks.sh`. Five stale items: + +1. Constants: hardcoded subnet ids `1 2 6 7 8 9` + old CIDR->role map -> resolve BY CIDR + (now in `lib-net.sh`; metal-internal is id=10 post-cutover, not id=6). +2. CHECK 1 / Step 1.3 deploy guard: provider-column-only VIP check -> triple-column + validator (provider/admin/internal, aligned, .50-.60). +3. CHECK 2: `enp8s0` + `10.12.12.0/22` (old "data") -> links BY CIDR; `enp8s0` now carries + `10.12.16.0/22` (data-tenant), metal-internal is on `br-internal`. +4. CHECK 3: hardcoded ids/DNS -> subnets BY CIDR. +5. EXIT GATE binding plane map (old: ceph->.16 / octavia->.12.1 / nova->.12.4x / vault->.8) + -> corrected per D-052: ceph public/osd/mon->storage(.32); octavia overlay->data-tenant + (.16); nova-compute neutron-plugin->data-tenant(.16); vault default->metal-admin(.8) + + cluster->metal-internal(.12). + +**Action at completion:** replace the inline CHECK blocks in phase-01 with +`bash scripts/pre-flight-checks.sh` (document expected PASS output) and add a post-add-model +`bash scripts/juju-spaces-check.sh openstack` as the per-model space gate (the old inline +CHECK 5 ran `juju spaces` pre-model and failed "model not found"; spaces are per-model). + +--- + +## Pending runbook / file edits (apply at completion) + +1. `runbooks/phase-01-bundle-deploy.md` -- DOCFIX-039 (above): swap inline pre-flight blocks + for `bash scripts/pre-flight-checks.sh`; add post-add-model `bash scripts/juju-spaces-check.sh + openstack`; fix the 5 stale gate items; document expected output. +2. `scripts/validate.sh` -- convert UTF-8 to ASCII when implementing the D-011 runner + (phase-08). `file` reports "Unicode text, UTF-8 text" (em-dashes from the placeholder); + violates the ASCII-only convention. Currently a placeholder, not yet run. +3. Teardown runbook -- reference `scripts/osd-blank-check.sh` for the OSD-blank verification + step (replaces the inline qemu-img loop). +4. `runbooks/` README / pre-flight references -- point at the new scripts where the old + inline discovery blocks were described. + +--- + +## Findings / process learnings (this session) + +- **Paste-corruption failure class.** A hand-built base64 pre-flight block shipped two + transcription defects: `[:space:]` (single bracket, must be `[[:space:]]`) on the grep + count line, and `ENV{` instead of `END{` on the awk tally (so the summary silently never + printed). Root cause: the base64 was hand-edited AFTER testing a clean version -- the + bytes sent were never round-tripped through the sandbox. Mitigation is now standard + practice (D-054): tested scripts committed to the repo, verified by sha256 on the jumphost. + +- **Juju spaces are per-model.** `juju spaces` / `juju reload-spaces` cannot run until after + `juju add-model`; the old phase-01 CHECK 5 ran pre-model and failed with "model not found". + Split into `juju-spaces-check.sh`, gated to run post-add-model. + +- **Default-space globally poisons network-get (deploy root cause).** The full D-052 + binding deploy failed universally (`network-get ... ERROR space "metal" not found`, + install hook dies on nearly every charm). Every static layer was correct -- bundle, + model bindings, MAAS spaces/VLANs/per-NIC space tags all read `metal-internal`. The + single stale value was controller `model-defaults default-space = metal` (a dead + pre-D-052 name). An INVALID default-space poisons `network-get` for ALL endpoints + regardless of their explicit binding. Fix: set `juju model-defaults + default-space=metal-admin` (a live space) before add-model. A `default-space`-resolves- + to-a-live-space gate is to be added to `pre-flight-checks.sh`. + +- **Teardown --destroy-storage on virsh DELETES machine objects (does NOT release).** + The phase-00 teardown (`juju destroy-model openstack --force --destroy-storage` then + per-host `maas machine release`) assumes release-to-Ready. On a virsh/KVM MAAS, + `--destroy-storage` DECOMPOSES (deletes) the VM-backed machine objects. All four + openstack hosts were removed from MAAS. Recoverable only because the libvirt domains + + disks (incl the blank OSD vdb) survived. See D-055. + +--- + +## Pending design-decisions.md appends (continued) + +### D-055 -- virsh teardown defect + host re-enrollment procedure (ADOPTED) + +**Defect:** `juju destroy-model --destroy-storage` against virsh-power MAAS machines +deletes (decomposes) the machine objects rather than releasing them to Ready. The +phase-00 teardown must NOT pass `--destroy-storage` for virsh hosts; release to Ready +without it. + +**Recovery (now a reusable procedure):** the libvirt domains survive, so re-enroll via +`maas admin machines create` per host with virsh power + the boot NIC MAC (NOT add-chassis +-- it would re-grab juju/lxd/tailscale). `machines create` auto-commissions +(New->Commissioning->Ready) by PXE off the 2_metal boot NIC. Then re-tag `openstack`, +then reconstruct the host interface tree (Strategy-B carve, from the captured as-built), +then verify (pre-flight), then redeploy with the default-space fix. + +**Artifacts:** `scripts/lib-hosts.sh`, `scripts/reenroll-hosts.sh`, +`docs/maas-as-built-reference.md`. Proven live on openstack0 (2026-06-26): created +virsh, commissioned, Ready, all six NICs discovered, boot NIC on 2_metal. + +### DOCFIX-040 -- host identity must be hostname-keyed, not system_id-keyed + +`lib-net.sh` lines 45-47 key the host maps (`SYSIDS`, `SYSID_HOST`, `SYSID_OCTET`) on +the system_ids 4na83t/qdbqd6/h8frng/tmsafc -- which DIED on re-enrollment (new random +ids). Any script keyed on them silently breaks. New `scripts/lib-hosts.sh` keys all host +identity on hostname (stable) and resolves system_id at runtime (`host_sysid`). At +completion: retire the SYSID-keyed maps from lib-net.sh (or repoint them to lib-hosts). + +--- + +## Security note (action required) + +The libvirt SSH password (`logxen@10.12.64.1`) was printed in plaintext on 2026-06-26 by +`maas admin machine power-parameters` during virsh power-template discovery. Treat as +exposed: **rotate the libvirt SSH credential after the rebuild** and scrub terminal +scrollback. Runbook rule added: never use `machine power-parameters` for templating; read +`power_type` and reconstruct the address pattern instead. `reenroll-hosts.sh` reads the +password interactively (never a CLI arg, never logged, never in the repo). + +--- + +## Scripts / docs added (this batch) + +- `scripts/lib-hosts.sh` -- hostname-keyed host identity + virsh power constants (no secret). +- `scripts/reenroll-hosts.sh` -- gated/idempotent re-enrollment (auto-commission, poll Ready, + boot-NIC-on-2_metal verify; --check read-only mode). Tested: bash -n, shellcheck clean, + mock-maas behavior test of --check (discover-by-hostname, NOT-ENROLLED detection, exit 0). +- `docs/maas-as-built-reference.md` -- captured MAAS substrate + per-host NIC inventory + + interface-carve target + virsh template, for DC-DC replay. +- Pending next artifact: the Strategy-B interface-carve script (built once all four are Ready; + bridge_type pulled verbatim from captured release JSON) -> then consolidate into + `runbooks/phase-00b-host-reenrollment.md`. diff --git a/scripts/lib-hosts.sh b/scripts/lib-hosts.sh new file mode 100644 index 0000000..4bec6f6 --- /dev/null +++ b/scripts/lib-hosts.sh @@ -0,0 +1,58 @@ +# scripts/lib-hosts.sh +# +# Single source of truth for the four OpenStack KVM host VMs (VR0 / Baldurkeep): +# their enrollment identity and virsh power parameters. SOURCED, not executed. +# ASCII + LF. Contains constants + read-only helpers ONLY; no mutations. +# +# WHY hostname-keyed (NOT system_id-keyed): MAAS system_ids are minted fresh on +# every (re-)enrollment. The old 4na83t/qdbqd6/h8frng/tmsafc died when the hosts +# were decomposed on 2026-06-26 and re-enrollment assigned new random ids. The +# stable identities are the hostname and the libvirt domain name, so every map +# here keys on hostname and the live system_id is resolved AT RUNTIME via +# host_sysid(). This SUPERSEDES the SYSID-keyed maps in lib-net.sh (DOCFIX-040). +# +# NO SECRET lives here. The libvirt SSH password is read interactively at run time +# (reenroll-hosts.sh) and is never written to a file, a command line, or the repo. + +# shellcheck shell=bash +# shellcheck disable=SC2034 # constants are consumed by sourcing scripts + +# Guard: sourced only. +if [ "${BASH_SOURCE[0]:-}" = "${0}" ]; then + echo "lib-hosts.sh is a sourced library; do not run it directly." >&2 + exit 2 +fi + +# The four OpenStack KVM hosts. libvirt domain name == MAAS hostname == power_id. +HOSTS=( openstack0 openstack1 openstack2 openstack3 ) + +# host -> last IPv4 octet on every plane (.40-.43). Stable by design (D-052 index). +declare -A HOST_OCTET=( [openstack0]=40 [openstack1]=41 [openstack2]=42 [openstack3]=43 ) + +# host -> PXE/boot NIC MAC == the 2_metal interface (libvirt ). +# Fixed in the libvirt domain XML; captured 2026-06-26. The boot NIC MUST be the +# metal/PXE plane or commissioning cannot DHCP/PXE. +declare -A HOST_BOOT_MAC=( + [openstack0]=52:54:00:4f:1c:0b + [openstack1]=52:54:00:83:25:1f + [openstack2]=52:54:00:23:bd:72 + [openstack3]=52:54:00:b2:7b:30 +) + +# virsh power (non-secret). MAAS reaches libvirt over SSH on the OOB host address. +# Mirrors the surviving juju/lxd/tailscale virsh machines' live config (read +# 2026-06-26). power_id is set per host to the hostname by the caller. +VIRSH_POWER_ADDRESS="qemu+ssh://logxen@10.12.64.1/system" +HOST_ARCH="amd64" + +# MAAS tag the deploy bundle places units against (constraint tags=openstack). +# Must be (re-)applied to every host post-commission or the bundle cannot bind. +HOST_TAG="openstack" + +# host_sysid : resolve the LIVE MAAS system_id by hostname (never +# hardcode it). Echoes the system_id, or empty if the host is not enrolled. +host_sysid() { + local hn="$1" + maas "${MAAS_PROFILE:-admin}" machines read 2>/dev/null \ + | jq -r --arg h "$hn" '.[] | select(.hostname==$h) | .system_id' | head -1 +} diff --git a/scripts/reenroll-hosts.sh b/scripts/reenroll-hosts.sh new file mode 100644 index 0000000..dfefe2a --- /dev/null +++ b/scripts/reenroll-hosts.sh @@ -0,0 +1,149 @@ +#!/usr/bin/env bash +# scripts/reenroll-hosts.sh +# +# Re-enroll the four OpenStack KVM hosts into MAAS as virsh-power machines after +# they were deleted/decomposed from MAAS. The libvirt domains must still EXIST -- +# this does NOT recreate VMs, only their MAAS machine objects. MAAS auto-commissions +# on create, so each host goes New -> Commissioning -> Ready, PXE-booting off its +# 2_metal boot NIC. +# +# Modes: +# (default) create any of the four that are MISSING, then poll all four to Ready +# --check read-only: report current status of openstack0-3 (no mutation) +# +# Discover-assert-pin: never creates a host that already exists. Idempotent -- +# a re-run after a partial run only creates the still-missing hosts. +# +# The libvirt SSH password is READ INTERACTIVELY (never a CLI arg, never echoed, +# never logged, never in the repo). SECURITY NOTE: that credential was exposed in +# plaintext on 2026-06-26 (`maas machine power-parameters` echoes power_pass) -- +# ROTATE the libvirt SSH credential after this rebuild. +# +# Pinned values come from scripts/lib-hosts.sh (host identity) + lib-net.sh. +# Exit codes: 0 all four Ready | 1 fatal | 2 warning/timeout + +set -euo pipefail +shopt -s inherit_errexit 2>/dev/null || true +IFS=$'\n\t' + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=scripts/lib-net.sh +. "$SCRIPT_DIR/lib-net.sh" +# shellcheck source=scripts/lib-hosts.sh +. "$SCRIPT_DIR/lib-hosts.sh" + +MAAS_PROFILE="${MAAS_PROFILE:-admin}" +READY_TIMEOUT="${READY_TIMEOUT:-1200}" # seconds (~20 min) + +FATAL=0; WARN=0 +fail() { echo "FAIL: $*" >&2; FATAL=$((FATAL+1)); } +warn() { echo "WARN: $*" >&2; WARN=$((WARN+1)); } +pass() { echo "PASS: $*"; } +note() { echo "NOTE: $*"; } +hdr() { echo; echo "=== $* ==="; } +finish() { + echo; echo "Summary: ${FATAL} fatal, ${WARN} warning" + if [ "$FATAL" -gt 0 ]; then exit 1 + elif [ "$WARN" -gt 0 ]; then exit 2 + fi + exit 0 +} + +need_jq || exit 1 + +MODE="create" +[ "${1:-}" = "--check" ] && MODE="check" + +# mread -> machine JSON or empty (set -e safe) +mread() { maas "$MAAS_PROFILE" machine read "$1" 2>/dev/null || true; } + +# report: read-only status of the four hosts (system_id resolved live by hostname) +report() { + local hn sid j st pwr fab + for hn in "${HOSTS[@]}"; do + sid="$(host_sysid "$hn" || true)" + if [ -z "$sid" ]; then printf " %-11s NOT-ENROLLED\n" "$hn"; continue; fi + j="$(mread "$sid")" + st="$(printf '%s' "$j" | jq -r '.status_name // "?"' 2>/dev/null || echo '?')" + pwr="$(printf '%s' "$j" | jq -r '.power_state // "?"' 2>/dev/null || echo '?')" + fab="$(maas "$MAAS_PROFILE" interfaces read "$sid" 2>/dev/null \ + | jq -r --arg m "${HOST_BOOT_MAC[$hn]}" '.[]|select(.mac_address==$m)|.vlan.fabric' \ + | head -1 || true)" + printf " %-11s sid=%-8s %-12s power=%-4s bootnic_fabric=%s\n" \ + "$hn" "$sid" "$st" "$pwr" "${fab:-?}" + done +} + +hdr "Current host status (by hostname; system_id resolved live)" +report + +if [ "$MODE" = "check" ]; then + note "read-only check mode; no changes made" + finish +fi + +# --------------------------------------------------------------------------- +# CREATE mode: build the MISSING set (discover-assert-pin); create only those. +MISSING=() +for hn in "${HOSTS[@]}"; do + [ -z "$(host_sysid "$hn" || true)" ] && MISSING+=("$hn") +done + +if [ "${#MISSING[@]}" -eq 0 ]; then + pass "all four hosts already enrolled; nothing to create" +else + note "to create: ${MISSING[*]}" + # Secret read WITHOUT echo. Never placed on a command line; unset after use. + read -rsp "libvirt SSH password for ${VIRSH_POWER_ADDRESS}: " PPASS; echo + [ -n "${PPASS:-}" ] || { fail "empty password; aborting (nothing created)"; finish; } + for hn in "${MISSING[@]}"; do + echo " creating $hn (boot mac ${HOST_BOOT_MAC[$hn]})" + if maas "$MAAS_PROFILE" machines create \ + hostname="$hn" \ + architecture="$HOST_ARCH" \ + mac_addresses="${HOST_BOOT_MAC[$hn]}" \ + power_type=virsh \ + power_parameters_power_id="$hn" \ + power_parameters_power_address="$VIRSH_POWER_ADDRESS" \ + power_parameters_power_pass="$PPASS" \ + | jq '{system_id, hostname, status_name, power_type}'; then + pass "create accepted: $hn" + else + fail "create failed: $hn" + fi + done + unset PPASS +fi + +# --------------------------------------------------------------------------- +hdr "Waiting for all four Ready (timeout ${READY_TIMEOUT}s)" +deadline=$(( $(date +%s) + READY_TIMEOUT )) +while :; do + allready=1 + for hn in "${HOSTS[@]}"; do + sid="$(host_sysid "$hn" || true)" + st="MISSING" + [ -n "$sid" ] && st="$(printf '%s' "$(mread "$sid")" | jq -r '.status_name // "?"' 2>/dev/null || echo '?')" + [ "$st" = "Ready" ] || allready=0 + done + if [ "$allready" = 1 ]; then pass "all four Ready"; break; fi + if [ "$(date +%s)" -ge "$deadline" ]; then warn "timeout waiting for Ready"; break; fi + sleep 20 +done + +hdr "Final status" +report + +# Verify each boot NIC landed on the 2_metal (PXE/admin) fabric. +hdr "Boot-NIC fabric check (expect 2_metal)" +for hn in "${HOSTS[@]}"; do + sid="$(host_sysid "$hn" || true)" + [ -n "$sid" ] || { fail "$hn not enrolled"; continue; } + fab="$(maas "$MAAS_PROFILE" interfaces read "$sid" 2>/dev/null \ + | jq -r --arg m "${HOST_BOOT_MAC[$hn]}" '.[]|select(.mac_address==$m)|.vlan.fabric' \ + | head -1 || true)" + if [ "$fab" = "2_metal" ]; then pass "$hn boot NIC on 2_metal"; else fail "$hn boot NIC on '${fab:-?}' (want 2_metal)"; fi +done + +note "next: re-tag '${HOST_TAG}' on all four, then the Strategy-B interface carve" +finish