diff --git a/docs/audit/gauntlet-20260807-tailscale-carve.txt b/docs/audit/gauntlet-20260807-tailscale-carve.txt new file mode 100644 index 0000000..85539ca --- /dev/null +++ b/docs/audit/gauntlet-20260807-tailscale-carve.txt @@ -0,0 +1,30 @@ + phase-06-mgmt-vm ALL PASS + phase-06-net-setup ALL PASS + phase-07-conductor-graft ALL PASS + pre-flight-checks ALL PASS + preflight ALL PASS + prereqs prereqs: 32/32 PASS + provider-bundle-check ALL PASS + record-audit record-audit: 35/35 PASS + reenroll-hosts ALL PASS + render-baseline RESULT: PASS=10 FAIL=0 + render-dc-overlays ALL PASS + render-drift ALL PASS + repo-lint ALL PASS + roles-aggregates-import roles-aggregates-import: 24/24 PASS + sandbox-fidelity-check sandbox-fidelity-check: 14/14 PASS + sandbox-seed sandbox-seed: 22/22 PASS + site-baseleg site-baseleg: 24 passed, 0 failed + site-forward site-forward: 38/38 PASS + site-headend-install site-headend-install: 59/59 PASS + site-ssh-config site-ssh-config: 30/30 PASS + site-tailscale === 15 pass / 0 fail === + tenant-acceptance ALL PASS + tenant-assert ALL PASS + tenant-offboard ALL PASS (22/22) + tenant-onboard ALL PASS + trust_filter ALL PASS + validate ALL PASS (29 checks) + vault-kv-health ALL PASS (10/10) + +GAUNTLET: ALL GREEN (101 harnesses) diff --git a/docs/changelog-20260807-dc0-tailscale-provisioning.md b/docs/changelog-20260807-dc0-tailscale-provisioning.md new file mode 100644 index 0000000..4307484 --- /dev/null +++ b/docs/changelog-20260807-dc0-tailscale-provisioning.md @@ -0,0 +1,84 @@ +# Changelog 2026-08-07 -- dc0 Tailscale .7 subnet-router: MAAS provisioning + aux-carve tooling + +Session: pick up the 2026-08-06/07 close -- provision the per-DC Tailscale `.7` subnet-router +toward closing phase-03 Step 3.3 (Horizon over the tailnet). Branch `dc-dc-stage5-preconditions`. +Under blanket approval the changelog is the review surface: each item states WHAT, WHY +(evidence), and HOW TO REVERT. Live cloud mutations are individually operator-gated. + +Operator rulings this session (GA-R5; verbatim, dated 2026-08-07): +- dc1 approach: **"No migration. Build region on DC1 correctly."** -- dc1's `.7` is gated behind + building the dedicated `vr1-dc1-region` on `vr1-dc1-maas-01` (N3); do NOT enlist it into the + Office1 `admin` region. Consistent with the standing region-per-DC invariant (D-132 q1). +- Tailscale join key: operator has a **plain (not tag-scoped) pre-auth key**. The ruled design + (D-129(iii) amdt note 1) is TAGGED identity; a plain key cannot authorise + `--advertise-tags=tag:subnet-router`, so the JOIN stays blocked pending a tagged key + the + Headscale-side autoApprovers/star-ACL (N1). Provisioning to carved-and-ready is unblocked. + +As-executed log (O3, repeated): `run-logged.sh` opens an interactive `script -aqe` subshell, +which a background agent cannot drive; the log was NOT opened. Every mutation below is gated +and captured here with its read-back. Flagged, not silent. + +## Measured corrections to the record (GA-R1 C2 -- measurement wins) + +- The dc0 `.7` VM had **already self-enlisted** into `vr1-dc0-region` as `known-marten` + (system_id `n4parh`), status New, `power_type` UNSET, powered off -- its six NICs carry the + pinned tailscale MACs (`opentofu/vr1-dc0-substrate/main.tf:243-245`). The 2026-08-06/07 close + said "powered off"; the enlisted-already part was not recorded. virsh `domstate` = shut off. +- `subtle-grouse` is the dc0 juju controller (`power_id=vr1-dc0-juju-01`), never renamed in + MAAS; it carries the 2-plane carve because `dc-node-carve.sh` resolves ROLE from the stable + lib-hosts LOGICAL name and machine IDENTITY from the pinned boot MAC -- the MAAS random name + is irrelevant. This is the idiom the tailscale aux-carve reuses. +- `vr1-dc1-region` is NOT registered; dc1's rack (`nmpcq4`) + 9 nodes are in the Office1 `admin` + region. dc1 `.7` has no dedicated region -> the operator ruling above. + +## Item 1 -- dc0 .7 VM: MAAS power control set + commissioned to Ready (LIVE, gated) + +WHAT: on `known-marten` (n4parh) in `vr1-dc0-region`: + (a) `maas-node-power.sh --commit` set `power_type=virsh`, + `power_address=qemu+ssh://jessea123@10.12.8.2/system` (the DC-region-reachable metal-admin + form, read off the working juju controller's `power-parameters`), `power_id=vr1-dc0-tailscale-01`. + Read-back: `[ok] known-marten -> vr1-dc0-tailscale-01 (power state: off)` -- MAAS proved + power control by a live `query-power-state`. + (b) `maas vr1-dc0-region machine commission n4parh` -> polled Commissioning -> Testing -> + **Ready** at 02:55:33Z. + +WHY: the `.7` router must be MAAS-deployed to get an OS (it is NOT in the juju model, so nothing +else deploys it -- DOCFIX-200's "stay Ready" applies only to juju-managed role nodes). Power +MUST be set before commission (the unset-power silent drop-to-New trap). The power address is +the region-reachable `10.12.8.2` (measured OPEN from the DC-local region; the Office1-form +`172.31.0.2` is CLOSED from there); listed domains over the transit `172.31.0.2` via `VIRSH_URI`. + +REVERT: `maas vr1-dc0-region machine release n4parh` (back to Ready) or `... machine delete +n4parh` + power off the domain; the substrate VM is untouched. + +## Item 2 -- dc-node-carve.sh + lib-hosts.sh: AUX carve target for the Tailscale .7 router + +WHAT: taught the v4 node carve to carve the per-DC Tailscale `.7` VM, which is NOT an OpenStack +role node: +- `scripts/lib-hosts.sh`: new `CARVE_AUX_HOSTS` array (default empty; `( vr1-dc0-tailscale-01 )` + / `( vr1-dc1-tailscale-01 )` per DC) so the VM is a carve target WITHOUT entering `HOSTS` + (every HOSTS consumer -- maas-role-tags, pre-flight-checks `${#HOSTS[@]}`, phase-00-teardown, + reenroll-hosts, dc-node-v6-carve -- would otherwise treat it as a role node). Inert + `HOST_OCTET[..]=7` + `HOST_BOOT_MAC` entries (map keys are looked-up, never iterated). + New `NODE_CARVE_TAILSCALE_PLANES=2` + `TAILSCALE_HOST_SUFFIX`. +- `scripts/dc-node-carve.sh`: `is_tailscale_host` + `is_two_plane_host` (juju OR tailscale); + `planes_for_host` and BOTH br-ex branches + the "spare NICs / NO br-ex" verifier now key on + `is_two_plane_host` -- so the tailscale VM carves metal-admin + provider-public RAW with NO + OVS br-ex (identical to the juju controller, octet 7). `--host` accepts `CARVE_AUX_HOSTS` + members; a bare apply still iterates `HOSTS` only (aux = opt-in by `--host`). +- `tests/dc-node-carve/run-tests.sh`: `t01` machine fixture + `emit_tailscale` + T28..T33 -- + passes on the 2-plane no-br-ex end state at `10.12.8.7`/`10.12.4.7`, and FAILS (failing- + direction fixtures) on a tailscale VM grown a br-ex (`tsbrex`) or with a spare NIC carved + onto a plane (`tscarved`); T31 proves the aux `--host` allowance did not widen unknown hosts. + +WHY: hard rule 4 -- no tested artifact carved this 2-leg shape (the existing tool is role/juju +shaped; a mis-key silently falls through to a 6-plane br-ex role carve). Implements the ruled +D-129(iii) amendment (2026-08-07) + D-134 octet map; no new D-number. The carve must precede +MAAS-deploy (MAAS refuses interface changes on a Deployed machine). + +GATES: `tests/dc-node-carve/run-tests.sh` 58 passed / 0 failed; repo-lint 0 fail (1 legacy +warn); full gauntlet **ALL GREEN (101 harnesses)** -- `docs/audit/gauntlet-20260807-tailscale-carve.txt`. + +REVERT: `git revert` this commit -- removes `CARVE_AUX_HOSTS`, the tailscale predicates, and the +harness cases; `dc-node-carve.sh` returns to role+juju only. lib-hosts map keys become inert. +No live-cloud effect (this item is repo tooling; the carve APPLY is a separate gated mutation). diff --git a/scripts/dc-node-carve.sh b/scripts/dc-node-carve.sh index e6a4246..d8df314 100755 --- a/scripts/dc-node-carve.sh +++ b/scripts/dc-node-carve.sh @@ -371,8 +371,14 @@ } # --- role resolution -- DERIVED from the host token, never typed ----------- +# The token is the STABLE lib-hosts logical name (not the re-minted MAAS +# hostname); machine identity is resolved separately by pinned boot MAC. is_juju_host(){ case "$1" in *"$JUJU_HOST_SUFFIX") return 0 ;; *) return 1 ;; esac; } -planes_for_host(){ if is_juju_host "$1"; then echo "$NODE_CARVE_JUJU_PLANES"; else echo "$NODE_CARVE_ROLE_PLANES"; fi; } +is_tailscale_host(){ case "$1" in *"${TAILSCALE_HOST_SUFFIX:-tailscale-01}") return 0 ;; *) return 1 ;; esac; } +# Two-plane hosts (juju controller, Tailscale router) carve metal-admin + +# provider-public RAW with NO OVS br-ex -- only role nodes get br-ex. +is_two_plane_host(){ is_juju_host "$1" || is_tailscale_host "$1"; } +planes_for_host(){ if is_two_plane_host "$1"; then echo "$NODE_CARVE_JUJU_PLANES"; else echo "$NODE_CARVE_ROLE_PLANES"; fi; } # =========================================================================== reload_static @@ -393,8 +399,12 @@ HOSTS_TODO=() if [ -n "$ONLY_HOST" ]; then - _found=0; for h in "${HOSTS[@]}"; do [ "$h" = "$ONLY_HOST" ] && _found=1; done - [ "$_found" = 1 ] || refuse "--host '$ONLY_HOST' is not a $SITE host in lib-hosts.sh HOSTS" + # --host may target a role node (HOSTS) OR an aux carve target (CARVE_AUX_HOSTS, + # e.g. the Tailscale .7 router). Aux hosts are OPT-IN by --host only: a bare + # apply iterates HOSTS and never touches them. + _found=0 + for h in "${HOSTS[@]}" "${CARVE_AUX_HOSTS[@]:-}"; do [ "$h" = "$ONLY_HOST" ] && _found=1; done + [ "$_found" = 1 ] || refuse "--host '$ONLY_HOST' is not a $SITE host in lib-hosts.sh HOSTS or CARVE_AUX_HOSTS" HOSTS_TODO=( "$ONLY_HOST" ) else HOSTS_TODO=( "${HOSTS[@]}" ) @@ -422,7 +432,7 @@ [ "$IDX" -lt "$NPLANES" ] || break NIC="$(nic_for "$IDX")"; CIDR="${CIDR_OF[$PLANE]}"; IP="$(plane_ip "$CIDR" "$OCTET")" if [ "$ACTION" = check ]; then - if [ "$PLANE" = provider-public ] && ! is_juju_host "$HOST"; then + if [ "$PLANE" = provider-public ] && ! is_two_plane_host "$HOST"; then ck "$([ "$(if_link br-ex "$CIDR")" = "static:$IP" ] && echo 0 || echo 1)" "$HOST br-ex STATIC $IP on $PLANE (got '$(if_link br-ex "$CIDR")')" ck "$([ "$(if_field br-ex type)" = "bridge" ] && echo 0 || echo 1)" "$HOST br-ex is a bridge (got '$(if_field br-ex type)')" ck "$([ "$(if_field br-ex parents)" = "$NIC" ] && echo 0 || echo 1)" "$HOST br-ex parent is $NIC (got '$(if_field br-ex parents)')" @@ -435,7 +445,7 @@ ck "$([ "$(if_field "$NIC" vlan)" = "$(sub_field "$CIDR" vlan)" ] && echo 0 || echo 1)" "$HOST $NIC on the $PLANE VLAN (got '$(if_field "$NIC" vlan)' want '$(sub_field "$CIDR" vlan)')" fi else - if [ "$PLANE" = provider-public ] && ! is_juju_host "$HOST"; then + if [ "$PLANE" = provider-public ] && ! is_two_plane_host "$HOST"; then build_ovs_brex "$SID" "$NIC" "$CIDR" "$IP" || true else carve_raw "$SID" "$NIC" "$CIDR" "$IP" || true @@ -444,10 +454,11 @@ IDX=$((IDX+1)) done - # The controller's enp3s0..enp6s0 are DELIBERATELY left on auto VLANs. Assert - # that deliberate absence, so a later session cannot "helpfully" carve them - # and have every gate stay green. - if [ "$ACTION" = check ] && is_juju_host "$HOST"; then + # A two-plane host's enp3s0..enp6s0 are DELIBERATELY left on auto VLANs (the + # juju controller AND the Tailscale .7 router). Assert that deliberate absence, + # so a later session cannot "helpfully" carve them and have every gate stay + # green -- and that no OVS br-ex was grown on the raw provider leg. + if [ "$ACTION" = check ] && is_two_plane_host "$HOST"; then _stray=0 for _i in $(seq "$NPLANES" 5); do _n="$(nic_for "$_i")" @@ -456,7 +467,7 @@ done done ck "$([ "$_stray" = 0 ] && echo 0 || echo 1)" "$HOST enp$((NPLANES+1))s0..enp6s0 carry NO plane link ($_stray found) -- deliberately left on auto VLANs" - ck "$([ -z "$(if_field br-ex id)" ] && echo 0 || echo 1)" "$HOST has NO br-ex (got id '$(if_field br-ex id)') -- the controller takes the provider leg RAW" + ck "$([ -z "$(if_field br-ex id)" ] && echo 0 || echo 1)" "$HOST has NO br-ex (got id '$(if_field br-ex id)') -- it takes the provider leg RAW" fi done diff --git a/scripts/lib-hosts.sh b/scripts/lib-hosts.sh index f75f0a9..7d28db3 100644 --- a/scripts/lib-hosts.sh +++ b/scripts/lib-hosts.sh @@ -26,6 +26,13 @@ # The four OpenStack KVM hosts. libvirt domain name == MAAS hostname == power_id. HOSTS=( openstack0 openstack1 openstack2 openstack3 ) +# Auxiliary (non-Juju-managed) VMs that dc-node-carve.sh may carve via an +# EXPLICIT --host, but which MUST NOT enter HOSTS: every HOSTS consumer +# (maas-role-tags, pre-flight-checks ${#HOSTS[@]}, phase-00-teardown, reenroll, +# dc-node-v6-carve) would otherwise treat them as OpenStack role nodes. Populated +# per-DC below; empty at the flat/VR0 layer. (D-129(iii) amdt 2026-08-07.) +CARVE_AUX_HOSTS=() + # host -> last IPv4 octet on every plane (.40-.43). Stable by design (D-052 index). declare -A HOST_OCTET=( [openstack0]=40 [openstack1]=41 [openstack2]=42 [openstack3]=43 ) @@ -80,6 +87,11 @@ NODE_CARVE_ROLE_PLANES=6 # role nodes: every plane NODE_CARVE_JUJU_PLANES=2 # controller: metal-admin + provider-public only JUJU_HOST_SUFFIX="juju-01" # how the controller is identified in HOSTS +# The per-DC Tailscale subnet router carves the SAME shape as the controller -- +# metal-admin + provider-public, NO OVS br-ex -- just at the utility .7 octet +# (D-129(iii) amdt 2026-08-07; provider-public egress reaches Headscale). +NODE_CARVE_TAILSCALE_PLANES=2 +TAILSCALE_HOST_SUFFIX="tailscale-01" # how the aux Tailscale VM is identified # The per-region forms exist only for the VR1 DCs (D-132 q1). Declared EMPTY at # the flat/VR0 layer so a shell that never selects a VR1 DC cannot pick up a @@ -173,8 +185,8 @@ HOST_OCTET=( [vr1-dc0-control-01]=100 [vr1-dc0-control-02]=101 [vr1-dc0-control-03]=102 [vr1-dc0-compute-01]=120 [vr1-dc0-compute-02]=121 [vr1-dc0-storage-01]=150 [vr1-dc0-storage-02]=151 - [vr1-dc0-storage-03]=152 [vr1-dc0-storage-04]=153 - [vr1-dc0-juju-01]=5 ) + [vr1-dc0-storage-03]=152 [vr1-dc0-storage-04]=153 + [vr1-dc0-juju-01]=5 [vr1-dc0-tailscale-01]=7 ) HOST_BOOT_MAC=( [vr1-dc0-control-01]=52:54:00:be:69:c5 [vr1-dc0-control-02]=52:54:00:02:ff:57 [vr1-dc0-control-03]=52:54:00:4f:de:a9 @@ -183,12 +195,17 @@ [vr1-dc0-storage-01]=52:54:00:5f:8d:42 [vr1-dc0-storage-02]=52:54:00:48:86:2c [vr1-dc0-storage-03]=52:54:00:b1:94:d1 - [vr1-dc0-storage-04]=52:54:00:2b:ed:ab - [vr1-dc0-juju-01]=52:54:00:48:e7:1e ) + [vr1-dc0-storage-04]=52:54:00:2b:ed:ab + [vr1-dc0-juju-01]=52:54:00:48:e7:1e + [vr1-dc0-tailscale-01]=52:54:00:54:d5:5e ) VIRSH_POWER_ADDRESS_FROM_OFFICE1="qemu+ssh://jessea123@172.31.0.2/system" VIRSH_POWER_ADDRESS_FROM_DCREGION="qemu+ssh://jessea123@10.12.8.2/system" VIRSH_POWER_ADDRESS="$VIRSH_POWER_ADDRESS_FROM_OFFICE1" HOST_TAG="openstack-vr1-dc0" + # The per-DC Tailscale subnet router (.7): an AUX carve target reached via + # --host, NOT an OpenStack role node (stays out of HOSTS). Boot MAC + octet + # above pin its identity; the substrate pins the domain (main.tf:243). + CARVE_AUX_HOSTS=( vr1-dc0-tailscale-01 ) ;; vr1-dc1) # MEASURED 2026-07-23 (Stage 4): 9/9 Ready; MACs are the config pins @@ -201,8 +218,8 @@ HOST_OCTET=( [vr1-dc1-control-01]=100 [vr1-dc1-control-02]=101 [vr1-dc1-control-03]=102 [vr1-dc1-compute-01]=120 [vr1-dc1-compute-02]=121 [vr1-dc1-storage-01]=150 [vr1-dc1-storage-02]=151 - [vr1-dc1-storage-03]=152 [vr1-dc1-storage-04]=153 - [vr1-dc1-juju-01]=5 ) + [vr1-dc1-storage-03]=152 [vr1-dc1-storage-04]=153 + [vr1-dc1-juju-01]=5 [vr1-dc1-tailscale-01]=7 ) HOST_BOOT_MAC=( [vr1-dc1-control-01]=52:54:01:d1:01:01 [vr1-dc1-control-02]=52:54:01:d1:02:01 [vr1-dc1-control-03]=52:54:01:d1:03:01 @@ -211,8 +228,9 @@ [vr1-dc1-storage-01]=52:54:01:d1:06:01 [vr1-dc1-storage-02]=52:54:01:d1:07:01 [vr1-dc1-storage-03]=52:54:01:d1:08:01 - [vr1-dc1-storage-04]=52:54:01:d1:09:01 - [vr1-dc1-juju-01]=52:54:00:53:12:70 ) + [vr1-dc1-storage-04]=52:54:01:d1:09:01 + [vr1-dc1-juju-01]=52:54:00:53:12:70 + [vr1-dc1-tailscale-01]=52:54:00:f9:d6:df ) VIRSH_POWER_ADDRESS_FROM_OFFICE1="qemu+ssh://jessea123@172.31.0.6/system" # 10.12.68.2 MEASURED on the dc1 rack 2026-07-30 (`ip -4 -o addr`: virbr6 # inet 10.12.68.2/22), corroborated by dc-rack-net.sh's dc1 LEGS block. @@ -220,6 +238,11 @@ VIRSH_POWER_ADDRESS_FROM_DCREGION="qemu+ssh://jessea123@10.12.68.2/system" VIRSH_POWER_ADDRESS="$VIRSH_POWER_ADDRESS_FROM_OFFICE1" HOST_TAG="openstack-vr1-dc1" + # The per-DC Tailscale subnet router (.7): an AUX carve target reached via + # --host, NOT an OpenStack role node (stays out of HOSTS). MAC pinned from + # opentofu/vr1-dc1-substrate/main.tf:207. dc1 carve is gated on the + # dedicated vr1-dc1 region being built first (operator ruling 2026-08-07). + CARVE_AUX_HOSTS=( vr1-dc1-tailscale-01 ) ;; dc0|dc1|dc2) echo "FAIL: bare '$dc' is RETIRED (D-119). It was AMBIGUOUS ACROSS REGIONS: 'dc0' meant VR0's live DC0 here, but VR1's FIRST DC in the NetBox importer. Use the region-qualified selector: vr0-dc0 | vr1-dc0 | vr1-dc1" >&2 diff --git a/tests/dc-node-carve/run-tests.sh b/tests/dc-node-carve/run-tests.sh index d8e04d0..42971a6 100755 --- a/tests/dc-node-carve/run-tests.sh +++ b/tests/dc-node-carve/run-tests.sh @@ -70,7 +70,8 @@ {"system_id":"s02","hostname":"known-alpaca","status_name":"Ready","boot_interface":{"mac_address":"52:54:00:48:86:2c"}}, {"system_id":"s03","hostname":"loyal-oyster","status_name":"Ready","boot_interface":{"mac_address":"52:54:00:b1:94:d1"}}, {"system_id":"s04","hostname":"civil-bug","status_name":"${STUB_S04_STATUS:-Ready}","boot_interface":{"mac_address":"52:54:00:2b:ed:ab"}}, - {"system_id":"j01","hostname":"subtle-grouse","status_name":"Ready","boot_interface":{"mac_address":"52:54:00:48:e7:1e"}}] + {"system_id":"j01","hostname":"subtle-grouse","status_name":"Ready","boot_interface":{"mac_address":"52:54:00:48:e7:1e"}}, + {"system_id":"t01","hostname":"known-marten","status_name":"${STUB_T01_STATUS:-Ready}","boot_interface":{"mac_address":"52:54:00:54:d5:5e"}}] M } @@ -93,7 +94,7 @@ c01) echo 100 ;; c02) echo 101 ;; c03) echo 102 ;; m01) echo 120 ;; m02) echo 121 ;; s01) echo 150 ;; s02) echo 151 ;; s03) echo 152 ;; s04) echo 153 ;; - j01) echo 5 ;; + j01) echo 5 ;; t01) echo 7 ;; esac } @@ -158,6 +159,27 @@ JU } +# the TAILSCALE .7 ROUTER: same shape as the controller (two planes RAW, no +# br-ex, enp3-6 on auto VLANs) but at octet 7. +emit_tailscale() { + local extra='' brex='' + if [ "$S" = tscarved ]; then + extra='{"id":3,"name":"enp3s0","type":"physical","vlan":{"id":5004},"parents":[],"links":[{"id":3,"mode":"static","ip_address":"10.12.12.7","subnet":{"cidr":"10.12.12.0/22"}}]},' + else + extra='{"id":3,"name":"enp3s0","type":"physical","vlan":{"id":5301},"parents":[],"links":[{"id":3,"mode":"link_up","ip_address":null,"subnet":null}]},' + fi + [ "$S" = tsbrex ] && brex='{"id":90,"name":"br-ex","type":"bridge","vlan":{"id":5002},"parents":["enp2s0"],"links":[]},' + cat <