diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 6dff0eb..6f61dcc 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -160,9 +160,11 @@ pattern, repo-carried + part of DC standup definition-of-done; sub-2..4 OPEN -- status line in design-decisions.md is the authority). SEC-014 OPENED (rack cluster - secret exposure during diagnosis). Queued delivery: forwarder + - rack-legs repo-carried persistence, appendix-A entries, LP report, - stale pod object cleanup. **MAC pinning SHIPPED 2026-07-21** (54 MACs + secret exposure during diagnosis). Queued delivery: appendix-A + entries, LP report, stale pod object cleanup. Forwarder + rack-legs + persistence SHIPPED 2026-07-21 as `scripts/dc-rack-net.sh` (D-131 + sub-1 delivery; harness 14 cases; gauntlet 74 ALL GREEN); rack + install PENDING, operator-gated. **MAC pinning SHIPPED 2026-07-21** (54 MACs measured via `virsh domiflist` + pinned in modules/node-vm + vr1-dc0-substrate; harness 15 cases; gauntlet 73 ALL GREEN) together with an operator-ruled power-ownership guard (`ignore_changes = diff --git a/docs/changelog-20260721-close-and-delivery.md b/docs/changelog-20260721-close-and-delivery.md index 0f45ce9..093ef41 100644 --- a/docs/changelog-20260721-close-and-delivery.md +++ b/docs/changelog-20260721-close-and-delivery.md @@ -76,3 +76,30 @@ - **Revert:** git revert the pinning commits, re-plan/apply on voffice1 (returns MACs to provider-owned -- NOT recommended; reopens the drift hole that caused the 2026-07-21 incident). + +## 5. dc-rack-net.sh: D-131 sub-1 delivery (forwarder + rack legs, repo-carried) + +- D-131 sub-1 RULED this session (standing per-DC pattern; ruling commit + fad3007), then delivered: NEW scripts/dc-rack-net.sh + -- site-keyed (dc0 rows MEASURED), runs ON the rack via piped + ssh. Owns: -rack-legs.service oneshot (re-adds 10.12.8.2/22 + + 10.12.4.2/22 + the 10.12.8.3/22 forwarder alias on boot -- the D-126 + reboot-loss class), the D-131 dnsmasq forwarder unit+conf (generated, + content-checked by diff in check mode), and a boot helper that + resolves bridges FROM LIBVIRT NET NAMES at runtime (virbrN is + auto-assigned = drifting ID; measured: dc-planes does NOT pin bridge + names). Region-side MAAS subnet setting stays a runbook step (same + split as site-headend-install's DHCP note). +- Delta from the live hand-placed artifacts (intentional, minimal): the + legs move from bare `ip addr` (reboot-lost) into the oneshot unit; the + dns unit gains Requires/After on the legs unit instead of its own + ExecStartPre alias hack; config content otherwise identical. +- NEW tests/dc-rack-net harness (14 cases incl. no-virbrN-literal, + MEASURED-tag discipline, read-only check path). Gauntlet 73 -> 74 ALL + GREEN; repo-lint 0 fail. +- Rack install is a gated mutation (operator approval pending at time of + this entry); check EXIT 0 capture cited from CURRENT-STATE when run. +- **Revert:** git rm the script + harness; on the rack (if installed): + systemctl disable --now dc0-rack-legs dc0-node-dns; rm the four + installed files; re-add the legs + old unit per changelog-20260721- + commissioning-diag item 4's revert. diff --git a/scripts/dc-rack-net.sh b/scripts/dc-rack-net.sh new file mode 100755 index 0000000..c1c6273 --- /dev/null +++ b/scripts/dc-rack-net.sh @@ -0,0 +1,210 @@ +#!/usr/bin/env bash +# scripts/dc-rack-net.sh -- DC rack network persistence: +# rack bridge legs + the node-facing DNS forwarder (D-131 sub-1, RULED +# 2026-07-21: the forwarder is the STANDING per-DC pattern; this script is its +# repo-carried delivery, folding in rack-legs persistence per D-128 register +# item 20). +# +# RUNS ON THE DC RACK HOST (e.g. vvr1-dc0), not on vcloud/voffice1. Invoke +# from a repo host over ssh, piping the script (no repo clone needed there): +# +# check: ssh -i -J voffice1 @172.31.0.2 \ +# 'sudo bash -s' -- check dc0 < scripts/dc-rack-net.sh +# install: same with 'install dc0' (idempotent; safe to re-run). +# +# WHAT IT OWNS (site-keyed table below; every value MEASURED, never inferred): +# 1. -rack-legs.service -- oneshot, After=libvirtd: re-adds the rack's +# bridge IP legs on every boot. WHY: these addresses were added with bare +# `ip addr` at standup (2026-07-20/21 session changelogs) and are DROPPED +# on reboot -- same failure class as the D-126 office1 base leg. The rack's +# MAAS DHCP (metal-admin leg) and edge-LAN reach (provider-public leg) +# silently die with them. +# 2. -node-dns.service + /etc/dnsmasq--node.conf -- the D-131 +# forwarder: dnsmasq on a metal-admin alias, no-resolv, forwarding to the +# region's authoritative BIND over the rack's own transit connection +# (OUTPUT traffic only -- the SEC-010 FORWARD-drop is untouched). WHY: the +# MAAS 3.7 rack-only agent resolver SERVFAILs ALL queries on an +# internet-isolated rack, including its own authoritative maas-internal +# zone (walks public root hints, ignores resolv.conf; LP report queued), +# so commissioning cloud-config-url never resolves. Full chain: +# docs/audit/commissioning-diag-20260721.txt. +# 3. /usr/local/sbin/-rack-net-apply -- the boot-time helper both units +# use. It resolves bridge devices FROM LIBVIRT NETWORK NAMES at runtime +# (virsh net-dumpxml): the virbrN names are libvirt AUTO-ASSIGNED and +# would drift if a plane network were ever recreated -- a network NAME is +# the stable identity, a virbrN literal is exactly the drifting-ID class +# hard rule 3 forbids. No virbr literal appears in this file. +# +# REGION-SIDE COMPANION (runbook step, NOT this script -- same split as +# site-headend-install.sh's DHCP note): the MAAS metal-admin subnet must carry +# dns_servers= + allow_dns=false (applied + proven 2026-07-21). +# +# EXIT: 0 ok | 1 check failed | 2 bad args/unknown site | 4 install failed. +# ASCII + LF only. +set -uo pipefail + +MODE="${1:-}"; SITE="${2:-}" +case "$MODE" in check|install) ;; *) + echo "usage: dc-rack-net.sh " >&2; exit 2 ;; esac + +# --------------------------------------------------------------------------- +# Site table. ADD A SITE ONLY WITH MEASURED VALUES (hard rule 2): every leg row +# cites where its value was measured. The harness rejects rows without a +# MEASURED tag. +# --------------------------------------------------------------------------- +case "$SITE" in + dc0) + # MEASURED 2026-07-21 on vvr1-dc0 (ip -4 -o addr / virsh net-list; session + # changelog 2026-07-21 + adjudication commissioning-diag-20260721.txt): + # vr1-dc0-metal-admin 10.12.8.2/22 rack MAAS/DHCP leg + # vr1-dc0-metal-admin 10.12.8.3/22 node-DNS forwarder listen alias + # vr1-dc0-provider-public 10.12.4.2/22 edge-LAN leg (edge gw 10.12.4.1) + LEGS="vr1-dc0-metal-admin=10.12.8.2/22 +vr1-dc0-metal-admin=10.12.8.3/22 +vr1-dc0-provider-public=10.12.4.2/22" + DNS_LISTEN="10.12.8.3" # MEASURED: live forwarder listen address + DNS_UPSTREAM="10.10.0.20" # MEASURED: region BIND (voffice1) over transit + ;; + *) echo "FAIL: unknown site '$SITE' -- add a MEASURED row block first" >&2; exit 2 ;; +esac + +HELPER="/usr/local/sbin/${SITE}-rack-net-apply" +LEGS_UNIT="/etc/systemd/system/${SITE}-rack-legs.service" +DNS_CONF="/etc/dnsmasq-${SITE}-node.conf" +DNS_UNIT="/etc/systemd/system/${SITE}-node-dns.service" + +# --------------------------------------------------------------------------- +# Generated file contents (single source of truth for check AND install). +# --------------------------------------------------------------------------- +gen_helper() { + cat </dev/null 2>&1; then say "OK $1 matches"; else + miss "$1 DIFFERS from generated content"; fi +} + +do_check() { + check_file "$HELPER" gen_helper + check_file "$LEGS_UNIT" gen_legs_unit + check_file "$DNS_CONF" gen_dns_conf + check_file "$DNS_UNIT" gen_dns_unit + for u in "${SITE}-rack-legs" "${SITE}-node-dns"; do + [ "$(systemctl is-enabled "$u" 2>/dev/null)" = "enabled" ] \ + && say "OK $u enabled" || miss "$u not enabled" + done + systemctl is-active --quiet "${SITE}-node-dns" \ + && say "OK ${SITE}-node-dns active" || miss "${SITE}-node-dns not active" + echo "$LEGS" | while IFS='=' read -r net cidr; do + br="$(virsh -c qemu:///system net-dumpxml "$net" 2>/dev/null \ + | sed -n "s/.*bridge name='\([^']*\)'.*/\1/p")" + if [ -z "$br" ]; then echo " MISS libvirt net $net unresolvable"; exit 1; fi + ip -4 -o addr show dev "$br" 2>/dev/null | grep -q " ${cidr%%/*}/" \ + && echo " OK $cidr on $br ($net)" \ + || { echo " MISS $cidr absent on $br ($net)"; exit 1; } + done || FAILED=1 + [ "$FAILED" -eq 0 ] && { echo "dc-rack-net check ($SITE): PASS"; exit 0; } + echo "dc-rack-net check ($SITE): FAIL"; exit 1 +} + +# --------------------------------------------------------------------------- +# install -- idempotent; requires root. +# --------------------------------------------------------------------------- +do_install() { + [ "$(id -u)" = "0" ] || { echo "FAIL: install requires root" >&2; exit 4; } + command -v dnsmasq >/dev/null 2>&1 || { echo "FAIL: dnsmasq not installed" >&2; exit 4; } + umask 022 + gen_helper > "$HELPER" && chmod 755 "$HELPER" || exit 4 + gen_legs_unit > "$LEGS_UNIT" || exit 4 + gen_dns_conf > "$DNS_CONF" || exit 4 + gen_dns_unit > "$DNS_UNIT" || exit 4 + systemctl daemon-reload || exit 4 + systemctl enable --now "${SITE}-rack-legs.service" >/dev/null 2>&1 || exit 4 + systemctl enable "${SITE}-node-dns.service" >/dev/null 2>&1 || exit 4 + systemctl restart "${SITE}-node-dns.service" || exit 4 + echo "dc-rack-net install ($SITE): done -- running check:" + do_check +} + +case "$MODE" in + check) do_check ;; + install) do_install ;; +esac diff --git a/tests/dc-rack-net/run-tests.sh b/tests/dc-rack-net/run-tests.sh new file mode 100755 index 0000000..e73b2c7 --- /dev/null +++ b/tests/dc-rack-net/run-tests.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# tests/dc-rack-net/run-tests.sh -- guard for scripts/dc-rack-net.sh (D-131 +# sub-1 standing per-DC pattern + D-128 item 20 rack-legs persistence). +# Static: syntax, the dc0 identity values (measured 2026-07-21 -- changing one +# silently breaks node DNS or rack addressing), stable-identity keying (libvirt +# net names, never virbrN literals), forwarder confinement (no-resolv, single +# upstream), and the MEASURED-tag discipline on site rows. +# Exit: 0 all pass | 1 any case failed. ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SCRIPT="$(cd "$HERE/../.." && pwd)/scripts/dc-rack-net.sh" +PASS=0; FAIL=0 +ok(){ echo " PASS $1"; PASS=$((PASS+1)); } +no(){ echo " FAIL $1"; FAIL=$((FAIL+1)); } + +[ -f "$SCRIPT" ] && ok "T1 script present" || { no "T1 script present"; echo; echo "dc-rack-net: $PASS passed, $((FAIL)) failed"; exit 1; } +bash -n "$SCRIPT" 2>/dev/null && ok "T2 bash -n clean" || no "T2 bash -n clean" + +# T3-T5: dc0 identity values exactly as measured 2026-07-21 +grep -q 'vr1-dc0-metal-admin=10.12.8.2/22' "$SCRIPT" \ + && ok "T3 dc0 rack MAAS/DHCP leg 10.12.8.2/22 on metal-admin" \ + || no "T3 dc0 rack MAAS/DHCP leg 10.12.8.2/22 on metal-admin" +grep -q 'vr1-dc0-metal-admin=10.12.8.3/22' "$SCRIPT" \ + && grep -q 'DNS_LISTEN="10.12.8.3"' "$SCRIPT" \ + && ok "T4 dc0 forwarder alias + listen 10.12.8.3" \ + || no "T4 dc0 forwarder alias + listen 10.12.8.3" +grep -q 'vr1-dc0-provider-public=10.12.4.2/22' "$SCRIPT" \ + && grep -q 'DNS_UPSTREAM="10.10.0.20"' "$SCRIPT" \ + && ok "T5 dc0 edge-LAN leg + region upstream" \ + || no "T5 dc0 edge-LAN leg + region upstream" + +# T6: stable-identity keying -- bridges resolved from libvirt net names; NO +# virbrN literal anywhere (the auto-assigned name is a drifting ID) +grep -q 'net-dumpxml' "$SCRIPT" && ! grep -q 'virbr[0-9]' "$SCRIPT" \ + && ok "T6 bridges resolved from net names; no virbrN literal" \ + || no "T6 bridges resolved from net names; no virbrN literal" + +# T7-T9: forwarder confinement (the D-131 shape: closed resolver, one upstream, +# OUTPUT-only traffic -- SEC-010 depends on it staying a forwarder, not a router) +grep -q 'no-resolv' "$SCRIPT" && ok "T7 dnsmasq no-resolv" || no "T7 dnsmasq no-resolv" +grep -q 'bind-interfaces' "$SCRIPT" && ok "T8 dnsmasq bind-interfaces (alias only)" || no "T8 dnsmasq bind-interfaces (alias only)" +grep -q 'server=\${DNS_UPSTREAM}' "$SCRIPT" \ + && ok "T9 single templated upstream (region BIND)" \ + || no "T9 single templated upstream (region BIND)" + +# T10: MEASURED-tag discipline -- the dc0 site block carries MEASURED citations +SITE_BLOCK="$(awk '/^ dc0\)/{f=1} f{print} f&&/;;/{exit}' "$SCRIPT")" +grep -q 'MEASURED' <<<"$SITE_BLOCK" \ + && ok "T10 dc0 site rows cite MEASURED provenance" \ + || no "T10 dc0 site rows cite MEASURED provenance" + +# T11: unknown site refuses (hard rule 2 -- no inferred rows) +OUT="$(bash "$SCRIPT" check nosuchsite 2>&1)"; RC=$? +[ "$RC" -eq 2 ] && grep -q 'MEASURED' <<<"$OUT" \ + && ok "T11 unknown site exits 2 and demands measured rows" \ + || no "T11 unknown site exits 2 and demands measured rows (rc=$RC)" + +# T12: bad mode refuses +bash "$SCRIPT" frobnicate dc0 >/dev/null 2>&1; RC=$? +[ "$RC" -eq 2 ] && ok "T12 bad mode exits 2" || no "T12 bad mode exits 2 (rc=$RC)" + +# T13: units are ordered -- dns unit Requires + After the legs unit (the alias +# must exist before dnsmasq binds it) +grep -q 'Requires=\${SITE}-rack-legs.service' "$SCRIPT" \ + && grep -q 'After=\${SITE}-rack-legs.service' "$SCRIPT" \ + && ok "T13 dns unit Requires/After legs unit" \ + || no "T13 dns unit Requires/After legs unit" + +# T14: check path never mutates -- no ip/systemctl mutation verbs inside do_check +CHECK_BLOCK="$(awk '/^do_check\(\)/{f=1} f{print} f&&/^}/{exit}' "$SCRIPT")" +grep -qE 'addr (add|replace|del)|systemctl (enable|restart|start|disable)|daemon-reload|> *"?\$' <<<"$CHECK_BLOCK" \ + && no "T14 do_check is read-only" \ + || ok "T14 do_check is read-only" + +echo; echo "dc-rack-net: $PASS passed, $FAIL failed" +[ "$FAIL" -eq 0 ]