diff --git a/docs/v1-redeploy-changelog.md b/docs/v1-redeploy-changelog.md index a86b2ab..f342698 100644 --- a/docs/v1-redeploy-changelog.md +++ b/docs/v1-redeploy-changelog.md @@ -1087,6 +1087,51 @@ its own `machines:` 8-11, so 8/9/10 are internally consistent. The live 0-3 numbering was a deploy-time artifact; the bundle correctly uses 8-11. +### 2026-06-30 -- Phase-04 executed (network carve + internal-cert SAN gate); DOCFIX-059/060 + +PHASE-04 (network carve) -- PASS: +- Step 4.1 provider-ext + provider-ext-fip created idempotently (phase-04-network-create.sh); EXIT gate + PASS via phase-04-network-verify.sh (external/flat/physnet1/not-shared; subnet 10.12.4.0/22, gw + 10.12.4.1, no-dhcp, FIP pool 10.12.5.0-10.12.7.254). As-built this deploy: provider-ext = a4e1a7fa-..., + provider-ext-fip = f66e5bc5-... (runbook as-built refreshed). + +DOCFIX-059 -- internal-cert SAN gate + the VANTAGE correction (the substantive finding): +- The prompt's phase-04 item "confirm internal certs carry 10.12.12.5x SANs" was NOT implemented by any + committed artifact. Added scripts/phase-04-internal-cert-san-verify.sh (+ tests/phase-04-internal-cert-san/) + and runbook Step 4.2. +- VANTAGE (load-bearing): metal-internal (10.12.12.0/22, VID 103) is an ISOLATED service plane (D-052); the + jumphost is NOT on it. An s_client from the jumphost to 10.12.12.x TIMES OUT / conn-errors, and an + un-hardened check mislabels that as "no IP-SAN" -- a FALSE negative (observed live this session). The gate + must probe FROM a unit ON the plane (keystone/leader) via juju exec. Confirmed live: all 11 internal https + endpoints carry their own 10.12.12.5x IP-SAN (keystone/glance/nova/neutron/cinderv3/placement/barbican/ + octavia/magnum/swift/s3). Internal TLS is correct; the earlier failures were purely vantage. +- HARDENING in the gate: (a) every probe is timeout-bounded (an unbounded s_client hangs ~127s on a filtered + VIP -- proven at 6.02s vs 127s), classified TIMEOUT/CONN-ERR distinctly from a real NO-SAN; (b) non-https + endpoints (the plain-HTTP glance-simplestreams image-stream) are SKIPPED (no cert). Test covers PASS / + SKIP-http / NO-SAN / NO-CERT (fake openstack/juju + real jq; run on the jumphost). + +DOCFIX-060 -- phase-04-network-carve.md drift (the script was right; the md lagged): +- Inline Step 4.1 used the hardcoded `maas admin subnet read 1` -- a post-D-052 landmine (subnet ids drift + across cutovers). Corrected to gateway-by-CIDR, matching phase-04-network-create.sh (DOCFIX-047). +- IPAM reference carried the pre-D-052 single "Metal 10.12.8.0/22 = internal/admin VIPs" model. Corrected to + the D-052/053 split: metal-admin 10.12.8 (admin VIPs .8.5x + PXE), metal-internal 10.12.12 (VID 103, + internal VIPs .12.5x + all service east-west). +- Added a CANONICAL EXECUTION note (D-056) pointing to the three phase-04 scripts; refreshed as-built IDs. + +PROCESS lessons (recorded; no code change): +- PASTE SAFETY: a delivered block whose BEGIN/END label lines (which contain parentheses) were left inside + the fenced code region broke on paste -- bash rejected the parenthesis as an unexpected token. Label lines + carrying parens are NOT comments. RULE: put only valid bash inside a fenced block; labels go as # comments + or prose -- and run bash -n on EVERY delivered block first (that was the miss). The runbooks' own + bold-label convention (labels OUTSIDE the fence) is correct and unaffected. +- NETWORK-PROBE TIMEOUT: any s_client/curl/nc in a runbook step must be timeout-bounded with an explicit + timeout branch -- an unbounded probe is not acceptable in a deterministic gate. + +PHASE-05 (octavia) -- IN PROGRESS at handoff: config gate clear (retrofit use-internal-endpoints=true, +image-format=raw, amp-image-tag=octavia-amphora on both sides); octavia blocked, charm-octavia resources +0/0/0; PRE gate PROCEED. Step 5.1 configure-resources running (--wait=20m; do NOT re-fire on wait-timeout). + ### Next-free numbers -Design decision: D-063. Doc fix: DOCFIX-059. (D-061 teardown, D-062 mysql; DOCFIX-057 old-teardown -deprecation, DOCFIX-058 phase-03 3.3 HTTP-upstream both recorded above.) +Design decision: D-063. Doc fix: DOCFIX-061. (DOCFIX-059 internal-cert SAN gate, DOCFIX-060 phase-04 md +drift both recorded above; D-061 teardown, D-062 mysql; DOCFIX-057 old-teardown deprecation, DOCFIX-058 +phase-03 3.3 HTTP-upstream recorded earlier.) diff --git a/runbooks/phase-04-network-carve.md b/runbooks/phase-04-network-carve.md index fd1bd3e..dc897b5 100644 --- a/runbooks/phase-04-network-carve.md +++ b/runbooks/phase-04-network-carve.md @@ -27,11 +27,20 @@ - 10.12.4.64 - 10.12.4.254 host + container primaries (MAAS auto-static) - 10.12.5.0 - 10.12.7.254 FIP pool / ext_net allocation_pool (this phase's subnet) -- MAAS RESERVED -Metal 10.12.8.0/22 (role Metal; charm control plane + internal VIPs): -- 10.12.8.2 - 10.12.8.63 internal/admin API HA VIPs (front-loaded /26) -- MAAS RESERVED +Metal-admin 10.12.8.0/22 (role metal-admin; UNTAGGED; operator/MAAS/admin API -- D-052/053): +- 10.12.8.2 - 10.12.8.63 ADMIN API HA VIPs (front-loaded /26) -- MAAS RESERVED (admin endpoint only) - 10.12.8.64 - 10.12.8.254 host + container primaries (incl single-unit svc endpoints, e.g. radosgw) - 10.12.9.0 - 10.12.11.254 MAAS PXE/enlistment DHCP (dynamic; iprange id 1) +Metal-internal 10.12.12.0/22 (role metal-internal; TAGGED VID 103; all service east-west -- D-052/053): +- 10.12.12.2 - 10.12.12.63 INTERNAL API HA VIPs (front-loaded /26) -- MAAS RESERVED. Internal endpoints + land here (e.g. keystone .12.50); confirm the certs' IP-SANs FROM a unit ON + this plane, never from the jumphost (Step 4.2, DOCFIX-059). +- 10.12.12.64 - 10.12.12.254 host + container primaries on the internal plane + +DOCFIX-060: the pre-D-052 single "Metal 10.12.8.0/22 = internal/admin VIPs" model is SUPERSEDED -- +admin VIPs are on metal-admin (.8.5x), internal VIPs on metal-internal (.12.5x), split per D-052/D-053. + KI-P3-001 invariant: on every space carrying juju VIPs (provider AND metal), the VIP block is MAAS-reserved and DISTINCT from the primary range and any neutron allocation_pool, so a MAAS auto-static primary can never land on a configured VIP. @@ -67,6 +76,11 @@ - `> CAUTION:` -- marks a destructive, secret-handling, or irreversible step. +CANONICAL EXECUTION (D-056): the reviewed, hardened scripts are the execution path -- +`scripts/phase-04-network-verify.sh` (PRE/POST gates), `scripts/phase-04-network-create.sh` +(the idempotent create below), and `scripts/phase-04-internal-cert-san-verify.sh` (Step 4.2, +DOCFIX-059). The inline blocks below document what they do; prefer running the scripts. + ## Step 4.1 -- Create the external provider network (B29; idempotent) `--external` but NOT `--share` (usable as router gateway + FIP source, but tenants cannot attach instance ports to the provider segment -- Option B @@ -80,7 +94,8 @@ ```bash # RUN: jumphost (MAAS profile is 'admin'; never run 'maas list' -- it prints the API key, DOCFIX-016) maas admin ipranges read | jq -r '.[] | select(.type=="reserved") | "\(.start_ip)-\(.end_ip) subnet=\(.subnet.id) [\(.comment)]"' -# expect a reserved 10.12.5.0-10.12.7.254 on subnet id 1 (provider); + the front-loaded VIP /26 reservations. +# expect a reserved 10.12.5.0-10.12.7.254 on the PROVIDER subnet (resolve its id BY CIDR, not literal 1 -- +# subnet ids drift across cutovers, DOCFIX-047); + the front-loaded VIP /26 reservations. ``` Create (idempotent `( set -e )`; dynamic gateway; tags applied via `set`, not an inline `--tag` flag): @@ -91,7 +106,7 @@ ( set -e PHYSNET=physnet1; EXT_NET=provider-ext; EXT_SUBNET=provider-ext-fip EXT_CIDR=10.12.4.0/22; FIP_START=10.12.5.0; FIP_END=10.12.7.254 - GW=$(maas admin subnet read 1 | jq -r '.gateway_ip') # dynamic; never hardcode .1 + GW=$(maas admin subnets read | jq -r '.[] | select(.cidr=="10.12.4.0/22") | .gateway_ip') # DOCFIX-047: discover BY CIDR; never 'maas admin subnet read 1' (subnet ids drift post-D-052) [ "$GW" = "10.12.4.1" ] || { echo "GATE FAIL: MAAS provider gateway='$GW' (expected 10.12.4.1)"; exit 1; } echo "[OK] gateway $GW" if openstack network show "$EXT_NET" -f value -c id >/dev/null 2>&1; then @@ -122,16 +137,40 @@ --- +## Step 4.2 -- Internal-cert SAN check (DOCFIX-059; read-only; run FROM a unit on metal-internal) +Confirm every INTERNAL keystone-catalog endpoint's TLS cert carries its own metal-internal VIP IP as an +IP-SAN. Internal certs here are IP-based (no FQDN SAN -- D-019 / D-021), so service-to-service TLS on +metal-internal validates only if the internal VIP IP is present in the cert's subjectAltName. + +VANTAGE (load-bearing -- DOCFIX-059): metal-internal (10.12.12.0/22, VID 103) is an ISOLATED service +plane (D-052). The jumphost is NOT on it, so an s_client from the jumphost to 10.12.12.x TIMES OUT and +reports false "missing SAN" negatives. This check probes FROM a unit ON the plane (default +keystone/leader) via `juju exec`; each probe is `timeout`-bounded, and non-https endpoints (the plain-HTTP +glance-simplestreams image-stream) are SKIPPED. NEVER run an internal-cert check from the jumphost. + +**CHECK (read-only) -- jumphost** +```bash +source ~/admin-openrc +bash scripts/phase-04-internal-cert-san-verify.sh # optional args: [PROBE_UNIT] [MODEL] +``` +**GATE:** `Summary: PASS` -- every internal https endpoint OK (its own 10.12.12.5x IP is in the cert SAN). +A `NO-SAN` (cert present, missing its IP) or `NO-CERT` (no cert even from the on-plane unit) is a HOLD -- +investigate before proceeding. + +--- + ## EXIT GATE (phase-04 complete) - `provider-ext` (external, flat/physnet1, not shared) + `provider-ext-fip` (full /22, FIP allocation pool, no-dhcp) present and tagged role=provider. +- Internal-cert SAN check (Step 4.2) `PASS` -- every internal https endpoint's cert carries its + own metal-internal IP-SAN (run FROM a unit on the plane, DOCFIX-059). - FIP allocation + tenant router gateways are now possible (needed by phase-06 mgmt VM FIP, phase-08 cluster FIPs + LB validation). ## As-built reference (object IDs regenerate per deploy -- old IDs are dead post-teardown, not a discrepancy) -- network provider-ext = 0d00ddc1-d2bf-4849-a087-14c07d77f167 (06-03 snapshot: 70b34bb2-...) +- network provider-ext = a4e1a7fa-dedf-4256-8437-36582e857d7c (2026-06-30; 06-03 snapshot: 70b34bb2-...) (external, flat, physnet1, shared=false, role=provider) -- subnet provider-ext-fip = d27f196c-a2d9-4bb9-99f3-bcb8caea3165 (06-03 snapshot: e3afcbae-...) +- subnet provider-ext-fip = f66e5bc5-b2a3-446b-bd4c-9005515f23a8 (2026-06-30; 06-03 snapshot: e3afcbae-...) (cidr 10.12.4.0/22, gateway 10.12.4.1, enable_dhcp=false, alloc 10.12.5.0-10.12.7.254, tags role=provider + netbox-iprange=10.12.5.0-10.12.7.254) - Live MAAS reservations the IPAM draft + D-003 do NOT yet list (the DRAFT is incomplete, not diff --git a/scripts/phase-04-internal-cert-san-verify.sh b/scripts/phase-04-internal-cert-san-verify.sh new file mode 100644 index 0000000..a77dedc --- /dev/null +++ b/scripts/phase-04-internal-cert-san-verify.sh @@ -0,0 +1,82 @@ +#!/usr/bin/env bash +# scripts/phase-04-internal-cert-san-verify.sh [PROBE_UNIT] [MODEL] +# +# Read-only phase-04 gate (DOCFIX-059): confirm every INTERNAL keystone-catalog +# endpoint's TLS cert carries its own metal-internal VIP IP as an IP-SAN. Internal +# certs on this deploy are IP-based (no FQDN SAN -- D-019 / D-021), so service-to-service +# TLS on metal-internal (10.12.12.0/22) validates only if the internal VIP IP is present +# in the cert's subjectAltName. +# +# VANTAGE (the load-bearing correction -- DOCFIX-059): metal-internal is an ISOLATED +# service plane (D-052); the operator jumphost is NOT on it, so an s_client from the +# jumphost to 10.12.12.x times out / connection-errors and yields FALSE "missing SAN" +# negatives. This gate therefore probes FROM a unit that IS on the plane (default +# keystone/leader) via `juju exec`. NEVER run an internal-cert check from the jumphost. +# +# Each probe is bounded by `timeout` (an unbounded s_client can hang ~127s on a filtered +# VIP) and classified: OK / NO-SAN (cert present but missing its own IP) / NO-CERT (no +# cert returned even from the on-plane unit -> a real reachability/listener fault). Non- +# https catalog endpoints (e.g. the plain-HTTP glance-simplestreams image-stream) carry +# no cert and are SKIPPED. +# +# Requires: jumphost; jq; openstack + juju; admin-openrc sourced (OS_AUTH_URL set). +# Usage: source ~/admin-openrc && scripts/phase-04-internal-cert-san-verify.sh [PROBE] [MODEL] +# Env: TIMEOUT_S (per-probe bound, default 6) +# Exit: 0 every internal https endpoint OK | 1 a NO-SAN/NO-CERT (HOLD) | 2 precondition +# ASCII + LF. + +set -euo pipefail +shopt -s inherit_errexit 2>/dev/null || true +IFS=$'\n\t' + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=scripts/lib-net.sh +. "$SCRIPT_DIR/lib-net.sh" + +PROBE="${1:-keystone/leader}" # any unit ON metal-internal; keystone/leader is on the plane + has openssl +MODEL="${2:-openstack}" +TIMEOUT_S="${TIMEOUT_S:-6}" # bound the per-endpoint TLS probe (a LAN handshake is sub-second) + +# --- preconditions ------------------------------------------------------------------ +need_jq || exit 2 +command -v openstack >/dev/null 2>&1 || { echo "FAIL: openstack client not found" >&2; exit 2; } +command -v juju >/dev/null 2>&1 || { echo "FAIL: juju client not found" >&2; exit 2; } +[ -n "${OS_AUTH_URL:-}" ] || { echo "FAIL: OS_AUTH_URL unset -- 'source ~/admin-openrc' first" >&2; exit 2; } + +EPS_JSON="$(openstack endpoint list --interface internal -f json 2>/dev/null || true)" +printf '%s' "$EPS_JSON" | jq -e 'type=="array"' >/dev/null 2>&1 \ + || { echo "FAIL: 'openstack endpoint list --interface internal -f json' did not return JSON" >&2; exit 2; } + +echo "=== phase-04 internal-cert SAN verify (read-only; probing FROM $PROBE on metal-internal) ===" +echo + +FATAL=0 +while IFS=$'\t' read -r svc url; do + [ -n "$url" ] || continue + case "$url" in + https://*) ;; + *) printf 'SKIP %-12s %s (non-TLS endpoint -- no cert)\n' "$svc" "$url"; continue ;; + esac + hp="${url#https://}"; hp="${hp%%/*}"; host="${hp%%:*}"; esc="${host//./\\.}" + # x509 parse is done REMOTELY (inside the bash -c); only the extracted SAN text returns. + # `|| true` so a nonzero juju/timeout does not trip set -e; empty san -> NO-CERT branch. + san="$(juju exec -m "$MODEL" --unit "$PROBE" -- \ + bash -c "timeout $TIMEOUT_S openssl s_client -connect $hp /dev/null | openssl x509 -noout -ext subjectAltName 2>/dev/null" \ + /dev/null || true)" + if printf '%s' "$san" | grep -qE "IP Address:${esc}(\$|[, ])"; then + printf 'OK %-12s %s\n' "$svc" "$hp" + elif [ -z "$san" ]; then + printf 'NO-CERT %-12s %s (no cert returned from %s)\n' "$svc" "$hp" "$PROBE"; FATAL=$((FATAL + 1)) + else + printf 'NO-SAN %-12s %s (cert present, missing %s)\n' "$svc" "$hp" "$host"; FATAL=$((FATAL + 1)) + printf ' SAN: %s\n' "$(printf '%s' "$san" | tr '\n' ' ' | sed 's/ */ /g')" + fi +done < <(printf '%s' "$EPS_JSON" | jq -r '.[] | select(.URL|startswith("https://")) | "\(.["Service Name"])\t\(.URL)"') + +echo +if [ "$FATAL" -ne 0 ]; then + echo "Summary: HOLD/FAIL -- $FATAL internal endpoint(s) failed the SAN check." + exit 1 +fi +echo "Summary: PASS -- every internal https endpoint cert carries its own metal-internal IP-SAN." +exit 0 diff --git a/tests/phase-04-internal-cert-san/fakebin/juju b/tests/phase-04-internal-cert-san/fakebin/juju new file mode 100644 index 0000000..e01aca5 --- /dev/null +++ b/tests/phase-04-internal-cert-san/fakebin/juju @@ -0,0 +1,19 @@ +#!/usr/bin/env bash +# fake juju for phase-04-internal-cert-san-verify.sh tests. +# Handles 'exec -m MODEL --unit UNIT -- bash -c "... -connect HOST:PORT ..."' by +# returning a canned subjectAltName for the probed HOST:PORT (the real x509 parse runs +# remotely in the script's bash -c; here we emit what that parse would print). +# BARBICAN_MODE env toggles barbican's cert: ok (default) | nosan | nocert. +cmd="$*" +hp="$(printf '%s' "$cmd" | grep -oE '\-connect [0-9.]+:[0-9]+' | awk '{print $2}')" +case "$hp" in + 10.12.12.50:5000) + echo " IP Address:10.12.4.50, IP Address:10.12.8.50, IP Address:10.12.12.50, DNS:juju-a" ;; + 10.12.12.51:9311) + case "${BARBICAN_MODE:-ok}" in + nosan) echo " IP Address:10.12.4.51, IP Address:10.12.8.51, DNS:juju-b" ;; # missing 10.12.12.51 + nocert) : ;; # empty -> NO-CERT + *) echo " IP Address:10.12.4.51, IP Address:10.12.8.51, IP Address:10.12.12.51, DNS:juju-b" ;; + esac ;; +esac +exit 0 diff --git a/tests/phase-04-internal-cert-san/fakebin/openstack b/tests/phase-04-internal-cert-san/fakebin/openstack new file mode 100644 index 0000000..4727128 --- /dev/null +++ b/tests/phase-04-internal-cert-san/fakebin/openstack @@ -0,0 +1,15 @@ +#!/usr/bin/env bash +# fake openstack for phase-04-internal-cert-san-verify.sh tests. +# 'endpoint list --interface internal -f json' -> canned internal catalog. +# Includes one plain-HTTP endpoint (image-stream) that MUST be SKIPPED (no cert). +if [ "${1:-}" = endpoint ] && [ "${2:-}" = list ]; then + cat <<'JSON' +[ + {"Service Name":"keystone","URL":"https://10.12.12.50:5000/v3","Interface":"internal"}, + {"Service Name":"barbican","URL":"https://10.12.12.51:9311","Interface":"internal"}, + {"Service Name":"image-stream","URL":"http://10.12.8.166:8080","Interface":"internal"} +] +JSON + exit 0 +fi +exit 0 diff --git a/tests/phase-04-internal-cert-san/run-tests.sh b/tests/phase-04-internal-cert-san/run-tests.sh new file mode 100644 index 0000000..e85c5b2 --- /dev/null +++ b/tests/phase-04-internal-cert-san/run-tests.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +# tests/phase-04-internal-cert-san/run-tests.sh -- offline regression for +# scripts/phase-04-internal-cert-san-verify.sh (fake openstack/juju + real jq). +# Run on the jumphost (jq present), same as the other phase tests. +set -euo pipefail +IFS=$'\n\t' +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SCRIPTS="$(cd "$HERE/../../scripts" && pwd)" +TARGET="$SCRIPTS/phase-04-internal-cert-san-verify.sh" +BIN="$HERE/fakebin" +command -v jq >/dev/null 2>&1 || { echo "FAIL: jq required (run on the jumphost)" >&2; exit 1; } +[ -f "$TARGET" ] || { echo "FAIL: target missing" >&2; exit 1; } +chmod +x "$BIN"/* 2>/dev/null || true +WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT +rc_all=0 +run() { + local want="$1" re="$2" label="$3"; shift 3 + local rc + set +e + PATH="$BIN:$PATH" OS_AUTH_URL="x" env "$@" bash "$TARGET" >"$WORK/out" 2>&1 + rc=$?; set -e + if [ "$rc" -eq "$want" ] && grep -qE "$re" "$WORK/out"; then + printf ' [OK] %-42s exit %s\n' "$label" "$rc" + else + printf ' [XX] %-42s exit %s (want %s; /%s/)\n' "$label" "$rc" "$want" "$re" + sed 's/^/ /' "$WORK/out"; rc_all=1 + fi +} +echo "=== phase-04-internal-cert-san-verify.sh (fake openstack/juju + real jq) ===" +run 0 'Summary: PASS' "all internal certs carry own IP-SAN" +run 0 'SKIP .* image-stream' "plain-HTTP image-stream is SKIPPED" +run 1 'NO-SAN .* barbican' "barbican cert missing own IP-SAN -> HOLD" BARBICAN_MODE=nosan +run 1 'NO-CERT .* barbican' "barbican returns no cert -> HOLD" BARBICAN_MODE=nocert +echo +[ "$rc_all" -eq 0 ] && echo "ALL PASS" || echo "SOME FAILED" +exit "$rc_all"