diff --git a/creds-manifests/vr1-office1.manifest b/creds-manifests/vr1-office1.manifest new file mode 100644 index 0000000..3b6f8eb --- /dev/null +++ b/creds-manifests/vr1-office1.manifest @@ -0,0 +1,22 @@ +# vr1-office1 creds manifest -- what MUST be consolidated in ~/vr1-office1-creds/ (SEC-009). +# scripts/creds-audit.sh verifies this list against the live folder: presence, mode, +# non-empty. No secret VALUES are read -- only existence, mode, and declared provenance. +# +# fields: +# source = "local" -- generated/held on the jumphost (vcloud) +# = ":" -- MINTED ON A SITE VM; a working copy MUST be pulled into +# the folder at mint time (this is the class SEC-009 missed: +# office1-netbox:/root/netbox-secrets/api.token went +# un-consolidated until it was needed). +# +office1_svc_ed25519 600 local +office1_svc_ed25519.pub 644 local +opnsense-api.txt 600 local +opnsense-root-hash 600 local +opnsense-root-password 600 local +README 600 local +tailscale-authkey.txt 600 local +vr1-netbox.env 600 local +vr1-netbox-sandbox.env 600 office1-netbox:/root/netbox-secrets/api.token +vr1-office1.env 600 local +vr1-stage1.env 600 local diff --git a/docs/changelog-20260715-creds-audit.md b/docs/changelog-20260715-creds-audit.md new file mode 100644 index 0000000..69a6a7f --- /dev/null +++ b/docs/changelog-20260715-creds-audit.md @@ -0,0 +1,60 @@ +# 2026-07-15 -- creds-audit: enforce per-site credential consolidation (SEC-009), so the DC0/DC1 spread cannot recur + +## Why + +SEC-009 established the convention "ALL of a site's secrets live in `~/-creds/` +(0700, files 0600), no loose env files in `~`." But nothing ENFORCED it, and it had +two blind spots that a checklist alone would not catch: + +1. **VM-minted secrets.** A secret generated ON a site VM (office1-netbox's sandbox + NetBox token at `/root/netbox-secrets/api.token`) has no forcing function to get + pulled into the jumphost folder. It went un-consolidated until the C2 fidelity + re-check needed it (SEC-009 addendum, 2026-07-15). +2. **Silent sprawl.** A loose `*.env`/`*.token` in `~` outside a `*-creds/` folder -- + the ORIGINAL SEC-009 exposure -- had no standing detector. + +Standing up DC0 and DC1 multiplies both (each mints MAAS API keys, per-DC NetBox +tokens, node/edge SSH keys, OPNsense creds, tailscale authkeys on multiple VMs). The +operator asked directly how we stop the spread recurring for the DCs. Answer: a +manifest + an audit + a mint-time runbook step. + +## What was built + +- **`creds-manifests/.manifest`** -- declares EXACTLY the secrets a site must + hold: ` `, where source is `local` or `:` + for VM-minted secrets (making the easy-to-miss class explicit and provenance-tracked). + `creds-manifests/vr1-office1.manifest` seeded with 11 entries from the live folder; + it audits CLEAN. +- **`scripts/creds-audit.sh `** -- verifies the live `~/-creds/` against + the manifest: folder is 0700, every entry present with its declared mode and + non-empty; flags any UNDECLARED file in the folder; and scans the home root for + SPRAWL (`*.env`/`*.token`/`*authkey*`/`*appcred*`/`*-cred*.txt` loose outside any + `*-creds/`). **Reads NO secret values -- metadata only** (stat/size), verified by a + harness assertion (T7: no cat/head/tail/od/xxd in the script). Env overrides + `CREDS_ROOT`/`MANIFEST_DIR` make it fully offline-testable. +- **`tests/creds-audit/run-tests.sh`** (7/7) -- synthetic folders under a tmp root: + complete+correct -> CLEAN; missing (VM-sourced) entry -> fail; wrong mode -> fail; + undeclared file -> fail; home-root sprawl -> fail; non-0700 folder -> fail; + metadata-only. Auto-discovered by the gauntlet. +- **Runbook discipline** -- `runbooks/dc-dc-phase3-maas-enlist-deploy.md` GATE now + carries a non-negotiable close-out: consolidate + manifest each secret AT MINT TIME, + and `bash scripts/creds-audit.sh $DC` MUST be CLEAN before a DC is declared done. +- **`docs/security-ledger.md`** -- SEC-009 ENFORCEMENT note recording the above. + +## Verification + +- Harness: `tests/creds-audit/run-tests.sh` -> 7/7 PASS. +- Live: `bash scripts/creds-audit.sh vr1-office1` -> CLEAN (folder 0700, 11/11 entries + present + correct + non-empty, no undeclared files, no sprawl). This also CONFIRMS + the SEC-009 sandbox-token gap is now closed and tool-verified. +- Gauntlet ALL GREEN; repo-lint 0 fail. + +## Revert + +- `git rm scripts/creds-audit.sh tests/creds-audit/run-tests.sh creds-manifests/vr1-office1.manifest docs/changelog-20260715-creds-audit.md` + (remove `creds-manifests/` if now empty). +- `git checkout HEAD -- docs/security-ledger.md runbooks/dc-dc-phase3-maas-enlist-deploy.md` + reverts the SEC-009 enforcement note and the runbook close-out. + +Nothing here reads, moves, or mutates any secret; the audit is read-only metadata +inspection and the manifest holds only filenames/modes/provenance. diff --git a/docs/security-ledger.md b/docs/security-ledger.md index 76ed6a4..8f7329f 100644 --- a/docs/security-ledger.md +++ b/docs/security-ledger.md @@ -39,3 +39,17 @@ session. The `/root/netbox-secrets/` copy remains the source-of-record on the VM; this is a working copy on the jumphost, mirroring how `vr1-netbox.env` holds the upstream token. See `docs/changelog-20260715-fidelity-check-scope-correctness.md`. + +**ENFORCEMENT (2026-07-15) -- so the miss cannot recur, esp. for DC0/DC1.** The convention was never +ENFORCED; a VM-minted secret had no forcing function to get consolidated and no check to catch a miss. +Now built (`docs/changelog-20260715-creds-audit.md`): +- **`creds-manifests/.manifest`** -- declares EXACTLY the secrets a site must hold, each with its + mode and PROVENANCE (`local`, or `:` for VM-minted secrets -- the class that was missed). + `creds-manifests/vr1-office1.manifest` seeded (11 entries) and the live folder audits CLEAN. +- **`scripts/creds-audit.sh `** -- verifies the live `~/-creds/` against the manifest + (presence, 0700 folder / declared file modes, non-empty), flags UNDECLARED files in the folder, and + scans the home root for SPRAWL (loose `*.env`/`*.token`/`*authkey*` outside any `*-creds/`). Reads NO + secret values -- metadata only. Harness `tests/creds-audit/` (7/7). +- **Standing rule:** run `bash scripts/creds-audit.sh vr1-dcN` at the CLOSE of each DC standup, and + consolidate+manifest every secret AT MINT TIME (not "later" -- later is when it is lost). Wired into + the `dc-dc-phase3` runbook. diff --git a/runbooks/dc-dc-phase3-maas-enlist-deploy.md b/runbooks/dc-dc-phase3-maas-enlist-deploy.md index d6afa9a..00cbc70 100644 --- a/runbooks/dc-dc-phase3-maas-enlist-deploy.md +++ b/runbooks/dc-dc-phase3-maas-enlist-deploy.md @@ -491,6 +491,16 @@ second DC; the two DCs' completions are independent (D-100: no shared control plane), so one can be done before the other. +> **CREDS-CONSOLIDATION CLOSE-OUT (SEC-009 enforcement -- non-negotiable).** Standing up a DC MINTS +> secrets on its VMs and on the jumphost -- MAAS API key, per-DC NetBox token (if any), node/edge SSH +> keys, OPNsense root creds, tailscale authkey, Juju/Vault material. **The moment each is minted, +> consolidate it into `~/$DC-creds/` (0700; files 0600) and add it to `creds-manifests/$DC.manifest` +> with its provenance (`local` or `:`).** Do NOT defer this -- "later" is when secrets get +> lost (and after a compaction they are gone). This is the exact class that bit us: office1's sandbox +> NetBox token, minted on the VM at `/root/netbox-secrets/`, went un-consolidated until it was needed +> (SEC-009 addendum). Before declaring this DC done, `bash scripts/creds-audit.sh $DC` MUST report +> **CLEAN** (folder perms, every manifest entry present, no undeclared files, no home-root sprawl). + --- ## Next diff --git a/scripts/creds-audit.sh b/scripts/creds-audit.sh new file mode 100755 index 0000000..009f030 --- /dev/null +++ b/scripts/creds-audit.sh @@ -0,0 +1,81 @@ +#!/usr/bin/env bash +# scripts/creds-audit.sh -- per-site credential-consolidation audit (SEC-009 enforcement). +# +# Verifies a site's consolidated creds folder (~/-creds/) holds EXACTLY the +# secrets its manifest requires -- each present, non-empty, and with the right mode -- +# and that no sensitive file is sprawled loose in the home dir OUTSIDE a *-creds/ +# folder. It reads NO secret values: only existence, mode, size, and declared +# provenance (from creds-manifests/.manifest). +# +# WHY THIS EXISTS. SEC-009 established the "ALL site secrets live in ~/-creds/" +# convention, but nothing ENFORCED it, and a secret MINTED ON A SITE VM +# (office1-netbox:/root/netbox-secrets/api.token) was never pulled into the folder -- +# it went unnoticed until the fidelity re-check needed it. This audit is the forcing +# check: run it at the END of every site standup and periodically. For the DCs, run +# `creds-audit.sh vr1-dc0` / `vr1-dc1` at standup close so the same gap cannot recur. +# +# Usage: +# bash scripts/creds-audit.sh # e.g. vr1-office1 +# Env overrides (used by the harness; not for normal operation): +# CREDS_ROOT parent dir of -creds (default: $HOME) +# MANIFEST_DIR dir holding .manifest (default: /creds-manifests) +set -uo pipefail + +SITE="${1:-}" +[ -n "$SITE" ] || { echo "usage: creds-audit.sh (e.g. vr1-office1)"; exit 2; } + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO="$(cd "$HERE/.." && pwd)" +CREDS_ROOT="${CREDS_ROOT:-$HOME}" +MANIFEST_DIR="${MANIFEST_DIR:-$REPO/creds-manifests}" +CREDS="$CREDS_ROOT/${SITE}-creds" +MANIFEST="$MANIFEST_DIR/${SITE}.manifest" + +F=0 +fail() { echo " [FAIL] $1"; F=$((F+1)); } +okln() { echo " [ok] $1"; } + +[ -f "$MANIFEST" ] || { echo "FAIL: no manifest at $MANIFEST"; exit 1; } +[ -d "$CREDS" ] || { echo "FAIL: no creds folder at $CREDS"; exit 1; } + +echo "=== creds-audit: $SITE ($CREDS) ===" + +# 1. folder must be 0700 (owner-only) +fmode=$(stat -c '%a' "$CREDS") +[ "$fmode" = 700 ] && okln "folder mode 0700" || fail "folder mode $fmode (want 0700)" + +# 2. every manifest entry present, right mode, non-empty +declare -A expected=() +while read -r fname fmode2 fsrc; do + case "$fname" in ''|\#*) continue;; esac + expected["$fname"]=1 + path="$CREDS/$fname" + if [ ! -e "$path" ]; then + fail "MISSING: $fname (provenance: ${fsrc:-?})" + continue + fi + amode=$(stat -c '%a' "$path") + [ "$amode" = "$fmode2" ] || fail "$fname mode $amode (want $fmode2)" + [ -s "$path" ] || fail "$fname is EMPTY" +done < "$MANIFEST" +[ "$F" = 0 ] && okln "all ${#expected[@]} manifest entries present, correct mode, non-empty" + +# 3. no UNDECLARED file in the folder (an undeclared secret sitting here is spread) +for path in "$CREDS"/* "$CREDS"/.[!.]*; do + [ -e "$path" ] || continue + b=$(basename "$path") + [ -n "${expected[$b]:-}" ] || fail "UNDECLARED file in folder: $b (add it to the manifest, or remove it)" +done + +# 4. sprawl: sensitive-looking files loose in CREDS_ROOT, OUTSIDE any *-creds/ folder +shopt -s nullglob +for path in "$CREDS_ROOT"/*.env "$CREDS_ROOT"/.*.env "$CREDS_ROOT"/*appcred* \ + "$CREDS_ROOT"/*-cred*.txt "$CREDS_ROOT"/*authkey* "$CREDS_ROOT"/*.token; do + [ -f "$path" ] || continue + fail "SPRAWL: sensitive file loose in $CREDS_ROOT: $(basename "$path") (belongs in a *-creds/ folder)" +done +shopt -u nullglob + +echo +if [ "$F" = 0 ]; then echo "creds-audit $SITE: CLEAN"; exit 0; fi +echo "creds-audit $SITE: $F problem(s) -- consolidate/fix before proceeding"; exit 1 diff --git a/tests/creds-audit/run-tests.sh b/tests/creds-audit/run-tests.sh new file mode 100755 index 0000000..7318a57 --- /dev/null +++ b/tests/creds-audit/run-tests.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# tests/creds-audit/run-tests.sh +# +# Harness for scripts/creds-audit.sh. Offline: builds synthetic creds folders + +# manifests under a tmp CREDS_ROOT and asserts the audit's verdict. Touches no real +# secret and no real home dir (CREDS_ROOT / MANIFEST_DIR are redirected to tmp). +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO="$(cd "$HERE/../.." && pwd)" +AUDIT="$REPO/scripts/creds-audit.sh" +P=0; F=0 +ok() { echo " [PASS] $1"; P=$((P+1)); } +no() { echo " [FAIL] $1"; F=$((F+1)); } + +[ -f "$AUDIT" ] || { echo "FAIL: $AUDIT not found"; exit 1; } + +# fresh synthetic environment: CREDS_ROOT/-creds + MANIFEST_DIR/.manifest +SITE=testsite +mkenv() { # mkenv -> echoes the tmp root; caller sets CREDS_ROOT/MANIFEST_DIR to it + local root; root="$(mktemp -d)" + mkdir -p "$root/${SITE}-creds" "$root/manifests" + chmod 700 "$root/${SITE}-creds" + printf 'a_secret 600 local\nb_key.pub 644 local\nc.env 600 somevm:/root/x.token\n' \ + > "$root/manifests/${SITE}.manifest" + printf 'secret\n' > "$root/${SITE}-creds/a_secret"; chmod 600 "$root/${SITE}-creds/a_secret" + printf 'pub\n' > "$root/${SITE}-creds/b_key.pub"; chmod 644 "$root/${SITE}-creds/b_key.pub" + printf 'tok\n' > "$root/${SITE}-creds/c.env"; chmod 600 "$root/${SITE}-creds/c.env" + echo "$root" +} +run() { CREDS_ROOT="$1" MANIFEST_DIR="$1/manifests" bash "$AUDIT" "$SITE" >"$1/out" 2>&1; echo $?; } + +echo "=== creds-audit ===" + +# T1: a complete, correct, non-sprawled folder -> CLEAN, exit 0 +R="$(mkenv)"; rc=$(run "$R") +[ "$rc" = 0 ] && grep -q "CLEAN" "$R/out" && ok "T1 complete + correct folder -> CLEAN, exit 0" \ + || { no "T1 a correct folder should pass (rc=$rc)"; sed 's/^/ /' "$R/out"; } +rm -rf "$R" + +# T2: a manifest-required file MISSING (the SEC-009 gap: a VM secret never pulled in) +R="$(mkenv)"; rm -f "$R/${SITE}-creds/c.env"; rc=$(run "$R") +[ "$rc" = 1 ] && grep -q "MISSING: c.env" "$R/out" && ok "T2 a manifested (VM-sourced) secret absent -> MISSING, exit 1" \ + || { no "T2 a missing manifest entry should fail (rc=$rc)"; sed 's/^/ /' "$R/out"; } +rm -rf "$R" + +# T3: wrong permissions on a secret -> FAIL +R="$(mkenv)"; chmod 644 "$R/${SITE}-creds/a_secret"; rc=$(run "$R") +[ "$rc" = 1 ] && grep -q "a_secret mode 644" "$R/out" && ok "T3 a secret with loose mode (644) -> caught, exit 1" \ + || { no "T3 wrong mode should fail (rc=$rc)"; sed 's/^/ /' "$R/out"; } +rm -rf "$R" + +# T4: an UNDECLARED file in the folder -> FAIL (spread: a secret nobody declared) +R="$(mkenv)"; printf 'x\n' > "$R/${SITE}-creds/stray.env"; chmod 600 "$R/${SITE}-creds/stray.env"; rc=$(run "$R") +[ "$rc" = 1 ] && grep -q "UNDECLARED file in folder: stray.env" "$R/out" && ok "T4 an undeclared file in the folder -> caught, exit 1" \ + || { no "T4 an undeclared file should fail (rc=$rc)"; sed 's/^/ /' "$R/out"; } +rm -rf "$R" + +# T5: SPRAWL -- a sensitive file loose in CREDS_ROOT, outside any *-creds/ folder +R="$(mkenv)"; printf 'x\n' > "$R/loose.env"; chmod 600 "$R/loose.env"; rc=$(run "$R") +[ "$rc" = 1 ] && grep -q "SPRAWL: sensitive file loose" "$R/out" && ok "T5 a loose .env in the home root -> SPRAWL, exit 1" \ + || { no "T5 sprawl in the home root should fail (rc=$rc)"; sed 's/^/ /' "$R/out"; } +rm -rf "$R" + +# T6: the folder itself not 0700 -> FAIL +R="$(mkenv)"; chmod 755 "$R/${SITE}-creds"; rc=$(run "$R") +[ "$rc" = 1 ] && grep -q "folder mode 755" "$R/out" && ok "T6 a world-listable creds folder (0755) -> caught, exit 1" \ + || { no "T6 a non-0700 folder should fail (rc=$rc)"; sed 's/^/ /' "$R/out"; } +rm -rf "$R" + +# T7: the audit reads NO secret content -- no content-reading verbs in the script +grep -qE '\b(cat|head|tail|less|more|od|xxd|hexdump)\b' "$AUDIT" \ + && no "T7 the audit contains a content-reading verb -- it must inspect metadata only" \ + || ok "T7 metadata-only: no cat/head/tail/od/xxd in the audit" + +echo +if [ "$F" = 0 ]; then echo "creds-audit: $P/$P PASS"; exit 0; fi +echo "creds-audit: FAILURES: $F (passed $P)"; exit 1