diff --git a/docs/D-068-openbao-assessment-DRAFT.md b/docs/D-068-openbao-assessment-DRAFT.md index e442f32..4a6ec5d 100644 --- a/docs/D-068-openbao-assessment-DRAFT.md +++ b/docs/D-068-openbao-assessment-DRAFT.md @@ -49,13 +49,27 @@ Pro/PCB). Compensating controls: mgmt-plane-only API exposure, audit logging shipped, minimal policies, confirm userpass/cert auth methods and plugin registration absent, unseal/root discipline (already the norm -- D-069). - Hard re-decision deadline proposed: 2027-04. Two measured follow-ups (log - only, do not execute): (a) whether the reactive charm's vault SNAP CHANNEL - can be raised above 1.8 toward 1.14.x, the last MPL payload -- would de-EOL - the payload WITHOUT touching relation topology; Canonical support for this - UNVERIFIED; needs on-host measurement (charm config + snap tracks), a - read-only look; (b) track Sunbeam feature parity -- the durable exit from - the V0 certificates world is a PLATFORM decision, not a vault swap. + Hard re-decision deadline proposed: 2027-04. Two follow-ups; (a) is now + MEASURED (read-only, 2026-07-06, evidence + ~/openstack-baseline/vault-config-20260706.yaml + vault-snap-tracks-20260706.txt): + the reactive charm EXPOSES a `channel` config (currently 1.8/stable; its own + description warns a change seals ALL vault units via the snap refresh -- + manual 3-of-5 unseal is our norm, so that is a planned-window event, not a + blocker). The Canonical vault snap on the live unit publishes tracks + 1.1-1.12, then 1.15-1.20 and 2.0: **there is NO 1.13 or 1.14 track** -- the + last MPL-licensed payload reachable via the charm is **1.12.11** + (1.12/stable), not 1.14.x. So the in-charm payload-bump option is: + 1.8.8 -> 1.12.11 (stays MPL; still EOL upstream, but 4 minors newer), or + 1.15.6+ (BUSL -- licensing review required). Remaining UNVERIFIED before any + such move: charm-vs-newer-payload API compatibility (does the reactive charm + drive 1.12 correctly -- Canonical test coverage unknown), storage-format + stepping across 4 minors on vault-on-mysql, and whether MySQL backend + behavior changed in 1.9-1.12. This is a rehearse-first D-NNN of its own if + pursued; NOT a routine refresh. (b) track Sunbeam feature parity -- the + durable exit from the V0 certificates world is a PLATFORM decision, not a + vault swap. Also verified live 2026-07-06: the charm's Vault-listener TLS + options exist and are empty (ssl-ca / ssl-cert / ssl-chain / ssl-key) -- + the concrete D-068 item 2 mechanism, no fabricated option names. ## Decision hooks for the operator (nothing ruled yet) diff --git a/docs/session-ledger.md b/docs/session-ledger.md index b5ac555..f4bdda9 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -31,8 +31,8 @@ - **OPEN security rows:** SEC-001 (rotate at v1 close, re-ruled 2026-07-06), SEC-003 (custodians + second-person unseal -- DEFERRED; keeps d011-06 MANUAL), SEC-004 (flip repo to private at v1 close -- deferral reaffirmed). -- **Next-free numbers:** D = 074, DOCFIX = 094, BUNDLEFIX = 012. - (Scan and the addendum-25 changelog next-free pointer agree.) +- **Next-free numbers:** D = 074, DOCFIX = 095, BUNDLEFIX = 012. + (Scan and the addendum-26 changelog next-free pointer agree.) --- @@ -225,6 +225,16 @@ - REPO-SIDE QUEUE now EMPTY except: H4 policy-drift script (main-chat backlog item 6 -- coordinate before building) and the client-docs sweep-on-change rule. EVERYTHING ELSE waits on the operator-present live window (the do-doc). +- DELIVERED (operator-away batch part 2, addendum 26): DOCFIX-094 stderr-separation + fixes (cloud-assert A5 FAIL-OPEN closed + harness proof; onboard resolvers; + offboard blind-sweep guard exit 20; 3 runbook paste blocks; acceptance:43 rides + its future harness). Vault measurements: charm `channel` + ssl-* options + verified; snap has NO 1.13/1.14 track -- max MPL payload in-charm = 1.12.11; + OpenBao memo updated. Orphan-trustee sweep draft NOT started (queued). +- BLOCKER: juju controller macaroon EXPIRED ~17:10 2026-07-06 -- all juju cmds + prompt for password. OPERATOR: `juju login -u jessea123` at next presence. + Blocked reads queued: keystone policy.json PENDING-LIVE-READ trio; barbican + host_href look. (Vault reads above completed BEFORE expiry.) diff --git a/docs/v1-redeploy-changelog.md b/docs/v1-redeploy-changelog.md index 2819cf1..969692c 100644 --- a/docs/v1-redeploy-changelog.md +++ b/docs/v1-redeploy-changelog.md @@ -2188,3 +2188,41 @@ REVERT: git rm runbooks/d011-batch3-window-DRAFT.md docs/D-068-openbao-assessment-DRAFT.md; git checkout HEAD~ -- runbooks/tenant-onboarding-v2-DRAFT.md docs/handoff-20260705-open-items.md docs/session-ledger.md docs/v1-redeploy-changelog.md Next-free after this push (per scan, keep this line unwrapped): D-074, DOCFIX-094, BUNDLEFIX-012. + +### 2026-07-06 (addendum 26, jumphost stream) -- DOCFIX-094 stderr-separation fixes; vault snap-track measurements; juju auth expiry + +Operator-away batch, part 2. +- **DOCFIX-094 (fixes the addendum-25 inventory):** structured captures now take + stdout ONLY, stderr to a tempfile surfaced on failure. + - scripts/cloud-assert.sh A5/A6 (3 sites): the A5 compute-service variant was + genuinely FAIL-OPEN (jq ... 2>/dev/null || true -> any stderr noise or API + error silently printed "all enabled compute services up"). Now fail-closed + with the stderr head as evidence. Harness +2 cases (T11 noise->PASS proves + separation; T12 erroring service-list->FAIL proves the fail-open is closed); + 12/12 green. + - scripts/tenant-onboard.sh stage0 + stage5 image resolvers (stage7 pattern). + - scripts/tenant-offboard.sh trust capture: parse failure now PRECOND exit 20 + ("sweep would be blind") instead of silently emptying the trust sweep; + stderr NOTE line added. Harness 7/7 green (no fixture change needed). + - runbooks/tenant-onboarding-v2-DRAFT.md: 3 paste blocks drop the 2>&1 merge + (stderr goes to the operator terminal, parser gets clean stdout). + - NOT fixed: scripts/tenant-acceptance.sh:43 -- no harness exists yet (script + backlog item 8, main-chat); fix rides that harness's delivery. LOGGED. +- **Read-only live measurements (before auth expiry; evidence in + ~/openstack-baseline/ vault-config-20260706.yaml + vault-snap-tracks-20260706.txt):** + reactive vault charm HAS `channel` config (1.8/stable; description warns a + change SEALS all units -- planned-window event under our manual-unseal norm); + Canonical vault snap tracks measured on the live unit: 1.1-1.12, 1.15-1.20, + 2.0 -- **NO 1.13 or 1.14 track**; max MPL-licensed payload reachable in-charm + is **1.12.11**, not 1.14.x. Listener TLS options verified present and empty: + ssl-ca/ssl-cert/ssl-chain/ssl-key (the D-068 item 2 mechanism, names now + measured not assumed). docs/D-068-openbao-assessment-DRAFT.md updated with + these measurements (follow-up (a) is now MEASURED). +- **BLOCKER (operator action):** the juju controller macaroon EXPIRED mid-session; + every juju command now prompts for the jumphost user's controller password + (interactive). Operator: run `juju login -u jessea123` at next presence. + Blocked until then: keystone policy.json [PENDING-LIVE-READ] trio read; + barbican host_href config look. Both queued. +REVERT: git checkout HEAD~ -- scripts/cloud-assert.sh scripts/tenant-onboard.sh scripts/tenant-offboard.sh tests/cloud-assert/run-tests.sh runbooks/tenant-onboarding-v2-DRAFT.md docs/D-068-openbao-assessment-DRAFT.md docs/session-ledger.md docs/v1-redeploy-changelog.md + +Next-free after this push (per scan, keep this line unwrapped): D-074, DOCFIX-095, BUNDLEFIX-012. diff --git a/runbooks/tenant-onboarding-v2-DRAFT.md b/runbooks/tenant-onboarding-v2-DRAFT.md index 3380e68..ccb046e 100644 --- a/runbooks/tenant-onboarding-v2-DRAFT.md +++ b/runbooks/tenant-onboarding-v2-DRAFT.md @@ -120,14 +120,14 @@ # public Magnum-ready image (kube_version + os_distro + public) IMG=$(openstack image list --public -f value -c ID -c Name &1 | awk '/kube/{print $1;exit}') -openstack image show "$IMG" -f json &1 | python3 -c ' +openstack image show "$IMG" -f json =2 vcpu, >=2048 MB) -openstack flavor list --public -f json &1 | python3 -c ' +openstack flavor list --public -f json =2 and (f.get("RAM") or 0)>=2048] print("magnum-capable flavors:",[f["Name"] for f in ok]); sys.exit(0 if ok else 1)' || fail=1 @@ -356,7 +356,7 @@ source ~/admin-openrc P=$(openstack project show "${CLIENT}-prod" --domain "$CLIENT" -f value -c id &1) RID=$(openstack router list --project "$P" -f value -c ID &1 | head -1) -openstack router show "$RID" -f json &1 | python3 -c ' +openstack router show "$RID" -f json &1; } # capture-then-test everywhere (SIGPIPE rule) +# Structured (-f json) captures take stdout ONLY -- merged stderr lands at char 0 +# and kills the parse (DOCFIX-094; the A5-services variant was silently fail-OPEN). +OERR="$(mktemp)"; trap 'rm -f "$OERR"' EXIT +O() { openstack "$@" "$OERR"; } +oerr() { head -1 "$OERR" 2>/dev/null || true; } WORST=0 fail() { echo " [FAIL] $*"; WORST=1; } warn() { echo " [WARN] $*"; [ "$WORST" -ne 1 ] && WORST=2; } @@ -117,17 +122,19 @@ warn "A5-A7 HELD: no admin scope in env (source ~/admin-openrc, then re-run)" else echo "================ A5: compute plane ================" - H=$(openstack hypervisor list -f json &1 || true) + H=$(O hypervisor list -f json || true) if jq -e . >/dev/null 2>&1 <<<"$H"; then DOWN=$(jq -r '.[] | select((.State // ."state") != "up") | (."Hypervisor Hostname" // .hypervisor_hostname)' <<<"$H" || true) [ -z "$DOWN" ] && ok "all hypervisors up" || fail "hypervisors down: $(tr '\n' ' ' <<<"$DOWN") (restart nova-compute; see appendix-A)" - else fail "hypervisor list unreadable: $(head -1 <<<"$H")"; fi - S=$(openstack compute service list -f json &1 || true) - BADS=$(jq -r '.[] | select(.Status=="enabled" and .State!="up") | "\(.Binary)@\(.Host)"' <<<"$S" 2>/dev/null || true) - [ -z "$BADS" ] && ok "all enabled compute services up" || fail "compute services down: $(tr '\n' ' ' <<<"$BADS")" + else fail "hypervisor list unreadable: $(oerr)"; fi + S=$(O compute service list -f json || true) + if jq -e . >/dev/null 2>&1 <<<"$S"; then + BADS=$(jq -r '.[] | select(.Status=="enabled" and .State!="up") | "\(.Binary)@\(.Host)"' <<<"$S" || true) + [ -z "$BADS" ] && ok "all enabled compute services up" || fail "compute services down: $(tr '\n' ' ' <<<"$BADS")" + else fail "compute service list unreadable: $(oerr) (was silently fail-open pre-DOCFIX-094)"; fi echo "================ A6: octavia load balancers ================" - L=$(openstack loadbalancer list -f json &1 || true) + L=$(O loadbalancer list -f json || true) if jq -e . >/dev/null 2>&1 <<<"$L"; then CNT=$(jq 'length' <<<"$L") if [ "$CNT" -eq 0 ]; then ok "no LBs present (nothing to assert)" @@ -137,7 +144,7 @@ [ -z "$BADL" ] && ok "$CNT LB(s) ACTIVE/ONLINE" \ || fail "LBs unhealthy: $(tr '\n' ' | ' <<<"$BADL") (failover pattern: ops-restart-procedure Stage 6)" fi - else warn "octavia not answering (absent?): $(head -1 <<<"$L")"; fi + else warn "octavia not answering (absent?): $(oerr)"; fi echo "================ A7: identity + magnum behavior ================" # DOCFIX-087: read the PO: marker from status JSON messages. juju 3.6 diff --git a/scripts/tenant-offboard.sh b/scripts/tenant-offboard.sh index a12e407..ed55c94 100644 --- a/scripts/tenant-offboard.sh +++ b/scripts/tenant-offboard.sh @@ -70,7 +70,10 @@ fi UIDLIST=$(printf '%s\n' "$USERS" | awk '{print $1}') -TRUSTS=$( openstack trust list -f json &1 | python3 -c " +# DOCFIX-094: stdout-only structured capture -- merged stderr (deprecation notices) +# used to land at char 0, kill the JSON parse, and silently EMPTY the trust sweep. +TRERR=$(mktemp) +TRUSTS=$( openstack trust list -f json "$TRERR" | python3 -c " import sys,json uids=set('''$UIDLIST'''.split()) try: @@ -78,6 +81,9 @@ if t.get('Trustor User ID') in uids: print(t.get('ID'), t.get('Trustor User ID'), t.get('Trustee User ID')) except Exception as e: print('trust-parse-error', e)" ) || true +[ -s "$TRERR" ] && echo " NOTE: trust list stderr: $(head -1 "$TRERR")" +rm -f "$TRERR" +grep -q 'trust-parse-error' <<<"${TRUSTS:-}" && { echo "PRECOND: trust list unparseable -- audit/sweep would be blind; fix before offboarding"; exit 20; } echo "--- trusts with trustor in domain ---"; printf '%s\n' "${TRUSTS:-}" | sed 's/^/ /' for P in $(printf '%s\n' "$PROJS" | awk '{print $1}'); do diff --git a/scripts/tenant-onboard.sh b/scripts/tenant-onboard.sh index 57464c7..d90c85f 100644 --- a/scripts/tenant-onboard.sh +++ b/scripts/tenant-onboard.sh @@ -43,8 +43,10 @@ echo "== stage0: preflight ==" juju status keystone -m openstack --format=yaml 2>/dev/null | python3 -c 'import sys,yaml;m=yaml.safe_load(sys.stdin)["applications"]["keystone"]["units"]["keystone/0"].get("workload-status",{}).get("message","");print("keystone:",m);sys.exit(0 if m.startswith("PO:") else 1)' || { echo " keystone override not PO: active"; fail=1; } for R in manager member load-balancer_member; do is_id "$(openstack role show "$R" -f value -c id &1)" || { echo " role $R MISSING"; fail=1; }; done - local img; img=$(openstack image list --public -f json &1 | jq -r '[.[]|select(.Name|test("kube"))][0].ID // empty' || true) # -f json (column-order trap); ID-by-position rode alphabetical luck - is_id "${img//-/}" 2>/dev/null || [[ "$img" =~ ^[0-9a-f-]{36}$ ]] || { echo " no public kube image"; fail=1; } + local ierr; ierr=$(mktemp) # DOCFIX-094: stdout-only capture; merged stderr killed the jq parse + local img; img=$(openstack image list --public -f json "$ierr" | jq -r '[.[]|select(.Name|test("kube"))][0].ID // empty' || true) # -f json (column-order trap); ID-by-position rode alphabetical luck + is_id "${img//-/}" 2>/dev/null || [[ "$img" =~ ^[0-9a-f-]{36}$ ]] || { echo " no public kube image$( [ -s "$ierr" ] && echo " (stderr: $(head -1 "$ierr"))" )"; fail=1; } + rm -f "$ierr" is_id "$(openstack domain show "$CLIENT" -f value -c id &1)" && { echo " domain $CLIENT already EXISTS -- decide reuse/clean"; fail=1; } || true [ "$fail" = 0 ] && echo " PREFLIGHT PASS" || die "preflight failed" } @@ -175,8 +177,10 @@ stage5(){ # tenant template (image by UUID) admin_env - local IMG; IMG=$(openstack image list --public -f json &1 | jq -r '[.[]|select(.Name|test("kube"))][0].ID // empty' || true) # -f json (column-order trap) - [[ "$IMG" =~ ^[0-9a-f-]{36}$ ]] || die "kube image uuid resolve failed" + local IERR5; IERR5=$(mktemp) # DOCFIX-094: stdout-only capture + local IMG; IMG=$(openstack image list --public -f json "$IERR5" | jq -r '[.[]|select(.Name|test("kube"))][0].ID // empty' || true) # -f json (column-order trap) + [[ "$IMG" =~ ^[0-9a-f-]{36}$ ]] || { sed 's/^/ /' "$IERR5" 2>/dev/null; rm -f "$IERR5"; die "kube image uuid resolve failed"; } + rm -f "$IERR5" local ACF="$OUT/${CLIENT}-svc-appcred.txt" echo "== stage5: tenant cluster template (image by UUID) ==" ( for v in $(env|awk -F= '/^OS_/{print $1}'); do unset "$v"; done diff --git a/tests/cloud-assert/run-tests.sh b/tests/cloud-assert/run-tests.sh index b5fd50f..024a396 100644 --- a/tests/cloud-assert/run-tests.sh +++ b/tests/cloud-assert/run-tests.sh @@ -30,9 +30,18 @@ cat > "$BIN/openstack" <<'FB' #!/usr/bin/env bash A="$*" +# truthful-mock knobs (DOCFIX-094): FIXDIR/noise -> realistic deprecation notice on +# STDERR for the three list calls; FIXDIR/svc-err -> compute service list ERRORS +# (stderr + rc1, no stdout), the shape that was silently fail-open pre-fix. +case "$A" in + *"hypervisor list"*|*"compute service list"*|*"loadbalancer list"*) + [ -f "$FIXDIR/noise" ] && echo "WARNING: this CLI call is deprecated and will be removed" >&2 ;; +esac case "$A" in *"hypervisor list"*) cat "$FIXDIR/hypervisors.json" ;; - *"compute service list"*) cat "$FIXDIR/services.json" ;; + *"compute service list"*) + if [ -f "$FIXDIR/svc-err" ]; then echo "Unauthorized (HTTP 401)" >&2; exit 1; fi + cat "$FIXDIR/services.json" ;; *"loadbalancer list"*) cat "$FIXDIR/lbs.json" ;; *"domain show magnum"*) cat "$FIXDIR/domain.txt" ;; *"user show magnum_domain_admin"*) cat "$FIXDIR/user.txt" ;; @@ -119,6 +128,14 @@ "units":{"keystone/0":{"workload-status":{"current":"active","message":"Unit is ready"}}}}}} J run 1 'NO policy override' "T10 missing PO: marker FAILS (DOCFIX-087)" "$F" +# T11 (DOCFIX-094): stderr deprecation noise on healthy cloud must NOT break the +# structured captures (pre-fix: merged noise killed the A5 parse -> false FAIL) +F="$TMP/noise"; mkgood "$F"; : > "$F/noise" + run 0 'CLOUD-ASSERT: PASS' "T11 stderr noise does not break A5/A6 parsing" "$F" +# T12 (DOCFIX-094): compute service list ERRORING must FAIL (pre-fix: silently OK) +F="$TMP/svcerr"; mkgood "$F"; : > "$F/svc-err" + run 1 'compute service list unreadable' "T12 erroring service-list FAILS (was fail-open)" "$F" + # T9: no admin scope -> A5-A7 HELD, exit 2 (never a silent pass) F="$TMP/held"; mkgood "$F" out="$(cd "$TMP" && env -u OS_AUTH_URL FIXDIR="$F" PATH="$BIN:$PATH" bash "$REPO/scripts/cloud-assert.sh" 2>&1)"; rc=$?