diff --git a/.gitignore b/.gitignore index a00d016..60d04f5 100644 --- a/.gitignore +++ b/.gitignore @@ -37,7 +37,13 @@ .env # Octavia PKI artifacts - never commit -overlays/octavia-pki.yaml +# GLOB, not an exact path (F4, 2026-07-29): the D-109 amendment makes this overlay PER-DC +# (overlays/vr1-dc0-octavia-pki.yaml, overlays/vr1-dc1-octavia-pki.yaml). Pinning the single +# old exact name would have left every per-DC file UNIGNORED and therefore committable -- +# and it carries CA private-key blobs plus the issuing-CA passphrase in PLAINTEXT, in a repo +# SEC-004 records as PUBLIC. The trailing form also still covers the pre-rename name. +# phase-01 step 1.0-GEN.d asserts `git check-ignore` on its own output before writing. +overlays/*octavia-pki.yaml octavia-pki/ passphrase.txt # Bundle backups (timestamped) diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index afe22dc..bde2998 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1974,6 +1974,26 @@ file holding CA key blobs plus a plaintext passphrase would become committable in a repo SEC-004 records as PUBLIC. Recorded so the F1 fix cannot be taken without widening the glob in the SAME commit, plus a `git check-ignore` self-assert in the generator. + **F1 + F4 ARE NOW BUILT (2026-07-29, same session).** Step 1.0-GEN is per-DC end to end: + `DC`/`DC_LABEL`/`REPO`/`VIP_OVERLAY`/`OCTAVIA_PKI_OVERLAY` export ONCE in a rewritten + 1.0-GEN.0, `WORKDIR="$HOME/octavia-pki/$DC"`, and all four `WORKDIR` re-derivations plus the + overlay write now carry `${DC:?}` so an unset DC REFUSES instead of silently reusing the old + shared path. Two new gates that did not exist before: a **REFUSE-IF-PRESENT** check that + aborts when this DC already has PKI material (regeneration invalidates every amphora already + issued against that CA, so it must be deliberate, not the default outcome of re-running a + step), and the **F4 gate** -- `git check-ignore -q "$OUT"` immediately before any key + material is written, asserting the ACTUAL ignore decision for that path rather than trusting + a pattern anybody must remember. `.gitignore` widened to `overlays/*octavia-pki.yaml`, and + the widening was PROVEN BOTH WAYS rather than assumed: all three per-DC and legacy names + read IGNORED, and a negative control (`overlays/vr1-dc0-vips.yaml`) reads NOT ignored, so + the glob is not silently swallowing tracked overlays. Also fixed in the same pass, because + it was the same dc0-freeze: **Step 1.3's VIP guard was a hand-rolled `grep -hcE` triple + anchored on the literal `10\.12\.4\.`** -- it hard-ABORTED on dc1, counted IPv4 ONLY so R2's + ruled v6 legs were guarded by nothing (chain-audit finding 22), and its prose said `11/11/0` + while its code demanded `13`. It now calls `provider-bundle-check.py` on the merged per-DC + input, which already encodes the bands, the count and the 2026-07-28 dual-family arity + coupling -- one source of truth instead of a second copy that drifts. The 2026-06-03 as-built + line retains the old command verbatim: it is HISTORY, not instruction. **F5 (MEDIUM) -- preflight's `DC` selector does not reach P4.** `preflight.sh:99` sets `DC` without exporting it, so propagation depends on the caller's invocation form; and it is moot because `pre-flight-checks.sh` never reads `DC` at all (one hit, in a comment). diff --git a/runbooks/phase-01-bundle-deploy.md b/runbooks/phase-01-bundle-deploy.md index 422291b..5967d3b 100644 --- a/runbooks/phase-01-bundle-deploy.md +++ b/runbooks/phase-01-bundle-deploy.md @@ -31,7 +31,8 @@ - phase-00 done: 4 machines Ready/power=off; MAAS carve applied (front-loaded VIP /26 reserved, FIP pool reserved, stale iprange gone); enp8s0 data NIC linked on ALL four hosts; OSD `/dev/vdb` wiped blank. -- `overlays/octavia-pki.yaml` present (Step 1.0). +- `overlays/${DC}-octavia-pki.yaml` present (Step 1.0). PER-DC since F1 (2026-07-29) -- + export `DC` (region-qualified, e.g. `vr1-dc0`) before running any step in this phase. - Hardened `bundle.yaml` in the working dir (channels pinned; VIPs `.50-.60`; reserved-host-memory 8192; image-conversion; use-policyd-override). @@ -73,7 +74,7 @@ ## Step 1.0 -- Octavia PKI overlay (secret-handling prereq) DISCRETE -`overlays/octavia-pki.yaml` carries the 5 lb-mgmt-* PKI keys (controller CA/cert, +`overlays/${DC}-octavia-pki.yaml` carries the 5 lb-mgmt-* PKI keys (controller CA/cert, issuing CA key+passphrase+cert). It is the ONLY overlay in the deploy command and is secret-safe + ASCII. PRIMARY path: reuse the existing validated overlay (the CAs are 10y, so it survives rebuilds). REGENERATION path (fresh CAs): run the discrete secret @@ -83,8 +84,17 @@ **CHECK (read-only) -- jumphost** ```bash # RUN: jumphost -- sanity only (does NOT print key material) -[ -f overlays/octavia-pki.yaml ] && grep -cE 'lb-mgmt-' overlays/octavia-pki.yaml # expect 5 keys -LC_ALL=C grep -nP '[^\x00-\x7F]' overlays/octavia-pki.yaml && echo "NON-ASCII" || echo "ASCII clean" +DC="${DC:?export DC=vr1-dc0|vr1-dc1 -- the PKI overlay is per-DC (F1)}" +OVL="overlays/${DC}-octavia-pki.yaml" +# Assert on CONTENT, not existence: an absent file, a wrong key count and a non-ASCII +# byte are three DIFFERENT answers and none of them may read as OK. +if [ ! -f "$OVL" ]; then + echo "MISSING $OVL -- generate it with Step 1.0-GEN for DC=$DC" +else + K=$(grep -cE 'lb-mgmt-' "$OVL" || true) + [ "$K" -eq 5 ] && echo "OK 5 lb-mgmt-* keys" || echo "FAIL $K lb-mgmt-* keys (want 5)" + LC_ALL=C grep -qP '[^\x00-\x7F]' "$OVL" && echo "FAIL non-ASCII bytes" || echo "OK ASCII clean" +fi ``` ## Step 1.1 -- Pre-deploy verify (read-only; 4 checks) @@ -145,12 +155,12 @@ juju models 2>&1 | tee /tmp/jmodels.txt if grep -qE '(^|[[:space:]]|/)openstack([[:space:]*]|$)' /tmp/jmodels.txt; then echo "ABORT: an 'openstack' model already exists (teardown is phase-00)"; - elif [ ! -f overlays/octavia-pki.yaml ]; then - echo "ABORT: overlays/octavia-pki.yaml missing (Step 1.0)"; + elif [ ! -f "overlays/${DC:?export DC first}-octavia-pki.yaml" ]; then + echo "ABORT: overlays/${DC}-octavia-pki.yaml missing (Step 1.0-GEN for DC=$DC)"; else juju add-model openstack - juju deploy ./bundle.yaml --overlay overlays/vr1-dc0-vips.yaml \ - --overlay overlays/octavia-pki.yaml -m openstack --dry-run + juju deploy ./bundle.yaml --overlay "overlays/${DC}-vips.yaml" \ + --overlay "overlays/${DC}-octavia-pki.yaml" -m openstack --dry-run fi } ) ``` @@ -166,20 +176,40 @@ `variables`). ## Step 1.3 -- Deploy (VIP-guarded) -Re-run the VIP guard inline (the dry-run never echoes vip values), -then deploy only if 11/11/0. +Re-run the VIP guard inline (the dry-run never echoes vip values), then deploy only if +the guard exits 0. + +> **THE GUARD IS NOW THE MAINTAINED CHECKER, NOT A HAND-ROLLED GREP (2026-07-29).** This +> block used to compute a `13/13/0` triple with three `grep -hcE` counts anchored on the +> literal `10\.12\.4\.`. Three defects, all measured: it hard-ABORTED on `vr1-dc1`, whose +> provider band is `10.12.64.x`; it counted only IPv4, so after R2's dual-stack ADD the +> ruled v6 legs were checked by NOTHING (chain-audit finding 22); and its prose said +> "11/11/0" while its code demanded `13`, a drift that appeared the moment R11 added +> vault `.61` and designate `.62`. `scripts/provider-bundle-check.py` already encodes +> every one of those invariants -- per-DC bands, the VIP count, and since 2026-07-28 the +> dual-family ARITY coupling -- so the runbook defers to it instead of keeping a second +> copy that drifts. One source of truth; a new ruling updates the checker and this step +> inherits it. **RUN -- jumphost** ```bash ( { - TOT=$(grep -hcE '^[[:space:]]+vip:[[:space:]]*"?10\.12\.4\.' bundle.yaml overlays/vr1-dc0-vips.yaml 2>/dev/null | awk '{s+=$1} END{print s+0}') - HI=$(grep -hcE '^[[:space:]]+vip:[[:space:]]*"?10\.12\.4\.(5[0-9]|6[0-2])("|$|[[:space:]])' bundle.yaml overlays/vr1-dc0-vips.yaml 2>/dev/null | awk '{s+=$1} END{print s+0}') - LO=$(grep -hcE '^[[:space:]]+vip:[[:space:]]*"?10\.12\.4\.(1[0-9]|20)("|$|[[:space:]])' bundle.yaml overlays/vr1-dc0-vips.yaml 2>/dev/null | awk '{s+=$1} END{print s+0}') - if [ "$TOT" = 13 ] && [ "$HI" = 13 ] && [ "$LO" = 0 ]; then - juju deploy ./bundle.yaml --overlay overlays/vr1-dc0-vips.yaml \ - --overlay overlays/octavia-pki.yaml -m openstack + DC="${DC:?export DC=vr1-dc0|vr1-dc1 first}" + set -- ./bundle.yaml + for OVL in "overlays/${DC}-vips.yaml" "overlays/${DC}-machines.yaml" \ + "overlays/${DC}-octavia-pki.yaml"; do + [ -f "$OVL" ] && set -- "$@" --overlay "$OVL" + done + # The octavia-pki overlay is REQUIRED for deploy (it carries the 5 lb-mgmt-* keys) -- + # absence is an ABORT, not a silently-skipped overlay. + [ -f "overlays/${DC}-octavia-pki.yaml" ] || { + echo "ABORT: overlays/${DC}-octavia-pki.yaml missing -- run Step 1.0-GEN for DC=$DC"; exit 1; } + + echo "validating the MERGED $DC deploy input: $*" + if python3 scripts/provider-bundle-check.py "$@" --dc "$DC"; then + juju deploy "$@" -m "${DC_MODEL:?export DC_MODEL first}" else - echo "ABORT: VIP guard failed (total=$TOT hi=$HI lo=$LO; want 13/13/0)" + echo "ABORT: VIP/bundle invariants failed for $DC -- fix before deploying"; exit 1 fi } ) ``` @@ -271,7 +301,9 @@ > CAUTION: SECRET step -- generates Octavia CA private keys + passphrases. Do NOT echo or log key > material (only cert dates/subjects + verify-OK are printed). The overlay it writes is gitignored. -Run ONLY if you are not reusing an existing `overlays/octavia-pki.yaml`. Produces the +Run ONLY if this DC has no PKI yet -- 1.0-GEN.0 REFUSES if `overlays/${DC}-octavia-pki.yaml` +or this DC's workspace already exists, because regenerating invalidates every amphora +already issued against that CA. Each DC is generated separately. Produces the two-tier EC PKI for Charmed Octavia's amphora trust domain and writes the overlay. Decisions (Workstream 3a, 2026-05-22): fresh generation; EC P-384 CAs (SHA-384, 10y); EC P-256 controller cert (2y); overlay-file distribution (gitignored); artifacts under @@ -285,15 +317,49 @@ - `lb-mgmt-controller-cacert` = base64(controller CA cert) - `lb-mgmt-controller-cert` = base64(controller cert + key, concatenated) -### 1.0-GEN.0 -- workspace (openssl 3.x; $HOME only -- snap home-confinement, never /tmp) +### 1.0-GEN.0 -- select the DC, then the workspace (openssl 3.x; $HOME only -- snap home-confinement, never /tmp) + +> **EVERY PATH IN THIS STEP IS PER-DC (F1, 2026-07-29).** The D-109 amendment rules +> "each DC gets its own Octavia CA; no cross-DC amphora root-of-trust". The 2026-07-27 +> R7 work parameterised the CA SUBJECT and the VIP gate but NOT the paths, so the +> workspace and the overlay were single fixed locations: generating a second DC's PKI +> overwrote the first DC's issuing-CA key, controller-CA key and both passphrases, and +> left the one fixed-name overlay carrying the wrong DC's CA -- which a later redeploy +> of the first DC would then apply to it. Set `DC` ONCE here; everything below derives +> from it, paths included. **RUN -- jumphost** ```bash # RUN: jumphost -WORKDIR="$HOME/octavia-pki" +export DC="${DC:-vr1-dc0}" # vr1-dc0 | vr1-dc1 -- region-qualified, never bare dcN +export DC_LABEL="$(printf '%s' "$DC" | tr 'a-z-' 'A-Z ')" # e.g. "VR1 DC0" +export REPO="${REPO:?REPO not set -- export REPO= (see runbooks/README.md Conventions)}" +export VIP_OVERLAY="$REPO/overlays/${DC}-vips.yaml" +export OCTAVIA_PKI_OVERLAY="$REPO/overlays/${DC}-octavia-pki.yaml" +[ -r "$VIP_OVERLAY" ] || { echo "ABORT: no VIP overlay at $VIP_OVERLAY -- is DC=$DC correct?"; exit 1; } + +WORKDIR="$HOME/octavia-pki/$DC" # PER-DC (F1) -- never $HOME/octavia-pki directly mkdir -p "$WORKDIR"/issuing-ca "$WORKDIR"/controller-ca "$WORKDIR"/controller -chmod 700 "$WORKDIR" +chmod 700 "$HOME/octavia-pki" "$WORKDIR" openssl version # expect OpenSSL 3.x +echo "DC=$DC WORKDIR=$WORKDIR overlay=$OCTAVIA_PKI_OVERLAY" +``` + +**REFUSE-IF-PRESENT gate -- run before generating anything.** Regeneration destroys the +CA this DC's already-deployed amphorae trust, so it must be a deliberate act, not the +default outcome of re-running a step. + +```bash +# RUN: jumphost +if [ -e "$WORKDIR/issuing-ca/issuing-ca.key.enc" ] || [ -e "$OCTAVIA_PKI_OVERLAY" ]; then + echo "REFUSING: PKI material already exists for DC=$DC." + echo " workspace: $WORKDIR" + echo " overlay: $OCTAVIA_PKI_OVERLAY" + echo "Regenerating invalidates every amphora already issued against this CA." + echo "To deliberately regenerate: move the existing material aside first, gated." + exit 1 +fi +echo "OK: no existing PKI for DC=$DC -- safe to generate" ``` ### 1.0-GEN.a -- Issuing CA (EC P-384, AES-256 encrypted key, self-signed 10y) @@ -306,18 +372,14 @@ > which made every other DC undeployable. Reuse across DCs was refused on posture: the > overlay carries CA private keys plus a plaintext passphrase in a repo SEC-004 records > as PUBLIC, so one shared amphora CA across two clouds D-100 defines as independent -> would widen an existing exposure. Set `DC` ONCE before the blocks below; everything -> else derives from it. -> -> ```bash -> export DC="${DC:-vr1-dc0}" # vr1-dc0 | vr1-dc1 -> export DC_LABEL="$(printf '%s' "$DC" | tr 'a-z-' 'A-Z ')" # e.g. "VR1 DC0" -> export VIP_OVERLAY="$REPO/overlays/${DC}-vips.yaml" -> ``` +> would widen an existing exposure. `DC`, `DC_LABEL`, `VIP_OVERLAY` and +> `OCTAVIA_PKI_OVERLAY` are exported ONCE in **1.0-GEN.0** above -- along with the +> per-DC `WORKDIR` that F1 added -- and every block below derives from them. ```bash ( { - WORKDIR="$HOME/octavia-pki"; cd "$WORKDIR/issuing-ca" || exit 1 # dir from 1.0-GEN.a + WORKDIR="$HOME/octavia-pki/${DC:?DC not set -- run 1.0-GEN.0 first}" + cd "$WORKDIR/issuing-ca" || exit 1 # dir from 1.0-GEN.0 openssl rand -base64 32 | tr -d '\n' > passphrase.txt chmod 600 passphrase.txt test "$(wc -c < passphrase.txt)" -eq 44 || { echo "ABORT: issuing passphrase length != 44"; exit 1; } @@ -339,7 +401,8 @@ **RUN -- jumphost** ```bash ( { - WORKDIR="$HOME/octavia-pki"; cd "$WORKDIR/controller-ca" || exit 1 # dir from 1.0-GEN.a + WORKDIR="$HOME/octavia-pki/${DC:?DC not set -- run 1.0-GEN.0 first}" + cd "$WORKDIR/controller-ca" || exit 1 # dir from 1.0-GEN.0 openssl rand -base64 32 | tr -d '\n' > passphrase.txt chmod 600 passphrase.txt test "$(wc -c < passphrase.txt)" -eq 44 || { echo "ABORT: controller passphrase length != 44"; exit 1; } @@ -361,7 +424,8 @@ **RUN -- jumphost** ```bash ( { - WORKDIR="$HOME/octavia-pki"; cd "$WORKDIR/controller" || exit 1 # dir from 1.0-GEN.a + WORKDIR="$HOME/octavia-pki/${DC:?DC not set -- run 1.0-GEN.0 first}" + cd "$WORKDIR/controller" || exit 1 # dir from 1.0-GEN.0 openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out controller.key chmod 600 controller.key cat > controller.cnf <<'CNF' @@ -455,22 +519,38 @@ } ) ``` -### 1.0-GEN.d -- Write overlays/octavia-pki.yaml (base64 blobs + plaintext passphrase) +### 1.0-GEN.d -- Write overlays/${DC}-octavia-pki.yaml (base64 blobs + plaintext passphrase) Four values are base64(PEM); the issuing-CA passphrase is a PLAIN string. The file is -gitignored. Set `$REPO` to the jumphost clone (the dir holding bundle.yaml + overlays/). +gitignored. `$REPO` and `$OCTAVIA_PKI_OVERLAY` are exported in 1.0-GEN.0. + +> **THE GITIGNORE CHECK BELOW IS LOAD-BEARING, NOT HYGIENE (F4, 2026-07-29).** This file +> carries CA private-key blobs PLUS the issuing-CA passphrase in PLAINTEXT, and SEC-004 +> records this repo as currently PUBLIC -- the gitignore is the only thing between a CA +> private key and publication. `.gitignore` used to pin the single exact path +> `overlays/octavia-pki.yaml`, so the moment F1 made the name per-DC the new name stopped +> matching and the file would have been committable. The pattern is now a glob, and this +> block ASSERTS that fact at generation time rather than trusting anyone to remember it. **RUN -- jumphost** ```bash ( { - WORKDIR="$HOME/octavia-pki"; cd "$WORKDIR" || exit 1 # dir from 1.0-GEN.a + WORKDIR="$HOME/octavia-pki/${DC:?DC not set -- run 1.0-GEN.0 first}" + cd "$WORKDIR" || exit 1 # dir from 1.0-GEN.0 REPO="${REPO:?REPO not set -- export REPO= (see runbooks/README.md Conventions)}" + OUT="${OCTAVIA_PKI_OVERLAY:?OCTAVIA_PKI_OVERLAY not set -- run 1.0-GEN.0 first}" mkdir -p "$REPO/overlays" + # F4 GATE: refuse to write key material into a path git would track. Asserts on the + # ACTUAL ignore decision for THIS path, not on the presence of a pattern. + git -C "$REPO" check-ignore -q "$OUT" || { + echo "ABORT: $OUT is NOT gitignored -- writing CA key material here would make it" + echo " committable in a repo SEC-004 records as PUBLIC. Fix .gitignore first." + exit 1; } ISS_CERT=$(base64 -w0 issuing-ca/issuing-ca.cert.pem) ISS_KEY=$(base64 -w0 issuing-ca/issuing-ca.key.enc) ISS_PASS=$(cat issuing-ca/passphrase.txt) CON_CACERT=$(base64 -w0 controller-ca/controller-ca.cert.pem) CON_CERT=$(base64 -w0 controller/controller.bundle.pem) - cat > "$REPO/overlays/octavia-pki.yaml" < "$OUT" <