diff --git a/creds-manifests/vr1-dc0.manifest b/creds-manifests/vr1-dc0.manifest index 63f7ad5..18b296b 100644 --- a/creds-manifests/vr1-dc0.manifest +++ b/creds-manifests/vr1-dc0.manifest @@ -48,3 +48,6 @@ # no jumphost copy to re-assert from after a snap refresh. vr1-dc0-maas-power_ed25519 600 local vr1-dc0-maas-power_ed25519.pub 644 local + +# [octavia-pki-backup] see creds-matrix-notes.md ## n-pki-backup +octavia-pki-vr1-dc0.tar.gz 600 local diff --git a/creds-manifests/vr1-dc1.manifest b/creds-manifests/vr1-dc1.manifest index 161e4ed..d53a563 100644 --- a/creds-manifests/vr1-dc1.manifest +++ b/creds-manifests/vr1-dc1.manifest @@ -35,3 +35,6 @@ # (off-manifest -- that store is not this folder). Declared 2026-07-25; it was # an UNDECLARED creds-audit FAIL until then. maas-api-key.txt 600 local + +# [octavia-pki-backup] see creds-matrix-notes.md ## n-pki-backup +octavia-pki-vr1-dc1.tar.gz 600 local diff --git a/creds-matrix-notes.md b/creds-matrix-notes.md index 969861d..59b5574 100644 --- a/creds-matrix-notes.md +++ b/creds-matrix-notes.md @@ -222,6 +222,38 @@ radius reaches `~/vault-init/`, which CLAUDE.md designates operator-only one-shot territory, so it was deliberately NOT bundled into an R7-conformance edit. +## n-pki-backup +**Recovery copy of a per-DC Octavia PKI workspace, added 2026-07-30 (operator-directed).** +One gzipped archive per DC in that DC's jumphost creds folder, produced by +`phase-01-bundle-deploy.md` step 1.0-GEN.e. + +**WHY IT EXISTS.** Three surfaces asserted this material "MUST be in the per-DC backup set" / +"backed up securely" and **no backup set existed** -- `scripts/cloud-snapshot.sh` is a +juju-layer capture and mentions octavia, tfstate and terraform ZERO times. So both DCs' 10-year +amphora trust roots sat in exactly one place, on one VM, with no copy. Losing the ISSUING CA key +means Octavia can never sign another amphora certificate (no new load balancers, no amphora +replacement); losing the CONTROLLER CA key means the controller certificate can never be +reissued, which F9 establishes it must be at Stage 7. + +**WHAT IT COSTS.** A SECOND at-rest copy of both encrypted CA private keys BESIDE their +plaintext passphrases, so the archive's own contents defeat encryption-at-rest -- treat it +exactly as the live keys. This is the trade D-109 option (b) was refused for, and the +distinction is deliberate: that ruling governed where the AUTHORITATIVE artifact lives and +which host the deploy reads. A recovery copy is not a second source of truth. + +**Integrity is asserted, not assumed:** the archive's sha256 is compared against the source +before the staging copy is removed from the headend, because a truncated `scp` would otherwise +leave a verified-looking backup of nothing. + +**STANDING FORWARD REQUIREMENT -- carry this into the pinned secrets-storage design.** When the +secrets-storage solution is settled, it MUST include a documented PROCESS AND PROCEDURE for +CERTIFICATE and CREDENTIAL backup, and **this step is one of the steps that must be folded into +it.** The jumphost creds folder is the SEC-009 convention location and is the INTERIM home only. +Recorded here rather than in a runbook comment so the requirement is found by anyone reading the +register, and so it is not rediscovered the next time a 10-year CA turns out to exist in one +place. Related: R13 Part 2's reproducibility debt, and the 12 declared secrets with no mint +command anywhere in the repo. + ## n-tenant-dir Per-tenant credential directory outside the `*-creds/` convention (research FINDING 2). Hard tenant isolation applies: D-069 keeps custody off-repo, and D-137 ruling 3 bounds diff --git a/creds-matrix.tsv b/creds-matrix.tsv index 123cd25..77fedb6 100644 --- a/creds-matrix.tsv +++ b/creds-matrix.tsv @@ -110,6 +110,7 @@ octavia-controller-bundle per-DC vr1-dc0 headend controller.bundle.pem none service off-manifest-known stage5 runbook:runbooks/phase-01-bundle-deploy.md:517 SEC-023 n-outside-convention octavia-controller-cert per-DC vr1-dc0 headend controller.cert.pem none service off-manifest-known stage5 runbook:runbooks/phase-01-bundle-deploy.md:521 SEC-023 n-outside-convention octavia-controller-ca-serial per-DC vr1-dc0 headend controller-ca.cert.srl none service off-manifest-known stage5 runbook:runbooks/phase-01-bundle-deploy.md:519 SEC-023 n-outside-convention +octavia-pki-backup per-DC vr1-dc0 jumphost octavia-pki-vr1-dc0.tar.gz none service consolidated stage5 runbook:runbooks/phase-01-bundle-deploy.md:628 SEC-023 n-pki-backup octavia-pki-overlay per-DC vr1-dc0 headend vr1-dc0-octavia-pki.yaml none service off-manifest-known stage5 runbook:runbooks/phase-01-bundle-deploy.md:553 SEC-004 n-overlay-in-clone octavia-issuing-ca-passphrase per-DC vr1-dc1 headend passphrase.txt none service off-manifest-known stage5 runbook:runbooks/phase-01-bundle-deploy.md:383 SEC-023 n-outside-convention octavia-issuing-ca-key per-DC vr1-dc1 headend issuing-ca.key.enc none service off-manifest-known stage5 runbook:runbooks/phase-01-bundle-deploy.md:387 SEC-023 n-outside-convention @@ -121,6 +122,7 @@ octavia-controller-bundle per-DC vr1-dc1 headend controller.bundle.pem none service off-manifest-known stage5 runbook:runbooks/phase-01-bundle-deploy.md:517 SEC-023 n-outside-convention octavia-controller-cert per-DC vr1-dc1 headend controller.cert.pem none service off-manifest-known stage5 runbook:runbooks/phase-01-bundle-deploy.md:521 SEC-023 n-outside-convention octavia-controller-ca-serial per-DC vr1-dc1 headend controller-ca.cert.srl none service off-manifest-known stage5 runbook:runbooks/phase-01-bundle-deploy.md:519 SEC-023 n-outside-convention +octavia-pki-backup per-DC vr1-dc1 jumphost octavia-pki-vr1-dc1.tar.gz none service consolidated stage5 runbook:runbooks/phase-01-bundle-deploy.md:628 SEC-023 n-pki-backup octavia-pki-overlay per-DC vr1-dc1 headend vr1-dc1-octavia-pki.yaml none service off-manifest-known stage5 runbook:runbooks/phase-01-bundle-deploy.md:553 SEC-004 n-overlay-in-clone tenant-domain-admin per-tenant - jumphost -domain-admin-cred.txt gui human off-manifest-known tenant-onboard script:scripts/tenant-onboard.sh:64 SEC-023 n-tenant-dir tenant-cluster-user per-tenant - jumphost -cluster-cred.txt api human off-manifest-known tenant-onboard script:scripts/tenant-onboard.sh:90 SEC-023 n-tenant-dir diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index c3b78ce..7e7c88a 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2309,6 +2309,44 @@ IMPROVES posture):** the one-off `chmod 600` on the six existing certs, and removal of the four existing intermediates. Until both run, live `verify` reads 23/3 and P5 keeps 12 of its 18 findings. **F9 is next and remains untouched.** +- **THE PKI BACKUP PATH IS BUILT 2026-07-30, and it closes a gap where three surfaces asserted a + backup set that DID NOT EXIST.** Operator direction: back up to the per-DC jumphost creds + folder BEFORE running the cert cleanup, and record that the pinned secrets-storage solution + must carry a certificate/credential backup procedure with this step folded into it. + **THE GAP, MEASURED:** `phase-01:594` said the workspace must be "backed up securely", + `phase-01:605` said it "MUST be in the per-DC backup set", and this document said the inner + tfstate should be added to "the site backup set" -- while `scripts/cloud-snapshot.sh`, the only + candidate, is a juju-layer capture that mentions octavia, tfstate and terraform **ZERO + times**. Both DCs' 10-year amphora trust roots therefore sat in ONE place on ONE VM with no + copy. Losing the ISSUING CA key means Octavia can never sign another amphora certificate (no + new load balancers, no amphora replacement); losing the CONTROLLER CA key means the controller + certificate can never be reissued, which F9 says it must be at Stage 7. + **SHAPE: one gzipped archive per DC, not a file-by-file tree copy** -- 2 register rows instead + of ~20, ATOMIC (a half-copied tree is the dangerous state), and sha256-verifiable. The live + tree on the headend remains what `octavia-pki.sh verify` asserts; the archive is recovery only. + **COST STATED RATHER THAN GLOSSED:** this is a SECOND at-rest copy of both encrypted CA keys + BESIDE their plaintext passphrases, so the archive's own contents defeat encryption-at-rest. + That is the trade D-109 option (b) was REFUSED for, and the distinction is deliberate -- that + ruling governed where the AUTHORITATIVE artifact lives and which host the deploy reads; a + recovery copy is not a second source of truth. + DELIVERED: `phase-01` step **1.0-GEN.e** (build on the headend, pull to `~/-creds/`, + **compare sha256 against the source BEFORE removing the staging copy** -- a truncated `scp` + would otherwise leave a verified-looking backup of nothing); 2 register rows + (`octavia-pki-backup`, per-DC, jumphost, `custody=consolidated` since the creds folder IS the + SEC-009 location); notes key `n-pki-backup`. Register now **97 rows / 24 octavia rows**. + **THE REGISTER CAUGHT MY OWN INCOMPLETE CHANGE:** adding rows raised S2 EXPECTED-BUT-ABSENT + twice, because `creds-manifests/*.manifest` are DERIVED from the matrix (D-137 ruling 2, "do + not hand-edit a manifest") and I had not re-rendered them. Rendered rows appended to both + manifests; `S3 render drift` clean and findings back to the pre-existing **7**. + **STANDING FORWARD REQUIREMENT, recorded in the register rather than as a runbook comment so + it is found by anyone reading it:** when the secrets-storage solution is pinned, it MUST + include a documented process and procedure for CERTIFICATE and CREDENTIAL backup, and **this + step is one of the steps that must be folded into it**. The jumphost creds folder is the + INTERIM home only. Related: R13 Part 2's reproducibility debt and the 12 declared secrets with + no mint command anywhere in the repo. + **STILL NOT BACKED UP, and out of this step's scope:** the two inner `terraform.tfstate` files + on the headend -- gitignored, untracked, the substrate's state-of-record for 20 VMs, and named + by this document as owed to "the site backup set" that has just been shown not to exist. **F3 -- both `~/octavia-pki/` and `overlays/octavia-pki.yaml` are ABSENT here** (existence checked, no contents read). So this is generation FROM SCRATCH for both DCs: there is nothing to reuse, which retires the reuse-vs-regenerate choice diff --git a/runbooks/phase-01-bundle-deploy.md b/runbooks/phase-01-bundle-deploy.md index 19ff9f9..5506595 100644 --- a/runbooks/phase-01-bundle-deploy.md +++ b/runbooks/phase-01-bundle-deploy.md @@ -594,6 +594,64 @@ `$HOME/octavia-pki/` (the CA keys + passphrases) OFF the repo and backed up securely; the 10y CAs are reused across rebuilds -- regenerate only on key compromise or CA expiry. +### 1.0-GEN.e -- BACK UP the per-DC workspace to the jumphost creds folder DISCRETE / SECRET + +> **THIS STEP IS NOT OPTIONAL, AND THE REASON IS MEASURED.** Before it existed, three +> surfaces said this material "MUST be in the per-DC backup set" / "backed up securely" -- +> and **no backup set existed**: `scripts/cloud-snapshot.sh` is a juju-layer capture and +> mentions octavia, tfstate and terraform ZERO times. So both DCs' 10-year amphora trust +> roots lived in exactly one place, on one VM, with no copy. Losing the ISSUING CA key means +> Octavia can never sign another amphora certificate -- no new load balancers, no amphora +> replacement. Losing the CONTROLLER CA key means the controller certificate can never be +> reissued, which F9 establishes it must be at Stage 7. +> +> **WHAT THIS COSTS, stated plainly:** it creates a SECOND at-rest copy of both encrypted CA +> keys BESIDE their plaintext passphrases. Encryption-at-rest is therefore defeated by the +> archive's own contents -- treat the archive exactly as you would the live keys. This is the +> same trade D-109 option (b) was REFUSED for, and the distinction is deliberate: that ruling +> governed where the AUTHORITATIVE artifact lives and which host the deploy reads. A backup is +> a recovery copy, not a second source of truth. +> +> **INTERIM, PENDING THE PINNED SECRETS-STORAGE SOLUTION.** The jumphost creds folder is the +> SEC-009 convention location and is where this goes for now. When the secrets-storage solution +> is designed, it MUST carry a process and procedure for CERTIFICATE and CREDENTIAL backup, and +> THIS STEP IS ONE OF THE STEPS THAT MUST BE FOLDED INTO IT -- not left as a per-runbook +> one-off. Recorded as a standing forward item so the requirement is not rediscovered. + +**RUN -- voffice1, then vcloud (the archive moves headend -> jumphost)** +```bash +# RUN: voffice1 -- build the archive beside the workspace, never inside the repo +( { + DC="${DC:?run 1.0-GEN.0 first}" + cd "$HOME/octavia-pki" || exit 1 + umask 077 + tar czf "$HOME/octavia-pki-${DC}.tar.gz" "$DC" + chmod 600 "$HOME/octavia-pki-${DC}.tar.gz" + sha256sum "$HOME/octavia-pki-${DC}.tar.gz" # RECORD this; it is the integrity check + tar tzf "$HOME/octavia-pki-${DC}.tar.gz" | wc -l # expect the workspace file count +} ) +``` + +```bash +# RUN: vcloud -- pull it into the per-DC creds folder, verify, then remove the staging copy +( { + DC="${DC:?export DC=vr1-dc0|vr1-dc1}" + install -d -m 700 "$HOME/${DC}-creds" + scp -q "voffice1:~/octavia-pki-${DC}.tar.gz" "$HOME/${DC}-creds/octavia-pki-${DC}.tar.gz" + chmod 600 "$HOME/${DC}-creds/octavia-pki-${DC}.tar.gz" + # INTEGRITY: compare against the source, do not assume scp succeeded + A=$(sha256sum < "$HOME/${DC}-creds/octavia-pki-${DC}.tar.gz" | cut -d' ' -f1) + B=$(ssh voffice1 "sha256sum < ~/octavia-pki-${DC}.tar.gz" | cut -d' ' -f1) + [ "$A" = "$B" ] && echo "OK sha256 matches source" || { echo "ABORT: sha256 MISMATCH -- do not delete the source"; exit 1; } + ssh voffice1 "rm -f ~/octavia-pki-${DC}.tar.gz" # only after the match + echo "backed up: ~/${DC}-creds/octavia-pki-${DC}.tar.gz" +} ) +``` + +Run BOTH blocks once per DC. The staging copy is removed from the headend only after the +sha256 comparison passes -- an `scp` that silently truncated would otherwise leave you with a +verified-looking backup of nothing. + **Per-DC layout produced (F1).** Each DC is a wholly separate trust domain on disk, so a second DC's generation cannot reach the first DC's material: