v1 Do-Document 02 — Octavia LBaaS PKI Overlay Generation

Status: Second execution document of Batch A. Generates local-only crypto material (no cloud touched). Runs after v1-do-doc-01-prep.md (state check). Runs before runbooks/01-destroy-model.md re-verification (Batch B doc 03 will point to that runbook).

Replaces: runbooks/deprecated/01a-octavia-pki-generation.md (which had a buggy VIP-count grep pattern, an obsolete §12 bundle-housekeeping block, a self-reference off-by-one, and exit 1 blocks in operator-facing context that violate the no-exit-in-pasted-shell rule).

Cross-references:

• D-007 (Magnum / Octavia inclusion) — Octavia bundle integration
• Bundle octavia.options PKI material section
• overlays/octavia-pki.yaml (gitignored — output of this document)
• Workstream 3a decision (2026-05-22): generate fresh, EC P-384 CAs, overlay-file approach

1. Purpose & scope

Generate a complete two-tier PKI for Charmed Octavia's amphora load-balancer trust domain. The output is a single overlay file (overlays/octavia-pki.yaml) that the v1-do-doc-04 deploy step will pass to juju deploy --overlay.

Octavia uses two CAs:

• Issuing CA — signs each amphora's server certificate at LB-creation time. Octavia receives the private key and passphrase (so it can sign at runtime).
• Controller CA — trust anchor for connections from the Octavia controller to the amphorae. Octavia receives only the cert (no key needed at runtime; signing of controller certs is a humans-only rotation event).

Plus one controller certificate (cert + key bundled) signed by the Controller CA.

Five charm options on the octavia application consume the artifacts:

Scope: v1 testcloud (VR0 DC0 Omega Cloud). Roosevelt deltas in §14.

Out of scope:

• Octavia API TLS (issued by Vault via octavia:certificates relation in the bundle; separate concern)
• Rotation procedure (deferred to Roosevelt runbook; testcloud rotation pointer in §15)

2. Decisions captured

Per workstream 3a sign-off (2026-05-22):

Naming convention:

• Issuing CA CN: VR0 DC0 Omega Cloud Octavia Issuing CA
• Controller CA CN: VR0 DC0 Omega Cloud Octavia Controller CA
• Controller cert CN: octavia-controller.omega.dc0.vr0.cloud.neumatrix.local
• Controller cert SANs: above CN, plus octavia.omega.dc0.vr0.cloud.neumatrix.local, plus 10.12.4.233 (the Octavia API VIP)
• Organization (O): Neumatrix

3. Prerequisites

Shell context — paste once at start:

export REPO="$HOME/openstack-caracal-ipv4"
echo "REPO=$REPO"
test -d "$REPO/.git" && echo "[OK] repo present" || echo "[FAIL] repo missing"
cd "$REPO"

Verify pre-deploy fixes are present (smoketest against the corrected grep pattern):

echo "VIP grep (corrected pattern, expect 12):"
grep -cE "^[[:space:]]+vip: 10\.12\.4\." "$REPO/bundle.yaml"

If this returns anything other than 12, the bundle pre-deploy fix did not land — stop and reconcile before proceeding.

Note on the grep pattern: the deprecated runbook 01a used ^ vip: 10.12.4. with a single literal space, which matches zero lines because the bundle's vip: entries live inside options: blocks indented six spaces deep. The corrected pattern ^[[:space:]]+vip: 10\.12\.4\. matches any leading whitespace and escapes the dot literals.

4. Pre-flight: gitignore patch (DO THIS FIRST)

Critical: the .gitignore patch must be in main BEFORE any private key material exists on disk in the workspace. This minimizes the race window for an accidental commit.

The current .gitignore already catches *.key, *.crt, *.pem via wildcards, but does NOT catch overlays/octavia-pki.yaml (a .yaml file) or passphrase.txt (a .txt file). This step adds the missing patterns.

cd "$REPO"

# Idempotent patch — only add the block if the overlay path is not already protected
if ! grep -q "^overlays/octavia-pki.yaml" .gitignore; then
  cat >> .gitignore <<'EOF'

# Octavia PKI artifacts — never commit
overlays/octavia-pki.yaml
octavia-pki/
passphrase.txt
EOF
  echo "[OK] .gitignore patched"
else
  echo "[OK] .gitignore already has overlay protection (no change)"
fi

# Review the diff (will be empty if already patched)
git diff .gitignore

If the diff shows the new block, commit and push it before generating any keys:

git add .gitignore
git commit -m "gitignore: octavia PKI artifacts and overlay (v1-do-doc-02 §4)"
git push origin main

Verify the gitignore is effective (this is a safety smoketest — touch a fake overlay file and ensure git ignores it):

touch overlays/octavia-pki.yaml
STATUS=$(git status --short overlays/octavia-pki.yaml)
rm overlays/octavia-pki.yaml

if [ -z "$STATUS" ]; then
  echo "[OK] gitignore working — overlay file does not show as untracked"
else
  echo "[FAIL] gitignore not effective — git sees:"
  echo "  $STATUS"
  echo "Stop here. Fix .gitignore syntax before generating any secrets."
fi

If [FAIL], do not proceed. The error means a generated PKI overlay could be accidentally committed.

5. Workspace setup

WORKDIR="$HOME/octavia-pki"
mkdir -p "$WORKDIR"/{issuing-ca,controller-ca,controller,overlay-build}
chmod 700 "$WORKDIR"
cd "$WORKDIR"
echo "Working in: $WORKDIR"
ls -la "$WORKDIR"

Resulting layout:

$HOME/octavia-pki/
├── issuing-ca/           # passphrase.txt, .key.enc, .cert.pem
├── controller-ca/        # passphrase.txt, .key.enc, .cert.pem
├── controller/           # .key, .csr, .cert.pem, .bundle.pem, .cnf
└── overlay-build/        # base64 intermediates → consumed by §10

6. Generate Issuing CA

EC P-384 key encrypted with random 32-byte passphrase. Self-signed cert, 10-year validity.

cd "$WORKDIR/issuing-ca"

# Generate passphrase (no trailing newline — required for clean YAML embedding)
openssl rand -base64 32 | tr -d '\n' > passphrase.txt
chmod 600 passphrase.txt

# Sanity-check length (no exit; operator-decided)
PASS_LEN=$(wc -c < passphrase.txt)
if [ "$PASS_LEN" -ne 44 ]; then
  echo "[FAIL] passphrase length is $PASS_LEN bytes, expected 44 — investigate before continuing"
else
  echo "[OK] passphrase length: $PASS_LEN bytes"
fi

On the length check: openssl rand -base64 32 produces 32 random bytes encoded as base64. Base64 encoding of 32 bytes is 44 characters (including two = padding chars). tr -d '\n' strips the trailing newline that openssl adds. The resulting file is exactly 44 bytes — verified by wc -c.

If the length is wrong, stop and re-run the openssl rand line before proceeding. Do NOT continue with a wrong-length passphrase — it will silently break the overlay parsing later.

# Generate EC P-384 private key, encrypted with passphrase
openssl genpkey -algorithm EC \
  -pkeyopt ec_paramgen_curve:P-384 \
  -aes-256-cbc \
  -pass file:passphrase.txt \
  -out issuing-ca.key.enc
chmod 600 issuing-ca.key.enc

# Self-sign cert (10 years, SHA-384)
openssl req -new -x509 -sha384 \
  -key issuing-ca.key.enc \
  -passin file:passphrase.txt \
  -days 3650 \
  -subj "/CN=VR0 DC0 Omega Cloud Octavia Issuing CA/O=Neumatrix" \
  -out issuing-ca.cert.pem

# Verify (no exit; visible output for operator)
echo "=== Issuing CA verification ==="
openssl x509 -in issuing-ca.cert.pem -noout -dates -subject
openssl verify -CAfile issuing-ca.cert.pem issuing-ca.cert.pem
# Expect: issuing-ca.cert.pem: OK

ls -la

7. Generate Controller CA

Identical pattern; different CN.

cd "$WORKDIR/controller-ca"

openssl rand -base64 32 | tr -d '\n' > passphrase.txt
chmod 600 passphrase.txt

PASS_LEN=$(wc -c < passphrase.txt)
if [ "$PASS_LEN" -ne 44 ]; then
  echo "[FAIL] passphrase length is $PASS_LEN bytes, expected 44 — investigate before continuing"
else
  echo "[OK] passphrase length: $PASS_LEN bytes"
fi

openssl genpkey -algorithm EC \
  -pkeyopt ec_paramgen_curve:P-384 \
  -aes-256-cbc \
  -pass file:passphrase.txt \
  -out controller-ca.key.enc
chmod 600 controller-ca.key.enc

openssl req -new -x509 -sha384 \
  -key controller-ca.key.enc \
  -passin file:passphrase.txt \
  -days 3650 \
  -subj "/CN=VR0 DC0 Omega Cloud Octavia Controller CA/O=Neumatrix" \
  -out controller-ca.cert.pem

echo "=== Controller CA verification ==="
openssl x509 -in controller-ca.cert.pem -noout -dates -subject
openssl verify -CAfile controller-ca.cert.pem controller-ca.cert.pem
# Expect: controller-ca.cert.pem: OK

ls -la

Why Controller CA's key is encrypted even though Octavia never uses it: the Controller CA key is needed for future rotations of the controller cert. Encrypting it (with its own passphrase, separate from Issuing CA's) is defense in depth — if the jumphost is compromised, the key still requires the passphrase to be useful for forging controller certs.

8. Generate Controller certificate

EC P-256 key (no encryption — Octavia must read it at startup), CSR with SAN extensions, signed by Controller CA, 2-year validity.

cd "$WORKDIR/controller"

# Generate unencrypted EC P-256 key
openssl genpkey -algorithm EC \
  -pkeyopt ec_paramgen_curve:P-256 \
  -out controller.key
chmod 600 controller.key

# CSR config with SAN extensions
cat > controller.cnf <<'EOF'
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no

[req_distinguished_name]
CN = octavia-controller.omega.dc0.vr0.cloud.neumatrix.local
O = Neumatrix

[v3_req]
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1 = octavia-controller.omega.dc0.vr0.cloud.neumatrix.local
DNS.2 = octavia.omega.dc0.vr0.cloud.neumatrix.local
IP.1 = 10.12.4.233
EOF

# Generate CSR
openssl req -new -sha256 \
  -key controller.key \
  -config controller.cnf \
  -out controller.csr

# Sign with Controller CA (2 years)
openssl x509 -req -sha256 \
  -in controller.csr \
  -CA "$WORKDIR/controller-ca/controller-ca.cert.pem" \
  -CAkey "$WORKDIR/controller-ca/controller-ca.key.enc" \
  -passin file:"$WORKDIR/controller-ca/passphrase.txt" \
  -CAcreateserial \
  -days 730 \
  -extfile controller.cnf \
  -extensions v3_req \
  -out controller.cert.pem

# Bundle cert + key (the lb-mgmt-controller-cert option expects both in one PEM)
cat controller.cert.pem controller.key > controller.bundle.pem
chmod 600 controller.bundle.pem

Verify the chain and SAN:

echo "=== Chain verification ==="
openssl verify -CAfile "$WORKDIR/controller-ca/controller-ca.cert.pem" controller.cert.pem
# Expect: controller.cert.pem: OK

echo ""
echo "=== SAN extensions ==="
openssl x509 -in controller.cert.pem -noout -ext subjectAltName
# Expect:
#     DNS:octavia-controller.omega.dc0.vr0.cloud.neumatrix.local,
#     DNS:octavia.omega.dc0.vr0.cloud.neumatrix.local,
#     IP Address:10.12.4.233

echo ""
echo "=== Validity ==="
openssl x509 -in controller.cert.pem -noout -dates
# Expect: notAfter ~2 years from today

echo ""
echo "=== Bundle integrity (cert and key match) ==="
# Use $HOME paths (not /tmp) per snap-confinement convention used elsewhere in repo
openssl x509 -in controller.bundle.pem -noout -pubkey > "$WORKDIR/controller/.cert.pub"
openssl pkey -in controller.bundle.pem -pubout > "$WORKDIR/controller/.key.pub"
if diff -q "$WORKDIR/controller/.cert.pub" "$WORKDIR/controller/.key.pub" >/dev/null; then
  echo "[OK] bundle cert/key match"
else
  echo "[FAIL] bundle cert/key DO NOT match — investigate before continuing"
fi
rm -f "$WORKDIR/controller/.cert.pub" "$WORKDIR/controller/.key.pub"

9. Final chain verification

A standalone block to confirm the full chain is sound before consuming for Octavia. All three "verify" lines must show : OK. If any do not, stop and investigate before proceeding.

cd "$WORKDIR"

echo "=== Issuing CA ==="
openssl x509 -in issuing-ca/issuing-ca.cert.pem -noout -subject -dates
openssl verify -CAfile issuing-ca/issuing-ca.cert.pem issuing-ca/issuing-ca.cert.pem

echo ""
echo "=== Controller CA ==="
openssl x509 -in controller-ca/controller-ca.cert.pem -noout -subject -dates
openssl verify -CAfile controller-ca/controller-ca.cert.pem controller-ca/controller-ca.cert.pem

echo ""
echo "=== Controller cert ==="
openssl x509 -in controller/controller.cert.pem -noout -subject -dates
openssl verify -CAfile controller-ca/controller-ca.cert.pem controller/controller.cert.pem

Operator-visible check: the three verify lines must all end with : OK.

10. Base64-encode artifacts

Each base64 file is a single line (no wrapping); each becomes one YAML value.

cd "$WORKDIR/overlay-build"

# Issuing CA cert (base64)
base64 -w0 "$WORKDIR/issuing-ca/issuing-ca.cert.pem" > issuing-cacert.b64

# Issuing CA private key (already encrypted PEM → base64)
base64 -w0 "$WORKDIR/issuing-ca/issuing-ca.key.enc" > issuing-ca-private-key.b64

# Controller CA cert
base64 -w0 "$WORKDIR/controller-ca/controller-ca.cert.pem" > controller-cacert.b64

# Controller cert + key bundle
base64 -w0 "$WORKDIR/controller/controller.bundle.pem" > controller-cert.b64

# Sanity-check sizes (expect 500-2000 chars each)
wc -c *.b64

11. Assemble the overlay file

# Read each artifact into shell variables
ISSUING_CACERT=$(cat "$WORKDIR/overlay-build/issuing-cacert.b64")
ISSUING_CA_KEY=$(cat "$WORKDIR/overlay-build/issuing-ca-private-key.b64")
ISSUING_CA_PASS=$(cat "$WORKDIR/issuing-ca/passphrase.txt")
CONTROLLER_CACERT=$(cat "$WORKDIR/overlay-build/controller-cacert.b64")
CONTROLLER_CERT=$(cat "$WORKDIR/overlay-build/controller-cert.b64")

# Assemble overlay (passphrase is YAML-quoted; cert blobs are not — they're
# guaranteed-safe base64 without special chars)
mkdir -p "$REPO/overlays"
cat > "$REPO/overlays/octavia-pki.yaml" <<EOF
# Octavia LBaaS PKI overlay — SENSITIVE — NEVER COMMIT
# Generated: $(date -u +%Y-%m-%dT%H:%M:%SZ) UTC
# Source: docs/v1-do-doc-02-pki.md
# Issuing CA, Controller CA, Controller cert all generated fresh per workstream 3a.
#
# This file is gitignored. If you see it staged or committed, .gitignore is broken.

applications:
  octavia:
    options:
      lb-mgmt-issuing-cacert: ${ISSUING_CACERT}
      lb-mgmt-issuing-ca-private-key: ${ISSUING_CA_KEY}
      lb-mgmt-issuing-ca-key-passphrase: "${ISSUING_CA_PASS}"
      lb-mgmt-controller-cacert: ${CONTROLLER_CACERT}
      lb-mgmt-controller-cert: ${CONTROLLER_CERT}
EOF

chmod 600 "$REPO/overlays/octavia-pki.yaml"

# Unset the shell variables (they held key material)
unset ISSUING_CACERT ISSUING_CA_KEY ISSUING_CA_PASS CONTROLLER_CACERT CONTROLLER_CERT

Validate the overlay parses as YAML:

python3 - <<PY
import yaml
with open("$REPO/overlays/octavia-pki.yaml") as f:
    d = yaml.safe_load(f)
o = d["applications"]["octavia"]["options"]
print("Keys present:", sorted(o.keys()))
print("All values non-empty:", all(v for v in o.values()))
PY

Expected: 5 keys listed; All values non-empty: True.

Confirm gitignore is doing its job:

cd "$REPO"
git status --short
echo ""
echo "If overlays/octavia-pki.yaml appears above as ?? (untracked), STOP."
echo "Shred the file with: shred -uvz overlays/octavia-pki.yaml"
echo "Fix .gitignore and regenerate (§4 + §6-11)."

The overlay file must NOT show up in git status --short. If it does, the gitignore patch in §4 did not stick.

12. Sensitive-file backup

The Issuing CA private key plus its passphrase are the crown jewels of the LB trust domain. Loss → cannot sign new amphora certs (LBs gradually break). Exposure → attacker can forge amphora identities and intercept tenant LB traffic.

Minimum backup for testcloud:

cd "$HOME"
BACKUP_NAME="octavia-pki-backup-$(date +%Y%m%d-%H%M%S).tar.gz"

tar -czf "$BACKUP_NAME" -C "$HOME" octavia-pki/

# Encrypt with strong symmetric cipher (will prompt for passphrase interactively)
gpg --symmetric --cipher-algo AES256 --output "${BACKUP_NAME}.gpg" "$BACKUP_NAME"

# Shred the unencrypted tar (whether gpg succeeded or failed — gpg output is the asset of record)
if [ -f "${BACKUP_NAME}.gpg" ]; then
  shred -uvz "$BACKUP_NAME"
  ls -la "${BACKUP_NAME}.gpg"
  echo "[OK] backup created and unencrypted tar shredded"
else
  echo "[FAIL] gpg encryption did not produce ${BACKUP_NAME}.gpg"
  echo "       Unencrypted tar still present at: $BACKUP_NAME"
  echo "       Investigate gpg failure before continuing."
fi

Move ${BACKUP_NAME}.gpg off-host to your chosen secrets store (admin workstation encrypted drive, password-manager attachment, dedicated secrets vault). Do not leave it on the jumphost long-term — single point of compromise.

Roosevelt note: Vault PKI engine stores all of this; no manual backup required. This procedure is testcloud-only.

13. Cleanup of intermediates

After successful deploy and post-deploy verification (§14), shred files that are not needed for future rotation:

# Optional: shred the base64 intermediates (regeneratable from PEM sources)
shred -uvz "$WORKDIR/overlay-build/"*.b64
rmdir "$WORKDIR/overlay-build"

# Optional: shred the CSR (regeneratable if needed)
shred -uvz "$WORKDIR/controller/controller.csr"

# DO NOT shred any of the following — they are needed for future operations:
#   - issuing-ca/{issuing-ca.cert.pem, issuing-ca.key.enc, passphrase.txt}
#   - controller-ca/{controller-ca.cert.pem, controller-ca.key.enc, passphrase.txt}
#   - controller/{controller.key, controller.cert.pem, controller.bundle.pem, controller.cnf}
#
# Specifically:
#   - Issuing CA artifacts: required for signing new amphoras (Octavia uses them at runtime)
#   - Controller CA artifacts: required for signing new controller certs (rotation)
#   - Controller cert/key: required to repopulate the overlay if jumphost is rebuilt

This step runs AFTER §14 verification has confirmed the overlay was consumed correctly.

14. Post-deploy verification

After v1-do-doc-04-deploy.md completes (juju deploy with the overlay), verify Octavia is healthy and the PKI plumbing works. This section is referenced from §13 above as the verification gate.

# Octavia charm active/idle
juju status octavia
# Expect: octavia/0 active idle

# Octavia services running
juju ssh octavia/0 -- sudo systemctl is-active octavia-api octavia-worker octavia-housekeeping
# Expect: 3x "active"

# Confirm PKI files landed on the unit
juju ssh octavia/0 -- sudo ls -la /etc/octavia/certs/
# Expect: server_ca.cert.pem, server_ca.key.pem, client_ca.cert.pem, client.cert-and-key.pem
# (filenames are charm-controlled; presence is what matters)

# Confirm Octavia can use them — verbose health-check from the API
juju ssh octavia/0 -- sudo journalctl -u octavia-api --since "5 minutes ago" \
  | grep -iE "(cert|ssl|tls|amphora)" | head -20
# Expect: no errors related to cert loading

Smoketest — create a test LB once amphora image is available:

# After octavia-diskimage-retrofit has populated Glance with the amphora image,
# and the LBaaS Mgmt network is wired (these are downstream deploy steps),
# a test LB creation exercises the full PKI chain:

source "$HOME/admin-openrc"
openstack loadbalancer create --name pki-smoketest --vip-subnet-id <provider-subnet>

# Watch for amphora spawn (3-5 minutes typical)
watch -n5 'openstack loadbalancer show pki-smoketest'
# Wait for: provisioning_status=ACTIVE, operating_status=ONLINE

# Octavia-worker log should show successful amphora handshake (signed by Issuing CA,
# trusted via Controller CA):
juju ssh octavia/0 -- sudo journalctl -u octavia-worker --since "10 minutes ago" \
  | grep -iE "(amphora|cert)" | tail -20
# Expect: "amphora <UUID> connection established" or similar
# Expect: no TLS handshake errors, no cert validation errors

# Cleanup the smoketest LB
openstack loadbalancer delete pki-smoketest --cascade

If amphora handshake fails with cert errors, the most likely causes are:

1. SAN mismatch — the controller's connection to amphora uses the cert's CN/SAN; verify the controller cert SAN (§8) covers all addresses Octavia uses to reach amphorae.
2. Bundle/key mismatch — lb-mgmt-controller-cert bundle should contain BOTH the cert and the matching private key; if they're for different keys, handshake fails. (Verified in §8 with the pubkey diff.)
3. Encrypted Issuing CA key + wrong passphrase — verify the passphrase string in the overlay (§11) matches what was used at generation (§6).

15. Roosevelt deltas (forward-look)

When this procedure is adapted for Roosevelt bare-metal deploy:

The procedure structure (generate Issuing CA → Controller CA → Controller cert → encode → overlay → backup → deploy) remains identical. Roosevelt just sources the CA root from Vault instead of self-signing.

16. Rotation/renewal pointer

For testcloud, the 2-year controller cert and 10-year CAs are intentionally "set and forget" — they will outlive the cloud at this scale.

If rotation IS needed before testcloud teardown (e.g., a key leak event), the re-run procedure is:

1. Generate new Controller cert signed by existing Controller CA (re-run §8-9 only).
2. Regenerate the overlay (§11) with the new Controller cert; leave all other values unchanged.
3. juju config octavia lb-mgmt-controller-cert=<new-base64> (single-option update; does not require full bundle redeploy).
4. Octavia services may need a restart: juju ssh octavia/0 -- sudo systemctl restart octavia-api octavia-worker octavia-housekeeping.
5. Existing amphorae will need to reconnect using the new cert; in-flight LBs may briefly drop. This is acceptable for a security-event rotation.

For Roosevelt, this whole procedure is replaced by Vault automated rotation.

17. Acceptance criteria — go/no-go for next step

Before proceeding to Batch B (v1-do-doc-03-destroy.md):

• §4 .gitignore patch applied and effective (overlay file is ignored)
• §6 Issuing CA generated; cert verifies OK; passphrase is 44 bytes
• §7 Controller CA generated; cert verifies OK; passphrase is 44 bytes
• §8 Controller cert generated and signed; chain verifies OK; SAN extensions present; bundle cert/key match
• §9 Final chain verification: all three verify lines show : OK
• §10 Four base64 artifacts produced
• §11 Overlay file written to $REPO/overlays/octavia-pki.yaml; parses as YAML; 5 non-empty option values
• §11 git status --short does NOT show the overlay file
• §12 Encrypted backup created and unencrypted tar shredded; backup moved off-host
• §13 deferred until after the deploy step and §14 verification

If all checked, the overlay is ready for v1-do-doc-04-deploy.md (Batch B). The overlay is consumed by the deploy command:

juju deploy ./bundle.yaml --overlay overlays/octavia-pki.yaml --trust

(Note: only one overlay reference. The deprecated overlays/vr0-dc0-testcloud.yaml placeholder is not used; the bundle has its testcloud values inline.)

18. Change log