Fork: 0
OpenStack / openstack-caracal-ipv4
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-ipv4 / runbooks / phase-02-vault-bringup.md
@JANeumatrix JANeumatrix 1 day ago 6 KB repo: sanitation sweep -- canonical phase-NN runbook + appendices; design-decisions D-001..D-042; canonical bundle; superseded files moved to remove/ for review
Raw Blame History

Phase 02 -- Vault Bring-up (PKI root; secret-handling)

Initialize, unseal, and authorize Vault -- the cloud's PKI/CA root. This is the SECRET-HANDLING phase: every step is DISCRETE and individually gated (never batched), secrets go through hidden prompts (never on argv / in a var / in scrollback / in a juju action log), and the init key material is saved OFF-HOST immediately.

Decisions: vault-on-mysql backend (etcd/easyrsa dropped -- C1). Troubleshooting: appendix-A -- DOCFIX-006 (init one-shot capture), DOCFIX-011 (authorize-charm token), DOCFIX-014 (generate-root-ca required), L4 (unseal via hidden prompt), R3 (HA Enabled false is correct here).

!!! IRREVERSIBLE ONE-SHOT -- vault operator init runs EXACTLY ONCE per vault. It prints the 5 unseal-key shares and the root token ONE TIME ONLY. A stdout-only > redirect drops the key block to a file that loses stderr and you can be left with NOTHING (DOCFIX-006 / the B15 incident). Run the init command VERBATIM as written in 2.1 (2>&1 | tee), confirm the gate, and SAVE the captured file off-host before doing anything else. Lost shares = unrecoverable vault.

Prerequisites (must be true entering phase-02)

  • phase-01 done: bundle deployed; mysql-innodb-cluster ACTIVE (vault's backend -- it bootstrapped before vault init).
  • vault/0 sits BLOCKED needing initialization (a fresh, uninitialized vault).

Constants and env-literals

  • vault loopback: http://127.0.0.1:8200 (on the unit; NOT a VIP -- this is B14's on-unit loopback model, deliberately not the jumphost-CLI + unit-IP path).
  • key-shares=5, key-threshold=3.

Run-location legend

  • # RUN: on vault/0 -- inside an interactive juju ssh -m openstack vault/0 session (the init/unseal/token-mint steps need a tty for hidden prompts -- do NOT pipe them).
  • # RUN: jumphost -- juju run client calls (authorize / generate-root-ca / status).

Step 2.1 -- Vault init [IRREVERSIBLE ONE-SHOT -- run verbatim] DISCRETE

# RUN: on vault/0 Open the session, set the loopback addr, pre-check fresh, then init with the 2>&1 | tee capture (NOT >). Save ~/vault-init/init.txt off-host the moment the gate passes.

juju ssh -m openstack vault/0
# --- inside the vault/0 session: ---
export VAULT_ADDR=http://127.0.0.1:8200 ; umask 077 ; mkdir -p ~/vault-init
vault status 2>&1 | grep -E 'Initialized|Sealed|Storage Type|HA Enabled' || true   # pre-check: Initialized false (fresh)
vault operator init -key-shares=5 -key-threshold=3 2>&1 | tee ~/vault-init/init.txt # DOCFIX-006: 2>&1|tee, NEVER '>'
grep -c '^Unseal Key' ~/vault-init/init.txt                                         # GATE: MUST print 5
grep -q '^Initial Root Token:' ~/vault-init/init.txt && echo TOKEN_OK || echo MISSING

GATE: 5 unseal keys AND TOKEN_OK. If the count is not 5 or the token is MISSING, STOP -- do not proceed (the empty-file case is the DOCFIX-006 catch). Now SAVE the 5 shares + root token off-host (operator secret store) before continuing. Do NOT batch this with unseal.

Step 2.2 -- Vault unseal (3 of 5) DISCRETE (re-runnable)

# RUN: on vault/0 Use Vault's OWN hidden prompt -- the key is never on the command line, in a var, or in scrollback (appendix-A: L4). Do NOT use vault operator unseal $K (that puts the key in ps/argv).

# --- inside the vault/0 session: ---
export VAULT_ADDR=http://127.0.0.1:8200
vault operator unseal      # prompts hidden; paste share 1   -> Unseal Progress 1/3
vault operator unseal      # prompts hidden; paste share 2   -> 2/3
vault operator unseal      # prompts hidden; paste share 3   -> 3/3
vault status 2>&1 | grep -E 'Sealed|Initialized|Storage Type|HA Enabled'

GATE: progress 1/3 -> 2/3 -> 3/3, then Sealed false. Expected final: Initialized true / Sealed false / Storage Type mysql / HA Enabled false (CORRECT for single-unit vault-on-mysql -- appendix-A: R3; any "HA true / etcd" reference is stale).

NOTE (unseal policy, v1): MANUAL unseal is the v1 standard -- after any vault unit reboot, re-run this 3-of-5 step at the hidden prompt. Auto-unseal (e.g. a transit/KMS seal so the unit returns unsealed after a reboot) is an available option, adopted case-by-case; it is NOT configured in v1. D-011.6 (phase-08) re-confirms manual unseal.

Step 2.3 -- Authorize-charm + generate-root-ca DISCRETE

First confirm the action schema (DOCFIX-011), then authorize with a SHORT-LIVED CHILD token (not the root token -- juju run persists action params in the operation log, so a minutes-lived token self-limits), then generate the root CA (DOCFIX-014 -- without it vault stays blocked "Missing CA cert").

# RUN: jumphost -- schema (read-only): authorize-charm requires `token` (direct-token path)
juju actions vault --schema --format yaml -m openstack | sed -n '/authorize-charm:/,/^[a-z]/p'
# RUN: on vault/0 -- mint a short-lived child token (root entered hidden, never on argv/history)
juju ssh -m openstack vault/0
# --- inside the session: ---
export VAULT_ADDR=http://127.0.0.1:8200
read -s -p "root token: " VAULT_TOKEN; echo ; export VAULT_TOKEN
vault token create -ttl=10m -field=token        # prints ONLY the child token -- copy it
unset VAULT_TOKEN
exit
# RUN: jumphost -- authorize + root CA + status (each juju run blocks to completion)
juju run vault/leader authorize-charm token=<short-lived-child-token> -m openstack
juju run vault/leader generate-root-ca -m openstack
juju status vault -m openstack

GATE: authorize-charm completes; generate-root-ca returns the root CA PEM ("Vault Root Certificate Authority (charm-pki-local)"); vault/0 -> active/idle "Unit is ready". The "Missing CA cert" block clears straight to active (validates DOCFIX-014). (mlock: disabled is expected/benign for snap/container vault without IPC_LOCK.)

EXIT GATE (phase-02 complete)

  • Vault Initialized true / Sealed false; 5 shares + root token saved OFF-HOST.
  • vault/0 active/idle; root CA generated (the cloud's PKI anchor).
  • The narrow cert cascade to the Vault consumers (ovn-central x3, ovn-chassis x3, ovn-chassis-octavia, neutron-api-plugin-ovn, barbican-vault) now proceeds -- it is watched and accepted in phase-03.

As-built reference (2026-06-03 run -- audit trail)

  • init: 5 shares / threshold 3, "Vault initialized with 5 key shares and a key threshold of 3"; captured via 2>&1 | tee ~/vault-init/init.txt.
  • unseal: 1/3 -> 2/3 -> 3/3 -> Sealed false; Storage Type mysql; HA Enabled false; Version 1.8.8, vault-cluster-872a43d1.
  • authorize: op 1/task 2 OK (short-lived child token); generate-root-ca op 3/task 4 returned the root CA (valid 2026-06-03 -> 2036-05-31); vault/0 + vault-mysql-router active.

Next

phase-03 -- core verify (cert-cascade settle, admin-openrc, Horizon).