Newer
Older
openstack-caracal-dc-dc / docs / audit / vault-init-qol-proposal-20260805.md

Vault-init workflow QoL sweep -- engineering (PROPOSED, not executed)

Filed 2026-08-05. Governing decision: D-142 (design-decisions.md). Status: APPROVED IN PRINCIPLE, IMPLEMENTATION DEFERRED. Operator approved all of R1-R5 ("Yes, I approve of all the recommendations so far") and then directed, 2026-08-05: "We will complete a vault init using current commands, take the qol and updated script/steps developed and we can test them on the next opportunity." So the 2026-08-05 dc0 vault init was run on the CURRENT runbook, unchanged; this document is the saved engineering to implement + TEST next opportunity. R2's off-host transport is UNRESOLVED (open question to operator, below).

Target runbook: runbooks/phase-02-vault-bringup.md. This is a QoL / interaction-shape sweep -- distinct scope from D-068 (vault substrate hardening: version/listener-TLS/ AppRole -- Roosevelt) and D-011.6 (manual-unseal acceptance bar, a preserve-item). Neither governs the init workflow; hence a new number, not an amendment.


Operator safety constraint (verbatim -- binds every change here)

"when the keys are printed (to either file or in the terminal) that at no point is the data caught up with the hidden prompt options that could cause the keys to generate and be lost. It has happened in the past and so changes around this part of the process have to be doublechecked."

Everything below is subordinate to this constraint. The init one-shot's key generation must never be entangled with a hidden prompt, and no change may introduce that entanglement.


Where the friction actually is (the commands are correct; the clunk is the shape)

  1. Three separate interactive juju ssh vault/0 opens (2.1 init, 2.2 unseal, 2.3 token-mint), each carrying the DOCFIX-029 "paste this line alone, wait for the prompt, then paste the block" dance. Single biggest source of clunk.
  2. A cross-session secret hop: the child token is minted inside vault/0, printed, you copy it, exit manually, then paste it into a read -s on the other host. Manual copy of live secret material between two terminals.
  3. Per-line mental substitution: the runbook is VR0-shaped -- -m openstack and run-location "jumphost" throughout. For VR1 every command is really -m vr1-dc0 on the dc0 rack (D-138). It "functions" because you substitute in your head -- friction on every line and a wrong-model foot-gun.
  4. The highest-stakes step is unguarded: saving ~/vault-init/init.txt off-host is a manual "remember to do this" between the init gate and unseal -- and losing it is the unrecoverable B15 catastrophe (DOCFIX-006). Everything else has a GATE; this one relies on discipline. (See R2 -- and its still-open transport question.)

Recommendations (prioritized) -- ALL operator-approved in principle

R1 -- Parameterize for VR1 (HIGH, low risk). Replace -m openstack with -m "$DC_MODEL" and change the run-location legend from "jumphost" to "the DC rack (D-138)". Kills the per-line substitution and the wrong-model error class. This is a defect fix, not a nicety -- -m openstack targets a model that does not exist in VR1 (fails closed here, but against a stale controller could resolve unintended). Belongs on the runbook-fold-register (D-138 retrofit) regardless -- filed there as F13.

R2 -- Make the off-host save an enforced GATE (HIGH). After the init gate, add a checkpoint that will not let 2.2 proceed until the operator confirms the off-host copy exists (paste the secret-store path; the helper sha256-compares it to ~/vault-init/init.txt). Turns the one catastrophic manual step into a gate like every other step. OPEN QUESTION -- transport not yet defined: ~/vault-init/init.txt lives inside the vault/0 juju unit, whose only durable path is inside a container a redeploy destroys. R2 cannot gate on a step with no defined mechanism. Candidate: juju scp -m "$DC_MODEL" vault/0:vault-init/init.txt <operator-secret-store> run from the rack, with R2 verifying by sha256 only, never reading/storing the contents. Must learn the operator's actual off-host practice before building R2 -- build around real practice, not an assumed one. (During the 2026-08-05 run this gap was surfaced verbally at the hand-off and the operator handled the save by hand.)

R3 -- A scripts/vault-init-assist.sh <DC> scaffold (HIGH, guard-safe). A NON-SECRET helper that does everything AROUND the one-shots: runs phase-02-vault-preflight, prints the exact per-DC commands with model/location already substituted (copy-paste ready), runs the read-only vault status / action-schema checks, and does R2's off-host-save verification. It NEVER runs the init/unseal/token one-shots (the guard blocks those, correctly) -- it removes the boilerplate and substitution around them. This is the risky one (a script emitting paste-able commands, in the exact place that has burned us). Build constraints are mandatory -- see below.

R4 -- Cut interactive sessions from 3 -> the genuine minimum (MED). Use non-interactive juju ssh vault/0 -- 'VAULT_ADDR=... vault status' for the pre-check and schema read (no session open, no DOCFIX-029 dance). Reserve interactive ttys for only the three steps that truly need a hidden prompt (init, unseal, token-mint). Since the DISCRETE gate is only between init and unseal, unseal (2.2) and the child-token mint (2.3a) can share one vault/0 session -- one fewer open and one fewer copy hop. Init (2.1) stays its own isolated session -- untouched (see safety analysis).

R5 -- Small conveniences (MED). A read-only vault-status wrapper (the vault status ... | grep -E ... repeated verbatim 3x), and a one-page "at-a-glance" card -- the 3 gates + the save checkpoint -- so you are not scrolling a 179-line runbook mid-secret-op.


PRESERVE -- these are deliberate, not clunk (a QoL pass must NOT erode them)

  • The DISCRETE, un-batched gating (init -> save -> unseal -> authorize) -- a security invariant, not friction.
  • Hidden prompts / read -s, 2>&1 | tee (never >), the short-lived child token instead of the root token, and manual unseal (v1 standard, D-011.6). All intentional; keep them.

Safety analysis -- key/hidden-prompt entanglement (the operator's constraint)

Mechanism of loss. When multiple lines are pasted into an interactive session they BUFFER. If any buffered line opens a hidden prompt (vault operator unseal, read -s), the prompt reads from the buffer and SWALLOWS the next buffered line as the "secret." Two outcomes: (a) a following command line gets eaten as a secret (wrong secret entered; that command never runs), or (b) the dangerous inverse for init -- vault operator init runs (generates the 5 shares + token IRREVERSIBLY, printed once) but its capture is compromised.

Confirmation of the CURRENT process -- Step 2.1 (init) is structurally safe, because three things hold together and all three currently do:

  1. The session open (juju ssh vault/0) is pasted ALONE, waiting for the remote prompt (DOCFIX-029) -- the in-session block is not fed to the SSH connect.
  2. The in-session init block contains NO hidden prompt -- export, vault status, vault operator init ... 2>&1 | tee, two greps. Nothing reads a hidden secret, so nothing can swallow the init line.
  3. The init line is ONE atomic line with 2>&1 | tee and mkdir -p ~/vault-init ahead of it -- keys land in the file even if the display is mangled, and the line cannot be half-executed (swallowed whole = init did not run = no keys generated = safe; or run intact = captured). Because init runs isolated with no preceding hidden prompt, the irreversible key generation cannot be entangled with a hidden prompt.

Residual risk that IS real (advisor-caught -- NOT a hidden-prompt problem). vault operator init can succeed server-side and FAIL the client-side tee write (unwritable/full path). Keys are generated once, irreversibly -- if tee wrote nothing they are gone. The grep -c '^Unseal Key' == 5 GATE catches this only AFTER the fact, when the keys already exist and are already lost. Fix (bake in): a pre-init writability probe -- touch ~/vault-init/.probe && rm ~/vault-init/.probe (with mkdir -p + umask 077 already present) -- prove the capture path is writable BEFORE generating anything irreversible. Strictly additive. (This probe was offered and run manually during the 2026-08-05 dc0 init -- it printed PROBE_OK before the one-shot.)

Latent trap the current runbook mitigates only with discipline. Steps 2.2 (unseal) and 2.3 (token-mint) each present their hidden-prompt commands inside a SINGLE fenced code block (2.2: three vault operator unseal lines together; 2.3: read -s then vault token create). Bulk-pasting either block lets the first hidden prompt swallow the next command line. These are RECOVERABLE (unseal re-runs; token re-mints -- neither is the irreversible init), so not the catastrophic loss -- but the block presentation invites the exact bulk-paste the prose warns against. R3 turns this prose-only warning into a STRUCTURAL safeguard.

Double-check on the proposed changes themselves:

  • R4 does NOT combine init with anything. The discrete gate sits between init and unseal; init (2.1) stays its own isolated session with no hidden prompt. Init isolation preserved. R4 only merges unseal + token-mint (both post-init, both recoverable).
  • R3 will NEVER emit a bulk block containing a hidden-prompt command followed by more lines. It emits the init command as a STANDALONE atomic unit (with mkdir -p + 2>&1 | tee intact and a "paste this line ALONE" banner), and each unseal / read -s step as its OWN separate paste unit with a "wait for the prompt before the next paste" gate. It never runs the one-shots (guard blocks them) and never reads/echoes/stores key material.
  • R2 runs strictly AFTER the init file is confirmed populated -- reads/verifies ~/vault-init/init.txt, never re-runs or restructures the init capture. Compares checksums only.
  • R1 touches only -m openstack -> -m "$DC_MODEL" and the run-location legend.

Build constraints (mandatory when R3 is implemented -- same discipline as dc-node-etchosts T10)

  • The harness MUST mutation-prove that the emitted init unit contains BOTH 2>&1 | tee AND mkdir -p, and that no emitted block puts a hidden-prompt command ahead of another line. (Revert the property -> the case must FAIL.)
  • The script never reads, echoes, paths, or stores key material; R2 compares sha256 checksums only.
  • Assert the emitted artifact, never a comment or the script's own prose about itself.
  • Add the pre-init writability probe to the emitted init unit.

What was already delivered vs deferred

  • DELIVERED this session: this proposal, D-142 (PROPOSED), and F13 on the fold-register.
  • DEFERRED to next opportunity (operator directive): the runbook edits (R1, R4, R5), the scripts/vault-init-assist.sh scaffold + harness (R3), and the R2 save-gate -- the last blocked on the operator's off-host transport answer.