# Phase 01 -- Bundle Deploy

Deploy the hardened bundle + the octavia-pki overlay onto the freshly-prepped MAAS
machines, and verify it settles to the expected PRE-vault-init state (zero errors,
vault awaiting init, the TLS consumers awaiting vault certs). Vault init is phase-02.

Decisions: B5 (IP-only), D-019 (no designate -- SUPERSEDED 2026-07-10 by D-106,
see the callout below), D-020 (dual provider+metal VIPs), R14 (VIPs front-loaded
to .50-.60), Section-G NIC bindings. Troubleshooting: appendix-A -- R14 (VIP
relocation), R15 (.10 phantom resolver), L1 (no `set -e` on count-gate blocks),
L3 (metal-side dual-VIP eyeball check), DOCFIX-016 (maas list leak).

!!! `bundle.yaml` CHANGED 2026-07-10 (DOCFIX-167, VR1 DC-DC buildout, D-106/
    D-108 both ADOPTED): `designate`/`designate-bind`/`designate-mysql-router`/
    `designate-hacluster` and `cinder-backup`/`ceph-rbd-mirror` are now IN the
    same `bundle.yaml` this phase deploys -- the counts and "NO designate" GATE
    text below are updated to match. **Genuinely unresolved, flagged rather
    than decided:** this is the SAME shared bundle file used for both a plain
    VR0/DC0 single-DC rehearsal redeploy and for standing up VR1's DC1 (per
    the buildout design's own framing, VR0's testcloud IS becoming DC1, not
    staying a separate thing) -- so ANY phase-01 run from this point forward
    gets these new charms, whether the intent is a VR0-only continuation or a
    DC-DC DC1 build. If a future session needs a plain VR0-only redeploy
    WITHOUT Designate/cinder-backup/ceph-rbd-mirror, that needs an explicit
    decision (a bundle fragment/override removing them for that one run) --
    do not assume either interpretation silently.

---

## Prerequisites (must be true entering phase-01)
- phase-00 done: 4 machines Ready/power=off; MAAS carve applied (front-loaded VIP /26
  reserved, FIP pool reserved, stale iprange gone); enp8s0 data NIC linked on ALL four
  hosts; OSD `/dev/vdb` wiped blank.
- `overlays/${DC}-octavia-pki.yaml` present (Step 1.0). PER-DC since F1 (2026-07-29) --
  export `DC` (region-qualified, e.g. `vr1-dc0`) before running any step in this phase.
- Hardened `bundle.yaml` in the working dir (channels pinned; VIPs `.50-.60`;
  reserved-host-memory 8192; image-conversion; use-policyd-override).

**GATE (DOCFIX-073) -- run THE pre-deploy gate before add-model:**
```bash
bash scripts/preflight.sh
```
Expect `PREFLIGHT: PASS` (repo lint + bundle invariants + Charmhub channel
assert + live MAAS pre-flight, aggregated). WARN = review then decide; FAIL =
do not deploy. This is the single entry point that replaced the six separate
gate artifacts an operator previously had to remember.

## Constants and env-literals
- Host identity / system_ids: resolve LIVE via `scripts/lib-hosts.sh` (DOCFIX-040:
  system_ids are re-minted on every reenrollment -- a documented id is stale by
  definition; never paste one into a command).
- Planes / CIDRs / VIP bands: `scripts/lib-net.sh` is the single authority
  (D-052/D-053). Resolve MAAS subnets BY CIDR (PATTERN-1), never by numeric
  subnet id -- ids drift across MAAS lifecycles. (DOCFIX-068 removed a stale
  pre-D-052 literal map that sat here.)
- expected plan (as of `bundle.yaml` post-DOCFIX-167, 2026-07-10): 56 apps,
  108 relations, 4 machines (bundle 8/9/10/11 -> juju 0/1/2/3), 24 LXD. (Was
  50 apps/97 relations before the Designate + cinder-backup + ceph-rbd-mirror
  reactivation -- see the callout above.)

## Run-location legend
- `# RUN: jumphost` -- `juju` + `maas admin` (MAAS profile is `admin`; never `maas list` -- DOCFIX-016).

---

## Command-label convention
Every command block below is bracketed by bold labels, so a command line is never mistaken
for surrounding prose (these render in GitBucket and read clearly in a raw editor):
- **RUN -- LOC** -- the block CHANGES state; run it at LOC (e.g. `jumphost`, `vault/0`, `jumphost -> magnum/0`).
- **CHECK (read-only) -- LOC** -- a read-only verification; safe to re-run.
- **GATE:** -- a hard stop; do NOT proceed past the block unless the stated condition holds.
- **Expect:** -- what a passing result looks like.
- `> CAUTION:` -- marks a destructive, secret-handling, or irreversible step.


## Step 1.0 -- Octavia PKI overlay (secret-handling prereq)  DISCRETE
`overlays/${DC}-octavia-pki.yaml` carries the 5 lb-mgmt-* PKI keys (controller CA/cert,
issuing CA key+passphrase+cert). It is the ONLY overlay in the deploy command and is
secret-safe + ASCII. PRIMARY path: reuse the existing validated overlay (the CAs are
10y, so it survives rebuilds). REGENERATION path (fresh CAs): run the discrete secret
procedure inlined as "Step 1.0-GEN" at the end of this phase. Either way, confirm the
overlay parses and contains exactly the 5 keys (sanity block below) before deploying.

**CHECK (read-only) -- jumphost**
```bash
# RUN: jumphost -- sanity only (does NOT print key material)
DC="${DC:?export DC=vr1-dc0|vr1-dc1 -- the PKI overlay is per-DC (F1)}"
OVL="overlays/${DC}-octavia-pki.yaml"
# Assert on CONTENT, not existence: an absent file, a wrong key count and a non-ASCII
# byte are three DIFFERENT answers and none of them may read as OK.
if [ ! -f "$OVL" ]; then
  echo "MISSING $OVL -- generate it with Step 1.0-GEN for DC=$DC"
else
  K=$(grep -cE 'lb-mgmt-' "$OVL" || true)
  [ "$K" -eq 5 ] && echo "OK 5 lb-mgmt-* keys" || echo "FAIL $K lb-mgmt-* keys (want 5)"
  LC_ALL=C grep -qP '[^\x00-\x7F]' "$OVL" && echo "FAIL non-ASCII bytes" || echo "OK ASCII clean"
fi
```

## Step 1.1 -- Pre-deploy verify (read-only; 4 checks)
One consolidated read-only block. NO `set -e` (a guarded count of
0 is a valid answer, not a failure -- appendix-A: L1); count greps are `|| true`.

**CHECK (read-only) -- jumphost**
```bash
( {
  echo "=== CHECK 1: bundle VIPs (quote-tolerant, octet-anchored) ==="
  grep -nE '^[[:space:]]+vip:' bundle.yaml overlays/vr1-dc0-vips.yaml
  TOT=$(grep -hcE '^[[:space:]]+vip:[[:space:]]*"?10\.12\.4\.' bundle.yaml overlays/vr1-dc0-vips.yaml 2>/dev/null | awk '{s+=$1} END{print s+0}')
  HI=$(grep -hcE '^[[:space:]]+vip:[[:space:]]*"?10\.12\.4\.(5[0-9]|6[0-2])("|$|[[:space:]])' bundle.yaml overlays/vr1-dc0-vips.yaml 2>/dev/null | awk '{s+=$1} END{print s+0}')
  LO=$(grep -hcE '^[[:space:]]+vip:[[:space:]]*"?10\.12\.4\.(1[0-9]|20)("|$|[[:space:]])' bundle.yaml overlays/vr1-dc0-vips.yaml 2>/dev/null | awk '{s+=$1} END{print s+0}')
  echo "  provider VIPs total=$TOT  in .50-.60=$HI  in .10-.20(stale)=$LO   (want 13/13/0)"
  # RULING 3 (2026-07-25): the counts span bundle.yaml AND its per-DC VIP overlay,
  # because the base is now VIP-FREE. `grep -c` over two files prints one count PER
  # FILE, so these sum with awk -- a bare $( ) would capture a two-line string and
  # every numeric test below would silently fail.
  # metal side is the second token of each dual vip; eyeball that all 11 are .8.50-.60,
  # clear of metal infra .8.10(maas)/.8.20(lxd)/.8.21(capi)/.8.30(juju) -- appendix-A: L3.

  echo "=== CHECK 2: enp8s0 data NIC linked on ALL FOUR hosts (10.12.12.0/22) ==="
  for SID in 4na83t qdbqd6 h8frng tmsafc; do
    echo -n "  $SID: "
    maas admin interfaces read "$SID" | jq -r '.[] | select(.name=="enp8s0")
      | [.links[]? | select(.subnet.cidr=="10.12.12.0/22") | .ip_address] | join(",")'
  done   # expect 10.12.12.40 / .41 / .42 / .43 (select by .subnet.cidr -> robust to id drift)

  echo "=== CHECK 3: subnet DNS resolvers ==="
  for ID in 1 2 6 7 8 9; do maas admin subnet read "$ID" | jq -c '{id,cidr,dns_servers}'; done
  # expect subnet 1 (provider) -> [10.12.4.1]; 2/6/7/8/9 -> [10.12.8.1]

  echo "=== CHECK 4a: nodes Ready / power off ==="
  maas admin machines read | jq -r '.[] | select(.system_id|IN("4na83t","qdbqd6","h8frng","tmsafc"))
    | "\(.hostname) \(.status_name) power=\(.power_state)"'
} )
```

**CHECK (read-only) -- jumphost**
```bash
# CHECK 4b: OSD /dev/vdb blank (DOCFIX-027 -- LOCAL libvirt-host loop, NOT ssh: the four
# hosts are Released/powered-off entering phase-01, and /var/lib/libvirt/images is a
# hypervisor (jumphost) path that does not exist on the hosts. RUN: jumphost (libvirt host; sudo).
for h in openstack0 openstack1 openstack2 openstack3; do
  echo "== $h =="
  sudo qemu-img info "/var/lib/libvirt/images/${h}-1.qcow2" | grep -E 'virtual size|disk size'
done   # expect virtual 512 GiB, disk ~200 KiB (sparse/blank)
```
**GATE:** VIPs 11/11/0; enp8s0 linked on all 4; subnet DNS as above; 4 nodes Ready; OSD blank.

## Step 1.2 -- Dry-run (guarded)
Refuse to add a model if `openstack` already exists; require the overlay.

**RUN -- jumphost**
```bash
( {
  juju models 2>&1 | tee /tmp/jmodels.txt
  if grep -qE '(^|[[:space:]]|/)openstack([[:space:]*]|$)' /tmp/jmodels.txt; then
    echo "ABORT: an 'openstack' model already exists (teardown is phase-00)"; 
  elif [ ! -f "overlays/${DC:?export DC first}-octavia-pki.yaml" ]; then
    echo "ABORT: overlays/${DC}-octavia-pki.yaml missing (Step 1.0-GEN for DC=$DC)";
  else
    juju add-model openstack
    juju deploy ./bundle.yaml --overlay "overlays/${DC}-vips.yaml" \
      --overlay "overlays/${DC}-octavia-pki.yaml" -m openstack --dry-run
  fi
} )
```
**GATE:** (from the plan): 56 apps, 108 relations, 4 machines (8/9/10/11 -> 0/1/2/3), 24 LXD;
ceph-osd/0-3 one per node; nova-compute/0-2 on machines 1/2/3 ONLY (machine 0 =
OSD+LXD host, no compute); channels match the matrix; relations include
`octavia:certificates - vault:certificates`, `vault:shared-db - vault-mysql-router`,
`mysql-innodb-cluster:certificates - vault:certificates`; NO `vault:ha` (still
absent, unchanged); designate/designate-bind/designate-mysql-router/designate-
hacluster and cinder-backup/ceph-rbd-mirror NOW PRESENT (D-106/D-108,
2026-07-10 -- corrects this GATE's old "NO designate (D-019)" text, since
D-019 is superseded). Only the two benign R11 warnings (L34 `name`, L55
`variables`).

## Step 1.2b -- metal-internal /etc/hosts cloudinit  (VR1 multi-space DCs ONLY; MUST precede Step 1.3)
VR1's D-052 three-plane isolation puts the OVN NB/SB DB + certificates on the **isolated**
metal-internal plane (no gateway, no route to the region resolver). charm-ovn-central derives
its TLS common_name from `get_hostname(its metal-internal address)`; that reverse lookup has
no reachable resolver on the isolated plane, so the CN comes out EMPTY, vault issues no server
cert, and the OVN cluster never forms (6641/6642 never listen). PROVEN live 2026-08-04 (a
controlled single-unit test: an `/etc/hosts` reverse entry -> CN populated -> vault issued the
per-unit server cert -> `/etc/ovn/{cert_host,key_host,ovn-central.crt}` written; the two
control units without it stayed broken). Full record: `docs/audit/ovn-central-cert-reeval-20260803.md`.
This sets a model-wide `cloudinit-userdata` that, at machine provision, gives each node a LOCAL
`/etc/hosts` reverse mapping for its metal-internal address. It BORROWS D-008's shape but is a
NEW mechanism (D-008's /etc/hosts is FQDN->VIP for os-public-hostname, unset here). It MUST be
set AFTER `add-model` (Step 1.2) and BEFORE `juju deploy` (Step 1.3) -- it only affects machines
created afterwards. SKIP for VR0 (single flat `metal`, no isolated plane -- the script REFUSES
when a site has no metal-internal plane). Runs where the juju client is (the DC rack for VR1, D-138).

**RUN -- (DC rack, D-138)**
```bash
( {
  DC="${DC:?export DC=vr1-dc0|vr1-dc1 first}"
  JUJU_MODEL="${DC_MODEL:?export DC_MODEL first}" bash scripts/dc-node-etchosts.sh apply "$DC" \
    && JUJU_MODEL="${DC_MODEL}" bash scripts/dc-node-etchosts.sh check "$DC"
} )
```
**GATE:** `check` exits 0 -- the model's `cloudinit-userdata` carries this DC's metal-internal
CIDR (from `lib-net.sh`) and the `/etc/hosts` runcmd. VERIFY-AT-PROVISION: after the first
machines come up in Step 1.3/1.4, confirm on one unit that `/etc/hosts` carries its
`10.12.<internal>.x <hostname>` line and `get_hostname(<that ip>)` resolves; this is the only
point the DELIVERY (vs. the rendered content) is provable.

## Step 1.3 -- Deploy (VIP-guarded)
Re-run the VIP guard inline (the dry-run never echoes vip values), then deploy only if
the guard exits 0.

> **THE GUARD IS NOW THE MAINTAINED CHECKER, NOT A HAND-ROLLED GREP (2026-07-29).** This
> block used to compute a `13/13/0` triple with three `grep -hcE` counts anchored on the
> literal `10\.12\.4\.`. Three defects, all measured: it hard-ABORTED on `vr1-dc1`, whose
> provider band is `10.12.64.x`; it counted only IPv4, so after R2's dual-stack ADD the
> ruled v6 legs were checked by NOTHING (chain-audit finding 22); and its prose said
> "11/11/0" while its code demanded `13`, a drift that appeared the moment R11 added
> vault `.61` and designate `.62`. `scripts/provider-bundle-check.py` already encodes
> every one of those invariants -- per-DC bands, the VIP count, and since 2026-07-28 the
> dual-family ARITY coupling -- so the runbook defers to it instead of keeping a second
> copy that drifts. One source of truth; a new ruling updates the checker and this step
> inherits it.

**RUN -- jumphost**
```bash
( {
  DC="${DC:?export DC=vr1-dc0|vr1-dc1 first}"
  set -- ./bundle.yaml
  for OVL in "overlays/${DC}-vips.yaml" "overlays/${DC}-machines.yaml" \
             "overlays/${DC}-octavia-pki.yaml"; do
    [ -f "$OVL" ] && set -- "$@" --overlay "$OVL"
  done
  # The octavia-pki overlay is REQUIRED for deploy (it carries the 5 lb-mgmt-* keys) --
  # absence is an ABORT, not a silently-skipped overlay.
  [ -f "overlays/${DC}-octavia-pki.yaml" ] || {
    echo "ABORT: overlays/${DC}-octavia-pki.yaml missing -- run Step 1.0-GEN for DC=$DC"; exit 1; }

  echo "validating the MERGED $DC deploy input: $*"
  if python3 scripts/provider-bundle-check.py "$@" --dc "$DC"; then
    juju deploy "$@" -m "${DC_MODEL:?export DC_MODEL first}"
  else
    echo "ABORT: VIP/bundle invariants failed for $DC -- fix before deploying"; exit 1
  fi
} )
```

CONVERGENCE WATCH (ENHANCEMENT-1): keep two windows open for the whole deploy arc.
- Window 1 (detail): `juju status -m openstack --watch 5s` (always explicit -m; on a 50-app
  model the table exceeds one screen, so this is a slice -- optionally filter, e.g. `... 'mysql*' vault`).
- Window 2 (signal): `scripts/deploy-watch.sh openstack 15` -- compact machine/unit state counts
  + named error/blocked units. Health-at-a-glance: the error/blocked section stays EMPTY until
  the expected late blocks (vault needs-init, octavia awaiting-configure). Neither window
  descends into subordinates; neither replaces the phase gates.

## Step 1.4 -- DNS gate during deploy (as machines come up)
Run when machine 0 reaches `started`, then per LXD unit as they
appear (flag BEFORE the target; logic inside the remote quotes; no outer 2>/dev/null):

**CHECK (read-only) -- jumphost**
```bash
juju ssh -m openstack 0 -- 'resolvectl status | grep -i "DNS Server"; getent hosts api.snapcraft.io && echo OK || echo FAIL'
# repeat for ceph-mon/0, mysql-innodb-cluster/0 as they appear
```
**GATE:** each returns OK (api.snapcraft.io resolves -> the snap install storm proceeds
clean). FINDING (non-blocking, R15): the unreachable region resolver `10.12.8.10`
(MAAS region/rack controller, advertised on the metal VLAN independent of the subnet
field) may still appear in a node's resolver list -- resolution succeeds because
systemd-resolved deprioritizes `.10` and falls through to `.1`. Latent fragility if
`.1` ever drops; understand/eliminate for Roosevelt. (appendix-A: R15.)

---

## EXIT GATE (phase-01 complete)
- Deploy settled to the PRE-vault-init end state:
  * ZERO units in `error`.
  * mysql-innodb-cluster x3 ACTIVE ("Cluster is ONLINE").
  * vault/0 BLOCKED "Vault needs to be initialized" (the phase-02 trigger, not a fault).
  * Waiting on vault certs (expected pre-init): ovn-central x3, ovn-chassis x3
    (incl nova-compute subordinates), ovn-chassis-octavia, neutron-api-plugin-ovn, barbican-vault.
  * octavia BLOCKED "Awaiting configure-resources" (D-021); gss unknown (pre-run).
  * **NOT YET OBSERVED LIVE (2026-07-10, D-106/D-108 reactivation) -- add to this
    checklist once a real deploy confirms it:** `designate` and `designate-bind`
    have a `certificates: metal-internal` relation to vault (per `bundle.yaml`),
    so BY THE SAME PATTERN as the apps above they are LIKELY also waiting on
    vault certs pre-init -- inferred from the relation graph, not confirmed
    against a real deploy. `cinder-backup` (subordinate) and `ceph-rbd-mirror`
    have no certificates relation, so no equivalent expectation for them. If a
    real deploy shows anything different for designate/designate-bind, correct
    this list rather than trusting this inference.
  * magnum/0 BLOCKED "Ports which should be open, but are not: 9501" -- pre-vault posture:
    magnum-api is loopback-bound ([api] host not yet templated) and haproxy backends target
    unit IPs. EXPECTED phase-01 end-state; self-resolves at the phase-02 cert rollout (apache2
    takes *:9501). Confirmed self-resolving 2026-06-12 (FINDING-2); verify in the phase-02 post-init sweep.
  * keystone/0 "PO (broken): Unit is ready" -- expected while use-policyd-override=true with
    no policy zip attached (FINDING-1); keystone runs the DEFAULT policy. No mutation this arc.
- Section-G NIC payoff confirmed (no subset/binding errors): ceph-mon -> storage 10.12.16.x;
  octavia -> data 10.12.12.1; nova-compute -> data 10.12.12.4x; vault -> metal 10.12.8.x.
- Proceed to phase-02 (vault init).

## As-built reference (2026-06-03 second redeploy -- audit trail)
- `juju deploy ./bundle.yaml --overlay overlays/octavia-pki.yaml -m openstack` on maas/default (cred maas-api).
- Plan: 50 apps / 97 relations / 4 machines / 24 LXD; placement as above.
- Pre-deploy verify: VIPs 11/11/0; enp8s0 -> 10.12.12.40-43 (all 4); subnet DNS as above; nodes Ready; OSD blank.
- Settled: zero errors; mysql /0 R/W (10.12.8.173), /1 (.179) /2 (.185) R/O; vault blocked needs-init.

## Balance / stability observations (Roosevelt rebalancing inputs -- post-deploy item 6)
- Quorum triads (mysql-innodb-cluster, ovn-central, ceph-mon) all on machines 0/1/2: correct
  anti-affinity; machine 3 loss breaks no quorum; any single loss of 0/1/2 leaves a 2-of-3 majority.
- Machine 0 = no-compute control host, largest container count: prefigures the Roosevelt role split.
- FLAG: machine 3 concentrates six singletons (vault, glance, nova-cloud-controller, octavia,
  placement, barbican) + compute + OSD. Acceptable on testcloud; Roosevelt answer is role split + HA,
  informed by measured footprints at the 32 GiB envelope (DEVIATION-2 caveat).
- rabbitmq-server single unit: messaging SPOF, as designed for v1.

## PATTERN-1 (standing convention) -- dynamic lookup vs. pinned identifiers
READ/VERIFY ops discover values at runtime (never hardcode what resolves: hostname->system_id via
`maas admin machines read | jq`; subnet id by CIDR). DESTRUCTIVE/IRREVERSIBLE ops discover
dynamically, ASSERT against a pinned EXPECTED set, ABORT on mismatch, then operate on the pinned
values (a filter bug or an unexpected new machine must not become collateral damage). Retrofit
candidates (apply as fixture-tested gated blocks, NOT bulk edits): phase-00 release/host loops ->
discover-assert-pin; subnet ids -> resolve by CIDR; octet maps -> derive from hostname index.
Canonical statement in runbooks/README.md.

## Next
phase-02 -- vault bring-up.

---

## Step 1.0-GEN -- Octavia management-PKI generation (regeneration path)  DISCRETE / SECRET

> CAUTION: SECRET step -- generates Octavia CA private keys + passphrases. Do NOT echo or log key
> material (only cert dates/subjects + verify-OK are printed). The overlay it writes is gitignored.

Run ONLY if this DC has no PKI yet -- 1.0-GEN.0 REFUSES if `overlays/${DC}-octavia-pki.yaml`
or this DC's workspace already exists, because regenerating invalidates every amphora
already issued against that CA. Each DC is generated separately. Produces the
two-tier EC PKI for Charmed Octavia's amphora trust domain and writes the overlay.
Decisions (Workstream 3a, 2026-05-22): fresh generation; EC P-384 CAs (SHA-384, 10y);
EC P-256 controller cert (2y); overlay-file distribution (gitignored); artifacts under
`$HOME/octavia-pki/`; passphrases = 32 random bytes base64 (44 chars). SECRET step --
do NOT echo key material; the only printed values are cert dates/subjects and verify OK.

The five `octavia` charm options the overlay sets:
- `lb-mgmt-issuing-cacert`            = base64(issuing CA cert)
- `lb-mgmt-issuing-ca-private-key`    = base64(issuing CA ENCRYPTED key)
- `lb-mgmt-issuing-ca-key-passphrase` = the issuing CA passphrase (PLAIN string, NOT base64)
- `lb-mgmt-controller-cacert`         = base64(controller CA cert)
- `lb-mgmt-controller-cert`           = base64(controller cert + key, concatenated)

### 1.0-GEN.0 -- select the DC, then the workspace (openssl 3.x; $HOME only -- snap home-confinement, never /tmp)

> **EVERY PATH IN THIS STEP IS PER-DC (F1, 2026-07-29).** The D-109 amendment rules
> "each DC gets its own Octavia CA; no cross-DC amphora root-of-trust". The 2026-07-27
> R7 work parameterised the CA SUBJECT and the VIP gate but NOT the paths, so the
> workspace and the overlay were single fixed locations: generating a second DC's PKI
> overwrote the first DC's issuing-CA key, controller-CA key and both passphrases, and
> left the one fixed-name overlay carrying the wrong DC's CA -- which a later redeploy
> of the first DC would then apply to it. Set `DC` ONCE here; everything below derives
> from it, paths included.
>
> **RUN LOCATION RULED 2026-07-29 (D-109 ruling note (b)) -- this step runs ON `voffice1`,
> the D-128 Plane-2 headend, NOT the vcloud jumphost.** `juju` is ABSENT on vcloud and 3.6.27
> on voffice1, and `dc-dc-phase4` deploys from voffice1 -- so generating here on the jumphost
> would mint the PKI on a host the deploy cannot read it from. Generate-then-transfer was
> declined: a second at-rest copy of a CA private key plus a plaintext passphrase widens the
> SEC-004 exposure R7 refused to widen. **`$REPO` below therefore means the HEADEND clone**,
> and the artifacts are registered at `host-role=headend`, not `jumphost`.

**RUN -- voffice1 (the D-128 Plane-2 headend; RULED 2026-07-29)**
```bash
# RUN: voffice1
export DC="${DC:-vr1-dc0}"          # vr1-dc0 | vr1-dc1 -- region-qualified, never bare dcN
export DC_LABEL="$(printf '%s' "$DC" | tr 'a-z-' 'A-Z ')"   # e.g. "VR1 DC0"
export REPO="${REPO:?REPO not set -- export REPO=<path to your clone> (see runbooks/README.md Conventions)}"
export VIP_OVERLAY="$REPO/overlays/${DC}-vips.yaml"
export OCTAVIA_PKI_OVERLAY="$REPO/overlays/${DC}-octavia-pki.yaml"
[ -r "$VIP_OVERLAY" ] || { echo "ABORT: no VIP overlay at $VIP_OVERLAY -- is DC=$DC correct?"; exit 1; }

WORKDIR="$HOME/octavia-pki/$DC"     # PER-DC (F1) -- never $HOME/octavia-pki directly
mkdir -p "$WORKDIR"/issuing-ca "$WORKDIR"/controller-ca "$WORKDIR"/controller
chmod 700 "$HOME/octavia-pki" "$WORKDIR"
openssl version    # expect OpenSSL 3.x
echo "DC=$DC  WORKDIR=$WORKDIR  overlay=$OCTAVIA_PKI_OVERLAY"
```

**REFUSE-IF-PRESENT gate -- run before generating anything.** Regeneration destroys the
CA this DC's already-deployed amphorae trust, so it must be a deliberate act, not the
default outcome of re-running a step.

```bash
# RUN: voffice1
if [ -e "$WORKDIR/issuing-ca/issuing-ca.key.enc" ] || [ -e "$OCTAVIA_PKI_OVERLAY" ]; then
  echo "REFUSING: PKI material already exists for DC=$DC."
  echo "  workspace: $WORKDIR"
  echo "  overlay:   $OCTAVIA_PKI_OVERLAY"
  echo "Regenerating invalidates every amphora already issued against this CA."
  echo "To deliberately regenerate: move the existing material aside first, gated."
  exit 1
fi
echo "OK: no existing PKI for DC=$DC -- safe to generate"
```

### 1.0-GEN.a -- Issuing CA (EC P-384, AES-256 encrypted key, self-signed 10y)

**RUN -- voffice1 (the D-128 Plane-2 headend; RULED 2026-07-29)**

> **R7 (D-109 AMENDMENT, RULED 2026-07-27) -- this generator is PER-DC.** It used to
> bake `VR0 DC0` into both CA subjects and gate the VIP on `^10\.12\.4\.`, so it could
> only ever produce a dc0 artifact -- and `phase-01` hard-ABORTs without the overlay,
> which made every other DC undeployable. Reuse across DCs was refused on posture: the
> overlay carries CA private keys plus a plaintext passphrase in a repo SEC-004 records
> as PUBLIC, so one shared amphora CA across two clouds D-100 defines as independent
> would widen an existing exposure. `DC`, `DC_LABEL`, `VIP_OVERLAY` and
> `OCTAVIA_PKI_OVERLAY` are exported ONCE in **1.0-GEN.0** above -- along with the
> per-DC `WORKDIR` that F1 added -- and every block below derives from them.

```bash
( {
  WORKDIR="$HOME/octavia-pki/${DC:?DC not set -- run 1.0-GEN.0 first}"
  cd "$WORKDIR/issuing-ca" || exit 1   # dir from 1.0-GEN.0
  openssl rand -base64 32 | tr -d '\n' > passphrase.txt
  chmod 600 passphrase.txt
  test "$(wc -c < passphrase.txt)" -eq 44 || { echo "ABORT: issuing passphrase length != 44"; exit 1; }
  openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-384 \
    -aes-256-cbc -pass file:passphrase.txt -out issuing-ca.key.enc
  chmod 600 issuing-ca.key.enc
  openssl req -new -x509 -sha384 -key issuing-ca.key.enc -passin file:passphrase.txt \
    -days 3650 -subj "/CN=${DC_LABEL:?DC_LABEL not set -- run 1.0-GEN.0; an unset label bakes a DC-less CA subject with NO error} Omega Cloud Octavia Issuing CA/O=Neumatrix" \
    -out issuing-ca.cert.pem
  openssl x509 -in issuing-ca.cert.pem -noout -dates -subject
  openssl verify -CAfile issuing-ca.cert.pem issuing-ca.cert.pem    # expect: OK
} )
```

### 1.0-GEN.b -- Controller CA (EC P-384, AES-256 encrypted key, self-signed 10y; own passphrase)
The controller CA key is encrypted (its own passphrase) for future controller-cert
rotation -- Octavia never receives this key, only the controller CA cert.

**RUN -- voffice1 (the D-128 Plane-2 headend; RULED 2026-07-29)**
```bash
( {
  WORKDIR="$HOME/octavia-pki/${DC:?DC not set -- run 1.0-GEN.0 first}"
  cd "$WORKDIR/controller-ca" || exit 1   # dir from 1.0-GEN.0
  openssl rand -base64 32 | tr -d '\n' > passphrase.txt
  chmod 600 passphrase.txt
  test "$(wc -c < passphrase.txt)" -eq 44 || { echo "ABORT: controller passphrase length != 44"; exit 1; }
  openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-384 \
    -aes-256-cbc -pass file:passphrase.txt -out controller-ca.key.enc
  chmod 600 controller-ca.key.enc
  openssl req -new -x509 -sha384 -key controller-ca.key.enc -passin file:passphrase.txt \
    -days 3650 -subj "/CN=${DC_LABEL:?DC_LABEL not set -- run 1.0-GEN.0; an unset label bakes a DC-less CA subject with NO error} Omega Cloud Octavia Controller CA/O=Neumatrix" \
    -out controller-ca.cert.pem
  openssl x509 -in controller-ca.cert.pem -noout -dates -subject
  openssl verify -CAfile controller-ca.cert.pem controller-ca.cert.pem    # expect: OK
} )
```

### 1.0-GEN.c -- Controller cert (EC P-256 UNENCRYPTED, SAN, signed by Controller CA, 2y)
The P-256 key is unencrypted -- Octavia reads it at startup. SAN carries the controller
FQDN, the octavia API FQDN, and the Octavia API VIP (derived from the bundle at generation time -- DOCFIX-067; never a baked literal).

> ## CAUTION -- THE `CNF` HEREDOC MUST START AND END AT COLUMN 1 (F8, hit live 2026-07-29)
>
> The block below sits inside an indented `( { ... } )` subshell, but the heredoc body and its
> closing `CNF` are at column 1 and **must stay there**. Copy-pasting this step with the
> indentation preserved gives a terminator that no longer matches: the heredoc never closes,
> and the rest of the block is swallowed as config text.
>
> **The failure is silent, which is why this warning is a heading and not a footnote.** Depending
> where the paste breaks, `controller.cnf` can end up without its `[alt_names]` section -- which
> produces a controller certificate with **NO SANs AT ALL** while every `openssl` command in the
> chain still prints OK. The operator hit exactly this on 2026-07-29 and had to correct the block
> before it would run.
>
> **NOT rewritten as `printf` lines here, deliberately.** That would be more paste-proof, but it
> is an untested rewrite of a step that mints 10-year CA material and cannot be exercised
> end-to-end from an agent session (the PreToolUse guard blocks the mint by design). The residual
> risk is now bounded instead: `scripts/octavia-pki.sh verify` asserts the SAN set
> (`A9`), so a SAN-less certificate FAILS a gate rather than shipping -- harness case `T2`.
> Converting to `printf` is a reasonable follow-up when someone can run a real generation.

**RUN -- voffice1 (the D-128 Plane-2 headend; RULED 2026-07-29)**
```bash
( {
  WORKDIR="$HOME/octavia-pki/${DC:?DC not set -- run 1.0-GEN.0 first}"
  # AS EXECUTED 2026-07-30 (operator's correction, adopted): fail fast on an unset VIP_OVERLAY
  # rather than reaching the readability test with an empty path.
  VIP_OVERLAY="${VIP_OVERLAY:?VIP_OVERLAY not set -- run 1.0-GEN.0 first}"
  cd "$WORKDIR/controller" || exit 1   # dir from 1.0-GEN.0
  openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out controller.key
  chmod 600 controller.key
  # THE ZONE IS DERIVED FROM $DC, NEVER TYPED (F9 fixed AT SOURCE, 2026-07-30).
  # This block used to bake `omega.dc0.vr0` as a literal in THREE places -- the CN and both
  # DNS SANs -- so every DC generated from it inherited VR0's region, and vr1-dc1 inherited
  # dc0's LABEL as well. That is exactly what had to be repaired on both live DCs by Step
  # 1.0-REISSUE. Leaving the literal here would have recreated the defect at the next DC
  # standup, which is the whole reason it is derived now. Same two expansions the tool uses
  # (`scripts/octavia-pki.sh`), so the runbook and the gate cannot disagree:
  DC_REGION="${DC%%-*}"                      # vr1-dc0 -> vr1  (D-008 <region>)
  DC_DNS_LABEL="${DC#*-}"                    # vr1-dc0 -> dc0  (D-008 <dc>, 0-indexed per D-117)
  # The <cloud> label and base domain come from the ONE place they live (2026-07-30), so a
  # rebuild that renames the estate edits scripts/lib-identity.sh and nothing else.
  # ${REPO:?} guarded HERE, not only at :538 below: this is now the FIRST use of $REPO in
  # this step, and an unset one would try to source "/scripts/lib-identity.sh" and fail with
  # a path error instead of naming the actual mistake.
  source "${REPO:?REPO not set -- run 1.0-GEN.0 first}/scripts/lib-identity.sh" \
    || { echo "FAIL: cannot read \$REPO/scripts/lib-identity.sh -- stop"; exit 1; }
  DC_ZONE="${CLOUD_NAME}.${DC_DNS_LABEL}.${DC_REGION}.${CLOUD_DOMAIN}"
  echo "controller cert zone: $DC_ZONE"      # CONFIRM before the sign
  # ONE printf PER LINE, APPENDING. NO HEREDOC (F8 fixed at source, 2026-07-30).
  # The heredoc this replaces had its body and terminator at column 1 inside an indented
  # subshell, and pasting it with the indentation preserved gave a terminator that no longer
  # matched: the heredoc never closed and `controller.cnf` ended up with no [alt_names]
  # section, producing a certificate with NO SANs while every openssl command still printed
  # OK. The operator hit that live on 2026-07-29. A printf's correctness does not depend on
  # invisible leading whitespace on a line nobody reads.
  # Values are %s ARGUMENTS, never inlined, so a value containing a `%` cannot become a
  # format directive. This is the shape `octavia-pki.sh reissue` uses, and it has now been
  # exercised end to end by two real mints (both DCs, 2026-07-30) plus the harness.
  : > controller.cnf
  printf '[req]\n'                                                  >> controller.cnf
  printf 'distinguished_name = req_distinguished_name\n'            >> controller.cnf
  printf 'req_extensions = v3_req\n'                                >> controller.cnf
  printf 'prompt = no\n'                                            >> controller.cnf
  printf '\n'                                                       >> controller.cnf
  printf '[req_distinguished_name]\n'                               >> controller.cnf
  printf 'CN = octavia-controller.%s\n' "$DC_ZONE"                  >> controller.cnf
  printf 'O = Neumatrix\n'                                          >> controller.cnf
  printf '\n'                                                       >> controller.cnf
  printf '[v3_req]\n'                                               >> controller.cnf
  printf 'keyUsage = critical, digitalSignature, keyEncipherment\n' >> controller.cnf
  printf 'extendedKeyUsage = clientAuth, serverAuth\n'              >> controller.cnf
  printf 'subjectAltName = @alt_names\n'                            >> controller.cnf
  printf '\n'                                                       >> controller.cnf
  printf '[alt_names]\n'                                            >> controller.cnf
  printf 'DNS.1 = octavia-controller.%s\n' "$DC_ZONE"               >> controller.cnf
  printf 'DNS.2 = octavia.%s\n' "$DC_ZONE"                          >> controller.cnf
  # ASSERT THE CONFIG BEFORE SIGNING. printf removes the paste hazard; it does not remove the
  # failure CLASS (a typo'd section name still yields a SAN-less cert, and every openssl
  # command would still print OK). So the structure is checked, not trusted.
  for s in '[req]' '[req_distinguished_name]' '[v3_req]' '[alt_names]'; do
    grep -Fqx "$s" controller.cnf || { echo "FAIL: controller.cnf is missing section $s -- stop"; exit 1; }
  done
  grep -Fqx 'subjectAltName = @alt_names' controller.cnf || { echo "FAIL: subjectAltName not wired to [alt_names] -- stop"; exit 1; }
  grep -Fqx "CN = octavia-controller.$DC_ZONE" controller.cnf || { echo "FAIL: CN is not the derived zone -- stop"; exit 1; }
  [ "$(grep -c '^DNS\.[0-9]* = ' controller.cnf)" -eq 2 ] || { echo "FAIL: expected exactly 2 DNS entries -- stop"; exit 1; }
  # HISTORY OF THE LINES ABOVE (superseded 2026-07-30 -- kept because the reasoning is the
  # lesson). This note used to read "logged 2026-07-29, NOT changed -- outside R7's ruled
  # scope: the CN and the two DNS SANs above still carry the `dc0.vr0` identity ... Revisit
  # with D-106's hostname work, not here." That was a correct scope call at the time and it
  # is now DISCHARGED: the names are derived from $DC above, and the two live certificates
  # were reissued into their own zones on 2026-07-30 (Step 1.0-REISSUE; capture
  # docs/audit/octavia-reissue-executed-20260730.txt).
  # WORTH KEEPING: R7 recorded that this cert's SAN is "already DERIVED per-DC by design
  # (DOCFIX-067)", which was true of the IP SAN and NOT of the DNS names -- a claim that was
  # accurate about one half of a field and was read as covering both. That is how F9 survived.
  # STILL TRUE: the DNS names remain INERT while `os-public-hostname` is unset on every API
  # charm (bundle.yaml:11, IP-ONLY endpoints per B5), so nothing resolves them today. They
  # are simply correct in ADVANCE of D-106's Stage-7 work arming them, rather than wrong and
  # waiting to be noticed.
  #
  # (The "no IPv6 IP SAN" gap this block used to log was RULED and CLOSED 2026-07-29 --
  # operator: "Yes, add the v6 IP sans" -- see the D-109 ruling note. IP.2 is emitted
  # below. Still true and still worth knowing: amphorae reach the controller on o-hm0's
  # charm-generated fc00::/64 ULA, not on any VIP leg, so neither SAN matches THAT path;
  # settle by inspection at the Octavia step rather than assuming.)
  #
  # DOCFIX-067: the SAN IP is DERIVED from the bundle (provider leg of octavia's
  # vip triple), never a literal -- the old baked 10.12.4.233 went stale when R14
  # relocated the VIPs to .50-.60. Measured value is echoed; confirm it before
  # the sign. BUNDLE may be overridden if the hardened bundle lives elsewhere.
  REPO="${REPO:?REPO not set -- export REPO=<path to your clone> (see runbooks/README.md Conventions)}"
  BUNDLE="${BUNDLE:-$REPO/bundle.yaml}"
  [ -r "$BUNDLE" ] || { echo "FAIL: bundle not readable at $BUNDLE (set BUNDLE=)"; exit 1; }
  # RULING 3 (2026-07-25): octavia's vip lives in the per-DC OVERLAY now, not in
  # $BUNDLE -- reading the base alone yields a KeyError that `|| true` would swallow
  # into an empty VIP, so this reads the MERGED deploy input explicitly.
  #
  # R7 (2026-07-27) carried an explicit caveat: "the generator's VIP gate must read the
  # MERGED deploy input, not bundle.yaml, or it breaks again the moment ruling-3's VIP
  # extraction and R11's .61/.62 land." Both have since landed, so the overlay IS the
  # merged input for this key, and it is selected by $DC rather than hardcoded to dc0.
  #
  # The gate was `^10\.12\.4\.` -- dc0's provider prefix -- so it hard-ABORTED on dc1's
  # 10.12.64.x. It now derives the expected prefix from the SAME overlay it read the VIP
  # from, so the check still bites (a VIP outside its own DC's provider plane fails) but
  # is no longer frozen to one DC. `.split()[0]` still takes the PROVIDER v4 leg, which
  # is the SAN this cert wants; the v6 legs are deliberately not added here (see the
  # note below).
  [ -r "$VIP_OVERLAY" ] || { echo "FAIL: VIP overlay not readable at $VIP_OVERLAY (set DC=)"; exit 1; }
  OCTAVIA_VIP=$(python3 -c "import yaml,sys;print(yaml.safe_load(open(sys.argv[1]))['applications']['octavia']['options']['vip'].split()[0])" "$VIP_OVERLAY" 2>/dev/null || true)
  KEYSTONE_VIP=$(python3 -c "import yaml,sys;print(yaml.safe_load(open(sys.argv[1]))['applications']['keystone']['options']['vip'].split()[0])" "$VIP_OVERLAY" 2>/dev/null || true)
  VIP_PREFIX="${KEYSTONE_VIP%.*}"      # this DC's provider-public /24-ish prefix, measured
  echo "measured octavia provider VIP: ${OCTAVIA_VIP:-<none>} (DC=$DC, expected prefix ${VIP_PREFIX:-<none>}.)"
  [ -n "$VIP_PREFIX" ] || { echo "FAIL: could not derive this DC's provider prefix from $VIP_OVERLAY -- stop"; exit 1; }
  echo "$OCTAVIA_VIP" | grep -qE "^$(printf '%s' "$VIP_PREFIX" | sed 's/\./\\./g')\.[0-9]{1,3}$" || { echo "FAIL: implausible VIP -- stop"; exit 1; }
  # D-109 RULING NOTE (2026-07-29, operator: "Yes, add the v6 IP sans"): the cert gains
  # this DC's provider v6 leg alongside its v4 one. Both are DERIVED from the same
  # per-DC overlay; the admin and internal legs stay excluded exactly as before, because
  # DOCFIX-067's design has always been provider-leg-only. A v4-only (3-address) vip
  # yields IP.1 alone and no v6 SAN, so this stays correct on a pre-dual-stack tree.
  OCTAVIA_VIP6=$(python3 -c "import yaml,sys,ipaddress;v=yaml.safe_load(open(sys.argv[1]))['applications']['octavia']['options']['vip'].split();print(next((a for a in v if ipaddress.ip_address(a).version==6),''))" "$VIP_OVERLAY" 2>/dev/null || true)
  printf 'IP.1 = %s\n' "$OCTAVIA_VIP" >> controller.cnf
  if [ -n "${OCTAVIA_VIP6:-}" ]; then
    echo "measured octavia provider v6 VIP: $OCTAVIA_VIP6"
    printf 'IP.2 = %s\n' "$OCTAVIA_VIP6" >> controller.cnf
  else
    echo "NOTE: octavia vip is v4-only (3 addresses) -- no IPv6 IP SAN emitted (expected before the dual-stack ADD)"
  fi
  openssl req -new -sha256 -key controller.key -config controller.cnf -out controller.csr
  openssl x509 -req -sha256 -in controller.csr \
    -CA ../controller-ca/controller-ca.cert.pem \
    -CAkey ../controller-ca/controller-ca.key.enc \
    -passin file:../controller-ca/passphrase.txt \
    -CAcreateserial -days 730 \
    -extfile controller.cnf -extensions v3_req \
    -out controller.cert.pem
  openssl verify -CAfile ../controller-ca/controller-ca.cert.pem controller.cert.pem  # expect: OK
  openssl x509 -in controller.cert.pem -noout -ext subjectAltName     # DNS x2 + IP present
  openssl x509 -in controller.cert.pem -noout -dates
  cat controller.cert.pem controller.key > controller.bundle.pem
  chmod 600 controller.bundle.pem
  # E2 FIX AT SOURCE (2026-07-29). The three certificates landed 664 on both DCs from the
  # inherited umask, and `creds-matrix.py` E2 flagged them. A CA certificate that is
  # group-WRITABLE is a SUBSTITUTABLE TRUST ANCHOR -- the exposure is integrity, not
  # confidentiality, which is why "they're public certs" is not a defence. Nothing reads
  # these files at runtime (Octavia receives the material via the overlay), so 600 costs
  # nothing. Done here so the NEXT DC is correct by construction rather than by a
  # remembered follow-up chmod.
  # The SERIAL is included: it carries no secret, but a group member who rewrites it forces the
  # next issuance to reuse a serial. Omitted on the first pass and caught by creds-matrix E2.
  chmod 600 controller.cert.pem ../issuing-ca/issuing-ca.cert.pem \
            ../controller-ca/controller-ca.cert.pem ../controller-ca/controller-ca.cert.srl
  # E3 FIX AT SOURCE (2026-07-29). These two are BUILD INTERMEDIATES: the signing request
  # is spent the moment the certificate is issued, and the openssl config is fully derived
  # from $DC plus the VIP overlay, so the repo already determines it. Both were left behind
  # on both DCs with no register row, which is what `creds-matrix.py` E3 reported. Removing
  # them is preferred over declaring them: a register that accumulates rows for artifacts
  # that should not exist trains the reader to add a row rather than ask whether the file
  # belongs. RETAINED DELIBERATELY: `controller.cert.pem` (the record of what was issued,
  # and what `octavia-pki.sh verify` reads) and `../controller-ca/controller-ca.cert.srl`
  # (CA issuance STATE -- deleting it lets the same CA reissue a duplicate serial).
  rm -f controller.csr controller.cnf
} )
```

### 1.0-GEN.d -- Write overlays/${DC}-octavia-pki.yaml (base64 blobs + plaintext passphrase)
Four values are base64(PEM); the issuing-CA passphrase is a PLAIN string. The file is
gitignored. `$REPO` and `$OCTAVIA_PKI_OVERLAY` are exported in 1.0-GEN.0.

> **THE GITIGNORE CHECK BELOW IS LOAD-BEARING, NOT HYGIENE (F4, 2026-07-29).** This file
> carries CA private-key blobs PLUS the issuing-CA passphrase in PLAINTEXT, and SEC-004
> records this repo as currently PUBLIC -- the gitignore is the only thing between a CA
> private key and publication. `.gitignore` used to pin the single exact path
> `overlays/octavia-pki.yaml`, so the moment F1 made the name per-DC the new name stopped
> matching and the file would have been committable. The pattern is now a glob, and this
> block ASSERTS that fact at generation time rather than trusting anyone to remember it.

**RUN -- voffice1 (the D-128 Plane-2 headend; RULED 2026-07-29)**
```bash
( {
  WORKDIR="$HOME/octavia-pki/${DC:?DC not set -- run 1.0-GEN.0 first}"
  cd "$WORKDIR" || exit 1   # dir from 1.0-GEN.0
  REPO="${REPO:?REPO not set -- export REPO=<path to your clone> (see runbooks/README.md Conventions)}"
  OUT="${OCTAVIA_PKI_OVERLAY:?OCTAVIA_PKI_OVERLAY not set -- run 1.0-GEN.0 first}"
  mkdir -p "$REPO/overlays"
  # F4 GATE: refuse to write key material into a path git would track. Asserts on the
  # ACTUAL ignore decision for THIS path, not on the presence of a pattern.
  git -C "$REPO" check-ignore -q "$OUT" || {
    echo "ABORT: $OUT is NOT gitignored -- writing CA key material here would make it"
    echo "       committable in a repo SEC-004 records as PUBLIC. Fix .gitignore first."
    exit 1; }
  # AS EXECUTED 2026-07-30 -- the operator's correction, adopted VERBATIM because it is what
  # actually produced the live overlays for both DCs. `base64 -w0` is a GNU-only flag;
  # `base64 < f | tr -d '\r\n'` yields the identical single line and is portable. Keeping the
  # -w0 form would have left this runbook describing a command nobody ran.
  ISS_CERT=$(base64 < issuing-ca/issuing-ca.cert.pem | tr -d '\r\n')
  ISS_KEY=$(base64 < issuing-ca/issuing-ca.key.enc | tr -d '\r\n')
  ISS_PASS=$(cat issuing-ca/passphrase.txt)
  CON_CACERT=$(base64 < controller-ca/controller-ca.cert.pem | tr -d '\r\n')
  CON_CERT=$(base64 < controller/controller.bundle.pem | tr -d '\r\n')
  cat > "$OUT" <<OVL
applications:
  octavia:
    options:
      lb-mgmt-issuing-cacert: "$ISS_CERT"
      lb-mgmt-issuing-ca-private-key: "$ISS_KEY"
      lb-mgmt-issuing-ca-key-passphrase: "$ISS_PASS"
      lb-mgmt-controller-cacert: "$CON_CACERT"
      lb-mgmt-controller-cert: "$CON_CERT"
OVL
  chmod 600 "$OUT"
  echo "wrote $OUT"
} )
```
Then run the Step 1.0 sanity block (5 keys present; ASCII clean) before deploying. Keep
`$HOME/octavia-pki/` (the CA keys + passphrases) OFF the repo and backed up securely; the
10y CAs are reused across rebuilds -- regenerate only on key compromise or CA expiry.

### 1.0-GEN.e -- BACK UP the per-DC workspace to the jumphost creds folder  DISCRETE / SECRET

> **THIS STEP IS NOT OPTIONAL, AND THE REASON IS MEASURED.** Before it existed, three
> surfaces said this material "MUST be in the per-DC backup set" / "backed up securely" --
> and **no backup set existed**: `scripts/cloud-snapshot.sh` is a juju-layer capture and
> mentions octavia, tfstate and terraform ZERO times. So both DCs' 10-year amphora trust
> roots lived in exactly one place, on one VM, with no copy. Losing the ISSUING CA key means
> Octavia can never sign another amphora certificate -- no new load balancers, no amphora
> replacement. Losing the CONTROLLER CA key means the controller certificate can never be
> reissued, which F9 establishes it must be at Stage 7.
>
> **WHAT THIS COSTS, stated plainly:** it creates a SECOND at-rest copy of both encrypted CA
> keys BESIDE their plaintext passphrases. Encryption-at-rest is therefore defeated by the
> archive's own contents -- treat the archive exactly as you would the live keys. This is the
> same trade D-109 option (b) was REFUSED for, and the distinction is deliberate: that ruling
> governed where the AUTHORITATIVE artifact lives and which host the deploy reads. A backup is
> a recovery copy, not a second source of truth.
>
> **INTERIM, PENDING THE PINNED SECRETS-STORAGE SOLUTION.** The jumphost creds folder is the
> SEC-009 convention location and is where this goes for now. When the secrets-storage solution
> is designed, it MUST carry a process and procedure for CERTIFICATE and CREDENTIAL backup, and
> THIS STEP IS ONE OF THE STEPS THAT MUST BE FOLDED INTO IT -- not left as a per-runbook
> one-off. Recorded as a standing forward item so the requirement is not rediscovered.

**RUN -- voffice1, then vcloud (the archive moves headend -> jumphost)**
```bash
# RUN: voffice1 -- build the archive beside the workspace, never inside the repo
( {
  DC="${DC:?run 1.0-GEN.0 first}"
  cd "$HOME/octavia-pki" || exit 1
  umask 077
  tar czf "$HOME/octavia-pki-${DC}.tar.gz" "$DC"
  chmod 600 "$HOME/octavia-pki-${DC}.tar.gz"
  sha256sum "$HOME/octavia-pki-${DC}.tar.gz"      # RECORD this; it is the integrity check
  tar tzf "$HOME/octavia-pki-${DC}.tar.gz" | wc -l   # expect the workspace file count
} )
```

```bash
# RUN: vcloud -- pull it into the per-DC creds folder, verify, then remove the staging copy
( {
  DC="${DC:?export DC=vr1-dc0|vr1-dc1}"
  install -d -m 700 "$HOME/${DC}-creds"
  scp -q "voffice1:~/octavia-pki-${DC}.tar.gz" "$HOME/${DC}-creds/octavia-pki-${DC}.tar.gz"
  chmod 600 "$HOME/${DC}-creds/octavia-pki-${DC}.tar.gz"
  # INTEGRITY: compare against the source, do not assume scp succeeded
  A=$(sha256sum < "$HOME/${DC}-creds/octavia-pki-${DC}.tar.gz" | cut -d' ' -f1)
  B=$(ssh voffice1 "sha256sum < ~/octavia-pki-${DC}.tar.gz" | cut -d' ' -f1)
  [ "$A" = "$B" ] && echo "OK sha256 matches source" || { echo "ABORT: sha256 MISMATCH -- do not delete the source"; exit 1; }
  ssh voffice1 "rm -f ~/octavia-pki-${DC}.tar.gz"   # only after the match
  echo "backed up: ~/${DC}-creds/octavia-pki-${DC}.tar.gz"
} )
```

Run BOTH blocks once per DC. The staging copy is removed from the headend only after the
sha256 comparison passes -- an `scp` that silently truncated would otherwise leave you with a
verified-looking backup of nothing.

**Per-DC layout produced (F1).** Each DC is a wholly separate trust domain on disk, so a
second DC's generation cannot reach the first DC's material:

    $HOME/octavia-pki/vr1-dc0/{issuing-ca,controller-ca,controller}/
    $HOME/octavia-pki/vr1-dc1/{issuing-ca,controller-ca,controller}/
    $REPO/overlays/vr1-dc0-octavia-pki.yaml
    $REPO/overlays/vr1-dc1-octavia-pki.yaml

The workspace tree MUST be in the per-DC backup set (step 1.0-GEN.e): losing the controller-CA
key makes that DC's controller-cert rotation impossible. **The OVERLAY does not need separate
backup** -- corrected 2026-07-30, because this line previously said "both ... and the overlay"
and implied a gap that does not exist: 1.0-GEN.d regenerates the overlay ENTIRELY from the
archived workspace files, so backing up the workspace covers it.

---

## Step 1.0-REISSUE -- rotate the CONTROLLER LEAF certificate (tool-driven)  DISCRETE / SECRET

**Use this, NOT 1.0-GEN, whenever the controller certificate is wrong or expiring.** 1.0-GEN's
REFUSE-IF-PRESENT gate (1.0-GEN.0) deliberately blocks re-running generation, and its only
stated escape -- "move the existing material aside first" -- moves the **CAs** aside, which
regenerates the trust domain and invalidates every amphora already issued against it. That is
the opposite of what a leaf rotation needs. This step exists so the sanctioned path is not the
dangerous one.

**WHAT IT DOES.** Mints a FRESH EC P-256 key and a new controller certificate signed by the
**EXISTING** controller CA, then rewrites exactly ONE value in
`overlays/${DC}-octavia-pki.yaml` (`lb-mgmt-controller-cert`). **No CA is regenerated.** The
amphora trust domain is untouched, so anything already issued stays valid.

**WHY IT IS A SCRIPT AND NOT A PASTE BLOCK.** 1.0-GEN.c is a hand-pasted `openssl` chain, and
that shape has failed twice in measurable ways: F8 (an indented heredoc terminator silently
produced a certificate with **NO SANs** while every command printed OK) and F9 (the CN and both
DNS SANs are baked `dc0.vr0` literals that no gate read). The reissue path derives every value,
asserts the artifact it produced, and stages before it promotes. Operator ruling 2026-07-30,
exact utterance: **"Full script minter now"** -- taken with the measured fact that
`guard-destructive.py` is a Bash-command-string matcher that has no opinion on any script
wrapping `openssl`, and that D-137 sub-ruling 1 (2026-07-25, "Blocking in preflight") had
already declined to constrain agents at write time. This step does not change that posture.

> ## CAUTION -- this consumes a CA serial and rotates a private key
>
> A reissue burns one serial from `controller-ca.cert.srl` and replaces `controller.key`. Both
> are irreversible forward. A pre-reissue archive is taken automatically **before** the first
> byte is minted and its path + sha256 are printed; record them. **Never restore that archive
> over a workspace that has issued since** -- it carries the CA's issuance state, so restoring
> it rolls the serial backwards and the next mint reuses a serial the estate already holds.

### 1.0-REISSUE.0 -- decide, read-only, and look before you mint

**CHECK (read-only) -- voffice1 (the D-128 Plane-2 headend; the PKI lives here, RULED D-109 (b))**
```bash
# RUN: voffice1
cd "$REPO" || exit 1
bash scripts/octavia-pki.sh verify vr1-dc0
bash scripts/octavia-pki.sh verify vr1-dc1
```
**Expect:** a verdict line per DC. Read `A12`/`A13` -- they name this DC's DERIVED expected zone
(`omega.dc0.vr1.cloud.neumatrix.local` / `omega.dc1.vr1.cloud.neumatrix.local`), and say whether
the certificate is in it. `A15` covers keyUsage/EKU, `A16` expiry, `A17` whether the deploy
overlay actually carries the material this workspace holds.

**GATE:** you know, per DC, WHY you are reissuing. "verify said something" is not a reason.

### 1.0-REISSUE.1 -- dry run (mints nothing)

**CHECK (read-only) -- voffice1**
```bash
# RUN: voffice1
bash scripts/octavia-pki.sh reissue vr1-dc0 --dry-run
```
**Expect:** every precondition reported, the derived zone echoed, and the full openssl config
printed. It writes no key, no cert, no backup and no staging directory. Read the config: the CN,
`DNS.1`, `DNS.2` and the IP SANs are what the certificate will carry.

**GATE:** the printed zone is THIS DC's. `vr1-dc0` must print `dc0.vr1`, `vr1-dc1` must print
`dc1.vr1`. If those are transposed, STOP -- do not proceed, and do not "fix" it by hand.

### 1.0-REISSUE.2 -- the mint, ONE DC at a time  > CAUTION: mutation

**RUN -- voffice1**
```bash
# RUN: voffice1
bash scripts/octavia-pki.sh reissue vr1-dc0
```
**Expect:** exit 0, the archive path + sha256, `serial <old> -> <new>`, and the new SANs.

**Exit codes -- act on them, do not re-run blindly:**

| rc | meaning | what to do |
|----|---------|-----------|
| 0 | reissued and every post-mint assertion passed | proceed to 1.0-REISSUE.3 |
| 1 | FAIL -- a measured defect blocked it, or a staged assertion failed | **nothing was promoted.** Fix what it names |
| 2 | bad arguments | fix the command |
| 3 | REFUSE -- could not evaluate or could not make it safe | **nothing was minted.** Wrong host, unreadable CA, unparseable VIPs |
| 4 | REFUSE -- already correct | nothing to do. `--force` only if you intend a rotation anyway |
| 5 | **PARTIAL -- ROLLBACK REQUIRED** | the workspace moved and the overlay did not. Run the printed restore command, then **record the burned serial** |

**GATE:** exit 0. Do NOT run the second DC until the first is verified in 1.0-REISSUE.3 --
one DC at a time is what keeps a wrong `$DC` from being a two-DC incident.

Then repeat for the other DC:
```bash
# RUN: voffice1
bash scripts/octavia-pki.sh reissue vr1-dc1
```

### 1.0-REISSUE.3 -- verify, per DC

**CHECK (read-only) -- voffice1**
```bash
# RUN: voffice1
bash scripts/octavia-pki.sh verify vr1-dc0
bash scripts/octavia-pki.sh verify vr1-dc1
python3 scripts/creds-matrix.py --tier2
```
**GATE:** each DC PASSes with **0 failed**, AND the output carries the literal line
`A12 DNS SANs are all in this DC's expected zone '<this DC's zone>'`.

> **THE COUNT IS NOT THE GATE, THE LINE IS.** A `PASS` verdict alone is compatible with the
> wrong certificate: in today's unarmed posture (`os-public-hostname` set nowhere, B5 IP-only)
> A12 and A13 RECORD rather than fail, so a wrong-zone cert also reports PASS -- it just
> carries a "(recorded, not failed)" line instead of the positive one. Grep for the positive.
>
> **Both gates, not either.** `creds-matrix` (P5) and `octavia-pki.sh verify` check different
> things and have disagreed before in BOTH directions -- a cert at 644 passes A3 and fails P5;
> one at 660 passes P5 and fails A3. That disagreement is the point of running both.

**GATE:** `creds-matrix --tier2` shows no NEW finding attributable to this change.

### 1.0-REISSUE.4 -- backup hygiene

The reissue wrote `~/octavia-pki/backups/<site>-octavia-pki-<stamp>-<pid>.tar.gz` on the
headend. It contains **both CA private keys, both passphrases, and the overlay** -- treat it as
live key material. Pull it to the jumphost creds folder and remove the headend copy, exactly as
1.0-GEN.e does for the generation archive; compare sha256 before deleting.

**GATE:** the headend carries no `octavia-pki/backups/` residue once the jumphost copy is
sha256-verified. If the run also NOTED leftover `.reissue-<site>.*` staging directories, inspect
and remove them deliberately -- each may hold a CA-signed certificate and an unencrypted key
that no register declares.
