# Ops -- Full-Cloud Restart / Recovery Procedure

STATUS: committed to the repo per DOCFIX-075 (previously lived only as an
operator-local document + a jumphost-local health script -- unreachable by a
Roosevelt team and by any fresh clone). Procedure lineage: authored and
validated 2026-05-20 on the prior build; adapted 2026-07-03 to current repo
references (appendix-A, cloud-assert, D-043). Stages whose mechanics depend on
build state carry [REVALIDATE] markers -- clear them on the first executed
restart of the current build and record the as-executed date here.

Scope: PLANNED maintenance windows taking all four hosts down, or UNPLANNED
recovery from full power/network loss that took MySQL or Vault offline
together. For single-host partial failures, use the targeted entries in
`runbooks/appendix-A-troubleshooting.md` instead.

Conventions: RUN/CHECK/GATE labels per runbooks/README.md. One gated step at a
time. Invoke scripts as `bash scripts/<name>.sh` (no exec bits in the repo).

---

## Pre-shutdown (planned only)

**RUN -- jumphost** -- baseline capture (reference, not a restore path)
```bash
( {
  juju switch | grep -q 'openstack' || { echo "FAIL: switch to the openstack model first"; exit 1; }
  BAK=~/openstack-baseline/$(date -u +%Y%m%d-%H%M%S)-pre-maintenance
  mkdir -p "$BAK"
  juju export-bundle --filename "$BAK/bundle.yaml"
  juju status --format=yaml > "$BAK/juju-status.yaml"
  for v in $(env | awk -F= '/^OS_/{print $1}'); do unset "$v"; done
  # shellcheck source=/dev/null
  source ~/admin-openrc
  openstack server list --all-projects     </dev/null > "$BAK/servers.txt"
  openstack loadbalancer list              </dev/null > "$BAK/loadbalancers.txt"
  openstack loadbalancer amphora list      </dev/null > "$BAK/amphorae.txt"
  echo "Pre-maintenance snapshot at $BAK"
} )
```

**Guest auto-resume policy (D-043, ADOPTED):** `nova-compute
resume-guests-state-on-host-boot=true` is charm config in the bundle. Running
guests resume automatically on host boot, honoring per-guest pre-shutdown
state -- so a guest you want to stay down for forensics must be
`openstack server stop`ped BEFORE the window. NOTE the D-043 caveat: the
in-cloud CAPI mgmt VM (capi-mgmt-v2) is a control-plane component riding a
tenant VM; it WILL auto-resume with the rest. Its manual-start policy (D-041)
now governs deliberate stops only.

**VR1 host-VM autostart policy (D-127):** the vcloud host-level libvirt domains follow a separate rule from
the cloud guests above. FOUNDATIONAL service VMs autostart (`voffice1`; `office1-opnsense` after one verified
post-fix cold boot -- DOCFIX-188); DC CONTAINMENT VMs (`vvr1-dc0`, future `vr1-dc1`) stay MANUAL (gated
depth-4/site-down bring-up, never on host boot). Set live with `virsh autostart <domain>`; see D-127.

## Shutdown sequence

1. (Optional) stop guests you want held down post-maintenance.
2. Power down hosts via MAAS/IPMI/virsh. Order not significant among
   openstack0-3 (functional peers).

## Power-on / recovery

### Stage 0 -- DC EGRESS, before anything that fetches [VR1 DC-DC only]

> **RUN THIS FIRST AND DO NOT SKIP IT.** Added 2026-08-02 after a **19-hour dc0
> egress outage that every existing check missed.** A containment-VM reboot left
> the DC's OPNsense edge broken (a partially-applied update), so the DC had no
> path off-site -- and `dc-rack-net.sh check dc0` PASSED throughout (it asserts
> bridge legs and unit states; **a leg is not a path**), the mirror answered
> HTTP 200 from its own nginx (**a 200 from a LOCAL service says nothing about
> upstream**), and the session that verified that reboot cited five true facts,
> none of which tested egress. It surfaced only when a nightly debmirror timer
> failed, three layers away from the cause.

```bash
# ON THE RACK -- every value the check uses describes the rack, so running it
# anywhere else measures a different machine's path.
ssh <site rack> 'bash -s -- check <dc0|dc1>' < scripts/dc-egress-check.sh
```

**GATE:** exit 0. The check is LAYERED and reports the FIRST failure as the
cause rather than the last, so read the first FAIL and fix that:

| First FAIL | What it means |
|---|---|
| `A1` | no default route, or it points somewhere other than this site's edge |
| `A2 edge ... does NOT answer` | **the edge itself is down** -- this is the 2026-08-01 outage. Nothing below it can work; do not chase the downstream symptoms |
| `A3` | the edge answers but traffic is not leaving it -- forwarding/NAT on the edge |
| `A4` | egress works; one upstream artifact source is down. Named individually so a single dead source is not read as a total outage |

Exit 2 is REFUSE ("could not look"), never a pass -- most often it means the
check was run somewhere other than the rack.

**Why here and not later:** everything from Stage 1 onward fetches. Discovering
a dead edge after the agents start is how a half-built model happens, and unlike
bootstrap the recovery is not "re-open egress and retry".

### Stage 1 -- foundation
**GATE:** every juju machine agent reports `started` (typically 2-5 min).
```bash
juju machines --format=json | jq -r '.machines | to_entries[]
  | "\(.key)  agent=\(.value."juju-status".current)"' | grep -v 'agent=started' \
  || echo "all machines started"
```

### Stage 2 -- MySQL InnoDB cluster
Check FIRST; act only on a confirmed outage. `reboot-cluster-from-complete-outage`
is DESTRUCTIVE against an already-reformed cluster (it rewrites topology from
the unit it runs on). See appendix-A "mysql-innodb-cluster recovery" for the
signature table (D-062 material).

**CHECK (read-only) -- jumphost**
```bash
juju status mysql-innodb-cluster --format=json | jq -r \
  '.applications."mysql-innodb-cluster".units | to_entries[]
   | "\(.key)  \(.value."workload-status".message // "")"'
```
**GATE:** three units `Cluster is ONLINE`, exactly one `Mode: R/W`. If ALL
units report a complete outage (and only then):
```bash
juju run mysql-innodb-cluster/leader reboot-cluster-from-complete-outage
```

### Stage 3 -- Vault unseal (sealed-after-restart is BY DESIGN)
Vault stores in MySQL and always restarts sealed; manual 3-of-5 unseal is the
v1 standard (D-011 amendment). Keys are NOT in version control -- custody per
D-069 / docs/security-ledger.md. Missing keys = recovery BLOCKER: escalate,
never improvise (re-init wipes the TLS plane).

**RUN -- jumphost** (hidden prompts; keys never in argv/history)
```bash
( {
  for i in 1 2 3; do
    read -s -p "Unseal key $i: " K; echo
    juju ssh vault/0 -- "VAULT_ADDR=http://127.0.0.1:8200 vault operator unseal $K" </dev/null >/dev/null
    unset K
  done
  juju ssh vault/0 -- 'VAULT_ADDR=http://127.0.0.1:8200 vault status 2>&1 | grep -E "Sealed|Initialized"' </dev/null
} )
```
**GATE:** `Sealed=false`.

### Stage 4 -- OVN chassis
**CHECK:** `connection-status` on every chassis (cloud-assert A4 runs this).
If any is not `connected`, run the post-vault TLS restart sweep (appendix-A;
restart ovn-controller / ovs-vswitchd / neutron-server on affected units),
then re-probe. [REVALIDATE: sweep unit list against current topology]

### Stage 5 -- guests
With D-043 in force, pre-shutdown-running guests should be ACTIVE.
```bash
openstack server list --all-projects -c Name -c Status </dev/null
```
Unexpectedly SHUTOFF guests: `openstack server start <name>`; if MANY are
down, verify `juju config nova-compute resume-guests-state-on-host-boot`
(a regressed flag takes effect only on the NEXT boot -- start manually now).

### Stage 6 -- Octavia LBs
Expect auto-recovery once amphora VMs resume. Any LB stuck
`provisioning_status=ERROR` after ~5 min: `openstack loadbalancer failover
<lb>` and poll to ACTIVE (VIP + FIP are preserved). [REVALIDATE: timing
budget on current build]

### Stage 7 -- end-to-end behavioral gate
```bash
source ~/admin-openrc
bash scripts/cloud-assert.sh
```
**GATE:** `CLOUD-ASSERT: PASS`. Any FAIL routes to appendix-A by its message.
This replaces the old jumphost-local `post-maintenance-health-check.sh`.

---

## Quick reference (symptom -> fix)

| Symptom | Fix |
|---|---|
| Units in `error` (stuck relation-broken hook) | `juju resolved --no-retry <unit>` |
| MySQL will not reform | Stage 2 -- action ONLY on confirmed complete outage |
| Vault sealed | Expected; Stage 3 manual unseal |
| Chassis `not connected` | Stage 4 TLS sweep |
| Hypervisor `down`, juju green | restart nova-compute on the host; recheck in 60s |
| LB `provisioning_status=ERROR` | `loadbalancer failover`; appendix-A LB entries |
| Horizon login cookie failure | restore `_99_internal_http_cookies.py` (D-044) |

## Open question (carried for Roosevelt)
Amphora cert validity across long outages: a resumed amphora keeps pre-shutdown
cert material; if the amphora-control-plane CA rotated during the window it
will fail to health-manager and be failed-over. Theoretical under default TTLs
for short outages; validate explicitly on bare metal (rotate CA, hard-stop an
amphora host, observe resume vs ERROR->failover) before relying on resume.