name: openstack-cloud-ops

description: "Operate, install, extend, and troubleshoot the Omega Cloud - a commercial multi-tenant Charmed OpenStack (Caracal 2024.1) deployment managed with Juju and MAAS, with Vault TLS, OVN, Ceph, Octavia, and Magnum/CAPI tenant Kubernetes. Use this skill for ANY work touching OpenStack, Juju, MAAS, Magnum, CAPI, Ceph, OVN, Octavia, Keystone, Vault-for-OpenStack, tenant onboarding, or the repository this skill operates against - including writing or reviewing bash/python operational scripts, debugging failed deploys or cluster creates, runbook work, design-decision (D-NNN) discussion, and incident triage. Use it even for seemingly simple OpenStack questions: this deployment has strict operating discipline and known charm traps that make generic answers wrong."

openstack-cloud-ops

Operating skill for the Omega Cloud: a commercial, multi-tenant, tenant self-administered OpenStack cloud. Current phase: single-DC virtual rehearsal ("testcloud", VR0 DC0) on four KVM hosts, rehearsing a future bare-metal multi-datacenter deployment ("Roosevelt"). The governing design constraint is MINIMIZE DELTA TO ROOSEVELT: the runbooks and scripts are primary deliverables alongside the running cloud, so transferable answers beat quick fixes.

Step 0 - locate the source of truth

This repository (GitBucket, git.baldurkeep.com) is authoritative for everything: bundle, runbooks, scripts, design decisions, as-built values. This skill is a discipline-and-routing layer OVER that repo, not a substitute for it. Never hardcode the repo's own name/path in a command -- it has already been renamed/forked once (see D-110) and will again; discover it fresh each session.

Look for a local clone: a repo dir in the working tree, or a $HOME entry containing bundle.yaml + docs/design-decisions.md. If found, confirm identity with git remote -v, then git log -1 to note HEAD and work from it.
No clone and you have shell + network: ask the operator for the exact clone URL (org/group and repo name -- do not guess or reuse one from memory; it may have moved, per this same repo's own history). The repo may be private; if the clone fails, ask the operator to provide access or the relevant files.
No clone obtainable (e.g. chat without sandbox network): say so, ask the operator to paste the relevant runbook/script, and proceed only on what is actually in front of you.

Divergence rule: if this skill and repo HEAD disagree, the repo wins - but FLAG the divergence to the operator rather than silently following either. The repo is a living draft; this skill's invariants (discipline, hardening) change slowly, its facts (IPs, versions, phase status) go stale fast.

Long sessions get compacted. At session start (and after any compaction), read docs/session-ledger.md and run bash scripts/ledger-scan.sh to recover what is in flight and reconcile it against repo ground truth (open decisions, open security rows, next-free numbers). Update the ledger at every deliverable -- it is standing, like the changelog. See references/operating-discipline.md (Session continuity).

Step 1 - detect the environment

Live shell to the jumphost / infra (Claude Code on vopenstack-jesse or similar): you may RUN read-only audits directly. Every mutation remains individually human-gated - present the command, state what it changes, wait for approval. A live shell relaxes the transport, never the discipline.
Chat / no infra shell: operate the gated copy-paste model - prepare labeled blocks, the operator runs them and pastes output back. Never assume a block ran or succeeded; wait for the pasted evidence.

Read references/operating-discipline.md before doing either.

The three hard operating rules (non-negotiable)

Execute only the current runbook step, exactly as written. No added scope, no adjacent improvements, no live re-architecture mid-step. Findings and improvement ideas are LOGGED (changelog / D-NNN proposal), never executed live mid-step.
Never use an inferred value. No IP, ID, name, or scope goes into a command unless it was measured this session or carried from confirmed as-built. If a value would be inferred: stop and measure it. Never run a destructive or session-altering command from memory without confirming it is the minimal correct action for the current live state.
Prefer dynamic lookups over hardcoded literals. Discover VIPs, project names, IDs, and version sets at runtime. Where a literal is unavoidable it is tagged and centralized (scripts/lib-net.sh, lib-hosts.sh), keyed by stable identity (CIDR, hostname) - never by drifting IDs.

Corollary that governs everything: verify before mutate. A read-only audit precedes every mutation; destructive and secret-handling steps are gated individually, never batched.

Routing - where to go for what

Task	Read first
Any command block, script, or paste block you are about to write	`references/script-authoring.md`
Deploy / redeploy / teardown (VR0, single-DC testcloud)	repo `runbooks/README.md`, then the phase-NN runbook; conventions in `references/operating-discipline.md`
VR1 DC-DC buildout (two-DC + Office1 headend -- the current live mission)	repo `docs/dc-dc-deployment-workflow.md` FIRST (stage status + tooling gap register), then `runbooks/README.md`'s "VR1 DC-DC track" section for the `dc-dc-phase0..6` runbooks + `dc-dc-teardown-rollback.md`. SEPARATE track from VR0's phase-NN above -- do not conflate stage numbers between the two.
Something is broken (triage, incidents)	`references/troubleshooting.md`, then repo `runbooks/appendix-A-troubleshooting.md`
CAPI / Magnum / mgmt-VM recovery	repo `runbooks/ops-capi-recovery.md`
Deliver ANY repo change (script, runbook, doc)	run `bash scripts/repo-lint.sh` + the touched script's `tests/<name>/run-tests.sh` BEFORE handing it over
Pre-deploy gate (before add-model)	`bash scripts/preflight.sh` -- THE single entry; do not run the sub-gates piecemeal
Is the cloud actually healthy? (post-deploy, post-restart, pre-change baseline, incident)	`bash scripts/cloud-assert.sh` (add `--capture` at deploy completion for the committed BOM)
Full-cloud restart after outage/maintenance	repo `runbooks/ops-restart-procedure.md`
Starting any consequential live session	`bash scripts/run-logged.sh <label>` first (as-executed log; docs/as-executed-log-convention.md)
Credential exposures / security TODOs	repo `docs/security-ledger.md` -- add a row, never only a script comment
Resuming work / what is still in flight (esp. after compaction)	repo `docs/session-ledger.md`, then `bash scripts/ledger-scan.sh` to reconcile against repo ground truth
Tenant onboarding / tenant self-service	repo `scripts/tenant-onboard.sh` + `runbooks/tenant-onboarding-v2-DRAFT.md` + `appendix-C/D`
Network / plane / IPAM questions	`references/environment.md`, repo `scripts/lib-net.sh`, NetBox (the IPAM apex)
ANY change request to a built surface	grep repo `docs/design-decisions.md` for the governing D-NNN FIRST - PROPOSED/OPEN means the operator has not ruled: present options, do not implement
Why is it built this way? / proposing changes	repo `docs/design-decisions.md` (D-NNN); grep before assigning a new number
Versions, channels, pins	repo `runbooks/appendix-B-asbuilt-version-lock.md`
Environment facts (hosts, repo, planes)	`references/environment.md`
OpenTofu / libvirt / MAAS / OPNsense IaC work (repo `opentofu/`)	`references/opentofu-provider-docs.md` FIRST (sources + fetch method), then repo `opentofu/README.md` (per-module ground truth, schema notes, audit findings)

Standard loops (repeatable session shapes)

Session bootstrap (jumphost): git -C "$REPO" pull (set $REPO once per session -- see runbooks/README.md Conventions / references/environment.md) -> bash scripts/repo-lint.sh (0 fail expected) -> if touching the live cloud, bash scripts/run-logged.sh <label> to open the logged shell. Repo HEAD and a clean lint are the preconditions for everything else.

Change-delivery loop: grep for prior art (zeroth decision) -> grep design-decisions for the governing D-NNN -> edit -> bash scripts/repo-lint.sh -> run the touched script's harness (create one if missing -- no script change ships without its harness) -> deliver as repo-relative ZIP + a changelog entry with a per-item revert. Under blanket approval, the changelog IS the review surface: every item states what, why (evidence), and how to revert.

Deploy loop (VR0, single-DC testcloud): phase-00 runbook (D-061 destroy path) -> bash scripts/preflight.sh PASS -> phase-01..08 gated -> bash scripts/cloud-assert.sh --capture -> commit the asbuilt/ BOM.

Deploy loop (VR1 DC-DC, the current live mission): read docs/dc-dc-deployment-workflow.md's tooling gap register FIRST -- check whether the stage you're about to run has any open blocking item -> Stage 1 (dc-dc-phase0-vcloud-prep.md) gated -> Stage 2..7 gated in order, per runbooks/README.md's "VR1 DC-DC track" table -> if a stage's tofu apply fails partway, read runbooks/dc-dc-teardown-rollback.md's rollback decision tree BEFORE reaching for a teardown (fix-forward is usually correct). Every opentofu/ command is UNVALIDATED (no tofu binary has run in any session that authored this tree) -- run bash scripts/opentofu-validate.sh first if available, and treat every OpenTofu step as reviewed-but-unexercised.

Incident loop: capture the verbatim error -> bash scripts/cloud-assert.sh (the service-own-verdict sweep localizes the layer) -> appendix-A by exact message -> recorded fix, gated -> log the finding (new root causes become appendix-A/DOCFIX material).

Posture

This is a commercial multi-tenant cloud with HARD tenant isolation (SCS Domain Manager persona). Treat tenant-visible surfaces and cross-domain boundaries as security-relevant in every change.
The operator community here values debate and industry best practice over quick fixes. Push back with sources when you disagree; own mistakes plainly and concisely. Fabricated flags, values, or version numbers are the cardinal sin - if you have not verified an option name or version, say so and verify.
Responses stay concise. Decisions get explicit rationale.