name: openstack-cloud-ops
Operating skill for the Omega Cloud: a commercial, multi-tenant, tenant self-administered OpenStack cloud. Current phase: single-DC virtual rehearsal ("testcloud", VR0 DC0) on four KVM hosts, rehearsing a future bare-metal multi-datacenter deployment ("Roosevelt"). The governing design constraint is MINIMIZE DELTA TO ROOSEVELT: the runbooks and scripts are primary deliverables alongside the running cloud, so transferable answers beat quick fixes.
This repository (GitBucket, git.baldurkeep.com) is authoritative for everything: bundle, runbooks, scripts, design decisions, as-built values. This skill is a discipline-and-routing layer OVER that repo, not a substitute for it. Never hardcode the repo's own name/path in a command -- it has already been renamed/forked once (see D-110) and will again; discover it fresh each session.
$HOME entry containing bundle.yaml + docs/design-decisions.md. If found, confirm identity with git remote -v, then git log -1 to note HEAD and work from it.Divergence rule: if this skill and repo HEAD disagree, the repo wins - but FLAG the divergence to the operator rather than silently following either. The repo is a living draft; this skill's invariants (discipline, hardening) change slowly, its facts (IPs, versions, phase status) go stale fast.
Long sessions get compacted. At session start (and after any compaction), read docs/session-ledger.md and run bash scripts/ledger-scan.sh to recover what is in flight and reconcile it against repo ground truth (open decisions, open security rows, next-free numbers). Update the ledger at every deliverable -- it is standing, like the changelog. See references/operating-discipline.md (Session continuity).
vopenstack-jesse or similar): you may RUN read-only audits directly. Every mutation remains individually human-gated - present the command, state what it changes, wait for approval. A live shell relaxes the transport, never the discipline.Read references/operating-discipline.md before doing either.
scripts/lib-net.sh, lib-hosts.sh), keyed by stable identity (CIDR, hostname) - never by drifting IDs.Corollary that governs everything: verify before mutate. A read-only audit precedes every mutation; destructive and secret-handling steps are gated individually, never batched.
| Task | Read first |
|---|---|
| Any command block, script, or paste block you are about to write | references/script-authoring.md |
| Deploy / redeploy / teardown (VR0, single-DC testcloud) | repo runbooks/README.md, then the phase-NN runbook; conventions in references/operating-discipline.md |
| VR1 DC-DC buildout (two-DC + Office1 headend -- the current live mission) | repo docs/dc-dc-deployment-workflow.md FIRST (stage status + tooling gap register), then runbooks/README.md's "VR1 DC-DC track" section for the dc-dc-phase0..6 runbooks + dc-dc-teardown-rollback.md. SEPARATE track from VR0's phase-NN above -- do not conflate stage numbers between the two. |
| Something is broken (triage, incidents) | references/troubleshooting.md, then repo runbooks/appendix-A-troubleshooting.md |
| CAPI / Magnum / mgmt-VM recovery | repo runbooks/ops-capi-recovery.md |
| Deliver ANY repo change (script, runbook, doc) | run bash scripts/repo-lint.sh + the touched script's tests/<name>/run-tests.sh BEFORE handing it over |
| Pre-deploy gate (before add-model) | bash scripts/preflight.sh -- THE single entry; do not run the sub-gates piecemeal |
| Is the cloud actually healthy? (post-deploy, post-restart, pre-change baseline, incident) | bash scripts/cloud-assert.sh (add --capture at deploy completion for the committed BOM) |
| Full-cloud restart after outage/maintenance | repo runbooks/ops-restart-procedure.md |
VR1 site unreachable after a vcloud reboot (ssh voffice1 / office1-* time out) |
bash scripts/site-baseleg.sh check office1 -- office1-local is an ISOLATED libvirt net, so the host's base L3 leg (10.10.0.10/24 + compose route 10.10.1.0/24 via 10.10.0.20) is dropped every reboot; the After=libvirtd unit re-adds it (D-126 amendment / D-128). If MISS and the unit is not yet installed: sudo bash scripts/site-baseleg.sh apply office1. This is the vcloud path; the workstation reaches the same over the tailnet (D-107). |
| Starting any consequential live session | bash scripts/run-logged.sh <label> first (as-executed log; docs/as-executed-log-convention.md) |
| Credential exposures / security TODOs | repo docs/security-ledger.md -- add a row, never only a script comment |
| Resuming work / what is still in flight (esp. after compaction) | repo docs/session-ledger.md, then bash scripts/ledger-scan.sh to reconcile against repo ground truth |
| Tenant onboarding / tenant self-service | repo scripts/tenant-onboard.sh + runbooks/tenant-onboarding-v2-DRAFT.md + appendix-C/D |
| Network / plane / IPAM questions | references/environment.md, repo scripts/lib-net.sh, NetBox (the IPAM apex -- but during VR1 the WORKING apex is office1-netbox, 10.10.1.10, per DOCFIX-195; netbox.baldurkeep.com is a READ-ONLY v1 reference, write-back deferred to end-of-deployment) |
| ANY change request to a built surface | grep repo docs/design-decisions.md for the governing D-NNN FIRST - PROPOSED/OPEN means the operator has not ruled: present options, do not implement |
| Why is it built this way? / proposing changes | repo docs/design-decisions.md (D-NNN); grep before assigning a new number |
| Versions, channels, pins | repo runbooks/appendix-B-asbuilt-version-lock.md |
| Environment facts (hosts, repo, planes) | references/environment.md |
OpenTofu / libvirt / MAAS / OPNsense IaC work (repo opentofu/) |
references/opentofu-provider-docs.md FIRST (sources + fetch method), then repo opentofu/README.md (per-module ground truth, schema notes, audit findings -- but its OPNsense section is SUPERSEDED by D-112/D-113) |
Creating/changing a libvirt_domain; anything about nested KVM, a libvirt storage pool, LXD versions, an OPNsense edge, or a VM that boots wrong |
references/platform-traps.md -- per-tool traps + version pins + a verbatim-error index (KiB-vs-MiB memory, acpi=off, missing cpu block = no svm, "in-place" bounces the guest, AppArmor on non-default pools, MAAS-vs-LXD 6.7, lxdbr0's dnsmasq, tcsh on the edge) |
Session bootstrap (jumphost): git -C "$REPO" pull (set $REPO once per session -- see runbooks/README.md Conventions / references/environment.md) -> bash scripts/repo-lint.sh (0 fail expected) -> if touching the live cloud, bash scripts/run-logged.sh <label> to open the logged shell. Repo HEAD and a clean lint are the preconditions for everything else.
VR1 operating model (D-128 -- invariant; do not re-derive this each session): Claude runs ON the vcloud jumphost -- do NOT relocate the session to a workstation. Work splits in two, origins FIXED BY DESIGN: Plane 1 (substrate -- the OUTER tofu root qemu:///system + local virsh, creating the DC host VMs) runs on vcloud and has no Roosevelt analog; Plane 2 (MAAS / NetBox / the INNER tofu root qemu+ssh-from-Office1 / tenant tests) EXECUTES on voffice1 via ssh voffice1 '...' so runbooks read "from the Office1 headend" and transfer. Claude's host only picks the TRANSPORT, not where work runs. Durable Office1 reach = the D-126 base-leg oneshot (scripts/site-baseleg.sh check office1 FIRST if ssh voffice1 is dead post-reboot) -- NOT tailscale-on-vcloud (D-107 is the operator's WORKSTATION path / human GUI, left untouched). DCs sit outside the /22 carve and keep the D-126 -J-through-vvr1-dc0 shape.
Change-delivery loop: grep for prior art (zeroth decision) -> grep design-decisions for the governing D-NNN -> edit -> bash scripts/repo-lint.sh -> run the touched script's harness (create one if missing -- no script change ships without its harness) -> deliver as repo-relative ZIP + a changelog entry with a per-item revert. Under blanket approval, the changelog IS the review surface: every item states what, why (evidence), and how to revert.
Deploy loop (VR0, single-DC testcloud): phase-00 runbook (D-061 destroy path) -> bash scripts/preflight.sh PASS -> phase-01..08 gated -> bash scripts/cloud-assert.sh --capture -> commit the asbuilt/ BOM.
Deploy loop (VR1 DC-DC, the current live mission): read docs/dc-dc-deployment-workflow.md's tooling gap register FIRST -- check whether the stage you're about to run has any open blocking item -> Stage 1 (dc-dc-phase0-vcloud-prep.md) gated -> Stage 2..7 gated in order, per runbooks/README.md's "VR1 DC-DC track" table -> if a stage's tofu apply fails partway, read runbooks/dc-dc-teardown-rollback.md's rollback decision tree BEFORE reaching for a teardown (fix-forward is usually correct). Every opentofu/ command is UNVALIDATED (no tofu binary has run in any session that authored this tree) -- run bash scripts/opentofu-validate.sh first if available, and treat every OpenTofu step as reviewed-but-unexercised. At STAGE CLOSE (every close-out gate MET, gauntlet ALL GREEN, repo-lint 0-fail): MERGE the stage branch to main (merge commit, NOT squash; operator-gated) as PART of the stage's definition-of-done, then RETIRE the branch (local + remote). ALSO sweep THIS skill in the same close: fold in any new INVARIANT the stage established and reconcile any divergence from repo HEAD (facts delegate to the repo and go stale; invariants live here and must be kept current). The next stage branches off post-merge main. main is trunk-of-record only for stages that have merged (precedent: Stage 2 -> PR #1 e468d2e, 2026-07-15). See docs/dc-dc-deployment-workflow.md "Cross-cutting discipline".
Incident loop: capture the verbatim error -> bash scripts/cloud-assert.sh (the service-own-verdict sweep localizes the layer) -> appendix-A by exact message -> recorded fix, gated -> log the finding (new root causes become appendix-A/DOCFIX material).