name: openstack-cloud-ops

description: "Operate, install, extend, and troubleshoot the Omega Cloud - a commercial multi-tenant Charmed OpenStack (Caracal 2024.1) deployment managed with Juju and MAAS, with Vault TLS, OVN, Ceph, Octavia, and Magnum/CAPI tenant Kubernetes. Use this skill for ANY work touching OpenStack, Juju, MAAS, Magnum, CAPI, Ceph, OVN, Octavia, Keystone, Vault-for-OpenStack, tenant onboarding, or the repository this skill operates against - including writing or reviewing bash/python operational scripts, debugging failed deploys or cluster creates, runbook work, design-decision (D-NNN) discussion, and incident triage. Use it even for seemingly simple OpenStack questions: this deployment has strict operating discipline and known charm traps that make generic answers wrong."

openstack-cloud-ops

Operating skill for the Omega Cloud: a commercial, multi-tenant, tenant self-administered OpenStack cloud, built as virtual rehearsals (VR0 single-DC "testcloud"; VR1 two-DC + Office1 headend) for a future bare-metal multi-datacenter deployment ("Roosevelt"). The CURRENT phase/stage lives in repo docs/CURRENT-STATE.md ONLY (GA-R1) -- never trust a phase claim from this file or memory. The governing design constraint is MINIMIZE DELTA TO ROOSEVELT: the runbooks and scripts are primary deliverables alongside the running cloud, so transferable answers beat quick fixes.

Step 0 - locate the source of truth

This repository (GitBucket, git.baldurkeep.com) is authoritative for everything: bundle, runbooks, scripts, design decisions, as-built values. This skill is a discipline-and-routing layer OVER that repo, not a substitute for it. Never hardcode the repo's own name/path in a command -- it has already been renamed/forked once (see D-110) and will again; discover it fresh each session.

1. Look for a local clone: a repo dir in the working tree, or a $HOME entry containing bundle.yaml + docs/design-decisions.md. If found, confirm identity with git remote -v, then git log -1 to note HEAD and work from it.
2. No clone and you have shell + network: ask the operator for the exact clone URL (org/group and repo name -- do not guess or reuse one from memory; it may have moved, per this same repo's own history). The repo may be private; if the clone fails, ask the operator to provide access or the relevant files.
3. No clone obtainable (e.g. chat without sandbox network): say so, ask the operator to paste the relevant runbook/script, and proceed only on what is actually in front of you.

Divergence rule: if this skill and repo HEAD disagree, the repo wins - but FLAG the divergence to the operator rather than silently following either. The repo is a living draft; this skill's invariants (discipline, hardening) change slowly, its facts (IPs, versions, phase status) go stale fast.

Session bookends (GA-R1/GA-R4). OPEN: read docs/CURRENT-STATE.md in full BEFORE any other doc (it is the only status authority), then docs/session-ledger.md

• bash scripts/ledger-scan.sh to recover what is in flight and reconcile it against repo ground truth (open decisions, open SEC count, next-free numbers). Re-do this after any compaction. CLOSE: a bounded ledger summary (15-line hard cap, GA-R4 rule 1); the full session body goes to docs/archive/ in the same close commit; commit and push -- a session ending without its bookend is treated as NOT durable by the next session. See references/operating-discipline.md (Session continuity).

  Step 1 - detect the environment

• Live shell to the jumphost / infra (Claude Code on vcloud or similar): you may RUN read-only audits directly. Every mutation remains individually human-gated - present the command, state what it changes, wait for approval. A live shell relaxes the transport, never the discipline.

• Chat / no infra shell: operate the gated copy-paste model - prepare labeled blocks, the operator runs them and pastes output back. Never assume a block ran or succeeded; wait for the pasted evidence.

Read references/operating-discipline.md before doing either.

The three hard operating rules (non-negotiable)

1. Execute only the current runbook step, exactly as written. No added scope, no adjacent improvements, no live re-architecture mid-step. Findings and improvement ideas are LOGGED (changelog / D-NNN proposal), never executed live mid-step.
2. Never use an inferred value. No IP, ID, name, or scope goes into a command unless it was measured this session or carried from confirmed as-built. If a value would be inferred: stop and measure it. Never run a destructive or session-altering command from memory without confirming it is the minimal correct action for the current live state.
3. Prefer dynamic lookups over hardcoded literals. Discover VIPs, project names, IDs, and version sets at runtime. Where a literal is unavoidable it is tagged and centralized (scripts/lib-net.sh, lib-hosts.sh), keyed by stable identity (CIDR, hostname) - never by drifting IDs.

Corollary that governs everything: verify before mutate. A read-only audit precedes every mutation; destructive and secret-handling steps are gated individually, never batched.

Routing - where to go for what

Standard loops (repeatable session shapes)

Session bootstrap (jumphost): Claude Code sessions launch from the repo root, always (GA-R7 rule 6 -- a session launched from a subdirectory gets its own empty project dir and a divergent auto-memory; the GA-F06 origin). git -C "$REPO" pull (set $REPO once per session -- see runbooks/README.md Conventions / references/environment.md) -> bash scripts/repo-lint.sh (0 fail expected) -> if touching the live cloud, bash scripts/run-logged.sh <label> to open the logged shell. Repo HEAD and a clean lint are the preconditions for everything else.

VR1 operating model (D-128 -- invariant; do not re-derive this each session): Claude runs ON the vcloud jumphost -- do NOT relocate the session to a workstation. Work splits in two, origins FIXED BY DESIGN: Plane 1 (substrate -- the OUTER tofu root qemu:///system + local virsh, creating the DC host VMs) runs on vcloud and has no Roosevelt analog; Plane 2 (MAAS / NetBox / the INNER tofu root qemu+ssh-from-Office1 / tenant tests) EXECUTES on voffice1 via ssh voffice1 '...' so runbooks read "from the Office1 headend" and transfer. Claude's host only picks the TRANSPORT, not where work runs. Durable Office1 reach = the D-126 base-leg oneshot (scripts/site-baseleg.sh check office1 FIRST if ssh voffice1 is dead post-reboot) -- NOT tailscale-on-vcloud (D-107 is the operator's WORKSTATION path / human GUI, left untouched). DCs sit outside the /22 carve and keep the D-126 -J-through-vvr1-dc0 shape. Durable host->site reach is standing per-SITE tooling, DCs included (so the gap does not reopen per-DC): site-baseleg.sh is site-keyed. At each DC standup, MEASURE how vcloud reaches that DC's containment/ service net -- if it is an ISOLATED host-local net, add a MEASURED leg row + install the unit as part of standup definition-of-done (NEVER an inferred DC IP -- the harness fails on an un-commented DC supernet); a routed/NAT net or an already-reachable qemu+ssh path needs none. Register item 20 tracks this. **Rack-side twin invariant (D-131 sub-1, RULED 2026-07-21): every internet-isolated DC rack gets the repo-carried scripts/dc-rack-net.sh install <site> at standup (reboot-persistent rack bridge legs

• the node-DNS dnsmasq forwarder aliased on metal-admin, forwarding to region BIND over the rack's own transit) -- MAAS 3.7's rack-only agent resolver SERVFAILs on isolated racks, so WITHOUT this, commissioning silently times out (appendix-A carries the symptom pair).** Two measured MAAS traps with standing consequences: MAAS virsh PODS are refuted for this repo's DCs (per-machine power_type=virsh via scripts/maas-node-power.sh is the ruled shape -- D-103/D-123 amendments 2026-07-20), and a pod delete CASCADES to its linked machine records -- read the pod's machine list BEFORE any vm-host delete; non-empty = STOP (appendix-A, 2026-07-21 incident).

Change-delivery loop: grep for prior art (zeroth decision) -> grep design-decisions for the governing D-NNN -> edit -> bash scripts/repo-lint.sh -> run the touched script's harness (create one if missing -- no script change ships without its harness) -> deliver with an entry in the SESSION changelog (ONE changelog per session, GA-R2/D1 -- never per-topic files) carrying a per-item revert. Under blanket approval, the changelog IS the review surface: every item states what, why (evidence), and how to revert. Changelogs are session-scoped scratch: consolidated per-stage at stage close, NEVER citable as status or decision authority.

Deploy loop (VR0, single-DC testcloud): phase-00 runbook (D-061 destroy path) -> bash scripts/preflight.sh PASS -> phase-01..08 gated -> bash scripts/cloud-assert.sh --capture -> commit the asbuilt/ BOM.

Deploy loop (VR1 DC-DC, the current live mission): read docs/dc-dc-deployment-workflow.md's tooling gap register FIRST -- check whether the stage you're about to run has any open blocking item -> Stage 1 (dc-dc-phase0-vcloud-prep.md) gated -> Stage 2..7 gated in order, per runbooks/README.md's "VR1 DC-DC track" table -> if a stage's tofu apply fails partway, read runbooks/dc-dc-teardown-rollback.md's rollback decision tree BEFORE reaching for a teardown (fix-forward is usually correct). Run bash scripts/opentofu-validate.sh before any tofu step; whether a given root/step has been exercised for real is a STATUS question -- check CURRENT-STATE, never this file's memory of it. At STAGE CLOSE (GA-R6): a stage or verification-type gate CLOSES only on a NAMED executable check whose captured output the closing commit cites -- prose cannot close a stage, and there is NO conditional close (a remainder splits into its own gate row or the stage stays open, E3). Ruling-type gates close per GA-R5 (recorded, pushed ruling). The close-out set is executable: gauntlet ALL GREEN, repo-lint 0-fail, the stage's GA-R2 consolidation commit landed, the GA-R7 memory review run, MERGE the stage branch to main (merge commit, NOT squash; operator-gated), then RETIRE the branch (local + remote). ALSO sweep THIS skill in the same close: fold in any new INVARIANT the stage established and reconcile any divergence from repo HEAD (facts delegate to the repo and go stale; invariants live here and must be kept current). The next stage branches off post-merge main. main is trunk-of-record only for stages that have merged (precedents: Stage 2 -> PR #1 e468d2e 2026-07-15; Stage 3 -> merge commit 2026-07-21, vr1-dc0 scope, dc1 HELD under G12). See docs/dc-dc-deployment-workflow.md "Cross-cutting discipline".

Incident loop: capture the verbatim error -> bash scripts/cloud-assert.sh (the service-own-verdict sweep localizes the layer) -> appendix-A by exact message -> recorded fix, gated -> log the finding (new root causes become appendix-A/DOCFIX material).

Record authority and session discipline (GA-R1..R7, ratified 2026-07-18)

Ratified text: repo docs/audit/ga-rulings.md -- this is the operating summary.

• docs/CURRENT-STATE.md is the ONLY status authority (GA-R1). Stage, gates, plan counts, version pins, decision-queue state live THERE and nowhere else; other surfaces point or are history. Any commit that changes a status it carries updates it in the SAME commit (C1; repo-lint L10 enforces). It is authoritative over documents, never over measurement -- a fresh capture wins and corrects it in the capture's commit (C2).
• Plan counts and versions are quoted only from captured command output -- a dated capture file or a quoted command + output; never from memory or another document's prose (GA-R1 rule 2). Legal stage/gate status tokens are OPEN / BLOCKED / HELD / CLOSED, used only in CURRENT-STATE (vocabulary ruling, Option A: readiness is not a state; event words are history, never states).
• Rulings are gated like mutations (GA-R5). ONE decision per operator exchange -- batch adoptions are invalid. A ruling exists only when its Status line quotes the question AND the operator's exact utterance, dated, committed and pushed before dependent work. Ambiguous/template answers rule NOTHING: stop and ask. Never adopt from an inferred or reconstructed ruling -- an operator ruling is a value under hard rule 2.
• D-number admission (GA-R3): a new D-number needs architectural consequence beyond the stage, a Roosevelt-delta (the A1 test: a Roosevelt build session would grep it before touching a built surface), or supersession. Everything else is OPERATIONAL: runbook edit / session-changelog line / as-built row, never a D-number. Entries carry [ARCH]/[OPS] tags; doubt resolves DOWN to OPS.
• Memory governance (GA-R7): auto-memory records observations and conventions ONLY -- never operator policy, priorities, or postures. Any memory claiming operator intent is PROPOSED until confirmed verbatim; remove on discovery. Durable observations GRADUATE to their repo surface; memory is never the only copy. Stage close includes a full memory review (rules 1-4).
• Reconciliation circuit-breaker. If you have been fixing records to match records for more than one cycle without touching the actual deliverable, STOP and report to the operator. Record-churn presenting as progress is this project's measured failure mode (GA-F06); escalating is the correct move.

Posture

• This is a commercial multi-tenant cloud with HARD tenant isolation (SCS Domain Manager persona). Treat tenant-visible surfaces and cross-domain boundaries as security-relevant in every change.
• The operator community here values debate and industry best practice over quick fixes. Push back with sources when you disagree; own mistakes plainly and concisely. Fabricated flags, values, or version numbers are the cardinal sin - if you have not verified an option name or version, say so and verify.
• Responses stay concise. Decisions get explicit rationale.