---
name: openstack-cloud-ops
description: "Operate, install, extend, and troubleshoot the Omega Cloud - a commercial multi-tenant Charmed OpenStack (Caracal 2024.1) deployment managed with Juju and MAAS, with Vault TLS, OVN, Ceph, Octavia, and Magnum/CAPI tenant Kubernetes. Use this skill for ANY work touching OpenStack, Juju, MAAS, Magnum, CAPI, Ceph, OVN, Octavia, Keystone, Vault-for-OpenStack, tenant onboarding, or the repository this skill operates against - including writing or reviewing bash/python operational scripts, debugging failed deploys or cluster creates, runbook work, design-decision (D-NNN) discussion, and incident triage. Use it even for seemingly simple OpenStack questions: this deployment has strict operating discipline and known charm traps that make generic answers wrong."
---

# openstack-cloud-ops

Operating skill for the Omega Cloud: a commercial, multi-tenant, tenant
self-administered OpenStack cloud, built as virtual rehearsals (VR0 single-DC
"testcloud"; VR1 two-DC + Office1 headend) for a future bare-metal
multi-datacenter deployment ("Roosevelt"). The CURRENT phase/stage lives in
repo `docs/CURRENT-STATE.md` ONLY (GA-R1) -- never trust a phase claim from
this file or memory. The governing design constraint is
MINIMIZE DELTA TO ROOSEVELT: the runbooks and scripts are primary deliverables
alongside the running cloud, so transferable answers beat quick fixes.

## Step 0 - locate the source of truth

This repository (GitBucket, git.baldurkeep.com) is authoritative for
everything: bundle, runbooks, scripts, design decisions, as-built values. This
skill is a discipline-and-routing layer OVER that repo, not a substitute for
it. Never hardcode the repo's own name/path in a command -- it has already been
renamed/forked once (see D-110) and will again; discover it fresh each session.

1. Look for a local clone: a repo dir in the working tree, or a `$HOME` entry
   containing `bundle.yaml` + `docs/design-decisions.md`. If found, confirm
   identity with `git remote -v`, then `git log -1` to note HEAD and work from it.
2. No clone and you have shell + network: ask the operator for the exact clone
   URL (org/group and repo name -- do not guess or reuse one from memory; it
   may have moved, per this same repo's own history). The repo may be private;
   if the clone fails, ask the operator to provide access or the relevant files.
3. No clone obtainable (e.g. chat without sandbox network): say so, ask the
   operator to paste the relevant runbook/script, and proceed only on what is
   actually in front of you.

**Divergence rule:** if this skill and repo HEAD disagree, the repo wins -
but FLAG the divergence to the operator rather than silently following either.
The repo is a living draft; this skill's invariants (discipline, hardening)
change slowly, its facts (IPs, versions, phase status) go stale fast.

**Session bookends (GA-R1/GA-R4).** OPEN: read `docs/CURRENT-STATE.md` in full
BEFORE any other doc (it is the only status authority), then `docs/session-ledger.md`
+ `bash scripts/ledger-scan.sh` to recover what is in flight and reconcile it against
repo ground truth (open decisions, open SEC count, next-free numbers). Re-do this
after any compaction. CLOSE: a bounded ledger summary (15-line hard cap, GA-R4 rule
1); the full session body goes to `docs/archive/` in the same close commit; commit
and push -- a session ending without its bookend is treated as NOT durable by the
next session. See `references/operating-discipline.md` (Session continuity).
## Step 1 - detect the environment

- **Live shell to the jumphost / infra** (Claude Code on `vcloud` or
  similar): you may RUN read-only audits directly. Every mutation remains
  individually human-gated - present the command, state what it changes, wait
  for approval. A live shell relaxes the transport, never the discipline.
- **Chat / no infra shell**: operate the gated copy-paste model - prepare
  labeled blocks, the operator runs them and pastes output back. Never assume
  a block ran or succeeded; wait for the pasted evidence.

Read `references/operating-discipline.md` before doing either.

## The three hard operating rules (non-negotiable)

1. **Execute only the current runbook step, exactly as written.** No added
   scope, no adjacent improvements, no live re-architecture mid-step. Findings
   and improvement ideas are LOGGED (changelog / D-NNN proposal), never
   executed live mid-step.
2. **Never use an inferred value.** No IP, ID, name, or scope goes into a
   command unless it was measured this session or carried from confirmed
   as-built. If a value would be inferred: stop and measure it. Never run a
   destructive or session-altering command from memory without confirming it
   is the minimal correct action for the current live state.
3. **Prefer dynamic lookups over hardcoded literals.** Discover VIPs, project
   names, IDs, and version sets at runtime. Where a literal is unavoidable it
   is tagged and centralized (`scripts/lib-net.sh`, `lib-hosts.sh`), keyed by
   stable identity (CIDR, hostname) - never by drifting IDs.

Corollary that governs everything: **verify before mutate**. A read-only audit
precedes every mutation; destructive and secret-handling steps are gated
individually, never batched.

## Routing - where to go for what

| Task | Read first |
|---|---|
| Any command block, script, or paste block you are about to write | `references/script-authoring.md` |
| Deploy / redeploy / teardown (VR0, single-DC testcloud) | repo `runbooks/README.md`, then the phase-NN runbook; conventions in `references/operating-discipline.md` |
| VR1 DC-DC buildout (two-DC + Office1 headend -- the current live mission) | repo `docs/CURRENT-STATE.md` FIRST (stage/gate STATUS lives there only), then `docs/dc-dc-deployment-workflow.md` (stage identity, links, gap register), then `runbooks/README.md`'s "VR1 DC-DC track" section for the `dc-dc-phase0..6` runbooks + `dc-dc-teardown-rollback.md`. SEPARATE track from VR0's phase-NN above -- do not conflate stage numbers between the two. |
| Something is broken (triage, incidents) | `references/troubleshooting.md`, then repo `runbooks/appendix-A-troubleshooting.md` |
| CAPI / Magnum / mgmt-VM recovery | repo `runbooks/ops-capi-recovery.md` |
| Deliver ANY repo change (script, runbook, doc) | run `bash scripts/repo-lint.sh` + the touched script's `tests/<name>/run-tests.sh` BEFORE handing it over |
| Pre-deploy gate (before add-model) | `bash scripts/preflight.sh` -- THE single entry; do not run the sub-gates piecemeal |
| Is the cloud actually healthy? (post-deploy, post-restart, pre-change baseline, incident) | `bash scripts/cloud-assert.sh` (add `--capture` at deploy completion for the committed BOM) |
| Full-cloud restart after outage/maintenance | repo `runbooks/ops-restart-procedure.md` |
| VR1 site unreachable after a vcloud reboot (`ssh voffice1` / `office1-*` time out) | `bash scripts/site-baseleg.sh check office1` -- `office1-local` is an ISOLATED libvirt net, so the host's base L3 leg (`10.10.0.10/24` + compose route `10.10.1.0/24 via 10.10.0.20`) is dropped every reboot; the `After=libvirtd` unit re-adds it (D-126 amendment / D-128). If MISS and the unit is not yet installed: `sudo bash scripts/site-baseleg.sh apply office1`. This is the vcloud path; the workstation reaches the same over the tailnet (D-107). |
| Starting any consequential live session | `bash scripts/run-logged.sh <label>` first (as-executed log; docs/as-executed-log-convention.md) |
| Credential exposures / security TODOs | repo `docs/security-ledger.md` -- add a row, never only a script comment |
| Resuming work / what is still in flight (esp. after compaction) | repo `docs/CURRENT-STATE.md` first, then `docs/session-ledger.md` + `bash scripts/ledger-scan.sh` to reconcile against repo ground truth |
| Tenant onboarding / tenant self-service | repo `scripts/tenant-onboard.sh` + `runbooks/tenant-onboarding-v2-DRAFT.md` + `appendix-C/D` |
| Network / plane / IPAM questions | `references/environment.md`, repo `scripts/lib-net.sh`, NetBox (the IPAM apex -- but during VR1 the WORKING apex is `office1-netbox`, 10.10.1.10, per DOCFIX-195; `netbox.baldurkeep.com` is a READ-ONLY v1 reference, write-back deferred to end-of-deployment) |
| ANY change request to a built surface | grep repo `docs/design-decisions.md` for the governing D-NNN FIRST - PROPOSED/OPEN means the operator has not ruled: present options, do not implement |
| Why is it built this way? / proposing changes | repo `docs/design-decisions.md` (D-NNN); grep before assigning a new number |
| Versions, channels, pins | repo `docs/CURRENT-STATE.md` section 7 (VR1 substrate -- measured pin authority); `runbooks/appendix-B-asbuilt-version-lock.md` (VR0 charm channels) |
| Environment facts (hosts, repo, planes) | `references/environment.md` |
| OpenTofu / libvirt / MAAS / OPNsense IaC work (repo `opentofu/`) | `references/opentofu-provider-docs.md` FIRST (sources + fetch method), then repo `opentofu/README.md` (per-module ground truth, schema notes, audit findings -- but its OPNsense section is SUPERSEDED by D-112/D-113) |
| Creating/changing a `libvirt_domain`; anything about nested KVM, a libvirt storage pool, LXD versions, an OPNsense edge, or a VM that boots wrong | `references/platform-traps.md` -- per-tool traps + version pins + a verbatim-error index (KiB-vs-MiB memory, `acpi=off`, missing `cpu` block = no `svm`, "in-place" bounces the guest, AppArmor on non-default pools, MAAS-vs-LXD 6.7, lxdbr0's dnsmasq, tcsh on the edge) |

## Standard loops (repeatable session shapes)

**Session bootstrap (jumphost):** Claude Code sessions launch from the repo
root, always (GA-R7 rule 6 -- a session launched from a subdirectory gets its
own empty project dir and a divergent auto-memory; the GA-F06 origin).
`git -C "$REPO" pull` (set `$REPO` once per
session -- see `runbooks/README.md` Conventions / `references/environment.md`) ->
`bash scripts/repo-lint.sh` (0 fail expected) -> if touching the live cloud,
`bash scripts/run-logged.sh <label>` to open the logged shell. Repo HEAD and a
clean lint are the preconditions for everything else.

**VR1 operating model (D-128 -- invariant; do not re-derive this each session):**
Claude runs ON the vcloud jumphost -- do NOT relocate the session to a workstation.
Work splits in two, origins FIXED BY DESIGN: **Plane 1** (substrate -- the OUTER
`tofu` root `qemu:///system` + local `virsh`, creating the DC *host* VMs) runs on
vcloud and has no Roosevelt analog; **Plane 2** (MAAS / NetBox / the INNER `tofu`
root `qemu+ssh`-from-Office1 / tenant tests) EXECUTES on `voffice1` via `ssh
voffice1 '...'` so runbooks read "from the Office1 headend" and transfer. Claude's
host only picks the TRANSPORT, not where work runs. Durable Office1 reach = the
D-126 base-leg oneshot (`scripts/site-baseleg.sh check office1` FIRST if `ssh
voffice1` is dead post-reboot) -- NOT tailscale-on-vcloud (D-107 is the operator's
WORKSTATION path / human GUI, left untouched). DCs sit outside the `/22` carve and
keep the D-126 `-J`-through-`vvr1-dc0` shape. **Durable host->site reach is standing
per-SITE tooling, DCs included (so the gap does not reopen per-DC):** `site-baseleg.sh`
is site-keyed. At each DC standup, MEASURE how vcloud reaches that DC's containment/
service net -- if it is an ISOLATED host-local net, add a MEASURED leg row + `install`
the unit as part of standup definition-of-done (NEVER an inferred DC IP -- the harness
fails on an un-commented DC supernet); a routed/NAT net or an already-reachable
`qemu+ssh` path needs none. Register item 20 tracks this. **Rack-side twin invariant
(D-131 sub-1, RULED 2026-07-21): every internet-isolated DC rack gets the repo-carried
`scripts/dc-rack-net.sh install <site>` at standup (reboot-persistent rack bridge legs
+ the node-DNS dnsmasq forwarder aliased on metal-admin, forwarding to region BIND over
the rack's own transit) -- MAAS 3.7's rack-only agent resolver SERVFAILs on isolated
racks, so WITHOUT this, commissioning silently times out (appendix-A carries the
symptom pair).** Two measured MAAS traps with standing consequences: MAAS virsh PODS
are refuted for this repo's DCs (per-machine `power_type=virsh` via
`scripts/maas-node-power.sh` is the ruled shape -- D-103/D-123 amendments 2026-07-20),
and a pod delete CASCADES to its linked machine records -- read the pod's machine list
BEFORE any `vm-host delete`; non-empty = STOP (appendix-A, 2026-07-21 incident).

**Change-delivery loop:** grep for prior art (zeroth decision) -> grep
design-decisions for the governing D-NNN -> edit -> `bash scripts/repo-lint.sh`
-> run the touched script's harness (create one if missing -- no script change
ships without its harness) -> deliver with an entry in the SESSION changelog
(ONE changelog per session, GA-R2/D1 -- never per-topic files) carrying a
per-item revert. Under blanket approval, the changelog IS the review surface:
every item states what, why (evidence), and how to revert. Changelogs are
session-scoped scratch: consolidated per-stage at stage close, NEVER citable
as status or decision authority.

**Deploy loop (VR0, single-DC testcloud):** phase-00 runbook (D-061 destroy
path) -> `bash scripts/preflight.sh` PASS -> phase-01..08 gated -> `bash
scripts/cloud-assert.sh --capture` -> commit the asbuilt/ BOM.

**Deploy loop (VR1 DC-DC, the current live mission):** read
`docs/dc-dc-deployment-workflow.md`'s tooling gap register FIRST -- check
whether the stage you're about to run has any open blocking item -> Stage 1
(`dc-dc-phase0-vcloud-prep.md`) gated -> Stage 2..7 gated in order, per
`runbooks/README.md`'s "VR1 DC-DC track" table -> if a stage's `tofu apply`
fails partway, read `runbooks/dc-dc-teardown-rollback.md`'s rollback decision
tree BEFORE reaching for a teardown (fix-forward is usually correct). Run
`bash scripts/opentofu-validate.sh` before any tofu step; whether a given
root/step has been exercised for real is a STATUS question -- check
CURRENT-STATE, never this file's memory of it. **At STAGE
CLOSE** (GA-R6): a stage or verification-type gate CLOSES only on a NAMED executable
check whose captured output the closing commit cites -- prose cannot close a stage,
and there is NO conditional close (a remainder splits into its own gate row or the
stage stays open, E3). Ruling-type gates close per GA-R5 (recorded, pushed ruling).
The close-out set is executable: gauntlet ALL GREEN, repo-lint 0-fail, the stage's
GA-R2 consolidation commit landed, the GA-R7 memory review run, MERGE the stage
branch to `main` (merge commit, NOT squash; operator-gated), then RETIRE the branch
(local + remote). **ALSO sweep THIS skill in the same close:** fold in any new
INVARIANT the stage established and reconcile any divergence from repo HEAD (facts
delegate to the repo and go stale; invariants live here and must be kept current). The next stage branches off post-merge `main`. `main` is
trunk-of-record only for stages that have merged (precedents: Stage 2 -> PR #1 `e468d2e`
2026-07-15; Stage 3 -> merge commit 2026-07-21, vr1-dc0 scope, dc1 HELD under G12). See
`docs/dc-dc-deployment-workflow.md` "Cross-cutting discipline".

**Incident loop:** capture the verbatim error -> `bash scripts/cloud-assert.sh`
(the service-own-verdict sweep localizes the layer) -> appendix-A by exact
message -> recorded fix, gated -> log the finding (new root causes become
appendix-A/DOCFIX material).

## Record authority and session discipline (GA-R1..R7, ratified 2026-07-18)

Ratified text: repo `docs/audit/ga-rulings.md` -- this is the operating summary.

- **`docs/CURRENT-STATE.md` is the ONLY status authority (GA-R1).** Stage,
  gates, plan counts, version pins, decision-queue state live THERE and nowhere
  else; other surfaces point or are history. Any commit that changes a status it
  carries updates it in the SAME commit (C1; repo-lint L10 enforces). It is
  authoritative over documents, never over measurement -- a fresh capture wins
  and corrects it in the capture's commit (C2).
- **Plan counts and versions are quoted only from captured command output** --
  a dated capture file or a quoted command + output; never from memory or
  another document's prose (GA-R1 rule 2). Legal stage/gate status tokens are
  OPEN / BLOCKED / HELD / CLOSED, used only in CURRENT-STATE (vocabulary ruling,
  Option A: readiness is not a state; event words are history, never states).
- **Rulings are gated like mutations (GA-R5).** ONE decision per operator
  exchange -- batch adoptions are invalid. A ruling exists only when its Status
  line quotes the question AND the operator's exact utterance, dated, committed
  and pushed before dependent work. Ambiguous/template answers rule NOTHING:
  stop and ask. Never adopt from an inferred or reconstructed ruling -- an
  operator ruling is a value under hard rule 2.
- **D-number admission (GA-R3):** a new D-number needs architectural consequence
  beyond the stage, a Roosevelt-delta (the A1 test: a Roosevelt build session
  would grep it before touching a built surface), or supersession. Everything
  else is OPERATIONAL: runbook edit / session-changelog line / as-built row,
  never a D-number. Entries carry [ARCH]/[OPS] tags; doubt resolves DOWN to OPS.
- **Memory governance (GA-R7):** auto-memory records observations and
  conventions ONLY -- never operator policy, priorities, or postures. Any
  memory claiming operator intent is PROPOSED until confirmed verbatim; remove
  on discovery. Durable observations GRADUATE to their repo surface; memory is
  never the only copy. Stage close includes a full memory review (rules 1-4).
- **Reconciliation circuit-breaker.** If you have been fixing records to match
  records for more than one cycle without touching the actual deliverable,
  STOP and report to the operator. Record-churn presenting as progress is this
  project's measured failure mode (GA-F06); escalating is the correct move.

## Posture

- This is a commercial multi-tenant cloud with HARD tenant isolation (SCS
  Domain Manager persona). Treat tenant-visible surfaces and cross-domain
  boundaries as security-relevant in every change.
- The operator community here values debate and industry best practice over
  quick fixes. Push back with sources when you disagree; own mistakes plainly
  and concisely. Fabricated flags, values, or version numbers are the cardinal
  sin - if you have not verified an option name or version, say so and verify.
- Responses stay concise. Decisions get explicit rationale.