---
name: openstack-cloud-ops
description: "Operate, install, extend, and troubleshoot the Omega Cloud - a commercial multi-tenant Charmed OpenStack (Caracal 2024.1) deployment managed with Juju and MAAS, with Vault TLS, OVN, Ceph, Octavia, and Magnum/CAPI tenant Kubernetes. Use this skill for ANY work touching OpenStack, Juju, MAAS, Magnum, CAPI, Ceph, OVN, Octavia, Keystone, Vault-for-OpenStack, tenant onboarding, or the repository this skill operates against - including writing or reviewing bash/python operational scripts, debugging failed deploys or cluster creates, runbook work, design-decision (D-NNN) discussion, and incident triage. Use it even for seemingly simple OpenStack questions: this deployment has strict operating discipline and known charm traps that make generic answers wrong."
---

# openstack-cloud-ops

Operating skill for the Omega Cloud: a commercial, multi-tenant, tenant
self-administered OpenStack cloud, built as virtual rehearsals (VR0 single-DC
"testcloud"; VR1 two-DC + Office1 headend) for a future bare-metal
multi-datacenter deployment ("Roosevelt"). The CURRENT phase/stage lives in
repo `docs/CURRENT-STATE.md` ONLY (GA-R1) -- never trust a phase claim from
this file or memory. The governing design constraint is
MINIMIZE DELTA TO ROOSEVELT: the runbooks and scripts are primary deliverables
alongside the running cloud, so transferable answers beat quick fixes.

## Step 0 - locate the source of truth

This repository (GitBucket, git.baldurkeep.com) is authoritative for
everything: bundle, runbooks, scripts, design decisions, as-built values. This
skill is a discipline-and-routing layer OVER that repo, not a substitute for
it. Never hardcode the repo's own name/path in a command -- it has already been
renamed/forked once (see D-110) and will again; discover it fresh each session.

1. Look for a local clone: a repo dir in the working tree, or a `$HOME` entry
   containing `bundle.yaml` + `docs/design-decisions.md`. If found, confirm
   identity with `git remote -v`, then `git log -1` to note HEAD and work from it.
2. No clone and you have shell + network: ask the operator for the exact clone
   URL (org/group and repo name -- do not guess or reuse one from memory; it
   may have moved, per this same repo's own history). The repo may be private;
   if the clone fails, ask the operator to provide access or the relevant files.
3. No clone obtainable (e.g. chat without sandbox network): say so, ask the
   operator to paste the relevant runbook/script, and proceed only on what is
   actually in front of you.

**Divergence rule:** if this skill and repo HEAD disagree, the repo wins -
but FLAG the divergence to the operator rather than silently following either.
The repo is a living draft; this skill's invariants (discipline, hardening)
change slowly, its facts (IPs, versions, phase status) go stale fast.

**Session bookends (GA-R1/GA-R4).** OPEN: read `docs/CURRENT-STATE.md` in full
BEFORE any other doc (it is the only status authority), then `docs/session-ledger.md`
+ `bash scripts/ledger-scan.sh` to recover what is in flight and reconcile it against
repo ground truth (open decisions, open SEC count, next-free numbers). Re-do this
after any compaction. CLOSE: a bounded ledger summary (15-line hard cap, GA-R4 rule
1); the full session body goes to `docs/archive/` in the same close commit; commit
and push -- a session ending without its bookend is treated as NOT durable by the
next session. See `references/operating-discipline.md` (Session continuity).
## Step 1 - detect the environment

- **Live shell to the jumphost / infra** (Claude Code on `vcloud` or
  similar): you may RUN read-only audits directly. Every mutation remains
  individually human-gated - present the command, state what it changes, wait
  for approval. A live shell relaxes the transport, never the discipline.
- **Chat / no infra shell**: operate the gated copy-paste model - prepare
  labeled blocks, the operator runs them and pastes output back. Never assume
  a block ran or succeeded; wait for the pasted evidence.

Read `references/operating-discipline.md` before doing either.

## The three hard operating rules (non-negotiable)

1. **Execute only the current runbook step, exactly as written.** No added
   scope, no adjacent improvements, no live re-architecture mid-step. Findings
   and improvement ideas are LOGGED (changelog / D-NNN proposal), never
   executed live mid-step.
2. **Never use an inferred value.** No IP, ID, name, or scope goes into a
   command unless it was measured this session or carried from confirmed
   as-built. If a value would be inferred: stop and measure it. Never run a
   destructive or session-altering command from memory without confirming it
   is the minimal correct action for the current live state.
3. **Prefer dynamic lookups over hardcoded literals.** Discover VIPs, project
   names, IDs, and version sets at runtime. Where a literal is unavoidable it
   is tagged and centralized (`scripts/lib-net.sh`, `lib-hosts.sh`), keyed by
   stable identity (CIDR, hostname) - never by drifting IDs.

Corollary that governs everything: **verify before mutate**. A read-only audit
precedes every mutation; destructive and secret-handling steps are gated
individually, never batched.

## Routing - where to go for what

| Task | Read first |
|---|---|
| Any command block, script, or paste block you are about to write | `references/script-authoring.md` |
| Deploy / redeploy / teardown (VR0, single-DC testcloud) | repo `runbooks/README.md`, then the phase-NN runbook; conventions in `references/operating-discipline.md` |
| VR1 DC-DC buildout (two-DC + Office1 headend -- the current live mission) | repo `docs/CURRENT-STATE.md` FIRST (stage/gate STATUS lives there only), then `docs/dc-dc-deployment-workflow.md` (stage identity, links, gap register), then `runbooks/README.md`'s "VR1 DC-DC track" section for the `dc-dc-phase0..6` runbooks + `dc-dc-teardown-rollback.md`. SEPARATE track from VR0's phase-NN above -- do not conflate stage numbers between the two. |
| Something is broken (triage, incidents) | `references/troubleshooting.md`, then repo `runbooks/appendix-A-troubleshooting.md` |
| CAPI / Magnum / mgmt-VM recovery | repo `runbooks/ops-capi-recovery.md` |
| Deliver ANY repo change (script, runbook, doc) | run `bash scripts/repo-lint.sh` + the touched script's `tests/<name>/run-tests.sh` BEFORE handing it over |
| Pre-deploy gate (before add-model) | `bash scripts/preflight.sh` -- THE single entry; do not run the sub-gates piecemeal |
| Is the cloud actually healthy? (post-deploy, post-restart, pre-change baseline, incident) | `bash scripts/cloud-assert.sh` (add `--capture` at deploy completion for the committed BOM) |
| Full-cloud restart after outage/maintenance | repo `runbooks/ops-restart-procedure.md` |
| VR1 site unreachable after a vcloud reboot (`ssh voffice1` / `office1-*` time out) | `bash scripts/site-baseleg.sh check office1` -- `office1-local` is an ISOLATED libvirt net, so the host's base L3 leg (`10.10.0.10/24` + compose route `10.10.1.0/24 via 10.10.0.20`) is dropped every reboot; the `After=libvirtd` unit re-adds it (D-126 amendment / D-128). If MISS and the unit is not yet installed: `sudo bash scripts/site-baseleg.sh apply office1`. This is the vcloud path; the workstation reaches the same over the tailnet (D-107). |
| Starting any consequential live session | `bash scripts/run-logged.sh <label>` first (as-executed log; docs/as-executed-log-convention.md) |
| Credential exposures / security TODOs | repo `docs/security-ledger.md` -- add a row, never only a script comment. The credential-LIFECYCLE policy (mint -> consolidate -> audit) is **D-137**, not the ledger (SEC-009's convention block is now a pointer); the expected-state register is `creds-matrix.tsv` + `scripts/creds-matrix.py` (tier 1 static, tier 2 existence, tier 3 validity), enforced as a BLOCKING preflight `P5`. `creds-audit` CLEAN is NOT evidence of completeness -- it is declaration-based |
| Per-DC artifact delivery (apt mirror / caching proxy); "can the nodes get packages?" | dc0 = `bash scripts/dc-mirror.sh check dc0` (full mirror); dc1 = `bash scripts/dc-cache-proxy.sh check dc1` (caching proxy). The two DCs DIFFER BY RULING (D-135 amendments) -- checking dc1 as a mirror fails by design. Both run ON the rack host over `ssh -J voffice1` with `sudo bash -s`; take rack transit IPs from `scripts/lib-hosts.sh`, never memory |
| Resuming work / what is still in flight (esp. after compaction) | repo `docs/CURRENT-STATE.md` first, then `docs/session-ledger.md` + `bash scripts/ledger-scan.sh` to reconcile against repo ground truth |
| Tenant onboarding / tenant self-service | repo `scripts/tenant-onboard.sh` + `runbooks/tenant-onboarding-v2-DRAFT.md` + `appendix-C/D` |
| Network / plane / IPAM questions | `references/environment.md`, repo `scripts/lib-net.sh`, NetBox (the IPAM apex -- but during VR1 the WORKING apex is `office1-netbox`, 10.10.1.10, per DOCFIX-195; `netbox.baldurkeep.com` is a READ-ONLY v1 reference, write-back deferred to end-of-deployment) |
| ANY change request to a built surface | grep repo `docs/design-decisions.md` for the governing D-NNN FIRST - PROPOSED/OPEN means the operator has not ruled: present options, do not implement |
| Why is it built this way? / proposing changes | repo `docs/design-decisions.md` (D-NNN); grep before assigning a new number |
| Versions, channels, pins | repo `docs/CURRENT-STATE.md` section 7 (VR1 substrate -- measured pin authority); `runbooks/appendix-B-asbuilt-version-lock.md` (VR0 charm channels) |
| Environment facts (hosts, repo, planes) | `references/environment.md` |
| OpenTofu / libvirt / MAAS / OPNsense IaC work (repo `opentofu/`) | `references/opentofu-provider-docs.md` FIRST (sources + fetch method), then repo `opentofu/README.md` (per-module ground truth, schema notes, audit findings -- but its OPNsense section is SUPERSEDED by D-112/D-113) |
| Creating/changing a `libvirt_domain`; anything about nested KVM, a libvirt storage pool, LXD versions, an OPNsense edge, or a VM that boots wrong | `references/platform-traps.md` -- per-tool traps + version pins + a verbatim-error index (KiB-vs-MiB memory, `acpi=off`, missing `cpu` block = no `svm`, "in-place" bounces the guest, AppArmor on non-default pools, MAAS-vs-LXD 6.7, lxdbr0's dnsmasq, tcsh on the edge) |

## Standard loops (repeatable session shapes)

**Session bootstrap (jumphost):** Claude Code sessions launch from the repo
root, always (GA-R7 rule 6 -- a session launched from a subdirectory gets its
own empty project dir and a divergent auto-memory; the GA-F06 origin).
`git -C "$REPO" pull` (set `$REPO` once per
session -- see `runbooks/README.md` Conventions / `references/environment.md`) ->
`bash scripts/repo-lint.sh` (0 fail expected) -> if touching the live cloud,
`bash scripts/run-logged.sh <label>` to open the logged shell. Repo HEAD and a
clean lint are the preconditions for everything else.

**VR1 operating model (D-128 -- invariant; do not re-derive this each session):**
Claude runs ON the vcloud jumphost -- do NOT relocate the session to a workstation.
Work splits in two, origins FIXED BY DESIGN: **Plane 1** (substrate -- the OUTER
`tofu` root `qemu:///system` + local `virsh`, creating the DC *host* VMs) runs on
vcloud and has no Roosevelt analog; **Plane 2** (MAAS / NetBox / the INNER `tofu`
root `qemu+ssh`-from-Office1 / tenant tests) EXECUTES on `voffice1` via `ssh
voffice1 '...'` so runbooks read "from the Office1 headend" and transfer. Claude's
host only picks the TRANSPORT, not where work runs. Durable Office1 reach = the
D-126 base-leg oneshot (`scripts/site-baseleg.sh check office1` FIRST if `ssh
voffice1` is dead post-reboot) -- NOT tailscale-on-vcloud (D-107 is the operator's
WORKSTATION path / human GUI, left untouched). DCs sit outside the `/22` carve and
keep the D-126 `-J`-through-`vvr1-dc0` shape. **Durable host->site reach is standing
per-SITE tooling, DCs included (so the gap does not reopen per-DC):** `site-baseleg.sh`
is site-keyed. At each DC standup, MEASURE how vcloud reaches that DC's containment/
service net -- if it is an ISOLATED host-local net, add a MEASURED leg row + `install`
the unit as part of standup definition-of-done (NEVER an inferred DC IP -- the harness
fails on an un-commented DC supernet); a routed/NAT net or an already-reachable
`qemu+ssh` path needs none. Register item 20 tracks this. **Rack-side twin invariant
(D-131 sub-1, RULED 2026-07-21): every internet-isolated DC rack gets the repo-carried
`scripts/dc-rack-net.sh install <site>` at standup (reboot-persistent rack bridge legs
+ the node-DNS dnsmasq forwarder aliased on metal-admin, forwarding to region BIND over
the rack's own transit) -- MAAS 3.7's rack-only agent resolver SERVFAILs on isolated
racks, so WITHOUT this, commissioning silently times out (appendix-A carries the
symptom pair).** Two measured MAAS traps with standing consequences: MAAS virsh PODS
are refuted for this repo's DCs (per-machine `power_type=virsh` via
`scripts/maas-node-power.sh` is the ruled shape -- D-103/D-123 amendments 2026-07-20),
and a pod delete CASCADES to its linked machine records -- read the pod's machine list
BEFORE any `vm-host delete`; non-empty = STOP (appendix-A, 2026-07-21 incident).
**Per-DC power credential (SEC-012 dc0, SEC-016 dc1, RULED 2026-07-23 -- standing
DC-standup invariant): each DC gets its OWN dedicated MAAS->libvirt power key, NEVER
a cross-DC reuse.** The REGION's MAAS snap dials the power address, so the split is:
the maas-node-power SCRIPT's mapping virsh uses the DC SERVICE key (via the region's
`~/.ssh/config` Host <rack-transit-ip>), while MAAS's own power ops use the dedicated
key installed in the snap (`/var/snap/maas/current/root/.ssh/` + a per-host `ssh
config` block so each rack uses its own key). FRAGILITY: the snap-side key lives under
per-revision `/var/snap/maas/current/` and may not survive a snap refresh -- re-assert
after any refresh. Roosevelt analog: per-DC IPMI/BMC credentials.

**Per-DC ARTIFACT DELIVERY is a per-DC STRATEGY, and the two DCs deliberately differ
(D-135 + its 2026-07-24/27 amendments -- standing invariant).** dc0 tests a FULL MIRROR
(`dc-mirror.sh`, debmirror+nginx on the D-134 utility `.4`); dc1 tests an APT CACHING
PROXY (`dc-cache-proxy.sh`, apt-cacher-ng on `.4:3142`, consumed via
`juju apt-http-proxy`). This is the EXPERIMENT, not an interim state -- there is no
fallback and no later convergence, and dc1's partial mirror was REMOVED 2026-07-27.
Consequences that bite: **checking dc1 as a mirror FAILS BY DESIGN** (use its own
checker), and each DC's gate is its own script's `check <site>`.
**Corollary invariant, learned the hard way: a per-DC utility service OWNS ITS OWN
NETWORK PREREQUISITES.** `dc-cache-proxy.sh` originally REUSED `dc-mirror.sh`'s
`<site>-mirror-net.service` for the `.4` alias + edge default route, so on a proxy-only
DC three files named `*-mirror-*` were load-bearing FOR THE PROXY: the documented
mirror-removal procedure would have silently killed it, and the proxy could not be
REBUILT afterward without reinstalling the whole mirror apparatus (enabled daily
debmirror timer included). Never let one strategy's teardown be able to break the
other's; name artifacts for the service that owns them. Roosevelt analog: a DC must be
able to change artifact strategy without rebuilding the other strategy's apparatus.

**Stage 4 hands off READY nodes, NOT deployed ones (DOCFIX-200, RULED 2026-07-23 --
standing).** MAAS-deploy is SKIPPED; Juju provisions at Stage 5. Any surface saying
"nodes Deployed" as a Stage-4 exit condition is a DEFECT, not a fact -- four were found
and fixed on 2026-07-27 (workflow-doc gate cell, phase-3 DoD + Step 7, phase-4
prerequisites). Practical consequence: **nothing node-side can be probed in Stage 4**
(the nodes are powered off), so node-side checks belong at Stage 5 first boot -- that is
why gate G17 exists.

**A CHECKER THAT CANNOT FAIL IS NOT A GATE (this project's most productive audit move).**
GA-R6 lets a stage close only on a named executable check, so the check itself must be
verified capable of failing. Two measured instances, days apart: `creds-audit` is
DECLARATION-based and read CLEAN on three sites while the D-137 matrix returned 13
findings; `dc-mirror.sh check` asserted that `last-sync.status` EXISTED and printed its
contents behind an unconditional `OK`, so a `FAIL` status and a four-day-stale `RUNNING`
both passed. Working rules that came out of it: assert on CONTENT, never on existence;
an unrecognised state REFUSES rather than defaulting to success; "could not look" is
never "nothing there"; and **enumerate what EXISTS rather than only auditing what is
declared** -- a register cannot report the absence of something it has no row for, which
is how an unregistered credential at both DC edges survived every audit. When a test
asserts a literal finding string, remediating the finding turns the harness red: REPLACE
the assertion with the new invariant and say so, never delete it to go green.

**Change-delivery loop:** grep for prior art (zeroth decision) -> grep
design-decisions for the governing D-NNN -> edit -> `bash scripts/repo-lint.sh`
-> run the touched script's harness (create one if missing -- no script change
ships without its harness) -> deliver with an entry in the SESSION changelog
(ONE changelog per session, GA-R2/D1 -- never per-topic files) carrying a
per-item revert. Under blanket approval, the changelog IS the review surface:
every item states what, why (evidence), and how to revert. Changelogs are
session-scoped scratch: consolidated per-stage at stage close, NEVER citable
as status or decision authority.

**Deploy loop (VR0, single-DC testcloud):** phase-00 runbook (D-061 destroy
path) -> `bash scripts/preflight.sh` PASS -> phase-01..08 gated -> `bash
scripts/cloud-assert.sh --capture` -> commit the asbuilt/ BOM.

**Deploy loop (VR1 DC-DC, the current live mission):** read
`docs/dc-dc-deployment-workflow.md`'s tooling gap register FIRST -- check
whether the stage you're about to run has any open blocking item -> Stage 1
(`dc-dc-phase0-vcloud-prep.md`) gated -> Stage 2..7 gated in order, per
`runbooks/README.md`'s "VR1 DC-DC track" table -> if a stage's `tofu apply`
fails partway, read `runbooks/dc-dc-teardown-rollback.md`'s rollback decision
tree BEFORE reaching for a teardown (fix-forward is usually correct). Run
`bash scripts/opentofu-validate.sh` before any tofu step; whether a given
root/step has been exercised for real is a STATUS question -- check
CURRENT-STATE, never this file's memory of it. **At STAGE
CLOSE** (GA-R6): a stage or verification-type gate CLOSES only on a NAMED executable
check whose captured output the closing commit cites -- prose cannot close a stage,
and there is NO conditional close (a remainder splits into its own gate row or the
stage stays open, E3). Ruling-type gates close per GA-R5 (recorded, pushed ruling).
The close-out set is executable: gauntlet ALL GREEN, repo-lint 0-fail, the stage's
GA-R2 consolidation commit landed, the GA-R7 memory review run, MERGE the stage
branch to `main` (merge commit, NOT squash; operator-gated), then RETIRE the branch
(local + remote). **ALSO sweep THIS skill in the same close:** fold in any new
INVARIANT the stage established and reconcile any divergence from repo HEAD (facts
delegate to the repo and go stale; invariants live here and must be kept current). The
AUTHORITATIVE, versioned skill is THIS dir (`.claude/skills/openstack-cloud-ops/` -- SKILL.md
+ references); keep it current here. THEN REGENERATE the dated single-file Chat-upload
snapshot from it (concatenate SKILL.md + references -> `.claude/skills/openstack-cloud-ops-consolidated-<date>.md`,
ASCII/LF byte-verified) so the Chat/project skill surface can be refreshed -- the snapshot
is a DERIVED artifact, never the source, and the Chat mount going stale is a known failure
mode (it caused a failed clone + a mis-shaped handoff, 2026-07-25; docs/audit/skill-divergence-20260725.md).
The next stage branches off post-merge `main`. `main` is
trunk-of-record only for stages that have merged (precedents: Stage 2 -> PR #1 `e468d2e`
2026-07-15; Stage 3 -> merge commit 2026-07-21, vr1-dc0 scope, dc1 HELD under G12). See
`docs/dc-dc-deployment-workflow.md` "Cross-cutting discipline".

**Incident loop:** capture the verbatim error -> `bash scripts/cloud-assert.sh`
(the service-own-verdict sweep localizes the layer) -> appendix-A by exact
message -> recorded fix, gated -> log the finding (new root causes become
appendix-A/DOCFIX material).

## Record authority and session discipline (GA-R1..R7, ratified 2026-07-18)

Ratified text: repo `docs/audit/ga-rulings.md` -- this is the operating summary.

- **`docs/CURRENT-STATE.md` is the ONLY status authority (GA-R1).** Stage,
  gates, plan counts, version pins, decision-queue state live THERE and nowhere
  else; other surfaces point or are history. Any commit that changes a status it
  carries updates it in the SAME commit (C1; repo-lint L10 enforces). It is
  authoritative over documents, never over measurement -- a fresh capture wins
  and corrects it in the capture's commit (C2).
- **Plan counts and versions are quoted only from captured command output** --
  a dated capture file or a quoted command + output; never from memory or
  another document's prose (GA-R1 rule 2). Legal stage/gate status tokens are
  OPEN / BLOCKED / HELD / CLOSED, used only in CURRENT-STATE (vocabulary ruling,
  Option A: readiness is not a state; event words are history, never states).
- **Rulings are gated like mutations (GA-R5).** ONE decision per operator
  exchange -- batch adoptions are invalid. A ruling exists only when its Status
  line quotes the question AND the operator's exact utterance, dated, committed
  and pushed before dependent work. Ambiguous/template answers rule NOTHING:
  stop and ask. Never adopt from an inferred or reconstructed ruling -- an
  operator ruling is a value under hard rule 2.
- **D-number admission (GA-R3):** a new D-number needs architectural consequence
  beyond the stage, a Roosevelt-delta (the A1 test: a Roosevelt build session
  would grep it before touching a built surface), or supersession. Everything
  else is OPERATIONAL: runbook edit / session-changelog line / as-built row,
  never a D-number. Entries carry [ARCH]/[OPS] tags; doubt resolves DOWN to OPS.
- **Memory governance (GA-R7):** auto-memory records observations and
  conventions ONLY -- never operator policy, priorities, or postures. Any
  memory claiming operator intent is PROPOSED until confirmed verbatim; remove
  on discovery. Durable observations GRADUATE to their repo surface; memory is
  never the only copy. Stage close includes a full memory review (rules 1-4).
- **Reconciliation circuit-breaker.** If you have been fixing records to match
  records for more than one cycle without touching the actual deliverable,
  STOP and report to the operator. Record-churn presenting as progress is this
  project's measured failure mode (GA-F06); escalating is the correct move.

## Posture

- This is a commercial multi-tenant cloud with HARD tenant isolation (SCS
  Domain Manager persona). Treat tenant-visible surfaces and cross-domain
  boundaries as security-relevant in every change.
- The operator community here values debate and industry best practice over
  quick fixes. Push back with sources when you disagree; own mistakes plainly
  and concisely. Fabricated flags, values, or version numbers are the cardinal
  sin - if you have not verified an option name or version, say so and verify.
- Responses stay concise. Decisions get explicit rationale.
