---
name: openstack-cloud-ops
description: "Operate, install, extend, and troubleshoot the Omega Cloud - a commercial multi-tenant Charmed OpenStack (Caracal 2024.1) deployment managed with Juju and MAAS, with Vault TLS, OVN, Ceph, Octavia, and Magnum/CAPI tenant Kubernetes. Use this skill for ANY work touching OpenStack, Juju, MAAS, Magnum, CAPI, Ceph, OVN, Octavia, Keystone, Vault-for-OpenStack, tenant onboarding, or the repository this skill operates against - including writing or reviewing bash/python operational scripts, debugging failed deploys or cluster creates, runbook work, design-decision (D-NNN) discussion, and incident triage. Use it even for seemingly simple OpenStack questions: this deployment has strict operating discipline and known charm traps that make generic answers wrong."
---

# openstack-cloud-ops

Operating skill for the Omega Cloud: a commercial, multi-tenant, tenant
self-administered OpenStack cloud, built as virtual rehearsals (VR0 single-DC
"testcloud"; VR1 two-DC + Office1 headend) for a future bare-metal
multi-datacenter deployment ("Roosevelt"). The CURRENT phase/stage lives in
repo `docs/CURRENT-STATE.md` ONLY (GA-R1) -- never trust a phase claim from
this file or memory. The governing design constraint is
MINIMIZE DELTA TO ROOSEVELT: the runbooks and scripts are primary deliverables
alongside the running cloud, so transferable answers beat quick fixes.

## Step 0 - locate the source of truth

This repository (GitBucket, git.baldurkeep.com) is authoritative for
everything: bundle, runbooks, scripts, design decisions, as-built values. This
skill is a discipline-and-routing layer OVER that repo, not a substitute for
it. Never hardcode the repo's own name/path in a command -- it has already been
renamed/forked once (see D-110) and will again; discover it fresh each session.

1. Look for a local clone: a repo dir in the working tree, or a `$HOME` entry
   containing `bundle.yaml` + `docs/design-decisions.md`. If found, confirm
   identity with `git remote -v`, then `git log -1` to note HEAD and work from it.
2. No clone and you have shell + network: ask the operator for the exact clone
   URL (org/group and repo name -- do not guess or reuse one from memory; it
   may have moved, per this same repo's own history). The repo may be private;
   if the clone fails, ask the operator to provide access or the relevant files.
3. No clone obtainable (e.g. chat without sandbox network): say so, ask the
   operator to paste the relevant runbook/script, and proceed only on what is
   actually in front of you.

**Divergence rule:** if this skill and repo HEAD disagree, the repo wins -
but FLAG the divergence to the operator rather than silently following either.
The repo is a living draft; this skill's invariants (discipline, hardening)
change slowly, its facts (IPs, versions, phase status) go stale fast.

**Session bookends (GA-R1/GA-R4).** OPEN: read `docs/CURRENT-STATE.md` in full
BEFORE any other doc (it is the only status authority), then `docs/session-ledger.md`
+ `bash scripts/ledger-scan.sh` to recover what is in flight and reconcile it against
repo ground truth (open decisions, open SEC count, next-free numbers). Re-do this
after any compaction. CLOSE: a bounded <=15-line summary (GA-R4 rule 1) appended to
`docs/session-ledger.md` as a `## SESSION CLOSE <date>` block; the session's full body
is its changelog (`docs/changelog-<date>-<label>.md`, GA-R2 one-per-session), cited by
that block's `Body:` line. `docs/archive/` is NOT the per-session body store -- it holds
periodically-rotated full ledgers (`session-ledger-rotated-*.md`, GA-R4 rule 3) and
per-stage consolidated records (`archive/changelogs/`, `archive/stage-records/`, GA-R2
rule 2 at STAGE close). Commit and push -- a session ending without its bookend is
treated as NOT durable by the next session. See
`references/operating-discipline.md` (Session continuity).
## Step 1 - detect the environment

- **Live shell to the jumphost / infra** (Claude Code on `vcloud` or
  similar): you may RUN read-only audits directly. Every mutation remains
  individually human-gated - present the command, state what it changes, wait
  for approval. A live shell relaxes the transport, never the discipline.
- **Chat / no infra shell**: operate the gated copy-paste model - prepare
  labeled blocks, the operator runs them and pastes output back. Never assume
  a block ran or succeeded; wait for the pasted evidence.

Read `references/operating-discipline.md` before doing either.

## The three hard operating rules (non-negotiable)

1. **Execute only the current runbook step, exactly as written.** No added
   scope, no adjacent improvements, no live re-architecture mid-step. Findings
   and improvement ideas are LOGGED (changelog / D-NNN proposal), never
   executed live mid-step.
2. **Never use an inferred value.** No IP, ID, name, or scope goes into a
   command unless it was measured this session or carried from confirmed
   as-built. If a value would be inferred: stop and measure it. Never run a
   destructive or session-altering command from memory without confirming it
   is the minimal correct action for the current live state.
3. **Prefer dynamic lookups over hardcoded literals.** Discover VIPs, project
   names, IDs, and version sets at runtime. Where a literal is unavoidable it
   is tagged and centralized (`scripts/lib-net.sh`, `lib-hosts.sh`), keyed by
   stable identity (CIDR, hostname) - never by drifting IDs.

Corollary that governs everything: **verify before mutate**. A read-only audit
precedes every mutation; destructive and secret-handling steps are gated
individually, never batched.

## Routing - where to go for what

| Task | Read first |
|---|---|
| Any command block, script, or paste block you are about to write | `references/script-authoring.md` |
| Deploy / redeploy / teardown (VR0, single-DC testcloud) | repo `runbooks/README.md`, then the phase-NN runbook; conventions in `references/operating-discipline.md` |
| VR1 DC-DC buildout (two-DC + Office1 headend -- the current live mission) | repo `docs/CURRENT-STATE.md` FIRST (stage/gate STATUS lives there only), then `docs/dc-dc-deployment-workflow.md` (stage identity, links, gap register), then `runbooks/README.md`'s "VR1 DC-DC track" section for the `dc-dc-phase0..6` runbooks + `dc-dc-teardown-rollback.md`. SEPARATE track from VR0's phase-NN above -- do not conflate stage numbers between the two. |
| **ANY teardown, destroy, release or rebuild -- model, controller, or substrate** | **`docs/tool-index.md` FIRST**, then `runbooks/dc-dc-teardown-rollback.md` (Path M = juju model, Path C = juju controller, Paths A/B = tofu substrate) and **`scripts/phase-00-teardown-{release,destroy}.sh`** for the VR0 D-061 pair. **A bare `juju destroy-model` is NOT a validated form here** -- D-061 records `--force --no-wait` with no storage flag as a command that caused damage 3x, and both tested scripts always pass `--destroy-storage` or `--release-storage`. |
| **"What tool already exists for this job?" -- ask BEFORE typing any operational command** | **`docs/tool-index.md`** (93 scripts / 28 runbooks / 98 harnesses; CLAUDE.md hard rule 4). Typing a raw `juju`/`maas`/`tofu`/`virsh` command is the ERROR SIGNAL that the lookup was skipped. |
| Something is broken (triage, incidents) | `references/troubleshooting.md`, then repo `runbooks/appendix-A-troubleshooting.md` |
| CAPI / Magnum / mgmt-VM recovery | repo `runbooks/ops-capi-recovery.md` |
| Deliver ANY repo change (script, runbook, doc) | run `bash scripts/repo-lint.sh` + the touched script's `tests/<name>/run-tests.sh` BEFORE handing it over |
| Pre-deploy gate (before add-model) | `bash scripts/preflight.sh` -- THE single entry; do not run the sub-gates piecemeal |
| Is the cloud actually healthy? (post-deploy, post-restart, pre-change baseline, incident) | `bash scripts/cloud-assert.sh` (add `--capture` at deploy completion for the committed BOM) |
| Full-cloud restart after outage/maintenance | repo `runbooks/ops-restart-procedure.md` |
| VR1 site unreachable after a vcloud reboot (`ssh voffice1` / `office1-*` time out) | `bash scripts/site-baseleg.sh check office1` -- `office1-local` is an ISOLATED libvirt net, so the host's base L3 leg (`10.10.0.10/24` + compose route `10.10.1.0/24 via 10.10.0.20`) is dropped every reboot; the `After=libvirtd` unit re-adds it (D-126 amendment / D-128). If MISS and the unit is not yet installed: `sudo bash scripts/site-baseleg.sh apply office1`. This is the vcloud path; the workstation reaches the same over the tailnet (D-107). |
| Starting any consequential live session | `bash scripts/run-logged.sh <label>` first (as-executed log; docs/as-executed-log-convention.md) |
| Octavia amphora PKI -- is it healthy, and how do I ROTATE the controller cert? | `bash scripts/octavia-pki.sh verify <site>` on **voffice1** (the PKI lives on the headend, D-109 ruling note (b) -- it REFUSES on the jumphost rather than describing the wrong machine). To rotate the LEAF: `reissue <site>` + `runbooks/phase-01-bundle-deploy.md` Step **1.0-REISSUE** -- NEVER 1.0-GEN, whose refuse-if-present escape moves the CAs and destroys the amphora trust domain. **A `PASS` verdict alone is not acceptance**: while `os-public-hostname` is unset (B5 IP-only) A12/A13 RECORD rather than fail, so grep for the literal line `DNS SANs are all in this DC's expected zone`. Run `creds-matrix.py --tier2` as well -- the two gates check different things and have disagreed in BOTH directions |
| Credential exposures / security TODOs | repo `docs/security-ledger.md` -- add a row, never only a script comment. The credential-LIFECYCLE policy (mint -> consolidate -> audit) is **D-137**, not the ledger (SEC-009's convention block is now a pointer); the expected-state register is `creds-matrix.tsv` + `scripts/creds-matrix.py` (tier 1 static, tier 2 existence, tier 3 validity), enforced as a BLOCKING preflight `P5`. `creds-audit` CLEAN is NOT evidence of completeness -- it is declaration-based |
| Per-DC artifact delivery (apt mirror / caching proxy); "can the nodes get packages?" | dc0 = `bash scripts/dc-mirror.sh check dc0` (full mirror); dc1 = `bash scripts/dc-cache-proxy.sh check dc1` (caching proxy). The two DCs DIFFER BY RULING (D-135 amendments) -- checking dc1 as a mirror fails by design. Both run ON the rack host over `ssh -J voffice1` with `sudo bash -s`; take rack transit IPs from `scripts/lib-hosts.sh`, never memory |
| Resuming work / what is still in flight (esp. after compaction) | repo `docs/CURRENT-STATE.md` first, then `docs/session-ledger.md` + `bash scripts/ledger-scan.sh` to reconcile against repo ground truth |
| Tenant onboarding / tenant self-service | repo `scripts/tenant-onboard.sh` + `runbooks/tenant-onboarding-v2-DRAFT.md` + `appendix-C/D` |
| Network / plane / IPAM questions | `references/environment.md`, repo `scripts/lib-net.sh`, NetBox (the IPAM apex -- but during VR1 the WORKING apex is `office1-netbox`, 10.10.1.10, per DOCFIX-195; `netbox.baldurkeep.com` is a READ-ONLY v1 reference, write-back deferred to end-of-deployment) |
| ANY change request to a built surface | grep repo `docs/design-decisions.md` for the governing D-NNN FIRST - PROPOSED/OPEN means the operator has not ruled: present options, do not implement |
| Why is it built this way? / proposing changes | repo `docs/design-decisions.md` (D-NNN); grep before assigning a new number |
| Versions, channels, pins | repo `docs/CURRENT-STATE.md` section 7 (VR1 substrate -- measured pin authority); `runbooks/appendix-B-asbuilt-version-lock.md` (VR0 charm channels) |
| Environment facts (hosts, repo, planes) | `references/environment.md` |
| OpenTofu / libvirt / MAAS / OPNsense IaC work (repo `opentofu/`) | `references/opentofu-provider-docs.md` FIRST (sources + fetch method), then repo `opentofu/README.md` (per-module ground truth, schema notes, audit findings -- but its OPNsense section is SUPERSEDED by D-112/D-113) |
| Creating/changing a `libvirt_domain`; anything about nested KVM, a libvirt storage pool, LXD versions, an OPNsense edge, or a VM that boots wrong | `references/platform-traps.md` -- per-tool traps + version pins + a verbatim-error index (KiB-vs-MiB memory, `acpi=off`, missing `cpu` block = no `svm`, "in-place" bounces the guest, AppArmor on non-default pools, MAAS-vs-LXD 6.7, lxdbr0's dnsmasq, tcsh on the edge) |

## Standard loops (repeatable session shapes)

**Session bootstrap (jumphost):** Claude Code sessions launch from the repo
root, always (GA-R7 rule 6 -- a session launched from a subdirectory gets its
own empty project dir and a divergent auto-memory; the GA-F06 origin).
`git -C "$REPO" pull` (set `$REPO` once per
session -- see `runbooks/README.md` Conventions / `references/environment.md`) ->
`bash scripts/repo-lint.sh` (0 fail expected) -> if touching the live cloud,
`bash scripts/run-logged.sh <label>` to open the logged shell. Repo HEAD and a
clean lint are the preconditions for everything else.

**VR1 operating model (D-128 -- invariant; do not re-derive this each session):**
Claude runs ON the vcloud jumphost -- do NOT relocate the session to a workstation.
Work splits in two, origins FIXED BY DESIGN: **Plane 1** (substrate -- the OUTER
`tofu` root `qemu:///system` + local `virsh`, creating the DC *host* VMs) runs on
vcloud and has no Roosevelt analog; **Plane 2** (MAAS / NetBox / the INNER `tofu`
root `qemu+ssh`-from-Office1 / tenant tests) EXECUTES on `voffice1` via `ssh
voffice1 '...'` so runbooks read "from the Office1 headend" and transfer. Claude's
host only picks the TRANSPORT, not where work runs.
**>>> D-138 AMENDS PLANE 2, AND THIS IS THE MOST MISREAD FACT IN THE MODEL. <<<**
Plane 2 is now SPLIT by what a tool dials. **Anything that dials the CLOUD at L3 --
the `juju` client and `phase-03..06`'s `openstack` CLI -- runs FROM INSIDE THE DC (its
rack), NOT on `voffice1`.** `voffice1` keeps everything application-proxied or
`qemu+ssh`-mediated: the MAAS CLI, NetBox, the inner tofu roots. The reason is
structural, not preference: there is NO L3 path from `voffice1` to any DC node plane
(SEC-010 drops forwarding across the transit leg; the plane nets are isolated libvirt
nets), which is what cost three failed bootstrap attempts on 2026-07-30 before D-138
was ruled. **MEASURED 2026-08-02, the unambiguous test: `juju controllers` on
`voffice1` -> `ERROR No controllers registered`; the same command on the dc0 rack ->
the live `vr1-dc0-controller`.** A session that takes the un-amended reading runs
`juju` against nothing and reads the failure as a network fault. **Practical trap that
follows: the racks have NO repo clone**, so a script sourcing `lib-net.sh` relative to
its own path cannot be piped through `bash -s` -- use the staged copies in
`~/repo-stage/scripts/` and sha256-verify them against the repo before trusting them. Durable Office1 reach = the
D-126 base-leg oneshot (`scripts/site-baseleg.sh check office1` FIRST if `ssh
voffice1` is dead post-reboot) -- NOT tailscale-on-vcloud (D-107 is the operator's
WORKSTATION path / human GUI, left untouched). DCs sit outside the `/22` carve and
keep the D-126 `-J`-through-`vvr1-dc0` shape. **Durable host->site reach is standing
per-SITE tooling, DCs included (so the gap does not reopen per-DC):** `site-baseleg.sh`
is site-keyed. At each DC standup, MEASURE how vcloud reaches that DC's containment/
service net -- if it is an ISOLATED host-local net, add a MEASURED leg row + `install`
the unit as part of standup definition-of-done (NEVER an inferred DC IP -- the harness
fails on an un-commented DC supernet); a routed/NAT net or an already-reachable
`qemu+ssh` path needs none. Register item 20 tracks this. **Rack-side twin invariant
(D-131 sub-1, RULED 2026-07-21): every internet-isolated DC rack gets the repo-carried
`scripts/dc-rack-net.sh install <site>` at standup (reboot-persistent rack bridge legs
+ the node-DNS dnsmasq forwarder aliased on metal-admin, forwarding to region BIND over
the rack's own transit) -- MAAS 3.7's rack-only agent resolver SERVFAILs on isolated
racks, so WITHOUT this, commissioning silently times out (appendix-A carries the
symptom pair).** Two measured MAAS traps with standing consequences: MAAS virsh PODS
are refuted for this repo's DCs (per-machine `power_type=virsh` via
`scripts/maas-node-power.sh` is the ruled shape -- D-103/D-123 amendments 2026-07-20),
and a pod delete CASCADES to its linked machine records -- read the pod's machine list
BEFORE any `vm-host delete`; non-empty = STOP (appendix-A, 2026-07-21 incident).
**Per-DC power credential (SEC-012 dc0, SEC-016 dc1, RULED 2026-07-23 -- standing
DC-standup invariant): each DC gets its OWN dedicated MAAS->libvirt power key, NEVER
a cross-DC reuse.** The REGION's MAAS snap dials the power address, so the split is:
the maas-node-power SCRIPT's mapping virsh uses the DC SERVICE key (via the region's
`~/.ssh/config` Host <rack-transit-ip>), while MAAS's own power ops use the dedicated
key installed in the snap (`/var/snap/maas/current/root/.ssh/` + a per-host `ssh
config` block so each rack uses its own key). FRAGILITY: the snap-side key lives under
per-revision `/var/snap/maas/current/` and may not survive a snap refresh -- re-assert
after any refresh. Roosevelt analog: per-DC IPMI/BMC credentials.

**Per-DC ARTIFACT DELIVERY is a per-DC STRATEGY, and the two DCs deliberately differ
(D-135 + its 2026-07-24/27 amendments -- standing invariant).** dc0 tests a FULL MIRROR
(`dc-mirror.sh`, debmirror+nginx on the D-134 utility `.4`); dc1 tests an APT CACHING
PROXY (`dc-cache-proxy.sh`, apt-cacher-ng on `.4:3142`, consumed via
`juju apt-http-proxy`). This is the EXPERIMENT, not an interim state -- there is no
fallback and no later convergence, and dc1's partial mirror was REMOVED 2026-07-27.
Consequences that bite: **checking dc1 as a mirror FAILS BY DESIGN** (use its own
checker), and each DC's gate is its own script's `check <site>`.
**Corollary invariant, learned the hard way: a per-DC utility service OWNS ITS OWN
NETWORK PREREQUISITES.** `dc-cache-proxy.sh` originally REUSED `dc-mirror.sh`'s
`<site>-mirror-net.service` for the `.4` alias + edge default route, so on a proxy-only
DC three files named `*-mirror-*` were load-bearing FOR THE PROXY: the documented
mirror-removal procedure would have silently killed it, and the proxy could not be
REBUILT afterward without reinstalling the whole mirror apparatus (enabled daily
debmirror timer included). Never let one strategy's teardown be able to break the
other's; name artifacts for the service that owns them. Roosevelt analog: a DC must be
able to change artifact strategy without rebuilding the other strategy's apparatus.

**Stage 4 hands off READY nodes, NOT deployed ones (DOCFIX-200, RULED 2026-07-23 --
standing).** MAAS-deploy is SKIPPED; Juju provisions at Stage 5. Any surface saying
"nodes Deployed" as a Stage-4 exit condition is a DEFECT, not a fact -- four were found
and fixed on 2026-07-27 (workflow-doc gate cell, phase-3 DoD + Step 7, phase-4
prerequisites). Practical consequence: **nothing node-side can be probed in Stage 4**
(the nodes are powered off), so node-side checks belong at Stage 5 first boot -- that is
why gate G17 exists.

**EVERY PER-DC SECRET NEEDS A ROTATION TOOL, NOT JUST A GENERATION RECIPE (RULED 2026-07-30,
operator: "Full script minter now" -- standing DC-standup invariant; `runbooks/phase-01-bundle-deploy.md`
Step 1.0-REISSUE is the worked example, `scripts/octavia-pki.sh reissue` the tool).** A generation
step guarded by REFUSE-IF-PRESENT -- which is correct, since regenerating invalidates everything
issued -- leaves NO sanctioned way to rotate a leaf, so its only documented escape ("move the
existing material aside") moves the CAs and destroys the trust domain. **The dangerous path
becomes the only path.** At every DC standup, whatever mints a per-DC secret must ship with a
narrow rotation counterpart that touches the leaf and NOT the root. Four properties that made
this one trustworthy, all measured rather than assumed: (i) **DERIVE per-DC identity from the
site token, never type it** -- `${SITE%%-*}`/`${SITE#*-}` gave region and DC label, and a
transposition is caught by a per-DC harness case rather than by review; (ii) **assert the
ARTIFACT, never the config** -- `-extfile` is what decides an issued extension set, so a config
that says the right thing proves nothing (a cert with NO keyUsage/EKU minted, promoted and left
the harness fully green until an assertion read the cert); (iii) **stage, assert, then promote**
-- the hand-pasted original overwrote the key BEFORE signing with no `set -e`, so any mid-way
failure left a bundle whose cert and key did not match, which nothing detected; (iv) **join the
workspace to the deploy input** -- every assertion graded the workspace while the OVERLAY is
what reaches the charm, and an overlay carrying a different bundle read PASS. Roosevelt analog:
per-DC IPMI/BMC and PKI credentials all need a rotation path before first deploy, not after the
first expiry.

**A HAND-PASTED `openssl` CHAIN IS A DEFECT CLASS, AND THIS REPO HAS THE RECEIPTS.** Two
independent failures came out of ONE runbook block: F8 (an INDENTED heredoc terminator silently
yielded a certificate with NO SANs while every command printed OK) and F9 (CN and both DNS SANs
were baked literals that no gate read, wrong on both DCs for weeks). Build the config with
`printf` one line per statement, passing values as `%s` ARGUMENTS -- a heredoc's correctness
depends on invisible leading whitespace on a line nobody reads. **And the harness is a gate too:
prove each new assertion can FAIL.** A mutation pass on 2026-07-30 deleted three freshly-written
assertions one at a time and the suite stayed green every time; they were decoration until each
got a failing-direction fixture. The two properties with no assertion at all were the ones that
would have hurt most -- a missing `-days 730` (openssl defaults to **30 days**) and a dropped
EKU (one-directional mTLS failure at LB-create time, every gate reading PASS).

**>>> THE MAAS REGION IS INSTALLED DIRECTLY IN THE DC AT STANDUP. IT IS NEVER MIGRATED. THIS
IS THE STANDING PATH FOR dc0, dc1 AND EVERY FUTURE DC. <<<** (Operator correction 2026-08-02,
verbatim: *"MAAS region gets installed directly on the DC, it is not migrated. If you have read
this somewhere we need to make sure it is corrected so future deployments don't follow a
regional migration path."*) D-132 q1 ruled a region controller PER DC; at a DC standup you
STAND ONE UP THERE and the nodes enlist into it from the start. **There is no migration step in
the workflow, and a session that invents one is following a path this project does not have.**

**THE dc0 MIGRATION OF 2026-07-30 WAS A ONE-TIME HISTORICAL REMEDIATION, NOT A PROCEDURE.** It
existed for exactly one reason that cannot recur: dc0's machines had ALREADY been enrolled in
the Office1 region before D-132 q1 was ruled, so those existing records had to be moved. **Any
DC stood up after that ruling never enters this state.** The block below is retained ONLY as
(a) the record of what was done to dc0 and (b) a source of MAAS behaviours worth knowing --
NOT as a workflow to follow. **Do not cite it as a standup step. Do not plan a migration for a
new DC.** If you find yourself reading it while planning DC3/4/5, you are in the wrong place:
the standup path is a region installed in the DC.

Retained record -- the dc0 migration's ordering, which was not the obvious one, and four of
whose steps had no repo tool: MAAS cannot move machines between regions, so the carve was
RECREATED and the machines RE-ENROLLED. Hard-won shape: **(i) prove WHICH REGION a profile resolves to before every command** --
`scripts/maas-profile-assert.sh` asserts by RACK-CONTROLLER IDENTITY, because a machine COUNT
is not proof and against the wrong profile an apply is an idempotent no-op that prints PASS
while `machine delete` destroys the real record. Repo-wide `MAAS_PROFILE` defaults to `admin`
= the OFFICE1 region. **(ii) Numeric MAAS ids do NOT cross regions** -- VLAN row 5005 is dc0
metal-admin in one region and `vr1-dc0-data-tenant` in the other; the same host is `tw7ptw`
there and `c3aqh8` here. Stable keys are fabric NAME, space NAME, subnet CIDR, tag NAME, BOOT
MAC, interface NAME. A pre-migration capture is a REBUILD REFERENCE, never a diff target.
**(iii) Per-node ordering: `delete` (old region) -> `virsh start` BY HAND (nothing else can
start a machine the new region has never seen) -> self-enlist -> SET POWER -> `abort` ->
`commission`. NEVER `commission` a machine whose `power_type` is unset** -- it passes every
script and drops back to `New` with zero events. **(iv) Enlist in BATCHES OF THREE**; nine at
once produced 15 minutes of silence and zero records while egress, the MAAS proxy, region load
and node liveness all measured healthy. **(v) A fresh region has ZERO SSH KEYS** -- MAAS
injects them into deployed machines and `juju bootstrap` needs SSH, so this reproduces the
"cannot SSH the machine" bootstrap failure exactly; import before deploying. **(vi) The node
carve is ROLE-DEPENDENT** -- role nodes take all six planes plus an OVS `br-ex` parented on
`enp2s0` (the static lands on br-ex, the NIC holds no L3 link); the juju controller takes only
metal-admin + provider-public raw WITH the gateway, no br-ex, `enp3-6` left on auto VLANs.
`lib-hosts.sh` carries `NIC_PLANE_ORDER`, which is **NOT** `lib-net.sh`'s `PLANE_CIDRS` order:
walking PLANE_CIDRS positionally swaps `enp1s0`/`enp2s0` and strands PXE on provider-public.
**(vii) `maas-role-tags.sh` PASSES while the `openstack-<site>` PLACEMENT tag is still absent**
-- it is not in its ROLES set and has no creator in the repo, yet `bundle.yaml` places on it
and `dc-node-v6-carve.py` dies without it. **(viii) `phase-00-maas-standup.sh` must NOT be
pointed at a new region** -- unlike `carve-host-interfaces.sh` and `reenroll-hosts.sh`, which
REFUSE `vr1-*` outright, it does not refuse and would create space-named fabrics and a VID-103
metal-internal that D-133 abolished. Tools built for this: `maas-profile-assert.sh`,
`maas-region-power-key.sh`, `dc-region-topology.sh`; `dc-plane-ipam.sh` and `maas-role-tags.sh`
already worked once given `MAAS_PROFILE`.

**A FINDING IS AN OBSERVATION, NOT A CONCLUSION -- MEASURE BEFORE YOU PUT IT TO THE OPERATOR
(2026-07-27 Stage-5 grounding audit; the second-most productive move after the one below).**
A seven-lens read-only committee produced findings that were reliable AS OBSERVATIONS and
repeatedly wrong AS CONCLUSIONS. Measured before presenting, FOUR of the first six questions
changed shape and three had to be WITHDRAWN or re-scoped outright: the "unassigned" IPv6
literals were already assigned and apex-recorded (D-111); the lib-net consumer problem had ONE
failure mode rather than two and a Stage-5 blast radius of two scripts rather than twenty; the
credential register already ATTRIBUTED and EXPLAINED the findings a question proposed to
suppress, and its notes file already said "do not delete the row to make the checker green";
the Ceph OSD fix was 8 volumes not 18; and the MTU work was two lagging segments, not a
project. **Cost of the discipline: minutes per question. Cost of skipping it: the operator
rules on a false premise.** Corollary: when your own measurement DISAGREES with a prior
finding, check your instrument first -- `awk -F'\t'` on a space-aligned `.tsv` returns zero
rows and looks exactly like a real result.

**A CITATION IS AN EXISTENCE CLAIM; ONLY ITS CONTENT IS EVIDENCE (same audit).** A careful,
well-written in-repo comment block argued a design position on two Launchpad numbers. Read in
full: one was a DIFFERENT installer's bug, Fix Released five years earlier; the other was not
about the claimed subject at all, and its actual root cause was fixed in a package version
this deployment already installs. The conclusion INVERTED -- what the comment called the risky
option was the vendor's own default. **No gate reads prose, so nothing in a repo can catch
this.** When a decision rests on an external citation, open the citation, check its STATUS and
DATES, and weigh them against the versions actually deployed. Vendor documentation and the
upstream bug tracker beat a repo comment every time. The operator's formulation is the one to
remember: *"Research is cheap, guesswork is expensive."*

**RULED IS NOT BUILT -- CHECK THE ARTIFACT, NOT THE DECISION (same audit).** Two decisions
that had been properly RULED were found never IMPLEMENTED: D-134's per-DC address bands
existed only as a table in prose (MAAS held ZERO reserved ranges), and D-020's enumeration of
vault among the apps carrying VIPs had never produced a vault VIP. Both had passed every gate
for weeks because nothing compared the decision to the artifact. Before treating a ruling as
in effect, grep the thing it was supposed to change.

**A CHECKER THAT CANNOT FAIL IS NOT A GATE (this project's most productive audit move).**
GA-R6 lets a stage close only on a named executable check, so the check itself must be
verified capable of failing. Two measured instances, days apart: `creds-audit` is
DECLARATION-based and read CLEAN on three sites while the D-137 matrix returned 13
findings; `dc-mirror.sh check` asserted that `last-sync.status` EXISTED and printed its
contents behind an unconditional `OK`, so a `FAIL` status and a four-day-stale `RUNNING`
both passed. Working rules that came out of it: assert on CONTENT, never on existence;
an unrecognised state REFUSES rather than defaulting to success; "could not look" is
never "nothing there"; and **enumerate what EXISTS rather than only auditing what is
declared** -- a register cannot report the absence of something it has no row for, which
is how an unregistered credential at both DC edges survived every audit. When a test
asserts a literal finding string, remediating the finding turns the harness red: REPLACE
the assertion with the new invariant and say so, never delete it to go green.

**Change-delivery loop:** grep for prior art (zeroth decision) -> grep
design-decisions for the governing D-NNN -> edit -> `bash scripts/repo-lint.sh`
-> run the touched script's harness (create one if missing -- no script change
ships without its harness) -> deliver with an entry in the SESSION changelog
(ONE changelog per session, GA-R2/D1 -- never per-topic files) carrying a
per-item revert. Under blanket approval, the changelog IS the review surface:
every item states what, why (evidence), and how to revert. Changelogs are
session-scoped scratch: consolidated per-stage at stage close, NEVER citable
as status or decision authority.

**Deploy loop (VR0, single-DC testcloud):** phase-00 runbook (D-061 destroy
path) -> `bash scripts/preflight.sh` PASS -> phase-01..08 gated -> `bash
scripts/cloud-assert.sh --capture` -> commit the asbuilt/ BOM.

**Deploy loop (VR1 DC-DC, the current live mission):** read
`docs/dc-dc-deployment-workflow.md`'s tooling gap register FIRST -- check
whether the stage you're about to run has any open blocking item -> Stage 1
(`dc-dc-phase0-vcloud-prep.md`) gated -> Stage 2..7 gated in order, per
`runbooks/README.md`'s "VR1 DC-DC track" table -> if a stage's `tofu apply`
fails partway, read `runbooks/dc-dc-teardown-rollback.md`'s rollback decision
tree BEFORE reaching for a teardown (fix-forward is usually correct). Run
`bash scripts/opentofu-validate.sh` before any tofu step; whether a given
root/step has been exercised for real is a STATUS question -- check
CURRENT-STATE, never this file's memory of it. **At STAGE
CLOSE** (GA-R6): a stage or verification-type gate CLOSES only on a NAMED executable
check whose captured output the closing commit cites -- prose cannot close a stage,
and there is NO conditional close (a remainder splits into its own gate row or the
stage stays open, E3). Ruling-type gates close per GA-R5 (recorded, pushed ruling).
The close-out set is executable: gauntlet ALL GREEN, repo-lint 0-fail, the stage's
GA-R2 consolidation commit landed, the GA-R7 memory review run, MERGE the stage
branch to `main` (merge commit, NOT squash; operator-gated), then RETIRE the branch
(local + remote). **ALSO sweep THIS skill in the same close:** fold in any new
INVARIANT the stage established and reconcile any divergence from repo HEAD (facts
delegate to the repo and go stale; invariants live here and must be kept current). The
AUTHORITATIVE, versioned skill is THIS dir (`.claude/skills/openstack-cloud-ops/` -- SKILL.md
+ references); keep it current here. THEN REGENERATE the dated single-file Chat-upload
snapshot from it (concatenate SKILL.md + references -> `.claude/skills/openstack-cloud-ops-consolidated-<date>.md`,
ASCII/LF byte-verified) so the Chat/project skill surface can be refreshed -- the snapshot
is a DERIVED artifact, never the source, and the Chat mount going stale is a known failure
mode (it caused a failed clone + a mis-shaped handoff, 2026-07-25; docs/audit/skill-divergence-20260725.md).
The next stage branches off post-merge `main`. `main` is
trunk-of-record only for stages that have merged (precedents: Stage 2 -> PR #1 `e468d2e`
2026-07-15; Stage 3 -> merge commit 2026-07-21, vr1-dc0 scope, dc1 HELD under G12). See
`docs/dc-dc-deployment-workflow.md` "Cross-cutting discipline".

**Incident loop:** capture the verbatim error -> `bash scripts/cloud-assert.sh`
(the service-own-verdict sweep localizes the layer) -> appendix-A by exact
message -> recorded fix, gated -> log the finding (new root causes become
appendix-A/DOCFIX material).

## Record authority and session discipline (GA-R1..R7, ratified 2026-07-18)

Ratified text: repo `docs/audit/ga-rulings.md` -- this is the operating summary.

- **`docs/CURRENT-STATE.md` is the ONLY status authority (GA-R1).** Stage,
  gates, plan counts, version pins, decision-queue state live THERE and nowhere
  else; other surfaces point or are history. Any commit that changes a status it
  carries updates it in the SAME commit (C1; repo-lint L10 enforces). It is
  authoritative over documents, never over measurement -- a fresh capture wins
  and corrects it in the capture's commit (C2).
- **Plan counts and versions are quoted only from captured command output** --
  a dated capture file or a quoted command + output; never from memory or
  another document's prose (GA-R1 rule 2). Legal stage/gate status tokens are
  OPEN / BLOCKED / HELD / CLOSED, used only in CURRENT-STATE (vocabulary ruling,
  Option A: readiness is not a state; event words are history, never states).
- **Rulings are gated like mutations (GA-R5).** ONE decision per operator
  exchange -- batch adoptions are invalid. A ruling exists only when its Status
  line quotes the question AND the operator's exact utterance, dated, committed
  and pushed before dependent work. Ambiguous/template answers rule NOTHING:
  stop and ask. Never adopt from an inferred or reconstructed ruling -- an
  operator ruling is a value under hard rule 2.
- **D-number admission (GA-R3):** a new D-number needs architectural consequence
  beyond the stage, a Roosevelt-delta (the A1 test: a Roosevelt build session
  would grep it before touching a built surface), or supersession. Everything
  else is OPERATIONAL: runbook edit / session-changelog line / as-built row,
  never a D-number. Entries carry [ARCH]/[OPS] tags; doubt resolves DOWN to OPS.
- **Memory governance (GA-R7):** auto-memory records observations and
  conventions ONLY -- never operator policy, priorities, or postures. Any
  memory claiming operator intent is PROPOSED until confirmed verbatim; remove
  on discovery. Durable observations GRADUATE to their repo surface; memory is
  never the only copy. Stage close includes a full memory review (rules 1-4).
- **Reconciliation circuit-breaker.** If you have been fixing records to match
  records for more than one cycle without touching the actual deliverable,
  STOP and report to the operator. Record-churn presenting as progress is this
  project's measured failure mode (GA-F06); escalating is the correct move.

## Posture

- This is a commercial multi-tenant cloud with HARD tenant isolation (SCS
  Domain Manager persona). Treat tenant-visible surfaces and cross-domain
  boundaries as security-relevant in every change.
- The operator community here values debate and industry best practice over
  quick fixes. Push back with sources when you disagree; own mistakes plainly
  and concisely. Fabricated flags, values, or version numbers are the cardinal
  sin - if you have not verified an option name or version, say so and verify.
- Responses stay concise. Decisions get explicit rationale.
