Newer
Older
openstack-caracal-dc-dc / docs / tool-index.md

Tool index -- WHAT IS ALREADY BUILT AND TESTED FOR THE JOB YOU ARE ABOUT TO DO

READ THIS BEFORE TYPING AN OPERATIONAL COMMAND. Created 2026-08-03 on operator direction after a session in which an ad-hoc juju destroy-model was run instead of the D-061 teardown scripts, took the dc0 Juju controller down, and cost hours of research the repo had already done and saved.

Operator, verbatim, and it is the charter of this file:

"The teardown scripts and write up after the incident that preceded D061 was documented and artifacts were deliberately created because of the findings and conclusions from that incident. We have created and saved these commands and artifacts to allow them to be reused in the future, like right now, when problems come up and a tested path would save time, troubleshooting, and research. The entire point of the repo is to provide a fully tested set of instructions, tested code, and troubleshooting tools. This is supposed to allow for a fully error free deployment of services. This is supposed to allow for troubleshooting and systems management with documentation and tools that do not require a build."

THE RULE THIS FILE EXISTS TO ENFORCE: an operational action resolves to an EXISTING tested artifact FIRST. Typing a raw juju / maas / tofu / virsh command for an operation is the ERROR SIGNAL, not the starting point -- it means the lookup was skipped. If no artifact exists, say so explicitly and propose building one; do not silently improvise and do not derive a command from --help when a tested invocation is on disk.

Scale of what is here, measured 2026-08-03: 93 scripts, 28 runbooks, 98 test harnesses, 65 scripts with their own tests/<name>/ harness. This is not a small library you can hold in your head. GREP IT.

Fastest lookups:

ls scripts/ | grep -i <topic>          # is there a tool?
ls tests/   | grep -i <topic>          # is it harnessed?
grep -rn "<verb>" runbooks/ docs/design-decisions.md   # is there a ruled procedure?

TEARDOWN -- the gap that caused this file

I want to... USE THIS Never
Tear down the juju MODEL (VR0/openstack) scripts/phase-00-teardown-destroy.sh (full redeploy spine) or scripts/phase-00-teardown-release.sh (KEEP machines) -- D-061, harness tests/phase-00-teardown-d061 a bare juju destroy-model
Tear down the juju MODEL (VR1 per-DC) runbooks/dc-dc-teardown-rollback.md Path M, which now carries the measured --force hazard a bare juju destroy-model
Tear down / rebuild a juju CONTROLLER runbooks/dc-dc-teardown-rollback.md Path C (C.2b is the API-down path) kill-controller alone -- MEASURED not to work when the API is down
Tear down the OpenTofu SUBSTRATE runbooks/dc-dc-teardown-rollback.md Paths A / B ad-hoc tofu destroy
Reset MAAS after a teardown runbooks/phase-00-teardown-maas-reset.md --

>>> AGNOSTICISM REVIEW OF THE D-061 PAIR, 2026-08-03 -- THEY ARE VR0-ONLY AS WRITTEN <<<

Operator standard, verbatim: "When we use a script for a task we need to make sure that it is agnostic as possible since these are supposed to be tools that can be ran on any deployment without rewrites." This is SKILL.md hard rule 3 (dynamic lookups over hardcoded literals; unavoidable literals tagged and CENTRALISED in lib-net.sh / lib-hosts.sh, keyed by stable identity). Reviewed against it, the D-061 pair is HALF agnostic:

  • Parameterised, good: MAAS_PROFILE="${MAAS_PROFILE:-admin}", MODEL="${OPENSTACK_MODEL:-openstack}".
  • Hardcoded VR0 literals, NOT overridable and NOT in lib-hosts.sh: HOSTS=(openstack0 openstack1 openstack2 openstack3), ORPHANS=(capi-mgmt), PROTECTED=(juju lxd tailscale), and --canary pins openstack0. The header claims the roster is "resolved live from maas admin machines read" -- it is not; only the machines JSON is live, the roster itself is a literal array.

THE DEFECT THIS CREATES, and it is a safety defect, not a portability nicety. A host that does not resolve is a note, not a fail: if [ -z "$s" ]; then note "$h: already absent from MAAS"; continue; fi. FATAL stays 0, the safety gate passes, and the script proceeds to destroy-model --destroy-storage --force --no-wait --no-prompt anyway. Pointed at VR1 (OPENSTACK_MODEL=vr1-dc0 MAAS_PROFILE=dc0) every one of the four VR0 hostnames resolves to nothing, so the entire host-safety audit silently no-ops while the destructive step still fires -- and "already absent from MAAS" reads like a reassuring, normal line. This is the checker-that-cannot-fail class located in the safety rail itself.

CONSEQUENCE FOR USE TODAY: the D-061 pair is the tested path for VR0 and MUST NOT be pointed at a VR1 DC until the roster is derived. VR1 model/controller teardown goes through dc-dc-teardown-rollback.md Paths M / C.

FIX WHEN IT IS THE CURRENT STEP (logged, not executed -- hard rule 1): derive the roster from STABLE IDENTITY rather than hostnames -- VR1 nodes carry openstack-<site> and role tags, and MAAS auto-generated hostnames do not match ruled names anyway -- or site-key it through lib-hosts.sh; and make a roster that resolves to ZERO machines a FATAL, not a note. A teardown whose host audit found nothing has not proven safety, it has failed to look.

D-061 is the governing decision and it names the failure mode by flag combination: the damage occurred "with --destroy-storage --force, and AGAIN with --force --no-wait and NO storage flag at all". Both tested scripts ALWAYS pass a storage flag -- --destroy-storage on the destroy path, --release-storage on the release path. A destroy-model with neither is not a form this repo has ever validated.

GATES -- run these instead of inventing a check

Question Tested artifact
Is it safe to deploy? scripts/preflight.sh -- THE single entry, do not run sub-gates piecemeal
Is the cloud healthy? scripts/cloud-assert.sh (--capture for the BOM)
Does the DC egress work? scripts/dc-egress-check.sh (layered; first failure is the cause)
Are the bundle invariants intact? scripts/provider-bundle-check.py
Are the charm channels pinned? scripts/channel_assert.py
Are the juju spaces right? scripts/juju-spaces-check.sh
Is the repo deliverable-clean? scripts/repo-lint.sh + scripts/run-tests-all.sh
Is OpenTofu valid? scripts/opentofu-validate.sh

PER-DC SERVICES -- each owns its own network prerequisites

Service Tool Note
apt caching proxy `scripts/dc-cache-proxy.sh <check\ install\ node> <dc0\ dc1>` site token is bare dc0/dc1
apt full mirror scripts/dc-mirror.sh historical arm, D-135 amendment (b); install is a strategy REVERSAL
snap forward proxy scripts/dc-snap-proxy.sh consume via juju snap-http-proxy/snap-https-proxy
rack network layer scripts/dc-rack-net.sh install <site> reboot-persistent legs + node DNS
host->site base leg `scripts/site-baseleg.sh check\ apply ` first thing after a vcloud reboot

MAAS

Job Tool
Prove WHICH REGION a profile resolves to scripts/maas-profile-assert.sh -- by rack identity; a machine count is NOT proof
Per-machine power config scripts/maas-node-power.sh (pods are REFUTED for VR1)
Role / controller tags scripts/maas-role-tags.sh check <site>
Region power key scripts/maas-region-power-key.sh
Node interface carve scripts/carve-host-interfaces.sh, scripts/dc-node-carve.sh
IPv6 plane carve / verify scripts/dc-node-v6-carve.py, scripts/dc-node-v6-verify.sh
Plane IPAM incl. ULA retirement scripts/dc-plane-ipam.sh
Region topology scripts/dc-region-topology.sh

CREDENTIALS / SECURITY

Job Tool
Expected-state credential register scripts/creds-matrix.py (tiers 1-3; BLOCKING preflight P5)
Declared-credential audit scripts/creds-audit.sh -- declaration-based, NOT evidence of completeness
Octavia amphora PKI verify / ROTATE `scripts/octavia-pki.sh verify\ reissue ` -- runs on the headend
Vault KV health scripts/vault-kv-health.sh, scripts/vault-kv-inner-probe.sh

TENANTS / ACCEPTANCE

scripts/tenant-onboard.sh, tenant-offboard.sh, tenant-assert.sh, tenant-acceptance.sh; phase-03..07 scripts for the service walk.

SESSION / RECORD

scripts/run-logged.sh <label> (as-executed log), scripts/ledger-scan.sh (open-work ground truth), scripts/record-audit.py, scripts/cloud-snapshot.sh.


HOW THIS FILE STAYS TRUE

It is a POINTER index, deliberately not a description of behaviour -- behaviour lives in each script's header, which is the authority. When a tool is added or its job changes, add or fix the ROW. A row that names a script which no longer exists is a defect; regenerate the inventory with:

ls scripts/ ; ls -d tests/*/          # what exists, and what is harnessed

If you cannot find a tool for the job, that is a FINDING, not a licence to improvise. Say so, propose the tool, and get the operator's ruling -- the same standard every other gap in this repo is held to.