diff --git a/.claude/skills/openstack-cloud-ops-consolidated-20260727.md b/.claude/skills/openstack-cloud-ops-consolidated-20260727.md index f0065e6..51beed7 100644 --- a/.claude/skills/openstack-cloud-ops-consolidated-20260727.md +++ b/.claude/skills/openstack-cloud-ops-consolidated-20260727.md @@ -1,19 +1,11 @@ - +# openstack-cloud-ops -- CONSOLIDATED SNAPSHOT (regenerated 2026-07-27, Stage-5 grounding audit) -# openstack-cloud-ops -- consolidated skill (Chat upload snapshot, 2026-07-27) +DERIVED ARTIFACT. Source of truth is .claude/skills/openstack-cloud-ops/ (SKILL.md + +references/). Regenerate from that directory; never edit this file directly. A stale +snapshot being uploaded to the Chat surface is a recorded failure mode -- +docs/audit/skill-divergence-20260725.md. -================================================================================ -## SKILL.md (main entry -- always in context) -================================================================================ +--- --- name: openstack-cloud-ops @@ -197,6 +189,39 @@ (the nodes are powered off), so node-side checks belong at Stage 5 first boot -- that is why gate G17 exists. +**A FINDING IS AN OBSERVATION, NOT A CONCLUSION -- MEASURE BEFORE YOU PUT IT TO THE OPERATOR +(2026-07-27 Stage-5 grounding audit; the second-most productive move after the one below).** +A seven-lens read-only committee produced findings that were reliable AS OBSERVATIONS and +repeatedly wrong AS CONCLUSIONS. Measured before presenting, FOUR of the first six questions +changed shape and three had to be WITHDRAWN or re-scoped outright: the "unassigned" IPv6 +literals were already assigned and apex-recorded (D-111); the lib-net consumer problem had ONE +failure mode rather than two and a Stage-5 blast radius of two scripts rather than twenty; the +credential register already ATTRIBUTED and EXPLAINED the findings a question proposed to +suppress, and its notes file already said "do not delete the row to make the checker green"; +the Ceph OSD fix was 8 volumes not 18; and the MTU work was two lagging segments, not a +project. **Cost of the discipline: minutes per question. Cost of skipping it: the operator +rules on a false premise.** Corollary: when your own measurement DISAGREES with a prior +finding, check your instrument first -- `awk -F'\t'` on a space-aligned `.tsv` returns zero +rows and looks exactly like a real result. + +**A CITATION IS AN EXISTENCE CLAIM; ONLY ITS CONTENT IS EVIDENCE (same audit).** A careful, +well-written in-repo comment block argued a design position on two Launchpad numbers. Read in +full: one was a DIFFERENT installer's bug, Fix Released five years earlier; the other was not +about the claimed subject at all, and its actual root cause was fixed in a package version +this deployment already installs. The conclusion INVERTED -- what the comment called the risky +option was the vendor's own default. **No gate reads prose, so nothing in a repo can catch +this.** When a decision rests on an external citation, open the citation, check its STATUS and +DATES, and weigh them against the versions actually deployed. Vendor documentation and the +upstream bug tracker beat a repo comment every time. The operator's formulation is the one to +remember: *"Research is cheap, guesswork is expensive."* + +**RULED IS NOT BUILT -- CHECK THE ARTIFACT, NOT THE DECISION (same audit).** Two decisions +that had been properly RULED were found never IMPLEMENTED: D-134's per-DC address bands +existed only as a table in prose (MAAS held ZERO reserved ranges), and D-020's enumeration of +vault among the apps carrying VIPs had never produced a vault VIP. Both had passed every gate +for weeks because nothing compared the decision to the artifact. Before treating a ruling as +in effect, grep the thing it was supposed to change. + **A CHECKER THAT CANNOT FAIL IS NOT A GATE (this project's most productive audit move).** GA-R6 lets a stage close only on a named executable check, so the check itself must be verified capable of failing. Two measured instances, days apart: `creds-audit` is @@ -308,10 +333,9 @@ sin - if you have not verified an option name or version, say so and verify. - Responses stay concise. Decisions get explicit rationale. +--- -================================================================================ -## references/operating-discipline.md (load on demand) -================================================================================ +# ===== references/operating-discipline.md ===== # Operating discipline @@ -469,10 +493,272 @@ and the guard kept. (Rotated from the session ledger's standing-lessons block, 2026-07-19.) +## The probe PATH determines the answer (added 2026-07-27) -================================================================================ -## references/environment.md (load on demand) -================================================================================ +"Could not look" is never "nothing there" -- and the reachability of a host depends on WHERE +you probe from. A clone check run as `ssh voffice1 'ssh office1-netbox ...'` returned +**unreachable**; the identical check run directly from vcloud returned a real **NO-CLONE**. +Recording the first as a negative finding would have been wrong. Before reporting an absence, +confirm you probed from a host that can actually see the target -- and say which host you +probed from. Same family as the 2026-07-26 `/root/maas-secrets` case, where an unprivileged +`[ -d ]` on a root-owned directory read as "does not exist" and was corrected to UNREADABLE. + + +--- + +# ===== references/script-authoring.md ===== + +# Script authoring - house style and hardening + +Every script and paste block in this project follows these rules. They are +not style preferences: each one encodes a failure that actually happened. +Read this in full before writing ANY bash or python for this cloud. + +## Zeroth decision: does it already exist? + +Before writing ANY operational script or check block, search the repo: +`grep -rli scripts/ runbooks/` - the deploy/verify surface is heavily +scripted and duplicating an existing script creates drift (e.g. the haproxy +backend sweep already exists as `scripts/phase-03-core-verify.sh` 3.1b). +Route to, extend, or fix the existing artifact; write new only when nothing +covers the need, and say which search came up empty. + +## First decision: what kind of block is this? + +The error-handling regime depends on execution mode. Choosing wrong is itself +a bug: + +1. **Executed script** (`bash script.sh`, own process): `set -uo pipefail` + at minimum; `set -e` acceptable and usually right, with the capture + caveats below. Exit codes are the interface. +2. **Interactive paste block** (operator pastes into their shell): NEVER a + bare `exit` (it kills their shell) and no `set -e` (it can kill their + shell's options or abort mid-paste). Wrap the whole block in a subshell + `( { ...; } )` so a stray exit is contained; signal failure by printing + `FAIL: ...` lines the operator can read. +3. **Verify/count-gate block** (greps that legitimately return zero matches): + run WITHOUT `set -e`, and end every count-grep with `|| true` - a zero + count is a valid answer, not an error (L1). `bash -n` cannot catch this; + it is behavior, not syntax. + +## The header contract (executed scripts) + +Every script opens with a comment block stating: path + argument synopsis; +what it does (one paragraph, referencing the D-NNN/DOCFIX it implements); +**whether it mutates anything** ("Mutates NOTHING" / what it changes and the +gate protecting it); usage line; exit-code contract; and "ASCII + LF". +House exit codes: `0` PROCEED, `1` HOLD (a gate failed), `2` precondition +missing (tool absent, wrong model, helper not found). Then: + + set -euo pipefail # or set -uo pipefail; see regime above + shopt -s inherit_errexit 2>/dev/null || true + IFS=$'\n\t' + +Source shared constants instead of restating them: +`. "$SCRIPT_DIR/lib-net.sh"` (planes, VIP bands, helpers) and `lib-hosts.sh` +(hostnames, octets, system_id resolution). If a value you need is not in a +lib, consider adding it there rather than inlining a literal. + +## Hardening rules (each one is a scar) + +**SIGPIPE races break guards in BOTH directions.** `cmd | grep -q X` under +pipefail: on match, grep exits, the producer takes SIGPIPE (141), the pipeline +reports failure despite the match. In an `... || die` verify this FALSE-DIES on +success; in an `... && die` guard it FAILS OPEN -- the 2026-07 sweep found a +duplicate-CIDR guard that let collisions through exactly this way. Treat every +`| grep -q` on a live pipe as a defect regardless of which way the test points. + +**Pipefail + SIGPIPE race.** `cmd | grep -q X` under pipefail falsely fails: +`grep -q` closes the pipe on first match, SIGPIPE (141) kills the producer +(`juju ssh` especially). Capture, then test: + + OUT=$(cmd 2>&1 || true) + grep -q "pattern" <<<"$OUT" + +**`set -e` kills id-captures silently.** `ID=$(openstack ... || die ...)` - +if the subshell exits non-zero before your handler fires, `set -e` aborts the +assignment line with no message. Append `|| true` to each capture, then +validate the captured value explicitly (see whole-output validation below). + +**Whole-output validation, never extract-then-check.** Do not pipe raw output +through `awk`/`grep` to extract a field and then test the fragment - a +partial failure yields a plausible-looking fragment. Capture the WHOLE output, +validate its shape (e.g. `is_id(){ [[ "$1" =~ ^[0-9a-f]{32}$ ]]; }` for a +keystone id), and only then use it. + +**Centralize `&1` once; +every call site then stays clean and un-forgettable. Heredoc-payload ssh +(`ssh ... bash -s <<'EOF'`) is the ONE exemption -- stdin IS the delivery there. + +**Inner stdin consumption.** Any `ssh`/`sudo`/`juju ssh` inside a heredoc, +pipe, or loop eats the remaining stdin and truncates the block. Append +`` - the `-m` flag goes BEFORE +the target. `juju run ` output: use `--format json` for +anything captured; confirm long actions via `juju show-operation `, not +the streamed log (a wait-timeout does not mean the hook failed). + +**Snap confinement.** The openstack CLI snap cannot read `/tmp` - stage files +under `$HOME`. Same for `juju attach-resource` payloads. + +**Client output ordering.** `openstack -f value -c X -c Y` returns columns in +ALPHABETICAL order, not flag order. When order matters: `-f json | jq`, or +single-column queries. After any jq returning null, run `jq 'keys'` - key +casing is command-specific (Title-Case in lists, hyphenated in quota show). + +**Environment isolation.** Any block that switches identity runs in a +subshell that first unsets all OS_* vars: +`( for v in $(env | awk -F= '/^OS_/{print $1}'); do unset "$v"; done; export ... )` +Thread `OS_CACERT` explicitly into isolated subshells - it gets stripped. +Confirm scope before acting: which project/domain the token holds, not which +you intended. + +**Stable keys, not drifting IDs.** Look up MAAS subnets by CIDR, machines by +hostname, CAPI CRs by LISTING then operating on the exact returned name +(the OpenStackCluster suffix is random per create - a wrong-name patch +silently no-ops). system_ids are re-minted on re-enrollment. + +**Verify the launched cmdline, not the config text.** For OpenStack debs run +via LSB-init-wrapped systemd: a flag "present in a file" proves nothing. +Gate on behavior - the init script's `show-args`, `ps -ww -C -o args` +on the live process, and probe under the daemon's RESTRICTED PATH +(`env -i PATH=/usr/sbin:/usr/bin:/sbin:/bin sh -c 'command -v helm'`) - an +interactive shell's PATH masks daemon-PATH failures. + +**sed is not a verifier.** A non-matching `sed -i` exits 0 having changed +nothing. Assert the post-edit content; never trust sed's exit code as proof +of the edit. + +**ASCII + LF, validated.** Non-ASCII: `grep -nP '[^\x00-\x7F]'` or a Python +byte read. CR check: Python `data.count(b"\x0d")` - `grep $'\r'` false- +positives on `$r...` tokens. Non-ASCII in conf.d has silently killed daemons. + +**Python helpers live in .py files** tested against fixtures - no inline +python-in-bash beyond one-liners. Do not assume `jq` exists off-jumphost; +gate on it (`need_jq`) or use python3. + +**Secrets in scripts:** whitelist-write to 0600 files under a 0700 dir +(`umask 077` first), never echo, never argv, measure secret length from the +file rather than asserting an expected length (this deployment's app-cred +secrets are 86 chars, not the commonly assumed 43 - never hardcode either). + +**Structured-output captures take stdout ONLY; never merge stderr.** `x=$(openstack ... -f json 2>&1)` +is a defect: any stderr line (a `--long` deprecation warning, an SDK notice) lands at char 0 and +kills the JSON parse -- this fail-closed a live SG audit in 2026-07. Capture stdout alone; send +stderr to a tempfile; surface it on parse failure, never discard it (`2>/dev/null` trades the bug +for silent error-masking). The `vr_json` helper in `scripts/lib-validate.sh` encodes this. Also +drop deprecated flags that only add stderr noise (`--long` is unneeded; fields are present without it). + +**Fixture-test throwaway probe one-liners like real code.** A measurement `ss`/`tcpdump`/`awk` +pipeline is code, and it has the same parse traps: `ss` DROPS the State column under a +`state established` filter (peer is then `$NF`, not `$5`); `cut -d:` splits IPv4-mapped IPv6 +(`[::ffff:10.0.0.1]:443`) at the wrong colon; `grep -oE 'src [0-9.]+'` greedily eats the trailing +`.` into the next field; charmed magnum logs to `/var/log/magnum/magnum-conductor.log`, NOT +journald. One measurement chain in 2026-07 shipped four such bugs before each was caught. Run the +parse against a fixture line (mapped-v6, plain-v4, the real column layout) BEFORE handing the +operator a probe -- the same discipline as a committed script. + +## Testing: nothing ships on `bash -n` + +`bash -n` validates parse, not behavior - and most of the failures above are +behavioral. Every nontrivial script gets an offline regression harness at +`tests//run-tests.sh` in the established pattern: + +- `fakebin/` contains fake `juju` / `openstack` / `maas` / `kubectl` / + `ssh` executables that replay fixture output and log the calls they + receive; real coreutils stay real. +- The harness sets `PATH="$BIN:$PATH"`, a scratch `HOME`, and env-injected + fixture paths, runs the target, and asserts BOTH the exit code and an + output regex per case: `run