diff --git a/.claude/skills/openstack-cloud-ops/SKILL.md b/.claude/skills/openstack-cloud-ops/SKILL.md index 98a7f74..12ae95d 100644 --- a/.claude/skills/openstack-cloud-ops/SKILL.md +++ b/.claude/skills/openstack-cloud-ops/SKILL.md @@ -85,6 +85,8 @@ | Any command block, script, or paste block you are about to write | `references/script-authoring.md` | | Deploy / redeploy / teardown (VR0, single-DC testcloud) | repo `runbooks/README.md`, then the phase-NN runbook; conventions in `references/operating-discipline.md` | | VR1 DC-DC buildout (two-DC + Office1 headend -- the current live mission) | repo `docs/CURRENT-STATE.md` FIRST (stage/gate STATUS lives there only), then `docs/dc-dc-deployment-workflow.md` (stage identity, links, gap register), then `runbooks/README.md`'s "VR1 DC-DC track" section for the `dc-dc-phase0..6` runbooks + `dc-dc-teardown-rollback.md`. SEPARATE track from VR0's phase-NN above -- do not conflate stage numbers between the two. | +| **ANY teardown, destroy, release or rebuild -- model, controller, or substrate** | **`docs/tool-index.md` FIRST**, then `runbooks/dc-dc-teardown-rollback.md` (Path M = juju model, Path C = juju controller, Paths A/B = tofu substrate) and **`scripts/phase-00-teardown-{release,destroy}.sh`** for the VR0 D-061 pair. **A bare `juju destroy-model` is NOT a validated form here** -- D-061 records `--force --no-wait` with no storage flag as a command that caused damage 3x, and both tested scripts always pass `--destroy-storage` or `--release-storage`. | +| **"What tool already exists for this job?" -- ask BEFORE typing any operational command** | **`docs/tool-index.md`** (93 scripts / 28 runbooks / 98 harnesses; CLAUDE.md hard rule 4). Typing a raw `juju`/`maas`/`tofu`/`virsh` command is the ERROR SIGNAL that the lookup was skipped. | | Something is broken (triage, incidents) | `references/troubleshooting.md`, then repo `runbooks/appendix-A-troubleshooting.md` | | CAPI / Magnum / mgmt-VM recovery | repo `runbooks/ops-capi-recovery.md` | | Deliver ANY repo change (script, runbook, doc) | run `bash scripts/repo-lint.sh` + the touched script's `tests//run-tests.sh` BEFORE handing it over | diff --git a/CLAUDE.md b/CLAUDE.md index 84c58c6..3c7342c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -19,6 +19,17 @@ justified as the minimal correct action, and individually human-approved (the permission `ask` rules enforce this -- do not work around them). Destructive steps are never batched. +4. **USE THE TESTED ARTIFACT. Look it up BEFORE you type a command.** This repo + exists to make deployment and troubleshooting error-free WITHOUT a build: + 93 scripts, 28 runbooks, 98 harnesses, each written because an incident + proved it was needed. **`docs/tool-index.md` maps the job to the artifact -- + read it first.** Typing a raw `juju` / `maas` / `tofu` / `virsh` command for + an operation is the ERROR SIGNAL, not the starting point: it means the lookup + was skipped. Deriving an invocation from `--help` or from a prior session's + transcript when a tested script is on disk is the same error. If no artifact + exists, SAY SO and propose building one -- a gap is a finding, never a licence + to improvise. (Added 2026-08-03 after an ad-hoc `juju destroy-model` was run + instead of the D-061 teardown scripts and took the dc0 controller down.) ## Session bootstrap (run at the start of every session) diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 28a232f..5cdbdd7 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2013,6 +2013,63 @@ at `life: dead`** -- that moved it BACK to `dying` and re-armed the undertaker loop. It did not help and it changed state. Both are now written into Path M as explicit prohibitions. + **>>> CAUSATION SETTLED 2026-08-03, AND THE HEADLINE IS PROCEDURAL, NOT A FLAG: I RAN AN + AD-HOC COMMAND INSTEAD OF THE TESTED D-061 TEARDOWN SCRIPTS THIS REPO BUILT FOR EXACTLY + THIS OPERATION. <<<** + **THE DIFFERENCE FROM EVERY VALIDATED TEARDOWN IN THIS REPO, measured:** both tested + scripts ALWAYS pass a storage flag -- `scripts/phase-00-teardown-destroy.sh:116` uses + `--destroy-storage --force --no-wait --no-prompt`, and + `scripts/phase-00-teardown-release.sh:178` uses `--release-storage --force --no-wait + --no-prompt`. **I ran `juju destroy-model vr1-dc0 --force --no-wait --no-prompt` -- with + NO storage flag at all** -- and I removed that flag DELIBERATELY, having measured `juju + storage` as empty and reasoned from `juju help` that it was a no-op. **D-061's problem + statement names my exact form as a known failure:** the damage occurred "with + `--destroy-storage --force`, and AGAIN with `--force --no-wait` and NO storage flag at + all", observed 3x in one session. **I cannot prove the missing flag is the MECHANISM behind + the orphaned status doc, and I will not claim it** -- but it is the concrete delta, and the + procedural miss is the finding regardless of mechanism. + **THE SCRIPTS DO FAR MORE THAN THE ONE COMMAND.** The release path runs `juju + remove-machine --keep-instance --force --no-wait` PER HOST first, HARD-VERIFIES each + survived in MAAS, only THEN destroys the model, then sweeps orphans -- a five-stage guarded + procedure with a harness at `tests/phase-00-teardown-d061`. I ran one bare command. I had + READ the D-061 relationship section in `dc-dc-teardown-rollback.md` earlier the same + session, correctly concluded the pod-decompose hazard does not transfer to VR1 (it does + not -- nothing decomposed), and then let "that hazard does not apply" become "the scripted + procedure does not apply", which does not follow. + **RECORDED SEPARATELY BECAUSE IT IS THE PATTERN, NOT THE INCIDENT:** this was the third + time in one session I bypassed an existing tested path -- the `/tmp/ucacfg` config + injection instead of the per-DC overlay, documenting `kill-controller` from its `--help` + instead of a run, and this. Remediation: `CLAUDE.md` hard rule 4 and `docs/tool-index.md`. + **WHAT WAS RULED OUT, and it matters because the alternative hypothesis was reasonable.** + Operator asked directly whether the bad + document came from the juju command or from the 08-01 containment-VM resize that + hard-cut every inner guest (the same event that destroyed the dc0 edge via UFS + soft-update damage). + **(1) THE OBJECT DID NOT EXIST AT THE TIME OF THE RESIZE.** The reboot was + **2026-08-01 05:48**, and the controller VM's `mongod` start time is `Sat Aug 1 + 05:49:59` -- one minute later, matching the bounce exactly. The model whose status doc is + missing (`001f6745-ca54-4fae-84f6-73bbfe787f0f`) was created by `juju add-model vr1-dc0` + at **Step 3.5 on 2026-08-02**, a full day AFTER. A power cut cannot corrupt a document + belonging to an object that did not yet exist. + **(2) THE OTHER MODEL ON THE SAME DISK IS INTACT -- this is the decisive one.** The + crash-loop log shows `allwatcher loaded for model "ce31f9bb-..."` SUCCEEDING on every + iteration while `001f6745` fails on the missing status doc. Same mongo, same disk, same + 08-01 hard cut. **Storage damage has no mechanism to spare the controller model's status + doc hundreds of times and remove only the status doc of the model that was later + force-destroyed.** + **(3) THE DB SERVED A FULL DEPLOY HOURS BEFORE THE FAILURE.** `mongod` has been UP + CONTINUOUSLY since 08-01 05:49:59 with no restart (it survived the 08-03 01:08:59 jujud + restart untouched), and in that window it served the entire bundle deploy -- 56 + applications, 36 machines/containers, spaces and model-config -- with no error. A + storage-corrupted state DB does not do that cleanly and then fail on exactly one document. + **MECHANISM: removing the status doc is part of what `destroy-model` DOES.** The defect is + that the model document was left behind after its status document was removed -- a + teardown-ordering failure, which is precisely why upstream documents `--force` as a LAST + RESORT that can leave state juju cannot reconcile. **NOT RULED OUT, and stated rather than + hidden: my re-issue of `destroy-model` against an already-`dead` model may have contributed + to which documents survived.** Either way the agent is the juju destroy path, not the + memory change. **CONSEQUENCE FOR THE 08-01 INCIDENT RECORD: its blast radius stays what it + was measured to be -- the two OPNsense edges -- and does NOT extend to the juju controller.** **PATH M IS AMENDED** (`runbooks/dc-dc-teardown-rollback.md` M.2): try WITHOUT `--force` when agents are alive; reserve `--force --no-wait` for the measured stall condition; never re-issue destroy against a `dead` model; and if the controller is already down on this diff --git a/docs/repo-rebuild-project-brief.md b/docs/repo-rebuild-project-brief.md new file mode 100644 index 0000000..bc552df --- /dev/null +++ b/docs/repo-rebuild-project-brief.md @@ -0,0 +1,316 @@ +# Prompt for a separate Chat / Chat+Code project: OMEGA CLOUD REPO REBUILD + +Authored 2026-08-03 at operator direction, from inside the failure it describes. +Hand the block below to Chat as the opening prompt. Attach the inputs listed in section 9. + +--- + +## BEGIN PROMPT + +You are leading a **repository reconstruction project** for a live, commercial, multi-tenant +Charmed OpenStack deployment ("Omega Cloud"). You will NOT touch the running cloud. Your +product is a rebuilt information architecture: documents, decision records, and a tool +registry that an operator or an AI agent can act from **without research, without +archaeology, and without guessing.** + +### 1. What this repo is, and why it looks the way it does + +The cloud is built as staged virtual rehearsals (VR0 single-DC, VR1 two-DC + headend) for a +future bare-metal multi-datacenter build ("Roosevelt"). The governing constraint is +MINIMIZE DELTA TO ROOSEVELT: the runbooks and scripts are primary deliverables alongside the +running cloud. + +**It was deliberately built one hardened layer at a time, and that is not a mistake to +undo.** The team intentionally proved each layer before adding the next. Example, in the +operator's words: *"we used IPv4 to build out the initial deployment steps and yaml so we +could confirm software functionality. After the software package had been verified we +replaced IPv4 with an IPv6 and dual stack configuration so we are only troubleshooting +IPv6/dual stack problems rather than trying to determine if the problem is networking or +software."* That sequencing is sound engineering and produced a genuinely hardened base. + +**The side effect is the problem you are solving: information and tool SPREAD.** Every +hardening pass left its predecessor in place -- superseded rulings, amended amendments, +annotations on annotations, prose written for a layer that has since been replaced, and +tooling written for one generation sitting beside tooling for the next with nothing +distinguishing them. + +### 2. The failure mode, with evidence -- this is what you must design against + +The repo is 880 tracked files: **93 scripts, 28 runbooks, 98 test harnesses**, plus decision, +status, and security registers. The discipline is unusually strong (every claim is supposed +to cite measured evidence). **The defect is not sloppiness -- it is retrieval.** Correct +information is present and cannot be found or trusted at the moment of action. + +Concrete instances, all from a single recent week, each of which cost real time or caused a +real fault: + +1. **A status file that cannot be read.** The single status authority is **7,357 lines / + ~647 KB**. It is simultaneously current state and full history. Nothing that large can be + read before acting, so agents grep it and get whichever fragment matches first -- often a + superseded one. +2. **Stale prose beside a live artifact.** A deploy runbook instructed the operator to omit a + required overlay "because the file does not exist." It had existed for weeks and carried + 16 applications' worth of load-bearing config. Following the runbook would have produced a + silently broken deploy (dead provider egress + 15 unreachable package sources). +3. **A measurement generalised past its validity.** A ruling recorded "the signing key is + already on the nodes" -- true when measured, on nodes whose state a *previous* deploy's + configuration had created. After that configuration changed, it was false. Nothing marked + the claim as time-bound. It blocked a live deploy. +4. **Tested tools that could not be found.** A hardened, harnessed teardown script pair + existed (written after an incident, precisely so it would never recur). Nothing routed to + them by task; they appeared only inside a decision record's prose. An ad-hoc command was + run instead and **took a controller down.** There is no tool index anywhere. +5. **A one-time remediation recorded as a standing procedure.** The always-loaded operating + skill carried "per-DC MAAS region MIGRATION" as an invariant. It was a one-off fix for one + datacenter's history; the standing path is a region installed directly. A future build + would have followed a migration path that does not exist. +6. **Superseded decisions still readable as current.** A runbook said the bundle ships "NO + designate" per decision D-019. D-019 had been superseded by D-106 (reactivating it) and + the software had shipped in the bundle for weeks. This was raised to the operator as an + open question -- pure wasted cycles on a settled matter. +7. **Gates asserting things their own command cannot print.** A quality gate required + assertions on `juju deploy --dry-run` output. Measured: that command prints no application + options at any verbosity. Multiple gate items could never have failed. + +**The pattern:** claims are recorded with evidence but WITHOUT SCOPE -- no statement of what +generation, layer, or moment they are true for, and no forward link to what superseded them. +Retrieval therefore returns the first plausible match, not the current truth. + +**The cost, in the operator's words:** *"We have spent almost an entire week working on this +deployment and it has come to a painful crawl because of this issue."* + +### 3. What you must produce + +1. **A CURRENT-TRUTH LAYER that is small enough to read in full before acting.** Hard size + budget. Status only -- no history, no narrative, no rationale. Every line carries: the + claim, its evidence pointer, the date measured, and the generation/layer it applies to. +2. **A RESOLVED DECISION VIEW.** Every decision reduced to its CURRENT effective ruling, with + superseded text moved behind it, not inline. Answer in one lookup: "what governs X today?" + Preserve every original utterance verbatim -- see constraint (b). +3. **A TOOL & HARNESS REGISTRY**, task-first ("I want to do X -> this tested artifact"), not an + alphabetical listing. Must state which generation each tool is valid for and whether it is + harnessed. **This is the single highest-value deliverable** -- its absence caused the + controller outage. +4. **A HISTORY ARCHIVE.** Everything removed from (1) and (2), losslessly, addressable, clearly + marked as history so it can never be mistaken for current. +5. **A PROVENANCE / STALENESS CONVENTION.** A required form for time-bound claims so instance + (3) above cannot recur: what was measured, when, under what configuration, and what would + invalidate it. +6. **A MIGRATION PLAN** from the current repo to the new structure, executable incrementally, + with a verification gate per step. The live deployment continues during this work. + +### 3b. THE CENTRAL DISTINCTION THE REBUILD MUST ENCODE: BUILDOUT vs EXECUTION + +**Operator, verbatim, and this is the thesis of the whole project:** + +> *"The tasks we are doing are repeatable runs that require the use of proven and supplied +> data and code. The buildout of that data and code is what requires troubleshooting and +> changes while we find the more efficient and hardened steps that can be scripted and locked +> in place for future use without requiring the future operator to engineer or troubleshoot +> any step."* + +There are **TWO MODES**, and the repo currently conflates them in the same files: + +| | BUILDOUT mode | EXECUTION mode | +|---|---|---| +| What it is | discovering the hardened step | running the proven step | +| Expected activity | troubleshooting, research, measurement, revision | none -- follow the artifact | +| Correct output | a locked, scripted, harnessed artifact + its ruling | a completed run and its captured evidence | +| Who reads it | the engineer hardening the layer | the future operator, who must NOT have to engineer | + +**Every surface in the rebuilt repo must declare which mode it serves.** A future operator in +EXECUTION mode must never have to read BUILDOUT narrative to find the command; today they +must, and that is the whole failure. The buildout record is kept as evidence (it is why the +system is trustworthy) but is architecturally separated so it can never be mistaken for the +instruction. + +**Design test for every proposed structure:** can a competent operator who was not present +for the buildout execute a full DC standup end to end, correctly, without engineering or +troubleshooting a single step? If not, the structure is not done. + +### 3c. REQUIRED: rebuild the Claude Code operating layer for Opus 5 + +The repo is operated through Claude Code. **That operating layer is in scope and must be +rebuilt, not carried over.** Anthropic's Opus 5 guidance explicitly calls for revisiting +`CLAUDE.md`, skills, and harness scaffolding rather than inheriting prior-model prompts, and +Opus 5 is the right pivot point to re-found how Code is used here around documentation-based +infrastructure practice. + +**Deliverable 7: a rebuilt Claude Code operating layer** -- +`CLAUDE.md`, the operating skill(s), hooks, permission/gating rules, and the governance rules +(the GA-R ruleset), each rebuilt from current Anthropic documentation rather than accreted. + +**Authoritative sources to work from (read them; do not work from summaries):** +- `platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5` +- `platform.claude.com/docs/en/about-claude/models/whats-new-opus-5` +- the migration guide section on moving from Opus 4.8 to Opus 5 +- in-product: `/claude-api migrate` (reviews prompt diffs) and `/doctor` (rightsizes + `CLAUDE.md` and skills) + +**Specific guidance that applies to this repo, and what it means here:** + +1. **>>> WRITTEN-DELIVERABLE LENGTH -- THIS IS ACTIVELY COMPOUNDING THE PROBLEM YOU ARE + SOLVING. <<<** Anthropic: *"files that Claude Opus 5 writes to disk (reports, Markdown + documents, summaries) are often longer than on prior models"* and recommend explicit + length calibration. **MEASURED HERE: the status authority grew 6,938 -> 7,357 lines during + a SINGLE session, most of it agent-authored.** The information spread this project exists + to fix is being fed by the default authoring behaviour. The rebuilt operating layer MUST + carry explicit length calibration and per-surface size budgets, or the new structure + re-bloats within weeks. +2. **Over-verification instructions should be REMOVED -- but SEPARATE TWO KINDS, and get this + wrong and you cause an outage.** Anthropic: remove explicit verification instructions + ("include a final verification step", "use a subagent to verify") because Opus 5 verifies + its own work and these cause waste. **THAT TARGETS PROMPT SCAFFOLDING. IT DOES NOT TARGET + DOMAIN SAFETY RULES.** "Verify before mutate", "never use an inferred value", "destructive + steps are never batched" and the individual-approval gating are not model scaffolding -- + they are operational controls on a LIVE COMMERCIAL CLOUD, each written after a real + incident. **Classify every existing instruction as SCAFFOLDING (remove) or CONTROL (keep, + and justify by naming the incident it came from).** Deleting a control because a general + prompting guide said "remove verification instructions" would be a serious error. +3. **Scope constraint should be explicit.** Opus 5 can expand task scope by its own judgment. + This repo's hard rule 1 (execute only the current step; findings are logged, never executed + mid-step) is exactly the recommended control and should be kept and sharpened. +4. **Subagent delegation should be capped deliberately.** Opus 5 delegates readily; delegation + pays on large independent tracks and wastes on small ones. Set explicit policy. +5. **Effort levels should be re-swept.** Defaults carried from a prior model are stale; + `low`/`medium` are viable primary cost controls where quality holds. +6. **Correction narration should be bounded** to corrections that change the operator's + decisions. + +**Also rebuild the GOVERNANCE ruleset** (record authority, ruling procedure, memory +governance, stage-close requirements). It is sound in intent and has itself accreted; restate +it minimally, resolve internal conflicts, and keep every ruling's verbatim utterance. + +### 4. Non-negotiable constraints + +a. **You do not touch the running cloud.** No infrastructure commands. Documents only. +b. **Operator rulings are sacred and verbatim.** Every ruling has a recorded exact operator + utterance with a date. These are never paraphrased, never compressed, never "cleaned up." + Restructure the surface AROUND them. +c. **No lossy compression of any record.** This repo has an explicit prohibition on + automated prose-compression tooling because it destroys verbatim records. Moving text is + allowed; rewriting or summarising a record in place is not. +d. **Nothing is deleted, only relocated** -- with a forwarding pointer from the old location. +e. **Every claim in your output cites `path:line`** of its source. A claim you cannot cite is + a finding, not a conclusion. +f. **A finding is an observation, not a conclusion.** Verify before presenting. This project's + own hardest-won lesson: measure before you put something to the operator. +g. **Preserve the layered-hardening history as EVIDENCE.** It is why the deployment is solid. + The goal is to make the CURRENT layer instantly findable, not to erase the earlier ones. + +### 5. Project structure -- four phases, each gated + +- **Phase 0 -- CARTOGRAPHY (read-only, no judgment).** Inventory every file: type, size, + purpose, last-modified, whether it makes status claims, decision claims, or both. Produce + the measured map. **No recommendations in this phase** -- its output must be reusable by + agents who disagree with each other. +- **Phase 1 -- CLASSIFICATION & CONFLICT DISCOVERY.** Every status claim, decision, and tool + classified by generation/layer and by current-vs-superseded. Produce the CONFLICT REGISTER: + every place two surfaces disagree, with both citations. Expect this to be large; it is the + core dataset. +- **Phase 2 -- ARCHITECTURE DESIGN.** Propose the target structure. Multiple independent + proposals, scored, then synthesised -- do not converge early on one design. +- **Phase 3 -- MIGRATION PLAN.** Incremental, verifiable, reversible steps. Each step names its + own check. Operator gates each batch. + +**Do not begin a phase until the prior phase's review has passed.** + +### 6. Agent roster -- distinct roles, deliberately non-overlapping + +Producers (each owns one lens; none may grade their own work): + +1. **CARTOGRAPHER** -- inventory and measurement only. Forbidden from interpreting. +2. **DECISION ARCHAEOLOGIST** -- reconstructs every decision's supersession chain to its + current effective ruling. Owns deliverable (2). +3. **STATUS/HISTORY SEPARATOR** -- splits the status authority into current-truth vs history. + Owns deliverables (1) and (4). +4. **TOOL REGISTRAR** -- inventories scripts/runbooks/harnesses by TASK; flags generation + validity and harness coverage; flags tools whose safety rails no-op outside their + generation. Owns deliverable (3). +5. **PROVENANCE AUDITOR** -- samples claims and traces each to its evidence; identifies + time-bound claims presented as invariant. Owns deliverable (5). +6. **ROOSEVELT-TRANSFER REVIEWER** -- judges every proposal against "does this transfer to the + real build?" This project exists to serve that, and it is the tiebreaker on design + disputes. +6b. **CLAUDE CODE HARNESS ARCHITECT** -- owns deliverable 7. Rebuilds `CLAUDE.md`, the + operating skill, hooks, permission rules and the governance ruleset from current Anthropic + Opus 5 documentation. **Its hardest and most consequential task is the SCAFFOLDING-vs-CONTROL + classification in 3c(2)** -- it must justify every instruction it REMOVES, and every control + it KEEPS must name the incident that produced it. Its output is reviewed by the + ROOSEVELT-TRANSFER REVIEWER (does the discipline survive contact with a real build?) and by + the ADVERSARIAL VERIFIER (what breaks if this instruction is gone?). +6c. **EXECUTION-MODE SIMULATOR** -- role-plays the future operator from section 3b: given only + the rebuilt EXECUTION surfaces, attempts a full DC standup end to end and reports every + point where it had to engineer, troubleshoot, search, or ask. **Each such point is a + defect in the structure, not in the simulator.** This is the acceptance instrument for the + central distinction. + +Reviewers (independent; must not have produced what they review): + +7. **ADVERSARIAL VERIFIER** -- mandate is to REFUTE, not confirm. For each finding: attempt to + show it wrong, and default to "refuted" when uncertain. Every producer output passes + through this before the operator sees it. +8. **COMPLETENESS CRITIC** -- asks only "what is missing?" -- which surface was not read, which + claim not verified, which generation not represented. Its output becomes the next round of + work. +9. **SYNTHESIST / ARCHITECT** -- reconciles reviewed outputs into one coherent design. Owns + Phase 2 synthesis. May not also be a producer. +10. **OPERATOR LIAISON** -- formats every decision needing a human ruling as a single question + with concrete options and a recommendation, and records the answer verbatim. **One + decision per exchange -- batched rulings are invalid.** + +### 7. Review protocol + +- Every producer output -> ADVERSARIAL VERIFIER -> COMPLETENESS CRITIC -> operator. +- Findings that survive refutation are marked CONFIRMED; those that do not are struck **in + place with the reason**, never deleted. +- Any disagreement between two producers goes to ROOSEVELT-TRANSFER REVIEWER, not to whoever + argues hardest. +- **A conclusion that changed after measurement must be reported as changed.** In the prior + audit of this repo, four of the first six findings changed shape once measured and three had + to be withdrawn. Expect the same rate; that is the process working. +- If two consecutive rounds produce only record-fixing-record churn with no deliverable + progress, STOP and escalate. Record-churn presenting as progress is this project's measured + failure mode. + +### 8. Definition of done + +- A cold agent, given only the current-truth layer plus the tool registry, can correctly + answer: what stage is this, what governs decision X today, and what tested tool performs + operation Y -- **in under five lookups and with zero archaeology.** +- Re-run the seven evidence instances in section 2 against the new structure. **Each must now + resolve correctly and quickly.** That is the acceptance test; it is concrete, and it is + drawn from real failures rather than invented. +- Every operator utterance still present verbatim, addressable, and attributed. + +### 9. Inputs to attach + +- The repository (or a filtered export: `docs/`, `runbooks/`, `scripts/` headers, and the + operating-skill directory). +- The consolidated single-file operating-skill snapshot, which is the intended Chat-upload + artifact. +- The status authority, the decision register, and the security register -- the three largest + and most conflicted surfaces. +- This brief. + +**Begin with Phase 0. Produce the measured map. Do not propose a structure yet.** + +## END PROMPT + +--- + +## Notes for the operator (not part of the prompt) + +- **Scope honesty.** Part of this week's cost was my own method -- reaching for derivation + instead of the repo's tested artifacts, three times in one session, twice after being told + directly. The structural fix above is necessary; it is not sufficient on its own, and it + should not be sold as the whole cause. +- **Sequencing.** Phase 0 and Phase 1 are pure read-only analysis and can run in parallel with + the live deployment. Phase 3's migration is the only part that competes for attention. +- **The highest-value single deliverable is the tool registry.** If the project stalls, that + one artifact still repays it -- it is the piece whose absence produced a downed controller. +- **Already built here, and worth carrying in as prior art rather than rediscovering:** + `docs/tool-index.md` (a first-cut task-first tool index, including the finding that the + D-061 teardown pair's safety rail silently no-ops outside VR0), `CLAUDE.md` hard rule 4 + (look up the tested artifact before typing a command), and this session's teardown-runbook + Paths M and C. diff --git a/docs/tool-index.md b/docs/tool-index.md new file mode 100644 index 0000000..9578089 --- /dev/null +++ b/docs/tool-index.md @@ -0,0 +1,162 @@ +# Tool index -- WHAT IS ALREADY BUILT AND TESTED FOR THE JOB YOU ARE ABOUT TO DO + +**READ THIS BEFORE TYPING AN OPERATIONAL COMMAND.** Created 2026-08-03 on operator +direction after a session in which an ad-hoc `juju destroy-model` was run instead of the +D-061 teardown scripts, took the dc0 Juju controller down, and cost hours of research the +repo had already done and saved. + +**Operator, verbatim, and it is the charter of this file:** + +> *"The teardown scripts and write up after the incident that preceded D061 was documented +> and artifacts were deliberately created because of the findings and conclusions from that +> incident. We have created and saved these commands and artifacts to allow them to be +> reused in the future, like right now, when problems come up and a tested path would save +> time, troubleshooting, and research. The entire point of the repo is to provide a fully +> tested set of instructions, tested code, and troubleshooting tools. This is supposed to +> allow for a fully error free deployment of services. This is supposed to allow for +> troubleshooting and systems management with documentation and tools that do not require a +> build."* + +**THE RULE THIS FILE EXISTS TO ENFORCE:** an operational action resolves to an EXISTING +tested artifact FIRST. **Typing a raw `juju` / `maas` / `tofu` / `virsh` command for an +operation is the ERROR SIGNAL, not the starting point** -- it means the lookup was skipped. +If no artifact exists, say so explicitly and propose building one; do not silently improvise +and do not derive a command from `--help` when a tested invocation is on disk. + +**Scale of what is here, measured 2026-08-03:** 93 scripts, 28 runbooks, **98 test +harnesses**, 65 scripts with their own `tests//` harness. This is not a small library +you can hold in your head. GREP IT. + +**Fastest lookups:** +```bash +ls scripts/ | grep -i # is there a tool? +ls tests/ | grep -i # is it harnessed? +grep -rn "" runbooks/ docs/design-decisions.md # is there a ruled procedure? +``` + +--- + +## TEARDOWN -- the gap that caused this file + +| I want to... | USE THIS | Never | +|---|---|---| +| Tear down the juju MODEL (VR0/openstack) | **`scripts/phase-00-teardown-destroy.sh`** (full redeploy spine) or **`scripts/phase-00-teardown-release.sh`** (KEEP machines) -- D-061, harness `tests/phase-00-teardown-d061` | a bare `juju destroy-model` | +| Tear down the juju MODEL (VR1 per-DC) | `runbooks/dc-dc-teardown-rollback.md` **Path M**, which now carries the measured `--force` hazard | a bare `juju destroy-model` | +| Tear down / rebuild a juju CONTROLLER | `runbooks/dc-dc-teardown-rollback.md` **Path C** (C.2b is the API-down path) | `kill-controller` alone -- MEASURED not to work when the API is down | +| Tear down the OpenTofu SUBSTRATE | `runbooks/dc-dc-teardown-rollback.md` Paths A / B | ad-hoc `tofu destroy` | +| Reset MAAS after a teardown | `runbooks/phase-00-teardown-maas-reset.md` | -- | + +### >>> AGNOSTICISM REVIEW OF THE D-061 PAIR, 2026-08-03 -- THEY ARE VR0-ONLY AS WRITTEN <<< + +**Operator standard, verbatim:** *"When we use a script for a task we need to make sure that +it is agnostic as possible since these are supposed to be tools that can be ran on any +deployment without rewrites."* This is SKILL.md hard rule 3 (dynamic lookups over hardcoded +literals; unavoidable literals tagged and CENTRALISED in `lib-net.sh` / `lib-hosts.sh`, keyed +by stable identity). Reviewed against it, the D-061 pair is **HALF agnostic**: + +- **Parameterised, good:** `MAAS_PROFILE="${MAAS_PROFILE:-admin}"`, + `MODEL="${OPENSTACK_MODEL:-openstack}"`. +- **Hardcoded VR0 literals, NOT overridable and NOT in `lib-hosts.sh`:** + `HOSTS=(openstack0 openstack1 openstack2 openstack3)`, `ORPHANS=(capi-mgmt)`, + `PROTECTED=(juju lxd tailscale)`, and `--canary` pins `openstack0`. The header claims the + roster is *"resolved live from `maas admin machines read`"* -- **it is not**; only the + machines JSON is live, the roster itself is a literal array. + +**THE DEFECT THIS CREATES, and it is a safety defect, not a portability nicety.** A host that +does not resolve is a `note`, not a `fail`: +`if [ -z "$s" ]; then note "$h: already absent from MAAS"; continue; fi`. +`FATAL` stays 0, the safety gate passes, **and the script proceeds to +`destroy-model --destroy-storage --force --no-wait --no-prompt` anyway.** Pointed at VR1 +(`OPENSTACK_MODEL=vr1-dc0 MAAS_PROFILE=dc0`) every one of the four VR0 hostnames resolves to +nothing, so **the entire host-safety audit silently no-ops while the destructive step still +fires** -- and "already absent from MAAS" reads like a reassuring, normal line. This is the +checker-that-cannot-fail class located in the safety rail itself. + +**CONSEQUENCE FOR USE TODAY: the D-061 pair is the tested path for VR0 and MUST NOT be +pointed at a VR1 DC until the roster is derived.** VR1 model/controller teardown goes through +`dc-dc-teardown-rollback.md` Paths M / C. + +**FIX WHEN IT IS THE CURRENT STEP (logged, not executed -- hard rule 1):** derive the roster +from STABLE IDENTITY rather than hostnames -- VR1 nodes carry `openstack-` and role +tags, and MAAS auto-generated hostnames do not match ruled names anyway -- or site-key it +through `lib-hosts.sh`; and **make a roster that resolves to ZERO machines a FATAL, not a +note.** A teardown whose host audit found nothing has not proven safety, it has failed to +look. + +**D-061 is the governing decision and it names the failure mode by flag combination:** the +damage occurred *"with `--destroy-storage --force`, and AGAIN with `--force --no-wait` and +NO storage flag at all"*. **Both tested scripts ALWAYS pass a storage flag** -- +`--destroy-storage` on the destroy path, `--release-storage` on the release path. A +`destroy-model` with neither is not a form this repo has ever validated. + +## GATES -- run these instead of inventing a check + +| Question | Tested artifact | +|---|---| +| Is it safe to deploy? | `scripts/preflight.sh` -- THE single entry, do not run sub-gates piecemeal | +| Is the cloud healthy? | `scripts/cloud-assert.sh` (`--capture` for the BOM) | +| Does the DC egress work? | `scripts/dc-egress-check.sh` (layered; first failure is the cause) | +| Are the bundle invariants intact? | `scripts/provider-bundle-check.py` | +| Are the charm channels pinned? | `scripts/channel_assert.py` | +| Are the juju spaces right? | `scripts/juju-spaces-check.sh` | +| Is the repo deliverable-clean? | `scripts/repo-lint.sh` + `scripts/run-tests-all.sh` | +| Is OpenTofu valid? | `scripts/opentofu-validate.sh` | + +## PER-DC SERVICES -- each owns its own network prerequisites + +| Service | Tool | Note | +|---|---|---| +| apt caching proxy | `scripts/dc-cache-proxy.sh ` | site token is bare `dc0`/`dc1` | +| apt full mirror | `scripts/dc-mirror.sh` | historical arm, D-135 amendment (b); `install` is a strategy REVERSAL | +| snap forward proxy | `scripts/dc-snap-proxy.sh` | consume via juju `snap-http-proxy`/`snap-https-proxy` | +| rack network layer | `scripts/dc-rack-net.sh install ` | reboot-persistent legs + node DNS | +| host->site base leg | `scripts/site-baseleg.sh check\|apply ` | first thing after a vcloud reboot | + +## MAAS + +| Job | Tool | +|---|---| +| Prove WHICH REGION a profile resolves to | `scripts/maas-profile-assert.sh` -- by rack identity; a machine count is NOT proof | +| Per-machine power config | `scripts/maas-node-power.sh` (pods are REFUTED for VR1) | +| Role / controller tags | `scripts/maas-role-tags.sh check ` | +| Region power key | `scripts/maas-region-power-key.sh` | +| Node interface carve | `scripts/carve-host-interfaces.sh`, `scripts/dc-node-carve.sh` | +| IPv6 plane carve / verify | `scripts/dc-node-v6-carve.py`, `scripts/dc-node-v6-verify.sh` | +| Plane IPAM incl. ULA retirement | `scripts/dc-plane-ipam.sh` | +| Region topology | `scripts/dc-region-topology.sh` | + +## CREDENTIALS / SECURITY + +| Job | Tool | +|---|---| +| Expected-state credential register | `scripts/creds-matrix.py` (tiers 1-3; BLOCKING preflight P5) | +| Declared-credential audit | `scripts/creds-audit.sh` -- declaration-based, NOT evidence of completeness | +| Octavia amphora PKI verify / ROTATE | `scripts/octavia-pki.sh verify\|reissue ` -- runs on the headend | +| Vault KV health | `scripts/vault-kv-health.sh`, `scripts/vault-kv-inner-probe.sh` | + +## TENANTS / ACCEPTANCE + +`scripts/tenant-onboard.sh`, `tenant-offboard.sh`, `tenant-assert.sh`, +`tenant-acceptance.sh`; phase-03..07 scripts for the service walk. + +## SESSION / RECORD + +`scripts/run-logged.sh