diff --git a/.claude/skills/openstack-cloud-ops/SKILL.md b/.claude/skills/openstack-cloud-ops/SKILL.md index 29fcf3f..f524cce 100644 --- a/.claude/skills/openstack-cloud-ops/SKILL.md +++ b/.claude/skills/openstack-cloud-ops/SKILL.md @@ -78,7 +78,8 @@ | Task | Read first | |---|---| | Any command block, script, or paste block you are about to write | `references/script-authoring.md` | -| Deploy / redeploy / teardown | repo `runbooks/README.md`, then the phase-NN runbook; conventions in `references/operating-discipline.md` | +| Deploy / redeploy / teardown (VR0, single-DC testcloud) | repo `runbooks/README.md`, then the phase-NN runbook; conventions in `references/operating-discipline.md` | +| VR1 DC-DC buildout (two-DC + Office1 headend -- the current live mission) | repo `docs/dc-dc-deployment-workflow.md` FIRST (stage status + tooling gap register), then `runbooks/README.md`'s "VR1 DC-DC track" section for the `dc-dc-phase0..6` runbooks + `dc-dc-teardown-rollback.md`. SEPARATE track from VR0's phase-NN above -- do not conflate stage numbers between the two. | | Something is broken (triage, incidents) | `references/troubleshooting.md`, then repo `runbooks/appendix-A-troubleshooting.md` | | CAPI / Magnum / mgmt-VM recovery | repo `runbooks/ops-capi-recovery.md` | | Deliver ANY repo change (script, runbook, doc) | run `bash scripts/repo-lint.sh` + the touched script's `tests//run-tests.sh` BEFORE handing it over | @@ -111,10 +112,21 @@ with a per-item revert. Under blanket approval, the changelog IS the review surface: every item states what, why (evidence), and how to revert. -**Deploy loop:** phase-00 runbook (D-061 destroy path) -> `bash -scripts/preflight.sh` PASS -> phase-01..08 gated -> `bash +**Deploy loop (VR0, single-DC testcloud):** phase-00 runbook (D-061 destroy +path) -> `bash scripts/preflight.sh` PASS -> phase-01..08 gated -> `bash scripts/cloud-assert.sh --capture` -> commit the asbuilt/ BOM. +**Deploy loop (VR1 DC-DC, the current live mission):** read +`docs/dc-dc-deployment-workflow.md`'s tooling gap register FIRST -- check +whether the stage you're about to run has any open blocking item -> Stage 1 +(`dc-dc-phase0-vcloud-prep.md`) gated -> Stage 2..7 gated in order, per +`runbooks/README.md`'s "VR1 DC-DC track" table -> if a stage's `tofu apply` +fails partway, read `runbooks/dc-dc-teardown-rollback.md`'s rollback decision +tree BEFORE reaching for a teardown (fix-forward is usually correct). Every +`opentofu/` command is UNVALIDATED (no `tofu` binary has run in any session +that authored this tree) -- run `bash scripts/opentofu-validate.sh` first if +available, and treat every OpenTofu step as reviewed-but-unexercised. + **Incident loop:** capture the verbatim error -> `bash scripts/cloud-assert.sh` (the service-own-verdict sweep localizes the layer) -> appendix-A by exact message -> recorded fix, gated -> log the finding (new root causes become diff --git a/.claude/skills/openstack-cloud-ops/references/environment.md b/.claude/skills/openstack-cloud-ops/references/environment.md index 1efd8aa..8bcbe80 100644 --- a/.claude/skills/openstack-cloud-ops/references/environment.md +++ b/.claude/skills/openstack-cloud-ops/references/environment.md @@ -4,14 +4,25 @@ re-measured or re-read from the repo/live cloud before use in a command - hard rule 2 applies. Snapshot date: 2026-07. The repo is fresher than this file. -## The two deployments +## The three deployments (this file covers VR0 in detail; VR1 is a pointer) -- **Testcloud (now):** VR0 DC0, four KVM host VMs (openstack0-3) on a single - hypervisor, managed by MAAS + Juju. Single-DC virtual rehearsal. +- **Testcloud (VR0 DC0, now):** four KVM host VMs (openstack0-3) on a single + hypervisor, managed by MAAS + Juju. Single-DC virtual rehearsal. The rest + of THIS file is VR0's own ground truth. +- **VR1 DC-DC (in progress, D-100..D-110):** two-DC + Office1-headend + rehearsal -- VR0's testcloud host is being REPURPOSED into DC1 (not a + second concurrent cloud). This file does NOT carry VR1's own facts (Office1 + network, DC1/DC2 mesh links, per-DC MAAS/Vault roots, etc.) -- those live + in `docs/dc-dc-deployment-workflow.md` (master tracker: stage status + + tooling gap register, read this first for any VR1 work) and + `docs/dc-dc-buildout-design.md` (the design itself). Runbooks: + `runbooks/README.md`'s "VR1 DC-DC track" section. IaC: `opentofu/` + (UNVALIDATED -- see its own README). - **Roosevelt (future):** bare-metal, multi-DC, commercial production (3310 Roosevelt Blvd, Eugene OR). Dedicated node roles (gateway/controller/ compute split) - unlike the hyperconverged testcloud. Every design choice is - judged by its transfer to Roosevelt. + judged by its transfer to Roosevelt -- VR1 is the rehearsal for THIS, not + an end in itself. ## Stack (verify against appendix-B for pins) @@ -92,6 +103,13 @@ - No KVM snapshot restore path exists (D-070 superseded D-012): rebuild-from-runbooks IS the restore path; baselines come from cloud-assert `--capture`. +- **VR1 DC-DC additions (2026-07-09/10), not part of VR0's map above:** + `docs/dc-dc-deployment-workflow.md` (master tracker), `docs/dc-dc-buildout- + design.md` (design), `docs/design-decisions.md` D-100..D-110, + `runbooks/dc-dc-phase0..6-*.md` + `dc-dc-teardown-rollback.md`, + `opentofu/` (libvirt/MAAS/OPNsense IaC, UNVALIDATED -- see its own + README), `netbox/dc-dc-prefixes-import.py`, `scripts/dc-dc-*.sh` (MTU/Ceph + budget calculators, Ceph replication tooling). ## Identity / tenancy model (see appendix-C/D and D-051, D-064, D-066) diff --git a/.claude/skills/openstack-cloud-ops/references/troubleshooting.md b/.claude/skills/openstack-cloud-ops/references/troubleshooting.md index 32465ef..c5ba069 100644 --- a/.claude/skills/openstack-cloud-ops/references/troubleshooting.md +++ b/.claude/skills/openstack-cloud-ops/references/troubleshooting.md @@ -10,6 +10,15 @@ This file gives you the method and the reflexes that must fire BEFORE and WHILE you read those. +**VR1 DC-DC track:** appendix-A above is VR0-specific -- nothing in the VR1 +DC-DC track (`docs/dc-dc-deployment-workflow.md`) has been executed yet, so +there is no equivalent incident index for it (an OpenTofu/MAAS-`vm_host`/ +OPNsense-edge failure will not be in appendix-A). Apply the SAME triage +method below regardless, but do not expect a pre-catalogued fix for a VR1- +specific symptom -- log a genuinely new finding as a DOCFIX/D-NNN candidate +per this repo's own discipline rather than assuming it must already be +documented somewhere. + ## Triage method 1. **Reproduce and capture verbatim.** The exact error text routes you in diff --git a/docs/changelog-20260710-skill-vr1-awareness.md b/docs/changelog-20260710-skill-vr1-awareness.md new file mode 100644 index 0000000..703e6f6 --- /dev/null +++ b/docs/changelog-20260710-skill-vr1-awareness.md @@ -0,0 +1,74 @@ +# Changelog 2026-07-10 -- skill/reference VR1 DC-DC awareness (DOCFIX-177) + +No live infrastructure touched. Found while the operator was heading to the +vcloud host to begin real execution: `.claude/skills/openstack-cloud-ops/` +(loaded before any nontrivial task, per this repo's own `CLAUDE.md`) and +`runbooks/README.md` (what the skill's routing table points to for "Deploy / +redeploy / teardown") were both SILENT on the entire VR1 DC-DC track -- +zero mentions of `docs/dc-dc-deployment-workflow.md`, the seven +`dc-dc-phase*.md` runbooks, `dc-dc-teardown-rollback.md` (DOCFIX-176), or +`opentofu/`. A fresh session reading only the skill (which is exactly what +happens at session start / after compaction) would have been routed straight +to VR0's `phase-01..08` runbooks with no indication the DC-DC track -- the +actual live mission for today -- exists at all. + +## Fix + +1. **`runbooks/README.md`**: new "VR1 DC-DC track" section -- a table + mirroring the existing VR0 "Phases" table's format, listing all 7 Stage + runbooks + `dc-dc-teardown-rollback.md`, explicit that Stage numbering + here is a SEPARATE sequence from the VR0 phase-NN numbers above it (they + share a repo, not a numbering scheme), and a pointer to + `docs/dc-dc-deployment-workflow.md` as the actual master tracker (stage + status + tooling gap register) rather than duplicating that content here. +2. **`.claude/skills/openstack-cloud-ops/SKILL.md`**: routing table gained a + dedicated VR1 DC-DC row (previously the "Deploy / redeploy / teardown" + row only pointed at the VR0 sequence); "Standard loops" gained a second + "Deploy loop (VR1 DC-DC)" entry alongside the existing VR0 one, including + the fix-forward-before-teardown reminder from DOCFIX-176's rollback + decision tree and the opentofu UNVALIDATED caveat. +3. **`references/environment.md`**: "The two deployments" section (VR0 + + Roosevelt) became "The three deployments" -- VR1 DC-DC added as a + pointer-only entry (this file stays VR0's own detailed ground truth; + VR1's facts live in the master tracker, not duplicated here). "Repo map" + section gained a VR1-specific addition listing where the new docs/ + runbooks/opentofu/scripts actually live. +4. **`references/troubleshooting.md`**: one paragraph clarifying that + `appendix-A-troubleshooting.md` is VR0-specific and the VR1 track has no + incident index yet (nothing has been executed) -- so a future + troubleshooting session doesn't waste time searching appendix-A for a + VR1-specific symptom that was never logged there, and knows to log a + genuinely new VR1 finding as a DOCFIX/D-NNN candidate instead of assuming + it must already be documented. + +**Checked, not changed:** `references/operating-discipline.md`'s "Session +continuity" section was re-read against tonight's actual practice (ledger +updates at every deliverable, `ledger-scan.sh` reconciliation) and found +accurate -- no drift to fix. `references/script-authoring.md` is generic +command-block/gating discipline that applies equally to both tracks -- +correctly track-agnostic, left as-is. `references/opentofu-provider-docs.md` +is provider-research content (already VR1-authored this session), not +topology/routing content -- out of scope for this pass. + +## Also fixed before this delivery: unpushed commits + +Found `git status` showing `main` 2 commits ahead of `origin/main` +(DOCFIX-174 and DOCFIX-175/176) -- since the session bootstrap the operator +runs on the vcloud host starts with `git pull`, these would have been +invisible there this morning. Pushed to `origin/main` before starting this +delivery. + +## Verification + +`bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy warn (ASCII-only +convention held across all edits). No script logic changed -- doc/runbook/ +skill prose only, no test harness applies. Full gauntlet re-run: same +pre-existing environment gaps as every prior check tonight, zero +regressions (see this delivery's session-ledger entry for the exact count). + +REVERT: `git checkout HEAD~ -- runbooks/README.md +.claude/skills/openstack-cloud-ops/SKILL.md +.claude/skills/openstack-cloud-ops/references/environment.md +.claude/skills/openstack-cloud-ops/references/troubleshooting.md` (safe -- +routing/reference content only, no live infrastructure or script behavior +depends on any of it). diff --git a/docs/session-ledger.md b/docs/session-ledger.md index fd00c5f..92acb7e 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -1472,6 +1472,23 @@ BUNDLEFIX-012. Caught the same way as everything else tonight: by running the tool and checking whether its own output was plausible before trusting it, not by code review. +- **DOCFIX-177** (`docs/changelog-20260710-skill-vr1-awareness.md`): + operator was heading to the vcloud host to begin real execution; asked + "is the skill ready?" -- it wasn't fully. `.claude/skills/openstack-cloud- + ops/SKILL.md` and `runbooks/README.md` (what its routing table points to) + had ZERO mentions of the entire VR1 DC-DC track -- a fresh session reading + only the skill would be routed to VR0's phase-01..08 runbooks with no + indication the DC-DC work (today's actual mission) exists. Fixed: + `runbooks/README.md` gained a full "VR1 DC-DC track" index table (7 Stage + runbooks + `dc-dc-teardown-rollback.md`, explicit that its numbering is a + separate sequence from VR0's), `SKILL.md`'s routing table + standard loops + gained VR1-specific rows, `references/environment.md` gained VR1 pointers + (a third "deployment" entry + a repo-map addition), `references/ + troubleshooting.md` gained a note that appendix-A is VR0-only and VR1 has + no incident index yet. Also found and fixed first: `main` was 2 commits + ahead of `origin/main` (DOCFIX-174, 175/176) -- pushed before this + delivery, since the vcloud host's session bootstrap starts with `git + pull` and would otherwise have missed them this morning. - **DOCFIX-176** (`docs/changelog-20260710-vr1-teardown-rollback-runbook.md`): closes gap register item #19. New `runbooks/dc-dc-teardown-rollback.md` -- the OpenTofu/libvirt/MAAS-vm_host layer had no teardown runbook (D-061's diff --git a/runbooks/README.md b/runbooks/README.md index fbbd4fc..7770a69 100644 --- a/runbooks/README.md +++ b/runbooks/README.md @@ -48,6 +48,35 @@ | 07 | phase-07-conductor-graft.md | Trustee domain-setup + graft the magnum-capi-helm driver | D-031 / D-037 / D-042 / D-046 / D-047 | | 08 | phase-08-workload-cluster-acceptance.md | End-to-end tenant cluster + acceptance bar | D-011 (amended D-019) | +## VR1 DC-DC track (separate from the VR0 phases above) + +The `dc-dc-phase*.md` runbooks below are a SEPARATE track from the VR0 +`phase-NN` sequence above: they build the two-DC + Office1-headend buildout +("VR1"), per `docs/dc-dc-buildout-design.md` and D-100..D-110, with VR0's own +testcloud host becoming DC1 (repurposed, not a second concurrent cloud). The +master tracker -- current stage status, the full tooling gap register, exit +gates -- is `docs/dc-dc-deployment-workflow.md`; read that first, not just +this table. "Stage" numbering here matches that tracker, not this README's +own VR0 phase numbers (they are two independent sequences that happen to +share a repo). + +| Stage | File | Purpose | Decisions | +| ----- | -------------------------------------- | ----------------------------------------------------- | ----------- | +| 0 | (no runbook -- `docs/design-decisions.md`) | Decision ratification (D-100..D-110) | D-100..D-110 | +| 1 | dc-dc-phase0-vcloud-prep.md | vcloud host prep + first real `tofu apply` (DC1 planes/Office1 pool/mesh links) | D-101/D-102 | +| 2 | dc-dc-phase1-office1-standup.md | Office1 headend: MAAS region, OpenTofu, NetBox, GitBucket, Tailscale, Office1's own OPNsense edge | D-103/D-106 | +| 3 | dc-dc-phase2-tofu-dc-substrate.md | OpenTofu builds each DC substrate (node-VM domains, OPNsense edge, MAAS `vm_host`) | D-103 | +| 4 | dc-dc-phase3-maas-enlist-deploy.md | MAAS enlist / commission / deploy, per DC | | +| 5 | dc-dc-phase4-juju-bundle-per-dc.md | Juju controller + OpenStack bundle, per DC (high reuse from VR0's bundle.yaml) | D-106/D-108 | +| 6 | dc-dc-phase5-dr-failover-drill.md | DR wiring (radosgw multisite, rbd-mirror) + failover drill | D-108 | +| 7 | dc-dc-phase6-designate-cos-magnum.md | Designate, COS, Magnum/CAPI, per DC | | +| -- | dc-dc-teardown-rollback.md | Teardown/rollback for this track's OpenTofu-provisioned infrastructure specifically -- NOT the same as `phase-00-teardown-*.sh` (D-061), which covers VR0's juju/MAAS-machine layer | -- | + +**As of 2026-07-10: all seven Stage runbooks + the teardown/rollback runbook +are WRITTEN, NONE EXECUTED.** `docs/dc-dc-deployment-workflow.md`'s tooling +gap register tracks what's still open before each stage can run for real -- +read it before starting Stage 1, not just this index. + ## Appendices - **appendix-A-troubleshooting.md** -- symptom -> cause -> fix index, keyed by