# Session ledger -- in-flight work continuity record **Purpose.** Long ops sessions on this cloud routinely exceed a single context window and get COMPACTED (sometimes several times). Anything living only in the chat scrollback is lost at compaction. This ledger is the durable, committed record of what is in flight, so any session -- after a compaction, or a fresh one, or the parallel Claude Code stream -- can resume without losing pending work. **How to use it (standing practice).** 1. At session start: read this ledger AND run `bash scripts/ledger-scan.sh`. Reconcile. 2. `ledger-scan.sh` is the DRIFT CHECK -- it derives the open-work it reliably can (PROPOSED/OPEN decisions, OPEN security rows, next-free numbers) straight from the repo. This narrative must not claim CLOSED anything the scan shows OPEN, nor omit what it surfaces. 3. Update this ledger at every deliverable/commit -- it is a standing deliverable like the changelog. The changelog says what CHANGED; this ledger says what is still OPEN. 4. Numbers and the machine-derived section below are seeded from a scan; re-run the scan and re-seed rather than editing those by hand. --- ## Machine-derived (re-seed from `scripts/ledger-scan.sh`; do not hand-edit) _As of 2026-07-09 scan (post-DOCFIX-151, $DC selector convention closed); re-run the scan to refresh:_ - **PROPOSED / OPEN decisions:** D-068 (Vault substrate hardening, Roosevelt -- 1.16 ruled out per amendment; off-EOL path OPEN), D-071 (routine update cadence + controller patch policy -- operator to rule; gated on the pre-DC-DC HA/backup session). **D-100..D-110 are ALL ADOPTED as of 2026-07-09** (D-102 MERGED into D-101; D-109 confirmed standalone) -- Stage 0 of `docs/dc-dc-deployment-workflow.md` is CLEARED; see `docs/dc-dc-buildout-design.md` Section 10 for the ruling record. _(D-050 RESOLVED/CLOSED 2026-07-06 via D-051; D-073 APPLIED live 2026-07-06 -- not scan-tracked as open. D-076..D-099 are a RESERVED source-project band, never assigned in this repo, per D-110.)_ - **OPEN security rows:** SEC-001 (rotate at v1 close, re-ruled 2026-07-06), SEC-003 (custodians + second-person unseal -- DEFERRED; keeps d011-06 MANUAL), SEC-004 (flip repo to private at v1 close -- deferral reaffirmed). - **Next-free numbers:** D = 111, DOCFIX = 151, BUNDLEFIX = 012. (Seeded from the 2026-07-09 scan, post DOCFIX-151.) --- ## Active build (this session) - **validate.sh D-011 runner -- modular.** Foundation committed: `scripts/lib-validate.sh` (exit contract 0/1/2/3/4, emit, vr_json stderr-safe, run, env hygiene, disruptive gate) + `scripts/validate.sh` orchestrator (profiles, verdict, --stop-on-fail, --include-disruptive). - Batch 1 committed: `checks/d011-01-charms`, `checks/d011-06-vault-unseal` (MANUAL until the SEC-003 rehearsal closes). - Batch 2 committed: `checks/d011-02-vip-jumphost` (catalog-derive public origins, TLS+HTTP), `checks/d011-03-vip-tenant` (ephemeral agnhost pod egress -> keystone VIP, D-035 proof). `post-restart` profile now fully populated. - Batch 3 DRAFTED (mock-tested only, NEEDS LIVE VALIDATION): `d011-04-octavia-lb` (RR+member additive; amphora failover --disruptive, N+1 headroom-guarded) + `d011-05-magnum-e2e` (wraps tenant-acceptance + timing). full-d011 profile now COMPLETE. - **Next:** live-validate batch 3 (see Verify-live queue), then item-3/#5-#8 backlog. - D-011 AMENDED bar: 1 charms; 2 VIP jumphost; 3 VIP tenant; 4 octavia RR/failover/recovery; 5 magnum e2e + OCCM; 6 second-person manual unseal (attestation, D-069); 7 DROPPED (D-070 supersedes D-012 snapshots); 8 DROPPED (D-019, no Designate). ## Script backlog (changelog-tracked) - DONE: tenant-offboard.sh, vault-kv-health.sh, cloud-assert.sh (item 3 largely), validate foundation + batch 1. - **item 3 remainder:** fold `vault-kv-health.sh` into cloud-assert as a section; use the combined gate to regenerate the stale restart-procedure runbook. - **item 5:** `scripts/tenant-cluster-create.sh` -- generalize the t2-02 stage6 + watch + D-066-evidence wrapper. Not started. - **item 6:** `scripts/keystone-policy-drift.sh` -- D-051 base_* alignment (LIVE-READ PENDING). Not started. - **item 7:** `scripts/cloud-snapshot.sh` -- juju-export baseline capture (D-070 retired KVM snapshots but KEPT the export/inventory baseline). Not started. - **item 8:** `tests/tenant-acceptance/` harness -- pending (tenant-onboard harness landed). ## Register / operator-gated (see docs/handoff-20260703-open-items.md) - **R-1 / SEC-003:** Vault unseal-key custodian assignment (OPEN). Gates V-2 and closes `d011-06-vault-unseal` (currently MANUAL). Operator input, recorded off-repo. - **V-2:** second-person unseal rehearsal (blocked on R-1). - **R-2 (D-043 caveat):** capi-mgmt-v2 auto-resume exclusion -- rule or accept. - **R-3:** was D-063 -- now CLOSED. The handoff doc still lists R-3 OPEN; update it. - **D-1:** Pattern-A full redeploy (VR0 DC0). ## Verify-live queue (batch 3 drafts -- confirm on real cloud before trusting) - d011-04 amphora_headroom() field parsing (`server show` .flavor; `hypervisor list --long` vcpu/ram) -- safe default HOLDs on any parse miss, but confirm the OK path live. - d011-04 OCCM LB-name-contains-service-name assumption; agnhost /hostname round-robin behavior. - d011-05 needs a FOIL tenant (2nd tenant) for P3 isolation -- onboard one or set VR_FOIL_APPCRED (acme, the former foil, was offboarded). - Sequence: run d011-04 non-disruptively first (RR+member), then --include-disruptive once headroom confirmed. ## Logged-not-actioned (small; would vanish at compaction) - offboard v2 `--sweep-magnum-orphans` mode (orphan per-cluster trustee in the magnum domain). - offboard stage-3 app-cred idempotency re-run guard. - `host_href=None` barbican observation (exercised fine in stage 6; probably closeable). - DOCFIX candidate: sweep repo for other `-f json ... 2>&1` stderr-merge hazards (DOCFIX-085 class). - D-050 (keystone policyd override) and the D-063-adjacent `identity:list_trusts=""` hardening remain PROPOSED/OPEN, not actioned. ## Jumphost stream -- ops-update-20260705: WINDOW CLOSED 2026-07-05 (addenda 13-16) First execution of runbooks/ops-update-procedure.md (DOCFIX-086; addendum 13). Logged session ops-update-20260705; checkpointed here at each milestone for cross-stream sync. State as of the Section 2 entry checkpoint: - **DONE Section 1 (pre-flight):** client 3.6.25 / controller 3.6.24 / all 91 agents uniformly 3.6.24 -> target 3.6.25. Worklist = 17 apps. Quiesce clean. Pre-change BOM asbuilt/20260705-102951 committed on a TRUE cloud-assert PASS (required the DOCFIX-087 A7 false-negative fix, addendum 14 -- window was HELD on the operator's Option B ruling until it landed). - **DONE Section 2.1:** controller state backup via `juju create-backup -m admin/controller` (902MB, checksummed, ~/openstack-baseline/, jumphost-only). NOTE: controller model is admin/controller, NOT /controller. - **DONE Section 2.2/2.3:** controller upgraded 3.6.24 -> 3.6.25 (blind window ~1 min, as expected); post-controller cloud-assert PASS. - **DONE Section 3:** all 91 openstack-model agents + controller machine at 3.6.25, states clean (63 idle / 28 started), nothing stuck. - **DONE Section 4 G0:** keystone 778->817; token probe OK; cloud-assert PASS. - **DONE Section 4 G1 (11 apps, standing group approval, serial+gated):** placement 125->154, nova-cloud-controller 795->823, neutron-api 650->710, neutron-api-plugin-ovn 178->215, glance 642->681, glance-simplestreams-sync 124->152, octavia-diskimage-retrofit 196->232, cinder 733->820, cinder-ceph 533->568, barbican 209->265, barbican-vault 75->99. All probes OK; boundary cloud-assert PASS. - **DONE Section 4 G2:** octavia 441->542; LBs 2/2 ACTIVE/ONLINE; cloud-assert PASS. - **DONE Section 4 G3:** dashboards 728->750 / 59->122 / 120->168; D-044 override INTACT; HTTP+login healthy. RCA: VIP HTTPS dead SINCE DEPLOY (haproxy 443 backend targets vhost-less internal addr; L4 check masks; phase-03 3.3 check fails open) -- NOT a G3 regression; G3 stands (operator ruling; addendum 15). Upstream bug + 2 DOCFIX candidates queued post-window. - **DONE (coordinated):** vault bundle revert 1.16->1.8/stable (BUNDLEFIX-010) + D-068 AMENDMENT recorded; 1.16 ruled out (certs V0/V1, Raft-only, Ceph, BUSL); "off EOL 1.8" remains OPEN. Evidence: docs/D-068-vault-1.8-vs-1.16-analysis.md. - **DONE Section 4 G4:** nova-compute 827->894; hypervisors up; guests byte-identical pre/post. - **DONE Sections 5-6:** post cloud-assert PASS; post BOM asbuilt/20260705-194617 committed; version coherence exact (91 agents @ 3.6.25); BOM diff = 17 expected rev pairs + 1 explained metadata delta (cinder secrets-storage endpoint); appendix-B B.1 fully re-baselined; runbook as-executed corrections DOCFIX-088; D-071 amended (backup exists). WINDOW COMPLETE. - **DONE follow-on (addendum 17):** magnum 70->96 + vault 372->714 refreshed after download-and-diff analysis (mass-rebuild campaign upstream; vault stayed reactive/V0, handlers byte-identical, never sealed). Fleet FULLY CURRENT (zero can-upgrade-to); appendix-B re-baselined from asbuilt/20260705-201118. - **DONE (addendum 18):** D-072 ADOPTED+EXECUTED -- dashboard VIP TLS repaired (cluster binding -> metal-admin; BUNDLEFIX-011 durable; https 200 first time on this build). DOCFIX-089 phase-03 3.3 fail-closed + appendix-A entry. Upstream bug DRAFTED: docs/upstream-bug-draft-dashboard-tls.md. - **POST-SESSION QUEUE (operator):** submit the upstream bug + record LP number in D-072; rule on D-071 (still PROPOSED); Vault off-EOL path (Roosevelt-scale, 1.16 ruled out per D-068 amendment; in-channel 714 applied). - **Window findings logged for close-out (do not action mid-window):** 1. magnum can-upgrade-to anomaly REPRODUCED (points at magnum-dashboard-122; evidence ~/openstack-baseline/magnum-can-upgrade-anomaly-20260705.json) AND Charmhub magnum 2024.1/stable latest release (r101, 2026-07-01) is s390x-ONLY -- no valid amd64 target; magnum EXCLUDED; report upstream to OpenStack Charmers. 2. vault NEWLY offers in-channel 372->714 on 1.8/stable -- NOT refreshed (D-068); logged only. 3. `juju create-backup`/`download-backup` EXIST on juju 3.6 -- authoring assumption ("removed in 3.0, expected absent") is WRONG; correct ops-update-procedure 2.1 and the D-071 risk section at window close (DOCFIX candidate). 4. juju status --format=line omits workload messages on 3.6 (root cause of the A7 false negative; fixed by DOCFIX-087). ## Jumphost stream -- session 2026-07-05 (post-window resume) - Orientation + reconcile DONE: machine-derived block re-seeded from today's scan (DOCFIX next-free 090 -> 091 was stale); shared-section stale-facts correction appended per the handoff reconcile item. Fences validate 4/0/0. - FINDING (logged, not actioned): `repo_lint.py` L5 "next-free identifiers" info line counts ANY textual mention of D-/DOCFIX-/BUNDLEFIX-NNN across all files, so it is inflated by the deliberate high-numbered decoys in the tests/ledger-scan fixture ("must NOT inflate") and by next-free pointer lines themselves. ledger-scan (heading-based, pointer-excluding) is the numbering authority and is correct. DOCFIX candidate: make L5 reuse ledger-scan's exclusion rules or drop the line. No number consumed. NOTE (2026-07-06 correction): the first version of THIS bullet quoted the decoy tokens literally and thereby inflated ledger-scan's own DOCFIX/BUNDLEFIX next-free (docs/ prose is scan-scanned; only "next-free" lines are excluded). Tokens de-fanged; standing rule: never write identifier-shaped tokens above the real high-water mark in docs/ or runbooks/ prose. - Standing by for batch-3 live validation support (verify-live queue). ## Jumphost stream -- session 2026-07-06 (operator ruling sweep; addendum 20) - RECORDED: D-050 RESOLVED/CLOSED via D-051 (operator ruling); D-073 ADOPTED (identity:list_trusts tightened to modern keystone default -- IMPLEMENTATION + TESTING PENDING, nothing live yet; proposed sequencing: before batch-3 d011-05); D-072 amendment (upstream dashboard-TLS bug WITHDRAWN -- operator judges site misconfig, draft retained); SEC-001 -> rotate at v1 close; SEC-003/004 deferred. - PINNED (operator): D-071 ratification and D-068 off-EOL path choice held until A1/A2 (batch-3 entry) complete. Magnum can-upgrade-to upstream report pinned. - A1 foil-tenant onboarding APPROVED in principle; tenant-structure model under discussion first (SCS Domain Manager persona confirmation + operator/tenant responsibility split). A2 (d011-04 disruptive gating) held behind A1. - IN DISCUSSION (B2): OpenBao evaluation; D-068 item 2 (vault_url cleartext http); D-068 item 3 (AppRole TTL audit + proactive probe) scoping. - DELIVERED (addendum 21): DOCFIX-091 docs/tenant-onboarding-contract.md per A1 rulings 1-4 (boundary, roles, intake, dangers, hardening register H1-H5 -- proposals logged, none implemented). A1.4 interpretation pending operator confirm: "SSH access into their domain" read as canary access-proof (H3). Verify-live additions: Horizon manager-role GUI identity probe (contract 5.3). - RULED (2026-07-06 cont.): A1.3 H1-H5 ADOPTED as pre-v1-close tasks; A1.4 canary interpretation CONFIRMED + adopted as pre-deployment task. Full onboarding completion package (scripts + internal docs + client-facing drafts + live end-to-end validation) added to handoff section 6 v1-close checklist. - RULED (2026-07-06 cont.): clientdocs/ TOP-LEVEL for client-facing drafts; build INTERLEAVES with batch-3 (foil onboarding = live workflow validation); A2 disruptive half PRE-APPROVED conditional on live N+1 headroom PASS. - DELIVERED (addendum 22): H1 scripts/tenant-assert.sh + 7-case harness (green); clientdocs/ package DOCFIX-092 (README/intake/welcome/self-service drafts). Remaining in the interleave before foil onboarding: H2 idempotency guard, H3 canary stage, D-073 list_trusts zip implementation (then gated live apply). - DELIVERED (operator-away batch, addenda 23-25): H2 re-run guards + H3 canary stage7 (harness 12/12); D-073 implementation STAGED (39-rule zip, drift check PASS -- live apply GATED, block in the D-073 amendment); DOCFIX-093 runbooks/d011-batch3-window-DRAFT.md (the one-window do-doc: D-073 apply + foil stages 0-4 + tenant-assert + canary + d011-04/05 + close-out); H5 embedded; docs/D-068-openbao-assessment-DRAFT.md (B2 research input -- no OpenBao charm exists, MySQL backend removed upstream, V1 goes to Sunbeam not legacy charms; research rec: deadline-bound risk acceptance). Stderr-merge sweep DONE: 10-site inventory logged in addendum 25 (fixes = DOCFIX candidate, not actioned). - REPO-SIDE QUEUE now EMPTY except: H4 policy-drift script (main-chat backlog item 6 -- coordinate before building) and the client-docs sweep-on-change rule. EVERYTHING ELSE waits on the operator-present live window (the do-doc). - DELIVERED (operator-away batch part 2, addendum 26): DOCFIX-094 stderr-separation fixes (cloud-assert A5 FAIL-OPEN closed + harness proof; onboard resolvers; offboard blind-sweep guard exit 20; 3 runbook paste blocks; acceptance:43 rides its future harness). Vault measurements: charm `channel` + ssl-* options verified; snap has NO 1.13/1.14 track -- max MPL payload in-charm = 1.12.11; OpenBao memo updated. Orphan-trustee sweep draft NOT started (queued). - BLOCKER: juju controller macaroon EXPIRED ~17:10 2026-07-06 -- all juju cmds prompt for password. OPERATOR: `juju login -u jessea123` at next presence. Blocked reads queued: keystone policy.json PENDING-LIVE-READ trio; barbican host_href look. (Vault reads above completed BEFORE expiry.) CLEARED later 2026-07-06: auth verified working; both queued reads DONE (see addendum 27 closures). ## Jumphost stream -- session 2026-07-06 (cont.): d011-batch3 WINDOW EXECUTED (addendum 27) - **WINDOW COMPLETE, all gates green** (logged d011-batch3, per-command wrap -- operator-ruled Claude Code lane, now in the convention doc as DOCFIX-097): D-073 APPLIED live (verify pair + H4-live PASS proof); foil1 onboarded 10.20.29.0/24 (stages 0-4 first-try, tenant-assert PASS 15/15 after fix, H2 double-SKIP, H3 canary PASS FIP 10.12.7.177/ubuntu); d011-04 PASS both halves (A2 conditional honored; headroom OK on all 3 hypervisors); d011-05 PASS (P1/P2/P3 with foil1); baseline + close-out cloud-assert PASS. - **In-window checker fixes (each harness-proven then re-run live):** DOCFIX-095 tenant-assert owner-context app-cred probe (admin-side list is always-403 by base policy admin_or_owner -- measured, NOT a D-073 effect); DOCFIX-096 d011-04 amphora_headroom --long + display-name hypervisor keys. - **D-011: batch-3 CLOSED; D-011 CLOSED-except-item-6** (SEC-003 manual unseal rehearsal). v1-close checklist (handoff section 6) now QUEUED. - **Queued-read closures:** keystone trio CONFIRMED (D-064 values explicit in live base policy; overlay comment tags dischargeable -- DOCFIX candidate); barbican host_href CLOSED benign-with-corrected-mechanism (charm renders literal https://None:9312, suppressing the upstream request-derived fallback; catalog unaffected; charm-level fix = future item needing D-NNN). - **Operator rulings this session:** devteam pinned as next tenant, onboarding HELD until a cloud snapshot exists (item-7 cloud-snapshot.sh draft launched); orphan-sweep audit stays rc 0 report-only; clientdocs L7 receipt keeps all 14 files; H4 design defaults accepted as delivered. - **Worktree drafts DELIVERED (uncommitted, integration queued):** orphan-trustee sweep (14/14), clientdocs L7 (17/17), tenant-acceptance harness (16 checks; closes the last stderr-inventory site), H4 keystone-policy-drift (10/10 + first live PASS), clientdocs handover pack (lint clean), cloud-snapshot.sh (in flight). Integration order: after this window commit; tenant-acceptance + L7 receipt last (receipt must hash post-integration file states). - **Open (small):** Horizon manual checks on foil1 (operator, browser, non-blocking); capi-test-1 CREATE_FAILED leftover (orphan-sweep candidate); beta-cluster lbtest leftover; repo-lint worktree-exclusion DOCFIX candidate. - **SCRIPT-QUALITY BATCH LANDED (addendum 33, 2026-07-07):** repo-wide sweep (110 scripts) -> operator-ruled fix scope shipped as DOCFIX-112..117: stage2 anti-escalation guard fail-closed (SECURITY -- was fail-open both ways); tenant-offboard hardening cluster (stderr-as-argv, Phase B counter CLOSED after 2 days logged, LB-wait fails loud, cred-file auth_url); validate.sh nonstd-exit HOLD; juju-spaces-check macaroon-vs-absent; pipefail sweep + vault-kv-health rc-test; guard-hook blocklist extension. Gauntlet 38/38. Deferred S-class + minor R items in docs/script-quality-findings-20260707.md (operator has NOT ruled S1-S5/S7/S8). ## Jumphost stream -- session 2026-07-08: devteam cluster-template incident (addendum 39) - **DIAGNOSED + DELIVERED:** devteam "Unable to create cluster template" = Public/Hidden flag -> `clustertemplate:publish` (admin_api) refusal, BY DESIGN; read-only diagnosis, no cloud change. DOCFIX-121 (clientdocs template-creation guidance + Public/Hidden warning) + DOCFIX-122 (appendix-A symptom entry); L7 receipt re-recorded; ~/devteam-package re-rendered (kubernetes.md, self-service-guide.md) for tonight's push. - **OPEN (operator option, unruled): stage5 for devteam** -- `devteam-k8s` was never delivered (window ran stages 0-4); one gated command (`bash scripts/tenant-onboard.sh devteam 5`) closes the docs' delivered- template promise; else devteam self-creates per new docs. - **OPEN (operator, small):** role-less GUI test user in the devteam domain (grant member or delete); Role-dropdown question on the dashboard-override test (why load-balancer_admin was submitted) -- evidence folded into the D-076 draft (worktree, uncommitted). - **TestCluster NoValidHost + decommission audit (addendum 40):** devteam 1+3 cluster stalled on RAM capacity (NOT quota); operator deleted it, devteam re-created 1+1 (TestCluster2). QUEUED RULINGS from the audit: (a) beta-cluster teardown (~17GB reclaim) -- cross-stream, main-chat owns the resize-acceptance role; (b) lbtest LB leftover reclaim via kubectl (beta kubeconfig, main-chat asset); (c) foil1 offboard (RECOMMENDED -- dormant shell); (d) quota-vs-capacity expectation (trim testcloud quotas or document in clientdocs) -- unruled. ## Jumphost stream -- session 2026-07-08 (cont.): Jenkins+K8s guide overnight (addendum 41) - **DELIVERED (DOCFIX-123):** new clientdocs/jenkins-kubernetes-guide.md -- the single end-to-end Jenkins-to-Kubernetes workflow HUB (Pattern A deploy- into-cluster + Pattern B build-agents; self-create cluster path; kubeconfig- as-Jenkins-credential; NoValidHost sizing; flannel/Public-Hidden/keypair traps; troubleshooting; glossary; cross-links). Flannel hole also closed in tenant-skill kubernetes.md. Guide added to LEAK_FILES (harness change), README updated, L7 receipt 30 files, gauntlet 38/38. Instantiated to ~/devteam-package (devteam values). Operator pushes tonight. - **OPERATOR RULINGS (on return, 2026-07-08 PM) + status:** 1. devteam cluster: DELETE + DELIVER TEMPLATE ruled. DONE (addendum 42): TestCluster2 confirmed gone + devteam-k8s calico template delivered. 2b. Handover package RE-consolidation (operator re-ruling, addenda 46-47): glossary folded into handover-pack + glossary.md removed + account-table hard de-dup (DOCFIX-128); tenant AI skill aligned to the consolidated package -- 3 missing failure signatures added, capacity caveat, Jenkins- guide integration (DOCFIX-129). Top-level docs 9->8. Package re-instantiated (glossary appendix in handover-pack, skill refreshed). DONE. 2. Consolidation review M1-M7: ALL APPROVED. DONE (addendum 43, DOCFIX-124): M5 error-holes in self-service-guide; M3 entry point + goal-map in welcome; M1/M2 single-source-of-truth + pointers (softer than hard-delete -- flagged, protects the AI skill + P2 test); M6 Jenkinsfile path/stage fixes; M4 new glossary.md. Package re-instantiation for the changed set = next step. FLAGGED: package handover-pack.md has unfilled {{CUSTODIAN_1/2}} (operator value; pre-existing, devteam placeholder-custodian test). 3. Gated script fixes: BOTH DONE (addendum 45). DOCFIX-126 offboard E0 always-403 no longer FATALs before Phase F (harness case added, 21/21); DOCFIX-127 onboard cred files now carry dashboard_url (env-overridable, as-built https://10.12.4.58/horizon). LIVE VERIFY STILL GATED: next real onboard/offboard confirms end-to-end (onboard cred-write stages 1/2 have no unit coverage -- pre-existing; logged future harness item). 4. DOCFIX list CLOSED (operator); handover-pack.md fine as-is (already carries {{DASHBOARD_URL}}/{{AUTH_URL}}). 5. HANDOVER READY (operator confirmed 2026-07-08). Two flags ACCEPTED for the devteam FIRST INTERNAL TEST: (a) package handover-pack.md keeps literal {{CUSTODIAN_1/2}} -- fine for the test, fill with real names for a real client handover; (b) existing ~/tenant-devteam cred files predate the DOCFIX-127 dashboard_url field (written 2026-07-07) -- fine for today, the dashboard URL is in handover-pack section 3 which the team receives. Repo fully pushed (HEAD db284fc). REMAINING = operator-manual delivery only: copy ~/devteam-package + ~/tenant-devteam to the two custodians in person (guard blocks agent access to cred material). - **Offboard E0 defect (for the fix):** admin-side identity:list_application_credentials is always-403 (base admin_or_owner; measured), so offboard v2 --apply counts 3 false failures and FATALs at exit 23 BEFORE Phase F (cred-dir retirement). foil1 offboard hit this live; cred dir retired by hand. Fix: owner-context-or-skip E0 inventory, do not count known-403 as failure; + harness case. ## Jumphost stream -- session 2026-07-08 (cont.): devteam self-service cluster live (post-handover) - **devteam self-created their first cluster (read-only audit, no cloud change).** `TestCluster` (uuid 92b92eed-afa3-4aaf-ab33-13d3ddeea869) CREATE_COMPLETE + HEALTHY (cluster / infrastructure / controlplane / nodegroup all Ready), created 2026-07-08 17:18 UTC in devteam-prod. 1 master + 1 worker, keypair devteam-key, api https://10.12.7.130:6443. calico + small-first + correct keypair + no Public/Hidden flag -- the DOCFIX-121 dashboard symptom resolved IN PRACTICE on their first real self-service create. (Note: this fresh TestCluster reuses the name of the earlier deleted TestCluster/TestCluster2, addenda 40/42 -- cosmetic.) - **Built via HORIZON, not the CLI.** They created their own template `TestTemplate` (calico, public/hidden False, keypair baked in = devteam-key) in the dashboard and launched TestCluster from it, rather than the delivered CLI template devteam-k8s (keypair_id None; measured, correct-by-design, still unused). Both are valid Magnum objects. - **What the Horizon path changes (LOGGED, not actioned; DOCFIX candidate, no number consumed):** 1. Nothing about the running cluster -- same Magnum API object either way. 2. Explains the keypair-on-template confusion this session: Horizon's Create Cluster Template dialog exposes an OPTIONAL Keypair field, so their GUI-built TestTemplate shows a key while the delivered devteam-k8s (CLI stage5 omits it by design) shows none. Comparing the two in-GUI drove the "no keypair -> must be required" read. It is not; the key is supplied at cluster-create time and their live cluster already carries it. 3. Doc-gap: jenkins-kubernetes-guide.md is CLI-centric. Their next guide step (Section 3, kubeconfig fetch via `openstack coe cluster config`) has NO Horizon path -- the supported retrieval is the CLI. (Earlier draft of this bullet claimed magnum-ui "offers a cluster-config download" -- UNVERIFIED, retracted; do not assert it.) The Jenkins integration requires the CLI/app-cred path regardless. - **ADDRESSED: DOCFIX-130 (addendum 48, COMMITTED f0f11ae; ~/devteam-package guide RE-INSTANTIATED 2026-07-08, 34937 bytes, tokens filled from live-measured AUTH_URL/REGION).** Grew from 2 gaps to a full guide pass on operator instruction ("fold and add 1-7... make GUI/CLI paths better defined"). clientdocs/jenkins-kubernetes-guide.md, additive only, no token touched: - STRUCTURE: new Section 0 dashboard-vs-CLI selector + **On the command line** / **In the dashboard** markers; dashboard equivalents for create (2.3), resize (2.5 new), teardown (8). Rule: lifecycle = either path; everything Kubernetes (kubeconfig onward) = CLI only. - GAPS 1-7 closed: registry/imagePullSecret (5.0; VERIFIED no registry service in catalog), ingress-nginx how-to + no-DNS (5.4 new), cluster resize (2.5 new; VERIFIED syntax), persistent storage self-verify (5.5 new), cert expiry read one-liner (3), tooling install + kubectl-skew (1); + ImagePullBackOff and PVC-Pending troubleshooting rows (9). - Leak harness CAUGHT two self-inflicted client-doc leaks ("Horizon", "magnum") on first pass -- reworded, now PASS 8/0; gauntlet 38/38. - ~/devteam-package NOT re-instantiated (refresh at next package sweep). See the changelog DOCFIX-130 entry. Numbers: DOCFIX next-free now 131. - **STORAGE OPERATOR FOLLOW-UP (logged, not actioned):** delivered cluster template has volume_driver unset + no docker volume (MEASURED). Whether the built cluster gets a default StorageClass from the cinder-CSI default is UNVERIFIED (verifying = kubectl into devteam's live cluster; deferred for tenant isolation). If none, persistent volumes are unavailable out of the box -- candidate template improvement (would need a D-NNN). 5.5 is written self-verifying so the doc is correct either way. - **CLEANUP QUEUE RECONCILED against live (2026-07-08 read-only audit) -- LIVE IS AHEAD OF LEDGER, several "open" items already resolved:** - foil1 offboard (was RECOMMENDED, addendum-40 c): DONE -- no foil project/user, no orphan trustee. (Closes the dormant-shell item; also means the gated DOCFIX-126/127 offboard live-verify was NOT exercised by us -- still unverified end-to-end, next real offboard confirms.) - capi-test-1 CREATE_FAILED delete (REMAINING-OPEN item 3, was operator-executes): DONE -- no cluster record, trustee reaped. - beta-cluster teardown + lbtest LB (addendum-40 a/b, cross-stream/main-chat): DONE -- no beta cluster, no stray LB. (Main-chat to log its side.) - Audit state: only devteam TestCluster (CREATE_COMPLETE) + its kubeapi LB + its protected trustee remain; orphan sweep ORPHAN=0 PROTECTED=2 UNPROVEN=0; zero unattached floating IPs. - STILL OPEN from the addendum-40 audit: only (d) quota-vs-capacity expectation -- trim testcloud quotas or document the capacity ceiling in clientdocs (unruled, operator). Everything else in that audit is closed. ## Jumphost stream -- session 2026-07-08 (cont.): S-class script-quality batch (operator-greenlit) Operator greenlit the recommended sequence: (1) Phase A stderr fix, (2) S7+S4, (3) S8; HOLD S1/S2/S3/S5 (lib-validate adoption on tenant-onboard/offboard) until the DOCFIX-126/127 live-verify lands. Source: docs/script-quality-findings-20260707.md. - **DONE -- DOCFIX-131 (addendum 49): tenant-offboard.sh Phase A stdout-only.** New `tinv()` (tenant-scoped `inv()` twin) for the 3 Phase A `coe cluster list` captures; wait loop retries on list-error, final check fails closed. Latent bug: merged stderr could false-trip exit 21 on an already-clean tenant. Harness un- excludes coe from warnstderr + new coe-stderr-not-argv assertion (22/22). Gauntlet 38/38, lint 0 fail, L7 re-recorded. Committed + pushed. - **DONE -- S7 (DOCFIX-132) + S4 (DOCFIX-133), addendum 50.** S7: removed dead `CF` (d011-04) + subsumed `vr_is_ipv4` guard (d011-03). S4: three inline trust filters -> one fixture-tested scripts/trust_filter.py (+tests/trust_filter/ 9/9); SDIR added to offboard + acceptance. Gauntlet 39 ALL GREEN. Committed + pushed. - **S8 RE-SCOPED (bigger than its "S" estimate; operator decision needed before I wire it):** the KEYSTONE_VIP literal (10.12.4.50) lives in 6 files, and 3 of them (tenant-onboard/offboard/acceptance) do NOT source lib-net.sh -- so centralizing means adding `. lib-net.sh` sourcing into client-critical scripts I just modified (DOCFIX-131 + S4), which stacks risk. Breakdown: - S8a (FIP pool FIP_START/END -> lib-net.sh): DONE (DOCFIX-134). FIP_POOL_START/END added to lib-net.sh; phase-04 create/verify reference them. Gauntlet 39/39. - S8b (KEYSTONE_VIP -> lib-net.sh): DEFERRED -- needs sourcing plumbing into 6 scripts incl. the 3 tenant scripts (onboard/offboard/acceptance) that do NOT source lib-net. Operator decision needed before wiring client-critical scripts. - S8c (provider-bundle-check.py restates plane CIDRs): DEFERRED -- design change (python can't source bash -- needs a generated/parsed form). - **DONE -- DOCFIX-135 (addendum 51): tenant AI skill registry gap.** devteam asked (via the assistant) if there's a registry setup step; the DOCFIX-130 registry content was not mirrored into the skill (grep found zero). Added "Images: bring your own registry" to references/kubernetes.md + ImagePullBackOff signature to references/troubleshooting.md; skill harness 8/8, gauntlet 39/39; package skill files re-instantiated. Committed + pushed. (Skill-lag class; watch for it on future guide changes.) - **DONE -- DOCFIX-136 (addendum 52): tenant AI skill FULL SWEEP vs DOCFIX-130.** Operator asked for a full sweep after the registry gap. Grep-audit found 6 more guide topics absent; all added concise (pointer to guide for detail): resize command, kubeconfig cert-expiry (+openssl read), dashboard-vs-CLI split, no-DNS, NEW Persistent storage section, ServiceAccount pointer; + PVC-Pending and cert-expired troubleshooting signatures. Skill now covers the full DOCFIX-130 set (ingress stays recommendation+pointer by design). Harness 8/8, gauntlet 39/39, package re-instantiated. Committed + pushed. STANDING LESSON: guide changes must sweep the skill in the same change -- skill-lag caught twice (DOCFIX-135/136). - **DONE -- DOCFIX-137 (addendum 53): the standing lesson is now ENFORCEABLE.** repo-lint L8 guide<->skill coupling guard: clientdocs/guide-skill-coupling.txt pins the guide's sha256; any guide change FAILS lint (naming the skill files to review) until re-recorded with `bash scripts/repo-lint.sh --record-guide-skill-coupling`. Harness +5 (28/28), gauntlet 39/39. So the next guide edit CANNOT land without a conscious skill review. Skill-lag closed at the source. - **S-CLASS BATCH STATUS (operator ruled 2026-07-08):** DONE = S7, S4, S8a (DOCFIX-132/133/134). **S8b APPROVED -- DO IT (next actionable):** centralize KEYSTONE_VIP into lib-net.sh. **DONE (DOCFIX-138):** KEYSTONE_VIP_DEFAULT in lib-net.sh; all 6 sites reference it (3 forms: KEYSTONE_VIP, KEYSTONE_HOSTPORT, acceptance inline); lib-net source added where absent. Gauntlet 39/39, lint 0 fail. **S8 FULLY DONE (S8a+S8b).** **S8c RULED WON'T-DO** (operator: marginal value for as-built constants vs bash<->python emit/parse machinery). S1/S2/S3/S5 (lib-validate) STILL HELD -- operator keeps DOCFIX-126/127 live-verify open for an upcoming opportunity (devteam actively using resources; not now). S6 no-action. - **quota-vs-capacity (addendum-40 d): CLOSED -- operator ACCEPTS as-documented** (DOCFIX-130 sec 2.3/5.5 + skill cover NoValidHost/capacity thoroughly; no quota trim). - **StorageClass follow-up (DOCFIX-130 sec 5.5): RESOLVED 2026-07-08.** Operator authorized full read of devteam's cluster (fully internal domain); scoped via ~/devteam-cluster-openrc (snap needs $HOME for --dir; kubeconfig fetched under $HOME, used, DELETED). devteam TestCluster HAS a default StorageClass: `default (default)` provisioner cinder.csi.openstack.org, Retain, WaitForFirstConsumer, expansion=true. So the delivered template (volume_driver unset) DOES yield PV support via the cinder-CSI magnum default -- persistent storage works out of the box; no template change needed. sec 5.5 doc is correct and self-verifying; OPTIONAL enhancement: state the expected `default` cinder class (would trigger L8 skill sweep). Cleanup: kubeconfig removed, no residue. - **D-076 assessment (operator #6: "tested working w/ devteam domain-admin? check for additional work"):** NOT fully working -- optimistic recollection. Revision 1 FAILED (full-page 403s + forced logout), reverted. Revision 2 PARTIALLY passed: identity panels render for the manager persona + session intact, BUT the create-user GUI flow submits a NON-managed role (load-balancer_admin) whose grant 403s, stranding a role-less user (flow not atomic). ADDITIONAL WORK before it can advance: (1) root-cause why the Role dropdown submitted load-balancer_admin (form default-role preselection vs mis-select); (2) add a per-view SEMANTIC harness test asserting create-user grants exactly `member` (invisible to the existing 50 policy pass/fail assertions); (3) clean up the orphaned test user 9a62f88e331f41adb5c9e41225ec8698 (name `test1`, CONFIRMED still present in devteam, ZERO roles -- delete or grant member); (4) operator RULING: adopt rev2 (a) vs close GUI/CLI-only (b) + accept/reject the charmhelpers blacklist-hole reliance; (5) another operator-present gated window to re-test after rework. NOTE: live override state UNVERIFIED -- juju macaroon expired (needs `juju login`); confirm use-policyd-override is reverted to baseline. Draft in worktree agent-a73ec798 (uncommitted). Clients unaffected: CLI identity path is the documented behavior. - **D-076 PINNED (operator 2026-07-08):** do NOT work D-076 while devteam are running their own tests in the env. Deferred until devteam testing completes; no further D-076 investigation/window until then. Draft stays in worktree. - **test1 orphan SCHEDULED FOR REMOVAL (operator 2026-07-08):** approved to delete; staged, NOT run now. Exact gated command (re-verify role-less at execution): `openstack user delete 9a62f88e331f41adb5c9e41225ec8698` (name test1, devteam domain, 0 roles measured 2026-07-08). Run at the next convenient window; it is a role-less shell with no resources, so removal does not affect devteam's tests. ## REMAINING OPEN WORK (single list for session resume; 2026-07-07, updated post-devteam-window) 1. **Devteam delivery -- last two steps are OPERATOR-MANUAL (everything else DONE, addenda 34-35):** intake collected 2026-07-07 (internal test; placeholder custodians operator-accepted; filed jumphost-local ~/devteam-intake-record.md, NEVER commit); quota applied+verified (10/20/51200, runbook standard); pack instantiated to ~/devteam-package (17 files, tokens verified zero, banners stripped; templates in clientdocs/ untouched -- L7 receipt intact). REMAINING (operator): (a) copy the CA into the package: `cp ~/vault-init/vault-ca-root.pem ~/devteam-package/omega-cloud-ca.pem` (guard correctly blocks agent access to ~/vault-init); (b) deliver ~/devteam-package + the ~/tenant-devteam credential files in-person to the two named custodians. 2. **Horizon contract checks: CLOSED for devteam (2026-07-07 full three-account walkthrough, addendum 38).** 5.1 VERIFIED (login + Identity-only view + expected popups); 5.3 verdict: GUI identity probe FAILS as the contract anticipated -- CLOSED via the CLI-path clause (DOCFIX-120 contract amendment); D-075 TLS-leg redirect CONFIRMED (operator browser). Isolation, keypair asymmetry, per-user app-cred visibility, and OBJECT STORAGE (container create/delete -- first live proof, closes the acceptance-checklist caveat) all verified in-GUI. foil1's own 5.1/5.3 rows: surface now proven on devteam; operator may run or waive foil1's pass (non-blocking either way). 2c. **Dashboard policy override (GUI identity for manager persona): first live apply FAILED acceptance + REVERTED 2026-07-07; rework IN FLIGHT (background agent, worktree).** Zip mechanics + PO: activation worked; panels rendered but page-level checks refused (Domains/ Projects/Users/Groups "not authorized"); Roles panel popped two errors then FORCED LOGOUT. Reverted to baseline in one config cycle (operator-confirmed). Decision draft stays PROPOSED (uncommitted, agent worktree; next-free D claimed at commit). Next: agent rework (per-view rule/target mapping, logout root cause, scope-down to Users+Projects, offline per-view semantic tests), then a second gated apply in a later window. Clients are unaffected: CLI identity path is the documented behavior. 2b. **D-075 RULED (option c) + EXECUTED 2026-07-07** (was: root-autoindex finding): redirect / -> /horizon + Options -Indexes via conf-enabled/99-omega-root.conf on the dashboard unit. First attempt (RedirectMatch) caught leaking the backend port in verify and replaced with per-dir mod_rewrite. Delivered as D-075 entry + DOCFIX-118 (phase-03 PER-REBUILD block + appendix-A symptom entry). PER-REBUILD like the D-044 override. 3. **capi-test-1 CREATE_FAILED record delete: RULED YES (2026-07-07), execution handed to operator** -- the Claude Code permission classifier (correctly) refuses agent-side deletes of resources it did not create; exact command staged in-session (uuid fae7ec70-712c-4e58-88dd-da0d7b8c89b2, measured this window). After it runs: next `tenant-offboard.sh --sweep-magnum-orphans --audit` reaps its trustee. 4. **S-class consolidation + minor R items:** proposals in docs/script-quality-findings-20260707.md -- present options to operator before building (S1-S5/S7/S8 unruled; S6 ruled no-action). 5. **Pinned decisions (do NOT re-ask):** D-071 ratification (pre-DC-DC HA/ backup session); D-068 off-EOL path (Roosevelt-scale); SEC-001/003/004 at v1 close; d011-06 stays MANUAL until SEC-003 rehearsal (gates D-011 FULL close; v1-close checklist QUEUED per handoff section 6). 6. **Cross-stream:** main-chat to reconcile its ledger section against addenda 27-33 (its backlog items 6/7/8 delivered here; its Verify-live queue fully cleared; D-050/D-073/D-074 all resolved). - **Horizon three-account walkthrough + docs alignment (2026-07-07 cont., addendum 38):** operator live-tested all three devteam accounts in the GUI; every observation contract-correct (details in item 2 above). DOCFIX-119: clientdocs aligned to observed GUI reality (5 files by background docs agent -- first-login expectations, CLI user-creation commands, popup signatures, object-storage row de-hedged + acceptance script aligned; harnesses 8/8 + 47/47; L7 receipt re-recorded) + README instantiation guidance (DASHBOARD_URL includes webroot). DOCFIX-120: contract section 5 amended with live verdicts (5.1 verified; 5.3 closed-by-CLI-clause; stage-7 canary marked IMPLEMENTED). Package RE-INSTANTIATED to ~/devteam-package from updated templates (incl. /horizon URL); OPERATOR: re-copy to local + deliver. Dashboard override attempt/revert logged in item 2c. - **devteam intake + quota + pack instantiation (2026-07-07 cont., addendum 35):** intake walkthrough with operator (7 sections; CIDR + short name pre-settled by rulings); quota set devteam-prod 10/20/51200 (values equal nova class defaults -- set explicitly per runbook "documents the record"); first-ever clientdocs pack instantiation: 17 files to ~/devteam-package via scratchpad script (9 tokens: endpoints MEASURED in-session -- AUTH_URL https://10.12.4.50:5000/v3, RegionOne; dashboard https://10.12.4.58 carried from D-072 as-executed verification). Guard/classifier blocks honored: no ~/vault-init access (CA copy handed to operator), no curl -k probe. LOGGED-NOT-ACTIONED (DOCFIX candidate): tenant-skill/references/day2-operations.md line ~7 explains the {{THIS}} notation in DELIVERED prose, so the literal token survives instantiation; reword at next clientdocs sweep (L7 receipt re-record required). - **devteam ONBOARDED (2026-07-07 window ops-devteam-onboard, addendum 34):** stages 0-4 + tenant-assert (15/15 PASS) + stage7 canary (SSH via FIP 10.12.7.43, teardown clean) ALL FIRST-TRY on TENANT_CIDR=10.100.0.0/24 (D-074 ruled fallback -- first live use of the fallback path and first client-facing workflow execution; ZERO as-executed corrections needed). Stage-2 anti-escalation guard (DOCFIX-112 batch) first live client run: admin grant denied as designed. Baseline reused per ruling (asbuilt/20260706-224851, not stale). Open cloud-assert PASS (A5-A7 held on first run -- one-shot shell lacked admin scope, known jumphost trait, re-run sourced) and close cloud-assert PASS. Handover creds staged in ~/tenant-devteam (0600, jumphost-only). Pack instantiation HELD on intake answers (item 1 above). capi-test-1 delete handed to operator (item 3). - **CLIENT PACKAGE RELEASED (addendum 31, 2026-07-07):** operator-requested polish sweep integrated (DOCFIX-110) + banner ruling applied (DRAFT removed; the three per-client templates keep TEMPLATE; README says RELEASED-for-handover). Package is handover-ready pending only per-client instantiation. In flight: repo-wide script quality sweep (report-only agent; findings to operator triage next). - **CLIENT PACKAGE COMPLETE (addendum 30):** starter kit (DOCFIX-108, 6 deliverables incl. tenancy-audit from the skill delivery; pipeline-smoke + leftover-check deduped away) + tenant AI skill omega-cloud-tenant (DOCFIX-109, leakage-gated) integrated; L7 coverage recursive; receipt 29 files; gauntlet 37/37. REMAINING before handover: operator-requested polish pass over the final clientdocs set (in flight), then devteam onboarding next session instantiates the pack. - **D-074 EXECUTED (addendum 29):** operator ADOPTED the main-chat CIDR correction PLAN same-day (overlap-allowed tenant CIDRs; client-choice + 10.100.0.0/24 fallback; capi-mgmt stays reserved; L5 print removed). Phase 0 gates honored: allow_overlapping_ips=True measured; overlap+Magnum proof via foil1 (EXACT 10.20.28.0/24 overlap with beta; cluster ~14.5 min; 2 Ready nodes via API-LB while beta served the same CIDR); teardown clean; cloud-assert PASS. Guard rewritten (DOCFIX-106, net_overlap.py, harness 12->17), docs aligned (contract/runbook/intake), D-016 amended, repo_lint L5 next-free removed (DOCFIX-107). FINDING-doc BUNDLEFIX token de-fanged (counter was inflated to 052; restored 012). Phase 3 (devteam onboarding, fallback CIDR) remains NEXT SESSION. Corrections fed back to main-chat via addendum 29: northwind example fictional; runbook filename wrong in PLAN. - **INTEGRATED (addendum 28):** all six worktree drafts -> main as DOCFIX-098 (H4 drift detector), 099/100 (offboard orphan sweep + Phase E0 guard), 101 (clientdocs handover pack), 102 (cloud-snapshot.sh -- unblocks the devteam snapshot gate), 103 (tenant-acceptance harness + the last stderr-inventory fix), 104 (lint L7 sweep-on-change ENFORCED; receipt live, 17 files). Gauntlet 35/35 on the integrated tree. Main-chat backlog items 6/7/8 are now DELIVERED (by this stream, operator-assigned) -- main chat to reconcile its section. - **devteam gate SATISFIED (2026-07-06):** cloud-snapshot first live runs both green -- asbuilt/20260706-224851 committed (12 redactions verified; scrubber caught a real embedded CA private key) + 1.1GB controller backup jumphost-local (~/openstack-baseline/, NEVER commit). Operator ruled: devteam onboarding NOT this session -- next session runs stages 0-4 + tenant-assert + canary on 10.20.30.0/24 (recommended, unruled), then instantiates the handover pack with dev-team intake answers (7 questions in the DOCFIX-101 entry). Also still open: foil1 Horizon manual checks (contract 5.1/5.3). ## Jumphost stream -- session 2026-07-09: new repo `openstack-caracal-dc-dc` first load (repo-rename sweep) - **Environment note:** this session's shell is Windows (win32), not the Linux jumphost (`vopenstack-jesse`) CLAUDE.md describes -- flagged to operator. `run-tests-all.sh` gauntlet on this box shows 23/39 pre-existing failures (missing `jq`, tty/exit-code quirks) unrelated to any change made this session; `repo-lint` (the only suite touched) is ALL PASS standalone and in the gauntlet. Confirm actual jumphost behavior at next Linux session. - **DONE -- DOCFIX-139 + DOCFIX-140 (changelog `docs/changelog-20260709-dc-dc-repo-rename-sweep.md`):** repo-rename sweep (stale `openstack-caracal-ipv4` path/URL references repaired to `openstack-caracal-dc-dc` across the skill, 4 runbook RUN blocks, and the repo_lint.py docstring; clientdocs L7 receipt re-recorded after confirming no client-facing drift) + a real repo_lint.py bug fix (L1 Windows path-separator: the `docs/design-decisions.md` legacy-carve-out comparison used `str(p.relative_to(R))`, which never matches on Windows, so the documented WARN silently became a FAIL -- fixed to `.as_posix()` at all 3 affected sites; harness gained T3b covering the carve-out path directly, 29/29 green). NOT actioned (flagged, larger scope): README.md and environment.md's body content still describe VR0 DC0/v1 single-DC scope wholesale -- awaiting operator direction on whether/when to rewrite for VR1 DC-DC. Uncommitted -- operator has not yet asked to commit. - **DONE -- DOCFIX-141 (changelog `docs/changelog-20260709-repo-agnostic-sweep.md`):** operator asked to scope the "make scripts repo-agnostic" idea as its own sweep + a lint rule, following directly from the DOCFIX-139/140 experience. New repo-lint check L9 (self-reference guard) FAILS any script/RUN-block/ skill file that hardcodes the current repo's own directory name; added with a 5-case harness (34/34 green). Along the way, narrowed `all_text()`'s `.claude` exclusion from "skip all of `.claude/`" to "skip only `.claude/worktrees/`" -- the blanket exclusion meant checked-in `.claude/skills/` and `.claude/hooks/` content was invisible to lint entirely (why the SKILL.md hardcode from DOCFIX-139 wasn't caught). Established a `$REPO` session-variable convention (documented in `runbooks/README.md` Conventions) and converted the 8 remaining hardcoded clone-path references (SKILL.md + 4 runbooks) to require it explicitly (fail loud if unset, no inferred fallback, per hard rule 2). environment.md's repo URL is KEPT (documented anchor fact, same as its other facts) but tagged with the new `repo-lint: allow-selfref` opt-out. LOGGED, NOT ACTIONED: L3/L4/L6/L9 only recognize triple-backtick fences for "in code" detection; `d011-batch3-window-DRAFT.md` (indented pseudo-code) and `tenant-onboarding-v2-DRAFT.md` (BEGIN/END block comments) use different conventions and are invisible to these checks -- a pre-existing gap, not introduced here, DOCFIX candidate (teach the checks the other conventions, or standardize runbooks onto fences). Uncommitted -- operator has not yet asked to commit. - **DONE -- workflow doc + gap register (2026-07-09):** `docs/dc-dc-deployment-workflow.md` added (Stage 0-7 followable checklist + tooling gap register audited directly against the repo) + a companion visual Artifact (https://claude.ai/code/artifact/e752f6dd-23ff-4c59-9ce5-1dedb917e5ea). COMMITTED + PUSHED (374aef5). - **DONE -- DOCFIX-142 (changelog `docs/changelog-20260709-opentofu-scaffold.md`):** first OpenTofu content in the repo (gap register item 2). `opentofu/` network + storage-pool scaffold: `dc-planes` (six per-DC planes), `mesh-link` (D-100 dark-fiber triangle), `dc-storage-pool` modules; DC1 + Office1 wired, DC2 held back pending its CIDR assignment (D-101). Provider schema (`dmacvicar/libvirt` v0.9.8) verified against its actual current docs this session, not memory -- differs materially from older common examples (network isolation is `forward.mode`, nested, not top-level `mode`; domain resource restructured to ~40 args mirroring raw libvirt XML). Node-VM/volume resources, OPNsense, DC2 planes, and `tc netem` application DELIBERATELY DEFERRED -- schema too unfamiliar to author safely without a real `tofu providers schema` pass; see `opentofu/README.md`. Added `scripts/opentofu-validate.sh` + `tests/opentofu-validate/` (2/2 green -- only the guard-clause paths testable without a `tofu` binary, which is not available anywhere this repo has been worked in this session; the actual fmt/init/validate path is UNTESTED, logged not hidden). Extended `repo_lint.py`'s L1 file-extension set to include `.tf` (previously uncovered since no Terraform/OpenTofu files existed). repo-lint 34/34 harness green, 0 fail/1 documented warn. Uncommitted -- operator has not yet asked to commit this piece. - **NEXT ACTIONABLE STEP (logged):** on a machine with the `tofu` binary + registry network access, run `scripts/opentofu-validate.sh`, then `tofu providers schema -json` against the initialized provider, and read the real `libvirt_domain`/`libvirt_volume` schema directly before authoring the node-VM module. - **DONE -- OPNsense gap detail (2026-07-09):** gap register item 4 split into its 4 separable sub-gaps (network substrate done; VM/image blocked on the domain module; OPNsense's own routing config not started, no mechanism chosen; `tc netem` a separate OS-level thing blocked on an unruled D-100 parameter). Mirrored into `opentofu/README.md` + the visual tracker. COMMITTED + PUSHED (0bb6cff). - **DONE -- DOCFIX-143 (changelog `docs/changelog-20260709-repo-lint-crlf-fix.md`):** fixed the CRLF `write_text` bug flagged during the DOCFIX-141 commit -- `--record-clientdocs-sweep`/`--record-guide-skill-coupling` now `write_bytes()` with an explicit ascii-encode instead of relying on `.gitattributes` eol=lf to clean up CRLF after the fact. Verified 0 CR bytes on a live re-record; harness 34/34 unchanged; repo-lint 0 fail/1 warn. COMMITTED + PUSHED (f93dd0d). - **DONE -- DOCFIX-144 (changelog `docs/changelog-20260709-opentofu-node-vm.md`):** moved into the domain module as requested. Built `opentofu/modules/node-vm` (blank-disk PXE-boot MAAS-managed node VM, D-103's pattern for Stage 3) -- this time from the provider's own real example `.tf` files, not doc-summarization alone, after last session deferred `libvirt_domain` for exactly that confidence gap. **Found and fixed a real bug in the already-committed DOCFIX-142 delivery in the process:** `dc-planes`/ `mesh-link`/`dc-storage-pool` used classic HCL block syntax (`domain { ... }`) where the provider's current schema actually needs attribute-style objects (`domain = { ... }`) -- confirmed once real provider examples surfaced the convention; doc-summarization alone had NOT caught this. All three corrected; `opentofu/README.md`'s schema notes now state the attribute-object convention as a provider-wide assumption. UNVERIFIED, flagged: the per-device `boot`-order attribute's exact shape (an inference by pattern). Still deferred: the cloud-init/base-image VM pattern (Office1 services + OPNsense), DC2 planes, netem. COMMITTED + PUSHED (3324387). - **DONE -- DOCFIX-145 (changelog `docs/changelog-20260709-opentofu-cloudinit-vm.md`):** built `opentofu/modules/base-image` + `modules/cloudinit-vm` (the OTHER VM pattern, for Office1's own service VMs -- MAAS-region, NetBox, GitBucket). Same high-confidence source as DOCFIX-144 (`examples/alpine_cloudinit.tf`, fetched as real code). Threaded the base image's `.path` output across the module boundary as a real attribute reference rather than reconstructing a volume path from pool+name strings -- a cleaner design than the earlier draft, avoiding an unconfirmed-convention guess entirely. `user_data`/`meta_data`/`network_config` are required inputs with no default (Office1 VM configs aren't designed yet; a plausible DHCP default would also silently fail since these planes carry no libvirt DHCP). **OPNsense applicability explicitly NOT confirmed** -- FreeBSD-based, cloud-init/NoCloud support not the same guarantee as Linux images, flagged prominently rather than assumed. Neither module instantiated (no image source chosen for any service VM). Workflow doc + visual tracker updated. Uncommitted -- operator has not yet asked to commit this piece. - **DONE -- OPNsense deployment research (changelog `docs/changelog-20260709-opnsense-research.md`, no DOCFIX -- research delivery, matches the design-doc precedent):** operator asked to check OPNsense's actual documentation. Confirmed cloud-init is genuinely unreliable on OPNsense (FreeBSD; OPNsense's own forum consensus), closing the DOCFIX-145 open question. Found the real mechanism instead: the Configuration Importer (official docs) scans an attached volume for `/conf/config.xml` in a 2-3s boot window; ISO9660 support was added to it specifically for VM/cloud automation (a closed, milestone-targeted, core-dev-assigned GitHub issue) -- mechanically identical to `cloudinit-vm`'s cdrom-attach shape, different payload. Use the nano image for KVM (pre-installed, serial-ready), not the installer images. Fully sourced in `opentofu/README.md`. NOT built yet: a decompress/convert preprocessing script for the nano image, an ISO9660 config.xml-seed builder, and the real config.xml content -- itemized as the next actionable step. COMMITTED + PUSHED (75ca823). - **DONE -- DOCFIX-146 (changelog `docs/changelog-20260709-opnsense-edge-module.md`):** operator asked to continue with OPNsense while the research was fresh. Built all three itemized next-steps: `scripts/opnsense-prep-image.sh` (nano image decompress+convert, no hardcoded mirror -- `OPNSENSE_MIRROR_BASE` required), `scripts/opnsense-build-config-iso.sh` (ISO9660 config.xml seed), and `opentofu/modules/opnsense-edge` wiring both together -- mechanically identical to `cloudinit-vm`'s cdrom-attach shape, no `libvirt_cloudinit_disk` involved. Explicit `lan_network_name`/ `wan_network_name` variables (not an ordered list) to prevent a silent LAN/WAN swap. Both harnesses green (4/4, 3/3) exercising REAL missing-tool guards (`qemu-img`/`genisoimage`/`xorriso` are all genuinely absent this session). Two things flagged, not presented as fact: whether the Configuration Importer's ISO9660 behavior holds on a real boot, and whether `devices.interfaces` list order reliably maps to LAN/WAN NIC enumeration for this guest. `config_iso_path` has no default -- real config.xml content per site is still undesigned. Not instantiated in root `main.tf`. `opentofu/README.md` + workflow doc + visual tracker updated to reflect BUILT (not just researched). COMMITTED + PUSHED (a3ff07f). - **DONE -- DOCFIX-147 (changelog `docs/changelog-20260709-opentofu-maas-netem.md`):** operator asked to pull docs and validate before drafting the next modules -- the two remaining unbuilt mechanisms from the OpenTofu scope (MAAS VM-host registration, tc netem application). Researched a real official MAAS provider (`canonical/maas` v2.7.2) and OpenTofu's own `local-exec`/`terraform_data` docs BEFORE writing any code, per instruction. Caught a real design mistake before it happened: read both `maas_vm_host` (register) and `maas_vm_host_machine` (compose -- takes cores/memory/disks as INPUTS, MAAS's "Compose machine" pod feature) schemas, cross-checked against D-103's explicit "MAAS does NOT compose new ones" ruling, and built `modules/maas-vm-host` using ONLY `maas_vm_host` -- the other resource would have had MAAS and `node-vm` fighting over VM creation. Also confirmed `local-exec` runs on the machine invoking `tofu apply` (Office1 per D-103), not the vcloud host where bridges live -- `modules/netem-link` wraps its `tc qdisc` command in an explicit SSH hop because of this, using `terraform_data` (OpenTofu's own recommended replacement for `null_resource`) with a destroy-time provisioner to clean up. Neither module instantiated (MAAS zone/pool, vcloud SSH target, and real netem parameters all still pending). `opentofu/README.md` gained a new sourced research section; workflow doc + visual tracker updated. COMMITTED + PUSHED (ea276c5). - **DONE -- DOCFIX-148 (changelog `docs/changelog-20260709-opentofu-audit.md`):** operator asked to check back over all current modules, pull docs for anything drafted without it, and review for errors/assumptions. Grepped every "UNVERIFIED"/"inferred"/"assumed" marker across all 9 modules and researched each further rather than re-stating hedges unchanged. **Found and fixed a genuine documentation error, not just a hedge:** `opnsense-edge`'s comments claimed interface list-order directly sets OPNsense's LAN/WAN role -- researching OPNsense's own interface-assignment model showed role assignment is a SEPARATE, explicit config.xml mapping, not automatic from device order ("vtnet0=WAN" is a convention some guides use, not an enforced rule). Comments + both LAN/WAN variable descriptions corrected; no config.xml has been written yet so this fixes documentation accuracy for whoever writes it next, not current runtime behavior. Confidence UPGRADED (not fully resolved) on three items: `node-vm`'s `boot`-order shape (confirmed the provider 1:1-mirrors libvirt's own XML, and libvirt's own docs confirm ``'s single-attribute shape), `create.content.url` accepting local paths (second independent source), and the `genisoimage` flags (confirmed standard, cleanly separated from the still-open "does OPNsense's Importer actually read it" question). Confirmed safe: `maas_vm_host`'s `zone`/`pool` computed-if-unset behavior. `opentofu/README.md` gained a new "Audit pass" section. Deliberately did NOT re-research already-well-sourced findings (Importer mechanism, vm_host-vs-vm_host_machine, the syntax-bug fix) -- targeted only real gaps. Also added (same delivery): `.claude/skills/openstack-cloud-ops/references/ opentofu-provider-docs.md` -- indexes every provider/doc source used building `opentofu/` with exact URLs, confirmed versions, default branches, AND the fetch methodology that actually worked this session (Registry doc pages are JS-rendered, use GitHub instead; branch names vary, check `default_branch` via the API; real example `.tf` files beat doc-summarized prose for syntax questions). New SKILL.md routing entry; cross-referenced from `opentofu/README.md` in both directions, no content duplicated (README.md stays the single source of truth for per-module findings). COMMITTED + PUSHED (96ecbad). - **DONE -- DOCFIX-149 (changelog `docs/changelog-20260709-opnsense-config-design.md`):** operator asked to move into designing the real config.xml content. Fetched OPNsense's own real, currently-shipped `config.xml.sample` (`opnsense/core` repo) and its `Route.xml` static-routes model before drafting. Built `opentofu/templates/opnsense-config.xml.tmpl` (`{{TOKEN}}`-parameterized, reusing this repo's existing clientdocs convention) + `scripts/opnsense-render-config.sh`. The real sample fetch DIRECTLY CONFIRMED the DOCFIX-148 audit finding with an actual example: ships literal placeholder device names inside each interface's own `` element, proving LAN/WAN role assignment really is the explicit per-block mapping DOCFIX-148 concluded. **The renderer is tested END-TO-END, not just guard clauses** (8/8 -- it needs no external tool, a first among the opnsense-* scripts) -- and the harness caught a REAL bug before it shipped: the token `HOSTNAME` collides with bash's own built-in `$HOSTNAME` variable (`unset` doesn't actually clear it), silently passing a test that should have failed. Renamed to `OPNSENSE_HOSTNAME` throughout. Full token legend in `opentofu/templates/README.md`, marking exactly which values are real (NTP pool, D-106 naming) vs. still pending Stage 0 ratification (D-100/D-101/D-107) vs. only measurable on a real boot (`vtnetN` assignment) vs. a security requirement (root password hash must be freshly generated, never the stock sample's own shipped default). `opentofu/README.md` + workflow doc + visual tracker updated. Uncommitted -- operator has not yet asked to commit this piece. - **DONE -- Stage 0 (decision ratification) RATIFIED 2026-07-09 (changelog `docs/changelog-20260709-stage0-ratification.md`, DOCFIX-150):** operator walked through the 6 buildout-design redline items live (AskUserQuestion, 2 rounds): Ceph size=3-by-default (size=2 only as an explicit Phase-0 fallback); netem same-metro lean; D-102 folds into D-101, D-109 stays standalone; metal-admin GAINS a ULA leg (reverses the stated lean -- a real deviation, not a rubber-stamp); COS per-DC-only confirmed; mirror sync topology independent-per-DC confirmed. All 11 of D-100..D-110 individually flipped PROPOSED -> ADOPTED in `docs/design-decisions.md` (D-102's content preserved under a MERGED-INTO-D-101 status line, per the append-only discipline -- not deleted). `docs/dc-dc-buildout-design.md` Section 10 rewritten from "open items" to a ruling record; Section 3's Ceph gate and the stray "Section D-102" cross-reference corrected to D-101. **Verified via `ledger-scan.sh`: D-100..D-110 no longer appear under PROPOSED/OPEN** -- only D-068/D-071 remain (separate, pre-existing, unrelated to this gate). repo-lint 0 fail/1 documented warn, harness 34/34. Stage 0 of `docs/dc-dc-deployment-workflow.md` is CLEARED. - **Operator context, 2026-07-09 evening:** operator stepped away from the workstation for the night, granted blanket git permission for the session, and authorized autonomous subagent use to make maximum progress toward an end-to-end-ready project by morning. Operator clarified this session runs from the WORKSTATION, not the vopenstack-jesse jumphost -- tonight's scope is PREP ONLY (docs, scripts, tests, hardened runbooks); no live infra (juju/MAAS/OpenStack APIs) is reachable or in scope tonight. Operator will switch to the vcloud host in the morning to run actual installs/deployments. This explains (and validates) why `run-tests-all.sh` shows 23 pre-existing failures in this environment -- all either "jq required" (not installed on the workstation) or live-cloud-dependent suites (`cloud-assert`, `preflight`, `tenant-*`, etc.) -- confirmed via `git stash` A/B to be identical with and without this session's own changes, i.e. an environment gap, not a regression. - **DONE -- Tooling gap register #1 CLOSED 2026-07-09 ($DC selector convention, changelog `docs/changelog-20260709-dc-selector-convention.md`, DOCFIX-151):** added `lib_net_select_dc()` to `scripts/lib-net.sh` (dc0/dc1 no-op per D-101's inherited-layout ruling; dc2 fails loud -- gap #3 still open) and `lib_hosts_select_dc()` to `scripts/lib-hosts.sh` (dc0 no-op; dc1 AND dc2 BOTH fail loud -- no real per-DC host enrollment exists for either yet, an intentional asymmetry vs. the net selector, documented in-line in both files). Backward compatible by construction -- no existing caller of either library is affected. New harness `tests/dc-selector/run-tests.sh`: 21/21 PASS (caught and fixed a real `pipefail`-vs-nonzero-exit-code bug in the harness itself while writing it). repo-lint 0 fail/1 documented warn (unchanged); full gauntlet confirmed via `git stash` A/B to introduce zero new failures. Stage 5's runbook (not yet authored) still needs to actually CALL `lib_net_select_dc "$DC"` / `lib_hosts_select_dc "$DC"` -- the mechanism exists, the call sites don't yet. ## State facts to remember - beta cluster left at **node_count=2** (deliberate; bonus resize acceptance coverage). - repo is temporarily **PUBLIC** for Claude web_fetch (SEC-004) -- flip private at v1 close. - Jumphost stream: see "Active window" section above (ops-update-20260705 in flight). Vault stays 1.8/stable (D-068 unruled). - CORRECTION (2026-07-05, jumphost stream, per handoff reconcile item): the two bullets above are stale -- ops-update-20260705 is CLOSED (fleet fully current, juju 3.6.25); D-068 is RULED on the 1.16 question (1.16 ruled out; vault stays 1.8/stable rev 714; the off-EOL-1.8.8 path remains OPEN under D-068). ## Project-completion (execute after D-011 passes) - Consolidate 10 per-phase do-documents into `docs/v1-deploy-runbook.md`. - Set repo visibility PRIVATE (SEC-004). - v2-deferred: SSH on GitBucket (port 29418), IPv6 dual-stack, NetBox import bundle. ### Cross-session pins (append-only) - D-071 completion sweep (2026-07-05, DONE): the ops-update window closed; D-071 was AMENDED (create-backup premise corrected -- `juju create-backup`/`download-backup` DO exist on juju 3.6; a 902MB controller backup was taken). D-071 REMAINS PROPOSED -- not ratified to ADOPTED. Operator ratification is pending; the single-controller (no-HA) half of the risk is unresolved by backups alone. See D-071 + its 2026-07-05 amendment. - Pre-DC-DC controller HA/backup planning (PINNED): before the Roosevelt (DC-DC) phase, hold a dedicated planning session on Juju controller High-Availability / backup architecture. Backups now exist (per D-071 amendment); controller HA is the open design item. This resolves the single-controller risk recorded under D-071 for production.