Fork: 0
OpenStack / openstack-caracal-ipv4
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-ipv4 / docs / session-ledger.md
@Jesse Austin Jesse Austin 12 hours ago 16 KB addendum 25: DOCFIX-093 batch-3 window do-doc + OpenBao research draft + H5 embed + stderr-sweep inventory
Raw Blame History

Session ledger -- in-flight work continuity record

Purpose. Long ops sessions on this cloud routinely exceed a single context window and get COMPACTED (sometimes several times). Anything living only in the chat scrollback is lost at compaction. This ledger is the durable, committed record of what is in flight, so any session -- after a compaction, or a fresh one, or the parallel Claude Code stream -- can resume without losing pending work.

How to use it (standing practice).

  1. At session start: read this ledger AND run bash scripts/ledger-scan.sh. Reconcile.
  2. ledger-scan.sh is the DRIFT CHECK -- it derives the open-work it reliably can (PROPOSED/OPEN decisions, OPEN security rows, next-free numbers) straight from the repo. This narrative must not claim CLOSED anything the scan shows OPEN, nor omit what it surfaces.
  3. Update this ledger at every deliverable/commit -- it is a standing deliverable like the changelog. The changelog says what CHANGED; this ledger says what is still OPEN.
  4. Numbers and the machine-derived section below are seeded from a scan; re-run the scan and re-seed rather than editing those by hand.

Machine-derived (re-seed from scripts/ledger-scan.sh; do not hand-edit)

As of 2026-07-06 (addendum-20 ruling sweep; re-run the scan to refresh):

  • PROPOSED / OPEN decisions: D-068 (Vault substrate hardening, Roosevelt -- 1.16 ruled out per amendment; off-EOL path OPEN), D-071 (routine update cadence + controller patch policy -- operator to rule; gated on the pre-DC-DC HA/backup session). (D-050 RESOLVED/CLOSED 2026-07-06 via D-051; D-073 filed ADOPTED, implementation pending -- not scan-tracked as open.)
  • OPEN security rows: SEC-001 (rotate at v1 close, re-ruled 2026-07-06), SEC-003 (custodians + second-person unseal -- DEFERRED; keeps d011-06 MANUAL), SEC-004 (flip repo to private at v1 close -- deferral reaffirmed).
  • Next-free numbers: D = 074, DOCFIX = 094, BUNDLEFIX = 012. (Scan and the addendum-25 changelog next-free pointer agree.)

Active build (this session)

  • validate.sh D-011 runner -- modular. Foundation committed: scripts/lib-validate.sh (exit contract 0/1/2/3/4, emit, vr_json stderr-safe, run, env hygiene, disruptive gate) + scripts/validate.sh orchestrator (profiles, verdict, --stop-on-fail, --include-disruptive).
    • Batch 1 committed: checks/d011-01-charms, checks/d011-06-vault-unseal (MANUAL until the SEC-003 rehearsal closes).
    • Batch 2 committed: checks/d011-02-vip-jumphost (catalog-derive public origins, TLS+HTTP), checks/d011-03-vip-tenant (ephemeral agnhost pod egress -> keystone VIP, D-035 proof). post-restart profile now fully populated.
    • Batch 3 DRAFTED (mock-tested only, NEEDS LIVE VALIDATION): d011-04-octavia-lb (RR+member additive; amphora failover --disruptive, N+1 headroom-guarded) + d011-05-magnum-e2e (wraps tenant-acceptance + timing). full-d011 profile now COMPLETE.
    • Next: live-validate batch 3 (see Verify-live queue), then item-3/#5-#8 backlog.
    • D-011 AMENDED bar: 1 charms; 2 VIP jumphost; 3 VIP tenant; 4 octavia RR/failover/recovery; 5 magnum e2e + OCCM; 6 second-person manual unseal (attestation, D-069); 7 DROPPED (D-070 supersedes D-012 snapshots); 8 DROPPED (D-019, no Designate).

Script backlog (changelog-tracked)

  • DONE: tenant-offboard.sh, vault-kv-health.sh, cloud-assert.sh (item 3 largely), validate foundation + batch 1.
  • item 3 remainder: fold vault-kv-health.sh into cloud-assert as a section; use the combined gate to regenerate the stale restart-procedure runbook.
  • item 5: scripts/tenant-cluster-create.sh -- generalize the t2-02 stage6 + watch + D-066-evidence wrapper. Not started.
  • item 6: scripts/keystone-policy-drift.sh -- D-051 base_* alignment (LIVE-READ PENDING). Not started.
  • item 7: scripts/cloud-snapshot.sh -- juju-export baseline capture (D-070 retired KVM snapshots but KEPT the export/inventory baseline). Not started.
  • item 8: tests/tenant-acceptance/ harness -- pending (tenant-onboard harness landed).

Register / operator-gated (see docs/handoff-20260703-open-items.md)

  • R-1 / SEC-003: Vault unseal-key custodian assignment (OPEN). Gates V-2 and closes d011-06-vault-unseal (currently MANUAL). Operator input, recorded off-repo.
  • V-2: second-person unseal rehearsal (blocked on R-1).
  • R-2 (D-043 caveat): capi-mgmt-v2 auto-resume exclusion -- rule or accept.
  • R-3: was D-063 -- now CLOSED. The handoff doc still lists R-3 OPEN; update it.
  • D-1: Pattern-A full redeploy (VR0 DC0).

Verify-live queue (batch 3 drafts -- confirm on real cloud before trusting)

  • d011-04 amphora_headroom() field parsing (server show .flavor; hypervisor list --long vcpu/ram) -- safe default HOLDs on any parse miss, but confirm the OK path live.
  • d011-04 OCCM LB-name-contains-service-name assumption; agnhost /hostname round-robin behavior.
  • d011-05 needs a FOIL tenant (2nd tenant) for P3 isolation -- onboard one or set VR_FOIL_APPCRED (acme, the former foil, was offboarded).
  • Sequence: run d011-04 non-disruptively first (RR+member), then --include-disruptive once headroom confirmed.

Logged-not-actioned (small; would vanish at compaction)

  • offboard v2 --sweep-magnum-orphans mode (orphan per-cluster trustee in the magnum domain).
  • offboard stage-3 app-cred idempotency re-run guard.
  • host_href=None barbican observation (exercised fine in stage 6; probably closeable).
  • DOCFIX candidate: sweep repo for other -f json ... 2>&1 stderr-merge hazards (DOCFIX-085 class).
  • D-050 (keystone policyd override) and the D-063-adjacent identity:list_trusts="" hardening remain PROPOSED/OPEN, not actioned.

Jumphost stream -- ops-update-20260705: WINDOW CLOSED 2026-07-05 (addenda 13-16)

First execution of runbooks/ops-update-procedure.md (DOCFIX-086; addendum 13). Logged session ops-update-20260705; checkpointed here at each milestone for cross-stream sync. State as of the Section 2 entry checkpoint:

  • DONE Section 1 (pre-flight): client 3.6.25 / controller 3.6.24 / all 91 agents uniformly 3.6.24 -> target 3.6.25. Worklist = 17 apps. Quiesce clean. Pre-change BOM asbuilt/20260705-102951 committed on a TRUE cloud-assert PASS (required the DOCFIX-087 A7 false-negative fix, addendum 14 -- window was HELD on the operator's Option B ruling until it landed).
  • DONE Section 2.1: controller state backup via juju create-backup -m admin/controller (902MB, checksummed, ~/openstack-baseline/, jumphost-only). NOTE: controller model is admin/controller, NOT /controller.
  • DONE Section 2.2/2.3: controller upgraded 3.6.24 -> 3.6.25 (blind window ~1 min, as expected); post-controller cloud-assert PASS.
  • DONE Section 3: all 91 openstack-model agents + controller machine at 3.6.25, states clean (63 idle / 28 started), nothing stuck.
  • DONE Section 4 G0: keystone 778->817; token probe OK; cloud-assert PASS.
  • DONE Section 4 G1 (11 apps, standing group approval, serial+gated): placement 125->154, nova-cloud-controller 795->823, neutron-api 650->710, neutron-api-plugin-ovn 178->215, glance 642->681, glance-simplestreams-sync 124->152, octavia-diskimage-retrofit 196->232, cinder 733->820, cinder-ceph 533->568, barbican 209->265, barbican-vault 75->99. All probes OK; boundary cloud-assert PASS.
  • DONE Section 4 G2: octavia 441->542; LBs 2/2 ACTIVE/ONLINE; cloud-assert PASS.
  • DONE Section 4 G3: dashboards 728->750 / 59->122 / 120->168; D-044 override INTACT; HTTP+login healthy. RCA: VIP HTTPS dead SINCE DEPLOY (haproxy 443 backend targets vhost-less internal addr; L4 check masks; phase-03 3.3 check fails open) -- NOT a G3 regression; G3 stands (operator ruling; addendum 15). Upstream bug + 2 DOCFIX candidates queued post-window.
  • DONE (coordinated): vault bundle revert 1.16->1.8/stable (BUNDLEFIX-010) + D-068 AMENDMENT recorded; 1.16 ruled out (certs V0/V1, Raft-only, Ceph, BUSL); "off EOL 1.8" remains OPEN. Evidence: docs/D-068-vault-1.8-vs-1.16-analysis.md.
  • DONE Section 4 G4: nova-compute 827->894; hypervisors up; guests byte-identical pre/post.
  • DONE Sections 5-6: post cloud-assert PASS; post BOM asbuilt/20260705-194617 committed; version coherence exact (91 agents @ 3.6.25); BOM diff = 17 expected rev pairs + 1 explained metadata delta (cinder secrets-storage endpoint); appendix-B B.1 fully re-baselined; runbook as-executed corrections DOCFIX-088; D-071 amended (backup exists). WINDOW COMPLETE.
  • DONE follow-on (addendum 17): magnum 70->96 + vault 372->714 refreshed after download-and-diff analysis (mass-rebuild campaign upstream; vault stayed reactive/V0, handlers byte-identical, never sealed). Fleet FULLY CURRENT (zero can-upgrade-to); appendix-B re-baselined from asbuilt/20260705-201118.
  • DONE (addendum 18): D-072 ADOPTED+EXECUTED -- dashboard VIP TLS repaired (cluster binding -> metal-admin; BUNDLEFIX-011 durable; https 200 first time on this build). DOCFIX-089 phase-03 3.3 fail-closed + appendix-A entry. Upstream bug DRAFTED: docs/upstream-bug-draft-dashboard-tls.md.
  • POST-SESSION QUEUE (operator): submit the upstream bug + record LP number in D-072; rule on D-071 (still PROPOSED); Vault off-EOL path (Roosevelt-scale, 1.16 ruled out per D-068 amendment; in-channel 714 applied).
  • Window findings logged for close-out (do not action mid-window):
    1. magnum can-upgrade-to anomaly REPRODUCED (points at magnum-dashboard-122; evidence ~/openstack-baseline/magnum-can-upgrade-anomaly-20260705.json) AND Charmhub magnum 2024.1/stable latest release (r101, 2026-07-01) is s390x-ONLY -- no valid amd64 target; magnum EXCLUDED; report upstream to OpenStack Charmers.
    2. vault NEWLY offers in-channel 372->714 on 1.8/stable -- NOT refreshed (D-068); logged only.
    3. juju create-backup/download-backup EXIST on juju 3.6 -- authoring assumption ("removed in 3.0, expected absent") is WRONG; correct ops-update-procedure 2.1 and the D-071 risk section at window close (DOCFIX candidate).
    4. juju status --format=line omits workload messages on 3.6 (root cause of the A7 false negative; fixed by DOCFIX-087).

Jumphost stream -- session 2026-07-05 (post-window resume)

  • Orientation + reconcile DONE: machine-derived block re-seeded from today's scan (DOCFIX next-free 090 -> 091 was stale); shared-section stale-facts correction appended per the handoff reconcile item. Fences validate 4/0/0.
  • FINDING (logged, not actioned): repo_lint.py L5 "next-free identifiers" info line counts ANY textual mention of D-/DOCFIX-/BUNDLEFIX-NNN across all files, so it is inflated by the deliberate high-numbered decoys in the tests/ledger-scan fixture ("must NOT inflate") and by next-free pointer lines themselves. ledger-scan (heading-based, pointer-excluding) is the numbering authority and is correct. DOCFIX candidate: make L5 reuse ledger-scan's exclusion rules or drop the line. No number consumed. NOTE (2026-07-06 correction): the first version of THIS bullet quoted the decoy tokens literally and thereby inflated ledger-scan's own DOCFIX/BUNDLEFIX next-free (docs/ prose is scan-scanned; only "next-free" lines are excluded). Tokens de-fanged; standing rule: never write identifier-shaped tokens above the real high-water mark in docs/ or runbooks/ prose.
  • Standing by for batch-3 live validation support (verify-live queue).

Jumphost stream -- session 2026-07-06 (operator ruling sweep; addendum 20)

  • RECORDED: D-050 RESOLVED/CLOSED via D-051 (operator ruling); D-073 ADOPTED (identity:list_trusts tightened to modern keystone default -- IMPLEMENTATION + TESTING PENDING, nothing live yet; proposed sequencing: before batch-3 d011-05); D-072 amendment (upstream dashboard-TLS bug WITHDRAWN -- operator judges site misconfig, draft retained); SEC-001 -> rotate at v1 close; SEC-003/004 deferred.
  • PINNED (operator): D-071 ratification and D-068 off-EOL path choice held until A1/A2 (batch-3 entry) complete. Magnum can-upgrade-to upstream report pinned.
  • A1 foil-tenant onboarding APPROVED in principle; tenant-structure model under discussion first (SCS Domain Manager persona confirmation + operator/tenant responsibility split). A2 (d011-04 disruptive gating) held behind A1.
  • IN DISCUSSION (B2): OpenBao evaluation; D-068 item 2 (vault_url cleartext http); D-068 item 3 (AppRole TTL audit + proactive probe) scoping.
  • DELIVERED (addendum 21): DOCFIX-091 docs/tenant-onboarding-contract.md per A1 rulings 1-4 (boundary, roles, intake, dangers, hardening register H1-H5 -- proposals logged, none implemented). A1.4 interpretation pending operator confirm: "SSH access into their domain" read as canary access-proof (H3). Verify-live additions: Horizon manager-role GUI identity probe (contract 5.3).
  • RULED (2026-07-06 cont.): A1.3 H1-H5 ADOPTED as pre-v1-close tasks; A1.4 canary interpretation CONFIRMED + adopted as pre-deployment task. Full onboarding completion package (scripts + internal docs + client-facing drafts + live end-to-end validation) added to handoff section 6 v1-close checklist.
  • RULED (2026-07-06 cont.): clientdocs/ TOP-LEVEL for client-facing drafts; build INTERLEAVES with batch-3 (foil onboarding = live workflow validation); A2 disruptive half PRE-APPROVED conditional on live N+1 headroom PASS.
  • DELIVERED (addendum 22): H1 scripts/tenant-assert.sh + 7-case harness (green); clientdocs/ package DOCFIX-092 (README/intake/welcome/self-service drafts). Remaining in the interleave before foil onboarding: H2 idempotency guard, H3 canary stage, D-073 list_trusts zip implementation (then gated live apply).
  • DELIVERED (operator-away batch, addenda 23-25): H2 re-run guards + H3 canary stage7 (harness 12/12); D-073 implementation STAGED (39-rule zip, drift check PASS -- live apply GATED, block in the D-073 amendment); DOCFIX-093 runbooks/d011-batch3-window-DRAFT.md (the one-window do-doc: D-073 apply + foil stages 0-4 + tenant-assert + canary + d011-04/05 + close-out); H5 embedded; docs/D-068-openbao-assessment-DRAFT.md (B2 research input -- no OpenBao charm exists, MySQL backend removed upstream, V1 goes to Sunbeam not legacy charms; research rec: deadline-bound risk acceptance). Stderr-merge sweep DONE: 10-site inventory logged in addendum 25 (fixes = DOCFIX candidate, not actioned).
  • REPO-SIDE QUEUE now EMPTY except: H4 policy-drift script (main-chat backlog item 6 -- coordinate before building) and the client-docs sweep-on-change rule. EVERYTHING ELSE waits on the operator-present live window (the do-doc).

State facts to remember

  • beta cluster left at node_count=2 (deliberate; bonus resize acceptance coverage).
  • repo is temporarily PUBLIC for Claude web_fetch (SEC-004) -- flip private at v1 close.
  • Jumphost stream: see "Active window" section above (ops-update-20260705 in flight). Vault stays 1.8/stable (D-068 unruled).
  • CORRECTION (2026-07-05, jumphost stream, per handoff reconcile item): the two bullets above are stale -- ops-update-20260705 is CLOSED (fleet fully current, juju 3.6.25); D-068 is RULED on the 1.16 question (1.16 ruled out; vault stays 1.8/stable rev 714; the off-EOL-1.8.8 path remains OPEN under D-068).

Project-completion (execute after D-011 passes)

  • Consolidate 10 per-phase do-documents into docs/v1-deploy-runbook.md.
  • Set repo visibility PRIVATE (SEC-004).
  • v2-deferred: SSH on GitBucket (port 29418), IPv6 dual-stack, NetBox import bundle.

Cross-session pins (append-only)

  • D-071 completion sweep (2026-07-05, DONE): the ops-update window closed; D-071 was AMENDED (create-backup premise corrected -- juju create-backup/download-backup DO exist on juju 3.6; a 902MB controller backup was taken). D-071 REMAINS PROPOSED -- not ratified to ADOPTED. Operator ratification is pending; the single-controller (no-HA) half of the risk is unresolved by backups alone. See D-071 + its 2026-07-05 amendment.
  • Pre-DC-DC controller HA/backup planning (PINNED): before the Roosevelt (DC-DC) phase, hold a dedicated planning session on Juju controller High-Availability / backup architecture. Backups now exist (per D-071 amendment); controller HA is the open design item. This resolves the single-controller risk recorded under D-071 for production.