Fork: 0
OpenStack / openstack-caracal-ipv4
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-ipv4 / docs / session-ledger.md
@Jesse Austin Jesse Austin 20 hours ago 13 KB session-ledger: 2026-07-05 resume reconcile (jumphost lane)
Raw Blame History

Session ledger -- in-flight work continuity record

Purpose. Long ops sessions on this cloud routinely exceed a single context window and get COMPACTED (sometimes several times). Anything living only in the chat scrollback is lost at compaction. This ledger is the durable, committed record of what is in flight, so any session -- after a compaction, or a fresh one, or the parallel Claude Code stream -- can resume without losing pending work.

How to use it (standing practice).

  1. At session start: read this ledger AND run bash scripts/ledger-scan.sh. Reconcile.
  2. ledger-scan.sh is the DRIFT CHECK -- it derives the open-work it reliably can (PROPOSED/OPEN decisions, OPEN security rows, next-free numbers) straight from the repo. This narrative must not claim CLOSED anything the scan shows OPEN, nor omit what it surfaces.
  3. Update this ledger at every deliverable/commit -- it is a standing deliverable like the changelog. The changelog says what CHANGED; this ledger says what is still OPEN.
  4. Numbers and the machine-derived section below are seeded from a scan; re-run the scan and re-seed rather than editing those by hand.

Machine-derived (re-seed from scripts/ledger-scan.sh; do not hand-edit)

As of repo HEAD 968fe6c, 2026-07-05 (re-run the scan to refresh):

  • PROPOSED / OPEN decisions: D-050 (keystone use-policyd-override=true with no policy zip, FINDING-1), D-068 (Vault substrate hardening, Roosevelt -- 1.16 ruled out per amendment; off-EOL path OPEN), D-071 (routine update cadence + controller patch policy -- operator to rule; gated on the pre-DC-DC HA/backup session).
  • OPEN security rows: SEC-001 (rotate after the current rebuild completes), SEC-003 (assign custodians + rehearse second-person unseal), SEC-004 (flip repo to private at v1 close).
  • Next-free numbers: D = 073, DOCFIX = 091, BUNDLEFIX = 012. (Scan, handoff-20260705, and the changelog next-free pointer all agree.)

Active build (this session)

  • validate.sh D-011 runner -- modular. Foundation committed: scripts/lib-validate.sh (exit contract 0/1/2/3/4, emit, vr_json stderr-safe, run, env hygiene, disruptive gate) + scripts/validate.sh orchestrator (profiles, verdict, --stop-on-fail, --include-disruptive).
    • Batch 1 committed: checks/d011-01-charms, checks/d011-06-vault-unseal (MANUAL until the SEC-003 rehearsal closes).
    • Batch 2 committed: checks/d011-02-vip-jumphost (catalog-derive public origins, TLS+HTTP), checks/d011-03-vip-tenant (ephemeral agnhost pod egress -> keystone VIP, D-035 proof). post-restart profile now fully populated.
    • Batch 3 DRAFTED (mock-tested only, NEEDS LIVE VALIDATION): d011-04-octavia-lb (RR+member additive; amphora failover --disruptive, N+1 headroom-guarded) + d011-05-magnum-e2e (wraps tenant-acceptance + timing). full-d011 profile now COMPLETE.
    • Next: live-validate batch 3 (see Verify-live queue), then item-3/#5-#8 backlog.
    • D-011 AMENDED bar: 1 charms; 2 VIP jumphost; 3 VIP tenant; 4 octavia RR/failover/recovery; 5 magnum e2e + OCCM; 6 second-person manual unseal (attestation, D-069); 7 DROPPED (D-070 supersedes D-012 snapshots); 8 DROPPED (D-019, no Designate).

Script backlog (changelog-tracked)

  • DONE: tenant-offboard.sh, vault-kv-health.sh, cloud-assert.sh (item 3 largely), validate foundation + batch 1.
  • item 3 remainder: fold vault-kv-health.sh into cloud-assert as a section; use the combined gate to regenerate the stale restart-procedure runbook.
  • item 5: scripts/tenant-cluster-create.sh -- generalize the t2-02 stage6 + watch + D-066-evidence wrapper. Not started.
  • item 6: scripts/keystone-policy-drift.sh -- D-051 base_* alignment (LIVE-READ PENDING). Not started.
  • item 7: scripts/cloud-snapshot.sh -- juju-export baseline capture (D-070 retired KVM snapshots but KEPT the export/inventory baseline). Not started.
  • item 8: tests/tenant-acceptance/ harness -- pending (tenant-onboard harness landed).

Register / operator-gated (see docs/handoff-20260703-open-items.md)

  • R-1 / SEC-003: Vault unseal-key custodian assignment (OPEN). Gates V-2 and closes d011-06-vault-unseal (currently MANUAL). Operator input, recorded off-repo.
  • V-2: second-person unseal rehearsal (blocked on R-1).
  • R-2 (D-043 caveat): capi-mgmt-v2 auto-resume exclusion -- rule or accept.
  • R-3: was D-063 -- now CLOSED. The handoff doc still lists R-3 OPEN; update it.
  • D-1: Pattern-A full redeploy (VR0 DC0).

Verify-live queue (batch 3 drafts -- confirm on real cloud before trusting)

  • d011-04 amphora_headroom() field parsing (server show .flavor; hypervisor list --long vcpu/ram) -- safe default HOLDs on any parse miss, but confirm the OK path live.
  • d011-04 OCCM LB-name-contains-service-name assumption; agnhost /hostname round-robin behavior.
  • d011-05 needs a FOIL tenant (2nd tenant) for P3 isolation -- onboard one or set VR_FOIL_APPCRED (acme, the former foil, was offboarded).
  • Sequence: run d011-04 non-disruptively first (RR+member), then --include-disruptive once headroom confirmed.

Logged-not-actioned (small; would vanish at compaction)

  • offboard v2 --sweep-magnum-orphans mode (orphan per-cluster trustee in the magnum domain).
  • offboard stage-3 app-cred idempotency re-run guard.
  • host_href=None barbican observation (exercised fine in stage 6; probably closeable).
  • DOCFIX candidate: sweep repo for other -f json ... 2>&1 stderr-merge hazards (DOCFIX-085 class).
  • D-050 (keystone policyd override) and the D-063-adjacent identity:list_trusts="" hardening remain PROPOSED/OPEN, not actioned.

Jumphost stream -- ops-update-20260705: WINDOW CLOSED 2026-07-05 (addenda 13-16)

First execution of runbooks/ops-update-procedure.md (DOCFIX-086; addendum 13). Logged session ops-update-20260705; checkpointed here at each milestone for cross-stream sync. State as of the Section 2 entry checkpoint:

  • DONE Section 1 (pre-flight): client 3.6.25 / controller 3.6.24 / all 91 agents uniformly 3.6.24 -> target 3.6.25. Worklist = 17 apps. Quiesce clean. Pre-change BOM asbuilt/20260705-102951 committed on a TRUE cloud-assert PASS (required the DOCFIX-087 A7 false-negative fix, addendum 14 -- window was HELD on the operator's Option B ruling until it landed).
  • DONE Section 2.1: controller state backup via juju create-backup -m admin/controller (902MB, checksummed, ~/openstack-baseline/, jumphost-only). NOTE: controller model is admin/controller, NOT /controller.
  • DONE Section 2.2/2.3: controller upgraded 3.6.24 -> 3.6.25 (blind window ~1 min, as expected); post-controller cloud-assert PASS.
  • DONE Section 3: all 91 openstack-model agents + controller machine at 3.6.25, states clean (63 idle / 28 started), nothing stuck.
  • DONE Section 4 G0: keystone 778->817; token probe OK; cloud-assert PASS.
  • DONE Section 4 G1 (11 apps, standing group approval, serial+gated): placement 125->154, nova-cloud-controller 795->823, neutron-api 650->710, neutron-api-plugin-ovn 178->215, glance 642->681, glance-simplestreams-sync 124->152, octavia-diskimage-retrofit 196->232, cinder 733->820, cinder-ceph 533->568, barbican 209->265, barbican-vault 75->99. All probes OK; boundary cloud-assert PASS.
  • DONE Section 4 G2: octavia 441->542; LBs 2/2 ACTIVE/ONLINE; cloud-assert PASS.
  • DONE Section 4 G3: dashboards 728->750 / 59->122 / 120->168; D-044 override INTACT; HTTP+login healthy. RCA: VIP HTTPS dead SINCE DEPLOY (haproxy 443 backend targets vhost-less internal addr; L4 check masks; phase-03 3.3 check fails open) -- NOT a G3 regression; G3 stands (operator ruling; addendum 15). Upstream bug + 2 DOCFIX candidates queued post-window.
  • DONE (coordinated): vault bundle revert 1.16->1.8/stable (BUNDLEFIX-010) + D-068 AMENDMENT recorded; 1.16 ruled out (certs V0/V1, Raft-only, Ceph, BUSL); "off EOL 1.8" remains OPEN. Evidence: docs/D-068-vault-1.8-vs-1.16-analysis.md.
  • DONE Section 4 G4: nova-compute 827->894; hypervisors up; guests byte-identical pre/post.
  • DONE Sections 5-6: post cloud-assert PASS; post BOM asbuilt/20260705-194617 committed; version coherence exact (91 agents @ 3.6.25); BOM diff = 17 expected rev pairs + 1 explained metadata delta (cinder secrets-storage endpoint); appendix-B B.1 fully re-baselined; runbook as-executed corrections DOCFIX-088; D-071 amended (backup exists). WINDOW COMPLETE.
  • DONE follow-on (addendum 17): magnum 70->96 + vault 372->714 refreshed after download-and-diff analysis (mass-rebuild campaign upstream; vault stayed reactive/V0, handlers byte-identical, never sealed). Fleet FULLY CURRENT (zero can-upgrade-to); appendix-B re-baselined from asbuilt/20260705-201118.
  • DONE (addendum 18): D-072 ADOPTED+EXECUTED -- dashboard VIP TLS repaired (cluster binding -> metal-admin; BUNDLEFIX-011 durable; https 200 first time on this build). DOCFIX-089 phase-03 3.3 fail-closed + appendix-A entry. Upstream bug DRAFTED: docs/upstream-bug-draft-dashboard-tls.md.
  • POST-SESSION QUEUE (operator): submit the upstream bug + record LP number in D-072; rule on D-071 (still PROPOSED); Vault off-EOL path (Roosevelt-scale, 1.16 ruled out per D-068 amendment; in-channel 714 applied).
  • Window findings logged for close-out (do not action mid-window):
    1. magnum can-upgrade-to anomaly REPRODUCED (points at magnum-dashboard-122; evidence ~/openstack-baseline/magnum-can-upgrade-anomaly-20260705.json) AND Charmhub magnum 2024.1/stable latest release (r101, 2026-07-01) is s390x-ONLY -- no valid amd64 target; magnum EXCLUDED; report upstream to OpenStack Charmers.
    2. vault NEWLY offers in-channel 372->714 on 1.8/stable -- NOT refreshed (D-068); logged only.
    3. juju create-backup/download-backup EXIST on juju 3.6 -- authoring assumption ("removed in 3.0, expected absent") is WRONG; correct ops-update-procedure 2.1 and the D-071 risk section at window close (DOCFIX candidate).
    4. juju status --format=line omits workload messages on 3.6 (root cause of the A7 false negative; fixed by DOCFIX-087).

Jumphost stream -- session 2026-07-05 (post-window resume)

  • Orientation + reconcile DONE: machine-derived block re-seeded from today's scan (DOCFIX next-free 090 -> 091 was stale); shared-section stale-facts correction appended per the handoff reconcile item. Fences validate 4/0/0.
  • FINDING (logged, not actioned): repo_lint.py L5 "next-free identifiers" info line counts ANY textual mention of D-/DOCFIX-/BUNDLEFIX-NNN across all files, so it is inflated by the deliberate tests/ledger-scan fixture (DOCFIX-099 / BUNDLEFIX-050, "must NOT inflate") and by next-free pointer lines themselves (reports D-074/DOCFIX-100/BUNDLEFIX-051). ledger-scan (heading-based, pointer- excluding) is the numbering authority and is correct. DOCFIX candidate: make L5 reuse ledger-scan's exclusion rules or drop the info line. No number consumed.
  • Standing by for batch-3 live validation support (verify-live queue).

State facts to remember

  • beta cluster left at node_count=2 (deliberate; bonus resize acceptance coverage).
  • repo is temporarily PUBLIC for Claude web_fetch (SEC-004) -- flip private at v1 close.
  • Jumphost stream: see "Active window" section above (ops-update-20260705 in flight). Vault stays 1.8/stable (D-068 unruled).
  • CORRECTION (2026-07-05, jumphost stream, per handoff reconcile item): the two bullets above are stale -- ops-update-20260705 is CLOSED (fleet fully current, juju 3.6.25); D-068 is RULED on the 1.16 question (1.16 ruled out; vault stays 1.8/stable rev 714; the off-EOL-1.8.8 path remains OPEN under D-068).

Project-completion (execute after D-011 passes)

  • Consolidate 10 per-phase do-documents into docs/v1-deploy-runbook.md.
  • Set repo visibility PRIVATE (SEC-004).
  • v2-deferred: SSH on GitBucket (port 29418), IPv6 dual-stack, NetBox import bundle.

Cross-session pins (append-only)

  • D-071 completion sweep (2026-07-05, DONE): the ops-update window closed; D-071 was AMENDED (create-backup premise corrected -- juju create-backup/download-backup DO exist on juju 3.6; a 902MB controller backup was taken). D-071 REMAINS PROPOSED -- not ratified to ADOPTED. Operator ratification is pending; the single-controller (no-HA) half of the risk is unresolved by backups alone. See D-071 + its 2026-07-05 amendment.
  • Pre-DC-DC controller HA/backup planning (PINNED): before the Roosevelt (DC-DC) phase, hold a dedicated planning session on Juju controller High-Availability / backup architecture. Backups now exist (per D-071 amendment); controller HA is the open design item. This resolves the single-controller risk recorded under D-071 for production.