Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / session-ledger.md
@JANeumatrix JANeumatrix 3 hours ago 19 KB session-ledger: rotate to restore the GA-R4 cap (317 -> 260); re-seed the scan header
Raw Blame History

Session ledger -- in-flight work continuity record

Purpose. Long ops sessions on this cloud routinely exceed a single context window and get COMPACTED (sometimes several times). Anything living only in the chat scrollback is lost at compaction. This ledger is the durable, committed record of what is IN FLIGHT, so any session -- after a compaction, or a fresh one -- can resume without losing pending work.

How to use it (standing practice).

  1. At session start: read this ledger AND run bash scripts/ledger-scan.sh. Reconcile.
  2. ledger-scan.sh is the DRIFT CHECK -- it derives the open-work it reliably can (PROPOSED/OPEN decisions, OPEN security rows, next-free numbers) straight from the repo. This narrative must not claim CLOSED anything the scan shows OPEN, nor omit what it surfaces.
  3. Update this ledger at every deliverable/commit -- it is a standing deliverable like the changelog. The changelog says what CHANGED; this ledger says what is still OPEN.
  4. The machine-derived block below is seeded from a scan; re-run the scan and re-seed rather than editing it by hand.

SINGLE STREAM (collapsed 2026-07-13). This ledger previously carried three parallel, separately-owned stream sections (main-chat, jumphost, shared) plus ~30 append-only session narratives. Those streams are CLOSED and reconciled into the one list below. There is now ONE stream. Do not re-introduce per-stream sections.

Where the history went. The 2,189-line session-by-session narrative is NOT lost -- it is in git history (the parent of the collapse commit) and, in durable form, in the 65 docs/changelog-*.md files, docs/design-decisions.md, and the incident reports. This ledger deliberately carries only what is still OPEN, plus the facts that would otherwise be lost because they live nowhere else.

Machine-derived (re-seed from scripts/ledger-scan.sh; do not hand-edit)

Re-seeded from a 2026-07-27 scan at the STAGE 4 CLOSE (values verified against bash scripts/ledger-scan.sh in that session). Re-run the scan to refresh.

  • PROPOSED / OPEN decisions (4): D-068 (Vault substrate hardening, Roosevelt -- sole remainder is Q2 path selection at Roosevelt Vault design time), D-131 (node-facing DNS for rack-only controllers [ARCH] -- sub-4 open + the pinned DNS architectural review), D-132 (Roosevelt per-DC MAAS topology [ARCH], operator-pinned to the next deployment), D-136 (NetBox-coupled per-DC render pipeline [ARCH], non-gating). D-137 is ADOPTED and correctly drops off the scan. Status lines in docs/design-decisions.md are the only ruling authority.
  • OPEN security rows: 21 per bash scripts/ledger-scan.sh (was 20). SEC-025 opened 2026-07-27 -- the NetBox web-GUI admin password, a HUMAN login that had never left the VM it was minted on, now consolidated to ~/vr1-office1-creds/; the row covers the at-rest exposure the copy CREATES, not the gap it closed. (SEC-024, 2026-07-26, was the previous addition.) The SEC register of record is docs/security-ledger.md; row-level dispositions live THERE only (GA-R4/F3) -- this block carries pointer + count, never rows.
  • Next-free numbers: D = 138, DOCFIX = 205 (204 assigned 2026-07-27 -- the stale "NTP from edge" Stage-4 gate bullet, superseded by D-129(iv), repaired across four surfaces), BUNDLEFIX = 053.
  • Gates: G17 OPENED 2026-07-27 by operator ruling (node-side artifact reachability split out of Stage 4; the gate table in docs/CURRENT-STATE.md is the authority).
  • Standing numbering rule: never write an identifier-shaped token (D-/DOCFIX-/BUNDLEFIX-NNN) ABOVE the real high-water mark anywhere in docs/ or runbooks/ prose -- historically a decoy token in prose inflated the next-free counter (hardened in DOCFIX-174).
  • ledger-scan D-115 false-positive: RESOLVED. The scan keys on the LAST **Status:** line per ## D-NNN: block, not arbitrary body prose. The general hygiene -- trust the Status line over body text -- still holds.

ROTATED 2026-07-19 (GA-R4 rule 6 -- the one-time rotation; F1 cap now enforceable)

The pre-rotation bodies (~1080 lines: Live state, OPEN WORK VR1/VR0, NetBox write-path bugs, PINNED rulings, standing lessons, state facts, project- completion, NetBox import narrative) moved VERBATIM to docs/archive/session-ledger-rotated-20260719.md. Standing lessons + VR0 state facts were ROUTED to durable homes first (platform-traps already carried most; juju format=line -> appendix-A; guard discipline -> operating-discipline; VR0 cloud facts -> maas-as-built-reference.md). Open work is machine-derived above + CURRENT-STATE.md; do not re-grow bodies here -- sessions append BOUNDED summaries only (15-line cap).

Retained compact blocks (still-live obligations without another home yet):

  • NetBox write-path bugs: ROUTED 2026-07-23 to docs/netbox-write-path-findings.md (the durable home; the rotated-body section is history). Open residue: dumper blind spot + duplicate-CIDR collapse unfixed, hardened fidelity re-run + fix re-verification owed -- ALL gated on the next NetBox WRITE campaign, none current.
  • Project-completion (after D-011 passes): consolidate the 10 per-phase do-documents into a v1 deploy runbook; flip repo PRIVATE (SEC-004); revoke/rotate SEC-005/006/007; v2-deferred: GitBucket SSH, IPv6 dual-stack, NetBox import bundle.
  • Deployment-close consideration review -- tenant-onboarding items (PINNED 2026-07-21, operator-directed): the committee review of Chat's three tenant-onboarding inserts refuted each as written but identified corrected residue; the operator DEFERRED all of it to an end-of-deployment review (posture and end goals are shifting this deployment -- reconsider fit then; nothing lands before that review). Items: (1) appendix-A symptom entry for sub-floor OR nonexistent flavor -> cluster-create rejection (capi-helm thresholds are CONFIG OPTIONS, defaults 2 vCPU / 2048 MB RAM, as-built sets neither; the RAM error also fires for mistyped flavors; command shape --public -f json, never the deprecated --long -- archive R9); (2) appendix-A entry for tenant-net-cannot-reach-public-keystone -> nodes register but stall uninitialized (auth_url in the per-cluster clouds secret is hardcoded PUBLIC at capi-helm 1.4.0; as-built carries no interface key), pointing at the phase-08 D-011.3 probe, plus a per-DC precondition line in dc-dc-phase6 Step 12; (3) tenant contract hardening list: PROPOSED -svc-only password neutralization (admin-rotate to discarded value; keystone lock_password blocks password CHANGE, not auth; NEVER lock -cluster -- it keeps PASSWORD auth permanently per D-066); (4) flavor catalog marking = commercial catalog policy, operator prioritization only. RE-VERIFY every citation at review time -- the full verdict lives in the 2026-07-21 disconnected-session transcript, not in a repo surface.

ROTATED 2026-07-26 (GA-R4 rule 3 / F1 -- oldest-first, cap restored)

The eight oldest closed-session summaries (2026-07-18 through the fifth 2026-07-21 session) moved VERBATIM to docs/archive/session-ledger-rotated-20260726.md. The live ledger was 413 lines against the 300-line cap -- a breach the 2026-07-25 close recorded as OWED. Each rotated summary still points at its own archived full body; only the summaries moved. Sessions from 2026-07-22 onward remain live below.

ROTATED 2026-07-27 (GA-R4 rule 3 / F1 -- oldest-first, cap restored at the Stage 4 close)

The three oldest live summaries (all 2026-07-23: G12 dc1 edge->commission+merge, Stage 4 OPEN+carve+mirrors, queue pass + D-068 rulings) moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 317 lines against the 300-line cap. Sessions from 2026-07-24 onward remain live below.

SESSION CLOSE 2026-07-24 -- caveman guardrails + DC1 proxy-build start (bounded, GA-R4)

  • caveman plugin DISABLED (enabled:false both scopes) + SEC-017 + guardrails (repo-lint L11 *.original.md detector; deny compress; CLAUDE.md norm). commits up to 338c919..d0db020.
  • DC-mirror: DC0 full mirror still syncing; DC1 debmirror PAUSED ~330G (bandwidth NOT the bottleneck, measured) + switched to an INTERIM apt caching proxy (D-135 AMENDED 2026-07-24): scripts/dc-cache-proxy.sh shipped + INSTALLED on dc1 (serves archive+UCA 200). Two per-DC Stage-5 test methods (dc0 full-mirror / dc1 proxy) = gates on ONE branch, not sub-branches.
  • DC1 Stage-5 deploy STARTED: per-DC MAAS creds (SEC-018/-019); Juju 3.6.25 on voffice1 (headend, D-128); add-cloud vr1-maas + add-credential both DCs; VIP overlay overlays/vr1-dc1-vips.yaml; phase-4 Step 2.0 (MAAS-cred task) added. commits ..116e9b2.
  • BLOCKED before bootstrap (committee-reviewed): committed bundle.yaml is still VR0 4-node HYPERCONVERGED; VR1 dc1 is 9 ROLE-SEPARATED (D-121 Option C). Architecture DECIDED, deploy ARTIFACTS not rendered (dc-ha-scaleup.yaml partial/unrendered/dc0-named/stale/unwired; tag/ placement/num_units unhandled; phase-4 runbook stale). DNS confirmed GREEN for bootstrap (D-131 forwarder resolves external; hang can't recur) -- but bootstrap needs edge egress OPEN (juju agents/snaps unmirrored); do NOT narrow egress until after bootstrap.
  • OWNED: I fabricated a "D-193" reference in analysis (cardinal sin) -- corrected to D-121; verified NOT persisted to repo. NEXT SESSION: render vr1-dc1 role-separated HA bundle + wire overlays + de-stale phase-4, THEN bootstrap. Full detail: CURRENT-STATE Stage-4 bullet.
  • Status ONLY in CURRENT-STATE.md. Changelog: docs/archive/changelogs/changelog-20260724-caveman-disable.md.

SESSION CLOSE 2026-07-25 -- handoff-pack execution + recon + Chat D-136 coupling (bounded, GA-R4)

  • IMMEDIATE NEXT (do FIRST): MAAS WEB-GUI login does NOT exist -- SEC-018/019 are API/Juju-only keys; the juju-vr1-dc0/dc1 superusers have RANDOM UNSTORED passwords, so the operator is locked out of the MAAS webpage (never minted; no prior task did it). Fix, operator-run on voffice1: sudo maas createadmin --username=<you> --email=<you> (or maas changepassword <user>); +SEC row.-> FALSIFIED + RESOLVED 2026-07-25 (SEC-020). The premise was wrong: GUI login DID exist. The admin superuser's password was minted 2026-07-13 by site-headend-install.sh:452 and MEASURED working this session (login 204 with the stored value vs 400 wrong-password control) -- it was merely root-only on voffice1 and un-consolidated (the SEC-009 miss class, not a missing account). Only the two juju-* clause was correct. DONE under operator-ruled scope "New account + consolidate only": admin password consolidated byte-identical to ~/vr1-office1-creds/ (VM copy stays source-of-record, NOT rotated); NEW operator superuser minted for human GUI login (stdin, never argv); dc0/dc1 manifest backfill -> creds-audit CLEAN all three sites. The juju-vr1-dc0/dc1 passwords remain random+unstored BY RULING -- their API keys are load-bearing for the blocked Stage-5 bootstrap, and operator removes any need to log in as them.
  • RECORD PHASE DONE + pushed (thru 87cec53): Rulings 1/2/3 (GA-R5 verbatim), D-136 PROPOSED (NetBox render pipeline, opt C, non-gating), DOCFIX-201 (D-020 not D-036, 7 sites)/-202/-203; pack-review committee + Chat open-Q5 (Jenkins IS the runner) folded in. DC0 mirror COMPLETE (949G+342M). lint 0.
  • GATED EXECUTION pending (each gated): verify MAAS role tags LIVE (deploy-blocking) -> 10th controller VM -> dc0 VIP extract (2 commits) -> vault-VIP repair (BLOCKER-1/2 + VIP_OCTET_MAX) -> DC-aware gates -> bootstrap. KEEP _handoff-not-committed/ (item 60 vault values not yet consumed).
  • Open sub-rulings: octavia family, reserved-range registry, vault-vip-derivation file. Deferred: CURRENT-STATE s4 stale-HIGH, G14 SEC 12->15, phase-4 designate. Chat skill pkg: .claude/skills/openstack-cloud-ops-consolidated-20260725.md. Detail: changelog items 1-17.

SESSION CLOSE 2026-07-25 -- MAAS admin-account recovery (SEC-020; bounded, GA-R4)

  • PREMISE FALSIFIED before any mutation: MAAS GUI login DID exist -- admin pw minted 2026-07-13 by site-headend-install.sh:452, MEASURED working (204 vs 400 wrong-pw control). Defect was root-only + un-consolidated on voffice1 (SEC-009 class), NOT a missing account.
  • Scope RE-RULED by operator: "New account + consolidate only" -- no existing password rotated.
  • DONE: operator superuser minted (pw via stdin, never argv); admin pw consolidated byte-identical to ~/vr1-office1-creds/ (VM copy stays source-of-record); dc0/dc1 manifest backfill -> creds-audit CLEAN x3; README index lists both. juju-* UNTOUCHED by ruling (their SEC-018/019 keys gate the blocked Stage-5 bootstrap). SEC-020 opened (open rows 15->16); falsified IMMEDIATE NEXT struck; machine-derived block re-seeded.
  • OWNED: called a login probe read-only; it authenticated, writing last_login twice (benign).
  • LOGGED not executed: headend-install consolidate-at-mint; conflation proposable as next-free D [ARCH]; 3 /root secrets unmoved. GA-R4 F1: ledger 370 lines at OPEN vs 300 cap -- rotation OWED. (DISCHARGED 2026-07-26: rotated oldest-first to 280 lines; see the ROTATED block above.)
  • Gauntlet 79 GREEN, repo-lint 0-fail, b2b0c80 pushed. Bookend EARLY (07-21 precedent); further work appends a POST-CLOSE ADDENDUM. Details: docs/archive/changelogs/changelog-20260725-maas-admin-recovery.md.

POST-CLOSE ADDENDUM 2026-07-26 -- D-137 ADOPTED (GA-R4; 07-18/07-21 addendum precedent)

  • The 2026-07-25 close bookend landed early by design; this addendum records the work that followed it rather than re-opening the entry.
  • Operator asked for a better mint/save method + "a durable rule ... so credentials aren't misplaced or lost". Committee review (5 lenses) run; the operator's own insight drove the design: a discovery sweep can NEVER detect a credential that was never minted -- absence is invisible to discovery -- so an expected-state credential MATRIX is the missing half.
  • D-137 ADOPTED 2026-07-26, all five sub-rulings RULED individually (GA-R5, each pushed before the next was asked): enforcement = blocking in preflight; manifests DERIVED from the matrix; --remote bounded to declared locations; D-137 is the policy authority (SEC-009 demotes to a pointer); identity conflation FOLDED IN as the one-identity-one- principal invariant. (SUPERSEDED SAME-DAY: implementation was built, committee-audited and remediated later on 2026-07-26 -- see CURRENT-STATE, which is the status authority. This line is kept as the narrative record of where the session stood at the time.)
  • Research: 3 read-only agents. Capture docs/audit/creds-creation-points-20260725.md -- 55 MINT sites, and 12 declared secrets have NO mint command anywhere (ssh-keygen = ZERO hits repo-wide), plus three credential dirs outside the SEC-009 convention.
  • SEC-021/-022/-023 OPENED (rows 16 -> 19): a consolidated dc0 credential ABSENT from its recorded location; two UNAUDITED shadow *-creds/ stores on the headend; sprawl-glob blind spots incl. a PREDICTED Stage-5 admin-openrc exposure. All logged-not-actioned.
  • NEXT as recorded at the time: build per docs/D-137-implementation-plan.md. THAT HAPPENED in the same-day successor session (tiers 1-3 built, six-lens committee audit, remediation, preflight P5 wired, SEC-009 demoted). Current status: CURRENT-STATE only.
  • Status ONLY in CURRENT-STATE.md. Detail: docs/archive/changelogs/changelog-20260725-maas-admin-recovery.md.

SESSION CLOSE 2026-07-26 -- D-137 build + committee audit + remediation (bounded, GA-R4)

  • D-137 was ADOPTED-not-built at open; all THREE tiers now exist plus enforcement (preflight P5 blocking). 17 commits af1b682..1696d08 pushed. Stage 4 stays OPEN; no stage closed.
  • Six-lens committee audited the build; FOUR converged on one root cause -- tier 2 keyed existence on host-role alone, ignoring site-key, so one site's credential satisfied every site, MASKING SEC-021's dc0 on-disk absence. 8 false greens, all reproduced and fixed.
  • Owned: ~21 inferred filenames (reported six); systemic sec-ref mis-attribution; CURRENT-STATE cited a verdict file for measurements it lacked. 2 claims WITHDRAWN.
  • Five operator rulings (GA-R5, quoted on D-137): 12-column schema; tier 1 as blocking P5; tier 2 local-blocking + stages-reached coupled by new repo-lint L12; tier 3 probe boundary; SEC-024 chmod. Ruling 4 (SEC-009 -> pointer) executed.
  • SEC-024 opened then remediated (mode), severity corrected first. SEC 19 -> 20. Ledger rotated twice (GA-R4 F1). Gauntlet 79 -> 81 GREEN; repo-lint 0-fail.
  • NEXT: live mission has NOT moved -- resume the DC1 Stage-5 chain per CURRENT-STATE. Unbuilt/unruled advice: creds-mint.sh BEFORE Stage 5 (largest minting event).
  • Full body: docs/archive/session-20260726-d137-build.md. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-07-27 -- creds consolidation + STAGE 4 CLOSE-OUT (bounded, GA-R4)

  • Opened on a creds question; ended closing Stage 4. STAGE 4 IS CLOSED AND MERGED -- operator-gated merge commit 6f5701d on main (2 parents, not squashed, 77 commits), branch retired local + remote, post-merge gauntlet ALL GREEN (81) + repo-lint 0-fail on main, close recorded in CURRENT-STATE by 1023596. Next stage branches off main.
  • Creds: dc0 SEC-012 power key consolidated + .pub derived (SEC-021(b) as written -- measured first: it IS the dedicated key, and dc0 using the snap default is SEC-016's ruled design, so NO re-mint); NetBox GUI admin password consolidated (SEC-025, rows 20->21); dc1 svc .pub backfilled. Findings 13 -> 7. MAAS account set verified COMPLETE by enumeration.
  • V2 taught the ruled-deferral state -- reissuing SEC-006's token would have CONTRAVENED a standing 2026-07-13 ruling; the register was what needed changing. --ledger added; V2 had shipped with ZERO harness cases.
  • Two false greens fixed, both in controls that had passed for days: dc-mirror.sh check asserted last-sync EXISTED (dc0 FAIL, dc1 4-day-stale RUNNING both read OK); and creds-audit CLEAN x3 alongside 13 matrix findings. Lesson repo-carried in the skill.
  • Rulings (GA-R5, all quoted): G17 opened (node-side reachability split out -- powered-off nodes cannot be probed); SEC-024 retention "Keep both"; set-interface-v4 reload "a"; D-135 AMENDED (dc0 tests full mirror / dc1 tests proxy -- dc1's 330G mirror REMOVED, not paused).
  • DOCFIX-204: DoD bullet 6 was UNSATISFIABLE (D-129(iv) had ruled the opposite); 4 surfaces. Carve residue: 108 fabrics -> 17, cascade-checked. dc1 nginx purged.
  • OWNED: first draft of the pf reload sat inside the reconfigure heredoc -- would have fixed nothing in the very drop case that caused the bug. Caught pre-ship.
  • FINDING for the next session (logged, not actioned): tests/creds-matrix T24's finding-class baseline (expected-findings.txt) covers TIER 1 ONLY -- tier-2/3 classes (E1/E3/E4/V1/V2) have NO baselined red state, so a future false green there would not turn the gauntlet red. Same class as the two false greens this session fixed. Also still open from the D-137 close: creds-mint.sh is unbuilt/unruled advice and Stage 5 is the largest minting event, so it is worth ruling BEFORE the bundle deploy rather than after.
  • Gauntlet ALL GREEN (81), repo-lint 0-fail. Full body: docs/archive/changelogs/changelog-20260727-creds-consolidation.md; stage record docs/archive/stage-records/vr1-stage4-record.md. Status ONLY in CURRENT-STATE.md.