Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / session-ledger.md
@JANeumatrix JANeumatrix 4 hours ago 20 KB Session close: GA-R4 bookend, skill sweep, ledger rotation, permission block removed
Raw Blame History

Session ledger -- in-flight work continuity record

Purpose. Long ops sessions on this cloud routinely exceed a single context window and get COMPACTED (sometimes several times). Anything living only in the chat scrollback is lost at compaction. This ledger is the durable, committed record of what is IN FLIGHT, so any session -- after a compaction, or a fresh one -- can resume without losing pending work.

How to use it (standing practice).

  1. At session start: read this ledger AND run bash scripts/ledger-scan.sh. Reconcile.
  2. ledger-scan.sh is the DRIFT CHECK -- it derives the open-work it reliably can (PROPOSED/OPEN decisions, OPEN security rows, next-free numbers) straight from the repo. This narrative must not claim CLOSED anything the scan shows OPEN, nor omit what it surfaces.
  3. Update this ledger at every deliverable/commit -- it is a standing deliverable like the changelog. The changelog says what CHANGED; this ledger says what is still OPEN.
  4. The machine-derived block below is seeded from a scan; re-run the scan and re-seed rather than editing it by hand.

SINGLE STREAM (collapsed 2026-07-13). This ledger previously carried three parallel, separately-owned stream sections (main-chat, jumphost, shared) plus ~30 append-only session narratives. Those streams are CLOSED and reconciled into the one list below. There is now ONE stream. Do not re-introduce per-stream sections.

Where the history went. The 2,189-line session-by-session narrative is NOT lost -- it is in git history (the parent of the collapse commit) and, in durable form, in the 65 docs/changelog-*.md files, docs/design-decisions.md, and the incident reports. This ledger deliberately carries only what is still OPEN, plus the facts that would otherwise be lost because they live nowhere else.

Machine-derived (re-seed from scripts/ledger-scan.sh; do not hand-edit)

Re-seeded from a 2026-07-27 scan at the STAGE 4 CLOSE (values verified against bash scripts/ledger-scan.sh in that session). Re-run the scan to refresh.

  • PROPOSED / OPEN decisions (4): D-068 (Vault substrate hardening, Roosevelt -- sole remainder is Q2 path selection at Roosevelt Vault design time), D-131 (node-facing DNS for rack-only controllers [ARCH] -- sub-4 open + the pinned DNS architectural review), D-132 (Roosevelt per-DC MAAS topology [ARCH], operator-pinned to the next deployment), D-136 (NetBox-coupled per-DC render pipeline [ARCH], non-gating). D-137 is ADOPTED and correctly drops off the scan. Status lines in docs/design-decisions.md are the only ruling authority.
  • OPEN security rows: 21 per bash scripts/ledger-scan.sh (was 20). SEC-025 opened 2026-07-27 -- the NetBox web-GUI admin password, a HUMAN login that had never left the VM it was minted on, now consolidated to ~/vr1-office1-creds/; the row covers the at-rest exposure the copy CREATES, not the gap it closed. (SEC-024, 2026-07-26, was the previous addition.) The SEC register of record is docs/security-ledger.md; row-level dispositions live THERE only (GA-R4/F3) -- this block carries pointer + count, never rows.
  • Next-free numbers: D = 138, DOCFIX = 205 (204 assigned 2026-07-27 -- the stale "NTP from edge" Stage-4 gate bullet, superseded by D-129(iv), repaired across four surfaces), BUNDLEFIX = 053.
  • Gates: G17 OPENED 2026-07-27 by operator ruling (node-side artifact reachability split out of Stage 4; the gate table in docs/CURRENT-STATE.md is the authority).
  • Standing numbering rule: never write an identifier-shaped token (D-/DOCFIX-/BUNDLEFIX-NNN) ABOVE the real high-water mark anywhere in docs/ or runbooks/ prose -- historically a decoy token in prose inflated the next-free counter (hardened in DOCFIX-174).
  • ledger-scan D-115 false-positive: RESOLVED. The scan keys on the LAST **Status:** line per ## D-NNN: block, not arbitrary body prose. The general hygiene -- trust the Status line over body text -- still holds.

ROTATED 2026-07-19 (GA-R4 rule 6 -- the one-time rotation; F1 cap now enforceable)

The pre-rotation bodies (~1080 lines: Live state, OPEN WORK VR1/VR0, NetBox write-path bugs, PINNED rulings, standing lessons, state facts, project- completion, NetBox import narrative) moved VERBATIM to docs/archive/session-ledger-rotated-20260719.md. Standing lessons + VR0 state facts were ROUTED to durable homes first (platform-traps already carried most; juju format=line -> appendix-A; guard discipline -> operating-discipline; VR0 cloud facts -> maas-as-built-reference.md). Open work is machine-derived above + CURRENT-STATE.md; do not re-grow bodies here -- sessions append BOUNDED summaries only (15-line cap).

Retained compact blocks (still-live obligations without another home yet):

  • NetBox write-path bugs: ROUTED 2026-07-23 to docs/netbox-write-path-findings.md (the durable home; the rotated-body section is history). Open residue: dumper blind spot + duplicate-CIDR collapse unfixed, hardened fidelity re-run + fix re-verification owed -- ALL gated on the next NetBox WRITE campaign, none current.
  • Project-completion (after D-011 passes): consolidate the 10 per-phase do-documents into a v1 deploy runbook; flip repo PRIVATE (SEC-004); revoke/rotate SEC-005/006/007; v2-deferred: GitBucket SSH, IPv6 dual-stack, NetBox import bundle.
  • Deployment-close consideration review -- tenant-onboarding items (PINNED 2026-07-21, operator-directed): the committee review of Chat's three tenant-onboarding inserts refuted each as written but identified corrected residue; the operator DEFERRED all of it to an end-of-deployment review (posture and end goals are shifting this deployment -- reconsider fit then; nothing lands before that review). Items: (1) appendix-A symptom entry for sub-floor OR nonexistent flavor -> cluster-create rejection (capi-helm thresholds are CONFIG OPTIONS, defaults 2 vCPU / 2048 MB RAM, as-built sets neither; the RAM error also fires for mistyped flavors; command shape --public -f json, never the deprecated --long -- archive R9); (2) appendix-A entry for tenant-net-cannot-reach-public-keystone -> nodes register but stall uninitialized (auth_url in the per-cluster clouds secret is hardcoded PUBLIC at capi-helm 1.4.0; as-built carries no interface key), pointing at the phase-08 D-011.3 probe, plus a per-DC precondition line in dc-dc-phase6 Step 12; (3) tenant contract hardening list: PROPOSED -svc-only password neutralization (admin-rotate to discarded value; keystone lock_password blocks password CHANGE, not auth; NEVER lock -cluster -- it keeps PASSWORD auth permanently per D-066); (4) flavor catalog marking = commercial catalog policy, operator prioritization only. RE-VERIFY every citation at review time -- the full verdict lives in the 2026-07-21 disconnected-session transcript, not in a repo surface.

ROTATED 2026-07-26 (GA-R4 rule 3 / F1 -- oldest-first, cap restored)

The eight oldest closed-session summaries (2026-07-18 through the fifth 2026-07-21 session) moved VERBATIM to docs/archive/session-ledger-rotated-20260726.md. The live ledger was 413 lines against the 300-line cap -- a breach the 2026-07-25 close recorded as OWED. Each rotated summary still points at its own archived full body; only the summaries moved. Sessions from 2026-07-22 onward remain live below.

ROTATED 2026-07-27 (GA-R4 rule 3 / F1 -- oldest-first, cap restored at the Stage 4 close)

The three oldest live summaries (all 2026-07-23: G12 dc1 edge->commission+merge, Stage 4 OPEN+carve+mirrors, queue pass + D-068 rulings) moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 317 lines against the 300-line cap. Sessions from 2026-07-24 onward remain live below.

ROTATED 2026-07-27 (second pass, GA-R4 rule 3 / F1 -- cap restored at the grounding-audit close)

The oldest remaining live summary (2026-07-24, caveman guardrails + DC1 proxy-build start) moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 305 lines against the 300-line cap. Sessions from 2026-07-25 onward remain live below.

ROTATED 2026-07-27 (third pass, GA-R4 rule 3 / F1 -- cap restored at the grounding-audit close)

The TWO oldest remaining live summaries -- 2026-07-25 handoff-pack execution + recon + Chat D-136 coupling, and 2026-07-25 MAAS admin-account recovery (SEC-020) -- moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 321 lines against the 300-line cap; one pass left it at 3 work appends a POST-CLOSE ADDENDUM. Details: docs/archive/changelogs/changelog-20260725-maas-admin-recovery.md.

POST-CLOSE ADDENDUM 2026-07-26 -- D-137 ADOPTED (GA-R4; 07-18/07-21 addendum precedent)

  • The 2026-07-25 close bookend landed early by design; this addendum records the work that followed it rather than re-opening the entry.
  • Operator asked for a better mint/save method + "a durable rule ... so credentials aren't misplaced or lost". Committee review (5 lenses) run; the operator's own insight drove the design: a discovery sweep can NEVER detect a credential that was never minted -- absence is invisible to discovery -- so an expected-state credential MATRIX is the missing half.
  • D-137 ADOPTED 2026-07-26, all five sub-rulings RULED individually (GA-R5, each pushed before the next was asked): enforcement = blocking in preflight; manifests DERIVED from the matrix; --remote bounded to declared locations; D-137 is the policy authority (SEC-009 demotes to a pointer); identity conflation FOLDED IN as the one-identity-one- principal invariant. (SUPERSEDED SAME-DAY: implementation was built, committee-audited and remediated later on 2026-07-26 -- see CURRENT-STATE, which is the status authority. This line is kept as the narrative record of where the session stood at the time.)
  • Research: 3 read-only agents. Capture docs/audit/creds-creation-points-20260725.md -- 55 MINT sites, and 12 declared secrets have NO mint command anywhere (ssh-keygen = ZERO hits repo-wide), plus three credential dirs outside the SEC-009 convention.
  • SEC-021/-022/-023 OPENED (rows 16 -> 19): a consolidated dc0 credential ABSENT from its recorded location; two UNAUDITED shadow *-creds/ stores on the headend; sprawl-glob blind spots incl. a PREDICTED Stage-5 admin-openrc exposure. All logged-not-actioned.
  • NEXT as recorded at the time: build per docs/D-137-implementation-plan.md. THAT HAPPENED in the same-day successor session (tiers 1-3 built, six-lens committee audit, remediation, preflight P5 wired, SEC-009 demoted). Current status: CURRENT-STATE only.
  • Status ONLY in CURRENT-STATE.md. Detail: docs/archive/changelogs/changelog-20260725-maas-admin-recovery.md.

SESSION CLOSE 2026-07-26 -- D-137 build + committee audit + remediation (bounded, GA-R4)

  • D-137 was ADOPTED-not-built at open; all THREE tiers now exist plus enforcement (preflight P5 blocking). 17 commits af1b682..1696d08 pushed. Stage 4 stays OPEN; no stage closed.
  • Six-lens committee audited the build; FOUR converged on one root cause -- tier 2 keyed existence on host-role alone, ignoring site-key, so one site's credential satisfied every site, MASKING SEC-021's dc0 on-disk absence. 8 false greens, all reproduced and fixed.
  • Owned: ~21 inferred filenames (reported six); systemic sec-ref mis-attribution; CURRENT-STATE cited a verdict file for measurements it lacked. 2 claims WITHDRAWN.
  • Five operator rulings (GA-R5, quoted on D-137): 12-column schema; tier 1 as blocking P5; tier 2 local-blocking + stages-reached coupled by new repo-lint L12; tier 3 probe boundary; SEC-024 chmod. Ruling 4 (SEC-009 -> pointer) executed.
  • SEC-024 opened then remediated (mode), severity corrected first. SEC 19 -> 20. Ledger rotated twice (GA-R4 F1). Gauntlet 79 -> 81 GREEN; repo-lint 0-fail.
  • NEXT: live mission has NOT moved -- resume the DC1 Stage-5 chain per CURRENT-STATE. Unbuilt/unruled advice: creds-mint.sh BEFORE Stage 5 (largest minting event).
  • Full body: docs/archive/session-20260726-d137-build.md. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-07-27 -- creds consolidation + STAGE 4 CLOSE-OUT (bounded, GA-R4)

  • Opened on a creds question; ended closing Stage 4. STAGE 4 IS CLOSED AND MERGED -- operator-gated merge commit 6f5701d on main (2 parents, not squashed, 77 commits), branch retired local + remote, post-merge gauntlet ALL GREEN (81) + repo-lint 0-fail on main, close recorded in CURRENT-STATE by 1023596. Next stage branches off main.
  • Creds: dc0 SEC-012 power key consolidated + .pub derived (SEC-021(b) as written -- measured first: it IS the dedicated key, and dc0 using the snap default is SEC-016's ruled design, so NO re-mint); NetBox GUI admin password consolidated (SEC-025, rows 20->21); dc1 svc .pub backfilled. Findings 13 -> 7. MAAS account set verified COMPLETE by enumeration.
  • V2 taught the ruled-deferral state -- reissuing SEC-006's token would have CONTRAVENED a standing 2026-07-13 ruling; the register was what needed changing. --ledger added; V2 had shipped with ZERO harness cases.
  • Two false greens fixed, both in controls that had passed for days: dc-mirror.sh check asserted last-sync EXISTED (dc0 FAIL, dc1 4-day-stale RUNNING both read OK); and creds-audit CLEAN x3 alongside 13 matrix findings. Lesson repo-carried in the skill.
  • Rulings (GA-R5, all quoted): G17 opened (node-side reachability split out -- powered-off nodes cannot be probed); SEC-024 retention "Keep both"; set-interface-v4 reload "a"; D-135 AMENDED (dc0 tests full mirror / dc1 tests proxy -- dc1's 330G mirror REMOVED, not paused).
  • DOCFIX-204: DoD bullet 6 was UNSATISFIABLE (D-129(iv) had ruled the opposite); 4 surfaces. Carve residue: 108 fabrics -> 17, cascade-checked. dc1 nginx purged.
  • OWNED: first draft of the pf reload sat inside the reconfigure heredoc -- would have fixed nothing in the very drop case that caused the bug. Caught pre-ship.
  • FINDING for the next session (logged, not actioned): tests/creds-matrix T24's finding-class baseline (expected-findings.txt) covers TIER 1 ONLY -- tier-2/3 classes (E1/E3/E4/V1/V2) have NO baselined red state, so a future false green there would not turn the gauntlet red. Same class as the two false greens this session fixed. Also still open from the D-137 close: creds-mint.sh is unbuilt/unruled advice and Stage 5 is the largest minting event, so it is worth ruling BEFORE the bundle deploy rather than after.
  • POST-CLOSE DURABILITY SWEEP (operator-requested): 4 transcript-only items landed on surfaces -- worst was dc-mirror.sh's dc1 row carrying NO warning that dc1 is proxy-only, so install dc1 would silently rebuild the removed apparatus + enabled timer + ~950G pull. Also: a WRONG causal claim in the mirror-gate capture superseded by appended correction (reset-failed cannot re-arm an inactive timer; the vector was a REBOOT via Persistent=yes); platform-traps section 5 added; the two-net-units coexistence claim marked REASONED-NOT- MEASURED. Capture docs/audit/queued-findings-20260727.txt. QUEUED: a runtime install guard for a non-mirror site (a comment is strictly weaker and prose-only prevention has already failed twice here).
  • Gauntlet ALL GREEN (81), repo-lint 0-fail. Full body: docs/archive/changelogs/changelog-20260727-creds-consolidation.md; stage record docs/archive/stage-records/vr1-stage4-record.md. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-07-27 -- STAGE-5 GROUNDING AUDIT (bounded, GA-R4)

  • Operator-directed reconciliation before Stage 5 opens; ran AUTONOMOUSLY while they were away, read-only, rulings QUEUED not adopted. Branch dc-dc-stage5-grounding-audit off main at 71519a2; 4 commits 514f0aa..de11bf4 pushed. NO stage opened or closed.
  • VERDICT: Stage 5 would NOT run error-free today and would fail early. SUBSTRATE is excellent -- all THREE tofu roots ZERO DIFF (first time in one session), 18 nodes Ready exact to D-121 C, MACs/power/statics/fabrics/interfaces all verified, both artifact paths serving, gauntlet 81 GREEN. The gap is the layer between substrate and deploy.
  • Blockers, each measured: voffice1 -- the D-128 host Stage 5 RUNS FROM -- is 105 commits behind on a retired branch, both dc1 overlays ABSENT, bundle still VR0 4-node; NO openstack client on either host (10 scripts need it); 10th controller VM UNAUTHORED + no tag, and dc1 has exactly 9 nodes for a 9-machine bundle; osd-devices=/dev/vdb but every node has only vda; per-role tags authored nowhere; ZERO IPv6 vs D-101's RULED dual-stack; and phase-01, which Step 4 delegates to "verbatim", ABORTS on dc1.
  • Gates that should have caught these cannot fail: repo-lint PASSES over ZERO files on a one-char typo; cluster_count checked nowhere so decorative HA passes; preflight ignores any sub-gate rc not in {1,2}; P3 verified ZERO of 33 channel pins (no juju on PATH). Three Stage-5 gate commands cannot execute on juju 3.6.27; four of five owed VERIFY-LIVE gates have no runbook step at all.
  • Deliverables: docs/audit/stage5-readiness-20260727.md (READ FIRST), stage5-committee-raw-20260727.md (7 lenses verbatim), queued-rulings-20260727.md -- 11 blocking + 4 standing questions, ONE exchange each, NONE adopted, stage5-live-measurement-20260727.txt, charter stage5-grounding-audit-scope-*.md.
  • 21-item DOCFIX batch LOGGED NOT EXECUTED (interlocks with unanswered rulings); next-free unchanged D 138 / DOCFIX 205 / BUNDLEFIX 053. ONE mechanical fix: G3's stale FREEZE clause, which would have blocked the remediation batch itself.
  • OWNED: ran the rack checkers locally and got a false FAIL (they pipe to the rack); read tail's exit code not the script's; waved P3's 33 warns through as expected when the CAUSE was a missing binary; called the privileged creds sweep a newly-closed item when 07-26 had already done it. All four corrected in place, not silently.
  • A SESSION-SCOPED PERMISSION BLOCK IS LIVE in the gitignored settings.local.json -- REMOVE IT. Its only durable record is scope-doc section 6. SEC 21 open, unchanged.
  • Full body: docs/archive/changelogs/changelog-20260727-stage5-grounding-audit.md. Status ONLY in CURRENT-STATE.md.

POST-CLOSE ADDENDUM 2026-07-27 -- 14 RULINGS + SKILL SWEEP (GA-R4; 07-25/07-26 precedent)

  • The STAGE-5 GROUNDING AUDIT close bookend above landed while the operator was away, by design. They then returned and worked the queued rulings. This addendum records what followed rather than re-opening that entry -- the same shape as the 2026-07-26 addendum.

  • Operator-directed full reconciliation before Stage 5; 7-lens read-only committee + live measurement, run AUTONOMOUSLY while away. Branch dc-dc-stage5-grounding-audit, ~30 commits 514f0aa..HEAD pushed. NO stage opened or closed.

  • VERDICT: Stage 5 would NOT run error-free. Substrate excellent (all THREE tofu roots ZERO DIFF, 18 nodes exact to D-121 C, gauntlet 81 GREEN); the gap is between substrate and deploy.
  • Blockers: voffice1 -- the D-128 host Stage 5 RUNS FROM -- is 105 commits stale on a branch deleted upstream; no openstack client on EITHER host; the 10th controller VM is UNAUTHORED; osd-devices points at a device no node has; per-role tags authored nowhere; ZERO IPv6 vs D-101's ruling; and phase-01, which Step 4 delegates to "verbatim", ABORTS on dc1.
  • Gates that should have caught these CANNOT FAIL: repo-lint PASSES over ZERO files on a one-char typo; cluster_count checked nowhere; preflight fails open on rc not in {1,2}; P3 verified ZERO of 33 channel pins. All ruled fixed (R15).
  • 14 RULINGS, 2 WITHDRAWN, G18 OPENED (blocking, deferred until the cloud is live). Authority = each D-number's Status block; questions+utterances in docs/audit/queued-rulings-20260727.md. Execution ALL logged-not-executed.
  • Next-free UNCHANGED: D 138 / DOCFIX 205 / BUNDLEFIX 053. SEC rows unchanged at 21.
  • Two RULED-BUT-NEVER-BUILT decisions found (D-134 bands, D-020 vault VIP), neither detectable by any gate. R8's evidence base collapsed on reading the cited bugs -- the charm's own default was the opposite of the repo's draft.
  • OWNED: three of my own premises needed correcting by measurement (R2a, R9, R14); used a 2-day dump instead of polling the apex; pushed one red-lint commit; hit the repo-lint L5 heading trap twice. All corrected on-surface, none silently.
  • Close sweep found 3 stale surfaces THIS AUDIT created (readiness doc's 13 dead NEEDS-RULING markers, "nothing is adopted" header, 6 blank utterance lines) -- all fixed.
  • SKILL SWEEP done despite no stage close (operator-directed): 3 new invariants + 4 authoring traps folded in; snapshot regenerated 1695 lines, ASCII/LF verified.
  • Session-scoped permission block REMOVED at close. Full body: docs/archive/changelogs/changelog-20260727-stage5-grounding-audit.md. Status ONLY in CURRENT-STATE.md.