Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / session-ledger.md
@JANeumatrix JANeumatrix 16 hours ago 18 KB GA-R4 session close: bounded bookend + the 2026-07-29 findings sweep
Raw Blame History

Session ledger -- in-flight work continuity record

Purpose. Long ops sessions on this cloud routinely exceed a single context window and get COMPACTED (sometimes several times). Anything living only in the chat scrollback is lost at compaction. This ledger is the durable, committed record of what is IN FLIGHT, so any session -- after a compaction, or a fresh one -- can resume without losing pending work.

How to use it (standing practice).

  1. At session start: read this ledger AND run bash scripts/ledger-scan.sh. Reconcile.
  2. ledger-scan.sh is the DRIFT CHECK -- it derives the open-work it reliably can (PROPOSED/OPEN decisions, OPEN security rows, next-free numbers) straight from the repo. This narrative must not claim CLOSED anything the scan shows OPEN, nor omit what it surfaces.
  3. Update this ledger at every deliverable/commit -- it is a standing deliverable like the changelog. The changelog says what CHANGED; this ledger says what is still OPEN.
  4. The machine-derived block below is seeded from a scan; re-run the scan and re-seed rather than editing it by hand.

SINGLE STREAM (collapsed 2026-07-13). This ledger previously carried three parallel, separately-owned stream sections (main-chat, jumphost, shared) plus ~30 append-only session narratives. Those streams are CLOSED and reconciled into the one list below. There is now ONE stream. Do not re-introduce per-stream sections.

Where the history went. The 2,189-line session-by-session narrative is NOT lost -- it is in git history (the parent of the collapse commit) and, in durable form, in the 65 docs/changelog-*.md files, docs/design-decisions.md, and the incident reports. This ledger deliberately carries only what is still OPEN, plus the facts that would otherwise be lost because they live nowhere else.

Machine-derived (re-seed from scripts/ledger-scan.sh; do not hand-edit)

Re-seeded from a 2026-07-27 scan at the STAGE 4 CLOSE. RE-VERIFIED 2026-07-29 against bash scripts/ledger-scan.sh -- every value below unchanged, so this is a confirmation, not a re-seed. Re-run the scan to refresh.

  • PROPOSED / OPEN decisions (3): D-068 (Vault substrate hardening, Roosevelt -- sole remainder is Q2 path selection at Roosevelt Vault design time), D-131 (node-facing DNS for rack-only controllers [ARCH] -- sub-4 open + the pinned DNS architectural review), D-132 (Roosevelt per-DC MAAS topology [ARCH], operator-pinned to the next deployment). D-136 was ADOPTED 2026-07-27 at option (D) and correctly drops off the scan, as D-137 did before it (was 4). Status lines in docs/design-decisions.md are the only ruling authority.
  • OPEN security rows: 21 per bash scripts/ledger-scan.sh (was 20). SEC-025 opened 2026-07-27 -- the NetBox web-GUI admin password, a HUMAN login that had never left the VM it was minted on, now consolidated to ~/vr1-office1-creds/; the row covers the at-rest exposure the copy CREATES, not the gap it closed. (SEC-024, 2026-07-26, was the previous addition.) The SEC register of record is docs/security-ledger.md; row-level dispositions live THERE only (GA-R4/F3) -- this block carries pointer + count, never rows.
  • Next-free numbers: D = 138, DOCFIX = 205 (204 assigned 2026-07-27 -- the stale "NTP from edge" Stage-4 gate bullet, superseded by D-129(iv), repaired across four surfaces), BUNDLEFIX = 053.
  • Gates: G17 OPENED 2026-07-27 by operator ruling (node-side artifact reachability split out of Stage 4; the gate table in docs/CURRENT-STATE.md is the authority).
  • Standing numbering rule: never write an identifier-shaped token (D-/DOCFIX-/BUNDLEFIX-NNN) ABOVE the real high-water mark anywhere in docs/ or runbooks/ prose -- historically a decoy token in prose inflated the next-free counter (hardened in DOCFIX-174).
  • ledger-scan D-115 false-positive: RESOLVED. The scan keys on the LAST **Status:** line per ## D-NNN: block, not arbitrary body prose. The general hygiene -- trust the Status line over body text -- still holds.

ROTATED 2026-07-19 (GA-R4 rule 6 -- the one-time rotation; F1 cap now enforceable)

The pre-rotation bodies (~1080 lines: Live state, OPEN WORK VR1/VR0, NetBox write-path bugs, PINNED rulings, standing lessons, state facts, project- completion, NetBox import narrative) moved VERBATIM to docs/archive/session-ledger-rotated-20260719.md. Standing lessons + VR0 state facts were ROUTED to durable homes first (platform-traps already carried most; juju format=line -> appendix-A; guard discipline -> operating-discipline; VR0 cloud facts -> maas-as-built-reference.md). Open work is machine-derived above + CURRENT-STATE.md; do not re-grow bodies here -- sessions append BOUNDED summaries only (15-line cap).

Retained compact blocks (still-live obligations without another home yet):

  • NetBox write-path bugs: ROUTED 2026-07-23 to docs/netbox-write-path-findings.md (the durable home; the rotated-body section is history). Open residue: dumper blind spot + duplicate-CIDR collapse unfixed, hardened fidelity re-run + fix re-verification owed -- ALL gated on the next NetBox WRITE campaign, none current.
  • Project-completion (after D-011 passes): consolidate the 10 per-phase do-documents into a v1 deploy runbook; flip repo PRIVATE (SEC-004); revoke/rotate SEC-005/006/007; v2-deferred: GitBucket SSH, IPv6 dual-stack, NetBox import bundle.
  • Deployment-close consideration review -- tenant-onboarding items (PINNED 2026-07-21, operator-directed): the committee review of Chat's three tenant-onboarding inserts refuted each as written but identified corrected residue; the operator DEFERRED all of it to an end-of-deployment review (posture and end goals are shifting this deployment -- reconsider fit then; nothing lands before that review). Items: (1) appendix-A symptom entry for sub-floor OR nonexistent flavor -> cluster-create rejection (capi-helm thresholds are CONFIG OPTIONS, defaults 2 vCPU / 2048 MB RAM, as-built sets neither; the RAM error also fires for mistyped flavors; command shape --public -f json, never the deprecated --long -- archive R9); (2) appendix-A entry for tenant-net-cannot-reach-public-keystone -> nodes register but stall uninitialized (auth_url in the per-cluster clouds secret is hardcoded PUBLIC at capi-helm 1.4.0; as-built carries no interface key), pointing at the phase-08 D-011.3 probe, plus a per-DC precondition line in dc-dc-phase6 Step 12; (3) tenant contract hardening list: PROPOSED -svc-only password neutralization (admin-rotate to discarded value; keystone lock_password blocks password CHANGE, not auth; NEVER lock -cluster -- it keeps PASSWORD auth permanently per D-066); (4) flavor catalog marking = commercial catalog policy, operator prioritization only. RE-VERIFY every citation at review time -- the full verdict lives in the 2026-07-21 disconnected-session transcript, not in a repo surface.

ROTATED 2026-07-26 (GA-R4 rule 3 / F1 -- oldest-first, cap restored)

The eight oldest closed-session summaries (2026-07-18 through the fifth 2026-07-21 session) moved VERBATIM to docs/archive/session-ledger-rotated-20260726.md. The live ledger was 413 lines against the 300-line cap -- a breach the 2026-07-25 close recorded as OWED. Each rotated summary still points at its own archived full body; only the summaries moved. Sessions from 2026-07-22 onward remain live below.

ROTATED 2026-07-27 (GA-R4 rule 3 / F1 -- oldest-first, cap restored at the Stage 4 close)

The three oldest live summaries (all 2026-07-23: G12 dc1 edge->commission+merge, Stage 4 OPEN+carve+mirrors, queue pass + D-068 rulings) moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 317 lines against the 300-line cap. Sessions from 2026-07-24 onward remain live below.

ROTATED 2026-07-27 (second pass, GA-R4 rule 3 / F1 -- cap restored at the grounding-audit close)

The oldest remaining live summary (2026-07-24, caveman guardrails + DC1 proxy-build start) moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 305 lines against the 300-line cap. Sessions from 2026-07-25 onward remain live below.

ROTATED 2026-07-27 (third pass, GA-R4 rule 3 / F1 -- cap restored at the grounding-audit close)

The TWO oldest remaining live summaries -- 2026-07-25 handoff-pack execution + recon + Chat D-136 coupling, and 2026-07-25 MAAS admin-account recovery (SEC-020) -- moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 321 lines against the 300-line cap; one pass left it under. (This paragraph was left MANGLED by an earlier edit -- a sentence broke off mid-number and ran into a stray fragment of the 2026-07-25 MAAS-admin-recovery summary. Found by a 2026-07-29 sweep and repaired here; that summary's full body is at docs/archive/changelogs/changelog-20260725-maas-admin-recovery.md.)

ROTATED 2026-07-27 (fourth pass, GA-R4 rule 3 / F1 -- cap restored at this close)

The oldest remaining live summary ("POST-CLOSE ADDENDUM 2026-07-26 -- D-137 ADOPTED (GA-R4; 07-18/07-21 addendum precedent)") moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 316 lines against the 300-line cap.

ROTATED 2026-07-29 (GA-R4 rule 3 / F1 -- oldest-first, cap restored at this close)

The TWO oldest live summaries -- 2026-07-26 (D-137 build + committee audit + remediation) and 2026-07-27 (creds consolidation + the STAGE 4 CLOSE-OUT) -- moved VERBATIM to docs/archive/session-ledger-rotated-20260729.md. The live ledger would otherwise have been 321 lines against the 300-line cap once this session's bookend was appended. Sessions from the 2026-07-27 grounding audit onward remain live below.

SESSION CLOSE 2026-07-27 -- STAGE-5 GROUNDING AUDIT (bounded, GA-R4)

  • Operator-directed reconciliation before Stage 5 opens; ran AUTONOMOUSLY while they were away, read-only, rulings QUEUED not adopted. Branch dc-dc-stage5-grounding-audit off main at 71519a2; 4 commits 514f0aa..de11bf4 pushed. NO stage opened or closed.
  • VERDICT: Stage 5 would NOT run error-free today and would fail early. SUBSTRATE is excellent -- all THREE tofu roots ZERO DIFF (first time in one session), 18 nodes Ready exact to D-121 C, MACs/power/statics/fabrics/interfaces all verified, both artifact paths serving, gauntlet 81 GREEN. The gap is the layer between substrate and deploy.
  • Blockers, each measured: voffice1 -- the D-128 host Stage 5 RUNS FROM -- is 105 commits behind on a retired branch, both dc1 overlays ABSENT, bundle still VR0 4-node; NO openstack client on either host (10 scripts need it); 10th controller VM UNAUTHORED + no tag, and dc1 has exactly 9 nodes for a 9-machine bundle; osd-devices=/dev/vdb but every node has only vda; per-role tags authored nowhere; ZERO IPv6 vs D-101's RULED dual-stack; and phase-01, which Step 4 delegates to "verbatim", ABORTS on dc1.
  • Gates that should have caught these cannot fail: repo-lint PASSES over ZERO files on a one-char typo; cluster_count checked nowhere so decorative HA passes; preflight ignores any sub-gate rc not in {1,2}; P3 verified ZERO of 33 channel pins (no juju on PATH). Three Stage-5 gate commands cannot execute on juju 3.6.27; four of five owed VERIFY-LIVE gates have no runbook step at all.
  • Deliverables: docs/audit/stage5-readiness-20260727.md (READ FIRST), stage5-committee-raw-20260727.md (7 lenses verbatim), queued-rulings-20260727.md -- 11 blocking + 4 standing questions, ONE exchange each, NONE adopted, stage5-live-measurement-20260727.txt, charter stage5-grounding-audit-scope-*.md.
  • 21-item DOCFIX batch LOGGED NOT EXECUTED (interlocks with unanswered rulings); next-free unchanged D 138 / DOCFIX 205 / BUNDLEFIX 053. ONE mechanical fix: G3's stale FREEZE clause, which would have blocked the remediation batch itself.
  • OWNED: ran the rack checkers locally and got a false FAIL (they pipe to the rack); read tail's exit code not the script's; waved P3's 33 warns through as expected when the CAUSE was a missing binary; called the privileged creds sweep a newly-closed item when 07-26 had already done it. All four corrected in place, not silently.
  • A SESSION-SCOPED PERMISSION BLOCK IS LIVE in the gitignored settings.local.json -- REMOVE IT. Its only durable record is scope-doc section 6. SEC 21 open, unchanged.
  • Full body: docs/archive/changelogs/changelog-20260727-stage5-grounding-audit.md. Status ONLY in CURRENT-STATE.md.

POST-CLOSE ADDENDUM 2026-07-27 -- 14 RULINGS + SKILL SWEEP (GA-R4; 07-25/07-26 precedent)

  • The STAGE-5 GROUNDING AUDIT close bookend above landed while the operator was away, by design. They then returned and worked the queued rulings. This addendum records what followed rather than re-opening that entry -- the same shape as the 2026-07-26 addendum.

  • Operator-directed full reconciliation before Stage 5; 7-lens read-only committee + live measurement, run AUTONOMOUSLY while away. Branch dc-dc-stage5-grounding-audit, ~30 commits 514f0aa..HEAD pushed. NO stage opened or closed.

  • VERDICT: Stage 5 would NOT run error-free. Substrate excellent (all THREE tofu roots ZERO DIFF, 18 nodes exact to D-121 C, gauntlet 81 GREEN); the gap is between substrate and deploy.
  • Blockers: voffice1 -- the D-128 host Stage 5 RUNS FROM -- is 105 commits stale on a branch deleted upstream; no openstack client on EITHER host; the 10th controller VM is UNAUTHORED; osd-devices points at a device no node has; per-role tags authored nowhere; ZERO IPv6 vs D-101's ruling; and phase-01, which Step 4 delegates to "verbatim", ABORTS on dc1.
  • Gates that should have caught these CANNOT FAIL: repo-lint PASSES over ZERO files on a one-char typo; cluster_count checked nowhere; preflight fails open on rc not in {1,2}; P3 verified ZERO of 33 channel pins. All ruled fixed (R15).
  • 14 RULINGS, 2 WITHDRAWN, G18 OPENED (blocking, deferred until the cloud is live). Authority = each D-number's Status block; questions+utterances in docs/audit/queued-rulings-20260727.md. Execution ALL logged-not-executed.
  • Next-free UNCHANGED: D 138 / DOCFIX 205 / BUNDLEFIX 053. SEC rows unchanged at 21.
  • Two RULED-BUT-NEVER-BUILT decisions found (D-134 bands, D-020 vault VIP), neither detectable by any gate. R8's evidence base collapsed on reading the cited bugs -- the charm's own default was the opposite of the repo's draft.
  • OWNED: three of my own premises needed correcting by measurement (R2a, R9, R14); used a 2-day dump instead of polling the apex; pushed one red-lint commit; hit the repo-lint L5 heading trap twice. All corrected on-surface, none silently.
  • Close sweep found 3 stale surfaces THIS AUDIT created (readiness doc's 13 dead NEEDS-RULING markers, "nothing is adopted" header, 6 blank utterance lines) -- all fixed.
  • SKILL SWEEP done despite no stage close (operator-directed): 3 new invariants + 4 authoring traps folded in; snapshot regenerated 1695 lines, ASCII/LF verified.
  • Session-scoped permission block REMOVED at close. Full body: docs/archive/changelogs/changelog-20260727-stage5-grounding-audit.md. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-07-27 -- Phase 0, R15 gates, D-136, step 3 executed (bounded, GA-R4)

  • Merged the grounding-audit branch to main (607813b, recorded 6495cfb), retired it; 29 commits on dc-dc-stage5-preconditions. NO stage opened/closed. SEC 21; D 138/DOCFIX 205.
  • PHASE 0: voffice1 off a 105-commit-stale retired branch (both DCs' inner tfstate lives there -- proven gitignored, sha256 identical); openstack 6.6.0 (snap REFUTED: no Caracal channel).
  • ALL THREE R15 GATES FIXED, each reproduced first: repo-lint PASSED over ZERO files on a typo'd flag; preflight left "clear to deploy" on exits 127/126/130/3; the gauntlet pinned no names.
  • RULINGS (GA-R5, quoted): no DC ordering + NOC deferred (F1); D-136 ADOPTED option (D); v6 host part mirrors the v4 octet (F4); dc1 FIP pool; node v6 = MAAS static; D-101 GOVERNING RATIONALE.
  • STEP 3 EXECUTED both DCs: 12 v6 subnets, 24 bands + 2 FIP pools, apex 3->27 ranges / 4->160 addresses (156 VIPs), node carve 108 links. D-134 bands + D-020/R11 VIPs were RULED-BUT-NEVER-BUILT, now artifacts. 18 Ready unchanged. Found: gauntlet HOST-DEPENDENT, preflight P5 HOST-BLIND.
  • OWNED: called step 3 "complete" twice while the node layer was unbuilt; mis-filed P0-5; built the apex tool against a dump so it matched zero live. All corrected on-surface.
  • NEXT: gate host-authority; provider-bundle-check ARITY gap (imminent); voffice1 back to main at merge; then the renderer. Body: changelog-20260727-stage5-phase0.md.

SESSION CLOSE 2026-07-29 -- arity gate, renderer, ruling 3, chain audit, OSD carve LIVE (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions, 22 commits pushed. NO stage opened/closed. Scan unchanged: 3 decisions, SEC 21, D 138 / DOCFIX 205 / BUNDLEFIX 053.
  • VIP arity gap CLOSED + R11's three ruled gate changes. prefer-ipv6 is coupled to arity BOTH ways, closing the L3-9 merge order in which dropping the v6 legs exited 0.
  • RENDERER SHIPPED (D-136 (D)), reproducing a reviewed artifact BYTE-FOR-BYTE. RULING 3, both commits: bundle.yaml is VIP-FREE; both DCs dual-family, vault .61 / designate .62 BUILT.
  • CHAIN AUDIT (step 6): the adversarial lens made the chain go wrong-but-GREEN three times -- R9's ruled drift gate had never been built and dc0's overlay was gated by nothing. Built tests/render-drift, added 3 refusals, fixed 2 harness cases that could not fail.
  • LIVE, each gated and independently verified: 8 OSD volumes + 2 controller VMs applied both DCs; 9 re-commissions with skip_networking=1, every diff EXACTLY the new device; MAC adoption applied (drift 0 across 20 nodes, both roots ZERO DIFF); both controllers commissioned; all ruled MAAS tags created. THE STAGE-5 ALLOCATION BLOCKER IS CLOSED.
  • R1 precondition 3 CLOSED BY MEASUREMENT -- skip_networking=1 is the vendor control, and the MAC check sits BETWEEN apply and re-commission, which is what makes 2026-07-20 non-repeatable.
  • RULED (GA-R5): v6 IP SANs on the Octavia controller cert (D-109 note); juju controller at utility .5 AND the octet map is a STANDING CROSS-DC STANDARD (D-134 amendment) -- divergence between DCs at the same octet is now a DEFECT, not a local choice.
  • Octavia: clean negative. juju 3.6 DEEP-MERGES overlay options (charm v12.1.1 source), so octavia-pki cannot clobber the VIP; and nothing inside Octavia requires IPv4.
  • OWNED: three strict-bash quoting traps, two false-positive greps, one fabricated flag, one red-lint push. All corrected on-surface; traps in docs/audit/queued-findings-20260729.txt.
  • NEXT: octavia-pki generation -- the last item hard-failing preflight, and the generator is now per-DC. Gauntlet ALL GREEN (87) vcloud; repo-lint 0 fail.
  • Body: docs/changelog-20260728-vip-arity-gate.md. Status ONLY in CURRENT-STATE.md.