Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / session-ledger.md
@JANeumatrix JANeumatrix 12 hours ago 21 KB GA-R4 session close: F9 closed live, the reissue tool, P7, lib-identity
Raw Blame History

Session ledger -- in-flight work continuity record

Purpose. Long ops sessions on this cloud routinely exceed a single context window and get COMPACTED (sometimes several times). Anything living only in the chat scrollback is lost at compaction. This ledger is the durable, committed record of what is IN FLIGHT, so any session -- after a compaction, or a fresh one -- can resume without losing pending work.

How to use it (standing practice).

  1. At session start: read this ledger AND run bash scripts/ledger-scan.sh. Reconcile.
  2. ledger-scan.sh is the DRIFT CHECK -- it derives the open-work it reliably can (PROPOSED/OPEN decisions, OPEN security rows, next-free numbers) straight from the repo. This narrative must not claim CLOSED anything the scan shows OPEN, nor omit what it surfaces.
  3. Update this ledger at every deliverable/commit -- it is a standing deliverable like the changelog. The changelog says what CHANGED; this ledger says what is still OPEN.
  4. The machine-derived block below is seeded from a scan; re-run the scan and re-seed rather than editing it by hand.

SINGLE STREAM (collapsed 2026-07-13). This ledger previously carried three parallel, separately-owned stream sections (main-chat, jumphost, shared) plus ~30 append-only session narratives. Those streams are CLOSED and reconciled into the one list below. There is now ONE stream. Do not re-introduce per-stream sections.

Where the history went. The 2,189-line session-by-session narrative is NOT lost -- it is in git history (the parent of the collapse commit) and, in durable form, in the 65 docs/changelog-*.md files, docs/design-decisions.md, and the incident reports. This ledger deliberately carries only what is still OPEN, plus the facts that would otherwise be lost because they live nowhere else.

Machine-derived (re-seed from scripts/ledger-scan.sh; do not hand-edit)

Re-seeded from a 2026-07-27 scan at the STAGE 4 CLOSE. RE-VERIFIED 2026-07-29 against bash scripts/ledger-scan.sh -- every value below unchanged, so this is a confirmation, not a re-seed. Re-run the scan to refresh.

  • PROPOSED / OPEN decisions (3): D-068 (Vault substrate hardening, Roosevelt -- sole remainder is Q2 path selection at Roosevelt Vault design time), D-131 (node-facing DNS for rack-only controllers [ARCH] -- sub-4 open + the pinned DNS architectural review), D-132 (Roosevelt per-DC MAAS topology [ARCH], operator-pinned to the next deployment). D-136 was ADOPTED 2026-07-27 at option (D) and correctly drops off the scan, as D-137 did before it (was 4). Status lines in docs/design-decisions.md are the only ruling authority.
  • OPEN security rows: 21 per bash scripts/ledger-scan.sh (was 20). SEC-025 opened 2026-07-27 -- the NetBox web-GUI admin password, a HUMAN login that had never left the VM it was minted on, now consolidated to ~/vr1-office1-creds/; the row covers the at-rest exposure the copy CREATES, not the gap it closed. (SEC-024, 2026-07-26, was the previous addition.) The SEC register of record is docs/security-ledger.md; row-level dispositions live THERE only (GA-R4/F3) -- this block carries pointer + count, never rows.
  • Next-free numbers: D = 138, DOCFIX = 206, BUNDLEFIX = 053. RE-SEEDED from bash scripts/ledger-scan.sh at the 2026-07-30 close: DOCFIX-205 was assigned that day (the D-117 annotation batch -- four ratified decision texts annotated, D-117's own overstated "FULLY EXECUTED" Status corrected, phase-6's selector-into-FQDN defect fixed). 204 remains 2026-07-27's stale "NTP from edge" Stage-4 gate bullet.
  • Gates: G17 OPENED 2026-07-27 by operator ruling (node-side artifact reachability split out of Stage 4; the gate table in docs/CURRENT-STATE.md is the authority).
  • Standing numbering rule: never write an identifier-shaped token (D-/DOCFIX-/BUNDLEFIX-NNN) ABOVE the real high-water mark anywhere in docs/ or runbooks/ prose -- historically a decoy token in prose inflated the next-free counter (hardened in DOCFIX-174).
  • ledger-scan D-115 false-positive: RESOLVED. The scan keys on the LAST **Status:** line per ## D-NNN: block, not arbitrary body prose. The general hygiene -- trust the Status line over body text -- still holds.

ROTATED 2026-07-19 (GA-R4 rule 6 -- the one-time rotation; F1 cap now enforceable)

The pre-rotation bodies (~1080 lines: Live state, OPEN WORK VR1/VR0, NetBox write-path bugs, PINNED rulings, standing lessons, state facts, project- completion, NetBox import narrative) moved VERBATIM to docs/archive/session-ledger-rotated-20260719.md. Standing lessons + VR0 state facts were ROUTED to durable homes first (platform-traps already carried most; juju format=line -> appendix-A; guard discipline -> operating-discipline; VR0 cloud facts -> maas-as-built-reference.md). Open work is machine-derived above + CURRENT-STATE.md; do not re-grow bodies here -- sessions append BOUNDED summaries only (15-line cap).

Retained compact blocks (still-live obligations without another home yet):

  • NetBox write-path bugs: ROUTED 2026-07-23 to docs/netbox-write-path-findings.md (the durable home; the rotated-body section is history). Open residue: dumper blind spot + duplicate-CIDR collapse unfixed, hardened fidelity re-run + fix re-verification owed -- ALL gated on the next NetBox WRITE campaign, none current.
  • Project-completion (after D-011 passes): consolidate the 10 per-phase do-documents into a v1 deploy runbook; flip repo PRIVATE (SEC-004); revoke/rotate SEC-005/006/007; v2-deferred: GitBucket SSH, IPv6 dual-stack, NetBox import bundle.
  • Deployment-close consideration review -- tenant-onboarding items (PINNED 2026-07-21, operator-directed): the committee review of Chat's three tenant-onboarding inserts refuted each as written but identified corrected residue; the operator DEFERRED all of it to an end-of-deployment review (posture and end goals are shifting this deployment -- reconsider fit then; nothing lands before that review). Items: (1) appendix-A symptom entry for sub-floor OR nonexistent flavor -> cluster-create rejection (capi-helm thresholds are CONFIG OPTIONS, defaults 2 vCPU / 2048 MB RAM, as-built sets neither; the RAM error also fires for mistyped flavors; command shape --public -f json, never the deprecated --long -- archive R9); (2) appendix-A entry for tenant-net-cannot-reach-public-keystone -> nodes register but stall uninitialized (auth_url in the per-cluster clouds secret is hardcoded PUBLIC at capi-helm 1.4.0; as-built carries no interface key), pointing at the phase-08 D-011.3 probe, plus a per-DC precondition line in dc-dc-phase6 Step 12; (3) tenant contract hardening list: PROPOSED -svc-only password neutralization (admin-rotate to discarded value; keystone lock_password blocks password CHANGE, not auth; NEVER lock -cluster -- it keeps PASSWORD auth permanently per D-066); (4) flavor catalog marking = commercial catalog policy, operator prioritization only. RE-VERIFY every citation at review time -- the full verdict lives in the 2026-07-21 disconnected-session transcript, not in a repo surface.

ROTATED 2026-07-26 (GA-R4 rule 3 / F1 -- oldest-first, cap restored)

The eight oldest closed-session summaries (2026-07-18 through the fifth 2026-07-21 session) moved VERBATIM to docs/archive/session-ledger-rotated-20260726.md. The live ledger was 413 lines against the 300-line cap -- a breach the 2026-07-25 close recorded as OWED. Each rotated summary still points at its own archived full body; only the summaries moved. Sessions from 2026-07-22 onward remain live below.

ROTATED 2026-07-27 (GA-R4 rule 3 / F1 -- oldest-first, cap restored at the Stage 4 close)

The three oldest live summaries (all 2026-07-23: G12 dc1 edge->commission+merge, Stage 4 OPEN+carve+mirrors, queue pass + D-068 rulings) moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 317 lines against the 300-line cap. Sessions from 2026-07-24 onward remain live below.

ROTATED 2026-07-27 (second pass, GA-R4 rule 3 / F1 -- cap restored at the grounding-audit close)

The oldest remaining live summary (2026-07-24, caveman guardrails + DC1 proxy-build start) moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 305 lines against the 300-line cap. Sessions from 2026-07-25 onward remain live below.

ROTATED 2026-07-27 (third pass, GA-R4 rule 3 / F1 -- cap restored at the grounding-audit close)

The TWO oldest remaining live summaries -- 2026-07-25 handoff-pack execution + recon + Chat D-136 coupling, and 2026-07-25 MAAS admin-account recovery (SEC-020) -- moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 321 lines against the 300-line cap; one pass left it under. (This paragraph was left MANGLED by an earlier edit -- a sentence broke off mid-number and ran into a stray fragment of the 2026-07-25 MAAS-admin-recovery summary. Found by a 2026-07-29 sweep and repaired here; that summary's full body is at docs/archive/changelogs/changelog-20260725-maas-admin-recovery.md.)

ROTATED 2026-07-27 (fourth pass, GA-R4 rule 3 / F1 -- cap restored at this close)

The oldest remaining live summary ("POST-CLOSE ADDENDUM 2026-07-26 -- D-137 ADOPTED (GA-R4; 07-18/07-21 addendum precedent)") moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 316 lines against the 300-line cap.

ROTATED 2026-07-29 (GA-R4 rule 3 / F1 -- oldest-first, cap restored at this close)

The TWO oldest live summaries -- 2026-07-26 (D-137 build + committee audit + remediation) and 2026-07-27 (creds consolidation + the STAGE 4 CLOSE-OUT) -- moved VERBATIM to docs/archive/session-ledger-rotated-20260729.md. The live ledger would otherwise have been 321 lines against the 300-line cap once this session's bookend was appended. Sessions from the 2026-07-27 grounding audit onward remain live below.

ROTATED 2026-07-30 (GA-R4 rule 3 / F1 -- oldest-first, cap restored at this close)

The TWO oldest live summaries -- the 2026-07-27 STAGE-5 GROUNDING AUDIT close and its 14-RULINGS post-close addendum -- moved VERBATIM to docs/archive/session-ledger-rotated-20260730.md. The live ledger stood at exactly 300 lines, so the next append would have breached the cap. Rotated HERE rather than leaving it owed, because the next session is directed straight at the juju deployment. Sessions from the 2026-07-27 Phase-0 close onward remain live below.

SESSION CLOSE 2026-07-27 -- Phase 0, R15 gates, D-136, step 3 executed (bounded, GA-R4)

  • Merged the grounding-audit branch to main (607813b, recorded 6495cfb), retired it; 29 commits on dc-dc-stage5-preconditions. NO stage opened/closed. SEC 21; D 138/DOCFIX 205.
  • PHASE 0: voffice1 off a 105-commit-stale retired branch (both DCs' inner tfstate lives there -- proven gitignored, sha256 identical); openstack 6.6.0 (snap REFUTED: no Caracal channel).
  • ALL THREE R15 GATES FIXED, each reproduced first: repo-lint PASSED over ZERO files on a typo'd flag; preflight left "clear to deploy" on exits 127/126/130/3; the gauntlet pinned no names.
  • RULINGS (GA-R5, quoted): no DC ordering + NOC deferred (F1); D-136 ADOPTED option (D); v6 host part mirrors the v4 octet (F4); dc1 FIP pool; node v6 = MAAS static; D-101 GOVERNING RATIONALE.
  • STEP 3 EXECUTED both DCs: 12 v6 subnets, 24 bands + 2 FIP pools, apex 3->27 ranges / 4->160 addresses (156 VIPs), node carve 108 links. D-134 bands + D-020/R11 VIPs were RULED-BUT-NEVER-BUILT, now artifacts. 18 Ready unchanged. Found: gauntlet HOST-DEPENDENT, preflight P5 HOST-BLIND.
  • OWNED: called step 3 "complete" twice while the node layer was unbuilt; mis-filed P0-5; built the apex tool against a dump so it matched zero live. All corrected on-surface.
  • NEXT: gate host-authority; provider-bundle-check ARITY gap (imminent); voffice1 back to main at merge; then the renderer. Body: changelog-20260727-stage5-phase0.md.

SESSION CLOSE 2026-07-29 -- arity gate, renderer, ruling 3, chain audit, OSD carve LIVE (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions, 22 commits pushed. NO stage opened/closed. Scan unchanged: 3 decisions, SEC 21, D 138 / DOCFIX 205 / BUNDLEFIX 053.
  • VIP arity gap CLOSED + R11's three ruled gate changes. prefer-ipv6 is coupled to arity BOTH ways, closing the L3-9 merge order in which dropping the v6 legs exited 0.
  • RENDERER SHIPPED (D-136 (D)), reproducing a reviewed artifact BYTE-FOR-BYTE. RULING 3, both commits: bundle.yaml is VIP-FREE; both DCs dual-family, vault .61 / designate .62 BUILT.
  • CHAIN AUDIT (step 6): the adversarial lens made the chain go wrong-but-GREEN three times -- R9's ruled drift gate had never been built and dc0's overlay was gated by nothing. Built tests/render-drift, added 3 refusals, fixed 2 harness cases that could not fail.
  • LIVE, each gated and independently verified: 8 OSD volumes + 2 controller VMs applied both DCs; 9 re-commissions with skip_networking=1, every diff EXACTLY the new device; MAC adoption applied (drift 0 across 20 nodes, both roots ZERO DIFF); both controllers commissioned; all ruled MAAS tags created. THE STAGE-5 ALLOCATION BLOCKER IS CLOSED.
  • R1 precondition 3 CLOSED BY MEASUREMENT -- skip_networking=1 is the vendor control, and the MAC check sits BETWEEN apply and re-commission, which is what makes 2026-07-20 non-repeatable.
  • RULED (GA-R5): v6 IP SANs on the Octavia controller cert (D-109 note); juju controller at utility .5 AND the octet map is a STANDING CROSS-DC STANDARD (D-134 amendment) -- divergence between DCs at the same octet is now a DEFECT, not a local choice.
  • Octavia: clean negative. juju 3.6 DEEP-MERGES overlay options (charm v12.1.1 source), so octavia-pki cannot clobber the VIP; and nothing inside Octavia requires IPv4.
  • OWNED: three strict-bash quoting traps, two false-positive greps, one fabricated flag, one red-lint push. All corrected on-surface; traps in docs/audit/queued-findings-20260729.txt.
  • NEXT: octavia-pki generation -- the last item hard-failing preflight, and the generator is now per-DC. Gauntlet ALL GREEN (87) vcloud; repo-lint 0 fail.
  • Body: docs/changelog-20260728-vip-arity-gate.md. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-07-30 -- Stage-5 preconditions: 19 Phase-3 items, gate integrity, per-DC Octavia PKI LIVE (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions, 25 commits pushed (ab4c842..). NO stage opened/closed. Scan unchanged: 3 decisions, SEC 21, D 138 / DOCFIX 205 / BUNDLEFIX 053.
  • 6 agents (3 read-only recon, 4 delivery). Phase-3 re-measured 2 FIXED / 19 REMAIN / 5 NEW, not the 21 the 07-27 doc implied; 13 items + 3.9/3.10 + R5 delivered, phase-4 runbook 434 -> 929 lines.
  • F1 (HIGH): R7's per-DC Octavia PKI was HALF BUILT -- subject and VIP gate per-DC, $DC in NO path, so generating dc1 would have DESTROYED dc0's CA and left one fixed-name overlay applying dc1's CA to dc0. Fixed end to end + REFUSE-IF-PRESENT.
  • F4: the per-DC rename would have UN-IGNORED a CA private key in a repo SEC-004 calls PUBLIC. Glob + a check-ignore self-assert, proven both ways. F2: register was blind to a missing 2nd CA set (was RULED work, R13 Part 1). F6: P5 was probing the WRONG HOST'S filesystem and reporting it as fact -- 34 findings vs the true 7.
  • PKI GENERATED (operator-executed; guard blocks the mint by design, not worked around), both DCs verified PASS 29/0, independence proven by sha256, backups proven by restore (PKI 12/12 vs live; tfstate byte-identical, correct serials). E2 + E3 fully closed.
  • scripts/octavia-pki.sh verify NEW (harness 21/21): asserts the SAN set nothing asserted before (F8's SAN-less cert), and A12 ARMS ITSELF from os-public-hostname so F9 cannot be missed.
  • Gate integrity: decorative HA, machines-overlay no-op, cloud-assert arity, G17 node check, gauntlet locale. P4 DC-aware (DC refs 1 -> 45); 3.7 closed; lib-net dc1 arm.
  • NEW ruling-shaped, now BLOCKING: D-008 + D-106:2563 do not say whether substrate vr1-dc1 is dc1 by TOKEN or dc2 by POSITION -- item 3.1's ambiguity inside a ruling, deciding cert identity. A12 REFUSES rather than picking. ^ WITHDRAWN 2026-07-30 (DOCFIX-205): NOT a ruling gap. D-117 ruled it 2026-07-13 (dc1/dc2 retired for dc0/dc1) and says so in its own Status line; D-106 read as open only because D-117's annotation half was never executed (zero of four) while its Status line claimed "FULLY EXECUTED". Also not blocking -- A12 measures INERT and passes. Now an assertion on the derived zone; status in CURRENT-STATE.md.
  • Headend reconcile found a gate RED ON THE ONLY HOST THAT DEPLOYS (opentofu-validate fmt walking gitignored tfvars). Fixed + scoped; both hosts now ALL GREEN (89), repo-lint 0 fail.
  • Three wrong records found, none catchable by any gate: D-124 told a reader to source a provider MAC from a file holding only boot MACs (would have bridged br-ex onto the PXE NIC); the skill cites D-107 for a tailnet path it does not rule; office1-tailscale is labelled a subnet router and advertises none.
  • A "backup set" asserted on three surfaces did not exist. Built for the PKI (1.0-GEN.e) and the inner tfstates (phase-2 step 13), declared, restores proven. Forward requirement recorded: the pinned secrets-storage solution MUST absorb both.
  • OWNED: the operator's own working commands were NOT in the repo until the pre-bookend sweep asked; I waved through 664 CA certs as "harmless" (integrity, not secrecy); two silenced errors (| grep -q under pipefail, git pull -q &&); four consecutive cwd mistakes. All corrected on-surface.
  • NEXT: F8 residue (heredoc -> printf when a real generation can be run); D-137 fork-1 guard ruling (blocked correctly once, misfired 5x); preflight still has NO verified-green reading; G17 fixture-green only; the two phase-04-network-* scripts.
  • Sweep capture docs/audit/queued-findings-20260730.txt (F10-F13 + 2 ruling questions). Body: docs/changelog-20260729-stage5-preconditions.md. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-07-30 (part 2) -- F9 CLOSED live, the reissue tool, P7, lib-identity (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions, 19 commits pushed (1ddb078..). NO stage opened/closed. Scan unchanged: 3 decisions, SEC 21, D 138 / DOCFIX 206 / BUNDLEFIX 053.
  • >>> OPERATOR DIRECTIVE, STANDING: THE NEXT SESSION PROCEEDS TO THE JUJU DEPLOYMENT (Stage 5), NO MATTER WHAT. Verbatim: "we have to continue to juju deployment next session no matter what". Precondition polishing is DONE; further hardening is OUT OF SCOPE unless it blocks the deploy. This is the GA-F06 circuit-breaker made explicit -- do not spend the session on records.
  • DOCFIX-205: Q1 WITHDRAWN -- D-117 ruled it 2026-07-13; it resurfaced only because D-117's annotation half was never executed (0 of 4) while its own Status claimed "FULLY EXECUTED". All four annotated; Status corrected. Q2 withdrawn too (D-137 fork 1 ruled 2026-07-25). Third stale premise in one findings file.
  • F9 CLOSED LIVE, both DCs (gated per DC, never batched): controller certs reissued into their own zones -- omega.dc0.vr1 / omega.dc1.vr1, fresh P-256 keys, CAs untouched. Capture docs/audit/octavia-reissue-executed-20260730.txt.
  • octavia-pki.sh reissue SHIPPED (operator: "Full script minter now"): derived zone, printf config (F8 paid off), stage-assert-promote, single-value overlay surgery, backup-before-mint.
  • VALIDATION AGENTS EARNED THEIR COST. Mutation testing deleted my three new assertions and the harness stayed GREEN every time. Six defects closed: A15 (no keyUsage/EKU minted + passed everything), A16 (-days defaults to 30 and verify had NO validity check), A17 (overlay/workspace desync read PASS), the re-run guard REFUSING to fix broken certs, A8's vacuous negative, two overlay checks moved pre-mint.
  • F8 + F9 FIXED AT SOURCE in 1.0-GEN.c -- the generation path would otherwise recreate both at the next DC standup. F10 handled: 13 mint-refs re-anchored single-pass by row id; S4 clean.
  • P7: the PKI is now an actual GATE in preflight.sh -- headend-only, NOT EVALUATED elsewhere, rc3 mapped explicitly, and it requires the literal zone line because verify exits 0 on a wrong-zone cert while inert.
  • scripts/lib-identity.sh -- CLOUD_NAME + CLOUD_DOMAIN in ONE file; a rebuild that renames the estate edits one file. T47 catches shell-vs-OpenTofu drift, proven able to fail.
  • Backup custody done + REGISTERED (2 rows, manifests, n-reissue-backup); the secrets-storage PIN now covers this step's creds and certs. Hazard recorded: these archives carry CA issuance state -- never restore over a workspace that has issued since.
  • OWNED: I withdrew two of my own recommendations after measuring (the vr0-dc0 retirement, and the D-137 escalation framing); pushed one red-lint commit via ; instead of &&; two harness stub bugs. All corrected on-surface.
  • Guard misfire tally now SEVEN -- it blocked a command that only TESTED it, then the heredoc documenting its own misfires. Hardening it needs no ruling (fork 1 is ruled).
  • Gauntlet ALL GREEN (89) BOTH hosts; repo-lint 0 fail; octavia-pki 51/51; preflight 33/33; creds-matrix 65/65 + 5 pre-existing findings. verify 37/0 and P7 [ok] on both DCs.
  • NEXT (Stage 5 entry): preflight is RED only on P5's pre-existing credential register -- decide accept-or-remediate at the gate, do not re-audit it. G17 arms at first boot. Bodies: docs/changelog-20260730-docfix205-d117-annotation.md, -octavia-reissue-tool.md. Status ONLY in CURRENT-STATE.md.