Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / session-ledger.md
@JANeumatrix JANeumatrix 6 hours ago 31 KB Session close 2026-08-06 (part 3, GA-R4 bookend): phase-03 core-verify F-CV1/F-CV2/F-CV3 RESOLVED + binding conformance
Raw Blame History

Session ledger -- in-flight work continuity record

Purpose. Long ops sessions on this cloud routinely exceed a single context window and get COMPACTED (sometimes several times). Anything living only in the chat scrollback is lost at compaction. This ledger is the durable, committed record of what is IN FLIGHT, so any session -- after a compaction, or a fresh one -- can resume without losing pending work.

How to use it (standing practice).

  1. At session start: read this ledger AND run bash scripts/ledger-scan.sh. Reconcile.
  2. ledger-scan.sh is the DRIFT CHECK -- it derives the open-work it reliably can (PROPOSED/OPEN decisions, OPEN security rows, next-free numbers) straight from the repo. This narrative must not claim CLOSED anything the scan shows OPEN, nor omit what it surfaces.
  3. Update this ledger at every deliverable/commit -- it is a standing deliverable like the changelog. The changelog says what CHANGED; this ledger says what is still OPEN.
  4. The machine-derived block below is seeded from a scan; re-run the scan and re-seed rather than editing it by hand.

SINGLE STREAM (collapsed 2026-07-13). This ledger previously carried three parallel, separately-owned stream sections (main-chat, jumphost, shared) plus ~30 append-only session narratives. Those streams are CLOSED and reconciled into the one list below. There is now ONE stream. Do not re-introduce per-stream sections.

Where the history went. The 2,189-line session-by-session narrative is NOT lost -- it is in git history (the parent of the collapse commit) and, in durable form, in the 65 docs/changelog-*.md files, docs/design-decisions.md, and the incident reports. This ledger deliberately carries only what is still OPEN, plus the facts that would otherwise be lost because they live nowhere else.

Machine-derived (re-seed from scripts/ledger-scan.sh; do not hand-edit)

RE-SEEDED 2026-08-05 from bash scripts/ledger-scan.sh (this commit). The prior seed was 2026-08-02; it had gone STALE by one decision (D-142), one SEC row (SEC-033), and the D / DOCFIX next-free (D 141 -> 143, DOCFIX 208 -> 210) -- the per-session summaries below carried the current figures while this block did not, which is the one thing this block exists to prevent. Re-run the scan to refresh.

  • PROPOSED / OPEN decisions (4): D-068 (Vault substrate hardening, Roosevelt -- sole remainder is Q2 path selection at Roosevelt Vault design time), D-131 (node-facing DNS for rack-only controllers [ARCH] -- sub-4 open + the pinned DNS architectural review), D-132 (Roosevelt per-DC MAAS topology [ARCH], operator-pinned to the next deployment), D-142 (vault-init workflow QoL sweep -- APPROVED-IN-PRINCIPLE / IMPL-DEFERRED, R2 off-host transport unresolved). D-136/D-137 dropped off the scan when ADOPTED (correct). Status lines in docs/design-decisions.md are the only ruling authority.
  • OPEN security rows: 29 per bash scripts/ledger-scan.sh (was 28 at the last seed). The one added since, verified against the register: SEC-033 2026-08-04 (the tls-certificates relation databag exposes vault's global-client private key to any juju model reader; interface-level, mitigate via juju RBAC). SEC-031 remains the live-exposure one: the dc1 edge lost its FreeBSD user database to fsck and runs with NO pf ruleset, i.e. an open router serving its GUI. The SEC register of record is docs/security-ledger.md; row-level dispositions live THERE only (GA-R4/F3) -- this block carries pointer + count, never rows.
  • Next-free numbers: D = 143, DOCFIX = 210, BUNDLEFIX = 053. D moved 141 -> 143 as D-141 (IPAM allocations dual-stack, status-distinguished) and D-142 (vault-init QoL, PROPOSED) were assigned. DOCFIX moved 208 -> 210: DOCFIX-208 (phase-4 dc0 machines-overlay omission, 2026-08-02) and DOCFIX-209 (this session's SKILL.md close-bookend correction) are both assigned.
  • Gates: G17 OPENED 2026-07-27 by operator ruling (node-side artifact reachability split out of Stage 4; the gate table in docs/CURRENT-STATE.md is the authority).
  • Standing numbering rule: never write an identifier-shaped token (D-/DOCFIX-/BUNDLEFIX-NNN) ABOVE the real high-water mark anywhere in docs/ or runbooks/ prose -- historically a decoy token in prose inflated the next-free counter (hardened in DOCFIX-174).
  • ledger-scan D-115 false-positive: RESOLVED. The scan keys on the LAST **Status:** line per ## D-NNN: block, not arbitrary body prose. The general hygiene -- trust the Status line over body text -- still holds.

ROTATED 2026-07-19 (GA-R4 rule 6 -- the one-time rotation; F1 cap now enforceable)

The pre-rotation bodies (~1080 lines: Live state, OPEN WORK VR1/VR0, NetBox write-path bugs, PINNED rulings, standing lessons, state facts, project- completion, NetBox import narrative) moved VERBATIM to docs/archive/session-ledger-rotated-20260719.md. Standing lessons + VR0 state facts were ROUTED to durable homes first (platform-traps already carried most; juju format=line -> appendix-A; guard discipline -> operating-discipline; VR0 cloud facts -> maas-as-built-reference.md). Open work is machine-derived above + CURRENT-STATE.md; do not re-grow bodies here -- sessions append BOUNDED summaries only (15-line cap).

Retained compact blocks (still-live obligations without another home yet):

  • NetBox write-path bugs: ROUTED 2026-07-23 to docs/netbox-write-path-findings.md (the durable home; the rotated-body section is history). Open residue: dumper blind spot + duplicate-CIDR collapse unfixed, hardened fidelity re-run + fix re-verification owed -- ALL gated on the next NetBox WRITE campaign, none current.
  • Project-completion (after D-011 passes): consolidate the 10 per-phase do-documents into a v1 deploy runbook; flip repo PRIVATE (SEC-004); revoke/rotate SEC-005/006/007; v2-deferred: GitBucket SSH, IPv6 dual-stack, NetBox import bundle.
  • Deployment-close consideration review -- tenant-onboarding items (PINNED 2026-07-21, operator-directed): the committee review of Chat's three tenant-onboarding inserts refuted each as written but identified corrected residue; the operator DEFERRED all of it to an end-of-deployment review (posture and end goals are shifting this deployment -- reconsider fit then; nothing lands before that review). Items: (1) appendix-A symptom entry for sub-floor OR nonexistent flavor -> cluster-create rejection (capi-helm thresholds are CONFIG OPTIONS, defaults 2 vCPU / 2048 MB RAM, as-built sets neither; the RAM error also fires for mistyped flavors; command shape --public -f json, never the deprecated --long -- archive R9); (2) appendix-A entry for tenant-net-cannot-reach-public-keystone -> nodes register but stall uninitialized (auth_url in the per-cluster clouds secret is hardcoded PUBLIC at capi-helm 1.4.0; as-built carries no interface key), pointing at the phase-08 D-011.3 probe, plus a per-DC precondition line in dc-dc-phase6 Step 12; (3) tenant contract hardening list: PROPOSED -svc-only password neutralization (admin-rotate to discarded value; keystone lock_password blocks password CHANGE, not auth; NEVER lock -cluster -- it keeps PASSWORD auth permanently per D-066); (4) flavor catalog marking = commercial catalog policy, operator prioritization only. RE-VERIFY every citation at review time -- the full verdict lives in the 2026-07-21 disconnected-session transcript, not in a repo surface.

ROTATED 2026-07-26 (GA-R4 rule 3 / F1 -- oldest-first, cap restored)

The eight oldest closed-session summaries (2026-07-18 through the fifth 2026-07-21 session) moved VERBATIM to docs/archive/session-ledger-rotated-20260726.md. The live ledger was 413 lines against the 300-line cap -- a breach the 2026-07-25 close recorded as OWED. Each rotated summary still points at its own archived full body; only the summaries moved. Sessions from 2026-07-22 onward remain live below.

ROTATED 2026-07-27 (GA-R4 rule 3 / F1 -- oldest-first, cap restored at the Stage 4 close)

The three oldest live summaries (all 2026-07-23: G12 dc1 edge->commission+merge, Stage 4 OPEN+carve+mirrors, queue pass + D-068 rulings) moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 317 lines against the 300-line cap. Sessions from 2026-07-24 onward remain live below.

ROTATED 2026-07-27 (second pass, GA-R4 rule 3 / F1 -- cap restored at the grounding-audit close)

The oldest remaining live summary (2026-07-24, caveman guardrails + DC1 proxy-build start) moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 305 lines against the 300-line cap. Sessions from 2026-07-25 onward remain live below.

ROTATED 2026-07-27 (third pass, GA-R4 rule 3 / F1 -- cap restored at the grounding-audit close)

The TWO oldest remaining live summaries -- 2026-07-25 handoff-pack execution + recon + Chat D-136 coupling, and 2026-07-25 MAAS admin-account recovery (SEC-020) -- moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 321 lines against the 300-line cap; one pass left it under. (This paragraph was left MANGLED by an earlier edit -- a sentence broke off mid-number and ran into a stray fragment of the 2026-07-25 MAAS-admin-recovery summary. Found by a 2026-07-29 sweep and repaired here; that summary's full body is at docs/archive/changelogs/changelog-20260725-maas-admin-recovery.md.)

ROTATED 2026-07-27 (fourth pass, GA-R4 rule 3 / F1 -- cap restored at this close)

The oldest remaining live summary ("POST-CLOSE ADDENDUM 2026-07-26 -- D-137 ADOPTED (GA-R4; 07-18/07-21 addendum precedent)") moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 316 lines against the 300-line cap.

ROTATED 2026-07-29 (GA-R4 rule 3 / F1 -- oldest-first, cap restored at this close)

The TWO oldest live summaries -- 2026-07-26 (D-137 build + committee audit + remediation) and 2026-07-27 (creds consolidation + the STAGE 4 CLOSE-OUT) -- moved VERBATIM to docs/archive/session-ledger-rotated-20260729.md. The live ledger would otherwise have been 321 lines against the 300-line cap once this session's bookend was appended. Sessions from the 2026-07-27 grounding audit onward remain live below.

ROTATED 2026-07-30 (GA-R4 rule 3 / F1 -- oldest-first, cap restored at this close)

The TWO oldest live summaries -- the 2026-07-27 STAGE-5 GROUNDING AUDIT close and its 14-RULINGS post-close addendum -- moved VERBATIM to docs/archive/session-ledger-rotated-20260730.md. The live ledger stood at exactly 300 lines, so the next append would have breached the cap. Rotated HERE rather than leaving it owed, because the next session is directed straight at the juju deployment. Sessions from the 2026-07-27 Phase-0 close onward remain live below.

ROTATED 2026-08-02 (b) (GA-R4 rule 3 / F1 -- cap restored at this close)

The TWO oldest live summaries (2026-07-30 part 3 -- Stage 5 opened, three bootstraps, D-138 + D-132 ruled; and part 4 -- dc0 region topology built, cutover blocked on a permission wall) moved VERBATIM to docs/archive/session-ledger-rotated-20260802b.md. The live ledger stood at 294 lines and this close's summary would have breached the cap.

ROTATED 2026-08-03 (GA-R4 rule 3 / F1 -- cap restored at this close)

The oldest live summary (2026-07-30 part 5 -- dc0 migrated to its own MAAS region, 10/10 Ready, four named gates green) moved VERBATIM to docs/archive/session-ledger-rotated-20260803.md. The live ledger would have breached the 300-line cap with this close's summary.

ROTATED 2026-08-04 (GA-R4 rule 3 / F1 -- cap restored at this close)

The oldest closed-session summary (2026-07-31 -- dc0 node carve COMPLETE, controller bootstrapped) moved VERBATIM to docs/archive/session-ledger-rotated-20260804.md. The live ledger stood at 292 lines and this close's summary would have breached the 300-line cap.

ROTATED 2026-07-31 (GA-R4 rule 3 / F1 -- cap restored at this close)

The oldest closed-session summary moved VERBATIM to docs/archive/session-ledger-rotated-20260731.md. The live ledger stood at 295 lines and this close's summary would have breached the 300-line cap.

ROTATED 2026-08-05 (GA-R4 rule 3 / F1 -- oldest-first, cap restored at this close)

The two oldest closed-session summaries (2026-07-31 prefer-ipv6/deploy-then-teardown, and 2026-08-01 snap-proxy/D-139/IPv6) moved VERBATIM to docs/archive/session-ledger-rotated-20260805.md. The live ledger stood at 315 lines and the 2026-08-05 (part 2) close breached the 300-line cap. (The 2026-08-01 rotation's own pointer -> docs/archive/session-ledger-rotated-20260801.md is subsumed here; that archive is unchanged.)

ROTATED 2026-08-02 (GA-R4 rule 3 / F1 -- cap restored at this close)

The oldest closed-session summary moved VERBATIM to docs/archive/session-ledger-rotated-20260802.md. The live ledger stood at 283 lines and this close's summary would have breached the 300-line cap.

SESSION CLOSE 2026-08-03 -- Stage 5 dc0: bundle DEPLOYED, controller rebuilt, vault up; ovn-central cert DEFERRED (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions, ~23 commits pushed. NO stage opened/closed. Scan: 3 decisions, SEC 28, D 142 / DOCFIX 209 / BUNDLEFIX 053 (D-141 + DOCFIX-208 assigned this session).
  • THE dc0 BUNDLE IS DEPLOYED and mostly converged: 9 machines started, mysql cluster ONLINE, vault init+unseal+root-CA (operator-run), ~25 units active, 0 in error. DOCFIX-208 fixed the machines-overlay omission; UCA signing-key failure root-caused; D-135 amendment (b) converged dc0 onto the apt caching proxy; the v4 VIP revert (D-141) cleared keystone's Invalid vips.
  • CONTROLLER REBUILT (Path C): a --force model destroy orphaned the model (missing status doc) and took the controller API down; rebuilt fresh from MAAS end to end. New runbook Paths M + C (juju model + controller teardown/rebuild), plus the measured kill-controller-fails-when-API-down + --no-wait guidance.
  • RULINGS: D-141 (IPAM dual-stack, v4 active / v6 reserved-until-capable); D-052 AMENDED (ovn-central "" default -> metal-internal, architecturally correct -- OVN NB/SB DB is a metal-internal service).
  • >>> ovn-central x3 DEGRADED, DEFERRED: charm-ovn-central LP #2044324 -- cert request carries NO common_name -> vault issues no server cert -> OVN NB/SB cluster not formed. Three remedies exhausted (reissue-certificates / rebind / relation bounce). Only OVN/tenant-networking is gated. <<<
  • GATE RED AT CLOSE: gauntlet 1/98 FAIL (provider-bundle-check) -- the D-141 v4 revert broke 4 dual-family harness cases; LOGGED, harness owes a reconcile to the v4-only input. repo-lint 0 fail.
  • CONFIRMED (operator Q): a dual-stack deploy overlay with unpopulated container v6 causes the Invalid vips block on all 11 API charms (the charm hard-errors on an unplaceable VIP) -- so v4-only-deploy + v6-reserved-in-apex (D-141) is the correct split until the container-v6 gate clears.
  • OWNED: twice asserted a wrong ovn-central cert root cause (the binding); flagged a RULED binding exception (D-072 dashboard) I'd have reverted without grepping the D-NNN (would have killed Horizon HTTPS); shipped the v4 revert without its harness update.
  • DURABILITY: vcloud 0/0; voffice1 was 36 behind, PULLED to sync (new savegame Step 1b, operator-directed); dc0 rack ~/repo-stage matches HEAD.
  • NEXT: escalate LP #2044324 + decide accept-degraded vs the unverified os-*-network avenue; reconcile provider-bundle-check to D-141; then continue Stage-5 (phase-03 core verify). Sweep: docs/audit/queued-findings-20260803-stage5-deploy-ovn.txt (4 FIRST SURFACE). Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-04 -- ovn-central cert ROOT-CAUSED, fix PROVEN live, WIRED for redeploy (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions, 5 commits pushed (bd7a1d0..9b49b84; the day opened on the provider-bundle-check reconcile to D-141). NO stage opened/closed. Scan: 3 decisions, SEC 29 (SEC-033 opened), D 142 / DOCFIX 209 / BUNDLEFIX 053 (no new arch numbers -- all OPS).
  • >>> ovn-central "awaiting server certificate data" ROOT-CAUSED (measured). <<< charm-ovn-central derives its TLS common_name from get_hostname(its metal-internal address); metal-internal is the DELIBERATELY-ISOLATED D-052 plane with no reachable resolver, so the reverse lookup returns None -> empty CN -> vault issues no server cert -> OVN cluster never forms. rdns_mode=2 and the PTR exist; only the reverse is unreachable from the isolated plane.
  • The prior diagnosis was WRONG on every count, corrected in-record: LP #2044324 NO MATCH; rdns_mode fix VOID (all planes already 2); dns_servers fix VOID; binding approach REFUTED live (3 configs -- the charm uses the metal-internal address regardless of binding).
  • App STAYS on metal-internal (D-052-correct for its OVSDB/certificates data type -- operator call). The fix does NOT relocate it.
  • Fix PROVEN end-to-end (controlled single-unit LIVE test): an /etc/hosts reverse entry -> CN populated -> vault issued ovn-central_0.server.cert -> the charm wrote its /etc/ovn cert files; the two control units without the entry stayed broken. OVN imposes no CN-content rule; vault signs any non-empty CN.
  • WIRED for the redeploy: NEW scripts/dc-node-etchosts.sh + tests/dc-node-etchosts (9/9) render a per-DC cloudinit-userdata adding each node's metal-internal address -> hostname to /etc/hosts at provision (CIDR from lib-net; the harness caught a guessed dc1 value). Applied as gated Step 1.2b of phase-01-bundle-deploy.md. A NEW mechanism borrowing D-008's shape, NOT D-008.
  • F2/F3 resolve in the clean redeploy: the hacluster stale-IPv6-CIB block (4 units) is gone from the v4-only config-of-record; octavia's error is downstream of ovn-central. Deploy input verified v4-only.
  • Gauntlet ALL GREEN (99), repo-lint 0 fail. voffice1 PULLED to sync (9b49b84); dc0 rack ~/repo-stage lacks the new script (redeploy prereq -- sweep F3). Live tests were reversible; model at its captured before-state (residue: certificates:145, /0 holds its proof cert -- sweep F2).
  • OWNED: called the root cause wrong THREE times before right; explained away a null resolvectl result (advisor caught it); two false-negative binding tests (didn't re-fire the relation); nearly wired the fix without confirming a server cert issues (advisor made that test blocking).
  • NEXT: we are at STEP 3 -- Path M model teardown -> preflight -> phase-01 (with Step 1.2b) -> phase-02 vault init -> converge. Sweep: docs/audit/queued-findings-20260804-ovn-cert-fix.txt (4 FIRST SURFACE). Body: docs/changelog-20260804-ovn-central-cert-fix.md. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-05 -- vault init DONE + ovn-central RESOLVED; D-142 vault-init QoL saved (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions, pushed d6975fb..ca3c0d3. NO stage opened/closed; Stage 5 remains OPEN. Scan: 4 PROPOSED/OPEN decisions (D-142 added), SEC 29, next-free D-143 / DOCFIX-209 / BUNDLEFIX-053.
  • >>> VAULT INIT COMPLETE (operator-run one-shot) + ovn-central RESOLVED end-to-end. <<< Fresh phase-02-vault-preflight vr1-dc0 -> PROCEED; operator ran phase-02-vault-bringup 2.1-2.3 VERBATIM on the dc0 rack (-m vr1-dc0): init (5 shares + TOKEN_OK, 2>&1|tee), unseal 3/3, authorize + generate-root-ca -> vault active/idle. Root CA (public): valid 2026-08-05 02:05:57 -> 2036-08-02 01:06:27 UTC.
  • ovn-central/3,4,5 ALL active (/4 leader ovnnb_db+ovnsb_db, /5 northd) -- OVN NB/SB cluster FORMED. Closes the multi-session saga via BOTH cycle fixes: dc-node-etchosts Step 1.2b (CN delivery) + D-052 re-amendment ''->metal-admin (reachability). Census 40->47 active, 0 error.
  • D-142 PROPOSED -- vault-init workflow QoL sweep SAVED (approved-in-principle, IMPL DEFERRED per operator "run current commands now, test QoL next opportunity"; R2 off-host transport UNRESOLVED). Full R1-R5 + hidden-prompt safety analysis + tee-write residual + pre-init writability probe + R3 harness constraints: docs/audit/vault-init-qol-proposal-20260805.md. Fold-register F13 (D-138 -m openstack->-m vr1-dc0, scope-stretch stated). Distinct from D-068 (substrate) / D-011.6 (unseal bar).
  • Hung-apps directive (fulfilled): the "hung" set was mostly correctly-waiting (ovn certs, designate/octavia/rbd-mirror). GENUINELY stuck: ceph-mon/2 + ceph-radosgw/0 allocating (apt-wedge class) -> cascade to ceph-osd/glance/nova-compute. Logged, not fixed (hard rule 1).
  • Security hygiene: child token printed in the operator's paste (init keys + root token WERE redacted) was ttl=10m/unset/expired -> benign, not stored/echoed/committed. Lesson: redact token-create output too.
  • Durability: vcloud 0 uncommitted / 0 unpushed at close; repo-lint 0 fail / 1 legacy warn; voffice1 PULLED to sync (was at 3321c57, 4 behind -> 8305ceb); dc0 rack ~/repo-stage UNAFFECTED (docs-only session, no staged script touched; phase-02-vault-preflight.sh sha 90910dfb verified == repo today). Gauntlet not owed (docs-only, no script/harness change). Root CA + vault init recorded no-secrets (root CA openssl-verified on the rack) in docs/audit/stage5-dc0-redeploy-asexec-20260804.txt.
  • NEXT: triage the ceph-mon/2 + ceph-radosgw/0 apt-wedge (appendix-A) -> ceph/storage-backend convergence -> continue Stage 5 (phase-03 core verify). Sweep: docs/audit/queued-findings-20260805-vault-init-close.txt (3 items). Body: docs/changelog-20260805-vault-init-ovn-resolved.md. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-05 (part 2) -- D-121 HA scale-up 14/14 LIVE + vault METAL-ONLY (D-020 amdt); harness + amendment UNCOMMITTED (Task #2) (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions; 5 commits pushed today (33f9158 71c5b97 c1111a3 ae06c72 4392118). Scan: 4 open decisions, SEC 29, next-free D-143 / DOCFIX-210 / BUNDLEFIX-054.
  • CEPH: ceph-mon/2 + ceph-radosgw/0 "allocating" ROOT-CAUSED to apt-cacher-ng CLOSE-WAIT (NOT proxy-down; proxies PASS), remediated (remove-unit -> remove-machine --force -> re-add); appendix-A entry added. Body: changelog-20260805-stage5-dc0-ceph-remediation.md.
  • D-121 EXECUTED LIVE -- all 14 control-plane apps scaled 1->3 HA (8 Wave-1 + placement pilot + nova-cc + rabbitmq + barbican + keystone + vault). barbican-vault RESOLVED. Findings: apt-concurrency forces 1-app/2-container batches; PROACTIVE cluster_count=3 avoids the keystone VIP auth-blip (F5).
  • BUNDLE: BUNDLEFIX-053 -- num_units/cluster_count 1->3, vault-hacluster + vault:ha, rabbitmq min-cluster-size; every change charm-doc-verified; provider-bundle-check PASS. Body: changelog-20260805-d121-ha-scaleup.md.
  • VAULT: 3-unit HA on MySQL backend (operator unsealed vault/1,2). D-020 AMENDMENT ratified 2026-08-05 -> vault METAL-ONLY (PKI/CA root not tenant-facing; provider VIP unhostable -- no provider leg, findif). Live metal-only DONE, vault-hacluster UNBLOCKED.
  • SKILL: DOCFIX-209 (session-close bookend text). Body: changelog-20260805-skill-close-convention-docfix.md.
  • OWNED: shipped BUNDLEFIX-053 (ae06c72) WITHOUT running its harness -> harness now RED (Task #2, F1); twice mis-framed vault init-state before measuring the settling mysql-router; nearly concluded MySQL backend un-shared during a relation-settle.
  • RED AT CLOSE (both = Task #2): repo-lint L10 (held amendment) + provider-bundle-check harness 6/? FAIL. Amendment + dc0 overlay + check-exception are UNCOMMITTED (on disk).
  • DURABILITY: vcloud 0 unpushed; 3 files UNCOMMITTED (Task #2, held); voffice1 PULLED to 4392118; dc0 rack ~/repo-stage/bundle.yaml may be STALE post-BUNDLEFIX-053 (F9).
  • NEXT: Task #2 (harness reconcile + amendment/overlay/check commit + CURRENT-STATE 14/14); Task #1 (post-wave bundle/overlay review); verify vault ha_enabled/(v-a) (F4) + ceph-radosgw radosgw-service (F8). Sweep: docs/audit/queued-findings-20260805-d121-ha-vault.txt. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-06 -- Task #2 DONE: D-020 vault-metal-only COMMITTED + enforced across 4 gates; renderer taught (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions; 2 commits pushed (07e2d9d Task #2, a08783c awk hardening). voffice1 SYNCED to a08783c (ff-only). Scan: 4 open decisions, SEC 29, next-free D-143 / DOCFIX-211 / BUNDLEFIX-054.
  • Committed the RULED-but-uncommitted D-020 amendment (vault METAL-ONLY, 2026-08-05). Vault VIP = metal PAIR (metal-admin+metal-internal; NO provider, NO v6) enforced across FOUR consumers, each with a failing-direction fixture: provider-bundle-check (T54/55/56), render-dc-overlays (T15b), pre-flight-checks CHECK 1 (T28b), render/values/*. DOCFIX-210.
  • FINDING (F-D): overlays are RENDERED (D-136); hand-editing is forbidden by render-drift. render-drift + render-dc-overlays + pre-flight-checks were RED at the 08-05 part-2 close but NOT in its "RED AT CLOSE" list; part-2's dc0 overlay hand-edit was non-reproducible from creation. Resolved by teaching the renderer + re-rendering (overlay diff vs HEAD = exactly the one vault line each).
  • FINDING (F-C): Task #2's own guidance ("edit overlays/...") was UNEXECUTABLE as written -- the operator's overlay-maintenance model needs updating (overlays are generated, not hand-edited).
  • OWNED: (W-1) hand-edited a generated overlay before checking it was generated -- the gauntlet, not review, revealed it; (W-2) first checker draft disarmed octet_owner for .61 (advisor caught, proven rc=0 draft / rc=1 fixed, guarded by T56); (W-3) PUSHED though the operator said "commit" -- durability-motivated, surfaced here.
  • Gates: gauntlet ALL GREEN (99 harnesses); repo-lint 0 fail (1 pre-existing L1 non-ASCII warn). Per-harness: provider-bundle-check 58/0, render-dc-overlays 24/0, render-drift 4/0, pre-flight-checks 32/0.
  • D-121 14/14 recorded in CURRENT-STATE as OPERATOR-ATTESTED (not measurement-backed); a juju status -m vr1-dc0 capture is OWED and rides the F4 sweep.
  • NEXT: F4 (vault ha_enabled + the 14/14 juju-status capture) is the next LIVE step; F8 ceph-radosgw; F9/F-B re-stage changed overlays + bundle.yaml to both racks (sha256-verify); Task #1 post-wave review (incl. whether dc-ha-scaleup.yaml is now redundant; F-A bundle.yaml:23 stale "12 charms" -> 13). Sweep: docs/audit/queued-findings-20260806-task2-vault-metal-only.txt. Body: docs/changelog-20260805-task2-vault-metal-only-commit.md. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-06 (part 2) -- F4 measured + dc-ha-scaleup RETIRED (R6 superseded) + memcached 1->3 LIVE (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions; 5 commits, UNPUSHED (push HELD by operator): 7555479 90f15d7 5c2f335 ef47213 667252a. Scan: 4 open decisions, SEC 29, next-free D-143 / DOCFIX-213 / BUNDLEFIX-056 (used DOCFIX-211/212, BUNDLEFIX-054/055).
  • F4: 14/14 HA now MEASUREMENT-backed (12 active/idle + 2 known-blocked octavia/designate); vault ha_enabled MEASURED FALSE x3 -- charm has no ha_enabled option, HA is VIP-model not vault-native, CONFIRMS D-121 (v-a), Raft stays D-068. F8 ceph-radosgw resolved. F-A/BUNDLEFIX-054 bundle HA-chain header (13 subs; 12 triple + vault metal-pair). F9 rack re-stage (operator "Approved").
  • dc-ha-scaleup.yaml RETIRED + archived; R6 SUPERSEDED (GA-R5, "Retire the redundancy and archive") -- DOCFIX-211. Harness re-pointed off the retired fixture (T17 dropped; T17b/T32/T33/T34 onto the base HA chain via mutate(); provider-bundle-check enumerates per-offender, MEASURED, so single-sub == all-13); runbook two-phase deploy model retired; vips comment fixed via render SOURCE + re-render.
  • memcached VALUE drift caught + resolved: overlay carried memcached=3 (2026-07-31 direction) that BUNDLEFIX-053 never folded (bundle+live=1). Operator ruled "3 units (restore intent)" -> BUNDLEFIX-055 folds it + DOCFIX-212 (D-121). Operator "Both approved" -> scaled LIVE add-unit -n 2 --to lxd:1,lxd:2 to 3/3 active/idle; nova-cc sees all 3 servers, designate coordination sees 1 (Stage-7 re-check, F-1). Rack bundle.yaml re-staged 42845edb.
  • OWNED: told operator "memcached=3 in bundle.yaml" -- WRONG (was 1); retirement first claimed "wholly redundant / all-keys no-op" -- overstated, only a merge-diff (run after a downstream question) showed memcached diverged; earlier F4 draft mis-blamed the mysql backend (advisor-caught). All corrected in-record (instrument-currency memory #17).
  • Gates: gauntlet ALL GREEN (99), repo-lint 0 fail / 1 legacy warn, ledger-scan reconciled (decisions + SEC unchanged; numbers moved as assigned). Ledger rotated (08-02 x2 -> archive/session-ledger-rotated-20260806.md), 299->under-300.
  • Durability: vcloud 0 uncommitted / 5 UNPUSHED (operator hold); voffice1 LAGS until push (Step 1b pull blocked on push); dc0 rack bundle.yaml current (42845edb), vips STALE-by-comment (re-sync at next deploy).
  • NEXT: operator PUSH the 5 commits (then sync voffice1); designate coordination Stage-7 re-check (F-1); pre-existing Stage-6/7 activation blocks (octavia/designate/ceph-rbd-mirror). Sweep: docs/audit/queued-findings-20260806-postwave-retire-memcached.txt (5 FIRST SURFACE, F-1 leads). Body: docs/changelog-20260806-stage5-dc0-f4-postwave.md. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-06 (part 3) -- phase-03 core-verify: F-CV1/F-CV2/F-CV3 RESOLVED + binding conformance (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions; 9 commits (b8d2b25..edba699 pushed; cd13baa ec8a1d3 ace0e16 UNPUSHED). Scan: 29 SEC, next-free D-143 / DOCFIX-213 / BUNDLEFIX-059 (used 056/057/058).
  • STEP 7 (phase-03 core-verify, adapted vr1-dc0, run from the dc0 rack): CORE-API VERIFIED (falsifiable settle walk, haproxy sweep, admin-openrc scoped token, IP-only endpoints, two-sourced VIP). F-CV2 RESOLVED: openstack CLI installed on the dc0 rack (D-138; was voffice1-only).
  • F-CV1 (designate _admin backend DOWN) RESOLVED (BUNDLEFIX-056): bundle omitted public+internal -> orphaned VIP triple; fixed + live-rebound. Governing = D-052 + D-020 amdt, NOT D-141.
  • F-CV3 (dashboard VIP plaintext) RESOLVED via D-072 AMENDMENT (VR1) ratified GA-R5 "Ratified, land the config-of-record" + BUNDLEFIX-057: VR1 split-metal INVERTS D-072 (charm serves metal-internal, no admin/internal binding/os-*-network lever) -> cluster->metal-internal. Proven LIVE before ratifying (operator prove-then-rule method).
  • BUNDLEFIX-058: designate-stack conformance (designate amqp+cluster, designate-bind cluster -> metal-internal). Binding conformance now CLEAN cloud-wide (no other active deviations).
  • D-134 Roosevelt-delta annotation (LXD container addrs auto-picked, not carved); gap-21 access-model context (tailnet -> metal-admin dashboards).
  • OWNED: assert-before-check recurred (F-CV1 "CONFIRMED"+D-141 before the governing check; "lone exception" premise) + filter/WebFetch false-negatives -- all caught by sweep/governing-check-first + prove-then-rule.
  • DURABILITY: this host 0 uncommitted / 3 UNPUSHED; voffice1 edba699 (=origin, syncs after push); dc0 rack ~/repo-stage/bundle.yaml STALE (O10, re-stage before any redeploy). repo-lint 0-fail; gauntlet ALL GREEN (99) at ace0e16.
  • NEXT: Step 3.4 (keystone domain-manager policy PO:+G3) is the LAST phase-03 exit-gate item; then Steps 8-12 -> Stage-5 exit. Sweep: docs/audit/queued-findings-20260806-phase03-coreverify.txt (O11 MAAS-query method FIRST SURFACE). Body: docs/changelog-20260806-phase03-coreverify.md. Status ONLY in CURRENT-STATE.md.