Newer
Older
openstack-caracal-dc-dc / docs / audit / queued-findings-20260806-postwave-retire-memcached.txt
QUEUED FINDINGS -- session 2026-08-06 "Stage-5 dc0: F4/F8/F9 + F-A + retire dc-ha-scaleup + memcached 1->3 live"
Sweep per savegame Step 3. FIRST SURFACE items lead. Body: docs/changelog-20260806-stage5-dc0-f4-postwave.md
Commits (UNPUSHED, push held by operator): 7555479, 90f15d7, 5c2f335, ef47213, 667252a. Status ONLY in docs/CURRENT-STATE.md.

================================================================================
FIRST SURFACE -- originated in this session; recorded by this session's commits/captures
================================================================================

F-1. **[OPERATOR-FLAGGED] designate coordination points at ONE memcached unit, not 3.** After
     the memcached 1->3 live scale-up, nova-cloud-controller picked up all 3 servers
     (memcache_servers = .115,.166,.165:11211) but designate/leader designate.conf still has
     backend_url = memcached://10.12.12.115:11211 (memcached/0 only). designate is workload-BLOCKED
     (nameservers/Stage-7), so its config is pre-activation and may re-render on unblock -- OR the
     designate tooz-memcached driver uses a single URL by design. RE-CHECK AT STAGE-7 designate
     activation whether coordination should span all 3 memcached units (a single-URL coordination
     backend is a single point of failure for the 3 designate units). RECORDED, not fixed (hard
     rule 1). Surface: docs/audit/stage5-dc0-memcached-scaleup-20260806.txt + CURRENT-STATE
     (memcached note) + changelog Item 7.

F-2. **vault ha_enabled MEASURED FALSE on all 3 units (F4).** The vault charm renders storage
     "mysql" with no ha_enabled and exposes no such config option (only vip + dns-ha-access-record);
     HA is charm/VIP model, not vault-native. CONFIRMS D-121 (v-a) as-built; vault-native/Raft HA
     stays owned by D-068. Operator note: 3 unsealed actives on one backend is safe only while the
     hacluster VIP is the sole ingress. Surface: CURRENT-STATE (F4 block) + changelog Item 1 +
     capture stage5-dc0-juju-status-14of14-20260806.txt.

F-3. **memcached VALUE drift found + resolved.** The retired dc-ha-scaleup.yaml carried memcached
     num_units:3 (operator-directed 2026-07-31) which BUNDLEFIX-053 NEVER folded -- bundle.yaml +
     live sat at 1. Operator ruled "3 units (restore intent)"; BUNDLEFIX-055 folded it into the
     base bundle + DOCFIX-212 corrected D-121; live scaled to 3. Surface: changelog Items 5/6/7 +
     D-121 + CURRENT-STATE. This corrects two in-session misstatements (see OWNED).

F-4. **provider-bundle-check ENUMERATES per-offender (measured).** An all-13 cluster_count 3->1
     rewrite exits 1 with 13 DECORATIVE lines; a single-sub rewrite exits 1 with 1. So the
     re-pointed single-sub mutate() fixtures (T33/T34) are equivalent to the old whole-overlay
     rewrite for proving the arity gate FIRES -- not a narrowing. Surface: harness invariant-10
     comment (tests/provider-bundle-check/run-tests.sh).

F-5. **dc0 rack ~/repo-stage/overlays/vr1-dc0-vips.yaml is stale-by-ONE-COMMENT.** The vips overlay
     was re-rendered (comment-only) AFTER F9 staged it (3f403408); functionally identical, no
     VIP/data change. bundle.yaml WAS re-staged current (42845edb). Re-sync the vips comment at the
     next real deploy per D-138 (sha-verify). Surface: changelog Item 6.

================================================================================
ON SURFACE -- verified already recorded (where)
================================================================================

S-1. 14/14 HA measurement-backed (12 active/idle + 2 known-blocked); F8 ceph-radosgw resolved:
     CURRENT-STATE + changelog Items 1/2 + capture stage5-dc0-juju-status-14of14-20260806.txt.
S-2. F-A/BUNDLEFIX-054 bundle.yaml HA-chain header (13 subs; 12 triple + vault metal-pair):
     bundle.yaml:23 + changelog Item 4.
S-3. dc-ha-scaleup.yaml RETIRED + archived; R6 SUPERSEDED (GA-R5, "Retire the redundancy and
     archive"): design-decisions.md (R6 note) + docs/archive/dc-ha-scaleup-RETIRED-20260806.yaml +
     changelog Items 5/6 + DOCFIX-211. Harness re-pointed (T17 dropped; T17b/T32/T33/T34 onto base).
S-4. memcached 3 LIVE (add-unit, 3/3 active/idle) + re-stage (42845edb): capture + CURRENT-STATE +
     changelog Item 7.

================================================================================
OWED / NOT DONE (for the next session)
================================================================================

O-1. F-1 designate coordination Stage-7 re-check (above).
O-2. 5 UNPUSHED commits (push held by operator). voffice1 clone will lag until pushed; cannot be
     synced (savegame Step 1b) until this host pushes. dc0 rack ~/repo-stage: bundle.yaml current,
     vips stale-by-comment (F-5).
O-3. Pre-existing carried items unchanged this session: preflight P5 red (ruled-accepted, 6
     findings), the D-142 vault-init QoL (PROPOSED, impl deferred), F8-era ceph/octavia/designate
     blocks are Stage-6/7 activation work.

================================================================================
ALWAYS-SWEEP-5 (structurally invisible)
================================================================================

A-1. GITIGNORED STATE: no .claude/settings.local.json permission rule was added this session --
     the classifier gated the rack scp (F9) and the juju add-unit; both cleared by IN-BAND operator
     approval ("Approved" / "Both approved"), not a persisted rule. So NO gitignored permission
     state to preserve. The octavia-pki per-DC overlays remain gitignored (untouched this session).
A-2. DANGLING REFERENCES: all paths cited by this session's commits resolve -- the archived overlay,
     both new captures (juju-status-14of14, memcached-scaleup), the changelog. Verified.
A-3. RULING FIDELITY (exact utterances recorded): "Retire the redundancy and archive" (R6
     supersession, design-decisions); "3 units (restore intent)" (memcached, changelog Item 7 +
     D-121); "Both approved" (live add-unit + re-stage, changelog Item 7). All dated 2026-08-06.
A-4. AS-EXECUTED LOG GAP: run-logged.sh was NOT used (background session). The live mutations
     (juju add-unit memcached, the two rack scp re-stages) were run via direct `ssh vr1-dc0-rack`
     and are recorded in the changelog + captures rather than an as-executed script(1) log. DECLARED
     here as the log gap; the captures are the durable record.
A-5. CONTRADICTION DETECTOR: (a) bundle.yaml memcached=1 contradicted the claimed "=3" -> measured,
     folded to 3 (BUNDLEFIX-055). (b) D-121:55 "memcached=1 (optional 3)" contradicted the
     2026-07-31 operator direction -> DOCFIX-212. (c) vault "active:true" x3 vs HA Enabled=false ->
     resolved (charm VIP-HA, not vault-native). All reconciled, none left open.

================================================================================
OWNED (own-mistakes, corrected in-session)
================================================================================

W-1. Told the operator "memcached=3 is in bundle.yaml:1044" -- WRONG, it was num_units:1. Caught
     when reading the block directly for the currency fix. Corrected (changelog Items 5/6/7).
W-2. The retirement commit/changelog first claimed the overlay was "wholly redundant / all-keys
     no-op deep-merge." A full merge-diff (run only when the memcached question forced it) showed
     it diverged on memcached (3 vs 1). Corrected in the changelog; BUNDLEFIX-055 makes it true.
     LESSON: run the merge-diff BEFORE asserting redundancy, not after a downstream question forces it.
W-3. (earlier this session) First F4 draft mis-blamed the mysql backend as HA-incapable; the advisor
     caught it -- the real cause is the charm renders no ha_enabled option. Corrected before commit.
W-4. Nearly hand-edited the RENDERED vr1-dc0-vips.yaml comment before a render/ grep showed it is
     generated from render/values (D-136); edited the source + re-rendered instead (the W-1 render
     trap from the 2026-08-05 session, avoided this time by grepping render/ first).