Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260806-stage5-dc0-f4-postwave.md
@JANeumatrix JANeumatrix 14 hours ago 11 KB Stage 5 dc0: memcached 1->3 config-of-record [BUNDLEFIX-055 + DOCFIX-212]
Raw Blame History

Changelog 2026-08-06 -- Stage 5 dc0: F4 measurement-backed + F8/F9 + F-A header + Task #1 review

Session body (GA-R2, one per session). Branch dc-dc-stage5-preconditions. Stage 5 OPEN. Status lives ONLY in docs/CURRENT-STATE.md. All live reads were read-only (D-138 rack path); the one owed mutation (F9 rack re-stage) is gated on operator approval, NOT executed unattended (operator-away background session).

Item 1 -- F4: 14/14 HA now MEASUREMENT-BACKED + vault ha_enabled measured [commit 7555479]

  • Read-only juju status -m vr1-dc0 captured FROM the dc0 rack (D-138 path ssh vr1-dc0-rack -> juju). Capture: docs/audit/stage5-dc0-juju-status-14of14-20260806.txt.
  • 14/14 D-121-enumerated HA apps at scale=3 (graded against changelog-20260805-d121-ha-scaleup.md:24, NOT reverse-engineered). HONEST SPLIT (GA-R6 E3, not rounded): 12 active/idle; 2 blocked-at-scale=3 on KNOWN non-HA items -- octavia (configure-resources pending) + designate (nameservers must be set). Model-wide 152 active/idle, 7 blocked, 1 unknown (glance-simplestreams-sync, normal). Converted the CURRENT-STATE 14/14 line OPERATOR-ATTESTED -> MEASUREMENT-BACKED (C2).
  • vault ha_enabled MEASURED FALSE x3 (vault status; unauthenticated, non-secret). CAUSE (measured -- corrected a mid-session draft that mis-blamed the mysql backend as HA-incapable, advisor-caught): the vault charm renders storage "mysql" with NO ha_enabled and exposes NO such config option (HA options are only vip + dns-ha-access-record). Vault's HA here is the charm/VIP model, NOT vault-native active/standby -- all 3 units show active: true; the hacluster VIP constrains live traffic to one. This CONFIRMS D-121 (v-a) as-built; vault-native HA (Raft) stays owned by D-068 (already the (v-b) alternative in D-121). No new decision.
  • Revert: git revert 7555479 (record-only; no live-cloud change).

Item 2 -- F8: ceph-radosgw RESOLVED [in capture file]

  • 3 units active/idle "Unit is ready" (80/tcp). The prior "Services not running: radosgw" was the new-unit stale-status class; converged. No action. Revert: n/a (read-only).

Item 3 -- F9/F-B: staging drift MEASURED; re-stage GATED [task #3, pending operator approval]

  • dc0 rack ~/repo-stage: bundle.yaml (staged 213b51ba) and overlays/vr1-dc0-vips.yaml (staged cf42b937) are STALE vs HEAD (bundle 20ca73a6 pre-F-A / vips 3f403408); dc1 rack has NO ~/repo-stage (dc1 HELD/undeployed). Re-stage matters only at the NEXT dc0 deploy (running cloud already converged). Prepared + gated -- rack write not executed unattended.
  • NOTE: F-A (Item 4) changes bundle.yaml, so the re-stage MUST carry the post-F-A bundle.
  • First attempt DENIED by the auto-mode classifier (hard rule 3 gate). OPERATOR APPROVED 2026-08-06; re-run SUCCEEDED. Both files sha256-verified on the dc0 rack == repo HEAD: bundle.yaml=754f1d9e (post-F-A), vr1-dc0-vips.yaml=3f403408. dc0 rack ~/repo-stage now current. dc1 rack has no ~/repo-stage (HELD) -- future standup work, not owed now. Revert: re-stage any prior committed bundle/overlay from git (staging is a copy of tracked files).

Item 4 -- F-A / BUNDLEFIX-054: bundle.yaml HA-chain header corrected [OPS]

  • bundle.yaml:23 HA-chain header said "ACTIVE for 12 charms ... 11 carry the dual-VIP pattern (designate ... no HAProxy VIP)". THREE stale claims after BUNDLEFIX-053 + D-020 amendment + R11, all corrected FROM the measured merged dc0 deploy input: (a) 12 -> 13 hacluster subordinates/:ha relations (vault-hacluster added by BUNDLEFIX-053; 
    provider-bundle-check reports "13 hacluster subordinate(s)").
    (b) VIP-carriers: ALL 13 carry a VIP -- **12 the full provider+metal-admin+metal-internal triple, 
    vault a METAL-ONLY pair** (.61, no provider -- D-020 amendment). Measured from
`overlays/vr1-dc0-vips.yaml`.
    (c) The "designate has no HAProxy VIP" note is RETIRED -- designate gained a full triple VIP 
    (.62) under R11.
    Also clarified rabbitmq-server is the 14th HA app but native-clustered (no hacluster/VIP), so not in the 13. COMMENT-ONLY; no deploy semantics change. gauntlet + repo-lint re-run green.
  • Revert: restore the prior 6-line header block (git revert this commit).

Item 5 -- Task #1 (post-wave review): dc-ha-scaleup.yaml is now REDUNDANT -- RECOMMENDATION, not executed

  • MEASURED: bundle.yaml (via BUNDLEFIX-053) carries the HA content of overlays/dc-ha-scaleup.yaml -- num_units:3 + to:[lxd:0,1,2] on the 14 HA apps, cluster_count:3 on the 12 hacluster subs, vault-hacluster declared (line 1036), rabbitmq min-cluster-size:3 (170), and the [vault:ha, vault-hacluster:ha] relation (1058). CORRECTION 2026-08-06: NOT wholly redundant as first stated. A full merge-diff (Item 7) found ONE divergence -- memcached (overlay num_units:3, base 1), which BUNDLEFIX-053 did NOT fold. That is resolved by BUNDLEFIX-055 (Item 7); with it, the overlay is genuinely redundant.
  • The overlay header itself records "NOT YET APPLIED to any live model"; the live 14/14 scale-up was done via per-app juju add-unit, never via this overlay.
  • RECOMMENDATION was put to the operator; RULED 2026-08-06 -- see Item 6.

Item 6 -- RETIRE + ARCHIVE dc-ha-scaleup.yaml (operator-ruled 2026-08-06; R6 SUPERSEDED) [DOCFIX-211]

  • Operator ruling, exact utterance: "Retire the redundancy and archive." Recorded as a GA-R5 SUPERSESSION note on D-121's R6 ruling (docs/design-decisions.md), which retains the original 2026-07-27 record below it as history. The overlay was redundant EXCEPT for one field: BUNDLEFIX-053 folded the 14 HA apps + vault-hacluster + rabbitmq + the vault:ha relation (an all-keys no-op there); a full merge-diff (Item 7) later found it ALSO carried memcached num_units:3 which BUNDLEFIX-053 missed (base stayed 1). BUNDLEFIX-055 (Item 7) folds memcached, making the archived overlay fully redundant. The retirement stands; the fold completes it.
  • Archived: git mv overlays/dc-ha-scaleup.yaml docs/archive/dc-ha-scaleup-RETIRED-20260806.yaml
    • a RETIRED banner (do-not-deploy). NO prior precedent for archiving an overlay (the only past retirement, vr0-dc0-testcloud.yaml, went to a one-time remove/ dir); docs/archive/ chosen explicitly as the repo's archive location.
  • Harness (tests/provider-bundle-check/run-tests.sh) re-pointed off the retired fixture: T17 REMOVED (once vault-hacluster moved to base it became identical to T18); T17b/T32/T33/T34 RE-POINTED onto good.yaml (the base HA chain) -- T33/T34 now use the deterministic mutate() helper instead of sed-on-overlay (a wrong key path KeyErrors loudly; no silent no-op fixture). Each re-pointed FAIL case (run 1) proven to fire against the real checker message (captured standalone before editing). Count 58 -> 57 (T17 dropped). provider-bundle-check 57/0.
  • Docs reconciled (present-structure only; dated history left as history): design-decisions.md (R6 supersession note + :503 + the :4563 "overlay encodes v-a" ref, now bundle.yaml/BUNDLEFIX-053); runbook dc-dc-phase4 Step-4 note + Step-12.3(a) gate (the two-phase "base at cluster_count:1, then apply overlay to reach 3" model RETIRED -- base now deploys 3-unit HA directly, so cluster_count:3 is the expected post-Step-4 value); dc-dc-deployment-workflow.md item 22; scripts/provider-bundle-check.py + scripts/cloud-assert.sh comments.
  • Rendered vips overlays: the "land BEFORE dc-ha-scaleup.yaml" comment lives in the RENDER SOURCE (render/values/vr1-dc{0,1}-vips.yaml, D-136). Edited the SOURCE and RE-RENDERED (never hand-edited the overlay -- the W-1 trap); overlay diff vs HEAD is exactly the two comment lines; render-drift 4/0. Hand-maintained vr1-dc{0,1}-machines.yaml deploy-command comments edited direct.
  • CURRENT-STATE: NO edit needed -- its active Stage-5 status already lists the deploy input as "bundle + -vips/-machines/-octavia-pki overlays" (no HA overlay), consistent with BUNDLEFIX-053. The remaining dc-ha-scaleup mentions are dated historical narrative (2026-07-27 captures, NEW-7 findings) and are retained as history (not present status; GA-R1).
  • Staging note: the dc0 rack ~/repo-stage/overlays/vr1-dc0-vips.yaml re-staged in Item 3 is now stale-by-ONE-COMMENT vs the re-render (functionally identical -- no VIP/data change). Not re-staged (a comment does not warrant another gated rack write); it re-syncs + sha-verifies at the next real deploy per D-138. OWED-at-next-deploy, logged.
  • Gates: gauntlet run (see close); repo-lint 0 fail (1 pre-existing L1 warn); provider-bundle-check 57/0; render-drift 4/0.
  • Revert: git mv the overlay back to overlays/ (strip the banner); restore the 5 harness cases and the doc/comment blocks (git revert this commit); re-render the vips overlays from reverted values.

Item 7 -- memcached VALUE drift caught by the retirement; folded to 3 [BUNDLEFIX-055 + DOCFIX-212]

  • Found while reconciling a stale comment (operator asked for currency ahead of a possible teardown+redeploy). A full merge-diff of the archived overlay vs bundle.yaml (measured 2026-08-06) showed the ONLY divergence was memcached: overlay num_units:3 + to:[lxd:0,1,2] vs base num_units:1 + to:[lxd:0]. BUNDLEFIX-053 folded the HA chain but NOT memcached; the overlay's memcached:3 (operator-directed 2026-07-31) was never folded and never deployed (git: bundle.yaml memcached==1 at ae06c72 and HEAD; F4 capture: live memcached==1).
  • Owned: last turn I told the operator "memcached=3 is in bundle.yaml:1044" -- WRONG (it was 1); and the retirement changelog/commit overstated BUNDLEFIX-053 as folding memcached / the overlay as a full no-op. Both corrected above (Items 5, 6).
  • Operator ruling 2026-08-06 (GA-R5): "3 units (restore intent)." So BUNDLEFIX-055 folds memcached into bundle.yaml (num_units:1->3, to:[lxd:0]->[lxd:0,1,2]) -- completing the fold BUNDLEFIX-053 missed and making the archived overlay genuinely, fully redundant. NO hacluster/VIP (3 independent caches; clients hash across the full server list). The overlay's rationale comment (previously "the only place the exclusion set is written down") is now carried in bundle.yaml.
  • DOCFIX-212: design-decisions.md D-121 "Left single (NOT scaled)" list corrected -- memcached removed from it and recorded as =3.
  • LIVE/CONFIG GAP (recorded in CURRENT-STATE): live memcached is still 1; config-of-record is now 3. A teardown+redeploy from bundle.yaml brings it up at 3; alternatively a gated live juju add-unit memcached -n 2 --to lxd:1,lxd:2. Not done live this turn (hard rule 1; no operator instruction to mutate live).
  • Re-stage owed: this is a FUNCTIONAL bundle.yaml change (not a comment), so the dc0 rack staged bundle.yaml (754f1d9e) is now genuinely stale -- re-stage + sha-verify (gated) OR the redeploy re-stages. Logged.
  • Gates: gauntlet + repo-lint (see close); provider-bundle-check re-run (memcached has no hacluster/VIP/arity constraint, so the scale is orthogonal to those gates).
  • Revert: bundle.yaml memcached back to num_units:1 + to:[lxd:0]; revert the D-121 DOCFIX-212 line.