Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260806-phase03-coreverify.md
@JANeumatrix JANeumatrix 6 hours ago 8 KB BUNDLEFIX-057 / D-072 AMENDMENT (VR1): dashboard cluster -> metal-internal (F-CV3 RESOLVED)
Raw Blame History

Changelog 2026-08-06 -- Stage 5 dc0: Step 7 phase-03 core verify (core-API layer)

Session-scoped (GA-R2). Branch dc-dc-stage5-preconditions. Stage 5 remains OPEN (this is NOT a stage close). Under blanket approval the changelog is the review surface: each item = what / why (evidence) / revert.

Evidence capture (all read-only measurement + the one gated install): docs/audit/stage5-dc0-phase03-coreverify-20260806.txt.

Item 1 -- INFRA: openstack CLI installed on the dc0 rack (F-CV2 resolve)

What. sudo apt-get install -y python3-openstackclient on the dc0 rack (vr1-dc0-rack, 172.31.0.2). Landed python3-openstackclient 6.6.0-0ubuntu2

  • python3-openstacksdk 3.0.0-0ubuntu2 from noble/main. Verified openstack --version -> openstack 6.6.0. Why. Step 7 (phase-03) is the first phase to invoke the openstack CLI, and per D-138 that CLI runs FROM the DC rack (no L3 path from voffice1 to the cloud). The 07-27 install landed only on voffice1. phase-03-admin-openrc.sh failed "openstack not found" on the rack. This is the documented remediation of the 07-30 queued-finding F1 ("Install on the DC client host before Step 7"); pin is the measured 07-27 value, snap refuted, noble-native Caracal so no UCA. Revert. ssh vr1-dc0-rack 'sudo apt-get purge -y python3-openstackclient python3-openstacksdk' (client-only; no service impact).

Item 2 -- STAGE: rack repo-stage gained phase-03-admin-openrc.sh + extract helper

What. scp staged scripts/phase-03-admin-openrc.sh (sha 2aeb20b720d83944) and scripts/extract_admin_password.py (sha 7b9b25a8fafd75cb) into ~/repo-stage/scripts/ on the dc0 rack; both sha256-verified == repo HEAD. Why. D-138 rack-run discipline: Step 7 runs the tested phase-03 admin-openrc builder from the rack's staged copy, sha-verified before trust. Revert. ssh vr1-dc0-rack 'rm ~/repo-stage/scripts/phase-03-admin-openrc.sh ~/repo-stage/scripts/extract_admin_password.py' (redeploy inputs, not live state).

Item 3 -- RACK STATE: admin-openrc + vault root CA built on the dc0 rack

What. MODEL=vr1-dc0 KEYSTONE_VIP=10.12.4.50 phase-03-admin-openrc.sh wrote ~/admin-openrc (0600) + ~/vault-init/vault-ca-root.pem on the rack; scoped token issued. Secret-adjacent files, on-rack only; password never entered context (script prints length only). Why. phase-03 Step 3.2 -- the IP-only admin credential + vault CA for API verification. Two-source keystone VIP confirmed (overlay == live endpoint). Revert. ssh vr1-dc0-rack 'rm ~/admin-openrc' (regenerable from live state).

Item 4 -- DOC: CURRENT-STATE section 7 OpenStack-client row amended (GA-R1/C1)

What. In-row amendment: client now INSTALLED ON THE dc0 RACK; the D-138 correction that phase-03..06 run from the rack, not voffice1; dc1-rack install OWED. Verify command + host cell updated. Why. GA-R1/C1 -- a commit that changes a status CURRENT-STATE carries updates it in the same commit. Real measured status change (client presence on the rack). Revert. git revert this commit's CURRENT-STATE hunk.

Item 5 -- DOC: CURRENT-STATE section 1 Stage-5 progress note (Step 7)

What. Added the Step-7 phase-03 core-verify progress block: core-API VERIFIED, exit gate OPEN on F-CV3 (dashboard TLS) + Step 3.4; F-CV1 retraction; F-CV2 resolve. Why. Stage/gate status lives in CURRENT-STATE only (GA-R1). Revert. git revert this commit's CURRENT-STATE hunk.

Item 6 -- BUNDLEFIX-056: designate binding fix (F-CV1 RESOLVED)

What. designate's REST API public + internal endpoints were on the '' metal-admin fallback (omitted from the bundle), orphaning the provider + metal-internal legs of its ruled .62 VIP triple and leaving the _admin haproxy backend SSL-DOWN on the unserved metal-internal address (F-CV1).

  • bundle.yaml: designate bindings +public: provider-public +internal: metal-internal; corrected the misleading dnsaas comment (it wrongly claimed designate has no public binding -- the deployed charm metadata declares public/admin/internal extra-bindings).
  • scripts/provider-bundle-check.py: EXPECT_PUBLIC_VIP 11->12 (designate joins; vault stays out, metal-only) + header + rationale rewrite (preserves the "not 13" warning).
  • tests/provider-bundle-check/run-tests.sh: +T16c (count 11 FAILS) +T16d (count 13 FAILS) -- both failing directions proven; harness 57->59, ALL PASS.
  • docs/network-space-binding-reference.md: row 88 (1->2 prov-pub, 6->7 m-internal) + a section-6 note.
  • LIVE (operator-approved): juju bind designate public=provider-public internal=metal-internal on the dc0 rack (rc=0). Charm re-rendered + reissued cert; settled to idle. Why. Conformance to the D-020 amendment (2026-07-27, designate's established provider/admin/internal triple) + the generic binding rule + every sibling API charm. Governing: D-052 / D-020 amendment. Root of the defect: the prior reading that designate had "no public binding, only dnsaas" -- dnsaas is ADDITIONAL, not a replacement. Verify. Full haproxy sweep 0 DOWN cloud-wide; designate apache https vhosts span all 3 planes; cert SANs now include provider-public; catalog triple correct (public 10.12.4.62 / internal 10.12.12.62 / admin 10.12.8.62). Gauntlet ALL GREEN (99); repo-lint 0-fail. Evidence: docs/audit/stage5-dc0-phase03-coreverify-20260806.txt (F-CV1 FIX section). Revert. juju bind designate public=metal-admin internal=metal-admin (live) + git revert this commit (bundle/checker/harness/reference). Both halves needed.

Item 7 -- BUNDLEFIX-057 / D-072 AMENDMENT (VR1): dashboard cluster -> metal-internal (F-CV3 RESOLVED)

What. Retired the VR0 D-072 openstack-dashboard cluster: metal-admin exception for VR1.

  • bundle.yaml: dashboard cluster: metal-admin -> metal-internal + rewrote the BUNDLEFIX-011 comment to explain the VR1 inversion.
  • docs/design-decisions.md: new D-072 AMENDMENT (VR1, 2026-08-06), RULED GA-R5, operator utterance "Ratified, land the config-of-record"; refined invariant (cluster = the plane apache SERVES, = default only when the charm serves its internal vhost there).
  • docs/network-space-binding-reference.md: matrix row (m-internal 10->11, m-admin-nondefault 1->0), section-5 exception RETIRED, section-6 cross-ref updated.
  • LIVE (proven BEFORE ratifying, per operator process directive): juju bind openstack-dashboard cluster=metal-internal (rc=0). No harness asserts this binding (checked) -> none owed. Why. VR1 split metal into admin+internal; the dashboard charm declares no admin/internal extra-binding + no os--network (metadata + charmhub docs confirmed), so apache serves its SSL vhost on metal-INTERNAL, while haproxy dialed cluster=metal-admin -> vhost-less -> plaintext (the D-072 trap, inverted). Option A (serve metal-admin) unavailable (no charm lever). cluster -> metal-internal is both the generic HA/cluster-peer rule (14 apps) and the served plane. Verify. haproxy :433 backends moved to metal-internal (.12.108/.145/.146); provider VIP 10.12.4.58 AND operator metal-admin VIP 10.12.8.58 both TLS 200 CA-verified (were 000/plaintext); reissued cert covers all 3 VIP IPs incl 10.12.8.58 (resolves AH01909); 3 units active/idle. Evidence: docs/audit/stage5-dc0-phase03-coreverify-20260806.txt (F-CV3 REMEDIATION section). dc1 inherits via the shared bundle -> correct-by-deploy, no rebind. *Revert. juju bind openstack-dashboard cluster=metal-admin (live) + git revert this commit.

Findings logged (NOT executed -- hard rule 1)

  • F-CV1 designate-api plaintext on :8991 vs haproxy check-ssl -> backend DOWN -- RESOLVED this session, see Item 6 (BUNDLEFIX-056). designate-api is UP; the "collateral of the Stage-7 block" reading was RETRACTED.
  • F-CV3 dashboard VIP TLS plaintext -- RESOLVED this session, see Item 7 (BUNDLEFIX-057 / D-072 VR1 amendment). Root cause was the VR1 split-metal inversion of D-072, not the initial "charm frontend not effective" framing; proven live then ratified.
  • Shared shape: 2 services plaintext-vs-TLS-expecting while the other 12 serve TLS.
  • DOCFIX candidate phase-03-admin-openrc.sh / phase-04-* / vault-kv-health.sh read DC-dependent lib-net values without lib_net_select_dc (harmless on dc0, WRONG+silent on dc1). Fix before dc1's Step 7.
  • OWED Step 3.4 domain-manager policy gate (PO: + G3); dc1-rack client install; pending rack kernel 6.8.0-136->137 (a maintenance-window reboot, NOT acted on).