diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index f9bf1c4..f4faa7a 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2389,6 +2389,21 @@ Consumer verify: nova-cloud-controller sees all 3 servers; designate's coordination `backend_url` still shows ONE (memcached/0) -- designate is workload-BLOCKED pre-Stage-7, so re-check that at designate activation (recorded, not a scale-up defect). + **>>> STEP 7 (phase-03 CORE VERIFY, adapted for vr1-dc0) -- CORE-API LAYER VERIFIED 2026-08-06; + exit gate NOT MET (2 open items). Capture `docs/audit/stage5-dc0-phase03-coreverify-20260806.txt`. <<<** + Run read-only FROM the dc0 rack (D-138). PASS: settle walk (8/162 non-active/idle, ALL in a + prediction written before the read = octavia D-021 + designate Stage-7 + ceph-rbd-mirror Stage-6 + + gss; octavia's is its OWN configure-resources block, confirming ovn RESOLVED end-to-end); + haproxy sweep 0 DOWN across the 12 ACTIVATED VIP apps; admin-openrc scoped-token OK; endpoint list + IP-only; keystone VIP two-sourced (overlay 10.12.4.50 == live public endpoint); vault root CA TLS + to keystone OK. **F-CV2 RESOLVED:** openstack CLI was ABSENT on the dc0 rack (07-30 queued-F1, HIT + at Step 7) -> installed `python3-openstackclient 6.6.0-0ubuntu2` on the rack (see section 7 row). + **EXIT GATE OPEN on (a) F-CV3: dashboard VIP 10.12.4.58:443 serves PLAINTEXT (apache-SSL-inactive + despite certs present) -- Horizon exit-gate FAILS; and (b) Step 3.4 domain-manager policy NOT RUN. + F-CV1: designate-api is UP but plaintext on :8991 vs haproxy check-ssl (same SHAPE as F-CV3); + the earlier "collateral of the block" reading is RETRACTED.** F-CV1/F-CV3 = a shared-shape + plaintext-vs-TLS-expecting defect on 2 services while the other 12 serve TLS correctly; LOGGED, + not fixed (hard rule 1), triage authorized by operator 2026-08-06. **NAMED-GATE DEFECT found by measurement -- `phase-03-core-verify.md` Step 3.1 asserts expected non-active/idle = 1 (octavia only); the VR1 roster yields 4 deferred-by-design + gss.** That gate is STALE for VR1 and a DOCFIX is owed (also owed on that runbook: `-m openstack` -> `-m vr1-dc0` @@ -7540,7 +7555,7 @@ | OPNsense edge | 26.7.1 (FreeBSD base 15.1) | MEASURED 2026-07-23 via the gated API (`GET core/firmware/status` -> product_version 26.7.1, capture `docs/audit/g13-close-20260723.txt`); updated 26.7 -> 26.7.1 in the G13 bundle. DC edges (vr1-dc0/dc1) remain 26.7 | office1-opnsense | | NetBox (Office1 apex) | 4.6.4 per as-built `docs/vr1-office1-as-built.md:44`; service UP verified (HTTP 302) this session | `ssh office1-netbox 'curl ... localhost:8000'` | office1-netbox | | Juju | **3.6.27 (rev 35621, `3/stable`) MEASURED 2026-07-27 on voffice1** -- the headend is the D-128 Plane-2 execution host and this is the client that will bootstrap the controller. Supersedes the 3.6.25 figure recorded 2026-07-24 (also at line 162, kept there as history): an in-channel patch refresh, which D-071 ADOPTED 2026-07-21 explicitly permits (patch-only jumps, in-channel-only refreshes), so this is policy-compliant drift and NOT an incident. The jumphost has NO juju client (measured ABSENT). | `ssh voffice1 'snap list'` (capture `docs/audit/stage5-live-measurement-20260727.txt`) | voffice1 | -| OpenStack client | **6.6.0 (`python3-openstackclient 6.6.0-0ubuntu2`, noble/main) INSTALLED ON voffice1 2026-07-27** -- Stage-5 Phase 0 precondition 0.2, operator-approved. voffice1 is the D-128 Plane-2 host every Stage-5+ script runs from. Verified behaviourally, not by presence: `openstack --version` -> `openstack 6.6.0`, `--help` exit 0, and `server list` fails CLEANLY on absent auth config rather than crashing. Companion pins: `python3-openstacksdk 3.0.0-0ubuntu2`, `python3-novaclient 2:18.5.0-0ubuntu1`. **The snap was REFUTED by measurement, not preference:** `openstackclients` has NO Caracal channel (newest stable `zed`, 2023-03; `latest/stable` is `xena`, 2021), and `docs/design-decisions.md:638` already records its home-only confinement trap. 6.6.0 is the Caracal 2024.1 client, verified upstream rather than from memory. noble's native OpenStack release IS Caracal 2024.1, so no UCA is needed on this host. **STILL ABSENT ON vcloud** -- deliberately: D-128 puts this work on the headend. Capture `docs/audit/stage5-phase0-20260727.txt`. **Supersedes the "ABSENT ON BOTH HOSTS" figure measured earlier the same day** (kept here as history, per the Juju row's precedent for in-row supersession): the ten Stage-5/6/7 scripts that invoke it (`phase-03-admin-openrc.sh`, `phase-04-network-{create,verify}.sh`, `phase-04-internal-cert-san-verify.sh`, `phase-05-{amphora-pipeline,octavia-verify}.sh`, `phase-06-{bootstrap,capi-stack,mgmt-vm,net-setup}.sh`) now have a client on the host D-128 runs them from. | `ssh voffice1 'openstack --version; dpkg-query -W ...'` | voffice1 | +| OpenStack client | **6.6.0 (`python3-openstackclient 6.6.0-0ubuntu2`, noble/main) INSTALLED ON voffice1 2026-07-27** -- Stage-5 Phase 0 precondition 0.2, operator-approved. voffice1 is the D-128 Plane-2 host every Stage-5+ script runs from. Verified behaviourally, not by presence: `openstack --version` -> `openstack 6.6.0`, `--help` exit 0, and `server list` fails CLEANLY on absent auth config rather than crashing. Companion pins: `python3-openstacksdk 3.0.0-0ubuntu2`, `python3-novaclient 2:18.5.0-0ubuntu1`. **The snap was REFUTED by measurement, not preference:** `openstackclients` has NO Caracal channel (newest stable `zed`, 2023-03; `latest/stable` is `xena`, 2021), and `docs/design-decisions.md:638` already records its home-only confinement trap. 6.6.0 is the Caracal 2024.1 client, verified upstream rather than from memory. noble's native OpenStack release IS Caracal 2024.1, so no UCA is needed on this host. **STILL ABSENT ON vcloud** -- deliberately: D-128 puts this work on the headend. Capture `docs/audit/stage5-phase0-20260727.txt`. **Supersedes the "ABSENT ON BOTH HOSTS" figure measured earlier the same day** (kept here as history, per the Juju row's precedent for in-row supersession): the ten Stage-5/6/7 scripts that invoke it (`phase-03-admin-openrc.sh`, `phase-04-network-{create,verify}.sh`, `phase-04-internal-cert-san-verify.sh`, `phase-05-{amphora-pipeline,octavia-verify}.sh`, `phase-06-{bootstrap,capi-stack,mgmt-vm,net-setup}.sh`) now have a client on the host D-128 runs them from. **AMENDED 2026-08-06 (D-138 supersedes the "on voffice1 is sufficient" reading): these ten scripts DIAL THE CLOUD AT L3 and per D-138 run FROM THE DC RACK, not voffice1 (which has no L3 path to the cloud). At dc0's Step 7 (phase-03 core verify) `phase-03-admin-openrc.sh` failed "openstack not found" on the rack. `python3-openstackclient 6.6.0-0ubuntu2` (+ `python3-openstacksdk 3.0.0-0ubuntu2`) is now INSTALLED ON THE dc0 RACK (172.31.0.2), verified `openstack --version` -> `openstack 6.6.0`; snap still refuted, noble-native Caracal so no UCA. Capture `docs/audit/stage5-dc0-phase03-coreverify-20260806.txt`. The voffice1 copy remains (it serves the MAAS/NetBox/inner-tofu Plane-2 tools that stay on the headend). OWED: the SAME install on the dc1 rack before dc1's Step 7 (== the per-DC 07-30 queued-finding F1).** | `ssh vr1-dc0-rack 'openstack --version'`; `ssh voffice1 'openstack --version'` | dc0 rack + voffice1 | The known-stale pin sites this table used to enumerate (the GA-F03/F04/ F05 tofu, OPNsense, and jumphost-name values -- stated token-free here so diff --git a/docs/audit/stage5-dc0-phase03-coreverify-20260806.txt b/docs/audit/stage5-dc0-phase03-coreverify-20260806.txt new file mode 100644 index 0000000..bca6428 --- /dev/null +++ b/docs/audit/stage5-dc0-phase03-coreverify-20260806.txt @@ -0,0 +1,199 @@ +# Stage 5 dc0 -- Step 7: phase-03 core verify (ADAPTED for vr1-dc0), read-only pass +# Runbook: runbooks/phase-03-core-verify.md, adapted per dc-dc-phase4 Steps 7-9. +# Model: vr1-dc0. Run-location: dc0 rack (172.31.0.2 via ProxyJump voffice1) -- D-138. +# Started 2026-08-06. Author: Claude Code (background session), operator "continue". +# +# DISCIPLINE: hard rule 1 -- this is VERIFICATION. Findings are LOGGED here, not +# fixed mid-step (the sole sanctioned in-step remediation is the 3.1 haproxy +# `reload`, individually gated). Prediction is written BEFORE reading live status +# so the settle gate is falsifiable (advisor 2026-08-06; memory #17 / GA-R6 +# "a checker that cannot fail is not a gate"). +# +# ===================================================================== +# PREDICTION (written 2026-08-06 BEFORE any `juju status` this session) +# ===================================================================== +# Source of the prediction: +# - 08-06 part-2 close: "14/14 HA ... 12 active/idle + 2 known-blocked +# (octavia/designate)". +# - 08-05 close: designate/octavia/rbd-mirror correctly-waiting on later stages. +# - D-021: octavia BLOCKED "Awaiting configure-resources" until phase-05 (Stage 6). +# - Stage-7 (dc-dc-phase6): designate activation. +# - Stage-6 (dc-dc-phase5): ceph-rbd-mirror / radosgw multisite DR wiring. +# +# EXPECTED non-active/idle at this pass (the prediction to diff against): +# E1. octavia/* -- BLOCKED, D-021 (awaiting configure-resources; Stage 6) +# E2. designate/* -- waiting/blocked, Stage-7 activation +# E3. ceph-rbd-mirror/* -- waiting/blocked, Stage-6 DR wiring +# E4. (tolerated-transient) glance-simplestreams-sync -- pre-run, if present +# +# DIFF RULE (the gate): +# - Any live non-active/idle unit NOT in {E1,E2,E3,E4} => FINDING (log below). +# - Any unit in {E1,E2,E3} that came back ACTIVE/IDLE => FINDING (record stale). +# - octavia downstream-of-ovn note (08-04): ovn-central is now RESOLVED +# (08-05), so octavia's block should be its OWN D-021 block, not an ovn +# cascade. If octavia is waiting on ovn/cert, that is a FINDING. +# +# ===================================================================== +# MEASUREMENT (filled in below, this session) +# ===================================================================== + +## Step 3.1 acceptance walk -- MEASURED 2026-08-06 (read-only) +[vr1-dc0-rack] juju status -m vr1-dc0 --format=yaml (rc=0, 223979 bytes, 57 apps, 162 units) +Processed on jumphost. Non-active/idle units: 8 of 162 (154 active/idle): + ceph-rbd-mirror/0 blocked idle 'ceph-local' incomplete, 'ceph-remote' missing [E3 Stage-6 DR] + designate/0,1,2 blocked idle nameservers must be set [E2 Stage-7] + glance-simplestreams-sync/0 unknown idle (empty) [E4 transient] + octavia/0 blocked idle Awaiting end-user execution of `configure-resources` [E1 D-021] + octavia/1,2 blocked idle Awaiting leader to create required resources [E1 D-021] + +DIFF vs PREDICTION: MATCH. All 8 in {E1,E2,E3,E4}; none outside; none expected-blocked +came back active. GATE PASS (falsifiable -- prediction pre-dated the read). +NOTE: octavia/0 message is its OWN D-021 configure-resources block, NOT an ovn/cert +cascade -- confirms ovn-central RESOLVED (08-05) end-to-end; the 08-04 "octavia +downstream-of-ovn" concern is CLEARED. + +## Step 3.1 haproxy backend sweep (DOCFIX-031/D-045) -- MEASURED 2026-08-06 (read-only) +Scoped to the 13 VIP-fronted apps (those carrying an *-hacluster subordinate), +39 units. Loop run ON the rack (juju ssh controller-local). +RESULT: 3 DOWN lines, ALL on designate (the known Stage-7 block, E2): + [designate/0] designate-api_admin_10.12.12.110:8991 DOWN L6RSP "SSL handshake failure / Layer6 invalid response" + [designate/1] designate-api_admin_10.12.12.144:8991 DOWN L6RSP (same) + [designate/2] designate-api_admin_10.12.12.143:8991 DOWN L6RSP (same) +CLEAN for all 12 ACTIVATED VIP apps: barbican ceph-radosgw cinder glance keystone + magnum neutron-api nova-cloud-controller octavia openstack-dashboard placement vault + -> zero DOWN. + +FINDING F-CV1 (LOGGED, not fixed -- hard rule 1): designate-api _admin backend DOWN + with an SSL-handshake/L6 signature on all 3 units. designate is BLOCKED pre-Stage-7 + ("nameservers must be set"), so this is plausibly collateral of the un-activated API, + NOT necessarily the D-045 plaintext-vs-SSL defect. The sanctioned in-step haproxy + `reload` is WITHHELD: designate is not a healthy app, and reload would not clear a + Stage-7 activation gap. ACTION: re-run this sweep against designate AFTER Stage-7 + activation (dc-dc-phase6); if the L6/SSL DOWN persists on an ACTIVE designate, THAT + is a D-045-class defect to remediate then. Note the signature here is L6RSP (SSL + handshake), which differs from D-045's classic L7STS/400 plaintext-vs-SSL. + +GATE VERDICT: PASS for the 12 activated VIP apps (the phase-03 core-verify scope at + this pre-Stage-6/7 point). designate deferred to its Stage-7 activation, consistent + with the settle-walk E2 classification. + +## Step 3.2 admin-openrc -- MEASURED 2026-08-06 +Config source (dc0 vips overlay line 36): keystone provider VIP = 10.12.4.50 + (v4-only D-141 triple ".50 / .8.50 / .12.50"). +Rack state: admin-openrc + vault root CA both ABSENT (fresh build, nothing to clobber; + API reachability never yet verified via openrc on this deploy). phase-03-admin-openrc.sh + + extract_admin_password.py staged to rack, sha256 MATCH (2aeb20b720d83944 / + 7b9b25a8fafd75cb); lib-net.sh dep already staged, MATCH 3e465ae7de3bf92d. + +BLOCKER F-CV2 (== the KNOWN 07-30 queued-finding F1, HIT at Step 7) -- RESOLVED this session: + First run -> "FAIL: openstack not found". The openstack CLI was ABSENT on the dc0 rack + (no snap/apt/venv). D-138 puts phase-03..06's openstack CLI ON THE RACK, but the 07-27 + install landed it only on voffice1 (which cannot reach keystone's provider VIP L3). + Phase-01/02 were juju-only, so it first surfaced here. Documented remediation (07-30 F1): + "Install on the DC client host before Step 7." EXECUTED (gated): apt-get install -y + python3-openstackclient on the rack -> 6.6.0-0ubuntu2 (+ python3-openstacksdk + 3.0.0-0ubuntu2), matching the measured 07-27 pin exactly; snap correctly refuted, no UCA. + REVERT: sudo apt-get purge -y python3-openstackclient on the rack. + OWED DOC: add the dc0 rack to CURRENT-STATE section 7's OpenStack-client row (per F1's + own instruction). Same install is OWED on the dc1 rack before dc1's Step 7. + NOTE (pending kernel): rack reports 6.8.0-136 running vs 6.8.0-137 available -- NOT acted + on (a reboot would bounce the rack + libvirt + all nodes); logged for a maintenance window. + +BUILD (after unblock): MODEL=vr1-dc0 KEYSTONE_VIP=10.12.4.50 phase-03-admin-openrc.sh -> + vault root CA subject "Vault Root Certificate Authority (charm-pki-local)", notAfter + Aug 2 2036 (matches 08-05 init); admin project = admin (password len 16); wrote openrc + 0600; **[OK] scoped token issued** against https://10.12.4.50:5000/v3. + +TWO-SOURCE VIP (advisor): CONFIRMED. Config (overlay) keystone provider VIP = 10.12.4.50; + live artifact `openstack endpoint list --service keystone`: + public https://10.12.4.50:5000/v3 (provider) + admin https://10.12.8.50:35357/v3 (metal-admin, :35357) + internal https://10.12.12.50:5000/v3 (metal-internal) + Config == artifact. IP-only endpoint list PASS across ALL services (public .4.x / + admin .8.x / internal .12.x -- the VR1 dual-metal-plane split, D-141; not VR0's single + metal plane, and NOT a defect). s3/swift on radosgw VIP .60:443; image-stream HTTP on + metal .8.192 (gss, expected HTTP). GATE PASS. + +## Step 3.3 dashboard-VIP TLS probe (read-only; advisor: NOT deferred) -- MEASURED 2026-08-06 +FINDING F-CV3 (LOGGED, not fixed -- hard rule 1): the dc0 dashboard VIP does NOT serve + TLS on 443. Instrument checked first (advisor rule): OS_CACERT valid (the openstack TLS + calls above succeeded on it), 443 TCP OPEN on BOTH 10.12.4.58 and 10.12.8.58, so not a + connectivity/CA artifact. curl (even -k) -> http_code 000, errormsg "OpenSSL...wrong + version number" == the server on 443 answers PLAINTEXT, not TLS. Plain http :80 root + -> 200. This is the known D-072 / DOCFIX-089 "Horizon VIP https handshake death" pattern + (the exact defect the fail-closed rewrite was built to catch after it passed the old + fail-open gate for weeks). SCOPE: Horizon operator/tenant ACCESS, NOT a core-API blocker + -- every core API endpoint proved working TLS + a scoped token. NEEDS TRIAGE (likely a + dashboard-ssl/haproxy-TLS-not-terminating item; appendix-A "Horizon VIP https handshake + death" + D-072) before Horizon is usable. Deferred with it: the Step 3.3 nginx external + proxy repoint (genuinely VR0-external-proxy topology / different VR1 edge workstream). +NOTE: D-044 Secure-cookie override and D-075 root redirect (both PER-REBUILD apache + mutations) are NOT applied this rebuild -- root http :80 -> 200 (not the D-075 302 to + /horizon). Expected (per-rebuild steps not yet run), recorded so it is not mistaken for + a defect; they belong with the Horizon-access remediation once F-CV3 clears. + +## TLS-LAYER DISCRIMINATION (read-only, 2026-08-06) -- routes F-CV1 + F-CV3 +Advisor-directed: characterize the LAYER before routing (a citation is an existence +claim; do not pattern-match the root cause -- the ovn saga was called wrong 3x). +DASHBOARD (openstack-dashboard/leader): + - /etc/apache2/ssl/horizon/ EXISTS, dir mtime Aug 5 02:06 (== cert-cascade time). So + certs were delivered -- NOT a bare cascade gap. + - haproxy `bind *:443` with NO `ssl crt` clause; listeners 443 (haproxy), 433 + 70 (apache). + - curl -k to 127.0.0.1 :70, :433 AND :443 ALL return "wrong version number" == every + port answers PLAINTEXT. Apache's own :433 (the intended SSL backend) is plaintext too. + - ROUTING: NOT the classic D-045 (haproxy-not-reloaded) -- the apache backend itself is + plaintext. NOT the ovn cascade gap -- certs are present. It is an apache-SSL-NOT-ACTIVE + condition (ssl vhost/module or charm ssl wiring). Needs its own triage; do NOT file it + under D-072 by pattern-match (my earlier F-CV3 note did -- corrected here). +DESIGNATE (designate/leader): + - designate-api process ACTIVE/up (systemctl active, pid running). So the earlier + sweep DOWN is NOT "api down". + - designate-api LISTENS PLAINTEXT on :8991; haproxy backend uses `server ... :8991 check + check-ssl verify none` -- haproxy attempts an SSL handshake to a plaintext backend -> + L6RSP "SSL handshake failure" -> backend DOWN. This is a TLS-PRESENTATION MISMATCH, + same SHAPE as the dashboard. +CORRECTION to F-CV1: my "plausibly collateral of the Stage-7 nameservers block" guess is + WRONG -- designate-api IS up; the DOWN is a plaintext-vs-check-ssl mismatch, not a + not-yet-activated API. Whether the mismatch clears at Stage-7 activation is now an OPEN + question, not an assumption. Re-check at Stage-7 STILL applies, but the classification + "collateral" is RETRACTED. +COMMON SHAPE (F-CV1 + F-CV3): two services (designate-api, horizon-apache) serve plaintext + while their TLS layer expects SSL, on a cloud where the OTHER 12 VIP apps serve TLS + correctly. NOT asserted to be one identical root cause (dashboard = apache-ssl-inactive; + designate = api-plaintext-vs-haproxy-check-ssl), but the shape is shared and worth a + single triage thread. LOGGED, not fixed (hard rule 1). + +## Step 3.4 keystone domain-manager policy -- NOT RUN this session + Deferred with the openstack-client-dependent verification depth; the PO: stage-1 check + + the C.4 G3 behavioral probe (which mutates -- creates user/project) are owed. Recorded as + NOT RUN (not as passed), per the runbook's own "PO: proves parse, not that policy works". + +## ============ STEP 7 / phase-03 EXIT GATE: NOT MET ============ +GA-R6/E3: no conditional close -- the remainder keeps the step OPEN (or splits to its +own gate row). Verified vs the runbook EXIT GATE: + [PASS] Cloud settled: only expected Stage-6/7 blocks (falsifiable, matched prediction). + [PASS] haproxy backends UP -- for the 12 ACTIVATED VIP apps (designate = F-CV1). + [PASS] admin-openrc scoped token; endpoint list IP-only; two-source keystone VIP. + [PASS] vault root CA validates TLS to the keystone VIP (.4.50). + [FAIL] Horizon reachable AND login works -- F-CV3, dashboard VIP serves plaintext on 443. + [NOT RUN] Step 3.4 domain-manager policy (PO: stage-1 + G3 behavioral). +CORE-API layer: VERIFIED. Step 7 as a whole: OPEN (2 named items: F-CV3, Step 3.4). + +## OWED (survive-a-clear): + O1. CURRENT-STATE section 7 OpenStack-client row: now STALE. python3-openstackclient + 6.6.0-0ubuntu2 is INSTALLED ON THE dc0 RACK (this session) -- the row says client is + on voffice1 only. GA-R1/C1: update in the same commit that records this. (Not churn -- + a real measured status change.) + O2. dc1 rack needs the SAME python3-openstackclient install before dc1's Step 7 (F-CV2 + is per-DC; only dc0 done). + O3. F-CV3 (dashboard apache-SSL-inactive) + F-CV1 (designate api-plaintext-vs-check-ssl): + operator decision -- triage now, or defer with the Horizon-access workstream. + O4. Step 3.4 domain-manager policy gate (PO: + G3) still owed for phase-03 close. + O5. lib_net_select_dc DOCFIX candidate (below). + +## DOCFIX candidate (advisor -- log now while looking): phase-03-admin-openrc.sh, + phase-04-network-{create,verify}.sh, phase-04-internal-cert-san-verify.sh and + vault-kv-health.sh read DC-dependent lib-net values WITHOUT calling lib_net_select_dc + (measured 07-29, dc-dc-phase4 Steps 7-9 caveat). Harmless on dc0 (resolves dc0's + literals); on dc1 it is WRONG and SILENT. Fix before dc1's Step 7, or export every value + explicitly. This session used explicit KEYSTONE_VIP=10.12.4.50, so unaffected. diff --git a/docs/changelog-20260806-phase03-coreverify.md b/docs/changelog-20260806-phase03-coreverify.md new file mode 100644 index 0000000..d80b3aa --- /dev/null +++ b/docs/changelog-20260806-phase03-coreverify.md @@ -0,0 +1,71 @@ +# Changelog 2026-08-06 -- Stage 5 dc0: Step 7 phase-03 core verify (core-API layer) + +Session-scoped (GA-R2). Branch `dc-dc-stage5-preconditions`. Stage 5 remains OPEN +(this is NOT a stage close). Under blanket approval the changelog is the review +surface: each item = what / why (evidence) / revert. + +Evidence capture (all read-only measurement + the one gated install): +`docs/audit/stage5-dc0-phase03-coreverify-20260806.txt`. + +--- + +## Item 1 -- INFRA: openstack CLI installed on the dc0 rack (F-CV2 resolve) +**What.** `sudo apt-get install -y python3-openstackclient` on the dc0 rack +(`vr1-dc0-rack`, 172.31.0.2). Landed `python3-openstackclient 6.6.0-0ubuntu2` ++ `python3-openstacksdk 3.0.0-0ubuntu2` from noble/main. Verified `openstack +--version` -> `openstack 6.6.0`. +**Why.** Step 7 (phase-03) is the first phase to invoke the `openstack` CLI, and +per D-138 that CLI runs FROM the DC rack (no L3 path from voffice1 to the cloud). +The 07-27 install landed only on voffice1. `phase-03-admin-openrc.sh` failed +"openstack not found" on the rack. This is the documented remediation of the +07-30 queued-finding F1 ("Install on the DC client host before Step 7"); pin is +the measured 07-27 value, snap refuted, noble-native Caracal so no UCA. +**Revert.** `ssh vr1-dc0-rack 'sudo apt-get purge -y python3-openstackclient +python3-openstacksdk'` (client-only; no service impact). + +## Item 2 -- STAGE: rack repo-stage gained phase-03-admin-openrc.sh + extract helper +**What.** `scp` staged `scripts/phase-03-admin-openrc.sh` (sha 2aeb20b720d83944) +and `scripts/extract_admin_password.py` (sha 7b9b25a8fafd75cb) into +`~/repo-stage/scripts/` on the dc0 rack; both sha256-verified == repo HEAD. +**Why.** D-138 rack-run discipline: Step 7 runs the tested phase-03 admin-openrc +builder from the rack's staged copy, sha-verified before trust. +**Revert.** `ssh vr1-dc0-rack 'rm ~/repo-stage/scripts/phase-03-admin-openrc.sh +~/repo-stage/scripts/extract_admin_password.py'` (redeploy inputs, not live state). + +## Item 3 -- RACK STATE: admin-openrc + vault root CA built on the dc0 rack +**What.** `MODEL=vr1-dc0 KEYSTONE_VIP=10.12.4.50 phase-03-admin-openrc.sh` wrote +`~/admin-openrc` (0600) + `~/vault-init/vault-ca-root.pem` on the rack; scoped +token issued. Secret-adjacent files, on-rack only; password never entered context +(script prints length only). +**Why.** phase-03 Step 3.2 -- the IP-only admin credential + vault CA for API +verification. Two-source keystone VIP confirmed (overlay == live endpoint). +**Revert.** `ssh vr1-dc0-rack 'rm ~/admin-openrc'` (regenerable from live state). + +## Item 4 -- DOC: CURRENT-STATE section 7 OpenStack-client row amended (GA-R1/C1) +**What.** In-row amendment: client now INSTALLED ON THE dc0 RACK; the D-138 +correction that phase-03..06 run from the rack, not voffice1; dc1-rack install +OWED. Verify command + host cell updated. +**Why.** GA-R1/C1 -- a commit that changes a status CURRENT-STATE carries updates +it in the same commit. Real measured status change (client presence on the rack). +**Revert.** `git revert` this commit's CURRENT-STATE hunk. + +## Item 5 -- DOC: CURRENT-STATE section 1 Stage-5 progress note (Step 7) +**What.** Added the Step-7 phase-03 core-verify progress block: core-API VERIFIED, +exit gate OPEN on F-CV3 (dashboard TLS) + Step 3.4; F-CV1 retraction; F-CV2 resolve. +**Why.** Stage/gate status lives in CURRENT-STATE only (GA-R1). +**Revert.** `git revert` this commit's CURRENT-STATE hunk. + +--- + +## Findings logged (NOT executed -- hard rule 1) +- **F-CV1** designate-api plaintext on :8991 vs haproxy `check-ssl` -> backend DOWN. + designate-api is UP; the "collateral of the Stage-7 block" reading is RETRACTED. +- **F-CV3** dashboard VIP 10.12.4.58:443 serves plaintext (apache-SSL-inactive + despite certs under /etc/apache2/ssl/horizon/). Horizon exit-gate FAILS. NOT + D-072 by pattern-match; own triage owed. Operator authorized triage 2026-08-06. +- Shared shape: 2 services plaintext-vs-TLS-expecting while the other 12 serve TLS. +- **DOCFIX candidate** phase-03-admin-openrc.sh / phase-04-* / vault-kv-health.sh + read DC-dependent lib-net values without `lib_net_select_dc` (harmless on dc0, + WRONG+silent on dc1). Fix before dc1's Step 7. +- **OWED** Step 3.4 domain-manager policy gate (PO: + G3); dc1-rack client install; + pending rack kernel 6.8.0-136->137 (a maintenance-window reboot, NOT acted on).