# Queued findings -- Stage 5 dc0 Step 7 (phase-03 core verify), 2026-08-06
# Survives-a-clear sweep. Full evidence: docs/audit/stage5-dc0-phase03-coreverify-20260806.txt
# Status authority is CURRENT-STATE.md (section 1 Stage-5 block + section 7 client row).

STEP 7 EXIT GATE: NOT MET (GA-R6/E3, no conditional close). Core-API layer VERIFIED;
two named items keep it OPEN: F-CV3 (Horizon TLS) + Step 3.4 (not run).

--- FINDINGS (logged, NOT fixed -- hard rule 1; operator authorized triage 2026-08-06) ---

F-CV1  [RESOLVED 2026-08-06 -- BUNDLEFIX-056, operator-approved fix executed + verified:
       bundle +public:provider-public +internal:metal-internal + live `juju bind designate
       public=provider-public internal=metal-internal`. Full haproxy sweep 0 DOWN cloud-wide;
       apache vhosts span all 3 planes; cert reissued for provider-public; catalog triple
       correct (public .4.62 / internal .12.62 / admin .8.62). Checker EXPECT 11->12 (+T16c/d);
       binding-matrix row updated; gauntlet ALL GREEN 99. Details below retained as the record.]
       designate _admin haproxy backend DOWN. designate/0 apache https frontend binds ONLY
  10.12.8.198:8991 (metal-admin); haproxy's `designate-api_admin_10.12.12.110` backend dials
  10.12.12.110:8991 (METAL-INTERNAL) where apache has no SSL vhost -> check-ssl -> DOWN.
  ROOT CAUSE (researched 2026-08-06, governing = D-052 + the generic binding rule; NOT D-141):
  designate's endpoint bindings deviate from every sibling API app.
   - `:internal` -> metal-internal = CONFIRMED DEFECT (sits on the `''` metal-admin FALLBACK;
     every sibling binds internal->metal-internal; no ruled exception). designate has a
     metal-internal VIP 10.12.12.62 and its cert ALREADY carries the metal-internal SANs, but
     no apache vhost there. FIX (gated, bundle+live per D-072): add `internal: metal-internal`
     to bundle.yaml + `juju bind designate internal=metal-internal`; verify haproxy readback +
     https-200 + re-read cert SANs. All 3 units have metal-internal addrs (no --force needed).
   - `:public` -> provider-public = OPEN, needs operator RULING. designate uniquely has
     prov-pub=1 = `:dnsaas` (D-106 dual-VIP), not `:public`; cert doesn't cover provider-public.
     May be intentional (tenants consume DNS, not the designate REST API). DO NOT assume.
  Section-4 table (network-space-binding-reference.md:88) is "Generated from bundle.yaml"
  (descriptive of the defect), not intent. designate is Stage-7-blocked -> no urgency.

F-CV3  [RESOLVED 2026-08-06 -- D-072 AMENDMENT (VR1) ratified GA-R5 + BUNDLEFIX-057. Root cause:
       VR1 split-metal inverts D-072 -- dashboard charm declares no admin/internal binding + no
       os-*-network (metadata + charmhub docs), so apache serves metal-INTERNAL while haproxy dialed
       cluster=metal-admin -> plaintext. Option A (serve metal-admin) unavailable (no charm lever).
       Fix (proven LIVE before ratifying): cluster -> metal-internal. Verified: provider + metal-admin
       VIPs TLS 200 CA-verified; cert covers all 3 VIPs (resolves AH01909); units active/idle. Config-
       of-record landed; dc1 inherits via shared bundle (no rebind). Detail below retained as record.]
       dashboard VIP 10.12.4.58:443 serves PLAINTEXT (Horizon exit-gate
  FAILS). Certs present under /etc/apache2/ssl/horizon/; apache :433 served by Ubuntu
  default-ssl.conf, NOT the charm's openstack_https_frontend.conf (glance, working, uses the
  charm frontend w/ SSLEngine on). Charm https frontend not effective for dashboard. SEPARATE
  finding from F-CV1 -- do NOT chase a common fix. Needs focused triage + gated remediation.
  ACCESS-MODEL CONTEXT (2026-08-06): operators reach the dashboard via the metal-admin VIP over
  the tailnet (dc-dc-deployment-workflow.md gap-register item 21 access-model note) -> argues for
  the fix serving TLS on metal-admin (option A), NOT metal-internal (option B, which also reverses
  ruled D-072). A-vs-B + whether apache can be steered onto metal-admin is the open (a) work.

  NOTE both: certs ARE present -> NOT the ovn CN-issuance class; NOT a missing certificates
  relation. Charm apache-TLS-frontend layer. Also owed with the Horizon-access remediation:
  D-044 cookie override + D-075 root redirect (per-rebuild, not applied this rebuild).

--- OWED (not findings, just must-not-evaporate) ---

O2  dc1 RACK needs `apt-get install -y python3-openstackclient` (6.6.0-0ubuntu2) BEFORE dc1's
    Step 7 -- F-CV2 is per-DC; only the dc0 rack was done this session (== 07-30 queued-F1).
O4  Step 3.4 keystone domain-manager policy gate (PO: stage-1 + C.4 G3 behavioral, G3 mutates)
    still owed for phase-03 close.
O5  DOCFIX candidate: phase-03-admin-openrc.sh / phase-04-network-{create,verify}.sh /
    phase-04-internal-cert-san-verify.sh / vault-kv-health.sh read DC-dependent lib-net values
    WITHOUT lib_net_select_dc (harmless on dc0, WRONG+SILENT on dc1). Fix before dc1's Step 7.
O6  NAMED-GATE DEFECT (already in CURRENT-STATE): phase-03-core-verify.md Step 3.1 asserts
    non-active/idle == 1; VR1 roster yields the 4 deferred-by-design + gss. Runbook DOCFIX owed
    (-m openstack -> -m vr1-dc0; run-from-rack per D-138; the settle count).
O7  rack kernel 6.8.0-136 running vs 6.8.0-137 available -- reboot NOT taken (would bounce the
    rack + libvirt + all nodes); a maintenance-window item, logged.
O9  >>> STAGE-6 WATCH ITEM (record so it is not lost) <<< ceph-rbd-mirror carries
    `certificates` / `cluster` / `public` on the metal-admin default -- UNBOUND today (Stage-6
    DR wiring not yet connected), so no active deviation NOW. When rbd-mirror is wired at
    Stage-6 (dc-dc-phase5-dr-failover-drill), `certificates: metal-admin` could reproduce a
    TLS-cert-plane issue (the F-CV1/F-CV3 class). PRE-CHECK at Stage-6 before wiring: bind
    certificates->metal-internal + cluster->replication (ceph) as needed, per the generic rule.
    Surfaced by the 2026-08-06 binding-conformance sweep.

O8  OBSERVATION (NOT verified as a defect -- do the D-NNN check first): designate's `amqp`
    endpoint is bound to metal-admin, while the generic rule (network-space-binding-reference
    line 57) + glance put amqp on metal-internal. Surfaced during the BUNDLEFIX-056 bind output.
    NOT causing a measured failure (absent from every sweep); NOT touched (out of F-CV1 scope,
    hard rule 1). A future binding-conformance pass should check whether this is ruled/intentional
    (like it should have for public/internal) before treating it as a defect. Candidate only.
    >>> RESOLVED 2026-08-06 (BUNDLEFIX-058): the binding-conformance sweep confirmed designate
    amqp+cluster (and designate-bind cluster) were the only ACTIVE deviations; repaired to
    metal-internal, proven live, no regression. Not a ruled exception.

O10 OWED (re-stage for redeploy): the dc0 rack `~/repo-stage/bundle.yaml` is STALE vs repo
    (staged=42845edb repo=8dc19c9f) -- this session changed bundle.yaml 3x (BUNDLEFIX-056/057/058).
    Live dc0 is fixed via `juju bind` (bundle staleness does not affect the running cloud), but a
    REDEPLOY would use the stale staged bundle. Re-stage (scp + sha256-verify) before any dc0
    redeploy. Overlays unaffected this session.

O11 ENV FACT (FIRST SURFACE): querying the dc0 MAAS region. No maas CLI profile is logged in on
    the rack OR the region VM; the region admin API key lives ONLY in the vcloud consolidated creds
    `~/vr1-dc0-creds/maas-region-api-key.txt` (operator convention). Transient-use pattern that
    never persists the key: `ssh vr1-dc0-rack 'maas login <p> http://10.12.8.6:5240/MAAS/ - ;
    <read-only queries> ; maas logout <p>' < ~/vr1-dc0-creds/maas-region-api-key.txt` (key piped to
    stdin, logout removes the profile -- verified no ~/.maascli.db residue after). vcloud CANNOT
    reach 10.12.8.6:5240 (D-138); the rack can.

--- FIRST SURFACE in this file: F-CV1 (confirmed), F-CV3, O5, O7, O11 (MAAS-query method). ---
