diff --git a/docs/audit/ovn-central-cert-reeval-20260803.md b/docs/audit/ovn-central-cert-reeval-20260803.md new file mode 100644 index 0000000..9d944fa --- /dev/null +++ b/docs/audit/ovn-central-cert-reeval-20260803.md @@ -0,0 +1,138 @@ +# ovn-central "awaiting server certificate data" -- ROOT-CAUSE RE-EVALUATION, 2026-08-03 + +Re-evaluation of the prior 2026-08-03 diagnosis (the one the operator flagged as unreliable, +made under context-length pressure). Method: fresh live measurement (read-only, from the dc0 rack +per D-138) + three parallel deployed-source agents + one adversarial-review agent. Every claim +below is tagged MEASURED (read live/source this session) or INFERRED/UNKNOWN. Model `vr1-dc0`; +ovn-central rev 311 (24.03/stable); vault rev 724 (1.8/stable). + +## VERDICT: the prior root cause was WRONG. Corrected root cause below, confirmed end-to-end. + +### Corrected root cause (MEASURED at every hop) +ovn-central publishes a certificate request with an EMPTY common_name, so vault never creates a +server-cert request for it, so ovn-central blocks -- and the empty CN is because ovn-central's +address has no reverse-DNS PTR: + +1. **vault issues a per-unit SERVER cert ONLY** when the requester databag carries a truthy + top-level `common_name` OR a `cert_requests` map (vault charm `hooks/relations/tls-certificates/ + provides.py:179,191`). Otherwise it publishes ONLY the global client cert (ca+client.cert+ + client.key), unconditionally (`reactive/vault_handlers.py:1094-1121`). BY DESIGN. +2. **ovn-central uses the default charms.openstack cert path** (it does NOT override + `get_certificate_requests`; `reactive/layer_openstack.py:127-137` -> `charms_openstack/charm/ + classes.py:284-291` -> `charmhelpers/contrib/openstack/cert_utils.py`). That path sets + `cn = get_hostname(local_address(unit_get_fallback='private-address'))` (`cert_utils.py` + `add_hostname_cn`). +3. **`private-address` = eth1 `10.12.12.122` (metal-internal).** MEASURED `unit-get private-address`. + `get_hostname("10.12.12.122")` returns **None** -- `gethostbyaddr` raises herror; the + metal-internal plane 10.12.12.0/22 has NO PTR records. Contrast MEASURED: eth0 `10.12.8.185` + (metal-admin) reverse-resolves to `juju-237a74-0-lxd-15.maas`. +4. => published databag = `certificate_name`+`sans:["10.12.12.122"]`, **no common_name** => vault's + `all_requests` skips it => client-cert-only => `certificates.server.certs.available` never sets + => `/etc/ovn/{cert_host,key_host,ovn-central.crt}` never written (MEASURED absent) => raft NB/SB + cluster can't SSL-handshake on 6643/6644 (`cluster/status` = "joining cluster, Cluster ID: not + yet known") => the `pssl:` client listeners 6641/6642 open only on a cluster leader + (`configure_ovn_listener`, `if status.is_cluster_leader`) => nothing listens on 6641/6642. +5. **Sibling comparison that proves the mechanism generic (MEASURED):** `mysql-innodb-cluster` + publishes the IDENTICAL CN-less shape, gets the IDENTICAL client-cert-only response, and is + healthy -- it falls back to MySQL's own auto-generated self-signed cert (issuer + `CN=MySQL_Server_8.0.46_Auto_Generated_CA_Certificate`). `ovn-chassis` WORKS because it OVERRIDES + the CN with the OVS-DB hostname (`vr1-dc0-compute-01.maas`), never DNS. ovn-central has neither a + fallback nor an override. + +### What the prior diagnosis got wrong +- **"LP #2044324 CONFIRMED root cause"** -> **NO MATCH** (MEASURED against the bug's content). The + bug blocks on "Ports which should be open, but are not: 6641, 6642" (a post-cert port check) with + NO binding at all and "inconsistently"; ours is "awaiting server certificate data", binding + PRESENT, request PUBLISHED, DETERMINISTIC. The bug is untriaged 2.5y, zero comments, lxd-cloud. + No upstream fix exists (github.com/canonical/charm-ovn-central; opendev mirror is archived), and + no channel/revision changes cert-request construction (24.03/stable moved 311->332, nothing + relevant; no 24.09 track). A channel bump will NOT fix this. +- **"vault cannot sign because common_name is missing."** -> vault is never ASKED to sign; no server + request object is created for a CN-less databag. mysql demonstrates identical treatment + health. +- **The "Skipping request for certificate for ip in , no local address found" log line the + prior session leaned on** does NOT causally drive this, and per the reviewer's grep does not even + appear as that exact string in rev 311 -- the prior session likely quoted a different charm + version's log. + +### CORRECTION to an over-claim MADE DURING this re-eval (owned) +An interim framing here said "the default binding is NOT the lever, PROVEN by: broken at deploy under +metal-admin default + the rebind changed nothing." That "proven" is WITHDRAWN. Source shows the CN +derives from `private-address`, which FOLLOWS the default binding. So the D-052 amendment (default +'' metal-admin -> metal-internal, commit 53e046e) is ON the causal path: it moves private-address +from PTR-bearing eth0 to PTR-less eth1. Whether ovn-central was already CN-broken at deploy (default +still metal-admin) is **UNKNOWN and not reconstructable** (debug-log rotated; juju keeps no +historical relation data). The corrected root cause rests on the CURRENT measured state, which is +solid; the deploy-time story is not needed for it. + +## Remedies (ranked; all are MUTATIONS -> operator-gated, NOT executed) + +**PRIMARY -- add reverse-DNS for the metal-internal ovn-central addresses, then re-fire.** +- vault's PKI role signs with `allow_any_name=True`/`enforce_hostnames=False` (MEASURED by the + vault-source agent, `vault_pki.py`), and OVN peers connect by IP (already in the SAN), so the CN + CONTENT is near-irrelevant -- it only needs to be NON-EMPTY. +- Add PTRs in MAAS for 10.12.12.122/.108/.104 (the three ovn-central units). FEASIBILITY (MEASURED): + the reverse zone `12.12.10.in-addr.arpa` IS served by MAAS (has SOA), but has NO PTRs for these + juju-assigned addresses (MAAS did not allocate them), so this is "register records", not "already + managed". CONFIRM FIRST (operator, read-only): `maas subnets read` -> `rdns_mode`/ + `managed` for the 10.12.12.0/22 row. +- Re-fire: the request handler is gated `when_not is-update-status-hook`, so DNS change alone won't + re-run it -- a no-op `juju config` touch or a certificates relation cycle re-publishes; + `request_server_cert` overwrites the empty CN. +- APPLY TO ALL THREE UNITS. The raft cluster + 6641/6642 need all three to hold certs at once. +- **ACCEPTANCE CRITERIA (do NOT use "unit goes active" -- it is a FALSE GREEN; workload status gates + only on the cert flag, not cluster health):** (1) `ovn-central_N.server.cert` appears in the + databag; (2) `/etc/ovn/{cert_host,key_host,ovn-central.crt}` written on each unit; (3) after all + three, `ovs-appctl cluster/status OVN_Northbound` shows a real Cluster ID + a leader, and 6641/6642 + actually listen. + +**PROOF-FIRST DIAGNOSTIC (recommended before the DNS change; low-risk, one unit).** Add an +`/etc/hosts` line on ovn-central/0 mapping 10.12.12.122 -> a `.maas` name (nsswitch = files dns), +poke the certificates relation, and confirm criteria (1)+(2) appear. This proves the CN is the only +issuance blocker WITHOUT touching MAAS DNS. Caveats (MEASURED/reviewer): juju's machine agent +rewrites `/etc/hosts`, so it is EPHEMERAL -- read the result immediately after the poke; and it can +only establish (1)+(2), never cluster health. + +**ALTERNATIVE, D-NNN-shaped (NOT recommended without a ruling) -- revert the default binding to +metal-admin.** Since eth0/metal-admin already has a working PTR, moving private-address back there +would yield a resolvable CN with NO DNS change. BUT it reverses the standing metal-internal-binding +decision (commits 580fc06/53e046e) and has a LARGE blast radius: it moves ovsdb-peer bound-address, +the SAN set, the firewall `cluster_remote_addrs`, and the raft addresses already baked into the +joining DBs (`Address: ssl:10.12.12.122:6643`). Flag for the operator as a decision, not a fix. + +**RULED OUT (MEASURED):** charm-config CN lever -- rev 311 `config.yaml` has no `os-*-network`/ +`os-*-hostname`/`ssl_*` option, `metadata.yaml` declares no internal/admin/public extra-bindings. +Accept-degraded -- total OVN control-plane outage; no non-TLS or manual-cert escape in rev 311 +(`actions.yaml` has no cert action). + +## WHY NO PTR FOR metal-internal (operator asked; MEASURED from the unit resolver) +MAAS DID register the metal-internal addresses in FORWARD DNS but never generated their REVERSE +(PTR) records. Measured `getent hosts` on ovn-central/0 (resolver 127.0.0.53 -> maas): +- metal-admin `10.12.8.185` : forward `juju-237a74-0-lxd-15.maas` AND reverse PTR -> BOTH present. +- metal-internal `10.12.12.122`: forward `eth1.juju-237a74-0-lxd-15.maas` present; reverse PTR ABSENT. +- metal-internal `10.12.12.114` (keystone): forward `eth1.juju-237a74-0-lxd-7.maas`; reverse ABSENT. +So MAAS knows these addresses (forward A records exist under `eth1..maas`); it simply does not +publish PTRs for the metal-internal plane. Reverse-DNS GENERATION (MAAS subnet `rdns_mode`) is +effectively enabled for the metal-admin subnet (10.12.8.0/22, the PXE/provisioning plane MAAS fully +manages) and NOT for metal-internal (10.12.12.0/22, a juju-consumed L3 segment). This is consistent +with the deployment's deliberate metal-admin-centric DNS posture (D-131 scopes the node-DNS forwarder +to metal-admin only). It is default/plane-scoping behavior, not a defect -- it only became +load-bearing because ovn-central's default cert path derives its CN via a REVERSE lookup of its +metal-internal address, which the API charms' cert path does not do the same way. +CONSEQUENCE FOR THE REMEDY (refines Remedy 1): because the FORWARD records already exist, the +cleanest fix is to ENABLE reverse-DNS on the metal-internal subnet in MAAS (`rdns_mode`), which makes +MAAS auto-generate PTRs for the addresses it already knows -- one subnet-config toggle + DNS reload, +rather than hand-creating per-address PTRs. Confirm the current `rdns_mode`/`managed` for 10.12.12.0/22 +via `maas subnets read` first (operator; the MAAS API key is behind the secrets guard so +this session did not read it). NOTE the reverse zone itself is already MAAS-served (SOA present for +12.12.10.in-addr.arpa) but unpopulated -- consistent with rdns generation being off for that subnet. + +## Residual UNKNOWNs (before executing) +- `rdns_mode`/`managed` for 10.12.12.0/22 in MAAS (needs the MAAS API; not read this session -- + secrets rule kept the reviewer off the API key). Decides whether PTRs can simply be added. +- End-to-end confirmation that a non-empty CN yields a usable server cert AND a formed cluster -- the + proof-first diagnostic + all-three rollout establishes this. + +## SEC observation (file as SEC-033) +`juju show-unit ` exposes vault's GLOBAL-CLIENT private key +(`client.key` PEM) in the relation databag in cleartext to any juju model reader. Inherent to the +tls-certificates global-client model, not newly introduced; mitigate via juju model RBAC. diff --git a/docs/audit/ovn-central-cert-remediation-plan-20260803.md b/docs/audit/ovn-central-cert-remediation-plan-20260803.md new file mode 100644 index 0000000..4f3c804 --- /dev/null +++ b/docs/audit/ovn-central-cert-remediation-plan-20260803.md @@ -0,0 +1,98 @@ +# ovn-central server-cert remediation -- rdns_mode + re-fire (DRAFT, operator-gated) + +Companion to `docs/audit/ovn-central-cert-reeval-20260803.md`. Every mutating step is GATED +(present -> justify -> individual human approval). Nothing here is executed by the drafting session. +Root cause (measured): ovn-central derives its cert CN from a REVERSE lookup of its metal-internal +address (10.12.12.x); the metal-internal plane has forward A records but NO reverse PTRs, so +get_hostname() -> None -> empty common_name -> vault issues no server cert -> ovn-central blocks. +Fix: enable reverse-DNS generation on the metal-internal subnet so PTRs auto-populate for the +addresses MAAS already knows, then re-fire the cert request. + +## Access + guards (all MAAS ops) +- MAAS admin for dc0 runs FROM voffice1 via the SSH tunnel to the dc0 region API + (`10.12.8.6:5240`), profile **`vr1-dc0-region`** (CURRENT-STATE:447-456). voffice1 cannot reach + `10.12.8.6:5240` directly -- bring up the rack-originated tunnel `-L 127.0.0.1:5241:10.12.8.6:5240` + first (the profile points at the tunnel endpoint; if the tunnel is down the profile REFUSES, + exit 2 -- it cannot silently fall back to Office1). +- BEFORE ANY dc0 MAAS command, assert the profile resolves to the dc0 rack, NOT Office1 + (a wrong-region mutation is an idempotent no-op that prints success while destroying nothing -- + or worse; this is the repo's most-warned MAAS hazard): + bash scripts/maas-profile-assert.sh vr1-dc0-region hot-kid # expect exit 0 +- juju ops for ovn-central run FROM the dc0 rack (`ssh vr1-dc0-rack 'juju ...'`, D-138). +- The MAAS API key is behind the PreToolUse secrets guard; the operator runs the MAAS commands. + +## STEP 0 -- MEASURE FIRST (read-only; operator runs on voffice1 after the tunnel + assert) +Capture the metal-internal subnet id and its current DNS config, and the metal-admin values to +mirror. Do NOT proceed if `managed` is false on metal-internal (rdns needs a managed subnet -- that +is a bigger change; stop and re-scope). + maas vr1-dc0-region subnets read | \ + jq -r '.[] | select(.cidr|startswith("10.12.8.") or startswith("10.12.12.")) | + "\(.id)\t\(.cidr)\t\(.name)\tmanaged=\(.managed)\trdns_mode=\(.rdns_mode)\tallow_dns=\(.allow_dns)"' +Record: METAL_INTERNAL_ID, its current rdns_mode (expected 0/Disabled), and METAL_ADMIN rdns_mode +(the value to mirror -- expected 2). ACCEPTANCE for this step: metal-internal rdns_mode is currently +0 (or !=metal-admin) AND managed=true. If managed=false -> STOP, re-scope. + +## STEP 1 -- (OPTIONAL, RECOMMENDED) proof-first: confirm CN is the ONLY issuance blocker +Low-risk, one unit, no MAAS change. Proves a non-empty CN yields a server cert before touching DNS. +GATED. On ovn-central/0 (nsswitch = files dns, MEASURED): + ssh vr1-dc0-rack "juju exec --unit ovn-central/0 -- bash -c ' + echo \"10.12.12.122 ovn-central-0.proof.local\" | sudo tee -a /etc/hosts + '" +Then re-fire ONLY this unit's request (see STEP 3's per-unit form) and read the result IMMEDIATELY +(juju's machine agent rewrites /etc/hosts, so the entry is EPHEMERAL): + - PASS iff `ovn-central_0.server.cert` appears in `relation-get -r certificates:142 - vault/0`'s + view (or `juju show-unit ovn-central/0` shows it) AND `/etc/ovn/{cert_host,key_host,ovn-central.crt}` + get written. This proves ONLY cert issuance (criteria 1-2), never cluster health. +Revert: remove the /etc/hosts line (or let the next hook rewrite it). This is a diagnostic, NOT the fix. + +## STEP 2 -- ENABLE reverse-DNS on the metal-internal subnet (MUTATION, GATED) +Set metal-internal's rdns_mode to match metal-admin's captured value (expected 2 = Enabled + RFC2317). + # re-assert profile first + bash scripts/maas-profile-assert.sh vr1-dc0-region hot-kid + maas vr1-dc0-region subnet update rdns_mode= +MAAS regenerates DNS on the change. Then VERIFY PTRs populated (read-only, from a unit's resolver): + ssh vr1-dc0-rack "juju exec --unit ovn-central/0 -- bash -c ' + for ip in 10.12.12.122 10.12.12.108 10.12.12.104; do printf \"%s -> \" \$ip; getent hosts \$ip || echo NONE; done'" + ACCEPTANCE: all three now resolve to `eth1..maas` (was NONE). If still NONE after a minute, + the region BIND may need a reload -- capture, do NOT loop blindly; investigate before re-running. +Revert: `maas vr1-dc0-region subnet update rdns_mode=` (from STEP 0). + +## STEP 3 -- RE-FIRE the cert request on ALL THREE units (MUTATION, GATED) +The request handler is gated `when_not is-update-status-hook`, so DNS alone will not re-run it. Cycle +the app-level certificates relation once -- this re-runs the joined/changed handlers on all three +units together, and `request_server_cert` overwrites the empty CN with the now-resolvable name: + ssh vr1-dc0-rack "juju remove-relation ovn-central:certificates vault:certificates" + # wait for the relation to fully depart, then: + ssh vr1-dc0-rack "juju integrate ovn-central:certificates vault:certificates" +(Least-disruptive alternative if a full bounce is undesirable: `juju config ovn-central +ovsdb-server-election-timer=5` then back to `=4` to drive config-changed -- benign, the cluster is +already dead. The relation bounce is the deterministic, prev-session-proven trigger and is preferred +because it re-fires all three units at once.) NOTE the relation currently in use is certificates:142; +a bounce mints a new relation id -- expected. +Revert: re-integrate (the relation is self-healing); no data loss (ovn-central has no working server +cert to lose). + +## STEP 4 -- ACCEPTANCE (read-only). DO NOT use "unit active" as the gate -- it is a FALSE GREEN. +Workload status gates only on the cert flag, not cluster health; a single unit can read `active` while +the OVN cluster stays dead. Require, IN ORDER: + 1. Each unit publishes a server cert: `juju show-unit ovn-central/N` -> vault's view carries + `ovn-central_N.server.cert` (not just ca/client.cert). All three. + 2. `/etc/ovn/{cert_host,key_host,ovn-central.crt}` written on all three units. + 3. Cluster formed (only after ALL THREE hold certs): + `ssh vr1-dc0-rack "juju exec --unit ovn-central/0 -- ovs-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound"` + -> a real `Cluster ID` and a `Leader` (not "joining cluster / not yet known"); same for OVN_Southbound. + 4. Listeners up: `ss -tlnp | grep -E '6641|6642'` present on the cluster leader. + 5. THEN `juju status ovn-central` -> active (consequence, not criterion), and the OVN consumers + (neutron-api-plugin-ovn, ovn-chassis) settle. Re-run `bash scripts/cloud-assert.sh` for the + behavioral verdict. + +## Classification + follow-ups (operator) +- Is enabling rdns on metal-internal an OPS change or a D-NNN? It unblocks a charm (OPS-leaning) but + mildly adjusts the metal-admin-only DNS posture (D-131). Roosevelt-delta: EVERY DC where ovn-central + binds metal-internal needs this same rdns enablement, OR the CN derivation should not depend on a + reverse lookup. Recommend recording the ruling and, if adopted, folding rdns-on-east-west-planes into + the DC-standup definition-of-done (dc-plane-ipam.sh / the region MAAS config step). +- No tested artifact exists for "enable rdns + re-fire ovn-central cert" (raw maas/juju here, gated). If + this recurs per-DC, build one with a harness (repo norm). Flagged, not a licence to improvise beyond + this one-off. +- SEC-033 (relation-databag exposes vault's global-client private key) is independent -- file separately. diff --git a/docs/audit/stage5-sweep-misses-20260803.md b/docs/audit/stage5-sweep-misses-20260803.md new file mode 100644 index 0000000..645e7ab --- /dev/null +++ b/docs/audit/stage5-sweep-misses-20260803.md @@ -0,0 +1,75 @@ +# Stage-5 sweep for "other misses like the ovn-central one" -- 2026-08-03 + +Triggered after the ovn-central cert re-eval. Class hunted: silent failures / latent +misconfigurations, especially per-plane facility gaps (like metal-internal's missing reverse DNS) +and incomplete-transition fallout, that were not surfaced in CURRENT-STATE. Method: live census +from the dc0 rack (read-only) + per-plane DNS matrix + targeted probes. All MEASURED this session. + +## Census (juju status, model vr1-dc0): 55 active, 6 blocked, 4 waiting, 1 unknown, 1 ERROR. + +## FINDING 1 -- metal-internal is the ONLY plane without reverse DNS; standup tooling ignores rdns_mode +Per-plane reverse-DNS (MEASURED via a unit's resolver, one live addr each): + provider-public 10.12.4.x -> resolves (br-ex..maas) + metal-admin 10.12.8.x -> resolves (juju-.maas) + metal-internal 10.12.12.x -> **NONE** <-- the outlier + data-tenant 10.12.16.x -> resolves (eth2..maas) + storage 10.12.32.x -> resolves (eth1..maas) + 10.12.36.x -> resolves (eth2..maas) +So this is NOT a "metal-admin-only DNS" posture (my earlier framing, corrected): FIVE of six planes +have reverse DNS and metal-internal alone lacks it. Origin (MEASURED): `phase-00-maas-standup.sh` +sets per-subnet `gateway_ip`, `managed=true`, and `dns_servers`, but **never `rdns_mode`** -- so +reverse-DNS enablement is unmanaged and inconsistent across planes, and metal-internal ended up +without it. This is the ROOT of the ovn-central cert failure and a TOOLING GAP. +- Remediation for the live symptom: `docs/audit/ovn-central-cert-remediation-plan-20260803.md`. +- Systemic fix (Roosevelt-delta): have the standup tooling MANAGE `rdns_mode` per subnet + (assert/set it, with a harness), so every DC's planes are reverse-DNS-consistent by construction. + Every future DC would otherwise inherit the same inconsistency. Classify OPS-vs-D-NNN with the + operator (leans toward a standup-DoD addition). +- Exact per-subnet `rdns_mode` values need an operator MAAS read (`maas vr1-dc0-region subnets read`; + key behind the secrets guard). Behaviorally, metal-internal reads as rdns-off, the rest rds-on. + +## FINDING 2 -- 4 hacluster subordinates BLOCKED on stale/malformed IPv6 VIP resources (D-141 fallout) +barbican-hacluster, designate-hacluster, magnum-hacluster, placement-hacluster all blocked: +"Resource: res__None_ipv6addr_vip not running". MEASURED on barbican-hacluster/0: +- `grp_barbican_vips` contains the three v4 IPaddr2 VIPs (Started) PLUS **TWO** IPv6addr resources + for the SAME v6 address `2602:f3e2:f02:21::51`: + res_barbican_eth0_ipv6addr_vip (IPv6addr, nic=eth0) -> Started + res_barbican_None_ipv6addr_vip (IPv6addr, nic=None) -> NOT running <-- the block + An IPv6addr resource with `nic=None` cannot start; hence blocked. +- The `ha` relation data (json_groups / json_resource_params) still carries these v6 VIP resources, + even though the PRINCIPAL's `juju config barbican vip` is now v4-only ("10.12.4.51 10.12.8.51 + 10.12.12.51"). So the LIVE hacluster retains IPv6 VIP resources from the PRE-D-141 dual-stack + deploy that were never cleaned up when the model was reverted to v4-only. +- CLASS: incomplete D-141 v4-only transition on the LIVE model -- the SAME family as the + provider-bundle-check harness reconcile (a built-surface reverted without its companion change). + Not in CURRENT-STATE. Silently blocking 4 units. +- Follow-up (not done): determine why the v6 resources persist (stale pacemaker CIB not reconciled + after the vip config change, vs the charm still computing a v6 VIP from somewhere) and clean them + (likely `juju run /leader update-status` won't clear a CIB resource; may need the + hacluster charm to re-render, or a manual `crm configure delete res_*_None_ipv6addr_vip` GATED). + Also confirm whether the prefer-ipv6 charms' haclusters are clean (census: they were active). + +## FINDING 3 -- octavia/0 in ERROR (not a cert bug; downstream TLS to neutron) +octavia/0: hook failed "certificates-relation-changed", looping every ~5 min. Root (MEASURED +traceback): the hook calls neutron at `https://10.12.4.55:9696/v2.0/networks` and gets +`SSLError(SSLEOFError ... UNEXPECTED_EOF_WHILE_READING)`. Probing neutron-api's VIP directly: +`openssl s_client -connect 10.12.4.55:9696` -> CONNECTED but **"no peer certificate available"** -- +neutron-api's VIP is not presenting a server cert / not serving TLS cleanly. So octavia's error is +DOWNSTREAM of neutron-api's VIP TLS being unhealthy, itself plausibly tied to the OVN NB/SB outage +(neutron/OVN ML2 unhealthy) OR a separate neutron HAProxy/cert issue. Not in CURRENT-STATE. +- Follow-up (not done): determine why neutron's VIP presents no cert -- is HAProxy up with the vault + cert? is the neutron backend down because OVN is down? Fixing ovn-central may clear this; verify + after. Distinct from the ovn-central cert root cause. + +## KNOWN / EXPECTED TAIL (for completeness -- NOT new misses) +ceph-rbd-mirror/0 (cross-DC, 'ceph-remote' missing), barbican-vault/0 ('secrets-storage' +incomplete), designate/0 ('nameservers must be set' -- Stage 7), glance-simplestreams-sync/0 +(unknown) -- all previously known/expected. + +## META-OBSERVATION +Findings 1 and 2, plus the already-fixed provider-bundle-check harness, are all tails of INCOMPLETE +TRANSITIONS: the D-141 v4-only revert and the plane build-out each left companion changes undone +(hacluster CIB, harness) or a facility inconsistent (rdns per plane). Recommend a deliberate +"D-141 transition completeness" and "per-plane facility parity" pass rather than fixing each tail +as it surfaces. Severity order for action: F1 (blocks ovn-central + Roosevelt-delta) > F2 (blocks 4 +units) > F3 (1 unit, likely downstream).