# ovn-central "awaiting server certificate data" -- ROOT-CAUSE RE-EVALUATION, 2026-08-03 Re-evaluation of the prior 2026-08-03 diagnosis (the one the operator flagged as unreliable, made under context-length pressure). Method: fresh live measurement (read-only, from the dc0 rack per D-138) + three parallel deployed-source agents + one adversarial-review agent. Every claim below is tagged MEASURED (read live/source this session) or INFERRED/UNKNOWN. Model `vr1-dc0`; ovn-central rev 311 (24.03/stable); vault rev 724 (1.8/stable). ## VERDICT: the prior root cause was WRONG. Corrected root cause below, confirmed end-to-end. ### Corrected root cause (MEASURED at every hop) ovn-central publishes a certificate request with an EMPTY common_name, so vault never creates a server-cert request for it, so ovn-central blocks -- and the empty CN is because ovn-central's address has no reverse-DNS PTR: 1. **vault issues a per-unit SERVER cert ONLY** when the requester databag carries a truthy top-level `common_name` OR a `cert_requests` map (vault charm `hooks/relations/tls-certificates/ provides.py:179,191`). Otherwise it publishes ONLY the global client cert (ca+client.cert+ client.key), unconditionally (`reactive/vault_handlers.py:1094-1121`). BY DESIGN. 2. **ovn-central uses the default charms.openstack cert path** (it does NOT override `get_certificate_requests`; `reactive/layer_openstack.py:127-137` -> `charms_openstack/charm/ classes.py:284-291` -> `charmhelpers/contrib/openstack/cert_utils.py`). That path sets `cn = get_hostname(local_address(unit_get_fallback='private-address'))` (`cert_utils.py` `add_hostname_cn`). 3. **`private-address` = eth1 `10.12.12.122` (metal-internal).** MEASURED `unit-get private-address`. `get_hostname("10.12.12.122")` returns **None** -- `gethostbyaddr` raises herror; the metal-internal plane 10.12.12.0/22 has NO PTR records. Contrast MEASURED: eth0 `10.12.8.185` (metal-admin) reverse-resolves to `juju-237a74-0-lxd-15.maas`. 4. => published databag = `certificate_name`+`sans:["10.12.12.122"]`, **no common_name** => vault's `all_requests` skips it => client-cert-only => `certificates.server.certs.available` never sets => `/etc/ovn/{cert_host,key_host,ovn-central.crt}` never written (MEASURED absent) => raft NB/SB cluster can't SSL-handshake on 6643/6644 (`cluster/status` = "joining cluster, Cluster ID: not yet known") => the `pssl:` client listeners 6641/6642 open only on a cluster leader (`configure_ovn_listener`, `if status.is_cluster_leader`) => nothing listens on 6641/6642. 5. **Sibling comparison that proves the mechanism generic (MEASURED):** `mysql-innodb-cluster` publishes the IDENTICAL CN-less shape, gets the IDENTICAL client-cert-only response, and is healthy -- it falls back to MySQL's own auto-generated self-signed cert (issuer `CN=MySQL_Server_8.0.46_Auto_Generated_CA_Certificate`). `ovn-chassis` WORKS because it OVERRIDES the CN with the OVS-DB hostname (`vr1-dc0-compute-01.maas`), never DNS. ovn-central has neither a fallback nor an override. ### What the prior diagnosis got wrong - **"LP #2044324 CONFIRMED root cause"** -> **NO MATCH** (MEASURED against the bug's content). The bug blocks on "Ports which should be open, but are not: 6641, 6642" (a post-cert port check) with NO binding at all and "inconsistently"; ours is "awaiting server certificate data", binding PRESENT, request PUBLISHED, DETERMINISTIC. The bug is untriaged 2.5y, zero comments, lxd-cloud. No upstream fix exists (github.com/canonical/charm-ovn-central; opendev mirror is archived), and no channel/revision changes cert-request construction (24.03/stable moved 311->332, nothing relevant; no 24.09 track). A channel bump will NOT fix this. - **"vault cannot sign because common_name is missing."** -> vault is never ASKED to sign; no server request object is created for a CN-less databag. mysql demonstrates identical treatment + health. - **The "Skipping request for certificate for ip in , no local address found" log line the prior session leaned on** does NOT causally drive this, and per the reviewer's grep does not even appear as that exact string in rev 311 -- the prior session likely quoted a different charm version's log. ### CORRECTION to an over-claim MADE DURING this re-eval (owned) An interim framing here said "the default binding is NOT the lever, PROVEN by: broken at deploy under metal-admin default + the rebind changed nothing." That "proven" is WITHDRAWN. Source shows the CN derives from `private-address`, which FOLLOWS the default binding. So the D-052 amendment (default '' metal-admin -> metal-internal, commit 53e046e) is ON the causal path: it moves private-address from PTR-bearing eth0 to PTR-less eth1. Whether ovn-central was already CN-broken at deploy (default still metal-admin) is **UNKNOWN and not reconstructable** (debug-log rotated; juju keeps no historical relation data). The corrected root cause rests on the CURRENT measured state, which is solid; the deploy-time story is not needed for it. ## Remedies (ranked; all are MUTATIONS -> operator-gated, NOT executed) **PRIMARY -- add reverse-DNS for the metal-internal ovn-central addresses, then re-fire.** - vault's PKI role signs with `allow_any_name=True`/`enforce_hostnames=False` (MEASURED by the vault-source agent, `vault_pki.py`), and OVN peers connect by IP (already in the SAN), so the CN CONTENT is near-irrelevant -- it only needs to be NON-EMPTY. - Add PTRs in MAAS for 10.12.12.122/.108/.104 (the three ovn-central units). FEASIBILITY (MEASURED): the reverse zone `12.12.10.in-addr.arpa` IS served by MAAS (has SOA), but has NO PTRs for these juju-assigned addresses (MAAS did not allocate them), so this is "register records", not "already managed". CONFIRM FIRST (operator, read-only): `maas subnets read` -> `rdns_mode`/ `managed` for the 10.12.12.0/22 row. - Re-fire: the request handler is gated `when_not is-update-status-hook`, so DNS change alone won't re-run it -- a no-op `juju config` touch or a certificates relation cycle re-publishes; `request_server_cert` overwrites the empty CN. - APPLY TO ALL THREE UNITS. The raft cluster + 6641/6642 need all three to hold certs at once. - **ACCEPTANCE CRITERIA (do NOT use "unit goes active" -- it is a FALSE GREEN; workload status gates only on the cert flag, not cluster health):** (1) `ovn-central_N.server.cert` appears in the databag; (2) `/etc/ovn/{cert_host,key_host,ovn-central.crt}` written on each unit; (3) after all three, `ovs-appctl cluster/status OVN_Northbound` shows a real Cluster ID + a leader, and 6641/6642 actually listen. **PROOF-FIRST DIAGNOSTIC (recommended before the DNS change; low-risk, one unit).** Add an `/etc/hosts` line on ovn-central/0 mapping 10.12.12.122 -> a `.maas` name (nsswitch = files dns), poke the certificates relation, and confirm criteria (1)+(2) appear. This proves the CN is the only issuance blocker WITHOUT touching MAAS DNS. Caveats (MEASURED/reviewer): juju's machine agent rewrites `/etc/hosts`, so it is EPHEMERAL -- read the result immediately after the poke; and it can only establish (1)+(2), never cluster health. **ALTERNATIVE, D-NNN-shaped (NOT recommended without a ruling) -- revert the default binding to metal-admin.** Since eth0/metal-admin already has a working PTR, moving private-address back there would yield a resolvable CN with NO DNS change. BUT it reverses the standing metal-internal-binding decision (commits 580fc06/53e046e) and has a LARGE blast radius: it moves ovsdb-peer bound-address, the SAN set, the firewall `cluster_remote_addrs`, and the raft addresses already baked into the joining DBs (`Address: ssl:10.12.12.122:6643`). Flag for the operator as a decision, not a fix. **RULED OUT (MEASURED):** charm-config CN lever -- rev 311 `config.yaml` has no `os-*-network`/ `os-*-hostname`/`ssl_*` option, `metadata.yaml` declares no internal/admin/public extra-bindings. Accept-degraded -- total OVN control-plane outage; no non-TLS or manual-cert escape in rev 311 (`actions.yaml` has no cert action). ## WHY NO PTR FOR metal-internal (operator asked; MEASURED from the unit resolver) MAAS DID register the metal-internal addresses in FORWARD DNS but never generated their REVERSE (PTR) records. Measured `getent hosts` on ovn-central/0 (resolver 127.0.0.53 -> maas): - metal-admin `10.12.8.185` : forward `juju-237a74-0-lxd-15.maas` AND reverse PTR -> BOTH present. - metal-internal `10.12.12.122`: forward `eth1.juju-237a74-0-lxd-15.maas` present; reverse PTR ABSENT. - metal-internal `10.12.12.114` (keystone): forward `eth1.juju-237a74-0-lxd-7.maas`; reverse ABSENT. So MAAS knows these addresses (forward A records exist under `eth1..maas`); it simply does not publish PTRs for the metal-internal plane. Reverse-DNS GENERATION (MAAS subnet `rdns_mode`) is effectively enabled for the metal-admin subnet (10.12.8.0/22, the PXE/provisioning plane MAAS fully manages) and NOT for metal-internal (10.12.12.0/22, a juju-consumed L3 segment). This is consistent with the deployment's deliberate metal-admin-centric DNS posture (D-131 scopes the node-DNS forwarder to metal-admin only). It is default/plane-scoping behavior, not a defect -- it only became load-bearing because ovn-central's default cert path derives its CN via a REVERSE lookup of its metal-internal address, which the API charms' cert path does not do the same way. CONSEQUENCE FOR THE REMEDY (refines Remedy 1): because the FORWARD records already exist, the cleanest fix is to ENABLE reverse-DNS on the metal-internal subnet in MAAS (`rdns_mode`), which makes MAAS auto-generate PTRs for the addresses it already knows -- one subnet-config toggle + DNS reload, rather than hand-creating per-address PTRs. Confirm the current `rdns_mode`/`managed` for 10.12.12.0/22 via `maas subnets read` first (operator; the MAAS API key is behind the secrets guard so this session did not read it). NOTE the reverse zone itself is already MAAS-served (SOA present for 12.12.10.in-addr.arpa) but unpopulated -- consistent with rdns generation being off for that subnet. ## >>> CORRECTION 2026-08-04 (MEASURED): the "no PTR / enable rdns_mode" diagnosis above is WRONG. <<< Measured from the dc0 region (profile vr1-dc0-region, guarded) and the ovn-central/0 unit: - **`rdns_mode=2, managed=True` on ALL SIX planes**, metal-internal included. rdns is NOT the cause; the "enable rdns_mode" remediation is VOID (a no-op). - **The PTR EXISTS.** The region BIND at 10.12.8.6 answers `dig -x 10.12.12.122` -> `eth1.juju-...maas` correctly. The failure is the RESOLUTION PATH from the unit, not PTR generation. - **The metal-internal plane is ISOLATED and has NO reachable resolver.** The region controller `hot-kid` has interfaces ONLY on metal-admin (10.12.8.6) and provider-public (10.12.4.6) -- none on metal-internal. metal-internal subnet `dns_servers=[]`, so MAAS advertises the provider-public `10.12.4.6`, which is unroutable from metal-internal. On the unit, routes are LINK-SCOPED only (`10.12.8.0/22 dev eth0`, `10.12.12.0/22 dev eth1`); `ping -I eth1 10.12.8.6` -> **NO ROUTE**. - **systemd-resolved scopes the reverse query of a container's OWN metal-internal address to eth1**, which cannot route to any resolver -> "No route to host" -> `get_hostname()`=None -> empty CN. DISCRIMINATING TEST (advisor-directed): setting BOTH links to only 10.12.8.6 (confirmed applied) + flush STILL failed "No route to host" for 10.12.12.122, while 10.12.16.1 resolved via eth0. So it is NOT a `dns_servers`/`allow_dns` change either. Non-local addresses resolve via eth0; a container's OWN metal-internal address cannot, because metal-internal is isolated by design. - **CORRECTED root cause:** ovn-central (rev 311) derives its cert CN from a REVERSE lookup of its default-binding (private) address, which is on the DELIBERATELY-ISOLATED metal-internal plane that has no reachable resolver -- so the CN is empty and no server cert is issued. This is a structural incompatibility between charm-ovn-central's CN-from-reverse-DNS behaviour and binding the OVN DB plane to an isolated network. The refutation of LP #2044324 and the vault-issuance rule above STAND. - **CORRECTED fix framing (a real decision -- see the remediation plan's CORRECTION):** the fix must DECOUPLE ovn-central's cert CN from metal-internal reverse-DNS. It is a PRECONDITION for any redeploy (step 3 redeploys onto the SAME MAAS; fresh containers fail identically otherwise), and it must be verified at PROVISION time (what MAAS hands a NEW container), not just runtime-poked. ## Residual UNKNOWNs (before executing) - `rdns_mode`/`managed` for 10.12.12.0/22 in MAAS (needs the MAAS API; not read this session -- secrets rule kept the reviewer off the API key). Decides whether PTRs can simply be added. - End-to-end confirmation that a non-empty CN yields a usable server cert AND a formed cluster -- the proof-first diagnostic + all-three rollout establishes this. ## SEC observation (file as SEC-033) `juju show-unit ` exposes vault's GLOBAL-CLIENT private key (`client.key` PEM) in the relation databag in cleartext to any juju model reader. Inherent to the tls-certificates global-client model, not newly introduced; mitigate via juju model RBAC.