# ovn-central "awaiting server certificate data" -- ROOT-CAUSE RE-EVALUATION, 2026-08-03

Re-evaluation of the prior 2026-08-03 diagnosis (the one the operator flagged as unreliable,
made under context-length pressure). Method: fresh live measurement (read-only, from the dc0 rack
per D-138) + three parallel deployed-source agents + one adversarial-review agent. Every claim
below is tagged MEASURED (read live/source this session) or INFERRED/UNKNOWN. Model `vr1-dc0`;
ovn-central rev 311 (24.03/stable); vault rev 724 (1.8/stable).

## VERDICT: the prior root cause was WRONG. Corrected root cause below, confirmed end-to-end.

### Corrected root cause (MEASURED at every hop)
ovn-central publishes a certificate request with an EMPTY common_name, so vault never creates a
server-cert request for it, so ovn-central blocks -- and the empty CN is because ovn-central's
address has no reverse-DNS PTR:

1. **vault issues a per-unit SERVER cert ONLY** when the requester databag carries a truthy
   top-level `common_name` OR a `cert_requests` map (vault charm `hooks/relations/tls-certificates/
   provides.py:179,191`). Otherwise it publishes ONLY the global client cert (ca+client.cert+
   client.key), unconditionally (`reactive/vault_handlers.py:1094-1121`). BY DESIGN.
2. **ovn-central uses the default charms.openstack cert path** (it does NOT override
   `get_certificate_requests`; `reactive/layer_openstack.py:127-137` -> `charms_openstack/charm/
   classes.py:284-291` -> `charmhelpers/contrib/openstack/cert_utils.py`). That path sets
   `cn = get_hostname(local_address(unit_get_fallback='private-address'))` (`cert_utils.py`
   `add_hostname_cn`).
3. **`private-address` = eth1 `10.12.12.122` (metal-internal).** MEASURED `unit-get private-address`.
   `get_hostname("10.12.12.122")` returns **None** -- `gethostbyaddr` raises herror; the
   metal-internal plane 10.12.12.0/22 has NO PTR records. Contrast MEASURED: eth0 `10.12.8.185`
   (metal-admin) reverse-resolves to `juju-237a74-0-lxd-15.maas`.
4. => published databag = `certificate_name`+`sans:["10.12.12.122"]`, **no common_name** => vault's
   `all_requests` skips it => client-cert-only => `certificates.server.certs.available` never sets
   => `/etc/ovn/{cert_host,key_host,ovn-central.crt}` never written (MEASURED absent) => raft NB/SB
   cluster can't SSL-handshake on 6643/6644 (`cluster/status` = "joining cluster, Cluster ID: not
   yet known") => the `pssl:` client listeners 6641/6642 open only on a cluster leader
   (`configure_ovn_listener`, `if status.is_cluster_leader`) => nothing listens on 6641/6642.
5. **Sibling comparison that proves the mechanism generic (MEASURED):** `mysql-innodb-cluster`
   publishes the IDENTICAL CN-less shape, gets the IDENTICAL client-cert-only response, and is
   healthy -- it falls back to MySQL's own auto-generated self-signed cert (issuer
   `CN=MySQL_Server_8.0.46_Auto_Generated_CA_Certificate`). `ovn-chassis` WORKS because it OVERRIDES
   the CN with the OVS-DB hostname (`vr1-dc0-compute-01.maas`), never DNS. ovn-central has neither a
   fallback nor an override.

### What the prior diagnosis got wrong
- **"LP #2044324 CONFIRMED root cause"** -> **NO MATCH** (MEASURED against the bug's content). The
  bug blocks on "Ports which should be open, but are not: 6641, 6642" (a post-cert port check) with
  NO binding at all and "inconsistently"; ours is "awaiting server certificate data", binding
  PRESENT, request PUBLISHED, DETERMINISTIC. The bug is untriaged 2.5y, zero comments, lxd-cloud.
  No upstream fix exists (github.com/canonical/charm-ovn-central; opendev mirror is archived), and
  no channel/revision changes cert-request construction (24.03/stable moved 311->332, nothing
  relevant; no 24.09 track). A channel bump will NOT fix this.
- **"vault cannot sign because common_name is missing."** -> vault is never ASKED to sign; no server
  request object is created for a CN-less databag. mysql demonstrates identical treatment + health.
- **The "Skipping request for certificate for ip in <space>, no local address found" log line the
  prior session leaned on** does NOT causally drive this, and per the reviewer's grep does not even
  appear as that exact string in rev 311 -- the prior session likely quoted a different charm
  version's log.

### CORRECTION to an over-claim MADE DURING this re-eval (owned)
An interim framing here said "the default binding is NOT the lever, PROVEN by: broken at deploy under
metal-admin default + the rebind changed nothing." That "proven" is WITHDRAWN. Source shows the CN
derives from `private-address`, which FOLLOWS the default binding. So the D-052 amendment (default
'' metal-admin -> metal-internal, commit 53e046e) is ON the causal path: it moves private-address
from PTR-bearing eth0 to PTR-less eth1. Whether ovn-central was already CN-broken at deploy (default
still metal-admin) is **UNKNOWN and not reconstructable** (debug-log rotated; juju keeps no
historical relation data). The corrected root cause rests on the CURRENT measured state, which is
solid; the deploy-time story is not needed for it.

## Remedies (ranked; all are MUTATIONS -> operator-gated, NOT executed)

**PRIMARY -- add reverse-DNS for the metal-internal ovn-central addresses, then re-fire.**
- vault's PKI role signs with `allow_any_name=True`/`enforce_hostnames=False` (MEASURED by the
  vault-source agent, `vault_pki.py`), and OVN peers connect by IP (already in the SAN), so the CN
  CONTENT is near-irrelevant -- it only needs to be NON-EMPTY.
- Add PTRs in MAAS for 10.12.12.122/.108/.104 (the three ovn-central units). FEASIBILITY (MEASURED):
  the reverse zone `12.12.10.in-addr.arpa` IS served by MAAS (has SOA), but has NO PTRs for these
  juju-assigned addresses (MAAS did not allocate them), so this is "register records", not "already
  managed". CONFIRM FIRST (operator, read-only): `maas <profile> subnets read` -> `rdns_mode`/
  `managed` for the 10.12.12.0/22 row.
- Re-fire: the request handler is gated `when_not is-update-status-hook`, so DNS change alone won't
  re-run it -- a no-op `juju config` touch or a certificates relation cycle re-publishes;
  `request_server_cert` overwrites the empty CN.
- APPLY TO ALL THREE UNITS. The raft cluster + 6641/6642 need all three to hold certs at once.
- **ACCEPTANCE CRITERIA (do NOT use "unit goes active" -- it is a FALSE GREEN; workload status gates
  only on the cert flag, not cluster health):** (1) `ovn-central_N.server.cert` appears in the
  databag; (2) `/etc/ovn/{cert_host,key_host,ovn-central.crt}` written on each unit; (3) after all
  three, `ovs-appctl cluster/status OVN_Northbound` shows a real Cluster ID + a leader, and 6641/6642
  actually listen.

**PROOF-FIRST DIAGNOSTIC (recommended before the DNS change; low-risk, one unit).** Add an
`/etc/hosts` line on ovn-central/0 mapping 10.12.12.122 -> a `.maas` name (nsswitch = files dns),
poke the certificates relation, and confirm criteria (1)+(2) appear. This proves the CN is the only
issuance blocker WITHOUT touching MAAS DNS. Caveats (MEASURED/reviewer): juju's machine agent
rewrites `/etc/hosts`, so it is EPHEMERAL -- read the result immediately after the poke; and it can
only establish (1)+(2), never cluster health.

**ALTERNATIVE, D-NNN-shaped (NOT recommended without a ruling) -- revert the default binding to
metal-admin.** Since eth0/metal-admin already has a working PTR, moving private-address back there
would yield a resolvable CN with NO DNS change. BUT it reverses the standing metal-internal-binding
decision (commits 580fc06/53e046e) and has a LARGE blast radius: it moves ovsdb-peer bound-address,
the SAN set, the firewall `cluster_remote_addrs`, and the raft addresses already baked into the
joining DBs (`Address: ssl:10.12.12.122:6643`). Flag for the operator as a decision, not a fix.

**RULED OUT (MEASURED):** charm-config CN lever -- rev 311 `config.yaml` has no `os-*-network`/
`os-*-hostname`/`ssl_*` option, `metadata.yaml` declares no internal/admin/public extra-bindings.
Accept-degraded -- total OVN control-plane outage; no non-TLS or manual-cert escape in rev 311
(`actions.yaml` has no cert action).

## WHY NO PTR FOR metal-internal (operator asked; MEASURED from the unit resolver)
MAAS DID register the metal-internal addresses in FORWARD DNS but never generated their REVERSE
(PTR) records. Measured `getent hosts` on ovn-central/0 (resolver 127.0.0.53 -> maas):
- metal-admin `10.12.8.185`   : forward `juju-237a74-0-lxd-15.maas` AND reverse PTR -> BOTH present.
- metal-internal `10.12.12.122`: forward `eth1.juju-237a74-0-lxd-15.maas` present; reverse PTR ABSENT.
- metal-internal `10.12.12.114` (keystone): forward `eth1.juju-237a74-0-lxd-7.maas`; reverse ABSENT.
So MAAS knows these addresses (forward A records exist under `eth1.<node>.maas`); it simply does not
publish PTRs for the metal-internal plane. Reverse-DNS GENERATION (MAAS subnet `rdns_mode`) is
effectively enabled for the metal-admin subnet (10.12.8.0/22, the PXE/provisioning plane MAAS fully
manages) and NOT for metal-internal (10.12.12.0/22, a juju-consumed L3 segment). This is consistent
with the deployment's deliberate metal-admin-centric DNS posture (D-131 scopes the node-DNS forwarder
to metal-admin only). It is default/plane-scoping behavior, not a defect -- it only became
load-bearing because ovn-central's default cert path derives its CN via a REVERSE lookup of its
metal-internal address, which the API charms' cert path does not do the same way.
CONSEQUENCE FOR THE REMEDY (refines Remedy 1): because the FORWARD records already exist, the
cleanest fix is to ENABLE reverse-DNS on the metal-internal subnet in MAAS (`rdns_mode`), which makes
MAAS auto-generate PTRs for the addresses it already knows -- one subnet-config toggle + DNS reload,
rather than hand-creating per-address PTRs. Confirm the current `rdns_mode`/`managed` for 10.12.12.0/22
via `maas <profile> subnets read` first (operator; the MAAS API key is behind the secrets guard so
this session did not read it). NOTE the reverse zone itself is already MAAS-served (SOA present for
12.12.10.in-addr.arpa) but unpopulated -- consistent with rdns generation being off for that subnet.

## >>> CORRECTION 2026-08-04 (MEASURED): the "no PTR / enable rdns_mode" diagnosis above is WRONG. <<<
Measured from the dc0 region (profile vr1-dc0-region, guarded) and the ovn-central/0 unit:
- **`rdns_mode=2, managed=True` on ALL SIX planes**, metal-internal included. rdns is NOT the cause;
  the "enable rdns_mode" remediation is VOID (a no-op).
- **The PTR EXISTS.** The region BIND at 10.12.8.6 answers `dig -x 10.12.12.122` -> `eth1.juju-...maas`
  correctly. The failure is the RESOLUTION PATH from the unit, not PTR generation.
- **The metal-internal plane is ISOLATED and has NO reachable resolver.** The region controller
  `hot-kid` has interfaces ONLY on metal-admin (10.12.8.6) and provider-public (10.12.4.6) -- none on
  metal-internal. metal-internal subnet `dns_servers=[]`, so MAAS advertises the provider-public
  `10.12.4.6`, which is unroutable from metal-internal. On the unit, routes are LINK-SCOPED only
  (`10.12.8.0/22 dev eth0`, `10.12.12.0/22 dev eth1`); `ping -I eth1 10.12.8.6` -> **NO ROUTE**.
- **systemd-resolved scopes the reverse query of a container's OWN metal-internal address to eth1**,
  which cannot route to any resolver -> "No route to host" -> `get_hostname()`=None -> empty CN.
  DISCRIMINATING TEST (advisor-directed): setting BOTH links to only 10.12.8.6 (confirmed applied) +
  flush STILL failed "No route to host" for 10.12.12.122, while 10.12.16.1 resolved via eth0. So it
  is NOT a `dns_servers`/`allow_dns` change either. Non-local addresses resolve via eth0; a
  container's OWN metal-internal address cannot, because metal-internal is isolated by design.
- **CORRECTED root cause:** ovn-central (rev 311) derives its cert CN from a REVERSE lookup of its
  default-binding (private) address, which is on the DELIBERATELY-ISOLATED metal-internal plane that
  has no reachable resolver -- so the CN is empty and no server cert is issued. This is a structural
  incompatibility between charm-ovn-central's CN-from-reverse-DNS behaviour and binding the OVN DB
  plane to an isolated network. The refutation of LP #2044324 and the vault-issuance rule above STAND.
- **CORRECTED fix framing (a real decision -- see the remediation plan's CORRECTION):** the fix must
  DECOUPLE ovn-central's cert CN from metal-internal reverse-DNS. It is a PRECONDITION for any
  redeploy (step 3 redeploys onto the SAME MAAS; fresh containers fail identically otherwise), and it
  must be verified at PROVISION time (what MAAS hands a NEW container), not just runtime-poked.

## >>> RESOLUTION 2026-08-04 (PROVEN live, then wired for redeploy). <<<
**Root cause (final, measured):** charm-ovn-central (rev 311) computes its TLS `common_name` as
`get_hostname(local_address())`, and `local_address()` is the unit's metal-internal address
(10.12.12.x) -- the DELIBERATELY-ISOLATED D-052 plane with no gateway and no reachable DNS
resolver. `get_hostname()` -> None -> empty CN -> vault issues no server cert. rdns_mode=2 and
the MAAS forward+reverse records exist at the region BIND; the reverse is simply UNREACHABLE
from the isolated plane. The charm has NO config override for the CN (measured: config.yaml).

**Binding approach REFUTED, live (three configs):** `''`->metal-admin, `certificates`->metal-admin,
and `''`->metal-admin+bounce all left the CN empty -- the charm derives the CN from the
metal-internal address regardless of which endpoint is rebound. Also `juju`/MAAS default-route
selection is non-deterministic and not tied to the default binding (Juju charm-guide: a binding
"is not a mechanism that alters or verifies routing"; LP#1781856). So the app STAYS on
metal-internal (D-052-correct for its OVSDB/certificates data type); the fix does NOT move it.

**Fix PROVEN end-to-end, live 2026-08-04 (controlled single-unit test):** an `/etc/hosts` entry
`10.12.12.122 <name>` on ovn-central/0 made the charm's own `get_hostname("10.12.12.122")`
resolve, the charm published `common_name`, vault issued `ovn-central_0.server.cert`, and the
charm wrote `/etc/ovn/{cert_host,key_host,ovn-central.crt}` -- while control units /1 and /2
(no entry) stayed broken. The entry is the sole cause of the fix. OVN imposes NO CN-content
rule on the server/DB cert and vault signs any non-empty CN, so the name only needs to be
non-empty and stable.

**Wired for the redeploy (verify-at-provision):** `scripts/dc-node-etchosts.sh` renders a
model-wide `cloudinit-userdata` (harness `tests/dc-node-etchosts` 9/9) that, at machine
provision, adds each node's metal-internal address -> hostname to `/etc/hosts`, scoped to THIS
DC's metal-internal CIDR pinned from `lib-net.sh` (dc0 10.12.12.0/22, dc1 10.12.72.0/22). The
rendered runcmd was executed live and correctly added the metal-internal entry, skipped the
metal-admin address, and made `get_hostname` resolve. Applied as gated Step 1.2b in
`runbooks/phase-01-bundle-deploy.md` (after add-model, before deploy). **This is a NEW mechanism
that BORROWS D-008's static-/etc/hosts SHAPE but is NOT D-008** (D-008's /etc/hosts is FQDN->VIP
for os-public-hostname, unset in VR1; no /etc/hosts script existed before this). The DELIVERY
(does juju apply it to fresh containers) is only verifiable at the redeploy; the rendered
content and the cert-issuance chain are proven.

## Residual UNKNOWNs (before executing)
- `rdns_mode`/`managed` for 10.12.12.0/22 in MAAS (needs the MAAS API; not read this session --
  secrets rule kept the reviewer off the API key). Decides whether PTRs can simply be added.
- End-to-end confirmation that a non-empty CN yields a usable server cert AND a formed cluster -- the
  proof-first diagnostic + all-three rollout establishes this.

## SEC observation (file as SEC-033)
`juju show-unit <any tls-certificates consumer>` exposes vault's GLOBAL-CLIENT private key
(`client.key` PEM) in the relation databag in cleartext to any juju model reader. Inherent to the
tls-certificates global-client model, not newly introduced; mitigate via juju model RBAC.
