Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / audit / ovn-central-cert-reeval-20260803.md
@JANeumatrix JANeumatrix 22 hours ago 10 KB audit: ovn-central cert reeval + remediation plan + Stage-5 misses sweep
Raw Blame History

ovn-central "awaiting server certificate data" -- ROOT-CAUSE RE-EVALUATION, 2026-08-03

Re-evaluation of the prior 2026-08-03 diagnosis (the one the operator flagged as unreliable, made under context-length pressure). Method: fresh live measurement (read-only, from the dc0 rack per D-138) + three parallel deployed-source agents + one adversarial-review agent. Every claim below is tagged MEASURED (read live/source this session) or INFERRED/UNKNOWN. Model vr1-dc0; ovn-central rev 311 (24.03/stable); vault rev 724 (1.8/stable).

VERDICT: the prior root cause was WRONG. Corrected root cause below, confirmed end-to-end.

Corrected root cause (MEASURED at every hop)

ovn-central publishes a certificate request with an EMPTY common_name, so vault never creates a server-cert request for it, so ovn-central blocks -- and the empty CN is because ovn-central's address has no reverse-DNS PTR:

  1. vault issues a per-unit SERVER cert ONLY when the requester databag carries a truthy top-level common_name OR a cert_requests map (vault charm hooks/relations/tls-certificates/ provides.py:179,191). Otherwise it publishes ONLY the global client cert (ca+client.cert+ client.key), unconditionally (reactive/vault_handlers.py:1094-1121). BY DESIGN.
  2. ovn-central uses the default charms.openstack cert path (it does NOT override get_certificate_requests; reactive/layer_openstack.py:127-137 -> charms_openstack/charm/ classes.py:284-291 -> charmhelpers/contrib/openstack/cert_utils.py). That path sets cn = get_hostname(local_address(unit_get_fallback='private-address')) (cert_utils.py add_hostname_cn).
  3. private-address = eth1 10.12.12.122 (metal-internal). MEASURED unit-get private-address. get_hostname("10.12.12.122") returns None -- gethostbyaddr raises herror; the metal-internal plane 10.12.12.0/22 has NO PTR records. Contrast MEASURED: eth0 10.12.8.185 (metal-admin) reverse-resolves to juju-237a74-0-lxd-15.maas.
  4. => published databag = certificate_name+sans:["10.12.12.122"], no common_name => vault's all_requests skips it => client-cert-only => certificates.server.certs.available never sets => /etc/ovn/{cert_host,key_host,ovn-central.crt} never written (MEASURED absent) => raft NB/SB cluster can't SSL-handshake on 6643/6644 (cluster/status = "joining cluster, Cluster ID: not yet known") => the pssl: client listeners 6641/6642 open only on a cluster leader (configure_ovn_listener, if status.is_cluster_leader) => nothing listens on 6641/6642.
  5. Sibling comparison that proves the mechanism generic (MEASURED): mysql-innodb-cluster publishes the IDENTICAL CN-less shape, gets the IDENTICAL client-cert-only response, and is healthy -- it falls back to MySQL's own auto-generated self-signed cert (issuer CN=MySQL_Server_8.0.46_Auto_Generated_CA_Certificate). ovn-chassis WORKS because it OVERRIDES the CN with the OVS-DB hostname (vr1-dc0-compute-01.maas), never DNS. ovn-central has neither a fallback nor an override.

What the prior diagnosis got wrong

  • "LP #2044324 CONFIRMED root cause" -> NO MATCH (MEASURED against the bug's content). The bug blocks on "Ports which should be open, but are not: 6641, 6642" (a post-cert port check) with NO binding at all and "inconsistently"; ours is "awaiting server certificate data", binding PRESENT, request PUBLISHED, DETERMINISTIC. The bug is untriaged 2.5y, zero comments, lxd-cloud. No upstream fix exists (github.com/canonical/charm-ovn-central; opendev mirror is archived), and no channel/revision changes cert-request construction (24.03/stable moved 311->332, nothing relevant; no 24.09 track). A channel bump will NOT fix this.
  • "vault cannot sign because common_name is missing." -> vault is never ASKED to sign; no server request object is created for a CN-less databag. mysql demonstrates identical treatment + health.
  • The "Skipping request for certificate for ip in , no local address found" log line the prior session leaned on does NOT causally drive this, and per the reviewer's grep does not even appear as that exact string in rev 311 -- the prior session likely quoted a different charm version's log.

CORRECTION to an over-claim MADE DURING this re-eval (owned)

An interim framing here said "the default binding is NOT the lever, PROVEN by: broken at deploy under metal-admin default + the rebind changed nothing." That "proven" is WITHDRAWN. Source shows the CN derives from private-address, which FOLLOWS the default binding. So the D-052 amendment (default '' metal-admin -> metal-internal, commit 53e046e) is ON the causal path: it moves private-address from PTR-bearing eth0 to PTR-less eth1. Whether ovn-central was already CN-broken at deploy (default still metal-admin) is UNKNOWN and not reconstructable (debug-log rotated; juju keeps no historical relation data). The corrected root cause rests on the CURRENT measured state, which is solid; the deploy-time story is not needed for it.

Remedies (ranked; all are MUTATIONS -> operator-gated, NOT executed)

PRIMARY -- add reverse-DNS for the metal-internal ovn-central addresses, then re-fire.

  • vault's PKI role signs with allow_any_name=True/enforce_hostnames=False (MEASURED by the vault-source agent, vault_pki.py), and OVN peers connect by IP (already in the SAN), so the CN CONTENT is near-irrelevant -- it only needs to be NON-EMPTY.
  • Add PTRs in MAAS for 10.12.12.122/.108/.104 (the three ovn-central units). FEASIBILITY (MEASURED): the reverse zone 12.12.10.in-addr.arpa IS served by MAAS (has SOA), but has NO PTRs for these juju-assigned addresses (MAAS did not allocate them), so this is "register records", not "already managed". CONFIRM FIRST (operator, read-only): maas <profile> subnets read -> rdns_mode/ managed for the 10.12.12.0/22 row.
  • Re-fire: the request handler is gated when_not is-update-status-hook, so DNS change alone won't re-run it -- a no-op juju config touch or a certificates relation cycle re-publishes; request_server_cert overwrites the empty CN.
  • APPLY TO ALL THREE UNITS. The raft cluster + 6641/6642 need all three to hold certs at once.
  • ACCEPTANCE CRITERIA (do NOT use "unit goes active" -- it is a FALSE GREEN; workload status gates only on the cert flag, not cluster health): (1) ovn-central_N.server.cert appears in the databag; (2) /etc/ovn/{cert_host,key_host,ovn-central.crt} written on each unit; (3) after all three, ovs-appctl cluster/status OVN_Northbound shows a real Cluster ID + a leader, and 6641/6642 actually listen.

PROOF-FIRST DIAGNOSTIC (recommended before the DNS change; low-risk, one unit). Add an /etc/hosts line on ovn-central/0 mapping 10.12.12.122 -> a .maas name (nsswitch = files dns), poke the certificates relation, and confirm criteria (1)+(2) appear. This proves the CN is the only issuance blocker WITHOUT touching MAAS DNS. Caveats (MEASURED/reviewer): juju's machine agent rewrites /etc/hosts, so it is EPHEMERAL -- read the result immediately after the poke; and it can only establish (1)+(2), never cluster health.

ALTERNATIVE, D-NNN-shaped (NOT recommended without a ruling) -- revert the default binding to metal-admin. Since eth0/metal-admin already has a working PTR, moving private-address back there would yield a resolvable CN with NO DNS change. BUT it reverses the standing metal-internal-binding decision (commits 580fc06/53e046e) and has a LARGE blast radius: it moves ovsdb-peer bound-address, the SAN set, the firewall cluster_remote_addrs, and the raft addresses already baked into the joining DBs (Address: ssl:10.12.12.122:6643). Flag for the operator as a decision, not a fix.

RULED OUT (MEASURED): charm-config CN lever -- rev 311 config.yaml has no os-*-network/ os-*-hostname/ssl_* option, metadata.yaml declares no internal/admin/public extra-bindings. Accept-degraded -- total OVN control-plane outage; no non-TLS or manual-cert escape in rev 311 (actions.yaml has no cert action).

WHY NO PTR FOR metal-internal (operator asked; MEASURED from the unit resolver)

MAAS DID register the metal-internal addresses in FORWARD DNS but never generated their REVERSE (PTR) records. Measured getent hosts on ovn-central/0 (resolver 127.0.0.53 -> maas):

  • metal-admin 10.12.8.185 : forward juju-237a74-0-lxd-15.maas AND reverse PTR -> BOTH present.
  • metal-internal 10.12.12.122: forward eth1.juju-237a74-0-lxd-15.maas present; reverse PTR ABSENT.
  • metal-internal 10.12.12.114 (keystone): forward eth1.juju-237a74-0-lxd-7.maas; reverse ABSENT. So MAAS knows these addresses (forward A records exist under eth1.<node>.maas); it simply does not publish PTRs for the metal-internal plane. Reverse-DNS GENERATION (MAAS subnet rdns_mode) is effectively enabled for the metal-admin subnet (10.12.8.0/22, the PXE/provisioning plane MAAS fully manages) and NOT for metal-internal (10.12.12.0/22, a juju-consumed L3 segment). This is consistent with the deployment's deliberate metal-admin-centric DNS posture (D-131 scopes the node-DNS forwarder to metal-admin only). It is default/plane-scoping behavior, not a defect -- it only became load-bearing because ovn-central's default cert path derives its CN via a REVERSE lookup of its metal-internal address, which the API charms' cert path does not do the same way. CONSEQUENCE FOR THE REMEDY (refines Remedy 1): because the FORWARD records already exist, the cleanest fix is to ENABLE reverse-DNS on the metal-internal subnet in MAAS (rdns_mode), which makes MAAS auto-generate PTRs for the addresses it already knows -- one subnet-config toggle + DNS reload, rather than hand-creating per-address PTRs. Confirm the current rdns_mode/managed for 10.12.12.0/22 via maas <profile> subnets read first (operator; the MAAS API key is behind the secrets guard so this session did not read it). NOTE the reverse zone itself is already MAAS-served (SOA present for 12.12.10.in-addr.arpa) but unpopulated -- consistent with rdns generation being off for that subnet.

Residual UNKNOWNs (before executing)

  • rdns_mode/managed for 10.12.12.0/22 in MAAS (needs the MAAS API; not read this session -- secrets rule kept the reviewer off the API key). Decides whether PTRs can simply be added.
  • End-to-end confirmation that a non-empty CN yields a usable server cert AND a formed cluster -- the proof-first diagnostic + all-three rollout establishes this.

SEC observation (file as SEC-033)

juju show-unit <any tls-certificates consumer> exposes vault's GLOBAL-CLIENT private key (client.key PEM) in the relation databag in cleartext to any juju model reader. Inherent to the tls-certificates global-client model, not newly introduced; mitigate via juju model RBAC.