Ensure EVP_PKEY_set1_DH detects X9.42 keys

OpenSSL supports both PKCS#3 and X9.42 DH keys. By default we use PKCS#3
keys. The function `EVP_PKEY_set1_DH` was assuming that the supplied DH
key was a PKCS#3 key. It should detect what type of key it is and assign
the correct type as appropriate.

Fixes #10592

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/10593)

diff --git a/crypto/evp/p_lib.c b/crypto/evp/p_lib.c
index 8e6682fff4322e6f2738ab9657ed2f97233d5f86..99d6063e3ed40762642c373a47618e2dd6ce0be8 100644 (file)
--- a/crypto/evp/p_lib.c
+++ b/crypto/evp/p_lib.c
@@ -569,7 +569,9 @@ EC_KEY *EVP_PKEY_get1_EC_KEY(EVP_PKEY *pkey)
 
 int EVP_PKEY_set1_DH(EVP_PKEY *pkey, DH *key)
 {
-    int ret = EVP_PKEY_assign_DH(pkey, key);
+    int type = DH_get0_q(key) == NULL ? EVP_PKEY_DH : EVP_PKEY_DHX;
+    int ret = EVP_PKEY_assign(pkey, type, key);
+
     if (ret)
         DH_up_ref(key);
     return ret;