Fix DH_check() excessive time with over sized modulus

The DH_check() function checks numerous aspects of the key or parameters
that have been supplied. Some of those checks use the supplied modulus
value even if it is excessively large.

There is already a maximum DH modulus size (10,000 bits) over which
OpenSSL will not generate or derive keys. DH_check() will however still
perform various tests for validity on such a large modulus. We introduce a
new maximum (32,768) over which DH_check() will just fail.

An application that calls DH_check() and supplies a key or parameters
obtained from an untrusted source could be vulnerable to a Denial of
Service attack.

The function DH_check() is itself called by a number of other OpenSSL
functions. An application calling any of those other functions may
similarly be affected. The other functions affected by this are
DH_check_ex() and EVP_PKEY_param_check().

CVE-2023-3446

Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tom Cosgrove <tom.cosgrove@arm.com>
Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/21451)

diff --git a/crypto/dh/dh_check.c b/crypto/dh/dh_check.c
index 0b8a17c67582300d58eea2f326e8d72c660fe0c8..98014593b6debe384163ef9af810dcf4b9b71a5f 100644 (file)
--- a/crypto/dh/dh_check.c
+++ b/crypto/dh/dh_check.c
@@ -152,6 +152,12 @@ int DH_check(const DH *dh, int *ret)
     if (nid != NID_undef)
         return 1;
 
+    /* Don't do any checks at all with an excessively large modulus */
+    if (BN_num_bits(dh->params.p) > OPENSSL_DH_CHECK_MAX_MODULUS_BITS) {
+        ERR_raise(ERR_LIB_DH, DH_R_MODULUS_TOO_LARGE);
+        return 0;
+    }
+
     if (!DH_check_params(dh, ret))
         return 0;
 

diff --git a/include/openssl/dh.h b/include/openssl/dh.h
index 2c6219e7c2a6bc1e2ab8b822211efe9325a2287f..da6e7b06c879a8d62ff4bc2a39a9686b37403a69 100644 (file)
--- a/include/openssl/dh.h
+++ b/include/openssl/dh.h
@@ -92,7 +92,11 @@ int EVP_PKEY_CTX_get0_dh_kdf_ukm(EVP_PKEY_CTX *ctx, unsigned char **ukm);
 #  include <openssl/dherr.h>
 
 #  ifndef OPENSSL_DH_MAX_MODULUS_BITS
-#   define OPENSSL_DH_MAX_MODULUS_BITS    10000
+#   define OPENSSL_DH_MAX_MODULUS_BITS        10000
+#  endif
+
+#  ifndef OPENSSL_DH_CHECK_MAX_MODULUS_BITS
+#   define OPENSSL_DH_CHECK_MAX_MODULUS_BITS  32768
 #  endif
 
 #  define OPENSSL_DH_FIPS_MIN_MODULUS_BITS 1024