DH_check(): Do not try checking q properties if it is obviously invalid
authorTomas Mraz <tomas@openssl.org>
Tue, 25 Jul 2023 13:22:48 +0000 (15:22 +0200)
committerTodd Short <todd.short@me.com>
Thu, 27 Jul 2023 13:48:22 +0000 (09:48 -0400)
If  |q| >= |p| then the q value is obviously wrong as q
is supposed to be a prime divisor of p-1.

We check if p is overly large so this added test implies that
q is not large either when performing subsequent tests using that
q value.

Otherwise if it is too large these additional checks of the q value
such as the primality test can then trigger DoS by doing overly long
computations.

Fixes CVE-2023-3817

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tom Cosgrove <tom.cosgrove@arm.com>
Reviewed-by: Todd Short <todd.short@me.com>
(Merged from https://github.com/openssl/openssl/pull/21550)

crypto/dh/dh_check.c

index 6e1ea5349aaa967b95ec2ea04e3a1cb3644e9266..58c1d1798b6201fdc307a5e13de782561e03ac82 100644 (file)
@@ -143,7 +143,7 @@ int DH_check(const DH *dh, int *ret)
 #ifdef FIPS_MODULE
     return DH_check_params(dh, ret);
 #else
-    int ok = 0, r;
+    int ok = 0, r, q_good = 0;
     BN_CTX *ctx = NULL;
     BIGNUM *t1 = NULL, *t2 = NULL;
     int nid = DH_get_nid((DH *)dh);
@@ -172,6 +172,13 @@ int DH_check(const DH *dh, int *ret)
         goto err;
 
     if (dh->params.q != NULL) {
+        if (BN_ucmp(dh->params.p, dh->params.q) > 0)
+            q_good = 1;
+        else
+            *ret |= DH_CHECK_INVALID_Q_VALUE;
+    }
+
+    if (q_good) {
         if (BN_cmp(dh->params.g, BN_value_one()) <= 0)
             *ret |= DH_NOT_SUITABLE_GENERATOR;
         else if (BN_cmp(dh->params.g, dh->params.p) >= 0)