DH_check(): Do not try checking q properties if it is obviously invalid

If |q| >= |p| then the q value is obviously wrong as q is supposed to be a prime divisor of p-1. We check if p is overly large so this added test implies that q is not large either when performing subsequent tests using that q value. Otherwise if it is too large these additional checks of the q value such as the primality test can then trigger DoS by doing overly long computations. Fixes CVE-2023-3817 Reviewed-by: Paul Dale <pauli@openssl.org> Reviewed-by: Matt Caswell <matt@openssl.org> (Merged from #21551)
openssl · Jul 31, 2023 · 91ddeba · 91ddeba
1 parent eec805e
commit 91ddeba
Showing 1 changed file with 9 additions and 2 deletions.
diff --git a/crypto/dh/dh_check.c b/crypto/dh/dh_check.c
@@ -97,7 +97,7 @@ int DH_check_ex(const DH *dh)
 
 int DH_check(const DH *dh, int *ret)
 {
-    int ok = 0, r;
+    int ok = 0, r, q_good = 0;
     BN_CTX *ctx = NULL;
     BIGNUM *t1 = NULL, *t2 = NULL;
 
@@ -120,7 +120,14 @@ int DH_check(const DH *dh, int *ret)
     if (t2 == NULL)
         goto err;
 
-    if (dh->q) {
+    if (dh->q != NULL) {
+        if (BN_ucmp(dh->p, dh->q) > 0)
+            q_good = 1;
+        else
+            *ret |= DH_CHECK_INVALID_Q_VALUE;
+    }
+
+    if (q_good) {
         if (BN_cmp(dh->g, BN_value_one()) <= 0)
             *ret |= DH_NOT_SUITABLE_GENERATOR;
         else if (BN_cmp(dh->g, dh->p) >= 0)