If |q| >= |p| then the q value is obviously wrong as q
is supposed to be a prime divisor of p-1.
We check if p is overly large so this added test implies that
q is not large either when performing subsequent tests using that
q value.
Otherwise if it is too large these additional checks of the q value
such as the primality test can then trigger DoS by doing overly long
computations.
Fixes CVE-2023-3817
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tom Cosgrove <tom.cosgrove@arm.com>
Reviewed-by: Todd Short <todd.short@me.com>
(Merged from https://github.com/openssl/openssl/pull/21550)
(cherry picked from commit
1c16253f3c3a8d1e25918c3f404aae6a5b0893de)
#ifdef FIPS_MODULE
return DH_check_params(dh, ret);
#else
- int ok = 0, r;
+ int ok = 0, r, q_good = 0;
BN_CTX *ctx = NULL;
BIGNUM *t1 = NULL, *t2 = NULL;
int nid = DH_get_nid((DH *)dh);
goto err;
if (dh->params.q != NULL) {
+ if (BN_ucmp(dh->params.p, dh->params.q) > 0)
+ q_good = 1;
+ else
+ *ret |= DH_CHECK_INVALID_Q_VALUE;
+ }
+
+ if (q_good) {
if (BN_cmp(dh->params.g, BN_value_one()) <= 0)
*ret |= DH_NOT_SUITABLE_GENERATOR;
else if (BN_cmp(dh->params.g, dh->params.p) >= 0)