DH_check(): Do not try checking q properties if it is obviously invalid
If |q| >= |p| then the q value is obviously wrong as q
is supposed to be a prime divisor of p-1.
We check if p is overly large so this added test implies that
q is not large either when performing subsequent tests using that
q value.
Otherwise if it is too large these additional checks of the q value
such as the primality test can then trigger DoS by doing overly long
computations.
Fixes CVE-2023-3817
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tom Cosgrove <tom.cosgrove@arm.com>
Reviewed-by: Todd Short <todd.short@me.com>
(Merged from https://github.com/openssl/openssl/pull/21550)
(cherry picked from commit
1c16253f3c3a8d1e25918c3f404aae6a5b0893de)
(cherry picked from commit
6a1eb62c29db6cb5eec707f9338aee00f44e26f5)