DH_check(): Do not try checking q properties if it is obviously invalid
authorTomas Mraz <tomas@openssl.org>
Tue, 25 Jul 2023 13:22:48 +0000 (15:22 +0200)
committerTodd Short <todd.short@me.com>
Thu, 27 Jul 2023 13:57:58 +0000 (09:57 -0400)
commit9002fd07327a91f35ba6c1307e71fa6fd4409b7f
treee3dc8c7c1cb91d9cf56a67b70073f2a8945fd9ad
parente648db50d9a63f71cab5cb78424c2932d019a744
DH_check(): Do not try checking q properties if it is obviously invalid

If  |q| >= |p| then the q value is obviously wrong as q
is supposed to be a prime divisor of p-1.

We check if p is overly large so this added test implies that
q is not large either when performing subsequent tests using that
q value.

Otherwise if it is too large these additional checks of the q value
such as the primality test can then trigger DoS by doing overly long
computations.

Fixes CVE-2023-3817

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tom Cosgrove <tom.cosgrove@arm.com>
Reviewed-by: Todd Short <todd.short@me.com>
(Merged from https://github.com/openssl/openssl/pull/21550)

(cherry picked from commit 1c16253f3c3a8d1e25918c3f404aae6a5b0893de)
(cherry picked from commit 6a1eb62c29db6cb5eec707f9338aee00f44e26f5)
crypto/dh/dh_check.c