RSA keygen fixes

Fixes #18321

Increase the iteration factor used when 'Computing a Probable Prime Factor Based on Auxiliary Primes' from 5 to 20.
This matches the algorithm update made in FIPS 186-5.

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/18429)

diff --git a/crypto/bn/bn_rsa_fips186_4.c b/crypto/bn/bn_rsa_fips186_4.c
index 8faaaefe991f91d7c2a744e75873e7c7f701ef79..a451576bf9e31d709cb4c83d01ace24ffb92c89b 100644 (file)
--- a/crypto/bn/bn_rsa_fips186_4.c
+++ b/crypto/bn/bn_rsa_fips186_4.c
@@ -303,7 +303,14 @@ int ossl_bn_rsa_fips186_4_derive_prime(BIGNUM *Y, BIGNUM *X, const BIGNUM *Xin,
     if (BN_is_negative(R) && !BN_add(R, R, r1r2x2))
         goto err;
 
-    imax = 5 * bits; /* max = 5/2 * nbits */
+    /*
+     * In FIPS 186-4 imax was set to 5 * nlen/2.
+     * Analysis by Allen Roginsky (See https://csrc.nist.gov/CSRC/media/Publications/fips/186/4/final/documents/comments-received-fips186-4-december-2015.pdf
+     * page 68) indicates this has a 1 in 2 million chance of failure.
+     * The number has been updated to 20 * nlen/2 as used in
+     * FIPS186-5 Appendix B.9 Step 9.
+     */
+    imax = 20 * bits; /* max = 20/2 * nbits */
     for (;;) {
         if (Xin == NULL) {
             /*