Fix error propagatation in BN_check_prime()

BN_check_prime() is supposed to return 0 for a composite number and -1
on error. Properly translate the return value of the internal function
ossl_bn_miller_rabin_is_prime(), where 0 means an error.

The confusion prevented BN_GENCB callbacks from aborting the primality
test or key generation routines utilizing this.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/19314)

(cherry picked from commit 0b3867634f74f6cb7b60b3a0adde396421207214)

diff --git a/crypto/bn/bn_prime.c b/crypto/bn/bn_prime.c
index 9e2f6861a519bf16a4cababe3e794a0e7cb246f0..54f7ca611fd096a45bddf8b19132e52a4827045d 100644 (file)
--- a/crypto/bn/bn_prime.c
+++ b/crypto/bn/bn_prime.c
@@ -308,9 +308,10 @@ static int bn_is_prime_int(const BIGNUM *w, int checks, BN_CTX *ctx,
         goto err;
 #endif
 
-    ret = ossl_bn_miller_rabin_is_prime(w, checks, ctx, cb, 0, &status);
-    if (!ret)
+    if (!ossl_bn_miller_rabin_is_prime(w, checks, ctx, cb, 0, &status)) {
+        ret = -1;
         goto err;
+    }
     ret = (status == BN_PRIMETEST_PROBABLY_PRIME);
 err:
 #ifndef FIPS_MODULE