BN_to_montgomery expects its inputs to be in the interval 0 .. modulus-1,

so we have to reduce the random numbers used in test_mont.

Before this change, test_mont failed in [debug-]solaris-sparcv9-gcc
configurations ("Montgomery multiplication test failed!" because
the multiplication result obtained with Montgomery multiplication
differed from the result obtained by BN_mod_mul).
Substituing the old version of bn_gcd.c (BN_mod_inverse) did not avoid
the problem.

The strange thing is that it I did not observe any problems
when using debug-solaris-sparcv8-gcc and solaris-sparcv9-cc,
as well as when compiling OpenSSL 0.9.6 in the solaric-sparcv9-gcc
configuration on the same system.

diff --git a/crypto/bn/bntest.c b/crypto/bn/bntest.c
index 86d384428c6db4abab6e813a88a0d69d9ab60ddc..9e478dfe24b8ba1448ee86346edc3a44280ee12c 100644 (file)
--- a/crypto/bn/bntest.c
+++ b/crypto/bn/bntest.c
@@ -629,6 +629,9 @@ int test_mont(BIO *bp, BN_CTX *ctx)
                BN_rand(&n,bits,0,1);
                BN_MONT_CTX_set(mont,&n,ctx);
 
+               BN_nnmod(&a,&a,&n,ctx);
+               BN_nnmod(&b,&b,&n,ctx);
+
                BN_to_montgomery(&A,&a,mont,ctx);
                BN_to_montgomery(&B,&b,mont,ctx);