bignum: fix boundary condition in montgomery logic

It's not clear whether this inconsistency could lead to an actual
computation error, but it involved a BIGNUM being passed around the
montgomery logic in an inconsistent state. This was found using flags
-DBN_DEBUG -DBN_DEBUG_RAND, and working backwards from this assertion
in 'ectest';

ectest: bn_mul.c:960: BN_mul: Assertion `(_bnum2->top == 0) ||
(_bnum2->d[_bnum2->top - 1] != 0)' failed

Signed-off-by: Geoff Thorpe <geoff@openssl.org>
(cherry picked from commit a52926189155d906d8c11ff97cbc1e5191d202cd)

diff --git a/crypto/bn/bn_exp.c b/crypto/bn/bn_exp.c
index 5adb441870235a3d72bed617aa9965c288b09840..3fa8cda8428f20a87b1de55b9ee2f443d7520c52 100644 (file)
--- a/crypto/bn/bn_exp.c
+++ b/crypto/bn/bn_exp.c
@@ -493,6 +493,9 @@ int BN_mod_exp_mont(BIGNUM *rr, const BIGNUM *a, const BIGNUM *p,
                r->d[0] = (0-m->d[0])&BN_MASK2;
                for(i=1;i<j;i++) r->d[i] = (~m->d[i])&BN_MASK2;
                r->top = j;
+               /* Upper words will be zero if the corresponding words of 'm'
+                * were 0xfff[...], so decrement r->top accordingly. */
+               bn_correct_top(r);
                }
        else
 #endif