bignum: fix boundary condition in montgomery logic