Correct the check of RSA_FLAG_SIGN_VER

The wrong flags were being tested. It is the rsa->meth flags not the rsa
flags that should be tested.

wpa_supplicant has a bit of code that
1. Allocates and defines a RSA_METHOD structure.
2. calls RSA_new();
3. calls RSA_set_method().

In current versions of that code the rsa_sign and rsa_verify members of
the RSA_METHOD structure are not defined, thus making it compatible
with the really old versions of OpenSSL.

But should one change it use the rsa_sign method one must set the
RSA_FLAG_SIGN_VER bit of the RSA_METHOD structure to indicate that
one or both of those new methods are required.  In doing so, OpenSSL
will not call the new methods, not without this change.

CLA: trivial

Change-Id: I6e65a80f21399f25e966466ff676e3b21f85f360

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5971)

diff --git a/crypto/rsa/rsa_sign.c b/crypto/rsa/rsa_sign.c
index 82ca8324dfbc11c02736e0abffe5b7649b313b36..b7fff43530f2b488aa699842ae9450f791d384a1 100644 (file)
--- a/crypto/rsa/rsa_sign.c
+++ b/crypto/rsa/rsa_sign.c
@@ -84,7 +84,7 @@ int RSA_sign(int type, const unsigned char *m, unsigned int m_len,
         return 0;
     }
 #endif
-    if ((rsa->flags & RSA_FLAG_SIGN_VER) && rsa->meth->rsa_sign) {
+    if ((rsa->meth->flags & RSA_FLAG_SIGN_VER) && rsa->meth->rsa_sign) {
         return rsa->meth->rsa_sign(type, m, m_len, sigret, siglen, rsa);
     }
     /* Special case: SSL signature, just check the length */
@@ -293,7 +293,7 @@ int RSA_verify(int dtype, const unsigned char *m, unsigned int m_len,
                const unsigned char *sigbuf, unsigned int siglen, RSA *rsa)
 {
 
-    if ((rsa->flags & RSA_FLAG_SIGN_VER) && rsa->meth->rsa_verify) {
+    if ((rsa->meth->flags & RSA_FLAG_SIGN_VER) && rsa->meth->rsa_verify) {
         return rsa->meth->rsa_verify(dtype, m, m_len, sigbuf, siglen, rsa);
     }