An incompatibility has always existed between the format used for RSA

signatures and MDC2 using EVP or RSA_sign. This has become more apparent
when the dgst utility in OpenSSL 1.0.0 and later switched to using the
EVP_DigestSign functions which call RSA_sign.

This means that the signature format OpenSSL 1.0.0 and later used with
dgst -sign and MDC2 is incompatible with previous versions.

Add detection in RSA_verify so either format works.

Note: MDC2 is disabled by default in OpenSSL and very rarely used in practice.

diff --git a/CHANGES b/CHANGES
index 47d76cb81a0215d0075615f08a0019882d681ca2..cd3500f8445d325e571e029b9b700eabc488b45d 100644 (file)
--- a/CHANGES
+++ b/CHANGES
@@ -4,6 +4,13 @@
 
  Changes between 1.0.0f and 1.0.1  [xx XXX xxxx]
 
+  *) The format used for MDC2 RSA signatures is inconsistent between EVP
+     and the RSA_sign/RSA_verify functions. This was made more apparent when
+     OpenSSL used RSA_sign/RSA_verify for some RSA signatures in particular
+     those which went through EVP_PKEY_METHOD in 1.0.0 and later. Detect 
+     the correct format in RSA_verify so both forms transparently work.
+     [Steve Henson]
+
   *) Some servers which support TLS 1.0 can choke if we initially indicate
      support for TLS 1.2 and later renegotiate using TLS 1.0 in the RSA
      encrypted premaster secret. As a workaround use the maximum pemitted

diff --git a/crypto/rsa/rsa_sign.c b/crypto/rsa/rsa_sign.c
index 2ccadb73d850c193389090c2f88ba22ba268c9b3..b6f6037ae002df06a4fb078e8f8af0a0b6dd2dfb 100644 (file)
--- a/crypto/rsa/rsa_sign.c
+++ b/crypto/rsa/rsa_sign.c
@@ -199,6 +199,22 @@ int int_rsa_verify(int dtype, const unsigned char *m,
        i=RSA_public_decrypt((int)siglen,sigbuf,s,rsa,RSA_PKCS1_PADDING);
 
        if (i <= 0) goto err;
+       /* Oddball MDC2 case: signature can be OCTET STRING.
+        * check for correct tag and length octets.
+        */
+       if (dtype == NID_mdc2 && i == 18 && s[0] == 0x04 && s[1] == 0x10)
+               {
+               if (rm)
+                       {
+                       memcpy(rm, s + 2, 16);
+                       *prm_len = 16;
+                       ret = 1;
+                       }
+               else if(memcmp(m, s + 2, 16))
+                       RSAerr(RSA_F_INT_RSA_VERIFY,RSA_R_BAD_SIGNATURE);
+               else
+                       ret = 1;
+               }
 
        /* Special case: SSL signature */
        if(dtype == NID_md5_sha1) {