Fix the RSA_SSLV23_PADDING padding type
This also fixes the public function RSA_padding_check_SSLv23.
Commit
6555a89 changed the padding check logic in RSA_padding_check_SSLv23
so that padding is rejected if the nul delimiter byte is not immediately
preceded by at least 8 bytes containing 0x03. Prior to that commit the
padding is rejected if it *is* preceded by at least 8 bytes containing 0x03.
Presumably this change was made to be consistent with what it says in
appendix E.3 of RFC 5246. Unfortunately that RFC is in error, and the
original behaviour was correct. This is fixed in later errata issued for
that RFC.
This has no impact on libssl for modern versions of OpenSSL because
there is no protocol support for SSLv2 in these versions. However
applications that call RSA_paddin_check_SSLv23 directly, or use the
RSA_SSLV23_PADDING mode may still be impacted. The effect of the original
error is that an RSA message encrypted by an SSLv2 only client will fail to
be decrypted properly by a TLS capable server, or a message encrypted by a
TLS capable client will fail to decrypt on an SSLv2 only server. Most
significantly an RSA message encrypted by a TLS capable client will be
successfully decrypted by a TLS capable server. This last case should fail
due to a rollback being detected.
Thanks to D. Katz and Joel Luellwitz (both from Trustwave) for reporting
this issue.
CVE-2021-23839
Reviewed-by: Paul Dale <pauli@openssl.org>