cms: avoid intermittent test failure
authorJames Muir <james@openssl.org>
Thu, 14 Dec 2023 19:14:37 +0000 (14:14 -0500)
committerTomas Mraz <tomas@openssl.org>
Mon, 18 Dec 2023 11:18:24 +0000 (12:18 +0100)
commitffed597882baf2f07274e7eaa8f3c4fa9fa74ac1
tree2edf82c24f0bcf7522e4a99c117111c94dcfa788
parente1002c84725a64b6a097f3155dc6851b57f7ba8e
cms: avoid intermittent test failure

If you decrypt a random input using RSAES-PKCS-v1_5, then there is a
non-negligible chance that the result will look like a valid plaintext
(that is why RSAES-PKCS-v1_5 shouldn't be used anymore).  This was the
cause of an intermittent failure in a test that did a cms-encrypt
operation targetting multiple recipients.

The failure happened during key-only decrypt.  The recipient decrypts
every RSA ciphertext -- only one is supposed to decrypt successfully,
which would reveal the right content-key.  Occassionally, more than
one decrypted successfully.

Update the test by specifying the recipient cert in the decrypt op
(this avoids looping over all RSA ciphertexts).

Add a new test to get coverage for key-only decrypt, but use RSA-OAEP
during the encrypt op.

Fixes https://github.com/openssl/project/issues/380

Testing:

  $ make TESTS='test_cms' test

Reviewed-by: Neil Horman <nhorman@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/23055)
test/recipes/80-test_cms.t
test/smime-certs/mksmime-certs.sh
test/smime-certs/smrsa3-cert.pem [new file with mode: 0644]
test/smime-certs/smrsa3-key.pem [new file with mode: 0644]