ASN1: Fix i2d_provided() return value

i2d_provided() - which is the internal provider data function for
i2d_KeyParams(), i2d_PrivateKey(), i2d_PublicKey() - didn't treat the
returned length from OSSL_ENCODER_to_data() quite as well as it should
have.  A simple added flag that records the state of |*pp| before
calling OSSL_ENCODER_to_data() fixes the problem.

Fixes #14655

Reviewed-by: Ben Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/15277)

diff --git a/crypto/asn1/i2d_evp.c b/crypto/asn1/i2d_evp.c
index 2a101a6fa31a609fd3950d1157f9bc9c2f091ccb..f03dcb26669d13752f3489ed63d7e24cfd72c73b 100644 (file)
--- a/crypto/asn1/i2d_evp.c
+++ b/crypto/asn1/i2d_evp.c
@@ -48,6 +48,7 @@ static int i2d_provided(const EVP_PKEY *a, int selection,
          * down, when pp != NULL.
          */
         size_t len = INT_MAX;
+        int pp_was_NULL = (pp == NULL || *pp == NULL);
 
         ctx = OSSL_ENCODER_CTX_new_for_pkey(a, selection,
                                             output_info->output_type,
@@ -56,7 +57,7 @@ static int i2d_provided(const EVP_PKEY *a, int selection,
         if (ctx == NULL)
             return -1;
         if (OSSL_ENCODER_to_data(ctx, pp, &len)) {
-            if (pp == NULL)
+            if (pp_was_NULL)
                 ret = (int)len;
             else
                 ret = INT_MAX - (int)len;