Check ASN1_item_ndef_i2d() return value.

Return an error instead of trying to malloc a negative number.
The other usage in this file already had a similar check, and the caller
should have put an entry on the error stack already.

Note that we only check the initial calls to obtain the encoded length,
and assume that the follow-up call to actually encode to the allocated
storage will succeed if the first one did.

Fixes: #14177
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/14308)

diff --git a/crypto/asn1/bio_ndef.c b/crypto/asn1/bio_ndef.c
index 87c22e897c7042e0d1191a20abb1e56c51efef59..f1ad8d3e708904a04082691a8ca018f908467318 100644 (file)
--- a/crypto/asn1/bio_ndef.c
+++ b/crypto/asn1/bio_ndef.c
@@ -114,6 +114,8 @@ static int ndef_prefix(BIO *b, unsigned char **pbuf, int *plen, void *parg)
     ndef_aux = *(NDEF_SUPPORT **)parg;
 
     derlen = ASN1_item_ndef_i2d(ndef_aux->val, NULL, ndef_aux->it);
+    if (derlen < 0)
+        return 0;
     if ((p = OPENSSL_malloc(derlen)) == NULL) {
         ERR_raise(ERR_LIB_ASN1, ERR_R_MALLOC_FAILURE);
         return 0;