check i2d_ASN1_TYPE return value

add a length check to the return value of function i2d_ASN1_TYPE. Return an error instead of trying to malloc a negative number.

CLA: trivial

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/14828)

diff --git a/crypto/asn1/a_strex.c b/crypto/asn1/a_strex.c
index b9b6f1c8715b9e8065633488e5d223d3a4688d70..b31761aae6f52c1f9e75acea646b24a0f71942bd 100644 (file)
--- a/crypto/asn1/a_strex.c
+++ b/crypto/asn1/a_strex.c
@@ -280,6 +280,8 @@ static int do_dump(unsigned long lflags, char_io *io_ch, void *arg,
     t.type = str->type;
     t.value.ptr = (char *)str;
     der_len = i2d_ASN1_TYPE(&t, NULL);
+    if (der_len <= 0)
+        return -1;
     if ((der_buf = OPENSSL_malloc(der_len)) == NULL) {
         ERR_raise(ERR_LIB_ASN1, ERR_R_MALLOC_FAILURE);
         return -1;