Check ASN1_item_ndef_i2d() return value.

Check ASN1_item_ndef_i2d() return value.

Return an error instead of trying to malloc a negative number.
The other usage in this file already had a similar check, and the caller
should have put an entry on the error stack already.

Note that we only check the initial calls to obtain the encoded length,
and assume that the follow-up call to actually encode to the allocated
storage will succeed if the first one did.

Fixes: #14177
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/14308)