Move malloc fail checks closer to malloc

Move memory allocation failure checks closer to the site of the malloc in
dgst app. Only a problem if the debug flag is set...but still should be
fixed.

Reviewed-by: Tim Hudson <tjh@openssl.org>
(cherry picked from commit be1477adc97e76f4b83ed8075589f529069bd5d1)