Fix bug in X509_print_ex

Fix bug in X509_print_ex

If the user set nmflags == XN_FLAG_COMPAT and X509_NAME_print_ex(3)
failed, the error return value of 0 was misinterpreted as an indicator
of success, causing X509_print_ex(3) to ignore the error, continue
printing, and potentially return successfully even though not all
the content of the certificate was printed.

The X509_NAME_print_ex(3) manual page explains that this function
indicates failure by returning 0 if nmflags == XN_FLAG_COMPAT
and by returning -1 if nmflags != XN_FLAG_COMPAT.

Note that just checking for <= 0 in all cases would not be correct
either because X509_NAME_print_ex(3) returns 0 to indicate that it
successfully printed zero bytes in some cases, for example when all
three of the following conditions hold:
1. nmflags != XN_FLAG_COMPAT
2. indent == 0 (which X509_print_ex(3) does use in some cases)
3. the name object is NULL or empty

Thanks to Ingo Schwarze <schwarze@openbsd.org> for finding the bug,
and Joel Sing <jsing@openbsd.org> for contributing an idea for the
fix.

Reviewed-by: Ben Kaduk <kaduk@mit.edu>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16009)