From: Richard Levitte <levitte@openssl.org>
Date: Fri, 7 Jul 2017 09:11:33 +0000 (+0200)
Subject: test/run_tests.pl: Make sure to exit with a code that's understood universally
X-Git-Tag: OpenSSL_1_1_1-pre1~1089
X-Git-Url: https://git.openssl.org/gitweb/?p=openssl.git;a=commitdiff_plain;h=4549ed12ec3337313c14815438fa9aee88bf1359

test/run_tests.pl: Make sure to exit with a code that's understood universally

TAP::Parser::Aggregator::has_errors may return any number, not just 0
and 1.  With Perl on VMS, any number from 2 and on is interpreted as a
VMS status, the 3 lower bits are the encoded severity (1 = SUCCESS,
for example), so depending on what has_errors returns, a test failure
might be interpreted as a success.  Therefore, it's better to make
sure the exit code is 0 or 1, nothing else (they are special on VMS,
and mean SUCCESS or FAILURE, to match Unix conventions).

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3880)
---

diff --git a/test/run_tests.pl b/test/run_tests.pl
index a91d761900..66f620e216 100644
--- a/test/run_tests.pl
+++ b/test/run_tests.pl
@@ -85,7 +85,16 @@ my $harness = $TAP_Harness->new(\%tapargs);
 my $ret = $harness->runtests(map { abs2rel($_, rel2abs(curdir())); }
                                  sort keys %tests);
 
-exit $ret->has_errors if (ref($ret) eq "TAP::Parser::Aggregator");
+# $ret->has_errors may be any number, not just 0 or 1.  On VMS, numbers
+# from 2 and on are used as is as VMS statuses, which has severity encoded
+# in the lower 3 bits.  0 and 1, on the other hand, generate SUCCESS and
+# FAILURE, so for currect reporting on all platforms, we make sure the only
+# exit codes are 0 and 1.  Double-bang is the trick to do so.
+exit !!$ret->has_errors if (ref($ret) eq "TAP::Parser::Aggregator");
+
+# If this isn't a TAP::Parser::Aggregator, it's the pre-TAP test harness,
+# which simply dies at the end if any test failed, so we don't need to bother
+# with any exit code in that case.
 
 sub find_matching_tests {
     my ($glob) = @_;