From: Matt Caswell <matt@openssl.org>
Date: Sat, 15 Oct 2016 10:09:20 +0000 (+0100)
Subject: Add documentation on the BoringSSL test suite integration
X-Git-Tag: OpenSSL_1_1_1-pre1~3243
X-Git-Url: https://git.openssl.org/gitweb/?p=openssl.git;a=commitdiff_plain;h=5227337a7c340eeeebaced1230e8dfaa7a8ec7db;ds=sidebyside

Add documentation on the BoringSSL test suite integration

Added the file README.external which describes how to build and run OpenSSL
to use the BoringSSL test suite. Also updated INSTALL to point to it.

Reviewed-by: Richard Levitte <levitte@openssl.org>
---

diff --git a/INSTALL b/INSTALL
index e31431bfbe..36cb090090 100644
--- a/INSTALL
+++ b/INSTALL
@@ -314,7 +314,8 @@
                    Enable building of integration with external test suites.
                    This is a developer option and may not work on all platforms.
                    The only supported external test suite at the current time is
-                   the BoringSSL test suite.
+                   the BoringSSL test suite. See the file test/README.external
+                   for further details.
 
   no-filenames
                    Don't compile in filename and line number information (e.g.
diff --git a/test/README.external b/test/README.external
new file mode 100644
index 0000000000..68d84813bb
--- /dev/null
+++ b/test/README.external
@@ -0,0 +1,65 @@
+Running the BoringSSL test suite with OpenSSL
+=============================================
+
+It is possible to integrate external test suites into OpenSSL's "make test". At
+the current time the only supported external suite is the one used by
+BoringSSL.
+
+This capability is considered a developer option and may not work on all
+platforms.
+
+In order to run the BoringSSL tests with OpenSSL, first checkout the BoringSSL
+source code into an appropriate directory:
+
+$ git clone boringssl https://boringssl.googlesource.com/boringssl
+
+The BoringSSL tests are only confirmed to work at a specific commit in the
+BoringSSL repository. Later commits may or may not pass the test suite:
+
+$ cd boringssl
+$ git checkout 490469f850e
+
+From the OpenSSL source code configure to use the external tests:
+
+$ cd ../openssl
+$ ./config enable-ssl3 enable-ssl3-method enable-weak-ssl-ciphers \
+  enable-external-tests
+
+Note that using other config option than those given above may cause the tests
+to fail.
+
+Run the OpenSSL tests by providing the path to the BoringSSL test runner in the
+BORING_RUNNER_DIR environment variable:
+
+$ BORING_RUNNER_DIR=/path/to/boringssl/ssl/test/runner make test
+
+Note that the test suite may change directory while running so the path provided
+should be absolute and not relative to the current working directory.
+
+To see more detailed output you can run just the BoringSSL tests with the
+verbose option:
+
+$ VERBOSE=1 BORING_RUNNER_DIR=/path/to/boringssl/ssl/test/runner make \
+  TESTS="test_external" test
+
+
+Test failures and suppressions
+==============================
+
+A large number of the BoringSSL tests are known to fail. A test could fail
+because of many possible reasons. For example:
+
+- A bug in OpenSSL
+- Different interpretations of standards
+- Assumptions about the way BoringSSL works that do not apply to OpenSSL
+- The test uses APIs added to BoringSSL that are not present in OpenSSL
+- etc
+
+In order to provide a "clean" baseline run with all the tests passing a config
+file has been provided that suppresses the running of tests that are known to
+fail. These suppressions are held in the file "test/ossl_shim/ossl_config.json"
+within the OpenSSL source code.
+
+The community is encouraged to contribute patches which reduce the number of
+suppressions that are currently present.
+