Running external test suites with OpenSSL
=========================================
-It is possible to integrate external test suites into OpenSSL's "make test".
+It is possible to integrate external test suites into OpenSSL's `make test`.
This capability is considered a developer option and does not work on all
platforms.
-
-
-The BoringSSL test suite
-========================
-
-In order to run the BoringSSL tests with OpenSSL, first checkout the BoringSSL
-source code into an appropriate directory. This can be done in two ways:
-
-1) Separately from the OpenSSL checkout using:
-
- $ git clone https://boringssl.googlesource.com/boringssl boringssl
-
- The BoringSSL tests are only confirmed to work at a specific commit in the
- BoringSSL repository. Later commits may or may not pass the test suite:
-
- $ cd boringssl
- $ git checkout 490469f850e
-
-2) Using the already configured submodule settings in OpenSSL:
-
- $ git submodule update --init
-
-Configure the OpenSSL source code to enable the external tests:
-
-$ cd ../openssl
-$ ./config enable-ssl3 enable-ssl3-method enable-weak-ssl-ciphers \
- enable-external-tests
-
-Note that using other config options than those given above may cause the tests
-to fail.
-
-Run the OpenSSL tests by providing the path to the BoringSSL test runner in the
-BORING_RUNNER_DIR environment variable:
-
-$ BORING_RUNNER_DIR=/path/to/boringssl/ssl/test/runner make test
-
-Note that the test suite may change directory while running so the path provided
-should be absolute and not relative to the current working directory.
-
-To see more detailed output you can run just the BoringSSL tests with the
-verbose option:
-
-$ VERBOSE=1 BORING_RUNNER_DIR=/path/to/boringssl/ssl/test/runner make \
- TESTS="test_external_boringssl" test
-
-
-Test failures and suppressions
-------------------------------
-
-A large number of the BoringSSL tests are known to fail. A test could fail
-because of many possible reasons. For example:
-
-- A bug in OpenSSL
-- Different interpretations of standards
-- Assumptions about the way BoringSSL works that do not apply to OpenSSL
-- The test uses APIs added to BoringSSL that are not present in OpenSSL
-- etc
-
-In order to provide a "clean" baseline run with all the tests passing a config
-file has been provided that suppresses the running of tests that are known to
-fail. These suppressions are held in the file "test/ossl_shim/ossl_config.json"
-within the OpenSSL source code.
-
-The community is encouraged to contribute patches which reduce the number of
-suppressions that are currently present.
-
-
Python PYCA/Cryptography test suite
===================================
This python test suite runs cryptographic tests with a local OpenSSL build as
the implementation.
-First checkout the PYCA/Cryptography module into ./pyca-cryptography using:
+First checkout the `PYCA/Cryptography` module into `./pyca-cryptography` using:
-$ git submodule update --init
+ $ git submodule update --init
Then configure/build OpenSSL compatible with the python module:
-$ ./config shared enable-external-tests
-$ make
+ $ ./config shared enable-external-tests
+ $ make
The tests will run in a python virtual environment which requires virtualenv
to be installed.
-$ make test VERBOSE=1 TESTS=test_external_pyca
+ $ make test VERBOSE=1 TESTS=test_external_pyca
Test failures and suppressions
------------------------------
Some tests target older (<=1.0.2) versions so will not run. Other tests target
other crypto implementations so are not relevant. Currently no tests fail.
-
krb5 test suite
===============
You will need a git checkout of krb5 at the top level:
-$ git clone https://github.com/krb5/krb5
+ $ git clone https://github.com/krb5/krb5
krb5's master has to pass this same CI, but a known-good version is
krb5-1.15.1-final if you want to be sure.
-$ cd krb5
-$ git checkout krb5-1.15.1-final
-$ cd ..
+ $ cd krb5
+ $ git checkout krb5-1.15.1-final
+ $ cd ..
OpenSSL must be built with external tests enabled:
-$ ./config enable-external-tests
-$ make
+ $ ./config enable-external-tests
+ $ make
krb5's tests will then be run as part of the rest of the suite, or can be
explicitly run (with more debugging):
-$ VERBOSE=1 make TESTS=test_external_krb5 test
+ $ VERBOSE=1 make TESTS=test_external_krb5 test
Test-failures suppressions
--------------------------
of your system. Certain tests may require more installed packages to run. No
tests are expected to fail.
-
GOST engine test suite
===============
You will need a git checkout of gost-engine at the top level:
-$ git submodule update --init
+ $ git submodule update --init
Then configure/build OpenSSL enabling external tests:
-$ ./config shared enable-external-tests
-$ make
+ $ ./config shared enable-external-tests
+ $ make
GOST engine requires CMake for the build process.
GOST engine tests will then be run as part of the rest of the suite, or can be
explicitly run (with more debugging):
-$ make test VERBOSE=1 TESTS=test_external_gost_engine
+ $ make test VERBOSE=1 TESTS=test_external_gost_engine
+
+OQSprovider test suite
+======================
+
+Much like the PYCA/Cryptography test suite, this builds and runs the OQS
+(OpenQuantumSafe -- www.openquantumsafe.org) provider tests against the
+local OpenSSL build.
+
+You will need a git checkout of oqsprovider at the top level:
+
+ $ git submodule update --init
+
+Then configure/build OpenSSL enabling external tests:
+
+ $ ./config shared enable-external-tests
+ $ make
+
+oqsprovider requires CMake for the build process.
+
+OQSprovider tests will then be run as part of the rest of the suite, or can be
+explicitly run (with more debugging):
+
+ $ make test VERBOSE=1 TESTS=test_external_oqsprovider
+
+The environment variable `OQS_SKIP_TESTS` can be set to select tests and
+algorithms to be skipped. If not set, the "rainbow" algorithm set as well as
+the (OQS-)OpenSSL1.1.1 compatibility tests will not be executed. So, for
+example to exclude the "mceliece" and "kyber" algorithms execute
+
+ OQS_SKIP_TESTS=mceliece,kyber make test TESTS=test_external_oqsprovider
+
+The names of all supported quantum-safe algorithms are available at
+<https://github.com/open-quantum-safe/openssl#supported-algorithms>
Updating test suites
====================
- Make sure the submodules are cloned locally:
- $ git submodule update --init --recursive
+ $ git submodule update --init --recursive
- Enter subdirectory and pull from the repository (use a specific branch/tag if required):
- $ cd <submodule-dir>
- $ git pull origin master
+ $ cd `<submodule-dir>`
+ $ git pull origin master
- Go to root directory, there should be a new git status:
- $ cd ../
- $ git status
- ...
- # modified: <submodule-dir> (new commits)
- ...
+ $ cd ../
+ $ git status
+ ...
+ # modified: `<submodule-dir>` (new commits)
+ ...
- Add/commit/push the update
- git add <submodule-dir>
- git commit -m "Updated <submodule> to latest commit"
- git push
-
+ $ git add `<submodule-dir>`
+ $ git commit -m `"Updated <submodule> to latest commit"`
+ $ git push
projects / openssl.git / blobdiff