Make the fuzzers more reproducible

We want to be in the same global state each time we come in
FuzzerTestOneInput(). There are various reasons why we might not be that
include:
- Initialization that happens on first use. This is mostly the
  RUN_ONCE() things, or loading of error strings.
- Results that get cached. For instance a stack that is sorted, RSA
  blinding that has been set up, ...

So I try to trigger as much as possible in FuzzerInitialize(), and for
things I didn't find out how to trigger this it needs to happen in
FuzzerTestOneInput().

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2023

Move libfuzzer sanitizer options to README

This is something you might want to change depending on the version to
use, there is no point in us fixing this to something.

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2023

CMS fuzzer: also use id2

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2023

Make the random number generator predictable when fuzzing.

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2023

Use 8bit-counters when using libfuzzer

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2023

bndiv fuzzer: move new and free calls to the init and cleanup function.

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2023

bignum fuzzer: move new and free calls to the init and cleanup function.

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2023

asn1parse: create the out bio during init, free it during cleanup

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2023

Add a FuzzerClean() function

This allows to free everything we allocated, so we can detect memory
leaks.

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2023

FuzzerInitialize always exists

There was a time it could be NULL, but that was changed to always have it.

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2023

Fix formatting of fuzzers

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2023

Fix a typo in bio_read_intern

Reviewed-by: Stephen Henson <steve@openssl.org>

Make refdata in tls13encryptest static

Reviewed-by: Rich Salz <rsalz@openssl.org>

Convert tls13encryptiontest so that we pass around a pointer not an index

We also split the long string literals into 3 to avoid problems where we
go over the 509 character limit.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fix some style issues in the TLSv1.3 nonce construction code

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fix a double free in tls13encryptiontest

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fix a travis compilation error

Reviewed-by: Rich Salz <rsalz@openssl.org>

Add a test for TLSv1.3 encryption using the new nonce construction

Reviewed-by: Rich Salz <rsalz@openssl.org>

Use the TLSv1.3 nonce construction

This updates the record layer to use the TLSv1.3 style nonce construciton.
It also updates TLSProxy and ossltest to be able to recognise the new
layout.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fix mac-then-encrypt test with enable-tls1_3

Commit b3618f44 added a test for mac-then-encrypt. However the test fails
when running with "enable-tls1_3". The problem is that the test creates a
connection, which ends up being TLSv1.3. However it also restricts the
ciphers to a single mac-then-encrypt ciphersuite that is not TLSv1.3
compatible so the connection aborts and the test fails. Mac-then-encrypt
is not relevant to TLSv1.3, so the test should disable that protocol
version.

Reviewed-by: Emilia Käsper <emilia@openssl.org>

Ensure we are in accept state in DTLSv1_listen

Calling SSL_set_accept_state() after DTLSv1_listen() clears the state, so
SSL_accept() no longer works. In 1.0.2 calling DTLSv1_listen() would set
the accept state automatically. We should still do that.

Fixes #1989

Reviewed-by: Andy Polyakov <appro@openssl.org>

Make bntest be (mostly) file-based.

Test suite used from boring, written by David Benjamin.
Test driver converted from C++ to C.
Added a Perl program to check the testsuite file.
Extensive review feedback incorporated (thanks folks).

Reviewed-by: Emilia Käsper <emilia@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>

Test mac-then-encrypt

Verify that the encrypt-then-mac negotiation is handled
correctly. Additionally, when compiled with no-asm, this test ensures
coverage for the constant-time MAC copying code in
ssl3_cbc_copy_mac. The proxy-based CBC padding test covers that as
well but it's nevertheless better to have an explicit handshake test
for mac-then-encrypt.

Reviewed-by: Andy Polyakov <appro@openssl.org>

add CMS SHA1 signing test

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fix ctrl operation for SHA1/MD5SHA1.

This makes S/MIME and CMS signing in MIME format for SHA1 work again.

Reviewed-by: Rich Salz <rsalz@openssl.org>

INSTALL: clarify 386 and no-sse2 options.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>

modes/ctr128.c: fix false carry in counter increment procedure.

GH issue #1916 affects only big-endian platforms. TLS is not affected,
because TLS fragment is never big enough.

Reviewed-by: Matt Caswell <matt@openssl.org>

test/evptests.txt: add regression test for false carry in ctr128.c.

GH issue #1916 affects only big-endian platforms. TLS is not affected,
because TLS fragment is never big enough.

Reviewed-by: Matt Caswell <matt@openssl.org>

bn/asm/ppc-mont.pl: signal no-op in 32-bit bit build.

The bug was introduced in 80d27cdb84985c697f8fabb7649abf1f54714d13,
one too many instructions was removed. It went unnoticed, because
new subroutine introduced in previous commit is called in real-life
RSA/DSA/DH cases, while original code is called only in rare tests.
The bug was caught in test_fuzz.

Reviewed-by: Rich Salz <rsalz@openssl.org>

coveralls: Use gcov-5 since we build it using gcc-5

Reviewed-by: Emilia Käsper <emilia@openssl.org>
GH: #2003

Fix a bogus uninit var warning

Reviewed-by: Richard Levitte <levitte@openssl.org>

Fix a warning about an uninit var

Reviewed-by: Richard Levitte <levitte@openssl.org>

Run BoringSSL tests on Travis

Reviewed-by: Richard Levitte <levitte@openssl.org>

Fix a missing function prototype in AFALG engine

Reviewed-by: Richard Levitte <levitte@openssl.org>

Fix missing NULL checks in key_share processing

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fix missing NULL checks in CKE processing

Reviewed-by: Rich Salz <rsalz@openssl.org>

Use ClientHello.legacy_version for the RSA pre-master no matter what

Don't use what is in supported_versions for the RSA pre-master

Reviewed-by: Emilia Käsper <emilia@openssl.org>

Fix some defines in ossl_shim

ossl_shim had some TLS1.3 defines that are now in ssl.h so need to be
removed.

Reviewed-by: Emilia Käsper <emilia@openssl.org>

Fix a double ;; causing a travis failure

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fix an uninit variable usage

Reviewed-by: Rich Salz <rsalz@openssl.org>

Update tls13secretstest to use the new simpler test framework

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fix some style issues with TLSv1.3 state machine PR

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fix some style issues identified during review

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fix SSL_IS_TLS13(s)

The SSL_IS_TLS13() macro wasn't quite right. It would come back with true
in the case where we haven't yet negotiated TLSv1.3, but it could be
negotiated.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Extend test_tls13messages

Add various different handshake types that are possible.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Add an s_server capability to read an OCSP Response from a file

Current s_server can only get an OCSP Response from an OCSP responder. This
provides the capability to instead get the OCSP Response from a DER encoded
file.

This should make testing of OCSP easier.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Add a test for the TLSv1.3 state machine

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fix some TLSProxy warnings

After the client processes the server's initial flight in TLS1.3 it may
respond with either an encrypted, or an unencrypted alert. We needed to
teach TLSProxy about this so that it didn't issue spurious warnings.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Ensure unexpected messages are handled consistently

In one case we weren't always sending an unexpected message alert if we
don't get what we expect.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Ensure the end of first server flight processing is done

There is a set of miscellaneous processing for OCSP, CT etc at the end of
the ServerDone processing. In TLS1.3 we don't have a ServerDone, so this
needs to move elsewhere.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fix the tests following the state machine changes for TLSv1.3

Reviewed-by: Rich Salz <rsalz@openssl.org>

Create the Finished message payload

The previous commit had a dummy payload for the Finished data. This commit
fills it in with a real value.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Update state machine to be closer to TLS1.3

This is a major overhaul of the TLSv1.3 state machine. Currently it still
looks like TLSv1.2. This commit changes things around so that it starts
to look a bit less like TLSv1.2 and bit more like TLSv1.3.

After this commit we have:

ClientHello
+ key_share          ---->
                           ServerHello
                           +key_share
                           {CertificateRequest*}
                           {Certificate*}
                           {CertificateStatus*}
                     <---- {Finished}
{Certificate*}
{CertificateVerify*}
{Finished}           ---->
[ApplicationData]    <---> [Application Data]

Key differences between this intermediate position and the final TLSv1.3
position are:
- No EncryptedExtensions message yet
- No server side CertificateVerify message yet
- CertificateStatus still exists as a separate message
- A number of the messages are still in the TLSv1.2 format
- Still running on the TLSv1.2 record layer

Reviewed-by: Rich Salz <rsalz@openssl.org>

Implement tls13_change_cipher_state()

Reviewed-by: Rich Salz <rsalz@openssl.org>

Remove old style NewSessionTicket from TLSv1.3

TLSv1.3 has a NewSessionTicket message, but it is *completely* different to
the TLSv1.2 one and may as well have been called something else. This commit
removes the old style NewSessionTicket from TLSv1.3. We will have to add the
new style one back in later.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fix EXTMS error introduced by commit 94ed2c6

Commit 94ed2c6 dropped a ! operator by mistake, which causes extended
master secret connections to fail. This puts in back.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Clarify what X509_NAME_online does with the given buffer and size

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1980)

Add missing -zdelete for some linux arches

b6d5ba1a9f004d637acac18ae3519fe063b6b5e1 forgot to update some linux arches.

Reviewed-by: Richard Levitte <levitte@openssl.org>
GH: #1977

Skipping tests in evp_test leaks memory

When configured with "no-mdc2 enable-crypto-mdebug" the evp_test
will leak memory due to skipped tests, and error out.

Also fix a skip condition

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1968)

Make SSL_read and SSL_write return the old behaviour and document it.

This reverts commit 4880672a9b41a09a0984b55e219f02a2de7ab75e.

Fixes: #1903
Reviewed-by: Matt Caswell <matt@openssl.org>
GH: #1931

Add test to check EVP_PKEY method ordering.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Avoid warnings like unused enum value (as suggested by Rich Salz)

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1945)

Shift inside of switch back one level (code style change, as requested by Rich Salz)

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1945)

Add LMTP support (RFC 2033) to s_client ("-starttls lmtp")

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1945)

Use consistent variable names in example

In the X509_NAME_get_index_by_NID.pod example, the initialized variable is called
"loc", but the one used in the for loop is called "lastpos". Make the names match.

CLA: trivial
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1949)

Update the location of the pod files

CLA: trivial
Since 99d63d4 ("Move manpages to man[1357] structure.", 2016-10-26), the location
of the pod files has changed.
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1951)

Fix name of "locked" variable

It's called with 0 when it's already locked, with 1 when it's not.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #1500

Add support for reference counting using C11 atomics

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #1500

Support MSBLOB format if RC4 is disabled

Reviewed-by: Matt Caswell <matt@openssl.org>

sha/asm/sha512-armv8.pl: fix big-endian support in __KERNEL__ case.

In non-__KERNEL__ context 32-bit-style __ARMEB__/__ARMEL__ macros were
set in arm_arch.h, which is shared between 32- and 64-bit builds. Since
it's not included in __KERNEL__ case, we have to adhere to official
64-bit pre-defines, __AARCH64EB__/__AARCH64EL__.

[If we are to share more code, it would need similar adjustment.]

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>

Fix missing SSL_IS_TLS13(s) usage

We should use the macro for testing if we are using TLSv1.3 rather than
checking s->version directly.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Handle "int_ctx_new:unsupported algorithm" error

Calling EVP_PKEY_CTX_new_id(curve_NID, NULL) causes an error for most
curves that are implemented through the EC low-level API, and in the
last commit we call it for every curve to avoid treating X25519 as a
special case.

Last commit code already handles correctly this failure, but does not
remove these events from the thread error queue, thus some
false-positive warnings are printed at the end of execution.

This commit ensures that the error queue is clean, without flushing
other errors.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1658)

Do not handle R_EC_X25519 as a special case

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1658)

Remove leftover KDF pointer

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1658)

Use CRYPTO_memcmp for comparing derived secrets

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1658)

Reintroduce preliminary sanity check in ECDH speed and remove further checks in the benchmark loop.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1658)

Remove ECDH_EVP_derive_key wrapper function

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1658)

More coding style fixes

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1658)

Run util/openssl-format-source against apps/speed.c

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1658)

bugfix: calculate outlen for each curve

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1658)

Handle EVP_PKEY_derive errors and fix coding style issues

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1658)

Fix coding style and remove some stale code/comments

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1658)

Use EVP interface for ECDH in apps/speed.c

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1658)

Add conversion test for MSBLOB format.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Make MSBLOB format work with dsa utility.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fix MSBLOB format with RSA.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Raise an error on memory alloc failure.

Both strdup or malloc failure should raise à err.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1905)

Missing free item on push failure

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1905)

Merge find-undoc-api into find-doc-nits

Use \b on NOEXIST and EXPORT_VAR_AS_FUNC patterns as suggested by Andy.

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1912)

Move SCT_LIST_free definition into a more logical place

This reflects its position in include/openssl/ct.h.

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1548)

Make sure things get deleted when test setup fails in ct_test.c

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1548)

Use valid signature in test_decode_tls_sct()

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1548)

Pass a temporary pointer to o2i_SCT_signature from SCT_new_from_base64

Otherwise, |dec| gets moved past the end of the signature by
o2i_SCT_signature and then can't be correctly freed afterwards.

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1548)

Subtract padding from outlen in ct_base64_decode

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1548)

Construct SCT from base64 in ct_test

This gives better code coverage and is more representative of how a
user would likely construct an SCT (using the base64 returned by a CT log).

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1548)

On x86 machines where the compiler supports -m32, use 'linux-x86'

The rationale is that the linux-x86 is the most likely config target
to evolve and should therefore be chosen when possible, while
linux-elf is mostly reserved for older Linux machines.

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1924)

Add a modern linux-x86 config target

'linux-x86' is similar to 'linux-x86_64' but uses -m32 rather than -m64.

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1924)

Fix a "defined but not used" warning when enabling ssl-trace

Reviewed-by: Rich Salz <rsalz@openssl.org>

Remove a hack from ssl_test_old

ssl_test_old was reaching inside the SSL structure and changing the internal
BIO values. This is completely unneccessary, and was causing an abort in the
test when enabling TLSv1.3.

I also removed the need for ssl_test_old to include ssl_locl.h. This
required the addition of some missing accessors for SSL_COMP name and id
fields.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Add SSL_peek() and SSL_peek_ex() to NAME section

Reviewed-by: Rich Salz <rsalz@openssl.org>