test: fetching proper signature provider for non-exportable keys

Co-author: Selva Nair <selva.nair@gmail.com>

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16725)

CORE: Encure that cached fetches can be done per provider

This mostly entails passing around a provider pointer, and handling
queries that includes a pointer to a provider, where NULL means "any".

This also means that there's a need to pass the provider pointer, not
just down to the cache functions, but also be able to get it from
ossl_method_store_fetch().  To this end, that function's OSSL_PROVIDER
pointer argument is modified to be a pointer reference, so the
function can answer back what provider the method comes from.

Test added.

Fixes #16614

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16725)

EVP: For all operations that use an EVP_PKEY, check that there is one

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16725)

EVP: Allow a fallback for operations that work with an EVP_PKEY

Functions like EVP_PKEY_sign_init() do an implicit fetch of the
operation implementation (EVP_SIGNATURE in this case), then get the
KEYMGMT from the same provider, and tries to export the key there if
necessary.

If an export of the key isn't possible (because the provider that
holds the key is an HSM and therefore can't export), we would simply
fail without looking any further.

This change modifies the behaviour a bit by trying a second fetch of
the operation implementation, but specifically from the provider of
the EVP_PKEY that's being used.  This is done with the same properties
that were used with the initial operation implementation fetch, and
should therefore be safe, allowing only what those properties allow.

Fixes #16614

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16725)

EVP: Add internal functions to fetch type specific EVP methods from provider

Added functions:

evp_signature_fetch_from_prov(), evp_asym_cipher_fetch_from_prov(),
evp_keyexch_fetch_from_prov(), evp_kem_fetch_from_prov()

These are all like the public conterparts, except they all take a
provider instead of a library context as first argument.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16725)

EVP: Reverse the fetch logic in all pkey using functionality

In all initializing functions for functionality that use an EVP_PKEY, the
coded logic was to find an KEYMGMT implementation first, and then try to
find the operation method (for example, SIGNATURE implementation) in the
same provider.

This implies that in providers where there is a KEYMGMT implementation,
there must also be a SIGNATURE implementation, along with a KEYEXCH,
ASYM_CIPHER, etc implementation.

The intended design was, however, the opposite implication, i.e. that
where there is a SIGNATURE implementation, there must also be KEYMGMT.

This change reverses the logic of the code to be closer to the intended
design.

There is a consequence; we now use the query_operation_name function from
the KEYMGMT of the EVP_PKEY given by the EVP_PKEY_CTX (ultimately given by
the application).  Previously, we used the query_operation_name function
from the KEYMGMT found alongside the SIGNATURE implementation.

Another minor consequence is that the |keymgmt| field in EVP_PKEY_CTX
is now always a reference to the KEYMGMT of the |pkey| field if that
one is given (|pkey| isn't NULL) and is provided (|pkey->keymgmt|
isn't NULL).

Fixes #16614

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16725)

EVP: Add evp_keymgmt_fetch_from_prov()

This is an internal function to fetch a keymgmt method from a specific
provider.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16725)

EVP: Add the internal function evp_generic_fetch_from_prov()

This function leverages the generic possibility to fetch EVP methods
from a specific provider.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16725)

CORE: add a provider argument to ossl_method_construct()

This makes it possible to limit the search of methods to that
particular provider.  This uses already available possibilities in
ossl_algorithm_do_all().

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16725)

test/ssl_old_test.c: Do NULL pointer check before its use

In openssl-3.0.0 and system provided, it is not reasonable to
check null pointer after use. The order was accidentally reversed.
Therefore, it is better to correct it.

CLA: trivial

Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16915)

free the Post-Handshake Auth digest when there is an error saving the digest

Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16917)

Remove unused variable 'sctx'

In the openssl-3.0.0 and system provided, the variable 'sctx' is unused in test_509_dup_cert.
Therefore, it might be better to remove the definition and operation of it.

CLA: trivial

Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16916)

Test that a key is usable after an EVP_PKEY_fromdata call

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16911)

Don't crash encoding a public key with no public key value

If asked to encode an EC_KEY public key, but no public key value is present
in the structure, we should fail rather than crash.

Fixes the crash seen here:
https://mta.openssl.org/pipermail/openssl-users/2021-October/014479.html

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16911)

add checks for the return values of BN_new(), sk_RSA_PRIME_INFO_new_reserve(),
EVP_PKEY_CTX_new_from_pkey() and EVP_CIPHER_CTX_new().
Otherwise may result in memory errors.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16892)

speed: range check the argument given to -multi

For machines where sizeof(size_t) == sizeof(int) there is a possible overflow
which could cause a crash.

For machines where sizeof(size_t) > sizeof(int), the existing checks adequately
detect the situation.

Fixes #16899

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16904)

Doc: replace `NULL` terminated with `NUL`

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Ben Kaduk <kaduk@mit.edu>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16885)

doc: Fix some function signature errors

Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16897)

test-rand: return failure on not enough data, allow parent

The test-rand RNG was returning success when it had some but insufficient data.
Now, it returns failure and doesn't advance the data pointer.

The test-rand RNG was failing when a parent was specified.  This case is now
ignored.

Fixes #16785

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16905)

Note that SHA1 and MD5 x509 signatures are also forbidden at security level 1

The exclusion of SHA1 for X509 signatures is not obvious as the "intuative"
idea is that SHA1 should have 80 security bits. However the security bits
of SHA1 are explicitly set to 63 to avoid the it being strong enough for
security level 1. x509_set.c has the comment:

    /*
     * SHA1 and MD5 are known to be broken. Reduce security bits so that
     * they're no longer accepted at security level 1.
     * The real values don't really matter as long as they're lower than 80,
     * which is our security level 1.
     */

Signed-off-by: Arne Schwabe <arne@rfc2549.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16895)

migration_guide: Mention ERR_GET_FUNC() and function code removal

Fixes #16817

Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16879)

cmp.c: Avoid dereference with negative index and use memcpy

This prevents a compile-time warning on newer gcc.

Also fix the related warning message.

Fixes #16814

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16821)

X509_PUBKEY_dup: Do not just up-ref the EVP_PKEY

We try EVP_PKEY_dup() and if it fails we re-decode it using the
legacy method as provided keys should be duplicable.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16648)

X509_dup: Avoid duplicating the embedded EVP_PKEY

The EVP_PKEY will be recreated from scratch which is OK.

Fixes #16606

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16648)

tests: Add test for X509_dup with ENGINE based key

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16648)

Fix no-cmac

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16894)

OCSP_sendreq_bio: Avoid doublefree of mem BIO

Reviewed-by: David von Oheimb <david.von.oheimb@siemens.com>
(Merged from https://github.com/openssl/openssl/pull/16886)

doc: Fix some function signature errors

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16874)

OSSL_HTTP_transfer.pod: clarify that resulting BIO must be freed

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16888)

OSSL_HTTP_REQ_CTX.pod: clarify that resulting BIO must not be freed

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16888)

Permit no/empty digest in core_obj_add_sigid

Also add digest parameter documentation for add_sigid and
permit NULL as digest name in the provider upcall.

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16770)

Doc: be explicit about NUL in max_identity_len

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16877)

Fix the s_server psk_server_cb for use in DTLS

Commit 0007ff257c added a protocol version check to psk_server_cb but
failed to take account of DTLS causing DTLS based psk connections to
fail.

Fixes #16707

Reviewed-by: Ben Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/16838)

Clean up on failed BIO creation

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16316)

Fix documentation errors, mainly caused by return values of BIO_ctrl

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16767)

Fix some documentation errors related to return values

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16794)

doc: EVP_PKEY_get_utf8/octet_string_param() clarify NULL buffer behavior

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/16843)

doc: Document the type of label EVP_PKEY_CTX_set0_rsa_oaep_label properly

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/16869)

Update pyca-cryptography sub-module

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16789)

Fix acvp_test sig_gen

Ensure we set the size of the signature buffer before we call
EVP_DigestSign()

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16789)

Fix test_CMAC_keygen

Make sure we correctly pass through the size of the buffer to
EVP_DigestSignFinal

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16789)

Fix a bug in signature self tests in the FIPS module

When calling EVP_PKEY_sign(), the size of the signature buffer must
be passed in *siglen.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16789)

Add an additional note to EVP_DigestSign() documentation

Clarify what happens if it fails. Make it clear that you can pass a NULL
"sig" buffer to get the "siglen".

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16789)

Test short buffers

Test that calling EVP_DigestSign(), EVP_DigestSignFinal(),
EVP_PKEY_sign(), EVP_PKEY_get_raw_private_key(), or
EVP_PKEY_get_raw_public_key() with a short output buffer results in a
failure.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16789)

Fix SSKDF to not claim a buffer size that is too small for the MAC

We also check that our buffer is sufficiently sized for the MAC output

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16789)

Enforce a size check in EVP_MAC_final()

Make sure that the outsize for the buffer is large enough for the
output from the MAC.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16789)

Prevent an overflow if an application supplies a buffer that is too small

If an application bug means that a buffer smaller than is necessary is
passed to various functions then OpenSSL does not spot that the buffer
is too small and fills it anyway. This PR prevents that.

Since it requires an application bug to hit this problem, no CVE is
allocated.

Thanks to David Benjamin for reporting this issue.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16789)

APPS/req.c: Make -reqexts option an alias of -extensions option

This simplifies code, doc, and use.
Fixes issue ignoring one or the other.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16865)

Add missing define to enable AES-NI usage on x86 platform

Fixes #16858

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16866)

Fix function signature error

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/16852)

Add tests for ENGINE problems

Add some tests which would have caught the issues fixed in the previous
3 commits related to engine handling.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16846)

Update provider_util.c to correctly handle ENGINE references

provider_util.c failed to free ENGINE references when clearing a cipher
or a digest. Additionally ciphers and digests were not copied correctly,
which would lead to double-frees if it were not for the previously
mentioned leaks.

Fixes #16845

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16846)

Ensure pkey_set_type handles ENGINE references correctly

pkey_set_type should not consume the ENGINE references that may be
passed to it.

Fixes #16757

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16846)

Make sure EVP_CIPHER_CTX_copy works with the dasync engine

Ciphers in the daysnc engine were failing to copy their context properly
in the event of EVP_CIPHER_CTX_copy() because they did not define the
flag EVP_CIPH_CUSTOM_FLAG

Fixes #16844

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16846)

Avoid NULL+X UB in bss_mem.c

Fixes #16816

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16818)

Fix the signature newctx documentation

The documentation omitted the propq parameter

Fixes #16755

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16836)

Fix lock leak in evp_keymgmt_util_export_to_provider()

Fixes #16847

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16849)

Raise error when invalid digest used with SM2

Otherwise commands like openssl req -newkey sm2 fail silently without
reporting any error unless -sm3 option is added.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16833)

test/ssl_old_test.c: Fix potential leak

Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16806)

Fix test/recipes/01-test_symbol_presence.t to disregard version info

The output of 'nm -DPg' contains version info attached to the symbols,
which makes the test fail.  Simply dropping the version info makes the
test work again.

Fixes #16810 (followup)

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16840)

Fix another memory leak reported in CIFuzz

Direct leak of 2 byte(s) in 1 object(s) allocated from:
    #0 0x4a067d in __interceptor_malloc /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:129:3
    #1 0x57acd9 in CRYPTO_malloc /src/openssl/crypto/mem.c:184:12
    #2 0x57e106 in CRYPTO_strdup /src/openssl/crypto/o_str.c:24:11
    #3 0x5c139f in def_load_bio /src/openssl/crypto/conf/conf_def.c:427:45
    #4 0x56adf5 in NCONF_load_bio /src/openssl/crypto/conf/conf_lib.c:282:12
    #5 0x4d96cf in FuzzerTestOneInput /src/openssl/fuzz/conf.c:38:5
    #6 0x4d9830 in LLVMFuzzerTestOneInput /src/openssl/fuzz/driver.c:28:12
    #7 0x510c23 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) cxa_noexception.cpp
    #8 0x4fc4d2 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:324:6
    #9 0x501f85 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) cxa_noexception.cpp
    #10 0x52ac82 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
    #11 0x7f15336bf0b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16813)

Fix a memory leak reported in CIFuzz

Direct leak of 4 byte(s) in 1 object(s) allocated from:
     #0 0x4a067d in __interceptor_malloc /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:129:3
     #1 0x57af0d in CRYPTO_malloc /src/openssl/crypto/mem.c:184:12
     #2 0x57af0d in CRYPTO_realloc /src/openssl/crypto/mem.c:207:16
     #3 0x569d17 in BUF_MEM_grow /src/openssl/crypto/buffer/buffer.c:97:15
     #4 0x5c3629 in str_copy /src/openssl/crypto/conf/conf_def.c:642:10
     #5 0x5c1cc1 in def_load_bio /src/openssl/crypto/conf/conf_def.c:452:22
     #6 0x56adf5 in NCONF_load_bio /src/openssl/crypto/conf/conf_lib.c:282:12
     #7 0x4d96cf in FuzzerTestOneInput /src/openssl/fuzz/conf.c:38:5
     #8 0x4d9830 in LLVMFuzzerTestOneInput /src/openssl/fuzz/driver.c:28:12
     #9 0x510c23 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) cxa_noexception.cpp
     #10 0x4fc4d2 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:324:6
     #11 0x501f85 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) cxa_noexception.cpp
     #12 0x52ac82 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16813)

Fix test/recipes/01-test_symbol_presence.t to allow for stripped libraries

It's a small change to the 'nm' call, to have it look at dynamic symbols
rather than the normal ones.

Fixes #16810

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16822)

feat: Add sm2 signature test case from GM/T 0003.5-2012

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16802)

cmp_vfy.c, encoder_lib.c: Fix potential leak of a BIO

Fixes #16787

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/16804)

ctrl_params_translate: Fix leak of BN_CTX

Also add a missing allocation failure check.

Fixes #16788

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/16804)

req: Do not warn about using stdin when generating new request

Fixes #16773

Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16774)

Extend custom extension testing

Test the scenario where we add a custom extension to a cetificate
request and expect a response in the client's certificate message.

Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16634)

New extensions can be sent in a certificate request

Normally we expect a client to send new  extensions in the ClientHello,
which may be echoed back by the server in subsequent messages. However the
server can also send a new extension in the certificate request message to
be echoed back in a certificate message

Fixes #16632

Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16634)

doc: OPENSSL_CORE_CTX should never be cast to OSSL_LIB_CTX

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16771)

Bugfix: unsafe return check of EVP_PKEY_fromdata

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16783)

Bugfix: unsafe return check of EVP_PKEY_fromdata_init

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16783)

Update gost-engine to the latest version

Update the gost-engine submodule to pick up the latest version
including fixes for the default security level of 2.

Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/16760)

Update document for default security level change

Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/16760)

Fix tests for new default security level

Fix tests that were expecting a default security level of 1 to work with
the new default of 2.

Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/16760)

Increase the default security level to 2

OTC voted to increase the security level from 1 to 2

Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/16760)

doc: document that property names are unique

Both queries and definitions only support each individual name appearing once.
It is an error to have a name appear more than once.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16716)

test: add failure testing for property parsing

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16716)

property: produce error if a name is duplicated

Neither queries nor definitions handle duplicated property names well.
Make having such an error.

Fixes #16715

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16716)

Bindhost/bindport should be freed

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16775)

Fix unsafe BIO_get_md_ctx check

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16768)

Fix heading in random generator man7 page

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16753)

apps/x509: Fix self-signed check to happen before setting issuer name

Fixes #16720

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16747)

s_socket.c: Avoid possible NULL pointer dereference

Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/16736)

Replace the AES-128-CBC-HMAC-SHA1 cipher in e_ossltest.c

This replaces the AES-128-CBC-HMAC-SHA1 cipher with a
non-encrypting version for use the test suite.

[extended tests]

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16693)

Remove OPENSSL_ia32cap overrides in various test scripts

The removed override was: OPENSSL_ia32cap=~0x200000200000000
which disables AESNI codepaths and PCLMULQDQ (useful for ghash).
It is unclear why this was done, but it probably just hides bugs.

[extended tests]

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16693)

Fix a memory leak in the afalg engine

Fixes: #16743
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16744)

Fix for the dasync engine

Fixes: #16724
Fixes: #16735
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16734)

obj: Add SM4 GCM/CCM OID

Add the following OID:

  SM4-GCM: 1.2.156.10197.1.104.8
  SM4-CCM: 1.2.156.10197.1.104.9

Signed-off-by: Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16731)

RISC-V support for the SHA256

Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16710)

BIO_f_ssl.pod: Make clear where an SSL BIOs are expected as an argument

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16688)

apps/lib/s_socket.c: Fix mem leak on host name in init_client()

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16688)

Fix ssl_free() and thus BIO_free() to respect BIO_NOCLOSE

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16688)

aarch64: support BTI and pointer authentication in assembly

This change adds optional support for
- Armv8.3-A Pointer Authentication (PAuth) and
- Armv8.5-A Branch Target Identification (BTI)
features to the perl scripts.

Both features can be enabled with additional compiler flags.
Unless any of these are enabled explicitly there is no code change at
all.

The extensions are briefly described below. Please read the appropriate
chapters of the Arm Architecture Reference Manual for the complete
specification.

Scope
-----

This change only affects generated assembly code.

Armv8.3-A Pointer Authentication
--------------------------------

Pointer Authentication extension supports the authentication of the
contents of registers before they are used for indirect branching
or load.

PAuth provides a probabilistic method to detect corruption of register
values. PAuth signing instructions generate a Pointer Authentication
Code (PAC) based on the value of a register, a seed and a key.
The generated PAC is inserted into the original value in the register.
A PAuth authentication instruction recomputes the PAC, and if it matches
the PAC in the register, restores its original value. In case of a
mismatch, an architecturally unmapped address is generated instead.

With PAuth, mitigation against ROP (Return-oriented Programming) attacks
can be implemented. This is achieved by signing the contents of the
link-register (LR) before it is pushed to stack. Once LR is popped,
it is authenticated. This way a stack corruption which overwrites the
LR on the stack is detectable.

The PAuth extension adds several new instructions, some of which are not
recognized by older hardware. To support a single codebase for both pre
Armv8.3-A targets and newer ones, only NOP-space instructions are added
by this patch. These instructions are treated as NOPs on hardware
which does not support Armv8.3-A. Furthermore, this patch only considers
cases where LR is saved to the stack and then restored before branching
to its content. There are cases in the code where LR is pushed to stack
but it is not used later. We do not address these cases as they are not
affected by PAuth.

There are two keys available to sign an instruction address: A and B.
PACIASP and PACIBSP only differ in the used keys: A and B, respectively.
The keys are typically managed by the operating system.

To enable generating code for PAuth compile with
-mbranch-protection=<mode>:

- standard or pac-ret: add PACIASP and AUTIASP, also enables BTI
  (read below)
- pac-ret+b-key: add PACIBSP and AUTIBSP

Armv8.5-A Branch Target Identification
--------------------------------------

Branch Target Identification features some new instructions which
protect the execution of instructions on guarded pages which are not
intended branch targets.

If Armv8.5-A is supported by the hardware, execution of an instruction
changes the value of PSTATE.BTYPE field. If an indirect branch
lands on a guarded page the target instruction must be one of the
BTI <jc> flavors, or in case of a direct call or jump it can be any
other instruction. If the target instruction is not compatible with the
value of PSTATE.BTYPE a Branch Target Exception is generated.

In short, indirect jumps are compatible with BTI <j> and <jc> while
indirect calls are compatible with BTI <c> and <jc>. Please refer to the
specification for the details.

Armv8.3-A PACIASP and PACIBSP are implicit branch target
identification instructions which are equivalent with BTI c or BTI jc
depending on system register configuration.

BTI is used to mitigate JOP (Jump-oriented Programming) attacks by
limiting the set of instructions which can be jumped to.

BTI requires active linker support to mark the pages with BTI-enabled
code as guarded. For ELF64 files BTI compatibility is recorded in the
.note.gnu.property section. For a shared object or static binary it is
required that all linked units support BTI. This means that even a
single assembly file without the required note section turns-off BTI
for the whole binary or shared object.

The new BTI instructions are treated as NOPs on hardware which does
not support Armv8.5-A or on pages which are not guarded.

To insert this new and optional instruction compile with
-mbranch-protection=standard (also enables PAuth) or +bti.

When targeting a guarded page from a non-guarded page, weaker
compatibility restrictions apply to maintain compatibility between
legacy and new code. For detailed rules please refer to the Arm ARM.

Compiler support
----------------

Compiler support requires understanding '-mbranch-protection=<mode>'
and emitting the appropriate feature macros (__ARM_FEATURE_BTI_DEFAULT
and __ARM_FEATURE_PAC_DEFAULT). The current state is the following:

-------------------------------------------------------
| Compiler | -mbranch-protection | Feature macros     |
+----------+---------------------+--------------------+
| clang    | 9.0.0               | 11.0.0             |
+----------+---------------------+--------------------+
| gcc      | 9                   | expected in 10.1+  |
-------------------------------------------------------

Available Platforms
------------------

Arm Fast Model and QEMU support both extensions.

https://developer.arm.com/tools-and-software/simulation-models/fast-models
https://www.qemu.org/

Implementation Notes
--------------------

This change adds BTI landing pads even to assembly functions which are
likely to be directly called only. In these cases, landing pads might
be superfluous depending on what code the linker generates.
Code size and performance impact for these cases would be negligible.

Interaction with C code
-----------------------

Pointer Authentication is a per-frame protection while Branch Target
Identification can be turned on and off only for all code pages of a
whole shared object or static binary. Because of these properties if
C/C++ code is compiled without any of the above features but assembly
files support any of them unconditionally there is no incompatibility
between the two.

Useful Links
------------

To fully understand the details of both PAuth and BTI it is advised to
read the related chapters of the Arm Architecture Reference Manual
(Arm ARM):
https://developer.arm.com/documentation/ddi0487/latest/

Additional materials:

"Providing protection for complex software"
https://developer.arm.com/architectures/learn-the-architecture/providing-protection-for-complex-software

Arm Compiler Reference Guide Version 6.14: -mbranch-protection
https://developer.arm.com/documentation/101754/0614/armclang-Reference/armclang-Command-line-Options/-mbranch-protection?lang=en

Arm C Language Extensions (ACLE)
https://developer.arm.com/docs/101028/latest

Addional Notes
--------------

This patch is a copy of the work done by Tamas Petz in boringssl. It
contains the changes from the following commits:

aarch64: support BTI and pointer authentication in assembly
    Change-Id: I4335f92e2ccc8e209c7d68a0a79f1acdf3aeb791
    URL: https://boringssl-review.googlesource.com/c/boringssl/+/42084
aarch64: Improve conditional compilation
    Change-Id: I14902a64e5f403c2b6a117bc9f5fb1a4f4611ebf
    URL: https://boringssl-review.googlesource.com/c/boringssl/+/43524
aarch64: Fix name of gnu property note section
    Change-Id: I6c432d1c852129e9c273f6469a8b60e3983671ec
    URL: https://boringssl-review.googlesource.com/c/boringssl/+/44024

Change-Id: I2d95ebc5e4aeb5610d3b226f9754ee80cf74a9af

Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16674)

doc: crypto(7) - fix typo

CLA: trivial

Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16695)

Fully initialise cipher/digest app handles

This avoids a crash in e.g. `openssl chacha20` as reported by
Steffen Nurpmeso on openssl-users.

Resolves: #16713

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16714)

doc/man3/SSL_set_fd.pod: add note about Windows compiler warning

According to an old stackoverflow thread [1], citing an even older comment by
Andy Polyakov (1875e6db29, Pull up Win64 support from 0.9.8., 2005-07-05),
a cast of 'SOCKET' (UINT_PTR) to 'int' does not create a problem, because although
the documentation [2] claims that the upper limit is INVALID_SOCKET-1 (2^64 - 2),
in practice the socket() implementation on Windows returns an index into the kernel
handle table, the size of which is limited to 2^24 [3].

Add this note to the manual page to avoid unnecessary roundtrips to StackOverflow.

[1] https://stackoverflow.com/questions/1953639/is-it-safe-to-cast-socket-to-int-under-win64
[2] https://docs.microsoft.com/en-us/windows/win32/winsock/socket-data-type-2
[3] https://docs.microsoft.com/en-us/windows/win32/sysinfo/kernel-objects

Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16699)

RISC-V support for the SHA512

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16638)

Update manual to reference the IANA TLS Cipher Suites Registry

Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16179)

Allow cipher strings to be given using its standard name

Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16179)