Fix an assertion in the DTLS server code
This fixes an internal error alert from the server and
an unexpected connection failure in the release version,
but a failed assertion and a server crash in the
debug version.
Reproduce this issue with a DTLS server/client like that:
./openssl s_server -dtls -mtu 1500
./openssl s_client -dtls -maxfraglen 512
In the debug version a crash happens in the Server now:
./openssl s_server -dtls -mtu 1500
Using default temp DH parameters
ACCEPT
ssl/statem/statem_dtls.c:269: OpenSSL internal error: Assertion failed: len == written
Aborted (core dumped)
While in the release version the handshake exceeds the
negotiated max fragment size, and fails because of this:
$ ./openssl s_server -dtls -mtu 1500
Using default temp DH parameters
ACCEPT
ERROR
4057152ADA7F0000:error:
0A0000C2:SSL routines:do_dtls1_write:exceeds max fragment size:ssl/record/rec_layer_d1.c:826:
shutting down SSL
CONNECTION CLOSED
From the client's point of view the connection fails
with an Internal Error Alert:
$ ./openssl s_client -dtls -maxfraglen 512
Connecting to ::1
CONNECTED(
00000003)
40B76343377F0000:error:
0A000438:SSL routines:dtls1_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_d1.c:613:SSL alert number 80
and now the connection attempt fails unexpectedly.
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/18093)
(cherry picked from commit
e915c3f5381cd38ebdc1824c3ba9896ea7160103)