Disallow SSL_key_update() if there are writes pending

If an application is halfway through writing application data it should
not be allowed to attempt an SSL_key_update() operation. Instead the
SSL_write() operation should be completed.

Fixes #12485

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16077)

diff --git a/ssl/ssl_lib.c b/ssl/ssl_lib.c
index c1e8e41f02bc68581a110323c3e444cc6cd1bc6b..892a417d935437196b8cd352e8875271aef1005f 100644 (file)
--- a/ssl/ssl_lib.c
+++ b/ssl/ssl_lib.c
@@ -2262,6 +2262,11 @@ int SSL_key_update(SSL *s, int updatetype)
         return 0;
     }
 
+    if (RECORD_LAYER_write_pending(&s->rlayer)) {
+        ERR_raise(ERR_LIB_SSL, SSL_R_BAD_WRITE_RETRY);
+        return 0;
+    }
+
     ossl_statem_set_in_init(s, 1);
     s->key_update = updatetype;
     return 1;