zero pad DHE public key in ServerKeyExchange message for interop
authorrussor <russor@whatsapp.com>
Mon, 25 Jul 2016 17:30:13 +0000 (10:30 -0700)
committerRich Salz <rsalz@openssl.org>
Mon, 25 Jul 2016 17:41:33 +0000 (13:41 -0400)
commit78a01b3f69f563a1577a6f90edbd9ebde80d6b70
tree0fc2fa9107f0b1bb6af45c38803573ca3f8625d8
parentfe0169b09717b3c3d52c0fba96e1dcf5e8a60d94
zero pad DHE public key in ServerKeyExchange message for interop

Some versions of the Microsoft TLS stack have problems when the DHE public key
is encoded with fewer bytes than the DHE prime.

There's some public acknowledgement of the bug at these links:

https://connect.microsoft.com/IE/feedback/details/1253526/tls-serverkeyexchange-with-1024-dhe-may-encode-dh-y-as-127-bytes-breaking-internet-explorer-11
https://connect.microsoft.com/IE/feedback/details/1104905/wininet-calculation-of-mac-in-tls-handshake-intermittently-fails-for-dhe-rsa-key-exchange

This encoding issue also causes the same errors with 2048-bit DHE, if the
public key is encoded in fewer than 256 bytes and includes the TLS stack on
Windows Phone 8.x.

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1320)
ssl/statem/statem_srvr.c