Use an enum for the return value from a construction function

Use an enum for the return value from a construction function

Construction return values are no longer boolean but can return 3 different
values, so we use an enum to represent them.

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Hugo Landau <hlandau@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/18990)