Compiler warnings like one listed below are generated during server build on MacOS:
In file included from server-10.2-MDEV-23564/mysys_ssl/openssl.c:33:
In file included from /usr/local/include/openssl/evp.h:16:
In file included from /usr/local/include/openssl/bio.h:20:
/usr/local/include/openssl/crypto.h:206:10: warning: 'CRYPTO_cleanup_all_ex_data' macro redefined [-Wmacro-redefined]
^
/mariadb/server-10.2-MDEV-23564/include/ssl_compat.h:46:9: note: previous definition is here
^
In case MariaDB serer is build with -DCMAKE_BUILD_TYPE=Debug it results in
build error.
The reason of compiler warnings is that header file <ssl_compat.h>
included before the openssl system header files. File ssl_compat.h
contains some macros with the same names as SSL API functions declared
in the openssl system header files. It resulted in duplicate
symbols that produces compiler warnings.
To fix the issue the header file ssl_compat.h should be included
after a line where openssl system header is included.
for consistency with EVP_CIPHER_CTX_new() and EVP_CIPHER_CTX_free().
As the EVP_DIGESTINIT(3) man page says:
EVP_MD_CTX_create() and EVP_MD_CTX_destroy() were renamed to
EVP_MD_CTX_new() and EVP_MD_CTX_free() in OpenSSL 1.1.
Closes#621
If the CRYPTO_set_mem_functions fails just return success
from check_openssl_compatibility. The only case where
CRYPTO_set_mem_functions fails is the allow_customize==0 (aka FIPS
mode). The check_openssl_compatibility isn't able to complete unless
this function returns success.
ref: https://github.com/openssl/openssl/blob/OpenSSL_1_1_0g/crypto/mem.c#L34
CRYPTO_set_mem_functions() works only until the first allocation is done:
* remove the second CRYPTO_set_mem_functions() call
* check whether the first CRYPTO_set_mem_functions() call worked
* stricter memory checks (==1, not >1, etc)
* as coc_malloc cannot be removed, make the counter a bit cheaper
* only do the check for OpenSSL 1.1 (because of OpenSSL 1.0 bug)
Apparently openssl ignores CRYPTO_set_mem_functions() in fips mode,
so we cannot detect structure sizes. Don't freak out when
no mallocs were detected.
post-review fixes:
* move all ssl implementation related ifdefs/defines to one file
(ssl_compat.h)
* work around OpenSSL-1.1 desire to malloc every EVP context by
run-time checking that context allocated on the stack is big enough
(openssl.c)
* use newer version of the AWS SDK for OpenSSL 1.1
* use get_dh2048() function as generated by openssl 1.1
(viosslfactories.c)
