This gives a short overview over found/missing dependencies as well
as enabled/disabled features.
Initial author Heinz Wiesinger <heinz@m2mobi.com>
Additions by Vicențiu Ciorbaru <vicentiu@mariadb.org>
* Report all plugins enabled via MYSQL_ADD_PLUGIN
* Simplify code. Eliminate duplication by making use of WITH_xxx
variable values to set feature "ON" / "OFF" state.
Reviewed by: wlad@mariadb.com (code details) serg@mariadb.com (the idea)
mysql_config should never return -lzlib as linking flag.
Clients trying to link against mariadb that use the output of
mysql_config experience linking error because we return -lzlib
when we don't ship any libzlib.a in our binary tarballs, bundled
zlib static library is already linked.
added %config directive to /etc/logrotate.d/mysql and put it to server
package. if file is edited by customer and defaults are changed - new
.rpmnew file with defaults will be created next to old one.
Not all environments have 'diff' installed. Most notably CentOS 8
does not have diff out-of-the-box. Thus users running 'cmake .' and
'make' would fail to build MariaDB, and they would think the error
was in ABI incompatibilities due to the error message emitted by CMake
when in reality simply 'diff' was missing.
This fixes it and makes the developer experience better by simply skipping
the diffing if 'diff' is not found.
## Proof
Running in a clean CentOS 8 container.
### Before
```
$ cmake .
...
-- Looking for cpuid.h
-- Looking for cpuid.h - found
-- Looking for x86intrin.h
-- Looking for x86intrin.h - found
-- Looking for event.h
-- Looking for event.h - not found
-- Configuring done
-- Generating done
-- Build files have been written to: /build
$ make
Scanning dependencies of target abi_check
CMake Error at cmake/do_abi_check.cmake:82 (MESSAGE):
ABI check found difference between /build/include/mysql/plugin_audit.h.pp
and /build/abi_check.out
make[2]: *** [CMakeFiles/abi_check.dir/build.make:57: CMakeFiles/abi_check] Error 1
make[1]: *** [CMakeFiles/Makefile2:168: CMakeFiles/abi_check.dir/all] Error 2
make: *** [Makefile:163: all] Error 2
$ yum install -y diffutils
...
Installed:
diffutils-3.6-6.el8.x86_64
Complete!
$ make
[ 0%] Built target abi_check
Scanning dependencies of target INFO_BIN
[ 0%] Built target INFO_BIN
Scanning dependencies of target INFO_SRC
[ 0%] Built target INFO_SRC
Scanning dependencies of target wsrep_api_v26
[ 0%] Building C object wsrep-lib/wsrep-API/CMakeFiles/wsrep_api_v26.dir/v26/wsrep_dummy.c.o
[ 0%] Building C object wsrep-lib/wsrep-API/CMakeFiles/wsrep_api_v26.dir/v26/wsrep_gtid.c.o
[ 0%] Building C object wsrep-lib/wsrep-API/CMakeFiles/wsrep_api_v26.dir/v26/wsrep_loader.c.o
[ 0%] Building C object wsrep-lib/wsrep-API/CMakeFiles/wsrep_api_v26.dir/v26/wsrep_uuid.c.o
[ 0%] Linking C static library libwsrep_api_v26.a
[ 0%] Built target wsrep_api_v26
```
### After
```
$ make
Command 'diff' not found. ABI check for /build/server/include/mysql/plugin_audit.h skipped.
Command 'diff' not found. ABI check for /build/server/include/mysql/plugin_ftparser.h skipped.
Command 'diff' not found. ABI check for /build/server/include/mysql.h skipped.
Command 'diff' not found. ABI check for /build/server/include/mysql/psi/psi_abi_v1.h skipped.
Command 'diff' not found. ABI check for /build/server/include/mysql/psi/psi_abi_v2.h skipped.
Command 'diff' not found. ABI check for /build/server/include/mysql/client_plugin.h skipped.
Command 'diff' not found. ABI check for /build/server/include/mysql/plugin_auth.h skipped.
Command 'diff' not found. ABI check for /build/server/include/mysql/plugin_password_validation.h skipped.
Command 'diff' not found. ABI check for /build/server/include/mysql/plugin_encryption.h skipped.
Command 'diff' not found. ABI check for /build/server/include/mysql/plugin_data_type.h skipped.
Command 'diff' not found. ABI check for /build/server/include/mysql/plugin_function.h skipped.
[ 0%] Built target abi_check
[ 0%] Built target INFO_SRC
[ 0%] Built target INFO_BIN
[ 0%] Built target wsrep_api_v26
[ 0%] Building CXX object wsrep-lib/src/CMakeFiles/wsrep-lib.dir/server_state.cpp.o
```
If diff is installed, those warnings are simply not shown. Builds pass
without the need to install 'diff'.
Brad Smith made this OpenBSD file based of the
FreeBSD cmake directives in commit ab58904367
by the Monty Program Ab.
As such the Oracle Copyright header isn't really applicable.
Not all environments have 'diff' installed. Most notably CentOS 8
does not have diff out-of-the-box. Thus users running 'cmake .' and
'make' would fail to build MariaDB, and they would think the error
was in ABI incompatibilities due to the error message emitted by CMake
when in reality simply 'diff' was missing.
This fixes it and makes the developer experience better by simply
skipping the diffing if 'diff' is not found.
Closes#1846
RocksDB failed to package on fedora 32 and 33 with
*** ERROR: ambiguous python shebang in /usr/bin/myrocks_hotbackup: #!/usr/bin/env python. Change it to python3 (or python2) explicitly.
This fixed the MySQL bug# 20338 about misuse of double underscore
prefix __WIN__, which was old MySQL's idea of identifying Windows
Replace it by _WIN32 standard symbol for targeting Windows OS
(both 32 and 64 bit)
Not that connect storage engine is not fixed in this patch (must be
fixed in "upstream" branch)
- As solution `PLUGIN_CONNECT=NO` use early check to disable plugin:
Solution suggested by wlad@mariadb.com
- `JNI_FOUND` is a internal result variable and should be set with
cached library and header variables (like `JAVA_INCLUDE_PATH`) defined.
* Note: wrapper cmake/FindJNI.cmake runs first time and cmake native Find<module> returns only cached variable, like `JAVA_INCLUDE_PATH`, results variable are not cached).
Reviewed by: serg@mariadb.com
- As solution `PLUGIN_CONNECT=NO` use early check to disable plugin:
Solution suggested by wlad@mariadb.com
- `JNI_FOUND` is a internal result variable and should be set with
cached library and header variables (like `JAVA_INCLUDE_PATH`) defined.
* Note: wrapper cmake/FindJNI.cmake runs first time and cmake native Find<module> returns only cached variable, like `JAVA_INCLUDE_PATH`, results variable are not cached).
Reviewed by: serg@mariadb.com
If IPO is enabled, MSVC compiles with /GL which is not
compatible with cmake's WINDOWS_EXPORT_ALL_SYMBOLS.
Since server.dll can't autoexport symbols from /GL compiled objects,
IPO must be disabled for dbug, string,mysys, and sql targets (in case we
build "plugin" DLLs that need server.dll's symbols)
The easiest way to compile and test the server with UBSAN is to run:
./BUILD/compile-pentium64-ubsan
and then run mysql-test-run.
After this commit, one should be able to run this without any UBSAN
warnings. There is still a few compiler warnings that should be fixed
at some point, but these do not expose any real bugs.
The 'special' cases where we disable, suppress or circumvent UBSAN are:
- ref10 source (as here we intentionally do some shifts that UBSAN
complains about.
- x86 version of optimized int#korr() methods. UBSAN do not like unaligned
memory access of integers. Fixed by using byte_order_generic.h when
compiling with UBSAN
- We use smaller thread stack with ASAN and UBSAN, which forced me to
disable a few tests that prints the thread stack size.
- Verifying class types does not work for shared libraries. I added
suppression in mysql-test-run.pl for this case.
- Added '#ifdef WITH_UBSAN' when using integer arithmetic where it is
safe to have overflows (two cases, in item_func.cc).
Things fixed:
- Don't left shift signed values
(byte_order_generic.h, mysqltest.c, item_sum.cc and many more)
- Don't assign not non existing values to enum variables.
- Ensure that bool and enum values are properly initialized in
constructors. This was needed as UBSAN checks that these types has
correct values when one copies an object.
(gcalc_tools.h, ha_partition.cc, item_sum.cc, partition_element.h ...)
- Ensure we do not called handler functions on unallocated objects or
deleted objects.
(events.cc, sql_acl.cc).
- Fixed bugs in Item_sp::Item_sp() where we did not call constructor
on Query_arena object.
- Fixed several cast of objects to an incompatible class!
(Item.cc, Item_buff.cc, item_timefunc.cc, opt_subselect.cc, sql_acl.cc,
sql_select.cc ...)
- Ensure we do not do integer arithmetic that causes over or underflows.
This includes also ++ and -- of integers.
(Item_func.cc, Item_strfunc.cc, item_timefunc.cc, sql_base.cc ...)
- Added JSON_VALUE_UNITIALIZED to json_value_types and ensure that
value_type is initialized to this instead of to -1, which is not a valid
enum value for json_value_types.
- Ensure we do not call memcpy() when second argument could be null.
- Fixed that Item_func_str::make_empty_result() creates an empty string
instead of a null string (safer as it ensures we do not do arithmetic
on null strings).
Other things:
- Changed struct st_position to an OBJECT and added an initialization
function to it to ensure that we do not copy or use uninitialized
members. The change to a class was also motived that we used "struct
st_position" and POSITION randomly trough the code which was
confusing.
- Notably big rewrite in sql_acl.cc to avoid using deleted objects.
- Changed in sql_partition to use '^' instead of '-'. This is safe as
the operator is either 0 or 0x8000000000000000ULL.
- Added check for select_nr < INT_MAX in JOIN::build_explain() to
avoid bug when get_select() could return NULL.
- Reordered elements in POSITION for better alignment.
- Changed sql_test.cc::print_plan() to use pointers instead of objects.
- Fixed bug in find_set() where could could execute '1 << -1'.
- Added variable have_sanitizer, used by mtr. (This variable was before
only in 10.5 and up). It can now have one of two values:
ASAN or UBSAN.
- Moved ~Archive_share() from ha_archive.cc to ha_archive.h and marked
it virtual. This was an effort to get UBSAN to work with loaded storage
engines. I kept the change as the new place is better.
- Added in CONNECT engine COLBLK::SetName(), to get around a wrong cast
in tabutil.cpp.
- Added HAVE_REPLICATION around usage of rgi_slave, to get embedded
server to compile with UBSAN. (Patch from Marko).
- Added #ifdef for powerpc64 to avoid a bug in old gcc versions related
to integer arithmetic.
Changes that should not be needed but had to be done to suppress warnings
from UBSAN:
- Added static_cast<<uint16_t>> around shift to get rid of a LOT of
compiler warnings when using UBSAN.
- Had to change some '/' of 2 base integers to shift to get rid of
some compile time warnings.
Reviewed by:
- Json changes: Alexey Botchkov
- Charset changes in ctype-uca.c: Alexander Barkov
- InnoDB changes & Embedded server: Marko Mäkelä
- sql_acl.cc changes: Vicențiu Ciorbaru
- build_explain() changes: Sergey Petrunia
This fixes the MariaDB-client RPM to conflict with
< MariaDB-server-10.6 as a number of files where moved
from the server to client component.
see commits 09202b2e6d ->
fa0ad2fb11
Systemd has a socket activation feature where a mariadb.socket
definition defines the sockets to listen to, and passes those
file descriptors directly to mariadbd to use when a connection
occurs.
The new functionality is utilized when starting as follows:
systemctl start mariadb.socket
The mariadb.socket definition only needs to contain the network
information, ListenStream= directives, the mariadb.service
definition is still used for service instigation.
When mariadbd is started in this way, the socket, port, bind-address
backlog are all assumed to be self contained in the mariadb.socket
definition and as such the mariadb settings and command line
arguments of these network settings are ignored.
See man systemd.socket for how to limit this to specific ports.
Extra ports, those specified with extra_port in socket activation
mode, are those with a FileDescriptorName=extra. These need
to be in a separate service name like mariadb-extra.socket and
these require a Service={mariadb.service} directive to map to the
original service. Extra ports need systemd v227 or greater
(not RHEL/Centos7 - v219) when FileDescriptorName= was added,
otherwise the extra ports are treated like ordinary ports.
The number of sockets isn't limited when using systemd socket activation
(except by operating system limits on file descriptors and a minimal
amount of memory used per file descriptor). The systemd sockets passed
can include any ownership or permissions, including those the
mariadbd process wouldn't normally have the permission to create.
This implementation is compatible with mariadb.service definitions.
Those services started with:
systemctl start mariadb.service
does actually start the mariadb.service and used all the my.cnf
settings of sockets and ports like it previously did.
- use FIND_PACKAGE(LIBAIO) to find libaio
- Use standard CMake conventions in Find{PMEM,URING}.cmake
- Drop the LIB from LIB{PMEM,URING}_{INCLUDE_DIR,LIBRARIES}
It is cleaner, and consistent with how other packages are handled in CMake.
e.g successful FIND_PACKAGE(PMEM) now sets PMEM_FOUND, PMEM_LIBRARIES,
PMEM_INCLUDE_DIR, not LIBPMEM_{FOUND,LIBRARIES,INCLUDE_DIR}.
- Decrease the output. use FIND_PACKAGE with QUIET argument.
- for Linux packages, either liburing, or libaio is required
If liburing is installed, libaio does not need to be present .
Use FIND_PACKAGE([LIBAIO|URING] REQUIRED) if either library is required.
The new default values WITH_URING:BOOL=OFF, WITH_PMEM:BOOL=OFF imply
that the dependencies are optional.
An explicit request WITH_URING=ON or WITH_PMEM=ON will cause the
build to fail if the requested dependencies are not available.
Last, to prevent a feature to be built in even though the built-time
dependencies are available, the following can be used:
cmake -DCMAKE_DISABLE_FIND_PACKAGE_URING=1
cmake -DCMAKE_DISABLE_FIND_PACKAGE_PMEM=1
This cleanup was suggested by Vladislav Vaintroub.
std version has an advantage of a more convenient units implementation from
std::chrono. Now it's no need to multipy/divide to bring anything to
micro seconds.
WITH_PMEM: Change the parameter from BOOL to a ternary STRING
with a default value that allows to use libpmem when it is available.
In this way, a libpmem dependency will be automatically installed
on other than Debian-based systems, based on what is available on
the continuous integraton system for building the packages.
WITH_PMEM=auto (default): Use libpmem when available.
WITH_PMEM=yes: Require libpmem to be available.
WITH_PMEM=no: Do not depend on libpmem.
Updating the debian/control file will automatically update the dependencies
in all CI environments that directly read the debian/control file, such
as Salsa-CI and buildbot.mariadb.org to some degree.
(https://github.com/MariaDB/mariadb.org-tools/issues/43)
On Debian/Ubuntu releases that don't have liburing-dev available,
automatically downgrade to libaio-dev (just like libcurl4->3 is done).
This ensures the debian/control file is always up-to-date and works for
latest Debian and Ubuntu releases, while the backwards compatibility mods
are maintained in autobake-deb.sh separately, and can be dropped from there
once support for certain platforms end.
Debian/Ubuntu availability visible at:
- https://packages.debian.org/search?searchon=names&keywords=liburing-dev
- https://packages.ubuntu.com/search?searchon=names&keywords=liburing-dev
Also modify debian/rules to force a build without libaio. Use YES instead
of ON to make the flag more logical (=turning libaio check "off").
Stop running Salsa-CI for Debian Stretch-backports, as it does not have
liburing-dev available nor is the old-old Debian stable a relevant platform
for MariaDB 10.6 to test against anymore. Since the Stretch-backports build
can no longer be made, neither can the MySQL 5.7 on Bionic upgrade test be
run, as it depended on the Stretch binary.
This commit does not modify the .travis.yml file, as Travis-CI does not
have new enough Ubuntu releases available yet. Also Travis-CI.org is
practically dead now as build times have been shrunk to near zero.
The scope of this change is also Debian/Ubuntu only. No RPM or Windows or
Mac changes are included in this commit.
This commit does not update the external libmariadb or ColumnStore
CI pipelines, as those are maintained in different repositories.
liburing is a new optional dependency (WITH_URING=auto|yes|no)
that replaces libaio when it is available.
aio_uring: class which wraps io_uring stuff
aio_uring::bind()/unbind(): optional optimization
aio_uring::submit_io(): mutex prevents data race. liburing calls are
thread-unsafe. But if you look into it's implementation you'll see
atomic operations. They're used for synchronization between kernel and
user-space only. That's why our own synchronization is still needed.
For systemd, we add LimitMEMLOCK=524288 (ulimit -l 524288)
because the io_uring_setup system call that is invoked
by io_uring_queue_init() requests locked memory. The value
was found empirically; with 262144, we would occasionally
fail to enable io_uring when using the maximum values of
innodb_read_io_threads=64 and innodb_write_io_threads=64.
aio_uring::thread_routine(): Tolerate -EINTR return from
io_uring_wait_cqe(), because it may occur on shutdown
on Ubuntu 20.10 (Groovy Gorilla).
This was mostly implemented by Eugene Kosov. Systemd integration
and improved startup/shutdown error handling by Marko Mäkelä.
Introduced by 85828b8f22
This is running 2 git processes in parallel, which, if unlucky can cause
either of them to fail with "File already exists" error.
cmake has caught up and since version 3.18 it started supporting
CPACK_RPM_POST_TRANS_SCRIPT_FILE, something we've supported for
two years and cmake 2.8.11. Both implementation add %posttrans tag
and rpmbuild gets confused.
Disable our implementation for cmake 3.18+
It was only from CMake-3.14.0 that CMAKE_REQUIRED_LINK_OPTIONS
was used in CHECK_CXX_SOURCE_COMPILES. Without this, it could be
the case (as was on OSX) that a flag was never checked in
CHECK_CXX_SOURCE_COMPILES, the CHECK successfully passed, but
failed at link time.
As such we use CMAKE_REQUIRED_LIBRARIES to include the flags to check
as its compatible enough with the cmake versions for non-Windows
compilers/linkers.
Tested on x86_64 with:
* 3.11.4
* 3.17.4
Corrects: 7473e1841c
In the future:
* cmake >=3.14.0 can use CMAKE_REQUIRED_LINK_OPTIONS
* cmake >=3.18.0 can use CHECK_LINKER_FLAG (with policy CMP0057 NEW)
(e.g: commit c7ac2deff9a2c965887dcc67cbf2a3a7c3e0123d)
CMAKE_REQUIRED_LIBRARIES suggested by serg@mariadb.com
Reviewed-by: anel@mariadb.org
WORKING_DIRECTORY \$ENV{DESTDIR}\${CMAKE_INSTALL_PREFIX}
means that this path already exists, which is not necessarily the case
for make install. Create it `mkdir -p` style.
Largely based on MySQL commit
75271e51d6
MySQL Ref:
BUG#24566529: BACKPORT BUG#23575445 TO 5.6
(cut)
Also, the PTR_SANE macro which tries to check if a pointer
is invalid (used when printing pointer values in stack traces)
gave false negatives on OSX/FreeBSD. On these platforms we
now simply check if the pointer is non-null. This also removes
a sbrk() deprecation warning when building on OS X. (It was
before only disabled with building using XCode).
Removed execinfo path of MySQL patch that was already included.
sbrk doesn't exist on FreeBSD aarch64.
Removed HAVE_BSS_START based detection and replaced with __linux__
as it doesn't exist on OSX, Solaris or Windows. __bss_start
exists on mutiple Linux architectures.
Tested on FreeBSD and Linux x86_64. Being in FreeBSD ports for 2
years implies a good testing there on all FreeBSD architectures there
too. MySQL-8.0.21 code is functionally identical to original commit.
-Wl,-z,relro,-z,now are linker flags and should
be checked as such.
TODO: perform module, exe shared checks separately
rather than a pure linker check.
Cassandra has long been non-functional, doesn't compile, etc.,
and development on it has halted. Since MariaDB Server 10.5,
the build is disabled by default.
It makes sense to remove it as it's just taking up space.
* fix FindLZ4 to follow convention (LIBRARIES, not LIBRARY)
* remove redundant checks from rocksdb/CMakeLists.txt
* put all checks through the same macro that uniformly
checks for a package, prints the message, adds definition
The library finder needs to have capitals in its name so that FIND_PACKAGE
will load the correct finder and actually detect that libzstd is available.
Without this change the CMake would just always silently skip ZSTD since
it would never find it.
Simplify Debian autopkgtest RocksDB part and make it more verbose so that
future regressions like this are easier to debug.
Also remove QUIET from the RocksDB FIND_PACKAGE call so that it is easier
to read in build logs what libraries were detected. Also add missing
underscores to error messages.
On FreeBSD, perl isn't in /usr/bin, its in /usr/local/bin or
elsewhere in the path.
Like storage/{maria/unittest/,}ma_test_* , we use /usr/bin/env to
find perl and run it.
MRI scripts cannot handle + in paths, and ubuntu CI makes use of
these.
So we remove the top level build dir from the script and
transform it into a relative path script.
Because of common dependencies between the
static libraries list can contain duplicates.
We reduce these down to the single last one in
the list.
This reduces the relative time of a rebuild from:
$ (cd builddir/; time make -j)
...
real 0m30.789s
user 1m33.477s
sys 0m19.678s
and the LIB entries
$ grep ADDLIB builddir/libmysqld/mysqlserver-\$\<CONFIG\>.mri.tpl | wc -l
179
$ du -h builddir/libmysqld/libmariadbd.a
4.1G builddir/libmysqld/libmariadbd.a
To:
$ (cd builddir/; time make -j)
...
real 0m20.139s
user 1m32.423s
sys 0m12.208s
$ grep ADDLIB builddir/libmysqld/mysqlserver-\$\<CONFIG\>.mri.tpl | wc -l
25
$ du -h builddir/libmysqld/libmariadbd.a
688M builddir/libmysqld/libmariadbd.a
as cmake manual says
If a sequential execution of multiple commands is required, use multiple
``execute_process()`` calls with a single ``COMMAND`` argument.
This corrects build failures on ppc64{,le} with the
WITH_EMBEDDED_SERVER option enabled.
MDEV-22641 added an unusual case in which the same object
file in was included twice with a different function
defination. The original cmake/merge_archives_unix.cmake
did not tolerate such eventualities.
So we move to the highest voted answer on Stack Overflow
for the merging of static libraries.
https://stackoverflow.com/questions/3821916/how-to-merge-two-ar-static-libraries-into-one
Thin archives generated compile failures and the libtool
mechanism would of been another dependency and using .la
files that isn't part of a normal cmake output. The straight
Apple mechanism of libtool with static archives also failed
on Linux.
This leaves the MRI script mechansim which was implemented
in this change.
Existing implementation used my_checksum (from mysys)
for calculating table checksum and binlog checksum.
This implementation was optimized for powerpc only and lacked
SIMD implementation for x86 (using clmul) and ARM
(using ACLE) instead used zlib-crc32.
mariabackup had its own copy of the crc32 implementation
using hardware optimized implementation only for x86 and lagged
hardware based implementation for powerpc and ARM.
Patch helps unifies all such calls and help aggregate all of them
using an unified interface my_checksum().
Said unification also enables hardware optimized calls for all
architecture viz. x86, ARM, POWERPC.
Default always fallback to zlib crc32.
Thanks to Daniel Black for reviewing, fixing and testing
PowerPC changes. Thanks to Marko and Daniel for early code feedback.
The TokuDB storage engine has been deprecated by upstream
Percona Server 8.0 in favor of MyRocks and will not be available
in subsequent major upstream releases.
Let us remove it from MariaDB Server as well.
MyRocks is actively maintained, and it can be used instead.
setup alternative name for rhel8/centos8
use correct C/C version when providing mariadb-connector-c
and other changes for installation over distro packages to work:
dnf install mariadb-server
dnf install --allowerasing MariaDB-{server,common,client,shared}*.rpm
