The new default values WITH_URING:BOOL=OFF, WITH_PMEM:BOOL=OFF imply
that the dependencies are optional.
An explicit request WITH_URING=ON or WITH_PMEM=ON will cause the
build to fail if the requested dependencies are not available.
Last, to prevent a feature to be built in even though the built-time
dependencies are available, the following can be used:
cmake -DCMAKE_DISABLE_FIND_PACKAGE_URING=1
cmake -DCMAKE_DISABLE_FIND_PACKAGE_PMEM=1
This cleanup was suggested by Vladislav Vaintroub.
liburing is a new optional dependency (WITH_URING=auto|yes|no)
that replaces libaio when it is available.
aio_uring: class which wraps io_uring stuff
aio_uring::bind()/unbind(): optional optimization
aio_uring::submit_io(): mutex prevents data race. liburing calls are
thread-unsafe. But if you look into it's implementation you'll see
atomic operations. They're used for synchronization between kernel and
user-space only. That's why our own synchronization is still needed.
For systemd, we add LimitMEMLOCK=524288 (ulimit -l 524288)
because the io_uring_setup system call that is invoked
by io_uring_queue_init() requests locked memory. The value
was found empirically; with 262144, we would occasionally
fail to enable io_uring when using the maximum values of
innodb_read_io_threads=64 and innodb_write_io_threads=64.
aio_uring::thread_routine(): Tolerate -EINTR return from
io_uring_wait_cqe(), because it may occur on shutdown
on Ubuntu 20.10 (Groovy Gorilla).
This was mostly implemented by Eugene Kosov. Systemd integration
and improved startup/shutdown error handling by Marko Mäkelä.
The policy is not set for 10.2
If it is set, CMake would complain about bundled zlib for which the policy
is not set.
Fix:
- Set policy for 10.2 for the top level project.
For 10.3+ it was already set
- Cleanup zlib to remove unneeded stuff. It is an internal static library,
it needs none of PROJECT, library versioning, RC file on Windows.
The name of the library on Unix does not make any difference, since it is
static and compiled in.
-Wl,-z,relro,-z,now are linker flags and should
be checked as such.
TODO: perform module, exe shared checks separately
rather than a pure linker check.
Existing implementation used my_checksum (from mysys)
for calculating table checksum and binlog checksum.
This implementation was optimized for powerpc only and lacked
SIMD implementation for x86 (using clmul) and ARM
(using ACLE) instead used zlib-crc32.
mariabackup had its own copy of the crc32 implementation
using hardware optimized implementation only for x86 and lagged
hardware based implementation for powerpc and ARM.
Patch helps unifies all such calls and help aggregate all of them
using an unified interface my_checksum().
Said unification also enables hardware optimized calls for all
architecture viz. x86, ARM, POWERPC.
Default always fallback to zlib crc32.
Thanks to Daniel Black for reviewing, fixing and testing
PowerPC changes. Thanks to Marko and Daniel for early code feedback.
setup alternative name for rhel8/centos8
use correct C/C version when providing mariadb-connector-c
and other changes for installation over distro packages to work:
dnf install mariadb-server
dnf install --allowerasing MariaDB-{server,common,client,shared}*.rpm
Disable IPO (interprocedural optimization, aka /GL) on Windows
on libraries, from which server.dll exports symbols - exporting symbols
does not work for objects compiled with /GL.
This is a way do disable DBUG_ENTER()/DBUG_EXIT() stuff which is
needed to dbug trace. Those who doesn't need it may avoid tests
slowdown with -DWITH_DBUG_TRACE=OFF
dbug/tests.c: add define which is neede always in this test
innodb.log_file_name_debug.test: do not depend on DBUG trace stuff
in test
Benchmark results: each test eats less CPU and you can have more
parallel jobs in MTR.
patched:
./mtr -mem -par=8 -suite=innodb 185.34s user 86.85s system 133% cpu 3:23.27 total
./mtr -mem -par=8 -suite=main 80.96s user 36.01s system 182% cpu 1:04.07 total
main.select [ pass ] 1660
main.select [ pass ] 1513
main.select [ pass ] 1543
main.select [ pass ] 1660
main.select [ pass ] 1521
main.select [ pass ] 1511
main.select [ pass ] 1508
main.select [ pass ] 1520
main.select [ pass ] 1514
main.select [ pass ] 1522
vanilla:
./mtr -mem -par=8 -suite=innodb 203.61s user 92.16s system 140% cpu 3:30.16 total
./mtr -mem -par=8 -suite=main 94.11s user 35.51s system 206% cpu 1:02.69 total
main.select [ pass ] 2032
main.select [ pass ] 2017
main.select [ pass ] 2040
main.select [ pass ] 2183
main.select [ pass ] 2253
main.select [ pass ] 2075
main.select [ pass ] 2109
main.select [ pass ] 2080
main.select [ pass ] 2098
main.select [ pass ] 2114
This task deals with packing the sort key inside the sort buffer, which would
lead to efficient usage of the memory allocated for the sort buffer.
The changes brought by this feature are
1) Sort buffers would have sort keys of variable length
2) The format for sort keys inside the sort buffer would look like
|<sort_length><null_byte><key_part1><null_byte><key_part2>.......|
sort_length is the extra bytes that are required to store the variable
length of a sort key.
3) When packing of sort key is done we store the ORIGINAL VALUES inside
the sort buffer and not the STRXFRM form (mem-comparable sort keys).
4) Special comparison function packed_keys_comparison() is introduced
to compare 2 sort keys.
This patch also contains contributions from Sergei Petrunia.
Cleanup install_layout to account for multi-arch setup and remove
redundant defines in debian rules.
Signed-off-by: Vicențiu Ciorbaru <vicentiu@mariadb.org>
Also added support for MAP_SYNC. It allows to achieve decent performance
with DAX devices even when libpmem is unavailable.
Fixed Windows version of my_msync(): according to manual FlushViewOfFile()
may return before flush is actually completed. It is advised to issue
FlushFileBuffers() after FlushViewOfFile().
The library is capable of
- asynchronous execution of tasks (and optionally waiting for them)
- asynchronous file IO
This is implemented using libaio on Linux and completion ports on
Windows. Elsewhere, async io is "simulated", which means worker threads
are performing synchronous IO.
- timers, scheduling work asynchronously in some point of the future.
Also periodic timers are implemented.
The MDEV-20265 commit e746f451d5
introduces DBUG_ASSERT(right_op == r_tbl) in
st_select_lex::add_cross_joined_table(), and that assertion would
fail in several tests that exercise joins. That commit was skipped
in this merge, and a separate fix of MDEV-20265 will be necessary in 10.4.
MemorySanitizer is a compile-time instrumentation layer in clang and GCC.
Together with AddressSanitizer mostly makes the run-time instrumentation
of Valgrind redundant. It is a little more tricky to set up, because
running with uninstrumented libraries will lead into false positives.
You will need an instrumented libc++, and you should use
-stdlib=libc++ instead of the default libstdc++. To build the
instrumented library, you can refer to
https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo
or you can adapt these steps that worked for me, for clang-8 version 8.0.1:
cd /mariadb
sudo apt source libc++-8-dev
cd llvm-toolchain-8-8.0.1
mkdir libc++msan; cd libc++msan
cmake ../libcxx -DCMAKE_BUILD_TYPE=Release -DLLVM_USE_SANITIZER=Memory \
-DCMAKE_C_COMPILER=clang-8 -DCMAKE_CXX_COMPILER=clang++-8
Then, in your MariaDB build directory, you have to compile with
libc++ and bundled libraries, such as WITH_SSL=bundled, WITH_ZLIB=bundled.
For uninstrumented system libraries, you will get false positives for
uninitialized values. Like this:
cmake -DWITH_MSAN=ON -DWITH_SSL=bundled -DWITH_ZLIB=bundled \
-DCMAKE_CXX_FLAGS='-stdlib=libc++' ..
Note: you should also add -O2 to the compiler options, or you may
get crashes due to stack overflow.
Finally, to run tests, you must replace libc++ with the instrumented one:
LD_LIBRARY_PATH=/mariadb/llvm-toolchain-8-8.0.1/libc++msan/lib \
MSAN_OPTIONS=abort_on_error=1 \
./mtr --big-test --parallel=auto --force --retry=0
Failure to do so will report numerous false positives related to
operations on std::string and the like.
This is work in progress. Some issues will still have to be fixed
for WITH_MSAN to be usable. See MDEV-20377 for details.
Those two may work incorrectly together. Namely, ASAN may produce
false positives or false negatives. For details see
https://github.com/google/sanitizers/wiki/AddressSanitizer#faq
Make SECURITY_HARDENED disabled by default if WITH_ASAN=ON
Based on contribution by Eugene Kosov.
- Add new submodule for WolfSSL
- Build and use wolfssl and wolfcrypt instead of yassl/taocrypt
- Use HAVE_WOLFSSL instead of HAVE_YASSL
- Increase MY_AES_CTX_SIZE, to avoid compile time asserts in my_crypt.cc
(sizeof(EVP_CIPHER_CTX) is larger on WolfSSL)
With MAX_INDEXIES=64(default), key_map=Bitmap<64> is just a wrapper around
ulonglong and thus "trivial" (can be bzero-ed, or memcpy-ed, and stays
valid still)
With MAX_INDEXES=128, key_map = Bitmap<128> is not a "trivial" type
anymore. The implementation uses MY_BITMAP, and MY_BITMAP contains pointers
which make Bitmap invalid, when it is memcpy-ed/bzero-ed.
The problem in 10.4 is that there are many new key_map members, inside TABLE
or KEY, and those are often memcopied and bzeroed
The fix makes Bitmap "trivial", by inlining most of MY_BITMAP functionality.
pointers/heap allocations are not used anymore.
GitHub uses a library called Licensee to identify a project's license
type. It shows this information in the status bar and via the API if it
can unambiguously identify the license.
This commit modifies a few of MariaDB's docs so that Licensee is able
to recognize the repository's license type. It renames the README's
"License" section to "Licensing" and renames COPYING.thirdparty to
THIRDPARTY.
These changes allow Licensee to bypass both files when it
scans the repo for license files, which thus allows Licensee to
successfully identify the license type of MariaDB as GPL 2.0.
Signed-off-by: Andrea Kao <eirinikos@gmail.com>
automatic BuildRequires for source RPM: for every FILEPATH and
"Have library XXX" cached variable, detect what rpm package it comes from
and add it to the list of dependencies.
That is, the source RPM will BuildRequire all those packages that
were found by cmake when the source RPM was built. Presumably, our
CMakeLists.txt won't check for libraries that aren't needed for a build.
It supports libraries/executables/files found with
FIND_LIBRARY
FIND_FILE
FIND_PROGRAM
CHECK_LIBRARY_EXISTS
Disable LOAD DATA LOCAL INFILE suport by default and
auto-enable it for the duration of one query, if the query
string starts with the word "load". In all other cases the application
should enable LOAD DATA LOCAL INFILE support explicitly.
