Commit graph

190150 commits

Author SHA1 Message Date
Marko Mäkelä
79c166c56d Fix cmake -DWITH_MSAN=ON
commit ab4069909d accidentally
misplaced #endif directives.
2020-07-05 20:38:50 +03:00
Marko Mäkelä
ab4069909d After-merge fix for ASAN and MSAN
The merge commit 0fd89a1a89
of commit b6ec1e8bbf
seems to cause occasional MemorySanitizer failures,
because it failed to replace some MEM_UNDEFINED() calls
with MEM_MAKE_ADDRESSABLE().

my_large_free(): Correctly invoke MEM_MAKE_ADDRESSABLE() after
freeing memory. Failure to do so could cause bogus
AddressSanitizer failures for memory allocated by my_large_malloc().
On MemorySanitizer, we will do nothing.

buf_pool_t::chunk_t::create(): Replace the MEM_MAKE_ADDRESSABLE()
that had been added in commit 484931325e
to work around the issue.
2020-07-05 16:25:29 +03:00
Marko Mäkelä
90d5d90640 Merge 10.4 into 10.5 2020-07-04 22:19:49 +03:00
Marko Mäkelä
b99fa1e767 Merge 10.3 into 10.4 2020-07-04 22:11:16 +03:00
Marko Mäkelä
453dc4b300 Fixup the parent commit for MSAN and Valgrind
commit 484931325e was a necessary
fix for the buffer pool resizing tests in 10.5 in
AddressSanitizer. However, that change would break the tests
innodb.innodb_buffer_pool_resize and
innodb.innodb_buffer_pool_resize_with_chunks
when run in MemorySanitizer, or presumably in Valgrind as well.
(Those tests run "forever" in Valgrind.)

buf_pool_resize(): Cancel the effect of MEM_NOACCESS() in Valgrind
and ASAN. In MSAN, MEM_NOACCESS() is a no-op, and hence we must do
nothing special here.

MEM_MAKE_ADDRESSABLE() would declare the memory contents undefined.
In this particular case, we must actually declare the contents
defined for Valgrind.
2020-07-04 22:02:30 +03:00
Marko Mäkelä
a85f81af03 MDEV-22535 fixup: Define a single-caller function inline
Let us avoid any overhead in release builds, for an empty function.
2020-07-04 14:28:11 +03:00
Marko Mäkelä
2d00e003b2 After-merge fixes for ASAN
The merge commit 0fd89a1a89
of commit b6ec1e8bbf
was slightly incomplete.

ReadView::mem_valid(): Use the correct primitive
MEM_MAKE_ADDRESSABLE(), because MEM_UNDEFINED() now has
no effect on ASAN.

recv_sys_t::alloc(), recv_sys_t::add(): Use MEM_MAKE_ADDRESSABLE()
instead of MEM_UNDEFINED(), to get the correct behaviour for ASAN.
For Valgrind and MSAN, there is no change in behaviour.

recv_sys_t::free(), recv_sys_t::clear(): Before freeing memory to
buf_pool.free_list, invoke MEM_MAKE_ADDRESSABLE() on the entire
buf_block_t::frame, to cancel the effect of MEM_NOACCESS() in
recv_sys_t::alloc().
2020-07-04 14:28:11 +03:00
Sergei Golubchik
478591608e optimize ha_delete_table_force
first try discovering engines, then the rest.
otherwise every DROP TABLE non_existent; will do
lots of i/o trying to remove .MYI/.MYD/.MAI/.MAD/.CSV/etc files

this matches the old behavior where DROP TABLE always tried to discover
the table before dropping.
2020-07-04 01:44:47 +02:00
Sergei Golubchik
4227dd2ac6 continue DROP TEMPORARY TABLE t1, t2, t3 after error.
normal DROP TABLE with many tables continues after an error,
trying to drop as many tables as possible. But DROP TEMPORARY TABLE
was aborting on the first error. Change it to behave as DROP TABLE does.
2020-07-04 01:44:47 +02:00
Sergei Golubchik
6c52931680 replace HTON_AUTOMATIC_DELETE_TABLE with return -1 from drop_table() 2020-07-04 01:44:47 +02:00
Sergei Golubchik
4876651e0f remove mysql_declare_plugin declaration from some plugins 2020-07-04 01:44:47 +02:00
Sergei Golubchik
7c2ba9e9d7 MDEV-11412 Ensure that table is truly dropped when using DROP TABLE
don't do table discovery on DROP. DROP falls back to "force"
approach when a table isn't found and will try to drop in all
engines anyway. That is, trying to discover in all engines before
the drop is redundant and may be expensive.
2020-07-04 01:44:47 +02:00
Sergei Golubchik
79a3f96166 rewrite bug#26704 test case
drop-no_root needs DROP DATABASE to fail.
But `chmod 000` is not reliable after drop table force anymore.

Make DROP DATABASE to fail by creating an extra file in the db dir
2020-07-04 01:44:47 +02:00
Sergei Golubchik
529b6dffe9 small cleanup 2020-07-04 01:44:46 +02:00
Sergei Golubchik
35f566db8d cleanup: make dd_frm_type to work as documented
remove redundant argument, return all possible enum values
2020-07-04 01:44:46 +02:00
Sergei Golubchik
2bb5981c20 MDEV-11412 Ensure that table is truly dropped when using DROP TABLE
minor post-review fixes
* remove duplicate tests
* first file indicates table existance even with discovery
* don't duplicate trigger dropping code
2020-07-04 01:44:46 +02:00
Sergei Golubchik
b014720b6c optimization: use hton->drop_table in few simple cases 2020-07-04 01:44:46 +02:00
Sergei Golubchik
c55c292832 introduce hton->drop_table() method
first step in moving drop table out of the handler.
todo: other methods that don't need an open table

for now hton->drop_table is optional, for backward compatibility
reasons
2020-07-04 01:44:46 +02:00
Sergei Golubchik
f17f7a43ba test dropping of a MEMORY table 2020-07-04 01:44:46 +02:00
Sergei Golubchik
d2b852b4ca cleanup, less #ifdef's 2020-07-04 01:44:46 +02:00
Monty
7d75e43261 Disable rpl_parallel2 temporarly until we have a proper fix for it in 10.5 2020-07-04 02:19:02 +03:00
Monty
0fd89a1a89 Merge remote-tracking branch 'origin/10.4' into 10.5 2020-07-03 23:31:12 +03:00
Qi Wu
70684afef2 Atomic write support for ScaleFlux NVMe SSD's 2020-07-03 22:42:10 +03:00
Monty
e9f06b19e0 Merge remote-tracking branch 'origin/10.3' into 10.4 2020-07-03 20:26:09 +03:00
Monty
484931325e Fix for MSAN from Marko related to buf_pool_resize 2020-07-03 18:37:33 +03:00
Julius Goryavsky
1bf863a91a Merge branch '10.4-MDEV-22222' of https://github.com/codership/mariadb-server into 10.4-MDEV-22222 2020-07-03 16:17:59 +02:00
Sergei Golubchik
7a4afad969 compilation fix
include/my_valgrind.h:88:112: error: ‘void* memset(void*, int, size_t)’ writing to an object of non-trivial type ‘key_map’ {aka ‘class Bitmap<64>’}; use assignment instead [-Werror=class-memaccess]

in this case it's safe, Bitmap<> is trivial enough
2020-07-03 15:01:21 +02:00
Monty
e6595a06d6 Don't give errors for default value copy in create_tmp_table 2020-07-03 01:18:51 +03:00
Monty
6e81ba0c12 Don't give errors for default value copy in create_tmp_table 2020-07-03 01:16:31 +03:00
Monty
5211af1c16 Merge remote-tracking branch 'origin/10.3' into 10.4 2020-07-03 00:35:28 +03:00
Monty
53ecc354e3 Fixed errors found by MSAN 2020-07-02 23:50:56 +03:00
Marko Mäkelä
b6ec1e8bbf MDEV-20377 post-fix: Introduce MEM_MAKE_ADDRESSABLE
In AddressSanitizer, we only want memory poisoning to happen
in connection with custom memory allocation or freeing.

The primary use of MEM_UNDEFINED is for declaring memory uninitialized
in Valgrind or MemorySanitizer. We do not want MEM_UNDEFINED to
have the unwanted side effect that AddressSanitizer would no longer
be able to complain about accessing unallocated memory.

MEM_UNDEFINED(): Define as no-op for AddressSanitizer.

MEM_MAKE_ADDRESSABLE(): Define as MEM_UNDEFINED() or
ASAN_UNPOISON_MEMORY_REGION().

MEM_CHECK_ADDRESSABLE(): Wrap also __asan_region_is_poisoned().
2020-07-02 17:59:28 +03:00
Monty
65f831d17c Fixed bugs found by valgrind
- Some of the bug fixes are backports from 10.5!
- The fix in innobase/fil/fil0fil.cc is just a backport to get less
  error messages in mysqld.1.err when running with valgrind.
- Renamed HAVE_valgrind_or_MSAN to HAVE_valgrind
2020-07-02 17:57:34 +03:00
Monty
6cee9b1953 MDEV-22535 TABLE::initialize_quick_structures() takes 0.5% in oltp_read_only
Fixed by:
- Make all quick_* variable allocated according to real number keys instead
  of MAX_KEY
- Store all the quick* items in separated allocated structure (OPT_RANGE)
- Ensure we don't access any quick* variable without first checking
  opt_range_keys.is_set().  Thanks to this, we don't need any
  pre-initialization of quick* variables anymore.

Some renames was done to use the new structure:
table->quick_keys                -> table->opt_range_keys
table->quick_rows[X]             -> table->opt_range[X].rows
table->quick_key_parts[X]        -> table->opt_range[X].key_parts
table->quick_costs[X]            -> table->opt_range[X].cost
table->quick_index_only_costs[X] -> table->opt_range[X].index_only_cost
table->quick_n_ranges[X]         -> table->opt_range[X].ranges
table->quick_condition_rows      -> table->opt_range_condition_rows

This patch should both decrease memory needed for TABLE objects
(3528 -> 984 + keyinfo) and increase performance, thanks to less
initializations per query, and more localized memory, thanks to the
opt_range structure.
2020-07-02 16:59:14 +03:00
Monty
5cbb18cb44 Fixed typo in InnoDB when compiling with VALGRIND 2020-07-02 16:59:04 +03:00
Monty
29f9e679ad Don't copy uninitialized bytes when copying varstrings
When using field_conv(), which is called in case of field1=field2 copy in
fill_records(), full varstring's was copied, including unitialized bytes.
This caused valgrind to compilain about usage of unitialized bytes when
using Aria static length records.
Fixed by not using memcpy when copying varstrings but instead just copy
the real bytes.
2020-07-02 14:25:41 +03:00
Monty
3f2044ae99 MDEV-22535 TABLE::initialize_quick_structures() takes 0.5% in oltp_read_only
- Removed not needed bzero in void TABLE::initialize_quick_structures().
- Replaced bzero with TRASH_ALLOC() to have this change verfied with
  memory checkers
- Added missing table->quick_keys.is_set in table_cond_selectivity()
2020-07-02 14:25:41 +03:00
Marko Mäkelä
1813d92d0c Merge 10.4 into 10.5 2020-07-02 09:41:44 +03:00
Marko Mäkelä
0fe97c20b3 MDEV-23017: Regression due to unwanted read-ahead
MDEV-15053 inadvertently inverted the condition for invoking
buf_read_ahead_linear().

buf_page_get_low(): Apply the correct condition.

buf_page_optimistic_get(): Remove the read-ahead altogether.
This function should never access a block for the first time.
2020-07-02 07:45:30 +03:00
Marko Mäkelä
f347b3e0e6 Merge 10.3 into 10.4 2020-07-02 07:39:33 +03:00
Marko Mäkelä
1df1a63924 Merge 10.2 into 10.3 2020-07-02 06:17:51 +03:00
Marko Mäkelä
dba7e1e8e1 Merge 10.1 into 10.2 2020-07-02 06:05:13 +03:00
Marko Mäkelä
c43a666662 Revert "Fix result of merge."
This reverts commit e0793d3865.

In idiomatic C++, accessor functions should not discard qualifiers.
2020-07-02 06:04:42 +03:00
Marko Mäkelä
90d1e58ed0 MDEV-22941: Fix the DBUG_ENTER name 2020-07-02 06:04:31 +03:00
Marko Mäkelä
838a1046b2 MDEV-20377: Fix cmake -DPLUGIN_PERFSCHEMA=NO 2020-07-02 06:03:59 +03:00
Oleksandr Byelkin
b0f836053b MDEV-22983: Mariabackup's --help option disappeared
return --help option
2020-07-01 23:30:44 +03:00
Marko Mäkelä
69df4f834b MDEV-20377: Fix -Wunused-but-set-variable 2020-07-01 20:34:06 +03:00
Vladislav Vaintroub
41b0d98e69 MDEV-23067 Windows : manually registered services rejected mysql_upgrade_service
- service not using "--defaults-file" can have any name not just "MySQL"
- service with "--defaults-file", without datadir in them
use default datadir (install_root\data)
2020-07-01 18:43:21 +02:00
Marko Mäkelä
be51738465 MDEV-20428 after-merge fix: Stabilize the test 2020-07-01 17:43:44 +03:00
Marko Mäkelä
c36834c832 MDEV-20377: Make WITH_MSAN more usable
MemorySanitizer (clang -fsanitize=memory) requires that all code
be compiled with instrumentation enabled. The only exception is the
C runtime library. Failure to use instrumented libraries will cause
bogus messages about memory being uninitialized.

In WITH_MSAN builds, we must avoid calling getservbyname(),
because even though it is a standard library function, it is
not instrumented, not even in clang 10.

Note: Before MariaDB Server 10.5, ./mtr will typically fail
due to the old PCRE library, which was updated in MDEV-14024.

The following cmake options were tested on 10.5
in commit 94d0bb4dbe:

cmake \
-DCMAKE_C_FLAGS='-march=native -O2' \
-DCMAKE_CXX_FLAGS='-stdlib=libc++ -march=native -O2' \
-DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \
-DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \
-DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO \
-DWITH_SAFEMALLOC=OFF \
-DWITH_{ZLIB,SSL,PCRE}=bundled \
-DHAVE_LIBAIO_H=0 \
-DWITH_MSAN=ON

MEM_MAKE_DEFINED(): An alias for VALGRIND_MAKE_MEM_DEFINED()
and __msan_unpoison().

MEM_GET_VBITS(), MEM_SET_VBITS(): Aliases for
VALGRIND_GET_VBITS(), VALGRIND_SET_VBITS(), __msan_copy_shadow().

InnoDB: Replace the UNIV_MEM_ macros with corresponding MEM_ macros.

ut_crc32_8_hw(), ut_crc32_64_low_hw(): Use the compiler built-in
functions instead of inline assembler when building WITH_MSAN.
This will require at least -msse4.2 when building for IA-32 or AMD64.
The inline assembler would not be instrumented, and would thus cause
bogus failures.
2020-07-01 17:23:00 +03:00