Commit graph

181343 commits

Author SHA1 Message Date
Marko Mäkelä
d4187bdc51 Replace purge_iter_t with purge_sys_t::iterator
Also, remove the field undo_rseg_space.
Apparently its purpose was to avoid problems with
temporary undo logs, which MySQL 5.7 unnecessarily adds to
the purge system. (Temporary undo log records are not purged.)
MariaDB 10.2 fixed this in MDEV-12289 or earlier.
2018-02-21 19:16:47 +02:00
Marko Mäkelä
28d844fd07 Rename the purge_sys_t iterators
purge_iter_t::operator<=(): Ordering comparison.
This replaces trx_purge_check_limit() with the difference that
we are not comparing undo_rseg_space. (In MariaDB, temporary
undo logs do not enter the purge subsystem at all.)

purge_sys_t::done: Remove. This was not used for anything.

purge_sys_t::tail: Renamed from purge_sys_t::iter.

purge_sys_t::head: Renamed from purge_sys_t::limit.
2018-02-21 19:16:47 +02:00
Marko Mäkelä
7bfe33ee28 MDEV-10814 innodb large allocations - Don't dump
Merge pull request #364
2018-02-21 19:15:20 +02:00
Daniel Black
8f50a26eb7 MDEV-15356: tp_timeout_handler needs to call set_killed_no_mutex as it has the mutex
Regression introducted in c2118a08b1 where LOCK_thd_data was moveed
to LOCK_thd_kill
2018-02-21 18:07:43 +01:00
Vladislav Vaintroub
e2ac8d3ff1 Windows : Fix /permissive- compile error 2018-02-21 16:33:55 +00:00
Marko Mäkelä
094cf73045 Avoid some dead code 2018-02-21 09:46:51 +02:00
Vladislav Vaintroub
56e7b7eaed Make possible to use clang on Windows (clang-cl)
-DWITH_ASAN can be used as well now, on x64

Fix many clang-cl warnings.
2018-02-20 21:17:36 +00:00
Galina Shalygina
9d97e6010e MDEV-14835 Server crashes in Field_iterator_table::create_item when number of
elements of BIGINT or YEAR type in the IN list reaches in_predicate_conversion_threshold

The bug appears at the prepare stage when IN-predicate with the long list
of values is converted into IN-subquery. It happens because values in the
right operand of the IN-predicate that have BIGINT or YEAR types are converted
into the Item_int_with_ref.
To fix it in the procedure Item_func_in::create_value_list_for_tvc
real_item() is taken for each value in the right operand of the IN-predicate.
2018-02-20 21:56:52 +02:00
Marko Mäkelä
947efe17ed MDEV-15158 On commit, do not write to the TRX_SYS page
This is based on a prototype by
Thirunarayanan Balathandayuthapani <thiru@mariadb.com>.

Binlog and Galera write-set replication information was written into
TRX_SYS page on each commit. Instead of writing to the TRX_SYS during
normal operation, InnoDB can make use of rollback segment header pages,
which are already being written to during a commit.

The following list of fields in rollback segment header page are added:
    TRX_RSEG_BINLOG_OFFSET
    TRX_RSEG_BINLOG_NAME (NUL-terminated; empty name = not present)
    TRX_RSEG_WSREP_XID_FORMAT (0=not present; 1=present)
    TRX_RSEG_WSREP_XID_GTRID
    TRX_RSEG_WSREP_XID_BQUAL
    TRX_RSEG_WSREP_XID_DATA

trx_sys_t: Introduce the fields
recovered_binlog_filename, recovered_binlog_offset, recovered_wsrep_xid.

To facilitate upgrade from older mysql or mariaDB versions, we will read
the information in TRX_SYS page. It will be overridden by the
information that we find in rollback segment header pages.

Mariabackup --prepare will read the metadata from the rollback
segment header pages via trx_rseg_array_init(). It will still
not read any undo log pages or recover any transactions.
2018-02-20 21:36:36 +02:00
Marko Mäkelä
f6544353e0 Pull request #625: ASAN unpoison system versioning related buffer 2018-02-20 15:08:16 +02:00
Eugene Kosov
d4822bef04 remove uint 2018-02-20 15:09:31 +03:00
Marko Mäkelä
cd63f43c40 Fix the Windows build
my_atomic_load32() expects a pointer to non-const on Windows.
2018-02-20 13:08:43 +02:00
Eugene Kosov
8ed9ef54f9 ASAN unpoison system versioning related buffer
Fixes this report:
==3165==ERROR: AddressSanitizer: use-after-poison on address 0x61e0000270a0 at pc 0x00000114b78c bp 0x7f15d65fe120 sp 0x7f15d65fd8d0
WRITE of size 1366 at 0x61e0000270a0 thread T28
    #0 0x114b78b in __asan_memcpy fun/cpp_projects/llvm_toolchain/llvm/projects/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cc:23
    #1 0x208208d in TABLE::init(THD*, TABLE_LIST*) work/mariadb/sql/table.cc:4662:3
    #2 0x19df85b in open_table(THD*, TABLE_LIST*, Open_table_context*) work/mariadb/sql/sql_base.cc:1993:10
    #3 0x19eb968 in open_and_process_table(THD*, LEX*, TABLE_LIST*, unsigned int*, unsigned int, Prelocking_strategy*, bool, Open_table_context*) work/mariadb/sql/sql_base.cc:3483:14
    #4 0x19e7c05 in open_tables(THD*, DDL_options_st const&, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*) work/mariadb/sql/sql_base.cc:4001:14
    #5 0x19f4dac in open_and_lock_tables(THD*, DDL_options_st const&, TABLE_LIST*, bool, unsigned int, Prelocking_strategy*) work/mariadb/sql/sql_base.cc:4879:7
    #6 0x1627263 in open_and_lock_tables(THD*, TABLE_LIST*, bool, unsigned int) work/mariadb/sql/sql_base.h:487:10
    #7 0x1c3839c in mysql_execute_command(THD*) work/mariadb/sql/sql_parse.cc:5113:13
    #8 0x1c1b72c in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) work/mariadb/sql/sql_parse.cc:7980:18
    #9 0x1c13464 in handle_bootstrap_impl(THD*) work/mariadb/sql/sql_parse.cc:1044:5
    #10 0x1c11ff7 in do_handle_bootstrap(THD*) work/mariadb/sql/sql_parse.cc:1096:3
    #11 0x1c11d14 in handle_bootstrap work/mariadb/sql/sql_parse.cc:1079:3
    #12 0x115a6ae in __asan::AsanThread::ThreadStart(unsigned long, __sanitizer::atomic_uintptr_t*) fun/cpp_projects/llvm_toolchain/llvm/projects/compiler-rt/lib/asan/asan_thread.cc:259
    #13 0x7f15fe1407fb in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x77fb)
    #14 0x7f15fbb64b5e in clone /build/glibc-itYbWN/glibc-2.26/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
2018-02-20 13:50:23 +03:00
Marko Mäkelä
1fa14a7c08 Replace trx_undo_mem_free() with ut_free() 2018-02-20 10:30:14 +02:00
Marko Mäkelä
60a68fdf71 Clarify the access to trx_sys.rseg_history_len
trx_sys_t::rseg_history_len: Make private, and clarify the
documentation.

trx_sys_t::history_size(): Read rseg_history_len.

trx_sys_t::history_insert(), trx_sys_t::history_remove(),
trx_sys_t::history_add(): Update rseg_history_len.
2018-02-20 10:30:08 +02:00
Alexey Botchkov
d23fcc427c MDEV-14593 human-readable XA RECOVER.
sql_yacc_ora.yy fixed.
2018-02-19 23:29:51 +04:00
Marko Mäkelä
5521994ce2 Pull request #614: various small code changes 2018-02-19 11:37:45 +02:00
Marko Mäkelä
2ba487cfe8 Merge bb-10.2-ext into 10.3 2018-02-19 11:37:29 +02:00
Monty
ef3147b1d6 Fix for MDEV-15105 (memory loss with LOCK sequence)
MDEV 15105 "Bytes lost and Assertion `global_status_var.global_memory_used == 0' fails on shutdown after attempts
to LOCK/RENAME sequence"
2018-02-19 11:27:21 +02:00
Monty
06ba07c269 Test case for MDEV-12887 (bug fixed long ago)
MDEV-12887 UT_LIST_GET_LEN(trx->lock.trx_locks) == 0 when mysqldump
sequence
2018-02-19 11:26:25 +02:00
Marko Mäkelä
278c036275 Merge 10.2 into bb-10.2-ext 2018-02-19 09:01:06 +02:00
Marko Mäkelä
112cb56182 Add suppressions for background page read errors 2018-02-19 08:59:36 +02:00
Marko Mäkelä
3c419fde5f Cleanup after commit ac8e3c85a4
srv_conc_t::n_active: Correct the comment, and remove an
assertion that trivially holds now that the type is unsigned.
2018-02-19 08:58:22 +02:00
Vladislav Vaintroub
acab33a1f2 Merge branch '10.2-backup-fixes' into 10.2 2018-02-18 10:49:46 +00:00
Vladislav Vaintroub
e917188f99 Merge branch 'bb-10.2-wlad' into 10.2-backup-fixes 2018-02-18 10:47:23 +00:00
Alexander Barkov
fed51b80fb Adding "const" qualifier to the MYSQL_TIME parameter of Item_temporal_literal constructors 2018-02-17 19:16:56 +04:00
Marko Mäkelä
970ce270c9 Merge 10.1 into 10.2
Disable the test encryption.innodb_encryption-page-compression
because the wait_condition would seem to time out deterministically.
MDEV-14814 has to be addressed in 10.2 separately.

Datafile::validate_first_page(): Do not invoke
page_size_t::page_size_t(flags) before validating the tablespace flags.
This avoids a crash in MDEV-15333 innodb.restart test case.
FIXME: Reduce the number of error messages. The first one is enough.
2018-02-17 14:54:12 +02:00
Marko Mäkelä
9a46d97149 MDEV-15333 MariaDB (still) slow start
This performance regression was introduced in the MariaDB 10.1
file format incompatibility bug fix MDEV-11623 (MariaDB 10.1.21
and MariaDB 10.2.4) and partially fixed in MariaDB 10.1.25 in
MDEV-12610 without adding a regression test case.

On a normal startup (without crash recovery), InnoDB should not read
every .ibd data file, because this is slow. Like in MySQL, for now,
InnoDB will still open every data file (without reading), and it
will read every .ibd file for which an .isl file exists, or the
DATA DIRECTORY attribute has been specified for the table.

The test case shuts down InnoDB, moves data files, replaces them
with garbage, and then restarts InnoDB, expecting no messages to
be issued for the garbage files. (Some messages will for now be
issued for the table that uses the DATA DIRECTORY attribute.)
Finally, the test shuts down the server, restores the old data files,
and restarts again to drop the tables.

fil_open_single_table_tablespace(): Remove the condition on flags,
and only call fsp_flags_try_adjust() if validate==true
(reading the first page has been requested). The only caller with
validate==false is at server startup when we are processing all
records from SYS_TABLES. The flags passed to this function are
actually derived from SYS_TABLES.TYPE and SYS_TABLES.N_COLS,
and there never was any problem with SYS_TABLES in MariaDB 10.1.
The problem that MDEV-11623 was that incorrect tablespace flags
were computed and written to FSP_SPACE_FLAGS.
2018-02-17 14:20:33 +02:00
Daniel Black
b600f30786 MDEV-10814: Innodb large allocations - madvise - Don't dump
Note: Linux only

Core dumps of large buffer pool pages take time and space
and pose potential data expose in scenarios where data-at-rest
encryption is deployed.

Here we use madvise(MADV_DONT_DUMP) on large memory allocations
used by the innodb buffer pool, log_sys and recv_sys. The effect
of this system call is that these memory areas will not appear in
a core dump. Data from these buffers is rarely useful in fault
diagnosis.

log_sys and recv_sys structures now use large memory allocations
for their large buffer.

Debug builds don't include the madvise syscall and as such will
include full core dumps.

A function, buf_madvise_do_dump, is added but never called. It
is there to be called from a debugger to re-enable the core
dumping of all of these pages if for some reason the entire
contents of these buffers are needed.

Idea thanks to Hartmut Holzgraefe
2018-02-17 20:00:56 +11:00
Alexander Kuleshov
afc56a509c merge two the same consistently 'if' clauses into one
we have two the same 'if' clauses that check opt_bin_log argument
in mysqld.cc and both clauses go successively one after another.
Let's merge them into one
2018-02-17 12:46:16 +04:00
Eugene Kosov
f02f1eda7e review fixes 2018-02-16 22:15:51 +03:00
Marko Mäkelä
8bf2c08d54 Add a suppression for background page read error 2018-02-16 21:02:35 +02:00
Eugene Kosov
6de8f79b11 remove unneeded variable 2018-02-16 21:44:51 +03:00
Eugene Kosov
de4c9f460c change some ibool to bool 2018-02-16 21:44:51 +03:00
Eugene Kosov
e14790b89d let buf_page_hash_lock_get() be function, not macro 2018-02-16 21:44:51 +03:00
Eugene Kosov
365f478240 make buf_block_t::lock_hash_val uint32_t 2018-02-16 21:44:51 +03:00
Eugene Kosov
af2d260b37 merge btr_page_get_level_low() and btr_page_get_level() 2018-02-16 21:44:51 +03:00
Eugene Kosov
9d46bd8a24 flst_add_to_empty(): read len only in debug build 2018-02-16 21:44:50 +03:00
Eugene Kosov
aaf71116ee remove unused stuff:
que_cur_t
que_fork_t::cur_end
que_fork_t::cur_pos
que_fork_t::cur_on_row
2018-02-16 21:44:50 +03:00
Eugene Kosov
399ec84847 prettify lock_rec_has_to_wait() 2018-02-16 21:44:50 +03:00
Eugene Kosov
e4fa5492a9 prettify lock_has_to_wait 2018-02-16 21:44:50 +03:00
Alexander Kuleshov
f8bdf4d1ee remove duplicated inclusion of derror.h 2018-02-16 15:53:56 +04:00
Alexander Kuleshov
8ab37bbd69 fix comment in my_decimal_set_zero()
which says:

    We need the up-cast here, since my_decimal has sign() member functions,
    which conflicts with decimal_t::size

But decimal_t does not provide `size` field.
2018-02-16 15:52:25 +04:00
Sergey Vojtovich
a351f40cba MDEV-14541 - Workaround GCC ICE on ARM64 2018-02-16 14:14:43 +04:00
Alexander Barkov
a55ded1b89 Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext 2018-02-16 13:52:02 +04:00
Alexander Barkov
6668da2216 MDEV-15289 Binding an out-of-range DATETIME value in binary protocol breaks replication 2018-02-16 13:44:24 +04:00
Jan Lindström
21e5335154 MDEV-9962: encryption.innodb_encryption_filekeys stalled waiting for key encryption threads to decrypt all required spaces
Test changes.
2018-02-16 10:19:57 +02:00
Jan Lindström
d3fbff38b9 MDEV-14814: encryption.innodb_encryption-page-compression failed in buildbot with timeout on wait condition
Problem is that if pages are in buffer pool there is no need for
decrypt or page decompression operations.
2018-02-16 08:21:19 +02:00
Monty
144616034c Don't run sql_sequence.grant for embeddes server 2018-02-16 01:35:19 +02:00
Vladislav Vaintroub
2129eab7e2 MDEV-15071 backup does not store xtrabackup_info in the --extra-lsndir directory 2018-02-15 21:08:18 +00:00