Commit graph

5034 commits

Author SHA1 Message Date
Marko Mäkelä
4fde1361a6 MDEV-15553 Assertion failed in dict_table_get_col_name
dict_foreign_qualify_index(): Avoid a redundant and harmful
computation of col_name of a virtual column. This fixes the
assertion failure.

dict_foreign_push_index_error(): Do not call dict_table_get_col_name()
on a virtual column. (It is unclear if this condition is actually
reachable.)
2018-04-05 15:01:17 +03:00
Marko Mäkelä
3c21eccb8c MDEV-15764 InnoDB may write uninitialized garbage to redo log
log_write_up_to(): Erase the end of the current log block.
Simplify the computation of pad_size.

log_buffer_switch(): Evaluate a condition only once.
2018-04-03 15:58:13 +03:00
Vladislav Vaintroub
a1d68faa38 CMake : Move INNODB_DISALLOW_WRITES from top-level CMakeLists.txt to innodb 2018-04-01 14:26:06 +00:00
Marko Mäkelä
3eb73bf630 Remove unnecessary SysTablespace references 2018-03-29 16:54:05 +03:00
Marko Mäkelä
622d21e2b8 row_drop_table_for_mysql(): Use a constant string
It does not hurt to delete non-existing records from SYS_TABLESPACES
and SYS_DATAFILES. Because MariaDB does not support CREATE TABLESPACE,
only the system tablespace (space_id=0) can contain multiple tables.
But, there are no entries for the system tablespace in these tables
(which actually are stored inside the system tablespace).
2018-03-29 16:54:05 +03:00
Marko Mäkelä
6cccef21a6 MDEV-15720 ib_buffer_pool unnecessarily includes the temporary tablespace
The purpose of the InnoDB buffer pool dump is to allow InnoDB to be
restarted with the same persistent data pages in the buffer pool.

The InnoDB temporary tablespace that was introduced in MariaDB 10.2.2
is always reinitialized on restart. Therefore, it does not make sense
to attempt to dump or restore any pages of the temporary tablespace.
2018-03-29 13:22:16 +03:00
Marko Mäkelä
4d9969c216 MDEV-15719 ALTER TABLE…ALGORITHM=INPLACE is unnecessarily refused due to innodb_force_recovery
ha_innobase::check_if_supported_inplace_alter(): Only check for
high_level_read_only. Do not unnecessarily refuse
ALTER TABLE...ALGORITHM=INPLACE if innodb_force_recovery was
specified as 1, 2, or 3.

innobase_start_or_create_for_mysql(): Block all writes from SQL
if the system tablespace was initialized with 'newraw'.
2018-03-29 13:20:59 +03:00
Marko Mäkelä
d18a66147c recv_validate_tablespace(): Fix -Wmissing-fallthrough 2018-03-28 20:40:09 +03:00
Marko Mäkelä
5beddfa08c fil_node_open_file(): Add a missing space to message 2018-03-28 20:39:57 +03:00
Thirunarayanan Balathandayuthapani
73af8af094 MDEV-15325 Incomplete validation of missing tablespace during recovery
Problem:
=======
	During validation of missing tablespace, missing tablespace id is
being compared with hash table of redo logs (recv_sys->addr_hash). But if the
hash table ran out of memory then there is a possibility that it will not contain
the redo logs of all tablespace. In that case, Server will load the InnoDB
even though there is a missing tablespace.

Solution:
========
	If the recv_sys->addr_hash hash table ran out of memory then InnoDB needs
to scan the remaining redo log again to validate the missing tablespace.
2018-03-27 13:47:56 +05:30
Thirunarayanan Balathandayuthapani
e27535093d - Follow-up fix to MDEV-15229 2018-03-26 15:48:27 +05:30
Sergey Vojtovich
e147a4a067 Fixed build failure 2018-03-23 00:32:16 +04:00
Marko Mäkelä
e80a842000 Merge 10.1 into 10.2 2018-03-22 18:02:40 +02:00
Marko Mäkelä
2fb31821de MDEV-11984 Avoid accessing SYS_TABLESPACES unnecessarily
The following INFORMATION_SCHEMA views were unnecessarily retrieving
the data from the SYS_TABLESPACES table instead of directly fetching
it from the fil_system cache:

information_schema.innodb_tablespaces_encryption
information_schema.innodb_tablespaces_scrubbing

InnoDB always loads all tablespace metadata into memory at startup
and never evicts it while the tablespace exists.

With this fix, accessing these views will be much faster and use less
memory, and include data about all tablespaces, including undo
tablespaces.

The view information_schema.innodb_sys_tablespaces will still reflect
the contents of the SYS_TABLESPACES table.
2018-03-22 18:01:29 +02:00
Eugene Kosov
8d32959b09 fix data races
srv_last_monitor_time: make all accesses relaxed atomical

WARNING: ThreadSanitizer: data race (pid=12041)
  Write of size 8 at 0x000003949278 by thread T26 (mutexes: write M226445748578513120):
    #0 thd_destructor_proxy storage/innobase/handler/ha_innodb.cc:314:14 (mysqld+0x19b5505)

  Previous read of size 8 at 0x000003949278 by main thread:
    #0 innobase_init(void*) storage/innobase/handler/ha_innodb.cc:4180:11 (mysqld+0x1a03404)
    #1 ha_initialize_handlerton(st_plugin_int*) sql/handler.cc:522:31 (mysqld+0xc5ec73)
    #2 plugin_initialize(st_mem_root*, st_plugin_int*, int*, char**, bool) sql/sql_plugin.cc:1447:9 (mysqld+0x134908d)
    #3 plugin_init(int*, char**, int) sql/sql_plugin.cc:1729:15 (mysqld+0x13484f0)
    #4 init_server_components() sql/mysqld.cc:5345:7 (mysqld+0xbf720f)
    #5 mysqld_main(int, char**) sql/mysqld.cc:5940:7 (mysqld+0xbf107d)
    #6 main sql/main.cc:25:10 (mysqld+0xbe971b)

  Location is global 'srv_running' of size 8 at 0x000003949278 (mysqld+0x000003949278)

WARNING: ThreadSanitizer: data race (pid=27869)
  Atomic write of size 4 at 0x7b4800000c00 by thread T8:
    #0 __tsan_atomic32_exchange llvm/projects/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cc:589 (mysqld+0xbd4eac)
    #1 TTASEventMutex<GenericPolicy>::exit() storage/innobase/include/ib0mutex.h:467:7 (mysqld+0x1a8d4cb)
    #2 PolicyMutex<TTASEventMutex<GenericPolicy> >::exit() storage/innobase/include/ib0mutex.h:609:10 (mysqld+0x1a7839e)
    #3 fil_validate() storage/innobase/fil/fil0fil.cc:5535:2 (mysqld+0x1abd913)
    #4 fil_validate_skip() storage/innobase/fil/fil0fil.cc:204:9 (mysqld+0x1aba601)
    #5 fil_aio_wait(unsigned long) storage/innobase/fil/fil0fil.cc:5296:2 (mysqld+0x1abbae6)
    #6 io_handler_thread storage/innobase/srv/srv0start.cc:340:3 (mysqld+0x21abe1e)

  Previous read of size 4 at 0x7b4800000c00 by main thread (mutexes: write M1273, write M1271):
    #0 TTASEventMutex<GenericPolicy>::state() const storage/innobase/include/ib0mutex.h:530:10 (mysqld+0x21c66e2)
    #1 sync_array_detect_deadlock(sync_array_t*, sync_cell_t*, sync_cell_t*, unsigned long) storage/innobase/sync/sync0arr.cc:746:14 (mysqld+0x21c1c7a)
    #2 sync_array_wait_event(sync_array_t*, sync_cell_t*&) storage/innobase/sync/sync0arr.cc:465:6 (mysqld+0x21c1708)
    #3 TTASEventMutex<GenericPolicy>::enter(unsigned int, unsigned int, char const*, unsigned int) storage/innobase/include/ib0mutex.h:516:6 (mysqld+0x1a8c206)
    #4 PolicyMutex<TTASEventMutex<GenericPolicy> >::enter(unsigned int, unsigned int, char const*, unsigned int) storage/innobase/include/ib0mutex.h:635:10 (mysqld+0x1a782c3)
    #5 fil_mutex_enter_and_prepare_for_io(unsigned long) storage/innobase/fil/fil0fil.cc:1131:3 (mysqld+0x1a9a92e)
    #6 fil_io(IORequest const&, bool, page_id_t const&, page_size_t const&, unsigned long, unsigned long, void*, void*, bool) storage/innobase/fil/fil0fil.cc:5082:2 (mysqld+0x1ab8de2)
    #7 buf_flush_write_block_low(buf_page_t*, buf_flush_t, bool) storage/innobase/buf/buf0flu.cc:1112:3 (mysqld+0x1cb970a)
    #8 buf_flush_page(buf_pool_t*, buf_page_t*, buf_flush_t, bool) storage/innobase/buf/buf0flu.cc:1270:3 (mysqld+0x1cb7d70)
    #9 buf_flush_try_neighbors(page_id_t const&, buf_flush_t, unsigned long, unsigned long) storage/innobase/buf/buf0flu.cc:1493:9 (mysqld+0x1cc9674)
    #10 buf_flush_page_and_try_neighbors(buf_page_t*, buf_flush_t, unsigned long, unsigned long*) storage/innobase/buf/buf0flu.cc:1565:13 (mysqld+0x1cbadf3)
    #11 buf_do_flush_list_batch(buf_pool_t*, unsigned long, unsigned long) storage/innobase/buf/buf0flu.cc:1825:3 (mysqld+0x1cbbcb8)
    #12 buf_flush_batch(buf_pool_t*, buf_flush_t, unsigned long, unsigned long, flush_counters_t*) storage/innobase/buf/buf0flu.cc:1895:16 (mysqld+0x1cbb459)
    #13 buf_flush_do_batch(buf_pool_t*, buf_flush_t, unsigned long, unsigned long, flush_counters_t*) storage/innobase/buf/buf0flu.cc:2065:2 (mysqld+0x1cbcfe1)
    #14 buf_flush_lists(unsigned long, unsigned long, unsigned long*) storage/innobase/buf/buf0flu.cc:2167:8 (mysqld+0x1cbd5a3)
    #15 log_preflush_pool_modified_pages(unsigned long) storage/innobase/log/log0log.cc:1400:13 (mysqld+0x1eefc3b)
    #16 log_make_checkpoint_at(unsigned long, bool) storage/innobase/log/log0log.cc:1751:10 (mysqld+0x1eefb16)
    #17 buf_dblwr_create() storage/innobase/buf/buf0dblwr.cc:335:2 (mysqld+0x1cd2141)
    #18 innobase_start_or_create_for_mysql() storage/innobase/srv/srv0start.cc:2539:10 (mysqld+0x21b4d8e)
    #19 innobase_init(void*) storage/innobase/handler/ha_innodb.cc:4193:8 (mysqld+0x1a5e3d7)
    #20 ha_initialize_handlerton(st_plugin_int*) sql/handler.cc:522:31 (mysqld+0xc74d33)
    #21 plugin_initialize(st_mem_root*, st_plugin_int*, int*, char**, bool) sql/sql_plugin.cc:1447:9 (mysqld+0x1376d5d)
    #22 plugin_init(int*, char**, int) sql/sql_plugin.cc:1729:15 (mysqld+0x13761c0)
    #23 init_server_components() sql/mysqld.cc:5348:7 (mysqld+0xc0d0ff)
    #24 mysqld_main(int, char**) sql/mysqld.cc:5943:7 (mysqld+0xc06f9d)
    #25 main sql/main.cc:25:10 (mysqld+0xbff71b)

WARNING: ThreadSanitizer: data race (pid=29031)
  Write of size 8 at 0x0000039e48e0 by thread T15:
    #0 srv_monitor_thread storage/innobase/srv/srv0srv.cc:1699:24 (mysqld+0x21a254e)

  Previous write of size 8 at 0x0000039e48e0 by thread T14:
    #0 srv_refresh_innodb_monitor_stats() storage/innobase/srv/srv0srv.cc:1165:24 (mysqld+0x21a3124)
    #1 srv_error_monitor_thread storage/innobase/srv/srv0srv.cc:1836:3 (mysqld+0x21a2d40)

  Location is global 'srv_last_monitor_time' of size 8 at 0x0000039e48e0 (mysqld+0x0000039e48e0)
2018-03-22 14:42:15 +04:00
Marko Mäkelä
03a80e20f7 pfs_os_file_read_no_error_handling_int_fd_func(): Remove a variable 2018-03-21 22:58:56 +02:00
Marko Mäkelä
3d7915f000 Merge 10.1 into 10.2 2018-03-21 22:58:52 +02:00
Vicențiu Ciorbaru
82aeb6b596 Merge branch '10.1' into 10.2 2018-03-21 10:36:49 +02:00
Marko Mäkelä
613be24b7a Merge 10.0 into 10.1 2018-03-20 19:25:08 +02:00
Marko Mäkelä
0492100059 Merge 5.5 into 10.0 2018-03-20 18:36:03 +02:00
Marko Mäkelä
e0a0fe7d81 MDEV-12396 IMPORT TABLESPACE: Do not retry partial reads
fil_iterate(), fil_tablespace_iterate(): Replace os_file_read()
with os_file_read_no_error_handling().

os_file_read_func(), os_file_read_no_error_handling_func():
Do not retry partial reads. There used to be an infinite amount
of retries. Because InnoDB extends both data and log files upfront,
partial reads should be impossible during normal operation.
2018-03-20 15:31:39 +02:00
Marko Mäkelä
a80af35a85 MDEV-12396 IMPORT cleanup: ROW_FORMAT=COMPRESSED
Initialize block.page.zip only once.

PageConverter::update(): Initialize m_page_zip_ptr
as late as possible.
(We should really remove it at some point.)

PageConverter::operator(): Refer to block->page.zip instead of
m_page_zip_ptr.

AbstractCallback::get_frame(): Define static. Refer
to block->page.zip.data directly.

fil_iterate(): Refer to block->page.zip.data directly.

fil_tablespace_iterate(): Initialize block.page.zip.data as soon
as possible.
2018-03-20 15:31:39 +02:00
Marko Mäkelä
eaa7bfb59f MDEV-12396 IMPORT TABLESPACE: Simplify validation
fil_iterate(): Validate the pages directly.

import_page_status_t, PageConverter::validate(): Remove.

AbstractCallback::filename(): New accessor.

AbstractCallback::is_interrupted(): Replaces periodic_check().

PageConverter::trigger_corruption(): Remove.
2018-03-20 15:31:39 +02:00
Marko Mäkelä
6247c64c2a MDEV-12396 IMPORT TABLESPACE cleanup
Reduce unnecessary inter-module calls for IMPORT TABLESPACE.
Move some IMPORT-related code from fil0fil.cc to row0import.cc.

PageCallback: Remove. Make AbstractCallback the base class.

PageConverter: Define some member functions inline.
2018-03-20 15:31:39 +02:00
Thirunarayanan Balathandayuthapani
eee73ddfbb MDEV-12255 innodb_prefix_index_cluster_optimization hits debug build
assert on UTF-8 columns

Problem:
=======
(1) Multi-byte character cases are not considered during prefix index
cluster optimization check. It leads to fetch of improper results during
read operation.
(2) Strict assert in row_sel_field_store_in_mysql_format_func and it asserts
for prefix index record to mysql conversion.

Solution:
========
(1) Consider the case of multi-byte character during prefix index
cluster optimization check.
(2) Relax the assert in row_sel_field_store_in_mysql_format_func to allow
prefix index record to mysql format conversion.

The patch is taken from
1eee538087
2018-03-20 17:53:33 +05:30
Eugene Kosov
5a8f8f89d6 honor alignment rules and xtradb too 2018-03-20 10:46:57 +03:00
Vicențiu Ciorbaru
24b353162f Merge branch '10.0-galera' into 10.1 2018-03-19 15:21:01 +02:00
Eugene Kosov
75c76dbb06 MDEV-15030 Add ASAN instrumentation
Learn both valgrind and asan to catch this bug:

  mem_heap_t* heap = mem_heap_create(1024);
  byte* p = reinterpret_cast<byte*>(heap) + sizeof(mem_heap_t);
  *p = 123;

Overflows of the last allocation in a block will be catched too.

mem_heap_create_block(): poison newly allocated memory
2018-03-19 16:18:53 +03:00
Marko Mäkelä
27c54b77c1 Make some locking primitives inline
lock_rec_trx_wait(): Merge to the only caller lock_prdt_rec_move().

lock_rec_reset_nth_bit(), lock_set_lock_and_trx_wait(),
lock_reset_lock_and_trx_wait(): Define in lock0priv.h.
2018-03-16 15:50:05 +02:00
Marko Mäkelä
d2a15092c1 lock_table_create(), lock_rec_create(): Clean up the WSREP code
By definition, c_lock->trx->lock.wait_lock==c_lock cannot hold.
That is, the owner transaction of a lock cannot be waiting for that
particular lock. It must have been waiting for some other lock.
Remove the dead code related to that. Also, test c_lock for NULLness
only once.
2018-03-16 15:50:05 +02:00
Marko Mäkelä
61e192fa40 lock_reset_lock_and_trx_wait(): Remove diagnostics 2018-03-16 15:50:05 +02:00
Marko Mäkelä
27d4333cb9 MDEV-13935 INSERT stuck at state Unlocking tables
Refactor lock_grant(). With innodb_lock_schedule_algorithm=VATS
some callers were passing an incorrect parameter owns_trx_mutex
to lock_grant().

lock_grant_after_reset(): Refactored from lock_grant(), without
the call to lock_reset_lock_and_trx_wait().

lock_grant_have_trx_mutex(): A variant of lock_grant() where the
caller already holds the lock->trx->mutex. The normal lock_grant()
will acquire and release lock->trx->mutex.

lock_grant(): Define as a wrapper that will acquire lock->trx->mutex.
2018-03-16 15:50:04 +02:00
Marko Mäkelä
f93a219c72 MDEV-13935 INSERT stuck at state Unlocking tables
lock_rec_queue_validate(): Restore some assertions.

DeadlockChecker::select_victim(): Reduce the WSREP-related diff.
2018-03-16 15:50:04 +02:00
Marko Mäkelä
cac373f533 Add missing #ifdef WITH_WSREP
lock_table_create(): Move the WSREP parameter c_lock last,
and make it NULL by default, to avoid the need for a wrapper
function.

lock_table_enqueue_waiting(): Move the WSREP parameter c_lock last.
2018-03-16 15:50:04 +02:00
Marko Mäkelä
788b3ee86d Reduce the diff from 5.7 in DeadlockChecker::search()
This is a non-functional change.
2018-03-16 15:50:04 +02:00
Marko Mäkelä
bd7ed1b923 MDEV-13935 INSERT stuck at state Unlocking tables
Revert the dead code for MySQL 5.7 multi-master replication (GCS),
also known as
WL#6835: InnoDB: GCS Replication: Deterministic Deadlock Handling
(High Prio Transactions in InnoDB).

Also, make innodb_lock_schedule_algorithm=vats skip SPATIAL INDEX,
because the code does not seem to be compatible with them.

Add FIXME comments to some SPATIAL INDEX locking code. It looks
like Galera write-set replication might not work with SPATIAL INDEX.
2018-03-16 15:50:04 +02:00
Marko Mäkelä
e15e879fae Remove the unreachable error DB_QUE_THR_SUSPENDED 2018-03-16 15:50:03 +02:00
Marko Mäkelä
84129fb1b5 After-merge fix for commit 98eb9518db
The merge only covered 10.1 up to
commit 4d248974e0.

Actually merge the changes up to
commit 0a534348c7.

Also, remove the unused InnoDB field trx_t::abort_type.
2018-03-16 15:49:53 +02:00
Marko Mäkelä
7033af9e81 Conditionally define TRX_WSREP_ABORT 2018-03-16 08:38:48 +02:00
Marko Mäkelä
ca40330d1d Fix a deadlock in thd_report_wait_for()
Unlike commit a54abf0175 claimed,
the caller of THD::awake() may actually hold the InnoDB lock_sys->mutex.
That commit introduced a deadlock of threads in the replication slave
when running the test rpl.rpl_parallel_optimistic_nobinlog.

lock_trx_handle_wait(): Expect the callers to acquire and release
lock_sys->mutex and trx->mutex.

innobase_kill_query(): Restore the logic for conditionally acquiring
and releasing the mutexes. THD::awake() can be called from inside
InnoDB while holding one or both mutexes, via thd_report_wait_for() and
via wsrep_innobase_kill_one_trx().
2018-03-16 08:23:56 +02:00
Marko Mäkelä
dbb3960ad8 Follow-up to MDEV-11236/MDEV-14846 debug assertion
ha_innobase::unlock_row(): Use a relaxed version of the
trx_state_eq() debug assertion, because rr_unlock_row()
may be invoked after an error has been already reported
and the transaction has been rolled back.
2018-03-15 19:48:29 +02:00
Marko Mäkelä
723f87e9d3 lock_table_create(), lock_rec_create(): Clean up the WSREP code
By definition, c_lock->trx->lock.wait_lock==c_lock cannot hold.
That is, the owner transaction of a lock cannot be waiting for that
particular lock. It must have been waiting for some other lock.
Remove the dead code related to that. Also, test c_lock for NULLness
only once.
2018-03-15 19:44:24 +02:00
Marko Mäkelä
a54abf0175 innobase_kill_query(): Use lock_trx_handle_wait()
The caller of THD::awake() should never hold any InnoDB mutexes,
so we can always acquire lock_sys->mutex and trx->mutex.
2018-03-15 19:44:24 +02:00
Daniel Black
8b54c31486 MDEV-8743: where O_CLOEXEC is available, use for innodb buf_dump
As this is the only moderately critical fopened for writing file,
create an alternate path to use open and fdopen for non-glibc platforms
that support O_CLOEXEC (BSDs).

Tested on Linux (by modifing the GLIBC defination) to take this
alternate path:

$ cd /proc/23874
$ more fdinfo/71
pos:    0
flags:  02100001
mnt_id: 24
$ ls -la fd/71
l-wx------. 1 dan dan 64 Mar 14 13:30 fd/71 -> /dev/shm/var_auto_i7rl/mysqld.1/data/ib_buffer_pool.incomplete
2018-03-15 12:07:43 +02:00
Thirunarayanan Balathandayuthapani
76ae6e725d MDEV-15384 buf_flush_LRU_list_batch() always reports n->flushed=0, n->evicted=0
- buf_flush_LRU_list_batch() initializes the count to zero and updates them
correctly.
2018-03-13 15:25:38 +05:30
Thirunarayanan Balathandayuthapani
ff909acfa4 MDEV-14545 Backup fails due to MLOG_INDEX_LOAD record
Problem:
=======
  Mariabackup exits during prepare phase if it encounters
MLOG_INDEX_LOAD redo log record. MLOG_INDEX_LOAD record
informs Mariabackup that the backup cannot be completed based
on the redo log scan, because some information is purposely
omitted due to bulk index creation in ALTER TABLE.

Solution:
========
Detect the MLOG_INDEX_LOAD redo record during backup phase and
exit the mariabackup with the proper error message.
2018-03-13 15:25:38 +05:30
Marko Mäkelä
94e00da9f1 Merge 10.1 into 10.2 2018-03-13 09:43:13 +02:00
Marko Mäkelä
71f9cc1221 MDEV-15554 InnoDB page_cleaner shutdown sometimes hangs
buf_flush_page_cleaner_coordinator(): Signal the worker threads
to exit while waiting for them to exit. Apparently, signals are
sometimes lost, causing shutdown to occasionally hang when
multiple page cleaners (and buffer pool instances) are used,
that is, when innodb_buffer_pool_size is at least 1 GiB.

buf_flush_page_cleaner_close(): Merge with the only caller.
2018-03-13 09:41:42 +02:00
Sergey Vojtovich
12f9cf075f Removed unused variables.
Based on contribution by Daniel Black.
2018-03-12 16:49:50 +04:00
Marko Mäkelä
3a93ec53c1 Merge 10.1 into 10.2 2018-03-12 14:27:17 +02:00