Threads may fall asleep forever while acquiring InnoDB rw-lock on Power8. This
regression was introduced along with InnoDB atomic operations fixes.
The problem was that proper memory order wasn't enforced between "writers"
store and "lock_word" load:
my_atomic_store32((int32*) &lock->waiters, 1);
...
local_lock_word = lock->lock_word;
Locking protocol is such that store to "writers" must be completed before load
of "lock_word". my_atomic_store32() was expected to enforce memory order because
it issues strongest MY_MEMORY_ORDER_SEQ_CST memory barrier.
According to C11:
- any operation with this memory order is both an acquire operation and a
release operation
- for atomic store order must be one of memory_order_relaxed
memory_order_release or memory_order_seq_cst. Otherwise the behavior is
undefined.
That is it doesn't say explicitly that this expectation is wrong, but there are
indications that acquire (which is actually supposed to guarantee memory order
in this case) may not be issued along with MY_MEMORY_ORDER_SEQ_CST.
A good fix for this is to encode waiters into lock_word, but it is a bit too
intrusive. Instead we change atomic store to atomic fetch-and-store, which
does memory load and is guaranteed to issue acquire.
Simplify away recursive flag: it is not necessary for rw-locks to operate
properly. Now writer_thread != 0 means recursive.
As we only need correct value of writer_thread only in writer_thread itself
it is rather safe to load and update it non-atomically.
This patch also fixes potential reorder of "writer_thread" and "recursive"
loads (aka MDEV-7148), which was reopened along with InnoDB thread fences
simplification. Previous versions are unaffected, because they have os_rmb
in rw_lock_lock_word_decr(). It wasn't observed at the moment of writing
though.
In InnoDB and XtraDB functions that declare pointer parameters as nonnull,
remove nullness checks, because GCC would optimize them away anyway.
Use #ifdef instead of #if when checking for a configuration flag.
Clang says that left shifts of negative values are undefined.
So, use ~0U instead of ~0 in a number of macros.
Some functions that were defined as UNIV_INLINE were declared as
UNIV_INTERN. Consistently use the same type of linkage.
ibuf_merge_or_delete_for_page() could pass bitmap_page=NULL to
buf_page_print(), conflicting with the __attribute__((nonnull)).
Make TokuDB report row lock waits with thd_rpl_deadlock_check(). This allows
parallel replication to properly detect conflicts, and kill and retry the
offending transaction.
Merge into the MariaDB tree the pull request from Rich Prohaska for
PerconaFT. These changes are needed to get parallel replication to
work with TokuDB. Once the pull request is accepted by Percona and the new upstream version enters MariaDB, this commit can be superseded.
Original commit message from Rich Prohaska:
1. Fix the release before wait race
The release before wait race occurs when a lock is released by transaction A after transaction B tried to acquire it but before transaction B has a chance to register it's pending lock request. There are several ways to fix this problem, but we want to optimize for the common situation of minimal lock conflicts, which is what the lock acquisition algorithm currently does. Our solution to the release before wait race is for transaction B to retry its lock request after its lock request has been added to the pending lock set.
2. Fix the retry race
The retry race occurs in the current lock retry algorithm which assumes that if some transaction is running lock retry, then my transaction does not also need to run it. There is a chance that some pending lock requests will be skipped, but these lock requests will eventually time out. For applications with small numbers of concurrent transactions, timeouts will frequently occur, and the application throughput will be very small.
The solution to the retry race is to use a group retry algorithm. All threads run through the retry logic. Sequence numbers are used to group retries into batches such that one transaction can run the retry logic on behalf of several transactions. This amortizes the retry cost. The sequence numbers also ensure that when a transaction releases its locks, all of the pending lock requests that it is blocking are retried.
3. Implement a mechanism to find and kill a pending lock request
Tags lock requests with a client id, use the client id as a key into the pending lock requests sets to find a lock request, complete the lock request with a lock timeout error.
Copyright (c) 2016, Rich Prohaska
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Due to high memory reqirements spider tests fail often on automated testing
VM's due to rather limited resource allocation.
For example with 10.2 spider needs at least 200M * 8 mysqld instances = 1.6Gb
RAM per mtr instance. With --parallel=4 it needs 6.4Gb, while appropriate hosts
have just 3Gb.
This is not a fix, this is instrumentation to find out is MySQL frm dictionary
and InnoDB data dictionary really out-of-sync when this assertion is fired,
or is there some other reason (bug).
The crash is caused due to a race condition where wsrep
startup threads invoke ha_maria::implicit_commit() method
while maria_hton is partially initialized. The fix is to
skip this method if plugins are uninitialized.
With "WL#6047 - Do not allocate trx id for read-only transactions"
m_prebuilt->trx->id is always 0 for read-only transactions. This makes
it useless as an index for fuzzy counters.
Use server thread id instead similarly to MySQL.
for InnoDB tables"
Don't use thr_lock.c locks for InnoDB tables. Below is list of changes that
were needed to implement this:
- HANDLER OPEN acquireis MDL_SHARED_READ instead of MDL_SHARED
- HANDLER READ calls external_lock() even if SE is not going to be locked by
THR_LOCK
- InnoDB lock wait timeouts are now honored which are much shorter by default
than server lock wait timeouts (1 year vs 50 seconds)
- with @@autocommit= 1 LOCK TABLES disables autocommit implicitely, though
user still sees @@autocommt= 1
- the above starts implicit transaction
- transactions started by LOCK TABLES are now rolled back on disconnect
(previously everything was committed due to autocommit)
- transactions started by LOCK TABLES are now rolled back by ROLLBACK
(previously everything was committed due to autocommit)
- it is now impossible to change BINLOG_FORMAT under LOCK TABLES (at least
to statement) due to running transaction
- LOCK TABLES WRITE is additionally handled by MDL
- ...in contrast LOCK TABLES READ protection against DML is pure InnoDB
- combining transactional and non-transactional tables under LOCK TABLES
may cause rolled back changes in transactional table and "committed"
changes in non-transactional table
- user may disable innodb_table_locks, which will cause LOCK TABLES to be
noop basically
Removed tests for BUG#45143 and BUG#55930 which cover InnoDB + THR_LOCK. To
operate properly these tests require code flow to go through THR_LOCK debug
sync points, which is not the case after this patch. These tests are removed
by WL#6671 as well. An alternative is to port them to different storage engine.
Issue:
======
Currently the approach we take to find the chunk corresponding to a given
pointer uses srv_buf_pool_chunk_unit based on the assumption that
srv_buf_pool_chunk_unit is the total size of all pages in a buffer pool
chunk. We first step back by srv_buf_pool_chunk_unit bytes and use
std::map::upper_bound() to find the first chunk in the map whose key >= the
resulting pointer.
However, the real size of a chunk (and thus, the total size of its pages)
may differ from the value configured with innodb_buffer_pool_chunk_size
due to rounding up to the OS page size. So, in some cases the above logic
gives us the wrong chunk.
Fix:
====
We find out the chunk corresponding to the give pointer without using
srv_buf_pool_chunk_unit. This is done by using std::map::upper_bound()
to find the next chunk in the map which appears right after the pointer and
decrementing the iterator, which would give us the chunk the pointer
belongs to.
Contribution by Alexey Kopytov.
RB: 13347
Reviewed-by: Debarun Banerjee <debarun.banerjee@oracle.com>
No -DHAVE_LIBNUMA=1 was passed to the source compile (and the
global include/my_config.h wasn't used).
This also is Linux only so corrected the cmake macro.
Fixed indenting in cmake macro.
Removed NUMA defination from include/my_config.h as its only
in the storage engine.
Thanks Elena Stepanova and Vladislav Vaintroub for the detailed
list of bugs/questions.
Signed-off-by: Daniel Black <daniel.black@au.ibm.com>
Analysis: Problem is that page is encrypted but encryption information
on page 0 has already being changed.
Fix: If page header contains key_version != 0 and even if based on
current encryption information tablespace is not encrypted we
need to check is page corrupted. If it is not, then we know that
page is not encrypted. If page is corrupted, we need to try to
decrypt it and then compare the stored and calculated checksums
to see is page corrupted or not.
Clean-up nolock.h: it doesn't serve any purpose anymore. Appropriate code moved
to x86-gcc.h and my_atomic.h.
If gcc sync bultins were detected, we want to make use of them independently of
__GNUC__ definition. E.g. XLC simulates those, but doesn't define __GNUC__.
HS/Spider: According to AIX manual alloca() returns char*, which cannot be
casted to any type with static_cast. Use explicit cast instead.
MDL: Removed namemangling pragma, which didn't let MariaDB build with XLC.
WSREP: _int64 seem to be conflicting name with XLC, replaced with _integer64.
CONNECT: RTLD_NOLOAD is GNU extention. Removed rather meaningless check if
library is loaded. Multiple dlopen()'s of the same library are permitted,
and it never gets closed anyway. Except for error, which was a bug: it may
close library, which can still be referenced by other subsystems.
InnoDB: __ppc_get_timebase() is GNU extention. Only use it when __GLIBC__ is
defined.
Based on contribution by flynn1973.
Two problems:
(1) When pushing warning to sql-layer we need to check that thd != NULL
to avoid NULL-pointer reference.
(2) At tablespace key rotation if used key_id is not found from
encryption plugin tablespace should not be rotated.
MDEV-10394: Innodb system table space corrupted
Analysis: After we have read the page in buf_page_io_complete try to
find if the page is encrypted or corrupted. Encryption was determined
by reading FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION field from FIL-header
as a key_version. However, this field is not always zero even when
encryption is not used. Thus, incorrect key_version could lead situation where
decryption is tried to page that is not encrypted.
Fix: We still read key_version information from FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION
field but also check if tablespace has encryption information before trying
encrypt the page.
For Debian the rules file is the main makefile and assuming that a upstream
makefile will mangle the Debian packaging files creates false alerts
from static analysis tools and other problems.
The following directives to ignore warnings where in the PerconaFT build in tokudb.
These generate errors when g++ ... -o xxx.so is used to compile are shared object.
As these don't actually hit any warnings they have been removed.
* -Wno-ignored-attributes
* -Wno-pointer-bool-conversion
Signed-off-by: Daniel Black <daniel.black@au.ibm.com>
