TABLES <list> WITH READ LOCK are incompatible".
The problem was that FLUSH TABLES <list> WITH READ LOCK
which was issued when other connection has acquired global
read lock using FLUSH TABLES WITH READ LOCK was blocked
and has to wait until global read lock is released.
This issue stemmed from the fact that FLUSH TABLES <list>
WITH READ LOCK implementation has acquired X metadata locks
on tables to be flushed. Since these locks required acquiring
of global IX lock this statement was incompatible with global
read lock.
This patch addresses problem by using SNW metadata type of
lock for tables to be flushed by FLUSH TABLES <list> WITH
READ LOCK. It is OK to acquire them without global IX lock
as long as we won't try to upgrade those locks. Since SNW
locks allow concurrent statements using same table FLUSH
TABLE <list> WITH READ LOCK now has to wait until old
versions of tables to be flushed go away after acquiring
metadata locks. Since such waiting can lead to deadlock
MDL deadlock detector was extended to take into account
waits for flush and resolve such deadlocks.
As a bonus code in open_tables() which was responsible for
waiting old versions of tables to go away was refactored.
Now when we encounter old version of table in open_table()
we don't back-off and wait for all old version to go away,
but instead wait for this particular table to be flushed.
Such approach supported by deadlock detection should reduce
number of scenarios in which FLUSH TABLES aborts concurrent
multi-statement transactions.
Note that active FLUSH TABLES <list> WITH READ LOCK still
blocks concurrent FLUSH TABLES WITH READ LOCK statement
as the former keeps tables open and thus prevents the
latter statement from doing flush.
mysql-test/include/handler.inc:
Adjusted test case after changing status which is set
when FLUSH TABLES waits for tables to be flushed from
"Flushing tables" to "Waiting for table".
mysql-test/r/flush.result:
Added test which checks that "flush tables <list> with
read lock" is compatible with active "flush tables with
read lock" but not vice-versa. This test also covers
bug #52044 "FLUSH TABLES WITH READ LOCK and FLUSH TABLES
<list> WITH READ LOCK are incompatible".
mysql-test/r/mdl_sync.result:
Added scenarios in which wait for table to be flushed
causes deadlocks to the coverage of MDL deadlock detector.
mysql-test/suite/perfschema/r/dml_setup_instruments.result:
Adjusted test results after removal of COND_refresh
condition variable.
mysql-test/suite/perfschema/r/server_init.result:
Adjusted test and its results after removal of COND_refresh
condition variable.
mysql-test/suite/perfschema/t/server_init.test:
Adjusted test and its results after removal of COND_refresh
condition variable.
mysql-test/t/flush.test:
Added test which checks that "flush tables <list> with
read lock" is compatible with active "flush tables with
read lock" but not vice-versa. This test also covers
bug #52044 "FLUSH TABLES WITH READ LOCK and FLUSH TABLES
<list> WITH READ LOCK are incompatible".
mysql-test/t/kill.test:
Adjusted test case after changing status which is set
when FLUSH TABLES waits for tables to be flushed from
"Flushing tables" to "Waiting for table".
mysql-test/t/lock_multi.test:
Adjusted test case after changing status which is set
when FLUSH TABLES waits for tables to be flushed from
"Flushing tables" to "Waiting for table".
mysql-test/t/mdl_sync.test:
Added scenarios in which wait for table to be flushed
causes deadlocks to the coverage of MDL deadlock detector.
sql/ha_ndbcluster.cc:
Adjusted code after adding one more parameter for
close_cached_tables() call - timeout for waiting for
table to be flushed.
sql/ha_ndbcluster_binlog.cc:
Adjusted code after adding one more parameter for
close_cached_tables() call - timeout for waiting for
table to be flushed.
sql/lock.cc:
Removed COND_refresh condition variable. See comment
for sql_base.cc for details.
sql/mdl.cc:
Now MDL deadlock detector takes into account information
about waits for table flushes when searching for deadlock.
To implement this change:
- Declaration of enum_deadlock_weight and
Deadlock_detection_visitor were moved to mdl.h header
to make them available to the code in table.cc which
implements deadlock detector traversal through edges
of waiters graph representing waiting for flush.
- Since now MDL_context may wait not only for metadata
lock but also for table to be flushed an abstract
Wait_for_edge class was introduced. Its descendants
MDL_ticket and Flush_ticket incapsulate specifics
of inspecting waiters graph when following through
edge representing wait of particular type.
We no longer require global IX metadata lock when acquiring
SNW or SNRW locks. Such locks are needed only when metadata
locks of these types are upgraded to X locks. This allows
to use SNW locks in FLUSH TABLES <list> WITH READ LOCK
implementation and keep the latter compatible with global
read lock.
sql/mdl.h:
Now MDL deadlock detector takes into account information
about waits for table flushes when searching for deadlock.
To implement this change:
- Declaration of enum_deadlock_weight and
Deadlock_detection_visitor were moved to mdl.h header
to make them available to the code in table.cc which
implements deadlock detector traversal through edges
of waiters graph representing waiting for flush.
- Since now MDL_context may wait not only for metadata
lock but also for table to be flushed an abstract
Wait_for_edge class was introduced. Its descendants
MDL_ticket and Flush_ticket incapsulate specifics
of inspecting waiters graph when following through
edge representing wait of particular type.
- Deadlock_detection_visitor now has m_table_shares_visited
member which allows to support recursive locking for
LOCK_open. This is required when deadlock detector
inspects waiters graph which contains several edges
representing waits for flushes or needs to come through
the such edge more than once.
sql/mysqld.cc:
Removed COND_refresh condition variable. See comment
for sql_base.cc for details.
sql/mysqld.h:
Removed COND_refresh condition variable. See comment
for sql_base.cc for details.
sql/sql_base.cc:
Changed approach to how threads are waiting for table
to be flushed. Now thread that wants to wait for old
table to go away subscribes for notification by adding
Flush_ticket to table's share and waits using
MDL_context::m_wait object. Once table gets flushed
(i.e. all tables are closed and table share is ready
to be destroyed) all such waiters are notified
individually.
Thanks to this change MDL deadlock detector can take
such waits into account.
To implement this/as result of this change:
- tdc_wait_for_old_versions() was replaced with
tdc_wait_for_old_version() which waits for individual
old share to go away and which is called by open_table()
after finding out that share is outdated. We don't
need to perform back-off before such waiting thanks
to the fact that deadlock detector now sees such waits.
- As result Open_table_ctx::m_mdl_requests became
unnecessary and was removed. We no longer allocate
copies of MDL_request objects on MEM_ROOT when
MYSQL_OPEN_FORCE_SHARED/SHARED_HIGH_PRIO flags are
in effect.
- close_cached_tables() and tdc_wait_for_old_version()
share code which implements waiting for share to be
flushed - the both use TABLE_SHARE::wait_until_flush()
method. Thanks to this close_cached_tables() supports
timeouts and has extra parameter for this.
- Open_table_context::OT_MDL_CONFLICT enum element was
renamed to OT_CONFLICT as it is now also used in cases
when back-off is required to resolve deadlock caused
by waiting for flush and not metadata lock.
- In cases when we discover that current connection tries
to open tables from different generation we now simply
back-off and restart process of opening tables. To
support this Open_table_context::OT_REOPEN_TABLES enum
element was added.
- COND_refresh condition variable became unnecessary and
was removed.
- mysql_notify_thread_having_shared_lock() no longer wakes
up connections waiting for flush as all such connections
can be waken up by deadlock detector if necessary.
sql/sql_base.h:
- close_cached_tables() now has one more parameter -
timeout for waiting for table to be flushed.
- Open_table_context::OT_MDL_CONFLICT enum element was
renamed to OT_CONFLICT as it is now also used in cases
when back-off is required to resolve deadlock caused
by waiting for flush and not metadata lock.
Added new OT_REOPEN_TABLES enum element to be used in
cases when we need to restart open tables process even
in the middle of transaction.
- Open_table_ctx::m_mdl_requests became unnecessary and
was removed.
sql/sql_class.h:
Added assert ensuring that we won't use LOCK_open mutex
with THD::enter_cond(). Otherwise deadlocks can arise in
MDL deadlock detector.
sql/sql_parse.cc:
Changed FLUSH TABLES <list> WITH READ LOCK to take SNW
metadata locks instead of X locks on tables to be flushed.
Since we no longer require global IX lock to be taken
when SNW locks are taken this makes this statement
compatible with FLUSH TABLES WITH READ LOCK statement.
Since SNW locks allow other connections to have table
opened FLUSH TABLES <list> WITH READ LOCK now has to
wait during open_tables() for old version to go away.
Such waits can lead to deadlocks which will be detected
by MDL deadlock detector which now takes waits for table
to be flushed into account.
Also adjusted code after adding one more parameter for
close_cached_tables() call - timeout for waiting for
table to be flushed.
sql/sql_yacc.yy:
FLUSH TABLES <list> WITH READ LOCK now needs only SNW
metadata locks on tables.
sql/sys_vars.cc:
Adjusted code after adding one more parameter for
close_cached_tables() call - timeout for waiting for
table to be flushed.
sql/table.cc:
Implemented new approach to how threads are waiting for
table to be flushed. Now thread that wants to wait for
old table to go away subscribes for notification by
adding Flush_ticket to table's share and waits using
MDL_context::m_wait object. Once table gets flushed
(i.e. all tables are closed and table share is ready
to be destroyed) all such waiters are notified
individually. This change allows to make such waits
visible inside of MDL deadlock detector.
To do it:
- Added list of waiters/Flush_tickets to TABLE_SHARE
class.
- Changed free_table_share() to postpone freeing of
share memory until last waiter goes away and to
wake up subscribed waiters.
- Added TABLE_SHARE::wait_until_flushed() method which
implements subscription to the list of waiters for
table to be flushed and waiting for this event.
Implemented interface which allows to expose waits for
flushes to MDL deadlock detector:
- Introduced Flush_ticket class a descendant of
Wait_for_edge class.
- Added TABLE_SHARE::find_deadlock() method which allows
deadlock detector to find out what contexts are still
using old version of table in question (i.e. to find
out what contexts are waited for by owner of
Flush_ticket).
sql/table.h:
In order to support new strategy of waiting for table flush
(see comment for table.cc for details) added list of
waiters/Flush_tickets to TABLE_SHARE class.
Implemented interface which allows to expose waits for
flushes to MDL deadlock detector:
- Introduced Flush_ticket class a descendant of
Wait_for_edge class.
- Added TABLE_SHARE::find_deadlock() method which allows
deadlock detector to find out what contexts are still
using old version of table in question (i.e. to find
out what contexts are waited for by owner of
Flush_ticket).
DATABASE with open HANDLER"
Remove LOCK_create_db, database name locks, and use metadata locks instead.
This exposes CREATE/DROP/ALTER DATABASE statements to the graph-based
deadlock detector in MDL, and paves the way for a safe, deadlock-free
implementation of RENAME DATABASE.
Database DDL statements will now take exclusive metadata locks on
the database name, while table/view/routine DDL statements take
intention exclusive locks on the database name. This prevents race
conditions between database DDL and table/view/routine DDL.
(e.g. DROP DATABASE with concurrent CREATE/ALTER/DROP TABLE)
By adding database name locks, this patch implements
WL#4450 "DDL locking: CREATE/DROP DATABASE must use database locks" and
WL#4985 "DDL locking: namespace/hierarchical locks".
The patch also changes code to use init_one_table() where appropriate.
The new lock_table_names() function requires TABLE_LIST::db_length to
be set correctly, and this is taken care of by init_one_table().
This patch also adds a simple template to help work with
the mysys HASH data structure.
Most of the patch was written by Konstantin Osipov.
mysql-test/suite/perfschema/t/no_threads-master.opt:
Innodb will start multiple threads, which is
not compatible with --one-thread. Disable
innodb to be able to run the test case.
locks for DML statements and changes the way MDL locks
are acquired/granted in contended case.
Instead of backing-off when a lock conflict is encountered
and waiting for it to go away before restarting open_tables()
process we now wait for lock to be released without releasing
any previously acquired locks. If conflicting lock goes away
we resume opening tables. If waiting leads to a deadlock we
try to resolve it by backing-off and restarting open_tables()
immediately.
As result both waiting for possibility to acquire and
acquiring of a metadata lock now always happen within the
same MDL API call. This has allowed to make release of a lock
and granting it to the most appropriate pending request an
atomic operation.
Thanks to this it became possible to wake up during release
of lock only those waiters which requests can be satisfied
at the moment as well as wake up only one waiter in case
when granting its request would prevent all other requests
from being satisfied. This solves thundering herd problem
which occured in cases when we were releasing some lock and
woke up many waiters for SNRW or X locks (this was the issue
in bug#52289 "performance regression for MyISAM in sysbench
OLTP_RW test".
This also allowed to implement more fair (FIFO) scheduling
among waiters with the same priority.
It also opens the door for introducing new types of requests
for metadata locks such as low-prio SNRW lock which is
necessary in order to support LOCK TABLES LOW_PRIORITY WRITE.
Notice that after this sometimes can report ER_LOCK_DEADLOCK
error in cases in which it has not happened before.
Particularly we will always report this error if waiting for
conflicting lock has happened in the middle of transaction
and resulted in a deadlock. Before this patch the error was
not reported if deadlock could have been resolved by backing
off all metadata locks acquired by the current statement.
mysql-test/r/mdl_sync.result:
Added test coverage for some aspects of deadlock handling in
metadata locking subsystem.
Adjusted test case after removing back-off in general case
when conflicting metadata lock is encountered during
open_tables() (now this happens only if waiting for
conflicting lock to go away leads to a deadlock).
mysql-test/r/sp_sync.result:
Adjusted test case after removing back-off in general case
when conflicting metadata lock is encountered during
open_tables() (now this happens only if waiting for
conflicting lock to go away leads to a deadlock).
mysql-test/suite/perfschema/r/dml_setup_instruments.result:
Adjusted test results after renaming MDL_context::
m_waiting_for_lock rwlock to m_LOCK_waiting_for.
mysql-test/suite/rpl/r/rpl_sp.result:
Adjusted test case after implementing new approach to
acquiring metadata locks in open_tables(). We no longer
release all MDL locks acquired by statement before waiting
for conflicting lock to go away. As result DROP FUNCTION
statement has to wait for DML statement which managed to
acquire metadata lock on function being dropped and now
waits for other conflicting metadata lock to go away.
mysql-test/suite/rpl/t/rpl_sp.test:
Adjusted test case after implementing new approach to
acquiring metadata locks in open_tables(). We no longer
release all MDL locks acquired by statement before waiting
for conflicting lock to go away. As result DROP FUNCTION
statement has to wait for DML statement which managed to
acquire metadata lock on function being dropped and now
waits for other conflicting metadata lock to go away.
mysql-test/t/mdl_sync.test:
Added test coverage for some aspects of deadlock handling in
metadata locking subsystem.
Adjusted test case after removing back-off in general case
when conflicting metadata lock is encountered during
open_tables() (now this happens only if waiting for
conflicting lock to go away leads to a deadlock).
mysql-test/t/sp_sync.test:
Adjusted test case after removing back-off in general case
when conflicting metadata lock is encountered during
open_tables() (now this happens only if waiting for
conflicting lock to go away leads to a deadlock).
sql/mdl.cc:
Changed MDL subsystem to support new approach to acquring
metadata locks in open tables and more fair and efficient
scheduling of metadata locks. To implement this:
- Made releasing of the lock and granting it to the most
appropriate pending request atomic operation. As result it
became possible to wake up only those waiters requests from
which can be satisfied at the moment as well as wake-up
only one waiter in case when granting its request would
prevent all other requests from being satisfied.
This solved thundering herd problem which occured in cases
when we were releasing some lock and woke up many waiters
for SNRW or X locks (this was the issue in Bug #52289
"performance regression for MyISAM in sysbench OLTP_RW
test".
To emphasize above changes wake_up_waiters() was renamed
to MDL_context::reschedule_waiters().
- Changed code to add tickets for new requests to the back of
waiters queue and to select tickets to be satisfied from
the head of the queue if possible (this makes scheduling of
requests with the same priority fair). To be able to do
this efficiently we now use for waiting and granted queues
version of I_P_List class which provides fast push_back()
method.
- Members and methods of MDL_context related to sending
and waiting for signal were moved to separate MDL_wait
class.
- Since in order to avoid race conditions we must grant the
lock only to the context which was not chosen as a victim
of deadlock, killed or aborted due to timeout
MDL_wait::set_status() (former awake()) was changed not to
send signal if signal slot is already occupied and to
indicate this fact through its return value. As another
consequence MDL_wait::timed_wait() method was changed to
handle timeout (optionally) and abort due to kill as
signals which make signal slot occupied.
- Renamed MDL_context::acquire_lock_impl() to acquire_lock().
Changed it to be able correctly process requests for shared
locks when there are open HANDLERs, made this method more
optimized for acquisition of shared locks. As part of this
change moved code common between try_acquire_lock() and
acquire_lock() to new try_acquire_lock_impl() method.
Also adjusted acquire_lock()'s code to take into account
the fact that in cases when lock is granted as result of
MDL_context::reschedule_waiters() call (i.e. when it is
granted after waiting for lock conflict to go away)
updating MDL_lock state is responsibility of the thread
calling reschedule_waiters().
- Changed MDL_context::find_deadlock() to send VICTIM
signal even if victim is the context which has initiated
deadlock detection. This is required in order to avoid
races in cases when the same context simultaneously is
chosen as a victim and its request for lock is satisfied.
As result return value of this method became unnecessary
and it was changed to return void.
Adjusted MDL_lock::find_deadlock() method to take into
account that now there can be a discrepancy between
MDL_context::m_waiting_for value being set and real state
of the ticket this member points to.
- Renamed MDL_context::m_waiting_for_lock to m_LOCK_waiting_for
and MDL_context::stop_waiting() to done_waiting_for().
- Finally, removed MDL_context::wait_for_lock() method.
sql/mdl.h:
Changed MDL subsystem to support new approach to acquring
metadata locks in open tables and more fair and efficient
scheduling of metadata locks. To implement this:
- Members and methods of MDL_context related to sending
and waiting for signal were moved to separate MDL_wait
class.
- Since now in order to avoid race conditions we must grant
the lock only to the context which was not chosen as a
victim of deadlock, killed or aborted due to timeout
MDL_wait::set_status (former awake()) was changed not to
send signal if signal slot is already occupied and to
indicate this fact through its return value.
Also NORMAL_WAKE_UP signal became GRANTED, and timeouts
and aborts due to kill became full blown signals rather
than simple return values.
- MDL_wait::timed_wait() now takes extra parameter that
indicates whether signal should be set if timeout is
reached.
- Enabled fast push_back() operation in MDL_context::m_tickets
list to make move_ticket_after_trans_sentinel() method more
efficient.
- Removed MDL_context::wait_for_lock() method.
- Renamed MDL_context::m_waiting_for_lock to m_LOCK_waiting_for
and MDL_context::stop_waiting() to done_waiting_for().
- MDL_context::acquire_lock_impl() became acquire_lock().
- Introduced MDL_context::try_acquire_lock_impl() as a
place for code shared by try_acquire_lock and
acquire_lock().
- Due to fact that now VICTIM signal is sent even if victim
is the context which has initiated deadlock detection
find_deadlock() no longer needs a return value.
sql/sql_base.cc:
Implemented new approach to acquiring metadata locks in
open_tables(). We no longer perform back-off when conflicting
metadata lock is encountered. Instead we wait for this lock
to go away while holding all locks which were acquired so
far. Back-off is only used in situation when further waiting
will cause a deadlock which could be avoided by performing
back-off and restarting open_tables() process. Absence of
waiting between back-off and restart of acquiring metadata
locks can't lead to livelocks as MDL subsystem was changed
to make release of lock and granting it to waiting lock
an atomic action, so back-off will automatically give way
to other participants of deadlock loop.
Accordingly:
- open_table_get_mdl_lock() and open_and_process_routine()
were changed to wait for conflicting metadata lock to
go away without back-off. Only if such wait leads to a
deadlock back-off is requested. As part of this change
new error handler class was introduced which converts,
if possible, ER_LOCK_DEADLOCK error to a request for
back-off and re-start of open_tables() process.
- Open_table_context::recover_from_failed_open() was changed
not to wait in case of metadata lock conflict. Instead we
immediately proceed to re-acquiring locks.
- Open_table_context::request_backoff_action() now always
emits error if back-off is requested in the middle of
transaction as we can't be sure that releasing lock
which were acquired only by current statement will
resolve a deadlock. Before this patch such situations were
successfully detected thanks to the fact that we called
MDL_context::wait_for_lock() method in
recover_from_failed_open().
- In order to avoid deadlocks open_tables() code was adjusted
to flush open HANDLERs for which there are pending requests
for X locks before restarting the process of acquiring
metadata locks.
- Changed close_tables_for_reopen() not to reset MDL_request
for tables belonging to the tail of prelocking list. It is
no longer necessary as these MDL_request objects won't be
used for any waiting.
- Adjusted comment in tdc_wait_for_old_version() to avoid
mentioning removed MDL_context::wait_for_lock() method.
sql/sql_base.h:
As we no longer wait for conflicting metadata lock away in
Open_table_context::recover_from_failed_open() method,
Open_table_context::OT_WAIT_MDL_LOCK action was renamed to
OT_MDL_CONFLICT.
Also Open_table_context::m_failed_mdl_request became
unnecessary and was removed.
sql/sql_plist.h:
Extended I_P_List template to support efficient push_back()
operation if it is parameterized with an appropriate policy
class.
sql/sql_show.cc:
Adjusted code after removal of MDL_context::wait_for_lock()
method. Now if one needs to acquire metadata lock with waiting
one has to use a variant of MDL_context::acquire_lock() method.
Before this fix, the performance schema instrumentation
in mdl.h / mdl.cc was incomplete, causing:
- build warnings,
- no data collection for the performance schema
This fix:
- added instrumentation helpers for the new preferred
reader read write lock, mysql_prlock_*
- implemented completely the performance schema
instrumentation of mdl.h / mdl.cc
Before this fix, mysql_upgrade would always drop and re create
the performance_schema database.
This in theory could destroy user data created using 5.1 or older versions.
With this fix, mysql_upgrade checks the content of the
performance_schema database before droping it.
Before this fix, the performance schema file instrumentation would treat:
- a relative path to a file
- an absolute path to the same file
as two different files.
This would lead to:
- separate aggregation counters
- file leaks when a file is removed.
With this fix, a relative and absolute path are resolved to the same file instrument.
When compiling wiht ./configure --with-ssl=/usr,
which used OPEN_SSL but not YASSL, the code in sql/mysqld.cc
failed to build because of an incomplete performance schema instrumentation.
This fix implements properly the instrumentation for the rwlock
used in openssl_lock_t.
Verified that the code builds, and the ssl + performance schema tests
do pass.
Problem: The test case failed because: (i) warning text in
result file differed from the warning output by the
server, and (ii) binlog contents in result file did
not show the statements logged wrapped in BEGIN/COMMIT
as it is the case after WL 2687.
Solution: We update the result file, but first we change the
unsafe warning text to also refer to performance_schema
table(s). This required changing the result files for
existing test cases that provide output for warnings
related to ER_BINLOG_UNSAFE_SYSTEM_TABLE. "Grepping" in
result files, shows that only binlog_unsafe contained
reference to such a warning.
We also update the result file with the missing
BEGIN/COMMIT statements.
