mirror of
https://github.com/MariaDB/server.git
synced 2025-01-20 22:12:30 +01:00
2870bd7423
SHOW PROCESSLIST, SHOW BINLOGS
Problem: A deadlock was occurring when 4 threads were
involved in acquiring locks in the following way
Thread 1: Dump thread ( Slave is reconnecting, so on
Master, a new dump thread is trying kill
zombie dump threads. It acquired thread's
LOCK_thd_data and it is about to acquire
mysys_var->current_mutex ( which LOCK_log)
Thread 2: Application thread is executing show binlogs and
acquired LOCK_log and it is about to acquire
LOCK_index.
Thread 3: Application thread is executing Purge binary logs
and acquired LOCK_index and it is about to
acquire LOCK_thread_count.
Thread 4: Application thread is executing show processlist
and acquired LOCK_thread_count and it is
about to acquire zombie dump thread's
LOCK_thd_data.
Deadlock Cycle:
Thread 1 -> Thread 2 -> Thread 3-> Thread 4 ->Thread 1
The same above deadlock was observed even when thread 4 is
executing 'SELECT * FROM information_schema.processlist' command and
acquired LOCK_thread_count and it is about to acquire zombie
dump thread's LOCK_thd_data.
Analysis:
There are four locks involved in the deadlock. LOCK_log,
LOCK_thread_count, LOCK_index and LOCK_thd_data.
LOCK_log, LOCK_thread_count, LOCK_index are global mutexes
where as LOCK_thd_data is local to a thread.
We can divide these four locks in two groups.
Group 1 consists of LOCK_log and LOCK_index and the order
should be LOCK_log followed by LOCK_index.
Group 2 consists of other two mutexes
LOCK_thread_count, LOCK_thd_data and the order should
be LOCK_thread_count followed by LOCK_thd_data.
Unfortunately, there is no specific predefined lock order defined
to follow in the MySQL system when it comes to locks across these
two groups. In the above problematic example,
there is no problem in the way we are acquiring the locks
if you see each thread individually.
But If you combine all 4 threads, they end up in a deadlock.
Fix:
Since everything seems to be fine in the way threads are taking locks,
In this patch We are changing the duration of the locks in Thread 4
to break the deadlock. i.e., before the patch, Thread 4
('show processlist' command) mysqld_list_processes()
function acquires LOCK_thread_count for the complete duration
of the function and it also acquires/releases
each thread's LOCK_thd_data.
LOCK_thread_count is used to protect addition and
deletion of threads in global threads list. While show
process list is looping through all the existing threads,
it will be a problem if a thread is exited but there is no problem
if a new thread is added to the system. Hence a new mutex is
introduced "LOCK_thd_remove" which will protect deletion
of a thread from global threads list. All threads which are
getting exited should acquire LOCK_thd_remove
followed by LOCK_thread_count. (It should take LOCK_thread_count
also because other places of the code still thinks that exit thread
is protected with LOCK_thread_count. In this fix, we are changing
only 'show process list' query logic )
(Eg: unlink_thd logic will be protected with
LOCK_thd_remove).
Logic of mysqld_list_processes(or file_schema_processlist)
will now be protected with 'LOCK_thd_remove' instead of
'LOCK_thread_count'.
Now the new locking order after this patch is:
LOCK_thd_remove -> LOCK_thd_data -> LOCK_log ->
LOCK_index -> LOCK_thread_count
|
||
|---|---|---|
| .. | ||
| collections | ||
| extra | ||
| include | ||
| lib | ||
| r | ||
| std_data | ||
| suite | ||
| t | ||
| CMakeLists.txt | ||
| mtr.out-of-source | ||
| mysql-stress-test.pl | ||
| mysql-test-run.pl | ||
| purify.supp | ||
| README | ||
| README.gcov | ||
| README.stress | ||
| valgrind.supp | ||
This directory contains a test suite for the MySQL daemon. To run the currently existing test cases, simply execute ./mysql-test-run in this directory. It will fire up the newly built mysqld and test it. Note that you do not have to have to do "make install", and you could actually have a co-existing MySQL installation. The tests will not conflict with it. All tests must pass. If one or more of them fail on your system, please read the following manual section for instructions on how to report the problem: http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html If you want to use an already running MySQL server for specific tests, use the --extern option to mysql-test-run. Please note that in this mode, the test suite expects you to provide the names of the tests to run. For example, here is the command to run the "alias" and "analyze" tests with an external server: mysql-test-run --extern alias analyze To match your setup, you might also need to provide --socket, --user, and other relevant options. With no test cases named on the command line, mysql-test-run falls back to the normal "non-extern" behavior. The reason for this is that some tests cannot run with an external server. You can create your own test cases. To create a test case, create a new file in the t subdirectory using a text editor. The file should have a .test extension. For example: xemacs t/test_case_name.test In the file, put a set of SQL statements that create some tables, load test data, and run some queries to manipulate it. We would appreciate it if you name your test tables t1, t2, t3 ... (to not conflict too much with existing tables). Your test should begin by dropping the tables you are going to create and end by dropping them again. This ensures that you can run the test over and over again. If you are using mysqltest commands (like result file names) in your test case, you should create the result file as follows: mysql-test-run --record test_case_name or mysqltest --record < t/test_case_name.test If you only have a simple test cases consisting of SQL statements and comments, you can create the test case in one of the following ways: mysql-test-run --record test_case_name mysql test < t/test_case_name.test > r/test_case_name.result mysqltest --record --record-file=r/test_case_name.result < t/test_case_name.test When this is done, take a look at r/test_case_name.result - If the result is incorrect, you have found a bug. In this case, you should edit the test result to the correct results so that we can verify that the bug is corrected in future releases. To submit your test case, put your .test file and .result file(s) into a tar.gz or zip archive, create a bug report at http://bugs.mysql.com/ and attach the archive to the bug report.