2011-07-02 22:12:12 +02:00
|
|
|
#error don't use
|
2011-01-14 09:40:18 +01:00
|
|
|
/*
|
Bug#17283409 4-WAY DEADLOCK: ZOMBIES, PURGING BINLOGS,
SHOW PROCESSLIST, SHOW BINLOGS
Problem: A deadlock was occurring when 4 threads were
involved in acquiring locks in the following way
Thread 1: Dump thread ( Slave is reconnecting, so on
Master, a new dump thread is trying kill
zombie dump threads. It acquired thread's
LOCK_thd_data and it is about to acquire
mysys_var->current_mutex ( which LOCK_log)
Thread 2: Application thread is executing show binlogs and
acquired LOCK_log and it is about to acquire
LOCK_index.
Thread 3: Application thread is executing Purge binary logs
and acquired LOCK_index and it is about to
acquire LOCK_thread_count.
Thread 4: Application thread is executing show processlist
and acquired LOCK_thread_count and it is
about to acquire zombie dump thread's
LOCK_thd_data.
Deadlock Cycle:
Thread 1 -> Thread 2 -> Thread 3-> Thread 4 ->Thread 1
The same above deadlock was observed even when thread 4 is
executing 'SELECT * FROM information_schema.processlist' command and
acquired LOCK_thread_count and it is about to acquire zombie
dump thread's LOCK_thd_data.
Analysis:
There are four locks involved in the deadlock. LOCK_log,
LOCK_thread_count, LOCK_index and LOCK_thd_data.
LOCK_log, LOCK_thread_count, LOCK_index are global mutexes
where as LOCK_thd_data is local to a thread.
We can divide these four locks in two groups.
Group 1 consists of LOCK_log and LOCK_index and the order
should be LOCK_log followed by LOCK_index.
Group 2 consists of other two mutexes
LOCK_thread_count, LOCK_thd_data and the order should
be LOCK_thread_count followed by LOCK_thd_data.
Unfortunately, there is no specific predefined lock order defined
to follow in the MySQL system when it comes to locks across these
two groups. In the above problematic example,
there is no problem in the way we are acquiring the locks
if you see each thread individually.
But If you combine all 4 threads, they end up in a deadlock.
Fix:
Since everything seems to be fine in the way threads are taking locks,
In this patch We are changing the duration of the locks in Thread 4
to break the deadlock. i.e., before the patch, Thread 4
('show processlist' command) mysqld_list_processes()
function acquires LOCK_thread_count for the complete duration
of the function and it also acquires/releases
each thread's LOCK_thd_data.
LOCK_thread_count is used to protect addition and
deletion of threads in global threads list. While show
process list is looping through all the existing threads,
it will be a problem if a thread is exited but there is no problem
if a new thread is added to the system. Hence a new mutex is
introduced "LOCK_thd_remove" which will protect deletion
of a thread from global threads list. All threads which are
getting exited should acquire LOCK_thd_remove
followed by LOCK_thread_count. (It should take LOCK_thread_count
also because other places of the code still thinks that exit thread
is protected with LOCK_thread_count. In this fix, we are changing
only 'show process list' query logic )
(Eg: unlink_thd logic will be protected with
LOCK_thd_remove).
Logic of mysqld_list_processes(or file_schema_processlist)
will now be protected with 'LOCK_thd_remove' instead of
'LOCK_thread_count'.
Now the new locking order after this patch is:
LOCK_thd_remove -> LOCK_thd_data -> LOCK_log ->
LOCK_index -> LOCK_thread_count
2014-05-08 14:43:01 +02:00
|
|
|
Copyright (c) 2010, 2014, Oracle and/or its affiliates. All rights reserved.
|
2011-01-14 09:40:18 +01:00
|
|
|
|
|
|
|
This program is free software; you can redistribute it and/or modify
|
|
|
|
it under the terms of the GNU General Public License as published by
|
|
|
|
the Free Software Foundation; version 2 of the License.
|
|
|
|
|
|
|
|
This program is distributed in the hope that it will be useful,
|
|
|
|
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
GNU General Public License for more details.
|
|
|
|
|
|
|
|
You should have received a copy of the GNU General Public License
|
|
|
|
along with this program; if not, write to the Free Software
|
2011-06-30 17:46:53 +02:00
|
|
|
Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
|
2011-01-14 09:40:18 +01:00
|
|
|
*/
|
|
|
|
|
|
|
|
#ifndef THREAD_POOL_PRIV_INCLUDED
|
|
|
|
#define THREAD_POOL_PRIV_INCLUDED
|
|
|
|
|
2011-02-10 16:19:38 +01:00
|
|
|
/*
|
2011-02-17 10:48:44 +01:00
|
|
|
The thread pool requires access to some MySQL server error codes, this is
|
|
|
|
accessed from mysqld_error.h.
|
2011-02-17 10:57:57 +01:00
|
|
|
We need access to the struct that defines the thread pool plugin interface
|
|
|
|
which is accessed through scheduler.h.
|
2011-02-17 10:48:44 +01:00
|
|
|
All accesses to THD variables and functions are defined in this header file.
|
2011-02-10 16:19:38 +01:00
|
|
|
A thread pool can also use DEBUG_SYNC and must thus include
|
|
|
|
debug_sync.h
|
|
|
|
To handle definitions of Information Schema plugins it is also required
|
|
|
|
to include sql_profile.h and table.h.
|
|
|
|
*/
|
2011-02-17 10:48:44 +01:00
|
|
|
#include <mysqld_error.h> /* To get ER_ERROR_ON_READ */
|
2011-02-10 16:19:38 +01:00
|
|
|
#define MYSQL_SERVER 1
|
2011-02-17 10:50:51 +01:00
|
|
|
#include <scheduler.h>
|
2011-02-10 16:19:38 +01:00
|
|
|
#include <debug_sync.h>
|
|
|
|
#include <sql_profile.h>
|
|
|
|
#include <table.h>
|
2012-07-25 12:54:18 +02:00
|
|
|
#include <sql_list.h>
|
2011-02-10 16:19:38 +01:00
|
|
|
|
2011-02-10 19:24:31 +01:00
|
|
|
/* Needed to get access to scheduler variables */
|
2011-02-16 19:30:02 +01:00
|
|
|
void* thd_get_scheduler_data(THD *thd);
|
|
|
|
void thd_set_scheduler_data(THD *thd, void *data);
|
2011-02-10 19:24:31 +01:00
|
|
|
PSI_thread* thd_get_psi(THD *thd);
|
2011-02-16 19:30:02 +01:00
|
|
|
void thd_set_psi(THD *thd, PSI_thread *psi);
|
|
|
|
|
|
|
|
/* Interface to THD variables and functions */
|
|
|
|
void thd_set_killed(THD *thd);
|
|
|
|
void thd_clear_errors(THD *thd);
|
|
|
|
void thd_set_thread_stack(THD *thd, char *stack_start);
|
2011-04-15 13:02:29 +02:00
|
|
|
void thd_lock_thread_count(THD *thd);
|
|
|
|
void thd_unlock_thread_count(THD *thd);
|
Bug#17283409 4-WAY DEADLOCK: ZOMBIES, PURGING BINLOGS,
SHOW PROCESSLIST, SHOW BINLOGS
Problem: A deadlock was occurring when 4 threads were
involved in acquiring locks in the following way
Thread 1: Dump thread ( Slave is reconnecting, so on
Master, a new dump thread is trying kill
zombie dump threads. It acquired thread's
LOCK_thd_data and it is about to acquire
mysys_var->current_mutex ( which LOCK_log)
Thread 2: Application thread is executing show binlogs and
acquired LOCK_log and it is about to acquire
LOCK_index.
Thread 3: Application thread is executing Purge binary logs
and acquired LOCK_index and it is about to
acquire LOCK_thread_count.
Thread 4: Application thread is executing show processlist
and acquired LOCK_thread_count and it is
about to acquire zombie dump thread's
LOCK_thd_data.
Deadlock Cycle:
Thread 1 -> Thread 2 -> Thread 3-> Thread 4 ->Thread 1
The same above deadlock was observed even when thread 4 is
executing 'SELECT * FROM information_schema.processlist' command and
acquired LOCK_thread_count and it is about to acquire zombie
dump thread's LOCK_thd_data.
Analysis:
There are four locks involved in the deadlock. LOCK_log,
LOCK_thread_count, LOCK_index and LOCK_thd_data.
LOCK_log, LOCK_thread_count, LOCK_index are global mutexes
where as LOCK_thd_data is local to a thread.
We can divide these four locks in two groups.
Group 1 consists of LOCK_log and LOCK_index and the order
should be LOCK_log followed by LOCK_index.
Group 2 consists of other two mutexes
LOCK_thread_count, LOCK_thd_data and the order should
be LOCK_thread_count followed by LOCK_thd_data.
Unfortunately, there is no specific predefined lock order defined
to follow in the MySQL system when it comes to locks across these
two groups. In the above problematic example,
there is no problem in the way we are acquiring the locks
if you see each thread individually.
But If you combine all 4 threads, they end up in a deadlock.
Fix:
Since everything seems to be fine in the way threads are taking locks,
In this patch We are changing the duration of the locks in Thread 4
to break the deadlock. i.e., before the patch, Thread 4
('show processlist' command) mysqld_list_processes()
function acquires LOCK_thread_count for the complete duration
of the function and it also acquires/releases
each thread's LOCK_thd_data.
LOCK_thread_count is used to protect addition and
deletion of threads in global threads list. While show
process list is looping through all the existing threads,
it will be a problem if a thread is exited but there is no problem
if a new thread is added to the system. Hence a new mutex is
introduced "LOCK_thd_remove" which will protect deletion
of a thread from global threads list. All threads which are
getting exited should acquire LOCK_thd_remove
followed by LOCK_thread_count. (It should take LOCK_thread_count
also because other places of the code still thinks that exit thread
is protected with LOCK_thread_count. In this fix, we are changing
only 'show process list' query logic )
(Eg: unlink_thd logic will be protected with
LOCK_thd_remove).
Logic of mysqld_list_processes(or file_schema_processlist)
will now be protected with 'LOCK_thd_remove' instead of
'LOCK_thread_count'.
Now the new locking order after this patch is:
LOCK_thd_remove -> LOCK_thd_data -> LOCK_log ->
LOCK_index -> LOCK_thread_count
2014-05-08 14:43:01 +02:00
|
|
|
void thd_lock_thread_remove(THD *thd);
|
|
|
|
void thd_unlock_thread_remove(THD *thd);
|
2011-02-16 19:30:02 +01:00
|
|
|
void thd_close_connection(THD *thd);
|
|
|
|
THD *thd_get_current_thd();
|
|
|
|
void thd_lock_data(THD *thd);
|
|
|
|
void thd_unlock_data(THD *thd);
|
|
|
|
bool thd_is_transaction_active(THD *thd);
|
|
|
|
int thd_connection_has_data(THD *thd);
|
|
|
|
void thd_set_net_read_write(THD *thd, uint val);
|
2012-07-25 12:54:18 +02:00
|
|
|
uint thd_get_net_read_write(THD *thd);
|
2011-02-16 19:30:02 +01:00
|
|
|
void thd_set_mysys_var(THD *thd, st_my_thread_var *mysys_var);
|
2012-07-25 12:54:18 +02:00
|
|
|
ulong thd_get_net_wait_timeout(THD *thd);
|
2011-02-16 20:38:35 +01:00
|
|
|
my_socket thd_get_fd(THD *thd);
|
2012-11-09 10:24:35 +01:00
|
|
|
int thd_store_globals(THD* thd);
|
2011-02-10 19:24:31 +01:00
|
|
|
|
2012-08-28 16:13:03 +02:00
|
|
|
THD *first_global_thread();
|
|
|
|
THD *next_global_thread(THD *thd);
|
2012-07-25 12:54:18 +02:00
|
|
|
|
2011-02-17 10:48:44 +01:00
|
|
|
/* Print to the MySQL error log */
|
|
|
|
void sql_print_error(const char *format, ...);
|
|
|
|
|
|
|
|
/* Store a table record */
|
|
|
|
bool schema_table_store_record(THD *thd, TABLE *table);
|
|
|
|
|
2011-01-14 09:40:18 +01:00
|
|
|
/*
|
2011-02-17 10:48:44 +01:00
|
|
|
The thread pool must be able to execute statements using the connection
|
2011-01-14 09:40:18 +01:00
|
|
|
state in THD object. This is the main objective of the thread pool to
|
|
|
|
schedule the start of these commands.
|
|
|
|
*/
|
|
|
|
bool do_command(THD *thd);
|
|
|
|
|
|
|
|
/*
|
|
|
|
The thread pool requires an interface to the connection logic in the
|
|
|
|
MySQL Server since the thread pool will maintain the event logic on
|
|
|
|
the TCP connection of the MySQL Server. Thus new connections, dropped
|
|
|
|
connections will be discovered by the thread pool and it needs to
|
|
|
|
ensure that the proper MySQL Server logic attached to these events is
|
|
|
|
executed.
|
|
|
|
*/
|
2011-02-17 10:48:44 +01:00
|
|
|
/* Initialise a new connection handler thread */
|
|
|
|
bool init_new_connection_handler_thread();
|
|
|
|
/* Set up connection thread before use as execution thread */
|
|
|
|
bool setup_connection_thread_globals(THD *thd);
|
|
|
|
/* Prepare connection as part of connection set-up */
|
2011-02-10 18:41:02 +01:00
|
|
|
bool thd_prepare_connection(THD *thd);
|
2011-02-17 10:48:44 +01:00
|
|
|
/* Release auditing before executing statement */
|
|
|
|
void mysql_audit_release(THD *thd);
|
|
|
|
/* Check if connection is still alive */
|
2011-02-10 18:41:02 +01:00
|
|
|
bool thd_is_connection_alive(THD *thd);
|
2011-02-17 10:48:44 +01:00
|
|
|
/* Close connection with possible error code */
|
|
|
|
void close_connection(THD *thd, uint errcode);
|
|
|
|
/* End the connection before closing it */
|
2011-01-14 09:40:18 +01:00
|
|
|
void end_connection(THD *thd);
|
2011-05-25 12:17:27 +02:00
|
|
|
/* Cleanup the THD object */
|
|
|
|
void thd_cleanup(THD *thd);
|
2011-05-18 09:38:05 +02:00
|
|
|
/* Decrement connection counter */
|
|
|
|
void dec_connection_count();
|
2011-02-17 10:48:44 +01:00
|
|
|
/* Destroy THD object */
|
2011-05-18 09:38:05 +02:00
|
|
|
void delete_thd(THD *thd);
|
2011-01-14 09:40:18 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
thread_created is maintained by thread pool when activated since
|
|
|
|
user threads are created by the thread pool (and also special
|
2011-02-16 20:38:35 +01:00
|
|
|
threads to maintain the thread pool). This is done through
|
|
|
|
inc_thread_created.
|
2011-02-17 10:48:44 +01:00
|
|
|
|
2011-01-14 09:40:18 +01:00
|
|
|
max_connections is needed to calculate the maximum number of threads
|
2011-02-16 20:38:35 +01:00
|
|
|
that is allowed to be started by the thread pool. The method
|
|
|
|
get_max_connections() gets reference to this variable.
|
2011-02-17 10:48:44 +01:00
|
|
|
|
2011-02-16 20:38:35 +01:00
|
|
|
connection_attrib is the thread attributes for connection threads,
|
|
|
|
the method get_connection_attrib provides a reference to these
|
|
|
|
attributes.
|
2011-01-14 09:40:18 +01:00
|
|
|
*/
|
2011-02-16 20:38:35 +01:00
|
|
|
void inc_thread_created(void);
|
|
|
|
ulong get_max_connections(void);
|
2011-02-17 10:48:44 +01:00
|
|
|
pthread_attr_t *get_connection_attrib(void);
|
2011-01-14 09:40:18 +01:00
|
|
|
#endif
|