mirror of
https://github.com/MariaDB/server.git
synced 2025-04-07 15:55:41 +02:00
3f59bbeeae
The problem seems to be a deadlock between KILL command execution and BF abort issued by an applier, where: * KILL has locked victim's LOCK_thd_kill and LOCK_thd_data. * Applier has innodb side global lock mutex and victim trx mutex. * KILL is calling innobase_kill_query, and is blocked by innodb global lock mutex. * Applier is in wsrep_innobase_kill_one_trx and is blocked by victim's LOCK_thd_kill. The fix in this commit removes the TOI replication of KILL command and makes KILL execution less intrusive operation. Aborting the victim happens now by using awake_no_mutex() and ha_abort_transaction(). If the KILL happens when the transaction is committing, the KILL operation is postponed to happen after the statement has completed in order to avoid KILL to interrupt commit processing. Notable changes in this commit: * wsrep client connections's error state may remain sticky after client connection is closed. This error message will then pop up for the next client session issuing first SQL statement. This problem raised with test galera.galera_bf_kill. The fix is to reset wsrep client error state, before a THD is reused for next connetion. * Release THD locks in wsrep_abort_transaction when locking innodb mutexes. This guarantees same locking order as with applier BF aborting. * BF abort from MDL was changed to do BF abort on server/wsrep-lib side first, and only then do the BF abort on InnoDB side. This removes the need to call back from InnoDB for BF aborts which originate from MDL and simplifies the locking. * Removed wsrep_thd_set_wsrep_aborter() from service_wsrep.h. The manipulation of the wsrep_aborter can be done solely on server side. Moreover, it is now debug only variable and could be excluded from optimized builds. * Remove LOCK_thd_kill from wsrep_thd_LOCK/UNLOCK to allow more fine grained locking for SR BF abort which may require locking of victim LOCK_thd_kill. Added explicit call for wsrep_thd_kill_LOCK/UNLOCK where appropriate. * Wsrep-lib was updated to version which allows external locking for BF abort calls. Changes to MTR tests: * Disable galera_bf_abort_group_commit. This test is going to be removed (MDEV-30855). * Record galera_gcache_recover_manytrx as result file was incomplete. Trivial change. * Make galera_create_table_as_select more deterministic: Wait until CTAS execution has reached MDL wait for multi-master conflict case. Expected error from multi-master conflict is ER_QUERY_INTERRUPTED. This is because CTAS does not yet have open wsrep transaction when it is waiting for MDL, query gets interrupted instead of BF aborted. This should be addressed in separate task. * A new test galera_kill_group_commit to verify correct behavior when KILL is executed while the transaction is committing. Co-authored-by: Seppo Jaakola <seppo.jaakola@iki.fi> Co-authored-by: Jan Lindström <jan.lindstrom@galeracluster.com> Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
69 lines
2.3 KiB
Text
69 lines
2.3 KiB
Text
#
|
|
# Verify that transaction which has reached group commit queue
|
|
# cannot be killed. If the kill succeeds, assertion for
|
|
# wsrep transaction state will fail.
|
|
#
|
|
# If the bug is present, i.e. wsrep transaction gets killed during
|
|
# group commit wait, this test is enough to reproduce the crash
|
|
# most of the time.
|
|
#
|
|
|
|
--source include/have_innodb.inc
|
|
--source include/have_debug_sync.inc
|
|
--source include/galera_cluster.inc
|
|
|
|
# Connection for KILL commands
|
|
--connect node_1_kill, 127.0.0.1, root, , test, $NODE_MYPORT_1
|
|
# Connection for sync point control
|
|
--connect node_1_ctrl, 127.0.0.1, root, , test, $NODE_MYPORT_1
|
|
SET SESSION wsrep_sync_wait = 0;
|
|
# Connection for group commit follower
|
|
--connect node_1_follower, 127.0.0.1, root, , test, $NODE_MYPORT_1
|
|
# Need to disable sync wait to reach commit queue when leader
|
|
# is blocked.
|
|
SET SESSION wsrep_sync_wait = 0;
|
|
--let $follower_id = `SELECT CONNECTION_ID()`
|
|
|
|
--connection node_1
|
|
CREATE TABLE t1 (f1 INT PRIMARY KEY) ENGINE=InnoDB;
|
|
|
|
SET SESSION DEBUG_SYNC = "commit_before_enqueue SIGNAL leader_before_enqueue_reached WAIT_FOR leader_before_enqueue_continue";
|
|
--send INSERT INTO t1 VALUES (1)
|
|
|
|
--connection node_1_ctrl
|
|
SET DEBUG_SYNC = "now WAIT_FOR leader_before_enqueue_reached";
|
|
|
|
--connection node_1_follower
|
|
# SET SESSION DEBUG_SYNC = "group_commit_waiting_for_prior SIGNAL follower_waiting_for_prior_reached WAIT_FOR follower_waiting_for_prior_continue";
|
|
--send INSERT INTO t1 VALUES (2);
|
|
|
|
--connection node_1_ctrl
|
|
# TODO: Is it possible to use sync points to enforce group commit to happen?
|
|
# The leader will hold commit monitor in commit_before_enqueue sync point,
|
|
# which prevents the follower to reach the group commit wait state.
|
|
# We now sleep and expect the follower to reach group commit, but this
|
|
# may cause false negatives.
|
|
--sleep 1
|
|
|
|
--connection node_1_kill
|
|
--echo # Execute KILL QUERY for group commit follower
|
|
--disable_query_log
|
|
--disable_result_log
|
|
# Because it is currently impossible to verify that the
|
|
# follower has reached group commit queue, the KILL may
|
|
# sometimes return success.
|
|
--error 0,ER_KILL_DENIED_ERROR
|
|
--eval KILL QUERY $follower_id
|
|
--enable_result_log
|
|
--enable_query_log
|
|
|
|
SET DEBUG_SYNC = "now SIGNAL leader_before_enqueue_continue";
|
|
--connection node_1_follower
|
|
--reap
|
|
|
|
--connection node_1
|
|
--reap
|
|
SELECT * FROM t1;
|
|
|
|
SET DEBUG_SYNC = "RESET";
|
|
DROP TABLE t1;
|