The assertion happens when: (i) the master and slave are configured to
use the semisync plugin; (ii) the DBA disables semisync on the master;
(iii) and he also unsets the option to wait for slaves ACK even if the
semisync slave count reaches 0 during the waiting period. This
combination of factors makes the server run into an assertion as soon
as the last semisync slave disconnects and its dump thread exits.
The root of the problem is the fact that when the dump thread
disconnects and calls the observer hook transmit_stop, which ends up
calling ReplSemiSyncMaster::remove_slave, there is no check whether
the master has already disabled semisync or not. If it has, the then a
second call to the switch_off member function must be avoided.
The quick fix is to avoid calling switch_off if the DBA has disabled
the semisync plugin interactively on the master. Also, the switch_off
member function should only be called if the plugin has not been
switched off already. This is basically the pattern throughout the
rest of the semisync plugin and no other calls seem vulnerable to
similar crashes/assertions.
(This a backport of the patch to 5.5, which is also vulnerable.)
RPL_SEMI_SYNC_MASTER_ENABLED OFF.
Problem:
=======
If master is waiting for a reply from slave, at this time
set global rpl_semi_sync_master_enabled=OFF, the master
server will crash.
Analysis:
========
When master is waiting for a reply from slave, at this time
if semi sync is switched off on master, during switch off if
active transactions are present the transactions will be
cleared and "active_tranxs_" variable will be set to NULL.
When the waiting master connection finds that semi sync is
switched of it tries to access "active_tranxs_" without
checking if the transaction list exists or not. Accessing
NULL transaction list causes the crash.
Fix:
===
A check has been added to see a valid list exists before
accessing the "active_tranxs_".
plugin/semisync/semisync_master.cc:
Added check for the existence of valid 'active_transx_'.
If rpl_semi_sync_master_timeout is large, calculation of absolute waiting time in semi-sync plugin is inefficient. This error is specific to systems with 64 bit long values (all 64 bit Unixes)
In rpl_semi_sync_master_timeout has maximal value (= MAX_ULONGLONG), calculating abstime may require ~ 18 billion subtract operations.
The fix is to use division instead of subtraction-in-a-loop. Also fixed an integer overflow bug.
and small collateral changes
mysql-test/lib/My/Test.pm:
somehow with "print" we get truncated writes sometimes
mysql-test/suite/perfschema/r/digest_table_full.result:
md5 hashes of statement digests differ, because yacc token codes are different in mariadb
mysql-test/suite/perfschema/r/dml_handler.result:
host table is not ported over yet
mysql-test/suite/perfschema/r/information_schema.result:
host table is not ported over yet
mysql-test/suite/perfschema/r/nesting.result:
this differs, because we don't rewrite general log queries, and multi-statement
packets are logged as a one entry. this result file is identical to what mysql-5.6.5
produces with the --log-raw option.
mysql-test/suite/perfschema/r/relaylog.result:
MariaDB modifies the binlog index file directly, while MySQL 5.6 has a feature "crash-safe binlog index" and modifies a special "crash-safe" shadow copy of the index file and then moves it over. That's why this test shows "NONE" index file writes in MySQL and "MANY" in MariaDB.
mysql-test/suite/perfschema/r/server_init.result:
MariaDB initializes the "manager" resources from the "manager" thread, and starts this thread only when --flush-time is not 0. MySQL 5.6 initializes "manager" resources unconditionally on server startup.
mysql-test/suite/perfschema/r/stage_mdl_global.result:
this differs, because MariaDB disables query cache when query_cache_size=0. MySQL does not
do that, and this causes useless mutex locks and waits.
mysql-test/suite/perfschema/r/statement_digest.result:
md5 hashes of statement digests differ, because yacc token codes are different in mariadb
mysql-test/suite/perfschema/r/statement_digest_consumers.result:
md5 hashes of statement digests differ, because yacc token codes are different in mariadb
mysql-test/suite/perfschema/r/statement_digest_long_query.result:
md5 hashes of statement digests differ, because yacc token codes are different in mariadb
mysql-test/suite/rpl/r/rpl_mixed_drop_create_temp_table.result:
will be updated to match 5.6 when alfranio.correia@oracle.com-20110512172919-c1b5kmum4h52g0ni and anders.song@greatopensource.com-20110105052107-zoab0bsf5a6xxk2y are merged
mysql-test/suite/rpl/r/rpl_non_direct_mixed_mixing_engines.result:
will be updated to match 5.6 when anders.song@greatopensource.com-20110105052107-zoab0bsf5a6xxk2y is merged
client/mysqldump.c:
Slave needs to be initialized with 0
dbug/dbug.c:
Removed not existing function
plugin/semisync/semisync_master.cc:
Fixed compiler warning
sql/opt_range.cc:
thd needs to be set early as it's used in some error conditions.
sql/sql_table.cc:
Changed to use uchar* to make array indexing portable
storage/innobase/handler/ha_innodb.cc:
Removed not used variable
storage/maria/ma_delete.c:
Fixed compiler warning
storage/maria/ma_write.c:
Fixed compiler warning
The semisync code does a fast-but-unsafe check for enabled or not without lock,
followed by a slow-but-safe check under lock. However, if the slow check failed,
the code still referenced not valid data (in an assert() expression), causing a
crash.
Fixed by not running the incorrect assert when semisync is disabled.
bug in semisync plugin. It didn't check thd->killed before waiting on mysys->current_cond,
and thus an attepmt to kill the thread (on shutdown) was lost
plugin/semisync/semisync_master.cc:
mdev:201 lp:962540
The problem was because the gettimeofday function was incorrect
implemented for Windows, and so the semisync master did not wait
for slave reply properly on Windows.
Fixed by removing the gettimeofday function for Windows, and using
set_timespec function to get current time for all platforms.
plugin/semisync/semisync_master.cc:
removing the gettimeofday function for Windows, using set_timespec
function to get current time for all platforms.
The root cause of the crash is that a TranxNode is freed before it is used.
A TranxNode is allocated and inserted into the active list each time
a log event is written and flushed into the binlog file.
The memory for TranxNode is allocated with thd_alloc and will be freed
at the end of the statement. The after_commit/after_rollback callback
was supposed to be called before the end of each statement and remove the node from
the active list. However this assumption is not correct in all cases(e.g. call
'CREATE TEMPORARY TABLE myisam_t SELECT * FROM innodb_t' in a transaction
and delete all temporary tables automatically when a session closed),
and can cause the memory allocated for TranxNode be freed
before it was removed from the active list. So The TranxNode pointer in the active
list would become a wild pointer and cause the crash.
After this patch, We have a class called a TranxNodeAllocate which manages the memory
for allocating and freeing TranxNode. It uses my_malloc to allocate memory.
sql/rpl_handler.cc:
params are not initialized.
Added back n_frees, use 'clear' instead of 'free' since memory is
not freed here.
plugin/semisync/semisync_master.cc:
Added back n_frees, use 'clear' instead of 'free' in the message since memory is not freed here.
Before this patch, semisync assumed transactions running in parallel
can not be larger than max_connections, but this is not true when
the event scheduler is executing events, and cause semisync run out
of preallocated transaction nodes.
Fix the problem by allocating transaction nodes dynamically.
This patch also fixed a possible deadlock when running UNINSTALL
PLUGIN rpl_semi_sync_master and updating in parallel. Fixed by
releasing the internal Delegate lock before unlock the plugins.
mysql-test/suite/rpl/t/rpl_semi_sync_event.test:
Add test case for bug#49020
plugin/semisync/semisync_master.cc:
Allocating TranxNode dynamically
plugin/semisync/semisync_master.h:
Allocating TranxNode dynamically
sql/rpl_handler.cc:
Unlock plugins after we have released the Delegate lock to avoid possible deadlock when uninstalling semisync master plugin and doing update in parallel.
CMakeLists.txt:
Add plugin/semisync subdirectory
mysql-test/mysql-test-run.pl:
Check for semisync dll for Windows
mysql-test/suite/rpl/r/rpl_semi_sync.result:
Update result file
mysql-test/suite/rpl/t/rpl_semi_sync.test:
Test semi-sync on Windows
plugin/semisync/semisync_master.cc:
Define gettimeofday for Windows
Add an option to control whether the master should keep waiting
until timeout when it detected that there is no semi-sync slave
available.
The bool option 'rpl_semi_sync_master_wait_no_slave' is 1 by
defalt, and will keep waiting until timeout. When set to 0, the
master will switch to asynchronous replication immediately when
no semi-sync slave is available.
Semi-sync status were not reset by FLUSH STATUS, this was because
all semi-sync status variables are defined as SHOW_FUNC and FLUSH
STATUS could only reset SHOW_LONG type variables.
This problem is fixed by change all status variables that should
be reset by FLUSH STATUS from SHOW_FUNC to SHOW_LONG.
After the fix, the following status variables will be reset by
FLUSH STATUS:
Rpl_semi_sync_master_yes_tx
Rpl_semi_sync_master_no_tx
Note: normally, FLUSH STATUS itself will be written into binlog
and be replicated, so after FLUSH STATS, one of
Rpl_semi_sync_master_yes_tx
Rpl_semi_sync_master_no_tx
can be 1 dependent on the semi-sync status. So it's recommended
to use FLUSH NO_WRITE_TO_BINLOG STATUS to avoid this.
Semi-sync uses an extra connection from slave to master to send
replies, this is a normal client connection, and used a normal
SET query to set the reply information on master, which is visible
to user and may cause some confusion and complaining.
This problem is fixed by using the method of sending reply by
using the same connection that is used by master dump thread to
send binlog to slave. Since now the semi-sync plugins are integrated
with the server code, it is not a problem to use the internal net
interfaces to do this.
The master dump thread will mark the event requires a reply and
wait for the reply when the event just sent is the last event
of a transaction and semi-sync status is ON; And the slave will
send a reply to master when it received such an event that requires
a reply.
On sparc, semisync master/slave status is always showed as OFF, this
is fixed by change rpl_semisync_master/slave_status variables from
long to char.
plugin/semisync/semisync_master.cc:
Change rpl_semisync_master_status variables from long to char
plugin/semisync/semisync_master.h:
Change rpl_semisync_master_status variables from long to char
plugin/semisync/semisync_slave.cc:
Change rpl_semisync_slave_status variables from long to char
plugin/semisync/semisync_slave.h:
Change rpl_semisync_slave_status variables from long to char