The mentioned on the bug report set of bugs fixes have not be pushed to the main trees.
Fixed with extracting commits done to 6.0-rpl tree and applying them to the main 5.1.
Notes.
1. part of changes - the mtr's specific - were packported to the main 5.0 tree for mtr v1
as http://lists.mysql.com/commits/46562
However, there is no that fix anymore in the mtr v2. (This fact was mailed to mtr maintaining
people).
2. Bug@36929 crash in kill_zombie_dump_threads-> THD::awake() with replication tests
is not backported because the base code of the patch is libevent and that was removed
from the main trees due to its instability.
client/mysqlbinlog.cc:
fixes for BUG#35546
mysql-test/suite/rpl/r/rpl_bug41902.result:
the new tests result file is added.
mysql-test/suite/rpl/t/rpl_bug41902-slave.opt:
conf file for bug41902 testing is added.
mysql-test/suite/rpl/t/rpl_bug41902.test:
regression tests for Bug #41902 is added.
sql/log.cc:
collection of changes due to Bug #48463.
sql/log.h:
collection of changes due to Bug #48463.
sql/rpl_rli.h:
collection of changes due to Bug #48463.
sql/slave.cc:
collection of changes due to Bug #48463.
sql/sql_repl.cc:
collection of changes due to Bug #48463.
Before the patch, slaves only appear in the output of SHOW SLAVE HOSTS
when report-host option is set. If an expected slave does not appear in
the list, nobody knows whether the slave does not connect or has started
without the "report-host" option. The output also contains a strange
field "Rpl_recovery_rank" which has never been implemented and the manual
of MySQL5.4 declares that the field has been removed from MySQL5.4.
This patch is done with these,
According to the manual of MySQL5.4, "Rpl_recovery_rank" is removed.
Slaves will register themselves to master no matter if report_host option is set
or not. When slaves are registering themselves, their Server_ids, report_host
and other information are together sent to master. Sever_ids are never null
and is unique in one replication group. Slaves always can be identified with
different Server_ids no matter if report_host exists.
Implemented the server infrastructure for the fix:
1. Added a function LEX_STRING *thd_query_string(THD) to return
a LEX_STRING structure instead of char *.
This is the function that must be called in innodb instead of
thd_query()
2. Did some encapsulation in THD : aggregated thd_query and
thd_query_length into a LEX_STRING and made accessor and mutator
methods for easy code updating.
3. Updated the server code to use the new methods where applicable.
The BINLOG statement was sharing too much code with the slave SQL thread, introduced with
the patch for Bug#32407. This caused statements to be logged with the wrong server_id, the
id stored inside the events of the BINLOG statement rather than the id of the running
server.
Fix by rearranging code a bit so that only relevant parts of the code are executed by
the BINLOG statement, and the server_id of the server executing the statements will
not be overrided by the server_id stored in the 'format description BINLOG statement'.
mysql-test/extra/binlog_tests/binlog.test:
Added test to verify if the server_id stored in the 'format
description BINLOG statement' will override the server_id
of the server executing the statements.
mysql-test/suite/binlog/r/binlog_row_binlog.result:
Test result for bug#46640
mysql-test/suite/binlog/r/binlog_stm_binlog.result:
Test result for bug#46640
sql/log_event.cc:
Moved rows_event_stmt_clean() call from update_pos() to apply_event(). This in any case
makes more sense, and is needed as update_pos() is no longer called when executing
BINLOG statements.
Moved setting of rli->relay_log.description_event_for_exec from
Format_description_log_event::do_update_pos() to
Format_description_log_event::do_apply_event()
sql/log_event_old.cc:
Moved rows_event_stmt_clean() call from update_pos() to apply_event(). This in any case
makes more sense, and is needed as update_pos() is no longer called when executing
BINLOG statements.
sql/slave.cc:
The skip flag is no longer needed, as the code path for BINLOG statement has been
cleaned up.
sql/sql_binlog.cc:
Don't invoke the update_pos() code path for the BINLOG statement, as it contains code
that is redundant and/or harmful (especially setting thd->server_id).
mysql-test/suite/rpl/r/rpl_stop_middle_group.result:
the new result file
mysql-test/suite/rpl/t/rpl_stop_middle_group.test:
renamed from rpl_row_stop_middle_update and added a regression test for bug#45940.
There are three internal status for slave I/O thread, both
MYSQL_SLAVE_RUN_NOT_CONNECT and MYSQL_SLAVE_NOT_RUN are reported
as 'No' for Slave_IO_running of command SHOW SLAVE STATUS.
Change MYSQL_SLAVE_RUN_NOT_CONNECT to be reported as 'Connecting'.
NOTE: Backporting the patch to next-mr.
The slave was crashing while failing to execute the init_slave() function.
The issue stems from two different reasons:
1 - A failure while allocating the master info structure generated a
segfault due to a NULL pointer.
2 - A failure while recovering generated a segfault due to a non-initialized
relay log file. In other words, the mi->init and rli->init were both set to true
before executing the recovery process thus creating an inconsistent state as the
relay log file was not initialized.
To circumvent such problems, we refactored the recovery process which is now executed
while initializing the relay log. It is ensured that the master info structure is
created before accessing it and any error is propagated thus avoiding to set mi->init
and rli->init to true when for instance the relay log is not initialized or the relay
info is not flushed.
The changes related to the refactory are described below:
1 - Removed call to init_recovery from init_slave.
2 - Changed the signature of the function init_recovery.
3 - Removed flushes. They are called while initializing the relay log and master
info.
4 - Made sure that if the relay info is not flushed the mi-init and rli-init are not
set to true.
In this patch, we also replaced the exit(1) in the fault injection by DBUG_ABORT()
to make it compliant with the code guidelines.
On Mac OS X or Windows, sending a SIGHUP to the server or a
asynchronous flush (triggered by flush_time), would cause the
server to crash.
The problem was that a hook used to detach client API handles
wasn't prepared to handle cases where the thread does not have
a associated session.
The solution is to verify whether the thread has a associated
session before trying to detach a handle.
mysql-test/r/federated_debug.result:
Add test case result for Bug#47525
mysql-test/t/federated_debug-master.opt:
Debug point.
mysql-test/t/federated_debug.test:
Add test case for Bug#47525
sql/slave.cc:
Check whether a the thread has a associated session.
sql/sql_parse.cc:
Add debug code to simulate a reload without thread session.
NOTE: Backporting the patch to next-mr.
The fix proposed in BUG#35542 and BUG#31665 introduces a performance issue
when fsyncing the master.info, relay.info and relay-log.bin* after #th events.
Although such solution has been proposed to reduce the probability of corrupted
files due to a slave-crash, the performance penalty introduced by it has
made the approach impractical for highly intensive workloads.
In a nutshell, the option --syn-relay-log proposed in BUG#35542 and BUG#31665
simultaneously fsyncs master.info, relay-log.info and relay-log.bin* and
this is the main source of performance issues.
This patch introduces new options that give more control to the user on
what should be fsynced and how often:
1) (--sync-master-info, integer) which syncs the master.info after #th event;
2) (--sync-relay-log, integer) which syncs the relay-log.bin* after #th
events.
3) (--sync-relay-log-info, integer) which syncs the relay.info after #th
transactions.
To provide both performance and increased reliability, we recommend the following
setup:
1) --sync-master-info = 0 eventually the operating system will fsync it;
2) --sync-relay-log = 0 eventually the operating system will fsync it;
3) --sync-relay-log-info = 1 fsyncs it after every transaction;
Notice, that the previous setup does not reduce the probability of
corrupted master.info and relay-log.bin*. To overcome the issue, this patch also
introduces a recovery mechanism that right after restart throws away relay-log.bin*
retrieved from a master and updates the master.info based on the relay.info:
4) (--relay-log-recovery, boolean) which enables a recovery mechanism that
throws away relay-log.bin* after a crash.
However, it can only recover the incorrect binlog file and position in master.info,
if other informations (host, port password, etc) are corrupted or incorrect,
then this recovery mechanism will fail to work.
when compiled with Sun Studio compiler).
The thing is that Sun Studio compiler calls destructor of stack
objects when pthread_exit() is called. That triggered an assertion
in DBUG_ENTER()/DBUG_RETURN() validation logic (if DBUG_ENTER() is
used in the beginning of function, all returns should be replaced
by DBUG_RETURN/DBUG_VOID_RETURN macros).
A fix is to explicitly use DBUG_LEAVE macro.
But there is no Last_IO_Error reported.
On the master, if a binary log event is larger than max_allowed_packet,
ER_MASTER_FATAL_ERROR_READING_BINLOG and the specific reason of this error is
sent to a slave when it requests a dump from the master, thus leading
the I/O thread to stop.
On a slave, the I/O thread stops when receiving a packet larger than max_allowed_packet.
In both cases, however, there was no Last_IO_Error reported.
This patch adds code to report the Last_IO_Error and exact reason before stopping the
I/O thread and also reports the case the out memory pops up while
handling packets from the master.
sql/sql_repl.cc:
The master send the Specific reasons instead of "error reading log entry" to the slave which is requesting a dump.
if an fatal error is returned by read_log_event function.
If the SQL Thread fails to execute an event due to a temporary error (e.g.
ER_LOCK_DEADLOCK) and the option "--slave_transaction_retries" is set the SQL
Thread should not be aborted and the transaction should be restarted from the
beginning and re-executed.
Unfortunately, a wrong interpretation of the THD::is_fatal_error was preventing
this behavior. In a nutshell, "this variable is set to TRUE if an execution of a
compound statement cannot continue. In particular, it is used to disable access
to the CONTINUE or EXIT handlers of stored routines. So even temporary errors
may have this variable set.
To fix the bug, we have done what follows:
DBUG_ENTER("has_temporary_error");
- if (thd->is_fatal_error)
- DBUG_RETURN(0);
-
DBUG_EXECUTE_IF("all_errors_are_temporary_errors",
if (thd->main_da.is_error())
{
Bug#45243: crash on win in sql thread clear_tables_to_lock() -> free()
Bug#45242: crash on win in mysql_close() -> free()
Bug#45238: rpl_slave_skip, rpl_change_master failed (lost connection) for STOP SLAVE
Bug#46030: rpl_truncate_3innodb causes server crash on windows
Bug#46014: rpl_stm_reset_slave crashes the server sporadically in pb2
When killing a user session on the server, it's necessary to
interrupt (notify) the thread associated with the session that
the connection is being killed so that the thread is woken up
if waiting for I/O. On a few platforms (Mac, Windows and HP-UX)
where the SIGNAL_WITH_VIO_CLOSE flag is defined, this interruption
procedure is to asynchronously close the underlying socket of
the connection.
In order to enable this schema, each connection serving thread
registers its VIO (I/O interface) so that other threads can
access it and close the connection. But only the owner thread of
the VIO might delete it as to guarantee that other threads won't
see freed memory (the thread unregisters the VIO before deleting
it). A side note: closing the socket introduces a harmless race
that might cause a thread attempt to read from a closed socket,
but this is deemed acceptable.
The problem is that this infrastructure was meant to only be used
by server threads, but the slave I/O thread was registering the
VIO of a mysql handle (a client API structure that represents a
connection to another server instance) as a active connection of
the thread. But under some circumstances such as network failures,
the client API might destroy the VIO associated with a handle at
will, yet the VIO wouldn't be properly unregistered. This could
lead to accesses to freed data if a thread attempted to kill a
slave I/O thread whose connection was already broken.
There was a attempt to work around this by checking whether
the socket was being interrupted, but this hack didn't work as
intended due to the aforementioned race -- attempting to read
from the socket would yield a "bad file descriptor" error.
The solution is to add a hook to the client API that is called
from the client code before the VIO of a handle is deleted.
This hook allows the slave I/O thread to detach the active vio
so it does not point to freed memory.
server-tools/instance-manager/mysql_connection.cc:
Add stub method required for linking.
sql-common/client.c:
Invoke hook.
sql/client_settings.h:
Export hook.
sql/slave.cc:
Introduce hook that clears the active VIO before it is freed
by the client API.
procedures causes crashes!
The problem of that bugreport was mostly fixed by the
patch for bug 38691.
However, attached test case focused on another crash or
valgrind warning problem: SHOW PROCESSLIST query accesses
freed memory of SP instruction that run in a parallel
connection.
Changes of thd->query/thd->query_length in dangerous
places have been guarded with the per-thread
LOCK_thd_data mutex (the THD::LOCK_delete mutex has been
renamed to THD::LOCK_thd_data).
sql/ha_myisam.cc:
Bug #38816: kill + flush tables with read lock + stored
procedures causes crashes!
Modification of THD::query/query_length has been guarded
with the a THD::set_query() method call/LOCK_thd_data
mutex.
Unnecessary locking with the global LOCK_thread_count
mutex has been removed.
sql/log_event.cc:
Bug #38816: kill + flush tables with read lock + stored
procedures causes crashes!
Modification of THD::query/query_length has been guarded
with the THD::set_query()) method call/LOCK_thd_data
mutex.
sql/slave.cc:
Bug #38816: kill + flush tables with read lock + stored
procedures causes crashes!
Modification of THD::query/query_length has been guarded
with the THD::set_query() method call/LOCK_thd_data mutex.
The THD::LOCK_delete mutex has been renamed to
THD::LOCK_thd_data.
sql/sp_head.cc:
Bug #38816: kill + flush tables with read lock + stored
procedures causes crashes!
Modification of THD::query/query_length has been guarded
with the a THD::set_query() method call/LOCK_thd_data
mutex.
sql/sql_class.cc:
Bug #38816: kill + flush tables with read lock + stored
procedures causes crashes!
The new THD::LOCK_thd_data mutex and THD::set_query()
method has been added to guard modifications of THD::query/
THD::query_length fields, also the Statement::set_statement()
method has been overloaded in the THD class.
The THD::LOCK_delete mutex has been renamed to
THD::LOCK_thd_data.
sql/sql_class.h:
Bug #38816: kill + flush tables with read lock + stored
procedures causes crashes!
The new THD::LOCK_thd_data mutex and THD::set_query()
method has been added to guard modifications of THD::query/
THD::query_length fields, also the Statement::set_statement()
method has been overloaded in the THD class.
The THD::LOCK_delete mutex has been renamed to
THD::LOCK_thd_data.
sql/sql_insert.cc:
Bug #38816: kill + flush tables with read lock + stored
procedures causes crashes!
Modification of THD::query/query_length has been guarded
with the a THD::set_query() method call/LOCK_thd_data
mutex.
sql/sql_parse.cc:
Bug #38816: kill + flush tables with read lock + stored
procedures causes crashes!
Modification of THD::query/query_length has been guarded
with the a THD::set_query() method call/LOCK_thd_data mutex.
sql/sql_repl.cc:
Bug #38816: kill + flush tables with read lock + stored
procedures causes crashes!
The THD::LOCK_delete mutex has been renamed to
THD::LOCK_thd_data.
sql/sql_show.cc:
Bug #38816: kill + flush tables with read lock + stored
procedures causes crashes!
Inter-thread read of THD::query/query_length field has
been protected with a new per-thread LOCK_thd_data
mutex in the mysqld_list_processes function.
The "get_master_version_and_clock(...)" function in sql/slave.cc ignores
error and passes directly when queries fail, or queries succeed
but the result retrieved is empty.
The "get_master_version_and_clock(...)" function should try to reconnect master
if queries fail because of transient network problems, and fail otherwise.
The I/O thread should print a warning if the some system variables do not
exist on master (very old master)
mysql-test/extra/rpl_tests/rpl_get_master_version_and_clock.test:
Added test file for bug #45214
mysql-test/suite/rpl/r/rpl_get_master_version_and_clock.result:
Added test result for bug #45214
mysql-test/suite/rpl/t/rpl_get_master_version_and_clock.test:
Added test file for bug #45214
sql/slave.cc:
The 'is_network_error()' function is added for checking if the error is caused by network.
Added a new return value (2) to 'get_master_version_and_clock()' function result set
to indicate transient network errors when queries fail, and the caller should
try to reconnect in this case.
timeout
In STMT and MIXED modes, a statement that changes both non-transactional and
transactional tables must be written to the binary log whenever there are
changes to non-transactional tables. This means that the statement gets into the
binary log even when the changes to the transactional tables fail. In particular
, in the presence of a failure such statement is annotated with the error number
and wrapped in a begin/rollback. On the slave, while applying the statement, it
is expected the same failure and the rollback prevents the transactional changes
to be persisted.
Unfortunately, statements that fail due to concurrency issues (e.g. deadlocks,
timeouts) are logged in the same way causing the slave to stop as the statements
are applied sequentially by the SQL Thread. To fix this bug, we automatically
ignore concurrency failures on the slave. Specifically, the following failures
are ignored: ER_LOCK_WAIT_TIMEOUT, ER_LOCK_DEADLOCK and ER_XA_RBDEADLOCK.
The reason for the crash was rotate_relay_log (mi=0x0) did not verify
the passed value of active_mi. There are more cases where active_mi
is supposed to be non-zero e.g change_master(), stop_slave(), and it's
reasonable to protect from a similar crash all of them with common
fixes.
Fixed with spliting end_slave() in slave threads release and slave
data clean-up parts (a new close_active_mi()). The new function is
invoked at the very end of close_connections() so that all users of
active_mi are proven to have left.
sql/mysqld.cc:
added the 2nd part (data) of the slave's clean up.
sql/slave.cc:
end_slave() is split in two part to release the slave threads and the remained
resources separately.
The new close_active_mi() should be called after all possible users ofactive_mi
has left, i.e at the very end of close_connections().
sql/slave.h:
interface to the new end_active_mi() function is added.
with gcc 4.3.2
Compiling MySQL with gcc 4.3.2 and later produces a number of
warnings, many of which are new with the recent compiler
versions.
This bug will be resolved in more than one patch to limit the
size of changesets. This is the first patch, fixing a number
of the warnings, predominantly "suggest using parentheses
around && in ||", and empty for and while bodies.
with gcc 4.3.2
Compiling MySQL with gcc 4.3.2 and later produces a number of
warnings, many of which are new with the recent compiler
versions.
This bug will be resolved in more than one patch to limit the
size of changesets. This is the first patch, fixing a number
of the warnings, predominantly "suggest using parentheses
around && in ||", and empty for and while bodies.
The server was not cleaning the last IO error and error number when
resetting slave.
This patch addresses this issue by backporting into 5.1 part of the
patch in BUG 34654. A fix for this issue had already been pushed into
6.0 as part of the aforementioned bug, however the patch also included
some refactoring. The fix for 5.1 does not take into account the
refactoring part.
mysql-test/extra/rpl_tests/rpl_reset_slave.test:
Backported the test case and improved with deploying include/start_slave.inc
in relevant spots.
sql/slave.cc:
Backported part of patch from 6.0 that includes cleaning
mi->clear_error() at:
1. beginning of handle_slave_io
2. on successful connection
Also, backported the assertion added in the original patch.
sql/sql_repl.cc:
Backported the call to mi->clear_error() on reset_slave().
stop/start slave
When stopping and restarting the slave while it is replicating
temporary tables, the server would crash or raise an assertion
failure. This was due to the fact that although temporary tables are
saved between slave threads restart, the reference to the thread in
use (table->in_use) was not being properly updated when the restart
happened (it would still reference the old/invalid thread instead of
the new one).
This patch addresses this issue by resetting the reference to the new
slave thread on slave thread restart.
mysql-test/r/rpl_temporary.result:
Result file.
mysql-test/t/rpl_temporary.test:
Test case that checks that both failures go away.
sql/slave.cc:
Changed slave.cc to reset sql_thd reference in temporary tables.
165 changesets with 23 conflicts:
Text conflict in mysql-test/r/lock_multi.result
Text conflict in mysql-test/t/lock_multi.test
Text conflict in mysql-test/t/mysqldump.test
Text conflict in sql/item_strfunc.cc
Text conflict in sql/log.cc
Text conflict in sql/log_event.cc
Text conflict in sql/parse_file.cc
Text conflict in sql/slave.cc
Text conflict in sql/sp.cc
Text conflict in sql/sp_head.cc
Text conflict in sql/sql_acl.cc
Text conflict in sql/sql_base.cc
Text conflict in sql/sql_class.cc
Text conflict in sql/sql_crypt.cc
Text conflict in sql/sql_db.cc
Text conflict in sql/sql_lex.cc
Text conflict in sql/sql_parse.cc
Text conflict in sql/sql_select.cc
Text conflict in sql/sql_table.cc
Text conflict in sql/sql_view.cc
Text conflict in storage/innobase/handler/ha_innodb.cc
Text conflict in storage/myisam/mi_packrec.c
Text conflict in tests/mysql_client_test.c
Updates to Innobase, taken from main 5.1:
bzr: ERROR: Some change isn't sane:
File mysql-test/r/innodb-semi-consistent.result is owned by Innobase and should not be updated.
File mysql-test/t/innodb-semi-consistent.test is owned by Innobase and should not be updated.
File storage/innobase/handler/ha_innodb.cc is owned by Innobase and should not be updated.
File storage/innobase/ibuf/ibuf0ibuf.c is owned by Innobase and should not be updated.
File storage/innobase/include/row0mysql.h is owned by Innobase and should not be updated.
File storage/innobase/include/srv0srv.h is owned by Innobase and should not be updated.
File storage/innobase/include/trx0trx.h is owned by Innobase and should not be updated.
File storage/innobase/include/trx0trx.ic is owned by Innobase and should not be updated.
File storage/innobase/lock/lock0lock.c is owned by Innobase and should not be updated.
File storage/innobase/page/page0cur.c is owned by Innobase and should not be updated.
File storage/innobase/row/row0mysql.c is owned by Innobase and should not be updated.
File storage/innobase/row/row0sel.c is owned by Innobase and should not be updated.
File storage/innobase/srv/srv0srv.c is owned by Innobase and should not be updated.
File storage/innobase/trx/trx0trx.c is owned by Innobase and should not be updated.
(Set env var 'ALLOW_UPDATE_INNOBASE_OWNED' to override.)
The issue of the current bug is unguarded access to mi->slave_running
by the shutdown thread calling end_slave() that is bug#29968
(alas happened not to be cross-linked with the current bug)
Fixed:
with removing the unguarded read of the running status
and perform reading it in terminate_slave_thread()
at time run_lock is taken (mostly bug#29968 backporting, still with some
improvements over that patch - see the error reporting from
terminate_slave_thread()).
Issue of bug#38716 is fixed here for 5.0 branch as well.
Note:
There has been a separate artifact identified -
a race condition between init_slave() and end_slave() -
reported as Bug#44467.
mysql-test/r/rpl_bug38694.result:
a new results file is added.
mysql-test/t/rpl_bug38694-slave.opt:
simulating delay at slave threads shutdown.
mysql-test/t/rpl_bug38694.test:
A new test to check if a delay at the termination phase of slave threads
could cause any issue.
sql/slave.cc:
The unguarded read of the running status is removed. Its reading is done in
terminate_slave_thread() at time run_lock is taken;
Calling terminate_slave_threads(skip_lock := !need_slave_mutex) in the failing branch of start_slave_threads() which is bug#38716 issue.
sql/slave.h:
removing terminate_slave_thread() out of the global interface scope.
