It is well-known that due to concurrency issues, a slave can become
inconsistent when a transaction contains updates to both transaction and
non-transactional tables.
In a nutshell, the current code-base tries to preserve causality among the
statements by writing non-transactional statements to the txn-cache which
is flushed upon commit. However, modifications done to non-transactional
tables on behalf of a transaction become immediately visible to other
connections but may not immediately get into the binary log and therefore
consistency may be broken.
In general, it is impossible to automatically detect causality/dependency
among statements by just analyzing the statements sent to the server. This
happen because dependency may be hidden in the application code and it is
necessary to know a priori all the statements processed in the context of
a transaction such as in a procedure. Moreover, even for the few cases that
we could automatically address in the server, the computation effort
required could make the approach infeasible.
So, in this patch we introduce the option
- "--binlog-direct-non-transactional-updates" that can be used to bypass
the current behavior in order to write directly to binary log statements
that change non-transactional tables.
Besides, it is used to enable the WL#2687 which is disabled by default.
mysql-test/extra/rpl_tests/rpl_binlog_max_cache_size.test:
Changes the result set as the STMT mode behaves as both the ROW and MIXED modes and as such uses the non-trx-cache and takes longer to fill the trx-cache up and trigger an error.
mysql-test/extra/rpl_tests/rpl_implicit_commit_binlog.test:
Changes the result set as the STMT mode behaves as both the ROW and MIXED modes and as such uses the non-trx-cache. It also fixes comments.
mysql-test/extra/rpl_tests/rpl_mixing_engines.test:
The STMT mode is unsafe when mixed-statements are executed thus making slaves to go out of sync. For that reason, it checks consistency if not in STMT mode.
mysql-test/include/default_mysqld.cnf:
Makes binlog-direct-non-transactional-updates "TRUE" by default in the test
cases.
mysql-test/r/mysqld--help-notwin.result:
Updates the result file with the new option.
mysql-test/r/mysqld--help-win.result:
Updates the result file with the new option.
mysql-test/suite/binlog/r/binlog_multi_engine.result:
Updates the result file because non-trx-changes are written ahead of the
transaction.
mysql-test/suite/binlog/r/binlog_switch_inside_trans.result:
Verifies if the user cannot change the opion binlog_direct_non_transactional_updates
within a transaction or a procedure/function/trigger.
mysql-test/suite/binlog/t/binlog_stm_mix_innodb_myisam-master.opt:
Sets binlog_direct_non_transactional_updates to FALSE in order to avoid changing
the test case and its result file.
mysql-test/suite/binlog/t/binlog_switch_inside_trans.test:
Verifies if the user cannot change the opion binlog_direct_non_transactional_updates
within a transaction or a procedure/function/trigger.
mysql-test/suite/ndb/r/ndb_binlog_format.result:
Updates the result file because non-trx-changes are written ahead of the
transaction.
mysql-test/suite/rpl/r/rpl_begin_commit_rollback.result:
Sets binlog_direct_non_transactional_updates to FALSE in order to avoid changing the test case and its result file.
mysql-test/suite/rpl/r/rpl_concurrency_error.result:
Updates the result file because non-trx-changes are written ahead of the
transaction
mysql-test/suite/rpl/r/rpl_mixed_implicit_commit_binlog.result:
Updates the result file because non-trx-changes are written ahead of the
transaction.
mysql-test/suite/rpl/r/rpl_mixed_mixing_engines.result:
Updates the result file because non-trx-changes are written ahead of the
transaction.
mysql-test/suite/rpl/r/rpl_stm_binlog_max_cache_size.result:
Changes the result set as the STMT mode behaves as both the ROW and MIXED modes and as such uses the non-trx-cache and takes longer to fill the trx-cache up and trigger an error.
mysql-test/suite/rpl/r/rpl_stm_implicit_commit_binlog.result:
Updates the result file because non-trx-changes are written ahead of the
transaction.
mysql-test/suite/rpl/r/rpl_stm_mixing_engines.result:
Updates the result file because non-trx-changes are written ahead of the
transaction.
mysql-test/suite/rpl/r/rpl_stm_start_stop_slave.result:
Sets binlog_direct_non_transactional_updates to FALSE in order to avoid changing the test case and its result file.
mysql-test/suite/rpl/r/rpl_stm_stop_middle_group.result:
Sets binlog_direct_non_transactional_updates to FALSE in order to avoid changing the test case and its result file.
mysql-test/suite/rpl/t/rpl_begin_commit_rollback.test:
Sets binlog_direct_non_transactional_updates to FALSE in order to avoid changing the test case and its result file.
mysql-test/suite/rpl/t/rpl_stm_start_stop_slave.test:
Sets binlog_direct_non_transactional_updates to FALSE in order to avoid changing the test case and its result file.
mysql-test/suite/rpl/t/rpl_stm_stop_middle_group.test:
Sets binlog_direct_non_transactional_updates to FALSE in order to avoid changing the test case and its result file.
sql/log.cc:
Verifies if changes should be written to either the trx-cache or non-trx-cache through the use of the function use_trans_cache(). It also organizes the code.
sql/log.h:
Changes the signature of some functions by adding the modifier "const" to the thd parameter. Specifically, the following functions are changed:
bool trans_has_updated_trans_table(const THD* thd);
bool stmt_has_updated_trans_table(const THD *thd);
bool use_trans_cache(const THD*, bool is_transactional);
sql/share/errmsg-utf8.txt:
Creates error messages to report when an user tries to change the new option
binlog_direct_non_transactional_updates within a transaction or a procedure/
function/trigger.
sql/share/errmsg.txt:
Creates error messages to report when an user tries to change the new option
binlog_direct_non_transactional_updates within a transaction or a procedure/
function/trigger.
sql/sql_class.h:
Adds the new option binlog_direct_non_transactional_updates.
sql/sys_vars.cc:
Adds the new option binlog_direct_non_transactional_updates.
support-files/my-small.cnf.sh:
Adds binlog-direct-non-transactional-updates to the example file. By default
the option is disabled.
It is well-known that due to concurrency issues, a slave can become
inconsistent when a transaction contains updates to both transaction and
non-transactional tables.
In a nutshell, the current code-base tries to preserve causality among the
statements by writing non-transactional statements to the txn-cache which
is flushed upon commit. However, modifications done to non-transactional
tables on behalf of a transaction become immediately visible to other
connections but may not immediately get into the binary log and therefore
consistency may be broken.
In general, it is impossible to automatically detect causality/dependency
among statements by just analyzing the statements sent to the server. This
happen because dependency may be hidden in the application code and it is
necessary to know a priori all the statements processed in the context of
a transaction such as in a procedure. Moreover, even for the few cases that
we could automatically address in the server, the computation effort
required could make the approach infeasible.
So, in this patch we introduce the option
- "--binlog-direct-non-transactional-updates" that can be used to bypass
the current behavior in order to write directly to binary log statements
that change non-transactional tables.
Besides, it is used to enable the WL#2687 which is disabled by default.
mysql-test/extra/rpl_tests/rpl_binlog_max_cache_size.test:
Changes the result set as the STMT mode behaves as both the ROW and MIXED modes and as such uses the non-trx-cache and takes longer to fill the trx-cache up and trigger an error.
mysql-test/extra/rpl_tests/rpl_implicit_commit_binlog.test:
Changes the result set as the STMT mode behaves as both the ROW and MIXED modes and as such uses the non-trx-cache. It also fixes comments.
mysql-test/extra/rpl_tests/rpl_mixing_engines.test:
The STMT mode is unsafe when mixed-statements are executed thus making slaves to go out of sync. For that reason, it checks consistency if not in STMT mode.
mysql-test/include/default_mysqld.cnf:
Makes binlog-direct-non-transactional-updates "TRUE" by default in the test
cases.
mysql-test/r/mysqld--help-notwin.result:
Updates the result file with the new option.
mysql-test/r/mysqld--help-win.result:
Updates the result file with the new option.
mysql-test/suite/binlog/r/binlog_multi_engine.result:
Updates the result file because non-trx-changes are written ahead of the
transaction.
mysql-test/suite/binlog/r/binlog_switch_inside_trans.result:
Verifies if the user cannot change the opion binlog_direct_non_transactional_updates
within a transaction or a procedure/function/trigger.
mysql-test/suite/binlog/t/binlog_stm_mix_innodb_myisam-master.opt:
Sets binlog_direct_non_transactional_updates to FALSE in order to avoid changing
the test case and its result file.
mysql-test/suite/binlog/t/binlog_switch_inside_trans.test:
Verifies if the user cannot change the opion binlog_direct_non_transactional_updates
within a transaction or a procedure/function/trigger.
mysql-test/suite/ndb/r/ndb_binlog_format.result:
Updates the result file because non-trx-changes are written ahead of the
transaction.
mysql-test/suite/rpl/r/rpl_begin_commit_rollback.result:
Sets binlog_direct_non_transactional_updates to FALSE in order to avoid changing the test case and its result file.
mysql-test/suite/rpl/r/rpl_concurrency_error.result:
Updates the result file because non-trx-changes are written ahead of the
transaction
mysql-test/suite/rpl/r/rpl_mixed_implicit_commit_binlog.result:
Updates the result file because non-trx-changes are written ahead of the
transaction.
mysql-test/suite/rpl/r/rpl_mixed_mixing_engines.result:
Updates the result file because non-trx-changes are written ahead of the
transaction.
mysql-test/suite/rpl/r/rpl_stm_binlog_max_cache_size.result:
Changes the result set as the STMT mode behaves as both the ROW and MIXED modes and as such uses the non-trx-cache and takes longer to fill the trx-cache up and trigger an error.
mysql-test/suite/rpl/r/rpl_stm_implicit_commit_binlog.result:
Updates the result file because non-trx-changes are written ahead of the
transaction.
mysql-test/suite/rpl/r/rpl_stm_mixing_engines.result:
Updates the result file because non-trx-changes are written ahead of the
transaction.
mysql-test/suite/rpl/r/rpl_stm_start_stop_slave.result:
Sets binlog_direct_non_transactional_updates to FALSE in order to avoid changing the test case and its result file.
mysql-test/suite/rpl/r/rpl_stm_stop_middle_group.result:
Sets binlog_direct_non_transactional_updates to FALSE in order to avoid changing the test case and its result file.
mysql-test/suite/rpl/t/rpl_begin_commit_rollback.test:
Sets binlog_direct_non_transactional_updates to FALSE in order to avoid changing the test case and its result file.
mysql-test/suite/rpl/t/rpl_stm_start_stop_slave.test:
Sets binlog_direct_non_transactional_updates to FALSE in order to avoid changing the test case and its result file.
mysql-test/suite/rpl/t/rpl_stm_stop_middle_group.test:
Sets binlog_direct_non_transactional_updates to FALSE in order to avoid changing the test case and its result file.
sql/log.cc:
Verifies if changes should be written to either the trx-cache or non-trx-cache through the use of the function use_trans_cache(). It also organizes the code.
sql/log.h:
Changes the signature of some functions by adding the modifier "const" to the thd parameter. Specifically, the following functions are changed:
bool trans_has_updated_trans_table(const THD* thd);
bool stmt_has_updated_trans_table(const THD *thd);
bool use_trans_cache(const THD*, bool is_transactional);
sql/share/errmsg-utf8.txt:
Creates error messages to report when an user tries to change the new option
binlog_direct_non_transactional_updates within a transaction or a procedure/
function/trigger.
sql/share/errmsg.txt:
Creates error messages to report when an user tries to change the new option
binlog_direct_non_transactional_updates within a transaction or a procedure/
function/trigger.
sql/sql_class.h:
Adds the new option binlog_direct_non_transactional_updates.
sql/sys_vars.cc:
Adds the new option binlog_direct_non_transactional_updates.
support-files/my-small.cnf.sh:
Adds binlog-direct-non-transactional-updates to the example file. By default
the option is disabled.
The reason for the crash is using uinitialized mutex attribute (MY_MUTEX_FAST_INIT)
in pthread_mutex_init.
The fix is to initialize the attribute before the first use.
Problem: The test case failed because: (i) warning text in
result file differed from the warning output by the
server, and (ii) binlog contents in result file did
not show the statements logged wrapped in BEGIN/COMMIT
as it is the case after WL 2687.
Solution: We update the result file, but first we change the
unsafe warning text to also refer to performance_schema
table(s). This required changing the result files for
existing test cases that provide output for warnings
related to ER_BINLOG_UNSAFE_SYSTEM_TABLE. "Grepping" in
result files, shows that only binlog_unsafe contained
reference to such a warning.
We also update the result file with the missing
BEGIN/COMMIT statements.
When starting mysqld it did not recognize most of the options given on
the command line when it was compiled for 32-bit Solaris using Sun
Studio compiler. The cause for this was that most of the entries in
the my_long_options array contained "garbage" data. The garbage data
was caused by a compiler bug. When initilizing the def_value member
for the "default-storage-engine" entry it was initialized like this:
(longlong)"MyISAM"
i.e. casting a 32 bit pointer to a 64 bit integer value. Due to the
compiler bug only 4 bytes was allocated (instead of 8 bytes). This
caused everything following this entry to be stored at a location that
was 4 byte wrong.
The fix/work-around for this problem is initialize the def_value
for default-storage-engine in my_long_options to 0 and instead
initialize the default_storage_engine variable to "MyISAM" in
init_common_variables().
sql/mysqld.cc:
Due to a bug in Sun Studio compiler when generating 32 bit code the
initialization of the def_value member of the default-storage-engine entry
in my_long_options only got 4 bytes allocated instead of 8 bytes.
The compiler bug was triggered by casting a 32 bit pointer to a 64 bit
integer value in the initialization code for my_long_options. To avoid
triggering the compiler bug the intialization of the def_value in
my_long_options is set to 0 and instead the default_storage_engine
is initialized to "MyISAM" in init_common_variables().
replicating
Replace c_ptr() calls with c_ptr_safe() calls to
avoid valgrind warnings.
Adding code to to handle the case that no metadata
was present in the table map for the column.
Allow first parameter to unpack_row() to be NULL,
in which case no source tables is used and hence
no checks nor conversions are done.
Clarifying some comments and fixing documentation
for unpack_row().
- mysqld--help-win
Updated result so that it contains missing
value for slave-type-conversions
- rpl_idempotency
This seems a bad merge. In BUG#39934, the contents of
this file had been split into rpl_row_idempontency and
rpl_idempotency. The patch was pushed to 5.1-rep+3 which
was later merged in rep+2-delivery1 which in turn was
merged in 5.1-rpl-merge. Now while merging next-mr in
5.1-rpl-merge, the file got back it's old content (which
is in rpl_row_idempotency now because of BUG#39934). This
cset reverts the bad merge:
bzr merge -r revid:dao-gang.qu@sun.com-20100112120709-ioxp11yl9bvquaqd..\
before:revid:dao-gang.qu@sun.com-20100112120709-ioxp11yl9bvquaqd\
suite/rpl/t/rpl_idempotency.test
- sys_vars.all_vars:
Added test case for slave_type_conversions variable
- rpl_row_idempotency
Removed ER_SLAVE_AMBIGOUS_EXEC_MODE (which was removed by WL 4738)
from the test case. Using ER_WRONG_VALUE_FOR_VAR instead.
- mysqld--help-win
Added missing help for --slave-type-conversions from the
result file.
