Documentation of the feature can be found at: http://kb.askmonty.org/en/multi-source-replication/
This code is based on code from Taobao, developed by Plinux
BUILD/SETUP.sh:
Added -Wno-invalid-offsetof to get rid of warning of offsetof() on C++ class (safe in the contex we use it)
client/mysqltest.cc:
Added support for error names starting with 'W'
Added connection_name support to --sync_with_master
cmake/maintainer.cmake:
Added -Wno-invalid-offsetof to get rid of warning of offsetof() on C++ class (safe in the contex we use it)
mysql-test/r/mysqltest.result:
Updated results
mysql-test/r/parser.result:
Updated results
mysql-test/suite/multi_source/my.cnf:
Setup of multi-master tests
mysql-test/suite/multi_source/simple.result:
Simple basic test of multi-source functionality
mysql-test/suite/multi_source/simple.test:
Simple basic test of multi-source functionality
mysql-test/suite/multi_source/syntax.result:
Test of multi-source syntax
mysql-test/suite/multi_source/syntax.test:
Test of multi-source syntax
mysql-test/suite/rpl/r/rpl_rotate_logs.result:
Updated results because of new error messages
mysql-test/t/parser.test:
Updated test as master_pos_wait() now takes more arguments than before
sql/event_scheduler.cc:
No reason to initialize slave_thread (it's guaranteed to be zero here)
sql/item_create.cc:
Added connection_name argument to master_pos_wait()
Simplified code
sql/item_func.cc:
Added connection_name argument to master_pos_wait()
sql/item_func.h:
Added connection_name argument to master_pos_wait()
sql/log.cc:
Added tag "Master 'connection_name'" to slave errors that has a connection name.
sql/mysqld.cc:
Added variable mysqld_server_initialized so that other functions can test if server is fully initialized.
Free all slave data in one place (fewer ifdef's)
Removed not needed call to close_active_mi()
Initialize slaves() later in startup to ensure that everthing is really initialized when slaves start.
Made status variable slave_running multi-source safe
sql/mysqld.h:
Added mysqld_server_initialized
sql/rpl_mi.cc:
Store connection name and cmp_connection_name (only used for show full slave status) in Master_info
Added code for Master_info_index, which handles storage of multi-master information
Don't write the empty "" connection_name to multi-master.info file. This is handled by the original code.
sql/rpl_mi.h:
Added connection_name and Master_info_index
sql/rpl_rli.cc:
Added connection_name to relay log files.
sql/rpl_rli.h:
Fixed type of slave_skip_counter as we now access it directly in sys_vars.cc, so it must be uint
sql/share/errmsg-utf8.txt:
Added new error messages needed for multi-source
Added multi-source name to error ER_MASTER_INFO and WARN_NO_MASTER_INFO
sql/slave.cc:
Moved things a bit around to make it easier to handle error conditions.
Create a global master_info_index and add the "" connection to it
Ensure that new Master_info doesn't fail.
Don't call terminate_slave_threads(active_mi..) on end_slave() as this is now done automaticly when deleting master_info_index.
Delete not needed function close_active_mi(). One can achive same thing by calling end_slave().
Added support for SHOW FULL SLAVE STATUS (show status for all master connections with connection_name as first column)
sql/slave.h:
Added new prototypes
sql/sql_base.cc:
More DBUG_PRINT
sql/sql_class.cc:
Reset thd->connection_name and thd-->default_master_connection
sql/sql_class.h:
Added thd->connection_name and thd-->default_master_connection
Added slave_skip_count to variables to make changing the @@sql_slave_skip_count variable thread safe
sql/sql_const.h:
Added MAX_CONNECTION_NAME
sql/sql_lex.cc:
Reset 'lex->verbose' (to simplify some sql_yacc.yy code)
sql/sql_lex.h:
Added connection_name
sql/sql_parse.cc:
Added support for connection_name to all SLAVE commands.
- Instead of using active_mi, we now get the current Master_info from master_info_index.
- Create new replication threads with CHANGE MASTER
- Added support for show_all_master_info()
sql/sql_reload.cc:
Made reset/full slave use master_info_index->get_master_info() instead of active_mi.
If one uses 'RESET SLAVE "connection_name" all' the connection is removed from master_info_index.
sql/sql_repl.cc:
sql_slave_skip_counter is moved to thd->variables to make it thread safe and fix some bugs with it
Add connection name to relay log files.
Added connection name to errors.
Added some logging for multi-master if log_warnings > 1
stop_slave():
- Don't check if thd is set. It's guaranteed to always be set.
change_master():
- Check for duplicate connection names in change_master()
- Check for wrong arguments first in file (to simplify error handling)
- Register new connections in master_info_index
sql/sql_yacc.yy:
Added optional connection_name to a all relevant master/slave commands
sql/strfunc.cc:
my_global.h shoud always be included first.
sql/sys_vars.cc:
Added variable default_master_connection
Made variable sql_slave_skip_counter multi-source safe
sql/sys_vars.h:
Added Sys_var_session_lexstring (needed for default_master_connection)
Added Sys_var_multi_source_uint (needed for sql_slave_skip_counter).
and small collateral changes
mysql-test/lib/My/Test.pm:
somehow with "print" we get truncated writes sometimes
mysql-test/suite/perfschema/r/digest_table_full.result:
md5 hashes of statement digests differ, because yacc token codes are different in mariadb
mysql-test/suite/perfschema/r/dml_handler.result:
host table is not ported over yet
mysql-test/suite/perfschema/r/information_schema.result:
host table is not ported over yet
mysql-test/suite/perfschema/r/nesting.result:
this differs, because we don't rewrite general log queries, and multi-statement
packets are logged as a one entry. this result file is identical to what mysql-5.6.5
produces with the --log-raw option.
mysql-test/suite/perfschema/r/relaylog.result:
MariaDB modifies the binlog index file directly, while MySQL 5.6 has a feature "crash-safe binlog index" and modifies a special "crash-safe" shadow copy of the index file and then moves it over. That's why this test shows "NONE" index file writes in MySQL and "MANY" in MariaDB.
mysql-test/suite/perfschema/r/server_init.result:
MariaDB initializes the "manager" resources from the "manager" thread, and starts this thread only when --flush-time is not 0. MySQL 5.6 initializes "manager" resources unconditionally on server startup.
mysql-test/suite/perfschema/r/stage_mdl_global.result:
this differs, because MariaDB disables query cache when query_cache_size=0. MySQL does not
do that, and this causes useless mutex locks and waits.
mysql-test/suite/perfschema/r/statement_digest.result:
md5 hashes of statement digests differ, because yacc token codes are different in mariadb
mysql-test/suite/perfschema/r/statement_digest_consumers.result:
md5 hashes of statement digests differ, because yacc token codes are different in mariadb
mysql-test/suite/perfschema/r/statement_digest_long_query.result:
md5 hashes of statement digests differ, because yacc token codes are different in mariadb
mysql-test/suite/rpl/r/rpl_mixed_drop_create_temp_table.result:
will be updated to match 5.6 when alfranio.correia@oracle.com-20110512172919-c1b5kmum4h52g0ni and anders.song@greatopensource.com-20110105052107-zoab0bsf5a6xxk2y are merged
mysql-test/suite/rpl/r/rpl_non_direct_mixed_mixing_engines.result:
will be updated to match 5.6 when anders.song@greatopensource.com-20110105052107-zoab0bsf5a6xxk2y is merged
NUMBERS
If a system variable was declared as deprecated without mention of an
alternative, the message would look funny, e.g. for @@delayed_insert_limit:
Warning 1287 '@@delayed_insert_limit' is deprecated and
will be removed in MySQL .
The message was meant to display the version number, but it's not
possible to give one when declaring a system variable.
The fix does two things:
1) The definition of the message
ER_WARN_DEPRECATED_SYNTAX_NO_REPLACEMENT is changed so that it does
not display a version number. I.e. in English the message now reads:
Warning 1287 The syntax '@@delayed_insert_limit' is deprecated and
will be removed in a future version.
2) The message ER_WARN_DEPRECATED_SYNTAX_WITH_VER is discontinued in
favor of ER_WARN_DEPRECATED_SYNTAX for system variables. This change
was already done in versions 5.6 and above as part of wl#5265. This
part is simply back-ported from the worklog.
- Changed output to be error "error-text" instead of error - error-text
extra/perror.c:
Move my_handler_errors.h into include
include/my_handler_errors.h:
Move my_handler_errors.h into include
mysql-test/r/errors.result:
Updated result
mysql-test/r/innodb_mysql_sync.result:
Updated result
mysql-test/r/myisam-system.result:
Updated result
mysql-test/r/myisampack.result:
Updated result
mysql-test/r/partition_innodb_plugin.result:
Updated result
mysql-test/r/ps_1general.result:
Updated result
mysql-test/r/trigger.result:
Updated result
mysql-test/r/type_bit.result:
Updated result
mysql-test/r/type_bit_innodb.result:
Updated result
mysql-test/r/type_blob.result:
Updated result
mysql-test/suite/archive/archive.result:
Updated result
mysql-test/suite/binlog/r/binlog_index.result:
Updated result
mysql-test/suite/binlog/r/binlog_ioerr.result:
Updated result
mysql-test/suite/csv/csv.result:
Updated result
mysql-test/suite/engines/iuds/r/type_bit_iuds.result:
Updated result
mysql-test/suite/federated/federated_bug_35333.result:
Updated result
mysql-test/suite/innodb/r/innodb-create-options.result:
Updated result
mysql-test/suite/innodb/r/innodb-index.result:
Updated result
mysql-test/suite/innodb/r/innodb-zip.result:
Updated result
mysql-test/suite/innodb/r/innodb.result:
Updated result
mysql-test/suite/innodb/r/innodb_bug13635833.result:
Updated result
mysql-test/suite/innodb/r/innodb_bug21704.result:
Updated result
mysql-test/suite/innodb/r/innodb_bug46000.result:
Updated result
mysql-test/suite/parts/r/partition_bit_innodb.result:
Updated result
mysql-test/suite/parts/r/partition_bit_myisam.result:
Updated result
mysql-test/suite/percona/percona_innodb_fake_changes.result:
Updated result
mysql-test/suite/perfschema/r/misc.result:
Updated result
mysql-test/suite/perfschema/r/privilege.result:
Updated result
mysql-test/suite/rpl/r/rpl_EE_err.result:
Updated result
mysql-test/suite/rpl/r/rpl_binlog_errors.result:
Updated result
mysql-test/suite/rpl/r/rpl_drop_db.result:
Updated result
sql/share/errmsg-utf8.txt:
Removed 'column' from error text that was used in different context
strings/my_vsnprintf.c:
Move my_handler_errors.h into include
Minor cleanups
Changed output of %M to be error "error-text" instead of error - error-text
unittest/mysys/my_vsnprintf-t.c:
Updated error text
- Adding %M my_sprintf() modifier that prints error number - system-error-text
- Modified mysys, mysql_client and SQL error messages to use %M instead of %d
- Added my_strerror()
Updated handler errors to 5.6 error numbers
Updated text for a few error messages (to match 5.6)
Increased length of command name in error output
extra/comp_err.c:
Added support for %M
include/my_base.h:
Updated handler errors to 5.6 error numbers
include/my_sys.h:
Added my_strerror()
libmysql/errmsg.c:
Updated error messages to use %M
mysql-test/r/errors.result:
Updated result as error message have changed
mysql-test/r/innodb_mysql_sync.result:
Updated result with text for errno
mysql-test/r/myisam-system.result:
Updated result with text for errno
mysql-test/r/myisam.result:
Updated result as error message have changed
mysql-test/r/myisampack.result:
Updated result with text for errno
mysql-test/r/mysql.result:
Updated result with text for errno
mysql-test/r/mysql_upgrade.result:
Updated result with text for errno
mysql-test/r/partition_datatype.result:
Updated result as error message have changed
mysql-test/r/partition_innodb_plugin.result:
Updated result with text for errno
mysql-test/r/ps_1general.result:
Updated result with text for errno
mysql-test/r/trigger.result:
Updated result with text for errno
mysql-test/r/type_bit.result:
Updated result as error message have changed
mysql-test/r/type_bit_innodb.result:
Updated result as error message have changed
mysql-test/r/type_blob.result:
Updated result as error message have changed
mysql-test/suite/archive/archive.result:
Updated result with text for errno
mysql-test/suite/binlog/r/binlog_index.result:
Updated result with text for errno
mysql-test/suite/binlog/r/binlog_ioerr.result:
Updated result with text for errno
mysql-test/suite/csv/csv.result:
Updated result with text for errno
mysql-test/suite/federated/federated_bug_35333.result:
Updated result with text for errno
mysql-test/suite/innodb/r/innodb-create-options.result:
Updated result with text for errno
mysql-test/suite/innodb/r/innodb-index.result:
Updated result with text for errno
mysql-test/suite/innodb/r/innodb-zip.result:
Updated result as error message have changed
mysql-test/suite/innodb/r/innodb.result:
Updated result with text for errno
mysql-test/suite/innodb/r/innodb_bug21704.result:
Updated result with text for errno
mysql-test/suite/innodb/r/innodb_bug46000.result:
Updated result with text for errno
mysql-test/suite/innodb/r/innodb_bug53591.result:
Updated result as error message have changed
mysql-test/suite/innodb/r/innodb_corrupt_bit.result:
New error numbers
mysql-test/suite/innodb/r/innodb_prefix_index_liftedlimit.result:
Updated result as error message have changed
mysql-test/suite/innodb/t/innodb-create-options.test:
Added regexp to avoid system error text
mysql-test/suite/innodb/t/innodb-zip.test:
Added regexp to avoid system error text
mysql-test/suite/maria/maria-recovery2.result:
Updated supression rule
mysql-test/suite/maria/maria-recovery2.test:
Updated supression rule
mysql-test/suite/maria/maria.result:
Updated result as error message have changed
mysql-test/suite/parts/r/partition_bit_innodb.result:
Updated result as error message have changed
mysql-test/suite/parts/r/partition_bit_myisam.result:
Updated result as error message have changed
mysql-test/suite/percona/percona_innodb_fake_changes.result:
Updated result with text for errno
mysql-test/suite/perfschema/r/dml_cond_instances.result:
Updated result as error message have changed
mysql-test/suite/perfschema/r/dml_events_waits_current.result:
Updated result as error message have changed
mysql-test/suite/perfschema/r/dml_events_waits_history.result:
Updated result as error message have changed
mysql-test/suite/perfschema/r/dml_events_waits_history_long.result:
Updated result as error message have changed
mysql-test/suite/perfschema/r/dml_ews_by_instance.result:
Updated result as error message have changed
mysql-test/suite/perfschema/r/dml_ews_by_thread_by_event_name.result:
Updated result as error message have changed
mysql-test/suite/perfschema/r/dml_ews_global_by_event_name.result:
Updated result as error message have changed
mysql-test/suite/perfschema/r/dml_file_instances.result:
Updated result as error message have changed
mysql-test/suite/perfschema/r/dml_file_summary_by_event_name.result:
Updated result as error message have changed
mysql-test/suite/perfschema/r/dml_file_summary_by_instance.result:
Updated result as error message have changed
mysql-test/suite/perfschema/r/dml_mutex_instances.result:
Updated result as error message have changed
mysql-test/suite/perfschema/r/dml_performance_timers.result:
Updated result as error message have changed
mysql-test/suite/perfschema/r/dml_rwlock_instances.result:
Updated result as error message have changed
mysql-test/suite/perfschema/r/dml_threads.result:
Updated result as error message have changed
mysql-test/suite/perfschema/r/misc.result:
Updated result with text for errno
mysql-test/suite/perfschema/r/privilege.result:
Updated result with text for errno
mysql-test/suite/rpl/r/rpl_EE_err.result:
Updated result with text for errno
mysql-test/suite/rpl/r/rpl_binlog_errors.result:
Updated result with text for errno
mysql-test/suite/rpl/r/rpl_drop_db.result:
Updated result with text for errno
mysys/errors.c:
Updated error messages to use %M
Changed all errors to use Errcode: consistenly
mysys/my_handler_errors.h:
Updated handler errors to 5.6 error numbers
sql/share/errmsg-utf8.txt:
Updated error messages to use %M
sql/sys_vars.cc:
Added error number to ER_EVENT_SET_VAR_ERROR
strings/my_vsnprintf.c:
Added %M my_sprintf() modifier that prints error number - system-error-text
Simplify code
Movied common code to function
Removed some casts that was not necessary when reading integer/unsigned int stored in longlong
Added my_strerror()
unittest/mysys/my_vsnprintf-t.c:
Added testing of %M
TABLES IN INCORRECT ENGINE
PROBLEM:
CREATE/ALTER TABLE currently can move system tables like
mysql.db, user, host etc, to engines other than MyISAM. This is not
completely supported as of now, by mysqld. When some of system tables
like plugin, servers, event, func, *_priv, time_zone* are moved
to innodb, mysqld restart crashes. Currently system tables
can be moved to BLACKHOLE also!!!.
ANALYSIS:
The problem is that there is no check before creating or moving
a system table to some particular engine.
System tables are suppose to be residing in MyISAM. We can think
of restricting system tables to exist only in MyISAM. But, there could
be future needs of these system tables to be part of other engines
by design. For eg, NDB cluster expects some tables to be on innodb
or ndb engine. This calls for a solution, by which system
tables can be supported by any desired engine, with minimal effort.
FIX:
The solution provides a handlerton interface using which,
mysqld server can query particular storage engine handlerton for
system tables that it supports. This way each storage engine
layer can define their own system database and system tables.
The check_engine() function uses the new handlerton function
ha_check_if_supported_system_table() to check if db.tablename
provided in the DDL is supported by the SE.
Note: This fix has modified a test in help.test, which was moving
mysql.help_* to innodb. The primary intention of the test was not
to move them between engines.
Description: When the table has more than one unique or primary key,
INSERT... ON DUP KEY UPDATE statement is sensitive to the order in which
the storage engines checks the keys. Depending on this order, the storage
engine may determine different rows to mysql, and hence mysql can update
different rows on master and slave.
Solution: We mark INSERT...ON DUP KEY UPDATE on a table with more than on unique
key as unsafe therefore the event will be logged in row format if it is available
(ROW/MIXED). If only STATEMENT format is available, a warning will be thrown.
mysql-test/suite/binlog/r/binlog_unsafe.result:
Updated result file
mysql-test/suite/binlog/t/binlog_unsafe.test:
Added test to check for warning being thrown when the unsafe statement is executed
mysql-test/suite/rpl/r/rpl_known_bugs_detection.result:
Updated result file
sql/share/errmsg-utf8.txt:
Added new warning message
sql/sql_base.cc:
check for tables in the query with more than one UNIQUE KEY and INSERT ON DUPLICATE KEY UPDATE, and mark such statements unsafe.
mysql-test/suite/innodb/t/group_commit_crash.test:
remove autoincrement to avoid rbr being used for insert ... select
mysql-test/suite/innodb/t/group_commit_crash_no_optimize_thread.test:
remove autoincrement to avoid rbr being used for insert ... select
mysys/my_addr_resolve.c:
a pointer to a buffer is returned to the caller -> the buffer cannot be on the stack
mysys/stacktrace.c:
my_vsnprintf() is ok here, in 5.5
https://mariadb.atlassian.net/browse/MDEV-28
This task implements a new clause LIMIT ROWS EXAMINED <num>
as an extention to the ANSI LIMIT clause. This extension
allows to limit the number of rows and/or keys a query
would access (read and/or write) during query execution.
There are two threads. In one thread, dml operation is going on
involving cascaded update operation. In another thread, alter
table add foreign key constraint is happening. Under these
circumstances, it is possible for the dml thread to access a
dict_foreign_t object that has been freed by the ddl thread.
The debug sync test case provides the sequence of operations.
Without fix, the test case will crash the server (because of
newly added assert). With fix, the alter table stmt will return
an error message.
rb:947
approved by Jimmy Yang
Fixed compiler warnings found by buildbot
Makefile.am:
Removed extra empty line
cmd-line-utils/libedit/sys.h:
Fixed that strndup() doesn't give compiler warnings
mysql-test/Makefile.am:
Fixes for 'make distcheck'
plugin/auth_pam/auth_pam.c:
Ensure that prototype for strndup() is included on linux
sql/share/Makefile.am:
Fixes for 'make distcheck'
storage/innodb_plugin/btr/btr0sea.c:
Fixed compiler warning
support-files/Makefile.am:
Fixes for 'make distcheck'
Problem: Statements that write to tables with auto_increment columns
based on the selection from another table, may lead to master
and slave going out of sync, as the order in which the rows
are retrieved from the table may differ on master and slave.
Solution: We mark writing to a table with auto_increment table
based on the rows selected from another table as unsafe. This
will cause the execution of such statements to throw a warning
and forces the statement to be logged in ROW if the logging
format is mixed.
Changes:
1. All the statements that writes to a table with auto_increment
column(s) based on the rows fetched from another table, will now
be unsafe.
2. CREATE TABLE with SELECT will now be unsafe.
sql/share/errmsg-utf8.txt:
Added new warning messages.
sql/sql_base.cc:
-Created function to check statements that write to
tables with auto_increment column and has select.
-Marked all the statements that write to a table
with auto_increment column based on rows fetched
from other table(s) as unsafe.
sql/sql_table.cc:
mark CREATE TABLE[with auto_increment column] as unsafe.
Problem: Statements that write to tables with auto_increment columns
based on the selection from another table, may lead to master
and slave going out of sync, as the order in which the rows
are retrived from the table may differ on master and slave.
Solution: We mark writing to a table with auto_increment table
as unsafe. This will cause the execution of such statements to
throw a warning and forces the statement to be logged in ROW if
the logging format is mixed.
Changes:
1. All the statements that writes to a table with auto_increment
column(s) based on the rows fetched from another table, will now
be unsafe.
2. CREATE TABLE with SELECT will now be unsafe.
sql/share/errmsg-utf8.txt:
Added new Warning messages
sql/sql_base.cc:
created a new function that checks for select + write on a autoinc table
made all such statements to be unsafe.
sql/sql_parse.cc:
made create autoincremnet tabble + select unsafe
A follow-up patch corrects max sizes of printed strings and changes llstr() to %lld.
Credits go to Davi who provided a great feedback.
sql/share/errmsg-utf8.txt:
Max size for the whole message is 512 so a part of - like '%-.512s' should be less,
reduction to 320 is safe and with good chances won't cut off a part of a rather log
message in Last_IO_Error = 'Got fatal error 1236 ...'
sql/sql_repl.cc:
llstr() is replaced by %lld.
The server crashes when receiving a COM_BINLOG_DUMP command with a position of 0 or
larger than the file size.
The execution proceeds to an error block having the last read replication coordinates
pointer be NULL and its dereferencing crashed the server.
Fixed with making "public" previously used only for heartbeat coordinates.
mysql-test/extra/rpl_tests/rpl_start_stop_slave.test:
regression test for bug#3593869-64035 is added.
mysql-test/suite/rpl/r/rpl_cant_read_event_incident.result:
results updated (error mess format is changed).
mysql-test/suite/rpl/r/rpl_log_pos.result:
results updated (error mess format is changed).
mysql-test/suite/rpl/r/rpl_manual_change_index_file.result:
results updated (error mess format is changed).
mysql-test/suite/rpl/r/rpl_packet.result:
results updated (error mess format is changed).
mysql-test/suite/rpl/r/rpl_stm_start_stop_slave.result:
results updated (error mess format is changed).
mysql-test/suite/rpl/t/rpl_stm_start_stop_slave.test:
Slave is stopped by bug#3593869-64035 tests so
-let $rpl_only_running_threads= 1 is set prior to rpl_end.
sql/share/errmsg-utf8.txt:
Increasing the max length of explanatory message to 512.
sql/sql_repl.cc:
Making `coord' to carry the last read from binlog event coordinates
regardless of heartbeat.
Renaming, small cleanup and simplifying the code after if (coord) becomes unnecessary.
Adding yet another 3rd pair of coordinates - the starting replication -
into error text.
sql/share/errmsg-utf8.txt:
correct the error message, as query_cache_type variable is not read-ony anymore
sql/sql_cache.cc:
the caller should verify that query cache resize
is possible, before trying it
sql/sys_vars.cc:
* test if qc resize is possible in the sysvar on_check() funntion,
not in the on_update() function.
* use the error message that better describes the problem
* rename all debugging related command-line options
and variables to start from "debug-", and made them all
OFF by default.
* replace "MySQL" with "MariaDB" in error messages
* "Cast ... converted ... integer to it's ... complement"
is now a note, not a warning
* @@query_cache_strip_comments now has a session scope,
not global.
