'LOAD DATA CONCURRENT [LOCAL] INFILE ...' statment only is binlogged as
'LOAD DATA [LOCAL] INFILE ...' in SBR and MBR. As a result, if replication is on,
queries on slaves will be blocked by the replication SQL thread.
This patch write code to write 'CONCURRENT' into the log event if 'CONCURRENT' option
is in the original statement in SBR and MBR.
{PROCEDURE|FUNCTION} FROM ...'
The master would hit an assertion when binary log was
active. This was due to the fact that the thread's diagnostics
area was being cleared before writing to the binlog,
independently of mysql_routine_grant returning an error or
not. When mysql_routine_grant was to return an error, the return
value and the diagnostics area contents would
mismatch. Consequently, neither my_ok would be called nor an
error would be signaled in the diagnostics area, eventually
triggering the assertion in net_end_statement.
We fix this by not clearing the diagnostics area at binlogging
time.
The 'slave_patternload_file' is assigned to the real path of the load data file
when initializing the object of Relay_log_info. But the path of the load data
file is not formatted to real path when executing event from relay log. So the
error will be encountered if the path of the load data file is a symbolic link.
Actually the global 'opt_secure_file_priv' is not formatted to real path when
loading data from file. So the same thing will happen too.
To fix these errors, the path of the load data file should be formatted to
real path when executing event from relay log. And the 'opt_secure_file_priv'
should be formatted to real path when loading data infile.
mysql-test/suite/rpl/r/rpl_loaddata_symlink.result:
Test result for bug#43913.
mysql-test/suite/rpl/t/rpl_loaddata_symlink-master.sh:
Added the test file to create a link from $MYSQLTEST_VARDIR/std_data
to $MYSQLTEST_VARDIR/std_data_master_link
mysql-test/suite/rpl/t/rpl_loaddata_symlink-slave.sh:
Added the test file to create a link from $MYSQLTEST_VARDIR/std_data
to $MYSQLTEST_VARDIR/std_data_slave_link
mysql-test/suite/rpl/t/rpl_loaddata_symlink.test:
Added the test file to verify if loading data infile will work fine
if the path of the load data file is a symbolic link.
sql/rpl_rli.cc:
Added call 'my_realpath' function for avoiding sometimes the 'fn_format'
function can't format real path rightly.
<tmp_tbl> with RBL
When binlogging the statement, the server always handle the existing
object as a table, even though it is a view. However a view is
handled differently in other parts of the code thus leading the
statement to crash in RBL if the view exists.
This happens because the underlying tables for the view are not opened
when we try to call store_create_info() on the view in order to build
a CREATE TABLE statement.
This patch will only address the crash problem, other binlogging
problems related to CREATE TABLE IF NOT EXISTS LIKE when the existing
object is a view will be solved by BUG 47442.
In RBR, All statements operating on temporary tables should not be binlogged.
Despite this fact, after executing 'TRUNCATE... ' on a temporary table,
the command is still logged, even if in row-based mode. Consequently, this raises
problems in the slave as the table may not exist, resulting in an
execution failure. Ultimately, this causes the slave to report
an error and abort.
After this patch, 'TRUNCATE ...' statement on a temporary table will not be
binlogged in RBR.
Problem: Some system functions that could return different values on
master and slave were not marked unsafe. In particular:
GET_LOCK
IS_FREE_LOCK
IS_USED_LOCK
MASTER_POS_WAIT
RELEASE_LOCK
SLEEP
SYSDATE
VERSION
Fix: Mark these functions unsafe.
mysql-test/extra/rpl_tests/rpl_stm_000001.test:
- The test does not work in mixed mode any more, since it tries to
simulate an error in the sql thread in a query that uses get_lock.
Since get_lock now causes the query to be logged in row format,
the error didn't happen. Hence, we now force statement mode.
- Warnings must be disabled when the unsafe query is issued.
- Replaced some save_master_pos+connection slave+sync_with_master
by sync_slave_with_master.
mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result:
updated result file
mysql-test/suite/binlog/r/binlog_stm_row.result:
updated result file
mysql-test/suite/binlog/r/binlog_unsafe.result:
updated result file
mysql-test/suite/binlog/t/binlog_killed.test:
binlog_killed only works in statement format now, since
it switches to row mode in mixed mode.
mysql-test/suite/binlog/t/binlog_stm_mix_innodb_myisam.test:
suppress warnings for unsafe statements
mysql-test/suite/binlog/t/binlog_stm_row.test:
- Suppress warnings in test that causes warnings.
- The test sets binlog format explicitly, so no need to execute it
twice.
mysql-test/suite/binlog/t/binlog_unsafe.test:
Added test for all unsafe system functions. This test also includes
system functions that were unsafe prior to BUG#47995.
mysql-test/suite/rpl/r/rpl_err_ignoredtable.result:
updated result file
mysql-test/suite/rpl/r/rpl_get_lock.result:
updated result file
mysql-test/suite/rpl/r/rpl_nondeterministic_functions.result:
new result file
mysql-test/suite/rpl/r/rpl_stm_000001.result:
updated result file
mysql-test/suite/rpl/r/rpl_trigger.result:
updated result file
mysql-test/suite/rpl/t/rpl_err_ignoredtable.test:
- suppress warnings for unsafe statement
- replaced save_master_pos+connection slave+sync_with_master
with sync_slave_with_master
mysql-test/suite/rpl/t/rpl_get_lock.test:
update test case that causes new warnings
mysql-test/suite/rpl/t/rpl_nondeterministic_functions.test:
Added new test case for nondeterministic functions.
mysql-test/suite/rpl/t/rpl_trigger.test:
update test case that causes new warnings
sql/item_create.cc:
Marked some system functions unsafe.
sql/item_strfunc.cc:
Clarified comment related to this bug.
sql/sql_yacc.yy:
Marked sysdate unsafe.
The 'rpl_get_master_version_and_clock' test verifies if the slave I/O
thread tries to reconnect to master when it tries to get the values of
the UNIX_TIMESTAMP, SERVER_ID from master under network disconnection.
So the master server is restarted for making the transient network
disconnection. Restarting master server can bring two problems as following:
1. The time out error is encountered sporadically. The slave I/O thread tries
to reconnect master ten times, which is set in my.cnf. So in the test
framework sporadically the slave I/O thread really stoped when it can't
reconnect to master in the ten times successfully before the master starts,
then the time out error will be encountered while waiting for the slave to
start.
2. These warnings and errors are produced in server log file when
the slave I/O thread tries to get the values of the UNIX_TIMESTAMP,
SERVER_ID from master under the transient network disconnection.
To fix problem 1, increase the master retry count to sixty times,
so that the slave I/O thread has enough time to reconnect master
successfully.
To fix problem 2, suppress these warnings and errors by mtr suppression,
because they are expected.
mysql-test/suite/rpl/t/rpl_get_master_version_and_clock-slave.opt:
Added the *.opt file for increasing master retry count to
sixty times.
mysql-test/suite/rpl/t/rpl_get_master_version_and_clock.test:
Added mtr suppression for suppressing warnings and errors
in server log file.
binlog, replication aborts
In SBR or MBR, the schema name is not being written to the binlog
when executing a LOAD DATA statement. This becomes a problem when
the current database (lets call it db1) is different from the
table's schema (lets call it db2). For instance, take the
following statements:
use db1;
load data local infile 'infile.txt' into table db2.t
Should this statement be logged without t's schema (db2), when
replaying it, one can get db1.t populated instead of db2.t (if
db1.t exists). On the other hand, if there is no db1.t at all,
replication will stop.
We fix this by always logging the table (in load file) with fully
qualified name when its schema is different from the current
database or when no default database was selected.
Backporting BUG#43789 to mysql-5.1-bugteam
The replication was generating corrupted data, warning messages on Valgrind
and aborting on debug mode while replicating a "null" to "not null" field.
Specifically the unpack_row routine, was considering the slave's table
definition and trying to retrieve a field value, where there was nothing to be
retrieved, ignoring the fact that the value was defined as "null" by the master.
To fix the problem, we proceed as follows:
1 - If it is not STRICT sql_mode, implicit default values are used, regardless
if it is multi-row or single-row statement.
2 - However, if it is STRICT mode, then a we do what follows:
2.1 If it is a transactional engine, we do a rollback on the first NULL that is
to be set into a NOT NULL column and return an error.
2.2 If it is a non-transactional engine and it is the first row to be inserted
with multi-row, we also return the error. Otherwise, we proceed with the
execution, use implicit default values and print out warning messages.
Unfortunately, the current patch cannot mimic the behavior showed by the master
for updates on multi-tables and multi-row inserts. This happens because such
statements are unfolded in different row events. For instance, considering the
following updates and strict mode:
(master)
create table t1 (a int);
create table t2 (a int not null);
insert into t1 values (1);
insert into t2 values (2);
update t1, t2 SET t1.a=10, t2.a=NULL;
t1 would have (10) and t2 would have (0) as this would be handled as a
multi-row update. On the other hand, if we had the following updates:
(master)
create table t1 (a int);
create table t2 (a int);
(slave)
create table t1 (a int);
create table t2 (a int not null);
(master)
insert into t1 values (1);
insert into t2 values (2);
update t1, t2 SET t1.a=10, t2.a=NULL;
On the master t1 would have (10) and t2 would have (NULL). On
the slave, t1 would have (10) but the update on t1 would fail.
Backporting BUG#38173 to mysql-5.1-bugteam
The reason of the bug was incompatibile with the master side behaviour.
INSERT query on the master is allowed to insert into a table without specifying
values of DEFAULT-less fields if sql_mode is not strict.
Fixed with checking sql_mode by the sql thread to decide how to react.
Non-strict sql_mode should allow Write_rows event to complete.
todo: warnings can be shown via show slave status, still this is a
separate rather general issue how to show warnings for the slave threads.
Let
- T be a transactional table and N non-transactional table.
- B be begin, C commit and R rollback.
- N be a statement that accesses and changes only N-tables.
- T be a statement that accesses and changes only T-tables.
In RBR, changes to N-tables that happen early in a transaction are not immediately flushed
upon committing a statement. This behavior may, however, break consistency in the presence
of concurrency since changes done to N-tables become immediately visible to other
connections. To fix this problem, we do the following:
. B N N T C would log - B N C B N C B T C.
. B N N T R would log - B N C B N C B T R.
Note that we are not preserving history from the master as we are introducing a commit that
never happened. However, this seems to be more acceptable than the possibility of breaking
consistency in the presence of concurrency.
The problem is that there is only one autoinc value associated with
the query when binlogging. If more than one autoinc values are used
in the query, the autoinc values after the first one can be inserted
wrongly on slave. So these autoinc values can become inconsistent on
master and slave.
The problem is resolved by marking all the statements that invoke
a trigger or call a function that updated autoinc fields as unsafe,
and will switch to row-format in Mixed mode. Actually, the statement
is safe if just one autoinc value is used in sub-statement, but it's
impossible to check how many autoinc values are used in sub-statement.)
mysql-test/suite/rpl/r/rpl_auto_increment_update_failure.result:
Test result for bug#45677
mysql-test/suite/rpl/t/rpl_auto_increment_update_failure.test:
Added test to verify the following two properties:
P1) insert/update in an autoinc column causes statement to
be logged in row format if binlog_format=mixed
P2) if binlog_format=mixed, and a trigger or function contains
two or more inserts/updates in a table that has an autoinc
column, then the slave should not go out of sync, even if
there are concurrent transactions.
sql/sql_base.cc:
Added function 'has_write_table_with_auto_increment' to check
if one (or more) write tables have auto_increment columns.
Removed function 'has_two_write_locked_tables_with_auto_increment',
because the function is included in function
'has_write_table_with_auto_increment'.
"load data" statements were written to the binlog as a mix of the original statement
and bits recreated from parse-info. This relied on implementation details and broke
with IGNORE_SPACES and versioned comments.
We now completely resynthesize the query for LOAD DATA for binlog (which among other
things normalizes them somewhat with regard to case, spaces, etc.).
We have already parsed the query properly, so we make use of that rather
than mix-and-match string literals and parsed items.
This should make us safe with regard to versioned comments, even those
spanning multiple tokens. Also no longer affected by IGNORE_SPACES.
mysql-test/r/mysqlbinlog.result:
LOAD DATA INFILE normalized
mysql-test/suite/binlog/r/binlog_killed_simulate.result:
LOAD DATA INFILE normalized
mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result:
LOAD DATA INFILE normalized
mysql-test/suite/binlog/r/binlog_stm_blackhole.result:
LOAD DATA INFILE normalized
mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result:
LOAD DATA INFILE normalized
mysql-test/suite/rpl/r/rpl_innodb_mixed_dml.result:
LOAD DATA INFILE normalized
mysql-test/suite/rpl/r/rpl_loaddata.result:
LOAD DATA INFILE normalized
mysql-test/suite/rpl/r/rpl_loaddata_fatal.result:
LOAD DATA INFILE normalized; offsets adjusted to reflect that
mysql-test/suite/rpl/r/rpl_loaddata_map.result:
LOAD DATA INFILE normalized
mysql-test/suite/rpl/r/rpl_loaddatalocal.result:
test for #43746 - trying to break LOAD DATA part of parser
mysql-test/suite/rpl/r/rpl_stm_log.result:
LOAD DATA INFILE normalized
mysql-test/suite/rpl/t/rpl_loaddatalocal.test:
try to break the LOAD DATA part of the parser (test for #43746)
mysql-test/t/mysqlbinlog.test:
LOAD DATA INFILE normalized; adjust offsets to reflect that
sql/log_event.cc:
clean up Load_log_event::print_query and friends so they don't print
excess spaces. add support for printing charset names to print_query.
sql/log_event.h:
We already have three places where we synthesize LOAD DATA queries.
Better use one of those!
sql/sql_lex.h:
When binlogging LOAD DATA statements, we make up the statement to
be logged (from the parse-info, rather than substrings of the
original query) now. Consequently, we no longer need (string-)
pointers into the original query.
sql/sql_load.cc:
Completely rewrote write_execute_load_query_log_event() to synthesize the
LOAD DATA statement wholesale, rather than piece it together from
synthesized bits and literal excerpts from the original query. This
will not only give us a nice, normalized statement (all uppercase,
no excess spaces, etc.), it will also handle comments, including
versioned comments right, which is certainly more than we can say
about the previous incarnation.
sql/sql_yacc.yy:
We're no longer assembling LOAD DATA statements from bodyparts of the
original query, so some bookkeeping in the parser can go.
All statements executed by mysql_upgrade are binlogged and then are replicated to slave.
This will result in some errors. The report of this bug has demonstrated some examples.
Master and slave should be upgraded separately. All statements executed by
mysql_upgrade will not be binlogged.
--write-binlog and --skip-write-binlog options are added into mysql_upgrade.
These options control whether sql statements are binlogged or not.
HA_ERR_WRONG_INDEX
In RBR, disabling keys on slave table will break replication when
updating or deleting a record. When the slave thread tries to
find the row, by searching in the storage engine, it checks
whether the table has a key or not. If it has one, then the slave
thread uses it to search the record.
Nonetheless, the slave only checks whether the key exists or not,
it does not verify if it is active. Should the key be
disabled (eg, DBA has issued an ALTER TABLE ... DISABLE KEYS)
then it will result in error: HA_ERR_WRONG_INDEX.
This patch addresses this issue by making the slave thread also
check whether the key is active or not before actually using it.
The failure is not reproduced on 5.1, so enable the 'rpl_cross_version' test.
mysql-test/suite/rpl/t/disabled.def:
Got rid of the line for enabling 'rpl_cross_version' test.
Network error happened here, but it can be caused by CR_CONNECTION_ERROR,
CR_CONN_HOST_ERROR, CR_SERVER_GONE_ERROR, CR_SERVER_LOST, ER_CON_COUNT_ERROR,
and ER_SERVER_SHUTDOWN. We just check CR_SERVER_LOST here, so the test fails.
To fix the problem, check all errors that can be cause by the master shutdown.
mysql-test/extra/rpl_tests/rpl_get_master_version_and_clock.test:
Added a 'if' sentence to check all errors that can be cause by the master shutdown.
mysql-test/suite/rpl/r/rpl_get_master_version_and_clock.result:
Test result is updated duo to the patch of bug#46931
But there is no Last_IO_Error reported.
On the master, if a binary log event is larger than max_allowed_packet,
ER_MASTER_FATAL_ERROR_READING_BINLOG and the specific reason of this error is
sent to a slave when it requests a dump from the master, thus leading
the I/O thread to stop.
On a slave, the I/O thread stops when receiving a packet larger than max_allowed_packet.
In both cases, however, there was no Last_IO_Error reported.
This patch adds code to report the Last_IO_Error and exact reason before stopping the
I/O thread and also reports the case the out memory pops up while
handling packets from the master.
sql/sql_repl.cc:
The master send the Specific reasons instead of "error reading log entry" to the slave which is requesting a dump.
if an fatal error is returned by read_log_event function.
The test case rpl_do_grant fails sporadically on PB2 with "Access
denied for user 'create_rout_db'@'localhost' ...". Inspecting the
test case, one may find that if issues a GRANT on the master
connection and immediately after it creates two new connections
(one to the master and one to the slave) using the credentials
set with the GRANT.
Unfortunately, there is no synchronization between master and
slave after the grant and before the connections are
established. This can result in slave not having executed the
GRANT by the time the connection is attempted.
This patch fixes this by deploying a sync_slave_with_master
between the grant and the connections attempt.
The test case creates two temporary tables, then closes the
connection, waits for it to disconnect, then syncs the slave with
the master, checks for remaining opened temporary tables on
slave (which should be 0) and finally drops the used
database (mysqltest).
Unfortunately, sometimes, the test fails with one open table on
the slave. This seems to be caused by the fact that waiting for
the connection to be closed is not sufficient. The test needs to
wait for the DROP event to be logged and only then synchronize
the slave with the master and proceed with the check. This is
caused by the asynchronous nature of the disconnect wrt
binlogging of the DROP temporary table statement.
We fix this by deploying a call to wait_for_binlog_event.inc
on the test case, which makes execution to wait for the DROP
temp tables event before synchronizing master and slave.
In RBR, There is an inconsistency between slaves and master.
When INSERT statement which includes an auto_increment field is executed,
Store engine of master will check the value of the auto_increment field.
It will generate a sequence number and then replace the value, if its value is NULL or empty.
if the field's value is 0, the store engine will do like encountering the NULL values
unless NO_AUTO_VALUE_ON_ZERO is set into SQL_MODE.
In contrast, if the field's value is 0, Store engine of slave always generates a new sequence number
whether or not NO_AUTO_VALUE_ON_ZERO is set into SQL_MODE.
SQL MODE of slave sql thread is always consistency with master's.
Another variable is related to this bug.
If generateing a sequence number is decided by the values of
table->auto_increment_field_not_null and SQL_MODE(if includes MODE_NO_AUTO_VALUE_ON_ZERO)
The table->auto_increment_is_not_null is FALSE, which causes this bug to appear. ..
Essentially, Bug#45574 results in this bug. The 'CREATE DATABASE IF NOT EXISTS' statement was not
binlogged, when the database has existed.
Sometimes, the master and slaves become inconsistent. The "CREATE DATABASE
IF NOT EXISTS mysqltest1" statement is not binlogged
if the db 'mysqltest1' existed before the test case is executed.
So the db 'mysqltest1' can't be created on slave.
Patch of Bug#45574 has resolved this problem.
But I think it is better to replace 'mysqltest1' by default db 'test'.
If an EVENT is created without the DEFINER clause set explicitly or with it set
to CURRENT_USER, the master and slaves become inconsistent. This issue stems from
the fact that in both cases, the DEFINER is set to the CURRENT_USER of the current
thread. On the master, the CURRENT_USER is the mysqld's user, while on the slave,
the CURRENT_USER is empty for the SQL Thread which is responsible for executing
the statement.
To fix the problem, we do what follows. If the definer is not set explicitly,
a DEFINER clause is added when writing the query into binlog; if 'CURRENT_USER' is
used as the DEFINER, it is replaced with the value of the current user before
writing to binlog.
mysql-test/suite/rpl/r/rpl_create_if_not_exists.result:
Updated the result file after fixing bug#44331
mysql-test/suite/rpl/r/rpl_drop_if_exists.result:
Updated the result file after fixing bug#44331
mysql-test/suite/rpl/r/rpl_events.result:
Test result of Bug#44331
mysql-test/suite/rpl/r/rpl_innodb_mixed_dml.result:
Updated the result file after fixing bug#44331
mysql-test/suite/rpl/t/rpl_events.test:
Added test to verify if the definer is consistent between master and slave
when the event is created without the DEFINER clause set explicitly or the
DEFINER is set to CURRENT_USER
sql/events.cc:
The "create_query_string" function is added to create a new query string
for removing executable comments.
sql/sql_yacc.yy:
The remember_name token was added for recording the offset of EVENT_SYM.
Slave does not correctly handle "expected errors" leading to inconsistencies
between the mater and slave. Specifically, when a statement changes both
transactional and non-transactional tables, the transactional changes are
automatically rolled back on the master but the slave ignores the error and
does not roll them back thus leading to inconsistencies.
To fix the problem, we automatically roll back a statement that fails on
the slave but note that the transaction is not rolled back unless a "rollback"
command is in the relay log file.
mysql-test/extra/rpl_tests/rpl_mixing_engines.test:
Enabled item 13.e which was disabled because of the bug fixed by the
current and removed item 14 which was introduced by mistake.
binlog
Mixing transactional (T) and non-transactional (N) tables on behalf of a
transaction may lead to inconsistencies among masters and slaves in STATEMENT
mode. The problem stems from the fact that although modifications done to
non-transactional tables on behalf of a transaction become immediately visible
to other connections they do not immediately get to the binary log and therefore
consistency is broken. Although there may be issues in mixing T and M tables in
STATEMENT mode, there are safe combinations that clients find useful.
In this bug, we fix the following issue. Mixing N and T tables in multi-level
(e.g. a statement that fires a trigger) or multi-table table statements (e.g.
update t1, t2...) were not handled correctly. In such cases, it was not possible
to distinguish when a T table was updated if the sequence of changes was N and T.
In a nutshell, just the flag "modified_non_trans_table" was not enough to reflect
that both a N and T tables were changed. To circumvent this issue, we check if an
engine is registered in the handler's list and changed something which means that
a T table was modified.
Check WL 2687 for a full-fledged patch that will make the use of either the MIXED or
ROW modes completely safe.
mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result:
Truncate statement is wrapped in BEGIN/COMMIT.
mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result:
Truncate statement is wrapped in BEGIN/COMMIT.
