mysqlbinlog --database parameter was being ignored when processing
row events. As such no event filtering would take place.
This patch addresses this by deploying a call to shall_skip_database
when table_map_events are handled (as these contain also the name of
the database). All other rows events referencing the table id for the
filtered map event, will also be skipped.
Make the caller of Query_log_event, Execute_load_log_event
constructors and THD::binlog_query to provide the error code
instead of having the constructors to figure out the error code.
If secure-file-priv was set on slave, it became unable to execute
LOAD DATA INFILE statements sent from master using mixed or
statement-based replication.
This patch fixes the issue by ignoring this security restriction
and checking if the files are created and read by the slave in the
--slave-load-tmpdir while executing the SQL Thread.
- Remove bothersome warning messages. This change focuses on the warnings
that are covered by the ignore file: support-files/compiler_warnings.supp.
- Strings are guaranteed to be max uint in length
conflicts:
Text conflict in client/mysqltest.cc
Text conflict in mysql-test/include/wait_until_connected_again.inc
Text conflict in mysql-test/lib/mtr_report.pm
Text conflict in mysql-test/mysql-test-run.pl
Text conflict in mysql-test/r/events_bugs.result
Text conflict in mysql-test/r/log_state.result
Text conflict in mysql-test/r/myisam_data_pointer_size_func.result
Text conflict in mysql-test/r/mysqlcheck.result
Text conflict in mysql-test/r/query_cache.result
Text conflict in mysql-test/r/status.result
Text conflict in mysql-test/suite/binlog/r/binlog_index.result
Text conflict in mysql-test/suite/binlog/r/binlog_innodb.result
Text conflict in mysql-test/suite/rpl/r/rpl_packet.result
Text conflict in mysql-test/suite/rpl/t/rpl_packet.test
Text conflict in mysql-test/t/disabled.def
Text conflict in mysql-test/t/events_bugs.test
Text conflict in mysql-test/t/log_state.test
Text conflict in mysql-test/t/myisam_data_pointer_size_func.test
Text conflict in mysql-test/t/mysqlcheck.test
Text conflict in mysql-test/t/query_cache.test
Text conflict in mysql-test/t/rpl_init_slave_func.test
Text conflict in mysql-test/t/status.test
Problem: when the server reads a log_event from file, it should read
the post-header lengths from the format_description_log_event. Some
event types which currently have post-header length 0 did not do this,
and instead had a hard-coded zero length for the post-header. That
means the current server version will not be able to read future
versions of these events.
Fix: make the reader functions read the post-header.
Problem: When an Incident_log_event contains a bad incident number on disk,
the server crashes with an assertion.
Fix: Don't validate input with assertions. Use errors.
Rotate event is automatically generated and written when rotating binary
log or relay log. Rotate events for relay logs are usually ignored by slave
SQL thread becuase they have the same server id as that of the slave.
However, if --replicate-same-server-id is enabled, rotate event
for relay log would be treated as if it's a rotate event from master, and
would be executed by slave to update the rli->group_master_log_name and
rli->group_master_log_pos to a wrong value and cause the MASTER_POS_WAIT
function to fail and return NULL.
This patch fixed this problem by setting a flag bit (LOG_EVENT_RELAY_LOG_F)
in the event to tell the SQL thread to ignore these Rotate events generated
for relay logs.
This patch also added another binlog event flag bit (LOG_EVENT_ARTIFICIAL_F)
to distinquish faked events, the method used before this was by checking if
log_pos was zero.
- Implementing --base64-format=decode-rows, to display
SQL-alike decoded row events without their BINLOG statements.
- Adding --base64-format=decode-rows into tests when
calling mysqlbinlog to avoid non-deterministic results
- Removing resetting of last_table_id in "RESET MASTER",
which appeared to be dangerous.
Implementing -v command line parameter to mysqlbinlog
to decode and print row events.
mysql-test/include/mysqlbinlog_row_engine.inc
mysql-test/r/mysqlbinlog_row.result
mysql-test/r/mysqlbinlog_row_big.result
mysql-test/r/mysqlbinlog_row_innodb.result
mysql-test/r/mysqlbinlog_row_myisam.result
mysql-test/r/mysqlbinlog_row_trans.result
mysql-test/t/mysqlbinlog_row.test
mysql-test/t/mysqlbinlog_row_big.test
mysql-test/t/mysqlbinlog_row_innodb.test
mysql-test/t/mysqlbinlog_row_myisam.test
mysql-test/t/mysqlbinlog_row_trans.test
Adding tests
client/Makefile.am
Adding new files to symlink
client/mysqlbinlog.cc
Adding -v option
sql/log_event.cc
Impelentations of the new methods
sql/log_event.h
Declaration of the new methods and member
sql/mysql_priv.h
Adding new function prototype
sql/rpl_tblmap.cc
Adding pre-processor conditions
sql/rpl_tblmap.h
Adding pre-processor conditions
sql/rpl_utility.h
Adding pre-processor conditions
sql/sql_base.cc
Adding reset_table_id_sequence() function.
sql/sql_repl.cc
Resetting table_id on "RESET MASTER"
.bzrignore
Ignoring new symlinked files
The problem of this bug is that we need to get the list of tables
to be updated for a multi-table update statement, which requires to
open all the tables referenced by the statement and resolve all
the fields involved in update in order to figure out the list of
tables for update. However if there are replicate filter rules,
some tables might not exist on slave and result in a failure
before we could examine the filter rules.
I think the whole problem can not be solved on slave alone,
the master must record and send the information of tables
involved for update to slave, so that the slave do not need to
open all the tables referenced by the multi-table update statement to
figure out which tables are involved for update.
So a status variable is added to Query_log event to store the
value of table map for update on master. And on slave, it will
try to get the value of this variable and use it to examine
filter rules without opening any tables on slave, if this values
is not available, the old approach is used and thus the bug will
still occur for when replicating from old masters.
The bug allow multiple executing transactions working with non-transactional
to interfere with each others by interleaving the events of different trans-
actions.
Bug is fixed by writing non-transactional events to the transaction cache and
flushing the cache to the binary log at statement commit. To mimic the behavior
of normal statement-based replication, we flush the transaction cache in row-
based mode when there is no committed statements in the transaction cache,
which means we are committing the first one. This means that it will be written
to the binary log as a "mini-transaction" with just the rows for the statement.
Note that the changes here does not take effect when building the server with
HAVE_TRANSACTIONS set to false, but it is not clear if this was possible before
this patch either.
For row-based logging, we also have that when AUTOCOMMIT=1, the code now always
generates a BEGIN/COMMIT pair for single statements, or BEGIN/ROLLBACK pair in the
case of non-transactional changes in a statement that was rolled back. Note that
for the case where changes to a non-transactional table causes a rollback due
to error, the statement will now be logged with a BEGIN/ROLLBACK pair, even
though some changes has been committed to the non-transactional table.
Problem: in mixed and statement mode, a query that refers to a
system variable will use the slave's value when replayed on
slave. So if the value of a system variable is inserted into a
table, the slave will differ from the master.
Fix: mark statements that refer to a system variable as "unsafe",
meaning they will be replicated by row in mixed mode and produce a warning
in statement mode. There are some exceptions: some variables are actually
replicated. Those should *not* be marked as unsafe.
BUG#34732: mysqlbinlog does not print default values for auto_increment variables
Problem: mysqlbinlog does not print default values for some variables,
including auto_increment_increment and others. So if a client executing
the output of mysqlbinlog has different default values, replication will
be wrong.
Fix: Always print default values for all variables that are replicated.
I need to fix the two bugs at the same time, because the test cases would
fail if I only fixed one of them.
The error message due to lack of the default value for an extra field
was not as informative as it should be.
Fixed with improving the scheme of gathering, propagating and reporting
errors in applying rows events.
The scheme is in the following.
Any kind of error of processing of a row event incidents are to be
registered with my_error().
In the end Rows_log_event::do_apply_event() invokes rli->report() with the
message to display consisting of all the errors.
This mimics `show warnings' displaying.
A simple test checks three errors in processing an event.
Two hunks - a user level error and pushing it into the list -
have been devoted to already fixed Bug@31702.
Some open issues relating to this artifact listed on BUG@21842 page and
on WL@3679.
Todo: to synchronize the statement in the tests comments on Update and Delete
events may not stop when an extra field does not have a default with wl@3228 spec.
Main problem: mysql 5.1 cannot read binlogs from 4.1.
Subproblem 1: There is a mistake in sql_ex_info::init. The read_str()
function updates its first argument to point to the next character to
read. However, it is applied only to a copy of the buffer pointer, so the
real buffer pointer is not updated.
Fix 1: do not take a copy of the buffer pointer. The copy was needed
because sql_ex_info::init does not use the const attribute on some of its
arguments. So we add the const attribute, too.
Subproblem 2: The first BINLOG statement is asserted to be a
FORMAT_DESCRIPTION_LOG_EVENT, but 4.1 binlogs begin with START_EVENT_V3.
Fix 2: allow START_EVENT_V3 too.
Problem: Replication fails when master is mysql-5.1-wl2325-5.0-drop6 and
slave is mysql-5.1-new-rpl. The reason is that, in
mysql-5.1-wl2325-5.0-drop6, the event type id's were different than in
mysql-5.1-new-rpl.
Fix (in mysql-5.1-new-rpl):
(1) detect that the server that generated the events uses the old
format, by checking the server version of the format_description_log_event
This patch recognizes mysql-5.1-wl2325-5.0-drop6p13-alpha,
mysql-5.1-wl2325-5.0-drop6, mysql-5.1-wl2325-5.0, mysql-5.1-wl2325-no-dd.
(2) if the generating server is old, map old event types to new event
types using a permutation array.
I've also added a test case which reads binlogs for four different
versions.
Problem: it is unsafe to read base64-printed events without first
reading the Format_description_log_event (FD). Currently, mysqlbinlog
cannot print the FD.
As a side effect, another bug has also been fixed: When mysqlbinlog
--start-position=X was specified, no ROLLBACK was printed. I changed
this, so that ROLLBACK is always printed.
This patch does several things:
- Format_description_log_event (FD) now print themselves in base64
format.
- mysqlbinlog is now able to print FD events. It has three modes:
--base64-output=auto Print row events in base64 output, and print
FD event. The FD event is printed even if
it is outside the range specified with
--start-position, because it would not be
safe to read row events otherwise. This is
the default.
--base64-output=always Like --base64-output=auto, but also print
base64 output for query events. This is
like the old --base64-output flag, which
is also a shorthand for
--base64-output=always
--base64-output=never Never print base64 output, generate error if
row events occur in binlog. This is
useful to suppress the FD event in binlogs
known not to contain row events (e.g.,
because BINLOG statement is unsafe,
requires root privileges, is not SQL, etc)
- the BINLOG statement now handles FD events correctly, by setting
the thread's rli's relay log's description_event_for_exec to the
loaded event.
In fact, executing a BINLOG statement is almost the same as reading
an event from a relay log. Before my patch, the code for this was
separated (exec_relay_log_event in slave.cc executes events from
the relay log, mysql_client_binlog_statement in sql_binlog.cc
executes BINLOG statements). I needed to augment
mysql_client_binlog_statement to do parts of what
exec_relay_log_event does. Hence, I did a small refactoring and
moved parts of exec_relay_log_event to a new function, which I
named apply_event_and_update_pos. apply_event_and_update_pos is
called both from exec_relay_log_event and from
mysql_client_binlog_statement.
- When a non-FD event is executed in a BINLOG statement, without
previously executing a FD event in a BINLOG statement, it generates
an error, because that's unsafe. I took a new error code for that:
ER_NO_FORMAT_DESCRIPTION_EVENT_BEFORE_BINLOG_STATEMENTS.
In order to get a decent error message containing the name of the
event, I added the class method char*
Log_event::get_type_str(Log_event_type type), which returns a
string name for the given Log_event_type. This is just like the
existing char* Log_event::get_type_str(), except it is a class
method that takes the log event type as parameter.
I also added PRE_GA_*_ROWS_LOG_EVENT to Log_event::get_type_str(),
so that names of old rows event are properly printed.
- When reading an event, I added a check that the event type is known
by the current Format_description_log_event. Without this, it may
crash on bad input (and I was struck by this several times).
- I patched the following test cases, which all contain BINLOG
statements for row events which must be preceded by BINLOG
statements for FD events:
- rpl_bug31076
While I was here, I fixed some small things in log_event.cc:
- replaced hard-coded 4 by EVENT_TYPE_OFFSET in 3 places
- replaced return by DBUG_VOID_RETURN in one place
- The name of the logfile can be '-' to indicate stdin. Before my
patch, the code just checked if the first character is '-'; now it
does a full strcmp(). Probably, all arguments that begin with a -
are already handled somewhere else as flags, but I still think it
is better that the code reflects what it is supposed to do, with as
little dependencies as possible on other parts of the code. If we
one day implement that all command line arguments after -- are
files (as most unix tools do), then we need this.
I also fixed the following in slave.cc:
- next_event() was declared twice, and queue_event was not static but
should be static (not used outside the file).
without PK
Bug#31609 Not all RBR slave errors reported as errors
bug#32468 delete rows event on a table with foreign key constraint fails
The first two bugs comprise idempotency issues.
First, there was no error code reported under conditions of the bug
description although the slave sql thread halted.
Second, executions were different with and without presence of prim key in
the table.
Third, there was no way to instruct the slave whether to ignore an error
and skip to the following event or to halt.
Fourth, there are handler errors which might happen due to idempotent
applying of binlog but those were not listed among the "idempotent" error
list.
All the named issues are addressed.
Wrt to the 3rd, there is the new global system variable, changeble at run
time, which controls the slave sql thread behaviour.
The new variable allows further extensions to mimic the sql_mode
session/global variable.
To address the 4th, the new bug#32468 had to be fixed as it was staying
in the way.
Query_log_event::error_code
A query can perform completely having the local var error of mysql_$query
zero, where $query in insert, update, delete, load,
and be binlogged with error_code e.g KILLED_QUERY while there is no
reason do to so.
That can happen because Query_log_event consults thd->killed flag to
evaluate error_code.
Fixed with implementing a scheme suggested and partly implemented at
time of bug@22725 work-on. error_status is cached immediatly after the
control leaves the main rows-loop and that instance always corresponds
to `error' the local of mysql_$query functions. The cached value
is passed to Query_log_event constructor, not the default thd->killed
which can be changed in between of the caching and the constructing.
Documented some binlog events using doxygen. More will be done later.
Also fixed typos in other comments and added remarks about dubious code.
Only comments are affected, there is no change to the actual code.
When replicating an update pair (before image, after image) under row-based
replication, and the before image is not found on the slave, the after image
was not discared, and was hence read as a before image for the next row.
Eventually, this lead to an after image being read outside the block of rows
in the event, causing an assertion to fire.
This patch fixes this by reading the after image in the event that the row
was not found on the slave, adds some extra debug assertion to catch future
errors earlier, and also adds a few non-debug checks to prevent reading
outside the block of the event.
is possible):
When skipping the beginning of a transaction starting with BEGIN, the OPTION_BEGIN
flag was not set correctly, which caused the slave to not recognize that it was
inside a group. This patch sets the OPTION_BEGIN flag for BEGIN, COMMIT, ROLLBACK,
and XID events. It also adds checks if inside a group before decreasing the
slave skip counter to zero.
Begin_query_log_event was not marked that it could not end a group, which is now
corrected.
using TPC-B):
Problem: A RBR event can contain incomplete row data (only key value and
fields which have been changed). In that case, when the row is unpacked
into record and written to a table, the missing fields get incorrect NULL
values leading to master-slave inconsistency.
Solution: Use values found in slave's table for columns which are not given
in the rows event. The code for writing a single row uses the following
algorithm:
1. unpack row_data into table->record[0],
2. try to insert record,
3. if duplicate record found, fetch it into table->record[0],
4. unpack row_data into table->record[0],
5. write table->record[0] into the table.
Where row_data is the row as stored in the data area of a rows event.
Thus:
a) unpacking of row_data happens at the time when row is written into
a table,
b) when unpacking (in step 4), only columns present in row_data are
overwritten - all other columns remain as they were found in the table.
Since all data needed for the above algorithm is stored inside
Rows_log_event class, functions which locate and write rows are turned
into methods of that class.
replace_record() -> Rows_log_event::write_row()
find_and_fetch_row() -> Rows_log_event::find_row()
Both methods take row data from event's data buffer - the row being
processed is pointed by m_curr_row. They unpack the data as needed into
table's record buffers record[0] or record[1]. When row is unpacked,
m_curr_row_end is set to point at next row in the data buffer.
Other changes introduced in this changeset:
- Change signature of unpack_row(): don't report errors and don't
setup table's rw_set here. Errors can happen only when setting default
values in prepare_record() function and are detected there.
- In Rows_log_event and derived classes, don't pass arguments to
the execution primitives (do_...() member functions) but use class
members instead.
- Move old row handling code into log_event_old.cc to be used by
*_rows_log_event_old classes.
Also, a new test rpl_ndb_2other is added which tests basic replication
from master using ndb tables to slave storing the same tables using
(possibly) different engine (myisam,innodb).
Test is based on existing tests rpl_ndb_2myisam and rpl_ndb_2innodb.
However, these tests doesn't work for various reasons and currently are
disabled (see BUG#19227).
The new test differs from the ones it is based on as follows:
1. Single test tests replication with different storage engines on slave
(myisam, innodb, ndb).
2. Include file extra/rpl_tests/rpl_ndb_2multi_eng.test containing
original tests is replaced by extra/rpl_tests/rpl_ndb_2multi_basic.test
which doesn't contain tests using partitioned tables as these don't work
currently. Instead, it tests replication to a slave which has more or
less columns than master.
3. Include file include/rpl_multi_engine3.inc is replaced with
include/rpl_multi_engine2.inc. The later differs by performing slightly
different operations (updating more than one row in the table) and
clearing table with "TRUNCATE TABLE" statement instead of "DELETE FROM"
as replication of "DELETE" doesn't work well in this setting.
4. Slave must use option --log-slave-updates=0 as otherwise execution of
replication events generated by ndb fails if table uses a different
storage engine on slave (see BUG#29569).
Fixed failing func_misc test for embedded server
Added casts to avoid compiler warnings
Removed Table_locks_immediate as it's depending on log file cacheing
Changed type of get_time() to avoid warnings
Removed testing if purger master logs succeded as this is not deterministic
