(MYSQLBINLOG -V CRASHES WITH THAT BINLOG)
Problem: If slave receives a corrupted row event,
slave server is crashing.
Analysis: When slave is unpacking the row event, it is
not validating the data before applying the event. If the
data is corrupted for eg: the length of a field is wrong,
it could end up reading wrong data leading to a crash.
A similar problem happens when mysqlbinlog tool is used
against a corrupted binlog using '-v' option. Due to -v
option, the tool tries to print the values of all the
fields. Corrupted field length could lead to a crash.
Fix: Before unpacking the field, a verification
will be made on the length. If it falls into the event
range, only then it will be unpacked. Otherwise,
"ER_SLAVE_CORRUPT_EVENT" error will be thrown.
Incase mysqlbinlog -v case, the field value will not be
printed and the processing of the file will be stopped.
sql/field.h:
Removed a function which is not required anymore
sql/log_event.cc:
Adding a validation on the field length before
the tool tries to print the value.
sql/log_event.h:
Changing unpack_row call according to the new arguments
sql/log_event_old.h:
Changing unpack_row call according to the new arguments
sql/rpl_record.cc:
Adding a new argument 'row_end' which tells
the end position of the complete data in the
row event. It will be used to do validation
before doing 'unpack' field.
sql/rpl_record.h:
Adding a new argument 'row_end' which tells
the end position of the complete data in the
row event. It will be used to do validation
before doing 'unpack' field.
sql/rpl_utility.cc:
Now calc_field_size() is required for client too.
Added 'from_end' as extra parameter to Field::unpack() to detect wrong from data.
Change ha_archive::unpack_row() to detect wrong field lengths.
Replication code changed to detect wrong field information in events.
mysql-test/r/archive.result:
dded test case for lp:917689
sql/field.cc:
Added 'from_end' as extra parameter to Field::unpack() to detect wrong from data.
Removed not used 'unpack_key' functions.
sql/field.h:
Added 'from_end' as extra parameter to Field::unpack() to detect wrong from data.
Removed not used 'unpack_key' functions.
Removed some not needed unpack() functions.
sql/filesort.cc:
Added buffer end parameter to unpack_addon_fields()
sql/log_event.h:
Added end of buffer argument to unpack_row()
sql/log_event_old.cc:
Added end of buffer argument to unpack_row()
sql/log_event_old.h:
Added end of buffer argument to unpack_row()
sql/records.cc:
Added buffer end parameter to unpack_addon_fields()
sql/rpl_record.cc:
Added end of buffer argument to unpack_row()
Added detection of wrong field information in events
sql/rpl_record.h:
Added end of buffer argument to unpack_row()
sql/rpl_record_old.cc:
Added end of buffer argument to unpack_row()
Added detection of wrong field information in events
sql/rpl_record_old.h:
Added end of buffer argument to unpack_row()
sql/table.h:
Added buffer end parameter to unpack()
storage/archive/ha_archive.cc:
Change ha_archive::unpack_row() to detect wrong field lengths.
This fixes lp:917689
sql/sql_insert.cc:
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
******
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
sql/sql_table.cc:
small cleanup
******
small cleanup
A lot of small fixes and new test cases.
client/mysqlbinlog.cc:
Cast removed
client/mysqltest.cc:
Added missing DBUG_RETURN
include/my_pthread.h:
set_timespec_time_nsec() now only takes one argument
mysql-test/t/date_formats.test:
Remove --disable_ps_protocl as now also ps supports microseconds
mysys/my_uuid.c:
Changed to use my_interval_timer() instead of my_getsystime()
mysys/waiting_threads.c:
Changed to use my_hrtime()
sql/field.h:
Added bool special_const_compare() for fields that may convert values before compare (like year)
sql/field_conv.cc:
Added test to get optimal copying of identical temporal values.
sql/item.cc:
Return that item_int is equal if it's positive, even if unsigned flag is different.
Fixed Item_cache_str::save_in_field() to have identical null check as other similar functions
Added proper NULL check to Item_cache_int::save_in_field()
sql/item_cmpfunc.cc:
Don't call convert_constant_item() if there is nothing that is worth converting.
Simplified test when years should be converted
sql/item_sum.cc:
Mark cache values in Item_sum_hybrid as not constants to ensure they are not replaced by other cache values in compare_datetime()
sql/item_timefunc.cc:
Changed sec_to_time() to take a my_decimal argument to ensure we don't loose any sub seconds.
Added Item_temporal_func::get_time() (This simplifies some things)
sql/mysql_priv.h:
Added Lazy_string_decimal()
sql/mysqld.cc:
Added my_decimal constants max_seconds_for_time_type, time_second_part_factor
sql/table.cc:
Changed expr_arena to be of type CONVENTIONAL_EXECUTION to ensure that we don't loose any items that are created by fix_fields()
sql/tztime.cc:
TIME_to_gmt_sec() now sets *in_dst_time_gap in case of errors
This is needed to be able to detect if timestamp is 0
storage/maria/lockman.c:
Changed from my_getsystime() to set_timespec_time_nsec()
storage/maria/ma_loghandler.c:
Changed from my_getsystime() to my_hrtime()
storage/maria/ma_recovery.c:
Changed from my_getsystime() to mmicrosecond_interval_timer()
storage/maria/unittest/trnman-t.c:
Changed from my_getsystime() to mmicrosecond_interval_timer()
storage/xtradb/handler/ha_innodb.cc:
Added support for new time,datetime and timestamp
unittest/mysys/thr_template.c:
my_getsystime() -> my_interval_timer()
unittest/mysys/waiting_threads-t.c:
my_getsystime() -> my_interval_timer()
- Fixed some issues with partitions and connection_string, which also fixed lp:716890 "Pre- and post-recovery crash in Aria"
- Fixed wrong assert in Aria
Now need to merge with latest xtradb before pushing
sql/ha_partition.cc:
Ensure that m_ordered_rec_buffer is not freed before close.
sql/mysqld.cc:
Changed to use opt_stack_trace instead of opt_pstack.
Removed references to pstack
sql/partition_element.h:
Ensure that connect_string is initialized
storage/maria/ma_key_recover.c:
Fixed wrong assert
Normally, auto_increment value is generated for the column by
inserting either NULL or 0 into it. NO_AUTO_VALUE_ON_ZERO
suppresses this behavior for 0 so that only NULL generates
the auto_increment value. This behavior is also followed by
a slave, specifically by the SQL Thread, when applying events
in the statement format from a master. However, when applying
events in the row format, the flag was ignored thus causing
an assertion failure:
"Assertion failed: next_insert_id == 0, file .\handler.cc"
In fact, we never need to generate a auto_increment value for
the column when applying events in row format on slave. So we
don't allow it to happen by using 'MODE_NO_AUTO_VALUE_ON_ZERO'.
Refactoring: Get rid of all the sql_mode checks to rows_log_event
when applying it for avoiding problems caused by the inconsistency
of the sql_mode on slave and master as the sql_mode is not set for
Rows_log_event.
mysql-test/extra/rpl_tests/rpl_auto_increment.test:
Added test to verify if the assertion of "next_insert_id == 0"
will fail in ha_external_lock() function.
mysql-test/suite/rpl/r/rpl_auto_increment.result:
Test result for bug#56662.
sql/log_event.cc:
Added code to not allow generation of auto_increment value when
processing rows event by adding 'MODE_NO_AUTO_VALUE_ON_ZERO' to
sql_mode.
sql/rpl_record.cc:
Added code to get rid of the 'MODE_STRICT_TRANS_TABLES' and
MODE_STRICT_ALL_TABLES check to the table when appling the
rows event and treat it as no strict.
Normally, auto_increment value is generated for the column by
inserting either NULL or 0 into it. NO_AUTO_VALUE_ON_ZERO
suppresses this behavior for 0 so that only NULL generates
the auto_increment value. This behavior is also followed by
a slave, specifically by the SQL Thread, when applying events
in the statement format from a master. However, when applying
events in the row format, the flag was ignored thus causing
an assertion failure:
"Assertion failed: next_insert_id == 0, file .\handler.cc"
In fact, we never need to generate a auto_increment value for
the column when applying events in row format on slave. So we
don't allow it to happen by using 'MODE_NO_AUTO_VALUE_ON_ZERO'.
Refactoring: Get rid of all the sql_mode checks to rows_log_event
when applying it for avoiding problems caused by the inconsistency
of the sql_mode on slave and master as the sql_mode is not set for
Rows_log_event.
mysql-test/extra/rpl_tests/rpl_auto_increment.test:
Added test to verify if the assertion of "next_insert_id == 0"
will fail in ha_external_lock() function.
mysql-test/suite/rpl/r/rpl_auto_increment.result:
Test result for bug#56662.
sql/log_event.cc:
Added code to not allow generation of auto_increment value when
processing rows event by adding 'MODE_NO_AUTO_VALUE_ON_ZERO' to
sql_mode.
sql/rpl_record.cc:
Added code to get rid of the 'MODE_STRICT_TRANS_TABLES' and
MODE_STRICT_ALL_TABLES check to the table when appling the
rows event and treat it as no strict.
Fixed compiler warnings in xtradb
Added back resetting of null bitmap but now in row_search_for_mysql()
storage/xtradb/row/row0sel.c:
Added back resetting of null bitmap but now in row_search_for_mysql()
Replication SET and ENUM fields from a big-endian to a little-
endian machine (or the opposite) that are represented using
more than 1 byte (SET fields with more than 8 members or ENUM
fields with more than 256 constants) will fail to replicate
correctly when using row-based replication.
The reason is that there are no pack() or unpack() functions
for Field_set or Field_enum, which make them rely on Field::pack
and Field::unpack. These functions pack data as strings, but
since Field_set and Field_enum use integral types for
representation, the fields are stored incorrectly on big-endian
machines.
This patch adds Field_enum::pack and Field_enum::unpack
functions that store the integral value correctly in the binary
log even on big-endian machines. Since Field_set inherits from
Field_enum, it will use the same functions for packing and
unpacking the field.
sql/field.cc:
Removing some obsolete debug printouts and adding Field_enum::pack
and Field_enum::unpack functions.
sql/field.h:
Adding helper functions for packing and unpacking 16- and
24-bit integral types.
Field_short::pack and Field_short::unpack now use these functions.
sql/rpl_record.cc:
Removing some obsolete debug printouts and adding some
more useful ones.
This patch:
- Moves all definitions from the mysql_priv.h file into
header files for the component where the variable is
defined
- Creates header files if the component lacks one
- Eliminates all include directives from mysql_priv.h
- Eliminates all circular include cycles
- Rename time.cc to sql_time.cc
- Rename mysql_priv.h to sql_priv.h
cant find record
Some engines return data for the record. Despite the fact that
the null bit is set for some fields, their old value may still in
the row. This can happen when unpacking an AI from the binlog on
top of a previous record in which a field is set to NULL, which
previously contained a value. Ultimately, this may cause the
comparison of records to fail when the slave is doing an index or
range scan.
We fix this by deploying a call to reset() for each field that is
set to null while unpacking a row from the binary log.
Furthermore, we also add mixed mode test case to cover the
scenario where updating and setting a field to null through a
Query event and later searching it through a rows event will
succeed.
Finally, we also change the reset() method, from Field_bit class,
so that it takes into account bits stored among the null bits and
not only the ones stored in the record.
mysql-test/suite/rpl/t/rpl_set_null_innodb.test:
InnoDB test.
mysql-test/suite/rpl/t/rpl_set_null_myisam.test:
MyISAM test.
mysql-test/suite/rpl_ndb/t/rpl_ndb_set_null.test:
NDB test.
sql/field.h:
Changed reset so that it also clears the bits
among the null_bits for the Field_bit class.
sql/rpl_record.cc:
Resetting field after setting it to null when unpacking
row.
Rename method as to not hide a base.
Reorder attributes initialization.
Remove unused variable.
Rework code to silence a warning due to assignment used as truth value.
sql/item_strfunc.cc:
Rename method as to not hide a base.
sql/item_strfunc.h:
Rename method as to not hide a base.
sql/log_event.cc:
Reorder attributes initialization.
sql/rpl_injector.cc:
Rework code to silence a warning due to assignment used as truth value.
sql/rpl_record.cc:
Remove unused variable.
sql/sql_db.cc:
Rework code to silence a warning due to assignment used as truth value.
sql/sql_parse.cc:
Rework code to silence a warning due to assignment used as truth value.
sql/sql_table.cc:
Rework code to silence a warning due to assignment used as truth value.
replicating
Replace c_ptr() calls with c_ptr_safe() calls to
avoid valgrind warnings.
Adding code to to handle the case that no metadata
was present in the table map for the column.
Allow first parameter to unpack_row() to be NULL,
in which case no source tables is used and hence
no checks nor conversions are done.
Clarifying some comments and fixing documentation
for unpack_row().
This patch fixes these warnings and some compile time warnings.
On top of that, it also fixes rpl_err_ignoredtable test failure.
This test was failing because the warning suppression text was not
matching the latest text. We fix this by making them match.
mysql-test/suite/rpl/t/rpl_err_ignoredtable.test:
Replaced the suppression text.
sql/rpl_record.cc:
- Fixed some compile time warnings (replaced %d --> %ld and
removed unused mas mask variable.
- Fixed valgrind warnings when using c_ptr(). Replaced with
c_ptr_safe().
sql/rpl_utility.cc:
- Fixed valgrind warnings when using c_ptr(). Replaced with
c_ptr_safe().
Row-based replication requires the types of columns on the
master and slave to be approximately the same (some safe
conversions between strings are allowed), but does not
allow safe conversions between fields of similar types such
as TINYINT and INT.
This patch implement type conversions between similar fields
on the master and slave.
The conversions are controlled using a new variable
SLAVE_TYPE_CONVERSIONS of type SET('ALL_LOSSY','ALL_NON_LOSSY').
Non-lossy conversions are any conversions that do not run the
risk of losing any information, while lossy conversions can
potentially truncate the value. The column definitions are
checked to decide if the conversion is acceptable.
If neither conversion is enabled, it is required that the
definitions of the columns are identical on master and slave.
Conversion is done by creating an internal conversion table,
unpacking the master data into it, and then copy the data to
the real table on the slave.
.bzrignore:
New files added
client/Makefile.am:
New files added
client/mysqlbinlog.cc:
Functions in rpl_utility.cc is now needed by mysqlbinlog.cc.
libmysqld/Makefile.am:
New files added
mysql-test/extra/rpl_tests/check_type.inc:
Test include file to check a single type conversion.
mysql-test/extra/rpl_tests/rpl_extraSlave_Col.test:
Switching to use INT instead of TEXT for column that should not have matching types.
mysql-test/extra/rpl_tests/rpl_row_basic.test:
Adding code to enable type conversions for BIT tests since InnoDB
cannot handle them properly due to incorrect information stored as
metadata.
mysql-test/extra/rpl_tests/type_conversions.test:
Test file to check a set of type conversions
with current settings of slave_type_conversions.
mysql-test/suite/rpl/t/rpl_typeconv.test:
Test file to test conversions from master to slave with
all possible values for slave_type_conversions.
The test also checks that the slave_type_conversions
variable works as expected.
sql/field.cc:
Changing definition of compatible_field_size to both check if
two field with identical base types are compatible and give an
order between them if they are compatible.
This only implement checking on the slave, so it will not affect
replication from an old master to a new slave.
sql/field.h:
Changing prototypes for functions:
- compatible_field_size()
- init_for_tmp_table()
- row_pack_length()
sql/log_event.cc:
Changing compability checks to build a conversion table if the fields
are compatible, but does not have the same base type.
sql/log_event_old.cc:
Changing compability checks to build a conversion table if the fields
are compatible, but does not have the same base type.
sql/mysql_priv.h:
Adding global option variable for SLAVE_TYPE_CONVERSIONS
sql/mysqld.cc:
Adding SLAVE_TYPE_CONVERSIONS global server variable.
sql/rpl_record.cc:
Changing unpack_row to use the conversion table if present.
sql/rpl_rli.h:
Removing function get_tabledef and replacing it with get_table_data().
This function retrieve data for table opened for replication, not just
table definition.
sql/rpl_utility.cc:
Function table_def::compatible_with is changed to compare table on master
and slave for compatibility and generate a conversions table if they are
compatible.
Computing real type of fields from metadata for ENUM and SET types.
Computing pack_length correctly for ENUM, SET, and BLOB types.
Adding optimization to not check compatibility if no
slave type conversions are enabled.
sql/rpl_utility.h:
Changing prototypes since implementation has changed.
Modifying table_def::type() to return real type instead of stored type.
sql/set_var.cc:
Adding SLAVE_TYPE_CONVERSIONS variable.
sql/set_var.h:
Adding SLAVE_TYPE_CONVERSIONS variable.
sql/share/errmsg.txt:
Adding error messages for slave type conversions.
sql/sql_class.h:
Adding SLAVE_TYPE_CONVERSIONS variable.
sql/sql_select.cc:
Correcting create_virtual_tmp_table() to compute null bit positions
correctly in the presence of bit fields.
Backporting BUG#43789 to mysql-5.1-bugteam
The replication was generating corrupted data, warning messages on Valgrind
and aborting on debug mode while replicating a "null" to "not null" field.
Specifically the unpack_row routine, was considering the slave's table
definition and trying to retrieve a field value, where there was nothing to be
retrieved, ignoring the fact that the value was defined as "null" by the master.
To fix the problem, we proceed as follows:
1 - If it is not STRICT sql_mode, implicit default values are used, regardless
if it is multi-row or single-row statement.
2 - However, if it is STRICT mode, then a we do what follows:
2.1 If it is a transactional engine, we do a rollback on the first NULL that is
to be set into a NOT NULL column and return an error.
2.2 If it is a non-transactional engine and it is the first row to be inserted
with multi-row, we also return the error. Otherwise, we proceed with the
execution, use implicit default values and print out warning messages.
Unfortunately, the current patch cannot mimic the behavior showed by the master
for updates on multi-tables and multi-row inserts. This happens because such
statements are unfolded in different row events. For instance, considering the
following updates and strict mode:
(master)
create table t1 (a int);
create table t2 (a int not null);
insert into t1 values (1);
insert into t2 values (2);
update t1, t2 SET t1.a=10, t2.a=NULL;
t1 would have (10) and t2 would have (0) as this would be handled as a
multi-row update. On the other hand, if we had the following updates:
(master)
create table t1 (a int);
create table t2 (a int);
(slave)
create table t1 (a int);
create table t2 (a int not null);
(master)
insert into t1 values (1);
insert into t2 values (2);
update t1, t2 SET t1.a=10, t2.a=NULL;
On the master t1 would have (10) and t2 would have (NULL). On
the slave, t1 would have (10) but the update on t1 would fail.
Backporting BUG#38173 to mysql-5.1-bugteam
The reason of the bug was incompatibile with the master side behaviour.
INSERT query on the master is allowed to insert into a table without specifying
values of DEFAULT-less fields if sql_mode is not strict.
Fixed with checking sql_mode by the sql thread to decide how to react.
Non-strict sql_mode should allow Write_rows event to complete.
todo: warnings can be shown via show slave status, still this is a
separate rather general issue how to show warnings for the slave threads.
vs not null
NOTE: Backporting the patch to next-mr.
The replication was generating corrupted data, warning messages on Valgrind
and aborting on debug mode while replicating a "null" to "not null" field.
Specifically the unpack_row routine, was considering the slave's table
definition and trying to retrieve a field value, where there was nothing to be
retrieved, ignoring the fact that the value was defined as "null" by the master.
To fix the problem, we proceed as follows:
1 - If it is not STRICT sql_mode, implicit default values are used, regardless
if it is multi-row or single-row statement.
2 - However, if it is STRICT mode, then a we do what follows:
2.1 If it is a transactional engine, we do a rollback on the first NULL that is
to be set into a NOT NULL column and return an error.
2.2 If it is a non-transactional engine and it is the first row to be inserted
with multi-row, we also return the error. Otherwise, we proceed with the
execution, use implicit default values and print out warning messages.
Unfortunately, the current patch cannot mimic the behavior showed by the master
for updates on multi-tables and multi-row inserts. This happens because such
statements are unfolded in different row events. For instance, considering the
following updates and strict mode:
(master)
create table t1 (a int);
create table t2 (a int not null);
insert into t1 values (1);
insert into t2 values (2);
update t1, t2 SET t1.a=10, t2.a=NULL;
t1 would have (10) and t2 would have (0) as this would be handled as a
multi-row update. On the other hand, if we had the following updates:
(master)
create table t1 (a int);
create table t2 (a int);
(slave)
create table t1 (a int);
create table t2 (a int not null);
(master)
insert into t1 values (1);
insert into t2 values (2);
update t1, t2 SET t1.a=10, t2.a=NULL;
On the master t1 would have (10) and t2 would have (NULL). On
the slave, t1 would have (10) but the update on t1 would fail.
NOTE: Backporting the patch to next-mr.
The reason of the bug was incompatibile with the master side behaviour.
INSERT query on the master is allowed to insert into a table without specifying
values of DEFAULT-less fields if sql_mode is not strict.
Fixed with checking sql_mode by the sql thread to decide how to react.
Non-strict sql_mode should allow Write_rows event to complete.
todo: warnings can be shown via show slave status, still this is a
separate rather general issue how to show warnings for the slave threads.
When using mixed mode the record values stored inside the storage
engine differed from the ones computed from the row event. This
happened because the prepare_record function was calling
empty_record macro causing some don't care bits to be left set.
Replacing the empty_record plus explicitly setting defaults with
restore_record to restore the record default values fixes this.
The error message due to lack of the default value for an extra field
was not as informative as it should be.
Fixed with improving the scheme of gathering, propagating and reporting
errors in applying rows events.
The scheme is in the following.
Any kind of error of processing of a row event incidents are to be
registered with my_error().
In the end Rows_log_event::do_apply_event() invokes rli->report() with the
message to display consisting of all the errors.
This mimics `show warnings' displaying.
A simple test checks three errors in processing an event.
Two hunks - a user level error and pushing it into the list -
have been devoted to already fixed Bug@31702.
Some open issues relating to this artifact listed on BUG@21842 page and
on WL@3679.
Todo: to synchronize the statement in the tests comments on Update and Delete
events may not stop when an extra field does not have a default with wl@3228 spec.
include/my_base.h:
A new handler level error code that is supposed to be mapped to a set of more
specific ER_ user level errors.
mysql-test/extra/rpl_tests/rpl_row_tabledefs.test:
Adding yet another extra fields to see more than one error in show
slave status' report.
mysql-test/suite/rpl/r/rpl_row_tabledefs_2myisam.result:
results changed (the error message etc)
mysql-test/suite/rpl/r/rpl_row_tabledefs_3innodb.result:
results changed
sql/log_event.cc:
Refining slave_rows_error_report to iterate on the list of gathered errors;
Simplifying signature of prepare_record as the function does not call
rli->report to leave that duty to the event's top level code.
sql/log_event.h:
adding a corrupt event error pushing. The error will be seen with
show slave status.
sql/log_event_old.cc:
similar to log_event.cc changes
sql/rpl_record.cc:
prepare_record only pushes an error to the list
sql/rpl_record.h:
signature changed
sql/share/errmsg.txt:
The user level error code that corresponds to HA_ERR_CORRUPT_EVENT.
The error will be reported in show slave status if such a failure happens.
storage engines do not set the unused null bits (i.e., the
filler bits and the X bit) correctly. Also adding some casts
to debug printouts to eliminate compiler warnings.
sql/ha_ndbcluster.cc:
Changing debug enter names to use full names for member functions.
sql/records.cc:
Emptying records before starting to read records since some engines do
not set the unused null bits, leading to valgrind errors.
sql/rpl_record.cc:
Adding casts to debug printouts to eliminate compiler warnings.
into kindahl-laptop.dnsalias.net:/home/bkroot/mysql-5.1-rpl
sql/field.cc:
Auto merged
sql/field.h:
Auto merged
sql/log_event.cc:
Auto merged
sql/sql_show.cc:
Auto merged
sql/rpl_record.cc:
Manual merge.
Refactoring code to add parameter to pack() and unpack() functions with
purpose of indicating if data should be packed in little-endian or
native order. Using new functions to always pack data for binary log
in little-endian order. The purpose of this refactoring is to allow
proper implementation of endian-agnostic pack() and unpack() functions.
Eliminating several versions of virtual pack() and unpack() functions
in favor for one single virtual function which is overridden in
subclasses.
Implementing pack() and unpack() functions for some field types that
packed data in native format regardless of the value of the
st_table_share::db_low_byte_first flag.
The field types that were packed in native format regardless are:
Field_real, Field_decimal, Field_tiny, Field_short, Field_medium,
Field_long, Field_longlong, and Field_blob.
Before the patch, row-based logging wrote the rows incorrectly on
big-endian machines where the storage engine defined its own
low_byte_first() to be FALSE on big-endian machines (the default
is TRUE), while little-endian machines wrote the fields in correct
order. The only known storage engine that does this is NDB. In effect,
this means that row-based replication from or to a big-endian
machine where the table was using NDB as storage engine failed if the
other engine was either non-NDB or on a little-endian machine.
With this patch, row-based logging is now always done in little-endian
order, while ORDER BY uses the native order if the storage engine
defines low_byte_first() to return FALSE for big-endian machines.
In addition, the max_data_length() function available in Field_blob
was generalized to the entire Field hierarchy to give the maximum
number of bytes that Field::pack() will write.
mysql-test/extra/rpl_tests/rpl_extraMaster_Col.test:
Sorting by columns that produces deterministic order.
mysql-test/suite/rpl/r/rpl_extraColmaster_innodb.result:
Result change.
mysql-test/suite/rpl/r/rpl_extraColmaster_myisam.result:
Result change.
mysql-test/suite/rpl/r/rpl_row_extraColmaster_ndb.result:
Result change.
mysql-test/suite/rpl/t/disabled.def:
Enabling tests.
mysql-test/suite/rpl/t/rpl_row_mysqlbinlog.test:
Adding missing sync_slave_with_master causing slave to keep tables
after shutdown.
mysql-test/suite/rpl_ndb/t/disabled.def:
Enabling tests.
mysql-test/suite/rpl_ndb/t/rpl_ndb_innodb2ndb-slave.opt:
Adding --new option
mysql-test/suite/rpl_ndb/t/rpl_ndb_innodb2ndb.test:
Adding have_log_bin.
mysql-test/suite/rpl_ndb/t/rpl_ndb_myisam2ndb-slave.opt:
Adding --new option
mysql-test/suite/rpl_ndb/t/rpl_ndb_myisam2ndb.test:
Adding have_log_bin
mysql-test/t/partition.test:
Adding have_archive, since that is used in the test.
sql/field.cc:
Eliminating all two-argument pack() and unpack() functions and moving
functionality into the four-argument version. The four argument version
is introduced so that it is possible to avoid using the storage engine
default when writing and reading the packed format (the unpacked format
still uses the storage engine's default). This is used by row-based
replication to write the fields in a storage engine- and endian-agnostic
format.
Packing integral and floating-point numbers in little-endian format
(if requested).
Using pad_char for the field instead of spaces (0x20) when unpacking.
Adding some Doxygen documentation.
---
Adding max_data_length() to denote the maximum number of bytes that
pack() will write.
Adding casts to remove warnings for debug printouts.
sql/field.h:
Eliminating all virtual pack() and unpack() functions except the four-
argument version, which now is the function that shall be overridden.
The two-argument versions are convenience functions, to prevent changes
to code that uses these.
Adding code to pack integer numbers and floating-point numbers in
little-endian format, if requested.
---
Adding max_data_length() to denote the maximum number of bytes that
pack() will write.
sql/log.cc:
Removing debug printout causing crash when starting NDB on Solaris.
sql/log_event.cc:
Adding missing #ifndef causing compile failure. Adding debug printouts.
sql/rpl_record.cc:
Debriding code. Using new pack() and unpack() functions to always pack
fields little-endian. Adding debug printouts.
---
Using max_data_length() when packing field into row.
Adding casts to debug printouts.
sql/sql_show.cc:
Adding code that causes crash on Solaris machines since printf() cannot
handle NULL values for strings properly.
mysql-test/suite/rpl_ndb/r/rpl_ndb_innodb2ndb.result:
New BitKeeper file ``mysql-test/suite/rpl_ndb/r/rpl_ndb_innodb2ndb.result''
mysql-test/suite/rpl_ndb/r/rpl_ndb_myisam2ndb.result:
New BitKeeper file ``mysql-test/suite/rpl_ndb/r/rpl_ndb_myisam2ndb.result''
into sita.local:/Users/tsmith/m/bk/maint/51
sql/field.cc:
Auto merged
sql/log_event_old.cc:
Auto merged
sql/rpl_record.h:
Auto merged
sql/rpl_utility.cc:
Auto merged
sql/rpl_utility.h:
Auto merged
sql/slave.h:
Auto merged
storage/innobase/handler/ha_innodb.cc:
Auto merged
sql/log_event.cc:
Manual merge
sql/log_event.h:
Manual merge
sql/log_event_old.h:
Manual merge
sql/rpl_record.cc:
Manual merge
sql/slave.cc:
Manual merge
into quant.(none):/ext/mysql/bk/mysql-5.1-bug21842-rpl
mysql-test/suite/ndb/r/ndb_dd_basic.result:
Auto merged
sql/field.cc:
Auto merged
sql/handler.cc:
Auto merged
sql/log.cc:
Auto merged
sql/log_event_old.cc:
Auto merged
sql/mysqld.cc:
Auto merged
sql/rpl_record.cc:
Auto merged
sql/rpl_record.h:
Auto merged
sql/rpl_utility.cc:
Auto merged
sql/rpl_utility.h:
Auto merged
sql/sp_head.cc:
Auto merged
sql/sql_class.cc:
Auto merged
sql/sql_class.h:
Auto merged
sql/sql_insert.cc:
Auto merged
sql/sql_select.cc:
Auto merged
sql/sql_string.cc:
Auto merged
sql/sql_table.cc:
Auto merged
sql/unireg.h:
Auto merged
sql/log_event.cc:
Manual merge.
sql/log_event.h:
Manual merge.
sql/log_event_old.h:
Manual merge.
using TPC-B):
Problem: A RBR event can contain incomplete row data (only key value and
fields which have been changed). In that case, when the row is unpacked
into record and written to a table, the missing fields get incorrect NULL
values leading to master-slave inconsistency.
Solution: Use values found in slave's table for columns which are not given
in the rows event. The code for writing a single row uses the following
algorithm:
1. unpack row_data into table->record[0],
2. try to insert record,
3. if duplicate record found, fetch it into table->record[0],
4. unpack row_data into table->record[0],
5. write table->record[0] into the table.
Where row_data is the row as stored in the data area of a rows event.
Thus:
a) unpacking of row_data happens at the time when row is written into
a table,
b) when unpacking (in step 4), only columns present in row_data are
overwritten - all other columns remain as they were found in the table.
Since all data needed for the above algorithm is stored inside
Rows_log_event class, functions which locate and write rows are turned
into methods of that class.
replace_record() -> Rows_log_event::write_row()
find_and_fetch_row() -> Rows_log_event::find_row()
Both methods take row data from event's data buffer - the row being
processed is pointed by m_curr_row. They unpack the data as needed into
table's record buffers record[0] or record[1]. When row is unpacked,
m_curr_row_end is set to point at next row in the data buffer.
Other changes introduced in this changeset:
- Change signature of unpack_row(): don't report errors and don't
setup table's rw_set here. Errors can happen only when setting default
values in prepare_record() function and are detected there.
- In Rows_log_event and derived classes, don't pass arguments to
the execution primitives (do_...() member functions) but use class
members instead.
- Move old row handling code into log_event_old.cc to be used by
*_rows_log_event_old classes.
Also, a new test rpl_ndb_2other is added which tests basic replication
from master using ndb tables to slave storing the same tables using
(possibly) different engine (myisam,innodb).
Test is based on existing tests rpl_ndb_2myisam and rpl_ndb_2innodb.
However, these tests doesn't work for various reasons and currently are
disabled (see BUG#19227).
The new test differs from the ones it is based on as follows:
1. Single test tests replication with different storage engines on slave
(myisam, innodb, ndb).
2. Include file extra/rpl_tests/rpl_ndb_2multi_eng.test containing
original tests is replaced by extra/rpl_tests/rpl_ndb_2multi_basic.test
which doesn't contain tests using partitioned tables as these don't work
currently. Instead, it tests replication to a slave which has more or
less columns than master.
3. Include file include/rpl_multi_engine3.inc is replaced with
include/rpl_multi_engine2.inc. The later differs by performing slightly
different operations (updating more than one row in the table) and
clearing table with "TRUNCATE TABLE" statement instead of "DELETE FROM"
as replication of "DELETE" doesn't work well in this setting.
4. Slave must use option --log-slave-updates=0 as otherwise execution of
replication events generated by ndb fails if table uses a different
storage engine on slave (see BUG#29569).
sql/log_event.cc:
- Initialization of new Rows_log_event members.
- Fixing some typos in documentation.
In Rows_log_event::do_apply_event:
- Set COMPLETE_ROWS_F flag (when master and slave have the same number of
columns and all colums are present in the row)
- Move initialization of tables write/read sets here, outside the rows
processing loop (and out of unpack_row() function).
- Remove calls to do_prepare_row() - no longer needed.
- Add code managing m_curr_row and m_curr_row_end pointers.
- Change signatures of row processing methods of Rows_log_event and it
descendants - now most arguments are taken from class members.
- Remove do_prepare_row() methods which are no longer used.
- The auto_afree_ptr template is moved to rpl_utility.h (so that it can
be used in log_event_old.cc).
- Removed copy_extra_fields() function - no longer used.
In Rows_log_event::write_row (former replace_record):
- The old code is moved to log_event_old.cc.
- Use prepare_record() and non-destructive unpack_current_row() to fill record
with data.
- In case a record being inserted already exists on slave and row data is
incomplete use the record found and non-destructive unpack_current_row() to
combine new column values with existing ones.
- More debug info added.
In Rows_log_event::find_row (former find_and_fetch_row function):
- The old code is moved to log_event_old.cc.
- Unpacking of the row is moved here.
- In case of search using PK, the key data is prepared here.
- More debug info added.
- Remove initialization of Rows_log_event::m_after_image buffer which is no
longer used.
- Use new row unpacking methods in Update_rows_log_event::do_exec_row() to
create before and after image.
Note: all existing code used by Rows_log_event::do_apply_event() has been moved
to log_event_old.cc to be used by *_rows_log_event_old classes.
sql/log_event.h:
- Add new COMPLETE_ROWS_F flag in Rows_log_event.
- Add Rows_log_event members describing the row being processed.
- Add a pointer to key buffer which is used in derived classes.
- Add new methods: find__row(), write_row() and unpack_current_row().
- Change signatures of do_...() methods (replace method arguments by
class members).
- Remove do_prepare_row() method which is no longer used.
- Update method documentation.
- Add Old_rows_log_event class, which contains the old row processing code, as
a friend of Rows_log_event so that it can access all members of an event
instance.
sql/log_event_old.cc:
Move here old implementation of Rows_log_event::do_apply_event() and
helper methods.
sql/log_event_old.h:
- Define new class Old_rows_log_event encapsulating old version of
Rows_log_event::do_apply_event() and the helper methods.
- Add the Old_rows_log_event class as a base for *_old versions of RBR event
classes, ensure that the old version of do_apply_event() is called.
- For *_old classes, declare the helper methods used in the old version of
do_apply_event().
sql/rpl_record.cc:
- Make unpack_row non-destructive for columns not present in the row.
- Don't fill read/write set here as it is done outside these functions.
- Move initialization of a record with default values to a separate
function prepare_record().
sql/rpl_record.h:
- Change signature of unpack_row().
- Declare function prepare_record().
sql/rpl_utility.cc:
Make tabe_def::calc_field_size() a const method.
sql/rpl_utility.h:
Make table_def::calc_field_size() a const method.
Move auto_afree_ptr template here so that it can be re-used (currently
in log_event.cc and log_event_old.cc). Similar with DBUG_PRINT_BITSET
macro.
mysql-test/extra/rpl_tests/rpl_ndb_2multi_basic.test:
Modification of rpl_ndb_2multi_eng test. Tests with partitioned tables
are removed and a setup with slave having different number of columns
than master is added.
mysql-test/include/rpl_multi_engine2.inc:
Modification of rpl_multi_engine3.inc which operates on more rows and
replaces "DELETE FROM t1" with "TRUNCATE TABLE t1" as the first form
doesn't replicate in NDB -> non-NDB setting (BUG#28538).
mysql-test/suite/rpl_ndb/r/rpl_ndb_2other.result:
Results of the test.
mysql-test/suite/rpl_ndb/t/rpl_ndb_2other-slave.opt:
Test options. --log-slave-updates=0 is compulsory as otherwise non-NDB
slave applying row events from NDB master will fail when trying to log
them.
mysql-test/suite/rpl_ndb/t/rpl_ndb_2other.test:
Test replication of NDB table to slave using other engine. The main test
is in extra/rpl_tests/rpl_ndb_2multi_basic.test. It is included here
several times with different settings of default storage engine on slave.
into weblab.(none):/home/marcsql/TREE/mysql-5.1-rt-merge
sql/field.cc:
Auto merged
sql/field.h:
Auto merged
sql/log_event.h:
Auto merged
sql/rpl_record.cc:
Auto merged
sql/slave.h:
Auto merged
sql/sql_base.cc:
Auto merged
sql/sql_class.cc:
Auto merged
sql/sql_handler.cc:
Auto merged
sql/sql_parse.cc:
Auto merged
sql/sql_select.cc:
Auto merged
(and be more friendly to Doxygen by removing unnecessary typedefs).
sql/log.cc:
Renaming struct st_relay_log_info to class Relay_log_info.
sql/log.h:
Renaming struct st_relay_log_info to class Relay_log_info.
sql/log_event.cc:
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/log_event.h:
Renaming struct st_relay_log_info to class Relay_log_info.
Renaming RELAY_LOG_INFO to Relay_log_info.
Removing typedef RELAY_LOG_INFO.
sql/log_event_old.cc:
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/log_event_old.h:
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/rpl_mi.h:
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/rpl_record.cc:
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/rpl_record.h:
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/rpl_record_old.cc:
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/rpl_record_old.h:
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/rpl_rli.cc:
Renaming struct st_relay_log_info to class Relay_log_info.
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/rpl_rli.h:
Renaming struct st_relay_log_info to class Relay_log_info.
Renaming RELAY_LOG_INFO to Relay_log_info.
Removing typedef RELAY_LOG_INFO.
sql/rpl_utility.cc:
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/rpl_utility.h:
Renaming struct st_relay_log_info to class Relay_log_info.
Renaming RELAY_LOG_INFO to Relay_log_info.
Removing typedef RELAY_LOG_INFO.
sql/slave.cc:
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/slave.h:
Renaming struct st_relay_log_info to class Relay_log_info.
Renaming RELAY_LOG_INFO to Relay_log_info.
Removing typedef RELAY_LOG_INFO.
sql/sql_binlog.cc:
Renaming RELAY_LOG_INFO to Relay_log_info.
sql/sql_class.h:
Renaming struct st_relay_log_info to class Relay_log_info.
Renaming RELAY_LOG_INFO to Relay_log_info.
Removing typedef RELAY_LOG_INFO.
sql/sql_repl.cc:
Renaming RELAY_LOG_INFO to Relay_log_info.
