2007-09-20 16:31:05 +02:00
|
|
|
stop slave;
|
|
|
|
|
drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t9;
|
|
|
|
|
reset master;
|
|
|
|
|
reset slave;
|
|
|
|
|
drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t9;
|
|
|
|
|
start slave;
|
WL#5151: Conversion between different types when replicating
Row-based replication requires the types of columns on the
master and slave to be approximately the same (some safe
conversions between strings are allowed), but does not
allow safe conversions between fields of similar types such
as TINYINT and INT.
This patch implement type conversions between similar fields
on the master and slave.
The conversions are controlled using a new variable
SLAVE_TYPE_CONVERSIONS of type SET('ALL_LOSSY','ALL_NON_LOSSY').
Non-lossy conversions are any conversions that do not run the
risk of losing any information, while lossy conversions can
potentially truncate the value. The column definitions are
checked to decide if the conversion is acceptable.
If neither conversion is enabled, it is required that the
definitions of the columns are identical on master and slave.
Conversion is done by creating an internal conversion table,
unpacking the master data into it, and then copy the data to
the real table on the slave.
.bzrignore:
New files added
client/Makefile.am:
New files added
client/mysqlbinlog.cc:
Functions in rpl_utility.cc is now needed by mysqlbinlog.cc.
libmysqld/Makefile.am:
New files added
mysql-test/extra/rpl_tests/check_type.inc:
Test include file to check a single type conversion.
mysql-test/extra/rpl_tests/rpl_extraSlave_Col.test:
Switching to use INT instead of TEXT for column that should not have matching types.
mysql-test/extra/rpl_tests/rpl_row_basic.test:
Adding code to enable type conversions for BIT tests since InnoDB
cannot handle them properly due to incorrect information stored as
metadata.
mysql-test/extra/rpl_tests/type_conversions.test:
Test file to check a set of type conversions
with current settings of slave_type_conversions.
mysql-test/suite/rpl/t/rpl_typeconv.test:
Test file to test conversions from master to slave with
all possible values for slave_type_conversions.
The test also checks that the slave_type_conversions
variable works as expected.
sql/field.cc:
Changing definition of compatible_field_size to both check if
two field with identical base types are compatible and give an
order between them if they are compatible.
This only implement checking on the slave, so it will not affect
replication from an old master to a new slave.
sql/field.h:
Changing prototypes for functions:
- compatible_field_size()
- init_for_tmp_table()
- row_pack_length()
sql/log_event.cc:
Changing compability checks to build a conversion table if the fields
are compatible, but does not have the same base type.
sql/log_event_old.cc:
Changing compability checks to build a conversion table if the fields
are compatible, but does not have the same base type.
sql/mysql_priv.h:
Adding global option variable for SLAVE_TYPE_CONVERSIONS
sql/mysqld.cc:
Adding SLAVE_TYPE_CONVERSIONS global server variable.
sql/rpl_record.cc:
Changing unpack_row to use the conversion table if present.
sql/rpl_rli.h:
Removing function get_tabledef and replacing it with get_table_data().
This function retrieve data for table opened for replication, not just
table definition.
sql/rpl_utility.cc:
Function table_def::compatible_with is changed to compare table on master
and slave for compatibility and generate a conversions table if they are
compatible.
Computing real type of fields from metadata for ENUM and SET types.
Computing pack_length correctly for ENUM, SET, and BLOB types.
Adding optimization to not check compatibility if no
slave type conversions are enabled.
sql/rpl_utility.h:
Changing prototypes since implementation has changed.
Modifying table_def::type() to return real type instead of stored type.
sql/set_var.cc:
Adding SLAVE_TYPE_CONVERSIONS variable.
sql/set_var.h:
Adding SLAVE_TYPE_CONVERSIONS variable.
sql/share/errmsg.txt:
Adding error messages for slave type conversions.
sql/sql_class.h:
Adding SLAVE_TYPE_CONVERSIONS variable.
sql/sql_select.cc:
Correcting create_virtual_tmp_table() to compute null bit positions
correctly in the presence of bit fields.
2009-12-14 12:04:55 +01:00
|
|
|
SET @saved_slave_type_conversions = @@slave_type_conversions;
|
|
|
|
|
SET GLOBAL SLAVE_TYPE_CONVERSIONS = 'ALL_NON_LOSSY';
|
2007-09-20 16:31:05 +02:00
|
|
|
CREATE DATABASE track;
|
|
|
|
|
USE track;
|
|
|
|
|
CREATE TABLE `visits` (
|
|
|
|
|
`visits_id` int(11) unsigned NOT NULL AUTO_INCREMENT,
|
|
|
|
|
`myid` varchar(32) NOT NULL DEFAULT '',
|
|
|
|
|
`src` varchar(64) NOT NULL DEFAULT '',
|
|
|
|
|
`ip` int(10) unsigned NOT NULL DEFAULT '0',
|
|
|
|
|
`cc` char(2) NOT NULL DEFAULT '',
|
|
|
|
|
`org` varchar(80) DEFAULT NULL,
|
|
|
|
|
`ref` varchar(255) NOT NULL DEFAULT '',
|
|
|
|
|
`time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
|
|
|
|
`host` varchar(30) NOT NULL DEFAULT '',
|
|
|
|
|
`entry` varchar(255) NOT NULL DEFAULT '',
|
|
|
|
|
`visit_exit` varchar(255) NOT NULL DEFAULT '',
|
|
|
|
|
`user_id` int(11) unsigned NOT NULL DEFAULT '0',
|
|
|
|
|
`visit_start` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
|
|
|
|
|
PRIMARY KEY (`visits_id`),
|
|
|
|
|
KEY `ip` (`ip`),
|
|
|
|
|
KEY `time` (`time`),
|
|
|
|
|
KEY `user_id` (`user_id`)
|
|
|
|
|
) ENGINE=MyISAM AUTO_INCREMENT=21293381 DEFAULT CHARSET=latin1;
|
|
|
|
|
CREATE TABLE `visits_events` (
|
|
|
|
|
`event_id` mediumint(8) unsigned NOT NULL DEFAULT '0',
|
|
|
|
|
`visit_id` int(11) unsigned NOT NULL DEFAULT '0',
|
|
|
|
|
`timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
|
|
|
|
`src` varchar(64) NOT NULL DEFAULT '',
|
|
|
|
|
`data` varchar(255) NOT NULL DEFAULT '',
|
|
|
|
|
`visits_events_id` int(11) unsigned NOT NULL AUTO_INCREMENT,
|
|
|
|
|
PRIMARY KEY (`visits_events_id`),
|
|
|
|
|
KEY `event_id` (`event_id`),
|
|
|
|
|
KEY `visit_id` (`visit_id`),
|
|
|
|
|
KEY `data` (`data`)
|
|
|
|
|
) ENGINE=MyISAM AUTO_INCREMENT=33900731 DEFAULT CHARSET=latin1;
|
|
|
|
|
/*!40019 SET @@session.max_insert_delayed_threads=0*/;
|
|
|
|
|
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
|
|
|
|
|
BINLOG '
|
BUG#32407: Impossible to do point-in-time recovery from older binlog
Problem: it is unsafe to read base64-printed events without first
reading the Format_description_log_event (FD). Currently, mysqlbinlog
cannot print the FD.
As a side effect, another bug has also been fixed: When mysqlbinlog
--start-position=X was specified, no ROLLBACK was printed. I changed
this, so that ROLLBACK is always printed.
This patch does several things:
- Format_description_log_event (FD) now print themselves in base64
format.
- mysqlbinlog is now able to print FD events. It has three modes:
--base64-output=auto Print row events in base64 output, and print
FD event. The FD event is printed even if
it is outside the range specified with
--start-position, because it would not be
safe to read row events otherwise. This is
the default.
--base64-output=always Like --base64-output=auto, but also print
base64 output for query events. This is
like the old --base64-output flag, which
is also a shorthand for
--base64-output=always
--base64-output=never Never print base64 output, generate error if
row events occur in binlog. This is
useful to suppress the FD event in binlogs
known not to contain row events (e.g.,
because BINLOG statement is unsafe,
requires root privileges, is not SQL, etc)
- the BINLOG statement now handles FD events correctly, by setting
the thread's rli's relay log's description_event_for_exec to the
loaded event.
In fact, executing a BINLOG statement is almost the same as reading
an event from a relay log. Before my patch, the code for this was
separated (exec_relay_log_event in slave.cc executes events from
the relay log, mysql_client_binlog_statement in sql_binlog.cc
executes BINLOG statements). I needed to augment
mysql_client_binlog_statement to do parts of what
exec_relay_log_event does. Hence, I did a small refactoring and
moved parts of exec_relay_log_event to a new function, which I
named apply_event_and_update_pos. apply_event_and_update_pos is
called both from exec_relay_log_event and from
mysql_client_binlog_statement.
- When a non-FD event is executed in a BINLOG statement, without
previously executing a FD event in a BINLOG statement, it generates
an error, because that's unsafe. I took a new error code for that:
ER_NO_FORMAT_DESCRIPTION_EVENT_BEFORE_BINLOG_STATEMENTS.
In order to get a decent error message containing the name of the
event, I added the class method char*
Log_event::get_type_str(Log_event_type type), which returns a
string name for the given Log_event_type. This is just like the
existing char* Log_event::get_type_str(), except it is a class
method that takes the log event type as parameter.
I also added PRE_GA_*_ROWS_LOG_EVENT to Log_event::get_type_str(),
so that names of old rows event are properly printed.
- When reading an event, I added a check that the event type is known
by the current Format_description_log_event. Without this, it may
crash on bad input (and I was struck by this several times).
- I patched the following test cases, which all contain BINLOG
statements for row events which must be preceded by BINLOG
statements for FD events:
- rpl_bug31076
While I was here, I fixed some small things in log_event.cc:
- replaced hard-coded 4 by EVENT_TYPE_OFFSET in 3 places
- replaced return by DBUG_VOID_RETURN in one place
- The name of the logfile can be '-' to indicate stdin. Before my
patch, the code just checked if the first character is '-'; now it
does a full strcmp(). Probably, all arguments that begin with a -
are already handled somewhere else as flags, but I still think it
is better that the code reflects what it is supposed to do, with as
little dependencies as possible on other parts of the code. If we
one day implement that all command line arguments after -- are
files (as most unix tools do), then we need this.
I also fixed the following in slave.cc:
- next_event() was declared twice, and queue_event was not static but
should be static (not used outside the file).
client/client_priv.h:
Declared the new option for base64 output.
client/mysqlbinlog.cc:
- Change from using the two-state command line option
"default/--base64-output" to the three-state
"--base64-output=[never|auto|always]"
- Print the FD event even if it is outside the --start-position range.
- Stop if a row event is about to be printed without a preceding FD
event.
- Minor fixes:
* changed 4 to EVENT_TYPE_OFFSET in some places
* Added comments
* before, "mysqlbinlog -xyz" read from stdin; now it does not
(only "mysqlbinlog -" reads stdin).
mysql-test/r/mysqlbinlog2.result:
Updated result file: mysqlbinlog now prints ROLLBACK always.
mysql-test/suite/binlog/t/disabled.def:
The test must be disabled since it reveals another bug: see BUG#33247.
mysql-test/suite/rpl/r/rpl_bug31076.result:
Updated result file
mysql-test/suite/rpl/r/rpl_row_mysqlbinlog.result:
Updated result file
mysql-test/suite/rpl/t/rpl_bug31076.test:
Had to add explicit Format_description_log_event before other BINLOG
statements
mysql-test/t/mysqlbinlog2.test:
we must suppress base64 output in result file because it contains a
timestamp
sql/log_event.cc:
- Made FD events able to print themselves
- Added check that the current FD event knows about the event type, when
an event is about to be read. (Hint to reviewers: I had to re-indent
a big block because of this; use diff -b)
* To get a decent error message, I also added a class method
const char* Log_event::get_type_str(Log_event_type)
which converts number to event type string without having a
Log_event object.
* Made Log_event::get_type_str aware of PRE_GA_*_ROWS_LOG_EVENT.
- Minor fixes:
* Changed return to DBUG_VOID_RETURN
sql/log_event.h:
- Declared enum to describe the three base64_output modes
- Use the enum instead of a flag
- Declare the new class method get_type_str (see log_event.cc)
sql/share/errmsg.txt:
Added error msg.
sql/slave.cc:
- Factored out part of exec_relay_log_event to the new function
apply_event_and_update_pos, because that code is needed when executing
BINLOG statements. (this is be functionally equivalent to the
previous code, except: (1) skipping events is now optional, controlled
by a parameter to the new function (2) the return value of
exec_relay_log_event has changed; see next item).
- Changed returned error value to always be 1. Before, it would return
the error value from apply_log_event, which was unnecessary. This
change is safe because the exact return value of exec_relay_log_event
is never examined; it is only tested to be ==0 or !=0.
- Added comments describing exec_relay_log_event and
apply_event_and_update_pos.
- Minor fixes:
* Removed duplicate declaration of next_event, made queue_event
static.
* Added doxygen code to include this file.
sql/slave.h:
Declared the new apply_event_and_update_pos
sql/sql_binlog.cc:
- Made mysql_binlog_statement set the current FD event when the given
event is an FD event. This entails using the new function
apply_event_and_update_pos from slave.cc instead of just calling the
ev->apply method.
- Made mysql_binlog_statement fail if the first BINLOG statement is not
an FD event.
mysql-test/suite/binlog/r/binlog_base64_flag.result:
New test file needs new result file
mysql-test/suite/binlog/t/binlog_base64_flag.test:
Added test case to verify that:
- my patch fixes the bug
- the new --base64-output flag works as expected
- base64 events not preceded by an FD event give an error
- an event of a type not known by the current FD event fails cleanly.
mysql-test/suite/binlog/std_data/binlog-bug32407.000001:
BitKeeper file /home/sven/bk/b32407-5.1-new-rpl-mysqlbinlog_base64/mysql-test/suite/binlog/std_data/binlog-bug32407.000001
2007-12-14 19:02:02 +01:00
|
|
|
O1ZVRw8BAAAAZgAAAGoAAAAAAAQANS4xLjIzLXJjLWRlYnVnLWxvZwAAAAAAAAAAAAAAAAAAAAAA
|
|
|
|
|
AAAAAAAAAAAAAAAAAAA7VlVHEzgNAAgAEgAEBAQEEgAAUwAEGggAAAAICAgC
|
|
|
|
|
'/*!*/;
|
|
|
|
|
BINLOG '
|
2007-09-20 16:31:05 +02:00
|
|
|
Bk3vRhO0AQAAOAAAALcLyQkAAJlXFwIAAAAABXRyYWNrAA12aXNpdHNfZXZlbnRzAAYJAwcPDwM=
|
|
|
|
|
Bk3vRhe0AQAAWgAAABEMyQkQAJlXFwIAAAEABv/AIE4AvvVDAQZN70YAK0Rvd25sb2Fkcy9NeVNR
|
|
|
|
|
TC00LjEvbXlzcWwtNC4xLjEyYS13aW4zMi56aXBPaAIC
|
|
|
|
|
'/*!*/;
|
|
|
|
|
SET INSERT_ID=21231039/*!*/;
|
|
|
|
|
use track/*!*/;
|
|
|
|
|
SET TIMESTAMP=1190087942/*!*/;
|
|
|
|
|
SET @@session.foreign_key_checks=1, @@session.sql_auto_is_null=1, @@session.unique_checks=1/*!*/;
|
|
|
|
|
SET @@session.sql_mode=0/*!*/;
|
|
|
|
|
SET @@session.character_set_client=8,@@session.collation_connection=8,@@session.collation_server=8/*!*/;
|
|
|
|
|
SET @@session.time_zone='UTC'/*!*/;
|
|
|
|
|
INSERT INTO visits (myid, user_id, src, ip, cc, org, ref, time, host, entry, visit_exit, visit_start)
|
|
|
|
|
VALUES ('3m3l4rhs6do0sf5p1i9lr94g928a272v', '', '', INET_ATON('71.118.124.98'), '', '', 'http://dev.mysql.com/downloads/connector/j/3.0.html', NULL, 'dev.mysql.com', '/get/Downloads/Connector-J/mysql-connector-java-3.0.17-ga.zip/from/pick', '/get/Downloads/Connector-J/mysql-connector-java-3.0.17-ga.zip/from/pick', NOW())/*!*/;
|
|
|
|
|
Warnings:
|
|
|
|
|
Warning 1366 Incorrect integer value: '' for column 'user_id' at row 1
|
|
|
|
|
SELECT * FROM visits;
|
|
|
|
|
visits_id myid src ip cc org ref time host entry visit_exit user_id visit_start
|
|
|
|
|
21231039 3m3l4rhs6do0sf5p1i9lr94g928a272v 1198947426 http://dev.mysql.com/downloads/connector/j/3.0.html 2007-09-18 03:59:02 dev.mysql.com /get/Downloads/Connector-J/mysql-connector-java-3.0.17-ga.zip/from/pick /get/Downloads/Connector-J/mysql-connector-java-3.0.17-ga.zip/from/pick 0 2007-09-18 03:59:02
|
|
|
|
|
SELECT * FROM visits_events;
|
|
|
|
|
event_id visit_id timestamp src data visits_events_id
|
|
|
|
|
20000 21231038 2007-09-18 03:59:02 Downloads/MySQL-4.1/mysql-4.1.12a-win32.zip 33712207
|
WL#5151: Conversion between different types when replicating
Row-based replication requires the types of columns on the
master and slave to be approximately the same (some safe
conversions between strings are allowed), but does not
allow safe conversions between fields of similar types such
as TINYINT and INT.
This patch implement type conversions between similar fields
on the master and slave.
The conversions are controlled using a new variable
SLAVE_TYPE_CONVERSIONS of type SET('ALL_LOSSY','ALL_NON_LOSSY').
Non-lossy conversions are any conversions that do not run the
risk of losing any information, while lossy conversions can
potentially truncate the value. The column definitions are
checked to decide if the conversion is acceptable.
If neither conversion is enabled, it is required that the
definitions of the columns are identical on master and slave.
Conversion is done by creating an internal conversion table,
unpacking the master data into it, and then copy the data to
the real table on the slave.
.bzrignore:
New files added
client/Makefile.am:
New files added
client/mysqlbinlog.cc:
Functions in rpl_utility.cc is now needed by mysqlbinlog.cc.
libmysqld/Makefile.am:
New files added
mysql-test/extra/rpl_tests/check_type.inc:
Test include file to check a single type conversion.
mysql-test/extra/rpl_tests/rpl_extraSlave_Col.test:
Switching to use INT instead of TEXT for column that should not have matching types.
mysql-test/extra/rpl_tests/rpl_row_basic.test:
Adding code to enable type conversions for BIT tests since InnoDB
cannot handle them properly due to incorrect information stored as
metadata.
mysql-test/extra/rpl_tests/type_conversions.test:
Test file to check a set of type conversions
with current settings of slave_type_conversions.
mysql-test/suite/rpl/t/rpl_typeconv.test:
Test file to test conversions from master to slave with
all possible values for slave_type_conversions.
The test also checks that the slave_type_conversions
variable works as expected.
sql/field.cc:
Changing definition of compatible_field_size to both check if
two field with identical base types are compatible and give an
order between them if they are compatible.
This only implement checking on the slave, so it will not affect
replication from an old master to a new slave.
sql/field.h:
Changing prototypes for functions:
- compatible_field_size()
- init_for_tmp_table()
- row_pack_length()
sql/log_event.cc:
Changing compability checks to build a conversion table if the fields
are compatible, but does not have the same base type.
sql/log_event_old.cc:
Changing compability checks to build a conversion table if the fields
are compatible, but does not have the same base type.
sql/mysql_priv.h:
Adding global option variable for SLAVE_TYPE_CONVERSIONS
sql/mysqld.cc:
Adding SLAVE_TYPE_CONVERSIONS global server variable.
sql/rpl_record.cc:
Changing unpack_row to use the conversion table if present.
sql/rpl_rli.h:
Removing function get_tabledef and replacing it with get_table_data().
This function retrieve data for table opened for replication, not just
table definition.
sql/rpl_utility.cc:
Function table_def::compatible_with is changed to compare table on master
and slave for compatibility and generate a conversions table if they are
compatible.
Computing real type of fields from metadata for ENUM and SET types.
Computing pack_length correctly for ENUM, SET, and BLOB types.
Adding optimization to not check compatibility if no
slave type conversions are enabled.
sql/rpl_utility.h:
Changing prototypes since implementation has changed.
Modifying table_def::type() to return real type instead of stored type.
sql/set_var.cc:
Adding SLAVE_TYPE_CONVERSIONS variable.
sql/set_var.h:
Adding SLAVE_TYPE_CONVERSIONS variable.
sql/share/errmsg.txt:
Adding error messages for slave type conversions.
sql/sql_class.h:
Adding SLAVE_TYPE_CONVERSIONS variable.
sql/sql_select.cc:
Correcting create_virtual_tmp_table() to compute null bit positions
correctly in the presence of bit fields.
2009-12-14 12:04:55 +01:00
|
|
|
SET GLOBAL SLAVE_TYPE_CONVERSIONS = @saved_slave_type_conversions;
|
2007-10-01 12:10:46 +02:00
|
|
|
DROP DATABASE track;
|
|
|
|
|
End of 5.1 tests
|