Problem:
========
1) Drop table queries are re-generated by server
before writing the events(queries) into binlog
for various reasons. If table name/db name contains
a non regular characters (like latin characters),
the generated query is wrong. Hence it breaks the
replication.
2) In the edge case, when table name/db name contains
64 characters, server is throwing an assert
assert(M_TBLLEN < 128)
3) In the edge case, when db name contains 64 latin
characters, binlog content is interpreted badly
which is leading replication failure.
Analysis & Fix :
================
1) Parser reads the table name from the query and converts
it to standard charset(utf8) and stores it in table_name variable.
When drop table query is regenerated with the same table_name
variable, it should be converted back to the original charset
from standard charset(utf8).
2) Latin character takes two bytes for each character. Limit
of the identifier is 64. SYSTEM_CHARSET_MBMAXLEN is set to '3'.
So there is a possiblity that tablename/dbname contains 3 * 64.
Hence assert is changed to
(M_TBLLEN <= NAME_CHAR_LEN*SYSTEM_CHARSET_MBMAXLEN)
3) db_len in the binlog event header is taking 1 byte.
db_len is ranged from 0 to 192 bytes (3 * 64).
While reading the db_len from the event, server
is casting to uint instead of uchar which is leading
to bad db_len. This problem is fixed by changing the
cast type to uchar.
Description: On an example MySQL instance with 28k empty
InnoDB tables, a specific query to information_schema.tables
and information_schema.columns leads to memory consumption
over 38GB RSS.
Analysis: In get_all_tables() call, we fill the I_S tables
from frm files and storage engine. As part of that process
we call make_table_name_list() and allocate memory for all
the 28k frm file names in the THD mem_root through
make_lex_string_root(). Since it has been called around
28k * 28k times there is a huge memory getting hogged in
THD mem_root. This causes the RSS to grow to 38GB.
Fix: As part of fix we are creating a temporary mem_root
in get_all_tables and passing it to fill_fiels(). There we
replace the THD mem_root with the temporary mem_root and
allocates the file names in temporary mem_root and frees
it once we fill the I_S tables in get_all_tables and
re-assign the original mem_root back to THD mem_root.
Note: Checked the massif out put with the fix now the memory growth is just around 580MB at peak.
Problem: Uninstallation of semi sync plugin causes replication to
break.
Analysis: A semisync enabled replication is mutual agreement between
Master and Slave when the connection (I/O thread) is established.
Once I/O thread is started and if semisync is enabled on both
master and slave, master appends special magic header to events
using semisync plugin functions and sends it to slave. And slave
expects that each event will have that special magic header format
and reads those bytes using semisync plugin functions.
When semi sync replication is in use if users execute
uninstallation of the plugin on master, slave gets confused while
interpreting that event's content because it expects special
magic header at the beginning of the event. Slave SQL thread will
be stopped with "Missing magic number in the header" error.
Similar problem will happen if uninstallation of the plugin happens
on slave when semi sync replication is in in use. Master sends
the events with magic header and slave does not know about the
added magic header and thinks that it received a corrupted event.
Hence slave SQL thread stops with "Found corrupted event" error.
Fix: Uninstallation of semisync plugin will be blocked when semisync
replication is in use and will throw 'ER_UNKNOWN_ERROR' error.
To detect that semisync replication is in use, this patch uses
semisync status variable values.
> On Master, it checks for 'Rpl_semi_sync_master_status' to be OFF
before allowing the uninstallation of rpl_semi_sync_master plugin.
>> Rpl_semi_sync_master_status is OFF when
>>> there is no dump thread running
>>> there are no semisync slaves
> On Slave, it checks for 'Rpl_semi_sync_slave_status' to be OFF
before allowing the uninstallation of rpl_semi_sync_slave plugin.
>> Rpl_semi_sync_slave_status is OFF when
>>> there is no I/O thread running
>>> replication is asynchronous replication.
The feature was backported from MySQL 5.6.
Some code was added to make commands as
SELECT * FROM ignored_db.t1;
CALL ignored_db.proc();
USE ignored_db;
to take that option into account.
per-file comments:
mysql-test/r/ignore_db_dirs_basic.result
test result added.
mysql-test/t/ignore_db_dirs_basic-master.opt
options for the test,
actually the set of --ignore-db-dir lines.
mysql-test/t/ignore_db_dirs_basic.test
test for the feature.
Same test from 5.6 was taken as a basis,
then tests for SELECT, CALL etc were added.
per-file comments:
sql/mysql_priv.h
MDEV-495 backport --ignore-db-dir.
interface for db_name_is_in_ignore_list() added.
sql/mysqld.cc
MDEV-495 backport --ignore-db-dir.
--ignore-db-dir handling.
sql/set_var.cc
MDEV-495 backport --ignore-db-dir.
the @@ignore_db_dirs variable added.
sql/sql_show.cc
MDEV-495 backport --ignore-db-dir.
check if the directory is ignored.
sql/sql_show.h
MDEV-495 backport --ignore-db-dir.
interface added for opt_ignored_db_dirs.
sql/table.cc
MDEV-495 backport --ignore-db-dir.
check if the directory is ignored.
QUOTING IN REPLICATION
Problem: Misquoting or unquoted identifiers may lead to
incorrect statements to be logged to the binary log.
Fix: we use specialized functions to append quoted identifiers in
the statements generated by the server.
This patch:
- Moves all definitions from the mysql_priv.h file into
header files for the component where the variable is
defined
- Creates header files if the component lacks one
- Eliminates all include directives from mysql_priv.h
- Eliminates all circular include cycles
- Rename time.cc to sql_time.cc
- Rename mysql_priv.h to sql_priv.h
This patch:
- Moves all definitions from the mysql_priv.h file into
header files for the component where the variable is
defined
- Creates header files if the component lacks one
- Eliminates all include directives from mysql_priv.h
- Eliminates all circular include cycles
- Rename time.cc to sql_time.cc
- Rename mysql_priv.h to sql_priv.h
The failure was caused by executing a CREATE-SELECT statement that creates a
table in another database than the current one. In row-based logging, the
CREATE statement was written to the binary log without the database, hence
creating the table in the wrong database, causing the following inserts to
fail since the table didn't exist in the given database.
Fixed the bug by adding a parameter to store_create_info() that will make
the function print the database name before the table name and used that
in the calls that write the CREATE statement to the binary log. The database
name is only printed if it is different than the currently selected database.
The output of SHOW CREATE TABLE has not changed and is still printed without
the database name.
mysql-test/suite/rpl/t/rpl_row_create_table.test:
Added test to check that CREATE-SELECT into another database than the
current one replicates.
sql/sql_insert.cc:
Adding parameter to calls to store_create_info().
sql/sql_show.cc:
Adding parameter to calls to store_create_info().
Extending store_create_info() with parameter 'show_database' that will cause
the database to be written before the table name.
sql/sql_show.h:
Adding parameter to call to store_create_info() to tell if the database should be shown or not.
sql/sql_table.cc:
Adding parameter to calls to store_create_info().
The failure was caused by executing a CREATE-SELECT statement that creates a
table in another database than the current one. In row-based logging, the
CREATE statement was written to the binary log without the database, hence
creating the table in the wrong database, causing the following inserts to
fail since the table didn't exist in the given database.
Fixed the bug by adding a parameter to store_create_info() that will make
the function print the database name before the table name and used that
in the calls that write the CREATE statement to the binary log. The database
name is only printed if it is different than the currently selected database.
The output of SHOW CREATE TABLE has not changed and is still printed without
the database name.
client/mysqldump.c:
table type compare is changed to case insensitive
mysql-test/r/information_schema.result:
test result
mysql-test/r/information_schema_db.result:
result fix
mysql-test/suite/ndb/r/ndb_alter_table.result:
result fix
mysql-test/suite/ndb/r/ndb_temporary.result:
result fix
mysql-test/t/information_schema.test:
test case
sql/ha_ndbcluster.cc:
char* variables are changed to LEX_STRING
sql/ha_ndbcluster.h:
char* variables are changed to LEX_STRING
sql/ha_ndbcluster_binlog.cc:
char* variables are changed to LEX_STRING
sql/handler.cc:
char* variables are changed to LEX_STRING
sql/handler.h:
char* variables are changed to LEX_STRING
sql/sql_base.cc:
Modified functions which are used during open table process
according to table opening method and requested_object.
sql/sql_select.cc:
Add support for I_S tables into select_describe() function
sql/sql_show.cc:
1. Added initialization of 'open_method' to 'st_field_info' structs.
2. Added initialization of 'i_s_requested_object' to 'ST_SCHEMA_TABLE' structs.
3. New function which calculates database name and table name values
from 'where' condition if it's possible
void get_lookup_field_values(THD *thd, COND *cond, TABLE_LIST *table,
LOOKUP_FIELD_VALUES *lookup_field_vals);
4. New function which set table open method
setup_table_open_method(TABLE_LIST *tables,
ST_SCHEMA_TABLE *schema_table,
enum enum_schema_tables schema_table_idx)
5. New function
int make_db_list(THD *thd, List<LEX_STRING> *files,
LOOKUP_FIELD_VALUES *lookup_field_vals,
bool *with_i_schema)
6. New function
int make_table_name_list(THD *thd, List<LEX_STRING> *files, LEX *lex,
LOOKUP_FIELD_VALUES *lookup_field_vals,
bool with_i_schema, LEX_STRING *db_name)
7. Modified 'get_all_tables' function according to new schema(see wl#3732).
sql/sql_show.h:
char* variables are changed to LEX_STRING
sql/table.cc:
Modified functions which are used during open table process
according to table opening method and requested_object.
sql/table.h:
1. added new constants(open_method)
#define SKIP_OPEN_TABLE 0
#define OPEN_FRM_ONLY 1
#define OPEN_FULL_TABLE 2
2. Added new field 'open_method' into struct st_field_info;
uint open_method;
3. Added new field into ST_SCHEMA_TABLE struct
uint i_s_requested_object; /* the object we need to open(TABLE | VIEW) */.
4. Added new field to TABLE_LIST struct.
uint i_s_requested_object;
This field is set from ST_SCHEMA_TABLE.i_s_requested_object
for processed table before opening.
5. Added new fields to TABLE_LIST struct, used for 'explain select' for I_S table
bool has_db_lookup_value;
bool has_table_lookup_value;
uint table_open_method;
sql/unireg.h:
added new constants
mysql-test/r/innodb_mysql.result:
Update test results (merge from the main tree).
mysql-test/r/query_cache.result:
Update test results (merge from the main tree).
mysql-test/r/sp.result:
Update test results (merge from the main tree).
mysql-test/t/query_cache.test:
Use --echo End of to simplify future merges.
sql/handler.h:
st_table_list -> TABLE_LIST
sql/item_create.cc:
A post-merge fix (this code is in sql_yacc.yy in 5.0)
sql/rpl_utility.h:
st_table_list -> TABLE_LIST
sql/sp.cc:
A post-merge fix.
sql/sp_head.cc:
In 5.1 memdup_root returns void*.
sql/sql_show.cc:
st_table_list -> TABLE_LIST
sql/sql_show.h:
st_table_list -> TABLE_LIST
sql/sql_yacc.yy:
A post-merge fix.
sql/table.cc:
st_table_list -> TABLE_LIST
sql/table.h:
st_table_list -> TABLE_LIST
Removed "MySQL Finland AB & TCX DataKonsult AB" from copyright header
Adjusted year(s) in copyright header
Added GPL copyright text
my_vle.h, rpl_utility.h, my_vle.c, base64-t.c, rpl_utility.cc:
Changed copyright header formatting some
plugin_example.c, daemon_example.c:
Added "Copyright (C) 2006 MySQL AB" to GPL header
CMakeLists.txt:
Added GPL copyright text
client/CMakeLists.txt:
Added GPL copyright text
dbug/CMakeLists.txt:
Added GPL copyright text
extra/CMakeLists.txt:
Added GPL copyright text
extra/yassl/CMakeLists.txt:
Added GPL copyright text
extra/yassl/taocrypt/CMakeLists.txt:
Added GPL copyright text
libmysql/CMakeLists.txt:
Added GPL copyright text
libmysqld/CMakeLists.txt:
Added GPL copyright text
libmysqld/examples/CMakeLists.txt:
Added GPL copyright text
mysys/CMakeLists.txt:
Added GPL copyright text
regex/CMakeLists.txt:
Added GPL copyright text
server-tools/instance-manager/CMakeLists.txt:
Added GPL copyright text
sql/CMakeLists.txt:
Added GPL copyright text
storage/archive/CMakeLists.txt:
Added GPL copyright text
storage/blackhole/CMakeLists.txt:
Added GPL copyright text
storage/csv/CMakeLists.txt:
Added GPL copyright text
storage/example/CMakeLists.txt:
Added GPL copyright text
storage/federated/CMakeLists.txt:
Added GPL copyright text
storage/heap/CMakeLists.txt:
Added GPL copyright text
storage/innobase/CMakeLists.txt:
Added GPL copyright text
storage/myisam/CMakeLists.txt:
Added GPL copyright text
storage/myisammrg/CMakeLists.txt:
Added GPL copyright text
strings/CMakeLists.txt:
Added GPL copyright text
tests/CMakeLists.txt:
Added GPL copyright text
vio/CMakeLists.txt:
Added GPL copyright text
zlib/CMakeLists.txt:
Added GPL copyright text
include/my_vle.h:
Changed copyright header formatting some
mysys/my_vle.c:
Changed copyright header formatting some
plugin/daemon_example/daemon_example.c:
Added "Copyright (C) 2006 MySQL AB" to GPL header
plugin/fulltext/plugin_example.c:
Added "Copyright (C) 2006 MySQL AB" to GPL header
plugin/Makefile.am:
Removed "MySQL Finland AB & TCX DataKonsult AB" from copyright header
Adjusted year(s) in copyright header
sql/authors.h:
Removed "MySQL Finland AB & TCX DataKonsult AB" from copyright header
Adjusted year(s) in copyright header
sql/contributors.h:
Removed "MySQL Finland AB & TCX DataKonsult AB" from copyright header
Adjusted year(s) in copyright header
sql/item_xmlfunc.cc:
Removed "MySQL Finland AB & TCX DataKonsult AB" from copyright header
Adjusted year(s) in copyright header
sql/partition_element.h:
Removed "MySQL Finland AB & TCX DataKonsult AB" from copyright header
Adjusted year(s) in copyright header
sql/partition_info.h:
Removed "MySQL Finland AB & TCX DataKonsult AB" from copyright header
Adjusted year(s) in copyright header
sql/rpl_injector.cc:
Removed "MySQL Finland AB & TCX DataKonsult AB" from copyright header
Adjusted year(s) in copyright header
sql/rpl_injector.h:
Removed "MySQL Finland AB & TCX DataKonsult AB" from copyright header
Adjusted year(s) in copyright header
sql/sql_binlog.cc:
Removed "MySQL Finland AB & TCX DataKonsult AB" from copyright header
Adjusted year(s) in copyright header
sql/sql_servers.h:
Removed "MySQL Finland AB & TCX DataKonsult AB" from copyright header
Adjusted year(s) in copyright header
storage/Makefile.am:
Removed "MySQL Finland AB & TCX DataKonsult AB" from copyright header
Adjusted year(s) in copyright header
storage/archive/Makefile.am:
Removed "MySQL Finland AB & TCX DataKonsult AB" from copyright header
Adjusted year(s) in copyright header
storage/blackhole/Makefile.am:
Removed "MySQL Finland AB & TCX DataKonsult AB" from copyright header
Adjusted year(s) in copyright header
storage/csv/Makefile.am:
Removed "MySQL Finland AB & TCX DataKonsult AB" from copyright header
Adjusted year(s) in copyright header
storage/example/Makefile.am:
Removed "MySQL Finland AB & TCX DataKonsult AB" from copyright header
Adjusted year(s) in copyright header
storage/federated/Makefile.am:
Removed "MySQL Finland AB & TCX DataKonsult AB" from copyright header
Adjusted year(s) in copyright header
win/Makefile.am:
Removed "MySQL Finland AB & TCX DataKonsult AB" from copyright header
Adjusted year(s) in copyright header
sql/rpl_utility.cc:
Changed copyright header formatting some
sql/rpl_utility.h:
Changed copyright header formatting some
unittest/mysys/base64-t.c:
Changed copyright header formatting some
include/my_uctype.h:
Added GPL copyright text
plugin/daemon_example/Makefile.am:
Added GPL copyright text
plugin/fulltext/Makefile.am:
Added GPL copyright text
scripts/make_win_bin_dist:
Added GPL copyright text
server-tools/instance-manager/user_management_commands.cc:
Added GPL copyright text
sql/sql_builtin.cc.in:
Added GPL copyright text
sql/sql_show.h:
Added GPL copyright text
storage/archive/archive_test.c:
Added GPL copyright text
storage/ndb/src/kernel/blocks/dbtup/test_varpage.cpp:
Added GPL copyright text
storage/ndb/src/kernel/blocks/diskpage.cpp:
Added GPL copyright text
storage/ndb/src/kernel/vm/Rope.cpp:
Added GPL copyright text
storage/ndb/src/mgmsrv/ParamInfo.hpp:
Added GPL copyright text
strings/uctypedump.c:
Added GPL copyright text
unittest/Makefile.am:
Added GPL copyright text
unittest/examples/Makefile.am:
Added GPL copyright text
unittest/examples/core-t.c:
Added GPL copyright text
unittest/examples/no_plan-t.c:
Added GPL copyright text
unittest/examples/simple-t.c:
Added GPL copyright text
unittest/examples/skip-t.c:
Added GPL copyright text
unittest/examples/skip_all-t.c:
Added GPL copyright text
unittest/examples/todo-t.c:
Added GPL copyright text
unittest/mysys/Makefile.am:
Added GPL copyright text
unittest/mytap/Makefile.am:
Added GPL copyright text
unittest/mytap/t/Makefile.am:
Added GPL copyright text
unittest/mytap/t/basic-t.c:
Added GPL copyright text
unittest/unit.pl:
Added GPL copyright text
win/build-vs71.bat:
Added GPL copyright text
win/build-vs8.bat:
Added GPL copyright text
win/configure.js:
Added GPL copyright text
Cut number 6. Move code from sql_show.cc to event_db_repository.cc
that more belongs to the latter.
sql/event_db_repository.cc:
move code that works with mysql.event from sql_show.cc to
event_db_repository.cc . Route through Event_db_repository's interface
which is proxied by class Events. The code relies on a function from
sql_show.cc which does the actual storage in the schema table. I think
it's better to leave the function there because the structure of
I_S.EVENTS is defined in sql_show.cc
sql/event_db_repository.h:
I_S / SHOW EVENTS handling hooks
sql/event_scheduler.cc:
use the pointer to db_repository which Event_scheduler already has
sql/events.cc:
Put a comment to get_instance
sql/events.h:
callback for I_S (sql_show.cc)
sql/sql_show.cc:
move code that belongs more to Event_db_repository than to here.
Use a callback of class Events. Only 1 function is left here, because
it copies data into the actual rows of I_S.EVENTS and belongs to this file.
sql/sql_show.h:
export this function will be called from event_db_repository.cc
